From: Linus Torvalds Date: Thu, 21 Jun 2018 21:21:34 +0000 (+0900) Subject: Merge tag 'nfs-for-4.18-2' of git://git.linux-nfs.org/projects/trondmy/linux-nfs X-Git-Tag: v4.18-rc2~22 X-Git-Url: http://git.samba.org/samba.git/?p=sfrench%2Fcifs-2.6.git;a=commitdiff_plain;h=27db64f65f1be2f2ee741a1bf20d8d13d62c167f;hp=7b0df92ac12148098391bf53f3494af17812f264 Merge tag 'nfs-for-4.18-2' of git://git.linux-nfs.org/projects/trondmy/linux-nfs Pull NFS client bugfixes from Trond Myklebust: "Hightlights include: - fix an rcu deadlock in nfs_delegation_find_inode() - fix NFSv4 deadlocks due to not freeing the session slot in layoutget - don't send layoutreturn if the layout is already invalid - prevent duplicate XID allocation - flexfiles: Don't tie up all the rpciod threads in resends" * tag 'nfs-for-4.18-2' of git://git.linux-nfs.org/projects/trondmy/linux-nfs: pNFS/flexfiles: Process writeback resends from nfsiod context as well pNFS/flexfiles: Don't tie up all the rpciod threads in resends sunrpc: Prevent duplicate XID allocation pNFS: Don't send layoutreturn if the layout is already invalid pNFS: Always free the session slot on error in nfs4_layoutget_handle_exception NFS: Fix an rcu deadlock in nfs_delegation_find_inode() --- diff --git a/.mailmap b/.mailmap index 7fa9d41fbdaf..29ddeb1bf015 100644 --- a/.mailmap +++ b/.mailmap @@ -186,6 +186,9 @@ Uwe Kleine-König Uwe Kleine-König Uwe Kleine-König Valdis Kletnieks +Vinod Koul +Vinod Koul +Vinod Koul Viresh Kumar Viresh Kumar Viresh Kumar diff --git a/Documentation/00-INDEX b/Documentation/00-INDEX index 708dc4c166e4..2754fe83f0d4 100644 --- a/Documentation/00-INDEX +++ b/Documentation/00-INDEX @@ -64,8 +64,6 @@ auxdisplay/ - misc. LCD driver documentation (cfag12864b, ks0108). backlight/ - directory with info on controlling backlights in flat panel displays -bcache.txt - - Block-layer cache on fast SSDs to improve slow (raid) I/O performance. block/ - info on the Block I/O (BIO) layer. blockdev/ @@ -78,18 +76,10 @@ bus-devices/ - directory with info on TI GPMC (General Purpose Memory Controller) bus-virt-phys-mapping.txt - how to access I/O mapped memory from within device drivers. -cachetlb.txt - - describes the cache/TLB flushing interfaces Linux uses. cdrom/ - directory with information on the CD-ROM drivers that Linux has. cgroup-v1/ - cgroups v1 features, including cpusets and memory controller. -cgroup-v2.txt - - cgroups v2 features, including cpusets and memory controller. -circular-buffers.txt - - how to make use of the existing circular buffer infrastructure -clk.txt - - info on the common clock framework cma/ - Continuous Memory Area (CMA) debugfs interface. conf.py diff --git a/Documentation/ABI/obsolete/sysfs-gpio b/Documentation/ABI/obsolete/sysfs-gpio index 32513dc2eec9..40d41ea1a3f5 100644 --- a/Documentation/ABI/obsolete/sysfs-gpio +++ b/Documentation/ABI/obsolete/sysfs-gpio @@ -11,7 +11,7 @@ Description: Kernel code may export it for complete or partial access. GPIOs are identified as they are inside the kernel, using integers in - the range 0..INT_MAX. See Documentation/gpio/gpio.txt for more information. + the range 0..INT_MAX. See Documentation/gpio for more information. /sys/class/gpio /export ... asks the kernel to export a GPIO to userspace diff --git a/Documentation/ABI/removed/sysfs-bus-nfit b/Documentation/ABI/removed/sysfs-bus-nfit new file mode 100644 index 000000000000..ae8c1ca53828 --- /dev/null +++ b/Documentation/ABI/removed/sysfs-bus-nfit @@ -0,0 +1,17 @@ +What: /sys/bus/nd/devices/regionX/nfit/ecc_unit_size +Date: Aug, 2017 +KernelVersion: v4.14 (Removed v4.18) +Contact: linux-nvdimm@lists.01.org +Description: + (RO) Size of a write request to a DIMM that will not incur a + read-modify-write cycle at the memory controller. + + When the nfit driver initializes it runs an ARS (Address Range + Scrub) operation across every pmem range. Part of that process + involves determining the ARS capabilities of a given address + range. One of the capabilities that is reported is the 'Clear + Uncorrectable Error Range Length Unit Size' (see: ACPI 6.2 + section 9.20.7.4 Function Index 1 - Query ARS Capabilities). + This property indicates the boundary at which the NVDIMM may + need to perform read-modify-write cycles to maintain ECC (Error + Correcting Code) blocks. diff --git a/Documentation/ABI/stable/sysfs-bus-vmbus b/Documentation/ABI/stable/sysfs-bus-vmbus index 0c9d9dcd2151..3eaffbb2d468 100644 --- a/Documentation/ABI/stable/sysfs-bus-vmbus +++ b/Documentation/ABI/stable/sysfs-bus-vmbus @@ -1,25 +1,25 @@ -What: /sys/bus/vmbus/devices/vmbus_*/id +What: /sys/bus/vmbus/devices//id Date: Jul 2009 KernelVersion: 2.6.31 Contact: K. Y. Srinivasan Description: The VMBus child_relid of the device's primary channel Users: tools/hv/lsvmbus -What: /sys/bus/vmbus/devices/vmbus_*/class_id +What: /sys/bus/vmbus/devices//class_id Date: Jul 2009 KernelVersion: 2.6.31 Contact: K. Y. Srinivasan Description: The VMBus interface type GUID of the device Users: tools/hv/lsvmbus -What: /sys/bus/vmbus/devices/vmbus_*/device_id +What: /sys/bus/vmbus/devices//device_id Date: Jul 2009 KernelVersion: 2.6.31 Contact: K. Y. Srinivasan Description: The VMBus interface instance GUID of the device Users: tools/hv/lsvmbus -What: /sys/bus/vmbus/devices/vmbus_*/channel_vp_mapping +What: /sys/bus/vmbus/devices//channel_vp_mapping Date: Jul 2015 KernelVersion: 4.2.0 Contact: K. Y. Srinivasan @@ -28,112 +28,112 @@ Description: The mapping of which primary/sub channels are bound to which Format: Users: tools/hv/lsvmbus -What: /sys/bus/vmbus/devices/vmbus_*/device +What: /sys/bus/vmbus/devices//device Date: Dec. 2015 KernelVersion: 4.5 Contact: K. Y. Srinivasan Description: The 16 bit device ID of the device Users: tools/hv/lsvmbus and user level RDMA libraries -What: /sys/bus/vmbus/devices/vmbus_*/vendor +What: /sys/bus/vmbus/devices//vendor Date: Dec. 2015 KernelVersion: 4.5 Contact: K. Y. Srinivasan Description: The 16 bit vendor ID of the device Users: tools/hv/lsvmbus and user level RDMA libraries -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN +What: /sys/bus/vmbus/devices//channels/ Date: September. 2017 KernelVersion: 4.14 Contact: Stephen Hemminger Description: Directory for per-channel information NN is the VMBUS relid associtated with the channel. -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/cpu +What: /sys/bus/vmbus/devices//channels//cpu Date: September. 2017 KernelVersion: 4.14 Contact: Stephen Hemminger Description: VCPU (sub)channel is affinitized to Users: tools/hv/lsvmbus and other debugging tools -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/cpu +What: /sys/bus/vmbus/devices//channels//cpu Date: September. 2017 KernelVersion: 4.14 Contact: Stephen Hemminger Description: VCPU (sub)channel is affinitized to Users: tools/hv/lsvmbus and other debugging tools -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/in_mask +What: /sys/bus/vmbus/devices//channels//in_mask Date: September. 2017 KernelVersion: 4.14 Contact: Stephen Hemminger Description: Host to guest channel interrupt mask Users: Debugging tools -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/latency +What: /sys/bus/vmbus/devices//channels//latency Date: September. 2017 KernelVersion: 4.14 Contact: Stephen Hemminger Description: Channel signaling latency Users: Debugging tools -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/out_mask +What: /sys/bus/vmbus/devices//channels//out_mask Date: September. 2017 KernelVersion: 4.14 Contact: Stephen Hemminger Description: Guest to host channel interrupt mask Users: Debugging tools -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/pending +What: /sys/bus/vmbus/devices//channels//pending Date: September. 2017 KernelVersion: 4.14 Contact: Stephen Hemminger Description: Channel interrupt pending state Users: Debugging tools -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/read_avail +What: /sys/bus/vmbus/devices//channels//read_avail Date: September. 2017 KernelVersion: 4.14 Contact: Stephen Hemminger Description: Bytes available to read Users: Debugging tools -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/write_avail +What: /sys/bus/vmbus/devices//channels//write_avail Date: September. 2017 KernelVersion: 4.14 Contact: Stephen Hemminger Description: Bytes available to write Users: Debugging tools -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/events +What: /sys/bus/vmbus/devices//channels//events Date: September. 2017 KernelVersion: 4.14 Contact: Stephen Hemminger Description: Number of times we have signaled the host Users: Debugging tools -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/interrupts +What: /sys/bus/vmbus/devices//channels//interrupts Date: September. 2017 KernelVersion: 4.14 Contact: Stephen Hemminger Description: Number of times we have taken an interrupt (incoming) Users: Debugging tools -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/subchannel_id +What: /sys/bus/vmbus/devices//channels//subchannel_id Date: January. 2018 KernelVersion: 4.16 Contact: Stephen Hemminger Description: Subchannel ID associated with VMBUS channel Users: Debugging tools and userspace drivers -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/monitor_id +What: /sys/bus/vmbus/devices//channels//monitor_id Date: January. 2018 KernelVersion: 4.16 Contact: Stephen Hemminger Description: Monitor bit associated with channel Users: Debugging tools and userspace drivers -What: /sys/bus/vmbus/devices/vmbus_*/channels/NN/ring +What: /sys/bus/vmbus/devices//channels//ring Date: January. 2018 KernelVersion: 4.16 Contact: Stephen Hemminger diff --git a/Documentation/ABI/stable/sysfs-devices-node b/Documentation/ABI/stable/sysfs-devices-node index 5b2d0f08867c..3e90e1f3bf0a 100644 --- a/Documentation/ABI/stable/sysfs-devices-node +++ b/Documentation/ABI/stable/sysfs-devices-node @@ -90,4 +90,4 @@ Date: December 2009 Contact: Lee Schermerhorn Description: The node's huge page size control/query attributes. - See Documentation/vm/hugetlbpage.txt \ No newline at end of file + See Documentation/admin-guide/mm/hugetlbpage.rst \ No newline at end of file diff --git a/Documentation/ABI/testing/evm b/Documentation/ABI/testing/evm index d12cb2eae9ee..201d10319fa1 100644 --- a/Documentation/ABI/testing/evm +++ b/Documentation/ABI/testing/evm @@ -57,3 +57,16 @@ Description: dracut (via 97masterkey and 98integrity) and systemd (via core/ima-setup) have support for loading keys at boot time. + +What: security/integrity/evm/evm_xattrs +Date: April 2018 +Contact: Matthew Garrett +Description: + Shows the set of extended attributes used to calculate or + validate the EVM signature, and allows additional attributes + to be added at runtime. Any signatures generated after + additional attributes are added (and on files posessing those + additional attributes) will only be valid if the same + additional attributes are configured on system boot. Writing + a single period (.) will lock the xattr list from any further + modification. diff --git a/Documentation/ABI/testing/ima_policy b/Documentation/ABI/testing/ima_policy index b8465e00ba5f..74c6702de74e 100644 --- a/Documentation/ABI/testing/ima_policy +++ b/Documentation/ABI/testing/ima_policy @@ -21,7 +21,7 @@ Description: audit | hash | dont_hash condition:= base | lsm [option] base: [[func=] [mask=] [fsmagic=] [fsuuid=] [uid=] - [euid=] [fowner=]] + [euid=] [fowner=] [fsname=]] lsm: [[subj_user=] [subj_role=] [subj_type=] [obj_user=] [obj_role=] [obj_type=]] option: [[appraise_type=]] [permit_directio] diff --git a/Documentation/ABI/testing/sysfs-bus-iio b/Documentation/ABI/testing/sysfs-bus-iio index 6a5f34b4d5b9..731146c3b138 100644 --- a/Documentation/ABI/testing/sysfs-bus-iio +++ b/Documentation/ABI/testing/sysfs-bus-iio @@ -190,6 +190,13 @@ Description: but should match other such assignments on device). Units after application of scale and offset are m/s^2. +What: /sys/bus/iio/devices/iio:deviceX/in_angl_raw +KernelVersion: 4.17 +Contact: linux-iio@vger.kernel.org +Description: + Angle of rotation. Units after application of scale and offset + are radians. + What: /sys/bus/iio/devices/iio:deviceX/in_anglvel_x_raw What: /sys/bus/iio/devices/iio:deviceX/in_anglvel_y_raw What: /sys/bus/iio/devices/iio:deviceX/in_anglvel_z_raw @@ -297,6 +304,7 @@ What: /sys/bus/iio/devices/iio:deviceX/in_pressure_offset What: /sys/bus/iio/devices/iio:deviceX/in_humidityrelative_offset What: /sys/bus/iio/devices/iio:deviceX/in_magn_offset What: /sys/bus/iio/devices/iio:deviceX/in_rot_offset +What: /sys/bus/iio/devices/iio:deviceX/in_angl_offset KernelVersion: 2.6.35 Contact: linux-iio@vger.kernel.org Description: @@ -350,6 +358,7 @@ What: /sys/bus/iio/devices/iio:deviceX/in_humidityrelative_scale What: /sys/bus/iio/devices/iio:deviceX/in_velocity_sqrt(x^2+y^2+z^2)_scale What: /sys/bus/iio/devices/iio:deviceX/in_illuminance_scale What: /sys/bus/iio/devices/iio:deviceX/in_countY_scale +What: /sys/bus/iio/devices/iio:deviceX/in_angl_scale KernelVersion: 2.6.35 Contact: linux-iio@vger.kernel.org Description: diff --git a/Documentation/ABI/testing/sysfs-bus-nfit b/Documentation/ABI/testing/sysfs-bus-nfit index 619eb8ca0f99..a1cb44dcb908 100644 --- a/Documentation/ABI/testing/sysfs-bus-nfit +++ b/Documentation/ABI/testing/sysfs-bus-nfit @@ -212,22 +212,3 @@ Description: range. Used by NVDIMM Region Mapping Structure to uniquely refer to this structure. Value of 0 is reserved and not used as an index. - - -What: /sys/bus/nd/devices/regionX/nfit/ecc_unit_size -Date: Aug, 2017 -KernelVersion: v4.14 -Contact: linux-nvdimm@lists.01.org -Description: - (RO) Size of a write request to a DIMM that will not incur a - read-modify-write cycle at the memory controller. - - When the nfit driver initializes it runs an ARS (Address Range - Scrub) operation across every pmem range. Part of that process - involves determining the ARS capabilities of a given address - range. One of the capabilities that is reported is the 'Clear - Uncorrectable Error Range Length Unit Size' (see: ACPI 6.2 - section 9.20.7.4 Function Index 1 - Query ARS Capabilities). - This property indicates the boundary at which the NVDIMM may - need to perform read-modify-write cycles to maintain ECC (Error - Correcting Code) blocks. diff --git a/Documentation/ABI/testing/sysfs-bus-rpmsg b/Documentation/ABI/testing/sysfs-bus-rpmsg index 189e419a5a2d..990fcc420935 100644 --- a/Documentation/ABI/testing/sysfs-bus-rpmsg +++ b/Documentation/ABI/testing/sysfs-bus-rpmsg @@ -73,3 +73,23 @@ Description: This sysfs entry tells us whether the channel is a local server channel that is announced (values are either true or false). + +What: /sys/bus/rpmsg/devices/.../driver_override +Date: April 2018 +KernelVersion: 4.18 +Contact: Bjorn Andersson +Description: + Every rpmsg device is a communication channel with a remote + processor. Channels are identified by a textual name (see + /sys/bus/rpmsg/devices/.../name above) and have a local + ("source") rpmsg address, and remote ("destination") rpmsg + address. + + The listening entity (or client) which communicates with a + remote processor is referred as rpmsg driver. The rpmsg device + and rpmsg driver are matched based on rpmsg device name and + rpmsg driver ID table. + + This sysfs entry allows the rpmsg driver for a rpmsg device + to be specified which will override standard OF, ID table + and name matching. diff --git a/Documentation/ABI/testing/sysfs-bus-usb b/Documentation/ABI/testing/sysfs-bus-usb index c702c78f24d8..08d456e07b53 100644 --- a/Documentation/ABI/testing/sysfs-bus-usb +++ b/Documentation/ABI/testing/sysfs-bus-usb @@ -189,6 +189,28 @@ Description: The file will read "hotplug", "wired" and "not used" if the information is available, and "unknown" otherwise. +What: /sys/bus/usb/devices/.../(hub interface)/portX/quirks +Date: May 2018 +Contact: Nicolas Boichat +Description: + In some cases, we care about time-to-active for devices + connected on a specific port (e.g. non-standard USB port like + pogo pins), where the device to be connected is known in + advance, and behaves well according to the specification. + This attribute is a bit-field that controls the behavior of + a specific port: + - Bit 0 of this field selects the "old" enumeration scheme, + as it is considerably faster (it only causes one USB reset + instead of 2). + The old enumeration scheme can also be selected globally + using /sys/module/usbcore/parameters/old_scheme_first, but + it is often not desirable as the new scheme was introduced to + increase compatibility with more devices. + - Bit 1 reduces TRSTRCY to the 10 ms that are required by the + USB 2.0 specification, instead of the 50 ms that are normally + used to help make enumeration work better on some high speed + devices. + What: /sys/bus/usb/devices/.../(hub interface)/portX/over_current_count Date: February 2018 Contact: Richard Leitner @@ -236,3 +258,21 @@ Description: Supported values are 0 - 15. More information on how besl values map to microseconds can be found in USB 2.0 ECN Errata for Link Power Management, section 4.10) + +What: /sys/bus/usb/devices/.../rx_lanes +Date: March 2018 +Contact: Mathias Nyman +Description: + Number of rx lanes the device is using. + USB 3.2 adds Dual-lane support, 2 rx and 2 tx lanes over Type-C. + Inter-Chip SSIC devices support asymmetric lanes up to 4 lanes per + direction. Devices before USB 3.2 are single lane (rx_lanes = 1) + +What: /sys/bus/usb/devices/.../tx_lanes +Date: March 2018 +Contact: Mathias Nyman +Description: + Number of tx lanes the device is using. + USB 3.2 adds Dual-lane support, 2 rx and 2 tx -lanes over Type-C. + Inter-Chip SSIC devices support asymmetric lanes up to 4 lanes per + direction. Devices before USB 3.2 are single lane (tx_lanes = 1) diff --git a/Documentation/ABI/testing/sysfs-class-cxl b/Documentation/ABI/testing/sysfs-class-cxl index 8e69345c37cc..bbbabffc682a 100644 --- a/Documentation/ABI/testing/sysfs-class-cxl +++ b/Documentation/ABI/testing/sysfs-class-cxl @@ -69,7 +69,9 @@ Date: September 2014 Contact: linuxppc-dev@lists.ozlabs.org Description: read/write Set the mode for prefaulting in segments into the segment table - when performing the START_WORK ioctl. Possible values: + when performing the START_WORK ioctl. Only applicable when + running under hashed page table mmu. + Possible values: none: No prefaulting (default) work_element_descriptor: Treat the work element descriptor as an effective address and diff --git a/Documentation/ABI/testing/sysfs-class-mtd b/Documentation/ABI/testing/sysfs-class-mtd index f34e592301d1..3bc7c0a95c92 100644 --- a/Documentation/ABI/testing/sysfs-class-mtd +++ b/Documentation/ABI/testing/sysfs-class-mtd @@ -232,3 +232,11 @@ Description: of the parent (another partition or a flash device) in bytes. This attribute is absent on flash devices, so it can be used to distinguish them from partitions. + +What: /sys/class/mtd/mtdX/oobavail +Date: April 2018 +KernelVersion: 4.16 +Contact: linux-mtd@lists.infradead.org +Description: + Number of bytes available for a client to place data into + the out of band area. diff --git a/Documentation/ABI/testing/sysfs-class-power b/Documentation/ABI/testing/sysfs-class-power index f85ce9e327b9..5e23e22dce1b 100644 --- a/Documentation/ABI/testing/sysfs-class-power +++ b/Documentation/ABI/testing/sysfs-class-power @@ -1,3 +1,458 @@ +===== General Properties ===== + +What: /sys/class/power_supply//manufacturer +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports the name of the device manufacturer. + + Access: Read + Valid values: Represented as string + +What: /sys/class/power_supply//model_name +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports the name of the device model. + + Access: Read + Valid values: Represented as string + +What: /sys/class/power_supply//serial_number +Date: January 2008 +Contact: linux-pm@vger.kernel.org +Description: + Reports the serial number of the device. + + Access: Read + Valid values: Represented as string + +What: /sys/class/power_supply//type +Date: May 2010 +Contact: linux-pm@vger.kernel.org +Description: + Describes the main type of the supply. + + Access: Read + Valid values: "Battery", "UPS", "Mains", "USB" + +===== Battery Properties ===== + +What: /sys/class/power_supply//capacity +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Fine grain representation of battery capacity. + Access: Read + Valid values: 0 - 100 (percent) + +What: /sys/class/power_supply//capacity_alert_max +Date: July 2012 +Contact: linux-pm@vger.kernel.org +Description: + Maximum battery capacity trip-wire value where the supply will + notify user-space of the event. This is normally used for the + battery discharging scenario where user-space needs to know the + battery has dropped to an upper level so it can take + appropriate action (e.g. warning user that battery level is + low). + + Access: Read, Write + Valid values: 0 - 100 (percent) + +What: /sys/class/power_supply//capacity_alert_min +Date: July 2012 +Contact: linux-pm@vger.kernel.org +Description: + Minimum battery capacity trip-wire value where the supply will + notify user-space of the event. This is normally used for the + battery discharging scenario where user-space needs to know the + battery has dropped to a lower level so it can take + appropriate action (e.g. warning user that battery level is + critically low). + + Access: Read, Write + Valid values: 0 - 100 (percent) + +What: /sys/class/power_supply//capacity_level +Date: June 2009 +Contact: linux-pm@vger.kernel.org +Description: + Coarse representation of battery capacity. + + Access: Read + Valid values: "Unknown", "Critical", "Low", "Normal", "High", + "Full" + +What: /sys/class/power_supply//current_avg +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports an average IBAT current reading for the battery, over a + fixed period. Normally devices will provide a fixed interval in + which they average readings to smooth out the reported value. + + Access: Read + Valid values: Represented in microamps + +What: /sys/class/power_supply//current_max +Date: October 2010 +Contact: linux-pm@vger.kernel.org +Description: + Reports the maximum IBAT current allowed into the battery. + + Access: Read + Valid values: Represented in microamps + +What: /sys/class/power_supply//current_now +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports an instant, single IBAT current reading for the battery. + This value is not averaged/smoothed. + + Access: Read + Valid values: Represented in microamps + +What: /sys/class/power_supply//charge_type +Date: July 2009 +Contact: linux-pm@vger.kernel.org +Description: + Represents the type of charging currently being applied to the + battery. + + Access: Read + Valid values: "Unknown", "N/A", "Trickle", "Fast" + +What: /sys/class/power_supply//charge_term_current +Date: July 2014 +Contact: linux-pm@vger.kernel.org +Description: + Reports the charging current value which is used to determine + when the battery is considered full and charging should end. + + Access: Read + Valid values: Represented in microamps + +What: /sys/class/power_supply//health +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports the health of the battery or battery side of charger + functionality. + + Access: Read + Valid values: "Unknown", "Good", "Overheat", "Dead", + "Over voltage", "Unspecified failure", "Cold", + "Watchdog timer expire", "Safety timer expire" + +What: /sys/class/power_supply//precharge_current +Date: June 2017 +Contact: linux-pm@vger.kernel.org +Description: + Reports the charging current applied during pre-charging phase + for a battery charge cycle. + + Access: Read + Valid values: Represented in microamps + +What: /sys/class/power_supply//present +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports whether a battery is present or not in the system. + + Access: Read + Valid values: + 0: Absent + 1: Present + +What: /sys/class/power_supply//status +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Represents the charging status of the battery. Normally this + is read-only reporting although for some supplies this can be + used to enable/disable charging to the battery. + + Access: Read, Write + Valid values: "Unknown", "Charging", "Discharging", + "Not charging", "Full" + +What: /sys/class/power_supply//technology +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Describes the battery technology supported by the supply. + + Access: Read + Valid values: "Unknown", "NiMH", "Li-ion", "Li-poly", "LiFe", + "NiCd", "LiMn" + +What: /sys/class/power_supply//temp +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports the current TBAT battery temperature reading. + + Access: Read + Valid values: Represented in 1/10 Degrees Celsius + +What: /sys/class/power_supply//temp_alert_max +Date: July 2012 +Contact: linux-pm@vger.kernel.org +Description: + Maximum TBAT temperature trip-wire value where the supply will + notify user-space of the event. This is normally used for the + battery charging scenario where user-space needs to know the + battery temperature has crossed an upper threshold so it can + take appropriate action (e.g. warning user that battery level is + critically high, and charging has stopped). + + Access: Read + Valid values: Represented in 1/10 Degrees Celsius + +What: /sys/class/power_supply//temp_alert_min +Date: July 2012 +Contact: linux-pm@vger.kernel.org +Description: + Minimum TBAT temperature trip-wire value where the supply will + notify user-space of the event. This is normally used for the + battery charging scenario where user-space needs to know the + battery temperature has crossed a lower threshold so it can take + appropriate action (e.g. warning user that battery level is + high, and charging current has been reduced accordingly to + remedy the situation). + + Access: Read + Valid values: Represented in 1/10 Degrees Celsius + +What: /sys/class/power_supply//temp_max +Date: July 2014 +Contact: linux-pm@vger.kernel.org +Description: + Reports the maximum allowed TBAT battery temperature for + charging. + + Access: Read + Valid values: Represented in 1/10 Degrees Celsius + +What: /sys/class/power_supply//temp_min +Date: July 2014 +Contact: linux-pm@vger.kernel.org +Description: + Reports the minimum allowed TBAT battery temperature for + charging. + + Access: Read + Valid values: Represented in 1/10 Degrees Celsius + +What: /sys/class/power_supply//voltage_avg, +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports an average VBAT voltage reading for the battery, over a + fixed period. Normally devices will provide a fixed interval in + which they average readings to smooth out the reported value. + + Access: Read + Valid values: Represented in microvolts + +What: /sys/class/power_supply//voltage_max, +Date: January 2008 +Contact: linux-pm@vger.kernel.org +Description: + Reports the maximum safe VBAT voltage permitted for the battery, + during charging. + + Access: Read + Valid values: Represented in microvolts + +What: /sys/class/power_supply//voltage_min, +Date: January 2008 +Contact: linux-pm@vger.kernel.org +Description: + Reports the minimum safe VBAT voltage permitted for the battery, + during discharging. + + Access: Read + Valid values: Represented in microvolts + +What: /sys/class/power_supply//voltage_now, +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports an instant, single VBAT voltage reading for the battery. + This value is not averaged/smoothed. + + Access: Read + Valid values: Represented in microvolts + +===== USB Properties ===== + +What: /sys/class/power_supply//current_avg +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports an average IBUS current reading over a fixed period. + Normally devices will provide a fixed interval in which they + average readings to smooth out the reported value. + + Access: Read + Valid values: Represented in microamps + + +What: /sys/class/power_supply//current_max +Date: October 2010 +Contact: linux-pm@vger.kernel.org +Description: + Reports the maximum IBUS current the supply can support. + + Access: Read + Valid values: Represented in microamps + +What: /sys/class/power_supply//current_now +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports the IBUS current supplied now. This value is generally + read-only reporting, unless the 'online' state of the supply + is set to be programmable, in which case this value can be set + within the reported min/max range. + + Access: Read, Write + Valid values: Represented in microamps + +What: /sys/class/power_supply//input_current_limit +Date: July 2014 +Contact: linux-pm@vger.kernel.org +Description: + Details the incoming IBUS current limit currently set in the + supply. Normally this is configured based on the type of + connection made (e.g. A configured SDP should output a maximum + of 500mA so the input current limit is set to the same value). + + Access: Read, Write + Valid values: Represented in microamps + +What: /sys/class/power_supply//online, +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Indicates if VBUS is present for the supply. When the supply is + online, and the supply allows it, then it's possible to switch + between online states (e.g. Fixed -> Programmable for a PD_PPS + USB supply so voltage and current can be controlled). + + Access: Read, Write + Valid values: + 0: Offline + 1: Online Fixed - Fixed Voltage Supply + 2: Online Programmable - Programmable Voltage Supply + +What: /sys/class/power_supply//temp +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports the current supply temperature reading. This would + normally be the internal temperature of the device itself (e.g + TJUNC temperature of an IC) + + Access: Read + Valid values: Represented in 1/10 Degrees Celsius + +What: /sys/class/power_supply//temp_alert_max +Date: July 2012 +Contact: linux-pm@vger.kernel.org +Description: + Maximum supply temperature trip-wire value where the supply will + notify user-space of the event. This is normally used for the + charging scenario where user-space needs to know the supply + temperature has crossed an upper threshold so it can take + appropriate action (e.g. warning user that the supply + temperature is critically high, and charging has stopped to + remedy the situation). + + Access: Read + Valid values: Represented in 1/10 Degrees Celsius + +What: /sys/class/power_supply//temp_alert_min +Date: July 2012 +Contact: linux-pm@vger.kernel.org +Description: + Minimum supply temperature trip-wire value where the supply will + notify user-space of the event. This is normally used for the + charging scenario where user-space needs to know the supply + temperature has crossed a lower threshold so it can take + appropriate action (e.g. warning user that the supply + temperature is high, and charging current has been reduced + accordingly to remedy the situation). + + Access: Read + Valid values: Represented in 1/10 Degrees Celsius + +What: /sys/class/power_supply//temp_max +Date: July 2014 +Contact: linux-pm@vger.kernel.org +Description: + Reports the maximum allowed supply temperature for operation. + + Access: Read + Valid values: Represented in 1/10 Degrees Celsius + +What: /sys/class/power_supply//temp_min +Date: July 2014 +Contact: linux-pm@vger.kernel.org +Description: + Reports the mainimum allowed supply temperature for operation. + + Access: Read + Valid values: Represented in 1/10 Degrees Celsius + +What: /sys/class/power_supply//usb_type +Date: March 2018 +Contact: linux-pm@vger.kernel.org +Description: + Reports what type of USB connection is currently active for + the supply, for example it can show if USB-PD capable source + is attached. + + Access: Read-Only + Valid values: "Unknown", "SDP", "DCP", "CDP", "ACA", "C", "PD", + "PD_DRP", "PD_PPS", "BrickID" + +What: /sys/class/power_supply//voltage_max +Date: January 2008 +Contact: linux-pm@vger.kernel.org +Description: + Reports the maximum VBUS voltage the supply can support. + + Access: Read + Valid values: Represented in microvolts + +What: /sys/class/power_supply//voltage_min +Date: January 2008 +Contact: linux-pm@vger.kernel.org +Description: + Reports the minimum VBUS voltage the supply can support. + + Access: Read + Valid values: Represented in microvolts + +What: /sys/class/power_supply//voltage_now +Date: May 2007 +Contact: linux-pm@vger.kernel.org +Description: + Reports the VBUS voltage supplied now. This value is generally + read-only reporting, unless the 'online' state of the supply + is set to be programmable, in which case this value can be set + within the reported min/max range. + + Access: Read, Write + Valid values: Represented in microvolts + +===== Device Specific Properties ===== + What: /sys/class/power/ds2760-battery.*/charge_now Date: May 2010 KernelVersion: 2.6.35 diff --git a/Documentation/ABI/testing/sysfs-class-rc b/Documentation/ABI/testing/sysfs-class-rc index 8be1fd3760e0..6c0d6c8cb911 100644 --- a/Documentation/ABI/testing/sysfs-class-rc +++ b/Documentation/ABI/testing/sysfs-class-rc @@ -1,7 +1,7 @@ What: /sys/class/rc/ Date: Apr 2010 KernelVersion: 2.6.35 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab Description: The rc/ class sub-directory belongs to the Remote Controller core and provides a sysfs interface for configuring infrared @@ -10,7 +10,7 @@ Description: What: /sys/class/rc/rcN/ Date: Apr 2010 KernelVersion: 2.6.35 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab Description: A /sys/class/rc/rcN directory is created for each remote control receiver device where N is the number of the receiver. @@ -18,7 +18,7 @@ Description: What: /sys/class/rc/rcN/protocols Date: Jun 2010 KernelVersion: 2.6.36 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab Description: Reading this file returns a list of available protocols, something like: @@ -36,7 +36,7 @@ Description: What: /sys/class/rc/rcN/filter Date: Jan 2014 KernelVersion: 3.15 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab Description: Sets the scancode filter expected value. Use in combination with /sys/class/rc/rcN/filter_mask to set the @@ -49,7 +49,7 @@ Description: What: /sys/class/rc/rcN/filter_mask Date: Jan 2014 KernelVersion: 3.15 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab Description: Sets the scancode filter mask of bits to compare. Use in combination with /sys/class/rc/rcN/filter to set the bits @@ -64,7 +64,7 @@ Description: What: /sys/class/rc/rcN/wakeup_protocols Date: Feb 2017 KernelVersion: 4.11 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab Description: Reading this file returns a list of available protocols to use for the wakeup filter, something like: @@ -83,7 +83,7 @@ Description: What: /sys/class/rc/rcN/wakeup_filter Date: Jan 2014 KernelVersion: 3.15 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab Description: Sets the scancode wakeup filter expected value. Use in combination with /sys/class/rc/rcN/wakeup_filter_mask to @@ -98,7 +98,7 @@ Description: What: /sys/class/rc/rcN/wakeup_filter_mask Date: Jan 2014 KernelVersion: 3.15 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab Description: Sets the scancode wakeup filter mask of bits to compare. Use in combination with /sys/class/rc/rcN/wakeup_filter to set diff --git a/Documentation/ABI/testing/sysfs-class-rc-nuvoton b/Documentation/ABI/testing/sysfs-class-rc-nuvoton index 905bcdeedef2..d3abe45f8690 100644 --- a/Documentation/ABI/testing/sysfs-class-rc-nuvoton +++ b/Documentation/ABI/testing/sysfs-class-rc-nuvoton @@ -1,7 +1,7 @@ What: /sys/class/rc/rcN/wakeup_data Date: Mar 2016 KernelVersion: 4.6 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab Description: Reading this file returns the stored CIR wakeup sequence. It starts with a pulse, followed by a space, pulse etc. diff --git a/Documentation/ABI/testing/sysfs-devices-edac b/Documentation/ABI/testing/sysfs-devices-edac index 46ff929fd52a..256a9e990c0b 100644 --- a/Documentation/ABI/testing/sysfs-devices-edac +++ b/Documentation/ABI/testing/sysfs-devices-edac @@ -77,7 +77,7 @@ Description: Read/Write attribute file that controls memory scrubbing. What: /sys/devices/system/edac/mc/mc*/max_location Date: April 2012 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab linux-edac@vger.kernel.org Description: This attribute file displays the information about the last available memory slot in this memory controller. It is used by @@ -85,7 +85,7 @@ Description: This attribute file displays the information about the last What: /sys/devices/system/edac/mc/mc*/(dimm|rank)*/size Date: April 2012 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab linux-edac@vger.kernel.org Description: This attribute file will display the size of dimm or rank. For dimm*/size, this is the size, in MB of the DIMM memory @@ -96,14 +96,14 @@ Description: This attribute file will display the size of dimm or rank. What: /sys/devices/system/edac/mc/mc*/(dimm|rank)*/dimm_dev_type Date: April 2012 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab linux-edac@vger.kernel.org Description: This attribute file will display what type of DRAM device is being utilized on this DIMM (x1, x2, x4, x8, ...). What: /sys/devices/system/edac/mc/mc*/(dimm|rank)*/dimm_edac_mode Date: April 2012 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab linux-edac@vger.kernel.org Description: This attribute file will display what type of Error detection and correction is being utilized. For example: S4ECD4ED would @@ -111,7 +111,7 @@ Description: This attribute file will display what type of Error detection What: /sys/devices/system/edac/mc/mc*/(dimm|rank)*/dimm_label Date: April 2012 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab linux-edac@vger.kernel.org Description: This control file allows this DIMM to have a label assigned to it. With this label in the module, when errors occur @@ -126,14 +126,14 @@ Description: This control file allows this DIMM to have a label assigned What: /sys/devices/system/edac/mc/mc*/(dimm|rank)*/dimm_location Date: April 2012 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab linux-edac@vger.kernel.org Description: This attribute file will display the location (csrow/channel, branch/channel/slot or channel/slot) of the dimm or rank. What: /sys/devices/system/edac/mc/mc*/(dimm|rank)*/dimm_mem_type Date: April 2012 -Contact: Mauro Carvalho Chehab +Contact: Mauro Carvalho Chehab linux-edac@vger.kernel.org Description: This attribute file will display what type of memory is currently on this csrow. Normally, either buffered or diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu index bd4975e132d3..9c5e7732d249 100644 --- a/Documentation/ABI/testing/sysfs-devices-system-cpu +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu @@ -238,9 +238,6 @@ Description: Discover and change clock speed of CPUs See files in Documentation/cpu-freq/ for more information. - In particular, read Documentation/cpu-freq/user-guide.txt - to learn how to control the knobs. - What: /sys/devices/system/cpu/cpu#/cpufreq/freqdomain_cpus Date: June 2013 diff --git a/Documentation/ABI/testing/sysfs-fs-f2fs b/Documentation/ABI/testing/sysfs-fs-f2fs index 540553c933b6..9b0123388f18 100644 --- a/Documentation/ABI/testing/sysfs-fs-f2fs +++ b/Documentation/ABI/testing/sysfs-fs-f2fs @@ -101,6 +101,7 @@ Date: February 2015 Contact: "Jaegeuk Kim" Description: Controls the trimming rate in batch mode. + What: /sys/fs/f2fs//cp_interval Date: October 2015 @@ -140,7 +141,7 @@ Contact: "Shuoran Liu" Description: Shows total written kbytes issued to disk. -What: /sys/fs/f2fs//feature +What: /sys/fs/f2fs//features Date: July 2017 Contact: "Jaegeuk Kim" Description: diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-hugepages b/Documentation/ABI/testing/sysfs-kernel-mm-hugepages index e21c00571cf4..fdaa2162fae1 100644 --- a/Documentation/ABI/testing/sysfs-kernel-mm-hugepages +++ b/Documentation/ABI/testing/sysfs-kernel-mm-hugepages @@ -12,4 +12,4 @@ Description: free_hugepages surplus_hugepages resv_hugepages - See Documentation/vm/hugetlbpage.txt for details. + See Documentation/admin-guide/mm/hugetlbpage.rst for details. diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-ksm b/Documentation/ABI/testing/sysfs-kernel-mm-ksm index 73e653ee2481..dfc13244cda3 100644 --- a/Documentation/ABI/testing/sysfs-kernel-mm-ksm +++ b/Documentation/ABI/testing/sysfs-kernel-mm-ksm @@ -40,7 +40,7 @@ Description: Kernel Samepage Merging daemon sysfs interface sleep_millisecs: how many milliseconds ksm should sleep between scans. - See Documentation/vm/ksm.txt for more information. + See Documentation/vm/ksm.rst for more information. What: /sys/kernel/mm/ksm/merge_across_nodes Date: January 2013 diff --git a/Documentation/ABI/testing/sysfs-kernel-slab b/Documentation/ABI/testing/sysfs-kernel-slab index 2cc0a72b64be..29601d93a1c2 100644 --- a/Documentation/ABI/testing/sysfs-kernel-slab +++ b/Documentation/ABI/testing/sysfs-kernel-slab @@ -37,7 +37,7 @@ Description: The alloc_calls file is read-only and lists the kernel code locations from which allocations for this cache were performed. The alloc_calls file only contains information if debugging is - enabled for that cache (see Documentation/vm/slub.txt). + enabled for that cache (see Documentation/vm/slub.rst). What: /sys/kernel/slab/cache/alloc_fastpath Date: February 2008 @@ -219,7 +219,7 @@ Contact: Pekka Enberg , Description: The free_calls file is read-only and lists the locations of object frees if slab debugging is enabled (see - Documentation/vm/slub.txt). + Documentation/vm/slub.rst). What: /sys/kernel/slab/cache/free_fastpath Date: February 2008 diff --git a/Documentation/ABI/testing/sysfs-platform-ideapad-laptop b/Documentation/ABI/testing/sysfs-platform-ideapad-laptop index 597a2f3d1efc..1b31be3f996a 100644 --- a/Documentation/ABI/testing/sysfs-platform-ideapad-laptop +++ b/Documentation/ABI/testing/sysfs-platform-ideapad-laptop @@ -25,3 +25,16 @@ Description: Control touchpad mode. * 1 -> Switched On * 0 -> Switched Off + +What: /sys/bus/pci/devices///VPC2004:00/fn_lock +Date: May 2018 +KernelVersion: 4.18 +Contact: "Oleg Keri " +Description: + Control fn-lock mode. + * 1 -> Switched On + * 0 -> Switched Off + + For example: + # echo "0" > \ + /sys/bus/pci/devices/0000:00:1f.0/PNP0C09:00/VPC2004:00/fn_lock diff --git a/Documentation/PCI/pci-error-recovery.txt b/Documentation/PCI/pci-error-recovery.txt index 0b6bb3ef449e..688b69121e82 100644 --- a/Documentation/PCI/pci-error-recovery.txt +++ b/Documentation/PCI/pci-error-recovery.txt @@ -110,7 +110,7 @@ The actual steps taken by a platform to recover from a PCI error event will be platform-dependent, but will follow the general sequence described below. -STEP 0: Error Event +STEP 0: Error Event: ERR_NONFATAL ------------------- A PCI bus error is detected by the PCI hardware. On powerpc, the slot is isolated, in that all I/O is blocked: all reads return 0xffffffff, @@ -228,13 +228,7 @@ proceeds to either STEP3 (Link Reset) or to STEP 5 (Resume Operations). If any driver returned PCI_ERS_RESULT_NEED_RESET, then the platform proceeds to STEP 4 (Slot Reset) -STEP 3: Link Reset ------------------- -The platform resets the link. This is a PCI-Express specific step -and is done whenever a fatal error has been detected that can be -"solved" by resetting the link. - -STEP 4: Slot Reset +STEP 3: Slot Reset ------------------ In response to a return value of PCI_ERS_RESULT_NEED_RESET, the @@ -320,7 +314,7 @@ Failure). >>> However, it probably should. -STEP 5: Resume Operations +STEP 4: Resume Operations ------------------------- The platform will call the resume() callback on all affected device drivers if all drivers on the segment have returned @@ -332,7 +326,7 @@ a result code. At this point, if a new error happens, the platform will restart a new error recovery sequence. -STEP 6: Permanent Failure +STEP 5: Permanent Failure ------------------------- A "permanent failure" has occurred, and the platform cannot recover the device. The platform will call error_detected() with a @@ -355,6 +349,27 @@ errors. See the discussion in powerpc/eeh-pci-error-recovery.txt for additional detail on real-life experience of the causes of software errors. +STEP 0: Error Event: ERR_FATAL +------------------- +PCI bus error is detected by the PCI hardware. On powerpc, the slot is +isolated, in that all I/O is blocked: all reads return 0xffffffff, all +writes are ignored. + +STEP 1: Remove devices +-------------------- +Platform removes the devices depending on the error agent, it could be +this port for all subordinates or upstream component (likely downstream +port) + +STEP 2: Reset link +-------------------- +The platform resets the link. This is a PCI-Express specific step and is +done whenever a fatal error has been detected that can be "solved" by +resetting the link. + +STEP 3: Re-enumerate the devices +-------------------- +Initiates the re-enumeration. Conclusion; General Remarks --------------------------- diff --git a/Documentation/RCU/whatisRCU.txt b/Documentation/RCU/whatisRCU.txt index a27fbfb0efb8..65eb856526b7 100644 --- a/Documentation/RCU/whatisRCU.txt +++ b/Documentation/RCU/whatisRCU.txt @@ -1,3 +1,5 @@ +What is RCU? -- "Read, Copy, Update" + Please note that the "What is RCU?" LWN series is an excellent place to start learning about RCU: diff --git a/Documentation/accelerators/ocxl.rst b/Documentation/accelerators/ocxl.rst index ddcc58d01cfb..14cefc020e2d 100644 --- a/Documentation/accelerators/ocxl.rst +++ b/Documentation/accelerators/ocxl.rst @@ -157,6 +157,17 @@ OCXL_IOCTL_GET_METADATA: Obtains configuration information from the card, such at the size of MMIO areas, the AFU version, and the PASID for the current context. +OCXL_IOCTL_ENABLE_P9_WAIT: + + Allows the AFU to wake a userspace thread executing 'wait'. Returns + information to userspace to allow it to configure the AFU. Note that + this is only available on POWER9. + +OCXL_IOCTL_GET_FEATURES: + + Reports on which CPU features that affect OpenCAPI are usable from + userspace. + mmap ---- diff --git a/Documentation/acpi/cppc_sysfs.txt b/Documentation/acpi/cppc_sysfs.txt new file mode 100644 index 000000000000..f20fb445135d --- /dev/null +++ b/Documentation/acpi/cppc_sysfs.txt @@ -0,0 +1,69 @@ + + Collaborative Processor Performance Control (CPPC) + +CPPC defined in the ACPI spec describes a mechanism for the OS to manage the +performance of a logical processor on a contigious and abstract performance +scale. CPPC exposes a set of registers to describe abstract performance scale, +to request performance levels and to measure per-cpu delivered performance. + +For more details on CPPC please refer to the ACPI specification at: + +http://uefi.org/specifications + +Some of the CPPC registers are exposed via sysfs under: + +/sys/devices/system/cpu/cpuX/acpi_cppc/ + +for each cpu X + +-------------------------------------------------------------------------------- + +$ ls -lR /sys/devices/system/cpu/cpu0/acpi_cppc/ +/sys/devices/system/cpu/cpu0/acpi_cppc/: +total 0 +-r--r--r-- 1 root root 65536 Mar 5 19:38 feedback_ctrs +-r--r--r-- 1 root root 65536 Mar 5 19:38 highest_perf +-r--r--r-- 1 root root 65536 Mar 5 19:38 lowest_freq +-r--r--r-- 1 root root 65536 Mar 5 19:38 lowest_nonlinear_perf +-r--r--r-- 1 root root 65536 Mar 5 19:38 lowest_perf +-r--r--r-- 1 root root 65536 Mar 5 19:38 nominal_freq +-r--r--r-- 1 root root 65536 Mar 5 19:38 nominal_perf +-r--r--r-- 1 root root 65536 Mar 5 19:38 reference_perf +-r--r--r-- 1 root root 65536 Mar 5 19:38 wraparound_time + +-------------------------------------------------------------------------------- + +* highest_perf : Highest performance of this processor (abstract scale). +* nominal_perf : Highest sustained performance of this processor (abstract scale). +* lowest_nonlinear_perf : Lowest performance of this processor with nonlinear + power savings (abstract scale). +* lowest_perf : Lowest performance of this processor (abstract scale). + +* lowest_freq : CPU frequency corresponding to lowest_perf (in MHz). +* nominal_freq : CPU frequency corresponding to nominal_perf (in MHz). + The above frequencies should only be used to report processor performance in + freqency instead of abstract scale. These values should not be used for any + functional decisions. + +* feedback_ctrs : Includes both Reference and delivered performance counter. + Reference counter ticks up proportional to processor's reference performance. + Delivered counter ticks up proportional to processor's delivered performance. +* wraparound_time: Minimum time for the feedback counters to wraparound (seconds). +* reference_perf : Performance level at which reference performance counter + accumulates (abstract scale). + +-------------------------------------------------------------------------------- + + Computing Average Delivered Performance + +Below describes the steps to compute the average performance delivered by taking +two different snapshots of feedback counters at time T1 and T2. + +T1: Read feedback_ctrs as fbc_t1 + Wait or run some workload +T2: Read feedback_ctrs as fbc_t2 + +delivered_counter_delta = fbc_t2[del] - fbc_t1[del] +reference_counter_delta = fbc_t2[ref] - fbc_t1[ref] + +delivered_perf = (refernce_perf x delivered_counter_delta) / reference_counter_delta diff --git a/Documentation/acpi/method-customizing.txt b/Documentation/acpi/method-customizing.txt index a3f598e141f2..7235da975f23 100644 --- a/Documentation/acpi/method-customizing.txt +++ b/Documentation/acpi/method-customizing.txt @@ -16,7 +16,8 @@ control method rather than override the entire DSDT, because kernel rebuild/reboot is not needed and test result can be got in minutes. Note: Only ACPI METHOD can be overridden, any other object types like - "Device", "OperationRegion", are not recognized. + "Device", "OperationRegion", are not recognized. Methods + declared inside scope operators are also not supported. Note: The same ACPI control method can be overridden for many times, and it's always the latest one that used by Linux/kernel. Note: To get the ACPI debug object output (Store (AAAA, Debug)), @@ -32,8 +33,6 @@ Note: To get the ACPI debug object output (Store (AAAA, Debug)), DefinitionBlock ("", "SSDT", 1, "", "", 0x20080715) { - External (ACON) - Method (\_SB_.AC._PSR, 0, NotSerialized) { Store ("In AC _PSR", Debug) @@ -42,9 +41,10 @@ Note: To get the ACPI debug object output (Store (AAAA, Debug)), } Note that the full pathname of the method in ACPI namespace should be used. - And remember to use "External" to declare external objects. e) assemble the file to generate the AML code of the method. - e.g. "iasl psr.asl" (psr.aml is generated as a result) + e.g. "iasl -vw 6084 psr.asl" (psr.aml is generated as a result) + If parameter "-vw 6084" is not supported by your iASL compiler, + please try a newer version. f) mount debugfs by "mount -t debugfs none /sys/kernel/debug" g) override the old method via the debugfs by running "cat /tmp/psr.aml > /sys/kernel/debug/acpi/custom_method" diff --git a/Documentation/admin-guide/LSM/apparmor.rst b/Documentation/admin-guide/LSM/apparmor.rst index 3e9734bd0e05..6cf81bbd7ce8 100644 --- a/Documentation/admin-guide/LSM/apparmor.rst +++ b/Documentation/admin-guide/LSM/apparmor.rst @@ -44,8 +44,8 @@ Links Mailing List - apparmor@lists.ubuntu.com -Wiki - http://apparmor.wiki.kernel.org/ +Wiki - http://wiki.apparmor.net -User space tools - https://launchpad.net/apparmor +User space tools - https://gitlab.com/apparmor -Kernel module - git://git.kernel.org/pub/scm/linux/kernel/git/jj/apparmor-dev.git +Kernel module - git://git.kernel.org/pub/scm/linux/kernel/git/jj/linux-apparmor diff --git a/Documentation/bcache.txt b/Documentation/admin-guide/bcache.rst similarity index 100% rename from Documentation/bcache.txt rename to Documentation/admin-guide/bcache.rst diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst new file mode 100644 index 000000000000..8a2c52d5c53b --- /dev/null +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -0,0 +1,2040 @@ +================ +Control Group v2 +================ + +:Date: October, 2015 +:Author: Tejun Heo + +This is the authoritative documentation on the design, interface and +conventions of cgroup v2. It describes all userland-visible aspects +of cgroup including core and specific controller behaviors. All +future changes must be reflected in this document. Documentation for +v1 is available under Documentation/cgroup-v1/. + +.. CONTENTS + + 1. Introduction + 1-1. Terminology + 1-2. What is cgroup? + 2. Basic Operations + 2-1. Mounting + 2-2. Organizing Processes and Threads + 2-2-1. Processes + 2-2-2. Threads + 2-3. [Un]populated Notification + 2-4. Controlling Controllers + 2-4-1. Enabling and Disabling + 2-4-2. Top-down Constraint + 2-4-3. No Internal Process Constraint + 2-5. Delegation + 2-5-1. Model of Delegation + 2-5-2. Delegation Containment + 2-6. Guidelines + 2-6-1. Organize Once and Control + 2-6-2. Avoid Name Collisions + 3. Resource Distribution Models + 3-1. Weights + 3-2. Limits + 3-3. Protections + 3-4. Allocations + 4. Interface Files + 4-1. Format + 4-2. Conventions + 4-3. Core Interface Files + 5. Controllers + 5-1. CPU + 5-1-1. CPU Interface Files + 5-2. Memory + 5-2-1. Memory Interface Files + 5-2-2. Usage Guidelines + 5-2-3. Memory Ownership + 5-3. IO + 5-3-1. IO Interface Files + 5-3-2. Writeback + 5-4. PID + 5-4-1. PID Interface Files + 5-5. Device + 5-6. RDMA + 5-6-1. RDMA Interface Files + 5-7. Misc + 5-7-1. perf_event + 5-N. Non-normative information + 5-N-1. CPU controller root cgroup process behaviour + 5-N-2. IO controller root cgroup process behaviour + 6. Namespace + 6-1. Basics + 6-2. The Root and Views + 6-3. Migration and setns(2) + 6-4. Interaction with Other Namespaces + P. Information on Kernel Programming + P-1. Filesystem Support for Writeback + D. Deprecated v1 Core Features + R. Issues with v1 and Rationales for v2 + R-1. Multiple Hierarchies + R-2. Thread Granularity + R-3. Competition Between Inner Nodes and Threads + R-4. Other Interface Issues + R-5. Controller Issues and Remedies + R-5-1. Memory + + +Introduction +============ + +Terminology +----------- + +"cgroup" stands for "control group" and is never capitalized. The +singular form is used to designate the whole feature and also as a +qualifier as in "cgroup controllers". When explicitly referring to +multiple individual control groups, the plural form "cgroups" is used. + + +What is cgroup? +--------------- + +cgroup is a mechanism to organize processes hierarchically and +distribute system resources along the hierarchy in a controlled and +configurable manner. + +cgroup is largely composed of two parts - the core and controllers. +cgroup core is primarily responsible for hierarchically organizing +processes. A cgroup controller is usually responsible for +distributing a specific type of system resource along the hierarchy +although there are utility controllers which serve purposes other than +resource distribution. + +cgroups form a tree structure and every process in the system belongs +to one and only one cgroup. All threads of a process belong to the +same cgroup. On creation, all processes are put in the cgroup that +the parent process belongs to at the time. A process can be migrated +to another cgroup. Migration of a process doesn't affect already +existing descendant processes. + +Following certain structural constraints, controllers may be enabled or +disabled selectively on a cgroup. All controller behaviors are +hierarchical - if a controller is enabled on a cgroup, it affects all +processes which belong to the cgroups consisting the inclusive +sub-hierarchy of the cgroup. When a controller is enabled on a nested +cgroup, it always restricts the resource distribution further. The +restrictions set closer to the root in the hierarchy can not be +overridden from further away. + + +Basic Operations +================ + +Mounting +-------- + +Unlike v1, cgroup v2 has only single hierarchy. The cgroup v2 +hierarchy can be mounted with the following mount command:: + + # mount -t cgroup2 none $MOUNT_POINT + +cgroup2 filesystem has the magic number 0x63677270 ("cgrp"). All +controllers which support v2 and are not bound to a v1 hierarchy are +automatically bound to the v2 hierarchy and show up at the root. +Controllers which are not in active use in the v2 hierarchy can be +bound to other hierarchies. This allows mixing v2 hierarchy with the +legacy v1 multiple hierarchies in a fully backward compatible way. + +A controller can be moved across hierarchies only after the controller +is no longer referenced in its current hierarchy. Because per-cgroup +controller states are destroyed asynchronously and controllers may +have lingering references, a controller may not show up immediately on +the v2 hierarchy after the final umount of the previous hierarchy. +Similarly, a controller should be fully disabled to be moved out of +the unified hierarchy and it may take some time for the disabled +controller to become available for other hierarchies; furthermore, due +to inter-controller dependencies, other controllers may need to be +disabled too. + +While useful for development and manual configurations, moving +controllers dynamically between the v2 and other hierarchies is +strongly discouraged for production use. It is recommended to decide +the hierarchies and controller associations before starting using the +controllers after system boot. + +During transition to v2, system management software might still +automount the v1 cgroup filesystem and so hijack all controllers +during boot, before manual intervention is possible. To make testing +and experimenting easier, the kernel parameter cgroup_no_v1= allows +disabling controllers in v1 and make them always available in v2. + +cgroup v2 currently supports the following mount options. + + nsdelegate + + Consider cgroup namespaces as delegation boundaries. This + option is system wide and can only be set on mount or modified + through remount from the init namespace. The mount option is + ignored on non-init namespace mounts. Please refer to the + Delegation section for details. + + +Organizing Processes and Threads +-------------------------------- + +Processes +~~~~~~~~~ + +Initially, only the root cgroup exists to which all processes belong. +A child cgroup can be created by creating a sub-directory:: + + # mkdir $CGROUP_NAME + +A given cgroup may have multiple child cgroups forming a tree +structure. Each cgroup has a read-writable interface file +"cgroup.procs". When read, it lists the PIDs of all processes which +belong to the cgroup one-per-line. The PIDs are not ordered and the +same PID may show up more than once if the process got moved to +another cgroup and then back or the PID got recycled while reading. + +A process can be migrated into a cgroup by writing its PID to the +target cgroup's "cgroup.procs" file. Only one process can be migrated +on a single write(2) call. If a process is composed of multiple +threads, writing the PID of any thread migrates all threads of the +process. + +When a process forks a child process, the new process is born into the +cgroup that the forking process belongs to at the time of the +operation. After exit, a process stays associated with the cgroup +that it belonged to at the time of exit until it's reaped; however, a +zombie process does not appear in "cgroup.procs" and thus can't be +moved to another cgroup. + +A cgroup which doesn't have any children or live processes can be +destroyed by removing the directory. Note that a cgroup which doesn't +have any children and is associated only with zombie processes is +considered empty and can be removed:: + + # rmdir $CGROUP_NAME + +"/proc/$PID/cgroup" lists a process's cgroup membership. If legacy +cgroup is in use in the system, this file may contain multiple lines, +one for each hierarchy. The entry for cgroup v2 is always in the +format "0::$PATH":: + + # cat /proc/842/cgroup + ... + 0::/test-cgroup/test-cgroup-nested + +If the process becomes a zombie and the cgroup it was associated with +is removed subsequently, " (deleted)" is appended to the path:: + + # cat /proc/842/cgroup + ... + 0::/test-cgroup/test-cgroup-nested (deleted) + + +Threads +~~~~~~~ + +cgroup v2 supports thread granularity for a subset of controllers to +support use cases requiring hierarchical resource distribution across +the threads of a group of processes. By default, all threads of a +process belong to the same cgroup, which also serves as the resource +domain to host resource consumptions which are not specific to a +process or thread. The thread mode allows threads to be spread across +a subtree while still maintaining the common resource domain for them. + +Controllers which support thread mode are called threaded controllers. +The ones which don't are called domain controllers. + +Marking a cgroup threaded makes it join the resource domain of its +parent as a threaded cgroup. The parent may be another threaded +cgroup whose resource domain is further up in the hierarchy. The root +of a threaded subtree, that is, the nearest ancestor which is not +threaded, is called threaded domain or thread root interchangeably and +serves as the resource domain for the entire subtree. + +Inside a threaded subtree, threads of a process can be put in +different cgroups and are not subject to the no internal process +constraint - threaded controllers can be enabled on non-leaf cgroups +whether they have threads in them or not. + +As the threaded domain cgroup hosts all the domain resource +consumptions of the subtree, it is considered to have internal +resource consumptions whether there are processes in it or not and +can't have populated child cgroups which aren't threaded. Because the +root cgroup is not subject to no internal process constraint, it can +serve both as a threaded domain and a parent to domain cgroups. + +The current operation mode or type of the cgroup is shown in the +"cgroup.type" file which indicates whether the cgroup is a normal +domain, a domain which is serving as the domain of a threaded subtree, +or a threaded cgroup. + +On creation, a cgroup is always a domain cgroup and can be made +threaded by writing "threaded" to the "cgroup.type" file. The +operation is single direction:: + + # echo threaded > cgroup.type + +Once threaded, the cgroup can't be made a domain again. To enable the +thread mode, the following conditions must be met. + +- As the cgroup will join the parent's resource domain. The parent + must either be a valid (threaded) domain or a threaded cgroup. + +- When the parent is an unthreaded domain, it must not have any domain + controllers enabled or populated domain children. The root is + exempt from this requirement. + +Topology-wise, a cgroup can be in an invalid state. Please consider +the following topology:: + + A (threaded domain) - B (threaded) - C (domain, just created) + +C is created as a domain but isn't connected to a parent which can +host child domains. C can't be used until it is turned into a +threaded cgroup. "cgroup.type" file will report "domain (invalid)" in +these cases. Operations which fail due to invalid topology use +EOPNOTSUPP as the errno. + +A domain cgroup is turned into a threaded domain when one of its child +cgroup becomes threaded or threaded controllers are enabled in the +"cgroup.subtree_control" file while there are processes in the cgroup. +A threaded domain reverts to a normal domain when the conditions +clear. + +When read, "cgroup.threads" contains the list of the thread IDs of all +threads in the cgroup. Except that the operations are per-thread +instead of per-process, "cgroup.threads" has the same format and +behaves the same way as "cgroup.procs". While "cgroup.threads" can be +written to in any cgroup, as it can only move threads inside the same +threaded domain, its operations are confined inside each threaded +subtree. + +The threaded domain cgroup serves as the resource domain for the whole +subtree, and, while the threads can be scattered across the subtree, +all the processes are considered to be in the threaded domain cgroup. +"cgroup.procs" in a threaded domain cgroup contains the PIDs of all +processes in the subtree and is not readable in the subtree proper. +However, "cgroup.procs" can be written to from anywhere in the subtree +to migrate all threads of the matching process to the cgroup. + +Only threaded controllers can be enabled in a threaded subtree. When +a threaded controller is enabled inside a threaded subtree, it only +accounts for and controls resource consumptions associated with the +threads in the cgroup and its descendants. All consumptions which +aren't tied to a specific thread belong to the threaded domain cgroup. + +Because a threaded subtree is exempt from no internal process +constraint, a threaded controller must be able to handle competition +between threads in a non-leaf cgroup and its child cgroups. Each +threaded controller defines how such competitions are handled. + + +[Un]populated Notification +-------------------------- + +Each non-root cgroup has a "cgroup.events" file which contains +"populated" field indicating whether the cgroup's sub-hierarchy has +live processes in it. Its value is 0 if there is no live process in +the cgroup and its descendants; otherwise, 1. poll and [id]notify +events are triggered when the value changes. This can be used, for +example, to start a clean-up operation after all processes of a given +sub-hierarchy have exited. The populated state updates and +notifications are recursive. Consider the following sub-hierarchy +where the numbers in the parentheses represent the numbers of processes +in each cgroup:: + + A(4) - B(0) - C(1) + \ D(0) + +A, B and C's "populated" fields would be 1 while D's 0. After the one +process in C exits, B and C's "populated" fields would flip to "0" and +file modified events will be generated on the "cgroup.events" files of +both cgroups. + + +Controlling Controllers +----------------------- + +Enabling and Disabling +~~~~~~~~~~~~~~~~~~~~~~ + +Each cgroup has a "cgroup.controllers" file which lists all +controllers available for the cgroup to enable:: + + # cat cgroup.controllers + cpu io memory + +No controller is enabled by default. Controllers can be enabled and +disabled by writing to the "cgroup.subtree_control" file:: + + # echo "+cpu +memory -io" > cgroup.subtree_control + +Only controllers which are listed in "cgroup.controllers" can be +enabled. When multiple operations are specified as above, either they +all succeed or fail. If multiple operations on the same controller +are specified, the last one is effective. + +Enabling a controller in a cgroup indicates that the distribution of +the target resource across its immediate children will be controlled. +Consider the following sub-hierarchy. The enabled controllers are +listed in parentheses:: + + A(cpu,memory) - B(memory) - C() + \ D() + +As A has "cpu" and "memory" enabled, A will control the distribution +of CPU cycles and memory to its children, in this case, B. As B has +"memory" enabled but not "CPU", C and D will compete freely on CPU +cycles but their division of memory available to B will be controlled. + +As a controller regulates the distribution of the target resource to +the cgroup's children, enabling it creates the controller's interface +files in the child cgroups. In the above example, enabling "cpu" on B +would create the "cpu." prefixed controller interface files in C and +D. Likewise, disabling "memory" from B would remove the "memory." +prefixed controller interface files from C and D. This means that the +controller interface files - anything which doesn't start with +"cgroup." are owned by the parent rather than the cgroup itself. + + +Top-down Constraint +~~~~~~~~~~~~~~~~~~~ + +Resources are distributed top-down and a cgroup can further distribute +a resource only if the resource has been distributed to it from the +parent. This means that all non-root "cgroup.subtree_control" files +can only contain controllers which are enabled in the parent's +"cgroup.subtree_control" file. A controller can be enabled only if +the parent has the controller enabled and a controller can't be +disabled if one or more children have it enabled. + + +No Internal Process Constraint +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Non-root cgroups can distribute domain resources to their children +only when they don't have any processes of their own. In other words, +only domain cgroups which don't contain any processes can have domain +controllers enabled in their "cgroup.subtree_control" files. + +This guarantees that, when a domain controller is looking at the part +of the hierarchy which has it enabled, processes are always only on +the leaves. This rules out situations where child cgroups compete +against internal processes of the parent. + +The root cgroup is exempt from this restriction. Root contains +processes and anonymous resource consumption which can't be associated +with any other cgroups and requires special treatment from most +controllers. How resource consumption in the root cgroup is governed +is up to each controller (for more information on this topic please +refer to the Non-normative information section in the Controllers +chapter). + +Note that the restriction doesn't get in the way if there is no +enabled controller in the cgroup's "cgroup.subtree_control". This is +important as otherwise it wouldn't be possible to create children of a +populated cgroup. To control resource distribution of a cgroup, the +cgroup must create children and transfer all its processes to the +children before enabling controllers in its "cgroup.subtree_control" +file. + + +Delegation +---------- + +Model of Delegation +~~~~~~~~~~~~~~~~~~~ + +A cgroup can be delegated in two ways. First, to a less privileged +user by granting write access of the directory and its "cgroup.procs", +"cgroup.threads" and "cgroup.subtree_control" files to the user. +Second, if the "nsdelegate" mount option is set, automatically to a +cgroup namespace on namespace creation. + +Because the resource control interface files in a given directory +control the distribution of the parent's resources, the delegatee +shouldn't be allowed to write to them. For the first method, this is +achieved by not granting access to these files. For the second, the +kernel rejects writes to all files other than "cgroup.procs" and +"cgroup.subtree_control" on a namespace root from inside the +namespace. + +The end results are equivalent for both delegation types. Once +delegated, the user can build sub-hierarchy under the directory, +organize processes inside it as it sees fit and further distribute the +resources it received from the parent. The limits and other settings +of all resource controllers are hierarchical and regardless of what +happens in the delegated sub-hierarchy, nothing can escape the +resource restrictions imposed by the parent. + +Currently, cgroup doesn't impose any restrictions on the number of +cgroups in or nesting depth of a delegated sub-hierarchy; however, +this may be limited explicitly in the future. + + +Delegation Containment +~~~~~~~~~~~~~~~~~~~~~~ + +A delegated sub-hierarchy is contained in the sense that processes +can't be moved into or out of the sub-hierarchy by the delegatee. + +For delegations to a less privileged user, this is achieved by +requiring the following conditions for a process with a non-root euid +to migrate a target process into a cgroup by writing its PID to the +"cgroup.procs" file. + +- The writer must have write access to the "cgroup.procs" file. + +- The writer must have write access to the "cgroup.procs" file of the + common ancestor of the source and destination cgroups. + +The above two constraints ensure that while a delegatee may migrate +processes around freely in the delegated sub-hierarchy it can't pull +in from or push out to outside the sub-hierarchy. + +For an example, let's assume cgroups C0 and C1 have been delegated to +user U0 who created C00, C01 under C0 and C10 under C1 as follows and +all processes under C0 and C1 belong to U0:: + + ~~~~~~~~~~~~~ - C0 - C00 + ~ cgroup ~ \ C01 + ~ hierarchy ~ + ~~~~~~~~~~~~~ - C1 - C10 + +Let's also say U0 wants to write the PID of a process which is +currently in C10 into "C00/cgroup.procs". U0 has write access to the +file; however, the common ancestor of the source cgroup C10 and the +destination cgroup C00 is above the points of delegation and U0 would +not have write access to its "cgroup.procs" files and thus the write +will be denied with -EACCES. + +For delegations to namespaces, containment is achieved by requiring +that both the source and destination cgroups are reachable from the +namespace of the process which is attempting the migration. If either +is not reachable, the migration is rejected with -ENOENT. + + +Guidelines +---------- + +Organize Once and Control +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Migrating a process across cgroups is a relatively expensive operation +and stateful resources such as memory are not moved together with the +process. This is an explicit design decision as there often exist +inherent trade-offs between migration and various hot paths in terms +of synchronization cost. + +As such, migrating processes across cgroups frequently as a means to +apply different resource restrictions is discouraged. A workload +should be assigned to a cgroup according to the system's logical and +resource structure once on start-up. Dynamic adjustments to resource +distribution can be made by changing controller configuration through +the interface files. + + +Avoid Name Collisions +~~~~~~~~~~~~~~~~~~~~~ + +Interface files for a cgroup and its children cgroups occupy the same +directory and it is possible to create children cgroups which collide +with interface files. + +All cgroup core interface files are prefixed with "cgroup." and each +controller's interface files are prefixed with the controller name and +a dot. A controller's name is composed of lower case alphabets and +'_'s but never begins with an '_' so it can be used as the prefix +character for collision avoidance. Also, interface file names won't +start or end with terms which are often used in categorizing workloads +such as job, service, slice, unit or workload. + +cgroup doesn't do anything to prevent name collisions and it's the +user's responsibility to avoid them. + + +Resource Distribution Models +============================ + +cgroup controllers implement several resource distribution schemes +depending on the resource type and expected use cases. This section +describes major schemes in use along with their expected behaviors. + + +Weights +------- + +A parent's resource is distributed by adding up the weights of all +active children and giving each the fraction matching the ratio of its +weight against the sum. As only children which can make use of the +resource at the moment participate in the distribution, this is +work-conserving. Due to the dynamic nature, this model is usually +used for stateless resources. + +All weights are in the range [1, 10000] with the default at 100. This +allows symmetric multiplicative biases in both directions at fine +enough granularity while staying in the intuitive range. + +As long as the weight is in range, all configuration combinations are +valid and there is no reason to reject configuration changes or +process migrations. + +"cpu.weight" proportionally distributes CPU cycles to active children +and is an example of this type. + + +Limits +------ + +A child can only consume upto the configured amount of the resource. +Limits can be over-committed - the sum of the limits of children can +exceed the amount of resource available to the parent. + +Limits are in the range [0, max] and defaults to "max", which is noop. + +As limits can be over-committed, all configuration combinations are +valid and there is no reason to reject configuration changes or +process migrations. + +"io.max" limits the maximum BPS and/or IOPS that a cgroup can consume +on an IO device and is an example of this type. + + +Protections +----------- + +A cgroup is protected to be allocated upto the configured amount of +the resource if the usages of all its ancestors are under their +protected levels. Protections can be hard guarantees or best effort +soft boundaries. Protections can also be over-committed in which case +only upto the amount available to the parent is protected among +children. + +Protections are in the range [0, max] and defaults to 0, which is +noop. + +As protections can be over-committed, all configuration combinations +are valid and there is no reason to reject configuration changes or +process migrations. + +"memory.low" implements best-effort memory protection and is an +example of this type. + + +Allocations +----------- + +A cgroup is exclusively allocated a certain amount of a finite +resource. Allocations can't be over-committed - the sum of the +allocations of children can not exceed the amount of resource +available to the parent. + +Allocations are in the range [0, max] and defaults to 0, which is no +resource. + +As allocations can't be over-committed, some configuration +combinations are invalid and should be rejected. Also, if the +resource is mandatory for execution of processes, process migrations +may be rejected. + +"cpu.rt.max" hard-allocates realtime slices and is an example of this +type. + + +Interface Files +=============== + +Format +------ + +All interface files should be in one of the following formats whenever +possible:: + + New-line separated values + (when only one value can be written at once) + + VAL0\n + VAL1\n + ... + + Space separated values + (when read-only or multiple values can be written at once) + + VAL0 VAL1 ...\n + + Flat keyed + + KEY0 VAL0\n + KEY1 VAL1\n + ... + + Nested keyed + + KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01... + KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11... + ... + +For a writable file, the format for writing should generally match +reading; however, controllers may allow omitting later fields or +implement restricted shortcuts for most common use cases. + +For both flat and nested keyed files, only the values for a single key +can be written at a time. For nested keyed files, the sub key pairs +may be specified in any order and not all pairs have to be specified. + + +Conventions +----------- + +- Settings for a single feature should be contained in a single file. + +- The root cgroup should be exempt from resource control and thus + shouldn't have resource control interface files. Also, + informational files on the root cgroup which end up showing global + information available elsewhere shouldn't exist. + +- If a controller implements weight based resource distribution, its + interface file should be named "weight" and have the range [1, + 10000] with 100 as the default. The values are chosen to allow + enough and symmetric bias in both directions while keeping it + intuitive (the default is 100%). + +- If a controller implements an absolute resource guarantee and/or + limit, the interface files should be named "min" and "max" + respectively. If a controller implements best effort resource + guarantee and/or limit, the interface files should be named "low" + and "high" respectively. + + In the above four control files, the special token "max" should be + used to represent upward infinity for both reading and writing. + +- If a setting has a configurable default value and keyed specific + overrides, the default entry should be keyed with "default" and + appear as the first entry in the file. + + The default value can be updated by writing either "default $VAL" or + "$VAL". + + When writing to update a specific override, "default" can be used as + the value to indicate removal of the override. Override entries + with "default" as the value must not appear when read. + + For example, a setting which is keyed by major:minor device numbers + with integer values may look like the following:: + + # cat cgroup-example-interface-file + default 150 + 8:0 300 + + The default value can be updated by:: + + # echo 125 > cgroup-example-interface-file + + or:: + + # echo "default 125" > cgroup-example-interface-file + + An override can be set by:: + + # echo "8:16 170" > cgroup-example-interface-file + + and cleared by:: + + # echo "8:0 default" > cgroup-example-interface-file + # cat cgroup-example-interface-file + default 125 + 8:16 170 + +- For events which are not very high frequency, an interface file + "events" should be created which lists event key value pairs. + Whenever a notifiable event happens, file modified event should be + generated on the file. + + +Core Interface Files +-------------------- + +All cgroup core files are prefixed with "cgroup." + + cgroup.type + + A read-write single value file which exists on non-root + cgroups. + + When read, it indicates the current type of the cgroup, which + can be one of the following values. + + - "domain" : A normal valid domain cgroup. + + - "domain threaded" : A threaded domain cgroup which is + serving as the root of a threaded subtree. + + - "domain invalid" : A cgroup which is in an invalid state. + It can't be populated or have controllers enabled. It may + be allowed to become a threaded cgroup. + + - "threaded" : A threaded cgroup which is a member of a + threaded subtree. + + A cgroup can be turned into a threaded cgroup by writing + "threaded" to this file. + + cgroup.procs + A read-write new-line separated values file which exists on + all cgroups. + + When read, it lists the PIDs of all processes which belong to + the cgroup one-per-line. The PIDs are not ordered and the + same PID may show up more than once if the process got moved + to another cgroup and then back or the PID got recycled while + reading. + + A PID can be written to migrate the process associated with + the PID to the cgroup. The writer should match all of the + following conditions. + + - It must have write access to the "cgroup.procs" file. + + - It must have write access to the "cgroup.procs" file of the + common ancestor of the source and destination cgroups. + + When delegating a sub-hierarchy, write access to this file + should be granted along with the containing directory. + + In a threaded cgroup, reading this file fails with EOPNOTSUPP + as all the processes belong to the thread root. Writing is + supported and moves every thread of the process to the cgroup. + + cgroup.threads + A read-write new-line separated values file which exists on + all cgroups. + + When read, it lists the TIDs of all threads which belong to + the cgroup one-per-line. The TIDs are not ordered and the + same TID may show up more than once if the thread got moved to + another cgroup and then back or the TID got recycled while + reading. + + A TID can be written to migrate the thread associated with the + TID to the cgroup. The writer should match all of the + following conditions. + + - It must have write access to the "cgroup.threads" file. + + - The cgroup that the thread is currently in must be in the + same resource domain as the destination cgroup. + + - It must have write access to the "cgroup.procs" file of the + common ancestor of the source and destination cgroups. + + When delegating a sub-hierarchy, write access to this file + should be granted along with the containing directory. + + cgroup.controllers + A read-only space separated values file which exists on all + cgroups. + + It shows space separated list of all controllers available to + the cgroup. The controllers are not ordered. + + cgroup.subtree_control + A read-write space separated values file which exists on all + cgroups. Starts out empty. + + When read, it shows space separated list of the controllers + which are enabled to control resource distribution from the + cgroup to its children. + + Space separated list of controllers prefixed with '+' or '-' + can be written to enable or disable controllers. A controller + name prefixed with '+' enables the controller and '-' + disables. If a controller appears more than once on the list, + the last one is effective. When multiple enable and disable + operations are specified, either all succeed or all fail. + + cgroup.events + A read-only flat-keyed file which exists on non-root cgroups. + The following entries are defined. Unless specified + otherwise, a value change in this file generates a file + modified event. + + populated + 1 if the cgroup or its descendants contains any live + processes; otherwise, 0. + + cgroup.max.descendants + A read-write single value files. The default is "max". + + Maximum allowed number of descent cgroups. + If the actual number of descendants is equal or larger, + an attempt to create a new cgroup in the hierarchy will fail. + + cgroup.max.depth + A read-write single value files. The default is "max". + + Maximum allowed descent depth below the current cgroup. + If the actual descent depth is equal or larger, + an attempt to create a new child cgroup will fail. + + cgroup.stat + A read-only flat-keyed file with the following entries: + + nr_descendants + Total number of visible descendant cgroups. + + nr_dying_descendants + Total number of dying descendant cgroups. A cgroup becomes + dying after being deleted by a user. The cgroup will remain + in dying state for some time undefined time (which can depend + on system load) before being completely destroyed. + + A process can't enter a dying cgroup under any circumstances, + a dying cgroup can't revive. + + A dying cgroup can consume system resources not exceeding + limits, which were active at the moment of cgroup deletion. + + +Controllers +=========== + +CPU +--- + +The "cpu" controllers regulates distribution of CPU cycles. This +controller implements weight and absolute bandwidth limit models for +normal scheduling policy and absolute bandwidth allocation model for +realtime scheduling policy. + +WARNING: cgroup2 doesn't yet support control of realtime processes and +the cpu controller can only be enabled when all RT processes are in +the root cgroup. Be aware that system management software may already +have placed RT processes into nonroot cgroups during the system boot +process, and these processes may need to be moved to the root cgroup +before the cpu controller can be enabled. + + +CPU Interface Files +~~~~~~~~~~~~~~~~~~~ + +All time durations are in microseconds. + + cpu.stat + A read-only flat-keyed file which exists on non-root cgroups. + This file exists whether the controller is enabled or not. + + It always reports the following three stats: + + - usage_usec + - user_usec + - system_usec + + and the following three when the controller is enabled: + + - nr_periods + - nr_throttled + - throttled_usec + + cpu.weight + A read-write single value file which exists on non-root + cgroups. The default is "100". + + The weight in the range [1, 10000]. + + cpu.weight.nice + A read-write single value file which exists on non-root + cgroups. The default is "0". + + The nice value is in the range [-20, 19]. + + This interface file is an alternative interface for + "cpu.weight" and allows reading and setting weight using the + same values used by nice(2). Because the range is smaller and + granularity is coarser for the nice values, the read value is + the closest approximation of the current weight. + + cpu.max + A read-write two value file which exists on non-root cgroups. + The default is "max 100000". + + The maximum bandwidth limit. It's in the following format:: + + $MAX $PERIOD + + which indicates that the group may consume upto $MAX in each + $PERIOD duration. "max" for $MAX indicates no limit. If only + one number is written, $MAX is updated. + + +Memory +------ + +The "memory" controller regulates distribution of memory. Memory is +stateful and implements both limit and protection models. Due to the +intertwining between memory usage and reclaim pressure and the +stateful nature of memory, the distribution model is relatively +complex. + +While not completely water-tight, all major memory usages by a given +cgroup are tracked so that the total memory consumption can be +accounted and controlled to a reasonable extent. Currently, the +following types of memory usages are tracked. + +- Userland memory - page cache and anonymous memory. + +- Kernel data structures such as dentries and inodes. + +- TCP socket buffers. + +The above list may expand in the future for better coverage. + + +Memory Interface Files +~~~~~~~~~~~~~~~~~~~~~~ + +All memory amounts are in bytes. If a value which is not aligned to +PAGE_SIZE is written, the value may be rounded up to the closest +PAGE_SIZE multiple when read back. + + memory.current + A read-only single value file which exists on non-root + cgroups. + + The total amount of memory currently being used by the cgroup + and its descendants. + + memory.min + A read-write single value file which exists on non-root + cgroups. The default is "0". + + Hard memory protection. If the memory usage of a cgroup + is within its effective min boundary, the cgroup's memory + won't be reclaimed under any conditions. If there is no + unprotected reclaimable memory available, OOM killer + is invoked. + + Effective min boundary is limited by memory.min values of + all ancestor cgroups. If there is memory.min overcommitment + (child cgroup or cgroups are requiring more protected memory + than parent will allow), then each child cgroup will get + the part of parent's protection proportional to its + actual memory usage below memory.min. + + Putting more memory than generally available under this + protection is discouraged and may lead to constant OOMs. + + If a memory cgroup is not populated with processes, + its memory.min is ignored. + + memory.low + A read-write single value file which exists on non-root + cgroups. The default is "0". + + Best-effort memory protection. If the memory usage of a + cgroup is within its effective low boundary, the cgroup's + memory won't be reclaimed unless memory can be reclaimed + from unprotected cgroups. + + Effective low boundary is limited by memory.low values of + all ancestor cgroups. If there is memory.low overcommitment + (child cgroup or cgroups are requiring more protected memory + than parent will allow), then each child cgroup will get + the part of parent's protection proportional to its + actual memory usage below memory.low. + + Putting more memory than generally available under this + protection is discouraged. + + memory.high + A read-write single value file which exists on non-root + cgroups. The default is "max". + + Memory usage throttle limit. This is the main mechanism to + control memory usage of a cgroup. If a cgroup's usage goes + over the high boundary, the processes of the cgroup are + throttled and put under heavy reclaim pressure. + + Going over the high limit never invokes the OOM killer and + under extreme conditions the limit may be breached. + + memory.max + A read-write single value file which exists on non-root + cgroups. The default is "max". + + Memory usage hard limit. This is the final protection + mechanism. If a cgroup's memory usage reaches this limit and + can't be reduced, the OOM killer is invoked in the cgroup. + Under certain circumstances, the usage may go over the limit + temporarily. + + This is the ultimate protection mechanism. As long as the + high limit is used and monitored properly, this limit's + utility is limited to providing the final safety net. + + memory.events + A read-only flat-keyed file which exists on non-root cgroups. + The following entries are defined. Unless specified + otherwise, a value change in this file generates a file + modified event. + + low + The number of times the cgroup is reclaimed due to + high memory pressure even though its usage is under + the low boundary. This usually indicates that the low + boundary is over-committed. + + high + The number of times processes of the cgroup are + throttled and routed to perform direct memory reclaim + because the high memory boundary was exceeded. For a + cgroup whose memory usage is capped by the high limit + rather than global memory pressure, this event's + occurrences are expected. + + max + The number of times the cgroup's memory usage was + about to go over the max boundary. If direct reclaim + fails to bring it down, the cgroup goes to OOM state. + + oom + The number of time the cgroup's memory usage was + reached the limit and allocation was about to fail. + + Depending on context result could be invocation of OOM + killer and retrying allocation or failing allocation. + + Failed allocation in its turn could be returned into + userspace as -ENOMEM or silently ignored in cases like + disk readahead. For now OOM in memory cgroup kills + tasks iff shortage has happened inside page fault. + + oom_kill + The number of processes belonging to this cgroup + killed by any kind of OOM killer. + + memory.stat + A read-only flat-keyed file which exists on non-root cgroups. + + This breaks down the cgroup's memory footprint into different + types of memory, type-specific details, and other information + on the state and past events of the memory management system. + + All memory amounts are in bytes. + + The entries are ordered to be human readable, and new entries + can show up in the middle. Don't rely on items remaining in a + fixed position; use the keys to look up specific values! + + anon + Amount of memory used in anonymous mappings such as + brk(), sbrk(), and mmap(MAP_ANONYMOUS) + + file + Amount of memory used to cache filesystem data, + including tmpfs and shared memory. + + kernel_stack + Amount of memory allocated to kernel stacks. + + slab + Amount of memory used for storing in-kernel data + structures. + + sock + Amount of memory used in network transmission buffers + + shmem + Amount of cached filesystem data that is swap-backed, + such as tmpfs, shm segments, shared anonymous mmap()s + + file_mapped + Amount of cached filesystem data mapped with mmap() + + file_dirty + Amount of cached filesystem data that was modified but + not yet written back to disk + + file_writeback + Amount of cached filesystem data that was modified and + is currently being written back to disk + + inactive_anon, active_anon, inactive_file, active_file, unevictable + Amount of memory, swap-backed and filesystem-backed, + on the internal memory management lists used by the + page reclaim algorithm + + slab_reclaimable + Part of "slab" that might be reclaimed, such as + dentries and inodes. + + slab_unreclaimable + Part of "slab" that cannot be reclaimed on memory + pressure. + + pgfault + Total number of page faults incurred + + pgmajfault + Number of major page faults incurred + + workingset_refault + + Number of refaults of previously evicted pages + + workingset_activate + + Number of refaulted pages that were immediately activated + + workingset_nodereclaim + + Number of times a shadow node has been reclaimed + + pgrefill + + Amount of scanned pages (in an active LRU list) + + pgscan + + Amount of scanned pages (in an inactive LRU list) + + pgsteal + + Amount of reclaimed pages + + pgactivate + + Amount of pages moved to the active LRU list + + pgdeactivate + + Amount of pages moved to the inactive LRU lis + + pglazyfree + + Amount of pages postponed to be freed under memory pressure + + pglazyfreed + + Amount of reclaimed lazyfree pages + + memory.swap.current + A read-only single value file which exists on non-root + cgroups. + + The total amount of swap currently being used by the cgroup + and its descendants. + + memory.swap.max + A read-write single value file which exists on non-root + cgroups. The default is "max". + + Swap usage hard limit. If a cgroup's swap usage reaches this + limit, anonymous memory of the cgroup will not be swapped out. + + memory.swap.events + A read-only flat-keyed file which exists on non-root cgroups. + The following entries are defined. Unless specified + otherwise, a value change in this file generates a file + modified event. + + max + The number of times the cgroup's swap usage was about + to go over the max boundary and swap allocation + failed. + + fail + The number of times swap allocation failed either + because of running out of swap system-wide or max + limit. + + When reduced under the current usage, the existing swap + entries are reclaimed gradually and the swap usage may stay + higher than the limit for an extended period of time. This + reduces the impact on the workload and memory management. + + +Usage Guidelines +~~~~~~~~~~~~~~~~ + +"memory.high" is the main mechanism to control memory usage. +Over-committing on high limit (sum of high limits > available memory) +and letting global memory pressure to distribute memory according to +usage is a viable strategy. + +Because breach of the high limit doesn't trigger the OOM killer but +throttles the offending cgroup, a management agent has ample +opportunities to monitor and take appropriate actions such as granting +more memory or terminating the workload. + +Determining whether a cgroup has enough memory is not trivial as +memory usage doesn't indicate whether the workload can benefit from +more memory. For example, a workload which writes data received from +network to a file can use all available memory but can also operate as +performant with a small amount of memory. A measure of memory +pressure - how much the workload is being impacted due to lack of +memory - is necessary to determine whether a workload needs more +memory; unfortunately, memory pressure monitoring mechanism isn't +implemented yet. + + +Memory Ownership +~~~~~~~~~~~~~~~~ + +A memory area is charged to the cgroup which instantiated it and stays +charged to the cgroup until the area is released. Migrating a process +to a different cgroup doesn't move the memory usages that it +instantiated while in the previous cgroup to the new cgroup. + +A memory area may be used by processes belonging to different cgroups. +To which cgroup the area will be charged is in-deterministic; however, +over time, the memory area is likely to end up in a cgroup which has +enough memory allowance to avoid high reclaim pressure. + +If a cgroup sweeps a considerable amount of memory which is expected +to be accessed repeatedly by other cgroups, it may make sense to use +POSIX_FADV_DONTNEED to relinquish the ownership of memory areas +belonging to the affected files to ensure correct memory ownership. + + +IO +-- + +The "io" controller regulates the distribution of IO resources. This +controller implements both weight based and absolute bandwidth or IOPS +limit distribution; however, weight based distribution is available +only if cfq-iosched is in use and neither scheme is available for +blk-mq devices. + + +IO Interface Files +~~~~~~~~~~~~~~~~~~ + + io.stat + A read-only nested-keyed file which exists on non-root + cgroups. + + Lines are keyed by $MAJ:$MIN device numbers and not ordered. + The following nested keys are defined. + + ====== =================== + rbytes Bytes read + wbytes Bytes written + rios Number of read IOs + wios Number of write IOs + ====== =================== + + An example read output follows: + + 8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353 + 8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252 + + io.weight + A read-write flat-keyed file which exists on non-root cgroups. + The default is "default 100". + + The first line is the default weight applied to devices + without specific override. The rest are overrides keyed by + $MAJ:$MIN device numbers and not ordered. The weights are in + the range [1, 10000] and specifies the relative amount IO time + the cgroup can use in relation to its siblings. + + The default weight can be updated by writing either "default + $WEIGHT" or simply "$WEIGHT". Overrides can be set by writing + "$MAJ:$MIN $WEIGHT" and unset by writing "$MAJ:$MIN default". + + An example read output follows:: + + default 100 + 8:16 200 + 8:0 50 + + io.max + A read-write nested-keyed file which exists on non-root + cgroups. + + BPS and IOPS based IO limit. Lines are keyed by $MAJ:$MIN + device numbers and not ordered. The following nested keys are + defined. + + ===== ================================== + rbps Max read bytes per second + wbps Max write bytes per second + riops Max read IO operations per second + wiops Max write IO operations per second + ===== ================================== + + When writing, any number of nested key-value pairs can be + specified in any order. "max" can be specified as the value + to remove a specific limit. If the same key is specified + multiple times, the outcome is undefined. + + BPS and IOPS are measured in each IO direction and IOs are + delayed if limit is reached. Temporary bursts are allowed. + + Setting read limit at 2M BPS and write at 120 IOPS for 8:16:: + + echo "8:16 rbps=2097152 wiops=120" > io.max + + Reading returns the following:: + + 8:16 rbps=2097152 wbps=max riops=max wiops=120 + + Write IOPS limit can be removed by writing the following:: + + echo "8:16 wiops=max" > io.max + + Reading now returns the following:: + + 8:16 rbps=2097152 wbps=max riops=max wiops=max + + +Writeback +~~~~~~~~~ + +Page cache is dirtied through buffered writes and shared mmaps and +written asynchronously to the backing filesystem by the writeback +mechanism. Writeback sits between the memory and IO domains and +regulates the proportion of dirty memory by balancing dirtying and +write IOs. + +The io controller, in conjunction with the memory controller, +implements control of page cache writeback IOs. The memory controller +defines the memory domain that dirty memory ratio is calculated and +maintained for and the io controller defines the io domain which +writes out dirty pages for the memory domain. Both system-wide and +per-cgroup dirty memory states are examined and the more restrictive +of the two is enforced. + +cgroup writeback requires explicit support from the underlying +filesystem. Currently, cgroup writeback is implemented on ext2, ext4 +and btrfs. On other filesystems, all writeback IOs are attributed to +the root cgroup. + +There are inherent differences in memory and writeback management +which affects how cgroup ownership is tracked. Memory is tracked per +page while writeback per inode. For the purpose of writeback, an +inode is assigned to a cgroup and all IO requests to write dirty pages +from the inode are attributed to that cgroup. + +As cgroup ownership for memory is tracked per page, there can be pages +which are associated with different cgroups than the one the inode is +associated with. These are called foreign pages. The writeback +constantly keeps track of foreign pages and, if a particular foreign +cgroup becomes the majority over a certain period of time, switches +the ownership of the inode to that cgroup. + +While this model is enough for most use cases where a given inode is +mostly dirtied by a single cgroup even when the main writing cgroup +changes over time, use cases where multiple cgroups write to a single +inode simultaneously are not supported well. In such circumstances, a +significant portion of IOs are likely to be attributed incorrectly. +As memory controller assigns page ownership on the first use and +doesn't update it until the page is released, even if writeback +strictly follows page ownership, multiple cgroups dirtying overlapping +areas wouldn't work as expected. It's recommended to avoid such usage +patterns. + +The sysctl knobs which affect writeback behavior are applied to cgroup +writeback as follows. + + vm.dirty_background_ratio, vm.dirty_ratio + These ratios apply the same to cgroup writeback with the + amount of available memory capped by limits imposed by the + memory controller and system-wide clean memory. + + vm.dirty_background_bytes, vm.dirty_bytes + For cgroup writeback, this is calculated into ratio against + total available memory and applied the same way as + vm.dirty[_background]_ratio. + + +PID +--- + +The process number controller is used to allow a cgroup to stop any +new tasks from being fork()'d or clone()'d after a specified limit is +reached. + +The number of tasks in a cgroup can be exhausted in ways which other +controllers cannot prevent, thus warranting its own controller. For +example, a fork bomb is likely to exhaust the number of tasks before +hitting memory restrictions. + +Note that PIDs used in this controller refer to TIDs, process IDs as +used by the kernel. + + +PID Interface Files +~~~~~~~~~~~~~~~~~~~ + + pids.max + A read-write single value file which exists on non-root + cgroups. The default is "max". + + Hard limit of number of processes. + + pids.current + A read-only single value file which exists on all cgroups. + + The number of processes currently in the cgroup and its + descendants. + +Organisational operations are not blocked by cgroup policies, so it is +possible to have pids.current > pids.max. This can be done by either +setting the limit to be smaller than pids.current, or attaching enough +processes to the cgroup such that pids.current is larger than +pids.max. However, it is not possible to violate a cgroup PID policy +through fork() or clone(). These will return -EAGAIN if the creation +of a new process would cause a cgroup policy to be violated. + + +Device controller +----------------- + +Device controller manages access to device files. It includes both +creation of new device files (using mknod), and access to the +existing device files. + +Cgroup v2 device controller has no interface files and is implemented +on top of cgroup BPF. To control access to device files, a user may +create bpf programs of the BPF_CGROUP_DEVICE type and attach them +to cgroups. On an attempt to access a device file, corresponding +BPF programs will be executed, and depending on the return value +the attempt will succeed or fail with -EPERM. + +A BPF_CGROUP_DEVICE program takes a pointer to the bpf_cgroup_dev_ctx +structure, which describes the device access attempt: access type +(mknod/read/write) and device (type, major and minor numbers). +If the program returns 0, the attempt fails with -EPERM, otherwise +it succeeds. + +An example of BPF_CGROUP_DEVICE program may be found in the kernel +source tree in the tools/testing/selftests/bpf/dev_cgroup.c file. + + +RDMA +---- + +The "rdma" controller regulates the distribution and accounting of +of RDMA resources. + +RDMA Interface Files +~~~~~~~~~~~~~~~~~~~~ + + rdma.max + A readwrite nested-keyed file that exists for all the cgroups + except root that describes current configured resource limit + for a RDMA/IB device. + + Lines are keyed by device name and are not ordered. + Each line contains space separated resource name and its configured + limit that can be distributed. + + The following nested keys are defined. + + ========== ============================= + hca_handle Maximum number of HCA Handles + hca_object Maximum number of HCA Objects + ========== ============================= + + An example for mlx4 and ocrdma device follows:: + + mlx4_0 hca_handle=2 hca_object=2000 + ocrdma1 hca_handle=3 hca_object=max + + rdma.current + A read-only file that describes current resource usage. + It exists for all the cgroup except root. + + An example for mlx4 and ocrdma device follows:: + + mlx4_0 hca_handle=1 hca_object=20 + ocrdma1 hca_handle=1 hca_object=23 + + +Misc +---- + +perf_event +~~~~~~~~~~ + +perf_event controller, if not mounted on a legacy hierarchy, is +automatically enabled on the v2 hierarchy so that perf events can +always be filtered by cgroup v2 path. The controller can still be +moved to a legacy hierarchy after v2 hierarchy is populated. + + +Non-normative information +------------------------- + +This section contains information that isn't considered to be a part of +the stable kernel API and so is subject to change. + + +CPU controller root cgroup process behaviour +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When distributing CPU cycles in the root cgroup each thread in this +cgroup is treated as if it was hosted in a separate child cgroup of the +root cgroup. This child cgroup weight is dependent on its thread nice +level. + +For details of this mapping see sched_prio_to_weight array in +kernel/sched/core.c file (values from this array should be scaled +appropriately so the neutral - nice 0 - value is 100 instead of 1024). + + +IO controller root cgroup process behaviour +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Root cgroup processes are hosted in an implicit leaf child node. +When distributing IO resources this implicit child node is taken into +account as if it was a normal child cgroup of the root cgroup with a +weight value of 200. + + +Namespace +========= + +Basics +------ + +cgroup namespace provides a mechanism to virtualize the view of the +"/proc/$PID/cgroup" file and cgroup mounts. The CLONE_NEWCGROUP clone +flag can be used with clone(2) and unshare(2) to create a new cgroup +namespace. The process running inside the cgroup namespace will have +its "/proc/$PID/cgroup" output restricted to cgroupns root. The +cgroupns root is the cgroup of the process at the time of creation of +the cgroup namespace. + +Without cgroup namespace, the "/proc/$PID/cgroup" file shows the +complete path of the cgroup of a process. In a container setup where +a set of cgroups and namespaces are intended to isolate processes the +"/proc/$PID/cgroup" file may leak potential system level information +to the isolated processes. For Example:: + + # cat /proc/self/cgroup + 0::/batchjobs/container_id1 + +The path '/batchjobs/container_id1' can be considered as system-data +and undesirable to expose to the isolated processes. cgroup namespace +can be used to restrict visibility of this path. For example, before +creating a cgroup namespace, one would see:: + + # ls -l /proc/self/ns/cgroup + lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835] + # cat /proc/self/cgroup + 0::/batchjobs/container_id1 + +After unsharing a new namespace, the view changes:: + + # ls -l /proc/self/ns/cgroup + lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183] + # cat /proc/self/cgroup + 0::/ + +When some thread from a multi-threaded process unshares its cgroup +namespace, the new cgroupns gets applied to the entire process (all +the threads). This is natural for the v2 hierarchy; however, for the +legacy hierarchies, this may be unexpected. + +A cgroup namespace is alive as long as there are processes inside or +mounts pinning it. When the last usage goes away, the cgroup +namespace is destroyed. The cgroupns root and the actual cgroups +remain. + + +The Root and Views +------------------ + +The 'cgroupns root' for a cgroup namespace is the cgroup in which the +process calling unshare(2) is running. For example, if a process in +/batchjobs/container_id1 cgroup calls unshare, cgroup +/batchjobs/container_id1 becomes the cgroupns root. For the +init_cgroup_ns, this is the real root ('/') cgroup. + +The cgroupns root cgroup does not change even if the namespace creator +process later moves to a different cgroup:: + + # ~/unshare -c # unshare cgroupns in some cgroup + # cat /proc/self/cgroup + 0::/ + # mkdir sub_cgrp_1 + # echo 0 > sub_cgrp_1/cgroup.procs + # cat /proc/self/cgroup + 0::/sub_cgrp_1 + +Each process gets its namespace-specific view of "/proc/$PID/cgroup" + +Processes running inside the cgroup namespace will be able to see +cgroup paths (in /proc/self/cgroup) only inside their root cgroup. +From within an unshared cgroupns:: + + # sleep 100000 & + [1] 7353 + # echo 7353 > sub_cgrp_1/cgroup.procs + # cat /proc/7353/cgroup + 0::/sub_cgrp_1 + +From the initial cgroup namespace, the real cgroup path will be +visible:: + + $ cat /proc/7353/cgroup + 0::/batchjobs/container_id1/sub_cgrp_1 + +From a sibling cgroup namespace (that is, a namespace rooted at a +different cgroup), the cgroup path relative to its own cgroup +namespace root will be shown. For instance, if PID 7353's cgroup +namespace root is at '/batchjobs/container_id2', then it will see:: + + # cat /proc/7353/cgroup + 0::/../container_id2/sub_cgrp_1 + +Note that the relative path always starts with '/' to indicate that +its relative to the cgroup namespace root of the caller. + + +Migration and setns(2) +---------------------- + +Processes inside a cgroup namespace can move into and out of the +namespace root if they have proper access to external cgroups. For +example, from inside a namespace with cgroupns root at +/batchjobs/container_id1, and assuming that the global hierarchy is +still accessible inside cgroupns:: + + # cat /proc/7353/cgroup + 0::/sub_cgrp_1 + # echo 7353 > batchjobs/container_id2/cgroup.procs + # cat /proc/7353/cgroup + 0::/../container_id2 + +Note that this kind of setup is not encouraged. A task inside cgroup +namespace should only be exposed to its own cgroupns hierarchy. + +setns(2) to another cgroup namespace is allowed when: + +(a) the process has CAP_SYS_ADMIN against its current user namespace +(b) the process has CAP_SYS_ADMIN against the target cgroup + namespace's userns + +No implicit cgroup changes happen with attaching to another cgroup +namespace. It is expected that the someone moves the attaching +process under the target cgroup namespace root. + + +Interaction with Other Namespaces +--------------------------------- + +Namespace specific cgroup hierarchy can be mounted by a process +running inside a non-init cgroup namespace:: + + # mount -t cgroup2 none $MOUNT_POINT + +This will mount the unified cgroup hierarchy with cgroupns root as the +filesystem root. The process needs CAP_SYS_ADMIN against its user and +mount namespaces. + +The virtualization of /proc/self/cgroup file combined with restricting +the view of cgroup hierarchy by namespace-private cgroupfs mount +provides a properly isolated cgroup view inside the container. + + +Information on Kernel Programming +================================= + +This section contains kernel programming information in the areas +where interacting with cgroup is necessary. cgroup core and +controllers are not covered. + + +Filesystem Support for Writeback +-------------------------------- + +A filesystem can support cgroup writeback by updating +address_space_operations->writepage[s]() to annotate bio's using the +following two functions. + + wbc_init_bio(@wbc, @bio) + Should be called for each bio carrying writeback data and + associates the bio with the inode's owner cgroup. Can be + called anytime between bio allocation and submission. + + wbc_account_io(@wbc, @page, @bytes) + Should be called for each data segment being written out. + While this function doesn't care exactly when it's called + during the writeback session, it's the easiest and most + natural to call it as data segments are added to a bio. + +With writeback bio's annotated, cgroup support can be enabled per +super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for +selective disabling of cgroup writeback support which is helpful when +certain filesystem features, e.g. journaled data mode, are +incompatible. + +wbc_init_bio() binds the specified bio to its cgroup. Depending on +the configuration, the bio may be executed at a lower priority and if +the writeback session is holding shared resources, e.g. a journal +entry, may lead to priority inversion. There is no one easy solution +for the problem. Filesystems can try to work around specific problem +cases by skipping wbc_init_bio() or using bio_associate_blkcg() +directly. + + +Deprecated v1 Core Features +=========================== + +- Multiple hierarchies including named ones are not supported. + +- All v1 mount options are not supported. + +- The "tasks" file is removed and "cgroup.procs" is not sorted. + +- "cgroup.clone_children" is removed. + +- /proc/cgroups is meaningless for v2. Use "cgroup.controllers" file + at the root instead. + + +Issues with v1 and Rationales for v2 +==================================== + +Multiple Hierarchies +-------------------- + +cgroup v1 allowed an arbitrary number of hierarchies and each +hierarchy could host any number of controllers. While this seemed to +provide a high level of flexibility, it wasn't useful in practice. + +For example, as there is only one instance of each controller, utility +type controllers such as freezer which can be useful in all +hierarchies could only be used in one. The issue is exacerbated by +the fact that controllers couldn't be moved to another hierarchy once +hierarchies were populated. Another issue was that all controllers +bound to a hierarchy were forced to have exactly the same view of the +hierarchy. It wasn't possible to vary the granularity depending on +the specific controller. + +In practice, these issues heavily limited which controllers could be +put on the same hierarchy and most configurations resorted to putting +each controller on its own hierarchy. Only closely related ones, such +as the cpu and cpuacct controllers, made sense to be put on the same +hierarchy. This often meant that userland ended up managing multiple +similar hierarchies repeating the same steps on each hierarchy +whenever a hierarchy management operation was necessary. + +Furthermore, support for multiple hierarchies came at a steep cost. +It greatly complicated cgroup core implementation but more importantly +the support for multiple hierarchies restricted how cgroup could be +used in general and what controllers was able to do. + +There was no limit on how many hierarchies there might be, which meant +that a thread's cgroup membership couldn't be described in finite +length. The key might contain any number of entries and was unlimited +in length, which made it highly awkward to manipulate and led to +addition of controllers which existed only to identify membership, +which in turn exacerbated the original problem of proliferating number +of hierarchies. + +Also, as a controller couldn't have any expectation regarding the +topologies of hierarchies other controllers might be on, each +controller had to assume that all other controllers were attached to +completely orthogonal hierarchies. This made it impossible, or at +least very cumbersome, for controllers to cooperate with each other. + +In most use cases, putting controllers on hierarchies which are +completely orthogonal to each other isn't necessary. What usually is +called for is the ability to have differing levels of granularity +depending on the specific controller. In other words, hierarchy may +be collapsed from leaf towards root when viewed from specific +controllers. For example, a given configuration might not care about +how memory is distributed beyond a certain level while still wanting +to control how CPU cycles are distributed. + + +Thread Granularity +------------------ + +cgroup v1 allowed threads of a process to belong to different cgroups. +This didn't make sense for some controllers and those controllers +ended up implementing different ways to ignore such situations but +much more importantly it blurred the line between API exposed to +individual applications and system management interface. + +Generally, in-process knowledge is available only to the process +itself; thus, unlike service-level organization of processes, +categorizing threads of a process requires active participation from +the application which owns the target process. + +cgroup v1 had an ambiguously defined delegation model which got abused +in combination with thread granularity. cgroups were delegated to +individual applications so that they can create and manage their own +sub-hierarchies and control resource distributions along them. This +effectively raised cgroup to the status of a syscall-like API exposed +to lay programs. + +First of all, cgroup has a fundamentally inadequate interface to be +exposed this way. For a process to access its own knobs, it has to +extract the path on the target hierarchy from /proc/self/cgroup, +construct the path by appending the name of the knob to the path, open +and then read and/or write to it. This is not only extremely clunky +and unusual but also inherently racy. There is no conventional way to +define transaction across the required steps and nothing can guarantee +that the process would actually be operating on its own sub-hierarchy. + +cgroup controllers implemented a number of knobs which would never be +accepted as public APIs because they were just adding control knobs to +system-management pseudo filesystem. cgroup ended up with interface +knobs which were not properly abstracted or refined and directly +revealed kernel internal details. These knobs got exposed to +individual applications through the ill-defined delegation mechanism +effectively abusing cgroup as a shortcut to implementing public APIs +without going through the required scrutiny. + +This was painful for both userland and kernel. Userland ended up with +misbehaving and poorly abstracted interfaces and kernel exposing and +locked into constructs inadvertently. + + +Competition Between Inner Nodes and Threads +------------------------------------------- + +cgroup v1 allowed threads to be in any cgroups which created an +interesting problem where threads belonging to a parent cgroup and its +children cgroups competed for resources. This was nasty as two +different types of entities competed and there was no obvious way to +settle it. Different controllers did different things. + +The cpu controller considered threads and cgroups as equivalents and +mapped nice levels to cgroup weights. This worked for some cases but +fell flat when children wanted to be allocated specific ratios of CPU +cycles and the number of internal threads fluctuated - the ratios +constantly changed as the number of competing entities fluctuated. +There also were other issues. The mapping from nice level to weight +wasn't obvious or universal, and there were various other knobs which +simply weren't available for threads. + +The io controller implicitly created a hidden leaf node for each +cgroup to host the threads. The hidden leaf had its own copies of all +the knobs with ``leaf_`` prefixed. While this allowed equivalent +control over internal threads, it was with serious drawbacks. It +always added an extra layer of nesting which wouldn't be necessary +otherwise, made the interface messy and significantly complicated the +implementation. + +The memory controller didn't have a way to control what happened +between internal tasks and child cgroups and the behavior was not +clearly defined. There were attempts to add ad-hoc behaviors and +knobs to tailor the behavior to specific workloads which would have +led to problems extremely difficult to resolve in the long term. + +Multiple controllers struggled with internal tasks and came up with +different ways to deal with it; unfortunately, all the approaches were +severely flawed and, furthermore, the widely different behaviors +made cgroup as a whole highly inconsistent. + +This clearly is a problem which needs to be addressed from cgroup core +in a uniform way. + + +Other Interface Issues +---------------------- + +cgroup v1 grew without oversight and developed a large number of +idiosyncrasies and inconsistencies. One issue on the cgroup core side +was how an empty cgroup was notified - a userland helper binary was +forked and executed for each event. The event delivery wasn't +recursive or delegatable. The limitations of the mechanism also led +to in-kernel event delivery filtering mechanism further complicating +the interface. + +Controller interfaces were problematic too. An extreme example is +controllers completely ignoring hierarchical organization and treating +all cgroups as if they were all located directly under the root +cgroup. Some controllers exposed a large amount of inconsistent +implementation details to userland. + +There also was no consistency across controllers. When a new cgroup +was created, some controllers defaulted to not imposing extra +restrictions while others disallowed any resource usage until +explicitly configured. Configuration knobs for the same type of +control used widely differing naming schemes and formats. Statistics +and information knobs were named arbitrarily and used different +formats and units even in the same controller. + +cgroup v2 establishes common conventions where appropriate and updates +controllers so that they expose minimal and consistent interfaces. + + +Controller Issues and Remedies +------------------------------ + +Memory +~~~~~~ + +The original lower boundary, the soft limit, is defined as a limit +that is per default unset. As a result, the set of cgroups that +global reclaim prefers is opt-in, rather than opt-out. The costs for +optimizing these mostly negative lookups are so high that the +implementation, despite its enormous size, does not even provide the +basic desirable behavior. First off, the soft limit has no +hierarchical meaning. All configured groups are organized in a global +rbtree and treated like equal peers, regardless where they are located +in the hierarchy. This makes subtree delegation impossible. Second, +the soft limit reclaim pass is so aggressive that it not just +introduces high allocation latencies into the system, but also impacts +system performance due to overreclaim, to the point where the feature +becomes self-defeating. + +The memory.low boundary on the other hand is a top-down allocated +reserve. A cgroup enjoys reclaim protection when it's within its low, +which makes delegation of subtrees possible. + +The original high boundary, the hard limit, is defined as a strict +limit that can not budge, even if the OOM killer has to be called. +But this generally goes against the goal of making the most out of the +available memory. The memory consumption of workloads varies during +runtime, and that requires users to overcommit. But doing that with a +strict upper limit requires either a fairly accurate prediction of the +working set size or adding slack to the limit. Since working set size +estimation is hard and error prone, and getting it wrong results in +OOM kills, most users tend to err on the side of a looser limit and +end up wasting precious resources. + +The memory.high boundary on the other hand can be set much more +conservatively. When hit, it throttles allocations by forcing them +into direct reclaim to work off the excess, but it never invokes the +OOM killer. As a result, a high boundary that is chosen too +aggressively will not terminate the processes, but instead it will +lead to gradual performance degradation. The user can monitor this +and make corrections until the minimal memory footprint that still +gives acceptable performance is found. + +In extreme cases, with many concurrent allocations and a complete +breakdown of reclaim progress within the group, the high boundary can +be exceeded. But even then it's mostly better to satisfy the +allocation from the slack available in other groups or the rest of the +system than killing the group. Otherwise, memory.max is there to +limit this type of spillover and ultimately contain buggy or even +malicious applications. + +Setting the original memory.limit_in_bytes below the current usage was +subject to a race condition, where concurrent charges could cause the +limit setting to fail. memory.max on the other hand will first set the +limit to prevent new charges, and then reclaim and OOM kill until the +new limit is met - or the task writing to memory.max is killed. + +The combined memory+swap accounting and limiting is replaced by real +control over swap space. + +The main argument for a combined memory+swap facility in the original +cgroup design was that global or parental pressure would always be +able to swap all anonymous memory of a child group, regardless of the +child's own (possibly untrusted) configuration. However, untrusted +groups can sabotage swapping by other means - such as referencing its +anonymous memory in a tight loop - and an admin can not assume full +swappability when overcommitting untrusted jobs. + +For trusted jobs, on the other hand, a combined counter is not an +intuitive userspace interface, and it flies in the face of the idea +that cgroup controllers should account and limit specific physical +resources. Swap space is a resource like all others in the system, +and that's why unified hierarchy allows distributing it separately. diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst index 5bb9161dbe6a..48d70af11652 100644 --- a/Documentation/admin-guide/index.rst +++ b/Documentation/admin-guide/index.rst @@ -48,6 +48,7 @@ configure specific aspects of kernel behavior to your liking. :maxdepth: 1 initrd + cgroup-v2 serial-console braille-console parport @@ -60,9 +61,11 @@ configure specific aspects of kernel behavior to your liking. mono java ras + bcache pm/index thunderbolt LSM/index + mm/index .. only:: subproject and html diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index f2040d46f095..efc7aa7a0670 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -106,11 +106,11 @@ use by PCI Format: ,... - acpi_mask_gpe= [HW,ACPI] + acpi_mask_gpe= [HW,ACPI] Due to the existence of _Lxx/_Exx, some GPEs triggered by unsupported hardware/firmware features can result in - GPE floodings that cannot be automatically disabled by - the GPE dispatcher. + GPE floodings that cannot be automatically disabled by + the GPE dispatcher. This facility can be used to prevent such uncontrolled GPE floodings. Format: @@ -256,7 +256,7 @@ (may crash computer or cause data corruption) ALSA [HW,ALSA] - See Documentation/sound/alsa/alsa-parameters.txt + See Documentation/sound/alsa-configuration.rst alignment= [KNL,ARM] Allow the default userspace alignment fault handler @@ -472,10 +472,10 @@ for platform specific values (SB1, Loongson3 and others). - ccw_timeout_log [S390] + ccw_timeout_log [S390] See Documentation/s390/CommonIO for details. - cgroup_disable= [KNL] Disable a particular controller + cgroup_disable= [KNL] Disable a particular controller Format: {name of the controller(s) to disable} The effects of cgroup_disable=foo are: - foo isn't auto-mounted if you mount all cgroups in @@ -518,7 +518,7 @@ those clocks in any way. This parameter is useful for debug and development, but should not be needed on a platform with proper driver support. For more - information, see Documentation/clk.txt. + information, see Documentation/driver-api/clk.rst. clock= [BUGS=X86-32, HW] gettimeofday clocksource override. [Deprecated] @@ -587,11 +587,6 @@ Sets the size of memory pool for coherent, atomic dma allocations, by default set to 256K. - code_bytes [X86] How many bytes of object code to print - in an oops report. - Range: 0 - 8192 - Default: 64 - com20020= [HW,NET] ARCnet - COM20020 chipset Format: [,[,[,[,[,]]]]] @@ -641,8 +636,8 @@ hvc Use the hypervisor console device . This is for both Xen and PowerPC hypervisors. - If the device connected to the port is not a TTY but a braille - device, prepend "brl," before the device type, for instance + If the device connected to the port is not a TTY but a braille + device, prepend "brl," before the device type, for instance console=brl,ttyS0 For now, only VisioBraille is supported. @@ -662,7 +657,7 @@ consoleblank= [KNL] The console blank (screen saver) timeout in seconds. A value of 0 disables the blank timer. - Defaults to 0. + Defaults to 0. coredump_filter= [KNL] Change the default value for @@ -730,7 +725,7 @@ or memory reserved is below 4G. cryptomgr.notests - [KNL] Disable crypto self-tests + [KNL] Disable crypto self-tests cs89x0_dma= [HW,NET] Format: @@ -746,7 +741,7 @@ Format: , See also Documentation/input/devices/joystick-parport.rst - ddebug_query= [KNL,DYNAMIC_DEBUG] Enable debug messages at early boot + ddebug_query= [KNL,DYNAMIC_DEBUG] Enable debug messages at early boot time. See Documentation/admin-guide/dynamic-debug-howto.rst for details. Deprecated, see dyndbg. @@ -833,7 +828,7 @@ causing system reset or hang due to sending INIT from AP to BSP. - disable_ddw [PPC/PSERIES] + disable_ddw [PPC/PSERIES] Disable Dynamic DMA Window support. Use this if to workaround buggy firmware. @@ -1025,6 +1020,12 @@ address. The serial port must already be setup and configured. Options are not yet supported. + qcom_geni, + Start an early, polled-mode console on a Qualcomm + Generic Interface (GENI) based serial port at the + specified address. The serial port must already be + setup and configured. Options are not yet supported. + earlyprintk= [X86,SH,ARM,M68k,S390] earlyprintk=vga earlyprintk=efi @@ -1188,7 +1189,7 @@ parameter will force ia64_sal_cache_flush to call ia64_pal_cache_flush instead of SAL_CACHE_FLUSH. - forcepae [X86-32] + forcepae [X86-32] Forcefully enable Physical Address Extension (PAE). Many Pentium M systems disable PAE but may have a functionally usable PAE implementation. @@ -1247,7 +1248,7 @@ gamma= [HW,DRM] - gart_fix_e820= [X86_64] disable the fix e820 for K8 GART + gart_fix_e820= [X86_64] disable the fix e820 for K8 GART Format: off | on default: on @@ -1341,23 +1342,32 @@ x86-64 are 2M (when the CPU supports "pse") and 1G (when the CPU supports the "pdpe1gb" cpuinfo flag). - hvc_iucv= [S390] Number of z/VM IUCV hypervisor console (HVC) - terminal devices. Valid values: 0..8 - hvc_iucv_allow= [S390] Comma-separated list of z/VM user IDs. - If specified, z/VM IUCV HVC accepts connections - from listed z/VM user IDs only. + hung_task_panic= + [KNL] Should the hung task detector generate panics. + Format: + A nonzero value instructs the kernel to panic when a + hung task is detected. The default value is controlled + by the CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time + option. The value selected by this boot parameter can + be changed later by the kernel.hung_task_panic sysctl. + + hvc_iucv= [S390] Number of z/VM IUCV hypervisor console (HVC) + terminal devices. Valid values: 0..8 + hvc_iucv_allow= [S390] Comma-separated list of z/VM user IDs. + If specified, z/VM IUCV HVC accepts connections + from listed z/VM user IDs only. keep_bootcon [KNL] Do not unregister boot console at start. This is only useful for debugging when something happens in the window between unregistering the boot console and initializing the real console. - i2c_bus= [HW] Override the default board specific I2C bus speed - or register an additional I2C bus that is not - registered from board initialization code. - Format: - , + i2c_bus= [HW] Override the default board specific I2C bus speed + or register an additional I2C bus that is not + registered from board initialization code. + Format: + , i8042.debug [HW] Toggle i8042 debug mode i8042.unmask_kbd_data @@ -1386,7 +1396,7 @@ Default: only on s2r transitions on x86; most other architectures force reset to be always executed i8042.unlock [HW] Unlock (ignore) the keylock - i8042.kbdreset [HW] Reset device connected to KBD port + i8042.kbdreset [HW] Reset device connected to KBD port i810= [HW,DRM] @@ -1548,13 +1558,13 @@ programs exec'd, files mmap'd for exec, and all files opened for read by uid=0. - ima_template= [IMA] + ima_template= [IMA] Select one of defined IMA measurements template formats. Formats: { "ima" | "ima-ng" | "ima-sig" } Default: "ima-ng" ima_template_fmt= - [IMA] Define a custom template format. + [IMA] Define a custom template format. Format: { "field1|...|fieldN" } ima.ahash_minsize= [IMA] Minimum file size for asynchronous hash usage @@ -1597,7 +1607,7 @@ inport.irq= [HW] Inport (ATI XL and Microsoft) busmouse driver Format: - int_pln_enable [x86] Enable power limit notification interrupt + int_pln_enable [x86] Enable power limit notification interrupt integrity_audit=[IMA] Format: { "0" | "1" } @@ -1650,39 +1660,39 @@ 0 disables intel_idle and fall back on acpi_idle. 1 to 9 specify maximum depth of C-state. - intel_pstate= [X86] - disable - Do not enable intel_pstate as the default - scaling driver for the supported processors - passive - Use intel_pstate as a scaling driver, but configure it - to work with generic cpufreq governors (instead of - enabling its internal governor). This mode cannot be - used along with the hardware-managed P-states (HWP) - feature. - force - Enable intel_pstate on systems that prohibit it by default - in favor of acpi-cpufreq. Forcing the intel_pstate driver - instead of acpi-cpufreq may disable platform features, such - as thermal controls and power capping, that rely on ACPI - P-States information being indicated to OSPM and therefore - should be used with caution. This option does not work with - processors that aren't supported by the intel_pstate driver - or on platforms that use pcc-cpufreq instead of acpi-cpufreq. - no_hwp - Do not enable hardware P state control (HWP) - if available. - hwp_only - Only load intel_pstate on systems which support - hardware P state control (HWP) if available. - support_acpi_ppc - Enforce ACPI _PPC performance limits. If the Fixed ACPI - Description Table, specifies preferred power management - profile as "Enterprise Server" or "Performance Server", - then this feature is turned on by default. - per_cpu_perf_limits - Allow per-logical-CPU P-State performance control limits using - cpufreq sysfs interface + intel_pstate= [X86] + disable + Do not enable intel_pstate as the default + scaling driver for the supported processors + passive + Use intel_pstate as a scaling driver, but configure it + to work with generic cpufreq governors (instead of + enabling its internal governor). This mode cannot be + used along with the hardware-managed P-states (HWP) + feature. + force + Enable intel_pstate on systems that prohibit it by default + in favor of acpi-cpufreq. Forcing the intel_pstate driver + instead of acpi-cpufreq may disable platform features, such + as thermal controls and power capping, that rely on ACPI + P-States information being indicated to OSPM and therefore + should be used with caution. This option does not work with + processors that aren't supported by the intel_pstate driver + or on platforms that use pcc-cpufreq instead of acpi-cpufreq. + no_hwp + Do not enable hardware P state control (HWP) + if available. + hwp_only + Only load intel_pstate on systems which support + hardware P state control (HWP) if available. + support_acpi_ppc + Enforce ACPI _PPC performance limits. If the Fixed ACPI + Description Table, specifies preferred power management + profile as "Enterprise Server" or "Performance Server", + then this feature is turned on by default. + per_cpu_perf_limits + Allow per-logical-CPU P-State performance control limits using + cpufreq sysfs interface intremap= [X86-64, Intel-IOMMU] on enable Interrupt Remapping (default) @@ -1705,7 +1715,6 @@ nopanic merge nomerge - forcesac soft pt [x86, IA-64] nobypass [PPC/POWERNV] @@ -2027,7 +2036,7 @@ * [no]ncqtrim: Turn off queued DSM TRIM. * nohrst, nosrst, norst: suppress hard, soft - and both resets. + and both resets. * rstonce: only attempt one reset during hot-unplug link recovery @@ -2215,7 +2224,7 @@ [KNL,SH] Allow user to override the default size for per-device physically contiguous DMA buffers. - memhp_default_state=online/offline + memhp_default_state=online/offline [KNL] Set the initial state for the memory hotplug onlining policy. If not specified, the default value is set according to the @@ -2600,6 +2609,9 @@ emulation library even if a 387 maths coprocessor is present. + no5lvl [X86-64] Disable 5-level paging mode. Forces + kernel to use 4-level paging instead. + no_console_suspend [HW] Never suspend the console Disable suspending of consoles during suspend and @@ -2765,7 +2777,7 @@ [X86,PV_OPS] Disable paravirtualized VMware scheduler clock and use the default one. - no-steal-acc [X86,KVM] Disable paravirtualized steal time accounting. + no-steal-acc [X86,KVM] Disable paravirtualized steal time accounting. steal time is computed, but won't influence scheduler behaviour @@ -2826,7 +2838,7 @@ notsc [BUGS=X86-32] Disable Time Stamp Counter nowatchdog [KNL] Disable both lockup detectors, i.e. - soft-lockup and NMI watchdog (hard-lockup). + soft-lockup and NMI watchdog (hard-lockup). nowb [ARM] @@ -2846,7 +2858,7 @@ If the dependencies are under your control, you can turn on cpu0_hotplug. - nps_mtm_hs_ctr= [KNL,ARC] + nps_mtm_hs_ctr= [KNL,ARC] This parameter sets the maximum duration, in cycles, each HW thread of the CTOP can run without interruptions, before HW switches it. @@ -2914,9 +2926,6 @@ This will also cause panics on machine check exceptions. Useful together with panic=30 to trigger a reboot. - OSS [HW,OSS] - See Documentation/sound/oss/oss-parameters.txt - page_owner= [KNL] Boot-time page_owner enabling option. Storage of the information about who allocated each page is disabled in default. With this switch, @@ -2987,7 +2996,7 @@ pci=option[,option...] [PCI] various PCI subsystem options: earlydump [X86] dump PCI config space before the kernel - changes anything + changes anything off [X86] don't probe for the PCI bus bios [X86-32] force use of PCI BIOS, don't access the hardware directly. Use this if your machine @@ -3075,7 +3084,7 @@ is enabled by default. If you need to use this, please report a bug. nocrs [X86] Ignore PCI host bridge windows from ACPI. - If you need to use this, please report a bug. + If you need to use this, please report a bug. routeirq Do IRQ routing for all PCI devices. This is normally done in pci_enable_device(), so this option is a temporary workaround @@ -3150,6 +3159,8 @@ on: Turn realloc on realloc same as realloc=on noari do not use PCIe ARI. + noats [PCIE, Intel-IOMMU, AMD-IOMMU] + do not use PCIe ATS (and IOMMU device IOTLB). pcie_scan_all Scan all possible PCIe devices. Otherwise we only look for one device below a PCIe downstream port. @@ -3918,7 +3929,7 @@ cache (risks via metadata attacks are mostly unchanged). Debug options disable merging on their own. - For more information see Documentation/vm/slub.txt. + For more information see Documentation/vm/slub.rst. slab_max_order= [MM, SLAB] Determines the maximum allowed order for slabs. @@ -3932,7 +3943,7 @@ slub_debug can create guard zones around objects and may poison objects when not in use. Also tracks the last alloc / free. For more information see - Documentation/vm/slub.txt. + Documentation/vm/slub.rst. slub_memcg_sysfs= [MM, SLUB] Determines whether to enable sysfs directories for @@ -3946,7 +3957,7 @@ Determines the maximum allowed order for slabs. A high setting may cause OOMs due to memory fragmentation. For more information see - Documentation/vm/slub.txt. + Documentation/vm/slub.rst. slub_min_objects= [MM, SLUB] The minimum number of objects per slab. SLUB will @@ -3955,12 +3966,12 @@ the number of objects indicated. The higher the number of objects the smaller the overhead of tracking slabs and the less frequently locks need to be acquired. - For more information see Documentation/vm/slub.txt. + For more information see Documentation/vm/slub.rst. slub_min_order= [MM, SLUB] Determines the minimum page order for slabs. Must be lower than slub_max_order. - For more information see Documentation/vm/slub.txt. + For more information see Documentation/vm/slub.rst. slub_nomerge [MM, SLUB] Same with slab_nomerge. This is supported for legacy. @@ -4092,6 +4103,23 @@ expediting. Set to zero to disable automatic expediting. + ssbd= [ARM64,HW] + Speculative Store Bypass Disable control + + On CPUs that are vulnerable to the Speculative + Store Bypass vulnerability and offer a + firmware based mitigation, this parameter + indicates how the mitigation should be used: + + force-on: Unconditionally enable mitigation for + for both kernel and userspace + force-off: Unconditionally disable mitigation for + for both kernel and userspace + kernel: Always enable mitigation in the + kernel, and offer a prctl interface + to allow userspace to register its + interest in being mitigated too. + stack_guard_gap= [MM] override the default stack gap protection. The value is in page units and it defines how many pages prior @@ -4304,7 +4332,7 @@ [FTRACE] Set and start specified trace events in order to facilitate early boot debugging. The event-list is a comma separated list of trace events to enable. See - also Documentation/trace/events.txt + also Documentation/trace/events.rst trace_options=[option-list] [FTRACE] Enable or disable tracer options at boot. @@ -4319,7 +4347,7 @@ trace_options=stacktrace - See also Documentation/trace/ftrace.txt "trace options" + See also Documentation/trace/ftrace.rst "trace options" section. tp_printk[FTRACE] @@ -4358,7 +4386,8 @@ Format: [always|madvise|never] Can be used to control the default behavior of the system with respect to transparent hugepages. - See Documentation/vm/transhuge.txt for more details. + See Documentation/admin-guide/mm/transhuge.rst + for more details. tsc= Disable clocksource stability checks for TSC. Format: @@ -4436,7 +4465,7 @@ usbcore.initial_descriptor_timeout= [USB] Specifies timeout for the initial 64-byte - USB_REQ_GET_DESCRIPTOR request in milliseconds + USB_REQ_GET_DESCRIPTOR request in milliseconds (default 5000 = 5.0 seconds). usbcore.nousb [USB] Disable the USB subsystem diff --git a/Documentation/admin-guide/mm/concepts.rst b/Documentation/admin-guide/mm/concepts.rst new file mode 100644 index 000000000000..291699c810d4 --- /dev/null +++ b/Documentation/admin-guide/mm/concepts.rst @@ -0,0 +1,222 @@ +.. _mm_concepts: + +================= +Concepts overview +================= + +The memory management in Linux is complex system that evolved over the +years and included more and more functionality to support variety of +systems from MMU-less microcontrollers to supercomputers. The memory +management for systems without MMU is called ``nommu`` and it +definitely deserves a dedicated document, which hopefully will be +eventually written. Yet, although some of the concepts are the same, +here we assume that MMU is available and CPU can translate a virtual +address to a physical address. + +.. contents:: :local: + +Virtual Memory Primer +===================== + +The physical memory in a computer system is a limited resource and +even for systems that support memory hotplug there is a hard limit on +the amount of memory that can be installed. The physical memory is not +necessary contiguous, it might be accessible as a set of distinct +address ranges. Besides, different CPU architectures, and even +different implementations of the same architecture have different view +how these address ranges defined. + +All this makes dealing directly with physical memory quite complex and +to avoid this complexity a concept of virtual memory was developed. + +The virtual memory abstracts the details of physical memory from the +application software, allows to keep only needed information in the +physical memory (demand paging) and provides a mechanism for the +protection and controlled sharing of data between processes. + +With virtual memory, each and every memory access uses a virtual +address. When the CPU decodes the an instruction that reads (or +writes) from (or to) the system memory, it translates the `virtual` +address encoded in that instruction to a `physical` address that the +memory controller can understand. + +The physical system memory is divided into page frames, or pages. The +size of each page is architecture specific. Some architectures allow +selection of the page size from several supported values; this +selection is performed at the kernel build time by setting an +appropriate kernel configuration option. + +Each physical memory page can be mapped as one or more virtual +pages. These mappings are described by page tables that allow +translation from virtual address used by programs to real address in +the physical memory. The page tables organized hierarchically. + +The tables at the lowest level of the hierarchy contain physical +addresses of actual pages used by the software. The tables at higher +levels contain physical addresses of the pages belonging to the lower +levels. The pointer to the top level page table resides in a +register. When the CPU performs the address translation, it uses this +register to access the top level page table. The high bits of the +virtual address are used to index an entry in the top level page +table. That entry is then used to access the next level in the +hierarchy with the next bits of the virtual address as the index to +that level page table. The lowest bits in the virtual address define +the offset inside the actual page. + +Huge Pages +========== + +The address translation requires several memory accesses and memory +accesses are slow relatively to CPU speed. To avoid spending precious +processor cycles on the address translation, CPUs maintain a cache of +such translations called Translation Lookaside Buffer (or +TLB). Usually TLB is pretty scarce resource and applications with +large memory working set will experience performance hit because of +TLB misses. + +Many modern CPU architectures allow mapping of the memory pages +directly by the higher levels in the page table. For instance, on x86, +it is possible to map 2M and even 1G pages using entries in the second +and the third level page tables. In Linux such pages are called +`huge`. Usage of huge pages significantly reduces pressure on TLB, +improves TLB hit-rate and thus improves overall system performance. + +There are two mechanisms in Linux that enable mapping of the physical +memory with the huge pages. The first one is `HugeTLB filesystem`, or +hugetlbfs. It is a pseudo filesystem that uses RAM as its backing +store. For the files created in this filesystem the data resides in +the memory and mapped using huge pages. The hugetlbfs is described at +:ref:`Documentation/admin-guide/mm/hugetlbpage.rst `. + +Another, more recent, mechanism that enables use of the huge pages is +called `Transparent HugePages`, or THP. Unlike the hugetlbfs that +requires users and/or system administrators to configure what parts of +the system memory should and can be mapped by the huge pages, THP +manages such mappings transparently to the user and hence the +name. See +:ref:`Documentation/admin-guide/mm/transhuge.rst ` +for more details about THP. + +Zones +===== + +Often hardware poses restrictions on how different physical memory +ranges can be accessed. In some cases, devices cannot perform DMA to +all the addressable memory. In other cases, the size of the physical +memory exceeds the maximal addressable size of virtual memory and +special actions are required to access portions of the memory. Linux +groups memory pages into `zones` according to their possible +usage. For example, ZONE_DMA will contain memory that can be used by +devices for DMA, ZONE_HIGHMEM will contain memory that is not +permanently mapped into kernel's address space and ZONE_NORMAL will +contain normally addressed pages. + +The actual layout of the memory zones is hardware dependent as not all +architectures define all zones, and requirements for DMA are different +for different platforms. + +Nodes +===== + +Many multi-processor machines are NUMA - Non-Uniform Memory Access - +systems. In such systems the memory is arranged into banks that have +different access latency depending on the "distance" from the +processor. Each bank is referred as `node` and for each node Linux +constructs an independent memory management subsystem. A node has it's +own set of zones, lists of free and used pages and various statistics +counters. You can find more details about NUMA in +:ref:`Documentation/vm/numa.rst ` and in +:ref:`Documentation/admin-guide/mm/numa_memory_policy.rst `. + +Page cache +========== + +The physical memory is volatile and the common case for getting data +into the memory is to read it from files. Whenever a file is read, the +data is put into the `page cache` to avoid expensive disk access on +the subsequent reads. Similarly, when one writes to a file, the data +is placed in the page cache and eventually gets into the backing +storage device. The written pages are marked as `dirty` and when Linux +decides to reuse them for other purposes, it makes sure to synchronize +the file contents on the device with the updated data. + +Anonymous Memory +================ + +The `anonymous memory` or `anonymous mappings` represent memory that +is not backed by a filesystem. Such mappings are implicitly created +for program's stack and heap or by explicit calls to mmap(2) system +call. Usually, the anonymous mappings only define virtual memory areas +that the program is allowed to access. The read accesses will result +in creation of a page table entry that references a special physical +page filled with zeroes. When the program performs a write, regular +physical page will be allocated to hold the written data. The page +will be marked dirty and if the kernel will decide to repurpose it, +the dirty page will be swapped out. + +Reclaim +======= + +Throughout the system lifetime, a physical page can be used for storing +different types of data. It can be kernel internal data structures, +DMA'able buffers for device drivers use, data read from a filesystem, +memory allocated by user space processes etc. + +Depending on the page usage it is treated differently by the Linux +memory management. The pages that can be freed at any time, either +because they cache the data available elsewhere, for instance, on a +hard disk, or because they can be swapped out, again, to the hard +disk, are called `reclaimable`. The most notable categories of the +reclaimable pages are page cache and anonymous memory. + +In most cases, the pages holding internal kernel data and used as DMA +buffers cannot be repurposed, and they remain pinned until freed by +their user. Such pages are called `unreclaimable`. However, in certain +circumstances, even pages occupied with kernel data structures can be +reclaimed. For instance, in-memory caches of filesystem metadata can +be re-read from the storage device and therefore it is possible to +discard them from the main memory when system is under memory +pressure. + +The process of freeing the reclaimable physical memory pages and +repurposing them is called (surprise!) `reclaim`. Linux can reclaim +pages either asynchronously or synchronously, depending on the state +of the system. When system is not loaded, most of the memory is free +and allocation request will be satisfied immediately from the free +pages supply. As the load increases, the amount of the free pages goes +down and when it reaches a certain threshold (high watermark), an +allocation request will awaken the ``kswapd`` daemon. It will +asynchronously scan memory pages and either just free them if the data +they contain is available elsewhere, or evict to the backing storage +device (remember those dirty pages?). As memory usage increases even +more and reaches another threshold - min watermark - an allocation +will trigger the `direct reclaim`. In this case allocation is stalled +until enough memory pages are reclaimed to satisfy the request. + +Compaction +========== + +As the system runs, tasks allocate and free the memory and it becomes +fragmented. Although with virtual memory it is possible to present +scattered physical pages as virtually contiguous range, sometimes it is +necessary to allocate large physically contiguous memory areas. Such +need may arise, for instance, when a device driver requires large +buffer for DMA, or when THP allocates a huge page. Memory `compaction` +addresses the fragmentation issue. This mechanism moves occupied pages +from the lower part of a memory zone to free pages in the upper part +of the zone. When a compaction scan is finished free pages are grouped +together at the beginning of the zone and allocations of large +physically contiguous areas become possible. + +Like reclaim, the compaction may happen asynchronously in ``kcompactd`` +daemon or synchronously as a result of memory allocation request. + +OOM killer +========== + +It may happen, that on a loaded machine memory will be exhausted. When +the kernel detects that the system runs out of memory (OOM) it invokes +`OOM killer`. Its mission is simple: all it has to do is to select a +task to sacrifice for the sake of the overall system health. The +selected task is killed in a hope that after it exits enough memory +will be freed to continue normal operation. diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst new file mode 100644 index 000000000000..1cc0bc78d10e --- /dev/null +++ b/Documentation/admin-guide/mm/hugetlbpage.rst @@ -0,0 +1,382 @@ +.. _hugetlbpage: + +============= +HugeTLB Pages +============= + +Overview +======== + +The intent of this file is to give a brief summary of hugetlbpage support in +the Linux kernel. This support is built on top of multiple page size support +that is provided by most modern architectures. For example, x86 CPUs normally +support 4K and 2M (1G if architecturally supported) page sizes, ia64 +architecture supports multiple page sizes 4K, 8K, 64K, 256K, 1M, 4M, 16M, +256M and ppc64 supports 4K and 16M. A TLB is a cache of virtual-to-physical +translations. Typically this is a very scarce resource on processor. +Operating systems try to make best use of limited number of TLB resources. +This optimization is more critical now as bigger and bigger physical memories +(several GBs) are more readily available. + +Users can use the huge page support in Linux kernel by either using the mmap +system call or standard SYSV shared memory system calls (shmget, shmat). + +First the Linux kernel needs to be built with the CONFIG_HUGETLBFS +(present under "File systems") and CONFIG_HUGETLB_PAGE (selected +automatically when CONFIG_HUGETLBFS is selected) configuration +options. + +The ``/proc/meminfo`` file provides information about the total number of +persistent hugetlb pages in the kernel's huge page pool. It also displays +default huge page size and information about the number of free, reserved +and surplus huge pages in the pool of huge pages of default size. +The huge page size is needed for generating the proper alignment and +size of the arguments to system calls that map huge page regions. + +The output of ``cat /proc/meminfo`` will include lines like:: + + HugePages_Total: uuu + HugePages_Free: vvv + HugePages_Rsvd: www + HugePages_Surp: xxx + Hugepagesize: yyy kB + Hugetlb: zzz kB + +where: + +HugePages_Total + is the size of the pool of huge pages. +HugePages_Free + is the number of huge pages in the pool that are not yet + allocated. +HugePages_Rsvd + is short for "reserved," and is the number of huge pages for + which a commitment to allocate from the pool has been made, + but no allocation has yet been made. Reserved huge pages + guarantee that an application will be able to allocate a + huge page from the pool of huge pages at fault time. +HugePages_Surp + is short for "surplus," and is the number of huge pages in + the pool above the value in ``/proc/sys/vm/nr_hugepages``. The + maximum number of surplus huge pages is controlled by + ``/proc/sys/vm/nr_overcommit_hugepages``. +Hugepagesize + is the default hugepage size (in Kb). +Hugetlb + is the total amount of memory (in kB), consumed by huge + pages of all sizes. + If huge pages of different sizes are in use, this number + will exceed HugePages_Total \* Hugepagesize. To get more + detailed information, please, refer to + ``/sys/kernel/mm/hugepages`` (described below). + + +``/proc/filesystems`` should also show a filesystem of type "hugetlbfs" +configured in the kernel. + +``/proc/sys/vm/nr_hugepages`` indicates the current number of "persistent" huge +pages in the kernel's huge page pool. "Persistent" huge pages will be +returned to the huge page pool when freed by a task. A user with root +privileges can dynamically allocate more or free some persistent huge pages +by increasing or decreasing the value of ``nr_hugepages``. + +Pages that are used as huge pages are reserved inside the kernel and cannot +be used for other purposes. Huge pages cannot be swapped out under +memory pressure. + +Once a number of huge pages have been pre-allocated to the kernel huge page +pool, a user with appropriate privilege can use either the mmap system call +or shared memory system calls to use the huge pages. See the discussion of +:ref:`Using Huge Pages `, below. + +The administrator can allocate persistent huge pages on the kernel boot +command line by specifying the "hugepages=N" parameter, where 'N' = the +number of huge pages requested. This is the most reliable method of +allocating huge pages as memory has not yet become fragmented. + +Some platforms support multiple huge page sizes. To allocate huge pages +of a specific size, one must precede the huge pages boot command parameters +with a huge page size selection parameter "hugepagesz=". must +be specified in bytes with optional scale suffix [kKmMgG]. The default huge +page size may be selected with the "default_hugepagesz=" boot parameter. + +When multiple huge page sizes are supported, ``/proc/sys/vm/nr_hugepages`` +indicates the current number of pre-allocated huge pages of the default size. +Thus, one can use the following command to dynamically allocate/deallocate +default sized persistent huge pages:: + + echo 20 > /proc/sys/vm/nr_hugepages + +This command will try to adjust the number of default sized huge pages in the +huge page pool to 20, allocating or freeing huge pages, as required. + +On a NUMA platform, the kernel will attempt to distribute the huge page pool +over all the set of allowed nodes specified by the NUMA memory policy of the +task that modifies ``nr_hugepages``. The default for the allowed nodes--when the +task has default memory policy--is all on-line nodes with memory. Allowed +nodes with insufficient available, contiguous memory for a huge page will be +silently skipped when allocating persistent huge pages. See the +:ref:`discussion below ` +of the interaction of task memory policy, cpusets and per node attributes +with the allocation and freeing of persistent huge pages. + +The success or failure of huge page allocation depends on the amount of +physically contiguous memory that is present in system at the time of the +allocation attempt. If the kernel is unable to allocate huge pages from +some nodes in a NUMA system, it will attempt to make up the difference by +allocating extra pages on other nodes with sufficient available contiguous +memory, if any. + +System administrators may want to put this command in one of the local rc +init files. This will enable the kernel to allocate huge pages early in +the boot process when the possibility of getting physical contiguous pages +is still very high. Administrators can verify the number of huge pages +actually allocated by checking the sysctl or meminfo. To check the per node +distribution of huge pages in a NUMA system, use:: + + cat /sys/devices/system/node/node*/meminfo | fgrep Huge + +``/proc/sys/vm/nr_overcommit_hugepages`` specifies how large the pool of +huge pages can grow, if more huge pages than ``/proc/sys/vm/nr_hugepages`` are +requested by applications. Writing any non-zero value into this file +indicates that the hugetlb subsystem is allowed to try to obtain that +number of "surplus" huge pages from the kernel's normal page pool, when the +persistent huge page pool is exhausted. As these surplus huge pages become +unused, they are freed back to the kernel's normal page pool. + +When increasing the huge page pool size via ``nr_hugepages``, any existing +surplus pages will first be promoted to persistent huge pages. Then, additional +huge pages will be allocated, if necessary and if possible, to fulfill +the new persistent huge page pool size. + +The administrator may shrink the pool of persistent huge pages for +the default huge page size by setting the ``nr_hugepages`` sysctl to a +smaller value. The kernel will attempt to balance the freeing of huge pages +across all nodes in the memory policy of the task modifying ``nr_hugepages``. +Any free huge pages on the selected nodes will be freed back to the kernel's +normal page pool. + +Caveat: Shrinking the persistent huge page pool via ``nr_hugepages`` such that +it becomes less than the number of huge pages in use will convert the balance +of the in-use huge pages to surplus huge pages. This will occur even if +the number of surplus pages would exceed the overcommit value. As long as +this condition holds--that is, until ``nr_hugepages+nr_overcommit_hugepages`` is +increased sufficiently, or the surplus huge pages go out of use and are freed-- +no more surplus huge pages will be allowed to be allocated. + +With support for multiple huge page pools at run-time available, much of +the huge page userspace interface in ``/proc/sys/vm`` has been duplicated in +sysfs. +The ``/proc`` interfaces discussed above have been retained for backwards +compatibility. The root huge page control directory in sysfs is:: + + /sys/kernel/mm/hugepages + +For each huge page size supported by the running kernel, a subdirectory +will exist, of the form:: + + hugepages-${size}kB + +Inside each of these directories, the same set of files will exist:: + + nr_hugepages + nr_hugepages_mempolicy + nr_overcommit_hugepages + free_hugepages + resv_hugepages + surplus_hugepages + +which function as described above for the default huge page-sized case. + +.. _mem_policy_and_hp_alloc: + +Interaction of Task Memory Policy with Huge Page Allocation/Freeing +=================================================================== + +Whether huge pages are allocated and freed via the ``/proc`` interface or +the ``/sysfs`` interface using the ``nr_hugepages_mempolicy`` attribute, the +NUMA nodes from which huge pages are allocated or freed are controlled by the +NUMA memory policy of the task that modifies the ``nr_hugepages_mempolicy`` +sysctl or attribute. When the ``nr_hugepages`` attribute is used, mempolicy +is ignored. + +The recommended method to allocate or free huge pages to/from the kernel +huge page pool, using the ``nr_hugepages`` example above, is:: + + numactl --interleave echo 20 \ + >/proc/sys/vm/nr_hugepages_mempolicy + +or, more succinctly:: + + numactl -m echo 20 >/proc/sys/vm/nr_hugepages_mempolicy + +This will allocate or free ``abs(20 - nr_hugepages)`` to or from the nodes +specified in , depending on whether number of persistent huge pages +is initially less than or greater than 20, respectively. No huge pages will be +allocated nor freed on any node not included in the specified . + +When adjusting the persistent hugepage count via ``nr_hugepages_mempolicy``, any +memory policy mode--bind, preferred, local or interleave--may be used. The +resulting effect on persistent huge page allocation is as follows: + +#. Regardless of mempolicy mode [see + :ref:`Documentation/admin-guide/mm/numa_memory_policy.rst `], + persistent huge pages will be distributed across the node or nodes + specified in the mempolicy as if "interleave" had been specified. + However, if a node in the policy does not contain sufficient contiguous + memory for a huge page, the allocation will not "fallback" to the nearest + neighbor node with sufficient contiguous memory. To do this would cause + undesirable imbalance in the distribution of the huge page pool, or + possibly, allocation of persistent huge pages on nodes not allowed by + the task's memory policy. + +#. One or more nodes may be specified with the bind or interleave policy. + If more than one node is specified with the preferred policy, only the + lowest numeric id will be used. Local policy will select the node where + the task is running at the time the nodes_allowed mask is constructed. + For local policy to be deterministic, the task must be bound to a cpu or + cpus in a single node. Otherwise, the task could be migrated to some + other node at any time after launch and the resulting node will be + indeterminate. Thus, local policy is not very useful for this purpose. + Any of the other mempolicy modes may be used to specify a single node. + +#. The nodes allowed mask will be derived from any non-default task mempolicy, + whether this policy was set explicitly by the task itself or one of its + ancestors, such as numactl. This means that if the task is invoked from a + shell with non-default policy, that policy will be used. One can specify a + node list of "all" with numactl --interleave or --membind [-m] to achieve + interleaving over all nodes in the system or cpuset. + +#. Any task mempolicy specified--e.g., using numactl--will be constrained by + the resource limits of any cpuset in which the task runs. Thus, there will + be no way for a task with non-default policy running in a cpuset with a + subset of the system nodes to allocate huge pages outside the cpuset + without first moving to a cpuset that contains all of the desired nodes. + +#. Boot-time huge page allocation attempts to distribute the requested number + of huge pages over all on-lines nodes with memory. + +Per Node Hugepages Attributes +============================= + +A subset of the contents of the root huge page control directory in sysfs, +described above, will be replicated under each the system device of each +NUMA node with memory in:: + + /sys/devices/system/node/node[0-9]*/hugepages/ + +Under this directory, the subdirectory for each supported huge page size +contains the following attribute files:: + + nr_hugepages + free_hugepages + surplus_hugepages + +The free\_' and surplus\_' attribute files are read-only. They return the number +of free and surplus [overcommitted] huge pages, respectively, on the parent +node. + +The ``nr_hugepages`` attribute returns the total number of huge pages on the +specified node. When this attribute is written, the number of persistent huge +pages on the parent node will be adjusted to the specified value, if sufficient +resources exist, regardless of the task's mempolicy or cpuset constraints. + +Note that the number of overcommit and reserve pages remain global quantities, +as we don't know until fault time, when the faulting task's mempolicy is +applied, from which node the huge page allocation will be attempted. + +.. _using_huge_pages: + +Using Huge Pages +================ + +If the user applications are going to request huge pages using mmap system +call, then it is required that system administrator mount a file system of +type hugetlbfs:: + + mount -t hugetlbfs \ + -o uid=,gid=,mode=,pagesize=,size=,\ + min_size=,nr_inodes= none /mnt/huge + +This command mounts a (pseudo) filesystem of type hugetlbfs on the directory +``/mnt/huge``. Any file created on ``/mnt/huge`` uses huge pages. + +The ``uid`` and ``gid`` options sets the owner and group of the root of the +file system. By default the ``uid`` and ``gid`` of the current process +are taken. + +The ``mode`` option sets the mode of root of file system to value & 01777. +This value is given in octal. By default the value 0755 is picked. + +If the platform supports multiple huge page sizes, the ``pagesize`` option can +be used to specify the huge page size and associated pool. ``pagesize`` +is specified in bytes. If ``pagesize`` is not specified the platform's +default huge page size and associated pool will be used. + +The ``size`` option sets the maximum value of memory (huge pages) allowed +for that filesystem (``/mnt/huge``). The ``size`` option can be specified +in bytes, or as a percentage of the specified huge page pool (``nr_hugepages``). +The size is rounded down to HPAGE_SIZE boundary. + +The ``min_size`` option sets the minimum value of memory (huge pages) allowed +for the filesystem. ``min_size`` can be specified in the same way as ``size``, +either bytes or a percentage of the huge page pool. +At mount time, the number of huge pages specified by ``min_size`` are reserved +for use by the filesystem. +If there are not enough free huge pages available, the mount will fail. +As huge pages are allocated to the filesystem and freed, the reserve count +is adjusted so that the sum of allocated and reserved huge pages is always +at least ``min_size``. + +The option ``nr_inodes`` sets the maximum number of inodes that ``/mnt/huge`` +can use. + +If the ``size``, ``min_size`` or ``nr_inodes`` option is not provided on +command line then no limits are set. + +For ``pagesize``, ``size``, ``min_size`` and ``nr_inodes`` options, you can +use [G|g]/[M|m]/[K|k] to represent giga/mega/kilo. +For example, size=2K has the same meaning as size=2048. + +While read system calls are supported on files that reside on hugetlb +file systems, write system calls are not. + +Regular chown, chgrp, and chmod commands (with right permissions) could be +used to change the file attributes on hugetlbfs. + +Also, it is important to note that no such mount command is required if +applications are going to use only shmat/shmget system calls or mmap with +MAP_HUGETLB. For an example of how to use mmap with MAP_HUGETLB see +:ref:`map_hugetlb ` below. + +Users who wish to use hugetlb memory via shared memory segment should be +members of a supplementary group and system admin needs to configure that gid +into ``/proc/sys/vm/hugetlb_shm_group``. It is possible for same or different +applications to use any combination of mmaps and shm* calls, though the mount of +filesystem will be required for using mmap calls without MAP_HUGETLB. + +Syscalls that operate on memory backed by hugetlb pages only have their lengths +aligned to the native page size of the processor; they will normally fail with +errno set to EINVAL or exclude hugetlb pages that extend beyond the length if +not hugepage aligned. For example, munmap(2) will fail if memory is backed by +a hugetlb page and the length is smaller than the hugepage size. + + +Examples +======== + +.. _map_hugetlb: + +``map_hugetlb`` + see tools/testing/selftests/vm/map_hugetlb.c + +``hugepage-shm`` + see tools/testing/selftests/vm/hugepage-shm.c + +``hugepage-mmap`` + see tools/testing/selftests/vm/hugepage-mmap.c + +The `libhugetlbfs`_ library provides a wide range of userspace tools +to help with huge page usability, environment setup, and control. + +.. _libhugetlbfs: https://github.com/libhugetlbfs/libhugetlbfs diff --git a/Documentation/admin-guide/mm/idle_page_tracking.rst b/Documentation/admin-guide/mm/idle_page_tracking.rst new file mode 100644 index 000000000000..6f7b7ca1add3 --- /dev/null +++ b/Documentation/admin-guide/mm/idle_page_tracking.rst @@ -0,0 +1,116 @@ +.. _idle_page_tracking: + +================== +Idle Page Tracking +================== + +Motivation +========== + +The idle page tracking feature allows to track which memory pages are being +accessed by a workload and which are idle. This information can be useful for +estimating the workload's working set size, which, in turn, can be taken into +account when configuring the workload parameters, setting memory cgroup limits, +or deciding where to place the workload within a compute cluster. + +It is enabled by CONFIG_IDLE_PAGE_TRACKING=y. + +.. _user_api: + +User API +======== + +The idle page tracking API is located at ``/sys/kernel/mm/page_idle``. +Currently, it consists of the only read-write file, +``/sys/kernel/mm/page_idle/bitmap``. + +The file implements a bitmap where each bit corresponds to a memory page. The +bitmap is represented by an array of 8-byte integers, and the page at PFN #i is +mapped to bit #i%64 of array element #i/64, byte order is native. When a bit is +set, the corresponding page is idle. + +A page is considered idle if it has not been accessed since it was marked idle +(for more details on what "accessed" actually means see the :ref:`Implementation +Details ` section). +To mark a page idle one has to set the bit corresponding to +the page by writing to the file. A value written to the file is OR-ed with the +current bitmap value. + +Only accesses to user memory pages are tracked. These are pages mapped to a +process address space, page cache and buffer pages, swap cache pages. For other +page types (e.g. SLAB pages) an attempt to mark a page idle is silently ignored, +and hence such pages are never reported idle. + +For huge pages the idle flag is set only on the head page, so one has to read +``/proc/kpageflags`` in order to correctly count idle huge pages. + +Reading from or writing to ``/sys/kernel/mm/page_idle/bitmap`` will return +-EINVAL if you are not starting the read/write on an 8-byte boundary, or +if the size of the read/write is not a multiple of 8 bytes. Writing to +this file beyond max PFN will return -ENXIO. + +That said, in order to estimate the amount of pages that are not used by a +workload one should: + + 1. Mark all the workload's pages as idle by setting corresponding bits in + ``/sys/kernel/mm/page_idle/bitmap``. The pages can be found by reading + ``/proc/pid/pagemap`` if the workload is represented by a process, or by + filtering out alien pages using ``/proc/kpagecgroup`` in case the workload + is placed in a memory cgroup. + + 2. Wait until the workload accesses its working set. + + 3. Read ``/sys/kernel/mm/page_idle/bitmap`` and count the number of bits set. + If one wants to ignore certain types of pages, e.g. mlocked pages since they + are not reclaimable, he or she can filter them out using + ``/proc/kpageflags``. + +See :ref:`Documentation/admin-guide/mm/pagemap.rst ` for more +information about ``/proc/pid/pagemap``, ``/proc/kpageflags``, and +``/proc/kpagecgroup``. + +.. _impl_details: + +Implementation Details +====================== + +The kernel internally keeps track of accesses to user memory pages in order to +reclaim unreferenced pages first on memory shortage conditions. A page is +considered referenced if it has been recently accessed via a process address +space, in which case one or more PTEs it is mapped to will have the Accessed bit +set, or marked accessed explicitly by the kernel (see mark_page_accessed()). The +latter happens when: + + - a userspace process reads or writes a page using a system call (e.g. read(2) + or write(2)) + + - a page that is used for storing filesystem buffers is read or written, + because a process needs filesystem metadata stored in it (e.g. lists a + directory tree) + + - a page is accessed by a device driver using get_user_pages() + +When a dirty page is written to swap or disk as a result of memory reclaim or +exceeding the dirty memory limit, it is not marked referenced. + +The idle memory tracking feature adds a new page flag, the Idle flag. This flag +is set manually, by writing to ``/sys/kernel/mm/page_idle/bitmap`` (see the +:ref:`User API ` +section), and cleared automatically whenever a page is referenced as defined +above. + +When a page is marked idle, the Accessed bit must be cleared in all PTEs it is +mapped to, otherwise we will not be able to detect accesses to the page coming +from a process address space. To avoid interference with the reclaimer, which, +as noted above, uses the Accessed bit to promote actively referenced pages, one +more page flag is introduced, the Young flag. When the PTE Accessed bit is +cleared as a result of setting or updating a page's Idle flag, the Young flag +is set on the page. The reclaimer treats the Young flag as an extra PTE +Accessed bit and therefore will consider such a page as referenced. + +Since the idle memory tracking feature is based on the memory reclaimer logic, +it only works with pages that are on an LRU list, other pages are silently +ignored. That means it will ignore a user memory page if it is isolated, but +since there are usually not many of them, it should not affect the overall +result noticeably. In order not to stall scanning of the idle page bitmap, +locked pages may be skipped too. diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst new file mode 100644 index 000000000000..ceead68c2df7 --- /dev/null +++ b/Documentation/admin-guide/mm/index.rst @@ -0,0 +1,36 @@ +================= +Memory Management +================= + +Linux memory management subsystem is responsible, as the name implies, +for managing the memory in the system. This includes implemnetation of +virtual memory and demand paging, memory allocation both for kernel +internal structures and user space programms, mapping of files into +processes address space and many other cool things. + +Linux memory management is a complex system with many configurable +settings. Most of these settings are available via ``/proc`` +filesystem and can be quired and adjusted using ``sysctl``. These APIs +are described in Documentation/sysctl/vm.txt and in `man 5 proc`_. + +.. _man 5 proc: http://man7.org/linux/man-pages/man5/proc.5.html + +Linux memory management has its own jargon and if you are not yet +familiar with it, consider reading +:ref:`Documentation/admin-guide/mm/concepts.rst `. + +Here we document in detail how to interact with various mechanisms in +the Linux memory management. + +.. toctree:: + :maxdepth: 1 + + concepts + hugetlbpage + idle_page_tracking + ksm + numa_memory_policy + pagemap + soft-dirty + transhuge + userfaultfd diff --git a/Documentation/admin-guide/mm/ksm.rst b/Documentation/admin-guide/mm/ksm.rst new file mode 100644 index 000000000000..9303786632d1 --- /dev/null +++ b/Documentation/admin-guide/mm/ksm.rst @@ -0,0 +1,189 @@ +.. _admin_guide_ksm: + +======================= +Kernel Samepage Merging +======================= + +Overview +======== + +KSM is a memory-saving de-duplication feature, enabled by CONFIG_KSM=y, +added to the Linux kernel in 2.6.32. See ``mm/ksm.c`` for its implementation, +and http://lwn.net/Articles/306704/ and http://lwn.net/Articles/330589/ + +KSM was originally developed for use with KVM (where it was known as +Kernel Shared Memory), to fit more virtual machines into physical memory, +by sharing the data common between them. But it can be useful to any +application which generates many instances of the same data. + +The KSM daemon ksmd periodically scans those areas of user memory +which have been registered with it, looking for pages of identical +content which can be replaced by a single write-protected page (which +is automatically copied if a process later wants to update its +content). The amount of pages that KSM daemon scans in a single pass +and the time between the passes are configured using :ref:`sysfs +intraface ` + +KSM only merges anonymous (private) pages, never pagecache (file) pages. +KSM's merged pages were originally locked into kernel memory, but can now +be swapped out just like other user pages (but sharing is broken when they +are swapped back in: ksmd must rediscover their identity and merge again). + +Controlling KSM with madvise +============================ + +KSM only operates on those areas of address space which an application +has advised to be likely candidates for merging, by using the madvise(2) +system call:: + + int madvise(addr, length, MADV_MERGEABLE) + +The app may call + +:: + + int madvise(addr, length, MADV_UNMERGEABLE) + +to cancel that advice and restore unshared pages: whereupon KSM +unmerges whatever it merged in that range. Note: this unmerging call +may suddenly require more memory than is available - possibly failing +with EAGAIN, but more probably arousing the Out-Of-Memory killer. + +If KSM is not configured into the running kernel, madvise MADV_MERGEABLE +and MADV_UNMERGEABLE simply fail with EINVAL. If the running kernel was +built with CONFIG_KSM=y, those calls will normally succeed: even if the +the KSM daemon is not currently running, MADV_MERGEABLE still registers +the range for whenever the KSM daemon is started; even if the range +cannot contain any pages which KSM could actually merge; even if +MADV_UNMERGEABLE is applied to a range which was never MADV_MERGEABLE. + +If a region of memory must be split into at least one new MADV_MERGEABLE +or MADV_UNMERGEABLE region, the madvise may return ENOMEM if the process +will exceed ``vm.max_map_count`` (see Documentation/sysctl/vm.txt). + +Like other madvise calls, they are intended for use on mapped areas of +the user address space: they will report ENOMEM if the specified range +includes unmapped gaps (though working on the intervening mapped areas), +and might fail with EAGAIN if not enough memory for internal structures. + +Applications should be considerate in their use of MADV_MERGEABLE, +restricting its use to areas likely to benefit. KSM's scans may use a lot +of processing power: some installations will disable KSM for that reason. + +.. _ksm_sysfs: + +KSM daemon sysfs interface +========================== + +The KSM daemon is controlled by sysfs files in ``/sys/kernel/mm/ksm/``, +readable by all but writable only by root: + +pages_to_scan + how many pages to scan before ksmd goes to sleep + e.g. ``echo 100 > /sys/kernel/mm/ksm/pages_to_scan``. + + Default: 100 (chosen for demonstration purposes) + +sleep_millisecs + how many milliseconds ksmd should sleep before next scan + e.g. ``echo 20 > /sys/kernel/mm/ksm/sleep_millisecs`` + + Default: 20 (chosen for demonstration purposes) + +merge_across_nodes + specifies if pages from different NUMA nodes can be merged. + When set to 0, ksm merges only pages which physically reside + in the memory area of same NUMA node. That brings lower + latency to access of shared pages. Systems with more nodes, at + significant NUMA distances, are likely to benefit from the + lower latency of setting 0. Smaller systems, which need to + minimize memory usage, are likely to benefit from the greater + sharing of setting 1 (default). You may wish to compare how + your system performs under each setting, before deciding on + which to use. ``merge_across_nodes`` setting can be changed only + when there are no ksm shared pages in the system: set run 2 to + unmerge pages first, then to 1 after changing + ``merge_across_nodes``, to remerge according to the new setting. + + Default: 1 (merging across nodes as in earlier releases) + +run + * set to 0 to stop ksmd from running but keep merged pages, + * set to 1 to run ksmd e.g. ``echo 1 > /sys/kernel/mm/ksm/run``, + * set to 2 to stop ksmd and unmerge all pages currently merged, but + leave mergeable areas registered for next run. + + Default: 0 (must be changed to 1 to activate KSM, except if + CONFIG_SYSFS is disabled) + +use_zero_pages + specifies whether empty pages (i.e. allocated pages that only + contain zeroes) should be treated specially. When set to 1, + empty pages are merged with the kernel zero page(s) instead of + with each other as it would happen normally. This can improve + the performance on architectures with coloured zero pages, + depending on the workload. Care should be taken when enabling + this setting, as it can potentially degrade the performance of + KSM for some workloads, for example if the checksums of pages + candidate for merging match the checksum of an empty + page. This setting can be changed at any time, it is only + effective for pages merged after the change. + + Default: 0 (normal KSM behaviour as in earlier releases) + +max_page_sharing + Maximum sharing allowed for each KSM page. This enforces a + deduplication limit to avoid high latency for virtual memory + operations that involve traversal of the virtual mappings that + share the KSM page. The minimum value is 2 as a newly created + KSM page will have at least two sharers. The higher this value + the faster KSM will merge the memory and the higher the + deduplication factor will be, but the slower the worst case + virtual mappings traversal could be for any given KSM + page. Slowing down this traversal means there will be higher + latency for certain virtual memory operations happening during + swapping, compaction, NUMA balancing and page migration, in + turn decreasing responsiveness for the caller of those virtual + memory operations. The scheduler latency of other tasks not + involved with the VM operations doing the virtual mappings + traversal is not affected by this parameter as these + traversals are always schedule friendly themselves. + +stable_node_chains_prune_millisecs + specifies how frequently KSM checks the metadata of the pages + that hit the deduplication limit for stale information. + Smaller milllisecs values will free up the KSM metadata with + lower latency, but they will make ksmd use more CPU during the + scan. It's a noop if not a single KSM page hit the + ``max_page_sharing`` yet. + +The effectiveness of KSM and MADV_MERGEABLE is shown in ``/sys/kernel/mm/ksm/``: + +pages_shared + how many shared pages are being used +pages_sharing + how many more sites are sharing them i.e. how much saved +pages_unshared + how many pages unique but repeatedly checked for merging +pages_volatile + how many pages changing too fast to be placed in a tree +full_scans + how many times all mergeable areas have been scanned +stable_node_chains + the number of KSM pages that hit the ``max_page_sharing`` limit +stable_node_dups + number of duplicated KSM pages + +A high ratio of ``pages_sharing`` to ``pages_shared`` indicates good +sharing, but a high ratio of ``pages_unshared`` to ``pages_sharing`` +indicates wasted effort. ``pages_volatile`` embraces several +different kinds of activity, but a high proportion there would also +indicate poor use of madvise MADV_MERGEABLE. + +The maximum possible ``pages_sharing/pages_shared`` ratio is limited by the +``max_page_sharing`` tunable. To increase the ratio ``max_page_sharing`` must +be increased accordingly. + +-- +Izik Eidus, +Hugh Dickins, 17 Nov 2009 diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst new file mode 100644 index 000000000000..d78c5b315f72 --- /dev/null +++ b/Documentation/admin-guide/mm/numa_memory_policy.rst @@ -0,0 +1,495 @@ +.. _numa_memory_policy: + +================== +NUMA Memory Policy +================== + +What is NUMA Memory Policy? +============================ + +In the Linux kernel, "memory policy" determines from which node the kernel will +allocate memory in a NUMA system or in an emulated NUMA system. Linux has +supported platforms with Non-Uniform Memory Access architectures since 2.4.?. +The current memory policy support was added to Linux 2.6 around May 2004. This +document attempts to describe the concepts and APIs of the 2.6 memory policy +support. + +Memory policies should not be confused with cpusets +(``Documentation/cgroup-v1/cpusets.txt``) +which is an administrative mechanism for restricting the nodes from which +memory may be allocated by a set of processes. Memory policies are a +programming interface that a NUMA-aware application can take advantage of. When +both cpusets and policies are applied to a task, the restrictions of the cpuset +takes priority. See :ref:`Memory Policies and cpusets ` +below for more details. + +Memory Policy Concepts +====================== + +Scope of Memory Policies +------------------------ + +The Linux kernel supports _scopes_ of memory policy, described here from +most general to most specific: + +System Default Policy + this policy is "hard coded" into the kernel. It is the policy + that governs all page allocations that aren't controlled by + one of the more specific policy scopes discussed below. When + the system is "up and running", the system default policy will + use "local allocation" described below. However, during boot + up, the system default policy will be set to interleave + allocations across all nodes with "sufficient" memory, so as + not to overload the initial boot node with boot-time + allocations. + +Task/Process Policy + this is an optional, per-task policy. When defined for a + specific task, this policy controls all page allocations made + by or on behalf of the task that aren't controlled by a more + specific scope. If a task does not define a task policy, then + all page allocations that would have been controlled by the + task policy "fall back" to the System Default Policy. + + The task policy applies to the entire address space of a task. Thus, + it is inheritable, and indeed is inherited, across both fork() + [clone() w/o the CLONE_VM flag] and exec*(). This allows a parent task + to establish the task policy for a child task exec()'d from an + executable image that has no awareness of memory policy. See the + :ref:`Memory Policy APIs ` section, + below, for an overview of the system call + that a task may use to set/change its task/process policy. + + In a multi-threaded task, task policies apply only to the thread + [Linux kernel task] that installs the policy and any threads + subsequently created by that thread. Any sibling threads existing + at the time a new task policy is installed retain their current + policy. + + A task policy applies only to pages allocated after the policy is + installed. Any pages already faulted in by the task when the task + changes its task policy remain where they were allocated based on + the policy at the time they were allocated. + +.. _vma_policy: + +VMA Policy + A "VMA" or "Virtual Memory Area" refers to a range of a task's + virtual address space. A task may define a specific policy for a range + of its virtual address space. See the + :ref:`Memory Policy APIs ` section, + below, for an overview of the mbind() system call used to set a VMA + policy. + + A VMA policy will govern the allocation of pages that back + this region of the address space. Any regions of the task's + address space that don't have an explicit VMA policy will fall + back to the task policy, which may itself fall back to the + System Default Policy. + + VMA policies have a few complicating details: + + * VMA policy applies ONLY to anonymous pages. These include + pages allocated for anonymous segments, such as the task + stack and heap, and any regions of the address space + mmap()ed with the MAP_ANONYMOUS flag. If a VMA policy is + applied to a file mapping, it will be ignored if the mapping + used the MAP_SHARED flag. If the file mapping used the + MAP_PRIVATE flag, the VMA policy will only be applied when + an anonymous page is allocated on an attempt to write to the + mapping-- i.e., at Copy-On-Write. + + * VMA policies are shared between all tasks that share a + virtual address space--a.k.a. threads--independent of when + the policy is installed; and they are inherited across + fork(). However, because VMA policies refer to a specific + region of a task's address space, and because the address + space is discarded and recreated on exec*(), VMA policies + are NOT inheritable across exec(). Thus, only NUMA-aware + applications may use VMA policies. + + * A task may install a new VMA policy on a sub-range of a + previously mmap()ed region. When this happens, Linux splits + the existing virtual memory area into 2 or 3 VMAs, each with + it's own policy. + + * By default, VMA policy applies only to pages allocated after + the policy is installed. Any pages already faulted into the + VMA range remain where they were allocated based on the + policy at the time they were allocated. However, since + 2.6.16, Linux supports page migration via the mbind() system + call, so that page contents can be moved to match a newly + installed policy. + +Shared Policy + Conceptually, shared policies apply to "memory objects" mapped + shared into one or more tasks' distinct address spaces. An + application installs shared policies the same way as VMA + policies--using the mbind() system call specifying a range of + virtual addresses that map the shared object. However, unlike + VMA policies, which can be considered to be an attribute of a + range of a task's address space, shared policies apply + directly to the shared object. Thus, all tasks that attach to + the object share the policy, and all pages allocated for the + shared object, by any task, will obey the shared policy. + + As of 2.6.22, only shared memory segments, created by shmget() or + mmap(MAP_ANONYMOUS|MAP_SHARED), support shared policy. When shared + policy support was added to Linux, the associated data structures were + added to hugetlbfs shmem segments. At the time, hugetlbfs did not + support allocation at fault time--a.k.a lazy allocation--so hugetlbfs + shmem segments were never "hooked up" to the shared policy support. + Although hugetlbfs segments now support lazy allocation, their support + for shared policy has not been completed. + + As mentioned above in :ref:`VMA policies ` section, + allocations of page cache pages for regular files mmap()ed + with MAP_SHARED ignore any VMA policy installed on the virtual + address range backed by the shared file mapping. Rather, + shared page cache pages, including pages backing private + mappings that have not yet been written by the task, follow + task policy, if any, else System Default Policy. + + The shared policy infrastructure supports different policies on subset + ranges of the shared object. However, Linux still splits the VMA of + the task that installs the policy for each range of distinct policy. + Thus, different tasks that attach to a shared memory segment can have + different VMA configurations mapping that one shared object. This + can be seen by examining the /proc//numa_maps of tasks sharing + a shared memory region, when one task has installed shared policy on + one or more ranges of the region. + +Components of Memory Policies +----------------------------- + +A NUMA memory policy consists of a "mode", optional mode flags, and +an optional set of nodes. The mode determines the behavior of the +policy, the optional mode flags determine the behavior of the mode, +and the optional set of nodes can be viewed as the arguments to the +policy behavior. + +Internally, memory policies are implemented by a reference counted +structure, struct mempolicy. Details of this structure will be +discussed in context, below, as required to explain the behavior. + +NUMA memory policy supports the following 4 behavioral modes: + +Default Mode--MPOL_DEFAULT + This mode is only used in the memory policy APIs. Internally, + MPOL_DEFAULT is converted to the NULL memory policy in all + policy scopes. Any existing non-default policy will simply be + removed when MPOL_DEFAULT is specified. As a result, + MPOL_DEFAULT means "fall back to the next most specific policy + scope." + + For example, a NULL or default task policy will fall back to the + system default policy. A NULL or default vma policy will fall + back to the task policy. + + When specified in one of the memory policy APIs, the Default mode + does not use the optional set of nodes. + + It is an error for the set of nodes specified for this policy to + be non-empty. + +MPOL_BIND + This mode specifies that memory must come from the set of + nodes specified by the policy. Memory will be allocated from + the node in the set with sufficient free memory that is + closest to the node where the allocation takes place. + +MPOL_PREFERRED + This mode specifies that the allocation should be attempted + from the single node specified in the policy. If that + allocation fails, the kernel will search other nodes, in order + of increasing distance from the preferred node based on + information provided by the platform firmware. + + Internally, the Preferred policy uses a single node--the + preferred_node member of struct mempolicy. When the internal + mode flag MPOL_F_LOCAL is set, the preferred_node is ignored + and the policy is interpreted as local allocation. "Local" + allocation policy can be viewed as a Preferred policy that + starts at the node containing the cpu where the allocation + takes place. + + It is possible for the user to specify that local allocation + is always preferred by passing an empty nodemask with this + mode. If an empty nodemask is passed, the policy cannot use + the MPOL_F_STATIC_NODES or MPOL_F_RELATIVE_NODES flags + described below. + +MPOL_INTERLEAVED + This mode specifies that page allocations be interleaved, on a + page granularity, across the nodes specified in the policy. + This mode also behaves slightly differently, based on the + context where it is used: + + For allocation of anonymous pages and shared memory pages, + Interleave mode indexes the set of nodes specified by the + policy using the page offset of the faulting address into the + segment [VMA] containing the address modulo the number of + nodes specified by the policy. It then attempts to allocate a + page, starting at the selected node, as if the node had been + specified by a Preferred policy or had been selected by a + local allocation. That is, allocation will follow the per + node zonelist. + + For allocation of page cache pages, Interleave mode indexes + the set of nodes specified by the policy using a node counter + maintained per task. This counter wraps around to the lowest + specified node after it reaches the highest specified node. + This will tend to spread the pages out over the nodes + specified by the policy based on the order in which they are + allocated, rather than based on any page offset into an + address range or file. During system boot up, the temporary + interleaved system default policy works in this mode. + +NUMA memory policy supports the following optional mode flags: + +MPOL_F_STATIC_NODES + This flag specifies that the nodemask passed by + the user should not be remapped if the task or VMA's set of allowed + nodes changes after the memory policy has been defined. + + Without this flag, any time a mempolicy is rebound because of a + change in the set of allowed nodes, the node (Preferred) or + nodemask (Bind, Interleave) is remapped to the new set of + allowed nodes. This may result in nodes being used that were + previously undesired. + + With this flag, if the user-specified nodes overlap with the + nodes allowed by the task's cpuset, then the memory policy is + applied to their intersection. If the two sets of nodes do not + overlap, the Default policy is used. + + For example, consider a task that is attached to a cpuset with + mems 1-3 that sets an Interleave policy over the same set. If + the cpuset's mems change to 3-5, the Interleave will now occur + over nodes 3, 4, and 5. With this flag, however, since only node + 3 is allowed from the user's nodemask, the "interleave" only + occurs over that node. If no nodes from the user's nodemask are + now allowed, the Default behavior is used. + + MPOL_F_STATIC_NODES cannot be combined with the + MPOL_F_RELATIVE_NODES flag. It also cannot be used for + MPOL_PREFERRED policies that were created with an empty nodemask + (local allocation). + +MPOL_F_RELATIVE_NODES + This flag specifies that the nodemask passed + by the user will be mapped relative to the set of the task or VMA's + set of allowed nodes. The kernel stores the user-passed nodemask, + and if the allowed nodes changes, then that original nodemask will + be remapped relative to the new set of allowed nodes. + + Without this flag (and without MPOL_F_STATIC_NODES), anytime a + mempolicy is rebound because of a change in the set of allowed + nodes, the node (Preferred) or nodemask (Bind, Interleave) is + remapped to the new set of allowed nodes. That remap may not + preserve the relative nature of the user's passed nodemask to its + set of allowed nodes upon successive rebinds: a nodemask of + 1,3,5 may be remapped to 7-9 and then to 1-3 if the set of + allowed nodes is restored to its original state. + + With this flag, the remap is done so that the node numbers from + the user's passed nodemask are relative to the set of allowed + nodes. In other words, if nodes 0, 2, and 4 are set in the user's + nodemask, the policy will be effected over the first (and in the + Bind or Interleave case, the third and fifth) nodes in the set of + allowed nodes. The nodemask passed by the user represents nodes + relative to task or VMA's set of allowed nodes. + + If the user's nodemask includes nodes that are outside the range + of the new set of allowed nodes (for example, node 5 is set in + the user's nodemask when the set of allowed nodes is only 0-3), + then the remap wraps around to the beginning of the nodemask and, + if not already set, sets the node in the mempolicy nodemask. + + For example, consider a task that is attached to a cpuset with + mems 2-5 that sets an Interleave policy over the same set with + MPOL_F_RELATIVE_NODES. If the cpuset's mems change to 3-7, the + interleave now occurs over nodes 3,5-7. If the cpuset's mems + then change to 0,2-3,5, then the interleave occurs over nodes + 0,2-3,5. + + Thanks to the consistent remapping, applications preparing + nodemasks to specify memory policies using this flag should + disregard their current, actual cpuset imposed memory placement + and prepare the nodemask as if they were always located on + memory nodes 0 to N-1, where N is the number of memory nodes the + policy is intended to manage. Let the kernel then remap to the + set of memory nodes allowed by the task's cpuset, as that may + change over time. + + MPOL_F_RELATIVE_NODES cannot be combined with the + MPOL_F_STATIC_NODES flag. It also cannot be used for + MPOL_PREFERRED policies that were created with an empty nodemask + (local allocation). + +Memory Policy Reference Counting +================================ + +To resolve use/free races, struct mempolicy contains an atomic reference +count field. Internal interfaces, mpol_get()/mpol_put() increment and +decrement this reference count, respectively. mpol_put() will only free +the structure back to the mempolicy kmem cache when the reference count +goes to zero. + +When a new memory policy is allocated, its reference count is initialized +to '1', representing the reference held by the task that is installing the +new policy. When a pointer to a memory policy structure is stored in another +structure, another reference is added, as the task's reference will be dropped +on completion of the policy installation. + +During run-time "usage" of the policy, we attempt to minimize atomic operations +on the reference count, as this can lead to cache lines bouncing between cpus +and NUMA nodes. "Usage" here means one of the following: + +1) querying of the policy, either by the task itself [using the get_mempolicy() + API discussed below] or by another task using the /proc//numa_maps + interface. + +2) examination of the policy to determine the policy mode and associated node + or node lists, if any, for page allocation. This is considered a "hot + path". Note that for MPOL_BIND, the "usage" extends across the entire + allocation process, which may sleep during page reclaimation, because the + BIND policy nodemask is used, by reference, to filter ineligible nodes. + +We can avoid taking an extra reference during the usages listed above as +follows: + +1) we never need to get/free the system default policy as this is never + changed nor freed, once the system is up and running. + +2) for querying the policy, we do not need to take an extra reference on the + target task's task policy nor vma policies because we always acquire the + task's mm's mmap_sem for read during the query. The set_mempolicy() and + mbind() APIs [see below] always acquire the mmap_sem for write when + installing or replacing task or vma policies. Thus, there is no possibility + of a task or thread freeing a policy while another task or thread is + querying it. + +3) Page allocation usage of task or vma policy occurs in the fault path where + we hold them mmap_sem for read. Again, because replacing the task or vma + policy requires that the mmap_sem be held for write, the policy can't be + freed out from under us while we're using it for page allocation. + +4) Shared policies require special consideration. One task can replace a + shared memory policy while another task, with a distinct mmap_sem, is + querying or allocating a page based on the policy. To resolve this + potential race, the shared policy infrastructure adds an extra reference + to the shared policy during lookup while holding a spin lock on the shared + policy management structure. This requires that we drop this extra + reference when we're finished "using" the policy. We must drop the + extra reference on shared policies in the same query/allocation paths + used for non-shared policies. For this reason, shared policies are marked + as such, and the extra reference is dropped "conditionally"--i.e., only + for shared policies. + + Because of this extra reference counting, and because we must lookup + shared policies in a tree structure under spinlock, shared policies are + more expensive to use in the page allocation path. This is especially + true for shared policies on shared memory regions shared by tasks running + on different NUMA nodes. This extra overhead can be avoided by always + falling back to task or system default policy for shared memory regions, + or by prefaulting the entire shared memory region into memory and locking + it down. However, this might not be appropriate for all applications. + +.. _memory_policy_apis: + +Memory Policy APIs +================== + +Linux supports 3 system calls for controlling memory policy. These APIS +always affect only the calling task, the calling task's address space, or +some shared object mapped into the calling task's address space. + +.. note:: + the headers that define these APIs and the parameter data types for + user space applications reside in a package that is not part of the + Linux kernel. The kernel system call interfaces, with the 'sys\_' + prefix, are defined in ; the mode and flag + definitions are defined in . + +Set [Task] Memory Policy:: + + long set_mempolicy(int mode, const unsigned long *nmask, + unsigned long maxnode); + +Set's the calling task's "task/process memory policy" to mode +specified by the 'mode' argument and the set of nodes defined by +'nmask'. 'nmask' points to a bit mask of node ids containing at least +'maxnode' ids. Optional mode flags may be passed by combining the +'mode' argument with the flag (for example: MPOL_INTERLEAVE | +MPOL_F_STATIC_NODES). + +See the set_mempolicy(2) man page for more details + + +Get [Task] Memory Policy or Related Information:: + + long get_mempolicy(int *mode, + const unsigned long *nmask, unsigned long maxnode, + void *addr, int flags); + +Queries the "task/process memory policy" of the calling task, or the +policy or location of a specified virtual address, depending on the +'flags' argument. + +See the get_mempolicy(2) man page for more details + + +Install VMA/Shared Policy for a Range of Task's Address Space:: + + long mbind(void *start, unsigned long len, int mode, + const unsigned long *nmask, unsigned long maxnode, + unsigned flags); + +mbind() installs the policy specified by (mode, nmask, maxnodes) as a +VMA policy for the range of the calling task's address space specified +by the 'start' and 'len' arguments. Additional actions may be +requested via the 'flags' argument. + +See the mbind(2) man page for more details. + +Memory Policy Command Line Interface +==================================== + +Although not strictly part of the Linux implementation of memory policy, +a command line tool, numactl(8), exists that allows one to: + ++ set the task policy for a specified program via set_mempolicy(2), fork(2) and + exec(2) + ++ set the shared policy for a shared memory segment via mbind(2) + +The numactl(8) tool is packaged with the run-time version of the library +containing the memory policy system call wrappers. Some distributions +package the headers and compile-time libraries in a separate development +package. + +.. _mem_pol_and_cpusets: + +Memory Policies and cpusets +=========================== + +Memory policies work within cpusets as described above. For memory policies +that require a node or set of nodes, the nodes are restricted to the set of +nodes whose memories are allowed by the cpuset constraints. If the nodemask +specified for the policy contains nodes that are not allowed by the cpuset and +MPOL_F_RELATIVE_NODES is not used, the intersection of the set of nodes +specified for the policy and the set of nodes with memory is used. If the +result is the empty set, the policy is considered invalid and cannot be +installed. If MPOL_F_RELATIVE_NODES is used, the policy's nodes are mapped +onto and folded into the task's set of allowed nodes as previously described. + +The interaction of memory policies and cpusets can be problematic when tasks +in two cpusets share access to a memory region, such as shared memory segments +created by shmget() of mmap() with the MAP_ANONYMOUS and MAP_SHARED flags, and +any of the tasks install shared policy on the region, only nodes whose +memories are allowed in both cpusets may be used in the policies. Obtaining +this information requires "stepping outside" the memory policy APIs to use the +cpuset information and requires that one know in what cpusets other task might +be attaching to the shared region. Furthermore, if the cpusets' allowed +memory sets are disjoint, "local" allocation is the only valid policy. diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst new file mode 100644 index 000000000000..577af85beb41 --- /dev/null +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -0,0 +1,201 @@ +.. _pagemap: + +============================= +Examining Process Page Tables +============================= + +pagemap is a new (as of 2.6.25) set of interfaces in the kernel that allow +userspace programs to examine the page tables and related information by +reading files in ``/proc``. + +There are four components to pagemap: + + * ``/proc/pid/pagemap``. This file lets a userspace process find out which + physical frame each virtual page is mapped to. It contains one 64-bit + value for each virtual page, containing the following data (from + ``fs/proc/task_mmu.c``, above pagemap_read): + + * Bits 0-54 page frame number (PFN) if present + * Bits 0-4 swap type if swapped + * Bits 5-54 swap offset if swapped + * Bit 55 pte is soft-dirty (see + :ref:`Documentation/admin-guide/mm/soft-dirty.rst `) + * Bit 56 page exclusively mapped (since 4.2) + * Bits 57-60 zero + * Bit 61 page is file-page or shared-anon (since 3.5) + * Bit 62 page swapped + * Bit 63 page present + + Since Linux 4.0 only users with the CAP_SYS_ADMIN capability can get PFNs. + In 4.0 and 4.1 opens by unprivileged fail with -EPERM. Starting from + 4.2 the PFN field is zeroed if the user does not have CAP_SYS_ADMIN. + Reason: information about PFNs helps in exploiting Rowhammer vulnerability. + + If the page is not present but in swap, then the PFN contains an + encoding of the swap file number and the page's offset into the + swap. Unmapped pages return a null PFN. This allows determining + precisely which pages are mapped (or in swap) and comparing mapped + pages between processes. + + Efficient users of this interface will use ``/proc/pid/maps`` to + determine which areas of memory are actually mapped and llseek to + skip over unmapped regions. + + * ``/proc/kpagecount``. This file contains a 64-bit count of the number of + times each page is mapped, indexed by PFN. + + * ``/proc/kpageflags``. This file contains a 64-bit set of flags for each + page, indexed by PFN. + + The flags are (from ``fs/proc/page.c``, above kpageflags_read): + + 0. LOCKED + 1. ERROR + 2. REFERENCED + 3. UPTODATE + 4. DIRTY + 5. LRU + 6. ACTIVE + 7. SLAB + 8. WRITEBACK + 9. RECLAIM + 10. BUDDY + 11. MMAP + 12. ANON + 13. SWAPCACHE + 14. SWAPBACKED + 15. COMPOUND_HEAD + 16. COMPOUND_TAIL + 17. HUGE + 18. UNEVICTABLE + 19. HWPOISON + 20. NOPAGE + 21. KSM + 22. THP + 23. BALLOON + 24. ZERO_PAGE + 25. IDLE + + * ``/proc/kpagecgroup``. This file contains a 64-bit inode number of the + memory cgroup each page is charged to, indexed by PFN. Only available when + CONFIG_MEMCG is set. + +Short descriptions to the page flags +==================================== + +0 - LOCKED + page is being locked for exclusive access, e.g. by undergoing read/write IO +7 - SLAB + page is managed by the SLAB/SLOB/SLUB/SLQB kernel memory allocator + When compound page is used, SLUB/SLQB will only set this flag on the head + page; SLOB will not flag it at all. +10 - BUDDY + a free memory block managed by the buddy system allocator + The buddy system organizes free memory in blocks of various orders. + An order N block has 2^N physically contiguous pages, with the BUDDY flag + set for and _only_ for the first page. +15 - COMPOUND_HEAD + A compound page with order N consists of 2^N physically contiguous pages. + A compound page with order 2 takes the form of "HTTT", where H donates its + head page and T donates its tail page(s). The major consumers of compound + pages are hugeTLB pages + (:ref:`Documentation/admin-guide/mm/hugetlbpage.rst `), + the SLUB etc. memory allocators and various device drivers. + However in this interface, only huge/giga pages are made visible + to end users. +16 - COMPOUND_TAIL + A compound page tail (see description above). +17 - HUGE + this is an integral part of a HugeTLB page +19 - HWPOISON + hardware detected memory corruption on this page: don't touch the data! +20 - NOPAGE + no page frame exists at the requested address +21 - KSM + identical memory pages dynamically shared between one or more processes +22 - THP + contiguous pages which construct transparent hugepages +23 - BALLOON + balloon compaction page +24 - ZERO_PAGE + zero page for pfn_zero or huge_zero page +25 - IDLE + page has not been accessed since it was marked idle (see + :ref:`Documentation/admin-guide/mm/idle_page_tracking.rst `). + Note that this flag may be stale in case the page was accessed via + a PTE. To make sure the flag is up-to-date one has to read + ``/sys/kernel/mm/page_idle/bitmap`` first. + +IO related page flags +--------------------- + +1 - ERROR + IO error occurred +3 - UPTODATE + page has up-to-date data + ie. for file backed page: (in-memory data revision >= on-disk one) +4 - DIRTY + page has been written to, hence contains new data + i.e. for file backed page: (in-memory data revision > on-disk one) +8 - WRITEBACK + page is being synced to disk + +LRU related page flags +---------------------- + +5 - LRU + page is in one of the LRU lists +6 - ACTIVE + page is in the active LRU list +18 - UNEVICTABLE + page is in the unevictable (non-)LRU list It is somehow pinned and + not a candidate for LRU page reclaims, e.g. ramfs pages, + shmctl(SHM_LOCK) and mlock() memory segments +2 - REFERENCED + page has been referenced since last LRU list enqueue/requeue +9 - RECLAIM + page will be reclaimed soon after its pageout IO completed +11 - MMAP + a memory mapped page +12 - ANON + a memory mapped page that is not part of a file +13 - SWAPCACHE + page is mapped to swap space, i.e. has an associated swap entry +14 - SWAPBACKED + page is backed by swap/RAM + +The page-types tool in the tools/vm directory can be used to query the +above flags. + +Using pagemap to do something useful +==================================== + +The general procedure for using pagemap to find out about a process' memory +usage goes like this: + + 1. Read ``/proc/pid/maps`` to determine which parts of the memory space are + mapped to what. + 2. Select the maps you are interested in -- all of them, or a particular + library, or the stack or the heap, etc. + 3. Open ``/proc/pid/pagemap`` and seek to the pages you would like to examine. + 4. Read a u64 for each page from pagemap. + 5. Open ``/proc/kpagecount`` and/or ``/proc/kpageflags``. For each PFN you + just read, seek to that entry in the file, and read the data you want. + +For example, to find the "unique set size" (USS), which is the amount of +memory that a process is using that is not shared with any other process, +you can go through every map in the process, find the PFNs, look those up +in kpagecount, and tally up the number of pages that are only referenced +once. + +Other notes +=========== + +Reading from any of the files will return -EINVAL if you are not starting +the read on an 8-byte boundary (e.g., if you sought an odd number of bytes +into the file), or if the size of the read is not a multiple of 8 bytes. + +Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is +always 12 at most architectures). Since Linux 3.11 their meaning changes +after first clear of soft-dirty bits. Since Linux 4.2 they are used for +flags unconditionally. diff --git a/Documentation/admin-guide/mm/soft-dirty.rst b/Documentation/admin-guide/mm/soft-dirty.rst new file mode 100644 index 000000000000..cb0cfd6672fa --- /dev/null +++ b/Documentation/admin-guide/mm/soft-dirty.rst @@ -0,0 +1,47 @@ +.. _soft_dirty: + +=============== +Soft-Dirty PTEs +=============== + +The soft-dirty is a bit on a PTE which helps to track which pages a task +writes to. In order to do this tracking one should + + 1. Clear soft-dirty bits from the task's PTEs. + + This is done by writing "4" into the ``/proc/PID/clear_refs`` file of the + task in question. + + 2. Wait some time. + + 3. Read soft-dirty bits from the PTEs. + + This is done by reading from the ``/proc/PID/pagemap``. The bit 55 of the + 64-bit qword is the soft-dirty one. If set, the respective PTE was + written to since step 1. + + +Internally, to do this tracking, the writable bit is cleared from PTEs +when the soft-dirty bit is cleared. So, after this, when the task tries to +modify a page at some virtual address the #PF occurs and the kernel sets +the soft-dirty bit on the respective PTE. + +Note, that although all the task's address space is marked as r/o after the +soft-dirty bits clear, the #PF-s that occur after that are processed fast. +This is so, since the pages are still mapped to physical memory, and thus all +the kernel does is finds this fact out and puts both writable and soft-dirty +bits on the PTE. + +While in most cases tracking memory changes by #PF-s is more than enough +there is still a scenario when we can lose soft dirty bits -- a task +unmaps a previously mapped memory region and then maps a new one at exactly +the same place. When unmap is called, the kernel internally clears PTE values +including soft dirty bits. To notify user space application about such +memory region renewal the kernel always marks new memory regions (and +expanded regions) as soft dirty. + +This feature is actively used by the checkpoint-restore project. You +can find more details about it on http://criu.org + + +-- Pavel Emelyanov, Apr 9, 2013 diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst new file mode 100644 index 000000000000..7ab93a8404b9 --- /dev/null +++ b/Documentation/admin-guide/mm/transhuge.rst @@ -0,0 +1,418 @@ +.. _admin_guide_transhuge: + +============================ +Transparent Hugepage Support +============================ + +Objective +========= + +Performance critical computing applications dealing with large memory +working sets are already running on top of libhugetlbfs and in turn +hugetlbfs. Transparent HugePage Support (THP) is an alternative mean of +using huge pages for the backing of virtual memory with huge pages +that supports the automatic promotion and demotion of page sizes and +without the shortcomings of hugetlbfs. + +Currently THP only works for anonymous memory mappings and tmpfs/shmem. +But in the future it can expand to other filesystems. + +.. note:: + in the examples below we presume that the basic page size is 4K and + the huge page size is 2M, although the actual numbers may vary + depending on the CPU architecture. + +The reason applications are running faster is because of two +factors. The first factor is almost completely irrelevant and it's not +of significant interest because it'll also have the downside of +requiring larger clear-page copy-page in page faults which is a +potentially negative effect. The first factor consists in taking a +single page fault for each 2M virtual region touched by userland (so +reducing the enter/exit kernel frequency by a 512 times factor). This +only matters the first time the memory is accessed for the lifetime of +a memory mapping. The second long lasting and much more important +factor will affect all subsequent accesses to the memory for the whole +runtime of the application. The second factor consist of two +components: + +1) the TLB miss will run faster (especially with virtualization using + nested pagetables but almost always also on bare metal without + virtualization) + +2) a single TLB entry will be mapping a much larger amount of virtual + memory in turn reducing the number of TLB misses. With + virtualization and nested pagetables the TLB can be mapped of + larger size only if both KVM and the Linux guest are using + hugepages but a significant speedup already happens if only one of + the two is using hugepages just because of the fact the TLB miss is + going to run faster. + +THP can be enabled system wide or restricted to certain tasks or even +memory ranges inside task's address space. Unless THP is completely +disabled, there is ``khugepaged`` daemon that scans memory and +collapses sequences of basic pages into huge pages. + +The THP behaviour is controlled via :ref:`sysfs ` +interface and using madivse(2) and prctl(2) system calls. + +Transparent Hugepage Support maximizes the usefulness of free memory +if compared to the reservation approach of hugetlbfs by allowing all +unused memory to be used as cache or other movable (or even unmovable +entities). It doesn't require reservation to prevent hugepage +allocation failures to be noticeable from userland. It allows paging +and all other advanced VM features to be available on the +hugepages. It requires no modifications for applications to take +advantage of it. + +Applications however can be further optimized to take advantage of +this feature, like for example they've been optimized before to avoid +a flood of mmap system calls for every malloc(4k). Optimizing userland +is by far not mandatory and khugepaged already can take care of long +lived page allocations even for hugepage unaware applications that +deals with large amounts of memory. + +In certain cases when hugepages are enabled system wide, application +may end up allocating more memory resources. An application may mmap a +large region but only touch 1 byte of it, in that case a 2M page might +be allocated instead of a 4k page for no good. This is why it's +possible to disable hugepages system-wide and to only have them inside +MADV_HUGEPAGE madvise regions. + +Embedded systems should enable hugepages only inside madvise regions +to eliminate any risk of wasting any precious byte of memory and to +only run faster. + +Applications that gets a lot of benefit from hugepages and that don't +risk to lose memory by using hugepages, should use +madvise(MADV_HUGEPAGE) on their critical mmapped regions. + +.. _thp_sysfs: + +sysfs +===== + +Global THP controls +------------------- + +Transparent Hugepage Support for anonymous memory can be entirely disabled +(mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE +regions (to avoid the risk of consuming more memory resources) or enabled +system wide. This can be achieved with one of:: + + echo always >/sys/kernel/mm/transparent_hugepage/enabled + echo madvise >/sys/kernel/mm/transparent_hugepage/enabled + echo never >/sys/kernel/mm/transparent_hugepage/enabled + +It's also possible to limit defrag efforts in the VM to generate +anonymous hugepages in case they're not immediately free to madvise +regions or to never try to defrag memory and simply fallback to regular +pages unless hugepages are immediately available. Clearly if we spend CPU +time to defrag memory, we would expect to gain even more by the fact we +use hugepages later instead of regular pages. This isn't always +guaranteed, but it may be more likely in case the allocation is for a +MADV_HUGEPAGE region. + +:: + + echo always >/sys/kernel/mm/transparent_hugepage/defrag + echo defer >/sys/kernel/mm/transparent_hugepage/defrag + echo defer+madvise >/sys/kernel/mm/transparent_hugepage/defrag + echo madvise >/sys/kernel/mm/transparent_hugepage/defrag + echo never >/sys/kernel/mm/transparent_hugepage/defrag + +always + means that an application requesting THP will stall on + allocation failure and directly reclaim pages and compact + memory in an effort to allocate a THP immediately. This may be + desirable for virtual machines that benefit heavily from THP + use and are willing to delay the VM start to utilise them. + +defer + means that an application will wake kswapd in the background + to reclaim pages and wake kcompactd to compact memory so that + THP is available in the near future. It's the responsibility + of khugepaged to then install the THP pages later. + +defer+madvise + will enter direct reclaim and compaction like ``always``, but + only for regions that have used madvise(MADV_HUGEPAGE); all + other regions will wake kswapd in the background to reclaim + pages and wake kcompactd to compact memory so that THP is + available in the near future. + +madvise + will enter direct reclaim like ``always`` but only for regions + that are have used madvise(MADV_HUGEPAGE). This is the default + behaviour. + +never + should be self-explanatory. + +By default kernel tries to use huge zero page on read page fault to +anonymous mapping. It's possible to disable huge zero page by writing 0 +or enable it back by writing 1:: + + echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page + echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page + +Some userspace (such as a test program, or an optimized memory allocation +library) may want to know the size (in bytes) of a transparent hugepage:: + + cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size + +khugepaged will be automatically started when +transparent_hugepage/enabled is set to "always" or "madvise, and it'll +be automatically shutdown if it's set to "never". + +Khugepaged controls +------------------- + +khugepaged runs usually at low frequency so while one may not want to +invoke defrag algorithms synchronously during the page faults, it +should be worth invoking defrag at least in khugepaged. However it's +also possible to disable defrag in khugepaged by writing 0 or enable +defrag in khugepaged by writing 1:: + + echo 0 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag + echo 1 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag + +You can also control how many pages khugepaged should scan at each +pass:: + + /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan + +and how many milliseconds to wait in khugepaged between each pass (you +can set this to 0 to run khugepaged at 100% utilization of one core):: + + /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs + +and how many milliseconds to wait in khugepaged if there's an hugepage +allocation failure to throttle the next allocation attempt:: + + /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs + +The khugepaged progress can be seen in the number of pages collapsed:: + + /sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed + +for each pass:: + + /sys/kernel/mm/transparent_hugepage/khugepaged/full_scans + +``max_ptes_none`` specifies how many extra small pages (that are +not already mapped) can be allocated when collapsing a group +of small pages into one large page:: + + /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none + +A higher value leads to use additional memory for programs. +A lower value leads to gain less thp performance. Value of +max_ptes_none can waste cpu time very little, you can +ignore it. + +``max_ptes_swap`` specifies how many pages can be brought in from +swap when collapsing a group of pages into a transparent huge page:: + + /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_swap + +A higher value can cause excessive swap IO and waste +memory. A lower value can prevent THPs from being +collapsed, resulting fewer pages being collapsed into +THPs, and lower memory access performance. + +Boot parameter +============== + +You can change the sysfs boot time defaults of Transparent Hugepage +Support by passing the parameter ``transparent_hugepage=always`` or +``transparent_hugepage=madvise`` or ``transparent_hugepage=never`` +to the kernel command line. + +Hugepages in tmpfs/shmem +======================== + +You can control hugepage allocation policy in tmpfs with mount option +``huge=``. It can have following values: + +always + Attempt to allocate huge pages every time we need a new page; + +never + Do not allocate huge pages; + +within_size + Only allocate huge page if it will be fully within i_size. + Also respect fadvise()/madvise() hints; + +advise + Only allocate huge pages if requested with fadvise()/madvise(); + +The default policy is ``never``. + +``mount -o remount,huge= /mountpoint`` works fine after mount: remounting +``huge=never`` will not attempt to break up huge pages at all, just stop more +from being allocated. + +There's also sysfs knob to control hugepage allocation policy for internal +shmem mount: /sys/kernel/mm/transparent_hugepage/shmem_enabled. The mount +is used for SysV SHM, memfds, shared anonymous mmaps (of /dev/zero or +MAP_ANONYMOUS), GPU drivers' DRM objects, Ashmem. + +In addition to policies listed above, shmem_enabled allows two further +values: + +deny + For use in emergencies, to force the huge option off from + all mounts; +force + Force the huge option on for all - very useful for testing; + +Need of application restart +=========================== + +The transparent_hugepage/enabled values and tmpfs mount option only affect +future behavior. So to make them effective you need to restart any +application that could have been using hugepages. This also applies to the +regions registered in khugepaged. + +Monitoring usage +================ + +The number of anonymous transparent huge pages currently used by the +system is available by reading the AnonHugePages field in ``/proc/meminfo``. +To identify what applications are using anonymous transparent huge pages, +it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages fields +for each mapping. + +The number of file transparent huge pages mapped to userspace is available +by reading ShmemPmdMapped and ShmemHugePages fields in ``/proc/meminfo``. +To identify what applications are mapping file transparent huge pages, it +is necessary to read ``/proc/PID/smaps`` and count the FileHugeMapped fields +for each mapping. + +Note that reading the smaps file is expensive and reading it +frequently will incur overhead. + +There are a number of counters in ``/proc/vmstat`` that may be used to +monitor how successfully the system is providing huge pages for use. + +thp_fault_alloc + is incremented every time a huge page is successfully + allocated to handle a page fault. This applies to both the + first time a page is faulted and for COW faults. + +thp_collapse_alloc + is incremented by khugepaged when it has found + a range of pages to collapse into one huge page and has + successfully allocated a new huge page to store the data. + +thp_fault_fallback + is incremented if a page fault fails to allocate + a huge page and instead falls back to using small pages. + +thp_collapse_alloc_failed + is incremented if khugepaged found a range + of pages that should be collapsed into one huge page but failed + the allocation. + +thp_file_alloc + is incremented every time a file huge page is successfully + allocated. + +thp_file_mapped + is incremented every time a file huge page is mapped into + user address space. + +thp_split_page + is incremented every time a huge page is split into base + pages. This can happen for a variety of reasons but a common + reason is that a huge page is old and is being reclaimed. + This action implies splitting all PMD the page mapped with. + +thp_split_page_failed + is incremented if kernel fails to split huge + page. This can happen if the page was pinned by somebody. + +thp_deferred_split_page + is incremented when a huge page is put onto split + queue. This happens when a huge page is partially unmapped and + splitting it would free up some memory. Pages on split queue are + going to be split under memory pressure. + +thp_split_pmd + is incremented every time a PMD split into table of PTEs. + This can happen, for instance, when application calls mprotect() or + munmap() on part of huge page. It doesn't split huge page, only + page table entry. + +thp_zero_page_alloc + is incremented every time a huge zero page is + successfully allocated. It includes allocations which where + dropped due race with other allocation. Note, it doesn't count + every map of the huge zero page, only its allocation. + +thp_zero_page_alloc_failed + is incremented if kernel fails to allocate + huge zero page and falls back to using small pages. + +thp_swpout + is incremented every time a huge page is swapout in one + piece without splitting. + +thp_swpout_fallback + is incremented if a huge page has to be split before swapout. + Usually because failed to allocate some continuous swap space + for the huge page. + +As the system ages, allocating huge pages may be expensive as the +system uses memory compaction to copy data around memory to free a +huge page for use. There are some counters in ``/proc/vmstat`` to help +monitor this overhead. + +compact_stall + is incremented every time a process stalls to run + memory compaction so that a huge page is free for use. + +compact_success + is incremented if the system compacted memory and + freed a huge page for use. + +compact_fail + is incremented if the system tries to compact memory + but failed. + +compact_pages_moved + is incremented each time a page is moved. If + this value is increasing rapidly, it implies that the system + is copying a lot of data to satisfy the huge page allocation. + It is possible that the cost of copying exceeds any savings + from reduced TLB misses. + +compact_pagemigrate_failed + is incremented when the underlying mechanism + for moving a page failed. + +compact_blocks_moved + is incremented each time memory compaction examines + a huge page aligned range of pages. + +It is possible to establish how long the stalls were using the function +tracer to record how long was spent in __alloc_pages_nodemask and +using the mm_page_alloc tracepoint to identify which allocations were +for huge pages. + +Optimizing the applications +=========================== + +To be guaranteed that the kernel will map a 2M page immediately in any +memory region, the mmap region has to be hugepage naturally +aligned. posix_memalign() can provide that guarantee. + +Hugetlbfs +========= + +You can use hugetlbfs on a kernel that has transparent hugepage +support enabled just fine as always. No difference can be noted in +hugetlbfs other than there will be less overall fragmentation. All +usual features belonging to hugetlbfs are preserved and +unaffected. libhugetlbfs will also work fine as usual. diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst new file mode 100644 index 000000000000..5048cf661a8a --- /dev/null +++ b/Documentation/admin-guide/mm/userfaultfd.rst @@ -0,0 +1,241 @@ +.. _userfaultfd: + +=========== +Userfaultfd +=========== + +Objective +========= + +Userfaults allow the implementation of on-demand paging from userland +and more generally they allow userland to take control of various +memory page faults, something otherwise only the kernel code could do. + +For example userfaults allows a proper and more optimal implementation +of the PROT_NONE+SIGSEGV trick. + +Design +====== + +Userfaults are delivered and resolved through the userfaultfd syscall. + +The userfaultfd (aside from registering and unregistering virtual +memory ranges) provides two primary functionalities: + +1) read/POLLIN protocol to notify a userland thread of the faults + happening + +2) various UFFDIO_* ioctls that can manage the virtual memory regions + registered in the userfaultfd that allows userland to efficiently + resolve the userfaults it receives via 1) or to manage the virtual + memory in the background + +The real advantage of userfaults if compared to regular virtual memory +management of mremap/mprotect is that the userfaults in all their +operations never involve heavyweight structures like vmas (in fact the +userfaultfd runtime load never takes the mmap_sem for writing). + +Vmas are not suitable for page- (or hugepage) granular fault tracking +when dealing with virtual address spaces that could span +Terabytes. Too many vmas would be needed for that. + +The userfaultfd once opened by invoking the syscall, can also be +passed using unix domain sockets to a manager process, so the same +manager process could handle the userfaults of a multitude of +different processes without them being aware about what is going on +(well of course unless they later try to use the userfaultfd +themselves on the same region the manager is already tracking, which +is a corner case that would currently return -EBUSY). + +API +=== + +When first opened the userfaultfd must be enabled invoking the +UFFDIO_API ioctl specifying a uffdio_api.api value set to UFFD_API (or +a later API version) which will specify the read/POLLIN protocol +userland intends to speak on the UFFD and the uffdio_api.features +userland requires. The UFFDIO_API ioctl if successful (i.e. if the +requested uffdio_api.api is spoken also by the running kernel and the +requested features are going to be enabled) will return into +uffdio_api.features and uffdio_api.ioctls two 64bit bitmasks of +respectively all the available features of the read(2) protocol and +the generic ioctl available. + +The uffdio_api.features bitmask returned by the UFFDIO_API ioctl +defines what memory types are supported by the userfaultfd and what +events, except page fault notifications, may be generated. + +If the kernel supports registering userfaultfd ranges on hugetlbfs +virtual memory areas, UFFD_FEATURE_MISSING_HUGETLBFS will be set in +uffdio_api.features. Similarly, UFFD_FEATURE_MISSING_SHMEM will be +set if the kernel supports registering userfaultfd ranges on shared +memory (covering all shmem APIs, i.e. tmpfs, IPCSHM, /dev/zero +MAP_SHARED, memfd_create, etc). + +The userland application that wants to use userfaultfd with hugetlbfs +or shared memory need to set the corresponding flag in +uffdio_api.features to enable those features. + +If the userland desires to receive notifications for events other than +page faults, it has to verify that uffdio_api.features has appropriate +UFFD_FEATURE_EVENT_* bits set. These events are described in more +detail below in "Non-cooperative userfaultfd" section. + +Once the userfaultfd has been enabled the UFFDIO_REGISTER ioctl should +be invoked (if present in the returned uffdio_api.ioctls bitmask) to +register a memory range in the userfaultfd by setting the +uffdio_register structure accordingly. The uffdio_register.mode +bitmask will specify to the kernel which kind of faults to track for +the range (UFFDIO_REGISTER_MODE_MISSING would track missing +pages). The UFFDIO_REGISTER ioctl will return the +uffdio_register.ioctls bitmask of ioctls that are suitable to resolve +userfaults on the range registered. Not all ioctls will necessarily be +supported for all memory types depending on the underlying virtual +memory backend (anonymous memory vs tmpfs vs real filebacked +mappings). + +Userland can use the uffdio_register.ioctls to manage the virtual +address space in the background (to add or potentially also remove +memory from the userfaultfd registered range). This means a userfault +could be triggering just before userland maps in the background the +user-faulted page. + +The primary ioctl to resolve userfaults is UFFDIO_COPY. That +atomically copies a page into the userfault registered range and wakes +up the blocked userfaults (unless uffdio_copy.mode & +UFFDIO_COPY_MODE_DONTWAKE is set). Other ioctl works similarly to +UFFDIO_COPY. They're atomic as in guaranteeing that nothing can see an +half copied page since it'll keep userfaulting until the copy has +finished. + +QEMU/KVM +======== + +QEMU/KVM is using the userfaultfd syscall to implement postcopy live +migration. Postcopy live migration is one form of memory +externalization consisting of a virtual machine running with part or +all of its memory residing on a different node in the cloud. The +userfaultfd abstraction is generic enough that not a single line of +KVM kernel code had to be modified in order to add postcopy live +migration to QEMU. + +Guest async page faults, FOLL_NOWAIT and all other GUP features work +just fine in combination with userfaults. Userfaults trigger async +page faults in the guest scheduler so those guest processes that +aren't waiting for userfaults (i.e. network bound) can keep running in +the guest vcpus. + +It is generally beneficial to run one pass of precopy live migration +just before starting postcopy live migration, in order to avoid +generating userfaults for readonly guest regions. + +The implementation of postcopy live migration currently uses one +single bidirectional socket but in the future two different sockets +will be used (to reduce the latency of the userfaults to the minimum +possible without having to decrease /proc/sys/net/ipv4/tcp_wmem). + +The QEMU in the source node writes all pages that it knows are missing +in the destination node, into the socket, and the migration thread of +the QEMU running in the destination node runs UFFDIO_COPY|ZEROPAGE +ioctls on the userfaultfd in order to map the received pages into the +guest (UFFDIO_ZEROCOPY is used if the source page was a zero page). + +A different postcopy thread in the destination node listens with +poll() to the userfaultfd in parallel. When a POLLIN event is +generated after a userfault triggers, the postcopy thread read() from +the userfaultfd and receives the fault address (or -EAGAIN in case the +userfault was already resolved and waken by a UFFDIO_COPY|ZEROPAGE run +by the parallel QEMU migration thread). + +After the QEMU postcopy thread (running in the destination node) gets +the userfault address it writes the information about the missing page +into the socket. The QEMU source node receives the information and +roughly "seeks" to that page address and continues sending all +remaining missing pages from that new page offset. Soon after that +(just the time to flush the tcp_wmem queue through the network) the +migration thread in the QEMU running in the destination node will +receive the page that triggered the userfault and it'll map it as +usual with the UFFDIO_COPY|ZEROPAGE (without actually knowing if it +was spontaneously sent by the source or if it was an urgent page +requested through a userfault). + +By the time the userfaults start, the QEMU in the destination node +doesn't need to keep any per-page state bitmap relative to the live +migration around and a single per-page bitmap has to be maintained in +the QEMU running in the source node to know which pages are still +missing in the destination node. The bitmap in the source node is +checked to find which missing pages to send in round robin and we seek +over it when receiving incoming userfaults. After sending each page of +course the bitmap is updated accordingly. It's also useful to avoid +sending the same page twice (in case the userfault is read by the +postcopy thread just before UFFDIO_COPY|ZEROPAGE runs in the migration +thread). + +Non-cooperative userfaultfd +=========================== + +When the userfaultfd is monitored by an external manager, the manager +must be able to track changes in the process virtual memory +layout. Userfaultfd can notify the manager about such changes using +the same read(2) protocol as for the page fault notifications. The +manager has to explicitly enable these events by setting appropriate +bits in uffdio_api.features passed to UFFDIO_API ioctl: + +UFFD_FEATURE_EVENT_FORK + enable userfaultfd hooks for fork(). When this feature is + enabled, the userfaultfd context of the parent process is + duplicated into the newly created process. The manager + receives UFFD_EVENT_FORK with file descriptor of the new + userfaultfd context in the uffd_msg.fork. + +UFFD_FEATURE_EVENT_REMAP + enable notifications about mremap() calls. When the + non-cooperative process moves a virtual memory area to a + different location, the manager will receive + UFFD_EVENT_REMAP. The uffd_msg.remap will contain the old and + new addresses of the area and its original length. + +UFFD_FEATURE_EVENT_REMOVE + enable notifications about madvise(MADV_REMOVE) and + madvise(MADV_DONTNEED) calls. The event UFFD_EVENT_REMOVE will + be generated upon these calls to madvise. The uffd_msg.remove + will contain start and end addresses of the removed area. + +UFFD_FEATURE_EVENT_UNMAP + enable notifications about memory unmapping. The manager will + get UFFD_EVENT_UNMAP with uffd_msg.remove containing start and + end addresses of the unmapped area. + +Although the UFFD_FEATURE_EVENT_REMOVE and UFFD_FEATURE_EVENT_UNMAP +are pretty similar, they quite differ in the action expected from the +userfaultfd manager. In the former case, the virtual memory is +removed, but the area is not, the area remains monitored by the +userfaultfd, and if a page fault occurs in that area it will be +delivered to the manager. The proper resolution for such page fault is +to zeromap the faulting address. However, in the latter case, when an +area is unmapped, either explicitly (with munmap() system call), or +implicitly (e.g. during mremap()), the area is removed and in turn the +userfaultfd context for such area disappears too and the manager will +not get further userland page faults from the removed area. Still, the +notification is required in order to prevent manager from using +UFFDIO_COPY on the unmapped area. + +Unlike userland page faults which have to be synchronous and require +explicit or implicit wakeup, all the events are delivered +asynchronously and the non-cooperative process resumes execution as +soon as manager executes read(). The userfaultfd manager should +carefully synchronize calls to UFFDIO_COPY with the events +processing. To aid the synchronization, the UFFDIO_COPY ioctl will +return -ENOSPC when the monitored process exits at the time of +UFFDIO_COPY, and -ENOENT, when the non-cooperative process has changed +its virtual memory layout simultaneously with outstanding UFFDIO_COPY +operation. + +The current asynchronous model of the event delivery is optimal for +single threaded non-cooperative userfaultfd manager implementations. A +synchronous event delivery model can be added later as a new +userfaultfd feature to facilitate multithreading enhancements of the +non cooperative manager, for example to allow UFFDIO_COPY ioctls to +run in parallel to the event reception. Single threaded +implementations should continue to use the current async event +delivery model instead. diff --git a/Documentation/admin-guide/pm/intel_pstate.rst b/Documentation/admin-guide/pm/intel_pstate.rst index ab2fe0eda1d7..8b9164990956 100644 --- a/Documentation/admin-guide/pm/intel_pstate.rst +++ b/Documentation/admin-guide/pm/intel_pstate.rst @@ -410,7 +410,7 @@ argument is passed to the kernel in the command line. That only is supported in some configurations, though (for example, if the `HWP feature is enabled in the processor `_, the operation mode of the driver cannot be changed), and if it is not - supported in the current configuration, writes to this attribute with + supported in the current configuration, writes to this attribute will fail with an appropriate error. Interpretation of Policy Attributes diff --git a/Documentation/admin-guide/ramoops.rst b/Documentation/admin-guide/ramoops.rst index 4efd7ce77565..6dbcc5481000 100644 --- a/Documentation/admin-guide/ramoops.rst +++ b/Documentation/admin-guide/ramoops.rst @@ -61,7 +61,7 @@ Setting the ramoops parameters can be done in several different manners: mem=128M ramoops.mem_address=0x8000000 ramoops.ecc=1 B. Use Device Tree bindings, as described in - ``Documentation/device-tree/bindings/reserved-memory/admin-guide/ramoops.rst``. + ``Documentation/devicetree/bindings/reserved-memory/ramoops.txt``. For example:: reserved-memory { diff --git a/Documentation/arm/Marvell/README b/Documentation/arm/Marvell/README index b5bb7f518840..56ada27c53be 100644 --- a/Documentation/arm/Marvell/README +++ b/Documentation/arm/Marvell/README @@ -302,19 +302,15 @@ Berlin family (Multimedia Solutions) 88DE3010, Armada 1000 (no Linux support) Core: Marvell PJ1 (ARMv5TE), Dual-core Product Brief: http://www.marvell.com.cn/digital-entertainment/assets/armada_1000_pb.pdf - 88DE3005, Armada 1500-mini 88DE3005, Armada 1500 Mini Design name: BG2CD Core: ARM Cortex-A9, PL310 L2CC - Homepage: http://www.marvell.com/multimedia-solutions/armada-1500-mini/ - 88DE3006, Armada 1500 Mini Plus - Design name: BG2CDP - Core: Dual Core ARM Cortex-A7 - Homepage: http://www.marvell.com/multimedia-solutions/armada-1500-mini-plus/ + 88DE3006, Armada 1500 Mini Plus + Design name: BG2CDP + Core: Dual Core ARM Cortex-A7 88DE3100, Armada 1500 Design name: BG2 Core: Marvell PJ4B-MP (ARMv7), Tauros3 L2CC - Product Brief: http://www.marvell.com/digital-entertainment/armada-1500/assets/Marvell-ARMADA-1500-Product-Brief.pdf 88DE3114, Armada 1500 Pro Design name: BG2Q Core: Quad Core ARM Cortex-A9, PL310 L2CC @@ -324,13 +320,16 @@ Berlin family (Multimedia Solutions) 88DE3218, ARMADA 1500 Ultra Core: ARM Cortex-A53 - Homepage: http://www.marvell.com/multimedia-solutions/ + Homepage: https://www.synaptics.com/products/multimedia-solutions Directory: arch/arm/mach-berlin Comments: + * This line of SoCs is based on Marvell Sheeva or ARM Cortex CPUs with Synopsys DesignWare (IRQ, GPIO, Timers, ...) and PXA IP (SDHCI, USB, ETH, ...). + * The Berlin family was acquired by Synaptics from Marvell in 2017. + CPU Cores --------- diff --git a/Documentation/arm/OMAP/README b/Documentation/arm/OMAP/README index 75645c45d14a..90c6c57d61e8 100644 --- a/Documentation/arm/OMAP/README +++ b/Documentation/arm/OMAP/README @@ -5,3 +5,7 @@ KERNEL NEW DEPENDENCIES v4.3+ Update is needed for custom .config files to make sure CONFIG_REGULATOR_PBIAS is enabled for MMC1 to work properly. + +v4.18+ Update is needed for custom .config files to make sure + CONFIG_MMC_SDHCI_OMAP is enabled for all MMC instances + to work in DRA7 and K2G based boards. diff --git a/Documentation/misc-devices/lcd-panel-cgram.txt b/Documentation/auxdisplay/lcd-panel-cgram.txt similarity index 100% rename from Documentation/misc-devices/lcd-panel-cgram.txt rename to Documentation/auxdisplay/lcd-panel-cgram.txt diff --git a/Documentation/block/biodoc.txt b/Documentation/block/biodoc.txt index 86927029a52d..207eca58efaa 100644 --- a/Documentation/block/biodoc.txt +++ b/Documentation/block/biodoc.txt @@ -752,18 +752,6 @@ completion of the request to the block layer. This means ending tag operations before calling end_that_request_last()! For an example of a user of these helpers, see the IDE tagged command queueing support. -Certain hardware conditions may dictate a need to invalidate the block tag -queue. For instance, on IDE any tagged request error needs to clear both -the hardware and software block queue and enable the driver to sanely restart -all the outstanding requests. There's a third helper to do that: - - blk_queue_invalidate_tags(struct request_queue *q) - - Clear the internal block tag queue and re-add all the pending requests - to the request queue. The driver will receive them again on the - next request_fn run, just like it did the first time it encountered - them. - 3.2.5.2 Tag info Some block functions exist to query current tag status or to go from a @@ -805,8 +793,7 @@ Internally, block manages tags in the blk_queue_tag structure: Most of the above is simple and straight forward, however busy_list may need a bit of explaining. Normally we don't care too much about request ordering, but in the event of any barrier requests in the tag queue we need to ensure -that requests are restarted in the order they were queue. This may happen -if the driver needs to use blk_queue_invalidate_tags(). +that requests are restarted in the order they were queue. 3.3 I/O Submission diff --git a/Documentation/block/cmdline-partition.txt b/Documentation/block/cmdline-partition.txt index 525b9f6d7fb4..760a3f7c3ed4 100644 --- a/Documentation/block/cmdline-partition.txt +++ b/Documentation/block/cmdline-partition.txt @@ -1,7 +1,9 @@ Embedded device command line partition parsing ===================================================================== -Support for reading the block device partition table from the command line. +The "blkdevparts" command line option adds support for reading the +block device partition table from the kernel command line. + It is typically used for fixed block (eMMC) embedded devices. It has no MBR, so saves storage space. Bootloader can be easily accessed by absolute address of data on the block device. @@ -14,22 +16,27 @@ blkdevparts=[;] := [@](part-name) - block device disk name, embedded device used fixed block device, - it's disk name also fixed. such as: mmcblk0, mmcblk1, mmcblk0boot0. + block device disk name. Embedded device uses fixed block device. + Its disk name is also fixed, such as: mmcblk0, mmcblk1, mmcblk0boot0. partition size, in bytes, such as: 512, 1m, 1G. + size may contain an optional suffix of (upper or lower case): + K, M, G, T, P, E. + "-" is used to denote all remaining space. partition start address, in bytes. + offset may contain an optional suffix of (upper or lower case): + K, M, G, T, P, E. (part-name) - partition name, kernel send uevent with "PARTNAME". application can create - a link to block device partition with the name "PARTNAME". - user space application can access partition by partition name. + partition name. Kernel sends uevent with "PARTNAME". Application can + create a link to block device partition with the name "PARTNAME". + User space application can access partition by partition name. Example: - eMMC disk name is "mmcblk0" and "mmcblk0boot0" + eMMC disk names are "mmcblk0" and "mmcblk0boot0". bootargs: 'blkdevparts=mmcblk0:1G(data0),1G(data1),-;mmcblk0boot0:1m(boot),-(kernel)' diff --git a/Documentation/block/null_blk.txt b/Documentation/block/null_blk.txt index 733927a7b501..07f147381f32 100644 --- a/Documentation/block/null_blk.txt +++ b/Documentation/block/null_blk.txt @@ -71,13 +71,16 @@ use_per_node_hctx=[0/1]: Default: 0 1: The multi-queue block layer is instantiated with a hardware dispatch queue for each CPU node in the system. -use_lightnvm=[0/1]: Default: 0 - Register device with LightNVM. Requires blk-mq and CONFIG_NVM to be enabled. - no_sched=[0/1]: Default: 0 0: nullb* use default blk-mq io scheduler. 1: nullb* doesn't use io scheduler. +blocking=[0/1]: Default: 0 + 0: Register as a non-blocking blk-mq driver device. + 1: Register as a blocking blk-mq driver device, null_blk will set + the BLK_MQ_F_BLOCKING flag, indicating that it sometimes/always + needs to block in its ->queue_rq() function. + shared_tags=[0/1]: Default: 0 0: Tag set is not shared. 1: Tag set shared between devices for blk-mq. Only makes sense with diff --git a/Documentation/blockdev/zram.txt b/Documentation/blockdev/zram.txt index 257e65714c6a..875b2b56b87f 100644 --- a/Documentation/blockdev/zram.txt +++ b/Documentation/blockdev/zram.txt @@ -218,6 +218,7 @@ line of text and contains the following stats separated by whitespace: same_pages the number of same element filled pages written to this disk. No memory is allocated for such pages. pages_compacted the number of pages freed during compaction + huge_pages the number of incompressible pages 9) Deactivate: swapoff /dev/zram0 @@ -242,5 +243,29 @@ to backing storage rather than keeping it in memory. User should set up backing device via /sys/block/zramX/backing_dev before disksize setting. += memory tracking + +With CONFIG_ZRAM_MEMORY_TRACKING, user can know information of the +zram block. It could be useful to catch cold or incompressible +pages of the process with*pagemap. +If you enable the feature, you could see block state via +/sys/kernel/debug/zram/zram0/block_state". The output is as follows, + + 300 75.033841 .wh + 301 63.806904 s.. + 302 63.806919 ..h + +First column is zram's block index. +Second column is access time since the system was booted +Third column is state of the block. +(s: same page +w: written page to backing store +h: huge page) + +First line of above example says 300th block is accessed at 75.033841sec +and the block's state is huge so it is written back to the backing +storage. It's a debugging feature so anyone shouldn't rely on it to work +properly. + Nitin Gupta ngupta@vflare.org diff --git a/Documentation/bpf/README.rst b/Documentation/bpf/README.rst new file mode 100644 index 000000000000..b9a80c9e9392 --- /dev/null +++ b/Documentation/bpf/README.rst @@ -0,0 +1,36 @@ +================= +BPF documentation +================= + +This directory contains documentation for the BPF (Berkeley Packet +Filter) facility, with a focus on the extended BPF version (eBPF). + +This kernel side documentation is still work in progress. The main +textual documentation is (for historical reasons) described in +`Documentation/networking/filter.txt`_, which describe both classical +and extended BPF instruction-set. +The Cilium project also maintains a `BPF and XDP Reference Guide`_ +that goes into great technical depth about the BPF Architecture. + +The primary info for the bpf syscall is available in the `man-pages`_ +for `bpf(2)`_. + + + +Frequently asked questions (FAQ) +================================ + +Two sets of Questions and Answers (Q&A) are maintained. + +* QA for common questions about BPF see: bpf_design_QA_ + +* QA for developers interacting with BPF subsystem: bpf_devel_QA_ + + +.. Links: +.. _bpf_design_QA: bpf_design_QA.rst +.. _bpf_devel_QA: bpf_devel_QA.rst +.. _Documentation/networking/filter.txt: ../networking/filter.txt +.. _man-pages: https://www.kernel.org/doc/man-pages/ +.. _bpf(2): http://man7.org/linux/man-pages/man2/bpf.2.html +.. _BPF and XDP Reference Guide: http://cilium.readthedocs.io/en/latest/bpf/ diff --git a/Documentation/bpf/bpf_design_QA.rst b/Documentation/bpf/bpf_design_QA.rst new file mode 100644 index 000000000000..6780a6d81745 --- /dev/null +++ b/Documentation/bpf/bpf_design_QA.rst @@ -0,0 +1,221 @@ +============== +BPF Design Q&A +============== + +BPF extensibility and applicability to networking, tracing, security +in the linux kernel and several user space implementations of BPF +virtual machine led to a number of misunderstanding on what BPF actually is. +This short QA is an attempt to address that and outline a direction +of where BPF is heading long term. + +.. contents:: + :local: + :depth: 3 + +Questions and Answers +===================== + +Q: Is BPF a generic instruction set similar to x64 and arm64? +------------------------------------------------------------- +A: NO. + +Q: Is BPF a generic virtual machine ? +------------------------------------- +A: NO. + +BPF is generic instruction set *with* C calling convention. +----------------------------------------------------------- + +Q: Why C calling convention was chosen? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A: Because BPF programs are designed to run in the linux kernel +which is written in C, hence BPF defines instruction set compatible +with two most used architectures x64 and arm64 (and takes into +consideration important quirks of other architectures) and +defines calling convention that is compatible with C calling +convention of the linux kernel on those architectures. + +Q: can multiple return values be supported in the future? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +A: NO. BPF allows only register R0 to be used as return value. + +Q: can more than 5 function arguments be supported in the future? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +A: NO. BPF calling convention only allows registers R1-R5 to be used +as arguments. BPF is not a standalone instruction set. +(unlike x64 ISA that allows msft, cdecl and other conventions) + +Q: can BPF programs access instruction pointer or return address? +----------------------------------------------------------------- +A: NO. + +Q: can BPF programs access stack pointer ? +------------------------------------------ +A: NO. + +Only frame pointer (register R10) is accessible. +From compiler point of view it's necessary to have stack pointer. +For example LLVM defines register R11 as stack pointer in its +BPF backend, but it makes sure that generated code never uses it. + +Q: Does C-calling convention diminishes possible use cases? +----------------------------------------------------------- +A: YES. + +BPF design forces addition of major functionality in the form +of kernel helper functions and kernel objects like BPF maps with +seamless interoperability between them. It lets kernel call into +BPF programs and programs call kernel helpers with zero overhead. +As all of them were native C code. That is particularly the case +for JITed BPF programs that are indistinguishable from +native kernel C code. + +Q: Does it mean that 'innovative' extensions to BPF code are disallowed? +------------------------------------------------------------------------ +A: Soft yes. + +At least for now until BPF core has support for +bpf-to-bpf calls, indirect calls, loops, global variables, +jump tables, read only sections and all other normal constructs +that C code can produce. + +Q: Can loops be supported in a safe way? +---------------------------------------- +A: It's not clear yet. + +BPF developers are trying to find a way to +support bounded loops where the verifier can guarantee that +the program terminates in less than 4096 instructions. + +Instruction level questions +--------------------------- + +Q: LD_ABS and LD_IND instructions vs C code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Q: How come LD_ABS and LD_IND instruction are present in BPF whereas +C code cannot express them and has to use builtin intrinsics? + +A: This is artifact of compatibility with classic BPF. Modern +networking code in BPF performs better without them. +See 'direct packet access'. + +Q: BPF instructions mapping not one-to-one to native CPU +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Q: It seems not all BPF instructions are one-to-one to native CPU. +For example why BPF_JNE and other compare and jumps are not cpu-like? + +A: This was necessary to avoid introducing flags into ISA which are +impossible to make generic and efficient across CPU architectures. + +Q: why BPF_DIV instruction doesn't map to x64 div? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +A: Because if we picked one-to-one relationship to x64 it would have made +it more complicated to support on arm64 and other archs. Also it +needs div-by-zero runtime check. + +Q: why there is no BPF_SDIV for signed divide operation? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +A: Because it would be rarely used. llvm errors in such case and +prints a suggestion to use unsigned divide instead + +Q: Why BPF has implicit prologue and epilogue? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +A: Because architectures like sparc have register windows and in general +there are enough subtle differences between architectures, so naive +store return address into stack won't work. Another reason is BPF has +to be safe from division by zero (and legacy exception path +of LD_ABS insn). Those instructions need to invoke epilogue and +return implicitly. + +Q: Why BPF_JLT and BPF_JLE instructions were not introduced in the beginning? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +A: Because classic BPF didn't have them and BPF authors felt that compiler +workaround would be acceptable. Turned out that programs lose performance +due to lack of these compare instructions and they were added. +These two instructions is a perfect example what kind of new BPF +instructions are acceptable and can be added in the future. +These two already had equivalent instructions in native CPUs. +New instructions that don't have one-to-one mapping to HW instructions +will not be accepted. + +Q: BPF 32-bit subregister requirements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Q: BPF 32-bit subregisters have a requirement to zero upper 32-bits of BPF +registers which makes BPF inefficient virtual machine for 32-bit +CPU architectures and 32-bit HW accelerators. Can true 32-bit registers +be added to BPF in the future? + +A: NO. The first thing to improve performance on 32-bit archs is to teach +LLVM to generate code that uses 32-bit subregisters. Then second step +is to teach verifier to mark operations where zero-ing upper bits +is unnecessary. Then JITs can take advantage of those markings and +drastically reduce size of generated code and improve performance. + +Q: Does BPF have a stable ABI? +------------------------------ +A: YES. BPF instructions, arguments to BPF programs, set of helper +functions and their arguments, recognized return codes are all part +of ABI. However when tracing programs are using bpf_probe_read() helper +to walk kernel internal datastructures and compile with kernel +internal headers these accesses can and will break with newer +kernels. The union bpf_attr -> kern_version is checked at load time +to prevent accidentally loading kprobe-based bpf programs written +for a different kernel. Networking programs don't do kern_version check. + +Q: How much stack space a BPF program uses? +------------------------------------------- +A: Currently all program types are limited to 512 bytes of stack +space, but the verifier computes the actual amount of stack used +and both interpreter and most JITed code consume necessary amount. + +Q: Can BPF be offloaded to HW? +------------------------------ +A: YES. BPF HW offload is supported by NFP driver. + +Q: Does classic BPF interpreter still exist? +-------------------------------------------- +A: NO. Classic BPF programs are converted into extend BPF instructions. + +Q: Can BPF call arbitrary kernel functions? +------------------------------------------- +A: NO. BPF programs can only call a set of helper functions which +is defined for every program type. + +Q: Can BPF overwrite arbitrary kernel memory? +--------------------------------------------- +A: NO. + +Tracing bpf programs can *read* arbitrary memory with bpf_probe_read() +and bpf_probe_read_str() helpers. Networking programs cannot read +arbitrary memory, since they don't have access to these helpers. +Programs can never read or write arbitrary memory directly. + +Q: Can BPF overwrite arbitrary user memory? +------------------------------------------- +A: Sort-of. + +Tracing BPF programs can overwrite the user memory +of the current task with bpf_probe_write_user(). Every time such +program is loaded the kernel will print warning message, so +this helper is only useful for experiments and prototypes. +Tracing BPF programs are root only. + +Q: bpf_trace_printk() helper warning +------------------------------------ +Q: When bpf_trace_printk() helper is used the kernel prints nasty +warning message. Why is that? + +A: This is done to nudge program authors into better interfaces when +programs need to pass data to user space. Like bpf_perf_event_output() +can be used to efficiently stream data via perf ring buffer. +BPF maps can be used for asynchronous data sharing between kernel +and user space. bpf_trace_printk() should only be used for debugging. + +Q: New functionality via kernel modules? +---------------------------------------- +Q: Can BPF functionality such as new program or map types, new +helpers, etc be added out of kernel module code? + +A: NO. diff --git a/Documentation/bpf/bpf_design_QA.txt b/Documentation/bpf/bpf_design_QA.txt deleted file mode 100644 index f3e458a0bb2f..000000000000 --- a/Documentation/bpf/bpf_design_QA.txt +++ /dev/null @@ -1,156 +0,0 @@ -BPF extensibility and applicability to networking, tracing, security -in the linux kernel and several user space implementations of BPF -virtual machine led to a number of misunderstanding on what BPF actually is. -This short QA is an attempt to address that and outline a direction -of where BPF is heading long term. - -Q: Is BPF a generic instruction set similar to x64 and arm64? -A: NO. - -Q: Is BPF a generic virtual machine ? -A: NO. - -BPF is generic instruction set _with_ C calling convention. - -Q: Why C calling convention was chosen? -A: Because BPF programs are designed to run in the linux kernel - which is written in C, hence BPF defines instruction set compatible - with two most used architectures x64 and arm64 (and takes into - consideration important quirks of other architectures) and - defines calling convention that is compatible with C calling - convention of the linux kernel on those architectures. - -Q: can multiple return values be supported in the future? -A: NO. BPF allows only register R0 to be used as return value. - -Q: can more than 5 function arguments be supported in the future? -A: NO. BPF calling convention only allows registers R1-R5 to be used - as arguments. BPF is not a standalone instruction set. - (unlike x64 ISA that allows msft, cdecl and other conventions) - -Q: can BPF programs access instruction pointer or return address? -A: NO. - -Q: can BPF programs access stack pointer ? -A: NO. Only frame pointer (register R10) is accessible. - From compiler point of view it's necessary to have stack pointer. - For example LLVM defines register R11 as stack pointer in its - BPF backend, but it makes sure that generated code never uses it. - -Q: Does C-calling convention diminishes possible use cases? -A: YES. BPF design forces addition of major functionality in the form - of kernel helper functions and kernel objects like BPF maps with - seamless interoperability between them. It lets kernel call into - BPF programs and programs call kernel helpers with zero overhead. - As all of them were native C code. That is particularly the case - for JITed BPF programs that are indistinguishable from - native kernel C code. - -Q: Does it mean that 'innovative' extensions to BPF code are disallowed? -A: Soft yes. At least for now until BPF core has support for - bpf-to-bpf calls, indirect calls, loops, global variables, - jump tables, read only sections and all other normal constructs - that C code can produce. - -Q: Can loops be supported in a safe way? -A: It's not clear yet. BPF developers are trying to find a way to - support bounded loops where the verifier can guarantee that - the program terminates in less than 4096 instructions. - -Q: How come LD_ABS and LD_IND instruction are present in BPF whereas - C code cannot express them and has to use builtin intrinsics? -A: This is artifact of compatibility with classic BPF. Modern - networking code in BPF performs better without them. - See 'direct packet access'. - -Q: It seems not all BPF instructions are one-to-one to native CPU. - For example why BPF_JNE and other compare and jumps are not cpu-like? -A: This was necessary to avoid introducing flags into ISA which are - impossible to make generic and efficient across CPU architectures. - -Q: why BPF_DIV instruction doesn't map to x64 div? -A: Because if we picked one-to-one relationship to x64 it would have made - it more complicated to support on arm64 and other archs. Also it - needs div-by-zero runtime check. - -Q: why there is no BPF_SDIV for signed divide operation? -A: Because it would be rarely used. llvm errors in such case and - prints a suggestion to use unsigned divide instead - -Q: Why BPF has implicit prologue and epilogue? -A: Because architectures like sparc have register windows and in general - there are enough subtle differences between architectures, so naive - store return address into stack won't work. Another reason is BPF has - to be safe from division by zero (and legacy exception path - of LD_ABS insn). Those instructions need to invoke epilogue and - return implicitly. - -Q: Why BPF_JLT and BPF_JLE instructions were not introduced in the beginning? -A: Because classic BPF didn't have them and BPF authors felt that compiler - workaround would be acceptable. Turned out that programs lose performance - due to lack of these compare instructions and they were added. - These two instructions is a perfect example what kind of new BPF - instructions are acceptable and can be added in the future. - These two already had equivalent instructions in native CPUs. - New instructions that don't have one-to-one mapping to HW instructions - will not be accepted. - -Q: BPF 32-bit subregisters have a requirement to zero upper 32-bits of BPF - registers which makes BPF inefficient virtual machine for 32-bit - CPU architectures and 32-bit HW accelerators. Can true 32-bit registers - be added to BPF in the future? -A: NO. The first thing to improve performance on 32-bit archs is to teach - LLVM to generate code that uses 32-bit subregisters. Then second step - is to teach verifier to mark operations where zero-ing upper bits - is unnecessary. Then JITs can take advantage of those markings and - drastically reduce size of generated code and improve performance. - -Q: Does BPF have a stable ABI? -A: YES. BPF instructions, arguments to BPF programs, set of helper - functions and their arguments, recognized return codes are all part - of ABI. However when tracing programs are using bpf_probe_read() helper - to walk kernel internal datastructures and compile with kernel - internal headers these accesses can and will break with newer - kernels. The union bpf_attr -> kern_version is checked at load time - to prevent accidentally loading kprobe-based bpf programs written - for a different kernel. Networking programs don't do kern_version check. - -Q: How much stack space a BPF program uses? -A: Currently all program types are limited to 512 bytes of stack - space, but the verifier computes the actual amount of stack used - and both interpreter and most JITed code consume necessary amount. - -Q: Can BPF be offloaded to HW? -A: YES. BPF HW offload is supported by NFP driver. - -Q: Does classic BPF interpreter still exist? -A: NO. Classic BPF programs are converted into extend BPF instructions. - -Q: Can BPF call arbitrary kernel functions? -A: NO. BPF programs can only call a set of helper functions which - is defined for every program type. - -Q: Can BPF overwrite arbitrary kernel memory? -A: NO. Tracing bpf programs can _read_ arbitrary memory with bpf_probe_read() - and bpf_probe_read_str() helpers. Networking programs cannot read - arbitrary memory, since they don't have access to these helpers. - Programs can never read or write arbitrary memory directly. - -Q: Can BPF overwrite arbitrary user memory? -A: Sort-of. Tracing BPF programs can overwrite the user memory - of the current task with bpf_probe_write_user(). Every time such - program is loaded the kernel will print warning message, so - this helper is only useful for experiments and prototypes. - Tracing BPF programs are root only. - -Q: When bpf_trace_printk() helper is used the kernel prints nasty - warning message. Why is that? -A: This is done to nudge program authors into better interfaces when - programs need to pass data to user space. Like bpf_perf_event_output() - can be used to efficiently stream data via perf ring buffer. - BPF maps can be used for asynchronous data sharing between kernel - and user space. bpf_trace_printk() should only be used for debugging. - -Q: Can BPF functionality such as new program or map types, new - helpers, etc be added out of kernel module code? -A: NO. diff --git a/Documentation/bpf/bpf_devel_QA.rst b/Documentation/bpf/bpf_devel_QA.rst new file mode 100644 index 000000000000..0e7c1d946e83 --- /dev/null +++ b/Documentation/bpf/bpf_devel_QA.rst @@ -0,0 +1,640 @@ +================================= +HOWTO interact with BPF subsystem +================================= + +This document provides information for the BPF subsystem about various +workflows related to reporting bugs, submitting patches, and queueing +patches for stable kernels. + +For general information about submitting patches, please refer to +`Documentation/process/`_. This document only describes additional specifics +related to BPF. + +.. contents:: + :local: + :depth: 2 + +Reporting bugs +============== + +Q: How do I report bugs for BPF kernel code? +-------------------------------------------- +A: Since all BPF kernel development as well as bpftool and iproute2 BPF +loader development happens through the netdev kernel mailing list, +please report any found issues around BPF to the following mailing +list: + + netdev@vger.kernel.org + +This may also include issues related to XDP, BPF tracing, etc. + +Given netdev has a high volume of traffic, please also add the BPF +maintainers to Cc (from kernel MAINTAINERS_ file): + +* Alexei Starovoitov +* Daniel Borkmann + +In case a buggy commit has already been identified, make sure to keep +the actual commit authors in Cc as well for the report. They can +typically be identified through the kernel's git tree. + +**Please do NOT report BPF issues to bugzilla.kernel.org since it +is a guarantee that the reported issue will be overlooked.** + +Submitting patches +================== + +Q: To which mailing list do I need to submit my BPF patches? +------------------------------------------------------------ +A: Please submit your BPF patches to the netdev kernel mailing list: + + netdev@vger.kernel.org + +Historically, BPF came out of networking and has always been maintained +by the kernel networking community. Although these days BPF touches +many other subsystems as well, the patches are still routed mainly +through the networking community. + +In case your patch has changes in various different subsystems (e.g. +tracing, security, etc), make sure to Cc the related kernel mailing +lists and maintainers from there as well, so they are able to review +the changes and provide their Acked-by's to the patches. + +Q: Where can I find patches currently under discussion for BPF subsystem? +------------------------------------------------------------------------- +A: All patches that are Cc'ed to netdev are queued for review under netdev +patchwork project: + + http://patchwork.ozlabs.org/project/netdev/list/ + +Those patches which target BPF, are assigned to a 'bpf' delegate for +further processing from BPF maintainers. The current queue with +patches under review can be found at: + + https://patchwork.ozlabs.org/project/netdev/list/?delegate=77147 + +Once the patches have been reviewed by the BPF community as a whole +and approved by the BPF maintainers, their status in patchwork will be +changed to 'Accepted' and the submitter will be notified by mail. This +means that the patches look good from a BPF perspective and have been +applied to one of the two BPF kernel trees. + +In case feedback from the community requires a respin of the patches, +their status in patchwork will be set to 'Changes Requested', and purged +from the current review queue. Likewise for cases where patches would +get rejected or are not applicable to the BPF trees (but assigned to +the 'bpf' delegate). + +Q: How do the changes make their way into Linux? +------------------------------------------------ +A: There are two BPF kernel trees (git repositories). Once patches have +been accepted by the BPF maintainers, they will be applied to one +of the two BPF trees: + + * https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/ + * https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/ + +The bpf tree itself is for fixes only, whereas bpf-next for features, +cleanups or other kind of improvements ("next-like" content). This is +analogous to net and net-next trees for networking. Both bpf and +bpf-next will only have a master branch in order to simplify against +which branch patches should get rebased to. + +Accumulated BPF patches in the bpf tree will regularly get pulled +into the net kernel tree. Likewise, accumulated BPF patches accepted +into the bpf-next tree will make their way into net-next tree. net and +net-next are both run by David S. Miller. From there, they will go +into the kernel mainline tree run by Linus Torvalds. To read up on the +process of net and net-next being merged into the mainline tree, see +the `netdev FAQ`_ under: + + `Documentation/networking/netdev-FAQ.txt`_ + +Occasionally, to prevent merge conflicts, we might send pull requests +to other trees (e.g. tracing) with a small subset of the patches, but +net and net-next are always the main trees targeted for integration. + +The pull requests will contain a high-level summary of the accumulated +patches and can be searched on netdev kernel mailing list through the +following subject lines (``yyyy-mm-dd`` is the date of the pull +request):: + + pull-request: bpf yyyy-mm-dd + pull-request: bpf-next yyyy-mm-dd + +Q: How do I indicate which tree (bpf vs. bpf-next) my patch should be applied to? +--------------------------------------------------------------------------------- + +A: The process is the very same as described in the `netdev FAQ`_, so +please read up on it. The subject line must indicate whether the +patch is a fix or rather "next-like" content in order to let the +maintainers know whether it is targeted at bpf or bpf-next. + +For fixes eventually landing in bpf -> net tree, the subject must +look like:: + + git format-patch --subject-prefix='PATCH bpf' start..finish + +For features/improvements/etc that should eventually land in +bpf-next -> net-next, the subject must look like:: + + git format-patch --subject-prefix='PATCH bpf-next' start..finish + +If unsure whether the patch or patch series should go into bpf +or net directly, or bpf-next or net-next directly, it is not a +problem either if the subject line says net or net-next as target. +It is eventually up to the maintainers to do the delegation of +the patches. + +If it is clear that patches should go into bpf or bpf-next tree, +please make sure to rebase the patches against those trees in +order to reduce potential conflicts. + +In case the patch or patch series has to be reworked and sent out +again in a second or later revision, it is also required to add a +version number (``v2``, ``v3``, ...) into the subject prefix:: + + git format-patch --subject-prefix='PATCH net-next v2' start..finish + +When changes have been requested to the patch series, always send the +whole patch series again with the feedback incorporated (never send +individual diffs on top of the old series). + +Q: What does it mean when a patch gets applied to bpf or bpf-next tree? +----------------------------------------------------------------------- +A: It means that the patch looks good for mainline inclusion from +a BPF point of view. + +Be aware that this is not a final verdict that the patch will +automatically get accepted into net or net-next trees eventually: + +On the netdev kernel mailing list reviews can come in at any point +in time. If discussions around a patch conclude that they cannot +get included as-is, we will either apply a follow-up fix or drop +them from the trees entirely. Therefore, we also reserve to rebase +the trees when deemed necessary. After all, the purpose of the tree +is to: + +i) accumulate and stage BPF patches for integration into trees + like net and net-next, and + +ii) run extensive BPF test suite and + workloads on the patches before they make their way any further. + +Once the BPF pull request was accepted by David S. Miller, then +the patches end up in net or net-next tree, respectively, and +make their way from there further into mainline. Again, see the +`netdev FAQ`_ for additional information e.g. on how often they are +merged to mainline. + +Q: How long do I need to wait for feedback on my BPF patches? +------------------------------------------------------------- +A: We try to keep the latency low. The usual time to feedback will +be around 2 or 3 business days. It may vary depending on the +complexity of changes and current patch load. + +Q: How often do you send pull requests to major kernel trees like net or net-next? +---------------------------------------------------------------------------------- + +A: Pull requests will be sent out rather often in order to not +accumulate too many patches in bpf or bpf-next. + +As a rule of thumb, expect pull requests for each tree regularly +at the end of the week. In some cases pull requests could additionally +come also in the middle of the week depending on the current patch +load or urgency. + +Q: Are patches applied to bpf-next when the merge window is open? +----------------------------------------------------------------- +A: For the time when the merge window is open, bpf-next will not be +processed. This is roughly analogous to net-next patch processing, +so feel free to read up on the `netdev FAQ`_ about further details. + +During those two weeks of merge window, we might ask you to resend +your patch series once bpf-next is open again. Once Linus released +a ``v*-rc1`` after the merge window, we continue processing of bpf-next. + +For non-subscribers to kernel mailing lists, there is also a status +page run by David S. Miller on net-next that provides guidance: + + http://vger.kernel.org/~davem/net-next.html + +Q: Verifier changes and test cases +---------------------------------- +Q: I made a BPF verifier change, do I need to add test cases for +BPF kernel selftests_? + +A: If the patch has changes to the behavior of the verifier, then yes, +it is absolutely necessary to add test cases to the BPF kernel +selftests_ suite. If they are not present and we think they are +needed, then we might ask for them before accepting any changes. + +In particular, test_verifier.c is tracking a high number of BPF test +cases, including a lot of corner cases that LLVM BPF back end may +generate out of the restricted C code. Thus, adding test cases is +absolutely crucial to make sure future changes do not accidentally +affect prior use-cases. Thus, treat those test cases as: verifier +behavior that is not tracked in test_verifier.c could potentially +be subject to change. + +Q: samples/bpf preference vs selftests? +--------------------------------------- +Q: When should I add code to `samples/bpf/`_ and when to BPF kernel +selftests_ ? + +A: In general, we prefer additions to BPF kernel selftests_ rather than +`samples/bpf/`_. The rationale is very simple: kernel selftests are +regularly run by various bots to test for kernel regressions. + +The more test cases we add to BPF selftests, the better the coverage +and the less likely it is that those could accidentally break. It is +not that BPF kernel selftests cannot demo how a specific feature can +be used. + +That said, `samples/bpf/`_ may be a good place for people to get started, +so it might be advisable that simple demos of features could go into +`samples/bpf/`_, but advanced functional and corner-case testing rather +into kernel selftests. + +If your sample looks like a test case, then go for BPF kernel selftests +instead! + +Q: When should I add code to the bpftool? +----------------------------------------- +A: The main purpose of bpftool (under tools/bpf/bpftool/) is to provide +a central user space tool for debugging and introspection of BPF programs +and maps that are active in the kernel. If UAPI changes related to BPF +enable for dumping additional information of programs or maps, then +bpftool should be extended as well to support dumping them. + +Q: When should I add code to iproute2's BPF loader? +--------------------------------------------------- +A: For UAPI changes related to the XDP or tc layer (e.g. ``cls_bpf``), +the convention is that those control-path related changes are added to +iproute2's BPF loader as well from user space side. This is not only +useful to have UAPI changes properly designed to be usable, but also +to make those changes available to a wider user base of major +downstream distributions. + +Q: Do you accept patches as well for iproute2's BPF loader? +----------------------------------------------------------- +A: Patches for the iproute2's BPF loader have to be sent to: + + netdev@vger.kernel.org + +While those patches are not processed by the BPF kernel maintainers, +please keep them in Cc as well, so they can be reviewed. + +The official git repository for iproute2 is run by Stephen Hemminger +and can be found at: + + https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/ + +The patches need to have a subject prefix of '``[PATCH iproute2 +master]``' or '``[PATCH iproute2 net-next]``'. '``master``' or +'``net-next``' describes the target branch where the patch should be +applied to. Meaning, if kernel changes went into the net-next kernel +tree, then the related iproute2 changes need to go into the iproute2 +net-next branch, otherwise they can be targeted at master branch. The +iproute2 net-next branch will get merged into the master branch after +the current iproute2 version from master has been released. + +Like BPF, the patches end up in patchwork under the netdev project and +are delegated to 'shemminger' for further processing: + + http://patchwork.ozlabs.org/project/netdev/list/?delegate=389 + +Q: What is the minimum requirement before I submit my BPF patches? +------------------------------------------------------------------ +A: When submitting patches, always take the time and properly test your +patches *prior* to submission. Never rush them! If maintainers find +that your patches have not been properly tested, it is a good way to +get them grumpy. Testing patch submissions is a hard requirement! + +Note, fixes that go to bpf tree *must* have a ``Fixes:`` tag included. +The same applies to fixes that target bpf-next, where the affected +commit is in net-next (or in some cases bpf-next). The ``Fixes:`` tag is +crucial in order to identify follow-up commits and tremendously helps +for people having to do backporting, so it is a must have! + +We also don't accept patches with an empty commit message. Take your +time and properly write up a high quality commit message, it is +essential! + +Think about it this way: other developers looking at your code a month +from now need to understand *why* a certain change has been done that +way, and whether there have been flaws in the analysis or assumptions +that the original author did. Thus providing a proper rationale and +describing the use-case for the changes is a must. + +Patch submissions with >1 patch must have a cover letter which includes +a high level description of the series. This high level summary will +then be placed into the merge commit by the BPF maintainers such that +it is also accessible from the git log for future reference. + +Q: Features changing BPF JIT and/or LLVM +---------------------------------------- +Q: What do I need to consider when adding a new instruction or feature +that would require BPF JIT and/or LLVM integration as well? + +A: We try hard to keep all BPF JITs up to date such that the same user +experience can be guaranteed when running BPF programs on different +architectures without having the program punt to the less efficient +interpreter in case the in-kernel BPF JIT is enabled. + +If you are unable to implement or test the required JIT changes for +certain architectures, please work together with the related BPF JIT +developers in order to get the feature implemented in a timely manner. +Please refer to the git log (``arch/*/net/``) to locate the necessary +people for helping out. + +Also always make sure to add BPF test cases (e.g. test_bpf.c and +test_verifier.c) for new instructions, so that they can receive +broad test coverage and help run-time testing the various BPF JITs. + +In case of new BPF instructions, once the changes have been accepted +into the Linux kernel, please implement support into LLVM's BPF back +end. See LLVM_ section below for further information. + +Stable submission +================= + +Q: I need a specific BPF commit in stable kernels. What should I do? +-------------------------------------------------------------------- +A: In case you need a specific fix in stable kernels, first check whether +the commit has already been applied in the related ``linux-*.y`` branches: + + https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/ + +If not the case, then drop an email to the BPF maintainers with the +netdev kernel mailing list in Cc and ask for the fix to be queued up: + + netdev@vger.kernel.org + +The process in general is the same as on netdev itself, see also the +`netdev FAQ`_ document. + +Q: Do you also backport to kernels not currently maintained as stable? +---------------------------------------------------------------------- +A: No. If you need a specific BPF commit in kernels that are currently not +maintained by the stable maintainers, then you are on your own. + +The current stable and longterm stable kernels are all listed here: + + https://www.kernel.org/ + +Q: The BPF patch I am about to submit needs to go to stable as well +------------------------------------------------------------------- +What should I do? + +A: The same rules apply as with netdev patch submissions in general, see +`netdev FAQ`_ under: + + `Documentation/networking/netdev-FAQ.txt`_ + +Never add "``Cc: stable@vger.kernel.org``" to the patch description, but +ask the BPF maintainers to queue the patches instead. This can be done +with a note, for example, under the ``---`` part of the patch which does +not go into the git log. Alternatively, this can be done as a simple +request by mail instead. + +Q: Queue stable patches +----------------------- +Q: Where do I find currently queued BPF patches that will be submitted +to stable? + +A: Once patches that fix critical bugs got applied into the bpf tree, they +are queued up for stable submission under: + + http://patchwork.ozlabs.org/bundle/bpf/stable/?state=* + +They will be on hold there at minimum until the related commit made its +way into the mainline kernel tree. + +After having been under broader exposure, the queued patches will be +submitted by the BPF maintainers to the stable maintainers. + +Testing patches +=============== + +Q: How to run BPF selftests +--------------------------- +A: After you have booted into the newly compiled kernel, navigate to +the BPF selftests_ suite in order to test BPF functionality (current +working directory points to the root of the cloned git tree):: + + $ cd tools/testing/selftests/bpf/ + $ make + +To run the verifier tests:: + + $ sudo ./test_verifier + +The verifier tests print out all the current checks being +performed. The summary at the end of running all tests will dump +information of test successes and failures:: + + Summary: 418 PASSED, 0 FAILED + +In order to run through all BPF selftests, the following command is +needed:: + + $ sudo make run_tests + +See the kernels selftest `Documentation/dev-tools/kselftest.rst`_ +document for further documentation. + +Q: Which BPF kernel selftests version should I run my kernel against? +--------------------------------------------------------------------- +A: If you run a kernel ``xyz``, then always run the BPF kernel selftests +from that kernel ``xyz`` as well. Do not expect that the BPF selftest +from the latest mainline tree will pass all the time. + +In particular, test_bpf.c and test_verifier.c have a large number of +test cases and are constantly updated with new BPF test sequences, or +existing ones are adapted to verifier changes e.g. due to verifier +becoming smarter and being able to better track certain things. + +LLVM +==== + +Q: Where do I find LLVM with BPF support? +----------------------------------------- +A: The BPF back end for LLVM is upstream in LLVM since version 3.7.1. + +All major distributions these days ship LLVM with BPF back end enabled, +so for the majority of use-cases it is not required to compile LLVM by +hand anymore, just install the distribution provided package. + +LLVM's static compiler lists the supported targets through +``llc --version``, make sure BPF targets are listed. Example:: + + $ llc --version + LLVM (http://llvm.org/): + LLVM version 6.0.0svn + Optimized build. + Default target: x86_64-unknown-linux-gnu + Host CPU: skylake + + Registered Targets: + bpf - BPF (host endian) + bpfeb - BPF (big endian) + bpfel - BPF (little endian) + x86 - 32-bit X86: Pentium-Pro and above + x86-64 - 64-bit X86: EM64T and AMD64 + +For developers in order to utilize the latest features added to LLVM's +BPF back end, it is advisable to run the latest LLVM releases. Support +for new BPF kernel features such as additions to the BPF instruction +set are often developed together. + +All LLVM releases can be found at: http://releases.llvm.org/ + +Q: Got it, so how do I build LLVM manually anyway? +-------------------------------------------------- +A: You need cmake and gcc-c++ as build requisites for LLVM. Once you have +that set up, proceed with building the latest LLVM and clang version +from the git repositories:: + + $ git clone http://llvm.org/git/llvm.git + $ cd llvm/tools + $ git clone --depth 1 http://llvm.org/git/clang.git + $ cd ..; mkdir build; cd build + $ cmake .. -DLLVM_TARGETS_TO_BUILD="BPF;X86" \ + -DBUILD_SHARED_LIBS=OFF \ + -DCMAKE_BUILD_TYPE=Release \ + -DLLVM_BUILD_RUNTIME=OFF + $ make -j $(getconf _NPROCESSORS_ONLN) + +The built binaries can then be found in the build/bin/ directory, where +you can point the PATH variable to. + +Q: Reporting LLVM BPF issues +---------------------------- +Q: Should I notify BPF kernel maintainers about issues in LLVM's BPF code +generation back end or about LLVM generated code that the verifier +refuses to accept? + +A: Yes, please do! + +LLVM's BPF back end is a key piece of the whole BPF +infrastructure and it ties deeply into verification of programs from the +kernel side. Therefore, any issues on either side need to be investigated +and fixed whenever necessary. + +Therefore, please make sure to bring them up at netdev kernel mailing +list and Cc BPF maintainers for LLVM and kernel bits: + +* Yonghong Song +* Alexei Starovoitov +* Daniel Borkmann + +LLVM also has an issue tracker where BPF related bugs can be found: + + https://bugs.llvm.org/buglist.cgi?quicksearch=bpf + +However, it is better to reach out through mailing lists with having +maintainers in Cc. + +Q: New BPF instruction for kernel and LLVM +------------------------------------------ +Q: I have added a new BPF instruction to the kernel, how can I integrate +it into LLVM? + +A: LLVM has a ``-mcpu`` selector for the BPF back end in order to allow +the selection of BPF instruction set extensions. By default the +``generic`` processor target is used, which is the base instruction set +(v1) of BPF. + +LLVM has an option to select ``-mcpu=probe`` where it will probe the host +kernel for supported BPF instruction set extensions and selects the +optimal set automatically. + +For cross-compilation, a specific version can be select manually as well :: + + $ llc -march bpf -mcpu=help + Available CPUs for this target: + + generic - Select the generic processor. + probe - Select the probe processor. + v1 - Select the v1 processor. + v2 - Select the v2 processor. + [...] + +Newly added BPF instructions to the Linux kernel need to follow the same +scheme, bump the instruction set version and implement probing for the +extensions such that ``-mcpu=probe`` users can benefit from the +optimization transparently when upgrading their kernels. + +If you are unable to implement support for the newly added BPF instruction +please reach out to BPF developers for help. + +By the way, the BPF kernel selftests run with ``-mcpu=probe`` for better +test coverage. + +Q: clang flag for target bpf? +----------------------------- +Q: In some cases clang flag ``-target bpf`` is used but in other cases the +default clang target, which matches the underlying architecture, is used. +What is the difference and when I should use which? + +A: Although LLVM IR generation and optimization try to stay architecture +independent, ``-target `` still has some impact on generated code: + +- BPF program may recursively include header file(s) with file scope + inline assembly codes. The default target can handle this well, + while ``bpf`` target may fail if bpf backend assembler does not + understand these assembly codes, which is true in most cases. + +- When compiled without ``-g``, additional elf sections, e.g., + .eh_frame and .rela.eh_frame, may be present in the object file + with default target, but not with ``bpf`` target. + +- The default target may turn a C switch statement into a switch table + lookup and jump operation. Since the switch table is placed + in the global readonly section, the bpf program will fail to load. + The bpf target does not support switch table optimization. + The clang option ``-fno-jump-tables`` can be used to disable + switch table generation. + +- For clang ``-target bpf``, it is guaranteed that pointer or long / + unsigned long types will always have a width of 64 bit, no matter + whether underlying clang binary or default target (or kernel) is + 32 bit. However, when native clang target is used, then it will + compile these types based on the underlying architecture's conventions, + meaning in case of 32 bit architecture, pointer or long / unsigned + long types e.g. in BPF context structure will have width of 32 bit + while the BPF LLVM back end still operates in 64 bit. The native + target is mostly needed in tracing for the case of walking ``pt_regs`` + or other kernel structures where CPU's register width matters. + Otherwise, ``clang -target bpf`` is generally recommended. + +You should use default target when: + +- Your program includes a header file, e.g., ptrace.h, which eventually + pulls in some header files containing file scope host assembly codes. + +- You can add ``-fno-jump-tables`` to work around the switch table issue. + +Otherwise, you can use ``bpf`` target. Additionally, you *must* use bpf target +when: + +- Your program uses data structures with pointer or long / unsigned long + types that interface with BPF helpers or context data structures. Access + into these structures is verified by the BPF verifier and may result + in verification failures if the native architecture is not aligned with + the BPF architecture, e.g. 64-bit. An example of this is + BPF_PROG_TYPE_SK_MSG require ``-target bpf`` + + +.. Links +.. _Documentation/process/: https://www.kernel.org/doc/html/latest/process/ +.. _MAINTAINERS: ../../MAINTAINERS +.. _Documentation/networking/netdev-FAQ.txt: ../networking/netdev-FAQ.txt +.. _netdev FAQ: ../networking/netdev-FAQ.txt +.. _samples/bpf/: ../../samples/bpf/ +.. _selftests: ../../tools/testing/selftests/bpf/ +.. _Documentation/dev-tools/kselftest.rst: + https://www.kernel.org/doc/html/latest/dev-tools/kselftest.html + +Happy BPF hacking! diff --git a/Documentation/bpf/bpf_devel_QA.txt b/Documentation/bpf/bpf_devel_QA.txt deleted file mode 100644 index da57601153a0..000000000000 --- a/Documentation/bpf/bpf_devel_QA.txt +++ /dev/null @@ -1,570 +0,0 @@ -This document provides information for the BPF subsystem about various -workflows related to reporting bugs, submitting patches, and queueing -patches for stable kernels. - -For general information about submitting patches, please refer to -Documentation/process/. This document only describes additional specifics -related to BPF. - -Reporting bugs: ---------------- - -Q: How do I report bugs for BPF kernel code? - -A: Since all BPF kernel development as well as bpftool and iproute2 BPF - loader development happens through the netdev kernel mailing list, - please report any found issues around BPF to the following mailing - list: - - netdev@vger.kernel.org - - This may also include issues related to XDP, BPF tracing, etc. - - Given netdev has a high volume of traffic, please also add the BPF - maintainers to Cc (from kernel MAINTAINERS file): - - Alexei Starovoitov - Daniel Borkmann - - In case a buggy commit has already been identified, make sure to keep - the actual commit authors in Cc as well for the report. They can - typically be identified through the kernel's git tree. - - Please do *not* report BPF issues to bugzilla.kernel.org since it - is a guarantee that the reported issue will be overlooked. - -Submitting patches: -------------------- - -Q: To which mailing list do I need to submit my BPF patches? - -A: Please submit your BPF patches to the netdev kernel mailing list: - - netdev@vger.kernel.org - - Historically, BPF came out of networking and has always been maintained - by the kernel networking community. Although these days BPF touches - many other subsystems as well, the patches are still routed mainly - through the networking community. - - In case your patch has changes in various different subsystems (e.g. - tracing, security, etc), make sure to Cc the related kernel mailing - lists and maintainers from there as well, so they are able to review - the changes and provide their Acked-by's to the patches. - -Q: Where can I find patches currently under discussion for BPF subsystem? - -A: All patches that are Cc'ed to netdev are queued for review under netdev - patchwork project: - - http://patchwork.ozlabs.org/project/netdev/list/ - - Those patches which target BPF, are assigned to a 'bpf' delegate for - further processing from BPF maintainers. The current queue with - patches under review can be found at: - - https://patchwork.ozlabs.org/project/netdev/list/?delegate=77147 - - Once the patches have been reviewed by the BPF community as a whole - and approved by the BPF maintainers, their status in patchwork will be - changed to 'Accepted' and the submitter will be notified by mail. This - means that the patches look good from a BPF perspective and have been - applied to one of the two BPF kernel trees. - - In case feedback from the community requires a respin of the patches, - their status in patchwork will be set to 'Changes Requested', and purged - from the current review queue. Likewise for cases where patches would - get rejected or are not applicable to the BPF trees (but assigned to - the 'bpf' delegate). - -Q: How do the changes make their way into Linux? - -A: There are two BPF kernel trees (git repositories). Once patches have - been accepted by the BPF maintainers, they will be applied to one - of the two BPF trees: - - https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/ - https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/ - - The bpf tree itself is for fixes only, whereas bpf-next for features, - cleanups or other kind of improvements ("next-like" content). This is - analogous to net and net-next trees for networking. Both bpf and - bpf-next will only have a master branch in order to simplify against - which branch patches should get rebased to. - - Accumulated BPF patches in the bpf tree will regularly get pulled - into the net kernel tree. Likewise, accumulated BPF patches accepted - into the bpf-next tree will make their way into net-next tree. net and - net-next are both run by David S. Miller. From there, they will go - into the kernel mainline tree run by Linus Torvalds. To read up on the - process of net and net-next being merged into the mainline tree, see - the netdev FAQ under: - - Documentation/networking/netdev-FAQ.txt - - Occasionally, to prevent merge conflicts, we might send pull requests - to other trees (e.g. tracing) with a small subset of the patches, but - net and net-next are always the main trees targeted for integration. - - The pull requests will contain a high-level summary of the accumulated - patches and can be searched on netdev kernel mailing list through the - following subject lines (yyyy-mm-dd is the date of the pull request): - - pull-request: bpf yyyy-mm-dd - pull-request: bpf-next yyyy-mm-dd - -Q: How do I indicate which tree (bpf vs. bpf-next) my patch should be - applied to? - -A: The process is the very same as described in the netdev FAQ, so - please read up on it. The subject line must indicate whether the - patch is a fix or rather "next-like" content in order to let the - maintainers know whether it is targeted at bpf or bpf-next. - - For fixes eventually landing in bpf -> net tree, the subject must - look like: - - git format-patch --subject-prefix='PATCH bpf' start..finish - - For features/improvements/etc that should eventually land in - bpf-next -> net-next, the subject must look like: - - git format-patch --subject-prefix='PATCH bpf-next' start..finish - - If unsure whether the patch or patch series should go into bpf - or net directly, or bpf-next or net-next directly, it is not a - problem either if the subject line says net or net-next as target. - It is eventually up to the maintainers to do the delegation of - the patches. - - If it is clear that patches should go into bpf or bpf-next tree, - please make sure to rebase the patches against those trees in - order to reduce potential conflicts. - - In case the patch or patch series has to be reworked and sent out - again in a second or later revision, it is also required to add a - version number (v2, v3, ...) into the subject prefix: - - git format-patch --subject-prefix='PATCH net-next v2' start..finish - - When changes have been requested to the patch series, always send the - whole patch series again with the feedback incorporated (never send - individual diffs on top of the old series). - -Q: What does it mean when a patch gets applied to bpf or bpf-next tree? - -A: It means that the patch looks good for mainline inclusion from - a BPF point of view. - - Be aware that this is not a final verdict that the patch will - automatically get accepted into net or net-next trees eventually: - - On the netdev kernel mailing list reviews can come in at any point - in time. If discussions around a patch conclude that they cannot - get included as-is, we will either apply a follow-up fix or drop - them from the trees entirely. Therefore, we also reserve to rebase - the trees when deemed necessary. After all, the purpose of the tree - is to i) accumulate and stage BPF patches for integration into trees - like net and net-next, and ii) run extensive BPF test suite and - workloads on the patches before they make their way any further. - - Once the BPF pull request was accepted by David S. Miller, then - the patches end up in net or net-next tree, respectively, and - make their way from there further into mainline. Again, see the - netdev FAQ for additional information e.g. on how often they are - merged to mainline. - -Q: How long do I need to wait for feedback on my BPF patches? - -A: We try to keep the latency low. The usual time to feedback will - be around 2 or 3 business days. It may vary depending on the - complexity of changes and current patch load. - -Q: How often do you send pull requests to major kernel trees like - net or net-next? - -A: Pull requests will be sent out rather often in order to not - accumulate too many patches in bpf or bpf-next. - - As a rule of thumb, expect pull requests for each tree regularly - at the end of the week. In some cases pull requests could additionally - come also in the middle of the week depending on the current patch - load or urgency. - -Q: Are patches applied to bpf-next when the merge window is open? - -A: For the time when the merge window is open, bpf-next will not be - processed. This is roughly analogous to net-next patch processing, - so feel free to read up on the netdev FAQ about further details. - - During those two weeks of merge window, we might ask you to resend - your patch series once bpf-next is open again. Once Linus released - a v*-rc1 after the merge window, we continue processing of bpf-next. - - For non-subscribers to kernel mailing lists, there is also a status - page run by David S. Miller on net-next that provides guidance: - - http://vger.kernel.org/~davem/net-next.html - -Q: I made a BPF verifier change, do I need to add test cases for - BPF kernel selftests? - -A: If the patch has changes to the behavior of the verifier, then yes, - it is absolutely necessary to add test cases to the BPF kernel - selftests suite. If they are not present and we think they are - needed, then we might ask for them before accepting any changes. - - In particular, test_verifier.c is tracking a high number of BPF test - cases, including a lot of corner cases that LLVM BPF back end may - generate out of the restricted C code. Thus, adding test cases is - absolutely crucial to make sure future changes do not accidentally - affect prior use-cases. Thus, treat those test cases as: verifier - behavior that is not tracked in test_verifier.c could potentially - be subject to change. - -Q: When should I add code to samples/bpf/ and when to BPF kernel - selftests? - -A: In general, we prefer additions to BPF kernel selftests rather than - samples/bpf/. The rationale is very simple: kernel selftests are - regularly run by various bots to test for kernel regressions. - - The more test cases we add to BPF selftests, the better the coverage - and the less likely it is that those could accidentally break. It is - not that BPF kernel selftests cannot demo how a specific feature can - be used. - - That said, samples/bpf/ may be a good place for people to get started, - so it might be advisable that simple demos of features could go into - samples/bpf/, but advanced functional and corner-case testing rather - into kernel selftests. - - If your sample looks like a test case, then go for BPF kernel selftests - instead! - -Q: When should I add code to the bpftool? - -A: The main purpose of bpftool (under tools/bpf/bpftool/) is to provide - a central user space tool for debugging and introspection of BPF programs - and maps that are active in the kernel. If UAPI changes related to BPF - enable for dumping additional information of programs or maps, then - bpftool should be extended as well to support dumping them. - -Q: When should I add code to iproute2's BPF loader? - -A: For UAPI changes related to the XDP or tc layer (e.g. cls_bpf), the - convention is that those control-path related changes are added to - iproute2's BPF loader as well from user space side. This is not only - useful to have UAPI changes properly designed to be usable, but also - to make those changes available to a wider user base of major - downstream distributions. - -Q: Do you accept patches as well for iproute2's BPF loader? - -A: Patches for the iproute2's BPF loader have to be sent to: - - netdev@vger.kernel.org - - While those patches are not processed by the BPF kernel maintainers, - please keep them in Cc as well, so they can be reviewed. - - The official git repository for iproute2 is run by Stephen Hemminger - and can be found at: - - https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/ - - The patches need to have a subject prefix of '[PATCH iproute2 master]' - or '[PATCH iproute2 net-next]'. 'master' or 'net-next' describes the - target branch where the patch should be applied to. Meaning, if kernel - changes went into the net-next kernel tree, then the related iproute2 - changes need to go into the iproute2 net-next branch, otherwise they - can be targeted at master branch. The iproute2 net-next branch will get - merged into the master branch after the current iproute2 version from - master has been released. - - Like BPF, the patches end up in patchwork under the netdev project and - are delegated to 'shemminger' for further processing: - - http://patchwork.ozlabs.org/project/netdev/list/?delegate=389 - -Q: What is the minimum requirement before I submit my BPF patches? - -A: When submitting patches, always take the time and properly test your - patches *prior* to submission. Never rush them! If maintainers find - that your patches have not been properly tested, it is a good way to - get them grumpy. Testing patch submissions is a hard requirement! - - Note, fixes that go to bpf tree *must* have a Fixes: tag included. The - same applies to fixes that target bpf-next, where the affected commit - is in net-next (or in some cases bpf-next). The Fixes: tag is crucial - in order to identify follow-up commits and tremendously helps for people - having to do backporting, so it is a must have! - - We also don't accept patches with an empty commit message. Take your - time and properly write up a high quality commit message, it is - essential! - - Think about it this way: other developers looking at your code a month - from now need to understand *why* a certain change has been done that - way, and whether there have been flaws in the analysis or assumptions - that the original author did. Thus providing a proper rationale and - describing the use-case for the changes is a must. - - Patch submissions with >1 patch must have a cover letter which includes - a high level description of the series. This high level summary will - then be placed into the merge commit by the BPF maintainers such that - it is also accessible from the git log for future reference. - -Q: What do I need to consider when adding a new instruction or feature - that would require BPF JIT and/or LLVM integration as well? - -A: We try hard to keep all BPF JITs up to date such that the same user - experience can be guaranteed when running BPF programs on different - architectures without having the program punt to the less efficient - interpreter in case the in-kernel BPF JIT is enabled. - - If you are unable to implement or test the required JIT changes for - certain architectures, please work together with the related BPF JIT - developers in order to get the feature implemented in a timely manner. - Please refer to the git log (arch/*/net/) to locate the necessary - people for helping out. - - Also always make sure to add BPF test cases (e.g. test_bpf.c and - test_verifier.c) for new instructions, so that they can receive - broad test coverage and help run-time testing the various BPF JITs. - - In case of new BPF instructions, once the changes have been accepted - into the Linux kernel, please implement support into LLVM's BPF back - end. See LLVM section below for further information. - -Stable submission: ------------------- - -Q: I need a specific BPF commit in stable kernels. What should I do? - -A: In case you need a specific fix in stable kernels, first check whether - the commit has already been applied in the related linux-*.y branches: - - https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/ - - If not the case, then drop an email to the BPF maintainers with the - netdev kernel mailing list in Cc and ask for the fix to be queued up: - - netdev@vger.kernel.org - - The process in general is the same as on netdev itself, see also the - netdev FAQ document. - -Q: Do you also backport to kernels not currently maintained as stable? - -A: No. If you need a specific BPF commit in kernels that are currently not - maintained by the stable maintainers, then you are on your own. - - The current stable and longterm stable kernels are all listed here: - - https://www.kernel.org/ - -Q: The BPF patch I am about to submit needs to go to stable as well. What - should I do? - -A: The same rules apply as with netdev patch submissions in general, see - netdev FAQ under: - - Documentation/networking/netdev-FAQ.txt - - Never add "Cc: stable@vger.kernel.org" to the patch description, but - ask the BPF maintainers to queue the patches instead. This can be done - with a note, for example, under the "---" part of the patch which does - not go into the git log. Alternatively, this can be done as a simple - request by mail instead. - -Q: Where do I find currently queued BPF patches that will be submitted - to stable? - -A: Once patches that fix critical bugs got applied into the bpf tree, they - are queued up for stable submission under: - - http://patchwork.ozlabs.org/bundle/bpf/stable/?state=* - - They will be on hold there at minimum until the related commit made its - way into the mainline kernel tree. - - After having been under broader exposure, the queued patches will be - submitted by the BPF maintainers to the stable maintainers. - -Testing patches: ----------------- - -Q: Which BPF kernel selftests version should I run my kernel against? - -A: If you run a kernel xyz, then always run the BPF kernel selftests from - that kernel xyz as well. Do not expect that the BPF selftest from the - latest mainline tree will pass all the time. - - In particular, test_bpf.c and test_verifier.c have a large number of - test cases and are constantly updated with new BPF test sequences, or - existing ones are adapted to verifier changes e.g. due to verifier - becoming smarter and being able to better track certain things. - -LLVM: ------ - -Q: Where do I find LLVM with BPF support? - -A: The BPF back end for LLVM is upstream in LLVM since version 3.7.1. - - All major distributions these days ship LLVM with BPF back end enabled, - so for the majority of use-cases it is not required to compile LLVM by - hand anymore, just install the distribution provided package. - - LLVM's static compiler lists the supported targets through 'llc --version', - make sure BPF targets are listed. Example: - - $ llc --version - LLVM (http://llvm.org/): - LLVM version 6.0.0svn - Optimized build. - Default target: x86_64-unknown-linux-gnu - Host CPU: skylake - - Registered Targets: - bpf - BPF (host endian) - bpfeb - BPF (big endian) - bpfel - BPF (little endian) - x86 - 32-bit X86: Pentium-Pro and above - x86-64 - 64-bit X86: EM64T and AMD64 - - For developers in order to utilize the latest features added to LLVM's - BPF back end, it is advisable to run the latest LLVM releases. Support - for new BPF kernel features such as additions to the BPF instruction - set are often developed together. - - All LLVM releases can be found at: http://releases.llvm.org/ - -Q: Got it, so how do I build LLVM manually anyway? - -A: You need cmake and gcc-c++ as build requisites for LLVM. Once you have - that set up, proceed with building the latest LLVM and clang version - from the git repositories: - - $ git clone http://llvm.org/git/llvm.git - $ cd llvm/tools - $ git clone --depth 1 http://llvm.org/git/clang.git - $ cd ..; mkdir build; cd build - $ cmake .. -DLLVM_TARGETS_TO_BUILD="BPF;X86" \ - -DBUILD_SHARED_LIBS=OFF \ - -DCMAKE_BUILD_TYPE=Release \ - -DLLVM_BUILD_RUNTIME=OFF - $ make -j $(getconf _NPROCESSORS_ONLN) - - The built binaries can then be found in the build/bin/ directory, where - you can point the PATH variable to. - -Q: Should I notify BPF kernel maintainers about issues in LLVM's BPF code - generation back end or about LLVM generated code that the verifier - refuses to accept? - -A: Yes, please do! LLVM's BPF back end is a key piece of the whole BPF - infrastructure and it ties deeply into verification of programs from the - kernel side. Therefore, any issues on either side need to be investigated - and fixed whenever necessary. - - Therefore, please make sure to bring them up at netdev kernel mailing - list and Cc BPF maintainers for LLVM and kernel bits: - - Yonghong Song - Alexei Starovoitov - Daniel Borkmann - - LLVM also has an issue tracker where BPF related bugs can be found: - - https://bugs.llvm.org/buglist.cgi?quicksearch=bpf - - However, it is better to reach out through mailing lists with having - maintainers in Cc. - -Q: I have added a new BPF instruction to the kernel, how can I integrate - it into LLVM? - -A: LLVM has a -mcpu selector for the BPF back end in order to allow the - selection of BPF instruction set extensions. By default the 'generic' - processor target is used, which is the base instruction set (v1) of BPF. - - LLVM has an option to select -mcpu=probe where it will probe the host - kernel for supported BPF instruction set extensions and selects the - optimal set automatically. - - For cross-compilation, a specific version can be select manually as well. - - $ llc -march bpf -mcpu=help - Available CPUs for this target: - - generic - Select the generic processor. - probe - Select the probe processor. - v1 - Select the v1 processor. - v2 - Select the v2 processor. - [...] - - Newly added BPF instructions to the Linux kernel need to follow the same - scheme, bump the instruction set version and implement probing for the - extensions such that -mcpu=probe users can benefit from the optimization - transparently when upgrading their kernels. - - If you are unable to implement support for the newly added BPF instruction - please reach out to BPF developers for help. - - By the way, the BPF kernel selftests run with -mcpu=probe for better - test coverage. - -Q: In some cases clang flag "-target bpf" is used but in other cases the - default clang target, which matches the underlying architecture, is used. - What is the difference and when I should use which? - -A: Although LLVM IR generation and optimization try to stay architecture - independent, "-target " still has some impact on generated code: - - - BPF program may recursively include header file(s) with file scope - inline assembly codes. The default target can handle this well, - while bpf target may fail if bpf backend assembler does not - understand these assembly codes, which is true in most cases. - - - When compiled without -g, additional elf sections, e.g., - .eh_frame and .rela.eh_frame, may be present in the object file - with default target, but not with bpf target. - - - The default target may turn a C switch statement into a switch table - lookup and jump operation. Since the switch table is placed - in the global readonly section, the bpf program will fail to load. - The bpf target does not support switch table optimization. - The clang option "-fno-jump-tables" can be used to disable - switch table generation. - - - For clang -target bpf, it is guaranteed that pointer or long / - unsigned long types will always have a width of 64 bit, no matter - whether underlying clang binary or default target (or kernel) is - 32 bit. However, when native clang target is used, then it will - compile these types based on the underlying architecture's conventions, - meaning in case of 32 bit architecture, pointer or long / unsigned - long types e.g. in BPF context structure will have width of 32 bit - while the BPF LLVM back end still operates in 64 bit. The native - target is mostly needed in tracing for the case of walking pt_regs - or other kernel structures where CPU's register width matters. - Otherwise, clang -target bpf is generally recommended. - - You should use default target when: - - - Your program includes a header file, e.g., ptrace.h, which eventually - pulls in some header files containing file scope host assembly codes. - - You can add "-fno-jump-tables" to work around the switch table issue. - - Otherwise, you can use bpf target. Additionally, you _must_ use bpf target - when: - - - Your program uses data structures with pointer or long / unsigned long - types that interface with BPF helpers or context data structures. Access - into these structures is verified by the BPF verifier and may result - in verification failures if the native architecture is not aligned with - the BPF architecture, e.g. 64-bit. An example of this is - BPF_PROG_TYPE_SK_MSG require '-target bpf' - -Happy BPF hacking! diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt deleted file mode 100644 index 74cdeaed9f7a..000000000000 --- a/Documentation/cgroup-v2.txt +++ /dev/null @@ -1,1998 +0,0 @@ -================ -Control Group v2 -================ - -:Date: October, 2015 -:Author: Tejun Heo - -This is the authoritative documentation on the design, interface and -conventions of cgroup v2. It describes all userland-visible aspects -of cgroup including core and specific controller behaviors. All -future changes must be reflected in this document. Documentation for -v1 is available under Documentation/cgroup-v1/. - -.. CONTENTS - - 1. Introduction - 1-1. Terminology - 1-2. What is cgroup? - 2. Basic Operations - 2-1. Mounting - 2-2. Organizing Processes and Threads - 2-2-1. Processes - 2-2-2. Threads - 2-3. [Un]populated Notification - 2-4. Controlling Controllers - 2-4-1. Enabling and Disabling - 2-4-2. Top-down Constraint - 2-4-3. No Internal Process Constraint - 2-5. Delegation - 2-5-1. Model of Delegation - 2-5-2. Delegation Containment - 2-6. Guidelines - 2-6-1. Organize Once and Control - 2-6-2. Avoid Name Collisions - 3. Resource Distribution Models - 3-1. Weights - 3-2. Limits - 3-3. Protections - 3-4. Allocations - 4. Interface Files - 4-1. Format - 4-2. Conventions - 4-3. Core Interface Files - 5. Controllers - 5-1. CPU - 5-1-1. CPU Interface Files - 5-2. Memory - 5-2-1. Memory Interface Files - 5-2-2. Usage Guidelines - 5-2-3. Memory Ownership - 5-3. IO - 5-3-1. IO Interface Files - 5-3-2. Writeback - 5-4. PID - 5-4-1. PID Interface Files - 5-5. Device - 5-6. RDMA - 5-6-1. RDMA Interface Files - 5-7. Misc - 5-7-1. perf_event - 5-N. Non-normative information - 5-N-1. CPU controller root cgroup process behaviour - 5-N-2. IO controller root cgroup process behaviour - 6. Namespace - 6-1. Basics - 6-2. The Root and Views - 6-3. Migration and setns(2) - 6-4. Interaction with Other Namespaces - P. Information on Kernel Programming - P-1. Filesystem Support for Writeback - D. Deprecated v1 Core Features - R. Issues with v1 and Rationales for v2 - R-1. Multiple Hierarchies - R-2. Thread Granularity - R-3. Competition Between Inner Nodes and Threads - R-4. Other Interface Issues - R-5. Controller Issues and Remedies - R-5-1. Memory - - -Introduction -============ - -Terminology ------------ - -"cgroup" stands for "control group" and is never capitalized. The -singular form is used to designate the whole feature and also as a -qualifier as in "cgroup controllers". When explicitly referring to -multiple individual control groups, the plural form "cgroups" is used. - - -What is cgroup? ---------------- - -cgroup is a mechanism to organize processes hierarchically and -distribute system resources along the hierarchy in a controlled and -configurable manner. - -cgroup is largely composed of two parts - the core and controllers. -cgroup core is primarily responsible for hierarchically organizing -processes. A cgroup controller is usually responsible for -distributing a specific type of system resource along the hierarchy -although there are utility controllers which serve purposes other than -resource distribution. - -cgroups form a tree structure and every process in the system belongs -to one and only one cgroup. All threads of a process belong to the -same cgroup. On creation, all processes are put in the cgroup that -the parent process belongs to at the time. A process can be migrated -to another cgroup. Migration of a process doesn't affect already -existing descendant processes. - -Following certain structural constraints, controllers may be enabled or -disabled selectively on a cgroup. All controller behaviors are -hierarchical - if a controller is enabled on a cgroup, it affects all -processes which belong to the cgroups consisting the inclusive -sub-hierarchy of the cgroup. When a controller is enabled on a nested -cgroup, it always restricts the resource distribution further. The -restrictions set closer to the root in the hierarchy can not be -overridden from further away. - - -Basic Operations -================ - -Mounting --------- - -Unlike v1, cgroup v2 has only single hierarchy. The cgroup v2 -hierarchy can be mounted with the following mount command:: - - # mount -t cgroup2 none $MOUNT_POINT - -cgroup2 filesystem has the magic number 0x63677270 ("cgrp"). All -controllers which support v2 and are not bound to a v1 hierarchy are -automatically bound to the v2 hierarchy and show up at the root. -Controllers which are not in active use in the v2 hierarchy can be -bound to other hierarchies. This allows mixing v2 hierarchy with the -legacy v1 multiple hierarchies in a fully backward compatible way. - -A controller can be moved across hierarchies only after the controller -is no longer referenced in its current hierarchy. Because per-cgroup -controller states are destroyed asynchronously and controllers may -have lingering references, a controller may not show up immediately on -the v2 hierarchy after the final umount of the previous hierarchy. -Similarly, a controller should be fully disabled to be moved out of -the unified hierarchy and it may take some time for the disabled -controller to become available for other hierarchies; furthermore, due -to inter-controller dependencies, other controllers may need to be -disabled too. - -While useful for development and manual configurations, moving -controllers dynamically between the v2 and other hierarchies is -strongly discouraged for production use. It is recommended to decide -the hierarchies and controller associations before starting using the -controllers after system boot. - -During transition to v2, system management software might still -automount the v1 cgroup filesystem and so hijack all controllers -during boot, before manual intervention is possible. To make testing -and experimenting easier, the kernel parameter cgroup_no_v1= allows -disabling controllers in v1 and make them always available in v2. - -cgroup v2 currently supports the following mount options. - - nsdelegate - - Consider cgroup namespaces as delegation boundaries. This - option is system wide and can only be set on mount or modified - through remount from the init namespace. The mount option is - ignored on non-init namespace mounts. Please refer to the - Delegation section for details. - - -Organizing Processes and Threads --------------------------------- - -Processes -~~~~~~~~~ - -Initially, only the root cgroup exists to which all processes belong. -A child cgroup can be created by creating a sub-directory:: - - # mkdir $CGROUP_NAME - -A given cgroup may have multiple child cgroups forming a tree -structure. Each cgroup has a read-writable interface file -"cgroup.procs". When read, it lists the PIDs of all processes which -belong to the cgroup one-per-line. The PIDs are not ordered and the -same PID may show up more than once if the process got moved to -another cgroup and then back or the PID got recycled while reading. - -A process can be migrated into a cgroup by writing its PID to the -target cgroup's "cgroup.procs" file. Only one process can be migrated -on a single write(2) call. If a process is composed of multiple -threads, writing the PID of any thread migrates all threads of the -process. - -When a process forks a child process, the new process is born into the -cgroup that the forking process belongs to at the time of the -operation. After exit, a process stays associated with the cgroup -that it belonged to at the time of exit until it's reaped; however, a -zombie process does not appear in "cgroup.procs" and thus can't be -moved to another cgroup. - -A cgroup which doesn't have any children or live processes can be -destroyed by removing the directory. Note that a cgroup which doesn't -have any children and is associated only with zombie processes is -considered empty and can be removed:: - - # rmdir $CGROUP_NAME - -"/proc/$PID/cgroup" lists a process's cgroup membership. If legacy -cgroup is in use in the system, this file may contain multiple lines, -one for each hierarchy. The entry for cgroup v2 is always in the -format "0::$PATH":: - - # cat /proc/842/cgroup - ... - 0::/test-cgroup/test-cgroup-nested - -If the process becomes a zombie and the cgroup it was associated with -is removed subsequently, " (deleted)" is appended to the path:: - - # cat /proc/842/cgroup - ... - 0::/test-cgroup/test-cgroup-nested (deleted) - - -Threads -~~~~~~~ - -cgroup v2 supports thread granularity for a subset of controllers to -support use cases requiring hierarchical resource distribution across -the threads of a group of processes. By default, all threads of a -process belong to the same cgroup, which also serves as the resource -domain to host resource consumptions which are not specific to a -process or thread. The thread mode allows threads to be spread across -a subtree while still maintaining the common resource domain for them. - -Controllers which support thread mode are called threaded controllers. -The ones which don't are called domain controllers. - -Marking a cgroup threaded makes it join the resource domain of its -parent as a threaded cgroup. The parent may be another threaded -cgroup whose resource domain is further up in the hierarchy. The root -of a threaded subtree, that is, the nearest ancestor which is not -threaded, is called threaded domain or thread root interchangeably and -serves as the resource domain for the entire subtree. - -Inside a threaded subtree, threads of a process can be put in -different cgroups and are not subject to the no internal process -constraint - threaded controllers can be enabled on non-leaf cgroups -whether they have threads in them or not. - -As the threaded domain cgroup hosts all the domain resource -consumptions of the subtree, it is considered to have internal -resource consumptions whether there are processes in it or not and -can't have populated child cgroups which aren't threaded. Because the -root cgroup is not subject to no internal process constraint, it can -serve both as a threaded domain and a parent to domain cgroups. - -The current operation mode or type of the cgroup is shown in the -"cgroup.type" file which indicates whether the cgroup is a normal -domain, a domain which is serving as the domain of a threaded subtree, -or a threaded cgroup. - -On creation, a cgroup is always a domain cgroup and can be made -threaded by writing "threaded" to the "cgroup.type" file. The -operation is single direction:: - - # echo threaded > cgroup.type - -Once threaded, the cgroup can't be made a domain again. To enable the -thread mode, the following conditions must be met. - -- As the cgroup will join the parent's resource domain. The parent - must either be a valid (threaded) domain or a threaded cgroup. - -- When the parent is an unthreaded domain, it must not have any domain - controllers enabled or populated domain children. The root is - exempt from this requirement. - -Topology-wise, a cgroup can be in an invalid state. Please consider -the following topology:: - - A (threaded domain) - B (threaded) - C (domain, just created) - -C is created as a domain but isn't connected to a parent which can -host child domains. C can't be used until it is turned into a -threaded cgroup. "cgroup.type" file will report "domain (invalid)" in -these cases. Operations which fail due to invalid topology use -EOPNOTSUPP as the errno. - -A domain cgroup is turned into a threaded domain when one of its child -cgroup becomes threaded or threaded controllers are enabled in the -"cgroup.subtree_control" file while there are processes in the cgroup. -A threaded domain reverts to a normal domain when the conditions -clear. - -When read, "cgroup.threads" contains the list of the thread IDs of all -threads in the cgroup. Except that the operations are per-thread -instead of per-process, "cgroup.threads" has the same format and -behaves the same way as "cgroup.procs". While "cgroup.threads" can be -written to in any cgroup, as it can only move threads inside the same -threaded domain, its operations are confined inside each threaded -subtree. - -The threaded domain cgroup serves as the resource domain for the whole -subtree, and, while the threads can be scattered across the subtree, -all the processes are considered to be in the threaded domain cgroup. -"cgroup.procs" in a threaded domain cgroup contains the PIDs of all -processes in the subtree and is not readable in the subtree proper. -However, "cgroup.procs" can be written to from anywhere in the subtree -to migrate all threads of the matching process to the cgroup. - -Only threaded controllers can be enabled in a threaded subtree. When -a threaded controller is enabled inside a threaded subtree, it only -accounts for and controls resource consumptions associated with the -threads in the cgroup and its descendants. All consumptions which -aren't tied to a specific thread belong to the threaded domain cgroup. - -Because a threaded subtree is exempt from no internal process -constraint, a threaded controller must be able to handle competition -between threads in a non-leaf cgroup and its child cgroups. Each -threaded controller defines how such competitions are handled. - - -[Un]populated Notification --------------------------- - -Each non-root cgroup has a "cgroup.events" file which contains -"populated" field indicating whether the cgroup's sub-hierarchy has -live processes in it. Its value is 0 if there is no live process in -the cgroup and its descendants; otherwise, 1. poll and [id]notify -events are triggered when the value changes. This can be used, for -example, to start a clean-up operation after all processes of a given -sub-hierarchy have exited. The populated state updates and -notifications are recursive. Consider the following sub-hierarchy -where the numbers in the parentheses represent the numbers of processes -in each cgroup:: - - A(4) - B(0) - C(1) - \ D(0) - -A, B and C's "populated" fields would be 1 while D's 0. After the one -process in C exits, B and C's "populated" fields would flip to "0" and -file modified events will be generated on the "cgroup.events" files of -both cgroups. - - -Controlling Controllers ------------------------ - -Enabling and Disabling -~~~~~~~~~~~~~~~~~~~~~~ - -Each cgroup has a "cgroup.controllers" file which lists all -controllers available for the cgroup to enable:: - - # cat cgroup.controllers - cpu io memory - -No controller is enabled by default. Controllers can be enabled and -disabled by writing to the "cgroup.subtree_control" file:: - - # echo "+cpu +memory -io" > cgroup.subtree_control - -Only controllers which are listed in "cgroup.controllers" can be -enabled. When multiple operations are specified as above, either they -all succeed or fail. If multiple operations on the same controller -are specified, the last one is effective. - -Enabling a controller in a cgroup indicates that the distribution of -the target resource across its immediate children will be controlled. -Consider the following sub-hierarchy. The enabled controllers are -listed in parentheses:: - - A(cpu,memory) - B(memory) - C() - \ D() - -As A has "cpu" and "memory" enabled, A will control the distribution -of CPU cycles and memory to its children, in this case, B. As B has -"memory" enabled but not "CPU", C and D will compete freely on CPU -cycles but their division of memory available to B will be controlled. - -As a controller regulates the distribution of the target resource to -the cgroup's children, enabling it creates the controller's interface -files in the child cgroups. In the above example, enabling "cpu" on B -would create the "cpu." prefixed controller interface files in C and -D. Likewise, disabling "memory" from B would remove the "memory." -prefixed controller interface files from C and D. This means that the -controller interface files - anything which doesn't start with -"cgroup." are owned by the parent rather than the cgroup itself. - - -Top-down Constraint -~~~~~~~~~~~~~~~~~~~ - -Resources are distributed top-down and a cgroup can further distribute -a resource only if the resource has been distributed to it from the -parent. This means that all non-root "cgroup.subtree_control" files -can only contain controllers which are enabled in the parent's -"cgroup.subtree_control" file. A controller can be enabled only if -the parent has the controller enabled and a controller can't be -disabled if one or more children have it enabled. - - -No Internal Process Constraint -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Non-root cgroups can distribute domain resources to their children -only when they don't have any processes of their own. In other words, -only domain cgroups which don't contain any processes can have domain -controllers enabled in their "cgroup.subtree_control" files. - -This guarantees that, when a domain controller is looking at the part -of the hierarchy which has it enabled, processes are always only on -the leaves. This rules out situations where child cgroups compete -against internal processes of the parent. - -The root cgroup is exempt from this restriction. Root contains -processes and anonymous resource consumption which can't be associated -with any other cgroups and requires special treatment from most -controllers. How resource consumption in the root cgroup is governed -is up to each controller (for more information on this topic please -refer to the Non-normative information section in the Controllers -chapter). - -Note that the restriction doesn't get in the way if there is no -enabled controller in the cgroup's "cgroup.subtree_control". This is -important as otherwise it wouldn't be possible to create children of a -populated cgroup. To control resource distribution of a cgroup, the -cgroup must create children and transfer all its processes to the -children before enabling controllers in its "cgroup.subtree_control" -file. - - -Delegation ----------- - -Model of Delegation -~~~~~~~~~~~~~~~~~~~ - -A cgroup can be delegated in two ways. First, to a less privileged -user by granting write access of the directory and its "cgroup.procs", -"cgroup.threads" and "cgroup.subtree_control" files to the user. -Second, if the "nsdelegate" mount option is set, automatically to a -cgroup namespace on namespace creation. - -Because the resource control interface files in a given directory -control the distribution of the parent's resources, the delegatee -shouldn't be allowed to write to them. For the first method, this is -achieved by not granting access to these files. For the second, the -kernel rejects writes to all files other than "cgroup.procs" and -"cgroup.subtree_control" on a namespace root from inside the -namespace. - -The end results are equivalent for both delegation types. Once -delegated, the user can build sub-hierarchy under the directory, -organize processes inside it as it sees fit and further distribute the -resources it received from the parent. The limits and other settings -of all resource controllers are hierarchical and regardless of what -happens in the delegated sub-hierarchy, nothing can escape the -resource restrictions imposed by the parent. - -Currently, cgroup doesn't impose any restrictions on the number of -cgroups in or nesting depth of a delegated sub-hierarchy; however, -this may be limited explicitly in the future. - - -Delegation Containment -~~~~~~~~~~~~~~~~~~~~~~ - -A delegated sub-hierarchy is contained in the sense that processes -can't be moved into or out of the sub-hierarchy by the delegatee. - -For delegations to a less privileged user, this is achieved by -requiring the following conditions for a process with a non-root euid -to migrate a target process into a cgroup by writing its PID to the -"cgroup.procs" file. - -- The writer must have write access to the "cgroup.procs" file. - -- The writer must have write access to the "cgroup.procs" file of the - common ancestor of the source and destination cgroups. - -The above two constraints ensure that while a delegatee may migrate -processes around freely in the delegated sub-hierarchy it can't pull -in from or push out to outside the sub-hierarchy. - -For an example, let's assume cgroups C0 and C1 have been delegated to -user U0 who created C00, C01 under C0 and C10 under C1 as follows and -all processes under C0 and C1 belong to U0:: - - ~~~~~~~~~~~~~ - C0 - C00 - ~ cgroup ~ \ C01 - ~ hierarchy ~ - ~~~~~~~~~~~~~ - C1 - C10 - -Let's also say U0 wants to write the PID of a process which is -currently in C10 into "C00/cgroup.procs". U0 has write access to the -file; however, the common ancestor of the source cgroup C10 and the -destination cgroup C00 is above the points of delegation and U0 would -not have write access to its "cgroup.procs" files and thus the write -will be denied with -EACCES. - -For delegations to namespaces, containment is achieved by requiring -that both the source and destination cgroups are reachable from the -namespace of the process which is attempting the migration. If either -is not reachable, the migration is rejected with -ENOENT. - - -Guidelines ----------- - -Organize Once and Control -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Migrating a process across cgroups is a relatively expensive operation -and stateful resources such as memory are not moved together with the -process. This is an explicit design decision as there often exist -inherent trade-offs between migration and various hot paths in terms -of synchronization cost. - -As such, migrating processes across cgroups frequently as a means to -apply different resource restrictions is discouraged. A workload -should be assigned to a cgroup according to the system's logical and -resource structure once on start-up. Dynamic adjustments to resource -distribution can be made by changing controller configuration through -the interface files. - - -Avoid Name Collisions -~~~~~~~~~~~~~~~~~~~~~ - -Interface files for a cgroup and its children cgroups occupy the same -directory and it is possible to create children cgroups which collide -with interface files. - -All cgroup core interface files are prefixed with "cgroup." and each -controller's interface files are prefixed with the controller name and -a dot. A controller's name is composed of lower case alphabets and -'_'s but never begins with an '_' so it can be used as the prefix -character for collision avoidance. Also, interface file names won't -start or end with terms which are often used in categorizing workloads -such as job, service, slice, unit or workload. - -cgroup doesn't do anything to prevent name collisions and it's the -user's responsibility to avoid them. - - -Resource Distribution Models -============================ - -cgroup controllers implement several resource distribution schemes -depending on the resource type and expected use cases. This section -describes major schemes in use along with their expected behaviors. - - -Weights -------- - -A parent's resource is distributed by adding up the weights of all -active children and giving each the fraction matching the ratio of its -weight against the sum. As only children which can make use of the -resource at the moment participate in the distribution, this is -work-conserving. Due to the dynamic nature, this model is usually -used for stateless resources. - -All weights are in the range [1, 10000] with the default at 100. This -allows symmetric multiplicative biases in both directions at fine -enough granularity while staying in the intuitive range. - -As long as the weight is in range, all configuration combinations are -valid and there is no reason to reject configuration changes or -process migrations. - -"cpu.weight" proportionally distributes CPU cycles to active children -and is an example of this type. - - -Limits ------- - -A child can only consume upto the configured amount of the resource. -Limits can be over-committed - the sum of the limits of children can -exceed the amount of resource available to the parent. - -Limits are in the range [0, max] and defaults to "max", which is noop. - -As limits can be over-committed, all configuration combinations are -valid and there is no reason to reject configuration changes or -process migrations. - -"io.max" limits the maximum BPS and/or IOPS that a cgroup can consume -on an IO device and is an example of this type. - - -Protections ------------ - -A cgroup is protected to be allocated upto the configured amount of -the resource if the usages of all its ancestors are under their -protected levels. Protections can be hard guarantees or best effort -soft boundaries. Protections can also be over-committed in which case -only upto the amount available to the parent is protected among -children. - -Protections are in the range [0, max] and defaults to 0, which is -noop. - -As protections can be over-committed, all configuration combinations -are valid and there is no reason to reject configuration changes or -process migrations. - -"memory.low" implements best-effort memory protection and is an -example of this type. - - -Allocations ------------ - -A cgroup is exclusively allocated a certain amount of a finite -resource. Allocations can't be over-committed - the sum of the -allocations of children can not exceed the amount of resource -available to the parent. - -Allocations are in the range [0, max] and defaults to 0, which is no -resource. - -As allocations can't be over-committed, some configuration -combinations are invalid and should be rejected. Also, if the -resource is mandatory for execution of processes, process migrations -may be rejected. - -"cpu.rt.max" hard-allocates realtime slices and is an example of this -type. - - -Interface Files -=============== - -Format ------- - -All interface files should be in one of the following formats whenever -possible:: - - New-line separated values - (when only one value can be written at once) - - VAL0\n - VAL1\n - ... - - Space separated values - (when read-only or multiple values can be written at once) - - VAL0 VAL1 ...\n - - Flat keyed - - KEY0 VAL0\n - KEY1 VAL1\n - ... - - Nested keyed - - KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01... - KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11... - ... - -For a writable file, the format for writing should generally match -reading; however, controllers may allow omitting later fields or -implement restricted shortcuts for most common use cases. - -For both flat and nested keyed files, only the values for a single key -can be written at a time. For nested keyed files, the sub key pairs -may be specified in any order and not all pairs have to be specified. - - -Conventions ------------ - -- Settings for a single feature should be contained in a single file. - -- The root cgroup should be exempt from resource control and thus - shouldn't have resource control interface files. Also, - informational files on the root cgroup which end up showing global - information available elsewhere shouldn't exist. - -- If a controller implements weight based resource distribution, its - interface file should be named "weight" and have the range [1, - 10000] with 100 as the default. The values are chosen to allow - enough and symmetric bias in both directions while keeping it - intuitive (the default is 100%). - -- If a controller implements an absolute resource guarantee and/or - limit, the interface files should be named "min" and "max" - respectively. If a controller implements best effort resource - guarantee and/or limit, the interface files should be named "low" - and "high" respectively. - - In the above four control files, the special token "max" should be - used to represent upward infinity for both reading and writing. - -- If a setting has a configurable default value and keyed specific - overrides, the default entry should be keyed with "default" and - appear as the first entry in the file. - - The default value can be updated by writing either "default $VAL" or - "$VAL". - - When writing to update a specific override, "default" can be used as - the value to indicate removal of the override. Override entries - with "default" as the value must not appear when read. - - For example, a setting which is keyed by major:minor device numbers - with integer values may look like the following:: - - # cat cgroup-example-interface-file - default 150 - 8:0 300 - - The default value can be updated by:: - - # echo 125 > cgroup-example-interface-file - - or:: - - # echo "default 125" > cgroup-example-interface-file - - An override can be set by:: - - # echo "8:16 170" > cgroup-example-interface-file - - and cleared by:: - - # echo "8:0 default" > cgroup-example-interface-file - # cat cgroup-example-interface-file - default 125 - 8:16 170 - -- For events which are not very high frequency, an interface file - "events" should be created which lists event key value pairs. - Whenever a notifiable event happens, file modified event should be - generated on the file. - - -Core Interface Files --------------------- - -All cgroup core files are prefixed with "cgroup." - - cgroup.type - - A read-write single value file which exists on non-root - cgroups. - - When read, it indicates the current type of the cgroup, which - can be one of the following values. - - - "domain" : A normal valid domain cgroup. - - - "domain threaded" : A threaded domain cgroup which is - serving as the root of a threaded subtree. - - - "domain invalid" : A cgroup which is in an invalid state. - It can't be populated or have controllers enabled. It may - be allowed to become a threaded cgroup. - - - "threaded" : A threaded cgroup which is a member of a - threaded subtree. - - A cgroup can be turned into a threaded cgroup by writing - "threaded" to this file. - - cgroup.procs - A read-write new-line separated values file which exists on - all cgroups. - - When read, it lists the PIDs of all processes which belong to - the cgroup one-per-line. The PIDs are not ordered and the - same PID may show up more than once if the process got moved - to another cgroup and then back or the PID got recycled while - reading. - - A PID can be written to migrate the process associated with - the PID to the cgroup. The writer should match all of the - following conditions. - - - It must have write access to the "cgroup.procs" file. - - - It must have write access to the "cgroup.procs" file of the - common ancestor of the source and destination cgroups. - - When delegating a sub-hierarchy, write access to this file - should be granted along with the containing directory. - - In a threaded cgroup, reading this file fails with EOPNOTSUPP - as all the processes belong to the thread root. Writing is - supported and moves every thread of the process to the cgroup. - - cgroup.threads - A read-write new-line separated values file which exists on - all cgroups. - - When read, it lists the TIDs of all threads which belong to - the cgroup one-per-line. The TIDs are not ordered and the - same TID may show up more than once if the thread got moved to - another cgroup and then back or the TID got recycled while - reading. - - A TID can be written to migrate the thread associated with the - TID to the cgroup. The writer should match all of the - following conditions. - - - It must have write access to the "cgroup.threads" file. - - - The cgroup that the thread is currently in must be in the - same resource domain as the destination cgroup. - - - It must have write access to the "cgroup.procs" file of the - common ancestor of the source and destination cgroups. - - When delegating a sub-hierarchy, write access to this file - should be granted along with the containing directory. - - cgroup.controllers - A read-only space separated values file which exists on all - cgroups. - - It shows space separated list of all controllers available to - the cgroup. The controllers are not ordered. - - cgroup.subtree_control - A read-write space separated values file which exists on all - cgroups. Starts out empty. - - When read, it shows space separated list of the controllers - which are enabled to control resource distribution from the - cgroup to its children. - - Space separated list of controllers prefixed with '+' or '-' - can be written to enable or disable controllers. A controller - name prefixed with '+' enables the controller and '-' - disables. If a controller appears more than once on the list, - the last one is effective. When multiple enable and disable - operations are specified, either all succeed or all fail. - - cgroup.events - A read-only flat-keyed file which exists on non-root cgroups. - The following entries are defined. Unless specified - otherwise, a value change in this file generates a file - modified event. - - populated - 1 if the cgroup or its descendants contains any live - processes; otherwise, 0. - - cgroup.max.descendants - A read-write single value files. The default is "max". - - Maximum allowed number of descent cgroups. - If the actual number of descendants is equal or larger, - an attempt to create a new cgroup in the hierarchy will fail. - - cgroup.max.depth - A read-write single value files. The default is "max". - - Maximum allowed descent depth below the current cgroup. - If the actual descent depth is equal or larger, - an attempt to create a new child cgroup will fail. - - cgroup.stat - A read-only flat-keyed file with the following entries: - - nr_descendants - Total number of visible descendant cgroups. - - nr_dying_descendants - Total number of dying descendant cgroups. A cgroup becomes - dying after being deleted by a user. The cgroup will remain - in dying state for some time undefined time (which can depend - on system load) before being completely destroyed. - - A process can't enter a dying cgroup under any circumstances, - a dying cgroup can't revive. - - A dying cgroup can consume system resources not exceeding - limits, which were active at the moment of cgroup deletion. - - -Controllers -=========== - -CPU ---- - -The "cpu" controllers regulates distribution of CPU cycles. This -controller implements weight and absolute bandwidth limit models for -normal scheduling policy and absolute bandwidth allocation model for -realtime scheduling policy. - -WARNING: cgroup2 doesn't yet support control of realtime processes and -the cpu controller can only be enabled when all RT processes are in -the root cgroup. Be aware that system management software may already -have placed RT processes into nonroot cgroups during the system boot -process, and these processes may need to be moved to the root cgroup -before the cpu controller can be enabled. - - -CPU Interface Files -~~~~~~~~~~~~~~~~~~~ - -All time durations are in microseconds. - - cpu.stat - A read-only flat-keyed file which exists on non-root cgroups. - This file exists whether the controller is enabled or not. - - It always reports the following three stats: - - - usage_usec - - user_usec - - system_usec - - and the following three when the controller is enabled: - - - nr_periods - - nr_throttled - - throttled_usec - - cpu.weight - A read-write single value file which exists on non-root - cgroups. The default is "100". - - The weight in the range [1, 10000]. - - cpu.weight.nice - A read-write single value file which exists on non-root - cgroups. The default is "0". - - The nice value is in the range [-20, 19]. - - This interface file is an alternative interface for - "cpu.weight" and allows reading and setting weight using the - same values used by nice(2). Because the range is smaller and - granularity is coarser for the nice values, the read value is - the closest approximation of the current weight. - - cpu.max - A read-write two value file which exists on non-root cgroups. - The default is "max 100000". - - The maximum bandwidth limit. It's in the following format:: - - $MAX $PERIOD - - which indicates that the group may consume upto $MAX in each - $PERIOD duration. "max" for $MAX indicates no limit. If only - one number is written, $MAX is updated. - - -Memory ------- - -The "memory" controller regulates distribution of memory. Memory is -stateful and implements both limit and protection models. Due to the -intertwining between memory usage and reclaim pressure and the -stateful nature of memory, the distribution model is relatively -complex. - -While not completely water-tight, all major memory usages by a given -cgroup are tracked so that the total memory consumption can be -accounted and controlled to a reasonable extent. Currently, the -following types of memory usages are tracked. - -- Userland memory - page cache and anonymous memory. - -- Kernel data structures such as dentries and inodes. - -- TCP socket buffers. - -The above list may expand in the future for better coverage. - - -Memory Interface Files -~~~~~~~~~~~~~~~~~~~~~~ - -All memory amounts are in bytes. If a value which is not aligned to -PAGE_SIZE is written, the value may be rounded up to the closest -PAGE_SIZE multiple when read back. - - memory.current - A read-only single value file which exists on non-root - cgroups. - - The total amount of memory currently being used by the cgroup - and its descendants. - - memory.low - A read-write single value file which exists on non-root - cgroups. The default is "0". - - Best-effort memory protection. If the memory usages of a - cgroup and all its ancestors are below their low boundaries, - the cgroup's memory won't be reclaimed unless memory can be - reclaimed from unprotected cgroups. - - Putting more memory than generally available under this - protection is discouraged. - - memory.high - A read-write single value file which exists on non-root - cgroups. The default is "max". - - Memory usage throttle limit. This is the main mechanism to - control memory usage of a cgroup. If a cgroup's usage goes - over the high boundary, the processes of the cgroup are - throttled and put under heavy reclaim pressure. - - Going over the high limit never invokes the OOM killer and - under extreme conditions the limit may be breached. - - memory.max - A read-write single value file which exists on non-root - cgroups. The default is "max". - - Memory usage hard limit. This is the final protection - mechanism. If a cgroup's memory usage reaches this limit and - can't be reduced, the OOM killer is invoked in the cgroup. - Under certain circumstances, the usage may go over the limit - temporarily. - - This is the ultimate protection mechanism. As long as the - high limit is used and monitored properly, this limit's - utility is limited to providing the final safety net. - - memory.events - A read-only flat-keyed file which exists on non-root cgroups. - The following entries are defined. Unless specified - otherwise, a value change in this file generates a file - modified event. - - low - The number of times the cgroup is reclaimed due to - high memory pressure even though its usage is under - the low boundary. This usually indicates that the low - boundary is over-committed. - - high - The number of times processes of the cgroup are - throttled and routed to perform direct memory reclaim - because the high memory boundary was exceeded. For a - cgroup whose memory usage is capped by the high limit - rather than global memory pressure, this event's - occurrences are expected. - - max - The number of times the cgroup's memory usage was - about to go over the max boundary. If direct reclaim - fails to bring it down, the cgroup goes to OOM state. - - oom - The number of time the cgroup's memory usage was - reached the limit and allocation was about to fail. - - Depending on context result could be invocation of OOM - killer and retrying allocation or failing allocation. - - Failed allocation in its turn could be returned into - userspace as -ENOMEM or silently ignored in cases like - disk readahead. For now OOM in memory cgroup kills - tasks iff shortage has happened inside page fault. - - oom_kill - The number of processes belonging to this cgroup - killed by any kind of OOM killer. - - memory.stat - A read-only flat-keyed file which exists on non-root cgroups. - - This breaks down the cgroup's memory footprint into different - types of memory, type-specific details, and other information - on the state and past events of the memory management system. - - All memory amounts are in bytes. - - The entries are ordered to be human readable, and new entries - can show up in the middle. Don't rely on items remaining in a - fixed position; use the keys to look up specific values! - - anon - Amount of memory used in anonymous mappings such as - brk(), sbrk(), and mmap(MAP_ANONYMOUS) - - file - Amount of memory used to cache filesystem data, - including tmpfs and shared memory. - - kernel_stack - Amount of memory allocated to kernel stacks. - - slab - Amount of memory used for storing in-kernel data - structures. - - sock - Amount of memory used in network transmission buffers - - shmem - Amount of cached filesystem data that is swap-backed, - such as tmpfs, shm segments, shared anonymous mmap()s - - file_mapped - Amount of cached filesystem data mapped with mmap() - - file_dirty - Amount of cached filesystem data that was modified but - not yet written back to disk - - file_writeback - Amount of cached filesystem data that was modified and - is currently being written back to disk - - inactive_anon, active_anon, inactive_file, active_file, unevictable - Amount of memory, swap-backed and filesystem-backed, - on the internal memory management lists used by the - page reclaim algorithm - - slab_reclaimable - Part of "slab" that might be reclaimed, such as - dentries and inodes. - - slab_unreclaimable - Part of "slab" that cannot be reclaimed on memory - pressure. - - pgfault - Total number of page faults incurred - - pgmajfault - Number of major page faults incurred - - workingset_refault - - Number of refaults of previously evicted pages - - workingset_activate - - Number of refaulted pages that were immediately activated - - workingset_nodereclaim - - Number of times a shadow node has been reclaimed - - pgrefill - - Amount of scanned pages (in an active LRU list) - - pgscan - - Amount of scanned pages (in an inactive LRU list) - - pgsteal - - Amount of reclaimed pages - - pgactivate - - Amount of pages moved to the active LRU list - - pgdeactivate - - Amount of pages moved to the inactive LRU lis - - pglazyfree - - Amount of pages postponed to be freed under memory pressure - - pglazyfreed - - Amount of reclaimed lazyfree pages - - memory.swap.current - A read-only single value file which exists on non-root - cgroups. - - The total amount of swap currently being used by the cgroup - and its descendants. - - memory.swap.max - A read-write single value file which exists on non-root - cgroups. The default is "max". - - Swap usage hard limit. If a cgroup's swap usage reaches this - limit, anonymous memory of the cgroup will not be swapped out. - - -Usage Guidelines -~~~~~~~~~~~~~~~~ - -"memory.high" is the main mechanism to control memory usage. -Over-committing on high limit (sum of high limits > available memory) -and letting global memory pressure to distribute memory according to -usage is a viable strategy. - -Because breach of the high limit doesn't trigger the OOM killer but -throttles the offending cgroup, a management agent has ample -opportunities to monitor and take appropriate actions such as granting -more memory or terminating the workload. - -Determining whether a cgroup has enough memory is not trivial as -memory usage doesn't indicate whether the workload can benefit from -more memory. For example, a workload which writes data received from -network to a file can use all available memory but can also operate as -performant with a small amount of memory. A measure of memory -pressure - how much the workload is being impacted due to lack of -memory - is necessary to determine whether a workload needs more -memory; unfortunately, memory pressure monitoring mechanism isn't -implemented yet. - - -Memory Ownership -~~~~~~~~~~~~~~~~ - -A memory area is charged to the cgroup which instantiated it and stays -charged to the cgroup until the area is released. Migrating a process -to a different cgroup doesn't move the memory usages that it -instantiated while in the previous cgroup to the new cgroup. - -A memory area may be used by processes belonging to different cgroups. -To which cgroup the area will be charged is in-deterministic; however, -over time, the memory area is likely to end up in a cgroup which has -enough memory allowance to avoid high reclaim pressure. - -If a cgroup sweeps a considerable amount of memory which is expected -to be accessed repeatedly by other cgroups, it may make sense to use -POSIX_FADV_DONTNEED to relinquish the ownership of memory areas -belonging to the affected files to ensure correct memory ownership. - - -IO --- - -The "io" controller regulates the distribution of IO resources. This -controller implements both weight based and absolute bandwidth or IOPS -limit distribution; however, weight based distribution is available -only if cfq-iosched is in use and neither scheme is available for -blk-mq devices. - - -IO Interface Files -~~~~~~~~~~~~~~~~~~ - - io.stat - A read-only nested-keyed file which exists on non-root - cgroups. - - Lines are keyed by $MAJ:$MIN device numbers and not ordered. - The following nested keys are defined. - - ====== =================== - rbytes Bytes read - wbytes Bytes written - rios Number of read IOs - wios Number of write IOs - ====== =================== - - An example read output follows: - - 8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353 - 8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252 - - io.weight - A read-write flat-keyed file which exists on non-root cgroups. - The default is "default 100". - - The first line is the default weight applied to devices - without specific override. The rest are overrides keyed by - $MAJ:$MIN device numbers and not ordered. The weights are in - the range [1, 10000] and specifies the relative amount IO time - the cgroup can use in relation to its siblings. - - The default weight can be updated by writing either "default - $WEIGHT" or simply "$WEIGHT". Overrides can be set by writing - "$MAJ:$MIN $WEIGHT" and unset by writing "$MAJ:$MIN default". - - An example read output follows:: - - default 100 - 8:16 200 - 8:0 50 - - io.max - A read-write nested-keyed file which exists on non-root - cgroups. - - BPS and IOPS based IO limit. Lines are keyed by $MAJ:$MIN - device numbers and not ordered. The following nested keys are - defined. - - ===== ================================== - rbps Max read bytes per second - wbps Max write bytes per second - riops Max read IO operations per second - wiops Max write IO operations per second - ===== ================================== - - When writing, any number of nested key-value pairs can be - specified in any order. "max" can be specified as the value - to remove a specific limit. If the same key is specified - multiple times, the outcome is undefined. - - BPS and IOPS are measured in each IO direction and IOs are - delayed if limit is reached. Temporary bursts are allowed. - - Setting read limit at 2M BPS and write at 120 IOPS for 8:16:: - - echo "8:16 rbps=2097152 wiops=120" > io.max - - Reading returns the following:: - - 8:16 rbps=2097152 wbps=max riops=max wiops=120 - - Write IOPS limit can be removed by writing the following:: - - echo "8:16 wiops=max" > io.max - - Reading now returns the following:: - - 8:16 rbps=2097152 wbps=max riops=max wiops=max - - -Writeback -~~~~~~~~~ - -Page cache is dirtied through buffered writes and shared mmaps and -written asynchronously to the backing filesystem by the writeback -mechanism. Writeback sits between the memory and IO domains and -regulates the proportion of dirty memory by balancing dirtying and -write IOs. - -The io controller, in conjunction with the memory controller, -implements control of page cache writeback IOs. The memory controller -defines the memory domain that dirty memory ratio is calculated and -maintained for and the io controller defines the io domain which -writes out dirty pages for the memory domain. Both system-wide and -per-cgroup dirty memory states are examined and the more restrictive -of the two is enforced. - -cgroup writeback requires explicit support from the underlying -filesystem. Currently, cgroup writeback is implemented on ext2, ext4 -and btrfs. On other filesystems, all writeback IOs are attributed to -the root cgroup. - -There are inherent differences in memory and writeback management -which affects how cgroup ownership is tracked. Memory is tracked per -page while writeback per inode. For the purpose of writeback, an -inode is assigned to a cgroup and all IO requests to write dirty pages -from the inode are attributed to that cgroup. - -As cgroup ownership for memory is tracked per page, there can be pages -which are associated with different cgroups than the one the inode is -associated with. These are called foreign pages. The writeback -constantly keeps track of foreign pages and, if a particular foreign -cgroup becomes the majority over a certain period of time, switches -the ownership of the inode to that cgroup. - -While this model is enough for most use cases where a given inode is -mostly dirtied by a single cgroup even when the main writing cgroup -changes over time, use cases where multiple cgroups write to a single -inode simultaneously are not supported well. In such circumstances, a -significant portion of IOs are likely to be attributed incorrectly. -As memory controller assigns page ownership on the first use and -doesn't update it until the page is released, even if writeback -strictly follows page ownership, multiple cgroups dirtying overlapping -areas wouldn't work as expected. It's recommended to avoid such usage -patterns. - -The sysctl knobs which affect writeback behavior are applied to cgroup -writeback as follows. - - vm.dirty_background_ratio, vm.dirty_ratio - These ratios apply the same to cgroup writeback with the - amount of available memory capped by limits imposed by the - memory controller and system-wide clean memory. - - vm.dirty_background_bytes, vm.dirty_bytes - For cgroup writeback, this is calculated into ratio against - total available memory and applied the same way as - vm.dirty[_background]_ratio. - - -PID ---- - -The process number controller is used to allow a cgroup to stop any -new tasks from being fork()'d or clone()'d after a specified limit is -reached. - -The number of tasks in a cgroup can be exhausted in ways which other -controllers cannot prevent, thus warranting its own controller. For -example, a fork bomb is likely to exhaust the number of tasks before -hitting memory restrictions. - -Note that PIDs used in this controller refer to TIDs, process IDs as -used by the kernel. - - -PID Interface Files -~~~~~~~~~~~~~~~~~~~ - - pids.max - A read-write single value file which exists on non-root - cgroups. The default is "max". - - Hard limit of number of processes. - - pids.current - A read-only single value file which exists on all cgroups. - - The number of processes currently in the cgroup and its - descendants. - -Organisational operations are not blocked by cgroup policies, so it is -possible to have pids.current > pids.max. This can be done by either -setting the limit to be smaller than pids.current, or attaching enough -processes to the cgroup such that pids.current is larger than -pids.max. However, it is not possible to violate a cgroup PID policy -through fork() or clone(). These will return -EAGAIN if the creation -of a new process would cause a cgroup policy to be violated. - - -Device controller ------------------ - -Device controller manages access to device files. It includes both -creation of new device files (using mknod), and access to the -existing device files. - -Cgroup v2 device controller has no interface files and is implemented -on top of cgroup BPF. To control access to device files, a user may -create bpf programs of the BPF_CGROUP_DEVICE type and attach them -to cgroups. On an attempt to access a device file, corresponding -BPF programs will be executed, and depending on the return value -the attempt will succeed or fail with -EPERM. - -A BPF_CGROUP_DEVICE program takes a pointer to the bpf_cgroup_dev_ctx -structure, which describes the device access attempt: access type -(mknod/read/write) and device (type, major and minor numbers). -If the program returns 0, the attempt fails with -EPERM, otherwise -it succeeds. - -An example of BPF_CGROUP_DEVICE program may be found in the kernel -source tree in the tools/testing/selftests/bpf/dev_cgroup.c file. - - -RDMA ----- - -The "rdma" controller regulates the distribution and accounting of -of RDMA resources. - -RDMA Interface Files -~~~~~~~~~~~~~~~~~~~~ - - rdma.max - A readwrite nested-keyed file that exists for all the cgroups - except root that describes current configured resource limit - for a RDMA/IB device. - - Lines are keyed by device name and are not ordered. - Each line contains space separated resource name and its configured - limit that can be distributed. - - The following nested keys are defined. - - ========== ============================= - hca_handle Maximum number of HCA Handles - hca_object Maximum number of HCA Objects - ========== ============================= - - An example for mlx4 and ocrdma device follows:: - - mlx4_0 hca_handle=2 hca_object=2000 - ocrdma1 hca_handle=3 hca_object=max - - rdma.current - A read-only file that describes current resource usage. - It exists for all the cgroup except root. - - An example for mlx4 and ocrdma device follows:: - - mlx4_0 hca_handle=1 hca_object=20 - ocrdma1 hca_handle=1 hca_object=23 - - -Misc ----- - -perf_event -~~~~~~~~~~ - -perf_event controller, if not mounted on a legacy hierarchy, is -automatically enabled on the v2 hierarchy so that perf events can -always be filtered by cgroup v2 path. The controller can still be -moved to a legacy hierarchy after v2 hierarchy is populated. - - -Non-normative information -------------------------- - -This section contains information that isn't considered to be a part of -the stable kernel API and so is subject to change. - - -CPU controller root cgroup process behaviour -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When distributing CPU cycles in the root cgroup each thread in this -cgroup is treated as if it was hosted in a separate child cgroup of the -root cgroup. This child cgroup weight is dependent on its thread nice -level. - -For details of this mapping see sched_prio_to_weight array in -kernel/sched/core.c file (values from this array should be scaled -appropriately so the neutral - nice 0 - value is 100 instead of 1024). - - -IO controller root cgroup process behaviour -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Root cgroup processes are hosted in an implicit leaf child node. -When distributing IO resources this implicit child node is taken into -account as if it was a normal child cgroup of the root cgroup with a -weight value of 200. - - -Namespace -========= - -Basics ------- - -cgroup namespace provides a mechanism to virtualize the view of the -"/proc/$PID/cgroup" file and cgroup mounts. The CLONE_NEWCGROUP clone -flag can be used with clone(2) and unshare(2) to create a new cgroup -namespace. The process running inside the cgroup namespace will have -its "/proc/$PID/cgroup" output restricted to cgroupns root. The -cgroupns root is the cgroup of the process at the time of creation of -the cgroup namespace. - -Without cgroup namespace, the "/proc/$PID/cgroup" file shows the -complete path of the cgroup of a process. In a container setup where -a set of cgroups and namespaces are intended to isolate processes the -"/proc/$PID/cgroup" file may leak potential system level information -to the isolated processes. For Example:: - - # cat /proc/self/cgroup - 0::/batchjobs/container_id1 - -The path '/batchjobs/container_id1' can be considered as system-data -and undesirable to expose to the isolated processes. cgroup namespace -can be used to restrict visibility of this path. For example, before -creating a cgroup namespace, one would see:: - - # ls -l /proc/self/ns/cgroup - lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835] - # cat /proc/self/cgroup - 0::/batchjobs/container_id1 - -After unsharing a new namespace, the view changes:: - - # ls -l /proc/self/ns/cgroup - lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183] - # cat /proc/self/cgroup - 0::/ - -When some thread from a multi-threaded process unshares its cgroup -namespace, the new cgroupns gets applied to the entire process (all -the threads). This is natural for the v2 hierarchy; however, for the -legacy hierarchies, this may be unexpected. - -A cgroup namespace is alive as long as there are processes inside or -mounts pinning it. When the last usage goes away, the cgroup -namespace is destroyed. The cgroupns root and the actual cgroups -remain. - - -The Root and Views ------------------- - -The 'cgroupns root' for a cgroup namespace is the cgroup in which the -process calling unshare(2) is running. For example, if a process in -/batchjobs/container_id1 cgroup calls unshare, cgroup -/batchjobs/container_id1 becomes the cgroupns root. For the -init_cgroup_ns, this is the real root ('/') cgroup. - -The cgroupns root cgroup does not change even if the namespace creator -process later moves to a different cgroup:: - - # ~/unshare -c # unshare cgroupns in some cgroup - # cat /proc/self/cgroup - 0::/ - # mkdir sub_cgrp_1 - # echo 0 > sub_cgrp_1/cgroup.procs - # cat /proc/self/cgroup - 0::/sub_cgrp_1 - -Each process gets its namespace-specific view of "/proc/$PID/cgroup" - -Processes running inside the cgroup namespace will be able to see -cgroup paths (in /proc/self/cgroup) only inside their root cgroup. -From within an unshared cgroupns:: - - # sleep 100000 & - [1] 7353 - # echo 7353 > sub_cgrp_1/cgroup.procs - # cat /proc/7353/cgroup - 0::/sub_cgrp_1 - -From the initial cgroup namespace, the real cgroup path will be -visible:: - - $ cat /proc/7353/cgroup - 0::/batchjobs/container_id1/sub_cgrp_1 - -From a sibling cgroup namespace (that is, a namespace rooted at a -different cgroup), the cgroup path relative to its own cgroup -namespace root will be shown. For instance, if PID 7353's cgroup -namespace root is at '/batchjobs/container_id2', then it will see:: - - # cat /proc/7353/cgroup - 0::/../container_id2/sub_cgrp_1 - -Note that the relative path always starts with '/' to indicate that -its relative to the cgroup namespace root of the caller. - - -Migration and setns(2) ----------------------- - -Processes inside a cgroup namespace can move into and out of the -namespace root if they have proper access to external cgroups. For -example, from inside a namespace with cgroupns root at -/batchjobs/container_id1, and assuming that the global hierarchy is -still accessible inside cgroupns:: - - # cat /proc/7353/cgroup - 0::/sub_cgrp_1 - # echo 7353 > batchjobs/container_id2/cgroup.procs - # cat /proc/7353/cgroup - 0::/../container_id2 - -Note that this kind of setup is not encouraged. A task inside cgroup -namespace should only be exposed to its own cgroupns hierarchy. - -setns(2) to another cgroup namespace is allowed when: - -(a) the process has CAP_SYS_ADMIN against its current user namespace -(b) the process has CAP_SYS_ADMIN against the target cgroup - namespace's userns - -No implicit cgroup changes happen with attaching to another cgroup -namespace. It is expected that the someone moves the attaching -process under the target cgroup namespace root. - - -Interaction with Other Namespaces ---------------------------------- - -Namespace specific cgroup hierarchy can be mounted by a process -running inside a non-init cgroup namespace:: - - # mount -t cgroup2 none $MOUNT_POINT - -This will mount the unified cgroup hierarchy with cgroupns root as the -filesystem root. The process needs CAP_SYS_ADMIN against its user and -mount namespaces. - -The virtualization of /proc/self/cgroup file combined with restricting -the view of cgroup hierarchy by namespace-private cgroupfs mount -provides a properly isolated cgroup view inside the container. - - -Information on Kernel Programming -================================= - -This section contains kernel programming information in the areas -where interacting with cgroup is necessary. cgroup core and -controllers are not covered. - - -Filesystem Support for Writeback --------------------------------- - -A filesystem can support cgroup writeback by updating -address_space_operations->writepage[s]() to annotate bio's using the -following two functions. - - wbc_init_bio(@wbc, @bio) - Should be called for each bio carrying writeback data and - associates the bio with the inode's owner cgroup. Can be - called anytime between bio allocation and submission. - - wbc_account_io(@wbc, @page, @bytes) - Should be called for each data segment being written out. - While this function doesn't care exactly when it's called - during the writeback session, it's the easiest and most - natural to call it as data segments are added to a bio. - -With writeback bio's annotated, cgroup support can be enabled per -super_block by setting SB_I_CGROUPWB in ->s_iflags. This allows for -selective disabling of cgroup writeback support which is helpful when -certain filesystem features, e.g. journaled data mode, are -incompatible. - -wbc_init_bio() binds the specified bio to its cgroup. Depending on -the configuration, the bio may be executed at a lower priority and if -the writeback session is holding shared resources, e.g. a journal -entry, may lead to priority inversion. There is no one easy solution -for the problem. Filesystems can try to work around specific problem -cases by skipping wbc_init_bio() or using bio_associate_blkcg() -directly. - - -Deprecated v1 Core Features -=========================== - -- Multiple hierarchies including named ones are not supported. - -- All v1 mount options are not supported. - -- The "tasks" file is removed and "cgroup.procs" is not sorted. - -- "cgroup.clone_children" is removed. - -- /proc/cgroups is meaningless for v2. Use "cgroup.controllers" file - at the root instead. - - -Issues with v1 and Rationales for v2 -==================================== - -Multiple Hierarchies --------------------- - -cgroup v1 allowed an arbitrary number of hierarchies and each -hierarchy could host any number of controllers. While this seemed to -provide a high level of flexibility, it wasn't useful in practice. - -For example, as there is only one instance of each controller, utility -type controllers such as freezer which can be useful in all -hierarchies could only be used in one. The issue is exacerbated by -the fact that controllers couldn't be moved to another hierarchy once -hierarchies were populated. Another issue was that all controllers -bound to a hierarchy were forced to have exactly the same view of the -hierarchy. It wasn't possible to vary the granularity depending on -the specific controller. - -In practice, these issues heavily limited which controllers could be -put on the same hierarchy and most configurations resorted to putting -each controller on its own hierarchy. Only closely related ones, such -as the cpu and cpuacct controllers, made sense to be put on the same -hierarchy. This often meant that userland ended up managing multiple -similar hierarchies repeating the same steps on each hierarchy -whenever a hierarchy management operation was necessary. - -Furthermore, support for multiple hierarchies came at a steep cost. -It greatly complicated cgroup core implementation but more importantly -the support for multiple hierarchies restricted how cgroup could be -used in general and what controllers was able to do. - -There was no limit on how many hierarchies there might be, which meant -that a thread's cgroup membership couldn't be described in finite -length. The key might contain any number of entries and was unlimited -in length, which made it highly awkward to manipulate and led to -addition of controllers which existed only to identify membership, -which in turn exacerbated the original problem of proliferating number -of hierarchies. - -Also, as a controller couldn't have any expectation regarding the -topologies of hierarchies other controllers might be on, each -controller had to assume that all other controllers were attached to -completely orthogonal hierarchies. This made it impossible, or at -least very cumbersome, for controllers to cooperate with each other. - -In most use cases, putting controllers on hierarchies which are -completely orthogonal to each other isn't necessary. What usually is -called for is the ability to have differing levels of granularity -depending on the specific controller. In other words, hierarchy may -be collapsed from leaf towards root when viewed from specific -controllers. For example, a given configuration might not care about -how memory is distributed beyond a certain level while still wanting -to control how CPU cycles are distributed. - - -Thread Granularity ------------------- - -cgroup v1 allowed threads of a process to belong to different cgroups. -This didn't make sense for some controllers and those controllers -ended up implementing different ways to ignore such situations but -much more importantly it blurred the line between API exposed to -individual applications and system management interface. - -Generally, in-process knowledge is available only to the process -itself; thus, unlike service-level organization of processes, -categorizing threads of a process requires active participation from -the application which owns the target process. - -cgroup v1 had an ambiguously defined delegation model which got abused -in combination with thread granularity. cgroups were delegated to -individual applications so that they can create and manage their own -sub-hierarchies and control resource distributions along them. This -effectively raised cgroup to the status of a syscall-like API exposed -to lay programs. - -First of all, cgroup has a fundamentally inadequate interface to be -exposed this way. For a process to access its own knobs, it has to -extract the path on the target hierarchy from /proc/self/cgroup, -construct the path by appending the name of the knob to the path, open -and then read and/or write to it. This is not only extremely clunky -and unusual but also inherently racy. There is no conventional way to -define transaction across the required steps and nothing can guarantee -that the process would actually be operating on its own sub-hierarchy. - -cgroup controllers implemented a number of knobs which would never be -accepted as public APIs because they were just adding control knobs to -system-management pseudo filesystem. cgroup ended up with interface -knobs which were not properly abstracted or refined and directly -revealed kernel internal details. These knobs got exposed to -individual applications through the ill-defined delegation mechanism -effectively abusing cgroup as a shortcut to implementing public APIs -without going through the required scrutiny. - -This was painful for both userland and kernel. Userland ended up with -misbehaving and poorly abstracted interfaces and kernel exposing and -locked into constructs inadvertently. - - -Competition Between Inner Nodes and Threads -------------------------------------------- - -cgroup v1 allowed threads to be in any cgroups which created an -interesting problem where threads belonging to a parent cgroup and its -children cgroups competed for resources. This was nasty as two -different types of entities competed and there was no obvious way to -settle it. Different controllers did different things. - -The cpu controller considered threads and cgroups as equivalents and -mapped nice levels to cgroup weights. This worked for some cases but -fell flat when children wanted to be allocated specific ratios of CPU -cycles and the number of internal threads fluctuated - the ratios -constantly changed as the number of competing entities fluctuated. -There also were other issues. The mapping from nice level to weight -wasn't obvious or universal, and there were various other knobs which -simply weren't available for threads. - -The io controller implicitly created a hidden leaf node for each -cgroup to host the threads. The hidden leaf had its own copies of all -the knobs with ``leaf_`` prefixed. While this allowed equivalent -control over internal threads, it was with serious drawbacks. It -always added an extra layer of nesting which wouldn't be necessary -otherwise, made the interface messy and significantly complicated the -implementation. - -The memory controller didn't have a way to control what happened -between internal tasks and child cgroups and the behavior was not -clearly defined. There were attempts to add ad-hoc behaviors and -knobs to tailor the behavior to specific workloads which would have -led to problems extremely difficult to resolve in the long term. - -Multiple controllers struggled with internal tasks and came up with -different ways to deal with it; unfortunately, all the approaches were -severely flawed and, furthermore, the widely different behaviors -made cgroup as a whole highly inconsistent. - -This clearly is a problem which needs to be addressed from cgroup core -in a uniform way. - - -Other Interface Issues ----------------------- - -cgroup v1 grew without oversight and developed a large number of -idiosyncrasies and inconsistencies. One issue on the cgroup core side -was how an empty cgroup was notified - a userland helper binary was -forked and executed for each event. The event delivery wasn't -recursive or delegatable. The limitations of the mechanism also led -to in-kernel event delivery filtering mechanism further complicating -the interface. - -Controller interfaces were problematic too. An extreme example is -controllers completely ignoring hierarchical organization and treating -all cgroups as if they were all located directly under the root -cgroup. Some controllers exposed a large amount of inconsistent -implementation details to userland. - -There also was no consistency across controllers. When a new cgroup -was created, some controllers defaulted to not imposing extra -restrictions while others disallowed any resource usage until -explicitly configured. Configuration knobs for the same type of -control used widely differing naming schemes and formats. Statistics -and information knobs were named arbitrarily and used different -formats and units even in the same controller. - -cgroup v2 establishes common conventions where appropriate and updates -controllers so that they expose minimal and consistent interfaces. - - -Controller Issues and Remedies ------------------------------- - -Memory -~~~~~~ - -The original lower boundary, the soft limit, is defined as a limit -that is per default unset. As a result, the set of cgroups that -global reclaim prefers is opt-in, rather than opt-out. The costs for -optimizing these mostly negative lookups are so high that the -implementation, despite its enormous size, does not even provide the -basic desirable behavior. First off, the soft limit has no -hierarchical meaning. All configured groups are organized in a global -rbtree and treated like equal peers, regardless where they are located -in the hierarchy. This makes subtree delegation impossible. Second, -the soft limit reclaim pass is so aggressive that it not just -introduces high allocation latencies into the system, but also impacts -system performance due to overreclaim, to the point where the feature -becomes self-defeating. - -The memory.low boundary on the other hand is a top-down allocated -reserve. A cgroup enjoys reclaim protection when it and all its -ancestors are below their low boundaries, which makes delegation of -subtrees possible. Secondly, new cgroups have no reserve per default -and in the common case most cgroups are eligible for the preferred -reclaim pass. This allows the new low boundary to be efficiently -implemented with just a minor addition to the generic reclaim code, -without the need for out-of-band data structures and reclaim passes. -Because the generic reclaim code considers all cgroups except for the -ones running low in the preferred first reclaim pass, overreclaim of -individual groups is eliminated as well, resulting in much better -overall workload performance. - -The original high boundary, the hard limit, is defined as a strict -limit that can not budge, even if the OOM killer has to be called. -But this generally goes against the goal of making the most out of the -available memory. The memory consumption of workloads varies during -runtime, and that requires users to overcommit. But doing that with a -strict upper limit requires either a fairly accurate prediction of the -working set size or adding slack to the limit. Since working set size -estimation is hard and error prone, and getting it wrong results in -OOM kills, most users tend to err on the side of a looser limit and -end up wasting precious resources. - -The memory.high boundary on the other hand can be set much more -conservatively. When hit, it throttles allocations by forcing them -into direct reclaim to work off the excess, but it never invokes the -OOM killer. As a result, a high boundary that is chosen too -aggressively will not terminate the processes, but instead it will -lead to gradual performance degradation. The user can monitor this -and make corrections until the minimal memory footprint that still -gives acceptable performance is found. - -In extreme cases, with many concurrent allocations and a complete -breakdown of reclaim progress within the group, the high boundary can -be exceeded. But even then it's mostly better to satisfy the -allocation from the slack available in other groups or the rest of the -system than killing the group. Otherwise, memory.max is there to -limit this type of spillover and ultimately contain buggy or even -malicious applications. - -Setting the original memory.limit_in_bytes below the current usage was -subject to a race condition, where concurrent charges could cause the -limit setting to fail. memory.max on the other hand will first set the -limit to prevent new charges, and then reclaim and OOM kill until the -new limit is met - or the task writing to memory.max is killed. - -The combined memory+swap accounting and limiting is replaced by real -control over swap space. - -The main argument for a combined memory+swap facility in the original -cgroup design was that global or parental pressure would always be -able to swap all anonymous memory of a child group, regardless of the -child's own (possibly untrusted) configuration. However, untrusted -groups can sabotage swapping by other means - such as referencing its -anonymous memory in a tight loop - and an admin can not assume full -swappability when overcommitting untrusted jobs. - -For trusted jobs, on the other hand, a combined counter is not an -intuitive userspace interface, and it flies in the face of the idea -that cgroup controllers should account and limit specific physical -resources. Swap space is a resource like all others in the system, -and that's why unified hierarchy allows distributing it separately. diff --git a/Documentation/clk.txt b/Documentation/clk.txt deleted file mode 100644 index 511628bb3d3a..000000000000 --- a/Documentation/clk.txt +++ /dev/null @@ -1,307 +0,0 @@ -======================== -The Common Clk Framework -======================== - -:Author: Mike Turquette - -This document endeavours to explain the common clk framework details, -and how to port a platform over to this framework. It is not yet a -detailed explanation of the clock api in include/linux/clk.h, but -perhaps someday it will include that information. - -Introduction and interface split -================================ - -The common clk framework is an interface to control the clock nodes -available on various devices today. This may come in the form of clock -gating, rate adjustment, muxing or other operations. This framework is -enabled with the CONFIG_COMMON_CLK option. - -The interface itself is divided into two halves, each shielded from the -details of its counterpart. First is the common definition of struct -clk which unifies the framework-level accounting and infrastructure that -has traditionally been duplicated across a variety of platforms. Second -is a common implementation of the clk.h api, defined in -drivers/clk/clk.c. Finally there is struct clk_ops, whose operations -are invoked by the clk api implementation. - -The second half of the interface is comprised of the hardware-specific -callbacks registered with struct clk_ops and the corresponding -hardware-specific structures needed to model a particular clock. For -the remainder of this document any reference to a callback in struct -clk_ops, such as .enable or .set_rate, implies the hardware-specific -implementation of that code. Likewise, references to struct clk_foo -serve as a convenient shorthand for the implementation of the -hardware-specific bits for the hypothetical "foo" hardware. - -Tying the two halves of this interface together is struct clk_hw, which -is defined in struct clk_foo and pointed to within struct clk_core. This -allows for easy navigation between the two discrete halves of the common -clock interface. - -Common data structures and api -============================== - -Below is the common struct clk_core definition from -drivers/clk/clk.c, modified for brevity:: - - struct clk_core { - const char *name; - const struct clk_ops *ops; - struct clk_hw *hw; - struct module *owner; - struct clk_core *parent; - const char **parent_names; - struct clk_core **parents; - u8 num_parents; - u8 new_parent_index; - ... - }; - -The members above make up the core of the clk tree topology. The clk -api itself defines several driver-facing functions which operate on -struct clk. That api is documented in include/linux/clk.h. - -Platforms and devices utilizing the common struct clk_core use the struct -clk_ops pointer in struct clk_core to perform the hardware-specific parts of -the operations defined in clk-provider.h:: - - struct clk_ops { - int (*prepare)(struct clk_hw *hw); - void (*unprepare)(struct clk_hw *hw); - int (*is_prepared)(struct clk_hw *hw); - void (*unprepare_unused)(struct clk_hw *hw); - int (*enable)(struct clk_hw *hw); - void (*disable)(struct clk_hw *hw); - int (*is_enabled)(struct clk_hw *hw); - void (*disable_unused)(struct clk_hw *hw); - unsigned long (*recalc_rate)(struct clk_hw *hw, - unsigned long parent_rate); - long (*round_rate)(struct clk_hw *hw, - unsigned long rate, - unsigned long *parent_rate); - int (*determine_rate)(struct clk_hw *hw, - struct clk_rate_request *req); - int (*set_parent)(struct clk_hw *hw, u8 index); - u8 (*get_parent)(struct clk_hw *hw); - int (*set_rate)(struct clk_hw *hw, - unsigned long rate, - unsigned long parent_rate); - int (*set_rate_and_parent)(struct clk_hw *hw, - unsigned long rate, - unsigned long parent_rate, - u8 index); - unsigned long (*recalc_accuracy)(struct clk_hw *hw, - unsigned long parent_accuracy); - int (*get_phase)(struct clk_hw *hw); - int (*set_phase)(struct clk_hw *hw, int degrees); - void (*init)(struct clk_hw *hw); - int (*debug_init)(struct clk_hw *hw, - struct dentry *dentry); - }; - -Hardware clk implementations -============================ - -The strength of the common struct clk_core comes from its .ops and .hw pointers -which abstract the details of struct clk from the hardware-specific bits, and -vice versa. To illustrate consider the simple gateable clk implementation in -drivers/clk/clk-gate.c:: - - struct clk_gate { - struct clk_hw hw; - void __iomem *reg; - u8 bit_idx; - ... - }; - -struct clk_gate contains struct clk_hw hw as well as hardware-specific -knowledge about which register and bit controls this clk's gating. -Nothing about clock topology or accounting, such as enable_count or -notifier_count, is needed here. That is all handled by the common -framework code and struct clk_core. - -Let's walk through enabling this clk from driver code:: - - struct clk *clk; - clk = clk_get(NULL, "my_gateable_clk"); - - clk_prepare(clk); - clk_enable(clk); - -The call graph for clk_enable is very simple:: - - clk_enable(clk); - clk->ops->enable(clk->hw); - [resolves to...] - clk_gate_enable(hw); - [resolves struct clk gate with to_clk_gate(hw)] - clk_gate_set_bit(gate); - -And the definition of clk_gate_set_bit:: - - static void clk_gate_set_bit(struct clk_gate *gate) - { - u32 reg; - - reg = __raw_readl(gate->reg); - reg |= BIT(gate->bit_idx); - writel(reg, gate->reg); - } - -Note that to_clk_gate is defined as:: - - #define to_clk_gate(_hw) container_of(_hw, struct clk_gate, hw) - -This pattern of abstraction is used for every clock hardware -representation. - -Supporting your own clk hardware -================================ - -When implementing support for a new type of clock it is only necessary to -include the following header:: - - #include - -To construct a clk hardware structure for your platform you must define -the following:: - - struct clk_foo { - struct clk_hw hw; - ... hardware specific data goes here ... - }; - -To take advantage of your data you'll need to support valid operations -for your clk:: - - struct clk_ops clk_foo_ops { - .enable = &clk_foo_enable; - .disable = &clk_foo_disable; - }; - -Implement the above functions using container_of:: - - #define to_clk_foo(_hw) container_of(_hw, struct clk_foo, hw) - - int clk_foo_enable(struct clk_hw *hw) - { - struct clk_foo *foo; - - foo = to_clk_foo(hw); - - ... perform magic on foo ... - - return 0; - }; - -Below is a matrix detailing which clk_ops are mandatory based upon the -hardware capabilities of that clock. A cell marked as "y" means -mandatory, a cell marked as "n" implies that either including that -callback is invalid or otherwise unnecessary. Empty cells are either -optional or must be evaluated on a case-by-case basis. - -.. table:: clock hardware characteristics - - +----------------+------+-------------+---------------+-------------+------+ - | | gate | change rate | single parent | multiplexer | root | - +================+======+=============+===============+=============+======+ - |.prepare | | | | | | - +----------------+------+-------------+---------------+-------------+------+ - |.unprepare | | | | | | - +----------------+------+-------------+---------------+-------------+------+ - +----------------+------+-------------+---------------+-------------+------+ - |.enable | y | | | | | - +----------------+------+-------------+---------------+-------------+------+ - |.disable | y | | | | | - +----------------+------+-------------+---------------+-------------+------+ - |.is_enabled | y | | | | | - +----------------+------+-------------+---------------+-------------+------+ - +----------------+------+-------------+---------------+-------------+------+ - |.recalc_rate | | y | | | | - +----------------+------+-------------+---------------+-------------+------+ - |.round_rate | | y [1]_ | | | | - +----------------+------+-------------+---------------+-------------+------+ - |.determine_rate | | y [1]_ | | | | - +----------------+------+-------------+---------------+-------------+------+ - |.set_rate | | y | | | | - +----------------+------+-------------+---------------+-------------+------+ - +----------------+------+-------------+---------------+-------------+------+ - |.set_parent | | | n | y | n | - +----------------+------+-------------+---------------+-------------+------+ - |.get_parent | | | n | y | n | - +----------------+------+-------------+---------------+-------------+------+ - +----------------+------+-------------+---------------+-------------+------+ - |.recalc_accuracy| | | | | | - +----------------+------+-------------+---------------+-------------+------+ - +----------------+------+-------------+---------------+-------------+------+ - |.init | | | | | | - +----------------+------+-------------+---------------+-------------+------+ - -.. [1] either one of round_rate or determine_rate is required. - -Finally, register your clock at run-time with a hardware-specific -registration function. This function simply populates struct clk_foo's -data and then passes the common struct clk parameters to the framework -with a call to:: - - clk_register(...) - -See the basic clock types in ``drivers/clk/clk-*.c`` for examples. - -Disabling clock gating of unused clocks -======================================= - -Sometimes during development it can be useful to be able to bypass the -default disabling of unused clocks. For example, if drivers aren't enabling -clocks properly but rely on them being on from the bootloader, bypassing -the disabling means that the driver will remain functional while the issues -are sorted out. - -To bypass this disabling, include "clk_ignore_unused" in the bootargs to the -kernel. - -Locking -======= - -The common clock framework uses two global locks, the prepare lock and the -enable lock. - -The enable lock is a spinlock and is held across calls to the .enable, -.disable operations. Those operations are thus not allowed to sleep, -and calls to the clk_enable(), clk_disable() API functions are allowed in -atomic context. - -For clk_is_enabled() API, it is also designed to be allowed to be used in -atomic context. However, it doesn't really make any sense to hold the enable -lock in core, unless you want to do something else with the information of -the enable state with that lock held. Otherwise, seeing if a clk is enabled is -a one-shot read of the enabled state, which could just as easily change after -the function returns because the lock is released. Thus the user of this API -needs to handle synchronizing the read of the state with whatever they're -using it for to make sure that the enable state doesn't change during that -time. - -The prepare lock is a mutex and is held across calls to all other operations. -All those operations are allowed to sleep, and calls to the corresponding API -functions are not allowed in atomic context. - -This effectively divides operations in two groups from a locking perspective. - -Drivers don't need to manually protect resources shared between the operations -of one group, regardless of whether those resources are shared by multiple -clocks or not. However, access to resources that are shared between operations -of the two groups needs to be protected by the drivers. An example of such a -resource would be a register that controls both the clock rate and the clock -enable/disable state. - -The clock framework is reentrant, in that a driver is allowed to call clock -framework functions from within its implementation of clock operations. This -can for instance cause a .set_rate operation of one clock being called from -within the .set_rate operation of another clock. This case must be considered -in the driver implementations, but the code flow is usually controlled by the -driver in that case. - -Note that locking must also be considered when code outside of the common -clock framework needs to access resources used by the clock operations. This -is considered out of scope of this document. diff --git a/Documentation/core-api/atomic_ops.rst b/Documentation/core-api/atomic_ops.rst index fce929144ccd..2e7165f86f55 100644 --- a/Documentation/core-api/atomic_ops.rst +++ b/Documentation/core-api/atomic_ops.rst @@ -111,7 +111,6 @@ If the compiler can prove that do_something() does not store to the variable a, then the compiler is within its rights transforming this to the following:: - tmp = a; if (a > 0) for (;;) do_something(); @@ -119,7 +118,7 @@ the following:: If you don't want the compiler to do this (and you probably don't), then you should use something like the following:: - while (READ_ONCE(a) < 0) + while (READ_ONCE(a) > 0) do_something(); Alternatively, you could place a barrier() call in the loop. @@ -467,10 +466,12 @@ Like the above, except that these routines return a boolean which indicates whether the changed bit was set _BEFORE_ the atomic bit operation. -WARNING! It is incredibly important that the value be a boolean, -ie. "0" or "1". Do not try to be fancy and save a few instructions by -declaring the above to return "long" and just returning something like -"old_val & mask" because that will not work. + +.. warning:: + It is incredibly important that the value be a boolean, ie. "0" or "1". + Do not try to be fancy and save a few instructions by declaring the + above to return "long" and just returning something like "old_val & + mask" because that will not work. For one thing, this return value gets truncated to int in many code paths using these interfaces, so on 64-bit if the bit is set in the diff --git a/Documentation/cachetlb.txt b/Documentation/core-api/cachetlb.rst similarity index 100% rename from Documentation/cachetlb.txt rename to Documentation/core-api/cachetlb.rst diff --git a/Documentation/circular-buffers.txt b/Documentation/core-api/circular-buffers.rst similarity index 100% rename from Documentation/circular-buffers.txt rename to Documentation/core-api/circular-buffers.rst diff --git a/Documentation/core-api/gfp_mask-from-fs-io.rst b/Documentation/core-api/gfp_mask-from-fs-io.rst new file mode 100644 index 000000000000..e0df8f416582 --- /dev/null +++ b/Documentation/core-api/gfp_mask-from-fs-io.rst @@ -0,0 +1,66 @@ +================================= +GFP masks used from FS/IO context +================================= + +:Date: May, 2018 +:Author: Michal Hocko + +Introduction +============ + +Code paths in the filesystem and IO stacks must be careful when +allocating memory to prevent recursion deadlocks caused by direct +memory reclaim calling back into the FS or IO paths and blocking on +already held resources (e.g. locks - most commonly those used for the +transaction context). + +The traditional way to avoid this deadlock problem is to clear __GFP_FS +respectively __GFP_IO (note the latter implies clearing the first as well) in +the gfp mask when calling an allocator. GFP_NOFS respectively GFP_NOIO can be +used as shortcut. It turned out though that above approach has led to +abuses when the restricted gfp mask is used "just in case" without a +deeper consideration which leads to problems because an excessive use +of GFP_NOFS/GFP_NOIO can lead to memory over-reclaim or other memory +reclaim issues. + +New API +======== + +Since 4.12 we do have a generic scope API for both NOFS and NOIO context +``memalloc_nofs_save``, ``memalloc_nofs_restore`` respectively ``memalloc_noio_save``, +``memalloc_noio_restore`` which allow to mark a scope to be a critical +section from a filesystem or I/O point of view. Any allocation from that +scope will inherently drop __GFP_FS respectively __GFP_IO from the given +mask so no memory allocation can recurse back in the FS/IO. + +.. kernel-doc:: include/linux/sched/mm.h + :functions: memalloc_nofs_save memalloc_nofs_restore +.. kernel-doc:: include/linux/sched/mm.h + :functions: memalloc_noio_save memalloc_noio_restore + +FS/IO code then simply calls the appropriate save function before +any critical section with respect to the reclaim is started - e.g. +lock shared with the reclaim context or when a transaction context +nesting would be possible via reclaim. The restore function should be +called when the critical section ends. All that ideally along with an +explanation what is the reclaim context for easier maintenance. + +Please note that the proper pairing of save/restore functions +allows nesting so it is safe to call ``memalloc_noio_save`` or +``memalloc_noio_restore`` respectively from an existing NOIO or NOFS +scope. + +What about __vmalloc(GFP_NOFS) +============================== + +vmalloc doesn't support GFP_NOFS semantic because there are hardcoded +GFP_KERNEL allocations deep inside the allocator which are quite non-trivial +to fix up. That means that calling ``vmalloc`` with GFP_NOFS/GFP_NOIO is +almost always a bug. The good news is that the NOFS/NOIO semantic can be +achieved by the scope API. + +In the ideal world, upper layers should already mark dangerous contexts +and so no special care is required and vmalloc should be called without +any problems. Sometimes if the context is not really clear or there are +layering violations then the recommended way around that is to wrap ``vmalloc`` +by the scope API with a comment explaining the problem. diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst index c670a8031786..f5a66b72f984 100644 --- a/Documentation/core-api/index.rst +++ b/Documentation/core-api/index.rst @@ -14,6 +14,7 @@ Core utilities kernel-api assoc_array atomic_ops + cachetlb refcount-vs-atomic cpu_hotplug idr @@ -25,6 +26,8 @@ Core utilities genalloc errseq printk-formats + circular-buffers + gfp_mask-from-fs-io Interfaces for kernel debugging =============================== diff --git a/Documentation/core-api/kernel-api.rst b/Documentation/core-api/kernel-api.rst index 92f30006adae..76fe2d0f5e7d 100644 --- a/Documentation/core-api/kernel-api.rst +++ b/Documentation/core-api/kernel-api.rst @@ -39,17 +39,17 @@ String Manipulation .. kernel-doc:: lib/string.c :export: +Basic Kernel Library Functions +============================== + +The Linux kernel provides more basic utility functions. + Bit Operations -------------- .. kernel-doc:: arch/x86/include/asm/bitops.h :internal: -Basic Kernel Library Functions -============================== - -The Linux kernel provides more basic utility functions. - Bitmap Operations ----------------- @@ -80,6 +80,31 @@ Command-line Parsing .. kernel-doc:: lib/cmdline.c :export: +Sorting +------- + +.. kernel-doc:: lib/sort.c + :export: + +.. kernel-doc:: lib/list_sort.c + :export: + +Text Searching +-------------- + +.. kernel-doc:: lib/textsearch.c + :doc: ts_intro + +.. kernel-doc:: lib/textsearch.c + :export: + +.. kernel-doc:: include/linux/textsearch.h + :functions: textsearch_find textsearch_next \ + textsearch_get_pattern textsearch_get_pattern_len + +CRC and Math Functions in Linux +=============================== + CRC Functions ------------- @@ -103,9 +128,6 @@ CRC Functions .. kernel-doc:: lib/crc-itu-t.c :export: -Math Functions in Linux -======================= - Base 2 log and power Functions ------------------------------ @@ -127,28 +149,6 @@ Division Functions .. kernel-doc:: lib/gcd.c :export: -Sorting -------- - -.. kernel-doc:: lib/sort.c - :export: - -.. kernel-doc:: lib/list_sort.c - :export: - -Text Searching --------------- - -.. kernel-doc:: lib/textsearch.c - :doc: ts_intro - -.. kernel-doc:: lib/textsearch.c - :export: - -.. kernel-doc:: include/linux/textsearch.h - :functions: textsearch_find textsearch_next \ - textsearch_get_pattern textsearch_get_pattern_len - UUID/GUID --------- @@ -284,7 +284,7 @@ Resources Management MTRR Handling ------------- -.. kernel-doc:: arch/x86/kernel/cpu/mtrr/main.c +.. kernel-doc:: arch/x86/kernel/cpu/mtrr/mtrr.c :export: Security Framework diff --git a/Documentation/core-api/printk-formats.rst b/Documentation/core-api/printk-formats.rst index eb30efdd2e78..25dc591cb110 100644 --- a/Documentation/core-api/printk-formats.rst +++ b/Documentation/core-api/printk-formats.rst @@ -419,11 +419,10 @@ struct clk %pC pll1 %pCn pll1 - %pCr 1560000000 For printing struct clk structures. %pC and %pCn print the name (Common Clock Framework) or address (legacy clock framework) of the -structure; %pCr prints the current clock rate. +structure. Passed by reference. diff --git a/Documentation/core-api/refcount-vs-atomic.rst b/Documentation/core-api/refcount-vs-atomic.rst index 83351c258cdb..322851bada16 100644 --- a/Documentation/core-api/refcount-vs-atomic.rst +++ b/Documentation/core-api/refcount-vs-atomic.rst @@ -17,7 +17,7 @@ in order to help maintainers validate their code against the change in these memory ordering guarantees. The terms used through this document try to follow the formal LKMM defined in -github.com/aparri/memory-model/blob/master/Documentation/explanation.txt +tools/memory-model/Documentation/explanation.txt. memory-barriers.txt and atomic_t.txt provide more background to the memory ordering in general and for atomic operations specifically. diff --git a/Documentation/crypto/crypto_engine.rst b/Documentation/crypto/crypto_engine.rst index 8272ac92a14f..1d56221dfe35 100644 --- a/Documentation/crypto/crypto_engine.rst +++ b/Documentation/crypto/crypto_engine.rst @@ -8,11 +8,13 @@ The crypto engine API (CE), is a crypto queue manager. Requirement ----------- -You have to put at start of your tfm_ctx the struct crypto_engine_ctx -struct your_tfm_ctx { +You have to put at start of your tfm_ctx the struct crypto_engine_ctx:: + + struct your_tfm_ctx { struct crypto_engine_ctx enginectx; ... -}; + }; + Why: Since CE manage only crypto_async_request, it cannot know the underlying request_type and so have access only on the TFM. So using container_of for accessing __ctx is impossible. diff --git a/Documentation/crypto/index.rst b/Documentation/crypto/index.rst index 94c4786f2573..c4ff5d791233 100644 --- a/Documentation/crypto/index.rst +++ b/Documentation/crypto/index.rst @@ -20,5 +20,6 @@ for cryptographic use cases, as well as programming examples. architecture devel-algos userspace-if + crypto_engine api api-samples diff --git a/Documentation/dell_rbu.txt b/Documentation/dell_rbu.txt index 0fdb6aa2704c..5d1ce7bcd04d 100644 --- a/Documentation/dell_rbu.txt +++ b/Documentation/dell_rbu.txt @@ -121,10 +121,7 @@ read back the image downloaded. .. note:: - This driver requires a patch for firmware_class.c which has the modified - request_firmware_nowait function. - - Also after updating the BIOS image a user mode application needs to execute + After updating the BIOS image a user mode application needs to execute code which sends the BIOS update request to the BIOS. So on the next reboot the BIOS knows about the new image downloaded and it updates itself. Also don't unload the rbu driver if the image has to be updated. diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst index f7a18f274357..aabc8738b3d8 100644 --- a/Documentation/dev-tools/kasan.rst +++ b/Documentation/dev-tools/kasan.rst @@ -120,7 +120,7 @@ A typical out of bounds access report looks like this:: The header of the report discribe what kind of bug happened and what kind of access caused it. It's followed by the description of the accessed slub object -(see 'SLUB Debug output' section in Documentation/vm/slub.txt for details) and +(see 'SLUB Debug output' section in Documentation/vm/slub.rst for details) and the description of the accessed memory page. In the last section the report shows memory state around the accessed address. diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst index e80850eefe13..3bf371a938d0 100644 --- a/Documentation/dev-tools/kselftest.rst +++ b/Documentation/dev-tools/kselftest.rst @@ -151,6 +151,11 @@ Contributing new tests (details) TEST_FILES, TEST_GEN_FILES mean it is the file which is used by test. + * First use the headers inside the kernel source and/or git repo, and then the + system headers. Headers for the kernel release as opposed to headers + installed by the distro on the system should be the primary focus to be able + to find regressions. + Test Harness ============ diff --git a/Documentation/device-mapper/writecache.txt b/Documentation/device-mapper/writecache.txt new file mode 100644 index 000000000000..4424fa2c67d7 --- /dev/null +++ b/Documentation/device-mapper/writecache.txt @@ -0,0 +1,68 @@ +The writecache target caches writes on persistent memory or on SSD. It +doesn't cache reads because reads are supposed to be cached in page cache +in normal RAM. + +When the device is constructed, the first sector should be zeroed or the +first sector should contain valid superblock from previous invocation. + +Constructor parameters: +1. type of the cache device - "p" or "s" + p - persistent memory + s - SSD +2. the underlying device that will be cached +3. the cache device +4. block size (4096 is recommended; the maximum block size is the page + size) +5. the number of optional parameters (the parameters with an argument + count as two) + high_watermark n (default: 50) + start writeback when the number of used blocks reach this + watermark + low_watermark x (default: 45) + stop writeback when the number of used blocks drops below + this watermark + writeback_jobs n (default: unlimited) + limit the number of blocks that are in flight during + writeback. Setting this value reduces writeback + throughput, but it may improve latency of read requests + autocommit_blocks n (default: 64 for pmem, 65536 for ssd) + when the application writes this amount of blocks without + issuing the FLUSH request, the blocks are automatically + commited + autocommit_time ms (default: 1000) + autocommit time in milliseconds. The data is automatically + commited if this time passes and no FLUSH request is + received + fua (by default on) + applicable only to persistent memory - use the FUA flag + when writing data from persistent memory back to the + underlying device + nofua + applicable only to persistent memory - don't use the FUA + flag when writing back data and send the FLUSH request + afterwards + - some underlying devices perform better with fua, some + with nofua. The user should test it + +Status: +1. error indicator - 0 if there was no error, otherwise error number +2. the number of blocks +3. the number of free blocks +4. the number of blocks under writeback + +Messages: + flush + flush the cache device. The message returns successfully + if the cache device was flushed without an error + flush_on_suspend + flush the cache device on next suspend. Use this message + when you are going to remove the cache device. The proper + sequence for removing the cache device is: + 1. send the "flush_on_suspend" message + 2. load an inactive table with a linear target that maps + to the underlying device + 3. suspend the device + 4. ask for status and verify that there are no errors + 5. resume the device, so that it will use the linear + target + 6. the cache device is now inactive and it can be deleted diff --git a/Documentation/devicetree/bindings/arm/amlogic.txt b/Documentation/devicetree/bindings/arm/amlogic.txt index f747f47922c5..69880560c0f0 100644 --- a/Documentation/devicetree/bindings/arm/amlogic.txt +++ b/Documentation/devicetree/bindings/arm/amlogic.txt @@ -25,6 +25,10 @@ Boards with the Amlogic Meson8b SoC shall have the following properties: Required root node property: compatible: "amlogic,meson8b"; +Boards with the Amlogic Meson8m2 SoC shall have the following properties: + Required root node property: + compatible: "amlogic,meson8m2"; + Boards with the Amlogic Meson GXBaby SoC shall have the following properties: Required root node property: compatible: "amlogic,meson-gxbb"; @@ -54,6 +58,8 @@ Board compatible values (alphabetically, grouped by SoC): - "hardkernel,odroid-c1" (Meson8b) - "tronfy,mxq" (Meson8b) + - "tronsmart,mxiii-plus" (Meson8m2) + - "amlogic,p200" (Meson gxbb) - "amlogic,p201" (Meson gxbb) - "friendlyarm,nanopi-k2" (Meson gxbb) diff --git a/Documentation/devicetree/bindings/arm/bcm/brcm,bcm2835.txt b/Documentation/devicetree/bindings/arm/bcm/brcm,bcm2835.txt index 3e3efa046ac5..1e3e29a545e2 100644 --- a/Documentation/devicetree/bindings/arm/bcm/brcm,bcm2835.txt +++ b/Documentation/devicetree/bindings/arm/bcm/brcm,bcm2835.txt @@ -34,6 +34,10 @@ Raspberry Pi 3 Model B Required root node properties: compatible = "raspberrypi,3-model-b", "brcm,bcm2837"; +Raspberry Pi 3 Model B+ +Required root node properties: +compatible = "raspberrypi,3-model-b-plus", "brcm,bcm2837"; + Raspberry Pi Compute Module Required root node properties: compatible = "raspberrypi,compute-module", "brcm,bcm2835"; diff --git a/Documentation/devicetree/bindings/arm/mediatek/mediatek,g3dsys.txt b/Documentation/devicetree/bindings/arm/mediatek/mediatek,g3dsys.txt new file mode 100644 index 000000000000..7de43bf41fdc --- /dev/null +++ b/Documentation/devicetree/bindings/arm/mediatek/mediatek,g3dsys.txt @@ -0,0 +1,30 @@ +MediaTek g3dsys controller +============================ + +The MediaTek g3dsys controller provides various clocks and reset controller to +the GPU. + +Required Properties: + +- compatible: Should be: + - "mediatek,mt2701-g3dsys", "syscon": + for MT2701 SoC + - "mediatek,mt7623-g3dsys", "mediatek,mt2701-g3dsys", "syscon": + for MT7623 SoC +- #clock-cells: Must be 1 +- #reset-cells: Must be 1 + +The g3dsys controller uses the common clk binding from +Documentation/devicetree/bindings/clock/clock-bindings.txt +The available clocks are defined in dt-bindings/clock/mt*-clk.h. + +Example: + +g3dsys: clock-controller@13000000 { + compatible = "mediatek,mt7623-g3dsys", + "mediatek,mt2701-g3dsys", + "syscon"; + reg = <0 0x13000000 0 0x200>; + #clock-cells = <1>; + #reset-cells = <1>; +}; diff --git a/Documentation/devicetree/bindings/arm/samsung/samsung-boards.txt b/Documentation/devicetree/bindings/arm/samsung/samsung-boards.txt index 14510b215480..bdadc3da9556 100644 --- a/Documentation/devicetree/bindings/arm/samsung/samsung-boards.txt +++ b/Documentation/devicetree/bindings/arm/samsung/samsung-boards.txt @@ -21,8 +21,6 @@ Required root node properties: - "samsung,smdk5420" - for Exynos5420-based Samsung SMDK5420 eval board. - "samsung,tm2" - for Exynos5433-based Samsung TM2 board. - "samsung,tm2e" - for Exynos5433-based Samsung TM2E board. - - "samsung,sd5v1" - for Exynos5440-based Samsung board. - - "samsung,ssdk5440" - for Exynos5440-based Samsung board. * Other companies Exynos SoC based * FriendlyARM diff --git a/Documentation/devicetree/bindings/arm/shmobile.txt b/Documentation/devicetree/bindings/arm/shmobile.txt index d3d1df97834f..d8cf740132c6 100644 --- a/Documentation/devicetree/bindings/arm/shmobile.txt +++ b/Documentation/devicetree/bindings/arm/shmobile.txt @@ -21,6 +21,8 @@ SoCs: compatible = "renesas,r8a7744" - RZ/G1E (R8A77450) compatible = "renesas,r8a7745" + - RZ/G1C (R8A77470) + compatible = "renesas,r8a77470" - R-Car M1A (R8A77781) compatible = "renesas,r8a7778" - R-Car H1 (R8A77790) @@ -45,6 +47,8 @@ SoCs: compatible = "renesas,r8a77970" - R-Car V3H (R8A77980) compatible = "renesas,r8a77980" + - R-Car E3 (R8A77990) + compatible = "renesas,r8a77990" - R-Car D3 (R8A77995) compatible = "renesas,r8a77995" @@ -67,6 +71,8 @@ Boards: compatible = "renesas,draak", "renesas,r8a77995" - Eagle (RTP0RC77970SEB0010S) compatible = "renesas,eagle", "renesas,r8a77970" + - Ebisu (RTP0RC77990SEB0010S) + compatible = "renesas,ebisu", "renesas,r8a77990" - Genmai (RTK772100BC00000BR) compatible = "renesas,genmai", "renesas,r7s72100" - GR-Peach (X28A-M01-E/F) @@ -78,6 +84,8 @@ Boards: compatible = "renesas,h3ulcb", "renesas,r8a7795" - Henninger compatible = "renesas,henninger", "renesas,r8a7791" + - iWave Systems RZ/G1C Single Board Computer (iW-RainboW-G23S) + compatible = "iwave,g23s", "renesas,r8a77470" - iWave Systems RZ/G1E SODIMM SOM Development Platform (iW-RainboW-G22D) compatible = "iwave,g22d", "iwave,g22m", "renesas,r8a7745" - iWave Systems RZ/G1E SODIMM System On Module (iW-RainboW-G22M-SM) @@ -108,7 +116,7 @@ Boards: compatible = "renesas,salvator-x", "renesas,r8a7795" - Salvator-X (RTP0RC7796SIPB0011S) compatible = "renesas,salvator-x", "renesas,r8a7796" - - Salvator-X (RTP0RC7796SIPB0011S (M3N)) + - Salvator-X (RTP0RC7796SIPB0011S (M3-N)) compatible = "renesas,salvator-x", "renesas,r8a77965" - Salvator-XS (Salvator-X 2nd version, RTP0RC7795SIPB0012S) compatible = "renesas,salvator-xs", "renesas,r8a7795" @@ -124,6 +132,8 @@ Boards: compatible = "renesas,sk-rzg1m", "renesas,r8a7743" - Stout (ADAS Starterkit, Y-R-CAR-ADAS-SKH2-BOARD) compatible = "renesas,stout", "renesas,r8a7790" + - V3HSK (Y-ASK-RCAR-V3H-WS10) + compatible = "renesas,v3hsk", "renesas,r8a77980" - V3MSK (Y-ASK-RCAR-V3M-WS10) compatible = "renesas,v3msk", "renesas,r8a77970" - Wheat (RTP0RC7792ASKB0000JE) diff --git a/Documentation/devicetree/bindings/arm/stm32/stm32-syscon.txt b/Documentation/devicetree/bindings/arm/stm32/stm32-syscon.txt new file mode 100644 index 000000000000..99980aee26e5 --- /dev/null +++ b/Documentation/devicetree/bindings/arm/stm32/stm32-syscon.txt @@ -0,0 +1,14 @@ +STMicroelectronics STM32 Platforms System Controller + +Properties: + - compatible : should contain two values. First value must be : + - " st,stm32mp157-syscfg " - for stm32mp157 based SoCs, + second value must be always "syscon". + - reg : offset and length of the register set. + + Example: + syscfg: syscon@50020000 { + compatible = "st,stm32mp157-syscfg", "syscon"; + reg = <0x50020000 0x400>; + }; + diff --git a/Documentation/devicetree/bindings/arm/stm32.txt b/Documentation/devicetree/bindings/arm/stm32/stm32.txt similarity index 100% rename from Documentation/devicetree/bindings/arm/stm32.txt rename to Documentation/devicetree/bindings/arm/stm32/stm32.txt diff --git a/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra20-mc.txt b/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra20-mc.txt deleted file mode 100644 index f9632bacbd04..000000000000 --- a/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra20-mc.txt +++ /dev/null @@ -1,16 +0,0 @@ -NVIDIA Tegra20 MC(Memory Controller) - -Required properties: -- compatible : "nvidia,tegra20-mc" -- reg : Should contain 2 register ranges(address and length); see the - example below. Note that the MC registers are interleaved with the - GART registers, and hence must be represented as multiple ranges. -- interrupts : Should contain MC General interrupt. - -Example: - memory-controller@7000f000 { - compatible = "nvidia,tegra20-mc"; - reg = <0x7000f000 0x024 - 0x7000f03c 0x3c4>; - interrupts = <0 77 0x04>; - }; diff --git a/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra30-mc.txt b/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra30-mc.txt deleted file mode 100644 index bdf1a612422b..000000000000 --- a/Documentation/devicetree/bindings/arm/tegra/nvidia,tegra30-mc.txt +++ /dev/null @@ -1,18 +0,0 @@ -NVIDIA Tegra30 MC(Memory Controller) - -Required properties: -- compatible : "nvidia,tegra30-mc" -- reg : Should contain 4 register ranges(address and length); see the - example below. Note that the MC registers are interleaved with the - SMMU registers, and hence must be represented as multiple ranges. -- interrupts : Should contain MC General interrupt. - -Example: - memory-controller { - compatible = "nvidia,tegra30-mc"; - reg = <0x7000f000 0x010 - 0x7000f03c 0x1b4 - 0x7000f200 0x028 - 0x7000f284 0x17c>; - interrupts = <0 77 0x04>; - }; diff --git a/Documentation/devicetree/bindings/arm/ux500/boards.txt b/Documentation/devicetree/bindings/arm/ux500/boards.txt index 7334c24625fc..0fa429534f49 100644 --- a/Documentation/devicetree/bindings/arm/ux500/boards.txt +++ b/Documentation/devicetree/bindings/arm/ux500/boards.txt @@ -26,7 +26,7 @@ interrupt-controller: see binding for interrupt-controller/arm,gic.txt timer: - see binding for arm/twd.txt + see binding for timer/arm,twd.txt clocks: see binding for clocks/ux500.txt diff --git a/Documentation/devicetree/bindings/bus/ti-sysc.txt b/Documentation/devicetree/bindings/bus/ti-sysc.txt index 2957a9ae291f..d8ed5b780ed9 100644 --- a/Documentation/devicetree/bindings/bus/ti-sysc.txt +++ b/Documentation/devicetree/bindings/bus/ti-sysc.txt @@ -79,7 +79,11 @@ Optional properties: mode as for example omap4 L4_CFG_CLKCTRL - clock-names should contain at least "fck", and optionally also "ick" - depending on the SoC and the interconnect target module + depending on the SoC and the interconnect target module, + some interconnect target modules also need additional + optional clocks that can be specified as listed in TRM + for the related CLKCTRL register bits 8 to 15 such as + "dbclk" or "clk32k" depending on their role - ti,hwmods optional TI interconnect module name to use legacy hwmod platform data diff --git a/Documentation/devicetree/bindings/clock/actions,s900-cmu.txt b/Documentation/devicetree/bindings/clock/actions,s900-cmu.txt new file mode 100644 index 000000000000..93e4fb827cd6 --- /dev/null +++ b/Documentation/devicetree/bindings/clock/actions,s900-cmu.txt @@ -0,0 +1,47 @@ +* Actions S900 Clock Management Unit (CMU) + +The Actions S900 clock management unit generates and supplies clock to various +controllers within the SoC. The clock binding described here is applicable to +S900 SoC. + +Required Properties: + +- compatible: should be "actions,s900-cmu" +- reg: physical base address of the controller and length of memory mapped + region. +- clocks: Reference to the parent clocks ("hosc", "losc") +- #clock-cells: should be 1. + +Each clock is assigned an identifier, and client nodes can use this identifier +to specify the clock which they consume. + +All available clocks are defined as preprocessor macros in +dt-bindings/clock/actions,s900-cmu.h header and can be used in device +tree sources. + +External clocks: + +The hosc clock used as input for the plls is generated outside the SoC. It is +expected that it is defined using standard clock bindings as "hosc". + +Actions S900 CMU also requires one more clock: + - "losc" - internal low frequency oscillator + +Example: Clock Management Unit node: + + cmu: clock-controller@e0160000 { + compatible = "actions,s900-cmu"; + reg = <0x0 0xe0160000 0x0 0x1000>; + clocks = <&hosc>, <&losc>; + #clock-cells = <1>; + }; + +Example: UART controller node that consumes clock generated by the clock +management unit: + + uart: serial@e012a000 { + compatible = "actions,s900-uart", "actions,owl-uart"; + reg = <0x0 0xe012a000 0x0 0x2000>; + interrupts = ; + clocks = <&cmu CLK_UART5>; + }; diff --git a/Documentation/devicetree/bindings/clock/amlogic,gxbb-aoclkc.txt b/Documentation/devicetree/bindings/clock/amlogic,gxbb-aoclkc.txt index 786dc39ca904..3a880528030e 100644 --- a/Documentation/devicetree/bindings/clock/amlogic,gxbb-aoclkc.txt +++ b/Documentation/devicetree/bindings/clock/amlogic,gxbb-aoclkc.txt @@ -9,6 +9,7 @@ Required Properties: - GXBB (S905) : "amlogic,meson-gxbb-aoclkc" - GXL (S905X, S905D) : "amlogic,meson-gxl-aoclkc" - GXM (S912) : "amlogic,meson-gxm-aoclkc" + - AXG (A113D, A113X) : "amlogic,meson-axg-aoclkc" followed by the common "amlogic,meson-gx-aoclkc" - #clock-cells: should be 1. diff --git a/Documentation/devicetree/bindings/clock/amlogic,gxbb-clkc.txt b/Documentation/devicetree/bindings/clock/amlogic,gxbb-clkc.txt index e2b377ed6f91..e950599566a9 100644 --- a/Documentation/devicetree/bindings/clock/amlogic,gxbb-clkc.txt +++ b/Documentation/devicetree/bindings/clock/amlogic,gxbb-clkc.txt @@ -10,9 +10,6 @@ Required Properties: "amlogic,gxl-clkc" for GXL and GXM SoC, "amlogic,axg-clkc" for AXG SoC. -- reg: physical base address of the clock controller and length of memory - mapped region. - - #clock-cells: should be 1. Each clock is assigned an identifier and client nodes can use this identifier @@ -20,13 +17,22 @@ to specify the clock which they consume. All available clocks are defined as preprocessor macros in the dt-bindings/clock/gxbb-clkc.h header and can be used in device tree sources. +Parent node should have the following properties : +- compatible: "syscon", "simple-mfd, and "amlogic,meson-gx-hhi-sysctrl" or + "amlogic,meson-axg-hhi-sysctrl" +- reg: base address and size of the HHI system control register space. + Example: Clock controller node: - clkc: clock-controller@c883c000 { +sysctrl: system-controller@0 { + compatible = "amlogic,meson-gx-hhi-sysctrl", "syscon", "simple-mfd"; + reg = <0 0 0 0x400>; + + clkc: clock-controller { #clock-cells = <1>; compatible = "amlogic,gxbb-clkc"; - reg = <0x0 0xc883c000 0x0 0x3db>; }; +}; Example: UART controller node that consumes the clock generated by the clock controller: diff --git a/Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt b/Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt index f8e4a93466cb..ab730ea0a560 100644 --- a/Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt +++ b/Documentation/devicetree/bindings/clock/brcm,iproc-clocks.txt @@ -276,36 +276,38 @@ These clock IDs are defined in: clk_ts_500_ref genpll2 2 BCM_SR_GENPLL2_TS_500_REF_CLK clk_125_nitro genpll2 3 BCM_SR_GENPLL2_125_NITRO_CLK clk_chimp genpll2 4 BCM_SR_GENPLL2_CHIMP_CLK - clk_nic_flash genpll2 5 BCM_SR_GENPLL2_NIC_FLASH + clk_nic_flash genpll2 5 BCM_SR_GENPLL2_NIC_FLASH_CLK + clk_fs genpll2 6 BCM_SR_GENPLL2_FS_CLK genpll3 crystal 0 BCM_SR_GENPLL3 clk_hsls genpll3 1 BCM_SR_GENPLL3_HSLS_CLK clk_sdio genpll3 2 BCM_SR_GENPLL3_SDIO_CLK genpll4 crystal 0 BCM_SR_GENPLL4 - ccn genpll4 1 BCM_SR_GENPLL4_CCN_CLK + clk_ccn genpll4 1 BCM_SR_GENPLL4_CCN_CLK clk_tpiu_pll genpll4 2 BCM_SR_GENPLL4_TPIU_PLL_CLK - noc_clk genpll4 3 BCM_SR_GENPLL4_NOC_CLK + clk_noc genpll4 3 BCM_SR_GENPLL4_NOC_CLK clk_chclk_fs4 genpll4 4 BCM_SR_GENPLL4_CHCLK_FS4_CLK clk_bridge_fscpu genpll4 5 BCM_SR_GENPLL4_BRIDGE_FSCPU_CLK - genpll5 crystal 0 BCM_SR_GENPLL5 - fs4_hf_clk genpll5 1 BCM_SR_GENPLL5_FS4_HF_CLK - crypto_ae_clk genpll5 2 BCM_SR_GENPLL5_CRYPTO_AE_CLK - raid_ae_clk genpll5 3 BCM_SR_GENPLL5_RAID_AE_CLK + clk_fs4_hf genpll5 1 BCM_SR_GENPLL5_FS4_HF_CLK + clk_crypto_ae genpll5 2 BCM_SR_GENPLL5_CRYPTO_AE_CLK + clk_raid_ae genpll5 3 BCM_SR_GENPLL5_RAID_AE_CLK genpll6 crystal 0 BCM_SR_GENPLL6 - 48_usb genpll6 1 BCM_SR_GENPLL6_48_USB_CLK + clk_48_usb genpll6 1 BCM_SR_GENPLL6_48_USB_CLK lcpll0 crystal 0 BCM_SR_LCPLL0 clk_sata_refp lcpll0 1 BCM_SR_LCPLL0_SATA_REFP_CLK clk_sata_refn lcpll0 2 BCM_SR_LCPLL0_SATA_REFN_CLK - clk_usb_ref lcpll0 3 BCM_SR_LCPLL0_USB_REF_CLK - sata_refpn lcpll0 3 BCM_SR_LCPLL0_SATA_REFPN_CLK + clk_sata_350 lcpll0 3 BCM_SR_LCPLL0_SATA_350_CLK + clk_sata_500 lcpll0 4 BCM_SR_LCPLL0_SATA_500_CLK lcpll1 crystal 0 BCM_SR_LCPLL1 - wan lcpll1 1 BCM_SR_LCPLL0_WAN_CLK + clk_wan lcpll1 1 BCM_SR_LCPLL1_WAN_CLK + clk_usb_ref lcpll1 2 BCM_SR_LCPLL1_USB_REF_CLK + clk_crmu_ts lcpll1 3 BCM_SR_LCPLL1_CRMU_TS_CLK lcpll_pcie crystal 0 BCM_SR_LCPLL_PCIE - pcie_phy_ref lcpll1 1 BCM_SR_LCPLL_PCIE_PHY_REF_CLK + clk_pcie_phy_ref lcpll1 1 BCM_SR_LCPLL_PCIE_PHY_REF_CLK diff --git a/Documentation/devicetree/bindings/clock/nuvoton,npcm750-clk.txt b/Documentation/devicetree/bindings/clock/nuvoton,npcm750-clk.txt new file mode 100644 index 000000000000..f82064546d11 --- /dev/null +++ b/Documentation/devicetree/bindings/clock/nuvoton,npcm750-clk.txt @@ -0,0 +1,100 @@ +* Nuvoton NPCM7XX Clock Controller + +Nuvoton Poleg BMC NPCM7XX contains an integrated clock controller, which +generates and supplies clocks to all modules within the BMC. + +External clocks: + +There are six fixed clocks that are generated outside the BMC. All clocks are of +a known fixed value that cannot be changed. clk_refclk, clk_mcbypck and +clk_sysbypck are inputs to the clock controller. +clk_rg1refck, clk_rg2refck and clk_xin are external clocks suppling the +network. They are set on the device tree, but not used by the clock module. The +network devices use them directly. +Example can be found below. + +All available clocks are defined as preprocessor macros in: +dt-bindings/clock/nuvoton,npcm7xx-clock.h +and can be reused as DT sources. + +Required Properties of clock controller: + + - compatible: "nuvoton,npcm750-clk" : for clock controller of Nuvoton + Poleg BMC NPCM750 + + - reg: physical base address of the clock controller and length of + memory mapped region. + + - #clock-cells: should be 1. + +Example: Clock controller node: + + clk: clock-controller@f0801000 { + compatible = "nuvoton,npcm750-clk"; + #clock-cells = <1>; + reg = <0xf0801000 0x1000>; + clock-names = "refclk", "sysbypck", "mcbypck"; + clocks = <&clk_refclk>, <&clk_sysbypck>, <&clk_mcbypck>; + }; + +Example: Required external clocks for network: + + /* external reference clock */ + clk_refclk: clk-refclk { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <25000000>; + clock-output-names = "refclk"; + }; + + /* external reference clock for cpu. float in normal operation */ + clk_sysbypck: clk-sysbypck { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <800000000>; + clock-output-names = "sysbypck"; + }; + + /* external reference clock for MC. float in normal operation */ + clk_mcbypck: clk-mcbypck { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <800000000>; + clock-output-names = "mcbypck"; + }; + + /* external clock signal rg1refck, supplied by the phy */ + clk_rg1refck: clk-rg1refck { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <125000000>; + clock-output-names = "clk_rg1refck"; + }; + + /* external clock signal rg2refck, supplied by the phy */ + clk_rg2refck: clk-rg2refck { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <125000000>; + clock-output-names = "clk_rg2refck"; + }; + + clk_xin: clk-xin { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <50000000>; + clock-output-names = "clk_xin"; + }; + + +Example: GMAC controller node that consumes two clocks: a generated clk by the +clock controller and a fixed clock from DT (clk_rg1refck). + + ethernet0: ethernet@f0802000 { + compatible = "snps,dwmac"; + reg = <0xf0802000 0x2000>; + interrupts = <0 14 4>; + interrupt-names = "macirq"; + clocks = <&clk_rg1refck>, <&clk NPCM7XX_CLK_AHB>; + clock-names = "stmmaceth", "clk_gmac"; + }; diff --git a/Documentation/devicetree/bindings/clock/qcom,gcc.txt b/Documentation/devicetree/bindings/clock/qcom,gcc.txt index 551d03be9665..664ea1fd6c76 100644 --- a/Documentation/devicetree/bindings/clock/qcom,gcc.txt +++ b/Documentation/devicetree/bindings/clock/qcom,gcc.txt @@ -17,7 +17,9 @@ Required properties : "qcom,gcc-msm8974pro-ac" "qcom,gcc-msm8994" "qcom,gcc-msm8996" + "qcom,gcc-msm8998" "qcom,gcc-mdm9615" + "qcom,gcc-sdm845" - reg : shall contain base register location and length - #clock-cells : shall contain 1 diff --git a/Documentation/devicetree/bindings/clock/qcom,rpmh-clk.txt b/Documentation/devicetree/bindings/clock/qcom,rpmh-clk.txt new file mode 100644 index 000000000000..3c007653da31 --- /dev/null +++ b/Documentation/devicetree/bindings/clock/qcom,rpmh-clk.txt @@ -0,0 +1,22 @@ +Qualcomm Technologies, Inc. RPMh Clocks +------------------------------------------------------- + +Resource Power Manager Hardened (RPMh) manages shared resources on +some Qualcomm Technologies Inc. SoCs. It accepts clock requests from +other hardware subsystems via RSC to control clocks. + +Required properties : +- compatible : shall contain "qcom,sdm845-rpmh-clk" + +- #clock-cells : must contain 1 + +Example : + +#include + + &apps_rsc { + rpmhcc: clock-controller { + compatible = "qcom,sdm845-rpmh-clk"; + #clock-cells = <1>; + }; + }; diff --git a/Documentation/devicetree/bindings/clock/qcom,videocc.txt b/Documentation/devicetree/bindings/clock/qcom,videocc.txt new file mode 100644 index 000000000000..e7c035afa778 --- /dev/null +++ b/Documentation/devicetree/bindings/clock/qcom,videocc.txt @@ -0,0 +1,19 @@ +Qualcomm Video Clock & Reset Controller Binding +----------------------------------------------- + +Required properties : +- compatible : shall contain "qcom,sdm845-videocc" +- reg : shall contain base register location and length +- #clock-cells : from common clock binding, shall contain 1. +- #power-domain-cells : from generic power domain binding, shall contain 1. + +Optional properties : +- #reset-cells : from common reset binding, shall contain 1. + +Example: + videocc: clock-controller@ab00000 { + compatible = "qcom,sdm845-videocc"; + reg = <0xab00000 0x10000>; + #clock-cells = <1>; + #power-domain-cells = <1>; + }; diff --git a/Documentation/devicetree/bindings/clock/renesas,cpg-mssr.txt b/Documentation/devicetree/bindings/clock/renesas,cpg-mssr.txt index 773a5226342f..db542abadb75 100644 --- a/Documentation/devicetree/bindings/clock/renesas,cpg-mssr.txt +++ b/Documentation/devicetree/bindings/clock/renesas,cpg-mssr.txt @@ -15,6 +15,7 @@ Required Properties: - compatible: Must be one of: - "renesas,r8a7743-cpg-mssr" for the r8a7743 SoC (RZ/G1M) - "renesas,r8a7745-cpg-mssr" for the r8a7745 SoC (RZ/G1E) + - "renesas,r8a77470-cpg-mssr" for the r8a77470 SoC (RZ/G1C) - "renesas,r8a7790-cpg-mssr" for the r8a7790 SoC (R-Car H2) - "renesas,r8a7791-cpg-mssr" for the r8a7791 SoC (R-Car M2-W) - "renesas,r8a7792-cpg-mssr" for the r8a7792 SoC (R-Car V2H) @@ -25,6 +26,7 @@ Required Properties: - "renesas,r8a77965-cpg-mssr" for the r8a77965 SoC (R-Car M3-N) - "renesas,r8a77970-cpg-mssr" for the r8a77970 SoC (R-Car V3M) - "renesas,r8a77980-cpg-mssr" for the r8a77980 SoC (R-Car V3H) + - "renesas,r8a77990-cpg-mssr" for the r8a77990 SoC (R-Car E3) - "renesas,r8a77995-cpg-mssr" for the r8a77995 SoC (R-Car D3) - reg: Base address and length of the memory resource used by the CPG/MSSR @@ -33,10 +35,12 @@ Required Properties: - clocks: References to external parent clocks, one entry for each entry in clock-names - clock-names: List of external parent clock names. Valid names are: - - "extal" (r8a7743, r8a7745, r8a7790, r8a7791, r8a7792, r8a7793, r8a7794, - r8a7795, r8a7796, r8a77965, r8a77970, r8a77980, r8a77995) + - "extal" (r8a7743, r8a7745, r8a77470, r8a7790, r8a7791, r8a7792, + r8a7793, r8a7794, r8a7795, r8a7796, r8a77965, r8a77970, + r8a77980, r8a77990, r8a77995) - "extalr" (r8a7795, r8a7796, r8a77965, r8a77970, r8a77980) - - "usb_extal" (r8a7743, r8a7745, r8a7790, r8a7791, r8a7793, r8a7794) + - "usb_extal" (r8a7743, r8a7745, r8a77470, r8a7790, r8a7791, r8a7793, + r8a7794) - #clock-cells: Must be 2 - For CPG core clocks, the two clock specifier cells must be "CPG_CORE" diff --git a/Documentation/devicetree/bindings/clock/rockchip.txt b/Documentation/devicetree/bindings/clock/rockchip.txt deleted file mode 100644 index 22f6769e5d4a..000000000000 --- a/Documentation/devicetree/bindings/clock/rockchip.txt +++ /dev/null @@ -1,77 +0,0 @@ -Device Tree Clock bindings for arch-rockchip - -This binding uses the common clock binding[1]. - -[1] Documentation/devicetree/bindings/clock/clock-bindings.txt - -== Gate clocks == - -These bindings are deprecated! -Please use the soc specific CRU bindings instead. - -The gate registers form a continuos block which makes the dt node -structure a matter of taste, as either all gates can be put into -one gate clock spanning all registers or they can be divided into -the 10 individual gates containing 16 clocks each. -The code supports both approaches. - -Required properties: -- compatible : "rockchip,rk2928-gate-clk" -- reg : shall be the control register address(es) for the clock. -- #clock-cells : from common clock binding; shall be set to 1 -- clock-output-names : the corresponding gate names that the clock controls -- clocks : should contain the parent clock for each individual gate, - therefore the number of clocks elements should match the number of - clock-output-names - -Example using multiple gate clocks: - - clk_gates0: gate-clk@200000d0 { - compatible = "rockchip,rk2928-gate-clk"; - reg = <0x200000d0 0x4>; - clocks = <&dummy>, <&dummy>, - <&dummy>, <&dummy>, - <&dummy>, <&dummy>, - <&dummy>, <&dummy>, - <&dummy>, <&dummy>, - <&dummy>, <&dummy>, - <&dummy>, <&dummy>, - <&dummy>, <&dummy>; - - clock-output-names = - "gate_core_periph", "gate_cpu_gpll", - "gate_ddrphy", "gate_aclk_cpu", - "gate_hclk_cpu", "gate_pclk_cpu", - "gate_atclk_cpu", "gate_i2s0", - "gate_i2s0_frac", "gate_i2s1", - "gate_i2s1_frac", "gate_i2s2", - "gate_i2s2_frac", "gate_spdif", - "gate_spdif_frac", "gate_testclk"; - - #clock-cells = <1>; - }; - - clk_gates1: gate-clk@200000d4 { - compatible = "rockchip,rk2928-gate-clk"; - reg = <0x200000d4 0x4>; - clocks = <&xin24m>, <&xin24m>, - <&xin24m>, <&dummy>, - <&dummy>, <&xin24m>, - <&xin24m>, <&dummy>, - <&xin24m>, <&dummy>, - <&xin24m>, <&dummy>, - <&xin24m>, <&dummy>, - <&xin24m>, <&dummy>; - - clock-output-names = - "gate_timer0", "gate_timer1", - "gate_timer2", "gate_jtag", - "gate_aclk_lcdc1_src", "gate_otgphy0", - "gate_otgphy1", "gate_ddr_gpll", - "gate_uart0", "gate_frac_uart0", - "gate_uart1", "gate_frac_uart1", - "gate_uart2", "gate_frac_uart2", - "gate_uart3", "gate_frac_uart3"; - - #clock-cells = <1>; - }; diff --git a/Documentation/devicetree/bindings/clock/st/st,clkgen.txt b/Documentation/devicetree/bindings/clock/st/st,clkgen.txt index 7364953d0d0b..45ac19bfa0a9 100644 --- a/Documentation/devicetree/bindings/clock/st/st,clkgen.txt +++ b/Documentation/devicetree/bindings/clock/st/st,clkgen.txt @@ -31,10 +31,10 @@ This binding uses the common clock binding[1]. Each subnode should use the binding described in [2]..[7] [1] Documentation/devicetree/bindings/clock/clock-bindings.txt -[3] Documentation/devicetree/bindings/clock/st,clkgen-mux.txt -[4] Documentation/devicetree/bindings/clock/st,clkgen-pll.txt -[7] Documentation/devicetree/bindings/clock/st,quadfs.txt -[8] Documentation/devicetree/bindings/clock/st,flexgen.txt +[3] Documentation/devicetree/bindings/clock/st/st,clkgen-mux.txt +[4] Documentation/devicetree/bindings/clock/st/st,clkgen-pll.txt +[7] Documentation/devicetree/bindings/clock/st/st,quadfs.txt +[8] Documentation/devicetree/bindings/clock/st/st,flexgen.txt Required properties: diff --git a/Documentation/devicetree/bindings/clock/sunxi-ccu.txt b/Documentation/devicetree/bindings/clock/sunxi-ccu.txt index 460ef27b1008..47d2e902ced4 100644 --- a/Documentation/devicetree/bindings/clock/sunxi-ccu.txt +++ b/Documentation/devicetree/bindings/clock/sunxi-ccu.txt @@ -21,6 +21,7 @@ Required properties : - "allwinner,sun50i-a64-r-ccu" - "allwinner,sun50i-h5-ccu" - "allwinner,sun50i-h6-ccu" + - "allwinner,sun50i-h6-r-ccu" - "nextthing,gr8-ccu" - reg: Must contain the registers base address and length @@ -35,7 +36,7 @@ Required properties : For the main CCU on H6, one more clock is needed: - "iosc": the SoC's internal frequency oscillator -For the PRCM CCUs on A83T/H3/A64, two more clocks are needed: +For the PRCM CCUs on A83T/H3/A64/H6, two more clocks are needed: - "pll-periph": the SoC's peripheral PLL from the main CCU - "iosc": the SoC's internal frequency oscillator diff --git a/Documentation/devicetree/bindings/clock/ti/gate.txt b/Documentation/devicetree/bindings/clock/ti/gate.txt index 03f8fdee62a7..56d603c1f716 100644 --- a/Documentation/devicetree/bindings/clock/ti/gate.txt +++ b/Documentation/devicetree/bindings/clock/ti/gate.txt @@ -10,7 +10,7 @@ will be controlled instead and the corresponding hw-ops for that is used. [1] Documentation/devicetree/bindings/clock/clock-bindings.txt -[2] Documentation/devicetree/bindings/clock/gate-clock.txt +[2] Documentation/devicetree/bindings/clock/gpio-gate-clock.txt [3] Documentation/devicetree/bindings/clock/ti/clockdomain.txt Required properties: diff --git a/Documentation/devicetree/bindings/clock/ti/interface.txt b/Documentation/devicetree/bindings/clock/ti/interface.txt index 3111a409fea6..3f4704040140 100644 --- a/Documentation/devicetree/bindings/clock/ti/interface.txt +++ b/Documentation/devicetree/bindings/clock/ti/interface.txt @@ -9,7 +9,7 @@ companion clock finding (match corresponding functional gate clock) and hardware autoidle enable / disable. [1] Documentation/devicetree/bindings/clock/clock-bindings.txt -[2] Documentation/devicetree/bindings/clock/gate-clock.txt +[2] Documentation/devicetree/bindings/clock/gpio-gate-clock.txt Required properties: - compatible : shall be one of: diff --git a/Documentation/devicetree/bindings/cpufreq/cpufreq-mediatek.txt b/Documentation/devicetree/bindings/cpufreq/cpufreq-mediatek.txt index d36f07e0a2bb..0551c78619de 100644 --- a/Documentation/devicetree/bindings/cpufreq/cpufreq-mediatek.txt +++ b/Documentation/devicetree/bindings/cpufreq/cpufreq-mediatek.txt @@ -8,7 +8,7 @@ Required properties: "intermediate" - A parent of "cpu" clock which is used as "intermediate" clock source (usually MAINPLL) when the original CPU PLL is under transition and not stable yet. - Please refer to Documentation/devicetree/bindings/clk/clock-bindings.txt for + Please refer to Documentation/devicetree/bindings/clock/clock-bindings.txt for generic clock consumer properties. - operating-points-v2: Please refer to Documentation/devicetree/bindings/opp/opp.txt for detail. diff --git a/Documentation/devicetree/bindings/devfreq/rk3399_dmc.txt b/Documentation/devicetree/bindings/devfreq/rk3399_dmc.txt index d6d2833482c9..fc2bcbe26b1e 100644 --- a/Documentation/devicetree/bindings/devfreq/rk3399_dmc.txt +++ b/Documentation/devicetree/bindings/devfreq/rk3399_dmc.txt @@ -12,7 +12,7 @@ Required properties: - clocks: Phandles for clock specified in "clock-names" property - clock-names : The name of clock used by the DFI, must be "pclk_ddr_mon"; -- operating-points-v2: Refer to Documentation/devicetree/bindings/power/opp.txt +- operating-points-v2: Refer to Documentation/devicetree/bindings/opp/opp.txt for details. - center-supply: DMC supply node. - status: Marks the node enabled/disabled. diff --git a/Documentation/devicetree/bindings/display/bridge/adi,adv7511.txt b/Documentation/devicetree/bindings/display/bridge/adi,adv7511.txt index 0047b1394c70..2c887536258c 100644 --- a/Documentation/devicetree/bindings/display/bridge/adi,adv7511.txt +++ b/Documentation/devicetree/bindings/display/bridge/adi,adv7511.txt @@ -14,7 +14,13 @@ Required properties: "adi,adv7513" "adi,adv7533" -- reg: I2C slave address +- reg: I2C slave addresses + The ADV7511 internal registers are split into four pages exposed through + different I2C addresses, creating four register maps. Each map has it own + I2C address and acts as a standard slave device on the I2C bus. The main + address is mandatory, others are optional and revert to defaults if not + specified. + The ADV7511 supports a large number of input data formats that differ by their color depth, color format, clock mode, bit justification and random @@ -70,6 +76,9 @@ Optional properties: rather than generate its own timings for HDMI output. - clocks: from common clock binding: reference to the CEC clock. - clock-names: from common clock binding: must be "cec". +- reg-names : Names of maps with programmable addresses. + It can contain any map needing a non-default address. + Possible maps names are : "main", "edid", "cec", "packet" Required nodes: @@ -88,7 +97,12 @@ Example adv7511w: hdmi@39 { compatible = "adi,adv7511w"; - reg = <39>; + /* + * The EDID page will be accessible on address 0x66 on the I2C + * bus. All other maps continue to use their default addresses. + */ + reg = <0x39>, <0x66>; + reg-names = "main", "edid"; interrupt-parent = <&gpio3>; interrupts = <29 IRQ_TYPE_EDGE_FALLING>; clocks = <&cec_clock>; diff --git a/Documentation/devicetree/bindings/display/bridge/cdns,dsi.txt b/Documentation/devicetree/bindings/display/bridge/cdns,dsi.txt new file mode 100644 index 000000000000..f5725bb6c61c --- /dev/null +++ b/Documentation/devicetree/bindings/display/bridge/cdns,dsi.txt @@ -0,0 +1,133 @@ +Cadence DSI bridge +================== + +The Cadence DSI bridge is a DPI to DSI bridge supporting up to 4 DSI lanes. + +Required properties: +- compatible: should be set to "cdns,dsi". +- reg: physical base address and length of the controller's registers. +- interrupts: interrupt line connected to the DSI bridge. +- clocks: DSI bridge clocks. +- clock-names: must contain "dsi_p_clk" and "dsi_sys_clk". +- phys: phandle link to the MIPI D-PHY controller. +- phy-names: must contain "dphy". +- #address-cells: must be set to 1. +- #size-cells: must be set to 0. + +Optional properties: +- resets: DSI reset lines. +- reset-names: can contain "dsi_p_rst". + +Required subnodes: +- ports: Ports as described in Documentation/devicetree/bindings/graph.txt. + 2 ports are available: + * port 0: this port is only needed if some of your DSI devices are + controlled through an external bus like I2C or SPI. Can have at + most 4 endpoints. The endpoint number is directly encoding the + DSI virtual channel used by this device. + * port 1: represents the DPI input. + Other ports will be added later to support the new kind of inputs. + +- one subnode per DSI device connected on the DSI bus. Each DSI device should + contain a reg property encoding its virtual channel. + +Cadence DPHY +============ + +Cadence DPHY block. + +Required properties: +- compatible: should be set to "cdns,dphy". +- reg: physical base address and length of the DPHY registers. +- clocks: DPHY reference clocks. +- clock-names: must contain "psm" and "pll_ref". +- #phy-cells: must be set to 0. + + +Example: + dphy0: dphy@fd0e0000{ + compatible = "cdns,dphy"; + reg = <0x0 0xfd0e0000 0x0 0x1000>; + clocks = <&psm_clk>, <&pll_ref_clk>; + clock-names = "psm", "pll_ref"; + #phy-cells = <0>; + }; + + dsi0: dsi@fd0c0000 { + compatible = "cdns,dsi"; + reg = <0x0 0xfd0c0000 0x0 0x1000>; + clocks = <&pclk>, <&sysclk>; + clock-names = "dsi_p_clk", "dsi_sys_clk"; + interrupts = <1>; + phys = <&dphy0>; + phy-names = "dphy"; + #address-cells = <1>; + #size-cells = <0>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + reg = <1>; + dsi0_dpi_input: endpoint { + remote-endpoint = <&xxx_dpi_output>; + }; + }; + }; + + panel: dsi-dev@0 { + compatible = ""; + reg = <0>; + }; + }; + +or + + dsi0: dsi@fd0c0000 { + compatible = "cdns,dsi"; + reg = <0x0 0xfd0c0000 0x0 0x1000>; + clocks = <&pclk>, <&sysclk>; + clock-names = "dsi_p_clk", "dsi_sys_clk"; + interrupts = <1>; + phys = <&dphy1>; + phy-names = "dphy"; + #address-cells = <1>; + #size-cells = <0>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + #address-cells = <1>; + #size-cells = <0>; + + dsi0_output: endpoint@0 { + reg = <0>; + remote-endpoint = <&dsi_panel_input>; + }; + }; + + port@1 { + reg = <1>; + dsi0_dpi_input: endpoint { + remote-endpoint = <&xxx_dpi_output>; + }; + }; + }; + }; + + i2c@xxx { + panel: panel@59 { + compatible = ""; + reg = <0x59>; + + port { + dsi_panel_input: endpoint { + remote-endpoint = <&dsi0_output>; + }; + }; + }; + }; diff --git a/Documentation/devicetree/bindings/display/bridge/renesas,dw-hdmi.txt b/Documentation/devicetree/bindings/display/bridge/renesas,dw-hdmi.txt index 3a72a103a18a..a41d280c3f9f 100644 --- a/Documentation/devicetree/bindings/display/bridge/renesas,dw-hdmi.txt +++ b/Documentation/devicetree/bindings/display/bridge/renesas,dw-hdmi.txt @@ -14,6 +14,7 @@ Required properties: - compatible : Shall contain one or more of - "renesas,r8a7795-hdmi" for R8A7795 (R-Car H3) compatible HDMI TX - "renesas,r8a7796-hdmi" for R8A7796 (R-Car M3-W) compatible HDMI TX + - "renesas,r8a77965-hdmi" for R8A77965 (R-Car M3-N) compatible HDMI TX - "renesas,rcar-gen3-hdmi" for the generic R-Car Gen3 compatible HDMI TX When compatible with generic versions, nodes must list the SoC-specific diff --git a/Documentation/devicetree/bindings/display/bridge/tda998x.txt b/Documentation/devicetree/bindings/display/bridge/tda998x.txt index 24cc2466185a..f5a02f61dd36 100644 --- a/Documentation/devicetree/bindings/display/bridge/tda998x.txt +++ b/Documentation/devicetree/bindings/display/bridge/tda998x.txt @@ -27,7 +27,10 @@ Optional properties: in question is used. The implementation allows one or two DAIs. If two DAIs are defined, they must be of different type. -[1] Documentation/sound/alsa/soc/DAI.txt + - nxp,calib-gpios: calibration GPIO, which must correspond with the + gpio used for the TDA998x interrupt pin. + +[1] Documentation/sound/soc/dai.rst [2] include/dt-bindings/display/tda998x.h Example: diff --git a/Documentation/devicetree/bindings/display/bridge/thine,thc63lvd1024.txt b/Documentation/devicetree/bindings/display/bridge/thine,thc63lvd1024.txt new file mode 100644 index 000000000000..37f0c04d5a28 --- /dev/null +++ b/Documentation/devicetree/bindings/display/bridge/thine,thc63lvd1024.txt @@ -0,0 +1,60 @@ +Thine Electronics THC63LVD1024 LVDS decoder +------------------------------------------- + +The THC63LVD1024 is a dual link LVDS receiver designed to convert LVDS streams +to parallel data outputs. The chip supports single/dual input/output modes, +handling up to two LVDS input streams and up to two digital CMOS/TTL outputs. + +Single or dual operation mode, output data mapping and DDR output modes are +configured through input signals and the chip does not expose any control bus. + +Required properties: +- compatible: Shall be "thine,thc63lvd1024" +- vcc-supply: Power supply for TTL output, TTL CLOCKOUT signal, LVDS input, + PPL and digital circuitry + +Optional properties: +- powerdown-gpios: Power down GPIO signal, pin name "/PDWN". Active low +- oe-gpios: Output enable GPIO signal, pin name "OE". Active high + +The THC63LVD1024 video port connections are modeled according +to OF graph bindings specified by Documentation/devicetree/bindings/graph.txt + +Required video port nodes: +- port@0: First LVDS input port +- port@2: First digital CMOS/TTL parallel output + +Optional video port nodes: +- port@1: Second LVDS input port +- port@3: Second digital CMOS/TTL parallel output + +Example: +-------- + + thc63lvd1024: lvds-decoder { + compatible = "thine,thc63lvd1024"; + + vcc-supply = <®_lvds_vcc>; + powerdown-gpios = <&gpio4 15 GPIO_ACTIVE_LOW>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + + lvds_dec_in_0: endpoint { + remote-endpoint = <&lvds_out>; + }; + }; + + port@2{ + reg = <2>; + + lvds_dec_out_2: endpoint { + remote-endpoint = <&adv7511_in>; + }; + }; + }; + }; diff --git a/Documentation/devicetree/bindings/display/exynos/exynos5433-decon.txt b/Documentation/devicetree/bindings/display/exynos/exynos5433-decon.txt index fc2588292a68..775193e1c641 100644 --- a/Documentation/devicetree/bindings/display/exynos/exynos5433-decon.txt +++ b/Documentation/devicetree/bindings/display/exynos/exynos5433-decon.txt @@ -19,7 +19,8 @@ Required properties: clock-names property. - clock-names: list of clock names sorted in the same order as the clocks property. Must contain "pclk", "aclk_decon", "aclk_smmu_decon0x", - "aclk_xiu_decon0x", "pclk_smmu_decon0x", clk_decon_vclk", + "aclk_xiu_decon0x", "pclk_smmu_decon0x", "aclk_smmu_decon1x", + "aclk_xiu_decon1x", "pclk_smmu_decon1x", clk_decon_vclk", "sclk_decon_eclk" - ports: contains a port which is connected to mic node. address-cells and size-cells must 1 and 0, respectively. @@ -34,10 +35,14 @@ decon: decon@13800000 { clocks = <&cmu_disp CLK_ACLK_DECON>, <&cmu_disp CLK_ACLK_SMMU_DECON0X>, <&cmu_disp CLK_ACLK_XIU_DECON0X>, <&cmu_disp CLK_PCLK_SMMU_DECON0X>, + <&cmu_disp CLK_ACLK_SMMU_DECON1X>, + <&cmu_disp CLK_ACLK_XIU_DECON1X>, + <&cmu_disp CLK_PCLK_SMMU_DECON1X>, <&cmu_disp CLK_SCLK_DECON_VCLK>, <&cmu_disp CLK_SCLK_DECON_ECLK>; clock-names = "aclk_decon", "aclk_smmu_decon0x", "aclk_xiu_decon0x", - "pclk_smmu_decon0x", "sclk_decon_vclk", "sclk_decon_eclk"; + "pclk_smmu_decon0x", "aclk_smmu_decon1x", "aclk_xiu_decon1x", + "pclk_smmu_decon1x", "sclk_decon_vclk", "sclk_decon_eclk"; interrupt-names = "vsync", "lcd_sys"; interrupts = <0 202 0>, <0 203 0>; diff --git a/Documentation/devicetree/bindings/display/renesas,du.txt b/Documentation/devicetree/bindings/display/renesas,du.txt index c9cd17f99702..7c6854bd0a04 100644 --- a/Documentation/devicetree/bindings/display/renesas,du.txt +++ b/Documentation/devicetree/bindings/display/renesas,du.txt @@ -13,6 +13,7 @@ Required Properties: - "renesas,du-r8a7794" for R8A7794 (R-Car E2) compatible DU - "renesas,du-r8a7795" for R8A7795 (R-Car H3) compatible DU - "renesas,du-r8a7796" for R8A7796 (R-Car M3-W) compatible DU + - "renesas,du-r8a77965" for R8A77965 (R-Car M3-N) compatible DU - "renesas,du-r8a77970" for R8A77970 (R-Car V3M) compatible DU - "renesas,du-r8a77995" for R8A77995 (R-Car D3) compatible DU @@ -47,20 +48,21 @@ bindings specified in Documentation/devicetree/bindings/graph.txt. The following table lists for each supported model the port number corresponding to each DU output. - Port0 Port1 Port2 Port3 + Port0 Port1 Port2 Port3 ----------------------------------------------------------------------------- - R8A7743 (RZ/G1M) DPAD 0 LVDS 0 - - - R8A7745 (RZ/G1E) DPAD 0 DPAD 1 - - - R8A7779 (R-Car H1) DPAD 0 DPAD 1 - - - R8A7790 (R-Car H2) DPAD 0 LVDS 0 LVDS 1 - - R8A7791 (R-Car M2-W) DPAD 0 LVDS 0 - - - R8A7792 (R-Car V2H) DPAD 0 DPAD 1 - - - R8A7793 (R-Car M2-N) DPAD 0 LVDS 0 - - - R8A7794 (R-Car E2) DPAD 0 DPAD 1 - - - R8A7795 (R-Car H3) DPAD 0 HDMI 0 HDMI 1 LVDS 0 - R8A7796 (R-Car M3-W) DPAD 0 HDMI 0 LVDS 0 - - R8A77970 (R-Car V3M) DPAD 0 LVDS 0 - - - R8A77995 (R-Car D3) DPAD 0 LVDS 0 LVDS 1 - + R8A7743 (RZ/G1M) DPAD 0 LVDS 0 - - + R8A7745 (RZ/G1E) DPAD 0 DPAD 1 - - + R8A7779 (R-Car H1) DPAD 0 DPAD 1 - - + R8A7790 (R-Car H2) DPAD 0 LVDS 0 LVDS 1 - + R8A7791 (R-Car M2-W) DPAD 0 LVDS 0 - - + R8A7792 (R-Car V2H) DPAD 0 DPAD 1 - - + R8A7793 (R-Car M2-N) DPAD 0 LVDS 0 - - + R8A7794 (R-Car E2) DPAD 0 DPAD 1 - - + R8A7795 (R-Car H3) DPAD 0 HDMI 0 HDMI 1 LVDS 0 + R8A7796 (R-Car M3-W) DPAD 0 HDMI 0 LVDS 0 - + R8A77965 (R-Car M3-N) DPAD 0 HDMI 0 LVDS 0 - + R8A77970 (R-Car V3M) DPAD 0 LVDS 0 - - + R8A77995 (R-Car D3) DPAD 0 LVDS 0 LVDS 1 - Example: R8A7795 (R-Car H3) ES2.0 DU diff --git a/Documentation/devicetree/bindings/display/sunxi/sun6i-dsi.txt b/Documentation/devicetree/bindings/display/sunxi/sun6i-dsi.txt new file mode 100644 index 000000000000..6a6cf5de08b0 --- /dev/null +++ b/Documentation/devicetree/bindings/display/sunxi/sun6i-dsi.txt @@ -0,0 +1,93 @@ +Allwinner A31 DSI Encoder +========================= + +The DSI pipeline consists of two separate blocks: the DSI controller +itself, and its associated D-PHY. + +DSI Encoder +----------- + +The DSI Encoder generates the DSI signal from the TCON's. + +Required properties: + - compatible: value must be one of: + * allwinner,sun6i-a31-mipi-dsi + - reg: base address and size of memory-mapped region + - interrupts: interrupt associated to this IP + - clocks: phandles to the clocks feeding the DSI encoder + * bus: the DSI interface clock + * mod: the DSI module clock + - clock-names: the clock names mentioned above + - phys: phandle to the D-PHY + - phy-names: must be "dphy" + - resets: phandle to the reset controller driving the encoder + + - ports: A ports node with endpoint definitions as defined in + Documentation/devicetree/bindings/media/video-interfaces.txt. The + first port should be the input endpoint, usually coming from the + associated TCON. + +Any MIPI-DSI device attached to this should be described according to +the bindings defined in ../mipi-dsi-bus.txt + +D-PHY +----- + +Required properties: + - compatible: value must be one of: + * allwinner,sun6i-a31-mipi-dphy + - reg: base address and size of memory-mapped region + - clocks: phandles to the clocks feeding the DSI encoder + * bus: the DSI interface clock + * mod: the DSI module clock + - clock-names: the clock names mentioned above + - resets: phandle to the reset controller driving the encoder + +Example: + +dsi0: dsi@1ca0000 { + compatible = "allwinner,sun6i-a31-mipi-dsi"; + reg = <0x01ca0000 0x1000>; + interrupts = ; + clocks = <&ccu CLK_BUS_MIPI_DSI>, + <&ccu CLK_DSI_SCLK>; + clock-names = "bus", "mod"; + resets = <&ccu RST_BUS_MIPI_DSI>; + phys = <&dphy0>; + phy-names = "dphy"; + #address-cells = <1>; + #size-cells = <0>; + + panel@0 { + compatible = "bananapi,lhr050h41", "ilitek,ili9881c"; + reg = <0>; + power-gpios = <&pio 1 7 GPIO_ACTIVE_HIGH>; /* PB07 */ + reset-gpios = <&r_pio 0 5 GPIO_ACTIVE_LOW>; /* PL05 */ + backlight = <&pwm_bl>; + }; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + #address-cells = <1>; + #size-cells = <0>; + reg = <0>; + + dsi0_in_tcon0: endpoint { + remote-endpoint = <&tcon0_out_dsi0>; + }; + }; + }; +}; + +dphy0: d-phy@1ca1000 { + compatible = "allwinner,sun6i-a31-mipi-dphy"; + reg = <0x01ca1000 0x1000>; + clocks = <&ccu CLK_BUS_MIPI_DSI>, + <&ccu CLK_DSI_DPHY>; + clock-names = "bus", "mod"; + resets = <&ccu RST_BUS_MIPI_DSI>; + #phy-cells = <0>; +}; diff --git a/Documentation/devicetree/bindings/dma/k3dma.txt b/Documentation/devicetree/bindings/dma/k3dma.txt index 23f8d712c3ce..4945aeac4dc4 100644 --- a/Documentation/devicetree/bindings/dma/k3dma.txt +++ b/Documentation/devicetree/bindings/dma/k3dma.txt @@ -23,7 +23,6 @@ Controller: dma-requests = <27>; interrupts = <0 12 4>; clocks = <&pclk>; - status = "disable"; }; Client: diff --git a/Documentation/devicetree/bindings/dma/renesas,rcar-dmac.txt b/Documentation/devicetree/bindings/dma/renesas,rcar-dmac.txt index 61315eaa7660..b1ba639554c0 100644 --- a/Documentation/devicetree/bindings/dma/renesas,rcar-dmac.txt +++ b/Documentation/devicetree/bindings/dma/renesas,rcar-dmac.txt @@ -29,6 +29,7 @@ Required Properties: - "renesas,dmac-r8a77965" (R-Car M3-N) - "renesas,dmac-r8a77970" (R-Car V3M) - "renesas,dmac-r8a77980" (R-Car V3H) + - "renesas,dmac-r8a77995" (R-Car D3) - reg: base address and length of the registers block for the DMAC diff --git a/Documentation/devicetree/bindings/dma/renesas,usb-dmac.txt b/Documentation/devicetree/bindings/dma/renesas,usb-dmac.txt index 9dc935e24e55..482e54362d3e 100644 --- a/Documentation/devicetree/bindings/dma/renesas,usb-dmac.txt +++ b/Documentation/devicetree/bindings/dma/renesas,usb-dmac.txt @@ -12,6 +12,8 @@ Required Properties: - "renesas,r8a7795-usb-dmac" (R-Car H3) - "renesas,r8a7796-usb-dmac" (R-Car M3-W) - "renesas,r8a77965-usb-dmac" (R-Car M3-N) + - "renesas,r8a77990-usb-dmac" (R-Car E3) + - "renesas,r8a77995-usb-dmac" (R-Car D3) - reg: base address and length of the registers block for the DMAC - interrupts: interrupt specifiers for the DMAC, one for each entry in interrupt-names. diff --git a/Documentation/devicetree/bindings/dma/ti-edma.txt b/Documentation/devicetree/bindings/dma/ti-edma.txt index 66026dcf53e1..3f15f6644527 100644 --- a/Documentation/devicetree/bindings/dma/ti-edma.txt +++ b/Documentation/devicetree/bindings/dma/ti-edma.txt @@ -190,7 +190,6 @@ mmc0: mmc@23000000 { power-domains = <&k2g_pds 0xb>; clocks = <&k2g_clks 0xb 1>, <&k2g_clks 0xb 2>; clock-names = "fck", "mmchsdb_fck"; - status = "disabled"; }; ------------------------------------------------------------------------------ diff --git a/Documentation/devicetree/bindings/arm/altera/socfpga-eccmgr.txt b/Documentation/devicetree/bindings/edac/socfpga-eccmgr.txt similarity index 87% rename from Documentation/devicetree/bindings/arm/altera/socfpga-eccmgr.txt rename to Documentation/devicetree/bindings/edac/socfpga-eccmgr.txt index 4a1714f96bab..5626560a6cfd 100644 --- a/Documentation/devicetree/bindings/arm/altera/socfpga-eccmgr.txt +++ b/Documentation/devicetree/bindings/edac/socfpga-eccmgr.txt @@ -231,3 +231,38 @@ Example: <48 IRQ_TYPE_LEVEL_HIGH>; }; }; + +Stratix10 SoCFPGA ECC Manager +The Stratix10 SoC ECC Manager handles the IRQs for each peripheral +in a shared register similar to the Arria10. However, ECC requires +access to registers that can only be read from Secure Monitor with +SMC calls. Therefore the device tree is slightly different. + +Required Properties: +- compatible : Should be "altr,socfpga-s10-ecc-manager" +- interrupts : Should be single bit error interrupt, then double bit error + interrupt. +- interrupt-controller : boolean indicator that ECC Manager is an interrupt controller +- #interrupt-cells : must be set to 2. + +Subcomponents: + +SDRAM ECC +Required Properties: +- compatible : Should be "altr,sdram-edac-s10" +- interrupts : Should be single bit error interrupt, then double bit error + interrupt, in this order. + +Example: + + eccmgr { + compatible = "altr,socfpga-s10-ecc-manager"; + interrupts = <0 15 4>, <0 95 4>; + interrupt-controller; + #interrupt-cells = <2>; + + sdramedac { + compatible = "altr,sdram-edac-s10"; + interrupts = <16 4>, <48 4>; + }; + }; diff --git a/Documentation/devicetree/bindings/firmware/qcom,scm.txt b/Documentation/devicetree/bindings/firmware/qcom,scm.txt index 7b40054be0d8..fcf6979c0b6d 100644 --- a/Documentation/devicetree/bindings/firmware/qcom,scm.txt +++ b/Documentation/devicetree/bindings/firmware/qcom,scm.txt @@ -11,9 +11,10 @@ Required properties: * "qcom,scm-msm8660" for MSM8660 platforms * "qcom,scm-msm8690" for MSM8690 platforms * "qcom,scm-msm8996" for MSM8996 platforms + * "qcom,scm-ipq4019" for IPQ4019 platforms * "qcom,scm" for later processors (MSM8916, APQ8084, MSM8974, etc) - clocks: One to three clocks may be required based on compatible. - * No clock required for "qcom,scm-msm8996" + * No clock required for "qcom,scm-msm8996", "qcom,scm-ipq4019" * Only core clock required for "qcom,scm-apq8064", "qcom,scm-msm8660", and "qcom,scm-msm8960" * Core, iface, and bus clocks required for "qcom,scm" - clock-names: Must contain "core" for the core clock, "iface" for the interface diff --git a/Documentation/devicetree/bindings/fpga/lattice-machxo2-spi.txt b/Documentation/devicetree/bindings/fpga/lattice-machxo2-spi.txt new file mode 100644 index 000000000000..a8c362eb160c --- /dev/null +++ b/Documentation/devicetree/bindings/fpga/lattice-machxo2-spi.txt @@ -0,0 +1,29 @@ +Lattice MachXO2 Slave SPI FPGA Manager + +Lattice MachXO2 FPGAs support a method of loading the bitstream over +'slave SPI' interface. + +See 'MachXO2ProgrammingandConfigurationUsageGuide.pdf' on www.latticesemi.com + +Required properties: +- compatible: should contain "lattice,machxo2-slave-spi" +- reg: spi chip select of the FPGA + +Example for full FPGA configuration: + + fpga-region0 { + compatible = "fpga-region"; + fpga-mgr = <&fpga_mgr_spi>; + #address-cells = <0x1>; + #size-cells = <0x1>; + }; + + spi1: spi@2000 { + ... + + fpga_mgr_spi: fpga-mgr@0 { + compatible = "lattice,machxo2-slave-spi"; + spi-max-frequency = <8000000>; + reg = <0>; + }; + }; diff --git a/Documentation/devicetree/bindings/fsi/fsi-master-gpio.txt b/Documentation/devicetree/bindings/fsi/fsi-master-gpio.txt index a767259dedad..1e442450747f 100644 --- a/Documentation/devicetree/bindings/fsi/fsi-master-gpio.txt +++ b/Documentation/devicetree/bindings/fsi/fsi-master-gpio.txt @@ -11,6 +11,10 @@ Optional properties: - trans-gpios = ; : GPIO for voltage translator enable - mux-gpios = ; : GPIO for pin multiplexing with other functions (eg, external FSI masters) + - no-gpio-delays; : Don't add extra delays between GPIO + accesses. This is useful when the HW + GPIO block is running at a low enough + frequency. Examples: diff --git a/Documentation/devicetree/bindings/gpio/gpio-pca953x.txt b/Documentation/devicetree/bindings/gpio/gpio-pca953x.txt index d2a937682836..88f228665507 100644 --- a/Documentation/devicetree/bindings/gpio/gpio-pca953x.txt +++ b/Documentation/devicetree/bindings/gpio/gpio-pca953x.txt @@ -31,10 +31,15 @@ Required properties: ti,tca9554 onnn,pca9654 exar,xra1202 + - gpio-controller: if used as gpio expander. + - #gpio-cells: if used as gpio expander. + - interrupt-controller: if to be used as interrupt expander. + - #interrupt-cells: if to be used as interrupt expander. Optional properties: - reset-gpios: GPIO specification for the RESET input. This is an active low signal to the PCA953x. + - vcc-supply: power supply regulator. Example: @@ -47,3 +52,32 @@ Example: interrupt-parent = <&gpio3>; interrupts = <23 IRQ_TYPE_LEVEL_LOW>; }; + + +Example with Interrupts: + + + gpio99: gpio@22 { + compatible = "nxp,pcal6524"; + reg = <0x22>; + interrupt-parent = <&gpio6>; + interrupts = <1 IRQ_TYPE_EDGE_FALLING>; /* gpio6_161 */ + interrupt-controller; + #interrupt-cells = <2>; + vcc-supply = <&vdds_1v8_main>; + gpio-controller; + #gpio-cells = <2>; + gpio-line-names = + "hdmi-ct-hpd", "hdmi.ls-oe", "p02", "p03", "vibra", "fault2", "p06", "p07", + "en-usb", "en-host1", "en-host2", "chg-int", "p14", "p15", "mic-int", "en-modem", + "shdn-hs-amp", "chg-status+red", "green", "blue", "en-esata", "fault1", "p26", "p27"; + }; + + ts3a227@3b { + compatible = "ti,ts3a227e"; + reg = <0x3b>; + interrupt-parent = <&gpio99>; + interrupts = <14 IRQ_TYPE_EDGE_RISING>; + ti,micbias = <0>; /* 2.1V */ + }; + diff --git a/Documentation/devicetree/bindings/gpio/renesas,gpio-rcar.txt b/Documentation/devicetree/bindings/gpio/renesas,gpio-rcar.txt index 9474138d776e..378f1322211e 100644 --- a/Documentation/devicetree/bindings/gpio/renesas,gpio-rcar.txt +++ b/Documentation/devicetree/bindings/gpio/renesas,gpio-rcar.txt @@ -5,6 +5,7 @@ Required Properties: - compatible: should contain one or more of the following: - "renesas,gpio-r8a7743": for R8A7743 (RZ/G1M) compatible GPIO controller. - "renesas,gpio-r8a7745": for R8A7745 (RZ/G1E) compatible GPIO controller. + - "renesas,gpio-r8a77470": for R8A77470 (RZ/G1C) compatible GPIO controller. - "renesas,gpio-r8a7778": for R8A7778 (R-Car M1) compatible GPIO controller. - "renesas,gpio-r8a7779": for R8A7779 (R-Car H1) compatible GPIO controller. - "renesas,gpio-r8a7790": for R8A7790 (R-Car H2) compatible GPIO controller. @@ -14,7 +15,9 @@ Required Properties: - "renesas,gpio-r8a7794": for R8A7794 (R-Car E2) compatible GPIO controller. - "renesas,gpio-r8a7795": for R8A7795 (R-Car H3) compatible GPIO controller. - "renesas,gpio-r8a7796": for R8A7796 (R-Car M3-W) compatible GPIO controller. + - "renesas,gpio-r8a77965": for R8A77965 (R-Car M3-N) compatible GPIO controller. - "renesas,gpio-r8a77970": for R8A77970 (R-Car V3M) compatible GPIO controller. + - "renesas,gpio-r8a77990": for R8A77990 (R-Car E3) compatible GPIO controller. - "renesas,gpio-r8a77995": for R8A77995 (R-Car D3) compatible GPIO controller. - "renesas,rcar-gen1-gpio": for a generic R-Car Gen1 GPIO controller. - "renesas,rcar-gen2-gpio": for a generic R-Car Gen2 or RZ/G1 GPIO controller. diff --git a/Documentation/devicetree/bindings/gpio/snps-dwapb-gpio.txt b/Documentation/devicetree/bindings/gpio/snps-dwapb-gpio.txt index 4a75da7051bd..3c1118bc67f5 100644 --- a/Documentation/devicetree/bindings/gpio/snps-dwapb-gpio.txt +++ b/Documentation/devicetree/bindings/gpio/snps-dwapb-gpio.txt @@ -26,8 +26,13 @@ controller. the second encodes the triger flags encoded as described in Documentation/devicetree/bindings/interrupt-controller/interrupts.txt - interrupt-parent : The parent interrupt controller. -- interrupts : The interrupt to the parent controller raised when GPIOs - generate the interrupts. +- interrupts : The interrupts to the parent controller raised when GPIOs + generate the interrupts. If the controller provides one combined interrupt + for all GPIOs, specify a single interrupt. If the controller provides one + interrupt for each GPIO, provide a list of interrupts that correspond to each + of the GPIO pins. When specifying multiple interrupts, if any are unconnected, + use the interrupts-extended property to specify the interrupts and set the + interrupt controller handle for unused interrupts to 0. - snps,nr-gpios : The number of pins in the port, a single cell. - resets : Reset line for the controller. diff --git a/Documentation/devicetree/bindings/gpu/arm,mali-midgard.txt b/Documentation/devicetree/bindings/gpu/arm,mali-midgard.txt index 039219df05c5..18a2cde2e5f3 100644 --- a/Documentation/devicetree/bindings/gpu/arm,mali-midgard.txt +++ b/Documentation/devicetree/bindings/gpu/arm,mali-midgard.txt @@ -34,7 +34,7 @@ Optional properties: - mali-supply : Phandle to regulator for the Mali device. Refer to Documentation/devicetree/bindings/regulator/regulator.txt for details. -- operating-points-v2 : Refer to Documentation/devicetree/bindings/power/opp.txt +- operating-points-v2 : Refer to Documentation/devicetree/bindings/opp/opp.txt for details. diff --git a/Documentation/devicetree/bindings/gpu/arm,mali-utgard.txt b/Documentation/devicetree/bindings/gpu/arm,mali-utgard.txt index c1f65d1dac1d..63cd91176a68 100644 --- a/Documentation/devicetree/bindings/gpu/arm,mali-utgard.txt +++ b/Documentation/devicetree/bindings/gpu/arm,mali-utgard.txt @@ -44,7 +44,7 @@ Optional properties: - memory-region: Memory region to allocate from, as defined in - Documentation/devicetree/bindi/reserved-memory/reserved-memory.txt + Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt - mali-supply: Phandle to regulator for the Mali device, as defined in diff --git a/Documentation/devicetree/bindings/gpu/brcm,bcm-v3d.txt b/Documentation/devicetree/bindings/gpu/brcm,bcm-v3d.txt new file mode 100644 index 000000000000..c907aa8dd755 --- /dev/null +++ b/Documentation/devicetree/bindings/gpu/brcm,bcm-v3d.txt @@ -0,0 +1,28 @@ +Broadcom V3D GPU + +Only the Broadcom V3D 3.x and newer GPUs are covered by this binding. +For V3D 2.x, see brcm,bcm-vc4.txt. + +Required properties: +- compatible: Should be "brcm,7268-v3d" or "brcm,7278-v3d" +- reg: Physical base addresses and lengths of the register areas +- reg-names: Names for the register areas. The "hub", "bridge", and "core0" + register areas are always required. The "gca" register area + is required if the GCA cache controller is present. +- interrupts: The interrupt numbers. The first interrupt is for the hub, + while the following interrupts are for the cores. + See bindings/interrupt-controller/interrupts.txt + +Optional properties: +- clocks: The core clock the unit runs on + +v3d { + compatible = "brcm,7268-v3d"; + reg = <0xf1204000 0x100>, + <0xf1200000 0x4000>, + <0xf1208000 0x4000>, + <0xf1204100 0x100>; + reg-names = "bridge", "hub", "core0", "gca"; + interrupts = <0 78 4>, + <0 77 4>; +}; diff --git a/Documentation/devicetree/bindings/gpu/samsung-scaler.txt b/Documentation/devicetree/bindings/gpu/samsung-scaler.txt new file mode 100644 index 000000000000..9c3d98105dfd --- /dev/null +++ b/Documentation/devicetree/bindings/gpu/samsung-scaler.txt @@ -0,0 +1,27 @@ +* Samsung Exynos Image Scaler + +Required properties: + - compatible : value should be one of the following: + (a) "samsung,exynos5420-scaler" for Scaler IP in Exynos5420 + (b) "samsung,exynos5433-scaler" for Scaler IP in Exynos5433 + + - reg : Physical base address of the IP registers and length of memory + mapped region. + + - interrupts : Interrupt specifier for scaler interrupt, according to format + specific to interrupt parent. + + - clocks : Clock specifier for scaler clock, according to generic clock + bindings. (See Documentation/devicetree/bindings/clock/exynos*.txt) + + - clock-names : Names of clocks. For exynos scaler, it should be "mscl" + on 5420 and "pclk", "aclk" and "aclk_xiu" on 5433. + +Example: + scaler@12800000 { + compatible = "samsung,exynos5420-scaler"; + reg = <0x12800000 0x1294>; + interrupts = <0 220 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&clock CLK_MSCL0>; + clock-names = "mscl"; + }; diff --git a/Documentation/devicetree/bindings/hwmon/gpio-fan.txt b/Documentation/devicetree/bindings/hwmon/gpio-fan.txt index 439a7430fc68..2becdcfdc840 100644 --- a/Documentation/devicetree/bindings/hwmon/gpio-fan.txt +++ b/Documentation/devicetree/bindings/hwmon/gpio-fan.txt @@ -11,7 +11,7 @@ Optional properties: must have the RPM values in ascending order. - alarm-gpios: This pin going active indicates something is wrong with the fan, and a udev event will be fired. -- cooling-cells: If used as a cooling device, must be <2> +- #cooling-cells: If used as a cooling device, must be <2> Also see: Documentation/devicetree/bindings/thermal/thermal.txt min and max states are derived from the speed-map of the fan. diff --git a/Documentation/devicetree/bindings/hwmon/ltc2990.txt b/Documentation/devicetree/bindings/hwmon/ltc2990.txt new file mode 100644 index 000000000000..f92f54029e84 --- /dev/null +++ b/Documentation/devicetree/bindings/hwmon/ltc2990.txt @@ -0,0 +1,36 @@ +ltc2990: Linear Technology LTC2990 power monitor + +Required properties: +- compatible: Must be "lltc,ltc2990" +- reg: I2C slave address +- lltc,meas-mode: + An array of two integers for configuring the chip measurement mode. + + The first integer defines the bits 2..0 in the control register. In all + cases the internal temperature and supply voltage are measured. In + addition the following input measurements are enabled per mode: + + 0: V1, V2, TR2 + 1: V1-V2, TR2 + 2: V1-V2, V3, V4 + 3: TR1, V3, V4 + 4: TR1, V3-V4 + 5: TR1, TR2 + 6: V1-V2, V3-V4 + 7: V1, V2, V3, V4 + + The second integer defines the bits 4..3 in the control register. This + allows a subset of the measurements to be enabled: + + 0: Internal temperature and supply voltage only + 1: TR1, V1 or V1-V2 only per mode + 2: TR2, V3 or V3-V4 only per mode + 3: All measurements per mode + +Example: + +ltc2990@4c { + compatible = "lltc,ltc2990"; + reg = <0x4c>; + lltc,meas-mode = <7 3>; /* V1, V2, V3, V4 */ +}; diff --git a/Documentation/devicetree/bindings/i2c/i2c-davinci.txt b/Documentation/devicetree/bindings/i2c/i2c-davinci.txt index 64e6e656c345..b745f3706120 100644 --- a/Documentation/devicetree/bindings/i2c/i2c-davinci.txt +++ b/Documentation/devicetree/bindings/i2c/i2c-davinci.txt @@ -24,7 +24,7 @@ Recommended properties : - clock-frequency : desired I2C bus clock frequency in Hz. - ti,has-pfunc: boolean; if defined, it indicates that SoC supports PFUNC registers. PFUNC registers allow to switch I2C pins to function as - GPIOs, so they can by toggled manually. + GPIOs, so they can be toggled manually. Example (enbw_cmc board): i2c@1c22000 { diff --git a/Documentation/devicetree/bindings/i2c/i2c-rcar.txt b/Documentation/devicetree/bindings/i2c/i2c-rcar.txt index 4a7811ecd954..7ce8fae55537 100644 --- a/Documentation/devicetree/bindings/i2c/i2c-rcar.txt +++ b/Documentation/devicetree/bindings/i2c/i2c-rcar.txt @@ -15,6 +15,7 @@ Required properties: "renesas,i2c-r8a7796" if the device is a part of a R8A7796 SoC. "renesas,i2c-r8a77965" if the device is a part of a R8A77965 SoC. "renesas,i2c-r8a77970" if the device is a part of a R8A77970 SoC. + "renesas,i2c-r8a77980" if the device is a part of a R8A77980 SoC. "renesas,i2c-r8a77995" if the device is a part of a R8A77995 SoC. "renesas,rcar-gen1-i2c" for a generic R-Car Gen1 compatible device. "renesas,rcar-gen2-i2c" for a generic R-Car Gen2 or RZ/G1 compatible diff --git a/Documentation/devicetree/bindings/i2c/i2c-s3c2410.txt b/Documentation/devicetree/bindings/i2c/i2c-s3c2410.txt index 89b3250f049b..66ae46d3bc2f 100644 --- a/Documentation/devicetree/bindings/i2c/i2c-s3c2410.txt +++ b/Documentation/devicetree/bindings/i2c/i2c-s3c2410.txt @@ -8,9 +8,7 @@ Required properties: (b) "samsung, s3c2440-i2c", for i2c compatible with s3c2440 i2c. (c) "samsung, s3c2440-hdmiphy-i2c", for s3c2440-like i2c used inside HDMIPHY block found on several samsung SoCs - (d) "samsung, exynos5440-i2c", for s3c2440-like i2c used - on EXYNOS5440 which does not need GPIO configuration. - (e) "samsung, exynos5-sata-phy-i2c", for s3c2440-like i2c used as + (d) "samsung, exynos5-sata-phy-i2c", for s3c2440-like i2c used as a host to SATA PHY controller on an internal bus. - reg: physical base address of the controller and length of memory mapped region. diff --git a/Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt b/Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt index 1e6ee3deb4fa..d1acd5ea2737 100644 --- a/Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt +++ b/Documentation/devicetree/bindings/iio/adc/amlogic,meson-saradc.txt @@ -7,6 +7,7 @@ Required properties: - "amlogic,meson-gxbb-saradc" for GXBB - "amlogic,meson-gxl-saradc" for GXL - "amlogic,meson-gxm-saradc" for GXM + - "amlogic,meson-axg-saradc" for AXG along with the generic "amlogic,meson-saradc" - reg: the physical base address and length of the registers - interrupts: the interrupt indicating end of sampling diff --git a/Documentation/devicetree/bindings/iio/adc/mcp320x.txt b/Documentation/devicetree/bindings/iio/adc/mcp320x.txt index 7d64753df949..56373d643f76 100644 --- a/Documentation/devicetree/bindings/iio/adc/mcp320x.txt +++ b/Documentation/devicetree/bindings/iio/adc/mcp320x.txt @@ -49,7 +49,7 @@ Required properties: Examples: spi_controller { mcp3x0x@0 { - compatible = "mcp3002"; + compatible = "microchip,mcp3002"; reg = <0>; spi-max-frequency = <1000000>; vref-supply = <&vref_reg>; diff --git a/Documentation/devicetree/bindings/arm/samsung/exynos-adc.txt b/Documentation/devicetree/bindings/iio/adc/samsung,exynos-adc.txt similarity index 100% rename from Documentation/devicetree/bindings/arm/samsung/exynos-adc.txt rename to Documentation/devicetree/bindings/iio/adc/samsung,exynos-adc.txt diff --git a/Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt b/Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt index e8bb8243e92c..f1ead43a1a95 100644 --- a/Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt +++ b/Documentation/devicetree/bindings/iio/adc/st,stm32-adc.txt @@ -24,8 +24,11 @@ Required properties: - compatible: Should be one of: "st,stm32f4-adc-core" "st,stm32h7-adc-core" + "st,stm32mp1-adc-core" - reg: Offset and length of the ADC block register set. -- interrupts: Must contain the interrupt for ADC block. +- interrupts: One or more interrupts for ADC block. Some parts like stm32f4 + and stm32h7 share a common ADC interrupt line. stm32mp1 has two separate + interrupt lines, one for each ADC within ADC block. - clocks: Core can use up to two clocks, depending on part used: - "adc" clock: for the analog circuitry, common to all ADCs. It's required on stm32f4. @@ -53,6 +56,7 @@ Required properties: - compatible: Should be one of: "st,stm32f4-adc" "st,stm32h7-adc" + "st,stm32mp1-adc" - reg: Offset of ADC instance in ADC block (e.g. may be 0x0, 0x100, 0x200). - clocks: Input clock private to this ADC instance. It's required only on stm32f4, that has per instance clock input for registers access. diff --git a/Documentation/devicetree/bindings/iio/adc/st,stm32-dfsdm-adc.txt b/Documentation/devicetree/bindings/iio/adc/st,stm32-dfsdm-adc.txt index ed7520d1d051..75ba25d062e1 100644 --- a/Documentation/devicetree/bindings/iio/adc/st,stm32-dfsdm-adc.txt +++ b/Documentation/devicetree/bindings/iio/adc/st,stm32-dfsdm-adc.txt @@ -8,14 +8,16 @@ It is mainly targeted for: - PDM microphones (audio digital microphone) It features up to 8 serial digital interfaces (SPI or Manchester) and -up to 4 filters on stm32h7. +up to 4 filters on stm32h7 or 6 filters on stm32mp1. Each child node match with a filter instance. Contents of a STM32 DFSDM root node: ------------------------------------ Required properties: -- compatible: Should be "st,stm32h7-dfsdm". +- compatible: Should be one of: + "st,stm32h7-dfsdm" + "st,stm32mp1-dfsdm" - reg: Offset and length of the DFSDM block register set. - clocks: IP and serial interfaces clocking. Should be set according to rcc clock ID and "clock-names". @@ -45,6 +47,7 @@ Required properties: "st,stm32-dfsdm-adc" for sigma delta ADCs "st,stm32-dfsdm-dmic" for audio digital microphone. - reg: Specifies the DFSDM filter instance used. + Valid values are from 0 to 3 on stm32h7, 0 to 5 on stm32mp1. - interrupts: IRQ lines connected to each DFSDM filter instance. - st,adc-channels: List of single-ended channels muxed for this ADC. valid values: diff --git a/Documentation/devicetree/bindings/iio/afe/current-sense-amplifier.txt b/Documentation/devicetree/bindings/iio/afe/current-sense-amplifier.txt new file mode 100644 index 000000000000..821b61b8c542 --- /dev/null +++ b/Documentation/devicetree/bindings/iio/afe/current-sense-amplifier.txt @@ -0,0 +1,26 @@ +Current Sense Amplifier +======================= + +When an io-channel measures the output voltage from a current sense +amplifier, the interesting measurement is almost always the current +through the sense resistor, not the voltage output. This binding +describes such a current sense circuit. + +Required properties: +- compatible : "current-sense-amplifier" +- io-channels : Channel node of a voltage io-channel. +- sense-resistor-micro-ohms : The sense resistance in microohms. + +Optional properties: +- sense-gain-mult: Amplifier gain multiplier. The default is <1>. +- sense-gain-div: Amplifier gain divider. The default is <1>. + +Example: + +sysi { + compatible = "current-sense-amplifier"; + io-channels = <&tiadc 0>; + + sense-resistor-micro-ohms = <20000>; + sense-gain-mul = <50>; +}; diff --git a/Documentation/devicetree/bindings/iio/afe/current-sense-shunt.txt b/Documentation/devicetree/bindings/iio/afe/current-sense-shunt.txt new file mode 100644 index 000000000000..0f67108a07b6 --- /dev/null +++ b/Documentation/devicetree/bindings/iio/afe/current-sense-shunt.txt @@ -0,0 +1,41 @@ +Current Sense Shunt +=================== + +When an io-channel measures the voltage over a current sense shunt, +the interesting measurement is almost always the current through the +shunt, not the voltage over it. This binding describes such a current +sense circuit. + +Required properties: +- compatible : "current-sense-shunt" +- io-channels : Channel node of a voltage io-channel. +- shunt-resistor-micro-ohms : The shunt resistance in microohms. + +Example: +The system current is measured by measuring the voltage over a +3.3 ohms shunt resistor. + +sysi { + compatible = "current-sense-shunt"; + io-channels = <&tiadc 0>; + + /* Divide the voltage by 3300000/1000000 (or 3.3) for the current. */ + shunt-resistor-micro-ohms = <3300000>; +}; + +&i2c { + tiadc: adc@48 { + compatible = "ti,ads1015"; + reg = <0x48>; + #io-channel-cells = <1>; + + #address-cells = <1>; + #size-cells = <0>; + + channel@0 { /* IN0,IN1 differential */ + reg = <0>; + ti,gain = <1>; + ti,datarate = <4>; + }; + }; +}; diff --git a/Documentation/devicetree/bindings/iio/afe/voltage-divider.txt b/Documentation/devicetree/bindings/iio/afe/voltage-divider.txt new file mode 100644 index 000000000000..b452a8406107 --- /dev/null +++ b/Documentation/devicetree/bindings/iio/afe/voltage-divider.txt @@ -0,0 +1,53 @@ +Voltage divider +=============== + +When an io-channel measures the midpoint of a voltage divider, the +interesting voltage is often the voltage over the full resistance +of the divider. This binding describes the voltage divider in such +a curcuit. + + Vin ----. + | + .-----. + | R | + '-----' + | + +---- Vout + | + .-----. + | Rout| + '-----' + | + GND + +Required properties: +- compatible : "voltage-divider" +- io-channels : Channel node of a voltage io-channel measuring Vout. +- output-ohms : Resistance Rout over which the output voltage is measured. + See full-ohms. +- full-ohms : Resistance R + Rout for the full divider. The io-channel + is scaled by the Rout / (R + Rout) quotient. + +Example: +The system voltage is circa 12V, but divided down with a 22/222 +voltage divider (R = 200 Ohms, Rout = 22 Ohms) and fed to an ADC. + +sysv { + compatible = "voltage-divider"; + io-channels = <&maxadc 1>; + + /* Scale the system voltage by 22/222 to fit the ADC range. */ + output-ohms = <22>; + full-ohms = <222>; /* 200 + 22 */ +}; + +&spi { + maxadc: adc@0 { + compatible = "maxim,max1027"; + reg = <0>; + #io-channel-cells = <1>; + interrupt-parent = <&gpio5>; + interrupts = <15 IRQ_TYPE_EDGE_RISING>; + spi-max-frequency = <1000000>; + }; +}; diff --git a/Documentation/devicetree/bindings/iio/dac/ltc2632.txt b/Documentation/devicetree/bindings/iio/dac/ltc2632.txt index eb911e5a8ab4..e0d5fea33031 100644 --- a/Documentation/devicetree/bindings/iio/dac/ltc2632.txt +++ b/Documentation/devicetree/bindings/iio/dac/ltc2632.txt @@ -12,12 +12,26 @@ Required properties: Property rules described in Documentation/devicetree/bindings/spi/spi-bus.txt apply. In particular, "reg" and "spi-max-frequency" properties must be given. +Optional properties: + - vref-supply: Phandle to the external reference voltage supply. This should + only be set if there is an external reference voltage connected to the VREF + pin. If the property is not set the internal reference is used. + Example: + vref: regulator-vref { + compatible = "regulator-fixed"; + regulator-name = "vref-ltc2632"; + regulator-min-microvolt = <1250000>; + regulator-max-microvolt = <1250000>; + regulator-always-on; + }; + spi_master { dac: ltc2632@0 { compatible = "lltc,ltc2632-l12"; reg = <0>; /* CS0 */ spi-max-frequency = <1000000>; + vref-supply = <&vref>; /* optional */ }; }; diff --git a/Documentation/devicetree/bindings/iio/dac/ti,dac5571.txt b/Documentation/devicetree/bindings/iio/dac/ti,dac5571.txt new file mode 100644 index 000000000000..03af6b9a4d07 --- /dev/null +++ b/Documentation/devicetree/bindings/iio/dac/ti,dac5571.txt @@ -0,0 +1,24 @@ +* Texas Instruments DAC5571 Family + +Required properties: + - compatible: Should contain + "ti,dac5571" + "ti,dac6571" + "ti,dac7571" + "ti,dac5574" + "ti,dac6574" + "ti,dac7574" + "ti,dac5573" + "ti,dac6573" + "ti,dac7573" + - reg: Should contain the DAC I2C address + +Optional properties: + - vref-supply: The regulator supply for DAC reference voltage + +Example: +dac@0 { + compatible = "ti,dac5571"; + reg = <0x4C>; + vref-supply = <&vdd_supply>; +}; diff --git a/Documentation/devicetree/bindings/iio/imu/inv_mpu6050.txt b/Documentation/devicetree/bindings/iio/imu/inv_mpu6050.txt index 2b4514592f83..5f4777e8cc9e 100644 --- a/Documentation/devicetree/bindings/iio/imu/inv_mpu6050.txt +++ b/Documentation/devicetree/bindings/iio/imu/inv_mpu6050.txt @@ -8,10 +8,16 @@ Required properties: "invensense,mpu6500" "invensense,mpu9150" "invensense,mpu9250" + "invensense,mpu9255" "invensense,icm20608" - reg : the I2C address of the sensor - interrupt-parent : should be the phandle for the interrupt controller - - interrupts : interrupt mapping for GPIO IRQ + - interrupts: interrupt mapping for IRQ. It should be configured with flags + IRQ_TYPE_LEVEL_HIGH, IRQ_TYPE_EDGE_RISING, IRQ_TYPE_LEVEL_LOW or + IRQ_TYPE_EDGE_FALLING. + + Refer to interrupt-controller/interrupts.txt for generic interrupt client node + bindings. Optional properties: - mount-matrix: an optional 3x3 mounting rotation matrix @@ -24,7 +30,7 @@ Example: compatible = "invensense,mpu6050"; reg = <0x68>; interrupt-parent = <&gpio1>; - interrupts = <18 1>; + interrupts = <18 IRQ_TYPE_EDGE_RISING>; mount-matrix = "-0.984807753012208", /* x0 */ "0", /* y0 */ "-0.173648177666930", /* z0 */ @@ -41,7 +47,7 @@ Example: compatible = "invensense,mpu9250"; reg = <0x68>; interrupt-parent = <&gpio3>; - interrupts = <21 1>; + interrupts = <21 IRQ_TYPE_LEVEL_HIGH>; i2c-gate { #address-cells = <1>; #size-cells = <0>; diff --git a/Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt b/Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt index 1ff1af799c76..ef8a8566c63f 100644 --- a/Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt +++ b/Documentation/devicetree/bindings/iio/imu/st_lsm6dsx.txt @@ -6,6 +6,7 @@ Required properties: "st,lsm6ds3h" "st,lsm6dsl" "st,lsm6dsm" + "st,ism330dlc" - reg: i2c address of the sensor / spi cs line Optional properties: diff --git a/Documentation/devicetree/bindings/iio/potentiostat/lmp91000.txt b/Documentation/devicetree/bindings/iio/potentiostat/lmp91000.txt index b9b621e94cd7..e6d0c2eb345c 100644 --- a/Documentation/devicetree/bindings/iio/potentiostat/lmp91000.txt +++ b/Documentation/devicetree/bindings/iio/potentiostat/lmp91000.txt @@ -1,10 +1,13 @@ -* Texas Instruments LMP91000 potentiostat +* Texas Instruments LMP91000 series of potentiostats -http://www.ti.com/lit/ds/symlink/lmp91000.pdf +LMP91000: http://www.ti.com/lit/ds/symlink/lmp91000.pdf +LMP91002: http://www.ti.com/lit/ds/symlink/lmp91002.pdf Required properties: - - compatible: should be "ti,lmp91000" + - compatible: should be one of the following: + "ti,lmp91000" + "ti,lmp91002" - reg: the I2C address of the device - io-channels: the phandle of the iio provider diff --git a/Documentation/devicetree/bindings/input/elan_i2c.txt b/Documentation/devicetree/bindings/input/elan_i2c.txt index ee3242c4ba67..d80a83583238 100644 --- a/Documentation/devicetree/bindings/input/elan_i2c.txt +++ b/Documentation/devicetree/bindings/input/elan_i2c.txt @@ -14,6 +14,7 @@ Optional properties: - pinctrl-0: a phandle pointing to the pin settings for the device (see pinctrl binding [1]). - vcc-supply: a phandle for the regulator supplying 3.3V power. +- elan,trackpoint: touchpad can support a trackpoint (boolean) [0]: Documentation/devicetree/bindings/interrupt-controller/interrupts.txt [1]: Documentation/devicetree/bindings/pinctrl/pinctrl-bindings.txt diff --git a/Documentation/devicetree/bindings/input/mtk-pmic-keys.txt b/Documentation/devicetree/bindings/input/mtk-pmic-keys.txt new file mode 100644 index 000000000000..2888d07c2ef0 --- /dev/null +++ b/Documentation/devicetree/bindings/input/mtk-pmic-keys.txt @@ -0,0 +1,43 @@ +MediaTek MT6397/MT6323 PMIC Keys Device Driver + +There are two key functions provided by MT6397/MT6323 PMIC, pwrkey +and homekey. The key functions are defined as the subnode of the function +node provided by MT6397/MT6323 PMIC that is being defined as one kind +of Muti-Function Device (MFD) + +For MT6397/MT6323 MFD bindings see: +Documentation/devicetree/bindings/mfd/mt6397.txt + +Required properties: +- compatible: "mediatek,mt6397-keys" or "mediatek,mt6323-keys" +- linux,keycodes: See Documentation/devicetree/bindings/input/keys.txt + +Optional Properties: +- wakeup-source: See Documentation/devicetree/bindings/power/wakeup-source.txt +- mediatek,long-press-mode: Long press key shutdown setting, 1 for + pwrkey only, 2 for pwrkey/homekey together, others for disabled. +- power-off-time-sec: See Documentation/devicetree/bindings/input/keys.txt + +Example: + + pmic: mt6397 { + compatible = "mediatek,mt6397"; + + ... + + mt6397keys: mt6397keys { + compatible = "mediatek,mt6397-keys"; + mediatek,long-press-mode = <1>; + power-off-time-sec = <0>; + + power { + linux,keycodes = <116>; + wakeup-source; + }; + + home { + linux,keycodes = <114>; + }; + }; + + }; diff --git a/Documentation/devicetree/bindings/input/rmi4/rmi_2d_sensor.txt b/Documentation/devicetree/bindings/input/rmi4/rmi_2d_sensor.txt index f2c30c8b725d..9afffbdf6e28 100644 --- a/Documentation/devicetree/bindings/input/rmi4/rmi_2d_sensor.txt +++ b/Documentation/devicetree/bindings/input/rmi4/rmi_2d_sensor.txt @@ -12,7 +12,7 @@ Additional documentation for F11 can be found at: http://www.synaptics.com/sites/default/files/511-000136-01-Rev-E-RMI4-Interfacing-Guide.pdf Optional Touch Properties: -Description in Documentation/devicetree/bindings/input/touch +Description in Documentation/devicetree/bindings/input/touchscreen - touchscreen-inverted-x - touchscreen-inverted-y - touchscreen-swapped-x-y diff --git a/Documentation/devicetree/bindings/input/rotary-encoder.txt b/Documentation/devicetree/bindings/input/rotary-encoder.txt index f99fe5cdeaec..a644408b33b8 100644 --- a/Documentation/devicetree/bindings/input/rotary-encoder.txt +++ b/Documentation/devicetree/bindings/input/rotary-encoder.txt @@ -28,7 +28,7 @@ Deprecated properties: This property is deprecated. Instead, a 'steps-per-period ' value should be used, such as "rotary-encoder,steps-per-period = <2>". -See Documentation/input/rotary-encoder.txt for more information. +See Documentation/input/devices/rotary-encoder.rst for more information. Example: diff --git a/Documentation/devicetree/bindings/interrupt-controller/amlogic,meson-gpio-intc.txt b/Documentation/devicetree/bindings/interrupt-controller/amlogic,meson-gpio-intc.txt index a83f9a5734ca..89674ad8a097 100644 --- a/Documentation/devicetree/bindings/interrupt-controller/amlogic,meson-gpio-intc.txt +++ b/Documentation/devicetree/bindings/interrupt-controller/amlogic,meson-gpio-intc.txt @@ -9,11 +9,12 @@ number of interrupt exposed depends on the SoC. Required properties: -- compatible : must have "amlogic,meson8-gpio-intc” and either - “amlogic,meson8-gpio-intc” for meson8 SoCs (S802) or - “amlogic,meson8b-gpio-intc” for meson8b SoCs (S805) or - “amlogic,meson-gxbb-gpio-intc” for GXBB SoCs (S905) or - “amlogic,meson-gxl-gpio-intc” for GXL SoCs (S905X, S912) +- compatible : must have "amlogic,meson8-gpio-intc" and either + "amlogic,meson8-gpio-intc" for meson8 SoCs (S802) or + "amlogic,meson8b-gpio-intc" for meson8b SoCs (S805) or + "amlogic,meson-gxbb-gpio-intc" for GXBB SoCs (S905) or + "amlogic,meson-gxl-gpio-intc" for GXL SoCs (S905X, S912) + "amlogic,meson-axg-gpio-intc" for AXG SoCs (A113D, A113X) - interrupt-parent : a phandle to the GIC the interrupts are routed to. Usually this is provided at the root level of the device tree as it is common to most of the SoC. diff --git a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt index 0a57f2f4167d..3ea78c4ef887 100644 --- a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt +++ b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt @@ -57,6 +57,20 @@ Optional occupied by the redistributors. Required if more than one such region is present. +- msi-controller: Boolean property. Identifies the node as an MSI + controller. Only present if the Message Based Interrupt + functionnality is being exposed by the HW, and the mbi-ranges + property present. + +- mbi-ranges: A list of pairs , where "intid" is the first + SPI of a range that can be used an MBI, and "span" the size of that + range. Multiple ranges can be provided. Requires "msi-controller" to + be set. + +- mbi-alias: Address property. Base address of an alias of the GICD + region containing only the {SET,CLR}SPI registers to be used if + isolation is required, and if supported by the HW. + Sub-nodes: PPI affinity can be expressed as a single "ppi-partitions" node, @@ -99,6 +113,9 @@ Examples: <0x0 0x2c020000 0 0x2000>; // GICV interrupts = <1 9 4>; + msi-controller; + mbi-ranges = <256 128>; + gic-its@2c200000 { compatible = "arm,gic-v3-its"; msi-controller; diff --git a/Documentation/devicetree/bindings/interrupt-controller/st,stm32-exti.txt b/Documentation/devicetree/bindings/interrupt-controller/st,stm32-exti.txt index edf03f09244b..136bd612bd83 100644 --- a/Documentation/devicetree/bindings/interrupt-controller/st,stm32-exti.txt +++ b/Documentation/devicetree/bindings/interrupt-controller/st,stm32-exti.txt @@ -5,11 +5,14 @@ Required properties: - compatible: Should be: "st,stm32-exti" "st,stm32h7-exti" + "st,stm32mp1-exti" - reg: Specifies base physical address and size of the registers - interrupt-controller: Indentifies the node as an interrupt controller - #interrupt-cells: Specifies the number of cells to encode an interrupt specifier, shall be 2 - interrupts: interrupts references to primary interrupt controller + (only needed for exti controller with multiple exti under + same parent interrupt: st,stm32-exti and st,stm32h7-exti") Example: diff --git a/Documentation/devicetree/bindings/ipmi/npcm7xx-kcs-bmc.txt b/Documentation/devicetree/bindings/ipmi/npcm7xx-kcs-bmc.txt new file mode 100644 index 000000000000..3538a214fff1 --- /dev/null +++ b/Documentation/devicetree/bindings/ipmi/npcm7xx-kcs-bmc.txt @@ -0,0 +1,39 @@ +* Nuvoton NPCM7xx KCS (Keyboard Controller Style) IPMI interface + +The Nuvoton SOCs (NPCM7xx) are commonly used as BMCs +(Baseboard Management Controllers) and the KCS interface can be +used to perform in-band IPMI communication with their host. + +Required properties: +- compatible : should be one of + "nuvoton,npcm750-kcs-bmc" +- interrupts : interrupt generated by the controller +- kcs_chan : The KCS channel number in the controller + +Example: + + lpc_kcs: lpc_kcs@f0007000 { + compatible = "nuvoton,npcm750-lpc-kcs", "simple-mfd", "syscon"; + reg = <0xf0007000 0x40>; + reg-io-width = <1>; + + #address-cells = <1>; + #size-cells = <1>; + ranges = <0x0 0xf0007000 0x40>; + + kcs1: kcs1@0 { + compatible = "nuvoton,npcm750-kcs-bmc"; + reg = <0x0 0x40>; + interrupts = <0 9 4>; + kcs_chan = <1>; + status = "disabled"; + }; + + kcs2: kcs2@0 { + compatible = "nuvoton,npcm750-kcs-bmc"; + reg = <0x0 0x40>; + interrupts = <0 9 4>; + kcs_chan = <2>; + status = "disabled"; + }; + }; \ No newline at end of file diff --git a/Documentation/devicetree/bindings/leds/backlight/pwm-backlight.txt b/Documentation/devicetree/bindings/leds/backlight/pwm-backlight.txt index 764db86d441a..310810906613 100644 --- a/Documentation/devicetree/bindings/leds/backlight/pwm-backlight.txt +++ b/Documentation/devicetree/bindings/leds/backlight/pwm-backlight.txt @@ -17,6 +17,10 @@ Optional properties: "pwms" property (see PWM binding[0]) - enable-gpios: contains a single GPIO specifier for the GPIO which enables and disables the backlight (see GPIO binding[1]) + - post-pwm-on-delay-ms: Delay in ms between setting an initial (non-zero) PWM + and enabling the backlight using GPIO. + - pwm-off-delay-ms: Delay in ms between disabling the backlight using GPIO + and setting PWM value to 0. [0]: Documentation/devicetree/bindings/pwm/pwm.txt [1]: Documentation/devicetree/bindings/gpio/gpio.txt @@ -32,4 +36,6 @@ Example: power-supply = <&vdd_bl_reg>; enable-gpios = <&gpio 58 0>; + post-pwm-on-delay-ms = <10>; + pwm-off-delay-ms = <10>; }; diff --git a/Documentation/devicetree/bindings/leds/backlight/zii,rave-sp-backlight.txt b/Documentation/devicetree/bindings/leds/backlight/zii,rave-sp-backlight.txt new file mode 100644 index 000000000000..ff5c92138650 --- /dev/null +++ b/Documentation/devicetree/bindings/leds/backlight/zii,rave-sp-backlight.txt @@ -0,0 +1,23 @@ +Zodiac Inflight Innovations RAVE Supervisory Processor Backlight Bindings + +RAVE SP backlight device is a "MFD cell" device corresponding to +backlight functionality of RAVE Supervisory Processor. It is expected +that its Device Tree node is specified as a child of the node +corresponding to the parent RAVE SP device (as documented in +Documentation/devicetree/bindings/mfd/zii,rave-sp.txt) + +Required properties: + +- compatible: Should be "zii,rave-sp-backlight" + +Example: + + rave-sp { + compatible = "zii,rave-sp-rdu1"; + current-speed = <38400>; + + backlight { + compatible = "zii,rave-sp-backlight"; + }; + } + diff --git a/Documentation/devicetree/bindings/leds/leds-cr0014114.txt b/Documentation/devicetree/bindings/leds/leds-cr0014114.txt new file mode 100644 index 000000000000..4255b19ad25c --- /dev/null +++ b/Documentation/devicetree/bindings/leds/leds-cr0014114.txt @@ -0,0 +1,54 @@ +Crane Merchandising System - cr0014114 LED driver +------------------------------------------------- + +This LED Board is widely used in vending machines produced +by Crane Merchandising Systems. + +Required properties: +- compatible: "crane,cr0014114" + +Property rules described in Documentation/devicetree/bindings/spi/spi-bus.txt +apply. In particular, "reg" and "spi-max-frequency" properties must be given. + +LED sub-node properties: +- label : + see Documentation/devicetree/bindings/leds/common.txt +- linux,default-trigger : (optional) + see Documentation/devicetree/bindings/leds/common.txt + +Example +------- + +led-controller@0 { + compatible = "crane,cr0014114"; + reg = <0>; + spi-max-frequency = <50000>; + #address-cells = <1>; + #size-cells = <0>; + + led@0 { + reg = <0>; + label = "red:coin"; + }; + led@1 { + reg = <1>; + label = "green:coin"; + }; + led@2 { + reg = <2>; + label = "blue:coin"; + }; + led@3 { + reg = <3>; + label = "red:bill"; + }; + led@4 { + reg = <4>; + label = "green:bill"; + }; + led@5 { + reg = <5>; + label = "blue:bill"; + }; + ... +}; diff --git a/Documentation/devicetree/bindings/leds/leds-lm3601x.txt b/Documentation/devicetree/bindings/leds/leds-lm3601x.txt new file mode 100644 index 000000000000..a88b2c41e75b --- /dev/null +++ b/Documentation/devicetree/bindings/leds/leds-lm3601x.txt @@ -0,0 +1,45 @@ +* Texas Instruments - lm3601x Single-LED Flash Driver + +The LM3601X are ultra-small LED flash drivers that +provide a high level of adjustability. + +Required properties: + - compatible : Can be one of the following + "ti,lm36010" + "ti,lm36011" + - reg : I2C slave address + - #address-cells : 1 + - #size-cells : 0 + +Required child properties: + - reg : 0 - Indicates a IR mode + 1 - Indicates a Torch (white LED) mode + +Required properties for flash LED child nodes: + See Documentation/devicetree/bindings/leds/common.txt + - flash-max-microamp : Range from 11mA - 1.5A + - flash-max-timeout-us : Range from 40ms - 1600ms + - led-max-microamp : Range from 2.4mA - 376mA + +Optional child properties: + - label : see Documentation/devicetree/bindings/leds/common.txt + +Example: +led-controller@64 { + compatible = "ti,lm36010"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x64>; + + led@0 { + reg = <1>; + label = "white:torch"; + led-max-microamp = <376000>; + flash-max-microamp = <1500000>; + flash-max-timeout-us = <1600000>; + }; +} + +For more product information please see the links below: +http://www.ti.com/product/LM36010 +http://www.ti.com/product/LM36011 diff --git a/Documentation/devicetree/bindings/leds/leds-sc27xx-bltc.txt b/Documentation/devicetree/bindings/leds/leds-sc27xx-bltc.txt new file mode 100644 index 000000000000..dddf84f9c7ea --- /dev/null +++ b/Documentation/devicetree/bindings/leds/leds-sc27xx-bltc.txt @@ -0,0 +1,41 @@ +LEDs connected to Spreadtrum SC27XX PMIC breathing light controller + +The SC27xx breathing light controller supports to 3 outputs: +red LED, green LED and blue LED. Each LED can work at normal +PWM mode or breath light mode. + +Required properties: +- compatible: Should be "sprd,sc2731-bltc". +- #address-cells: Must be 1. +- #size-cells: Must be 0. +- reg: Specify the controller address. + +Required child properties: +- reg: Port this LED is connected to. + +Optional child properties: +- label: See Documentation/devicetree/bindings/leds/common.txt. + +Examples: + +led-controller@200 { + compatible = "sprd,sc2731-bltc"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x200>; + + led@0 { + label = "red"; + reg = <0x0>; + }; + + led@1 { + label = "green"; + reg = <0x1>; + }; + + led@2 { + label = "blue"; + reg = <0x2>; + }; +}; diff --git a/Documentation/devicetree/bindings/mailbox/qcom,apcs-kpss-global.txt b/Documentation/devicetree/bindings/mailbox/qcom,apcs-kpss-global.txt index 16964f0c1773..6e8a9ab0fdae 100644 --- a/Documentation/devicetree/bindings/mailbox/qcom,apcs-kpss-global.txt +++ b/Documentation/devicetree/bindings/mailbox/qcom,apcs-kpss-global.txt @@ -10,6 +10,8 @@ platforms. Definition: must be one of: "qcom,msm8916-apcs-kpss-global", "qcom,msm8996-apcs-hmss-global" + "qcom,msm8998-apcs-hmss-global" + "qcom,sdm845-apss-shared" - reg: Usage: required diff --git a/Documentation/devicetree/bindings/mailbox/stm32-ipcc.txt b/Documentation/devicetree/bindings/mailbox/stm32-ipcc.txt new file mode 100644 index 000000000000..1d2b7fee7b85 --- /dev/null +++ b/Documentation/devicetree/bindings/mailbox/stm32-ipcc.txt @@ -0,0 +1,47 @@ +* STMicroelectronics STM32 IPCC (Inter-Processor Communication Controller) + +The IPCC block provides a non blocking signaling mechanism to post and +retrieve messages in an atomic way between two processors. +It provides the signaling for N bidirectionnal channels. The number of channels +(N) can be read from a dedicated register. + +Required properties: +- compatible: Must be "st,stm32mp1-ipcc" +- reg: Register address range (base address and length) +- st,proc-id: Processor id using the mailbox (0 or 1) +- clocks: Input clock +- interrupt-names: List of names for the interrupts described by the interrupt + property. Must contain the following entries: + - "rx" + - "tx" + - "wakeup" +- interrupts: Interrupt specifiers for "rx channel occupied", "tx channel + free" and "system wakeup". +- #mbox-cells: Number of cells required for the mailbox specifier. Must be 1. + The data contained in the mbox specifier of the "mboxes" + property in the client node is the mailbox channel index. + +Optional properties: +- wakeup-source: Flag to indicate whether this device can wake up the system + + + +Example: + ipcc: mailbox@4c001000 { + compatible = "st,stm32mp1-ipcc"; + #mbox-cells = <1>; + reg = <0x4c001000 0x400>; + st,proc-id = <0>; + interrupts-extended = <&intc GIC_SPI 100 IRQ_TYPE_NONE>, + <&intc GIC_SPI 101 IRQ_TYPE_NONE>, + <&aiec 62 1>; + interrupt-names = "rx", "tx", "wakeup"; + clocks = <&rcc_clk IPCC>; + wakeup-source; + } + +Client: + mbox_test { + ... + mboxes = <&ipcc 0>, <&ipcc 1>; + }; diff --git a/Documentation/devicetree/bindings/marvell.txt b/Documentation/devicetree/bindings/marvell.txt deleted file mode 100644 index 7f722316458a..000000000000 --- a/Documentation/devicetree/bindings/marvell.txt +++ /dev/null @@ -1,516 +0,0 @@ -Marvell Discovery mv64[345]6x System Controller chips -=========================================================== - -The Marvell mv64[345]60 series of system controller chips contain -many of the peripherals needed to implement a complete computer -system. In this section, we define device tree nodes to describe -the system controller chip itself and each of the peripherals -which it contains. Compatible string values for each node are -prefixed with the string "marvell,", for Marvell Technology Group Ltd. - -1) The /system-controller node - - This node is used to represent the system-controller and must be - present when the system uses a system controller chip. The top-level - system-controller node contains information that is global to all - devices within the system controller chip. The node name begins - with "system-controller" followed by the unit address, which is - the base address of the memory-mapped register set for the system - controller chip. - - Required properties: - - - ranges : Describes the translation of system controller addresses - for memory mapped registers. - - clock-frequency: Contains the main clock frequency for the system - controller chip. - - reg : This property defines the address and size of the - memory-mapped registers contained within the system controller - chip. The address specified in the "reg" property should match - the unit address of the system-controller node. - - #address-cells : Address representation for system controller - devices. This field represents the number of cells needed to - represent the address of the memory-mapped registers of devices - within the system controller chip. - - #size-cells : Size representation for the memory-mapped - registers within the system controller chip. - - #interrupt-cells : Defines the width of cells used to represent - interrupts. - - Optional properties: - - - model : The specific model of the system controller chip. Such - as, "mv64360", "mv64460", or "mv64560". - - compatible : A string identifying the compatibility identifiers - of the system controller chip. - - The system-controller node contains child nodes for each system - controller device that the platform uses. Nodes should not be created - for devices which exist on the system controller chip but are not used - - Example Marvell Discovery mv64360 system-controller node: - - system-controller@f1000000 { /* Marvell Discovery mv64360 */ - #address-cells = <1>; - #size-cells = <1>; - model = "mv64360"; /* Default */ - compatible = "marvell,mv64360"; - clock-frequency = <133333333>; - reg = <0xf1000000 0x10000>; - virtual-reg = <0xf1000000>; - ranges = <0x88000000 0x88000000 0x1000000 /* PCI 0 I/O Space */ - 0x80000000 0x80000000 0x8000000 /* PCI 0 MEM Space */ - 0xa0000000 0xa0000000 0x4000000 /* User FLASH */ - 0x00000000 0xf1000000 0x0010000 /* Bridge's regs */ - 0xf2000000 0xf2000000 0x0040000>;/* Integrated SRAM */ - - [ child node definitions... ] - } - -2) Child nodes of /system-controller - - a) Marvell Discovery MDIO bus - - The MDIO is a bus to which the PHY devices are connected. For each - device that exists on this bus, a child node should be created. See - the definition of the PHY node below for an example of how to define - a PHY. - - Required properties: - - #address-cells : Should be <1> - - #size-cells : Should be <0> - - compatible : Should be "marvell,mv64360-mdio" - - Example: - - mdio { - #address-cells = <1>; - #size-cells = <0>; - compatible = "marvell,mv64360-mdio"; - - ethernet-phy@0 { - ...... - }; - }; - - - b) Marvell Discovery ethernet controller - - The Discover ethernet controller is described with two levels - of nodes. The first level describes an ethernet silicon block - and the second level describes up to 3 ethernet nodes within - that block. The reason for the multiple levels is that the - registers for the node are interleaved within a single set - of registers. The "ethernet-block" level describes the - shared register set, and the "ethernet" nodes describe ethernet - port-specific properties. - - Ethernet block node - - Required properties: - - #address-cells : <1> - - #size-cells : <0> - - compatible : "marvell,mv64360-eth-block" - - reg : Offset and length of the register set for this block - - Optional properties: - - clocks : Phandle to the clock control device and gate bit - - Example Discovery Ethernet block node: - ethernet-block@2000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "marvell,mv64360-eth-block"; - reg = <0x2000 0x2000>; - ethernet@0 { - ....... - }; - }; - - Ethernet port node - - Required properties: - - compatible : Should be "marvell,mv64360-eth". - - reg : Should be <0>, <1>, or <2>, according to which registers - within the silicon block the device uses. - - interrupts : where a is the interrupt number for the port. - - interrupt-parent : the phandle for the interrupt controller - that services interrupts for this device. - - phy : the phandle for the PHY connected to this ethernet - controller. - - local-mac-address : 6 bytes, MAC address - - Example Discovery Ethernet port node: - ethernet@0 { - compatible = "marvell,mv64360-eth"; - reg = <0>; - interrupts = <32>; - interrupt-parent = <&PIC>; - phy = <&PHY0>; - local-mac-address = [ 00 00 00 00 00 00 ]; - }; - - - - c) Marvell Discovery PHY nodes - - Required properties: - - interrupts : where a is the interrupt number for this phy. - - interrupt-parent : the phandle for the interrupt controller that - services interrupts for this device. - - reg : The ID number for the phy, usually a small integer - - Example Discovery PHY node: - ethernet-phy@1 { - compatible = "broadcom,bcm5421"; - interrupts = <76>; /* GPP 12 */ - interrupt-parent = <&PIC>; - reg = <1>; - }; - - - d) Marvell Discovery SDMA nodes - - Represent DMA hardware associated with the MPSC (multiprotocol - serial controllers). - - Required properties: - - compatible : "marvell,mv64360-sdma" - - reg : Offset and length of the register set for this device - - interrupts : where a is the interrupt number for the DMA - device. - - interrupt-parent : the phandle for the interrupt controller - that services interrupts for this device. - - Example Discovery SDMA node: - sdma@4000 { - compatible = "marvell,mv64360-sdma"; - reg = <0x4000 0xc18>; - virtual-reg = <0xf1004000>; - interrupts = <36>; - interrupt-parent = <&PIC>; - }; - - - e) Marvell Discovery BRG nodes - - Represent baud rate generator hardware associated with the MPSC - (multiprotocol serial controllers). - - Required properties: - - compatible : "marvell,mv64360-brg" - - reg : Offset and length of the register set for this device - - clock-src : A value from 0 to 15 which selects the clock - source for the baud rate generator. This value corresponds - to the CLKS value in the BRGx configuration register. See - the mv64x60 User's Manual. - - clock-frequence : The frequency (in Hz) of the baud rate - generator's input clock. - - current-speed : The current speed setting (presumably by - firmware) of the baud rate generator. - - Example Discovery BRG node: - brg@b200 { - compatible = "marvell,mv64360-brg"; - reg = <0xb200 0x8>; - clock-src = <8>; - clock-frequency = <133333333>; - current-speed = <9600>; - }; - - - f) Marvell Discovery CUNIT nodes - - Represent the Serial Communications Unit device hardware. - - Required properties: - - reg : Offset and length of the register set for this device - - Example Discovery CUNIT node: - cunit@f200 { - reg = <0xf200 0x200>; - }; - - - g) Marvell Discovery MPSCROUTING nodes - - Represent the Discovery's MPSC routing hardware - - Required properties: - - reg : Offset and length of the register set for this device - - Example Discovery CUNIT node: - mpscrouting@b500 { - reg = <0xb400 0xc>; - }; - - - h) Marvell Discovery MPSCINTR nodes - - Represent the Discovery's MPSC DMA interrupt hardware registers - (SDMA cause and mask registers). - - Required properties: - - reg : Offset and length of the register set for this device - - Example Discovery MPSCINTR node: - mpsintr@b800 { - reg = <0xb800 0x100>; - }; - - - i) Marvell Discovery MPSC nodes - - Represent the Discovery's MPSC (Multiprotocol Serial Controller) - serial port. - - Required properties: - - compatible : "marvell,mv64360-mpsc" - - reg : Offset and length of the register set for this device - - sdma : the phandle for the SDMA node used by this port - - brg : the phandle for the BRG node used by this port - - cunit : the phandle for the CUNIT node used by this port - - mpscrouting : the phandle for the MPSCROUTING node used by this port - - mpscintr : the phandle for the MPSCINTR node used by this port - - cell-index : the hardware index of this cell in the MPSC core - - max_idle : value needed for MPSC CHR3 (Maximum Frame Length) - register - - interrupts : where a is the interrupt number for the MPSC. - - interrupt-parent : the phandle for the interrupt controller - that services interrupts for this device. - - Example Discovery MPSCINTR node: - mpsc@8000 { - compatible = "marvell,mv64360-mpsc"; - reg = <0x8000 0x38>; - virtual-reg = <0xf1008000>; - sdma = <&SDMA0>; - brg = <&BRG0>; - cunit = <&CUNIT>; - mpscrouting = <&MPSCROUTING>; - mpscintr = <&MPSCINTR>; - cell-index = <0>; - max_idle = <40>; - interrupts = <40>; - interrupt-parent = <&PIC>; - }; - - - j) Marvell Discovery Watch Dog Timer nodes - - Represent the Discovery's watchdog timer hardware - - Required properties: - - compatible : "marvell,mv64360-wdt" - - reg : Offset and length of the register set for this device - - Example Discovery Watch Dog Timer node: - wdt@b410 { - compatible = "marvell,mv64360-wdt"; - reg = <0xb410 0x8>; - }; - - - k) Marvell Discovery I2C nodes - - Represent the Discovery's I2C hardware - - Required properties: - - device_type : "i2c" - - compatible : "marvell,mv64360-i2c" - - reg : Offset and length of the register set for this device - - interrupts : where a is the interrupt number for the I2C. - - interrupt-parent : the phandle for the interrupt controller - that services interrupts for this device. - - Example Discovery I2C node: - compatible = "marvell,mv64360-i2c"; - reg = <0xc000 0x20>; - virtual-reg = <0xf100c000>; - interrupts = <37>; - interrupt-parent = <&PIC>; - }; - - - l) Marvell Discovery PIC (Programmable Interrupt Controller) nodes - - Represent the Discovery's PIC hardware - - Required properties: - - #interrupt-cells : <1> - - #address-cells : <0> - - compatible : "marvell,mv64360-pic" - - reg : Offset and length of the register set for this device - - interrupt-controller - - Example Discovery PIC node: - pic { - #interrupt-cells = <1>; - #address-cells = <0>; - compatible = "marvell,mv64360-pic"; - reg = <0x0 0x88>; - interrupt-controller; - }; - - - m) Marvell Discovery MPP (Multipurpose Pins) multiplexing nodes - - Represent the Discovery's MPP hardware - - Required properties: - - compatible : "marvell,mv64360-mpp" - - reg : Offset and length of the register set for this device - - Example Discovery MPP node: - mpp@f000 { - compatible = "marvell,mv64360-mpp"; - reg = <0xf000 0x10>; - }; - - - n) Marvell Discovery GPP (General Purpose Pins) nodes - - Represent the Discovery's GPP hardware - - Required properties: - - compatible : "marvell,mv64360-gpp" - - reg : Offset and length of the register set for this device - - Example Discovery GPP node: - gpp@f000 { - compatible = "marvell,mv64360-gpp"; - reg = <0xf100 0x20>; - }; - - - o) Marvell Discovery PCI host bridge node - - Represents the Discovery's PCI host bridge device. The properties - for this node conform to Rev 2.1 of the PCI Bus Binding to IEEE - 1275-1994. A typical value for the compatible property is - "marvell,mv64360-pci". - - Example Discovery PCI host bridge node - pci@80000000 { - #address-cells = <3>; - #size-cells = <2>; - #interrupt-cells = <1>; - device_type = "pci"; - compatible = "marvell,mv64360-pci"; - reg = <0xcf8 0x8>; - ranges = <0x01000000 0x0 0x0 - 0x88000000 0x0 0x01000000 - 0x02000000 0x0 0x80000000 - 0x80000000 0x0 0x08000000>; - bus-range = <0 255>; - clock-frequency = <66000000>; - interrupt-parent = <&PIC>; - interrupt-map-mask = <0xf800 0x0 0x0 0x7>; - interrupt-map = < - /* IDSEL 0x0a */ - 0x5000 0 0 1 &PIC 80 - 0x5000 0 0 2 &PIC 81 - 0x5000 0 0 3 &PIC 91 - 0x5000 0 0 4 &PIC 93 - - /* IDSEL 0x0b */ - 0x5800 0 0 1 &PIC 91 - 0x5800 0 0 2 &PIC 93 - 0x5800 0 0 3 &PIC 80 - 0x5800 0 0 4 &PIC 81 - - /* IDSEL 0x0c */ - 0x6000 0 0 1 &PIC 91 - 0x6000 0 0 2 &PIC 93 - 0x6000 0 0 3 &PIC 80 - 0x6000 0 0 4 &PIC 81 - - /* IDSEL 0x0d */ - 0x6800 0 0 1 &PIC 93 - 0x6800 0 0 2 &PIC 80 - 0x6800 0 0 3 &PIC 81 - 0x6800 0 0 4 &PIC 91 - >; - }; - - - p) Marvell Discovery CPU Error nodes - - Represent the Discovery's CPU error handler device. - - Required properties: - - compatible : "marvell,mv64360-cpu-error" - - reg : Offset and length of the register set for this device - - interrupts : the interrupt number for this device - - interrupt-parent : the phandle for the interrupt controller - that services interrupts for this device. - - Example Discovery CPU Error node: - cpu-error@70 { - compatible = "marvell,mv64360-cpu-error"; - reg = <0x70 0x10 0x128 0x28>; - interrupts = <3>; - interrupt-parent = <&PIC>; - }; - - - q) Marvell Discovery SRAM Controller nodes - - Represent the Discovery's SRAM controller device. - - Required properties: - - compatible : "marvell,mv64360-sram-ctrl" - - reg : Offset and length of the register set for this device - - interrupts : the interrupt number for this device - - interrupt-parent : the phandle for the interrupt controller - that services interrupts for this device. - - Example Discovery SRAM Controller node: - sram-ctrl@380 { - compatible = "marvell,mv64360-sram-ctrl"; - reg = <0x380 0x80>; - interrupts = <13>; - interrupt-parent = <&PIC>; - }; - - - r) Marvell Discovery PCI Error Handler nodes - - Represent the Discovery's PCI error handler device. - - Required properties: - - compatible : "marvell,mv64360-pci-error" - - reg : Offset and length of the register set for this device - - interrupts : the interrupt number for this device - - interrupt-parent : the phandle for the interrupt controller - that services interrupts for this device. - - Example Discovery PCI Error Handler node: - pci-error@1d40 { - compatible = "marvell,mv64360-pci-error"; - reg = <0x1d40 0x40 0xc28 0x4>; - interrupts = <12>; - interrupt-parent = <&PIC>; - }; - - - s) Marvell Discovery Memory Controller nodes - - Represent the Discovery's memory controller device. - - Required properties: - - compatible : "marvell,mv64360-mem-ctrl" - - reg : Offset and length of the register set for this device - - interrupts : the interrupt number for this device - - interrupt-parent : the phandle for the interrupt controller - that services interrupts for this device. - - Example Discovery Memory Controller node: - mem-ctrl@1400 { - compatible = "marvell,mv64360-mem-ctrl"; - reg = <0x1400 0x60>; - interrupts = <17>; - interrupt-parent = <&PIC>; - }; - - diff --git a/Documentation/devicetree/bindings/media/cdns,csi2rx.txt b/Documentation/devicetree/bindings/media/cdns,csi2rx.txt new file mode 100644 index 000000000000..6b02a0657ad9 --- /dev/null +++ b/Documentation/devicetree/bindings/media/cdns,csi2rx.txt @@ -0,0 +1,100 @@ +Cadence MIPI-CSI2 RX controller +=============================== + +The Cadence MIPI-CSI2 RX controller is a CSI-2 bridge supporting up to 4 CSI +lanes in input, and 4 different pixel streams in output. + +Required properties: + - compatible: must be set to "cdns,csi2rx" and an SoC-specific compatible + - reg: base address and size of the memory mapped region + - clocks: phandles to the clocks driving the controller + - clock-names: must contain: + * sys_clk: main clock + * p_clk: register bank clock + * pixel_if[0-3]_clk: pixel stream output clock, one for each stream + implemented in hardware, between 0 and 3 + +Optional properties: + - phys: phandle to the external D-PHY, phy-names must be provided + - phy-names: must contain "dphy", if the implementation uses an + external D-PHY + +Required subnodes: + - ports: A ports node with one port child node per device input and output + port, in accordance with the video interface bindings defined in + Documentation/devicetree/bindings/media/video-interfaces.txt. The + port nodes are numbered as follows: + + Port Description + ----------------------------- + 0 CSI-2 input + 1 Stream 0 output + 2 Stream 1 output + 3 Stream 2 output + 4 Stream 3 output + + The stream output port nodes are optional if they are not + connected to anything at the hardware level or implemented + in the design.Since there is only one endpoint per port, + the endpoints are not numbered. + + +Example: + +csi2rx: csi-bridge@0d060000 { + compatible = "cdns,csi2rx"; + reg = <0x0d060000 0x1000>; + clocks = <&byteclock>, <&byteclock> + <&coreclock>, <&coreclock>, + <&coreclock>, <&coreclock>; + clock-names = "sys_clk", "p_clk", + "pixel_if0_clk", "pixel_if1_clk", + "pixel_if2_clk", "pixel_if3_clk"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + + csi2rx_in_sensor: endpoint { + remote-endpoint = <&sensor_out_csi2rx>; + clock-lanes = <0>; + data-lanes = <1 2>; + }; + }; + + port@1 { + reg = <1>; + + csi2rx_out_grabber0: endpoint { + remote-endpoint = <&grabber0_in_csi2rx>; + }; + }; + + port@2 { + reg = <2>; + + csi2rx_out_grabber1: endpoint { + remote-endpoint = <&grabber1_in_csi2rx>; + }; + }; + + port@3 { + reg = <3>; + + csi2rx_out_grabber2: endpoint { + remote-endpoint = <&grabber2_in_csi2rx>; + }; + }; + + port@4 { + reg = <4>; + + csi2rx_out_grabber3: endpoint { + remote-endpoint = <&grabber3_in_csi2rx>; + }; + }; + }; +}; diff --git a/Documentation/devicetree/bindings/media/cdns,csi2tx.txt b/Documentation/devicetree/bindings/media/cdns,csi2tx.txt new file mode 100644 index 000000000000..459c6e332f52 --- /dev/null +++ b/Documentation/devicetree/bindings/media/cdns,csi2tx.txt @@ -0,0 +1,98 @@ +Cadence MIPI-CSI2 TX controller +=============================== + +The Cadence MIPI-CSI2 TX controller is a CSI-2 bridge supporting up to +4 CSI lanes in output, and up to 4 different pixel streams in input. + +Required properties: + - compatible: must be set to "cdns,csi2tx" + - reg: base address and size of the memory mapped region + - clocks: phandles to the clocks driving the controller + - clock-names: must contain: + * esc_clk: escape mode clock + * p_clk: register bank clock + * pixel_if[0-3]_clk: pixel stream output clock, one for each stream + implemented in hardware, between 0 and 3 + +Optional properties + - phys: phandle to the D-PHY. If it is set, phy-names need to be set + - phy-names: must contain "dphy" + +Required subnodes: + - ports: A ports node with one port child node per device input and output + port, in accordance with the video interface bindings defined in + Documentation/devicetree/bindings/media/video-interfaces.txt. The + port nodes are numbered as follows. + + Port Description + ----------------------------- + 0 CSI-2 output + 1 Stream 0 input + 2 Stream 1 input + 3 Stream 2 input + 4 Stream 3 input + + The stream input port nodes are optional if they are not + connected to anything at the hardware level or implemented + in the design. Since there is only one endpoint per port, + the endpoints are not numbered. + +Example: + +csi2tx: csi-bridge@0d0e1000 { + compatible = "cdns,csi2tx"; + reg = <0x0d0e1000 0x1000>; + clocks = <&byteclock>, <&byteclock>, + <&coreclock>, <&coreclock>, + <&coreclock>, <&coreclock>; + clock-names = "p_clk", "esc_clk", + "pixel_if0_clk", "pixel_if1_clk", + "pixel_if2_clk", "pixel_if3_clk"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + + csi2tx_out: endpoint { + remote-endpoint = <&remote_in>; + clock-lanes = <0>; + data-lanes = <1 2>; + }; + }; + + port@1 { + reg = <1>; + + csi2tx_in_stream0: endpoint { + remote-endpoint = <&stream0_out>; + }; + }; + + port@2 { + reg = <2>; + + csi2tx_in_stream1: endpoint { + remote-endpoint = <&stream1_out>; + }; + }; + + port@3 { + reg = <3>; + + csi2tx_in_stream2: endpoint { + remote-endpoint = <&stream2_out>; + }; + }; + + port@4 { + reg = <4>; + + csi2tx_in_stream3: endpoint { + remote-endpoint = <&stream3_out>; + }; + }; + }; +}; diff --git a/Documentation/devicetree/bindings/media/i2c/ov7251.txt b/Documentation/devicetree/bindings/media/i2c/ov7251.txt new file mode 100644 index 000000000000..8281151f7493 --- /dev/null +++ b/Documentation/devicetree/bindings/media/i2c/ov7251.txt @@ -0,0 +1,52 @@ +* Omnivision 1/7.5-Inch B&W VGA CMOS Digital Image Sensor + +The Omnivision OV7251 is a 1/7.5-Inch CMOS active pixel digital image sensor +with an active array size of 640H x 480V. It is programmable through a serial +I2C interface. + +Required Properties: +- compatible: Value should be "ovti,ov7251". +- clocks: Reference to the xclk clock. +- clock-names: Should be "xclk". +- clock-frequency: Frequency of the xclk clock. +- enable-gpios: Chip enable GPIO. Polarity is GPIO_ACTIVE_HIGH. This corresponds + to the hardware pin XSHUTDOWN which is physically active low. +- vdddo-supply: Chip digital IO regulator. +- vdda-supply: Chip analog regulator. +- vddd-supply: Chip digital core regulator. + +The device node shall contain one 'port' child node with a single 'endpoint' +subnode for its digital output video port, in accordance with the video +interface bindings defined in +Documentation/devicetree/bindings/media/video-interfaces.txt. + +Example: + + &i2c1 { + ... + + ov7251: camera-sensor@60 { + compatible = "ovti,ov7251"; + reg = <0x60>; + + enable-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>; + pinctrl-names = "default"; + pinctrl-0 = <&camera_bw_default>; + + clocks = <&clks 200>; + clock-names = "xclk"; + clock-frequency = <24000000>; + + vdddo-supply = <&camera_dovdd_1v8>; + vdda-supply = <&camera_avdd_2v8>; + vddd-supply = <&camera_dvdd_1v2>; + + port { + ov7251_ep: endpoint { + clock-lanes = <1>; + data-lanes = <0>; + remote-endpoint = <&csi0_ep>; + }; + }; + }; + }; diff --git a/Documentation/devicetree/bindings/media/i2c/ov772x.txt b/Documentation/devicetree/bindings/media/i2c/ov772x.txt new file mode 100644 index 000000000000..0b3ede5b8e6a --- /dev/null +++ b/Documentation/devicetree/bindings/media/i2c/ov772x.txt @@ -0,0 +1,40 @@ +* Omnivision OV7720/OV7725 CMOS sensor + +The Omnivision OV7720/OV7725 sensor supports multiple resolutions output, +such as VGA, QVGA, and any size scaling down from CIF to 40x30. It also can +support the YUV422, RGB565/555/444, GRB422 or raw RGB output formats. + +Required Properties: +- compatible: shall be one of + "ovti,ov7720" + "ovti,ov7725" +- clocks: reference to the xclk input clock. + +Optional Properties: +- reset-gpios: reference to the GPIO connected to the RSTB pin which is + active low, if any. +- powerdown-gpios: reference to the GPIO connected to the PWDN pin which is + active high, if any. + +The device node shall contain one 'port' child node with one child 'endpoint' +subnode for its digital output video port, in accordance with the video +interface bindings defined in Documentation/devicetree/bindings/media/ +video-interfaces.txt. + +Example: + +&i2c0 { + ov772x: camera@21 { + compatible = "ovti,ov7725"; + reg = <0x21>; + reset-gpios = <&axi_gpio_0 0 GPIO_ACTIVE_LOW>; + powerdown-gpios = <&axi_gpio_0 1 GPIO_ACTIVE_LOW>; + clocks = <&xclk>; + + port { + ov772x_0: endpoint { + remote-endpoint = <&vcap1_in0>; + }; + }; + }; +}; diff --git a/Documentation/devicetree/bindings/media/i2c/panasonic,amg88xx.txt b/Documentation/devicetree/bindings/media/i2c/panasonic,amg88xx.txt new file mode 100644 index 000000000000..4a3181a3dd7e --- /dev/null +++ b/Documentation/devicetree/bindings/media/i2c/panasonic,amg88xx.txt @@ -0,0 +1,19 @@ +* Panasonic AMG88xx + +The Panasonic family of AMG88xx Grid-Eye sensors allow recording +8x8 10Hz video which consists of thermal datapoints + +Required Properties: + - compatible : Must be "panasonic,amg88xx" + - reg : i2c address of the device + +Example: + + i2c0@1c22000 { + ... + amg88xx@69 { + compatible = "panasonic,amg88xx"; + reg = <0x69>; + }; + ... + }; diff --git a/Documentation/devicetree/bindings/media/rcar_vin.txt b/Documentation/devicetree/bindings/media/rcar_vin.txt index 1ce7ff9449c5..a19517e1c669 100644 --- a/Documentation/devicetree/bindings/media/rcar_vin.txt +++ b/Documentation/devicetree/bindings/media/rcar_vin.txt @@ -2,20 +2,28 @@ Renesas R-Car Video Input driver (rcar_vin) ------------------------------------------- The rcar_vin device provides video input capabilities for the Renesas R-Car -family of devices. The current blocks are always slaves and suppot one input -channel which can be either RGB, YUYV or BT656. +family of devices. + +Each VIN instance has a single parallel input that supports RGB and YUV video, +with both external synchronization and BT.656 synchronization for the latter. +Depending on the instance the VIN input is connected to external SoC pins, or +on Gen3 platforms to a CSI-2 receiver. - compatible: Must be one or more of the following - - "renesas,vin-r8a7795" for the R8A7795 device - - "renesas,vin-r8a7794" for the R8A7794 device - - "renesas,vin-r8a7793" for the R8A7793 device - - "renesas,vin-r8a7792" for the R8A7792 device - - "renesas,vin-r8a7791" for the R8A7791 device - - "renesas,vin-r8a7790" for the R8A7790 device - - "renesas,vin-r8a7779" for the R8A7779 device + - "renesas,vin-r8a7743" for the R8A7743 device + - "renesas,vin-r8a7745" for the R8A7745 device - "renesas,vin-r8a7778" for the R8A7778 device - - "renesas,rcar-gen2-vin" for a generic R-Car Gen2 compatible device. - - "renesas,rcar-gen3-vin" for a generic R-Car Gen3 compatible device. + - "renesas,vin-r8a7779" for the R8A7779 device + - "renesas,vin-r8a7790" for the R8A7790 device + - "renesas,vin-r8a7791" for the R8A7791 device + - "renesas,vin-r8a7792" for the R8A7792 device + - "renesas,vin-r8a7793" for the R8A7793 device + - "renesas,vin-r8a7794" for the R8A7794 device + - "renesas,vin-r8a7795" for the R8A7795 device + - "renesas,vin-r8a7796" for the R8A7796 device + - "renesas,vin-r8a77970" for the R8A77970 device + - "renesas,rcar-gen2-vin" for a generic R-Car Gen2 or RZ/G1 compatible + device. When compatible with the generic version nodes must list the SoC-specific version corresponding to the platform first @@ -28,21 +36,38 @@ channel which can be either RGB, YUYV or BT656. Additionally, an alias named vinX will need to be created to specify which video input device this is. -The per-board settings: +The per-board settings Gen2 platforms: - port sub-node describing a single endpoint connected to the vin as described in video-interfaces.txt[1]. Only the first one will be considered as each vin interface has one input port. - These settings are used to work out video input format and widths - into the system. +The per-board settings Gen3 platforms: +Gen3 platforms can support both a single connected parallel input source +from external SoC pins (port0) and/or multiple parallel input sources +from local SoC CSI-2 receivers (port1) depending on SoC. -Device node example -------------------- +- renesas,id - ID number of the VIN, VINx in the documentation. +- ports + - port 0 - sub-node describing a single endpoint connected to the VIN + from external SoC pins described in video-interfaces.txt[1]. + Describing more then one endpoint in port 0 is invalid. Only VIN + instances that are connected to external pins should have port 0. + - port 1 - sub-nodes describing one or more endpoints connected to + the VIN from local SoC CSI-2 receivers. The endpoint numbers must + use the following schema. - aliases { - vin0 = &vin0; - }; + - Endpoint 0 - sub-node describing the endpoint connected to CSI20 + - Endpoint 1 - sub-node describing the endpoint connected to CSI21 + - Endpoint 2 - sub-node describing the endpoint connected to CSI40 + - Endpoint 3 - sub-node describing the endpoint connected to CSI41 + +Device node example for Gen2 platforms +-------------------------------------- + + aliases { + vin0 = &vin0; + }; vin0: vin@e6ef0000 { compatible = "renesas,vin-r8a7790", "renesas,rcar-gen2-vin"; @@ -52,8 +77,8 @@ Device node example status = "disabled"; }; -Board setup example (vin1 composite video input) ------------------------------------------------- +Board setup example for Gen2 platforms (vin1 composite video input) +------------------------------------------------------------------- &i2c2 { status = "okay"; @@ -92,6 +117,77 @@ Board setup example (vin1 composite video input) }; }; +Device node example for Gen3 platforms +-------------------------------------- + + vin0: video@e6ef0000 { + compatible = "renesas,vin-r8a7795"; + reg = <0 0xe6ef0000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 811>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 811>; + renesas,id = <0>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin0csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin0>; + }; + vin0csi21: endpoint@1 { + reg = <1>; + remote-endpoint= <&csi21vin0>; + }; + vin0csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin0>; + }; + }; + }; + }; + csi20: csi2@fea80000 { + compatible = "renesas,r8a7795-csi2"; + reg = <0 0xfea80000 0 0x10000>; + interrupts = ; + clocks = <&cpg CPG_MOD 714>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 714>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + csi20_in: endpoint { + clock-lanes = <0>; + data-lanes = <1>; + remote-endpoint = <&adv7482_txb>; + }; + }; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + csi20vin0: endpoint@0 { + reg = <0>; + remote-endpoint = <&vin0csi20>; + }; + }; + }; + }; [1] video-interfaces.txt common video media interface diff --git a/Documentation/devicetree/bindings/media/renesas,ceu.txt b/Documentation/devicetree/bindings/media/renesas,ceu.txt index 3fc66dfb192c..8a7a616e9019 100644 --- a/Documentation/devicetree/bindings/media/renesas,ceu.txt +++ b/Documentation/devicetree/bindings/media/renesas,ceu.txt @@ -2,14 +2,15 @@ Renesas Capture Engine Unit (CEU) ---------------------------------------------- The Capture Engine Unit is the image capture interface found in the Renesas -SH Mobile and RZ SoCs. +SH Mobile, R-Mobile and RZ SoCs. The interface supports a single parallel input with data bus width of 8 or 16 bits. Required properties: -- compatible: Shall be "renesas,r7s72100-ceu" for CEU units found in RZ/A1H - and RZ/A1M SoCs. +- compatible: Shall be one of the following values: + "renesas,r7s72100-ceu" for CEU units found in RZ/A1H and RZ/A1M SoCs + "renesas,r8a7740-ceu" for CEU units found in R-Mobile A1 R8A7740 SoCs - reg: Registers address base and size. - interrupts: The interrupt specifier. diff --git a/Documentation/devicetree/bindings/media/renesas,rcar-csi2.txt b/Documentation/devicetree/bindings/media/renesas,rcar-csi2.txt new file mode 100644 index 000000000000..2d385b65b275 --- /dev/null +++ b/Documentation/devicetree/bindings/media/renesas,rcar-csi2.txt @@ -0,0 +1,101 @@ +Renesas R-Car MIPI CSI-2 +------------------------ + +The R-Car CSI-2 receiver device provides MIPI CSI-2 capabilities for the +Renesas R-Car family of devices. It is used in conjunction with the +R-Car VIN module, which provides the video capture capabilities. + +Mandatory properties +-------------------- + - compatible: Must be one or more of the following + - "renesas,r8a7795-csi2" for the R8A7795 device. + - "renesas,r8a7796-csi2" for the R8A7796 device. + - "renesas,r8a77965-csi2" for the R8A77965 device. + - "renesas,r8a77970-csi2" for the R8A77970 device. + + - reg: the register base and size for the device registers + - interrupts: the interrupt for the device + - clocks: reference to the parent clock + +The device node shall contain two 'port' child nodes according to the +bindings defined in Documentation/devicetree/bindings/media/ +video-interfaces.txt. port@0 shall connect to the CSI-2 source. port@1 +shall connect to all the R-Car VIN modules that have a hardware +connection to the CSI-2 receiver. + +- port@0- Video source (mandatory) + - endpoint@0 - sub-node describing the endpoint that is the video source + +- port@1 - VIN instances (optional) + - One endpoint sub-node for every R-Car VIN instance which is connected + to the R-Car CSI-2 receiver. + +Example: + + csi20: csi2@fea80000 { + compatible = "renesas,r8a7796-csi2"; + reg = <0 0xfea80000 0 0x10000>; + interrupts = <0 184 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&cpg CPG_MOD 714>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 714>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <0>; + + csi20_in: endpoint@0 { + reg = <0>; + clock-lanes = <0>; + data-lanes = <1>; + remote-endpoint = <&adv7482_txb>; + }; + }; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + csi20vin0: endpoint@0 { + reg = <0>; + remote-endpoint = <&vin0csi20>; + }; + csi20vin1: endpoint@1 { + reg = <1>; + remote-endpoint = <&vin1csi20>; + }; + csi20vin2: endpoint@2 { + reg = <2>; + remote-endpoint = <&vin2csi20>; + }; + csi20vin3: endpoint@3 { + reg = <3>; + remote-endpoint = <&vin3csi20>; + }; + csi20vin4: endpoint@4 { + reg = <4>; + remote-endpoint = <&vin4csi20>; + }; + csi20vin5: endpoint@5 { + reg = <5>; + remote-endpoint = <&vin5csi20>; + }; + csi20vin6: endpoint@6 { + reg = <6>; + remote-endpoint = <&vin6csi20>; + }; + csi20vin7: endpoint@7 { + reg = <7>; + remote-endpoint = <&vin7csi20>; + }; + }; + }; + }; diff --git a/Documentation/devicetree/bindings/media/stih407-c8sectpfe.txt b/Documentation/devicetree/bindings/media/stih407-c8sectpfe.txt index c7888d6f6408..880d4d70c9fd 100644 --- a/Documentation/devicetree/bindings/media/stih407-c8sectpfe.txt +++ b/Documentation/devicetree/bindings/media/stih407-c8sectpfe.txt @@ -28,7 +28,7 @@ See: Documentation/devicetree/bindings/clock/clock-bindings.txt - pinctrl-names : a pinctrl state named tsin%d-serial or tsin%d-parallel (where %d is tsin-num) must be defined for each tsin child node. - pinctrl-0 : phandle referencing pin configuration for this tsin configuration -See: Documentation/devicetree/bindings/pinctrl/pinctrl-binding.txt +See: Documentation/devicetree/bindings/pinctrl/pinctrl-bindings.txt Required properties (tsin (child) node): diff --git a/Documentation/devicetree/bindings/memory-controllers/nvidia,tegra20-mc.txt b/Documentation/devicetree/bindings/memory-controllers/nvidia,tegra20-mc.txt new file mode 100644 index 000000000000..7d60a50a4fa1 --- /dev/null +++ b/Documentation/devicetree/bindings/memory-controllers/nvidia,tegra20-mc.txt @@ -0,0 +1,26 @@ +NVIDIA Tegra20 MC(Memory Controller) + +Required properties: +- compatible : "nvidia,tegra20-mc" +- reg : Should contain 2 register ranges(address and length); see the + example below. Note that the MC registers are interleaved with the + GART registers, and hence must be represented as multiple ranges. +- interrupts : Should contain MC General interrupt. +- #reset-cells : Should be 1. This cell represents memory client module ID. + The assignments may be found in header file + or in the TRM documentation. + +Example: + mc: memory-controller@7000f000 { + compatible = "nvidia,tegra20-mc"; + reg = <0x7000f000 0x024 + 0x7000f03c 0x3c4>; + interrupts = <0 77 0x04>; + #reset-cells = <1>; + }; + + video-codec@6001a000 { + compatible = "nvidia,tegra20-vde"; + ... + resets = <&mc TEGRA20_MC_RESET_VDE>; + }; diff --git a/Documentation/devicetree/bindings/memory-controllers/nvidia,tegra30-mc.txt b/Documentation/devicetree/bindings/memory-controllers/nvidia,tegra30-mc.txt index 14968b048cd3..a878b5908a4d 100644 --- a/Documentation/devicetree/bindings/memory-controllers/nvidia,tegra30-mc.txt +++ b/Documentation/devicetree/bindings/memory-controllers/nvidia,tegra30-mc.txt @@ -12,6 +12,9 @@ Required properties: - clock-names: Must include the following entries: - mc: the module's clock input - interrupts: The interrupt outputs from the controller. +- #reset-cells : Should be 1. This cell represents memory client module ID. + The assignments may be found in header file + or in the TRM documentation. Required properties for Tegra30, Tegra114, Tegra124, Tegra132 and Tegra210: - #iommu-cells: Should be 1. The single cell of the IOMMU specifier defines @@ -72,12 +75,14 @@ Example SoC include file: interrupts = ; #iommu-cells = <1>; + #reset-cells = <1>; }; sdhci@700b0000 { compatible = "nvidia,tegra124-sdhci"; ... iommus = <&mc TEGRA_SWGROUP_SDMMC1A>; + resets = <&mc TEGRA124_MC_RESET_SDMMC1>; }; }; diff --git a/Documentation/devicetree/bindings/mfd/arizona.txt b/Documentation/devicetree/bindings/mfd/arizona.txt index bdd017686ea5..a014afb07902 100644 --- a/Documentation/devicetree/bindings/mfd/arizona.txt +++ b/Documentation/devicetree/bindings/mfd/arizona.txt @@ -50,7 +50,7 @@ Required properties: Optional properties: - - wlf,reset : GPIO specifier for the GPIO controlling /RESET + - reset-gpios : GPIO specifier for the GPIO controlling /RESET - clocks: Should reference the clocks supplied on MCLK1 and MCLK2 - clock-names: Should contains two strings: @@ -70,6 +70,10 @@ Optional properties: Documentation/devicetree/bindings/regulator/regulator.txt (wm5102, wm5110, wm8280, wm8997, wm8998, wm1814) +Deprecated properties: + + - wlf,reset : GPIO specifier for the GPIO controlling /RESET + Also see child specific device properties: Regulator - ../regulator/arizona-regulator.txt Extcon - ../extcon/extcon-arizona.txt diff --git a/Documentation/devicetree/bindings/mfd/as3722.txt b/Documentation/devicetree/bindings/mfd/as3722.txt index 0b2a6099aa20..5297b2210704 100644 --- a/Documentation/devicetree/bindings/mfd/as3722.txt +++ b/Documentation/devicetree/bindings/mfd/as3722.txt @@ -46,7 +46,7 @@ is required: Following properties are require if pin control setting is required at boot. - pinctrl-names: A pinctrl state named "default" be defined, using the - bindings in pinctrl/pinctrl-binding.txt. + bindings in pinctrl/pinctrl-bindings.txt. - pinctrl[0...n]: Properties to contain the phandle that refer to different nodes of pin control settings. These nodes represents the pin control setting of state 0 to state n. Each of these diff --git a/Documentation/devicetree/bindings/mfd/axp20x.txt b/Documentation/devicetree/bindings/mfd/axp20x.txt index 9455503b0299..d1762f3b30af 100644 --- a/Documentation/devicetree/bindings/mfd/axp20x.txt +++ b/Documentation/devicetree/bindings/mfd/axp20x.txt @@ -43,7 +43,7 @@ Optional properties: regulator to drive the OTG VBus, rather then as an input pin which signals whether the board is driving OTG VBus or not. - (axp221 / axp223 / axp813 only) + (axp221 / axp223 / axp803/ axp813 only) - x-powers,master-mode: Boolean (axp806 only). Set this when the PMIC is wired for master mode. The default is slave mode. @@ -132,6 +132,7 @@ FLDO2 : LDO : fldoin-supply : shared supply LDO_IO0 : LDO : ips-supply : GPIO 0 LDO_IO1 : LDO : ips-supply : GPIO 1 RTC_LDO : LDO : ips-supply : always on +DRIVEVBUS : Enable output : drivevbus-supply : external regulator AXP806 regulators, type, and corresponding input supply names: diff --git a/Documentation/devicetree/bindings/mfd/bd9571mwv.txt b/Documentation/devicetree/bindings/mfd/bd9571mwv.txt index 9ab216a851d5..25d1f697eb25 100644 --- a/Documentation/devicetree/bindings/mfd/bd9571mwv.txt +++ b/Documentation/devicetree/bindings/mfd/bd9571mwv.txt @@ -25,6 +25,25 @@ Required properties: Each child node is defined using the standard binding for regulators. +Optional properties: + - rohm,ddr-backup-power : Value to use for DDR-Backup Power (default 0). + This is a bitmask that specifies which DDR power + rails need to be kept powered when backup mode is + entered, for system suspend: + - bit 0: DDR0 + - bit 1: DDR1 + - bit 2: DDR0C + - bit 3: DDR1C + These bits match the KEEPON_DDR* bits in the + documentation for the "BKUP Mode Cnt" register. + - rohm,rstbmode-level: The RSTB signal is configured for level mode, to + accommodate a toggle power switch (the RSTBMODE pin is + strapped low). + - rohm,rstbmode-pulse: The RSTB signal is configured for pulse mode, to + accommodate a momentary power switch (the RSTBMODE pin + is strapped high). + The two properties above are mutually exclusive. + Example: pmic: pmic@30 { @@ -36,6 +55,8 @@ Example: #interrupt-cells = <2>; gpio-controller; #gpio-cells = <2>; + rohm,ddr-backup-power = <0xf>; + rohm,rstbmode-pulse; regulators { dvfs: dvfs { diff --git a/Documentation/devicetree/bindings/mfd/da9063.txt b/Documentation/devicetree/bindings/mfd/da9063.txt index 05b21bcb8543..443e68286957 100644 --- a/Documentation/devicetree/bindings/mfd/da9063.txt +++ b/Documentation/devicetree/bindings/mfd/da9063.txt @@ -1,4 +1,4 @@ -* Dialog DA9063 Power Management Integrated Circuit (PMIC) +* Dialog DA9063/DA9063L Power Management Integrated Circuit (PMIC) DA9093 consists of a large and varied group of sub-devices (I2C Only): @@ -6,14 +6,14 @@ Device Supply Names Description ------ ------------ ----------- da9063-regulator : : LDOs & BUCKs da9063-onkey : : On Key -da9063-rtc : : Real-Time Clock +da9063-rtc : : Real-Time Clock (DA9063 only) da9063-watchdog : : Watchdog ====== Required properties: -- compatible : Should be "dlg,da9063" +- compatible : Should be "dlg,da9063" or "dlg,da9063l" - reg : Specifies the I2C slave address (this defaults to 0x58 but it can be modified to match the chip's OTP settings). - interrupt-parent : Specifies the reference to the interrupt controller for @@ -23,8 +23,8 @@ Required properties: Sub-nodes: -- regulators : This node defines the settings for the LDOs and BUCKs. The - DA9063 regulators are bound using their names listed below: +- regulators : This node defines the settings for the LDOs and BUCKs. + The DA9063(L) regulators are bound using their names listed below: bcore1 : BUCK CORE1 bcore2 : BUCK CORE2 @@ -32,16 +32,16 @@ Sub-nodes: bmem : BUCK MEM bio : BUCK IO bperi : BUCK PERI - ldo1 : LDO_1 - ldo2 : LDO_2 + ldo1 : LDO_1 (DA9063 only) + ldo2 : LDO_2 (DA9063 only) ldo3 : LDO_3 - ldo4 : LDO_4 - ldo5 : LDO_5 - ldo6 : LDO_6 + ldo4 : LDO_4 (DA9063 only) + ldo5 : LDO_5 (DA9063 only) + ldo6 : LDO_6 (DA9063 only) ldo7 : LDO_7 ldo8 : LDO_8 ldo9 : LDO_9 - ldo10 : LDO_10 + ldo10 : LDO_10 (DA9063 only) ldo11 : LDO_11 The component follows the standard regulator framework and the bindings @@ -49,8 +49,9 @@ Sub-nodes: Documentation/devicetree/bindings/regulator/regulator.txt - rtc : This node defines settings for the Real-Time Clock associated with - the DA9063. There are currently no entries in this binding, however - compatible = "dlg,da9063-rtc" should be added if a node is created. + the DA9063 only. The RTC is not present in DA9063L. There are currently + no entries in this binding, however compatible = "dlg,da9063-rtc" should + be added if a node is created. - onkey : This node defines the OnKey settings for controlling the key functionality of the device. The node should contain the compatible property @@ -65,8 +66,9 @@ Sub-nodes: and KEY_SLEEP. - watchdog : This node defines settings for the Watchdog timer associated - with the DA9063. There are currently no entries in this binding, however - compatible = "dlg,da9063-watchdog" should be added if a node is created. + with the DA9063 and DA9063L. There are currently no entries in this + binding, however compatible = "dlg,da9063-watchdog" should be added + if a node is created. Example: diff --git a/Documentation/devicetree/bindings/mfd/motorola-cpcap.txt b/Documentation/devicetree/bindings/mfd/motorola-cpcap.txt index 15bc885f9df4..c639705a98ef 100644 --- a/Documentation/devicetree/bindings/mfd/motorola-cpcap.txt +++ b/Documentation/devicetree/bindings/mfd/motorola-cpcap.txt @@ -12,6 +12,30 @@ Required properties: - spi-max-frequency : Typically set to 3000000 - spi-cs-high : SPI chip select direction +Optional subnodes: + +The sub-functions of CPCAP get their own node with their own compatible values, +which are described in the following files: + +- ../power/supply/cpcap-battery.txt +- ../power/supply/cpcap-charger.txt +- ../regulator/cpcap-regulator.txt +- ../phy/phy-cpcap-usb.txt +- ../input/cpcap-pwrbutton.txt +- ../rtc/cpcap-rtc.txt +- ../leds/leds-cpcap.txt +- ../iio/adc/cpcap-adc.txt + +The only exception is the audio codec. Instead of a compatible value its +node must be named "audio-codec". + +Required properties for the audio-codec subnode: + +- #sound-dai-cells = <1>; + +The audio-codec provides two DAIs. The first one is connected to the +Stereo HiFi DAC and the second one is connected to the Voice DAC. + Example: &mcspi1 { @@ -26,6 +50,24 @@ Example: #size-cells = <0>; spi-max-frequency = <3000000>; spi-cs-high; + + audio-codec { + #sound-dai-cells = <1>; + + /* HiFi */ + port@0 { + endpoint { + remote-endpoint = <&cpu_dai1>; + }; + }; + + /* Voice */ + port@1 { + endpoint { + remote-endpoint = <&cpu_dai2>; + }; + }; + }; }; }; diff --git a/Documentation/devicetree/bindings/mfd/mt6397.txt b/Documentation/devicetree/bindings/mfd/mt6397.txt index 522a3bbf1bac..0ebd08af777d 100644 --- a/Documentation/devicetree/bindings/mfd/mt6397.txt +++ b/Documentation/devicetree/bindings/mfd/mt6397.txt @@ -7,11 +7,12 @@ MT6397/MT6323 is a multifunction device with the following sub modules: - GPIO - Clock - LED +- Keys It is interfaced to host controller using SPI interface by a proprietary hardware called PMIC wrapper or pwrap. MT6397/MT6323 MFD is a child device of pwrap. See the following for pwarp node definitions: -Documentation/devicetree/bindings/soc/pwrap.txt +Documentation/devicetree/bindings/soc/mediatek/pwrap.txt This document describes the binding for MFD device and its sub module. @@ -40,6 +41,11 @@ Optional subnodes: - compatible: "mediatek,mt6323-led" see Documentation/devicetree/bindings/leds/leds-mt6323.txt +- keys + Required properties: + - compatible: "mediatek,mt6397-keys" or "mediatek,mt6323-keys" + see Documentation/devicetree/bindings/input/mtk-pmic-keys.txt + Example: pwrap: pwrap@1000f000 { compatible = "mediatek,mt8135-pwrap"; diff --git a/Documentation/devicetree/bindings/mfd/qcom,spmi-pmic.txt b/Documentation/devicetree/bindings/mfd/qcom,spmi-pmic.txt index 6ac06c1b9aec..143706222a51 100644 --- a/Documentation/devicetree/bindings/mfd/qcom,spmi-pmic.txt +++ b/Documentation/devicetree/bindings/mfd/qcom,spmi-pmic.txt @@ -29,6 +29,9 @@ Required properties: "qcom,pm8916", "qcom,pm8004", "qcom,pm8909", + "qcom,pm8998", + "qcom,pmi8998", + "qcom,pm8005", or generalized "qcom,spmi-pmic". - reg: Specifies the SPMI USID slave address for this device. For more information see: diff --git a/Documentation/devicetree/bindings/mfd/stm32-timers.txt b/Documentation/devicetree/bindings/mfd/stm32-timers.txt index 1db6e0057a63..0e900b52e895 100644 --- a/Documentation/devicetree/bindings/mfd/stm32-timers.txt +++ b/Documentation/devicetree/bindings/mfd/stm32-timers.txt @@ -19,6 +19,11 @@ Required parameters: Optional parameters: - resets: Phandle to the parent reset controller. See ../reset/st,stm32-rcc.txt +- dmas: List of phandle to dma channels that can be used for + this timer instance. There may be up to 7 dma channels. +- dma-names: List of dma names. Must match 'dmas' property. Valid + names are: "ch1", "ch2", "ch3", "ch4", "up", "trig", + "com". Optional subnodes: - pwm: See ../pwm/pwm-stm32.txt @@ -44,3 +49,18 @@ Example: reg = <0>; }; }; + +Example with all dmas: + timer@40010000 { + ... + dmas = <&dmamux1 11 0x400 0x0>, + <&dmamux1 12 0x400 0x0>, + <&dmamux1 13 0x400 0x0>, + <&dmamux1 14 0x400 0x0>, + <&dmamux1 15 0x400 0x0>, + <&dmamux1 16 0x400 0x0>, + <&dmamux1 17 0x400 0x0>; + dma-names = "ch1", "ch2", "ch3", "ch4", "up", "trig", "com"; + ... + child nodes... + }; diff --git a/Documentation/devicetree/bindings/mfd/sun6i-prcm.txt b/Documentation/devicetree/bindings/mfd/sun6i-prcm.txt index dd2c06540485..daa091c2e67b 100644 --- a/Documentation/devicetree/bindings/mfd/sun6i-prcm.txt +++ b/Documentation/devicetree/bindings/mfd/sun6i-prcm.txt @@ -8,8 +8,8 @@ Required properties: - reg: The PRCM registers range The prcm node may contain several subdevices definitions: - - see Documentation/devicetree/clk/sunxi.txt for clock devices - - see Documentation/devicetree/reset/allwinner,sunxi-clock-reset.txt for reset + - see Documentation/devicetree/bindings/clock/sunxi.txt for clock devices + - see Documentation/devicetree/bindings/reset/allwinner,sunxi-clock-reset.txt for reset controller devices diff --git a/Documentation/devicetree/bindings/mips/lantiq/rcu.txt b/Documentation/devicetree/bindings/mips/lantiq/rcu.txt index a086f1e1cdd7..7f0822b4beae 100644 --- a/Documentation/devicetree/bindings/mips/lantiq/rcu.txt +++ b/Documentation/devicetree/bindings/mips/lantiq/rcu.txt @@ -61,7 +61,6 @@ Example of the RCU bindings on a xRX200 SoC: usb_phy0: usb2-phy@18 { compatible = "lantiq,xrx200-usb2-phy"; reg = <0x18 4>, <0x38 4>; - status = "disabled"; resets = <&reset1 4 4>, <&reset0 4 4>; reset-names = "phy", "ctrl"; @@ -71,7 +70,6 @@ Example of the RCU bindings on a xRX200 SoC: usb_phy1: usb2-phy@34 { compatible = "lantiq,xrx200-usb2-phy"; reg = <0x34 4>, <0x3C 4>; - status = "disabled"; resets = <&reset1 5 4>, <&reset0 4 4>; reset-names = "phy", "ctrl"; diff --git a/Documentation/devicetree/bindings/mmc/amlogic,meson-gx.txt b/Documentation/devicetree/bindings/mmc/amlogic,meson-gx.txt index 50bf611a4d2c..13e70409e8ac 100644 --- a/Documentation/devicetree/bindings/mmc/amlogic,meson-gx.txt +++ b/Documentation/devicetree/bindings/mmc/amlogic,meson-gx.txt @@ -12,6 +12,7 @@ Required properties: - "amlogic,meson-gxbb-mmc" - "amlogic,meson-gxl-mmc" - "amlogic,meson-gxm-mmc" + - "amlogic,meson-axg-mmc" - clocks : A list of phandle + clock-specifier pairs for the clocks listed in clock-names. - clock-names: Should contain the following: "core" - Main peripheral bus clock @@ -19,6 +20,7 @@ Required properties: "clkin1" - Other parent clock of internal mux The driver has an internal mux clock which switches between clkin0 and clkin1 depending on the clock rate requested by the MMC core. +- resets : phandle of the internal reset line Example: @@ -29,4 +31,5 @@ Example: clocks = <&clkc CLKID_SD_EMMC_A>, <&xtal>, <&clkc CLKID_FCLK_DIV2>; clock-names = "core", "clkin0", "clkin1"; pinctrl-0 = <&emmc_pins>; + resets = <&reset RESET_SD_EMMC_A>; }; diff --git a/Documentation/devicetree/bindings/mmc/bluefield-dw-mshc.txt b/Documentation/devicetree/bindings/mmc/bluefield-dw-mshc.txt new file mode 100644 index 000000000000..b0f0999ea1a9 --- /dev/null +++ b/Documentation/devicetree/bindings/mmc/bluefield-dw-mshc.txt @@ -0,0 +1,29 @@ +* Mellanox Bluefield SoC specific extensions to the Synopsys Designware + Mobile Storage Host Controller + +Read synopsys-dw-mshc.txt for more details + +The Synopsys designware mobile storage host controller is used to interface +a SoC with storage medium such as eMMC or SD/MMC cards. This file documents +differences between the core Synopsys dw mshc controller properties described +by synopsys-dw-mshc.txt and the properties used by the Mellanox Bluefield SoC +specific extensions to the Synopsys Designware Mobile Storage Host Controller. + +Required Properties: + +* compatible: should be one of the following. + - "mellanox,bluefield-dw-mshc": for controllers with Mellanox Bluefield SoC + specific extensions. + +Example: + + /* Mellanox Bluefield SoC MMC */ + mmc@6008000 { + compatible = "mellanox,bluefield-dw-mshc"; + reg = <0x6008000 0x400>; + interrupts = <32>; + fifo-depth = <0x100>; + clock-frequency = <24000000>; + bus-width = <8>; + cap-mmc-highspeed; + }; diff --git a/Documentation/devicetree/bindings/mmc/exynos-dw-mshc.txt b/Documentation/devicetree/bindings/mmc/exynos-dw-mshc.txt index a58c173b7ab9..0419a63f73a0 100644 --- a/Documentation/devicetree/bindings/mmc/exynos-dw-mshc.txt +++ b/Documentation/devicetree/bindings/mmc/exynos-dw-mshc.txt @@ -62,7 +62,7 @@ Required properties for a slot (Deprecated - Recommend to use one slot per host) rest of the gpios (depending on the bus-width property) are the data lines in no particular order. The format of the gpio specifier depends on the gpio controller. -(Deprecated - Refer to Documentation/devicetree/binding/pinctrl/samsung-pinctrl.txt) +(Deprecated - Refer to Documentation/devicetree/bindings/pinctrl/samsung-pinctrl.txt) Example: diff --git a/Documentation/devicetree/bindings/mmc/jz4740.txt b/Documentation/devicetree/bindings/mmc/jz4740.txt new file mode 100644 index 000000000000..7cd8c432d7c8 --- /dev/null +++ b/Documentation/devicetree/bindings/mmc/jz4740.txt @@ -0,0 +1,38 @@ +* Ingenic JZ47xx MMC controllers + +This file documents the device tree properties used for the MMC controller in +Ingenic JZ4740/JZ4780 SoCs. These are in addition to the core MMC properties +described in mmc.txt. + +Required properties: +- compatible: Should be one of the following: + - "ingenic,jz4740-mmc" for the JZ4740 + - "ingenic,jz4780-mmc" for the JZ4780 +- reg: Should contain the MMC controller registers location and length. +- interrupts: Should contain the interrupt specifier of the MMC controller. +- clocks: Clock for the MMC controller. + +Optional properties: +- dmas: List of DMA specifiers with the controller specific format + as described in the generic DMA client binding. A tx and rx + specifier is required. +- dma-names: RX and TX DMA request names. + Should be "rx" and "tx", in that order. + +For additional details on DMA client bindings see ../dma/dma.txt. + +Example: + +mmc0: mmc@13450000 { + compatible = "ingenic,jz4780-mmc"; + reg = <0x13450000 0x1000>; + + interrupt-parent = <&intc>; + interrupts = <37>; + + clocks = <&cgu JZ4780_CLK_MSC0>; + clock-names = "mmc"; + + dmas = <&dma JZ4780_DMA_MSC0_RX 0xffffffff>, <&dma JZ4780_DMA_MSC0_TX 0xffffffff>; + dma-names = "rx", "tx"; +}; diff --git a/Documentation/devicetree/bindings/mmc/microchip,sdhci-pic32.txt b/Documentation/devicetree/bindings/mmc/microchip,sdhci-pic32.txt index 3149297b3933..f064528effed 100644 --- a/Documentation/devicetree/bindings/mmc/microchip,sdhci-pic32.txt +++ b/Documentation/devicetree/bindings/mmc/microchip,sdhci-pic32.txt @@ -12,7 +12,7 @@ Required properties: See: Documentation/devicetree/bindings/clock/clock-bindings.txt - pinctrl-names: A pinctrl state names "default" must be defined. - pinctrl-0: Phandle referencing pin configuration of the SDHCI controller. - See: Documentation/devicetree/bindings/pinctrl/pinctrl-binding.txt + See: Documentation/devicetree/bindings/pinctrl/pinctrl-bindings.txt Example: diff --git a/Documentation/devicetree/bindings/mmc/mmc.txt b/Documentation/devicetree/bindings/mmc/mmc.txt index 467cd7b147ce..f5a0923b34ca 100644 --- a/Documentation/devicetree/bindings/mmc/mmc.txt +++ b/Documentation/devicetree/bindings/mmc/mmc.txt @@ -19,6 +19,8 @@ Optional properties: - wp-gpios: Specify GPIOs for write protection, see gpio binding - cd-inverted: when present, polarity on the CD line is inverted. See the note below for the case, when a GPIO is used for the CD line +- cd-debounce-delay-ms: Set delay time before detecting card after card insert interrupt. + It's only valid when cd-gpios is present. - wp-inverted: when present, polarity on the WP line is inverted. See the note below for the case, when a GPIO is used for the WP line - disable-wp: When set no physical WP line is present. This property should @@ -56,6 +58,10 @@ Optional properties: - fixed-emmc-driver-type: for non-removable eMMC, enforce this driver type. The value is the driver type as specified in the eMMC specification (table 206 in spec version 5.1). +- post-power-on-delay-ms : It was invented for MMC pwrseq-simple which could + be referred to mmc-pwrseq-simple.txt. But now it's reused as a tunable delay + waiting for I/O signalling and card power supply to be stable, regardless of + whether pwrseq-simple is used. Default to 10ms if no available. *NOTE* on CD and WP polarity. To use common for all SD/MMC host controllers line polarity properties, we have to fix the meaning of the "normal" and "inverted" diff --git a/Documentation/devicetree/bindings/mmc/sdhci-omap.txt b/Documentation/devicetree/bindings/mmc/sdhci-omap.txt index 51775a372c06..393848c2138e 100644 --- a/Documentation/devicetree/bindings/mmc/sdhci-omap.txt +++ b/Documentation/devicetree/bindings/mmc/sdhci-omap.txt @@ -4,7 +4,14 @@ Refer to mmc.txt for standard MMC bindings. Required properties: - compatible: Should be "ti,dra7-sdhci" for DRA7 and DRA72 controllers + Should be "ti,k2g-sdhci" for K2G - ti,hwmods: Must be "mmc", is controller instance starting 1 + (Not required for K2G). +- pinctrl-names: Should be subset of "default", "hs", "sdr12", "sdr25", "sdr50", + "ddr50-rev11", "sdr104-rev11", "ddr50", "sdr104", + "ddr_1_8v-rev11", "ddr_1_8v" or "ddr_3_3v", "hs200_1_8v-rev11", + "hs200_1_8v", +- pinctrl- : Pinctrl states as described in bindings/pinctrl/pinctrl-bindings.txt Example: mmc1: mmc@4809c000 { diff --git a/Documentation/devicetree/bindings/mmc/sdhci-st.txt b/Documentation/devicetree/bindings/mmc/sdhci-st.txt index 6b3d40ca395e..ccf82b4ee838 100644 --- a/Documentation/devicetree/bindings/mmc/sdhci-st.txt +++ b/Documentation/devicetree/bindings/mmc/sdhci-st.txt @@ -20,7 +20,7 @@ Required properties: - pinctrl-names: A pinctrl state names "default" must be defined. - pinctrl-0: Phandle referencing pin configuration of the sd/emmc controller. - See: Documentation/devicetree/bindings/pinctrl/pinctrl-binding.txt + See: Documentation/devicetree/bindings/pinctrl/pinctrl-bindings.txt - reg: This must provide the host controller base address and it can also contain the FlashSS Top register for TX/RX delay used by the driver diff --git a/Documentation/devicetree/bindings/mmc/tmio_mmc.txt b/Documentation/devicetree/bindings/mmc/tmio_mmc.txt index 2d5287eeed95..839f469f4525 100644 --- a/Documentation/devicetree/bindings/mmc/tmio_mmc.txt +++ b/Documentation/devicetree/bindings/mmc/tmio_mmc.txt @@ -26,6 +26,8 @@ Required properties: "renesas,sdhi-r8a7794" - SDHI IP on R8A7794 SoC "renesas,sdhi-r8a7795" - SDHI IP on R8A7795 SoC "renesas,sdhi-r8a7796" - SDHI IP on R8A7796 SoC + "renesas,sdhi-r8a77965" - SDHI IP on R8A77965 SoC + "renesas,sdhi-r8a77980" - SDHI IP on R8A77980 SoC "renesas,sdhi-r8a77995" - SDHI IP on R8A77995 SoC "renesas,sdhi-shmobile" - a generic sh-mobile SDHI controller "renesas,rcar-gen1-sdhi" - a generic R-Car Gen1 SDHI controller @@ -67,7 +69,6 @@ Example: R8A7790 (R-Car H2) SDHI controller nodes max-frequency = <195000000>; power-domains = <&sysc R8A7790_PD_ALWAYS_ON>; resets = <&cpg 314>; - status = "disabled"; }; sdhi1: sd@ee120000 { @@ -81,7 +82,6 @@ Example: R8A7790 (R-Car H2) SDHI controller nodes max-frequency = <195000000>; power-domains = <&sysc R8A7790_PD_ALWAYS_ON>; resets = <&cpg 313>; - status = "disabled"; }; sdhi2: sd@ee140000 { @@ -95,7 +95,6 @@ Example: R8A7790 (R-Car H2) SDHI controller nodes max-frequency = <97500000>; power-domains = <&sysc R8A7790_PD_ALWAYS_ON>; resets = <&cpg 312>; - status = "disabled"; }; sdhi3: sd@ee160000 { @@ -109,5 +108,4 @@ Example: R8A7790 (R-Car H2) SDHI controller nodes max-frequency = <97500000>; power-domains = <&sysc R8A7790_PD_ALWAYS_ON>; resets = <&cpg 311>; - status = "disabled"; }; diff --git a/Documentation/devicetree/bindings/mtd/gpmi-nand.txt b/Documentation/devicetree/bindings/mtd/gpmi-nand.txt index b289ef3c1b7e..393588385c6e 100644 --- a/Documentation/devicetree/bindings/mtd/gpmi-nand.txt +++ b/Documentation/devicetree/bindings/mtd/gpmi-nand.txt @@ -47,6 +47,11 @@ Optional properties: partitions written from Linux with this feature turned on may not be accessible by the BootROM code. + - nand-ecc-strength: integer representing the number of bits to correct + per ECC step. Needs to be a multiple of 2. + - nand-ecc-step-size: integer representing the number of data bytes + that are covered by a single ECC step. The driver + supports 512 and 1024. The device tree may optionally contain sub-nodes describing partitions of the address space. See partition.txt for more detail. diff --git a/Documentation/devicetree/bindings/powerpc/4xx/ndfc.txt b/Documentation/devicetree/bindings/mtd/ibm,ndfc.txt similarity index 100% rename from Documentation/devicetree/bindings/powerpc/4xx/ndfc.txt rename to Documentation/devicetree/bindings/mtd/ibm,ndfc.txt diff --git a/Documentation/devicetree/bindings/mtd/mtk-nand.txt b/Documentation/devicetree/bindings/mtd/mtk-nand.txt index 1c88526dedfc..4d3ec5e4ff8a 100644 --- a/Documentation/devicetree/bindings/mtd/mtk-nand.txt +++ b/Documentation/devicetree/bindings/mtd/mtk-nand.txt @@ -20,7 +20,6 @@ Required NFI properties: - interrupts: Interrupts of NFI. - clocks: NFI required clocks. - clock-names: NFI clocks internal name. -- status: Disabled default. Then set "okay" by platform. - ecc-engine: Required ECC Engine node. - #address-cells: NAND chip index, should be 1. - #size-cells: Should be 0. @@ -34,7 +33,6 @@ Example: clocks = <&pericfg CLK_PERI_NFI>, <&pericfg CLK_PERI_NFI_PAD>; clock-names = "nfi_clk", "pad_clk"; - status = "disabled"; ecc-engine = <&bch>; #address-cells = <1>; #size-cells = <0>; @@ -50,14 +48,19 @@ Optional: - nand-on-flash-bbt: Store BBT on NAND Flash. - nand-ecc-mode: the NAND ecc mode (check driver for supported modes) - nand-ecc-step-size: Number of data bytes covered by a single ECC step. - valid values: 512 and 1024. + valid values: + 512 and 1024 on mt2701 and mt2712. + 512 only on mt7622. 1024 is recommended for large page NANDs. - nand-ecc-strength: Number of bits to correct per ECC step. - The valid values that the controller supports are: 4, 6, - 8, 10, 12, 14, 16, 18, 20, 22, 24, 28, 32, 36, 40, 44, - 48, 52, 56, 60. + The valid values that each controller supports: + mt2701: 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 28, + 32, 36, 40, 44, 48, 52, 56, 60. + mt2712: 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 28, + 32, 36, 40, 44, 48, 52, 56, 60, 68, 72, 80. + mt7622: 4, 6, 8, 10, 12, 14, 16. The strength should be calculated as follows: - E = (S - F) * 8 / 14 + E = (S - F) * 8 / B S = O / (P / Q) E : nand-ecc-strength. S : spare size per sector. @@ -66,6 +69,15 @@ Optional: O : oob size. P : page size. Q : nand-ecc-step-size. + B : number of parity bits needed to correct + 1 bitflip. + According to MTK NAND controller design, + this number depends on max ecc step size + that MTK NAND controller supports. + If max ecc step size supported is 1024, + then it should be always 14. And if max + ecc step size is 512, then it should be + always 13. If the result does not match any one of the listed choices above, please select the smaller valid value from the list. @@ -152,7 +164,6 @@ Required BCH properties: - interrupts: Interrupts of ECC. - clocks: ECC required clocks. - clock-names: ECC clocks internal name. -- status: Disabled default. Then set "okay" by platform. Example: @@ -162,5 +173,4 @@ Example: interrupts = ; clocks = <&pericfg CLK_PERI_NFI_ECC>; clock-names = "nfiecc_clk"; - status = "disabled"; }; diff --git a/Documentation/devicetree/bindings/mtd/partition.txt b/Documentation/devicetree/bindings/mtd/partition.txt index 36f3b769a626..a8f382642ba9 100644 --- a/Documentation/devicetree/bindings/mtd/partition.txt +++ b/Documentation/devicetree/bindings/mtd/partition.txt @@ -14,7 +14,7 @@ method is used for a given flash device. To describe the method there should be a subnode of the flash device that is named 'partitions'. It must have a 'compatible' property, which is used to identify the method to use. -We currently only document a binding for fixed layouts. +Available bindings are listed in the "partitions" subdirectory. Fixed Partitions diff --git a/Documentation/devicetree/bindings/mtd/partitions/brcm,bcm947xx-cfe-partitions.txt b/Documentation/devicetree/bindings/mtd/partitions/brcm,bcm947xx-cfe-partitions.txt new file mode 100644 index 000000000000..1d61a029395e --- /dev/null +++ b/Documentation/devicetree/bindings/mtd/partitions/brcm,bcm947xx-cfe-partitions.txt @@ -0,0 +1,42 @@ +Broadcom BCM47xx Partitions +=========================== + +Broadcom is one of hardware manufacturers providing SoCs (BCM47xx) used in +home routers. Their BCM947xx boards using CFE bootloader have several partitions +without any on-flash partition table. On some devices their sizes and/or +meanings can also vary so fixed partitioning can't be used. + +Discovering partitions on these devices is possible thanks to having a special +header and/or magic signature at the beginning of each of them. They are also +block aligned which is important for determinig a size. + +Most of partitions use ASCII text based magic for determining a type. More +complex partitions (like TRX with its HDR0 magic) may include extra header +containing some details, including a length. + +A list of supported partitions includes: +1) Bootloader with Broadcom's CFE (Common Firmware Environment) +2) NVRAM with configuration/calibration data +3) Device manufacturer's data with some default values (e.g. SSIDs) +4) TRX firmware container which can hold up to 4 subpartitions +5) Backup TRX firmware used after failed upgrade + +As mentioned earlier, role of some partitions may depend on extra configuration. +For example both: main firmware and backup firmware use the same TRX format with +the same header. To distinguish currently used firmware a CFE's environment +variable "bootpartition" is used. + + +Devices using Broadcom partitions described above should should have flash node +with a subnode named "partitions" using following properties: + +Required properties: +- compatible : (required) must be "brcm,bcm947xx-cfe-partitions" + +Example: + +flash@0 { + partitions { + compatible = "brcm,bcm947xx-cfe-partitions"; + }; +}; diff --git a/Documentation/devicetree/bindings/mtd/sunxi-nand.txt b/Documentation/devicetree/bindings/mtd/sunxi-nand.txt index 0734f03bf3d3..dcd5a5d80dc0 100644 --- a/Documentation/devicetree/bindings/mtd/sunxi-nand.txt +++ b/Documentation/devicetree/bindings/mtd/sunxi-nand.txt @@ -22,8 +22,6 @@ Optional properties: - reset : phandle + reset specifier pair - reset-names : must contain "ahb" - allwinner,rb : shall contain the native Ready/Busy ids. - or -- rb-gpios : shall contain the gpios used as R/B pins. - nand-ecc-mode : one of the supported ECC modes ("hw", "soft", "soft_bch" or "none") diff --git a/Documentation/devicetree/bindings/net/dsa/b53.txt b/Documentation/devicetree/bindings/net/dsa/b53.txt index 8acf51a4dfa8..47a6a7fe0b86 100644 --- a/Documentation/devicetree/bindings/net/dsa/b53.txt +++ b/Documentation/devicetree/bindings/net/dsa/b53.txt @@ -10,6 +10,7 @@ Required properties: "brcm,bcm53128" "brcm,bcm5365" "brcm,bcm5395" + "brcm,bcm5389" "brcm,bcm5397" "brcm,bcm5398" diff --git a/Documentation/devicetree/bindings/net/dsa/dsa.txt b/Documentation/devicetree/bindings/net/dsa/dsa.txt index cfe8f64eca4f..3ceeb8de1196 100644 --- a/Documentation/devicetree/bindings/net/dsa/dsa.txt +++ b/Documentation/devicetree/bindings/net/dsa/dsa.txt @@ -82,8 +82,6 @@ linked into one DSA cluster. switch0: switch0@0 { compatible = "marvell,mv88e6085"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; dsa,member = <0 0>; @@ -135,8 +133,6 @@ linked into one DSA cluster. switch1: switch1@0 { compatible = "marvell,mv88e6085"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; dsa,member = <0 1>; @@ -204,8 +200,6 @@ linked into one DSA cluster. switch2: switch2@0 { compatible = "marvell,mv88e6085"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; dsa,member = <0 2>; diff --git a/Documentation/devicetree/bindings/net/dsa/ksz.txt b/Documentation/devicetree/bindings/net/dsa/ksz.txt index fd23904ac68e..a700943218ca 100644 --- a/Documentation/devicetree/bindings/net/dsa/ksz.txt +++ b/Documentation/devicetree/bindings/net/dsa/ksz.txt @@ -6,7 +6,7 @@ Required properties: - compatible: For external switch chips, compatible string must be exactly one of: "microchip,ksz9477" -See Documentation/devicetree/bindings/dsa/dsa.txt for a list of additional +See Documentation/devicetree/bindings/net/dsa/dsa.txt for a list of additional required and optional properties. Examples: diff --git a/Documentation/devicetree/bindings/net/dsa/mt7530.txt b/Documentation/devicetree/bindings/net/dsa/mt7530.txt index a9bc27b93ee3..aa3527f71fdc 100644 --- a/Documentation/devicetree/bindings/net/dsa/mt7530.txt +++ b/Documentation/devicetree/bindings/net/dsa/mt7530.txt @@ -31,7 +31,7 @@ Required properties for the child nodes within ports container: - phy-mode: String, must be either "trgmii" or "rgmii" for port labeled "cpu". -See Documentation/devicetree/bindings/dsa/dsa.txt for a list of additional +See Documentation/devicetree/bindings/net/dsa/dsa.txt for a list of additional required, optional properties and how the integrated switch subnodes must be specified. diff --git a/Documentation/devicetree/bindings/net/dsa/qca8k.txt b/Documentation/devicetree/bindings/net/dsa/qca8k.txt index 9c67ee4890d7..bbcb255c3150 100644 --- a/Documentation/devicetree/bindings/net/dsa/qca8k.txt +++ b/Documentation/devicetree/bindings/net/dsa/qca8k.txt @@ -2,7 +2,10 @@ Required properties: -- compatible: should be "qca,qca8337" +- compatible: should be one of: + "qca,qca8334" + "qca,qca8337" + - #size-cells: must be 0 - #address-cells: must be 1 @@ -14,6 +17,20 @@ port and PHY id, each subnode describing a port needs to have a valid phandle referencing the internal PHY connected to it. The CPU port of this switch is always port 0. +A CPU port node has the following optional node: + +- fixed-link : Fixed-link subnode describing a link to a non-MDIO + managed entity. See + Documentation/devicetree/bindings/net/fixed-link.txt + for details. + +For QCA8K the 'fixed-link' sub-node supports only the following properties: + +- 'speed' (integer, mandatory), to indicate the link speed. Accepted + values are 10, 100 and 1000 +- 'full-duplex' (boolean, optional), to indicate that full duplex is + used. When absent, half duplex is assumed. + Example: @@ -53,6 +70,10 @@ Example: label = "cpu"; ethernet = <&gmac1>; phy-mode = "rgmii"; + fixed-link { + speed = 1000; + full-duplex; + }; }; port@1 { diff --git a/Documentation/devicetree/bindings/net/dwmac-sun8i.txt b/Documentation/devicetree/bindings/net/dwmac-sun8i.txt index 3d6d5fa0c4d5..cfe724398a12 100644 --- a/Documentation/devicetree/bindings/net/dwmac-sun8i.txt +++ b/Documentation/devicetree/bindings/net/dwmac-sun8i.txt @@ -7,6 +7,7 @@ Required properties: - compatible: must be one of the following string: "allwinner,sun8i-a83t-emac" "allwinner,sun8i-h3-emac" + "allwinner,sun8i-r40-gmac" "allwinner,sun8i-v3s-emac" "allwinner,sun50i-a64-emac" - reg: address and length of the register for the device. @@ -20,18 +21,18 @@ Required properties: - phy-handle: See ethernet.txt - #address-cells: shall be 1 - #size-cells: shall be 0 -- syscon: A phandle to the syscon of the SoC with one of the following - compatible string: - - allwinner,sun8i-h3-system-controller - - allwinner,sun8i-v3s-system-controller - - allwinner,sun50i-a64-system-controller - - allwinner,sun8i-a83t-system-controller +- syscon: A phandle to the device containing the EMAC or GMAC clock register Optional properties: -- allwinner,tx-delay-ps: TX clock delay chain value in ps. Range value is 0-700. Default is 0) -- allwinner,rx-delay-ps: RX clock delay chain value in ps. Range value is 0-3100. Default is 0) -Both delay properties need to be a multiple of 100. They control the delay for -external PHY. +- allwinner,tx-delay-ps: TX clock delay chain value in ps. + Range is 0-700. Default is 0. + Unavailable for allwinner,sun8i-r40-gmac +- allwinner,rx-delay-ps: RX clock delay chain value in ps. + Range is 0-3100. Default is 0. + Range is 0-700 for allwinner,sun8i-r40-gmac +Both delay properties need to be a multiple of 100. They control the +clock delay for external RGMII PHY. They do not apply to the internal +PHY or external non-RGMII PHYs. Optional properties for the following compatibles: - "allwinner,sun8i-h3-emac", diff --git a/Documentation/devicetree/bindings/net/fsl-tsec-phy.txt b/Documentation/devicetree/bindings/net/fsl-tsec-phy.txt index 79bf352e659c..047bdf7bdd2f 100644 --- a/Documentation/devicetree/bindings/net/fsl-tsec-phy.txt +++ b/Documentation/devicetree/bindings/net/fsl-tsec-phy.txt @@ -86,70 +86,4 @@ Example: * Gianfar PTP clock nodes -General Properties: - - - compatible Should be "fsl,etsec-ptp" - - reg Offset and length of the register set for the device - - interrupts There should be at least two interrupts. Some devices - have as many as four PTP related interrupts. - -Clock Properties: - - - fsl,cksel Timer reference clock source. - - fsl,tclk-period Timer reference clock period in nanoseconds. - - fsl,tmr-prsc Prescaler, divides the output clock. - - fsl,tmr-add Frequency compensation value. - - fsl,tmr-fiper1 Fixed interval period pulse generator. - - fsl,tmr-fiper2 Fixed interval period pulse generator. - - fsl,max-adj Maximum frequency adjustment in parts per billion. - - These properties set the operational parameters for the PTP - clock. You must choose these carefully for the clock to work right. - Here is how to figure good values: - - TimerOsc = selected reference clock MHz - tclk_period = desired clock period nanoseconds - NominalFreq = 1000 / tclk_period MHz - FreqDivRatio = TimerOsc / NominalFreq (must be greater that 1.0) - tmr_add = ceil(2^32 / FreqDivRatio) - OutputClock = NominalFreq / tmr_prsc MHz - PulseWidth = 1 / OutputClock microseconds - FiperFreq1 = desired frequency in Hz - FiperDiv1 = 1000000 * OutputClock / FiperFreq1 - tmr_fiper1 = tmr_prsc * tclk_period * FiperDiv1 - tclk_period - max_adj = 1000000000 * (FreqDivRatio - 1.0) - 1 - - The calculation for tmr_fiper2 is the same as for tmr_fiper1. The - driver expects that tmr_fiper1 will be correctly set to produce a 1 - Pulse Per Second (PPS) signal, since this will be offered to the PPS - subsystem to synchronize the Linux clock. - - Reference clock source is determined by the value, which is holded - in CKSEL bits in TMR_CTRL register. "fsl,cksel" property keeps the - value, which will be directly written in those bits, that is why, - according to reference manual, the next clock sources can be used: - - <0> - external high precision timer reference clock (TSEC_TMR_CLK - input is used for this purpose); - <1> - eTSEC system clock; - <2> - eTSEC1 transmit clock; - <3> - RTC clock input. - - When this attribute is not used, eTSEC system clock will serve as - IEEE 1588 timer reference clock. - -Example: - - ptp_clock@24e00 { - compatible = "fsl,etsec-ptp"; - reg = <0x24E00 0xB0>; - interrupts = <12 0x8 13 0x8>; - interrupt-parent = < &ipic >; - fsl,cksel = <1>; - fsl,tclk-period = <10>; - fsl,tmr-prsc = <100>; - fsl,tmr-add = <0x999999A4>; - fsl,tmr-fiper1 = <0x3B9AC9F6>; - fsl,tmr-fiper2 = <0x00018696>; - fsl,max-adj = <659999998>; - }; +Refer to Documentation/devicetree/bindings/ptp/ptp-qoriq.txt diff --git a/Documentation/devicetree/bindings/powerpc/4xx/emac.txt b/Documentation/devicetree/bindings/net/ibm,emac.txt similarity index 100% rename from Documentation/devicetree/bindings/powerpc/4xx/emac.txt rename to Documentation/devicetree/bindings/net/ibm,emac.txt diff --git a/Documentation/devicetree/bindings/net/meson-dwmac.txt b/Documentation/devicetree/bindings/net/meson-dwmac.txt index 61cada22ae6c..1321bb194ed9 100644 --- a/Documentation/devicetree/bindings/net/meson-dwmac.txt +++ b/Documentation/devicetree/bindings/net/meson-dwmac.txt @@ -11,6 +11,7 @@ Required properties on all platforms: - "amlogic,meson8b-dwmac" - "amlogic,meson8m2-dwmac" - "amlogic,meson-gxbb-dwmac" + - "amlogic,meson-axg-dwmac" Additionally "snps,dwmac" and any applicable more detailed version number described in net/stmmac.txt should be used. diff --git a/Documentation/devicetree/bindings/net/microchip,lan78xx.txt b/Documentation/devicetree/bindings/net/microchip,lan78xx.txt new file mode 100644 index 000000000000..76786a0f6d3d --- /dev/null +++ b/Documentation/devicetree/bindings/net/microchip,lan78xx.txt @@ -0,0 +1,54 @@ +Microchip LAN78xx Gigabit Ethernet controller + +The LAN78XX devices are usually configured by programming their OTP or with +an external EEPROM, but some platforms (e.g. Raspberry Pi 3 B+) have neither. +The Device Tree properties, if present, override the OTP and EEPROM. + +Required properties: +- compatible: Should be one of "usb424,7800", "usb424,7801" or "usb424,7850". + +Optional properties: +- local-mac-address: see ethernet.txt +- mac-address: see ethernet.txt + +Optional properties of the embedded PHY: +- microchip,led-modes: a 0..4 element vector, with each element configuring + the operating mode of an LED. Omitted LEDs are turned off. Allowed values + are defined in "include/dt-bindings/net/microchip-lan78xx.h". + +Example: + +/* Based on the configuration for a Raspberry Pi 3 B+ */ +&usb { + usb-port@1 { + compatible = "usb424,2514"; + reg = <1>; + #address-cells = <1>; + #size-cells = <0>; + + usb-port@1 { + compatible = "usb424,2514"; + reg = <1>; + #address-cells = <1>; + #size-cells = <0>; + + ethernet: ethernet@1 { + compatible = "usb424,7800"; + reg = <1>; + local-mac-address = [ 00 11 22 33 44 55 ]; + + mdio { + #address-cells = <0x1>; + #size-cells = <0x0>; + eth_phy: ethernet-phy@1 { + reg = <1>; + microchip,led-modes = < + LAN78XX_LINK_1000_ACTIVITY + LAN78XX_LINK_10_100_ACTIVITY + >; + }; + }; + }; + }; + }; +}; diff --git a/Documentation/devicetree/bindings/net/mscc-miim.txt b/Documentation/devicetree/bindings/net/mscc-miim.txt new file mode 100644 index 000000000000..7104679cf59d --- /dev/null +++ b/Documentation/devicetree/bindings/net/mscc-miim.txt @@ -0,0 +1,26 @@ +Microsemi MII Management Controller (MIIM) / MDIO +================================================= + +Properties: +- compatible: must be "mscc,ocelot-miim" +- reg: The base address of the MDIO bus controller register bank. Optionally, a + second register bank can be defined if there is an associated reset register + for internal PHYs +- #address-cells: Must be <1>. +- #size-cells: Must be <0>. MDIO addresses have no size component. +- interrupts: interrupt specifier (refer to the interrupt binding) + +Typically an MDIO bus might have several children. + +Example: + mdio@107009c { + #address-cells = <1>; + #size-cells = <0>; + compatible = "mscc,ocelot-miim"; + reg = <0x107009c 0x36>, <0x10700f0 0x8>; + interrupts = <14>; + + phy0: ethernet-phy@0 { + reg = <0>; + }; + }; diff --git a/Documentation/devicetree/bindings/net/mscc-ocelot.txt b/Documentation/devicetree/bindings/net/mscc-ocelot.txt new file mode 100644 index 000000000000..0a84711abece --- /dev/null +++ b/Documentation/devicetree/bindings/net/mscc-ocelot.txt @@ -0,0 +1,82 @@ +Microsemi Ocelot network Switch +=============================== + +The Microsemi Ocelot network switch can be found on Microsemi SoCs (VSC7513, +VSC7514) + +Required properties: +- compatible: Should be "mscc,vsc7514-switch" +- reg: Must contain an (offset, length) pair of the register set for each + entry in reg-names. +- reg-names: Must include the following entries: + - "sys" + - "rew" + - "qs" + - "hsio" + - "qsys" + - "ana" + - "portX" with X from 0 to the number of last port index available on that + switch +- interrupts: Should contain the switch interrupts for frame extraction and + frame injection +- interrupt-names: should contain the interrupt names: "xtr", "inj" +- ethernet-ports: A container for child nodes representing switch ports. + +The ethernet-ports container has the following properties + +Required properties: + +- #address-cells: Must be 1 +- #size-cells: Must be 0 + +Each port node must have the following mandatory properties: +- reg: Describes the port address in the switch + +Port nodes may also contain the following optional standardised +properties, described in binding documents: + +- phy-handle: Phandle to a PHY on an MDIO bus. See + Documentation/devicetree/bindings/net/ethernet.txt for details. + +Example: + + switch@1010000 { + compatible = "mscc,vsc7514-switch"; + reg = <0x1010000 0x10000>, + <0x1030000 0x10000>, + <0x1080000 0x100>, + <0x10d0000 0x10000>, + <0x11e0000 0x100>, + <0x11f0000 0x100>, + <0x1200000 0x100>, + <0x1210000 0x100>, + <0x1220000 0x100>, + <0x1230000 0x100>, + <0x1240000 0x100>, + <0x1250000 0x100>, + <0x1260000 0x100>, + <0x1270000 0x100>, + <0x1280000 0x100>, + <0x1800000 0x80000>, + <0x1880000 0x10000>; + reg-names = "sys", "rew", "qs", "hsio", "port0", + "port1", "port2", "port3", "port4", "port5", + "port6", "port7", "port8", "port9", "port10", + "qsys", "ana"; + interrupts = <21 22>; + interrupt-names = "xtr", "inj"; + + ethernet-ports { + #address-cells = <1>; + #size-cells = <0>; + + port0: port@0 { + reg = <0>; + phy-handle = <&phy0>; + }; + port1: port@1 { + reg = <1>; + phy-handle = <&phy1>; + }; + }; + }; diff --git a/Documentation/devicetree/bindings/net/qualcomm-bluetooth.txt b/Documentation/devicetree/bindings/net/qualcomm-bluetooth.txt new file mode 100644 index 000000000000..0ea18a53cc29 --- /dev/null +++ b/Documentation/devicetree/bindings/net/qualcomm-bluetooth.txt @@ -0,0 +1,30 @@ +Qualcomm Bluetooth Chips +--------------------- + +This documents the binding structure and common properties for serial +attached Qualcomm devices. + +Serial attached Qualcomm devices shall be a child node of the host UART +device the slave device is attached to. + +Required properties: + - compatible: should contain one of the following: + * "qcom,qca6174-bt" + +Optional properties: + - enable-gpios: gpio specifier used to enable chip + - clocks: clock provided to the controller (SUSCLK_32KHZ) + +Example: + +serial@7570000 { + label = "BT-UART"; + status = "okay"; + + bluetooth { + compatible = "qcom,qca6174-bt"; + + enable-gpios = <&pm8994_gpios 19 GPIO_ACTIVE_HIGH>; + clocks = <&divclk4>; + }; +}; diff --git a/Documentation/devicetree/bindings/net/renesas,ravb.txt b/Documentation/devicetree/bindings/net/renesas,ravb.txt index 890526dbfc26..fac897d54423 100644 --- a/Documentation/devicetree/bindings/net/renesas,ravb.txt +++ b/Documentation/devicetree/bindings/net/renesas,ravb.txt @@ -21,6 +21,7 @@ Required properties: - "renesas,etheravb-r8a77965" for the R8A77965 SoC. - "renesas,etheravb-r8a77970" for the R8A77970 SoC. - "renesas,etheravb-r8a77980" for the R8A77980 SoC. + - "renesas,etheravb-r8a77990" for the R8A77990 SoC. - "renesas,etheravb-r8a77995" for the R8A77995 SoC. - "renesas,etheravb-rcar-gen3" as a fallback for the above R-Car Gen3 devices. diff --git a/Documentation/devicetree/bindings/net/sff,sfp.txt b/Documentation/devicetree/bindings/net/sff,sfp.txt index 929591d52ed6..832139919f20 100644 --- a/Documentation/devicetree/bindings/net/sff,sfp.txt +++ b/Documentation/devicetree/bindings/net/sff,sfp.txt @@ -7,11 +7,11 @@ Required properties: "sff,sfp" for SFP modules "sff,sff" for soldered down SFF modules -Optional Properties: - - i2c-bus : phandle of an I2C bus controller for the SFP two wire serial interface +Optional Properties: + - mod-def0-gpios : GPIO phandle and a specifier of the MOD-DEF0 (AKA Mod_ABS) module presence input gpio signal, active (module absent) high. Must not be present for SFF modules diff --git a/Documentation/devicetree/bindings/net/sh_eth.txt b/Documentation/devicetree/bindings/net/sh_eth.txt index 5172799a7f1a..82a4cf2c145d 100644 --- a/Documentation/devicetree/bindings/net/sh_eth.txt +++ b/Documentation/devicetree/bindings/net/sh_eth.txt @@ -14,6 +14,7 @@ Required properties: "renesas,ether-r8a7791" if the device is a part of R8A7791 SoC. "renesas,ether-r8a7793" if the device is a part of R8A7793 SoC. "renesas,ether-r8a7794" if the device is a part of R8A7794 SoC. + "renesas,gether-r8a77980" if the device is a part of R8A77980 SoC. "renesas,ether-r7s72100" if the device is a part of R7S72100 SoC. "renesas,rcar-gen1-ether" for a generic R-Car Gen1 device. "renesas,rcar-gen2-ether" for a generic R-Car Gen2 or RZ/G1 diff --git a/Documentation/devicetree/bindings/net/socionext,uniphier-ave4.txt b/Documentation/devicetree/bindings/net/socionext,uniphier-ave4.txt index 96398cc2982f..fc8f01718690 100644 --- a/Documentation/devicetree/bindings/net/socionext,uniphier-ave4.txt +++ b/Documentation/devicetree/bindings/net/socionext,uniphier-ave4.txt @@ -13,13 +13,25 @@ Required properties: - reg: Address where registers are mapped and size of region. - interrupts: Should contain the MAC interrupt. - phy-mode: See ethernet.txt in the same directory. Allow to choose - "rgmii", "rmii", or "mii" according to the PHY. + "rgmii", "rmii", "mii", or "internal" according to the PHY. + The acceptable mode is SoC-dependent. - phy-handle: Should point to the external phy device. See ethernet.txt file in the same directory. - clocks: A phandle to the clock for the MAC. + For Pro4 SoC, that is "socionext,uniphier-pro4-ave4", + another MAC clock, GIO bus clock and PHY clock are also required. + - clock-names: Should contain + - "ether", "ether-gb", "gio", "ether-phy" for Pro4 SoC + - "ether" for others + - resets: A phandle to the reset control for the MAC. For Pro4 SoC, + GIO bus reset is also required. + - reset-names: Should contain + - "ether", "gio" for Pro4 SoC + - "ether" for others + - socionext,syscon-phy-mode: A phandle to syscon with one argument + that configures phy mode. The argument is the ID of MAC instance. Optional properties: - - resets: A phandle to the reset control for the MAC. - local-mac-address: See ethernet.txt in the same directory. Required subnode: @@ -34,8 +46,11 @@ Example: interrupts = <0 66 4>; phy-mode = "rgmii"; phy-handle = <ðphy>; + clock-names = "ether"; clocks = <&sys_clk 6>; + reset-names = "ether"; resets = <&sys_rst 6>; + socionext,syscon-phy-mode = <&soc_glue 0>; local-mac-address = [00 00 00 00 00 00]; mdio { diff --git a/Documentation/devicetree/bindings/net/stm32-dwmac.txt b/Documentation/devicetree/bindings/net/stm32-dwmac.txt index 489dbcb66c5a..1341012722aa 100644 --- a/Documentation/devicetree/bindings/net/stm32-dwmac.txt +++ b/Documentation/devicetree/bindings/net/stm32-dwmac.txt @@ -6,14 +6,28 @@ Please see stmmac.txt for the other unchanged properties. The device node has following properties. Required properties: -- compatible: Should be "st,stm32-dwmac" to select glue, and +- compatible: For MCU family should be "st,stm32-dwmac" to select glue, and "snps,dwmac-3.50a" to select IP version. + For MPU family should be "st,stm32mp1-dwmac" to select + glue, and "snps,dwmac-4.20a" to select IP version. - clocks: Must contain a phandle for each entry in clock-names. - clock-names: Should be "stmmaceth" for the host clock. Should be "mac-clk-tx" for the MAC TX clock. Should be "mac-clk-rx" for the MAC RX clock. + For MPU family need to add also "ethstp" for power mode clock and, + "syscfg-clk" for SYSCFG clock. +- interrupt-names: Should contain a list of interrupt names corresponding to + the interrupts in the interrupts property, if available. + Should be "macirq" for the main MAC IRQ + Should be "eth_wake_irq" for the IT which wake up system - st,syscon : Should be phandle/offset pair. The phandle to the syscon node which - encompases the glue register, and the offset of the control register. + encompases the glue register, and the offset of the control register. + +Optional properties: +- clock-names: For MPU family "mac-clk-ck" for PHY without quartz +- st,int-phyclk (boolean) : valid only where PHY do not have quartz and need to be clock + by RCC + Example: ethernet@40028000 { diff --git a/Documentation/devicetree/bindings/net/wireless/qcom,ath10k.txt b/Documentation/devicetree/bindings/net/wireless/qcom,ath10k.txt index 3d2a031217da..7fd4e8ce4149 100644 --- a/Documentation/devicetree/bindings/net/wireless/qcom,ath10k.txt +++ b/Documentation/devicetree/bindings/net/wireless/qcom,ath10k.txt @@ -4,6 +4,7 @@ Required properties: - compatible: Should be one of the following: * "qcom,ath10k" * "qcom,ipq4019-wifi" + * "qcom,wcn3990-wifi" PCI based devices uses compatible string "qcom,ath10k" and takes calibration data along with board specific data via "qcom,ath10k-calibration-data". @@ -18,8 +19,12 @@ In general, entry "qcom,ath10k-pre-calibration-data" and "qcom,ath10k-calibration-data" conflict with each other and only one can be provided per device. +SNOC based devices (i.e. wcn3990) uses compatible string "qcom,wcn3990-wifi". + Optional properties: - reg: Address and length of the register set for the device. +- reg-names: Must include the list of following reg names, + "membase" - resets: Must contain an entry for each entry in reset-names. See ../reset/reseti.txt for details. - reset-names: Must include the list of following reset names, @@ -49,6 +54,8 @@ Optional properties: hw versions. - qcom,ath10k-pre-calibration-data : pre calibration data as an array, the length can vary between hw versions. +- -supply: handle to the regulator device tree node + optional "supply-name" is "vdd-0.8-cx-mx". Example (to supply the calibration data alone): @@ -119,3 +126,27 @@ wifi0: wifi@a000000 { qcom,msi_base = <0x40>; qcom,ath10k-pre-calibration-data = [ 01 02 03 ... ]; }; + +Example (to supply wcn3990 SoC wifi block details): + +wifi@18000000 { + compatible = "qcom,wcn3990-wifi"; + reg = <0x18800000 0x800000>; + reg-names = "membase"; + clocks = <&clock_gcc clk_aggre2_noc_clk>; + clock-names = "smmu_aggre2_noc_clk" + interrupts = + <0 130 0 /* CE0 */ >, + <0 131 0 /* CE1 */ >, + <0 132 0 /* CE2 */ >, + <0 133 0 /* CE3 */ >, + <0 134 0 /* CE4 */ >, + <0 135 0 /* CE5 */ >, + <0 136 0 /* CE6 */ >, + <0 137 0 /* CE7 */ >, + <0 138 0 /* CE8 */ >, + <0 139 0 /* CE9 */ >, + <0 140 0 /* CE10 */ >, + <0 141 0 /* CE11 */ >; + vdd-0.8-cx-mx-supply = <&pm8998_l5>; +}; diff --git a/Documentation/devicetree/bindings/nvmem/allwinner,sunxi-sid.txt b/Documentation/devicetree/bindings/nvmem/allwinner,sunxi-sid.txt index d69543701d5d..e319fe5e205a 100644 --- a/Documentation/devicetree/bindings/nvmem/allwinner,sunxi-sid.txt +++ b/Documentation/devicetree/bindings/nvmem/allwinner,sunxi-sid.txt @@ -4,6 +4,7 @@ Required properties: - compatible: Should be one of the following: "allwinner,sun4i-a10-sid" "allwinner,sun7i-a20-sid" + "allwinner,sun8i-a83t-sid" "allwinner,sun8i-h3-sid" "allwinner,sun50i-a64-sid" diff --git a/Documentation/devicetree/bindings/nvmem/zii,rave-sp-eeprom.txt b/Documentation/devicetree/bindings/nvmem/zii,rave-sp-eeprom.txt new file mode 100644 index 000000000000..0df79d9e07ec --- /dev/null +++ b/Documentation/devicetree/bindings/nvmem/zii,rave-sp-eeprom.txt @@ -0,0 +1,40 @@ +Zodiac Inflight Innovations RAVE EEPROM Bindings + +RAVE SP EEPROM device is a "MFD cell" device exposing physical EEPROM +attached to RAVE Supervisory Processor. It is expected that its Device +Tree node is specified as a child of the node corresponding to the +parent RAVE SP device (as documented in +Documentation/devicetree/bindings/mfd/zii,rave-sp.txt) + +Required properties: + +- compatible: Should be "zii,rave-sp-eeprom" + +Optional properties: + +- zii,eeprom-name: Unique EEPROM identifier describing its function in the + system. Will be used as created NVMEM deivce's name. + +Data cells: + +Data cells are child nodes of eerpom node, bindings for which are +documented in Documentation/devicetree/bindings/nvmem/nvmem.txt + +Example: + + rave-sp { + compatible = "zii,rave-sp-rdu1"; + current-speed = <38400>; + + eeprom@a4 { + compatible = "zii,rave-sp-eeprom"; + reg = <0xa4 0x4000>; + #address-cells = <1>; + #size-cells = <1>; + zii,eeprom-name = "main-eeprom"; + + wdt_timeout: wdt-timeout@81 { + reg = <0x81 2>; + }; + }; + } diff --git a/Documentation/devicetree/bindings/opp/kryo-cpufreq.txt b/Documentation/devicetree/bindings/opp/kryo-cpufreq.txt new file mode 100644 index 000000000000..c2127b96805a --- /dev/null +++ b/Documentation/devicetree/bindings/opp/kryo-cpufreq.txt @@ -0,0 +1,680 @@ +Qualcomm Technologies, Inc. KRYO CPUFreq and OPP bindings +=================================== + +In Certain Qualcomm Technologies, Inc. SoCs like apq8096 and msm8996 +that have KRYO processors, the CPU ferequencies subset and voltage value +of each OPP varies based on the silicon variant in use. +Qualcomm Technologies, Inc. Process Voltage Scaling Tables +defines the voltage and frequency value based on the msm-id in SMEM +and speedbin blown in the efuse combination. +The qcom-cpufreq-kryo driver reads the msm-id and efuse value from the SoC +to provide the OPP framework with required information (existing HW bitmap). +This is used to determine the voltage and frequency value for each OPP of +operating-points-v2 table when it is parsed by the OPP framework. + +Required properties: +-------------------- +In 'cpus' nodes: +- operating-points-v2: Phandle to the operating-points-v2 table to use. + +In 'operating-points-v2' table: +- compatible: Should be + - 'operating-points-v2-kryo-cpu' for apq8096 and msm8996. +- nvmem-cells: A phandle pointing to a nvmem-cells node representing the + efuse registers that has information about the + speedbin that is used to select the right frequency/voltage + value pair. + Please refer the for nvmem-cells + bindings Documentation/devicetree/bindings/nvmem/nvmem.txt + and also examples below. + +In every OPP node: +- opp-supported-hw: A single 32 bit bitmap value, representing compatible HW. + Bitmap: + 0: MSM8996 V3, speedbin 0 + 1: MSM8996 V3, speedbin 1 + 2: MSM8996 V3, speedbin 2 + 3: unused + 4: MSM8996 SG, speedbin 0 + 5: MSM8996 SG, speedbin 1 + 6: MSM8996 SG, speedbin 2 + 7-31: unused + +Example 1: +--------- + + cpus { + #address-cells = <2>; + #size-cells = <0>; + + CPU0: cpu@0 { + device_type = "cpu"; + compatible = "qcom,kryo"; + reg = <0x0 0x0>; + enable-method = "psci"; + clocks = <&kryocc 0>; + cpu-supply = <&pm8994_s11_saw>; + operating-points-v2 = <&cluster0_opp>; + #cooling-cells = <2>; + next-level-cache = <&L2_0>; + L2_0: l2-cache { + compatible = "cache"; + cache-level = <2>; + }; + }; + + CPU1: cpu@1 { + device_type = "cpu"; + compatible = "qcom,kryo"; + reg = <0x0 0x1>; + enable-method = "psci"; + clocks = <&kryocc 0>; + cpu-supply = <&pm8994_s11_saw>; + operating-points-v2 = <&cluster0_opp>; + #cooling-cells = <2>; + next-level-cache = <&L2_0>; + }; + + CPU2: cpu@100 { + device_type = "cpu"; + compatible = "qcom,kryo"; + reg = <0x0 0x100>; + enable-method = "psci"; + clocks = <&kryocc 1>; + cpu-supply = <&pm8994_s11_saw>; + operating-points-v2 = <&cluster1_opp>; + #cooling-cells = <2>; + next-level-cache = <&L2_1>; + L2_1: l2-cache { + compatible = "cache"; + cache-level = <2>; + }; + }; + + CPU3: cpu@101 { + device_type = "cpu"; + compatible = "qcom,kryo"; + reg = <0x0 0x101>; + enable-method = "psci"; + clocks = <&kryocc 1>; + cpu-supply = <&pm8994_s11_saw>; + operating-points-v2 = <&cluster1_opp>; + #cooling-cells = <2>; + next-level-cache = <&L2_1>; + }; + + cpu-map { + cluster0 { + core0 { + cpu = <&CPU0>; + }; + + core1 { + cpu = <&CPU1>; + }; + }; + + cluster1 { + core0 { + cpu = <&CPU2>; + }; + + core1 { + cpu = <&CPU3>; + }; + }; + }; + }; + + cluster0_opp: opp_table0 { + compatible = "operating-points-v2-kryo-cpu"; + nvmem-cells = <&speedbin_efuse>; + opp-shared; + + opp-307200000 { + opp-hz = /bits/ 64 <307200000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x77>; + clock-latency-ns = <200000>; + }; + opp-384000000 { + opp-hz = /bits/ 64 <384000000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-422400000 { + opp-hz = /bits/ 64 <422400000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-460800000 { + opp-hz = /bits/ 64 <460800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-480000000 { + opp-hz = /bits/ 64 <480000000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-537600000 { + opp-hz = /bits/ 64 <537600000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-556800000 { + opp-hz = /bits/ 64 <556800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-614400000 { + opp-hz = /bits/ 64 <614400000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-652800000 { + opp-hz = /bits/ 64 <652800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-691200000 { + opp-hz = /bits/ 64 <691200000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-729600000 { + opp-hz = /bits/ 64 <729600000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-768000000 { + opp-hz = /bits/ 64 <768000000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-844800000 { + opp-hz = /bits/ 64 <844800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x77>; + clock-latency-ns = <200000>; + }; + opp-902400000 { + opp-hz = /bits/ 64 <902400000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-960000000 { + opp-hz = /bits/ 64 <960000000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-979200000 { + opp-hz = /bits/ 64 <979200000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1036800000 { + opp-hz = /bits/ 64 <1036800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1056000000 { + opp-hz = /bits/ 64 <1056000000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1113600000 { + opp-hz = /bits/ 64 <1113600000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1132800000 { + opp-hz = /bits/ 64 <1132800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1190400000 { + opp-hz = /bits/ 64 <1190400000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1209600000 { + opp-hz = /bits/ 64 <1209600000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1228800000 { + opp-hz = /bits/ 64 <1228800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1286400000 { + opp-hz = /bits/ 64 <1286400000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1324800000 { + opp-hz = /bits/ 64 <1324800000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x5>; + clock-latency-ns = <200000>; + }; + opp-1363200000 { + opp-hz = /bits/ 64 <1363200000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x72>; + clock-latency-ns = <200000>; + }; + opp-1401600000 { + opp-hz = /bits/ 64 <1401600000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x5>; + clock-latency-ns = <200000>; + }; + opp-1440000000 { + opp-hz = /bits/ 64 <1440000000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1478400000 { + opp-hz = /bits/ 64 <1478400000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x1>; + clock-latency-ns = <200000>; + }; + opp-1497600000 { + opp-hz = /bits/ 64 <1497600000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x4>; + clock-latency-ns = <200000>; + }; + opp-1516800000 { + opp-hz = /bits/ 64 <1516800000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1593600000 { + opp-hz = /bits/ 64 <1593600000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x71>; + clock-latency-ns = <200000>; + }; + opp-1996800000 { + opp-hz = /bits/ 64 <1996800000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x20>; + clock-latency-ns = <200000>; + }; + opp-2188800000 { + opp-hz = /bits/ 64 <2188800000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x10>; + clock-latency-ns = <200000>; + }; + }; + + cluster1_opp: opp_table1 { + compatible = "operating-points-v2-kryo-cpu"; + nvmem-cells = <&speedbin_efuse>; + opp-shared; + + opp-307200000 { + opp-hz = /bits/ 64 <307200000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x77>; + clock-latency-ns = <200000>; + }; + opp-384000000 { + opp-hz = /bits/ 64 <384000000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-403200000 { + opp-hz = /bits/ 64 <403200000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-460800000 { + opp-hz = /bits/ 64 <460800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-480000000 { + opp-hz = /bits/ 64 <480000000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-537600000 { + opp-hz = /bits/ 64 <537600000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-556800000 { + opp-hz = /bits/ 64 <556800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-614400000 { + opp-hz = /bits/ 64 <614400000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-652800000 { + opp-hz = /bits/ 64 <652800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-691200000 { + opp-hz = /bits/ 64 <691200000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-729600000 { + opp-hz = /bits/ 64 <729600000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-748800000 { + opp-hz = /bits/ 64 <748800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-806400000 { + opp-hz = /bits/ 64 <806400000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-825600000 { + opp-hz = /bits/ 64 <825600000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-883200000 { + opp-hz = /bits/ 64 <883200000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-902400000 { + opp-hz = /bits/ 64 <902400000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-940800000 { + opp-hz = /bits/ 64 <940800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-979200000 { + opp-hz = /bits/ 64 <979200000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1036800000 { + opp-hz = /bits/ 64 <1036800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1056000000 { + opp-hz = /bits/ 64 <1056000000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1113600000 { + opp-hz = /bits/ 64 <1113600000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1132800000 { + opp-hz = /bits/ 64 <1132800000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1190400000 { + opp-hz = /bits/ 64 <1190400000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1209600000 { + opp-hz = /bits/ 64 <1209600000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1248000000 { + opp-hz = /bits/ 64 <1248000000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1286400000 { + opp-hz = /bits/ 64 <1286400000>; + opp-microvolt = <905000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1324800000 { + opp-hz = /bits/ 64 <1324800000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1363200000 { + opp-hz = /bits/ 64 <1363200000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1401600000 { + opp-hz = /bits/ 64 <1401600000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1440000000 { + opp-hz = /bits/ 64 <1440000000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1478400000 { + opp-hz = /bits/ 64 <1478400000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1516800000 { + opp-hz = /bits/ 64 <1516800000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1555200000 { + opp-hz = /bits/ 64 <1555200000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1593600000 { + opp-hz = /bits/ 64 <1593600000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1632000000 { + opp-hz = /bits/ 64 <1632000000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1670400000 { + opp-hz = /bits/ 64 <1670400000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1708800000 { + opp-hz = /bits/ 64 <1708800000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1747200000 { + opp-hz = /bits/ 64 <1747200000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x70>; + clock-latency-ns = <200000>; + }; + opp-1785600000 { + opp-hz = /bits/ 64 <1785600000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x7>; + clock-latency-ns = <200000>; + }; + opp-1804800000 { + opp-hz = /bits/ 64 <1804800000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x6>; + clock-latency-ns = <200000>; + }; + opp-1824000000 { + opp-hz = /bits/ 64 <1824000000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x71>; + clock-latency-ns = <200000>; + }; + opp-1900800000 { + opp-hz = /bits/ 64 <1900800000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x74>; + clock-latency-ns = <200000>; + }; + opp-1920000000 { + opp-hz = /bits/ 64 <1920000000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x1>; + clock-latency-ns = <200000>; + }; + opp-1977600000 { + opp-hz = /bits/ 64 <1977600000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x30>; + clock-latency-ns = <200000>; + }; + opp-1996800000 { + opp-hz = /bits/ 64 <1996800000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x1>; + clock-latency-ns = <200000>; + }; + opp-2054400000 { + opp-hz = /bits/ 64 <2054400000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x30>; + clock-latency-ns = <200000>; + }; + opp-2073600000 { + opp-hz = /bits/ 64 <2073600000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x1>; + clock-latency-ns = <200000>; + }; + opp-2150400000 { + opp-hz = /bits/ 64 <2150400000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x31>; + clock-latency-ns = <200000>; + }; + opp-2246400000 { + opp-hz = /bits/ 64 <2246400000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x10>; + clock-latency-ns = <200000>; + }; + opp-2342400000 { + opp-hz = /bits/ 64 <2342400000>; + opp-microvolt = <1140000 905000 1140000>; + opp-supported-hw = <0x10>; + clock-latency-ns = <200000>; + }; + }; + +.... + +reserved-memory { + #address-cells = <2>; + #size-cells = <2>; + ranges; +.... + smem_mem: smem-mem@86000000 { + reg = <0x0 0x86000000 0x0 0x200000>; + no-map; + }; +.... +}; + +smem { + compatible = "qcom,smem"; + memory-region = <&smem_mem>; + hwlocks = <&tcsr_mutex 3>; +}; + +soc { +.... + qfprom: qfprom@74000 { + compatible = "qcom,qfprom"; + reg = <0x00074000 0x8ff>; + #address-cells = <1>; + #size-cells = <1>; + .... + speedbin_efuse: speedbin@133 { + reg = <0x133 0x1>; + bits = <5 3>; + }; + }; +}; diff --git a/Documentation/devicetree/bindings/opp/opp.txt b/Documentation/devicetree/bindings/opp/opp.txt index 4e4f30288c8b..c396c4c0af92 100644 --- a/Documentation/devicetree/bindings/opp/opp.txt +++ b/Documentation/devicetree/bindings/opp/opp.txt @@ -82,7 +82,10 @@ This defines voltage-current-frequency combinations along with other related properties. Required properties: -- opp-hz: Frequency in Hz, expressed as a 64-bit big-endian integer. +- opp-hz: Frequency in Hz, expressed as a 64-bit big-endian integer. This is a + required property for all device nodes but devices like power domains. The + power domain nodes must have another (implementation dependent) property which + uniquely identifies the OPP nodes. Optional properties: - opp-microvolt: voltage in micro Volts. @@ -159,7 +162,7 @@ Optional properties: - status: Marks the node enabled/disabled. -- required-opp: This contains phandle to an OPP node in another device's OPP +- required-opps: This contains phandle to an OPP node in another device's OPP table. It may contain an array of phandles, where each phandle points to an OPP of a different device. It should not contain multiple phandles to the OPP nodes in the same OPP table. This specifies the minimum required OPP of the diff --git a/Documentation/devicetree/bindings/pci/designware-pcie.txt b/Documentation/devicetree/bindings/pci/designware-pcie.txt index 1da7ade3183c..c124f9bc11f3 100644 --- a/Documentation/devicetree/bindings/pci/designware-pcie.txt +++ b/Documentation/devicetree/bindings/pci/designware-pcie.txt @@ -1,7 +1,9 @@ * Synopsys DesignWare PCIe interface Required properties: -- compatible: should contain "snps,dw-pcie" to identify the core. +- compatible: + "snps,dw-pcie" for RC mode; + "snps,dw-pcie-ep" for EP mode; - reg: Should contain the configuration address space. - reg-names: Must be "config" for the PCIe configuration space. (The old way of getting the configuration address space from "ranges" @@ -41,11 +43,11 @@ EP mode: Example configuration: - pcie: pcie@dffff000 { + pcie: pcie@dfc00000 { compatible = "snps,dw-pcie"; - reg = <0xdffff000 0x1000>, /* Controller registers */ - <0xd0000000 0x2000>; /* PCI config space */ - reg-names = "ctrlreg", "config"; + reg = <0xdfc00000 0x0001000>, /* IP registers */ + <0xd0000000 0x0002000>; /* Configuration space */ + reg-names = "dbi", "config"; #address-cells = <3>; #size-cells = <2>; device_type = "pci"; @@ -54,5 +56,15 @@ Example configuration: interrupts = <25>, <24>; #interrupt-cells = <1>; num-lanes = <1>; - num-viewport = <3>; + }; +or + pcie: pcie@dfc00000 { + compatible = "snps,dw-pcie-ep"; + reg = <0xdfc00000 0x0001000>, /* IP registers 1 */ + <0xdfc01000 0x0001000>, /* IP registers 2 */ + <0xd0000000 0x2000000>; /* Configuration space */ + reg-names = "dbi", "dbi2", "addr_space"; + num-ib-windows = <6>; + num-ob-windows = <2>; + num-lanes = <1>; }; diff --git a/Documentation/devicetree/bindings/pci/hisilicon-pcie.txt b/Documentation/devicetree/bindings/pci/hisilicon-pcie.txt index 7bf9df047a1e..0dcb87d6554f 100644 --- a/Documentation/devicetree/bindings/pci/hisilicon-pcie.txt +++ b/Documentation/devicetree/bindings/pci/hisilicon-pcie.txt @@ -3,7 +3,7 @@ HiSilicon Hip05 and Hip06 PCIe host bridge DT description HiSilicon PCIe host controller is based on the Synopsys DesignWare PCI core. It shares common functions with the PCIe DesignWare core driver and inherits common properties defined in -Documentation/devicetree/bindings/pci/designware-pci.txt. +Documentation/devicetree/bindings/pci/designware-pcie.txt. Additional properties are described here: diff --git a/Documentation/devicetree/bindings/pci/kirin-pcie.txt b/Documentation/devicetree/bindings/pci/kirin-pcie.txt index 6e217c63123d..6bbe43818ad5 100644 --- a/Documentation/devicetree/bindings/pci/kirin-pcie.txt +++ b/Documentation/devicetree/bindings/pci/kirin-pcie.txt @@ -3,7 +3,7 @@ HiSilicon Kirin SoCs PCIe host DT description Kirin PCIe host controller is based on the Synopsys DesignWare PCI core. It shares common functions with the PCIe DesignWare core driver and inherits common properties defined in -Documentation/devicetree/bindings/pci/designware-pci.txt. +Documentation/devicetree/bindings/pci/designware-pcie.txt. Additional properties are described here: diff --git a/Documentation/devicetree/bindings/pci/mobiveil-pcie.txt b/Documentation/devicetree/bindings/pci/mobiveil-pcie.txt new file mode 100644 index 000000000000..65038aa642e5 --- /dev/null +++ b/Documentation/devicetree/bindings/pci/mobiveil-pcie.txt @@ -0,0 +1,73 @@ +* Mobiveil AXI PCIe Root Port Bridge DT description + +Mobiveil's GPEX 4.0 is a PCIe Gen4 root port bridge IP. This configurable IP +has up to 8 outbound and inbound windows for the address translation. + +Required properties: +- #address-cells: Address representation for root ports, set to <3> +- #size-cells: Size representation for root ports, set to <2> +- #interrupt-cells: specifies the number of cells needed to encode an + interrupt source. The value must be 1. +- compatible: Should contain "mbvl,gpex40-pcie" +- reg: Should contain PCIe registers location and length + "config_axi_slave": PCIe controller registers + "csr_axi_slave" : Bridge config registers + "gpio_slave" : GPIO registers to control slot power + "apb_csr" : MSI registers + +- device_type: must be "pci" +- apio-wins : number of requested apio outbound windows + default 2 outbound windows are configured - + 1. Config window + 2. Memory window +- ppio-wins : number of requested ppio inbound windows + default 1 inbound memory window is configured. +- bus-range: PCI bus numbers covered +- interrupt-controller: identifies the node as an interrupt controller +- #interrupt-cells: specifies the number of cells needed to encode an + interrupt source. The value must be 1. +- interrupt-parent : phandle to the interrupt controller that + it is attached to, it should be set to gic to point to + ARM's Generic Interrupt Controller node in system DT. +- interrupts: The interrupt line of the PCIe controller + last cell of this field is set to 4 to + denote it as IRQ_TYPE_LEVEL_HIGH type interrupt. +- interrupt-map-mask, + interrupt-map: standard PCI properties to define the mapping of the + PCI interface to interrupt numbers. +- ranges: ranges for the PCI memory regions (I/O space region is not + supported by hardware) + Please refer to the standard PCI bus binding document for a more + detailed explanation + + +Example: +++++++++ + pcie0: pcie@a0000000 { + #address-cells = <3>; + #size-cells = <2>; + compatible = "mbvl,gpex40-pcie"; + reg = <0xa0000000 0x00001000>, + <0xb0000000 0x00010000>, + <0xff000000 0x00200000>, + <0xb0010000 0x00001000>; + reg-names = "config_axi_slave", + "csr_axi_slave", + "gpio_slave", + "apb_csr"; + device_type = "pci"; + apio-wins = <2>; + ppio-wins = <1>; + bus-range = <0x00000000 0x000000ff>; + interrupt-controller; + interrupt-parent = <&gic>; + #interrupt-cells = <1>; + interrupts = < 0 89 4 >; + interrupt-map-mask = <0 0 0 7>; + interrupt-map = <0 0 0 0 &pci_express 0>, + <0 0 0 1 &pci_express 1>, + <0 0 0 2 &pci_express 2>, + <0 0 0 3 &pci_express 3>; + ranges = < 0x83000000 0 0x00000000 0xa8000000 0 0x8000000>; + + }; diff --git a/Documentation/devicetree/bindings/pci/pci-armada8k.txt b/Documentation/devicetree/bindings/pci/pci-armada8k.txt index c1e4c3d10a74..9e3fc15e1af8 100644 --- a/Documentation/devicetree/bindings/pci/pci-armada8k.txt +++ b/Documentation/devicetree/bindings/pci/pci-armada8k.txt @@ -12,7 +12,10 @@ Required properties: - "ctrl" for the control register region - "config" for the config space region - interrupts: Interrupt specifier for the PCIe controler -- clocks: reference to the PCIe controller clock +- clocks: reference to the PCIe controller clocks +- clock-names: mandatory if there is a second clock, in this case the + name must be "core" for the first clock and "reg" for the second + one Example: diff --git a/Documentation/devicetree/bindings/pci/pci-keystone.txt b/Documentation/devicetree/bindings/pci/pci-keystone.txt index 7e05487544ed..3d4a209b0fd0 100644 --- a/Documentation/devicetree/bindings/pci/pci-keystone.txt +++ b/Documentation/devicetree/bindings/pci/pci-keystone.txt @@ -3,9 +3,9 @@ TI Keystone PCIe interface Keystone PCI host Controller is based on the Synopsys DesignWare PCI hardware version 3.65. It shares common functions with the PCIe DesignWare core driver and inherits common properties defined in -Documentation/devicetree/bindings/pci/designware-pci.txt +Documentation/devicetree/bindings/pci/designware-pcie.txt -Please refer to Documentation/devicetree/bindings/pci/designware-pci.txt +Please refer to Documentation/devicetree/bindings/pci/designware-pcie.txt for the details of DesignWare DT bindings. Additional properties are described here as well as properties that are not applicable. diff --git a/Documentation/devicetree/bindings/pci/rcar-pci.txt b/Documentation/devicetree/bindings/pci/rcar-pci.txt index 1fb614e615da..a5f7fc62d10e 100644 --- a/Documentation/devicetree/bindings/pci/rcar-pci.txt +++ b/Documentation/devicetree/bindings/pci/rcar-pci.txt @@ -8,6 +8,7 @@ compatible: "renesas,pcie-r8a7743" for the R8A7743 SoC; "renesas,pcie-r8a7793" for the R8A7793 SoC; "renesas,pcie-r8a7795" for the R8A7795 SoC; "renesas,pcie-r8a7796" for the R8A7796 SoC; + "renesas,pcie-r8a77980" for the R8A77980 SoC; "renesas,pcie-rcar-gen2" for a generic R-Car Gen2 or RZ/G1 compatible device. "renesas,pcie-rcar-gen3" for a generic R-Car Gen3 compatible device. @@ -32,6 +33,11 @@ compatible: "renesas,pcie-r8a7743" for the R8A7743 SoC; and PCIe bus clocks. - clock-names: from common clock binding: should be "pcie" and "pcie_bus". +Optional properties: +- phys: from common PHY binding: PHY phandle and specifier (only make sense + for R-Car gen3 SoCs where the PCIe PHYs have their own register blocks). +- phy-names: from common PHY binding: should be "pcie". + Example: SoC-specific DT Entry: diff --git a/Documentation/devicetree/bindings/pci/rockchip-pcie-ep.txt b/Documentation/devicetree/bindings/pci/rockchip-pcie-ep.txt new file mode 100644 index 000000000000..778467307a93 --- /dev/null +++ b/Documentation/devicetree/bindings/pci/rockchip-pcie-ep.txt @@ -0,0 +1,62 @@ +* Rockchip AXI PCIe Endpoint Controller DT description + +Required properties: +- compatible: Should contain "rockchip,rk3399-pcie-ep" +- reg: Two register ranges as listed in the reg-names property +- reg-names: Must include the following names + - "apb-base" + - "mem-base" +- clocks: Must contain an entry for each entry in clock-names. + See ../clocks/clock-bindings.txt for details. +- clock-names: Must include the following entries: + - "aclk" + - "aclk-perf" + - "hclk" + - "pm" +- resets: Must contain seven entries for each entry in reset-names. + See ../reset/reset.txt for details. +- reset-names: Must include the following names + - "core" + - "mgmt" + - "mgmt-sticky" + - "pipe" + - "pm" + - "aclk" + - "pclk" +- pinctrl-names : The pin control state names +- pinctrl-0: The "default" pinctrl state +- phys: Must contain an phandle to a PHY for each entry in phy-names. +- phy-names: Must include 4 entries for all 4 lanes even if some of + them won't be used for your cases. Entries are of the form "pcie-phy-N": + where N ranges from 0 to 3. + (see example below and you MUST also refer to ../phy/rockchip-pcie-phy.txt + for changing the #phy-cells of phy node to support it) +- rockchip,max-outbound-regions: Maximum number of outbound regions + +Optional Property: +- num-lanes: number of lanes to use +- max-functions: Maximum number of functions that can be configured (default 1). + +pcie0-ep: pcie@f8000000 { + compatible = "rockchip,rk3399-pcie-ep"; + #address-cells = <3>; + #size-cells = <2>; + rockchip,max-outbound-regions = <16>; + clocks = <&cru ACLK_PCIE>, <&cru ACLK_PERF_PCIE>, + <&cru PCLK_PCIE>, <&cru SCLK_PCIE_PM>; + clock-names = "aclk", "aclk-perf", + "hclk", "pm"; + max-functions = /bits/ 8 <8>; + num-lanes = <4>; + reg = <0x0 0xfd000000 0x0 0x1000000>, <0x0 0x80000000 0x0 0x20000>; + reg-names = "apb-base", "mem-base"; + resets = <&cru SRST_PCIE_CORE>, <&cru SRST_PCIE_MGMT>, + <&cru SRST_PCIE_MGMT_STICKY>, <&cru SRST_PCIE_PIPE> , + <&cru SRST_PCIE_PM>, <&cru SRST_P_PCIE>, <&cru SRST_A_PCIE>; + reset-names = "core", "mgmt", "mgmt-sticky", "pipe", + "pm", "pclk", "aclk"; + phys = <&pcie_phy 0>, <&pcie_phy 1>, <&pcie_phy 2>, <&pcie_phy 3>; + phy-names = "pcie-phy-0", "pcie-phy-1", "pcie-phy-2", "pcie-phy-3"; + pinctrl-names = "default"; + pinctrl-0 = <&pcie_clkreq>; +}; diff --git a/Documentation/devicetree/bindings/pci/rockchip-pcie.txt b/Documentation/devicetree/bindings/pci/rockchip-pcie-host.txt similarity index 100% rename from Documentation/devicetree/bindings/pci/rockchip-pcie.txt rename to Documentation/devicetree/bindings/pci/rockchip-pcie-host.txt diff --git a/Documentation/devicetree/bindings/pci/xgene-pci.txt b/Documentation/devicetree/bindings/pci/xgene-pci.txt index 6fd2decfa66c..92490330dc1c 100644 --- a/Documentation/devicetree/bindings/pci/xgene-pci.txt +++ b/Documentation/devicetree/bindings/pci/xgene-pci.txt @@ -25,8 +25,6 @@ Optional properties: Example: -SoC-specific DT Entry: - pcie0: pcie@1f2b0000 { status = "disabled"; device_type = "pci"; @@ -50,8 +48,3 @@ SoC-specific DT Entry: clocks = <&pcie0clk 0>; }; - -Board-specific DT Entry: - &pcie0 { - status = "ok"; - }; diff --git a/Documentation/devicetree/bindings/phy/phy-mtk-xsphy.txt b/Documentation/devicetree/bindings/phy/phy-mtk-xsphy.txt new file mode 100644 index 000000000000..e7caefa0b9c2 --- /dev/null +++ b/Documentation/devicetree/bindings/phy/phy-mtk-xsphy.txt @@ -0,0 +1,109 @@ +MediaTek XS-PHY binding +-------------------------- + +The XS-PHY controller supports physical layer functionality for USB3.1 +GEN2 controller on MediaTek SoCs. + +Required properties (controller (parent) node): + - compatible : should be "mediatek,-xsphy", "mediatek,xsphy", + soc-model is the name of SoC, such as mt3611 etc; + when using "mediatek,xsphy" compatible string, you need SoC specific + ones in addition, one of: + - "mediatek,mt3611-xsphy" + + - #address-cells, #size-cells : should use the same values as the root node + - ranges: must be present + +Optional properties (controller (parent) node): + - reg : offset and length of register shared by multiple U3 ports, + exclude port's private register, if only U2 ports provided, + shouldn't use the property. + - mediatek,src-ref-clk-mhz : u32, frequency of reference clock for slew rate + calibrate + - mediatek,src-coef : u32, coefficient for slew rate calibrate, depends on + SoC process + +Required nodes : a sub-node is required for each port the controller + provides. Address range information including the usual + 'reg' property is used inside these nodes to describe + the controller's topology. + +Required properties (port (child) node): +- reg : address and length of the register set for the port. +- clocks : a list of phandle + clock-specifier pairs, one for each + entry in clock-names +- clock-names : must contain + "ref": 48M reference clock for HighSpeed analog phy; and 26M + reference clock for SuperSpeedPlus analog phy, sometimes is + 24M, 25M or 27M, depended on platform. +- #phy-cells : should be 1 + cell after port phandle is phy type from: + - PHY_TYPE_USB2 + - PHY_TYPE_USB3 + +The following optional properties are only for debug or HQA test +Optional properties (PHY_TYPE_USB2 port (child) node): +- mediatek,eye-src : u32, the value of slew rate calibrate +- mediatek,eye-vrt : u32, the selection of VRT reference voltage +- mediatek,eye-term : u32, the selection of HS_TX TERM reference voltage +- mediatek,efuse-intr : u32, the selection of Internal Resistor + +Optional properties (PHY_TYPE_USB3 port (child) node): +- mediatek,efuse-intr : u32, the selection of Internal Resistor +- mediatek,efuse-tx-imp : u32, the selection of TX Impedance +- mediatek,efuse-rx-imp : u32, the selection of RX Impedance + +Banks layout of xsphy +------------------------------------------------------------- +port offset bank +u2 port0 0x0000 MISC + 0x0100 FMREG + 0x0300 U2PHY_COM +u2 port1 0x1000 MISC + 0x1100 FMREG + 0x1300 U2PHY_COM +u2 port2 0x2000 MISC + ... +u31 common 0x3000 DIG_GLB + 0x3100 PHYA_GLB +u31 port0 0x3400 DIG_LN_TOP + 0x3500 DIG_LN_TX0 + 0x3600 DIG_LN_RX0 + 0x3700 DIG_LN_DAIF + 0x3800 PHYA_LN +u31 port1 0x3a00 DIG_LN_TOP + 0x3b00 DIG_LN_TX0 + 0x3c00 DIG_LN_RX0 + 0x3d00 DIG_LN_DAIF + 0x3e00 PHYA_LN + ... + +DIG_GLB & PHYA_GLB are shared by U31 ports. + +Example: + +u3phy: usb-phy@11c40000 { + compatible = "mediatek,mt3611-xsphy", "mediatek,xsphy"; + reg = <0 0x11c43000 0 0x0200>; + mediatek,src-ref-clk-mhz = <26>; + mediatek,src-coef = <17>; + #address-cells = <2>; + #size-cells = <2>; + ranges; + + u2port0: usb-phy@11c40000 { + reg = <0 0x11c40000 0 0x0400>; + clocks = <&clk48m>; + clock-names = "ref"; + mediatek,eye-src = <4>; + #phy-cells = <1>; + }; + + u3port0: usb-phy@11c43000 { + reg = <0 0x11c43400 0 0x0500>; + clocks = <&clk26m>; + clock-names = "ref"; + mediatek,efuse-intr = <28>; + #phy-cells = <1>; + }; +}; diff --git a/Documentation/devicetree/bindings/phy/qcom-qmp-phy.txt b/Documentation/devicetree/bindings/phy/qcom-qmp-phy.txt index dcf1b8f691d5..266a1bb8bb6e 100644 --- a/Documentation/devicetree/bindings/phy/qcom-qmp-phy.txt +++ b/Documentation/devicetree/bindings/phy/qcom-qmp-phy.txt @@ -9,7 +9,8 @@ Required properties: "qcom,ipq8074-qmp-pcie-phy" for PCIe phy on IPQ8074 "qcom,msm8996-qmp-pcie-phy" for 14nm PCIe phy on msm8996, "qcom,msm8996-qmp-usb3-phy" for 14nm USB3 phy on msm8996, - "qcom,qmp-v3-usb3-phy" for USB3 QMP V3 phy. + "qcom,sdm845-qmp-usb3-phy" for USB3 QMP V3 phy on sdm845, + "qcom,sdm845-qmp-usb3-uni-phy" for USB3 QMP V3 UNI phy on sdm845. - reg: offset and length of register set for PHY's common serdes block. diff --git a/Documentation/devicetree/bindings/phy/qcom-qusb2-phy.txt b/Documentation/devicetree/bindings/phy/qcom-qusb2-phy.txt index 42c97426836e..03025d97998b 100644 --- a/Documentation/devicetree/bindings/phy/qcom-qusb2-phy.txt +++ b/Documentation/devicetree/bindings/phy/qcom-qusb2-phy.txt @@ -6,7 +6,7 @@ QUSB2 controller supports LS/FS/HS usb connectivity on Qualcomm chipsets. Required properties: - compatible: compatible list, contains "qcom,msm8996-qusb2-phy" for 14nm PHY on msm8996, - "qcom,qusb2-v2-phy" for QUSB2 V2 PHY. + "qcom,sdm845-qusb2-phy" for 10nm PHY on sdm845. - reg: offset and length of the PHY register set. - #phy-cells: must be 0. @@ -27,6 +27,27 @@ Optional properties: tuning parameter value for qusb2 phy. - qcom,tcsr-syscon: Phandle to TCSR syscon register region. + - qcom,imp-res-offset-value: It is a 6 bit value that specifies offset to be + added to PHY refgen RESCODE via IMP_CTRL1 register. It is a PHY + tuning parameter that may vary for different boards of same SOC. + This property is applicable to only QUSB2 v2 PHY (sdm845). + - qcom,hstx-trim-value: It is a 4 bit value that specifies tuning for HSTX + output current. + Possible range is - 15mA to 24mA (stepsize of 600 uA). + See dt-bindings/phy/phy-qcom-qusb2.h for applicable values. + This property is applicable to only QUSB2 v2 PHY (sdm845). + Default value is 22.2mA for sdm845. + - qcom,preemphasis-level: It is a 2 bit value that specifies pre-emphasis level. + Possible range is 0 to 15% (stepsize of 5%). + See dt-bindings/phy/phy-qcom-qusb2.h for applicable values. + This property is applicable to only QUSB2 v2 PHY (sdm845). + Default value is 10% for sdm845. +- qcom,preemphasis-width: It is a 1 bit value that specifies how long the HSTX + pre-emphasis (specified using qcom,preemphasis-level) must be in + effect. Duration could be half-bit of full-bit. + See dt-bindings/phy/phy-qcom-qusb2.h for applicable values. + This property is applicable to only QUSB2 v2 PHY (sdm845). + Default value is full-bit width for sdm845. Example: hsusb_phy: phy@7411000 { diff --git a/Documentation/devicetree/bindings/pinctrl/actions,s900-pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/actions,s900-pinctrl.txt index fb87c7d74f2e..8fb5a53775e8 100644 --- a/Documentation/devicetree/bindings/pinctrl/actions,s900-pinctrl.txt +++ b/Documentation/devicetree/bindings/pinctrl/actions,s900-pinctrl.txt @@ -8,6 +8,17 @@ Required Properties: - reg: Should contain the register base address and size of the pin controller. - clocks: phandle of the clock feeding the pin controller +- gpio-controller: Marks the device node as a GPIO controller. +- gpio-ranges: Specifies the mapping between gpio controller and + pin-controller pins. +- #gpio-cells: Should be two. The first cell is the gpio pin number + and the second cell is used for optional parameters. +- interrupt-controller: Marks the device node as an interrupt controller. +- #interrupt-cells: Specifies the number of cells needed to encode an + interrupt. Shall be set to 2. The first cell + defines the interrupt number, the second encodes + the trigger flags described in + bindings/interrupt-controller/interrupts.txt Please refer to pinctrl-bindings.txt in this directory for details of the common pinctrl bindings used by client devices, including the meaning of the @@ -164,6 +175,11 @@ Example: compatible = "actions,s900-pinctrl"; reg = <0x0 0xe01b0000 0x0 0x1000>; clocks = <&cmu CLK_GPIO>; + gpio-controller; + gpio-ranges = <&pinctrl 0 0 146>; + #gpio-cells = <2>; + interrupt-controller; + #interrupt-cells = <2>; uart2-default: uart2-default { pinmux { diff --git a/Documentation/devicetree/bindings/pinctrl/allwinner,sunxi-pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/allwinner,sunxi-pinctrl.txt index 64bc5c2a76da..258a4648ab81 100644 --- a/Documentation/devicetree/bindings/pinctrl/allwinner,sunxi-pinctrl.txt +++ b/Documentation/devicetree/bindings/pinctrl/allwinner,sunxi-pinctrl.txt @@ -28,6 +28,7 @@ Required properties: "allwinner,sun50i-a64-r-pinctrl" "allwinner,sun50i-h5-pinctrl" "allwinner,sun50i-h6-pinctrl" + "allwinner,sun50i-h6-r-pinctrl" "nextthing,gr8-pinctrl" - reg: Should contain the register physical address and length for the diff --git a/Documentation/devicetree/bindings/pinctrl/brcm,bcm2835-gpio.txt b/Documentation/devicetree/bindings/pinctrl/brcm,bcm2835-gpio.txt index 2569866c692f..3fac0a061bcc 100644 --- a/Documentation/devicetree/bindings/pinctrl/brcm,bcm2835-gpio.txt +++ b/Documentation/devicetree/bindings/pinctrl/brcm,bcm2835-gpio.txt @@ -36,6 +36,24 @@ listed. In other words, a subnode that lists only a mux function implies no information about any pull configuration. Similarly, a subnode that lists only a pul parameter implies no information about the mux function. +The BCM2835 pin configuration and multiplexing supports the generic bindings. +For details on each properties, you can refer to ./pinctrl-bindings.txt. + +Required sub-node properties: + - pins + - function + +Optional sub-node properties: + - bias-disable + - bias-pull-up + - bias-pull-down + - output-high + - output-low + +Legacy pin configuration and multiplexing binding: +*** (Its use is deprecated, use generic multiplexing and configuration +bindings instead) + Required subnode-properties: - brcm,pins: An array of cells. Each cell contains the ID of a pin. Valid IDs are the integer GPIO IDs; 0==GPIO0, 1==GPIO1, ... 53==GPIO53. diff --git a/Documentation/devicetree/bindings/pinctrl/meson,pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/meson,pinctrl.txt index 2c12f9789116..54ecb8ab7788 100644 --- a/Documentation/devicetree/bindings/pinctrl/meson,pinctrl.txt +++ b/Documentation/devicetree/bindings/pinctrl/meson,pinctrl.txt @@ -3,8 +3,10 @@ Required properties for the root node: - compatible: one of "amlogic,meson8-cbus-pinctrl" "amlogic,meson8b-cbus-pinctrl" + "amlogic,meson8m2-cbus-pinctrl" "amlogic,meson8-aobus-pinctrl" "amlogic,meson8b-aobus-pinctrl" + "amlogic,meson8m2-aobus-pinctrl" "amlogic,meson-gxbb-periphs-pinctrl" "amlogic,meson-gxbb-aobus-pinctrl" "amlogic,meson-gxl-periphs-pinctrl" diff --git a/Documentation/devicetree/bindings/pinctrl/pinctrl-max77620.txt b/Documentation/devicetree/bindings/pinctrl/pinctrl-max77620.txt index ad4fce3552bb..511fc234558b 100644 --- a/Documentation/devicetree/bindings/pinctrl/pinctrl-max77620.txt +++ b/Documentation/devicetree/bindings/pinctrl/pinctrl-max77620.txt @@ -11,9 +11,9 @@ Optional Pinmux properties: -------------------------- Following properties are required if default setting of pins are required at boot. -- pinctrl-names: A pinctrl state named per . +- pinctrl-names: A pinctrl state named per . - pinctrl[0...n]: Properties to contain the phandle for pinctrl states per - . + . The pin configurations are defined as child of the pinctrl states node. Each sub-node have following properties: diff --git a/Documentation/devicetree/bindings/pinctrl/pinctrl-mcp23s08.txt b/Documentation/devicetree/bindings/pinctrl/pinctrl-mcp23s08.txt index a5a8322a31bd..625a22e2f211 100644 --- a/Documentation/devicetree/bindings/pinctrl/pinctrl-mcp23s08.txt +++ b/Documentation/devicetree/bindings/pinctrl/pinctrl-mcp23s08.txt @@ -18,7 +18,9 @@ Required properties: removed. - #gpio-cells : Should be two. - first cell is the pin number - - second cell is used to specify flags. Flags are currently unused. + - second cell is used to specify flags as described in + 'Documentation/devicetree/bindings/gpio/gpio.txt'. Allowed values defined by + 'include/dt-bindings/gpio/gpio.h' (e.g. GPIO_ACTIVE_LOW). - gpio-controller : Marks the device node as a GPIO controller. - reg : For an address on its bus. I2C uses this a the I2C address of the chip. SPI uses this to specify the chipselect line which the chip is @@ -99,9 +101,9 @@ Optional Pinmux properties: -------------------------- Following properties are required if default setting of pins are required at boot. -- pinctrl-names: A pinctrl state named per . +- pinctrl-names: A pinctrl state named per . - pinctrl[0...n]: Properties to contain the phandle for pinctrl states per - . + . The pin configurations are defined as child of the pinctrl states node. Each sub-node have following properties: diff --git a/Documentation/devicetree/bindings/pinctrl/pinctrl-mt7622.txt b/Documentation/devicetree/bindings/pinctrl/pinctrl-mt7622.txt index f18ed99f6e14..def8fcad8941 100644 --- a/Documentation/devicetree/bindings/pinctrl/pinctrl-mt7622.txt +++ b/Documentation/devicetree/bindings/pinctrl/pinctrl-mt7622.txt @@ -9,6 +9,16 @@ Required properties for the root node: - #gpio-cells: Should be two. The first cell is the pin number and the second is the GPIO flags. +Optional properties: +- interrupt-controller : Marks the device node as an interrupt controller + +If the property interrupt-controller is defined, following property is required +- reg-names: A string describing the "reg" entries. Must contain "eint". +- interrupts : The interrupt output from the controller. +- #interrupt-cells: Should be two. +- interrupt-parent: Phandle of the interrupt parent to which the external + GPIO interrupts are forwarded to. + Please refer to pinctrl-bindings.txt in this directory for details of the common pinctrl bindings used by client devices, including the meaning of the phrase "pin configuration node". diff --git a/Documentation/devicetree/bindings/pinctrl/pinctrl-rk805.txt b/Documentation/devicetree/bindings/pinctrl/pinctrl-rk805.txt index eee3dc260934..cbcbd31e3ce8 100644 --- a/Documentation/devicetree/bindings/pinctrl/pinctrl-rk805.txt +++ b/Documentation/devicetree/bindings/pinctrl/pinctrl-rk805.txt @@ -10,9 +10,9 @@ Optional Pinmux properties: -------------------------- Following properties are required if default setting of pins are required at boot. -- pinctrl-names: A pinctrl state named per . +- pinctrl-names: A pinctrl state named per . - pinctrl[0...n]: Properties to contain the phandle for pinctrl states per - . + . The pin configurations are defined as child of the pinctrl states node. Each sub-node have following properties: diff --git a/Documentation/devicetree/bindings/pinctrl/renesas,pfc-pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/renesas,pfc-pinctrl.txt index 892d8fd7b700..abd8fbcf1e62 100644 --- a/Documentation/devicetree/bindings/pinctrl/renesas,pfc-pinctrl.txt +++ b/Documentation/devicetree/bindings/pinctrl/renesas,pfc-pinctrl.txt @@ -15,6 +15,7 @@ Required Properties: - "renesas,pfc-r8a7740": for R8A7740 (R-Mobile A1) compatible pin-controller. - "renesas,pfc-r8a7743": for R8A7743 (RZ/G1M) compatible pin-controller. - "renesas,pfc-r8a7745": for R8A7745 (RZ/G1E) compatible pin-controller. + - "renesas,pfc-r8a77470": for R8A77470 (RZ/G1C) compatible pin-controller. - "renesas,pfc-r8a7778": for R8A7778 (R-Car M1) compatible pin-controller. - "renesas,pfc-r8a7779": for R8A7779 (R-Car H1) compatible pin-controller. - "renesas,pfc-r8a7790": for R8A7790 (R-Car H2) compatible pin-controller. @@ -27,6 +28,7 @@ Required Properties: - "renesas,pfc-r8a77965": for R8A77965 (R-Car M3-N) compatible pin-controller. - "renesas,pfc-r8a77970": for R8A77970 (R-Car V3M) compatible pin-controller. - "renesas,pfc-r8a77980": for R8A77980 (R-Car V3H) compatible pin-controller. + - "renesas,pfc-r8a77990": for R8A77990 (R-Car E3) compatible pin-controller. - "renesas,pfc-r8a77995": for R8A77995 (R-Car D3) compatible pin-controller. - "renesas,pfc-sh73a0": for SH73A0 (SH-Mobile AG5) compatible pin-controller. diff --git a/Documentation/devicetree/bindings/pinctrl/rockchip,pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/rockchip,pinctrl.txt index a01a3b8a2363..0919db294c17 100644 --- a/Documentation/devicetree/bindings/pinctrl/rockchip,pinctrl.txt +++ b/Documentation/devicetree/bindings/pinctrl/rockchip,pinctrl.txt @@ -20,6 +20,7 @@ defined as gpio sub-nodes of the pinmux controller. Required properties for iomux controller: - compatible: should be + "rockchip,px30-pinctrl": for Rockchip PX30 "rockchip,rv1108-pinctrl": for Rockchip RV1108 "rockchip,rk2928-pinctrl": for Rockchip RK2928 "rockchip,rk3066a-pinctrl": for Rockchip RK3066a diff --git a/Documentation/devicetree/bindings/power/fsl,imx-gpc.txt b/Documentation/devicetree/bindings/power/fsl,imx-gpc.txt index b31d6bbeee16..726ec2875223 100644 --- a/Documentation/devicetree/bindings/power/fsl,imx-gpc.txt +++ b/Documentation/devicetree/bindings/power/fsl,imx-gpc.txt @@ -14,7 +14,7 @@ Required properties: datasheet - interrupts: Should contain one interrupt specifier for the GPC interrupt - clocks: Must contain an entry for each entry in clock-names. - See Documentation/devicetree/bindings/clocks/clock-bindings.txt for details. + See Documentation/devicetree/bindings/clock/clock-bindings.txt for details. - clock-names: Must include the following entries: - ipg diff --git a/Documentation/devicetree/bindings/power/pd-samsung.txt b/Documentation/devicetree/bindings/power/pd-samsung.txt index 549f7dee9b9d..92ef355e8f64 100644 --- a/Documentation/devicetree/bindings/power/pd-samsung.txt +++ b/Documentation/devicetree/bindings/power/pd-samsung.txt @@ -15,23 +15,13 @@ Required Properties: Optional Properties: - label: Human readable string with domain name. Will be visible in userspace to let user to distinguish between multiple domains in SoC. -- clocks: List of clock handles. The parent clocks of the input clocks to the - devices in this power domain are set to oscclk before power gating - and restored back after powering on a domain. This is required for - all domains which are powered on and off and not required for unused - domains. -- clock-names: The following clocks can be specified: - - oscclk: Oscillator clock. - - clkN: Input clocks to the devices in this power domain. These clocks - will be reparented to oscclk before switching power domain off. - Their original parent will be brought back after turning on - the domain. Maximum of 4 clocks (N = 0 to 3) are supported. - - asbN: Clocks required by asynchronous bridges (ASB) present in - the power domain. These clock should be enabled during power - domain on/off operations. - power-domains: phandle pointing to the parent power domain, for more details see Documentation/devicetree/bindings/power/power_domain.txt +Deprecated Properties: +- clocks +- clock-names + Node of a device using power domains must have a power-domains property defined with a phandle to respective power domain. @@ -47,8 +37,6 @@ Example: mfc_pd: power-domain@10044060 { compatible = "samsung,exynos4210-pd"; reg = <0x10044060 0x20>; - clocks = <&clock CLK_FIN_PLL>, <&clock CLK_MOUT_USER_ACLK333>; - clock-names = "oscclk", "clk0"; #power-domain-cells = <0>; label = "MFC"; }; diff --git a/Documentation/devicetree/bindings/power/power_domain.txt b/Documentation/devicetree/bindings/power/power_domain.txt index f3355313c020..9b387f861aed 100644 --- a/Documentation/devicetree/bindings/power/power_domain.txt +++ b/Documentation/devicetree/bindings/power/power_domain.txt @@ -111,8 +111,8 @@ Example 3: ==PM domain consumers== Required properties: - - power-domains : A phandle and PM domain specifier as defined by bindings of - the power controller specified by phandle. + - power-domains : A list of PM domain specifiers, as defined by bindings of + the power controller that is the PM domain provider. Example: @@ -122,12 +122,21 @@ Example: power-domains = <&power 0>; }; -The node above defines a typical PM domain consumer device, which is located -inside a PM domain with index 0 of a power controller represented by a node -with the label "power". + leaky-device@12351000 { + compatible = "foo,i-leak-current"; + reg = <0x12351000 0x1000>; + power-domains = <&power 0>, <&power 1> ; + }; + +The first example above defines a typical PM domain consumer device, which is +located inside a PM domain with index 0 of a power controller represented by a +node with the label "power". +In the second example the consumer device are partitioned across two PM domains, +the first with index 0 and the second with index 1, of a power controller that +is represented by a node with the label "power. Optional properties: -- required-opp: This contains phandle to an OPP node in another device's OPP +- required-opps: This contains phandle to an OPP node in another device's OPP table. It may contain an array of phandles, where each phandle points to an OPP of a different device. It should not contain multiple phandles to the OPP nodes in the same OPP table. This specifies the minimum required OPP of the @@ -175,14 +184,14 @@ Example: compatible = "foo,i-leak-current"; reg = <0x12350000 0x1000>; power-domains = <&power 0>; - required-opp = <&domain0_opp_0>; + required-opps = <&domain0_opp_0>; }; leaky-device1@12350000 { compatible = "foo,i-leak-current"; reg = <0x12350000 0x1000>; power-domains = <&power 1>; - required-opp = <&domain1_opp_1>; + required-opps = <&domain1_opp_1>; }; [1]. Documentation/devicetree/bindings/power/domain-idle-state.txt diff --git a/Documentation/devicetree/bindings/power/renesas,rcar-sysc.txt b/Documentation/devicetree/bindings/power/renesas,rcar-sysc.txt index ab399e559257..180ae65be753 100644 --- a/Documentation/devicetree/bindings/power/renesas,rcar-sysc.txt +++ b/Documentation/devicetree/bindings/power/renesas,rcar-sysc.txt @@ -9,6 +9,7 @@ Required properties: - compatible: Must contain exactly one of the following: - "renesas,r8a7743-sysc" (RZ/G1M) - "renesas,r8a7745-sysc" (RZ/G1E) + - "renesas,r8a77470-sysc" (RZ/G1C) - "renesas,r8a7779-sysc" (R-Car H1) - "renesas,r8a7790-sysc" (R-Car H2) - "renesas,r8a7791-sysc" (R-Car M2-W) @@ -20,6 +21,7 @@ Required properties: - "renesas,r8a77965-sysc" (R-Car M3-N) - "renesas,r8a77970-sysc" (R-Car V3M) - "renesas,r8a77980-sysc" (R-Car V3H) + - "renesas,r8a77990-sysc" (R-Car E3) - "renesas,r8a77995-sysc" (R-Car D3) - reg: Address start and address range for the device. - #power-domain-cells: Must be 1. diff --git a/Documentation/devicetree/bindings/power/rockchip-io-domain.txt b/Documentation/devicetree/bindings/power/rockchip-io-domain.txt index 4a4766e9c254..e66fd4eab71c 100644 --- a/Documentation/devicetree/bindings/power/rockchip-io-domain.txt +++ b/Documentation/devicetree/bindings/power/rockchip-io-domain.txt @@ -31,6 +31,8 @@ SoC is on the same page. Required properties: - compatible: should be one of: + - "rockchip,px30-io-voltage-domain" for px30 + - "rockchip,px30-pmu-io-voltage-domain" for px30 pmu-domains - "rockchip,rk3188-io-voltage-domain" for rk3188 - "rockchip,rk3228-io-voltage-domain" for rk3228 - "rockchip,rk3288-io-voltage-domain" for rk3288 @@ -51,6 +53,19 @@ a phandle the relevant regulator. All specified supplies must be able to report their voltage. The IO Voltage Domain for any non-specified supplies will be not be touched. +Possible supplies for PX30: +- vccio6-supply: The supply connected to VCCIO6. +- vccio1-supply: The supply connected to VCCIO1. +- vccio2-supply: The supply connected to VCCIO2. +- vccio3-supply: The supply connected to VCCIO3. +- vccio4-supply: The supply connected to VCCIO4. +- vccio5-supply: The supply connected to VCCIO5. +- vccio-oscgpi-supply: The supply connected to VCCIO_OSCGPI. + +Possible supplies for PX30 pmu-domains: +- pmuio1-supply: The supply connected to PMUIO1. +- pmuio2-supply: The supply connected to PMUIO2. + Possible supplies for rk3188: - ap0-supply: The supply connected to AP0_VCC. - ap1-supply: The supply connected to AP1_VCC. diff --git a/Documentation/devicetree/bindings/power/supply/ab8500/btemp.txt b/Documentation/devicetree/bindings/power/supply/ab8500/btemp.txt index 0ba1bcc7f33a..f181e46d8e07 100644 --- a/Documentation/devicetree/bindings/power/supply/ab8500/btemp.txt +++ b/Documentation/devicetree/bindings/power/supply/ab8500/btemp.txt @@ -13,4 +13,4 @@ Required Properties: }; For information on battery specific node, Ref: -Documentation/devicetree/bindings/power_supply/ab8500/fg.txt +Documentation/devicetree/bindings/power/supply/ab8500/fg.txt diff --git a/Documentation/devicetree/bindings/power/supply/ab8500/chargalg.txt b/Documentation/devicetree/bindings/power/supply/ab8500/chargalg.txt index ef5328371122..56636f927203 100644 --- a/Documentation/devicetree/bindings/power/supply/ab8500/chargalg.txt +++ b/Documentation/devicetree/bindings/power/supply/ab8500/chargalg.txt @@ -13,4 +13,4 @@ ab8500_chargalg { }; For information on battery specific node, Ref: -Documentation/devicetree/bindings/power_supply/ab8500/fg.txt +Documentation/devicetree/bindings/power/supply/ab8500/fg.txt diff --git a/Documentation/devicetree/bindings/power/supply/ab8500/charger.txt b/Documentation/devicetree/bindings/power/supply/ab8500/charger.txt index 6bdbb08ea9e0..24ada03e07b4 100644 --- a/Documentation/devicetree/bindings/power/supply/ab8500/charger.txt +++ b/Documentation/devicetree/bindings/power/supply/ab8500/charger.txt @@ -22,4 +22,4 @@ Required Properties: }; For information on battery specific node, Ref: -Documentation/devicetree/bindings/power_supply/ab8500/fg.txt +Documentation/devicetree/bindings/power/supply/ab8500/fg.txt diff --git a/Documentation/devicetree/bindings/power/supply/bq27xxx.txt b/Documentation/devicetree/bindings/power/supply/bq27xxx.txt index 615c1cb6889f..37994fdb18ca 100644 --- a/Documentation/devicetree/bindings/power/supply/bq27xxx.txt +++ b/Documentation/devicetree/bindings/power/supply/bq27xxx.txt @@ -25,6 +25,7 @@ Required properties: * "ti,bq27545" - BQ27545 * "ti,bq27421" - BQ27421 * "ti,bq27425" - BQ27425 + * "ti,bq27426" - BQ27426 * "ti,bq27441" - BQ27441 * "ti,bq27621" - BQ27621 - reg: integer, I2C address of the fuel gauge. diff --git a/Documentation/devicetree/bindings/power/wakeup-source.txt b/Documentation/devicetree/bindings/power/wakeup-source.txt index 5d254ab13ebf..cfd74659fbed 100644 --- a/Documentation/devicetree/bindings/power/wakeup-source.txt +++ b/Documentation/devicetree/bindings/power/wakeup-source.txt @@ -22,7 +22,7 @@ List of legacy properties and respective binding document 3. "has-tpo" Documentation/devicetree/bindings/rtc/rtc-opal.txt 4. "linux,wakeup" Documentation/devicetree/bindings/input/gpio-matrix-keypad.txt Documentation/devicetree/bindings/mfd/tc3589x.txt - Documentation/devicetree/bindings/input/ads7846.txt + Documentation/devicetree/bindings/input/touchscreen/ads7846.txt 5. "linux,keypad-wakeup" Documentation/devicetree/bindings/input/qcom,pm8xxx-keypad.txt 6. "linux,input-wakeup" Documentation/devicetree/bindings/input/samsung-keypad.txt 7. "nvidia,wakeup-source" Documentation/devicetree/bindings/input/nvidia,tegra20-kbc.txt diff --git a/Documentation/devicetree/bindings/pps/pps-gpio.txt b/Documentation/devicetree/bindings/pps/pps-gpio.txt index 0de23b793657..3683874832ae 100644 --- a/Documentation/devicetree/bindings/pps/pps-gpio.txt +++ b/Documentation/devicetree/bindings/pps/pps-gpio.txt @@ -20,5 +20,4 @@ Example: assert-falling-edge; compatible = "pps-gpio"; - status = "okay"; }; diff --git a/Documentation/devicetree/bindings/ptp/ptp-qoriq.txt b/Documentation/devicetree/bindings/ptp/ptp-qoriq.txt new file mode 100644 index 000000000000..0f569d8e73a3 --- /dev/null +++ b/Documentation/devicetree/bindings/ptp/ptp-qoriq.txt @@ -0,0 +1,69 @@ +* Freescale QorIQ 1588 timer based PTP clock + +General Properties: + + - compatible Should be "fsl,etsec-ptp" + - reg Offset and length of the register set for the device + - interrupts There should be at least two interrupts. Some devices + have as many as four PTP related interrupts. + +Clock Properties: + + - fsl,cksel Timer reference clock source. + - fsl,tclk-period Timer reference clock period in nanoseconds. + - fsl,tmr-prsc Prescaler, divides the output clock. + - fsl,tmr-add Frequency compensation value. + - fsl,tmr-fiper1 Fixed interval period pulse generator. + - fsl,tmr-fiper2 Fixed interval period pulse generator. + - fsl,max-adj Maximum frequency adjustment in parts per billion. + + These properties set the operational parameters for the PTP + clock. You must choose these carefully for the clock to work right. + Here is how to figure good values: + + TimerOsc = selected reference clock MHz + tclk_period = desired clock period nanoseconds + NominalFreq = 1000 / tclk_period MHz + FreqDivRatio = TimerOsc / NominalFreq (must be greater that 1.0) + tmr_add = ceil(2^32 / FreqDivRatio) + OutputClock = NominalFreq / tmr_prsc MHz + PulseWidth = 1 / OutputClock microseconds + FiperFreq1 = desired frequency in Hz + FiperDiv1 = 1000000 * OutputClock / FiperFreq1 + tmr_fiper1 = tmr_prsc * tclk_period * FiperDiv1 - tclk_period + max_adj = 1000000000 * (FreqDivRatio - 1.0) - 1 + + The calculation for tmr_fiper2 is the same as for tmr_fiper1. The + driver expects that tmr_fiper1 will be correctly set to produce a 1 + Pulse Per Second (PPS) signal, since this will be offered to the PPS + subsystem to synchronize the Linux clock. + + Reference clock source is determined by the value, which is holded + in CKSEL bits in TMR_CTRL register. "fsl,cksel" property keeps the + value, which will be directly written in those bits, that is why, + according to reference manual, the next clock sources can be used: + + <0> - external high precision timer reference clock (TSEC_TMR_CLK + input is used for this purpose); + <1> - eTSEC system clock; + <2> - eTSEC1 transmit clock; + <3> - RTC clock input. + + When this attribute is not used, eTSEC system clock will serve as + IEEE 1588 timer reference clock. + +Example: + + ptp_clock@24e00 { + compatible = "fsl,etsec-ptp"; + reg = <0x24E00 0xB0>; + interrupts = <12 0x8 13 0x8>; + interrupt-parent = < &ipic >; + fsl,cksel = <1>; + fsl,tclk-period = <10>; + fsl,tmr-prsc = <100>; + fsl,tmr-add = <0x999999A4>; + fsl,tmr-fiper1 = <0x3B9AC9F6>; + fsl,tmr-fiper2 = <0x00018696>; + fsl,max-adj = <659999998>; + }; diff --git a/Documentation/devicetree/bindings/pwm/pwm-omap-dmtimer.txt b/Documentation/devicetree/bindings/pwm/pwm-omap-dmtimer.txt index 2e53324fb720..5ccfcc82da08 100644 --- a/Documentation/devicetree/bindings/pwm/pwm-omap-dmtimer.txt +++ b/Documentation/devicetree/bindings/pwm/pwm-omap-dmtimer.txt @@ -2,7 +2,7 @@ Required properties: - compatible: Shall contain "ti,omap-dmtimer-pwm". -- ti,timers: phandle to PWM capable OMAP timer. See arm/omap/timer.txt for info +- ti,timers: phandle to PWM capable OMAP timer. See timer/ti,timer.txt for info about these timers. - #pwm-cells: Should be 3. See pwm.txt in this directory for a description of the cells format. diff --git a/Documentation/devicetree/bindings/regulator/pfuze100.txt b/Documentation/devicetree/bindings/regulator/pfuze100.txt index c6dd3f5e485b..f0ada3b14d70 100644 --- a/Documentation/devicetree/bindings/regulator/pfuze100.txt +++ b/Documentation/devicetree/bindings/regulator/pfuze100.txt @@ -21,7 +21,7 @@ Each regulator is defined using the standard binding for regulators. Example 1: PFUZE100 - pmic: pfuze100@8 { + pfuze100: pmic@8 { compatible = "fsl,pfuze100"; reg = <0x08>; @@ -122,7 +122,7 @@ Example 1: PFUZE100 Example 2: PFUZE200 - pmic: pfuze200@8 { + pfuze200: pmic@8 { compatible = "fsl,pfuze200"; reg = <0x08>; @@ -216,7 +216,7 @@ Example 2: PFUZE200 Example 3: PFUZE3000 - pmic: pfuze3000@8 { + pfuze3000: pmic@8 { compatible = "fsl,pfuze3000"; reg = <0x08>; diff --git a/Documentation/devicetree/bindings/regulator/qcom,spmi-regulator.txt b/Documentation/devicetree/bindings/regulator/qcom,spmi-regulator.txt index 57d2c65899df..406f2e570c50 100644 --- a/Documentation/devicetree/bindings/regulator/qcom,spmi-regulator.txt +++ b/Documentation/devicetree/bindings/regulator/qcom,spmi-regulator.txt @@ -110,6 +110,11 @@ Qualcomm SPMI Regulators Definition: Reference to regulator supplying the input pin, as described in the data sheet. +- qcom,saw-reg: + Usage: optional + Value type: + Description: Reference to syscon node defining the SAW registers. + The regulator node houses sub-nodes for each regulator within the device. Each sub-node is identified using the node's name, with valid values listed for each @@ -201,6 +206,17 @@ see regulator.txt - with additional custom properties described below: 2 = 0.55 uA 3 = 0.75 uA +- qcom,saw-slave: + Usage: optional + Value type: + Description: SAW controlled gang slave. Will not be configured. + +- qcom,saw-leader: + Usage: optional + Value type: + Description: SAW controlled gang leader. Will be configured as + SAW regulator. + Example: regulators { @@ -221,3 +237,32 @@ Example: .... }; + +Example 2: + + saw3: syscon@9A10000 { + compatible = "syscon"; + reg = <0x9A10000 0x1000>; + }; + + ... + + spm-regulators { + compatible = "qcom,pm8994-regulators"; + qcom,saw-reg = <&saw3>; + s8 { + qcom,saw-slave; + }; + s9 { + qcom,saw-slave; + }; + s10 { + qcom,saw-slave; + }; + pm8994_s11_saw: s11 { + qcom,saw-leader; + regulator-always-on; + regulator-min-microvolt = <900000>; + regulator-max-microvolt = <1140000>; + }; + }; diff --git a/Documentation/devicetree/bindings/regulator/regulator.txt b/Documentation/devicetree/bindings/regulator/regulator.txt index 2babe15b618d..a7cd36877bfe 100644 --- a/Documentation/devicetree/bindings/regulator/regulator.txt +++ b/Documentation/devicetree/bindings/regulator/regulator.txt @@ -59,6 +59,11 @@ Optional properties: - regulator-initial-mode: initial operating mode. The set of possible operating modes depends on the capabilities of every hardware so each device binding documentation explains which values the regulator supports. +- regulator-allowed-modes: list of operating modes that software is allowed to + configure for the regulator at run-time. Elements may be specified in any + order. The set of possible operating modes depends on the capabilities of + every hardware so each device binding document explains which values the + regulator supports. - regulator-system-load: Load in uA present on regulator that is not captured by any consumer request. - regulator-pull-down: Enable pull down resistor when the regulator is disabled. @@ -68,6 +73,11 @@ Optional properties: 0: Disable active discharge. 1: Enable active discharge. Absence of this property will leave configuration to default. +- regulator-coupled-with: Regulators with which the regulator + is coupled. The linkage is 2-way - all coupled regulators should be linked + with each other. A regulator should not be coupled with its supplier. +- regulator-coupled-max-spread: Max spread between voltages of coupled regulators + in microvolts. Deprecated properties: - regulator-compatible: If a regulator chip contains multiple diff --git a/Documentation/devicetree/bindings/regulator/rohm,bd71837-regulator.txt b/Documentation/devicetree/bindings/regulator/rohm,bd71837-regulator.txt new file mode 100644 index 000000000000..4edf3137d9f7 --- /dev/null +++ b/Documentation/devicetree/bindings/regulator/rohm,bd71837-regulator.txt @@ -0,0 +1,126 @@ +ROHM BD71837 Power Management Integrated Circuit (PMIC) regulator bindings + +BD71837MWV is a programmable Power Management +IC (PMIC) for powering single-core, dual-core, and +quad-core SoC’s such as NXP-i.MX 8M. It is optimized +for low BOM cost and compact solution footprint. It +integrates 8 Buck regulators and 7 LDO’s to provide all +the power rails required by the SoC and the commonly +used peripherals. + +Required properties: + - regulator-name: should be "buck1", ..., "buck8" and "ldo1", ..., "ldo7" + +List of regulators provided by this controller. BD71837 regulators node +should be sub node of the BD71837 MFD node. See BD71837 MFD bindings at +Documentation/devicetree/bindings/mfd/rohm,bd71837-pmic.txt +Regulator nodes should be named to BUCK_ and LDO_. The +definition for each of these nodes is defined using the standard +binding for regulators at +Documentation/devicetree/bindings/regulator/regulator.txt. +Note that if BD71837 starts at RUN state you probably want to use +regulator-boot-on at least for BUCK6 and BUCK7 so that those are not +disabled by driver at startup. LDO5 and LDO6 are supplied by those and +if they are disabled at startup the voltage monitoring for LDO5/LDO6 will +cause PMIC to reset. + +The valid names for regulator nodes are: +BUCK1, BUCK2, BUCK3, BUCK4, BUCK5, BUCK6, BUCK7, BUCK8 +LDO1, LDO2, LDO3, LDO4, LDO5, LDO6, LDO7 + +Optional properties: +- Any optional property defined in bindings/regulator/regulator.txt + +Example: +regulators { + buck1: BUCK1 { + regulator-name = "buck1"; + regulator-min-microvolt = <700000>; + regulator-max-microvolt = <1300000>; + regulator-boot-on; + regulator-ramp-delay = <1250>; + }; + buck2: BUCK2 { + regulator-name = "buck2"; + regulator-min-microvolt = <700000>; + regulator-max-microvolt = <1300000>; + regulator-boot-on; + regulator-always-on; + regulator-ramp-delay = <1250>; + }; + buck3: BUCK3 { + regulator-name = "buck3"; + regulator-min-microvolt = <700000>; + regulator-max-microvolt = <1300000>; + regulator-boot-on; + }; + buck4: BUCK4 { + regulator-name = "buck4"; + regulator-min-microvolt = <700000>; + regulator-max-microvolt = <1300000>; + regulator-boot-on; + }; + buck5: BUCK5 { + regulator-name = "buck5"; + regulator-min-microvolt = <700000>; + regulator-max-microvolt = <1350000>; + regulator-boot-on; + }; + buck6: BUCK6 { + regulator-name = "buck6"; + regulator-min-microvolt = <3000000>; + regulator-max-microvolt = <3300000>; + regulator-boot-on; + }; + buck7: BUCK7 { + regulator-name = "buck7"; + regulator-min-microvolt = <1605000>; + regulator-max-microvolt = <1995000>; + regulator-boot-on; + }; + buck8: BUCK8 { + regulator-name = "buck8"; + regulator-min-microvolt = <800000>; + regulator-max-microvolt = <1400000>; + }; + + ldo1: LDO1 { + regulator-name = "ldo1"; + regulator-min-microvolt = <3000000>; + regulator-max-microvolt = <3300000>; + regulator-boot-on; + }; + ldo2: LDO2 { + regulator-name = "ldo2"; + regulator-min-microvolt = <900000>; + regulator-max-microvolt = <900000>; + regulator-boot-on; + }; + ldo3: LDO3 { + regulator-name = "ldo3"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <3300000>; + }; + ldo4: LDO4 { + regulator-name = "ldo4"; + regulator-min-microvolt = <900000>; + regulator-max-microvolt = <1800000>; + }; + ldo5: LDO5 { + regulator-name = "ldo5"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <3300000>; + }; + ldo6: LDO6 { + regulator-name = "ldo6"; + regulator-min-microvolt = <900000>; + regulator-max-microvolt = <1800000>; + }; + ldo7_reg: LDO7 { + regulator-name = "ldo7"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <3300000>; + }; +}; + + diff --git a/Documentation/devicetree/bindings/regulator/sy8106a-regulator.txt b/Documentation/devicetree/bindings/regulator/sy8106a-regulator.txt new file mode 100644 index 000000000000..39a8ca73f572 --- /dev/null +++ b/Documentation/devicetree/bindings/regulator/sy8106a-regulator.txt @@ -0,0 +1,23 @@ +SY8106A Voltage regulator + +Required properties: +- compatible: Must be "silergy,sy8106a" +- reg: I2C slave address - must be <0x65> +- silergy,fixed-microvolt - the voltage when I2C regulating is disabled (set + by external resistor like a fixed voltage) + +Any property defined as part of the core regulator binding, defined in +./regulator.txt, can also be used. + +Example: + + sy8106a { + compatible = "silergy,sy8106a"; + reg = <0x65>; + regulator-name = "sy8106a-vdd"; + silergy,fixed-microvolt = <1200000>; + regulator-min-microvolt = <1000000>; + regulator-max-microvolt = <1400000>; + regulator-boot-on; + regulator-always-on; + }; diff --git a/Documentation/devicetree/bindings/remoteproc/qcom,q6v5.txt b/Documentation/devicetree/bindings/remoteproc/qcom,q6v5.txt index 00d3d58a102f..d90182425450 100644 --- a/Documentation/devicetree/bindings/remoteproc/qcom,q6v5.txt +++ b/Documentation/devicetree/bindings/remoteproc/qcom,q6v5.txt @@ -11,6 +11,7 @@ on the Qualcomm Hexagon core. "qcom,msm8916-mss-pil", "qcom,msm8974-mss-pil" "qcom,msm8996-mss-pil" + "qcom,sdm845-mss-pil" - reg: Usage: required diff --git a/Documentation/devicetree/bindings/reserved-memory/qcom,cmd-db.txt b/Documentation/devicetree/bindings/reserved-memory/qcom,cmd-db.txt new file mode 100644 index 000000000000..68395530c0a5 --- /dev/null +++ b/Documentation/devicetree/bindings/reserved-memory/qcom,cmd-db.txt @@ -0,0 +1,37 @@ +Command DB +--------- + +Command DB is a database that provides a mapping between resource key and the +resource address for a system resource managed by a remote processor. The data +is stored in a shared memory region and is loaded by the remote processor. + +Some of the Qualcomm Technologies Inc SoC's have hardware accelerators for +controlling shared resources. Depending on the board configuration the shared +resource properties may change. These properties are dynamically probed by the +remote processor and made available in the shared memory. + +The bindings for Command DB is specified in the reserved-memory section in +devicetree. The devicetree representation of the command DB driver should be: + +Properties: +- compatible: + Usage: required + Value type: + Definition: Should be "qcom,cmd-db" + +- reg: + Usage: required + Value type: + Definition: The register address that points to the actual location of + the Command DB in memory. + +Example: + + reserved-memory { + [...] + reserved-memory@85fe0000 { + reg = <0x0 0x85fe0000 0x0 0x20000>; + compatible = "qcom,cmd-db"; + no-map; + }; + }; diff --git a/Documentation/devicetree/bindings/reset/renesas,rst.txt b/Documentation/devicetree/bindings/reset/renesas,rst.txt index 294a0dae106a..67e83b02e10b 100644 --- a/Documentation/devicetree/bindings/reset/renesas,rst.txt +++ b/Documentation/devicetree/bindings/reset/renesas,rst.txt @@ -17,6 +17,7 @@ Required properties: Examples with soctypes are: - "renesas,r8a7743-rst" (RZ/G1M) - "renesas,r8a7745-rst" (RZ/G1E) + - "renesas,r8a77470-rst" (RZ/G1C) - "renesas,r8a7778-reset-wdt" (R-Car M1A) - "renesas,r8a7779-reset-wdt" (R-Car H1) - "renesas,r8a7790-rst" (R-Car H2) @@ -29,6 +30,7 @@ Required properties: - "renesas,r8a77965-rst" (R-Car M3-N) - "renesas,r8a77970-rst" (R-Car V3M) - "renesas,r8a77980-rst" (R-Car V3H) + - "renesas,r8a77990-rst" (R-Car E3) - "renesas,r8a77995-rst" (R-Car D3) - reg: Address start and address range for the device. diff --git a/Documentation/devicetree/bindings/rng/brcm,bcm2835.txt b/Documentation/devicetree/bindings/rng/brcm,bcm2835.txt index 627b29531a32..aaac7975f61c 100644 --- a/Documentation/devicetree/bindings/rng/brcm,bcm2835.txt +++ b/Documentation/devicetree/bindings/rng/brcm,bcm2835.txt @@ -14,11 +14,16 @@ Optional properties: - clocks : phandle to clock-controller plus clock-specifier pair - clock-names : "ipsec" as a clock name +Optional properties: + +- interrupts: specify the interrupt for the RNG block + Example: rng { - compatible = "brcm,bcm2835-rng"; - reg = <0x7e104000 0x10>; + compatible = "brcm,bcm2835-rng"; + reg = <0x7e104000 0x10>; + interrupts = <2 29>; }; rng@18033000 { diff --git a/Documentation/devicetree/bindings/crypto/samsung,exynos-rng4.txt b/Documentation/devicetree/bindings/rng/samsung,exynos4-rng.txt similarity index 100% rename from Documentation/devicetree/bindings/crypto/samsung,exynos-rng4.txt rename to Documentation/devicetree/bindings/rng/samsung,exynos4-rng.txt diff --git a/Documentation/devicetree/bindings/sparc_sun_oracle_rng.txt b/Documentation/devicetree/bindings/rng/sparc_sun_oracle_rng.txt similarity index 100% rename from Documentation/devicetree/bindings/sparc_sun_oracle_rng.txt rename to Documentation/devicetree/bindings/rng/sparc_sun_oracle_rng.txt diff --git a/Documentation/devicetree/bindings/rtc/nxp,rtc-2123.txt b/Documentation/devicetree/bindings/rtc/nxp,rtc-2123.txt index 5cbc0b145a61..811124a36d16 100644 --- a/Documentation/devicetree/bindings/rtc/nxp,rtc-2123.txt +++ b/Documentation/devicetree/bindings/rtc/nxp,rtc-2123.txt @@ -9,7 +9,7 @@ Optional properties: Example: -rtc: nxp,rtc-pcf2123@3 { +pcf2123: rtc@3 { compatible = "nxp,rtc-pcf2123" reg = <3> spi-cs-high; diff --git a/Documentation/devicetree/bindings/rtc/st,stm32-rtc.txt b/Documentation/devicetree/bindings/rtc/st,stm32-rtc.txt index a66692a08ace..c920e2736991 100644 --- a/Documentation/devicetree/bindings/rtc/st,stm32-rtc.txt +++ b/Documentation/devicetree/bindings/rtc/st,stm32-rtc.txt @@ -1,23 +1,29 @@ STM32 Real Time Clock Required properties: -- compatible: can be either "st,stm32-rtc" or "st,stm32h7-rtc", depending on - the device is compatible with stm32(f4/f7) or stm32h7. +- compatible: can be one of the following: + - "st,stm32-rtc" for devices compatible with stm32(f4/f7). + - "st,stm32h7-rtc" for devices compatible with stm32h7. + - "st,stm32mp1-rtc" for devices compatible with stm32mp1. - reg: address range of rtc register set. - clocks: can use up to two clocks, depending on part used: - "rtc_ck": RTC clock source. - It is required on stm32(f4/f7) and stm32h7. - "pclk": RTC APB interface clock. It is not present on stm32(f4/f7). - It is required on stm32h7. + It is required on stm32(h7/mp1). - clock-names: must be "rtc_ck" and "pclk". - It is required only on stm32h7. + It is required on stm32(h7/mp1). - interrupt-parent: phandle for the interrupt controller. -- interrupts: rtc alarm interrupt. -- st,syscfg: phandle for pwrcfg, mandatory to disable/enable backup domain - (RTC registers) write protection. + It is required on stm32(f4/f7/h7). +- interrupts: rtc alarm interrupt. On stm32mp1, a second interrupt is required + for rtc alarm wakeup interrupt. +- st,syscfg: phandle/offset/mask triplet. The phandle to pwrcfg used to + access control register at offset, and change the dbp (Disable Backup + Protection) bit represented by the mask, mandatory to disable/enable backup + domain (RTC registers) write protection. + It is required on stm32(f4/f7/h7). -Optional properties (to override default rtc_ck parent clock): +Optional properties (to override default rtc_ck parent clock on stm32(f4/f7/h7): - assigned-clocks: reference to the rtc_ck clock entry. - assigned-clock-parents: phandle of the new parent clock of rtc_ck. @@ -31,7 +37,7 @@ Example: assigned-clock-parents = <&rcc 1 CLK_LSE>; interrupt-parent = <&exti>; interrupts = <17 1>; - st,syscfg = <&pwrcfg>; + st,syscfg = <&pwrcfg 0x00 0x100>; }; rtc: rtc@58004000 { @@ -44,5 +50,14 @@ Example: interrupt-parent = <&exti>; interrupts = <17 1>; interrupt-names = "alarm"; - st,syscfg = <&pwrcfg>; + st,syscfg = <&pwrcfg 0x00 0x100>; + }; + + rtc: rtc@5c004000 { + compatible = "st,stm32mp1-rtc"; + reg = <0x5c004000 0x400>; + clocks = <&rcc RTCAPB>, <&rcc RTC>; + clock-names = "pclk", "rtc_ck"; + interrupts-extended = <&intc GIC_SPI 3 IRQ_TYPE_NONE>, + <&exti 19 1>; }; diff --git a/Documentation/devicetree/bindings/serial/microchip,pic32-uart.txt b/Documentation/devicetree/bindings/serial/microchip,pic32-uart.txt index 7a34345d0ca3..c8dd440e9747 100644 --- a/Documentation/devicetree/bindings/serial/microchip,pic32-uart.txt +++ b/Documentation/devicetree/bindings/serial/microchip,pic32-uart.txt @@ -8,7 +8,7 @@ Required properties: See: Documentation/devicetree/bindings/clock/clock-bindings.txt - pinctrl-names: A pinctrl state names "default" must be defined. - pinctrl-0: Phandle referencing pin configuration of the UART peripheral. - See: Documentation/devicetree/bindings/pinctrl/pinctrl-binding.txt + See: Documentation/devicetree/bindings/pinctrl/pinctrl-bindings.txt Optional properties: - cts-gpios: CTS pin for UART diff --git a/Documentation/devicetree/bindings/soc/qcom/qcom,apr.txt b/Documentation/devicetree/bindings/soc/qcom/qcom,apr.txt new file mode 100644 index 000000000000..bcc612cc7423 --- /dev/null +++ b/Documentation/devicetree/bindings/soc/qcom/qcom,apr.txt @@ -0,0 +1,84 @@ +Qualcomm APR (Asynchronous Packet Router) binding + +This binding describes the Qualcomm APR. APR is a IPC protocol for +communication between Application processor and QDSP. APR is mainly +used for audio/voice services on the QDSP. + +- compatible: + Usage: required + Value type: + Definition: must be "qcom,apr-v", example "qcom,apr-v2" + +- reg + Usage: required + Value type: + Definition: Destination processor ID. + Possible values are : + 1 - APR simulator + 2 - PC + 3 - MODEM + 4 - ADSP + 5 - APPS + 6 - MODEM2 + 7 - APPS2 + += APR SERVICES +Each subnode of the APR node represents service tied to this apr. The name +of the nodes are not important. The properties of these nodes are defined +by the individual bindings for the specific service +- All APR services MUST contain the following property: + +- reg + Usage: required + Value type: + Definition: APR Service ID + Possible values are : + 3 - DSP Core Service + 4 - Audio Front End Service. + 5 - Voice Stream Manager Service. + 6 - Voice processing manager. + 7 - Audio Stream Manager Service. + 8 - Audio Device Manager Service. + 9 - Multimode voice manager. + 10 - Core voice stream. + 11 - Core voice processor. + 12 - Ultrasound stream manager. + 13 - Listen stream manager. + += EXAMPLE +The following example represents a QDSP based sound card on a MSM8996 device +which uses apr as communication between Apps and QDSP. + + apr@4 { + compatible = "qcom,apr-v2"; + reg = ; + + q6core@3 { + compatible = "qcom,q6core"; + reg = ; + }; + + q6afe@4 { + compatible = "qcom,q6afe"; + reg = ; + + dais { + #sound-dai-cells = <1>; + hdmi@1 { + reg = <1>; + }; + }; + }; + + q6asm@7 { + compatible = "qcom,q6asm"; + reg = ; + ... + }; + + q6adm@8 { + compatible = "qcom,q6adm"; + reg = ; + ... + }; + }; diff --git a/Documentation/devicetree/bindings/soc/qcom/qcom,geni-se.txt b/Documentation/devicetree/bindings/soc/qcom/qcom,geni-se.txt new file mode 100644 index 000000000000..d330c73de9a2 --- /dev/null +++ b/Documentation/devicetree/bindings/soc/qcom/qcom,geni-se.txt @@ -0,0 +1,119 @@ +Qualcomm Technologies, Inc. GENI Serial Engine QUP Wrapper Controller + +Generic Interface (GENI) based Qualcomm Universal Peripheral (QUP) wrapper +is a programmable module for supporting a wide range of serial interfaces +like UART, SPI, I2C, I3C, etc. A single QUP module can provide upto 8 Serial +Interfaces, using its internal Serial Engines. The GENI Serial Engine QUP +Wrapper controller is modeled as a node with zero or more child nodes each +representing a serial engine. + +Required properties: +- compatible: Must be "qcom,geni-se-qup". +- reg: Must contain QUP register address and length. +- clock-names: Must contain "m-ahb" and "s-ahb". +- clocks: AHB clocks needed by the device. + +Required properties if child node exists: +- #address-cells: Must be <1> for Serial Engine Address +- #size-cells: Must be <1> for Serial Engine Address Size +- ranges: Must be present + +Properties for children: + +A GENI based QUP wrapper controller node can contain 0 or more child nodes +representing serial devices. These serial devices can be a QCOM UART, I2C +controller, SPI controller, or some combination of aforementioned devices. +Please refer below the child node definitions for the supported serial +interface protocols. + +Qualcomm Technologies Inc. GENI Serial Engine based I2C Controller + +Required properties: +- compatible: Must be "qcom,geni-i2c". +- reg: Must contain QUP register address and length. +- interrupts: Must contain I2C interrupt. +- clock-names: Must contain "se". +- clocks: Serial engine core clock needed by the device. +- #address-cells: Must be <1> for I2C device address. +- #size-cells: Must be <0> as I2C addresses have no size component. + +Optional property: +- clock-frequency: Desired I2C bus clock frequency in Hz. + When missing default to 400000Hz. + +Child nodes should conform to I2C bus binding as described in i2c.txt. + +Qualcomm Technologies Inc. GENI Serial Engine based UART Controller + +Required properties: +- compatible: Must be "qcom,geni-debug-uart". +- reg: Must contain UART register location and length. +- interrupts: Must contain UART core interrupts. +- clock-names: Must contain "se". +- clocks: Serial engine core clock needed by the device. + +Qualcomm Technologies Inc. GENI Serial Engine based SPI Controller + +Required properties: +- compatible: Must contain "qcom,geni-spi". +- reg: Must contain SPI register location and length. +- interrupts: Must contain SPI controller interrupts. +- clock-names: Must contain "se". +- clocks: Serial engine core clock needed by the device. +- spi-max-frequency: Specifies maximum SPI clock frequency, units - Hz. +- #address-cells: Must be <1> to define a chip select address on + the SPI bus. +- #size-cells: Must be <0>. + +SPI slave nodes must be children of the SPI master node and conform to SPI bus +binding as described in Documentation/devicetree/bindings/spi/spi-bus.txt. + +Example: + geniqup@8c0000 { + compatible = "qcom,geni-se-qup"; + reg = <0x8c0000 0x6000>; + clock-names = "m-ahb", "s-ahb"; + clocks = <&clock_gcc GCC_QUPV3_WRAP_0_M_AHB_CLK>, + <&clock_gcc GCC_QUPV3_WRAP_0_S_AHB_CLK>; + #address-cells = <1>; + #size-cells = <1>; + ranges; + + i2c0: i2c@a94000 { + compatible = "qcom,geni-i2c"; + reg = <0xa94000 0x4000>; + interrupts = ; + clock-names = "se"; + clocks = <&clock_gcc GCC_QUPV3_WRAP0_S5_CLK>; + pinctrl-names = "default", "sleep"; + pinctrl-0 = <&qup_1_i2c_5_active>; + pinctrl-1 = <&qup_1_i2c_5_sleep>; + #address-cells = <1>; + #size-cells = <0>; + }; + + uart0: serial@a88000 { + compatible = "qcom,geni-debug-uart"; + reg = <0xa88000 0x7000>; + interrupts = ; + clock-names = "se"; + clocks = <&clock_gcc GCC_QUPV3_WRAP0_S0_CLK>; + pinctrl-names = "default", "sleep"; + pinctrl-0 = <&qup_1_uart_3_active>; + pinctrl-1 = <&qup_1_uart_3_sleep>; + }; + + spi0: spi@a84000 { + compatible = "qcom,geni-spi"; + reg = <0xa84000 0x4000>; + interrupts = ; + clock-names = "se"; + clocks = <&clock_gcc GCC_QUPV3_WRAP0_S0_CLK>; + pinctrl-names = "default", "sleep"; + pinctrl-0 = <&qup_1_spi_2_active>; + pinctrl-1 = <&qup_1_spi_2_sleep>; + spi-max-frequency = <19200000>; + #address-cells = <1>; + #size-cells = <0>; + }; + } diff --git a/Documentation/devicetree/bindings/soc/qcom/qcom,smd-rpm.txt b/Documentation/devicetree/bindings/soc/qcom/qcom,smd-rpm.txt index a48049ccf6d0..89e1cb9212f6 100644 --- a/Documentation/devicetree/bindings/soc/qcom/qcom,smd-rpm.txt +++ b/Documentation/devicetree/bindings/soc/qcom/qcom,smd-rpm.txt @@ -22,6 +22,7 @@ resources. "qcom,rpm-apq8084" "qcom,rpm-msm8916" "qcom,rpm-msm8974" + "qcom,rpm-msm8998" - qcom,smd-channels: Usage: required diff --git a/Documentation/devicetree/bindings/soc/qcom/qcom,smd.txt b/Documentation/devicetree/bindings/soc/qcom/qcom,smd.txt index ea1dc75ec9ea..234ae2256501 100644 --- a/Documentation/devicetree/bindings/soc/qcom/qcom,smd.txt +++ b/Documentation/devicetree/bindings/soc/qcom/qcom,smd.txt @@ -22,9 +22,15 @@ The edge is described by the following properties: Definition: should specify the IRQ used by the remote processor to signal this processor about communication related updates -- qcom,ipc: +- mboxes: Usage: required Value type: + Definition: reference to the associated doorbell in APCS, as described + in mailbox/mailbox.txt + +- qcom,ipc: + Usage: required, unless mboxes is specified + Value type: Definition: three entries specifying the outgoing ipc bit used for signaling the remote processor: - phandle to a syscon node representing the apcs registers diff --git a/Documentation/devicetree/bindings/soc/rockchip/power_domain.txt b/Documentation/devicetree/bindings/soc/rockchip/power_domain.txt index 301d2a9bc1b8..5d49d0a2ff29 100644 --- a/Documentation/devicetree/bindings/soc/rockchip/power_domain.txt +++ b/Documentation/devicetree/bindings/soc/rockchip/power_domain.txt @@ -5,6 +5,10 @@ powered up/down by software based on different application scenes to save power. Required properties for power domain controller: - compatible: Should be one of the following. + "rockchip,px30-power-controller" - for PX30 SoCs. + "rockchip,rk3036-power-controller" - for RK3036 SoCs. + "rockchip,rk3128-power-controller" - for RK3128 SoCs. + "rockchip,rk3228-power-controller" - for RK3228 SoCs. "rockchip,rk3288-power-controller" - for RK3288 SoCs. "rockchip,rk3328-power-controller" - for RK3328 SoCs. "rockchip,rk3366-power-controller" - for RK3366 SoCs. @@ -17,6 +21,10 @@ Required properties for power domain controller: Required properties for power domain sub nodes: - reg: index of the power domain, should use macros in: + "include/dt-bindings/power/px30-power.h" - for PX30 type power domain. + "include/dt-bindings/power/rk3036-power.h" - for RK3036 type power domain. + "include/dt-bindings/power/rk3128-power.h" - for RK3128 type power domain. + "include/dt-bindings/power/rk3228-power.h" - for RK3228 type power domain. "include/dt-bindings/power/rk3288-power.h" - for RK3288 type power domain. "include/dt-bindings/power/rk3328-power.h" - for RK3328 type power domain. "include/dt-bindings/power/rk3366-power.h" - for RK3366 type power domain. @@ -93,6 +101,10 @@ Node of a device using power domains must have a power-domains property, containing a phandle to the power device node and an index specifying which power domain to use. The index should use macros in: + "include/dt-bindings/power/px30-power.h" - for px30 type power domain. + "include/dt-bindings/power/rk3036-power.h" - for rk3036 type power domain. + "include/dt-bindings/power/rk3128-power.h" - for rk3128 type power domain. + "include/dt-bindings/power/rk3128-power.h" - for rk3228 type power domain. "include/dt-bindings/power/rk3288-power.h" - for rk3288 type power domain. "include/dt-bindings/power/rk3328-power.h" - for rk3328 type power domain. "include/dt-bindings/power/rk3366-power.h" - for rk3366 type power domain. diff --git a/Documentation/devicetree/bindings/soc/ti/keystone-navigator-qmss.txt b/Documentation/devicetree/bindings/soc/ti/keystone-navigator-qmss.txt index 77cd42cc5f54..b025770eeb92 100644 --- a/Documentation/devicetree/bindings/soc/ti/keystone-navigator-qmss.txt +++ b/Documentation/devicetree/bindings/soc/ti/keystone-navigator-qmss.txt @@ -17,7 +17,8 @@ pool management. Required properties: -- compatible : Must be "ti,keystone-navigator-qmss"; +- compatible : Must be "ti,keystone-navigator-qmss". + : Must be "ti,66ak2g-navss-qm" for QMSS on K2G SoC. - clocks : phandle to the reference clock for this device. - queue-range : total range of queue numbers for the device. - linkram0 :
for internal link ram, where size is the total @@ -39,6 +40,12 @@ Required properties: - Descriptor memory setup region. - Queue Management/Queue Proxy region for queue Push. - Queue Management/Queue Proxy region for queue Pop. + +For QMSS on K2G SoC, following QM reg indexes are used in that order + - Queue Peek region. + - Queue configuration region. + - Queue Management/Queue Proxy region for queue Push/Pop. + - queue-pools : child node classifying the queue ranges into pools. Queue ranges are grouped into 3 type of pools: - qpend : pool of qpend(interruptible) queues diff --git a/Documentation/devicetree/bindings/sound/adi,ssm2305.txt b/Documentation/devicetree/bindings/sound/adi,ssm2305.txt new file mode 100644 index 000000000000..a9c9d83c8a30 --- /dev/null +++ b/Documentation/devicetree/bindings/sound/adi,ssm2305.txt @@ -0,0 +1,14 @@ +Analog Devices SSM2305 Speaker Amplifier +======================================== + +Required properties: + - compatible : "adi,ssm2305" + - shutdown-gpios : The gpio connected to the shutdown pin. + The gpio signal is ACTIVE_LOW. + +Example: + +ssm2305: analog-amplifier { + compatible = "adi,ssm2305"; + shutdown-gpios = <&gpio3 20 GPIO_ACTIVE_LOW>; +}; diff --git a/Documentation/devicetree/bindings/sound/atmel-i2s.txt b/Documentation/devicetree/bindings/sound/atmel-i2s.txt new file mode 100644 index 000000000000..735368b8a73f --- /dev/null +++ b/Documentation/devicetree/bindings/sound/atmel-i2s.txt @@ -0,0 +1,47 @@ +* Atmel I2S controller + +Required properties: +- compatible: Should be "atmel,sama5d2-i2s". +- reg: Should be the physical base address of the controller and the + length of memory mapped region. +- interrupts: Should contain the interrupt for the controller. +- dmas: Should be one per channel name listed in the dma-names property, + as described in atmel-dma.txt and dma.txt files. +- dma-names: Two dmas have to be defined, "tx" and "rx". + This IP also supports one shared channel for both rx and tx; + if this mode is used, one "rx-tx" name must be used. +- clocks: Must contain an entry for each entry in clock-names. + Please refer to clock-bindings.txt. +- clock-names: Should be one of each entry matching the clocks phandles list: + - "pclk" (peripheral clock) Required. + - "gclk" (generated clock) Optional (1). + - "aclk" (Audio PLL clock) Optional (1). + - "muxclk" (I2S mux clock) Optional (1). + +Optional properties: +- pinctrl-0: Should specify pin control groups used for this controller. +- princtrl-names: Should contain only one value - "default". + + +(1) : Only the peripheral clock is required. The generated clock, the Audio + PLL clock adn the I2S mux clock are optional and should only be set + together, when Master Mode is required. + +Example: + + i2s@f8050000 { + compatible = "atmel,sama5d2-i2s"; + reg = <0xf8050000 0x300>; + interrupts = <54 IRQ_TYPE_LEVEL_HIGH 7>; + dmas = <&dma0 + (AT91_XDMAC_DT_MEM_IF(0) | AT91_XDMAC_DT_PER_IF(1) | + AT91_XDMAC_DT_PERID(31))>, + <&dma0 + (AT91_XDMAC_DT_MEM_IF(0) | AT91_XDMAC_DT_PER_IF(1) | + AT91_XDMAC_DT_PERID(32))>; + dma-names = "tx", "rx"; + clocks = <&i2s0_clk>, <&i2s0_gclk>, <&audio_pll_pmc>, <&i2s0muxck>; + clock-names = "pclk", "gclk", "aclk", "muxclk"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_i2s0_default>; + }; diff --git a/Documentation/devicetree/bindings/sound/cs42xx8.txt b/Documentation/devicetree/bindings/sound/cs42xx8.txt index f631fbca6284..8619a156d038 100644 --- a/Documentation/devicetree/bindings/sound/cs42xx8.txt +++ b/Documentation/devicetree/bindings/sound/cs42xx8.txt @@ -16,7 +16,7 @@ Required properties: Example: -codec: cs42888@48 { +cs42888: codec@48 { compatible = "cirrus,cs42888"; reg = <0x48>; clocks = <&codec_mclk 0>; diff --git a/Documentation/devicetree/bindings/sound/fsl,asrc.txt b/Documentation/devicetree/bindings/sound/fsl,asrc.txt index f5a14115b459..1d4d9f938689 100644 --- a/Documentation/devicetree/bindings/sound/fsl,asrc.txt +++ b/Documentation/devicetree/bindings/sound/fsl,asrc.txt @@ -31,14 +31,16 @@ Required properties: it. This property is optional depending on the SoC design. - - big-endian : If this property is absent, the little endian mode - will be in use as default. Otherwise, the big endian - mode will be in use for all the device registers. - - fsl,asrc-rate : Defines a mutual sample rate used by DPCM Back Ends. - fsl,asrc-width : Defines a mutual sample width used by DPCM Back Ends. +Optional properties: + + - big-endian : If this property is absent, the little endian mode + will be in use as default. Otherwise, the big endian + mode will be in use for all the device registers. + Example: asrc: asrc@2034000 { diff --git a/Documentation/devicetree/bindings/sound/fsl,esai.txt b/Documentation/devicetree/bindings/sound/fsl,esai.txt index cacd18bb9ba6..5b9914367610 100644 --- a/Documentation/devicetree/bindings/sound/fsl,esai.txt +++ b/Documentation/devicetree/bindings/sound/fsl,esai.txt @@ -42,6 +42,8 @@ Required properties: means all the settings for Receiving would be duplicated from Transmition related registers. +Optional properties: + - big-endian : If this property is absent, the native endian mode will be in use as default, or the big endian mode will be in use for all the device registers. diff --git a/Documentation/devicetree/bindings/sound/fsl,spdif.txt b/Documentation/devicetree/bindings/sound/fsl,spdif.txt index 38cfa7573441..8b324f82a782 100644 --- a/Documentation/devicetree/bindings/sound/fsl,spdif.txt +++ b/Documentation/devicetree/bindings/sound/fsl,spdif.txt @@ -33,6 +33,8 @@ Required properties: it. This property is optional depending on the SoC design. +Optional properties: + - big-endian : If this property is absent, the native endian mode will be in use as default, or the big endian mode will be in use for all the device registers. diff --git a/Documentation/devicetree/bindings/sound/fsl-sai.txt b/Documentation/devicetree/bindings/sound/fsl-sai.txt index 740b467adf7d..dd9e59738e08 100644 --- a/Documentation/devicetree/bindings/sound/fsl-sai.txt +++ b/Documentation/devicetree/bindings/sound/fsl-sai.txt @@ -28,9 +28,6 @@ Required properties: pinctrl-names. See ../pinctrl/pinctrl-bindings.txt for details of the property values. - - big-endian : Boolean property, required if all the FTM_PWM - registers are big-endian rather than little-endian. - - lsb-first : Configures whether the LSB or the MSB is transmitted first for the fifo data. If this property is absent, the MSB is transmitted first as default, or the LSB @@ -48,6 +45,11 @@ Required properties: receive data by following their own bit clocks and frame sync clocks separately. +Optional properties: + + - big-endian : Boolean property, required if all the SAI + registers are big-endian rather than little-endian. + Optional properties (for mx6ul): - fsl,sai-mclk-direction-output: This is a boolean property. If present, diff --git a/Documentation/devicetree/bindings/sound/mt2701-afe-pcm.txt b/Documentation/devicetree/bindings/sound/mt2701-afe-pcm.txt index e2f7f4951215..560762e0a168 100644 --- a/Documentation/devicetree/bindings/sound/mt2701-afe-pcm.txt +++ b/Documentation/devicetree/bindings/sound/mt2701-afe-pcm.txt @@ -1,7 +1,9 @@ Mediatek AFE PCM controller for mt2701 Required properties: -- compatible = "mediatek,mt2701-audio"; +- compatible: should be one of the followings. + - "mediatek,mt2701-audio" + - "mediatek,mt7622-audio" - interrupts: should contain AFE and ASYS interrupts - interrupt-names: should be "afe" and "asys" - power-domains: should define the power domain diff --git a/Documentation/devicetree/bindings/sound/mt6351.txt b/Documentation/devicetree/bindings/sound/mt6351.txt new file mode 100644 index 000000000000..7fb2cb99245e --- /dev/null +++ b/Documentation/devicetree/bindings/sound/mt6351.txt @@ -0,0 +1,16 @@ +Mediatek MT6351 Audio Codec + +The communication between MT6351 and SoC is through Mediatek PMIC wrapper. +For more detail, please visit Mediatek PMIC wrapper documentation. + +Must be a child node of PMIC wrapper. + +Required properties: + +- compatible : "mediatek,mt6351-sound". + +Example: + +mt6351_snd { + compatible = "mediatek,mt6351-sound"; +}; diff --git a/Documentation/devicetree/bindings/sound/mt6797-afe-pcm.txt b/Documentation/devicetree/bindings/sound/mt6797-afe-pcm.txt new file mode 100644 index 000000000000..0ae29de15bfd --- /dev/null +++ b/Documentation/devicetree/bindings/sound/mt6797-afe-pcm.txt @@ -0,0 +1,42 @@ +Mediatek AFE PCM controller for mt6797 + +Required properties: +- compatible = "mediatek,mt6797-audio"; +- reg: register location and size +- interrupts: should contain AFE interrupt +- power-domains: should define the power domain +- clocks: Must contain an entry for each entry in clock-names +- clock-names: should have these clock names: + "infra_sys_audio_clk", + "infra_sys_audio_26m", + "mtkaif_26m_clk", + "top_mux_audio", + "top_mux_aud_intbus", + "top_sys_pll3_d4", + "top_sys_pll1_d4", + "top_clk26m_clk"; + +Example: + + afe: mt6797-afe-pcm@11220000 { + compatible = "mediatek,mt6797-audio"; + reg = <0 0x11220000 0 0x1000>; + interrupts = ; + power-domains = <&scpsys MT6797_POWER_DOMAIN_AUDIO>; + clocks = <&infrasys CLK_INFRA_AUDIO>, + <&infrasys CLK_INFRA_AUDIO_26M>, + <&infrasys CLK_INFRA_AUDIO_26M_PAD_TOP>, + <&topckgen CLK_TOP_MUX_AUDIO>, + <&topckgen CLK_TOP_MUX_AUD_INTBUS>, + <&topckgen CLK_TOP_SYSPLL3_D4>, + <&topckgen CLK_TOP_SYSPLL1_D4>, + <&clk26m>; + clock-names = "infra_sys_audio_clk", + "infra_sys_audio_26m", + "mtkaif_26m_clk", + "top_mux_audio", + "top_mux_aud_intbus", + "top_sys_pll3_d4", + "top_sys_pll1_d4", + "top_clk26m_clk"; + }; diff --git a/Documentation/devicetree/bindings/sound/mt6797-mt6351.txt b/Documentation/devicetree/bindings/sound/mt6797-mt6351.txt new file mode 100644 index 000000000000..1d95a8840f19 --- /dev/null +++ b/Documentation/devicetree/bindings/sound/mt6797-mt6351.txt @@ -0,0 +1,14 @@ +MT6797 with MT6351 CODEC + +Required properties: +- compatible: "mediatek,mt6797-mt6351-sound" +- mediatek,platform: the phandle of MT6797 ASoC platform +- mediatek,audio-codec: the phandles of MT6351 codec + +Example: + + sound { + compatible = "mediatek,mt6797-mt6351-sound"; + mediatek,audio-codec = <&mt6351_snd>; + mediatek,platform = <&afe>; + }; diff --git a/Documentation/devicetree/bindings/sound/qcom,apq8096.txt b/Documentation/devicetree/bindings/sound/qcom,apq8096.txt new file mode 100644 index 000000000000..aa54e49fc8a2 --- /dev/null +++ b/Documentation/devicetree/bindings/sound/qcom,apq8096.txt @@ -0,0 +1,109 @@ +* Qualcomm Technologies APQ8096 ASoC sound card driver + +This binding describes the APQ8096 sound card, which uses qdsp for audio. + +- compatible: + Usage: required + Value type: + Definition: must be "qcom,apq8096-sndcard" + +- qcom,audio-routing: + Usage: Optional + Value type: + Definition: A list of the connections between audio components. + Each entry is a pair of strings, the first being the + connection's sink, the second being the connection's + source. Valid names could be power supplies, MicBias + of codec and the jacks on the board: + Valid names include: + + Board Connectors: + "Headphone Left" + "Headphone Right" + "Earphone" + "Line Out1" + "Line Out2" + "Line Out3" + "Line Out4" + "Analog Mic1" + "Analog Mic2" + "Analog Mic3" + "Analog Mic4" + "Analog Mic5" + "Analog Mic6" + "Digital Mic2" + "Digital Mic3" + + Audio pins and MicBias on WCD9335 Codec: + "MIC_BIAS1 + "MIC_BIAS2" + "MIC_BIAS3" + "MIC_BIAS4" + "AMIC1" + "AMIC2" + "AMIC3" + "AMIC4" + "AMIC5" + "AMIC6" + "AMIC6" + "DMIC1" + "DMIC2" + "DMIC3" += dailinks +Each subnode of sndcard represents either a dailink, and subnodes of each +dailinks would be cpu/codec/platform dais. + +- link-name: + Usage: required + Value type: + Definition: User friendly name for dai link + += CPU, PLATFORM, CODEC dais subnodes +- cpu: + Usage: required + Value type: + Definition: cpu dai sub-node + +- codec: + Usage: Optional + Value type: + Definition: codec dai sub-node + +- platform: + Usage: Optional + Value type: + Definition: platform dai sub-node + +- sound-dai: + Usage: required + Value type: + Definition: dai phandle/s and port of CPU/CODEC/PLATFORM node. + +Example: + +audio { + compatible = "qcom,apq8096-sndcard"; + qcom,model = "DB820c"; + + mm1-dai-link { + link-name = "MultiMedia1"; + cpu { + sound-dai = <&q6asmdai MSM_FRONTEND_DAI_MULTIMEDIA1>; + }; + }; + + hdmi-dai-link { + link-name = "HDMI Playback"; + cpu { + sound-dai = <&q6afe HDMI_RX>; + }; + + platform { + sound-dai = <&q6adm>; + }; + + codec { + sound-dai = <&hdmi 0>; + }; + }; +}; diff --git a/Documentation/devicetree/bindings/sound/qcom,q6adm.txt b/Documentation/devicetree/bindings/sound/qcom,q6adm.txt new file mode 100644 index 000000000000..cb709e5dbc44 --- /dev/null +++ b/Documentation/devicetree/bindings/sound/qcom,q6adm.txt @@ -0,0 +1,33 @@ +Qualcomm Audio Device Manager (Q6ADM) binding + +Q6ADM is one of the APR audio service on Q6DSP. +Please refer to qcom,apr.txt for details of the coommon apr service bindings +used by the apr service device. + +- but must contain the following property: + +- compatible: + Usage: required + Value type: + Definition: must be "qcom,q6adm-v.". + Or "qcom,q6adm" where the version number can be queried + from DSP. + example "qcom,q6adm-v2.0" + + += ADM routing +"routing" subnode of the ADM node represents adm routing specific configuration + +- #sound-dai-cells + Usage: required + Value type: + Definition: Must be 0 + += EXAMPLE +q6adm@8 { + compatible = "qcom,q6adm"; + reg = ; + q6routing: routing { + #sound-dai-cells = <0>; + }; +}; diff --git a/Documentation/devicetree/bindings/sound/qcom,q6afe.txt b/Documentation/devicetree/bindings/sound/qcom,q6afe.txt new file mode 100644 index 000000000000..bdbf87df8c0b --- /dev/null +++ b/Documentation/devicetree/bindings/sound/qcom,q6afe.txt @@ -0,0 +1,172 @@ +Qualcomm Audio Front End (Q6AFE) binding + +AFE is one of the APR audio service on Q6DSP +Please refer to qcom,apr.txt for details of the common apr service bindings +used by all apr services. Must contain the following properties. + +- compatible: + Usage: required + Value type: + Definition: must be "qcom,q6afe-v." + Or "qcom,q6afe" where the version number can be queried + from DSP. + example "qcom,q6afe" + += AFE DAIs (Digial Audio Interface) +"dais" subnode of the AFE node. It represents afe dais, each afe dai is a +subnode of "dais" representing board specific dai setup. +"dais" node should have following properties followed by dai children. + +- #sound-dai-cells + Usage: required + Value type: + Definition: Must be 1 + +- #address-cells + Usage: required + Value type: + Definition: Must be 1 + +- #size-cells + Usage: required + Value type: + Definition: Must be 0 + +== AFE DAI is subnode of "dais" and represent a dai, it includes board specific +configuration of each dai. Must contain the following properties. + +- reg + Usage: required + Value type: + Definition: Must be dai id + +- qcom,sd-lines + Usage: required for mi2s interface + Value type: + Definition: Must be list of serial data lines used by this dai. + should be one or more of the 1-4 sd lines. + + - qcom,tdm-sync-mode: + Usage: required for tdm interface + Value type: + Definition: Synchronization mode. + 0 - Short sync bit mode + 1 - Long sync mode + 2 - Short sync slot mode + + - qcom,tdm-sync-src: + Usage: required for tdm interface + Value type: + Definition: Synchronization source. + 0 - External source + 1 - Internal source + + - qcom,tdm-data-out: + Usage: required for tdm interface + Value type: + Definition: Data out signal to drive with other masters. + 0 - Disable + 1 - Enable + + - qcom,tdm-invert-sync: + Usage: required for tdm interface + Value type: + Definition: Invert the sync. + 0 - Normal + 1 - Invert + + - qcom,tdm-data-delay: + Usage: required for tdm interface + Value type: + Definition: Number of bit clock to delay data + with respect to sync edge. + 0 - 0 bit clock cycle + 1 - 1 bit clock cycle + 2 - 2 bit clock cycle + + - qcom,tdm-data-align: + Usage: required for tdm interface + Value type: + Definition: Indicate how data is packed + within the slot. For example, 32 slot width in case of + sample bit width is 24. + 0 - MSB + 1 - LSB + += EXAMPLE + +q6afe@4 { + compatible = "qcom,q6afe"; + reg = ; + + dais { + #sound-dai-cells = <1>; + #address-cells = <1>; + #size-cells = <0>; + + hdmi@1 { + reg = <1>; + }; + + tdm@24 { + reg = <24>; + qcom,tdm-sync-mode = <1>: + qcom,tdm-sync-src = <1>; + qcom,tdm-data-out = <0>; + qcom,tdm-invert-sync = <1>; + qcom,tdm-data-delay = <1>; + qcom,tdm-data-align = <0>; + + }; + + tdm@25 { + reg = <25>; + qcom,tdm-sync-mode = <1>: + qcom,tdm-sync-src = <1>; + qcom,tdm-data-out = <0>; + qcom,tdm-invert-sync = <1>; + qcom,tdm-data-delay <1>: + qcom,tdm-data-align = <0>; + }; + + prim-mi2s-rx@16 { + reg = <16>; + qcom,sd-lines = <1 3>; + }; + + prim-mi2s-tx@17 { + reg = <17>; + qcom,sd-lines = <2>; + }; + + sec-mi2s-rx@18 { + reg = <18>; + qcom,sd-lines = <1 4>; + }; + + sec-mi2s-tx@19 { + reg = <19>; + qcom,sd-lines = <2>; + }; + + tert-mi2s-rx@20 { + reg = <20>; + qcom,sd-lines = <2 4>; + }; + + tert-mi2s-tx@21 { + reg = <21>; + qcom,sd-lines = <1>; + }; + + quat-mi2s-rx@22 { + reg = <22>; + qcom,sd-lines = <1>; + }; + + quat-mi2s-tx@23 { + reg = <23>; + qcom,sd-lines = <2>; + }; + }; +}; diff --git a/Documentation/devicetree/bindings/sound/qcom,q6asm.txt b/Documentation/devicetree/bindings/sound/qcom,q6asm.txt new file mode 100644 index 000000000000..2178eb91146f --- /dev/null +++ b/Documentation/devicetree/bindings/sound/qcom,q6asm.txt @@ -0,0 +1,33 @@ +Qualcomm Audio Stream Manager (Q6ASM) binding + +Q6ASM is one of the APR audio service on Q6DSP. +Please refer to qcom,apr.txt for details of the common apr service bindings +used by the apr service device. + +- but must contain the following property: + +- compatible: + Usage: required + Value type: + Definition: must be "qcom,q6asm-v.". + Or "qcom,q6asm" where the version number can be queried + from DSP. + example "qcom,q6asm-v2.0" + += ASM DAIs (Digial Audio Interface) +"dais" subnode of the ASM node represents dai specific configuration + +- #sound-dai-cells + Usage: required + Value type: + Definition: Must be 1 + += EXAMPLE + +q6asm@7 { + compatible = "qcom,q6asm"; + reg = ; + q6asmdai: dais { + #sound-dai-cells = <1>; + }; +}; diff --git a/Documentation/devicetree/bindings/sound/qcom,q6core.txt b/Documentation/devicetree/bindings/sound/qcom,q6core.txt new file mode 100644 index 000000000000..7f36ff8bec18 --- /dev/null +++ b/Documentation/devicetree/bindings/sound/qcom,q6core.txt @@ -0,0 +1,21 @@ +Qualcomm ADSP Core service binding + +Q6CORE is one of the APR audio service on Q6DSP. +Please refer to qcom,apr.txt for details of the common apr service bindings +used by the apr service device. + +- but must contain the following property: + +- compatible: + Usage: required + Value type: + Definition: must be "qcom,q6core-v.". + Or "qcom,q6core" where the version number can be queried + from DSP. + example "qcom,q6core-v2.0" + += EXAMPLE +q6core@3 { + compatible = "qcom,q6core"; + reg = ; +}; diff --git a/Documentation/devicetree/bindings/sound/rt274.txt b/Documentation/devicetree/bindings/sound/rt274.txt index e9a6178c78cf..791a1bd767b9 100644 --- a/Documentation/devicetree/bindings/sound/rt274.txt +++ b/Documentation/devicetree/bindings/sound/rt274.txt @@ -26,7 +26,7 @@ Pins on the device (for linking into audio routes) for RT274: Example: -codec: rt274@1c { +rt274: codec@1c { compatible = "realtek,rt274"; reg = <0x1c>; interrupts = <7 IRQ_TYPE_EDGE_FALLING>; diff --git a/Documentation/devicetree/bindings/sound/rt5514.txt b/Documentation/devicetree/bindings/sound/rt5514.txt index 4f33b0d96afe..b25ed08c7a5a 100644 --- a/Documentation/devicetree/bindings/sound/rt5514.txt +++ b/Documentation/devicetree/bindings/sound/rt5514.txt @@ -32,7 +32,7 @@ Pins on the device (for linking into audio routes) for I2C: Example: -codec: rt5514@57 { +rt5514: codec@57 { compatible = "realtek,rt5514"; reg = <0x57>; }; diff --git a/Documentation/devicetree/bindings/sound/rt5616.txt b/Documentation/devicetree/bindings/sound/rt5616.txt index e41085818559..540a4bf252e4 100644 --- a/Documentation/devicetree/bindings/sound/rt5616.txt +++ b/Documentation/devicetree/bindings/sound/rt5616.txt @@ -26,7 +26,7 @@ Pins on the device (for linking into audio routes) for RT5616: Example: -codec: rt5616@1b { +rt5616: codec@1b { compatible = "realtek,rt5616"; reg = <0x1b>; }; diff --git a/Documentation/devicetree/bindings/sound/rt5640.txt b/Documentation/devicetree/bindings/sound/rt5640.txt index 57fe64643050..e40e4893eed8 100644 --- a/Documentation/devicetree/bindings/sound/rt5640.txt +++ b/Documentation/devicetree/bindings/sound/rt5640.txt @@ -22,6 +22,41 @@ Optional properties: - realtek,ldo1-en-gpios : The GPIO that controls the CODEC's LDO1_EN pin. +- realtek,dmic1-data-pin + 0: dmic1 is not used + 1: using IN1P pin as dmic1 data pin + 2: using GPIO3 pin as dmic1 data pin + +- realtek,dmic2-data-pin + 0: dmic2 is not used + 1: using IN1N pin as dmic2 data pin + 2: using GPIO4 pin as dmic2 data pin + +- realtek,jack-detect-source + u32. Valid values: + 0: jack-detect is not used + 1: Use GPIO1 for jack-detect + 2: Use JD1_IN4P for jack-detect + 3: Use JD2_IN4N for jack-detect + 4: Use GPIO2 for jack-detect + 5: Use GPIO3 for jack-detect + 6: Use GPIO4 for jack-detect + +- realtek,jack-detect-not-inverted + bool. Normal jack-detect switches give an inverted signal, set this bool + in the rare case you've a jack-detect switch which is not inverted. + +- realtek,over-current-threshold-microamp + u32, micbias over-current detection threshold in µA, valid values are + 600, 1500 and 2000µA. + +- realtek,over-current-scale-factor + u32, micbias over-current detection scale-factor, valid values are: + 0: Scale current by 0.5 + 1: Scale current by 0.75 + 2: Scale current by 1.0 + 3: Scale current by 1.5 + Pins on the device (for linking into audio routes) for RT5639/RT5640: * DMIC1 diff --git a/Documentation/devicetree/bindings/sound/rt5645.txt b/Documentation/devicetree/bindings/sound/rt5645.txt index 7cee1f518f59..a03f9a872a71 100644 --- a/Documentation/devicetree/bindings/sound/rt5645.txt +++ b/Documentation/devicetree/bindings/sound/rt5645.txt @@ -69,4 +69,4 @@ codec: rt5650@1a { realtek,dmic-en = "true"; realtek,en-jd-func = "true"; realtek,jd-mode = <3>; -}; \ No newline at end of file +}; diff --git a/Documentation/devicetree/bindings/sound/rt5651.txt b/Documentation/devicetree/bindings/sound/rt5651.txt index b85221864cec..a41199a5cd79 100644 --- a/Documentation/devicetree/bindings/sound/rt5651.txt +++ b/Documentation/devicetree/bindings/sound/rt5651.txt @@ -50,7 +50,7 @@ Pins on the device (for linking into audio routes) for RT5651: Example: -codec: rt5651@1a { +rt5651: codec@1a { compatible = "realtek,rt5651"; reg = <0x1a>; realtek,dmic-en = "true"; diff --git a/Documentation/devicetree/bindings/sound/rt5663.txt b/Documentation/devicetree/bindings/sound/rt5663.txt index 497bcfc58b71..23386446c63d 100644 --- a/Documentation/devicetree/bindings/sound/rt5663.txt +++ b/Documentation/devicetree/bindings/sound/rt5663.txt @@ -47,7 +47,7 @@ Pins on the device (for linking into audio routes) for RT5663: Example: -codec: rt5663@12 { +rt5663: codec@12 { compatible = "realtek,rt5663"; reg = <0x12>; interrupts = <7 IRQ_TYPE_EDGE_FALLING>; diff --git a/Documentation/devicetree/bindings/sound/rt5668.txt b/Documentation/devicetree/bindings/sound/rt5668.txt new file mode 100644 index 000000000000..c88b96e7764b --- /dev/null +++ b/Documentation/devicetree/bindings/sound/rt5668.txt @@ -0,0 +1,50 @@ +RT5668B audio CODEC + +This device supports I2C only. + +Required properties: + +- compatible : "realtek,rt5668b" + +- reg : The I2C address of the device. + +Optional properties: + +- interrupts : The CODEC's interrupt output. + +- realtek,dmic1-data-pin + 0: dmic1 is not used + 1: using GPIO2 pin as dmic1 data pin + 2: using GPIO5 pin as dmic1 data pin + +- realtek,dmic1-clk-pin + 0: using GPIO1 pin as dmic1 clock pin + 1: using GPIO3 pin as dmic1 clock pin + +- realtek,jd-src + 0: No JD is used + 1: using JD1 as JD source + +- realtek,ldo1-en-gpios : The GPIO that controls the CODEC's LDO1_EN pin. + +Pins on the device (for linking into audio routes) for RT5668B: + + * DMIC L1 + * DMIC R1 + * IN1P + * HPOL + * HPOR + +Example: + +rt5668 { + compatible = "realtek,rt5668b"; + reg = <0x1a>; + interrupt-parent = <&gpio>; + interrupts = ; + realtek,ldo1-en-gpios = + <&gpio TEGRA_GPIO(R, 2) GPIO_ACTIVE_HIGH>; + realtek,dmic1-data-pin = <1>; + realtek,dmic1-clk-pin = <1>; + realtek,jd-src = <1>; +}; diff --git a/Documentation/devicetree/bindings/sound/sgtl5000.txt b/Documentation/devicetree/bindings/sound/sgtl5000.txt index 9a36c7e2a143..0f214457476f 100644 --- a/Documentation/devicetree/bindings/sound/sgtl5000.txt +++ b/Documentation/devicetree/bindings/sound/sgtl5000.txt @@ -39,7 +39,7 @@ VDDIO 1.8V 2.5V 3.3V Example: -codec: sgtl5000@a { +sgtl5000: codec@a { compatible = "fsl,sgtl5000"; reg = <0x0a>; #sound-dai-cells = <0>; diff --git a/Documentation/devicetree/bindings/sound/simple-card.txt b/Documentation/devicetree/bindings/sound/simple-card.txt index 17c13e74667d..a4c72d09cd45 100644 --- a/Documentation/devicetree/bindings/sound/simple-card.txt +++ b/Documentation/devicetree/bindings/sound/simple-card.txt @@ -86,6 +86,11 @@ Optional CPU/CODEC subnodes properties: in dai startup() and disabled with clk_disable_unprepare() in dai shutdown(). + If a clock is specified and a + multiplication factor is given with + mclk-fs, the clock will be set to the + calculated mclk frequency when the + stream starts. - system-clock-direction-out : specifies clock direction as 'out' on initialization. It is useful for some aCPUs with fixed clocks. diff --git a/Documentation/devicetree/bindings/sound/st,stm32-i2s.txt b/Documentation/devicetree/bindings/sound/st,stm32-i2s.txt index 4bda52042402..58c341300552 100644 --- a/Documentation/devicetree/bindings/sound/st,stm32-i2s.txt +++ b/Documentation/devicetree/bindings/sound/st,stm32-i2s.txt @@ -18,7 +18,7 @@ Required properties: See Documentation/devicetree/bindings/dma/stm32-dma.txt. - dma-names: Identifier for each DMA request line. Must be "tx" and "rx". - pinctrl-names: should contain only value "default" - - pinctrl-0: see Documentation/devicetree/bindings/pinctrl/pinctrl-stm32.txt + - pinctrl-0: see Documentation/devicetree/bindings/pinctrl/st,stm32-pinctrl.txt Optional properties: - resets: Reference to a reset controller asserting the reset controller diff --git a/Documentation/devicetree/bindings/sound/st,stm32-sai.txt b/Documentation/devicetree/bindings/sound/st,stm32-sai.txt index f301cdf0b7e6..3a3fc506e43a 100644 --- a/Documentation/devicetree/bindings/sound/st,stm32-sai.txt +++ b/Documentation/devicetree/bindings/sound/st,stm32-sai.txt @@ -37,7 +37,7 @@ SAI subnodes required properties: "tx": if sai sub-block is configured as playback DAI "rx": if sai sub-block is configured as capture DAI - pinctrl-names: should contain only value "default" - - pinctrl-0: see Documentation/devicetree/bindings/pinctrl/pinctrl-stm32.txt + - pinctrl-0: see Documentation/devicetree/bindings/pinctrl/st,stm32-pinctrl.txt SAI subnodes Optional properties: - st,sync: specify synchronization mode. diff --git a/Documentation/devicetree/bindings/sound/ti,tas6424.txt b/Documentation/devicetree/bindings/sound/ti,tas6424.txt index 1c4ada0eef4e..eacb54f34188 100644 --- a/Documentation/devicetree/bindings/sound/ti,tas6424.txt +++ b/Documentation/devicetree/bindings/sound/ti,tas6424.txt @@ -6,6 +6,8 @@ Required properties: - compatible: "ti,tas6424" - TAS6424 - reg: I2C slave address - sound-dai-cells: must be equal to 0 + - standby-gpios: GPIO used to shut the TAS6424 down. + - mute-gpios: GPIO used to mute all the outputs Example: diff --git a/Documentation/devicetree/bindings/sound/tscs42xx.txt b/Documentation/devicetree/bindings/sound/tscs42xx.txt index 2ac2f0996697..7eea32e9d078 100644 --- a/Documentation/devicetree/bindings/sound/tscs42xx.txt +++ b/Documentation/devicetree/bindings/sound/tscs42xx.txt @@ -8,9 +8,15 @@ Required Properties: - reg : <0x71> for analog mic <0x69> for digital mic + - clock-names: Must one of the following "mclk1", "xtal", "mclk2" + + - clocks: phandle of the clock that provides the codec sysclk + Example: wookie: codec@69 { compatible = "tempo,tscs42A2"; reg = <0x69>; + clock-names = "xtal"; + clocks = <&audio_xtal>; }; diff --git a/Documentation/devicetree/bindings/sound/tscs454.txt b/Documentation/devicetree/bindings/sound/tscs454.txt new file mode 100644 index 000000000000..3ba3e2d2c206 --- /dev/null +++ b/Documentation/devicetree/bindings/sound/tscs454.txt @@ -0,0 +1,23 @@ +TSCS454 Audio CODEC + +Required Properties: + + - compatible : "tempo,tscs454" + + - reg : <0x69> + + - clock-names: Must one of the following "xtal", "mclk1", "mclk2" + + - clocks: phandle of the clock that provides the codec sysclk + + Note: If clock is not provided then bit clock is assumed + +Example: + +redwood: codec@69 { + #sound-dai-cells = <1>; + compatible = "tempo,tscs454"; + reg = <0x69>; + clock-names = "mclk1"; + clocks = <&audio_mclk>; +}; diff --git a/Documentation/devicetree/bindings/sound/wm8510.txt b/Documentation/devicetree/bindings/sound/wm8510.txt index fa1a32b85577..e6b6cc041f89 100644 --- a/Documentation/devicetree/bindings/sound/wm8510.txt +++ b/Documentation/devicetree/bindings/sound/wm8510.txt @@ -12,7 +12,7 @@ Required properties: Example: -codec: wm8510@1a { +wm8510: codec@1a { compatible = "wlf,wm8510"; reg = <0x1a>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8523.txt b/Documentation/devicetree/bindings/sound/wm8523.txt index 04746186b283..f3a6485f4b8a 100644 --- a/Documentation/devicetree/bindings/sound/wm8523.txt +++ b/Documentation/devicetree/bindings/sound/wm8523.txt @@ -10,7 +10,7 @@ Required properties: Example: -codec: wm8523@1a { +wm8523: codec@1a { compatible = "wlf,wm8523"; reg = <0x1a>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8524.txt b/Documentation/devicetree/bindings/sound/wm8524.txt index 0f0553563fc1..f6c0c263b135 100644 --- a/Documentation/devicetree/bindings/sound/wm8524.txt +++ b/Documentation/devicetree/bindings/sound/wm8524.txt @@ -10,7 +10,7 @@ Required properties: Example: -codec: wm8524 { +wm8524: codec { compatible = "wlf,wm8524"; wlf,mute-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8580.txt b/Documentation/devicetree/bindings/sound/wm8580.txt index 78fce9b14954..ff3f9f5f2111 100644 --- a/Documentation/devicetree/bindings/sound/wm8580.txt +++ b/Documentation/devicetree/bindings/sound/wm8580.txt @@ -10,7 +10,7 @@ Required properties: Example: -codec: wm8580@1a { +wm8580: codec@1a { compatible = "wlf,wm8580"; reg = <0x1a>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8711.txt b/Documentation/devicetree/bindings/sound/wm8711.txt index 8ed9998cd23c..c30a1387c4bf 100644 --- a/Documentation/devicetree/bindings/sound/wm8711.txt +++ b/Documentation/devicetree/bindings/sound/wm8711.txt @@ -12,7 +12,7 @@ Required properties: Example: -codec: wm8711@1a { +wm8711: codec@1a { compatible = "wlf,wm8711"; reg = <0x1a>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8728.txt b/Documentation/devicetree/bindings/sound/wm8728.txt index a8b5c3668e60..a3608b4c78b9 100644 --- a/Documentation/devicetree/bindings/sound/wm8728.txt +++ b/Documentation/devicetree/bindings/sound/wm8728.txt @@ -12,7 +12,7 @@ Required properties: Example: -codec: wm8728@1a { +wm8728: codec@1a { compatible = "wlf,wm8728"; reg = <0x1a>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8731.txt b/Documentation/devicetree/bindings/sound/wm8731.txt index 236690e99b87..f660d9bb0e69 100644 --- a/Documentation/devicetree/bindings/sound/wm8731.txt +++ b/Documentation/devicetree/bindings/sound/wm8731.txt @@ -12,7 +12,7 @@ Required properties: Example: -codec: wm8731@1a { +wm8731: codec@1a { compatible = "wlf,wm8731"; reg = <0x1a>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8737.txt b/Documentation/devicetree/bindings/sound/wm8737.txt index 4bc2cea3b140..eda1ec6a7563 100644 --- a/Documentation/devicetree/bindings/sound/wm8737.txt +++ b/Documentation/devicetree/bindings/sound/wm8737.txt @@ -12,7 +12,7 @@ Required properties: Example: -codec: wm8737@1a { +wm8737: codec@1a { compatible = "wlf,wm8737"; reg = <0x1a>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8741.txt b/Documentation/devicetree/bindings/sound/wm8741.txt index a13315408719..b69e196c741c 100644 --- a/Documentation/devicetree/bindings/sound/wm8741.txt +++ b/Documentation/devicetree/bindings/sound/wm8741.txt @@ -21,7 +21,7 @@ Optional properties: Example: -codec: wm8741@1a { +wm8741: codec@1a { compatible = "wlf,wm8741"; reg = <0x1a>; diff --git a/Documentation/devicetree/bindings/sound/wm8750.txt b/Documentation/devicetree/bindings/sound/wm8750.txt index 8db239fd5ecd..682f221f6f38 100644 --- a/Documentation/devicetree/bindings/sound/wm8750.txt +++ b/Documentation/devicetree/bindings/sound/wm8750.txt @@ -12,7 +12,7 @@ Required properties: Example: -codec: wm8750@1a { +wm8750: codec@1a { compatible = "wlf,wm8750"; reg = <0x1a>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8753.txt b/Documentation/devicetree/bindings/sound/wm8753.txt index 8eee61282105..eca9e5a825a9 100644 --- a/Documentation/devicetree/bindings/sound/wm8753.txt +++ b/Documentation/devicetree/bindings/sound/wm8753.txt @@ -34,7 +34,7 @@ Pins on the device (for linking into audio routes): Example: -codec: wm8753@1a { +wm8753: codec@1a { compatible = "wlf,wm8753"; reg = <0x1a>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8770.txt b/Documentation/devicetree/bindings/sound/wm8770.txt index 866e00ca150b..cac762a1105d 100644 --- a/Documentation/devicetree/bindings/sound/wm8770.txt +++ b/Documentation/devicetree/bindings/sound/wm8770.txt @@ -10,7 +10,7 @@ Required properties: Example: -codec: wm8770@1 { +wm8770: codec@1 { compatible = "wlf,wm8770"; reg = <1>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8776.txt b/Documentation/devicetree/bindings/sound/wm8776.txt index 3b9ca49abc2b..01173369c3ed 100644 --- a/Documentation/devicetree/bindings/sound/wm8776.txt +++ b/Documentation/devicetree/bindings/sound/wm8776.txt @@ -12,7 +12,7 @@ Required properties: Example: -codec: wm8776@1a { +wm8776: codec@1a { compatible = "wlf,wm8776"; reg = <0x1a>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8804.txt b/Documentation/devicetree/bindings/sound/wm8804.txt index 6fd124b16496..2c1641c17a91 100644 --- a/Documentation/devicetree/bindings/sound/wm8804.txt +++ b/Documentation/devicetree/bindings/sound/wm8804.txt @@ -19,7 +19,7 @@ Optional properties: Example: -codec: wm8804@1a { +wm8804: codec@1a { compatible = "wlf,wm8804"; reg = <0x1a>; }; diff --git a/Documentation/devicetree/bindings/sound/wm8903.txt b/Documentation/devicetree/bindings/sound/wm8903.txt index afc51caf1137..6371c2434afe 100644 --- a/Documentation/devicetree/bindings/sound/wm8903.txt +++ b/Documentation/devicetree/bindings/sound/wm8903.txt @@ -57,7 +57,7 @@ Pins on the device (for linking into audio routes): Example: -codec: wm8903@1a { +wm8903: codec@1a { compatible = "wlf,wm8903"; reg = <0x1a>; interrupts = < 347 >; diff --git a/Documentation/devicetree/bindings/sound/wm8960.txt b/Documentation/devicetree/bindings/sound/wm8960.txt index 2deb8a3da9c5..6d29ac3750ee 100644 --- a/Documentation/devicetree/bindings/sound/wm8960.txt +++ b/Documentation/devicetree/bindings/sound/wm8960.txt @@ -23,7 +23,7 @@ Optional properties: Example: -codec: wm8960@1a { +wm8960: codec@1a { compatible = "wlf,wm8960"; reg = <0x1a>; diff --git a/Documentation/devicetree/bindings/sound/wm8962.txt b/Documentation/devicetree/bindings/sound/wm8962.txt index 7f82b59ec8f9..dcfa9a3369fd 100644 --- a/Documentation/devicetree/bindings/sound/wm8962.txt +++ b/Documentation/devicetree/bindings/sound/wm8962.txt @@ -24,7 +24,7 @@ Optional properties: Example: -codec: wm8962@1a { +wm8962: codec@1a { compatible = "wlf,wm8962"; reg = <0x1a>; diff --git a/Documentation/devicetree/bindings/sound/wm8994.txt b/Documentation/devicetree/bindings/sound/wm8994.txt index 68c4e8d96bed..4a9dead1b7d3 100644 --- a/Documentation/devicetree/bindings/sound/wm8994.txt +++ b/Documentation/devicetree/bindings/sound/wm8994.txt @@ -59,7 +59,7 @@ Optional properties: Example: -codec: wm8994@1a { +wm8994: codec@1a { compatible = "wlf,wm8994"; reg = <0x1a>; diff --git a/Documentation/devicetree/bindings/spi/spi-st-ssc.txt b/Documentation/devicetree/bindings/spi/spi-st-ssc.txt index fe54959ec957..1bdc4709e474 100644 --- a/Documentation/devicetree/bindings/spi/spi-st-ssc.txt +++ b/Documentation/devicetree/bindings/spi/spi-st-ssc.txt @@ -9,7 +9,7 @@ Required properties: - clocks : Must contain an entry for each name in clock-names See ../clk/* - pinctrl-names : Uses "default", can use "sleep" if provided - See ../pinctrl/pinctrl-binding.txt + See ../pinctrl/pinctrl-bindings.txt Optional properties: - cs-gpios : List of GPIO chip selects diff --git a/Documentation/devicetree/bindings/submitting-patches.txt b/Documentation/devicetree/bindings/submitting-patches.txt index 274058c583dd..de0d6090c0fd 100644 --- a/Documentation/devicetree/bindings/submitting-patches.txt +++ b/Documentation/devicetree/bindings/submitting-patches.txt @@ -6,7 +6,14 @@ I. For patch submitters 0) Normal patch submission rules from Documentation/process/submitting-patches.rst applies. - 1) The Documentation/ portion of the patch should be a separate patch. + 1) The Documentation/ and include/dt-bindings/ portion of the patch should + be a separate patch. The preferred subject prefix for binding patches is: + + "dt-bindings: : ..." + + The 80 characters of the subject are precious. It is recommended to not + use "Documentation" or "doc" because that is implied. All bindings are + docs. Repeating "binding" again should also be avoided. 2) Submit the entire series to the devicetree mailinglist at diff --git a/Documentation/devicetree/bindings/thermal/exynos-thermal.txt b/Documentation/devicetree/bindings/thermal/exynos-thermal.txt index b957acff57aa..ad648d93d961 100644 --- a/Documentation/devicetree/bindings/thermal/exynos-thermal.txt +++ b/Documentation/devicetree/bindings/thermal/exynos-thermal.txt @@ -12,7 +12,6 @@ "samsung,exynos5420-tmu-ext-triminfo" for TMU channels 2, 3 and 4 Exynos5420 (Must pass triminfo base and triminfo clock) "samsung,exynos5433-tmu" - "samsung,exynos5440-tmu" "samsung,exynos7-tmu" - interrupt-parent : The phandle for the interrupt controller - reg : Address range of the thermal registers. For soc's which has multiple @@ -68,18 +67,7 @@ Example 1): #thermal-sensor-cells = <0>; }; -Example 2): - - tmuctrl_0: tmuctrl@160118 { - compatible = "samsung,exynos5440-tmu"; - reg = <0x160118 0x230>, <0x160368 0x10>; - interrupts = <0 58 0>; - clocks = <&clock 21>; - clock-names = "tmu_apbif"; - #thermal-sensor-cells = <0>; - }; - -Example 3): (In case of Exynos5420 "with misplaced TRIMINFO register") +Example 2): (In case of Exynos5420 "with misplaced TRIMINFO register") tmu_cpu2: tmu@10068000 { compatible = "samsung,exynos5420-tmu-ext-triminfo"; reg = <0x10068000 0x100>, <0x1006c000 0x4>; diff --git a/Documentation/devicetree/bindings/thermal/imx-thermal.txt b/Documentation/devicetree/bindings/thermal/imx-thermal.txt index 379eb763073e..823e4176eef8 100644 --- a/Documentation/devicetree/bindings/thermal/imx-thermal.txt +++ b/Documentation/devicetree/bindings/thermal/imx-thermal.txt @@ -1,8 +1,13 @@ * Temperature Monitor (TEMPMON) on Freescale i.MX SoCs Required properties: -- compatible : "fsl,imx6q-tempmon" for i.MX6Q, "fsl,imx6sx-tempmon" for i.MX6SX. - i.MX6SX has two more IRQs than i.MX6Q, one is IRQ_LOW and the other is IRQ_PANIC, +- compatible : must be one of following: + - "fsl,imx6q-tempmon" for i.MX6Q, + - "fsl,imx6sx-tempmon" for i.MX6SX, + - "fsl,imx7d-tempmon" for i.MX7S/D. +- interrupts : the interrupt output of the controller: + i.MX6Q has one IRQ which will be triggered when temperature is higher than high threshold, + i.MX6SX and i.MX7S/D have two more IRQs than i.MX6Q, one is IRQ_LOW and the other is IRQ_PANIC, when temperature is below than low threshold, IRQ_LOW will be triggered, when temperature is higher than panic threshold, system will auto reboot by SRC module. - fsl,tempmon : phandle pointer to system controller that contains TEMPMON diff --git a/Documentation/devicetree/bindings/thermal/mediatek-thermal.txt b/Documentation/devicetree/bindings/thermal/mediatek-thermal.txt index 0d73ea5e9c0c..41d6a443ad66 100644 --- a/Documentation/devicetree/bindings/thermal/mediatek-thermal.txt +++ b/Documentation/devicetree/bindings/thermal/mediatek-thermal.txt @@ -12,6 +12,7 @@ Required properties: - "mediatek,mt8173-thermal" : For MT8173 family of SoCs - "mediatek,mt2701-thermal" : For MT2701 family of SoCs - "mediatek,mt2712-thermal" : For MT2712 family of SoCs + - "mediatek,mt7622-thermal" : For MT7622 SoC - reg: Address range of the thermal controller - interrupts: IRQ for the thermal controller - clocks, clock-names: Clocks needed for the thermal controller. required diff --git a/Documentation/devicetree/bindings/thermal/qcom-tsens.txt b/Documentation/devicetree/bindings/thermal/qcom-tsens.txt index 292ed89d900b..06195e8f35e2 100644 --- a/Documentation/devicetree/bindings/thermal/qcom-tsens.txt +++ b/Documentation/devicetree/bindings/thermal/qcom-tsens.txt @@ -8,6 +8,7 @@ Required properties: - reg: Address range of the thermal registers - #thermal-sensor-cells : Should be 1. See ./thermal.txt for a description. +- #qcom,sensors: Number of sensors in tsens block - Refer to Documentation/devicetree/bindings/nvmem/nvmem.txt to know how to specify nvmem cells diff --git a/Documentation/devicetree/bindings/thermal/rcar-gen3-thermal.txt b/Documentation/devicetree/bindings/thermal/rcar-gen3-thermal.txt index fdf5caa6229b..cfa154bb0fa7 100644 --- a/Documentation/devicetree/bindings/thermal/rcar-gen3-thermal.txt +++ b/Documentation/devicetree/bindings/thermal/rcar-gen3-thermal.txt @@ -9,6 +9,7 @@ Required properties: Examples with soctypes are: - "renesas,r8a7795-thermal" (R-Car H3) - "renesas,r8a7796-thermal" (R-Car M3-W) + - "renesas,r8a77965-thermal" (R-Car M3-N) - reg : Address ranges of the thermal registers. Each sensor needs one address range. Sorting must be done in increasing order according to datasheet, i.e. @@ -18,7 +19,7 @@ Required properties: Optional properties: -- interrupts : interrupts routed to the TSC (3 for H3 and M3-W) +- interrupts : interrupts routed to the TSC (3 for H3, M3-W and M3-N) - power-domain : Must contain a reference to the power domain. This property is mandatory if the thermal sensor instance is part of a controllable power domain. @@ -27,9 +28,9 @@ Example: tsc: thermal@e6198000 { compatible = "renesas,r8a7795-thermal"; - reg = <0 0xe6198000 0 0x68>, - <0 0xe61a0000 0 0x5c>, - <0 0xe61a8000 0 0x5c>; + reg = <0 0xe6198000 0 0x100>, + <0 0xe61a0000 0 0x100>, + <0 0xe61a8000 0 0x100>; interrupts = , , ; diff --git a/Documentation/devicetree/bindings/thermal/rcar-thermal.txt b/Documentation/devicetree/bindings/thermal/rcar-thermal.txt index 349e635f2d87..67c563f1b4c4 100644 --- a/Documentation/devicetree/bindings/thermal/rcar-thermal.txt +++ b/Documentation/devicetree/bindings/thermal/rcar-thermal.txt @@ -3,7 +3,8 @@ Required properties: - compatible : "renesas,thermal-", "renesas,rcar-gen2-thermal" (with thermal-zone) or - "renesas,rcar-thermal" (without thermal-zone) as fallback. + "renesas,rcar-thermal" (without thermal-zone) as + fallback except R-Car D3. Examples with soctypes are: - "renesas,thermal-r8a73a4" (R-Mobile APE6) - "renesas,thermal-r8a7743" (RZ/G1M) @@ -12,13 +13,15 @@ Required properties: - "renesas,thermal-r8a7791" (R-Car M2-W) - "renesas,thermal-r8a7792" (R-Car V2H) - "renesas,thermal-r8a7793" (R-Car M2-N) + - "renesas,thermal-r8a77995" (R-Car D3) - reg : Address range of the thermal registers. The 1st reg will be recognized as common register if it has "interrupts". Option properties: -- interrupts : use interrupt +- interrupts : If present should contain 3 interrupts for + R-Car D3 or 1 interrupt otherwise. Example (non interrupt support): diff --git a/Documentation/devicetree/bindings/thermal/uniphier-thermal.txt b/Documentation/devicetree/bindings/thermal/uniphier-thermal.txt index 686c0b42ed3f..ceb92a95727a 100644 --- a/Documentation/devicetree/bindings/thermal/uniphier-thermal.txt +++ b/Documentation/devicetree/bindings/thermal/uniphier-thermal.txt @@ -8,6 +8,7 @@ Required properties: - compatible : - "socionext,uniphier-pxs2-thermal" : For UniPhier PXs2 SoC - "socionext,uniphier-ld20-thermal" : For UniPhier LD20 SoC + - "socionext,uniphier-pxs3-thermal" : For UniPhier PXs3 SoC - interrupts : IRQ for the temperature alarm - #thermal-sensor-cells : Should be 0. See ./thermal.txt for details. diff --git a/Documentation/devicetree/bindings/nios2/timer.txt b/Documentation/devicetree/bindings/timer/altr,timer-1.0.txt similarity index 100% rename from Documentation/devicetree/bindings/nios2/timer.txt rename to Documentation/devicetree/bindings/timer/altr,timer-1.0.txt diff --git a/Documentation/devicetree/bindings/arm/arch_timer.txt b/Documentation/devicetree/bindings/timer/arm,arch_timer.txt similarity index 100% rename from Documentation/devicetree/bindings/arm/arch_timer.txt rename to Documentation/devicetree/bindings/timer/arm,arch_timer.txt diff --git a/Documentation/devicetree/bindings/arm/armv7m_systick.txt b/Documentation/devicetree/bindings/timer/arm,armv7m-systick.txt similarity index 100% rename from Documentation/devicetree/bindings/arm/armv7m_systick.txt rename to Documentation/devicetree/bindings/timer/arm,armv7m-systick.txt diff --git a/Documentation/devicetree/bindings/arm/global_timer.txt b/Documentation/devicetree/bindings/timer/arm,global_timer.txt similarity index 100% rename from Documentation/devicetree/bindings/arm/global_timer.txt rename to Documentation/devicetree/bindings/timer/arm,global_timer.txt diff --git a/Documentation/devicetree/bindings/arm/twd.txt b/Documentation/devicetree/bindings/timer/arm,twd.txt similarity index 100% rename from Documentation/devicetree/bindings/arm/twd.txt rename to Documentation/devicetree/bindings/timer/arm,twd.txt diff --git a/Documentation/devicetree/bindings/powerpc/fsl/gtm.txt b/Documentation/devicetree/bindings/timer/fsl,gtm.txt similarity index 100% rename from Documentation/devicetree/bindings/powerpc/fsl/gtm.txt rename to Documentation/devicetree/bindings/timer/fsl,gtm.txt diff --git a/Documentation/devicetree/bindings/arm/mrvl/timer.txt b/Documentation/devicetree/bindings/timer/mrvl,mmp-timer.txt similarity index 100% rename from Documentation/devicetree/bindings/arm/mrvl/timer.txt rename to Documentation/devicetree/bindings/timer/mrvl,mmp-timer.txt diff --git a/Documentation/devicetree/bindings/arm/msm/timer.txt b/Documentation/devicetree/bindings/timer/qcom,msm-timer.txt similarity index 100% rename from Documentation/devicetree/bindings/arm/msm/timer.txt rename to Documentation/devicetree/bindings/timer/qcom,msm-timer.txt diff --git a/Documentation/devicetree/bindings/timer/renesas,cmt.txt b/Documentation/devicetree/bindings/timer/renesas,cmt.txt index d740989eb569..b40add2d9bb4 100644 --- a/Documentation/devicetree/bindings/timer/renesas,cmt.txt +++ b/Documentation/devicetree/bindings/timer/renesas,cmt.txt @@ -22,6 +22,10 @@ Required Properties: - "renesas,r8a73a4-cmt0" for the 32-bit CMT0 device included in r8a73a4. - "renesas,r8a73a4-cmt1" for the 48-bit CMT1 device included in r8a73a4. + - "renesas,r8a7743-cmt0" for the 32-bit CMT0 device included in r8a7743. + - "renesas,r8a7743-cmt1" for the 48-bit CMT1 device included in r8a7743. + - "renesas,r8a7745-cmt0" for the 32-bit CMT0 device included in r8a7745. + - "renesas,r8a7745-cmt1" for the 48-bit CMT1 device included in r8a7745. - "renesas,r8a7790-cmt0" for the 32-bit CMT0 device included in r8a7790. - "renesas,r8a7790-cmt1" for the 48-bit CMT1 device included in r8a7790. - "renesas,r8a7791-cmt0" for the 32-bit CMT0 device included in r8a7791. @@ -31,10 +35,12 @@ Required Properties: - "renesas,r8a7794-cmt0" for the 32-bit CMT0 device included in r8a7794. - "renesas,r8a7794-cmt1" for the 48-bit CMT1 device included in r8a7794. - - "renesas,rcar-gen2-cmt0" for 32-bit CMT0 devices included in R-Car Gen2. - - "renesas,rcar-gen2-cmt1" for 48-bit CMT1 devices included in R-Car Gen2. - These are fallbacks for r8a73a4 and all the R-Car Gen2 - entries listed above. + - "renesas,rcar-gen2-cmt0" for 32-bit CMT0 devices included in R-Car Gen2 + and RZ/G1. + - "renesas,rcar-gen2-cmt1" for 48-bit CMT1 devices included in R-Car Gen2 + and RZ/G1. + These are fallbacks for r8a73a4, R-Car Gen2 and RZ/G1 entries + listed above. - reg: base address and length of the registers block for the timer module. - interrupts: interrupt-specifier for the timer, one per channel. diff --git a/Documentation/devicetree/bindings/arm/spear-timer.txt b/Documentation/devicetree/bindings/timer/st,spear-timer.txt similarity index 100% rename from Documentation/devicetree/bindings/arm/spear-timer.txt rename to Documentation/devicetree/bindings/timer/st,spear-timer.txt diff --git a/Documentation/devicetree/bindings/c6x/timer64.txt b/Documentation/devicetree/bindings/timer/ti,c64x+timer64.txt similarity index 100% rename from Documentation/devicetree/bindings/c6x/timer64.txt rename to Documentation/devicetree/bindings/timer/ti,c64x+timer64.txt diff --git a/Documentation/devicetree/bindings/arm/omap/timer.txt b/Documentation/devicetree/bindings/timer/ti,timer.txt similarity index 100% rename from Documentation/devicetree/bindings/arm/omap/timer.txt rename to Documentation/devicetree/bindings/timer/ti,timer.txt diff --git a/Documentation/devicetree/bindings/arm/vt8500/via,vt8500-timer.txt b/Documentation/devicetree/bindings/timer/via,vt8500-timer.txt similarity index 100% rename from Documentation/devicetree/bindings/arm/vt8500/via,vt8500-timer.txt rename to Documentation/devicetree/bindings/timer/via,vt8500-timer.txt diff --git a/Documentation/devicetree/bindings/usb/ci-hdrc-usb2.txt b/Documentation/devicetree/bindings/usb/ci-hdrc-usb2.txt index 0e03344e2e8b..2e9318151df7 100644 --- a/Documentation/devicetree/bindings/usb/ci-hdrc-usb2.txt +++ b/Documentation/devicetree/bindings/usb/ci-hdrc-usb2.txt @@ -76,6 +76,10 @@ Optional properties: needs to make sure it does not send more than 90% maximum_periodic_data_per_frame. The use case is multiple transactions, but less frame rate. +- mux-controls: The mux control for toggling host/device output of this + controller. It's expected that a mux state of 0 indicates device mode and a + mux state of 1 indicates host mode. +- mux-control-names: Shall be "usb_switch" if mux-controls is specified. i.mx specific properties - fsl,usbmisc: phandler of non-core register device, with one @@ -102,4 +106,6 @@ Example: rx-burst-size-dword = <0x10>; extcon = <0>, <&usb_id>; phy-clkgate-delay-us = <400>; + mux-controls = <&usb_switch>; + mux-control-names = "usb_switch"; }; diff --git a/Documentation/devicetree/bindings/usb/dwc3.txt b/Documentation/devicetree/bindings/usb/dwc3.txt index 0dbd3083e7dd..7f13ebef06cb 100644 --- a/Documentation/devicetree/bindings/usb/dwc3.txt +++ b/Documentation/devicetree/bindings/usb/dwc3.txt @@ -7,6 +7,26 @@ Required properties: - compatible: must be "snps,dwc3" - reg : Address and length of the register set for the device - interrupts: Interrupts used by the dwc3 controller. + - clock-names: should contain "ref", "bus_early", "suspend" + - clocks: list of phandle and clock specifier pairs corresponding to + entries in the clock-names property. + +Exception for clocks: + clocks are optional if the parent node (i.e. glue-layer) is compatible to + one of the following: + "amlogic,meson-axg-dwc3" + "amlogic,meson-gxl-dwc3" + "cavium,octeon-7130-usb-uctl" + "qcom,dwc3" + "samsung,exynos5250-dwusb3" + "samsung,exynos7-dwusb3" + "sprd,sc9860-dwc3" + "st,stih407-dwc3" + "ti,am437x-dwc3" + "ti,dwc3" + "ti,keystone-dwc3" + "rockchip,rk3399-dwc3" + "xlnx,zynqmp-dwc3" Optional properties: - usb-phy : array of phandle for the PHY device. The first element @@ -15,6 +35,7 @@ Optional properties: - phys: from the *Generic PHY* bindings - phy-names: from the *Generic PHY* bindings; supported names are "usb2-phy" or "usb3-phy". + - resets: a single pair of phandle and reset specifier - snps,usb3_lpm_capable: determines if platform is USB3 LPM capable - snps,disable_scramble_quirk: true when SW should disable data scrambling. Only really useful for FPGA builds. diff --git a/Documentation/devicetree/bindings/usb/fcs,fusb302.txt b/Documentation/devicetree/bindings/usb/fcs,fusb302.txt index 472facfa5a71..6087dc7f209e 100644 --- a/Documentation/devicetree/bindings/usb/fcs,fusb302.txt +++ b/Documentation/devicetree/bindings/usb/fcs,fusb302.txt @@ -6,12 +6,6 @@ Required properties : - interrupts : Interrupt specifier Optional properties : -- fcs,max-sink-microvolt : Maximum voltage to negotiate when configured as sink -- fcs,max-sink-microamp : Maximum current to negotiate when configured as sink -- fcs,max-sink-microwatt : Maximum power to negotiate when configured as sink - If this is less then max-sink-microvolt * - max-sink-microamp then the configured current will - be clamped. - fcs,operating-sink-microwatt : Minimum amount of power accepted from a sink when negotiating diff --git a/Documentation/devicetree/bindings/usb/hisilicon,histb-xhci.txt b/Documentation/devicetree/bindings/usb/hisilicon,histb-xhci.txt new file mode 100644 index 000000000000..f4633496b122 --- /dev/null +++ b/Documentation/devicetree/bindings/usb/hisilicon,histb-xhci.txt @@ -0,0 +1,45 @@ +HiSilicon STB xHCI + +The device node for HiSilicon STB xHCI host controller + +Required properties: + - compatible: should be "hisilicon,hi3798cv200-xhci" + - reg: specifies physical base address and size of the registers + - interrupts : interrupt used by the controller + - clocks: a list of phandle + clock-specifier pairs, one for each + entry in clock-names + - clock-names: must contain + "bus": for bus clock + "utmi": for utmi clock + "pipe": for pipe clock + "suspend": for suspend clock + - resets: a list of phandle and reset specifier pairs as listed in + reset-names property. + - reset-names: must contain + "soft": for soft reset + - phys: a list of phandle + phy specifier pairs + - phy-names: must contain at least one of following: + "inno": for inno phy + "combo": for combo phy + +Optional properties: + - usb2-lpm-disable: indicate if we don't want to enable USB2 HW LPM + - usb3-lpm-capable: determines if platform is USB3 LPM capable + - imod-interval-ns: default interrupt moderation interval is 40000ns + +Example: + +xhci0: xchi@f98a0000 { + compatible = "hisilicon,hi3798cv200-xhci"; + reg = <0xf98a0000 0x10000>; + interrupts = ; + clocks = <&crg HISTB_USB3_BUS_CLK>, + <&crg HISTB_USB3_UTMI_CLK>, + <&crg HISTB_USB3_PIPE_CLK>, + <&crg HISTB_USB3_SUSPEND_CLK>; + clock-names = "bus", "utmi", "pipe", "suspend"; + resets = <&crg 0xb0 12>; + reset-names = "soft"; + phys = <&usb2_phy1_port1 0>, <&combphy0 PHY_TYPE_USB3>; + phy-names = "inno", "combo"; +}; diff --git a/Documentation/devicetree/bindings/usb/qcom,dwc3.txt b/Documentation/devicetree/bindings/usb/qcom,dwc3.txt index bc8a2fa5d2bf..95afdcf3c337 100644 --- a/Documentation/devicetree/bindings/usb/qcom,dwc3.txt +++ b/Documentation/devicetree/bindings/usb/qcom,dwc3.txt @@ -1,54 +1,95 @@ Qualcomm SuperSpeed DWC3 USB SoC controller Required properties: -- compatible: should contain "qcom,dwc3" +- compatible: Compatible list, contains + "qcom,dwc3" + "qcom,msm8996-dwc3" for msm8996 SOC. + "qcom,sdm845-dwc3" for sdm845 SOC. +- reg: Offset and length of register set for QSCRATCH wrapper +- power-domains: specifies a phandle to PM domain provider node - clocks: A list of phandle + clock-specifier pairs for the clocks listed in clock-names -- clock-names: Should contain the following: +- clock-names: Should contain the following: "core" Master/Core clock, have to be >= 125 MHz for SS operation and >= 60MHz for HS operation + "mock_utmi" Mock utmi clock needed for ITP/SOF generation in + host mode. Its frequency should be 19.2MHz. + "sleep" Sleep clock, used for wakeup when USB3 core goes + into low power mode (U3). Optional clocks: - "iface" System bus AXI clock. Not present on all platforms - "sleep" Sleep clock, used when USB3 core goes into low - power mode (U3). + "iface" System bus AXI clock. + Not present on "qcom,msm8996-dwc3" compatible. + "cfg_noc" System Config NOC clock. + Not present on "qcom,msm8996-dwc3" compatible. +- assigned-clocks: Should be: + MOCK_UTMI_CLK + MASTER_CLK +- assigned-clock-rates: Should be: + 19.2Mhz (192000000) for MOCK_UTMI_CLK + >=125Mhz (125000000) for MASTER_CLK in SS mode + >=60Mhz (60000000) for MASTER_CLK in HS mode + +Optional properties: +- resets: Phandle to reset control that resets core and wrapper. +- interrupts: specifies interrupts from controller wrapper used + to wakeup from low power/susepnd state. Must contain + one or more entry for interrupt-names property +- interrupt-names: Must include the following entries: + - "hs_phy_irq": The interrupt that is asserted when a + wakeup event is received on USB2 bus + - "ss_phy_irq": The interrupt that is asserted when a + wakeup event is received on USB3 bus + - "dm_hs_phy_irq" and "dp_hs_phy_irq": Separate + interrupts for any wakeup event on DM and DP lines +- qcom,select-utmi-as-pipe-clk: if present, disable USB3 pipe_clk requirement. + Used when dwc3 operates without SSPHY and only + HS/FS/LS modes are supported. Required child node: A child node must exist to represent the core DWC3 IP block. The name of the node is not important. The content of the node is defined in dwc3.txt. Phy documentation is provided in the following places: -Documentation/devicetree/bindings/phy/qcom-dwc3-usb-phy.txt +Documentation/devicetree/bindings/phy/qcom-qmp-phy.txt - USB3 QMP PHY +Documentation/devicetree/bindings/phy/qcom-qusb2-phy.txt - USB2 QUSB2 PHY Example device nodes: hs_phy: phy@100f8800 { - compatible = "qcom,dwc3-hs-usb-phy"; - reg = <0x100f8800 0x30>; - clocks = <&gcc USB30_0_UTMI_CLK>; - clock-names = "ref"; - #phy-cells = <0>; - + compatible = "qcom,qusb2-v2-phy"; + ... }; ss_phy: phy@100f8830 { - compatible = "qcom,dwc3-ss-usb-phy"; - reg = <0x100f8830 0x30>; - clocks = <&gcc USB30_0_MASTER_CLK>; - clock-names = "ref"; - #phy-cells = <0>; - + compatible = "qcom,qmp-v3-usb3-phy"; + ... }; - usb3_0: usb30@0 { + usb3_0: usb30@a6f8800 { compatible = "qcom,dwc3"; + reg = <0xa6f8800 0x400>; #address-cells = <1>; #size-cells = <1>; - clocks = <&gcc USB30_0_MASTER_CLK>; - clock-names = "core"; - ranges; + interrupts = <0 131 0>, <0 486 0>, <0 488 0>, <0 489 0>; + interrupt-names = "hs_phy_irq", "ss_phy_irq", + "dm_hs_phy_irq", "dp_hs_phy_irq"; + + clocks = <&gcc GCC_USB30_PRIM_MASTER_CLK>, + <&gcc GCC_USB30_PRIM_MOCK_UTMI_CLK>, + <&gcc GCC_USB30_PRIM_SLEEP_CLK>; + clock-names = "core", "mock_utmi", "sleep"; + + assigned-clocks = <&gcc GCC_USB30_PRIM_MOCK_UTMI_CLK>, + <&gcc GCC_USB30_PRIM_MASTER_CLK>; + assigned-clock-rates = <19200000>, <133000000>; + + resets = <&gcc GCC_USB30_PRIM_BCR>; + reset-names = "core_reset"; + power-domains = <&gcc USB30_PRIM_GDSC>; + qcom,select-utmi-as-pipe-clk; dwc3@10000000 { compatible = "snps,dwc3"; diff --git a/Documentation/devicetree/bindings/usb/richtek,rt1711h.txt b/Documentation/devicetree/bindings/usb/richtek,rt1711h.txt new file mode 100644 index 000000000000..09e847e92e5e --- /dev/null +++ b/Documentation/devicetree/bindings/usb/richtek,rt1711h.txt @@ -0,0 +1,17 @@ +Richtek RT1711H TypeC PD Controller. + +Required properties: + - compatible : Must be "richtek,rt1711h". + - reg : Must be 0x4e, it's slave address of RT1711H. + - interrupt-parent : the phandle for the interrupt controller that + provides interrupts for this device. + - interrupts : where a is the interrupt number and b represents an + encoding of the sense and level information for the interrupt. + +Example : +rt1711h@4e { + compatible = "richtek,rt1711h"; + reg = <0x4e>; + interrupt-parent = <&gpio26>; + interrupts = <0 IRQ_TYPE_LEVEL_LOW>; +}; diff --git a/Documentation/devicetree/bindings/usb/rockchip,dwc3.txt b/Documentation/devicetree/bindings/usb/rockchip,dwc3.txt index 50a31536e975..252a05c5d976 100644 --- a/Documentation/devicetree/bindings/usb/rockchip,dwc3.txt +++ b/Documentation/devicetree/bindings/usb/rockchip,dwc3.txt @@ -16,7 +16,7 @@ A child node must exist to represent the core DWC3 IP block. The name of the node is not important. The content of the node is defined in dwc3.txt. Phy documentation is provided in the following places: -Documentation/devicetree/bindings/phy/rockchip,dwc3-usb-phy.txt +Documentation/devicetree/bindings/phy/qcom-dwc3-usb-phy.txt Example device nodes: diff --git a/Documentation/devicetree/bindings/vendor-prefixes.txt b/Documentation/devicetree/bindings/vendor-prefixes.txt index a38d8bfae19c..7cad066191ee 100644 --- a/Documentation/devicetree/bindings/vendor-prefixes.txt +++ b/Documentation/devicetree/bindings/vendor-prefixes.txt @@ -32,6 +32,7 @@ andestech Andes Technology Corporation apm Applied Micro Circuits Corporation (APM) aptina Aptina Imaging arasan Arasan Chip Systems +archermind ArcherMind Technology (Nanjing) Co., Ltd. arctic Arctic Sand aries Aries Embedded GmbH arm ARM Ltd. @@ -47,6 +48,7 @@ auvidea Auvidea GmbH avago Avago Technologies avia avia semiconductor avic Shanghai AVIC Optoelectronics Co., Ltd. +avnet Avnet, Inc. axentia Axentia Technologies AB axis Axis Communications AB bananapi BIPAI KEJI LIMITED @@ -56,6 +58,7 @@ bosch Bosch Sensortec GmbH boundary Boundary Devices Inc. brcm Broadcom Corporation buffalo Buffalo, Inc. +bticino Bticino International calxeda Calxeda capella Capella Microsystems, Inc cascoda Cascoda, Ltd. @@ -75,6 +78,7 @@ cnxt Conexant Systems, Inc. compulab CompuLab Ltd. cortina Cortina Systems, Inc. cosmic Cosmic Circuits +crane Crane Connectivity Solutions creative Creative Technology Ltd crystalfontz Crystalfontz America, Inc. cubietech Cubietech, Ltd. @@ -185,6 +189,7 @@ khadas Khadas kiebackpeter Kieback & Peter GmbH kinetic Kinetic Technologies kingnovel Kingnovel Technology Co., Ltd. +koe Kaohsiung Opto-Electronics Inc. kosagi Sutajio Ko-Usagi PTE Ltd. kyo Kyocera Corporation lacie LaCie @@ -199,11 +204,13 @@ linaro Linaro Limited linksys Belkin International, Inc. (Linksys) linux Linux-specific binding lltc Linear Technology Corporation +logicpd Logic PD, Inc. lsi LSI Corp. (LSI Logic) lwn Liebherr-Werk Nenzing GmbH macnica Macnica Americas marvell Marvell Technology Group Ltd. maxim Maxim Integrated Products +mbvl Mobiveil Inc. mcube mCube meas Measurement Specialties mediatek MediaTek Inc. @@ -279,6 +286,7 @@ pine64 Pine64 pixcir PIXCIR MICROELECTRONICS Co., Ltd plathome Plat'Home Co., Ltd. plda PLDA +portwell Portwell Inc. poslab Poslab Technology Co., Ltd. powervr PowerVR (deprecated, use img) probox2 PROBOX2 (by W2COMP Co., Ltd.) @@ -318,6 +326,7 @@ sgx SGX Sensortech sharp Sharp Corporation shimafuji Shimafuji Electric, Inc. si-en Si-En Technology Ltd. +sifive SiFive, Inc. sigma Sigma Designs, Inc. sii Seiko Instruments, Inc. sil Silicon Image @@ -393,6 +402,7 @@ vot Vision Optical Technology Co., Ltd. wd Western Digital Corp. wetek WeTek Electronics, limited. wexler Wexler +wi2wi Wi2Wi, Inc. winbond Winbond Electronics corp. winstar Winstar Display Corp. wlf Wolfson Microelectronics diff --git a/Documentation/devicetree/bindings/watchdog/ingenic,jz4740-wdt.txt b/Documentation/devicetree/bindings/watchdog/ingenic,jz4740-wdt.txt index cb44918f01a8..ce1cb72d5345 100644 --- a/Documentation/devicetree/bindings/watchdog/ingenic,jz4740-wdt.txt +++ b/Documentation/devicetree/bindings/watchdog/ingenic,jz4740-wdt.txt @@ -3,10 +3,15 @@ Ingenic Watchdog Timer (WDT) Controller for JZ4740 & JZ4780 Required properties: compatible: "ingenic,jz4740-watchdog" or "ingenic,jz4780-watchdog" reg: Register address and length for watchdog registers +clocks: phandle to the RTC clock +clock-names: should be "rtc" Example: watchdog: jz4740-watchdog@10002000 { compatible = "ingenic,jz4740-watchdog"; - reg = <0x10002000 0x100>; + reg = <0x10002000 0x10>; + + clocks = <&cgu JZ4740_CLK_RTC>; + clock-names = "rtc"; }; diff --git a/Documentation/devicetree/bindings/watchdog/renesas-wdt.txt b/Documentation/devicetree/bindings/watchdog/renesas-wdt.txt index 74b2f03c1515..f24d802b8056 100644 --- a/Documentation/devicetree/bindings/watchdog/renesas-wdt.txt +++ b/Documentation/devicetree/bindings/watchdog/renesas-wdt.txt @@ -1,18 +1,27 @@ Renesas Watchdog Timer (WDT) Controller Required properties: -- compatible : Should be "renesas,-wdt", and - "renesas,rcar-gen3-wdt" or "renesas,rza-wdt" as fallback. + - compatible : Must be "renesas,-wdt", followed by a generic + fallback compatible string when compatible with the generic + version. Examples with soctypes are: - - "renesas,r7s72100-wdt" (RZ/A1) + - "renesas,r8a7743-wdt" (RZ/G1M) + - "renesas,r8a7745-wdt" (RZ/G1E) + - "renesas,r8a7790-wdt" (R-Car H2) + - "renesas,r8a7791-wdt" (R-Car M2-W) + - "renesas,r8a7792-wdt" (R-Car V2H) + - "renesas,r8a7793-wdt" (R-Car M2-N) + - "renesas,r8a7794-wdt" (R-Car E2) - "renesas,r8a7795-wdt" (R-Car H3) - "renesas,r8a7796-wdt" (R-Car M3-W) + - "renesas,r8a77965-wdt" (R-Car M3-N) - "renesas,r8a77970-wdt" (R-Car V3M) - "renesas,r8a77995-wdt" (R-Car D3) - - When compatible with the generic version, nodes must list the SoC-specific - version corresponding to the platform first, followed by the generic - version. + - "renesas,r7s72100-wdt" (RZ/A1) + The generic compatible string must be: + - "renesas,rza-wdt" for RZ/A + - "renesas,rcar-gen2-wdt" for R-Car Gen2 and RZ/G + - "renesas,rcar-gen3-wdt" for R-Car Gen3 - reg : Should contain WDT registers location and length - clocks : the clock feeding the watchdog timer. diff --git a/Documentation/driver-api/clk.rst b/Documentation/driver-api/clk.rst new file mode 100644 index 000000000000..593cca5058b1 --- /dev/null +++ b/Documentation/driver-api/clk.rst @@ -0,0 +1,307 @@ +======================== +The Common Clk Framework +======================== + +:Author: Mike Turquette + +This document endeavours to explain the common clk framework details, +and how to port a platform over to this framework. It is not yet a +detailed explanation of the clock api in include/linux/clk.h, but +perhaps someday it will include that information. + +Introduction and interface split +================================ + +The common clk framework is an interface to control the clock nodes +available on various devices today. This may come in the form of clock +gating, rate adjustment, muxing or other operations. This framework is +enabled with the CONFIG_COMMON_CLK option. + +The interface itself is divided into two halves, each shielded from the +details of its counterpart. First is the common definition of struct +clk which unifies the framework-level accounting and infrastructure that +has traditionally been duplicated across a variety of platforms. Second +is a common implementation of the clk.h api, defined in +drivers/clk/clk.c. Finally there is struct clk_ops, whose operations +are invoked by the clk api implementation. + +The second half of the interface is comprised of the hardware-specific +callbacks registered with struct clk_ops and the corresponding +hardware-specific structures needed to model a particular clock. For +the remainder of this document any reference to a callback in struct +clk_ops, such as .enable or .set_rate, implies the hardware-specific +implementation of that code. Likewise, references to struct clk_foo +serve as a convenient shorthand for the implementation of the +hardware-specific bits for the hypothetical "foo" hardware. + +Tying the two halves of this interface together is struct clk_hw, which +is defined in struct clk_foo and pointed to within struct clk_core. This +allows for easy navigation between the two discrete halves of the common +clock interface. + +Common data structures and api +============================== + +Below is the common struct clk_core definition from +drivers/clk/clk.c, modified for brevity:: + + struct clk_core { + const char *name; + const struct clk_ops *ops; + struct clk_hw *hw; + struct module *owner; + struct clk_core *parent; + const char **parent_names; + struct clk_core **parents; + u8 num_parents; + u8 new_parent_index; + ... + }; + +The members above make up the core of the clk tree topology. The clk +api itself defines several driver-facing functions which operate on +struct clk. That api is documented in include/linux/clk.h. + +Platforms and devices utilizing the common struct clk_core use the struct +clk_ops pointer in struct clk_core to perform the hardware-specific parts of +the operations defined in clk-provider.h:: + + struct clk_ops { + int (*prepare)(struct clk_hw *hw); + void (*unprepare)(struct clk_hw *hw); + int (*is_prepared)(struct clk_hw *hw); + void (*unprepare_unused)(struct clk_hw *hw); + int (*enable)(struct clk_hw *hw); + void (*disable)(struct clk_hw *hw); + int (*is_enabled)(struct clk_hw *hw); + void (*disable_unused)(struct clk_hw *hw); + unsigned long (*recalc_rate)(struct clk_hw *hw, + unsigned long parent_rate); + long (*round_rate)(struct clk_hw *hw, + unsigned long rate, + unsigned long *parent_rate); + int (*determine_rate)(struct clk_hw *hw, + struct clk_rate_request *req); + int (*set_parent)(struct clk_hw *hw, u8 index); + u8 (*get_parent)(struct clk_hw *hw); + int (*set_rate)(struct clk_hw *hw, + unsigned long rate, + unsigned long parent_rate); + int (*set_rate_and_parent)(struct clk_hw *hw, + unsigned long rate, + unsigned long parent_rate, + u8 index); + unsigned long (*recalc_accuracy)(struct clk_hw *hw, + unsigned long parent_accuracy); + int (*get_phase)(struct clk_hw *hw); + int (*set_phase)(struct clk_hw *hw, int degrees); + void (*init)(struct clk_hw *hw); + void (*debug_init)(struct clk_hw *hw, + struct dentry *dentry); + }; + +Hardware clk implementations +============================ + +The strength of the common struct clk_core comes from its .ops and .hw pointers +which abstract the details of struct clk from the hardware-specific bits, and +vice versa. To illustrate consider the simple gateable clk implementation in +drivers/clk/clk-gate.c:: + + struct clk_gate { + struct clk_hw hw; + void __iomem *reg; + u8 bit_idx; + ... + }; + +struct clk_gate contains struct clk_hw hw as well as hardware-specific +knowledge about which register and bit controls this clk's gating. +Nothing about clock topology or accounting, such as enable_count or +notifier_count, is needed here. That is all handled by the common +framework code and struct clk_core. + +Let's walk through enabling this clk from driver code:: + + struct clk *clk; + clk = clk_get(NULL, "my_gateable_clk"); + + clk_prepare(clk); + clk_enable(clk); + +The call graph for clk_enable is very simple:: + + clk_enable(clk); + clk->ops->enable(clk->hw); + [resolves to...] + clk_gate_enable(hw); + [resolves struct clk gate with to_clk_gate(hw)] + clk_gate_set_bit(gate); + +And the definition of clk_gate_set_bit:: + + static void clk_gate_set_bit(struct clk_gate *gate) + { + u32 reg; + + reg = __raw_readl(gate->reg); + reg |= BIT(gate->bit_idx); + writel(reg, gate->reg); + } + +Note that to_clk_gate is defined as:: + + #define to_clk_gate(_hw) container_of(_hw, struct clk_gate, hw) + +This pattern of abstraction is used for every clock hardware +representation. + +Supporting your own clk hardware +================================ + +When implementing support for a new type of clock it is only necessary to +include the following header:: + + #include + +To construct a clk hardware structure for your platform you must define +the following:: + + struct clk_foo { + struct clk_hw hw; + ... hardware specific data goes here ... + }; + +To take advantage of your data you'll need to support valid operations +for your clk:: + + struct clk_ops clk_foo_ops { + .enable = &clk_foo_enable; + .disable = &clk_foo_disable; + }; + +Implement the above functions using container_of:: + + #define to_clk_foo(_hw) container_of(_hw, struct clk_foo, hw) + + int clk_foo_enable(struct clk_hw *hw) + { + struct clk_foo *foo; + + foo = to_clk_foo(hw); + + ... perform magic on foo ... + + return 0; + }; + +Below is a matrix detailing which clk_ops are mandatory based upon the +hardware capabilities of that clock. A cell marked as "y" means +mandatory, a cell marked as "n" implies that either including that +callback is invalid or otherwise unnecessary. Empty cells are either +optional or must be evaluated on a case-by-case basis. + +.. table:: clock hardware characteristics + + +----------------+------+-------------+---------------+-------------+------+ + | | gate | change rate | single parent | multiplexer | root | + +================+======+=============+===============+=============+======+ + |.prepare | | | | | | + +----------------+------+-------------+---------------+-------------+------+ + |.unprepare | | | | | | + +----------------+------+-------------+---------------+-------------+------+ + +----------------+------+-------------+---------------+-------------+------+ + |.enable | y | | | | | + +----------------+------+-------------+---------------+-------------+------+ + |.disable | y | | | | | + +----------------+------+-------------+---------------+-------------+------+ + |.is_enabled | y | | | | | + +----------------+------+-------------+---------------+-------------+------+ + +----------------+------+-------------+---------------+-------------+------+ + |.recalc_rate | | y | | | | + +----------------+------+-------------+---------------+-------------+------+ + |.round_rate | | y [1]_ | | | | + +----------------+------+-------------+---------------+-------------+------+ + |.determine_rate | | y [1]_ | | | | + +----------------+------+-------------+---------------+-------------+------+ + |.set_rate | | y | | | | + +----------------+------+-------------+---------------+-------------+------+ + +----------------+------+-------------+---------------+-------------+------+ + |.set_parent | | | n | y | n | + +----------------+------+-------------+---------------+-------------+------+ + |.get_parent | | | n | y | n | + +----------------+------+-------------+---------------+-------------+------+ + +----------------+------+-------------+---------------+-------------+------+ + |.recalc_accuracy| | | | | | + +----------------+------+-------------+---------------+-------------+------+ + +----------------+------+-------------+---------------+-------------+------+ + |.init | | | | | | + +----------------+------+-------------+---------------+-------------+------+ + +.. [1] either one of round_rate or determine_rate is required. + +Finally, register your clock at run-time with a hardware-specific +registration function. This function simply populates struct clk_foo's +data and then passes the common struct clk parameters to the framework +with a call to:: + + clk_register(...) + +See the basic clock types in ``drivers/clk/clk-*.c`` for examples. + +Disabling clock gating of unused clocks +======================================= + +Sometimes during development it can be useful to be able to bypass the +default disabling of unused clocks. For example, if drivers aren't enabling +clocks properly but rely on them being on from the bootloader, bypassing +the disabling means that the driver will remain functional while the issues +are sorted out. + +To bypass this disabling, include "clk_ignore_unused" in the bootargs to the +kernel. + +Locking +======= + +The common clock framework uses two global locks, the prepare lock and the +enable lock. + +The enable lock is a spinlock and is held across calls to the .enable, +.disable operations. Those operations are thus not allowed to sleep, +and calls to the clk_enable(), clk_disable() API functions are allowed in +atomic context. + +For clk_is_enabled() API, it is also designed to be allowed to be used in +atomic context. However, it doesn't really make any sense to hold the enable +lock in core, unless you want to do something else with the information of +the enable state with that lock held. Otherwise, seeing if a clk is enabled is +a one-shot read of the enabled state, which could just as easily change after +the function returns because the lock is released. Thus the user of this API +needs to handle synchronizing the read of the state with whatever they're +using it for to make sure that the enable state doesn't change during that +time. + +The prepare lock is a mutex and is held across calls to all other operations. +All those operations are allowed to sleep, and calls to the corresponding API +functions are not allowed in atomic context. + +This effectively divides operations in two groups from a locking perspective. + +Drivers don't need to manually protect resources shared between the operations +of one group, regardless of whether those resources are shared by multiple +clocks or not. However, access to resources that are shared between operations +of the two groups needs to be protected by the drivers. An example of such a +resource would be a register that controls both the clock rate and the clock +enable/disable state. + +The clock framework is reentrant, in that a driver is allowed to call clock +framework functions from within its implementation of clock operations. This +can for instance cause a .set_rate operation of one clock being called from +within the .set_rate operation of another clock. This case must be considered +in the driver implementations, but the code flow is usually controlled by the +driver in that case. + +Note that locking must also be considered when code outside of the common +clock framework needs to access resources used by the clock operations. This +is considered out of scope of this document. diff --git a/Documentation/driver-api/device_connection.rst b/Documentation/driver-api/device_connection.rst index affbc5566ab0..ba364224c349 100644 --- a/Documentation/driver-api/device_connection.rst +++ b/Documentation/driver-api/device_connection.rst @@ -40,4 +40,4 @@ API --- .. kernel-doc:: drivers/base/devcon.c - : functions: device_connection_find_match device_connection_find device_connection_add device_connection_remove + :functions: device_connection_find_match device_connection_find device_connection_add device_connection_remove diff --git a/Documentation/driver-api/firmware/fallback-mechanisms.rst b/Documentation/driver-api/firmware/fallback-mechanisms.rst index f353783ae0be..d35fed65eae9 100644 --- a/Documentation/driver-api/firmware/fallback-mechanisms.rst +++ b/Documentation/driver-api/firmware/fallback-mechanisms.rst @@ -72,9 +72,12 @@ the firmware requested, and establishes it in the device hierarchy by associating the device used to make the request as the device's parent. The sysfs directory's file attributes are defined and controlled through the new device's class (firmware_class) and group (fw_dev_attr_groups). -This is actually where the original firmware_class.c file name comes from, -as originally the only firmware loading mechanism available was the -mechanism we now use as a fallback mechanism. +This is actually where the original firmware_class module name came from, +given that originally the only firmware loading mechanism available was the +mechanism we now use as a fallback mechanism, which registers a struct class +firmware_class. Because the attributes exposed are part of the module name, the +module name firmware_class cannot be renamed in the future, to ensure backward +compatibility with old userspace. To load firmware using the sysfs interface we expose a loading indicator, and a file upload firmware into: @@ -83,7 +86,7 @@ and a file upload firmware into: * /sys/$DEVPATH/data To upload firmware you will echo 1 onto the loading file to indicate -you are loading firmware. You then cat the firmware into the data file, +you are loading firmware. You then write the firmware into the data file, and you notify the kernel the firmware is ready by echo'ing 0 onto the loading file. @@ -136,7 +139,8 @@ by kobject uevents. This is specially exacerbated due to the fact that most distributions today disable CONFIG_FW_LOADER_USER_HELPER_FALLBACK. Refer to do_firmware_uevent() for details of the kobject event variables -setup. Variables passwdd with a kobject add event: +setup. The variables currently passed to userspace with a "kobject add" +event are: * FIRMWARE=firmware name * TIMEOUT=timeout value diff --git a/Documentation/driver-api/firmware/firmware_cache.rst b/Documentation/driver-api/firmware/firmware_cache.rst index 2210e5bfb332..c2e69d9c6bf1 100644 --- a/Documentation/driver-api/firmware/firmware_cache.rst +++ b/Documentation/driver-api/firmware/firmware_cache.rst @@ -29,8 +29,8 @@ Some implementation details about the firmware cache setup: * If an asynchronous call is used the firmware cache is only set up for a device if if the second argument (uevent) to request_firmware_nowait() is true. When uevent is true it requests that a kobject uevent be sent to - userspace for the firmware request. For details refer to the Fackback - mechanism documented below. + userspace for the firmware request through the sysfs fallback mechanism + if the firmware file is not found. * If the firmware cache is determined to be needed as per the above two criteria the firmware cache is setup by adding a devres entry for the diff --git a/Documentation/driver-api/firmware/request_firmware.rst b/Documentation/driver-api/firmware/request_firmware.rst index d5ec95a7195b..f62bdcbfed5b 100644 --- a/Documentation/driver-api/firmware/request_firmware.rst +++ b/Documentation/driver-api/firmware/request_firmware.rst @@ -20,6 +20,11 @@ request_firmware .. kernel-doc:: drivers/base/firmware_loader/main.c :functions: request_firmware +firmware_request_nowarn +----------------------- +.. kernel-doc:: drivers/base/firmware_loader/main.c + :functions: firmware_request_nowarn + request_firmware_direct ----------------------- .. kernel-doc:: drivers/base/firmware_loader/main.c diff --git a/Documentation/driver-api/fpga/fpga-bridge.rst b/Documentation/driver-api/fpga/fpga-bridge.rst new file mode 100644 index 000000000000..2c2aaca894bf --- /dev/null +++ b/Documentation/driver-api/fpga/fpga-bridge.rst @@ -0,0 +1,49 @@ +FPGA Bridge +=========== + +API to implement a new FPGA bridge +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. kernel-doc:: include/linux/fpga/fpga-bridge.h + :functions: fpga_bridge + +.. kernel-doc:: include/linux/fpga/fpga-bridge.h + :functions: fpga_bridge_ops + +.. kernel-doc:: drivers/fpga/fpga-bridge.c + :functions: fpga_bridge_create + +.. kernel-doc:: drivers/fpga/fpga-bridge.c + :functions: fpga_bridge_free + +.. kernel-doc:: drivers/fpga/fpga-bridge.c + :functions: fpga_bridge_register + +.. kernel-doc:: drivers/fpga/fpga-bridge.c + :functions: fpga_bridge_unregister + +API to control an FPGA bridge +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You probably won't need these directly. FPGA regions should handle this. + +.. kernel-doc:: drivers/fpga/fpga-bridge.c + :functions: of_fpga_bridge_get + +.. kernel-doc:: drivers/fpga/fpga-bridge.c + :functions: fpga_bridge_get + +.. kernel-doc:: drivers/fpga/fpga-bridge.c + :functions: fpga_bridge_put + +.. kernel-doc:: drivers/fpga/fpga-bridge.c + :functions: fpga_bridge_get_to_list + +.. kernel-doc:: drivers/fpga/fpga-bridge.c + :functions: of_fpga_bridge_get_to_list + +.. kernel-doc:: drivers/fpga/fpga-bridge.c + :functions: fpga_bridge_enable + +.. kernel-doc:: drivers/fpga/fpga-bridge.c + :functions: fpga_bridge_disable diff --git a/Documentation/driver-api/fpga/fpga-mgr.rst b/Documentation/driver-api/fpga/fpga-mgr.rst new file mode 100644 index 000000000000..bcf2dd24e179 --- /dev/null +++ b/Documentation/driver-api/fpga/fpga-mgr.rst @@ -0,0 +1,220 @@ +FPGA Manager +============ + +Overview +-------- + +The FPGA manager core exports a set of functions for programming an FPGA with +an image. The API is manufacturer agnostic. All manufacturer specifics are +hidden away in a low level driver which registers a set of ops with the core. +The FPGA image data itself is very manufacturer specific, but for our purposes +it's just binary data. The FPGA manager core won't parse it. + +The FPGA image to be programmed can be in a scatter gather list, a single +contiguous buffer, or a firmware file. Because allocating contiguous kernel +memory for the buffer should be avoided, users are encouraged to use a scatter +gather list instead if possible. + +The particulars for programming the image are presented in a structure (struct +fpga_image_info). This struct contains parameters such as pointers to the +FPGA image as well as image-specific particulars such as whether the image was +built for full or partial reconfiguration. + +How to support a new FPGA device +-------------------------------- + +To add another FPGA manager, write a driver that implements a set of ops. The +probe function calls fpga_mgr_register(), such as:: + + static const struct fpga_manager_ops socfpga_fpga_ops = { + .write_init = socfpga_fpga_ops_configure_init, + .write = socfpga_fpga_ops_configure_write, + .write_complete = socfpga_fpga_ops_configure_complete, + .state = socfpga_fpga_ops_state, + }; + + static int socfpga_fpga_probe(struct platform_device *pdev) + { + struct device *dev = &pdev->dev; + struct socfpga_fpga_priv *priv; + struct fpga_manager *mgr; + int ret; + + priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL); + if (!priv) + return -ENOMEM; + + /* + * do ioremaps, get interrupts, etc. and save + * them in priv + */ + + mgr = fpga_mgr_create(dev, "Altera SOCFPGA FPGA Manager", + &socfpga_fpga_ops, priv); + if (!mgr) + return -ENOMEM; + + platform_set_drvdata(pdev, mgr); + + ret = fpga_mgr_register(mgr); + if (ret) + fpga_mgr_free(mgr); + + return ret; + } + + static int socfpga_fpga_remove(struct platform_device *pdev) + { + struct fpga_manager *mgr = platform_get_drvdata(pdev); + + fpga_mgr_unregister(mgr); + + return 0; + } + + +The ops will implement whatever device specific register writes are needed to +do the programming sequence for this particular FPGA. These ops return 0 for +success or negative error codes otherwise. + +The programming sequence is:: + 1. .write_init + 2. .write or .write_sg (may be called once or multiple times) + 3. .write_complete + +The .write_init function will prepare the FPGA to receive the image data. The +buffer passed into .write_init will be atmost .initial_header_size bytes long, +if the whole bitstream is not immediately available then the core code will +buffer up at least this much before starting. + +The .write function writes a buffer to the FPGA. The buffer may be contain the +whole FPGA image or may be a smaller chunk of an FPGA image. In the latter +case, this function is called multiple times for successive chunks. This interface +is suitable for drivers which use PIO. + +The .write_sg version behaves the same as .write except the input is a sg_table +scatter list. This interface is suitable for drivers which use DMA. + +The .write_complete function is called after all the image has been written +to put the FPGA into operating mode. + +The ops include a .state function which will read the hardware FPGA manager and +return a code of type enum fpga_mgr_states. It doesn't result in a change in +hardware state. + +How to write an image buffer to a supported FPGA +------------------------------------------------ + +Some sample code:: + + #include + + struct fpga_manager *mgr; + struct fpga_image_info *info; + int ret; + + /* + * Get a reference to FPGA manager. The manager is not locked, so you can + * hold onto this reference without it preventing programming. + * + * This example uses the device node of the manager. Alternatively, use + * fpga_mgr_get(dev) instead if you have the device. + */ + mgr = of_fpga_mgr_get(mgr_node); + + /* struct with information about the FPGA image to program. */ + info = fpga_image_info_alloc(dev); + + /* flags indicates whether to do full or partial reconfiguration */ + info->flags = FPGA_MGR_PARTIAL_RECONFIG; + + /* + * At this point, indicate where the image is. This is pseudo-code; you're + * going to use one of these three. + */ + if (image is in a scatter gather table) { + + info->sgt = [your scatter gather table] + + } else if (image is in a buffer) { + + info->buf = [your image buffer] + info->count = [image buffer size] + + } else if (image is in a firmware file) { + + info->firmware_name = devm_kstrdup(dev, firmware_name, GFP_KERNEL); + + } + + /* Get exclusive control of FPGA manager */ + ret = fpga_mgr_lock(mgr); + + /* Load the buffer to the FPGA */ + ret = fpga_mgr_buf_load(mgr, &info, buf, count); + + /* Release the FPGA manager */ + fpga_mgr_unlock(mgr); + fpga_mgr_put(mgr); + + /* Deallocate the image info if you're done with it */ + fpga_image_info_free(info); + +API for implementing a new FPGA Manager driver +---------------------------------------------- + +.. kernel-doc:: include/linux/fpga/fpga-mgr.h + :functions: fpga_manager + +.. kernel-doc:: include/linux/fpga/fpga-mgr.h + :functions: fpga_manager_ops + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: fpga_mgr_create + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: fpga_mgr_free + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: fpga_mgr_register + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: fpga_mgr_unregister + +API for programming a FPGA +-------------------------- + +.. kernel-doc:: include/linux/fpga/fpga-mgr.h + :functions: fpga_image_info + +.. kernel-doc:: include/linux/fpga/fpga-mgr.h + :functions: fpga_mgr_states + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: fpga_image_info_alloc + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: fpga_image_info_free + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: of_fpga_mgr_get + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: fpga_mgr_get + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: fpga_mgr_put + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: fpga_mgr_lock + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: fpga_mgr_unlock + +.. kernel-doc:: include/linux/fpga/fpga-mgr.h + :functions: fpga_mgr_states + +Note - use :c:func:`fpga_region_program_fpga()` instead of :c:func:`fpga_mgr_load()` + +.. kernel-doc:: drivers/fpga/fpga-mgr.c + :functions: fpga_mgr_load diff --git a/Documentation/driver-api/fpga/fpga-region.rst b/Documentation/driver-api/fpga/fpga-region.rst new file mode 100644 index 000000000000..f89e4a311722 --- /dev/null +++ b/Documentation/driver-api/fpga/fpga-region.rst @@ -0,0 +1,102 @@ +FPGA Region +=========== + +Overview +-------- + +This document is meant to be an brief overview of the FPGA region API usage. A +more conceptual look at regions can be found in the Device Tree binding +document [#f1]_. + +For the purposes of this API document, let's just say that a region associates +an FPGA Manager and a bridge (or bridges) with a reprogrammable region of an +FPGA or the whole FPGA. The API provides a way to register a region and to +program a region. + +Currently the only layer above fpga-region.c in the kernel is the Device Tree +support (of-fpga-region.c) described in [#f1]_. The DT support layer uses regions +to program the FPGA and then DT to handle enumeration. The common region code +is intended to be used by other schemes that have other ways of accomplishing +enumeration after programming. + +An fpga-region can be set up to know the following things: + + * which FPGA manager to use to do the programming + + * which bridges to disable before programming and enable afterwards. + +Additional info needed to program the FPGA image is passed in the struct +fpga_image_info including: + + * pointers to the image as either a scatter-gather buffer, a contiguous + buffer, or the name of firmware file + + * flags indicating specifics such as whether the image if for partial + reconfiguration. + +How to program a FPGA using a region +------------------------------------ + +First, allocate the info struct:: + + info = fpga_image_info_alloc(dev); + if (!info) + return -ENOMEM; + +Set flags as needed, i.e.:: + + info->flags |= FPGA_MGR_PARTIAL_RECONFIG; + +Point to your FPGA image, such as:: + + info->sgt = &sgt; + +Add info to region and do the programming:: + + region->info = info; + ret = fpga_region_program_fpga(region); + +:c:func:`fpga_region_program_fpga()` operates on info passed in the +fpga_image_info (region->info). This function will attempt to: + + * lock the region's mutex + * lock the region's FPGA manager + * build a list of FPGA bridges if a method has been specified to do so + * disable the bridges + * program the FPGA + * re-enable the bridges + * release the locks + +Then you will want to enumerate whatever hardware has appeared in the FPGA. + +How to add a new FPGA region +---------------------------- + +An example of usage can be seen in the probe function of [#f2]_. + +.. [#f1] ../devicetree/bindings/fpga/fpga-region.txt +.. [#f2] ../../drivers/fpga/of-fpga-region.c + +API to program a FGPA +--------------------- + +.. kernel-doc:: drivers/fpga/fpga-region.c + :functions: fpga_region_program_fpga + +API to add a new FPGA region +---------------------------- + +.. kernel-doc:: include/linux/fpga/fpga-region.h + :functions: fpga_region + +.. kernel-doc:: drivers/fpga/fpga-region.c + :functions: fpga_region_create + +.. kernel-doc:: drivers/fpga/fpga-region.c + :functions: fpga_region_free + +.. kernel-doc:: drivers/fpga/fpga-region.c + :functions: fpga_region_register + +.. kernel-doc:: drivers/fpga/fpga-region.c + :functions: fpga_region_unregister diff --git a/Documentation/driver-api/fpga/index.rst b/Documentation/driver-api/fpga/index.rst new file mode 100644 index 000000000000..c51e5ebd544a --- /dev/null +++ b/Documentation/driver-api/fpga/index.rst @@ -0,0 +1,13 @@ +============== +FPGA Subsystem +============== + +:Author: Alan Tull + +.. toctree:: + :maxdepth: 2 + + intro + fpga-mgr + fpga-bridge + fpga-region diff --git a/Documentation/driver-api/fpga/intro.rst b/Documentation/driver-api/fpga/intro.rst new file mode 100644 index 000000000000..51cd81dbb4dc --- /dev/null +++ b/Documentation/driver-api/fpga/intro.rst @@ -0,0 +1,54 @@ +Introduction +============ + +The FPGA subsystem supports reprogramming FPGAs dynamically under +Linux. Some of the core intentions of the FPGA subsystems are: + +* The FPGA subsystem is vendor agnostic. + +* The FPGA subsystem separates upper layers (userspace interfaces and + enumeration) from lower layers that know how to program a specific + FPGA. + +* Code should not be shared between upper and lower layers. This + should go without saying. If that seems necessary, there's probably + framework functionality that that can be added that will benefit + other users. Write the linux-fpga mailing list and maintainers and + seek out a solution that expands the framework for broad reuse. + +* Generally, when adding code, think of the future. Plan for re-use. + +The framework in the kernel is divided into: + +FPGA Manager +------------ + +If you are adding a new FPGA or a new method of programming a FPGA, +this is the subsystem for you. Low level FPGA manager drivers contain +the knowledge of how to program a specific device. This subsystem +includes the framework in fpga-mgr.c and the low level drivers that +are registered with it. + +FPGA Bridge +----------- + +FPGA Bridges prevent spurious signals from going out of a FPGA or a +region of a FPGA during programming. They are disabled before +programming begins and re-enabled afterwards. An FPGA bridge may be +actual hard hardware that gates a bus to a cpu or a soft ("freeze") +bridge in FPGA fabric that surrounds a partial reconfiguration region +of an FPGA. This subsystem includes fpga-bridge.c and the low level +drivers that are registered with it. + +FPGA Region +----------- + +If you are adding a new interface to the FPGA framework, add it on top +of a FPGA region to allow the most reuse of your interface. + +The FPGA Region framework (fpga-region.c) associates managers and +bridges as reconfigurable regions. A region may refer to the whole +FPGA in full reconfiguration or to a partial reconfiguration region. + +The Device Tree FPGA Region support (of-fpga-region.c) handles +reprogramming FPGAs when device tree overlays are applied. diff --git a/Documentation/driver-api/gpio/board.rst b/Documentation/driver-api/gpio/board.rst index 25d62b2e9fd0..2c112553df84 100644 --- a/Documentation/driver-api/gpio/board.rst +++ b/Documentation/driver-api/gpio/board.rst @@ -177,3 +177,19 @@ mapping and is thus transparent to GPIO consumers. A set of functions such as gpiod_set_value() is available to work with the new descriptor-oriented interface. + +Boards using platform data can also hog GPIO lines by defining GPIO hog tables. + +.. code-block:: c + + struct gpiod_hog gpio_hog_table[] = { + GPIO_HOG("gpio.0", 10, "foo", GPIO_ACTIVE_LOW, GPIOD_OUT_HIGH), + { } + }; + +And the table can be added to the board code as follows:: + + gpiod_add_hogs(gpio_hog_table); + +The line will be hogged as soon as the gpiochip is created or - in case the +chip was created earlier - when the hog table is registered. diff --git a/Documentation/driver-api/gpio/consumer.rst b/Documentation/driver-api/gpio/consumer.rst index c71a50d85b50..aa03f389d41d 100644 --- a/Documentation/driver-api/gpio/consumer.rst +++ b/Documentation/driver-api/gpio/consumer.rst @@ -57,7 +57,7 @@ device that displays digits), an additional index argument can be specified:: enum gpiod_flags flags) For a more detailed description of the con_id parameter in the DeviceTree case -see Documentation/gpio/board.txt +see Documentation/driver-api/gpio/board.rst The flags parameter is used to optionally specify a direction and initial value for the GPIO. Values can be: diff --git a/Documentation/driver-api/gpio/driver.rst b/Documentation/driver-api/gpio/driver.rst index 505ee906d7d9..cbe0242842d1 100644 --- a/Documentation/driver-api/gpio/driver.rst +++ b/Documentation/driver-api/gpio/driver.rst @@ -44,7 +44,7 @@ common to each controller of that type: - methods to establish GPIO line direction - methods used to access GPIO line values - - method to set electrical configuration to a a given GPIO line + - method to set electrical configuration for a given GPIO line - method to return the IRQ number associated to a given GPIO line - flag saying whether calls to its methods may sleep - optional line names array to identify lines @@ -143,7 +143,7 @@ resistor will make the line tend to high level unless one of the transistors on the rail actively pulls it down. The level on the line will go as high as the VDD on the pull-up resistor, which -may be higher than the level supported by the transistor, achieveing a +may be higher than the level supported by the transistor, achieving a level-shift to the higher VDD. Integrated electronics often have an output driver stage in the form of a CMOS @@ -382,7 +382,7 @@ Real-Time compliance for GPIO IRQ chips Any provider of irqchips needs to be carefully tailored to support Real Time preemption. It is desirable that all irqchips in the GPIO subsystem keep this -in mind and does the proper testing to assure they are real time-enabled. +in mind and do the proper testing to assure they are real time-enabled. So, pay attention on above " RT_FULL:" notes, please. The following is a checklist to follow when preparing a driver for real time-compliance: diff --git a/Documentation/driver-api/gpio/drivers-on-gpio.rst b/Documentation/driver-api/gpio/drivers-on-gpio.rst index 7da0c1dd1f7a..f3a189320e11 100644 --- a/Documentation/driver-api/gpio/drivers-on-gpio.rst +++ b/Documentation/driver-api/gpio/drivers-on-gpio.rst @@ -85,6 +85,10 @@ hardware descriptions such as device tree or ACPI: any other serio bus to the system and makes it possible to connect drivers for e.g. keyboards and other PS/2 protocol based devices. +- cec-gpio: drivers/media/platform/cec-gpio/ is used to interact with a CEC + Consumer Electronics Control bus using only GPIO. It is used to communicate + with devices on the HDMI bus. + Apart from this there are special GPIO drivers in subsystems like MMC/SD to read card detect and write protect GPIO lines, and in the TTY serial subsystem to emulate MCTRL (modem control) signals CTS/RTS by using two GPIO lines. The diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst index 6d8352c0f354..6d9f2f9fe20e 100644 --- a/Documentation/driver-api/index.rst +++ b/Documentation/driver-api/index.rst @@ -17,7 +17,9 @@ available subsections can be seen below. basics infrastructure pm/index + clk device-io + device_connection dma-buf device_link message-based @@ -34,6 +36,7 @@ available subsections can be seen below. edac scsi libata + target mtdnand miscellaneous w1 @@ -49,6 +52,7 @@ available subsections can be seen below. dmaengine/index slimbus soundwire/index + fpga/index .. only:: subproject and html diff --git a/Documentation/driver-api/infrastructure.rst b/Documentation/driver-api/infrastructure.rst index bee1b9a1702f..6172f3cc3d0b 100644 --- a/Documentation/driver-api/infrastructure.rst +++ b/Documentation/driver-api/infrastructure.rst @@ -49,10 +49,10 @@ Device Drivers Base Device Drivers DMA Management ----------------------------- -.. kernel-doc:: drivers/base/dma-coherent.c +.. kernel-doc:: kernel/dma/coherent.c :export: -.. kernel-doc:: drivers/base/dma-mapping.c +.. kernel-doc:: kernel/dma/mapping.c :export: Device drivers PnP support diff --git a/Documentation/driver-api/scsi.rst b/Documentation/driver-api/scsi.rst index 31ad0fed6763..64b231d125e0 100644 --- a/Documentation/driver-api/scsi.rst +++ b/Documentation/driver-api/scsi.rst @@ -334,5 +334,5 @@ todo ~~~~ Parallel (fast/wide/ultra) SCSI, USB, SATA, SAS, Fibre Channel, -FireWire, ATAPI devices, Infiniband, I2O, iSCSI, Parallel ports, +FireWire, ATAPI devices, Infiniband, I2O, Parallel ports, netlink... diff --git a/Documentation/driver-api/soundwire/error_handling.rst b/Documentation/driver-api/soundwire/error_handling.rst new file mode 100644 index 000000000000..aa3a0a23a066 --- /dev/null +++ b/Documentation/driver-api/soundwire/error_handling.rst @@ -0,0 +1,65 @@ +======================== +SoundWire Error Handling +======================== + +The SoundWire PHY was designed with care and errors on the bus are going to +be very unlikely, and if they happen it should be limited to single bit +errors. Examples of this design can be found in the synchronization +mechanism (sync loss after two errors) and short CRCs used for the Bulk +Register Access. + +The errors can be detected with multiple mechanisms: + +1. Bus clash or parity errors: This mechanism relies on low-level detectors + that are independent of the payload and usages, and they cover both control + and audio data. The current implementation only logs such errors. + Improvements could be invalidating an entire programming sequence and + restarting from a known position. In the case of such errors outside of a + control/command sequence, there is no concealment or recovery for audio + data enabled by the SoundWire protocol, the location of the error will also + impact its audibility (most-significant bits will be more impacted in PCM), + and after a number of such errors are detected the bus might be reset. Note + that bus clashes due to programming errors (two streams using the same bit + slots) or electrical issues during the transmit/receive transition cannot + be distinguished, although a recurring bus clash when audio is enabled is a + indication of a bus allocation issue. The interrupt mechanism can also help + identify Slaves which detected a Bus Clash or a Parity Error, but they may + not be responsible for the errors so resetting them individually is not a + viable recovery strategy. + +2. Command status: Each command is associated with a status, which only + covers transmission of the data between devices. The ACK status indicates + that the command was received and will be executed by the end of the + current frame. A NAK indicates that the command was in error and will not + be applied. In case of a bad programming (command sent to non-existent + Slave or to a non-implemented register) or electrical issue, no response + signals the command was ignored. Some Master implementations allow for a + command to be retransmitted several times. If the retransmission fails, + backtracking and restarting the entire programming sequence might be a + solution. Alternatively some implementations might directly issue a bus + reset and re-enumerate all devices. + +3. Timeouts: In a number of cases such as ChannelPrepare or + ClockStopPrepare, the bus driver is supposed to poll a register field until + it transitions to a NotFinished value of zero. The MIPI SoundWire spec 1.1 + does not define timeouts but the MIPI SoundWire DisCo document adds + recommendation on timeouts. If such configurations do not complete, the + driver will return a -ETIMEOUT. Such timeouts are symptoms of a faulty + Slave device and are likely impossible to recover from. + +Errors during global reconfiguration sequences are extremely difficult to +handle: + +1. BankSwitch: An error during the last command issuing a BankSwitch is + difficult to backtrack from. Retransmitting the Bank Switch command may be + possible in a single segment setup, but this can lead to synchronization + problems when enabling multiple bus segments (a command with side effects + such as frame reconfiguration would be handled at different times). A global + hard-reset might be the best solution. + +Note that SoundWire does not provide a mechanism to detect illegal values +written in valid registers. In a number of cases the standard even mentions +that the Slave might behave in implementation-defined ways. The bus +implementation does not provide a recovery mechanism for such errors, Slave +or Master driver implementers are responsible for writing valid values in +valid registers and implement additional range checking if needed. diff --git a/Documentation/driver-api/soundwire/index.rst b/Documentation/driver-api/soundwire/index.rst index 647e94654752..6db026028f27 100644 --- a/Documentation/driver-api/soundwire/index.rst +++ b/Documentation/driver-api/soundwire/index.rst @@ -6,6 +6,9 @@ SoundWire Documentation :maxdepth: 1 summary + stream + error_handling + locking .. only:: subproject diff --git a/Documentation/driver-api/soundwire/locking.rst b/Documentation/driver-api/soundwire/locking.rst new file mode 100644 index 000000000000..253f73555255 --- /dev/null +++ b/Documentation/driver-api/soundwire/locking.rst @@ -0,0 +1,106 @@ +================= +SoundWire Locking +================= + +This document explains locking mechanism of the SoundWire Bus. Bus uses +following locks in order to avoid race conditions in Bus operations on +shared resources. + + - Bus lock + + - Message lock + +Bus lock +======== + +SoundWire Bus lock is a mutex and is part of Bus data structure +(sdw_bus) which is used for every Bus instance. This lock is used to +serialize each of the following operations(s) within SoundWire Bus instance. + + - Addition and removal of Slave(s), changing Slave status. + + - Prepare, Enable, Disable and De-prepare stream operations. + + - Access of Stream data structure. + +Message lock +============ + +SoundWire message transfer lock. This mutex is part of +Bus data structure (sdw_bus). This lock is used to serialize the message +transfers (read/write) within a SoundWire Bus instance. + +Below examples show how locks are acquired. + +Example 1 +--------- + +Message transfer. + + 1. For every message transfer + + a. Acquire Message lock. + + b. Transfer message (Read/Write) to Slave1 or broadcast message on + Bus in case of bank switch. + + c. Release Message lock :: + + +----------+ +---------+ + | | | | + | Bus | | Master | + | | | Driver | + | | | | + +----+-----+ +----+----+ + | | + | bus->ops->xfer_msg() | + <-------------------------------+ a. Acquire Message lock + | | b. Transfer message + | | + +-------------------------------> c. Release Message lock + | return success/error | d. Return success/error + | | + + + + +Example 2 +--------- + +Prepare operation. + + 1. Acquire lock for Bus instance associated with Master 1. + + 2. For every message transfer in Prepare operation + + a. Acquire Message lock. + + b. Transfer message (Read/Write) to Slave1 or broadcast message on + Bus in case of bank switch. + + c. Release Message lock. + + 3. Release lock for Bus instance associated with Master 1 :: + + +----------+ +---------+ + | | | | + | Bus | | Master | + | | | Driver | + | | | | + +----+-----+ +----+----+ + | | + | sdw_prepare_stream() | + <-------------------------------+ 1. Acquire bus lock + | | 2. Perform stream prepare + | | + | | + | bus->ops->xfer_msg() | + <-------------------------------+ a. Acquire Message lock + | | b. Transfer message + | | + +-------------------------------> c. Release Message lock + | return success/error | d. Return success/error + | | + | | + | return success/error | 3. Release bus lock + +-------------------------------> 4. Return success/error + | | + + + diff --git a/Documentation/driver-api/soundwire/stream.rst b/Documentation/driver-api/soundwire/stream.rst new file mode 100644 index 000000000000..29121aa55fb9 --- /dev/null +++ b/Documentation/driver-api/soundwire/stream.rst @@ -0,0 +1,372 @@ +========================= +Audio Stream in SoundWire +========================= + +An audio stream is a logical or virtual connection created between + + (1) System memory buffer(s) and Codec(s) + + (2) DSP memory buffer(s) and Codec(s) + + (3) FIFO(s) and Codec(s) + + (4) Codec(s) and Codec(s) + +which is typically driven by a DMA(s) channel through the data link. An +audio stream contains one or more channels of data. All channels within +stream must have same sample rate and same sample size. + +Assume a stream with two channels (Left & Right) is opened using SoundWire +interface. Below are some ways a stream can be represented in SoundWire. + +Stream Sample in memory (System memory, DSP memory or FIFOs) :: + + ------------------------- + | L | R | L | R | L | R | + ------------------------- + +Example 1: Stereo Stream with L and R channels is rendered from Master to +Slave. Both Master and Slave is using single port. :: + + +---------------+ Clock Signal +---------------+ + | Master +----------------------------------+ Slave | + | Interface | | Interface | + | | | 1 | + | | Data Signal | | + | L + R +----------------------------------+ L + R | + | (Data) | Data Direction | (Data) | + +---------------+ +-----------------------> +---------------+ + + +Example 2: Stereo Stream with L and R channels is captured from Slave to +Master. Both Master and Slave is using single port. :: + + + +---------------+ Clock Signal +---------------+ + | Master +----------------------------------+ Slave | + | Interface | | Interface | + | | | 1 | + | | Data Signal | | + | L + R +----------------------------------+ L + R | + | (Data) | Data Direction | (Data) | + +---------------+ <-----------------------+ +---------------+ + + +Example 3: Stereo Stream with L and R channels is rendered by Master. Each +of the L and R channel is received by two different Slaves. Master and both +Slaves are using single port. :: + + +---------------+ Clock Signal +---------------+ + | Master +---------+------------------------+ Slave | + | Interface | | | Interface | + | | | | 1 | + | | | Data Signal | | + | L + R +---+------------------------------+ L | + | (Data) | | | Data Direction | (Data) | + +---------------+ | | +-------------> +---------------+ + | | + | | + | | +---------------+ + | +----------------------> | Slave | + | | Interface | + | | 2 | + | | | + +----------------------------> | R | + | (Data) | + +---------------+ + + +Example 4: Stereo Stream with L and R channel is rendered by two different +Ports of the Master and is received by only single Port of the Slave +interface. :: + + +--------------------+ + | | + | +--------------+ +----------------+ + | | || | | + | | Data Port || L Channel | | + | | 1 |------------+ | | + | | L Channel || | +-----+----+ | + | | (Data) || | L + R Channel || Data | | + | Master +----------+ | +---+---------> || Port | | + | Interface | | || 1 | | + | +--------------+ | || | | + | | || | +----------+ | + | | Data Port |------------+ | | + | | 2 || R Channel | Slave | + | | R Channel || | Interface | + | | (Data) || | 1 | + | +--------------+ Clock Signal | L + R | + | +---------------------------> | (Data) | + +--------------------+ | | + +----------------+ + +SoundWire Stream Management flow +================================ + +Stream definitions +------------------ + + (1) Current stream: This is classified as the stream on which operation has + to be performed like prepare, enable, disable, de-prepare etc. + + (2) Active stream: This is classified as the stream which is already active + on Bus other than current stream. There can be multiple active streams + on the Bus. + +SoundWire Bus manages stream operations for each stream getting +rendered/captured on the SoundWire Bus. This section explains Bus operations +done for each of the stream allocated/released on Bus. Following are the +stream states maintained by the Bus for each of the audio stream. + + +SoundWire stream states +----------------------- + +Below shows the SoundWire stream states and state transition diagram. :: + + +-----------+ +------------+ +----------+ +----------+ + | ALLOCATED +---->| CONFIGURED +---->| PREPARED +---->| ENABLED | + | STATE | | STATE | | STATE | | STATE | + +-----------+ +------------+ +----------+ +----+-----+ + ^ + | + | + v + +----------+ +------------+ +----+-----+ + | RELEASED |<----------+ DEPREPARED |<-------+ DISABLED | + | STATE | | STATE | | STATE | + +----------+ +------------+ +----------+ + +NOTE: State transition between prepare and deprepare is supported in Spec +but not in the software (subsystem) + +NOTE2: Stream state transition checks need to be handled by caller +framework, for example ALSA/ASoC. No checks for stream transition exist in +SoundWire subsystem. + +Stream State Operations +----------------------- + +Below section explains the operations done by the Bus on Master(s) and +Slave(s) as part of stream state transitions. + +SDW_STREAM_ALLOCATED +~~~~~~~~~~~~~~~~~~~~ + +Allocation state for stream. This is the entry state +of the stream. Operations performed before entering in this state: + + (1) A stream runtime is allocated for the stream. This stream + runtime is used as a reference for all the operations performed + on the stream. + + (2) The resources required for holding stream runtime information are + allocated and initialized. This holds all stream related information + such as stream type (PCM/PDM) and parameters, Master and Slave + interface associated with the stream, stream state etc. + +After all above operations are successful, stream state is set to +``SDW_STREAM_ALLOCATED``. + +Bus implements below API for allocate a stream which needs to be called once +per stream. From ASoC DPCM framework, this stream state maybe linked to +.startup() operation. + + .. code-block:: c + int sdw_alloc_stream(char * stream_name); + + +SDW_STREAM_CONFIGURED +~~~~~~~~~~~~~~~~~~~~~ + +Configuration state of stream. Operations performed before entering in +this state: + + (1) The resources allocated for stream information in SDW_STREAM_ALLOCATED + state are updated here. This includes stream parameters, Master(s) + and Slave(s) runtime information associated with current stream. + + (2) All the Master(s) and Slave(s) associated with current stream provide + the port information to Bus which includes port numbers allocated by + Master(s) and Slave(s) for current stream and their channel mask. + +After all above operations are successful, stream state is set to +``SDW_STREAM_CONFIGURED``. + +Bus implements below APIs for CONFIG state which needs to be called by +the respective Master(s) and Slave(s) associated with stream. These APIs can +only be invoked once by respective Master(s) and Slave(s). From ASoC DPCM +framework, this stream state is linked to .hw_params() operation. + + .. code-block:: c + int sdw_stream_add_master(struct sdw_bus * bus, + struct sdw_stream_config * stream_config, + struct sdw_ports_config * ports_config, + struct sdw_stream_runtime * stream); + + int sdw_stream_add_slave(struct sdw_slave * slave, + struct sdw_stream_config * stream_config, + struct sdw_ports_config * ports_config, + struct sdw_stream_runtime * stream); + + +SDW_STREAM_PREPARED +~~~~~~~~~~~~~~~~~~~ + +Prepare state of stream. Operations performed before entering in this state: + + (1) Bus parameters such as bandwidth, frame shape, clock frequency, + are computed based on current stream as well as already active + stream(s) on Bus. Re-computation is required to accommodate current + stream on the Bus. + + (2) Transport and port parameters of all Master(s) and Slave(s) port(s) are + computed for the current as well as already active stream based on frame + shape and clock frequency computed in step 1. + + (3) Computed Bus and transport parameters are programmed in Master(s) and + Slave(s) registers. The banked registers programming is done on the + alternate bank (bank currently unused). Port(s) are enabled for the + already active stream(s) on the alternate bank (bank currently unused). + This is done in order to not disrupt already active stream(s). + + (4) Once all the values are programmed, Bus initiates switch to alternate + bank where all new values programmed gets into effect. + + (5) Ports of Master(s) and Slave(s) for current stream are prepared by + programming PrepareCtrl register. + +After all above operations are successful, stream state is set to +``SDW_STREAM_PREPARED``. + +Bus implements below API for PREPARE state which needs to be called once per +stream. From ASoC DPCM framework, this stream state is linked to +.prepare() operation. + + .. code-block:: c + int sdw_prepare_stream(struct sdw_stream_runtime * stream); + + +SDW_STREAM_ENABLED +~~~~~~~~~~~~~~~~~~ + +Enable state of stream. The data port(s) are enabled upon entering this state. +Operations performed before entering in this state: + + (1) All the values computed in SDW_STREAM_PREPARED state are programmed + in alternate bank (bank currently unused). It includes programming of + already active stream(s) as well. + + (2) All the Master(s) and Slave(s) port(s) for the current stream are + enabled on alternate bank (bank currently unused) by programming + ChannelEn register. + + (3) Once all the values are programmed, Bus initiates switch to alternate + bank where all new values programmed gets into effect and port(s) + associated with current stream are enabled. + +After all above operations are successful, stream state is set to +``SDW_STREAM_ENABLED``. + +Bus implements below API for ENABLE state which needs to be called once per +stream. From ASoC DPCM framework, this stream state is linked to +.trigger() start operation. + + .. code-block:: c + int sdw_enable_stream(struct sdw_stream_runtime * stream); + +SDW_STREAM_DISABLED +~~~~~~~~~~~~~~~~~~~ + +Disable state of stream. The data port(s) are disabled upon exiting this state. +Operations performed before entering in this state: + + (1) All the Master(s) and Slave(s) port(s) for the current stream are + disabled on alternate bank (bank currently unused) by programming + ChannelEn register. + + (2) All the current configuration of Bus and active stream(s) are programmed + into alternate bank (bank currently unused). + + (3) Once all the values are programmed, Bus initiates switch to alternate + bank where all new values programmed gets into effect and port(s) associated + with current stream are disabled. + +After all above operations are successful, stream state is set to +``SDW_STREAM_DISABLED``. + +Bus implements below API for DISABLED state which needs to be called once +per stream. From ASoC DPCM framework, this stream state is linked to +.trigger() stop operation. + + .. code-block:: c + int sdw_disable_stream(struct sdw_stream_runtime * stream); + + +SDW_STREAM_DEPREPARED +~~~~~~~~~~~~~~~~~~~~~ + +De-prepare state of stream. Operations performed before entering in this +state: + + (1) All the port(s) of Master(s) and Slave(s) for current stream are + de-prepared by programming PrepareCtrl register. + + (2) The payload bandwidth of current stream is reduced from the total + bandwidth requirement of bus and new parameters calculated and + applied by performing bank switch etc. + +After all above operations are successful, stream state is set to +``SDW_STREAM_DEPREPARED``. + +Bus implements below API for DEPREPARED state which needs to be called once +per stream. From ASoC DPCM framework, this stream state is linked to +.trigger() stop operation. + + .. code-block:: c + int sdw_deprepare_stream(struct sdw_stream_runtime * stream); + + +SDW_STREAM_RELEASED +~~~~~~~~~~~~~~~~~~~ + +Release state of stream. Operations performed before entering in this state: + + (1) Release port resources for all Master(s) and Slave(s) port(s) + associated with current stream. + + (2) Release Master(s) and Slave(s) runtime resources associated with + current stream. + + (3) Release stream runtime resources associated with current stream. + +After all above operations are successful, stream state is set to +``SDW_STREAM_RELEASED``. + +Bus implements below APIs for RELEASE state which needs to be called by +all the Master(s) and Slave(s) associated with stream. From ASoC DPCM +framework, this stream state is linked to .hw_free() operation. + + .. code-block:: c + int sdw_stream_remove_master(struct sdw_bus * bus, + struct sdw_stream_runtime * stream); + int sdw_stream_remove_slave(struct sdw_slave * slave, + struct sdw_stream_runtime * stream); + + +The .shutdown() ASoC DPCM operation calls below Bus API to release +stream assigned as part of ALLOCATED state. + +In .shutdown() the data structure maintaining stream state are freed up. + + .. code-block:: c + void sdw_release_stream(struct sdw_stream_runtime * stream); + +Not Supported +============= + +1. A single port with multiple channels supported cannot be used between two +streams or across stream. For example a port with 4 channels cannot be used +to handle 2 independent stereo streams even though it's possible in theory +in SoundWire. diff --git a/Documentation/driver-api/target.rst b/Documentation/driver-api/target.rst new file mode 100644 index 000000000000..4363611dd86d --- /dev/null +++ b/Documentation/driver-api/target.rst @@ -0,0 +1,64 @@ +================================= +target and iSCSI Interfaces Guide +================================= + +Introduction and Overview +========================= + +TBD + +Target core device interfaces +============================= + +.. kernel-doc:: drivers/target/target_core_device.c + :export: + +Target core transport interfaces +================================ + +.. kernel-doc:: drivers/target/target_core_transport.c + :export: + +Target-supported userspace I/O +============================== + +.. kernel-doc:: drivers/target/target_core_user.c + :doc: Userspace I/O + +.. kernel-doc:: include/uapi/linux/target_core_user.h + :doc: Ring Design + +iSCSI helper functions +====================== + +.. kernel-doc:: drivers/scsi/libiscsi.c + :export: + + +iSCSI boot information +====================== + +.. kernel-doc:: drivers/scsi/iscsi_boot_sysfs.c + :export: + + +iSCSI transport class +===================== + +The file drivers/scsi/scsi_transport_iscsi.c defines transport +attributes for the iSCSI class, which sends SCSI packets over TCP/IP +connections. + +.. kernel-doc:: drivers/scsi/scsi_transport_iscsi.c + :export: + + +iSCSI TCP interfaces +==================== + +.. kernel-doc:: drivers/scsi/iscsi_tcp.c + :internal: + +.. kernel-doc:: drivers/scsi/libiscsi_tcp.c + :export: + diff --git a/Documentation/driver-api/uio-howto.rst b/Documentation/driver-api/uio-howto.rst index 92056c20e070..fb2eb73be4a3 100644 --- a/Documentation/driver-api/uio-howto.rst +++ b/Documentation/driver-api/uio-howto.rst @@ -711,7 +711,8 @@ The vmbus device regions are mapped into uio device resources: If a subchannel is created by a request to host, then the uio_hv_generic device driver will create a sysfs binary file for the per-channel ring buffer. -For example: +For example:: + /sys/bus/vmbus/devices/3811fe4d-0fa0-4b62-981a-74fc1084c757/channels/21/ring Further information diff --git a/Documentation/driver-api/usb/dwc3.rst b/Documentation/driver-api/usb/dwc3.rst index c3dc84a50ce5..8b36ff11cef9 100644 --- a/Documentation/driver-api/usb/dwc3.rst +++ b/Documentation/driver-api/usb/dwc3.rst @@ -674,9 +674,8 @@ operations, both of which can be traced. Format is:: __entry->flags & DWC3_EP_ENABLED ? 'E' : 'e', __entry->flags & DWC3_EP_STALL ? 'S' : 's', __entry->flags & DWC3_EP_WEDGE ? 'W' : 'w', - __entry->flags & DWC3_EP_BUSY ? 'B' : 'b', + __entry->flags & DWC3_EP_TRANSFER_STARTED ? 'B' : 'b', __entry->flags & DWC3_EP_PENDING_REQUEST ? 'P' : 'p', - __entry->flags & DWC3_EP_MISSED_ISOC ? 'M' : 'm', __entry->flags & DWC3_EP_END_TRANSFER_PENDING ? 'E' : 'e', __entry->direction ? '<' : '>' ) diff --git a/Documentation/features/core/BPF-JIT/arch-support.txt b/Documentation/features/core/BPF-JIT/arch-support.txt deleted file mode 100644 index 0b96b4e1e7d4..000000000000 --- a/Documentation/features/core/BPF-JIT/arch-support.txt +++ /dev/null @@ -1,31 +0,0 @@ -# -# Feature name: BPF-JIT -# Kconfig: HAVE_BPF_JIT -# description: arch supports BPF JIT optimizations -# - ----------------------- - | arch |status| - ----------------------- - | alpha: | TODO | - | arc: | TODO | - | arm: | ok | - | arm64: | ok | - | c6x: | TODO | - | h8300: | TODO | - | hexagon: | TODO | - | ia64: | TODO | - | m68k: | TODO | - | microblaze: | TODO | - | mips: | ok | - | nios2: | TODO | - | openrisc: | TODO | - | parisc: | TODO | - | powerpc: | ok | - | s390: | ok | - | sh: | TODO | - | sparc: | ok | - | um: | TODO | - | unicore32: | TODO | - | x86: | ok | - | xtensa: | TODO | - ----------------------- diff --git a/Documentation/features/core/cBPF-JIT/arch-support.txt b/Documentation/features/core/cBPF-JIT/arch-support.txt new file mode 100644 index 000000000000..90459cdde314 --- /dev/null +++ b/Documentation/features/core/cBPF-JIT/arch-support.txt @@ -0,0 +1,33 @@ +# +# Feature name: cBPF-JIT +# Kconfig: HAVE_CBPF_JIT +# description: arch supports cBPF JIT optimizations +# + ----------------------- + | arch |status| + ----------------------- + | alpha: | TODO | + | arc: | TODO | + | arm: | TODO | + | arm64: | TODO | + | c6x: | TODO | + | h8300: | TODO | + | hexagon: | TODO | + | ia64: | TODO | + | m68k: | TODO | + | microblaze: | TODO | + | mips: | ok | + | nds32: | TODO | + | nios2: | TODO | + | openrisc: | TODO | + | parisc: | TODO | + | powerpc: | ok | + | riscv: | TODO | + | s390: | TODO | + | sh: | TODO | + | sparc: | ok | + | um: | TODO | + | unicore32: | TODO | + | x86: | TODO | + | xtensa: | TODO | + ----------------------- diff --git a/Documentation/features/core/eBPF-JIT/arch-support.txt b/Documentation/features/core/eBPF-JIT/arch-support.txt new file mode 100644 index 000000000000..c90a0382fe66 --- /dev/null +++ b/Documentation/features/core/eBPF-JIT/arch-support.txt @@ -0,0 +1,33 @@ +# +# Feature name: eBPF-JIT +# Kconfig: HAVE_EBPF_JIT +# description: arch supports eBPF JIT optimizations +# + ----------------------- + | arch |status| + ----------------------- + | alpha: | TODO | + | arc: | TODO | + | arm: | ok | + | arm64: | ok | + | c6x: | TODO | + | h8300: | TODO | + | hexagon: | TODO | + | ia64: | TODO | + | m68k: | TODO | + | microblaze: | TODO | + | mips: | ok | + | nds32: | TODO | + | nios2: | TODO | + | openrisc: | TODO | + | parisc: | TODO | + | powerpc: | ok | + | riscv: | TODO | + | s390: | ok | + | sh: | TODO | + | sparc: | ok | + | um: | TODO | + | unicore32: | TODO | + | x86: | ok | + | xtensa: | TODO | + ----------------------- diff --git a/Documentation/features/core/generic-idle-thread/arch-support.txt b/Documentation/features/core/generic-idle-thread/arch-support.txt index 372a2b18a617..0ef6acdb991c 100644 --- a/Documentation/features/core/generic-idle-thread/arch-support.txt +++ b/Documentation/features/core/generic-idle-thread/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | - | openrisc: | TODO | + | openrisc: | ok | | parisc: | ok | | powerpc: | ok | + | riscv: | ok | | s390: | ok | | sh: | ok | | sparc: | ok | diff --git a/Documentation/features/core/jump-labels/arch-support.txt b/Documentation/features/core/jump-labels/arch-support.txt index ad97217b003b..27cbd63abfd2 100644 --- a/Documentation/features/core/jump-labels/arch-support.txt +++ b/Documentation/features/core/jump-labels/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | TODO | | sparc: | ok | diff --git a/Documentation/features/core/tracehook/arch-support.txt b/Documentation/features/core/tracehook/arch-support.txt index 36ee7bef5d18..f44c274e40ed 100644 --- a/Documentation/features/core/tracehook/arch-support.txt +++ b/Documentation/features/core/tracehook/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | ok | | nios2: | ok | | openrisc: | ok | | parisc: | ok | | powerpc: | ok | + | riscv: | ok | | s390: | ok | | sh: | ok | | sparc: | ok | diff --git a/Documentation/features/debug/KASAN/arch-support.txt b/Documentation/features/debug/KASAN/arch-support.txt index f5c99fa576d3..282ecc8ea1da 100644 --- a/Documentation/features/debug/KASAN/arch-support.txt +++ b/Documentation/features/debug/KASAN/arch-support.txt @@ -17,15 +17,17 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | TODO | + | riscv: | TODO | | s390: | TODO | | sh: | TODO | | sparc: | TODO | | um: | TODO | | unicore32: | TODO | - | x86: | ok | 64-bit only + | x86: | ok | | xtensa: | ok | ----------------------- diff --git a/Documentation/features/debug/gcov-profile-all/arch-support.txt b/Documentation/features/debug/gcov-profile-all/arch-support.txt index 5170a9934843..01b2b3004e0a 100644 --- a/Documentation/features/debug/gcov-profile-all/arch-support.txt +++ b/Documentation/features/debug/gcov-profile-all/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | ok | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | ok | | sparc: | TODO | diff --git a/Documentation/features/debug/kgdb/arch-support.txt b/Documentation/features/debug/kgdb/arch-support.txt index 13b6e994ae1f..3b4dff22329f 100644 --- a/Documentation/features/debug/kgdb/arch-support.txt +++ b/Documentation/features/debug/kgdb/arch-support.txt @@ -11,16 +11,18 @@ | arm: | ok | | arm64: | ok | | c6x: | TODO | - | h8300: | TODO | + | h8300: | ok | | hexagon: | ok | | ia64: | TODO | | m68k: | TODO | | microblaze: | ok | | mips: | ok | + | nds32: | TODO | | nios2: | ok | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | TODO | | sh: | ok | | sparc: | ok | diff --git a/Documentation/features/debug/kprobes-on-ftrace/arch-support.txt b/Documentation/features/debug/kprobes-on-ftrace/arch-support.txt index 419bb38820e7..7e963d0ae646 100644 --- a/Documentation/features/debug/kprobes-on-ftrace/arch-support.txt +++ b/Documentation/features/debug/kprobes-on-ftrace/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | TODO | | sh: | TODO | | sparc: | TODO | diff --git a/Documentation/features/debug/kprobes/arch-support.txt b/Documentation/features/debug/kprobes/arch-support.txt index 52b3ace0a030..4ada027faf16 100644 --- a/Documentation/features/debug/kprobes/arch-support.txt +++ b/Documentation/features/debug/kprobes/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | ok | | arm: | ok | - | arm64: | TODO | + | arm64: | ok | | c6x: | TODO | | h8300: | TODO | | hexagon: | TODO | @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | ok | | s390: | ok | | sh: | ok | | sparc: | ok | diff --git a/Documentation/features/debug/kretprobes/arch-support.txt b/Documentation/features/debug/kretprobes/arch-support.txt index 180d24419518..044e13fcca5d 100644 --- a/Documentation/features/debug/kretprobes/arch-support.txt +++ b/Documentation/features/debug/kretprobes/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | ok | | arm: | ok | - | arm64: | TODO | + | arm64: | ok | | c6x: | TODO | | h8300: | TODO | | hexagon: | TODO | @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | ok | | sparc: | ok | diff --git a/Documentation/features/debug/optprobes/arch-support.txt b/Documentation/features/debug/optprobes/arch-support.txt index 0a1241f45e41..dce7669c918f 100644 --- a/Documentation/features/debug/optprobes/arch-support.txt +++ b/Documentation/features/debug/optprobes/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | - | powerpc: | TODO | + | powerpc: | ok | + | riscv: | TODO | | s390: | TODO | | sh: | TODO | | sparc: | TODO | diff --git a/Documentation/features/debug/stackprotector/arch-support.txt b/Documentation/features/debug/stackprotector/arch-support.txt index 570019572383..954ac1c95553 100644 --- a/Documentation/features/debug/stackprotector/arch-support.txt +++ b/Documentation/features/debug/stackprotector/arch-support.txt @@ -1,6 +1,6 @@ # # Feature name: stackprotector -# Kconfig: HAVE_CC_STACKPROTECTOR +# Kconfig: HAVE_STACKPROTECTOR # description: arch supports compiler driven stack overflow protection # ----------------------- @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | TODO | + | riscv: | TODO | | s390: | TODO | | sh: | ok | | sparc: | TODO | diff --git a/Documentation/features/debug/uprobes/arch-support.txt b/Documentation/features/debug/uprobes/arch-support.txt index 0b8d922eb799..1a3f9d3229bf 100644 --- a/Documentation/features/debug/uprobes/arch-support.txt +++ b/Documentation/features/debug/uprobes/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | TODO | | arm: | ok | - | arm64: | TODO | + | arm64: | ok | | c6x: | TODO | | h8300: | TODO | | hexagon: | TODO | @@ -17,13 +17,15 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | TODO | - | sparc: | TODO | + | sparc: | ok | | um: | TODO | | unicore32: | TODO | | x86: | ok | diff --git a/Documentation/features/debug/user-ret-profiler/arch-support.txt b/Documentation/features/debug/user-ret-profiler/arch-support.txt index 13852ae62e9e..1d78d1069a5f 100644 --- a/Documentation/features/debug/user-ret-profiler/arch-support.txt +++ b/Documentation/features/debug/user-ret-profiler/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | TODO | + | riscv: | TODO | | s390: | TODO | | sh: | TODO | | sparc: | TODO | diff --git a/Documentation/features/io/dma-api-debug/arch-support.txt b/Documentation/features/io/dma-api-debug/arch-support.txt deleted file mode 100644 index e438ed675623..000000000000 --- a/Documentation/features/io/dma-api-debug/arch-support.txt +++ /dev/null @@ -1,31 +0,0 @@ -# -# Feature name: dma-api-debug -# Kconfig: HAVE_DMA_API_DEBUG -# description: arch supports DMA debug facilities -# - ----------------------- - | arch |status| - ----------------------- - | alpha: | TODO | - | arc: | TODO | - | arm: | ok | - | arm64: | ok | - | c6x: | ok | - | h8300: | TODO | - | hexagon: | TODO | - | ia64: | ok | - | m68k: | TODO | - | microblaze: | ok | - | mips: | ok | - | nios2: | TODO | - | openrisc: | TODO | - | parisc: | TODO | - | powerpc: | ok | - | s390: | ok | - | sh: | ok | - | sparc: | ok | - | um: | TODO | - | unicore32: | TODO | - | x86: | ok | - | xtensa: | ok | - ----------------------- diff --git a/Documentation/features/io/dma-contiguous/arch-support.txt b/Documentation/features/io/dma-contiguous/arch-support.txt index 47f64a433df0..30c072d2b67c 100644 --- a/Documentation/features/io/dma-contiguous/arch-support.txt +++ b/Documentation/features/io/dma-contiguous/arch-support.txt @@ -17,11 +17,13 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | TODO | - | s390: | TODO | + | riscv: | ok | + | s390: | ok | | sh: | TODO | | sparc: | TODO | | um: | TODO | diff --git a/Documentation/features/io/sg-chain/arch-support.txt b/Documentation/features/io/sg-chain/arch-support.txt index 07f357fadbff..6554f0372c3f 100644 --- a/Documentation/features/io/sg-chain/arch-support.txt +++ b/Documentation/features/io/sg-chain/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | TODO | | sparc: | ok | diff --git a/Documentation/features/lib/strncasecmp/arch-support.txt b/Documentation/features/lib/strncasecmp/arch-support.txt deleted file mode 100644 index 4f3a6a0e4e68..000000000000 --- a/Documentation/features/lib/strncasecmp/arch-support.txt +++ /dev/null @@ -1,31 +0,0 @@ -# -# Feature name: strncasecmp -# Kconfig: __HAVE_ARCH_STRNCASECMP -# description: arch provides an optimized strncasecmp() function -# - ----------------------- - | arch |status| - ----------------------- - | alpha: | TODO | - | arc: | TODO | - | arm: | TODO | - | arm64: | TODO | - | c6x: | TODO | - | h8300: | TODO | - | hexagon: | TODO | - | ia64: | TODO | - | m68k: | TODO | - | microblaze: | TODO | - | mips: | TODO | - | nios2: | TODO | - | openrisc: | TODO | - | parisc: | TODO | - | powerpc: | TODO | - | s390: | TODO | - | sh: | TODO | - | sparc: | TODO | - | um: | TODO | - | unicore32: | TODO | - | x86: | TODO | - | xtensa: | TODO | - ----------------------- diff --git a/Documentation/features/locking/cmpxchg-local/arch-support.txt b/Documentation/features/locking/cmpxchg-local/arch-support.txt index 482a0b09d1f8..51704a2dc8d1 100644 --- a/Documentation/features/locking/cmpxchg-local/arch-support.txt +++ b/Documentation/features/locking/cmpxchg-local/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | TODO | | arm: | TODO | - | arm64: | TODO | + | arm64: | ok | | c6x: | TODO | | h8300: | TODO | | hexagon: | TODO | @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | TODO | + | riscv: | TODO | | s390: | ok | | sh: | TODO | | sparc: | TODO | diff --git a/Documentation/features/locking/lockdep/arch-support.txt b/Documentation/features/locking/lockdep/arch-support.txt index bb35c5ba6286..bd39c5edd460 100644 --- a/Documentation/features/locking/lockdep/arch-support.txt +++ b/Documentation/features/locking/lockdep/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | ok | | mips: | ok | + | nds32: | ok | | nios2: | TODO | - | openrisc: | TODO | + | openrisc: | ok | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | ok | | sparc: | ok | diff --git a/Documentation/features/locking/queued-rwlocks/arch-support.txt b/Documentation/features/locking/queued-rwlocks/arch-support.txt index 627e9a6b2db9..da7aff3bee0b 100644 --- a/Documentation/features/locking/queued-rwlocks/arch-support.txt +++ b/Documentation/features/locking/queued-rwlocks/arch-support.txt @@ -9,21 +9,23 @@ | alpha: | TODO | | arc: | TODO | | arm: | TODO | - | arm64: | TODO | + | arm64: | ok | | c6x: | TODO | | h8300: | TODO | | hexagon: | TODO | | ia64: | TODO | | m68k: | TODO | | microblaze: | TODO | - | mips: | TODO | + | mips: | ok | + | nds32: | TODO | | nios2: | TODO | - | openrisc: | TODO | + | openrisc: | ok | | parisc: | TODO | | powerpc: | TODO | + | riscv: | TODO | | s390: | TODO | | sh: | TODO | - | sparc: | TODO | + | sparc: | ok | | um: | TODO | | unicore32: | TODO | | x86: | ok | diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.txt b/Documentation/features/locking/queued-spinlocks/arch-support.txt index 9edda216cdfb..478e9101322c 100644 --- a/Documentation/features/locking/queued-spinlocks/arch-support.txt +++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt @@ -16,14 +16,16 @@ | ia64: | TODO | | m68k: | TODO | | microblaze: | TODO | - | mips: | TODO | + | mips: | ok | + | nds32: | TODO | | nios2: | TODO | - | openrisc: | TODO | + | openrisc: | ok | | parisc: | TODO | | powerpc: | TODO | + | riscv: | TODO | | s390: | TODO | | sh: | TODO | - | sparc: | TODO | + | sparc: | ok | | um: | TODO | | unicore32: | TODO | | x86: | ok | diff --git a/Documentation/features/locking/rwsem-optimized/arch-support.txt b/Documentation/features/locking/rwsem-optimized/arch-support.txt index 8d9afb10b16e..e54b1f1a8091 100644 --- a/Documentation/features/locking/rwsem-optimized/arch-support.txt +++ b/Documentation/features/locking/rwsem-optimized/arch-support.txt @@ -1,6 +1,6 @@ # # Feature name: rwsem-optimized -# Kconfig: Optimized asm/rwsem.h +# Kconfig: !RWSEM_GENERIC_SPINLOCK # description: arch provides optimized rwsem APIs # ----------------------- @@ -8,8 +8,8 @@ ----------------------- | alpha: | ok | | arc: | TODO | - | arm: | TODO | - | arm64: | TODO | + | arm: | ok | + | arm64: | ok | | c6x: | TODO | | h8300: | TODO | | hexagon: | TODO | @@ -17,14 +17,16 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | TODO | + | riscv: | TODO | | s390: | ok | | sh: | ok | | sparc: | ok | - | um: | TODO | + | um: | ok | | unicore32: | TODO | | x86: | ok | | xtensa: | ok | diff --git a/Documentation/features/perf/kprobes-event/arch-support.txt b/Documentation/features/perf/kprobes-event/arch-support.txt index d01239ee34b3..7331402d1887 100644 --- a/Documentation/features/perf/kprobes-event/arch-support.txt +++ b/Documentation/features/perf/kprobes-event/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | TODO | | arm: | ok | - | arm64: | TODO | + | arm64: | ok | | c6x: | TODO | | h8300: | TODO | | hexagon: | ok | @@ -17,13 +17,15 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | ok | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | ok | - | sparc: | TODO | + | sparc: | ok | | um: | TODO | | unicore32: | TODO | | x86: | ok | diff --git a/Documentation/features/perf/perf-regs/arch-support.txt b/Documentation/features/perf/perf-regs/arch-support.txt index 458faba5311a..53feeee6cdad 100644 --- a/Documentation/features/perf/perf-regs/arch-support.txt +++ b/Documentation/features/perf/perf-regs/arch-support.txt @@ -17,11 +17,13 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | - | s390: | TODO | + | riscv: | TODO | + | s390: | ok | | sh: | TODO | | sparc: | TODO | | um: | TODO | diff --git a/Documentation/features/perf/perf-stackdump/arch-support.txt b/Documentation/features/perf/perf-stackdump/arch-support.txt index 545d01c69c88..16164348e0ea 100644 --- a/Documentation/features/perf/perf-stackdump/arch-support.txt +++ b/Documentation/features/perf/perf-stackdump/arch-support.txt @@ -17,11 +17,13 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | - | s390: | TODO | + | riscv: | TODO | + | s390: | ok | | sh: | TODO | | sparc: | TODO | | um: | TODO | diff --git a/Documentation/features/sched/membarrier-sync-core/arch-support.txt b/Documentation/features/sched/membarrier-sync-core/arch-support.txt index 85a6c9d4571c..dbdf62907703 100644 --- a/Documentation/features/sched/membarrier-sync-core/arch-support.txt +++ b/Documentation/features/sched/membarrier-sync-core/arch-support.txt @@ -40,10 +40,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | TODO | + | riscv: | TODO | | s390: | TODO | | sh: | TODO | | sparc: | TODO | diff --git a/Documentation/features/sched/numa-balancing/arch-support.txt b/Documentation/features/sched/numa-balancing/arch-support.txt index 347508863872..c68bb2c2cb62 100644 --- a/Documentation/features/sched/numa-balancing/arch-support.txt +++ b/Documentation/features/sched/numa-balancing/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | .. | | arm: | .. | - | arm64: | .. | + | arm64: | ok | | c6x: | .. | | h8300: | .. | | hexagon: | .. | @@ -17,11 +17,13 @@ | m68k: | .. | | microblaze: | .. | | mips: | TODO | + | nds32: | TODO | | nios2: | .. | | openrisc: | .. | | parisc: | .. | | powerpc: | ok | - | s390: | .. | + | riscv: | TODO | + | s390: | ok | | sh: | .. | | sparc: | TODO | | um: | .. | diff --git a/Documentation/features/scripts/features-refresh.sh b/Documentation/features/scripts/features-refresh.sh new file mode 100755 index 000000000000..9e72d38a0720 --- /dev/null +++ b/Documentation/features/scripts/features-refresh.sh @@ -0,0 +1,98 @@ +# +# Small script that refreshes the kernel feature support status in place. +# + +for F_FILE in Documentation/features/*/*/arch-support.txt; do + F=$(grep "^# Kconfig:" "$F_FILE" | cut -c26-) + + # + # Each feature F is identified by a pair (O, K), where 'O' can + # be either the empty string (for 'nop') or "not" (the logical + # negation operator '!'); other operators are not supported. + # + O="" + K=$F + if [[ "$F" == !* ]]; then + O="not" + K=$(echo $F | sed -e 's/^!//g') + fi + + # + # F := (O, K) is 'valid' iff there is a Kconfig file (for some + # arch) which contains K. + # + # Notice that this definition entails an 'asymmetry' between + # the case 'O = ""' and the case 'O = "not"'. E.g., F may be + # _invalid_ if: + # + # [case 'O = ""'] + # 1) no arch provides support for F, + # 2) K does not exist (e.g., it was renamed/mis-typed); + # + # [case 'O = "not"'] + # 3) all archs provide support for F, + # 4) as in (2). + # + # The rationale for adopting this definition (and, thus, for + # keeping the asymmetry) is: + # + # We want to be able to 'detect' (2) (or (4)). + # + # (1) and (3) may further warn the developers about the fact + # that K can be removed. + # + F_VALID="false" + for ARCH_DIR in arch/*/; do + K_FILES=$(find $ARCH_DIR -name "Kconfig*") + K_GREP=$(grep "$K" $K_FILES) + if [ ! -z "$K_GREP" ]; then + F_VALID="true" + break + fi + done + if [ "$F_VALID" = "false" ]; then + printf "WARNING: '%s' is not a valid Kconfig\n" "$F" + fi + + T_FILE="$F_FILE.tmp" + grep "^#" $F_FILE > $T_FILE + echo " -----------------------" >> $T_FILE + echo " | arch |status|" >> $T_FILE + echo " -----------------------" >> $T_FILE + for ARCH_DIR in arch/*/; do + ARCH=$(echo $ARCH_DIR | sed -e 's/arch//g' | sed -e 's/\///g') + K_FILES=$(find $ARCH_DIR -name "Kconfig*") + K_GREP=$(grep "$K" $K_FILES) + # + # Arch support status values for (O, K) are updated according + # to the following rules. + # + # - ("", K) is 'supported by a given arch', if there is a + # Kconfig file for that arch which contains K; + # + # - ("not", K) is 'supported by a given arch', if there is + # no Kconfig file for that arch which contains K; + # + # - otherwise: preserve the previous status value (if any), + # default to 'not yet supported'. + # + # Notice that, according these rules, invalid features may be + # updated/modified. + # + if [ "$O" = "" ] && [ ! -z "$K_GREP" ]; then + printf " |%12s: | ok |\n" "$ARCH" >> $T_FILE + elif [ "$O" = "not" ] && [ -z "$K_GREP" ]; then + printf " |%12s: | ok |\n" "$ARCH" >> $T_FILE + else + S=$(grep -v "^#" "$F_FILE" | grep " $ARCH:") + if [ ! -z "$S" ]; then + echo "$S" >> $T_FILE + else + printf " |%12s: | TODO |\n" "$ARCH" \ + >> $T_FILE + fi + fi + done + echo " -----------------------" >> $T_FILE + mv $T_FILE $F_FILE +done diff --git a/Documentation/features/seccomp/seccomp-filter/arch-support.txt b/Documentation/features/seccomp/seccomp-filter/arch-support.txt index e4fad58a05e5..d4271b493b41 100644 --- a/Documentation/features/seccomp/seccomp-filter/arch-support.txt +++ b/Documentation/features/seccomp/seccomp-filter/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | - | parisc: | TODO | - | powerpc: | TODO | + | parisc: | ok | + | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | TODO | | sparc: | TODO | diff --git a/Documentation/features/time/arch-tick-broadcast/arch-support.txt b/Documentation/features/time/arch-tick-broadcast/arch-support.txt index 8052904b25fc..83d9e68462bb 100644 --- a/Documentation/features/time/arch-tick-broadcast/arch-support.txt +++ b/Documentation/features/time/arch-tick-broadcast/arch-support.txt @@ -17,12 +17,14 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | TODO | - | sh: | TODO | + | sh: | ok | | sparc: | TODO | | um: | TODO | | unicore32: | TODO | diff --git a/Documentation/features/time/clockevents/arch-support.txt b/Documentation/features/time/clockevents/arch-support.txt index 7c76b946297e..3d4908fce6da 100644 --- a/Documentation/features/time/clockevents/arch-support.txt +++ b/Documentation/features/time/clockevents/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | ok | | microblaze: | ok | | mips: | ok | + | nds32: | ok | | nios2: | ok | | openrisc: | ok | - | parisc: | TODO | + | parisc: | ok | | powerpc: | ok | + | riscv: | ok | | s390: | ok | | sh: | ok | | sparc: | ok | diff --git a/Documentation/features/time/context-tracking/arch-support.txt b/Documentation/features/time/context-tracking/arch-support.txt index 9433b3e523b3..c29974afffaa 100644 --- a/Documentation/features/time/context-tracking/arch-support.txt +++ b/Documentation/features/time/context-tracking/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | TODO | | sh: | TODO | | sparc: | ok | diff --git a/Documentation/features/time/irq-time-acct/arch-support.txt b/Documentation/features/time/irq-time-acct/arch-support.txt index 212dde0b578c..8d73c463ec27 100644 --- a/Documentation/features/time/irq-time-acct/arch-support.txt +++ b/Documentation/features/time/irq-time-acct/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | .. | - | powerpc: | .. | + | powerpc: | ok | + | riscv: | TODO | | s390: | .. | | sh: | TODO | | sparc: | .. | diff --git a/Documentation/features/time/modern-timekeeping/arch-support.txt b/Documentation/features/time/modern-timekeeping/arch-support.txt index 4074028f72f7..e7c6ea6b8fb3 100644 --- a/Documentation/features/time/modern-timekeeping/arch-support.txt +++ b/Documentation/features/time/modern-timekeeping/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | ok | | mips: | ok | + | nds32: | ok | | nios2: | ok | | openrisc: | ok | | parisc: | ok | | powerpc: | ok | + | riscv: | ok | | s390: | ok | | sh: | ok | | sparc: | ok | diff --git a/Documentation/features/time/virt-cpuacct/arch-support.txt b/Documentation/features/time/virt-cpuacct/arch-support.txt index a394d8820517..4646457461cf 100644 --- a/Documentation/features/time/virt-cpuacct/arch-support.txt +++ b/Documentation/features/time/virt-cpuacct/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | ok | | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | TODO | | sparc: | ok | diff --git a/Documentation/features/vm/ELF-ASLR/arch-support.txt b/Documentation/features/vm/ELF-ASLR/arch-support.txt index 082f93d5b40e..1f71d090ff2c 100644 --- a/Documentation/features/vm/ELF-ASLR/arch-support.txt +++ b/Documentation/features/vm/ELF-ASLR/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | ok | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | - | parisc: | TODO | + | parisc: | ok | | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | TODO | | sparc: | TODO | diff --git a/Documentation/features/vm/PG_uncached/arch-support.txt b/Documentation/features/vm/PG_uncached/arch-support.txt index 605e0abb756d..fbd5aa463b0a 100644 --- a/Documentation/features/vm/PG_uncached/arch-support.txt +++ b/Documentation/features/vm/PG_uncached/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | TODO | + | riscv: | TODO | | s390: | TODO | | sh: | TODO | | sparc: | TODO | diff --git a/Documentation/features/vm/THP/arch-support.txt b/Documentation/features/vm/THP/arch-support.txt index 7a8eb0bd5ca8..5d7ecc378f29 100644 --- a/Documentation/features/vm/THP/arch-support.txt +++ b/Documentation/features/vm/THP/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | .. | | microblaze: | .. | | mips: | ok | + | nds32: | TODO | | nios2: | .. | | openrisc: | .. | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | .. | | sparc: | ok | diff --git a/Documentation/features/vm/TLB/arch-support.txt b/Documentation/features/vm/TLB/arch-support.txt index 35fb99b2b3ea..f7af9678eb66 100644 --- a/Documentation/features/vm/TLB/arch-support.txt +++ b/Documentation/features/vm/TLB/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | .. | | microblaze: | .. | | mips: | TODO | + | nds32: | TODO | | nios2: | .. | | openrisc: | .. | | parisc: | TODO | | powerpc: | TODO | + | riscv: | TODO | | s390: | TODO | | sh: | TODO | | sparc: | TODO | diff --git a/Documentation/features/vm/huge-vmap/arch-support.txt b/Documentation/features/vm/huge-vmap/arch-support.txt index ed8b943ad8fc..d0713ccc7117 100644 --- a/Documentation/features/vm/huge-vmap/arch-support.txt +++ b/Documentation/features/vm/huge-vmap/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | TODO | + | riscv: | TODO | | s390: | TODO | | sh: | TODO | | sparc: | TODO | diff --git a/Documentation/features/vm/ioremap_prot/arch-support.txt b/Documentation/features/vm/ioremap_prot/arch-support.txt index 589947bdf0a8..8527601a3739 100644 --- a/Documentation/features/vm/ioremap_prot/arch-support.txt +++ b/Documentation/features/vm/ioremap_prot/arch-support.txt @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | TODO | | sh: | ok | | sparc: | TODO | diff --git a/Documentation/features/vm/numa-memblock/arch-support.txt b/Documentation/features/vm/numa-memblock/arch-support.txt index 8b8bea0318a0..1a988052cd24 100644 --- a/Documentation/features/vm/numa-memblock/arch-support.txt +++ b/Documentation/features/vm/numa-memblock/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | .. | | arm: | .. | - | arm64: | .. | + | arm64: | ok | | c6x: | .. | | h8300: | .. | | hexagon: | .. | @@ -17,10 +17,12 @@ | m68k: | .. | | microblaze: | ok | | mips: | ok | + | nds32: | TODO | | nios2: | .. | | openrisc: | .. | | parisc: | .. | | powerpc: | ok | + | riscv: | ok | | s390: | ok | | sh: | ok | | sparc: | ok | diff --git a/Documentation/features/vm/pte_special/arch-support.txt b/Documentation/features/vm/pte_special/arch-support.txt index 055004f467d2..a8378424bc98 100644 --- a/Documentation/features/vm/pte_special/arch-support.txt +++ b/Documentation/features/vm/pte_special/arch-support.txt @@ -1,6 +1,6 @@ # # Feature name: pte_special -# Kconfig: __HAVE_ARCH_PTE_SPECIAL +# Kconfig: ARCH_HAS_PTE_SPECIAL # description: arch supports the pte_special()/pte_mkspecial() VM APIs # ----------------------- @@ -17,10 +17,12 @@ | m68k: | TODO | | microblaze: | TODO | | mips: | TODO | + | nds32: | TODO | | nios2: | TODO | | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | + | riscv: | TODO | | s390: | ok | | sh: | ok | | sparc: | ok | diff --git a/Documentation/filesystems/00-INDEX b/Documentation/filesystems/00-INDEX index b7bd6c9009cc..0937bade1099 100644 --- a/Documentation/filesystems/00-INDEX +++ b/Documentation/filesystems/00-INDEX @@ -10,8 +10,8 @@ afs.txt - info and examples for the distributed AFS (Andrew File System) fs. affs.txt - info and mount options for the Amiga Fast File System. -autofs4-mount-control.txt - - info on device control operations for autofs4 module. +autofs-mount-control.txt + - info on device control operations for autofs module. automount-support.txt - information about filesystem automount support. befs.txt @@ -89,8 +89,6 @@ locks.txt - info on file locking implementations, flock() vs. fcntl(), etc. mandatory-locking.txt - info on the Linux implementation of Sys V mandatory file locking. -ncpfs.txt - - info on Novell Netware(tm) filesystem using NCP protocol. nfs/ - nfs-related documentation. nilfs2.txt diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking index 75d2d57e2c44..2c391338c675 100644 --- a/Documentation/filesystems/Locking +++ b/Documentation/filesystems/Locking @@ -69,31 +69,31 @@ prototypes: locking rules: all may block - i_mutex(inode) -lookup: yes -create: yes -link: yes (both) -mknod: yes -symlink: yes -mkdir: yes -unlink: yes (both) -rmdir: yes (both) (see below) -rename: yes (all) (see below) + i_rwsem(inode) +lookup: shared +create: exclusive +link: exclusive (both) +mknod: exclusive +symlink: exclusive +mkdir: exclusive +unlink: exclusive (both) +rmdir: exclusive (both)(see below) +rename: exclusive (all) (see below) readlink: no get_link: no -setattr: yes +setattr: exclusive permission: no (may not block if called in rcu-walk mode) get_acl: no getattr: no listxattr: no fiemap: no update_time: no -atomic_open: yes +atomic_open: exclusive tmpfile: no - Additionally, ->rmdir(), ->unlink() and ->rename() have ->i_mutex on -victim. + Additionally, ->rmdir(), ->unlink() and ->rename() have ->i_rwsem + exclusive on victim. cross-directory ->rename() has (per-superblock) ->s_vfs_rename_sem. See Documentation/filesystems/directory-locking for more detailed discussion @@ -111,10 +111,10 @@ prototypes: locking rules: all may block - i_mutex(inode) + i_rwsem(inode) list: no get: no -set: yes +set: exclusive --------------------------- super_operations --------------------------- prototypes: @@ -217,14 +217,14 @@ prototypes: locking rules: All except set_page_dirty and freepage may block - PageLocked(page) i_mutex + PageLocked(page) i_rwsem writepage: yes, unlocks (see below) readpage: yes, unlocks writepages: set_page_dirty no readpages: -write_begin: locks the page yes -write_end: yes, unlocks yes +write_begin: locks the page exclusive +write_end: yes, unlocks exclusive bmap: invalidatepage: yes releasepage: yes @@ -439,7 +439,10 @@ prototypes: ssize_t (*read_iter) (struct kiocb *, struct iov_iter *); ssize_t (*write_iter) (struct kiocb *, struct iov_iter *); int (*iterate) (struct file *, struct dir_context *); - unsigned int (*poll) (struct file *, struct poll_table_struct *); + int (*iterate_shared) (struct file *, struct dir_context *); + __poll_t (*poll) (struct file *, struct poll_table_struct *); + struct wait_queue_head * (*get_poll_head)(struct file *, __poll_t); + __poll_t (*poll_mask) (struct file *, __poll_t); long (*unlocked_ioctl) (struct file *, unsigned int, unsigned long); long (*compat_ioctl) (struct file *, unsigned int, unsigned long); int (*mmap) (struct file *, struct vm_area_struct *); @@ -470,7 +473,7 @@ prototypes: }; locking rules: - All may block. + All except for ->poll_mask may block. ->llseek() locking has moved from llseek to the individual llseek implementations. If your fs is not using generic_file_llseek, you @@ -480,6 +483,10 @@ mutex or just to use i_size_read() instead. Note: this does not protect the file->f_pos against concurrent modifications since this is something the userspace has to take care about. +->iterate() is called with i_rwsem exclusive. + +->iterate_shared() is called with i_rwsem at least shared. + ->fasync() is responsible for maintaining the FASYNC bit in filp->f_flags. Most instances call fasync_helper(), which does that maintenance, so it's not normally something one needs to worry about. Return values > 0 will be @@ -498,6 +505,9 @@ in sys_read() and friends. the lease within the individual filesystem to record the result of the operation +->poll_mask can be called with or without the waitqueue lock for the waitqueue +returned from ->get_poll_head. + --------------------------- dquot_operations ------------------------------- prototypes: int (*write_dquot) (struct dquot *); diff --git a/Documentation/filesystems/autofs-mount-control.txt b/Documentation/filesystems/autofs-mount-control.txt new file mode 100644 index 000000000000..45edad6933cc --- /dev/null +++ b/Documentation/filesystems/autofs-mount-control.txt @@ -0,0 +1,406 @@ + +Miscellaneous Device control operations for the autofs kernel module +==================================================================== + +The problem +=========== + +There is a problem with active restarts in autofs (that is to say +restarting autofs when there are busy mounts). + +During normal operation autofs uses a file descriptor opened on the +directory that is being managed in order to be able to issue control +operations. Using a file descriptor gives ioctl operations access to +autofs specific information stored in the super block. The operations +are things such as setting an autofs mount catatonic, setting the +expire timeout and requesting expire checks. As is explained below, +certain types of autofs triggered mounts can end up covering an autofs +mount itself which prevents us being able to use open(2) to obtain a +file descriptor for these operations if we don't already have one open. + +Currently autofs uses "umount -l" (lazy umount) to clear active mounts +at restart. While using lazy umount works for most cases, anything that +needs to walk back up the mount tree to construct a path, such as +getcwd(2) and the proc file system /proc//cwd, no longer works +because the point from which the path is constructed has been detached +from the mount tree. + +The actual problem with autofs is that it can't reconnect to existing +mounts. Immediately one thinks of just adding the ability to remount +autofs file systems would solve it, but alas, that can't work. This is +because autofs direct mounts and the implementation of "on demand mount +and expire" of nested mount trees have the file system mounted directly +on top of the mount trigger directory dentry. + +For example, there are two types of automount maps, direct (in the kernel +module source you will see a third type called an offset, which is just +a direct mount in disguise) and indirect. + +Here is a master map with direct and indirect map entries: + +/- /etc/auto.direct +/test /etc/auto.indirect + +and the corresponding map files: + +/etc/auto.direct: + +/automount/dparse/g6 budgie:/autofs/export1 +/automount/dparse/g1 shark:/autofs/export1 +and so on. + +/etc/auto.indirect: + +g1 shark:/autofs/export1 +g6 budgie:/autofs/export1 +and so on. + +For the above indirect map an autofs file system is mounted on /test and +mounts are triggered for each sub-directory key by the inode lookup +operation. So we see a mount of shark:/autofs/export1 on /test/g1, for +example. + +The way that direct mounts are handled is by making an autofs mount on +each full path, such as /automount/dparse/g1, and using it as a mount +trigger. So when we walk on the path we mount shark:/autofs/export1 "on +top of this mount point". Since these are always directories we can +use the follow_link inode operation to trigger the mount. + +But, each entry in direct and indirect maps can have offsets (making +them multi-mount map entries). + +For example, an indirect mount map entry could also be: + +g1 \ + / shark:/autofs/export5/testing/test \ + /s1 shark:/autofs/export/testing/test/s1 \ + /s2 shark:/autofs/export5/testing/test/s2 \ + /s1/ss1 shark:/autofs/export1 \ + /s2/ss2 shark:/autofs/export2 + +and a similarly a direct mount map entry could also be: + +/automount/dparse/g1 \ + / shark:/autofs/export5/testing/test \ + /s1 shark:/autofs/export/testing/test/s1 \ + /s2 shark:/autofs/export5/testing/test/s2 \ + /s1/ss1 shark:/autofs/export2 \ + /s2/ss2 shark:/autofs/export2 + +One of the issues with version 4 of autofs was that, when mounting an +entry with a large number of offsets, possibly with nesting, we needed +to mount and umount all of the offsets as a single unit. Not really a +problem, except for people with a large number of offsets in map entries. +This mechanism is used for the well known "hosts" map and we have seen +cases (in 2.4) where the available number of mounts are exhausted or +where the number of privileged ports available is exhausted. + +In version 5 we mount only as we go down the tree of offsets and +similarly for expiring them which resolves the above problem. There is +somewhat more detail to the implementation but it isn't needed for the +sake of the problem explanation. The one important detail is that these +offsets are implemented using the same mechanism as the direct mounts +above and so the mount points can be covered by a mount. + +The current autofs implementation uses an ioctl file descriptor opened +on the mount point for control operations. The references held by the +descriptor are accounted for in checks made to determine if a mount is +in use and is also used to access autofs file system information held +in the mount super block. So the use of a file handle needs to be +retained. + + +The Solution +============ + +To be able to restart autofs leaving existing direct, indirect and +offset mounts in place we need to be able to obtain a file handle +for these potentially covered autofs mount points. Rather than just +implement an isolated operation it was decided to re-implement the +existing ioctl interface and add new operations to provide this +functionality. + +In addition, to be able to reconstruct a mount tree that has busy mounts, +the uid and gid of the last user that triggered the mount needs to be +available because these can be used as macro substitution variables in +autofs maps. They are recorded at mount request time and an operation +has been added to retrieve them. + +Since we're re-implementing the control interface, a couple of other +problems with the existing interface have been addressed. First, when +a mount or expire operation completes a status is returned to the +kernel by either a "send ready" or a "send fail" operation. The +"send fail" operation of the ioctl interface could only ever send +ENOENT so the re-implementation allows user space to send an actual +status. Another expensive operation in user space, for those using +very large maps, is discovering if a mount is present. Usually this +involves scanning /proc/mounts and since it needs to be done quite +often it can introduce significant overhead when there are many entries +in the mount table. An operation to lookup the mount status of a mount +point dentry (covered or not) has also been added. + +Current kernel development policy recommends avoiding the use of the +ioctl mechanism in favor of systems such as Netlink. An implementation +using this system was attempted to evaluate its suitability and it was +found to be inadequate, in this case. The Generic Netlink system was +used for this as raw Netlink would lead to a significant increase in +complexity. There's no question that the Generic Netlink system is an +elegant solution for common case ioctl functions but it's not a complete +replacement probably because its primary purpose in life is to be a +message bus implementation rather than specifically an ioctl replacement. +While it would be possible to work around this there is one concern +that lead to the decision to not use it. This is that the autofs +expire in the daemon has become far to complex because umount +candidates are enumerated, almost for no other reason than to "count" +the number of times to call the expire ioctl. This involves scanning +the mount table which has proved to be a big overhead for users with +large maps. The best way to improve this is try and get back to the +way the expire was done long ago. That is, when an expire request is +issued for a mount (file handle) we should continually call back to +the daemon until we can't umount any more mounts, then return the +appropriate status to the daemon. At the moment we just expire one +mount at a time. A Generic Netlink implementation would exclude this +possibility for future development due to the requirements of the +message bus architecture. + + +autofs Miscellaneous Device mount control interface +==================================================== + +The control interface is opening a device node, typically /dev/autofs. + +All the ioctls use a common structure to pass the needed parameter +information and return operation results: + +struct autofs_dev_ioctl { + __u32 ver_major; + __u32 ver_minor; + __u32 size; /* total size of data passed in + * including this struct */ + __s32 ioctlfd; /* automount command fd */ + + /* Command parameters */ + union { + struct args_protover protover; + struct args_protosubver protosubver; + struct args_openmount openmount; + struct args_ready ready; + struct args_fail fail; + struct args_setpipefd setpipefd; + struct args_timeout timeout; + struct args_requester requester; + struct args_expire expire; + struct args_askumount askumount; + struct args_ismountpoint ismountpoint; + }; + + char path[0]; +}; + +The ioctlfd field is a mount point file descriptor of an autofs mount +point. It is returned by the open call and is used by all calls except +the check for whether a given path is a mount point, where it may +optionally be used to check a specific mount corresponding to a given +mount point file descriptor, and when requesting the uid and gid of the +last successful mount on a directory within the autofs file system. + +The union is used to communicate parameters and results of calls made +as described below. + +The path field is used to pass a path where it is needed and the size field +is used account for the increased structure length when translating the +structure sent from user space. + +This structure can be initialized before setting specific fields by using +the void function call init_autofs_dev_ioctl(struct autofs_dev_ioctl *). + +All of the ioctls perform a copy of this structure from user space to +kernel space and return -EINVAL if the size parameter is smaller than +the structure size itself, -ENOMEM if the kernel memory allocation fails +or -EFAULT if the copy itself fails. Other checks include a version check +of the compiled in user space version against the module version and a +mismatch results in a -EINVAL return. If the size field is greater than +the structure size then a path is assumed to be present and is checked to +ensure it begins with a "/" and is NULL terminated, otherwise -EINVAL is +returned. Following these checks, for all ioctl commands except +AUTOFS_DEV_IOCTL_VERSION_CMD, AUTOFS_DEV_IOCTL_OPENMOUNT_CMD and +AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD the ioctlfd is validated and if it is +not a valid descriptor or doesn't correspond to an autofs mount point +an error of -EBADF, -ENOTTY or -EINVAL (not an autofs descriptor) is +returned. + + +The ioctls +========== + +An example of an implementation which uses this interface can be seen +in autofs version 5.0.4 and later in file lib/dev-ioctl-lib.c of the +distribution tar available for download from kernel.org in directory +/pub/linux/daemons/autofs/v5. + +The device node ioctl operations implemented by this interface are: + + +AUTOFS_DEV_IOCTL_VERSION +------------------------ + +Get the major and minor version of the autofs device ioctl kernel module +implementation. It requires an initialized struct autofs_dev_ioctl as an +input parameter and sets the version information in the passed in structure. +It returns 0 on success or the error -EINVAL if a version mismatch is +detected. + + +AUTOFS_DEV_IOCTL_PROTOVER_CMD and AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD +------------------------------------------------------------------ + +Get the major and minor version of the autofs protocol version understood +by loaded module. This call requires an initialized struct autofs_dev_ioctl +with the ioctlfd field set to a valid autofs mount point descriptor +and sets the requested version number in version field of struct args_protover +or sub_version field of struct args_protosubver. These commands return +0 on success or one of the negative error codes if validation fails. + + +AUTOFS_DEV_IOCTL_OPENMOUNT and AUTOFS_DEV_IOCTL_CLOSEMOUNT +---------------------------------------------------------- + +Obtain and release a file descriptor for an autofs managed mount point +path. The open call requires an initialized struct autofs_dev_ioctl with +the path field set and the size field adjusted appropriately as well +as the devid field of struct args_openmount set to the device number of +the autofs mount. The device number can be obtained from the mount options +shown in /proc/mounts. The close call requires an initialized struct +autofs_dev_ioct with the ioctlfd field set to the descriptor obtained +from the open call. The release of the file descriptor can also be done +with close(2) so any open descriptors will also be closed at process exit. +The close call is included in the implemented operations largely for +completeness and to provide for a consistent user space implementation. + + +AUTOFS_DEV_IOCTL_READY_CMD and AUTOFS_DEV_IOCTL_FAIL_CMD +-------------------------------------------------------- + +Return mount and expire result status from user space to the kernel. +Both of these calls require an initialized struct autofs_dev_ioctl +with the ioctlfd field set to the descriptor obtained from the open +call and the token field of struct args_ready or struct args_fail set +to the wait queue token number, received by user space in the foregoing +mount or expire request. The status field of struct args_fail is set to +the errno of the operation. It is set to 0 on success. + + +AUTOFS_DEV_IOCTL_SETPIPEFD_CMD +------------------------------ + +Set the pipe file descriptor used for kernel communication to the daemon. +Normally this is set at mount time using an option but when reconnecting +to a existing mount we need to use this to tell the autofs mount about +the new kernel pipe descriptor. In order to protect mounts against +incorrectly setting the pipe descriptor we also require that the autofs +mount be catatonic (see next call). + +The call requires an initialized struct autofs_dev_ioctl with the +ioctlfd field set to the descriptor obtained from the open call and +the pipefd field of struct args_setpipefd set to descriptor of the pipe. +On success the call also sets the process group id used to identify the +controlling process (eg. the owning automount(8) daemon) to the process +group of the caller. + + +AUTOFS_DEV_IOCTL_CATATONIC_CMD +------------------------------ + +Make the autofs mount point catatonic. The autofs mount will no longer +issue mount requests, the kernel communication pipe descriptor is released +and any remaining waits in the queue released. + +The call requires an initialized struct autofs_dev_ioctl with the +ioctlfd field set to the descriptor obtained from the open call. + + +AUTOFS_DEV_IOCTL_TIMEOUT_CMD +---------------------------- + +Set the expire timeout for mounts within an autofs mount point. + +The call requires an initialized struct autofs_dev_ioctl with the +ioctlfd field set to the descriptor obtained from the open call. + + +AUTOFS_DEV_IOCTL_REQUESTER_CMD +------------------------------ + +Return the uid and gid of the last process to successfully trigger a the +mount on the given path dentry. + +The call requires an initialized struct autofs_dev_ioctl with the path +field set to the mount point in question and the size field adjusted +appropriately. Upon return the uid field of struct args_requester contains +the uid and gid field the gid. + +When reconstructing an autofs mount tree with active mounts we need to +re-connect to mounts that may have used the original process uid and +gid (or string variations of them) for mount lookups within the map entry. +This call provides the ability to obtain this uid and gid so they may be +used by user space for the mount map lookups. + + +AUTOFS_DEV_IOCTL_EXPIRE_CMD +--------------------------- + +Issue an expire request to the kernel for an autofs mount. Typically +this ioctl is called until no further expire candidates are found. + +The call requires an initialized struct autofs_dev_ioctl with the +ioctlfd field set to the descriptor obtained from the open call. In +addition an immediate expire, independent of the mount timeout, can be +requested by setting the how field of struct args_expire to 1. If no +expire candidates can be found the ioctl returns -1 with errno set to +EAGAIN. + +This call causes the kernel module to check the mount corresponding +to the given ioctlfd for mounts that can be expired, issues an expire +request back to the daemon and waits for completion. + +AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD +------------------------------ + +Checks if an autofs mount point is in use. + +The call requires an initialized struct autofs_dev_ioctl with the +ioctlfd field set to the descriptor obtained from the open call and +it returns the result in the may_umount field of struct args_askumount, +1 for busy and 0 otherwise. + + +AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD +--------------------------------- + +Check if the given path is a mountpoint. + +The call requires an initialized struct autofs_dev_ioctl. There are two +possible variations. Both use the path field set to the path of the mount +point to check and the size field adjusted appropriately. One uses the +ioctlfd field to identify a specific mount point to check while the other +variation uses the path and optionally in.type field of struct args_ismountpoint +set to an autofs mount type. The call returns 1 if this is a mount point +and sets out.devid field to the device number of the mount and out.magic +field to the relevant super block magic number (described below) or 0 if +it isn't a mountpoint. In both cases the the device number (as returned +by new_encode_dev()) is returned in out.devid field. + +If supplied with a file descriptor we're looking for a specific mount, +not necessarily at the top of the mounted stack. In this case the path +the descriptor corresponds to is considered a mountpoint if it is itself +a mountpoint or contains a mount, such as a multi-mount without a root +mount. In this case we return 1 if the descriptor corresponds to a mount +point and and also returns the super magic of the covering mount if there +is one or 0 if it isn't a mountpoint. + +If a path is supplied (and the ioctlfd field is set to -1) then the path +is looked up and is checked to see if it is the root of a mount. If a +type is also given we are looking for a particular autofs mount and if +a match isn't found a fail is returned. If the the located path is the +root of a mount 1 is returned along with the super magic of the mount +or 0 otherwise. diff --git a/Documentation/filesystems/autofs.txt b/Documentation/filesystems/autofs.txt new file mode 100644 index 000000000000..373ad25852d3 --- /dev/null +++ b/Documentation/filesystems/autofs.txt @@ -0,0 +1,529 @@ + + + + +autofs - how it works +===================== + +Purpose +------- + +The goal of autofs is to provide on-demand mounting and race free +automatic unmounting of various other filesystems. This provides two +key advantages: + +1. There is no need to delay boot until all filesystems that + might be needed are mounted. Processes that try to access those + slow filesystems might be delayed but other processes can + continue freely. This is particularly important for + network filesystems (e.g. NFS) or filesystems stored on + media with a media-changing robot. + +2. The names and locations of filesystems can be stored in + a remote database and can change at any time. The content + in that data base at the time of access will be used to provide + a target for the access. The interpretation of names in the + filesystem can even be programmatic rather than database-backed, + allowing wildcards for example, and can vary based on the user who + first accessed a name. + +Context +------- + +The "autofs" filesystem module is only one part of an autofs system. +There also needs to be a user-space program which looks up names +and mounts filesystems. This will often be the "automount" program, +though other tools including "systemd" can make use of "autofs". +This document describes only the kernel module and the interactions +required with any user-space program. Subsequent text refers to this +as the "automount daemon" or simply "the daemon". + +"autofs" is a Linux kernel module with provides the "autofs" +filesystem type. Several "autofs" filesystems can be mounted and they +can each be managed separately, or all managed by the same daemon. + +Content +------- + +An autofs filesystem can contain 3 sorts of objects: directories, +symbolic links and mount traps. Mount traps are directories with +extra properties as described in the next section. + +Objects can only be created by the automount daemon: symlinks are +created with a regular `symlink` system call, while directories and +mount traps are created with `mkdir`. The determination of whether a +directory should be a mount trap or not is quite _ad hoc_, largely for +historical reasons, and is determined in part by the +*direct*/*indirect*/*offset* mount options, and the *maxproto* mount option. + +If neither the *direct* or *offset* mount options are given (so the +mount is considered to be *indirect*), then the root directory is +always a regular directory, otherwise it is a mount trap when it is +empty and a regular directory when not empty. Note that *direct* and +*offset* are treated identically so a concise summary is that the root +directory is a mount trap only if the filesystem is mounted *direct* +and the root is empty. + +Directories created in the root directory are mount traps only if the +filesystem is mounted *indirect* and they are empty. + +Directories further down the tree depend on the *maxproto* mount +option and particularly whether it is less than five or not. +When *maxproto* is five, no directories further down the +tree are ever mount traps, they are always regular directories. When +the *maxproto* is four (or three), these directories are mount traps +precisely when they are empty. + +So: non-empty (i.e. non-leaf) directories are never mount traps. Empty +directories are sometimes mount traps, and sometimes not depending on +where in the tree they are (root, top level, or lower), the *maxproto*, +and whether the mount was *indirect* or not. + +Mount Traps +--------------- + +A core element of the implementation of autofs is the Mount Traps +which are provided by the Linux VFS. Any directory provided by a +filesystem can be designated as a trap. This involves two separate +features that work together to allow autofs to do its job. + +**DCACHE_NEED_AUTOMOUNT** + +If a dentry has the DCACHE_NEED_AUTOMOUNT flag set (which gets set if +the inode has S_AUTOMOUNT set, or can be set directly) then it is +(potentially) a mount trap. Any access to this directory beyond a +"`stat`" will (normally) cause the `d_op->d_automount()` dentry operation +to be called. The task of this method is to find the filesystem that +should be mounted on the directory and to return it. The VFS is +responsible for actually mounting the root of this filesystem on the +directory. + +autofs doesn't find the filesystem itself but sends a message to the +automount daemon asking it to find and mount the filesystem. The +autofs `d_automount` method then waits for the daemon to report that +everything is ready. It will then return "`NULL`" indicating that the +mount has already happened. The VFS doesn't try to mount anything but +follows down the mount that is already there. + +This functionality is sufficient for some users of mount traps such +as NFS which creates traps so that mountpoints on the server can be +reflected on the client. However it is not sufficient for autofs. As +mounting onto a directory is considered to be "beyond a `stat`", the +automount daemon would not be able to mount a filesystem on the 'trap' +directory without some way to avoid getting caught in the trap. For +that purpose there is another flag. + +**DCACHE_MANAGE_TRANSIT** + +If a dentry has DCACHE_MANAGE_TRANSIT set then two very different but +related behaviors are invoked, both using the `d_op->d_manage()` +dentry operation. + +Firstly, before checking to see if any filesystem is mounted on the +directory, d_manage() will be called with the `rcu_walk` parameter set +to `false`. It may return one of three things: + +- A return value of zero indicates that there is nothing special + about this dentry and normal checks for mounts and automounts + should proceed. + + autofs normally returns zero, but first waits for any + expiry (automatic unmounting of the mounted filesystem) to + complete. This avoids races. + +- A return value of `-EISDIR` tells the VFS to ignore any mounts + on the directory and to not consider calling `->d_automount()`. + This effectively disables the **DCACHE_NEED_AUTOMOUNT** flag + causing the directory not be a mount trap after all. + + autofs returns this if it detects that the process performing the + lookup is the automount daemon and that the mount has been + requested but has not yet completed. How it determines this is + discussed later. This allows the automount daemon not to get + caught in the mount trap. + + There is a subtlety here. It is possible that a second autofs + filesystem can be mounted below the first and for both of them to + be managed by the same daemon. For the daemon to be able to mount + something on the second it must be able to "walk" down past the + first. This means that d_manage cannot *always* return -EISDIR for + the automount daemon. It must only return it when a mount has + been requested, but has not yet completed. + + `d_manage` also returns `-EISDIR` if the dentry shouldn't be a + mount trap, either because it is a symbolic link or because it is + not empty. + +- Any other negative value is treated as an error and returned + to the caller. + + autofs can return + + - -ENOENT if the automount daemon failed to mount anything, + - -ENOMEM if it ran out of memory, + - -EINTR if a signal arrived while waiting for expiry to + complete + - or any other error sent down by the automount daemon. + + +The second use case only occurs during an "RCU-walk" and so `rcu_walk` +will be set. + +An RCU-walk is a fast and lightweight process for walking down a +filename path (i.e. it is like running on tip-toes). RCU-walk cannot +cope with all situations so when it finds a difficulty it falls back +to "REF-walk", which is slower but more robust. + +RCU-walk will never call `->d_automount`; the filesystems must already +be mounted or RCU-walk cannot handle the path. +To determine if a mount-trap is safe for RCU-walk mode it calls +`->d_manage()` with `rcu_walk` set to `true`. + +In this case `d_manage()` must avoid blocking and should avoid taking +spinlocks if at all possible. Its sole purpose is to determine if it +would be safe to follow down into any mounted directory and the only +reason that it might not be is if an expiry of the mount is +underway. + +In the `rcu_walk` case, `d_manage()` cannot return -EISDIR to tell the +VFS that this is a directory that doesn't require d_automount. If +`rcu_walk` sees a dentry with DCACHE_NEED_AUTOMOUNT set but nothing +mounted, it *will* fall back to REF-walk. `d_manage()` cannot make the +VFS remain in RCU-walk mode, but can only tell it to get out of +RCU-walk mode by returning `-ECHILD`. + +So `d_manage()`, when called with `rcu_walk` set, should either return +-ECHILD if there is any reason to believe it is unsafe to end the +mounted filesystem, and otherwise should return 0. + +autofs will return `-ECHILD` if an expiry of the filesystem has been +initiated or is being considered, otherwise it returns 0. + + +Mountpoint expiry +----------------- + +The VFS has a mechanism for automatically expiring unused mounts, +much as it can expire any unused dentry information from the dcache. +This is guided by the MNT_SHRINKABLE flag. This only applies to +mounts that were created by `d_automount()` returning a filesystem to be +mounted. As autofs doesn't return such a filesystem but leaves the +mounting to the automount daemon, it must involve the automount daemon +in unmounting as well. This also means that autofs has more control +of expiry. + +The VFS also supports "expiry" of mounts using the MNT_EXPIRE flag to +the `umount` system call. Unmounting with MNT_EXPIRE will fail unless +a previous attempt had been made, and the filesystem has been inactive +and untouched since that previous attempt. autofs does not depend on +this but has its own internal tracking of whether filesystems were +recently used. This allows individual names in the autofs directory +to expire separately. + +With version 4 of the protocol, the automount daemon can try to +unmount any filesystems mounted on the autofs filesystem or remove any +symbolic links or empty directories any time it likes. If the unmount +or removal is successful the filesystem will be returned to the state +it was before the mount or creation, so that any access of the name +will trigger normal auto-mount processing. In particlar, `rmdir` and +`unlink` do not leave negative entries in the dcache as a normal +filesystem would, so an attempt to access a recently-removed object is +passed to autofs for handling. + +With version 5, this is not safe except for unmounting from top-level +directories. As lower-level directories are never mount traps, other +processes will see an empty directory as soon as the filesystem is +unmounted. So it is generally safest to use the autofs expiry +protocol described below. + +Normally the daemon only wants to remove entries which haven't been +used for a while. For this purpose autofs maintains a "`last_used`" +time stamp on each directory or symlink. For symlinks it genuinely +does record the last time the symlink was "used" or followed to find +out where it points to. For directories the field is a slight +misnomer. It actually records the last time that autofs checked if +the directory or one of its descendents was busy and found that it +was. This is just as useful and doesn't require updating the field so +often. + +The daemon is able to ask autofs if anything is due to be expired, +using an `ioctl` as discussed later. For a *direct* mount, autofs +considers if the entire mount-tree can be unmounted or not. For an +*indirect* mount, autofs considers each of the names in the top level +directory to determine if any of those can be unmounted and cleaned +up. + +There is an option with indirect mounts to consider each of the leaves +that has been mounted on instead of considering the top-level names. +This is intended for compatability with version 4 of autofs and should +be considered as deprecated. + +When autofs considers a directory it checks the `last_used` time and +compares it with the "timeout" value set when the filesystem was +mounted, though this check is ignored in some cases. It also checks if +the directory or anything below it is in use. For symbolic links, +only the `last_used` time is ever considered. + +If both appear to support expiring the directory or symlink, an action +is taken. + +There are two ways to ask autofs to consider expiry. The first is to +use the **AUTOFS_IOC_EXPIRE** ioctl. This only works for indirect +mounts. If it finds something in the root directory to expire it will +return the name of that thing. Once a name has been returned the +automount daemon needs to unmount any filesystems mounted below the +name normally. As described above, this is unsafe for non-toplevel +mounts in a version-5 autofs. For this reason the current `automountd` +does not use this ioctl. + +The second mechanism uses either the **AUTOFS_DEV_IOCTL_EXPIRE_CMD** or +the **AUTOFS_IOC_EXPIRE_MULTI** ioctl. This will work for both direct and +indirect mounts. If it selects an object to expire, it will notify +the daemon using the notification mechanism described below. This +will block until the daemon acknowledges the expiry notification. +This implies that the "`EXPIRE`" ioctl must be sent from a different +thread than the one which handles notification. + +While the ioctl is blocking, the entry is marked as "expiring" and +`d_manage` will block until the daemon affirms that the unmount has +completed (together with removing any directories that might have been +necessary), or has been aborted. + +Communicating with autofs: detecting the daemon +----------------------------------------------- + +There are several forms of communication between the automount daemon +and the filesystem. As we have already seen, the daemon can create and +remove directories and symlinks using normal filesystem operations. +autofs knows whether a process requesting some operation is the daemon +or not based on its process-group id number (see getpgid(1)). + +When an autofs filesystem is mounted the pgid of the mounting +processes is recorded unless the "pgrp=" option is given, in which +case that number is recorded instead. Any request arriving from a +process in that process group is considered to come from the daemon. +If the daemon ever has to be stopped and restarted a new pgid can be +provided through an ioctl as will be described below. + +Communicating with autofs: the event pipe +----------------------------------------- + +When an autofs filesystem is mounted, the 'write' end of a pipe must +be passed using the 'fd=' mount option. autofs will write +notification messages to this pipe for the daemon to respond to. +For version 5, the format of the message is: + + struct autofs_v5_packet { + int proto_version; /* Protocol version */ + int type; /* Type of packet */ + autofs_wqt_t wait_queue_token; + __u32 dev; + __u64 ino; + __u32 uid; + __u32 gid; + __u32 pid; + __u32 tgid; + __u32 len; + char name[NAME_MAX+1]; + }; + +where the type is one of + + autofs_ptype_missing_indirect + autofs_ptype_expire_indirect + autofs_ptype_missing_direct + autofs_ptype_expire_direct + +so messages can indicate that a name is missing (something tried to +access it but it isn't there) or that it has been selected for expiry. + +The pipe will be set to "packet mode" (equivalent to passing +`O_DIRECT`) to _pipe2(2)_ so that a read from the pipe will return at +most one packet, and any unread portion of a packet will be discarded. + +The `wait_queue_token` is a unique number which can identify a +particular request to be acknowledged. When a message is sent over +the pipe the affected dentry is marked as either "active" or +"expiring" and other accesses to it block until the message is +acknowledged using one of the ioctls below and the relevant +`wait_queue_token`. + +Communicating with autofs: root directory ioctls +------------------------------------------------ + +The root directory of an autofs filesystem will respond to a number of +ioctls. The process issuing the ioctl must have the CAP_SYS_ADMIN +capability, or must be the automount daemon. + +The available ioctl commands are: + +- **AUTOFS_IOC_READY**: a notification has been handled. The argument + to the ioctl command is the "wait_queue_token" number + corresponding to the notification being acknowledged. +- **AUTOFS_IOC_FAIL**: similar to above, but indicates failure with + the error code `ENOENT`. +- **AUTOFS_IOC_CATATONIC**: Causes the autofs to enter "catatonic" + mode meaning that it stops sending notifications to the daemon. + This mode is also entered if a write to the pipe fails. +- **AUTOFS_IOC_PROTOVER**: This returns the protocol version in use. +- **AUTOFS_IOC_PROTOSUBVER**: Returns the protocol sub-version which + is really a version number for the implementation. It is + currently 2. +- **AUTOFS_IOC_SETTIMEOUT**: This passes a pointer to an unsigned + long. The value is used to set the timeout for expiry, and + the current timeout value is stored back through the pointer. +- **AUTOFS_IOC_ASKUMOUNT**: Returns, in the pointed-to `int`, 1 if + the filesystem could be unmounted. This is only a hint as + the situation could change at any instant. This call can be + use to avoid a more expensive full unmount attempt. +- **AUTOFS_IOC_EXPIRE**: as described above, this asks if there is + anything suitable to expire. A pointer to a packet: + + struct autofs_packet_expire_multi { + int proto_version; /* Protocol version */ + int type; /* Type of packet */ + autofs_wqt_t wait_queue_token; + int len; + char name[NAME_MAX+1]; + }; + + is required. This is filled in with the name of something + that can be unmounted or removed. If nothing can be expired, + `errno` is set to `EAGAIN`. Even though a `wait_queue_token` + is present in the structure, no "wait queue" is established + and no acknowledgment is needed. +- **AUTOFS_IOC_EXPIRE_MULTI**: This is similar to + **AUTOFS_IOC_EXPIRE** except that it causes notification to be + sent to the daemon, and it blocks until the daemon acknowledges. + The argument is an integer which can contain two different flags. + + **AUTOFS_EXP_IMMEDIATE** causes `last_used` time to be ignored + and objects are expired if the are not in use. + + **AUTOFS_EXP_LEAVES** will select a leaf rather than a top-level + name to expire. This is only safe when *maxproto* is 4. + +Communicating with autofs: char-device ioctls +--------------------------------------------- + +It is not always possible to open the root of an autofs filesystem, +particularly a *direct* mounted filesystem. If the automount daemon +is restarted there is no way for it to regain control of existing +mounts using any of the above communication channels. To address this +need there is a "miscellaneous" character device (major 10, minor 235) +which can be used to communicate directly with the autofs filesystem. +It requires CAP_SYS_ADMIN for access. + +The `ioctl`s that can be used on this device are described in a separate +document `autofs-mount-control.txt`, and are summarized briefly here. +Each ioctl is passed a pointer to an `autofs_dev_ioctl` structure: + + struct autofs_dev_ioctl { + __u32 ver_major; + __u32 ver_minor; + __u32 size; /* total size of data passed in + * including this struct */ + __s32 ioctlfd; /* automount command fd */ + + /* Command parameters */ + union { + struct args_protover protover; + struct args_protosubver protosubver; + struct args_openmount openmount; + struct args_ready ready; + struct args_fail fail; + struct args_setpipefd setpipefd; + struct args_timeout timeout; + struct args_requester requester; + struct args_expire expire; + struct args_askumount askumount; + struct args_ismountpoint ismountpoint; + }; + + char path[0]; + }; + +For the **OPEN_MOUNT** and **IS_MOUNTPOINT** commands, the target +filesystem is identified by the `path`. All other commands identify +the filesystem by the `ioctlfd` which is a file descriptor open on the +root, and which can be returned by **OPEN_MOUNT**. + +The `ver_major` and `ver_minor` are in/out parameters which check that +the requested version is supported, and report the maximum version +that the kernel module can support. + +Commands are: + +- **AUTOFS_DEV_IOCTL_VERSION_CMD**: does nothing, except validate and + set version numbers. +- **AUTOFS_DEV_IOCTL_OPENMOUNT_CMD**: return an open file descriptor + on the root of an autofs filesystem. The filesystem is identified + by name and device number, which is stored in `openmount.devid`. + Device numbers for existing filesystems can be found in + `/proc/self/mountinfo`. +- **AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD**: same as `close(ioctlfd)`. +- **AUTOFS_DEV_IOCTL_SETPIPEFD_CMD**: if the filesystem is in + catatonic mode, this can provide the write end of a new pipe + in `setpipefd.pipefd` to re-establish communication with a daemon. + The process group of the calling process is used to identify the + daemon. +- **AUTOFS_DEV_IOCTL_REQUESTER_CMD**: `path` should be a + name within the filesystem that has been auto-mounted on. + On successful return, `requester.uid` and `requester.gid` will be + the UID and GID of the process which triggered that mount. +- **AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD**: Check if path is a + mountpoint of a particular type - see separate documentation for + details. +- **AUTOFS_DEV_IOCTL_PROTOVER_CMD**: +- **AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD**: +- **AUTOFS_DEV_IOCTL_READY_CMD**: +- **AUTOFS_DEV_IOCTL_FAIL_CMD**: +- **AUTOFS_DEV_IOCTL_CATATONIC_CMD**: +- **AUTOFS_DEV_IOCTL_TIMEOUT_CMD**: +- **AUTOFS_DEV_IOCTL_EXPIRE_CMD**: +- **AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD**: These all have the same + function as the similarly named **AUTOFS_IOC** ioctls, except + that **FAIL** can be given an explicit error number in `fail.status` + instead of assuming `ENOENT`, and this **EXPIRE** command + corresponds to **AUTOFS_IOC_EXPIRE_MULTI**. + +Catatonic mode +-------------- + +As mentioned, an autofs mount can enter "catatonic" mode. This +happens if a write to the notification pipe fails, or if it is +explicitly requested by an `ioctl`. + +When entering catatonic mode, the pipe is closed and any pending +notifications are acknowledged with the error `ENOENT`. + +Once in catatonic mode attempts to access non-existing names will +result in `ENOENT` while attempts to access existing directories will +be treated in the same way as if they came from the daemon, so mount +traps will not fire. + +When the filesystem is mounted a _uid_ and _gid_ can be given which +set the ownership of directories and symbolic links. When the +filesystem is in catatonic mode, any process with a matching UID can +create directories or symlinks in the root directory, but not in other +directories. + +Catatonic mode can only be left via the +**AUTOFS_DEV_IOCTL_OPENMOUNT_CMD** ioctl on the `/dev/autofs`. + +autofs, name spaces, and shared mounts +-------------------------------------- + +With bind mounts and name spaces it is possible for an autofs +filesystem to appear at multiple places in one or more filesystem +name spaces. For this to work sensibly, the autofs filesystem should +always be mounted "shared". e.g. + +> `mount --make-shared /autofs/mount/point` + +The automount daemon is only able to manage a single mount location for +an autofs filesystem and if mounts on that are not 'shared', other +locations will not behave as expected. In particular access to those +other locations will likely result in the `ELOOP` error + +> Too many levels of symbolic links diff --git a/Documentation/filesystems/autofs4-mount-control.txt b/Documentation/filesystems/autofs4-mount-control.txt deleted file mode 100644 index e5177cb31a04..000000000000 --- a/Documentation/filesystems/autofs4-mount-control.txt +++ /dev/null @@ -1,407 +0,0 @@ - -Miscellaneous Device control operations for the autofs4 kernel module -==================================================================== - -The problem -=========== - -There is a problem with active restarts in autofs (that is to say -restarting autofs when there are busy mounts). - -During normal operation autofs uses a file descriptor opened on the -directory that is being managed in order to be able to issue control -operations. Using a file descriptor gives ioctl operations access to -autofs specific information stored in the super block. The operations -are things such as setting an autofs mount catatonic, setting the -expire timeout and requesting expire checks. As is explained below, -certain types of autofs triggered mounts can end up covering an autofs -mount itself which prevents us being able to use open(2) to obtain a -file descriptor for these operations if we don't already have one open. - -Currently autofs uses "umount -l" (lazy umount) to clear active mounts -at restart. While using lazy umount works for most cases, anything that -needs to walk back up the mount tree to construct a path, such as -getcwd(2) and the proc file system /proc//cwd, no longer works -because the point from which the path is constructed has been detached -from the mount tree. - -The actual problem with autofs is that it can't reconnect to existing -mounts. Immediately one thinks of just adding the ability to remount -autofs file systems would solve it, but alas, that can't work. This is -because autofs direct mounts and the implementation of "on demand mount -and expire" of nested mount trees have the file system mounted directly -on top of the mount trigger directory dentry. - -For example, there are two types of automount maps, direct (in the kernel -module source you will see a third type called an offset, which is just -a direct mount in disguise) and indirect. - -Here is a master map with direct and indirect map entries: - -/- /etc/auto.direct -/test /etc/auto.indirect - -and the corresponding map files: - -/etc/auto.direct: - -/automount/dparse/g6 budgie:/autofs/export1 -/automount/dparse/g1 shark:/autofs/export1 -and so on. - -/etc/auto.indirect: - -g1 shark:/autofs/export1 -g6 budgie:/autofs/export1 -and so on. - -For the above indirect map an autofs file system is mounted on /test and -mounts are triggered for each sub-directory key by the inode lookup -operation. So we see a mount of shark:/autofs/export1 on /test/g1, for -example. - -The way that direct mounts are handled is by making an autofs mount on -each full path, such as /automount/dparse/g1, and using it as a mount -trigger. So when we walk on the path we mount shark:/autofs/export1 "on -top of this mount point". Since these are always directories we can -use the follow_link inode operation to trigger the mount. - -But, each entry in direct and indirect maps can have offsets (making -them multi-mount map entries). - -For example, an indirect mount map entry could also be: - -g1 \ - / shark:/autofs/export5/testing/test \ - /s1 shark:/autofs/export/testing/test/s1 \ - /s2 shark:/autofs/export5/testing/test/s2 \ - /s1/ss1 shark:/autofs/export1 \ - /s2/ss2 shark:/autofs/export2 - -and a similarly a direct mount map entry could also be: - -/automount/dparse/g1 \ - / shark:/autofs/export5/testing/test \ - /s1 shark:/autofs/export/testing/test/s1 \ - /s2 shark:/autofs/export5/testing/test/s2 \ - /s1/ss1 shark:/autofs/export2 \ - /s2/ss2 shark:/autofs/export2 - -One of the issues with version 4 of autofs was that, when mounting an -entry with a large number of offsets, possibly with nesting, we needed -to mount and umount all of the offsets as a single unit. Not really a -problem, except for people with a large number of offsets in map entries. -This mechanism is used for the well known "hosts" map and we have seen -cases (in 2.4) where the available number of mounts are exhausted or -where the number of privileged ports available is exhausted. - -In version 5 we mount only as we go down the tree of offsets and -similarly for expiring them which resolves the above problem. There is -somewhat more detail to the implementation but it isn't needed for the -sake of the problem explanation. The one important detail is that these -offsets are implemented using the same mechanism as the direct mounts -above and so the mount points can be covered by a mount. - -The current autofs implementation uses an ioctl file descriptor opened -on the mount point for control operations. The references held by the -descriptor are accounted for in checks made to determine if a mount is -in use and is also used to access autofs file system information held -in the mount super block. So the use of a file handle needs to be -retained. - - -The Solution -============ - -To be able to restart autofs leaving existing direct, indirect and -offset mounts in place we need to be able to obtain a file handle -for these potentially covered autofs mount points. Rather than just -implement an isolated operation it was decided to re-implement the -existing ioctl interface and add new operations to provide this -functionality. - -In addition, to be able to reconstruct a mount tree that has busy mounts, -the uid and gid of the last user that triggered the mount needs to be -available because these can be used as macro substitution variables in -autofs maps. They are recorded at mount request time and an operation -has been added to retrieve them. - -Since we're re-implementing the control interface, a couple of other -problems with the existing interface have been addressed. First, when -a mount or expire operation completes a status is returned to the -kernel by either a "send ready" or a "send fail" operation. The -"send fail" operation of the ioctl interface could only ever send -ENOENT so the re-implementation allows user space to send an actual -status. Another expensive operation in user space, for those using -very large maps, is discovering if a mount is present. Usually this -involves scanning /proc/mounts and since it needs to be done quite -often it can introduce significant overhead when there are many entries -in the mount table. An operation to lookup the mount status of a mount -point dentry (covered or not) has also been added. - -Current kernel development policy recommends avoiding the use of the -ioctl mechanism in favor of systems such as Netlink. An implementation -using this system was attempted to evaluate its suitability and it was -found to be inadequate, in this case. The Generic Netlink system was -used for this as raw Netlink would lead to a significant increase in -complexity. There's no question that the Generic Netlink system is an -elegant solution for common case ioctl functions but it's not a complete -replacement probably because its primary purpose in life is to be a -message bus implementation rather than specifically an ioctl replacement. -While it would be possible to work around this there is one concern -that lead to the decision to not use it. This is that the autofs -expire in the daemon has become far to complex because umount -candidates are enumerated, almost for no other reason than to "count" -the number of times to call the expire ioctl. This involves scanning -the mount table which has proved to be a big overhead for users with -large maps. The best way to improve this is try and get back to the -way the expire was done long ago. That is, when an expire request is -issued for a mount (file handle) we should continually call back to -the daemon until we can't umount any more mounts, then return the -appropriate status to the daemon. At the moment we just expire one -mount at a time. A Generic Netlink implementation would exclude this -possibility for future development due to the requirements of the -message bus architecture. - - -autofs4 Miscellaneous Device mount control interface -==================================================== - -The control interface is opening a device node, typically /dev/autofs. - -All the ioctls use a common structure to pass the needed parameter -information and return operation results: - -struct autofs_dev_ioctl { - __u32 ver_major; - __u32 ver_minor; - __u32 size; /* total size of data passed in - * including this struct */ - __s32 ioctlfd; /* automount command fd */ - - /* Command parameters */ - union { - struct args_protover protover; - struct args_protosubver protosubver; - struct args_openmount openmount; - struct args_ready ready; - struct args_fail fail; - struct args_setpipefd setpipefd; - struct args_timeout timeout; - struct args_requester requester; - struct args_expire expire; - struct args_askumount askumount; - struct args_ismountpoint ismountpoint; - }; - - char path[0]; -}; - -The ioctlfd field is a mount point file descriptor of an autofs mount -point. It is returned by the open call and is used by all calls except -the check for whether a given path is a mount point, where it may -optionally be used to check a specific mount corresponding to a given -mount point file descriptor, and when requesting the uid and gid of the -last successful mount on a directory within the autofs file system. - -The union is used to communicate parameters and results of calls made -as described below. - -The path field is used to pass a path where it is needed and the size field -is used account for the increased structure length when translating the -structure sent from user space. - -This structure can be initialized before setting specific fields by using -the void function call init_autofs_dev_ioctl(struct autofs_dev_ioctl *). - -All of the ioctls perform a copy of this structure from user space to -kernel space and return -EINVAL if the size parameter is smaller than -the structure size itself, -ENOMEM if the kernel memory allocation fails -or -EFAULT if the copy itself fails. Other checks include a version check -of the compiled in user space version against the module version and a -mismatch results in a -EINVAL return. If the size field is greater than -the structure size then a path is assumed to be present and is checked to -ensure it begins with a "/" and is NULL terminated, otherwise -EINVAL is -returned. Following these checks, for all ioctl commands except -AUTOFS_DEV_IOCTL_VERSION_CMD, AUTOFS_DEV_IOCTL_OPENMOUNT_CMD and -AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD the ioctlfd is validated and if it is -not a valid descriptor or doesn't correspond to an autofs mount point -an error of -EBADF, -ENOTTY or -EINVAL (not an autofs descriptor) is -returned. - - -The ioctls -========== - -An example of an implementation which uses this interface can be seen -in autofs version 5.0.4 and later in file lib/dev-ioctl-lib.c of the -distribution tar available for download from kernel.org in directory -/pub/linux/daemons/autofs/v5. - -The device node ioctl operations implemented by this interface are: - - -AUTOFS_DEV_IOCTL_VERSION ------------------------- - -Get the major and minor version of the autofs4 device ioctl kernel module -implementation. It requires an initialized struct autofs_dev_ioctl as an -input parameter and sets the version information in the passed in structure. -It returns 0 on success or the error -EINVAL if a version mismatch is -detected. - - -AUTOFS_DEV_IOCTL_PROTOVER_CMD and AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD ------------------------------------------------------------------- - -Get the major and minor version of the autofs4 protocol version understood -by loaded module. This call requires an initialized struct autofs_dev_ioctl -with the ioctlfd field set to a valid autofs mount point descriptor -and sets the requested version number in version field of struct args_protover -or sub_version field of struct args_protosubver. These commands return -0 on success or one of the negative error codes if validation fails. - - -AUTOFS_DEV_IOCTL_OPENMOUNT and AUTOFS_DEV_IOCTL_CLOSEMOUNT ----------------------------------------------------------- - -Obtain and release a file descriptor for an autofs managed mount point -path. The open call requires an initialized struct autofs_dev_ioctl with -the path field set and the size field adjusted appropriately as well -as the devid field of struct args_openmount set to the device number of -the autofs mount. The device number can be obtained from the mount options -shown in /proc/mounts. The close call requires an initialized struct -autofs_dev_ioct with the ioctlfd field set to the descriptor obtained -from the open call. The release of the file descriptor can also be done -with close(2) so any open descriptors will also be closed at process exit. -The close call is included in the implemented operations largely for -completeness and to provide for a consistent user space implementation. - - -AUTOFS_DEV_IOCTL_READY_CMD and AUTOFS_DEV_IOCTL_FAIL_CMD --------------------------------------------------------- - -Return mount and expire result status from user space to the kernel. -Both of these calls require an initialized struct autofs_dev_ioctl -with the ioctlfd field set to the descriptor obtained from the open -call and the token field of struct args_ready or struct args_fail set -to the wait queue token number, received by user space in the foregoing -mount or expire request. The status field of struct args_fail is set to -the errno of the operation. It is set to 0 on success. - - -AUTOFS_DEV_IOCTL_SETPIPEFD_CMD ------------------------------- - -Set the pipe file descriptor used for kernel communication to the daemon. -Normally this is set at mount time using an option but when reconnecting -to a existing mount we need to use this to tell the autofs mount about -the new kernel pipe descriptor. In order to protect mounts against -incorrectly setting the pipe descriptor we also require that the autofs -mount be catatonic (see next call). - -The call requires an initialized struct autofs_dev_ioctl with the -ioctlfd field set to the descriptor obtained from the open call and -the pipefd field of struct args_setpipefd set to descriptor of the pipe. -On success the call also sets the process group id used to identify the -controlling process (eg. the owning automount(8) daemon) to the process -group of the caller. - - -AUTOFS_DEV_IOCTL_CATATONIC_CMD ------------------------------- - -Make the autofs mount point catatonic. The autofs mount will no longer -issue mount requests, the kernel communication pipe descriptor is released -and any remaining waits in the queue released. - -The call requires an initialized struct autofs_dev_ioctl with the -ioctlfd field set to the descriptor obtained from the open call. - - -AUTOFS_DEV_IOCTL_TIMEOUT_CMD ----------------------------- - -Set the expire timeout for mounts within an autofs mount point. - -The call requires an initialized struct autofs_dev_ioctl with the -ioctlfd field set to the descriptor obtained from the open call. - - -AUTOFS_DEV_IOCTL_REQUESTER_CMD ------------------------------- - -Return the uid and gid of the last process to successfully trigger a the -mount on the given path dentry. - -The call requires an initialized struct autofs_dev_ioctl with the path -field set to the mount point in question and the size field adjusted -appropriately. Upon return the uid field of struct args_requester contains -the uid and gid field the gid. - -When reconstructing an autofs mount tree with active mounts we need to -re-connect to mounts that may have used the original process uid and -gid (or string variations of them) for mount lookups within the map entry. -This call provides the ability to obtain this uid and gid so they may be -used by user space for the mount map lookups. - - -AUTOFS_DEV_IOCTL_EXPIRE_CMD ---------------------------- - -Issue an expire request to the kernel for an autofs mount. Typically -this ioctl is called until no further expire candidates are found. - -The call requires an initialized struct autofs_dev_ioctl with the -ioctlfd field set to the descriptor obtained from the open call. In -addition an immediate expire, independent of the mount timeout, can be -requested by setting the how field of struct args_expire to 1. If no -expire candidates can be found the ioctl returns -1 with errno set to -EAGAIN. - -This call causes the kernel module to check the mount corresponding -to the given ioctlfd for mounts that can be expired, issues an expire -request back to the daemon and waits for completion. - -AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD ------------------------------- - -Checks if an autofs mount point is in use. - -The call requires an initialized struct autofs_dev_ioctl with the -ioctlfd field set to the descriptor obtained from the open call and -it returns the result in the may_umount field of struct args_askumount, -1 for busy and 0 otherwise. - - -AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD ---------------------------------- - -Check if the given path is a mountpoint. - -The call requires an initialized struct autofs_dev_ioctl. There are two -possible variations. Both use the path field set to the path of the mount -point to check and the size field adjusted appropriately. One uses the -ioctlfd field to identify a specific mount point to check while the other -variation uses the path and optionally in.type field of struct args_ismountpoint -set to an autofs mount type. The call returns 1 if this is a mount point -and sets out.devid field to the device number of the mount and out.magic -field to the relevant super block magic number (described below) or 0 if -it isn't a mountpoint. In both cases the the device number (as returned -by new_encode_dev()) is returned in out.devid field. - -If supplied with a file descriptor we're looking for a specific mount, -not necessarily at the top of the mounted stack. In this case the path -the descriptor corresponds to is considered a mountpoint if it is itself -a mountpoint or contains a mount, such as a multi-mount without a root -mount. In this case we return 1 if the descriptor corresponds to a mount -point and and also returns the super magic of the covering mount if there -is one or 0 if it isn't a mountpoint. - -If a path is supplied (and the ioctlfd field is set to -1) then the path -is looked up and is checked to see if it is the root of a mount. If a -type is also given we are looking for a particular autofs mount and if -a match isn't found a fail is returned. If the the located path is the -root of a mount 1 is returned along with the super magic of the mount -or 0 otherwise. - diff --git a/Documentation/filesystems/autofs4.txt b/Documentation/filesystems/autofs4.txt deleted file mode 100644 index f10dd590f69f..000000000000 --- a/Documentation/filesystems/autofs4.txt +++ /dev/null @@ -1,529 +0,0 @@ - - - - -autofs - how it works -===================== - -Purpose -------- - -The goal of autofs is to provide on-demand mounting and race free -automatic unmounting of various other filesystems. This provides two -key advantages: - -1. There is no need to delay boot until all filesystems that - might be needed are mounted. Processes that try to access those - slow filesystems might be delayed but other processes can - continue freely. This is particularly important for - network filesystems (e.g. NFS) or filesystems stored on - media with a media-changing robot. - -2. The names and locations of filesystems can be stored in - a remote database and can change at any time. The content - in that data base at the time of access will be used to provide - a target for the access. The interpretation of names in the - filesystem can even be programmatic rather than database-backed, - allowing wildcards for example, and can vary based on the user who - first accessed a name. - -Context -------- - -The "autofs4" filesystem module is only one part of an autofs system. -There also needs to be a user-space program which looks up names -and mounts filesystems. This will often be the "automount" program, -though other tools including "systemd" can make use of "autofs4". -This document describes only the kernel module and the interactions -required with any user-space program. Subsequent text refers to this -as the "automount daemon" or simply "the daemon". - -"autofs4" is a Linux kernel module with provides the "autofs" -filesystem type. Several "autofs" filesystems can be mounted and they -can each be managed separately, or all managed by the same daemon. - -Content -------- - -An autofs filesystem can contain 3 sorts of objects: directories, -symbolic links and mount traps. Mount traps are directories with -extra properties as described in the next section. - -Objects can only be created by the automount daemon: symlinks are -created with a regular `symlink` system call, while directories and -mount traps are created with `mkdir`. The determination of whether a -directory should be a mount trap or not is quite _ad hoc_, largely for -historical reasons, and is determined in part by the -*direct*/*indirect*/*offset* mount options, and the *maxproto* mount option. - -If neither the *direct* or *offset* mount options are given (so the -mount is considered to be *indirect*), then the root directory is -always a regular directory, otherwise it is a mount trap when it is -empty and a regular directory when not empty. Note that *direct* and -*offset* are treated identically so a concise summary is that the root -directory is a mount trap only if the filesystem is mounted *direct* -and the root is empty. - -Directories created in the root directory are mount traps only if the -filesystem is mounted *indirect* and they are empty. - -Directories further down the tree depend on the *maxproto* mount -option and particularly whether it is less than five or not. -When *maxproto* is five, no directories further down the -tree are ever mount traps, they are always regular directories. When -the *maxproto* is four (or three), these directories are mount traps -precisely when they are empty. - -So: non-empty (i.e. non-leaf) directories are never mount traps. Empty -directories are sometimes mount traps, and sometimes not depending on -where in the tree they are (root, top level, or lower), the *maxproto*, -and whether the mount was *indirect* or not. - -Mount Traps ---------------- - -A core element of the implementation of autofs is the Mount Traps -which are provided by the Linux VFS. Any directory provided by a -filesystem can be designated as a trap. This involves two separate -features that work together to allow autofs to do its job. - -**DCACHE_NEED_AUTOMOUNT** - -If a dentry has the DCACHE_NEED_AUTOMOUNT flag set (which gets set if -the inode has S_AUTOMOUNT set, or can be set directly) then it is -(potentially) a mount trap. Any access to this directory beyond a -"`stat`" will (normally) cause the `d_op->d_automount()` dentry operation -to be called. The task of this method is to find the filesystem that -should be mounted on the directory and to return it. The VFS is -responsible for actually mounting the root of this filesystem on the -directory. - -autofs doesn't find the filesystem itself but sends a message to the -automount daemon asking it to find and mount the filesystem. The -autofs `d_automount` method then waits for the daemon to report that -everything is ready. It will then return "`NULL`" indicating that the -mount has already happened. The VFS doesn't try to mount anything but -follows down the mount that is already there. - -This functionality is sufficient for some users of mount traps such -as NFS which creates traps so that mountpoints on the server can be -reflected on the client. However it is not sufficient for autofs. As -mounting onto a directory is considered to be "beyond a `stat`", the -automount daemon would not be able to mount a filesystem on the 'trap' -directory without some way to avoid getting caught in the trap. For -that purpose there is another flag. - -**DCACHE_MANAGE_TRANSIT** - -If a dentry has DCACHE_MANAGE_TRANSIT set then two very different but -related behaviors are invoked, both using the `d_op->d_manage()` -dentry operation. - -Firstly, before checking to see if any filesystem is mounted on the -directory, d_manage() will be called with the `rcu_walk` parameter set -to `false`. It may return one of three things: - -- A return value of zero indicates that there is nothing special - about this dentry and normal checks for mounts and automounts - should proceed. - - autofs normally returns zero, but first waits for any - expiry (automatic unmounting of the mounted filesystem) to - complete. This avoids races. - -- A return value of `-EISDIR` tells the VFS to ignore any mounts - on the directory and to not consider calling `->d_automount()`. - This effectively disables the **DCACHE_NEED_AUTOMOUNT** flag - causing the directory not be a mount trap after all. - - autofs returns this if it detects that the process performing the - lookup is the automount daemon and that the mount has been - requested but has not yet completed. How it determines this is - discussed later. This allows the automount daemon not to get - caught in the mount trap. - - There is a subtlety here. It is possible that a second autofs - filesystem can be mounted below the first and for both of them to - be managed by the same daemon. For the daemon to be able to mount - something on the second it must be able to "walk" down past the - first. This means that d_manage cannot *always* return -EISDIR for - the automount daemon. It must only return it when a mount has - been requested, but has not yet completed. - - `d_manage` also returns `-EISDIR` if the dentry shouldn't be a - mount trap, either because it is a symbolic link or because it is - not empty. - -- Any other negative value is treated as an error and returned - to the caller. - - autofs can return - - - -ENOENT if the automount daemon failed to mount anything, - - -ENOMEM if it ran out of memory, - - -EINTR if a signal arrived while waiting for expiry to - complete - - or any other error sent down by the automount daemon. - - -The second use case only occurs during an "RCU-walk" and so `rcu_walk` -will be set. - -An RCU-walk is a fast and lightweight process for walking down a -filename path (i.e. it is like running on tip-toes). RCU-walk cannot -cope with all situations so when it finds a difficulty it falls back -to "REF-walk", which is slower but more robust. - -RCU-walk will never call `->d_automount`; the filesystems must already -be mounted or RCU-walk cannot handle the path. -To determine if a mount-trap is safe for RCU-walk mode it calls -`->d_manage()` with `rcu_walk` set to `true`. - -In this case `d_manage()` must avoid blocking and should avoid taking -spinlocks if at all possible. Its sole purpose is to determine if it -would be safe to follow down into any mounted directory and the only -reason that it might not be is if an expiry of the mount is -underway. - -In the `rcu_walk` case, `d_manage()` cannot return -EISDIR to tell the -VFS that this is a directory that doesn't require d_automount. If -`rcu_walk` sees a dentry with DCACHE_NEED_AUTOMOUNT set but nothing -mounted, it *will* fall back to REF-walk. `d_manage()` cannot make the -VFS remain in RCU-walk mode, but can only tell it to get out of -RCU-walk mode by returning `-ECHILD`. - -So `d_manage()`, when called with `rcu_walk` set, should either return --ECHILD if there is any reason to believe it is unsafe to end the -mounted filesystem, and otherwise should return 0. - -autofs will return `-ECHILD` if an expiry of the filesystem has been -initiated or is being considered, otherwise it returns 0. - - -Mountpoint expiry ------------------ - -The VFS has a mechanism for automatically expiring unused mounts, -much as it can expire any unused dentry information from the dcache. -This is guided by the MNT_SHRINKABLE flag. This only applies to -mounts that were created by `d_automount()` returning a filesystem to be -mounted. As autofs doesn't return such a filesystem but leaves the -mounting to the automount daemon, it must involve the automount daemon -in unmounting as well. This also means that autofs has more control -of expiry. - -The VFS also supports "expiry" of mounts using the MNT_EXPIRE flag to -the `umount` system call. Unmounting with MNT_EXPIRE will fail unless -a previous attempt had been made, and the filesystem has been inactive -and untouched since that previous attempt. autofs4 does not depend on -this but has its own internal tracking of whether filesystems were -recently used. This allows individual names in the autofs directory -to expire separately. - -With version 4 of the protocol, the automount daemon can try to -unmount any filesystems mounted on the autofs filesystem or remove any -symbolic links or empty directories any time it likes. If the unmount -or removal is successful the filesystem will be returned to the state -it was before the mount or creation, so that any access of the name -will trigger normal auto-mount processing. In particlar, `rmdir` and -`unlink` do not leave negative entries in the dcache as a normal -filesystem would, so an attempt to access a recently-removed object is -passed to autofs for handling. - -With version 5, this is not safe except for unmounting from top-level -directories. As lower-level directories are never mount traps, other -processes will see an empty directory as soon as the filesystem is -unmounted. So it is generally safest to use the autofs expiry -protocol described below. - -Normally the daemon only wants to remove entries which haven't been -used for a while. For this purpose autofs maintains a "`last_used`" -time stamp on each directory or symlink. For symlinks it genuinely -does record the last time the symlink was "used" or followed to find -out where it points to. For directories the field is a slight -misnomer. It actually records the last time that autofs checked if -the directory or one of its descendents was busy and found that it -was. This is just as useful and doesn't require updating the field so -often. - -The daemon is able to ask autofs if anything is due to be expired, -using an `ioctl` as discussed later. For a *direct* mount, autofs -considers if the entire mount-tree can be unmounted or not. For an -*indirect* mount, autofs considers each of the names in the top level -directory to determine if any of those can be unmounted and cleaned -up. - -There is an option with indirect mounts to consider each of the leaves -that has been mounted on instead of considering the top-level names. -This is intended for compatability with version 4 of autofs and should -be considered as deprecated. - -When autofs considers a directory it checks the `last_used` time and -compares it with the "timeout" value set when the filesystem was -mounted, though this check is ignored in some cases. It also checks if -the directory or anything below it is in use. For symbolic links, -only the `last_used` time is ever considered. - -If both appear to support expiring the directory or symlink, an action -is taken. - -There are two ways to ask autofs to consider expiry. The first is to -use the **AUTOFS_IOC_EXPIRE** ioctl. This only works for indirect -mounts. If it finds something in the root directory to expire it will -return the name of that thing. Once a name has been returned the -automount daemon needs to unmount any filesystems mounted below the -name normally. As described above, this is unsafe for non-toplevel -mounts in a version-5 autofs. For this reason the current `automountd` -does not use this ioctl. - -The second mechanism uses either the **AUTOFS_DEV_IOCTL_EXPIRE_CMD** or -the **AUTOFS_IOC_EXPIRE_MULTI** ioctl. This will work for both direct and -indirect mounts. If it selects an object to expire, it will notify -the daemon using the notification mechanism described below. This -will block until the daemon acknowledges the expiry notification. -This implies that the "`EXPIRE`" ioctl must be sent from a different -thread than the one which handles notification. - -While the ioctl is blocking, the entry is marked as "expiring" and -`d_manage` will block until the daemon affirms that the unmount has -completed (together with removing any directories that might have been -necessary), or has been aborted. - -Communicating with autofs: detecting the daemon ------------------------------------------------ - -There are several forms of communication between the automount daemon -and the filesystem. As we have already seen, the daemon can create and -remove directories and symlinks using normal filesystem operations. -autofs knows whether a process requesting some operation is the daemon -or not based on its process-group id number (see getpgid(1)). - -When an autofs filesystem is mounted the pgid of the mounting -processes is recorded unless the "pgrp=" option is given, in which -case that number is recorded instead. Any request arriving from a -process in that process group is considered to come from the daemon. -If the daemon ever has to be stopped and restarted a new pgid can be -provided through an ioctl as will be described below. - -Communicating with autofs: the event pipe ------------------------------------------ - -When an autofs filesystem is mounted, the 'write' end of a pipe must -be passed using the 'fd=' mount option. autofs will write -notification messages to this pipe for the daemon to respond to. -For version 5, the format of the message is: - - struct autofs_v5_packet { - int proto_version; /* Protocol version */ - int type; /* Type of packet */ - autofs_wqt_t wait_queue_token; - __u32 dev; - __u64 ino; - __u32 uid; - __u32 gid; - __u32 pid; - __u32 tgid; - __u32 len; - char name[NAME_MAX+1]; - }; - -where the type is one of - - autofs_ptype_missing_indirect - autofs_ptype_expire_indirect - autofs_ptype_missing_direct - autofs_ptype_expire_direct - -so messages can indicate that a name is missing (something tried to -access it but it isn't there) or that it has been selected for expiry. - -The pipe will be set to "packet mode" (equivalent to passing -`O_DIRECT`) to _pipe2(2)_ so that a read from the pipe will return at -most one packet, and any unread portion of a packet will be discarded. - -The `wait_queue_token` is a unique number which can identify a -particular request to be acknowledged. When a message is sent over -the pipe the affected dentry is marked as either "active" or -"expiring" and other accesses to it block until the message is -acknowledged using one of the ioctls below and the relevant -`wait_queue_token`. - -Communicating with autofs: root directory ioctls ------------------------------------------------- - -The root directory of an autofs filesystem will respond to a number of -ioctls. The process issuing the ioctl must have the CAP_SYS_ADMIN -capability, or must be the automount daemon. - -The available ioctl commands are: - -- **AUTOFS_IOC_READY**: a notification has been handled. The argument - to the ioctl command is the "wait_queue_token" number - corresponding to the notification being acknowledged. -- **AUTOFS_IOC_FAIL**: similar to above, but indicates failure with - the error code `ENOENT`. -- **AUTOFS_IOC_CATATONIC**: Causes the autofs to enter "catatonic" - mode meaning that it stops sending notifications to the daemon. - This mode is also entered if a write to the pipe fails. -- **AUTOFS_IOC_PROTOVER**: This returns the protocol version in use. -- **AUTOFS_IOC_PROTOSUBVER**: Returns the protocol sub-version which - is really a version number for the implementation. It is - currently 2. -- **AUTOFS_IOC_SETTIMEOUT**: This passes a pointer to an unsigned - long. The value is used to set the timeout for expiry, and - the current timeout value is stored back through the pointer. -- **AUTOFS_IOC_ASKUMOUNT**: Returns, in the pointed-to `int`, 1 if - the filesystem could be unmounted. This is only a hint as - the situation could change at any instant. This call can be - use to avoid a more expensive full unmount attempt. -- **AUTOFS_IOC_EXPIRE**: as described above, this asks if there is - anything suitable to expire. A pointer to a packet: - - struct autofs_packet_expire_multi { - int proto_version; /* Protocol version */ - int type; /* Type of packet */ - autofs_wqt_t wait_queue_token; - int len; - char name[NAME_MAX+1]; - }; - - is required. This is filled in with the name of something - that can be unmounted or removed. If nothing can be expired, - `errno` is set to `EAGAIN`. Even though a `wait_queue_token` - is present in the structure, no "wait queue" is established - and no acknowledgment is needed. -- **AUTOFS_IOC_EXPIRE_MULTI**: This is similar to - **AUTOFS_IOC_EXPIRE** except that it causes notification to be - sent to the daemon, and it blocks until the daemon acknowledges. - The argument is an integer which can contain two different flags. - - **AUTOFS_EXP_IMMEDIATE** causes `last_used` time to be ignored - and objects are expired if the are not in use. - - **AUTOFS_EXP_LEAVES** will select a leaf rather than a top-level - name to expire. This is only safe when *maxproto* is 4. - -Communicating with autofs: char-device ioctls ---------------------------------------------- - -It is not always possible to open the root of an autofs filesystem, -particularly a *direct* mounted filesystem. If the automount daemon -is restarted there is no way for it to regain control of existing -mounts using any of the above communication channels. To address this -need there is a "miscellaneous" character device (major 10, minor 235) -which can be used to communicate directly with the autofs filesystem. -It requires CAP_SYS_ADMIN for access. - -The `ioctl`s that can be used on this device are described in a separate -document `autofs4-mount-control.txt`, and are summarized briefly here. -Each ioctl is passed a pointer to an `autofs_dev_ioctl` structure: - - struct autofs_dev_ioctl { - __u32 ver_major; - __u32 ver_minor; - __u32 size; /* total size of data passed in - * including this struct */ - __s32 ioctlfd; /* automount command fd */ - - /* Command parameters */ - union { - struct args_protover protover; - struct args_protosubver protosubver; - struct args_openmount openmount; - struct args_ready ready; - struct args_fail fail; - struct args_setpipefd setpipefd; - struct args_timeout timeout; - struct args_requester requester; - struct args_expire expire; - struct args_askumount askumount; - struct args_ismountpoint ismountpoint; - }; - - char path[0]; - }; - -For the **OPEN_MOUNT** and **IS_MOUNTPOINT** commands, the target -filesystem is identified by the `path`. All other commands identify -the filesystem by the `ioctlfd` which is a file descriptor open on the -root, and which can be returned by **OPEN_MOUNT**. - -The `ver_major` and `ver_minor` are in/out parameters which check that -the requested version is supported, and report the maximum version -that the kernel module can support. - -Commands are: - -- **AUTOFS_DEV_IOCTL_VERSION_CMD**: does nothing, except validate and - set version numbers. -- **AUTOFS_DEV_IOCTL_OPENMOUNT_CMD**: return an open file descriptor - on the root of an autofs filesystem. The filesystem is identified - by name and device number, which is stored in `openmount.devid`. - Device numbers for existing filesystems can be found in - `/proc/self/mountinfo`. -- **AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD**: same as `close(ioctlfd)`. -- **AUTOFS_DEV_IOCTL_SETPIPEFD_CMD**: if the filesystem is in - catatonic mode, this can provide the write end of a new pipe - in `setpipefd.pipefd` to re-establish communication with a daemon. - The process group of the calling process is used to identify the - daemon. -- **AUTOFS_DEV_IOCTL_REQUESTER_CMD**: `path` should be a - name within the filesystem that has been auto-mounted on. - On successful return, `requester.uid` and `requester.gid` will be - the UID and GID of the process which triggered that mount. -- **AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD**: Check if path is a - mountpoint of a particular type - see separate documentation for - details. -- **AUTOFS_DEV_IOCTL_PROTOVER_CMD**: -- **AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD**: -- **AUTOFS_DEV_IOCTL_READY_CMD**: -- **AUTOFS_DEV_IOCTL_FAIL_CMD**: -- **AUTOFS_DEV_IOCTL_CATATONIC_CMD**: -- **AUTOFS_DEV_IOCTL_TIMEOUT_CMD**: -- **AUTOFS_DEV_IOCTL_EXPIRE_CMD**: -- **AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD**: These all have the same - function as the similarly named **AUTOFS_IOC** ioctls, except - that **FAIL** can be given an explicit error number in `fail.status` - instead of assuming `ENOENT`, and this **EXPIRE** command - corresponds to **AUTOFS_IOC_EXPIRE_MULTI**. - -Catatonic mode --------------- - -As mentioned, an autofs mount can enter "catatonic" mode. This -happens if a write to the notification pipe fails, or if it is -explicitly requested by an `ioctl`. - -When entering catatonic mode, the pipe is closed and any pending -notifications are acknowledged with the error `ENOENT`. - -Once in catatonic mode attempts to access non-existing names will -result in `ENOENT` while attempts to access existing directories will -be treated in the same way as if they came from the daemon, so mount -traps will not fire. - -When the filesystem is mounted a _uid_ and _gid_ can be given which -set the ownership of directories and symbolic links. When the -filesystem is in catatonic mode, any process with a matching UID can -create directories or symlinks in the root directory, but not in other -directories. - -Catatonic mode can only be left via the -**AUTOFS_DEV_IOCTL_OPENMOUNT_CMD** ioctl on the `/dev/autofs`. - -autofs, name spaces, and shared mounts --------------------------------------- - -With bind mounts and name spaces it is possible for an autofs -filesystem to appear at multiple places in one or more filesystem -name spaces. For this to work sensibly, the autofs filesystem should -always be mounted "shared". e.g. - -> `mount --make-shared /autofs/mount/point` - -The automount daemon is only able to manage a single mount location for -an autofs filesystem and if mounts on that are not 'shared', other -locations will not behave as expected. In particular access to those -other locations will likely result in the `ELOOP` error - -> Too many levels of symbolic links diff --git a/Documentation/filesystems/automount-support.txt b/Documentation/filesystems/automount-support.txt index 7eb762eb3136..b0afd3d55eaf 100644 --- a/Documentation/filesystems/automount-support.txt +++ b/Documentation/filesystems/automount-support.txt @@ -9,7 +9,7 @@ also be requested by userspace. IN-KERNEL AUTOMOUNTING ====================== -See section "Mount Traps" of Documentation/filesystems/autofs4.txt +See section "Mount Traps" of Documentation/filesystems/autofs.txt Then from userspace, you can just do something like: diff --git a/Documentation/filesystems/ceph.txt b/Documentation/filesystems/ceph.txt index d7f011ddc150..8bf62240e10d 100644 --- a/Documentation/filesystems/ceph.txt +++ b/Documentation/filesystems/ceph.txt @@ -105,15 +105,13 @@ Mount Options address its connection to the monitor originates from. wsize=X - Specify the maximum write size in bytes. By default there is no - maximum. Ceph will normally size writes based on the file stripe - size. + Specify the maximum write size in bytes. Default: 16 MB. rsize=X - Specify the maximum read size in bytes. Default: 64 MB. + Specify the maximum read size in bytes. Default: 16 MB. rasize=X - Specify the maximum readahead. Default: 8 MB. + Specify the maximum readahead size in bytes. Default: 8 MB. mount_timeout=X Specify the timeout value for mount (in seconds), in the case diff --git a/Documentation/filesystems/cifs/AUTHORS b/Documentation/filesystems/cifs/AUTHORS index 9f4f87e16240..75865da2ce14 100644 --- a/Documentation/filesystems/cifs/AUTHORS +++ b/Documentation/filesystems/cifs/AUTHORS @@ -42,9 +42,11 @@ Jeff Layton (many, many fixes, as well as great work on the cifs Kerberos code) Scott Lovenberg Pavel Shilovsky (for great work adding SMB2 support, and various SMB3 features) Aurelien Aptel (for DFS SMB3 work and some key bug fixes) -Ronnie Sahlberg (for SMB3 xattr work and bug fixes) +Ronnie Sahlberg (for SMB3 xattr work, bug fixes, and lots of great work on compounding) Shirish Pargaonkar (for many ACL patches over the years) Sachin Prabhu (many bug fixes, including for reconnect, copy offload and security) +Paulo Alcantara +Long Li (some great work on RDMA, SMB Direct) Test case and Bug Report contributors @@ -58,5 +60,4 @@ mention to the Stanford Checker (SWAT) which pointed out many minor bugs in error paths. Valuable suggestions also have come from Al Viro and Dave Miller. -And thanks to the IBM LTC and Power test teams and SuSE testers for -finding multiple bugs during excellent stress test runs. +And thanks to the IBM LTC and Power test teams and SuSE and Citrix and RedHat testers for finding multiple bugs during excellent stress test runs. diff --git a/Documentation/filesystems/cifs/CHANGES b/Documentation/filesystems/cifs/CHANGES index bc0025cdd1c9..455e1cc494a9 100644 --- a/Documentation/filesystems/cifs/CHANGES +++ b/Documentation/filesystems/cifs/CHANGES @@ -1,3 +1,6 @@ +See https://wiki.samba.org/index.php/LinuxCIFSKernel for +more current information. + Version 1.62 ------------ Add sockopt=TCP_NODELAY mount option. EA (xattr) routines hardened diff --git a/Documentation/filesystems/cifs/TODO b/Documentation/filesystems/cifs/TODO index c5adf149b57f..852499aed64b 100644 --- a/Documentation/filesystems/cifs/TODO +++ b/Documentation/filesystems/cifs/TODO @@ -9,14 +9,14 @@ is a partial list of the known problems and missing features: a) SMB3 (and SMB3.02) missing optional features: - multichannel (started), integration with RDMA - - directory leases (improved metadata caching) - - T10 copy offload (copy chunk, and "Duplicate Extents" ioctl + - directory leases (improved metadata caching), started (root dir only) + - T10 copy offload ie "ODX" (copy chunk, and "Duplicate Extents" ioctl currently the only two server side copy mechanisms supported) b) improved sparse file support c) Directory entry caching relies on a 1 second timer, rather than -using Directory Leases +using Directory Leases, currently only the root file handle is cached longer d) quota support (needs minor kernel change since quota calls to make it to network filesystems or deviceless filesystems) @@ -42,6 +42,8 @@ mount or a per server basis to client UIDs or nobody if no mapping exists. Also better integration with winbind for resolving SID owners k) Add tools to take advantage of more smb3 specific ioctls and features +(passthrough ioctl/fsctl for sending various SMB3 fsctls to the server +is in progress) l) encrypted file support @@ -71,9 +73,8 @@ t) split cifs and smb3 support into separate modules so legacy (and less secure) CIFS dialect can be disabled in environments that don't need it and simplify the code. -u) Finish up SMB3.1.1 dialect support - -v) POSIX Extensions for SMB3.1.1 +v) POSIX Extensions for SMB3.1.1 (started, create and mkdir support added +so far). KNOWN BUGS ==================================== @@ -92,8 +93,8 @@ Misc testing to do 1) check out max path names and max path name components against various server types. Try nested symlinks (8 deep). Return max path name in stat -f information -2) Improve xfstest's cifs enablement and adapt xfstests where needed to test -cifs better +2) Improve xfstest's cifs/smb3 enablement and adapt xfstests where needed to test +cifs/smb3 better 3) Additional performance testing and optimization using iozone and similar - there are some easy changes that can be done to parallelize sequential writes, diff --git a/Documentation/filesystems/f2fs.txt b/Documentation/filesystems/f2fs.txt index 12a147c9f87f..69f8de995739 100644 --- a/Documentation/filesystems/f2fs.txt +++ b/Documentation/filesystems/f2fs.txt @@ -182,13 +182,15 @@ whint_mode=%s Control which write hints are passed down to block passes down hints with its policy. alloc_mode=%s Adjust block allocation policy, which supports "reuse" and "default". -fsync_mode=%s Control the policy of fsync. Currently supports "posix" - and "strict". In "posix" mode, which is default, fsync - will follow POSIX semantics and does a light operation - to improve the filesystem performance. In "strict" mode, - fsync will be heavy and behaves in line with xfs, ext4 - and btrfs, where xfstest generic/342 will pass, but the - performance will regress. +fsync_mode=%s Control the policy of fsync. Currently supports "posix", + "strict", and "nobarrier". In "posix" mode, which is + default, fsync will follow POSIX semantics and does a + light operation to improve the filesystem performance. + In "strict" mode, fsync will be heavy and behaves in line + with xfs, ext4 and btrfs, where xfstest generic/342 will + pass, but the performance will regress. "nobarrier" is + based on "posix", but doesn't issue flush command for + non-atomic files likewise "nobarrier" mount option. test_dummy_encryption Enable dummy encryption, which provides a fake fscrypt context. The fake fscrypt context is used by xfstests. diff --git a/Documentation/filesystems/fscrypt.rst b/Documentation/filesystems/fscrypt.rst index cfbc18f0d9c9..48b424de85bb 100644 --- a/Documentation/filesystems/fscrypt.rst +++ b/Documentation/filesystems/fscrypt.rst @@ -191,11 +191,21 @@ Currently, the following pairs of encryption modes are supported: - AES-256-XTS for contents and AES-256-CTS-CBC for filenames - AES-128-CBC for contents and AES-128-CTS-CBC for filenames +- Speck128/256-XTS for contents and Speck128/256-CTS-CBC for filenames It is strongly recommended to use AES-256-XTS for contents encryption. AES-128-CBC was added only for low-powered embedded devices with crypto accelerators such as CAAM or CESA that do not support XTS. +Similarly, Speck128/256 support was only added for older or low-end +CPUs which cannot do AES fast enough -- especially ARM CPUs which have +NEON instructions but not the Cryptography Extensions -- and for which +it would not otherwise be feasible to use encryption at all. It is +not recommended to use Speck on CPUs that have AES instructions. +Speck support is only available if it has been enabled in the crypto +API via CONFIG_CRYPTO_SPECK. Also, on ARM platforms, to get +acceptable performance CONFIG_CRYPTO_SPECK_NEON must be enabled. + New encryption modes can be added relatively easily, without changes to individual filesystems. However, authenticated encryption (AE) modes are not currently supported because of the difficulty of dealing diff --git a/Documentation/filesystems/fuse-io.txt b/Documentation/filesystems/fuse-io.txt new file mode 100644 index 000000000000..07b8f73f100f --- /dev/null +++ b/Documentation/filesystems/fuse-io.txt @@ -0,0 +1,38 @@ +Fuse supports the following I/O modes: + +- direct-io +- cached + + write-through + + writeback-cache + +The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the +FUSE_OPEN reply. + +In direct-io mode the page cache is completely bypassed for reads and writes. +No read-ahead takes place. Shared mmap is disabled. + +In cached mode reads may be satisfied from the page cache, and data may be +read-ahead by the kernel to fill the cache. The cache is always kept consistent +after any writes to the file. All mmap modes are supported. + +The cached mode has two sub modes controlling how writes are handled. The +write-through mode is the default and is supported on all kernels. The +writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the +FUSE_INIT reply. + +In write-through mode each write is immediately sent to userspace as one or more +WRITE requests, as well as updating any cached pages (and caching previously +uncached, but fully written pages). No READ requests are ever sent for writes, +so when an uncached page is partially written, the page is discarded. + +In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to +the cache only, which means that the write(2) syscall can often complete very +fast. Dirty pages are written back implicitly (background writeback or page +reclaim on memory pressure) or explicitly (invoked by close(2), fsync(2) and +when the last ref to the file is being released on munmap(2)). This mode +assumes that all changes to the filesystem go through the FUSE kernel module +(size and atime/ctime/mtime attributes are kept up-to-date by the kernel), so +it's generally not suitable for network filesystems. If a partial page is +written, then the page needs to be first read from userspace. This means, that +even for files opened for O_WRONLY it is possible that READ requests will be +generated by the kernel. diff --git a/Documentation/filesystems/ncpfs.txt b/Documentation/filesystems/ncpfs.txt deleted file mode 100644 index 5af164f4b37b..000000000000 --- a/Documentation/filesystems/ncpfs.txt +++ /dev/null @@ -1,12 +0,0 @@ -The ncpfs filesystem understands the NCP protocol, designed by the -Novell Corporation for their NetWare(tm) product. NCP is functionally -similar to the NFS used in the TCP/IP community. -To mount a NetWare filesystem, you need a special mount program, which -can be found in the ncpfs package. The home site for ncpfs is -ftp.gwdg.de/pub/linux/misc/ncpfs, but sunsite and its many mirrors -will have it as well. - -Related products are linware and mars_nwe, which will give Linux partial -NetWare server functionality. - -mars_nwe can be found on ftp.gwdg.de/pub/linux/misc/ncpfs. diff --git a/Documentation/filesystems/nfs/nfsroot.txt b/Documentation/filesystems/nfs/nfsroot.txt index 5efae00f6c7f..d2963123eb1c 100644 --- a/Documentation/filesystems/nfs/nfsroot.txt +++ b/Documentation/filesystems/nfs/nfsroot.txt @@ -5,6 +5,7 @@ Written 1996 by Gero Kuhlmann Updated 1997 by Martin Mares Updated 2006 by Nico Schottelius Updated 2006 by Horms +Updated 2018 by Chris Novakovic @@ -79,7 +80,7 @@ nfsroot=[:][,] ip=::::::: - : + :: This parameter tells the kernel how to configure IP addresses of devices and also how to set up the IP routing table. It was originally called @@ -110,6 +111,9 @@ ip=::::::: will not be triggered if it is missing and NFS root is not in operation. + Value is exported to /proc/net/pnp with the prefix "bootserver " + (see below). + Default: Determined using autoconfiguration. The address of the autoconfiguration server is used. @@ -123,10 +127,13 @@ ip=::::::: Default: Determined using autoconfiguration. - Name of the client. May be supplied by autoconfiguration, - but its absence will not trigger autoconfiguration. - If specified and DHCP is used, the user provided hostname will - be carried in the DHCP request to hopefully update DNS record. + Name of the client. If a '.' character is present, anything + before the first '.' is used as the client's hostname, and anything + after it is used as its NIS domain name. May be supplied by + autoconfiguration, but its absence will not trigger autoconfiguration. + If specified and DHCP is used, the user-provided hostname (and NIS + domain name, if present) will be carried in the DHCP request; this + may cause a DNS record to be created or updated for the client. Default: Client IP address is used in ASCII notation. @@ -162,12 +169,55 @@ ip=::::::: Default: any - IP address of first nameserver. - Value gets exported by /proc/net/pnp which is often linked - on embedded systems by /etc/resolv.conf. + IP address of primary nameserver. + Value is exported to /proc/net/pnp with the prefix "nameserver " + (see below). + + Default: None if not using autoconfiguration; determined + automatically if using autoconfiguration. + + IP address of secondary nameserver. + See . + + IP address of a Network Time Protocol (NTP) server. + Value is exported to /proc/net/ipconfig/ntp_servers, but is + otherwise unused (see below). + + Default: None if not using autoconfiguration; determined + automatically if using autoconfiguration. + + After configuration (whether manual or automatic) is complete, two files + are created in the following format; lines are omitted if their respective + value is empty following configuration: + + - /proc/net/pnp: + + #PROTO: (depending on configuration method) + domain (if autoconfigured, the DNS domain) + nameserver (primary name server IP) + nameserver (secondary name server IP) + nameserver (tertiary name server IP) + bootserver (NFS server IP) + + - /proc/net/ipconfig/ntp_servers: + + (NTP server IP) + (NTP server IP) + (NTP server IP) + + and (in /proc/net/pnp) and and + (in /proc/net/ipconfig/ntp_servers) are requested during autoconfiguration; + they cannot be specified as part of the "ip=" kernel command line parameter. + + Because the "domain" and "nameserver" options are recognised by DNS + resolvers, /etc/resolv.conf is often linked to /proc/net/pnp on systems + that use an NFS root filesystem. - IP address of second nameserver. - Same as above. + Note that the kernel will not synchronise the system time with any NTP + servers it discovers; this is the responsibility of a user space process + (e.g. an initrd/initramfs script that passes the IP addresses listed in + /proc/net/ipconfig/ntp_servers to an NTP client before mounting the real + root filesystem if it is on NFS). nfsrootdebug diff --git a/Documentation/filesystems/overlayfs.txt b/Documentation/filesystems/overlayfs.txt index 961b287ef323..72615a2c0752 100644 --- a/Documentation/filesystems/overlayfs.txt +++ b/Documentation/filesystems/overlayfs.txt @@ -429,11 +429,12 @@ This verification may cause significant overhead in some cases. Testsuite --------- -There's testsuite developed by David Howells at: +There's a testsuite originally developed by David Howells and currently +maintained by Amir Goldstein at: - git://git.infradead.org/users/dhowells/unionmount-testsuite.git + https://github.com/amir73il/unionmount-testsuite.git Run as root: # cd unionmount-testsuite - # ./run --ov + # ./run --ov --verify diff --git a/Documentation/filesystems/path-lookup.md b/Documentation/filesystems/path-lookup.md index 1933ef734e63..e2edd45c4bc0 100644 --- a/Documentation/filesystems/path-lookup.md +++ b/Documentation/filesystems/path-lookup.md @@ -460,7 +460,7 @@ this retry process in the next article. Automount points are locations in the filesystem where an attempt to lookup a name can trigger changes to how that lookup should be handled, in particular by mounting a filesystem there. These are -covered in greater detail in autofs4.txt in the Linux documentation +covered in greater detail in autofs.txt in the Linux documentation tree, but a few notes specifically related to path lookup are in order here. diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt index 2a84bb334894..520f6a84cf50 100644 --- a/Documentation/filesystems/proc.txt +++ b/Documentation/filesystems/proc.txt @@ -515,7 +515,8 @@ guarantees: The /proc/PID/clear_refs is used to reset the PG_Referenced and ACCESSED/YOUNG bits on both physical and virtual pages associated with a process, and the -soft-dirty bit on pte (see Documentation/vm/soft-dirty.txt for details). +soft-dirty bit on pte (see Documentation/admin-guide/mm/soft-dirty.rst +for details). To clear the bits for all the pages associated with the process > echo 1 > /proc/PID/clear_refs @@ -536,7 +537,8 @@ Any other value written to /proc/PID/clear_refs will have no effect. The /proc/pid/pagemap gives the PFN, which can be used to find the pageflags using /proc/kpageflags and number of times a page is mapped using -/proc/kpagecount. For detailed explanation, see Documentation/vm/pagemap.txt. +/proc/kpagecount. For detailed explanation, see +Documentation/admin-guide/mm/pagemap.rst. The /proc/pid/numa_maps is an extension based on maps, showing the memory locality and binding policy, as well as the memory usage (in pages) of @@ -564,7 +566,7 @@ address policy mapping details Where: "address" is the starting address for the mapping; -"policy" reports the NUMA memory policy set for the mapping (see vm/numa_memory_policy.txt); +"policy" reports the NUMA memory policy set for the mapping (see Documentation/admin-guide/mm/numa_memory_policy.rst); "mapping details" summarizes mapping data such as mapping type, page usage counters, node locality page counters (N0 == node0, N1 == node1, ...) and the kernel page size, in KB, that is backing the mapping up. diff --git a/Documentation/filesystems/tmpfs.txt b/Documentation/filesystems/tmpfs.txt index a85355cf85f4..d06e9a59a9f4 100644 --- a/Documentation/filesystems/tmpfs.txt +++ b/Documentation/filesystems/tmpfs.txt @@ -105,8 +105,9 @@ policy for the file will revert to "default" policy. NUMA memory allocation policies have optional flags that can be used in conjunction with their modes. These optional flags can be specified when tmpfs is mounted by appending them to the mode before the NodeList. -See Documentation/vm/numa_memory_policy.txt for a list of all available -memory allocation policy mode flags and their effect on memory policy. +See Documentation/admin-guide/mm/numa_memory_policy.rst for a list of +all available memory allocation policy mode flags and their effect on +memory policy. =static is equivalent to MPOL_F_STATIC_NODES =relative is equivalent to MPOL_F_RELATIVE_NODES diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt index 5fd325df59e2..829a7b7857a4 100644 --- a/Documentation/filesystems/vfs.txt +++ b/Documentation/filesystems/vfs.txt @@ -856,7 +856,9 @@ struct file_operations { ssize_t (*read_iter) (struct kiocb *, struct iov_iter *); ssize_t (*write_iter) (struct kiocb *, struct iov_iter *); int (*iterate) (struct file *, struct dir_context *); - unsigned int (*poll) (struct file *, struct poll_table_struct *); + __poll_t (*poll) (struct file *, struct poll_table_struct *); + struct wait_queue_head * (*get_poll_head)(struct file *, __poll_t); + __poll_t (*poll_mask) (struct file *, __poll_t); long (*unlocked_ioctl) (struct file *, unsigned int, unsigned long); long (*compat_ioctl) (struct file *, unsigned int, unsigned long); int (*mmap) (struct file *, struct vm_area_struct *); @@ -901,6 +903,17 @@ otherwise noted. activity on this file and (optionally) go to sleep until there is activity. Called by the select(2) and poll(2) system calls + get_poll_head: Returns the struct wait_queue_head that callers can + wait on. Callers need to check the returned events using ->poll_mask + once woken. Can return NULL to indicate polling is not supported, + or any error code using the ERR_PTR convention to indicate that a + grave error occured and ->poll_mask shall not be called. + + poll_mask: return the mask of EPOLL* values describing the file descriptor + state. Called either before going to sleep on the waitqueue returned by + get_poll_head, or after it has been woken. If ->get_poll_head and + ->poll_mask are implemented ->poll does not need to be implement. + unlocked_ioctl: called by the ioctl(2) system call. compat_ioctl: called by the ioctl(2) system call when 32 bit system calls diff --git a/Documentation/fpga/fpga-mgr.txt b/Documentation/fpga/fpga-mgr.txt deleted file mode 100644 index cc6413ed6fc9..000000000000 --- a/Documentation/fpga/fpga-mgr.txt +++ /dev/null @@ -1,199 +0,0 @@ -FPGA Manager Core - -Alan Tull 2015 - -Overview -======== - -The FPGA manager core exports a set of functions for programming an FPGA with -an image. The API is manufacturer agnostic. All manufacturer specifics are -hidden away in a low level driver which registers a set of ops with the core. -The FPGA image data itself is very manufacturer specific, but for our purposes -it's just binary data. The FPGA manager core won't parse it. - -The FPGA image to be programmed can be in a scatter gather list, a single -contiguous buffer, or a firmware file. Because allocating contiguous kernel -memory for the buffer should be avoided, users are encouraged to use a scatter -gather list instead if possible. - -The particulars for programming the image are presented in a structure (struct -fpga_image_info). This struct contains parameters such as pointers to the -FPGA image as well as image-specific particulars such as whether the image was -built for full or partial reconfiguration. - -API Functions: -============== - -To program the FPGA: --------------------- - - int fpga_mgr_load(struct fpga_manager *mgr, - struct fpga_image_info *info); - -Load the FPGA from an image which is indicated in the info. If successful, -the FPGA ends up in operating mode. Return 0 on success or a negative error -code. - -To allocate or free a struct fpga_image_info: ---------------------------------------------- - - struct fpga_image_info *fpga_image_info_alloc(struct device *dev); - - void fpga_image_info_free(struct fpga_image_info *info); - -To get/put a reference to a FPGA manager: ------------------------------------------ - - struct fpga_manager *of_fpga_mgr_get(struct device_node *node); - struct fpga_manager *fpga_mgr_get(struct device *dev); - void fpga_mgr_put(struct fpga_manager *mgr); - -Given a DT node or device, get a reference to a FPGA manager. This pointer -can be saved until you are ready to program the FPGA. fpga_mgr_put releases -the reference. - - -To get exclusive control of a FPGA manager: -------------------------------------------- - - int fpga_mgr_lock(struct fpga_manager *mgr); - void fpga_mgr_unlock(struct fpga_manager *mgr); - -The user should call fpga_mgr_lock and verify that it returns 0 before -attempting to program the FPGA. Likewise, the user should call -fpga_mgr_unlock when done programming the FPGA. - - -To register or unregister the low level FPGA-specific driver: -------------------------------------------------------------- - - int fpga_mgr_register(struct device *dev, const char *name, - const struct fpga_manager_ops *mops, - void *priv); - - void fpga_mgr_unregister(struct device *dev); - -Use of these two functions is described below in "How To Support a new FPGA -device." - - -How to write an image buffer to a supported FPGA -================================================ -#include - -struct fpga_manager *mgr; -struct fpga_image_info *info; -int ret; - -/* - * Get a reference to FPGA manager. The manager is not locked, so you can - * hold onto this reference without it preventing programming. - * - * This example uses the device node of the manager. Alternatively, use - * fpga_mgr_get(dev) instead if you have the device. - */ -mgr = of_fpga_mgr_get(mgr_node); - -/* struct with information about the FPGA image to program. */ -info = fpga_image_info_alloc(dev); - -/* flags indicates whether to do full or partial reconfiguration */ -info->flags = FPGA_MGR_PARTIAL_RECONFIG; - -/* - * At this point, indicate where the image is. This is pseudo-code; you're - * going to use one of these three. - */ -if (image is in a scatter gather table) { - - info->sgt = [your scatter gather table] - -} else if (image is in a buffer) { - - info->buf = [your image buffer] - info->count = [image buffer size] - -} else if (image is in a firmware file) { - - info->firmware_name = devm_kstrdup(dev, firmware_name, GFP_KERNEL); - -} - -/* Get exclusive control of FPGA manager */ -ret = fpga_mgr_lock(mgr); - -/* Load the buffer to the FPGA */ -ret = fpga_mgr_buf_load(mgr, &info, buf, count); - -/* Release the FPGA manager */ -fpga_mgr_unlock(mgr); -fpga_mgr_put(mgr); - -/* Deallocate the image info if you're done with it */ -fpga_image_info_free(info); - -How to support a new FPGA device -================================ -To add another FPGA manager, write a driver that implements a set of ops. The -probe function calls fpga_mgr_register(), such as: - -static const struct fpga_manager_ops socfpga_fpga_ops = { - .write_init = socfpga_fpga_ops_configure_init, - .write = socfpga_fpga_ops_configure_write, - .write_complete = socfpga_fpga_ops_configure_complete, - .state = socfpga_fpga_ops_state, -}; - -static int socfpga_fpga_probe(struct platform_device *pdev) -{ - struct device *dev = &pdev->dev; - struct socfpga_fpga_priv *priv; - int ret; - - priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL); - if (!priv) - return -ENOMEM; - - /* ... do ioremaps, get interrupts, etc. and save - them in priv... */ - - return fpga_mgr_register(dev, "Altera SOCFPGA FPGA Manager", - &socfpga_fpga_ops, priv); -} - -static int socfpga_fpga_remove(struct platform_device *pdev) -{ - fpga_mgr_unregister(&pdev->dev); - - return 0; -} - - -The ops will implement whatever device specific register writes are needed to -do the programming sequence for this particular FPGA. These ops return 0 for -success or negative error codes otherwise. - -The programming sequence is: - 1. .write_init - 2. .write or .write_sg (may be called once or multiple times) - 3. .write_complete - -The .write_init function will prepare the FPGA to receive the image data. The -buffer passed into .write_init will be atmost .initial_header_size bytes long, -if the whole bitstream is not immediately available then the core code will -buffer up at least this much before starting. - -The .write function writes a buffer to the FPGA. The buffer may be contain the -whole FPGA image or may be a smaller chunk of an FPGA image. In the latter -case, this function is called multiple times for successive chunks. This interface -is suitable for drivers which use PIO. - -The .write_sg version behaves the same as .write except the input is a sg_table -scatter list. This interface is suitable for drivers which use DMA. - -The .write_complete function is called after all the image has been written -to put the FPGA into operating mode. - -The ops include a .state function which will read the hardware FPGA manager and -return a code of type enum fpga_mgr_states. It doesn't result in a change in -hardware state. diff --git a/Documentation/fpga/fpga-region.txt b/Documentation/fpga/fpga-region.txt deleted file mode 100644 index 139a02ba1ff6..000000000000 --- a/Documentation/fpga/fpga-region.txt +++ /dev/null @@ -1,95 +0,0 @@ -FPGA Regions - -Alan Tull 2017 - -CONTENTS - - Introduction - - The FPGA region API - - Usage example - -Introduction -============ - -This document is meant to be an brief overview of the FPGA region API usage. A -more conceptual look at regions can be found in [1]. - -For the purposes of this API document, let's just say that a region associates -an FPGA Manager and a bridge (or bridges) with a reprogrammable region of an -FPGA or the whole FPGA. The API provides a way to register a region and to -program a region. - -Currently the only layer above fpga-region.c in the kernel is the Device Tree -support (of-fpga-region.c) described in [1]. The DT support layer uses regions -to program the FPGA and then DT to handle enumeration. The common region code -is intended to be used by other schemes that have other ways of accomplishing -enumeration after programming. - -An fpga-region can be set up to know the following things: -* which FPGA manager to use to do the programming -* which bridges to disable before programming and enable afterwards. - -Additional info needed to program the FPGA image is passed in the struct -fpga_image_info [2] including: -* pointers to the image as either a scatter-gather buffer, a contiguous - buffer, or the name of firmware file -* flags indicating specifics such as whether the image if for partial - reconfiguration. - -=================== -The FPGA region API -=================== - -To register or unregister a region: ------------------------------------ - - int fpga_region_register(struct device *dev, - struct fpga_region *region); - int fpga_region_unregister(struct fpga_region *region); - -An example of usage can be seen in the probe function of [3] - -To program an FPGA: -------------------- - int fpga_region_program_fpga(struct fpga_region *region); - -This function operates on info passed in the fpga_image_info -(region->info). - -This function will attempt to: - * lock the region's mutex - * lock the region's FPGA manager - * build a list of FPGA bridges if a method has been specified to do so - * disable the bridges - * program the FPGA - * re-enable the bridges - * release the locks - -============= -Usage example -============= - -First, allocate the info struct: - - info = fpga_image_info_alloc(dev); - if (!info) - return -ENOMEM; - -Set flags as needed, i.e. - - info->flags |= FPGA_MGR_PARTIAL_RECONFIG; - -Point to your FPGA image, such as: - - info->sgt = &sgt; - -Add info to region and do the programming: - - region->info = info; - ret = fpga_region_program_fpga(region); - -Then enumerate whatever hardware has appeared in the FPGA. - --- -[1] ../devicetree/bindings/fpga/fpga-region.txt -[2] ./fpga-mgr.txt -[3] ../../drivers/fpga/of-fpga-region.c diff --git a/Documentation/fpga/overview.txt b/Documentation/fpga/overview.txt deleted file mode 100644 index 0f1236e7e675..000000000000 --- a/Documentation/fpga/overview.txt +++ /dev/null @@ -1,23 +0,0 @@ -Linux kernel FPGA support - -Alan Tull 2017 - -The main point of this project has been to separate the out the upper layers -that know when to reprogram a FPGA from the lower layers that know how to -reprogram a specific FPGA device. The intention is to make this manufacturer -agnostic, understanding that of course the FPGA images are very device specific -themselves. - -The framework in the kernel includes: -* low level FPGA manager drivers that know how to program a specific device -* the fpga-mgr framework they are registered with -* low level FPGA bridge drivers for hard/soft bridges which are intended to - be disable during FPGA programming -* the fpga-bridge framework they are registered with -* the fpga-region framework which associates and controls managers and bridges - as reconfigurable regions -* the of-fpga-region support for reprogramming FPGAs when device tree overlays - are applied. - -I would encourage you the user to add code that creates FPGA regions rather -that trying to control managers and bridges separately. diff --git a/Documentation/gpu/drivers.rst b/Documentation/gpu/drivers.rst index e8c84419a2a1..f982558fc25d 100644 --- a/Documentation/gpu/drivers.rst +++ b/Documentation/gpu/drivers.rst @@ -10,8 +10,10 @@ GPU Driver Documentation tegra tinydrm tve200 + v3d vc4 bridge/dw-hdmi + xen-front .. only:: subproject and html diff --git a/Documentation/gpu/i915.rst b/Documentation/gpu/i915.rst index 41dc881b00dc..055df45596c1 100644 --- a/Documentation/gpu/i915.rst +++ b/Documentation/gpu/i915.rst @@ -58,6 +58,12 @@ Intel GVT-g Host Support(vGPU device model) .. kernel-doc:: drivers/gpu/drm/i915/intel_gvt.c :internal: +Workarounds +----------- + +.. kernel-doc:: drivers/gpu/drm/i915/intel_workarounds.c + :doc: Hardware workarounds + Display Hardware Handling ========================= @@ -249,6 +255,103 @@ Memory Management and Command Submission This sections covers all things related to the GEM implementation in the i915 driver. +Intel GPU Basics +---------------- + +An Intel GPU has multiple engines. There are several engine types. + +- RCS engine is for rendering 3D and performing compute, this is named + `I915_EXEC_RENDER` in user space. +- BCS is a blitting (copy) engine, this is named `I915_EXEC_BLT` in user + space. +- VCS is a video encode and decode engine, this is named `I915_EXEC_BSD` + in user space +- VECS is video enhancement engine, this is named `I915_EXEC_VEBOX` in user + space. +- The enumeration `I915_EXEC_DEFAULT` does not refer to specific engine; + instead it is to be used by user space to specify a default rendering + engine (for 3D) that may or may not be the same as RCS. + +The Intel GPU family is a family of integrated GPU's using Unified +Memory Access. For having the GPU "do work", user space will feed the +GPU batch buffers via one of the ioctls `DRM_IOCTL_I915_GEM_EXECBUFFER2` +or `DRM_IOCTL_I915_GEM_EXECBUFFER2_WR`. Most such batchbuffers will +instruct the GPU to perform work (for example rendering) and that work +needs memory from which to read and memory to which to write. All memory +is encapsulated within GEM buffer objects (usually created with the ioctl +`DRM_IOCTL_I915_GEM_CREATE`). An ioctl providing a batchbuffer for the GPU +to create will also list all GEM buffer objects that the batchbuffer reads +and/or writes. For implementation details of memory management see +`GEM BO Management Implementation Details`_. + +The i915 driver allows user space to create a context via the ioctl +`DRM_IOCTL_I915_GEM_CONTEXT_CREATE` which is identified by a 32-bit +integer. Such a context should be viewed by user-space as -loosely- +analogous to the idea of a CPU process of an operating system. The i915 +driver guarantees that commands issued to a fixed context are to be +executed so that writes of a previously issued command are seen by +reads of following commands. Actions issued between different contexts +(even if from the same file descriptor) are NOT given that guarantee +and the only way to synchronize across contexts (even from the same +file descriptor) is through the use of fences. At least as far back as +Gen4, also have that a context carries with it a GPU HW context; +the HW context is essentially (most of atleast) the state of a GPU. +In addition to the ordering guarantees, the kernel will restore GPU +state via HW context when commands are issued to a context, this saves +user space the need to restore (most of atleast) the GPU state at the +start of each batchbuffer. The non-deprecated ioctls to submit batchbuffer +work can pass that ID (in the lower bits of drm_i915_gem_execbuffer2::rsvd1) +to identify what context to use with the command. + +The GPU has its own memory management and address space. The kernel +driver maintains the memory translation table for the GPU. For older +GPUs (i.e. those before Gen8), there is a single global such translation +table, a global Graphics Translation Table (GTT). For newer generation +GPUs each context has its own translation table, called Per-Process +Graphics Translation Table (PPGTT). Of important note, is that although +PPGTT is named per-process it is actually per context. When user space +submits a batchbuffer, the kernel walks the list of GEM buffer objects +used by the batchbuffer and guarantees that not only is the memory of +each such GEM buffer object resident but it is also present in the +(PP)GTT. If the GEM buffer object is not yet placed in the (PP)GTT, +then it is given an address. Two consequences of this are: the kernel +needs to edit the batchbuffer submitted to write the correct value of +the GPU address when a GEM BO is assigned a GPU address and the kernel +might evict a different GEM BO from the (PP)GTT to make address room +for another GEM BO. Consequently, the ioctls submitting a batchbuffer +for execution also include a list of all locations within buffers that +refer to GPU-addresses so that the kernel can edit the buffer correctly. +This process is dubbed relocation. + +GEM BO Management Implementation Details +---------------------------------------- + +.. kernel-doc:: drivers/gpu/drm/i915/i915_vma.h + :doc: Virtual Memory Address + +Buffer Object Eviction +---------------------- + +This section documents the interface functions for evicting buffer +objects to make space available in the virtual gpu address spaces. Note +that this is mostly orthogonal to shrinking buffer objects caches, which +has the goal to make main memory (shared with the gpu through the +unified memory architecture) available. + +.. kernel-doc:: drivers/gpu/drm/i915/i915_gem_evict.c + :internal: + +Buffer Object Memory Shrinking +------------------------------ + +This section documents the interface function for shrinking memory usage +of buffer object caches. Shrinking is used to make main memory +available. Note that this is mostly orthogonal to evicting buffer +objects, which has the goal to make space in gpu virtual address spaces. + +.. kernel-doc:: drivers/gpu/drm/i915/i915_gem_shrinker.c + :internal: + Batchbuffer Parsing ------------------- @@ -267,6 +370,12 @@ Batchbuffer Pools .. kernel-doc:: drivers/gpu/drm/i915/i915_gem_batch_pool.c :internal: +User Batchbuffer Execution +-------------------------- + +.. kernel-doc:: drivers/gpu/drm/i915/i915_gem_execbuffer.c + :doc: User command execution + Logical Rings, Logical Ring Contexts and Execlists -------------------------------------------------- @@ -312,28 +421,14 @@ Object Tiling IOCTLs .. kernel-doc:: drivers/gpu/drm/i915/i915_gem_tiling.c :doc: buffer object tiling -Buffer Object Eviction ----------------------- - -This section documents the interface functions for evicting buffer -objects to make space available in the virtual gpu address spaces. Note -that this is mostly orthogonal to shrinking buffer objects caches, which -has the goal to make main memory (shared with the gpu through the -unified memory architecture) available. - -.. kernel-doc:: drivers/gpu/drm/i915/i915_gem_evict.c - :internal: - -Buffer Object Memory Shrinking ------------------------------- +WOPCM +===== -This section documents the interface function for shrinking memory usage -of buffer object caches. Shrinking is used to make main memory -available. Note that this is mostly orthogonal to evicting buffer -objects, which has the goal to make space in gpu virtual address spaces. +WOPCM Layout +------------ -.. kernel-doc:: drivers/gpu/drm/i915/i915_gem_shrinker.c - :internal: +.. kernel-doc:: drivers/gpu/drm/i915/intel_wopcm.c + :doc: WOPCM Layout GuC === @@ -359,6 +454,12 @@ GuC Firmware Layout .. kernel-doc:: drivers/gpu/drm/i915/intel_guc_fwif.h :doc: GuC Firmware Layout +GuC Address Space +----------------- + +.. kernel-doc:: drivers/gpu/drm/i915/intel_guc.c + :doc: GuC Address Space + Tracing ======= diff --git a/Documentation/gpu/kms-properties.csv b/Documentation/gpu/kms-properties.csv index 6b28b014cb7d..07ed22ea3bd6 100644 --- a/Documentation/gpu/kms-properties.csv +++ b/Documentation/gpu/kms-properties.csv @@ -98,5 +98,4 @@ radeon,DVI-I,“coherent”,RANGE,"Min=0, Max=1",Connector,TBD ,,"""underscan vborder""",RANGE,"Min=0, Max=128",Connector,TBD ,Audio,“audio”,ENUM,"{ ""off"", ""on"", ""auto"" }",Connector,TBD ,FMT Dithering,“dither”,ENUM,"{ ""off"", ""on"" }",Connector,TBD -rcar-du,Generic,"""alpha""",RANGE,"Min=0, Max=255",Plane,TBD ,,"""colorkey""",RANGE,"Min=0, Max=0x01ffffff",Plane,TBD diff --git a/Documentation/gpu/todo.rst b/Documentation/gpu/todo.rst index f4d0b3476d9c..a7c150d6b63f 100644 --- a/Documentation/gpu/todo.rst +++ b/Documentation/gpu/todo.rst @@ -212,6 +212,24 @@ probably use drm_fb_helper_fbdev_teardown(). Contact: Maintainer of the driver you plan to convert +Clean up mmap forwarding +------------------------ + +A lot of drivers forward gem mmap calls to dma-buf mmap for imported buffers. +And also a lot of them forward dma-buf mmap to the gem mmap implementations. +Would be great to refactor this all into a set of small common helpers. + +Contact: Daniel Vetter + +Put a reservation_object into drm_gem_object +-------------------------------------------- + +This would remove the need for the ->gem_prime_res_obj callback. It would also +allow us to implement generic helpers for waiting for a bo, allowing for quite a +bit of refactoring in the various wait ioctl implementations. + +Contact: Daniel Vetter + idr_init_base() --------------- diff --git a/Documentation/gpu/xen-front.rst b/Documentation/gpu/xen-front.rst new file mode 100644 index 000000000000..d988da7d1983 --- /dev/null +++ b/Documentation/gpu/xen-front.rst @@ -0,0 +1,31 @@ +==================================================== + drm/xen-front Xen para-virtualized frontend driver +==================================================== + +This frontend driver implements Xen para-virtualized display +according to the display protocol described at +include/xen/interface/io/displif.h + +Driver modes of operation in terms of display buffers used +========================================================== + +.. kernel-doc:: drivers/gpu/drm/xen/xen_drm_front.h + :doc: Driver modes of operation in terms of display buffers used + +Buffers allocated by the frontend driver +---------------------------------------- + +.. kernel-doc:: drivers/gpu/drm/xen/xen_drm_front.h + :doc: Buffers allocated by the frontend driver + +Buffers allocated by the backend +-------------------------------- + +.. kernel-doc:: drivers/gpu/drm/xen/xen_drm_front.h + :doc: Buffers allocated by the backend + +Driver limitations +================== + +.. kernel-doc:: drivers/gpu/drm/xen/xen_drm_front.h + :doc: Driver limitations diff --git a/Documentation/hwmon/hwmon-kernel-api.txt b/Documentation/hwmon/hwmon-kernel-api.txt index 53a806696c64..eb7a78aebb38 100644 --- a/Documentation/hwmon/hwmon-kernel-api.txt +++ b/Documentation/hwmon/hwmon-kernel-api.txt @@ -71,7 +71,8 @@ hwmon_device_register_with_info is the most comprehensive and preferred means to register a hardware monitoring device. It creates the standard sysfs attributes in the hardware monitoring core, letting the driver focus on reading from and writing to the chip instead of having to bother with sysfs attributes. -Its parameters are described in more detail below. +The parent device parameter cannot be NULL with non-NULL chip info. Its +parameters are described in more detail below. devm_hwmon_device_register_with_info is similar to hwmon_device_register_with_info. However, it is device managed, meaning the diff --git a/Documentation/hwmon/ina2xx b/Documentation/hwmon/ina2xx index cfd31d94c872..72d16f08e431 100644 --- a/Documentation/hwmon/ina2xx +++ b/Documentation/hwmon/ina2xx @@ -53,7 +53,7 @@ bus supply voltage. The shunt value in micro-ohms can be set via platform data or device tree at compile-time or via the shunt_resistor attribute in sysfs at run-time. Please -refer to the Documentation/devicetree/bindings/i2c/ina2xx.txt for bindings +refer to the Documentation/devicetree/bindings/hwmon/ina2xx.txt for bindings if the device tree is used. Additionally ina226 supports update_interval attribute as described in diff --git a/Documentation/hwmon/ltc2990 b/Documentation/hwmon/ltc2990 index c25211e90bdc..3ed68f676c0f 100644 --- a/Documentation/hwmon/ltc2990 +++ b/Documentation/hwmon/ltc2990 @@ -8,6 +8,7 @@ Supported chips: Datasheet: http://www.linear.com/product/ltc2990 Author: Mike Looijmans + Tom Levens Description @@ -16,10 +17,8 @@ Description LTC2990 is a Quad I2C Voltage, Current and Temperature Monitor. The chip's inputs can measure 4 voltages, or two inputs together (1+2 and 3+4) can be combined to measure a differential voltage, which is typically used to -measure current through a series resistor, or a temperature. - -This driver currently uses the 2x differential mode only. In order to support -other modes, the driver will need to be expanded. +measure current through a series resistor, or a temperature with an external +diode. Usage Notes @@ -32,12 +31,19 @@ devices explicitly. Sysfs attributes ---------------- +in0_input Voltage at Vcc pin in millivolt (range 2.5V to 5V) +temp1_input Internal chip temperature in millidegrees Celcius + +A subset of the following attributes are visible, depending on the measurement +mode of the chip. + +in[1-4]_input Voltage at V[1-4] pin in millivolt +temp2_input External temperature sensor TR1 in millidegrees Celcius +temp3_input External temperature sensor TR2 in millidegrees Celcius +curr1_input Current in mA across V1-V2 assuming a 1mOhm sense resistor +curr2_input Current in mA across V3-V4 assuming a 1mOhm sense resistor + The "curr*_input" measurements actually report the voltage drop across the input pins in microvolts. This is equivalent to the current through a 1mOhm sense resistor. Divide the reported value by the actual sense resistor value in mOhm to get the actual value. - -in0_input Voltage at Vcc pin in millivolt (range 2.5V to 5V) -temp1_input Internal chip temperature in millidegrees Celcius -curr1_input Current in mA across v1-v2 assuming a 1mOhm sense resistor. -curr2_input Current in mA across v3-v4 assuming a 1mOhm sense resistor. diff --git a/Documentation/i2c/busses/i2c-mlxcpld b/Documentation/i2c/busses/i2c-mlxcpld index 4e46c440b38d..925904aa9b57 100644 --- a/Documentation/i2c/busses/i2c-mlxcpld +++ b/Documentation/i2c/busses/i2c-mlxcpld @@ -20,6 +20,10 @@ The next transaction types are supported: - Write Byte/Block. Registers: +CPBLTY 0x0 - capability reg. + Bits [6:5] - transaction length. b01 - 72B is supported, + 36B in other case. + Bit 7 - SMBus block read support. CTRL 0x1 - control reg. Resets all the registers. HALF_CYC 0x4 - cycle reg. diff --git a/Documentation/i2c/busses/i2c-ocores b/Documentation/i2c/busses/i2c-ocores index c269aaa2f26a..4e713f4cdb2f 100644 --- a/Documentation/i2c/busses/i2c-ocores +++ b/Documentation/i2c/busses/i2c-ocores @@ -2,7 +2,7 @@ Kernel driver i2c-ocores Supported adapters: * OpenCores.org I2C controller by Richard Herveille (see datasheet link) - Datasheet: http://www.opencores.org/projects.cgi/web/i2c/overview + https://opencores.org/project/i2c/overview Author: Peter Korsgaard @@ -18,7 +18,7 @@ Usage i2c-ocores uses the platform bus, so you need to provide a struct platform_device with the base address and interrupt number. The dev.platform_data of the device should also point to a struct -ocores_i2c_platform_data (see linux/i2c-ocores.h) describing the +ocores_i2c_platform_data (see linux/platform_data/i2c-ocores.h) describing the distance between registers and the input clock speed. There is also a possibility to attach a list of i2c_board_info which the i2c-ocores driver will add to the bus upon creation. diff --git a/Documentation/i2c/muxes/i2c-mux-gpio b/Documentation/i2c/muxes/i2c-mux-gpio index 7a8d7d261632..893ecdfe6e43 100644 --- a/Documentation/i2c/muxes/i2c-mux-gpio +++ b/Documentation/i2c/muxes/i2c-mux-gpio @@ -30,12 +30,12 @@ i2c-mux-gpio uses the platform bus, so you need to provide a struct platform_device with the platform_data pointing to a struct i2c_mux_gpio_platform_data with the I2C adapter number of the master bus, the number of bus segments to create and the GPIO pins used -to control it. See include/linux/i2c-mux-gpio.h for details. +to control it. See include/linux/platform_data/i2c-mux-gpio.h for details. E.G. something like this for a MUX providing 4 bus segments controlled through 3 GPIO pins: -#include +#include #include static const unsigned myboard_gpiomux_gpios[] = { diff --git a/Documentation/index.rst b/Documentation/index.rst index 3b99ab931d41..fdc585703498 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -45,7 +45,7 @@ the kernel interface as seen by application developers. .. toctree:: :maxdepth: 2 - userspace-api/index + userspace-api/index Introduction to kernel development @@ -89,6 +89,7 @@ needed). sound/index crypto/index filesystems/index + vm/index Architecture-specific documentation ----------------------------------- diff --git a/Documentation/ioctl/botching-up-ioctls.txt b/Documentation/ioctl/botching-up-ioctls.txt index d02cfb48901c..883fb034bd04 100644 --- a/Documentation/ioctl/botching-up-ioctls.txt +++ b/Documentation/ioctl/botching-up-ioctls.txt @@ -73,7 +73,9 @@ will have a second iteration or at least an extension for any given interface. future extensions is going right down the gutters since someone will submit an ioctl struct with random stack garbage in the yet unused parts. Which then bakes in the ABI that those fields can never be used for anything else - but garbage. + but garbage. This is also the reason why you must explicitly pad all + structures, even if you never use them in an array - the padding the compiler + might insert could contain garbage. * Have simple testcases for all of the above. diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt index 7f7413e597f3..480c8609dc58 100644 --- a/Documentation/ioctl/ioctl-number.txt +++ b/Documentation/ioctl/ioctl-number.txt @@ -151,7 +151,7 @@ Code Seq#(hex) Include File Comments 'J' 00-1F drivers/scsi/gdth_ioctl.h 'K' all linux/kd.h 'L' 00-1F linux/loop.h conflict! -'L' 10-1F drivers/scsi/mpt2sas/mpt2sas_ctl.h conflict! +'L' 10-1F drivers/scsi/mpt3sas/mpt3sas_ctl.h conflict! 'L' 20-2F linux/lightnvm.h 'L' E0-FF linux/ppdd.h encrypted disk device driver @@ -296,7 +296,8 @@ Code Seq#(hex) Include File Comments 0x90 00 drivers/cdrom/sbpcd.h 0x92 00-0F drivers/usb/mon/mon_bin.c 0x93 60-7F linux/auto_fs.h -0x94 all fs/btrfs/ioctl.h +0x94 all fs/btrfs/ioctl.h Btrfs filesystem + and linux/fs.h some lifted to vfs/generic 0x97 00-7F fs/ceph/ioctl.h Ceph file system 0x99 00-0F 537-Addinboard driver @@ -327,6 +328,7 @@ Code Seq#(hex) Include File Comments 0xCA 80-BF uapi/scsi/cxlflash_ioctl.h 0xCB 00-1F CBM serial IEC bus in development: +0xCC 00-0F drivers/misc/ibmvmc.h pseries VMC driver 0xCD 01 linux/reiserfs_fs.h 0xCF 02 fs/cifs/ioctl.c 0xDB 00-0F drivers/char/mwave/mwavepub.h diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt index f5b9493f04ad..3534a84d206c 100644 --- a/Documentation/kbuild/kconfig-language.txt +++ b/Documentation/kbuild/kconfig-language.txt @@ -198,14 +198,6 @@ applicable everywhere (see syntax). enables the third modular state for all config symbols. At most one symbol may have the "modules" option set. - - "env"= - This imports the environment variable into Kconfig. It behaves like - a default, except that the value comes from the environment, this - also means that the behaviour when mixing it with normal defaults is - undefined at this point. The symbol is currently not exported back - to the build environment (if this is desired, it can be done via - another symbol). - - "allnoconfig_y" This declares the symbol as one that should have the value y when using "allnoconfig". Used for symbols that hide other symbols. @@ -481,6 +473,24 @@ config option to 'y' no matter the dependencies. The dependencies are moved to the symbol GENERIC_IOMAP and we avoid the situation where select forces a symbol equals to 'y'. +Adding features that need compiler support +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are several features that need compiler support. The recommended way +to describe the dependency on the compiler feature is to use "depends on" +followed by a test macro. + +config STACKPROTECTOR + bool "Stack Protector buffer overflow detection" + depends on $(cc-option,-fstack-protector) + ... + +If you need to expose a compiler capability to makefiles and/or C source files, +CC_HAS_ is the recommended prefix for the config option. + +config CC_HAS_STACKPROTECTOR_NONE + def_bool $(cc-option,-fno-stack-protector) + Build as module only ~~~~~~~~~~~~~~~~~~~~ To restrict a component build to module-only, qualify its config symbol diff --git a/Documentation/kbuild/kconfig-macro-language.txt b/Documentation/kbuild/kconfig-macro-language.txt new file mode 100644 index 000000000000..07da2ea68dce --- /dev/null +++ b/Documentation/kbuild/kconfig-macro-language.txt @@ -0,0 +1,242 @@ +Concept +------- + +The basic idea was inspired by Make. When we look at Make, we notice sort of +two languages in one. One language describes dependency graphs consisting of +targets and prerequisites. The other is a macro language for performing textual +substitution. + +There is clear distinction between the two language stages. For example, you +can write a makefile like follows: + + APP := foo + SRC := foo.c + CC := gcc + + $(APP): $(SRC) + $(CC) -o $(APP) $(SRC) + +The macro language replaces the variable references with their expanded form, +and handles as if the source file were input like follows: + + foo: foo.c + gcc -o foo foo.c + +Then, Make analyzes the dependency graph and determines the targets to be +updated. + +The idea is quite similar in Kconfig - it is possible to describe a Kconfig +file like this: + + CC := gcc + + config CC_HAS_FOO + def_bool $(shell, $(srctree)/scripts/gcc-check-foo.sh $(CC)) + +The macro language in Kconfig processes the source file into the following +intermediate: + + config CC_HAS_FOO + def_bool y + +Then, Kconfig moves onto the evaluation stage to resolve inter-symbol +dependency as explained in kconfig-language.txt. + + +Variables +--------- + +Like in Make, a variable in Kconfig works as a macro variable. A macro +variable is expanded "in place" to yield a text string that may then be +expanded further. To get the value of a variable, enclose the variable name in +$( ). The parentheses are required even for single-letter variable names; $X is +a syntax error. The curly brace form as in ${CC} is not supported either. + +There are two types of variables: simply expanded variables and recursively +expanded variables. + +A simply expanded variable is defined using the := assignment operator. Its +righthand side is expanded immediately upon reading the line from the Kconfig +file. + +A recursively expanded variable is defined using the = assignment operator. +Its righthand side is simply stored as the value of the variable without +expanding it in any way. Instead, the expansion is performed when the variable +is used. + +There is another type of assignment operator; += is used to append text to a +variable. The righthand side of += is expanded immediately if the lefthand +side was originally defined as a simple variable. Otherwise, its evaluation is +deferred. + +The variable reference can take parameters, in the following form: + + $(name,arg1,arg2,arg3) + +You can consider the parameterized reference as a function. (more precisely, +"user-defined function" in contrast to "built-in function" listed below). + +Useful functions must be expanded when they are used since the same function is +expanded differently if different parameters are passed. Hence, a user-defined +function is defined using the = assignment operator. The parameters are +referenced within the body definition with $(1), $(2), etc. + +In fact, recursively expanded variables and user-defined functions are the same +internally. (In other words, "variable" is "function with zero argument".) +When we say "variable" in a broad sense, it includes "user-defined function". + + +Built-in functions +------------------ + +Like Make, Kconfig provides several built-in functions. Every function takes a +particular number of arguments. + +In Make, every built-in function takes at least one argument. Kconfig allows +zero argument for built-in functions, such as $(fileno), $(lineno). You could +consider those as "built-in variable", but it is just a matter of how we call +it after all. Let's say "built-in function" here to refer to natively supported +functionality. + +Kconfig currently supports the following built-in functions. + + - $(shell,command) + + The "shell" function accepts a single argument that is expanded and passed + to a subshell for execution. The standard output of the command is then read + and returned as the value of the function. Every newline in the output is + replaced with a space. Any trailing newlines are deleted. The standard error + is not returned, nor is any program exit status. + + - $(info,text) + + The "info" function takes a single argument and prints it to stdout. + It evaluates to an empty string. + + - $(warning-if,condition,text) + + The "warning-if" function takes two arguments. If the condition part is "y", + the text part is sent to stderr. The text is prefixed with the name of the + current Kconfig file and the current line number. + + - $(error-if,condition,text) + + The "error-if" function is similar to "warning-if", but it terminates the + parsing immediately if the condition part is "y". + + - $(filename) + + The 'filename' takes no argument, and $(filename) is expanded to the file + name being parsed. + + - $(lineno) + + The 'lineno' takes no argument, and $(lineno) is expanded to the line number + being parsed. + + +Make vs Kconfig +--------------- + +Kconfig adopts Make-like macro language, but the function call syntax is +slightly different. + +A function call in Make looks like this: + + $(func-name arg1,arg2,arg3) + +The function name and the first argument are separated by at least one +whitespace. Then, leading whitespaces are trimmed from the first argument, +while whitespaces in the other arguments are kept. You need to use a kind of +trick to start the first parameter with spaces. For example, if you want +to make "info" function print " hello", you can write like follows: + + empty := + space := $(empty) $(empty) + $(info $(space)$(space)hello) + +Kconfig uses only commas for delimiters, and keeps all whitespaces in the +function call. Some people prefer putting a space after each comma delimiter: + + $(func-name, arg1, arg2, arg3) + +In this case, "func-name" will receive " arg1", " arg2", " arg3". The presence +of leading spaces may matter depending on the function. The same applies to +Make - for example, $(subst .c, .o, $(sources)) is a typical mistake; it +replaces ".c" with " .o". + +In Make, a user-defined function is referenced by using a built-in function, +'call', like this: + + $(call my-func,arg1,arg2,arg3) + +Kconfig invokes user-defined functions and built-in functions in the same way. +The omission of 'call' makes the syntax shorter. + +In Make, some functions treat commas verbatim instead of argument separators. +For example, $(shell echo hello, world) runs the command "echo hello, world". +Likewise, $(info hello, world) prints "hello, world" to stdout. You could say +this is _useful_ inconsistency. + +In Kconfig, for simpler implementation and grammatical consistency, commas that +appear in the $( ) context are always delimiters. It means + + $(shell, echo hello, world) + +is an error because it is passing two parameters where the 'shell' function +accepts only one. To pass commas in arguments, you can use the following trick: + + comma := , + $(shell, echo hello$(comma) world) + + +Caveats +------- + +A variable (or function) cannot be expanded across tokens. So, you cannot use +a variable as a shorthand for an expression that consists of multiple tokens. +The following works: + + RANGE_MIN := 1 + RANGE_MAX := 3 + + config FOO + int "foo" + range $(RANGE_MIN) $(RANGE_MAX) + +But, the following does not work: + + RANGES := 1 3 + + config FOO + int "foo" + range $(RANGES) + +A variable cannot be expanded to any keyword in Kconfig. The following does +not work: + + MY_TYPE := tristate + + config FOO + $(MY_TYPE) "foo" + default y + +Obviously from the design, $(shell command) is expanded in the textual +substitution phase. You cannot pass symbols to the 'shell' function. +The following does not work as expected. + + config ENDIAN_FLAG + string + default "-mbig-endian" if CPU_BIG_ENDIAN + default "-mlittle-endian" if CPU_LITTLE_ENDIAN + + config CC_HAS_ENDIAN_FLAG + def_bool $(shell $(srctree)/scripts/gcc-check-flag ENDIAN_FLAG) + +Instead, you can do like follows so that any function call is statically +expanded. + + config CC_HAS_ENDIAN_FLAG + bool + default $(shell $(srctree)/scripts/gcc-check-flag -mbig-endian) if CPU_BIG_ENDIAN + default $(shell $(srctree)/scripts/gcc-check-flag -mlittle-endian) if CPU_LITTLE_ENDIAN diff --git a/Documentation/kprobes.txt b/Documentation/kprobes.txt index 22208bf2386d..cb3b0de83fc6 100644 --- a/Documentation/kprobes.txt +++ b/Documentation/kprobes.txt @@ -724,8 +724,8 @@ migrate your tool to one of the following options: See following documents: - - Documentation/trace/kprobetrace.txt - - Documentation/trace/events.txt + - Documentation/trace/kprobetrace.rst + - Documentation/trace/events.rst - tools/perf/Documentation/perf-probe.txt diff --git a/Documentation/laptops/thinkpad-acpi.txt b/Documentation/laptops/thinkpad-acpi.txt index 00b6dfed573c..6cced88de6da 100644 --- a/Documentation/laptops/thinkpad-acpi.txt +++ b/Documentation/laptops/thinkpad-acpi.txt @@ -540,8 +540,10 @@ Events that are propagated by the driver to userspace: 0x6021 ALARM: a sensor is too hot 0x6022 ALARM: a sensor is extremely hot 0x6030 System thermal table changed +0x6032 Thermal Control command set completion (DYTC, Windows) 0x6040 Nvidia Optimus/AC adapter related (TO BE VERIFIED) 0x60C0 X1 Yoga 2016, Tablet mode status changed +0x60F0 Thermal Transformation changed (GMTS, Windows) Battery nearly empty alarms are a last resort attempt to get the operating system to hibernate or shutdown cleanly (0x2313), or shutdown diff --git a/Documentation/livepatch/livepatch.txt b/Documentation/livepatch/livepatch.txt index 1ae2de758c08..2d7ed09dbd59 100644 --- a/Documentation/livepatch/livepatch.txt +++ b/Documentation/livepatch/livepatch.txt @@ -429,30 +429,6 @@ See Documentation/ABI/testing/sysfs-kernel-livepatch for more details. The current Livepatch implementation has several limitations: - - + The patch must not change the semantic of the patched functions. - - The current implementation guarantees only that either the old - or the new function is called. The functions are patched one - by one. It means that the patch must _not_ change the semantic - of the function. - - - + Data structures can not be patched. - - There is no support to version data structures or anyhow migrate - one structure into another. Also the simple consistency model does - not allow to switch more functions atomically. - - Once there is more complex consistency mode, it will be possible to - use some workarounds. For example, it will be possible to use a hole - for a new member because the data structure is aligned. Or it will - be possible to use an existing member for something else. - - There are no plans to add more generic support for modified structures - at the moment. - - + Only functions that can be traced could be patched. Livepatch is based on the dynamic ftrace. In particular, functions diff --git a/Documentation/maintainer/pull-requests.rst b/Documentation/maintainer/pull-requests.rst index a19db3458b56..22b271de0304 100644 --- a/Documentation/maintainer/pull-requests.rst +++ b/Documentation/maintainer/pull-requests.rst @@ -41,7 +41,7 @@ named ``char-misc-next``, you would be using the following command:: that will create a signed tag called ``char-misc-4.15-rc1`` based on the last commit in the ``char-misc-next`` branch, and sign it with your gpg key -(see :ref:`Documentation/maintainer/configure_git.rst `). +(see :ref:`Documentation/maintainer/configure-git.rst `). Linus will only accept pull requests based on a signed tag. Other maintainers may differ. diff --git a/Documentation/media/kapi/cec-core.rst b/Documentation/media/kapi/cec-core.rst index a9f53f069a2d..1d989c544370 100644 --- a/Documentation/media/kapi/cec-core.rst +++ b/Documentation/media/kapi/cec-core.rst @@ -246,7 +246,10 @@ CEC_TX_STATUS_LOW_DRIVE: CEC_TX_STATUS_ERROR: some unspecified error occurred: this can be one of ARB_LOST or LOW_DRIVE if the hardware cannot differentiate or something - else entirely. + else entirely. Some hardware only supports OK and FAIL as the + result of a transmit, i.e. there is no way to differentiate + between the different possible errors. In that case map FAIL + to CEC_TX_STATUS_NACK and not to CEC_TX_STATUS_ERROR. CEC_TX_STATUS_MAX_RETRIES: could not transmit the message after trying multiple times. diff --git a/Documentation/media/uapi/cec/cec-ioc-receive.rst b/Documentation/media/uapi/cec/cec-ioc-receive.rst index bdad4b197bcd..e964074cd15b 100644 --- a/Documentation/media/uapi/cec/cec-ioc-receive.rst +++ b/Documentation/media/uapi/cec/cec-ioc-receive.rst @@ -231,26 +231,32 @@ View On' messages from initiator 0xf ('Unregistered') to destination 0 ('TV'). - ``CEC_TX_STATUS_OK`` - 0x01 - The message was transmitted successfully. This is mutually - exclusive with :ref:`CEC_TX_STATUS_MAX_RETRIES `. Other bits can still - be set if earlier attempts met with failure before the transmit - was eventually successful. + exclusive with :ref:`CEC_TX_STATUS_MAX_RETRIES `. + Other bits can still be set if earlier attempts met with failure before + the transmit was eventually successful. * .. _`CEC-TX-STATUS-ARB-LOST`: - ``CEC_TX_STATUS_ARB_LOST`` - 0x02 - - CEC line arbitration was lost. + - CEC line arbitration was lost, i.e. another transmit started at the + same time with a higher priority. Optional status, not all hardware + can detect this error condition. * .. _`CEC-TX-STATUS-NACK`: - ``CEC_TX_STATUS_NACK`` - 0x04 - - Message was not acknowledged. + - Message was not acknowledged. Note that some hardware cannot tell apart + a 'Not Acknowledged' status from other error conditions, i.e. the result + of a transmit is just OK or FAIL. In that case this status will be + returned when the transmit failed. * .. _`CEC-TX-STATUS-LOW-DRIVE`: - ``CEC_TX_STATUS_LOW_DRIVE`` - 0x08 - Low drive was detected on the CEC bus. This indicates that a follower detected an error on the bus and requests a - retransmission. + retransmission. Optional status, not all hardware can detect this + error condition. * .. _`CEC-TX-STATUS-ERROR`: - ``CEC_TX_STATUS_ERROR`` @@ -258,14 +264,14 @@ View On' messages from initiator 0xf ('Unregistered') to destination 0 ('TV'). - Some error occurred. This is used for any errors that do not fit ``CEC_TX_STATUS_ARB_LOST`` or ``CEC_TX_STATUS_LOW_DRIVE``, either because the hardware could not tell which error occurred, or because the hardware - tested for other conditions besides those two. + tested for other conditions besides those two. Optional status. * .. _`CEC-TX-STATUS-MAX-RETRIES`: - ``CEC_TX_STATUS_MAX_RETRIES`` - 0x20 - The transmit failed after one or more retries. This status bit is - mutually exclusive with :ref:`CEC_TX_STATUS_OK `. Other bits can still - be set to explain which failures were seen. + mutually exclusive with :ref:`CEC_TX_STATUS_OK `. + Other bits can still be set to explain which failures were seen. .. tabularcolumns:: |p{5.6cm}|p{0.9cm}|p{11.0cm}| diff --git a/Documentation/media/uapi/dvb/dvbapi.rst b/Documentation/media/uapi/dvb/dvbapi.rst index 18c86b3a3af1..89ddca38626f 100644 --- a/Documentation/media/uapi/dvb/dvbapi.rst +++ b/Documentation/media/uapi/dvb/dvbapi.rst @@ -62,7 +62,7 @@ Authors: - Original author of the Digital TV API documentation. -- Carvalho Chehab, Mauro +- Carvalho Chehab, Mauro - Ported document to Docbook XML, addition of DVBv5 API, documentation gaps fix. diff --git a/Documentation/media/uapi/rc/lirc-dev-intro.rst b/Documentation/media/uapi/rc/lirc-dev-intro.rst index 698e4f80270e..11516c8bff62 100644 --- a/Documentation/media/uapi/rc/lirc-dev-intro.rst +++ b/Documentation/media/uapi/rc/lirc-dev-intro.rst @@ -18,7 +18,7 @@ Example dmesg output upon a driver registering w/LIRC: .. code-block:: none $ dmesg |grep lirc_dev - rc rc0: lirc_dev: driver mceusb registered at minor = 0 + rc rc0: lirc_dev: driver mceusb registered at minor = 0, raw IR receiver, raw IR transmitter What you should see for a chardev: diff --git a/Documentation/media/uapi/rc/lirc-set-rec-timeout.rst b/Documentation/media/uapi/rc/lirc-set-rec-timeout.rst index b3e16bbdbc90..a833a6a4c25a 100644 --- a/Documentation/media/uapi/rc/lirc-set-rec-timeout.rst +++ b/Documentation/media/uapi/rc/lirc-set-rec-timeout.rst @@ -1,19 +1,23 @@ .. -*- coding: utf-8; mode: rst -*- .. _lirc_set_rec_timeout: +.. _lirc_get_rec_timeout: -************************** -ioctl LIRC_SET_REC_TIMEOUT -************************** +*************************************************** +ioctl LIRC_GET_REC_TIMEOUT and LIRC_SET_REC_TIMEOUT +*************************************************** Name ==== -LIRC_SET_REC_TIMEOUT - sets the integer value for IR inactivity timeout. +LIRC_GET_REC_TIMEOUT/LIRC_SET_REC_TIMEOUT - Get/set the integer value for IR inactivity timeout. Synopsis ======== +.. c:function:: int ioctl( int fd, LIRC_GET_REC_TIMEOUT, __u32 *timeout ) + :name: LIRC_GET_REC_TIMEOUT + .. c:function:: int ioctl( int fd, LIRC_SET_REC_TIMEOUT, __u32 *timeout ) :name: LIRC_SET_REC_TIMEOUT @@ -30,7 +34,7 @@ Arguments Description =========== -Sets the integer value for IR inactivity timeout. +Get and set the integer value for IR inactivity timeout. If supported by the hardware, setting it to 0 disables all hardware timeouts and data should be reported as soon as possible. If the exact value diff --git a/Documentation/media/uapi/v4l/common.rst b/Documentation/media/uapi/v4l/common.rst index 13f2ed3fc5a6..5f93e71122ef 100644 --- a/Documentation/media/uapi/v4l/common.rst +++ b/Documentation/media/uapi/v4l/common.rst @@ -41,6 +41,6 @@ applicable to all devices. extended-controls format planar-apis - crop selection-api + crop streaming-par diff --git a/Documentation/media/uapi/v4l/crop.rst b/Documentation/media/uapi/v4l/crop.rst index 182565b9ace4..45e8a895a320 100644 --- a/Documentation/media/uapi/v4l/crop.rst +++ b/Documentation/media/uapi/v4l/crop.rst @@ -2,9 +2,18 @@ .. _crop: -************************************* -Image Cropping, Insertion and Scaling -************************************* +***************************************************** +Image Cropping, Insertion and Scaling -- the CROP API +***************************************************** + +.. note:: + + The CROP API is mostly superseded by the newer :ref:`SELECTION API + `. The new API should be preferred in most cases, + with the exception of pixel aspect ratio detection, which is + implemented by :ref:`VIDIOC_CROPCAP ` and has no + equivalent in the SELECTION API. See :ref:`selection-vs-crop` for a + comparison of the two APIs. Some video capture devices can sample a subsection of the picture and shrink or enlarge it to an image of arbitrary size. We call these @@ -42,10 +51,9 @@ where applicable) will be fixed in this case. .. note:: - All capture and output devices must support the - :ref:`VIDIOC_CROPCAP ` ioctl such that applications - can determine if scaling takes place. - + All capture and output devices that support the CROP or SELECTION + API will also support the :ref:`VIDIOC_CROPCAP ` + ioctl. Cropping Structures =================== diff --git a/Documentation/media/uapi/v4l/selection-api-004.rst b/Documentation/media/uapi/v4l/selection-api-004.rst deleted file mode 100644 index d782cd5b2117..000000000000 --- a/Documentation/media/uapi/v4l/selection-api-004.rst +++ /dev/null @@ -1,137 +0,0 @@ -.. -*- coding: utf-8; mode: rst -*- - -************* -Configuration -************* - -Applications can use the :ref:`selection API ` to -select an area in a video signal or a buffer, and to query for default -settings and hardware limits. - -Video hardware can have various cropping, composing and scaling -limitations. It may only scale up or down, support only discrete scaling -factors, or have different scaling abilities in the horizontal and -vertical directions. Also it may not support scaling at all. At the same -time the cropping/composing rectangles may have to be aligned, and both -the source and the sink may have arbitrary upper and lower size limits. -Therefore, as usual, drivers are expected to adjust the requested -parameters and return the actual values selected. An application can -control the rounding behaviour using -:ref:`constraint flags `. - - -Configuration of video capture -============================== - -See figure :ref:`sel-targets-capture` for examples of the selection -targets available for a video capture device. It is recommended to -configure the cropping targets before to the composing targets. - -The range of coordinates of the top left corner, width and height of -areas that can be sampled is given by the ``V4L2_SEL_TGT_CROP_BOUNDS`` -target. It is recommended for the driver developers to put the top/left -corner at position ``(0,0)``. The rectangle's coordinates are expressed -in pixels. - -The top left corner, width and height of the source rectangle, that is -the area actually sampled, is given by the ``V4L2_SEL_TGT_CROP`` target. -It uses the same coordinate system as ``V4L2_SEL_TGT_CROP_BOUNDS``. The -active cropping area must lie completely inside the capture boundaries. -The driver may further adjust the requested size and/or position -according to hardware limitations. - -Each capture device has a default source rectangle, given by the -``V4L2_SEL_TGT_CROP_DEFAULT`` target. This rectangle shall over what the -driver writer considers the complete picture. Drivers shall set the -active crop rectangle to the default when the driver is first loaded, -but not later. - -The composing targets refer to a memory buffer. The limits of composing -coordinates are obtained using ``V4L2_SEL_TGT_COMPOSE_BOUNDS``. All -coordinates are expressed in pixels. The rectangle's top/left corner -must be located at position ``(0,0)``. The width and height are equal to -the image size set by :ref:`VIDIOC_S_FMT `. - -The part of a buffer into which the image is inserted by the hardware is -controlled by the ``V4L2_SEL_TGT_COMPOSE`` target. The rectangle's -coordinates are also expressed in the same coordinate system as the -bounds rectangle. The composing rectangle must lie completely inside -bounds rectangle. The driver must adjust the composing rectangle to fit -to the bounding limits. Moreover, the driver can perform other -adjustments according to hardware limitations. The application can -control rounding behaviour using -:ref:`constraint flags `. - -For capture devices the default composing rectangle is queried using -``V4L2_SEL_TGT_COMPOSE_DEFAULT``. It is usually equal to the bounding -rectangle. - -The part of a buffer that is modified by the hardware is given by -``V4L2_SEL_TGT_COMPOSE_PADDED``. It contains all pixels defined using -``V4L2_SEL_TGT_COMPOSE`` plus all padding data modified by hardware -during insertion process. All pixels outside this rectangle *must not* -be changed by the hardware. The content of pixels that lie inside the -padded area but outside active area is undefined. The application can -use the padded and active rectangles to detect where the rubbish pixels -are located and remove them if needed. - - -Configuration of video output -============================= - -For output devices targets and ioctls are used similarly to the video -capture case. The *composing* rectangle refers to the insertion of an -image into a video signal. The cropping rectangles refer to a memory -buffer. It is recommended to configure the composing targets before to -the cropping targets. - -The cropping targets refer to the memory buffer that contains an image -to be inserted into a video signal or graphical screen. The limits of -cropping coordinates are obtained using ``V4L2_SEL_TGT_CROP_BOUNDS``. -All coordinates are expressed in pixels. The top/left corner is always -point ``(0,0)``. The width and height is equal to the image size -specified using :ref:`VIDIOC_S_FMT ` ioctl. - -The top left corner, width and height of the source rectangle, that is -the area from which image date are processed by the hardware, is given -by the ``V4L2_SEL_TGT_CROP``. Its coordinates are expressed in in the -same coordinate system as the bounds rectangle. The active cropping area -must lie completely inside the crop boundaries and the driver may -further adjust the requested size and/or position according to hardware -limitations. - -For output devices the default cropping rectangle is queried using -``V4L2_SEL_TGT_CROP_DEFAULT``. It is usually equal to the bounding -rectangle. - -The part of a video signal or graphics display where the image is -inserted by the hardware is controlled by ``V4L2_SEL_TGT_COMPOSE`` -target. The rectangle's coordinates are expressed in pixels. The -composing rectangle must lie completely inside the bounds rectangle. The -driver must adjust the area to fit to the bounding limits. Moreover, the -driver can perform other adjustments according to hardware limitations. - -The device has a default composing rectangle, given by the -``V4L2_SEL_TGT_COMPOSE_DEFAULT`` target. This rectangle shall cover what -the driver writer considers the complete picture. It is recommended for -the driver developers to put the top/left corner at position ``(0,0)``. -Drivers shall set the active composing rectangle to the default one when -the driver is first loaded. - -The devices may introduce additional content to video signal other than -an image from memory buffers. It includes borders around an image. -However, such a padded area is driver-dependent feature not covered by -this document. Driver developers are encouraged to keep padded rectangle -equal to active one. The padded target is accessed by the -``V4L2_SEL_TGT_COMPOSE_PADDED`` identifier. It must contain all pixels -from the ``V4L2_SEL_TGT_COMPOSE`` target. - - -Scaling control -=============== - -An application can detect if scaling is performed by comparing the width -and the height of rectangles obtained using ``V4L2_SEL_TGT_CROP`` and -``V4L2_SEL_TGT_COMPOSE`` targets. If these are not equal then the -scaling is applied. The application can compute the scaling ratios using -these values. diff --git a/Documentation/media/uapi/v4l/selection-api-005.rst b/Documentation/media/uapi/v4l/selection-api-005.rst deleted file mode 100644 index 5b47a28ac6d7..000000000000 --- a/Documentation/media/uapi/v4l/selection-api-005.rst +++ /dev/null @@ -1,34 +0,0 @@ -.. -*- coding: utf-8; mode: rst -*- - -******************************** -Comparison with old cropping API -******************************** - -The selection API was introduced to cope with deficiencies of previous -:ref:`API `, that was designed to control simple capture -devices. Later the cropping API was adopted by video output drivers. The -ioctls are used to select a part of the display were the video signal is -inserted. It should be considered as an API abuse because the described -operation is actually the composing. The selection API makes a clear -distinction between composing and cropping operations by setting the -appropriate targets. The V4L2 API lacks any support for composing to and -cropping from an image inside a memory buffer. The application could -configure a capture device to fill only a part of an image by abusing -V4L2 API. Cropping a smaller image from a larger one is achieved by -setting the field ``bytesperline`` at struct -:c:type:`v4l2_pix_format`. -Introducing an image offsets could be done by modifying field ``m_userptr`` -at struct -:c:type:`v4l2_buffer` before calling -:ref:`VIDIOC_QBUF`. Those operations should be avoided because they are not -portable (endianness), and do not work for macroblock and Bayer formats -and mmap buffers. The selection API deals with configuration of buffer -cropping/composing in a clear, intuitive and portable way. Next, with -the selection API the concepts of the padded target and constraints -flags are introduced. Finally, struct :c:type:`v4l2_crop` -and struct :c:type:`v4l2_cropcap` have no reserved -fields. Therefore there is no way to extend their functionality. The new -struct :c:type:`v4l2_selection` provides a lot of place -for future extensions. Driver developers are encouraged to implement -only selection API. The former cropping API would be simulated using the -new one. diff --git a/Documentation/media/uapi/v4l/selection-api-configuration.rst b/Documentation/media/uapi/v4l/selection-api-configuration.rst new file mode 100644 index 000000000000..0a4ddc2d71db --- /dev/null +++ b/Documentation/media/uapi/v4l/selection-api-configuration.rst @@ -0,0 +1,137 @@ +.. -*- coding: utf-8; mode: rst -*- + +************* +Configuration +************* + +Applications can use the :ref:`selection API ` to +select an area in a video signal or a buffer, and to query for default +settings and hardware limits. + +Video hardware can have various cropping, composing and scaling +limitations. It may only scale up or down, support only discrete scaling +factors, or have different scaling abilities in the horizontal and +vertical directions. Also it may not support scaling at all. At the same +time the cropping/composing rectangles may have to be aligned, and both +the source and the sink may have arbitrary upper and lower size limits. +Therefore, as usual, drivers are expected to adjust the requested +parameters and return the actual values selected. An application can +control the rounding behaviour using +:ref:`constraint flags `. + + +Configuration of video capture +============================== + +See figure :ref:`sel-targets-capture` for examples of the selection +targets available for a video capture device. It is recommended to +configure the cropping targets before to the composing targets. + +The range of coordinates of the top left corner, width and height of +areas that can be sampled is given by the ``V4L2_SEL_TGT_CROP_BOUNDS`` +target. It is recommended for the driver developers to put the top/left +corner at position ``(0,0)``. The rectangle's coordinates are expressed +in pixels. + +The top left corner, width and height of the source rectangle, that is +the area actually sampled, is given by the ``V4L2_SEL_TGT_CROP`` target. +It uses the same coordinate system as ``V4L2_SEL_TGT_CROP_BOUNDS``. The +active cropping area must lie completely inside the capture boundaries. +The driver may further adjust the requested size and/or position +according to hardware limitations. + +Each capture device has a default source rectangle, given by the +``V4L2_SEL_TGT_CROP_DEFAULT`` target. This rectangle shall cover what the +driver writer considers the complete picture. Drivers shall set the +active crop rectangle to the default when the driver is first loaded, +but not later. + +The composing targets refer to a memory buffer. The limits of composing +coordinates are obtained using ``V4L2_SEL_TGT_COMPOSE_BOUNDS``. All +coordinates are expressed in pixels. The rectangle's top/left corner +must be located at position ``(0,0)``. The width and height are equal to +the image size set by :ref:`VIDIOC_S_FMT `. + +The part of a buffer into which the image is inserted by the hardware is +controlled by the ``V4L2_SEL_TGT_COMPOSE`` target. The rectangle's +coordinates are also expressed in the same coordinate system as the +bounds rectangle. The composing rectangle must lie completely inside +bounds rectangle. The driver must adjust the composing rectangle to fit +to the bounding limits. Moreover, the driver can perform other +adjustments according to hardware limitations. The application can +control rounding behaviour using +:ref:`constraint flags `. + +For capture devices the default composing rectangle is queried using +``V4L2_SEL_TGT_COMPOSE_DEFAULT``. It is usually equal to the bounding +rectangle. + +The part of a buffer that is modified by the hardware is given by +``V4L2_SEL_TGT_COMPOSE_PADDED``. It contains all pixels defined using +``V4L2_SEL_TGT_COMPOSE`` plus all padding data modified by hardware +during insertion process. All pixels outside this rectangle *must not* +be changed by the hardware. The content of pixels that lie inside the +padded area but outside active area is undefined. The application can +use the padded and active rectangles to detect where the rubbish pixels +are located and remove them if needed. + + +Configuration of video output +============================= + +For output devices targets and ioctls are used similarly to the video +capture case. The *composing* rectangle refers to the insertion of an +image into a video signal. The cropping rectangles refer to a memory +buffer. It is recommended to configure the composing targets before to +the cropping targets. + +The cropping targets refer to the memory buffer that contains an image +to be inserted into a video signal or graphical screen. The limits of +cropping coordinates are obtained using ``V4L2_SEL_TGT_CROP_BOUNDS``. +All coordinates are expressed in pixels. The top/left corner is always +point ``(0,0)``. The width and height is equal to the image size +specified using :ref:`VIDIOC_S_FMT ` ioctl. + +The top left corner, width and height of the source rectangle, that is +the area from which image date are processed by the hardware, is given +by the ``V4L2_SEL_TGT_CROP``. Its coordinates are expressed in in the +same coordinate system as the bounds rectangle. The active cropping area +must lie completely inside the crop boundaries and the driver may +further adjust the requested size and/or position according to hardware +limitations. + +For output devices the default cropping rectangle is queried using +``V4L2_SEL_TGT_CROP_DEFAULT``. It is usually equal to the bounding +rectangle. + +The part of a video signal or graphics display where the image is +inserted by the hardware is controlled by ``V4L2_SEL_TGT_COMPOSE`` +target. The rectangle's coordinates are expressed in pixels. The +composing rectangle must lie completely inside the bounds rectangle. The +driver must adjust the area to fit to the bounding limits. Moreover, the +driver can perform other adjustments according to hardware limitations. + +The device has a default composing rectangle, given by the +``V4L2_SEL_TGT_COMPOSE_DEFAULT`` target. This rectangle shall cover what +the driver writer considers the complete picture. It is recommended for +the driver developers to put the top/left corner at position ``(0,0)``. +Drivers shall set the active composing rectangle to the default one when +the driver is first loaded. + +The devices may introduce additional content to video signal other than +an image from memory buffers. It includes borders around an image. +However, such a padded area is driver-dependent feature not covered by +this document. Driver developers are encouraged to keep padded rectangle +equal to active one. The padded target is accessed by the +``V4L2_SEL_TGT_COMPOSE_PADDED`` identifier. It must contain all pixels +from the ``V4L2_SEL_TGT_COMPOSE`` target. + + +Scaling control +=============== + +An application can detect if scaling is performed by comparing the width +and the height of rectangles obtained using ``V4L2_SEL_TGT_CROP`` and +``V4L2_SEL_TGT_COMPOSE`` targets. If these are not equal then the +scaling is applied. The application can compute the scaling ratios using +these values. diff --git a/Documentation/media/uapi/v4l/selection-api-006.rst b/Documentation/media/uapi/v4l/selection-api-examples.rst similarity index 100% rename from Documentation/media/uapi/v4l/selection-api-006.rst rename to Documentation/media/uapi/v4l/selection-api-examples.rst diff --git a/Documentation/media/uapi/v4l/selection-api-002.rst b/Documentation/media/uapi/v4l/selection-api-intro.rst similarity index 100% rename from Documentation/media/uapi/v4l/selection-api-002.rst rename to Documentation/media/uapi/v4l/selection-api-intro.rst diff --git a/Documentation/media/uapi/v4l/selection-api-003.rst b/Documentation/media/uapi/v4l/selection-api-targets.rst similarity index 100% rename from Documentation/media/uapi/v4l/selection-api-003.rst rename to Documentation/media/uapi/v4l/selection-api-targets.rst diff --git a/Documentation/media/uapi/v4l/selection-api-vs-crop-api.rst b/Documentation/media/uapi/v4l/selection-api-vs-crop-api.rst new file mode 100644 index 000000000000..e7455fb1e572 --- /dev/null +++ b/Documentation/media/uapi/v4l/selection-api-vs-crop-api.rst @@ -0,0 +1,39 @@ +.. -*- coding: utf-8; mode: rst -*- + +.. _selection-vs-crop: + +******************************** +Comparison with old cropping API +******************************** + +The selection API was introduced to cope with deficiencies of the +older :ref:`CROP API `, that was designed to control simple +capture devices. Later the cropping API was adopted by video output +drivers. The ioctls are used to select a part of the display were the +video signal is inserted. It should be considered as an API abuse +because the described operation is actually the composing. The +selection API makes a clear distinction between composing and cropping +operations by setting the appropriate targets. + +The CROP API lacks any support for composing to and cropping from an +image inside a memory buffer. The application could configure a +capture device to fill only a part of an image by abusing V4L2 +API. Cropping a smaller image from a larger one is achieved by setting +the field ``bytesperline`` at struct :c:type:`v4l2_pix_format`. +Introducing an image offsets could be done by modifying field +``m_userptr`` at struct :c:type:`v4l2_buffer` before calling +:ref:`VIDIOC_QBUF `. Those operations should be avoided +because they are not portable (endianness), and do not work for +macroblock and Bayer formats and mmap buffers. + +The selection API deals with configuration of buffer +cropping/composing in a clear, intuitive and portable way. Next, with +the selection API the concepts of the padded target and constraints +flags are introduced. Finally, struct :c:type:`v4l2_crop` and struct +:c:type:`v4l2_cropcap` have no reserved fields. Therefore there is no +way to extend their functionality. The new struct +:c:type:`v4l2_selection` provides a lot of place for future +extensions. + +Driver developers are encouraged to implement only selection API. The +former cropping API would be simulated using the new one. diff --git a/Documentation/media/uapi/v4l/selection-api.rst b/Documentation/media/uapi/v4l/selection-api.rst index 81ea52d785b9..390233f704a3 100644 --- a/Documentation/media/uapi/v4l/selection-api.rst +++ b/Documentation/media/uapi/v4l/selection-api.rst @@ -2,15 +2,15 @@ .. _selection-api: -API for cropping, composing and scaling -======================================= +Cropping, composing and scaling -- the SELECTION API +==================================================== .. toctree:: :maxdepth: 1 - selection-api-002 - selection-api-003 - selection-api-004 - selection-api-005 - selection-api-006 + selection-api-intro.rst + selection-api-targets.rst + selection-api-configuration.rst + selection-api-vs-crop-api.rst + selection-api-examples.rst diff --git a/Documentation/media/uapi/v4l/selection.svg b/Documentation/media/uapi/v4l/selection.svg index a93e3b59786d..911062bd2844 100644 --- a/Documentation/media/uapi/v4l/selection.svg +++ b/Documentation/media/uapi/v4l/selection.svg @@ -1128,11 +1128,11 @@ - COMPOSE_BONDS + COMPOSE_BOUNDS - CROP_BONDS + CROP_BOUNDS overscan area diff --git a/Documentation/media/uapi/v4l/v4l2.rst b/Documentation/media/uapi/v4l/v4l2.rst index 2128717299b3..b89e5621ae69 100644 --- a/Documentation/media/uapi/v4l/v4l2.rst +++ b/Documentation/media/uapi/v4l/v4l2.rst @@ -45,7 +45,7 @@ Authors, in alphabetical order: - Subdev selections API. -- Carvalho Chehab, Mauro +- Carvalho Chehab, Mauro - Documented libv4l, designed and added v4l2grab example, Remote Controller chapter. diff --git a/Documentation/media/v4l-drivers/cx23885-cardlist.rst b/Documentation/media/v4l-drivers/cx23885-cardlist.rst index 3129ef04ddd3..8c24df8e0423 100644 --- a/Documentation/media/v4l-drivers/cx23885-cardlist.rst +++ b/Documentation/media/v4l-drivers/cx23885-cardlist.rst @@ -186,7 +186,7 @@ cx23885 cards list * - 43 - Hauppauge ImpactVCB-e - - 0070:7133 + - 0070:7133, 0070:7137 * - 44 - DViCO FusionHDTV DVB-T Dual Express2 @@ -243,3 +243,19 @@ cx23885 cards list * - 57 - Hauppauge WinTV-QuadHD-ATSC - 0070:6a18, 0070:6b18 + + * - 58 + - Hauppauge WinTV-HVR-1265(161111) + - 0070:2a18 + + * - 59 + - Hauppauge WinTV-Starburst2 + - 0070:f02a + + * - 60 + - Hauppauge WinTV-QuadHD-DVB(885) + - + + * - 61 + - Hauppauge WinTV-QuadHD-ATSC(885) + - diff --git a/Documentation/media/v4l-drivers/em28xx-cardlist.rst b/Documentation/media/v4l-drivers/em28xx-cardlist.rst index ec938c08f43d..dfe882ca945f 100644 --- a/Documentation/media/v4l-drivers/em28xx-cardlist.rst +++ b/Documentation/media/v4l-drivers/em28xx-cardlist.rst @@ -391,7 +391,7 @@ EM28xx cards list * - 94 - PCTV tripleStick (292e) - em28178 - - 2013:025f, 2040:0264 + - 2013:025f, 2013:0264, 2040:0264, 2040:8264, 2040:8268, 2040:8268 * - 95 - Leadtek VC100 - em2861 @@ -411,12 +411,16 @@ EM28xx cards list * - 99 - Hauppauge WinTV-dualHD DVB - em28174 - - 2040:0265 + - 2040:0265, 2040:8265 * - 100 - Hauppauge WinTV-dualHD 01595 ATSC/QAM - em28174 - - 2040:026d + - 2040:026d, 2040:826d * - 101 - Terratec Cinergy H6 rev. 2 - em2884 - 0ccd:10b2 + * - 102 + - :ZOLID HYBRID TV STICK + - em2882 + - diff --git a/Documentation/memory-barriers.txt b/Documentation/memory-barriers.txt index 6dafc8085acc..a02d6bbfc9d0 100644 --- a/Documentation/memory-barriers.txt +++ b/Documentation/memory-barriers.txt @@ -1920,9 +1920,6 @@ There are some more advanced barrier functions: /* assign ownership */ desc->status = DEVICE_OWN; - /* force memory to sync before notifying device via MMIO */ - wmb(); - /* notify device of new descriptors */ writel(DESC_NOTIFY, doorbell); } @@ -1930,11 +1927,15 @@ There are some more advanced barrier functions: The dma_rmb() allows us guarantee the device has released ownership before we read the data from the descriptor, and the dma_wmb() allows us to guarantee the data is written to the descriptor before the device - can see it now has ownership. The wmb() is needed to guarantee that the - cache coherent memory writes have completed before attempting a write to - the cache incoherent MMIO region. + can see it now has ownership. Note that, when using writel(), a prior + wmb() is not needed to guarantee that the cache coherent memory writes + have completed before writing to the MMIO region. The cheaper + writel_relaxed() does not provide this guarantee and must not be used + here. - See Documentation/DMA-API.txt for more information on consistent memory. + See the subsection "Kernel I/O barrier effects" for more information on + relaxed I/O accessors and the Documentation/DMA-API.txt file for more + information on consistent memory. MMIO WRITE BARRIER @@ -2903,7 +2904,7 @@ is discarded from the CPU's cache and reloaded. To deal with this, the appropriate part of the kernel must invalidate the overlapping bits of the cache on each CPU. -See Documentation/cachetlb.txt for more information on cache management. +See Documentation/core-api/cachetlb.rst for more information on cache management. CACHE COHERENCY VS MMIO @@ -3083,7 +3084,7 @@ CIRCULAR BUFFERS Memory barriers can be used to implement circular buffering without the need of a lock to serialise the producer with the consumer. See: - Documentation/circular-buffers.txt + Documentation/core-api/circular-buffers.rst for details. diff --git a/Documentation/misc-devices/ibmvmc.rst b/Documentation/misc-devices/ibmvmc.rst new file mode 100644 index 000000000000..46ded79554d4 --- /dev/null +++ b/Documentation/misc-devices/ibmvmc.rst @@ -0,0 +1,226 @@ +.. SPDX-License-Identifier: GPL-2.0+ +====================================================== +IBM Virtual Management Channel Kernel Driver (IBMVMC) +====================================================== + +:Authors: + Dave Engebretsen , + Adam Reznechek , + Steven Royer , + Bryant G. Ly , + +Introduction +============ + +Note: Knowledge of virtualization technology is required to understand +this document. + +A good reference document would be: + +https://openpowerfoundation.org/wp-content/uploads/2016/05/LoPAPR_DRAFT_v11_24March2016_cmt1.pdf + +The Virtual Management Channel (VMC) is a logical device which provides an +interface between the hypervisor and a management partition. This interface +is like a message passing interface. This management partition is intended +to provide an alternative to systems that use a Hardware Management +Console (HMC) - based system management. + +The primary hardware management solution that is developed by IBM relies +on an appliance server named the Hardware Management Console (HMC), +packaged as an external tower or rack-mounted personal computer. In a +Power Systems environment, a single HMC can manage multiple POWER +processor-based systems. + +Management Application +---------------------- + +In the management partition, a management application exists which enables +a system administrator to configure the system’s partitioning +characteristics via a command line interface (CLI) or Representational +State Transfer Application (REST API's). + +The management application runs on a Linux logical partition on a +POWER8 or newer processor-based server that is virtualized by PowerVM. +System configuration, maintenance, and control functions which +traditionally require an HMC can be implemented in the management +application using a combination of HMC to hypervisor interfaces and +existing operating system methods. This tool provides a subset of the +functions implemented by the HMC and enables basic partition configuration. +The set of HMC to hypervisor messages supported by the management +application component are passed to the hypervisor over a VMC interface, +which is defined below. + +The VMC enables the management partition to provide basic partitioning +functions: + +- Logical Partitioning Configuration +- Start, and stop actions for individual partitions +- Display of partition status +- Management of virtual Ethernet +- Management of virtual Storage +- Basic system management + +Virtual Management Channel (VMC) +-------------------------------- + +A logical device, called the Virtual Management Channel (VMC), is defined +for communicating between the management application and the hypervisor. It +basically creates the pipes that enable virtualization management +software. This device is presented to a designated management partition as +a virtual device. + +This communication device uses Command/Response Queue (CRQ) and the +Remote Direct Memory Access (RDMA) interfaces. A three-way handshake is +defined that must take place to establish that both the hypervisor and +management partition sides of the channel are running prior to +sending/receiving any of the protocol messages. + +This driver also utilizes Transport Event CRQs. CRQ messages are sent +when the hypervisor detects one of the peer partitions has abnormally +terminated, or one side has called H_FREE_CRQ to close their CRQ. +Two new classes of CRQ messages are introduced for the VMC device. VMC +Administrative messages are used for each partition using the VMC to +communicate capabilities to their partner. HMC Interface messages are used +for the actual flow of HMC messages between the management partition and +the hypervisor. As most HMC messages far exceed the size of a CRQ buffer, +a virtual DMA (RMDA) of the HMC message data is done prior to each HMC +Interface CRQ message. Only the management partition drives RDMA +operations; hypervisors never directly cause the movement of message data. + + +Terminology +----------- +RDMA + Remote Direct Memory Access is DMA transfer from the server to its + client or from the server to its partner partition. DMA refers + to both physical I/O to and from memory operations and to memory + to memory move operations. +CRQ + Command/Response Queue a facility which is used to communicate + between partner partitions. Transport events which are signaled + from the hypervisor to partition are also reported in this queue. + +Example Management Partition VMC Driver Interface +================================================= + +This section provides an example for the management application +implementation where a device driver is used to interface to the VMC +device. This driver consists of a new device, for example /dev/ibmvmc, +which provides interfaces to open, close, read, write, and perform +ioctl’s against the VMC device. + +VMC Interface Initialization +---------------------------- + +The device driver is responsible for initializing the VMC when the driver +is loaded. It first creates and initializes the CRQ. Next, an exchange of +VMC capabilities is performed to indicate the code version and number of +resources available in both the management partition and the hypervisor. +Finally, the hypervisor requests that the management partition create an +initial pool of VMC buffers, one buffer for each possible HMC connection, +which will be used for management application session initialization. +Prior to completion of this initialization sequence, the device returns +EBUSY to open() calls. EIO is returned for all open() failures. + +:: + + Management Partition Hypervisor + CRQ INIT + ----------------------------------------> + CRQ INIT COMPLETE + <---------------------------------------- + CAPABILITIES + ----------------------------------------> + CAPABILITIES RESPONSE + <---------------------------------------- + ADD BUFFER (HMC IDX=0,1,..) _ + <---------------------------------------- | + ADD BUFFER RESPONSE | - Perform # HMCs Iterations + ----------------------------------------> - + +VMC Interface Open +------------------ + +After the basic VMC channel has been initialized, an HMC session level +connection can be established. The application layer performs an open() to +the VMC device and executes an ioctl() against it, indicating the HMC ID +(32 bytes of data) for this session. If the VMC device is in an invalid +state, EIO will be returned for the ioctl(). The device driver creates a +new HMC session value (ranging from 1 to 255) and HMC index value (starting +at index 0 and ranging to 254) for this HMC ID. The driver then does an +RDMA of the HMC ID to the hypervisor, and then sends an Interface Open +message to the hypervisor to establish the session over the VMC. After the +hypervisor receives this information, it sends Add Buffer messages to the +management partition to seed an initial pool of buffers for the new HMC +connection. Finally, the hypervisor sends an Interface Open Response +message, to indicate that it is ready for normal runtime messaging. The +following illustrates this VMC flow: + +:: + + Management Partition Hypervisor + RDMA HMC ID + ----------------------------------------> + Interface Open + ----------------------------------------> + Add Buffer _ + <---------------------------------------- | + Add Buffer Response | - Perform N Iterations + ----------------------------------------> - + Interface Open Response + <---------------------------------------- + +VMC Interface Runtime +--------------------- + +During normal runtime, the management application and the hypervisor +exchange HMC messages via the Signal VMC message and RDMA operations. When +sending data to the hypervisor, the management application performs a +write() to the VMC device, and the driver RDMA’s the data to the hypervisor +and then sends a Signal Message. If a write() is attempted before VMC +device buffers have been made available by the hypervisor, or no buffers +are currently available, EBUSY is returned in response to the write(). A +write() will return EIO for all other errors, such as an invalid device +state. When the hypervisor sends a message to the management, the data is +put into a VMC buffer and an Signal Message is sent to the VMC driver in +the management partition. The driver RDMA’s the buffer into the partition +and passes the data up to the appropriate management application via a +read() to the VMC device. The read() request blocks if there is no buffer +available to read. The management application may use select() to wait for +the VMC device to become ready with data to read. + +:: + + Management Partition Hypervisor + MSG RDMA + ----------------------------------------> + SIGNAL MSG + ----------------------------------------> + SIGNAL MSG + <---------------------------------------- + MSG RDMA + <---------------------------------------- + +VMC Interface Close +------------------- + +HMC session level connections are closed by the management partition when +the application layer performs a close() against the device. This action +results in an Interface Close message flowing to the hypervisor, which +causes the session to be terminated. The device driver must free any +storage allocated for buffers for this HMC connection. + +:: + + Management Partition Hypervisor + INTERFACE CLOSE + ----------------------------------------> + INTERFACE CLOSE RESPONSE + <---------------------------------------- + +Additional Information +====================== + +For more information on the documentation for CRQ Messages, VMC Messages, +HMC interface Buffers, and signal messages please refer to the Linux on +Power Architecture Platform Reference. Section F. diff --git a/Documentation/networking/6lowpan.txt b/Documentation/networking/6lowpan.txt index a7dc7e939c7a..2e5a939d7e6f 100644 --- a/Documentation/networking/6lowpan.txt +++ b/Documentation/networking/6lowpan.txt @@ -24,10 +24,10 @@ enum lowpan_lltypes. Example to evaluate the private usually you can do: -static inline sturct lowpan_priv_foobar * +static inline struct lowpan_priv_foobar * lowpan_foobar_priv(struct net_device *dev) { - return (sturct lowpan_priv_foobar *)lowpan_priv(dev)->priv; + return (struct lowpan_priv_foobar *)lowpan_priv(dev)->priv; } switch (dev->type) { diff --git a/Documentation/networking/af_xdp.rst b/Documentation/networking/af_xdp.rst new file mode 100644 index 000000000000..ff929cfab4f4 --- /dev/null +++ b/Documentation/networking/af_xdp.rst @@ -0,0 +1,312 @@ +.. SPDX-License-Identifier: GPL-2.0 + +====== +AF_XDP +====== + +Overview +======== + +AF_XDP is an address family that is optimized for high performance +packet processing. + +This document assumes that the reader is familiar with BPF and XDP. If +not, the Cilium project has an excellent reference guide at +http://cilium.readthedocs.io/en/latest/bpf/. + +Using the XDP_REDIRECT action from an XDP program, the program can +redirect ingress frames to other XDP enabled netdevs, using the +bpf_redirect_map() function. AF_XDP sockets enable the possibility for +XDP programs to redirect frames to a memory buffer in a user-space +application. + +An AF_XDP socket (XSK) is created with the normal socket() +syscall. Associated with each XSK are two rings: the RX ring and the +TX ring. A socket can receive packets on the RX ring and it can send +packets on the TX ring. These rings are registered and sized with the +setsockopts XDP_RX_RING and XDP_TX_RING, respectively. It is mandatory +to have at least one of these rings for each socket. An RX or TX +descriptor ring points to a data buffer in a memory area called a +UMEM. RX and TX can share the same UMEM so that a packet does not have +to be copied between RX and TX. Moreover, if a packet needs to be kept +for a while due to a possible retransmit, the descriptor that points +to that packet can be changed to point to another and reused right +away. This again avoids copying data. + +The UMEM consists of a number of equally sized chunks. A descriptor in +one of the rings references a frame by referencing its addr. The addr +is simply an offset within the entire UMEM region. The user space +allocates memory for this UMEM using whatever means it feels is most +appropriate (malloc, mmap, huge pages, etc). This memory area is then +registered with the kernel using the new setsockopt XDP_UMEM_REG. The +UMEM also has two rings: the FILL ring and the COMPLETION ring. The +fill ring is used by the application to send down addr for the kernel +to fill in with RX packet data. References to these frames will then +appear in the RX ring once each packet has been received. The +completion ring, on the other hand, contains frame addr that the +kernel has transmitted completely and can now be used again by user +space, for either TX or RX. Thus, the frame addrs appearing in the +completion ring are addrs that were previously transmitted using the +TX ring. In summary, the RX and FILL rings are used for the RX path +and the TX and COMPLETION rings are used for the TX path. + +The socket is then finally bound with a bind() call to a device and a +specific queue id on that device, and it is not until bind is +completed that traffic starts to flow. + +The UMEM can be shared between processes, if desired. If a process +wants to do this, it simply skips the registration of the UMEM and its +corresponding two rings, sets the XDP_SHARED_UMEM flag in the bind +call and submits the XSK of the process it would like to share UMEM +with as well as its own newly created XSK socket. The new process will +then receive frame addr references in its own RX ring that point to +this shared UMEM. Note that since the ring structures are +single-consumer / single-producer (for performance reasons), the new +process has to create its own socket with associated RX and TX rings, +since it cannot share this with the other process. This is also the +reason that there is only one set of FILL and COMPLETION rings per +UMEM. It is the responsibility of a single process to handle the UMEM. + +How is then packets distributed from an XDP program to the XSKs? There +is a BPF map called XSKMAP (or BPF_MAP_TYPE_XSKMAP in full). The +user-space application can place an XSK at an arbitrary place in this +map. The XDP program can then redirect a packet to a specific index in +this map and at this point XDP validates that the XSK in that map was +indeed bound to that device and ring number. If not, the packet is +dropped. If the map is empty at that index, the packet is also +dropped. This also means that it is currently mandatory to have an XDP +program loaded (and one XSK in the XSKMAP) to be able to get any +traffic to user space through the XSK. + +AF_XDP can operate in two different modes: XDP_SKB and XDP_DRV. If the +driver does not have support for XDP, or XDP_SKB is explicitly chosen +when loading the XDP program, XDP_SKB mode is employed that uses SKBs +together with the generic XDP support and copies out the data to user +space. A fallback mode that works for any network device. On the other +hand, if the driver has support for XDP, it will be used by the AF_XDP +code to provide better performance, but there is still a copy of the +data into user space. + +Concepts +======== + +In order to use an AF_XDP socket, a number of associated objects need +to be setup. + +Jonathan Corbet has also written an excellent article on LWN, +"Accelerating networking with AF_XDP". It can be found at +https://lwn.net/Articles/750845/. + +UMEM +---- + +UMEM is a region of virtual contiguous memory, divided into +equal-sized frames. An UMEM is associated to a netdev and a specific +queue id of that netdev. It is created and configured (chunk size, +headroom, start address and size) by using the XDP_UMEM_REG setsockopt +system call. A UMEM is bound to a netdev and queue id, via the bind() +system call. + +An AF_XDP is socket linked to a single UMEM, but one UMEM can have +multiple AF_XDP sockets. To share an UMEM created via one socket A, +the next socket B can do this by setting the XDP_SHARED_UMEM flag in +struct sockaddr_xdp member sxdp_flags, and passing the file descriptor +of A to struct sockaddr_xdp member sxdp_shared_umem_fd. + +The UMEM has two single-producer/single-consumer rings, that are used +to transfer ownership of UMEM frames between the kernel and the +user-space application. + +Rings +----- + +There are a four different kind of rings: Fill, Completion, RX and +TX. All rings are single-producer/single-consumer, so the user-space +application need explicit synchronization of multiple +processes/threads are reading/writing to them. + +The UMEM uses two rings: Fill and Completion. Each socket associated +with the UMEM must have an RX queue, TX queue or both. Say, that there +is a setup with four sockets (all doing TX and RX). Then there will be +one Fill ring, one Completion ring, four TX rings and four RX rings. + +The rings are head(producer)/tail(consumer) based rings. A producer +writes the data ring at the index pointed out by struct xdp_ring +producer member, and increasing the producer index. A consumer reads +the data ring at the index pointed out by struct xdp_ring consumer +member, and increasing the consumer index. + +The rings are configured and created via the _RING setsockopt system +calls and mmapped to user-space using the appropriate offset to mmap() +(XDP_PGOFF_RX_RING, XDP_PGOFF_TX_RING, XDP_UMEM_PGOFF_FILL_RING and +XDP_UMEM_PGOFF_COMPLETION_RING). + +The size of the rings need to be of size power of two. + +UMEM Fill Ring +~~~~~~~~~~~~~~ + +The Fill ring is used to transfer ownership of UMEM frames from +user-space to kernel-space. The UMEM addrs are passed in the ring. As +an example, if the UMEM is 64k and each chunk is 4k, then the UMEM has +16 chunks and can pass addrs between 0 and 64k. + +Frames passed to the kernel are used for the ingress path (RX rings). + +The user application produces UMEM addrs to this ring. Note that the +kernel will mask the incoming addr. E.g. for a chunk size of 2k, the +log2(2048) LSB of the addr will be masked off, meaning that 2048, 2050 +and 3000 refers to the same chunk. + + +UMEM Completetion Ring +~~~~~~~~~~~~~~~~~~~~~~ + +The Completion Ring is used transfer ownership of UMEM frames from +kernel-space to user-space. Just like the Fill ring, UMEM indicies are +used. + +Frames passed from the kernel to user-space are frames that has been +sent (TX ring) and can be used by user-space again. + +The user application consumes UMEM addrs from this ring. + + +RX Ring +~~~~~~~ + +The RX ring is the receiving side of a socket. Each entry in the ring +is a struct xdp_desc descriptor. The descriptor contains UMEM offset +(addr) and the length of the data (len). + +If no frames have been passed to kernel via the Fill ring, no +descriptors will (or can) appear on the RX ring. + +The user application consumes struct xdp_desc descriptors from this +ring. + +TX Ring +~~~~~~~ + +The TX ring is used to send frames. The struct xdp_desc descriptor is +filled (index, length and offset) and passed into the ring. + +To start the transfer a sendmsg() system call is required. This might +be relaxed in the future. + +The user application produces struct xdp_desc descriptors to this +ring. + +XSKMAP / BPF_MAP_TYPE_XSKMAP +---------------------------- + +On XDP side there is a BPF map type BPF_MAP_TYPE_XSKMAP (XSKMAP) that +is used in conjunction with bpf_redirect_map() to pass the ingress +frame to a socket. + +The user application inserts the socket into the map, via the bpf() +system call. + +Note that if an XDP program tries to redirect to a socket that does +not match the queue configuration and netdev, the frame will be +dropped. E.g. an AF_XDP socket is bound to netdev eth0 and +queue 17. Only the XDP program executing for eth0 and queue 17 will +successfully pass data to the socket. Please refer to the sample +application (samples/bpf/) in for an example. + +Usage +===== + +In order to use AF_XDP sockets there are two parts needed. The +user-space application and the XDP program. For a complete setup and +usage example, please refer to the sample application. The user-space +side is xdpsock_user.c and the XDP side xdpsock_kern.c. + +Naive ring dequeue and enqueue could look like this:: + + // struct xdp_rxtx_ring { + // __u32 *producer; + // __u32 *consumer; + // struct xdp_desc *desc; + // }; + + // struct xdp_umem_ring { + // __u32 *producer; + // __u32 *consumer; + // __u64 *desc; + // }; + + // typedef struct xdp_rxtx_ring RING; + // typedef struct xdp_umem_ring RING; + + // typedef struct xdp_desc RING_TYPE; + // typedef __u64 RING_TYPE; + + int dequeue_one(RING *ring, RING_TYPE *item) + { + __u32 entries = *ring->producer - *ring->consumer; + + if (entries == 0) + return -1; + + // read-barrier! + + *item = ring->desc[*ring->consumer & (RING_SIZE - 1)]; + (*ring->consumer)++; + return 0; + } + + int enqueue_one(RING *ring, const RING_TYPE *item) + { + u32 free_entries = RING_SIZE - (*ring->producer - *ring->consumer); + + if (free_entries == 0) + return -1; + + ring->desc[*ring->producer & (RING_SIZE - 1)] = *item; + + // write-barrier! + + (*ring->producer)++; + return 0; + } + + +For a more optimized version, please refer to the sample application. + +Sample application +================== + +There is a xdpsock benchmarking/test application included that +demonstrates how to use AF_XDP sockets with both private and shared +UMEMs. Say that you would like your UDP traffic from port 4242 to end +up in queue 16, that we will enable AF_XDP on. Here, we use ethtool +for this:: + + ethtool -N p3p2 rx-flow-hash udp4 fn + ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \ + action 16 + +Running the rxdrop benchmark in XDP_DRV mode can then be done +using:: + + samples/bpf/xdpsock -i p3p2 -q 16 -r -N + +For XDP_SKB mode, use the switch "-S" instead of "-N" and all options +can be displayed with "-h", as usual. + +Credits +======= + +- Björn Töpel (AF_XDP core) +- Magnus Karlsson (AF_XDP core) +- Alexander Duyck +- Alexei Starovoitov +- Daniel Borkmann +- Jesper Dangaard Brouer +- John Fastabend +- Jonathan Corbet (LWN coverage) +- Michael S. Tsirkin +- Qi Z Zhang +- Willem de Bruijn + diff --git a/Documentation/networking/bonding.txt b/Documentation/networking/bonding.txt index 9ba04c0bab8d..c13214d073a4 100644 --- a/Documentation/networking/bonding.txt +++ b/Documentation/networking/bonding.txt @@ -140,7 +140,7 @@ bonding module at load time, or are specified via sysfs. Module options may be given as command line arguments to the insmod or modprobe command, but are usually specified in either the -/etc/modrobe.d/*.conf configuration files, or in a distro-specific +/etc/modprobe.d/*.conf configuration files, or in a distro-specific configuration file (some of which are detailed in the next section). Details on bonding support for sysfs is provided in the diff --git a/Documentation/networking/can.rst b/Documentation/networking/can.rst index d23c51abf8c6..2fd0b51a8c52 100644 --- a/Documentation/networking/can.rst +++ b/Documentation/networking/can.rst @@ -164,7 +164,7 @@ The Linux network devices (by default) just can handle the transmission and reception of media dependent frames. Due to the arbitration on the CAN bus the transmission of a low prio CAN-ID may be delayed by the reception of a high prio CAN frame. To -reflect the correct [*]_ traffic on the node the loopback of the sent +reflect the correct [#f1]_ traffic on the node the loopback of the sent data has to be performed right after a successful transmission. If the CAN network interface is not capable of performing the loopback for some reason the SocketCAN core can do this task as a fallback solution. @@ -175,7 +175,7 @@ networking behaviour for CAN applications. Due to some requests from the RT-SocketCAN group the loopback optionally may be disabled for each separate socket. See sockopts from the CAN RAW sockets in :ref:`socketcan-raw-sockets`. -.. [*] you really like to have this when you're running analyser +.. [#f1] you really like to have this when you're running analyser tools like 'candump' or 'cansniffer' on the (same) node. diff --git a/Documentation/networking/e100.rst b/Documentation/networking/e100.rst new file mode 100644 index 000000000000..d4d837027925 --- /dev/null +++ b/Documentation/networking/e100.rst @@ -0,0 +1,177 @@ +Linux* Base Driver for the Intel(R) PRO/100 Family of Adapters +============================================================== + +June 1, 2018 + +Contents +======== + +- In This Release +- Identifying Your Adapter +- Building and Installation +- Driver Configuration Parameters +- Additional Configurations +- Known Issues +- Support + + +In This Release +=============== + +This file describes the Linux* Base Driver for the Intel(R) PRO/100 Family of +Adapters. This driver includes support for Itanium(R)2-based systems. + +For questions related to hardware requirements, refer to the documentation +supplied with your Intel PRO/100 adapter. + +The following features are now available in supported kernels: + - Native VLANs + - Channel Bonding (teaming) + - SNMP + +Channel Bonding documentation can be found in the Linux kernel source: +/Documentation/networking/bonding.txt + + +Identifying Your Adapter +======================== + +For information on how to identify your adapter, and for the latest Intel +network drivers, refer to the Intel Support website: +http://www.intel.com/support + +Driver Configuration Parameters +=============================== + +The default value for each parameter is generally the recommended setting, +unless otherwise noted. + +Rx Descriptors: Number of receive descriptors. A receive descriptor is a data + structure that describes a receive buffer and its attributes to the network + controller. The data in the descriptor is used by the controller to write + data from the controller to host memory. In the 3.x.x driver the valid range + for this parameter is 64-256. The default value is 256. This parameter can be + changed using the command:: + + ethtool -G eth? rx n + + Where n is the number of desired Rx descriptors. + +Tx Descriptors: Number of transmit descriptors. A transmit descriptor is a data + structure that describes a transmit buffer and its attributes to the network + controller. The data in the descriptor is used by the controller to read + data from the host memory to the controller. In the 3.x.x driver the valid + range for this parameter is 64-256. The default value is 128. This parameter + can be changed using the command:: + + ethtool -G eth? tx n + + Where n is the number of desired Tx descriptors. + +Speed/Duplex: The driver auto-negotiates the link speed and duplex settings by + default. The ethtool utility can be used as follows to force speed/duplex.:: + + ethtool -s eth? autoneg off speed {10|100} duplex {full|half} + + NOTE: setting the speed/duplex to incorrect values will cause the link to + fail. + +Event Log Message Level: The driver uses the message level flag to log events + to syslog. The message level can be set at driver load time. It can also be + set using the command:: + + ethtool -s eth? msglvl n + + +Additional Configurations +========================= + + Configuring the Driver on Different Distributions + ------------------------------------------------- + + Configuring a network driver to load properly when the system is started is + distribution dependent. Typically, the configuration process involves adding + an alias line to /etc/modprobe.d/*.conf as well as editing other system + startup scripts and/or configuration files. Many popular Linux + distributions ship with tools to make these changes for you. To learn the + proper way to configure a network device for your system, refer to your + distribution documentation. If during this process you are asked for the + driver or module name, the name for the Linux Base Driver for the Intel + PRO/100 Family of Adapters is e100. + + As an example, if you install the e100 driver for two PRO/100 adapters + (eth0 and eth1), add the following to a configuration file in /etc/modprobe.d/ + + alias eth0 e100 + alias eth1 e100 + + Viewing Link Messages + --------------------- + In order to see link messages and other Intel driver information on your + console, you must set the dmesg level up to six. This can be done by + entering the following on the command line before loading the e100 driver:: + + dmesg -n 6 + + If you wish to see all messages issued by the driver, including debug + messages, set the dmesg level to eight. + + NOTE: This setting is not saved across reboots. + + + ethtool + ------- + + The driver utilizes the ethtool interface for driver configuration and + diagnostics, as well as displaying statistical information. The ethtool + version 1.6 or later is required for this functionality. + + The latest release of ethtool can be found from + https://www.kernel.org/pub/software/network/ethtool/ + + Enabling Wake on LAN* (WoL) + --------------------------- + WoL is provided through the ethtool* utility. For instructions on enabling + WoL with ethtool, refer to the ethtool man page. + + WoL will be enabled on the system during the next shut down or reboot. For + this driver version, in order to enable WoL, the e100 driver must be + loaded when shutting down or rebooting the system. + + NAPI + ---- + + NAPI (Rx polling mode) is supported in the e100 driver. + + See https://wiki.linuxfoundation.org/networking/napi for more information + on NAPI. + + Multiple Interfaces on Same Ethernet Broadcast Network + ------------------------------------------------------ + + Due to the default ARP behavior on Linux, it is not possible to have + one system on two IP networks in the same Ethernet broadcast domain + (non-partitioned switch) behave as expected. All Ethernet interfaces + will respond to IP traffic for any IP address assigned to the system. + This results in unbalanced receive traffic. + + If you have multiple interfaces in a server, either turn on ARP + filtering by + + (1) entering:: echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter + (this only works if your kernel's version is higher than 2.4.5), or + + (2) installing the interfaces in separate broadcast domains (either + in different switches or in a switch partitioned to VLANs). + + +Support +======= +For general information, go to the Intel support website at: +http://www.intel.com/support/ + +or the Intel Wired Networking project hosted by Sourceforge at: +http://sourceforge.net/projects/e1000 +If an issue is identified with the released source code on a supported kernel +with a supported adapter, email the specific information related to the issue +to e1000-devel@lists.sf.net. diff --git a/Documentation/networking/e100.txt b/Documentation/networking/e100.txt deleted file mode 100644 index 54810b82c01a..000000000000 --- a/Documentation/networking/e100.txt +++ /dev/null @@ -1,183 +0,0 @@ -Linux* Base Driver for the Intel(R) PRO/100 Family of Adapters -============================================================== - -March 15, 2011 - -Contents -======== - -- In This Release -- Identifying Your Adapter -- Building and Installation -- Driver Configuration Parameters -- Additional Configurations -- Known Issues -- Support - - -In This Release -=============== - -This file describes the Linux* Base Driver for the Intel(R) PRO/100 Family of -Adapters. This driver includes support for Itanium(R)2-based systems. - -For questions related to hardware requirements, refer to the documentation -supplied with your Intel PRO/100 adapter. - -The following features are now available in supported kernels: - - Native VLANs - - Channel Bonding (teaming) - - SNMP - -Channel Bonding documentation can be found in the Linux kernel source: -/Documentation/networking/bonding.txt - - -Identifying Your Adapter -======================== - -For more information on how to identify your adapter, go to the Adapter & -Driver ID Guide at: - - http://support.intel.com/support/network/adapter/pro100/21397.htm - -For the latest Intel network drivers for Linux, refer to the following -website. In the search field, enter your adapter name or type, or use the -networking link on the left to search for your adapter: - - http://downloadfinder.intel.com/scripts-df/support_intel.asp - -Driver Configuration Parameters -=============================== - -The default value for each parameter is generally the recommended setting, -unless otherwise noted. - -Rx Descriptors: Number of receive descriptors. A receive descriptor is a data - structure that describes a receive buffer and its attributes to the network - controller. The data in the descriptor is used by the controller to write - data from the controller to host memory. In the 3.x.x driver the valid range - for this parameter is 64-256. The default value is 64. This parameter can be - changed using the command: - - ethtool -G eth? rx n, where n is the number of desired rx descriptors. - -Tx Descriptors: Number of transmit descriptors. A transmit descriptor is a data - structure that describes a transmit buffer and its attributes to the network - controller. The data in the descriptor is used by the controller to read - data from the host memory to the controller. In the 3.x.x driver the valid - range for this parameter is 64-256. The default value is 64. This parameter - can be changed using the command: - - ethtool -G eth? tx n, where n is the number of desired tx descriptors. - -Speed/Duplex: The driver auto-negotiates the link speed and duplex settings by - default. The ethtool utility can be used as follows to force speed/duplex. - - ethtool -s eth? autoneg off speed {10|100} duplex {full|half} - - NOTE: setting the speed/duplex to incorrect values will cause the link to - fail. - -Event Log Message Level: The driver uses the message level flag to log events - to syslog. The message level can be set at driver load time. It can also be - set using the command: - - ethtool -s eth? msglvl n - - -Additional Configurations -========================= - - Configuring the Driver on Different Distributions - ------------------------------------------------- - - Configuring a network driver to load properly when the system is started is - distribution dependent. Typically, the configuration process involves adding - an alias line to /etc/modprobe.d/*.conf as well as editing other system - startup scripts and/or configuration files. Many popular Linux - distributions ship with tools to make these changes for you. To learn the - proper way to configure a network device for your system, refer to your - distribution documentation. If during this process you are asked for the - driver or module name, the name for the Linux Base Driver for the Intel - PRO/100 Family of Adapters is e100. - - As an example, if you install the e100 driver for two PRO/100 adapters - (eth0 and eth1), add the following to a configuration file in /etc/modprobe.d/ - - alias eth0 e100 - alias eth1 e100 - - Viewing Link Messages - --------------------- - In order to see link messages and other Intel driver information on your - console, you must set the dmesg level up to six. This can be done by - entering the following on the command line before loading the e100 driver: - - dmesg -n 8 - - If you wish to see all messages issued by the driver, including debug - messages, set the dmesg level to eight. - - NOTE: This setting is not saved across reboots. - - - ethtool - ------- - - The driver utilizes the ethtool interface for driver configuration and - diagnostics, as well as displaying statistical information. The ethtool - version 1.6 or later is required for this functionality. - - The latest release of ethtool can be found from - https://www.kernel.org/pub/software/network/ethtool/ - - Enabling Wake on LAN* (WoL) - --------------------------- - WoL is provided through the ethtool* utility. For instructions on enabling - WoL with ethtool, refer to the ethtool man page. - - WoL will be enabled on the system during the next shut down or reboot. For - this driver version, in order to enable WoL, the e100 driver must be - loaded when shutting down or rebooting the system. - - NAPI - ---- - - NAPI (Rx polling mode) is supported in the e100 driver. - - See www.cyberus.ca/~hadi/usenix-paper.tgz for more information on NAPI. - - Multiple Interfaces on Same Ethernet Broadcast Network - ------------------------------------------------------ - - Due to the default ARP behavior on Linux, it is not possible to have - one system on two IP networks in the same Ethernet broadcast domain - (non-partitioned switch) behave as expected. All Ethernet interfaces - will respond to IP traffic for any IP address assigned to the system. - This results in unbalanced receive traffic. - - If you have multiple interfaces in a server, either turn on ARP - filtering by - - (1) entering: echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter - (this only works if your kernel's version is higher than 2.4.5), or - - (2) installing the interfaces in separate broadcast domains (either - in different switches or in a switch partitioned to VLANs). - - -Support -======= - -For general information, go to the Intel support website at: - - http://support.intel.com - - or the Intel Wired Networking project hosted by Sourceforge at: - - http://sourceforge.net/projects/e1000 - -If an issue is identified with the released source code on the supported -kernel with a supported adapter, email the specific information related to the -issue to e1000-devel@lists.sourceforge.net. diff --git a/Documentation/networking/e1000.rst b/Documentation/networking/e1000.rst new file mode 100644 index 000000000000..616848940e63 --- /dev/null +++ b/Documentation/networking/e1000.rst @@ -0,0 +1,422 @@ +Linux* Base Driver for Intel(R) Ethernet Network Connection +=========================================================== + +Intel Gigabit Linux driver. +Copyright(c) 1999 - 2013 Intel Corporation. + +Contents +======== + +- Identifying Your Adapter +- Command Line Parameters +- Speed and Duplex Configuration +- Additional Configurations +- Support + +Identifying Your Adapter +======================== + +For more information on how to identify your adapter, go to the Adapter & +Driver ID Guide at: + + http://support.intel.com/support/go/network/adapter/idguide.htm + +For the latest Intel network drivers for Linux, refer to the following +website. In the search field, enter your adapter name or type, or use the +networking link on the left to search for your adapter: + + http://support.intel.com/support/go/network/adapter/home.htm + +Command Line Parameters +======================= + +The default value for each parameter is generally the recommended setting, +unless otherwise noted. + +NOTES: For more information about the AutoNeg, Duplex, and Speed + parameters, see the "Speed and Duplex Configuration" section in + this document. + + For more information about the InterruptThrottleRate, + RxIntDelay, TxIntDelay, RxAbsIntDelay, and TxAbsIntDelay + parameters, see the application note at: + http://www.intel.com/design/network/applnots/ap450.htm + +AutoNeg +------- +(Supported only on adapters with copper connections) +Valid Range: 0x01-0x0F, 0x20-0x2F +Default Value: 0x2F + +This parameter is a bit-mask that specifies the speed and duplex settings +advertised by the adapter. When this parameter is used, the Speed and +Duplex parameters must not be specified. + +NOTE: Refer to the Speed and Duplex section of this readme for more + information on the AutoNeg parameter. + +Duplex +------ +(Supported only on adapters with copper connections) +Valid Range: 0-2 (0=auto-negotiate, 1=half, 2=full) +Default Value: 0 + +This defines the direction in which data is allowed to flow. Can be +either one or two-directional. If both Duplex and the link partner are +set to auto-negotiate, the board auto-detects the correct duplex. If the +link partner is forced (either full or half), Duplex defaults to half- +duplex. + +FlowControl +----------- +Valid Range: 0-3 (0=none, 1=Rx only, 2=Tx only, 3=Rx&Tx) +Default Value: Reads flow control settings from the EEPROM + +This parameter controls the automatic generation(Tx) and response(Rx) +to Ethernet PAUSE frames. + +InterruptThrottleRate +--------------------- +(not supported on Intel(R) 82542, 82543 or 82544-based adapters) +Valid Range: 0,1,3,4,100-100000 (0=off, 1=dynamic, 3=dynamic conservative, + 4=simplified balancing) +Default Value: 3 + +The driver can limit the amount of interrupts per second that the adapter +will generate for incoming packets. It does this by writing a value to the +adapter that is based on the maximum amount of interrupts that the adapter +will generate per second. + +Setting InterruptThrottleRate to a value greater or equal to 100 +will program the adapter to send out a maximum of that many interrupts +per second, even if more packets have come in. This reduces interrupt +load on the system and can lower CPU utilization under heavy load, +but will increase latency as packets are not processed as quickly. + +The default behaviour of the driver previously assumed a static +InterruptThrottleRate value of 8000, providing a good fallback value for +all traffic types,but lacking in small packet performance and latency. +The hardware can handle many more small packets per second however, and +for this reason an adaptive interrupt moderation algorithm was implemented. + +Since 7.3.x, the driver has two adaptive modes (setting 1 or 3) in which +it dynamically adjusts the InterruptThrottleRate value based on the traffic +that it receives. After determining the type of incoming traffic in the last +timeframe, it will adjust the InterruptThrottleRate to an appropriate value +for that traffic. + +The algorithm classifies the incoming traffic every interval into +classes. Once the class is determined, the InterruptThrottleRate value is +adjusted to suit that traffic type the best. There are three classes defined: +"Bulk traffic", for large amounts of packets of normal size; "Low latency", +for small amounts of traffic and/or a significant percentage of small +packets; and "Lowest latency", for almost completely small packets or +minimal traffic. + +In dynamic conservative mode, the InterruptThrottleRate value is set to 4000 +for traffic that falls in class "Bulk traffic". If traffic falls in the "Low +latency" or "Lowest latency" class, the InterruptThrottleRate is increased +stepwise to 20000. This default mode is suitable for most applications. + +For situations where low latency is vital such as cluster or +grid computing, the algorithm can reduce latency even more when +InterruptThrottleRate is set to mode 1. In this mode, which operates +the same as mode 3, the InterruptThrottleRate will be increased stepwise to +70000 for traffic in class "Lowest latency". + +In simplified mode the interrupt rate is based on the ratio of TX and +RX traffic. If the bytes per second rate is approximately equal, the +interrupt rate will drop as low as 2000 interrupts per second. If the +traffic is mostly transmit or mostly receive, the interrupt rate could +be as high as 8000. + +Setting InterruptThrottleRate to 0 turns off any interrupt moderation +and may improve small packet latency, but is generally not suitable +for bulk throughput traffic. + +NOTE: InterruptThrottleRate takes precedence over the TxAbsIntDelay and + RxAbsIntDelay parameters. In other words, minimizing the receive + and/or transmit absolute delays does not force the controller to + generate more interrupts than what the Interrupt Throttle Rate + allows. + +CAUTION: If you are using the Intel(R) PRO/1000 CT Network Connection + (controller 82547), setting InterruptThrottleRate to a value + greater than 75,000, may hang (stop transmitting) adapters + under certain network conditions. If this occurs a NETDEV + WATCHDOG message is logged in the system event log. In + addition, the controller is automatically reset, restoring + the network connection. To eliminate the potential for the + hang, ensure that InterruptThrottleRate is set no greater + than 75,000 and is not set to 0. + +NOTE: When e1000 is loaded with default settings and multiple adapters + are in use simultaneously, the CPU utilization may increase non- + linearly. In order to limit the CPU utilization without impacting + the overall throughput, we recommend that you load the driver as + follows:: + + modprobe e1000 InterruptThrottleRate=3000,3000,3000 + + This sets the InterruptThrottleRate to 3000 interrupts/sec for + the first, second, and third instances of the driver. The range + of 2000 to 3000 interrupts per second works on a majority of + systems and is a good starting point, but the optimal value will + be platform-specific. If CPU utilization is not a concern, use + RX_POLLING (NAPI) and default driver settings. + +RxDescriptors +------------- +Valid Range: 48-256 for 82542 and 82543-based adapters + 48-4096 for all other supported adapters +Default Value: 256 + +This value specifies the number of receive buffer descriptors allocated +by the driver. Increasing this value allows the driver to buffer more +incoming packets, at the expense of increased system memory utilization. + +Each descriptor is 16 bytes. A receive buffer is also allocated for each +descriptor and can be either 2048, 4096, 8192, or 16384 bytes, depending +on the MTU setting. The maximum MTU size is 16110. + +NOTE: MTU designates the frame size. It only needs to be set for Jumbo + Frames. Depending on the available system resources, the request + for a higher number of receive descriptors may be denied. In this + case, use a lower number. + +RxIntDelay +---------- +Valid Range: 0-65535 (0=off) +Default Value: 0 + +This value delays the generation of receive interrupts in units of 1.024 +microseconds. Receive interrupt reduction can improve CPU efficiency if +properly tuned for specific network traffic. Increasing this value adds +extra latency to frame reception and can end up decreasing the throughput +of TCP traffic. If the system is reporting dropped receives, this value +may be set too high, causing the driver to run out of available receive +descriptors. + +CAUTION: When setting RxIntDelay to a value other than 0, adapters may + hang (stop transmitting) under certain network conditions. If + this occurs a NETDEV WATCHDOG message is logged in the system + event log. In addition, the controller is automatically reset, + restoring the network connection. To eliminate the potential + for the hang ensure that RxIntDelay is set to 0. + +RxAbsIntDelay +------------- +(This parameter is supported only on 82540, 82545 and later adapters.) +Valid Range: 0-65535 (0=off) +Default Value: 128 + +This value, in units of 1.024 microseconds, limits the delay in which a +receive interrupt is generated. Useful only if RxIntDelay is non-zero, +this value ensures that an interrupt is generated after the initial +packet is received within the set amount of time. Proper tuning, +along with RxIntDelay, may improve traffic throughput in specific network +conditions. + +Speed +----- +(This parameter is supported only on adapters with copper connections.) +Valid Settings: 0, 10, 100, 1000 +Default Value: 0 (auto-negotiate at all supported speeds) + +Speed forces the line speed to the specified value in megabits per second +(Mbps). If this parameter is not specified or is set to 0 and the link +partner is set to auto-negotiate, the board will auto-detect the correct +speed. Duplex should also be set when Speed is set to either 10 or 100. + +TxDescriptors +------------- +Valid Range: 48-256 for 82542 and 82543-based adapters + 48-4096 for all other supported adapters +Default Value: 256 + +This value is the number of transmit descriptors allocated by the driver. +Increasing this value allows the driver to queue more transmits. Each +descriptor is 16 bytes. + +NOTE: Depending on the available system resources, the request for a + higher number of transmit descriptors may be denied. In this case, + use a lower number. + +TxIntDelay +---------- +Valid Range: 0-65535 (0=off) +Default Value: 8 + +This value delays the generation of transmit interrupts in units of +1.024 microseconds. Transmit interrupt reduction can improve CPU +efficiency if properly tuned for specific network traffic. If the +system is reporting dropped transmits, this value may be set too high +causing the driver to run out of available transmit descriptors. + +TxAbsIntDelay +------------- +(This parameter is supported only on 82540, 82545 and later adapters.) +Valid Range: 0-65535 (0=off) +Default Value: 32 + +This value, in units of 1.024 microseconds, limits the delay in which a +transmit interrupt is generated. Useful only if TxIntDelay is non-zero, +this value ensures that an interrupt is generated after the initial +packet is sent on the wire within the set amount of time. Proper tuning, +along with TxIntDelay, may improve traffic throughput in specific +network conditions. + +XsumRX +------ +(This parameter is NOT supported on the 82542-based adapter.) +Valid Range: 0-1 +Default Value: 1 + +A value of '1' indicates that the driver should enable IP checksum +offload for received packets (both UDP and TCP) to the adapter hardware. + +Copybreak +--------- +Valid Range: 0-xxxxxxx (0=off) +Default Value: 256 +Usage: modprobe e1000.ko copybreak=128 + +Driver copies all packets below or equaling this size to a fresh RX +buffer before handing it up the stack. + +This parameter is different than other parameters, in that it is a +single (not 1,1,1 etc.) parameter applied to all driver instances and +it is also available during runtime at +/sys/module/e1000/parameters/copybreak + +SmartPowerDownEnable +-------------------- +Valid Range: 0-1 +Default Value: 0 (disabled) + +Allows PHY to turn off in lower power states. The user can turn off +this parameter in supported chipsets. + +Speed and Duplex Configuration +============================== + +Three keywords are used to control the speed and duplex configuration. +These keywords are Speed, Duplex, and AutoNeg. + +If the board uses a fiber interface, these keywords are ignored, and the +fiber interface board only links at 1000 Mbps full-duplex. + +For copper-based boards, the keywords interact as follows: + + The default operation is auto-negotiate. The board advertises all + supported speed and duplex combinations, and it links at the highest + common speed and duplex mode IF the link partner is set to auto-negotiate. + + If Speed = 1000, limited auto-negotiation is enabled and only 1000 Mbps + is advertised (The 1000BaseT spec requires auto-negotiation.) + + If Speed = 10 or 100, then both Speed and Duplex should be set. Auto- + negotiation is disabled, and the AutoNeg parameter is ignored. Partner + SHOULD also be forced. + +The AutoNeg parameter is used when more control is required over the +auto-negotiation process. It should be used when you wish to control which +speed and duplex combinations are advertised during the auto-negotiation +process. + +The parameter may be specified as either a decimal or hexadecimal value as +determined by the bitmap below. + +Bit position 7 6 5 4 3 2 1 0 +Decimal Value 128 64 32 16 8 4 2 1 +Hex value 80 40 20 10 8 4 2 1 +Speed (Mbps) N/A N/A 1000 N/A 100 100 10 10 +Duplex Full Full Half Full Half + +Some examples of using AutoNeg: + + modprobe e1000 AutoNeg=0x01 (Restricts autonegotiation to 10 Half) + modprobe e1000 AutoNeg=1 (Same as above) + modprobe e1000 AutoNeg=0x02 (Restricts autonegotiation to 10 Full) + modprobe e1000 AutoNeg=0x03 (Restricts autonegotiation to 10 Half or 10 Full) + modprobe e1000 AutoNeg=0x04 (Restricts autonegotiation to 100 Half) + modprobe e1000 AutoNeg=0x05 (Restricts autonegotiation to 10 Half or 100 + Half) + modprobe e1000 AutoNeg=0x020 (Restricts autonegotiation to 1000 Full) + modprobe e1000 AutoNeg=32 (Same as above) + +Note that when this parameter is used, Speed and Duplex must not be specified. + +If the link partner is forced to a specific speed and duplex, then this +parameter should not be used. Instead, use the Speed and Duplex parameters +previously mentioned to force the adapter to the same speed and duplex. + +Additional Configurations +========================= + + Jumbo Frames + ------------ + Jumbo Frames support is enabled by changing the MTU to a value larger than + the default of 1500. Use the ifconfig command to increase the MTU size. + For example:: + + ifconfig eth mtu 9000 up + + This setting is not saved across reboots. It can be made permanent if + you add:: + + MTU=9000 + + to the file /etc/sysconfig/network-scripts/ifcfg-eth. This example + applies to the Red Hat distributions; other distributions may store this + setting in a different location. + + Notes: + Degradation in throughput performance may be observed in some Jumbo frames + environments. If this is observed, increasing the application's socket buffer + size and/or increasing the /proc/sys/net/ipv4/tcp_*mem entry values may help. + See the specific application manual and /usr/src/linux*/Documentation/ + networking/ip-sysctl.txt for more details. + + - The maximum MTU setting for Jumbo Frames is 16110. This value coincides + with the maximum Jumbo Frames size of 16128. + + - Using Jumbo frames at 10 or 100 Mbps is not supported and may result in + poor performance or loss of link. + + - Adapters based on the Intel(R) 82542 and 82573V/E controller do not + support Jumbo Frames. These correspond to the following product names: + Intel(R) PRO/1000 Gigabit Server Adapter + Intel(R) PRO/1000 PM Network Connection + + ethtool + ------- + The driver utilizes the ethtool interface for driver configuration and + diagnostics, as well as displaying statistical information. The ethtool + version 1.6 or later is required for this functionality. + + The latest release of ethtool can be found from + https://www.kernel.org/pub/software/network/ethtool/ + + Enabling Wake on LAN* (WoL) + --------------------------- + WoL is configured through the ethtool* utility. + + WoL will be enabled on the system during the next shut down or reboot. + For this driver version, in order to enable WoL, the e1000 driver must be + loaded when shutting down or rebooting the system. + +Support +======= + +For general information, go to the Intel support website at: + + http://support.intel.com + +or the Intel Wired Networking project hosted by Sourceforge at: + + http://sourceforge.net/projects/e1000 + +If an issue is identified with the released source code on the supported +kernel with a supported adapter, email the specific information related +to the issue to e1000-devel@lists.sf.net diff --git a/Documentation/networking/e1000.txt b/Documentation/networking/e1000.txt deleted file mode 100644 index 1f6ed848363d..000000000000 --- a/Documentation/networking/e1000.txt +++ /dev/null @@ -1,461 +0,0 @@ -Linux* Base Driver for Intel(R) Ethernet Network Connection -=========================================================== - -Intel Gigabit Linux driver. -Copyright(c) 1999 - 2013 Intel Corporation. - -Contents -======== - -- Identifying Your Adapter -- Command Line Parameters -- Speed and Duplex Configuration -- Additional Configurations -- Support - -Identifying Your Adapter -======================== - -For more information on how to identify your adapter, go to the Adapter & -Driver ID Guide at: - - http://support.intel.com/support/go/network/adapter/idguide.htm - -For the latest Intel network drivers for Linux, refer to the following -website. In the search field, enter your adapter name or type, or use the -networking link on the left to search for your adapter: - - http://support.intel.com/support/go/network/adapter/home.htm - -Command Line Parameters -======================= - -The default value for each parameter is generally the recommended setting, -unless otherwise noted. - -NOTES: For more information about the AutoNeg, Duplex, and Speed - parameters, see the "Speed and Duplex Configuration" section in - this document. - - For more information about the InterruptThrottleRate, - RxIntDelay, TxIntDelay, RxAbsIntDelay, and TxAbsIntDelay - parameters, see the application note at: - http://www.intel.com/design/network/applnots/ap450.htm - -AutoNeg -------- -(Supported only on adapters with copper connections) -Valid Range: 0x01-0x0F, 0x20-0x2F -Default Value: 0x2F - -This parameter is a bit-mask that specifies the speed and duplex settings -advertised by the adapter. When this parameter is used, the Speed and -Duplex parameters must not be specified. - -NOTE: Refer to the Speed and Duplex section of this readme for more - information on the AutoNeg parameter. - -Duplex ------- -(Supported only on adapters with copper connections) -Valid Range: 0-2 (0=auto-negotiate, 1=half, 2=full) -Default Value: 0 - -This defines the direction in which data is allowed to flow. Can be -either one or two-directional. If both Duplex and the link partner are -set to auto-negotiate, the board auto-detects the correct duplex. If the -link partner is forced (either full or half), Duplex defaults to half- -duplex. - -FlowControl ------------ -Valid Range: 0-3 (0=none, 1=Rx only, 2=Tx only, 3=Rx&Tx) -Default Value: Reads flow control settings from the EEPROM - -This parameter controls the automatic generation(Tx) and response(Rx) -to Ethernet PAUSE frames. - -InterruptThrottleRate ---------------------- -(not supported on Intel(R) 82542, 82543 or 82544-based adapters) -Valid Range: 0,1,3,4,100-100000 (0=off, 1=dynamic, 3=dynamic conservative, - 4=simplified balancing) -Default Value: 3 - -The driver can limit the amount of interrupts per second that the adapter -will generate for incoming packets. It does this by writing a value to the -adapter that is based on the maximum amount of interrupts that the adapter -will generate per second. - -Setting InterruptThrottleRate to a value greater or equal to 100 -will program the adapter to send out a maximum of that many interrupts -per second, even if more packets have come in. This reduces interrupt -load on the system and can lower CPU utilization under heavy load, -but will increase latency as packets are not processed as quickly. - -The default behaviour of the driver previously assumed a static -InterruptThrottleRate value of 8000, providing a good fallback value for -all traffic types,but lacking in small packet performance and latency. -The hardware can handle many more small packets per second however, and -for this reason an adaptive interrupt moderation algorithm was implemented. - -Since 7.3.x, the driver has two adaptive modes (setting 1 or 3) in which -it dynamically adjusts the InterruptThrottleRate value based on the traffic -that it receives. After determining the type of incoming traffic in the last -timeframe, it will adjust the InterruptThrottleRate to an appropriate value -for that traffic. - -The algorithm classifies the incoming traffic every interval into -classes. Once the class is determined, the InterruptThrottleRate value is -adjusted to suit that traffic type the best. There are three classes defined: -"Bulk traffic", for large amounts of packets of normal size; "Low latency", -for small amounts of traffic and/or a significant percentage of small -packets; and "Lowest latency", for almost completely small packets or -minimal traffic. - -In dynamic conservative mode, the InterruptThrottleRate value is set to 4000 -for traffic that falls in class "Bulk traffic". If traffic falls in the "Low -latency" or "Lowest latency" class, the InterruptThrottleRate is increased -stepwise to 20000. This default mode is suitable for most applications. - -For situations where low latency is vital such as cluster or -grid computing, the algorithm can reduce latency even more when -InterruptThrottleRate is set to mode 1. In this mode, which operates -the same as mode 3, the InterruptThrottleRate will be increased stepwise to -70000 for traffic in class "Lowest latency". - -In simplified mode the interrupt rate is based on the ratio of TX and -RX traffic. If the bytes per second rate is approximately equal, the -interrupt rate will drop as low as 2000 interrupts per second. If the -traffic is mostly transmit or mostly receive, the interrupt rate could -be as high as 8000. - -Setting InterruptThrottleRate to 0 turns off any interrupt moderation -and may improve small packet latency, but is generally not suitable -for bulk throughput traffic. - -NOTE: InterruptThrottleRate takes precedence over the TxAbsIntDelay and - RxAbsIntDelay parameters. In other words, minimizing the receive - and/or transmit absolute delays does not force the controller to - generate more interrupts than what the Interrupt Throttle Rate - allows. - -CAUTION: If you are using the Intel(R) PRO/1000 CT Network Connection - (controller 82547), setting InterruptThrottleRate to a value - greater than 75,000, may hang (stop transmitting) adapters - under certain network conditions. If this occurs a NETDEV - WATCHDOG message is logged in the system event log. In - addition, the controller is automatically reset, restoring - the network connection. To eliminate the potential for the - hang, ensure that InterruptThrottleRate is set no greater - than 75,000 and is not set to 0. - -NOTE: When e1000 is loaded with default settings and multiple adapters - are in use simultaneously, the CPU utilization may increase non- - linearly. In order to limit the CPU utilization without impacting - the overall throughput, we recommend that you load the driver as - follows: - - modprobe e1000 InterruptThrottleRate=3000,3000,3000 - - This sets the InterruptThrottleRate to 3000 interrupts/sec for - the first, second, and third instances of the driver. The range - of 2000 to 3000 interrupts per second works on a majority of - systems and is a good starting point, but the optimal value will - be platform-specific. If CPU utilization is not a concern, use - RX_POLLING (NAPI) and default driver settings. - -RxDescriptors -------------- -Valid Range: 80-256 for 82542 and 82543-based adapters - 80-4096 for all other supported adapters -Default Value: 256 - -This value specifies the number of receive buffer descriptors allocated -by the driver. Increasing this value allows the driver to buffer more -incoming packets, at the expense of increased system memory utilization. - -Each descriptor is 16 bytes. A receive buffer is also allocated for each -descriptor and can be either 2048, 4096, 8192, or 16384 bytes, depending -on the MTU setting. The maximum MTU size is 16110. - -NOTE: MTU designates the frame size. It only needs to be set for Jumbo - Frames. Depending on the available system resources, the request - for a higher number of receive descriptors may be denied. In this - case, use a lower number. - -RxIntDelay ----------- -Valid Range: 0-65535 (0=off) -Default Value: 0 - -This value delays the generation of receive interrupts in units of 1.024 -microseconds. Receive interrupt reduction can improve CPU efficiency if -properly tuned for specific network traffic. Increasing this value adds -extra latency to frame reception and can end up decreasing the throughput -of TCP traffic. If the system is reporting dropped receives, this value -may be set too high, causing the driver to run out of available receive -descriptors. - -CAUTION: When setting RxIntDelay to a value other than 0, adapters may - hang (stop transmitting) under certain network conditions. If - this occurs a NETDEV WATCHDOG message is logged in the system - event log. In addition, the controller is automatically reset, - restoring the network connection. To eliminate the potential - for the hang ensure that RxIntDelay is set to 0. - -RxAbsIntDelay -------------- -(This parameter is supported only on 82540, 82545 and later adapters.) -Valid Range: 0-65535 (0=off) -Default Value: 128 - -This value, in units of 1.024 microseconds, limits the delay in which a -receive interrupt is generated. Useful only if RxIntDelay is non-zero, -this value ensures that an interrupt is generated after the initial -packet is received within the set amount of time. Proper tuning, -along with RxIntDelay, may improve traffic throughput in specific network -conditions. - -Speed ------ -(This parameter is supported only on adapters with copper connections.) -Valid Settings: 0, 10, 100, 1000 -Default Value: 0 (auto-negotiate at all supported speeds) - -Speed forces the line speed to the specified value in megabits per second -(Mbps). If this parameter is not specified or is set to 0 and the link -partner is set to auto-negotiate, the board will auto-detect the correct -speed. Duplex should also be set when Speed is set to either 10 or 100. - -TxDescriptors -------------- -Valid Range: 80-256 for 82542 and 82543-based adapters - 80-4096 for all other supported adapters -Default Value: 256 - -This value is the number of transmit descriptors allocated by the driver. -Increasing this value allows the driver to queue more transmits. Each -descriptor is 16 bytes. - -NOTE: Depending on the available system resources, the request for a - higher number of transmit descriptors may be denied. In this case, - use a lower number. - -TxDescriptorStep ----------------- -Valid Range: 1 (use every Tx Descriptor) - 4 (use every 4th Tx Descriptor) - -Default Value: 1 (use every Tx Descriptor) - -On certain non-Intel architectures, it has been observed that intense TX -traffic bursts of short packets may result in an improper descriptor -writeback. If this occurs, the driver will report a "TX Timeout" and reset -the adapter, after which the transmit flow will restart, though data may -have stalled for as much as 10 seconds before it resumes. - -The improper writeback does not occur on the first descriptor in a system -memory cache-line, which is typically 32 bytes, or 4 descriptors long. - -Setting TxDescriptorStep to a value of 4 will ensure that all TX descriptors -are aligned to the start of a system memory cache line, and so this problem -will not occur. - -NOTES: Setting TxDescriptorStep to 4 effectively reduces the number of - TxDescriptors available for transmits to 1/4 of the normal allocation. - This has a possible negative performance impact, which may be - compensated for by allocating more descriptors using the TxDescriptors - module parameter. - - There are other conditions which may result in "TX Timeout", which will - not be resolved by the use of the TxDescriptorStep parameter. As the - issue addressed by this parameter has never been observed on Intel - Architecture platforms, it should not be used on Intel platforms. - -TxIntDelay ----------- -Valid Range: 0-65535 (0=off) -Default Value: 64 - -This value delays the generation of transmit interrupts in units of -1.024 microseconds. Transmit interrupt reduction can improve CPU -efficiency if properly tuned for specific network traffic. If the -system is reporting dropped transmits, this value may be set too high -causing the driver to run out of available transmit descriptors. - -TxAbsIntDelay -------------- -(This parameter is supported only on 82540, 82545 and later adapters.) -Valid Range: 0-65535 (0=off) -Default Value: 64 - -This value, in units of 1.024 microseconds, limits the delay in which a -transmit interrupt is generated. Useful only if TxIntDelay is non-zero, -this value ensures that an interrupt is generated after the initial -packet is sent on the wire within the set amount of time. Proper tuning, -along with TxIntDelay, may improve traffic throughput in specific -network conditions. - -XsumRX ------- -(This parameter is NOT supported on the 82542-based adapter.) -Valid Range: 0-1 -Default Value: 1 - -A value of '1' indicates that the driver should enable IP checksum -offload for received packets (both UDP and TCP) to the adapter hardware. - -Copybreak ---------- -Valid Range: 0-xxxxxxx (0=off) -Default Value: 256 -Usage: insmod e1000.ko copybreak=128 - -Driver copies all packets below or equaling this size to a fresh RX -buffer before handing it up the stack. - -This parameter is different than other parameters, in that it is a -single (not 1,1,1 etc.) parameter applied to all driver instances and -it is also available during runtime at -/sys/module/e1000/parameters/copybreak - -SmartPowerDownEnable --------------------- -Valid Range: 0-1 -Default Value: 0 (disabled) - -Allows PHY to turn off in lower power states. The user can turn off -this parameter in supported chipsets. - -KumeranLockLoss ---------------- -Valid Range: 0-1 -Default Value: 1 (enabled) - -This workaround skips resetting the PHY at shutdown for the initial -silicon releases of ICH8 systems. - -Speed and Duplex Configuration -============================== - -Three keywords are used to control the speed and duplex configuration. -These keywords are Speed, Duplex, and AutoNeg. - -If the board uses a fiber interface, these keywords are ignored, and the -fiber interface board only links at 1000 Mbps full-duplex. - -For copper-based boards, the keywords interact as follows: - - The default operation is auto-negotiate. The board advertises all - supported speed and duplex combinations, and it links at the highest - common speed and duplex mode IF the link partner is set to auto-negotiate. - - If Speed = 1000, limited auto-negotiation is enabled and only 1000 Mbps - is advertised (The 1000BaseT spec requires auto-negotiation.) - - If Speed = 10 or 100, then both Speed and Duplex should be set. Auto- - negotiation is disabled, and the AutoNeg parameter is ignored. Partner - SHOULD also be forced. - -The AutoNeg parameter is used when more control is required over the -auto-negotiation process. It should be used when you wish to control which -speed and duplex combinations are advertised during the auto-negotiation -process. - -The parameter may be specified as either a decimal or hexadecimal value as -determined by the bitmap below. - -Bit position 7 6 5 4 3 2 1 0 -Decimal Value 128 64 32 16 8 4 2 1 -Hex value 80 40 20 10 8 4 2 1 -Speed (Mbps) N/A N/A 1000 N/A 100 100 10 10 -Duplex Full Full Half Full Half - -Some examples of using AutoNeg: - - modprobe e1000 AutoNeg=0x01 (Restricts autonegotiation to 10 Half) - modprobe e1000 AutoNeg=1 (Same as above) - modprobe e1000 AutoNeg=0x02 (Restricts autonegotiation to 10 Full) - modprobe e1000 AutoNeg=0x03 (Restricts autonegotiation to 10 Half or 10 Full) - modprobe e1000 AutoNeg=0x04 (Restricts autonegotiation to 100 Half) - modprobe e1000 AutoNeg=0x05 (Restricts autonegotiation to 10 Half or 100 - Half) - modprobe e1000 AutoNeg=0x020 (Restricts autonegotiation to 1000 Full) - modprobe e1000 AutoNeg=32 (Same as above) - -Note that when this parameter is used, Speed and Duplex must not be specified. - -If the link partner is forced to a specific speed and duplex, then this -parameter should not be used. Instead, use the Speed and Duplex parameters -previously mentioned to force the adapter to the same speed and duplex. - -Additional Configurations -========================= - - Jumbo Frames - ------------ - Jumbo Frames support is enabled by changing the MTU to a value larger than - the default of 1500. Use the ifconfig command to increase the MTU size. - For example: - - ifconfig eth mtu 9000 up - - This setting is not saved across reboots. It can be made permanent if - you add: - - MTU=9000 - - to the file /etc/sysconfig/network-scripts/ifcfg-eth. This example - applies to the Red Hat distributions; other distributions may store this - setting in a different location. - - Notes: - Degradation in throughput performance may be observed in some Jumbo frames - environments. If this is observed, increasing the application's socket buffer - size and/or increasing the /proc/sys/net/ipv4/tcp_*mem entry values may help. - See the specific application manual and /usr/src/linux*/Documentation/ - networking/ip-sysctl.txt for more details. - - - The maximum MTU setting for Jumbo Frames is 16110. This value coincides - with the maximum Jumbo Frames size of 16128. - - - Using Jumbo frames at 10 or 100 Mbps is not supported and may result in - poor performance or loss of link. - - - Adapters based on the Intel(R) 82542 and 82573V/E controller do not - support Jumbo Frames. These correspond to the following product names: - Intel(R) PRO/1000 Gigabit Server Adapter - Intel(R) PRO/1000 PM Network Connection - - ethtool - ------- - The driver utilizes the ethtool interface for driver configuration and - diagnostics, as well as displaying statistical information. The ethtool - version 1.6 or later is required for this functionality. - - The latest release of ethtool can be found from - https://www.kernel.org/pub/software/network/ethtool/ - - Enabling Wake on LAN* (WoL) - --------------------------- - WoL is configured through the ethtool* utility. - - WoL will be enabled on the system during the next shut down or reboot. - For this driver version, in order to enable WoL, the e1000 driver must be - loaded when shutting down or rebooting the system. - -Support -======= - -For general information, go to the Intel support website at: - - http://support.intel.com - -or the Intel Wired Networking project hosted by Sourceforge at: - - http://sourceforge.net/projects/e1000 - -If an issue is identified with the released source code on the supported -kernel with a supported adapter, email the specific information related -to the issue to e1000-devel@lists.sf.net diff --git a/Documentation/networking/failover.rst b/Documentation/networking/failover.rst new file mode 100644 index 000000000000..f0c8483cdbf5 --- /dev/null +++ b/Documentation/networking/failover.rst @@ -0,0 +1,18 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======== +FAILOVER +======== + +Overview +======== + +The failover module provides a generic interface for paravirtual drivers +to register a netdev and a set of ops with a failover instance. The ops +are used as event handlers that get called to handle netdev register/ +unregister/link change/name change events on slave pci ethernet devices +with the same mac address as the failover netdev. + +This enables paravirtual drivers to use a VF as an accelerated low latency +datapath. It also allows live migration of VMs with direct attached VFs by +failing over to the paravirtual datapath when the VF is unplugged. diff --git a/Documentation/networking/filter.txt b/Documentation/networking/filter.txt index fd55c7de9991..e6b4ebb2b243 100644 --- a/Documentation/networking/filter.txt +++ b/Documentation/networking/filter.txt @@ -483,6 +483,12 @@ Example output from dmesg: [ 3389.935851] JIT code: 00000030: 00 e8 28 94 ff e0 83 f8 01 75 07 b8 ff ff 00 00 [ 3389.935852] JIT code: 00000040: eb 02 31 c0 c9 c3 +When CONFIG_BPF_JIT_ALWAYS_ON is enabled, bpf_jit_enable is permanently set to 1 and +setting any other value than that will return in failure. This is even the case for +setting bpf_jit_enable to 2, since dumping the final JIT image into the kernel log +is discouraged and introspection through bpftool (under tools/bpf/bpftool/) is the +generally recommended approach instead. + In the kernel source tree under tools/bpf/, there's bpf_jit_disasm for generating disassembly out of the kernel log's hexdump: @@ -1136,6 +1142,7 @@ into a register from memory, the register's top 56 bits are known zero, while the low 8 are unknown - which is represented as the tnum (0x0; 0xff). If we then OR this with 0x40, we get (0x40; 0xbf), then if we add 1 we get (0x0; 0x1ff), because of potential carries. + Besides arithmetic, the register state can also be updated by conditional branches. For instance, if a SCALAR_VALUE is compared > 8, in the 'true' branch it will have a umin_value (unsigned minimum value) of 9, whereas in the 'false' @@ -1144,14 +1151,16 @@ BPF_JSGE) would instead update the signed minimum/maximum values. Information from the signed and unsigned bounds can be combined; for instance if a value is first tested < 8 and then tested s> 4, the verifier will conclude that the value is also > 4 and s< 8, since the bounds prevent crossing the sign boundary. + PTR_TO_PACKETs with a variable offset part have an 'id', which is common to all pointers sharing that same variable offset. This is important for packet range -checks: after adding some variable to a packet pointer, if you then copy it to -another register and (say) add a constant 4, both registers will share the same -'id' but one will have a fixed offset of +4. Then if it is bounds-checked and -found to be less than a PTR_TO_PACKET_END, the other register is now known to -have a safe range of at least 4 bytes. See 'Direct packet access', below, for -more on PTR_TO_PACKET ranges. +checks: after adding a variable to a packet pointer register A, if you then copy +it to another register B and then add a constant 4 to A, both registers will +share the same 'id' but the A will have a fixed offset of +4. Then if A is +bounds-checked and found to be less than a PTR_TO_PACKET_END, the register B is +now known to have a safe range of at least 4 bytes. See 'Direct packet access', +below, for more on PTR_TO_PACKET ranges. + The 'id' field is also used on PTR_TO_MAP_VALUE_OR_NULL, common to all copies of the pointer returned from a map lookup. This means that when one copy is checked and found to be non-NULL, all copies can become PTR_TO_MAP_VALUEs. diff --git a/Documentation/networking/gtp.txt b/Documentation/networking/gtp.txt index 0d9c18f05ec6..6966bbec1ecb 100644 --- a/Documentation/networking/gtp.txt +++ b/Documentation/networking/gtp.txt @@ -67,7 +67,7 @@ Don't be confused by terminology: The GTP User Plane goes through kernel accelerated path, while the GTP Control Plane goes to Userspace :) -The official homepge of the module is at +The official homepage of the module is at https://osmocom.org/projects/linux-kernel-gtp-u/wiki == Userspace Programs with Linux Kernel GTP-U support == @@ -120,7 +120,7 @@ If yo have questions regarding how to use the Kernel GTP module from your own software, or want to contribute to the code, please use the osmocom-net-grps mailing list for related discussion. The list can be reached at osmocom-net-gprs@lists.osmocom.org and the mailman -interface for managign your subscription is at +interface for managing your subscription is at https://lists.osmocom.org/mailman/listinfo/osmocom-net-gprs == Issue Tracker == diff --git a/Documentation/networking/ila.txt b/Documentation/networking/ila.txt index 78df879abd26..a17dac9dc915 100644 --- a/Documentation/networking/ila.txt +++ b/Documentation/networking/ila.txt @@ -121,7 +121,7 @@ three options to deal with this: - checksum neutral mapping When an address is translated the difference can be offset - elsewhere in a part of the packet that is covered by the + elsewhere in a part of the packet that is covered by the checksum. The low order sixteen bits of the identifier are used. This method is preferred since it doesn't require parsing a packet beyond the IP header and in most cases the diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst index f204eaff657d..fec8588a588e 100644 --- a/Documentation/networking/index.rst +++ b/Documentation/networking/index.rst @@ -6,9 +6,12 @@ Contents: .. toctree:: :maxdepth: 2 + af_xdp batman-adv can dpaa2/index + e100 + e1000 kapi z8530book msg_zerocopy diff --git a/Documentation/networking/ip-sysctl.txt b/Documentation/networking/ip-sysctl.txt index 35ffaa281b26..ce8fbf5aa63c 100644 --- a/Documentation/networking/ip-sysctl.txt +++ b/Documentation/networking/ip-sysctl.txt @@ -26,7 +26,7 @@ ip_no_pmtu_disc - INTEGER discarded. Outgoing frames are handled the same as in mode 1, implicitly setting IP_PMTUDISC_DONT on every created socket. - Mode 3 is a hardend pmtu discover mode. The kernel will only + Mode 3 is a hardened pmtu discover mode. The kernel will only accept fragmentation-needed errors if the underlying protocol can verify them besides a plain socket lookup. Current protocols for which pmtu events will be honored are TCP, SCTP @@ -449,8 +449,10 @@ tcp_recovery - INTEGER features. RACK: 0x1 enables the RACK loss detection for fast detection of lost - retransmissions and tail drops. + retransmissions and tail drops. It also subsumes and disables + RFC6675 recovery for SACK connections. RACK: 0x2 makes RACK's reordering window static (min_rtt/4). + RACK: 0x4 disables RACK's DUPACK threshold heuristic Default: 0x1 @@ -523,6 +525,19 @@ tcp_rmem - vector of 3 INTEGERs: min, default, max tcp_sack - BOOLEAN Enable select acknowledgments (SACKS). +tcp_comp_sack_delay_ns - LONG INTEGER + TCP tries to reduce number of SACK sent, using a timer + based on 5% of SRTT, capped by this sysctl, in nano seconds. + The default is 1ms, based on TSO autosizing period. + + Default : 1,000,000 ns (1 ms) + +tcp_comp_sack_nr - INTEGER + Max numer of SACK that can be compressed. + Using 0 disables SACK compression. + + Detault : 44 + tcp_slow_start_after_idle - BOOLEAN If set, provide RFC2861 behavior and time out the congestion window after an idle period. An idle period is defined at @@ -652,11 +667,15 @@ tcp_tso_win_divisor - INTEGER building larger TSO frames. Default: 3 -tcp_tw_reuse - BOOLEAN - Allow to reuse TIME-WAIT sockets for new connections when it is - safe from protocol viewpoint. Default value is 0. +tcp_tw_reuse - INTEGER + Enable reuse of TIME-WAIT sockets for new connections when it is + safe from protocol viewpoint. + 0 - disable + 1 - global enable + 2 - enable for loopback traffic only It should not be changed without advice/request of technical experts. + Default: 2 tcp_window_scaling - BOOLEAN Enable window scaling as defined in RFC1323. @@ -1428,6 +1447,19 @@ ip6frag_low_thresh - INTEGER ip6frag_time - INTEGER Time in seconds to keep an IPv6 fragment in memory. +IPv6 Segment Routing: + +seg6_flowlabel - INTEGER + Controls the behaviour of computing the flowlabel of outer + IPv6 header in case of SR T.encaps + + -1 set flowlabel to zero. + 0 copy flowlabel from Inner packet in case of Inner IPv6 + (Set flowlabel to 0 in case IPv4/L2) + 1 Compute the flowlabel using seg6_make_flowlabel() + + Default is 0. + conf/default/*: Change the interface-specific default settings. diff --git a/Documentation/networking/ipsec.txt b/Documentation/networking/ipsec.txt index 8dbc08b7e431..ba794b7e51be 100644 --- a/Documentation/networking/ipsec.txt +++ b/Documentation/networking/ipsec.txt @@ -25,8 +25,8 @@ Quote from RFC3173: is implementation dependent. Current IPComp implementation is indeed by the book, while as in practice -when sending non-compressed packet to the peer(whether or not packet len -is smaller than the threshold or the compressed len is large than original +when sending non-compressed packet to the peer (whether or not packet len +is smaller than the threshold or the compressed len is larger than original packet len), the packet is dropped when checking the policy as this packet matches the selector but not coming from any XFRM layer, i.e., with no security path. Such naked packet will not eventually make it to upper layer. diff --git a/Documentation/networking/ipvlan.txt b/Documentation/networking/ipvlan.txt index 812ef003e0a8..27a38e50c287 100644 --- a/Documentation/networking/ipvlan.txt +++ b/Documentation/networking/ipvlan.txt @@ -73,11 +73,11 @@ mode to make conn-tracking work. This is the default option. To configure the IPvlan port in this mode, user can choose to either add this option on the command-line or don't specify anything. This is the traditional mode where slaves can cross-talk among -themseleves apart from talking through the master device. +themselves apart from talking through the master device. 5.2 private: If this option is added to the command-line, the port is set in private -mode. i.e. port wont allow cross communication between slaves. +mode. i.e. port won't allow cross communication between slaves. 5.3 vepa: If this is added to the command-line, the port is set in VEPA mode. diff --git a/Documentation/networking/kcm.txt b/Documentation/networking/kcm.txt index 9a513295b07c..b773a5278ac4 100644 --- a/Documentation/networking/kcm.txt +++ b/Documentation/networking/kcm.txt @@ -1,4 +1,4 @@ -Kernel Connection Mulitplexor +Kernel Connection Multiplexor ----------------------------- Kernel Connection Multiplexor (KCM) is a mechanism that provides a message based @@ -31,7 +31,7 @@ KCM implements an NxM multiplexor in the kernel as diagrammed below: KCM sockets ----------- -The KCM sockets provide the user interface to the muliplexor. All the KCM sockets +The KCM sockets provide the user interface to the multiplexor. All the KCM sockets bound to a multiplexor are considered to have equivalent function, and I/O operations in different sockets may be done in parallel without the need for synchronization between threads in userspace. @@ -199,7 +199,7 @@ while. Example use: BFP programs for message delineation ------------------------------------ -BPF programs can be compiled using the BPF LLVM backend. For exmple, +BPF programs can be compiled using the BPF LLVM backend. For example, the BPF program for parsing Thrift is: #include "bpf.h" /* for __sk_buff */ @@ -222,7 +222,7 @@ messages. The kernel provides necessary assurances that messages are sent and received atomically. This relieves much of the burden applications have in mapping a message based protocol onto the TCP stream. KCM also make application layer messages a unit of work in the kernel for the purposes of -steerng and scheduling, which in turn allows a simpler networking model in +steering and scheduling, which in turn allows a simpler networking model in multithreaded applications. Configurations @@ -272,7 +272,7 @@ on the socket thus waking up the application thread. When the application sees the error (which may just be a disconnect) it should unattach the socket from KCM and then close it. It is assumed that once an error is posted on the TCP socket the data stream is unrecoverable (i.e. an error -may have occurred in the middle of receiving a messssge). +may have occurred in the middle of receiving a message). TCP connection monitoring ------------------------- diff --git a/Documentation/networking/net_failover.rst b/Documentation/networking/net_failover.rst new file mode 100644 index 000000000000..70ca2f5800c4 --- /dev/null +++ b/Documentation/networking/net_failover.rst @@ -0,0 +1,116 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============ +NET_FAILOVER +============ + +Overview +======== + +The net_failover driver provides an automated failover mechanism via APIs +to create and destroy a failover master netdev and mananges a primary and +standby slave netdevs that get registered via the generic failover +infrastructrure. + +The failover netdev acts a master device and controls 2 slave devices. The +original paravirtual interface is registered as 'standby' slave netdev and +a passthru/vf device with the same MAC gets registered as 'primary' slave +netdev. Both 'standby' and 'failover' netdevs are associated with the same +'pci' device. The user accesses the network interface via 'failover' netdev. +The 'failover' netdev chooses 'primary' netdev as default for transmits when +it is available with link up and running. + +This can be used by paravirtual drivers to enable an alternate low latency +datapath. It also enables hypervisor controlled live migration of a VM with +direct attached VF by failing over to the paravirtual datapath when the VF +is unplugged. + +virtio-net accelerated datapath: STANDBY mode +============================================= + +net_failover enables hypervisor controlled accelerated datapath to virtio-net +enabled VMs in a transparent manner with no/minimal guest userspace chanages. + +To support this, the hypervisor needs to enable VIRTIO_NET_F_STANDBY +feature on the virtio-net interface and assign the same MAC address to both +virtio-net and VF interfaces. + +Here is an example XML snippet that shows such configuration. + + + + + + + + +
+ + + + +
+ +
+ + +Booting a VM with the above configuration will result in the following 3 +netdevs created in the VM. + +4: ens10: mtu 1500 qdisc noqueue state UP group default qlen 1000 + link/ether 52:54:00:00:12:53 brd ff:ff:ff:ff:ff:ff + inet 192.168.12.53/24 brd 192.168.12.255 scope global dynamic ens10 + valid_lft 42482sec preferred_lft 42482sec + inet6 fe80::97d8:db2:8c10:b6d6/64 scope link + valid_lft forever preferred_lft forever +5: ens10nsby: mtu 1500 qdisc fq_codel master ens10 state UP group default qlen 1000 + link/ether 52:54:00:00:12:53 brd ff:ff:ff:ff:ff:ff +7: ens11: mtu 1500 qdisc mq master ens10 state UP group default qlen 1000 + link/ether 52:54:00:00:12:53 brd ff:ff:ff:ff:ff:ff + +ens10 is the 'failover' master netdev, ens10nsby and ens11 are the slave +'standby' and 'primary' netdevs respectively. + +Live Migration of a VM with SR-IOV VF & virtio-net in STANDBY mode +================================================================== + +net_failover also enables hypervisor controlled live migration to be supported +with VMs that have direct attached SR-IOV VF devices by automatic failover to +the paravirtual datapath when the VF is unplugged. + +Here is a sample script that shows the steps to initiate live migration on +the source hypervisor. + +# cat vf_xml + + + +
+ +
+ + +# Source Hypervisor +#!/bin/bash + +DOMAIN=fedora27-tap01 +PF=enp66s0f0 +VF_NUM=5 +TAP_IF=tap01 +VF_XML= + +MAC=52:54:00:00:12:53 +ZERO_MAC=00:00:00:00:00:00 + +virsh domif-setlink $DOMAIN $TAP_IF up +bridge fdb del $MAC dev $PF master +virsh detach-device $DOMAIN $VF_XML +ip link set $PF vf $VF_NUM mac $ZERO_MAC + +virsh migrate --live $DOMAIN qemu+ssh://$REMOTE_HOST/system + +# Destination Hypervisor +#!/bin/bash + +virsh attach-device $DOMAIN $VF_XML +virsh domif-setlink $DOMAIN $TAP_IF down diff --git a/Documentation/networking/netdev-FAQ.txt b/Documentation/networking/netdev-FAQ.txt index 2a3278d5cf35..fa951b820b25 100644 --- a/Documentation/networking/netdev-FAQ.txt +++ b/Documentation/networking/netdev-FAQ.txt @@ -179,6 +179,15 @@ A: No. See above answer. In short, if you think it really belongs in dash marker line as described in Documentation/process/submitting-patches.rst to temporarily embed that information into the patch that you send. +Q: Are all networking bug fixes backported to all stable releases? + +A: Due to capacity, Dave could only take care of the backports for the last + 2 stable releases. For earlier stable releases, each stable branch maintainer + is supposed to take care of them. If you find any patch is missing from an + earlier stable branch, please notify stable@vger.kernel.org with either a + commit ID or a formal patch backported, and CC Dave and other relevant + networking developers. + Q: Someone said that the comment style and coding convention is different for the networking content. Is this true? diff --git a/Documentation/networking/netdev-features.txt b/Documentation/networking/netdev-features.txt index c77f9d57eb91..c4a54c162547 100644 --- a/Documentation/networking/netdev-features.txt +++ b/Documentation/networking/netdev-features.txt @@ -113,6 +113,13 @@ whatever headers there might be. NETIF_F_TSO_ECN means that hardware can properly split packets with CWR bit set, be it TCPv4 (when NETIF_F_TSO is enabled) or TCPv6 (NETIF_F_TSO6). + * Transmit UDP segmentation offload + +NETIF_F_GSO_UDP_GSO_L4 accepts a single UDP header with a payload that exceeds +gso_size. On segmentation, it segments the payload on gso_size boundaries and +replicates the network and UDP headers (fixing up the last one if less than +gso_size). + * Transmit DMA from high memory On platforms where this is relevant, NETIF_F_HIGHDMA signals that diff --git a/Documentation/networking/nf_conntrack-sysctl.txt b/Documentation/networking/nf_conntrack-sysctl.txt index 433b6724797a..1669dc2419fd 100644 --- a/Documentation/networking/nf_conntrack-sysctl.txt +++ b/Documentation/networking/nf_conntrack-sysctl.txt @@ -156,7 +156,7 @@ nf_conntrack_timestamp - BOOLEAN nf_conntrack_udp_timeout - INTEGER (seconds) default 30 -nf_conntrack_udp_timeout_stream2 - INTEGER (seconds) +nf_conntrack_udp_timeout_stream - INTEGER (seconds) default 180 This extended timeout will be used in case there is an UDP stream diff --git a/Documentation/process/2.Process.rst b/Documentation/process/2.Process.rst index ce5561bb3f8e..a9c46dd0706b 100644 --- a/Documentation/process/2.Process.rst +++ b/Documentation/process/2.Process.rst @@ -18,17 +18,17 @@ major kernel release happening every two or three months. The recent release history looks like this: ====== ================= - 2.6.38 March 14, 2011 - 2.6.37 January 4, 2011 - 2.6.36 October 20, 2010 - 2.6.35 August 1, 2010 - 2.6.34 May 15, 2010 - 2.6.33 February 24, 2010 + 4.11 April 30, 2017 + 4.12 July 2, 2017 + 4.13 September 3, 2017 + 4.14 November 12, 2017 + 4.15 January 28, 2018 + 4.16 April 1, 2018 ====== ================= -Every 2.6.x release is a major kernel release with new features, internal -API changes, and more. A typical 2.6 release can contain nearly 10,000 -changesets with changes to several hundred thousand lines of code. 2.6 is +Every 4.x release is a major kernel release with new features, internal +API changes, and more. A typical 4.x release contain about 13,000 +changesets with changes to several hundred thousand lines of code. 4.x is thus the leading edge of Linux kernel development; the kernel uses a rolling development model which is continually integrating major changes. @@ -70,20 +70,19 @@ will get up to somewhere between -rc6 and -rc9 before the kernel is considered to be sufficiently stable and the final 2.6.x release is made. At that point the whole process starts over again. -As an example, here is how the 2.6.38 development cycle went (all dates in -2011): +As an example, here is how the 4.16 development cycle went (all dates in +2018): ============== =============================== - January 4 2.6.37 stable release - January 18 2.6.38-rc1, merge window closes - January 21 2.6.38-rc2 - February 1 2.6.38-rc3 - February 7 2.6.38-rc4 - February 15 2.6.38-rc5 - February 21 2.6.38-rc6 - March 1 2.6.38-rc7 - March 7 2.6.38-rc8 - March 14 2.6.38 stable release + January 28 4.15 stable release + February 11 4.16-rc1, merge window closes + February 18 4.16-rc2 + February 25 4.16-rc3 + March 4 4.16-rc4 + March 11 4.16-rc5 + March 18 4.16-rc6 + March 25 4.16-rc7 + April 1 4.17 stable release ============== =============================== How do the developers decide when to close the development cycle and create @@ -99,37 +98,42 @@ release is made. In the real world, this kind of perfection is hard to achieve; there are just too many variables in a project of this size. There comes a point where delaying the final release just makes the problem worse; the pile of changes waiting for the next merge window will grow -larger, creating even more regressions the next time around. So most 2.6.x +larger, creating even more regressions the next time around. So most 4.x kernels go out with a handful of known regressions though, hopefully, none of them are serious. Once a stable release is made, its ongoing maintenance is passed off to the "stable team," currently consisting of Greg Kroah-Hartman. The stable team -will release occasional updates to the stable release using the 2.6.x.y +will release occasional updates to the stable release using the 4.x.y numbering scheme. To be considered for an update release, a patch must (1) fix a significant bug, and (2) already be merged into the mainline for the next development kernel. Kernels will typically receive stable updates for a little more than one development cycle past their initial release. So, -for example, the 2.6.36 kernel's history looked like: +for example, the 4.13 kernel's history looked like: ============== =============================== - October 10 2.6.36 stable release - November 22 2.6.36.1 - December 9 2.6.36.2 - January 7 2.6.36.3 - February 17 2.6.36.4 + September 3 4.13 stable release + September 13 4.13.1 + September 20 4.13.2 + September 27 4.13.3 + October 5 4.13.4 + October 12 4.13.5 + ... ... + November 24 4.13.16 ============== =============================== -2.6.36.4 was the final stable update for the 2.6.36 release. +4.13.16 was the final stable update of the 4.13 release. Some kernels are designated "long term" kernels; they will receive support for a longer period. As of this writing, the current long term kernels and their maintainers are: - ====== ====================== =========================== - 2.6.27 Willy Tarreau (Deep-frozen stable kernel) - 2.6.32 Greg Kroah-Hartman - 2.6.35 Andi Kleen (Embedded flag kernel) + ====== ====================== ============================== + 3.16 Ben Hutchings (very long-term stable kernel) + 4.1 Sasha Levin + 4.4 Greg Kroah-Hartman (very long-term stable kernel) + 4.9 Greg Kroah-Hartman + 4.14 Greg Kroah-Hartman ====== ====================== =========================== The selection of a kernel for long-term support is purely a matter of a diff --git a/Documentation/process/5.Posting.rst b/Documentation/process/5.Posting.rst index c209d70da66f..c418c5d6cae4 100644 --- a/Documentation/process/5.Posting.rst +++ b/Documentation/process/5.Posting.rst @@ -10,8 +10,8 @@ of conventions and procedures which are used in the posting of patches; following them will make life much easier for everybody involved. This document will attempt to cover these expectations in reasonable detail; more information can also be found in the files process/submitting-patches.rst, -process/submitting-drivers.rst, and process/submit-checklist.rst in the kernel documentation -directory. +process/submitting-drivers.rst, and process/submit-checklist.rst in the kernel +documentation directory. When to post @@ -198,8 +198,8 @@ pass it to diff with the "-X" option. The tags mentioned above are used to describe how various developers have been associated with the development of this patch. They are described in -detail in the process/submitting-patches.rst document; what follows here is a brief -summary. Each of these lines has the format: +detail in the process/submitting-patches.rst document; what follows here is a +brief summary. Each of these lines has the format: :: @@ -210,8 +210,8 @@ The tags in common use are: - Signed-off-by: this is a developer's certification that he or she has the right to submit the patch for inclusion into the kernel. It is an agreement to the Developer's Certificate of Origin, the full text of - which can be found in Documentation/process/submitting-patches.rst. Code without a - proper signoff cannot be merged into the mainline. + which can be found in Documentation/process/submitting-patches.rst. Code + without a proper signoff cannot be merged into the mainline. - Co-developed-by: states that the patch was also created by another developer along with the original author. This is useful at times when multiple @@ -226,8 +226,8 @@ The tags in common use are: it to work. - Reviewed-by: the named developer has reviewed the patch for correctness; - see the reviewer's statement in Documentation/process/submitting-patches.rst for more - detail. + see the reviewer's statement in Documentation/process/submitting-patches.rst + for more detail. - Reported-by: names a user who reported a problem which is fixed by this patch; this tag is used to give credit to the (often underappreciated) diff --git a/Documentation/process/index.rst b/Documentation/process/index.rst index 1c9fe657ed01..37bd0628b6ee 100644 --- a/Documentation/process/index.rst +++ b/Documentation/process/index.rst @@ -52,6 +52,7 @@ lack of a better place. adding-syscalls magic-number volatile-considered-harmful + clang-format .. only:: subproject and html diff --git a/Documentation/process/maintainer-pgp-guide.rst b/Documentation/process/maintainer-pgp-guide.rst index b453561a7148..aff9b1a4d77b 100644 --- a/Documentation/process/maintainer-pgp-guide.rst +++ b/Documentation/process/maintainer-pgp-guide.rst @@ -219,7 +219,7 @@ Our goal is to protect your master key by moving it to offline media, so if you only have a combined **[SC]** key, then you should create a separate signing subkey:: - $ gpg --quick-add-key [fpr] ed25519 sign + $ gpg --quick-addkey [fpr] ed25519 sign Remember to tell the keyservers about this change, so others can pull down your new subkey:: @@ -450,11 +450,18 @@ functionality. There are several options available: others. If you want to use ECC keys, your best bet among commercially available devices is the Nitrokey Start. +.. note:: + + If you are listed in MAINTAINERS or have an account at kernel.org, + you `qualify for a free Nitrokey Start`_ courtesy of The Linux + Foundation. + .. _`Nitrokey Start`: https://shop.nitrokey.com/shop/product/nitrokey-start-6 .. _`Nitrokey Pro`: https://shop.nitrokey.com/shop/product/nitrokey-pro-3 .. _`Yubikey 4`: https://www.yubico.com/product/yubikey-4-series/ .. _Gnuk: http://www.fsij.org/doc-gnuk/ .. _`LWN has a good review`: https://lwn.net/Articles/736231/ +.. _`qualify for a free Nitrokey Start`: https://www.kernel.org/nitrokey-digital-tokens-for-kernel-developers.html Configure your smartcard device ------------------------------- @@ -482,7 +489,7 @@ there are no convenient command-line switches:: You should set the user PIN (1), Admin PIN (3), and the Reset Code (4). Please make sure to record and store these in a safe place -- especially the Admin PIN and the Reset Code (which allows you to completely wipe -the smartcard). You so rarely need to use the Admin PIN, that you will +the smartcard). You so rarely need to use the Admin PIN, that you will inevitably forget what it is if you do not record it. Getting back to the main card menu, you can also set other values (such @@ -494,6 +501,12 @@ additionally leak information about your smartcard should you lose it. Despite having the name "PIN", neither the user PIN nor the admin PIN on the card need to be numbers. +.. warning:: + + Some devices may require that you move the subkeys onto the device + before you can change the passphrase. Please check the documentation + provided by the device manufacturer. + Move the subkeys to your smartcard ---------------------------------- @@ -655,6 +668,20 @@ want to import these changes back into your regular working directory:: $ gpg --export | gpg --homedir ~/.gnupg --import $ unset GNUPGHOME +Using gpg-agent over ssh +~~~~~~~~~~~~~~~~~~~~~~~~ + +You can forward your gpg-agent over ssh if you need to sign tags or +commits on a remote system. Please refer to the instructions provided +on the GnuPG wiki: + +- `Agent Forwarding over SSH`_ + +It works more smoothly if you can modify the sshd server settings on the +remote end. + +.. _`Agent Forwarding over SSH`: https://wiki.gnupg.org/AgentForwarding + Using PGP with Git ================== @@ -692,6 +719,7 @@ should be used (``[fpr]`` is the fingerprint of your key):: tell git to always use it instead of the legacy ``gpg`` from version 1:: $ git config --global gpg.program gpg2 + $ git config --global gpgv.program gpgv2 How to work with signed tags ---------------------------- @@ -731,6 +759,13 @@ If you are verifying someone else's git tag, then you will need to import their PGP key. Please refer to the ":ref:`verify_identities`" section below. +.. note:: + + If you get "``gpg: Can't check signature: unknown pubkey + algorithm``" error, you need to tell git to use gpgv2 for + verification, so it properly processes signatures made by ECC keys. + See instructions at the start of this section. + Configure git to always sign annotated tags ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/Documentation/process/submitting-patches.rst b/Documentation/process/submitting-patches.rst index f7152ed565e5..908bb55be407 100644 --- a/Documentation/process/submitting-patches.rst +++ b/Documentation/process/submitting-patches.rst @@ -761,7 +761,7 @@ requests, especially from new, unknown developers. If in doubt you can use the pull request as the cover letter for a normal posting of the patch series, giving the maintainer the option of using either. -A pull request should have [GIT] or [PULL] in the subject line. The +A pull request should have [GIT PULL] in the subject line. The request itself should include the repository name and the branch of interest on a single line; it should look something like:: diff --git a/Documentation/riscv/pmu.txt b/Documentation/riscv/pmu.txt new file mode 100644 index 000000000000..b29f03a6d82f --- /dev/null +++ b/Documentation/riscv/pmu.txt @@ -0,0 +1,249 @@ +Supporting PMUs on RISC-V platforms +========================================== +Alan Kao , Mar 2018 + +Introduction +------------ + +As of this writing, perf_event-related features mentioned in The RISC-V ISA +Privileged Version 1.10 are as follows: +(please check the manual for more details) + +* [m|s]counteren +* mcycle[h], cycle[h] +* minstret[h], instret[h] +* mhpeventx, mhpcounterx[h] + +With such function set only, porting perf would require a lot of work, due to +the lack of the following general architectural performance monitoring features: + +* Enabling/Disabling counters + Counters are just free-running all the time in our case. +* Interrupt caused by counter overflow + No such feature in the spec. +* Interrupt indicator + It is not possible to have many interrupt ports for all counters, so an + interrupt indicator is required for software to tell which counter has + just overflowed. +* Writing to counters + There will be an SBI to support this since the kernel cannot modify the + counters [1]. Alternatively, some vendor considers to implement + hardware-extension for M-S-U model machines to write counters directly. + +This document aims to provide developers a quick guide on supporting their +PMUs in the kernel. The following sections briefly explain perf' mechanism +and todos. + +You may check previous discussions here [1][2]. Also, it might be helpful +to check the appendix for related kernel structures. + + +1. Initialization +----------------- + +*riscv_pmu* is a global pointer of type *struct riscv_pmu*, which contains +various methods according to perf's internal convention and PMU-specific +parameters. One should declare such instance to represent the PMU. By default, +*riscv_pmu* points to a constant structure *riscv_base_pmu*, which has very +basic support to a baseline QEMU model. + +Then he/she can either assign the instance's pointer to *riscv_pmu* so that +the minimal and already-implemented logic can be leveraged, or invent his/her +own *riscv_init_platform_pmu* implementation. + +In other words, existing sources of *riscv_base_pmu* merely provide a +reference implementation. Developers can flexibly decide how many parts they +can leverage, and in the most extreme case, they can customize every function +according to their needs. + + +2. Event Initialization +----------------------- + +When a user launches a perf command to monitor some events, it is first +interpreted by the userspace perf tool into multiple *perf_event_open* +system calls, and then each of them calls to the body of *event_init* +member function that was assigned in the previous step. In *riscv_base_pmu*'s +case, it is *riscv_event_init*. + +The main purpose of this function is to translate the event provided by user +into bitmap, so that HW-related control registers or counters can directly be +manipulated. The translation is based on the mappings and methods provided in +*riscv_pmu*. + +Note that some features can be done in this stage as well: + +(1) interrupt setting, which is stated in the next section; +(2) privilege level setting (user space only, kernel space only, both); +(3) destructor setting. Normally it is sufficient to apply *riscv_destroy_event*; +(4) tweaks for non-sampling events, which will be utilized by functions such as +*perf_adjust_period*, usually something like the follows: + +if (!is_sampling_event(event)) { + hwc->sample_period = x86_pmu.max_period; + hwc->last_period = hwc->sample_period; + local64_set(&hwc->period_left, hwc->sample_period); +} + +In the case of *riscv_base_pmu*, only (3) is provided for now. + + +3. Interrupt +------------ + +3.1. Interrupt Initialization + +This often occurs at the beginning of the *event_init* method. In common +practice, this should be a code segment like + +int x86_reserve_hardware(void) +{ + int err = 0; + + if (!atomic_inc_not_zero(&pmc_refcount)) { + mutex_lock(&pmc_reserve_mutex); + if (atomic_read(&pmc_refcount) == 0) { + if (!reserve_pmc_hardware()) + err = -EBUSY; + else + reserve_ds_buffers(); + } + if (!err) + atomic_inc(&pmc_refcount); + mutex_unlock(&pmc_reserve_mutex); + } + + return err; +} + +And the magic is in *reserve_pmc_hardware*, which usually does atomic +operations to make implemented IRQ accessible from some global function pointer. +*release_pmc_hardware* serves the opposite purpose, and it is used in event +destructors mentioned in previous section. + +(Note: From the implementations in all the architectures, the *reserve/release* +pair are always IRQ settings, so the *pmc_hardware* seems somehow misleading. +It does NOT deal with the binding between an event and a physical counter, +which will be introduced in the next section.) + +3.2. IRQ Structure + +Basically, a IRQ runs the following pseudo code: + +for each hardware counter that triggered this overflow + + get the event of this counter + + // following two steps are defined as *read()*, + // check the section Reading/Writing Counters for details. + count the delta value since previous interrupt + update the event->count (# event occurs) by adding delta, and + event->hw.period_left by subtracting delta + + if the event overflows + sample data + set the counter appropriately for the next overflow + + if the event overflows again + too frequently, throttle this event + fi + fi + +end for + +However as of this writing, none of the RISC-V implementations have designed an +interrupt for perf, so the details are to be completed in the future. + +4. Reading/Writing Counters +--------------------------- + +They seem symmetric but perf treats them quite differently. For reading, there +is a *read* interface in *struct pmu*, but it serves more than just reading. +According to the context, the *read* function not only reads the content of the +counter (event->count), but also updates the left period to the next interrupt +(event->hw.period_left). + +But the core of perf does not need direct write to counters. Writing counters +is hidden behind the abstraction of 1) *pmu->start*, literally start counting so one +has to set the counter to a good value for the next interrupt; 2) inside the IRQ +it should set the counter to the same resonable value. + +Reading is not a problem in RISC-V but writing would need some effort, since +counters are not allowed to be written by S-mode. + + +5. add()/del()/start()/stop() +----------------------------- + +Basic idea: add()/del() adds/deletes events to/from a PMU, and start()/stop() +starts/stop the counter of some event in the PMU. All of them take the same +arguments: *struct perf_event *event* and *int flag*. + +Consider perf as a state machine, then you will find that these functions serve +as the state transition process between those states. +Three states (event->hw.state) are defined: + +* PERF_HES_STOPPED: the counter is stopped +* PERF_HES_UPTODATE: the event->count is up-to-date +* PERF_HES_ARCH: arch-dependent usage ... we don't need this for now + +A normal flow of these state transitions are as follows: + +* A user launches a perf event, resulting in calling to *event_init*. +* When being context-switched in, *add* is called by the perf core, with a flag + PERF_EF_START, which means that the event should be started after it is added. + At this stage, a general event is bound to a physical counter, if any. + The state changes to PERF_HES_STOPPED and PERF_HES_UPTODATE, because it is now + stopped, and the (software) event count does not need updating. +** *start* is then called, and the counter is enabled. + With flag PERF_EF_RELOAD, it writes an appropriate value to the counter (check + previous section for detail). + Nothing is written if the flag does not contain PERF_EF_RELOAD. + The state now is reset to none, because it is neither stopped nor updated + (the counting already started) +* When being context-switched out, *del* is called. It then checks out all the + events in the PMU and calls *stop* to update their counts. +** *stop* is called by *del* + and the perf core with flag PERF_EF_UPDATE, and it often shares the same + subroutine as *read* with the same logic. + The state changes to PERF_HES_STOPPED and PERF_HES_UPTODATE, again. + +** Life cycle of these two pairs: *add* and *del* are called repeatedly as + tasks switch in-and-out; *start* and *stop* is also called when the perf core + needs a quick stop-and-start, for instance, when the interrupt period is being + adjusted. + +Current implementation is sufficient for now and can be easily extended to +features in the future. + +A. Related Structures +--------------------- + +* struct pmu: include/linux/perf_event.h +* struct riscv_pmu: arch/riscv/include/asm/perf_event.h + + Both structures are designed to be read-only. + + *struct pmu* defines some function pointer interfaces, and most of them take +*struct perf_event* as a main argument, dealing with perf events according to +perf's internal state machine (check kernel/events/core.c for details). + + *struct riscv_pmu* defines PMU-specific parameters. The naming follows the +convention of all other architectures. + +* struct perf_event: include/linux/perf_event.h +* struct hw_perf_event + + The generic structure that represents perf events, and the hardware-related +details. + +* struct riscv_hw_events: arch/riscv/include/asm/perf_event.h + + The structure that holds the status of events, has two fixed members: +the number of events and the array of the events. + +References +---------- + +[1] https://github.com/riscv/riscv-linux/pull/124 +[2] https://groups.google.com/a/groups.riscv.org/forum/#!topic/sw-dev/f19TmCNP6yA diff --git a/Documentation/scheduler/sched-deadline.txt b/Documentation/scheduler/sched-deadline.txt index 8ce78f82ae23..b14e03ff3528 100644 --- a/Documentation/scheduler/sched-deadline.txt +++ b/Documentation/scheduler/sched-deadline.txt @@ -49,7 +49,7 @@ CONTENTS 2.1 Main algorithm ------------------ - SCHED_DEADLINE uses three parameters, named "runtime", "period", and + SCHED_DEADLINE [18] uses three parameters, named "runtime", "period", and "deadline", to schedule tasks. A SCHED_DEADLINE task should receive "runtime" microseconds of execution time every "period" microseconds, and these "runtime" microseconds are available within "deadline" microseconds @@ -117,6 +117,10 @@ CONTENTS scheduling deadline = scheduling deadline + period remaining runtime = remaining runtime + runtime + The SCHED_FLAG_DL_OVERRUN flag in sched_attr's sched_flags field allows a task + to get informed about runtime overruns through the delivery of SIGXCPU + signals. + 2.2 Bandwidth reclaiming ------------------------ @@ -279,6 +283,19 @@ CONTENTS running_bw is incremented. +2.3 Energy-aware scheduling +------------------------ + + When cpufreq's schedutil governor is selected, SCHED_DEADLINE implements the + GRUB-PA [19] algorithm, reducing the CPU operating frequency to the minimum + value that still allows to meet the deadlines. This behavior is currently + implemented only for ARM architectures. + + A particular care must be taken in case the time needed for changing frequency + is of the same order of magnitude of the reservation period. In such cases, + setting a fixed CPU frequency results in a lower amount of deadline misses. + + 3. Scheduling Real-Time Tasks ============================= @@ -505,6 +522,12 @@ CONTENTS 17 - L. Abeni, G. Lipari, A. Parri, Y. Sun, Multicore CPU reclaiming: parallel or sequential?. In Proceedings of the 31st Annual ACM Symposium on Applied Computing, 2016. + 18 - J. Lelli, C. Scordino, L. Abeni, D. Faggioli, Deadline scheduling in the + Linux kernel, Software: Practice and Experience, 46(6): 821-839, June + 2016. + 19 - C. Scordino, L. Abeni, J. Lelli, Energy-Aware Real-Time Scheduling in + the Linux Kernel, 33rd ACM/SIGAPP Symposium On Applied Computing (SAC + 2018), Pau, France, April 2018. 4. Bandwidth management diff --git a/Documentation/scsi/scsi_eh.txt b/Documentation/scsi/scsi_eh.txt index 11e447bdb3a5..1b7436932a2b 100644 --- a/Documentation/scsi/scsi_eh.txt +++ b/Documentation/scsi/scsi_eh.txt @@ -82,24 +82,13 @@ function 1. invokes optional hostt->eh_timed_out() callback. Return value can be one of - - BLK_EH_HANDLED - This indicates that eh_timed_out() dealt with the timeout. - The command is passed back to the block layer and completed - via __blk_complete_requests(). - - *NOTE* After returning BLK_EH_HANDLED the SCSI layer is - assumed to be finished with the command, and no other - functions from the SCSI layer will be called. So this - should typically only be returned if the eh_timed_out() - handler raced with normal completion. - - BLK_EH_RESET_TIMER This indicates that more time is required to finish the command. Timer is restarted. This action is counted as a retry and only allowed scmd->allowed + 1(!) times. Once the - limit is reached, action for BLK_EH_NOT_HANDLED is taken instead. + limit is reached, action for BLK_EH_DONE is taken instead. - - BLK_EH_NOT_HANDLED + - BLK_EH_DONE eh_timed_out() callback did not handle the command. Step #2 is taken. diff --git a/Documentation/security/index.rst b/Documentation/security/index.rst index 298a94a33f05..85492bfca530 100644 --- a/Documentation/security/index.rst +++ b/Documentation/security/index.rst @@ -9,5 +9,7 @@ Security Documentation IMA-templates keys/index LSM + LSM-sctp + SELinux-sctp self-protection tpm/index diff --git a/Documentation/security/self-protection.rst b/Documentation/security/self-protection.rst index 0f53826c78b9..e1ca698e0006 100644 --- a/Documentation/security/self-protection.rst +++ b/Documentation/security/self-protection.rst @@ -156,7 +156,7 @@ The classic stack buffer overflow involves writing past the expected end of a variable stored on the stack, ultimately writing a controlled value to the stack frame's stored return address. The most widely used defense is the presence of a stack canary between the stack variables and the -return address (``CONFIG_CC_STACKPROTECTOR``), which is verified just before +return address (``CONFIG_STACKPROTECTOR``), which is verified just before the function returns. Other defenses include things like shadow stacks. Stack depth overflow diff --git a/Documentation/sound/alsa-configuration.rst b/Documentation/sound/alsa-configuration.rst index aed6b4fb8e46..4d83c1c0ca04 100644 --- a/Documentation/sound/alsa-configuration.rst +++ b/Documentation/sound/alsa-configuration.rst @@ -1062,7 +1062,7 @@ output (with ``--no-upload`` option) to kernel bugzilla or alsa-devel ML (see the section `Links and Addresses`_). ``power_save`` and ``power_save_controller`` options are for power-saving -mode. See powersave.txt for details. +mode. See powersave.rst for details. Note 2: If you get click noises on output, try the module option ``position_fix=1`` or ``2``. ``position_fix=1`` will use the SD_LPIB @@ -1133,7 +1133,7 @@ line_outs_monitor enable_monitor Enable Analog Out on Channel 63/64 by default. -See hdspm.txt for details. +See hdspm.rst for details. Module snd-ice1712 ------------------ @@ -2224,6 +2224,13 @@ quirk_alias Quirk alias list, pass strings like ``0123abcd:5678beef``, which applies the existing quirk for the device 5678:beef to a new device 0123:abcd. +use_vmalloc + Use vmalloc() for allocations of the PCM buffers (default: yes). + For architectures with non-coherent memory like ARM or MIPS, the + mmap access may give inconsistent results with vmalloc'ed + buffers. If mmap is used on such architectures, turn off this + option, so that the DMA-coherent buffers are allocated and used + instead. This module supports multiple devices, autoprobe and hotplugging. diff --git a/Documentation/sound/hd-audio/models.rst b/Documentation/sound/hd-audio/models.rst index 1fee5a4f6660..7c2d37571af0 100644 --- a/Documentation/sound/hd-audio/models.rst +++ b/Documentation/sound/hd-audio/models.rst @@ -263,6 +263,8 @@ hp-dock HP dock support mute-led-gpio Mute LED control via GPIO +hp-mic-fix + Fix for headset mic pin on HP boxes STAC9200 ======== diff --git a/Documentation/sound/soc/codec.rst b/Documentation/sound/soc/codec.rst index f87612b94812..8a9737eb7597 100644 --- a/Documentation/sound/soc/codec.rst +++ b/Documentation/sound/soc/codec.rst @@ -139,7 +139,7 @@ DAPM description ---------------- The Dynamic Audio Power Management description describes the codec power components and their relationships and registers to the ASoC core. -Please read dapm.txt for details of building the description. +Please read dapm.rst for details of building the description. Please also see the examples in other codec drivers. @@ -179,12 +179,12 @@ i.e. static int wm8974_mute(struct snd_soc_dai *dai, int mute) { - struct snd_soc_codec *codec = dai->codec; - u16 mute_reg = snd_soc_read(codec, WM8974_DAC) & 0xffbf; + struct snd_soc_component *component = dai->component; + u16 mute_reg = snd_soc_component_read32(component, WM8974_DAC) & 0xffbf; if (mute) - snd_soc_write(codec, WM8974_DAC, mute_reg | 0x40); + snd_soc_component_write(component, WM8974_DAC, mute_reg | 0x40); else - snd_soc_write(codec, WM8974_DAC, mute_reg); + snd_soc_component_write(component, WM8974_DAC, mute_reg); return 0; } diff --git a/Documentation/sound/soc/platform.rst b/Documentation/sound/soc/platform.rst index d5574904d981..c1badea53d3d 100644 --- a/Documentation/sound/soc/platform.rst +++ b/Documentation/sound/soc/platform.rst @@ -23,30 +23,26 @@ The platform DMA driver optionally supports the following ALSA operations:- }; The platform driver exports its DMA functionality via struct -snd_soc_platform_driver:- +snd_soc_component_driver:- :: - struct snd_soc_platform_driver { - char *name; + struct snd_soc_component_driver { + const char *name; - int (*probe)(struct platform_device *pdev); - int (*remove)(struct platform_device *pdev); - int (*suspend)(struct platform_device *pdev, struct snd_soc_cpu_dai *cpu_dai); - int (*resume)(struct platform_device *pdev, struct snd_soc_cpu_dai *cpu_dai); + ... + int (*probe)(struct snd_soc_component *); + void (*remove)(struct snd_soc_component *); + int (*suspend)(struct snd_soc_component *); + int (*resume)(struct snd_soc_component *); /* pcm creation and destruction */ - int (*pcm_new)(struct snd_card *, struct snd_soc_codec_dai *, struct snd_pcm *); + int (*pcm_new)(struct snd_soc_pcm_runtime *); void (*pcm_free)(struct snd_pcm *); - /* - * For platform caused delay reporting. - * Optional. - */ - snd_pcm_sframes_t (*delay)(struct snd_pcm_substream *, - struct snd_soc_dai *); - - /* platform stream ops */ - struct snd_pcm_ops *pcm_ops; + ... + const struct snd_pcm_ops *ops; + const struct snd_compr_ops *compr_ops; + ... }; Please refer to the ALSA driver documentation for details of audio DMA. @@ -66,7 +62,7 @@ Each SoC DAI driver must provide the following features:- 4. SYSCLK configuration 5. Suspend and resume (optional) -Please see codec.txt for a description of items 1 - 4. +Please see codec.rst for a description of items 1 - 4. SoC DSP Drivers diff --git a/Documentation/sphinx/rstFlatTable.py b/Documentation/sphinx/rstFlatTable.py index 25feb0d35e7a..2019a55f6b18 100755 --- a/Documentation/sphinx/rstFlatTable.py +++ b/Documentation/sphinx/rstFlatTable.py @@ -53,8 +53,6 @@ from docutils.utils import SystemMessagePropagation # common globals # ============================================================================== -# The version numbering follows numbering of the specification -# (Documentation/books/kernel-doc-HOWTO). __version__ = '1.0' PY3 = sys.version_info[0] == 3 diff --git a/Documentation/sysctl/net.txt b/Documentation/sysctl/net.txt index 5992602469d8..9ecde517728c 100644 --- a/Documentation/sysctl/net.txt +++ b/Documentation/sysctl/net.txt @@ -45,6 +45,7 @@ through bpf(2) and passing a verifier in the kernel, a JIT will then translate these BPF proglets into native CPU instructions. There are two flavors of JITs, the newer eBPF JIT currently supported on: - x86_64 + - x86_32 - arm64 - arm32 - ppc64 diff --git a/Documentation/sysctl/vm.txt b/Documentation/sysctl/vm.txt index 17256f2ad919..697ef8c225df 100644 --- a/Documentation/sysctl/vm.txt +++ b/Documentation/sysctl/vm.txt @@ -515,7 +515,7 @@ nr_hugepages Change the minimum size of the hugepage pool. -See Documentation/vm/hugetlbpage.txt +See Documentation/admin-guide/mm/hugetlbpage.rst ============================================================== @@ -524,7 +524,7 @@ nr_overcommit_hugepages Change the maximum size of the hugepage pool. The maximum is nr_hugepages + nr_overcommit_hugepages. -See Documentation/vm/hugetlbpage.txt +See Documentation/admin-guide/mm/hugetlbpage.rst ============================================================== @@ -667,7 +667,7 @@ and don't use much of it. The default value is 0. -See Documentation/vm/overcommit-accounting and +See Documentation/vm/overcommit-accounting.rst and mm/mmap.c::__vm_enough_memory() for more information. ============================================================== diff --git a/Documentation/trace/coresight-cpu-debug.txt b/Documentation/trace/coresight-cpu-debug.txt index 2b9b51cd501e..89ab09e78e8d 100644 --- a/Documentation/trace/coresight-cpu-debug.txt +++ b/Documentation/trace/coresight-cpu-debug.txt @@ -177,11 +177,11 @@ Here is an example of the debugging output format: ARM external debug module: coresight-cpu-debug 850000.debug: CPU[0]: coresight-cpu-debug 850000.debug: EDPRSR: 00000001 (Power:On DLK:Unlock) -coresight-cpu-debug 850000.debug: EDPCSR: [] handle_IPI+0x174/0x1d8 +coresight-cpu-debug 850000.debug: EDPCSR: handle_IPI+0x174/0x1d8 coresight-cpu-debug 850000.debug: EDCIDSR: 00000000 coresight-cpu-debug 850000.debug: EDVIDSR: 90000000 (State:Non-secure Mode:EL1/0 Width:64bits VMID:0) coresight-cpu-debug 852000.debug: CPU[1]: coresight-cpu-debug 852000.debug: EDPRSR: 00000001 (Power:On DLK:Unlock) -coresight-cpu-debug 852000.debug: EDPCSR: [] debug_notifier_call+0x23c/0x358 +coresight-cpu-debug 852000.debug: EDPCSR: debug_notifier_call+0x23c/0x358 coresight-cpu-debug 852000.debug: EDCIDSR: 00000000 coresight-cpu-debug 852000.debug: EDVIDSR: 90000000 (State:Non-secure Mode:EL1/0 Width:64bits VMID:0) diff --git a/Documentation/trace/coresight.txt b/Documentation/trace/coresight.txt index 6f0120c3a4f1..efbc832146e7 100644 --- a/Documentation/trace/coresight.txt +++ b/Documentation/trace/coresight.txt @@ -187,13 +187,19 @@ that can be performed on them (see "struct coresight_ops"). The specific to that component only. "Implementation defined" customisations are expected to be accessed and controlled using those entries. -Last but not least, "struct module *owner" is expected to be set to reflect -the information carried in "THIS_MODULE". How to use the tracer modules ----------------------------- -Before trace collection can start, a coresight sink needs to be identify. +There are two ways to use the Coresight framework: 1) using the perf cmd line +tools and 2) interacting directly with the Coresight devices using the sysFS +interface. Preference is given to the former as using the sysFS interface +requires a deep understanding of the Coresight HW. The following sections +provide details on using both methods. + +1) Using the sysFS interface: + +Before trace collection can start, a coresight sink needs to be identified. There is no limit on the amount of sinks (nor sources) that can be enabled at any given moment. As a generic operation, all device pertaining to the sink class will have an "active" entry in sysfs: @@ -298,42 +304,48 @@ Instruction 13570831 0x8026B584 E28DD00C false ADD Instruction 0 0x8026B588 E8BD8000 true LDM sp!,{pc} Timestamp Timestamp: 17107041535 -How to use the STM module -------------------------- +2) Using perf framework: -Using the System Trace Macrocell module is the same as the tracers - the only -difference is that clients are driving the trace capture rather -than the program flow through the code. +Coresight tracers are represented using the Perf framework's Performance +Monitoring Unit (PMU) abstraction. As such the perf framework takes charge of +controlling when tracing gets enabled based on when the process of interest is +scheduled. When configured in a system, Coresight PMUs will be listed when +queried by the perf command line tool: -As with any other CoreSight component, specifics about the STM tracer can be -found in sysfs with more information on each entry being found in [1]: + linaro@linaro-nano:~$ ./perf list pmu -root@genericarmv8:~# ls /sys/bus/coresight/devices/20100000.stm -enable_source hwevent_select port_enable subsystem uevent -hwevent_enable mgmt port_select traceid -root@genericarmv8:~# + List of pre-defined events (to be used in -e): -Like any other source a sink needs to be identified and the STM enabled before -being used: + cs_etm// [Kernel PMU event] -root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/20010000.etf/enable_sink -root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/20100000.stm/enable_source + linaro@linaro-nano:~$ -From there user space applications can request and use channels using the devfs -interface provided for that purpose by the generic STM API: +Regardless of the number of tracers available in a system (usually equal to the +amount of processor cores), the "cs_etm" PMU will be listed only once. -root@genericarmv8:~# ls -l /dev/20100000.stm -crw------- 1 root root 10, 61 Jan 3 18:11 /dev/20100000.stm -root@genericarmv8:~# +A Coresight PMU works the same way as any other PMU, i.e the name of the PMU is +listed along with configuration options within forward slashes '/'. Since a +Coresight system will typically have more than one sink, the name of the sink to +work with needs to be specified as an event option. Names for sink to choose +from are listed in sysFS under ($SYSFS)/bus/coresight/devices: -Details on how to use the generic STM API can be found here [2]. + root@linaro-nano:~# ls /sys/bus/coresight/devices/ + 20010000.etf 20040000.funnel 20100000.stm 22040000.etm + 22140000.etm 230c0000.funnel 23240000.etm 20030000.tpiu + 20070000.etr 20120000.replicator 220c0000.funnel + 23040000.etm 23140000.etm 23340000.etm -[1]. Documentation/ABI/testing/sysfs-bus-coresight-devices-stm -[2]. Documentation/trace/stm.txt + root@linaro-nano:~# perf record -e cs_etm/@20070000.etr/u --per-thread program +The syntax within the forward slashes '/' is important. The '@' character +tells the parser that a sink is about to be specified and that this is the sink +to use for the trace session. -Using perf tools ----------------- +More information on the above and other example on how to use Coresight with +the perf tools can be found in the "HOWTO.md" file of the openCSD gitHub +repository [3]. + +2.1) AutoFDO analysis using the perf tools: perf can be used to record and analyze trace of programs. @@ -381,3 +393,38 @@ sort example is from the AutoFDO tutorial (https://gcc.gnu.org/wiki/AutoFDO/Tuto $ taskset -c 2 ./sort_autofdo Bubble sorting array of 30000 elements 5806 ms + + +How to use the STM module +------------------------- + +Using the System Trace Macrocell module is the same as the tracers - the only +difference is that clients are driving the trace capture rather +than the program flow through the code. + +As with any other CoreSight component, specifics about the STM tracer can be +found in sysfs with more information on each entry being found in [1]: + +root@genericarmv8:~# ls /sys/bus/coresight/devices/20100000.stm +enable_source hwevent_select port_enable subsystem uevent +hwevent_enable mgmt port_select traceid +root@genericarmv8:~# + +Like any other source a sink needs to be identified and the STM enabled before +being used: + +root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/20010000.etf/enable_sink +root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/20100000.stm/enable_source + +From there user space applications can request and use channels using the devfs +interface provided for that purpose by the generic STM API: + +root@genericarmv8:~# ls -l /dev/20100000.stm +crw------- 1 root root 10, 61 Jan 3 18:11 /dev/20100000.stm +root@genericarmv8:~# + +Details on how to use the generic STM API can be found here [2]. + +[1]. Documentation/ABI/testing/sysfs-bus-coresight-devices-stm +[2]. Documentation/trace/stm.rst +[3]. https://github.com/Linaro/perf-opencsd diff --git a/Documentation/trace/events.rst b/Documentation/trace/events.rst index a5ea2cb0082b..696dc69b8158 100644 --- a/Documentation/trace/events.rst +++ b/Documentation/trace/events.rst @@ -8,7 +8,7 @@ Event Tracing 1. Introduction =============== -Tracepoints (see Documentation/trace/tracepoints.txt) can be used +Tracepoints (see Documentation/trace/tracepoints.rst) can be used without creating custom kernel modules to register probe functions using the event tracing infrastructure. @@ -338,10 +338,14 @@ used for conditionally invoking triggers. The syntax for event triggers is roughly based on the syntax for set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands' -section of Documentation/trace/ftrace.txt), but there are major +section of Documentation/trace/ftrace.rst), but there are major differences and the implementation isn't currently tied to it in any way, so beware about making generalizations between the two. +Note: Writing into trace_marker (See Documentation/trace/ftrace.rst) + can also enable triggers that are written into + /sys/kernel/tracing/events/ftrace/print/trigger + 6.1 Expression syntax --------------------- diff --git a/Documentation/trace/ftrace-uses.rst b/Documentation/trace/ftrace-uses.rst index 998a60a93015..1fbc69894eed 100644 --- a/Documentation/trace/ftrace-uses.rst +++ b/Documentation/trace/ftrace-uses.rst @@ -12,7 +12,7 @@ Written for: 4.14 Introduction ============ -The ftrace infrastructure was originially created to attach callbacks to the +The ftrace infrastructure was originally created to attach callbacks to the beginning of functions in order to record and trace the flow of the kernel. But callbacks to the start of a function can have other use cases. Either for live kernel patching, or for security monitoring. This document describes @@ -30,7 +30,7 @@ The ftrace context This requires extra care to what can be done inside a callback. A callback can be called outside the protective scope of RCU. -The ftrace infrastructure has some protections agains recursions and RCU +The ftrace infrastructure has some protections against recursions and RCU but one must still be very careful how they use the callbacks. @@ -199,7 +199,7 @@ If @buf is NULL and reset is set, all functions will be enabled for tracing. The @buf can also be a glob expression to enable all functions that match a specific pattern. -See Filter Commands in :file:`Documentation/trace/ftrace.txt`. +See Filter Commands in :file:`Documentation/trace/ftrace.rst`. To just trace the schedule function: diff --git a/Documentation/trace/ftrace.rst b/Documentation/trace/ftrace.rst index 67d9c38e95eb..a20d34955333 100644 --- a/Documentation/trace/ftrace.rst +++ b/Documentation/trace/ftrace.rst @@ -224,6 +224,8 @@ of ftrace. Here is a list of some of the key files: has a side effect of enabling or disabling specific functions to be traced. Echoing names of functions into this file will limit the trace to only those functions. + This influences the tracers "function" and "function_graph" + and thus also function profiling (see "function_profile_enabled"). The functions listed in "available_filter_functions" are what can be written into this file. @@ -265,6 +267,8 @@ of ftrace. Here is a list of some of the key files: Functions listed in this file will cause the function graph tracer to only trace these functions and the functions that they call. (See the section "dynamic ftrace" for more details). + Note, set_ftrace_filter and set_ftrace_notrace still affects + what functions are being traced. set_graph_notrace: @@ -277,7 +281,8 @@ of ftrace. Here is a list of some of the key files: This lists the functions that ftrace has processed and can trace. These are the function names that you can pass to - "set_ftrace_filter" or "set_ftrace_notrace". + "set_ftrace_filter", "set_ftrace_notrace", + "set_graph_function", or "set_graph_notrace". (See the section "dynamic ftrace" below for more details.) dyn_ftrace_total_info: @@ -507,6 +512,11 @@ of ftrace. Here is a list of some of the key files: trace_fd = open("trace_marker", WR_ONLY); + Note: Writing into the trace_marker file can also initiate triggers + that are written into /sys/kernel/tracing/events/ftrace/print/trigger + See "Event triggers" in Documentation/trace/events.rst and an + example in Documentation/trace/histogram.rst (Section 3.) + trace_marker_raw: This is similar to trace_marker above, but is meant for for binary data diff --git a/Documentation/trace/histogram.txt b/Documentation/trace/histogram.txt index 6e05510afc28..e73bcf9cb5f3 100644 --- a/Documentation/trace/histogram.txt +++ b/Documentation/trace/histogram.txt @@ -7,7 +7,7 @@ Histogram triggers are special event triggers that can be used to aggregate trace event data into histograms. For information on - trace events and event triggers, see Documentation/trace/events.txt. + trace events and event triggers, see Documentation/trace/events.rst. 2. Histogram Trigger Command @@ -1604,7 +1604,6 @@ Entries: 7 Dropped: 0 - 2.2 Inter-event hist triggers ----------------------------- @@ -1993,3 +1992,547 @@ hist trigger specification. Hits: 12970 Entries: 2 Dropped: 0 + +3. User space creating a trigger +-------------------------------- + +Writing into /sys/kernel/tracing/trace_marker writes into the ftrace +ring buffer. This can also act like an event, by writing into the trigger +file located in /sys/kernel/tracing/events/ftrace/print/ + +Modifying cyclictest to write into the trace_marker file before it sleeps +and after it wakes up, something like this: + +static void traceputs(char *str) +{ + /* tracemark_fd is the trace_marker file descriptor */ + if (tracemark_fd < 0) + return; + /* write the tracemark message */ + write(tracemark_fd, str, strlen(str)); +} + +And later add something like: + + traceputs("start"); + clock_nanosleep(...); + traceputs("end"); + +We can make a histogram from this: + + # cd /sys/kernel/tracing + # echo 'latency u64 lat' > synthetic_events + # echo 'hist:keys=common_pid:ts0=common_timestamp.usecs if buf == "start"' > events/ftrace/print/trigger + # echo 'hist:keys=common_pid:lat=common_timestamp.usecs-$ts0:onmatch(ftrace.print).latency($lat) if buf == "end"' >> events/ftrace/print/trigger + # echo 'hist:keys=lat,common_pid:sort=lat' > events/synthetic/latency/trigger + +The above created a synthetic event called "latency" and two histograms +against the trace_marker, one gets triggered when "start" is written into the +trace_marker file and the other when "end" is written. If the pids match, then +it will call the "latency" synthetic event with the calculated latency as its +parameter. Finally, a histogram is added to the latency synthetic event to +record the calculated latency along with the pid. + +Now running cyclictest with: + + # ./cyclictest -p80 -d0 -i250 -n -a -t --tracemark -b 1000 + + -p80 : run threads at priority 80 + -d0 : have all threads run at the same interval + -i250 : start the interval at 250 microseconds (all threads will do this) + -n : sleep with nanosleep + -a : affine all threads to a separate CPU + -t : one thread per available CPU + --tracemark : enable trace mark writing + -b 1000 : stop if any latency is greater than 1000 microseconds + +Note, the -b 1000 is used just to make --tracemark available. + +Then we can see the histogram created by this with: + + # cat events/synthetic/latency/hist +# event histogram +# +# trigger info: hist:keys=lat,common_pid:vals=hitcount:sort=lat:size=2048 [active] +# + +{ lat: 107, common_pid: 2039 } hitcount: 1 +{ lat: 122, common_pid: 2041 } hitcount: 1 +{ lat: 166, common_pid: 2039 } hitcount: 1 +{ lat: 174, common_pid: 2039 } hitcount: 1 +{ lat: 194, common_pid: 2041 } hitcount: 1 +{ lat: 196, common_pid: 2036 } hitcount: 1 +{ lat: 197, common_pid: 2038 } hitcount: 1 +{ lat: 198, common_pid: 2039 } hitcount: 1 +{ lat: 199, common_pid: 2039 } hitcount: 1 +{ lat: 200, common_pid: 2041 } hitcount: 1 +{ lat: 201, common_pid: 2039 } hitcount: 2 +{ lat: 202, common_pid: 2038 } hitcount: 1 +{ lat: 202, common_pid: 2043 } hitcount: 1 +{ lat: 203, common_pid: 2039 } hitcount: 1 +{ lat: 203, common_pid: 2036 } hitcount: 1 +{ lat: 203, common_pid: 2041 } hitcount: 1 +{ lat: 206, common_pid: 2038 } hitcount: 2 +{ lat: 207, common_pid: 2039 } hitcount: 1 +{ lat: 207, common_pid: 2036 } hitcount: 1 +{ lat: 208, common_pid: 2040 } hitcount: 1 +{ lat: 209, common_pid: 2043 } hitcount: 1 +{ lat: 210, common_pid: 2039 } hitcount: 1 +{ lat: 211, common_pid: 2039 } hitcount: 4 +{ lat: 212, common_pid: 2043 } hitcount: 1 +{ lat: 212, common_pid: 2039 } hitcount: 2 +{ lat: 213, common_pid: 2039 } hitcount: 1 +{ lat: 214, common_pid: 2038 } hitcount: 1 +{ lat: 214, common_pid: 2039 } hitcount: 2 +{ lat: 214, common_pid: 2042 } hitcount: 1 +{ lat: 215, common_pid: 2039 } hitcount: 1 +{ lat: 217, common_pid: 2036 } hitcount: 1 +{ lat: 217, common_pid: 2040 } hitcount: 1 +{ lat: 217, common_pid: 2039 } hitcount: 1 +{ lat: 218, common_pid: 2039 } hitcount: 6 +{ lat: 219, common_pid: 2039 } hitcount: 9 +{ lat: 220, common_pid: 2039 } hitcount: 11 +{ lat: 221, common_pid: 2039 } hitcount: 5 +{ lat: 221, common_pid: 2042 } hitcount: 1 +{ lat: 222, common_pid: 2039 } hitcount: 7 +{ lat: 223, common_pid: 2036 } hitcount: 1 +{ lat: 223, common_pid: 2039 } hitcount: 3 +{ lat: 224, common_pid: 2039 } hitcount: 4 +{ lat: 224, common_pid: 2037 } hitcount: 1 +{ lat: 224, common_pid: 2036 } hitcount: 2 +{ lat: 225, common_pid: 2039 } hitcount: 5 +{ lat: 225, common_pid: 2042 } hitcount: 1 +{ lat: 226, common_pid: 2039 } hitcount: 7 +{ lat: 226, common_pid: 2036 } hitcount: 4 +{ lat: 227, common_pid: 2039 } hitcount: 6 +{ lat: 227, common_pid: 2036 } hitcount: 12 +{ lat: 227, common_pid: 2043 } hitcount: 1 +{ lat: 228, common_pid: 2039 } hitcount: 7 +{ lat: 228, common_pid: 2036 } hitcount: 14 +{ lat: 229, common_pid: 2039 } hitcount: 9 +{ lat: 229, common_pid: 2036 } hitcount: 8 +{ lat: 229, common_pid: 2038 } hitcount: 1 +{ lat: 230, common_pid: 2039 } hitcount: 11 +{ lat: 230, common_pid: 2036 } hitcount: 6 +{ lat: 230, common_pid: 2043 } hitcount: 1 +{ lat: 230, common_pid: 2042 } hitcount: 2 +{ lat: 231, common_pid: 2041 } hitcount: 1 +{ lat: 231, common_pid: 2036 } hitcount: 6 +{ lat: 231, common_pid: 2043 } hitcount: 1 +{ lat: 231, common_pid: 2039 } hitcount: 8 +{ lat: 232, common_pid: 2037 } hitcount: 1 +{ lat: 232, common_pid: 2039 } hitcount: 6 +{ lat: 232, common_pid: 2040 } hitcount: 2 +{ lat: 232, common_pid: 2036 } hitcount: 5 +{ lat: 232, common_pid: 2043 } hitcount: 1 +{ lat: 233, common_pid: 2036 } hitcount: 5 +{ lat: 233, common_pid: 2039 } hitcount: 11 +{ lat: 234, common_pid: 2039 } hitcount: 4 +{ lat: 234, common_pid: 2038 } hitcount: 2 +{ lat: 234, common_pid: 2043 } hitcount: 2 +{ lat: 234, common_pid: 2036 } hitcount: 11 +{ lat: 234, common_pid: 2040 } hitcount: 1 +{ lat: 235, common_pid: 2037 } hitcount: 2 +{ lat: 235, common_pid: 2036 } hitcount: 8 +{ lat: 235, common_pid: 2043 } hitcount: 2 +{ lat: 235, common_pid: 2039 } hitcount: 5 +{ lat: 235, common_pid: 2042 } hitcount: 2 +{ lat: 235, common_pid: 2040 } hitcount: 4 +{ lat: 235, common_pid: 2041 } hitcount: 1 +{ lat: 236, common_pid: 2036 } hitcount: 7 +{ lat: 236, common_pid: 2037 } hitcount: 1 +{ lat: 236, common_pid: 2041 } hitcount: 5 +{ lat: 236, common_pid: 2039 } hitcount: 3 +{ lat: 236, common_pid: 2043 } hitcount: 9 +{ lat: 236, common_pid: 2040 } hitcount: 7 +{ lat: 237, common_pid: 2037 } hitcount: 1 +{ lat: 237, common_pid: 2040 } hitcount: 1 +{ lat: 237, common_pid: 2036 } hitcount: 9 +{ lat: 237, common_pid: 2039 } hitcount: 3 +{ lat: 237, common_pid: 2043 } hitcount: 8 +{ lat: 237, common_pid: 2042 } hitcount: 2 +{ lat: 237, common_pid: 2041 } hitcount: 2 +{ lat: 238, common_pid: 2043 } hitcount: 10 +{ lat: 238, common_pid: 2040 } hitcount: 1 +{ lat: 238, common_pid: 2037 } hitcount: 9 +{ lat: 238, common_pid: 2038 } hitcount: 1 +{ lat: 238, common_pid: 2039 } hitcount: 1 +{ lat: 238, common_pid: 2042 } hitcount: 3 +{ lat: 238, common_pid: 2036 } hitcount: 7 +{ lat: 239, common_pid: 2041 } hitcount: 1 +{ lat: 239, common_pid: 2043 } hitcount: 11 +{ lat: 239, common_pid: 2037 } hitcount: 11 +{ lat: 239, common_pid: 2038 } hitcount: 6 +{ lat: 239, common_pid: 2036 } hitcount: 7 +{ lat: 239, common_pid: 2040 } hitcount: 1 +{ lat: 239, common_pid: 2042 } hitcount: 9 +{ lat: 240, common_pid: 2037 } hitcount: 29 +{ lat: 240, common_pid: 2043 } hitcount: 15 +{ lat: 240, common_pid: 2040 } hitcount: 44 +{ lat: 240, common_pid: 2039 } hitcount: 1 +{ lat: 240, common_pid: 2041 } hitcount: 2 +{ lat: 240, common_pid: 2038 } hitcount: 1 +{ lat: 240, common_pid: 2036 } hitcount: 10 +{ lat: 240, common_pid: 2042 } hitcount: 13 +{ lat: 241, common_pid: 2036 } hitcount: 21 +{ lat: 241, common_pid: 2041 } hitcount: 36 +{ lat: 241, common_pid: 2037 } hitcount: 34 +{ lat: 241, common_pid: 2042 } hitcount: 14 +{ lat: 241, common_pid: 2040 } hitcount: 94 +{ lat: 241, common_pid: 2039 } hitcount: 12 +{ lat: 241, common_pid: 2038 } hitcount: 2 +{ lat: 241, common_pid: 2043 } hitcount: 28 +{ lat: 242, common_pid: 2040 } hitcount: 109 +{ lat: 242, common_pid: 2041 } hitcount: 506 +{ lat: 242, common_pid: 2039 } hitcount: 155 +{ lat: 242, common_pid: 2042 } hitcount: 21 +{ lat: 242, common_pid: 2037 } hitcount: 52 +{ lat: 242, common_pid: 2043 } hitcount: 21 +{ lat: 242, common_pid: 2036 } hitcount: 16 +{ lat: 242, common_pid: 2038 } hitcount: 156 +{ lat: 243, common_pid: 2037 } hitcount: 46 +{ lat: 243, common_pid: 2039 } hitcount: 40 +{ lat: 243, common_pid: 2042 } hitcount: 119 +{ lat: 243, common_pid: 2041 } hitcount: 611 +{ lat: 243, common_pid: 2036 } hitcount: 69 +{ lat: 243, common_pid: 2038 } hitcount: 784 +{ lat: 243, common_pid: 2040 } hitcount: 323 +{ lat: 243, common_pid: 2043 } hitcount: 14 +{ lat: 244, common_pid: 2043 } hitcount: 35 +{ lat: 244, common_pid: 2042 } hitcount: 305 +{ lat: 244, common_pid: 2039 } hitcount: 8 +{ lat: 244, common_pid: 2040 } hitcount: 4515 +{ lat: 244, common_pid: 2038 } hitcount: 371 +{ lat: 244, common_pid: 2037 } hitcount: 31 +{ lat: 244, common_pid: 2036 } hitcount: 114 +{ lat: 244, common_pid: 2041 } hitcount: 3396 +{ lat: 245, common_pid: 2036 } hitcount: 700 +{ lat: 245, common_pid: 2041 } hitcount: 2772 +{ lat: 245, common_pid: 2037 } hitcount: 268 +{ lat: 245, common_pid: 2039 } hitcount: 472 +{ lat: 245, common_pid: 2038 } hitcount: 2758 +{ lat: 245, common_pid: 2042 } hitcount: 3833 +{ lat: 245, common_pid: 2040 } hitcount: 3105 +{ lat: 245, common_pid: 2043 } hitcount: 645 +{ lat: 246, common_pid: 2038 } hitcount: 3451 +{ lat: 246, common_pid: 2041 } hitcount: 142 +{ lat: 246, common_pid: 2037 } hitcount: 5101 +{ lat: 246, common_pid: 2040 } hitcount: 68 +{ lat: 246, common_pid: 2043 } hitcount: 5099 +{ lat: 246, common_pid: 2039 } hitcount: 5608 +{ lat: 246, common_pid: 2042 } hitcount: 3723 +{ lat: 246, common_pid: 2036 } hitcount: 4738 +{ lat: 247, common_pid: 2042 } hitcount: 312 +{ lat: 247, common_pid: 2043 } hitcount: 2385 +{ lat: 247, common_pid: 2041 } hitcount: 452 +{ lat: 247, common_pid: 2038 } hitcount: 792 +{ lat: 247, common_pid: 2040 } hitcount: 78 +{ lat: 247, common_pid: 2036 } hitcount: 2375 +{ lat: 247, common_pid: 2039 } hitcount: 1834 +{ lat: 247, common_pid: 2037 } hitcount: 2655 +{ lat: 248, common_pid: 2037 } hitcount: 36 +{ lat: 248, common_pid: 2042 } hitcount: 11 +{ lat: 248, common_pid: 2038 } hitcount: 122 +{ lat: 248, common_pid: 2036 } hitcount: 135 +{ lat: 248, common_pid: 2039 } hitcount: 26 +{ lat: 248, common_pid: 2041 } hitcount: 503 +{ lat: 248, common_pid: 2043 } hitcount: 66 +{ lat: 248, common_pid: 2040 } hitcount: 46 +{ lat: 249, common_pid: 2037 } hitcount: 29 +{ lat: 249, common_pid: 2038 } hitcount: 1 +{ lat: 249, common_pid: 2043 } hitcount: 29 +{ lat: 249, common_pid: 2039 } hitcount: 8 +{ lat: 249, common_pid: 2042 } hitcount: 56 +{ lat: 249, common_pid: 2040 } hitcount: 27 +{ lat: 249, common_pid: 2041 } hitcount: 11 +{ lat: 249, common_pid: 2036 } hitcount: 27 +{ lat: 250, common_pid: 2038 } hitcount: 1 +{ lat: 250, common_pid: 2036 } hitcount: 30 +{ lat: 250, common_pid: 2040 } hitcount: 19 +{ lat: 250, common_pid: 2043 } hitcount: 22 +{ lat: 250, common_pid: 2042 } hitcount: 20 +{ lat: 250, common_pid: 2041 } hitcount: 1 +{ lat: 250, common_pid: 2039 } hitcount: 6 +{ lat: 250, common_pid: 2037 } hitcount: 48 +{ lat: 251, common_pid: 2037 } hitcount: 43 +{ lat: 251, common_pid: 2039 } hitcount: 1 +{ lat: 251, common_pid: 2036 } hitcount: 12 +{ lat: 251, common_pid: 2042 } hitcount: 2 +{ lat: 251, common_pid: 2041 } hitcount: 1 +{ lat: 251, common_pid: 2043 } hitcount: 15 +{ lat: 251, common_pid: 2040 } hitcount: 3 +{ lat: 252, common_pid: 2040 } hitcount: 1 +{ lat: 252, common_pid: 2036 } hitcount: 12 +{ lat: 252, common_pid: 2037 } hitcount: 21 +{ lat: 252, common_pid: 2043 } hitcount: 14 +{ lat: 253, common_pid: 2037 } hitcount: 21 +{ lat: 253, common_pid: 2039 } hitcount: 2 +{ lat: 253, common_pid: 2036 } hitcount: 9 +{ lat: 253, common_pid: 2043 } hitcount: 6 +{ lat: 253, common_pid: 2040 } hitcount: 1 +{ lat: 254, common_pid: 2036 } hitcount: 8 +{ lat: 254, common_pid: 2043 } hitcount: 3 +{ lat: 254, common_pid: 2041 } hitcount: 1 +{ lat: 254, common_pid: 2042 } hitcount: 1 +{ lat: 254, common_pid: 2039 } hitcount: 1 +{ lat: 254, common_pid: 2037 } hitcount: 12 +{ lat: 255, common_pid: 2043 } hitcount: 1 +{ lat: 255, common_pid: 2037 } hitcount: 2 +{ lat: 255, common_pid: 2036 } hitcount: 2 +{ lat: 255, common_pid: 2039 } hitcount: 8 +{ lat: 256, common_pid: 2043 } hitcount: 1 +{ lat: 256, common_pid: 2036 } hitcount: 4 +{ lat: 256, common_pid: 2039 } hitcount: 6 +{ lat: 257, common_pid: 2039 } hitcount: 5 +{ lat: 257, common_pid: 2036 } hitcount: 4 +{ lat: 258, common_pid: 2039 } hitcount: 5 +{ lat: 258, common_pid: 2036 } hitcount: 2 +{ lat: 259, common_pid: 2036 } hitcount: 7 +{ lat: 259, common_pid: 2039 } hitcount: 7 +{ lat: 260, common_pid: 2036 } hitcount: 8 +{ lat: 260, common_pid: 2039 } hitcount: 6 +{ lat: 261, common_pid: 2036 } hitcount: 5 +{ lat: 261, common_pid: 2039 } hitcount: 7 +{ lat: 262, common_pid: 2039 } hitcount: 5 +{ lat: 262, common_pid: 2036 } hitcount: 5 +{ lat: 263, common_pid: 2039 } hitcount: 7 +{ lat: 263, common_pid: 2036 } hitcount: 7 +{ lat: 264, common_pid: 2039 } hitcount: 9 +{ lat: 264, common_pid: 2036 } hitcount: 9 +{ lat: 265, common_pid: 2036 } hitcount: 5 +{ lat: 265, common_pid: 2039 } hitcount: 1 +{ lat: 266, common_pid: 2036 } hitcount: 1 +{ lat: 266, common_pid: 2039 } hitcount: 3 +{ lat: 267, common_pid: 2036 } hitcount: 1 +{ lat: 267, common_pid: 2039 } hitcount: 3 +{ lat: 268, common_pid: 2036 } hitcount: 1 +{ lat: 268, common_pid: 2039 } hitcount: 6 +{ lat: 269, common_pid: 2036 } hitcount: 1 +{ lat: 269, common_pid: 2043 } hitcount: 1 +{ lat: 269, common_pid: 2039 } hitcount: 2 +{ lat: 270, common_pid: 2040 } hitcount: 1 +{ lat: 270, common_pid: 2039 } hitcount: 6 +{ lat: 271, common_pid: 2041 } hitcount: 1 +{ lat: 271, common_pid: 2039 } hitcount: 5 +{ lat: 272, common_pid: 2039 } hitcount: 10 +{ lat: 273, common_pid: 2039 } hitcount: 8 +{ lat: 274, common_pid: 2039 } hitcount: 2 +{ lat: 275, common_pid: 2039 } hitcount: 1 +{ lat: 276, common_pid: 2039 } hitcount: 2 +{ lat: 276, common_pid: 2037 } hitcount: 1 +{ lat: 276, common_pid: 2038 } hitcount: 1 +{ lat: 277, common_pid: 2039 } hitcount: 1 +{ lat: 277, common_pid: 2042 } hitcount: 1 +{ lat: 278, common_pid: 2039 } hitcount: 1 +{ lat: 279, common_pid: 2039 } hitcount: 4 +{ lat: 279, common_pid: 2043 } hitcount: 1 +{ lat: 280, common_pid: 2039 } hitcount: 3 +{ lat: 283, common_pid: 2036 } hitcount: 2 +{ lat: 284, common_pid: 2039 } hitcount: 1 +{ lat: 284, common_pid: 2043 } hitcount: 1 +{ lat: 288, common_pid: 2039 } hitcount: 1 +{ lat: 289, common_pid: 2039 } hitcount: 1 +{ lat: 300, common_pid: 2039 } hitcount: 1 +{ lat: 384, common_pid: 2039 } hitcount: 1 + +Totals: + Hits: 67625 + Entries: 278 + Dropped: 0 + +Note, the writes are around the sleep, so ideally they will all be of 250 +microseconds. If you are wondering how there are several that are under +250 microseconds, that is because the way cyclictest works, is if one +iteration comes in late, the next one will set the timer to wake up less that +250. That is, if an iteration came in 50 microseconds late, the next wake up +will be at 200 microseconds. + +But this could easily be done in userspace. To make this even more +interesting, we can mix the histogram between events that happened in the +kernel with trace_marker. + + # cd /sys/kernel/tracing + # echo 'latency u64 lat' > synthetic_events + # echo 'hist:keys=pid:ts0=common_timestamp.usecs' > events/sched/sched_waking/trigger + # echo 'hist:keys=common_pid:lat=common_timestamp.usecs-$ts0:onmatch(sched.sched_waking).latency($lat) if buf == "end"' > events/ftrace/print/trigger + # echo 'hist:keys=lat,common_pid:sort=lat' > events/synthetic/latency/trigger + +The difference this time is that instead of using the trace_marker to start +the latency, the sched_waking event is used, matching the common_pid for the +trace_marker write with the pid that is being woken by sched_waking. + +After running cyclictest again with the same parameters, we now have: + + # cat events/synthetic/latency/hist +# event histogram +# +# trigger info: hist:keys=lat,common_pid:vals=hitcount:sort=lat:size=2048 [active] +# + +{ lat: 7, common_pid: 2302 } hitcount: 640 +{ lat: 7, common_pid: 2299 } hitcount: 42 +{ lat: 7, common_pid: 2303 } hitcount: 18 +{ lat: 7, common_pid: 2305 } hitcount: 166 +{ lat: 7, common_pid: 2306 } hitcount: 1 +{ lat: 7, common_pid: 2301 } hitcount: 91 +{ lat: 7, common_pid: 2300 } hitcount: 17 +{ lat: 8, common_pid: 2303 } hitcount: 8296 +{ lat: 8, common_pid: 2304 } hitcount: 6864 +{ lat: 8, common_pid: 2305 } hitcount: 9464 +{ lat: 8, common_pid: 2301 } hitcount: 9213 +{ lat: 8, common_pid: 2306 } hitcount: 6246 +{ lat: 8, common_pid: 2302 } hitcount: 8797 +{ lat: 8, common_pid: 2299 } hitcount: 8771 +{ lat: 8, common_pid: 2300 } hitcount: 8119 +{ lat: 9, common_pid: 2305 } hitcount: 1519 +{ lat: 9, common_pid: 2299 } hitcount: 2346 +{ lat: 9, common_pid: 2303 } hitcount: 2841 +{ lat: 9, common_pid: 2301 } hitcount: 1846 +{ lat: 9, common_pid: 2304 } hitcount: 3861 +{ lat: 9, common_pid: 2302 } hitcount: 1210 +{ lat: 9, common_pid: 2300 } hitcount: 2762 +{ lat: 9, common_pid: 2306 } hitcount: 4247 +{ lat: 10, common_pid: 2299 } hitcount: 16 +{ lat: 10, common_pid: 2306 } hitcount: 333 +{ lat: 10, common_pid: 2303 } hitcount: 16 +{ lat: 10, common_pid: 2304 } hitcount: 168 +{ lat: 10, common_pid: 2302 } hitcount: 240 +{ lat: 10, common_pid: 2301 } hitcount: 28 +{ lat: 10, common_pid: 2300 } hitcount: 95 +{ lat: 10, common_pid: 2305 } hitcount: 18 +{ lat: 11, common_pid: 2303 } hitcount: 5 +{ lat: 11, common_pid: 2305 } hitcount: 8 +{ lat: 11, common_pid: 2306 } hitcount: 221 +{ lat: 11, common_pid: 2302 } hitcount: 76 +{ lat: 11, common_pid: 2304 } hitcount: 26 +{ lat: 11, common_pid: 2300 } hitcount: 125 +{ lat: 11, common_pid: 2299 } hitcount: 2 +{ lat: 12, common_pid: 2305 } hitcount: 3 +{ lat: 12, common_pid: 2300 } hitcount: 6 +{ lat: 12, common_pid: 2306 } hitcount: 90 +{ lat: 12, common_pid: 2302 } hitcount: 4 +{ lat: 12, common_pid: 2303 } hitcount: 1 +{ lat: 12, common_pid: 2304 } hitcount: 122 +{ lat: 13, common_pid: 2300 } hitcount: 12 +{ lat: 13, common_pid: 2301 } hitcount: 1 +{ lat: 13, common_pid: 2306 } hitcount: 32 +{ lat: 13, common_pid: 2302 } hitcount: 5 +{ lat: 13, common_pid: 2305 } hitcount: 1 +{ lat: 13, common_pid: 2303 } hitcount: 1 +{ lat: 13, common_pid: 2304 } hitcount: 61 +{ lat: 14, common_pid: 2303 } hitcount: 4 +{ lat: 14, common_pid: 2306 } hitcount: 5 +{ lat: 14, common_pid: 2305 } hitcount: 4 +{ lat: 14, common_pid: 2304 } hitcount: 62 +{ lat: 14, common_pid: 2302 } hitcount: 19 +{ lat: 14, common_pid: 2300 } hitcount: 33 +{ lat: 14, common_pid: 2299 } hitcount: 1 +{ lat: 14, common_pid: 2301 } hitcount: 4 +{ lat: 15, common_pid: 2305 } hitcount: 1 +{ lat: 15, common_pid: 2302 } hitcount: 25 +{ lat: 15, common_pid: 2300 } hitcount: 11 +{ lat: 15, common_pid: 2299 } hitcount: 5 +{ lat: 15, common_pid: 2301 } hitcount: 1 +{ lat: 15, common_pid: 2304 } hitcount: 8 +{ lat: 15, common_pid: 2303 } hitcount: 1 +{ lat: 15, common_pid: 2306 } hitcount: 6 +{ lat: 16, common_pid: 2302 } hitcount: 31 +{ lat: 16, common_pid: 2306 } hitcount: 3 +{ lat: 16, common_pid: 2300 } hitcount: 5 +{ lat: 17, common_pid: 2302 } hitcount: 6 +{ lat: 17, common_pid: 2303 } hitcount: 1 +{ lat: 18, common_pid: 2304 } hitcount: 1 +{ lat: 18, common_pid: 2302 } hitcount: 8 +{ lat: 18, common_pid: 2299 } hitcount: 1 +{ lat: 18, common_pid: 2301 } hitcount: 1 +{ lat: 19, common_pid: 2303 } hitcount: 4 +{ lat: 19, common_pid: 2304 } hitcount: 5 +{ lat: 19, common_pid: 2302 } hitcount: 4 +{ lat: 19, common_pid: 2299 } hitcount: 3 +{ lat: 19, common_pid: 2306 } hitcount: 1 +{ lat: 19, common_pid: 2300 } hitcount: 4 +{ lat: 19, common_pid: 2305 } hitcount: 5 +{ lat: 20, common_pid: 2299 } hitcount: 2 +{ lat: 20, common_pid: 2302 } hitcount: 3 +{ lat: 20, common_pid: 2305 } hitcount: 1 +{ lat: 20, common_pid: 2300 } hitcount: 2 +{ lat: 20, common_pid: 2301 } hitcount: 2 +{ lat: 20, common_pid: 2303 } hitcount: 3 +{ lat: 21, common_pid: 2305 } hitcount: 1 +{ lat: 21, common_pid: 2299 } hitcount: 5 +{ lat: 21, common_pid: 2303 } hitcount: 4 +{ lat: 21, common_pid: 2302 } hitcount: 7 +{ lat: 21, common_pid: 2300 } hitcount: 1 +{ lat: 21, common_pid: 2301 } hitcount: 5 +{ lat: 21, common_pid: 2304 } hitcount: 2 +{ lat: 22, common_pid: 2302 } hitcount: 5 +{ lat: 22, common_pid: 2303 } hitcount: 1 +{ lat: 22, common_pid: 2306 } hitcount: 3 +{ lat: 22, common_pid: 2301 } hitcount: 2 +{ lat: 22, common_pid: 2300 } hitcount: 1 +{ lat: 22, common_pid: 2299 } hitcount: 1 +{ lat: 22, common_pid: 2305 } hitcount: 1 +{ lat: 22, common_pid: 2304 } hitcount: 1 +{ lat: 23, common_pid: 2299 } hitcount: 1 +{ lat: 23, common_pid: 2306 } hitcount: 2 +{ lat: 23, common_pid: 2302 } hitcount: 6 +{ lat: 24, common_pid: 2302 } hitcount: 3 +{ lat: 24, common_pid: 2300 } hitcount: 1 +{ lat: 24, common_pid: 2306 } hitcount: 2 +{ lat: 24, common_pid: 2305 } hitcount: 1 +{ lat: 24, common_pid: 2299 } hitcount: 1 +{ lat: 25, common_pid: 2300 } hitcount: 1 +{ lat: 25, common_pid: 2302 } hitcount: 4 +{ lat: 26, common_pid: 2302 } hitcount: 2 +{ lat: 27, common_pid: 2305 } hitcount: 1 +{ lat: 27, common_pid: 2300 } hitcount: 1 +{ lat: 27, common_pid: 2302 } hitcount: 3 +{ lat: 28, common_pid: 2306 } hitcount: 1 +{ lat: 28, common_pid: 2302 } hitcount: 4 +{ lat: 29, common_pid: 2302 } hitcount: 1 +{ lat: 29, common_pid: 2300 } hitcount: 2 +{ lat: 29, common_pid: 2306 } hitcount: 1 +{ lat: 29, common_pid: 2304 } hitcount: 1 +{ lat: 30, common_pid: 2302 } hitcount: 4 +{ lat: 31, common_pid: 2302 } hitcount: 6 +{ lat: 32, common_pid: 2302 } hitcount: 1 +{ lat: 33, common_pid: 2299 } hitcount: 1 +{ lat: 33, common_pid: 2302 } hitcount: 3 +{ lat: 34, common_pid: 2302 } hitcount: 2 +{ lat: 35, common_pid: 2302 } hitcount: 1 +{ lat: 35, common_pid: 2304 } hitcount: 1 +{ lat: 36, common_pid: 2302 } hitcount: 4 +{ lat: 37, common_pid: 2302 } hitcount: 6 +{ lat: 38, common_pid: 2302 } hitcount: 2 +{ lat: 39, common_pid: 2302 } hitcount: 2 +{ lat: 39, common_pid: 2304 } hitcount: 1 +{ lat: 40, common_pid: 2304 } hitcount: 2 +{ lat: 40, common_pid: 2302 } hitcount: 5 +{ lat: 41, common_pid: 2304 } hitcount: 1 +{ lat: 41, common_pid: 2302 } hitcount: 8 +{ lat: 42, common_pid: 2302 } hitcount: 6 +{ lat: 42, common_pid: 2304 } hitcount: 1 +{ lat: 43, common_pid: 2302 } hitcount: 3 +{ lat: 43, common_pid: 2304 } hitcount: 4 +{ lat: 44, common_pid: 2302 } hitcount: 6 +{ lat: 45, common_pid: 2302 } hitcount: 5 +{ lat: 46, common_pid: 2302 } hitcount: 5 +{ lat: 47, common_pid: 2302 } hitcount: 7 +{ lat: 48, common_pid: 2301 } hitcount: 1 +{ lat: 48, common_pid: 2302 } hitcount: 9 +{ lat: 49, common_pid: 2302 } hitcount: 3 +{ lat: 50, common_pid: 2302 } hitcount: 1 +{ lat: 50, common_pid: 2301 } hitcount: 1 +{ lat: 51, common_pid: 2302 } hitcount: 2 +{ lat: 51, common_pid: 2301 } hitcount: 1 +{ lat: 61, common_pid: 2302 } hitcount: 1 +{ lat: 110, common_pid: 2302 } hitcount: 1 + +Totals: + Hits: 89565 + Entries: 158 + Dropped: 0 + +This doesn't tell us any information about how late cyclictest may have +woken up, but it does show us a nice histogram of how long it took from +the time that cyclictest was woken to the time it made it into user space. diff --git a/Documentation/trace/intel_th.rst b/Documentation/trace/intel_th.rst index 990f13265178..19e2d633f3c7 100644 --- a/Documentation/trace/intel_th.rst +++ b/Documentation/trace/intel_th.rst @@ -38,7 +38,7 @@ description is at Documentation/ABI/testing/sysfs-bus-intel_th-devices-gth. STH registers an stm class device, through which it provides interface to userspace and kernelspace software trace sources. See -Documentation/trace/stm.txt for more information on that. +Documentation/trace/stm.rst for more information on that. MSU can be configured to collect trace data into a system memory buffer, which can later on be read from its device nodes via read() or diff --git a/Documentation/trace/tracepoint-analysis.rst b/Documentation/trace/tracepoint-analysis.rst index a4d3ff2e5efb..716326b9f152 100644 --- a/Documentation/trace/tracepoint-analysis.rst +++ b/Documentation/trace/tracepoint-analysis.rst @@ -6,7 +6,7 @@ Notes on Analysing Behaviour Using Events and Tracepoints 1. Introduction =============== -Tracepoints (see Documentation/trace/tracepoints.txt) can be used without +Tracepoints (see Documentation/trace/tracepoints.rst) can be used without creating custom kernel modules to register probe functions using the event tracing infrastructure. @@ -55,7 +55,7 @@ simple case of:: 3.1 System-Wide Event Enabling ------------------------------ -See Documentation/trace/events.txt for a proper description on how events +See Documentation/trace/events.rst for a proper description on how events can be enabled system-wide. A short example of enabling all events related to page allocation would look something like:: @@ -112,7 +112,7 @@ at that point. 3.4 Local Event Enabling ------------------------ -Documentation/trace/ftrace.txt describes how to enable events on a per-thread +Documentation/trace/ftrace.rst describes how to enable events on a per-thread basis using set_ftrace_pid. 3.5 Local Event Enablement with PCL @@ -137,7 +137,7 @@ basis using PCL such as follows. 4. Event Filtering ================== -Documentation/trace/ftrace.txt covers in-depth how to filter events in +Documentation/trace/ftrace.rst covers in-depth how to filter events in ftrace. Obviously using grep and awk of trace_pipe is an option as well as any script reading trace_pipe. diff --git a/Documentation/translations/ja_JP/howto.rst b/Documentation/translations/ja_JP/howto.rst index 8d7ed0cbbf5f..f3116381c26b 100644 --- a/Documentation/translations/ja_JP/howto.rst +++ b/Documentation/translations/ja_JP/howto.rst @@ -1,5 +1,5 @@ NOTE: -This is a version of Documentation/HOWTO translated into Japanese. +This is a version of Documentation/process/howto.rst translated into Japanese. This document is maintained by Tsugikazu Shibata If you find any difference between this document and the original file or a problem with the translation, please contact the maintainer of this file. @@ -109,7 +109,7 @@ linux-api@vger.kernel.org に送ることを勧めます。 ています。 カーネルに関して初めての人はここからスタートすると良い でしょう。 - :ref:`Documentation/Process/changes.rst ` + :ref:`Documentation/process/changes.rst ` このファイルはカーネルをうまく生成(訳注 build )し、走らせるのに最 小限のレベルで必要な数々のソフトウェアパッケージの一覧を示してい ます。 diff --git a/Documentation/translations/ko_KR/howto.rst b/Documentation/translations/ko_KR/howto.rst index 624654bdcd8a..a8197e072599 100644 --- a/Documentation/translations/ko_KR/howto.rst +++ b/Documentation/translations/ko_KR/howto.rst @@ -160,7 +160,7 @@ mtk.manpages@gmail.com의 메인테이너에게 보낼 것을 권장한다. 독특한 행동에 관하여 흔히 있는 오해들과 혼란들을 해소하고 있기 때문이다. - :ref:`Documentation/process/stable_kernel_rules.rst ` + :ref:`Documentation/process/stable-kernel-rules.rst ` 이 문서는 안정적인 커널 배포가 이루어지는 규칙을 설명하고 있으며 여러분들이 이러한 배포들 중 하나에 변경을 하길 원한다면 무엇을 해야 하는지를 설명한다. diff --git a/Documentation/translations/ko_KR/memory-barriers.txt b/Documentation/translations/ko_KR/memory-barriers.txt index 0a0930ab4156..921739d00f69 100644 --- a/Documentation/translations/ko_KR/memory-barriers.txt +++ b/Documentation/translations/ko_KR/memory-barriers.txt @@ -36,6 +36,9 @@ Documentation/memory-barriers.txt 부분도 있고, 의도하진 않았지만 사람에 의해 쓰였다보니 불완전한 부분도 있습니다. 이 문서는 리눅스에서 제공하는 다양한 메모리 배리어들을 사용하기 위한 안내서입니다만, 뭔가 이상하다 싶으면 (그런게 많을 겁니다) 질문을 부탁드립니다. +일부 이상한 점들은 공식적인 메모리 일관성 모델과 tools/memory-model/ 에 있는 +관련 문서를 참고해서 해결될 수 있을 겁니다. 그러나, 이 메모리 모델조차도 그 +관리자들의 의견의 집합으로 봐야지, 절대 옳은 예언자로 신봉해선 안될 겁니다. 다시 말하지만, 이 문서는 리눅스가 하드웨어에 기대하는 사항에 대한 명세서가 아닙니다. @@ -77,7 +80,7 @@ Documentation/memory-barriers.txt - 메모리 배리어의 종류. - 메모리 배리어에 대해 가정해선 안될 것. - - 데이터 의존성 배리어. + - 데이터 의존성 배리어 (역사적). - 컨트롤 의존성. - SMP 배리어 짝맞추기. - 메모리 배리어 시퀀스의 예. @@ -255,17 +258,20 @@ CPU 에게 기대할 수 있는 최소한의 보장사항 몇가지가 있습니 (*) 어떤 CPU 든, 의존성이 존재하는 메모리 액세스들은 해당 CPU 자신에게 있어서는 순서대로 메모리 시스템에 수행 요청됩니다. 즉, 다음에 대해서: - Q = READ_ONCE(P); smp_read_barrier_depends(); D = READ_ONCE(*Q); + Q = READ_ONCE(P); D = READ_ONCE(*Q); CPU 는 다음과 같은 메모리 오퍼레이션 시퀀스를 수행 요청합니다: Q = LOAD P, D = LOAD *Q - 그리고 그 시퀀스 내에서의 순서는 항상 지켜집니다. 대부분의 시스템에서 - smp_read_barrier_depends() 는 아무일도 안하지만 DEC Alpha 에서는 - 명시적으로 사용되어야 합니다. 보통의 경우에는 smp_read_barrier_depends() - 를 직접 사용하는 대신 rcu_dereference() 같은 것들을 사용해야 함을 - 알아두세요. + 그리고 그 시퀀스 내에서의 순서는 항상 지켜집니다. 하지만, DEC Alpha 에서 + READ_ONCE() 는 메모리 배리어 명령도 내게 되어 있어서, DEC Alpha CPU 는 + 다음과 같은 메모리 오퍼레이션들을 내놓게 됩니다: + + Q = LOAD P, MEMORY_BARRIER, D = LOAD *Q, MEMORY_BARRIER + + DEC Alpha 에서 수행되든 아니든, READ_ONCE() 는 컴파일러로부터의 악영향 + 또한 제거합니다. (*) 특정 CPU 내에서 겹치는 영역의 메모리에 행해지는 로드와 스토어 들은 해당 CPU 안에서는 순서가 바뀌지 않은 것으로 보여집니다. 즉, 다음에 대해서: @@ -421,8 +427,8 @@ CPU 에게 기대할 수 있는 최소한의 보장사항 몇가지가 있습니 데이터 의존성 배리어는 읽기 배리어의 보다 완화된 형태입니다. 두개의 로드 오퍼레이션이 있고 두번째 것이 첫번째 것의 결과에 의존하고 있을 때(예: 두번째 로드가 참조할 주소를 첫번째 로드가 읽는 경우), 두번째 로드가 읽어올 - 데이터는 첫번째 로드에 의해 그 주소가 얻어지기 전에 업데이트 되어 있음을 - 보장하기 위해서 데이터 의존성 배리어가 필요할 수 있습니다. + 데이터는 첫번째 로드에 의해 그 주소가 얻어진 뒤에 업데이트 됨을 보장하기 + 위해서 데이터 의존성 배리어가 필요할 수 있습니다. 데이터 의존성 배리어는 상호 의존적인 로드 오퍼레이션들 사이의 부분적 순서 세우기입니다; 스토어 오퍼레이션들이나 독립적인 로드들, 또는 중복되는 @@ -570,8 +576,14 @@ ACQUIRE 는 해당 오퍼레이션의 로드 부분에만 적용되고 RELEASE Documentation/DMA-API.txt -데이터 의존성 배리어 --------------------- +데이터 의존성 배리어 (역사적) +----------------------------- + +리눅스 커널 v4.15 기준으로, smp_read_barrier_depends() 가 READ_ONCE() 에 +추가되었는데, 이는 이 섹션에 주의를 기울여야 하는 사람들은 DEC Alpha 아키텍쳐 +전용 코드를 만드는 사람들과 READ_ONCE() 자체를 만드는 사람들 뿐임을 의미합니다. +그런 분들을 위해, 그리고 역사에 관심 있는 분들을 위해, 여기 데이터 의존성 +배리어에 대한 이야기를 적습니다. 데이터 의존성 배리어의 사용에 있어 지켜야 하는 사항들은 약간 미묘하고, 데이터 의존성 배리어가 사용되어야 하는 상황도 항상 명백하지는 않습니다. 설명을 위해 @@ -1787,7 +1799,7 @@ CPU 메모리 배리어 범용 mb() smp_mb() 쓰기 wmb() smp_wmb() 읽기 rmb() smp_rmb() - 데이터 의존성 read_barrier_depends() smp_read_barrier_depends() + 데이터 의존성 READ_ONCE() 데이터 의존성 배리어를 제외한 모든 메모리 배리어는 컴파일러 배리어를 @@ -2796,8 +2808,9 @@ CPU 2 는 C/D 를 갖습니다)가 병렬로 연결되어 있는 시스템을 여기에 개입하기 위해선, 데이터 의존성 배리어나 읽기 배리어를 로드 오퍼레이션들 -사이에 넣어야 합니다. 이렇게 함으로써 캐시가 다음 요청을 처리하기 전에 일관성 -큐를 처리하도록 강제하게 됩니다. +사이에 넣어야 합니다 (v4.15 부터는 READ_ONCE() 매크로에 의해 무조건적으로 +그렇게 됩니다). 이렇게 함으로써 캐시가 다음 요청을 처리하기 전에 일관성 큐를 +처리하도록 강제하게 됩니다. CPU 1 CPU 2 COMMENT =============== =============== ======================================= @@ -2826,7 +2839,10 @@ CPU 2 는 C/D 를 갖습니다)가 병렬로 연결되어 있는 시스템을 다른 CPU 들도 분할된 캐시를 가지고 있을 수 있지만, 그런 CPU 들은 평범한 메모리 액세스를 위해서도 이 분할된 캐시들 사이의 조정을 해야만 합니다. Alpha 는 가장 약한 메모리 순서 시맨틱 (semantic) 을 선택함으로써 메모리 배리어가 명시적으로 -사용되지 않았을 때에는 그런 조정이 필요하지 않게 했습니다. +사용되지 않았을 때에는 그런 조정이 필요하지 않게 했으며, 이는 Alpha 가 당시에 +더 높은 CPU 클락 속도를 가질 수 있게 했습니다. 하지만, (다시 말하건대, v4.15 +이후부터는) Alpha 아키텍쳐 전용 코드와 READ_ONCE() 매크로 내부에서를 제외하고는 +smp_read_barrier_depends() 가 사용되지 않아야 함을 알아두시기 바랍니다. 캐시 일관성 VS DMA @@ -2846,7 +2862,7 @@ CPU 의 캐시에서 RAM 으로 쓰여지는 더티 캐시 라인에 의해 덮 문제를 해결하기 위해선, 커널의 적절한 부분에서 각 CPU 의 캐시 안의 문제가 되는 비트들을 무효화 시켜야 합니다. -캐시 관리에 대한 더 많은 정보를 위해선 Documentation/cachetlb.txt 를 +캐시 관리에 대한 더 많은 정보를 위해선 Documentation/core-api/cachetlb.rst 를 참고하세요. @@ -2988,7 +3004,9 @@ Alpha CPU 의 일부 버전은 분할된 데이터 캐시를 가지고 있어서 메모리 일관성 시스템과 함께 두개의 캐시를 동기화 시켜서, 포인터 변경과 새로운 데이터의 발견을 올바른 순서로 일어나게 하기 때문입니다. -리눅스 커널의 메모리 배리어 모델은 Alpha 에 기초해서 정의되었습니다. +리눅스 커널의 메모리 배리어 모델은 Alpha 에 기초해서 정의되었습니다만, v4.15 +부터는 리눅스 커널이 READ_ONCE() 내에 smp_read_barrier_depends() 를 추가해서 +Alpha 의 메모리 모델로의 영향력이 크게 줄어들긴 했습니다. 위의 "캐시 일관성" 서브섹션을 참고하세요. @@ -3023,7 +3041,7 @@ smp_mb() 가 아니라 virt_mb() 를 사용해야 합니다. 동기화에 락을 사용하지 않고 구현하는데에 사용될 수 있습니다. 더 자세한 내용을 위해선 다음을 참고하세요: - Documentation/circular-buffers.txt + Documentation/core-api/circular-buffers.rst ========= diff --git a/Documentation/translations/zh_CN/SubmittingDrivers b/Documentation/translations/zh_CN/SubmittingDrivers index 929385e4b194..15e73562f710 100644 --- a/Documentation/translations/zh_CN/SubmittingDrivers +++ b/Documentation/translations/zh_CN/SubmittingDrivers @@ -107,7 +107,7 @@ Linux 2.6: 程序测试的指导，请参阅 Documentation/power/drivers-testing.txt。有关驱动程序电 源管理问题相对全面的概述，请参阅 - Documentation/power/admin-guide/devices.rst。 + Documentation/driver-api/pm/devices.rst。 管理： 如果一个驱动程序的作者还在进行有效的维护，那么通常除了那 些明显正确且不需要任何检查的补丁以外，其他所有的补丁都会 diff --git a/Documentation/translations/zh_CN/gpio.txt b/Documentation/translations/zh_CN/gpio.txt index 4f8bf30a41dc..4cb1ba8b8fed 100644 --- a/Documentation/translations/zh_CN/gpio.txt +++ b/Documentation/translations/zh_CN/gpio.txt @@ -1,4 +1,4 @@ -Chinese translated version of Documentation/gpio.txt +Chinese translated version of Documentation/gpio If you have any comment or update to the content, please contact the original document maintainer directly. However, if you have a problem @@ -10,7 +10,7 @@ Maintainer: Grant Likely Linus Walleij Chinese maintainer: Fu Wei --------------------------------------------------------------------- -Documentation/gpio.txt 的中文翻译 +Documentation/gpio 的中文翻译 如果想评论或更新本文的内容，请直接联系原文档的维护者。如果你使用英文 交流有困难的话，也可以向中文版维护者求助。如果本翻译更新不及时或者翻 diff --git a/Documentation/translations/zh_CN/io_ordering.txt b/Documentation/translations/zh_CN/io_ordering.txt index e592daf4e014..1f8127bdd415 100644 --- a/Documentation/translations/zh_CN/io_ordering.txt +++ b/Documentation/translations/zh_CN/io_ordering.txt @@ -1,4 +1,4 @@ -Chinese translated version of Documentation/io_orderings.txt +Chinese translated version of Documentation/io_ordering.txt If you have any comment or update to the content, please contact the original document maintainer directly. However, if you have a problem diff --git a/Documentation/translations/zh_CN/magic-number.txt b/Documentation/translations/zh_CN/magic-number.txt index e9db693c0a23..7159cec04090 100644 --- a/Documentation/translations/zh_CN/magic-number.txt +++ b/Documentation/translations/zh_CN/magic-number.txt @@ -1,4 +1,4 @@ -Chinese translated version of Documentation/magic-number.txt +Chinese translated version of Documentation/process/magic-number.rst If you have any comment or update to the content, please post to LKML directly. However, if you have problem communicating in English you can also ask the @@ -7,7 +7,7 @@ translation is outdated or there is problem with translation. Chinese maintainer: Jia Wei Wei --------------------------------------------------------------------- -Documentation/magic-number.txt的中文翻译 +Documentation/process/magic-number.rst的中文翻译 如果想评论或更新本文的内容，请直接发信到LKML。如果你使用英文交流有困难的话，也可 以向中文版维护者求助。如果本翻译更新不及时或者翻译存在问题，请联系中文版维护者。 diff --git a/Documentation/translations/zh_CN/video4linux/omap3isp.txt b/Documentation/translations/zh_CN/video4linux/omap3isp.txt index 67ffbf352ae0..e9f29375aa95 100644 --- a/Documentation/translations/zh_CN/video4linux/omap3isp.txt +++ b/Documentation/translations/zh_CN/video4linux/omap3isp.txt @@ -1,4 +1,4 @@ -Chinese translated version of Documentation/video4linux/omap3isp.txt +Chinese translated version of Documentation/media/v4l-drivers/omap3isp.rst If you have any comment or update to the content, please contact the original document maintainer directly. However, if you have a problem @@ -11,7 +11,7 @@ Maintainer: Laurent Pinchart David Cohen Chinese maintainer: Fu Wei --------------------------------------------------------------------- -Documentation/video4linux/omap3isp.txt 的中文翻译 +Documentation/media/v4l-drivers/omap3isp.rst 的中文翻译 如果想评论或更新本文的内容，请直接联系原文档的维护者。如果你使用英文 交流有困难的话，也可以向中文版维护者求助。如果本翻译更新不及时或者翻 diff --git a/Documentation/translations/zh_CN/video4linux/v4l2-framework.txt b/Documentation/translations/zh_CN/video4linux/v4l2-framework.txt index c77c0f060864..66c7c568bd86 100644 --- a/Documentation/translations/zh_CN/video4linux/v4l2-framework.txt +++ b/Documentation/translations/zh_CN/video4linux/v4l2-framework.txt @@ -1,4 +1,4 @@ -Chinese translated version of Documentation/video4linux/v4l2-framework.txt +Chinese translated version of Documentation/media/media_kapi.rst If you have any comment or update to the content, please contact the original document maintainer directly. However, if you have a problem @@ -9,7 +9,7 @@ or if there is a problem with the translation. Maintainer: Mauro Carvalho Chehab Chinese maintainer: Fu Wei --------------------------------------------------------------------- -Documentation/video4linux/v4l2-framework.txt 的中文翻译 +Documentation/media/media_kapi.rst 的中文翻译 如果想评论或更新本文的内容，请直接联系原文档的维护者。如果你使用英文 交流有困难的话，也可以向中文版维护者求助。如果本翻译更新不及时或者翻 @@ -777,7 +777,7 @@ v4l2 核心 API 提供了一个处理视频缓冲的标准方法(称为“videob 线性 DMA(videobuf-dma-contig)以及大多用于 USB 设备的用 vmalloc 分配的缓冲(videobuf-vmalloc)。 -请参阅 Documentation/video4linux/videobuf，以获得更多关于 videobuf +请参阅 Documentation/media/kapi/v4l2-videobuf.rst，以获得更多关于 videobuf 层的使用信息。 v4l2_fh 结构体 diff --git a/Documentation/userspace-api/seccomp_filter.rst b/Documentation/userspace-api/seccomp_filter.rst index 099c412951d6..82a468bc7560 100644 --- a/Documentation/userspace-api/seccomp_filter.rst +++ b/Documentation/userspace-api/seccomp_filter.rst @@ -207,13 +207,6 @@ directory. Here's a description of each file in that directory: to the file do not need to be in ordered form but reads from the file will be ordered in the same way as the actions_avail sysctl. - It is important to note that the value of ``actions_logged`` does not - prevent certain actions from being logged when the audit subsystem is - configured to audit a task. If the action is not found in - ``actions_logged`` list, the final decision on whether to audit the - action for that task is ultimately left up to the audit subsystem to - decide for all seccomp return values other than ``SECCOMP_RET_ALLOW``. - The ``allow`` string is not accepted in the ``actions_logged`` sysctl as it is not possible to log ``SECCOMP_RET_ALLOW`` actions. Attempting to write ``allow`` to the sysctl will result in an EINVAL being diff --git a/Documentation/vfio-mediated-device.txt b/Documentation/vfio-mediated-device.txt index 1b3950346532..c3f69bcaf96e 100644 --- a/Documentation/vfio-mediated-device.txt +++ b/Documentation/vfio-mediated-device.txt @@ -145,6 +145,11 @@ The functions in the mdev_parent_ops structure are as follows: * create: allocate basic resources in a driver for a mediated device * remove: free resources in a driver when a mediated device is destroyed +(Note that mdev-core provides no implicit serialization of create/remove +callbacks per mdev parent device, per mdev type, or any other categorization. +Vendor drivers are expected to be fully asynchronous in this respect or +provide their own internal resource protection.) + The callbacks in the mdev_parent_ops structure are as follows: * open: open callback of mediated device diff --git a/Documentation/vfio.txt b/Documentation/vfio.txt index ef6a5111eaa1..f1a4d3c3ba0b 100644 --- a/Documentation/vfio.txt +++ b/Documentation/vfio.txt @@ -252,15 +252,14 @@ into VFIO core. When devices are bound and unbound to the driver, the driver should call vfio_add_group_dev() and vfio_del_group_dev() respectively:: - extern int vfio_add_group_dev(struct iommu_group *iommu_group, - struct device *dev, + extern int vfio_add_group_dev(struct device *dev, const struct vfio_device_ops *ops, void *device_data); extern void *vfio_del_group_dev(struct device *dev); vfio_add_group_dev() indicates to the core to begin tracking the -specified iommu_group and register the specified dev as owned by +iommu_group of the specified dev and register the dev as owned by a VFIO bus driver. The driver provides an ops structure for callbacks similar to a file operations structure:: diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt index 758bf403a169..495b7742ab58 100644 --- a/Documentation/virtual/kvm/api.txt +++ b/Documentation/virtual/kvm/api.txt @@ -1269,12 +1269,18 @@ struct kvm_cpuid_entry2 { __u32 padding[3]; }; -This ioctl returns x86 cpuid features which are supported by both the hardware -and kvm. Userspace can use the information returned by this ioctl to -construct cpuid information (for KVM_SET_CPUID2) that is consistent with -hardware, kernel, and userspace capabilities, and with user requirements (for -example, the user may wish to constrain cpuid to emulate older hardware, -or for feature consistency across a cluster). +This ioctl returns x86 cpuid features which are supported by both the +hardware and kvm in its default configuration. Userspace can use the +information returned by this ioctl to construct cpuid information (for +KVM_SET_CPUID2) that is consistent with hardware, kernel, and +userspace capabilities, and with user requirements (for example, the +user may wish to constrain cpuid to emulate older hardware, or for +feature consistency across a cluster). + +Note that certain capabilities, such as KVM_CAP_X86_DISABLE_EXITS, may +expose cpuid features (e.g. MONITOR) which are not supported by kvm in +its default configuration. If userspace enables such capabilities, it +is responsible for modifying the results of this ioctl appropriately. Userspace invokes KVM_GET_SUPPORTED_CPUID by passing a kvm_cpuid2 structure with the 'nent' field indicating the number of entries in the variable-size @@ -4603,3 +4609,12 @@ Architectures: s390 This capability indicates that kvm will implement the interfaces to handle reset, migration and nested KVM for branch prediction blocking. The stfle facility 82 should not be provided to the guest without this capability. + +8.14 KVM_CAP_HYPERV_TLBFLUSH + +Architectures: x86 + +This capability indicates that KVM supports paravirtualized Hyper-V TLB Flush +hypercalls: +HvFlushVirtualAddressSpace, HvFlushVirtualAddressSpaceEx, +HvFlushVirtualAddressList, HvFlushVirtualAddressListEx. diff --git a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt index 9293b45abdb9..2408ab720ef7 100644 --- a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt +++ b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt @@ -27,16 +27,42 @@ Groups: VCPU and all of the redistributor pages are contiguous. Only valid for KVM_DEV_TYPE_ARM_VGIC_V3. This address needs to be 64K aligned. + + KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION (rw, 64-bit) + The attribute data pointed to by kvm_device_attr.addr is a __u64 value: + bits: | 63 .... 52 | 51 .... 16 | 15 - 12 |11 - 0 + values: | count | base | flags | index + - index encodes the unique redistributor region index + - flags: reserved for future use, currently 0 + - base field encodes bits [51:16] of the guest physical base address + of the first redistributor in the region. + - count encodes the number of redistributors in the region. Must be + greater than 0. + There are two 64K pages for each redistributor in the region and + redistributors are laid out contiguously within the region. Regions + are filled with redistributors in the index order. The sum of all + region count fields must be greater than or equal to the number of + VCPUs. Redistributor regions must be registered in the incremental + index order, starting from index 0. + The characteristics of a specific redistributor region can be read + by presetting the index field in the attr data. + Only valid for KVM_DEV_TYPE_ARM_VGIC_V3. + + It is invalid to mix calls with KVM_VGIC_V3_ADDR_TYPE_REDIST and + KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION attributes. + Errors: -E2BIG: Address outside of addressable IPA range - -EINVAL: Incorrectly aligned address + -EINVAL: Incorrectly aligned address, bad redistributor region + count/index, mixed redistributor region attribute usage -EEXIST: Address already configured + -ENOENT: Attempt to read the characteristics of a non existing + redistributor region -ENXIO: The group or attribute is unknown/unsupported for this device or hardware support is missing. -EFAULT: Invalid user pointer for attr->addr. - KVM_DEV_ARM_VGIC_GRP_DIST_REGS KVM_DEV_ARM_VGIC_GRP_REDIST_REGS Attributes: diff --git a/Documentation/virtual/kvm/mmu.txt b/Documentation/virtual/kvm/mmu.txt index f50d45b1e967..e507a9e0421e 100644 --- a/Documentation/virtual/kvm/mmu.txt +++ b/Documentation/virtual/kvm/mmu.txt @@ -49,8 +49,8 @@ The mmu supports first-generation mmu hardware, which allows an atomic switch of the current paging mode and cr3 during guest entry, as well as two-dimensional paging (AMD's NPT and Intel's EPT). The emulated hardware it exposes is the traditional 2/3/4 level x86 mmu, with support for global -pages, pae, pse, pse36, cr0.wp, and 1GB pages. Work is in progress to support -exposing NPT capable hardware on NPT capable hosts. +pages, pae, pse, pse36, cr0.wp, and 1GB pages. Emulated hardware also +able to expose NPT capable hardware on NPT capable hosts. Translation =========== @@ -465,5 +465,5 @@ Further reading =============== - NPT presentation from KVM Forum 2008 - http://www.linux-kvm.org/wiki/images/c/c8/KvmForum2008%24kdf2008_21.pdf + http://www.linux-kvm.org/images/c/c8/KvmForum2008%24kdf2008_21.pdf diff --git a/Documentation/virtual/kvm/nested-vmx.txt b/Documentation/virtual/kvm/nested-vmx.txt index 8ed937de1163..97eb1353e962 100644 --- a/Documentation/virtual/kvm/nested-vmx.txt +++ b/Documentation/virtual/kvm/nested-vmx.txt @@ -31,17 +31,6 @@ L0, the guest hypervisor, which we call L1, and its nested guest, which we call L2. -Known limitations ------------------ - -The current code supports running Linux guests under KVM guests. -Only 64-bit guest hypervisors are supported. - -Additional patches for running Windows under guest KVM, and Linux under -guest VMware server, and support for nested EPT, are currently running in -the lab, and will be sent as follow-on patchsets. - - Running nested VMX ------------------ diff --git a/Documentation/vm/00-INDEX b/Documentation/vm/00-INDEX index 0278f2c85efb..f4a4f3e884cf 100644 --- a/Documentation/vm/00-INDEX +++ b/Documentation/vm/00-INDEX @@ -1,62 +1,50 @@ 00-INDEX - this file. -active_mm.txt +active_mm.rst - An explanation from Linus about tsk->active_mm vs tsk->mm. -balance +balance.rst - various information on memory balancing. -cleancache.txt +cleancache.rst - Intro to cleancache and page-granularity victim cache. -frontswap.txt +frontswap.rst - Outline frontswap, part of the transcendent memory frontend. -highmem.txt +highmem.rst - Outline of highmem and common issues. -hmm.txt +hmm.rst - Documentation of heterogeneous memory management -hugetlbpage.txt - - a brief summary of hugetlbpage support in the Linux kernel. -hugetlbfs_reserv.txt +hugetlbfs_reserv.rst - A brief overview of hugetlbfs reservation design/implementation. -hwpoison.txt +hwpoison.rst - explains what hwpoison is -idle_page_tracking.txt - - description of the idle page tracking feature. -ksm.txt +ksm.rst - how to use the Kernel Samepage Merging feature. -mmu_notifier.txt +mmu_notifier.rst - a note about clearing pte/pmd and mmu notifications -numa +numa.rst - information about NUMA specific code in the Linux vm. -numa_memory_policy.txt - - documentation of concepts and APIs of the 2.6 memory policy support. -overcommit-accounting +overcommit-accounting.rst - description of the Linux kernels overcommit handling modes. -page_frags +page_frags.rst - description of page fragments allocator -page_migration +page_migration.rst - description of page migration in NUMA systems. -pagemap.txt - - pagemap, from the userspace perspective -page_owner.txt +page_owner.rst - tracking about who allocated each page -remap_file_pages.txt +remap_file_pages.rst - a note about remap_file_pages() system call -slub.txt +slub.rst - a short users guide for SLUB. -soft-dirty.txt - - short explanation for soft-dirty PTEs -split_page_table_lock +split_page_table_lock.rst - Separate per-table lock to improve scalability of the old page_table_lock. -swap_numa.txt +swap_numa.rst - automatic binding of swap device to numa node -transhuge.txt +transhuge.rst - Transparent Hugepage Support, alternative way of using hugepages. -unevictable-lru.txt +unevictable-lru.rst - Unevictable LRU infrastructure -userfaultfd.txt - - description of userfaultfd system call z3fold.txt - outline of z3fold allocator for storing compressed pages -zsmalloc.txt +zsmalloc.rst - outline of zsmalloc allocator for storing compressed pages -zswap.txt +zswap.rst - Intro to compressed cache for swap pages diff --git a/Documentation/vm/active_mm.rst b/Documentation/vm/active_mm.rst new file mode 100644 index 000000000000..c84471b180f8 --- /dev/null +++ b/Documentation/vm/active_mm.rst @@ -0,0 +1,91 @@ +.. _active_mm: + +========= +Active MM +========= + +:: + + List: linux-kernel + Subject: Re: active_mm + From: Linus Torvalds + Date: 1999-07-30 21:36:24 + + Cc'd to linux-kernel, because I don't write explanations all that often, + and when I do I feel better about more people reading them. + + On Fri, 30 Jul 1999, David Mosberger wrote: + > + > Is there a brief description someplace on how "mm" vs. "active_mm" in + > the task_struct are supposed to be used? (My apologies if this was + > discussed on the mailing lists---I just returned from vacation and + > wasn't able to follow linux-kernel for a while). + + Basically, the new setup is: + + - we have "real address spaces" and "anonymous address spaces". The + difference is that an anonymous address space doesn't care about the + user-level page tables at all, so when we do a context switch into an + anonymous address space we just leave the previous address space + active. + + The obvious use for a "anonymous address space" is any thread that + doesn't need any user mappings - all kernel threads basically fall into + this category, but even "real" threads can temporarily say that for + some amount of time they are not going to be interested in user space, + and that the scheduler might as well try to avoid wasting time on + switching the VM state around. Currently only the old-style bdflush + sync does that. + + - "tsk->mm" points to the "real address space". For an anonymous process, + tsk->mm will be NULL, for the logical reason that an anonymous process + really doesn't _have_ a real address space at all. + + - however, we obviously need to keep track of which address space we + "stole" for such an anonymous user. For that, we have "tsk->active_mm", + which shows what the currently active address space is. + + The rule is that for a process with a real address space (ie tsk->mm is + non-NULL) the active_mm obviously always has to be the same as the real + one. + + For a anonymous process, tsk->mm == NULL, and tsk->active_mm is the + "borrowed" mm while the anonymous process is running. When the + anonymous process gets scheduled away, the borrowed address space is + returned and cleared. + + To support all that, the "struct mm_struct" now has two counters: a + "mm_users" counter that is how many "real address space users" there are, + and a "mm_count" counter that is the number of "lazy" users (ie anonymous + users) plus one if there are any real users. + + Usually there is at least one real user, but it could be that the real + user exited on another CPU while a lazy user was still active, so you do + actually get cases where you have a address space that is _only_ used by + lazy users. That is often a short-lived state, because once that thread + gets scheduled away in favour of a real thread, the "zombie" mm gets + released because "mm_users" becomes zero. + + Also, a new rule is that _nobody_ ever has "init_mm" as a real MM any + more. "init_mm" should be considered just a "lazy context when no other + context is available", and in fact it is mainly used just at bootup when + no real VM has yet been created. So code that used to check + + if (current->mm == &init_mm) + + should generally just do + + if (!current->mm) + + instead (which makes more sense anyway - the test is basically one of "do + we have a user context", and is generally done by the page fault handler + and things like that). + + Anyway, I put a pre-patch-2.3.13-1 on ftp.kernel.org just a moment ago, + because it slightly changes the interfaces to accommodate the alpha (who + would have thought it, but the alpha actually ends up having one of the + ugliest context switch codes - unlike the other architectures where the MM + and register state is separate, the alpha PALcode joins the two, and you + need to switch both together). + + (From http://marc.info/?l=linux-kernel&m=93337278602211&w=2) diff --git a/Documentation/vm/active_mm.txt b/Documentation/vm/active_mm.txt deleted file mode 100644 index dbf45817405f..000000000000 --- a/Documentation/vm/active_mm.txt +++ /dev/null @@ -1,83 +0,0 @@ -List: linux-kernel -Subject: Re: active_mm -From: Linus Torvalds -Date: 1999-07-30 21:36:24 - -Cc'd to linux-kernel, because I don't write explanations all that often, -and when I do I feel better about more people reading them. - -On Fri, 30 Jul 1999, David Mosberger wrote: -> -> Is there a brief description someplace on how "mm" vs. "active_mm" in -> the task_struct are supposed to be used? (My apologies if this was -> discussed on the mailing lists---I just returned from vacation and -> wasn't able to follow linux-kernel for a while). - -Basically, the new setup is: - - - we have "real address spaces" and "anonymous address spaces". The - difference is that an anonymous address space doesn't care about the - user-level page tables at all, so when we do a context switch into an - anonymous address space we just leave the previous address space - active. - - The obvious use for a "anonymous address space" is any thread that - doesn't need any user mappings - all kernel threads basically fall into - this category, but even "real" threads can temporarily say that for - some amount of time they are not going to be interested in user space, - and that the scheduler might as well try to avoid wasting time on - switching the VM state around. Currently only the old-style bdflush - sync does that. - - - "tsk->mm" points to the "real address space". For an anonymous process, - tsk->mm will be NULL, for the logical reason that an anonymous process - really doesn't _have_ a real address space at all. - - - however, we obviously need to keep track of which address space we - "stole" for such an anonymous user. For that, we have "tsk->active_mm", - which shows what the currently active address space is. - - The rule is that for a process with a real address space (ie tsk->mm is - non-NULL) the active_mm obviously always has to be the same as the real - one. - - For a anonymous process, tsk->mm == NULL, and tsk->active_mm is the - "borrowed" mm while the anonymous process is running. When the - anonymous process gets scheduled away, the borrowed address space is - returned and cleared. - -To support all that, the "struct mm_struct" now has two counters: a -"mm_users" counter that is how many "real address space users" there are, -and a "mm_count" counter that is the number of "lazy" users (ie anonymous -users) plus one if there are any real users. - -Usually there is at least one real user, but it could be that the real -user exited on another CPU while a lazy user was still active, so you do -actually get cases where you have a address space that is _only_ used by -lazy users. That is often a short-lived state, because once that thread -gets scheduled away in favour of a real thread, the "zombie" mm gets -released because "mm_users" becomes zero. - -Also, a new rule is that _nobody_ ever has "init_mm" as a real MM any -more. "init_mm" should be considered just a "lazy context when no other -context is available", and in fact it is mainly used just at bootup when -no real VM has yet been created. So code that used to check - - if (current->mm == &init_mm) - -should generally just do - - if (!current->mm) - -instead (which makes more sense anyway - the test is basically one of "do -we have a user context", and is generally done by the page fault handler -and things like that). - -Anyway, I put a pre-patch-2.3.13-1 on ftp.kernel.org just a moment ago, -because it slightly changes the interfaces to accommodate the alpha (who -would have thought it, but the alpha actually ends up having one of the -ugliest context switch codes - unlike the other architectures where the MM -and register state is separate, the alpha PALcode joins the two, and you -need to switch both together). - -(From http://marc.info/?l=linux-kernel&m=93337278602211&w=2) diff --git a/Documentation/vm/balance b/Documentation/vm/balance deleted file mode 100644 index 964595481af6..000000000000 --- a/Documentation/vm/balance +++ /dev/null @@ -1,95 +0,0 @@ -Started Jan 2000 by Kanoj Sarcar - -Memory balancing is needed for !__GFP_ATOMIC and !__GFP_KSWAPD_RECLAIM as -well as for non __GFP_IO allocations. - -The first reason why a caller may avoid reclaim is that the caller can not -sleep due to holding a spinlock or is in interrupt context. The second may -be that the caller is willing to fail the allocation without incurring the -overhead of page reclaim. This may happen for opportunistic high-order -allocation requests that have order-0 fallback options. In such cases, -the caller may also wish to avoid waking kswapd. - -__GFP_IO allocation requests are made to prevent file system deadlocks. - -In the absence of non sleepable allocation requests, it seems detrimental -to be doing balancing. Page reclamation can be kicked off lazily, that -is, only when needed (aka zone free memory is 0), instead of making it -a proactive process. - -That being said, the kernel should try to fulfill requests for direct -mapped pages from the direct mapped pool, instead of falling back on -the dma pool, so as to keep the dma pool filled for dma requests (atomic -or not). A similar argument applies to highmem and direct mapped pages. -OTOH, if there is a lot of free dma pages, it is preferable to satisfy -regular memory requests by allocating one from the dma pool, instead -of incurring the overhead of regular zone balancing. - -In 2.2, memory balancing/page reclamation would kick off only when the -_total_ number of free pages fell below 1/64 th of total memory. With the -right ratio of dma and regular memory, it is quite possible that balancing -would not be done even when the dma zone was completely empty. 2.2 has -been running production machines of varying memory sizes, and seems to be -doing fine even with the presence of this problem. In 2.3, due to -HIGHMEM, this problem is aggravated. - -In 2.3, zone balancing can be done in one of two ways: depending on the -zone size (and possibly of the size of lower class zones), we can decide -at init time how many free pages we should aim for while balancing any -zone. The good part is, while balancing, we do not need to look at sizes -of lower class zones, the bad part is, we might do too frequent balancing -due to ignoring possibly lower usage in the lower class zones. Also, -with a slight change in the allocation routine, it is possible to reduce -the memclass() macro to be a simple equality. - -Another possible solution is that we balance only when the free memory -of a zone _and_ all its lower class zones falls below 1/64th of the -total memory in the zone and its lower class zones. This fixes the 2.2 -balancing problem, and stays as close to 2.2 behavior as possible. Also, -the balancing algorithm works the same way on the various architectures, -which have different numbers and types of zones. If we wanted to get -fancy, we could assign different weights to free pages in different -zones in the future. - -Note that if the size of the regular zone is huge compared to dma zone, -it becomes less significant to consider the free dma pages while -deciding whether to balance the regular zone. The first solution -becomes more attractive then. - -The appended patch implements the second solution. It also "fixes" two -problems: first, kswapd is woken up as in 2.2 on low memory conditions -for non-sleepable allocations. Second, the HIGHMEM zone is also balanced, -so as to give a fighting chance for replace_with_highmem() to get a -HIGHMEM page, as well as to ensure that HIGHMEM allocations do not -fall back into regular zone. This also makes sure that HIGHMEM pages -are not leaked (for example, in situations where a HIGHMEM page is in -the swapcache but is not being used by anyone) - -kswapd also needs to know about the zones it should balance. kswapd is -primarily needed in a situation where balancing can not be done, -probably because all allocation requests are coming from intr context -and all process contexts are sleeping. For 2.3, kswapd does not really -need to balance the highmem zone, since intr context does not request -highmem pages. kswapd looks at the zone_wake_kswapd field in the zone -structure to decide whether a zone needs balancing. - -Page stealing from process memory and shm is done if stealing the page would -alleviate memory pressure on any zone in the page's node that has fallen below -its watermark. - -watemark[WMARK_MIN/WMARK_LOW/WMARK_HIGH]/low_on_memory/zone_wake_kswapd: These -are per-zone fields, used to determine when a zone needs to be balanced. When -the number of pages falls below watermark[WMARK_MIN], the hysteric field -low_on_memory gets set. This stays set till the number of free pages becomes -watermark[WMARK_HIGH]. When low_on_memory is set, page allocation requests will -try to free some pages in the zone (providing GFP_WAIT is set in the request). -Orthogonal to this, is the decision to poke kswapd to free some zone pages. -That decision is not hysteresis based, and is done when the number of free -pages is below watermark[WMARK_LOW]; in which case zone_wake_kswapd is also set. - - -(Good) Ideas that I have heard: -1. Dynamic experience should influence balancing: number of failed requests -for a zone can be tracked and fed into the balancing scheme (jalvo@mbay.net) -2. Implement a replace_with_highmem()-like replace_with_regular() to preserve -dma pages. (lkd@tantalophile.demon.co.uk) diff --git a/Documentation/vm/balance.rst b/Documentation/vm/balance.rst new file mode 100644 index 000000000000..6a1fadf3e173 --- /dev/null +++ b/Documentation/vm/balance.rst @@ -0,0 +1,102 @@ +.. _balance: + +================ +Memory Balancing +================ + +Started Jan 2000 by Kanoj Sarcar + +Memory balancing is needed for !__GFP_ATOMIC and !__GFP_KSWAPD_RECLAIM as +well as for non __GFP_IO allocations. + +The first reason why a caller may avoid reclaim is that the caller can not +sleep due to holding a spinlock or is in interrupt context. The second may +be that the caller is willing to fail the allocation without incurring the +overhead of page reclaim. This may happen for opportunistic high-order +allocation requests that have order-0 fallback options. In such cases, +the caller may also wish to avoid waking kswapd. + +__GFP_IO allocation requests are made to prevent file system deadlocks. + +In the absence of non sleepable allocation requests, it seems detrimental +to be doing balancing. Page reclamation can be kicked off lazily, that +is, only when needed (aka zone free memory is 0), instead of making it +a proactive process. + +That being said, the kernel should try to fulfill requests for direct +mapped pages from the direct mapped pool, instead of falling back on +the dma pool, so as to keep the dma pool filled for dma requests (atomic +or not). A similar argument applies to highmem and direct mapped pages. +OTOH, if there is a lot of free dma pages, it is preferable to satisfy +regular memory requests by allocating one from the dma pool, instead +of incurring the overhead of regular zone balancing. + +In 2.2, memory balancing/page reclamation would kick off only when the +_total_ number of free pages fell below 1/64 th of total memory. With the +right ratio of dma and regular memory, it is quite possible that balancing +would not be done even when the dma zone was completely empty. 2.2 has +been running production machines of varying memory sizes, and seems to be +doing fine even with the presence of this problem. In 2.3, due to +HIGHMEM, this problem is aggravated. + +In 2.3, zone balancing can be done in one of two ways: depending on the +zone size (and possibly of the size of lower class zones), we can decide +at init time how many free pages we should aim for while balancing any +zone. The good part is, while balancing, we do not need to look at sizes +of lower class zones, the bad part is, we might do too frequent balancing +due to ignoring possibly lower usage in the lower class zones. Also, +with a slight change in the allocation routine, it is possible to reduce +the memclass() macro to be a simple equality. + +Another possible solution is that we balance only when the free memory +of a zone _and_ all its lower class zones falls below 1/64th of the +total memory in the zone and its lower class zones. This fixes the 2.2 +balancing problem, and stays as close to 2.2 behavior as possible. Also, +the balancing algorithm works the same way on the various architectures, +which have different numbers and types of zones. If we wanted to get +fancy, we could assign different weights to free pages in different +zones in the future. + +Note that if the size of the regular zone is huge compared to dma zone, +it becomes less significant to consider the free dma pages while +deciding whether to balance the regular zone. The first solution +becomes more attractive then. + +The appended patch implements the second solution. It also "fixes" two +problems: first, kswapd is woken up as in 2.2 on low memory conditions +for non-sleepable allocations. Second, the HIGHMEM zone is also balanced, +so as to give a fighting chance for replace_with_highmem() to get a +HIGHMEM page, as well as to ensure that HIGHMEM allocations do not +fall back into regular zone. This also makes sure that HIGHMEM pages +are not leaked (for example, in situations where a HIGHMEM page is in +the swapcache but is not being used by anyone) + +kswapd also needs to know about the zones it should balance. kswapd is +primarily needed in a situation where balancing can not be done, +probably because all allocation requests are coming from intr context +and all process contexts are sleeping. For 2.3, kswapd does not really +need to balance the highmem zone, since intr context does not request +highmem pages. kswapd looks at the zone_wake_kswapd field in the zone +structure to decide whether a zone needs balancing. + +Page stealing from process memory and shm is done if stealing the page would +alleviate memory pressure on any zone in the page's node that has fallen below +its watermark. + +watemark[WMARK_MIN/WMARK_LOW/WMARK_HIGH]/low_on_memory/zone_wake_kswapd: These +are per-zone fields, used to determine when a zone needs to be balanced. When +the number of pages falls below watermark[WMARK_MIN], the hysteric field +low_on_memory gets set. This stays set till the number of free pages becomes +watermark[WMARK_HIGH]. When low_on_memory is set, page allocation requests will +try to free some pages in the zone (providing GFP_WAIT is set in the request). +Orthogonal to this, is the decision to poke kswapd to free some zone pages. +That decision is not hysteresis based, and is done when the number of free +pages is below watermark[WMARK_LOW]; in which case zone_wake_kswapd is also set. + + +(Good) Ideas that I have heard: + +1. Dynamic experience should influence balancing: number of failed requests + for a zone can be tracked and fed into the balancing scheme (jalvo@mbay.net) +2. Implement a replace_with_highmem()-like replace_with_regular() to preserve + dma pages. (lkd@tantalophile.demon.co.uk) diff --git a/Documentation/vm/cleancache.rst b/Documentation/vm/cleancache.rst new file mode 100644 index 000000000000..68cba9131c31 --- /dev/null +++ b/Documentation/vm/cleancache.rst @@ -0,0 +1,296 @@ +.. _cleancache: + +========== +Cleancache +========== + +Motivation +========== + +Cleancache is a new optional feature provided by the VFS layer that +potentially dramatically increases page cache effectiveness for +many workloads in many environments at a negligible cost. + +Cleancache can be thought of as a page-granularity victim cache for clean +pages that the kernel's pageframe replacement algorithm (PFRA) would like +to keep around, but can't since there isn't enough memory. So when the +PFRA "evicts" a page, it first attempts to use cleancache code to +put the data contained in that page into "transcendent memory", memory +that is not directly accessible or addressable by the kernel and is +of unknown and possibly time-varying size. + +Later, when a cleancache-enabled filesystem wishes to access a page +in a file on disk, it first checks cleancache to see if it already +contains it; if it does, the page of data is copied into the kernel +and a disk access is avoided. + +Transcendent memory "drivers" for cleancache are currently implemented +in Xen (using hypervisor memory) and zcache (using in-kernel compressed +memory) and other implementations are in development. + +:ref:`FAQs ` are included below. + +Implementation Overview +======================= + +A cleancache "backend" that provides transcendent memory registers itself +to the kernel's cleancache "frontend" by calling cleancache_register_ops, +passing a pointer to a cleancache_ops structure with funcs set appropriately. +The functions provided must conform to certain semantics as follows: + +Most important, cleancache is "ephemeral". Pages which are copied into +cleancache have an indefinite lifetime which is completely unknowable +by the kernel and so may or may not still be in cleancache at any later time. +Thus, as its name implies, cleancache is not suitable for dirty pages. +Cleancache has complete discretion over what pages to preserve and what +pages to discard and when. + +Mounting a cleancache-enabled filesystem should call "init_fs" to obtain a +pool id which, if positive, must be saved in the filesystem's superblock; +a negative return value indicates failure. A "put_page" will copy a +(presumably about-to-be-evicted) page into cleancache and associate it with +the pool id, a file key, and a page index into the file. (The combination +of a pool id, a file key, and an index is sometimes called a "handle".) +A "get_page" will copy the page, if found, from cleancache into kernel memory. +An "invalidate_page" will ensure the page no longer is present in cleancache; +an "invalidate_inode" will invalidate all pages associated with the specified +file; and, when a filesystem is unmounted, an "invalidate_fs" will invalidate +all pages in all files specified by the given pool id and also surrender +the pool id. + +An "init_shared_fs", like init_fs, obtains a pool id but tells cleancache +to treat the pool as shared using a 128-bit UUID as a key. On systems +that may run multiple kernels (such as hard partitioned or virtualized +systems) that may share a clustered filesystem, and where cleancache +may be shared among those kernels, calls to init_shared_fs that specify the +same UUID will receive the same pool id, thus allowing the pages to +be shared. Note that any security requirements must be imposed outside +of the kernel (e.g. by "tools" that control cleancache). Or a +cleancache implementation can simply disable shared_init by always +returning a negative value. + +If a get_page is successful on a non-shared pool, the page is invalidated +(thus making cleancache an "exclusive" cache). On a shared pool, the page +is NOT invalidated on a successful get_page so that it remains accessible to +other sharers. The kernel is responsible for ensuring coherency between +cleancache (shared or not), the page cache, and the filesystem, using +cleancache invalidate operations as required. + +Note that cleancache must enforce put-put-get coherency and get-get +coherency. For the former, if two puts are made to the same handle but +with different data, say AAA by the first put and BBB by the second, a +subsequent get can never return the stale data (AAA). For get-get coherency, +if a get for a given handle fails, subsequent gets for that handle will +never succeed unless preceded by a successful put with that handle. + +Last, cleancache provides no SMP serialization guarantees; if two +different Linux threads are simultaneously putting and invalidating a page +with the same handle, the results are indeterminate. Callers must +lock the page to ensure serial behavior. + +Cleancache Performance Metrics +============================== + +If properly configured, monitoring of cleancache is done via debugfs in +the `/sys/kernel/debug/cleancache` directory. The effectiveness of cleancache +can be measured (across all filesystems) with: + +``succ_gets`` + number of gets that were successful + +``failed_gets`` + number of gets that failed + +``puts`` + number of puts attempted (all "succeed") + +``invalidates`` + number of invalidates attempted + +A backend implementation may provide additional metrics. + +.. _faq: + +FAQ +=== + +* Where's the value? (Andrew Morton) + +Cleancache provides a significant performance benefit to many workloads +in many environments with negligible overhead by improving the +effectiveness of the pagecache. Clean pagecache pages are +saved in transcendent memory (RAM that is otherwise not directly +addressable to the kernel); fetching those pages later avoids "refaults" +and thus disk reads. + +Cleancache (and its sister code "frontswap") provide interfaces for +this transcendent memory (aka "tmem"), which conceptually lies between +fast kernel-directly-addressable RAM and slower DMA/asynchronous devices. +Disallowing direct kernel or userland reads/writes to tmem +is ideal when data is transformed to a different form and size (such +as with compression) or secretly moved (as might be useful for write- +balancing for some RAM-like devices). Evicted page-cache pages (and +swap pages) are a great use for this kind of slower-than-RAM-but-much- +faster-than-disk transcendent memory, and the cleancache (and frontswap) +"page-object-oriented" specification provides a nice way to read and +write -- and indirectly "name" -- the pages. + +In the virtual case, the whole point of virtualization is to statistically +multiplex physical resources across the varying demands of multiple +virtual machines. This is really hard to do with RAM and efforts to +do it well with no kernel change have essentially failed (except in some +well-publicized special-case workloads). Cleancache -- and frontswap -- +with a fairly small impact on the kernel, provide a huge amount +of flexibility for more dynamic, flexible RAM multiplexing. +Specifically, the Xen Transcendent Memory backend allows otherwise +"fallow" hypervisor-owned RAM to not only be "time-shared" between multiple +virtual machines, but the pages can be compressed and deduplicated to +optimize RAM utilization. And when guest OS's are induced to surrender +underutilized RAM (e.g. with "self-ballooning"), page cache pages +are the first to go, and cleancache allows those pages to be +saved and reclaimed if overall host system memory conditions allow. + +And the identical interface used for cleancache can be used in +physical systems as well. The zcache driver acts as a memory-hungry +device that stores pages of data in a compressed state. And +the proposed "RAMster" driver shares RAM across multiple physical +systems. + +* Why does cleancache have its sticky fingers so deep inside the + filesystems and VFS? (Andrew Morton and Christoph Hellwig) + +The core hooks for cleancache in VFS are in most cases a single line +and the minimum set are placed precisely where needed to maintain +coherency (via cleancache_invalidate operations) between cleancache, +the page cache, and disk. All hooks compile into nothingness if +cleancache is config'ed off and turn into a function-pointer- +compare-to-NULL if config'ed on but no backend claims the ops +functions, or to a compare-struct-element-to-negative if a +backend claims the ops functions but a filesystem doesn't enable +cleancache. + +Some filesystems are built entirely on top of VFS and the hooks +in VFS are sufficient, so don't require an "init_fs" hook; the +initial implementation of cleancache didn't provide this hook. +But for some filesystems (such as btrfs), the VFS hooks are +incomplete and one or more hooks in fs-specific code are required. +And for some other filesystems, such as tmpfs, cleancache may +be counterproductive. So it seemed prudent to require a filesystem +to "opt in" to use cleancache, which requires adding a hook in +each filesystem. Not all filesystems are supported by cleancache +only because they haven't been tested. The existing set should +be sufficient to validate the concept, the opt-in approach means +that untested filesystems are not affected, and the hooks in the +existing filesystems should make it very easy to add more +filesystems in the future. + +The total impact of the hooks to existing fs and mm files is only +about 40 lines added (not counting comments and blank lines). + +* Why not make cleancache asynchronous and batched so it can more + easily interface with real devices with DMA instead of copying each + individual page? (Minchan Kim) + +The one-page-at-a-time copy semantics simplifies the implementation +on both the frontend and backend and also allows the backend to +do fancy things on-the-fly like page compression and +page deduplication. And since the data is "gone" (copied into/out +of the pageframe) before the cleancache get/put call returns, +a great deal of race conditions and potential coherency issues +are avoided. While the interface seems odd for a "real device" +or for real kernel-addressable RAM, it makes perfect sense for +transcendent memory. + +* Why is non-shared cleancache "exclusive"? And where is the + page "invalidated" after a "get"? (Minchan Kim) + +The main reason is to free up space in transcendent memory and +to avoid unnecessary cleancache_invalidate calls. If you want inclusive, +the page can be "put" immediately following the "get". If +put-after-get for inclusive becomes common, the interface could +be easily extended to add a "get_no_invalidate" call. + +The invalidate is done by the cleancache backend implementation. + +* What's the performance impact? + +Performance analysis has been presented at OLS'09 and LCA'10. +Briefly, performance gains can be significant on most workloads, +especially when memory pressure is high (e.g. when RAM is +overcommitted in a virtual workload); and because the hooks are +invoked primarily in place of or in addition to a disk read/write, +overhead is negligible even in worst case workloads. Basically +cleancache replaces I/O with memory-copy-CPU-overhead; on older +single-core systems with slow memory-copy speeds, cleancache +has little value, but in newer multicore machines, especially +consolidated/virtualized machines, it has great value. + +* How do I add cleancache support for filesystem X? (Boaz Harrash) + +Filesystems that are well-behaved and conform to certain +restrictions can utilize cleancache simply by making a call to +cleancache_init_fs at mount time. Unusual, misbehaving, or +poorly layered filesystems must either add additional hooks +and/or undergo extensive additional testing... or should just +not enable the optional cleancache. + +Some points for a filesystem to consider: + + - The FS should be block-device-based (e.g. a ram-based FS such + as tmpfs should not enable cleancache) + - To ensure coherency/correctness, the FS must ensure that all + file removal or truncation operations either go through VFS or + add hooks to do the equivalent cleancache "invalidate" operations + - To ensure coherency/correctness, either inode numbers must + be unique across the lifetime of the on-disk file OR the + FS must provide an "encode_fh" function. + - The FS must call the VFS superblock alloc and deactivate routines + or add hooks to do the equivalent cleancache calls done there. + - To maximize performance, all pages fetched from the FS should + go through the do_mpag_readpage routine or the FS should add + hooks to do the equivalent (cf. btrfs) + - Currently, the FS blocksize must be the same as PAGESIZE. This + is not an architectural restriction, but no backends currently + support anything different. + - A clustered FS should invoke the "shared_init_fs" cleancache + hook to get best performance for some backends. + +* Why not use the KVA of the inode as the key? (Christoph Hellwig) + +If cleancache would use the inode virtual address instead of +inode/filehandle, the pool id could be eliminated. But, this +won't work because cleancache retains pagecache data pages +persistently even when the inode has been pruned from the +inode unused list, and only invalidates the data page if the file +gets removed/truncated. So if cleancache used the inode kva, +there would be potential coherency issues if/when the inode +kva is reused for a different file. Alternately, if cleancache +invalidated the pages when the inode kva was freed, much of the value +of cleancache would be lost because the cache of pages in cleanache +is potentially much larger than the kernel pagecache and is most +useful if the pages survive inode cache removal. + +* Why is a global variable required? + +The cleancache_enabled flag is checked in all of the frequently-used +cleancache hooks. The alternative is a function call to check a static +variable. Since cleancache is enabled dynamically at runtime, systems +that don't enable cleancache would suffer thousands (possibly +tens-of-thousands) of unnecessary function calls per second. So the +global variable allows cleancache to be enabled by default at compile +time, but have insignificant performance impact when cleancache remains +disabled at runtime. + +* Does cleanache work with KVM? + +The memory model of KVM is sufficiently different that a cleancache +backend may have less value for KVM. This remains to be tested, +especially in an overcommitted system. + +* Does cleancache work in userspace? It sounds useful for + memory hungry caches like web browsers. (Jamie Lokier) + +No plans yet, though we agree it sounds useful, at least for +apps that bypass the page cache (e.g. O_DIRECT). + +Last updated: Dan Magenheimer, April 13 2011 diff --git a/Documentation/vm/cleancache.txt b/Documentation/vm/cleancache.txt deleted file mode 100644 index e4b49df7a048..000000000000 --- a/Documentation/vm/cleancache.txt +++ /dev/null @@ -1,277 +0,0 @@ -MOTIVATION - -Cleancache is a new optional feature provided by the VFS layer that -potentially dramatically increases page cache effectiveness for -many workloads in many environments at a negligible cost. - -Cleancache can be thought of as a page-granularity victim cache for clean -pages that the kernel's pageframe replacement algorithm (PFRA) would like -to keep around, but can't since there isn't enough memory. So when the -PFRA "evicts" a page, it first attempts to use cleancache code to -put the data contained in that page into "transcendent memory", memory -that is not directly accessible or addressable by the kernel and is -of unknown and possibly time-varying size. - -Later, when a cleancache-enabled filesystem wishes to access a page -in a file on disk, it first checks cleancache to see if it already -contains it; if it does, the page of data is copied into the kernel -and a disk access is avoided. - -Transcendent memory "drivers" for cleancache are currently implemented -in Xen (using hypervisor memory) and zcache (using in-kernel compressed -memory) and other implementations are in development. - -FAQs are included below. - -IMPLEMENTATION OVERVIEW - -A cleancache "backend" that provides transcendent memory registers itself -to the kernel's cleancache "frontend" by calling cleancache_register_ops, -passing a pointer to a cleancache_ops structure with funcs set appropriately. -The functions provided must conform to certain semantics as follows: - -Most important, cleancache is "ephemeral". Pages which are copied into -cleancache have an indefinite lifetime which is completely unknowable -by the kernel and so may or may not still be in cleancache at any later time. -Thus, as its name implies, cleancache is not suitable for dirty pages. -Cleancache has complete discretion over what pages to preserve and what -pages to discard and when. - -Mounting a cleancache-enabled filesystem should call "init_fs" to obtain a -pool id which, if positive, must be saved in the filesystem's superblock; -a negative return value indicates failure. A "put_page" will copy a -(presumably about-to-be-evicted) page into cleancache and associate it with -the pool id, a file key, and a page index into the file. (The combination -of a pool id, a file key, and an index is sometimes called a "handle".) -A "get_page" will copy the page, if found, from cleancache into kernel memory. -An "invalidate_page" will ensure the page no longer is present in cleancache; -an "invalidate_inode" will invalidate all pages associated with the specified -file; and, when a filesystem is unmounted, an "invalidate_fs" will invalidate -all pages in all files specified by the given pool id and also surrender -the pool id. - -An "init_shared_fs", like init_fs, obtains a pool id but tells cleancache -to treat the pool as shared using a 128-bit UUID as a key. On systems -that may run multiple kernels (such as hard partitioned or virtualized -systems) that may share a clustered filesystem, and where cleancache -may be shared among those kernels, calls to init_shared_fs that specify the -same UUID will receive the same pool id, thus allowing the pages to -be shared. Note that any security requirements must be imposed outside -of the kernel (e.g. by "tools" that control cleancache). Or a -cleancache implementation can simply disable shared_init by always -returning a negative value. - -If a get_page is successful on a non-shared pool, the page is invalidated -(thus making cleancache an "exclusive" cache). On a shared pool, the page -is NOT invalidated on a successful get_page so that it remains accessible to -other sharers. The kernel is responsible for ensuring coherency between -cleancache (shared or not), the page cache, and the filesystem, using -cleancache invalidate operations as required. - -Note that cleancache must enforce put-put-get coherency and get-get -coherency. For the former, if two puts are made to the same handle but -with different data, say AAA by the first put and BBB by the second, a -subsequent get can never return the stale data (AAA). For get-get coherency, -if a get for a given handle fails, subsequent gets for that handle will -never succeed unless preceded by a successful put with that handle. - -Last, cleancache provides no SMP serialization guarantees; if two -different Linux threads are simultaneously putting and invalidating a page -with the same handle, the results are indeterminate. Callers must -lock the page to ensure serial behavior. - -CLEANCACHE PERFORMANCE METRICS - -If properly configured, monitoring of cleancache is done via debugfs in -the /sys/kernel/debug/cleancache directory. The effectiveness of cleancache -can be measured (across all filesystems) with: - -succ_gets - number of gets that were successful -failed_gets - number of gets that failed -puts - number of puts attempted (all "succeed") -invalidates - number of invalidates attempted - -A backend implementation may provide additional metrics. - -FAQ - -1) Where's the value? (Andrew Morton) - -Cleancache provides a significant performance benefit to many workloads -in many environments with negligible overhead by improving the -effectiveness of the pagecache. Clean pagecache pages are -saved in transcendent memory (RAM that is otherwise not directly -addressable to the kernel); fetching those pages later avoids "refaults" -and thus disk reads. - -Cleancache (and its sister code "frontswap") provide interfaces for -this transcendent memory (aka "tmem"), which conceptually lies between -fast kernel-directly-addressable RAM and slower DMA/asynchronous devices. -Disallowing direct kernel or userland reads/writes to tmem -is ideal when data is transformed to a different form and size (such -as with compression) or secretly moved (as might be useful for write- -balancing for some RAM-like devices). Evicted page-cache pages (and -swap pages) are a great use for this kind of slower-than-RAM-but-much- -faster-than-disk transcendent memory, and the cleancache (and frontswap) -"page-object-oriented" specification provides a nice way to read and -write -- and indirectly "name" -- the pages. - -In the virtual case, the whole point of virtualization is to statistically -multiplex physical resources across the varying demands of multiple -virtual machines. This is really hard to do with RAM and efforts to -do it well with no kernel change have essentially failed (except in some -well-publicized special-case workloads). Cleancache -- and frontswap -- -with a fairly small impact on the kernel, provide a huge amount -of flexibility for more dynamic, flexible RAM multiplexing. -Specifically, the Xen Transcendent Memory backend allows otherwise -"fallow" hypervisor-owned RAM to not only be "time-shared" between multiple -virtual machines, but the pages can be compressed and deduplicated to -optimize RAM utilization. And when guest OS's are induced to surrender -underutilized RAM (e.g. with "self-ballooning"), page cache pages -are the first to go, and cleancache allows those pages to be -saved and reclaimed if overall host system memory conditions allow. - -And the identical interface used for cleancache can be used in -physical systems as well. The zcache driver acts as a memory-hungry -device that stores pages of data in a compressed state. And -the proposed "RAMster" driver shares RAM across multiple physical -systems. - -2) Why does cleancache have its sticky fingers so deep inside the - filesystems and VFS? (Andrew Morton and Christoph Hellwig) - -The core hooks for cleancache in VFS are in most cases a single line -and the minimum set are placed precisely where needed to maintain -coherency (via cleancache_invalidate operations) between cleancache, -the page cache, and disk. All hooks compile into nothingness if -cleancache is config'ed off and turn into a function-pointer- -compare-to-NULL if config'ed on but no backend claims the ops -functions, or to a compare-struct-element-to-negative if a -backend claims the ops functions but a filesystem doesn't enable -cleancache. - -Some filesystems are built entirely on top of VFS and the hooks -in VFS are sufficient, so don't require an "init_fs" hook; the -initial implementation of cleancache didn't provide this hook. -But for some filesystems (such as btrfs), the VFS hooks are -incomplete and one or more hooks in fs-specific code are required. -And for some other filesystems, such as tmpfs, cleancache may -be counterproductive. So it seemed prudent to require a filesystem -to "opt in" to use cleancache, which requires adding a hook in -each filesystem. Not all filesystems are supported by cleancache -only because they haven't been tested. The existing set should -be sufficient to validate the concept, the opt-in approach means -that untested filesystems are not affected, and the hooks in the -existing filesystems should make it very easy to add more -filesystems in the future. - -The total impact of the hooks to existing fs and mm files is only -about 40 lines added (not counting comments and blank lines). - -3) Why not make cleancache asynchronous and batched so it can - more easily interface with real devices with DMA instead - of copying each individual page? (Minchan Kim) - -The one-page-at-a-time copy semantics simplifies the implementation -on both the frontend and backend and also allows the backend to -do fancy things on-the-fly like page compression and -page deduplication. And since the data is "gone" (copied into/out -of the pageframe) before the cleancache get/put call returns, -a great deal of race conditions and potential coherency issues -are avoided. While the interface seems odd for a "real device" -or for real kernel-addressable RAM, it makes perfect sense for -transcendent memory. - -4) Why is non-shared cleancache "exclusive"? And where is the - page "invalidated" after a "get"? (Minchan Kim) - -The main reason is to free up space in transcendent memory and -to avoid unnecessary cleancache_invalidate calls. If you want inclusive, -the page can be "put" immediately following the "get". If -put-after-get for inclusive becomes common, the interface could -be easily extended to add a "get_no_invalidate" call. - -The invalidate is done by the cleancache backend implementation. - -5) What's the performance impact? - -Performance analysis has been presented at OLS'09 and LCA'10. -Briefly, performance gains can be significant on most workloads, -especially when memory pressure is high (e.g. when RAM is -overcommitted in a virtual workload); and because the hooks are -invoked primarily in place of or in addition to a disk read/write, -overhead is negligible even in worst case workloads. Basically -cleancache replaces I/O with memory-copy-CPU-overhead; on older -single-core systems with slow memory-copy speeds, cleancache -has little value, but in newer multicore machines, especially -consolidated/virtualized machines, it has great value. - -6) How do I add cleancache support for filesystem X? (Boaz Harrash) - -Filesystems that are well-behaved and conform to certain -restrictions can utilize cleancache simply by making a call to -cleancache_init_fs at mount time. Unusual, misbehaving, or -poorly layered filesystems must either add additional hooks -and/or undergo extensive additional testing... or should just -not enable the optional cleancache. - -Some points for a filesystem to consider: - -- The FS should be block-device-based (e.g. a ram-based FS such - as tmpfs should not enable cleancache) -- To ensure coherency/correctness, the FS must ensure that all - file removal or truncation operations either go through VFS or - add hooks to do the equivalent cleancache "invalidate" operations -- To ensure coherency/correctness, either inode numbers must - be unique across the lifetime of the on-disk file OR the - FS must provide an "encode_fh" function. -- The FS must call the VFS superblock alloc and deactivate routines - or add hooks to do the equivalent cleancache calls done there. -- To maximize performance, all pages fetched from the FS should - go through the do_mpag_readpage routine or the FS should add - hooks to do the equivalent (cf. btrfs) -- Currently, the FS blocksize must be the same as PAGESIZE. This - is not an architectural restriction, but no backends currently - support anything different. -- A clustered FS should invoke the "shared_init_fs" cleancache - hook to get best performance for some backends. - -7) Why not use the KVA of the inode as the key? (Christoph Hellwig) - -If cleancache would use the inode virtual address instead of -inode/filehandle, the pool id could be eliminated. But, this -won't work because cleancache retains pagecache data pages -persistently even when the inode has been pruned from the -inode unused list, and only invalidates the data page if the file -gets removed/truncated. So if cleancache used the inode kva, -there would be potential coherency issues if/when the inode -kva is reused for a different file. Alternately, if cleancache -invalidated the pages when the inode kva was freed, much of the value -of cleancache would be lost because the cache of pages in cleanache -is potentially much larger than the kernel pagecache and is most -useful if the pages survive inode cache removal. - -8) Why is a global variable required? - -The cleancache_enabled flag is checked in all of the frequently-used -cleancache hooks. The alternative is a function call to check a static -variable. Since cleancache is enabled dynamically at runtime, systems -that don't enable cleancache would suffer thousands (possibly -tens-of-thousands) of unnecessary function calls per second. So the -global variable allows cleancache to be enabled by default at compile -time, but have insignificant performance impact when cleancache remains -disabled at runtime. - -9) Does cleanache work with KVM? - -The memory model of KVM is sufficiently different that a cleancache -backend may have less value for KVM. This remains to be tested, -especially in an overcommitted system. - -10) Does cleancache work in userspace? It sounds useful for - memory hungry caches like web browsers. (Jamie Lokier) - -No plans yet, though we agree it sounds useful, at least for -apps that bypass the page cache (e.g. O_DIRECT). - -Last updated: Dan Magenheimer, April 13 2011 diff --git a/Documentation/vm/conf.py b/Documentation/vm/conf.py new file mode 100644 index 000000000000..3b0b601af558 --- /dev/null +++ b/Documentation/vm/conf.py @@ -0,0 +1,10 @@ +# -*- coding: utf-8; mode: python -*- + +project = "Linux Memory Management Documentation" + +tags.add("subproject") + +latex_documents = [ + ('index', 'memory-management.tex', project, + 'The kernel development community', 'manual'), +] diff --git a/Documentation/vm/frontswap.rst b/Documentation/vm/frontswap.rst new file mode 100644 index 000000000000..1979f430c1c5 --- /dev/null +++ b/Documentation/vm/frontswap.rst @@ -0,0 +1,293 @@ +.. _frontswap: + +========= +Frontswap +========= + +Frontswap provides a "transcendent memory" interface for swap pages. +In some environments, dramatic performance savings may be obtained because +swapped pages are saved in RAM (or a RAM-like device) instead of a swap disk. + +(Note, frontswap -- and :ref:`cleancache` (merged at 3.0) -- are the "frontends" +and the only necessary changes to the core kernel for transcendent memory; +all other supporting code -- the "backends" -- is implemented as drivers. +See the LWN.net article `Transcendent memory in a nutshell`_ +for a detailed overview of frontswap and related kernel parts) + +.. _Transcendent memory in a nutshell: https://lwn.net/Articles/454795/ + +Frontswap is so named because it can be thought of as the opposite of +a "backing" store for a swap device. The storage is assumed to be +a synchronous concurrency-safe page-oriented "pseudo-RAM device" conforming +to the requirements of transcendent memory (such as Xen's "tmem", or +in-kernel compressed memory, aka "zcache", or future RAM-like devices); +this pseudo-RAM device is not directly accessible or addressable by the +kernel and is of unknown and possibly time-varying size. The driver +links itself to frontswap by calling frontswap_register_ops to set the +frontswap_ops funcs appropriately and the functions it provides must +conform to certain policies as follows: + +An "init" prepares the device to receive frontswap pages associated +with the specified swap device number (aka "type"). A "store" will +copy the page to transcendent memory and associate it with the type and +offset associated with the page. A "load" will copy the page, if found, +from transcendent memory into kernel memory, but will NOT remove the page +from transcendent memory. An "invalidate_page" will remove the page +from transcendent memory and an "invalidate_area" will remove ALL pages +associated with the swap type (e.g., like swapoff) and notify the "device" +to refuse further stores with that swap type. + +Once a page is successfully stored, a matching load on the page will normally +succeed. So when the kernel finds itself in a situation where it needs +to swap out a page, it first attempts to use frontswap. If the store returns +success, the data has been successfully saved to transcendent memory and +a disk write and, if the data is later read back, a disk read are avoided. +If a store returns failure, transcendent memory has rejected the data, and the +page can be written to swap as usual. + +If a backend chooses, frontswap can be configured as a "writethrough +cache" by calling frontswap_writethrough(). In this mode, the reduction +in swap device writes is lost (and also a non-trivial performance advantage) +in order to allow the backend to arbitrarily "reclaim" space used to +store frontswap pages to more completely manage its memory usage. + +Note that if a page is stored and the page already exists in transcendent memory +(a "duplicate" store), either the store succeeds and the data is overwritten, +or the store fails AND the page is invalidated. This ensures stale data may +never be obtained from frontswap. + +If properly configured, monitoring of frontswap is done via debugfs in +the `/sys/kernel/debug/frontswap` directory. The effectiveness of +frontswap can be measured (across all swap devices) with: + +``failed_stores`` + how many store attempts have failed + +``loads`` + how many loads were attempted (all should succeed) + +``succ_stores`` + how many store attempts have succeeded + +``invalidates`` + how many invalidates were attempted + +A backend implementation may provide additional metrics. + +FAQ +=== + +* Where's the value? + +When a workload starts swapping, performance falls through the floor. +Frontswap significantly increases performance in many such workloads by +providing a clean, dynamic interface to read and write swap pages to +"transcendent memory" that is otherwise not directly addressable to the kernel. +This interface is ideal when data is transformed to a different form +and size (such as with compression) or secretly moved (as might be +useful for write-balancing for some RAM-like devices). Swap pages (and +evicted page-cache pages) are a great use for this kind of slower-than-RAM- +but-much-faster-than-disk "pseudo-RAM device" and the frontswap (and +cleancache) interface to transcendent memory provides a nice way to read +and write -- and indirectly "name" -- the pages. + +Frontswap -- and cleancache -- with a fairly small impact on the kernel, +provides a huge amount of flexibility for more dynamic, flexible RAM +utilization in various system configurations: + +In the single kernel case, aka "zcache", pages are compressed and +stored in local memory, thus increasing the total anonymous pages +that can be safely kept in RAM. Zcache essentially trades off CPU +cycles used in compression/decompression for better memory utilization. +Benchmarks have shown little or no impact when memory pressure is +low while providing a significant performance improvement (25%+) +on some workloads under high memory pressure. + +"RAMster" builds on zcache by adding "peer-to-peer" transcendent memory +support for clustered systems. Frontswap pages are locally compressed +as in zcache, but then "remotified" to another system's RAM. This +allows RAM to be dynamically load-balanced back-and-forth as needed, +i.e. when system A is overcommitted, it can swap to system B, and +vice versa. RAMster can also be configured as a memory server so +many servers in a cluster can swap, dynamically as needed, to a single +server configured with a large amount of RAM... without pre-configuring +how much of the RAM is available for each of the clients! + +In the virtual case, the whole point of virtualization is to statistically +multiplex physical resources across the varying demands of multiple +virtual machines. This is really hard to do with RAM and efforts to do +it well with no kernel changes have essentially failed (except in some +well-publicized special-case workloads). +Specifically, the Xen Transcendent Memory backend allows otherwise +"fallow" hypervisor-owned RAM to not only be "time-shared" between multiple +virtual machines, but the pages can be compressed and deduplicated to +optimize RAM utilization. And when guest OS's are induced to surrender +underutilized RAM (e.g. with "selfballooning"), sudden unexpected +memory pressure may result in swapping; frontswap allows those pages +to be swapped to and from hypervisor RAM (if overall host system memory +conditions allow), thus mitigating the potentially awful performance impact +of unplanned swapping. + +A KVM implementation is underway and has been RFC'ed to lkml. And, +using frontswap, investigation is also underway on the use of NVM as +a memory extension technology. + +* Sure there may be performance advantages in some situations, but + what's the space/time overhead of frontswap? + +If CONFIG_FRONTSWAP is disabled, every frontswap hook compiles into +nothingness and the only overhead is a few extra bytes per swapon'ed +swap device. If CONFIG_FRONTSWAP is enabled but no frontswap "backend" +registers, there is one extra global variable compared to zero for +every swap page read or written. If CONFIG_FRONTSWAP is enabled +AND a frontswap backend registers AND the backend fails every "store" +request (i.e. provides no memory despite claiming it might), +CPU overhead is still negligible -- and since every frontswap fail +precedes a swap page write-to-disk, the system is highly likely +to be I/O bound and using a small fraction of a percent of a CPU +will be irrelevant anyway. + +As for space, if CONFIG_FRONTSWAP is enabled AND a frontswap backend +registers, one bit is allocated for every swap page for every swap +device that is swapon'd. This is added to the EIGHT bits (which +was sixteen until about 2.6.34) that the kernel already allocates +for every swap page for every swap device that is swapon'd. (Hugh +Dickins has observed that frontswap could probably steal one of +the existing eight bits, but let's worry about that minor optimization +later.) For very large swap disks (which are rare) on a standard +4K pagesize, this is 1MB per 32GB swap. + +When swap pages are stored in transcendent memory instead of written +out to disk, there is a side effect that this may create more memory +pressure that can potentially outweigh the other advantages. A +backend, such as zcache, must implement policies to carefully (but +dynamically) manage memory limits to ensure this doesn't happen. + +* OK, how about a quick overview of what this frontswap patch does + in terms that a kernel hacker can grok? + +Let's assume that a frontswap "backend" has registered during +kernel initialization; this registration indicates that this +frontswap backend has access to some "memory" that is not directly +accessible by the kernel. Exactly how much memory it provides is +entirely dynamic and random. + +Whenever a swap-device is swapon'd frontswap_init() is called, +passing the swap device number (aka "type") as a parameter. +This notifies frontswap to expect attempts to "store" swap pages +associated with that number. + +Whenever the swap subsystem is readying a page to write to a swap +device (c.f swap_writepage()), frontswap_store is called. Frontswap +consults with the frontswap backend and if the backend says it does NOT +have room, frontswap_store returns -1 and the kernel swaps the page +to the swap device as normal. Note that the response from the frontswap +backend is unpredictable to the kernel; it may choose to never accept a +page, it could accept every ninth page, or it might accept every +page. But if the backend does accept a page, the data from the page +has already been copied and associated with the type and offset, +and the backend guarantees the persistence of the data. In this case, +frontswap sets a bit in the "frontswap_map" for the swap device +corresponding to the page offset on the swap device to which it would +otherwise have written the data. + +When the swap subsystem needs to swap-in a page (swap_readpage()), +it first calls frontswap_load() which checks the frontswap_map to +see if the page was earlier accepted by the frontswap backend. If +it was, the page of data is filled from the frontswap backend and +the swap-in is complete. If not, the normal swap-in code is +executed to obtain the page of data from the real swap device. + +So every time the frontswap backend accepts a page, a swap device read +and (potentially) a swap device write are replaced by a "frontswap backend +store" and (possibly) a "frontswap backend loads", which are presumably much +faster. + +* Can't frontswap be configured as a "special" swap device that is + just higher priority than any real swap device (e.g. like zswap, + or maybe swap-over-nbd/NFS)? + +No. First, the existing swap subsystem doesn't allow for any kind of +swap hierarchy. Perhaps it could be rewritten to accommodate a hierarchy, +but this would require fairly drastic changes. Even if it were +rewritten, the existing swap subsystem uses the block I/O layer which +assumes a swap device is fixed size and any page in it is linearly +addressable. Frontswap barely touches the existing swap subsystem, +and works around the constraints of the block I/O subsystem to provide +a great deal of flexibility and dynamicity. + +For example, the acceptance of any swap page by the frontswap backend is +entirely unpredictable. This is critical to the definition of frontswap +backends because it grants completely dynamic discretion to the +backend. In zcache, one cannot know a priori how compressible a page is. +"Poorly" compressible pages can be rejected, and "poorly" can itself be +defined dynamically depending on current memory constraints. + +Further, frontswap is entirely synchronous whereas a real swap +device is, by definition, asynchronous and uses block I/O. The +block I/O layer is not only unnecessary, but may perform "optimizations" +that are inappropriate for a RAM-oriented device including delaying +the write of some pages for a significant amount of time. Synchrony is +required to ensure the dynamicity of the backend and to avoid thorny race +conditions that would unnecessarily and greatly complicate frontswap +and/or the block I/O subsystem. That said, only the initial "store" +and "load" operations need be synchronous. A separate asynchronous thread +is free to manipulate the pages stored by frontswap. For example, +the "remotification" thread in RAMster uses standard asynchronous +kernel sockets to move compressed frontswap pages to a remote machine. +Similarly, a KVM guest-side implementation could do in-guest compression +and use "batched" hypercalls. + +In a virtualized environment, the dynamicity allows the hypervisor +(or host OS) to do "intelligent overcommit". For example, it can +choose to accept pages only until host-swapping might be imminent, +then force guests to do their own swapping. + +There is a downside to the transcendent memory specifications for +frontswap: Since any "store" might fail, there must always be a real +slot on a real swap device to swap the page. Thus frontswap must be +implemented as a "shadow" to every swapon'd device with the potential +capability of holding every page that the swap device might have held +and the possibility that it might hold no pages at all. This means +that frontswap cannot contain more pages than the total of swapon'd +swap devices. For example, if NO swap device is configured on some +installation, frontswap is useless. Swapless portable devices +can still use frontswap but a backend for such devices must configure +some kind of "ghost" swap device and ensure that it is never used. + +* Why this weird definition about "duplicate stores"? If a page + has been previously successfully stored, can't it always be + successfully overwritten? + +Nearly always it can, but no, sometimes it cannot. Consider an example +where data is compressed and the original 4K page has been compressed +to 1K. Now an attempt is made to overwrite the page with data that +is non-compressible and so would take the entire 4K. But the backend +has no more space. In this case, the store must be rejected. Whenever +frontswap rejects a store that would overwrite, it also must invalidate +the old data and ensure that it is no longer accessible. Since the +swap subsystem then writes the new data to the read swap device, +this is the correct course of action to ensure coherency. + +* What is frontswap_shrink for? + +When the (non-frontswap) swap subsystem swaps out a page to a real +swap device, that page is only taking up low-value pre-allocated disk +space. But if frontswap has placed a page in transcendent memory, that +page may be taking up valuable real estate. The frontswap_shrink +routine allows code outside of the swap subsystem to force pages out +of the memory managed by frontswap and back into kernel-addressable memory. +For example, in RAMster, a "suction driver" thread will attempt +to "repatriate" pages sent to a remote machine back to the local machine; +this is driven using the frontswap_shrink mechanism when memory pressure +subsides. + +* Why does the frontswap patch create the new include file swapfile.h? + +The frontswap code depends on some swap-subsystem-internal data +structures that have, over the years, moved back and forth between +static and global. This seemed a reasonable compromise: Define +them as global but declare them in a new include file that isn't +included by the large number of source files that include swap.h. + +Dan Magenheimer, last updated April 9, 2012 diff --git a/Documentation/vm/frontswap.txt b/Documentation/vm/frontswap.txt deleted file mode 100644 index c71a019be600..000000000000 --- a/Documentation/vm/frontswap.txt +++ /dev/null @@ -1,278 +0,0 @@ -Frontswap provides a "transcendent memory" interface for swap pages. -In some environments, dramatic performance savings may be obtained because -swapped pages are saved in RAM (or a RAM-like device) instead of a swap disk. - -(Note, frontswap -- and cleancache (merged at 3.0) -- are the "frontends" -and the only necessary changes to the core kernel for transcendent memory; -all other supporting code -- the "backends" -- is implemented as drivers. -See the LWN.net article "Transcendent memory in a nutshell" for a detailed -overview of frontswap and related kernel parts: -https://lwn.net/Articles/454795/ ) - -Frontswap is so named because it can be thought of as the opposite of -a "backing" store for a swap device. The storage is assumed to be -a synchronous concurrency-safe page-oriented "pseudo-RAM device" conforming -to the requirements of transcendent memory (such as Xen's "tmem", or -in-kernel compressed memory, aka "zcache", or future RAM-like devices); -this pseudo-RAM device is not directly accessible or addressable by the -kernel and is of unknown and possibly time-varying size. The driver -links itself to frontswap by calling frontswap_register_ops to set the -frontswap_ops funcs appropriately and the functions it provides must -conform to certain policies as follows: - -An "init" prepares the device to receive frontswap pages associated -with the specified swap device number (aka "type"). A "store" will -copy the page to transcendent memory and associate it with the type and -offset associated with the page. A "load" will copy the page, if found, -from transcendent memory into kernel memory, but will NOT remove the page -from transcendent memory. An "invalidate_page" will remove the page -from transcendent memory and an "invalidate_area" will remove ALL pages -associated with the swap type (e.g., like swapoff) and notify the "device" -to refuse further stores with that swap type. - -Once a page is successfully stored, a matching load on the page will normally -succeed. So when the kernel finds itself in a situation where it needs -to swap out a page, it first attempts to use frontswap. If the store returns -success, the data has been successfully saved to transcendent memory and -a disk write and, if the data is later read back, a disk read are avoided. -If a store returns failure, transcendent memory has rejected the data, and the -page can be written to swap as usual. - -If a backend chooses, frontswap can be configured as a "writethrough -cache" by calling frontswap_writethrough(). In this mode, the reduction -in swap device writes is lost (and also a non-trivial performance advantage) -in order to allow the backend to arbitrarily "reclaim" space used to -store frontswap pages to more completely manage its memory usage. - -Note that if a page is stored and the page already exists in transcendent memory -(a "duplicate" store), either the store succeeds and the data is overwritten, -or the store fails AND the page is invalidated. This ensures stale data may -never be obtained from frontswap. - -If properly configured, monitoring of frontswap is done via debugfs in -the /sys/kernel/debug/frontswap directory. The effectiveness of -frontswap can be measured (across all swap devices) with: - -failed_stores - how many store attempts have failed -loads - how many loads were attempted (all should succeed) -succ_stores - how many store attempts have succeeded -invalidates - how many invalidates were attempted - -A backend implementation may provide additional metrics. - -FAQ - -1) Where's the value? - -When a workload starts swapping, performance falls through the floor. -Frontswap significantly increases performance in many such workloads by -providing a clean, dynamic interface to read and write swap pages to -"transcendent memory" that is otherwise not directly addressable to the kernel. -This interface is ideal when data is transformed to a different form -and size (such as with compression) or secretly moved (as might be -useful for write-balancing for some RAM-like devices). Swap pages (and -evicted page-cache pages) are a great use for this kind of slower-than-RAM- -but-much-faster-than-disk "pseudo-RAM device" and the frontswap (and -cleancache) interface to transcendent memory provides a nice way to read -and write -- and indirectly "name" -- the pages. - -Frontswap -- and cleancache -- with a fairly small impact on the kernel, -provides a huge amount of flexibility for more dynamic, flexible RAM -utilization in various system configurations: - -In the single kernel case, aka "zcache", pages are compressed and -stored in local memory, thus increasing the total anonymous pages -that can be safely kept in RAM. Zcache essentially trades off CPU -cycles used in compression/decompression for better memory utilization. -Benchmarks have shown little or no impact when memory pressure is -low while providing a significant performance improvement (25%+) -on some workloads under high memory pressure. - -"RAMster" builds on zcache by adding "peer-to-peer" transcendent memory -support for clustered systems. Frontswap pages are locally compressed -as in zcache, but then "remotified" to another system's RAM. This -allows RAM to be dynamically load-balanced back-and-forth as needed, -i.e. when system A is overcommitted, it can swap to system B, and -vice versa. RAMster can also be configured as a memory server so -many servers in a cluster can swap, dynamically as needed, to a single -server configured with a large amount of RAM... without pre-configuring -how much of the RAM is available for each of the clients! - -In the virtual case, the whole point of virtualization is to statistically -multiplex physical resources across the varying demands of multiple -virtual machines. This is really hard to do with RAM and efforts to do -it well with no kernel changes have essentially failed (except in some -well-publicized special-case workloads). -Specifically, the Xen Transcendent Memory backend allows otherwise -"fallow" hypervisor-owned RAM to not only be "time-shared" between multiple -virtual machines, but the pages can be compressed and deduplicated to -optimize RAM utilization. And when guest OS's are induced to surrender -underutilized RAM (e.g. with "selfballooning"), sudden unexpected -memory pressure may result in swapping; frontswap allows those pages -to be swapped to and from hypervisor RAM (if overall host system memory -conditions allow), thus mitigating the potentially awful performance impact -of unplanned swapping. - -A KVM implementation is underway and has been RFC'ed to lkml. And, -using frontswap, investigation is also underway on the use of NVM as -a memory extension technology. - -2) Sure there may be performance advantages in some situations, but - what's the space/time overhead of frontswap? - -If CONFIG_FRONTSWAP is disabled, every frontswap hook compiles into -nothingness and the only overhead is a few extra bytes per swapon'ed -swap device. If CONFIG_FRONTSWAP is enabled but no frontswap "backend" -registers, there is one extra global variable compared to zero for -every swap page read or written. If CONFIG_FRONTSWAP is enabled -AND a frontswap backend registers AND the backend fails every "store" -request (i.e. provides no memory despite claiming it might), -CPU overhead is still negligible -- and since every frontswap fail -precedes a swap page write-to-disk, the system is highly likely -to be I/O bound and using a small fraction of a percent of a CPU -will be irrelevant anyway. - -As for space, if CONFIG_FRONTSWAP is enabled AND a frontswap backend -registers, one bit is allocated for every swap page for every swap -device that is swapon'd. This is added to the EIGHT bits (which -was sixteen until about 2.6.34) that the kernel already allocates -for every swap page for every swap device that is swapon'd. (Hugh -Dickins has observed that frontswap could probably steal one of -the existing eight bits, but let's worry about that minor optimization -later.) For very large swap disks (which are rare) on a standard -4K pagesize, this is 1MB per 32GB swap. - -When swap pages are stored in transcendent memory instead of written -out to disk, there is a side effect that this may create more memory -pressure that can potentially outweigh the other advantages. A -backend, such as zcache, must implement policies to carefully (but -dynamically) manage memory limits to ensure this doesn't happen. - -3) OK, how about a quick overview of what this frontswap patch does - in terms that a kernel hacker can grok? - -Let's assume that a frontswap "backend" has registered during -kernel initialization; this registration indicates that this -frontswap backend has access to some "memory" that is not directly -accessible by the kernel. Exactly how much memory it provides is -entirely dynamic and random. - -Whenever a swap-device is swapon'd frontswap_init() is called, -passing the swap device number (aka "type") as a parameter. -This notifies frontswap to expect attempts to "store" swap pages -associated with that number. - -Whenever the swap subsystem is readying a page to write to a swap -device (c.f swap_writepage()), frontswap_store is called. Frontswap -consults with the frontswap backend and if the backend says it does NOT -have room, frontswap_store returns -1 and the kernel swaps the page -to the swap device as normal. Note that the response from the frontswap -backend is unpredictable to the kernel; it may choose to never accept a -page, it could accept every ninth page, or it might accept every -page. But if the backend does accept a page, the data from the page -has already been copied and associated with the type and offset, -and the backend guarantees the persistence of the data. In this case, -frontswap sets a bit in the "frontswap_map" for the swap device -corresponding to the page offset on the swap device to which it would -otherwise have written the data. - -When the swap subsystem needs to swap-in a page (swap_readpage()), -it first calls frontswap_load() which checks the frontswap_map to -see if the page was earlier accepted by the frontswap backend. If -it was, the page of data is filled from the frontswap backend and -the swap-in is complete. If not, the normal swap-in code is -executed to obtain the page of data from the real swap device. - -So every time the frontswap backend accepts a page, a swap device read -and (potentially) a swap device write are replaced by a "frontswap backend -store" and (possibly) a "frontswap backend loads", which are presumably much -faster. - -4) Can't frontswap be configured as a "special" swap device that is - just higher priority than any real swap device (e.g. like zswap, - or maybe swap-over-nbd/NFS)? - -No. First, the existing swap subsystem doesn't allow for any kind of -swap hierarchy. Perhaps it could be rewritten to accommodate a hierarchy, -but this would require fairly drastic changes. Even if it were -rewritten, the existing swap subsystem uses the block I/O layer which -assumes a swap device is fixed size and any page in it is linearly -addressable. Frontswap barely touches the existing swap subsystem, -and works around the constraints of the block I/O subsystem to provide -a great deal of flexibility and dynamicity. - -For example, the acceptance of any swap page by the frontswap backend is -entirely unpredictable. This is critical to the definition of frontswap -backends because it grants completely dynamic discretion to the -backend. In zcache, one cannot know a priori how compressible a page is. -"Poorly" compressible pages can be rejected, and "poorly" can itself be -defined dynamically depending on current memory constraints. - -Further, frontswap is entirely synchronous whereas a real swap -device is, by definition, asynchronous and uses block I/O. The -block I/O layer is not only unnecessary, but may perform "optimizations" -that are inappropriate for a RAM-oriented device including delaying -the write of some pages for a significant amount of time. Synchrony is -required to ensure the dynamicity of the backend and to avoid thorny race -conditions that would unnecessarily and greatly complicate frontswap -and/or the block I/O subsystem. That said, only the initial "store" -and "load" operations need be synchronous. A separate asynchronous thread -is free to manipulate the pages stored by frontswap. For example, -the "remotification" thread in RAMster uses standard asynchronous -kernel sockets to move compressed frontswap pages to a remote machine. -Similarly, a KVM guest-side implementation could do in-guest compression -and use "batched" hypercalls. - -In a virtualized environment, the dynamicity allows the hypervisor -(or host OS) to do "intelligent overcommit". For example, it can -choose to accept pages only until host-swapping might be imminent, -then force guests to do their own swapping. - -There is a downside to the transcendent memory specifications for -frontswap: Since any "store" might fail, there must always be a real -slot on a real swap device to swap the page. Thus frontswap must be -implemented as a "shadow" to every swapon'd device with the potential -capability of holding every page that the swap device might have held -and the possibility that it might hold no pages at all. This means -that frontswap cannot contain more pages than the total of swapon'd -swap devices. For example, if NO swap device is configured on some -installation, frontswap is useless. Swapless portable devices -can still use frontswap but a backend for such devices must configure -some kind of "ghost" swap device and ensure that it is never used. - -5) Why this weird definition about "duplicate stores"? If a page - has been previously successfully stored, can't it always be - successfully overwritten? - -Nearly always it can, but no, sometimes it cannot. Consider an example -where data is compressed and the original 4K page has been compressed -to 1K. Now an attempt is made to overwrite the page with data that -is non-compressible and so would take the entire 4K. But the backend -has no more space. In this case, the store must be rejected. Whenever -frontswap rejects a store that would overwrite, it also must invalidate -the old data and ensure that it is no longer accessible. Since the -swap subsystem then writes the new data to the read swap device, -this is the correct course of action to ensure coherency. - -6) What is frontswap_shrink for? - -When the (non-frontswap) swap subsystem swaps out a page to a real -swap device, that page is only taking up low-value pre-allocated disk -space. But if frontswap has placed a page in transcendent memory, that -page may be taking up valuable real estate. The frontswap_shrink -routine allows code outside of the swap subsystem to force pages out -of the memory managed by frontswap and back into kernel-addressable memory. -For example, in RAMster, a "suction driver" thread will attempt -to "repatriate" pages sent to a remote machine back to the local machine; -this is driven using the frontswap_shrink mechanism when memory pressure -subsides. - -7) Why does the frontswap patch create the new include file swapfile.h? - -The frontswap code depends on some swap-subsystem-internal data -structures that have, over the years, moved back and forth between -static and global. This seemed a reasonable compromise: Define -them as global but declare them in a new include file that isn't -included by the large number of source files that include swap.h. - -Dan Magenheimer, last updated April 9, 2012 diff --git a/Documentation/vm/highmem.rst b/Documentation/vm/highmem.rst new file mode 100644 index 000000000000..0f69a9fec34d --- /dev/null +++ b/Documentation/vm/highmem.rst @@ -0,0 +1,147 @@ +.. _highmem: + +==================== +High Memory Handling +==================== + +By: Peter Zijlstra + +.. contents:: :local: + +What Is High Memory? +==================== + +High memory (highmem) is used when the size of physical memory approaches or +exceeds the maximum size of virtual memory. At that point it becomes +impossible for the kernel to keep all of the available physical memory mapped +at all times. This means the kernel needs to start using temporary mappings of +the pieces of physical memory that it wants to access. + +The part of (physical) memory not covered by a permanent mapping is what we +refer to as 'highmem'. There are various architecture dependent constraints on +where exactly that border lies. + +In the i386 arch, for example, we choose to map the kernel into every process's +VM space so that we don't have to pay the full TLB invalidation costs for +kernel entry/exit. This means the available virtual memory space (4GiB on +i386) has to be divided between user and kernel space. + +The traditional split for architectures using this approach is 3:1, 3GiB for +userspace and the top 1GiB for kernel space:: + + +--------+ 0xffffffff + | Kernel | + +--------+ 0xc0000000 + | | + | User | + | | + +--------+ 0x00000000 + +This means that the kernel can at most map 1GiB of physical memory at any one +time, but because we need virtual address space for other things - including +temporary maps to access the rest of the physical memory - the actual direct +map will typically be less (usually around ~896MiB). + +Other architectures that have mm context tagged TLBs can have separate kernel +and user maps. Some hardware (like some ARMs), however, have limited virtual +space when they use mm context tags. + + +Temporary Virtual Mappings +========================== + +The kernel contains several ways of creating temporary mappings: + +* vmap(). This can be used to make a long duration mapping of multiple + physical pages into a contiguous virtual space. It needs global + synchronization to unmap. + +* kmap(). This permits a short duration mapping of a single page. It needs + global synchronization, but is amortized somewhat. It is also prone to + deadlocks when using in a nested fashion, and so it is not recommended for + new code. + +* kmap_atomic(). This permits a very short duration mapping of a single + page. Since the mapping is restricted to the CPU that issued it, it + performs well, but the issuing task is therefore required to stay on that + CPU until it has finished, lest some other task displace its mappings. + + kmap_atomic() may also be used by interrupt contexts, since it is does not + sleep and the caller may not sleep until after kunmap_atomic() is called. + + It may be assumed that k[un]map_atomic() won't fail. + + +Using kmap_atomic +================= + +When and where to use kmap_atomic() is straightforward. It is used when code +wants to access the contents of a page that might be allocated from high memory +(see __GFP_HIGHMEM), for example a page in the pagecache. The API has two +functions, and they can be used in a manner similar to the following:: + + /* Find the page of interest. */ + struct page *page = find_get_page(mapping, offset); + + /* Gain access to the contents of that page. */ + void *vaddr = kmap_atomic(page); + + /* Do something to the contents of that page. */ + memset(vaddr, 0, PAGE_SIZE); + + /* Unmap that page. */ + kunmap_atomic(vaddr); + +Note that the kunmap_atomic() call takes the result of the kmap_atomic() call +not the argument. + +If you need to map two pages because you want to copy from one page to +another you need to keep the kmap_atomic calls strictly nested, like:: + + vaddr1 = kmap_atomic(page1); + vaddr2 = kmap_atomic(page2); + + memcpy(vaddr1, vaddr2, PAGE_SIZE); + + kunmap_atomic(vaddr2); + kunmap_atomic(vaddr1); + + +Cost of Temporary Mappings +========================== + +The cost of creating temporary mappings can be quite high. The arch has to +manipulate the kernel's page tables, the data TLB and/or the MMU's registers. + +If CONFIG_HIGHMEM is not set, then the kernel will try and create a mapping +simply with a bit of arithmetic that will convert the page struct address into +a pointer to the page contents rather than juggling mappings about. In such a +case, the unmap operation may be a null operation. + +If CONFIG_MMU is not set, then there can be no temporary mappings and no +highmem. In such a case, the arithmetic approach will also be used. + + +i386 PAE +======== + +The i386 arch, under some circumstances, will permit you to stick up to 64GiB +of RAM into your 32-bit machine. This has a number of consequences: + +* Linux needs a page-frame structure for each page in the system and the + pageframes need to live in the permanent mapping, which means: + +* you can have 896M/sizeof(struct page) page-frames at most; with struct + page being 32-bytes that would end up being something in the order of 112G + worth of pages; the kernel, however, needs to store more than just + page-frames in that memory... + +* PAE makes your page tables larger - which slows the system down as more + data has to be accessed to traverse in TLB fills and the like. One + advantage is that PAE has more PTE bits and can provide advanced features + like NX and PAT. + +The general recommendation is that you don't use more than 8GiB on a 32-bit +machine - although more might work for you and your workload, you're pretty +much on your own - don't expect kernel developers to really care much if things +come apart. diff --git a/Documentation/vm/highmem.txt b/Documentation/vm/highmem.txt deleted file mode 100644 index 4324d24ffacd..000000000000 --- a/Documentation/vm/highmem.txt +++ /dev/null @@ -1,162 +0,0 @@ - - ==================== - HIGH MEMORY HANDLING - ==================== - -By: Peter Zijlstra - -Contents: - - (*) What is high memory? - - (*) Temporary virtual mappings. - - (*) Using kmap_atomic. - - (*) Cost of temporary mappings. - - (*) i386 PAE. - - -==================== -WHAT IS HIGH MEMORY? -==================== - -High memory (highmem) is used when the size of physical memory approaches or -exceeds the maximum size of virtual memory. At that point it becomes -impossible for the kernel to keep all of the available physical memory mapped -at all times. This means the kernel needs to start using temporary mappings of -the pieces of physical memory that it wants to access. - -The part of (physical) memory not covered by a permanent mapping is what we -refer to as 'highmem'. There are various architecture dependent constraints on -where exactly that border lies. - -In the i386 arch, for example, we choose to map the kernel into every process's -VM space so that we don't have to pay the full TLB invalidation costs for -kernel entry/exit. This means the available virtual memory space (4GiB on -i386) has to be divided between user and kernel space. - -The traditional split for architectures using this approach is 3:1, 3GiB for -userspace and the top 1GiB for kernel space: - - +--------+ 0xffffffff - | Kernel | - +--------+ 0xc0000000 - | | - | User | - | | - +--------+ 0x00000000 - -This means that the kernel can at most map 1GiB of physical memory at any one -time, but because we need virtual address space for other things - including -temporary maps to access the rest of the physical memory - the actual direct -map will typically be less (usually around ~896MiB). - -Other architectures that have mm context tagged TLBs can have separate kernel -and user maps. Some hardware (like some ARMs), however, have limited virtual -space when they use mm context tags. - - -========================== -TEMPORARY VIRTUAL MAPPINGS -========================== - -The kernel contains several ways of creating temporary mappings: - - (*) vmap(). This can be used to make a long duration mapping of multiple - physical pages into a contiguous virtual space. It needs global - synchronization to unmap. - - (*) kmap(). This permits a short duration mapping of a single page. It needs - global synchronization, but is amortized somewhat. It is also prone to - deadlocks when using in a nested fashion, and so it is not recommended for - new code. - - (*) kmap_atomic(). This permits a very short duration mapping of a single - page. Since the mapping is restricted to the CPU that issued it, it - performs well, but the issuing task is therefore required to stay on that - CPU until it has finished, lest some other task displace its mappings. - - kmap_atomic() may also be used by interrupt contexts, since it is does not - sleep and the caller may not sleep until after kunmap_atomic() is called. - - It may be assumed that k[un]map_atomic() won't fail. - - -================= -USING KMAP_ATOMIC -================= - -When and where to use kmap_atomic() is straightforward. It is used when code -wants to access the contents of a page that might be allocated from high memory -(see __GFP_HIGHMEM), for example a page in the pagecache. The API has two -functions, and they can be used in a manner similar to the following: - - /* Find the page of interest. */ - struct page *page = find_get_page(mapping, offset); - - /* Gain access to the contents of that page. */ - void *vaddr = kmap_atomic(page); - - /* Do something to the contents of that page. */ - memset(vaddr, 0, PAGE_SIZE); - - /* Unmap that page. */ - kunmap_atomic(vaddr); - -Note that the kunmap_atomic() call takes the result of the kmap_atomic() call -not the argument. - -If you need to map two pages because you want to copy from one page to -another you need to keep the kmap_atomic calls strictly nested, like: - - vaddr1 = kmap_atomic(page1); - vaddr2 = kmap_atomic(page2); - - memcpy(vaddr1, vaddr2, PAGE_SIZE); - - kunmap_atomic(vaddr2); - kunmap_atomic(vaddr1); - - -========================== -COST OF TEMPORARY MAPPINGS -========================== - -The cost of creating temporary mappings can be quite high. The arch has to -manipulate the kernel's page tables, the data TLB and/or the MMU's registers. - -If CONFIG_HIGHMEM is not set, then the kernel will try and create a mapping -simply with a bit of arithmetic that will convert the page struct address into -a pointer to the page contents rather than juggling mappings about. In such a -case, the unmap operation may be a null operation. - -If CONFIG_MMU is not set, then there can be no temporary mappings and no -highmem. In such a case, the arithmetic approach will also be used. - - -======== -i386 PAE -======== - -The i386 arch, under some circumstances, will permit you to stick up to 64GiB -of RAM into your 32-bit machine. This has a number of consequences: - - (*) Linux needs a page-frame structure for each page in the system and the - pageframes need to live in the permanent mapping, which means: - - (*) you can have 896M/sizeof(struct page) page-frames at most; with struct - page being 32-bytes that would end up being something in the order of 112G - worth of pages; the kernel, however, needs to store more than just - page-frames in that memory... - - (*) PAE makes your page tables larger - which slows the system down as more - data has to be accessed to traverse in TLB fills and the like. One - advantage is that PAE has more PTE bits and can provide advanced features - like NX and PAT. - -The general recommendation is that you don't use more than 8GiB on a 32-bit -machine - although more might work for you and your workload, you're pretty -much on your own - don't expect kernel developers to really care much if things -come apart. diff --git a/Documentation/vm/hmm.rst b/Documentation/vm/hmm.rst new file mode 100644 index 000000000000..cdf3911582c8 --- /dev/null +++ b/Documentation/vm/hmm.rst @@ -0,0 +1,386 @@ +.. hmm: + +===================================== +Heterogeneous Memory Management (HMM) +===================================== + +Provide infrastructure and helpers to integrate non-conventional memory (device +memory like GPU on board memory) into regular kernel path, with the cornerstone +of this being specialized struct page for such memory (see sections 5 to 7 of +this document). + +HMM also provides optional helpers for SVM (Share Virtual Memory), i.e., +allowing a device to transparently access program address coherently with +the CPU meaning that any valid pointer on the CPU is also a valid pointer +for the device. This is becoming mandatory to simplify the use of advanced +heterogeneous computing where GPU, DSP, or FPGA are used to perform various +computations on behalf of a process. + +This document is divided as follows: in the first section I expose the problems +related to using device specific memory allocators. In the second section, I +expose the hardware limitations that are inherent to many platforms. The third +section gives an overview of the HMM design. The fourth section explains how +CPU page-table mirroring works and the purpose of HMM in this context. The +fifth section deals with how device memory is represented inside the kernel. +Finally, the last section presents a new migration helper that allows lever- +aging the device DMA engine. + +.. contents:: :local: + +Problems of using a device specific memory allocator +==================================================== + +Devices with a large amount of on board memory (several gigabytes) like GPUs +have historically managed their memory through dedicated driver specific APIs. +This creates a disconnect between memory allocated and managed by a device +driver and regular application memory (private anonymous, shared memory, or +regular file backed memory). From here on I will refer to this aspect as split +address space. I use shared address space to refer to the opposite situation: +i.e., one in which any application memory region can be used by a device +transparently. + +Split address space happens because device can only access memory allocated +through device specific API. This implies that all memory objects in a program +are not equal from the device point of view which complicates large programs +that rely on a wide set of libraries. + +Concretely this means that code that wants to leverage devices like GPUs needs +to copy object between generically allocated memory (malloc, mmap private, mmap +share) and memory allocated through the device driver API (this still ends up +with an mmap but of the device file). + +For flat data sets (array, grid, image, ...) this isn't too hard to achieve but +complex data sets (list, tree, ...) are hard to get right. Duplicating a +complex data set needs to re-map all the pointer relations between each of its +elements. This is error prone and program gets harder to debug because of the +duplicate data set and addresses. + +Split address space also means that libraries cannot transparently use data +they are getting from the core program or another library and thus each library +might have to duplicate its input data set using the device specific memory +allocator. Large projects suffer from this and waste resources because of the +various memory copies. + +Duplicating each library API to accept as input or output memory allocated by +each device specific allocator is not a viable option. It would lead to a +combinatorial explosion in the library entry points. + +Finally, with the advance of high level language constructs (in C++ but in +other languages too) it is now possible for the compiler to leverage GPUs and +other devices without programmer knowledge. Some compiler identified patterns +are only do-able with a shared address space. It is also more reasonable to use +a shared address space for all other patterns. + + +I/O bus, device memory characteristics +====================================== + +I/O buses cripple shared address spaces due to a few limitations. Most I/O +buses only allow basic memory access from device to main memory; even cache +coherency is often optional. Access to device memory from CPU is even more +limited. More often than not, it is not cache coherent. + +If we only consider the PCIE bus, then a device can access main memory (often +through an IOMMU) and be cache coherent with the CPUs. However, it only allows +a limited set of atomic operations from device on main memory. This is worse +in the other direction: the CPU can only access a limited range of the device +memory and cannot perform atomic operations on it. Thus device memory cannot +be considered the same as regular memory from the kernel point of view. + +Another crippling factor is the limited bandwidth (~32GBytes/s with PCIE 4.0 +and 16 lanes). This is 33 times less than the fastest GPU memory (1 TBytes/s). +The final limitation is latency. Access to main memory from the device has an +order of magnitude higher latency than when the device accesses its own memory. + +Some platforms are developing new I/O buses or additions/modifications to PCIE +to address some of these limitations (OpenCAPI, CCIX). They mainly allow two- +way cache coherency between CPU and device and allow all atomic operations the +architecture supports. Sadly, not all platforms are following this trend and +some major architectures are left without hardware solutions to these problems. + +So for shared address space to make sense, not only must we allow devices to +access any memory but we must also permit any memory to be migrated to device +memory while device is using it (blocking CPU access while it happens). + + +Shared address space and migration +================================== + +HMM intends to provide two main features. First one is to share the address +space by duplicating the CPU page table in the device page table so the same +address points to the same physical memory for any valid main memory address in +the process address space. + +To achieve this, HMM offers a set of helpers to populate the device page table +while keeping track of CPU page table updates. Device page table updates are +not as easy as CPU page table updates. To update the device page table, you must +allocate a buffer (or use a pool of pre-allocated buffers) and write GPU +specific commands in it to perform the update (unmap, cache invalidations, and +flush, ...). This cannot be done through common code for all devices. Hence +why HMM provides helpers to factor out everything that can be while leaving the +hardware specific details to the device driver. + +The second mechanism HMM provides is a new kind of ZONE_DEVICE memory that +allows allocating a struct page for each page of the device memory. Those pages +are special because the CPU cannot map them. However, they allow migrating +main memory to device memory using existing migration mechanisms and everything +looks like a page is swapped out to disk from the CPU point of view. Using a +struct page gives the easiest and cleanest integration with existing mm mech- +anisms. Here again, HMM only provides helpers, first to hotplug new ZONE_DEVICE +memory for the device memory and second to perform migration. Policy decisions +of what and when to migrate things is left to the device driver. + +Note that any CPU access to a device page triggers a page fault and a migration +back to main memory. For example, when a page backing a given CPU address A is +migrated from a main memory page to a device page, then any CPU access to +address A triggers a page fault and initiates a migration back to main memory. + +With these two features, HMM not only allows a device to mirror process address +space and keeping both CPU and device page table synchronized, but also lever- +ages device memory by migrating the part of the data set that is actively being +used by the device. + + +Address space mirroring implementation and API +============================================== + +Address space mirroring's main objective is to allow duplication of a range of +CPU page table into a device page table; HMM helps keep both synchronized. A +device driver that wants to mirror a process address space must start with the +registration of an hmm_mirror struct:: + + int hmm_mirror_register(struct hmm_mirror *mirror, + struct mm_struct *mm); + int hmm_mirror_register_locked(struct hmm_mirror *mirror, + struct mm_struct *mm); + + +The locked variant is to be used when the driver is already holding mmap_sem +of the mm in write mode. The mirror struct has a set of callbacks that are used +to propagate CPU page tables:: + + struct hmm_mirror_ops { + /* sync_cpu_device_pagetables() - synchronize page tables + * + * @mirror: pointer to struct hmm_mirror + * @update_type: type of update that occurred to the CPU page table + * @start: virtual start address of the range to update + * @end: virtual end address of the range to update + * + * This callback ultimately originates from mmu_notifiers when the CPU + * page table is updated. The device driver must update its page table + * in response to this callback. The update argument tells what action + * to perform. + * + * The device driver must not return from this callback until the device + * page tables are completely updated (TLBs flushed, etc); this is a + * synchronous call. + */ + void (*update)(struct hmm_mirror *mirror, + enum hmm_update action, + unsigned long start, + unsigned long end); + }; + +The device driver must perform the update action to the range (mark range +read only, or fully unmap, ...). The device must be done with the update before +the driver callback returns. + +When the device driver wants to populate a range of virtual addresses, it can +use either:: + + int hmm_vma_get_pfns(struct vm_area_struct *vma, + struct hmm_range *range, + unsigned long start, + unsigned long end, + hmm_pfn_t *pfns); + int hmm_vma_fault(struct vm_area_struct *vma, + struct hmm_range *range, + unsigned long start, + unsigned long end, + hmm_pfn_t *pfns, + bool write, + bool block); + +The first one (hmm_vma_get_pfns()) will only fetch present CPU page table +entries and will not trigger a page fault on missing or non-present entries. +The second one does trigger a page fault on missing or read-only entry if the +write parameter is true. Page faults use the generic mm page fault code path +just like a CPU page fault. + +Both functions copy CPU page table entries into their pfns array argument. Each +entry in that array corresponds to an address in the virtual range. HMM +provides a set of flags to help the driver identify special CPU page table +entries. + +Locking with the update() callback is the most important aspect the driver must +respect in order to keep things properly synchronized. The usage pattern is:: + + int driver_populate_range(...) + { + struct hmm_range range; + ... + again: + ret = hmm_vma_get_pfns(vma, &range, start, end, pfns); + if (ret) + return ret; + take_lock(driver->update); + if (!hmm_vma_range_done(vma, &range)) { + release_lock(driver->update); + goto again; + } + + // Use pfns array content to update device page table + + release_lock(driver->update); + return 0; + } + +The driver->update lock is the same lock that the driver takes inside its +update() callback. That lock must be held before hmm_vma_range_done() to avoid +any race with a concurrent CPU page table update. + +HMM implements all this on top of the mmu_notifier API because we wanted a +simpler API and also to be able to perform optimizations latter on like doing +concurrent device updates in multi-devices scenario. + +HMM also serves as an impedance mismatch between how CPU page table updates +are done (by CPU write to the page table and TLB flushes) and how devices +update their own page table. Device updates are a multi-step process. First, +appropriate commands are written to a buffer, then this buffer is scheduled for +execution on the device. It is only once the device has executed commands in +the buffer that the update is done. Creating and scheduling the update command +buffer can happen concurrently for multiple devices. Waiting for each device to +report commands as executed is serialized (there is no point in doing this +concurrently). + + +Represent and manage device memory from core kernel point of view +================================================================= + +Several different designs were tried to support device memory. First one used +a device specific data structure to keep information about migrated memory and +HMM hooked itself in various places of mm code to handle any access to +addresses that were backed by device memory. It turns out that this ended up +replicating most of the fields of struct page and also needed many kernel code +paths to be updated to understand this new kind of memory. + +Most kernel code paths never try to access the memory behind a page +but only care about struct page contents. Because of this, HMM switched to +directly using struct page for device memory which left most kernel code paths +unaware of the difference. We only need to make sure that no one ever tries to +map those pages from the CPU side. + +HMM provides a set of helpers to register and hotplug device memory as a new +region needing a struct page. This is offered through a very simple API:: + + struct hmm_devmem *hmm_devmem_add(const struct hmm_devmem_ops *ops, + struct device *device, + unsigned long size); + void hmm_devmem_remove(struct hmm_devmem *devmem); + +The hmm_devmem_ops is where most of the important things are:: + + struct hmm_devmem_ops { + void (*free)(struct hmm_devmem *devmem, struct page *page); + int (*fault)(struct hmm_devmem *devmem, + struct vm_area_struct *vma, + unsigned long addr, + struct page *page, + unsigned flags, + pmd_t *pmdp); + }; + +The first callback (free()) happens when the last reference on a device page is +dropped. This means the device page is now free and no longer used by anyone. +The second callback happens whenever the CPU tries to access a device page +which it cannot do. This second callback must trigger a migration back to +system memory. + + +Migration to and from device memory +=================================== + +Because the CPU cannot access device memory, migration must use the device DMA +engine to perform copy from and to device memory. For this we need a new +migration helper:: + + int migrate_vma(const struct migrate_vma_ops *ops, + struct vm_area_struct *vma, + unsigned long mentries, + unsigned long start, + unsigned long end, + unsigned long *src, + unsigned long *dst, + void *private); + +Unlike other migration functions it works on a range of virtual address, there +are two reasons for that. First, device DMA copy has a high setup overhead cost +and thus batching multiple pages is needed as otherwise the migration overhead +makes the whole exercise pointless. The second reason is because the +migration might be for a range of addresses the device is actively accessing. + +The migrate_vma_ops struct defines two callbacks. First one (alloc_and_copy()) +controls destination memory allocation and copy operation. Second one is there +to allow the device driver to perform cleanup operations after migration:: + + struct migrate_vma_ops { + void (*alloc_and_copy)(struct vm_area_struct *vma, + const unsigned long *src, + unsigned long *dst, + unsigned long start, + unsigned long end, + void *private); + void (*finalize_and_map)(struct vm_area_struct *vma, + const unsigned long *src, + const unsigned long *dst, + unsigned long start, + unsigned long end, + void *private); + }; + +It is important to stress that these migration helpers allow for holes in the +virtual address range. Some pages in the range might not be migrated for all +the usual reasons (page is pinned, page is locked, ...). This helper does not +fail but just skips over those pages. + +The alloc_and_copy() might decide to not migrate all pages in the +range (for reasons under the callback control). For those, the callback just +has to leave the corresponding dst entry empty. + +Finally, the migration of the struct page might fail (for file backed page) for +various reasons (failure to freeze reference, or update page cache, ...). If +that happens, then the finalize_and_map() can catch any pages that were not +migrated. Note those pages were still copied to a new page and thus we wasted +bandwidth but this is considered as a rare event and a price that we are +willing to pay to keep all the code simpler. + + +Memory cgroup (memcg) and rss accounting +======================================== + +For now device memory is accounted as any regular page in rss counters (either +anonymous if device page is used for anonymous, file if device page is used for +file backed page or shmem if device page is used for shared memory). This is a +deliberate choice to keep existing applications, that might start using device +memory without knowing about it, running unimpacted. + +A drawback is that the OOM killer might kill an application using a lot of +device memory and not a lot of regular system memory and thus not freeing much +system memory. We want to gather more real world experience on how applications +and system react under memory pressure in the presence of device memory before +deciding to account device memory differently. + + +Same decision was made for memory cgroup. Device memory pages are accounted +against same memory cgroup a regular page would be accounted to. This does +simplify migration to and from device memory. This also means that migration +back from device memory to regular memory cannot fail because it would +go above memory cgroup limit. We might revisit this choice latter on once we +get more experience in how device memory is used and its impact on memory +resource control. + + +Note that device memory can never be pinned by device driver nor through GUP +and thus such memory is always free upon process exit. Or when last reference +is dropped in case of shared memory or file backed memory. diff --git a/Documentation/vm/hmm.txt b/Documentation/vm/hmm.txt deleted file mode 100644 index 2d1d6f69e91b..000000000000 --- a/Documentation/vm/hmm.txt +++ /dev/null @@ -1,396 +0,0 @@ -Heterogeneous Memory Management (HMM) - -Provide infrastructure and helpers to integrate non-conventional memory (device -memory like GPU on board memory) into regular kernel path, with the cornerstone -of this being specialized struct page for such memory (see sections 5 to 7 of -this document). - -HMM also provides optional helpers for SVM (Share Virtual Memory), i.e., -allowing a device to transparently access program address coherently with the -CPU meaning that any valid pointer on the CPU is also a valid pointer for the -device. This is becoming mandatory to simplify the use of advanced hetero- -geneous computing where GPU, DSP, or FPGA are used to perform various -computations on behalf of a process. - -This document is divided as follows: in the first section I expose the problems -related to using device specific memory allocators. In the second section, I -expose the hardware limitations that are inherent to many platforms. The third -section gives an overview of the HMM design. The fourth section explains how -CPU page-table mirroring works and the purpose of HMM in this context. The -fifth section deals with how device memory is represented inside the kernel. -Finally, the last section presents a new migration helper that allows lever- -aging the device DMA engine. - - -1) Problems of using a device specific memory allocator: -2) I/O bus, device memory characteristics -3) Shared address space and migration -4) Address space mirroring implementation and API -5) Represent and manage device memory from core kernel point of view -6) Migration to and from device memory -7) Memory cgroup (memcg) and rss accounting - - -------------------------------------------------------------------------------- - -1) Problems of using a device specific memory allocator: - -Devices with a large amount of on board memory (several gigabytes) like GPUs -have historically managed their memory through dedicated driver specific APIs. -This creates a disconnect between memory allocated and managed by a device -driver and regular application memory (private anonymous, shared memory, or -regular file backed memory). From here on I will refer to this aspect as split -address space. I use shared address space to refer to the opposite situation: -i.e., one in which any application memory region can be used by a device -transparently. - -Split address space happens because device can only access memory allocated -through device specific API. This implies that all memory objects in a program -are not equal from the device point of view which complicates large programs -that rely on a wide set of libraries. - -Concretely this means that code that wants to leverage devices like GPUs needs -to copy object between generically allocated memory (malloc, mmap private, mmap -share) and memory allocated through the device driver API (this still ends up -with an mmap but of the device file). - -For flat data sets (array, grid, image, ...) this isn't too hard to achieve but -complex data sets (list, tree, ...) are hard to get right. Duplicating a -complex data set needs to re-map all the pointer relations between each of its -elements. This is error prone and program gets harder to debug because of the -duplicate data set and addresses. - -Split address space also means that libraries cannot transparently use data -they are getting from the core program or another library and thus each library -might have to duplicate its input data set using the device specific memory -allocator. Large projects suffer from this and waste resources because of the -various memory copies. - -Duplicating each library API to accept as input or output memory allocated by -each device specific allocator is not a viable option. It would lead to a -combinatorial explosion in the library entry points. - -Finally, with the advance of high level language constructs (in C++ but in -other languages too) it is now possible for the compiler to leverage GPUs and -other devices without programmer knowledge. Some compiler identified patterns -are only do-able with a shared address space. It is also more reasonable to use -a shared address space for all other patterns. - - -------------------------------------------------------------------------------- - -2) I/O bus, device memory characteristics - -I/O buses cripple shared address spaces due to a few limitations. Most I/O -buses only allow basic memory access from device to main memory; even cache -coherency is often optional. Access to device memory from CPU is even more -limited. More often than not, it is not cache coherent. - -If we only consider the PCIE bus, then a device can access main memory (often -through an IOMMU) and be cache coherent with the CPUs. However, it only allows -a limited set of atomic operations from device on main memory. This is worse -in the other direction: the CPU can only access a limited range of the device -memory and cannot perform atomic operations on it. Thus device memory cannot -be considered the same as regular memory from the kernel point of view. - -Another crippling factor is the limited bandwidth (~32GBytes/s with PCIE 4.0 -and 16 lanes). This is 33 times less than the fastest GPU memory (1 TBytes/s). -The final limitation is latency. Access to main memory from the device has an -order of magnitude higher latency than when the device accesses its own memory. - -Some platforms are developing new I/O buses or additions/modifications to PCIE -to address some of these limitations (OpenCAPI, CCIX). They mainly allow two- -way cache coherency between CPU and device and allow all atomic operations the -architecture supports. Sadly, not all platforms are following this trend and -some major architectures are left without hardware solutions to these problems. - -So for shared address space to make sense, not only must we allow devices to -access any memory but we must also permit any memory to be migrated to device -memory while device is using it (blocking CPU access while it happens). - - -------------------------------------------------------------------------------- - -3) Shared address space and migration - -HMM intends to provide two main features. First one is to share the address -space by duplicating the CPU page table in the device page table so the same -address points to the same physical memory for any valid main memory address in -the process address space. - -To achieve this, HMM offers a set of helpers to populate the device page table -while keeping track of CPU page table updates. Device page table updates are -not as easy as CPU page table updates. To update the device page table, you must -allocate a buffer (or use a pool of pre-allocated buffers) and write GPU -specific commands in it to perform the update (unmap, cache invalidations, and -flush, ...). This cannot be done through common code for all devices. Hence -why HMM provides helpers to factor out everything that can be while leaving the -hardware specific details to the device driver. - -The second mechanism HMM provides is a new kind of ZONE_DEVICE memory that -allows allocating a struct page for each page of the device memory. Those pages -are special because the CPU cannot map them. However, they allow migrating -main memory to device memory using existing migration mechanisms and everything -looks like a page is swapped out to disk from the CPU point of view. Using a -struct page gives the easiest and cleanest integration with existing mm mech- -anisms. Here again, HMM only provides helpers, first to hotplug new ZONE_DEVICE -memory for the device memory and second to perform migration. Policy decisions -of what and when to migrate things is left to the device driver. - -Note that any CPU access to a device page triggers a page fault and a migration -back to main memory. For example, when a page backing a given CPU address A is -migrated from a main memory page to a device page, then any CPU access to -address A triggers a page fault and initiates a migration back to main memory. - -With these two features, HMM not only allows a device to mirror process address -space and keeping both CPU and device page table synchronized, but also lever- -ages device memory by migrating the part of the data set that is actively being -used by the device. - - -------------------------------------------------------------------------------- - -4) Address space mirroring implementation and API - -Address space mirroring's main objective is to allow duplication of a range of -CPU page table into a device page table; HMM helps keep both synchronized. A -device driver that wants to mirror a process address space must start with the -registration of an hmm_mirror struct: - - int hmm_mirror_register(struct hmm_mirror *mirror, - struct mm_struct *mm); - int hmm_mirror_register_locked(struct hmm_mirror *mirror, - struct mm_struct *mm); - -The locked variant is to be used when the driver is already holding mmap_sem -of the mm in write mode. The mirror struct has a set of callbacks that are used -to propagate CPU page tables: - - struct hmm_mirror_ops { - /* sync_cpu_device_pagetables() - synchronize page tables - * - * @mirror: pointer to struct hmm_mirror - * @update_type: type of update that occurred to the CPU page table - * @start: virtual start address of the range to update - * @end: virtual end address of the range to update - * - * This callback ultimately originates from mmu_notifiers when the CPU - * page table is updated. The device driver must update its page table - * in response to this callback. The update argument tells what action - * to perform. - * - * The device driver must not return from this callback until the device - * page tables are completely updated (TLBs flushed, etc); this is a - * synchronous call. - */ - void (*update)(struct hmm_mirror *mirror, - enum hmm_update action, - unsigned long start, - unsigned long end); - }; - -The device driver must perform the update action to the range (mark range -read only, or fully unmap, ...). The device must be done with the update before -the driver callback returns. - - -When the device driver wants to populate a range of virtual addresses, it can -use either: - int hmm_vma_get_pfns(struct vm_area_struct *vma, - struct hmm_range *range, - unsigned long start, - unsigned long end, - hmm_pfn_t *pfns); - int hmm_vma_fault(struct vm_area_struct *vma, - struct hmm_range *range, - unsigned long start, - unsigned long end, - hmm_pfn_t *pfns, - bool write, - bool block); - -The first one (hmm_vma_get_pfns()) will only fetch present CPU page table -entries and will not trigger a page fault on missing or non-present entries. -The second one does trigger a page fault on missing or read-only entry if the -write parameter is true. Page faults use the generic mm page fault code path -just like a CPU page fault. - -Both functions copy CPU page table entries into their pfns array argument. Each -entry in that array corresponds to an address in the virtual range. HMM -provides a set of flags to help the driver identify special CPU page table -entries. - -Locking with the update() callback is the most important aspect the driver must -respect in order to keep things properly synchronized. The usage pattern is: - - int driver_populate_range(...) - { - struct hmm_range range; - ... - again: - ret = hmm_vma_get_pfns(vma, &range, start, end, pfns); - if (ret) - return ret; - take_lock(driver->update); - if (!hmm_vma_range_done(vma, &range)) { - release_lock(driver->update); - goto again; - } - - // Use pfns array content to update device page table - - release_lock(driver->update); - return 0; - } - -The driver->update lock is the same lock that the driver takes inside its -update() callback. That lock must be held before hmm_vma_range_done() to avoid -any race with a concurrent CPU page table update. - -HMM implements all this on top of the mmu_notifier API because we wanted a -simpler API and also to be able to perform optimizations latter on like doing -concurrent device updates in multi-devices scenario. - -HMM also serves as an impedance mismatch between how CPU page table updates -are done (by CPU write to the page table and TLB flushes) and how devices -update their own page table. Device updates are a multi-step process. First, -appropriate commands are written to a buffer, then this buffer is scheduled for -execution on the device. It is only once the device has executed commands in -the buffer that the update is done. Creating and scheduling the update command -buffer can happen concurrently for multiple devices. Waiting for each device to -report commands as executed is serialized (there is no point in doing this -concurrently). - - -------------------------------------------------------------------------------- - -5) Represent and manage device memory from core kernel point of view - -Several different designs were tried to support device memory. First one used -a device specific data structure to keep information about migrated memory and -HMM hooked itself in various places of mm code to handle any access to -addresses that were backed by device memory. It turns out that this ended up -replicating most of the fields of struct page and also needed many kernel code -paths to be updated to understand this new kind of memory. - -Most kernel code paths never try to access the memory behind a page -but only care about struct page contents. Because of this, HMM switched to -directly using struct page for device memory which left most kernel code paths -unaware of the difference. We only need to make sure that no one ever tries to -map those pages from the CPU side. - -HMM provides a set of helpers to register and hotplug device memory as a new -region needing a struct page. This is offered through a very simple API: - - struct hmm_devmem *hmm_devmem_add(const struct hmm_devmem_ops *ops, - struct device *device, - unsigned long size); - void hmm_devmem_remove(struct hmm_devmem *devmem); - -The hmm_devmem_ops is where most of the important things are: - - struct hmm_devmem_ops { - void (*free)(struct hmm_devmem *devmem, struct page *page); - int (*fault)(struct hmm_devmem *devmem, - struct vm_area_struct *vma, - unsigned long addr, - struct page *page, - unsigned flags, - pmd_t *pmdp); - }; - -The first callback (free()) happens when the last reference on a device page is -dropped. This means the device page is now free and no longer used by anyone. -The second callback happens whenever the CPU tries to access a device page -which it cannot do. This second callback must trigger a migration back to -system memory. - - -------------------------------------------------------------------------------- - -6) Migration to and from device memory - -Because the CPU cannot access device memory, migration must use the device DMA -engine to perform copy from and to device memory. For this we need a new -migration helper: - - int migrate_vma(const struct migrate_vma_ops *ops, - struct vm_area_struct *vma, - unsigned long mentries, - unsigned long start, - unsigned long end, - unsigned long *src, - unsigned long *dst, - void *private); - -Unlike other migration functions it works on a range of virtual address, there -are two reasons for that. First, device DMA copy has a high setup overhead cost -and thus batching multiple pages is needed as otherwise the migration overhead -makes the whole exercise pointless. The second reason is because the -migration might be for a range of addresses the device is actively accessing. - -The migrate_vma_ops struct defines two callbacks. First one (alloc_and_copy()) -controls destination memory allocation and copy operation. Second one is there -to allow the device driver to perform cleanup operations after migration. - - struct migrate_vma_ops { - void (*alloc_and_copy)(struct vm_area_struct *vma, - const unsigned long *src, - unsigned long *dst, - unsigned long start, - unsigned long end, - void *private); - void (*finalize_and_map)(struct vm_area_struct *vma, - const unsigned long *src, - const unsigned long *dst, - unsigned long start, - unsigned long end, - void *private); - }; - -It is important to stress that these migration helpers allow for holes in the -virtual address range. Some pages in the range might not be migrated for all -the usual reasons (page is pinned, page is locked, ...). This helper does not -fail but just skips over those pages. - -The alloc_and_copy() might decide to not migrate all pages in the -range (for reasons under the callback control). For those, the callback just -has to leave the corresponding dst entry empty. - -Finally, the migration of the struct page might fail (for file backed page) for -various reasons (failure to freeze reference, or update page cache, ...). If -that happens, then the finalize_and_map() can catch any pages that were not -migrated. Note those pages were still copied to a new page and thus we wasted -bandwidth but this is considered as a rare event and a price that we are -willing to pay to keep all the code simpler. - - -------------------------------------------------------------------------------- - -7) Memory cgroup (memcg) and rss accounting - -For now device memory is accounted as any regular page in rss counters (either -anonymous if device page is used for anonymous, file if device page is used for -file backed page or shmem if device page is used for shared memory). This is a -deliberate choice to keep existing applications, that might start using device -memory without knowing about it, running unimpacted. - -A drawback is that the OOM killer might kill an application using a lot of -device memory and not a lot of regular system memory and thus not freeing much -system memory. We want to gather more real world experience on how applications -and system react under memory pressure in the presence of device memory before -deciding to account device memory differently. - - -Same decision was made for memory cgroup. Device memory pages are accounted -against same memory cgroup a regular page would be accounted to. This does -simplify migration to and from device memory. This also means that migration -back from device memory to regular memory cannot fail because it would -go above memory cgroup limit. We might revisit this choice latter on once we -get more experience in how device memory is used and its impact on memory -resource control. - - -Note that device memory can never be pinned by device driver nor through GUP -and thus such memory is always free upon process exit. Or when last reference -is dropped in case of shared memory or file backed memory. diff --git a/Documentation/vm/hugetlbfs_reserv.rst b/Documentation/vm/hugetlbfs_reserv.rst new file mode 100644 index 000000000000..9d200762114f --- /dev/null +++ b/Documentation/vm/hugetlbfs_reserv.rst @@ -0,0 +1,595 @@ +.. _hugetlbfs_reserve: + +===================== +Hugetlbfs Reservation +===================== + +Overview +======== + +Huge pages as described at :ref:`hugetlbpage` are typically +preallocated for application use. These huge pages are instantiated in a +task's address space at page fault time if the VMA indicates huge pages are +to be used. If no huge page exists at page fault time, the task is sent +a SIGBUS and often dies an unhappy death. Shortly after huge page support +was added, it was determined that it would be better to detect a shortage +of huge pages at mmap() time. The idea is that if there were not enough +huge pages to cover the mapping, the mmap() would fail. This was first +done with a simple check in the code at mmap() time to determine if there +were enough free huge pages to cover the mapping. Like most things in the +kernel, the code has evolved over time. However, the basic idea was to +'reserve' huge pages at mmap() time to ensure that huge pages would be +available for page faults in that mapping. The description below attempts to +describe how huge page reserve processing is done in the v4.10 kernel. + + +Audience +======== +This description is primarily targeted at kernel developers who are modifying +hugetlbfs code. + + +The Data Structures +=================== + +resv_huge_pages + This is a global (per-hstate) count of reserved huge pages. Reserved + huge pages are only available to the task which reserved them. + Therefore, the number of huge pages generally available is computed + as (``free_huge_pages - resv_huge_pages``). +Reserve Map + A reserve map is described by the structure:: + + struct resv_map { + struct kref refs; + spinlock_t lock; + struct list_head regions; + long adds_in_progress; + struct list_head region_cache; + long region_cache_count; + }; + + There is one reserve map for each huge page mapping in the system. + The regions list within the resv_map describes the regions within + the mapping. A region is described as:: + + struct file_region { + struct list_head link; + long from; + long to; + }; + + The 'from' and 'to' fields of the file region structure are huge page + indices into the mapping. Depending on the type of mapping, a + region in the reserv_map may indicate reservations exist for the + range, or reservations do not exist. +Flags for MAP_PRIVATE Reservations + These are stored in the bottom bits of the reservation map pointer. + + ``#define HPAGE_RESV_OWNER (1UL << 0)`` + Indicates this task is the owner of the reservations + associated with the mapping. + ``#define HPAGE_RESV_UNMAPPED (1UL << 1)`` + Indicates task originally mapping this range (and creating + reserves) has unmapped a page from this task (the child) + due to a failed COW. +Page Flags + The PagePrivate page flag is used to indicate that a huge page + reservation must be restored when the huge page is freed. More + details will be discussed in the "Freeing huge pages" section. + + +Reservation Map Location (Private or Shared) +============================================ + +A huge page mapping or segment is either private or shared. If private, +it is typically only available to a single address space (task). If shared, +it can be mapped into multiple address spaces (tasks). The location and +semantics of the reservation map is significantly different for two types +of mappings. Location differences are: + +- For private mappings, the reservation map hangs off the the VMA structure. + Specifically, vma->vm_private_data. This reserve map is created at the + time the mapping (mmap(MAP_PRIVATE)) is created. +- For shared mappings, the reservation map hangs off the inode. Specifically, + inode->i_mapping->private_data. Since shared mappings are always backed + by files in the hugetlbfs filesystem, the hugetlbfs code ensures each inode + contains a reservation map. As a result, the reservation map is allocated + when the inode is created. + + +Creating Reservations +===================== +Reservations are created when a huge page backed shared memory segment is +created (shmget(SHM_HUGETLB)) or a mapping is created via mmap(MAP_HUGETLB). +These operations result in a call to the routine hugetlb_reserve_pages():: + + int hugetlb_reserve_pages(struct inode *inode, + long from, long to, + struct vm_area_struct *vma, + vm_flags_t vm_flags) + +The first thing hugetlb_reserve_pages() does is check for the NORESERVE +flag was specified in either the shmget() or mmap() call. If NORESERVE +was specified, then this routine returns immediately as no reservation +are desired. + +The arguments 'from' and 'to' are huge page indices into the mapping or +underlying file. For shmget(), 'from' is always 0 and 'to' corresponds to +the length of the segment/mapping. For mmap(), the offset argument could +be used to specify the offset into the underlying file. In such a case +the 'from' and 'to' arguments have been adjusted by this offset. + +One of the big differences between PRIVATE and SHARED mappings is the way +in which reservations are represented in the reservation map. + +- For shared mappings, an entry in the reservation map indicates a reservation + exists or did exist for the corresponding page. As reservations are + consumed, the reservation map is not modified. +- For private mappings, the lack of an entry in the reservation map indicates + a reservation exists for the corresponding page. As reservations are + consumed, entries are added to the reservation map. Therefore, the + reservation map can also be used to determine which reservations have + been consumed. + +For private mappings, hugetlb_reserve_pages() creates the reservation map and +hangs it off the VMA structure. In addition, the HPAGE_RESV_OWNER flag is set +to indicate this VMA owns the reservations. + +The reservation map is consulted to determine how many huge page reservations +are needed for the current mapping/segment. For private mappings, this is +always the value (to - from). However, for shared mappings it is possible that some reservations may already exist within the range (to - from). See the +section :ref:`Reservation Map Modifications ` +for details on how this is accomplished. + +The mapping may be associated with a subpool. If so, the subpool is consulted +to ensure there is sufficient space for the mapping. It is possible that the +subpool has set aside reservations that can be used for the mapping. See the +section :ref:`Subpool Reservations ` for more details. + +After consulting the reservation map and subpool, the number of needed new +reservations is known. The routine hugetlb_acct_memory() is called to check +for and take the requested number of reservations. hugetlb_acct_memory() +calls into routines that potentially allocate and adjust surplus page counts. +However, within those routines the code is simply checking to ensure there +are enough free huge pages to accommodate the reservation. If there are, +the global reservation count resv_huge_pages is adjusted something like the +following:: + + if (resv_needed <= (resv_huge_pages - free_huge_pages)) + resv_huge_pages += resv_needed; + +Note that the global lock hugetlb_lock is held when checking and adjusting +these counters. + +If there were enough free huge pages and the global count resv_huge_pages +was adjusted, then the reservation map associated with the mapping is +modified to reflect the reservations. In the case of a shared mapping, a +file_region will exist that includes the range 'from' 'to'. For private +mappings, no modifications are made to the reservation map as lack of an +entry indicates a reservation exists. + +If hugetlb_reserve_pages() was successful, the global reservation count and +reservation map associated with the mapping will be modified as required to +ensure reservations exist for the range 'from' - 'to'. + +.. _consume_resv: + +Consuming Reservations/Allocating a Huge Page +============================================= + +Reservations are consumed when huge pages associated with the reservations +are allocated and instantiated in the corresponding mapping. The allocation +is performed within the routine alloc_huge_page():: + + struct page *alloc_huge_page(struct vm_area_struct *vma, + unsigned long addr, int avoid_reserve) + +alloc_huge_page is passed a VMA pointer and a virtual address, so it can +consult the reservation map to determine if a reservation exists. In addition, +alloc_huge_page takes the argument avoid_reserve which indicates reserves +should not be used even if it appears they have been set aside for the +specified address. The avoid_reserve argument is most often used in the case +of Copy on Write and Page Migration where additional copies of an existing +page are being allocated. + +The helper routine vma_needs_reservation() is called to determine if a +reservation exists for the address within the mapping(vma). See the section +:ref:`Reservation Map Helper Routines ` for detailed +information on what this routine does. +The value returned from vma_needs_reservation() is generally +0 or 1. 0 if a reservation exists for the address, 1 if no reservation exists. +If a reservation does not exist, and there is a subpool associated with the +mapping the subpool is consulted to determine if it contains reservations. +If the subpool contains reservations, one can be used for this allocation. +However, in every case the avoid_reserve argument overrides the use of +a reservation for the allocation. After determining whether a reservation +exists and can be used for the allocation, the routine dequeue_huge_page_vma() +is called. This routine takes two arguments related to reservations: + +- avoid_reserve, this is the same value/argument passed to alloc_huge_page() +- chg, even though this argument is of type long only the values 0 or 1 are + passed to dequeue_huge_page_vma. If the value is 0, it indicates a + reservation exists (see the section "Memory Policy and Reservations" for + possible issues). If the value is 1, it indicates a reservation does not + exist and the page must be taken from the global free pool if possible. + +The free lists associated with the memory policy of the VMA are searched for +a free page. If a page is found, the value free_huge_pages is decremented +when the page is removed from the free list. If there was a reservation +associated with the page, the following adjustments are made:: + + SetPagePrivate(page); /* Indicates allocating this page consumed + * a reservation, and if an error is + * encountered such that the page must be + * freed, the reservation will be restored. */ + resv_huge_pages--; /* Decrement the global reservation count */ + +Note, if no huge page can be found that satisfies the VMA's memory policy +an attempt will be made to allocate one using the buddy allocator. This +brings up the issue of surplus huge pages and overcommit which is beyond +the scope reservations. Even if a surplus page is allocated, the same +reservation based adjustments as above will be made: SetPagePrivate(page) and +resv_huge_pages--. + +After obtaining a new huge page, (page)->private is set to the value of +the subpool associated with the page if it exists. This will be used for +subpool accounting when the page is freed. + +The routine vma_commit_reservation() is then called to adjust the reserve +map based on the consumption of the reservation. In general, this involves +ensuring the page is represented within a file_region structure of the region +map. For shared mappings where the the reservation was present, an entry +in the reserve map already existed so no change is made. However, if there +was no reservation in a shared mapping or this was a private mapping a new +entry must be created. + +It is possible that the reserve map could have been changed between the call +to vma_needs_reservation() at the beginning of alloc_huge_page() and the +call to vma_commit_reservation() after the page was allocated. This would +be possible if hugetlb_reserve_pages was called for the same page in a shared +mapping. In such cases, the reservation count and subpool free page count +will be off by one. This rare condition can be identified by comparing the +return value from vma_needs_reservation and vma_commit_reservation. If such +a race is detected, the subpool and global reserve counts are adjusted to +compensate. See the section +:ref:`Reservation Map Helper Routines ` for more +information on these routines. + + +Instantiate Huge Pages +====================== + +After huge page allocation, the page is typically added to the page tables +of the allocating task. Before this, pages in a shared mapping are added +to the page cache and pages in private mappings are added to an anonymous +reverse mapping. In both cases, the PagePrivate flag is cleared. Therefore, +when a huge page that has been instantiated is freed no adjustment is made +to the global reservation count (resv_huge_pages). + + +Freeing Huge Pages +================== + +Huge page freeing is performed by the routine free_huge_page(). This routine +is the destructor for hugetlbfs compound pages. As a result, it is only +passed a pointer to the page struct. When a huge page is freed, reservation +accounting may need to be performed. This would be the case if the page was +associated with a subpool that contained reserves, or the page is being freed +on an error path where a global reserve count must be restored. + +The page->private field points to any subpool associated with the page. +If the PagePrivate flag is set, it indicates the global reserve count should +be adjusted (see the section +:ref:`Consuming Reservations/Allocating a Huge Page ` +for information on how these are set). + +The routine first calls hugepage_subpool_put_pages() for the page. If this +routine returns a value of 0 (which does not equal the value passed 1) it +indicates reserves are associated with the subpool, and this newly free page +must be used to keep the number of subpool reserves above the minimum size. +Therefore, the global resv_huge_pages counter is incremented in this case. + +If the PagePrivate flag was set in the page, the global resv_huge_pages counter +will always be incremented. + +.. _sub_pool_resv: + +Subpool Reservations +==================== + +There is a struct hstate associated with each huge page size. The hstate +tracks all huge pages of the specified size. A subpool represents a subset +of pages within a hstate that is associated with a mounted hugetlbfs +filesystem. + +When a hugetlbfs filesystem is mounted a min_size option can be specified +which indicates the minimum number of huge pages required by the filesystem. +If this option is specified, the number of huge pages corresponding to +min_size are reserved for use by the filesystem. This number is tracked in +the min_hpages field of a struct hugepage_subpool. At mount time, +hugetlb_acct_memory(min_hpages) is called to reserve the specified number of +huge pages. If they can not be reserved, the mount fails. + +The routines hugepage_subpool_get/put_pages() are called when pages are +obtained from or released back to a subpool. They perform all subpool +accounting, and track any reservations associated with the subpool. +hugepage_subpool_get/put_pages are passed the number of huge pages by which +to adjust the subpool 'used page' count (down for get, up for put). Normally, +they return the same value that was passed or an error if not enough pages +exist in the subpool. + +However, if reserves are associated with the subpool a return value less +than the passed value may be returned. This return value indicates the +number of additional global pool adjustments which must be made. For example, +suppose a subpool contains 3 reserved huge pages and someone asks for 5. +The 3 reserved pages associated with the subpool can be used to satisfy part +of the request. But, 2 pages must be obtained from the global pools. To +relay this information to the caller, the value 2 is returned. The caller +is then responsible for attempting to obtain the additional two pages from +the global pools. + + +COW and Reservations +==================== + +Since shared mappings all point to and use the same underlying pages, the +biggest reservation concern for COW is private mappings. In this case, +two tasks can be pointing at the same previously allocated page. One task +attempts to write to the page, so a new page must be allocated so that each +task points to its own page. + +When the page was originally allocated, the reservation for that page was +consumed. When an attempt to allocate a new page is made as a result of +COW, it is possible that no free huge pages are free and the allocation +will fail. + +When the private mapping was originally created, the owner of the mapping +was noted by setting the HPAGE_RESV_OWNER bit in the pointer to the reservation +map of the owner. Since the owner created the mapping, the owner owns all +the reservations associated with the mapping. Therefore, when a write fault +occurs and there is no page available, different action is taken for the owner +and non-owner of the reservation. + +In the case where the faulting task is not the owner, the fault will fail and +the task will typically receive a SIGBUS. + +If the owner is the faulting task, we want it to succeed since it owned the +original reservation. To accomplish this, the page is unmapped from the +non-owning task. In this way, the only reference is from the owning task. +In addition, the HPAGE_RESV_UNMAPPED bit is set in the reservation map pointer +of the non-owning task. The non-owning task may receive a SIGBUS if it later +faults on a non-present page. But, the original owner of the +mapping/reservation will behave as expected. + + +.. _resv_map_modifications: + +Reservation Map Modifications +============================= + +The following low level routines are used to make modifications to a +reservation map. Typically, these routines are not called directly. Rather, +a reservation map helper routine is called which calls one of these low level +routines. These low level routines are fairly well documented in the source +code (mm/hugetlb.c). These routines are:: + + long region_chg(struct resv_map *resv, long f, long t); + long region_add(struct resv_map *resv, long f, long t); + void region_abort(struct resv_map *resv, long f, long t); + long region_count(struct resv_map *resv, long f, long t); + +Operations on the reservation map typically involve two operations: + +1) region_chg() is called to examine the reserve map and determine how + many pages in the specified range [f, t) are NOT currently represented. + + The calling code performs global checks and allocations to determine if + there are enough huge pages for the operation to succeed. + +2) + a) If the operation can succeed, region_add() is called to actually modify + the reservation map for the same range [f, t) previously passed to + region_chg(). + b) If the operation can not succeed, region_abort is called for the same + range [f, t) to abort the operation. + +Note that this is a two step process where region_add() and region_abort() +are guaranteed to succeed after a prior call to region_chg() for the same +range. region_chg() is responsible for pre-allocating any data structures +necessary to ensure the subsequent operations (specifically region_add())) +will succeed. + +As mentioned above, region_chg() determines the number of pages in the range +which are NOT currently represented in the map. This number is returned to +the caller. region_add() returns the number of pages in the range added to +the map. In most cases, the return value of region_add() is the same as the +return value of region_chg(). However, in the case of shared mappings it is +possible for changes to the reservation map to be made between the calls to +region_chg() and region_add(). In this case, the return value of region_add() +will not match the return value of region_chg(). It is likely that in such +cases global counts and subpool accounting will be incorrect and in need of +adjustment. It is the responsibility of the caller to check for this condition +and make the appropriate adjustments. + +The routine region_del() is called to remove regions from a reservation map. +It is typically called in the following situations: + +- When a file in the hugetlbfs filesystem is being removed, the inode will + be released and the reservation map freed. Before freeing the reservation + map, all the individual file_region structures must be freed. In this case + region_del is passed the range [0, LONG_MAX). +- When a hugetlbfs file is being truncated. In this case, all allocated pages + after the new file size must be freed. In addition, any file_region entries + in the reservation map past the new end of file must be deleted. In this + case, region_del is passed the range [new_end_of_file, LONG_MAX). +- When a hole is being punched in a hugetlbfs file. In this case, huge pages + are removed from the middle of the file one at a time. As the pages are + removed, region_del() is called to remove the corresponding entry from the + reservation map. In this case, region_del is passed the range + [page_idx, page_idx + 1). + +In every case, region_del() will return the number of pages removed from the +reservation map. In VERY rare cases, region_del() can fail. This can only +happen in the hole punch case where it has to split an existing file_region +entry and can not allocate a new structure. In this error case, region_del() +will return -ENOMEM. The problem here is that the reservation map will +indicate that there is a reservation for the page. However, the subpool and +global reservation counts will not reflect the reservation. To handle this +situation, the routine hugetlb_fix_reserve_counts() is called to adjust the +counters so that they correspond with the reservation map entry that could +not be deleted. + +region_count() is called when unmapping a private huge page mapping. In +private mappings, the lack of a entry in the reservation map indicates that +a reservation exists. Therefore, by counting the number of entries in the +reservation map we know how many reservations were consumed and how many are +outstanding (outstanding = (end - start) - region_count(resv, start, end)). +Since the mapping is going away, the subpool and global reservation counts +are decremented by the number of outstanding reservations. + +.. _resv_map_helpers: + +Reservation Map Helper Routines +=============================== + +Several helper routines exist to query and modify the reservation maps. +These routines are only interested with reservations for a specific huge +page, so they just pass in an address instead of a range. In addition, +they pass in the associated VMA. From the VMA, the type of mapping (private +or shared) and the location of the reservation map (inode or VMA) can be +determined. These routines simply call the underlying routines described +in the section "Reservation Map Modifications". However, they do take into +account the 'opposite' meaning of reservation map entries for private and +shared mappings and hide this detail from the caller:: + + long vma_needs_reservation(struct hstate *h, + struct vm_area_struct *vma, + unsigned long addr) + +This routine calls region_chg() for the specified page. If no reservation +exists, 1 is returned. If a reservation exists, 0 is returned:: + + long vma_commit_reservation(struct hstate *h, + struct vm_area_struct *vma, + unsigned long addr) + +This calls region_add() for the specified page. As in the case of region_chg +and region_add, this routine is to be called after a previous call to +vma_needs_reservation. It will add a reservation entry for the page. It +returns 1 if the reservation was added and 0 if not. The return value should +be compared with the return value of the previous call to +vma_needs_reservation. An unexpected difference indicates the reservation +map was modified between calls:: + + void vma_end_reservation(struct hstate *h, + struct vm_area_struct *vma, + unsigned long addr) + +This calls region_abort() for the specified page. As in the case of region_chg +and region_abort, this routine is to be called after a previous call to +vma_needs_reservation. It will abort/end the in progress reservation add +operation:: + + long vma_add_reservation(struct hstate *h, + struct vm_area_struct *vma, + unsigned long addr) + +This is a special wrapper routine to help facilitate reservation cleanup +on error paths. It is only called from the routine restore_reserve_on_error(). +This routine is used in conjunction with vma_needs_reservation in an attempt +to add a reservation to the reservation map. It takes into account the +different reservation map semantics for private and shared mappings. Hence, +region_add is called for shared mappings (as an entry present in the map +indicates a reservation), and region_del is called for private mappings (as +the absence of an entry in the map indicates a reservation). See the section +"Reservation cleanup in error paths" for more information on what needs to +be done on error paths. + + +Reservation Cleanup in Error Paths +================================== + +As mentioned in the section +:ref:`Reservation Map Helper Routines `, reservation +map modifications are performed in two steps. First vma_needs_reservation +is called before a page is allocated. If the allocation is successful, +then vma_commit_reservation is called. If not, vma_end_reservation is called. +Global and subpool reservation counts are adjusted based on success or failure +of the operation and all is well. + +Additionally, after a huge page is instantiated the PagePrivate flag is +cleared so that accounting when the page is ultimately freed is correct. + +However, there are several instances where errors are encountered after a huge +page is allocated but before it is instantiated. In this case, the page +allocation has consumed the reservation and made the appropriate subpool, +reservation map and global count adjustments. If the page is freed at this +time (before instantiation and clearing of PagePrivate), then free_huge_page +will increment the global reservation count. However, the reservation map +indicates the reservation was consumed. This resulting inconsistent state +will cause the 'leak' of a reserved huge page. The global reserve count will +be higher than it should and prevent allocation of a pre-allocated page. + +The routine restore_reserve_on_error() attempts to handle this situation. It +is fairly well documented. The intention of this routine is to restore +the reservation map to the way it was before the page allocation. In this +way, the state of the reservation map will correspond to the global reservation +count after the page is freed. + +The routine restore_reserve_on_error itself may encounter errors while +attempting to restore the reservation map entry. In this case, it will +simply clear the PagePrivate flag of the page. In this way, the global +reserve count will not be incremented when the page is freed. However, the +reservation map will continue to look as though the reservation was consumed. +A page can still be allocated for the address, but it will not use a reserved +page as originally intended. + +There is some code (most notably userfaultfd) which can not call +restore_reserve_on_error. In this case, it simply modifies the PagePrivate +so that a reservation will not be leaked when the huge page is freed. + + +Reservations and Memory Policy +============================== +Per-node huge page lists existed in struct hstate when git was first used +to manage Linux code. The concept of reservations was added some time later. +When reservations were added, no attempt was made to take memory policy +into account. While cpusets are not exactly the same as memory policy, this +comment in hugetlb_acct_memory sums up the interaction between reservations +and cpusets/memory policy:: + + /* + * When cpuset is configured, it breaks the strict hugetlb page + * reservation as the accounting is done on a global variable. Such + * reservation is completely rubbish in the presence of cpuset because + * the reservation is not checked against page availability for the + * current cpuset. Application can still potentially OOM'ed by kernel + * with lack of free htlb page in cpuset that the task is in. + * Attempt to enforce strict accounting with cpuset is almost + * impossible (or too ugly) because cpuset is too fluid that + * task or memory node can be dynamically moved between cpusets. + * + * The change of semantics for shared hugetlb mapping with cpuset is + * undesirable. However, in order to preserve some of the semantics, + * we fall back to check against current free page availability as + * a best attempt and hopefully to minimize the impact of changing + * semantics that cpuset has. + */ + +Huge page reservations were added to prevent unexpected page allocation +failures (OOM) at page fault time. However, if an application makes use +of cpusets or memory policy there is no guarantee that huge pages will be +available on the required nodes. This is true even if there are a sufficient +number of global reservations. + +Hugetlbfs regression testing +============================ + +The most complete set of hugetlb tests are in the libhugetlbfs repository. +If you modify any hugetlb related code, use the libhugetlbfs test suite +to check for regressions. In addition, if you add any new hugetlb +functionality, please add appropriate tests to libhugetlbfs. + +-- +Mike Kravetz, 7 April 2017 diff --git a/Documentation/vm/hugetlbfs_reserv.txt b/Documentation/vm/hugetlbfs_reserv.txt deleted file mode 100644 index 9aca09a76bed..000000000000 --- a/Documentation/vm/hugetlbfs_reserv.txt +++ /dev/null @@ -1,529 +0,0 @@ -Hugetlbfs Reservation Overview ------------------------------- -Huge pages as described at 'Documentation/vm/hugetlbpage.txt' are typically -preallocated for application use. These huge pages are instantiated in a -task's address space at page fault time if the VMA indicates huge pages are -to be used. If no huge page exists at page fault time, the task is sent -a SIGBUS and often dies an unhappy death. Shortly after huge page support -was added, it was determined that it would be better to detect a shortage -of huge pages at mmap() time. The idea is that if there were not enough -huge pages to cover the mapping, the mmap() would fail. This was first -done with a simple check in the code at mmap() time to determine if there -were enough free huge pages to cover the mapping. Like most things in the -kernel, the code has evolved over time. However, the basic idea was to -'reserve' huge pages at mmap() time to ensure that huge pages would be -available for page faults in that mapping. The description below attempts to -describe how huge page reserve processing is done in the v4.10 kernel. - - -Audience --------- -This description is primarily targeted at kernel developers who are modifying -hugetlbfs code. - - -The Data Structures -------------------- -resv_huge_pages - This is a global (per-hstate) count of reserved huge pages. Reserved - huge pages are only available to the task which reserved them. - Therefore, the number of huge pages generally available is computed - as (free_huge_pages - resv_huge_pages). -Reserve Map - A reserve map is described by the structure: - struct resv_map { - struct kref refs; - spinlock_t lock; - struct list_head regions; - long adds_in_progress; - struct list_head region_cache; - long region_cache_count; - }; - There is one reserve map for each huge page mapping in the system. - The regions list within the resv_map describes the regions within - the mapping. A region is described as: - struct file_region { - struct list_head link; - long from; - long to; - }; - The 'from' and 'to' fields of the file region structure are huge page - indices into the mapping. Depending on the type of mapping, a - region in the reserv_map may indicate reservations exist for the - range, or reservations do not exist. -Flags for MAP_PRIVATE Reservations - These are stored in the bottom bits of the reservation map pointer. - #define HPAGE_RESV_OWNER (1UL << 0) Indicates this task is the - owner of the reservations associated with the mapping. - #define HPAGE_RESV_UNMAPPED (1UL << 1) Indicates task originally - mapping this range (and creating reserves) has unmapped a - page from this task (the child) due to a failed COW. -Page Flags - The PagePrivate page flag is used to indicate that a huge page - reservation must be restored when the huge page is freed. More - details will be discussed in the "Freeing huge pages" section. - - -Reservation Map Location (Private or Shared) --------------------------------------------- -A huge page mapping or segment is either private or shared. If private, -it is typically only available to a single address space (task). If shared, -it can be mapped into multiple address spaces (tasks). The location and -semantics of the reservation map is significantly different for two types -of mappings. Location differences are: -- For private mappings, the reservation map hangs off the the VMA structure. - Specifically, vma->vm_private_data. This reserve map is created at the - time the mapping (mmap(MAP_PRIVATE)) is created. -- For shared mappings, the reservation map hangs off the inode. Specifically, - inode->i_mapping->private_data. Since shared mappings are always backed - by files in the hugetlbfs filesystem, the hugetlbfs code ensures each inode - contains a reservation map. As a result, the reservation map is allocated - when the inode is created. - - -Creating Reservations ---------------------- -Reservations are created when a huge page backed shared memory segment is -created (shmget(SHM_HUGETLB)) or a mapping is created via mmap(MAP_HUGETLB). -These operations result in a call to the routine hugetlb_reserve_pages() - -int hugetlb_reserve_pages(struct inode *inode, - long from, long to, - struct vm_area_struct *vma, - vm_flags_t vm_flags) - -The first thing hugetlb_reserve_pages() does is check for the NORESERVE -flag was specified in either the shmget() or mmap() call. If NORESERVE -was specified, then this routine returns immediately as no reservation -are desired. - -The arguments 'from' and 'to' are huge page indices into the mapping or -underlying file. For shmget(), 'from' is always 0 and 'to' corresponds to -the length of the segment/mapping. For mmap(), the offset argument could -be used to specify the offset into the underlying file. In such a case -the 'from' and 'to' arguments have been adjusted by this offset. - -One of the big differences between PRIVATE and SHARED mappings is the way -in which reservations are represented in the reservation map. -- For shared mappings, an entry in the reservation map indicates a reservation - exists or did exist for the corresponding page. As reservations are - consumed, the reservation map is not modified. -- For private mappings, the lack of an entry in the reservation map indicates - a reservation exists for the corresponding page. As reservations are - consumed, entries are added to the reservation map. Therefore, the - reservation map can also be used to determine which reservations have - been consumed. - -For private mappings, hugetlb_reserve_pages() creates the reservation map and -hangs it off the VMA structure. In addition, the HPAGE_RESV_OWNER flag is set -to indicate this VMA owns the reservations. - -The reservation map is consulted to determine how many huge page reservations -are needed for the current mapping/segment. For private mappings, this is -always the value (to - from). However, for shared mappings it is possible that some reservations may already exist within the range (to - from). See the -section "Reservation Map Modifications" for details on how this is accomplished. - -The mapping may be associated with a subpool. If so, the subpool is consulted -to ensure there is sufficient space for the mapping. It is possible that the -subpool has set aside reservations that can be used for the mapping. See the -section "Subpool Reservations" for more details. - -After consulting the reservation map and subpool, the number of needed new -reservations is known. The routine hugetlb_acct_memory() is called to check -for and take the requested number of reservations. hugetlb_acct_memory() -calls into routines that potentially allocate and adjust surplus page counts. -However, within those routines the code is simply checking to ensure there -are enough free huge pages to accommodate the reservation. If there are, -the global reservation count resv_huge_pages is adjusted something like the -following. - if (resv_needed <= (resv_huge_pages - free_huge_pages)) - resv_huge_pages += resv_needed; -Note that the global lock hugetlb_lock is held when checking and adjusting -these counters. - -If there were enough free huge pages and the global count resv_huge_pages -was adjusted, then the reservation map associated with the mapping is -modified to reflect the reservations. In the case of a shared mapping, a -file_region will exist that includes the range 'from' 'to'. For private -mappings, no modifications are made to the reservation map as lack of an -entry indicates a reservation exists. - -If hugetlb_reserve_pages() was successful, the global reservation count and -reservation map associated with the mapping will be modified as required to -ensure reservations exist for the range 'from' - 'to'. - - -Consuming Reservations/Allocating a Huge Page ---------------------------------------------- -Reservations are consumed when huge pages associated with the reservations -are allocated and instantiated in the corresponding mapping. The allocation -is performed within the routine alloc_huge_page(). -struct page *alloc_huge_page(struct vm_area_struct *vma, - unsigned long addr, int avoid_reserve) -alloc_huge_page is passed a VMA pointer and a virtual address, so it can -consult the reservation map to determine if a reservation exists. In addition, -alloc_huge_page takes the argument avoid_reserve which indicates reserves -should not be used even if it appears they have been set aside for the -specified address. The avoid_reserve argument is most often used in the case -of Copy on Write and Page Migration where additional copies of an existing -page are being allocated. - -The helper routine vma_needs_reservation() is called to determine if a -reservation exists for the address within the mapping(vma). See the section -"Reservation Map Helper Routines" for detailed information on what this -routine does. The value returned from vma_needs_reservation() is generally -0 or 1. 0 if a reservation exists for the address, 1 if no reservation exists. -If a reservation does not exist, and there is a subpool associated with the -mapping the subpool is consulted to determine if it contains reservations. -If the subpool contains reservations, one can be used for this allocation. -However, in every case the avoid_reserve argument overrides the use of -a reservation for the allocation. After determining whether a reservation -exists and can be used for the allocation, the routine dequeue_huge_page_vma() -is called. This routine takes two arguments related to reservations: -- avoid_reserve, this is the same value/argument passed to alloc_huge_page() -- chg, even though this argument is of type long only the values 0 or 1 are - passed to dequeue_huge_page_vma. If the value is 0, it indicates a - reservation exists (see the section "Memory Policy and Reservations" for - possible issues). If the value is 1, it indicates a reservation does not - exist and the page must be taken from the global free pool if possible. -The free lists associated with the memory policy of the VMA are searched for -a free page. If a page is found, the value free_huge_pages is decremented -when the page is removed from the free list. If there was a reservation -associated with the page, the following adjustments are made: - SetPagePrivate(page); /* Indicates allocating this page consumed - * a reservation, and if an error is - * encountered such that the page must be - * freed, the reservation will be restored. */ - resv_huge_pages--; /* Decrement the global reservation count */ -Note, if no huge page can be found that satisfies the VMA's memory policy -an attempt will be made to allocate one using the buddy allocator. This -brings up the issue of surplus huge pages and overcommit which is beyond -the scope reservations. Even if a surplus page is allocated, the same -reservation based adjustments as above will be made: SetPagePrivate(page) and -resv_huge_pages--. - -After obtaining a new huge page, (page)->private is set to the value of -the subpool associated with the page if it exists. This will be used for -subpool accounting when the page is freed. - -The routine vma_commit_reservation() is then called to adjust the reserve -map based on the consumption of the reservation. In general, this involves -ensuring the page is represented within a file_region structure of the region -map. For shared mappings where the the reservation was present, an entry -in the reserve map already existed so no change is made. However, if there -was no reservation in a shared mapping or this was a private mapping a new -entry must be created. - -It is possible that the reserve map could have been changed between the call -to vma_needs_reservation() at the beginning of alloc_huge_page() and the -call to vma_commit_reservation() after the page was allocated. This would -be possible if hugetlb_reserve_pages was called for the same page in a shared -mapping. In such cases, the reservation count and subpool free page count -will be off by one. This rare condition can be identified by comparing the -return value from vma_needs_reservation and vma_commit_reservation. If such -a race is detected, the subpool and global reserve counts are adjusted to -compensate. See the section "Reservation Map Helper Routines" for more -information on these routines. - - -Instantiate Huge Pages ----------------------- -After huge page allocation, the page is typically added to the page tables -of the allocating task. Before this, pages in a shared mapping are added -to the page cache and pages in private mappings are added to an anonymous -reverse mapping. In both cases, the PagePrivate flag is cleared. Therefore, -when a huge page that has been instantiated is freed no adjustment is made -to the global reservation count (resv_huge_pages). - - -Freeing Huge Pages ------------------- -Huge page freeing is performed by the routine free_huge_page(). This routine -is the destructor for hugetlbfs compound pages. As a result, it is only -passed a pointer to the page struct. When a huge page is freed, reservation -accounting may need to be performed. This would be the case if the page was -associated with a subpool that contained reserves, or the page is being freed -on an error path where a global reserve count must be restored. - -The page->private field points to any subpool associated with the page. -If the PagePrivate flag is set, it indicates the global reserve count should -be adjusted (see the section "Consuming Reservations/Allocating a Huge Page" -for information on how these are set). - -The routine first calls hugepage_subpool_put_pages() for the page. If this -routine returns a value of 0 (which does not equal the value passed 1) it -indicates reserves are associated with the subpool, and this newly free page -must be used to keep the number of subpool reserves above the minimum size. -Therefore, the global resv_huge_pages counter is incremented in this case. - -If the PagePrivate flag was set in the page, the global resv_huge_pages counter -will always be incremented. - - -Subpool Reservations --------------------- -There is a struct hstate associated with each huge page size. The hstate -tracks all huge pages of the specified size. A subpool represents a subset -of pages within a hstate that is associated with a mounted hugetlbfs -filesystem. - -When a hugetlbfs filesystem is mounted a min_size option can be specified -which indicates the minimum number of huge pages required by the filesystem. -If this option is specified, the number of huge pages corresponding to -min_size are reserved for use by the filesystem. This number is tracked in -the min_hpages field of a struct hugepage_subpool. At mount time, -hugetlb_acct_memory(min_hpages) is called to reserve the specified number of -huge pages. If they can not be reserved, the mount fails. - -The routines hugepage_subpool_get/put_pages() are called when pages are -obtained from or released back to a subpool. They perform all subpool -accounting, and track any reservations associated with the subpool. -hugepage_subpool_get/put_pages are passed the number of huge pages by which -to adjust the subpool 'used page' count (down for get, up for put). Normally, -they return the same value that was passed or an error if not enough pages -exist in the subpool. - -However, if reserves are associated with the subpool a return value less -than the passed value may be returned. This return value indicates the -number of additional global pool adjustments which must be made. For example, -suppose a subpool contains 3 reserved huge pages and someone asks for 5. -The 3 reserved pages associated with the subpool can be used to satisfy part -of the request. But, 2 pages must be obtained from the global pools. To -relay this information to the caller, the value 2 is returned. The caller -is then responsible for attempting to obtain the additional two pages from -the global pools. - - -COW and Reservations --------------------- -Since shared mappings all point to and use the same underlying pages, the -biggest reservation concern for COW is private mappings. In this case, -two tasks can be pointing at the same previously allocated page. One task -attempts to write to the page, so a new page must be allocated so that each -task points to its own page. - -When the page was originally allocated, the reservation for that page was -consumed. When an attempt to allocate a new page is made as a result of -COW, it is possible that no free huge pages are free and the allocation -will fail. - -When the private mapping was originally created, the owner of the mapping -was noted by setting the HPAGE_RESV_OWNER bit in the pointer to the reservation -map of the owner. Since the owner created the mapping, the owner owns all -the reservations associated with the mapping. Therefore, when a write fault -occurs and there is no page available, different action is taken for the owner -and non-owner of the reservation. - -In the case where the faulting task is not the owner, the fault will fail and -the task will typically receive a SIGBUS. - -If the owner is the faulting task, we want it to succeed since it owned the -original reservation. To accomplish this, the page is unmapped from the -non-owning task. In this way, the only reference is from the owning task. -In addition, the HPAGE_RESV_UNMAPPED bit is set in the reservation map pointer -of the non-owning task. The non-owning task may receive a SIGBUS if it later -faults on a non-present page. But, the original owner of the -mapping/reservation will behave as expected. - - -Reservation Map Modifications ------------------------------ -The following low level routines are used to make modifications to a -reservation map. Typically, these routines are not called directly. Rather, -a reservation map helper routine is called which calls one of these low level -routines. These low level routines are fairly well documented in the source -code (mm/hugetlb.c). These routines are: -long region_chg(struct resv_map *resv, long f, long t); -long region_add(struct resv_map *resv, long f, long t); -void region_abort(struct resv_map *resv, long f, long t); -long region_count(struct resv_map *resv, long f, long t); - -Operations on the reservation map typically involve two operations: -1) region_chg() is called to examine the reserve map and determine how - many pages in the specified range [f, t) are NOT currently represented. - - The calling code performs global checks and allocations to determine if - there are enough huge pages for the operation to succeed. - -2a) If the operation can succeed, region_add() is called to actually modify - the reservation map for the same range [f, t) previously passed to - region_chg(). -2b) If the operation can not succeed, region_abort is called for the same range - [f, t) to abort the operation. - -Note that this is a two step process where region_add() and region_abort() -are guaranteed to succeed after a prior call to region_chg() for the same -range. region_chg() is responsible for pre-allocating any data structures -necessary to ensure the subsequent operations (specifically region_add())) -will succeed. - -As mentioned above, region_chg() determines the number of pages in the range -which are NOT currently represented in the map. This number is returned to -the caller. region_add() returns the number of pages in the range added to -the map. In most cases, the return value of region_add() is the same as the -return value of region_chg(). However, in the case of shared mappings it is -possible for changes to the reservation map to be made between the calls to -region_chg() and region_add(). In this case, the return value of region_add() -will not match the return value of region_chg(). It is likely that in such -cases global counts and subpool accounting will be incorrect and in need of -adjustment. It is the responsibility of the caller to check for this condition -and make the appropriate adjustments. - -The routine region_del() is called to remove regions from a reservation map. -It is typically called in the following situations: -- When a file in the hugetlbfs filesystem is being removed, the inode will - be released and the reservation map freed. Before freeing the reservation - map, all the individual file_region structures must be freed. In this case - region_del is passed the range [0, LONG_MAX). -- When a hugetlbfs file is being truncated. In this case, all allocated pages - after the new file size must be freed. In addition, any file_region entries - in the reservation map past the new end of file must be deleted. In this - case, region_del is passed the range [new_end_of_file, LONG_MAX). -- When a hole is being punched in a hugetlbfs file. In this case, huge pages - are removed from the middle of the file one at a time. As the pages are - removed, region_del() is called to remove the corresponding entry from the - reservation map. In this case, region_del is passed the range - [page_idx, page_idx + 1). -In every case, region_del() will return the number of pages removed from the -reservation map. In VERY rare cases, region_del() can fail. This can only -happen in the hole punch case where it has to split an existing file_region -entry and can not allocate a new structure. In this error case, region_del() -will return -ENOMEM. The problem here is that the reservation map will -indicate that there is a reservation for the page. However, the subpool and -global reservation counts will not reflect the reservation. To handle this -situation, the routine hugetlb_fix_reserve_counts() is called to adjust the -counters so that they correspond with the reservation map entry that could -not be deleted. - -region_count() is called when unmapping a private huge page mapping. In -private mappings, the lack of a entry in the reservation map indicates that -a reservation exists. Therefore, by counting the number of entries in the -reservation map we know how many reservations were consumed and how many are -outstanding (outstanding = (end - start) - region_count(resv, start, end)). -Since the mapping is going away, the subpool and global reservation counts -are decremented by the number of outstanding reservations. - - -Reservation Map Helper Routines -------------------------------- -Several helper routines exist to query and modify the reservation maps. -These routines are only interested with reservations for a specific huge -page, so they just pass in an address instead of a range. In addition, -they pass in the associated VMA. From the VMA, the type of mapping (private -or shared) and the location of the reservation map (inode or VMA) can be -determined. These routines simply call the underlying routines described -in the section "Reservation Map Modifications". However, they do take into -account the 'opposite' meaning of reservation map entries for private and -shared mappings and hide this detail from the caller. - -long vma_needs_reservation(struct hstate *h, - struct vm_area_struct *vma, unsigned long addr) -This routine calls region_chg() for the specified page. If no reservation -exists, 1 is returned. If a reservation exists, 0 is returned. - -long vma_commit_reservation(struct hstate *h, - struct vm_area_struct *vma, unsigned long addr) -This calls region_add() for the specified page. As in the case of region_chg -and region_add, this routine is to be called after a previous call to -vma_needs_reservation. It will add a reservation entry for the page. It -returns 1 if the reservation was added and 0 if not. The return value should -be compared with the return value of the previous call to -vma_needs_reservation. An unexpected difference indicates the reservation -map was modified between calls. - -void vma_end_reservation(struct hstate *h, - struct vm_area_struct *vma, unsigned long addr) -This calls region_abort() for the specified page. As in the case of region_chg -and region_abort, this routine is to be called after a previous call to -vma_needs_reservation. It will abort/end the in progress reservation add -operation. - -long vma_add_reservation(struct hstate *h, - struct vm_area_struct *vma, unsigned long addr) -This is a special wrapper routine to help facilitate reservation cleanup -on error paths. It is only called from the routine restore_reserve_on_error(). -This routine is used in conjunction with vma_needs_reservation in an attempt -to add a reservation to the reservation map. It takes into account the -different reservation map semantics for private and shared mappings. Hence, -region_add is called for shared mappings (as an entry present in the map -indicates a reservation), and region_del is called for private mappings (as -the absence of an entry in the map indicates a reservation). See the section -"Reservation cleanup in error paths" for more information on what needs to -be done on error paths. - - -Reservation Cleanup in Error Paths ----------------------------------- -As mentioned in the section "Reservation Map Helper Routines", reservation -map modifications are performed in two steps. First vma_needs_reservation -is called before a page is allocated. If the allocation is successful, -then vma_commit_reservation is called. If not, vma_end_reservation is called. -Global and subpool reservation counts are adjusted based on success or failure -of the operation and all is well. - -Additionally, after a huge page is instantiated the PagePrivate flag is -cleared so that accounting when the page is ultimately freed is correct. - -However, there are several instances where errors are encountered after a huge -page is allocated but before it is instantiated. In this case, the page -allocation has consumed the reservation and made the appropriate subpool, -reservation map and global count adjustments. If the page is freed at this -time (before instantiation and clearing of PagePrivate), then free_huge_page -will increment the global reservation count. However, the reservation map -indicates the reservation was consumed. This resulting inconsistent state -will cause the 'leak' of a reserved huge page. The global reserve count will -be higher than it should and prevent allocation of a pre-allocated page. - -The routine restore_reserve_on_error() attempts to handle this situation. It -is fairly well documented. The intention of this routine is to restore -the reservation map to the way it was before the page allocation. In this -way, the state of the reservation map will correspond to the global reservation -count after the page is freed. - -The routine restore_reserve_on_error itself may encounter errors while -attempting to restore the reservation map entry. In this case, it will -simply clear the PagePrivate flag of the page. In this way, the global -reserve count will not be incremented when the page is freed. However, the -reservation map will continue to look as though the reservation was consumed. -A page can still be allocated for the address, but it will not use a reserved -page as originally intended. - -There is some code (most notably userfaultfd) which can not call -restore_reserve_on_error. In this case, it simply modifies the PagePrivate -so that a reservation will not be leaked when the huge page is freed. - - -Reservations and Memory Policy ------------------------------- -Per-node huge page lists existed in struct hstate when git was first used -to manage Linux code. The concept of reservations was added some time later. -When reservations were added, no attempt was made to take memory policy -into account. While cpusets are not exactly the same as memory policy, this -comment in hugetlb_acct_memory sums up the interaction between reservations -and cpusets/memory policy. - /* - * When cpuset is configured, it breaks the strict hugetlb page - * reservation as the accounting is done on a global variable. Such - * reservation is completely rubbish in the presence of cpuset because - * the reservation is not checked against page availability for the - * current cpuset. Application can still potentially OOM'ed by kernel - * with lack of free htlb page in cpuset that the task is in. - * Attempt to enforce strict accounting with cpuset is almost - * impossible (or too ugly) because cpuset is too fluid that - * task or memory node can be dynamically moved between cpusets. - * - * The change of semantics for shared hugetlb mapping with cpuset is - * undesirable. However, in order to preserve some of the semantics, - * we fall back to check against current free page availability as - * a best attempt and hopefully to minimize the impact of changing - * semantics that cpuset has. - */ - -Huge page reservations were added to prevent unexpected page allocation -failures (OOM) at page fault time. However, if an application makes use -of cpusets or memory policy there is no guarantee that huge pages will be -available on the required nodes. This is true even if there are a sufficient -number of global reservations. - - -Mike Kravetz, 7 April 2017 diff --git a/Documentation/vm/hugetlbpage.txt b/Documentation/vm/hugetlbpage.txt deleted file mode 100644 index faf077d50d42..000000000000 --- a/Documentation/vm/hugetlbpage.txt +++ /dev/null @@ -1,351 +0,0 @@ - -The intent of this file is to give a brief summary of hugetlbpage support in -the Linux kernel. This support is built on top of multiple page size support -that is provided by most modern architectures. For example, x86 CPUs normally -support 4K and 2M (1G if architecturally supported) page sizes, ia64 -architecture supports multiple page sizes 4K, 8K, 64K, 256K, 1M, 4M, 16M, -256M and ppc64 supports 4K and 16M. A TLB is a cache of virtual-to-physical -translations. Typically this is a very scarce resource on processor. -Operating systems try to make best use of limited number of TLB resources. -This optimization is more critical now as bigger and bigger physical memories -(several GBs) are more readily available. - -Users can use the huge page support in Linux kernel by either using the mmap -system call or standard SYSV shared memory system calls (shmget, shmat). - -First the Linux kernel needs to be built with the CONFIG_HUGETLBFS -(present under "File systems") and CONFIG_HUGETLB_PAGE (selected -automatically when CONFIG_HUGETLBFS is selected) configuration -options. - -The /proc/meminfo file provides information about the total number of -persistent hugetlb pages in the kernel's huge page pool. It also displays -default huge page size and information about the number of free, reserved -and surplus huge pages in the pool of huge pages of default size. -The huge page size is needed for generating the proper alignment and -size of the arguments to system calls that map huge page regions. - -The output of "cat /proc/meminfo" will include lines like: - -..... -HugePages_Total: uuu -HugePages_Free: vvv -HugePages_Rsvd: www -HugePages_Surp: xxx -Hugepagesize: yyy kB -Hugetlb: zzz kB - -where: -HugePages_Total is the size of the pool of huge pages. -HugePages_Free is the number of huge pages in the pool that are not yet - allocated. -HugePages_Rsvd is short for "reserved," and is the number of huge pages for - which a commitment to allocate from the pool has been made, - but no allocation has yet been made. Reserved huge pages - guarantee that an application will be able to allocate a - huge page from the pool of huge pages at fault time. -HugePages_Surp is short for "surplus," and is the number of huge pages in - the pool above the value in /proc/sys/vm/nr_hugepages. The - maximum number of surplus huge pages is controlled by - /proc/sys/vm/nr_overcommit_hugepages. -Hugepagesize is the default hugepage size (in Kb). -Hugetlb is the total amount of memory (in kB), consumed by huge - pages of all sizes. - If huge pages of different sizes are in use, this number - will exceed HugePages_Total * Hugepagesize. To get more - detailed information, please, refer to - /sys/kernel/mm/hugepages (described below). - - -/proc/filesystems should also show a filesystem of type "hugetlbfs" configured -in the kernel. - -/proc/sys/vm/nr_hugepages indicates the current number of "persistent" huge -pages in the kernel's huge page pool. "Persistent" huge pages will be -returned to the huge page pool when freed by a task. A user with root -privileges can dynamically allocate more or free some persistent huge pages -by increasing or decreasing the value of 'nr_hugepages'. - -Pages that are used as huge pages are reserved inside the kernel and cannot -be used for other purposes. Huge pages cannot be swapped out under -memory pressure. - -Once a number of huge pages have been pre-allocated to the kernel huge page -pool, a user with appropriate privilege can use either the mmap system call -or shared memory system calls to use the huge pages. See the discussion of -Using Huge Pages, below. - -The administrator can allocate persistent huge pages on the kernel boot -command line by specifying the "hugepages=N" parameter, where 'N' = the -number of huge pages requested. This is the most reliable method of -allocating huge pages as memory has not yet become fragmented. - -Some platforms support multiple huge page sizes. To allocate huge pages -of a specific size, one must precede the huge pages boot command parameters -with a huge page size selection parameter "hugepagesz=". must -be specified in bytes with optional scale suffix [kKmMgG]. The default huge -page size may be selected with the "default_hugepagesz=" boot parameter. - -When multiple huge page sizes are supported, /proc/sys/vm/nr_hugepages -indicates the current number of pre-allocated huge pages of the default size. -Thus, one can use the following command to dynamically allocate/deallocate -default sized persistent huge pages: - - echo 20 > /proc/sys/vm/nr_hugepages - -This command will try to adjust the number of default sized huge pages in the -huge page pool to 20, allocating or freeing huge pages, as required. - -On a NUMA platform, the kernel will attempt to distribute the huge page pool -over all the set of allowed nodes specified by the NUMA memory policy of the -task that modifies nr_hugepages. The default for the allowed nodes--when the -task has default memory policy--is all on-line nodes with memory. Allowed -nodes with insufficient available, contiguous memory for a huge page will be -silently skipped when allocating persistent huge pages. See the discussion -below of the interaction of task memory policy, cpusets and per node attributes -with the allocation and freeing of persistent huge pages. - -The success or failure of huge page allocation depends on the amount of -physically contiguous memory that is present in system at the time of the -allocation attempt. If the kernel is unable to allocate huge pages from -some nodes in a NUMA system, it will attempt to make up the difference by -allocating extra pages on other nodes with sufficient available contiguous -memory, if any. - -System administrators may want to put this command in one of the local rc -init files. This will enable the kernel to allocate huge pages early in -the boot process when the possibility of getting physical contiguous pages -is still very high. Administrators can verify the number of huge pages -actually allocated by checking the sysctl or meminfo. To check the per node -distribution of huge pages in a NUMA system, use: - - cat /sys/devices/system/node/node*/meminfo | fgrep Huge - -/proc/sys/vm/nr_overcommit_hugepages specifies how large the pool of -huge pages can grow, if more huge pages than /proc/sys/vm/nr_hugepages are -requested by applications. Writing any non-zero value into this file -indicates that the hugetlb subsystem is allowed to try to obtain that -number of "surplus" huge pages from the kernel's normal page pool, when the -persistent huge page pool is exhausted. As these surplus huge pages become -unused, they are freed back to the kernel's normal page pool. - -When increasing the huge page pool size via nr_hugepages, any existing surplus -pages will first be promoted to persistent huge pages. Then, additional -huge pages will be allocated, if necessary and if possible, to fulfill -the new persistent huge page pool size. - -The administrator may shrink the pool of persistent huge pages for -the default huge page size by setting the nr_hugepages sysctl to a -smaller value. The kernel will attempt to balance the freeing of huge pages -across all nodes in the memory policy of the task modifying nr_hugepages. -Any free huge pages on the selected nodes will be freed back to the kernel's -normal page pool. - -Caveat: Shrinking the persistent huge page pool via nr_hugepages such that -it becomes less than the number of huge pages in use will convert the balance -of the in-use huge pages to surplus huge pages. This will occur even if -the number of surplus pages it would exceed the overcommit value. As long as -this condition holds--that is, until nr_hugepages+nr_overcommit_hugepages is -increased sufficiently, or the surplus huge pages go out of use and are freed-- -no more surplus huge pages will be allowed to be allocated. - -With support for multiple huge page pools at run-time available, much of -the huge page userspace interface in /proc/sys/vm has been duplicated in sysfs. -The /proc interfaces discussed above have been retained for backwards -compatibility. The root huge page control directory in sysfs is: - - /sys/kernel/mm/hugepages - -For each huge page size supported by the running kernel, a subdirectory -will exist, of the form: - - hugepages-${size}kB - -Inside each of these directories, the same set of files will exist: - - nr_hugepages - nr_hugepages_mempolicy - nr_overcommit_hugepages - free_hugepages - resv_hugepages - surplus_hugepages - -which function as described above for the default huge page-sized case. - - -Interaction of Task Memory Policy with Huge Page Allocation/Freeing -=================================================================== - -Whether huge pages are allocated and freed via the /proc interface or -the /sysfs interface using the nr_hugepages_mempolicy attribute, the NUMA -nodes from which huge pages are allocated or freed are controlled by the -NUMA memory policy of the task that modifies the nr_hugepages_mempolicy -sysctl or attribute. When the nr_hugepages attribute is used, mempolicy -is ignored. - -The recommended method to allocate or free huge pages to/from the kernel -huge page pool, using the nr_hugepages example above, is: - - numactl --interleave echo 20 \ - >/proc/sys/vm/nr_hugepages_mempolicy - -or, more succinctly: - - numactl -m echo 20 >/proc/sys/vm/nr_hugepages_mempolicy - -This will allocate or free abs(20 - nr_hugepages) to or from the nodes -specified in , depending on whether number of persistent huge pages -is initially less than or greater than 20, respectively. No huge pages will be -allocated nor freed on any node not included in the specified . - -When adjusting the persistent hugepage count via nr_hugepages_mempolicy, any -memory policy mode--bind, preferred, local or interleave--may be used. The -resulting effect on persistent huge page allocation is as follows: - -1) Regardless of mempolicy mode [see Documentation/vm/numa_memory_policy.txt], - persistent huge pages will be distributed across the node or nodes - specified in the mempolicy as if "interleave" had been specified. - However, if a node in the policy does not contain sufficient contiguous - memory for a huge page, the allocation will not "fallback" to the nearest - neighbor node with sufficient contiguous memory. To do this would cause - undesirable imbalance in the distribution of the huge page pool, or - possibly, allocation of persistent huge pages on nodes not allowed by - the task's memory policy. - -2) One or more nodes may be specified with the bind or interleave policy. - If more than one node is specified with the preferred policy, only the - lowest numeric id will be used. Local policy will select the node where - the task is running at the time the nodes_allowed mask is constructed. - For local policy to be deterministic, the task must be bound to a cpu or - cpus in a single node. Otherwise, the task could be migrated to some - other node at any time after launch and the resulting node will be - indeterminate. Thus, local policy is not very useful for this purpose. - Any of the other mempolicy modes may be used to specify a single node. - -3) The nodes allowed mask will be derived from any non-default task mempolicy, - whether this policy was set explicitly by the task itself or one of its - ancestors, such as numactl. This means that if the task is invoked from a - shell with non-default policy, that policy will be used. One can specify a - node list of "all" with numactl --interleave or --membind [-m] to achieve - interleaving over all nodes in the system or cpuset. - -4) Any task mempolicy specified--e.g., using numactl--will be constrained by - the resource limits of any cpuset in which the task runs. Thus, there will - be no way for a task with non-default policy running in a cpuset with a - subset of the system nodes to allocate huge pages outside the cpuset - without first moving to a cpuset that contains all of the desired nodes. - -5) Boot-time huge page allocation attempts to distribute the requested number - of huge pages over all on-lines nodes with memory. - -Per Node Hugepages Attributes -============================= - -A subset of the contents of the root huge page control directory in sysfs, -described above, will be replicated under each the system device of each -NUMA node with memory in: - - /sys/devices/system/node/node[0-9]*/hugepages/ - -Under this directory, the subdirectory for each supported huge page size -contains the following attribute files: - - nr_hugepages - free_hugepages - surplus_hugepages - -The free_' and surplus_' attribute files are read-only. They return the number -of free and surplus [overcommitted] huge pages, respectively, on the parent -node. - -The nr_hugepages attribute returns the total number of huge pages on the -specified node. When this attribute is written, the number of persistent huge -pages on the parent node will be adjusted to the specified value, if sufficient -resources exist, regardless of the task's mempolicy or cpuset constraints. - -Note that the number of overcommit and reserve pages remain global quantities, -as we don't know until fault time, when the faulting task's mempolicy is -applied, from which node the huge page allocation will be attempted. - - -Using Huge Pages -================ - -If the user applications are going to request huge pages using mmap system -call, then it is required that system administrator mount a file system of -type hugetlbfs: - - mount -t hugetlbfs \ - -o uid=,gid=,mode=,pagesize=,size=,\ - min_size=,nr_inodes= none /mnt/huge - -This command mounts a (pseudo) filesystem of type hugetlbfs on the directory -/mnt/huge. Any files created on /mnt/huge uses huge pages. The uid and gid -options sets the owner and group of the root of the file system. By default -the uid and gid of the current process are taken. The mode option sets the -mode of root of file system to value & 01777. This value is given in octal. -By default the value 0755 is picked. If the platform supports multiple huge -page sizes, the pagesize option can be used to specify the huge page size and -associated pool. pagesize is specified in bytes. If pagesize is not specified -the platform's default huge page size and associated pool will be used. The -size option sets the maximum value of memory (huge pages) allowed for that -filesystem (/mnt/huge). The size option can be specified in bytes, or as a -percentage of the specified huge page pool (nr_hugepages). The size is -rounded down to HPAGE_SIZE boundary. The min_size option sets the minimum -value of memory (huge pages) allowed for the filesystem. min_size can be -specified in the same way as size, either bytes or a percentage of the -huge page pool. At mount time, the number of huge pages specified by -min_size are reserved for use by the filesystem. If there are not enough -free huge pages available, the mount will fail. As huge pages are allocated -to the filesystem and freed, the reserve count is adjusted so that the sum -of allocated and reserved huge pages is always at least min_size. The option -nr_inodes sets the maximum number of inodes that /mnt/huge can use. If the -size, min_size or nr_inodes option is not provided on command line then -no limits are set. For pagesize, size, min_size and nr_inodes options, you -can use [G|g]/[M|m]/[K|k] to represent giga/mega/kilo. For example, size=2K -has the same meaning as size=2048. - -While read system calls are supported on files that reside on hugetlb -file systems, write system calls are not. - -Regular chown, chgrp, and chmod commands (with right permissions) could be -used to change the file attributes on hugetlbfs. - -Also, it is important to note that no such mount command is required if -applications are going to use only shmat/shmget system calls or mmap with -MAP_HUGETLB. For an example of how to use mmap with MAP_HUGETLB see map_hugetlb -below. - -Users who wish to use hugetlb memory via shared memory segment should be a -member of a supplementary group and system admin needs to configure that gid -into /proc/sys/vm/hugetlb_shm_group. It is possible for same or different -applications to use any combination of mmaps and shm* calls, though the mount of -filesystem will be required for using mmap calls without MAP_HUGETLB. - -Syscalls that operate on memory backed by hugetlb pages only have their lengths -aligned to the native page size of the processor; they will normally fail with -errno set to EINVAL or exclude hugetlb pages that extend beyond the length if -not hugepage aligned. For example, munmap(2) will fail if memory is backed by -a hugetlb page and the length is smaller than the hugepage size. - - -Examples -======== - -1) map_hugetlb: see tools/testing/selftests/vm/map_hugetlb.c - -2) hugepage-shm: see tools/testing/selftests/vm/hugepage-shm.c - -3) hugepage-mmap: see tools/testing/selftests/vm/hugepage-mmap.c - -4) The libhugetlbfs (https://github.com/libhugetlbfs/libhugetlbfs) library - provides a wide range of userspace tools to help with huge page usability, - environment setup, and control. - -Kernel development regression testing -===================================== - -The most complete set of hugetlb tests are in the libhugetlbfs repository. -If you modify any hugetlb related code, use the libhugetlbfs test suite -to check for regressions. In addition, if you add any new hugetlb -functionality, please add appropriate tests to libhugetlbfs. diff --git a/Documentation/vm/hwpoison.rst b/Documentation/vm/hwpoison.rst new file mode 100644 index 000000000000..09bd24a92784 --- /dev/null +++ b/Documentation/vm/hwpoison.rst @@ -0,0 +1,186 @@ +.. hwpoison: + +======== +hwpoison +======== + +What is hwpoison? +================= + +Upcoming Intel CPUs have support for recovering from some memory errors +(``MCA recovery``). This requires the OS to declare a page "poisoned", +kill the processes associated with it and avoid using it in the future. + +This patchkit implements the necessary infrastructure in the VM. + +To quote the overview comment: + + * High level machine check handler. Handles pages reported by the + * hardware as being corrupted usually due to a 2bit ECC memory or cache + * failure. + * + * This focusses on pages detected as corrupted in the background. + * When the current CPU tries to consume corruption the currently + * running process can just be killed directly instead. This implies + * that if the error cannot be handled for some reason it's safe to + * just ignore it because no corruption has been consumed yet. Instead + * when that happens another machine check will happen. + * + * Handles page cache pages in various states. The tricky part + * here is that we can access any page asynchronous to other VM + * users, because memory failures could happen anytime and anywhere, + * possibly violating some of their assumptions. This is why this code + * has to be extremely careful. Generally it tries to use normal locking + * rules, as in get the standard locks, even if that means the + * error handling takes potentially a long time. + * + * Some of the operations here are somewhat inefficient and have non + * linear algorithmic complexity, because the data structures have not + * been optimized for this case. This is in particular the case + * for the mapping from a vma to a process. Since this case is expected + * to be rare we hope we can get away with this. + +The code consists of a the high level handler in mm/memory-failure.c, +a new page poison bit and various checks in the VM to handle poisoned +pages. + +The main target right now is KVM guests, but it works for all kinds +of applications. KVM support requires a recent qemu-kvm release. + +For the KVM use there was need for a new signal type so that +KVM can inject the machine check into the guest with the proper +address. This in theory allows other applications to handle +memory failures too. The expection is that near all applications +won't do that, but some very specialized ones might. + +Failure recovery modes +====================== + +There are two (actually three) modes memory failure recovery can be in: + +vm.memory_failure_recovery sysctl set to zero: + All memory failures cause a panic. Do not attempt recovery. + (on x86 this can be also affected by the tolerant level of the + MCE subsystem) + +early kill + (can be controlled globally and per process) + Send SIGBUS to the application as soon as the error is detected + This allows applications who can process memory errors in a gentle + way (e.g. drop affected object) + This is the mode used by KVM qemu. + +late kill + Send SIGBUS when the application runs into the corrupted page. + This is best for memory error unaware applications and default + Note some pages are always handled as late kill. + +User control +============ + +vm.memory_failure_recovery + See sysctl.txt + +vm.memory_failure_early_kill + Enable early kill mode globally + +PR_MCE_KILL + Set early/late kill mode/revert to system default + + arg1: PR_MCE_KILL_CLEAR: + Revert to system default + arg1: PR_MCE_KILL_SET: + arg2 defines thread specific mode + + PR_MCE_KILL_EARLY: + Early kill + PR_MCE_KILL_LATE: + Late kill + PR_MCE_KILL_DEFAULT + Use system global default + + Note that if you want to have a dedicated thread which handles + the SIGBUS(BUS_MCEERR_AO) on behalf of the process, you should + call prctl(PR_MCE_KILL_EARLY) on the designated thread. Otherwise, + the SIGBUS is sent to the main thread. + +PR_MCE_KILL_GET + return current mode + +Testing +======= + +* madvise(MADV_HWPOISON, ....) (as root) - Poison a page in the + process for testing + +* hwpoison-inject module through debugfs ``/sys/kernel/debug/hwpoison/`` + + corrupt-pfn + Inject hwpoison fault at PFN echoed into this file. This does + some early filtering to avoid corrupted unintended pages in test suites. + + unpoison-pfn + Software-unpoison page at PFN echoed into this file. This way + a page can be reused again. This only works for Linux + injected failures, not for real memory failures. + + Note these injection interfaces are not stable and might change between + kernel versions + + corrupt-filter-dev-major, corrupt-filter-dev-minor + Only handle memory failures to pages associated with the file + system defined by block device major/minor. -1U is the + wildcard value. This should be only used for testing with + artificial injection. + + corrupt-filter-memcg + Limit injection to pages owned by memgroup. Specified by inode + number of the memcg. + + Example:: + + mkdir /sys/fs/cgroup/mem/hwpoison + + usemem -m 100 -s 1000 & + echo `jobs -p` > /sys/fs/cgroup/mem/hwpoison/tasks + + memcg_ino=$(ls -id /sys/fs/cgroup/mem/hwpoison | cut -f1 -d' ') + echo $memcg_ino > /debug/hwpoison/corrupt-filter-memcg + + page-types -p `pidof init` --hwpoison # shall do nothing + page-types -p `pidof usemem` --hwpoison # poison its pages + + corrupt-filter-flags-mask, corrupt-filter-flags-value + When specified, only poison pages if ((page_flags & mask) == + value). This allows stress testing of many kinds of + pages. The page_flags are the same as in /proc/kpageflags. The + flag bits are defined in include/linux/kernel-page-flags.h and + documented in Documentation/admin-guide/mm/pagemap.rst + +* Architecture specific MCE injector + + x86 has mce-inject, mce-test + + Some portable hwpoison test programs in mce-test, see below. + +References +========== + +http://halobates.de/mce-lc09-2.pdf + Overview presentation from LinuxCon 09 + +git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git + Test suite (hwpoison specific portable tests in tsrc) + +git://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git + x86 specific injector + + +Limitations +=========== +- Not all page types are supported and never will. Most kernel internal + objects cannot be recovered, only LRU pages for now. +- Right now hugepage support is missing. + +--- +Andi Kleen, Oct 2009 diff --git a/Documentation/vm/hwpoison.txt b/Documentation/vm/hwpoison.txt deleted file mode 100644 index e912d7eee769..000000000000 --- a/Documentation/vm/hwpoison.txt +++ /dev/null @@ -1,187 +0,0 @@ -What is hwpoison? - -Upcoming Intel CPUs have support for recovering from some memory errors -(``MCA recovery''). This requires the OS to declare a page "poisoned", -kill the processes associated with it and avoid using it in the future. - -This patchkit implements the necessary infrastructure in the VM. - -To quote the overview comment: - - * High level machine check handler. Handles pages reported by the - * hardware as being corrupted usually due to a 2bit ECC memory or cache - * failure. - * - * This focusses on pages detected as corrupted in the background. - * When the current CPU tries to consume corruption the currently - * running process can just be killed directly instead. This implies - * that if the error cannot be handled for some reason it's safe to - * just ignore it because no corruption has been consumed yet. Instead - * when that happens another machine check will happen. - * - * Handles page cache pages in various states. The tricky part - * here is that we can access any page asynchronous to other VM - * users, because memory failures could happen anytime and anywhere, - * possibly violating some of their assumptions. This is why this code - * has to be extremely careful. Generally it tries to use normal locking - * rules, as in get the standard locks, even if that means the - * error handling takes potentially a long time. - * - * Some of the operations here are somewhat inefficient and have non - * linear algorithmic complexity, because the data structures have not - * been optimized for this case. This is in particular the case - * for the mapping from a vma to a process. Since this case is expected - * to be rare we hope we can get away with this. - -The code consists of a the high level handler in mm/memory-failure.c, -a new page poison bit and various checks in the VM to handle poisoned -pages. - -The main target right now is KVM guests, but it works for all kinds -of applications. KVM support requires a recent qemu-kvm release. - -For the KVM use there was need for a new signal type so that -KVM can inject the machine check into the guest with the proper -address. This in theory allows other applications to handle -memory failures too. The expection is that near all applications -won't do that, but some very specialized ones might. - ---- - -There are two (actually three) modi memory failure recovery can be in: - -vm.memory_failure_recovery sysctl set to zero: - All memory failures cause a panic. Do not attempt recovery. - (on x86 this can be also affected by the tolerant level of the - MCE subsystem) - -early kill - (can be controlled globally and per process) - Send SIGBUS to the application as soon as the error is detected - This allows applications who can process memory errors in a gentle - way (e.g. drop affected object) - This is the mode used by KVM qemu. - -late kill - Send SIGBUS when the application runs into the corrupted page. - This is best for memory error unaware applications and default - Note some pages are always handled as late kill. - ---- - -User control: - -vm.memory_failure_recovery - See sysctl.txt - -vm.memory_failure_early_kill - Enable early kill mode globally - -PR_MCE_KILL - Set early/late kill mode/revert to system default - arg1: PR_MCE_KILL_CLEAR: Revert to system default - arg1: PR_MCE_KILL_SET: arg2 defines thread specific mode - PR_MCE_KILL_EARLY: Early kill - PR_MCE_KILL_LATE: Late kill - PR_MCE_KILL_DEFAULT: Use system global default - Note that if you want to have a dedicated thread which handles - the SIGBUS(BUS_MCEERR_AO) on behalf of the process, you should - call prctl(PR_MCE_KILL_EARLY) on the designated thread. Otherwise, - the SIGBUS is sent to the main thread. - -PR_MCE_KILL_GET - return current mode - - ---- - -Testing: - -madvise(MADV_HWPOISON, ....) - (as root) - Poison a page in the process for testing - - -hwpoison-inject module through debugfs - -/sys/kernel/debug/hwpoison/ - -corrupt-pfn - -Inject hwpoison fault at PFN echoed into this file. This does -some early filtering to avoid corrupted unintended pages in test suites. - -unpoison-pfn - -Software-unpoison page at PFN echoed into this file. This -way a page can be reused again. -This only works for Linux injected failures, not for real -memory failures. - -Note these injection interfaces are not stable and might change between -kernel versions - -corrupt-filter-dev-major -corrupt-filter-dev-minor - -Only handle memory failures to pages associated with the file system defined -by block device major/minor. -1U is the wildcard value. -This should be only used for testing with artificial injection. - -corrupt-filter-memcg - -Limit injection to pages owned by memgroup. Specified by inode number -of the memcg. - -Example: - mkdir /sys/fs/cgroup/mem/hwpoison - - usemem -m 100 -s 1000 & - echo `jobs -p` > /sys/fs/cgroup/mem/hwpoison/tasks - - memcg_ino=$(ls -id /sys/fs/cgroup/mem/hwpoison | cut -f1 -d' ') - echo $memcg_ino > /debug/hwpoison/corrupt-filter-memcg - - page-types -p `pidof init` --hwpoison # shall do nothing - page-types -p `pidof usemem` --hwpoison # poison its pages - -corrupt-filter-flags-mask -corrupt-filter-flags-value - -When specified, only poison pages if ((page_flags & mask) == value). -This allows stress testing of many kinds of pages. The page_flags -are the same as in /proc/kpageflags. The flag bits are defined in -include/linux/kernel-page-flags.h and documented in -Documentation/vm/pagemap.txt - -Architecture specific MCE injector - -x86 has mce-inject, mce-test - -Some portable hwpoison test programs in mce-test, see blow. - ---- - -References: - -http://halobates.de/mce-lc09-2.pdf - Overview presentation from LinuxCon 09 - -git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git - Test suite (hwpoison specific portable tests in tsrc) - -git://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git - x86 specific injector - - ---- - -Limitations: - -- Not all page types are supported and never will. Most kernel internal -objects cannot be recovered, only LRU pages for now. -- Right now hugepage support is missing. - ---- -Andi Kleen, Oct 2009 - diff --git a/Documentation/vm/idle_page_tracking.txt b/Documentation/vm/idle_page_tracking.txt deleted file mode 100644 index 85dcc3bb85dc..000000000000 --- a/Documentation/vm/idle_page_tracking.txt +++ /dev/null @@ -1,98 +0,0 @@ -MOTIVATION - -The idle page tracking feature allows to track which memory pages are being -accessed by a workload and which are idle. This information can be useful for -estimating the workload's working set size, which, in turn, can be taken into -account when configuring the workload parameters, setting memory cgroup limits, -or deciding where to place the workload within a compute cluster. - -It is enabled by CONFIG_IDLE_PAGE_TRACKING=y. - -USER API - -The idle page tracking API is located at /sys/kernel/mm/page_idle. Currently, -it consists of the only read-write file, /sys/kernel/mm/page_idle/bitmap. - -The file implements a bitmap where each bit corresponds to a memory page. The -bitmap is represented by an array of 8-byte integers, and the page at PFN #i is -mapped to bit #i%64 of array element #i/64, byte order is native. When a bit is -set, the corresponding page is idle. - -A page is considered idle if it has not been accessed since it was marked idle -(for more details on what "accessed" actually means see the IMPLEMENTATION -DETAILS section). To mark a page idle one has to set the bit corresponding to -the page by writing to the file. A value written to the file is OR-ed with the -current bitmap value. - -Only accesses to user memory pages are tracked. These are pages mapped to a -process address space, page cache and buffer pages, swap cache pages. For other -page types (e.g. SLAB pages) an attempt to mark a page idle is silently ignored, -and hence such pages are never reported idle. - -For huge pages the idle flag is set only on the head page, so one has to read -/proc/kpageflags in order to correctly count idle huge pages. - -Reading from or writing to /sys/kernel/mm/page_idle/bitmap will return --EINVAL if you are not starting the read/write on an 8-byte boundary, or -if the size of the read/write is not a multiple of 8 bytes. Writing to -this file beyond max PFN will return -ENXIO. - -That said, in order to estimate the amount of pages that are not used by a -workload one should: - - 1. Mark all the workload's pages as idle by setting corresponding bits in - /sys/kernel/mm/page_idle/bitmap. The pages can be found by reading - /proc/pid/pagemap if the workload is represented by a process, or by - filtering out alien pages using /proc/kpagecgroup in case the workload is - placed in a memory cgroup. - - 2. Wait until the workload accesses its working set. - - 3. Read /sys/kernel/mm/page_idle/bitmap and count the number of bits set. If - one wants to ignore certain types of pages, e.g. mlocked pages since they - are not reclaimable, he or she can filter them out using /proc/kpageflags. - -See Documentation/vm/pagemap.txt for more information about /proc/pid/pagemap, -/proc/kpageflags, and /proc/kpagecgroup. - -IMPLEMENTATION DETAILS - -The kernel internally keeps track of accesses to user memory pages in order to -reclaim unreferenced pages first on memory shortage conditions. A page is -considered referenced if it has been recently accessed via a process address -space, in which case one or more PTEs it is mapped to will have the Accessed bit -set, or marked accessed explicitly by the kernel (see mark_page_accessed()). The -latter happens when: - - - a userspace process reads or writes a page using a system call (e.g. read(2) - or write(2)) - - - a page that is used for storing filesystem buffers is read or written, - because a process needs filesystem metadata stored in it (e.g. lists a - directory tree) - - - a page is accessed by a device driver using get_user_pages() - -When a dirty page is written to swap or disk as a result of memory reclaim or -exceeding the dirty memory limit, it is not marked referenced. - -The idle memory tracking feature adds a new page flag, the Idle flag. This flag -is set manually, by writing to /sys/kernel/mm/page_idle/bitmap (see the USER API -section), and cleared automatically whenever a page is referenced as defined -above. - -When a page is marked idle, the Accessed bit must be cleared in all PTEs it is -mapped to, otherwise we will not be able to detect accesses to the page coming -from a process address space. To avoid interference with the reclaimer, which, -as noted above, uses the Accessed bit to promote actively referenced pages, one -more page flag is introduced, the Young flag. When the PTE Accessed bit is -cleared as a result of setting or updating a page's Idle flag, the Young flag -is set on the page. The reclaimer treats the Young flag as an extra PTE -Accessed bit and therefore will consider such a page as referenced. - -Since the idle memory tracking feature is based on the memory reclaimer logic, -it only works with pages that are on an LRU list, other pages are silently -ignored. That means it will ignore a user memory page if it is isolated, but -since there are usually not many of them, it should not affect the overall -result noticeably. In order not to stall scanning of the idle page bitmap, -locked pages may be skipped too. diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst new file mode 100644 index 000000000000..c4ded22197ca --- /dev/null +++ b/Documentation/vm/index.rst @@ -0,0 +1,50 @@ +===================================== +Linux Memory Management Documentation +===================================== + +This is a collection of documents about Linux memory management (mm) subsystem. + +User guides for MM features +=========================== + +The following documents provide guides for controlling and tuning +various features of the Linux memory management + +.. toctree:: + :maxdepth: 1 + + swap_numa + zswap + +Kernel developers MM documentation +================================== + +The below documents describe MM internals with different level of +details ranging from notes and mailing list responses to elaborate +descriptions of data structures and algorithms. + +.. toctree:: + :maxdepth: 1 + + active_mm + balance + cleancache + frontswap + highmem + hmm + hwpoison + hugetlbfs_reserv + ksm + mmu_notifier + numa + overcommit-accounting + page_migration + page_frags + page_owner + remap_file_pages + slub + split_page_table_lock + transhuge + unevictable-lru + z3fold + zsmalloc diff --git a/Documentation/vm/ksm.rst b/Documentation/vm/ksm.rst new file mode 100644 index 000000000000..d32016d9be2c --- /dev/null +++ b/Documentation/vm/ksm.rst @@ -0,0 +1,87 @@ +.. _ksm: + +======================= +Kernel Samepage Merging +======================= + +KSM is a memory-saving de-duplication feature, enabled by CONFIG_KSM=y, +added to the Linux kernel in 2.6.32. See ``mm/ksm.c`` for its implementation, +and http://lwn.net/Articles/306704/ and http://lwn.net/Articles/330589/ + +The userspace interface of KSM is described in :ref:`Documentation/admin-guide/mm/ksm.rst ` + +Design +====== + +Overview +-------- + +.. kernel-doc:: mm/ksm.c + :DOC: Overview + +Reverse mapping +--------------- +KSM maintains reverse mapping information for KSM pages in the stable +tree. + +If a KSM page is shared between less than ``max_page_sharing`` VMAs, +the node of the stable tree that represents such KSM page points to a +list of :c:type:`struct rmap_item` and the ``page->mapping`` of the +KSM page points to the stable tree node. + +When the sharing passes this threshold, KSM adds a second dimension to +the stable tree. The tree node becomes a "chain" that links one or +more "dups". Each "dup" keeps reverse mapping information for a KSM +page with ``page->mapping`` pointing to that "dup". + +Every "chain" and all "dups" linked into a "chain" enforce the +invariant that they represent the same write protected memory content, +even if each "dup" will be pointed by a different KSM page copy of +that content. + +This way the stable tree lookup computational complexity is unaffected +if compared to an unlimited list of reverse mappings. It is still +enforced that there cannot be KSM page content duplicates in the +stable tree itself. + +The deduplication limit enforced by ``max_page_sharing`` is required +to avoid the virtual memory rmap lists to grow too large. The rmap +walk has O(N) complexity where N is the number of rmap_items +(i.e. virtual mappings) that are sharing the page, which is in turn +capped by ``max_page_sharing``. So this effectively spreads the linear +O(N) computational complexity from rmap walk context over different +KSM pages. The ksmd walk over the stable_node "chains" is also O(N), +but N is the number of stable_node "dups", not the number of +rmap_items, so it has not a significant impact on ksmd performance. In +practice the best stable_node "dup" candidate will be kept and found +at the head of the "dups" list. + +High values of ``max_page_sharing`` result in faster memory merging +(because there will be fewer stable_node dups queued into the +stable_node chain->hlist to check for pruning) and higher +deduplication factor at the expense of slower worst case for rmap +walks for any KSM page which can happen during swapping, compaction, +NUMA balancing and page migration. + +The ``stable_node_dups/stable_node_chains`` ratio is also affected by the +``max_page_sharing`` tunable, and an high ratio may indicate fragmentation +in the stable_node dups, which could be solved by introducing +fragmentation algorithms in ksmd which would refile rmap_items from +one stable_node dup to another stable_node dup, in order to free up +stable_node "dups" with few rmap_items in them, but that may increase +the ksmd CPU usage and possibly slowdown the readonly computations on +the KSM pages of the applications. + +The whole list of stable_node "dups" linked in the stable_node +"chains" is scanned periodically in order to prune stale stable_nodes. +The frequency of such scans is defined by +``stable_node_chains_prune_millisecs`` sysfs tunable. + +Reference +--------- +.. kernel-doc:: mm/ksm.c + :functions: mm_slot ksm_scan stable_node rmap_item + +-- +Izik Eidus, +Hugh Dickins, 17 Nov 2009 diff --git a/Documentation/vm/ksm.txt b/Documentation/vm/ksm.txt deleted file mode 100644 index 6686bd267dc9..000000000000 --- a/Documentation/vm/ksm.txt +++ /dev/null @@ -1,178 +0,0 @@ -How to use the Kernel Samepage Merging feature ----------------------------------------------- - -KSM is a memory-saving de-duplication feature, enabled by CONFIG_KSM=y, -added to the Linux kernel in 2.6.32. See mm/ksm.c for its implementation, -and http://lwn.net/Articles/306704/ and http://lwn.net/Articles/330589/ - -The KSM daemon ksmd periodically scans those areas of user memory which -have been registered with it, looking for pages of identical content which -can be replaced by a single write-protected page (which is automatically -copied if a process later wants to update its content). - -KSM was originally developed for use with KVM (where it was known as -Kernel Shared Memory), to fit more virtual machines into physical memory, -by sharing the data common between them. But it can be useful to any -application which generates many instances of the same data. - -KSM only merges anonymous (private) pages, never pagecache (file) pages. -KSM's merged pages were originally locked into kernel memory, but can now -be swapped out just like other user pages (but sharing is broken when they -are swapped back in: ksmd must rediscover their identity and merge again). - -KSM only operates on those areas of address space which an application -has advised to be likely candidates for merging, by using the madvise(2) -system call: int madvise(addr, length, MADV_MERGEABLE). - -The app may call int madvise(addr, length, MADV_UNMERGEABLE) to cancel -that advice and restore unshared pages: whereupon KSM unmerges whatever -it merged in that range. Note: this unmerging call may suddenly require -more memory than is available - possibly failing with EAGAIN, but more -probably arousing the Out-Of-Memory killer. - -If KSM is not configured into the running kernel, madvise MADV_MERGEABLE -and MADV_UNMERGEABLE simply fail with EINVAL. If the running kernel was -built with CONFIG_KSM=y, those calls will normally succeed: even if the -the KSM daemon is not currently running, MADV_MERGEABLE still registers -the range for whenever the KSM daemon is started; even if the range -cannot contain any pages which KSM could actually merge; even if -MADV_UNMERGEABLE is applied to a range which was never MADV_MERGEABLE. - -If a region of memory must be split into at least one new MADV_MERGEABLE -or MADV_UNMERGEABLE region, the madvise may return ENOMEM if the process -will exceed vm.max_map_count (see Documentation/sysctl/vm.txt). - -Like other madvise calls, they are intended for use on mapped areas of -the user address space: they will report ENOMEM if the specified range -includes unmapped gaps (though working on the intervening mapped areas), -and might fail with EAGAIN if not enough memory for internal structures. - -Applications should be considerate in their use of MADV_MERGEABLE, -restricting its use to areas likely to benefit. KSM's scans may use a lot -of processing power: some installations will disable KSM for that reason. - -The KSM daemon is controlled by sysfs files in /sys/kernel/mm/ksm/, -readable by all but writable only by root: - -pages_to_scan - how many present pages to scan before ksmd goes to sleep - e.g. "echo 100 > /sys/kernel/mm/ksm/pages_to_scan" - Default: 100 (chosen for demonstration purposes) - -sleep_millisecs - how many milliseconds ksmd should sleep before next scan - e.g. "echo 20 > /sys/kernel/mm/ksm/sleep_millisecs" - Default: 20 (chosen for demonstration purposes) - -merge_across_nodes - specifies if pages from different numa nodes can be merged. - When set to 0, ksm merges only pages which physically - reside in the memory area of same NUMA node. That brings - lower latency to access of shared pages. Systems with more - nodes, at significant NUMA distances, are likely to benefit - from the lower latency of setting 0. Smaller systems, which - need to minimize memory usage, are likely to benefit from - the greater sharing of setting 1 (default). You may wish to - compare how your system performs under each setting, before - deciding on which to use. merge_across_nodes setting can be - changed only when there are no ksm shared pages in system: - set run 2 to unmerge pages first, then to 1 after changing - merge_across_nodes, to remerge according to the new setting. - Default: 1 (merging across nodes as in earlier releases) - -run - set 0 to stop ksmd from running but keep merged pages, - set 1 to run ksmd e.g. "echo 1 > /sys/kernel/mm/ksm/run", - set 2 to stop ksmd and unmerge all pages currently merged, - but leave mergeable areas registered for next run - Default: 0 (must be changed to 1 to activate KSM, - except if CONFIG_SYSFS is disabled) - -use_zero_pages - specifies whether empty pages (i.e. allocated pages - that only contain zeroes) should be treated specially. - When set to 1, empty pages are merged with the kernel - zero page(s) instead of with each other as it would - happen normally. This can improve the performance on - architectures with coloured zero pages, depending on - the workload. Care should be taken when enabling this - setting, as it can potentially degrade the performance - of KSM for some workloads, for example if the checksums - of pages candidate for merging match the checksum of - an empty page. This setting can be changed at any time, - it is only effective for pages merged after the change. - Default: 0 (normal KSM behaviour as in earlier releases) - -max_page_sharing - Maximum sharing allowed for each KSM page. This - enforces a deduplication limit to avoid the virtual - memory rmap lists to grow too large. The minimum - value is 2 as a newly created KSM page will have at - least two sharers. The rmap walk has O(N) - complexity where N is the number of rmap_items - (i.e. virtual mappings) that are sharing the page, - which is in turn capped by max_page_sharing. So - this effectively spread the the linear O(N) - computational complexity from rmap walk context - over different KSM pages. The ksmd walk over the - stable_node "chains" is also O(N), but N is the - number of stable_node "dups", not the number of - rmap_items, so it has not a significant impact on - ksmd performance. In practice the best stable_node - "dup" candidate will be kept and found at the head - of the "dups" list. The higher this value the - faster KSM will merge the memory (because there - will be fewer stable_node dups queued into the - stable_node chain->hlist to check for pruning) and - the higher the deduplication factor will be, but - the slowest the worst case rmap walk could be for - any given KSM page. Slowing down the rmap_walk - means there will be higher latency for certain - virtual memory operations happening during - swapping, compaction, NUMA balancing and page - migration, in turn decreasing responsiveness for - the caller of those virtual memory operations. The - scheduler latency of other tasks not involved with - the VM operations doing the rmap walk is not - affected by this parameter as the rmap walks are - always schedule friendly themselves. - -stable_node_chains_prune_millisecs - How frequently to walk the whole - list of stable_node "dups" linked in the - stable_node "chains" in order to prune stale - stable_nodes. Smaller milllisecs values will free - up the KSM metadata with lower latency, but they - will make ksmd use more CPU during the scan. This - only applies to the stable_node chains so it's a - noop if not a single KSM page hit the - max_page_sharing yet (there would be no stable_node - chains in such case). - -The effectiveness of KSM and MADV_MERGEABLE is shown in /sys/kernel/mm/ksm/: - -pages_shared - how many shared pages are being used -pages_sharing - how many more sites are sharing them i.e. how much saved -pages_unshared - how many pages unique but repeatedly checked for merging -pages_volatile - how many pages changing too fast to be placed in a tree -full_scans - how many times all mergeable areas have been scanned - -stable_node_chains - number of stable node chains allocated, this is - effectively the number of KSM pages that hit the - max_page_sharing limit -stable_node_dups - number of stable node dups queued into the - stable_node chains - -A high ratio of pages_sharing to pages_shared indicates good sharing, but -a high ratio of pages_unshared to pages_sharing indicates wasted effort. -pages_volatile embraces several different kinds of activity, but a high -proportion there would also indicate poor use of madvise MADV_MERGEABLE. - -The maximum possible page_sharing/page_shared ratio is limited by the -max_page_sharing tunable. To increase the ratio max_page_sharing must -be increased accordingly. - -The stable_node_dups/stable_node_chains ratio is also affected by the -max_page_sharing tunable, and an high ratio may indicate fragmentation -in the stable_node dups, which could be solved by introducing -fragmentation algorithms in ksmd which would refile rmap_items from -one stable_node dup to another stable_node dup, in order to freeup -stable_node "dups" with few rmap_items in them, but that may increase -the ksmd CPU usage and possibly slowdown the readonly computations on -the KSM pages of the applications. - -Izik Eidus, -Hugh Dickins, 17 Nov 2009 diff --git a/Documentation/vm/mmu_notifier.rst b/Documentation/vm/mmu_notifier.rst new file mode 100644 index 000000000000..47baa1cf28c5 --- /dev/null +++ b/Documentation/vm/mmu_notifier.rst @@ -0,0 +1,99 @@ +.. _mmu_notifier: + +When do you need to notify inside page table lock ? +=================================================== + +When clearing a pte/pmd we are given a choice to notify the event through +(notify version of \*_clear_flush call mmu_notifier_invalidate_range) under +the page table lock. But that notification is not necessary in all cases. + +For secondary TLB (non CPU TLB) like IOMMU TLB or device TLB (when device use +thing like ATS/PASID to get the IOMMU to walk the CPU page table to access a +process virtual address space). There is only 2 cases when you need to notify +those secondary TLB while holding page table lock when clearing a pte/pmd: + + A) page backing address is free before mmu_notifier_invalidate_range_end() + B) a page table entry is updated to point to a new page (COW, write fault + on zero page, __replace_page(), ...) + +Case A is obvious you do not want to take the risk for the device to write to +a page that might now be used by some completely different task. + +Case B is more subtle. For correctness it requires the following sequence to +happen: + + - take page table lock + - clear page table entry and notify ([pmd/pte]p_huge_clear_flush_notify()) + - set page table entry to point to new page + +If clearing the page table entry is not followed by a notify before setting +the new pte/pmd value then you can break memory model like C11 or C++11 for +the device. + +Consider the following scenario (device use a feature similar to ATS/PASID): + +Two address addrA and addrB such that \|addrA - addrB\| >= PAGE_SIZE we assume +they are write protected for COW (other case of B apply too). + +:: + + [Time N] -------------------------------------------------------------------- + CPU-thread-0 {try to write to addrA} + CPU-thread-1 {try to write to addrB} + CPU-thread-2 {} + CPU-thread-3 {} + DEV-thread-0 {read addrA and populate device TLB} + DEV-thread-2 {read addrB and populate device TLB} + [Time N+1] ------------------------------------------------------------------ + CPU-thread-0 {COW_step0: {mmu_notifier_invalidate_range_start(addrA)}} + CPU-thread-1 {COW_step0: {mmu_notifier_invalidate_range_start(addrB)}} + CPU-thread-2 {} + CPU-thread-3 {} + DEV-thread-0 {} + DEV-thread-2 {} + [Time N+2] ------------------------------------------------------------------ + CPU-thread-0 {COW_step1: {update page table to point to new page for addrA}} + CPU-thread-1 {COW_step1: {update page table to point to new page for addrB}} + CPU-thread-2 {} + CPU-thread-3 {} + DEV-thread-0 {} + DEV-thread-2 {} + [Time N+3] ------------------------------------------------------------------ + CPU-thread-0 {preempted} + CPU-thread-1 {preempted} + CPU-thread-2 {write to addrA which is a write to new page} + CPU-thread-3 {} + DEV-thread-0 {} + DEV-thread-2 {} + [Time N+3] ------------------------------------------------------------------ + CPU-thread-0 {preempted} + CPU-thread-1 {preempted} + CPU-thread-2 {} + CPU-thread-3 {write to addrB which is a write to new page} + DEV-thread-0 {} + DEV-thread-2 {} + [Time N+4] ------------------------------------------------------------------ + CPU-thread-0 {preempted} + CPU-thread-1 {COW_step3: {mmu_notifier_invalidate_range_end(addrB)}} + CPU-thread-2 {} + CPU-thread-3 {} + DEV-thread-0 {} + DEV-thread-2 {} + [Time N+5] ------------------------------------------------------------------ + CPU-thread-0 {preempted} + CPU-thread-1 {} + CPU-thread-2 {} + CPU-thread-3 {} + DEV-thread-0 {read addrA from old page} + DEV-thread-2 {read addrB from new page} + +So here because at time N+2 the clear page table entry was not pair with a +notification to invalidate the secondary TLB, the device see the new value for +addrB before seing the new value for addrA. This break total memory ordering +for the device. + +When changing a pte to write protect or to point to a new write protected page +with same content (KSM) it is fine to delay the mmu_notifier_invalidate_range +call to mmu_notifier_invalidate_range_end() outside the page table lock. This +is true even if the thread doing the page table update is preempted right after +releasing page table lock but before call mmu_notifier_invalidate_range_end(). diff --git a/Documentation/vm/mmu_notifier.txt b/Documentation/vm/mmu_notifier.txt deleted file mode 100644 index 23b462566bb7..000000000000 --- a/Documentation/vm/mmu_notifier.txt +++ /dev/null @@ -1,93 +0,0 @@ -When do you need to notify inside page table lock ? - -When clearing a pte/pmd we are given a choice to notify the event through -(notify version of *_clear_flush call mmu_notifier_invalidate_range) under -the page table lock. But that notification is not necessary in all cases. - -For secondary TLB (non CPU TLB) like IOMMU TLB or device TLB (when device use -thing like ATS/PASID to get the IOMMU to walk the CPU page table to access a -process virtual address space). There is only 2 cases when you need to notify -those secondary TLB while holding page table lock when clearing a pte/pmd: - - A) page backing address is free before mmu_notifier_invalidate_range_end() - B) a page table entry is updated to point to a new page (COW, write fault - on zero page, __replace_page(), ...) - -Case A is obvious you do not want to take the risk for the device to write to -a page that might now be used by some completely different task. - -Case B is more subtle. For correctness it requires the following sequence to -happen: - - take page table lock - - clear page table entry and notify ([pmd/pte]p_huge_clear_flush_notify()) - - set page table entry to point to new page - -If clearing the page table entry is not followed by a notify before setting -the new pte/pmd value then you can break memory model like C11 or C++11 for -the device. - -Consider the following scenario (device use a feature similar to ATS/PASID): - -Two address addrA and addrB such that |addrA - addrB| >= PAGE_SIZE we assume -they are write protected for COW (other case of B apply too). - -[Time N] -------------------------------------------------------------------- -CPU-thread-0 {try to write to addrA} -CPU-thread-1 {try to write to addrB} -CPU-thread-2 {} -CPU-thread-3 {} -DEV-thread-0 {read addrA and populate device TLB} -DEV-thread-2 {read addrB and populate device TLB} -[Time N+1] ------------------------------------------------------------------ -CPU-thread-0 {COW_step0: {mmu_notifier_invalidate_range_start(addrA)}} -CPU-thread-1 {COW_step0: {mmu_notifier_invalidate_range_start(addrB)}} -CPU-thread-2 {} -CPU-thread-3 {} -DEV-thread-0 {} -DEV-thread-2 {} -[Time N+2] ------------------------------------------------------------------ -CPU-thread-0 {COW_step1: {update page table to point to new page for addrA}} -CPU-thread-1 {COW_step1: {update page table to point to new page for addrB}} -CPU-thread-2 {} -CPU-thread-3 {} -DEV-thread-0 {} -DEV-thread-2 {} -[Time N+3] ------------------------------------------------------------------ -CPU-thread-0 {preempted} -CPU-thread-1 {preempted} -CPU-thread-2 {write to addrA which is a write to new page} -CPU-thread-3 {} -DEV-thread-0 {} -DEV-thread-2 {} -[Time N+3] ------------------------------------------------------------------ -CPU-thread-0 {preempted} -CPU-thread-1 {preempted} -CPU-thread-2 {} -CPU-thread-3 {write to addrB which is a write to new page} -DEV-thread-0 {} -DEV-thread-2 {} -[Time N+4] ------------------------------------------------------------------ -CPU-thread-0 {preempted} -CPU-thread-1 {COW_step3: {mmu_notifier_invalidate_range_end(addrB)}} -CPU-thread-2 {} -CPU-thread-3 {} -DEV-thread-0 {} -DEV-thread-2 {} -[Time N+5] ------------------------------------------------------------------ -CPU-thread-0 {preempted} -CPU-thread-1 {} -CPU-thread-2 {} -CPU-thread-3 {} -DEV-thread-0 {read addrA from old page} -DEV-thread-2 {read addrB from new page} - -So here because at time N+2 the clear page table entry was not pair with a -notification to invalidate the secondary TLB, the device see the new value for -addrB before seing the new value for addrA. This break total memory ordering -for the device. - -When changing a pte to write protect or to point to a new write protected page -with same content (KSM) it is fine to delay the mmu_notifier_invalidate_range -call to mmu_notifier_invalidate_range_end() outside the page table lock. This -is true even if the thread doing the page table update is preempted right after -releasing page table lock but before call mmu_notifier_invalidate_range_end(). diff --git a/Documentation/vm/numa b/Documentation/vm/numa deleted file mode 100644 index a31b85b9bb88..000000000000 --- a/Documentation/vm/numa +++ /dev/null @@ -1,146 +0,0 @@ -Started Nov 1999 by Kanoj Sarcar - -What is NUMA? - -This question can be answered from a couple of perspectives: the -hardware view and the Linux software view. - -From the hardware perspective, a NUMA system is a computer platform that -comprises multiple components or assemblies each of which may contain 0 -or more CPUs, local memory, and/or IO buses. For brevity and to -disambiguate the hardware view of these physical components/assemblies -from the software abstraction thereof, we'll call the components/assemblies -'cells' in this document. - -Each of the 'cells' may be viewed as an SMP [symmetric multi-processor] subset -of the system--although some components necessary for a stand-alone SMP system -may not be populated on any given cell. The cells of the NUMA system are -connected together with some sort of system interconnect--e.g., a crossbar or -point-to-point link are common types of NUMA system interconnects. Both of -these types of interconnects can be aggregated to create NUMA platforms with -cells at multiple distances from other cells. - -For Linux, the NUMA platforms of interest are primarily what is known as Cache -Coherent NUMA or ccNUMA systems. With ccNUMA systems, all memory is visible -to and accessible from any CPU attached to any cell and cache coherency -is handled in hardware by the processor caches and/or the system interconnect. - -Memory access time and effective memory bandwidth varies depending on how far -away the cell containing the CPU or IO bus making the memory access is from the -cell containing the target memory. For example, access to memory by CPUs -attached to the same cell will experience faster access times and higher -bandwidths than accesses to memory on other, remote cells. NUMA platforms -can have cells at multiple remote distances from any given cell. - -Platform vendors don't build NUMA systems just to make software developers' -lives interesting. Rather, this architecture is a means to provide scalable -memory bandwidth. However, to achieve scalable memory bandwidth, system and -application software must arrange for a large majority of the memory references -[cache misses] to be to "local" memory--memory on the same cell, if any--or -to the closest cell with memory. - -This leads to the Linux software view of a NUMA system: - -Linux divides the system's hardware resources into multiple software -abstractions called "nodes". Linux maps the nodes onto the physical cells -of the hardware platform, abstracting away some of the details for some -architectures. As with physical cells, software nodes may contain 0 or more -CPUs, memory and/or IO buses. And, again, memory accesses to memory on -"closer" nodes--nodes that map to closer cells--will generally experience -faster access times and higher effective bandwidth than accesses to more -remote cells. - -For some architectures, such as x86, Linux will "hide" any node representing a -physical cell that has no memory attached, and reassign any CPUs attached to -that cell to a node representing a cell that does have memory. Thus, on -these architectures, one cannot assume that all CPUs that Linux associates with -a given node will see the same local memory access times and bandwidth. - -In addition, for some architectures, again x86 is an example, Linux supports -the emulation of additional nodes. For NUMA emulation, linux will carve up -the existing nodes--or the system memory for non-NUMA platforms--into multiple -nodes. Each emulated node will manage a fraction of the underlying cells' -physical memory. NUMA emluation is useful for testing NUMA kernel and -application features on non-NUMA platforms, and as a sort of memory resource -management mechanism when used together with cpusets. -[see Documentation/cgroup-v1/cpusets.txt] - -For each node with memory, Linux constructs an independent memory management -subsystem, complete with its own free page lists, in-use page lists, usage -statistics and locks to mediate access. In addition, Linux constructs for -each memory zone [one or more of DMA, DMA32, NORMAL, HIGH_MEMORY, MOVABLE], -an ordered "zonelist". A zonelist specifies the zones/nodes to visit when a -selected zone/node cannot satisfy the allocation request. This situation, -when a zone has no available memory to satisfy a request, is called -"overflow" or "fallback". - -Because some nodes contain multiple zones containing different types of -memory, Linux must decide whether to order the zonelists such that allocations -fall back to the same zone type on a different node, or to a different zone -type on the same node. This is an important consideration because some zones, -such as DMA or DMA32, represent relatively scarce resources. Linux chooses -a default Node ordered zonelist. This means it tries to fallback to other zones -from the same node before using remote nodes which are ordered by NUMA distance. - -By default, Linux will attempt to satisfy memory allocation requests from the -node to which the CPU that executes the request is assigned. Specifically, -Linux will attempt to allocate from the first node in the appropriate zonelist -for the node where the request originates. This is called "local allocation." -If the "local" node cannot satisfy the request, the kernel will examine other -nodes' zones in the selected zonelist looking for the first zone in the list -that can satisfy the request. - -Local allocation will tend to keep subsequent access to the allocated memory -"local" to the underlying physical resources and off the system interconnect-- -as long as the task on whose behalf the kernel allocated some memory does not -later migrate away from that memory. The Linux scheduler is aware of the -NUMA topology of the platform--embodied in the "scheduling domains" data -structures [see Documentation/scheduler/sched-domains.txt]--and the scheduler -attempts to minimize task migration to distant scheduling domains. However, -the scheduler does not take a task's NUMA footprint into account directly. -Thus, under sufficient imbalance, tasks can migrate between nodes, remote -from their initial node and kernel data structures. - -System administrators and application designers can restrict a task's migration -to improve NUMA locality using various CPU affinity command line interfaces, -such as taskset(1) and numactl(1), and program interfaces such as -sched_setaffinity(2). Further, one can modify the kernel's default local -allocation behavior using Linux NUMA memory policy. -[see Documentation/vm/numa_memory_policy.txt.] - -System administrators can restrict the CPUs and nodes' memories that a non- -privileged user can specify in the scheduling or NUMA commands and functions -using control groups and CPUsets. [see Documentation/cgroup-v1/cpusets.txt] - -On architectures that do not hide memoryless nodes, Linux will include only -zones [nodes] with memory in the zonelists. This means that for a memoryless -node the "local memory node"--the node of the first zone in CPU's node's -zonelist--will not be the node itself. Rather, it will be the node that the -kernel selected as the nearest node with memory when it built the zonelists. -So, default, local allocations will succeed with the kernel supplying the -closest available memory. This is a consequence of the same mechanism that -allows such allocations to fallback to other nearby nodes when a node that -does contain memory overflows. - -Some kernel allocations do not want or cannot tolerate this allocation fallback -behavior. Rather they want to be sure they get memory from the specified node -or get notified that the node has no free memory. This is usually the case when -a subsystem allocates per CPU memory resources, for example. - -A typical model for making such an allocation is to obtain the node id of the -node to which the "current CPU" is attached using one of the kernel's -numa_node_id() or CPU_to_node() functions and then request memory from only -the node id returned. When such an allocation fails, the requesting subsystem -may revert to its own fallback path. The slab kernel memory allocator is an -example of this. Or, the subsystem may choose to disable or not to enable -itself on allocation failure. The kernel profiling subsystem is an example of -this. - -If the architecture supports--does not hide--memoryless nodes, then CPUs -attached to memoryless nodes would always incur the fallback path overhead -or some subsystems would fail to initialize if they attempted to allocated -memory exclusively from a node without memory. To support such -architectures transparently, kernel subsystems can use the numa_mem_id() -or cpu_to_mem() function to locate the "local memory node" for the calling or -specified CPU. Again, this is the same node from which default, local page -allocations will be attempted. diff --git a/Documentation/vm/numa.rst b/Documentation/vm/numa.rst new file mode 100644 index 000000000000..185d8a568168 --- /dev/null +++ b/Documentation/vm/numa.rst @@ -0,0 +1,150 @@ +.. _numa: + +Started Nov 1999 by Kanoj Sarcar + +============= +What is NUMA? +============= + +This question can be answered from a couple of perspectives: the +hardware view and the Linux software view. + +From the hardware perspective, a NUMA system is a computer platform that +comprises multiple components or assemblies each of which may contain 0 +or more CPUs, local memory, and/or IO buses. For brevity and to +disambiguate the hardware view of these physical components/assemblies +from the software abstraction thereof, we'll call the components/assemblies +'cells' in this document. + +Each of the 'cells' may be viewed as an SMP [symmetric multi-processor] subset +of the system--although some components necessary for a stand-alone SMP system +may not be populated on any given cell. The cells of the NUMA system are +connected together with some sort of system interconnect--e.g., a crossbar or +point-to-point link are common types of NUMA system interconnects. Both of +these types of interconnects can be aggregated to create NUMA platforms with +cells at multiple distances from other cells. + +For Linux, the NUMA platforms of interest are primarily what is known as Cache +Coherent NUMA or ccNUMA systems. With ccNUMA systems, all memory is visible +to and accessible from any CPU attached to any cell and cache coherency +is handled in hardware by the processor caches and/or the system interconnect. + +Memory access time and effective memory bandwidth varies depending on how far +away the cell containing the CPU or IO bus making the memory access is from the +cell containing the target memory. For example, access to memory by CPUs +attached to the same cell will experience faster access times and higher +bandwidths than accesses to memory on other, remote cells. NUMA platforms +can have cells at multiple remote distances from any given cell. + +Platform vendors don't build NUMA systems just to make software developers' +lives interesting. Rather, this architecture is a means to provide scalable +memory bandwidth. However, to achieve scalable memory bandwidth, system and +application software must arrange for a large majority of the memory references +[cache misses] to be to "local" memory--memory on the same cell, if any--or +to the closest cell with memory. + +This leads to the Linux software view of a NUMA system: + +Linux divides the system's hardware resources into multiple software +abstractions called "nodes". Linux maps the nodes onto the physical cells +of the hardware platform, abstracting away some of the details for some +architectures. As with physical cells, software nodes may contain 0 or more +CPUs, memory and/or IO buses. And, again, memory accesses to memory on +"closer" nodes--nodes that map to closer cells--will generally experience +faster access times and higher effective bandwidth than accesses to more +remote cells. + +For some architectures, such as x86, Linux will "hide" any node representing a +physical cell that has no memory attached, and reassign any CPUs attached to +that cell to a node representing a cell that does have memory. Thus, on +these architectures, one cannot assume that all CPUs that Linux associates with +a given node will see the same local memory access times and bandwidth. + +In addition, for some architectures, again x86 is an example, Linux supports +the emulation of additional nodes. For NUMA emulation, linux will carve up +the existing nodes--or the system memory for non-NUMA platforms--into multiple +nodes. Each emulated node will manage a fraction of the underlying cells' +physical memory. NUMA emluation is useful for testing NUMA kernel and +application features on non-NUMA platforms, and as a sort of memory resource +management mechanism when used together with cpusets. +[see Documentation/cgroup-v1/cpusets.txt] + +For each node with memory, Linux constructs an independent memory management +subsystem, complete with its own free page lists, in-use page lists, usage +statistics and locks to mediate access. In addition, Linux constructs for +each memory zone [one or more of DMA, DMA32, NORMAL, HIGH_MEMORY, MOVABLE], +an ordered "zonelist". A zonelist specifies the zones/nodes to visit when a +selected zone/node cannot satisfy the allocation request. This situation, +when a zone has no available memory to satisfy a request, is called +"overflow" or "fallback". + +Because some nodes contain multiple zones containing different types of +memory, Linux must decide whether to order the zonelists such that allocations +fall back to the same zone type on a different node, or to a different zone +type on the same node. This is an important consideration because some zones, +such as DMA or DMA32, represent relatively scarce resources. Linux chooses +a default Node ordered zonelist. This means it tries to fallback to other zones +from the same node before using remote nodes which are ordered by NUMA distance. + +By default, Linux will attempt to satisfy memory allocation requests from the +node to which the CPU that executes the request is assigned. Specifically, +Linux will attempt to allocate from the first node in the appropriate zonelist +for the node where the request originates. This is called "local allocation." +If the "local" node cannot satisfy the request, the kernel will examine other +nodes' zones in the selected zonelist looking for the first zone in the list +that can satisfy the request. + +Local allocation will tend to keep subsequent access to the allocated memory +"local" to the underlying physical resources and off the system interconnect-- +as long as the task on whose behalf the kernel allocated some memory does not +later migrate away from that memory. The Linux scheduler is aware of the +NUMA topology of the platform--embodied in the "scheduling domains" data +structures [see Documentation/scheduler/sched-domains.txt]--and the scheduler +attempts to minimize task migration to distant scheduling domains. However, +the scheduler does not take a task's NUMA footprint into account directly. +Thus, under sufficient imbalance, tasks can migrate between nodes, remote +from their initial node and kernel data structures. + +System administrators and application designers can restrict a task's migration +to improve NUMA locality using various CPU affinity command line interfaces, +such as taskset(1) and numactl(1), and program interfaces such as +sched_setaffinity(2). Further, one can modify the kernel's default local +allocation behavior using Linux NUMA memory policy. +[see Documentation/admin-guide/mm/numa_memory_policy.rst.] + +System administrators can restrict the CPUs and nodes' memories that a non- +privileged user can specify in the scheduling or NUMA commands and functions +using control groups and CPUsets. [see Documentation/cgroup-v1/cpusets.txt] + +On architectures that do not hide memoryless nodes, Linux will include only +zones [nodes] with memory in the zonelists. This means that for a memoryless +node the "local memory node"--the node of the first zone in CPU's node's +zonelist--will not be the node itself. Rather, it will be the node that the +kernel selected as the nearest node with memory when it built the zonelists. +So, default, local allocations will succeed with the kernel supplying the +closest available memory. This is a consequence of the same mechanism that +allows such allocations to fallback to other nearby nodes when a node that +does contain memory overflows. + +Some kernel allocations do not want or cannot tolerate this allocation fallback +behavior. Rather they want to be sure they get memory from the specified node +or get notified that the node has no free memory. This is usually the case when +a subsystem allocates per CPU memory resources, for example. + +A typical model for making such an allocation is to obtain the node id of the +node to which the "current CPU" is attached using one of the kernel's +numa_node_id() or CPU_to_node() functions and then request memory from only +the node id returned. When such an allocation fails, the requesting subsystem +may revert to its own fallback path. The slab kernel memory allocator is an +example of this. Or, the subsystem may choose to disable or not to enable +itself on allocation failure. The kernel profiling subsystem is an example of +this. + +If the architecture supports--does not hide--memoryless nodes, then CPUs +attached to memoryless nodes would always incur the fallback path overhead +or some subsystems would fail to initialize if they attempted to allocated +memory exclusively from a node without memory. To support such +architectures transparently, kernel subsystems can use the numa_mem_id() +or cpu_to_mem() function to locate the "local memory node" for the calling or +specified CPU. Again, this is the same node from which default, local page +allocations will be attempted. diff --git a/Documentation/vm/numa_memory_policy.txt b/Documentation/vm/numa_memory_policy.txt deleted file mode 100644 index 622b927816e7..000000000000 --- a/Documentation/vm/numa_memory_policy.txt +++ /dev/null @@ -1,452 +0,0 @@ - -What is Linux Memory Policy? - -In the Linux kernel, "memory policy" determines from which node the kernel will -allocate memory in a NUMA system or in an emulated NUMA system. Linux has -supported platforms with Non-Uniform Memory Access architectures since 2.4.?. -The current memory policy support was added to Linux 2.6 around May 2004. This -document attempts to describe the concepts and APIs of the 2.6 memory policy -support. - -Memory policies should not be confused with cpusets -(Documentation/cgroup-v1/cpusets.txt) -which is an administrative mechanism for restricting the nodes from which -memory may be allocated by a set of processes. Memory policies are a -programming interface that a NUMA-aware application can take advantage of. When -both cpusets and policies are applied to a task, the restrictions of the cpuset -takes priority. See "MEMORY POLICIES AND CPUSETS" below for more details. - -MEMORY POLICY CONCEPTS - -Scope of Memory Policies - -The Linux kernel supports _scopes_ of memory policy, described here from -most general to most specific: - - System Default Policy: this policy is "hard coded" into the kernel. It - is the policy that governs all page allocations that aren't controlled - by one of the more specific policy scopes discussed below. When the - system is "up and running", the system default policy will use "local - allocation" described below. However, during boot up, the system - default policy will be set to interleave allocations across all nodes - with "sufficient" memory, so as not to overload the initial boot node - with boot-time allocations. - - Task/Process Policy: this is an optional, per-task policy. When defined - for a specific task, this policy controls all page allocations made by or - on behalf of the task that aren't controlled by a more specific scope. - If a task does not define a task policy, then all page allocations that - would have been controlled by the task policy "fall back" to the System - Default Policy. - - The task policy applies to the entire address space of a task. Thus, - it is inheritable, and indeed is inherited, across both fork() - [clone() w/o the CLONE_VM flag] and exec*(). This allows a parent task - to establish the task policy for a child task exec()'d from an - executable image that has no awareness of memory policy. See the - MEMORY POLICY APIS section, below, for an overview of the system call - that a task may use to set/change its task/process policy. - - In a multi-threaded task, task policies apply only to the thread - [Linux kernel task] that installs the policy and any threads - subsequently created by that thread. Any sibling threads existing - at the time a new task policy is installed retain their current - policy. - - A task policy applies only to pages allocated after the policy is - installed. Any pages already faulted in by the task when the task - changes its task policy remain where they were allocated based on - the policy at the time they were allocated. - - VMA Policy: A "VMA" or "Virtual Memory Area" refers to a range of a task's - virtual address space. A task may define a specific policy for a range - of its virtual address space. See the MEMORY POLICIES APIS section, - below, for an overview of the mbind() system call used to set a VMA - policy. - - A VMA policy will govern the allocation of pages that back this region of - the address space. Any regions of the task's address space that don't - have an explicit VMA policy will fall back to the task policy, which may - itself fall back to the System Default Policy. - - VMA policies have a few complicating details: - - VMA policy applies ONLY to anonymous pages. These include pages - allocated for anonymous segments, such as the task stack and heap, and - any regions of the address space mmap()ed with the MAP_ANONYMOUS flag. - If a VMA policy is applied to a file mapping, it will be ignored if - the mapping used the MAP_SHARED flag. If the file mapping used the - MAP_PRIVATE flag, the VMA policy will only be applied when an - anonymous page is allocated on an attempt to write to the mapping-- - i.e., at Copy-On-Write. - - VMA policies are shared between all tasks that share a virtual address - space--a.k.a. threads--independent of when the policy is installed; and - they are inherited across fork(). However, because VMA policies refer - to a specific region of a task's address space, and because the address - space is discarded and recreated on exec*(), VMA policies are NOT - inheritable across exec(). Thus, only NUMA-aware applications may - use VMA policies. - - A task may install a new VMA policy on a sub-range of a previously - mmap()ed region. When this happens, Linux splits the existing virtual - memory area into 2 or 3 VMAs, each with it's own policy. - - By default, VMA policy applies only to pages allocated after the policy - is installed. Any pages already faulted into the VMA range remain - where they were allocated based on the policy at the time they were - allocated. However, since 2.6.16, Linux supports page migration via - the mbind() system call, so that page contents can be moved to match - a newly installed policy. - - Shared Policy: Conceptually, shared policies apply to "memory objects" - mapped shared into one or more tasks' distinct address spaces. An - application installs a shared policies the same way as VMA policies--using - the mbind() system call specifying a range of virtual addresses that map - the shared object. However, unlike VMA policies, which can be considered - to be an attribute of a range of a task's address space, shared policies - apply directly to the shared object. Thus, all tasks that attach to the - object share the policy, and all pages allocated for the shared object, - by any task, will obey the shared policy. - - As of 2.6.22, only shared memory segments, created by shmget() or - mmap(MAP_ANONYMOUS|MAP_SHARED), support shared policy. When shared - policy support was added to Linux, the associated data structures were - added to hugetlbfs shmem segments. At the time, hugetlbfs did not - support allocation at fault time--a.k.a lazy allocation--so hugetlbfs - shmem segments were never "hooked up" to the shared policy support. - Although hugetlbfs segments now support lazy allocation, their support - for shared policy has not been completed. - - As mentioned above [re: VMA policies], allocations of page cache - pages for regular files mmap()ed with MAP_SHARED ignore any VMA - policy installed on the virtual address range backed by the shared - file mapping. Rather, shared page cache pages, including pages backing - private mappings that have not yet been written by the task, follow - task policy, if any, else System Default Policy. - - The shared policy infrastructure supports different policies on subset - ranges of the shared object. However, Linux still splits the VMA of - the task that installs the policy for each range of distinct policy. - Thus, different tasks that attach to a shared memory segment can have - different VMA configurations mapping that one shared object. This - can be seen by examining the /proc//numa_maps of tasks sharing - a shared memory region, when one task has installed shared policy on - one or more ranges of the region. - -Components of Memory Policies - - A Linux memory policy consists of a "mode", optional mode flags, and an - optional set of nodes. The mode determines the behavior of the policy, - the optional mode flags determine the behavior of the mode, and the - optional set of nodes can be viewed as the arguments to the policy - behavior. - - Internally, memory policies are implemented by a reference counted - structure, struct mempolicy. Details of this structure will be discussed - in context, below, as required to explain the behavior. - - Linux memory policy supports the following 4 behavioral modes: - - Default Mode--MPOL_DEFAULT: This mode is only used in the memory - policy APIs. Internally, MPOL_DEFAULT is converted to the NULL - memory policy in all policy scopes. Any existing non-default policy - will simply be removed when MPOL_DEFAULT is specified. As a result, - MPOL_DEFAULT means "fall back to the next most specific policy scope." - - For example, a NULL or default task policy will fall back to the - system default policy. A NULL or default vma policy will fall - back to the task policy. - - When specified in one of the memory policy APIs, the Default mode - does not use the optional set of nodes. - - It is an error for the set of nodes specified for this policy to - be non-empty. - - MPOL_BIND: This mode specifies that memory must come from the - set of nodes specified by the policy. Memory will be allocated from - the node in the set with sufficient free memory that is closest to - the node where the allocation takes place. - - MPOL_PREFERRED: This mode specifies that the allocation should be - attempted from the single node specified in the policy. If that - allocation fails, the kernel will search other nodes, in order of - increasing distance from the preferred node based on information - provided by the platform firmware. - - Internally, the Preferred policy uses a single node--the - preferred_node member of struct mempolicy. When the internal - mode flag MPOL_F_LOCAL is set, the preferred_node is ignored and - the policy is interpreted as local allocation. "Local" allocation - policy can be viewed as a Preferred policy that starts at the node - containing the cpu where the allocation takes place. - - It is possible for the user to specify that local allocation is - always preferred by passing an empty nodemask with this mode. - If an empty nodemask is passed, the policy cannot use the - MPOL_F_STATIC_NODES or MPOL_F_RELATIVE_NODES flags described - below. - - MPOL_INTERLEAVED: This mode specifies that page allocations be - interleaved, on a page granularity, across the nodes specified in - the policy. This mode also behaves slightly differently, based on - the context where it is used: - - For allocation of anonymous pages and shared memory pages, - Interleave mode indexes the set of nodes specified by the policy - using the page offset of the faulting address into the segment - [VMA] containing the address modulo the number of nodes specified - by the policy. It then attempts to allocate a page, starting at - the selected node, as if the node had been specified by a Preferred - policy or had been selected by a local allocation. That is, - allocation will follow the per node zonelist. - - For allocation of page cache pages, Interleave mode indexes the set - of nodes specified by the policy using a node counter maintained - per task. This counter wraps around to the lowest specified node - after it reaches the highest specified node. This will tend to - spread the pages out over the nodes specified by the policy based - on the order in which they are allocated, rather than based on any - page offset into an address range or file. During system boot up, - the temporary interleaved system default policy works in this - mode. - - Linux memory policy supports the following optional mode flags: - - MPOL_F_STATIC_NODES: This flag specifies that the nodemask passed by - the user should not be remapped if the task or VMA's set of allowed - nodes changes after the memory policy has been defined. - - Without this flag, anytime a mempolicy is rebound because of a - change in the set of allowed nodes, the node (Preferred) or - nodemask (Bind, Interleave) is remapped to the new set of - allowed nodes. This may result in nodes being used that were - previously undesired. - - With this flag, if the user-specified nodes overlap with the - nodes allowed by the task's cpuset, then the memory policy is - applied to their intersection. If the two sets of nodes do not - overlap, the Default policy is used. - - For example, consider a task that is attached to a cpuset with - mems 1-3 that sets an Interleave policy over the same set. If - the cpuset's mems change to 3-5, the Interleave will now occur - over nodes 3, 4, and 5. With this flag, however, since only node - 3 is allowed from the user's nodemask, the "interleave" only - occurs over that node. If no nodes from the user's nodemask are - now allowed, the Default behavior is used. - - MPOL_F_STATIC_NODES cannot be combined with the - MPOL_F_RELATIVE_NODES flag. It also cannot be used for - MPOL_PREFERRED policies that were created with an empty nodemask - (local allocation). - - MPOL_F_RELATIVE_NODES: This flag specifies that the nodemask passed - by the user will be mapped relative to the set of the task or VMA's - set of allowed nodes. The kernel stores the user-passed nodemask, - and if the allowed nodes changes, then that original nodemask will - be remapped relative to the new set of allowed nodes. - - Without this flag (and without MPOL_F_STATIC_NODES), anytime a - mempolicy is rebound because of a change in the set of allowed - nodes, the node (Preferred) or nodemask (Bind, Interleave) is - remapped to the new set of allowed nodes. That remap may not - preserve the relative nature of the user's passed nodemask to its - set of allowed nodes upon successive rebinds: a nodemask of - 1,3,5 may be remapped to 7-9 and then to 1-3 if the set of - allowed nodes is restored to its original state. - - With this flag, the remap is done so that the node numbers from - the user's passed nodemask are relative to the set of allowed - nodes. In other words, if nodes 0, 2, and 4 are set in the user's - nodemask, the policy will be effected over the first (and in the - Bind or Interleave case, the third and fifth) nodes in the set of - allowed nodes. The nodemask passed by the user represents nodes - relative to task or VMA's set of allowed nodes. - - If the user's nodemask includes nodes that are outside the range - of the new set of allowed nodes (for example, node 5 is set in - the user's nodemask when the set of allowed nodes is only 0-3), - then the remap wraps around to the beginning of the nodemask and, - if not already set, sets the node in the mempolicy nodemask. - - For example, consider a task that is attached to a cpuset with - mems 2-5 that sets an Interleave policy over the same set with - MPOL_F_RELATIVE_NODES. If the cpuset's mems change to 3-7, the - interleave now occurs over nodes 3,5-7. If the cpuset's mems - then change to 0,2-3,5, then the interleave occurs over nodes - 0,2-3,5. - - Thanks to the consistent remapping, applications preparing - nodemasks to specify memory policies using this flag should - disregard their current, actual cpuset imposed memory placement - and prepare the nodemask as if they were always located on - memory nodes 0 to N-1, where N is the number of memory nodes the - policy is intended to manage. Let the kernel then remap to the - set of memory nodes allowed by the task's cpuset, as that may - change over time. - - MPOL_F_RELATIVE_NODES cannot be combined with the - MPOL_F_STATIC_NODES flag. It also cannot be used for - MPOL_PREFERRED policies that were created with an empty nodemask - (local allocation). - -MEMORY POLICY REFERENCE COUNTING - -To resolve use/free races, struct mempolicy contains an atomic reference -count field. Internal interfaces, mpol_get()/mpol_put() increment and -decrement this reference count, respectively. mpol_put() will only free -the structure back to the mempolicy kmem cache when the reference count -goes to zero. - -When a new memory policy is allocated, its reference count is initialized -to '1', representing the reference held by the task that is installing the -new policy. When a pointer to a memory policy structure is stored in another -structure, another reference is added, as the task's reference will be dropped -on completion of the policy installation. - -During run-time "usage" of the policy, we attempt to minimize atomic operations -on the reference count, as this can lead to cache lines bouncing between cpus -and NUMA nodes. "Usage" here means one of the following: - -1) querying of the policy, either by the task itself [using the get_mempolicy() - API discussed below] or by another task using the /proc//numa_maps - interface. - -2) examination of the policy to determine the policy mode and associated node - or node lists, if any, for page allocation. This is considered a "hot - path". Note that for MPOL_BIND, the "usage" extends across the entire - allocation process, which may sleep during page reclaimation, because the - BIND policy nodemask is used, by reference, to filter ineligible nodes. - -We can avoid taking an extra reference during the usages listed above as -follows: - -1) we never need to get/free the system default policy as this is never - changed nor freed, once the system is up and running. - -2) for querying the policy, we do not need to take an extra reference on the - target task's task policy nor vma policies because we always acquire the - task's mm's mmap_sem for read during the query. The set_mempolicy() and - mbind() APIs [see below] always acquire the mmap_sem for write when - installing or replacing task or vma policies. Thus, there is no possibility - of a task or thread freeing a policy while another task or thread is - querying it. - -3) Page allocation usage of task or vma policy occurs in the fault path where - we hold them mmap_sem for read. Again, because replacing the task or vma - policy requires that the mmap_sem be held for write, the policy can't be - freed out from under us while we're using it for page allocation. - -4) Shared policies require special consideration. One task can replace a - shared memory policy while another task, with a distinct mmap_sem, is - querying or allocating a page based on the policy. To resolve this - potential race, the shared policy infrastructure adds an extra reference - to the shared policy during lookup while holding a spin lock on the shared - policy management structure. This requires that we drop this extra - reference when we're finished "using" the policy. We must drop the - extra reference on shared policies in the same query/allocation paths - used for non-shared policies. For this reason, shared policies are marked - as such, and the extra reference is dropped "conditionally"--i.e., only - for shared policies. - - Because of this extra reference counting, and because we must lookup - shared policies in a tree structure under spinlock, shared policies are - more expensive to use in the page allocation path. This is especially - true for shared policies on shared memory regions shared by tasks running - on different NUMA nodes. This extra overhead can be avoided by always - falling back to task or system default policy for shared memory regions, - or by prefaulting the entire shared memory region into memory and locking - it down. However, this might not be appropriate for all applications. - -MEMORY POLICY APIs - -Linux supports 3 system calls for controlling memory policy. These APIS -always affect only the calling task, the calling task's address space, or -some shared object mapped into the calling task's address space. - - Note: the headers that define these APIs and the parameter data types - for user space applications reside in a package that is not part of - the Linux kernel. The kernel system call interfaces, with the 'sys_' - prefix, are defined in ; the mode and flag - definitions are defined in . - -Set [Task] Memory Policy: - - long set_mempolicy(int mode, const unsigned long *nmask, - unsigned long maxnode); - - Set's the calling task's "task/process memory policy" to mode - specified by the 'mode' argument and the set of nodes defined - by 'nmask'. 'nmask' points to a bit mask of node ids containing - at least 'maxnode' ids. Optional mode flags may be passed by - combining the 'mode' argument with the flag (for example: - MPOL_INTERLEAVE | MPOL_F_STATIC_NODES). - - See the set_mempolicy(2) man page for more details - - -Get [Task] Memory Policy or Related Information - - long get_mempolicy(int *mode, - const unsigned long *nmask, unsigned long maxnode, - void *addr, int flags); - - Queries the "task/process memory policy" of the calling task, or - the policy or location of a specified virtual address, depending - on the 'flags' argument. - - See the get_mempolicy(2) man page for more details - - -Install VMA/Shared Policy for a Range of Task's Address Space - - long mbind(void *start, unsigned long len, int mode, - const unsigned long *nmask, unsigned long maxnode, - unsigned flags); - - mbind() installs the policy specified by (mode, nmask, maxnodes) as - a VMA policy for the range of the calling task's address space - specified by the 'start' and 'len' arguments. Additional actions - may be requested via the 'flags' argument. - - See the mbind(2) man page for more details. - -MEMORY POLICY COMMAND LINE INTERFACE - -Although not strictly part of the Linux implementation of memory policy, -a command line tool, numactl(8), exists that allows one to: - -+ set the task policy for a specified program via set_mempolicy(2), fork(2) and - exec(2) - -+ set the shared policy for a shared memory segment via mbind(2) - -The numactl(8) tool is packaged with the run-time version of the library -containing the memory policy system call wrappers. Some distributions -package the headers and compile-time libraries in a separate development -package. - - -MEMORY POLICIES AND CPUSETS - -Memory policies work within cpusets as described above. For memory policies -that require a node or set of nodes, the nodes are restricted to the set of -nodes whose memories are allowed by the cpuset constraints. If the nodemask -specified for the policy contains nodes that are not allowed by the cpuset and -MPOL_F_RELATIVE_NODES is not used, the intersection of the set of nodes -specified for the policy and the set of nodes with memory is used. If the -result is the empty set, the policy is considered invalid and cannot be -installed. If MPOL_F_RELATIVE_NODES is used, the policy's nodes are mapped -onto and folded into the task's set of allowed nodes as previously described. - -The interaction of memory policies and cpusets can be problematic when tasks -in two cpusets share access to a memory region, such as shared memory segments -created by shmget() of mmap() with the MAP_ANONYMOUS and MAP_SHARED flags, and -any of the tasks install shared policy on the region, only nodes whose -memories are allowed in both cpusets may be used in the policies. Obtaining -this information requires "stepping outside" the memory policy APIs to use the -cpuset information and requires that one know in what cpusets other task might -be attaching to the shared region. Furthermore, if the cpusets' allowed -memory sets are disjoint, "local" allocation is the only valid policy. diff --git a/Documentation/vm/overcommit-accounting b/Documentation/vm/overcommit-accounting deleted file mode 100644 index cbfaaa674118..000000000000 --- a/Documentation/vm/overcommit-accounting +++ /dev/null @@ -1,80 +0,0 @@ -The Linux kernel supports the following overcommit handling modes - -0 - Heuristic overcommit handling. Obvious overcommits of - address space are refused. Used for a typical system. It - ensures a seriously wild allocation fails while allowing - overcommit to reduce swap usage. root is allowed to - allocate slightly more memory in this mode. This is the - default. - -1 - Always overcommit. Appropriate for some scientific - applications. Classic example is code using sparse arrays - and just relying on the virtual memory consisting almost - entirely of zero pages. - -2 - Don't overcommit. The total address space commit - for the system is not permitted to exceed swap + a - configurable amount (default is 50%) of physical RAM. - Depending on the amount you use, in most situations - this means a process will not be killed while accessing - pages but will receive errors on memory allocation as - appropriate. - - Useful for applications that want to guarantee their - memory allocations will be available in the future - without having to initialize every page. - -The overcommit policy is set via the sysctl `vm.overcommit_memory'. - -The overcommit amount can be set via `vm.overcommit_ratio' (percentage) -or `vm.overcommit_kbytes' (absolute value). - -The current overcommit limit and amount committed are viewable in -/proc/meminfo as CommitLimit and Committed_AS respectively. - -Gotchas -------- - -The C language stack growth does an implicit mremap. If you want absolute -guarantees and run close to the edge you MUST mmap your stack for the -largest size you think you will need. For typical stack usage this does -not matter much but it's a corner case if you really really care - -In mode 2 the MAP_NORESERVE flag is ignored. - - -How It Works ------------- - -The overcommit is based on the following rules - -For a file backed map - SHARED or READ-only - 0 cost (the file is the map not swap) - PRIVATE WRITABLE - size of mapping per instance - -For an anonymous or /dev/zero map - SHARED - size of mapping - PRIVATE READ-only - 0 cost (but of little use) - PRIVATE WRITABLE - size of mapping per instance - -Additional accounting - Pages made writable copies by mmap - shmfs memory drawn from the same pool - -Status ------- - -o We account mmap memory mappings -o We account mprotect changes in commit -o We account mremap changes in size -o We account brk -o We account munmap -o We report the commit status in /proc -o Account and check on fork -o Review stack handling/building on exec -o SHMfs accounting -o Implement actual limit enforcement - -To Do ------ -o Account ptrace pages (this is hard) diff --git a/Documentation/vm/overcommit-accounting.rst b/Documentation/vm/overcommit-accounting.rst new file mode 100644 index 000000000000..0dd54bbe4afa --- /dev/null +++ b/Documentation/vm/overcommit-accounting.rst @@ -0,0 +1,87 @@ +.. _overcommit_accounting: + +===================== +Overcommit Accounting +===================== + +The Linux kernel supports the following overcommit handling modes + +0 + Heuristic overcommit handling. Obvious overcommits of address + space are refused. Used for a typical system. It ensures a + seriously wild allocation fails while allowing overcommit to + reduce swap usage. root is allowed to allocate slightly more + memory in this mode. This is the default. + +1 + Always overcommit. Appropriate for some scientific + applications. Classic example is code using sparse arrays and + just relying on the virtual memory consisting almost entirely + of zero pages. + +2 + Don't overcommit. The total address space commit for the + system is not permitted to exceed swap + a configurable amount + (default is 50%) of physical RAM. Depending on the amount you + use, in most situations this means a process will not be + killed while accessing pages but will receive errors on memory + allocation as appropriate. + + Useful for applications that want to guarantee their memory + allocations will be available in the future without having to + initialize every page. + +The overcommit policy is set via the sysctl ``vm.overcommit_memory``. + +The overcommit amount can be set via ``vm.overcommit_ratio`` (percentage) +or ``vm.overcommit_kbytes`` (absolute value). + +The current overcommit limit and amount committed are viewable in +``/proc/meminfo`` as CommitLimit and Committed_AS respectively. + +Gotchas +======= + +The C language stack growth does an implicit mremap. If you want absolute +guarantees and run close to the edge you MUST mmap your stack for the +largest size you think you will need. For typical stack usage this does +not matter much but it's a corner case if you really really care + +In mode 2 the MAP_NORESERVE flag is ignored. + + +How It Works +============ + +The overcommit is based on the following rules + +For a file backed map + | SHARED or READ-only - 0 cost (the file is the map not swap) + | PRIVATE WRITABLE - size of mapping per instance + +For an anonymous or ``/dev/zero`` map + | SHARED - size of mapping + | PRIVATE READ-only - 0 cost (but of little use) + | PRIVATE WRITABLE - size of mapping per instance + +Additional accounting + | Pages made writable copies by mmap + | shmfs memory drawn from the same pool + +Status +====== + +* We account mmap memory mappings +* We account mprotect changes in commit +* We account mremap changes in size +* We account brk +* We account munmap +* We report the commit status in /proc +* Account and check on fork +* Review stack handling/building on exec +* SHMfs accounting +* Implement actual limit enforcement + +To Do +===== +* Account ptrace pages (this is hard) diff --git a/Documentation/vm/page_frags b/Documentation/vm/page_frags deleted file mode 100644 index a6714565dbf9..000000000000 --- a/Documentation/vm/page_frags +++ /dev/null @@ -1,42 +0,0 @@ -Page fragments --------------- - -A page fragment is an arbitrary-length arbitrary-offset area of memory -which resides within a 0 or higher order compound page. Multiple -fragments within that page are individually refcounted, in the page's -reference counter. - -The page_frag functions, page_frag_alloc and page_frag_free, provide a -simple allocation framework for page fragments. This is used by the -network stack and network device drivers to provide a backing region of -memory for use as either an sk_buff->head, or to be used in the "frags" -portion of skb_shared_info. - -In order to make use of the page fragment APIs a backing page fragment -cache is needed. This provides a central point for the fragment allocation -and tracks allows multiple calls to make use of a cached page. The -advantage to doing this is that multiple calls to get_page can be avoided -which can be expensive at allocation time. However due to the nature of -this caching it is required that any calls to the cache be protected by -either a per-cpu limitation, or a per-cpu limitation and forcing interrupts -to be disabled when executing the fragment allocation. - -The network stack uses two separate caches per CPU to handle fragment -allocation. The netdev_alloc_cache is used by callers making use of the -__netdev_alloc_frag and __netdev_alloc_skb calls. The napi_alloc_cache is -used by callers of the __napi_alloc_frag and __napi_alloc_skb calls. The -main difference between these two calls is the context in which they may be -called. The "netdev" prefixed functions are usable in any context as these -functions will disable interrupts, while the "napi" prefixed functions are -only usable within the softirq context. - -Many network device drivers use a similar methodology for allocating page -fragments, but the page fragments are cached at the ring or descriptor -level. In order to enable these cases it is necessary to provide a generic -way of tearing down a page cache. For this reason __page_frag_cache_drain -was implemented. It allows for freeing multiple references from a single -page via a single call. The advantage to doing this is that it allows for -cleaning up the multiple references that were added to a page in order to -avoid calling get_page per allocation. - -Alexander Duyck, Nov 29, 2016. diff --git a/Documentation/vm/page_frags.rst b/Documentation/vm/page_frags.rst new file mode 100644 index 000000000000..637cc49d1b2f --- /dev/null +++ b/Documentation/vm/page_frags.rst @@ -0,0 +1,45 @@ +.. _page_frags: + +============== +Page fragments +============== + +A page fragment is an arbitrary-length arbitrary-offset area of memory +which resides within a 0 or higher order compound page. Multiple +fragments within that page are individually refcounted, in the page's +reference counter. + +The page_frag functions, page_frag_alloc and page_frag_free, provide a +simple allocation framework for page fragments. This is used by the +network stack and network device drivers to provide a backing region of +memory for use as either an sk_buff->head, or to be used in the "frags" +portion of skb_shared_info. + +In order to make use of the page fragment APIs a backing page fragment +cache is needed. This provides a central point for the fragment allocation +and tracks allows multiple calls to make use of a cached page. The +advantage to doing this is that multiple calls to get_page can be avoided +which can be expensive at allocation time. However due to the nature of +this caching it is required that any calls to the cache be protected by +either a per-cpu limitation, or a per-cpu limitation and forcing interrupts +to be disabled when executing the fragment allocation. + +The network stack uses two separate caches per CPU to handle fragment +allocation. The netdev_alloc_cache is used by callers making use of the +__netdev_alloc_frag and __netdev_alloc_skb calls. The napi_alloc_cache is +used by callers of the __napi_alloc_frag and __napi_alloc_skb calls. The +main difference between these two calls is the context in which they may be +called. The "netdev" prefixed functions are usable in any context as these +functions will disable interrupts, while the "napi" prefixed functions are +only usable within the softirq context. + +Many network device drivers use a similar methodology for allocating page +fragments, but the page fragments are cached at the ring or descriptor +level. In order to enable these cases it is necessary to provide a generic +way of tearing down a page cache. For this reason __page_frag_cache_drain +was implemented. It allows for freeing multiple references from a single +page via a single call. The advantage to doing this is that it allows for +cleaning up the multiple references that were added to a page in order to +avoid calling get_page per allocation. + +Alexander Duyck, Nov 29, 2016. diff --git a/Documentation/vm/page_migration b/Documentation/vm/page_migration deleted file mode 100644 index 496868072e24..000000000000 --- a/Documentation/vm/page_migration +++ /dev/null @@ -1,252 +0,0 @@ -Page migration --------------- - -Page migration allows the moving of the physical location of pages between -nodes in a numa system while the process is running. This means that the -virtual addresses that the process sees do not change. However, the -system rearranges the physical location of those pages. - -The main intend of page migration is to reduce the latency of memory access -by moving pages near to the processor where the process accessing that memory -is running. - -Page migration allows a process to manually relocate the node on which its -pages are located through the MF_MOVE and MF_MOVE_ALL options while setting -a new memory policy via mbind(). The pages of process can also be relocated -from another process using the sys_migrate_pages() function call. The -migrate_pages function call takes two sets of nodes and moves pages of a -process that are located on the from nodes to the destination nodes. -Page migration functions are provided by the numactl package by Andi Kleen -(a version later than 0.9.3 is required. Get it from -ftp://oss.sgi.com/www/projects/libnuma/download/). numactl provides libnuma -which provides an interface similar to other numa functionality for page -migration. cat /proc//numa_maps allows an easy review of where the -pages of a process are located. See also the numa_maps documentation in the -proc(5) man page. - -Manual migration is useful if for example the scheduler has relocated -a process to a processor on a distant node. A batch scheduler or an -administrator may detect the situation and move the pages of the process -nearer to the new processor. The kernel itself does only provide -manual page migration support. Automatic page migration may be implemented -through user space processes that move pages. A special function call -"move_pages" allows the moving of individual pages within a process. -A NUMA profiler may f.e. obtain a log showing frequent off node -accesses and may use the result to move pages to more advantageous -locations. - -Larger installations usually partition the system using cpusets into -sections of nodes. Paul Jackson has equipped cpusets with the ability to -move pages when a task is moved to another cpuset (See -Documentation/cgroup-v1/cpusets.txt). -Cpusets allows the automation of process locality. If a task is moved to -a new cpuset then also all its pages are moved with it so that the -performance of the process does not sink dramatically. Also the pages -of processes in a cpuset are moved if the allowed memory nodes of a -cpuset are changed. - -Page migration allows the preservation of the relative location of pages -within a group of nodes for all migration techniques which will preserve a -particular memory allocation pattern generated even after migrating a -process. This is necessary in order to preserve the memory latencies. -Processes will run with similar performance after migration. - -Page migration occurs in several steps. First a high level -description for those trying to use migrate_pages() from the kernel -(for userspace usage see the Andi Kleen's numactl package mentioned above) -and then a low level description of how the low level details work. - -A. In kernel use of migrate_pages() ------------------------------------ - -1. Remove pages from the LRU. - - Lists of pages to be migrated are generated by scanning over - pages and moving them into lists. This is done by - calling isolate_lru_page(). - Calling isolate_lru_page increases the references to the page - so that it cannot vanish while the page migration occurs. - It also prevents the swapper or other scans to encounter - the page. - -2. We need to have a function of type new_page_t that can be - passed to migrate_pages(). This function should figure out - how to allocate the correct new page given the old page. - -3. The migrate_pages() function is called which attempts - to do the migration. It will call the function to allocate - the new page for each page that is considered for - moving. - -B. How migrate_pages() works ----------------------------- - -migrate_pages() does several passes over its list of pages. A page is moved -if all references to a page are removable at the time. The page has -already been removed from the LRU via isolate_lru_page() and the refcount -is increased so that the page cannot be freed while page migration occurs. - -Steps: - -1. Lock the page to be migrated - -2. Ensure that writeback is complete. - -3. Lock the new page that we want to move to. It is locked so that accesses to - this (not yet uptodate) page immediately lock while the move is in progress. - -4. All the page table references to the page are converted to migration - entries. This decreases the mapcount of a page. If the resulting - mapcount is not zero then we do not migrate the page. All user space - processes that attempt to access the page will now wait on the page lock. - -5. The i_pages lock is taken. This will cause all processes trying - to access the page via the mapping to block on the spinlock. - -6. The refcount of the page is examined and we back out if references remain - otherwise we know that we are the only one referencing this page. - -7. The radix tree is checked and if it does not contain the pointer to this - page then we back out because someone else modified the radix tree. - -8. The new page is prepped with some settings from the old page so that - accesses to the new page will discover a page with the correct settings. - -9. The radix tree is changed to point to the new page. - -10. The reference count of the old page is dropped because the address space - reference is gone. A reference to the new page is established because - the new page is referenced by the address space. - -11. The i_pages lock is dropped. With that lookups in the mapping - become possible again. Processes will move from spinning on the lock - to sleeping on the locked new page. - -12. The page contents are copied to the new page. - -13. The remaining page flags are copied to the new page. - -14. The old page flags are cleared to indicate that the page does - not provide any information anymore. - -15. Queued up writeback on the new page is triggered. - -16. If migration entries were page then replace them with real ptes. Doing - so will enable access for user space processes not already waiting for - the page lock. - -19. The page locks are dropped from the old and new page. - Processes waiting on the page lock will redo their page faults - and will reach the new page. - -20. The new page is moved to the LRU and can be scanned by the swapper - etc again. - -C. Non-LRU page migration -------------------------- - -Although original migration aimed for reducing the latency of memory access -for NUMA, compaction who want to create high-order page is also main customer. - -Current problem of the implementation is that it is designed to migrate only -*LRU* pages. However, there are potential non-lru pages which can be migrated -in drivers, for example, zsmalloc, virtio-balloon pages. - -For virtio-balloon pages, some parts of migration code path have been hooked -up and added virtio-balloon specific functions to intercept migration logics. -It's too specific to a driver so other drivers who want to make their pages -movable would have to add own specific hooks in migration path. - -To overclome the problem, VM supports non-LRU page migration which provides -generic functions for non-LRU movable pages without driver specific hooks -migration path. - -If a driver want to make own pages movable, it should define three functions -which are function pointers of struct address_space_operations. - -1. bool (*isolate_page) (struct page *page, isolate_mode_t mode); - -What VM expects on isolate_page function of driver is to return *true* -if driver isolates page successfully. On returing true, VM marks the page -as PG_isolated so concurrent isolation in several CPUs skip the page -for isolation. If a driver cannot isolate the page, it should return *false*. - -Once page is successfully isolated, VM uses page.lru fields so driver -shouldn't expect to preserve values in that fields. - -2. int (*migratepage) (struct address_space *mapping, - struct page *newpage, struct page *oldpage, enum migrate_mode); - -After isolation, VM calls migratepage of driver with isolated page. -The function of migratepage is to move content of the old page to new page -and set up fields of struct page newpage. Keep in mind that you should -indicate to the VM the oldpage is no longer movable via __ClearPageMovable() -under page_lock if you migrated the oldpage successfully and returns -MIGRATEPAGE_SUCCESS. If driver cannot migrate the page at the moment, driver -can return -EAGAIN. On -EAGAIN, VM will retry page migration in a short time -because VM interprets -EAGAIN as "temporal migration failure". On returning -any error except -EAGAIN, VM will give up the page migration without retrying -in this time. - -Driver shouldn't touch page.lru field VM using in the functions. - -3. void (*putback_page)(struct page *); - -If migration fails on isolated page, VM should return the isolated page -to the driver so VM calls driver's putback_page with migration failed page. -In this function, driver should put the isolated page back to the own data -structure. - -4. non-lru movable page flags - -There are two page flags for supporting non-lru movable page. - -* PG_movable - -Driver should use the below function to make page movable under page_lock. - - void __SetPageMovable(struct page *page, struct address_space *mapping) - -It needs argument of address_space for registering migration family functions -which will be called by VM. Exactly speaking, PG_movable is not a real flag of -struct page. Rather than, VM reuses page->mapping's lower bits to represent it. - - #define PAGE_MAPPING_MOVABLE 0x2 - page->mapping = page->mapping | PAGE_MAPPING_MOVABLE; - -so driver shouldn't access page->mapping directly. Instead, driver should -use page_mapping which mask off the low two bits of page->mapping under -page lock so it can get right struct address_space. - -For testing of non-lru movable page, VM supports __PageMovable function. -However, it doesn't guarantee to identify non-lru movable page because -page->mapping field is unified with other variables in struct page. -As well, if driver releases the page after isolation by VM, page->mapping -doesn't have stable value although it has PAGE_MAPPING_MOVABLE -(Look at __ClearPageMovable). But __PageMovable is cheap to catch whether -page is LRU or non-lru movable once the page has been isolated. Because -LRU pages never can have PAGE_MAPPING_MOVABLE in page->mapping. It is also -good for just peeking to test non-lru movable pages before more expensive -checking with lock_page in pfn scanning to select victim. - -For guaranteeing non-lru movable page, VM provides PageMovable function. -Unlike __PageMovable, PageMovable functions validates page->mapping and -mapping->a_ops->isolate_page under lock_page. The lock_page prevents sudden -destroying of page->mapping. - -Driver using __SetPageMovable should clear the flag via __ClearMovablePage -under page_lock before the releasing the page. - -* PG_isolated - -To prevent concurrent isolation among several CPUs, VM marks isolated page -as PG_isolated under lock_page. So if a CPU encounters PG_isolated non-lru -movable page, it can skip it. Driver doesn't need to manipulate the flag -because VM will set/clear it automatically. Keep in mind that if driver -sees PG_isolated page, it means the page have been isolated by VM so it -shouldn't touch page.lru field. -PG_isolated is alias with PG_reclaim flag so driver shouldn't use the flag -for own purpose. - -Christoph Lameter, May 8, 2006. -Minchan Kim, Mar 28, 2016. diff --git a/Documentation/vm/page_migration.rst b/Documentation/vm/page_migration.rst new file mode 100644 index 000000000000..f68d61335abb --- /dev/null +++ b/Documentation/vm/page_migration.rst @@ -0,0 +1,257 @@ +.. _page_migration: + +============== +Page migration +============== + +Page migration allows the moving of the physical location of pages between +nodes in a numa system while the process is running. This means that the +virtual addresses that the process sees do not change. However, the +system rearranges the physical location of those pages. + +The main intend of page migration is to reduce the latency of memory access +by moving pages near to the processor where the process accessing that memory +is running. + +Page migration allows a process to manually relocate the node on which its +pages are located through the MF_MOVE and MF_MOVE_ALL options while setting +a new memory policy via mbind(). The pages of process can also be relocated +from another process using the sys_migrate_pages() function call. The +migrate_pages function call takes two sets of nodes and moves pages of a +process that are located on the from nodes to the destination nodes. +Page migration functions are provided by the numactl package by Andi Kleen +(a version later than 0.9.3 is required. Get it from +ftp://oss.sgi.com/www/projects/libnuma/download/). numactl provides libnuma +which provides an interface similar to other numa functionality for page +migration. cat ``/proc//numa_maps`` allows an easy review of where the +pages of a process are located. See also the numa_maps documentation in the +proc(5) man page. + +Manual migration is useful if for example the scheduler has relocated +a process to a processor on a distant node. A batch scheduler or an +administrator may detect the situation and move the pages of the process +nearer to the new processor. The kernel itself does only provide +manual page migration support. Automatic page migration may be implemented +through user space processes that move pages. A special function call +"move_pages" allows the moving of individual pages within a process. +A NUMA profiler may f.e. obtain a log showing frequent off node +accesses and may use the result to move pages to more advantageous +locations. + +Larger installations usually partition the system using cpusets into +sections of nodes. Paul Jackson has equipped cpusets with the ability to +move pages when a task is moved to another cpuset (See +Documentation/cgroup-v1/cpusets.txt). +Cpusets allows the automation of process locality. If a task is moved to +a new cpuset then also all its pages are moved with it so that the +performance of the process does not sink dramatically. Also the pages +of processes in a cpuset are moved if the allowed memory nodes of a +cpuset are changed. + +Page migration allows the preservation of the relative location of pages +within a group of nodes for all migration techniques which will preserve a +particular memory allocation pattern generated even after migrating a +process. This is necessary in order to preserve the memory latencies. +Processes will run with similar performance after migration. + +Page migration occurs in several steps. First a high level +description for those trying to use migrate_pages() from the kernel +(for userspace usage see the Andi Kleen's numactl package mentioned above) +and then a low level description of how the low level details work. + +In kernel use of migrate_pages() +================================ + +1. Remove pages from the LRU. + + Lists of pages to be migrated are generated by scanning over + pages and moving them into lists. This is done by + calling isolate_lru_page(). + Calling isolate_lru_page increases the references to the page + so that it cannot vanish while the page migration occurs. + It also prevents the swapper or other scans to encounter + the page. + +2. We need to have a function of type new_page_t that can be + passed to migrate_pages(). This function should figure out + how to allocate the correct new page given the old page. + +3. The migrate_pages() function is called which attempts + to do the migration. It will call the function to allocate + the new page for each page that is considered for + moving. + +How migrate_pages() works +========================= + +migrate_pages() does several passes over its list of pages. A page is moved +if all references to a page are removable at the time. The page has +already been removed from the LRU via isolate_lru_page() and the refcount +is increased so that the page cannot be freed while page migration occurs. + +Steps: + +1. Lock the page to be migrated + +2. Ensure that writeback is complete. + +3. Lock the new page that we want to move to. It is locked so that accesses to + this (not yet uptodate) page immediately lock while the move is in progress. + +4. All the page table references to the page are converted to migration + entries. This decreases the mapcount of a page. If the resulting + mapcount is not zero then we do not migrate the page. All user space + processes that attempt to access the page will now wait on the page lock. + +5. The i_pages lock is taken. This will cause all processes trying + to access the page via the mapping to block on the spinlock. + +6. The refcount of the page is examined and we back out if references remain + otherwise we know that we are the only one referencing this page. + +7. The radix tree is checked and if it does not contain the pointer to this + page then we back out because someone else modified the radix tree. + +8. The new page is prepped with some settings from the old page so that + accesses to the new page will discover a page with the correct settings. + +9. The radix tree is changed to point to the new page. + +10. The reference count of the old page is dropped because the address space + reference is gone. A reference to the new page is established because + the new page is referenced by the address space. + +11. The i_pages lock is dropped. With that lookups in the mapping + become possible again. Processes will move from spinning on the lock + to sleeping on the locked new page. + +12. The page contents are copied to the new page. + +13. The remaining page flags are copied to the new page. + +14. The old page flags are cleared to indicate that the page does + not provide any information anymore. + +15. Queued up writeback on the new page is triggered. + +16. If migration entries were page then replace them with real ptes. Doing + so will enable access for user space processes not already waiting for + the page lock. + +19. The page locks are dropped from the old and new page. + Processes waiting on the page lock will redo their page faults + and will reach the new page. + +20. The new page is moved to the LRU and can be scanned by the swapper + etc again. + +Non-LRU page migration +====================== + +Although original migration aimed for reducing the latency of memory access +for NUMA, compaction who want to create high-order page is also main customer. + +Current problem of the implementation is that it is designed to migrate only +*LRU* pages. However, there are potential non-lru pages which can be migrated +in drivers, for example, zsmalloc, virtio-balloon pages. + +For virtio-balloon pages, some parts of migration code path have been hooked +up and added virtio-balloon specific functions to intercept migration logics. +It's too specific to a driver so other drivers who want to make their pages +movable would have to add own specific hooks in migration path. + +To overclome the problem, VM supports non-LRU page migration which provides +generic functions for non-LRU movable pages without driver specific hooks +migration path. + +If a driver want to make own pages movable, it should define three functions +which are function pointers of struct address_space_operations. + +1. ``bool (*isolate_page) (struct page *page, isolate_mode_t mode);`` + + What VM expects on isolate_page function of driver is to return *true* + if driver isolates page successfully. On returing true, VM marks the page + as PG_isolated so concurrent isolation in several CPUs skip the page + for isolation. If a driver cannot isolate the page, it should return *false*. + + Once page is successfully isolated, VM uses page.lru fields so driver + shouldn't expect to preserve values in that fields. + +2. ``int (*migratepage) (struct address_space *mapping,`` +| ``struct page *newpage, struct page *oldpage, enum migrate_mode);`` + + After isolation, VM calls migratepage of driver with isolated page. + The function of migratepage is to move content of the old page to new page + and set up fields of struct page newpage. Keep in mind that you should + indicate to the VM the oldpage is no longer movable via __ClearPageMovable() + under page_lock if you migrated the oldpage successfully and returns + MIGRATEPAGE_SUCCESS. If driver cannot migrate the page at the moment, driver + can return -EAGAIN. On -EAGAIN, VM will retry page migration in a short time + because VM interprets -EAGAIN as "temporal migration failure". On returning + any error except -EAGAIN, VM will give up the page migration without retrying + in this time. + + Driver shouldn't touch page.lru field VM using in the functions. + +3. ``void (*putback_page)(struct page *);`` + + If migration fails on isolated page, VM should return the isolated page + to the driver so VM calls driver's putback_page with migration failed page. + In this function, driver should put the isolated page back to the own data + structure. + +4. non-lru movable page flags + + There are two page flags for supporting non-lru movable page. + + * PG_movable + + Driver should use the below function to make page movable under page_lock:: + + void __SetPageMovable(struct page *page, struct address_space *mapping) + + It needs argument of address_space for registering migration + family functions which will be called by VM. Exactly speaking, + PG_movable is not a real flag of struct page. Rather than, VM + reuses page->mapping's lower bits to represent it. + +:: + #define PAGE_MAPPING_MOVABLE 0x2 + page->mapping = page->mapping | PAGE_MAPPING_MOVABLE; + + so driver shouldn't access page->mapping directly. Instead, driver should + use page_mapping which mask off the low two bits of page->mapping under + page lock so it can get right struct address_space. + + For testing of non-lru movable page, VM supports __PageMovable function. + However, it doesn't guarantee to identify non-lru movable page because + page->mapping field is unified with other variables in struct page. + As well, if driver releases the page after isolation by VM, page->mapping + doesn't have stable value although it has PAGE_MAPPING_MOVABLE + (Look at __ClearPageMovable). But __PageMovable is cheap to catch whether + page is LRU or non-lru movable once the page has been isolated. Because + LRU pages never can have PAGE_MAPPING_MOVABLE in page->mapping. It is also + good for just peeking to test non-lru movable pages before more expensive + checking with lock_page in pfn scanning to select victim. + + For guaranteeing non-lru movable page, VM provides PageMovable function. + Unlike __PageMovable, PageMovable functions validates page->mapping and + mapping->a_ops->isolate_page under lock_page. The lock_page prevents sudden + destroying of page->mapping. + + Driver using __SetPageMovable should clear the flag via __ClearMovablePage + under page_lock before the releasing the page. + + * PG_isolated + + To prevent concurrent isolation among several CPUs, VM marks isolated page + as PG_isolated under lock_page. So if a CPU encounters PG_isolated non-lru + movable page, it can skip it. Driver doesn't need to manipulate the flag + because VM will set/clear it automatically. Keep in mind that if driver + sees PG_isolated page, it means the page have been isolated by VM so it + shouldn't touch page.lru field. + PG_isolated is alias with PG_reclaim flag so driver shouldn't use the flag + for own purpose. + +Christoph Lameter, May 8, 2006. +Minchan Kim, Mar 28, 2016. diff --git a/Documentation/vm/page_owner.rst b/Documentation/vm/page_owner.rst new file mode 100644 index 000000000000..0ed5ab8c7ab4 --- /dev/null +++ b/Documentation/vm/page_owner.rst @@ -0,0 +1,90 @@ +.. _page_owner: + +================================================== +page owner: Tracking about who allocated each page +================================================== + +Introduction +============ + +page owner is for the tracking about who allocated each page. +It can be used to debug memory leak or to find a memory hogger. +When allocation happens, information about allocation such as call stack +and order of pages is stored into certain storage for each page. +When we need to know about status of all pages, we can get and analyze +this information. + +Although we already have tracepoint for tracing page allocation/free, +using it for analyzing who allocate each page is rather complex. We need +to enlarge the trace buffer for preventing overlapping until userspace +program launched. And, launched program continually dump out the trace +buffer for later analysis and it would change system behviour with more +possibility rather than just keeping it in memory, so bad for debugging. + +page owner can also be used for various purposes. For example, accurate +fragmentation statistics can be obtained through gfp flag information of +each page. It is already implemented and activated if page owner is +enabled. Other usages are more than welcome. + +page owner is disabled in default. So, if you'd like to use it, you need +to add "page_owner=on" into your boot cmdline. If the kernel is built +with page owner and page owner is disabled in runtime due to no enabling +boot option, runtime overhead is marginal. If disabled in runtime, it +doesn't require memory to store owner information, so there is no runtime +memory overhead. And, page owner inserts just two unlikely branches into +the page allocator hotpath and if not enabled, then allocation is done +like as the kernel without page owner. These two unlikely branches should +not affect to allocation performance, especially if the static keys jump +label patching functionality is available. Following is the kernel's code +size change due to this facility. + +- Without page owner:: + + text data bss dec hex filename + 40662 1493 644 42799 a72f mm/page_alloc.o + +- With page owner:: + + text data bss dec hex filename + 40892 1493 644 43029 a815 mm/page_alloc.o + 1427 24 8 1459 5b3 mm/page_ext.o + 2722 50 0 2772 ad4 mm/page_owner.o + +Although, roughly, 4 KB code is added in total, page_alloc.o increase by +230 bytes and only half of it is in hotpath. Building the kernel with +page owner and turning it on if needed would be great option to debug +kernel memory problem. + +There is one notice that is caused by implementation detail. page owner +stores information into the memory from struct page extension. This memory +is initialized some time later than that page allocator starts in sparse +memory system, so, until initialization, many pages can be allocated and +they would have no owner information. To fix it up, these early allocated +pages are investigated and marked as allocated in initialization phase. +Although it doesn't mean that they have the right owner information, +at least, we can tell whether the page is allocated or not, +more accurately. On 2GB memory x86-64 VM box, 13343 early allocated pages +are catched and marked, although they are mostly allocated from struct +page extension feature. Anyway, after that, no page is left in +un-tracking state. + +Usage +===== + +1) Build user-space helper:: + + cd tools/vm + make page_owner_sort + +2) Enable page owner: add "page_owner=on" to boot cmdline. + +3) Do the job what you want to debug + +4) Analyze information from page owner:: + + cat /sys/kernel/debug/page_owner > page_owner_full.txt + grep -v ^PFN page_owner_full.txt > page_owner.txt + ./page_owner_sort page_owner.txt sorted_page_owner.txt + + See the result about who allocated each page + in the ``sorted_page_owner.txt``. diff --git a/Documentation/vm/page_owner.txt b/Documentation/vm/page_owner.txt deleted file mode 100644 index ffff1439076a..000000000000 --- a/Documentation/vm/page_owner.txt +++ /dev/null @@ -1,82 +0,0 @@ -page owner: Tracking about who allocated each page ------------------------------------------------------------ - -* Introduction - -page owner is for the tracking about who allocated each page. -It can be used to debug memory leak or to find a memory hogger. -When allocation happens, information about allocation such as call stack -and order of pages is stored into certain storage for each page. -When we need to know about status of all pages, we can get and analyze -this information. - -Although we already have tracepoint for tracing page allocation/free, -using it for analyzing who allocate each page is rather complex. We need -to enlarge the trace buffer for preventing overlapping until userspace -program launched. And, launched program continually dump out the trace -buffer for later analysis and it would change system behviour with more -possibility rather than just keeping it in memory, so bad for debugging. - -page owner can also be used for various purposes. For example, accurate -fragmentation statistics can be obtained through gfp flag information of -each page. It is already implemented and activated if page owner is -enabled. Other usages are more than welcome. - -page owner is disabled in default. So, if you'd like to use it, you need -to add "page_owner=on" into your boot cmdline. If the kernel is built -with page owner and page owner is disabled in runtime due to no enabling -boot option, runtime overhead is marginal. If disabled in runtime, it -doesn't require memory to store owner information, so there is no runtime -memory overhead. And, page owner inserts just two unlikely branches into -the page allocator hotpath and if not enabled, then allocation is done -like as the kernel without page owner. These two unlikely branches should -not affect to allocation performance, especially if the static keys jump -label patching functionality is available. Following is the kernel's code -size change due to this facility. - -- Without page owner - text data bss dec hex filename - 40662 1493 644 42799 a72f mm/page_alloc.o - -- With page owner - text data bss dec hex filename - 40892 1493 644 43029 a815 mm/page_alloc.o - 1427 24 8 1459 5b3 mm/page_ext.o - 2722 50 0 2772 ad4 mm/page_owner.o - -Although, roughly, 4 KB code is added in total, page_alloc.o increase by -230 bytes and only half of it is in hotpath. Building the kernel with -page owner and turning it on if needed would be great option to debug -kernel memory problem. - -There is one notice that is caused by implementation detail. page owner -stores information into the memory from struct page extension. This memory -is initialized some time later than that page allocator starts in sparse -memory system, so, until initialization, many pages can be allocated and -they would have no owner information. To fix it up, these early allocated -pages are investigated and marked as allocated in initialization phase. -Although it doesn't mean that they have the right owner information, -at least, we can tell whether the page is allocated or not, -more accurately. On 2GB memory x86-64 VM box, 13343 early allocated pages -are catched and marked, although they are mostly allocated from struct -page extension feature. Anyway, after that, no page is left in -un-tracking state. - -* Usage - -1) Build user-space helper - cd tools/vm - make page_owner_sort - -2) Enable page owner - Add "page_owner=on" to boot cmdline. - -3) Do the job what you want to debug - -4) Analyze information from page owner - cat /sys/kernel/debug/page_owner > page_owner_full.txt - grep -v ^PFN page_owner_full.txt > page_owner.txt - ./page_owner_sort page_owner.txt sorted_page_owner.txt - - See the result about who allocated each page - in the sorted_page_owner.txt. diff --git a/Documentation/vm/pagemap.txt b/Documentation/vm/pagemap.txt deleted file mode 100644 index eafcefa15261..000000000000 --- a/Documentation/vm/pagemap.txt +++ /dev/null @@ -1,183 +0,0 @@ -pagemap, from the userspace perspective ---------------------------------------- - -pagemap is a new (as of 2.6.25) set of interfaces in the kernel that allow -userspace programs to examine the page tables and related information by -reading files in /proc. - -There are four components to pagemap: - - * /proc/pid/pagemap. This file lets a userspace process find out which - physical frame each virtual page is mapped to. It contains one 64-bit - value for each virtual page, containing the following data (from - fs/proc/task_mmu.c, above pagemap_read): - - * Bits 0-54 page frame number (PFN) if present - * Bits 0-4 swap type if swapped - * Bits 5-54 swap offset if swapped - * Bit 55 pte is soft-dirty (see Documentation/vm/soft-dirty.txt) - * Bit 56 page exclusively mapped (since 4.2) - * Bits 57-60 zero - * Bit 61 page is file-page or shared-anon (since 3.5) - * Bit 62 page swapped - * Bit 63 page present - - Since Linux 4.0 only users with the CAP_SYS_ADMIN capability can get PFNs. - In 4.0 and 4.1 opens by unprivileged fail with -EPERM. Starting from - 4.2 the PFN field is zeroed if the user does not have CAP_SYS_ADMIN. - Reason: information about PFNs helps in exploiting Rowhammer vulnerability. - - If the page is not present but in swap, then the PFN contains an - encoding of the swap file number and the page's offset into the - swap. Unmapped pages return a null PFN. This allows determining - precisely which pages are mapped (or in swap) and comparing mapped - pages between processes. - - Efficient users of this interface will use /proc/pid/maps to - determine which areas of memory are actually mapped and llseek to - skip over unmapped regions. - - * /proc/kpagecount. This file contains a 64-bit count of the number of - times each page is mapped, indexed by PFN. - - * /proc/kpageflags. This file contains a 64-bit set of flags for each - page, indexed by PFN. - - The flags are (from fs/proc/page.c, above kpageflags_read): - - 0. LOCKED - 1. ERROR - 2. REFERENCED - 3. UPTODATE - 4. DIRTY - 5. LRU - 6. ACTIVE - 7. SLAB - 8. WRITEBACK - 9. RECLAIM - 10. BUDDY - 11. MMAP - 12. ANON - 13. SWAPCACHE - 14. SWAPBACKED - 15. COMPOUND_HEAD - 16. COMPOUND_TAIL - 17. HUGE - 18. UNEVICTABLE - 19. HWPOISON - 20. NOPAGE - 21. KSM - 22. THP - 23. BALLOON - 24. ZERO_PAGE - 25. IDLE - - * /proc/kpagecgroup. This file contains a 64-bit inode number of the - memory cgroup each page is charged to, indexed by PFN. Only available when - CONFIG_MEMCG is set. - -Short descriptions to the page flags: - - 0. LOCKED - page is being locked for exclusive access, eg. by undergoing read/write IO - - 7. SLAB - page is managed by the SLAB/SLOB/SLUB/SLQB kernel memory allocator - When compound page is used, SLUB/SLQB will only set this flag on the head - page; SLOB will not flag it at all. - -10. BUDDY - a free memory block managed by the buddy system allocator - The buddy system organizes free memory in blocks of various orders. - An order N block has 2^N physically contiguous pages, with the BUDDY flag - set for and _only_ for the first page. - -15. COMPOUND_HEAD -16. COMPOUND_TAIL - A compound page with order N consists of 2^N physically contiguous pages. - A compound page with order 2 takes the form of "HTTT", where H donates its - head page and T donates its tail page(s). The major consumers of compound - pages are hugeTLB pages (Documentation/vm/hugetlbpage.txt), the SLUB etc. - memory allocators and various device drivers. However in this interface, - only huge/giga pages are made visible to end users. -17. HUGE - this is an integral part of a HugeTLB page - -19. HWPOISON - hardware detected memory corruption on this page: don't touch the data! - -20. NOPAGE - no page frame exists at the requested address - -21. KSM - identical memory pages dynamically shared between one or more processes - -22. THP - contiguous pages which construct transparent hugepages - -23. BALLOON - balloon compaction page - -24. ZERO_PAGE - zero page for pfn_zero or huge_zero page - -25. IDLE - page has not been accessed since it was marked idle (see - Documentation/vm/idle_page_tracking.txt). Note that this flag may be - stale in case the page was accessed via a PTE. To make sure the flag - is up-to-date one has to read /sys/kernel/mm/page_idle/bitmap first. - - [IO related page flags] - 1. ERROR IO error occurred - 3. UPTODATE page has up-to-date data - ie. for file backed page: (in-memory data revision >= on-disk one) - 4. DIRTY page has been written to, hence contains new data - ie. for file backed page: (in-memory data revision > on-disk one) - 8. WRITEBACK page is being synced to disk - - [LRU related page flags] - 5. LRU page is in one of the LRU lists - 6. ACTIVE page is in the active LRU list -18. UNEVICTABLE page is in the unevictable (non-)LRU list - It is somehow pinned and not a candidate for LRU page reclaims, - eg. ramfs pages, shmctl(SHM_LOCK) and mlock() memory segments - 2. REFERENCED page has been referenced since last LRU list enqueue/requeue - 9. RECLAIM page will be reclaimed soon after its pageout IO completed -11. MMAP a memory mapped page -12. ANON a memory mapped page that is not part of a file -13. SWAPCACHE page is mapped to swap space, ie. has an associated swap entry -14. SWAPBACKED page is backed by swap/RAM - -The page-types tool in the tools/vm directory can be used to query the -above flags. - -Using pagemap to do something useful: - -The general procedure for using pagemap to find out about a process' memory -usage goes like this: - - 1. Read /proc/pid/maps to determine which parts of the memory space are - mapped to what. - 2. Select the maps you are interested in -- all of them, or a particular - library, or the stack or the heap, etc. - 3. Open /proc/pid/pagemap and seek to the pages you would like to examine. - 4. Read a u64 for each page from pagemap. - 5. Open /proc/kpagecount and/or /proc/kpageflags. For each PFN you just - read, seek to that entry in the file, and read the data you want. - -For example, to find the "unique set size" (USS), which is the amount of -memory that a process is using that is not shared with any other process, -you can go through every map in the process, find the PFNs, look those up -in kpagecount, and tally up the number of pages that are only referenced -once. - -Other notes: - -Reading from any of the files will return -EINVAL if you are not starting -the read on an 8-byte boundary (e.g., if you sought an odd number of bytes -into the file), or if the size of the read is not a multiple of 8 bytes. - -Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is -always 12 at most architectures). Since Linux 3.11 their meaning changes -after first clear of soft-dirty bits. Since Linux 4.2 they are used for -flags unconditionally. diff --git a/Documentation/vm/remap_file_pages.rst b/Documentation/vm/remap_file_pages.rst new file mode 100644 index 000000000000..7bef6718e3a9 --- /dev/null +++ b/Documentation/vm/remap_file_pages.rst @@ -0,0 +1,33 @@ +.. _remap_file_pages: + +============================== +remap_file_pages() system call +============================== + +The remap_file_pages() system call is used to create a nonlinear mapping, +that is, a mapping in which the pages of the file are mapped into a +nonsequential order in memory. The advantage of using remap_file_pages() +over using repeated calls to mmap(2) is that the former approach does not +require the kernel to create additional VMA (Virtual Memory Area) data +structures. + +Supporting of nonlinear mapping requires significant amount of non-trivial +code in kernel virtual memory subsystem including hot paths. Also to get +nonlinear mapping work kernel need a way to distinguish normal page table +entries from entries with file offset (pte_file). Kernel reserves flag in +PTE for this purpose. PTE flags are scarce resource especially on some CPU +architectures. It would be nice to free up the flag for other usage. + +Fortunately, there are not many users of remap_file_pages() in the wild. +It's only known that one enterprise RDBMS implementation uses the syscall +on 32-bit systems to map files bigger than can linearly fit into 32-bit +virtual address space. This use-case is not critical anymore since 64-bit +systems are widely available. + +The syscall is deprecated and replaced it with an emulation now. The +emulation creates new VMAs instead of nonlinear mappings. It's going to +work slower for rare users of remap_file_pages() but ABI is preserved. + +One side effect of emulation (apart from performance) is that user can hit +vm.max_map_count limit more easily due to additional VMAs. See comment for +DEFAULT_MAX_MAP_COUNT for more details on the limit. diff --git a/Documentation/vm/remap_file_pages.txt b/Documentation/vm/remap_file_pages.txt deleted file mode 100644 index f609142f406a..000000000000 --- a/Documentation/vm/remap_file_pages.txt +++ /dev/null @@ -1,27 +0,0 @@ -The remap_file_pages() system call is used to create a nonlinear mapping, -that is, a mapping in which the pages of the file are mapped into a -nonsequential order in memory. The advantage of using remap_file_pages() -over using repeated calls to mmap(2) is that the former approach does not -require the kernel to create additional VMA (Virtual Memory Area) data -structures. - -Supporting of nonlinear mapping requires significant amount of non-trivial -code in kernel virtual memory subsystem including hot paths. Also to get -nonlinear mapping work kernel need a way to distinguish normal page table -entries from entries with file offset (pte_file). Kernel reserves flag in -PTE for this purpose. PTE flags are scarce resource especially on some CPU -architectures. It would be nice to free up the flag for other usage. - -Fortunately, there are not many users of remap_file_pages() in the wild. -It's only known that one enterprise RDBMS implementation uses the syscall -on 32-bit systems to map files bigger than can linearly fit into 32-bit -virtual address space. This use-case is not critical anymore since 64-bit -systems are widely available. - -The syscall is deprecated and replaced it with an emulation now. The -emulation creates new VMAs instead of nonlinear mappings. It's going to -work slower for rare users of remap_file_pages() but ABI is preserved. - -One side effect of emulation (apart from performance) is that user can hit -vm.max_map_count limit more easily due to additional VMAs. See comment for -DEFAULT_MAX_MAP_COUNT for more details on the limit. diff --git a/Documentation/vm/slub.rst b/Documentation/vm/slub.rst new file mode 100644 index 000000000000..3a775fd64e2d --- /dev/null +++ b/Documentation/vm/slub.rst @@ -0,0 +1,361 @@ +.. _slub: + +========================== +Short users guide for SLUB +========================== + +The basic philosophy of SLUB is very different from SLAB. SLAB +requires rebuilding the kernel to activate debug options for all +slab caches. SLUB always includes full debugging but it is off by default. +SLUB can enable debugging only for selected slabs in order to avoid +an impact on overall system performance which may make a bug more +difficult to find. + +In order to switch debugging on one can add an option ``slub_debug`` +to the kernel command line. That will enable full debugging for +all slabs. + +Typically one would then use the ``slabinfo`` command to get statistical +data and perform operation on the slabs. By default ``slabinfo`` only lists +slabs that have data in them. See "slabinfo -h" for more options when +running the command. ``slabinfo`` can be compiled with +:: + + gcc -o slabinfo tools/vm/slabinfo.c + +Some of the modes of operation of ``slabinfo`` require that slub debugging +be enabled on the command line. F.e. no tracking information will be +available without debugging on and validation can only partially +be performed if debugging was not switched on. + +Some more sophisticated uses of slub_debug: +------------------------------------------- + +Parameters may be given to ``slub_debug``. If none is specified then full +debugging is enabled. Format: + +slub_debug= + Enable options for all slabs +slub_debug=, + Enable options only for select slabs + + +Possible debug options are:: + + F Sanity checks on (enables SLAB_DEBUG_CONSISTENCY_CHECKS + Sorry SLAB legacy issues) + Z Red zoning + P Poisoning (object and padding) + U User tracking (free and alloc) + T Trace (please only use on single slabs) + A Toggle failslab filter mark for the cache + O Switch debugging off for caches that would have + caused higher minimum slab orders + - Switch all debugging off (useful if the kernel is + configured with CONFIG_SLUB_DEBUG_ON) + +F.e. in order to boot just with sanity checks and red zoning one would specify:: + + slub_debug=FZ + +Trying to find an issue in the dentry cache? Try:: + + slub_debug=,dentry + +to only enable debugging on the dentry cache. + +Red zoning and tracking may realign the slab. We can just apply sanity checks +to the dentry cache with:: + + slub_debug=F,dentry + +Debugging options may require the minimum possible slab order to increase as +a result of storing the metadata (for example, caches with PAGE_SIZE object +sizes). This has a higher liklihood of resulting in slab allocation errors +in low memory situations or if there's high fragmentation of memory. To +switch off debugging for such caches by default, use:: + + slub_debug=O + +In case you forgot to enable debugging on the kernel command line: It is +possible to enable debugging manually when the kernel is up. Look at the +contents of:: + + /sys/kernel/slab// + +Look at the writable files. Writing 1 to them will enable the +corresponding debug option. All options can be set on a slab that does +not contain objects. If the slab already contains objects then sanity checks +and tracing may only be enabled. The other options may cause the realignment +of objects. + +Careful with tracing: It may spew out lots of information and never stop if +used on the wrong slab. + +Slab merging +============ + +If no debug options are specified then SLUB may merge similar slabs together +in order to reduce overhead and increase cache hotness of objects. +``slabinfo -a`` displays which slabs were merged together. + +Slab validation +=============== + +SLUB can validate all object if the kernel was booted with slub_debug. In +order to do so you must have the ``slabinfo`` tool. Then you can do +:: + + slabinfo -v + +which will test all objects. Output will be generated to the syslog. + +This also works in a more limited way if boot was without slab debug. +In that case ``slabinfo -v`` simply tests all reachable objects. Usually +these are in the cpu slabs and the partial slabs. Full slabs are not +tracked by SLUB in a non debug situation. + +Getting more performance +======================== + +To some degree SLUB's performance is limited by the need to take the +list_lock once in a while to deal with partial slabs. That overhead is +governed by the order of the allocation for each slab. The allocations +can be influenced by kernel parameters: + +.. slub_min_objects=x (default 4) +.. slub_min_order=x (default 0) +.. slub_max_order=x (default 3 (PAGE_ALLOC_COSTLY_ORDER)) + +``slub_min_objects`` + allows to specify how many objects must at least fit into one + slab in order for the allocation order to be acceptable. In + general slub will be able to perform this number of + allocations on a slab without consulting centralized resources + (list_lock) where contention may occur. + +``slub_min_order`` + specifies a minim order of slabs. A similar effect like + ``slub_min_objects``. + +``slub_max_order`` + specified the order at which ``slub_min_objects`` should no + longer be checked. This is useful to avoid SLUB trying to + generate super large order pages to fit ``slub_min_objects`` + of a slab cache with large object sizes into one high order + page. Setting command line parameter + ``debug_guardpage_minorder=N`` (N > 0), forces setting + ``slub_max_order`` to 0, what cause minimum possible order of + slabs allocation. + +SLUB Debug output +================= + +Here is a sample of slub debug output:: + + ==================================================================== + BUG kmalloc-8: Redzone overwritten + -------------------------------------------------------------------- + + INFO: 0xc90f6d28-0xc90f6d2b. First byte 0x00 instead of 0xcc + INFO: Slab 0xc528c530 flags=0x400000c3 inuse=61 fp=0xc90f6d58 + INFO: Object 0xc90f6d20 @offset=3360 fp=0xc90f6d58 + INFO: Allocated in get_modalias+0x61/0xf5 age=53 cpu=1 pid=554 + + Bytes b4 0xc90f6d10: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ + Object 0xc90f6d20: 31 30 31 39 2e 30 30 35 1019.005 + Redzone 0xc90f6d28: 00 cc cc cc . + Padding 0xc90f6d50: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ + + [] dump_trace+0x63/0x1eb + [] show_trace_log_lvl+0x1a/0x2f + [] show_trace+0x12/0x14 + [] dump_stack+0x16/0x18 + [] object_err+0x143/0x14b + [] check_object+0x66/0x234 + [] __slab_free+0x239/0x384 + [] kfree+0xa6/0xc6 + [] get_modalias+0xb9/0xf5 + [] dmi_dev_uevent+0x27/0x3c + [] dev_uevent+0x1ad/0x1da + [] kobject_uevent_env+0x20a/0x45b + [] kobject_uevent+0xa/0xf + [] store_uevent+0x4f/0x58 + [] dev_attr_store+0x29/0x2f + [] sysfs_write_file+0x16e/0x19c + [] vfs_write+0xd1/0x15a + [] sys_write+0x3d/0x72 + [] sysenter_past_esp+0x5f/0x99 + [] 0xb7f7b410 + ======================= + + FIX kmalloc-8: Restoring Redzone 0xc90f6d28-0xc90f6d2b=0xcc + +If SLUB encounters a corrupted object (full detection requires the kernel +to be booted with slub_debug) then the following output will be dumped +into the syslog: + +1. Description of the problem encountered + + This will be a message in the system log starting with:: + + =============================================== + BUG : + ----------------------------------------------- + + INFO: - + INFO: Slab
+ INFO: Object
+ INFO: Allocated in age= cpu= pid= + INFO: Freed in age= cpu= + pid= + + (Object allocation / free information is only available if SLAB_STORE_USER is + set for the slab. slub_debug sets that option) + +2. The object contents if an object was involved. + + Various types of lines can follow the BUG SLUB line: + + Bytes b4
: + Shows a few bytes before the object where the problem was detected. + Can be useful if the corruption does not stop with the start of the + object. + + Object
: + The bytes of the object. If the object is inactive then the bytes + typically contain poison values. Any non-poison value shows a + corruption by a write after free. + + Redzone
: + The Redzone following the object. The Redzone is used to detect + writes after the object. All bytes should always have the same + value. If there is any deviation then it is due to a write after + the object boundary. + + (Redzone information is only available if SLAB_RED_ZONE is set. + slub_debug sets that option) + + Padding
: + Unused data to fill up the space in order to get the next object + properly aligned. In the debug case we make sure that there are + at least 4 bytes of padding. This allows the detection of writes + before the object. + +3. A stackdump + + The stackdump describes the location where the error was detected. The cause + of the corruption is may be more likely found by looking at the function that + allocated or freed the object. + +4. Report on how the problem was dealt with in order to ensure the continued + operation of the system. + + These are messages in the system log beginning with:: + + FIX : + + In the above sample SLUB found that the Redzone of an active object has + been overwritten. Here a string of 8 characters was written into a slab that + has the length of 8 characters. However, a 8 character string needs a + terminating 0. That zero has overwritten the first byte of the Redzone field. + After reporting the details of the issue encountered the FIX SLUB message + tells us that SLUB has restored the Redzone to its proper value and then + system operations continue. + +Emergency operations +==================== + +Minimal debugging (sanity checks alone) can be enabled by booting with:: + + slub_debug=F + +This will be generally be enough to enable the resiliency features of slub +which will keep the system running even if a bad kernel component will +keep corrupting objects. This may be important for production systems. +Performance will be impacted by the sanity checks and there will be a +continual stream of error messages to the syslog but no additional memory +will be used (unlike full debugging). + +No guarantees. The kernel component still needs to be fixed. Performance +may be optimized further by locating the slab that experiences corruption +and enabling debugging only for that cache + +I.e.:: + + slub_debug=F,dentry + +If the corruption occurs by writing after the end of the object then it +may be advisable to enable a Redzone to avoid corrupting the beginning +of other objects:: + + slub_debug=FZ,dentry + +Extended slabinfo mode and plotting +=================================== + +The ``slabinfo`` tool has a special 'extended' ('-X') mode that includes: + - Slabcache Totals + - Slabs sorted by size (up to -N slabs, default 1) + - Slabs sorted by loss (up to -N slabs, default 1) + +Additionally, in this mode ``slabinfo`` does not dynamically scale +sizes (G/M/K) and reports everything in bytes (this functionality is +also available to other slabinfo modes via '-B' option) which makes +reporting more precise and accurate. Moreover, in some sense the `-X' +mode also simplifies the analysis of slabs' behaviour, because its +output can be plotted using the ``slabinfo-gnuplot.sh`` script. So it +pushes the analysis from looking through the numbers (tons of numbers) +to something easier -- visual analysis. + +To generate plots: + +a) collect slabinfo extended records, for example:: + + while [ 1 ]; do slabinfo -X >> FOO_STATS; sleep 1; done + +b) pass stats file(-s) to ``slabinfo-gnuplot.sh`` script:: + + slabinfo-gnuplot.sh FOO_STATS [FOO_STATS2 .. FOO_STATSN] + + The ``slabinfo-gnuplot.sh`` script will pre-processes the collected records + and generates 3 png files (and 3 pre-processing cache files) per STATS + file: + - Slabcache Totals: FOO_STATS-totals.png + - Slabs sorted by size: FOO_STATS-slabs-by-size.png + - Slabs sorted by loss: FOO_STATS-slabs-by-loss.png + +Another use case, when ``slabinfo-gnuplot.sh`` can be useful, is when you +need to compare slabs' behaviour "prior to" and "after" some code +modification. To help you out there, ``slabinfo-gnuplot.sh`` script +can 'merge' the `Slabcache Totals` sections from different +measurements. To visually compare N plots: + +a) Collect as many STATS1, STATS2, .. STATSN files as you need:: + + while [ 1 ]; do slabinfo -X >> STATS; sleep 1; done + +b) Pre-process those STATS files:: + + slabinfo-gnuplot.sh STATS1 STATS2 .. STATSN + +c) Execute ``slabinfo-gnuplot.sh`` in '-t' mode, passing all of the + generated pre-processed \*-totals:: + + slabinfo-gnuplot.sh -t STATS1-totals STATS2-totals .. STATSN-totals + + This will produce a single plot (png file). + + Plots, expectedly, can be large so some fluctuations or small spikes + can go unnoticed. To deal with that, ``slabinfo-gnuplot.sh`` has two + options to 'zoom-in'/'zoom-out': + + a) ``-s %d,%d`` -- overwrites the default image width and heigh + b) ``-r %d,%d`` -- specifies a range of samples to use (for example, + in ``slabinfo -X >> FOO_STATS; sleep 1;`` case, using a ``-r + 40,60`` range will plot only samples collected between 40th and + 60th seconds). + +Christoph Lameter, May 30, 2007 +Sergey Senozhatsky, October 23, 2015 diff --git a/Documentation/vm/slub.txt b/Documentation/vm/slub.txt deleted file mode 100644 index 84652419bff2..000000000000 --- a/Documentation/vm/slub.txt +++ /dev/null @@ -1,342 +0,0 @@ -Short users guide for SLUB --------------------------- - -The basic philosophy of SLUB is very different from SLAB. SLAB -requires rebuilding the kernel to activate debug options for all -slab caches. SLUB always includes full debugging but it is off by default. -SLUB can enable debugging only for selected slabs in order to avoid -an impact on overall system performance which may make a bug more -difficult to find. - -In order to switch debugging on one can add an option "slub_debug" -to the kernel command line. That will enable full debugging for -all slabs. - -Typically one would then use the "slabinfo" command to get statistical -data and perform operation on the slabs. By default slabinfo only lists -slabs that have data in them. See "slabinfo -h" for more options when -running the command. slabinfo can be compiled with - -gcc -o slabinfo tools/vm/slabinfo.c - -Some of the modes of operation of slabinfo require that slub debugging -be enabled on the command line. F.e. no tracking information will be -available without debugging on and validation can only partially -be performed if debugging was not switched on. - -Some more sophisticated uses of slub_debug: -------------------------------------------- - -Parameters may be given to slub_debug. If none is specified then full -debugging is enabled. Format: - -slub_debug= Enable options for all slabs -slub_debug=, - Enable options only for select slabs - -Possible debug options are - F Sanity checks on (enables SLAB_DEBUG_CONSISTENCY_CHECKS - Sorry SLAB legacy issues) - Z Red zoning - P Poisoning (object and padding) - U User tracking (free and alloc) - T Trace (please only use on single slabs) - A Toggle failslab filter mark for the cache - O Switch debugging off for caches that would have - caused higher minimum slab orders - - Switch all debugging off (useful if the kernel is - configured with CONFIG_SLUB_DEBUG_ON) - -F.e. in order to boot just with sanity checks and red zoning one would specify: - - slub_debug=FZ - -Trying to find an issue in the dentry cache? Try - - slub_debug=,dentry - -to only enable debugging on the dentry cache. - -Red zoning and tracking may realign the slab. We can just apply sanity checks -to the dentry cache with - - slub_debug=F,dentry - -Debugging options may require the minimum possible slab order to increase as -a result of storing the metadata (for example, caches with PAGE_SIZE object -sizes). This has a higher liklihood of resulting in slab allocation errors -in low memory situations or if there's high fragmentation of memory. To -switch off debugging for such caches by default, use - - slub_debug=O - -In case you forgot to enable debugging on the kernel command line: It is -possible to enable debugging manually when the kernel is up. Look at the -contents of: - -/sys/kernel/slab// - -Look at the writable files. Writing 1 to them will enable the -corresponding debug option. All options can be set on a slab that does -not contain objects. If the slab already contains objects then sanity checks -and tracing may only be enabled. The other options may cause the realignment -of objects. - -Careful with tracing: It may spew out lots of information and never stop if -used on the wrong slab. - -Slab merging ------------- - -If no debug options are specified then SLUB may merge similar slabs together -in order to reduce overhead and increase cache hotness of objects. -slabinfo -a displays which slabs were merged together. - -Slab validation ---------------- - -SLUB can validate all object if the kernel was booted with slub_debug. In -order to do so you must have the slabinfo tool. Then you can do - -slabinfo -v - -which will test all objects. Output will be generated to the syslog. - -This also works in a more limited way if boot was without slab debug. -In that case slabinfo -v simply tests all reachable objects. Usually -these are in the cpu slabs and the partial slabs. Full slabs are not -tracked by SLUB in a non debug situation. - -Getting more performance ------------------------- - -To some degree SLUB's performance is limited by the need to take the -list_lock once in a while to deal with partial slabs. That overhead is -governed by the order of the allocation for each slab. The allocations -can be influenced by kernel parameters: - -slub_min_objects=x (default 4) -slub_min_order=x (default 0) -slub_max_order=x (default 3 (PAGE_ALLOC_COSTLY_ORDER)) - -slub_min_objects allows to specify how many objects must at least fit -into one slab in order for the allocation order to be acceptable. -In general slub will be able to perform this number of allocations -on a slab without consulting centralized resources (list_lock) where -contention may occur. - -slub_min_order specifies a minim order of slabs. A similar effect like -slub_min_objects. - -slub_max_order specified the order at which slub_min_objects should no -longer be checked. This is useful to avoid SLUB trying to generate -super large order pages to fit slub_min_objects of a slab cache with -large object sizes into one high order page. Setting command line -parameter debug_guardpage_minorder=N (N > 0), forces setting -slub_max_order to 0, what cause minimum possible order of slabs -allocation. - -SLUB Debug output ------------------ - -Here is a sample of slub debug output: - -==================================================================== -BUG kmalloc-8: Redzone overwritten --------------------------------------------------------------------- - -INFO: 0xc90f6d28-0xc90f6d2b. First byte 0x00 instead of 0xcc -INFO: Slab 0xc528c530 flags=0x400000c3 inuse=61 fp=0xc90f6d58 -INFO: Object 0xc90f6d20 @offset=3360 fp=0xc90f6d58 -INFO: Allocated in get_modalias+0x61/0xf5 age=53 cpu=1 pid=554 - -Bytes b4 0xc90f6d10: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ - Object 0xc90f6d20: 31 30 31 39 2e 30 30 35 1019.005 - Redzone 0xc90f6d28: 00 cc cc cc . - Padding 0xc90f6d50: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ - - [] dump_trace+0x63/0x1eb - [] show_trace_log_lvl+0x1a/0x2f - [] show_trace+0x12/0x14 - [] dump_stack+0x16/0x18 - [] object_err+0x143/0x14b - [] check_object+0x66/0x234 - [] __slab_free+0x239/0x384 - [] kfree+0xa6/0xc6 - [] get_modalias+0xb9/0xf5 - [] dmi_dev_uevent+0x27/0x3c - [] dev_uevent+0x1ad/0x1da - [] kobject_uevent_env+0x20a/0x45b - [] kobject_uevent+0xa/0xf - [] store_uevent+0x4f/0x58 - [] dev_attr_store+0x29/0x2f - [] sysfs_write_file+0x16e/0x19c - [] vfs_write+0xd1/0x15a - [] sys_write+0x3d/0x72 - [] sysenter_past_esp+0x5f/0x99 - [] 0xb7f7b410 - ======================= - -FIX kmalloc-8: Restoring Redzone 0xc90f6d28-0xc90f6d2b=0xcc - -If SLUB encounters a corrupted object (full detection requires the kernel -to be booted with slub_debug) then the following output will be dumped -into the syslog: - -1. Description of the problem encountered - -This will be a message in the system log starting with - -=============================================== -BUG : ------------------------------------------------ - -INFO: - -INFO: Slab
-INFO: Object
-INFO: Allocated in age= cpu= pid= -INFO: Freed in age= cpu= - pid= - -(Object allocation / free information is only available if SLAB_STORE_USER is -set for the slab. slub_debug sets that option) - -2. The object contents if an object was involved. - -Various types of lines can follow the BUG SLUB line: - -Bytes b4
: - Shows a few bytes before the object where the problem was detected. - Can be useful if the corruption does not stop with the start of the - object. - -Object
: - The bytes of the object. If the object is inactive then the bytes - typically contain poison values. Any non-poison value shows a - corruption by a write after free. - -Redzone
: - The Redzone following the object. The Redzone is used to detect - writes after the object. All bytes should always have the same - value. If there is any deviation then it is due to a write after - the object boundary. - - (Redzone information is only available if SLAB_RED_ZONE is set. - slub_debug sets that option) - -Padding
: - Unused data to fill up the space in order to get the next object - properly aligned. In the debug case we make sure that there are - at least 4 bytes of padding. This allows the detection of writes - before the object. - -3. A stackdump - -The stackdump describes the location where the error was detected. The cause -of the corruption is may be more likely found by looking at the function that -allocated or freed the object. - -4. Report on how the problem was dealt with in order to ensure the continued -operation of the system. - -These are messages in the system log beginning with - -FIX : - -In the above sample SLUB found that the Redzone of an active object has -been overwritten. Here a string of 8 characters was written into a slab that -has the length of 8 characters. However, a 8 character string needs a -terminating 0. That zero has overwritten the first byte of the Redzone field. -After reporting the details of the issue encountered the FIX SLUB message -tells us that SLUB has restored the Redzone to its proper value and then -system operations continue. - -Emergency operations: ---------------------- - -Minimal debugging (sanity checks alone) can be enabled by booting with - - slub_debug=F - -This will be generally be enough to enable the resiliency features of slub -which will keep the system running even if a bad kernel component will -keep corrupting objects. This may be important for production systems. -Performance will be impacted by the sanity checks and there will be a -continual stream of error messages to the syslog but no additional memory -will be used (unlike full debugging). - -No guarantees. The kernel component still needs to be fixed. Performance -may be optimized further by locating the slab that experiences corruption -and enabling debugging only for that cache - -I.e. - - slub_debug=F,dentry - -If the corruption occurs by writing after the end of the object then it -may be advisable to enable a Redzone to avoid corrupting the beginning -of other objects. - - slub_debug=FZ,dentry - -Extended slabinfo mode and plotting ------------------------------------ - -The slabinfo tool has a special 'extended' ('-X') mode that includes: - - Slabcache Totals - - Slabs sorted by size (up to -N slabs, default 1) - - Slabs sorted by loss (up to -N slabs, default 1) - -Additionally, in this mode slabinfo does not dynamically scale sizes (G/M/K) -and reports everything in bytes (this functionality is also available to -other slabinfo modes via '-B' option) which makes reporting more precise and -accurate. Moreover, in some sense the `-X' mode also simplifies the analysis -of slabs' behaviour, because its output can be plotted using the -slabinfo-gnuplot.sh script. So it pushes the analysis from looking through -the numbers (tons of numbers) to something easier -- visual analysis. - -To generate plots: -a) collect slabinfo extended records, for example: - - while [ 1 ]; do slabinfo -X >> FOO_STATS; sleep 1; done - -b) pass stats file(-s) to slabinfo-gnuplot.sh script: - slabinfo-gnuplot.sh FOO_STATS [FOO_STATS2 .. FOO_STATSN] - -The slabinfo-gnuplot.sh script will pre-processes the collected records -and generates 3 png files (and 3 pre-processing cache files) per STATS -file: - - Slabcache Totals: FOO_STATS-totals.png - - Slabs sorted by size: FOO_STATS-slabs-by-size.png - - Slabs sorted by loss: FOO_STATS-slabs-by-loss.png - -Another use case, when slabinfo-gnuplot can be useful, is when you need -to compare slabs' behaviour "prior to" and "after" some code modification. -To help you out there, slabinfo-gnuplot.sh script can 'merge' the -`Slabcache Totals` sections from different measurements. To visually -compare N plots: - -a) Collect as many STATS1, STATS2, .. STATSN files as you need - while [ 1 ]; do slabinfo -X >> STATS; sleep 1; done - -b) Pre-process those STATS files - slabinfo-gnuplot.sh STATS1 STATS2 .. STATSN - -c) Execute slabinfo-gnuplot.sh in '-t' mode, passing all of the -generated pre-processed *-totals - slabinfo-gnuplot.sh -t STATS1-totals STATS2-totals .. STATSN-totals - -This will produce a single plot (png file). - -Plots, expectedly, can be large so some fluctuations or small spikes -can go unnoticed. To deal with that, `slabinfo-gnuplot.sh' has two -options to 'zoom-in'/'zoom-out': - a) -s %d,%d overwrites the default image width and heigh - b) -r %d,%d specifies a range of samples to use (for example, - in `slabinfo -X >> FOO_STATS; sleep 1;' case, using - a "-r 40,60" range will plot only samples collected - between 40th and 60th seconds). - -Christoph Lameter, May 30, 2007 -Sergey Senozhatsky, October 23, 2015 diff --git a/Documentation/vm/soft-dirty.txt b/Documentation/vm/soft-dirty.txt deleted file mode 100644 index 55684d11a1e8..000000000000 --- a/Documentation/vm/soft-dirty.txt +++ /dev/null @@ -1,43 +0,0 @@ - SOFT-DIRTY PTEs - - The soft-dirty is a bit on a PTE which helps to track which pages a task -writes to. In order to do this tracking one should - - 1. Clear soft-dirty bits from the task's PTEs. - - This is done by writing "4" into the /proc/PID/clear_refs file of the - task in question. - - 2. Wait some time. - - 3. Read soft-dirty bits from the PTEs. - - This is done by reading from the /proc/PID/pagemap. The bit 55 of the - 64-bit qword is the soft-dirty one. If set, the respective PTE was - written to since step 1. - - - Internally, to do this tracking, the writable bit is cleared from PTEs -when the soft-dirty bit is cleared. So, after this, when the task tries to -modify a page at some virtual address the #PF occurs and the kernel sets -the soft-dirty bit on the respective PTE. - - Note, that although all the task's address space is marked as r/o after the -soft-dirty bits clear, the #PF-s that occur after that are processed fast. -This is so, since the pages are still mapped to physical memory, and thus all -the kernel does is finds this fact out and puts both writable and soft-dirty -bits on the PTE. - - While in most cases tracking memory changes by #PF-s is more than enough -there is still a scenario when we can lose soft dirty bits -- a task -unmaps a previously mapped memory region and then maps a new one at exactly -the same place. When unmap is called, the kernel internally clears PTE values -including soft dirty bits. To notify user space application about such -memory region renewal the kernel always marks new memory regions (and -expanded regions) as soft dirty. - - This feature is actively used by the checkpoint-restore project. You -can find more details about it on http://criu.org - - --- Pavel Emelyanov, Apr 9, 2013 diff --git a/Documentation/vm/split_page_table_lock b/Documentation/vm/split_page_table_lock deleted file mode 100644 index 62842a857dab..000000000000 --- a/Documentation/vm/split_page_table_lock +++ /dev/null @@ -1,94 +0,0 @@ -Split page table lock -===================== - -Originally, mm->page_table_lock spinlock protected all page tables of the -mm_struct. But this approach leads to poor page fault scalability of -multi-threaded applications due high contention on the lock. To improve -scalability, split page table lock was introduced. - -With split page table lock we have separate per-table lock to serialize -access to the table. At the moment we use split lock for PTE and PMD -tables. Access to higher level tables protected by mm->page_table_lock. - -There are helpers to lock/unlock a table and other accessor functions: - - pte_offset_map_lock() - maps pte and takes PTE table lock, returns pointer to the taken - lock; - - pte_unmap_unlock() - unlocks and unmaps PTE table; - - pte_alloc_map_lock() - allocates PTE table if needed and take the lock, returns pointer - to taken lock or NULL if allocation failed; - - pte_lockptr() - returns pointer to PTE table lock; - - pmd_lock() - takes PMD table lock, returns pointer to taken lock; - - pmd_lockptr() - returns pointer to PMD table lock; - -Split page table lock for PTE tables is enabled compile-time if -CONFIG_SPLIT_PTLOCK_CPUS (usually 4) is less or equal to NR_CPUS. -If split lock is disabled, all tables guaded by mm->page_table_lock. - -Split page table lock for PMD tables is enabled, if it's enabled for PTE -tables and the architecture supports it (see below). - -Hugetlb and split page table lock ---------------------------------- - -Hugetlb can support several page sizes. We use split lock only for PMD -level, but not for PUD. - -Hugetlb-specific helpers: - - huge_pte_lock() - takes pmd split lock for PMD_SIZE page, mm->page_table_lock - otherwise; - - huge_pte_lockptr() - returns pointer to table lock; - -Support of split page table lock by an architecture ---------------------------------------------------- - -There's no need in special enabling of PTE split page table lock: -everything required is done by pgtable_page_ctor() and pgtable_page_dtor(), -which must be called on PTE table allocation / freeing. - -Make sure the architecture doesn't use slab allocator for page table -allocation: slab uses page->slab_cache for its pages. -This field shares storage with page->ptl. - -PMD split lock only makes sense if you have more than two page table -levels. - -PMD split lock enabling requires pgtable_pmd_page_ctor() call on PMD table -allocation and pgtable_pmd_page_dtor() on freeing. - -Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and -pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing -paths: i.e X86_PAE preallocate few PMDs on pgd_alloc(). - -With everything in place you can set CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK. - -NOTE: pgtable_page_ctor() and pgtable_pmd_page_ctor() can fail -- it must -be handled properly. - -page->ptl ---------- - -page->ptl is used to access split page table lock, where 'page' is struct -page of page containing the table. It shares storage with page->private -(and few other fields in union). - -To avoid increasing size of struct page and have best performance, we use a -trick: - - if spinlock_t fits into long, we use page->ptr as spinlock, so we - can avoid indirect access and save a cache line. - - if size of spinlock_t is bigger then size of long, we use page->ptl as - pointer to spinlock_t and allocate it dynamically. This allows to use - split lock with enabled DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC, but costs - one more cache line for indirect access; - -The spinlock_t allocated in pgtable_page_ctor() for PTE table and in -pgtable_pmd_page_ctor() for PMD table. - -Please, never access page->ptl directly -- use appropriate helper. diff --git a/Documentation/vm/split_page_table_lock.rst b/Documentation/vm/split_page_table_lock.rst new file mode 100644 index 000000000000..889b00be469f --- /dev/null +++ b/Documentation/vm/split_page_table_lock.rst @@ -0,0 +1,100 @@ +.. _split_page_table_lock: + +===================== +Split page table lock +===================== + +Originally, mm->page_table_lock spinlock protected all page tables of the +mm_struct. But this approach leads to poor page fault scalability of +multi-threaded applications due high contention on the lock. To improve +scalability, split page table lock was introduced. + +With split page table lock we have separate per-table lock to serialize +access to the table. At the moment we use split lock for PTE and PMD +tables. Access to higher level tables protected by mm->page_table_lock. + +There are helpers to lock/unlock a table and other accessor functions: + + - pte_offset_map_lock() + maps pte and takes PTE table lock, returns pointer to the taken + lock; + - pte_unmap_unlock() + unlocks and unmaps PTE table; + - pte_alloc_map_lock() + allocates PTE table if needed and take the lock, returns pointer + to taken lock or NULL if allocation failed; + - pte_lockptr() + returns pointer to PTE table lock; + - pmd_lock() + takes PMD table lock, returns pointer to taken lock; + - pmd_lockptr() + returns pointer to PMD table lock; + +Split page table lock for PTE tables is enabled compile-time if +CONFIG_SPLIT_PTLOCK_CPUS (usually 4) is less or equal to NR_CPUS. +If split lock is disabled, all tables guaded by mm->page_table_lock. + +Split page table lock for PMD tables is enabled, if it's enabled for PTE +tables and the architecture supports it (see below). + +Hugetlb and split page table lock +================================= + +Hugetlb can support several page sizes. We use split lock only for PMD +level, but not for PUD. + +Hugetlb-specific helpers: + + - huge_pte_lock() + takes pmd split lock for PMD_SIZE page, mm->page_table_lock + otherwise; + - huge_pte_lockptr() + returns pointer to table lock; + +Support of split page table lock by an architecture +=================================================== + +There's no need in special enabling of PTE split page table lock: +everything required is done by pgtable_page_ctor() and pgtable_page_dtor(), +which must be called on PTE table allocation / freeing. + +Make sure the architecture doesn't use slab allocator for page table +allocation: slab uses page->slab_cache for its pages. +This field shares storage with page->ptl. + +PMD split lock only makes sense if you have more than two page table +levels. + +PMD split lock enabling requires pgtable_pmd_page_ctor() call on PMD table +allocation and pgtable_pmd_page_dtor() on freeing. + +Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and +pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing +paths: i.e X86_PAE preallocate few PMDs on pgd_alloc(). + +With everything in place you can set CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK. + +NOTE: pgtable_page_ctor() and pgtable_pmd_page_ctor() can fail -- it must +be handled properly. + +page->ptl +========= + +page->ptl is used to access split page table lock, where 'page' is struct +page of page containing the table. It shares storage with page->private +(and few other fields in union). + +To avoid increasing size of struct page and have best performance, we use a +trick: + + - if spinlock_t fits into long, we use page->ptr as spinlock, so we + can avoid indirect access and save a cache line. + - if size of spinlock_t is bigger then size of long, we use page->ptl as + pointer to spinlock_t and allocate it dynamically. This allows to use + split lock with enabled DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC, but costs + one more cache line for indirect access; + +The spinlock_t allocated in pgtable_page_ctor() for PTE table and in +pgtable_pmd_page_ctor() for PMD table. + +Please, never access page->ptl directly -- use appropriate helper. diff --git a/Documentation/vm/swap_numa.rst b/Documentation/vm/swap_numa.rst new file mode 100644 index 000000000000..e0466f2db8fa --- /dev/null +++ b/Documentation/vm/swap_numa.rst @@ -0,0 +1,80 @@ +.. _swap_numa: + +=========================================== +Automatically bind swap device to numa node +=========================================== + +If the system has more than one swap device and swap device has the node +information, we can make use of this information to decide which swap +device to use in get_swap_pages() to get better performance. + + +How to use this feature +======================= + +Swap device has priority and that decides the order of it to be used. To make +use of automatically binding, there is no need to manipulate priority settings +for swap devices. e.g. on a 2 node machine, assume 2 swap devices swapA and +swapB, with swapA attached to node 0 and swapB attached to node 1, are going +to be swapped on. Simply swapping them on by doing:: + + # swapon /dev/swapA + # swapon /dev/swapB + +Then node 0 will use the two swap devices in the order of swapA then swapB and +node 1 will use the two swap devices in the order of swapB then swapA. Note +that the order of them being swapped on doesn't matter. + +A more complex example on a 4 node machine. Assume 6 swap devices are going to +be swapped on: swapA and swapB are attached to node 0, swapC is attached to +node 1, swapD and swapE are attached to node 2 and swapF is attached to node3. +The way to swap them on is the same as above:: + + # swapon /dev/swapA + # swapon /dev/swapB + # swapon /dev/swapC + # swapon /dev/swapD + # swapon /dev/swapE + # swapon /dev/swapF + +Then node 0 will use them in the order of:: + + swapA/swapB -> swapC -> swapD -> swapE -> swapF + +swapA and swapB will be used in a round robin mode before any other swap device. + +node 1 will use them in the order of:: + + swapC -> swapA -> swapB -> swapD -> swapE -> swapF + +node 2 will use them in the order of:: + + swapD/swapE -> swapA -> swapB -> swapC -> swapF + +Similaly, swapD and swapE will be used in a round robin mode before any +other swap devices. + +node 3 will use them in the order of:: + + swapF -> swapA -> swapB -> swapC -> swapD -> swapE + + +Implementation details +====================== + +The current code uses a priority based list, swap_avail_list, to decide +which swap device to use and if multiple swap devices share the same +priority, they are used round robin. This change here replaces the single +global swap_avail_list with a per-numa-node list, i.e. for each numa node, +it sees its own priority based list of available swap devices. Swap +device's priority can be promoted on its matching node's swap_avail_list. + +The current swap device's priority is set as: user can set a >=0 value, +or the system will pick one starting from -1 then downwards. The priority +value in the swap_avail_list is the negated value of the swap device's +due to plist being sorted from low to high. The new policy doesn't change +the semantics for priority >=0 cases, the previous starting from -1 then +downwards now becomes starting from -2 then downwards and -1 is reserved +as the promoted value. So if multiple swap devices are attached to the same +node, they will all be promoted to priority -1 on that node's plist and will +be used round robin before any other swap devices. diff --git a/Documentation/vm/swap_numa.txt b/Documentation/vm/swap_numa.txt deleted file mode 100644 index d5960c9124f5..000000000000 --- a/Documentation/vm/swap_numa.txt +++ /dev/null @@ -1,69 +0,0 @@ -Automatically bind swap device to numa node -------------------------------------------- - -If the system has more than one swap device and swap device has the node -information, we can make use of this information to decide which swap -device to use in get_swap_pages() to get better performance. - - -How to use this feature ------------------------ - -Swap device has priority and that decides the order of it to be used. To make -use of automatically binding, there is no need to manipulate priority settings -for swap devices. e.g. on a 2 node machine, assume 2 swap devices swapA and -swapB, with swapA attached to node 0 and swapB attached to node 1, are going -to be swapped on. Simply swapping them on by doing: -# swapon /dev/swapA -# swapon /dev/swapB - -Then node 0 will use the two swap devices in the order of swapA then swapB and -node 1 will use the two swap devices in the order of swapB then swapA. Note -that the order of them being swapped on doesn't matter. - -A more complex example on a 4 node machine. Assume 6 swap devices are going to -be swapped on: swapA and swapB are attached to node 0, swapC is attached to -node 1, swapD and swapE are attached to node 2 and swapF is attached to node3. -The way to swap them on is the same as above: -# swapon /dev/swapA -# swapon /dev/swapB -# swapon /dev/swapC -# swapon /dev/swapD -# swapon /dev/swapE -# swapon /dev/swapF - -Then node 0 will use them in the order of: -swapA/swapB -> swapC -> swapD -> swapE -> swapF -swapA and swapB will be used in a round robin mode before any other swap device. - -node 1 will use them in the order of: -swapC -> swapA -> swapB -> swapD -> swapE -> swapF - -node 2 will use them in the order of: -swapD/swapE -> swapA -> swapB -> swapC -> swapF -Similaly, swapD and swapE will be used in a round robin mode before any -other swap devices. - -node 3 will use them in the order of: -swapF -> swapA -> swapB -> swapC -> swapD -> swapE - - -Implementation details ----------------------- - -The current code uses a priority based list, swap_avail_list, to decide -which swap device to use and if multiple swap devices share the same -priority, they are used round robin. This change here replaces the single -global swap_avail_list with a per-numa-node list, i.e. for each numa node, -it sees its own priority based list of available swap devices. Swap -device's priority can be promoted on its matching node's swap_avail_list. - -The current swap device's priority is set as: user can set a >=0 value, -or the system will pick one starting from -1 then downwards. The priority -value in the swap_avail_list is the negated value of the swap device's -due to plist being sorted from low to high. The new policy doesn't change -the semantics for priority >=0 cases, the previous starting from -1 then -downwards now becomes starting from -2 then downwards and -1 is reserved -as the promoted value. So if multiple swap devices are attached to the same -node, they will all be promoted to priority -1 on that node's plist and will -be used round robin before any other swap devices. diff --git a/Documentation/vm/transhuge.rst b/Documentation/vm/transhuge.rst new file mode 100644 index 000000000000..a8cf6809e36e --- /dev/null +++ b/Documentation/vm/transhuge.rst @@ -0,0 +1,197 @@ +.. _transhuge: + +============================ +Transparent Hugepage Support +============================ + +This document describes design principles Transparent Hugepage (THP) +Support and its interaction with other parts of the memory management. + +Design principles +================= + +- "graceful fallback": mm components which don't have transparent hugepage + knowledge fall back to breaking huge pmd mapping into table of ptes and, + if necessary, split a transparent hugepage. Therefore these components + can continue working on the regular pages or regular pte mappings. + +- if a hugepage allocation fails because of memory fragmentation, + regular pages should be gracefully allocated instead and mixed in + the same vma without any failure or significant delay and without + userland noticing + +- if some task quits and more hugepages become available (either + immediately in the buddy or through the VM), guest physical memory + backed by regular pages should be relocated on hugepages + automatically (with khugepaged) + +- it doesn't require memory reservation and in turn it uses hugepages + whenever possible (the only possible reservation here is kernelcore= + to avoid unmovable pages to fragment all the memory but such a tweak + is not specific to transparent hugepage support and it's a generic + feature that applies to all dynamic high order allocations in the + kernel) + +get_user_pages and follow_page +============================== + +get_user_pages and follow_page if run on a hugepage, will return the +head or tail pages as usual (exactly as they would do on +hugetlbfs). Most gup users will only care about the actual physical +address of the page and its temporary pinning to release after the I/O +is complete, so they won't ever notice the fact the page is huge. But +if any driver is going to mangle over the page structure of the tail +page (like for checking page->mapping or other bits that are relevant +for the head page and not the tail page), it should be updated to jump +to check head page instead. Taking reference on any head/tail page would +prevent page from being split by anyone. + +.. note:: + these aren't new constraints to the GUP API, and they match the + same constrains that applies to hugetlbfs too, so any driver capable + of handling GUP on hugetlbfs will also work fine on transparent + hugepage backed mappings. + +In case you can't handle compound pages if they're returned by +follow_page, the FOLL_SPLIT bit can be specified as parameter to +follow_page, so that it will split the hugepages before returning +them. Migration for example passes FOLL_SPLIT as parameter to +follow_page because it's not hugepage aware and in fact it can't work +at all on hugetlbfs (but it instead works fine on transparent +hugepages thanks to FOLL_SPLIT). migration simply can't deal with +hugepages being returned (as it's not only checking the pfn of the +page and pinning it during the copy but it pretends to migrate the +memory in regular page sizes and with regular pte/pmd mappings). + +Graceful fallback +================= + +Code walking pagetables but unaware about huge pmds can simply call +split_huge_pmd(vma, pmd, addr) where the pmd is the one returned by +pmd_offset. It's trivial to make the code transparent hugepage aware +by just grepping for "pmd_offset" and adding split_huge_pmd where +missing after pmd_offset returns the pmd. Thanks to the graceful +fallback design, with a one liner change, you can avoid to write +hundred if not thousand of lines of complex code to make your code +hugepage aware. + +If you're not walking pagetables but you run into a physical hugepage +but you can't handle it natively in your code, you can split it by +calling split_huge_page(page). This is what the Linux VM does before +it tries to swapout the hugepage for example. split_huge_page() can fail +if the page is pinned and you must handle this correctly. + +Example to make mremap.c transparent hugepage aware with a one liner +change:: + + diff --git a/mm/mremap.c b/mm/mremap.c + --- a/mm/mremap.c + +++ b/mm/mremap.c + @@ -41,6 +41,7 @@ static pmd_t *get_old_pmd(struct mm_stru + return NULL; + + pmd = pmd_offset(pud, addr); + + split_huge_pmd(vma, pmd, addr); + if (pmd_none_or_clear_bad(pmd)) + return NULL; + +Locking in hugepage aware code +============================== + +We want as much code as possible hugepage aware, as calling +split_huge_page() or split_huge_pmd() has a cost. + +To make pagetable walks huge pmd aware, all you need to do is to call +pmd_trans_huge() on the pmd returned by pmd_offset. You must hold the +mmap_sem in read (or write) mode to be sure an huge pmd cannot be +created from under you by khugepaged (khugepaged collapse_huge_page +takes the mmap_sem in write mode in addition to the anon_vma lock). If +pmd_trans_huge returns false, you just fallback in the old code +paths. If instead pmd_trans_huge returns true, you have to take the +page table lock (pmd_lock()) and re-run pmd_trans_huge. Taking the +page table lock will prevent the huge pmd to be converted into a +regular pmd from under you (split_huge_pmd can run in parallel to the +pagetable walk). If the second pmd_trans_huge returns false, you +should just drop the page table lock and fallback to the old code as +before. Otherwise you can proceed to process the huge pmd and the +hugepage natively. Once finished you can drop the page table lock. + +Refcounts and transparent huge pages +==================================== + +Refcounting on THP is mostly consistent with refcounting on other compound +pages: + + - get_page()/put_page() and GUP operate in head page's ->_refcount. + + - ->_refcount in tail pages is always zero: get_page_unless_zero() never + succeed on tail pages. + + - map/unmap of the pages with PTE entry increment/decrement ->_mapcount + on relevant sub-page of the compound page. + + - map/unmap of the whole compound page accounted in compound_mapcount + (stored in first tail page). For file huge pages, we also increment + ->_mapcount of all sub-pages in order to have race-free detection of + last unmap of subpages. + +PageDoubleMap() indicates that the page is *possibly* mapped with PTEs. + +For anonymous pages PageDoubleMap() also indicates ->_mapcount in all +subpages is offset up by one. This additional reference is required to +get race-free detection of unmap of subpages when we have them mapped with +both PMDs and PTEs. + +This is optimization required to lower overhead of per-subpage mapcount +tracking. The alternative is alter ->_mapcount in all subpages on each +map/unmap of the whole compound page. + +For anonymous pages, we set PG_double_map when a PMD of the page got split +for the first time, but still have PMD mapping. The additional references +go away with last compound_mapcount. + +File pages get PG_double_map set on first map of the page with PTE and +goes away when the page gets evicted from page cache. + +split_huge_page internally has to distribute the refcounts in the head +page to the tail pages before clearing all PG_head/tail bits from the page +structures. It can be done easily for refcounts taken by page table +entries. But we don't have enough information on how to distribute any +additional pins (i.e. from get_user_pages). split_huge_page() fails any +requests to split pinned huge page: it expects page count to be equal to +sum of mapcount of all sub-pages plus one (split_huge_page caller must +have reference for head page). + +split_huge_page uses migration entries to stabilize page->_refcount and +page->_mapcount of anonymous pages. File pages just got unmapped. + +We safe against physical memory scanners too: the only legitimate way +scanner can get reference to a page is get_page_unless_zero(). + +All tail pages have zero ->_refcount until atomic_add(). This prevents the +scanner from getting a reference to the tail page up to that point. After the +atomic_add() we don't care about the ->_refcount value. We already known how +many references should be uncharged from the head page. + +For head page get_page_unless_zero() will succeed and we don't mind. It's +clear where reference should go after split: it will stay on head page. + +Note that split_huge_pmd() doesn't have any limitation on refcounting: +pmd can be split at any point and never fails. + +Partial unmap and deferred_split_huge_page() +============================================ + +Unmapping part of THP (with munmap() or other way) is not going to free +memory immediately. Instead, we detect that a subpage of THP is not in use +in page_remove_rmap() and queue the THP for splitting if memory pressure +comes. Splitting will free up unused subpages. + +Splitting the page right away is not an option due to locking context in +the place where we can detect partial unmap. It's also might be +counterproductive since in many cases partial unmap happens during exit(2) if +a THP crosses a VMA boundary. + +Function deferred_split_huge_page() is used to queue page for splitting. +The splitting itself will happen when we get memory pressure via shrinker +interface. diff --git a/Documentation/vm/transhuge.txt b/Documentation/vm/transhuge.txt deleted file mode 100644 index 4dde03b44ad1..000000000000 --- a/Documentation/vm/transhuge.txt +++ /dev/null @@ -1,527 +0,0 @@ -= Transparent Hugepage Support = - -== Objective == - -Performance critical computing applications dealing with large memory -working sets are already running on top of libhugetlbfs and in turn -hugetlbfs. Transparent Hugepage Support is an alternative means of -using huge pages for the backing of virtual memory with huge pages -that supports the automatic promotion and demotion of page sizes and -without the shortcomings of hugetlbfs. - -Currently it only works for anonymous memory mappings and tmpfs/shmem. -But in the future it can expand to other filesystems. - -The reason applications are running faster is because of two -factors. The first factor is almost completely irrelevant and it's not -of significant interest because it'll also have the downside of -requiring larger clear-page copy-page in page faults which is a -potentially negative effect. The first factor consists in taking a -single page fault for each 2M virtual region touched by userland (so -reducing the enter/exit kernel frequency by a 512 times factor). This -only matters the first time the memory is accessed for the lifetime of -a memory mapping. The second long lasting and much more important -factor will affect all subsequent accesses to the memory for the whole -runtime of the application. The second factor consist of two -components: 1) the TLB miss will run faster (especially with -virtualization using nested pagetables but almost always also on bare -metal without virtualization) and 2) a single TLB entry will be -mapping a much larger amount of virtual memory in turn reducing the -number of TLB misses. With virtualization and nested pagetables the -TLB can be mapped of larger size only if both KVM and the Linux guest -are using hugepages but a significant speedup already happens if only -one of the two is using hugepages just because of the fact the TLB -miss is going to run faster. - -== Design == - -- "graceful fallback": mm components which don't have transparent hugepage - knowledge fall back to breaking huge pmd mapping into table of ptes and, - if necessary, split a transparent hugepage. Therefore these components - can continue working on the regular pages or regular pte mappings. - -- if a hugepage allocation fails because of memory fragmentation, - regular pages should be gracefully allocated instead and mixed in - the same vma without any failure or significant delay and without - userland noticing - -- if some task quits and more hugepages become available (either - immediately in the buddy or through the VM), guest physical memory - backed by regular pages should be relocated on hugepages - automatically (with khugepaged) - -- it doesn't require memory reservation and in turn it uses hugepages - whenever possible (the only possible reservation here is kernelcore= - to avoid unmovable pages to fragment all the memory but such a tweak - is not specific to transparent hugepage support and it's a generic - feature that applies to all dynamic high order allocations in the - kernel) - -Transparent Hugepage Support maximizes the usefulness of free memory -if compared to the reservation approach of hugetlbfs by allowing all -unused memory to be used as cache or other movable (or even unmovable -entities). It doesn't require reservation to prevent hugepage -allocation failures to be noticeable from userland. It allows paging -and all other advanced VM features to be available on the -hugepages. It requires no modifications for applications to take -advantage of it. - -Applications however can be further optimized to take advantage of -this feature, like for example they've been optimized before to avoid -a flood of mmap system calls for every malloc(4k). Optimizing userland -is by far not mandatory and khugepaged already can take care of long -lived page allocations even for hugepage unaware applications that -deals with large amounts of memory. - -In certain cases when hugepages are enabled system wide, application -may end up allocating more memory resources. An application may mmap a -large region but only touch 1 byte of it, in that case a 2M page might -be allocated instead of a 4k page for no good. This is why it's -possible to disable hugepages system-wide and to only have them inside -MADV_HUGEPAGE madvise regions. - -Embedded systems should enable hugepages only inside madvise regions -to eliminate any risk of wasting any precious byte of memory and to -only run faster. - -Applications that gets a lot of benefit from hugepages and that don't -risk to lose memory by using hugepages, should use -madvise(MADV_HUGEPAGE) on their critical mmapped regions. - -== sysfs == - -Transparent Hugepage Support for anonymous memory can be entirely disabled -(mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE -regions (to avoid the risk of consuming more memory resources) or enabled -system wide. This can be achieved with one of: - -echo always >/sys/kernel/mm/transparent_hugepage/enabled -echo madvise >/sys/kernel/mm/transparent_hugepage/enabled -echo never >/sys/kernel/mm/transparent_hugepage/enabled - -It's also possible to limit defrag efforts in the VM to generate -anonymous hugepages in case they're not immediately free to madvise -regions or to never try to defrag memory and simply fallback to regular -pages unless hugepages are immediately available. Clearly if we spend CPU -time to defrag memory, we would expect to gain even more by the fact we -use hugepages later instead of regular pages. This isn't always -guaranteed, but it may be more likely in case the allocation is for a -MADV_HUGEPAGE region. - -echo always >/sys/kernel/mm/transparent_hugepage/defrag -echo defer >/sys/kernel/mm/transparent_hugepage/defrag -echo defer+madvise >/sys/kernel/mm/transparent_hugepage/defrag -echo madvise >/sys/kernel/mm/transparent_hugepage/defrag -echo never >/sys/kernel/mm/transparent_hugepage/defrag - -"always" means that an application requesting THP will stall on allocation -failure and directly reclaim pages and compact memory in an effort to -allocate a THP immediately. This may be desirable for virtual machines -that benefit heavily from THP use and are willing to delay the VM start -to utilise them. - -"defer" means that an application will wake kswapd in the background -to reclaim pages and wake kcompactd to compact memory so that THP is -available in the near future. It's the responsibility of khugepaged -to then install the THP pages later. - -"defer+madvise" will enter direct reclaim and compaction like "always", but -only for regions that have used madvise(MADV_HUGEPAGE); all other regions -will wake kswapd in the background to reclaim pages and wake kcompactd to -compact memory so that THP is available in the near future. - -"madvise" will enter direct reclaim like "always" but only for regions -that are have used madvise(MADV_HUGEPAGE). This is the default behaviour. - -"never" should be self-explanatory. - -By default kernel tries to use huge zero page on read page fault to -anonymous mapping. It's possible to disable huge zero page by writing 0 -or enable it back by writing 1: - -echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page -echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page - -Some userspace (such as a test program, or an optimized memory allocation -library) may want to know the size (in bytes) of a transparent hugepage: - -cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size - -khugepaged will be automatically started when -transparent_hugepage/enabled is set to "always" or "madvise, and it'll -be automatically shutdown if it's set to "never". - -khugepaged runs usually at low frequency so while one may not want to -invoke defrag algorithms synchronously during the page faults, it -should be worth invoking defrag at least in khugepaged. However it's -also possible to disable defrag in khugepaged by writing 0 or enable -defrag in khugepaged by writing 1: - -echo 0 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag -echo 1 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag - -You can also control how many pages khugepaged should scan at each -pass: - -/sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan - -and how many milliseconds to wait in khugepaged between each pass (you -can set this to 0 to run khugepaged at 100% utilization of one core): - -/sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs - -and how many milliseconds to wait in khugepaged if there's an hugepage -allocation failure to throttle the next allocation attempt. - -/sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs - -The khugepaged progress can be seen in the number of pages collapsed: - -/sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed - -for each pass: - -/sys/kernel/mm/transparent_hugepage/khugepaged/full_scans - -max_ptes_none specifies how many extra small pages (that are -not already mapped) can be allocated when collapsing a group -of small pages into one large page. - -/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none - -A higher value leads to use additional memory for programs. -A lower value leads to gain less thp performance. Value of -max_ptes_none can waste cpu time very little, you can -ignore it. - -max_ptes_swap specifies how many pages can be brought in from -swap when collapsing a group of pages into a transparent huge page. - -/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_swap - -A higher value can cause excessive swap IO and waste -memory. A lower value can prevent THPs from being -collapsed, resulting fewer pages being collapsed into -THPs, and lower memory access performance. - -== Boot parameter == - -You can change the sysfs boot time defaults of Transparent Hugepage -Support by passing the parameter "transparent_hugepage=always" or -"transparent_hugepage=madvise" or "transparent_hugepage=never" -(without "") to the kernel command line. - -== Hugepages in tmpfs/shmem == - -You can control hugepage allocation policy in tmpfs with mount option -"huge=". It can have following values: - - - "always": - Attempt to allocate huge pages every time we need a new page; - - - "never": - Do not allocate huge pages; - - - "within_size": - Only allocate huge page if it will be fully within i_size. - Also respect fadvise()/madvise() hints; - - - "advise: - Only allocate huge pages if requested with fadvise()/madvise(); - -The default policy is "never". - -"mount -o remount,huge= /mountpoint" works fine after mount: remounting -huge=never will not attempt to break up huge pages at all, just stop more -from being allocated. - -There's also sysfs knob to control hugepage allocation policy for internal -shmem mount: /sys/kernel/mm/transparent_hugepage/shmem_enabled. The mount -is used for SysV SHM, memfds, shared anonymous mmaps (of /dev/zero or -MAP_ANONYMOUS), GPU drivers' DRM objects, Ashmem. - -In addition to policies listed above, shmem_enabled allows two further -values: - - - "deny": - For use in emergencies, to force the huge option off from - all mounts; - - "force": - Force the huge option on for all - very useful for testing; - -== Need of application restart == - -The transparent_hugepage/enabled values and tmpfs mount option only affect -future behavior. So to make them effective you need to restart any -application that could have been using hugepages. This also applies to the -regions registered in khugepaged. - -== Monitoring usage == - -The number of anonymous transparent huge pages currently used by the -system is available by reading the AnonHugePages field in /proc/meminfo. -To identify what applications are using anonymous transparent huge pages, -it is necessary to read /proc/PID/smaps and count the AnonHugePages fields -for each mapping. - -The number of file transparent huge pages mapped to userspace is available -by reading ShmemPmdMapped and ShmemHugePages fields in /proc/meminfo. -To identify what applications are mapping file transparent huge pages, it -is necessary to read /proc/PID/smaps and count the FileHugeMapped fields -for each mapping. - -Note that reading the smaps file is expensive and reading it -frequently will incur overhead. - -There are a number of counters in /proc/vmstat that may be used to -monitor how successfully the system is providing huge pages for use. - -thp_fault_alloc is incremented every time a huge page is successfully - allocated to handle a page fault. This applies to both the - first time a page is faulted and for COW faults. - -thp_collapse_alloc is incremented by khugepaged when it has found - a range of pages to collapse into one huge page and has - successfully allocated a new huge page to store the data. - -thp_fault_fallback is incremented if a page fault fails to allocate - a huge page and instead falls back to using small pages. - -thp_collapse_alloc_failed is incremented if khugepaged found a range - of pages that should be collapsed into one huge page but failed - the allocation. - -thp_file_alloc is incremented every time a file huge page is successfully - allocated. - -thp_file_mapped is incremented every time a file huge page is mapped into - user address space. - -thp_split_page is incremented every time a huge page is split into base - pages. This can happen for a variety of reasons but a common - reason is that a huge page is old and is being reclaimed. - This action implies splitting all PMD the page mapped with. - -thp_split_page_failed is incremented if kernel fails to split huge - page. This can happen if the page was pinned by somebody. - -thp_deferred_split_page is incremented when a huge page is put onto split - queue. This happens when a huge page is partially unmapped and - splitting it would free up some memory. Pages on split queue are - going to be split under memory pressure. - -thp_split_pmd is incremented every time a PMD split into table of PTEs. - This can happen, for instance, when application calls mprotect() or - munmap() on part of huge page. It doesn't split huge page, only - page table entry. - -thp_zero_page_alloc is incremented every time a huge zero page is - successfully allocated. It includes allocations which where - dropped due race with other allocation. Note, it doesn't count - every map of the huge zero page, only its allocation. - -thp_zero_page_alloc_failed is incremented if kernel fails to allocate - huge zero page and falls back to using small pages. - -As the system ages, allocating huge pages may be expensive as the -system uses memory compaction to copy data around memory to free a -huge page for use. There are some counters in /proc/vmstat to help -monitor this overhead. - -compact_stall is incremented every time a process stalls to run - memory compaction so that a huge page is free for use. - -compact_success is incremented if the system compacted memory and - freed a huge page for use. - -compact_fail is incremented if the system tries to compact memory - but failed. - -compact_pages_moved is incremented each time a page is moved. If - this value is increasing rapidly, it implies that the system - is copying a lot of data to satisfy the huge page allocation. - It is possible that the cost of copying exceeds any savings - from reduced TLB misses. - -compact_pagemigrate_failed is incremented when the underlying mechanism - for moving a page failed. - -compact_blocks_moved is incremented each time memory compaction examines - a huge page aligned range of pages. - -It is possible to establish how long the stalls were using the function -tracer to record how long was spent in __alloc_pages_nodemask and -using the mm_page_alloc tracepoint to identify which allocations were -for huge pages. - -== get_user_pages and follow_page == - -get_user_pages and follow_page if run on a hugepage, will return the -head or tail pages as usual (exactly as they would do on -hugetlbfs). Most gup users will only care about the actual physical -address of the page and its temporary pinning to release after the I/O -is complete, so they won't ever notice the fact the page is huge. But -if any driver is going to mangle over the page structure of the tail -page (like for checking page->mapping or other bits that are relevant -for the head page and not the tail page), it should be updated to jump -to check head page instead. Taking reference on any head/tail page would -prevent page from being split by anyone. - -NOTE: these aren't new constraints to the GUP API, and they match the -same constrains that applies to hugetlbfs too, so any driver capable -of handling GUP on hugetlbfs will also work fine on transparent -hugepage backed mappings. - -In case you can't handle compound pages if they're returned by -follow_page, the FOLL_SPLIT bit can be specified as parameter to -follow_page, so that it will split the hugepages before returning -them. Migration for example passes FOLL_SPLIT as parameter to -follow_page because it's not hugepage aware and in fact it can't work -at all on hugetlbfs (but it instead works fine on transparent -hugepages thanks to FOLL_SPLIT). migration simply can't deal with -hugepages being returned (as it's not only checking the pfn of the -page and pinning it during the copy but it pretends to migrate the -memory in regular page sizes and with regular pte/pmd mappings). - -== Optimizing the applications == - -To be guaranteed that the kernel will map a 2M page immediately in any -memory region, the mmap region has to be hugepage naturally -aligned. posix_memalign() can provide that guarantee. - -== Hugetlbfs == - -You can use hugetlbfs on a kernel that has transparent hugepage -support enabled just fine as always. No difference can be noted in -hugetlbfs other than there will be less overall fragmentation. All -usual features belonging to hugetlbfs are preserved and -unaffected. libhugetlbfs will also work fine as usual. - -== Graceful fallback == - -Code walking pagetables but unaware about huge pmds can simply call -split_huge_pmd(vma, pmd, addr) where the pmd is the one returned by -pmd_offset. It's trivial to make the code transparent hugepage aware -by just grepping for "pmd_offset" and adding split_huge_pmd where -missing after pmd_offset returns the pmd. Thanks to the graceful -fallback design, with a one liner change, you can avoid to write -hundred if not thousand of lines of complex code to make your code -hugepage aware. - -If you're not walking pagetables but you run into a physical hugepage -but you can't handle it natively in your code, you can split it by -calling split_huge_page(page). This is what the Linux VM does before -it tries to swapout the hugepage for example. split_huge_page() can fail -if the page is pinned and you must handle this correctly. - -Example to make mremap.c transparent hugepage aware with a one liner -change: - -diff --git a/mm/mremap.c b/mm/mremap.c ---- a/mm/mremap.c -+++ b/mm/mremap.c -@@ -41,6 +41,7 @@ static pmd_t *get_old_pmd(struct mm_stru - return NULL; - - pmd = pmd_offset(pud, addr); -+ split_huge_pmd(vma, pmd, addr); - if (pmd_none_or_clear_bad(pmd)) - return NULL; - -== Locking in hugepage aware code == - -We want as much code as possible hugepage aware, as calling -split_huge_page() or split_huge_pmd() has a cost. - -To make pagetable walks huge pmd aware, all you need to do is to call -pmd_trans_huge() on the pmd returned by pmd_offset. You must hold the -mmap_sem in read (or write) mode to be sure an huge pmd cannot be -created from under you by khugepaged (khugepaged collapse_huge_page -takes the mmap_sem in write mode in addition to the anon_vma lock). If -pmd_trans_huge returns false, you just fallback in the old code -paths. If instead pmd_trans_huge returns true, you have to take the -page table lock (pmd_lock()) and re-run pmd_trans_huge. Taking the -page table lock will prevent the huge pmd to be converted into a -regular pmd from under you (split_huge_pmd can run in parallel to the -pagetable walk). If the second pmd_trans_huge returns false, you -should just drop the page table lock and fallback to the old code as -before. Otherwise you can proceed to process the huge pmd and the -hugepage natively. Once finished you can drop the page table lock. - -== Refcounts and transparent huge pages == - -Refcounting on THP is mostly consistent with refcounting on other compound -pages: - - - get_page()/put_page() and GUP operate in head page's ->_refcount. - - - ->_refcount in tail pages is always zero: get_page_unless_zero() never - succeed on tail pages. - - - map/unmap of the pages with PTE entry increment/decrement ->_mapcount - on relevant sub-page of the compound page. - - - map/unmap of the whole compound page accounted in compound_mapcount - (stored in first tail page). For file huge pages, we also increment - ->_mapcount of all sub-pages in order to have race-free detection of - last unmap of subpages. - -PageDoubleMap() indicates that the page is *possibly* mapped with PTEs. - -For anonymous pages PageDoubleMap() also indicates ->_mapcount in all -subpages is offset up by one. This additional reference is required to -get race-free detection of unmap of subpages when we have them mapped with -both PMDs and PTEs. - -This is optimization required to lower overhead of per-subpage mapcount -tracking. The alternative is alter ->_mapcount in all subpages on each -map/unmap of the whole compound page. - -For anonymous pages, we set PG_double_map when a PMD of the page got split -for the first time, but still have PMD mapping. The additional references -go away with last compound_mapcount. - -File pages get PG_double_map set on first map of the page with PTE and -goes away when the page gets evicted from page cache. - -split_huge_page internally has to distribute the refcounts in the head -page to the tail pages before clearing all PG_head/tail bits from the page -structures. It can be done easily for refcounts taken by page table -entries. But we don't have enough information on how to distribute any -additional pins (i.e. from get_user_pages). split_huge_page() fails any -requests to split pinned huge page: it expects page count to be equal to -sum of mapcount of all sub-pages plus one (split_huge_page caller must -have reference for head page). - -split_huge_page uses migration entries to stabilize page->_refcount and -page->_mapcount of anonymous pages. File pages just got unmapped. - -We safe against physical memory scanners too: the only legitimate way -scanner can get reference to a page is get_page_unless_zero(). - -All tail pages have zero ->_refcount until atomic_add(). This prevents the -scanner from getting a reference to the tail page up to that point. After the -atomic_add() we don't care about the ->_refcount value. We already known how -many references should be uncharged from the head page. - -For head page get_page_unless_zero() will succeed and we don't mind. It's -clear where reference should go after split: it will stay on head page. - -Note that split_huge_pmd() doesn't have any limitation on refcounting: -pmd can be split at any point and never fails. - -== Partial unmap and deferred_split_huge_page() == - -Unmapping part of THP (with munmap() or other way) is not going to free -memory immediately. Instead, we detect that a subpage of THP is not in use -in page_remove_rmap() and queue the THP for splitting if memory pressure -comes. Splitting will free up unused subpages. - -Splitting the page right away is not an option due to locking context in -the place where we can detect partial unmap. It's also might be -counterproductive since in many cases partial unmap happens during exit(2) if -a THP crosses a VMA boundary. - -Function deferred_split_huge_page() is used to queue page for splitting. -The splitting itself will happen when we get memory pressure via shrinker -interface. diff --git a/Documentation/vm/unevictable-lru.rst b/Documentation/vm/unevictable-lru.rst new file mode 100644 index 000000000000..fdd84cb8d511 --- /dev/null +++ b/Documentation/vm/unevictable-lru.rst @@ -0,0 +1,614 @@ +.. _unevictable_lru: + +============================== +Unevictable LRU Infrastructure +============================== + +.. contents:: :local: + + +Introduction +============ + +This document describes the Linux memory manager's "Unevictable LRU" +infrastructure and the use of this to manage several types of "unevictable" +pages. + +The document attempts to provide the overall rationale behind this mechanism +and the rationale for some of the design decisions that drove the +implementation. The latter design rationale is discussed in the context of an +implementation description. Admittedly, one can obtain the implementation +details - the "what does it do?" - by reading the code. One hopes that the +descriptions below add value by provide the answer to "why does it do that?". + + + +The Unevictable LRU +=================== + +The Unevictable LRU facility adds an additional LRU list to track unevictable +pages and to hide these pages from vmscan. This mechanism is based on a patch +by Larry Woodman of Red Hat to address several scalability problems with page +reclaim in Linux. The problems have been observed at customer sites on large +memory x86_64 systems. + +To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of +main memory will have over 32 million 4k pages in a single zone. When a large +fraction of these pages are not evictable for any reason [see below], vmscan +will spend a lot of time scanning the LRU lists looking for the small fraction +of pages that are evictable. This can result in a situation where all CPUs are +spending 100% of their time in vmscan for hours or days on end, with the system +completely unresponsive. + +The unevictable list addresses the following classes of unevictable pages: + + * Those owned by ramfs. + + * Those mapped into SHM_LOCK'd shared memory regions. + + * Those mapped into VM_LOCKED [mlock()ed] VMAs. + +The infrastructure may also be able to handle other conditions that make pages +unevictable, either by definition or by circumstance, in the future. + + +The Unevictable Page List +------------------------- + +The Unevictable LRU infrastructure consists of an additional, per-zone, LRU list +called the "unevictable" list and an associated page flag, PG_unevictable, to +indicate that the page is being managed on the unevictable list. + +The PG_unevictable flag is analogous to, and mutually exclusive with, the +PG_active flag in that it indicates on which LRU list a page resides when +PG_lru is set. + +The Unevictable LRU infrastructure maintains unevictable pages on an additional +LRU list for a few reasons: + + (1) We get to "treat unevictable pages just like we treat other pages in the + system - which means we get to use the same code to manipulate them, the + same code to isolate them (for migrate, etc.), the same code to keep track + of the statistics, etc..." [Rik van Riel] + + (2) We want to be able to migrate unevictable pages between nodes for memory + defragmentation, workload management and memory hotplug. The linux kernel + can only migrate pages that it can successfully isolate from the LRU + lists. If we were to maintain pages elsewhere than on an LRU-like list, + where they can be found by isolate_lru_page(), we would prevent their + migration, unless we reworked migration code to find the unevictable pages + itself. + + +The unevictable list does not differentiate between file-backed and anonymous, +swap-backed pages. This differentiation is only important while the pages are, +in fact, evictable. + +The unevictable list benefits from the "arrayification" of the per-zone LRU +lists and statistics originally proposed and posted by Christoph Lameter. + +The unevictable list does not use the LRU pagevec mechanism. Rather, +unevictable pages are placed directly on the page's zone's unevictable list +under the zone lru_lock. This allows us to prevent the stranding of pages on +the unevictable list when one task has the page isolated from the LRU and other +tasks are changing the "evictability" state of the page. + + +Memory Control Group Interaction +-------------------------------- + +The unevictable LRU facility interacts with the memory control group [aka +memory controller; see Documentation/cgroup-v1/memory.txt] by extending the +lru_list enum. + +The memory controller data structure automatically gets a per-zone unevictable +list as a result of the "arrayification" of the per-zone LRU lists (one per +lru_list enum element). The memory controller tracks the movement of pages to +and from the unevictable list. + +When a memory control group comes under memory pressure, the controller will +not attempt to reclaim pages on the unevictable list. This has a couple of +effects: + + (1) Because the pages are "hidden" from reclaim on the unevictable list, the + reclaim process can be more efficient, dealing only with pages that have a + chance of being reclaimed. + + (2) On the other hand, if too many of the pages charged to the control group + are unevictable, the evictable portion of the working set of the tasks in + the control group may not fit into the available memory. This can cause + the control group to thrash or to OOM-kill tasks. + + +.. _mark_addr_space_unevict: + +Marking Address Spaces Unevictable +---------------------------------- + +For facilities such as ramfs none of the pages attached to the address space +may be evicted. To prevent eviction of any such pages, the AS_UNEVICTABLE +address space flag is provided, and this can be manipulated by a filesystem +using a number of wrapper functions: + + * ``void mapping_set_unevictable(struct address_space *mapping);`` + + Mark the address space as being completely unevictable. + + * ``void mapping_clear_unevictable(struct address_space *mapping);`` + + Mark the address space as being evictable. + + * ``int mapping_unevictable(struct address_space *mapping);`` + + Query the address space, and return true if it is completely + unevictable. + +These are currently used in two places in the kernel: + + (1) By ramfs to mark the address spaces of its inodes when they are created, + and this mark remains for the life of the inode. + + (2) By SYSV SHM to mark SHM_LOCK'd address spaces until SHM_UNLOCK is called. + + Note that SHM_LOCK is not required to page in the locked pages if they're + swapped out; the application must touch the pages manually if it wants to + ensure they're in memory. + + +Detecting Unevictable Pages +--------------------------- + +The function page_evictable() in vmscan.c determines whether a page is +evictable or not using the query function outlined above [see section +:ref:`Marking address spaces unevictable `] +to check the AS_UNEVICTABLE flag. + +For address spaces that are so marked after being populated (as SHM regions +might be), the lock action (eg: SHM_LOCK) can be lazy, and need not populate +the page tables for the region as does, for example, mlock(), nor need it make +any special effort to push any pages in the SHM_LOCK'd area to the unevictable +list. Instead, vmscan will do this if and when it encounters the pages during +a reclamation scan. + +On an unlock action (such as SHM_UNLOCK), the unlocker (eg: shmctl()) must scan +the pages in the region and "rescue" them from the unevictable list if no other +condition is keeping them unevictable. If an unevictable region is destroyed, +the pages are also "rescued" from the unevictable list in the process of +freeing them. + +page_evictable() also checks for mlocked pages by testing an additional page +flag, PG_mlocked (as wrapped by PageMlocked()), which is set when a page is +faulted into a VM_LOCKED vma, or found in a vma being VM_LOCKED. + + +Vmscan's Handling of Unevictable Pages +-------------------------------------- + +If unevictable pages are culled in the fault path, or moved to the unevictable +list at mlock() or mmap() time, vmscan will not encounter the pages until they +have become evictable again (via munlock() for example) and have been "rescued" +from the unevictable list. However, there may be situations where we decide, +for the sake of expediency, to leave a unevictable page on one of the regular +active/inactive LRU lists for vmscan to deal with. vmscan checks for such +pages in all of the shrink_{active|inactive|page}_list() functions and will +"cull" such pages that it encounters: that is, it diverts those pages to the +unevictable list for the zone being scanned. + +There may be situations where a page is mapped into a VM_LOCKED VMA, but the +page is not marked as PG_mlocked. Such pages will make it all the way to +shrink_page_list() where they will be detected when vmscan walks the reverse +map in try_to_unmap(). If try_to_unmap() returns SWAP_MLOCK, +shrink_page_list() will cull the page at that point. + +To "cull" an unevictable page, vmscan simply puts the page back on the LRU list +using putback_lru_page() - the inverse operation to isolate_lru_page() - after +dropping the page lock. Because the condition which makes the page unevictable +may change once the page is unlocked, putback_lru_page() will recheck the +unevictable state of a page that it places on the unevictable list. If the +page has become unevictable, putback_lru_page() removes it from the list and +retries, including the page_unevictable() test. Because such a race is a rare +event and movement of pages onto the unevictable list should be rare, these +extra evictabilty checks should not occur in the majority of calls to +putback_lru_page(). + + +MLOCKED Pages +============= + +The unevictable page list is also useful for mlock(), in addition to ramfs and +SYSV SHM. Note that mlock() is only available in CONFIG_MMU=y situations; in +NOMMU situations, all mappings are effectively mlocked. + + +History +------- + +The "Unevictable mlocked Pages" infrastructure is based on work originally +posted by Nick Piggin in an RFC patch entitled "mm: mlocked pages off LRU". +Nick posted his patch as an alternative to a patch posted by Christoph Lameter +to achieve the same objective: hiding mlocked pages from vmscan. + +In Nick's patch, he used one of the struct page LRU list link fields as a count +of VM_LOCKED VMAs that map the page. This use of the link field for a count +prevented the management of the pages on an LRU list, and thus mlocked pages +were not migratable as isolate_lru_page() could not find them, and the LRU list +link field was not available to the migration subsystem. + +Nick resolved this by putting mlocked pages back on the lru list before +attempting to isolate them, thus abandoning the count of VM_LOCKED VMAs. When +Nick's patch was integrated with the Unevictable LRU work, the count was +replaced by walking the reverse map to determine whether any VM_LOCKED VMAs +mapped the page. More on this below. + + +Basic Management +---------------- + +mlocked pages - pages mapped into a VM_LOCKED VMA - are a class of unevictable +pages. When such a page has been "noticed" by the memory management subsystem, +the page is marked with the PG_mlocked flag. This can be manipulated using the +PageMlocked() functions. + +A PG_mlocked page will be placed on the unevictable list when it is added to +the LRU. Such pages can be "noticed" by memory management in several places: + + (1) in the mlock()/mlockall() system call handlers; + + (2) in the mmap() system call handler when mmapping a region with the + MAP_LOCKED flag; + + (3) mmapping a region in a task that has called mlockall() with the MCL_FUTURE + flag + + (4) in the fault path, if mlocked pages are "culled" in the fault path, + and when a VM_LOCKED stack segment is expanded; or + + (5) as mentioned above, in vmscan:shrink_page_list() when attempting to + reclaim a page in a VM_LOCKED VMA via try_to_unmap() + +all of which result in the VM_LOCKED flag being set for the VMA if it doesn't +already have it set. + +mlocked pages become unlocked and rescued from the unevictable list when: + + (1) mapped in a range unlocked via the munlock()/munlockall() system calls; + + (2) munmap()'d out of the last VM_LOCKED VMA that maps the page, including + unmapping at task exit; + + (3) when the page is truncated from the last VM_LOCKED VMA of an mmapped file; + or + + (4) before a page is COW'd in a VM_LOCKED VMA. + + +mlock()/mlockall() System Call Handling +--------------------------------------- + +Both [do\_]mlock() and [do\_]mlockall() system call handlers call mlock_fixup() +for each VMA in the range specified by the call. In the case of mlockall(), +this is the entire active address space of the task. Note that mlock_fixup() +is used for both mlocking and munlocking a range of memory. A call to mlock() +an already VM_LOCKED VMA, or to munlock() a VMA that is not VM_LOCKED is +treated as a no-op, and mlock_fixup() simply returns. + +If the VMA passes some filtering as described in "Filtering Special Vmas" +below, mlock_fixup() will attempt to merge the VMA with its neighbors or split +off a subset of the VMA if the range does not cover the entire VMA. Once the +VMA has been merged or split or neither, mlock_fixup() will call +populate_vma_page_range() to fault in the pages via get_user_pages() and to +mark the pages as mlocked via mlock_vma_page(). + +Note that the VMA being mlocked might be mapped with PROT_NONE. In this case, +get_user_pages() will be unable to fault in the pages. That's okay. If pages +do end up getting faulted into this VM_LOCKED VMA, we'll handle them in the +fault path or in vmscan. + +Also note that a page returned by get_user_pages() could be truncated or +migrated out from under us, while we're trying to mlock it. To detect this, +populate_vma_page_range() checks page_mapping() after acquiring the page lock. +If the page is still associated with its mapping, we'll go ahead and call +mlock_vma_page(). If the mapping is gone, we just unlock the page and move on. +In the worst case, this will result in a page mapped in a VM_LOCKED VMA +remaining on a normal LRU list without being PageMlocked(). Again, vmscan will +detect and cull such pages. + +mlock_vma_page() will call TestSetPageMlocked() for each page returned by +get_user_pages(). We use TestSetPageMlocked() because the page might already +be mlocked by another task/VMA and we don't want to do extra work. We +especially do not want to count an mlocked page more than once in the +statistics. If the page was already mlocked, mlock_vma_page() need do nothing +more. + +If the page was NOT already mlocked, mlock_vma_page() attempts to isolate the +page from the LRU, as it is likely on the appropriate active or inactive list +at that time. If the isolate_lru_page() succeeds, mlock_vma_page() will put +back the page - by calling putback_lru_page() - which will notice that the page +is now mlocked and divert the page to the zone's unevictable list. If +mlock_vma_page() is unable to isolate the page from the LRU, vmscan will handle +it later if and when it attempts to reclaim the page. + + +Filtering Special VMAs +---------------------- + +mlock_fixup() filters several classes of "special" VMAs: + +1) VMAs with VM_IO or VM_PFNMAP set are skipped entirely. The pages behind + these mappings are inherently pinned, so we don't need to mark them as + mlocked. In any case, most of the pages have no struct page in which to so + mark the page. Because of this, get_user_pages() will fail for these VMAs, + so there is no sense in attempting to visit them. + +2) VMAs mapping hugetlbfs page are already effectively pinned into memory. We + neither need nor want to mlock() these pages. However, to preserve the + prior behavior of mlock() - before the unevictable/mlock changes - + mlock_fixup() will call make_pages_present() in the hugetlbfs VMA range to + allocate the huge pages and populate the ptes. + +3) VMAs with VM_DONTEXPAND are generally userspace mappings of kernel pages, + such as the VDSO page, relay channel pages, etc. These pages + are inherently unevictable and are not managed on the LRU lists. + mlock_fixup() treats these VMAs the same as hugetlbfs VMAs. It calls + make_pages_present() to populate the ptes. + +Note that for all of these special VMAs, mlock_fixup() does not set the +VM_LOCKED flag. Therefore, we won't have to deal with them later during +munlock(), munmap() or task exit. Neither does mlock_fixup() account these +VMAs against the task's "locked_vm". + +.. _munlock_munlockall_handling: + +munlock()/munlockall() System Call Handling +------------------------------------------- + +The munlock() and munlockall() system calls are handled by the same functions - +do_mlock[all]() - as the mlock() and mlockall() system calls with the unlock vs +lock operation indicated by an argument. So, these system calls are also +handled by mlock_fixup(). Again, if called for an already munlocked VMA, +mlock_fixup() simply returns. Because of the VMA filtering discussed above, +VM_LOCKED will not be set in any "special" VMAs. So, these VMAs will be +ignored for munlock. + +If the VMA is VM_LOCKED, mlock_fixup() again attempts to merge or split off the +specified range. The range is then munlocked via the function +populate_vma_page_range() - the same function used to mlock a VMA range - +passing a flag to indicate that munlock() is being performed. + +Because the VMA access protections could have been changed to PROT_NONE after +faulting in and mlocking pages, get_user_pages() was unreliable for visiting +these pages for munlocking. Because we don't want to leave pages mlocked, +get_user_pages() was enhanced to accept a flag to ignore the permissions when +fetching the pages - all of which should be resident as a result of previous +mlocking. + +For munlock(), populate_vma_page_range() unlocks individual pages by calling +munlock_vma_page(). munlock_vma_page() unconditionally clears the PG_mlocked +flag using TestClearPageMlocked(). As with mlock_vma_page(), +munlock_vma_page() use the Test*PageMlocked() function to handle the case where +the page might have already been unlocked by another task. If the page was +mlocked, munlock_vma_page() updates that zone statistics for the number of +mlocked pages. Note, however, that at this point we haven't checked whether +the page is mapped by other VM_LOCKED VMAs. + +We can't call try_to_munlock(), the function that walks the reverse map to +check for other VM_LOCKED VMAs, without first isolating the page from the LRU. +try_to_munlock() is a variant of try_to_unmap() and thus requires that the page +not be on an LRU list [more on these below]. However, the call to +isolate_lru_page() could fail, in which case we couldn't try_to_munlock(). So, +we go ahead and clear PG_mlocked up front, as this might be the only chance we +have. If we can successfully isolate the page, we go ahead and +try_to_munlock(), which will restore the PG_mlocked flag and update the zone +page statistics if it finds another VMA holding the page mlocked. If we fail +to isolate the page, we'll have left a potentially mlocked page on the LRU. +This is fine, because we'll catch it later if and if vmscan tries to reclaim +the page. This should be relatively rare. + + +Migrating MLOCKED Pages +----------------------- + +A page that is being migrated has been isolated from the LRU lists and is held +locked across unmapping of the page, updating the page's address space entry +and copying the contents and state, until the page table entry has been +replaced with an entry that refers to the new page. Linux supports migration +of mlocked pages and other unevictable pages. This involves simply moving the +PG_mlocked and PG_unevictable states from the old page to the new page. + +Note that page migration can race with mlocking or munlocking of the same page. +This has been discussed from the mlock/munlock perspective in the respective +sections above. Both processes (migration and m[un]locking) hold the page +locked. This provides the first level of synchronization. Page migration +zeros out the page_mapping of the old page before unlocking it, so m[un]lock +can skip these pages by testing the page mapping under page lock. + +To complete page migration, we place the new and old pages back onto the LRU +after dropping the page lock. The "unneeded" page - old page on success, new +page on failure - will be freed when the reference count held by the migration +process is released. To ensure that we don't strand pages on the unevictable +list because of a race between munlock and migration, page migration uses the +putback_lru_page() function to add migrated pages back to the LRU. + + +Compacting MLOCKED Pages +------------------------ + +The unevictable LRU can be scanned for compactable regions and the default +behavior is to do so. /proc/sys/vm/compact_unevictable_allowed controls +this behavior (see Documentation/sysctl/vm.txt). Once scanning of the +unevictable LRU is enabled, the work of compaction is mostly handled by +the page migration code and the same work flow as described in MIGRATING +MLOCKED PAGES will apply. + +MLOCKING Transparent Huge Pages +------------------------------- + +A transparent huge page is represented by a single entry on an LRU list. +Therefore, we can only make unevictable an entire compound page, not +individual subpages. + +If a user tries to mlock() part of a huge page, we want the rest of the +page to be reclaimable. + +We cannot just split the page on partial mlock() as split_huge_page() can +fail and new intermittent failure mode for the syscall is undesirable. + +We handle this by keeping PTE-mapped huge pages on normal LRU lists: the +PMD on border of VM_LOCKED VMA will be split into PTE table. + +This way the huge page is accessible for vmscan. Under memory pressure the +page will be split, subpages which belong to VM_LOCKED VMAs will be moved +to unevictable LRU and the rest can be reclaimed. + +See also comment in follow_trans_huge_pmd(). + +mmap(MAP_LOCKED) System Call Handling +------------------------------------- + +In addition the mlock()/mlockall() system calls, an application can request +that a region of memory be mlocked supplying the MAP_LOCKED flag to the mmap() +call. There is one important and subtle difference here, though. mmap() + mlock() +will fail if the range cannot be faulted in (e.g. because mm_populate fails) +and returns with ENOMEM while mmap(MAP_LOCKED) will not fail. The mmaped +area will still have properties of the locked area - aka. pages will not get +swapped out - but major page faults to fault memory in might still happen. + +Furthermore, any mmap() call or brk() call that expands the heap by a +task that has previously called mlockall() with the MCL_FUTURE flag will result +in the newly mapped memory being mlocked. Before the unevictable/mlock +changes, the kernel simply called make_pages_present() to allocate pages and +populate the page table. + +To mlock a range of memory under the unevictable/mlock infrastructure, the +mmap() handler and task address space expansion functions call +populate_vma_page_range() specifying the vma and the address range to mlock. + +The callers of populate_vma_page_range() will have already added the memory range +to be mlocked to the task's "locked_vm". To account for filtered VMAs, +populate_vma_page_range() returns the number of pages NOT mlocked. All of the +callers then subtract a non-negative return value from the task's locked_vm. A +negative return value represent an error - for example, from get_user_pages() +attempting to fault in a VMA with PROT_NONE access. In this case, we leave the +memory range accounted as locked_vm, as the protections could be changed later +and pages allocated into that region. + + +munmap()/exit()/exec() System Call Handling +------------------------------------------- + +When unmapping an mlocked region of memory, whether by an explicit call to +munmap() or via an internal unmap from exit() or exec() processing, we must +munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. +Before the unevictable/mlock changes, mlocking did not mark the pages in any +way, so unmapping them required no processing. + +To munlock a range of memory under the unevictable/mlock infrastructure, the +munmap() handler and task address space call tear down function +munlock_vma_pages_all(). The name reflects the observation that one always +specifies the entire VMA range when munlock()ing during unmap of a region. +Because of the VMA filtering when mlocking() regions, only "normal" VMAs that +actually contain mlocked pages will be passed to munlock_vma_pages_all(). + +munlock_vma_pages_all() clears the VM_LOCKED VMA flag and, like mlock_fixup() +for the munlock case, calls __munlock_vma_pages_range() to walk the page table +for the VMA's memory range and munlock_vma_page() each resident page mapped by +the VMA. This effectively munlocks the page, only if this is the last +VM_LOCKED VMA that maps the page. + + +try_to_unmap() +-------------- + +Pages can, of course, be mapped into multiple VMAs. Some of these VMAs may +have VM_LOCKED flag set. It is possible for a page mapped into one or more +VM_LOCKED VMAs not to have the PG_mlocked flag set and therefore reside on one +of the active or inactive LRU lists. This could happen if, for example, a task +in the process of munlocking the page could not isolate the page from the LRU. +As a result, vmscan/shrink_page_list() might encounter such a page as described +in section "vmscan's handling of unevictable pages". To handle this situation, +try_to_unmap() checks for VM_LOCKED VMAs while it is walking a page's reverse +map. + +try_to_unmap() is always called, by either vmscan for reclaim or for page +migration, with the argument page locked and isolated from the LRU. Separate +functions handle anonymous and mapped file and KSM pages, as these types of +pages have different reverse map lookup mechanisms, with different locking. +In each case, whether rmap_walk_anon() or rmap_walk_file() or rmap_walk_ksm(), +it will call try_to_unmap_one() for every VMA which might contain the page. + +When trying to reclaim, if try_to_unmap_one() finds the page in a VM_LOCKED +VMA, it will then mlock the page via mlock_vma_page() instead of unmapping it, +and return SWAP_MLOCK to indicate that the page is unevictable: and the scan +stops there. + +mlock_vma_page() is called while holding the page table's lock (in addition +to the page lock, and the rmap lock): to serialize against concurrent mlock or +munlock or munmap system calls, mm teardown (munlock_vma_pages_all), reclaim, +holepunching, and truncation of file pages and their anonymous COWed pages. + + +try_to_munlock() Reverse Map Scan +--------------------------------- + +.. warning:: + [!] TODO/FIXME: a better name might be page_mlocked() - analogous to the + page_referenced() reverse map walker. + +When munlock_vma_page() [see section :ref:`munlock()/munlockall() System Call +Handling ` above] tries to munlock a +page, it needs to determine whether or not the page is mapped by any +VM_LOCKED VMA without actually attempting to unmap all PTEs from the +page. For this purpose, the unevictable/mlock infrastructure +introduced a variant of try_to_unmap() called try_to_munlock(). + +try_to_munlock() calls the same functions as try_to_unmap() for anonymous and +mapped file and KSM pages with a flag argument specifying unlock versus unmap +processing. Again, these functions walk the respective reverse maps looking +for VM_LOCKED VMAs. When such a VMA is found, as in the try_to_unmap() case, +the functions mlock the page via mlock_vma_page() and return SWAP_MLOCK. This +undoes the pre-clearing of the page's PG_mlocked done by munlock_vma_page. + +Note that try_to_munlock()'s reverse map walk must visit every VMA in a page's +reverse map to determine that a page is NOT mapped into any VM_LOCKED VMA. +However, the scan can terminate when it encounters a VM_LOCKED VMA. +Although try_to_munlock() might be called a great many times when munlocking a +large region or tearing down a large address space that has been mlocked via +mlockall(), overall this is a fairly rare event. + + +Page Reclaim in shrink_*_list() +------------------------------- + +shrink_active_list() culls any obviously unevictable pages - i.e. +!page_evictable(page) - diverting these to the unevictable list. +However, shrink_active_list() only sees unevictable pages that made it onto the +active/inactive lru lists. Note that these pages do not have PageUnevictable +set - otherwise they would be on the unevictable list and shrink_active_list +would never see them. + +Some examples of these unevictable pages on the LRU lists are: + + (1) ramfs pages that have been placed on the LRU lists when first allocated. + + (2) SHM_LOCK'd shared memory pages. shmctl(SHM_LOCK) does not attempt to + allocate or fault in the pages in the shared memory region. This happens + when an application accesses the page the first time after SHM_LOCK'ing + the segment. + + (3) mlocked pages that could not be isolated from the LRU and moved to the + unevictable list in mlock_vma_page(). + +shrink_inactive_list() also diverts any unevictable pages that it finds on the +inactive lists to the appropriate zone's unevictable list. + +shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd +after shrink_active_list() had moved them to the inactive list, or pages mapped +into VM_LOCKED VMAs that munlock_vma_page() couldn't isolate from the LRU to +recheck via try_to_munlock(). shrink_inactive_list() won't notice the latter, +but will pass on to shrink_page_list(). + +shrink_page_list() again culls obviously unevictable pages that it could +encounter for similar reason to shrink_inactive_list(). Pages mapped into +VM_LOCKED VMAs but without PG_mlocked set will make it all the way to +try_to_unmap(). shrink_page_list() will divert them to the unevictable list +when try_to_unmap() returns SWAP_MLOCK, as discussed above. diff --git a/Documentation/vm/unevictable-lru.txt b/Documentation/vm/unevictable-lru.txt deleted file mode 100644 index e14718572476..000000000000 --- a/Documentation/vm/unevictable-lru.txt +++ /dev/null @@ -1,633 +0,0 @@ - ============================== - UNEVICTABLE LRU INFRASTRUCTURE - ============================== - -======== -CONTENTS -======== - - (*) The Unevictable LRU - - - The unevictable page list. - - Memory control group interaction. - - Marking address spaces unevictable. - - Detecting Unevictable Pages. - - vmscan's handling of unevictable pages. - - (*) mlock()'d pages. - - - History. - - Basic management. - - mlock()/mlockall() system call handling. - - Filtering special vmas. - - munlock()/munlockall() system call handling. - - Migrating mlocked pages. - - Compacting mlocked pages. - - mmap(MAP_LOCKED) system call handling. - - munmap()/exit()/exec() system call handling. - - try_to_unmap(). - - try_to_munlock() reverse map scan. - - Page reclaim in shrink_*_list(). - - -============ -INTRODUCTION -============ - -This document describes the Linux memory manager's "Unevictable LRU" -infrastructure and the use of this to manage several types of "unevictable" -pages. - -The document attempts to provide the overall rationale behind this mechanism -and the rationale for some of the design decisions that drove the -implementation. The latter design rationale is discussed in the context of an -implementation description. Admittedly, one can obtain the implementation -details - the "what does it do?" - by reading the code. One hopes that the -descriptions below add value by provide the answer to "why does it do that?". - - -=================== -THE UNEVICTABLE LRU -=================== - -The Unevictable LRU facility adds an additional LRU list to track unevictable -pages and to hide these pages from vmscan. This mechanism is based on a patch -by Larry Woodman of Red Hat to address several scalability problems with page -reclaim in Linux. The problems have been observed at customer sites on large -memory x86_64 systems. - -To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of -main memory will have over 32 million 4k pages in a single zone. When a large -fraction of these pages are not evictable for any reason [see below], vmscan -will spend a lot of time scanning the LRU lists looking for the small fraction -of pages that are evictable. This can result in a situation where all CPUs are -spending 100% of their time in vmscan for hours or days on end, with the system -completely unresponsive. - -The unevictable list addresses the following classes of unevictable pages: - - (*) Those owned by ramfs. - - (*) Those mapped into SHM_LOCK'd shared memory regions. - - (*) Those mapped into VM_LOCKED [mlock()ed] VMAs. - -The infrastructure may also be able to handle other conditions that make pages -unevictable, either by definition or by circumstance, in the future. - - -THE UNEVICTABLE PAGE LIST -------------------------- - -The Unevictable LRU infrastructure consists of an additional, per-zone, LRU list -called the "unevictable" list and an associated page flag, PG_unevictable, to -indicate that the page is being managed on the unevictable list. - -The PG_unevictable flag is analogous to, and mutually exclusive with, the -PG_active flag in that it indicates on which LRU list a page resides when -PG_lru is set. - -The Unevictable LRU infrastructure maintains unevictable pages on an additional -LRU list for a few reasons: - - (1) We get to "treat unevictable pages just like we treat other pages in the - system - which means we get to use the same code to manipulate them, the - same code to isolate them (for migrate, etc.), the same code to keep track - of the statistics, etc..." [Rik van Riel] - - (2) We want to be able to migrate unevictable pages between nodes for memory - defragmentation, workload management and memory hotplug. The linux kernel - can only migrate pages that it can successfully isolate from the LRU - lists. If we were to maintain pages elsewhere than on an LRU-like list, - where they can be found by isolate_lru_page(), we would prevent their - migration, unless we reworked migration code to find the unevictable pages - itself. - - -The unevictable list does not differentiate between file-backed and anonymous, -swap-backed pages. This differentiation is only important while the pages are, -in fact, evictable. - -The unevictable list benefits from the "arrayification" of the per-zone LRU -lists and statistics originally proposed and posted by Christoph Lameter. - -The unevictable list does not use the LRU pagevec mechanism. Rather, -unevictable pages are placed directly on the page's zone's unevictable list -under the zone lru_lock. This allows us to prevent the stranding of pages on -the unevictable list when one task has the page isolated from the LRU and other -tasks are changing the "evictability" state of the page. - - -MEMORY CONTROL GROUP INTERACTION --------------------------------- - -The unevictable LRU facility interacts with the memory control group [aka -memory controller; see Documentation/cgroup-v1/memory.txt] by extending the -lru_list enum. - -The memory controller data structure automatically gets a per-zone unevictable -list as a result of the "arrayification" of the per-zone LRU lists (one per -lru_list enum element). The memory controller tracks the movement of pages to -and from the unevictable list. - -When a memory control group comes under memory pressure, the controller will -not attempt to reclaim pages on the unevictable list. This has a couple of -effects: - - (1) Because the pages are "hidden" from reclaim on the unevictable list, the - reclaim process can be more efficient, dealing only with pages that have a - chance of being reclaimed. - - (2) On the other hand, if too many of the pages charged to the control group - are unevictable, the evictable portion of the working set of the tasks in - the control group may not fit into the available memory. This can cause - the control group to thrash or to OOM-kill tasks. - - -MARKING ADDRESS SPACES UNEVICTABLE ----------------------------------- - -For facilities such as ramfs none of the pages attached to the address space -may be evicted. To prevent eviction of any such pages, the AS_UNEVICTABLE -address space flag is provided, and this can be manipulated by a filesystem -using a number of wrapper functions: - - (*) void mapping_set_unevictable(struct address_space *mapping); - - Mark the address space as being completely unevictable. - - (*) void mapping_clear_unevictable(struct address_space *mapping); - - Mark the address space as being evictable. - - (*) int mapping_unevictable(struct address_space *mapping); - - Query the address space, and return true if it is completely - unevictable. - -These are currently used in two places in the kernel: - - (1) By ramfs to mark the address spaces of its inodes when they are created, - and this mark remains for the life of the inode. - - (2) By SYSV SHM to mark SHM_LOCK'd address spaces until SHM_UNLOCK is called. - - Note that SHM_LOCK is not required to page in the locked pages if they're - swapped out; the application must touch the pages manually if it wants to - ensure they're in memory. - - -DETECTING UNEVICTABLE PAGES ---------------------------- - -The function page_evictable() in vmscan.c determines whether a page is -evictable or not using the query function outlined above [see section "Marking -address spaces unevictable"] to check the AS_UNEVICTABLE flag. - -For address spaces that are so marked after being populated (as SHM regions -might be), the lock action (eg: SHM_LOCK) can be lazy, and need not populate -the page tables for the region as does, for example, mlock(), nor need it make -any special effort to push any pages in the SHM_LOCK'd area to the unevictable -list. Instead, vmscan will do this if and when it encounters the pages during -a reclamation scan. - -On an unlock action (such as SHM_UNLOCK), the unlocker (eg: shmctl()) must scan -the pages in the region and "rescue" them from the unevictable list if no other -condition is keeping them unevictable. If an unevictable region is destroyed, -the pages are also "rescued" from the unevictable list in the process of -freeing them. - -page_evictable() also checks for mlocked pages by testing an additional page -flag, PG_mlocked (as wrapped by PageMlocked()), which is set when a page is -faulted into a VM_LOCKED vma, or found in a vma being VM_LOCKED. - - -VMSCAN'S HANDLING OF UNEVICTABLE PAGES --------------------------------------- - -If unevictable pages are culled in the fault path, or moved to the unevictable -list at mlock() or mmap() time, vmscan will not encounter the pages until they -have become evictable again (via munlock() for example) and have been "rescued" -from the unevictable list. However, there may be situations where we decide, -for the sake of expediency, to leave a unevictable page on one of the regular -active/inactive LRU lists for vmscan to deal with. vmscan checks for such -pages in all of the shrink_{active|inactive|page}_list() functions and will -"cull" such pages that it encounters: that is, it diverts those pages to the -unevictable list for the zone being scanned. - -There may be situations where a page is mapped into a VM_LOCKED VMA, but the -page is not marked as PG_mlocked. Such pages will make it all the way to -shrink_page_list() where they will be detected when vmscan walks the reverse -map in try_to_unmap(). If try_to_unmap() returns SWAP_MLOCK, -shrink_page_list() will cull the page at that point. - -To "cull" an unevictable page, vmscan simply puts the page back on the LRU list -using putback_lru_page() - the inverse operation to isolate_lru_page() - after -dropping the page lock. Because the condition which makes the page unevictable -may change once the page is unlocked, putback_lru_page() will recheck the -unevictable state of a page that it places on the unevictable list. If the -page has become unevictable, putback_lru_page() removes it from the list and -retries, including the page_unevictable() test. Because such a race is a rare -event and movement of pages onto the unevictable list should be rare, these -extra evictabilty checks should not occur in the majority of calls to -putback_lru_page(). - - -============= -MLOCKED PAGES -============= - -The unevictable page list is also useful for mlock(), in addition to ramfs and -SYSV SHM. Note that mlock() is only available in CONFIG_MMU=y situations; in -NOMMU situations, all mappings are effectively mlocked. - - -HISTORY -------- - -The "Unevictable mlocked Pages" infrastructure is based on work originally -posted by Nick Piggin in an RFC patch entitled "mm: mlocked pages off LRU". -Nick posted his patch as an alternative to a patch posted by Christoph Lameter -to achieve the same objective: hiding mlocked pages from vmscan. - -In Nick's patch, he used one of the struct page LRU list link fields as a count -of VM_LOCKED VMAs that map the page. This use of the link field for a count -prevented the management of the pages on an LRU list, and thus mlocked pages -were not migratable as isolate_lru_page() could not find them, and the LRU list -link field was not available to the migration subsystem. - -Nick resolved this by putting mlocked pages back on the lru list before -attempting to isolate them, thus abandoning the count of VM_LOCKED VMAs. When -Nick's patch was integrated with the Unevictable LRU work, the count was -replaced by walking the reverse map to determine whether any VM_LOCKED VMAs -mapped the page. More on this below. - - -BASIC MANAGEMENT ----------------- - -mlocked pages - pages mapped into a VM_LOCKED VMA - are a class of unevictable -pages. When such a page has been "noticed" by the memory management subsystem, -the page is marked with the PG_mlocked flag. This can be manipulated using the -PageMlocked() functions. - -A PG_mlocked page will be placed on the unevictable list when it is added to -the LRU. Such pages can be "noticed" by memory management in several places: - - (1) in the mlock()/mlockall() system call handlers; - - (2) in the mmap() system call handler when mmapping a region with the - MAP_LOCKED flag; - - (3) mmapping a region in a task that has called mlockall() with the MCL_FUTURE - flag - - (4) in the fault path, if mlocked pages are "culled" in the fault path, - and when a VM_LOCKED stack segment is expanded; or - - (5) as mentioned above, in vmscan:shrink_page_list() when attempting to - reclaim a page in a VM_LOCKED VMA via try_to_unmap() - -all of which result in the VM_LOCKED flag being set for the VMA if it doesn't -already have it set. - -mlocked pages become unlocked and rescued from the unevictable list when: - - (1) mapped in a range unlocked via the munlock()/munlockall() system calls; - - (2) munmap()'d out of the last VM_LOCKED VMA that maps the page, including - unmapping at task exit; - - (3) when the page is truncated from the last VM_LOCKED VMA of an mmapped file; - or - - (4) before a page is COW'd in a VM_LOCKED VMA. - - -mlock()/mlockall() SYSTEM CALL HANDLING ---------------------------------------- - -Both [do_]mlock() and [do_]mlockall() system call handlers call mlock_fixup() -for each VMA in the range specified by the call. In the case of mlockall(), -this is the entire active address space of the task. Note that mlock_fixup() -is used for both mlocking and munlocking a range of memory. A call to mlock() -an already VM_LOCKED VMA, or to munlock() a VMA that is not VM_LOCKED is -treated as a no-op, and mlock_fixup() simply returns. - -If the VMA passes some filtering as described in "Filtering Special Vmas" -below, mlock_fixup() will attempt to merge the VMA with its neighbors or split -off a subset of the VMA if the range does not cover the entire VMA. Once the -VMA has been merged or split or neither, mlock_fixup() will call -populate_vma_page_range() to fault in the pages via get_user_pages() and to -mark the pages as mlocked via mlock_vma_page(). - -Note that the VMA being mlocked might be mapped with PROT_NONE. In this case, -get_user_pages() will be unable to fault in the pages. That's okay. If pages -do end up getting faulted into this VM_LOCKED VMA, we'll handle them in the -fault path or in vmscan. - -Also note that a page returned by get_user_pages() could be truncated or -migrated out from under us, while we're trying to mlock it. To detect this, -populate_vma_page_range() checks page_mapping() after acquiring the page lock. -If the page is still associated with its mapping, we'll go ahead and call -mlock_vma_page(). If the mapping is gone, we just unlock the page and move on. -In the worst case, this will result in a page mapped in a VM_LOCKED VMA -remaining on a normal LRU list without being PageMlocked(). Again, vmscan will -detect and cull such pages. - -mlock_vma_page() will call TestSetPageMlocked() for each page returned by -get_user_pages(). We use TestSetPageMlocked() because the page might already -be mlocked by another task/VMA and we don't want to do extra work. We -especially do not want to count an mlocked page more than once in the -statistics. If the page was already mlocked, mlock_vma_page() need do nothing -more. - -If the page was NOT already mlocked, mlock_vma_page() attempts to isolate the -page from the LRU, as it is likely on the appropriate active or inactive list -at that time. If the isolate_lru_page() succeeds, mlock_vma_page() will put -back the page - by calling putback_lru_page() - which will notice that the page -is now mlocked and divert the page to the zone's unevictable list. If -mlock_vma_page() is unable to isolate the page from the LRU, vmscan will handle -it later if and when it attempts to reclaim the page. - - -FILTERING SPECIAL VMAS ----------------------- - -mlock_fixup() filters several classes of "special" VMAs: - -1) VMAs with VM_IO or VM_PFNMAP set are skipped entirely. The pages behind - these mappings are inherently pinned, so we don't need to mark them as - mlocked. In any case, most of the pages have no struct page in which to so - mark the page. Because of this, get_user_pages() will fail for these VMAs, - so there is no sense in attempting to visit them. - -2) VMAs mapping hugetlbfs page are already effectively pinned into memory. We - neither need nor want to mlock() these pages. However, to preserve the - prior behavior of mlock() - before the unevictable/mlock changes - - mlock_fixup() will call make_pages_present() in the hugetlbfs VMA range to - allocate the huge pages and populate the ptes. - -3) VMAs with VM_DONTEXPAND are generally userspace mappings of kernel pages, - such as the VDSO page, relay channel pages, etc. These pages - are inherently unevictable and are not managed on the LRU lists. - mlock_fixup() treats these VMAs the same as hugetlbfs VMAs. It calls - make_pages_present() to populate the ptes. - -Note that for all of these special VMAs, mlock_fixup() does not set the -VM_LOCKED flag. Therefore, we won't have to deal with them later during -munlock(), munmap() or task exit. Neither does mlock_fixup() account these -VMAs against the task's "locked_vm". - - -munlock()/munlockall() SYSTEM CALL HANDLING -------------------------------------------- - -The munlock() and munlockall() system calls are handled by the same functions - -do_mlock[all]() - as the mlock() and mlockall() system calls with the unlock vs -lock operation indicated by an argument. So, these system calls are also -handled by mlock_fixup(). Again, if called for an already munlocked VMA, -mlock_fixup() simply returns. Because of the VMA filtering discussed above, -VM_LOCKED will not be set in any "special" VMAs. So, these VMAs will be -ignored for munlock. - -If the VMA is VM_LOCKED, mlock_fixup() again attempts to merge or split off the -specified range. The range is then munlocked via the function -populate_vma_page_range() - the same function used to mlock a VMA range - -passing a flag to indicate that munlock() is being performed. - -Because the VMA access protections could have been changed to PROT_NONE after -faulting in and mlocking pages, get_user_pages() was unreliable for visiting -these pages for munlocking. Because we don't want to leave pages mlocked, -get_user_pages() was enhanced to accept a flag to ignore the permissions when -fetching the pages - all of which should be resident as a result of previous -mlocking. - -For munlock(), populate_vma_page_range() unlocks individual pages by calling -munlock_vma_page(). munlock_vma_page() unconditionally clears the PG_mlocked -flag using TestClearPageMlocked(). As with mlock_vma_page(), -munlock_vma_page() use the Test*PageMlocked() function to handle the case where -the page might have already been unlocked by another task. If the page was -mlocked, munlock_vma_page() updates that zone statistics for the number of -mlocked pages. Note, however, that at this point we haven't checked whether -the page is mapped by other VM_LOCKED VMAs. - -We can't call try_to_munlock(), the function that walks the reverse map to -check for other VM_LOCKED VMAs, without first isolating the page from the LRU. -try_to_munlock() is a variant of try_to_unmap() and thus requires that the page -not be on an LRU list [more on these below]. However, the call to -isolate_lru_page() could fail, in which case we couldn't try_to_munlock(). So, -we go ahead and clear PG_mlocked up front, as this might be the only chance we -have. If we can successfully isolate the page, we go ahead and -try_to_munlock(), which will restore the PG_mlocked flag and update the zone -page statistics if it finds another VMA holding the page mlocked. If we fail -to isolate the page, we'll have left a potentially mlocked page on the LRU. -This is fine, because we'll catch it later if and if vmscan tries to reclaim -the page. This should be relatively rare. - - -MIGRATING MLOCKED PAGES ------------------------ - -A page that is being migrated has been isolated from the LRU lists and is held -locked across unmapping of the page, updating the page's address space entry -and copying the contents and state, until the page table entry has been -replaced with an entry that refers to the new page. Linux supports migration -of mlocked pages and other unevictable pages. This involves simply moving the -PG_mlocked and PG_unevictable states from the old page to the new page. - -Note that page migration can race with mlocking or munlocking of the same page. -This has been discussed from the mlock/munlock perspective in the respective -sections above. Both processes (migration and m[un]locking) hold the page -locked. This provides the first level of synchronization. Page migration -zeros out the page_mapping of the old page before unlocking it, so m[un]lock -can skip these pages by testing the page mapping under page lock. - -To complete page migration, we place the new and old pages back onto the LRU -after dropping the page lock. The "unneeded" page - old page on success, new -page on failure - will be freed when the reference count held by the migration -process is released. To ensure that we don't strand pages on the unevictable -list because of a race between munlock and migration, page migration uses the -putback_lru_page() function to add migrated pages back to the LRU. - - -COMPACTING MLOCKED PAGES ------------------------- - -The unevictable LRU can be scanned for compactable regions and the default -behavior is to do so. /proc/sys/vm/compact_unevictable_allowed controls -this behavior (see Documentation/sysctl/vm.txt). Once scanning of the -unevictable LRU is enabled, the work of compaction is mostly handled by -the page migration code and the same work flow as described in MIGRATING -MLOCKED PAGES will apply. - -MLOCKING TRANSPARENT HUGE PAGES -------------------------------- - -A transparent huge page is represented by a single entry on an LRU list. -Therefore, we can only make unevictable an entire compound page, not -individual subpages. - -If a user tries to mlock() part of a huge page, we want the rest of the -page to be reclaimable. - -We cannot just split the page on partial mlock() as split_huge_page() can -fail and new intermittent failure mode for the syscall is undesirable. - -We handle this by keeping PTE-mapped huge pages on normal LRU lists: the -PMD on border of VM_LOCKED VMA will be split into PTE table. - -This way the huge page is accessible for vmscan. Under memory pressure the -page will be split, subpages which belong to VM_LOCKED VMAs will be moved -to unevictable LRU and the rest can be reclaimed. - -See also comment in follow_trans_huge_pmd(). - -mmap(MAP_LOCKED) SYSTEM CALL HANDLING -------------------------------------- - -In addition the mlock()/mlockall() system calls, an application can request -that a region of memory be mlocked supplying the MAP_LOCKED flag to the mmap() -call. There is one important and subtle difference here, though. mmap() + mlock() -will fail if the range cannot be faulted in (e.g. because mm_populate fails) -and returns with ENOMEM while mmap(MAP_LOCKED) will not fail. The mmaped -area will still have properties of the locked area - aka. pages will not get -swapped out - but major page faults to fault memory in might still happen. - -Furthermore, any mmap() call or brk() call that expands the heap by a -task that has previously called mlockall() with the MCL_FUTURE flag will result -in the newly mapped memory being mlocked. Before the unevictable/mlock -changes, the kernel simply called make_pages_present() to allocate pages and -populate the page table. - -To mlock a range of memory under the unevictable/mlock infrastructure, the -mmap() handler and task address space expansion functions call -populate_vma_page_range() specifying the vma and the address range to mlock. - -The callers of populate_vma_page_range() will have already added the memory range -to be mlocked to the task's "locked_vm". To account for filtered VMAs, -populate_vma_page_range() returns the number of pages NOT mlocked. All of the -callers then subtract a non-negative return value from the task's locked_vm. A -negative return value represent an error - for example, from get_user_pages() -attempting to fault in a VMA with PROT_NONE access. In this case, we leave the -memory range accounted as locked_vm, as the protections could be changed later -and pages allocated into that region. - - -munmap()/exit()/exec() SYSTEM CALL HANDLING -------------------------------------------- - -When unmapping an mlocked region of memory, whether by an explicit call to -munmap() or via an internal unmap from exit() or exec() processing, we must -munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. -Before the unevictable/mlock changes, mlocking did not mark the pages in any -way, so unmapping them required no processing. - -To munlock a range of memory under the unevictable/mlock infrastructure, the -munmap() handler and task address space call tear down function -munlock_vma_pages_all(). The name reflects the observation that one always -specifies the entire VMA range when munlock()ing during unmap of a region. -Because of the VMA filtering when mlocking() regions, only "normal" VMAs that -actually contain mlocked pages will be passed to munlock_vma_pages_all(). - -munlock_vma_pages_all() clears the VM_LOCKED VMA flag and, like mlock_fixup() -for the munlock case, calls __munlock_vma_pages_range() to walk the page table -for the VMA's memory range and munlock_vma_page() each resident page mapped by -the VMA. This effectively munlocks the page, only if this is the last -VM_LOCKED VMA that maps the page. - - -try_to_unmap() --------------- - -Pages can, of course, be mapped into multiple VMAs. Some of these VMAs may -have VM_LOCKED flag set. It is possible for a page mapped into one or more -VM_LOCKED VMAs not to have the PG_mlocked flag set and therefore reside on one -of the active or inactive LRU lists. This could happen if, for example, a task -in the process of munlocking the page could not isolate the page from the LRU. -As a result, vmscan/shrink_page_list() might encounter such a page as described -in section "vmscan's handling of unevictable pages". To handle this situation, -try_to_unmap() checks for VM_LOCKED VMAs while it is walking a page's reverse -map. - -try_to_unmap() is always called, by either vmscan for reclaim or for page -migration, with the argument page locked and isolated from the LRU. Separate -functions handle anonymous and mapped file and KSM pages, as these types of -pages have different reverse map lookup mechanisms, with different locking. -In each case, whether rmap_walk_anon() or rmap_walk_file() or rmap_walk_ksm(), -it will call try_to_unmap_one() for every VMA which might contain the page. - -When trying to reclaim, if try_to_unmap_one() finds the page in a VM_LOCKED -VMA, it will then mlock the page via mlock_vma_page() instead of unmapping it, -and return SWAP_MLOCK to indicate that the page is unevictable: and the scan -stops there. - -mlock_vma_page() is called while holding the page table's lock (in addition -to the page lock, and the rmap lock): to serialize against concurrent mlock or -munlock or munmap system calls, mm teardown (munlock_vma_pages_all), reclaim, -holepunching, and truncation of file pages and their anonymous COWed pages. - - -try_to_munlock() REVERSE MAP SCAN ---------------------------------- - - [!] TODO/FIXME: a better name might be page_mlocked() - analogous to the - page_referenced() reverse map walker. - -When munlock_vma_page() [see section "munlock()/munlockall() System Call -Handling" above] tries to munlock a page, it needs to determine whether or not -the page is mapped by any VM_LOCKED VMA without actually attempting to unmap -all PTEs from the page. For this purpose, the unevictable/mlock infrastructure -introduced a variant of try_to_unmap() called try_to_munlock(). - -try_to_munlock() calls the same functions as try_to_unmap() for anonymous and -mapped file and KSM pages with a flag argument specifying unlock versus unmap -processing. Again, these functions walk the respective reverse maps looking -for VM_LOCKED VMAs. When such a VMA is found, as in the try_to_unmap() case, -the functions mlock the page via mlock_vma_page() and return SWAP_MLOCK. This -undoes the pre-clearing of the page's PG_mlocked done by munlock_vma_page. - -Note that try_to_munlock()'s reverse map walk must visit every VMA in a page's -reverse map to determine that a page is NOT mapped into any VM_LOCKED VMA. -However, the scan can terminate when it encounters a VM_LOCKED VMA. -Although try_to_munlock() might be called a great many times when munlocking a -large region or tearing down a large address space that has been mlocked via -mlockall(), overall this is a fairly rare event. - - -PAGE RECLAIM IN shrink_*_list() -------------------------------- - -shrink_active_list() culls any obviously unevictable pages - i.e. -!page_evictable(page) - diverting these to the unevictable list. -However, shrink_active_list() only sees unevictable pages that made it onto the -active/inactive lru lists. Note that these pages do not have PageUnevictable -set - otherwise they would be on the unevictable list and shrink_active_list -would never see them. - -Some examples of these unevictable pages on the LRU lists are: - - (1) ramfs pages that have been placed on the LRU lists when first allocated. - - (2) SHM_LOCK'd shared memory pages. shmctl(SHM_LOCK) does not attempt to - allocate or fault in the pages in the shared memory region. This happens - when an application accesses the page the first time after SHM_LOCK'ing - the segment. - - (3) mlocked pages that could not be isolated from the LRU and moved to the - unevictable list in mlock_vma_page(). - -shrink_inactive_list() also diverts any unevictable pages that it finds on the -inactive lists to the appropriate zone's unevictable list. - -shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd -after shrink_active_list() had moved them to the inactive list, or pages mapped -into VM_LOCKED VMAs that munlock_vma_page() couldn't isolate from the LRU to -recheck via try_to_munlock(). shrink_inactive_list() won't notice the latter, -but will pass on to shrink_page_list(). - -shrink_page_list() again culls obviously unevictable pages that it could -encounter for similar reason to shrink_inactive_list(). Pages mapped into -VM_LOCKED VMAs but without PG_mlocked set will make it all the way to -try_to_unmap(). shrink_page_list() will divert them to the unevictable list -when try_to_unmap() returns SWAP_MLOCK, as discussed above. diff --git a/Documentation/vm/userfaultfd.txt b/Documentation/vm/userfaultfd.txt deleted file mode 100644 index bb2f945f87ab..000000000000 --- a/Documentation/vm/userfaultfd.txt +++ /dev/null @@ -1,229 +0,0 @@ -= Userfaultfd = - -== Objective == - -Userfaults allow the implementation of on-demand paging from userland -and more generally they allow userland to take control of various -memory page faults, something otherwise only the kernel code could do. - -For example userfaults allows a proper and more optimal implementation -of the PROT_NONE+SIGSEGV trick. - -== Design == - -Userfaults are delivered and resolved through the userfaultfd syscall. - -The userfaultfd (aside from registering and unregistering virtual -memory ranges) provides two primary functionalities: - -1) read/POLLIN protocol to notify a userland thread of the faults - happening - -2) various UFFDIO_* ioctls that can manage the virtual memory regions - registered in the userfaultfd that allows userland to efficiently - resolve the userfaults it receives via 1) or to manage the virtual - memory in the background - -The real advantage of userfaults if compared to regular virtual memory -management of mremap/mprotect is that the userfaults in all their -operations never involve heavyweight structures like vmas (in fact the -userfaultfd runtime load never takes the mmap_sem for writing). - -Vmas are not suitable for page- (or hugepage) granular fault tracking -when dealing with virtual address spaces that could span -Terabytes. Too many vmas would be needed for that. - -The userfaultfd once opened by invoking the syscall, can also be -passed using unix domain sockets to a manager process, so the same -manager process could handle the userfaults of a multitude of -different processes without them being aware about what is going on -(well of course unless they later try to use the userfaultfd -themselves on the same region the manager is already tracking, which -is a corner case that would currently return -EBUSY). - -== API == - -When first opened the userfaultfd must be enabled invoking the -UFFDIO_API ioctl specifying a uffdio_api.api value set to UFFD_API (or -a later API version) which will specify the read/POLLIN protocol -userland intends to speak on the UFFD and the uffdio_api.features -userland requires. The UFFDIO_API ioctl if successful (i.e. if the -requested uffdio_api.api is spoken also by the running kernel and the -requested features are going to be enabled) will return into -uffdio_api.features and uffdio_api.ioctls two 64bit bitmasks of -respectively all the available features of the read(2) protocol and -the generic ioctl available. - -The uffdio_api.features bitmask returned by the UFFDIO_API ioctl -defines what memory types are supported by the userfaultfd and what -events, except page fault notifications, may be generated. - -If the kernel supports registering userfaultfd ranges on hugetlbfs -virtual memory areas, UFFD_FEATURE_MISSING_HUGETLBFS will be set in -uffdio_api.features. Similarly, UFFD_FEATURE_MISSING_SHMEM will be -set if the kernel supports registering userfaultfd ranges on shared -memory (covering all shmem APIs, i.e. tmpfs, IPCSHM, /dev/zero -MAP_SHARED, memfd_create, etc). - -The userland application that wants to use userfaultfd with hugetlbfs -or shared memory need to set the corresponding flag in -uffdio_api.features to enable those features. - -If the userland desires to receive notifications for events other than -page faults, it has to verify that uffdio_api.features has appropriate -UFFD_FEATURE_EVENT_* bits set. These events are described in more -detail below in "Non-cooperative userfaultfd" section. - -Once the userfaultfd has been enabled the UFFDIO_REGISTER ioctl should -be invoked (if present in the returned uffdio_api.ioctls bitmask) to -register a memory range in the userfaultfd by setting the -uffdio_register structure accordingly. The uffdio_register.mode -bitmask will specify to the kernel which kind of faults to track for -the range (UFFDIO_REGISTER_MODE_MISSING would track missing -pages). The UFFDIO_REGISTER ioctl will return the -uffdio_register.ioctls bitmask of ioctls that are suitable to resolve -userfaults on the range registered. Not all ioctls will necessarily be -supported for all memory types depending on the underlying virtual -memory backend (anonymous memory vs tmpfs vs real filebacked -mappings). - -Userland can use the uffdio_register.ioctls to manage the virtual -address space in the background (to add or potentially also remove -memory from the userfaultfd registered range). This means a userfault -could be triggering just before userland maps in the background the -user-faulted page. - -The primary ioctl to resolve userfaults is UFFDIO_COPY. That -atomically copies a page into the userfault registered range and wakes -up the blocked userfaults (unless uffdio_copy.mode & -UFFDIO_COPY_MODE_DONTWAKE is set). Other ioctl works similarly to -UFFDIO_COPY. They're atomic as in guaranteeing that nothing can see an -half copied page since it'll keep userfaulting until the copy has -finished. - -== QEMU/KVM == - -QEMU/KVM is using the userfaultfd syscall to implement postcopy live -migration. Postcopy live migration is one form of memory -externalization consisting of a virtual machine running with part or -all of its memory residing on a different node in the cloud. The -userfaultfd abstraction is generic enough that not a single line of -KVM kernel code had to be modified in order to add postcopy live -migration to QEMU. - -Guest async page faults, FOLL_NOWAIT and all other GUP features work -just fine in combination with userfaults. Userfaults trigger async -page faults in the guest scheduler so those guest processes that -aren't waiting for userfaults (i.e. network bound) can keep running in -the guest vcpus. - -It is generally beneficial to run one pass of precopy live migration -just before starting postcopy live migration, in order to avoid -generating userfaults for readonly guest regions. - -The implementation of postcopy live migration currently uses one -single bidirectional socket but in the future two different sockets -will be used (to reduce the latency of the userfaults to the minimum -possible without having to decrease /proc/sys/net/ipv4/tcp_wmem). - -The QEMU in the source node writes all pages that it knows are missing -in the destination node, into the socket, and the migration thread of -the QEMU running in the destination node runs UFFDIO_COPY|ZEROPAGE -ioctls on the userfaultfd in order to map the received pages into the -guest (UFFDIO_ZEROCOPY is used if the source page was a zero page). - -A different postcopy thread in the destination node listens with -poll() to the userfaultfd in parallel. When a POLLIN event is -generated after a userfault triggers, the postcopy thread read() from -the userfaultfd and receives the fault address (or -EAGAIN in case the -userfault was already resolved and waken by a UFFDIO_COPY|ZEROPAGE run -by the parallel QEMU migration thread). - -After the QEMU postcopy thread (running in the destination node) gets -the userfault address it writes the information about the missing page -into the socket. The QEMU source node receives the information and -roughly "seeks" to that page address and continues sending all -remaining missing pages from that new page offset. Soon after that -(just the time to flush the tcp_wmem queue through the network) the -migration thread in the QEMU running in the destination node will -receive the page that triggered the userfault and it'll map it as -usual with the UFFDIO_COPY|ZEROPAGE (without actually knowing if it -was spontaneously sent by the source or if it was an urgent page -requested through a userfault). - -By the time the userfaults start, the QEMU in the destination node -doesn't need to keep any per-page state bitmap relative to the live -migration around and a single per-page bitmap has to be maintained in -the QEMU running in the source node to know which pages are still -missing in the destination node. The bitmap in the source node is -checked to find which missing pages to send in round robin and we seek -over it when receiving incoming userfaults. After sending each page of -course the bitmap is updated accordingly. It's also useful to avoid -sending the same page twice (in case the userfault is read by the -postcopy thread just before UFFDIO_COPY|ZEROPAGE runs in the migration -thread). - -== Non-cooperative userfaultfd == - -When the userfaultfd is monitored by an external manager, the manager -must be able to track changes in the process virtual memory -layout. Userfaultfd can notify the manager about such changes using -the same read(2) protocol as for the page fault notifications. The -manager has to explicitly enable these events by setting appropriate -bits in uffdio_api.features passed to UFFDIO_API ioctl: - -UFFD_FEATURE_EVENT_FORK - enable userfaultfd hooks for fork(). When -this feature is enabled, the userfaultfd context of the parent process -is duplicated into the newly created process. The manager receives -UFFD_EVENT_FORK with file descriptor of the new userfaultfd context in -the uffd_msg.fork. - -UFFD_FEATURE_EVENT_REMAP - enable notifications about mremap() -calls. When the non-cooperative process moves a virtual memory area to -a different location, the manager will receive UFFD_EVENT_REMAP. The -uffd_msg.remap will contain the old and new addresses of the area and -its original length. - -UFFD_FEATURE_EVENT_REMOVE - enable notifications about -madvise(MADV_REMOVE) and madvise(MADV_DONTNEED) calls. The event -UFFD_EVENT_REMOVE will be generated upon these calls to madvise. The -uffd_msg.remove will contain start and end addresses of the removed -area. - -UFFD_FEATURE_EVENT_UNMAP - enable notifications about memory -unmapping. The manager will get UFFD_EVENT_UNMAP with uffd_msg.remove -containing start and end addresses of the unmapped area. - -Although the UFFD_FEATURE_EVENT_REMOVE and UFFD_FEATURE_EVENT_UNMAP -are pretty similar, they quite differ in the action expected from the -userfaultfd manager. In the former case, the virtual memory is -removed, but the area is not, the area remains monitored by the -userfaultfd, and if a page fault occurs in that area it will be -delivered to the manager. The proper resolution for such page fault is -to zeromap the faulting address. However, in the latter case, when an -area is unmapped, either explicitly (with munmap() system call), or -implicitly (e.g. during mremap()), the area is removed and in turn the -userfaultfd context for such area disappears too and the manager will -not get further userland page faults from the removed area. Still, the -notification is required in order to prevent manager from using -UFFDIO_COPY on the unmapped area. - -Unlike userland page faults which have to be synchronous and require -explicit or implicit wakeup, all the events are delivered -asynchronously and the non-cooperative process resumes execution as -soon as manager executes read(). The userfaultfd manager should -carefully synchronize calls to UFFDIO_COPY with the events -processing. To aid the synchronization, the UFFDIO_COPY ioctl will -return -ENOSPC when the monitored process exits at the time of -UFFDIO_COPY, and -ENOENT, when the non-cooperative process has changed -its virtual memory layout simultaneously with outstanding UFFDIO_COPY -operation. - -The current asynchronous model of the event delivery is optimal for -single threaded non-cooperative userfaultfd manager implementations. A -synchronous event delivery model can be added later as a new -userfaultfd feature to facilitate multithreading enhancements of the -non cooperative manager, for example to allow UFFDIO_COPY ioctls to -run in parallel to the event reception. Single threaded -implementations should continue to use the current async event -delivery model instead. diff --git a/Documentation/vm/z3fold.rst b/Documentation/vm/z3fold.rst new file mode 100644 index 000000000000..224e3c61d686 --- /dev/null +++ b/Documentation/vm/z3fold.rst @@ -0,0 +1,30 @@ +.. _z3fold: + +====== +z3fold +====== + +z3fold is a special purpose allocator for storing compressed pages. +It is designed to store up to three compressed pages per physical page. +It is a zbud derivative which allows for higher compression +ratio keeping the simplicity and determinism of its predecessor. + +The main differences between z3fold and zbud are: + +* unlike zbud, z3fold allows for up to PAGE_SIZE allocations +* z3fold can hold up to 3 compressed pages in its page +* z3fold doesn't export any API itself and is thus intended to be used + via the zpool API. + +To keep the determinism and simplicity, z3fold, just like zbud, always +stores an integral number of compressed pages per page, but it can store +up to 3 pages unlike zbud which can store at most 2. Therefore the +compression ratio goes to around 2.7x while zbud's one is around 1.7x. + +Unlike zbud (but like zsmalloc for that matter) z3fold_alloc() does not +return a dereferenceable pointer. Instead, it returns an unsigned long +handle which encodes actual location of the allocated object. + +Keeping effective compression ratio close to zsmalloc's, z3fold doesn't +depend on MMU enabled and provides more predictable reclaim behavior +which makes it a better fit for small and response-critical systems. diff --git a/Documentation/vm/z3fold.txt b/Documentation/vm/z3fold.txt deleted file mode 100644 index 38e4dac810b6..000000000000 --- a/Documentation/vm/z3fold.txt +++ /dev/null @@ -1,26 +0,0 @@ -z3fold ------- - -z3fold is a special purpose allocator for storing compressed pages. -It is designed to store up to three compressed pages per physical page. -It is a zbud derivative which allows for higher compression -ratio keeping the simplicity and determinism of its predecessor. - -The main differences between z3fold and zbud are: -* unlike zbud, z3fold allows for up to PAGE_SIZE allocations -* z3fold can hold up to 3 compressed pages in its page -* z3fold doesn't export any API itself and is thus intended to be used - via the zpool API. - -To keep the determinism and simplicity, z3fold, just like zbud, always -stores an integral number of compressed pages per page, but it can store -up to 3 pages unlike zbud which can store at most 2. Therefore the -compression ratio goes to around 2.7x while zbud's one is around 1.7x. - -Unlike zbud (but like zsmalloc for that matter) z3fold_alloc() does not -return a dereferenceable pointer. Instead, it returns an unsigned long -handle which encodes actual location of the allocated object. - -Keeping effective compression ratio close to zsmalloc's, z3fold doesn't -depend on MMU enabled and provides more predictable reclaim behavior -which makes it a better fit for small and response-critical systems. diff --git a/Documentation/vm/zsmalloc.rst b/Documentation/vm/zsmalloc.rst new file mode 100644 index 000000000000..6e79893d6132 --- /dev/null +++ b/Documentation/vm/zsmalloc.rst @@ -0,0 +1,82 @@ +.. _zsmalloc: + +======== +zsmalloc +======== + +This allocator is designed for use with zram. Thus, the allocator is +supposed to work well under low memory conditions. In particular, it +never attempts higher order page allocation which is very likely to +fail under memory pressure. On the other hand, if we just use single +(0-order) pages, it would suffer from very high fragmentation -- +any object of size PAGE_SIZE/2 or larger would occupy an entire page. +This was one of the major issues with its predecessor (xvmalloc). + +To overcome these issues, zsmalloc allocates a bunch of 0-order pages +and links them together using various 'struct page' fields. These linked +pages act as a single higher-order page i.e. an object can span 0-order +page boundaries. The code refers to these linked pages as a single entity +called zspage. + +For simplicity, zsmalloc can only allocate objects of size up to PAGE_SIZE +since this satisfies the requirements of all its current users (in the +worst case, page is incompressible and is thus stored "as-is" i.e. in +uncompressed form). For allocation requests larger than this size, failure +is returned (see zs_malloc). + +Additionally, zs_malloc() does not return a dereferenceable pointer. +Instead, it returns an opaque handle (unsigned long) which encodes actual +location of the allocated object. The reason for this indirection is that +zsmalloc does not keep zspages permanently mapped since that would cause +issues on 32-bit systems where the VA region for kernel space mappings +is very small. So, before using the allocating memory, the object has to +be mapped using zs_map_object() to get a usable pointer and subsequently +unmapped using zs_unmap_object(). + +stat +==== + +With CONFIG_ZSMALLOC_STAT, we could see zsmalloc internal information via +``/sys/kernel/debug/zsmalloc/``. Here is a sample of stat output:: + + # cat /sys/kernel/debug/zsmalloc/zram0/classes + + class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage + ... + ... + 9 176 0 1 186 129 8 4 + 10 192 1 0 2880 2872 135 3 + 11 208 0 1 819 795 42 2 + 12 224 0 1 219 159 12 4 + ... + ... + + +class + index +size + object size zspage stores +almost_empty + the number of ZS_ALMOST_EMPTY zspages(see below) +almost_full + the number of ZS_ALMOST_FULL zspages(see below) +obj_allocated + the number of objects allocated +obj_used + the number of objects allocated to the user +pages_used + the number of pages allocated for the class +pages_per_zspage + the number of 0-order pages to make a zspage + +We assign a zspage to ZS_ALMOST_EMPTY fullness group when n <= N / f, where + +* n = number of allocated objects +* N = total number of objects zspage can store +* f = fullness_threshold_frac(ie, 4 at the moment) + +Similarly, we assign zspage to: + +* ZS_ALMOST_FULL when n > N / f +* ZS_EMPTY when n == 0 +* ZS_FULL when n == N diff --git a/Documentation/vm/zsmalloc.txt b/Documentation/vm/zsmalloc.txt deleted file mode 100644 index 64ed63c4f69d..000000000000 --- a/Documentation/vm/zsmalloc.txt +++ /dev/null @@ -1,70 +0,0 @@ -zsmalloc --------- - -This allocator is designed for use with zram. Thus, the allocator is -supposed to work well under low memory conditions. In particular, it -never attempts higher order page allocation which is very likely to -fail under memory pressure. On the other hand, if we just use single -(0-order) pages, it would suffer from very high fragmentation -- -any object of size PAGE_SIZE/2 or larger would occupy an entire page. -This was one of the major issues with its predecessor (xvmalloc). - -To overcome these issues, zsmalloc allocates a bunch of 0-order pages -and links them together using various 'struct page' fields. These linked -pages act as a single higher-order page i.e. an object can span 0-order -page boundaries. The code refers to these linked pages as a single entity -called zspage. - -For simplicity, zsmalloc can only allocate objects of size up to PAGE_SIZE -since this satisfies the requirements of all its current users (in the -worst case, page is incompressible and is thus stored "as-is" i.e. in -uncompressed form). For allocation requests larger than this size, failure -is returned (see zs_malloc). - -Additionally, zs_malloc() does not return a dereferenceable pointer. -Instead, it returns an opaque handle (unsigned long) which encodes actual -location of the allocated object. The reason for this indirection is that -zsmalloc does not keep zspages permanently mapped since that would cause -issues on 32-bit systems where the VA region for kernel space mappings -is very small. So, before using the allocating memory, the object has to -be mapped using zs_map_object() to get a usable pointer and subsequently -unmapped using zs_unmap_object(). - -stat ----- - -With CONFIG_ZSMALLOC_STAT, we could see zsmalloc internal information via -/sys/kernel/debug/zsmalloc/. Here is a sample of stat output: - -# cat /sys/kernel/debug/zsmalloc/zram0/classes - - class size almost_full almost_empty obj_allocated obj_used pages_used pages_per_zspage - .. - .. - 9 176 0 1 186 129 8 4 - 10 192 1 0 2880 2872 135 3 - 11 208 0 1 819 795 42 2 - 12 224 0 1 219 159 12 4 - .. - .. - - -class: index -size: object size zspage stores -almost_empty: the number of ZS_ALMOST_EMPTY zspages(see below) -almost_full: the number of ZS_ALMOST_FULL zspages(see below) -obj_allocated: the number of objects allocated -obj_used: the number of objects allocated to the user -pages_used: the number of pages allocated for the class -pages_per_zspage: the number of 0-order pages to make a zspage - -We assign a zspage to ZS_ALMOST_EMPTY fullness group when: - n <= N / f, where -n = number of allocated objects -N = total number of objects zspage can store -f = fullness_threshold_frac(ie, 4 at the moment) - -Similarly, we assign zspage to: - ZS_ALMOST_FULL when n > N / f - ZS_EMPTY when n == 0 - ZS_FULL when n == N diff --git a/Documentation/vm/zswap.rst b/Documentation/vm/zswap.rst new file mode 100644 index 000000000000..1444ecd40911 --- /dev/null +++ b/Documentation/vm/zswap.rst @@ -0,0 +1,135 @@ +.. _zswap: + +===== +zswap +===== + +Overview +======== + +Zswap is a lightweight compressed cache for swap pages. It takes pages that are +in the process of being swapped out and attempts to compress them into a +dynamically allocated RAM-based memory pool. zswap basically trades CPU cycles +for potentially reduced swap I/O.  This trade-off can also result in a +significant performance improvement if reads from the compressed cache are +faster than reads from a swap device. + +.. note:: + Zswap is a new feature as of v3.11 and interacts heavily with memory + reclaim. This interaction has not been fully explored on the large set of + potential configurations and workloads that exist. For this reason, zswap + is a work in progress and should be considered experimental. + + Some potential benefits: + +* Desktop/laptop users with limited RAM capacities can mitigate the + performance impact of swapping. +* Overcommitted guests that share a common I/O resource can + dramatically reduce their swap I/O pressure, avoiding heavy handed I/O + throttling by the hypervisor. This allows more work to get done with less + impact to the guest workload and guests sharing the I/O subsystem +* Users with SSDs as swap devices can extend the life of the device by + drastically reducing life-shortening writes. + +Zswap evicts pages from compressed cache on an LRU basis to the backing swap +device when the compressed pool reaches its size limit. This requirement had +been identified in prior community discussions. + +Zswap is disabled by default but can be enabled at boot time by setting +the ``enabled`` attribute to 1 at boot time. ie: ``zswap.enabled=1``. Zswap +can also be enabled and disabled at runtime using the sysfs interface. +An example command to enable zswap at runtime, assuming sysfs is mounted +at ``/sys``, is:: + + echo 1 > /sys/module/zswap/parameters/enabled + +When zswap is disabled at runtime it will stop storing pages that are +being swapped out. However, it will _not_ immediately write out or fault +back into memory all of the pages stored in the compressed pool. The +pages stored in zswap will remain in the compressed pool until they are +either invalidated or faulted back into memory. In order to force all +pages out of the compressed pool, a swapoff on the swap device(s) will +fault back into memory all swapped out pages, including those in the +compressed pool. + +Design +====== + +Zswap receives pages for compression through the Frontswap API and is able to +evict pages from its own compressed pool on an LRU basis and write them back to +the backing swap device in the case that the compressed pool is full. + +Zswap makes use of zpool for the managing the compressed memory pool. Each +allocation in zpool is not directly accessible by address. Rather, a handle is +returned by the allocation routine and that handle must be mapped before being +accessed. The compressed memory pool grows on demand and shrinks as compressed +pages are freed. The pool is not preallocated. By default, a zpool +of type zbud is created, but it can be selected at boot time by +setting the ``zpool`` attribute, e.g. ``zswap.zpool=zbud``. It can +also be changed at runtime using the sysfs ``zpool`` attribute, e.g.:: + + echo zbud > /sys/module/zswap/parameters/zpool + +The zbud type zpool allocates exactly 1 page to store 2 compressed pages, which +means the compression ratio will always be 2:1 or worse (because of half-full +zbud pages). The zsmalloc type zpool has a more complex compressed page +storage method, and it can achieve greater storage densities. However, +zsmalloc does not implement compressed page eviction, so once zswap fills it +cannot evict the oldest page, it can only reject new pages. + +When a swap page is passed from frontswap to zswap, zswap maintains a mapping +of the swap entry, a combination of the swap type and swap offset, to the zpool +handle that references that compressed swap page. This mapping is achieved +with a red-black tree per swap type. The swap offset is the search key for the +tree nodes. + +During a page fault on a PTE that is a swap entry, frontswap calls the zswap +load function to decompress the page into the page allocated by the page fault +handler. + +Once there are no PTEs referencing a swap page stored in zswap (i.e. the count +in the swap_map goes to 0) the swap code calls the zswap invalidate function, +via frontswap, to free the compressed entry. + +Zswap seeks to be simple in its policies. Sysfs attributes allow for one user +controlled policy: + +* max_pool_percent - The maximum percentage of memory that the compressed + pool can occupy. + +The default compressor is lzo, but it can be selected at boot time by +setting the ``compressor`` attribute, e.g. ``zswap.compressor=lzo``. +It can also be changed at runtime using the sysfs "compressor" +attribute, e.g.:: + + echo lzo > /sys/module/zswap/parameters/compressor + +When the zpool and/or compressor parameter is changed at runtime, any existing +compressed pages are not modified; they are left in their own zpool. When a +request is made for a page in an old zpool, it is uncompressed using its +original compressor. Once all pages are removed from an old zpool, the zpool +and its compressor are freed. + +Some of the pages in zswap are same-value filled pages (i.e. contents of the +page have same value or repetitive pattern). These pages include zero-filled +pages and they are handled differently. During store operation, a page is +checked if it is a same-value filled page before compressing it. If true, the +compressed length of the page is set to zero and the pattern or same-filled +value is stored. + +Same-value filled pages identification feature is enabled by default and can be +disabled at boot time by setting the ``same_filled_pages_enabled`` attribute +to 0, e.g. ``zswap.same_filled_pages_enabled=0``. It can also be enabled and +disabled at runtime using the sysfs ``same_filled_pages_enabled`` +attribute, e.g.:: + + echo 1 > /sys/module/zswap/parameters/same_filled_pages_enabled + +When zswap same-filled page identification is disabled at runtime, it will stop +checking for the same-value filled pages during store operation. However, the +existing pages which are marked as same-value filled pages remain stored +unchanged in zswap until they are either loaded or invalidated. + +A debugfs interface is provided for various statistic about pool size, number +of pages stored, same-value filled pages and various counters for the reasons +pages are rejected. diff --git a/Documentation/vm/zswap.txt b/Documentation/vm/zswap.txt deleted file mode 100644 index 0b3a1148f9f0..000000000000 --- a/Documentation/vm/zswap.txt +++ /dev/null @@ -1,122 +0,0 @@ -Overview: - -Zswap is a lightweight compressed cache for swap pages. It takes pages that are -in the process of being swapped out and attempts to compress them into a -dynamically allocated RAM-based memory pool. zswap basically trades CPU cycles -for potentially reduced swap I/O.  This trade-off can also result in a -significant performance improvement if reads from the compressed cache are -faster than reads from a swap device. - -NOTE: Zswap is a new feature as of v3.11 and interacts heavily with memory -reclaim. This interaction has not been fully explored on the large set of -potential configurations and workloads that exist. For this reason, zswap -is a work in progress and should be considered experimental. - -Some potential benefits: -* Desktop/laptop users with limited RAM capacities can mitigate the -    performance impact of swapping. -* Overcommitted guests that share a common I/O resource can -    dramatically reduce their swap I/O pressure, avoiding heavy handed I/O - throttling by the hypervisor. This allows more work to get done with less - impact to the guest workload and guests sharing the I/O subsystem -* Users with SSDs as swap devices can extend the life of the device by -    drastically reducing life-shortening writes. - -Zswap evicts pages from compressed cache on an LRU basis to the backing swap -device when the compressed pool reaches its size limit. This requirement had -been identified in prior community discussions. - -Zswap is disabled by default but can be enabled at boot time by setting -the "enabled" attribute to 1 at boot time. ie: zswap.enabled=1. Zswap -can also be enabled and disabled at runtime using the sysfs interface. -An example command to enable zswap at runtime, assuming sysfs is mounted -at /sys, is: - -echo 1 > /sys/module/zswap/parameters/enabled - -When zswap is disabled at runtime it will stop storing pages that are -being swapped out. However, it will _not_ immediately write out or fault -back into memory all of the pages stored in the compressed pool. The -pages stored in zswap will remain in the compressed pool until they are -either invalidated or faulted back into memory. In order to force all -pages out of the compressed pool, a swapoff on the swap device(s) will -fault back into memory all swapped out pages, including those in the -compressed pool. - -Design: - -Zswap receives pages for compression through the Frontswap API and is able to -evict pages from its own compressed pool on an LRU basis and write them back to -the backing swap device in the case that the compressed pool is full. - -Zswap makes use of zpool for the managing the compressed memory pool. Each -allocation in zpool is not directly accessible by address. Rather, a handle is -returned by the allocation routine and that handle must be mapped before being -accessed. The compressed memory pool grows on demand and shrinks as compressed -pages are freed. The pool is not preallocated. By default, a zpool of type -zbud is created, but it can be selected at boot time by setting the "zpool" -attribute, e.g. zswap.zpool=zbud. It can also be changed at runtime using the -sysfs "zpool" attribute, e.g. - -echo zbud > /sys/module/zswap/parameters/zpool - -The zbud type zpool allocates exactly 1 page to store 2 compressed pages, which -means the compression ratio will always be 2:1 or worse (because of half-full -zbud pages). The zsmalloc type zpool has a more complex compressed page -storage method, and it can achieve greater storage densities. However, -zsmalloc does not implement compressed page eviction, so once zswap fills it -cannot evict the oldest page, it can only reject new pages. - -When a swap page is passed from frontswap to zswap, zswap maintains a mapping -of the swap entry, a combination of the swap type and swap offset, to the zpool -handle that references that compressed swap page. This mapping is achieved -with a red-black tree per swap type. The swap offset is the search key for the -tree nodes. - -During a page fault on a PTE that is a swap entry, frontswap calls the zswap -load function to decompress the page into the page allocated by the page fault -handler. - -Once there are no PTEs referencing a swap page stored in zswap (i.e. the count -in the swap_map goes to 0) the swap code calls the zswap invalidate function, -via frontswap, to free the compressed entry. - -Zswap seeks to be simple in its policies. Sysfs attributes allow for one user -controlled policy: -* max_pool_percent - The maximum percentage of memory that the compressed - pool can occupy. - -The default compressor is lzo, but it can be selected at boot time by setting -the “compressor” attribute, e.g. zswap.compressor=lzo. It can also be changed -at runtime using the sysfs "compressor" attribute, e.g. - -echo lzo > /sys/module/zswap/parameters/compressor - -When the zpool and/or compressor parameter is changed at runtime, any existing -compressed pages are not modified; they are left in their own zpool. When a -request is made for a page in an old zpool, it is uncompressed using its -original compressor. Once all pages are removed from an old zpool, the zpool -and its compressor are freed. - -Some of the pages in zswap are same-value filled pages (i.e. contents of the -page have same value or repetitive pattern). These pages include zero-filled -pages and they are handled differently. During store operation, a page is -checked if it is a same-value filled page before compressing it. If true, the -compressed length of the page is set to zero and the pattern or same-filled -value is stored. - -Same-value filled pages identification feature is enabled by default and can be -disabled at boot time by setting the "same_filled_pages_enabled" attribute to 0, -e.g. zswap.same_filled_pages_enabled=0. It can also be enabled and disabled at -runtime using the sysfs "same_filled_pages_enabled" attribute, e.g. - -echo 1 > /sys/module/zswap/parameters/same_filled_pages_enabled - -When zswap same-filled page identification is disabled at runtime, it will stop -checking for the same-value filled pages during store operation. However, the -existing pages which are marked as same-value filled pages remain stored -unchanged in zswap until they are either loaded or invalidated. - -A debugfs interface is provided for various statistic about pool size, number -of pages stored, same-value filled pages and various counters for the reasons -pages are rejected. diff --git a/Documentation/x86/intel_rdt_ui.txt b/Documentation/x86/intel_rdt_ui.txt index 71c30984e94d..a16aa2113840 100644 --- a/Documentation/x86/intel_rdt_ui.txt +++ b/Documentation/x86/intel_rdt_ui.txt @@ -17,12 +17,14 @@ MBA (Memory Bandwidth Allocation) - "mba" To use the feature mount the file system: - # mount -t resctrl resctrl [-o cdp[,cdpl2]] /sys/fs/resctrl + # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps]] /sys/fs/resctrl mount options are: "cdp": Enable code/data prioritization in L3 cache allocations. "cdpl2": Enable code/data prioritization in L2 cache allocations. +"mba_MBps": Enable the MBA Software Controller(mba_sc) to specify MBA + bandwidth in MBps L2 and L3 CDP are controlled seperately. @@ -270,10 +272,11 @@ and 0xA are not. On a system with a 20-bit mask each bit represents 5% of the capacity of the cache. You could partition the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000. -Memory bandwidth(b/w) percentage --------------------------------- -For Memory b/w resource, user controls the resource by indicating the -percentage of total memory b/w. +Memory bandwidth Allocation and monitoring +------------------------------------------ + +For Memory bandwidth resource, by default the user controls the resource +by indicating the percentage of total memory bandwidth. The minimum bandwidth percentage value for each cpu model is predefined and can be looked up through "info/MB/min_bandwidth". The bandwidth @@ -285,7 +288,47 @@ to the next control step available on the hardware. The bandwidth throttling is a core specific mechanism on some of Intel SKUs. Using a high bandwidth and a low bandwidth setting on two threads sharing a core will result in both threads being throttled to use the -low bandwidth. +low bandwidth. The fact that Memory bandwidth allocation(MBA) is a core +specific mechanism where as memory bandwidth monitoring(MBM) is done at +the package level may lead to confusion when users try to apply control +via the MBA and then monitor the bandwidth to see if the controls are +effective. Below are such scenarios: + +1. User may *not* see increase in actual bandwidth when percentage + values are increased: + +This can occur when aggregate L2 external bandwidth is more than L3 +external bandwidth. Consider an SKL SKU with 24 cores on a package and +where L2 external is 10GBps (hence aggregate L2 external bandwidth is +240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20 +threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3 +bandwidth of 100GBps although the percentage value specified is only 50% +<< 100%. Hence increasing the bandwidth percentage will not yeild any +more bandwidth. This is because although the L2 external bandwidth still +has capacity, the L3 external bandwidth is fully used. Also note that +this would be dependent on number of cores the benchmark is run on. + +2. Same bandwidth percentage may mean different actual bandwidth + depending on # of threads: + +For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4 +thread, with 10% bandwidth' can consume upto 10GBps and 40GBps although +they have same percentage bandwidth of 10%. This is simply because as +threads start using more cores in an rdtgroup, the actual bandwidth may +increase or vary although user specified bandwidth percentage is same. + +In order to mitigate this and make the interface more user friendly, +resctrl added support for specifying the bandwidth in MBps as well. The +kernel underneath would use a software feedback mechanism or a "Software +Controller(mba_sc)" which reads the actual bandwidth using MBM counters +and adjust the memowy bandwidth percentages to ensure + + "actual bandwidth < user specified bandwidth". + +By default, the schemata would take the bandwidth percentage values +where as user can switch to the "MBA software controller" mode using +a mount option 'mba_MBps'. The schemata format is specified in the below +sections. L3 schemata file details (code and data prioritization disabled) ---------------------------------------------------------------- @@ -308,13 +351,20 @@ schemata format is always: L2:=;=;... -Memory b/w Allocation details ------------------------------ +Memory bandwidth Allocation (default mode) +------------------------------------------ Memory b/w domain is L3 cache. MB:=bandwidth0;=bandwidth1;... +Memory bandwidth Allocation specified in MBps +--------------------------------------------- + +Memory bandwidth domain is L3 cache. + + MB:=bw_MBps0;=bw_MBps1;... + Reading/writing the schemata file --------------------------------- Reading the schemata file will show the state of all resources @@ -358,6 +408,15 @@ allocations can overlap or not. The allocations specifies the maximum b/w that the group may be able to use and the system admin can configure the b/w accordingly. +If the MBA is specified in MB(megabytes) then user can enter the max b/w in MB +rather than the percentage values. + +# echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata +# echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata + +In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w +of 1024MB where as on socket 1 they would use 500MB. + Example 2 --------- Again two sockets, but this time with a more realistic 20-bit mask. diff --git a/Documentation/x86/x86_64/boot-options.txt b/Documentation/x86/x86_64/boot-options.txt index b297c48389b9..8d109ef67ab6 100644 --- a/Documentation/x86/x86_64/boot-options.txt +++ b/Documentation/x86/x86_64/boot-options.txt @@ -187,9 +187,9 @@ PCI IOMMU (input/output memory management unit) - Currently four x86-64 PCI-DMA mapping implementations exist: + Multiple x86-64 PCI-DMA mapping implementations exist, for example: - 1. : use no hardware/software IOMMU at all + 1. : use no hardware/software IOMMU at all (e.g. because you have < 3 GB memory). Kernel boot message: "PCI-DMA: Disabling IOMMU" @@ -208,7 +208,7 @@ IOMMU (input/output memory management unit) Kernel boot message: "PCI-DMA: Using Calgary IOMMU" iommu=[][,noagp][,off][,force][,noforce][,leak[=] - [,memaper[=]][,merge][,forcesac][,fullflush][,nomerge] + [,memaper[=]][,merge][,fullflush][,nomerge] [,noaperture][,calgary] General iommu options: @@ -235,14 +235,7 @@ IOMMU (input/output memory management unit) (experimental). nomerge Don't do scatter-gather (SG) merging. noaperture Ask the IOMMU not to touch the aperture for AGP. - forcesac Force single-address cycle (SAC) mode for masks <40bits - (experimental). noagp Don't initialize the AGP driver and use full aperture. - allowdac Allow double-address cycle (DAC) mode, i.e. DMA >4GB. - DAC is used with 32-bit PCI to push a 64-bit address in - two cycles. When off all DMA over >4GB is forced through - an IOMMU or software bounce buffering. - nodac Forbid DAC mode, i.e. DMA >4GB. panic Always panic when IOMMU overflows. calgary Use the Calgary IOMMU if it is available diff --git a/Kconfig b/Kconfig index 8c4c1cb0f9cd..a90d9f9e268b 100644 --- a/Kconfig +++ b/Kconfig @@ -3,10 +3,10 @@ # For a description of the syntax of this configuration file, # see Documentation/kbuild/kconfig-language.txt. # -mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration" +mainmenu "Linux/$(ARCH) $(KERNELVERSION) Kernel Configuration" -config SRCARCH - string - option env="SRCARCH" +comment "Compiler: $(CC_VERSION_TEXT)" -source "arch/$SRCARCH/Kconfig" +source "scripts/Kconfig.include" + +source "arch/$(SRCARCH)/Kconfig" diff --git a/LICENSES/exceptions/Linux-syscall-note b/LICENSES/exceptions/Linux-syscall-note index 6b60b61be4e9..9abdad71fafd 100644 --- a/LICENSES/exceptions/Linux-syscall-note +++ b/LICENSES/exceptions/Linux-syscall-note @@ -1,6 +1,6 @@ SPDX-Exception-Identifier: Linux-syscall-note SPDX-URL: https://spdx.org/licenses/Linux-syscall-note.html -SPDX-Licenses: GPL-2.0, GPL-2.0+, GPL-1.0+, LGPL-2.0, LGPL-2.0+, LGPL-2.1, LGPL-2.1+ +SPDX-Licenses: GPL-2.0, GPL-2.0+, GPL-1.0+, LGPL-2.0, LGPL-2.0+, LGPL-2.1, LGPL-2.1+, GPL-2.0-only, GPL-2.0-or-later Usage-Guide: This exception is used together with one of the above SPDX-Licenses to mark user space API (uapi) header files so they can be included diff --git a/LICENSES/other/Apache-2.0 b/LICENSES/other/Apache-2.0 new file mode 100644 index 000000000000..7cd903f573e5 --- /dev/null +++ b/LICENSES/other/Apache-2.0 @@ -0,0 +1,183 @@ +Valid-License-Identifier: Apache-2.0 +SPDX-URL: https://spdx.org/licenses/Apache-2.0.html +Usage-Guide: + To use the Apache License version 2.0 put the following SPDX tag/value + pair into a comment according to the placement guidelines in the + licensing rules documentation: + SPDX-License-Identifier: Apache-2.0 +License-Text: + +Apache License + +Version 2.0, January 2004 + +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, and +distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the +copyright owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other +entities that control, are controlled by, or are under common control with +that entity. For the purposes of this definition, "control" means (i) the +power, direct or indirect, to cause the direction or management of such +entity, whether by contract or otherwise, or (ii) ownership of fifty +percent (50%) or more of the outstanding shares, or (iii) beneficial +ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising +permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, +including but not limited to software source code, documentation source, +and configuration files. + +"Object" form shall mean any form resulting from mechanical transformation +or translation of a Source form, including but not limited to compiled +object code, generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, +made available under the License, as indicated by a copyright notice that +is included in or attached to the work (an example is provided in the +Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object form, +that is based on (or derived from) the Work and for which the editorial +revisions, annotations, elaborations, or other modifications represent, as +a whole, an original work of authorship. For the purposes of this License, +Derivative Works shall not include works that remain separable from, or +merely link (or bind by name) to the interfaces of, the Work and Derivative +Works thereof. + +"Contribution" shall mean any work of authorship, including the original +version of the Work and any modifications or additions to that Work or +Derivative Works thereof, that is intentionally submitted to Licensor for +inclusion in the Work by the copyright owner or by an individual or Legal +Entity authorized to submit on behalf of the copyright owner. For the +purposes of this definition, "submitted" means any form of electronic, +verbal, or written communication sent to the Licensor or its +representatives, including but not limited to communication on electronic +mailing lists, source code control systems, and issue tracking systems that +are managed by, or on behalf of, the Licensor for the purpose of discussing +and improving the Work, but excluding communication that is conspicuously +marked or otherwise designated in writing by the copyright owner as "Not a +Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on +behalf of whom a Contribution has been received by Licensor and +subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of this + License, each Contributor hereby grants to You a perpetual, worldwide, + non-exclusive, no-charge, royalty-free, irrevocable copyright license to + reproduce, prepare Derivative Works of, publicly display, publicly + perform, sublicense, and distribute the Work and such Derivative Works + in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of this + License, each Contributor hereby grants to You a perpetual, worldwide, + non-exclusive, no-charge, royalty-free, irrevocable (except as stated in + this section) patent license to make, have made, use, offer to sell, + sell, import, and otherwise transfer the Work, where such license + applies only to those patent claims licensable by such Contributor that + are necessarily infringed by their Contribution(s) alone or by + combination of their Contribution(s) with the Work to which such + Contribution(s) was submitted. If You institute patent litigation + against any entity (including a cross-claim or counterclaim in a + lawsuit) alleging that the Work or a Contribution incorporated within + the Work constitutes direct or contributory patent infringement, then + any patent licenses granted to You under this License for that Work + shall terminate as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the Work or + Derivative Works thereof in any medium, with or without modifications, + and in Source or Object form, provided that You meet the following + conditions: + + a. You must give any other recipients of the Work or Derivative Works a + copy of this License; and + + b. You must cause any modified files to carry prominent notices stating + that You changed the files; and + + c. You must retain, in the Source form of any Derivative Works that You + distribute, all copyright, patent, trademark, and attribution notices + from the Source form of the Work, excluding those notices that do not + pertain to any part of the Derivative Works; and + + d. If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained within + such NOTICE file, excluding those notices that do not pertain to any + part of the Derivative Works, in at least one of the following + places: within a NOTICE text file distributed as part of the + Derivative Works; within the Source form or documentation, if + provided along with the Derivative Works; or, within a display + generated by the Derivative Works, if and wherever such third-party + notices normally appear. The contents of the NOTICE file are for + informational purposes only and do not modify the License. You may + add Your own attribution notices within Derivative Works that You + distribute, alongside or as an addendum to the NOTICE text from the + Work, provided that such additional attribution notices cannot be + construed as modifying the License. + + You may add Your own copyright statement to Your modifications and may + provide additional or different license terms and conditions for use, + reproduction, or distribution of Your modifications, or for any such + Derivative Works as a whole, provided Your use, reproduction, and + distribution of the Work otherwise complies with the conditions stated + in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, any + Contribution intentionally submitted for inclusion in the Work by You to + the Licensor shall be under the terms and conditions of this License, + without any additional terms or conditions. Notwithstanding the above, + nothing herein shall supersede or modify the terms of any separate + license agreement you may have executed with Licensor regarding such + Contributions. + +6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or agreed to + in writing, Licensor provides the Work (and each Contributor provides + its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS + OF ANY KIND, either express or implied, including, without limitation, + any warranties or conditions of TITLE, NON-INFRINGEMENT, + MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely + responsible for determining the appropriateness of using or + redistributing the Work and assume any risks associated with Your + exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, whether + in tort (including negligence), contract, or otherwise, unless required + by applicable law (such as deliberate and grossly negligent acts) or + agreed to in writing, shall any Contributor be liable to You for + damages, including any direct, indirect, special, incidental, or + consequential damages of any character arising as a result of this + License or out of the use or inability to use the Work (including but + not limited to damages for loss of goodwill, work stoppage, computer + failure or malfunction, or any and all other commercial damages or + losses), even if such Contributor has been advised of the possibility of + such damages. + +9. Accepting Warranty or Additional Liability. While redistributing the + Work or Derivative Works thereof, You may choose to offer, and charge a + fee for, acceptance of support, warranty, indemnity, or other liability + obligations and/or rights consistent with this License. However, in + accepting such obligations, You may act only on Your own behalf and on + Your sole responsibility, not on behalf of any other Contributor, and + only if You agree to indemnify, defend, and hold each Contributor + harmless for any liability incurred by, or claims asserted against, such + Contributor by reason of your accepting any such warranty or additional + liability. + +END OF TERMS AND CONDITIONS diff --git a/LICENSES/other/CC-BY-SA-4.0 b/LICENSES/other/CC-BY-SA-4.0 new file mode 100644 index 000000000000..f9158e831e79 --- /dev/null +++ b/LICENSES/other/CC-BY-SA-4.0 @@ -0,0 +1,397 @@ +Valid-License-Identifier: CC-BY-SA-4.0 +SPDX-URL: https://spdx.org/licenses/CC-BY-SA-4.0 +Usage-Guide: + To use the Creative Commons Attribution Share Alike 4.0 International + license put the following SPDX tag/value pair into a comment according to + the placement guidelines in the licensing rules documentation: + SPDX-License-Identifier: CC-BY-SA-4.0 +License-Text: + +Creative Commons Attribution-ShareAlike 4.0 International + +Creative Commons Corporation ("Creative Commons") is not a law firm and +does not provide legal services or legal advice. Distribution of Creative +Commons public licenses does not create a lawyer-client or other +relationship. Creative Commons makes its licenses and related information +available on an "as-is" basis. Creative Commons gives no warranties +regarding its licenses, any material licensed under their terms and +conditions, or any related information. Creative Commons disclaims all +liability for damages resulting from their use to the fullest extent +possible. + +Using Creative Commons Public Licenses + +Creative Commons public licenses provide a standard set of terms and +conditions that creators and other rights holders may use to share original +works of authorship and other material subject to copyright and certain +other rights specified in the public license below. The following +considerations are for informational purposes only, are not exhaustive, and +do not form part of our licenses. + +Considerations for licensors: Our public licenses are intended for use by +those authorized to give the public permission to use material in ways +otherwise restricted by copyright and certain other rights. Our licenses +are irrevocable. Licensors should read and understand the terms and +conditions of the license they choose before applying it. Licensors should +also secure all rights necessary before applying our licenses so that the +public can reuse the material as expected. Licensors should clearly mark +any material not subject to the license. This includes other CC-licensed +material, or material used under an exception or limitation to +copyright. More considerations for licensors : +wiki.creativecommons.org/Considerations_for_licensors + +Considerations for the public: By using one of our public licenses, a +licensor grants the public permission to use the licensed material under +specified terms and conditions. If the licensor's permission is not +necessary for any reason - for example, because of any applicable exception +or limitation to copyright - then that use is not regulated by the +license. Our licenses grant only permissions under copyright and certain +other rights that a licensor has authority to grant. Use of the licensed +material may still be restricted for other reasons, including because +others have copyright or other rights in the material. A licensor may make +special requests, such as asking that all changes be marked or described. + +Although not required by our licenses, you are encouraged to respect those +requests where reasonable. More considerations for the public : +wiki.creativecommons.org/Considerations_for_licensees + +Creative Commons Attribution-ShareAlike 4.0 International Public License + +By exercising the Licensed Rights (defined below), You accept and agree to +be bound by the terms and conditions of this Creative Commons +Attribution-ShareAlike 4.0 International Public License ("Public +License"). To the extent this Public License may be interpreted as a +contract, You are granted the Licensed Rights in consideration of Your +acceptance of these terms and conditions, and the Licensor grants You such +rights in consideration of benefits the Licensor receives from making the +Licensed Material available under these terms and conditions. + +Section 1 - Definitions. + + a. Adapted Material means material subject to Copyright and Similar + Rights that is derived from or based upon the Licensed Material and + in which the Licensed Material is translated, altered, arranged, + transformed, or otherwise modified in a manner requiring permission + under the Copyright and Similar Rights held by the Licensor. For + purposes of this Public License, where the Licensed Material is a + musical work, performance, or sound recording, Adapted Material is + always produced where the Licensed Material is synched in timed + relation with a moving image. + + b. Adapter's License means the license You apply to Your Copyright and + Similar Rights in Your contributions to Adapted Material in + accordance with the terms and conditions of this Public License. + + c. BY-SA Compatible License means a license listed at + creativecommons.org/compatiblelicenses, approved by Creative Commons + as essentially the equivalent of this Public License. + + d. Copyright and Similar Rights means copyright and/or similar rights + closely related to copyright including, without limitation, + performance, broadcast, sound recording, and Sui Generis Database + Rights, without regard to how the rights are labeled or + categorized. For purposes of this Public License, the rights + specified in Section 2(b)(1)-(2) are not Copyright and Similar + Rights. + + e. Effective Technological Measures means those measures that, in the + absence of proper authority, may not be circumvented under laws + fulfilling obligations under Article 11 of the WIPO Copyright Treaty + adopted on December 20, 1996, and/or similar international + agreements. + + f. Exceptions and Limitations means fair use, fair dealing, and/or any + other exception or limitation to Copyright and Similar Rights that + applies to Your use of the Licensed Material. + + g. License Elements means the license attributes listed in the name of + a Creative Commons Public License. The License Elements of this + Public License are Attribution and ShareAlike. + + h. Licensed Material means the artistic or literary work, database, or + other material to which the Licensor applied this Public License. + + i. Licensed Rights means the rights granted to You subject to the terms + and conditions of this Public License, which are limited to all + Copyright and Similar Rights that apply to Your use of the Licensed + Material and that the Licensor has authority to license. + + j. Licensor means the individual(s) or entity(ies) granting rights + under this Public License. + + k. Share means to provide material to the public by any means or + process that requires permission under the Licensed Rights, such as + reproduction, public display, public performance, distribution, + dissemination, communication, or importation, and to make material + available to the public including in ways that members of the public + may access the material from a place and at a time individually + chosen by them. + + l. Sui Generis Database Rights means rights other than copyright + resulting from Directive 96/9/EC of the European Parliament and of + the Council of 11 March 1996 on the legal protection of databases, + as amended and/or succeeded, as well as other essentially equivalent + rights anywhere in the world. m. You means the individual or entity + exercising the Licensed Rights under this Public License. Your has a + corresponding meaning. + +Section 2 - Scope. + + a. License grant. + + 1. Subject to the terms and conditions of this Public License, the + Licensor hereby grants You a worldwide, royalty-free, + non-sublicensable, non-exclusive, irrevocable license to + exercise the Licensed Rights in the Licensed Material to: + + A. reproduce and Share the Licensed Material, in whole or in part; and + + B. produce, reproduce, and Share Adapted Material. + + 2. Exceptions and Limitations. For the avoidance of doubt, where + Exceptions and Limitations apply to Your use, this Public + License does not apply, and You do not need to comply with its + terms and conditions. + + 3. Term. The term of this Public License is specified in Section 6(a). + + 4. Media and formats; technical modifications allowed. The Licensor + authorizes You to exercise the Licensed Rights in all media and + formats whether now known or hereafter created, and to make + technical modifications necessary to do so. The Licensor waives + and/or agrees not to assert any right or authority to forbid You + from making technical modifications necessary to exercise the + Licensed Rights, including technical modifications necessary to + circumvent Effective Technological Measures. For purposes of + this Public License, simply making modifications authorized by + this Section 2(a)(4) never produces Adapted Material. + + 5. Downstream recipients. + + A. Offer from the Licensor - Licensed Material. Every recipient + of the Licensed Material automatically receives an offer + from the Licensor to exercise the Licensed Rights under the + terms and conditions of this Public License. + + B. Additional offer from the Licensor - Adapted Material. Every + recipient of Adapted Material from You automatically + receives an offer from the Licensor to exercise the Licensed + Rights in the Adapted Material under the conditions of the + Adapter's License You apply. + + C. No downstream restrictions. You may not offer or impose any + additional or different terms or conditions on, or apply any + Effective Technological Measures to, the Licensed Material + if doing so restricts exercise of the Licensed Rights by any + recipient of the Licensed Material. + + 6. No endorsement. Nothing in this Public License constitutes or + may be construed as permission to assert or imply that You are, + or that Your use of the Licensed Material is, connected with, or + sponsored, endorsed, or granted official status by, the Licensor + or others designated to receive attribution as provided in + Section 3(a)(1)(A)(i). + + b. Other rights. + + 1. Moral rights, such as the right of integrity, are not licensed + under this Public License, nor are publicity, privacy, and/or + other similar personality rights; however, to the extent + possible, the Licensor waives and/or agrees not to assert any + such rights held by the Licensor to the limited extent necessary + to allow You to exercise the Licensed Rights, but not otherwise. + + 2. Patent and trademark rights are not licensed under this Public + License. + + 3. To the extent possible, the Licensor waives any right to collect + royalties from You for the exercise of the Licensed Rights, + whether directly or through a collecting society under any + voluntary or waivable statutory or compulsory licensing + scheme. In all other cases the Licensor expressly reserves any + right to collect such royalties. + +Section 3 - License Conditions. + +Your exercise of the Licensed Rights is expressly made subject to the +following conditions. + + a. Attribution. + + 1. If You Share the Licensed Material (including in modified form), + You must: + + A. retain the following if it is supplied by the Licensor with + the Licensed Material: + + i. identification of the creator(s) of the Licensed + Material and any others designated to receive + attribution, in any reasonable manner requested by the + Licensor (including by pseudonym if designated); + + ii. a copyright notice; + + iii. a notice that refers to this Public License; + + iv. a notice that refers to the disclaimer of warranties; + + v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable; + + B. indicate if You modified the Licensed Material and retain an + indication of any previous modifications; and + + C. indicate the Licensed Material is licensed under this Public + License, and include the text of, or the URI or hyperlink to, + this Public License. + + 2. You may satisfy the conditions in Section 3(a)(1) in any + reasonable manner based on the medium, means, and context in + which You Share the Licensed Material. For example, it may be + reasonable to satisfy the conditions by providing a URI or + hyperlink to a resource that includes the required information. + + 3. If requested by the Licensor, You must remove any of the + information required by Section 3(a)(1)(A) to the extent + reasonably practicable. b. ShareAlike.In addition to the + conditions in Section 3(a), if You Share Adapted Material You + produce, the following conditions also apply. + + 1. The Adapter's License You apply must be a Creative Commons + license with the same License Elements, this version or + later, or a BY-SA Compatible License. + + 2. You must include the text of, or the URI or hyperlink to, the + Adapter's License You apply. You may satisfy this condition + in any reasonable manner based on the medium, means, and + context in which You Share Adapted Material. + + 3. You may not offer or impose any additional or different terms + or conditions on, or apply any Effective Technological + Measures to, Adapted Material that restrict exercise of the + rights granted under the Adapter's License You apply. + +Section 4 - Sui Generis Database Rights. + +Where the Licensed Rights include Sui Generis Database Rights that apply to +Your use of the Licensed Material: + + a. for the avoidance of doubt, Section 2(a)(1) grants You the right to + extract, reuse, reproduce, and Share all or a substantial portion of + the contents of the database; + + b. if You include all or a substantial portion of the database contents + in a database in which You have Sui Generis Database Rights, then + the database in which You have Sui Generis Database Rights (but not + its individual contents) is Adapted Material, including for purposes + of Section 3(b); and + + c. You must comply with the conditions in Section 3(a) if You Share all + or a substantial portion of the contents of the database. + + For the avoidance of doubt, this Section 4 supplements and does not + replace Your obligations under this Public License where the Licensed + Rights include other Copyright and Similar Rights. + +Section 5 - Disclaimer of Warranties and Limitation of Liability. + + a. Unless otherwise separately undertaken by the Licensor, to the + extent possible, the Licensor offers the Licensed Material as-is and + as-available, and makes no representations or warranties of any kind + concerning the Licensed Material, whether express, implied, + statutory, or other. This includes, without limitation, warranties + of title, merchantability, fitness for a particular purpose, + non-infringement, absence of latent or other defects, accuracy, or + the presence or absence of errors, whether or not known or + discoverable. Where disclaimers of warranties are not allowed in + full or in part, this disclaimer may not apply to You. + + b. To the extent possible, in no event will the Licensor be liable to + You on any legal theory (including, without limitation, negligence) + or otherwise for any direct, special, indirect, incidental, + consequential, punitive, exemplary, or other losses, costs, + expenses, or damages arising out of this Public License or use of + the Licensed Material, even if the Licensor has been advised of the + possibility of such losses, costs, expenses, or damages. Where a + limitation of liability is not allowed in full or in part, this + limitation may not apply to You. + + c. The disclaimer of warranties and limitation of liability provided + above shall be interpreted in a manner that, to the extent possible, + most closely approximates an absolute disclaimer and waiver of all + liability. + +Section 6 - Term and Termination. + + a. This Public License applies for the term of the Copyright and + Similar Rights licensed here. However, if You fail to comply with + this Public License, then Your rights under this Public License + terminate automatically. + + b. Where Your right to use the Licensed Material has terminated under + Section 6(a), it reinstates: + + 1. automatically as of the date the violation is cured, provided it + is cured within 30 days of Your discovery of the violation; or + + 2. upon express reinstatement by the Licensor. + + c. For the avoidance of doubt, this Section 6(b) does not affect any + right the Licensor may have to seek remedies for Your violations of + this Public License. + + d. For the avoidance of doubt, the Licensor may also offer the Licensed + Material under separate terms or conditions or stop distributing the + Licensed Material at any time; however, doing so will not terminate + this Public License. + + e. Sections 1, 5, 6, 7, and 8 survive termination of this Public License. + +Section 7 - Other Terms and Conditions. + + a. The Licensor shall not be bound by any additional or different terms + or conditions communicated by You unless expressly agreed. + + b. Any arrangements, understandings, or agreements regarding the + Licensed Material not stated herein are separate from and + independent of the terms and conditions of this Public License. + +Section 8 - Interpretation. + + a. For the avoidance of doubt, this Public License does not, and shall + not be interpreted to, reduce, limit, restrict, or impose conditions + on any use of the Licensed Material that could lawfully be made + without permission under this Public License. + + b. To the extent possible, if any provision of this Public License is + deemed unenforceable, it shall be automatically reformed to the + minimum extent necessary to make it enforceable. If the provision + cannot be reformed, it shall be severed from this Public License + without affecting the enforceability of the remaining terms and + conditions. + + c. No term or condition of this Public License will be waived and no + failure to comply consented to unless expressly agreed to by the + Licensor. + + d. Nothing in this Public License constitutes or may be interpreted as + a limitation upon, or waiver of, any privileges and immunities that + apply to the Licensor or You, including from the legal processes of + any jurisdiction or authority. + +Creative Commons is not a party to its public licenses. Notwithstanding, +Creative Commons may elect to apply one of its public licenses to material +it publishes and in those instances will be considered the "Licensor." The +text of the Creative Commons public licenses is dedicated to the public +domain under the CC0 Public Domain Dedication. Except for the limited +purpose of indicating that material is shared under a Creative Commons +public license or as otherwise permitted by the Creative Commons policies +published at creativecommons.org/policies, Creative Commons does not +authorize the use of the trademark "Creative Commons" or any other +trademark or logo of Creative Commons without its prior written consent +including, without limitation, in connection with any unauthorized +modifications to any of its public licenses or any other arrangements, +understandings, or agreements concerning use of licensed material. For the +avoidance of doubt, this paragraph does not form part of the public +licenses. + +Creative Commons may be contacted at creativecommons.org. diff --git a/LICENSES/other/CDDL-1.0 b/LICENSES/other/CDDL-1.0 new file mode 100644 index 000000000000..195a1687930a --- /dev/null +++ b/LICENSES/other/CDDL-1.0 @@ -0,0 +1,364 @@ +Valid-License-Identifier: CDDL-1.0 +SPDX-URL: https://spdx.org/licenses/CDDL-1.0.html +Usage-Guide: + To use the Common Development and Distribution License 1.0 put the + following SPDX tag/value pair into a comment according to the placement + guidelines in the licensing rules documentation: + SPDX-License-Identifier: CDDL-1.0 + +License-Text: + +COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) +Version 1.0 + + 1. Definitions. + + 1.1. "Contributor" means each individual or entity that creates or + contributes to the creation of Modifications. + + 1.2. "Contributor Version" means the combination of the Original + Software, prior Modifications used by a Contributor (if any), + and the Modifications made by that particular Contributor. + + 1.3. "Covered Software" means (a) the Original Software, or (b) + Modifications, or (c) the combination of files containing + Original Software with files containing Modifications, in each + case including portions thereof. + + 1.4. "Executable" means the Covered Software in any form other than + Source Code. + + 1.5. "Initial Developer" means the individual or entity that first + makes Original Software available under this License. + + 1.6. "Larger Work" means a work which combines Covered Software or + portions thereof with code not governed by the terms of this + License. + + 1.7. "License" means this document. + + 1.8. "Licensable" means having the right to grant, to the maximum + extent possible, whether at the time of the initial grant or + subsequently acquired, any and all of the rights conveyed herein. + + 1.9. "Modifications" means the Source Code and Executable form of + any of the following: + + A. Any file that results from an addition to, deletion from or + modification of the contents of a file containing Original + Software or previous Modifications; + + B. Any new file that contains any part of the Original Software + or previous Modification; or + + C. Any new file that is contributed or otherwise made available + under the terms of this License. + + 1.10. "Original Software" means the Source Code and Executable form + of computer software code that is originally released under + this License. + + 1.11. "Patent Claims" means any patent claim(s), now owned or + hereafter acquired, including without limitation, method, + process, and apparatus claims, in any patent Licensable by + grantor. + + 1.12. "Source Code" means (a) the common form of computer software + code in which modifications are made and (b) associated + documentation included in or with such code. + + 1.13. "You" (or "Your") means an individual or a legal entity + exercising rights under, and complying with all of the terms + of, this License. For legal entities, "You" includes any + entity which controls, is controlled by, or is under common + control with You. For purposes of this definition, "control" + means (a) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract + or otherwise, or (b) ownership of more than fifty percent + (50%) of the outstanding shares or beneficial ownership of + such entity. + + 2. License Grants. + 2.1. The Initial Developer Grant. + + Conditioned upon Your compliance with Section 3.1 below and subject + to third party intellectual property claims, the Initial Developer + hereby grants You a world-wide, royalty-free, non-exclusive + license: + + (a) under intellectual property rights (other than patent or + trademark) Licensable by Initial Developer, to use, + reproduce, modify, display, perform, sublicense and + distribute the Original Software (or portions thereof), + with or without Modifications, and/or as part of a Larger + Work; and + + (b) under Patent Claims infringed by the making, using or + selling of Original Software, to make, have made, use, + practice, sell, and offer for sale, and/or otherwise + dispose of the Original Software (or portions thereof). + + (c) The licenses granted in Sections 2.1(a) and (b) are + effective on the date Initial Developer first distributes + or otherwise makes the Original Software available to a + third party under the terms of this License. + + (d) Notwithstanding Section 2.1(b) above, no patent license is + granted: (1) for code that You delete from the Original + Software, or (2) for infringements caused by: (i) the + modification of the Original Software, or (ii) the + combination of the Original Software with other software or + devices. + + 2.2. Contributor Grant. + + Conditioned upon Your compliance with Section 3.1 below and subject + to third party intellectual property claims, each Contributor + hereby grants You a world-wide, royalty-free, non-exclusive + license: + + (a) under intellectual property rights (other than patent or + trademark) Licensable by Contributor to use, reproduce, + modify, display, perform, sublicense and distribute the + Modifications created by such Contributor (or portions + thereof), either on an unmodified basis, with other + Modifications, as Covered Software and/or as part of a + Larger Work; and + + (b) under Patent Claims infringed by the making, using, or + selling of Modifications made by that Contributor either + alone and/or in combination with its Contributor Version + (or portions of such combination), to make, use, sell, + offer for sale, have made, and/or otherwise dispose of: (1) + Modifications made by that Contributor (or portions + thereof); and (2) the combination of Modifications made by + that Contributor with its Contributor Version (or portions + of such combination). + + (c) The licenses granted in Sections 2.2(a) and 2.2(b) are + effective on the date Contributor first distributes or + otherwise makes the Modifications available to a third + party. + + (d) Notwithstanding Section 2.2(b) above, no patent license is + granted: (1) for any code that Contributor has deleted from + the Contributor Version; (2) for infringements caused by: + (i) third party modifications of Contributor Version, or + (ii) the combination of Modifications made by that + Contributor with other software (except as part of the + Contributor Version) or other devices; or (3) under Patent + Claims infringed by Covered Software in the absence of + Modifications made by that Contributor. + + 3. Distribution Obligations. + 3.1. Availability of Source Code. + + Any Covered Software that You distribute or otherwise make + available in Executable form must also be made available in Source + Code form and that Source Code form must be distributed only under + the terms of this License. You must include a copy of this License + with every copy of the Source Code form of the Covered Software You + distribute or otherwise make available. You must inform recipients + of any such Covered Software in Executable form as to how they can + obtain such Covered Software in Source Code form in a reasonable + manner on or through a medium customarily used for software + exchange. + + 3.2. Modifications. + + The Modifications that You create or to which You contribute are + governed by the terms of this License. You represent that You + believe Your Modifications are Your original creation(s) and/or You + have sufficient rights to grant the rights conveyed by this + License. + + 3.3. Required Notices. + + You must include a notice in each of Your Modifications that + identifies You as the Contributor of the Modification. You may not + remove or alter any copyright, patent or trademark notices + contained within the Covered Software, or any notices of licensing + or any descriptive text giving attribution to any Contributor or + the Initial Developer. + + 3.4. Application of Additional Terms. + + You may not offer or impose any terms on any Covered Software in + Source Code form that alters or restricts the applicable version of + this License or the recipients' rights hereunder. You may choose to + offer, and to charge a fee for, warranty, support, indemnity or + liability obligations to one or more recipients of Covered + Software. However, you may do so only on Your own behalf, and not + on behalf of the Initial Developer or any Contributor. You must + make it absolutely clear that any such warranty, support, indemnity + or liability obligation is offered by You alone, and You hereby + agree to indemnify the Initial Developer and every Contributor for + any liability incurred by the Initial Developer or such Contributor + as a result of warranty, support, indemnity or liability terms You + offer. + + 3.5. Distribution of Executable Versions. + + You may distribute the Executable form of the Covered Software + under the terms of this License or under the terms of a license of + Your choice, which may contain terms different from this License, + provided that You are in compliance with the terms of this License + and that the license for the Executable form does not attempt to + limit or alter the recipient's rights in the Source Code form from + the rights set forth in this License. If You distribute the Covered + Software in Executable form under a different license, You must + make it absolutely clear that any terms which differ from this + License are offered by You alone, not by the Initial Developer or + Contributor. You hereby agree to indemnify the Initial Developer + and every Contributor for any liability incurred by the Initial + Developer or such Contributor as a result of any such terms You + offer. + + 3.6. Larger Works. + + You may create a Larger Work by combining Covered Software with + other code not governed by the terms of this License and distribute + the Larger Work as a single product. In such a case, You must make + sure the requirements of this License are fulfilled for the Covered + Software. + + 4. Versions of the License. + 4.1. New Versions. + + Sun Microsystems, Inc. is the initial license steward and may + publish revised and/or new versions of this License from time to + time. Each version will be given a distinguishing version + number. Except as provided in Section 4.3, no one other than the + license steward has the right to modify this License. + + 4.2. Effect of New Versions. + + You may always continue to use, distribute or otherwise make the + Covered Software available under the terms of the version of the + License under which You originally received the Covered + Software. If the Initial Developer includes a notice in the + Original Software prohibiting it from being distributed or + otherwise made available under any subsequent version of the + License, You must distribute and make the Covered Software + available under the terms of the version of the License under which + You originally received the Covered Software. Otherwise, You may + also choose to use, distribute or otherwise make the Covered + Software available under the terms of any subsequent version of the + License published by the license steward. + + 4.3. Modified Versions. + + When You are an Initial Developer and You want to create a new + license for Your Original Software, You may create and use a + modified version of this License if You: (a) rename the license and + remove any references to the name of the license steward (except to + note that the license differs from this License); and (b) otherwise + make it clear that the license contains terms which differ from + this License. + + 5. DISCLAIMER OF WARRANTY. + + COVERED SOFTWARE IS PROVIDED UNDER THIS LICENSE ON AN "AS IS" BASIS, + WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, + WITHOUT LIMITATION, WARRANTIES THAT THE COVERED SOFTWARE IS FREE OF + DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR PURPOSE OR + NON-INFRINGING. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF + THE COVERED SOFTWARE IS WITH YOU. SHOULD ANY COVERED SOFTWARE PROVE + DEFECTIVE IN ANY RESPECT, YOU (NOT THE INITIAL DEVELOPER OR ANY OTHER + CONTRIBUTOR) ASSUME THE COST OF ANY NECESSARY SERVICING, REPAIR OR + CORRECTION. THIS DISCLAIMER OF WARRANTY CONSTITUTES AN ESSENTIAL PART + OF THIS LICENSE. NO USE OF ANY COVERED SOFTWARE IS AUTHORIZED HEREUNDER + EXCEPT UNDER THIS DISCLAIMER. + + 6. TERMINATION. + + 6.1. This License and the rights granted hereunder will terminate + automatically if You fail to comply with terms herein and fail to + cure such breach within 30 days of becoming aware of the + breach. Provisions which, by their nature, must remain in effect + beyond the termination of this License shall survive. + + 6.2. If You assert a patent infringement claim (excluding + declaratory judgment actions) against Initial Developer or a + Contributor (the Initial Developer or Contributor against whom You + assert such claim is referred to as "Participant") alleging that + the Participant Software (meaning the Contributor Version where the + Participant is a Contributor or the Original Software where the + Participant is the Initial Developer) directly or indirectly + infringes any patent, then any and all rights granted directly or + indirectly to You by such Participant, the Initial Developer (if + the Initial Developer is not the Participant) and all Contributors + under Sections 2.1 and/or 2.2 of this License shall, upon 60 days + notice from Participant terminate prospectively and automatically + at the expiration of such 60 day notice period, unless if within + such 60 day period You withdraw Your claim with respect to the + Participant Software against such Participant either unilaterally + or pursuant to a written agreement with Participant. + + 6.3. In the event of termination under Sections 6.1 or 6.2 above, + all end user licenses that have been validly granted by You or any + distributor hereunder prior to termination (excluding licenses + granted to You by any distributor) shall survive termination. + + 7. LIMITATION OF LIABILITY. + + UNDER NO CIRCUMSTANCES AND UNDER NO LEGAL THEORY, WHETHER TORT + (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE, SHALL YOU, THE INITIAL + DEVELOPER, ANY OTHER CONTRIBUTOR, OR ANY DISTRIBUTOR OF COVERED + SOFTWARE, OR ANY SUPPLIER OF ANY OF SUCH PARTIES, BE LIABLE TO ANY + PERSON FOR ANY INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES + OF ANY CHARACTER INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOST + PROFITS, LOSS OF GOODWILL, WORK STOPPAGE, COMPUTER FAILURE OR + MALFUNCTION, OR ANY AND ALL OTHER COMMERCIAL DAMAGES OR LOSSES, EVEN IF + SUCH PARTY SHALL HAVE BEEN INFORMED OF THE POSSIBILITY OF SUCH + DAMAGES. THIS LIMITATION OF LIABILITY SHALL NOT APPLY TO LIABILITY FOR + DEATH OR PERSONAL INJURY RESULTING FROM SUCH PARTY'S NEGLIGENCE TO THE + EXTENT APPLICABLE LAW PROHIBITS SUCH LIMITATION. SOME JURISDICTIONS DO + NOT ALLOW THE EXCLUSION OR LIMITATION OF INCIDENTAL OR CONSEQUENTIAL + DAMAGES, SO THIS EXCLUSION AND LIMITATION MAY NOT APPLY TO YOU. + + 8. U.S. GOVERNMENT END USERS. + + The Covered Software is a "commercial item," as that term is defined in + 48 C.F.R. 2.101 (Oct. 1995), consisting of "commercial computer + software" (as that term is defined at 48 C.F.R. $ 252.227-7014(a)(1)) + and "commercial computer software documentation" as such terms are used + in 48 C.F.R. 12.212 (Sept. 1995). Consistent with 48 C.F.R. 12.212 and + 48 C.F.R. 227.7202-1 through 227.7202-4 (June 1995), all + U.S. Government End Users acquire Covered Software with only those + rights set forth herein. This U.S. Government Rights clause is in lieu + of, and supersedes, any other FAR, DFAR, or other clause or provision + that addresses Government rights in computer software under this + License. + + 9. MISCELLANEOUS. + + This License represents the complete agreement concerning subject + matter hereof. If any provision of this License is held to be + unenforceable, such provision shall be reformed only to the extent + necessary to make it enforceable. This License shall be governed by the + law of the jurisdiction specified in a notice contained within the + Original Software (except to the extent applicable law, if any, + provides otherwise), excluding such jurisdiction's conflict-of-law + provisions. Any litigation relating to this License shall be subject to + the jurisdiction of the courts located in the jurisdiction and venue + specified in a notice contained within the Original Software, with the + losing party responsible for costs, including, without limitation, + court costs and reasonable attorneys' fees and expenses. The + application of the United Nations Convention on Contracts for the + International Sale of Goods is expressly excluded. Any law or + regulation which provides that the language of a contract shall be + construed against the drafter shall not apply to this License. You + agree that You alone are responsible for compliance with the United + States export administration regulations (and the export control laws + and regulation of any other countries) when You use, distribute or + otherwise make available any Covered Software. + + 10. RESPONSIBILITY FOR CLAIMS. + + As between Initial Developer and the Contributors, each party is + responsible for claims and damages arising, directly or indirectly, out + of its utilization of rights under this License and You agree to work + with Initial Developer and Contributors to distribute such + responsibility on an equitable basis. Nothing herein is intended or + shall be deemed to constitute any admission of liability. diff --git a/LICENSES/other/Linux-OpenIB b/LICENSES/other/Linux-OpenIB new file mode 100644 index 000000000000..1ad85f6b3a89 --- /dev/null +++ b/LICENSES/other/Linux-OpenIB @@ -0,0 +1,26 @@ +Valid-License-Identifier: Linux-OpenIB +SPDX-URL: https://spdx.org/licenses/Linux-OpenIB.html +Usage-Guide: + To use the Linux Kernel Variant of OpenIB.org license put the following + SPDX tag/value pair into a comment according to the placement guidelines + in the licensing rules documentation: + SPDX-License-Identifier: Linux-OpenIB +License-Text: + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + - Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + - Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. diff --git a/LICENSES/other/X11 b/LICENSES/other/X11 new file mode 100644 index 000000000000..fe4353fd0000 --- /dev/null +++ b/LICENSES/other/X11 @@ -0,0 +1,37 @@ +Valid-License-Identifier: X11 +SPDX-URL: https://spdx.org/licenses/X11.html +Usage-Guide: + To use the X11 put the following SPDX tag/value pair into a comment + according to the placement guidelines in the licensing rules + documentation: + SPDX-License-Identifier: X11 +License-Text: + + +X11 License + +Copyright (C) 1996 X Consortium + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +X CONSORTIUM BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Except as contained in this notice, the name of the X Consortium shall not +be used in advertising or otherwise to promote the sale, use or other +dealings in this Software without prior written authorization from the X +Consortium. + +X Window System is a trademark of X Consortium, Inc. diff --git a/LICENSES/preferred/GPL-2.0 b/LICENSES/preferred/GPL-2.0 index b8db91d3a1cb..ff0812fd89cc 100644 --- a/LICENSES/preferred/GPL-2.0 +++ b/LICENSES/preferred/GPL-2.0 @@ -1,5 +1,7 @@ Valid-License-Identifier: GPL-2.0 +Valid-License-Identifier: GPL-2.0-only Valid-License-Identifier: GPL-2.0+ +Valid-License-Identifier: GPL-2.0-or-later SPDX-URL: https://spdx.org/licenses/GPL-2.0.html Usage-Guide: To use this license in source code, put one of the following SPDX @@ -7,8 +9,12 @@ Usage-Guide: guidelines in the licensing rules documentation. For 'GNU General Public License (GPL) version 2 only' use: SPDX-License-Identifier: GPL-2.0 + or + SPDX-License-Identifier: GPL-2.0-only For 'GNU General Public License (GPL) version 2 or any later version' use: SPDX-License-Identifier: GPL-2.0+ + or + SPDX-License-Identifier: GPL-2.0-or-later License-Text: GNU GENERAL PUBLIC LICENSE diff --git a/MAINTAINERS b/MAINTAINERS index ca4afd68530c..edf3cf5ea691 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -767,12 +767,14 @@ F: drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c F: drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h F: drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v7.c F: drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v8.c +F: drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c F: drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c F: drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c F: drivers/gpu/drm/amd/amdkfd/ F: drivers/gpu/drm/amd/include/cik_structs.h F: drivers/gpu/drm/amd/include/kgd_kfd_interface.h F: drivers/gpu/drm/amd/include/vi_structs.h +F: drivers/gpu/drm/amd/include/v9_structs.h F: include/uapi/linux/kfd_ioctl.h AMD SEATTLE DEVICE TREE SUPPORT @@ -794,6 +796,14 @@ M: Michael Hanselmann S: Supported F: drivers/macintosh/ams/ +ANALOG DEVICES INC AD5686 DRIVER +M: Stefan Popa +L: linux-pm@vger.kernel.org +W: http://ez.analog.com/community/linux-device-drivers +S: Supported +F: drivers/iio/dac/ad5686* +F: drivers/iio/dac/ad5696* + ANALOG DEVICES INC AD9389B DRIVER M: Hans Verkuil L: linux-media@vger.kernel.org @@ -1133,10 +1143,12 @@ F: arch/arm/mach-actions/ F: arch/arm/boot/dts/owl-* F: arch/arm64/boot/dts/actions/ F: drivers/clocksource/owl-* +F: drivers/pinctrl/actions/* F: drivers/soc/actions/ F: include/dt-bindings/power/owl-* F: include/linux/soc/actions/ F: Documentation/devicetree/bindings/arm/actions.txt +F: Documentation/devicetree/bindings/pinctrl/actions,s900-pinctrl.txt F: Documentation/devicetree/bindings/power/actions,owl-sps.txt F: Documentation/devicetree/bindings/timer/actions,owl-timer.txt @@ -1407,6 +1419,7 @@ M: Shawn Guo M: Sascha Hauer R: Pengutronix Kernel Team R: Fabio Estevam +R: NXP Linux Team L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/shawnguo/linux.git @@ -1719,7 +1732,8 @@ F: arch/arm/mach-npcm/ F: arch/arm/boot/dts/nuvoton-npcm* F: include/dt-bindings/clock/nuvoton,npcm7xx-clks.h F: drivers/*/*npcm* -F: Documentation/*/*npcm* +F: Documentation/devicetree/bindings/*/*npcm* +F: Documentation/devicetree/bindings/*/*/*npcm* ARM/NUVOTON W90X900 ARM ARCHITECTURE M: Wan ZongShun @@ -1817,7 +1831,7 @@ F: drivers/spi/spi-qup.c F: drivers/tty/serial/msm_serial.c F: drivers/*/pm8???-* F: drivers/mfd/ssbi.c -F: drivers/firmware/qcom_scm.c +F: drivers/firmware/qcom_scm* T: git git://git.kernel.org/pub/scm/linux/kernel/git/agross/linux.git ARM/RADISYS ENP2611 MACHINE SUPPORT @@ -2331,6 +2345,14 @@ S: Maintained F: drivers/gpio/gpio-ath79.c F: Documentation/devicetree/bindings/gpio/gpio-ath79.txt +ATHEROS 71XX/9XXX USB PHY DRIVER +M: Alban Bedel +W: https://github.com/AlbanBedel/linux +T: git git://github.com/AlbanBedel/linux +S: Maintained +F: drivers/phy/qualcomm/phy-ath79-usb.c +F: Documentation/devicetree/bindings/phy/phy-ath79-usb.txt + ATHEROS ATH GENERIC UTILITIES M: Kalle Valo L: linux-wireless@vger.kernel.org @@ -2521,8 +2543,6 @@ F: kernel/audit* AUXILIARY DISPLAY DRIVERS M: Miguel Ojeda Sandonis -W: http://miguelojeda.es/auxdisplay.htm -W: http://jair.lab.fi.uva.es/~migojed/auxdisplay.htm S: Maintained F: drivers/auxdisplay/ F: include/linux/cfag12864b.h @@ -2588,6 +2608,7 @@ BACKLIGHT CLASS/SUBSYSTEM M: Lee Jones M: Daniel Thompson M: Jingoo Han +L: dri-devel@lists.freedesktop.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/lee/backlight.git S: Maintained F: drivers/video/backlight/ @@ -2722,13 +2743,13 @@ L: netdev@vger.kernel.org L: linux-kernel@vger.kernel.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git T: git git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git +Q: https://patchwork.ozlabs.org/project/netdev/list/?delegate=77147 S: Supported F: arch/x86/net/bpf_jit* F: Documentation/networking/filter.txt F: Documentation/bpf/ F: include/linux/bpf* F: include/linux/filter.h -F: include/trace/events/bpf.h F: include/trace/events/xdp.h F: include/uapi/linux/bpf* F: include/uapi/linux/filter.h @@ -2741,6 +2762,7 @@ F: net/sched/act_bpf.c F: net/sched/cls_bpf.c F: samples/bpf/ F: tools/bpf/ +F: tools/lib/bpf/ F: tools/testing/selftests/bpf/ BROADCOM B44 10/100 ETHERNET DRIVER @@ -3058,7 +3080,7 @@ M: Clemens Ladisch L: alsa-devel@alsa-project.org (moderated for non-subscribers) T: git git://git.alsa-project.org/alsa-kernel.git S: Maintained -F: Documentation/sound/alsa/Bt87x.txt +F: Documentation/sound/cards/bt87x.rst F: sound/pci/bt87x.c BT8XXGPIO DRIVER @@ -3136,6 +3158,13 @@ S: Supported F: Documentation/filesystems/caching/cachefiles.txt F: fs/cachefiles/ +CADENCE MIPI-CSI2 BRIDGES +M: Maxime Ripard +L: linux-media@vger.kernel.org +S: Maintained +F: Documentation/devicetree/bindings/media/cdns,*.txt +F: drivers/media/platform/cadence/cdns-csi2* + CADET FM/AM RADIO RECEIVER DRIVER M: Hans Verkuil L: linux-media@vger.kernel.org @@ -3347,7 +3376,7 @@ M: David Howells M: David Woodhouse L: keyrings@vger.kernel.org S: Maintained -F: Documentation/module-signing.txt +F: Documentation/admin-guide/module-signing.rst F: certs/ F: scripts/sign-file.c F: scripts/extract-cert.c @@ -3364,16 +3393,12 @@ F: include/linux/usb/wusb* CFAG12864B LCD DRIVER M: Miguel Ojeda Sandonis -W: http://miguelojeda.es/auxdisplay.htm -W: http://jair.lab.fi.uva.es/~migojed/auxdisplay.htm S: Maintained F: drivers/auxdisplay/cfag12864b.c F: include/linux/cfag12864b.h CFAG12864BFB LCD FRAMEBUFFER DRIVER M: Miguel Ojeda Sandonis -W: http://miguelojeda.es/auxdisplay.htm -W: http://jair.lab.fi.uva.es/~migojed/auxdisplay.htm S: Maintained F: drivers/auxdisplay/cfag12864bfb.c F: include/linux/cfag12864b.h @@ -3431,6 +3456,12 @@ S: Maintained F: Documentation/devicetree/bindings/input/touchscreen/chipone_icn8318.txt F: drivers/input/touchscreen/chipone_icn8318.c +CHIPONE ICN8505 I2C TOUCHSCREEN DRIVER +M: Hans de Goede +L: linux-input@vger.kernel.org +S: Maintained +F: drivers/input/touchscreen/chipone_icn8505.c + CHROME HARDWARE PLATFORM SUPPORT M: Benson Leung M: Olof Johansson @@ -3559,6 +3590,7 @@ F: drivers/clk/ X: drivers/clk/clkdev.c F: include/linux/clk-pr* F: include/linux/clk/ +F: include/linux/of_clk.h COMMON INTERNET FILE SYSTEM (CIFS) M: Steve French @@ -4328,14 +4360,11 @@ L: iommu@lists.linux-foundation.org T: git git://git.infradead.org/users/hch/dma-mapping.git W: http://git.infradead.org/users/hch/dma-mapping.git S: Supported -F: lib/dma-debug.c -F: lib/dma-direct.c -F: lib/dma-virt.c -F: drivers/base/dma-mapping.c -F: drivers/base/dma-coherent.c +F: kernel/dma/ F: include/asm-generic/dma-mapping.h F: include/linux/dma-direct.h F: include/linux/dma-mapping.h +F: include/linux/dma-noncoherent.h DME1737 HARDWARE MONITOR DRIVER M: Juerg Haefliger @@ -4399,6 +4428,12 @@ L: linux-kernel@vger.kernel.org S: Maintained F: drivers/staging/fsl-dpaa2/ethsw +DPAA2 PTP CLOCK DRIVER +M: Yangbo Lu +L: linux-kernel@vger.kernel.org +S: Maintained +F: drivers/staging/fsl-dpaa2/rtc + DPT_I2O SCSI RAID DRIVER M: Adaptec OEM Raid Solutions L: linux-scsi@vger.kernel.org @@ -4474,7 +4509,7 @@ DRM DRIVER FOR ILITEK ILI9225 PANELS M: David Lechner S: Maintained F: drivers/gpu/drm/tinydrm/ili9225.c -F: Documentation/devicetree/bindings/display/ili9225.txt +F: Documentation/devicetree/bindings/display/ilitek,ili9225.txt DRM DRIVER FOR INTEL I810 VIDEO CARDS S: Orphan / Obsolete @@ -4560,13 +4595,13 @@ DRM DRIVER FOR SITRONIX ST7586 PANELS M: David Lechner S: Maintained F: drivers/gpu/drm/tinydrm/st7586.c -F: Documentation/devicetree/bindings/display/st7586.txt +F: Documentation/devicetree/bindings/display/sitronix,st7586.txt DRM DRIVER FOR SITRONIX ST7735R PANELS M: David Lechner S: Maintained F: drivers/gpu/drm/tinydrm/st7735r.c -F: Documentation/devicetree/bindings/display/st7735r.txt +F: Documentation/devicetree/bindings/display/sitronix,st7735r.txt DRM DRIVER FOR TDFX VIDEO CARDS S: Orphan / Obsolete @@ -4599,7 +4634,6 @@ F: drivers/gpu/drm/ F: drivers/gpu/vga/ F: Documentation/devicetree/bindings/display/ F: Documentation/devicetree/bindings/gpu/ -F: Documentation/devicetree/bindings/video/ F: Documentation/gpu/ F: include/drm/ F: include/uapi/drm/ @@ -4644,7 +4678,7 @@ M: Boris Brezillon L: dri-devel@lists.freedesktop.org S: Supported F: drivers/gpu/drm/atmel-hlcdc/ -F: Documentation/devicetree/bindings/drm/atmel/ +F: Documentation/devicetree/bindings/display/atmel/ T: git git://anongit.freedesktop.org/drm/drm-misc DRM DRIVERS FOR BRIDGE CHIPS @@ -4669,13 +4703,13 @@ F: Documentation/devicetree/bindings/display/exynos/ DRM DRIVERS FOR FREESCALE DCU M: Stefan Agner -M: Alison Wang +M: Alison Wang L: dri-devel@lists.freedesktop.org S: Supported F: drivers/gpu/drm/fsl-dcu/ F: Documentation/devicetree/bindings/display/fsl,dcu.txt F: Documentation/devicetree/bindings/display/fsl,tcon.txt -F: Documentation/devicetree/bindings/display/panel/nec,nl4827hc19_05b.txt +F: Documentation/devicetree/bindings/display/panel/nec,nl4827hc19-05b.txt DRM DRIVERS FOR FREESCALE IMX M: Philipp Zabel @@ -4780,6 +4814,14 @@ S: Maintained F: drivers/gpu/drm/omapdrm/ F: Documentation/devicetree/bindings/display/ti/ +DRM DRIVERS FOR V3D +M: Eric Anholt +S: Supported +F: drivers/gpu/drm/v3d/ +F: include/uapi/drm/v3d_drm.h +F: Documentation/devicetree/bindings/gpu/brcm,bcm-v3d.txt +T: git git://anongit.freedesktop.org/drm/drm-misc + DRM DRIVERS FOR VC4 M: Eric Anholt T: git git://github.com/anholt/linux @@ -4826,6 +4868,15 @@ S: Maintained F: drivers/gpu/drm/tinydrm/ F: include/drm/tinydrm/ +DRM DRIVERS FOR XEN +M: Oleksandr Andrushchenko +T: git git://anongit.freedesktop.org/drm/drm-misc +L: dri-devel@lists.freedesktop.org +L: xen-devel@lists.xen.org +S: Supported +F: drivers/gpu/drm/xen/ +F: Documentation/gpu/xen-front.rst + DRM TTM SUBSYSTEM M: Christian Koenig M: Roger He @@ -5317,7 +5368,6 @@ F: include/linux/*mdio*.h F: include/linux/of_net.h F: include/linux/phy.h F: include/linux/phy_fixed.h -F: include/linux/platform_data/mdio-gpio.h F: include/linux/platform_data/mdio-bcm-unimac.h F: include/trace/events/mdio.h F: include/uapi/linux/mdio.h @@ -5413,6 +5463,19 @@ S: Maintained F: Documentation/hwmon/f71805f F: drivers/hwmon/f71805f.c +FADDR2LINE +M: Josh Poimboeuf +S: Maintained +F: scripts/faddr2line + +FAILOVER MODULE +M: Sridhar Samudrala +L: netdev@vger.kernel.org +S: Supported +F: net/core/failover.c +F: include/net/failover.h +F: Documentation/networking/failover.rst + FANOTIFY M: Jan Kara R: Amir Goldstein @@ -5565,6 +5628,7 @@ S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/atull/linux-fpga.git Q: http://patchwork.kernel.org/project/linux-fpga/list/ F: Documentation/fpga/ +F: Documentation/driver-api/fpga/ F: Documentation/devicetree/bindings/fpga/ F: drivers/fpga/ F: include/linux/fpga/ @@ -5622,7 +5686,6 @@ M: Claudiu Manoil L: netdev@vger.kernel.org S: Maintained F: drivers/net/ethernet/freescale/gianfar* -X: drivers/net/ethernet/freescale/gianfar_ptp.c F: Documentation/devicetree/bindings/net/fsl-tsec-phy.txt FREESCALE GPMI NAND DRIVER @@ -5667,7 +5730,15 @@ M: Madalin Bucur L: netdev@vger.kernel.org S: Maintained F: drivers/net/ethernet/freescale/fman -F: Documentation/devicetree/bindings/powerpc/fsl/fman.txt +F: Documentation/devicetree/bindings/net/fsl-fman.txt + +FREESCALE QORIQ PTP CLOCK DRIVER +M: Yangbo Lu +L: netdev@vger.kernel.org +S: Maintained +F: drivers/ptp/ptp_qoriq.c +F: include/linux/fsl/ptp_qoriq.h +F: Documentation/devicetree/bindings/ptp/ptp-qoriq.txt FREESCALE QUAD SPI DRIVER M: Han Xu @@ -5877,14 +5948,14 @@ GENERIC GPIO I2C DRIVER M: Haavard Skinnemoen S: Supported F: drivers/i2c/busses/i2c-gpio.c -F: include/linux/i2c-gpio.h +F: include/linux/platform_data/i2c-gpio.h GENERIC GPIO I2C MULTIPLEXER DRIVER M: Peter Korsgaard L: linux-i2c@vger.kernel.org S: Supported F: drivers/i2c/muxes/i2c-mux-gpio.c -F: include/linux/i2c-mux-gpio.h +F: include/linux/platform_data/i2c-mux-gpio.h F: Documentation/i2c/muxes/i2c-mux-gpio GENERIC HDLC (WAN) DRIVERS @@ -5948,8 +6019,8 @@ S: Maintained F: scripts/get_maintainer.pl GFS2 FILE SYSTEM -M: Steven Whitehouse M: Bob Peterson +M: Andreas Gruenbacher L: cluster-devel@redhat.com W: http://sources.redhat.com/cluster/ T: git git://git.kernel.org/pub/scm/linux/kernel/git/gfs2/linux-gfs2.git @@ -6217,6 +6288,7 @@ L: linux-hwmon@vger.kernel.org W: http://hwmon.wiki.kernel.org/ T: git git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging.git S: Maintained +F: Documentation/devicetree/bindings/hwmon/ F: Documentation/hwmon/ F: drivers/hwmon/ F: include/linux/hwmon*.h @@ -6424,7 +6496,7 @@ L: linux-mm@kvack.org S: Maintained F: mm/hmm* F: include/linux/hmm* -F: Documentation/vm/hmm.txt +F: Documentation/vm/hmm.rst HOST AP DRIVER M: Jouni Malinen @@ -6543,7 +6615,7 @@ F: arch/x86/hyperv F: drivers/hid/hid-hyperv.c F: drivers/hv/ F: drivers/input/serio/hyperv-keyboard.c -F: drivers/pci/host/pci-hyperv.c +F: drivers/pci/controller/pci-hyperv.c F: drivers/net/hyperv/ F: drivers/scsi/storvsc_drv.c F: drivers/uio/uio_hv_generic.c @@ -6751,6 +6823,12 @@ L: linux-scsi@vger.kernel.org S: Supported F: drivers/scsi/ibmvscsi/ibmvfc* +IBM Power Virtual Management Channel Driver +M: Bryant G. Ly +M: Steven Royer +S: Supported +F: drivers/misc/ibmvmc.* + IBM Power Virtual SCSI Device Drivers M: Tyrel Datwyler L: linux-scsi@vger.kernel.org @@ -6883,7 +6961,7 @@ IIO MULTIPLEXER M: Peter Rosin L: linux-iio@vger.kernel.org S: Maintained -F: Documentation/devicetree/bindings/iio/multiplexer/iio-mux.txt +F: Documentation/devicetree/bindings/iio/multiplexer/io-channel-mux.txt F: drivers/iio/multiplexer/iio-mux.c IIO SUBSYSTEM AND DRIVERS @@ -6902,6 +6980,15 @@ F: drivers/staging/iio/ F: include/linux/iio/ F: tools/iio/ +IIO UNIT CONVERTER +M: Peter Rosin +L: linux-iio@vger.kernel.org +S: Maintained +F: Documentation/devicetree/bindings/iio/afe/current-sense-amplifier.txt +F: Documentation/devicetree/bindings/iio/afe/current-sense-shunt.txt +F: Documentation/devicetree/bindings/iio/afe/voltage-divider.txt +F: drivers/iio/afe/iio-rescale.c + IKANOS/ADI EAGLE ADSL USB DRIVER M: Matthieu Castet M: Stanislaw Gruszka @@ -7034,14 +7121,13 @@ L: linux-fbdev@vger.kernel.org S: Maintained F: drivers/video/fbdev/i810/ -INTEL ASoC BDW/HSW DRIVERS +INTEL ASoC DRIVERS +M: Pierre-Louis Bossart +M: Liam Girdwood M: Jie Yang L: alsa-devel@alsa-project.org (moderated for non-subscribers) S: Supported -F: sound/soc/intel/common/sst-dsp* -F: sound/soc/intel/common/sst-firmware.c -F: sound/soc/intel/boards/broadwell.c -F: sound/soc/intel/haswell/ +F: sound/soc/intel/ INTEL C600 SERIES SAS CONTROLLER DRIVER M: Intel SCU Linux support @@ -7076,8 +7162,8 @@ Q: http://patchwork.ozlabs.org/project/intel-wired-lan/list/ T: git git://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/net-queue.git T: git git://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/next-queue.git S: Supported -F: Documentation/networking/e100.txt -F: Documentation/networking/e1000.txt +F: Documentation/networking/e100.rst +F: Documentation/networking/e1000.rst F: Documentation/networking/e1000e.txt F: Documentation/networking/igb.txt F: Documentation/networking/igbvf.txt @@ -7310,7 +7396,7 @@ F: drivers/platform/x86/intel-wmi-thunderbolt.c INTEL(R) TRACE HUB M: Alexander Shishkin S: Supported -F: Documentation/trace/intel_th.txt +F: Documentation/trace/intel_th.rst F: drivers/hwtracing/intel_th/ INTEL(R) TRUSTED EXECUTION TECHNOLOGY (TXT) @@ -7334,7 +7420,7 @@ M: Linus Walleij L: linux-iio@vger.kernel.org S: Maintained F: drivers/iio/gyro/mpu3050* -F: Documentation/devicetree/bindings/iio/gyroscope/inv,mpu3050.txt +F: Documentation/devicetree/bindings/iio/gyroscope/invensense,mpu3050.txt IOC3 ETHERNET DRIVER M: Ralf Baechle @@ -7637,8 +7723,9 @@ M: Masahiro Yamada T: git git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git kconfig L: linux-kbuild@vger.kernel.org S: Maintained -F: Documentation/kbuild/kconfig-language.txt +F: Documentation/kbuild/kconfig* F: scripts/kconfig/ +F: scripts/Kconfig.include KDUMP M: Dave Young @@ -7657,11 +7744,11 @@ W: https://linuxtv.org S: Maintained F: drivers/media/radio/radio-keene* -KERNEL AUTOMOUNTER v4 (AUTOFS4) +KERNEL AUTOMOUNTER M: Ian Kent L: autofs@vger.kernel.org S: Maintained -F: fs/autofs4/ +F: fs/autofs/ KERNEL BUILD + files below scripts/ (unless maintained elsewhere) M: Masahiro Yamada @@ -7905,8 +7992,6 @@ F: kernel/kprobes.c KS0108 LCD CONTROLLER DRIVER M: Miguel Ojeda Sandonis -W: http://miguelojeda.es/auxdisplay.htm -W: http://jair.lab.fi.uva.es/~migojed/auxdisplay.htm S: Maintained F: Documentation/auxdisplay/ks0108 F: drivers/auxdisplay/ks0108.c @@ -7964,6 +8049,13 @@ S: Maintained F: Documentation/misc-devices/eeprom F: drivers/misc/eeprom/eeprom.c +LEGO MINDSTORMS EV3 +R: David Lechner +S: Maintained +F: arch/arm/boot/dts/da850-lego-ev3.dts +F: Documentation/devicetree/bindings/power/supply/lego_ev3_battery.txt +F: drivers/power/supply/lego_ev3_battery.c + LEGO USB Tower driver M: Juergen Stuber L: legousb-devel@lists.sourceforge.net @@ -8209,7 +8301,7 @@ F: drivers/misc/lkdtm/* LINUX KERNEL MEMORY CONSISTENCY MODEL (LKMM) M: Alan Stern -M: Andrea Parri +M: Andrea Parri M: Will Deacon M: Peter Zijlstra M: Boqun Feng @@ -8316,6 +8408,7 @@ F: Documentation/admin-guide/LSM/LoadPin.rst LOCKING PRIMITIVES M: Peter Zijlstra M: Ingo Molnar +M: Will Deacon L: linux-kernel@vger.kernel.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git locking/core S: Maintained @@ -8350,7 +8443,6 @@ L: linux-scsi@vger.kernel.org W: http://www.avagotech.com/support/ S: Supported F: drivers/message/fusion/ -F: drivers/scsi/mpt2sas/ F: drivers/scsi/mpt3sas/ LSILOGIC/SYMBIOS/NCR 53C8XX and 53C1010 PCI-SCSI drivers @@ -8472,6 +8564,7 @@ M: Vivien Didelot L: netdev@vger.kernel.org S: Maintained F: drivers/net/dsa/mv88e6xxx/ +F: linux/platform_data/mv88e6xxx.h F: Documentation/devicetree/bindings/net/dsa/marvell.txt MARVELL ARMADA DRM SUPPORT @@ -8602,7 +8695,7 @@ M: Guenter Roeck L: linux-hwmon@vger.kernel.org S: Maintained F: Documentation/hwmon/max6697 -F: Documentation/devicetree/bindings/i2c/max6697.txt +F: Documentation/devicetree/bindings/hwmon/max6697.txt F: drivers/hwmon/max6697.c F: include/linux/platform_data/max6697.h @@ -8829,6 +8922,7 @@ L: linux-media@vger.kernel.org L: linux-renesas-soc@vger.kernel.org T: git git://linuxtv.org/media_tree.git S: Supported +F: Documentation/devicetree/bindings/media/renesas,rcar-csi2.txt F: Documentation/devicetree/bindings/media/rcar_vin.txt F: drivers/media/platform/rcar-vin/ @@ -8955,6 +9049,13 @@ L: linux-wireless@vger.kernel.org S: Maintained F: drivers/net/wireless/mediatek/mt7601u/ +MEDIATEK NAND CONTROLLER DRIVER +M: Xiaolei Li +L: linux-mtd@lists.infradead.org +S: Maintained +F: drivers/mtd/nand/raw/mtk_* +F: Documentation/devicetree/bindings/mtd/mtk-nand.txt + MEDIATEK RANDOM NUMBER GENERATOR SUPPORT M: Sean Wang S: Maintained @@ -8974,7 +9075,7 @@ M: Martin Donnelly M: Martyn Welch S: Maintained F: drivers/gpu/drm/bridge/megachips-stdpxxxx-ge-b850v3-fw.c -F: Documentation/devicetree/bindings/video/bridge/megachips-stdpxxxx-ge-b850v3-fw.txt +F: Documentation/devicetree/bindings/display/bridge/megachips-stdpxxxx-ge-b850v3-fw.txt MEGARAID SCSI/SAS DRIVERS M: Kashyap Desai @@ -9025,12 +9126,14 @@ W: http://www.mellanox.com Q: http://patchwork.ozlabs.org/project/netdev/list/ F: drivers/net/ethernet/mellanox/mlx5/core/en_* -MELLANOX ETHERNET INNOVA DRIVER +MELLANOX ETHERNET INNOVA DRIVERS R: Boris Pismenny L: netdev@vger.kernel.org S: Supported W: http://www.mellanox.com Q: http://patchwork.ozlabs.org/project/netdev/list/ +F: drivers/net/ethernet/mellanox/mlx5/core/en_accel/* +F: drivers/net/ethernet/mellanox/mlx5/core/accel/* F: drivers/net/ethernet/mellanox/mlx5/core/fpga/* F: include/linux/mlx5/mlx5_ifc_fpga.h @@ -9290,6 +9393,12 @@ F: include/linux/cciss*.h F: include/uapi/linux/cciss*.h F: Documentation/scsi/smartpqi.txt +MICROSEMI ETHERNET SWITCH DRIVER +M: Alexandre Belloni +L: netdev@vger.kernel.org +S: Supported +F: drivers/net/ethernet/mscc/ + MICROSOFT SURFACE PRO 3 BUTTON DRIVER M: Chen Yu L: platform-driver-x86@vger.kernel.org @@ -9303,10 +9412,12 @@ F: drivers/usb/image/microtek.* MIPS M: Ralf Baechle +M: Paul Burton M: James Hogan L: linux-mips@linux-mips.org W: http://www.linux-mips.org/ T: git git://git.linux-mips.org/pub/scm/ralf/linux.git +T: git git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git Q: http://patchwork.linux-mips.org/project/linux-mips/list/ S: Supported F: Documentation/devicetree/bindings/mips/ @@ -9403,6 +9514,13 @@ Q: http://patchwork.linuxtv.org/project/linux-media/list/ S: Maintained F: drivers/media/dvb-frontends/mn88473* +PCI DRIVER FOR MOBIVEIL PCIE IP +M: Subrahmanya Lingappa +L: linux-pci@vger.kernel.org +S: Supported +F: Documentation/devicetree/bindings/pci/mobiveil-pcie.txt +F: drivers/pci/controller/pcie-mobiveil.c + MODULE SUPPORT M: Jessica Yu T: git git://git.kernel.org/pub/scm/linux/kernel/git/jeyu/linux.git modules-next @@ -9542,7 +9660,7 @@ F: include/uapi/linux/mmc/ MULTIPLEXER SUBSYSTEM M: Peter Rosin S: Maintained -F: Documentation/ABI/testing/mux/sysfs-class-mux* +F: Documentation/ABI/testing/sysfs-class-mux* F: Documentation/devicetree/bindings/mux/ F: include/linux/dt-bindings/mux/ F: include/linux/mux/ @@ -9573,7 +9691,7 @@ MXSFB DRM DRIVER M: Marek Vasut S: Supported F: drivers/gpu/drm/mxsfb/ -F: Documentation/devicetree/bindings/display/mxsfb-drm.txt +F: Documentation/devicetree/bindings/display/mxsfb.txt MYRICOM MYRI-10G 10GbE DRIVER (MYRI10GE) M: Chris Lee @@ -9584,6 +9702,7 @@ F: drivers/net/ethernet/myricom/myri10ge/ NAND FLASH SUBSYSTEM M: Boris Brezillon +M: Miquel Raynal R: Richard Weinberger L: linux-mtd@lists.infradead.org W: http://www.linux-mtd.infradead.org/ @@ -9632,6 +9751,11 @@ L: linux-scsi@vger.kernel.org S: Maintained F: drivers/scsi/NCR_D700.* +NCSI LIBRARY: +M: Samuel Mendoza-Jonas +S: Maintained +F: net/ncsi/ + NCT6775 HARDWARE MONITOR DRIVER M: Guenter Roeck L: linux-hwmon@vger.kernel.org @@ -9639,6 +9763,14 @@ S: Maintained F: Documentation/hwmon/nct6775 F: drivers/hwmon/nct6775.c +NET_FAILOVER MODULE +M: Sridhar Samudrala +L: netdev@vger.kernel.org +S: Supported +F: driver/net/net_failover.c +F: include/net/net_failover.h +F: Documentation/networking/net_failover.rst + NETEFFECT IWARP RNIC DRIVER (IW_NES) M: Faisal Latif L: linux-rdma@vger.kernel.org @@ -9700,7 +9832,7 @@ S: Maintained F: drivers/net/ethernet/netronome/ NETWORK BLOCK DEVICE (NBD) -M: Josef Bacik +M: Josef Bacik S: Maintained L: linux-block@vger.kernel.org L: nbd@other.debian.org @@ -9832,7 +9964,21 @@ F: net/ipv6/calipso.c F: net/netfilter/xt_CONNSECMARK.c F: net/netfilter/xt_SECMARK.c +NETWORKING [TCP] +M: Eric Dumazet +L: netdev@vger.kernel.org +S: Maintained +F: net/ipv4/tcp*.c +F: net/ipv4/syncookies.c +F: net/ipv6/tcp*.c +F: net/ipv6/syncookies.c +F: include/uapi/linux/tcp.h +F: include/net/tcp.h +F: include/linux/tcp.h +F: include/trace/events/tcp.h + NETWORKING [TLS] +M: Boris Pismenny M: Aviad Yehezkel M: Dave Watson L: netdev@vger.kernel.org @@ -10098,7 +10244,7 @@ F: arch/powerpc/include/asm/pnv-ocxl.h F: drivers/misc/ocxl/ F: include/misc/ocxl* F: include/uapi/misc/ocxl.h -F: Documentation/accelerators/ocxl.txt +F: Documentation/accelerators/ocxl.rst OMAP AUDIO SUPPORT M: Peter Ujfalusi @@ -10127,18 +10273,16 @@ F: arch/arm/boot/dts/*am5* F: arch/arm/boot/dts/*dra7* OMAP DISPLAY SUBSYSTEM and FRAMEBUFFER SUPPORT (DSS2) -M: Tomi Valkeinen L: linux-omap@vger.kernel.org L: linux-fbdev@vger.kernel.org -S: Maintained +S: Orphan F: drivers/video/fbdev/omap2/ F: Documentation/arm/OMAP/DSS OMAP FRAMEBUFFER SUPPORT -M: Tomi Valkeinen L: linux-fbdev@vger.kernel.org L: linux-omap@vger.kernel.org -S: Maintained +S: Orphan F: drivers/video/fbdev/omap/ OMAP GENERAL PURPOSE MEMORY CONTROLLER SUPPORT @@ -10246,7 +10390,7 @@ F: arch/arm/mach-omap1/ F: arch/arm/plat-omap/ F: arch/arm/configs/omap1_defconfig F: drivers/i2c/busses/i2c-omap.c -F: include/linux/i2c-omap.h +F: include/linux/platform_data/i2c-omap.h OMAP2+ SUPPORT M: Tony Lindgren @@ -10278,7 +10422,7 @@ F: drivers/regulator/tps65218-regulator.c F: drivers/regulator/tps65910-regulator.c F: drivers/regulator/twl-regulator.c F: drivers/regulator/twl6030-regulator.c -F: include/linux/i2c-omap.h +F: include/linux/platform_data/i2c-omap.h ONION OMEGA2+ BOARD M: Harvey Hunt @@ -10355,6 +10499,7 @@ T: git git://linuxtv.org/media_tree.git S: Odd fixes F: drivers/media/i2c/ov772x.c F: include/media/i2c/ov772x.h +F: Documentation/devicetree/bindings/media/i2c/ov772x.txt OMNIVISION OV7740 SENSOR DRIVER M: Wenyou Yang @@ -10405,12 +10550,14 @@ F: drivers/infiniband/ulp/opa_vnic OPEN FIRMWARE AND DEVICE TREE OVERLAYS M: Pantelis Antoniou +M: Frank Rowand L: devicetree@vger.kernel.org S: Maintained F: Documentation/devicetree/dynamic-resolution-notes.txt F: Documentation/devicetree/overlay-notes.txt F: drivers/of/overlay.c F: drivers/of/resolver.c +K: of_overlay_notifier_ OPEN FIRMWARE AND FLATTENED DEVICE TREE M: Rob Herring @@ -10579,7 +10726,7 @@ PARALLEL LCD/KEYPAD PANEL DRIVER M: Willy Tarreau M: Ksenija Stanojevic S: Odd Fixes -F: Documentation/misc-devices/lcd-panel-cgram.txt +F: Documentation/auxdisplay/lcd-panel-cgram.txt F: drivers/misc/panel.c PARALLEL PORT SUBSYSTEM @@ -10680,7 +10827,7 @@ L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained F: Documentation/devicetree/bindings/pci/aardvark-pci.txt -F: drivers/pci/host/pci-aardvark.c +F: drivers/pci/controller/pci-aardvark.c PCI DRIVER FOR ALTERA PCIE IP M: Ley Foon Tan @@ -10688,7 +10835,7 @@ L: rfi@lists.rocketboards.org (moderated for non-subscribers) L: linux-pci@vger.kernel.org S: Supported F: Documentation/devicetree/bindings/pci/altera-pcie.txt -F: drivers/pci/host/pcie-altera.c +F: drivers/pci/controller/pcie-altera.c PCI DRIVER FOR APPLIEDMICRO XGENE M: Tanmay Inamdar @@ -10696,7 +10843,7 @@ L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org S: Maintained F: Documentation/devicetree/bindings/pci/xgene-pci.txt -F: drivers/pci/host/pci-xgene.c +F: drivers/pci/controller/pci-xgene.c PCI DRIVER FOR ARM VERSATILE PLATFORM M: Rob Herring @@ -10704,7 +10851,7 @@ L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org S: Maintained F: Documentation/devicetree/bindings/pci/versatile.txt -F: drivers/pci/host/pci-versatile.c +F: drivers/pci/controller/pci-versatile.c PCI DRIVER FOR ARMADA 8K M: Thomas Petazzoni @@ -10712,24 +10859,24 @@ L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org S: Maintained F: Documentation/devicetree/bindings/pci/pci-armada8k.txt -F: drivers/pci/dwc/pcie-armada8k.c +F: drivers/pci/controller/dwc/pcie-armada8k.c PCI DRIVER FOR CADENCE PCIE IP M: Alan Douglas L: linux-pci@vger.kernel.org S: Maintained F: Documentation/devicetree/bindings/pci/cdns,*.txt -F: drivers/pci/cadence/pcie-cadence* +F: drivers/pci/controller/pcie-cadence* PCI DRIVER FOR FREESCALE LAYERSCAPE -M: Minghuan Lian -M: Mingkai Hu -M: Roy Zang +M: Minghuan Lian +M: Mingkai Hu +M: Roy Zang L: linuxppc-dev@lists.ozlabs.org L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org S: Maintained -F: drivers/pci/dwc/*layerscape* +F: drivers/pci/controller/dwc/*layerscape* PCI DRIVER FOR GENERIC OF HOSTS M: Will Deacon @@ -10737,8 +10884,8 @@ L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained F: Documentation/devicetree/bindings/pci/host-generic-pci.txt -F: drivers/pci/host/pci-host-common.c -F: drivers/pci/host/pci-host-generic.c +F: drivers/pci/controller/pci-host-common.c +F: drivers/pci/controller/pci-host-generic.c PCI DRIVER FOR IMX6 M: Richard Zhu @@ -10747,14 +10894,14 @@ L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained F: Documentation/devicetree/bindings/pci/fsl,imx6q-pcie.txt -F: drivers/pci/dwc/*imx6* +F: drivers/pci/controller/dwc/*imx6* PCI DRIVER FOR INTEL VOLUME MANAGEMENT DEVICE (VMD) M: Keith Busch M: Jonathan Derrick L: linux-pci@vger.kernel.org S: Supported -F: drivers/pci/host/vmd.c +F: drivers/pci/controller/vmd.c PCI DRIVER FOR MICROSEMI SWITCHTEC M: Kurt Schwemmer @@ -10774,7 +10921,7 @@ M: Jason Cooper L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained -F: drivers/pci/host/*mvebu* +F: drivers/pci/controller/*mvebu* PCI DRIVER FOR NVIDIA TEGRA M: Thierry Reding @@ -10782,14 +10929,14 @@ L: linux-tegra@vger.kernel.org L: linux-pci@vger.kernel.org S: Supported F: Documentation/devicetree/bindings/pci/nvidia,tegra20-pcie.txt -F: drivers/pci/host/pci-tegra.c +F: drivers/pci/controller/pci-tegra.c PCI DRIVER FOR RENESAS R-CAR M: Simon Horman L: linux-pci@vger.kernel.org L: linux-renesas-soc@vger.kernel.org S: Maintained -F: drivers/pci/host/*rcar* +F: drivers/pci/controller/*rcar* PCI DRIVER FOR SAMSUNG EXYNOS M: Jingoo Han @@ -10797,7 +10944,7 @@ L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) L: linux-samsung-soc@vger.kernel.org (moderated for non-subscribers) S: Maintained -F: drivers/pci/dwc/pci-exynos.c +F: drivers/pci/controller/dwc/pci-exynos.c PCI DRIVER FOR SYNOPSYS DESIGNWARE M: Jingoo Han @@ -10805,7 +10952,7 @@ M: Joao Pinto L: linux-pci@vger.kernel.org S: Maintained F: Documentation/devicetree/bindings/pci/designware-pcie.txt -F: drivers/pci/dwc/*designware* +F: drivers/pci/controller/dwc/*designware* PCI DRIVER FOR TI DRA7XX M: Kishon Vijay Abraham I @@ -10813,14 +10960,14 @@ L: linux-omap@vger.kernel.org L: linux-pci@vger.kernel.org S: Supported F: Documentation/devicetree/bindings/pci/ti-pci.txt -F: drivers/pci/dwc/pci-dra7xx.c +F: drivers/pci/controller/dwc/pci-dra7xx.c PCI DRIVER FOR TI KEYSTONE M: Murali Karicheri L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained -F: drivers/pci/dwc/*keystone* +F: drivers/pci/controller/dwc/*keystone* PCI ENDPOINT SUBSYSTEM M: Kishon Vijay Abraham I @@ -10853,7 +11000,7 @@ L: rfi@lists.rocketboards.org (moderated for non-subscribers) L: linux-pci@vger.kernel.org S: Supported F: Documentation/devicetree/bindings/pci/altera-pcie-msi.txt -F: drivers/pci/host/pcie-altera-msi.c +F: drivers/pci/controller/pcie-altera-msi.c PCI MSI DRIVER FOR APPLIEDMICRO XGENE M: Duc Dang @@ -10861,7 +11008,7 @@ L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org S: Maintained F: Documentation/devicetree/bindings/pci/xgene-pci-msi.txt -F: drivers/pci/host/pci-xgene-msi.c +F: drivers/pci/controller/pci-xgene-msi.c PCI SUBSYSTEM M: Bjorn Helgaas @@ -10887,9 +11034,7 @@ L: linux-pci@vger.kernel.org Q: http://patchwork.ozlabs.org/project/linux-pci/list/ T: git git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/pci.git/ S: Supported -F: drivers/pci/cadence/ -F: drivers/pci/host/ -F: drivers/pci/dwc/ +F: drivers/pci/controller/ PCIE DRIVER FOR AXIS ARTPEC M: Jesper Nilsson @@ -10897,7 +11042,7 @@ L: linux-arm-kernel@axis.com L: linux-pci@vger.kernel.org S: Maintained F: Documentation/devicetree/bindings/pci/axis,artpec* -F: drivers/pci/dwc/*artpec* +F: drivers/pci/controller/dwc/*artpec* PCIE DRIVER FOR CAVIUM THUNDERX M: David Daney @@ -10905,22 +11050,22 @@ L: linux-pci@vger.kernel.org L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Supported F: Documentation/devicetree/bindings/pci/pci-thunder-* -F: drivers/pci/host/pci-thunder-* +F: drivers/pci/controller/pci-thunder-* PCIE DRIVER FOR HISILICON M: Zhou Wang L: linux-pci@vger.kernel.org S: Maintained F: Documentation/devicetree/bindings/pci/hisilicon-pcie.txt -F: drivers/pci/dwc/pcie-hisi.c +F: drivers/pci/controller/dwc/pcie-hisi.c PCIE DRIVER FOR HISILICON KIRIN M: Xiaowei Song M: Binghui Wang L: linux-pci@vger.kernel.org S: Maintained -F: Documentation/devicetree/bindings/pci/pcie-kirin.txt -F: drivers/pci/dwc/pcie-kirin.c +F: Documentation/devicetree/bindings/pci/kirin-pcie.txt +F: drivers/pci/controller/dwc/pcie-kirin.c PCIE DRIVER FOR HISILICON STB M: Jianguo Sun @@ -10928,7 +11073,7 @@ M: Shawn Guo L: linux-pci@vger.kernel.org S: Maintained F: Documentation/devicetree/bindings/pci/hisilicon-histb-pcie.txt -F: drivers/pci/dwc/pcie-histb.c +F: drivers/pci/controller/dwc/pcie-histb.c PCIE DRIVER FOR MEDIATEK M: Ryder Lee @@ -10936,35 +11081,35 @@ L: linux-pci@vger.kernel.org L: linux-mediatek@lists.infradead.org S: Supported F: Documentation/devicetree/bindings/pci/mediatek* -F: drivers/pci/host/*mediatek* +F: drivers/pci/controller/*mediatek* PCIE DRIVER FOR QUALCOMM MSM M: Stanimir Varbanov L: linux-pci@vger.kernel.org L: linux-arm-msm@vger.kernel.org S: Maintained -F: drivers/pci/dwc/*qcom* +F: drivers/pci/controller/dwc/*qcom* PCIE DRIVER FOR ROCKCHIP M: Shawn Lin L: linux-pci@vger.kernel.org L: linux-rockchip@lists.infradead.org S: Maintained -F: Documentation/devicetree/bindings/pci/rockchip-pcie.txt -F: drivers/pci/host/pcie-rockchip.c +F: Documentation/devicetree/bindings/pci/rockchip-pcie* +F: drivers/pci/controller/pcie-rockchip* PCI DRIVER FOR V3 SEMICONDUCTOR V360EPC M: Linus Walleij L: linux-pci@vger.kernel.org S: Maintained F: Documentation/devicetree/bindings/pci/v3-v360epc-pci.txt -F: drivers/pci/host/pci-v3-semi.c +F: drivers/pci/controller/pci-v3-semi.c PCIE DRIVER FOR ST SPEAR13XX M: Pratyush Anand L: linux-pci@vger.kernel.org S: Maintained -F: drivers/pci/dwc/*spear* +F: drivers/pci/controller/dwc/*spear* PCMCIA SUBSYSTEM M: Dominik Brodowski @@ -11114,6 +11259,7 @@ L: linux-mediatek@lists.infradead.org (moderated for non-subscribers) S: Maintained F: Documentation/devicetree/bindings/pinctrl/pinctrl-mt65xx.txt F: Documentation/devicetree/bindings/pinctrl/pinctrl-mt7622.txt +F: drivers/pinctrl/mediatek/mtk-eint.* F: drivers/pinctrl/mediatek/pinctrl-mtk-common.* F: drivers/pinctrl/mediatek/pinctrl-mt2701.c F: drivers/pinctrl/mediatek/pinctrl-mt7622.c @@ -11252,6 +11398,7 @@ M: Sebastian Reichel L: linux-pm@vger.kernel.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-power-supply.git S: Maintained +F: Documentation/ABI/testing/sysfs-class-power F: Documentation/devicetree/bindings/power/supply/ F: include/linux/power_supply.h F: drivers/power/supply/ @@ -11397,7 +11544,6 @@ S: Maintained W: http://linuxptp.sourceforge.net/ F: Documentation/ABI/testing/sysfs-ptp F: Documentation/ptp/* -F: drivers/net/ethernet/freescale/gianfar_ptp.c F: drivers/net/phy/dp83640* F: drivers/ptp/* F: include/linux/ptp_cl* @@ -11654,6 +11800,13 @@ F: Documentation/devicetree/bindings/media/qcom,camss.txt F: Documentation/media/v4l-drivers/qcom_camss.rst F: drivers/media/platform/qcom/camss-8x16/ +QUALCOMM CPUFREQ DRIVER MSM8996/APQ8096 +M: Ilia Lin +L: linux-pm@vger.kernel.org +S: Maintained +F: Documentation/devicetree/bindings/opp/kryo-cpufreq.txt +F: drivers/cpufreq/qcom-cpufreq-kryo.c + QUALCOMM EMAC GIGABIT ETHERNET DRIVER M: Timur Tabi L: netdev@vger.kernel.org @@ -11667,6 +11820,14 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/rkuo/linux-hexagon-kernel.g S: Supported F: arch/hexagon/ +QUALCOMM HIDMA DRIVER +M: Sinan Kaya +L: linux-arm-kernel@lists.infradead.org +L: linux-arm-msm@vger.kernel.org +L: dmaengine@vger.kernel.org +S: Supported +F: drivers/dma/qcom/hidma* + QUALCOMM IOMMU M: Rob Clark L: iommu@lists.linux-foundation.org @@ -11878,7 +12039,7 @@ F: include/linux/rtc.h F: include/uapi/linux/rtc.h F: include/linux/rtc/ F: include/linux/platform_data/rtc-* -F: tools/testing/selftests/timers/rtctest.c +F: tools/testing/selftests/rtc/ REALTEK AUDIO CODECS M: Bard Liao @@ -11972,6 +12133,18 @@ F: include/dt-bindings/reset/ F: include/linux/reset.h F: include/linux/reset-controller.h +RESTARTABLE SEQUENCES SUPPORT +M: Mathieu Desnoyers +M: Peter Zijlstra +M: "Paul E. McKenney" +M: Boqun Feng +L: linux-kernel@vger.kernel.org +S: Supported +F: kernel/rseq.c +F: include/uapi/linux/rseq.h +F: include/trace/events/rseq.h +F: tools/testing/selftests/rseq/ + RFKILL M: Johannes Berg L: linux-wireless@vger.kernel.org @@ -12004,7 +12177,7 @@ F: drivers/mtd/nand/raw/r852.h RISC-V ARCHITECTURE M: Palmer Dabbelt -M: Albert Ou +M: Albert Ou L: linux-riscv@lists.infradead.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/palmer/riscv-linux.git S: Supported @@ -12282,7 +12455,7 @@ L: linux-crypto@vger.kernel.org L: linux-samsung-soc@vger.kernel.org S: Maintained F: drivers/crypto/exynos-rng.c -F: Documentation/devicetree/bindings/crypto/samsung,exynos-rng4.txt +F: Documentation/devicetree/bindings/rng/samsung,exynos4-rng.txt SAMSUNG EXYNOS TRUE RANDOM NUMBER GENERATOR (TRNG) DRIVER M: Łukasz Stelmach @@ -12764,6 +12937,14 @@ F: drivers/media/usb/siano/ F: drivers/media/usb/siano/ F: drivers/media/mmc/siano/ +SIFIVE DRIVERS +M: Palmer Dabbelt +L: linux-riscv@lists.infradead.org +T: git git://git.kernel.org/pub/scm/linux/kernel/git/palmer/riscv-linux.git +S: Supported +K: sifive +N: sifive + SILEAD TOUCHSCREEN DRIVER M: Hans de Goede L: linux-input@vger.kernel.org @@ -13061,6 +13242,13 @@ S: Maintained F: drivers/ssb/ F: include/linux/ssb/ +SONY IMX258 SENSOR DRIVER +M: Sakari Ailus +L: linux-media@vger.kernel.org +T: git git://linuxtv.org/media_tree.git +S: Maintained +F: drivers/media/i2c/imx258.c + SONY IMX274 SENSOR DRIVER M: Leon Luo L: linux-media@vger.kernel.org @@ -13105,11 +13293,11 @@ F: include/uapi/sound/ F: sound/ SOUND - COMPRESSED AUDIO -M: Vinod Koul +M: Vinod Koul L: alsa-devel@alsa-project.org (moderated for non-subscribers) T: git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git S: Supported -F: Documentation/sound/alsa/compress_offload.txt +F: Documentation/sound/designs/compress-offload.rst F: include/sound/compress_driver.h F: include/uapi/sound/compress_* F: sound/core/compress_offload.c @@ -13130,7 +13318,7 @@ L: alsa-devel@alsa-project.org (moderated for non-subscribers) W: http://alsa-project.org/main/index.php/ASoC S: Supported F: Documentation/devicetree/bindings/sound/ -F: Documentation/sound/alsa/soc/ +F: Documentation/sound/soc/ F: sound/soc/ F: include/sound/soc* @@ -13277,13 +13465,6 @@ L: stable@vger.kernel.org S: Supported F: Documentation/process/stable-kernel-rules.rst -STAGING - ATOMISP DRIVER -M: Alan Cox -M: Sakari Ailus -L: linux-media@vger.kernel.org -S: Maintained -F: drivers/staging/media/atomisp/ - STAGING - COMEDI M: Ian Abbott M: H Hartley Sweeten @@ -13302,15 +13483,6 @@ S: Odd Fixes F: Documentation/devicetree/bindings/staging/iio/ F: drivers/staging/iio/ -STAGING - LUSTRE PARALLEL FILESYSTEM -M: Oleg Drokin -M: Andreas Dilger -M: James Simmons -L: lustre-devel@lists.lustre.org (moderated for non-subscribers) -W: http://wiki.lustre.org/ -S: Maintained -F: drivers/staging/lustre - STAGING - NVIDIA COMPLIANT EMBEDDED CONTROLLER INTERFACE (nvec) M: Marc Dietrich L: ac100@lists.launchpad.net (moderated for non-subscribers) @@ -13398,9 +13570,20 @@ T: git git://linuxtv.org/media_tree.git S: Maintained F: drivers/media/usb/stk1160/ +STM32 TIMER/LPTIMER DRIVERS +M: Fabrice Gasnier +S: Maintained +F: drivers/*/stm32-*timer* +F: drivers/pwm/pwm-stm32* +F: include/linux/*/stm32-*tim* +F: Documentation/ABI/testing/*timer-stm32 +F: Documentation/devicetree/bindings/*/stm32-*timer* +F: Documentation/devicetree/bindings/pwm/pwm-stm32* + STMMAC ETHERNET DRIVER M: Giuseppe Cavallaro M: Alexandre Torgue +M: Jose Abreu L: netdev@vger.kernel.org W: http://www.stlinux.com S: Supported @@ -13465,7 +13648,7 @@ M: Konrad Rzeszutek Wilk L: iommu@lists.linux-foundation.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb.git S: Supported -F: lib/swiotlb.c +F: kernel/dma/swiotlb.c F: arch/*/kernel/pci-swiotlb.c F: include/linux/swiotlb.h @@ -13477,6 +13660,12 @@ S: Supported F: net/switchdev/ F: include/net/switchdev.h +SY8106A REGULATOR DRIVER +M: Icenowy Zheng +S: Maintained +F: drivers/regulator/sy8106a-regulator.c +F: Documentation/devicetree/bindings/regulator/sy8106a-regulator.txt + SYNC FILE FRAMEWORK M: Sumit Semwal R: Gustavo Padovan @@ -13611,7 +13800,7 @@ SYSTEM TRACE MODULE CLASS M: Alexander Shishkin S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/ash/stm.git -F: Documentation/trace/stm.txt +F: Documentation/trace/stm.rst F: drivers/hwtracing/stm/ F: include/linux/stm.h F: include/uapi/linux/stm.h @@ -14288,7 +14477,7 @@ M: Steven Rostedt M: Ingo Molnar T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git perf/core S: Maintained -F: Documentation/trace/ftrace.txt +F: Documentation/trace/ftrace.rst F: arch/*/*/*/ftrace.h F: arch/*/kernel/ftrace.c F: include/*/ftrace.h @@ -14371,6 +14560,15 @@ S: Maintained F: drivers/tc/ F: include/linux/tc.h +TURBOSTAT UTILITY +M: "Len Brown" +L: linux-pm@vger.kernel.org +B: https://bugzilla.kernel.org +Q: https://patchwork.kernel.org/project/linux-pm/list/ +T: git git://git.kernel.org/pub/scm/linux/kernel/git/lenb/linux.git turbostat +S: Supported +F: tools/power/x86/turbostat/ + TW5864 VIDEO4LINUX DRIVER M: Bluecherry Maintainers M: Anton Sviridenko @@ -14617,7 +14815,9 @@ M: Woojung Huh M: Microchip Linux Driver Support L: netdev@vger.kernel.org S: Maintained +F: Documentation/devicetree/bindings/net/microchip,lan78xx.txt F: drivers/net/usb/lan78xx.* +F: include/dt-bindings/net/microchip-lan78xx.h USB MASS STORAGE DRIVER M: Alan Stern @@ -14661,6 +14861,7 @@ S: Maintained F: Documentation/usb/usbip_protocol.txt F: drivers/usb/usbip/ F: tools/usb/usbip/ +F: tools/testing/selftests/drivers/usb/usbip/ USB PEGASUS DRIVER M: Petko Manolov @@ -14745,7 +14946,7 @@ M: Heikki Krogerus L: linux-usb@vger.kernel.org S: Maintained F: Documentation/ABI/testing/sysfs-class-typec -F: Documentation/usb/typec.rst +F: Documentation/driver-api/usb/typec.rst F: drivers/usb/typec/ F: include/linux/usb/typec.h @@ -14812,8 +15013,7 @@ F: drivers/media/usb/zr364xx/ USER-MODE LINUX (UML) M: Jeff Dike M: Richard Weinberger -L: user-mode-linux-devel@lists.sourceforge.net -L: user-mode-linux-user@lists.sourceforge.net +L: linux-um@lists.infradead.org W: http://user-mode-linux.sourceforge.net T: git git://git.kernel.org/pub/scm/linux/kernel/git/rw/uml.git S: Maintained @@ -14933,6 +15133,12 @@ L: linux-media@vger.kernel.org S: Maintained F: drivers/media/platform/video-mux.c +VIDEO I2C POLLING DRIVER +M: Matt Ranostay +L: linux-media@vger.kernel.org +S: Maintained +F: drivers/media/i2c/video-i2c.c + VIDEOBUF2 FRAMEWORK M: Pawel Osciak M: Marek Szyprowski @@ -15369,6 +15575,13 @@ S: Maintained F: Documentation/x86/ F: arch/x86/ +X86 ENTRY CODE +M: Andy Lutomirski +L: linux-kernel@vger.kernel.org +T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/asm +S: Maintained +F: arch/x86/entry/ + X86 MCE INFRASTRUCTURE M: Tony Luck M: Borislav Petkov @@ -15391,7 +15604,7 @@ F: drivers/platform/x86/ F: drivers/platform/olpc/ X86 VDSO -M: Andy Lutomirski +M: Andy Lutomirski L: linux-kernel@vger.kernel.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/vdso S: Maintained @@ -15405,6 +15618,14 @@ T: git git://linuxtv.org/media_tree.git S: Maintained F: drivers/media/tuners/tuner-xc2028.* +XDP SOCKETS (AF_XDP) +M: Björn Töpel +M: Magnus Karlsson +L: netdev@vger.kernel.org +S: Maintained +F: kernel/bpf/xskmap.c +F: net/xdp/ + XEN BLOCK SUBSYSTEM M: Konrad Rzeszutek Wilk M: Roger Pau Monné @@ -15475,6 +15696,13 @@ S: Supported F: arch/x86/xen/*swiotlb* F: drivers/xen/*swiotlb* +XEN SOUND FRONTEND DRIVER +M: Oleksandr Andrushchenko +L: xen-devel@lists.xenproject.org (moderated for non-subscribers) +L: alsa-devel@alsa-project.org (moderated for non-subscribers) +S: Supported +F: sound/xen/* + XFS FILESYSTEM M: Darrick J. Wong M: linux-xfs@vger.kernel.org @@ -15513,6 +15741,14 @@ L: linux-kernel@vger.kernel.org S: Supported F: drivers/char/xillybus/ +XLP9XX I2C DRIVER +M: George Cherian +M: Jan Glauber +L: linux-i2c@vger.kernel.org +W: http://www.cavium.com +S: Supported +F: drivers/i2c/busses/i2c-xlp9xx.c + XRA1403 GPIO EXPANDER M: Nandor Han M: Semi Malinen @@ -15546,7 +15782,7 @@ YEALINK PHONE DRIVER M: Henk Vergonet L: usbb2k-api-dev@nongnu.org S: Maintained -F: Documentation/input/yealink.rst +F: Documentation/input/devices/yealink.rst F: drivers/input/misc/yealink.* Z8530 DRIVER FOR AX.25 @@ -15607,7 +15843,7 @@ L: linux-media@vger.kernel.org W: http://mjpeg.sourceforge.net/driver-zoran/ T: hg https://linuxtv.org/hg/v4l-dvb S: Odd Fixes -F: drivers/media/pci/zoran/ +F: drivers/staging/media/zoran/ ZRAM COMPRESSED RAM BLOCK DEVICE DRVIER M: Minchan Kim @@ -15631,7 +15867,7 @@ L: linux-mm@kvack.org S: Maintained F: mm/zsmalloc.c F: include/linux/zsmalloc.h -F: Documentation/vm/zsmalloc.txt +F: Documentation/vm/zsmalloc.rst ZSWAP COMPRESSED SWAP CACHING M: Seth Jennings diff --git a/Makefile b/Makefile index 56ba070dfa09..ca2af1ab91eb 100644 --- a/Makefile +++ b/Makefile @@ -1,8 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 VERSION = 4 -PATCHLEVEL = 17 +PATCHLEVEL = 18 SUBLEVEL = 0 -EXTRAVERSION = -rc7 +EXTRAVERSION = -rc1 NAME = Merciless Moray # *DOCUMENTATION* @@ -316,12 +316,9 @@ SUBARCH := $(shell uname -m | sed -e s/i.86/x86/ -e s/x86_64/x86/ \ # CROSS_COMPILE can be set on the command line # make CROSS_COMPILE=ia64-linux- # Alternatively CROSS_COMPILE can be set in the environment. -# A third alternative is to store a setting in .config so that plain -# "make" in the configured kernel build directory always uses that. # Default value for CROSS_COMPILE is not to prefix executables # Note: Some architectures assign CROSS_COMPILE in their arch/*/Makefile ARCH ?= $(SUBARCH) -CROSS_COMPILE ?= $(CONFIG_CROSS_COMPILE:"%"=%) # Architecture as present in compile.h UTS_MACHINE := $(ARCH) @@ -504,12 +501,23 @@ KBUILD_CFLAGS += $(call cc-option,-fno-PIE) KBUILD_AFLAGS += $(call cc-option,-fno-PIE) # check for 'asm goto' -ifeq ($(call shell-cached,$(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC) $(KBUILD_CFLAGS)), y) +ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC) $(KBUILD_CFLAGS)), y) CC_HAVE_ASM_GOTO := 1 KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO KBUILD_AFLAGS += -DCC_HAVE_ASM_GOTO endif +ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/cc-can-link.sh $(CC)), y) + CC_CAN_LINK := y + export CC_CAN_LINK +endif + +# The expansion should be delayed until arch/$(SRCARCH)/Makefile is included. +# Some architectures define CROSS_COMPILE in arch/$(SRCARCH)/Makefile. +# CC_VERSION_TEXT is referenced from Kconfig (so it needs export), +# and from include/config/auto.conf.cmd to detect the compiler upgrade. +CC_VERSION_TEXT = $(shell $(CC) --version | head -n 1) + ifeq ($(config-targets),1) # =========================================================================== # *config targets only - make sure prerequisites are updated, and descend @@ -519,7 +527,7 @@ ifeq ($(config-targets),1) # KBUILD_DEFCONFIG may point out an alternative default configuration # used for 'make defconfig' include arch/$(SRCARCH)/Makefile -export KBUILD_DEFCONFIG KBUILD_KCONFIG +export KBUILD_DEFCONFIG KBUILD_KCONFIG CC_VERSION_TEXT config: scripts_basic outputmakefile FORCE $(Q)$(MAKE) $(build)=scripts/kconfig $@ @@ -581,12 +589,32 @@ virt-y := virt/ endif # KBUILD_EXTMOD ifeq ($(dot-config),1) -# Read in config -include include/config/auto.conf +endif + +# The all: target is the default when no target is given on the +# command line. +# This allow a user to issue only 'make' to build a kernel including modules +# Defaults to vmlinux, but the arch makefile usually adds further targets +all: vmlinux + +CFLAGS_GCOV := -fprofile-arcs -ftest-coverage \ + $(call cc-option,-fno-tree-loop-im) \ + $(call cc-disable-warning,maybe-uninitialized,) +export CFLAGS_GCOV + +# The arch Makefile can set ARCH_{CPP,A,C}FLAGS to override the default +# values of the respective KBUILD_* variables +ARCH_CPPFLAGS := +ARCH_AFLAGS := +ARCH_CFLAGS := +include arch/$(SRCARCH)/Makefile +ifeq ($(dot-config),1) ifeq ($(KBUILD_EXTMOD),) -# Read in dependencies to all Kconfig* files, make sure to run -# oldconfig if changes are detected. +# Read in dependencies to all Kconfig* files, make sure to run syncconfig if +# changes are detected. This should be included after arch/$(SRCARCH)/Makefile +# because some architectures define CROSS_COMPILE there. -include include/config/auto.conf.cmd # To avoid any implicit rule to kick in, define an empty command @@ -618,24 +646,6 @@ else include/config/auto.conf: ; endif # $(dot-config) -# The all: target is the default when no target is given on the -# command line. -# This allow a user to issue only 'make' to build a kernel including modules -# Defaults to vmlinux, but the arch makefile usually adds further targets -all: vmlinux - -CFLAGS_GCOV := -fprofile-arcs -ftest-coverage \ - $(call cc-option,-fno-tree-loop-im) \ - $(call cc-disable-warning,maybe-uninitialized,) -export CFLAGS_GCOV CFLAGS_KCOV - -# The arch Makefile can set ARCH_{CPP,A,C}FLAGS to override the default -# values of the respective KBUILD_* variables -ARCH_CPPFLAGS := -ARCH_AFLAGS := -ARCH_CFLAGS := -include arch/$(SRCARCH)/Makefile - KBUILD_CFLAGS += $(call cc-option,-fno-delete-null-pointer-checks,) KBUILD_CFLAGS += $(call cc-disable-warning,frame-address,) KBUILD_CFLAGS += $(call cc-disable-warning, format-truncation) @@ -676,55 +686,11 @@ ifneq ($(CONFIG_FRAME_WARN),0) KBUILD_CFLAGS += $(call cc-option,-Wframe-larger-than=${CONFIG_FRAME_WARN}) endif -# This selects the stack protector compiler flag. Testing it is delayed -# until after .config has been reprocessed, in the prepare-compiler-check -# target. -ifdef CONFIG_CC_STACKPROTECTOR_AUTO - stackp-flag := $(call cc-option,-fstack-protector-strong,$(call cc-option,-fstack-protector)) - stackp-name := AUTO -else -ifdef CONFIG_CC_STACKPROTECTOR_REGULAR - stackp-flag := -fstack-protector - stackp-name := REGULAR -else -ifdef CONFIG_CC_STACKPROTECTOR_STRONG - stackp-flag := -fstack-protector-strong - stackp-name := STRONG -else - # If either there is no stack protector for this architecture or - # CONFIG_CC_STACKPROTECTOR_NONE is selected, we're done, and $(stackp-name) - # is empty, skipping all remaining stack protector tests. - # - # Force off for distro compilers that enable stack protector by default. - KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector) -endif -endif -endif -# Find arch-specific stack protector compiler sanity-checking script. -ifdef stackp-name -ifneq ($(stackp-flag),) - stackp-path := $(srctree)/scripts/gcc-$(SRCARCH)_$(BITS)-has-stack-protector.sh - stackp-check := $(wildcard $(stackp-path)) - # If the wildcard test matches a test script, run it to check functionality. - ifdef stackp-check - ifneq ($(shell $(CONFIG_SHELL) $(stackp-check) $(CC) $(KBUILD_CPPFLAGS) $(biarch)),y) - stackp-broken := y - endif - endif - ifndef stackp-broken - # If the stack protector is functional, enable code that depends on it. - KBUILD_CPPFLAGS += -DCONFIG_CC_STACKPROTECTOR - # Either we've already detected the flag (for AUTO) or we'll fail the - # build in the prepare-compiler-check rule (for specific flag). - KBUILD_CFLAGS += $(stackp-flag) - else - # We have to make sure stack protector is unconditionally disabled if - # the compiler is broken (in case we're going to continue the build in - # AUTO mode). - KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector) - endif -endif -endif +stackp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector +stackp-flags-$(CONFIG_STACKPROTECTOR) := -fstack-protector +stackp-flags-$(CONFIG_STACKPROTECTOR_STRONG) := -fstack-protector-strong + +KBUILD_CFLAGS += $(stackp-flags-y) ifeq ($(cc-name),clang) KBUILD_CPPFLAGS += $(call cc-option,-Qunused-arguments,) @@ -802,13 +768,12 @@ KBUILD_CFLAGS += $(call cc-option, -fno-inline-functions-called-once) endif ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION -KBUILD_CFLAGS += $(call cc-option,-ffunction-sections,) -KBUILD_CFLAGS += $(call cc-option,-fdata-sections,) +KBUILD_CFLAGS_KERNEL += $(call cc-option,-ffunction-sections,) +KBUILD_CFLAGS_KERNEL += $(call cc-option,-fdata-sections,) endif # arch Makefile may override CC so keep this after arch Makefile is included -NOSTDINC_FLAGS += -nostdinc -isystem $(call shell-cached,$(CC) -print-file-name=include) -CHECKFLAGS += $(NOSTDINC_FLAGS) +NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include) # warn about C99 declaration after statement KBUILD_CFLAGS += $(call cc-option,-Wdeclaration-after-statement,) @@ -878,6 +843,12 @@ ifeq ($(CONFIG_STRIP_ASM_SYMS),y) LDFLAGS_vmlinux += $(call ld-option, -X,) endif +# insure the checker run with the right endianness +CHECKFLAGS += $(if $(CONFIG_CPU_BIG_ENDIAN),-mbig-endian,-mlittle-endian) + +# the checker needs the correct machine size +CHECKFLAGS += $(if $(CONFIG_64BIT),-m64,-m32) + # Default kernel image to build when no specific target is given. # KBUILD_IMAGE may be overruled on the command line or # set in the environment @@ -1103,7 +1074,7 @@ endif # prepare2 creates a makefile if using a separate output directory. # From this point forward, .config has been reprocessed, so any rules # that need to depend on updated CONFIG_* values can be checked here. -prepare2: prepare3 prepare-compiler-check outputmakefile asm-generic +prepare2: prepare3 outputmakefile asm-generic prepare1: prepare2 $(version_h) $(autoksyms_h) include/generated/utsrelease.h \ include/config/auto.conf @@ -1129,43 +1100,6 @@ uapi-asm-generic: PHONY += prepare-objtool prepare-objtool: $(objtool_target) -# Check for CONFIG flags that require compiler support. Abort the build -# after .config has been processed, but before the kernel build starts. -# -# For security-sensitive CONFIG options, we don't want to fallback and/or -# silently change which compiler flags will be used, since that leads to -# producing kernels with different security feature characteristics -# depending on the compiler used. (For example, "But I selected -# CC_STACKPROTECTOR_STRONG! Why did it build with _REGULAR?!") -PHONY += prepare-compiler-check -prepare-compiler-check: FORCE -# Make sure compiler supports requested stack protector flag. -ifdef stackp-name - # Warn about CONFIG_CC_STACKPROTECTOR_AUTO having found no option. - ifeq ($(stackp-flag),) - @echo CONFIG_CC_STACKPROTECTOR_$(stackp-name): \ - Compiler does not support any known stack-protector >&2 - else - # Fail if specifically requested stack protector is missing. - ifeq ($(call cc-option, $(stackp-flag)),) - @echo Cannot use CONFIG_CC_STACKPROTECTOR_$(stackp-name): \ - $(stackp-flag) not supported by compiler >&2 && exit 1 - endif - endif -endif -# Make sure compiler does not have buggy stack-protector support. If a -# specific stack-protector was requested, fail the build, otherwise warn. -ifdef stackp-broken - ifeq ($(stackp-name),AUTO) - @echo CONFIG_CC_STACKPROTECTOR_$(stackp-name): \ - $(stackp-flag) available but compiler is broken: disabling >&2 - else - @echo Cannot use CONFIG_CC_STACKPROTECTOR_$(stackp-name): \ - $(stackp-flag) available but compiler is broken >&2 && exit 1 - endif -endif - @: - # Generate some files # --------------------------------------------------------------------------- @@ -1625,7 +1559,6 @@ clean: $(clean-dirs) -o -name '*.asn1.[ch]' \ -o -name '*.symtypes' -o -name 'modules.order' \ -o -name modules.builtin -o -name '.tmp_*.o.*' \ - -o -name .cache.mk \ -o -name '*.c.[012]*.*' \ -o -name '*.ll' \ -o -name '*.gcno' \) -type f -print | xargs rm -f @@ -1763,7 +1696,7 @@ quiet_cmd_rmfiles = $(if $(wildcard $(rm-files)),CLEAN $(wildcard $(rm-files)) # Run depmod only if we have System.map and depmod is executable quiet_cmd_depmod = DEPMOD $(KERNELRELEASE) cmd_depmod = $(CONFIG_SHELL) $(srctree)/scripts/depmod.sh $(DEPMOD) \ - $(KERNELRELEASE) "$(patsubst y,_,$(CONFIG_HAVE_UNDERSCORE_SYMBOL_PREFIX))" + $(KERNELRELEASE) # Create temporary dir for module support files # clean it up only when building all modules diff --git a/arch/Kconfig b/arch/Kconfig index 75dd23acf133..1aa59063f1fd 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -272,15 +272,19 @@ config HAVE_REGS_AND_STACK_ACCESS_API declared in asm/ptrace.h For example the kprobes-based event tracer needs this API. +config HAVE_RSEQ + bool + depends on HAVE_REGS_AND_STACK_ACCESS_API + help + This symbol should be selected by an architecture if it + supports an implementation of restartable sequences. + config HAVE_CLK bool help The calls support software clock gating and thus are a key power management tool on many systems. -config HAVE_DMA_API_DEBUG - bool - config HAVE_HW_BREAKPOINT bool depends on PERF_EVENTS @@ -399,7 +403,16 @@ config SECCOMP_FILTER in terms of Berkeley Packet Filter programs which implement task-defined system call filtering polices. - See Documentation/prctl/seccomp_filter.txt for details. + See Documentation/userspace-api/seccomp_filter.rst for details. + +preferred-plugin-hostcc := $(if-success,[ $(gcc-version) -ge 40800 ],$(HOSTCXX),$(HOSTCC)) + +config PLUGIN_HOSTCC + string + default "$(shell,$(srctree)/scripts/gcc-plugin.sh "$(preferred-plugin-hostcc)" "$(HOSTCXX)" "$(CC)")" + help + Host compiler used to build GCC plugins. This can be $(HOSTCXX), + $(HOSTCC), or a null string if GCC plugin is unsupported. config HAVE_GCC_PLUGINS bool @@ -410,7 +423,7 @@ config HAVE_GCC_PLUGINS menuconfig GCC_PLUGINS bool "GCC plugins" depends on HAVE_GCC_PLUGINS - depends on !COMPILE_TEST + depends on PLUGIN_HOSTCC != "" help GCC plugins are loadable modules that provide extra features to the compiler. They are useful for runtime instrumentation and static analysis. @@ -420,7 +433,7 @@ menuconfig GCC_PLUGINS config GCC_PLUGIN_CYC_COMPLEXITY bool "Compute the cyclomatic complexity of a function" if EXPERT depends on GCC_PLUGINS - depends on !COMPILE_TEST + depends on !COMPILE_TEST # too noisy help The complexity M of a function's control flow graph is defined as: M = E - N + 2P @@ -480,6 +493,7 @@ config GCC_PLUGIN_STRUCTLEAK config GCC_PLUGIN_STRUCTLEAK_BYREF_ALL bool "Force initialize all struct type variables passed by reference" depends on GCC_PLUGIN_STRUCTLEAK + depends on !COMPILE_TEST help Zero initialize any struct type local variable that may be passed by reference without having been initialized. @@ -487,7 +501,7 @@ config GCC_PLUGIN_STRUCTLEAK_BYREF_ALL config GCC_PLUGIN_STRUCTLEAK_VERBOSE bool "Report forcefully initialized variables" depends on GCC_PLUGIN_STRUCTLEAK - depends on !COMPILE_TEST + depends on !COMPILE_TEST # too noisy help This option will cause a warning to be printed each time the structleak plugin finds a variable it thinks needs to be @@ -527,7 +541,7 @@ config GCC_PLUGIN_RANDSTRUCT config GCC_PLUGIN_RANDSTRUCT_PERFORMANCE bool "Use cacheline-aware structure randomization" depends on GCC_PLUGIN_RANDSTRUCT - depends on !COMPILE_TEST + depends on !COMPILE_TEST # do not reduce test coverage help If you say Y here, the RANDSTRUCT randomization will make a best effort at restricting randomization to cacheline-sized @@ -535,17 +549,20 @@ config GCC_PLUGIN_RANDSTRUCT_PERFORMANCE in structures. This reduces the performance hit of RANDSTRUCT at the cost of weakened randomization. -config HAVE_CC_STACKPROTECTOR +config HAVE_STACKPROTECTOR bool help An arch should select this symbol if: - - its compiler supports the -fstack-protector option - it has implemented a stack canary (e.g. __stack_chk_guard) -choice - prompt "Stack Protector buffer overflow detection" - depends on HAVE_CC_STACKPROTECTOR - default CC_STACKPROTECTOR_AUTO +config CC_HAS_STACKPROTECTOR_NONE + def_bool $(cc-option,-fno-stack-protector) + +config STACKPROTECTOR + bool "Stack Protector buffer overflow detection" + depends on HAVE_STACKPROTECTOR + depends on $(cc-option,-fstack-protector) + default y help This option turns on the "stack-protector" GCC feature. This feature puts, at the beginning of functions, a canary value on @@ -555,14 +572,6 @@ choice overwrite the canary, which gets detected and the attack is then neutralized via a kernel panic. -config CC_STACKPROTECTOR_NONE - bool "None" - help - Disable "stack-protector" GCC feature. - -config CC_STACKPROTECTOR_REGULAR - bool "Regular" - help Functions will have the stack-protector canary logic added if they have an 8-byte or larger character array on the stack. @@ -573,8 +582,11 @@ config CC_STACKPROTECTOR_REGULAR about 3% of all kernel functions, which increases kernel code size by about 0.3%. -config CC_STACKPROTECTOR_STRONG - bool "Strong" +config STACKPROTECTOR_STRONG + bool "Strong Stack Protector" + depends on STACKPROTECTOR + depends on $(cc-option,-fstack-protector-strong) + default y help Functions will have the stack-protector canary logic added in any of the following conditions: @@ -592,29 +604,6 @@ config CC_STACKPROTECTOR_STRONG about 20% of all kernel functions, which increases the kernel code size by about 2%. -config CC_STACKPROTECTOR_AUTO - bool "Automatic" - help - If the compiler supports it, the best available stack-protector - option will be chosen. - -endchoice - -config LD_DEAD_CODE_DATA_ELIMINATION - bool - help - Select this if the architecture wants to do dead code and - data elimination with the linker by compiling with - -ffunction-sections -fdata-sections and linking with - --gc-sections. - - This requires that the arch annotates or otherwise protects - its external entry points from being discarded. Linker scripts - must also merge .text.*, .data.*, and .bss.* correctly into - output sections. Care must be taken not to pull in unrelated - sections (e.g., '.text.init'). Typically '.' in section names - is used to distinguish them from label names / C identifiers. - config HAVE_ARCH_WITHIN_STACK_FRAMES bool help @@ -690,12 +679,6 @@ config MODULES_USE_ELF_REL Modules only use ELF REL relocations. Modules with ELF RELA relocations will give an error. -config HAVE_UNDERSCORE_SYMBOL_PREFIX - bool - help - Some architectures generate an _ in front of C symbols; things like - module loading and assembly files need to know about this. - config HAVE_IRQ_EXIT_ON_IRQ_STACK bool help @@ -874,6 +857,21 @@ config OLD_SIGACTION config COMPAT_OLD_SIGACTION bool +config 64BIT_TIME + def_bool ARCH_HAS_64BIT_TIME + help + This should be selected by all architectures that need to support + new system calls with a 64-bit time_t. This is relevant on all 32-bit + architectures, and 64-bit architectures as part of compat syscall + handling. + +config COMPAT_32BIT_TIME + def_bool (!64BIT && 64BIT_TIME) || COMPAT + help + This enables 32 bit time_t support in addition to 64 bit time_t support. + This is relevant on all 32-bit architectures, and 64-bit architectures + as part of compat syscall handling. + config ARCH_NO_COHERENT_DMA_MMAP bool diff --git a/arch/alpha/Kconfig b/arch/alpha/Kconfig index f19dc31288c8..0c4805a572c8 100644 --- a/arch/alpha/Kconfig +++ b/arch/alpha/Kconfig @@ -10,6 +10,8 @@ config ALPHA select HAVE_OPROFILE select HAVE_PCSPKR_PLATFORM select HAVE_PERF_EVENTS + select NEED_DMA_MAP_STATE + select NEED_SG_DMA_LENGTH select VIRT_TO_BUS select GENERIC_IRQ_PROBE select AUTO_IRQ_AFFINITY if SMP @@ -64,15 +66,6 @@ config ZONE_DMA bool default y -config ARCH_DMA_ADDR_T_64BIT - def_bool y - -config NEED_DMA_MAP_STATE - def_bool y - -config NEED_SG_DMA_LENGTH - def_bool y - config GENERIC_ISA_DMA bool default y @@ -346,9 +339,6 @@ config PCI_DOMAINS config PCI_SYSCALL def_bool PCI -config IOMMU_HELPER - def_bool PCI - config ALPHA_NONAME bool depends on ALPHA_BOOK1 || ALPHA_NONAME_CH @@ -586,7 +576,7 @@ config ARCH_DISCONTIGMEM_ENABLE Say Y to support efficient handling of discontiguous physical memory, for architectures which are either NUMA (Non-Uniform Memory Access) or have huge holes in the physical address space for other reasons. - See for more. + See for more. source "mm/Kconfig" diff --git a/arch/alpha/Makefile b/arch/alpha/Makefile index 2cc3cc519c54..c5ec8c09c0c6 100644 --- a/arch/alpha/Makefile +++ b/arch/alpha/Makefile @@ -11,7 +11,7 @@ NM := $(NM) -B LDFLAGS_vmlinux := -static -N #-relax -CHECKFLAGS += -D__alpha__ -m64 +CHECKFLAGS += -D__alpha__ cflags-y := -pipe -mno-fp-regs -ffixed-8 cflags-y += $(call cc-option, -fno-jump-tables) diff --git a/arch/alpha/include/asm/Kbuild b/arch/alpha/include/asm/Kbuild index 9b68790013e2..0580cb8c84b2 100644 --- a/arch/alpha/include/asm/Kbuild +++ b/arch/alpha/include/asm/Kbuild @@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 +generic-y += compat.h generic-y += exec.h generic-y += export.h generic-y += fb.h diff --git a/arch/alpha/include/asm/pci.h b/arch/alpha/include/asm/pci.h index b9ec55351924..cf6bc1e64d66 100644 --- a/arch/alpha/include/asm/pci.h +++ b/arch/alpha/include/asm/pci.h @@ -56,11 +56,6 @@ struct pci_controller { /* IOMMU controls. */ -/* The PCI address space does not equal the physical memory address space. - The networking and block device layers use this boolean for bounce buffer - decisions. */ -#define PCI_DMA_BUS_IS_PHYS 0 - /* TODO: integrate with include/asm-generic/pci.h ? */ static inline int pci_get_legacy_ide_irq(struct pci_dev *dev, int channel) { diff --git a/arch/alpha/include/uapi/asm/Kbuild b/arch/alpha/include/uapi/asm/Kbuild index 9afaba5e5503..1a5b75310cf4 100644 --- a/arch/alpha/include/uapi/asm/Kbuild +++ b/arch/alpha/include/uapi/asm/Kbuild @@ -2,4 +2,8 @@ include include/uapi/asm-generic/Kbuild.asm generic-y += bpf_perf_event.h +generic-y += ipcbuf.h +generic-y += msgbuf.h generic-y += poll.h +generic-y += sembuf.h +generic-y += shmbuf.h diff --git a/arch/alpha/include/uapi/asm/ipcbuf.h b/arch/alpha/include/uapi/asm/ipcbuf.h deleted file mode 100644 index 90d6445a14df..000000000000 --- a/arch/alpha/include/uapi/asm/ipcbuf.h +++ /dev/null @@ -1,2 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#include diff --git a/arch/alpha/include/uapi/asm/msgbuf.h b/arch/alpha/include/uapi/asm/msgbuf.h deleted file mode 100644 index 8c5d4d8c1b16..000000000000 --- a/arch/alpha/include/uapi/asm/msgbuf.h +++ /dev/null @@ -1,28 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _ALPHA_MSGBUF_H -#define _ALPHA_MSGBUF_H - -/* - * The msqid64_ds structure for alpha architecture. - * Note extra padding because this structure is passed back and forth - * between kernel and user space. - * - * Pad space is left for: - * - 2 miscellaneous 64-bit values - */ - -struct msqid64_ds { - struct ipc64_perm msg_perm; - __kernel_time_t msg_stime; /* last msgsnd time */ - __kernel_time_t msg_rtime; /* last msgrcv time */ - __kernel_time_t msg_ctime; /* last change time */ - unsigned long msg_cbytes; /* current number of bytes on queue */ - unsigned long msg_qnum; /* number of messages in queue */ - unsigned long msg_qbytes; /* max number of bytes on queue */ - __kernel_pid_t msg_lspid; /* pid of last msgsnd */ - __kernel_pid_t msg_lrpid; /* last receive pid */ - unsigned long __unused1; - unsigned long __unused2; -}; - -#endif /* _ALPHA_MSGBUF_H */ diff --git a/arch/alpha/include/uapi/asm/sembuf.h b/arch/alpha/include/uapi/asm/sembuf.h deleted file mode 100644 index f28ffa668b2f..000000000000 --- a/arch/alpha/include/uapi/asm/sembuf.h +++ /dev/null @@ -1,23 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _ALPHA_SEMBUF_H -#define _ALPHA_SEMBUF_H - -/* - * The semid64_ds structure for alpha architecture. - * Note extra padding because this structure is passed back and forth - * between kernel and user space. - * - * Pad space is left for: - * - 2 miscellaneous 64-bit values - */ - -struct semid64_ds { - struct ipc64_perm sem_perm; /* permissions .. see ipc.h */ - __kernel_time_t sem_otime; /* last semop time */ - __kernel_time_t sem_ctime; /* last change time */ - unsigned long sem_nsems; /* no. of semaphores in array */ - unsigned long __unused1; - unsigned long __unused2; -}; - -#endif /* _ALPHA_SEMBUF_H */ diff --git a/arch/alpha/include/uapi/asm/shmbuf.h b/arch/alpha/include/uapi/asm/shmbuf.h deleted file mode 100644 index 7e041ca2eb40..000000000000 --- a/arch/alpha/include/uapi/asm/shmbuf.h +++ /dev/null @@ -1,39 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _ALPHA_SHMBUF_H -#define _ALPHA_SHMBUF_H - -/* - * The shmid64_ds structure for alpha architecture. - * Note extra padding because this structure is passed back and forth - * between kernel and user space. - * - * Pad space is left for: - * - 2 miscellaneous 64-bit values - */ - -struct shmid64_ds { - struct ipc64_perm shm_perm; /* operation perms */ - size_t shm_segsz; /* size of segment (bytes) */ - __kernel_time_t shm_atime; /* last attach time */ - __kernel_time_t shm_dtime; /* last detach time */ - __kernel_time_t shm_ctime; /* last change time */ - __kernel_pid_t shm_cpid; /* pid of creator */ - __kernel_pid_t shm_lpid; /* pid of last operator */ - unsigned long shm_nattch; /* no. of current attaches */ - unsigned long __unused1; - unsigned long __unused2; -}; - -struct shminfo64 { - unsigned long shmmax; - unsigned long shmmin; - unsigned long shmmni; - unsigned long shmseg; - unsigned long shmall; - unsigned long __unused1; - unsigned long __unused2; - unsigned long __unused3; - unsigned long __unused4; -}; - -#endif /* _ALPHA_SHMBUF_H */ diff --git a/arch/alpha/include/uapi/asm/siginfo.h b/arch/alpha/include/uapi/asm/siginfo.h index 0cf3b527b274..db3f0138536f 100644 --- a/arch/alpha/include/uapi/asm/siginfo.h +++ b/arch/alpha/include/uapi/asm/siginfo.h @@ -7,18 +7,4 @@ #include -/* - * SIGFPE si_codes - */ -#ifdef __KERNEL__ -#define FPE_FIXME 0 /* Broken dup of SI_USER */ -#endif /* __KERNEL__ */ - -/* - * SIGTRAP si_codes - */ -#ifdef __KERNEL__ -#define TRAP_FIXME 0 /* Broken dup of SI_USER */ -#endif /* __KERNEL__ */ - #endif diff --git a/arch/alpha/kernel/osf_sys.c b/arch/alpha/kernel/osf_sys.c index 89faa6f4de47..6e921754c8fc 100644 --- a/arch/alpha/kernel/osf_sys.c +++ b/arch/alpha/kernel/osf_sys.c @@ -871,8 +871,7 @@ SYSCALL_DEFINE5(osf_setsysinfo, unsigned long, op, void __user *, buffer, send a signal. Old exceptions are not signaled. */ fex = (exc >> IEEE_STATUS_TO_EXCSUM_SHIFT) & swcr; if (fex) { - siginfo_t info; - int si_code = 0; + int si_code = FPE_FLTUNK; if (fex & IEEE_TRAP_ENABLE_DNO) si_code = FPE_FLTUND; if (fex & IEEE_TRAP_ENABLE_INE) si_code = FPE_FLTRES; @@ -881,11 +880,9 @@ SYSCALL_DEFINE5(osf_setsysinfo, unsigned long, op, void __user *, buffer, if (fex & IEEE_TRAP_ENABLE_DZE) si_code = FPE_FLTDIV; if (fex & IEEE_TRAP_ENABLE_INV) si_code = FPE_FLTINV; - info.si_signo = SIGFPE; - info.si_errno = 0; - info.si_code = si_code; - info.si_addr = NULL; /* FIXME */ - send_sig_info(SIGFPE, &info, current); + send_sig_fault(SIGFPE, si_code, + (void __user *)NULL, /* FIXME */ + 0, current); } return 0; } diff --git a/arch/alpha/kernel/signal.c b/arch/alpha/kernel/signal.c index 9ebb3bcbc626..8c0c4ee0be6e 100644 --- a/arch/alpha/kernel/signal.c +++ b/arch/alpha/kernel/signal.c @@ -219,14 +219,8 @@ do_sigreturn(struct sigcontext __user *sc) /* Send SIGTRAP if we're single-stepping: */ if (ptrace_cancel_bpt (current)) { - siginfo_t info; - - info.si_signo = SIGTRAP; - info.si_errno = 0; - info.si_code = TRAP_BRKPT; - info.si_addr = (void __user *) regs->pc; - info.si_trapno = 0; - send_sig_info(SIGTRAP, &info, current); + send_sig_fault(SIGTRAP, TRAP_BRKPT, (void __user *) regs->pc, 0, + current); } return; @@ -253,14 +247,8 @@ do_rt_sigreturn(struct rt_sigframe __user *frame) /* Send SIGTRAP if we're single-stepping: */ if (ptrace_cancel_bpt (current)) { - siginfo_t info; - - info.si_signo = SIGTRAP; - info.si_errno = 0; - info.si_code = TRAP_BRKPT; - info.si_addr = (void __user *) regs->pc; - info.si_trapno = 0; - send_sig_info(SIGTRAP, &info, current); + send_sig_fault(SIGTRAP, TRAP_BRKPT, (void __user *) regs->pc, 0, + current); } return; diff --git a/arch/alpha/kernel/traps.c b/arch/alpha/kernel/traps.c index f43bd05dede2..bc9627698796 100644 --- a/arch/alpha/kernel/traps.c +++ b/arch/alpha/kernel/traps.c @@ -213,7 +213,6 @@ do_entArith(unsigned long summary, unsigned long write_mask, struct pt_regs *regs) { long si_code = FPE_FLTINV; - siginfo_t info; if (summary & 1) { /* Software-completion summary bit is set, so try to @@ -228,17 +227,12 @@ do_entArith(unsigned long summary, unsigned long write_mask, } die_if_kernel("Arithmetic fault", regs, 0, NULL); - info.si_signo = SIGFPE; - info.si_errno = 0; - info.si_code = si_code; - info.si_addr = (void __user *) regs->pc; - send_sig_info(SIGFPE, &info, current); + send_sig_fault(SIGFPE, si_code, (void __user *) regs->pc, 0, current); } asmlinkage void do_entIF(unsigned long type, struct pt_regs *regs) { - siginfo_t info; int signo, code; if ((regs->ps & ~IPL_MAX) == 0) { @@ -270,31 +264,20 @@ do_entIF(unsigned long type, struct pt_regs *regs) switch (type) { case 0: /* breakpoint */ - info.si_signo = SIGTRAP; - info.si_errno = 0; - info.si_code = TRAP_BRKPT; - info.si_trapno = 0; - info.si_addr = (void __user *) regs->pc; - if (ptrace_cancel_bpt(current)) { regs->pc -= 4; /* make pc point to former bpt */ } - send_sig_info(SIGTRAP, &info, current); + send_sig_fault(SIGTRAP, TRAP_BRKPT, (void __user *)regs->pc, 0, + current); return; case 1: /* bugcheck */ - info.si_signo = SIGTRAP; - info.si_errno = 0; - info.si_code = TRAP_FIXME; - info.si_addr = (void __user *) regs->pc; - info.si_trapno = 0; - send_sig_info(SIGTRAP, &info, current); + send_sig_fault(SIGTRAP, TRAP_UNK, (void __user *) regs->pc, 0, + current); return; case 2: /* gentrap */ - info.si_addr = (void __user *) regs->pc; - info.si_trapno = regs->r16; switch ((long) regs->r16) { case GEN_INTOVF: signo = SIGFPE; @@ -326,7 +309,7 @@ do_entIF(unsigned long type, struct pt_regs *regs) break; case GEN_ROPRAND: signo = SIGFPE; - code = FPE_FIXME; + code = FPE_FLTUNK; break; case GEN_DECOVF: @@ -348,15 +331,12 @@ do_entIF(unsigned long type, struct pt_regs *regs) case GEN_SUBRNG7: default: signo = SIGTRAP; - code = TRAP_FIXME; + code = TRAP_UNK; break; } - info.si_signo = signo; - info.si_errno = 0; - info.si_code = code; - info.si_addr = (void __user *) regs->pc; - send_sig_info(signo, &info, current); + send_sig_fault(signo, code, (void __user *) regs->pc, regs->r16, + current); return; case 4: /* opDEC */ @@ -380,11 +360,9 @@ do_entIF(unsigned long type, struct pt_regs *regs) if (si_code == 0) return; if (si_code > 0) { - info.si_signo = SIGFPE; - info.si_errno = 0; - info.si_code = si_code; - info.si_addr = (void __user *) regs->pc; - send_sig_info(SIGFPE, &info, current); + send_sig_fault(SIGFPE, si_code, + (void __user *) regs->pc, 0, + current); return; } } @@ -409,11 +387,7 @@ do_entIF(unsigned long type, struct pt_regs *regs) ; } - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_ILLOPC; - info.si_addr = (void __user *) regs->pc; - send_sig_info(SIGILL, &info, current); + send_sig_fault(SIGILL, ILL_ILLOPC, (void __user *)regs->pc, 0, current); } /* There is an ifdef in the PALcode in MILO that enables a @@ -426,15 +400,9 @@ do_entIF(unsigned long type, struct pt_regs *regs) asmlinkage void do_entDbg(struct pt_regs *regs) { - siginfo_t info; - die_if_kernel("Instruction fault", regs, 0, NULL); - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_ILLOPC; - info.si_addr = (void __user *) regs->pc; - force_sig_info(SIGILL, &info, current); + force_sig_fault(SIGILL, ILL_ILLOPC, (void __user *)regs->pc, 0, current); } @@ -758,7 +726,7 @@ do_entUnaUser(void __user * va, unsigned long opcode, unsigned long tmp1, tmp2, tmp3, tmp4; unsigned long fake_reg, *reg_addr = &fake_reg; - siginfo_t info; + int si_code; long error; /* Check the UAC bits to decide what the user wants us to do @@ -981,34 +949,27 @@ do_entUnaUser(void __user * va, unsigned long opcode, give_sigsegv: regs->pc -= 4; /* make pc point to faulting insn */ - info.si_signo = SIGSEGV; - info.si_errno = 0; /* We need to replicate some of the logic in mm/fault.c, since we don't have access to the fault code in the exception handling return path. */ if ((unsigned long)va >= TASK_SIZE) - info.si_code = SEGV_ACCERR; + si_code = SEGV_ACCERR; else { struct mm_struct *mm = current->mm; down_read(&mm->mmap_sem); if (find_vma(mm, (unsigned long)va)) - info.si_code = SEGV_ACCERR; + si_code = SEGV_ACCERR; else - info.si_code = SEGV_MAPERR; + si_code = SEGV_MAPERR; up_read(&mm->mmap_sem); } - info.si_addr = va; - send_sig_info(SIGSEGV, &info, current); + send_sig_fault(SIGSEGV, si_code, va, 0, current); return; give_sigbus: regs->pc -= 4; - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRALN; - info.si_addr = va; - send_sig_info(SIGBUS, &info, current); + send_sig_fault(SIGBUS, BUS_ADRALN, va, 0, current); return; } diff --git a/arch/alpha/mm/fault.c b/arch/alpha/mm/fault.c index cd3c572ee912..de2bd217adad 100644 --- a/arch/alpha/mm/fault.c +++ b/arch/alpha/mm/fault.c @@ -88,7 +88,6 @@ do_page_fault(unsigned long address, unsigned long mmcsr, struct mm_struct *mm = current->mm; const struct exception_table_entry *fixup; int fault, si_code = SEGV_MAPERR; - siginfo_t info; unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; /* As of EV6, a load into $31/$f31 is a prefetch, and never faults @@ -221,21 +220,13 @@ retry: up_read(&mm->mmap_sem); /* Send a sigbus, regardless of whether we were in kernel or user mode. */ - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRERR; - info.si_addr = (void __user *) address; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGBUS, BUS_ADRERR, (void __user *) address, 0, current); if (!user_mode(regs)) goto no_context; return; do_sigsegv: - info.si_signo = SIGSEGV; - info.si_errno = 0; - info.si_code = si_code; - info.si_addr = (void __user *) address; - force_sig_info(SIGSEGV, &info, current); + force_sig_fault(SIGSEGV, si_code, (void __user *) address, 0, current); return; #ifdef CONFIG_ALPHA_LARGE_VMALLOC diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig index d76bf4a83740..e81bcd271be7 100644 --- a/arch/arc/Kconfig +++ b/arch/arc/Kconfig @@ -9,11 +9,15 @@ config ARC def_bool y select ARC_TIMERS + select ARCH_HAS_SYNC_DMA_FOR_CPU + select ARCH_HAS_SYNC_DMA_FOR_DEVICE select ARCH_HAS_SG_CHAIN select ARCH_SUPPORTS_ATOMIC_RMW if ARC_HAS_LLSC select BUILDTIME_EXTABLE_SORT select CLONE_BACKWARDS select COMMON_CLK + select DMA_NONCOHERENT_OPS + select DMA_NONCOHERENT_MMAP select GENERIC_ATOMIC64 if !ISA_ARCV2 || !(ARC_HAS_LL64 && ARC_HAS_LLSC) select GENERIC_CLOCKEVENTS select GENERIC_FIND_FIRST_BIT @@ -44,6 +48,7 @@ config ARC select HAVE_GENERIC_DMA_COHERENT select HAVE_KERNEL_GZIP select HAVE_KERNEL_LZMA + select ARCH_HAS_PTE_SPECIAL config MIGHT_HAVE_PCI bool @@ -453,16 +458,11 @@ config ARC_HAS_PAE40 default n depends on ISA_ARCV2 select HIGHMEM + select PHYS_ADDR_T_64BIT help Enable access to physical memory beyond 4G, only supported on ARC cores with 40 bit Physical Addressing support -config ARCH_PHYS_ADDR_T_64BIT - def_bool ARC_HAS_PAE40 - -config ARCH_DMA_ADDR_T_64BIT - bool - config ARC_KVADDR_SIZE int "Kernel Virtual Address Space size (MB)" range 0 512 diff --git a/arch/arc/include/asm/Kbuild b/arch/arc/include/asm/Kbuild index 4bd5d4369e05..feed50ce89fa 100644 --- a/arch/arc/include/asm/Kbuild +++ b/arch/arc/include/asm/Kbuild @@ -1,7 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 generic-y += bugs.h +generic-y += compat.h generic-y += device.h generic-y += div64.h +generic-y += dma-mapping.h generic-y += emergency-restart.h generic-y += extable.h generic-y += fb.h diff --git a/arch/arc/include/asm/dma-mapping.h b/arch/arc/include/asm/dma-mapping.h deleted file mode 100644 index 7a16824bfe98..000000000000 --- a/arch/arc/include/asm/dma-mapping.h +++ /dev/null @@ -1,21 +0,0 @@ -/* - * DMA Mapping glue for ARC - * - * Copyright (C) 2004, 2007-2010, 2011-2012 Synopsys, Inc. (www.synopsys.com) - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ - -#ifndef ASM_ARC_DMA_MAPPING_H -#define ASM_ARC_DMA_MAPPING_H - -extern const struct dma_map_ops arc_dma_ops; - -static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus) -{ - return &arc_dma_ops; -} - -#endif diff --git a/arch/arc/include/asm/pci.h b/arch/arc/include/asm/pci.h index ba56c23c1b20..4ff53c041c64 100644 --- a/arch/arc/include/asm/pci.h +++ b/arch/arc/include/asm/pci.h @@ -16,12 +16,6 @@ #define PCIBIOS_MIN_MEM 0x100000 #define pcibios_assign_all_busses() 1 -/* - * The PCI address space does equal the physical memory address space. - * The networking and block device layers use this boolean for bounce - * buffer decisions. - */ -#define PCI_DMA_BUS_IS_PHYS 1 #endif /* __KERNEL__ */ diff --git a/arch/arc/include/asm/pgtable.h b/arch/arc/include/asm/pgtable.h index 08fe33830d4b..8ec5599a0957 100644 --- a/arch/arc/include/asm/pgtable.h +++ b/arch/arc/include/asm/pgtable.h @@ -320,8 +320,6 @@ PTE_BIT_FUNC(mkexec, |= (_PAGE_EXECUTE)); PTE_BIT_FUNC(mkspecial, |= (_PAGE_SPECIAL)); PTE_BIT_FUNC(mkhuge, |= (_PAGE_HW_SZ)); -#define __HAVE_ARCH_PTE_SPECIAL - static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) { return __pte((pte_val(pte) & _PAGE_CHG_MASK) | pgprot_val(newprot)); diff --git a/arch/arc/mm/dma.c b/arch/arc/mm/dma.c index 1dcc404b5aec..8c1071840979 100644 --- a/arch/arc/mm/dma.c +++ b/arch/arc/mm/dma.c @@ -16,13 +16,12 @@ * The default DMA address == Phy address which is 0x8000_0000 based. */ -#include +#include #include #include - -static void *arc_dma_alloc(struct device *dev, size_t size, - dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) +void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle, + gfp_t gfp, unsigned long attrs) { unsigned long order = get_order(size); struct page *page; @@ -89,7 +88,7 @@ static void *arc_dma_alloc(struct device *dev, size_t size, return kvaddr; } -static void arc_dma_free(struct device *dev, size_t size, void *vaddr, +void arch_dma_free(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_handle, unsigned long attrs) { phys_addr_t paddr = dma_handle; @@ -105,9 +104,9 @@ static void arc_dma_free(struct device *dev, size_t size, void *vaddr, __free_pages(page, get_order(size)); } -static int arc_dma_mmap(struct device *dev, struct vm_area_struct *vma, - void *cpu_addr, dma_addr_t dma_addr, size_t size, - unsigned long attrs) +int arch_dma_mmap(struct device *dev, struct vm_area_struct *vma, + void *cpu_addr, dma_addr_t dma_addr, size_t size, + unsigned long attrs) { unsigned long user_count = vma_pages(vma); unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT; @@ -130,149 +129,14 @@ static int arc_dma_mmap(struct device *dev, struct vm_area_struct *vma, return ret; } -/* - * streaming DMA Mapping API... - * CPU accesses page via normal paddr, thus needs to explicitly made - * consistent before each use - */ -static void _dma_cache_sync(phys_addr_t paddr, size_t size, - enum dma_data_direction dir) +void arch_sync_dma_for_device(struct device *dev, phys_addr_t paddr, + size_t size, enum dma_data_direction dir) { - switch (dir) { - case DMA_FROM_DEVICE: - dma_cache_inv(paddr, size); - break; - case DMA_TO_DEVICE: - dma_cache_wback(paddr, size); - break; - case DMA_BIDIRECTIONAL: - dma_cache_wback_inv(paddr, size); - break; - default: - pr_err("Invalid DMA dir [%d] for OP @ %pa[p]\n", dir, &paddr); - } + dma_cache_wback(paddr, size); } -/* - * arc_dma_map_page - map a portion of a page for streaming DMA - * - * Ensure that any data held in the cache is appropriately discarded - * or written back. - * - * The device owns this memory once this call has completed. The CPU - * can regain ownership by calling dma_unmap_page(). - * - * Note: while it takes struct page as arg, caller can "abuse" it to pass - * a region larger than PAGE_SIZE, provided it is physically contiguous - * and this still works correctly - */ -static dma_addr_t arc_dma_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs) +void arch_sync_dma_for_cpu(struct device *dev, phys_addr_t paddr, + size_t size, enum dma_data_direction dir) { - phys_addr_t paddr = page_to_phys(page) + offset; - - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - _dma_cache_sync(paddr, size, dir); - - return paddr; -} - -/* - * arc_dma_unmap_page - unmap a buffer previously mapped through dma_map_page() - * - * After this call, reads by the CPU to the buffer are guaranteed to see - * whatever the device wrote there. - * - * Note: historically this routine was not implemented for ARC - */ -static void arc_dma_unmap_page(struct device *dev, dma_addr_t handle, - size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - phys_addr_t paddr = handle; - - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - _dma_cache_sync(paddr, size, dir); + dma_cache_inv(paddr, size); } - -static int arc_dma_map_sg(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, unsigned long attrs) -{ - struct scatterlist *s; - int i; - - for_each_sg(sg, s, nents, i) - s->dma_address = dma_map_page(dev, sg_page(s), s->offset, - s->length, dir); - - return nents; -} - -static void arc_dma_unmap_sg(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, - unsigned long attrs) -{ - struct scatterlist *s; - int i; - - for_each_sg(sg, s, nents, i) - arc_dma_unmap_page(dev, sg_dma_address(s), sg_dma_len(s), dir, - attrs); -} - -static void arc_dma_sync_single_for_cpu(struct device *dev, - dma_addr_t dma_handle, size_t size, enum dma_data_direction dir) -{ - _dma_cache_sync(dma_handle, size, DMA_FROM_DEVICE); -} - -static void arc_dma_sync_single_for_device(struct device *dev, - dma_addr_t dma_handle, size_t size, enum dma_data_direction dir) -{ - _dma_cache_sync(dma_handle, size, DMA_TO_DEVICE); -} - -static void arc_dma_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sglist, int nelems, - enum dma_data_direction dir) -{ - int i; - struct scatterlist *sg; - - for_each_sg(sglist, sg, nelems, i) - _dma_cache_sync(sg_phys(sg), sg->length, dir); -} - -static void arc_dma_sync_sg_for_device(struct device *dev, - struct scatterlist *sglist, int nelems, - enum dma_data_direction dir) -{ - int i; - struct scatterlist *sg; - - for_each_sg(sglist, sg, nelems, i) - _dma_cache_sync(sg_phys(sg), sg->length, dir); -} - -static int arc_dma_supported(struct device *dev, u64 dma_mask) -{ - /* Support 32 bit DMA mask exclusively */ - return dma_mask == DMA_BIT_MASK(32); -} - -const struct dma_map_ops arc_dma_ops = { - .alloc = arc_dma_alloc, - .free = arc_dma_free, - .mmap = arc_dma_mmap, - .map_page = arc_dma_map_page, - .unmap_page = arc_dma_unmap_page, - .map_sg = arc_dma_map_sg, - .unmap_sg = arc_dma_unmap_sg, - .sync_single_for_device = arc_dma_sync_single_for_device, - .sync_single_for_cpu = arc_dma_sync_single_for_cpu, - .sync_sg_for_cpu = arc_dma_sync_sg_for_cpu, - .sync_sg_for_device = arc_dma_sync_sg_for_device, - .dma_supported = arc_dma_supported, -}; -EXPORT_SYMBOL(arc_dma_ops); diff --git a/arch/arc/mm/fault.c b/arch/arc/mm/fault.c index a0b7bd6d030d..b884bbd6f354 100644 --- a/arch/arc/mm/fault.c +++ b/arch/arc/mm/fault.c @@ -70,6 +70,8 @@ void do_page_fault(unsigned long address, struct pt_regs *regs) int write = regs->ecr_cause & ECR_C_PROTV_STORE; /* ST/EX */ unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; + clear_siginfo(&info); + /* * We fault-in kernel-space virtual memory on-demand. The * 'reference' page table is init_mm.pgd. diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index a7f8e7f4b88f..54eeb8d00bc6 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -8,8 +8,10 @@ config ARM select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_ELF_RANDOMIZE select ARCH_HAS_FORTIFY_SOURCE - select ARCH_HAS_SET_MEMORY + select ARCH_HAS_KCOV + select ARCH_HAS_PTE_SPECIAL if ARM_LPAE select ARCH_HAS_PHYS_TO_DMA + select ARCH_HAS_SET_MEMORY select ARCH_HAS_STRICT_KERNEL_RWX if MMU && !XIP_KERNEL select ARCH_HAS_STRICT_MODULE_RWX if MMU select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST @@ -56,11 +58,9 @@ config ARM select HAVE_ARCH_TRACEHOOK select HAVE_ARM_SMCCC if CPU_V7 select HAVE_EBPF_JIT if !CPU_ENDIAN_BE32 - select HAVE_CC_STACKPROTECTOR select HAVE_CONTEXT_TRACKING select HAVE_C_RECORDMCOUNT select HAVE_DEBUG_KMEMLEAK - select HAVE_DMA_API_DEBUG select HAVE_DMA_CONTIGUOUS if MMU select HAVE_DYNAMIC_FTRACE if (!XIP_KERNEL) && !CPU_ENDIAN_BE32 && MMU select HAVE_DYNAMIC_FTRACE_WITH_REGS if HAVE_DYNAMIC_FTRACE @@ -91,11 +91,14 @@ config ARM select HAVE_PERF_USER_STACK_DUMP select HAVE_RCU_TABLE_FREE if (SMP && ARM_LPAE) select HAVE_REGS_AND_STACK_ACCESS_API + select HAVE_RSEQ + select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_UID16 select HAVE_VIRT_CPU_ACCOUNTING_GEN select IRQ_FORCED_THREADING select MODULES_USE_ELF_REL + select NEED_DMA_MAP_STATE select NO_BOOTMEM select OF_EARLY_FLATTREE if OF select OF_RESERVED_MEM if OF @@ -119,9 +122,6 @@ config ARM_HAS_SG_CHAIN select ARCH_HAS_SG_CHAIN bool -config NEED_SG_DMA_LENGTH - bool - config ARM_DMA_USE_IOMMU bool select ARM_HAS_SG_CHAIN @@ -224,9 +224,6 @@ config ARCH_MAY_HAVE_PC_FDC config ZONE_DMA bool -config NEED_DMA_MAP_STATE - def_bool y - config ARCH_SUPPORTS_UPROBES def_bool y @@ -1305,7 +1302,7 @@ config SMP will run faster if you say N here. See also , - and the SMP-HOWTO available at + and the SMP-HOWTO available at . If you don't know what to do here, say N. @@ -1467,7 +1464,7 @@ config ARM_PSCI config ARCH_NR_GPIO int default 2048 if ARCH_SOCFPGA - default 1024 if ARCH_BRCMSTB || ARCH_SHMOBILE || ARCH_TEGRA || \ + default 1024 if ARCH_BRCMSTB || ARCH_RENESAS || ARCH_TEGRA || \ ARCH_ZYNQ default 512 if ARCH_EXYNOS || ARCH_KEYSTONE || SOC_OMAP5 || \ SOC_DRA7XX || ARCH_S3C24XX || ARCH_S3C64XX || ARCH_S5PV210 @@ -1704,6 +1701,7 @@ config ARCH_WANT_GENERAL_HUGETLB config ARM_MODULE_PLTS bool "Use PLTs to allow module memory to spill over into vmalloc area" depends on MODULES + default y help Allocate PLTs when loading modules so that jumps and calls whose targets are too far away for their relative offsets to be encoded @@ -1714,7 +1712,8 @@ config ARM_MODULE_PLTS rounding up to page size, the actual memory footprint is usually the same. - Say y if you are getting out of memory errors while loading modules + Disabling this is usually safe for small single-platform + configurations. If unsure, say y. source "mm/Kconfig" @@ -1778,12 +1777,6 @@ config SECCOMP and the task is only allowed to execute a few safe syscalls defined by each seccomp mode. -config SWIOTLB - def_bool y - -config IOMMU_HELPER - def_bool SWIOTLB - config PARAVIRT bool "Enable paravirtualization code" help @@ -1815,6 +1808,7 @@ config XEN depends on MMU select ARCH_DMA_ADDR_T_64BIT select ARM_PSCI + select SWIOTLB select SWIOTLB_XEN select PARAVIRT help diff --git a/arch/arm/Kconfig.debug b/arch/arm/Kconfig.debug index 199ebc1c4538..693f84392f1b 100644 --- a/arch/arm/Kconfig.debug +++ b/arch/arm/Kconfig.debug @@ -942,6 +942,13 @@ choice via SCIF0 on Renesas RZ/G1M (R8A7743), R-Car H2 (R8A7790), M2-W (R8A7791), V2H (R8A7792), or M2-N (R8A7793). + config DEBUG_RCAR_GEN2_SCIF1 + bool "Kernel low-level debugging messages via SCIF1 on R8A77470" + depends on ARCH_R8A77470 + help + Say Y here if you want kernel low-level debugging support + via SCIF1 on Renesas RZ/G1C (R8A77470). + config DEBUG_RCAR_GEN2_SCIF2 bool "Kernel low-level debugging messages via SCIF2 on R8A7794" depends on ARCH_R8A7794 @@ -1495,6 +1502,7 @@ config DEBUG_LL_INCLUDE default "debug/renesas-scif.S" if DEBUG_RCAR_GEN1_SCIF0 default "debug/renesas-scif.S" if DEBUG_RCAR_GEN1_SCIF2 default "debug/renesas-scif.S" if DEBUG_RCAR_GEN2_SCIF0 + default "debug/renesas-scif.S" if DEBUG_RCAR_GEN2_SCIF1 default "debug/renesas-scif.S" if DEBUG_RCAR_GEN2_SCIF2 default "debug/renesas-scif.S" if DEBUG_RCAR_GEN2_SCIF4 default "debug/renesas-scif.S" if DEBUG_RMOBILE_SCIFA0 @@ -1617,6 +1625,7 @@ config DEBUG_UART_PHYS default 0xe6c80000 if DEBUG_RMOBILE_SCIFA4 default 0xe6e58000 if DEBUG_RCAR_GEN2_SCIF2 default 0xe6e60000 if DEBUG_RCAR_GEN2_SCIF0 + default 0xe6e68000 if DEBUG_RCAR_GEN2_SCIF1 default 0xe6ee0000 if DEBUG_RCAR_GEN2_SCIF4 default 0xe8008000 if DEBUG_R7S72100_SCIF2 default 0xf0000be0 if ARCH_EBSA110 @@ -1651,8 +1660,8 @@ config DEBUG_UART_PHYS DEBUG_NETX_UART || \ DEBUG_QCOM_UARTDM || DEBUG_R7S72100_SCIF2 || \ DEBUG_RCAR_GEN1_SCIF0 || DEBUG_RCAR_GEN1_SCIF2 || \ - DEBUG_RCAR_GEN2_SCIF0 || DEBUG_RCAR_GEN2_SCIF2 || \ - DEBUG_RCAR_GEN2_SCIF4 || \ + DEBUG_RCAR_GEN2_SCIF0 || DEBUG_RCAR_GEN2_SCIF1 || \ + DEBUG_RCAR_GEN2_SCIF2 || DEBUG_RCAR_GEN2_SCIF4 || \ DEBUG_RMOBILE_SCIFA0 || DEBUG_RMOBILE_SCIFA1 || \ DEBUG_RMOBILE_SCIFA4 || DEBUG_S3C24XX_UART || \ DEBUG_S3C64XX_UART || \ diff --git a/arch/arm/Makefile b/arch/arm/Makefile index e4e537f27339..fc26c3d7b9b6 100644 --- a/arch/arm/Makefile +++ b/arch/arm/Makefile @@ -106,7 +106,7 @@ tune-$(CONFIG_CPU_V6K) =$(call cc-option,-mtune=arm1136j-s,-mtune=strongarm) tune-y := $(tune-y) ifeq ($(CONFIG_AEABI),y) -CFLAGS_ABI :=-mabi=aapcs-linux -mno-thumb-interwork -mfpu=vfp +CFLAGS_ABI :=-mabi=aapcs-linux -mfpu=vfp else CFLAGS_ABI :=$(call cc-option,-mapcs-32,-mabi=apcs-gnu) $(call cc-option,-mno-thumb-interwork,) endif @@ -135,7 +135,7 @@ endif KBUILD_CFLAGS +=$(CFLAGS_ABI) $(CFLAGS_ISA) $(arch-y) $(tune-y) $(call cc-option,-mshort-load-bytes,$(call cc-option,-malignment-traps,)) -msoft-float -Uarm KBUILD_AFLAGS +=$(CFLAGS_ABI) $(AFLAGS_ISA) $(arch-y) $(tune-y) -include asm/unified.h -msoft-float -CHECKFLAGS += -D__arm__ -m32 +CHECKFLAGS += -D__arm__ #Default value head-y := arch/arm/kernel/head$(MMUEXT).o @@ -212,7 +212,7 @@ machine-$(CONFIG_ARCH_S3C24XX) += s3c24xx machine-$(CONFIG_ARCH_S3C64XX) += s3c64xx machine-$(CONFIG_ARCH_S5PV210) += s5pv210 machine-$(CONFIG_ARCH_SA1100) += sa1100 -machine-$(CONFIG_ARCH_SHMOBILE) += shmobile +machine-$(CONFIG_ARCH_RENESAS) += shmobile machine-$(CONFIG_ARCH_SIRF) += prima2 machine-$(CONFIG_ARCH_SOCFPGA) += socfpga machine-$(CONFIG_ARCH_STI) += sti diff --git a/arch/arm/boot/compressed/Makefile b/arch/arm/boot/compressed/Makefile index 6a4e7341ecd3..1f5a5ffe7fcf 100644 --- a/arch/arm/boot/compressed/Makefile +++ b/arch/arm/boot/compressed/Makefile @@ -25,6 +25,9 @@ endif GCOV_PROFILE := n +# Prevents link failures: __sanitizer_cov_trace_pc() is not linked in. +KCOV_INSTRUMENT := n + # # Architecture dependencies # @@ -113,7 +116,7 @@ CFLAGS_fdt_ro.o := $(nossp_flags) CFLAGS_fdt_rw.o := $(nossp_flags) CFLAGS_fdt_wip.o := $(nossp_flags) -ccflags-y := -fpic -mno-single-pic-base -fno-builtin -I$(obj) +ccflags-y := -fpic $(call cc-option,-mno-single-pic-base,) -fno-builtin -I$(obj) asflags-y := -DZIMAGE # Supply kernel BSS size to the decompressor via a linker symbol. diff --git a/arch/arm/boot/dts/Makefile b/arch/arm/boot/dts/Makefile index 7e2424957809..37a3de760d40 100644 --- a/arch/arm/boot/dts/Makefile +++ b/arch/arm/boot/dts/Makefile @@ -75,6 +75,7 @@ dtb-$(CONFIG_ARCH_BCM2835) += \ bcm2835-rpi-a-plus.dtb \ bcm2836-rpi-2-b.dtb \ bcm2837-rpi-3-b.dtb \ + bcm2837-rpi-3-b-plus.dtb \ bcm2835-rpi-zero.dtb \ bcm2835-rpi-zero-w.dtb dtb-$(CONFIG_ARCH_BCM_5301X) += \ @@ -102,8 +103,10 @@ dtb-$(CONFIG_ARCH_BCM_5301X) += \ bcm47094-dlink-dir-885l.dtb \ bcm47094-linksys-panamera.dtb \ bcm47094-luxul-abr-4500.dtb \ + bcm47094-luxul-xap-1610.dtb \ bcm47094-luxul-xbr-4500.dtb \ bcm47094-luxul-xwr-3100.dtb \ + bcm47094-luxul-xwr-3150-v1.dtb \ bcm47094-netgear-r8500.dtb \ bcm94708.dtb \ bcm94709.dtb \ @@ -140,6 +143,7 @@ dtb-$(CONFIG_ARCH_BCM_NSP) += \ dtb-$(CONFIG_ARCH_BERLIN) += \ berlin2-sony-nsz-gs7.dtb \ berlin2cd-google-chromecast.dtb \ + berlin2cd-valve-steamlink.dtb \ berlin2q-marvell-dmp.dtb dtb-$(CONFIG_ARCH_BRCMSTB) += \ bcm7445-bcm97445svmb.dtb @@ -190,8 +194,6 @@ dtb-$(CONFIG_ARCH_EXYNOS5) += \ exynos5422-odroidxu3.dtb \ exynos5422-odroidxu3-lite.dtb \ exynos5422-odroidxu4.dtb \ - exynos5440-sd5v1.dtb \ - exynos5440-ssdk5440.dtb \ exynos5800-peach-pi.dtb dtb-$(CONFIG_ARCH_GEMINI) += \ gemini-dlink-dir-685.dtb \ @@ -312,14 +314,14 @@ dtb-$(CONFIG_ARCH_NPCM7XX) += \ dtb-$(CONFIG_MACH_MESON6) += \ meson6-atv1200.dtb dtb-$(CONFIG_MACH_MESON8) += \ - meson8-minix-neo-x8.dtb + meson8-minix-neo-x8.dtb \ + meson8b-mxq.dtb \ + meson8b-odroidc1.dtb \ + meson8m2-mxiii-plus.dtb dtb-$(CONFIG_ARCH_MMP) += \ pxa168-aspenite.dtb \ pxa910-dkb.dtb \ mmp2-brownstone.dtb -dtb-$(CONFIG_MACH_MESON8B) += \ - meson8b-mxq.dtb \ - meson8b-odroidc1.dtb dtb-$(CONFIG_ARCH_MPS2) += \ mps2-an385.dtb \ mps2-an399.dtb @@ -399,6 +401,7 @@ dtb-$(CONFIG_SOC_IMX6Q) += \ imx6dl-hummingboard2-som-v15.dtb \ imx6dl-icore.dtb \ imx6dl-icore-rqs.dtb \ + imx6dl-mamoj.dtb \ imx6dl-nit6xlite.dtb \ imx6dl-nitrogen6x.dtb \ imx6dl-phytec-mira-rdk-nand.dtb \ @@ -439,6 +442,7 @@ dtb-$(CONFIG_SOC_IMX6Q) += \ imx6q-cubox-i-emmc-som-v15.dtb \ imx6q-cubox-i-som-v15.dtb \ imx6q-dfi-fs700-m60.dtb \ + imx6q-dhcom-pdk2.dtb \ imx6q-display5-tianma-tm070-1280x768.dtb \ imx6q-dmo-edmqmx6.dtb \ imx6q-dms-ba16.dtb \ @@ -463,9 +467,11 @@ dtb-$(CONFIG_SOC_IMX6Q) += \ imx6q-hummingboard2-emmc-som-v15.dtb \ imx6q-hummingboard2-som-v15.dtb \ imx6q-icore.dtb \ + imx6q-icore-mipi.dtb \ imx6q-icore-ofcap10.dtb \ imx6q-icore-ofcap12.dtb \ imx6q-icore-rqs.dtb \ + imx6q-kp-tpc.dtb \ imx6q-marsboard.dtb \ imx6q-mccmon6.dtb \ imx6q-nitrogen6x.dtb \ @@ -688,6 +694,7 @@ dtb-$(CONFIG_SOC_AM33XX) += \ am335x-pdu001.dtb \ am335x-pepper.dtb \ am335x-phycore-rdk.dtb \ + am335x-pocketbeagle.dtb \ am335x-shc.dtb \ am335x-sbc-t335.dtb \ am335x-sl50.dtb \ @@ -760,12 +767,17 @@ dtb-$(CONFIG_ARCH_QCOM) += \ qcom-apq8084-ifc6540.dtb \ qcom-apq8084-mtp.dtb \ qcom-ipq4019-ap.dk01.1-c1.dtb \ + qcom-ipq4019-ap.dk04.1-c1.dtb \ + qcom-ipq4019-ap.dk04.1-c3.dtb \ + qcom-ipq4019-ap.dk07.1-c1.dtb \ + qcom-ipq4019-ap.dk07.1-c2.dtb \ qcom-ipq8064-ap148.dtb \ qcom-msm8660-surf.dtb \ qcom-msm8960-cdp.dtb \ qcom-msm8974-fairphone-fp2.dtb \ qcom-msm8974-lge-nexus5-hammerhead.dtb \ qcom-msm8974-samsung-klte.dtb \ + qcom-msm8974-sony-xperia-amami.dtb \ qcom-msm8974-sony-xperia-castor.dtb \ qcom-msm8974-sony-xperia-honami.dtb \ qcom-mdm9615-wp8548-mangoh-green.dtb @@ -795,6 +807,7 @@ dtb-$(CONFIG_ARCH_RENESAS) += \ r8a7745-iwg22d-sodimm.dtb \ r8a7745-iwg22d-sodimm-dbhd-ca.dtb \ r8a7745-sk-rzg1e.dtb \ + r8a77470-iwg23s-sbc.dtb \ r8a7778-bockw.dtb \ r8a7779-marzen.dtb \ r8a7790-lager.dtb \ @@ -959,6 +972,7 @@ dtb-$(CONFIG_MACH_SUN7I) += \ sun7i-a20-m3.dtb \ sun7i-a20-mk808c.dtb \ sun7i-a20-olimex-som-evb.dtb \ + sun7i-a20-olimex-som-evb-emmc.dtb \ sun7i-a20-olimex-som204-evb.dtb \ sun7i-a20-olimex-som204-evb-emmc.dtb \ sun7i-a20-olinuxino-lime.dtb \ @@ -992,8 +1006,9 @@ dtb-$(CONFIG_MACH_SUN8I) += \ sun8i-a83t-bananapi-m3.dtb \ sun8i-a83t-cubietruck-plus.dtb \ sun8i-a83t-tbs-a711.dtb \ - sun8i-h2-plus-orangepi-r1.dtb \ sun8i-h2-plus-bananapi-m2-zero.dtb \ + sun8i-h2-plus-libretech-all-h3-cc.dtb \ + sun8i-h2-plus-orangepi-r1.dtb \ sun8i-h2-plus-orangepi-zero.dtb \ sun8i-h3-bananapi-m2-plus.dtb \ sun8i-h3-beelink-x2.dtb \ @@ -1010,6 +1025,8 @@ dtb-$(CONFIG_MACH_SUN8I) += \ sun8i-h3-orangepi-plus.dtb \ sun8i-h3-orangepi-plus2e.dtb \ sun8i-r16-bananapi-m2m.dtb \ + sun8i-r16-nintendo-nes-classic.dtb \ + sun8i-r16-nintendo-super-nes-classic.dtb \ sun8i-r16-parrot.dtb \ sun8i-r40-bananapi-m2-ultra.dtb \ sun8i-v3s-licheepi-zero.dtb \ @@ -1054,9 +1071,7 @@ dtb-$(CONFIG_ARCH_U8500) += \ ste-hrefprev60-stuib.dtb \ ste-hrefprev60-tvk.dtb \ ste-hrefv60plus-stuib.dtb \ - ste-hrefv60plus-tvk.dtb \ - ste-ccu8540.dtb \ - ste-ccu9540.dtb + ste-hrefv60plus-tvk.dtb dtb-$(CONFIG_ARCH_UNIPHIER) += \ uniphier-ld4-ref.dtb \ uniphier-ld6b-ref.dtb \ @@ -1150,6 +1165,9 @@ dtb-$(CONFIG_ARCH_MEDIATEK) += \ mt6580-evbp1.dtb \ mt6589-aquaris5.dtb \ mt6592-evb.dtb \ + mt7623a-rfb-emmc.dtb \ + mt7623a-rfb-nand.dtb \ + mt7623n-rfb-emmc.dtb \ mt7623n-rfb-nand.dtb \ mt7623n-bananapi-bpi-r2.dtb \ mt8127-moose.dtb \ @@ -1158,8 +1176,11 @@ dtb-$(CONFIG_ARCH_ZX) += zx296702-ad1.dtb dtb-$(CONFIG_ARCH_ASPEED) += \ aspeed-ast2500-evb.dtb \ aspeed-bmc-arm-centriq2400-rep.dtb \ + aspeed-bmc-intel-s2600wf.dtb \ + aspeed-bmc-opp-lanyang.dtb \ aspeed-bmc-opp-palmetto.dtb \ aspeed-bmc-opp-romulus.dtb \ aspeed-bmc-opp-witherspoon.dtb \ aspeed-bmc-opp-zaius.dtb \ + aspeed-bmc-portwell-neptune.dtb \ aspeed-bmc-quanta-q71l.dtb diff --git a/arch/arm/boot/dts/am335x-baltos-ir3220.dts b/arch/arm/boot/dts/am335x-baltos-ir3220.dts index 46df1b22022c..1b215c425c57 100644 --- a/arch/arm/boot/dts/am335x-baltos-ir3220.dts +++ b/arch/arm/boot/dts/am335x-baltos-ir3220.dts @@ -85,7 +85,7 @@ gpio-controller; #gpio-cells = <2>; interrupt-parent = <&gpio0>; - interrupts = <20 GPIO_ACTIVE_LOW>; + interrupts = <20 IRQ_TYPE_EDGE_RISING>; pinctrl-names = "default"; pinctrl-0 = <&tca6416_pins>; }; diff --git a/arch/arm/boot/dts/am335x-baltos-ir5221.dts b/arch/arm/boot/dts/am335x-baltos-ir5221.dts index 5d56355ba040..832ead864dc5 100644 --- a/arch/arm/boot/dts/am335x-baltos-ir5221.dts +++ b/arch/arm/boot/dts/am335x-baltos-ir5221.dts @@ -94,7 +94,7 @@ gpio-controller; #gpio-cells = <2>; interrupt-parent = <&gpio0>; - interrupts = <20 GPIO_ACTIVE_LOW>; + interrupts = <20 IRQ_TYPE_EDGE_RISING>; pinctrl-names = "default"; pinctrl-0 = <&tca6416_pins>; }; diff --git a/arch/arm/boot/dts/am335x-baltos.dtsi b/arch/arm/boot/dts/am335x-baltos.dtsi index ec6052c521ef..ed7a5a3daa42 100644 --- a/arch/arm/boot/dts/am335x-baltos.dtsi +++ b/arch/arm/boot/dts/am335x-baltos.dtsi @@ -249,7 +249,7 @@ gpio-controller; #gpio-cells = <2>; interrupt-parent = <&gpio1>; - interrupts = <28 GPIO_ACTIVE_LOW>; + interrupts = <28 IRQ_TYPE_EDGE_RISING>; pinctrl-names = "default"; pinctrl-0 = <&tps65910_pins>; }; diff --git a/arch/arm/boot/dts/am335x-bone-common.dtsi b/arch/arm/boot/dts/am335x-bone-common.dtsi index e67b4d65c8d0..f9e8667f5886 100644 --- a/arch/arm/boot/dts/am335x-bone-common.dtsi +++ b/arch/arm/boot/dts/am335x-bone-common.dtsi @@ -161,7 +161,14 @@ mmc1_pins: pinmux_mmc1_pins { pinctrl-single,pins = < - AM33XX_IOPAD(0x960, PIN_INPUT | MUX_MODE7) /* GPIO0_6 */ + AM33XX_IOPAD(0x960, PIN_INPUT | MUX_MODE7) /* spio0_cs1.gpio0_6 */ + AM33XX_IOPAD(0x8fc, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat0.mmc0_dat0 */ + AM33XX_IOPAD(0x8f8, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat1.mmc0_dat1 */ + AM33XX_IOPAD(0x8f4, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat2.mmc0_dat2 */ + AM33XX_IOPAD(0x8f0, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat3.mmc0_dat3 */ + AM33XX_IOPAD(0x904, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_cmd.mmc0_cmd */ + AM33XX_IOPAD(0x900, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_clk.mmc0_clk */ + AM33XX_IOPAD(0x9a0, PIN_INPUT | MUX_MODE4) /* mcasp0_aclkr.mmc0_sdwp */ >; }; diff --git a/arch/arm/boot/dts/am335x-boneblue.dts b/arch/arm/boot/dts/am335x-boneblue.dts index 58baee158e64..7bcd72691f06 100644 --- a/arch/arm/boot/dts/am335x-boneblue.dts +++ b/arch/arm/boot/dts/am335x-boneblue.dts @@ -364,7 +364,7 @@ compatible = "invensense,mpu9250"; reg = <0x68>; interrupt-parent = <&gpio3>; - interrupts = <21 GPIO_ACTIVE_LOW>; + interrupts = <21 IRQ_TYPE_EDGE_RISING>; i2c-gate { #address-cells = <1>; #size-cells = <0>; diff --git a/arch/arm/boot/dts/am335x-evm.dts b/arch/arm/boot/dts/am335x-evm.dts index fee6b3ee1741..1356fd6f8da3 100644 --- a/arch/arm/boot/dts/am335x-evm.dts +++ b/arch/arm/boot/dts/am335x-evm.dts @@ -303,7 +303,14 @@ mmc1_pins: pinmux_mmc1_pins { pinctrl-single,pins = < - AM33XX_IOPAD(0x960, PIN_INPUT | MUX_MODE7) /* spi0_cs1.gpio0_6 */ + AM33XX_IOPAD(0x960, PIN_INPUT | MUX_MODE7) /* spi0_cs1.gpio0_6 */ + AM33XX_IOPAD(0x8fc, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat0.mmc0_dat0 */ + AM33XX_IOPAD(0x8f8, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat1.mmc0_dat1 */ + AM33XX_IOPAD(0x8f4, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat2.mmc0_dat2 */ + AM33XX_IOPAD(0x8f0, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat3.mmc0_dat3 */ + AM33XX_IOPAD(0x904, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_cmd.mmc0_cmd */ + AM33XX_IOPAD(0x900, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_clk.mmc0_clk */ + AM33XX_IOPAD(0x9a0, PIN_INPUT | MUX_MODE4) /* mcasp0_aclkr.mmc0_sdwp */ >; }; diff --git a/arch/arm/boot/dts/am335x-evmsk.dts b/arch/arm/boot/dts/am335x-evmsk.dts index fa608cd5dc14..0c096a795e37 100644 --- a/arch/arm/boot/dts/am335x-evmsk.dts +++ b/arch/arm/boot/dts/am335x-evmsk.dts @@ -137,7 +137,7 @@ }; }; - backlight { + lcd_bl: backlight { compatible = "pwm-backlight"; pwms = <&ecap2 0 50000 PWM_POLARITY_INVERTED>; brightness-levels = <0 58 61 66 75 90 125 170 255>; @@ -172,6 +172,7 @@ pinctrl-names = "default", "sleep"; pinctrl-0 = <&lcd_pins_default>; pinctrl-1 = <&lcd_pins_sleep>; + backlight = <&lcd_bl>; status = "okay"; panel-info { ac-bias = <255>; @@ -399,7 +400,14 @@ mmc1_pins: pinmux_mmc1_pins { pinctrl-single,pins = < - AM33XX_IOPAD(0x960, PIN_INPUT | MUX_MODE7) /* spi0_cs1.gpio0_6 */ + AM33XX_IOPAD(0x960, PIN_INPUT | MUX_MODE7) /* spi0_cs1.gpio0_6 */ + AM33XX_IOPAD(0x8fc, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat0.mmc0_dat0 */ + AM33XX_IOPAD(0x8f8, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat1.mmc0_dat1 */ + AM33XX_IOPAD(0x8f4, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat2.mmc0_dat2 */ + AM33XX_IOPAD(0x8f0, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_dat3.mmc0_dat3 */ + AM33XX_IOPAD(0x904, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_cmd.mmc0_cmd */ + AM33XX_IOPAD(0x900, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc0_clk.mmc0_clk */ + AM33XX_IOPAD(0x9a0, PIN_INPUT | MUX_MODE4) /* mcasp0_aclkr.mmc0_sdwp */ >; }; diff --git a/arch/arm/boot/dts/am335x-osd335x-common.dtsi b/arch/arm/boot/dts/am335x-osd335x-common.dtsi new file mode 100644 index 000000000000..f8ff473f94f0 --- /dev/null +++ b/arch/arm/boot/dts/am335x-osd335x-common.dtsi @@ -0,0 +1,124 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2012 Texas Instruments Incorporated - http://www.ti.com/ + * + * Author: Robert Nelson + */ + +/ { + cpus { + cpu@0 { + cpu0-supply = <&dcdc2_reg>; + }; + }; + + memory@80000000 { + device_type = "memory"; + reg = <0x80000000 0x20000000>; /* 512 MB */ + }; +}; + +&cpu0_opp_table { + /* + * Octavo Systems: + * The EFUSE_SMA register is not programmed for any of the AM335x wafers + * we get and we are not programming them during our production test. + * Therefore, from a DEVICE_ID revision point of view, the silicon looks + * like it is Revision 2.1. However, from an EFUSE_SMA point of view for + * the HW OPP table, the silicon looks like it is Revision 1.0 (ie the + * EFUSE_SMA register reads as all zeros). + */ + oppnitro-1000000000 { + opp-supported-hw = <0x06 0x0100>; + }; +}; + +&am33xx_pinmux { + i2c0_pins: pinmux-i2c0-pins { + pinctrl-single,pins = < + AM33XX_IOPAD(0x988, PIN_INPUT_PULLUP | MUX_MODE0) /* (C17) I2C0_SDA.I2C0_SDA */ + AM33XX_IOPAD(0x98c, PIN_INPUT_PULLUP | MUX_MODE0) /* (C16) I2C0_SCL.I2C0_SCL */ + >; + }; +}; + +&i2c0 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c0_pins>; + + status = "okay"; + clock-frequency = <400000>; + + tps: tps@24 { + reg = <0x24>; + }; +}; + +/include/ "tps65217.dtsi" + +&tps { + interrupts = <7>; /* NMI */ + interrupt-parent = <&intc>; + + ti,pmic-shutdown-controller; + + pwrbutton { + interrupts = <2>; + status = "okay"; + }; + + regulators { + dcdc1_reg: regulator@0 { + regulator-name = "vdds_dpr"; + regulator-always-on; + }; + + dcdc2_reg: regulator@1 { + /* VDD_MPU voltage limits 0.95V - 1.26V with +/-4% tolerance */ + regulator-name = "vdd_mpu"; + regulator-min-microvolt = <925000>; + regulator-max-microvolt = <1351500>; + regulator-boot-on; + regulator-always-on; + }; + + dcdc3_reg: regulator@2 { + /* VDD_CORE voltage limits 0.95V - 1.1V with +/-4% tolerance */ + regulator-name = "vdd_core"; + regulator-min-microvolt = <925000>; + regulator-max-microvolt = <1150000>; + regulator-boot-on; + regulator-always-on; + }; + + ldo1_reg: regulator@3 { + regulator-name = "vio,vrtc,vdds"; + regulator-always-on; + }; + + ldo2_reg: regulator@4 { + regulator-name = "vdd_3v3aux"; + regulator-always-on; + }; + + ldo3_reg: regulator@5 { + regulator-name = "vdd_1v8"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-always-on; + }; + + ldo4_reg: regulator@6 { + regulator-name = "vdd_3v3a"; + regulator-always-on; + }; + }; +}; + +&aes { + status = "okay"; +}; + +&sham { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/am335x-pocketbeagle.dts b/arch/arm/boot/dts/am335x-pocketbeagle.dts new file mode 100644 index 000000000000..62fe5cab9fae --- /dev/null +++ b/arch/arm/boot/dts/am335x-pocketbeagle.dts @@ -0,0 +1,237 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2012 Texas Instruments Incorporated - http://www.ti.com/ + * + * Author: Robert Nelson + */ +/dts-v1/; + +#include "am33xx.dtsi" +#include "am335x-osd335x-common.dtsi" + +/ { + model = "TI AM335x PocketBeagle"; + compatible = "ti,am335x-pocketbeagle", "ti,am335x-bone", "ti,am33xx"; + + chosen { + stdout-path = &uart0; + }; + + leds { + pinctrl-names = "default"; + pinctrl-0 = <&usr_leds_pins>; + + compatible = "gpio-leds"; + + usr0 { + label = "beaglebone:green:usr0"; + gpios = <&gpio1 21 GPIO_ACTIVE_HIGH>; + linux,default-trigger = "heartbeat"; + default-state = "off"; + }; + + usr1 { + label = "beaglebone:green:usr1"; + gpios = <&gpio1 22 GPIO_ACTIVE_HIGH>; + linux,default-trigger = "mmc0"; + default-state = "off"; + }; + + usr2 { + label = "beaglebone:green:usr2"; + gpios = <&gpio1 23 GPIO_ACTIVE_HIGH>; + linux,default-trigger = "cpu0"; + default-state = "off"; + }; + + usr3 { + label = "beaglebone:green:usr3"; + gpios = <&gpio1 24 GPIO_ACTIVE_HIGH>; + default-state = "off"; + }; + }; + + vmmcsd_fixed: fixedregulator0 { + compatible = "regulator-fixed"; + regulator-name = "vmmcsd_fixed"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + }; +}; + +&am33xx_pinmux { + i2c2_pins: pinmux-i2c2-pins { + pinctrl-single,pins = < + AM33XX_IOPAD(0x97c, PIN_INPUT_PULLUP | MUX_MODE3) /* (D17) uart1_rtsn.I2C2_SCL */ + AM33XX_IOPAD(0x978, PIN_INPUT_PULLUP | MUX_MODE3) /* (D18) uart1_ctsn.I2C2_SDA */ + >; + }; + + ehrpwm0_pins: pinmux-ehrpwm0-pins { + pinctrl-single,pins = < + AM33XX_IOPAD(0x990, PIN_OUTPUT_PULLDOWN | MUX_MODE1) /* (A13) mcasp0_aclkx.ehrpwm0A */ + >; + }; + + ehrpwm1_pins: pinmux-ehrpwm1-pins { + pinctrl-single,pins = < + AM33XX_IOPAD(0x848, PIN_OUTPUT_PULLDOWN | MUX_MODE6) /* (U14) gpmc_a2.ehrpwm1A */ + >; + }; + + mmc0_pins: pinmux-mmc0-pins { + pinctrl-single,pins = < + AM33XX_IOPAD(0x960, PIN_INPUT | MUX_MODE7) /* (C15) spi0_cs1.gpio0[6] */ + AM33XX_IOPAD(0x8fc, PIN_INPUT_PULLUP | MUX_MODE0) /* (G16) mmc0_dat0.mmc0_dat0 */ + AM33XX_IOPAD(0x8f8, PIN_INPUT_PULLUP | MUX_MODE0) /* (G15) mmc0_dat1.mmc0_dat1 */ + AM33XX_IOPAD(0x8f4, PIN_INPUT_PULLUP | MUX_MODE0) /* (F18) mmc0_dat2.mmc0_dat2 */ + AM33XX_IOPAD(0x8f0, PIN_INPUT_PULLUP | MUX_MODE0) /* (F17) mmc0_dat3.mmc0_dat3 */ + AM33XX_IOPAD(0x904, PIN_INPUT_PULLUP | MUX_MODE0) /* (G18) mmc0_cmd.mmc0_cmd */ + AM33XX_IOPAD(0x900, PIN_INPUT_PULLUP | MUX_MODE0) /* (G17) mmc0_clk.mmc0_clk */ + AM33XX_IOPAD(0x9a0, PIN_INPUT | MUX_MODE4) /* (B12) mcasp0_aclkr.mmc0_sdwp */ + >; + }; + + spi0_pins: pinmux-spi0-pins { + pinctrl-single,pins = < + AM33XX_IOPAD(0x950, PIN_INPUT_PULLUP | MUX_MODE0) /* (A17) spi0_sclk.spi0_sclk */ + AM33XX_IOPAD(0x954, PIN_INPUT_PULLUP | MUX_MODE0) /* (B17) spi0_d0.spi0_d0 */ + AM33XX_IOPAD(0x958, PIN_INPUT_PULLUP | MUX_MODE0) /* (B16) spi0_d1.spi0_d1 */ + AM33XX_IOPAD(0x95c, PIN_INPUT_PULLUP | MUX_MODE0) /* (A16) spi0_cs0.spi0_cs0 */ + >; + }; + + spi1_pins: pinmux-spi1-pins { + pinctrl-single,pins = < + AM33XX_IOPAD(0x964, PIN_INPUT_PULLUP | MUX_MODE4) /* (C18) eCAP0_in_PWM0_out.spi1_sclk */ + AM33XX_IOPAD(0x968, PIN_INPUT_PULLUP | MUX_MODE4) /* (E18) uart0_ctsn.spi1_d0 */ + AM33XX_IOPAD(0x96c, PIN_INPUT_PULLUP | MUX_MODE4) /* (E17) uart0_rtsn.spi1_d1 */ + AM33XX_IOPAD(0x9b0, PIN_INPUT_PULLUP | MUX_MODE4) /* (A15) xdma_event_intr0.spi1_cs1 */ + >; + }; + + usr_leds_pins: pinmux-usr-leds-pins { + pinctrl-single,pins = < + AM33XX_IOPAD(0x854, PIN_OUTPUT | MUX_MODE7) /* (V15) gpmc_a5.gpio1[21] - USR_LED_0 */ + AM33XX_IOPAD(0x858, PIN_OUTPUT | MUX_MODE7) /* (U15) gpmc_a6.gpio1[22] - USR_LED_1 */ + AM33XX_IOPAD(0x85c, PIN_OUTPUT | MUX_MODE7) /* (T15) gpmc_a7.gpio1[23] - USR_LED_2 */ + AM33XX_IOPAD(0x860, PIN_OUTPUT | MUX_MODE7) /* (V16) gpmc_a8.gpio1[24] - USR_LED_3 */ + >; + }; + + uart0_pins: pinmux-uart0-pins { + pinctrl-single,pins = < + AM33XX_IOPAD(0x970, PIN_INPUT_PULLUP | MUX_MODE0) /* (E15) uart0_rxd.uart0_rxd */ + AM33XX_IOPAD(0x974, PIN_OUTPUT_PULLDOWN | MUX_MODE0) /* (E16) uart0_txd.uart0_txd */ + >; + }; + + uart4_pins: pinmux-uart4-pins { + pinctrl-single,pins = < + AM33XX_IOPAD(0x870, PIN_INPUT_PULLUP | MUX_MODE6) /* (T17) gpmc_wait0.uart4_rxd */ + AM33XX_IOPAD(0x874, PIN_OUTPUT_PULLDOWN | MUX_MODE6) /* (U17) gpmc_wpn.uart4_txd */ + >; + }; +}; + +&epwmss0 { + status = "okay"; +}; + +&ehrpwm0 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&ehrpwm0_pins>; +}; + +&epwmss1 { + status = "okay"; +}; + +&ehrpwm1 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&ehrpwm1_pins>; +}; + +&i2c0 { + eeprom: eeprom@50 { + compatible = "atmel,24c256"; + reg = <0x50>; + }; +}; + +&i2c2 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c2_pins>; + + status = "okay"; + clock-frequency = <400000>; +}; + +&mmc1 { + status = "okay"; + vmmc-supply = <&vmmcsd_fixed>; + bus-width = <4>; + pinctrl-names = "default"; + pinctrl-0 = <&mmc0_pins>; + cd-gpios = <&gpio0 6 GPIO_ACTIVE_LOW>; +}; + +&rtc { + system-power-controller; +}; + +&tscadc { + status = "okay"; + adc { + ti,adc-channels = <0 1 2 3 4 5 6 7>; + ti,chan-step-avg = <16 16 16 16 16 16 16 16>; + ti,chan-step-opendelay = <0x98 0x98 0x98 0x98 0x98 0x98 0x98 0x98>; + ti,chan-step-sampledelay = <0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0>; + }; +}; + +&uart0 { + pinctrl-names = "default"; + pinctrl-0 = <&uart0_pins>; + + status = "okay"; +}; + +&uart4 { + pinctrl-names = "default"; + pinctrl-0 = <&uart4_pins>; + + status = "okay"; +}; + +&usb { + status = "okay"; +}; + +&usb_ctrl_mod { + status = "okay"; +}; + +&usb0_phy { + status = "okay"; +}; + +&usb0 { + status = "okay"; + dr_mode = "otg"; +}; + +&usb1_phy { + status = "okay"; +}; + +&usb1 { + status = "okay"; + dr_mode = "host"; +}; + +&cppi41dma { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/am3517-evm.dts b/arch/arm/boot/dts/am3517-evm.dts index 0e4a125f78e3..98aadb0f81c5 100644 --- a/arch/arm/boot/dts/am3517-evm.dts +++ b/arch/arm/boot/dts/am3517-evm.dts @@ -8,11 +8,17 @@ /dts-v1/; #include "am3517.dtsi" +#include "am3517-som.dtsi" +#include / { model = "TI AM3517 EVM (AM3517/05 TMDSEVM3517)"; compatible = "ti,am3517-evm", "ti,am3517", "ti,omap3"; + aliases { + display0 = &lcd0; + }; + memory@80000000 { device_type = "memory"; reg = <0x80000000 0x10000000>; /* 256 MB */ @@ -24,6 +30,144 @@ regulator-min-microvolt = <3300000>; regulator-max-microvolt = <3300000>; }; + + gpio-keys { + compatible = "gpio-keys-polled"; + poll-interval = <100>; + + user_pb { + label = "User Push Button"; + linux,code = ; + gpios = <&tca6416 5 GPIO_ACTIVE_LOW>; + }; + + user_sw_1 { + label = "User Switch 1"; + linux,code = ; + gpios = <&tca6416 8 GPIO_ACTIVE_LOW>; + }; + + user_sw_2 { + label = "User Switch 2"; + linux,code = ; + gpios = <&tca6416 9 GPIO_ACTIVE_LOW>; + }; + + user_sw_3 { + label = "User Switch 3"; + linux,code = ; + gpios = <&tca6416 10 GPIO_ACTIVE_LOW>; + }; + + user_sw_4 { + label = "User Switch 4"; + linux,code = ; + gpios = <&tca6416 11 GPIO_ACTIVE_LOW>; + }; + + user_sw_5 { + label = "User Switch 5"; + linux,code = ; + gpios = <&tca6416 12 GPIO_ACTIVE_LOW>; + }; + + user_sw_6 { + label = "User Switch 6"; + linux,code = ; + gpios = <&tca6416 13 GPIO_ACTIVE_LOW>; + }; + + user_sw_7 { + label = "User Switch 7"; + linux,code = ; + gpios = <&tca6416 14 GPIO_ACTIVE_LOW>; + }; + + user_sw_8 { + label = "User Switch 8"; + linux,code = ; + gpios = <&tca6416 15 GPIO_ACTIVE_LOW>; + }; + }; + + gpio-leds { + compatible = "gpio-leds"; + + pinctrl-names = "default"; + pinctrl-0 = <&leds_pins>; + + user_led_1 { + label = "am3517evm:green:user_led_1"; + gpios = <&tca6416 7 GPIO_ACTIVE_LOW>; + default-state = "on"; + }; + + user_led_2 { + label = "am3517evm:green:user_led_2"; + gpios = <&tca6416 6 GPIO_ACTIVE_LOW>; + default-state = "on"; + }; + + user_led_3 { + label = "am3517evm:green:user_led_3"; + gpios = <&gpio1 11 GPIO_ACTIVE_HIGH>; + linux,default-trigger = "mmc0"; /* SD/MMC card activity */ + }; + + user_led_4 { + label = "am3517evm:green:user_led_4"; + gpios = <&gpio1 31 GPIO_ACTIVE_HIGH>; + linux,default-trigger = "heartbeat"; + }; + }; + + lcd0: display@0 { + compatible = "panel-dpi"; + label = "15"; + status = "okay"; + pinctrl-names = "default"; + enable-gpios = <&gpio6 16 GPIO_ACTIVE_HIGH>; /* gpio176, lcd INI */ + + port { + lcd_in: endpoint { + remote-endpoint = <&dpi_out>; + }; + }; + + panel-timing { + clock-frequency = <9000000>; + hactive = <480>; + vactive = <272>; + hfront-porch = <3>; + hback-porch = <2>; + hsync-len = <42>; + vback-porch = <3>; + vfront-porch = <4>; + vsync-len = <11>; + hsync-active = <0>; + vsync-active = <0>; + de-active = <1>; + pixelclk-active = <1>; + }; + }; + + bl: backlight { + compatible = "pwm-backlight"; + pinctrl-names = "default"; + pinctrl-0 = <&backlight_pins>; + pwms = <&pwm11 0 5000000 0>; + brightness-levels = <0 10 20 30 40 50 60 70 80 90 100>; + default-brightness-level = <7>; + enable-gpios = <&gpio6 22 GPIO_ACTIVE_HIGH>; /* gpio_182 */ + }; + + pwm11: dmtimer-pwm@11 { + compatible = "ti,omap-dmtimer-pwm"; + pinctrl-names = "default"; + pinctrl-0 = <&pwm_pins>; + ti,timers = <&timer11>; + #pwm-cells = <3>; + }; }; &davinci_emac { @@ -34,12 +178,32 @@ status = "okay"; }; -&i2c1 { - clock-frequency = <400000>; +&dss { + status = "ok"; + + pinctrl-names = "default"; + pinctrl-0 = <&dss_dpi_pins>; + + vdds_dsi-supply = <&vdd_io_reg>; + vdda_video-supply = <&vdd_io_reg>; + + port { + dpi_out: endpoint { + remote-endpoint = <&lcd_in>; + data-lines = <16>; + }; + }; }; &i2c2 { clock-frequency = <400000>; + /* User DIP swithes [1:8] / User LEDS [1:2] */ + tca6416: gpio@21 { + compatible = "ti,tca6416"; + reg = <0x21>; + gpio-controller; + #gpio-cells = <2>; + }; }; &i2c3 { @@ -47,8 +211,13 @@ }; &mmc1 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&mmc1_pins>; vmmc-supply = <&vmmc_fixed>; bus-width = <4>; + wp-gpios = <&gpio4 30 GPIO_ACTIVE_HIGH>; /* gpio_126 */ + cd-gpios = <&gpio4 31 GPIO_ACTIVE_HIGH>; /* gpio_127 */ }; &mmc2 { @@ -59,3 +228,63 @@ status = "disabled"; }; +&omap3_pmx_core { + + leds_pins: pinmux_leds_pins { + pinctrl-single,pins = < + OMAP3_WKUP_IOPAD(0x2a24, PIN_OUTPUT_PULLUP | MUX_MODE4) /* jtag_emu0.gpio_11 */ + OMAP3_WKUP_IOPAD(0x2a26, PIN_OUTPUT_PULLUP | MUX_MODE4) /* jtag_emu1.gpio_31 */ + >; + }; + + mmc1_pins: pinmux_mmc1_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x2144, PIN_INPUT_PULLUP | MUX_MODE0) /* sdmmc1_clk.sdmmc1_clk */ + OMAP3_CORE1_IOPAD(0x2146, PIN_INPUT_PULLUP | MUX_MODE0) /* sdmmc1_cmd.sdmmc1_cmd */ + OMAP3_CORE1_IOPAD(0x2148, PIN_INPUT_PULLUP | MUX_MODE0) /* sdmmc1_dat0.sdmmc1_dat0 */ + OMAP3_CORE1_IOPAD(0x214a, PIN_INPUT_PULLUP | MUX_MODE0) /* sdmmc1_dat1.sdmmc1_dat1 */ + OMAP3_CORE1_IOPAD(0x214c, PIN_INPUT_PULLUP | MUX_MODE0) /* sdmmc1_dat2.sdmmc1_dat2 */ + OMAP3_CORE1_IOPAD(0x214e, PIN_INPUT_PULLUP | MUX_MODE0) /* sdmmc1_dat3.sdmmc1_dat3 */ + OMAP3_CORE1_IOPAD(0x2150, PIN_INPUT_PULLUP | MUX_MODE4) /* sdmmc1_dat4.gpio_126 */ + OMAP3_CORE1_IOPAD(0x2152, PIN_INPUT_PULLUP | MUX_MODE4) /* sdmmc1_dat5.gpio_127 */ + >; + }; + + pwm_pins: pinmux_pwm_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x21dc, PIN_OUTPUT | MUX_MODE1) /* mcspi2_cs0.gpt11_pwm */ + >; + }; + + backlight_pins: pinmux_backlight_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x21de, PIN_OUTPUT | MUX_MODE4) /* mcspi2_cs1.gpio_182 */ + >; + }; + + dss_dpi_pins: pinmux_dss_dpi_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x21d2, PIN_OUTPUT | MUX_MODE4) /* mcspi1_cs2.gpio_176 */ + OMAP3_CORE1_IOPAD(0x20d4, PIN_OUTPUT | MUX_MODE0) /* dss_pclk.dss_pclk */ + OMAP3_CORE1_IOPAD(0x20d6, PIN_OUTPUT | MUX_MODE0) /* dss_hsync.dss_hsync */ + OMAP3_CORE1_IOPAD(0x20d8, PIN_OUTPUT | MUX_MODE0) /* dss_vsync.dss_vsync */ + OMAP3_CORE1_IOPAD(0x20da, PIN_OUTPUT | MUX_MODE0) /* dss_acbias.dss_acbias */ + OMAP3_CORE1_IOPAD(0x20dc, PIN_OUTPUT | MUX_MODE0) /* dss_data0.dss_data0 */ + OMAP3_CORE1_IOPAD(0x20de, PIN_OUTPUT | MUX_MODE0) /* dss_data1.dss_data1 */ + OMAP3_CORE1_IOPAD(0x20e0, PIN_OUTPUT | MUX_MODE0) /* dss_data2.dss_data2 */ + OMAP3_CORE1_IOPAD(0x20e2, PIN_OUTPUT | MUX_MODE0) /* dss_data3.dss_data3 */ + OMAP3_CORE1_IOPAD(0x20e4, PIN_OUTPUT | MUX_MODE0) /* dss_data4.dss_data4 */ + OMAP3_CORE1_IOPAD(0x20e6, PIN_OUTPUT | MUX_MODE0) /* dss_data5.dss_data5 */ + OMAP3_CORE1_IOPAD(0x20e8, PIN_OUTPUT | MUX_MODE0) /* dss_data6.dss_data6 */ + OMAP3_CORE1_IOPAD(0x20ea, PIN_OUTPUT | MUX_MODE0) /* dss_data7.dss_data7 */ + OMAP3_CORE1_IOPAD(0x20ec, PIN_OUTPUT | MUX_MODE0) /* dss_data8.dss_data8 */ + OMAP3_CORE1_IOPAD(0x20ee, PIN_OUTPUT | MUX_MODE0) /* dss_data9.dss_data9 */ + OMAP3_CORE1_IOPAD(0x20f0, PIN_OUTPUT | MUX_MODE0) /* dss_data10.dss_data10 */ + OMAP3_CORE1_IOPAD(0x20f2, PIN_OUTPUT | MUX_MODE0) /* dss_data11.dss_data11 */ + OMAP3_CORE1_IOPAD(0x20f4, PIN_OUTPUT | MUX_MODE0) /* dss_data12.dss_data12 */ + OMAP3_CORE1_IOPAD(0x20f6, PIN_OUTPUT | MUX_MODE0) /* dss_data13.dss_data13 */ + OMAP3_CORE1_IOPAD(0x20f8, PIN_OUTPUT | MUX_MODE0) /* dss_data14.dss_data14 */ + OMAP3_CORE1_IOPAD(0x20fa, PIN_OUTPUT | MUX_MODE0) /* dss_data15.dss_data15 */ + >; + }; +}; diff --git a/arch/arm/boot/dts/am3517-som.dtsi b/arch/arm/boot/dts/am3517-som.dtsi new file mode 100644 index 000000000000..a6d5ff73c163 --- /dev/null +++ b/arch/arm/boot/dts/am3517-som.dtsi @@ -0,0 +1,142 @@ +/* + * Copyright (C) 2016 Derald D. Woods + * + * Based on am3517-evm.dts + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +/ { + cpus { + cpu@0 { + cpu0-supply = <&vdd_core_reg>; + }; + }; +}; + +&gpmc { + ranges = <0 0 0x30000000 0x1000000>; /* CS0: 16MB for NAND */ + + nand@0,0 { + compatible = "ti,omap2-nand"; + linux,mtd-name = "micron,mt29f4g16abchch"; + reg = <0 0 4>; /* CS0, offset 0, IO size 4 */ + nand-bus-width = <16>; + ti,nand-ecc-opt = "bch8"; + gpmc,sync-clk-ps = <0>; + gpmc,cs-on-ns = <0>; + gpmc,cs-rd-off-ns = <44>; + gpmc,cs-wr-off-ns = <44>; + gpmc,adv-on-ns = <6>; + gpmc,adv-rd-off-ns = <34>; + gpmc,adv-wr-off-ns = <44>; + gpmc,we-off-ns = <40>; + gpmc,oe-off-ns = <54>; + gpmc,access-ns = <64>; + gpmc,rd-cycle-ns = <82>; + gpmc,wr-cycle-ns = <82>; + gpmc,wr-access-ns = <40>; + gpmc,wr-data-mux-bus-ns = <0>; + gpmc,device-width = <2>; + #address-cells = <1>; + #size-cells = <1>; + }; +}; + +&i2c1 { + clock-frequency = <400000>; + + s35390a: s35390a@30 { + compatible = "sii,s35390a"; + reg = <0x30>; + + pinctrl-names = "default"; + pinctrl-0 = <&rtc_pins>; + interrupts-extended = <&gpio2 23 IRQ_TYPE_EDGE_FALLING>; /* gpio_55 */ + }; + + tps: tps65023@48 { + compatible = "ti,tps65023"; + reg = <0x48>; + + regulators { + vdd_core_reg: VDCDC1 { + regulator-name = "vdd_core"; + compatible = "regulator-fixed"; + regulator-always-on; + regulator-min-microvolt = <1200000>; + regulator-max-microvolt = <1200000>; + }; + + vdd_io_reg: VDCDC2 { + regulator-name = "vdd_io"; + compatible = "regulator-fixed"; + regulator-always-on; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + }; + + vdd_1v8_reg: VDCDC3 { + regulator-name = "vdd_1v8"; + compatible = "regulator-fixed"; + regulator-always-on; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + }; + + vdd_usb18_reg: LDO1 { + regulator-name = "vdd_usb18"; + compatible = "regulator-fixed"; + regulator-always-on; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + }; + + vdd_usb33_reg: LDO2 { + regulator-name = "vdd_usb33"; + compatible = "regulator-fixed"; + regulator-always-on; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + }; + }; + }; + + touchscreen: tsc2004@4b { + compatible = "ti,tsc2004"; + reg = <0x4b>; + + vio-supply = <&vdd_io_reg>; + + pinctrl-names = "default"; + pinctrl-0 = <&tsc2004_pins>; + interrupts-extended = <&gpio3 1 IRQ_TYPE_EDGE_RISING>; /* gpio_65 */ + + touchscreen-fuzz-x = <4>; + touchscreen-fuzz-y = <7>; + touchscreen-fuzz-pressure = <2>; + touchscreen-size-x = <480>; + touchscreen-size-y = <272>; + touchscreen-max-pressure = <2048>; + + ti,x-plate-ohms = <280>; + ti,esd-recovery-timeout-ms = <8000>; + }; +}; + +&omap3_pmx_core { + + rtc_pins: pinmux_rtc_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x20b6, PIN_INPUT_PULLUP | MUX_MODE4) /* gpmc_ncs4.gpio_55 */ + >; + }; + + tsc2004_pins: pinmux_tsc2004_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x20d2, PIN_INPUT | MUX_MODE4) /* gpmc_wait3.gpio_65 */ + >; + }; +}; diff --git a/arch/arm/boot/dts/am437x-cm-t43.dts b/arch/arm/boot/dts/am437x-cm-t43.dts index 3b9a94c274a7..bff5abe69bdb 100644 --- a/arch/arm/boot/dts/am437x-cm-t43.dts +++ b/arch/arm/boot/dts/am437x-cm-t43.dts @@ -203,7 +203,7 @@ tps65218: tps65218@24 { compatible = "ti,tps65218"; reg = <0x24>; - interrupts = ; /* NMIn */ + interrupts = ; /* NMIn */ interrupt-parent = <&gic>; interrupt-controller; #interrupt-cells = <2>; diff --git a/arch/arm/boot/dts/am437x-gp-evm.dts b/arch/arm/boot/dts/am437x-gp-evm.dts index 8fe95cd7232a..60414b1ca404 100644 --- a/arch/arm/boot/dts/am437x-gp-evm.dts +++ b/arch/arm/boot/dts/am437x-gp-evm.dts @@ -543,7 +543,7 @@ tps65218: tps65218@24 { reg = <0x24>; compatible = "ti,tps65218"; - interrupts = ; /* NMIn */ + interrupts = ; /* NMIn */ interrupt-controller; #interrupt-cells = <2>; diff --git a/arch/arm/boot/dts/am437x-sk-evm.dts b/arch/arm/boot/dts/am437x-sk-evm.dts index 4118802b7fea..440351ad0b80 100644 --- a/arch/arm/boot/dts/am437x-sk-evm.dts +++ b/arch/arm/boot/dts/am437x-sk-evm.dts @@ -15,6 +15,7 @@ #include #include #include +#include / { model = "TI AM437x SK EVM"; @@ -158,6 +159,22 @@ }; }; }; + + vmmcwl_fixed: fixedregulator-mmcwl { + /* + * WL_EN is not SDIO standard compliant. It is an out of band + * signal and hard to be dealt with in a standard way by the + * SDIO core driver. + * So modelling the WL_EN line as a regulator was a natural + * choice as the MMC core already deals with MMC supplies. + */ + compatible = "regulator-fixed"; + regulator-name = "vmmcwl_fixed"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + gpio = <&gpio4 8 GPIO_ACTIVE_HIGH>; + enable-active-high; + }; }; &am43xx_pinmux { @@ -418,6 +435,62 @@ AM4372_IOPAD(0xac4, PIN_OUTPUT | MUX_MODE0) /* usb0_drvvbus.usb0_drvvbus */ >; }; + + mmc3_pins_default: pinmux_mmc3_pins_default { + pinctrl-single,pins = < + AM4372_IOPAD(0x9f0, PIN_INPUT_PULLUP | MUX_MODE3) /* (AD21) cam1_data2.mmc2_clk */ + AM4372_IOPAD(0x9f4, PIN_INPUT_PULLUP | MUX_MODE3) /* (AE22) cam1_data3.mmc2_cmd */ + AM4372_IOPAD(0x9f8, PIN_INPUT_PULLUP | MUX_MODE3) /* (AD22) cam1_data4.mmc2_dat0 */ + AM4372_IOPAD(0x9fc, PIN_INPUT_PULLUP | MUX_MODE3) /* (AE23) cam1_data5.mmc2_dat1 */ + AM4372_IOPAD(0xa00, PIN_INPUT_PULLUP | MUX_MODE3) /* (AD23) cam1_data6.mmc2_dat2 */ + AM4372_IOPAD(0xa04, PIN_INPUT_PULLUP | MUX_MODE3) /* (AE24) cam1_data7.mmc2_dat3 */ + >; + }; + + mmc3_pins_sleep: pinmux_mmc3_pins_sleep { + pinctrl-single,pins = < + AM4372_IOPAD(0x9f0, PIN_INPUT_PULLDOWN | MUX_MODE7) /* (AD21) cam1_data2.mmc2_clk */ + AM4372_IOPAD(0x9f4, PIN_INPUT_PULLDOWN | MUX_MODE7) /* (AE22) cam1_data3.mmc2_cmd */ + AM4372_IOPAD(0x9f8, PIN_INPUT_PULLDOWN | MUX_MODE7) /* (AD22) cam1_data4.mmc2_dat0 */ + AM4372_IOPAD(0x9fc, PIN_INPUT_PULLDOWN | MUX_MODE7) /* (AE23) cam1_data5.mmc2_dat1 */ + AM4372_IOPAD(0xa00, PIN_INPUT_PULLDOWN | MUX_MODE7) /* (AD23) cam1_data6.mmc2_dat2 */ + AM4372_IOPAD(0xa04, PIN_INPUT_PULLDOWN | MUX_MODE7) /* (AE24) cam1_data7.mmc2_dat3 */ + >; + }; + + wlan_pins_default: pinmux_wlan_pins_default { + pinctrl-single,pins = < + AM4372_IOPAD(0x9d0, PIN_OUTPUT_PULLDOWN | MUX_MODE7) /* cam1_data8.gpio4_8 WL_EN */ + AM4372_IOPAD(0x9e4, PIN_INPUT | WAKEUP_ENABLE | MUX_MODE7) /* cam1_wen.gpio4_13 WL_IRQ */ + >; + }; + + wlan_pins_sleep: pinmux_wlan_pins_sleep { + pinctrl-single,pins = < + AM4372_IOPAD(0x9d0, PIN_OUTPUT_PULLDOWN | MUX_MODE7) /* cam1_data8.gpio4_8 WL_EN */ + AM4372_IOPAD(0x9e4, PIN_INPUT | WAKEUP_ENABLE | MUX_MODE7) /* cam1_wen.gpio4_13 WL_IRQ */ + >; + }; + + uart1_bt_pins_default: pinmux_uart1_bt_pins_default { + pinctrl-single,pins = < + AM4372_IOPAD(0x980, PIN_INPUT | MUX_MODE0) /* uart1_rxd.uart1_rxd */ + AM4372_IOPAD(0x984, PIN_OUTPUT_PULLDOWN | MUX_MODE0) /* uart1_txd.uart1_txd */ + AM4372_IOPAD(0x978, PIN_INPUT_PULLUP | MUX_MODE0) /* uart1_ctsn.uart1_ctsn */ + AM4372_IOPAD(0x97c, PIN_OUTPUT_PULLDOWN | MUX_MODE0) /* uart1_rtsn.uart1_rtsn */ + AM4372_IOPAD(0x9cc, PIN_OUTPUT_PULLDOWN | MUX_MODE7) /* cam1_data9.gpio4_7 BT_EN */ + >; + }; + + uart1_bt_pins_sleep: pinmux_uart1_bt_pins_sleep { + pinctrl-single,pins = < + AM4372_IOPAD(0x980, PIN_OUTPUT_PULLDOWN | MUX_MODE7) /* uart1_rxd.uart1_rxd */ + AM4372_IOPAD(0x984, PIN_OUTPUT_PULLDOWN | MUX_MODE7) /* uart1_txd.uart1_txd */ + AM4372_IOPAD(0x978, PIN_OUTPUT_PULLDOWN | MUX_MODE7) /* uart1_ctsn.uart1_ctsn */ + AM4372_IOPAD(0x97c, PIN_OUTPUT_PULLDOWN | MUX_MODE7) /* uart1_rtsn.uart1_rtsn */ + AM4372_IOPAD(0x9cc, PIN_OUTPUT_PULLUP | MUX_MODE7) /* cam1_data9.gpio4_7 BT_EN */ + >; + }; }; &i2c0 { @@ -581,6 +654,10 @@ status = "okay"; }; +&gpio4 { + status = "okay"; +}; + &gpio5 { status = "okay"; }; @@ -595,6 +672,44 @@ cd-gpios = <&gpio0 6 GPIO_ACTIVE_LOW>; }; +&uart1 { + status = "okay"; + pinctrl-names = "default", "sleep"; + pinctrl-0 = <&uart1_bt_pins_default>; + pinctrl-1 = <&uart1_bt_pins_sleep>; +}; + +&mmc3 { + status = "okay"; + /* + * these are on the crossbar and are outlined in the + * xbar-event-map element + */ + dmas = <&edma_xbar 30 0 1>, + <&edma_xbar 31 0 2>; + dma-names = "tx", "rx"; + vmmc-supply = <&vmmcwl_fixed>; + bus-width = <4>; + pinctrl-names = "default", "sleep"; + pinctrl-0 = <&mmc3_pins_default>; + pinctrl-1 = <&mmc3_pins_sleep>; + cap-power-off-card; + keep-power-in-suspend; + ti,non-removable; + + #address-cells = <1>; + #size-cells = <0>; + wlcore: wlcore@2 { + compatible = "ti,wl1835"; + pinctrl-names = "default", "sleep"; + pinctrl-0 = <&wlan_pins_default>; + pinctrl-1 = <&wlan_pins_sleep>; + reg = <2>; + interrupt-parent = <&gpio4>; + interrupts = <13 IRQ_TYPE_LEVEL_HIGH>; + }; +}; + &usb2_phy1 { status = "okay"; }; diff --git a/arch/arm/boot/dts/am43x-epos-evm.dts b/arch/arm/boot/dts/am43x-epos-evm.dts index a66941885c11..6502d3397653 100644 --- a/arch/arm/boot/dts/am43x-epos-evm.dts +++ b/arch/arm/boot/dts/am43x-epos-evm.dts @@ -595,7 +595,7 @@ tps65218: tps65218@24 { reg = <0x24>; compatible = "ti,tps65218"; - interrupts = ; /* NMIn */ + interrupts = ; /* NMIn */ interrupt-controller; #interrupt-cells = <2>; diff --git a/arch/arm/boot/dts/am571x-idk.dts b/arch/arm/boot/dts/am571x-idk.dts index a2555140babc..5bb9d68d6e90 100644 --- a/arch/arm/boot/dts/am571x-idk.dts +++ b/arch/arm/boot/dts/am571x-idk.dts @@ -10,6 +10,7 @@ #include "dra72x.dtsi" #include #include +#include "dra7-mmc-iodelay.dtsi" #include "dra72x-mmc-iodelay.dtsi" #include "am57xx-idk-common.dtsi" @@ -102,7 +103,7 @@ &mmc1 { pinctrl-names = "default", "hs", "sdr12", "sdr25", "sdr50", "ddr50", "sdr104"; - pinctrl-0 = <&mmc1_pins_default>; + pinctrl-0 = <&mmc1_pins_default_no_clk_pu>; pinctrl-1 = <&mmc1_pins_hs>; pinctrl-2 = <&mmc1_pins_sdr12>; pinctrl-3 = <&mmc1_pins_sdr25>; @@ -112,7 +113,7 @@ }; &mmc2 { - pinctrl-names = "default", "hs", "ddr_1_8v"; + pinctrl-names = "default", "hs", "ddr_3_3v"; pinctrl-0 = <&mmc2_pins_default>; pinctrl-1 = <&mmc2_pins_hs>; pinctrl-2 = <&mmc2_pins_ddr_rev20 &mmc2_iodelay_ddr_conf>; diff --git a/arch/arm/boot/dts/am572x-idk.dts b/arch/arm/boot/dts/am572x-idk.dts index 3a02ed720957..3ef9111d0e8b 100644 --- a/arch/arm/boot/dts/am572x-idk.dts +++ b/arch/arm/boot/dts/am572x-idk.dts @@ -9,6 +9,7 @@ /dts-v1/; #include "dra74x.dtsi" +#include "dra7-mmc-iodelay.dtsi" #include "dra74x-mmc-iodelay.dtsi" #include "am572x-idk-common.dtsi" @@ -20,7 +21,7 @@ &mmc1 { pinctrl-names = "default", "hs", "sdr12", "sdr25", "sdr50", "ddr50", "sdr104"; - pinctrl-0 = <&mmc1_pins_default>; + pinctrl-0 = <&mmc1_pins_default_no_clk_pu>; pinctrl-1 = <&mmc1_pins_hs>; pinctrl-2 = <&mmc1_pins_sdr12>; pinctrl-3 = <&mmc1_pins_sdr25>; @@ -30,7 +31,7 @@ }; &mmc2 { - pinctrl-names = "default", "hs", "ddr_1_8v"; + pinctrl-names = "default", "hs", "ddr_3_3v"; pinctrl-0 = <&mmc2_pins_default>; pinctrl-1 = <&mmc2_pins_hs>; pinctrl-2 = <&mmc2_pins_ddr_rev20>; diff --git a/arch/arm/boot/dts/am574x-idk.dts b/arch/arm/boot/dts/am574x-idk.dts index 41e12a382d2f..378dfa780ac1 100644 --- a/arch/arm/boot/dts/am574x-idk.dts +++ b/arch/arm/boot/dts/am574x-idk.dts @@ -7,6 +7,8 @@ /dts-v1/; #include "dra76x.dtsi" +#include "dra7-mmc-iodelay.dtsi" +#include "dra76x-mmc-iodelay.dtsi" #include "am572x-idk-common.dtsi" / { @@ -20,3 +22,21 @@ spi-max-frequency = <96000000>; }; }; + +&mmc1 { + pinctrl-names = "default", "hs", "sdr12", "sdr25", "sdr50", "ddr50", "sdr104"; + pinctrl-0 = <&mmc1_pins_default_no_clk_pu>; + pinctrl-1 = <&mmc1_pins_hs>; + pinctrl-2 = <&mmc1_pins_default>; + pinctrl-3 = <&mmc1_pins_hs>; + pinctrl-4 = <&mmc1_pins_sdr50>; + pinctrl-5 = <&mmc1_pins_ddr50 &mmc1_iodelay_ddr_conf>; + pinctrl-6 = <&mmc1_pins_ddr50 &mmc1_iodelay_sdr104_conf>; +}; + +&mmc2 { + pinctrl-names = "default", "hs", "ddr_3_3v"; + pinctrl-0 = <&mmc2_pins_default>; + pinctrl-1 = <&mmc2_pins_default>; + pinctrl-2 = <&mmc2_pins_default>; +}; diff --git a/arch/arm/boot/dts/am57xx-beagle-x15-common.dtsi b/arch/arm/boot/dts/am57xx-beagle-x15-common.dtsi index 6204a266212a..ad953113cefb 100644 --- a/arch/arm/boot/dts/am57xx-beagle-x15-common.dtsi +++ b/arch/arm/boot/dts/am57xx-beagle-x15-common.dtsi @@ -444,8 +444,8 @@ vmmc-supply = <&vdd_3v3>; vqmmc-supply = <&vdd_3v3>; bus-width = <8>; - ti,non-removable; - cap-mmc-dual-data-rate; + non-removable; + no-1-8-v; }; &sata { diff --git a/arch/arm/boot/dts/am57xx-beagle-x15.dts b/arch/arm/boot/dts/am57xx-beagle-x15.dts index d6689106d2a8..70a71c641066 100644 --- a/arch/arm/boot/dts/am57xx-beagle-x15.dts +++ b/arch/arm/boot/dts/am57xx-beagle-x15.dts @@ -25,10 +25,11 @@ pinctrl-1 = <&mmc1_pins_hs>; vmmc-supply = <&ldo1_reg>; + no-1-8-v; }; &mmc2 { - pinctrl-names = "default", "hs", "ddr_1_8v"; + pinctrl-names = "default", "hs", "ddr_3_3v"; pinctrl-0 = <&mmc2_pins_default>; pinctrl-1 = <&mmc2_pins_hs>; pinctrl-2 = <&mmc2_pins_ddr_3_3v_rev11 &mmc2_iodelay_ddr_3_3v_rev11_conf>; diff --git a/arch/arm/boot/dts/am57xx-idk-common.dtsi b/arch/arm/boot/dts/am57xx-idk-common.dtsi index 43cdf523a8a0..ad87f1ae904d 100644 --- a/arch/arm/boot/dts/am57xx-idk-common.dtsi +++ b/arch/arm/boot/dts/am57xx-idk-common.dtsi @@ -115,17 +115,6 @@ DRA7XX_CORE_IOPAD(0x37d4, MUX_MODE15 | PULL_UP) /* dcan1_rx.off */ >; }; - - mmc1_pins_default: mmc1_pins_default { - pinctrl-single,pins = < - DRA7XX_CORE_IOPAD(0x3754, PIN_INPUT_PULLDOWN | MUX_MODE0) /* mmc1_clk.clk */ - DRA7XX_CORE_IOPAD(0x3758, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_cmd.cmd */ - DRA7XX_CORE_IOPAD(0x375c, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat0.dat0 */ - DRA7XX_CORE_IOPAD(0x3760, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat1.dat1 */ - DRA7XX_CORE_IOPAD(0x3764, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat2.dat2 */ - DRA7XX_CORE_IOPAD(0x3768, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat3.dat3 */ - >; - }; }; &i2c1 { @@ -423,8 +412,9 @@ vmmc-supply = <&v3_3d>; vqmmc-supply = <&v3_3d>; bus-width = <8>; - ti,non-removable; + non-removable; max-frequency = <96000000>; + no-1-8-v; }; &dcan1 { diff --git a/arch/arm/boot/dts/armada-370-db.dts b/arch/arm/boot/dts/armada-370-db.dts index afe46097a403..77261a2fb949 100644 --- a/arch/arm/boot/dts/armada-370-db.dts +++ b/arch/arm/boot/dts/armada-370-db.dts @@ -105,33 +105,6 @@ usb@51000 { status = "okay"; }; - - nand@d0000 { - status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - - partitions { - compatible = "fixed-partitions"; - #address-cells = <1>; - #size-cells = <1>; - - partition@0 { - label = "U-Boot"; - reg = <0 0x800000>; - }; - partition@800000 { - label = "Linux"; - reg = <0x800000 0x800000>; - }; - partition@1000000 { - label = "Filesystem"; - reg = <0x1000000 0x3f000000>; - }; - }; - }; }; }; @@ -239,3 +212,33 @@ }; }; +&nand_controller { + status = "okay"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "U-Boot"; + reg = <0 0x800000>; + }; + partition@800000 { + label = "Linux"; + reg = <0x800000 0x800000>; + }; + partition@1000000 { + label = "Filesystem"; + reg = <0x1000000 0x3f000000>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-370-dlink-dns327l.dts b/arch/arm/boot/dts/armada-370-dlink-dns327l.dts index 8e46f63cbaa1..baa459dd51e4 100644 --- a/arch/arm/boot/dts/armada-370-dlink-dns327l.dts +++ b/arch/arm/boot/dts/armada-370-dlink-dns327l.dts @@ -44,61 +44,6 @@ usb@50000 { status = "okay"; }; - - nand@d0000 { - status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - nand-ecc-strength = <4>; - nand-ecc-step-size = <512>; - - partition@0 { - label = "u-boot"; - /* 1.0 MiB */ - reg = <0x0000000 0x100000>; - read-only; - }; - - partition@100000 { - label = "u-boot-env"; - /* 128 KiB */ - reg = <0x100000 0x20000>; - read-only; - }; - - partition@120000 { - label = "uImage"; - /* 7 MiB */ - reg = <0x120000 0x700000>; - }; - - partition@820000 { - label = "ubifs"; - /* ~ 84 MiB */ - reg = <0x820000 0x54e0000>; - }; - - /* Hardcoded into stock bootloader */ - partition@5d00000 { - label = "failsafe-uImage"; - /* 5 MiB */ - reg = <0x5d00000 0x500000>; - }; - - partition@6200000 { - label = "failsafe-fs"; - /* 29 MiB */ - reg = <0x6200000 0x1d00000>; - }; - - partition@7f00000 { - label = "bbt"; - /* 1 MiB for BBT */ - reg = <0x7f00000 0x100000>; - }; - }; }; }; @@ -319,3 +264,68 @@ clock-frequency = <100000>; status = "okay"; }; + +&nand_controller { + status = "okay"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "u-boot"; + /* 1.0 MiB */ + reg = <0x0000000 0x100000>; + read-only; + }; + + partition@100000 { + label = "u-boot-env"; + /* 128 KiB */ + reg = <0x100000 0x20000>; + read-only; + }; + + partition@120000 { + label = "uImage"; + /* 7 MiB */ + reg = <0x120000 0x700000>; + }; + + partition@820000 { + label = "ubifs"; + /* ~ 84 MiB */ + reg = <0x820000 0x54e0000>; + }; + + /* Hardcoded into stock bootloader */ + partition@5d00000 { + label = "failsafe-uImage"; + /* 5 MiB */ + reg = <0x5d00000 0x500000>; + }; + + partition@6200000 { + label = "failsafe-fs"; + /* 29 MiB */ + reg = <0x6200000 0x1d00000>; + }; + + partition@7f00000 { + label = "bbt"; + /* 1 MiB for BBT */ + reg = <0x7f00000 0x100000>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-370-mirabox.dts b/arch/arm/boot/dts/armada-370-mirabox.dts index 996f31b00729..7c2f5a79b50d 100644 --- a/arch/arm/boot/dts/armada-370-mirabox.dts +++ b/arch/arm/boot/dts/armada-370-mirabox.dts @@ -108,27 +108,6 @@ reg = <0x25>; }; }; - - nand@d0000 { - status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - - partition@0 { - label = "U-Boot"; - reg = <0 0x400000>; - }; - partition@400000 { - label = "Linux"; - reg = <0x400000 0x400000>; - }; - partition@800000 { - label = "Filesystem"; - reg = <0x800000 0x3f800000>; - }; - }; }; }; }; @@ -173,3 +152,33 @@ }; }; +&nand_controller { + status = "okay"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "U-Boot"; + reg = <0 0x400000>; + }; + partition@400000 { + label = "Linux"; + reg = <0x400000 0x400000>; + }; + partition@800000 { + label = "Filesystem"; + reg = <0x800000 0x3f800000>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-370-netgear-rn102.dts b/arch/arm/boot/dts/armada-370-netgear-rn102.dts index 56634803e16b..b0b640b7de40 100644 --- a/arch/arm/boot/dts/armada-370-netgear-rn102.dts +++ b/arch/arm/boot/dts/armada-370-netgear-rn102.dts @@ -81,46 +81,6 @@ pwm_polarity = <0>; }; }; - - nand@d0000 { - status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - - /* Use Hardware BCH ECC */ - nand-ecc-strength = <4>; - nand-ecc-step-size = <512>; - - partition@0 { - label = "u-boot"; - reg = <0x0000000 0x180000>; /* 1.5MB */ - read-only; - }; - - partition@180000 { - label = "u-boot-env"; - reg = <0x180000 0x20000>; /* 128KB */ - read-only; - }; - - partition@200000 { - label = "uImage"; - reg = <0x0200000 0x600000>; /* 6MB */ - }; - - partition@800000 { - label = "minirootfs"; - reg = <0x0800000 0x400000>; /* 4MB */ - }; - - /* Last MB is for the BBT, i.e. not writable */ - partition@c00000 { - label = "ubifs"; - reg = <0x0c00000 0x7400000>; /* 116MB */ - }; - }; }; }; @@ -264,3 +224,53 @@ marvell,function = "gpio"; }; }; + +&nand_controller { + status = "okay"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + + /* Use Hardware BCH ECC */ + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "u-boot"; + reg = <0x0000000 0x180000>; /* 1.5MB */ + read-only; + }; + + partition@180000 { + label = "u-boot-env"; + reg = <0x180000 0x20000>; /* 128KB */ + read-only; + }; + + partition@200000 { + label = "uImage"; + reg = <0x0200000 0x600000>; /* 6MB */ + }; + + partition@800000 { + label = "minirootfs"; + reg = <0x0800000 0x400000>; /* 4MB */ + }; + + /* Last MB is for the BBT, i.e. not writable */ + partition@c00000 { + label = "ubifs"; + reg = <0x0c00000 0x7400000>; /* 116MB */ + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-370-netgear-rn104.dts b/arch/arm/boot/dts/armada-370-netgear-rn104.dts index 16d0307f786a..9fd1cb9f4992 100644 --- a/arch/arm/boot/dts/armada-370-netgear-rn104.dts +++ b/arch/arm/boot/dts/armada-370-netgear-rn104.dts @@ -90,46 +90,6 @@ reg = <0x23>; }; }; - - nand@d0000 { - status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - - /* Use Hardware BCH ECC */ - nand-ecc-strength = <4>; - nand-ecc-step-size = <512>; - - partition@0 { - label = "u-boot"; - reg = <0x0000000 0x180000>; /* 1.5MB */ - read-only; - }; - - partition@180000 { - label = "u-boot-env"; - reg = <0x180000 0x20000>; /* 128KB */ - read-only; - }; - - partition@200000 { - label = "uImage"; - reg = <0x0200000 0x600000>; /* 6MB */ - }; - - partition@800000 { - label = "minirootfs"; - reg = <0x0800000 0x400000>; /* 4MB */ - }; - - /* Last MB is for the BBT, i.e. not writable */ - partition@c00000 { - label = "ubifs"; - reg = <0x0c00000 0x7400000>; /* 116MB */ - }; - }; }; }; @@ -276,3 +236,53 @@ marvell,function = "gpio"; }; }; + +&nand_controller { + status = "okay"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + + /* Use Hardware BCH ECC */ + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "u-boot"; + reg = <0x0000000 0x180000>; /* 1.5MB */ + read-only; + }; + + partition@180000 { + label = "u-boot-env"; + reg = <0x180000 0x20000>; /* 128KB */ + read-only; + }; + + partition@200000 { + label = "uImage"; + reg = <0x0200000 0x600000>; /* 6MB */ + }; + + partition@800000 { + label = "minirootfs"; + reg = <0x0800000 0x400000>; /* 4MB */ + }; + + /* Last MB is for the BBT, i.e. not writable */ + partition@c00000 { + label = "ubifs"; + reg = <0x0c00000 0x7400000>; /* 116MB */ + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-370-rd.dts b/arch/arm/boot/dts/armada-370-rd.dts index cc2f774eb267..2bfb3108b5b2 100644 --- a/arch/arm/boot/dts/armada-370-rd.dts +++ b/arch/arm/boot/dts/armada-370-rd.dts @@ -112,27 +112,6 @@ default-state = "keep"; }; }; - - nand@d0000 { - status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - - partition@0 { - label = "U-Boot"; - reg = <0 0x800000>; - }; - partition@800000 { - label = "Linux"; - reg = <0x800000 0x800000>; - }; - partition@1000000 { - label = "Filesystem"; - reg = <0x1000000 0x3f000000>; - }; - }; }; }; @@ -288,3 +267,34 @@ marvell,function = "gpio"; }; }; + +&nand_controller { + status = "okay"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "U-Boot"; + reg = <0 0x800000>; + }; + partition@800000 { + label = "Linux"; + reg = <0x800000 0x800000>; + }; + partition@1000000 { + label = "Filesystem"; + reg = <0x1000000 0x3f000000>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-370-seagate-nas-xbay.dtsi b/arch/arm/boot/dts/armada-370-seagate-nas-xbay.dtsi index a5206db0ebbd..b52634ecf1d9 100644 --- a/arch/arm/boot/dts/armada-370-seagate-nas-xbay.dtsi +++ b/arch/arm/boot/dts/armada-370-seagate-nas-xbay.dtsi @@ -66,33 +66,6 @@ interrupts = <110>; }; }; - - nand@d0000 { - status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - nand-ecc-strength = <4>; - nand-ecc-step-size = <512>; - - partition@0 { - label = "u-boot"; - reg = <0x0 0x300000>; - }; - partition@300000 { - label = "device-tree"; - reg = <0x300000 0x20000>; - }; - partition@320000 { - label = "linux"; - reg = <0x320000 0x2000000>; - }; - partition@2320000 { - label = "rootfs"; - reg = <0x2320000 0xdce0000>; - }; - }; }; }; @@ -227,3 +200,40 @@ marvell,function = "gpio"; }; }; + +&nand_controller { + status = "okay"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "u-boot"; + reg = <0x0 0x300000>; + }; + partition@300000 { + label = "device-tree"; + reg = <0x300000 0x20000>; + }; + partition@320000 { + label = "linux"; + reg = <0x320000 0x2000000>; + }; + partition@2320000 { + label = "rootfs"; + reg = <0x2320000 0xdce0000>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-370-xp.dtsi b/arch/arm/boot/dts/armada-370-xp.dtsi index 11fc3271dad4..c15f5e92f97f 100644 --- a/arch/arm/boot/dts/armada-370-xp.dtsi +++ b/arch/arm/boot/dts/armada-370-xp.dtsi @@ -244,11 +244,11 @@ status = "disabled"; }; - nand: nand@d0000 { - compatible = "marvell,armada370-nand"; + nand_controller: nand-controller@d0000 { + compatible = "marvell,armada370-nand-controller"; reg = <0xd0000 0x54>; #address-cells = <1>; - #size-cells = <1>; + #size-cells = <0>; interrupts = <113>; clocks = <&coredivclk 0>; status = "disabled"; diff --git a/arch/arm/boot/dts/armada-375-db.dts b/arch/arm/boot/dts/armada-375-db.dts index e4ecd7e75644..0e679465cbb5 100644 --- a/arch/arm/boot/dts/armada-375-db.dts +++ b/arch/arm/boot/dts/armada-375-db.dts @@ -103,28 +103,38 @@ nr-ports = <2>; }; -&nand { +&nand_controller { + status = "okay"; pinctrl-0 = <&nand_pins>; pinctrl-names = "default"; - status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - nand-ecc-strength = <4>; - nand-ecc-step-size = <512>; - - partition@0 { - label = "U-Boot"; - reg = <0 0x800000>; - }; - partition@800000 { - label = "Linux"; - reg = <0x800000 0x800000>; - }; - partition@1000000 { - label = "Filesystem"; - reg = <0x1000000 0x3f000000>; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "U-Boot"; + reg = <0 0x800000>; + }; + partition@800000 { + label = "Linux"; + reg = <0x800000 0x800000>; + }; + partition@1000000 { + label = "Filesystem"; + reg = <0x1000000 0x3f000000>; + }; + }; }; }; diff --git a/arch/arm/boot/dts/armada-375.dtsi b/arch/arm/boot/dts/armada-375.dtsi index 53ead6f26a0e..2932a29ae272 100644 --- a/arch/arm/boot/dts/armada-375.dtsi +++ b/arch/arm/boot/dts/armada-375.dtsi @@ -502,11 +502,11 @@ status = "disabled"; }; - nand: nand@d0000 { - compatible = "marvell,armada370-nand"; + nand_controller: nand-controller@d0000 { + compatible = "marvell,armada370-nand-controller"; reg = <0xd0000 0x54>; #address-cells = <1>; - #size-cells = <1>; + #size-cells = <0>; interrupts = ; clocks = <&gateclk 11>; status = "disabled"; diff --git a/arch/arm/boot/dts/armada-385-db-ap.dts b/arch/arm/boot/dts/armada-385-db-ap.dts index d294f24281a5..0e4613bb56ee 100644 --- a/arch/arm/boot/dts/armada-385-db-ap.dts +++ b/arch/arm/boot/dts/armada-385-db-ap.dts @@ -135,39 +135,6 @@ status = "okay"; }; - nfc: flash@d0000 { - status = "okay"; - num-cs = <1>; - nand-ecc-strength = <4>; - nand-ecc-step-size = <512>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - - partitions { - compatible = "fixed-partitions"; - #address-cells = <1>; - #size-cells = <1>; - - partition@0 { - label = "U-Boot"; - reg = <0x00000000 0x00800000>; - read-only; - }; - - partition@800000 { - label = "uImage"; - reg = <0x00800000 0x00400000>; - read-only; - }; - - partition@c00000 { - label = "Root"; - reg = <0x00c00000 0x3f400000>; - }; - }; - }; - usb3@f0000 { status = "okay"; usb-phy = <&usb3_phy>; @@ -233,3 +200,39 @@ spi-max-frequency = <54000000>; }; }; + +&nand_controller { + status = "okay"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + nand-on-flash-bbt; + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "U-Boot"; + reg = <0x00000000 0x00800000>; + read-only; + }; + + partition@800000 { + label = "uImage"; + reg = <0x00800000 0x00400000>; + read-only; + }; + + partition@c00000 { + label = "Root"; + reg = <0x00c00000 0x3f400000>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-385-linksys-caiman.dts b/arch/arm/boot/dts/armada-385-linksys-caiman.dts index 1f30993af405..a03050c97084 100644 --- a/arch/arm/boot/dts/armada-385-linksys-caiman.dts +++ b/arch/arm/boot/dts/armada-385-linksys-caiman.dts @@ -73,67 +73,72 @@ &nand { /* 128MiB */ - - partition@0 { - label = "u-boot"; - reg = <0x0000000 0x200000>; /* 2MiB */ - read-only; - }; - - partition@100000 { - label = "u_env"; - reg = <0x200000 0x40000>; /* 256KiB */ - }; - - partition@140000 { - label = "s_env"; - reg = <0x240000 0x40000>; /* 256KiB */ - }; - - partition@900000 { - label = "devinfo"; - reg = <0x900000 0x100000>; /* 1MiB */ - read-only; - }; - - /* kernel1 overlaps with rootfs1 by design */ - partition@a00000 { - label = "kernel1"; - reg = <0xa00000 0x2800000>; /* 40MiB */ - }; - - partition@1000000 { - label = "rootfs1"; - reg = <0x1000000 0x2200000>; /* 34MiB */ - }; - - /* kernel2 overlaps with rootfs2 by design */ - partition@3200000 { - label = "kernel2"; - reg = <0x3200000 0x2800000>; /* 40MiB */ - }; - - partition@3800000 { - label = "rootfs2"; - reg = <0x3800000 0x2200000>; /* 34MiB */ - }; - - /* - * 38MiB, last MiB is for the BBT, not writable - */ - partition@5a00000 { - label = "syscfg"; - reg = <0x5a00000 0x2600000>; - }; - - /* - * Unused area between "s_env" and "devinfo". - * Moved here because otherwise the renumbered - * partitions would break the bootloader - * supplied bootargs - */ - partition@180000 { - label = "unused_area"; - reg = <0x280000 0x680000>; /* 6.5MiB */ + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "u-boot"; + reg = <0x0000000 0x200000>; /* 2MiB */ + read-only; + }; + + partition@100000 { + label = "u_env"; + reg = <0x200000 0x40000>; /* 256KiB */ + }; + + partition@140000 { + label = "s_env"; + reg = <0x240000 0x40000>; /* 256KiB */ + }; + + partition@900000 { + label = "devinfo"; + reg = <0x900000 0x100000>; /* 1MiB */ + read-only; + }; + + /* kernel1 overlaps with rootfs1 by design */ + partition@a00000 { + label = "kernel1"; + reg = <0xa00000 0x2800000>; /* 40MiB */ + }; + + partition@1000000 { + label = "rootfs1"; + reg = <0x1000000 0x2200000>; /* 34MiB */ + }; + + /* kernel2 overlaps with rootfs2 by design */ + partition@3200000 { + label = "kernel2"; + reg = <0x3200000 0x2800000>; /* 40MiB */ + }; + + partition@3800000 { + label = "rootfs2"; + reg = <0x3800000 0x2200000>; /* 34MiB */ + }; + + /* + * 38MiB, last MiB is for the BBT, not writable + */ + partition@5a00000 { + label = "syscfg"; + reg = <0x5a00000 0x2600000>; + }; + + /* + * Unused area between "s_env" and "devinfo". + * Moved here because otherwise the renumbered + * partitions would break the bootloader + * supplied bootargs + */ + partition@180000 { + label = "unused_area"; + reg = <0x280000 0x680000>; /* 6.5MiB */ + }; }; }; diff --git a/arch/arm/boot/dts/armada-385-linksys-cobra.dts b/arch/arm/boot/dts/armada-385-linksys-cobra.dts index bc34802ce6bc..e3e4877a6f49 100644 --- a/arch/arm/boot/dts/armada-385-linksys-cobra.dts +++ b/arch/arm/boot/dts/armada-385-linksys-cobra.dts @@ -73,67 +73,72 @@ &nand { /* 128MiB */ - - partition@0 { - label = "u-boot"; - reg = <0x0000000 0x200000>; /* 2MiB */ - read-only; - }; - - partition@100000 { - label = "u_env"; - reg = <0x200000 0x40000>; /* 256KiB */ - }; - - partition@140000 { - label = "s_env"; - reg = <0x240000 0x40000>; /* 256KiB */ - }; - - partition@900000 { - label = "devinfo"; - reg = <0x900000 0x100000>; /* 1MiB */ - read-only; - }; - - /* kernel1 overlaps with rootfs1 by design */ - partition@a00000 { - label = "kernel1"; - reg = <0xa00000 0x2800000>; /* 40MiB */ - }; - - partition@1000000 { - label = "rootfs1"; - reg = <0x1000000 0x2200000>; /* 34MiB */ - }; - - /* kernel2 overlaps with rootfs2 by design */ - partition@3200000 { - label = "kernel2"; - reg = <0x3200000 0x2800000>; /* 40MiB */ - }; - - partition@3800000 { - label = "rootfs2"; - reg = <0x3800000 0x2200000>; /* 34MiB */ - }; - - /* - * 38MiB, last MiB is for the BBT, not writable - */ - partition@5a00000 { - label = "syscfg"; - reg = <0x5a00000 0x2600000>; - }; - - /* - * Unused area between "s_env" and "devinfo". - * Moved here because otherwise the renumbered - * partitions would break the bootloader - * supplied bootargs - */ - partition@180000 { - label = "unused_area"; - reg = <0x280000 0x680000>; /* 6.5MiB */ + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "u-boot"; + reg = <0x0000000 0x200000>; /* 2MiB */ + read-only; + }; + + partition@100000 { + label = "u_env"; + reg = <0x200000 0x40000>; /* 256KiB */ + }; + + partition@140000 { + label = "s_env"; + reg = <0x240000 0x40000>; /* 256KiB */ + }; + + partition@900000 { + label = "devinfo"; + reg = <0x900000 0x100000>; /* 1MiB */ + read-only; + }; + + /* kernel1 overlaps with rootfs1 by design */ + partition@a00000 { + label = "kernel1"; + reg = <0xa00000 0x2800000>; /* 40MiB */ + }; + + partition@1000000 { + label = "rootfs1"; + reg = <0x1000000 0x2200000>; /* 34MiB */ + }; + + /* kernel2 overlaps with rootfs2 by design */ + partition@3200000 { + label = "kernel2"; + reg = <0x3200000 0x2800000>; /* 40MiB */ + }; + + partition@3800000 { + label = "rootfs2"; + reg = <0x3800000 0x2200000>; /* 34MiB */ + }; + + /* + * 38MiB, last MiB is for the BBT, not writable + */ + partition@5a00000 { + label = "syscfg"; + reg = <0x5a00000 0x2600000>; + }; + + /* + * Unused area between "s_env" and "devinfo". + * Moved here because otherwise the renumbered + * partitions would break the bootloader + * supplied bootargs + */ + partition@180000 { + label = "unused_area"; + reg = <0x280000 0x680000>; /* 6.5MiB */ + }; }; }; diff --git a/arch/arm/boot/dts/armada-385-linksys-rango.dts b/arch/arm/boot/dts/armada-385-linksys-rango.dts index 5b745a0ccce5..3c4af57ec2b9 100644 --- a/arch/arm/boot/dts/armada-385-linksys-rango.dts +++ b/arch/arm/boot/dts/armada-385-linksys-rango.dts @@ -81,74 +81,79 @@ &nand { /* AMD/Spansion S34ML02G2 256MiB, OEM Layout */ - - partition@0 { - label = "u-boot"; - reg = <0x0000000 0x200000>; /* 2MiB */ - read-only; - }; - - partition@200000 { - label = "u_env"; - reg = <0x200000 0x20000>; /* 128KiB */ - }; - - partition@220000 { - label = "s_env"; - reg = <0x220000 0x40000>; /* 256KiB */ - }; - - partition@7e0000 { - label = "devinfo"; - reg = <0x7e0000 0x40000>; /* 256KiB */ - read-only; - }; - - partition@820000 { - label = "sysdiag"; - reg = <0x820000 0x1e0000>; /* 1920KiB */ - read-only; - }; - - /* kernel1 overlaps with rootfs1 by design */ - partition@a00000 { - label = "kernel1"; - reg = <0xa00000 0x5000000>; /* 80MiB */ - }; - - partition@1000000 { - label = "rootfs1"; - reg = <0x1000000 0x4a00000>; /* 74MiB */ - }; - - /* kernel2 overlaps with rootfs2 by design */ - partition@5a00000 { - label = "kernel2"; - reg = <0x5a00000 0x5000000>; /* 80MiB */ - }; - - partition@6000000 { - label = "rootfs2"; - reg = <0x6000000 0x4a00000>; /* 74MiB */ - }; - - /* - * 86MiB, last MiB is for the BBT, not writable - */ - partition@aa00000 { - label = "syscfg"; - reg = <0xaa00000 0x5600000>; - }; - - /* - * Unused area between "s_env" and "devinfo". - * Moved here because otherwise the renumbered - * partitions would break the bootloader - * supplied bootargs - */ - partition@180000 { - label = "unused_area"; - reg = <0x260000 0x5c0000>; /* 5.75MiB */ + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "u-boot"; + reg = <0x0000000 0x200000>; /* 2MiB */ + read-only; + }; + + partition@200000 { + label = "u_env"; + reg = <0x200000 0x20000>; /* 128KiB */ + }; + + partition@220000 { + label = "s_env"; + reg = <0x220000 0x40000>; /* 256KiB */ + }; + + partition@7e0000 { + label = "devinfo"; + reg = <0x7e0000 0x40000>; /* 256KiB */ + read-only; + }; + + partition@820000 { + label = "sysdiag"; + reg = <0x820000 0x1e0000>; /* 1920KiB */ + read-only; + }; + + /* kernel1 overlaps with rootfs1 by design */ + partition@a00000 { + label = "kernel1"; + reg = <0xa00000 0x5000000>; /* 80MiB */ + }; + + partition@1000000 { + label = "rootfs1"; + reg = <0x1000000 0x4a00000>; /* 74MiB */ + }; + + /* kernel2 overlaps with rootfs2 by design */ + partition@5a00000 { + label = "kernel2"; + reg = <0x5a00000 0x5000000>; /* 80MiB */ + }; + + partition@6000000 { + label = "rootfs2"; + reg = <0x6000000 0x4a00000>; /* 74MiB */ + }; + + /* + * 86MiB, last MiB is for the BBT, not writable + */ + partition@aa00000 { + label = "syscfg"; + reg = <0xaa00000 0x5600000>; + }; + + /* + * Unused area between "s_env" and "devinfo". + * Moved here because otherwise the renumbered + * partitions would break the bootloader + * supplied bootargs + */ + partition@180000 { + label = "unused_area"; + reg = <0x260000 0x5c0000>; /* 5.75MiB */ + }; }; }; diff --git a/arch/arm/boot/dts/armada-385-linksys-shelby.dts b/arch/arm/boot/dts/armada-385-linksys-shelby.dts index 44f5aeb5fc33..3451cd3e5dff 100644 --- a/arch/arm/boot/dts/armada-385-linksys-shelby.dts +++ b/arch/arm/boot/dts/armada-385-linksys-shelby.dts @@ -73,67 +73,72 @@ &nand { /* 128MiB */ - - partition@0 { - label = "u-boot"; - reg = <0x0000000 0x200000>; /* 2MiB */ - read-only; - }; - - partition@100000 { - label = "u_env"; - reg = <0x200000 0x40000>; /* 256KiB */ - }; - - partition@140000 { - label = "s_env"; - reg = <0x240000 0x40000>; /* 256KiB */ - }; - - partition@900000 { - label = "devinfo"; - reg = <0x900000 0x100000>; /* 1MiB */ - read-only; - }; - - /* kernel1 overlaps with rootfs1 by design */ - partition@a00000 { - label = "kernel1"; - reg = <0xa00000 0x2800000>; /* 40MiB */ - }; - - partition@1000000 { - label = "rootfs1"; - reg = <0x1000000 0x2200000>; /* 34MiB */ - }; - - /* kernel2 overlaps with rootfs2 by design */ - partition@3200000 { - label = "kernel2"; - reg = <0x3200000 0x2800000>; /* 40MiB */ - }; - - partition@3800000 { - label = "rootfs2"; - reg = <0x3800000 0x2200000>; /* 34MiB */ - }; - - /* - * 38MiB, last MiB is for the BBT, not writable - */ - partition@5a00000 { - label = "syscfg"; - reg = <0x5a00000 0x2600000>; - }; - - /* - * Unused area between "s_env" and "devinfo". - * Moved here because otherwise the renumbered - * partitions would break the bootloader - * supplied bootargs - */ - partition@180000 { - label = "unused_area"; - reg = <0x280000 0x680000>; /* 6.5MiB */ + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "u-boot"; + reg = <0x0000000 0x200000>; /* 2MiB */ + read-only; + }; + + partition@100000 { + label = "u_env"; + reg = <0x200000 0x40000>; /* 256KiB */ + }; + + partition@140000 { + label = "s_env"; + reg = <0x240000 0x40000>; /* 256KiB */ + }; + + partition@900000 { + label = "devinfo"; + reg = <0x900000 0x100000>; /* 1MiB */ + read-only; + }; + + /* kernel1 overlaps with rootfs1 by design */ + partition@a00000 { + label = "kernel1"; + reg = <0xa00000 0x2800000>; /* 40MiB */ + }; + + partition@1000000 { + label = "rootfs1"; + reg = <0x1000000 0x2200000>; /* 34MiB */ + }; + + /* kernel2 overlaps with rootfs2 by design */ + partition@3200000 { + label = "kernel2"; + reg = <0x3200000 0x2800000>; /* 40MiB */ + }; + + partition@3800000 { + label = "rootfs2"; + reg = <0x3800000 0x2200000>; /* 34MiB */ + }; + + /* + * 38MiB, last MiB is for the BBT, not writable + */ + partition@5a00000 { + label = "syscfg"; + reg = <0x5a00000 0x2600000>; + }; + + /* + * Unused area between "s_env" and "devinfo". + * Moved here because otherwise the renumbered + * partitions would break the bootloader + * supplied bootargs + */ + partition@180000 { + label = "unused_area"; + reg = <0x280000 0x680000>; /* 6.5MiB */ + }; }; }; diff --git a/arch/arm/boot/dts/armada-385-linksys.dtsi b/arch/arm/boot/dts/armada-385-linksys.dtsi index 4a0d7360110b..827e82be2201 100644 --- a/arch/arm/boot/dts/armada-385-linksys.dtsi +++ b/arch/arm/boot/dts/armada-385-linksys.dtsi @@ -138,13 +138,19 @@ }; }; -&nand { +&nand_controller { /* 128MiB or 256MiB */ status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; + #address-cells = <1>; + #size-cells = <0>; + + nand: nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + }; }; &mdio { diff --git a/arch/arm/boot/dts/armada-388-db.dts b/arch/arm/boot/dts/armada-388-db.dts index 05250d426dc4..a2bec07bf4c5 100644 --- a/arch/arm/boot/dts/armada-388-db.dts +++ b/arch/arm/boot/dts/armada-388-db.dts @@ -91,29 +91,6 @@ status = "okay"; }; - flash@d0000 { - status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - nand-ecc-strength = <4>; - nand-ecc-step-size = <512>; - - partition@0 { - label = "U-Boot"; - reg = <0 0x800000>; - }; - partition@800000 { - label = "Linux"; - reg = <0x800000 0x800000>; - }; - partition@1000000 { - label = "Filesystem"; - reg = <0x1000000 0x3f000000>; - }; - }; - sdhci@d8000 { broken-cd; wp-inverted; @@ -165,3 +142,35 @@ }; }; +&nand_controller { + status = "okay"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "U-Boot"; + reg = <0 0x800000>; + }; + partition@800000 { + label = "Linux"; + reg = <0x800000 0x800000>; + }; + partition@1000000 { + label = "Filesystem"; + reg = <0x1000000 0x3f000000>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-38x.dtsi b/arch/arm/boot/dts/armada-38x.dtsi index 4cc09e43eea2..18edc9bc7927 100644 --- a/arch/arm/boot/dts/armada-38x.dtsi +++ b/arch/arm/boot/dts/armada-38x.dtsi @@ -163,7 +163,7 @@ }; uart0: serial@12000 { - compatible = "snps,dw-apb-uart"; + compatible = "marvell,armada-38x-uart"; reg = <0x12000 0x100>; reg-shift = <2>; interrupts = ; @@ -173,7 +173,7 @@ }; uart1: serial@12100 { - compatible = "snps,dw-apb-uart"; + compatible = "marvell,armada-38x-uart"; reg = <0x12100 0x100>; reg-shift = <2>; interrupts = ; @@ -551,11 +551,11 @@ status = "okay"; }; - nand: flash@d0000 { - compatible = "marvell,armada370-nand"; + nand_controller: nand-controller@d0000 { + compatible = "marvell,armada370-nand-controller"; reg = <0xd0000 0x54>; #address-cells = <1>; - #size-cells = <1>; + #size-cells = <0>; interrupts = ; clocks = <&coredivclk 0>; status = "disabled"; diff --git a/arch/arm/boot/dts/armada-390-db.dts b/arch/arm/boot/dts/armada-390-db.dts index 1b2362e4c831..0e29474ae9a2 100644 --- a/arch/arm/boot/dts/armada-390-db.dts +++ b/arch/arm/boot/dts/armada-390-db.dts @@ -49,37 +49,6 @@ status = "okay"; }; - flash@d0000 { - status = "okay"; - pinctrl-0 = <&nand_pins>; - pinctrl-names = "default"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - nand-ecc-strength = <8>; - nand-ecc-step-size = <512>; - - partitions { - compatible = "fixed-partitions"; - #address-cells = <1>; - #size-cells = <1>; - - partition@0 { - label = "U-Boot"; - reg = <0 0x800000>; - }; - partition@800000 { - label = "Linux"; - reg = <0x800000 0x800000>; - }; - partition@1000000 { - label = "Filesystem"; - reg = <0x1000000 0x3f000000>; - }; - }; - }; - /* CON98 */ usb3@f8000 { status = "okay"; @@ -136,3 +105,38 @@ }; }; }; + +&nand_controller { + status = "okay"; + pinctrl-0 = <&nand_pins>; + pinctrl-names = "default"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + nand-ecc-strength = <8>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "U-Boot"; + reg = <0 0x800000>; + }; + partition@800000 { + label = "Linux"; + reg = <0x800000 0x800000>; + }; + partition@1000000 { + label = "Filesystem"; + reg = <0x1000000 0x3f000000>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-395-gp.dts b/arch/arm/boot/dts/armada-395-gp.dts index 2a9de192b423..6dd9e9077f84 100644 --- a/arch/arm/boot/dts/armada-395-gp.dts +++ b/arch/arm/boot/dts/armada-395-gp.dts @@ -57,41 +57,6 @@ status = "okay"; }; - flash@d0000 { - status = "okay"; - pinctrl-0 = <&nand_pins>; - pinctrl-names = "default"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - nand-ecc-strength = <4>; - nand-ecc-step-size = <512>; - - partitions { - compatible = "fixed-partitions"; - #address-cells = <1>; - #size-cells = <1>; - - partition@0 { - label = "U-Boot"; - reg = <0x00000000 0x00600000>; - read-only; - }; - - partition@800000 { - label = "uImage"; - reg = <0x00600000 0x00400000>; - read-only; - }; - - partition@1000000 { - label = "Root"; - reg = <0x00a00000 0x3f600000>; - }; - }; - }; - /* CON18 */ sdhci@d8000 { clock-frequency = <200000000>; @@ -130,3 +95,42 @@ }; }; }; + +&nand_controller { + status = "okay"; + pinctrl-0 = <&nand_pins>; + pinctrl-names = "default"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "U-Boot"; + reg = <0x00000000 0x00600000>; + read-only; + }; + + partition@800000 { + label = "uImage"; + reg = <0x00600000 0x00400000>; + read-only; + }; + + partition@1000000 { + label = "Root"; + reg = <0x00a00000 0x3f600000>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-398-db.dts b/arch/arm/boot/dts/armada-398-db.dts index 2337f24784f7..fc28308e5bc5 100644 --- a/arch/arm/boot/dts/armada-398-db.dts +++ b/arch/arm/boot/dts/armada-398-db.dts @@ -51,31 +51,6 @@ status = "okay"; }; - flash@d0000 { - status = "okay"; - pinctrl-0 = <&nand_pins>; - pinctrl-names = "default"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - nand-ecc-strength = <8>; - nand-ecc-step-size = <512>; - - partition@0 { - label = "U-Boot"; - reg = <0 0x800000>; - }; - partition@800000 { - label = "Linux"; - reg = <0x800000 0x800000>; - }; - partition@1000000 { - label = "Filesystem"; - reg = <0x1000000 0x3f000000>; - }; - }; - usb3@f8000 { status = "okay"; }; @@ -122,3 +97,38 @@ }; }; }; + +&nand_controller { + status = "okay"; + pinctrl-0 = <&nand_pins>; + pinctrl-names = "default"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + nand-ecc-strength = <8>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "U-Boot"; + reg = <0 0x800000>; + }; + partition@800000 { + label = "Linux"; + reg = <0x800000 0x800000>; + }; + partition@1000000 { + label = "Filesystem"; + reg = <0x1000000 0x3f000000>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-39x.dtsi b/arch/arm/boot/dts/armada-39x.dtsi index c1737c0a8325..f0c949831efb 100644 --- a/arch/arm/boot/dts/armada-39x.dtsi +++ b/arch/arm/boot/dts/armada-39x.dtsi @@ -367,11 +367,11 @@ interrupts = ; }; - flash@d0000 { - compatible = "marvell,armada370-nand"; + nand_controller: nand-controller@d0000 { + compatible = "marvell,armada370-nand-controller"; reg = <0xd0000 0x54>; #address-cells = <1>; - #size-cells = <1>; + #size-cells = <0>; interrupts = ; clocks = <&coredivclk 0>; status = "disabled"; diff --git a/arch/arm/boot/dts/armada-xp-98dx3236.dtsi b/arch/arm/boot/dts/armada-xp-98dx3236.dtsi index a5da44fb35ed..8d708cc22495 100644 --- a/arch/arm/boot/dts/armada-xp-98dx3236.dtsi +++ b/arch/arm/boot/dts/armada-xp-98dx3236.dtsi @@ -306,6 +306,19 @@ &pinctrl { compatible = "marvell,98dx3236-pinctrl"; + nand_pins: nand-pins { + marvell,pins = "mpp20", "mpp21", "mpp22", + "mpp23", "mpp24", "mpp25", + "mpp26", "mpp27", "mpp28", + "mpp29", "mpp30"; + marvell,function = "dev"; + }; + + nand_rb: nand-rb { + marvell,pins = "mpp19"; + marvell,function = "nand"; + }; + spi0_pins: spi0-pins { marvell,pins = "mpp0", "mpp1", "mpp2", "mpp3"; diff --git a/arch/arm/boot/dts/armada-xp-db-dxbc2.dts b/arch/arm/boot/dts/armada-xp-db-dxbc2.dts index 4c64923f1c52..f42fc6118b7c 100644 --- a/arch/arm/boot/dts/armada-xp-db-dxbc2.dts +++ b/arch/arm/boot/dts/armada-xp-db-dxbc2.dts @@ -70,9 +70,9 @@ &nand { status = "okay"; + label = "pxa3xx_nand-0"; num-cs = <1>; marvell,nand-keep-config; - marvell,nand-enable-arbiter; nand-on-flash-bbt; nand-ecc-strength = <4>; nand-ecc-step-size = <512>; diff --git a/arch/arm/boot/dts/armada-xp-db-xc3-24g4xg.dts b/arch/arm/boot/dts/armada-xp-db-xc3-24g4xg.dts index a0ebb52683f1..8432f517e346 100644 --- a/arch/arm/boot/dts/armada-xp-db-xc3-24g4xg.dts +++ b/arch/arm/boot/dts/armada-xp-db-xc3-24g4xg.dts @@ -69,9 +69,9 @@ &nand { status = "okay"; + label = "pxa3xx_nand-0"; num-cs = <1>; marvell,nand-keep-config; - marvell,nand-enable-arbiter; nand-on-flash-bbt; nand-ecc-strength = <4>; nand-ecc-step-size = <512>; diff --git a/arch/arm/boot/dts/armada-xp-db.dts b/arch/arm/boot/dts/armada-xp-db.dts index 73d3f5cb9828..f3ac7483afed 100644 --- a/arch/arm/boot/dts/armada-xp-db.dts +++ b/arch/arm/boot/dts/armada-xp-db.dts @@ -146,9 +146,9 @@ nand@d0000 { status = "okay"; + label = "pxa3xx_nand-0"; num-cs = <1>; marvell,nand-keep-config; - marvell,nand-enable-arbiter; nand-on-flash-bbt; partitions { diff --git a/arch/arm/boot/dts/armada-xp-gp.dts b/arch/arm/boot/dts/armada-xp-gp.dts index c143556bbb7b..1139e9469a83 100644 --- a/arch/arm/boot/dts/armada-xp-gp.dts +++ b/arch/arm/boot/dts/armada-xp-gp.dts @@ -162,9 +162,9 @@ nand@d0000 { status = "okay"; + label = "pxa3xx_nand-0"; num-cs = <1>; marvell,nand-keep-config; - marvell,nand-enable-arbiter; nand-on-flash-bbt; }; }; diff --git a/arch/arm/boot/dts/armada-xp-lenovo-ix4-300d.dts b/arch/arm/boot/dts/armada-xp-lenovo-ix4-300d.dts index def62e9e835b..bbbb38888bb8 100644 --- a/arch/arm/boot/dts/armada-xp-lenovo-ix4-300d.dts +++ b/arch/arm/boot/dts/armada-xp-lenovo-ix4-300d.dts @@ -83,9 +83,9 @@ nand@d0000 { status = "okay"; + label = "pxa3xx_nand-0"; num-cs = <1>; marvell,nand-keep-config; - marvell,nand-enable-arbiter; nand-on-flash-bbt; partitions { diff --git a/arch/arm/boot/dts/armada-xp-linksys-mamba.dts b/arch/arm/boot/dts/armada-xp-linksys-mamba.dts index f8b60d937818..7a2606c3b62e 100644 --- a/arch/arm/boot/dts/armada-xp-linksys-mamba.dts +++ b/arch/arm/boot/dts/armada-xp-linksys-mamba.dts @@ -165,79 +165,6 @@ bm@c8000 { status = "okay"; }; - - nand@d0000 { - status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - nand-ecc-strength = <4>; - nand-ecc-step-size = <512>; - - partition@0 { - label = "u-boot"; - reg = <0x0000000 0x100000>; /* 1MB */ - read-only; - }; - - partition@100000 { - label = "u_env"; - reg = <0x100000 0x40000>; /* 256KB */ - }; - - partition@140000 { - label = "s_env"; - reg = <0x140000 0x40000>; /* 256KB */ - }; - - partition@900000 { - label = "devinfo"; - reg = <0x900000 0x100000>; /* 1MB */ - read-only; - }; - - /* kernel1 overlaps with rootfs1 by design */ - partition@a00000 { - label = "kernel1"; - reg = <0xa00000 0x2800000>; /* 40MB */ - }; - - partition@d00000 { - label = "rootfs1"; - reg = <0xd00000 0x2500000>; /* 37MB */ - }; - - /* kernel2 overlaps with rootfs2 by design */ - partition@3200000 { - label = "kernel2"; - reg = <0x3200000 0x2800000>; /* 40MB */ - }; - - partition@3500000 { - label = "rootfs2"; - reg = <0x3500000 0x2500000>; /* 37MB */ - }; - - /* - * 38MB, last MB is for the BBT, not writable - */ - partition@5a00000 { - label = "syscfg"; - reg = <0x5a00000 0x2600000>; - }; - - /* - * Unused area between "s_env" and "devinfo". - * Moved here because otherwise the renumbered - * partitions would break the bootloader - * supplied bootargs - */ - partition@180000 { - label = "unused_area"; - reg = <0x180000 0x780000>; /* 7.5MB */ - }; - }; }; bm-bppi { @@ -434,3 +361,86 @@ }; }; }; + +&nand_controller { + status = "okay"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "u-boot"; + reg = <0x0000000 0x100000>; /* 1MB */ + read-only; + }; + + partition@100000 { + label = "u_env"; + reg = <0x100000 0x40000>; /* 256KB */ + }; + + partition@140000 { + label = "s_env"; + reg = <0x140000 0x40000>; /* 256KB */ + }; + + partition@900000 { + label = "devinfo"; + reg = <0x900000 0x100000>; /* 1MB */ + read-only; + }; + + /* kernel1 overlaps with rootfs1 by design */ + partition@a00000 { + label = "kernel1"; + reg = <0xa00000 0x2800000>; /* 40MB */ + }; + + partition@d00000 { + label = "rootfs1"; + reg = <0xd00000 0x2500000>; /* 37MB */ + }; + + /* kernel2 overlaps with rootfs2 by design */ + partition@3200000 { + label = "kernel2"; + reg = <0x3200000 0x2800000>; /* 40MB */ + }; + + partition@3500000 { + label = "rootfs2"; + reg = <0x3500000 0x2500000>; /* 37MB */ + }; + + /* + * 38MB, last MB is for the BBT, not writable + */ + partition@5a00000 { + label = "syscfg"; + reg = <0x5a00000 0x2600000>; + }; + + /* + * Unused area between "s_env" and "devinfo". + * Moved here because otherwise the renumbered + * partitions would break the bootloader + * supplied bootargs + */ + partition@180000 { + label = "unused_area"; + reg = <0x180000 0x780000>; /* 7.5MB */ + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/armada-xp-netgear-rn2120.dts b/arch/arm/boot/dts/armada-xp-netgear-rn2120.dts index c350b1cf5201..8ea73587db81 100644 --- a/arch/arm/boot/dts/armada-xp-netgear-rn2120.dts +++ b/arch/arm/boot/dts/armada-xp-netgear-rn2120.dts @@ -117,46 +117,6 @@ nr-ports = <2>; status = "okay"; }; - - nand@d0000 { - status = "okay"; - num-cs = <1>; - marvell,nand-keep-config; - marvell,nand-enable-arbiter; - nand-on-flash-bbt; - - /* Use Hardware BCH ECC */ - nand-ecc-strength = <4>; - nand-ecc-step-size = <512>; - - partition@0 { - label = "u-boot"; - reg = <0x0000000 0x180000>; /* 1.5MB */ - read-only; - }; - - partition@180000 { - label = "u-boot-env"; - reg = <0x180000 0x20000>; /* 128KB */ - read-only; - }; - - partition@200000 { - label = "uImage"; - reg = <0x0200000 0x600000>; /* 6MB */ - }; - - partition@800000 { - label = "minirootfs"; - reg = <0x0800000 0x400000>; /* 4MB */ - }; - - /* Last MB is for the BBT, i.e. not writable */ - partition@c00000 { - label = "ubifs"; - reg = <0x0c00000 0x7400000>; /* 116MB */ - }; - }; }; }; @@ -345,3 +305,53 @@ marvell,function = "gpio"; }; }; + +&nand_controller { + status = "okay"; + + nand@0 { + reg = <0>; + label = "pxa3xx_nand-0"; + nand-rb = <0>; + marvell,nand-keep-config; + nand-on-flash-bbt; + + /* Use Hardware BCH ECC */ + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "u-boot"; + reg = <0x0000000 0x180000>; /* 1.5MB */ + read-only; + }; + + partition@180000 { + label = "u-boot-env"; + reg = <0x180000 0x20000>; /* 128KB */ + read-only; + }; + + partition@200000 { + label = "uImage"; + reg = <0x0200000 0x600000>; /* 6MB */ + }; + + partition@800000 { + label = "minirootfs"; + reg = <0x0800000 0x400000>; /* 4MB */ + }; + + /* Last MB is for the BBT, i.e. not writable */ + partition@c00000 { + label = "ubifs"; + reg = <0x0c00000 0x7400000>; /* 116MB */ + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/aspeed-ast2500-evb.dts b/arch/arm/boot/dts/aspeed-ast2500-evb.dts index 91a36c1f029b..ede11c597673 100644 --- a/arch/arm/boot/dts/aspeed-ast2500-evb.dts +++ b/arch/arm/boot/dts/aspeed-ast2500-evb.dts @@ -79,3 +79,21 @@ reg = <0x4d>; }; }; + +&ehci0 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_usb2ah_default>; +}; + +&ehci1 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_usb2bh_default>; +}; + +&uhci { + status = "okay"; + + /* No pinctrl, this follows the above EHCI settings */ +}; diff --git a/arch/arm/boot/dts/aspeed-bmc-intel-s2600wf.dts b/arch/arm/boot/dts/aspeed-bmc-intel-s2600wf.dts new file mode 100644 index 000000000000..7a291de02543 --- /dev/null +++ b/arch/arm/boot/dts/aspeed-bmc-intel-s2600wf.dts @@ -0,0 +1,129 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2017 Intel Corporation +/dts-v1/; + +#include "aspeed-g5.dtsi" + +/ { + model = "S2600WF BMC"; + compatible = "intel,s2600wf-bmc", "aspeed,ast2500"; + + chosen { + stdout-path = &uart5; + bootargs = "earlyprintk"; + }; + + memory { + reg = <0x80000000 0x20000000>; + }; + + reserved-memory { + #address-cells = <1>; + #size-cells = <1>; + ranges; + + vga_memory: framebuffer@7f000000 { + no-map; + reg = <0x7f000000 0x01000000>; + }; + }; + + iio-hwmon { + compatible = "iio-hwmon"; + io-channels = <&adc 0>, <&adc 1>, <&adc 2>, <&adc 3>, + <&adc 4>, <&adc 5>, <&adc 6>, <&adc 7>, + <&adc 8>, <&adc 9>, <&adc 10>, <&adc 11>, + <&adc 12>, <&adc 13>, <&adc 14>, <&adc 15>; + }; + +}; + +&fmc { + status = "okay"; + flash@0 { + status = "okay"; + m25p,fast-read; + label = "bmc"; +#include "openbmc-flash-layout.dtsi" + }; +}; + +&spi1 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_spi1_default>; + + flash@0 { + status = "okay"; + m25p,fast-read; + label = "pnor"; + }; +}; + +&uart5 { + status = "okay"; +}; + +&mac0 { + status = "okay"; + + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_rmii1_default>; + use-ncsi; +}; + +&mac1 { + status = "okay"; + + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_rgmii2_default &pinctrl_mdio2_default>; +}; + +&i2c1 { + status = "okay"; +}; + +&i2c2 { + status = "okay"; +}; + +&i2c3 { + status = "okay"; +}; + +&i2c4 { + status = "okay"; +}; + +&i2c5 { + status = "okay"; +}; + +&i2c6 { + status = "okay"; +}; + +&i2c7 { + status = "okay"; +}; + +&i2c13 { + status = "okay"; +}; + +&gfx { + status = "okay"; +}; + +&pinctrl { + aspeed,external-nodes = <&gfx &lhc>; +}; + +&pwm_tacho { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_pwm0_default &pinctrl_pwm1_default + &pinctrl_pwm2_default &pinctrl_pwm3_default + &pinctrl_pwm4_default &pinctrl_pwm5_default + &pinctrl_pwm6_default &pinctrl_pwm7_default>; +}; diff --git a/arch/arm/boot/dts/aspeed-bmc-opp-lanyang.dts b/arch/arm/boot/dts/aspeed-bmc-opp-lanyang.dts new file mode 100644 index 000000000000..d598b6391362 --- /dev/null +++ b/arch/arm/boot/dts/aspeed-bmc-opp-lanyang.dts @@ -0,0 +1,325 @@ +// SPDX-License-Identifier: GPL-2.0+ +// Copyright (c) 2018 Inventec Corporation +/dts-v1/; + +#include "aspeed-g5.dtsi" +#include + +/ { + model = "Lanyang BMC"; + compatible = "inventec,lanyang-bmc", "aspeed,ast2500"; + + chosen { + stdout-path = &uart5; + bootargs = "console=ttyS4,115200 earlyprintk"; + }; + + memory { + reg = <0x80000000 0x40000000>; + }; + + reserved-memory { + #address-cells = <1>; + #size-cells = <1>; + ranges; + + flash_memory: region@98000000 { + no-map; + reg = <0x98000000 0x04000000>; /* 64M */ + }; + }; + + leds { + compatible = "gpio-leds"; + + sys_boot_status { + label = "System_boot_status"; + gpios = <&gpio ASPEED_GPIO(B, 6) GPIO_ACTIVE_LOW>; + }; + + attention { + label = "Attention_locator"; + gpios = <&gpio ASPEED_GPIO(B, 7) GPIO_ACTIVE_HIGH>; + }; + + plt_fault { + label = "Platform_fault"; + gpios = <&gpio ASPEED_GPIO(B, 1) GPIO_ACTIVE_HIGH>; + }; + + hdd_fault { + label = "Onboard_drive_fault"; + gpios = <&gpio ASPEED_GPIO(B, 3) GPIO_ACTIVE_HIGH>; + }; + bmc_err { + lable = "BMC_fault"; + gpios = <&gpio ASPEED_GPIO(H, 6) GPIO_ACTIVE_HIGH>; + }; + + sys_err { + lable = "Sys_fault"; + gpios = <&gpio ASPEED_GPIO(H, 7) GPIO_ACTIVE_HIGH>; + }; + }; + + fsi: gpio-fsi { + compatible = "fsi-master-gpio", "fsi-master"; + #address-cells = <2>; + #size-cells = <0>; + + clock-gpios = <&gpio ASPEED_GPIO(J, 0) GPIO_ACTIVE_HIGH>; + data-gpios = <&gpio ASPEED_GPIO(J, 1) GPIO_ACTIVE_HIGH>; + trans-gpios = <&gpio ASPEED_GPIO(D, 5) GPIO_ACTIVE_HIGH>; + enable-gpios = <&gpio ASPEED_GPIO(D, 0) GPIO_ACTIVE_HIGH>; + mux-gpios = <&gpio ASPEED_GPIO(H, 2) GPIO_ACTIVE_HIGH>; + }; + + iio-hwmon { + compatible = "iio-hwmon"; + io-channels = <&adc 0>, <&adc 1>, <&adc 2>, <&adc 3>, + <&adc 4>, <&adc 5>, <&adc 6>, <&adc 7>, + <&adc 8>, <&adc 9>, <&adc 10>, <&adc 11>, + <&adc 13>, <&adc 14>, <&adc 15>; + }; + + iio-hwmon-battery { + compatible = "iio-hwmon"; + io-channels = <&adc 12>; + }; +}; + +&pwm_tacho { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_pwm0_default &pinctrl_pwm1_default + &pinctrl_pwm2_default &pinctrl_pwm3_default>; + + fan@0 { + reg = <0x00>; + aspeed,fan-tach-ch = /bits/ 8 <0x00>; + }; + + fan@1 { + reg = <0x01>; + aspeed,fan-tach-ch = /bits/ 8 <0x01>; + }; + + fan@2 { + reg = <0x02>; + aspeed,fan-tach-ch = /bits/ 8 <0x02>; + }; + + fan@3 { + reg = <0x03>; + aspeed,fan-tach-ch = /bits/ 8 <0x03>; + }; +}; + +&fmc { + status = "okay"; + flash@0 { + status = "okay"; + m25p,fast-read; + label = "bmc"; +#include "openbmc-flash-layout.dtsi" + }; +}; + +&spi1 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_spi1_default>; + + flash@0 { + status = "okay"; + label = "pnor"; + m25p,fast-read; + }; +}; + +&spi2 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_spi2ck_default + &pinctrl_spi2cs0_default + &pinctrl_spi2cs1_default + &pinctrl_spi2miso_default + &pinctrl_spi2mosi_default>; + + flash@0 { + status = "okay"; + }; +}; + +&uart1 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_txd1_default + &pinctrl_rxd1_default>; +}; + +&lpc_ctrl { + status = "okay"; + memory-region = <&flash_memory>; + flash = <&spi1>; +}; + +&lpc_snoop { + status = "okay"; + snoop-ports = <0x80>; +}; + +&uart5 { + status = "okay"; +}; + +&mac0 { + status = "okay"; + + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_rmii1_default>; + use-ncsi; +}; + +&mac1 { + status = "okay"; + + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_rgmii2_default &pinctrl_mdio2_default>; +}; + +&i2c0 { + status = "okay"; + + eeprom@55 { + compatible = "atmel,24c64"; + reg = <0x55>; + pagesize = <32>; + }; + + rtc@68 { + compatible = "nxp,pcf8523"; + reg = <0x68>; + }; + + tmp75@48 { + compatible = "ti,tmp75"; + reg = <0x48>; + }; +}; + +&i2c1 { + status = "okay"; +}; + +&i2c2 { + status = "okay"; +}; + +&i2c3 { + status = "okay"; +}; + +&i2c4 { + status = "okay"; +}; + +&i2c5 { + status = "okay"; +}; + +&i2c6 { + status = "okay"; +}; + +&i2c7 { + status = "okay"; +}; + +&i2c8 { + status = "okay"; +}; + +&i2c9 { + status = "okay"; +}; + +&i2c10 { + status = "okay"; +}; + +&i2c11 { + status = "okay"; +}; + +&vuart { + status = "okay"; +}; + +&gfx { + status = "okay"; +}; + +&pinctrl { + aspeed,external-nodes = <&gfx &lhc>; +}; + +&gpio { + pin_gpio_b0 { + gpio-hog; + gpios = ; + output-high; + line-name = "BMC_HDD1_PWR_EN"; + }; + + pin_gpio_b5 { + gpio-hog; + gpios = ; + input; + line-name = "BMC_USB1_OCI2"; + }; + + pin_gpio_h5 { + gpio-hog; + gpios = ; + output-high; + line-name = "BMC_CP0_PERST_ENABLE_R"; + }; + + pin_gpio_z2 { + gpio-hog; + gpios = ; + output-high; + line-name = "RST_PCA9546_U177_N"; + }; + + pin_gpio_aa6 { + gpio-hog; + gpios = ; + output-high; + line-name = "BMC_CP0_RESET_N"; + }; + + pin_gpio_aa7 { + gpio-hog; + gpios = ; + output-high; + line-name = "BMC_TPM_RESET_N"; + }; + + pin_gpio_ab0 { + gpio-hog; + gpios = ; + output-high; + line-name = "BMC_USB_PWRON_N"; + }; +}; + +&ibt { + status = "okay"; +}; + +&adc { + status = "okay"; +}; + diff --git a/arch/arm/boot/dts/aspeed-bmc-opp-romulus.dts b/arch/arm/boot/dts/aspeed-bmc-opp-romulus.dts index 51bc6a2e9dd5..389f5f83bef9 100644 --- a/arch/arm/boot/dts/aspeed-bmc-opp-romulus.dts +++ b/arch/arm/boot/dts/aspeed-bmc-opp-romulus.dts @@ -68,6 +68,12 @@ gpios = <&gpio ASPEED_GPIO(J, 2) GPIO_ACTIVE_LOW>; linux,code = ; }; + + id-button { + label = "id-button"; + gpios = <&gpio ASPEED_GPIO(Q, 7) GPIO_ACTIVE_LOW>; + linux,code = ; + }; }; }; diff --git a/arch/arm/boot/dts/aspeed-bmc-opp-witherspoon.dts b/arch/arm/boot/dts/aspeed-bmc-opp-witherspoon.dts index 7056231cbee6..78a511e6e482 100644 --- a/arch/arm/boot/dts/aspeed-bmc-opp-witherspoon.dts +++ b/arch/arm/boot/dts/aspeed-bmc-opp-witherspoon.dts @@ -28,6 +28,34 @@ }; }; + gpio-keys { + compatible = "gpio-keys"; + + air-water { + label = "air-water"; + gpios = <&gpio ASPEED_GPIO(B, 5) GPIO_ACTIVE_LOW>; + linux,code = ; + }; + + checkstop { + label = "checkstop"; + gpios = <&gpio ASPEED_GPIO(J, 2) GPIO_ACTIVE_LOW>; + linux,code = ; + }; + + ps0-presence { + label = "ps0-presence"; + gpios = <&gpio ASPEED_GPIO(P, 7) GPIO_ACTIVE_LOW>; + linux,code = ; + }; + + ps1-presence { + label = "ps1-presence"; + gpios = <&gpio ASPEED_GPIO(N, 0) GPIO_ACTIVE_LOW>; + linux,code = ; + }; + }; + gpio-keys-polled { compatible = "gpio-keys-polled"; #address-cells = <1>; @@ -547,6 +575,10 @@ pinctrl-0 = <&pinctrl_wdtrst1_default>; }; +&wdt2 { + aspeed,alt-boot; +}; + &ibt { status = "okay"; }; diff --git a/arch/arm/boot/dts/aspeed-bmc-opp-zaius.dts b/arch/arm/boot/dts/aspeed-bmc-opp-zaius.dts index ebe726a0d311..ccbf645ab84d 100644 --- a/arch/arm/boot/dts/aspeed-bmc-opp-zaius.dts +++ b/arch/arm/boot/dts/aspeed-bmc-opp-zaius.dts @@ -55,6 +55,12 @@ gpios = <&gpio ASPEED_GPIO(F, 7) GPIO_ACTIVE_LOW>; linux,code = ; }; + + pcie-e2b-present{ + label = "pcie-e2b-present"; + gpios = <&gpio ASPEED_GPIO(E, 7) GPIO_ACTIVE_LOW>; + linux,code = ; + }; }; leds { diff --git a/arch/arm/boot/dts/aspeed-bmc-portwell-neptune.dts b/arch/arm/boot/dts/aspeed-bmc-portwell-neptune.dts new file mode 100644 index 000000000000..43ed13963d35 --- /dev/null +++ b/arch/arm/boot/dts/aspeed-bmc-portwell-neptune.dts @@ -0,0 +1,159 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2017 Facebook Inc. +/dts-v1/; + +#include "aspeed-g5.dtsi" +#include + +/ { + model = "Portwell Neptune BMC"; + compatible = "portwell,neptune-bmc", "aspeed,ast2500"; + aliases { + serial0 = &uart1; + serial4 = &uart5; + }; + chosen { + stdout-path = &uart5; + bootargs = "console=ttyS4,115200 earlyprintk"; + }; + + memory { + reg = <0x80000000 0x20000000>; + }; + + leds { + compatible = "gpio-leds"; + postcode0 { + label="BMC_UP"; + gpios = <&gpio ASPEED_GPIO(H, 0) GPIO_ACTIVE_HIGH>; + default-state = "on"; + }; + postcode1 { + label="BMC_HB"; + gpios = <&gpio ASPEED_GPIO(H, 1) GPIO_ACTIVE_HIGH>; + linux,default-trigger = "heartbeat"; + }; + postcode2 { + label="FAULT"; + gpios = <&gpio ASPEED_GPIO(H, 2) GPIO_ACTIVE_HIGH>; + }; + // postcode3-7 are GPIOH3-H7 + }; +}; + +&fmc { + status = "okay"; + flash@0 { + status = "okay"; + m25p,fast-read; +#include "openbmc-flash-layout.dtsi" + }; +}; + +&spi1 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_spi1_default>; + flash@0 { + status = "okay"; + m25p,fast-read; + label = "pnor"; + }; +}; + +&uart1 { + // Host Console + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_txd1_default + &pinctrl_rxd1_default>; +}; + +&uart5 { + // BMC Console + status = "okay"; +}; + +&mac0 { + status = "okay"; + + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_rmii1_default + &pinctrl_mdio1_default>; +}; + +&mac1 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_rmii2_default>; + use-ncsi; +}; + +&i2c1 { + status = "okay"; + // To PCIe slot SMBUS +}; + +&i2c2 { + status = "okay"; + // To LAN I210 +}; + +&i2c3 { + status = "okay"; + // SMBus to COMe AB +}; + +&i2c4 { + status = "okay"; + // I2C to COMe AB +}; + +&i2c5 { + status = "okay"; +// USB Debug card + pca9555@27 { + compatible = "nxp,pca9555"; + reg = <0x27>; + }; +}; + +&i2c6 { + status = "okay"; + tpm@20 { + compatible = "infineon,slb9645tt"; + reg = <0x20>; + }; + tmp421@4e { + compatible = "ti,tmp421"; + reg = <0x4e>; + }; + tmp421@4f { + compatible = "ti,tmp421"; + reg = <0x4f>; + }; +}; + +&i2c8 { + status = "okay"; + eeprom@51 { + compatible = "atmel,24c128"; + reg = <0x51>; + pagesize = <32>; + }; +}; + +&pwm_tacho { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_pwm0_default &pinctrl_pwm1_default>; + fan@0 { + reg = <0x00>; + aspeed,fan-tach-ch = /bits/ 8 <0x00>; + }; + + fan@1 { + reg = <0x00>; + aspeed,fan-tach-ch = /bits/ 8 <0x01>; + }; +}; diff --git a/arch/arm/boot/dts/aspeed-g4.dtsi b/arch/arm/boot/dts/aspeed-g4.dtsi index 518d2bc7c7fc..75df1573380e 100644 --- a/arch/arm/boot/dts/aspeed-g4.dtsi +++ b/arch/arm/boot/dts/aspeed-g4.dtsi @@ -108,6 +108,23 @@ status = "disabled"; }; + ehci0: usb@1e6a1000 { + compatible = "aspeed,ast2400-ehci", "generic-ehci"; + reg = <0x1e6a1000 0x100>; + interrupts = <5>; + clocks = <&syscon ASPEED_CLK_GATE_USBPORT1CLK>; + status = "disabled"; + }; + + uhci: usb@1e6b0000 { + compatible = "aspeed,ast2400-uhci", "generic-uhci"; + reg = <0x1e6b0000 0x100>; + interrupts = <14>; + #ports = <3>; + clocks = <&syscon ASPEED_CLK_GATE_USBUHCICLK>; + status = "disabled"; + }; + apb { compatible = "simple-bus"; #address-cells = <1>; @@ -125,6 +142,14 @@ pinctrl: pinctrl { compatible = "aspeed,g4-pinctrl"; }; + + }; + + rng: hwrng@1e6e2078 { + compatible = "timeriomem_rng"; + reg = <0x1e6e2078 0x4>; + period = <1>; + quality = <100>; }; adc: adc@1e6e9000 { @@ -1250,6 +1275,16 @@ groups = "USBCKI"; }; + pinctrl_usb2h_default: usb2h_default { + function = "USB2H1"; + groups = "USB2H1"; + }; + + pinctrl_usb2d_default: usb2d_default { + function = "USB2D1"; + groups = "USB2D1"; + }; + pinctrl_vgabios_rom_default: vgabios_rom_default { function = "VGABIOS_ROM"; groups = "VGABIOS_ROM"; diff --git a/arch/arm/boot/dts/aspeed-g5.dtsi b/arch/arm/boot/dts/aspeed-g5.dtsi index f9917717dd08..17f2714d18a7 100644 --- a/arch/arm/boot/dts/aspeed-g5.dtsi +++ b/arch/arm/boot/dts/aspeed-g5.dtsi @@ -143,6 +143,31 @@ status = "disabled"; }; + ehci0: usb@1e6a1000 { + compatible = "aspeed,ast2500-ehci", "generic-ehci"; + reg = <0x1e6a1000 0x100>; + interrupts = <5>; + clocks = <&syscon ASPEED_CLK_GATE_USBPORT1CLK>; + status = "disabled"; + }; + + ehci1: usb@1e6a3000 { + compatible = "aspeed,ast2500-ehci", "generic-ehci"; + reg = <0x1e6a3000 0x100>; + interrupts = <13>; + clocks = <&syscon ASPEED_CLK_GATE_USBPORT2CLK>; + status = "disabled"; + }; + + uhci: usb@1e6b0000 { + compatible = "aspeed,ast2500-uhci", "generic-uhci"; + reg = <0x1e6b0000 0x100>; + interrupts = <14>; + #ports = <2>; + clocks = <&syscon ASPEED_CLK_GATE_USBUHCICLK>; + status = "disabled"; + }; + apb { compatible = "simple-bus"; #address-cells = <1>; @@ -164,6 +189,13 @@ }; }; + rng: hwrng@1e6e2078 { + compatible = "timeriomem_rng"; + reg = <0x1e6e2078 0x4>; + period = <1>; + quality = <100>; + }; + gfx: display@1e6e6000 { compatible = "aspeed,ast2500-gfx", "syscon"; reg = <0x1e6e6000 0x1000>; @@ -1380,6 +1412,21 @@ groups = "USBCKI"; }; + pinctrl_usb2ah_default: usb2ah_default { + function = "USB2AH"; + groups = "USB2AH"; + }; + + pinctrl_usb11bhid_default: usb11bhid_default { + function = "USB11BHID"; + groups = "USB11BHID"; + }; + + pinctrl_usb2bh_default: usb2bh_default { + function = "USB2BH"; + groups = "USB2BH"; + }; + pinctrl_vgabiosrom_default: vgabiosrom_default { function = "VGABIOSROM"; groups = "VGABIOSROM"; diff --git a/arch/arm/boot/dts/at91-sama5d2_xplained.dts b/arch/arm/boot/dts/at91-sama5d2_xplained.dts index e4bbb7e0f793..fcc85d70f36e 100644 --- a/arch/arm/boot/dts/at91-sama5d2_xplained.dts +++ b/arch/arm/boot/dts/at91-sama5d2_xplained.dts @@ -232,7 +232,7 @@ pinctrl-names = "default"; pinctrl-0 = <&pinctrl_charger_chglev &pinctrl_charger_lbo &pinctrl_charger_irq>; interrupt-parent = <&pioA>; - interrupts = ; + interrupts = ; active-semi,chglev-gpios = <&pioA PIN_PA12 GPIO_ACTIVE_HIGH>; active-semi,lbo-gpios = <&pioA PIN_PC8 GPIO_ACTIVE_LOW>; diff --git a/arch/arm/boot/dts/at91-sama5d4ek.dts b/arch/arm/boot/dts/at91-sama5d4ek.dts index 7887a7160a54..0702a2f2b336 100644 --- a/arch/arm/boot/dts/at91-sama5d4ek.dts +++ b/arch/arm/boot/dts/at91-sama5d4ek.dts @@ -129,8 +129,8 @@ wakeup-source; }; - atmel_mxt_ts@4c { - compatible = "atmel,atmel_mxt_ts"; + touchscreen@4c { + compatible = "atmel,maxtouch"; reg = <0x4c>; interrupt-parent = <&pioE>; interrupts = <24 0x0>; diff --git a/arch/arm/boot/dts/bcm2835-rpi-a-plus.dts b/arch/arm/boot/dts/bcm2835-rpi-a-plus.dts index aa1fc7babfea..2cd9c5e4f892 100644 --- a/arch/arm/boot/dts/bcm2835-rpi-a-plus.dts +++ b/arch/arm/boot/dts/bcm2835-rpi-a-plus.dts @@ -101,6 +101,12 @@ hpd-gpios = <&gpio 46 GPIO_ACTIVE_LOW>; }; +&pwm { + pinctrl-names = "default"; + pinctrl-0 = <&pwm0_gpio40 &pwm1_gpio45>; + status = "okay"; +}; + &uart0 { pinctrl-names = "default"; pinctrl-0 = <&uart0_gpio14>; diff --git a/arch/arm/boot/dts/bcm2835-rpi-a.dts b/arch/arm/boot/dts/bcm2835-rpi-a.dts index 425f6b0a5ef8..067d1f07a2d3 100644 --- a/arch/arm/boot/dts/bcm2835-rpi-a.dts +++ b/arch/arm/boot/dts/bcm2835-rpi-a.dts @@ -96,6 +96,12 @@ hpd-gpios = <&gpio 46 GPIO_ACTIVE_HIGH>; }; +&pwm { + pinctrl-names = "default"; + pinctrl-0 = <&pwm0_gpio40 &pwm1_gpio45>; + status = "okay"; +}; + &uart0 { pinctrl-names = "default"; pinctrl-0 = <&uart0_gpio14>; diff --git a/arch/arm/boot/dts/bcm2835-rpi-b-plus.dts b/arch/arm/boot/dts/bcm2835-rpi-b-plus.dts index effa195e7895..cfbdaacbaeba 100644 --- a/arch/arm/boot/dts/bcm2835-rpi-b-plus.dts +++ b/arch/arm/boot/dts/bcm2835-rpi-b-plus.dts @@ -103,6 +103,12 @@ hpd-gpios = <&gpio 46 GPIO_ACTIVE_LOW>; }; +&pwm { + pinctrl-names = "default"; + pinctrl-0 = <&pwm0_gpio40 &pwm1_gpio45>; + status = "okay"; +}; + &uart0 { pinctrl-names = "default"; pinctrl-0 = <&uart0_gpio14>; diff --git a/arch/arm/boot/dts/bcm2835-rpi-b-rev2.dts b/arch/arm/boot/dts/bcm2835-rpi-b-rev2.dts index 772ec3b48231..5641d162dfdb 100644 --- a/arch/arm/boot/dts/bcm2835-rpi-b-rev2.dts +++ b/arch/arm/boot/dts/bcm2835-rpi-b-rev2.dts @@ -96,6 +96,12 @@ hpd-gpios = <&gpio 46 GPIO_ACTIVE_LOW>; }; +&pwm { + pinctrl-names = "default"; + pinctrl-0 = <&pwm0_gpio40 &pwm1_gpio45>; + status = "okay"; +}; + &uart0 { pinctrl-names = "default"; pinctrl-0 = <&uart0_gpio14>; diff --git a/arch/arm/boot/dts/bcm2835-rpi-b.dts b/arch/arm/boot/dts/bcm2835-rpi-b.dts index 434483d6fc14..31ff602e2cd3 100644 --- a/arch/arm/boot/dts/bcm2835-rpi-b.dts +++ b/arch/arm/boot/dts/bcm2835-rpi-b.dts @@ -91,6 +91,12 @@ hpd-gpios = <&gpio 46 GPIO_ACTIVE_HIGH>; }; +&pwm { + pinctrl-names = "default"; + pinctrl-0 = <&pwm0_gpio40 &pwm1_gpio45>; + status = "okay"; +}; + &uart0 { pinctrl-names = "default"; pinctrl-0 = <&uart0_gpio14>; diff --git a/arch/arm/boot/dts/bcm2835-rpi.dtsi b/arch/arm/boot/dts/bcm2835-rpi.dtsi index 6c3cfaa77f3d..cb2d6d78a7fb 100644 --- a/arch/arm/boot/dts/bcm2835-rpi.dtsi +++ b/arch/arm/boot/dts/bcm2835-rpi.dtsi @@ -83,12 +83,6 @@ bus-width = <4>; }; -&pwm { - pinctrl-names = "default"; - pinctrl-0 = <&pwm0_gpio40 &pwm1_gpio45>; - status = "okay"; -}; - &usb { power-domains = <&power RPI_POWER_DOMAIN_USB>; }; diff --git a/arch/arm/boot/dts/bcm2836-rpi-2-b.dts b/arch/arm/boot/dts/bcm2836-rpi-2-b.dts index 5c339adabdf0..2fef70a09953 100644 --- a/arch/arm/boot/dts/bcm2836-rpi-2-b.dts +++ b/arch/arm/boot/dts/bcm2836-rpi-2-b.dts @@ -41,6 +41,12 @@ hpd-gpios = <&gpio 46 GPIO_ACTIVE_LOW>; }; +&pwm { + pinctrl-names = "default"; + pinctrl-0 = <&pwm0_gpio40 &pwm1_gpio45>; + status = "okay"; +}; + &uart0 { pinctrl-names = "default"; pinctrl-0 = <&uart0_gpio14>; diff --git a/arch/arm/boot/dts/bcm2837-rpi-3-b-plus.dts b/arch/arm/boot/dts/bcm2837-rpi-3-b-plus.dts new file mode 100644 index 000000000000..4adb85e66be3 --- /dev/null +++ b/arch/arm/boot/dts/bcm2837-rpi-3-b-plus.dts @@ -0,0 +1,108 @@ +// SPDX-License-Identifier: GPL-2.0 +/dts-v1/; +#include "bcm2837.dtsi" +#include "bcm2835-rpi.dtsi" +#include "bcm283x-rpi-lan7515.dtsi" +#include "bcm283x-rpi-usb-host.dtsi" + +/ { + compatible = "raspberrypi,3-model-b-plus", "brcm,bcm2837"; + model = "Raspberry Pi 3 Model B+"; + + chosen { + /* 8250 auxiliary UART instead of pl011 */ + stdout-path = "serial1:115200n8"; + }; + + memory { + reg = <0 0x40000000>; + }; + + leds { + act { + gpios = <&gpio 29 GPIO_ACTIVE_HIGH>; + }; + + pwr { + label = "PWR"; + gpios = <&expgpio 2 GPIO_ACTIVE_LOW>; + }; + }; + + wifi_pwrseq: wifi-pwrseq { + compatible = "mmc-pwrseq-simple"; + reset-gpios = <&expgpio 1 GPIO_ACTIVE_HIGH>; + }; +}; + +&firmware { + expgpio: gpio { + compatible = "raspberrypi,firmware-gpio"; + gpio-controller; + #gpio-cells = <2>; + gpio-line-names = "BT_ON", + "WL_ON", + "STATUS_LED", + "LAN_RUN", + "", + "CAM_GPIO0", + "CAM_GPIO1", + ""; + status = "okay"; + }; +}; + +&hdmi { + hpd-gpios = <&gpio 28 GPIO_ACTIVE_LOW>; +}; + +&pwm { + pinctrl-names = "default"; + pinctrl-0 = <&pwm0_gpio40 &pwm1_gpio41>; + status = "okay"; +}; + +/* SDHCI is used to control the SDIO for wireless */ +&sdhci { + #address-cells = <1>; + #size-cells = <0>; + pinctrl-names = "default"; + pinctrl-0 = <&emmc_gpio34>; + status = "okay"; + bus-width = <4>; + non-removable; + mmc-pwrseq = <&wifi_pwrseq>; + + brcmf: wifi@1 { + reg = <1>; + compatible = "brcm,bcm4329-fmac"; + }; +}; + +/* SDHOST is used to drive the SD card */ +&sdhost { + pinctrl-names = "default"; + pinctrl-0 = <&sdhost_gpio48>; + status = "okay"; + bus-width = <4>; +}; + +/* uart0 communicates with the BT module */ +&uart0 { + pinctrl-names = "default"; + pinctrl-0 = <&uart0_ctsrts_gpio30 &uart0_gpio32 &gpclk2_gpio43>; + status = "okay"; + + bluetooth { + compatible = "brcm,bcm43438-bt"; + max-speed = <2000000>; + shutdown-gpios = <&expgpio 0 GPIO_ACTIVE_HIGH>; + }; +}; + +/* uart1 is mapped to the pin header */ +&uart1 { + pinctrl-names = "default"; + pinctrl-0 = <&uart1_gpio14>; + status = "okay"; +}; diff --git a/arch/arm/boot/dts/bcm2837-rpi-3-b.dts b/arch/arm/boot/dts/bcm2837-rpi-3-b.dts index 0b31d995a066..c318bcbc6ba7 100644 --- a/arch/arm/boot/dts/bcm2837-rpi-3-b.dts +++ b/arch/arm/boot/dts/bcm2837-rpi-3-b.dts @@ -20,9 +20,14 @@ leds { act { - gpios = <&gpio 47 GPIO_ACTIVE_HIGH>; + gpios = <&expgpio 2 GPIO_ACTIVE_HIGH>; }; }; + + wifi_pwrseq: wifi-pwrseq { + compatible = "mmc-pwrseq-simple"; + reset-gpios = <&expgpio 1 GPIO_ACTIVE_HIGH>; + }; }; &firmware { @@ -42,6 +47,16 @@ }; }; +&pwm { + pinctrl-names = "default"; + pinctrl-0 = <&pwm0_gpio40 &pwm1_gpio41>; + status = "okay"; +}; + +&hdmi { + hpd-gpios = <&expgpio 4 GPIO_ACTIVE_LOW>; +}; + /* uart0 communicates with the BT module */ &uart0 { pinctrl-names = "default"; @@ -51,6 +66,7 @@ bluetooth { compatible = "brcm,bcm43438-bt"; max-speed = <2000000>; + shutdown-gpios = <&expgpio 0 GPIO_ACTIVE_HIGH>; }; }; @@ -63,11 +79,19 @@ /* SDHCI is used to control the SDIO for wireless */ &sdhci { + #address-cells = <1>; + #size-cells = <0>; pinctrl-names = "default"; pinctrl-0 = <&emmc_gpio34>; status = "okay"; bus-width = <4>; non-removable; + mmc-pwrseq = <&wifi_pwrseq>; + + brcmf: wifi@1 { + reg = <1>; + compatible = "brcm,bcm4329-fmac"; + }; }; /* SDHOST is used to drive the SD card */ diff --git a/arch/arm/boot/dts/bcm283x-rpi-lan7515.dtsi b/arch/arm/boot/dts/bcm283x-rpi-lan7515.dtsi new file mode 100644 index 000000000000..9403da0990d0 --- /dev/null +++ b/arch/arm/boot/dts/bcm283x-rpi-lan7515.dtsi @@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0 +/ { + aliases { + ethernet0 = ðernet; + }; +}; + +&usb { + usb-port@1 { + compatible = "usb424,2514"; + reg = <1>; + #address-cells = <1>; + #size-cells = <0>; + + usb-port@1 { + compatible = "usb424,2514"; + reg = <1>; + #address-cells = <1>; + #size-cells = <0>; + + ethernet: ethernet@1 { + compatible = "usb424,7800"; + reg = <1>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/bcm283x.dtsi b/arch/arm/boot/dts/bcm283x.dtsi index ac00e730f898..61315cf734ef 100644 --- a/arch/arm/boot/dts/bcm283x.dtsi +++ b/arch/arm/boot/dts/bcm283x.dtsi @@ -136,6 +136,7 @@ rng@7e104000 { compatible = "brcm,bcm2835-rng"; reg = <0x7e104000 0x10>; + interrupts = <2 29>; }; mailbox: mailbox@7e00b880 { diff --git a/arch/arm/boot/dts/bcm4708-asus-rt-ac56u.dts b/arch/arm/boot/dts/bcm4708-asus-rt-ac56u.dts index 4175174e589a..ff2e551b9058 100644 --- a/arch/arm/boot/dts/bcm4708-asus-rt-ac56u.dts +++ b/arch/arm/boot/dts/bcm4708-asus-rt-ac56u.dts @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for Asus RT-AC56U * * Copyright (C) 2015 Rafał Miłecki - * - * Permission to use, copy, modify, and/or distribute this software for any - * purpose with or without fee is hereby granted, provided that the above - * copyright notice and this permission notice appear in all copies. - * - * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH - * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY - * AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, - * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM - * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE - * OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR - * PERFORMANCE OF THIS SOFTWARE. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm4708-asus-rt-ac68u.dts b/arch/arm/boot/dts/bcm4708-asus-rt-ac68u.dts index 8fa033fea959..3bcc03788f38 100644 --- a/arch/arm/boot/dts/bcm4708-asus-rt-ac68u.dts +++ b/arch/arm/boot/dts/bcm4708-asus-rt-ac68u.dts @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for Asus RT-AC68U * * Copyright (C) 2015 Rafał Miłecki - * - * Permission to use, copy, modify, and/or distribute this software for any - * purpose with or without fee is hereby granted, provided that the above - * copyright notice and this permission notice appear in all copies. - * - * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH - * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY - * AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, - * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM - * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE - * OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR - * PERFORMANCE OF THIS SOFTWARE. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm4708-buffalo-wzr-1750dhp.dts b/arch/arm/boot/dts/bcm4708-buffalo-wzr-1750dhp.dts index 8b64caabaad8..a587384f8e40 100644 --- a/arch/arm/boot/dts/bcm4708-buffalo-wzr-1750dhp.dts +++ b/arch/arm/boot/dts/bcm4708-buffalo-wzr-1750dhp.dts @@ -1,10 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for Buffalo WZR-1750DHP * * Copyright (C) 2014 Rafał Miłecki - * - * Licensed under the GNU/GPL. See COPYING for details. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm4708-linksys-ea6300-v1.dts b/arch/arm/boot/dts/bcm4708-linksys-ea6300-v1.dts index 126ab5867772..6c8f0ad82332 100644 --- a/arch/arm/boot/dts/bcm4708-linksys-ea6300-v1.dts +++ b/arch/arm/boot/dts/bcm4708-linksys-ea6300-v1.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright (C) 2017 Rafał Miłecki - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm4708-luxul-xap-1510.dts b/arch/arm/boot/dts/bcm4708-luxul-xap-1510.dts index f591b0f256d8..ebda45fe9ea4 100644 --- a/arch/arm/boot/dts/bcm4708-luxul-xap-1510.dts +++ b/arch/arm/boot/dts/bcm4708-luxul-xap-1510.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright 2016 Luxul Inc. - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm4708-luxul-xwc-1000.dts b/arch/arm/boot/dts/bcm4708-luxul-xwc-1000.dts index 50d65d8fbd9a..9dd0e22c906a 100644 --- a/arch/arm/boot/dts/bcm4708-luxul-xwc-1000.dts +++ b/arch/arm/boot/dts/bcm4708-luxul-xwc-1000.dts @@ -1,10 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for Luxul XWC-1000 * * Copyright 2014 Luxul Inc. - * - * Licensed under the GNU/GPL. See COPYING for details. */ /dts-v1/; @@ -26,9 +25,15 @@ nand: nand@18028000 { nandcs@0 { - partition@0 { - label = "ubi"; - reg = <0x00000000 0x08000000>; + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "ubi"; + reg = <0x00000000 0x08000000>; + }; }; }; }; diff --git a/arch/arm/boot/dts/bcm4708-netgear-r6300-v2.dts b/arch/arm/boot/dts/bcm4708-netgear-r6300-v2.dts index bb66cebe0bd8..2642494c97a1 100644 --- a/arch/arm/boot/dts/bcm4708-netgear-r6300-v2.dts +++ b/arch/arm/boot/dts/bcm4708-netgear-r6300-v2.dts @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for Netgear R6300 V2 * * Copyright (C) 2014 Rafał Miłecki - * - * Permission to use, copy, modify, and/or distribute this software for any - * purpose with or without fee is hereby granted, provided that the above - * copyright notice and this permission notice appear in all copies. - * - * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH - * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY - * AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, - * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM - * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE - * OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR - * PERFORMANCE OF THIS SOFTWARE. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm4708-smartrg-sr400ac.dts b/arch/arm/boot/dts/bcm4708-smartrg-sr400ac.dts index 83a4c60bb431..e7b09b7b7d25 100644 --- a/arch/arm/boot/dts/bcm4708-smartrg-sr400ac.dts +++ b/arch/arm/boot/dts/bcm4708-smartrg-sr400ac.dts @@ -1,10 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X arm platform code. * DTS for SmartRG SR400ac * * Copyright (C) 2015 Rafał Miłecki - * - * Licensed under the GNU/GPL. See COPYING for details. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm47081-asus-rt-n18u.dts b/arch/arm/boot/dts/bcm47081-asus-rt-n18u.dts index 0800a964f2fe..16314fcc6e56 100644 --- a/arch/arm/boot/dts/bcm47081-asus-rt-n18u.dts +++ b/arch/arm/boot/dts/bcm47081-asus-rt-n18u.dts @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for Asus RT-N18U * * Copyright (C) 2014 Rafał Miłecki - * - * Permission to use, copy, modify, and/or distribute this software for any - * purpose with or without fee is hereby granted, provided that the above - * copyright notice and this permission notice appear in all copies. - * - * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH - * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY - * AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, - * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM - * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE - * OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR - * PERFORMANCE OF THIS SOFTWARE. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm47081-buffalo-wzr-600dhp2.dts b/arch/arm/boot/dts/bcm47081-buffalo-wzr-600dhp2.dts index c2af33eb47de..328aa90240ce 100644 --- a/arch/arm/boot/dts/bcm47081-buffalo-wzr-600dhp2.dts +++ b/arch/arm/boot/dts/bcm47081-buffalo-wzr-600dhp2.dts @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for Buffalo WZR-600DHP2 * * Copyright (C) 2014 Rafał Miłecki - * - * Permission to use, copy, modify, and/or distribute this software for any - * purpose with or without fee is hereby granted, provided that the above - * copyright notice and this permission notice appear in all copies. - * - * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH - * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY - * AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, - * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM - * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE - * OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR - * PERFORMANCE OF THIS SOFTWARE. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm47081-buffalo-wzr-900dhp.dts b/arch/arm/boot/dts/bcm47081-buffalo-wzr-900dhp.dts index 87ea6ba664f5..8ea46eed26e2 100644 --- a/arch/arm/boot/dts/bcm47081-buffalo-wzr-900dhp.dts +++ b/arch/arm/boot/dts/bcm47081-buffalo-wzr-900dhp.dts @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for Buffalo WZR-900DHP * * Copyright (C) 2015 Rafał Miłecki - * - * Permission to use, copy, modify, and/or distribute this software for any - * purpose with or without fee is hereby granted, provided that the above - * copyright notice and this permission notice appear in all copies. - * - * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH - * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY - * AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, - * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM - * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE - * OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR - * PERFORMANCE OF THIS SOFTWARE. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm47081-luxul-xap-1410.dts b/arch/arm/boot/dts/bcm47081-luxul-xap-1410.dts index 9b5759849983..5eeac7302329 100644 --- a/arch/arm/boot/dts/bcm47081-luxul-xap-1410.dts +++ b/arch/arm/boot/dts/bcm47081-luxul-xap-1410.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright 2017 Luxul Inc. - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm47081-luxul-xwr-1200.dts b/arch/arm/boot/dts/bcm47081-luxul-xwr-1200.dts index ba1c19b1b3eb..da4d9ec62fc6 100644 --- a/arch/arm/boot/dts/bcm47081-luxul-xwr-1200.dts +++ b/arch/arm/boot/dts/bcm47081-luxul-xwr-1200.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright 2017 Luxul Inc. - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm47081-tplink-archer-c5-v2.dts b/arch/arm/boot/dts/bcm47081-tplink-archer-c5-v2.dts index 3ed8de42cb48..c94c732188fb 100644 --- a/arch/arm/boot/dts/bcm47081-tplink-archer-c5-v2.dts +++ b/arch/arm/boot/dts/bcm47081-tplink-archer-c5-v2.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright (C) 2017 Rafał Miłecki - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm4709-asus-rt-ac87u.dts b/arch/arm/boot/dts/bcm4709-asus-rt-ac87u.dts index df473cc41572..22271818f901 100644 --- a/arch/arm/boot/dts/bcm4709-asus-rt-ac87u.dts +++ b/arch/arm/boot/dts/bcm4709-asus-rt-ac87u.dts @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for Asus RT-AC87U * * Copyright (C) 2015 Rafał Miłecki - * - * Permission to use, copy, modify, and/or distribute this software for any - * purpose with or without fee is hereby granted, provided that the above - * copyright notice and this permission notice appear in all copies. - * - * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH - * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY - * AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, - * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM - * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE - * OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR - * PERFORMANCE OF THIS SOFTWARE. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm4709-buffalo-wxr-1900dhp.dts b/arch/arm/boot/dts/bcm4709-buffalo-wxr-1900dhp.dts index 92058c73ee59..79a9633ec417 100644 --- a/arch/arm/boot/dts/bcm4709-buffalo-wxr-1900dhp.dts +++ b/arch/arm/boot/dts/bcm4709-buffalo-wxr-1900dhp.dts @@ -1,10 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for Buffalo WXR-1900DHP * * Copyright (C) 2015 Felix Fietkau - * - * Licensed under the GNU/GPL. See COPYING for details. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm4709-linksys-ea9200.dts b/arch/arm/boot/dts/bcm4709-linksys-ea9200.dts index 3d1d9c2c4efc..db744a5e122d 100644 --- a/arch/arm/boot/dts/bcm4709-linksys-ea9200.dts +++ b/arch/arm/boot/dts/bcm4709-linksys-ea9200.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright (C) 2017 Rafał Miłecki - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm4709-netgear-r7000.dts b/arch/arm/boot/dts/bcm4709-netgear-r7000.dts index f43ab4721456..9e267d38df4c 100644 --- a/arch/arm/boot/dts/bcm4709-netgear-r7000.dts +++ b/arch/arm/boot/dts/bcm4709-netgear-r7000.dts @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for Netgear R7000 * * Copyright (C) 2015 Rafał Miłecki - * - * Permission to use, copy, modify, and/or distribute this software for any - * purpose with or without fee is hereby granted, provided that the above - * copyright notice and this permission notice appear in all copies. - * - * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH - * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY - * AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, - * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM - * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE - * OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR - * PERFORMANCE OF THIS SOFTWARE. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm4709-tplink-archer-c9-v1.dts b/arch/arm/boot/dts/bcm4709-tplink-archer-c9-v1.dts index ec4a50e440f6..f5bf6586ae07 100644 --- a/arch/arm/boot/dts/bcm4709-tplink-archer-c9-v1.dts +++ b/arch/arm/boot/dts/bcm4709-tplink-archer-c9-v1.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright (C) 2016 Rafał Miłecki - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm47094-dlink-dir-885l.dts b/arch/arm/boot/dts/bcm47094-dlink-dir-885l.dts index 7cc7d344fe5b..d173bcd93b91 100644 --- a/arch/arm/boot/dts/bcm47094-dlink-dir-885l.dts +++ b/arch/arm/boot/dts/bcm47094-dlink-dir-885l.dts @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X ARM platform code. * DTS for D-Link DIR-885L * * Copyright (C) 2016 Rafał Miłecki - * - * Permission to use, copy, modify, and/or distribute this software for any - * purpose with or without fee is hereby granted, provided that the above - * copyright notice and this permission notice appear in all copies. - * - * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH - * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY - * AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, - * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM - * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE - * OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR - * PERFORMANCE OF THIS SOFTWARE. */ /dts-v1/; @@ -37,9 +26,15 @@ nand: nand@18028000 { nandcs@0 { - partition@0 { - label = "firmware"; - reg = <0x00000000 0x08000000>; + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "firmware"; + reg = <0x00000000 0x08000000>; + }; }; }; }; diff --git a/arch/arm/boot/dts/bcm47094-linksys-panamera.dts b/arch/arm/boot/dts/bcm47094-linksys-panamera.dts index b6750f70dffb..f47afe36d857 100644 --- a/arch/arm/boot/dts/bcm47094-linksys-panamera.dts +++ b/arch/arm/boot/dts/bcm47094-linksys-panamera.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright (C) 2017 Rafał Miłecki - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm47094-luxul-abr-4500.dts b/arch/arm/boot/dts/bcm47094-luxul-abr-4500.dts index ecd22a246746..a5cef51cfe4f 100644 --- a/arch/arm/boot/dts/bcm47094-luxul-abr-4500.dts +++ b/arch/arm/boot/dts/bcm47094-luxul-abr-4500.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright (C) 2017 Luxul Inc. - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm47094-luxul-xap-1610.dts b/arch/arm/boot/dts/bcm47094-luxul-xap-1610.dts new file mode 100644 index 000000000000..7fd85475893d --- /dev/null +++ b/arch/arm/boot/dts/bcm47094-luxul-xap-1610.dts @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT +/* + * Copyright 2018 Luxul Inc. + */ + +/dts-v1/; + +#include "bcm47094.dtsi" + +/ { + compatible = "luxul,xap-1610-v1", "brcm,bcm47094", "brcm,bcm4708"; + model = "Luxul XAP-1610 V1"; + + chosen { + bootargs = "earlycon"; + }; + + memory { + reg = <0x00000000 0x08000000>; + }; + + leds { + compatible = "gpio-leds"; + + status { + label = "bcm53xx:green:status"; + gpios = <&chipcommon 0 GPIO_ACTIVE_LOW>; + linux,default-trigger = "timer"; + }; + + 2ghz { + label = "bcm53xx:blue:2ghz"; + gpios = <&chipcommon 13 GPIO_ACTIVE_LOW>; + }; + + 5ghz { + label = "bcm53xx:blue:5ghz"; + gpios = <&chipcommon 14 GPIO_ACTIVE_LOW>; + }; + }; + + gpio-keys { + compatible = "gpio-keys"; + #address-cells = <1>; + #size-cells = <0>; + + restart { + label = "Reset"; + linux,code = ; + gpios = <&chipcommon 17 GPIO_ACTIVE_LOW>; + }; + }; +}; + +&spi_nor { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/bcm47094-luxul-xbr-4500.dts b/arch/arm/boot/dts/bcm47094-luxul-xbr-4500.dts index 15ffb1abc440..7496aabf8f77 100644 --- a/arch/arm/boot/dts/bcm47094-luxul-xbr-4500.dts +++ b/arch/arm/boot/dts/bcm47094-luxul-xbr-4500.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright (C) 2017 Luxul Inc. - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm47094-luxul-xwr-3100.dts b/arch/arm/boot/dts/bcm47094-luxul-xwr-3100.dts index bc1d1e10d4ac..53aaa5212610 100644 --- a/arch/arm/boot/dts/bcm47094-luxul-xwr-3100.dts +++ b/arch/arm/boot/dts/bcm47094-luxul-xwr-3100.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright 2016 Luxul Inc. - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm47094-luxul-xwr-3150-v1.dts b/arch/arm/boot/dts/bcm47094-luxul-xwr-3150-v1.dts new file mode 100644 index 000000000000..bdad7267255a --- /dev/null +++ b/arch/arm/boot/dts/bcm47094-luxul-xwr-3150-v1.dts @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT +/* + * Copyright 2018 Luxul Inc. + */ + +/dts-v1/; + +#include "bcm47094.dtsi" +#include "bcm5301x-nand-cs0-bch8.dtsi" + +/ { + compatible = "luxul,xwr-3150-v1", "brcm,bcm47094", "brcm,bcm4708"; + model = "Luxul XWR-3150 V1"; + + chosen { + bootargs = "earlycon"; + }; + + memory { + reg = <0x00000000 0x08000000 + 0x88000000 0x18000000>; + }; + + leds { + compatible = "gpio-leds"; + + power { + label = "bcm53xx:green:power"; + gpios = <&chipcommon 0 GPIO_ACTIVE_LOW>; + linux,default-trigger = "default-on"; + }; + + usb3 { + label = "bcm53xx:green:usb3"; + gpios = <&chipcommon 8 GPIO_ACTIVE_LOW>; + trigger-sources = <&ohci_port1>, <&ehci_port1>, + <&xhci_port1>; + linux,default-trigger = "usbport"; + }; + + status { + label = "bcm53xx:green:status"; + gpios = <&chipcommon 10 GPIO_ACTIVE_LOW>; + linux,default-trigger = "timer"; + }; + + 2ghz { + label = "bcm53xx:green:2ghz"; + gpios = <&chipcommon 13 GPIO_ACTIVE_LOW>; + }; + + 5ghz { + label = "bcm53xx:green:5ghz"; + gpios = <&chipcommon 14 GPIO_ACTIVE_LOW>; + }; + }; + + gpio-keys { + compatible = "gpio-keys"; + #address-cells = <1>; + #size-cells = <0>; + + restart { + label = "Reset"; + linux,code = ; + gpios = <&chipcommon 17 GPIO_ACTIVE_LOW>; + }; + }; +}; + +&usb3 { + vcc-gpio = <&chipcommon 18 GPIO_ACTIVE_HIGH>; +}; + +&spi_nor { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/bcm47094-netgear-r8500.dts b/arch/arm/boot/dts/bcm47094-netgear-r8500.dts index 859929973158..0e718edc065a 100644 --- a/arch/arm/boot/dts/bcm47094-netgear-r8500.dts +++ b/arch/arm/boot/dts/bcm47094-netgear-r8500.dts @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright (C) 2016 Rafał Miłecki - * - * Licensed under the ISC license. */ /dts-v1/; diff --git a/arch/arm/boot/dts/bcm5301x-nand-cs0-bch1.dtsi b/arch/arm/boot/dts/bcm5301x-nand-cs0-bch1.dtsi index 24b099c00f13..c349e8f0afc5 100644 --- a/arch/arm/boot/dts/bcm5301x-nand-cs0-bch1.dtsi +++ b/arch/arm/boot/dts/bcm5301x-nand-cs0-bch1.dtsi @@ -1,9 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom Northstar NAND. * * Copyright (C) 2016 Rafał Miłecki - * - * Licensed under the ISC license. */ #include "bcm5301x-nand-cs0.dtsi" diff --git a/arch/arm/boot/dts/bcm5301x-nand-cs0-bch4.dtsi b/arch/arm/boot/dts/bcm5301x-nand-cs0-bch4.dtsi index b4e875df9528..18e25e302b13 100644 --- a/arch/arm/boot/dts/bcm5301x-nand-cs0-bch4.dtsi +++ b/arch/arm/boot/dts/bcm5301x-nand-cs0-bch4.dtsi @@ -1,7 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Copyright 2016 Luxul Inc. - * - * Licensed under the ISC license. */ #include "bcm5301x-nand-cs0.dtsi" diff --git a/arch/arm/boot/dts/bcm5301x-nand-cs0-bch8.dtsi b/arch/arm/boot/dts/bcm5301x-nand-cs0-bch8.dtsi index 9a9630ded306..c8e56d30bd6f 100644 --- a/arch/arm/boot/dts/bcm5301x-nand-cs0-bch8.dtsi +++ b/arch/arm/boot/dts/bcm5301x-nand-cs0-bch8.dtsi @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom BCM470X / BCM5301X Nand chip defaults. * @@ -5,8 +6,6 @@ * and uses 8 bit ECC. * * Copyright (C) 2015 Hauke Mehrtens - * - * Licensed under the GNU/GPL. See COPYING for details. */ #include "bcm5301x-nand-cs0.dtsi" diff --git a/arch/arm/boot/dts/bcm5301x-nand-cs0.dtsi b/arch/arm/boot/dts/bcm5301x-nand-cs0.dtsi index 168495106b82..e5a2d62daf92 100644 --- a/arch/arm/boot/dts/bcm5301x-nand-cs0.dtsi +++ b/arch/arm/boot/dts/bcm5301x-nand-cs0.dtsi @@ -1,9 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0-or-later OR MIT /* * Broadcom Northstar NAND. * * Copyright (C) 2015 Hauke Mehrtens - * - * Licensed under the GNU/GPL. See COPYING for details. */ / { diff --git a/arch/arm/boot/dts/berlin2-sony-nsz-gs7.dts b/arch/arm/boot/dts/berlin2-sony-nsz-gs7.dts index 1c475796d17f..64a297759eb9 100644 --- a/arch/arm/boot/dts/berlin2-sony-nsz-gs7.dts +++ b/arch/arm/boot/dts/berlin2-sony-nsz-gs7.dts @@ -1,39 +1,8 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) /* * Device Tree file for Sony NSZ-GS7 * * Sebastian Hesselbarth - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. */ /dts-v1/; diff --git a/arch/arm/boot/dts/berlin2.dtsi b/arch/arm/boot/dts/berlin2.dtsi index d575823c5750..db67377af266 100644 --- a/arch/arm/boot/dts/berlin2.dtsi +++ b/arch/arm/boot/dts/berlin2.dtsi @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) /* * Device Tree Include file for Marvell Armada 1500 (Berlin BG2) SoC * @@ -5,38 +6,6 @@ * * based on GPL'ed 2.6 kernel sources * (c) Marvell International Ltd. - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. */ #include @@ -149,7 +118,7 @@ local-timer@ad0600 { compatible = "arm,cortex-a9-twd-timer"; reg = <0xad0600 0x20>; - interrupts = ; + interrupts = ; clocks = <&chip_clk CLKID_TWD>; }; diff --git a/arch/arm/boot/dts/berlin2cd-google-chromecast.dts b/arch/arm/boot/dts/berlin2cd-google-chromecast.dts index ca24def0ce13..56fa951bc86f 100644 --- a/arch/arm/boot/dts/berlin2cd-google-chromecast.dts +++ b/arch/arm/boot/dts/berlin2cd-google-chromecast.dts @@ -1,39 +1,8 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) /* * Device Tree file for Google Chromecast * * Sebastian Hesselbarth - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. */ /dts-v1/; @@ -52,22 +21,35 @@ memory@0 { device_type = "memory"; - reg = <0x00000000 0x20000000>; /* 512 MB */ + + /* + * We're using "linux,usable-memory" instead of "reg" here + * because the (signed and encrypted) bootloader that shipped + * with this device provides an incorrect memory range in + * ATAG_MEM. Linux helpfully overrides the "reg" property with + * data from the ATAG, so we can't specify the proper range + * normally. Fortunately, this alternate property is checked + * first by the OF driver, so we can (ab)use it instead. + */ + linux,usable-memory = <0x00000000 0x20000000>; /* 512 MB */ }; leds { - compatible = "gpio-leds"; + compatible = "pwm-leds"; + pinctrl-0 = <&ledpwm_pmux>; + pinctrl-names = "default"; white { label = "white"; - gpios = <&portc 1 GPIO_ACTIVE_HIGH>; - default-state = "keep"; + pwms = <&pwm 0 600000 0>; + max-brightness = <255>; + linux,default-trigger = "default-on"; }; red { label = "red"; - gpios = <&portc 2 GPIO_ACTIVE_HIGH>; - default-state = "keep"; + pwms = <&pwm 1 600000 0>; + max-brightness = <255>; }; }; }; @@ -86,3 +68,10 @@ &usb_phy1 { status = "okay"; }; &usb1 { status = "okay"; }; + +&soc_pinctrl { + ledpwm_pmux: ledpwm-pmux { + groups = "G0"; + function = "pwm"; + }; +}; diff --git a/arch/arm/boot/dts/berlin2cd-valve-steamlink.dts b/arch/arm/boot/dts/berlin2cd-valve-steamlink.dts new file mode 100644 index 000000000000..79ac842ae461 --- /dev/null +++ b/arch/arm/boot/dts/berlin2cd-valve-steamlink.dts @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) +/* + * Copyright 2018 Alexander Monakov + */ +/dts-v1/; + +#include "berlin2cd.dtsi" +#include + +/ { + model = "Valve Steam Link"; + compatible = "valve,steamlink", "marvell,berlin2cd", "marvell,berlin"; + + memory@0 { + device_type = "memory"; + reg = <0x00000000 0x20000000>; /* 512 MB */ + }; + + gpio-restart { + compatible = "gpio-restart"; + gpios = <&porta 6 GPIO_ACTIVE_HIGH>; + active-delay = <100>; + inactive-delay = <10>; + wait-delay = <100>; + priority = <200>; + }; +}; + +&cpu { + cpu-supply = <&vcpu>; + operating-points = < + /* kHz uV */ + 1000000 1325000 + >; +}; + +&i2c0 { + status = "okay"; + + /* There are two regulators on the board. One is accessible via I2C, + * with buck1 providing SoC power (set up by bootloader to 1.325V or + * less depending on leakage value in OTP), and buck2 likely used for + * DRAM (providing 1.35V). The other regulator on the opposite side + * of the board is probably supplying SDIO and NAND fixed voltages. */ + regulator@19 { + compatible = "marvell,88pg868"; + reg = <0x19>; + + vcpu: buck1 { + regulator-boot-on; + regulator-always-on; + regulator-min-microvolt = <1000000>; + regulator-max-microvolt = <1325000>; + }; + }; +}; + +/* Fixed interface to on-board Marvell 8897 Wi-Fi/Bluetooth/NFC chip. */ +&sdhci0 { + keep-power-in-suspend; + non-removable; + status = "okay"; +}; + +&uart0 { + /* RX/TX are routed to TP50/TP51 on the board. */ + status = "okay"; +}; + +/* The SoC is connected to on-board USB hub that in turn has one downstream + * port wired to the on-board Steam Controller wireless receiver chip. */ +&usb_phy1 { status = "okay"; }; + +&usb1 { + dr_mode = "host"; + status = "okay"; +}; + +ð1 { status = "okay"; }; diff --git a/arch/arm/boot/dts/berlin2cd.dtsi b/arch/arm/boot/dts/berlin2cd.dtsi index 501c59d97eae..e5c1f4213ff9 100644 --- a/arch/arm/boot/dts/berlin2cd.dtsi +++ b/arch/arm/boot/dts/berlin2cd.dtsi @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) /* * Device Tree Include file for Marvell Armada 1500-mini (Berlin BG2CD) SoC * @@ -5,38 +6,6 @@ * * based on GPL'ed 2.6 kernel sources * (c) Marvell International Ltd. - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. */ #include @@ -57,7 +26,7 @@ #address-cells = <1>; #size-cells = <0>; - cpu@0 { + cpu: cpu@0 { compatible = "arm,cortex-a9"; device_type = "cpu"; next-level-cache = <&l2>; @@ -73,6 +42,12 @@ }; }; + pmu { + compatible = "arm,cortex-a9-pmu"; + interrupt-parent = <&gic>; + interrupts = ; + }; + refclk: oscillator { compatible = "fixed-clock"; #clock-cells = <0>; @@ -87,11 +62,6 @@ ranges = <0 0xf7000000 0x1000000>; - pmu { - compatible = "arm,cortex-a9-pmu"; - interrupts = ; - }; - sdhci0: sdhci@ab0000 { compatible = "mrvl,pxav3-mmc"; reg = <0xab0000 0x200>; @@ -108,6 +78,11 @@ cache-level = <2>; }; + snoop-control-unit@ad0000 { + compatible = "arm,cortex-a9-scu"; + reg = <0xad0000 0x100>; + }; + gic: interrupt-controller@ad1000 { compatible = "arm,cortex-a9-gic"; reg = <0xad1000 0x1000>, <0xad0100 0x0100>; @@ -115,10 +90,24 @@ #interrupt-cells = <3>; }; + global-timer@ad0200 { + compatible = "arm,cortex-a9-global-timer"; + reg = <0xad0200 0x20>; + interrupts = ; + clocks = <&chip_clk CLKID_TWD>; + }; + local-timer@ad0600 { compatible = "arm,cortex-a9-twd-timer"; reg = <0xad0600 0x20>; - interrupts = ; + interrupts = ; + clocks = <&chip_clk CLKID_TWD>; + }; + + local-wdt@ad0620 { + compatible = "arm,cortex-a9-twd-wdt"; + reg = <0xad0620 0x20>; + interrupts = ; clocks = <&chip_clk CLKID_TWD>; }; @@ -254,6 +243,60 @@ }; }; + i2c0: i2c@1400 { + compatible = "snps,designware-i2c"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x1400 0x100>; + interrupts = <16>; + clocks = <&chip_clk CLKID_CFG>; + status = "disabled"; + }; + + i2c1: i2c@1800 { + compatible = "snps,designware-i2c"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x1800 0x100>; + interrupts = <17>; + clocks = <&chip_clk CLKID_CFG>; + status = "disabled"; + }; + + spi0: spi@1c00 { + compatible = "snps,dw-apb-ssi"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x1c00 0x100>; + interrupts = <4>; + clocks = <&chip_clk CLKID_CFG>; + status = "disabled"; + }; + + wdt4: watchdog@2000 { + compatible = "snps,dw-wdt"; + reg = <0x2000 0x100>; + clocks = <&chip_clk CLKID_CFG>; + interrupts = <5>; + status = "disabled"; + }; + + wdt5: watchdog@2400 { + compatible = "snps,dw-wdt"; + reg = <0x2400 0x100>; + clocks = <&chip_clk CLKID_CFG>; + interrupts = <6>; + status = "disabled"; + }; + + wdt6: watchdog@2800 { + compatible = "snps,dw-wdt"; + reg = <0x2800 0x100>; + clocks = <&chip_clk CLKID_CFG>; + interrupts = <7>; + status = "disabled"; + }; + timer0: timer@2c00 { compatible = "snps,dw-apb-timer"; reg = <0x2c00 0x14>; @@ -435,6 +478,36 @@ }; }; + spi1: spi@6000 { + compatible = "snps,dw-apb-ssi"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x6000 0x100>; + clocks = <&refclk>; + interrupts = <5>; + status = "disabled"; + }; + + i2c2: i2c@7000 { + compatible = "snps,designware-i2c"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x7000 0x100>; + interrupts = <6>; + clocks = <&refclk>; + status = "disabled"; + }; + + i2c3: i2c@8000 { + compatible = "snps,designware-i2c"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x8000 0x100>; + interrupts = <7>; + clocks = <&refclk>; + status = "disabled"; + }; + sm_gpio0: gpio@c000 { compatible = "snps,dw-apb-gpio"; reg = <0xc000 0x400>; @@ -472,6 +545,16 @@ status = "disabled"; }; + uart2: serial@b000 { + compatible = "snps,dw-apb-uart"; + reg = <0xb000 0x100>; + reg-shift = <2>; + reg-io-width = <1>; + interrupts = <10>; + clocks = <&refclk>; + status = "disabled"; + }; + sysctrl: system-controller@d000 { compatible = "simple-mfd", "syscon"; reg = <0xd000 0x100>; @@ -479,6 +562,12 @@ sys_pinctrl: pin-controller { compatible = "marvell,berlin2cd-system-pinctrl"; }; + + adc: adc { + compatible = "marvell,berlin2-adc"; + interrupts = <12>, <14>; + interrupt-names = "adc", "tsen"; + }; }; sic: interrupt-controller@e000 { diff --git a/arch/arm/boot/dts/berlin2q-marvell-dmp.dts b/arch/arm/boot/dts/berlin2q-marvell-dmp.dts index 57aa5f8a7c77..c162f98cb8e8 100644 --- a/arch/arm/boot/dts/berlin2q-marvell-dmp.dts +++ b/arch/arm/boot/dts/berlin2q-marvell-dmp.dts @@ -1,37 +1,6 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) /* * Copyright (C) 2014 Antoine Ténart - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. */ /dts-v1/; diff --git a/arch/arm/boot/dts/berlin2q.dtsi b/arch/arm/boot/dts/berlin2q.dtsi index bf3a6c9a1d34..516a7ce25791 100644 --- a/arch/arm/boot/dts/berlin2q.dtsi +++ b/arch/arm/boot/dts/berlin2q.dtsi @@ -1,37 +1,6 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) /* * Copyright (C) 2014 Antoine Ténart - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. */ #include @@ -53,7 +22,7 @@ #size-cells = <0>; enable-method = "marvell,berlin-smp"; - cpu@0 { + cpu0: cpu@0 { compatible = "arm,cortex-a9"; device_type = "cpu"; next-level-cache = <&l2>; @@ -71,21 +40,21 @@ >; }; - cpu@1 { + cpu1: cpu@1 { compatible = "arm,cortex-a9"; device_type = "cpu"; next-level-cache = <&l2>; reg = <1>; }; - cpu@2 { + cpu2: cpu@2 { compatible = "arm,cortex-a9"; device_type = "cpu"; next-level-cache = <&l2>; reg = <2>; }; - cpu@3 { + cpu3: cpu@3 { compatible = "arm,cortex-a9"; device_type = "cpu"; next-level-cache = <&l2>; @@ -93,6 +62,19 @@ }; }; + pmu { + compatible = "arm,cortex-a9-pmu"; + interrupt-parent = <&gic>; + interrupts = , + , + , + ; + interrupt-affinity = <&cpu0>, + <&cpu1>, + <&cpu2>, + <&cpu3>; + }; + refclk: oscillator { compatible = "fixed-clock"; #clock-cells = <0>; @@ -107,14 +89,6 @@ ranges = <0 0xf7000000 0x1000000>; interrupt-parent = <&gic>; - pmu { - compatible = "arm,cortex-a9-pmu"; - interrupts = , - , - , - ; - }; - sdhci0: sdhci@ab0000 { compatible = "mrvl,pxav3-mmc"; reg = <0xab0000 0x200>; @@ -145,6 +119,7 @@ l2: l2-cache-controller@ac0000 { compatible = "arm,pl310-cache"; reg = <0xac0000 0x1000>; + cache-unified; cache-level = <2>; arm,data-latency = <2 2 2>; arm,tag-latency = <2 2 2>; @@ -159,7 +134,7 @@ compatible = "arm,cortex-a9-twd-timer"; reg = <0xad0600 0x20>; clocks = <&chip_clk CLKID_TWD>; - interrupts = ; + interrupts = ; }; gic: interrupt-controller@ad1000 { diff --git a/arch/arm/boot/dts/da850-evm.dts b/arch/arm/boot/dts/da850-evm.dts index 3962fa4b07f5..0e82bb988fde 100644 --- a/arch/arm/boot/dts/da850-evm.dts +++ b/arch/arm/boot/dts/da850-evm.dts @@ -27,143 +27,6 @@ spi0 = &spi1; }; - soc@1c00000 { - pmx_core: pinmux@14120 { - status = "okay"; - - mcasp0_pins: pinmux_mcasp0_pins { - pinctrl-single,bits = < - /* - * AHCLKX, ACLKX, AFSX, AHCLKR, ACLKR, - * AFSR, AMUTE - */ - 0x00 0x11111111 0xffffffff - /* AXR11, AXR12 */ - 0x04 0x00011000 0x000ff000 - >; - }; - nand_pins: nand_pins { - pinctrl-single,bits = < - /* EMA_WAIT[0], EMA_OE, EMA_WE, EMA_CS[4], EMA_CS[3] */ - 0x1c 0x10110110 0xf0ff0ff0 - /* - * EMA_D[0], EMA_D[1], EMA_D[2], - * EMA_D[3], EMA_D[4], EMA_D[5], - * EMA_D[6], EMA_D[7] - */ - 0x24 0x11111111 0xffffffff - /* EMA_A[1], EMA_A[2] */ - 0x30 0x01100000 0x0ff00000 - >; - }; - }; - serial0: serial@42000 { - status = "okay"; - }; - serial1: serial@10c000 { - status = "okay"; - }; - serial2: serial@10d000 { - status = "okay"; - }; - rtc0: rtc@23000 { - status = "okay"; - }; - i2c0: i2c@22000 { - status = "okay"; - clock-frequency = <100000>; - pinctrl-names = "default"; - pinctrl-0 = <&i2c0_pins>; - - tps: tps@48 { - reg = <0x48>; - }; - tlv320aic3106: tlv320aic3106@18 { - #sound-dai-cells = <0>; - compatible = "ti,tlv320aic3106"; - reg = <0x18>; - status = "okay"; - - /* Regulators */ - IOVDD-supply = <&vdcdc2_reg>; - /* Derived from VBAT: Baseboard 3.3V / 1.8V */ - AVDD-supply = <&vbat>; - DRVDD-supply = <&vbat>; - DVDD-supply = <&vbat>; - }; - tca6416: gpio@20 { - compatible = "ti,tca6416"; - reg = <0x20>; - gpio-controller; - #gpio-cells = <2>; - }; - }; - wdt: wdt@21000 { - status = "okay"; - }; - mmc0: mmc@40000 { - max-frequency = <50000000>; - bus-width = <4>; - status = "okay"; - pinctrl-names = "default"; - pinctrl-0 = <&mmc0_pins>; - }; - spi1: spi@30e000 { - status = "okay"; - pinctrl-names = "default"; - pinctrl-0 = <&spi1_pins &spi1_cs0_pin>; - flash: m25p80@0 { - #address-cells = <1>; - #size-cells = <1>; - compatible = "m25p64"; - spi-max-frequency = <30000000>; - m25p,fast-read; - reg = <0>; - partition@0 { - label = "U-Boot-SPL"; - reg = <0x00000000 0x00010000>; - read-only; - }; - partition@1 { - label = "U-Boot"; - reg = <0x00010000 0x00080000>; - read-only; - }; - partition@2 { - label = "U-Boot-Env"; - reg = <0x00090000 0x00010000>; - read-only; - }; - partition@3 { - label = "Kernel"; - reg = <0x000a0000 0x00280000>; - }; - partition@4 { - label = "Filesystem"; - reg = <0x00320000 0x00400000>; - }; - partition@5 { - label = "MAC-Address"; - reg = <0x007f0000 0x00010000>; - read-only; - }; - }; - }; - mdio: mdio@224000 { - status = "okay"; - pinctrl-names = "default"; - pinctrl-0 = <&mdio_pins>; - bus_freq = <2200000>; - }; - eth0: ethernet@220000 { - status = "okay"; - pinctrl-names = "default"; - pinctrl-0 = <&mii_pins>; - }; - gpio: gpio@226000 { - status = "okay"; - }; - }; vbat: fixedregulator0 { compatible = "regulator-fixed"; regulator-name = "vbat"; @@ -200,6 +63,155 @@ }; }; +&pmx_core { + status = "okay"; + + mcasp0_pins: pinmux_mcasp0_pins { + pinctrl-single,bits = < + /* + * AHCLKX, ACLKX, AFSX, AHCLKR, ACLKR, + * AFSR, AMUTE + */ + 0x00 0x11111111 0xffffffff + /* AXR11, AXR12 */ + 0x04 0x00011000 0x000ff000 + >; + }; + nand_pins: nand_pins { + pinctrl-single,bits = < + /* EMA_WAIT[0], EMA_OE, EMA_WE, EMA_CS[4], EMA_CS[3] */ + 0x1c 0x10110110 0xf0ff0ff0 + /* + * EMA_D[0], EMA_D[1], EMA_D[2], + * EMA_D[3], EMA_D[4], EMA_D[5], + * EMA_D[6], EMA_D[7] + */ + 0x24 0x11111111 0xffffffff + /* EMA_A[1], EMA_A[2] */ + 0x30 0x01100000 0x0ff00000 + >; + }; +}; + +&serial0 { + status = "okay"; +}; + +&serial1 { + status = "okay"; +}; + +&serial2 { + status = "okay"; +}; + +&rtc0 { + status = "okay"; +}; + +&i2c0 { + status = "okay"; + clock-frequency = <100000>; + pinctrl-names = "default"; + pinctrl-0 = <&i2c0_pins>; + + tps: tps@48 { + reg = <0x48>; + }; + tlv320aic3106: tlv320aic3106@18 { + #sound-dai-cells = <0>; + compatible = "ti,tlv320aic3106"; + reg = <0x18>; + status = "okay"; + + /* Regulators */ + IOVDD-supply = <&vdcdc2_reg>; + /* Derived from VBAT: Baseboard 3.3V / 1.8V */ + AVDD-supply = <&vbat>; + DRVDD-supply = <&vbat>; + DVDD-supply = <&vbat>; + }; + tca6416: gpio@20 { + compatible = "ti,tca6416"; + reg = <0x20>; + gpio-controller; + #gpio-cells = <2>; + }; +}; + +&wdt { + status = "okay"; +}; + +&mmc0 { + max-frequency = <50000000>; + bus-width = <4>; + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&mmc0_pins>; + cd-gpios = <&gpio 64 GPIO_ACTIVE_LOW>; + wp-gpios = <&gpio 65 GPIO_ACTIVE_HIGH>; +}; + +&spi1 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&spi1_pins &spi1_cs0_pin>; + flash: m25p80@0 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "m25p64"; + spi-max-frequency = <30000000>; + m25p,fast-read; + reg = <0>; + partition@0 { + label = "U-Boot-SPL"; + reg = <0x00000000 0x00010000>; + read-only; + }; + partition@1 { + label = "U-Boot"; + reg = <0x00010000 0x00080000>; + read-only; + }; + partition@2 { + label = "U-Boot-Env"; + reg = <0x00090000 0x00010000>; + read-only; + }; + partition@3 { + label = "Kernel"; + reg = <0x000a0000 0x00280000>; + }; + partition@4 { + label = "Filesystem"; + reg = <0x00320000 0x00400000>; + }; + partition@5 { + label = "MAC-Address"; + reg = <0x007f0000 0x00010000>; + read-only; + }; + }; +}; + +&mdio { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&mdio_pins>; + bus_freq = <2200000>; +}; + +ð0 { + status = "okay"; + pinctrl-names = "default"; + pinctrl-0 = <&mii_pins>; +}; + +&gpio { + status = "okay"; +}; + /include/ "tps6507x.dtsi" &tps { @@ -309,6 +321,18 @@ }; }; +&usb_phy { + status = "okay"; +}; + +&usb0 { + status = "okay"; +}; + +&usb1 { + status = "okay"; +}; + &vpif { pinctrl-names = "default"; pinctrl-0 = <&vpif_capture_pins>, <&vpif_display_pins>; diff --git a/arch/arm/boot/dts/da850-lego-ev3.dts b/arch/arm/boot/dts/da850-lego-ev3.dts index 1ffd87796cac..ee3932475ce7 100644 --- a/arch/arm/boot/dts/da850-lego-ev3.dts +++ b/arch/arm/boot/dts/da850-lego-ev3.dts @@ -33,11 +33,9 @@ */ gpio_keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; label = "EV3 Brick Buttons"; pinctrl-names = "default"; - pinctrl-0 = <&button_pins>, <&button_bias>; + pinctrl-0 = <&button_bias>; center { label = "Center"; @@ -81,8 +79,6 @@ */ leds { compatible = "gpio-leds"; - pinctrl-names = "default"; - pinctrl-0 = <&led_pins>; left_green { label = "led0:green:brick-status"; @@ -119,8 +115,6 @@ gpio-poweroff { compatible = "gpio-poweroff"; gpios = <&gpio 107 GPIO_ACTIVE_LOW>; - pinctrl-names = "default"; - pinctrl-0 = <&system_power_pin>; }; sound { @@ -136,8 +130,6 @@ * the sensor (input) ports, the motor (output) ports and the A/DC. */ vcc5v: regulator1 { - pinctrl-names = "default"; - pinctrl-0 = <&vcc5v_pins>; compatible = "regulator-fixed"; regulator-name = "vcc5v"; regulator-min-microvolt = <5000000>; @@ -165,8 +157,6 @@ * This is the amplifier for the speaker. */ amp: regulator3 { - pinctrl-names = "default"; - pinctrl-0 = <&_pins>; compatible = "regulator-fixed"; regulator-name = "amp"; gpio = <&gpio 111 GPIO_ACTIVE_HIGH>; @@ -177,8 +167,6 @@ * The EV3 can use 6-AA batteries or a rechargeable Li-ion battery pack. */ battery { - pinctrl-names = "default"; - pinctrl-0 = <&battery_pins>; compatible = "lego,ev3-battery"; io-channels = <&adc 4>, <&adc 3>; io-channel-names = "voltage", "current"; @@ -206,73 +194,10 @@ &pmx_core { status = "okay"; - mmc0_cd_pin: pinmux_mmc0_cd { - pinctrl-single,bits = < - /* GP5[14] */ - 0x2C 0x00000080 0x000000f0 - >; - }; - - button_pins: pinmux_button_pins { - pinctrl-single,bits = < - /* GP1[13] */ - 0x8 0x00000800 0x00000f00 - /* GP6[10] */ - 0x34 0x00800000 0x00f00000 - /* GP6[6] */ - 0x38 0x00000080 0x000000f0 - /* GP7[12], GP7[14], GP7[15] */ - 0x40 0x00808800 0x00f0ff00 - >; - }; - - led_pins: pinmux_led_pins { - pinctrl-single,bits = < - /* GP6[12], GP6[13], GP6[14] */ - 0x34 0x00008880 0x0000fff0 - /* GP6[7] */ - 0x38 0x00000008 0x0000000f - >; - }; - - system_power_pin: pinmux_system_power { - pinctrl-single,bits = < - /* GP6[11] */ - 0x34 0x00080000 0x000f0000 - >; - }; - - vcc5v_pins: pinmux_vcc5v { - pinctrl-single,bits = < - /* GP6[5] */ - 0x40 0x00000080 0x000000f0 - /* GP6[3] */ - 0x4c 0x00008000 0x0000f000 - >; - }; - - amp_pins: pinmux_amp_pins { - pinctrl-single,bits = < - /* GP6[15] */ - 0x34 0x00000008 0x0000000f - >; - }; - - battery_pins: pinmux_battery_pins { - pinctrl-single,bits = < - /* GP0[6] */ - 0x04 0x00000080 0x000000f0 - /* GP8[8] */ - 0x4c 0x00000080 0x000000f0 - >; - }; - ev3_lcd_pins: pinmux_lcd { pinctrl-single,bits = < - /* SIMO, GP2[11], GP2[12], CLK */ - 0x14 0x00188100 0x00ffff00 - /* GP5[0] */ - 0x30 0x80000000 0xf0000000 + /* SIMO, CLK */ + 0x14 0x00100100 0x00f00f00 >; }; }; @@ -327,7 +252,7 @@ bus-width = <4>; cd-gpios = <&gpio 94 GPIO_ACTIVE_LOW>; pinctrl-names = "default"; - pinctrl-0 = <&mmc0_pins>, <&mmc0_cd_pin>; + pinctrl-0 = <&mmc0_pins>; }; &spi0 { diff --git a/arch/arm/boot/dts/da850.dtsi b/arch/arm/boot/dts/da850.dtsi index 12010002dbdb..f6f1597b03df 100644 --- a/arch/arm/boot/dts/da850.dtsi +++ b/arch/arm/boot/dts/da850.dtsi @@ -59,8 +59,18 @@ pinctrl-single,bit-per-mux; pinctrl-single,register-width = <32>; pinctrl-single,function-mask = <0xf>; + /* pin base, nr pins & gpio function */ + pinctrl-single,gpio-range = <&range 0 17 0x8>, + <&range 17 8 0x4>, + <&range 26 8 0x4>, + <&range 34 80 0x8>, + <&range 129 31 0x8>; status = "disabled"; + range: gpio-range { + #pinctrl-single,gpio-range-cells = <3>; + }; + serial0_rtscts_pins: pinmux_serial0_rtscts_pins { pinctrl-single,bits = < /* UART0_RTS UART0_CTS */ @@ -549,6 +559,150 @@ status = "disabled"; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = <&pmx_core 0 15 1>, + <&pmx_core 1 14 1>, + <&pmx_core 2 13 1>, + <&pmx_core 3 12 1>, + <&pmx_core 4 11 1>, + <&pmx_core 5 10 1>, + <&pmx_core 6 9 1>, + <&pmx_core 7 8 1>, + <&pmx_core 8 7 1>, + <&pmx_core 9 6 1>, + <&pmx_core 10 5 1>, + <&pmx_core 11 4 1>, + <&pmx_core 12 3 1>, + <&pmx_core 13 2 1>, + <&pmx_core 14 1 1>, + <&pmx_core 15 0 1>, + <&pmx_core 16 39 1>, + <&pmx_core 17 38 1>, + <&pmx_core 18 37 1>, + <&pmx_core 19 36 1>, + <&pmx_core 20 35 1>, + <&pmx_core 21 34 1>, + <&pmx_core 22 33 1>, + <&pmx_core 23 32 1>, + <&pmx_core 24 24 1>, + <&pmx_core 25 22 1>, + <&pmx_core 26 21 1>, + <&pmx_core 27 20 1>, + <&pmx_core 28 19 1>, + <&pmx_core 29 18 1>, + <&pmx_core 30 17 1>, + <&pmx_core 31 16 1>, + <&pmx_core 32 55 1>, + <&pmx_core 33 54 1>, + <&pmx_core 34 53 1>, + <&pmx_core 35 52 1>, + <&pmx_core 36 51 1>, + <&pmx_core 37 50 1>, + <&pmx_core 38 49 1>, + <&pmx_core 39 48 1>, + <&pmx_core 40 47 1>, + <&pmx_core 41 46 1>, + <&pmx_core 42 45 1>, + <&pmx_core 43 44 1>, + <&pmx_core 44 43 1>, + <&pmx_core 45 42 1>, + <&pmx_core 46 41 1>, + <&pmx_core 47 40 1>, + <&pmx_core 48 71 1>, + <&pmx_core 49 70 1>, + <&pmx_core 50 69 1>, + <&pmx_core 51 68 1>, + <&pmx_core 52 67 1>, + <&pmx_core 53 66 1>, + <&pmx_core 54 65 1>, + <&pmx_core 55 64 1>, + <&pmx_core 56 63 1>, + <&pmx_core 57 62 1>, + <&pmx_core 58 61 1>, + <&pmx_core 59 60 1>, + <&pmx_core 60 59 1>, + <&pmx_core 61 58 1>, + <&pmx_core 62 57 1>, + <&pmx_core 63 56 1>, + <&pmx_core 64 87 1>, + <&pmx_core 65 86 1>, + <&pmx_core 66 85 1>, + <&pmx_core 67 84 1>, + <&pmx_core 68 83 1>, + <&pmx_core 69 82 1>, + <&pmx_core 70 81 1>, + <&pmx_core 71 80 1>, + <&pmx_core 72 70 1>, + <&pmx_core 73 78 1>, + <&pmx_core 74 77 1>, + <&pmx_core 75 76 1>, + <&pmx_core 76 75 1>, + <&pmx_core 77 74 1>, + <&pmx_core 78 73 1>, + <&pmx_core 79 72 1>, + <&pmx_core 80 103 1>, + <&pmx_core 81 102 1>, + <&pmx_core 82 101 1>, + <&pmx_core 83 100 1>, + <&pmx_core 84 99 1>, + <&pmx_core 85 98 1>, + <&pmx_core 86 97 1>, + <&pmx_core 87 96 1>, + <&pmx_core 88 95 1>, + <&pmx_core 89 94 1>, + <&pmx_core 90 93 1>, + <&pmx_core 91 92 1>, + <&pmx_core 92 91 1>, + <&pmx_core 93 90 1>, + <&pmx_core 94 89 1>, + <&pmx_core 95 88 1>, + <&pmx_core 96 158 1>, + <&pmx_core 97 157 1>, + <&pmx_core 98 156 1>, + <&pmx_core 99 155 1>, + <&pmx_core 100 154 1>, + <&pmx_core 101 129 1>, + <&pmx_core 102 113 1>, + <&pmx_core 103 112 1>, + <&pmx_core 104 111 1>, + <&pmx_core 105 110 1>, + <&pmx_core 106 109 1>, + <&pmx_core 107 108 1>, + <&pmx_core 108 107 1>, + <&pmx_core 109 106 1>, + <&pmx_core 110 105 1>, + <&pmx_core 111 104 1>, + <&pmx_core 112 145 1>, + <&pmx_core 113 144 1>, + <&pmx_core 114 143 1>, + <&pmx_core 115 142 1>, + <&pmx_core 116 141 1>, + <&pmx_core 117 140 1>, + <&pmx_core 118 139 1>, + <&pmx_core 119 138 1>, + <&pmx_core 120 137 1>, + <&pmx_core 121 136 1>, + <&pmx_core 122 135 1>, + <&pmx_core 123 134 1>, + <&pmx_core 124 133 1>, + <&pmx_core 125 132 1>, + <&pmx_core 126 131 1>, + <&pmx_core 127 130 1>, + <&pmx_core 128 159 1>, + <&pmx_core 129 31 1>, + <&pmx_core 130 30 1>, + <&pmx_core 131 20 1>, + <&pmx_core 132 28 1>, + <&pmx_core 133 27 1>, + <&pmx_core 134 26 1>, + <&pmx_core 135 23 1>, + <&pmx_core 136 153 1>, + <&pmx_core 137 152 1>, + <&pmx_core 138 151 1>, + <&pmx_core 139 150 1>, + <&pmx_core 140 149 1>, + <&pmx_core 141 148 1>, + <&pmx_core 142 147 1>, + <&pmx_core 143 146 1>; }; pinconf: pin-controller@22c00c { compatible = "ti,da850-pupd"; diff --git a/arch/arm/boot/dts/dm8148-t410.dts b/arch/arm/boot/dts/dm8148-t410.dts index 6418f9cdbe83..c46a227b543d 100644 --- a/arch/arm/boot/dts/dm8148-t410.dts +++ b/arch/arm/boot/dts/dm8148-t410.dts @@ -77,7 +77,7 @@ DM814X_IOPAD(0x09dc, PIN_INPUT_PULLUP | 0x1) /* SD2_DAT[0] */ DM814X_IOPAD(0x09e0, PIN_INPUT | 0x1) /* SD2_CLK */ DM814X_IOPAD(0x09f4, PIN_INPUT_PULLUP | 0x2) /* SD2_CMD */ - DM814X_IOPAD(0x0920, PIN_INPUT | 40) /* SD2_SDCD */ + DM814X_IOPAD(0x0920, PIN_INPUT | 0x40) /* SD2_SDCD */ >; }; diff --git a/arch/arm/boot/dts/dra7-evm-common.dtsi b/arch/arm/boot/dts/dra7-evm-common.dtsi index 05a7b1a01bc3..33230c8b2951 100644 --- a/arch/arm/boot/dts/dra7-evm-common.dtsi +++ b/arch/arm/boot/dts/dra7-evm-common.dtsi @@ -260,3 +260,18 @@ &pcie1_rc { status = "okay"; }; + +&mmc4 { + bus-width = <4>; + cap-power-off-card; + keep-power-in-suspend; + non-removable; + #address-cells = <1>; + #size-cells = <0>; + wifi@2 { + compatible = "ti,wl1835"; + reg = <2>; + interrupt-parent = <&gpio5>; + interrupts = <7 IRQ_TYPE_EDGE_RISING>; + }; +}; diff --git a/arch/arm/boot/dts/dra7-evm.dts b/arch/arm/boot/dts/dra7-evm.dts index f1425b0f3a54..0894593860d6 100644 --- a/arch/arm/boot/dts/dra7-evm.dts +++ b/arch/arm/boot/dts/dra7-evm.dts @@ -20,6 +20,16 @@ reg = <0x0 0x80000000 0x0 0x60000000>; /* 1536 MB */ }; + evm_12v0: fixedregulator-evm_12v0 { + /* main supply */ + compatible = "regulator-fixed"; + regulator-name = "evm_12v0"; + regulator-min-microvolt = <12000000>; + regulator-max-microvolt = <12000000>; + regulator-always-on; + regulator-boot-on; + }; + evm_1v8_sw: fixedregulator-evm_1v8 { compatible = "regulator-fixed"; regulator-name = "evm_1v8"; @@ -54,6 +64,48 @@ regulator-max-microvolt = <1800000>; }; + evm_3v3: fixedregulator-evm3v3 { + /* Output of Cntlr A of TPS43351-Q1 on dra7-evm */ + compatible = "regulator-fixed"; + regulator-name = "evm_3v3"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + vin-supply = <&evm_12v0>; + regulator-always-on; + regulator-boot-on; + }; + + evm_5v0: fixedregulator-evm_5v0 { + /* Output of Cntlr B of TPS43351-Q1 on dra7-evm */ + compatible = "regulator-fixed"; + regulator-name = "evm_5v0"; + regulator-min-microvolt = <5000000>; + regulator-max-microvolt = <5000000>; + vin-supply = <&evm_12v0>; + regulator-always-on; + regulator-boot-on; + }; + + evm_3v6: fixedregulator-evm_3v6 { + compatible = "regulator-fixed"; + regulator-name = "evm_3v6"; + regulator-min-microvolt = <3600000>; + regulator-max-microvolt = <3600000>; + vin-supply = <&evm_5v0>; + regulator-always-on; + regulator-boot-on; + }; + + vmmcwl_fixed: fixedregulator-mmcwl { + compatible = "regulator-fixed"; + regulator-name = "vmmcwl_fixed"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + gpio = <&gpio5 8 0>; + startup-delay-us = <70000>; + enable-active-high; + }; + extcon_usb2: extcon_usb2 { compatible = "linux,extcon-usb-gpio"; id-gpio = <&pcf_gpio_21 2 GPIO_ACTIVE_HIGH>; @@ -325,6 +377,7 @@ vmmc-supply = <&evm_1v8_sw>; vqmmc-supply = <&evm_1v8_sw>; bus-width = <8>; + non-removable; pinctrl-names = "default", "hs", "ddr_1_8v-rev11", "ddr_1_8v", "hs200_1_8v-rev11", "hs200_1_8v"; pinctrl-0 = <&mmc2_pins_default>; pinctrl-1 = <&mmc2_pins_hs>; @@ -334,6 +387,21 @@ pinctrl-5 = <&mmc2_pins_hs200 &mmc2_iodelay_hs200_rev20_conf>; }; +&mmc4 { + status = "okay"; + vmmc-supply = <&evm_3v6>; + vqmmc-supply = <&vmmcwl_fixed>; + pinctrl-names = "default-rev11", "default", "hs-rev11", "hs", "sdr12-rev11", "sdr12", "sdr25-rev11", "sdr25"; + pinctrl-0 = <&mmc4_pins_default &mmc4_iodelay_ds_rev11_conf>; + pinctrl-1 = <&mmc4_pins_default &mmc4_iodelay_ds_rev20_conf>; + pinctrl-2 = <&mmc4_pins_hs &mmc4_iodelay_sdr12_hs_sdr25_rev11_conf>; + pinctrl-3 = <&mmc4_pins_hs &mmc4_iodelay_sdr12_hs_sdr25_rev20_conf>; + pinctrl-4 = <&mmc4_pins_sdr12 &mmc4_iodelay_sdr12_hs_sdr25_rev11_conf>; + pinctrl-5 = <&mmc4_pins_sdr12 &mmc4_iodelay_sdr12_hs_sdr25_rev20_conf>; + pinctrl-6 = <&mmc4_pins_sdr25 &mmc4_iodelay_sdr12_hs_sdr25_rev11_conf>; + pinctrl-7 = <&mmc4_pins_sdr25 &mmc4_iodelay_sdr12_hs_sdr25_rev20_conf>; +}; + &cpu0 { vdd-supply = <&smps123_reg>; }; diff --git a/arch/arm/boot/dts/dra7-mmc-iodelay.dtsi b/arch/arm/boot/dts/dra7-mmc-iodelay.dtsi new file mode 100644 index 000000000000..aa0947266526 --- /dev/null +++ b/arch/arm/boot/dts/dra7-mmc-iodelay.dtsi @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * MMC IOdelay values for TI's DRA7xx SoCs. + * Copyright (C) 2018 Texas Instruments + * Author: Kishon Vijay Abraham I + */ + +&dra7_pmx_core { + mmc1_pins_default_no_clk_pu: mmc1_pins_default_no_clk_pu { + pinctrl-single,pins = < + DRA7XX_CORE_IOPAD(0x3754, PIN_INPUT_PULLDOWN | MUX_MODE0) /* mmc1_clk.clk */ + DRA7XX_CORE_IOPAD(0x3758, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_cmd.cmd */ + DRA7XX_CORE_IOPAD(0x375c, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat0.dat0 */ + DRA7XX_CORE_IOPAD(0x3760, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat1.dat1 */ + DRA7XX_CORE_IOPAD(0x3764, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat2.dat2 */ + DRA7XX_CORE_IOPAD(0x3768, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat3.dat3 */ + >; + }; +}; diff --git a/arch/arm/boot/dts/dra7.dtsi b/arch/arm/boot/dts/dra7.dtsi index f4ddd86f2c77..9dcd14edc202 100644 --- a/arch/arm/boot/dts/dra7.dtsi +++ b/arch/arm/boot/dts/dra7.dtsi @@ -1079,17 +1079,15 @@ }; mmc1: mmc@4809c000 { - compatible = "ti,omap4-hsmmc"; + compatible = "ti,dra7-sdhci"; reg = <0x4809c000 0x400>; interrupts = ; ti,hwmods = "mmc1"; - ti,dual-volt; - ti,needs-special-reset; - dmas = <&sdma_xbar 61>, <&sdma_xbar 62>; - dma-names = "tx", "rx"; status = "disabled"; pbias-supply = <&pbias_mmc_reg>; max-frequency = <192000000>; + mmc-ddr-1_8v; + mmc-ddr-3_3v; }; hdqw1w: 1w@480b2000 { @@ -1100,40 +1098,40 @@ }; mmc2: mmc@480b4000 { - compatible = "ti,omap4-hsmmc"; + compatible = "ti,dra7-sdhci"; reg = <0x480b4000 0x400>; interrupts = ; ti,hwmods = "mmc2"; - ti,needs-special-reset; - dmas = <&sdma_xbar 47>, <&sdma_xbar 48>; - dma-names = "tx", "rx"; status = "disabled"; max-frequency = <192000000>; + /* SDR104/DDR50/SDR50 bits in CAPA2 is not supported */ + sdhci-caps-mask = <0x7 0x0>; + mmc-hs200-1_8v; + mmc-ddr-1_8v; + mmc-ddr-3_3v; }; mmc3: mmc@480ad000 { - compatible = "ti,omap4-hsmmc"; + compatible = "ti,dra7-sdhci"; reg = <0x480ad000 0x400>; interrupts = ; ti,hwmods = "mmc3"; - ti,needs-special-reset; - dmas = <&sdma_xbar 77>, <&sdma_xbar 78>; - dma-names = "tx", "rx"; status = "disabled"; /* Errata i887 limits max-frequency of MMC3 to 64 MHz */ max-frequency = <64000000>; + /* SDMA is not supported */ + sdhci-caps-mask = <0x0 0x400000>; }; mmc4: mmc@480d1000 { - compatible = "ti,omap4-hsmmc"; + compatible = "ti,dra7-sdhci"; reg = <0x480d1000 0x400>; interrupts = ; ti,hwmods = "mmc4"; - ti,needs-special-reset; - dmas = <&sdma_xbar 57>, <&sdma_xbar 58>; - dma-names = "tx", "rx"; status = "disabled"; max-frequency = <192000000>; + /* SDMA is not supported */ + sdhci-caps-mask = <0x0 0x400000>; }; mmu0_dsp1: mmu@40d01000 { diff --git a/arch/arm/boot/dts/dra71-evm.dts b/arch/arm/boot/dts/dra71-evm.dts index ebc4bbae981e..b7aeaeeead3b 100644 --- a/arch/arm/boot/dts/dra71-evm.dts +++ b/arch/arm/boot/dts/dra71-evm.dts @@ -7,6 +7,7 @@ */ #include "dra72-evm-common.dtsi" +#include "dra7-mmc-iodelay.dtsi" #include "dra72x-mmc-iodelay.dtsi" #include @@ -50,19 +51,6 @@ }; }; -&dra7_pmx_core { - mmc1_pins_default: mmc1_pins_default { - pinctrl-single,pins = < - DRA7XX_CORE_IOPAD(0x3754, PIN_INPUT_PULLDOWN | MUX_MODE0) /* mmc1_clk.clk */ - DRA7XX_CORE_IOPAD(0x3758, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_cmd.cmd */ - DRA7XX_CORE_IOPAD(0x375c, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat0.dat0 */ - DRA7XX_CORE_IOPAD(0x3760, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat1.dat1 */ - DRA7XX_CORE_IOPAD(0x3764, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat2.dat2 */ - DRA7XX_CORE_IOPAD(0x3768, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat3.dat3 */ - >; - }; -}; - &i2c1 { status = "okay"; clock-frequency = <400000>; @@ -187,7 +175,7 @@ &mmc1 { pinctrl-names = "default", "hs", "sdr12", "sdr25", "sdr50", "ddr50", "sdr104"; - pinctrl-0 = <&mmc1_pins_default>; + pinctrl-0 = <&mmc1_pins_default_no_clk_pu>; pinctrl-1 = <&mmc1_pins_hs>; pinctrl-2 = <&mmc1_pins_sdr12>; pinctrl-3 = <&mmc1_pins_sdr25>; @@ -204,6 +192,7 @@ pinctrl-2 = <&mmc2_pins_ddr_rev20 &mmc2_iodelay_ddr_conf>; pinctrl-3 = <&mmc2_pins_hs200 &mmc2_iodelay_hs200_rev20_conf>; vmmc-supply = <&evm_1v8_sw>; + vqmmc-supply = <&evm_1v8_sw>; }; &mac { diff --git a/arch/arm/boot/dts/dra72-evm-common.dtsi b/arch/arm/boot/dts/dra72-evm-common.dtsi index e85f560a2f78..df174f5c15d1 100644 --- a/arch/arm/boot/dts/dra72-evm-common.dtsi +++ b/arch/arm/boot/dts/dra72-evm-common.dtsi @@ -44,6 +44,16 @@ regulator-boot-on; }; + evm_3v6: fixedregulator-evm_3v6 { + compatible = "regulator-fixed"; + regulator-name = "evm_3v6"; + regulator-min-microvolt = <3600000>; + regulator-max-microvolt = <3600000>; + vin-supply = <&evm_5v0>; + regulator-always-on; + regulator-boot-on; + }; + vsys_3v3: fixedregulator-vsys3v3 { /* Output 2 of TPS43351QDAPRQ1 on dra72-evm */ /* Output 2 of LM5140QRWGTQ1 on dra71-evm */ @@ -171,36 +181,18 @@ clocks = <&atl_clkin2_ck>; }; }; -}; - -&dra7_pmx_core { - mmc1_pins_default: mmc1_pins_default { - pinctrl-single,pins = < - DRA7XX_CORE_IOPAD(0x376c, PIN_INPUT | MUX_MODE14) /* mmc1sdcd.gpio219 */ - DRA7XX_CORE_IOPAD(0x3754, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_clk.clk */ - DRA7XX_CORE_IOPAD(0x3758, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_cmd.cmd */ - DRA7XX_CORE_IOPAD(0x375c, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat0.dat0 */ - DRA7XX_CORE_IOPAD(0x3760, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat1.dat1 */ - DRA7XX_CORE_IOPAD(0x3764, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat2.dat2 */ - DRA7XX_CORE_IOPAD(0x3768, PIN_INPUT_PULLUP | MUX_MODE0) /* mmc1_dat3.dat3 */ - >; - }; - mmc2_pins_default: mmc2_pins_default { - pinctrl-single,pins = < - DRA7XX_CORE_IOPAD(0x349c, PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_a23.mmc2_clk */ - DRA7XX_CORE_IOPAD(0x34b0, PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_cs1.mmc2_cmd */ - DRA7XX_CORE_IOPAD(0x34a0, PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_a24.mmc2_dat0 */ - DRA7XX_CORE_IOPAD(0x34a4, PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_a25.mmc2_dat1 */ - DRA7XX_CORE_IOPAD(0x34a8, PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_a26.mmc2_dat2 */ - DRA7XX_CORE_IOPAD(0x34ac, PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_a27.mmc2_dat3 */ - DRA7XX_CORE_IOPAD(0x348c, PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_a19.mmc2_dat4 */ - DRA7XX_CORE_IOPAD(0x3490, PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_a20.mmc2_dat5 */ - DRA7XX_CORE_IOPAD(0x3494, PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_a21.mmc2_dat6 */ - DRA7XX_CORE_IOPAD(0x3498, PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_a22.mmc2_dat7 */ - >; + vmmcwl_fixed: fixedregulator-mmcwl { + compatible = "regulator-fixed"; + regulator-name = "vmmcwl_fixed"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + gpio = <&gpio5 8 GPIO_ACTIVE_HIGH>; + enable-active-high; }; +}; +&dra7_pmx_core { dcan1_pins_default: dcan1_pins_default { pinctrl-single,pins = < DRA7XX_CORE_IOPAD(0x37d0, PIN_OUTPUT_PULLUP | MUX_MODE0) /* dcan1_tx */ @@ -421,10 +413,33 @@ pinctrl-names = "default"; pinctrl-0 = <&mmc2_pins_default>; bus-width = <8>; - ti,non-removable; + non-removable; max-frequency = <192000000>; }; +&mmc4 { + status = "okay"; + vmmc-supply = <&evm_3v6>; + vqmmc-supply = <&vmmcwl_fixed>; + bus-width = <4>; + cap-power-off-card; + keep-power-in-suspend; + non-removable; + pinctrl-names = "default", "hs", "sdr12", "sdr25"; + pinctrl-0 = <&mmc4_pins_default>; + pinctrl-1 = <&mmc4_pins_default>; + pinctrl-2 = <&mmc4_pins_default>; + pinctrl-3 = <&mmc4_pins_default>; + #address-cells = <1>; + #size-cells = <0>; + wifi@2 { + compatible = "ti,wl1835"; + reg = <2>; + interrupt-parent = <&gpio5>; + interrupts = <7 IRQ_TYPE_EDGE_RISING>; + }; +}; + &mac { status = "okay"; }; diff --git a/arch/arm/boot/dts/dra72x-mmc-iodelay.dtsi b/arch/arm/boot/dts/dra72x-mmc-iodelay.dtsi index 088013c6dc6e..edad87c4292c 100644 --- a/arch/arm/boot/dts/dra72x-mmc-iodelay.dtsi +++ b/arch/arm/boot/dts/dra72x-mmc-iodelay.dtsi @@ -202,6 +202,17 @@ DRA7XX_CORE_IOPAD(0x3498, PIN_INPUT_PULLUP | MODE_SELECT | MUX_MODE1) /* gpmc_a22.mmc2_dat7 */ >; }; + + mmc4_pins_default: mmc4_pins_default { + pinctrl-single,pins = < + DRA7XX_CORE_IOPAD(0x37e8, PIN_INPUT_PULLUP | MUX_MODE3) /* uart1_ctsn.mmc4_clk */ + DRA7XX_CORE_IOPAD(0x37ec, PIN_INPUT_PULLUP | MUX_MODE3) /* uart1_rtsn.mmc4_cmd */ + DRA7XX_CORE_IOPAD(0x37f0, PIN_INPUT_PULLUP | MUX_MODE3) /* uart2_rxd.mmc4_dat0 */ + DRA7XX_CORE_IOPAD(0x37f4, PIN_INPUT_PULLUP | MUX_MODE3) /* uart2_txd.mmc4_dat1 */ + DRA7XX_CORE_IOPAD(0x37f8, PIN_INPUT_PULLUP | MUX_MODE3) /* uart2_ctsn.mmc4_dat2 */ + DRA7XX_CORE_IOPAD(0x37fc, PIN_INPUT_PULLUP | MUX_MODE3) /* uart2_rtsn.mmc4_dat3 */ + >; + }; }; &dra7_iodelay_core { diff --git a/arch/arm/boot/dts/dra76-evm.dts b/arch/arm/boot/dts/dra76-evm.dts index 2deb96405d06..c07f0051844d 100644 --- a/arch/arm/boot/dts/dra76-evm.dts +++ b/arch/arm/boot/dts/dra76-evm.dts @@ -42,6 +42,16 @@ regulator-boot-on; }; + vio_3v6: fixedregulator-vio_3v6 { + compatible = "regulator-fixed"; + regulator-name = "vio_3v6"; + regulator-min-microvolt = <3600000>; + regulator-max-microvolt = <3600000>; + vin-supply = <&vsys_5v0>; + regulator-always-on; + regulator-boot-on; + }; + vsys_3v3: fixedregulator-vsys3v3 { /* Output of Cntlr A of TPS43351-Q1 on dra76-evm */ compatible = "regulator-fixed"; @@ -81,6 +91,16 @@ vin-supply = <&smps5_reg>; }; + vmmcwl_fixed: fixedregulator-mmcwl { + compatible = "regulator-fixed"; + regulator-name = "vmmcwl_fixed"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + gpio = <&gpio5 8 0>; /* gpio5_8 */ + startup-delay-us = <70000>; + enable-active-high; + }; + vtt_fixed: fixedregulator-vtt { compatible = "regulator-fixed"; regulator-name = "vtt_fixed"; @@ -307,7 +327,7 @@ &mmc1 { status = "okay"; vmmc-supply = <&vio_3v3_sd>; - vmmc_aux-supply = <&ldo4_reg>; + vqmmc-supply = <&ldo4_reg>; bus-width = <4>; /* * SDCD signal is not being used here - using the fact that GPIO mode @@ -324,6 +344,7 @@ vmmc-supply = <&vio_1v8>; vqmmc-supply = <&vio_1v8>; bus-width = <8>; + non-removable; pinctrl-names = "default", "hs", "ddr_1_8v", "hs200_1_8v"; pinctrl-0 = <&mmc2_pins_default>; pinctrl-1 = <&mmc2_pins_default>; @@ -331,6 +352,17 @@ pinctrl-3 = <&mmc2_pins_hs200 &mmc2_iodelay_hs200_conf>; }; +&mmc4 { + status = "okay"; + vmmc-supply = <&vio_3v6>; + vqmmc-supply = <&vmmcwl_fixed>; + pinctrl-names = "default", "hs", "sdr12", "sdr25"; + pinctrl-0 = <&mmc4_pins_hs &mmc4_iodelay_default_conf>; + pinctrl-1 = <&mmc4_pins_hs &mmc4_iodelay_manual1_conf>; + pinctrl-2 = <&mmc4_pins_hs &mmc4_iodelay_manual1_conf>; + pinctrl-3 = <&mmc4_pins_hs &mmc4_iodelay_manual1_conf>; +}; + /* No RTC on this device */ &rtc { status = "disabled"; diff --git a/arch/arm/boot/dts/emev2-kzm9d.dts b/arch/arm/boot/dts/emev2-kzm9d.dts index c238407133bf..0af44b7eadb9 100644 --- a/arch/arm/boot/dts/emev2-kzm9d.dts +++ b/arch/arm/boot/dts/emev2-kzm9d.dts @@ -34,9 +34,6 @@ gpio_keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; - one { debounce-interval = <50>; wakeup-source; diff --git a/arch/arm/boot/dts/emev2.dtsi b/arch/arm/boot/dts/emev2.dtsi index 42ea246e71cb..fec1241b858f 100644 --- a/arch/arm/boot/dts/emev2.dtsi +++ b/arch/arm/boot/dts/emev2.dtsi @@ -31,13 +31,13 @@ #address-cells = <1>; #size-cells = <0>; - cpu@0 { + cpu0: cpu@0 { device_type = "cpu"; compatible = "arm,cortex-a9"; reg = <0>; clock-frequency = <533000000>; }; - cpu@1 { + cpu1: cpu@1 { device_type = "cpu"; compatible = "arm,cortex-a9"; reg = <1>; @@ -57,6 +57,7 @@ compatible = "arm,cortex-a9-pmu"; interrupts = , ; + interrupt-affinity = <&cpu0>, <&cpu1>; }; clocks@e0110000 { diff --git a/arch/arm/boot/dts/exynos-syscon-restart.dtsi b/arch/arm/boot/dts/exynos-syscon-restart.dtsi index 4b3dd0549a54..ecf416690a15 100644 --- a/arch/arm/boot/dts/exynos-syscon-restart.dtsi +++ b/arch/arm/boot/dts/exynos-syscon-restart.dtsi @@ -3,22 +3,18 @@ * Samsung's Exynos SoC syscon reboot/poweroff nodes common definition. */ -/ { - soc { - compatible = "simple-bus"; - - poweroff: syscon-poweroff { - compatible = "syscon-poweroff"; - regmap = <&pmu_system_controller>; - offset = <0x330C>; /* PS_HOLD_CONTROL */ - mask = <0x5200>; /* reset value */ - }; +&pmu_system_controller { + poweroff: syscon-poweroff { + compatible = "syscon-poweroff"; + regmap = <&pmu_system_controller>; + offset = <0x330C>; /* PS_HOLD_CONTROL */ + mask = <0x5200>; /* reset value */ + }; - reboot: syscon-reboot { - compatible = "syscon-reboot"; - regmap = <&pmu_system_controller>; - offset = <0x0400>; /* SWRESET */ - mask = <0x1>; - }; + reboot: syscon-reboot { + compatible = "syscon-reboot"; + regmap = <&pmu_system_controller>; + offset = <0x0400>; /* SWRESET */ + mask = <0x1>; }; }; diff --git a/arch/arm/boot/dts/exynos3250-rinato.dts b/arch/arm/boot/dts/exynos3250-rinato.dts index 2c3460781cc6..2a6b828c01b7 100644 --- a/arch/arm/boot/dts/exynos3250-rinato.dts +++ b/arch/arm/boot/dts/exynos3250-rinato.dts @@ -24,6 +24,10 @@ i2c7 = &i2c_max77836; }; + chosen { + stdout-path = &serial_1; + }; + memory@40000000 { device_type = "memory"; reg = <0x40000000 0x1ff00000>; diff --git a/arch/arm/boot/dts/exynos3250.dtsi b/arch/arm/boot/dts/exynos3250.dtsi index 0a5f989d963b..962af97c1883 100644 --- a/arch/arm/boot/dts/exynos3250.dtsi +++ b/arch/arm/boot/dts/exynos3250.dtsi @@ -15,7 +15,6 @@ */ #include "exynos4-cpu-thermal.dtsi" -#include "exynos-syscon-restart.dtsi" #include #include #include @@ -919,3 +918,4 @@ }; #include "exynos3250-pinctrl.dtsi" +#include "exynos-syscon-restart.dtsi" diff --git a/arch/arm/boot/dts/exynos4.dtsi b/arch/arm/boot/dts/exynos4.dtsi index 909a9f2bf5be..dfe41b698745 100644 --- a/arch/arm/boot/dts/exynos4.dtsi +++ b/arch/arm/boot/dts/exynos4.dtsi @@ -20,7 +20,6 @@ #include #include #include -#include "exynos-syscon-restart.dtsi" / { interrupt-parent = <&gic>; @@ -1025,3 +1024,5 @@ }; }; }; + +#include "exynos-syscon-restart.dtsi" diff --git a/arch/arm/boot/dts/exynos4210-origen.dts b/arch/arm/boot/dts/exynos4210-origen.dts index 9a310e841d5d..2ab99f9f3d0a 100644 --- a/arch/arm/boot/dts/exynos4210-origen.dts +++ b/arch/arm/boot/dts/exynos4210-origen.dts @@ -34,26 +34,17 @@ stdout-path = &serial_2; }; - regulators { - compatible = "simple-bus"; - #address-cells = <1>; - #size-cells = <0>; - - mmc_reg: regulator@0 { - compatible = "regulator-fixed"; - reg = <0>; - regulator-name = "VMEM_VDD_2.8V"; - regulator-min-microvolt = <2800000>; - regulator-max-microvolt = <2800000>; - gpio = <&gpx1 1 GPIO_ACTIVE_HIGH>; - enable-active-high; - }; + mmc_reg: voltage-regulator { + compatible = "regulator-fixed"; + regulator-name = "VMEM_VDD_2.8V"; + regulator-min-microvolt = <2800000>; + regulator-max-microvolt = <2800000>; + gpio = <&gpx1 1 GPIO_ACTIVE_HIGH>; + enable-active-high; }; gpio_keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; up { label = "Up"; @@ -131,12 +122,23 @@ cpu0-supply = <&buck1_reg>; }; +&exynos_usbphy { + status = "okay"; +}; + &fimd { pinctrl-0 = <&lcd_en &lcd_clk &lcd_data24 &pwm0_out>; pinctrl-names = "default"; status = "okay"; }; +&hsotg { + vusb_d-supply = <&ldo3_reg>; + vusb_a-supply = <&ldo8_reg>; + dr_mode = "peripheral"; + status = "okay"; +}; + &i2c_0 { status = "okay"; samsung,i2c-sda-delay = <100>; diff --git a/arch/arm/boot/dts/exynos4210-trats.dts b/arch/arm/boot/dts/exynos4210-trats.dts index eaeeb4f6b84a..6f1d76cb7951 100644 --- a/arch/arm/boot/dts/exynos4210-trats.dts +++ b/arch/arm/boot/dts/exynos4210-trats.dts @@ -259,8 +259,8 @@ reg = <0x48>; interrupt-parent = <&gpx0>; interrupts = <4 IRQ_TYPE_EDGE_FALLING>; - x-size = <720>; - y-size = <1280>; + touchscreen-size-x = <720>; + touchscreen-size-y = <1280>; avdd-supply = <&tsp_reg>; vdd-supply = <&tsp_reg>; }; diff --git a/arch/arm/boot/dts/exynos4412-galaxy-s3.dtsi b/arch/arm/boot/dts/exynos4412-galaxy-s3.dtsi index ee8e1f445370..30eee5942eff 100644 --- a/arch/arm/boot/dts/exynos4412-galaxy-s3.dtsi +++ b/arch/arm/boot/dts/exynos4412-galaxy-s3.dtsi @@ -15,24 +15,22 @@ i2c10 = &i2c_cm36651; }; - regulators { - lcd_vdd3_reg: voltage-regulator-2 { - compatible = "regulator-fixed"; - regulator-name = "LCD_VDD_2.2V"; - regulator-min-microvolt = <2200000>; - regulator-max-microvolt = <2200000>; - gpio = <&gpc0 1 GPIO_ACTIVE_HIGH>; - enable-active-high; - }; + lcd_vdd3_reg: voltage-regulator-6 { + compatible = "regulator-fixed"; + regulator-name = "LCD_VDD_2.2V"; + regulator-min-microvolt = <2200000>; + regulator-max-microvolt = <2200000>; + gpio = <&gpc0 1 GPIO_ACTIVE_HIGH>; + enable-active-high; + }; - ps_als_reg: voltage-regulator-5 { - compatible = "regulator-fixed"; - regulator-name = "LED_A_3.0V"; - regulator-min-microvolt = <3000000>; - regulator-max-microvolt = <3000000>; - gpio = <&gpj0 5 GPIO_ACTIVE_HIGH>; - enable-active-high; - }; + ps_als_reg: voltage-regulator-7 { + compatible = "regulator-fixed"; + regulator-name = "LED_A_3.0V"; + regulator-min-microvolt = <3000000>; + regulator-max-microvolt = <3000000>; + gpio = <&gpj0 5 GPIO_ACTIVE_HIGH>; + enable-active-high; }; i2c_ak8975: i2c-gpio-0 { @@ -120,8 +118,8 @@ reg = <0x48>; interrupt-parent = <&gpm2>; interrupts = <3 IRQ_TYPE_EDGE_FALLING>; - x-size = <720>; - y-size = <1280>; + touchscreen-size-x = <720>; + touchscreen-size-y = <1280>; avdd-supply = <&ldo23_reg>; vdd-supply = <&ldo24_reg>; }; diff --git a/arch/arm/boot/dts/exynos4412-midas.dtsi b/arch/arm/boot/dts/exynos4412-midas.dtsi index 76f2b30f1731..dc11ca1673e8 100644 --- a/arch/arm/boot/dts/exynos4412-midas.dtsi +++ b/arch/arm/boot/dts/exynos4412-midas.dtsi @@ -46,56 +46,50 @@ }; }; - regulators { - compatible = "simple-bus"; - #address-cells = <1>; - #size-cells = <0>; - - cam_io_reg: voltage-regulator-1 { - compatible = "regulator-fixed"; - regulator-name = "CAM_SENSOR_A"; - regulator-min-microvolt = <2800000>; - regulator-max-microvolt = <2800000>; - enable-active-high; - status = "disabled"; - }; + cam_io_reg: voltage-regulator-1 { + compatible = "regulator-fixed"; + regulator-name = "CAM_SENSOR_A"; + regulator-min-microvolt = <2800000>; + regulator-max-microvolt = <2800000>; + enable-active-high; + status = "disabled"; + }; - cam_af_reg: voltage-regulator-3 { - compatible = "regulator-fixed"; - regulator-name = "CAM_AF"; - regulator-min-microvolt = <2800000>; - regulator-max-microvolt = <2800000>; - enable-active-high; - status = "disabled"; - }; + cam_af_reg: voltage-regulator-2 { + compatible = "regulator-fixed"; + regulator-name = "CAM_AF"; + regulator-min-microvolt = <2800000>; + regulator-max-microvolt = <2800000>; + enable-active-high; + status = "disabled"; + }; - vsil12: voltage-regulator-6 { - compatible = "regulator-fixed"; - regulator-name = "VSIL_1.2V"; - regulator-min-microvolt = <1200000>; - regulator-max-microvolt = <1200000>; - gpio = <&gpl0 4 GPIO_ACTIVE_HIGH>; - enable-active-high; - vin-supply = <&buck7_reg>; - }; + vsil12: voltage-regulator-3 { + compatible = "regulator-fixed"; + regulator-name = "VSIL_1.2V"; + regulator-min-microvolt = <1200000>; + regulator-max-microvolt = <1200000>; + gpio = <&gpl0 4 GPIO_ACTIVE_HIGH>; + enable-active-high; + vin-supply = <&buck7_reg>; + }; - vcc33mhl: voltage-regulator-7 { - compatible = "regulator-fixed"; - regulator-name = "VCC_3.3_MHL"; - regulator-min-microvolt = <3300000>; - regulator-max-microvolt = <3300000>; - gpio = <&gpl0 4 GPIO_ACTIVE_HIGH>; - enable-active-high; - }; + vcc33mhl: voltage-regulator-4 { + compatible = "regulator-fixed"; + regulator-name = "VCC_3.3_MHL"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + gpio = <&gpl0 4 GPIO_ACTIVE_HIGH>; + enable-active-high; + }; - vcc18mhl: voltage-regulator-8 { - compatible = "regulator-fixed"; - regulator-name = "VCC_1.8_MHL"; - regulator-min-microvolt = <1800000>; - regulator-max-microvolt = <1800000>; - gpio = <&gpl0 4 GPIO_ACTIVE_HIGH>; - enable-active-high; - }; + vcc18mhl: voltage-regulator-5 { + compatible = "regulator-fixed"; + regulator-name = "VCC_1.8_MHL"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + gpio = <&gpl0 4 GPIO_ACTIVE_HIGH>; + enable-active-high; }; gpio-keys { diff --git a/arch/arm/boot/dts/exynos4412-n710x.dts b/arch/arm/boot/dts/exynos4412-n710x.dts index eb402a0d6651..fe2bfd76cc4e 100644 --- a/arch/arm/boot/dts/exynos4412-n710x.dts +++ b/arch/arm/boot/dts/exynos4412-n710x.dts @@ -13,15 +13,13 @@ /* bootargs are passed in by bootloader */ - regulators { - cam_vdda_reg: voltage-regulator-9 { - compatible = "regulator-fixed"; - regulator-name = "CAM_SENSOR_CORE_1.2V"; - regulator-min-microvolt = <1200000>; - regulator-max-microvolt = <1200000>; - gpio = <&gpm4 1 GPIO_ACTIVE_HIGH>; - enable-active-high; - }; + cam_vdda_reg: voltage-regulator-6 { + compatible = "regulator-fixed"; + regulator-name = "CAM_SENSOR_CORE_1.2V"; + regulator-min-microvolt = <1200000>; + regulator-max-microvolt = <1200000>; + gpio = <&gpm4 1 GPIO_ACTIVE_HIGH>; + enable-active-high; }; }; diff --git a/arch/arm/boot/dts/exynos4412-odroid-common.dtsi b/arch/arm/boot/dts/exynos4412-odroid-common.dtsi index d7ad07fd48f9..a09e46c9dbc0 100644 --- a/arch/arm/boot/dts/exynos4412-odroid-common.dtsi +++ b/arch/arm/boot/dts/exynos4412-odroid-common.dtsi @@ -37,20 +37,14 @@ }; sound: sound { - compatible = "simple-audio-card"; + compatible = "hardkernel,odroid-xu4-audio"; - simple-audio-card,format = "i2s"; - simple-audio-card,bitclock-master = <&link0_codec>; - simple-audio-card,frame-master = <&link0_codec>; - - simple-audio-card,cpu { + cpu { sound-dai = <&i2s0 0>; - system-clock-frequency = <19200000>; }; - link0_codec: simple-audio-card,codec { - sound-dai = <&max98090>; - clocks = <&i2s0 CLK_I2S_CDCLK>; + codec { + sound-dai = <&hdmi>, <&max98090>; }; }; @@ -142,14 +136,25 @@ pinctrl-0 = <>; }; +&clock { + assigned-clocks = <&clock CLK_FOUT_EPLL>; + assigned-clock-rates = <45158401>; +}; + &clock_audss { assigned-clocks = <&clock_audss EXYNOS_MOUT_AUDSS>, <&clock_audss EXYNOS_MOUT_I2S>, <&clock_audss EXYNOS_DOUT_SRP>, - <&clock_audss EXYNOS_DOUT_AUD_BUS>; + <&clock_audss EXYNOS_DOUT_AUD_BUS>, + <&clock_audss EXYNOS_DOUT_I2S>; + assigned-clock-parents = <&clock CLK_FOUT_EPLL>, - <&clock_audss EXYNOS_MOUT_AUDSS>; - assigned-clock-rates = <0>, <0>, <192000000>, <19200000>; + <&clock_audss EXYNOS_MOUT_AUDSS>; + + assigned-clock-rates = <0>, <0>, + <196608001>, + <(196608001 / 2)>, + <(196608001 / 8)>; }; &cpu0 { @@ -498,6 +503,8 @@ pinctrl-0 = <&i2s0_bus>; pinctrl-names = "default"; status = "okay"; + assigned-clocks = <&i2s0 CLK_I2S_RCLK_SRC>; + assigned-clock-parents = <&clock_audss EXYNOS_SCLK_I2S>; }; &mixer { diff --git a/arch/arm/boot/dts/exynos4412-odroidu3.dts b/arch/arm/boot/dts/exynos4412-odroidu3.dts index bdcd4523cc1c..459919b65df8 100644 --- a/arch/arm/boot/dts/exynos4412-odroidu3.dts +++ b/arch/arm/boot/dts/exynos4412-odroidu3.dts @@ -113,11 +113,11 @@ }; &sound { - simple-audio-card,name = "Odroid-U3"; - simple-audio-card,widgets = + model = "Odroid-U3"; + samsung,audio-widgets = "Headphone", "Headphone Jack", "Speakers", "Speakers"; - simple-audio-card,routing = + samsung,audio-routing = "Headphone Jack", "HPL", "Headphone Jack", "HPR", "Headphone Jack", "MICBIAS", diff --git a/arch/arm/boot/dts/exynos4412-odroidx.dts b/arch/arm/boot/dts/exynos4412-odroidx.dts index 2dff129bc2ad..348556fcdd9d 100644 --- a/arch/arm/boot/dts/exynos4412-odroidx.dts +++ b/arch/arm/boot/dts/exynos4412-odroidx.dts @@ -97,12 +97,12 @@ }; &sound { - simple-audio-card,name = "Odroid-X"; - simple-audio-card,widgets = + model = "Odroid-X"; + samsung,audio-widgets = "Headphone", "Headphone Jack", "Microphone", "Mic Jack", "Microphone", "DMIC"; - simple-audio-card,routing = + samsung,audio-routing = "Headphone Jack", "HPL", "Headphone Jack", "HPR", "IN1", "Mic Jack", diff --git a/arch/arm/boot/dts/exynos4412-origen.dts b/arch/arm/boot/dts/exynos4412-origen.dts index 1514f0dbaff8..346f71932457 100644 --- a/arch/arm/boot/dts/exynos4412-origen.dts +++ b/arch/arm/boot/dts/exynos4412-origen.dts @@ -90,7 +90,7 @@ samsung,vbus-gpio = <&gpx3 5 1>; status = "okay"; - port@1{ + port@1 { status = "okay"; }; port@2 { diff --git a/arch/arm/boot/dts/exynos4412.dtsi b/arch/arm/boot/dts/exynos4412.dtsi index 2ae1ab602f4b..7b43c10c510b 100644 --- a/arch/arm/boot/dts/exynos4412.dtsi +++ b/arch/arm/boot/dts/exynos4412.dtsi @@ -298,7 +298,7 @@ status = "disabled"; }; - sysmmu_g2d: sysmmu@10A40000{ + sysmmu_g2d: sysmmu@10a40000 { compatible = "samsung,exynos-sysmmu"; reg = <0x10A40000 0x1000>; interrupt-parent = <&combiner>; diff --git a/arch/arm/boot/dts/exynos5.dtsi b/arch/arm/boot/dts/exynos5.dtsi index f8d7de1144f1..5d2f0a06fbef 100644 --- a/arch/arm/boot/dts/exynos5.dtsi +++ b/arch/arm/boot/dts/exynos5.dtsi @@ -12,7 +12,6 @@ #include #include -#include "exynos-syscon-restart.dtsi" / { interrupt-parent = <&gic>; @@ -197,8 +196,6 @@ reg = <0x145B0000 0x1000>; interrupts = <10 3>; interrupt-parent = <&combiner>; - #address-cells = <1>; - #size-cells = <0>; status = "disabled"; }; diff --git a/arch/arm/boot/dts/exynos5250.dtsi b/arch/arm/boot/dts/exynos5250.dtsi index 45283a6c5eee..2daf505b3d08 100644 --- a/arch/arm/boot/dts/exynos5250.dtsi +++ b/arch/arm/boot/dts/exynos5250.dtsi @@ -132,10 +132,6 @@ reg = <0x100440A0 0x20>; #power-domain-cells = <0>; label = "DISP1"; - clocks = <&clock CLK_FIN_PLL>, - <&clock CLK_MOUT_ACLK200_DISP1_SUB>, - <&clock CLK_MOUT_ACLK300_DISP1_SUB>; - clock-names = "oscclk", "clk0", "clk1"; }; pd_mau: power-domain@100440c0 { @@ -882,7 +878,7 @@ #iommu-cells = <0>; }; - sysmmu_fimc_dis1: sysmmu@132E0000{ + sysmmu_fimc_dis1: sysmmu@132e0000 { compatible = "samsung,exynos-sysmmu"; reg = <0x132E0000 0x1000>; interrupt-parent = <&combiner>; @@ -1117,3 +1113,4 @@ }; #include "exynos5250-pinctrl.dtsi" +#include "exynos-syscon-restart.dtsi" diff --git a/arch/arm/boot/dts/exynos5410.dtsi b/arch/arm/boot/dts/exynos5410.dtsi index 55509c690328..20e789ea136f 100644 --- a/arch/arm/boot/dts/exynos5410.dtsi +++ b/arch/arm/boot/dts/exynos5410.dtsi @@ -439,3 +439,4 @@ }; #include "exynos5410-pinctrl.dtsi" +#include "exynos-syscon-restart.dtsi" diff --git a/arch/arm/boot/dts/exynos5420-peach-pit.dts b/arch/arm/boot/dts/exynos5420-peach-pit.dts index 244f0091c21f..57c2332bf282 100644 --- a/arch/arm/boot/dts/exynos5420-peach-pit.dts +++ b/arch/arm/boot/dts/exynos5420-peach-pit.dts @@ -29,7 +29,7 @@ aliases { /* Assign 20 so we don't get confused w/ builtin ones */ - i2c20 = "/spi@12d40000/cros-ec@0/i2c-tunnel"; + i2c20 = &i2c_tunnel; }; backlight: backlight { @@ -970,7 +970,7 @@ samsung,spi-feedback-delay = <1>; }; - i2c-tunnel { + i2c_tunnel: i2c-tunnel { compatible = "google,cros-ec-i2c-tunnel"; #address-cells = <1>; #size-cells = <0>; diff --git a/arch/arm/boot/dts/exynos5420.dtsi b/arch/arm/boot/dts/exynos5420.dtsi index 2f3cb2a97f71..f4e8c5823bc2 100644 --- a/arch/arm/boot/dts/exynos5420.dtsi +++ b/arch/arm/boot/dts/exynos5420.dtsi @@ -276,10 +276,6 @@ reg = <0x10044000 0x20>; #power-domain-cells = <0>; label = "GSC"; - clocks = <&clock CLK_FIN_PLL>, - <&clock CLK_MOUT_USER_ACLK300_GSCL>, - <&clock CLK_GSCL0>, <&clock CLK_GSCL1>; - clock-names = "oscclk", "clk0", "asb0", "asb1"; }; isp_pd: power-domain@10044020 { @@ -292,10 +288,6 @@ mfc_pd: power-domain@10044060 { compatible = "samsung,exynos4210-pd"; reg = <0x10044060 0x20>; - clocks = <&clock CLK_FIN_PLL>, - <&clock CLK_MOUT_USER_ACLK333>, - <&clock CLK_ACLK333>; - clock-names = "oscclk", "clk0","asb0"; #power-domain-cells = <0>; label = "MFC"; }; @@ -312,12 +304,6 @@ reg = <0x100440C0 0x20>; #power-domain-cells = <0>; label = "DISP"; - clocks = <&clock CLK_FIN_PLL>, - <&clock CLK_MOUT_USER_ACLK200_DISP1>, - <&clock CLK_MOUT_USER_ACLK300_DISP1>, - <&clock CLK_MOUT_USER_ACLK400_DISP1>, - <&clock CLK_FIMD1>, <&clock CLK_MIXER>; - clock-names = "oscclk", "clk0", "clk1", "clk2", "asb0", "asb1"; }; mau_pd: power-domain@100440e0 { @@ -687,6 +673,36 @@ iommus = <&sysmmu_gscl1>; }; + scaler_0: scaler@12800000 { + compatible = "samsung,exynos5420-scaler"; + reg = <0x12800000 0x1294>; + interrupts = <0 220 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&clock CLK_MSCL0>; + clock-names = "mscl"; + power-domains = <&msc_pd>; + iommus = <&sysmmu_scaler0r>, <&sysmmu_scaler0w>; + }; + + scaler_1: scaler@12810000 { + compatible = "samsung,exynos5420-scaler"; + reg = <0x12810000 0x1294>; + interrupts = <0 221 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&clock CLK_MSCL1>; + clock-names = "mscl"; + power-domains = <&msc_pd>; + iommus = <&sysmmu_scaler1r>, <&sysmmu_scaler1w>; + }; + + scaler_2: scaler@12820000 { + compatible = "samsung,exynos5420-scaler"; + reg = <0x12820000 0x1294>; + interrupts = <0 222 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&clock CLK_MSCL2>; + clock-names = "mscl"; + power-domains = <&msc_pd>; + iommus = <&sysmmu_scaler2r>, <&sysmmu_scaler2w>; + }; + jpeg_0: jpeg@11f50000 { compatible = "samsung,exynos5420-jpeg"; reg = <0x11F50000 0x1000>; @@ -761,7 +777,7 @@ #include "exynos5420-tmu-sensor-conf.dtsi" }; - sysmmu_g2dr: sysmmu@0x10A60000 { + sysmmu_g2dr: sysmmu@10a60000 { compatible = "samsung,exynos-sysmmu"; reg = <0x10A60000 0x1000>; interrupt-parent = <&combiner>; @@ -771,7 +787,7 @@ #iommu-cells = <0>; }; - sysmmu_g2dw: sysmmu@0x10A70000 { + sysmmu_g2dw: sysmmu@10a70000 { compatible = "samsung,exynos-sysmmu"; reg = <0x10A70000 0x1000>; interrupt-parent = <&combiner>; @@ -781,7 +797,7 @@ #iommu-cells = <0>; }; - sysmmu_tv: sysmmu@0x14650000 { + sysmmu_tv: sysmmu@14650000 { compatible = "samsung,exynos-sysmmu"; reg = <0x14650000 0x1000>; interrupt-parent = <&combiner>; @@ -792,7 +808,7 @@ #iommu-cells = <0>; }; - sysmmu_gscl0: sysmmu@0x13E80000 { + sysmmu_gscl0: sysmmu@13e80000 { compatible = "samsung,exynos-sysmmu"; reg = <0x13E80000 0x1000>; interrupt-parent = <&combiner>; @@ -803,7 +819,7 @@ #iommu-cells = <0>; }; - sysmmu_gscl1: sysmmu@0x13E90000 { + sysmmu_gscl1: sysmmu@13e90000 { compatible = "samsung,exynos-sysmmu"; reg = <0x13E90000 0x1000>; interrupt-parent = <&combiner>; @@ -814,65 +830,71 @@ #iommu-cells = <0>; }; - sysmmu_scaler0r: sysmmu@0x12880000 { + sysmmu_scaler0r: sysmmu@12880000 { compatible = "samsung,exynos-sysmmu"; reg = <0x12880000 0x1000>; interrupt-parent = <&combiner>; interrupts = <22 4>; clock-names = "sysmmu", "master"; clocks = <&clock CLK_SMMU_MSCL0>, <&clock CLK_MSCL0>; + power-domains = <&msc_pd>; #iommu-cells = <0>; }; - sysmmu_scaler1r: sysmmu@0x12890000 { + sysmmu_scaler1r: sysmmu@12890000 { compatible = "samsung,exynos-sysmmu"; reg = <0x12890000 0x1000>; interrupts = ; clock-names = "sysmmu", "master"; clocks = <&clock CLK_SMMU_MSCL1>, <&clock CLK_MSCL1>; + power-domains = <&msc_pd>; #iommu-cells = <0>; }; - sysmmu_scaler2r: sysmmu@0x128A0000 { + sysmmu_scaler2r: sysmmu@128a0000 { compatible = "samsung,exynos-sysmmu"; reg = <0x128A0000 0x1000>; interrupts = ; clock-names = "sysmmu", "master"; clocks = <&clock CLK_SMMU_MSCL2>, <&clock CLK_MSCL2>; + power-domains = <&msc_pd>; #iommu-cells = <0>; }; - sysmmu_scaler0w: sysmmu@0x128C0000 { + sysmmu_scaler0w: sysmmu@128c0000 { compatible = "samsung,exynos-sysmmu"; reg = <0x128C0000 0x1000>; interrupt-parent = <&combiner>; interrupts = <27 2>; clock-names = "sysmmu", "master"; clocks = <&clock CLK_SMMU_MSCL0>, <&clock CLK_MSCL0>; + power-domains = <&msc_pd>; #iommu-cells = <0>; }; - sysmmu_scaler1w: sysmmu@0x128D0000 { + sysmmu_scaler1w: sysmmu@128d0000 { compatible = "samsung,exynos-sysmmu"; reg = <0x128D0000 0x1000>; interrupt-parent = <&combiner>; interrupts = <22 6>; clock-names = "sysmmu", "master"; clocks = <&clock CLK_SMMU_MSCL1>, <&clock CLK_MSCL1>; + power-domains = <&msc_pd>; #iommu-cells = <0>; }; - sysmmu_scaler2w: sysmmu@0x128E0000 { + sysmmu_scaler2w: sysmmu@128e0000 { compatible = "samsung,exynos-sysmmu"; reg = <0x128E0000 0x1000>; interrupt-parent = <&combiner>; interrupts = <19 6>; clock-names = "sysmmu", "master"; clocks = <&clock CLK_SMMU_MSCL2>, <&clock CLK_MSCL2>; + power-domains = <&msc_pd>; #iommu-cells = <0>; }; - sysmmu_rotator: sysmmu@0x11D40000 { + sysmmu_rotator: sysmmu@11d40000 { compatible = "samsung,exynos-sysmmu"; reg = <0x11D40000 0x1000>; interrupt-parent = <&combiner>; @@ -882,7 +904,7 @@ #iommu-cells = <0>; }; - sysmmu_jpeg0: sysmmu@0x11F10000 { + sysmmu_jpeg0: sysmmu@11f10000 { compatible = "samsung,exynos-sysmmu"; reg = <0x11F10000 0x1000>; interrupt-parent = <&combiner>; @@ -892,7 +914,7 @@ #iommu-cells = <0>; }; - sysmmu_jpeg1: sysmmu@0x11F20000 { + sysmmu_jpeg1: sysmmu@11f20000 { compatible = "samsung,exynos-sysmmu"; reg = <0x11F20000 0x1000>; interrupts = ; @@ -901,7 +923,7 @@ #iommu-cells = <0>; }; - sysmmu_mfc_l: sysmmu@0x11200000 { + sysmmu_mfc_l: sysmmu@11200000 { compatible = "samsung,exynos-sysmmu"; reg = <0x11200000 0x1000>; interrupt-parent = <&combiner>; @@ -912,7 +934,7 @@ #iommu-cells = <0>; }; - sysmmu_mfc_r: sysmmu@0x11210000 { + sysmmu_mfc_r: sysmmu@11210000 { compatible = "samsung,exynos-sysmmu"; reg = <0x11210000 0x1000>; interrupt-parent = <&combiner>; @@ -923,7 +945,7 @@ #iommu-cells = <0>; }; - sysmmu_fimd1_0: sysmmu@0x14640000 { + sysmmu_fimd1_0: sysmmu@14640000 { compatible = "samsung,exynos-sysmmu"; reg = <0x14640000 0x1000>; interrupt-parent = <&combiner>; @@ -934,7 +956,7 @@ #iommu-cells = <0>; }; - sysmmu_fimd1_1: sysmmu@0x14680000 { + sysmmu_fimd1_1: sysmmu@14680000 { compatible = "samsung,exynos-sysmmu"; reg = <0x14680000 0x1000>; interrupt-parent = <&combiner>; @@ -1531,3 +1553,4 @@ }; #include "exynos5420-pinctrl.dtsi" +#include "exynos-syscon-restart.dtsi" diff --git a/arch/arm/boot/dts/exynos5422-odroid-core.dtsi b/arch/arm/boot/dts/exynos5422-odroid-core.dtsi index d31249f25ccf..2f4f40882dab 100644 --- a/arch/arm/boot/dts/exynos5422-odroid-core.dtsi +++ b/arch/arm/boot/dts/exynos5422-odroid-core.dtsi @@ -23,7 +23,7 @@ stdout-path = "serial2:115200n8"; }; - firmware@02073000 { + firmware@2073000 { compatible = "samsung,secure-firmware"; reg = <0x02073000 0x1000>; }; diff --git a/arch/arm/boot/dts/exynos5440-sd5v1.dts b/arch/arm/boot/dts/exynos5440-sd5v1.dts deleted file mode 100644 index c4b8392d1ae1..000000000000 --- a/arch/arm/boot/dts/exynos5440-sd5v1.dts +++ /dev/null @@ -1,42 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * SAMSUNG SD5v1 board device tree source - * - * Copyright (c) 2013 Samsung Electronics Co., Ltd. - * http://www.samsung.com - */ - -/dts-v1/; -#include "exynos5440.dtsi" - -/ { - model = "SAMSUNG SD5v1 board based on EXYNOS5440"; - compatible = "samsung,sd5v1", "samsung,exynos5440", "samsung,exynos5"; - - chosen { - bootargs = "root=/dev/sda2 rw rootwait ignore_loglevel earlyprintk no_console_suspend mem=2048M@0x80000000 mem=6144M@0x100000000 console=ttySAC0,115200"; - }; - - /* FIXME: set reg property with correct start address and size */ - memory@0 { - device_type = "memory"; - reg = <0 0>; - }; - - fixed-rate-clocks { - xtal { - compatible = "samsung,clock-xtal"; - clock-frequency = <50000000>; - }; - }; - - spi { - status = "disabled"; - }; - -}; - -&gmac { - fixed_phy; - phy_addr = <1>; -}; diff --git a/arch/arm/boot/dts/exynos5440-ssdk5440.dts b/arch/arm/boot/dts/exynos5440-ssdk5440.dts deleted file mode 100644 index a33c4fc29ae5..000000000000 --- a/arch/arm/boot/dts/exynos5440-ssdk5440.dts +++ /dev/null @@ -1,81 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * SAMSUNG SSDK5440 board device tree source - * - * Copyright (c) 2012 Samsung Electronics Co., Ltd. - * http://www.samsung.com - */ - -/dts-v1/; -#include "exynos5440.dtsi" -#include - -/ { - model = "SAMSUNG SSDK5440 board based on EXYNOS5440"; - compatible = "samsung,ssdk5440", "samsung,exynos5440", "samsung,exynos5"; - - chosen { - bootargs = "root=/dev/sda2 rw rootwait ignore_loglevel earlyprintk no_console_suspend mem=2048M@0x80000000 mem=6144M@0x100000000 console=ttySAC0,115200"; - }; - - /* FIXME: set reg property with correct start address and size */ - memory@0 { - device_type = "memory"; - reg = <0 0>; - }; - - fixed-rate-clocks { - xtal { - compatible = "samsung,clock-xtal"; - clock-frequency = <50000000>; - }; - }; -}; - -&pcie_0 { - reset-gpio = <&pin_ctrl 5 GPIO_ACTIVE_HIGH>; - status = "okay"; -}; - -&pcie_1 { - reset-gpio = <&pin_ctrl 22 GPIO_ACTIVE_HIGH>; - status = "okay"; -}; - -&spi_0 { - flash: w25q128@0 { - #address-cells = <1>; - #size-cells = <1>; - compatible = "winbond,w25q128"; - spi-max-frequency = <15625000>; - reg = <0>; - controller-data { - samsung,spi-feedback-delay = <0>; - }; - - partition@0 { - label = "BootLoader"; - reg = <0x60000 0x80000>; - read-only; - }; - - partition@e0000 { - label = "Recovery-Kernel"; - reg = <0xe0000 0x300000>; - read-only; - }; - - partition@3e0000 { - label = "CRAM-FS"; - reg = <0x3e0000 0x700000>; - read-only; - }; - - partition@ae0000 { - label = "User-Data"; - reg = <0xae0000 0x520000>; - }; - - }; - -}; diff --git a/arch/arm/boot/dts/exynos5440-tmu-sensor-conf.dtsi b/arch/arm/boot/dts/exynos5440-tmu-sensor-conf.dtsi deleted file mode 100644 index 0421c3d42905..000000000000 --- a/arch/arm/boot/dts/exynos5440-tmu-sensor-conf.dtsi +++ /dev/null @@ -1,20 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * Device tree sources for Exynos5440 TMU sensor configuration - * - * Copyright (c) 2014 Lukasz Majewski - */ - -#include - -#thermal-sensor-cells = <0>; -samsung,tmu_gain = <5>; -samsung,tmu_reference_voltage = <16>; -samsung,tmu_noise_cancel_mode = <4>; -samsung,tmu_efuse_value = <0x5d2d>; -samsung,tmu_min_efuse_value = <16>; -samsung,tmu_max_efuse_value = <76>; -samsung,tmu_first_point_trim = <25>; -samsung,tmu_second_point_trim = <70>; -samsung,tmu_default_temp_offset = <25>; -samsung,tmu_cal_type = ; diff --git a/arch/arm/boot/dts/exynos5440-trip-points.dtsi b/arch/arm/boot/dts/exynos5440-trip-points.dtsi deleted file mode 100644 index a2b04fed7d0b..000000000000 --- a/arch/arm/boot/dts/exynos5440-trip-points.dtsi +++ /dev/null @@ -1,21 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * Device tree sources for default Exynos5440 thermal zone definition - * - * Copyright (c) 2014 Lukasz Majewski - */ - -polling-delay-passive = <0>; -polling-delay = <0>; -trips { - cpu-alert-0 { - temperature = <100000>; /* millicelsius */ - hysteresis = <0>; /* millicelsius */ - type = "active"; - }; - cpu-crit-0 { - temperature = <105000>; /* millicelsius */ - hysteresis = <0>; /* millicelsius */ - type = "critical"; - }; -}; diff --git a/arch/arm/boot/dts/exynos5440.dtsi b/arch/arm/boot/dts/exynos5440.dtsi deleted file mode 100644 index f3abecc44657..000000000000 --- a/arch/arm/boot/dts/exynos5440.dtsi +++ /dev/null @@ -1,355 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * SAMSUNG EXYNOS5440 SoC device tree source - * - * Copyright (c) 2012 Samsung Electronics Co., Ltd. - * http://www.samsung.com - */ - -#include -#include -#include - -/ { - compatible = "samsung,exynos5440", "samsung,exynos5"; - - interrupt-parent = <&gic>; - #address-cells = <1>; - #size-cells = <1>; - - aliases { - serial0 = &serial_0; - serial1 = &serial_1; - spi0 = &spi_0; - tmuctrl0 = &tmuctrl_0; - tmuctrl1 = &tmuctrl_1; - tmuctrl2 = &tmuctrl_2; - }; - - cpus { - #address-cells = <1>; - #size-cells = <0>; - - cpu@0 { - device_type = "cpu"; - compatible = "arm,cortex-a15"; - reg = <0>; - }; - cpu@1 { - device_type = "cpu"; - compatible = "arm,cortex-a15"; - reg = <1>; - }; - cpu@2 { - device_type = "cpu"; - compatible = "arm,cortex-a15"; - reg = <2>; - }; - cpu@3 { - device_type = "cpu"; - compatible = "arm,cortex-a15"; - reg = <3>; - }; - }; - - soc: soc { - compatible = "simple-bus"; - #address-cells = <1>; - #size-cells = <1>; - ranges; - - clock: clock-controller@160000 { - compatible = "samsung,exynos5440-clock"; - reg = <0x160000 0x1000>; - #clock-cells = <1>; - }; - - gic: interrupt-controller@2e0000 { - compatible = "arm,cortex-a15-gic"; - #interrupt-cells = <3>; - interrupt-controller; - reg = <0x2E1000 0x1000>, - <0x2E2000 0x2000>, - <0x2E4000 0x2000>, - <0x2E6000 0x2000>; - interrupts = ; - }; - - - arm-pmu { - compatible = "arm,cortex-a15-pmu", "arm,cortex-a9-pmu"; - interrupts = , - , - , - ; - }; - - timer { - compatible = "arm,cortex-a15-timer", - "arm,armv7-timer"; - interrupts = , - , - , - ; - clock-frequency = <50000000>; - }; - - cpufreq@160000 { - compatible = "samsung,exynos5440-cpufreq"; - reg = <0x160000 0x1000>; - interrupts = ; - operating-points = < - /* KHz uV */ - 1500000 1100000 - 1400000 1075000 - 1300000 1050000 - 1200000 1025000 - 1100000 1000000 - 1000000 975000 - 900000 950000 - 800000 925000 - >; - }; - - serial_0: serial@b0000 { - compatible = "samsung,exynos4210-uart"; - reg = <0xB0000 0x1000>; - interrupts = ; - clocks = <&clock CLK_B_125>, <&clock CLK_B_125>; - clock-names = "uart", "clk_uart_baud0"; - }; - - serial_1: serial@c0000 { - compatible = "samsung,exynos4210-uart"; - reg = <0xC0000 0x1000>; - interrupts = ; - clocks = <&clock CLK_B_125>, <&clock CLK_B_125>; - clock-names = "uart", "clk_uart_baud0"; - }; - - spi_0: spi@d0000 { - compatible = "samsung,exynos5440-spi"; - reg = <0xD0000 0x100>; - interrupts = ; - #address-cells = <1>; - #size-cells = <0>; - samsung,spi-src-clk = <0>; - num-cs = <1>; - clocks = <&clock CLK_B_125>, <&clock CLK_SPI_BAUD>; - clock-names = "spi", "spi_busclk0"; - }; - - pin_ctrl: pinctrl@e0000 { - compatible = "samsung,exynos5440-pinctrl"; - reg = <0xE0000 0x1000>; - interrupts = , - , - , - , - , - , - , - ; - interrupt-controller; - #interrupt-cells = <2>; - #gpio-cells = <2>; - - fan: fan { - samsung,exynos5440-pin-function = <1>; - }; - - hdd_led0: hdd_led0 { - samsung,exynos5440-pin-function = <2>; - }; - - hdd_led1: hdd_led1 { - samsung,exynos5440-pin-function = <3>; - }; - - uart1: uart1 { - samsung,exynos5440-pin-function = <4>; - }; - }; - - i2c@f0000 { - compatible = "samsung,exynos5440-i2c"; - reg = <0xF0000 0x1000>; - interrupts = ; - #address-cells = <1>; - #size-cells = <0>; - clocks = <&clock CLK_B_125>; - clock-names = "i2c"; - }; - - i2c@100000 { - compatible = "samsung,exynos5440-i2c"; - reg = <0x100000 0x1000>; - interrupts = ; - #address-cells = <1>; - #size-cells = <0>; - clocks = <&clock CLK_B_125>; - clock-names = "i2c"; - }; - - watchdog@110000 { - compatible = "samsung,s3c6410-wdt"; - reg = <0x110000 0x1000>; - interrupts = ; - clocks = <&clock CLK_B_125>; - clock-names = "watchdog"; - }; - - gmac: ethernet@230000 { - compatible = "snps,dwmac-3.70a", "snps,dwmac"; - reg = <0x00230000 0x8000>; - interrupt-parent = <&gic>; - interrupts = ; - interrupt-names = "macirq"; - phy-mode = "sgmii"; - clocks = <&clock CLK_GMAC0>; - clock-names = "stmmaceth"; - }; - - amba { - #address-cells = <1>; - #size-cells = <1>; - compatible = "simple-bus"; - interrupt-parent = <&gic>; - ranges; - }; - - rtc@130000 { - compatible = "samsung,s3c6410-rtc"; - reg = <0x130000 0x1000>; - interrupts = , - ; - clocks = <&clock CLK_B_125>; - clock-names = "rtc"; - }; - - tmuctrl_0: tmuctrl@160118 { - compatible = "samsung,exynos5440-tmu"; - reg = <0x160118 0x230>, <0x160368 0x10>; - interrupts = ; - clocks = <&clock CLK_B_125>; - clock-names = "tmu_apbif"; - #include "exynos5440-tmu-sensor-conf.dtsi" - }; - - tmuctrl_1: tmuctrl@16011c { - compatible = "samsung,exynos5440-tmu"; - reg = <0x16011C 0x230>, <0x160368 0x10>; - interrupts = ; - clocks = <&clock CLK_B_125>; - clock-names = "tmu_apbif"; - #include "exynos5440-tmu-sensor-conf.dtsi" - }; - - tmuctrl_2: tmuctrl@160120 { - compatible = "samsung,exynos5440-tmu"; - reg = <0x160120 0x230>, <0x160368 0x10>; - interrupts = ; - clocks = <&clock CLK_B_125>; - clock-names = "tmu_apbif"; - #include "exynos5440-tmu-sensor-conf.dtsi" - }; - - sata@210000 { - compatible = "snps,exynos5440-ahci"; - reg = <0x210000 0x10000>; - interrupts = ; - clocks = <&clock CLK_SATA>; - clock-names = "sata"; - }; - - ohci@220000 { - compatible = "samsung,exynos5440-ohci"; - reg = <0x220000 0x1000>; - interrupts = ; - clocks = <&clock CLK_USB>; - clock-names = "usbhost"; - }; - - ehci@221000 { - compatible = "samsung,exynos5440-ehci"; - reg = <0x221000 0x1000>; - interrupts = ; - clocks = <&clock CLK_USB>; - clock-names = "usbhost"; - }; - - pcie_phy0: pcie-phy@270000 { - #phy-cells = <0>; - compatible = "samsung,exynos5440-pcie-phy"; - reg = <0x270000 0x1000>, <0x271000 0x40>; - }; - - pcie_phy1: pcie-phy@272000 { - #phy-cells = <0>; - compatible = "samsung,exynos5440-pcie-phy"; - reg = <0x272000 0x1000>, <0x271040 0x40>; - }; - - pcie_0: pcie@290000 { - compatible = "samsung,exynos5440-pcie", "snps,dw-pcie"; - reg = <0x290000 0x1000>, <0x40000000 0x1000>; - reg-names = "elbi", "config"; - interrupts = , - , - ; - clocks = <&clock CLK_PR0_250_O>, <&clock CLK_PB0_250_O>; - clock-names = "pcie", "pcie_bus"; - #address-cells = <3>; - #size-cells = <2>; - device_type = "pci"; - phys = <&pcie_phy0>; - ranges = <0x81000000 0 0 0x40001000 0 0x00010000 /* downstream I/O */ - 0x82000000 0 0x40011000 0x40011000 0 0x1ffef000>; /* non-prefetchable memory */ - bus-range = <0x00 0xff>; - #interrupt-cells = <1>; - interrupt-map-mask = <0 0 0 0>; - interrupt-map = <0x0 0 &gic 53>; - num-lanes = <4>; - status = "disabled"; - }; - - pcie_1: pcie@2a0000 { - compatible = "samsung,exynos5440-pcie", "snps,dw-pcie"; - reg = <0x2a0000 0x1000>, <0x60000000 0x1000>; - reg-names = "elbi", "config"; - interrupts = , - , - ; - clocks = <&clock CLK_PR1_250_O>, <&clock CLK_PB0_250_O>; - clock-names = "pcie", "pcie_bus"; - #address-cells = <3>; - #size-cells = <2>; - device_type = "pci"; - phys = <&pcie_phy1>; - ranges = <0x81000000 0 0 0x60001000 0 0x00010000 /* downstream I/O */ - 0x82000000 0 0x60011000 0x60011000 0 0x1ffef000>; /* non-prefetchable memory */ - bus-range = <0x00 0xff>; - #interrupt-cells = <1>; - interrupt-map-mask = <0 0 0 0>; - interrupt-map = <0x0 0 &gic 56>; - num-lanes = <4>; - status = "disabled"; - }; - }; - - thermal-zones { - cpu0_thermal: cpu0-thermal { - thermal-sensors = <&tmuctrl_0>; - #include "exynos5440-trip-points.dtsi" - }; - cpu1_thermal: cpu1-thermal { - thermal-sensors = <&tmuctrl_1>; - #include "exynos5440-trip-points.dtsi" - }; - cpu2_thermal: cpu2-thermal { - thermal-sensors = <&tmuctrl_2>; - #include "exynos5440-trip-points.dtsi" - }; - }; -}; diff --git a/arch/arm/boot/dts/exynos5800-peach-pi.dts b/arch/arm/boot/dts/exynos5800-peach-pi.dts index 2f8df9244f72..d80ab9085da1 100644 --- a/arch/arm/boot/dts/exynos5800-peach-pi.dts +++ b/arch/arm/boot/dts/exynos5800-peach-pi.dts @@ -27,7 +27,7 @@ aliases { /* Assign 20 so we don't get confused w/ builtin ones */ - i2c20 = "/spi@12d40000/cros-ec@0/i2c-tunnel"; + i2c20 = &i2c_tunnel; }; backlight: backlight { @@ -939,7 +939,7 @@ samsung,spi-feedback-delay = <1>; }; - i2c-tunnel { + i2c_tunnel: i2c-tunnel { compatible = "google,cros-ec-i2c-tunnel"; #address-cells = <1>; #size-cells = <0>; diff --git a/arch/arm/boot/dts/gemini-dlink-dir-685.dts b/arch/arm/boot/dts/gemini-dlink-dir-685.dts index cadde92bc6b5..fb5c954ab95a 100644 --- a/arch/arm/boot/dts/gemini-dlink-dir-685.dts +++ b/arch/arm/boot/dts/gemini-dlink-dir-685.dts @@ -13,22 +13,22 @@ #address-cells = <1>; #size-cells = <1>; - memory { + memory@0 { /* 128 MB SDRAM in 2 x Hynix HY5DU121622DTP-D43 */ device_type = "memory"; reg = <0x00000000 0x8000000>; }; chosen { - stdout-path = "uart0:115200n8"; + bootargs = "console=ttyS0,19200n8 root=/dev/sda1 rw rootwait"; + stdout-path = "uart0:19200n8"; }; gpio_keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; + button-esc { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = ; label = "reset"; @@ -36,7 +36,7 @@ gpios = <&gpio0 8 GPIO_ACTIVE_LOW>; }; button-eject { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = ; label = "unmount"; @@ -98,7 +98,7 @@ /* * These two LEDs are on the side of the device. * For electrical reasons, both LEDs cannot be active - * at the same time so only blue or orange can on at + * at the same time so only blue or orange can be on at * one time. Enabling both makes the LED go dark. * The LEDs both sit inside the unmount button and the * label on the case says "unmount". @@ -108,12 +108,14 @@ /* Collides with LPC_SERIRQ, UART DTR, SSP FSC pins */ gpios = <&gpio0 11 GPIO_ACTIVE_HIGH>; default-state = "off"; + linux,default-trigger = "disk-read"; }; led-orange-hd { label = "dir685:orange:HD"; /* Collides with LPC_LAD[2], UART DSR, SSP ECLK pins */ gpios = <&gpio0 12 GPIO_ACTIVE_HIGH>; default-state = "off"; + linux,default-trigger = "disk-write"; }; }; diff --git a/arch/arm/boot/dts/gemini-dlink-dns-313.dts b/arch/arm/boot/dts/gemini-dlink-dns-313.dts index 403364a7aab9..d1329322b968 100644 --- a/arch/arm/boot/dts/gemini-dlink-dns-313.dts +++ b/arch/arm/boot/dts/gemini-dlink-dns-313.dts @@ -15,7 +15,7 @@ #address-cells = <1>; #size-cells = <1>; - memory { + memory@0 { /* 64 MB SDRAM in a Nanya NT5DS32M16BS-6K package */ device_type = "memory"; reg = <0x00000000 0x4000000>; @@ -26,15 +26,15 @@ }; chosen { + bootargs = "console=ttyS0,19200n8 root=/dev/sda4 rw rootwait"; stdout-path = "uart0:19200n8"; }; gpio_keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; + button-esc { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = ; label = "reset"; @@ -59,14 +59,13 @@ label = "dns313:green:disk"; gpios = <&gpio0 3 GPIO_ACTIVE_HIGH>; default-state = "off"; - linux,default-trigger = "ide-disk"; - /* Ideally should activate while reading */ + linux,default-trigger = "disk-read"; }; led-disk-red { label = "dns313:red:disk"; gpios = <&gpio0 4 GPIO_ACTIVE_HIGH>; default-state = "off"; - /* Ideally should activate while writing */ + linux,default-trigger = "disk-write"; }; }; @@ -158,8 +157,12 @@ soc { flash@30000000 { + /* + * This is a Eon EN29LV400AB 512 KiB flash with + * three partitions. + */ + compatible = "cortina,gemini-flash", "jedec-flash"; status = "okay"; - /* 512KB of flash */ reg = <0x30000000 0x00080000>; /* diff --git a/arch/arm/boot/dts/gemini-nas4220b.dts b/arch/arm/boot/dts/gemini-nas4220b.dts index 4785fbcc41ed..963ea890c87f 100644 --- a/arch/arm/boot/dts/gemini-nas4220b.dts +++ b/arch/arm/boot/dts/gemini-nas4220b.dts @@ -14,7 +14,7 @@ #address-cells = <1>; #size-cells = <1>; - memory { /* 128 MB */ + memory@0 { /* 128 MB */ device_type = "memory"; reg = <0x00000000 0x8000000>; }; @@ -26,19 +26,17 @@ gpio_keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; - button@29 { - debounce_interval = <50>; + button-setup { + debounce-interval = <50>; wakeup-source; linux,code = ; label = "Backup button"; /* Conflict with TVC */ gpios = <&gpio1 29 GPIO_ACTIVE_LOW>; }; - button@31 { - debounce_interval = <50>; + button-restart { + debounce-interval = <50>; wakeup-source; linux,code = ; label = "Softreset button"; @@ -49,13 +47,13 @@ leds { compatible = "gpio-leds"; - led@28 { + led-orange-hdd { label = "nas4220b:orange:hdd"; /* Conflict with TVC */ gpios = <&gpio1 28 GPIO_ACTIVE_HIGH>; default-state = "on"; }; - led@30 { + led-green-os { label = "nas4220b:green:os"; /* Conflict with TVC */ gpios = <&gpio1 30 GPIO_ACTIVE_HIGH>; @@ -202,5 +200,9 @@ ata@63000000 { status = "okay"; }; + + ata@63400000 { + status = "okay"; + }; }; }; diff --git a/arch/arm/boot/dts/gemini-rut1xx.dts b/arch/arm/boot/dts/gemini-rut1xx.dts index 15f20178642c..eb4f0bf074da 100644 --- a/arch/arm/boot/dts/gemini-rut1xx.dts +++ b/arch/arm/boot/dts/gemini-rut1xx.dts @@ -14,7 +14,7 @@ #address-cells = <1>; #size-cells = <1>; - memory { /* 128 MB */ + memory@0 { /* 128 MB */ device_type = "memory"; reg = <0x00000000 0x8000000>; }; @@ -26,11 +26,9 @@ gpio_keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; - button@28 { - debounce_interval = <50>; + button-setup { + debounce-interval = <50>; wakeup-source; linux,code = ; label = "Reset to defaults"; @@ -41,14 +39,14 @@ leds { compatible = "gpio-leds"; - led@7 { + led-gsm { /* FIXME: add the LED color */ label = "rut1xx::gsm"; /* Conflict with ICE */ gpios = <&gpio0 7 GPIO_ACTIVE_HIGH>; default-state = "on"; }; - led@31 { + led-power { /* FIXME: add the LED color */ label = "rut1xx::power"; /* Conflict with NAND CE0 */ diff --git a/arch/arm/boot/dts/gemini-sq201.dts b/arch/arm/boot/dts/gemini-sq201.dts index 63c02ca9513c..e5cf9d1a98cd 100644 --- a/arch/arm/boot/dts/gemini-sq201.dts +++ b/arch/arm/boot/dts/gemini-sq201.dts @@ -14,7 +14,7 @@ #address-cells = <1>; #size-cells = <1>; - memory { /* 128 MB */ + memory@0 { /* 128 MB */ device_type = "memory"; reg = <0x00000000 0x8000000>; }; @@ -26,11 +26,9 @@ gpio_keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; - button@18 { - debounce_interval = <50>; + button-setup { + debounce-interval = <50>; wakeup-source; linux,code = ; label = "factory reset"; @@ -41,14 +39,14 @@ leds { compatible = "gpio-leds"; - led@20 { + led-green-info { label = "sq201:green:info"; /* Conflict with parallel flash */ gpios = <&gpio0 20 GPIO_ACTIVE_HIGH>; default-state = "on"; linux,default-trigger = "heartbeat"; }; - led@31 { + led-green-usb { label = "sq201:green:usb"; /* Conflict with parallel and NAND flash */ gpios = <&gpio0 31 GPIO_ACTIVE_HIGH>; diff --git a/arch/arm/boot/dts/gemini-wbd111.dts b/arch/arm/boot/dts/gemini-wbd111.dts index b4ec9ad85d72..29af86cd10f7 100644 --- a/arch/arm/boot/dts/gemini-wbd111.dts +++ b/arch/arm/boot/dts/gemini-wbd111.dts @@ -14,7 +14,8 @@ #address-cells = <1>; #size-cells = <1>; - memory { /* 128 MB */ + memory@0 { + /* 128 MB */ device_type = "memory"; reg = <0x00000000 0x8000000>; }; @@ -26,11 +27,9 @@ gpio_keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; - button@5 { - debounce_interval = <50>; + button-setup { + debounce-interval = <50>; wakeup-source; linux,code = ; label = "reset"; @@ -42,25 +41,25 @@ leds { compatible = "gpio-leds"; - led@1 { + led-red-l3 { label = "wbd111:red:L3"; /* Conflict with TVC and extended parallel flash */ gpios = <&gpio0 1 GPIO_ACTIVE_HIGH>; default-state = "off"; }; - led@2 { + led-green-l4 { label = "wbd111:green:L4"; /* Conflict with TVC and extended parallel flash */ gpios = <&gpio0 2 GPIO_ACTIVE_HIGH>; default-state = "off"; }; - led@3 { + led-red-l4 { label = "wbd111:red:L4"; /* Conflict with TVC and extended parallel flash */ gpios = <&gpio0 3 GPIO_ACTIVE_HIGH>; default-state = "off"; }; - led@5 { + led-greeb-l3 { label = "wbd111:green:L3"; /* Conflict with TVC and extended parallel flash */ gpios = <&gpio0 5 GPIO_ACTIVE_HIGH>; diff --git a/arch/arm/boot/dts/gemini-wbd222.dts b/arch/arm/boot/dts/gemini-wbd222.dts index 6d25bcc046e7..24e6ae3616f7 100644 --- a/arch/arm/boot/dts/gemini-wbd222.dts +++ b/arch/arm/boot/dts/gemini-wbd222.dts @@ -14,7 +14,7 @@ #address-cells = <1>; #size-cells = <1>; - memory { /* 128 MB */ + memory@0 { /* 128 MB */ device_type = "memory"; reg = <0x00000000 0x8000000>; }; @@ -26,11 +26,9 @@ gpio_keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; - button@5 { - debounce_interval = <50>; + button-setup { + debounce-interval = <50>; wakeup-source; linux,code = ; label = "reset"; @@ -42,25 +40,25 @@ leds { compatible = "gpio-leds"; - led@1 { + led-red-l3 { label = "wbd111:red:L3"; /* Conflict with TVC and extended parallel flash */ gpios = <&gpio0 1 GPIO_ACTIVE_HIGH>; default-state = "off"; }; - led@2 { + led-green-l4 { label = "wbd111:green:L4"; /* Conflict with TVC and extended parallel flash */ gpios = <&gpio0 2 GPIO_ACTIVE_HIGH>; default-state = "off"; }; - led@3 { + led-red-l4 { label = "wbd111:red:L4"; /* Conflict with TVC and extended parallel flash */ gpios = <&gpio0 3 GPIO_ACTIVE_HIGH>; default-state = "off"; }; - led@5 { + led-green-l3 { label = "wbd111:green:L3"; /* Conflict with TVC and extended parallel flash */ gpios = <&gpio0 5 GPIO_ACTIVE_HIGH>; diff --git a/arch/arm/boot/dts/gemini.dtsi b/arch/arm/boot/dts/gemini.dtsi index 0568baca500a..eb752e9495de 100644 --- a/arch/arm/boot/dts/gemini.dtsi +++ b/arch/arm/boot/dts/gemini.dtsi @@ -3,8 +3,6 @@ * Device Tree file for Cortina systems Gemini SoC */ -/include/ "skeleton.dtsi" - #include #include #include diff --git a/arch/arm/boot/dts/imx1-ads.dts b/arch/arm/boot/dts/imx1-ads.dts index 6354e4c87313..a1d81badb5c8 100644 --- a/arch/arm/boot/dts/imx1-ads.dts +++ b/arch/arm/boot/dts/imx1-ads.dts @@ -23,17 +23,6 @@ memory@8000000 { reg = <0x08000000 0x04000000>; }; - - clocks { - #address-cells = <1>; - #size-cells = <0>; - - clk32 { - compatible = "fsl,imx-clk32", "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <32000>; - }; - }; }; &cspi1 { diff --git a/arch/arm/boot/dts/imx1.dtsi b/arch/arm/boot/dts/imx1.dtsi index f7b9edf93f5e..3edc7b5550d8 100644 --- a/arch/arm/boot/dts/imx1.dtsi +++ b/arch/arm/boot/dts/imx1.dtsi @@ -1,13 +1,6 @@ -/* - * Copyright (C) 2014 Alexander Shiyan - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright (C) 2014 Alexander Shiyan #include "imx1-pinfunc.h" @@ -62,6 +55,14 @@ }; }; + clocks { + clk32 { + compatible = "fsl,imx-clk32", "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <32000>; + }; + }; + soc { #address-cells = <1>; #size-cells = <1>; diff --git a/arch/arm/boot/dts/imx23-evk.dts b/arch/arm/boot/dts/imx23-evk.dts index 9d92ece82560..9fb47724b9c1 100644 --- a/arch/arm/boot/dts/imx23-evk.dts +++ b/arch/arm/boot/dts/imx23-evk.dts @@ -1,13 +1,6 @@ -/* - * Copyright 2012 Freescale Semiconductor, Inc. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Freescale Semiconductor, Inc. /dts-v1/; #include "imx23.dtsi" diff --git a/arch/arm/boot/dts/imx23.dtsi b/arch/arm/boot/dts/imx23.dtsi index cb0a3fe32718..71bfd2b15609 100644 --- a/arch/arm/boot/dts/imx23.dtsi +++ b/arch/arm/boot/dts/imx23.dtsi @@ -1,13 +1,6 @@ -/* - * Copyright 2012 Freescale Semiconductor, Inc. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Freescale Semiconductor, Inc. #include "imx23-pinfunc.h" diff --git a/arch/arm/boot/dts/imx25-pdk.dts b/arch/arm/boot/dts/imx25-pdk.dts index 7f9bd052b84e..a5626b46ac4e 100644 --- a/arch/arm/boot/dts/imx25-pdk.dts +++ b/arch/arm/boot/dts/imx25-pdk.dts @@ -1,13 +1,6 @@ -/* - * Copyright 2013 Freescale Semiconductor, Inc. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2013 Freescale Semiconductor, Inc. /dts-v1/; #include @@ -291,7 +284,6 @@ }; &ssi1 { - codec-handle = <&codec>; status = "okay"; }; diff --git a/arch/arm/boot/dts/imx25.dtsi b/arch/arm/boot/dts/imx25.dtsi index cf70df20b19c..85c15ee63272 100644 --- a/arch/arm/boot/dts/imx25.dtsi +++ b/arch/arm/boot/dts/imx25.dtsi @@ -1,13 +1,6 @@ -/* - * Copyright 2012 Sascha Hauer, Pengutronix - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Sascha Hauer, Pengutronix #include #include "imx25-pinfunc.h" @@ -70,9 +63,6 @@ }; clocks { - #address-cells = <1>; - #size-cells = <0>; - osc { compatible = "fsl,imx-osc", "fixed-clock"; #clock-cells = <0>; diff --git a/arch/arm/boot/dts/imx27-apf27.dts b/arch/arm/boot/dts/imx27-apf27.dts index 66941cdbf244..3eddd805a793 100644 --- a/arch/arm/boot/dts/imx27-apf27.dts +++ b/arch/arm/boot/dts/imx27-apf27.dts @@ -22,17 +22,10 @@ memory@a0000000 { reg = <0xa0000000 0x04000000>; }; +}; - clocks { - #address-cells = <1>; - #size-cells = <0>; - - osc26m { - compatible = "fsl,imx-osc26m", "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; - }; - }; +&clk_osc26m { + clock-frequency = <0>; }; &iomuxc { diff --git a/arch/arm/boot/dts/imx27-pdk.dts b/arch/arm/boot/dts/imx27-pdk.dts index 924b90c9985d..f9a882d99132 100644 --- a/arch/arm/boot/dts/imx27-pdk.dts +++ b/arch/arm/boot/dts/imx27-pdk.dts @@ -1,13 +1,6 @@ -/* - * Copyright 2012 Sascha Hauer, Pengutronix - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Sascha Hauer, Pengutronix /dts-v1/; #include "imx27.dtsi" diff --git a/arch/arm/boot/dts/imx27.dtsi b/arch/arm/boot/dts/imx27.dtsi index 6585b00c3917..753d88df1627 100644 --- a/arch/arm/boot/dts/imx27.dtsi +++ b/arch/arm/boot/dts/imx27.dtsi @@ -1,13 +1,6 @@ -/* - * Copyright 2012 Sascha Hauer, Pengutronix - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Sascha Hauer, Pengutronix #include "imx27-pinfunc.h" @@ -57,10 +50,7 @@ }; clocks { - #address-cells = <1>; - #size-cells = <0>; - - osc26m { + clk_osc26m: osc26m { compatible = "fsl,imx-osc26m", "fixed-clock"; #clock-cells = <0>; clock-frequency = <26000000>; diff --git a/arch/arm/boot/dts/imx28-cfa10049.dts b/arch/arm/boot/dts/imx28-cfa10049.dts index 60e5c7fd5035..f1c8315b3e01 100644 --- a/arch/arm/boot/dts/imx28-cfa10049.dts +++ b/arch/arm/boot/dts/imx28-cfa10049.dts @@ -398,8 +398,6 @@ compatible = "gpio-keys"; pinctrl-names = "default"; pinctrl-0 = <&rotary_btn_pins_cfa10049>; - #address-cells = <1>; - #size-cells = <0>; rotary_button { label = "rotary_button"; diff --git a/arch/arm/boot/dts/imx28-duckbill-2-enocean.dts b/arch/arm/boot/dts/imx28-duckbill-2-enocean.dts index 7f8d40a9c67e..22215337f72a 100644 --- a/arch/arm/boot/dts/imx28-duckbill-2-enocean.dts +++ b/arch/arm/boot/dts/imx28-duckbill-2-enocean.dts @@ -206,8 +206,6 @@ gpio-keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; pinctrl-names = "default"; pinctrl-0 = <&enocean_button>; diff --git a/arch/arm/boot/dts/imx28-evk.dts b/arch/arm/boot/dts/imx28-evk.dts index b0d39654aeb3..6b0ae667640f 100644 --- a/arch/arm/boot/dts/imx28-evk.dts +++ b/arch/arm/boot/dts/imx28-evk.dts @@ -1,13 +1,6 @@ -/* - * Copyright 2012 Freescale Semiconductor, Inc. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Freescale Semiconductor, Inc. /dts-v1/; #include "imx28.dtsi" diff --git a/arch/arm/boot/dts/imx28-tx28.dts b/arch/arm/boot/dts/imx28-tx28.dts index 687186358c18..b8f46432e2a2 100644 --- a/arch/arm/boot/dts/imx28-tx28.dts +++ b/arch/arm/boot/dts/imx28-tx28.dts @@ -140,15 +140,10 @@ regulator-boot-on; }; - clocks { - #address-cells = <1>; - #size-cells = <0>; - mclk: clock@0 { - compatible = "fixed-clock"; - reg = <0>; - #clock-cells = <0>; - clock-frequency = <26000000>; - }; + mclk: clock-mclk { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <26000000>; }; sound { @@ -345,6 +340,7 @@ interrupts = <5 IRQ_TYPE_EDGE_FALLING>; reset-gpios = <&gpio2 6 GPIO_ACTIVE_LOW>; wake-gpios = <&gpio4 9 GPIO_ACTIVE_HIGH>; + wakeup-source; }; touchscreen: tsc2007@48 { diff --git a/arch/arm/boot/dts/imx28.dtsi b/arch/arm/boot/dts/imx28.dtsi index 9ad8d3556859..5107fdc482ea 100644 --- a/arch/arm/boot/dts/imx28.dtsi +++ b/arch/arm/boot/dts/imx28.dtsi @@ -1,13 +1,6 @@ -/* - * Copyright 2012 Freescale Semiconductor, Inc. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Freescale Semiconductor, Inc. #include #include "imx28-pinfunc.h" diff --git a/arch/arm/boot/dts/imx31.dtsi b/arch/arm/boot/dts/imx31.dtsi index ebc3f2dbb6fd..4642c8169a65 100644 --- a/arch/arm/boot/dts/imx31.dtsi +++ b/arch/arm/boot/dts/imx31.dtsi @@ -1,13 +1,6 @@ -/* - * Copyright 2012 Denis 'GNUtoo' Carikli - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Denis 'GNUtoo' Carikli / { #address-cells = <1>; diff --git a/arch/arm/boot/dts/imx35-pdk.dts b/arch/arm/boot/dts/imx35-pdk.dts index 646b1257bba2..df613e88fd2c 100644 --- a/arch/arm/boot/dts/imx35-pdk.dts +++ b/arch/arm/boot/dts/imx35-pdk.dts @@ -1,14 +1,7 @@ -/* - * Copyright 2013 Eukréa Electromatique - * Copyright 2014 Freescale Semiconductor, Inc. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2013 Eukréa Electromatique +// Copyright 2014 Freescale Semiconductor, Inc. /dts-v1/; #include "imx35.dtsi" diff --git a/arch/arm/boot/dts/imx35.dtsi b/arch/arm/boot/dts/imx35.dtsi index 54111ed218b1..1c50b785cad4 100644 --- a/arch/arm/boot/dts/imx35.dtsi +++ b/arch/arm/boot/dts/imx35.dtsi @@ -1,12 +1,8 @@ -/* - * Copyright 2012 Steffen Trumtrar, Pengutronix - * - * based on imx27.dtsi - * - * This program is free software; you can redistribute it and/or modify it under - * the terms of the GNU General Public License version 2 as published by the - * Free Software Foundation. - */ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright 2012 Steffen Trumtrar, Pengutronix +// +// based on imx27.dtsi #include "imx35-pinfunc.h" diff --git a/arch/arm/boot/dts/imx50-evk.dts b/arch/arm/boot/dts/imx50-evk.dts index 23f1833e23fa..f0622ec4ba9c 100644 --- a/arch/arm/boot/dts/imx50-evk.dts +++ b/arch/arm/boot/dts/imx50-evk.dts @@ -1,15 +1,8 @@ -/* - * Copyright 2013 Greg Ungerer - * Copyright 2011 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2013 Greg Ungerer +// Copyright 2011 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. /dts-v1/; #include "imx50.dtsi" diff --git a/arch/arm/boot/dts/imx50.dtsi b/arch/arm/boot/dts/imx50.dtsi index 7954e79d0a16..a9b712db9f6c 100644 --- a/arch/arm/boot/dts/imx50.dtsi +++ b/arch/arm/boot/dts/imx50.dtsi @@ -60,9 +60,6 @@ }; clocks { - #address-cells = <1>; - #size-cells = <0>; - ckil { compatible = "fsl,imx-ckil", "fixed-clock"; #clock-cells = <0>; diff --git a/arch/arm/boot/dts/imx51-babbage.dts b/arch/arm/boot/dts/imx51-babbage.dts index cf7a1963df25..b8ca73d3d379 100644 --- a/arch/arm/boot/dts/imx51-babbage.dts +++ b/arch/arm/boot/dts/imx51-babbage.dts @@ -1,14 +1,7 @@ -/* - * Copyright 2011 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2011 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. /dts-v1/; #include "imx51.dtsi" diff --git a/arch/arm/boot/dts/imx51-zii-rdu1.dts b/arch/arm/boot/dts/imx51-zii-rdu1.dts index 6464f2560e06..df9eca94d812 100644 --- a/arch/arm/boot/dts/imx51-zii-rdu1.dts +++ b/arch/arm/boot/dts/imx51-zii-rdu1.dts @@ -207,8 +207,6 @@ switch@0 { compatible = "marvell,mv88e6085"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; dsa,member = <0 0>; @@ -462,7 +460,10 @@ pinctrl-names = "default"; pinctrl-0 = <&pinctrl_esdhc1>; bus-width = <4>; + no-1-8-v; non-removable; + no-sdio; + no-sd; status = "okay"; }; @@ -591,6 +592,7 @@ phy_type = "ulpi"; fsl,usbphy = <&usbh1phy>; disable-over-current; + maximum-speed = "full-speed"; vbus-supply = <®_5p0v_main>; status = "okay"; }; diff --git a/arch/arm/boot/dts/imx51.dtsi b/arch/arm/boot/dts/imx51.dtsi index 5d390a64e976..fe01b890c715 100644 --- a/arch/arm/boot/dts/imx51.dtsi +++ b/arch/arm/boot/dts/imx51.dtsi @@ -1,14 +1,7 @@ -/* - * Copyright 2011 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2011 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. #include "imx51-pinfunc.h" #include @@ -56,9 +49,6 @@ }; clocks { - #address-cells = <1>; - #size-cells = <0>; - ckil { compatible = "fsl,imx-ckil", "fixed-clock"; #clock-cells = <0>; diff --git a/arch/arm/boot/dts/imx53-ard.dts b/arch/arm/boot/dts/imx53-ard.dts index 80fc00705d92..117bd002dd1d 100644 --- a/arch/arm/boot/dts/imx53-ard.dts +++ b/arch/arm/boot/dts/imx53-ard.dts @@ -11,6 +11,7 @@ */ /dts-v1/; +#include #include "imx53.dtsi" / { @@ -68,34 +69,34 @@ home { label = "Home"; gpios = <&gpio5 10 0>; - linux,code = <102>; /* KEY_HOME */ + linux,code = ; wakeup-source; }; back { label = "Back"; gpios = <&gpio5 11 0>; - linux,code = <158>; /* KEY_BACK */ + linux,code = ; wakeup-source; }; program { label = "Program"; gpios = <&gpio5 12 0>; - linux,code = <362>; /* KEY_PROGRAM */ + linux,code = ; wakeup-source; }; volume-up { label = "Volume Up"; gpios = <&gpio5 13 0>; - linux,code = <115>; /* KEY_VOLUMEUP */ + linux,code = ; }; volume-down { label = "Volume Down"; gpios = <&gpio4 0 0>; - linux,code = <114>; /* KEY_VOLUMEDOWN */ + linux,code = ; }; }; }; diff --git a/arch/arm/boot/dts/imx53-m53.dtsi b/arch/arm/boot/dts/imx53-m53.dtsi index 3da6dd5edb79..ce45f08e3051 100644 --- a/arch/arm/boot/dts/imx53-m53.dtsi +++ b/arch/arm/boot/dts/imx53-m53.dtsi @@ -53,8 +53,6 @@ stmpe610@41 { compatible = "st,stmpe610"; - #address-cells = <1>; - #size-cells = <0>; reg = <0x41>; id = <0>; blocks = <0x5>; diff --git a/arch/arm/boot/dts/imx53-ppd.dts b/arch/arm/boot/dts/imx53-ppd.dts index d5628af2e301..3aa6f693fa9f 100644 --- a/arch/arm/boot/dts/imx53-ppd.dts +++ b/arch/arm/boot/dts/imx53-ppd.dts @@ -180,8 +180,6 @@ power-gpio-keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; power-button { label = "Power button"; @@ -192,8 +190,6 @@ touch-lock-key { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; touch-lock-button { label = "Touch lock button"; @@ -300,7 +296,7 @@ compatible = "dlg,da9053-aa"; reg = <0>; interrupt-parent = <&gpio3>; - interrupts = <12 0x8>; + interrupts = <12 IRQ_TYPE_LEVEL_LOW>; spi-max-frequency = <1000000>; dlg,tsi-as-adc; tsiref-supply = <®_tsiref>; @@ -473,7 +469,7 @@ compatible = "fsl,mma8453"; reg = <0x1c>; interrupt-parent = <&gpio1>; - interrupts = <6 0>; + interrupts = <6 IRQ_TYPE_NONE>; interrupt-names = "INT1"; }; @@ -539,7 +535,7 @@ reset-gpio = <&gpio5 19 GPIO_ACTIVE_HIGH>; reg = <0x4b>; interrupt-parent = <&gpio5>; - interrupts = <4 0x8>; + interrupts = <4 IRQ_TYPE_LEVEL_LOW>; }; }; @@ -559,8 +555,6 @@ status = "okay"; port@2 { - reg = <2>; - lvds0_out: endpoint { remote-endpoint = <&panel_in_lvds0>; }; diff --git a/arch/arm/boot/dts/imx53-qsb-common.dtsi b/arch/arm/boot/dts/imx53-qsb-common.dtsi index 485a69d45e1c..ef7658a78836 100644 --- a/arch/arm/boot/dts/imx53-qsb-common.dtsi +++ b/arch/arm/boot/dts/imx53-qsb-common.dtsi @@ -1,14 +1,7 @@ -/* - * Copyright 2011 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2011 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. #include "imx53.dtsi" diff --git a/arch/arm/boot/dts/imx53-qsb.dts b/arch/arm/boot/dts/imx53-qsb.dts index d3d662e37677..6831836bd726 100644 --- a/arch/arm/boot/dts/imx53-qsb.dts +++ b/arch/arm/boot/dts/imx53-qsb.dts @@ -1,14 +1,7 @@ -/* - * Copyright 2011 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2011 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. /dts-v1/; #include "imx53-qsb-common.dtsi" @@ -23,7 +16,7 @@ compatible = "dlg,da9053-aa", "dlg,da9052"; reg = <0x48>; interrupt-parent = <&gpio7>; - interrupts = <11 0x8>; /* low-level active IRQ at GPIO7_11 */ + interrupts = <11 IRQ_TYPE_LEVEL_LOW>; /* low-level active IRQ at GPIO7_11 */ regulators { buck1_reg: buck1 { diff --git a/arch/arm/boot/dts/imx53-qsrb.dts b/arch/arm/boot/dts/imx53-qsrb.dts index 4e103a905dc9..1bbf24ad308a 100644 --- a/arch/arm/boot/dts/imx53-qsrb.dts +++ b/arch/arm/boot/dts/imx53-qsrb.dts @@ -1,14 +1,7 @@ -/* - * Copyright 2011 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2011 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. /dts-v1/; diff --git a/arch/arm/boot/dts/imx53-smd.dts b/arch/arm/boot/dts/imx53-smd.dts index fd030128666c..462071c9ddd7 100644 --- a/arch/arm/boot/dts/imx53-smd.dts +++ b/arch/arm/boot/dts/imx53-smd.dts @@ -1,16 +1,10 @@ -/* - * Copyright 2011 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2011 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. /dts-v1/; +#include #include "imx53.dtsi" / { @@ -27,13 +21,13 @@ volume-up { label = "Volume Up"; gpios = <&gpio2 14 0>; - linux,code = <115>; /* KEY_VOLUMEUP */ + linux,code = ; }; volume-down { label = "Volume Down"; gpios = <&gpio2 15 0>; - linux,code = <114>; /* KEY_VOLUMEDOWN */ + linux,code = ; }; }; }; diff --git a/arch/arm/boot/dts/imx53-tx53-x03x.dts b/arch/arm/boot/dts/imx53-tx53-x03x.dts index af8ec5e4417b..a7f77527269d 100644 --- a/arch/arm/boot/dts/imx53-tx53-x03x.dts +++ b/arch/arm/boot/dts/imx53-tx53-x03x.dts @@ -245,6 +245,7 @@ interrupts = <15 IRQ_TYPE_EDGE_FALLING>; reset-gpios = <&gpio2 22 GPIO_ACTIVE_LOW>; wake-gpios = <&gpio2 21 GPIO_ACTIVE_HIGH>; + wakeup-source; }; touchscreen: tsc2007@48 { diff --git a/arch/arm/boot/dts/imx53-tx53.dtsi b/arch/arm/boot/dts/imx53-tx53.dtsi index 69a2af7d6c11..54cf3e67069a 100644 --- a/arch/arm/boot/dts/imx53-tx53.dtsi +++ b/arch/arm/boot/dts/imx53-tx53.dtsi @@ -58,7 +58,7 @@ can0 = &can2; /* Make the can interface indices consistent with TX28/TX48 modules */ can1 = &can1; ipu = &ipu; - reg_can_xcvr = ®_can_xcvr; + reg-can-xcvr = ®_can_xcvr; usbh1 = &usbh1; usbotg = &usbotg; }; @@ -67,13 +67,12 @@ ckih1 { clock-frequency = <0>; }; + }; - mclk: clock@0 { - compatible = "fixed-clock"; - reg = <0>; - #clock-cells = <0>; - clock-frequency = <26000000>; - }; + mclk: clock-mclk { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <26000000>; }; gpio-keys { @@ -550,7 +549,6 @@ }; &ssi1 { - codec-handle = <&sgtl5000>; status = "okay"; }; diff --git a/arch/arm/boot/dts/imx53-voipac-dmm-668.dtsi b/arch/arm/boot/dts/imx53-voipac-dmm-668.dtsi index df8dafe2564d..f83a8c62ea53 100644 --- a/arch/arm/boot/dts/imx53-voipac-dmm-668.dtsi +++ b/arch/arm/boot/dts/imx53-voipac-dmm-668.dtsi @@ -150,7 +150,7 @@ compatible = "dlg,da9053-aa", "dlg,da9052"; reg = <0x48>; interrupt-parent = <&gpio7>; - interrupts = <11 0x8>; /* low-level active IRQ at GPIO7_11 */ + interrupts = <11 IRQ_TYPE_LEVEL_LOW>; /* low-level active IRQ at GPIO7_11 */ regulators { buck1_reg: buck1 { diff --git a/arch/arm/boot/dts/imx53.dtsi b/arch/arm/boot/dts/imx53.dtsi index 3d65c0192f69..1a7a7bb3df45 100644 --- a/arch/arm/boot/dts/imx53.dtsi +++ b/arch/arm/boot/dts/imx53.dtsi @@ -88,9 +88,6 @@ }; clocks { - #address-cells = <1>; - #size-cells = <0>; - ckil { compatible = "fsl,imx-ckil", "fixed-clock"; #clock-cells = <0>; @@ -488,6 +485,10 @@ remote-endpoint = <&ipu_di0_lvds0>; }; }; + + port@2 { + reg = <2>; + }; }; lvds-channel@1 { @@ -503,6 +504,10 @@ remote-endpoint = <&ipu_di1_lvds1>; }; }; + + port@2 { + reg = <2>; + }; }; }; diff --git a/arch/arm/boot/dts/imx6dl-aristainetos2_4.dts b/arch/arm/boot/dts/imx6dl-aristainetos2_4.dts index 7128c76d5721..29940ba215a8 100644 --- a/arch/arm/boot/dts/imx6dl-aristainetos2_4.dts +++ b/arch/arm/boot/dts/imx6dl-aristainetos2_4.dts @@ -78,8 +78,6 @@ &ecspi1 { lcd_panel: display@0 { - #address-cells = <1>; - #size-cells = <1>; compatible = "lg,lg4573"; spi-max-frequency = <10000000>; reg = <0>; diff --git a/arch/arm/boot/dts/imx6dl-colibri-eval-v3.dts b/arch/arm/boot/dts/imx6dl-colibri-eval-v3.dts index ea184d108491..3dee3af1a4c1 100644 --- a/arch/arm/boot/dts/imx6dl-colibri-eval-v3.dts +++ b/arch/arm/boot/dts/imx6dl-colibri-eval-v3.dts @@ -72,15 +72,12 @@ stdout-path = "serial0:115200n8"; }; - clocks { - /* Fixed crystal dedicated to mcp251x */ - clk16m: clk@1 { - compatible = "fixed-clock"; - reg = <1>; - #clock-cells = <0>; - clock-frequency = <16000000>; - clock-output-names = "clk16m"; - }; + /* Fixed crystal dedicated to mcp251x */ + clk16m: clock-16m { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <16000000>; + clock-output-names = "clk16m"; }; gpio-keys { diff --git a/arch/arm/boot/dts/imx6dl-mamoj.dts b/arch/arm/boot/dts/imx6dl-mamoj.dts new file mode 100644 index 000000000000..6b2d29138bed --- /dev/null +++ b/arch/arm/boot/dts/imx6dl-mamoj.dts @@ -0,0 +1,224 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR MIT) +/* + * Copyright (C) 2018 BTicino + * Copyright (C) 2018 Amarula Solutions B.V. + */ + +/dts-v1/; + +#include "imx6dl.dtsi" + +/ { + model = "BTicino i.MX6DL Mamoj board"; + compatible = "bticino,imx6dl-mamoj", "fsl,imx6dl"; +}; + +&fec { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_enet>; + phy-mode = "mii"; + status = "okay"; +}; + +&i2c3 { + clock-frequency = <400000>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_i2c3>; + status = "okay"; +}; + +&i2c4 { + clock-frequency = <100000>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_i2c4>; + status = "okay"; + + pfuze100: pmic@8 { + compatible = "fsl,pfuze100"; + reg = <0x08>; + + regulators { + /* CPU vdd_arm core */ + sw1a_reg: sw1ab { + regulator-min-microvolt = <300000>; + regulator-max-microvolt = <1875000>; + regulator-boot-on; + regulator-always-on; + regulator-ramp-delay = <6250>; + }; + + /* SOC vdd_soc */ + sw1c_reg: sw1c { + regulator-min-microvolt = <300000>; + regulator-max-microvolt = <1875000>; + regulator-boot-on; + regulator-always-on; + regulator-ramp-delay = <6250>; + }; + + /* I/O power GEN_3V3 */ + sw2_reg: sw2 { + regulator-min-microvolt = <800000>; + regulator-max-microvolt = <3300000>; + regulator-boot-on; + regulator-always-on; + }; + + /* DDR memory */ + sw3a_reg: sw3a { + regulator-min-microvolt = <400000>; + regulator-max-microvolt = <1975000>; + regulator-boot-on; + regulator-always-on; + }; + + /* DDR memory */ + sw3b_reg: sw3b { + regulator-min-microvolt = <400000>; + regulator-max-microvolt = <1975000>; + regulator-boot-on; + regulator-always-on; + }; + + /* not used */ + sw4_reg: sw4 { + regulator-min-microvolt = <800000>; + regulator-max-microvolt = <3300000>; + }; + + /* not used */ + swbst_reg: swbst { + regulator-min-microvolt = <5000000>; + regulator-max-microvolt = <5150000>; + }; + + /* PMIC vsnvs. EX boot mode */ + snvs_reg: vsnvs { + regulator-min-microvolt = <1000000>; + regulator-max-microvolt = <3000000>; + regulator-boot-on; + regulator-always-on; + }; + + vref_reg: vrefddr { + regulator-boot-on; + regulator-always-on; + }; + + /* not used */ + vgen1_reg: vgen1 { + regulator-min-microvolt = <800000>; + regulator-max-microvolt = <1550000>; + }; + + /* not used */ + vgen2_reg: vgen2 { + regulator-min-microvolt = <800000>; + regulator-max-microvolt = <1550000>; + }; + + /* not used */ + vgen3_reg: vgen3 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <3300000>; + }; + + /* 1v8 general power */ + vgen4_reg: vgen4 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; + + /* 2v8 general power IMX6 */ + vgen5_reg: vgen5 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; + + /* 3v3 Ethernet */ + vgen6_reg: vgen6 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; + }; + }; +}; + +&uart3 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_uart3>; + status = "okay"; +}; + +&usdhc3 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_usdhc3>; + bus-width = <8>; + non-removable; + keep-power-in-suspend; + status = "okay"; +}; + +&iomuxc { + pinctrl_enet: enetgrp { + fsl,pins = < + MX6QDL_PAD_ENET_MDIO__ENET_MDIO 0x1b0b0 + MX6QDL_PAD_ENET_MDC__ENET_MDC 0x1b0b0 + MX6QDL_PAD_ENET_REF_CLK__ENET_TX_CLK 0x1b0b1 + MX6QDL_PAD_ENET_TXD0__ENET_TX_DATA0 0x1b0b0 + MX6QDL_PAD_ENET_TXD1__ENET_TX_DATA1 0x1b0b0 + MX6QDL_PAD_KEY_ROW2__ENET_TX_DATA2 0x1b0b0 + MX6QDL_PAD_KEY_ROW0__ENET_TX_DATA3 0x1b0b0 + MX6QDL_PAD_ENET_TX_EN__ENET_TX_EN 0x1b0b0 + MX6QDL_PAD_GPIO_19__ENET_TX_ER 0x1b0b0 + MX6QDL_PAD_GPIO_18__ENET_RX_CLK 0x1b0b1 + MX6QDL_PAD_ENET_RXD0__ENET_RX_DATA0 0x1b0b0 + MX6QDL_PAD_ENET_RXD1__ENET_RX_DATA1 0x1b0b0 + MX6QDL_PAD_KEY_COL2__ENET_RX_DATA2 0x1b0b0 + MX6QDL_PAD_KEY_COL0__ENET_RX_DATA3 0x1b0b0 + MX6QDL_PAD_ENET_CRS_DV__ENET_RX_EN 0x1b0b0 + MX6QDL_PAD_ENET_RX_ER__ENET_RX_ER 0x1b0b0 + MX6QDL_PAD_KEY_COL3__ENET_CRS 0x1b0b0 + MX6QDL_PAD_KEY_ROW1__ENET_COL 0x1b0b0 + >; + }; + + pinctrl_i2c3: i2c3grp { + fsl,pins = < + MX6QDL_PAD_GPIO_3__I2C3_SCL 0x4001b8b1 + MX6QDL_PAD_GPIO_6__I2C3_SDA 0x4001b8b1 + >; + }; + + pinctrl_i2c4: i2c4grp { + fsl,pins = < + MX6QDL_PAD_GPIO_7__I2C4_SCL 0x4001b8b1 + MX6QDL_PAD_GPIO_8__I2C4_SDA 0x4001b8b1 + >; + }; + + pinctrl_uart3: uart3grp { + fsl,pins = < + MX6QDL_PAD_EIM_D24__UART3_TX_DATA 0x1b0b1 + MX6QDL_PAD_EIM_D25__UART3_RX_DATA 0x1b0b1 + >; + }; + + pinctrl_usdhc3: usdhc3grp { + fsl,pins = < + MX6QDL_PAD_SD3_CMD__SD3_CMD 0x17059 + MX6QDL_PAD_SD3_CLK__SD3_CLK 0x10059 + MX6QDL_PAD_SD3_DAT0__SD3_DATA0 0x17059 + MX6QDL_PAD_SD3_DAT1__SD3_DATA1 0x17059 + MX6QDL_PAD_SD3_DAT2__SD3_DATA2 0x17059 + MX6QDL_PAD_SD3_DAT3__SD3_DATA3 0x17059 + MX6QDL_PAD_SD3_DAT4__SD3_DATA4 0x17059 + MX6QDL_PAD_SD3_DAT5__SD3_DATA5 0x17059 + MX6QDL_PAD_SD3_DAT6__SD3_DATA6 0x17059 + MX6QDL_PAD_SD3_DAT7__SD3_DATA7 0x17059 + >; + }; +}; diff --git a/arch/arm/boot/dts/imx6dl-sabreauto.dts b/arch/arm/boot/dts/imx6dl-sabreauto.dts index a6ce7b487ad7..660d52a245ba 100644 --- a/arch/arm/boot/dts/imx6dl-sabreauto.dts +++ b/arch/arm/boot/dts/imx6dl-sabreauto.dts @@ -1,10 +1,6 @@ -/* - * Copyright (C) 2013 Freescale Semiconductor, Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright (C) 2013 Freescale Semiconductor, Inc. /dts-v1/; diff --git a/arch/arm/boot/dts/imx6dl-sabresd.dts b/arch/arm/boot/dts/imx6dl-sabresd.dts index 9607afe088fc..cd6bbf22a16f 100644 --- a/arch/arm/boot/dts/imx6dl-sabresd.dts +++ b/arch/arm/boot/dts/imx6dl-sabresd.dts @@ -1,10 +1,6 @@ -/* - * Copyright (C) 2013 Freescale Semiconductor, Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright (C) 2013 Freescale Semiconductor, Inc. /dts-v1/; diff --git a/arch/arm/boot/dts/imx6dl-udoo.dts b/arch/arm/boot/dts/imx6dl-udoo.dts index e3713f00e819..d871cac1711f 100644 --- a/arch/arm/boot/dts/imx6dl-udoo.dts +++ b/arch/arm/boot/dts/imx6dl-udoo.dts @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright 2013 Freescale Semiconductor, Inc. * * Author: Fabio Estevam - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * */ /dts-v1/; #include "imx6dl.dtsi" diff --git a/arch/arm/boot/dts/imx6dl-wandboard-revb1.dts b/arch/arm/boot/dts/imx6dl-wandboard-revb1.dts index 5727fa48cfd5..738db4fc7702 100644 --- a/arch/arm/boot/dts/imx6dl-wandboard-revb1.dts +++ b/arch/arm/boot/dts/imx6dl-wandboard-revb1.dts @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright 2013 Freescale Semiconductor, Inc. * * Author: Fabio Estevam - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * */ /dts-v1/; #include "imx6dl.dtsi" diff --git a/arch/arm/boot/dts/imx6dl-wandboard-revd1.dts b/arch/arm/boot/dts/imx6dl-wandboard-revd1.dts index a72c07db7dda..51de6b4bd7d8 100644 --- a/arch/arm/boot/dts/imx6dl-wandboard-revd1.dts +++ b/arch/arm/boot/dts/imx6dl-wandboard-revd1.dts @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright 2013 Freescale Semiconductor, Inc. * * Author: Fabio Estevam - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * */ /dts-v1/; #include "imx6dl.dtsi" diff --git a/arch/arm/boot/dts/imx6dl-wandboard.dts b/arch/arm/boot/dts/imx6dl-wandboard.dts index a09f274cd1f4..b43454deaa1a 100644 --- a/arch/arm/boot/dts/imx6dl-wandboard.dts +++ b/arch/arm/boot/dts/imx6dl-wandboard.dts @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright 2013 Freescale Semiconductor, Inc. * * Author: Fabio Estevam - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * */ /dts-v1/; #include "imx6dl.dtsi" diff --git a/arch/arm/boot/dts/imx6dl.dtsi b/arch/arm/boot/dts/imx6dl.dtsi index 558bce81209d..b384913c34dd 100644 --- a/arch/arm/boot/dts/imx6dl.dtsi +++ b/arch/arm/boot/dts/imx6dl.dtsi @@ -1,12 +1,6 @@ - -/* - * Copyright 2013 Freescale Semiconductor, Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - */ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright 2013 Freescale Semiconductor, Inc. #include #include "imx6dl-pinfunc.h" diff --git a/arch/arm/boot/dts/imx6q-b850v3.dts b/arch/arm/boot/dts/imx6q-b850v3.dts index 35edbdc7bcd1..044a5bebe1c5 100644 --- a/arch/arm/boot/dts/imx6q-b850v3.dts +++ b/arch/arm/boot/dts/imx6q-b850v3.dts @@ -156,8 +156,6 @@ stdp2690@72 { compatible = "megachips,stdp2690-ge-b850v3-fw"; - #address-cells = <1>; - #size-cells = <0>; reg = <0x72>; ports { @@ -184,8 +182,6 @@ stdp4028@73 { compatible = "megachips,stdp4028-ge-b850v3-fw"; - #address-cells = <1>; - #size-cells = <0>; reg = <0x73>; interrupt-parent = <&gpio2>; interrupts = <0 IRQ_TYPE_LEVEL_HIGH>; diff --git a/arch/arm/boot/dts/imx6q-ba16.dtsi b/arch/arm/boot/dts/imx6q-ba16.dtsi index bf4bdb385de9..e903c488287b 100644 --- a/arch/arm/boot/dts/imx6q-ba16.dtsi +++ b/arch/arm/boot/dts/imx6q-ba16.dtsi @@ -157,7 +157,12 @@ partition@d0000 { label = "spare"; - reg = <0xd0000 0x130000>; + reg = <0xd0000 0x320000>; + }; + + partition@3f0000 { + label = "mfg"; + reg = <0x3f0000 0x10000>; }; }; }; diff --git a/arch/arm/boot/dts/imx6q-bx50v3.dtsi b/arch/arm/boot/dts/imx6q-bx50v3.dtsi index 990e411cbca0..d3cba09be0cb 100644 --- a/arch/arm/boot/dts/imx6q-bx50v3.dtsi +++ b/arch/arm/boot/dts/imx6q-bx50v3.dtsi @@ -43,13 +43,10 @@ #include "imx6q-ba16.dtsi" / { - clocks { - mclk: clock@0 { - compatible = "fixed-clock"; - reg = <0>; - #clock-cells = <0>; - clock-frequency = <22000000>; - }; + mclk: clock-mclk { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <22000000>; }; gpio-poweroff { @@ -107,8 +104,6 @@ switch@0 { compatible = "marvell,mv88e6085"; /* 88e6240*/ - #address-cells = <1>; - #size-cells = <0>; reg = <0>; switch_ports: ports { diff --git a/arch/arm/boot/dts/imx6q-dhcom-pdk2.dts b/arch/arm/boot/dts/imx6q-dhcom-pdk2.dts new file mode 100644 index 000000000000..9c61e3be2d9a --- /dev/null +++ b/arch/arm/boot/dts/imx6q-dhcom-pdk2.dts @@ -0,0 +1,151 @@ +// SPDX-License-Identifier: (GPL-2.0+) +/* + * Copyright (C) 2015 DH electronics GmbH + * Copyright (C) 2018 Marek Vasut + */ + +/dts-v1/; + +#include "imx6q-dhcom-som.dtsi" + +/ { + model = "Freescale i.MX6 Quad DHCOM Premium Developer Kit (2)"; + compatible = "dh,imx6q-dhcom-pdk2", "dh,imx6q-dhcom-som", "fsl,imx6q"; + + chosen { + stdout-path = &uart1; + }; + + clk_ext_audio_codec: clock-codec { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <24000000>; + }; + + sound { + compatible = "fsl,imx-audio-sgtl5000"; + model = "imx-sgtl5000"; + ssi-controller = <&ssi1>; + audio-codec = <&sgtl5000>; + audio-routing = + "MIC_IN", "Mic Jack", + "Mic Jack", "Mic Bias", + "LINE_IN", "Line In Jack", + "Headphone Jack", "HP_OUT"; + mux-int-port = <1>; + mux-ext-port = <3>; + }; +}; + +&audmux { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_audmux_ext>; + status = "okay"; +}; + +&hdmi { + ddc-i2c-bus = <&i2c2>; + status = "okay"; +}; + +&i2c2 { + sgtl5000: codec@a { + compatible = "fsl,sgtl5000"; + reg = <0x0a>; + #sound-dai-cells = <0>; + clocks = <&clk_ext_audio_codec>; + VDDA-supply = <®_3p3v>; + VDDIO-supply = <®_3p3v>; + }; +}; + +&iomuxc { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_hog_base &pinctrl_hog>; + + pinctrl_hog: hog-grp { + fsl,pins = < + MX6QDL_PAD_GPIO_2__GPIO1_IO02 0x400120b0 + MX6QDL_PAD_GPIO_4__GPIO1_IO04 0x400120b0 + MX6QDL_PAD_GPIO_5__GPIO1_IO05 0x400120b0 + MX6QDL_PAD_CSI0_DAT17__GPIO6_IO03 0x400120b0 + MX6QDL_PAD_GPIO_19__GPIO4_IO05 0x120b0 + MX6QDL_PAD_DI0_PIN4__GPIO4_IO20 0x400120b0 + MX6QDL_PAD_EIM_D27__GPIO3_IO27 0x120b0 + MX6QDL_PAD_KEY_ROW0__GPIO4_IO07 0x120b0 + MX6QDL_PAD_KEY_COL1__GPIO4_IO08 0x400120b0 + MX6QDL_PAD_NANDF_CS1__GPIO6_IO14 0x400120b0 + MX6QDL_PAD_NANDF_CS2__GPIO6_IO15 0x400120b0 + MX6QDL_PAD_KEY_ROW1__GPIO4_IO09 0x400120b0 + MX6QDL_PAD_SD3_DAT5__GPIO7_IO00 0x400120b0 + MX6QDL_PAD_SD3_DAT4__GPIO7_IO01 0x400120b0 + MX6QDL_PAD_CSI0_VSYNC__GPIO5_IO21 0x400120b0 + MX6QDL_PAD_GPIO_18__GPIO7_IO13 0x400120b0 + MX6QDL_PAD_SD1_CMD__GPIO1_IO18 0x400120b0 + MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x400120b0 + MX6QDL_PAD_SD1_DAT1__GPIO1_IO17 0x400120b0 + MX6QDL_PAD_SD1_DAT2__GPIO1_IO19 0x400120b0 + MX6QDL_PAD_SD1_CLK__GPIO1_IO20 0x400120b0 + MX6QDL_PAD_CSI0_PIXCLK__GPIO5_IO18 0x400120b0 + MX6QDL_PAD_CSI0_MCLK__GPIO5_IO19 0x400120b0 + MX6QDL_PAD_KEY_COL0__GPIO4_IO06 0x400120b0 + >; + }; + + pinctrl_audmux_ext: audmux-ext-grp { + fsl,pins = < + MX6QDL_PAD_CSI0_DAT7__AUD3_RXD 0x130b0 + MX6QDL_PAD_CSI0_DAT4__AUD3_TXC 0x130b0 + MX6QDL_PAD_CSI0_DAT5__AUD3_TXD 0x110b0 + MX6QDL_PAD_CSI0_DAT6__AUD3_TXFS 0x130b0 + >; + }; + + pinctrl_enet_1G: enet-1G-grp { + fsl,pins = < + MX6QDL_PAD_ENET_MDIO__ENET_MDIO 0x100b0 + MX6QDL_PAD_ENET_MDC__ENET_MDC 0x100b0 + MX6QDL_PAD_RGMII_TXC__RGMII_TXC 0x100b0 + MX6QDL_PAD_RGMII_TD0__RGMII_TD0 0x100b0 + MX6QDL_PAD_RGMII_TD1__RGMII_TD1 0x100b0 + MX6QDL_PAD_RGMII_TD2__RGMII_TD2 0x100b0 + MX6QDL_PAD_RGMII_TD3__RGMII_TD3 0x100b0 + MX6QDL_PAD_RGMII_TX_CTL__RGMII_TX_CTL 0x100b0 + MX6QDL_PAD_ENET_REF_CLK__ENET_TX_CLK 0x100b0 + MX6QDL_PAD_RGMII_RXC__RGMII_RXC 0x1b0b0 + MX6QDL_PAD_RGMII_RD0__RGMII_RD0 0x1b0b0 + MX6QDL_PAD_RGMII_RD1__RGMII_RD1 0x1b0b0 + MX6QDL_PAD_RGMII_RD2__RGMII_RD2 0x1b0b0 + MX6QDL_PAD_RGMII_RD3__RGMII_RD3 0x1b0b0 + MX6QDL_PAD_RGMII_RX_CTL__RGMII_RX_CTL 0x1b0b0 + MX6QDL_PAD_EIM_D29__GPIO3_IO29 0x000b0 + MX6QDL_PAD_GPIO_0__GPIO1_IO00 0x000b1 + MX6QDL_PAD_EIM_D26__GPIO3_IO26 0x000b1 + >; + }; + + pinctrl_pcie: pcie-grp { + fsl,pins = < + MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20 0x1b0b1 + >; + }; +}; + +&pcie { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_pcie>; + reset-gpio = <&gpio6 14 GPIO_ACTIVE_LOW>; + status = "okay"; +}; + +&ssi1 { + status = "okay"; +}; + +&sata { + status = "okay"; +}; + +&usdhc3 { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/imx6q-dhcom-som.dtsi b/arch/arm/boot/dts/imx6q-dhcom-som.dtsi new file mode 100644 index 000000000000..bbba0671f0f4 --- /dev/null +++ b/arch/arm/boot/dts/imx6q-dhcom-som.dtsi @@ -0,0 +1,476 @@ +// SPDX-License-Identifier: (GPL-2.0+) +/* + * Copyright (C) 2015 DH electronics GmbH + * Copyright (C) 2018 Marek Vasut + */ + +#include "imx6q.dtsi" +#include +#include +#include +#include + +/ { + aliases { + mmc0 = &usdhc2; + mmc1 = &usdhc3; + mmc2 = &usdhc4; + mmc3 = &usdhc1; + }; + + memory@10000000 { + reg = <0x10000000 0x40000000>; + }; + + reg_usb_otg_vbus: regulator-usb-otg-vbus { + compatible = "regulator-fixed"; + regulator-name = "usb_otg_vbus"; + regulator-min-microvolt = <5000000>; + regulator-max-microvolt = <5000000>; + }; + + reg_usb_h1_vbus: regulator-usb-h1-vbus { + compatible = "regulator-fixed"; + regulator-name = "usb_h1_vbus"; + regulator-min-microvolt = <5000000>; + regulator-max-microvolt = <5000000>; + gpio = <&gpio3 31 GPIO_ACTIVE_HIGH>; + enable-active-high; + }; + + reg_3p3v: regulator-3P3V { + compatible = "regulator-fixed"; + regulator-name = "3P3V"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; +}; + +&can1 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_flexcan1>; + status = "okay"; +}; + +&can2 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_flexcan2>; + status = "okay"; +}; + +&ecspi1 { + cs-gpios = <&gpio2 30 GPIO_ACTIVE_HIGH>, <&gpio4 11 GPIO_ACTIVE_HIGH>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_ecspi1>; + status = "okay"; + + flash@0 { /* S25FL116K */ + #address-cells = <1>; + #size-cells = <1>; + compatible = "jedec,spi-nor"; + spi-max-frequency = <50000000>; + reg = <0>; + m25p,fast-read; + }; +}; + +&ecspi2 { + cs-gpios = <&gpio5 29 GPIO_ACTIVE_LOW>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_ecspi2>; + status = "okay"; +}; + +&fec { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_enet_100M>; + phy-mode = "rmii"; + phy-handle = <ðphy0>; + status = "okay"; + + mdio { + #address-cells = <1>; + #size-cells = <0>; + + ethphy0: ethernet-phy@0 { /* SMSC LAN8710Ai */ + reg = <0>; + max-speed = <100>; + reset-gpios = <&gpio5 0 GPIO_ACTIVE_LOW>; + reset-delay-us = <1000>; + reset-post-delay-us = <1000>; + }; + }; +}; + +&i2c1 { + clock-frequency = <100000>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_i2c1>; + status = "okay"; +}; + +&i2c2 { + clock-frequency = <100000>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_i2c2>; + status = "okay"; +}; + +&i2c3 { + clock-frequency = <100000>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_i2c3>; + status = "okay"; + + ltc3676: pmic@3c { + compatible = "lltc,ltc3676"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_pmic_hw300>; + reg = <0x3c>; + interrupt-parent = <&gpio5>; + interrupts = <2 IRQ_TYPE_EDGE_FALLING>; + + regulators { + sw1_reg: sw1 { + regulator-min-microvolt = <787500>; + regulator-max-microvolt = <1527272>; + lltc,fb-voltage-divider = <100000 110000>; + regulator-suspend-mem-microvolt = <1040000>; + regulator-ramp-delay = <7000>; + regulator-boot-on; + regulator-always-on; + }; + + sw2_reg: sw2 { + regulator-min-microvolt = <1885714>; + regulator-max-microvolt = <3657142>; + lltc,fb-voltage-divider = <100000 28000>; + regulator-ramp-delay = <7000>; + regulator-boot-on; + regulator-always-on; + }; + + sw3_reg: sw3 { + regulator-min-microvolt = <787500>; + regulator-max-microvolt = <1527272>; + lltc,fb-voltage-divider = <100000 110000>; + regulator-suspend-mem-microvolt = <980000>; + regulator-ramp-delay = <7000>; + regulator-boot-on; + regulator-always-on; + }; + + sw4_reg: sw4 { + regulator-min-microvolt = <855571>; + regulator-max-microvolt = <1659291>; + lltc,fb-voltage-divider = <100000 93100>; + regulator-ramp-delay = <7000>; + regulator-boot-on; + regulator-always-on; + }; + + ldo1_reg: ldo1 { + regulator-min-microvolt = <3240306>; + regulator-max-microvolt = <3240306>; + lltc,fb-voltage-divider = <102000 29400>; + regulator-boot-on; + regulator-always-on; + }; + + ldo2_reg: ldo2 { + regulator-min-microvolt = <2484708>; + regulator-max-microvolt = <2484708>; + lltc,fb-voltage-divider = <100000 41200>; + regulator-boot-on; + regulator-always-on; + }; + }; + }; + + touchscreen@49 { /* TSC2004 */ + compatible = "ti,tsc2004"; + reg = <0x49>; + vio-supply = <®_3p3v>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_tsc2004_hw300>; + interrupts-extended = <&gpio4 14 IRQ_TYPE_EDGE_FALLING>; + status = "disabled"; + }; + + eeprom@50 { + compatible = "atmel,24c02"; + reg = <0x50>; + pagesize = <16>; + }; + + rtc@56 { + compatible = "rv3029c2"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_rtc_hw300>; + reg = <0x56>; + interrupt-parent = <&gpio7>; + interrupts = <12 2>; + }; +}; + +&iomuxc { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_hog_base>; + + pinctrl_hog_base: hog-base-grp { + fsl,pins = < + MX6QDL_PAD_EIM_A19__GPIO2_IO19 0x120b0 + MX6QDL_PAD_EIM_A23__GPIO6_IO06 0x120b0 + MX6QDL_PAD_EIM_A22__GPIO2_IO16 0x120b0 + MX6QDL_PAD_EIM_A16__GPIO2_IO22 0x120b0 + MX6QDL_PAD_EIM_A17__GPIO2_IO21 0x120b0 + >; + }; + + pinctrl_ecspi1: ecspi1-grp { + fsl,pins = < + MX6QDL_PAD_EIM_D17__ECSPI1_MISO 0x100b1 + MX6QDL_PAD_EIM_D18__ECSPI1_MOSI 0x100b1 + MX6QDL_PAD_EIM_D16__ECSPI1_SCLK 0x100b1 + MX6QDL_PAD_EIM_EB2__GPIO2_IO30 0x1b0b0 + MX6QDL_PAD_KEY_ROW2__GPIO4_IO11 0x1b0b0 + >; + }; + + pinctrl_ecspi2: ecspi2-grp { + fsl,pins = < + MX6QDL_PAD_CSI0_DAT10__ECSPI2_MISO 0x100b1 + MX6QDL_PAD_CSI0_DAT9__ECSPI2_MOSI 0x100b1 + MX6QDL_PAD_CSI0_DAT8__ECSPI2_SCLK 0x100b1 + MX6QDL_PAD_CSI0_DAT11__GPIO5_IO29 0x1b0b0 + >; + }; + + pinctrl_enet_100M: enet-100M-grp { + fsl,pins = < + MX6QDL_PAD_ENET_MDIO__ENET_MDIO 0x1b0b0 + MX6QDL_PAD_ENET_MDC__ENET_MDC 0x1b0b0 + MX6QDL_PAD_ENET_CRS_DV__ENET_RX_EN 0x1b0b0 + MX6QDL_PAD_ENET_RX_ER__ENET_RX_ER 0x1b0b0 + MX6QDL_PAD_ENET_RXD0__ENET_RX_DATA0 0x1b0b0 + MX6QDL_PAD_ENET_RXD1__ENET_RX_DATA1 0x1b0b0 + MX6QDL_PAD_ENET_TX_EN__ENET_TX_EN 0x1b0b0 + MX6QDL_PAD_ENET_TXD0__ENET_TX_DATA0 0x1b0b0 + MX6QDL_PAD_ENET_TXD1__ENET_TX_DATA1 0x1b0b0 + MX6QDL_PAD_GPIO_16__ENET_REF_CLK 0x4001b0a8 + MX6QDL_PAD_EIM_WAIT__GPIO5_IO00 0x000b0 + MX6QDL_PAD_KEY_ROW4__GPIO4_IO15 0x000b1 + MX6QDL_PAD_GPIO_7__GPIO1_IO07 0x120b0 + >; + }; + + pinctrl_flexcan1: flexcan1-grp { + fsl,pins = < + MX6QDL_PAD_KEY_COL2__FLEXCAN1_TX 0x1b0b0 + MX6QDL_PAD_GPIO_8__FLEXCAN1_RX 0x1b0b0 + >; + }; + + pinctrl_flexcan2: flexcan2-grp { + fsl,pins = < + MX6QDL_PAD_SD3_DAT0__FLEXCAN2_TX 0x1b0b0 + MX6QDL_PAD_SD3_DAT1__FLEXCAN2_RX 0x1b0b0 + >; + }; + + pinctrl_i2c1: i2c1-grp { + fsl,pins = < + MX6QDL_PAD_EIM_D21__I2C1_SCL 0x4001b8b1 + MX6QDL_PAD_EIM_D28__I2C1_SDA 0x4001b8b1 + >; + }; + + pinctrl_i2c2: i2c2-grp { + fsl,pins = < + MX6QDL_PAD_KEY_COL3__I2C2_SCL 0x4001b8b1 + MX6QDL_PAD_KEY_ROW3__I2C2_SDA 0x4001b8b1 + >; + }; + + pinctrl_i2c3: i2c3-grp { + fsl,pins = < + MX6QDL_PAD_GPIO_3__I2C3_SCL 0x4001b8b1 + MX6QDL_PAD_GPIO_6__I2C3_SDA 0x4001b8b1 + >; + }; + + pinctrl_pmic_hw300: pmic-hw300-grp { + fsl,pins = < + MX6QDL_PAD_EIM_A25__GPIO5_IO02 0x1B0B0 + >; + }; + + pinctrl_rtc_hw300: rtc-hw300-grp { + fsl,pins = < + MX6QDL_PAD_GPIO_17__GPIO7_IO12 0x120B0 + >; + }; + + pinctrl_tsc2004_hw300: tsc2004-hw300-grp { + fsl,pins = < + MX6QDL_PAD_KEY_COL4__GPIO4_IO14 0x120B0 + >; + }; + + pinctrl_uart1: uart1-grp { + fsl,pins = < + MX6QDL_PAD_SD3_DAT7__UART1_TX_DATA 0x1b0b1 + MX6QDL_PAD_SD3_DAT6__UART1_RX_DATA 0x1b0b1 + MX6QDL_PAD_EIM_D20__UART1_RTS_B 0x1b0b1 + MX6QDL_PAD_EIM_D19__UART1_CTS_B 0x4001b0b1 + MX6QDL_PAD_EIM_D23__GPIO3_IO23 0x4001b0b1 + MX6QDL_PAD_EIM_D24__GPIO3_IO24 0x4001b0b1 + MX6QDL_PAD_EIM_D25__GPIO3_IO25 0x4001b0b1 + MX6QDL_PAD_EIM_EB3__GPIO2_IO31 0x4001b0b1 + >; + }; + + pinctrl_uart4: uart4-grp { + fsl,pins = < + MX6QDL_PAD_CSI0_DAT12__UART4_TX_DATA 0x1b0b1 + MX6QDL_PAD_CSI0_DAT13__UART4_RX_DATA 0x1b0b1 + >; + }; + + pinctrl_uart5: uart5-grp { + fsl,pins = < + MX6QDL_PAD_CSI0_DAT14__UART5_TX_DATA 0x1b0b1 + MX6QDL_PAD_CSI0_DAT15__UART5_RX_DATA 0x1b0b1 + MX6QDL_PAD_CSI0_DAT18__UART5_RTS_B 0x1b0b1 + MX6QDL_PAD_CSI0_DAT19__UART5_CTS_B 0x4001b0b1 + >; + }; + + pinctrl_usbh1: usbh1-grp { + fsl,pins = < + MX6QDL_PAD_EIM_D31__GPIO3_IO31 0x120B0 + >; + }; + + pinctrl_usbotg: usbotg-grp { + fsl,pins = < + MX6QDL_PAD_GPIO_1__USB_OTG_ID 0x17059 + >; + }; + + pinctrl_usdhc2: usdhc2-grp { + fsl,pins = < + MX6QDL_PAD_SD2_CMD__SD2_CMD 0x17059 + MX6QDL_PAD_SD2_CLK__SD2_CLK 0x10059 + MX6QDL_PAD_SD2_DAT0__SD2_DATA0 0x17059 + MX6QDL_PAD_SD2_DAT1__SD2_DATA1 0x17059 + MX6QDL_PAD_SD2_DAT2__SD2_DATA2 0x17059 + MX6QDL_PAD_SD2_DAT3__SD2_DATA3 0x17059 + MX6QDL_PAD_NANDF_CS3__GPIO6_IO16 0x120B0 + >; + }; + + pinctrl_usdhc3: usdhc3-grp { + fsl,pins = < + MX6QDL_PAD_SD3_CMD__SD3_CMD 0x17059 + MX6QDL_PAD_SD3_CLK__SD3_CLK 0x10059 + MX6QDL_PAD_SD3_DAT0__SD3_DATA0 0x17059 + MX6QDL_PAD_SD3_DAT1__SD3_DATA1 0x17059 + MX6QDL_PAD_SD3_DAT2__SD3_DATA2 0x17059 + MX6QDL_PAD_SD3_DAT3__SD3_DATA3 0x17059 + MX6QDL_PAD_SD3_RST__GPIO7_IO08 0x120B0 + >; + }; + + pinctrl_usdhc4: usdhc4-grp { + fsl,pins = < + MX6QDL_PAD_SD4_CMD__SD4_CMD 0x17059 + MX6QDL_PAD_SD4_CLK__SD4_CLK 0x10059 + MX6QDL_PAD_SD4_DAT0__SD4_DATA0 0x17059 + MX6QDL_PAD_SD4_DAT1__SD4_DATA1 0x17059 + MX6QDL_PAD_SD4_DAT2__SD4_DATA2 0x17059 + MX6QDL_PAD_SD4_DAT3__SD4_DATA3 0x17059 + MX6QDL_PAD_SD4_DAT4__SD4_DATA4 0x17059 + MX6QDL_PAD_SD4_DAT5__SD4_DATA5 0x17059 + MX6QDL_PAD_SD4_DAT6__SD4_DATA6 0x17059 + MX6QDL_PAD_SD4_DAT7__SD4_DATA7 0x17059 + >; + }; +}; + +®_arm { + vin-supply = <&sw3_reg>; +}; + +®_soc { + vin-supply = <&sw1_reg>; +}; + +&uart1 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_uart1>; + uart-has-rtscts; + dtr-gpios = <&gpio3 24 GPIO_ACTIVE_LOW>; + dsr-gpios = <&gpio3 25 GPIO_ACTIVE_LOW>; + dcd-gpios = <&gpio3 23 GPIO_ACTIVE_LOW>; + rng-gpios = <&gpio2 31 GPIO_ACTIVE_LOW>; + status = "okay"; +}; + +&uart4 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_uart4>; + status = "okay"; +}; + +&uart5 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_uart5>; + uart-has-rtscts; + status = "okay"; +}; + +&usbh1 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_usbh1>; + vbus-supply = <®_usb_h1_vbus>; + dr_mode = "host"; + status = "okay"; +}; + +&usbotg { + vbus-supply = <®_usb_otg_vbus>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_usbotg>; + disable-over-current; + dr_mode = "otg"; + status = "okay"; +}; + +&usdhc2 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_usdhc2>; + cd-gpios = <&gpio6 16 GPIO_ACTIVE_HIGH>; + keep-power-in-suspend; + status = "okay"; +}; + +&usdhc3 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_usdhc3>; + cd-gpios = <&gpio7 8 GPIO_ACTIVE_LOW>; + fsl,wp-controller; + keep-power-in-suspend; + status = "disabled"; +}; + +&usdhc4 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_usdhc4>; + non-removable; + bus-width = <8>; + no-1-8-v; + keep-power-in-suspend; + status = "okay"; +}; diff --git a/arch/arm/boot/dts/imx6q-gk802.dts b/arch/arm/boot/dts/imx6q-gk802.dts index 0be375611382..84d3540b3a97 100644 --- a/arch/arm/boot/dts/imx6q-gk802.dts +++ b/arch/arm/boot/dts/imx6q-gk802.dts @@ -8,6 +8,7 @@ /dts-v1/; #include +#include #include "imx6q.dtsi" / { @@ -43,7 +44,7 @@ recovery-button { label = "recovery"; gpios = <&gpio3 16 1>; - linux,code = <0x198>; /* KEY_RESTART */ + linux,code = ; wakeup-source; }; }; diff --git a/arch/arm/boot/dts/imx6q-icore-mipi.dts b/arch/arm/boot/dts/imx6q-icore-mipi.dts new file mode 100644 index 000000000000..acd3d33476d4 --- /dev/null +++ b/arch/arm/boot/dts/imx6q-icore-mipi.dts @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR MIT) +/* + * Copyright (C) 2017 Engicam S.r.l. + * Copyright (C) 2017 Amarula Solutions B.V. + * Author: Jagan Teki + */ + +/dts-v1/; + +#include "imx6q.dtsi" +#include "imx6qdl-icore.dtsi" + +/ { + model = "Engicam i.CoreM6 Quad/Dual MIPI Starter Kit"; + compatible = "engicam,imx6-icore", "fsl,imx6q"; +}; + +&hdmi { + ddc-i2c-bus = <&i2c2>; + status = "okay"; +}; + +&usdhc3 { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/imx6q-icore-ofcap12.dts b/arch/arm/boot/dts/imx6q-icore-ofcap12.dts index 9e230f56c5fb..6e27c8143f82 100644 --- a/arch/arm/boot/dts/imx6q-icore-ofcap12.dts +++ b/arch/arm/boot/dts/imx6q-icore-ofcap12.dts @@ -48,28 +48,31 @@ / { model = "Engicam i.CoreM6 Quad/Dual OpenFrame Capacitive touch 12 Kit"; compatible = "engicam,imx6-icore", "fsl,imx6q"; + + panel { + compatible = "koe,tx31d200vm0baa"; + backlight = <&backlight_lvds>; + + port { + panel_in: endpoint { + remote-endpoint = <&lvds0_out>; + }; + }; + }; }; &ldb { status = "okay"; lvds-channel@0 { - fsl,data-mapping = "spwg"; - fsl,data-width = <18>; + reg = <0>; status = "okay"; - display-timings { - native-mode = <&timing0>; - timing0: timing0 { - clock-frequency = <46800000>; - hactive = <1280>; - vactive = <480>; - hback-porch = <353>; - hfront-porch = <47>; - vback-porch = <39>; - vfront-porch = <4>; - hsync-len = <8>; - vsync-len = <2>; + port@4 { + reg = <4>; + + lvds0_out: endpoint { + remote-endpoint = <&panel_in>; }; }; }; diff --git a/arch/arm/boot/dts/imx6q-kp-tpc.dts b/arch/arm/boot/dts/imx6q-kp-tpc.dts new file mode 100644 index 000000000000..302d8d06e4cc --- /dev/null +++ b/arch/arm/boot/dts/imx6q-kp-tpc.dts @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR MIT) +/* + * Copyright 2018 + * Lukasz Majewski, DENX Software Engineering, lukma@denx.de + */ + +/dts-v1/; + +#include "imx6q-kp.dtsi" + +/ { + model = "Freescale i.MX6 Qwuad K+P TPC Board"; + compatible = "kiebackpeter,imx6q-tpc", "fsl,imx6q"; + + memory@10000000 { + reg = <0x10000000 0x40000000>; + }; +}; + +&ipu1_di0_disp0 { + remote-endpoint = <&lcd_display_in>; +}; diff --git a/arch/arm/boot/dts/imx6q-kp.dtsi b/arch/arm/boot/dts/imx6q-kp.dtsi new file mode 100644 index 000000000000..24c8169baf44 --- /dev/null +++ b/arch/arm/boot/dts/imx6q-kp.dtsi @@ -0,0 +1,432 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR MIT) +/* + * Copyright 2018 + * Lukasz Majewski, DENX Software Engineering, lukma@denx.de + */ + +/dts-v1/; + +#include "imx6q.dtsi" + +#include +#include +#include + +/ { + backlight_lcd: backlight-lcd { + compatible = "pwm-backlight"; + pwms = <&pwm1 0 5000000>; + brightness-levels = <0 255>; + num-interpolated-steps = <255>; + default-brightness-level = <250>; + }; + + beeper { + compatible = "pwm-beeper"; + pwms = <&pwm2 0 500000>; + }; + + lcd_display: display { + compatible = "fsl,imx-parallel-display"; + #address-cells = <1>; + #size-cells = <0>; + interface-pix-fmt = "rgb24"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_ipu1>; + + port@0 { + reg = <0>; + + lcd_display_in: endpoint { + remote-endpoint = <&ipu1_di0_disp0>; + }; + }; + + port@1 { + reg = <1>; + + lcd_display_out: endpoint { + remote-endpoint = <&lcd_panel_in>; + }; + }; + }; + + lcd_panel: lcd-panel { + compatible = "auo,g070vvn01"; + backlight = <&backlight_lcd>; + power-supply = <®_display>; + + port { + lcd_panel_in: endpoint { + remote-endpoint = <&lcd_display_out>; + }; + }; + }; + + leds { + compatible = "gpio-leds"; + + green { + label = "led1"; + gpios = <&gpio3 16 GPIO_ACTIVE_HIGH>; + linux,default-trigger = "gpio"; + default-state = "off"; + }; + + red { + label = "led0"; + gpios = <&gpio3 23 GPIO_ACTIVE_HIGH>; + linux,default-trigger = "gpio"; + default-state = "off"; + }; + }; + + reg_3p3v: regulator-3p3v { + compatible = "regulator-fixed"; + regulator-name = "3P3V"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; + + reg_audio: regulator-audio { + compatible = "regulator-fixed"; + regulator-name = "sgtl5000-supply"; + gpio = <&gpio6 31 GPIO_ACTIVE_HIGH>; + enable-active-high; + regulator-always-on; + }; + + reg_display: regulator-display { + compatible = "regulator-fixed"; + regulator-name = "display-supply"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; + + reg_usb_h1_vbus: regulator-usb_h1_vbus { + compatible = "regulator-fixed"; + regulator-name = "usb_h1_vbus"; + regulator-min-microvolt = <5000000>; + regulator-max-microvolt = <5000000>; + enable-active-high; + }; + + sound { + compatible = "simple-audio-card"; + simple-audio-card,name = "imx6q-sgtl5000-audio"; + simple-audio-card,format = "i2s"; + simple-audio-card,bitclock-master = <&codec_dai>; + simple-audio-card,frame-master = <&codec_dai>; + + cpu_dai: simple-audio-card,cpu { + sound-dai = <&ssi1>; + }; + + codec_dai: simple-audio-card,codec { + sound-dai = <&sgtl5000>; + }; + }; +}; + +&audmux { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_audmux>; + status = "okay"; + + ssi1 { + fsl,audmux-port = <0>; + fsl,port-config = < + (IMX_AUDMUX_V2_PTCR_SYN | + IMX_AUDMUX_V2_PTCR_TFSEL(2) | + IMX_AUDMUX_V2_PTCR_TCSEL(2) | + IMX_AUDMUX_V2_PTCR_TFSDIR | + IMX_AUDMUX_V2_PTCR_TCLKDIR) + IMX_AUDMUX_V2_PDCR_RXDSEL(2) + >; + }; + + aud3 { + fsl,audmux-port = <2>; + fsl,port-config = < + IMX_AUDMUX_V2_PTCR_SYN + IMX_AUDMUX_V2_PDCR_RXDSEL(0) + >; + }; +}; + +&can1 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_flexcan1>; +}; + +&can2 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_flexcan2>; +}; + +&fec { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_enet>; + phy-mode = "rgmii"; + fsl,magic-packet; + status = "okay"; +}; + +&i2c1 { + clock-frequency = <400000>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_i2c1>; + status = "okay"; + + touchscreen@5d { + compatible = "goodix,gt911"; + reg = <0x5d>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_ts>; + interrupt-parent = <&gpio1>; + interrupts = <9 IRQ_TYPE_EDGE_FALLING>; + irq-gpios = <&gpio1 9 GPIO_ACTIVE_HIGH>; + reset-gpios = <&gpio5 2 GPIO_ACTIVE_HIGH>; + }; + + ds1307: rtc@32 { + compatible = "dallas,ds1307"; + reg = <0x32>; + }; +}; + +&i2c2 { + clock-frequency = <400000>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_i2c2>; + status = "okay"; + + sgtl5000: audio-codec@a { + compatible = "fsl,sgtl5000"; + #sound-dai-cells = <0>; + reg = <0x0a>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_codec>; + clocks = <&clks IMX6QDL_CLK_CKO>; + VDDA-supply = <®_3p3v>; + VDDIO-supply = <®_3p3v>; + }; +}; + +&iomuxc { + pinctrl_audmux: audmuxgrp { + fsl,pins = < + MX6QDL_PAD_CSI0_DAT7__AUD3_RXD 0x130b0 + MX6QDL_PAD_CSI0_DAT4__AUD3_TXC 0x130b0 + MX6QDL_PAD_CSI0_DAT5__AUD3_TXD 0x110b0 + MX6QDL_PAD_CSI0_DAT6__AUD3_TXFS 0x130b0 + >; + }; + + pinctrl_codec: codecgrp { + fsl,pins = < + MX6QDL_PAD_EIM_BCLK__GPIO6_IO31 0x1b0b0 + /* sgtl5000 sys_mclk clock routed to CLKO1 */ + MX6QDL_PAD_GPIO_0__CCM_CLKO1 0x000b0 + >; + }; + + pinctrl_enet: enetgrp { + fsl,pins = < + MX6QDL_PAD_ENET_MDIO__ENET_MDIO 0x1b0b0 + MX6QDL_PAD_ENET_MDC__ENET_MDC 0x1b0b0 + MX6QDL_PAD_RGMII_TXC__RGMII_TXC 0x1b0b0 + MX6QDL_PAD_RGMII_TD0__RGMII_TD0 0x1b0b0 + MX6QDL_PAD_RGMII_TD1__RGMII_TD1 0x1b0b0 + MX6QDL_PAD_RGMII_TD2__RGMII_TD2 0x1b0b0 + MX6QDL_PAD_RGMII_TD3__RGMII_TD3 0x1b0b0 + MX6QDL_PAD_RGMII_TX_CTL__RGMII_TX_CTL 0x1b0b0 + MX6QDL_PAD_ENET_REF_CLK__ENET_TX_CLK 0x1b0b0 + MX6QDL_PAD_RGMII_RXC__RGMII_RXC 0x1b0b0 + MX6QDL_PAD_RGMII_RD0__RGMII_RD0 0x1b0b0 + MX6QDL_PAD_RGMII_RD1__RGMII_RD1 0x1b0b0 + MX6QDL_PAD_RGMII_RD2__RGMII_RD2 0x1b0b0 + MX6QDL_PAD_RGMII_RD3__RGMII_RD3 0x1b0b0 + MX6QDL_PAD_RGMII_RX_CTL__RGMII_RX_CTL 0x1b0b0 + MX6QDL_PAD_GPIO_16__ENET_REF_CLK 0x4001b0a8 + >; + }; + + pinctrl_flexcan1: can1grp { + fsl,pins = < + MX6QDL_PAD_GPIO_7__FLEXCAN1_TX 0x1b0b0 + MX6QDL_PAD_GPIO_8__FLEXCAN1_RX 0x1b0b0 + >; + }; + + pinctrl_flexcan2: can2grp { + fsl,pins = < + MX6QDL_PAD_KEY_COL4__FLEXCAN2_TX 0x1b0b0 + MX6QDL_PAD_KEY_ROW4__FLEXCAN2_RX 0x1b0b0 + >; + }; + + pinctrl_i2c1: i2c1grp { + fsl,pins = < + MX6QDL_PAD_CSI0_DAT8__I2C1_SDA 0x4001b8b1 + MX6QDL_PAD_CSI0_DAT9__I2C1_SCL 0x4001b8b1 + >; + }; + + pinctrl_i2c2: i2c2grp { + fsl,pins = < + MX6QDL_PAD_KEY_COL3__I2C2_SCL 0x4001b8b1 + MX6QDL_PAD_KEY_ROW3__I2C2_SDA 0x4001b8b1 + >; + }; + + pinctrl_ipu1: ipu1grp { + fsl,pins = < + MX6QDL_PAD_DI0_DISP_CLK__IPU1_DI0_DISP_CLK 0x10 + MX6QDL_PAD_DI0_PIN15__IPU1_DI0_PIN15 0x10 + MX6QDL_PAD_DI0_PIN2__IPU1_DI0_PIN02 0x10 + MX6QDL_PAD_DI0_PIN3__IPU1_DI0_PIN03 0x10 + MX6QDL_PAD_DISP0_DAT0__IPU1_DISP0_DATA00 0x10 + MX6QDL_PAD_DISP0_DAT1__IPU1_DISP0_DATA01 0x10 + MX6QDL_PAD_DISP0_DAT2__IPU1_DISP0_DATA02 0x10 + MX6QDL_PAD_DISP0_DAT3__IPU1_DISP0_DATA03 0x10 + MX6QDL_PAD_DISP0_DAT4__IPU1_DISP0_DATA04 0x10 + MX6QDL_PAD_DISP0_DAT5__IPU1_DISP0_DATA05 0x10 + MX6QDL_PAD_DISP0_DAT6__IPU1_DISP0_DATA06 0x10 + MX6QDL_PAD_DISP0_DAT7__IPU1_DISP0_DATA07 0x10 + MX6QDL_PAD_DISP0_DAT8__IPU1_DISP0_DATA08 0x10 + MX6QDL_PAD_DISP0_DAT9__IPU1_DISP0_DATA09 0x10 + MX6QDL_PAD_DISP0_DAT10__IPU1_DISP0_DATA10 0x10 + MX6QDL_PAD_DISP0_DAT11__IPU1_DISP0_DATA11 0x10 + MX6QDL_PAD_DISP0_DAT12__IPU1_DISP0_DATA12 0x10 + MX6QDL_PAD_DISP0_DAT13__IPU1_DISP0_DATA13 0x10 + MX6QDL_PAD_DISP0_DAT14__IPU1_DISP0_DATA14 0x10 + MX6QDL_PAD_DISP0_DAT15__IPU1_DISP0_DATA15 0x10 + MX6QDL_PAD_DISP0_DAT16__IPU1_DISP0_DATA16 0x10 + MX6QDL_PAD_DISP0_DAT17__IPU1_DISP0_DATA17 0x10 + MX6QDL_PAD_DISP0_DAT18__IPU1_DISP0_DATA18 0x10 + MX6QDL_PAD_DISP0_DAT19__IPU1_DISP0_DATA19 0x10 + MX6QDL_PAD_DISP0_DAT20__IPU1_DISP0_DATA20 0x10 + MX6QDL_PAD_DISP0_DAT21__IPU1_DISP0_DATA21 0x10 + MX6QDL_PAD_DISP0_DAT22__IPU1_DISP0_DATA22 0x10 + MX6QDL_PAD_DISP0_DAT23__IPU1_DISP0_DATA23 0x10 + >; + }; + + pinctrl_pwm1: pwm1grp { + fsl,pins = < + MX6QDL_PAD_SD1_DAT3__PWM1_OUT 0x1b0b1 + >; + }; + + pinctrl_pwm2: pwm2grp { + fsl,pins = < + MX6QDL_PAD_SD1_DAT2__PWM2_OUT 0x1b0b1 + >; + }; + + pinctrl_ts: tsgrp { + fsl,pins = < + MX6QDL_PAD_GPIO_9__GPIO1_IO09 0x1b0b0 + MX6QDL_PAD_EIM_A25__GPIO5_IO02 0x1b0b0 + >; + }; + + pinctrl_uart1: uart1grp { + fsl,pins = < + MX6QDL_PAD_SD3_DAT7__UART1_TX_DATA 0x1b0b1 + MX6QDL_PAD_SD3_DAT6__UART1_RX_DATA 0x1b0b1 + >; + }; + + pinctrl_uart2: uart2grp { + fsl,pins = < + MX6QDL_PAD_EIM_D26__UART2_TX_DATA 0x1b0b1 + MX6QDL_PAD_EIM_D27__UART2_RX_DATA 0x1b0b1 + MX6QDL_PAD_EIM_D28__UART2_CTS_B 0x1b0b1 + MX6QDL_PAD_EIM_D29__UART2_RTS_B 0x1b0b1 + >; + }; + + pinctrl_usdhc2: usdhc2grp { + fsl,pins = < + MX6QDL_PAD_SD2_CMD__SD2_CMD 0x17059 + MX6QDL_PAD_SD2_CLK__SD2_CLK 0x10059 + MX6QDL_PAD_SD2_DAT0__SD2_DATA0 0x17059 + MX6QDL_PAD_SD2_DAT1__SD2_DATA1 0x17059 + MX6QDL_PAD_SD2_DAT2__SD2_DATA2 0x17059 + MX6QDL_PAD_SD2_DAT3__SD2_DATA3 0x17059 + >; + }; + + pinctrl_usdhc4: usdhc4grp { + fsl,pins = < + MX6QDL_PAD_SD4_CMD__SD4_CMD 0x17059 + MX6QDL_PAD_SD4_CLK__SD4_CLK 0x10059 + MX6QDL_PAD_SD4_DAT0__SD4_DATA0 0x17059 + MX6QDL_PAD_SD4_DAT1__SD4_DATA1 0x17059 + MX6QDL_PAD_SD4_DAT2__SD4_DATA2 0x17059 + MX6QDL_PAD_SD4_DAT3__SD4_DATA3 0x17059 + MX6QDL_PAD_SD4_DAT4__SD4_DATA4 0x17059 + MX6QDL_PAD_SD4_DAT5__SD4_DATA5 0x17059 + MX6QDL_PAD_SD4_DAT6__SD4_DATA6 0x17059 + MX6QDL_PAD_SD4_DAT7__SD4_DATA7 0x17059 + >; + }; +}; + +&pwm1 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_pwm1>; + status = "okay"; +}; + +&pwm2 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_pwm2>; + status = "okay"; +}; + +&ssi1 { + status = "okay"; +}; + +&uart1 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_uart1>; + status = "okay"; +}; + +&uart2 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_uart2>; + uart-has-rtscts; +}; + +&usbh1 { + status = "okay"; +}; + +&usdhc2 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_usdhc2>; + bus-width = <4>; + cd-gpios = <&gpio1 4 GPIO_ACTIVE_LOW>; + status = "okay"; +}; + +&usdhc4 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_usdhc4>; + bus-width = <8>; + non-removable; + no-1-8-v; + keep-power-in-suspend; + status = "okay"; +}; + +&wdog1 { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/imx6q-novena.dts b/arch/arm/boot/dts/imx6q-novena.dts index 52f39371188d..fcd824dc485b 100644 --- a/arch/arm/boot/dts/imx6q-novena.dts +++ b/arch/arm/boot/dts/imx6q-novena.dts @@ -268,8 +268,6 @@ touch: stmpe811@44 { compatible = "st,stmpe811"; reg = <0x44>; - #address-cells = <1>; - #size-cells = <0>; irq-gpio = <&gpio5 13 GPIO_ACTIVE_HIGH>; id = <0>; blocks = <0x5>; diff --git a/arch/arm/boot/dts/imx6q-pistachio.dts b/arch/arm/boot/dts/imx6q-pistachio.dts index bd57b3b74db7..a31e83cd07a3 100644 --- a/arch/arm/boot/dts/imx6q-pistachio.dts +++ b/arch/arm/boot/dts/imx6q-pistachio.dts @@ -614,7 +614,7 @@ &uart5 { pinctrl-names = "default"; pinctrl-0 = <&pinctrl_uart5>; - fsl,uart-has-rtscts; + uart-has-rtscts; status = "okay"; }; diff --git a/arch/arm/boot/dts/imx6q-sabreauto.dts b/arch/arm/boot/dts/imx6q-sabreauto.dts index 334b9247e78c..6e981a3e0a83 100644 --- a/arch/arm/boot/dts/imx6q-sabreauto.dts +++ b/arch/arm/boot/dts/imx6q-sabreauto.dts @@ -1,14 +1,7 @@ -/* - * Copyright 2012 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. /dts-v1/; diff --git a/arch/arm/boot/dts/imx6q-sabresd.dts b/arch/arm/boot/dts/imx6q-sabresd.dts index 527772b62fee..eec944673c0b 100644 --- a/arch/arm/boot/dts/imx6q-sabresd.dts +++ b/arch/arm/boot/dts/imx6q-sabresd.dts @@ -1,14 +1,7 @@ -/* - * Copyright 2012 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. /dts-v1/; diff --git a/arch/arm/boot/dts/imx6q-udoo.dts b/arch/arm/boot/dts/imx6q-udoo.dts index c3e64ff3d544..52e9f4a211d0 100644 --- a/arch/arm/boot/dts/imx6q-udoo.dts +++ b/arch/arm/boot/dts/imx6q-udoo.dts @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright 2013 Freescale Semiconductor, Inc. * * Author: Fabio Estevam - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * */ /dts-v1/; #include "imx6q.dtsi" diff --git a/arch/arm/boot/dts/imx6q-utilite-pro.dts b/arch/arm/boot/dts/imx6q-utilite-pro.dts index f5d9c34b0d39..d16ff2083d62 100644 --- a/arch/arm/boot/dts/imx6q-utilite-pro.dts +++ b/arch/arm/boot/dts/imx6q-utilite-pro.dts @@ -61,8 +61,6 @@ encoder { compatible = "ti,tfp410"; - #address-cells = <1>; - #size-cells = <0>; ports { #address-cells = <1>; diff --git a/arch/arm/boot/dts/imx6q-var-dt6customboard.dts b/arch/arm/boot/dts/imx6q-var-dt6customboard.dts index e0728d475f6f..f2368a073d07 100644 --- a/arch/arm/boot/dts/imx6q-var-dt6customboard.dts +++ b/arch/arm/boot/dts/imx6q-var-dt6customboard.dts @@ -26,8 +26,6 @@ gpio-keys { compatible = "gpio-keys"; - #address-cells = <1>; - #size-cells = <0>; autorepeat; back { diff --git a/arch/arm/boot/dts/imx6q-wandboard-revb1.dts b/arch/arm/boot/dts/imx6q-wandboard-revb1.dts index b763352cddae..be85b980bdfe 100644 --- a/arch/arm/boot/dts/imx6q-wandboard-revb1.dts +++ b/arch/arm/boot/dts/imx6q-wandboard-revb1.dts @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright 2013 Freescale Semiconductor, Inc. * * Author: Fabio Estevam - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * */ /dts-v1/; #include "imx6q.dtsi" diff --git a/arch/arm/boot/dts/imx6q-wandboard-revd1.dts b/arch/arm/boot/dts/imx6q-wandboard-revd1.dts index 8691fab21058..fcfba28764d4 100644 --- a/arch/arm/boot/dts/imx6q-wandboard-revd1.dts +++ b/arch/arm/boot/dts/imx6q-wandboard-revd1.dts @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright 2013 Freescale Semiconductor, Inc. * * Author: Fabio Estevam - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * */ /dts-v1/; #include "imx6q.dtsi" diff --git a/arch/arm/boot/dts/imx6q-wandboard.dts b/arch/arm/boot/dts/imx6q-wandboard.dts index 2a3d98c1489a..fa36fe183fc0 100644 --- a/arch/arm/boot/dts/imx6q-wandboard.dts +++ b/arch/arm/boot/dts/imx6q-wandboard.dts @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright 2013 Freescale Semiconductor, Inc. * * Author: Fabio Estevam - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * */ /dts-v1/; #include "imx6q.dtsi" diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi index ae7b3f107893..70483ce72ba6 100644 --- a/arch/arm/boot/dts/imx6q.dtsi +++ b/arch/arm/boot/dts/imx6q.dtsi @@ -1,12 +1,6 @@ - -/* - * Copyright 2013 Freescale Semiconductor, Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - */ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright 2013 Freescale Semiconductor, Inc. #include #include "imx6q-pinfunc.h" @@ -162,22 +156,27 @@ #size-cells = <0>; reg = <2>; - ipu2_di0_disp0: disp0-endpoint { + ipu2_di0_disp0: endpoint@0 { + reg = <0>; }; - ipu2_di0_hdmi: hdmi-endpoint { + ipu2_di0_hdmi: endpoint@1 { + reg = <1>; remote-endpoint = <&hdmi_mux_2>; }; - ipu2_di0_mipi: mipi-endpoint { + ipu2_di0_mipi: endpoint@2 { + reg = <2>; remote-endpoint = <&mipi_mux_2>; }; - ipu2_di0_lvds0: lvds0-endpoint { + ipu2_di0_lvds0: endpoint@3 { + reg = <3>; remote-endpoint = <&lvds0_mux_2>; }; - ipu2_di0_lvds1: lvds1-endpoint { + ipu2_di0_lvds1: endpoint@4 { + reg = <4>; remote-endpoint = <&lvds1_mux_2>; }; }; @@ -187,19 +186,23 @@ #size-cells = <0>; reg = <3>; - ipu2_di1_hdmi: hdmi-endpoint { + ipu2_di1_hdmi: endpoint@1 { + reg = <1>; remote-endpoint = <&hdmi_mux_3>; }; - ipu2_di1_mipi: mipi-endpoint { + ipu2_di1_mipi: endpoint@2 { + reg = <2>; remote-endpoint = <&mipi_mux_3>; }; - ipu2_di1_lvds0: lvds0-endpoint { + ipu2_di1_lvds0: endpoint@3 { + reg = <3>; remote-endpoint = <&lvds0_mux_3>; }; - ipu2_di1_lvds1: lvds1-endpoint { + ipu2_di1_lvds1: endpoint@4 { + reg = <4>; remote-endpoint = <&lvds1_mux_3>; }; }; diff --git a/arch/arm/boot/dts/imx6qdl-apalis.dtsi b/arch/arm/boot/dts/imx6qdl-apalis.dtsi index 8206683172d2..64fbee61de44 100644 --- a/arch/arm/boot/dts/imx6qdl-apalis.dtsi +++ b/arch/arm/boot/dts/imx6qdl-apalis.dtsi @@ -331,8 +331,6 @@ compatible = "st,stmpe811"; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_touch_int>; - #address-cells = <1>; - #size-cells = <0>; reg = <0x41>; interrupts = <10 IRQ_TYPE_LEVEL_LOW>; interrupt-parent = <&gpio4>; diff --git a/arch/arm/boot/dts/imx6qdl-colibri.dtsi b/arch/arm/boot/dts/imx6qdl-colibri.dtsi index e4eb300549d4..76035db96f67 100644 --- a/arch/arm/boot/dts/imx6qdl-colibri.dtsi +++ b/arch/arm/boot/dts/imx6qdl-colibri.dtsi @@ -262,8 +262,6 @@ compatible = "st,stmpe811"; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_touch_int>; - #address-cells = <1>; - #size-cells = <0>; reg = <0x41>; interrupts = <20 IRQ_TYPE_LEVEL_LOW>; interrupt-parent = <&gpio6>; diff --git a/arch/arm/boot/dts/imx6qdl-gw5904.dtsi b/arch/arm/boot/dts/imx6qdl-gw5904.dtsi index 58124adfd65b..3c52bdb453f3 100644 --- a/arch/arm/boot/dts/imx6qdl-gw5904.dtsi +++ b/arch/arm/boot/dts/imx6qdl-gw5904.dtsi @@ -162,8 +162,6 @@ switch@0 { compatible = "marvell,mv88e6085"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; ports { diff --git a/arch/arm/boot/dts/imx6qdl-hummingboard.dtsi b/arch/arm/boot/dts/imx6qdl-hummingboard.dtsi index 7e20b47de839..0e64016e765f 100644 --- a/arch/arm/boot/dts/imx6qdl-hummingboard.dtsi +++ b/arch/arm/boot/dts/imx6qdl-hummingboard.dtsi @@ -38,6 +38,7 @@ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR * OTHER DEALINGS IN THE SOFTWARE. */ +#include / { /* Will be filled by the bootloader */ @@ -110,17 +111,27 @@ vin-supply = <&v_5v0>; }; - sound-sgtl5000 { - audio-codec = <&sgtl5000>; - audio-routing = - "MIC_IN", "Mic Jack", - "Mic Jack", "Mic Bias", + audio: sound-sgtl5000 { + compatible = "simple-audio-card"; + simple-audio-card,name = "On-board Codec"; + simple-audio-card,format = "i2s"; + simple-audio-card,bitclock-master = <&sound_codec>; + simple-audio-card,frame-master = <&sound_codec>; + simple-audio-card,widgets = + "Microphone", "Headphone Jack", + "Headphone", "Headphone Jack"; + simple-audio-card,routing = + "MIC_IN", "Headphone Jack", + "Headphone Jack", "Mic Bias", "Headphone Jack", "HP_OUT"; - compatible = "fsl,imx-audio-sgtl5000"; - model = "On-board Codec"; - mux-ext-port = <5>; - mux-int-port = <1>; - ssi-controller = <&ssi1>; + + sound_cpu: simple-audio-card,cpu { + sound-dai = <&ssi1>; + }; + + sound_codec: simple-audio-card,codec { + sound-dai = <&sgtl5000>; + }; }; sound-spdif { @@ -134,6 +145,26 @@ &audmux { status = "okay"; + + ssi1 { + fsl,audmux-port = <0>; + fsl,port-config = < + (IMX_AUDMUX_V2_PTCR_SYN | + IMX_AUDMUX_V2_PTCR_TFSEL(4) | + IMX_AUDMUX_V2_PTCR_TCSEL(4) | + IMX_AUDMUX_V2_PTCR_TFSDIR | + IMX_AUDMUX_V2_PTCR_TCLKDIR) + IMX_AUDMUX_V2_PDCR_RXDSEL(4) + >; + }; + + pins5 { + fsl,audmux-port = <4>; + fsl,port-config = < + IMX_AUDMUX_V2_PTCR_SYN + IMX_AUDMUX_V2_PDCR_RXDSEL(0) + >; + }; }; &can1 { @@ -166,6 +197,7 @@ compatible = "fsl,sgtl5000"; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_hummingboard_sgtl5000>; + #sound-dai-cells = <0>; reg = <0x0a>; VDDA-supply = <&v_3v2>; VDDIO-supply = <&v_3v2>; diff --git a/arch/arm/boot/dts/imx6qdl-hummingboard2.dtsi b/arch/arm/boot/dts/imx6qdl-hummingboard2.dtsi index 98241acb08a6..c413f9c3540f 100644 --- a/arch/arm/boot/dts/imx6qdl-hummingboard2.dtsi +++ b/arch/arm/boot/dts/imx6qdl-hummingboard2.dtsi @@ -38,6 +38,7 @@ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR * OTHER DEALINGS IN THE SOFTWARE. */ +#include / { /* Will be filled by the bootloader */ @@ -150,22 +151,52 @@ vin-supply = <&v_5v0>; }; - sound-sgtl5000 { - audio-codec = <&sgtl5000>; - audio-routing = + audio: sound-sgtl5000 { + compatible = "simple-audio-card"; + simple-audio-card,name = "On-board Codec"; + simple-audio-card,format = "i2s"; + simple-audio-card,bitclock-master = <&sound_codec>; + simple-audio-card,frame-master = <&sound_codec>; + simple-audio-card,widgets = + "Microphone", "Mic Jack", + "Headphone", "Headphone Jack"; + simple-audio-card,routing = "MIC_IN", "Mic Jack", "Mic Jack", "Mic Bias", "Headphone Jack", "HP_OUT"; - compatible = "fsl,imx-audio-sgtl5000"; - model = "On-board Codec"; - mux-ext-port = <5>; - mux-int-port = <1>; - ssi-controller = <&ssi1>; + + sound_cpu: simple-audio-card,cpu { + sound-dai = <&ssi1>; + }; + + sound_codec: simple-audio-card,codec { + sound-dai = <&sgtl5000>; + }; }; }; &audmux { status = "okay"; + + ssi1 { + fsl,audmux-port = <0>; + fsl,port-config = < + (IMX_AUDMUX_V2_PTCR_SYN | + IMX_AUDMUX_V2_PTCR_TFSEL(4) | + IMX_AUDMUX_V2_PTCR_TCSEL(4) | + IMX_AUDMUX_V2_PTCR_TFSDIR | + IMX_AUDMUX_V2_PTCR_TCLKDIR) + IMX_AUDMUX_V2_PDCR_RXDSEL(4) + >; + }; + + pins5 { + fsl,audmux-port = <4>; + fsl,port-config = < + IMX_AUDMUX_V2_PTCR_SYN + IMX_AUDMUX_V2_PDCR_RXDSEL(0) + >; + }; }; &ecspi2 { diff --git a/arch/arm/boot/dts/imx6qdl-icore.dtsi b/arch/arm/boot/dts/imx6qdl-icore.dtsi index b3a463a5908b..0a1574998fc6 100644 --- a/arch/arm/boot/dts/imx6qdl-icore.dtsi +++ b/arch/arm/boot/dts/imx6qdl-icore.dtsi @@ -49,7 +49,7 @@ reg = <0x10000000 0x80000000>; }; - backlight { + backlight_lvds: backlight-lvds { compatible = "pwm-backlight"; pwms = <&pwm3 0 100000>; brightness-levels = <0 4 8 16 32 64 128 255>; @@ -265,6 +265,14 @@ status = "okay"; }; +&usdhc3 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_usdhc3>; + no-1-8-v; + non-removable; + status = "disabled"; +}; + &iomuxc { pinctrl_audmux: audmux { fsl,pins = < @@ -378,4 +386,19 @@ MX6QDL_PAD_SD1_DAT3__SD1_DATA3 0x17070 >; }; + + pinctrl_usdhc3: usdhc3grp { + fsl,pins = < + MX6QDL_PAD_SD3_CMD__SD3_CMD 0x17059 + MX6QDL_PAD_SD3_CLK__SD3_CLK 0x10059 + MX6QDL_PAD_SD3_DAT0__SD3_DATA0 0x17059 + MX6QDL_PAD_SD3_DAT1__SD3_DATA1 0x17059 + MX6QDL_PAD_SD3_DAT2__SD3_DATA2 0x17059 + MX6QDL_PAD_SD3_DAT3__SD3_DATA3 0x17059 + MX6QDL_PAD_SD3_DAT4__SD3_DATA4 0x17059 + MX6QDL_PAD_SD3_DAT5__SD3_DATA5 0x17059 + MX6QDL_PAD_SD3_DAT6__SD3_DATA6 0x17059 + MX6QDL_PAD_SD3_DAT7__SD3_DATA7 0x17059 + >; + }; }; diff --git a/arch/arm/boot/dts/imx6qdl-phytec-pfla02.dtsi b/arch/arm/boot/dts/imx6qdl-phytec-pfla02.dtsi index c58f3443d55d..ed1aafd56973 100644 --- a/arch/arm/boot/dts/imx6qdl-phytec-pfla02.dtsi +++ b/arch/arm/boot/dts/imx6qdl-phytec-pfla02.dtsi @@ -115,7 +115,7 @@ compatible = "dlg,da9063"; reg = <0x58>; interrupt-parent = <&gpio2>; - interrupts = <9 0x8>; /* active-low GPIO2_9 */ + interrupts = <9 IRQ_TYPE_LEVEL_LOW>; /* active-low GPIO2_9 */ regulators { vddcore_reg: bcore1 { diff --git a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi index 54b0139e978d..0e28e36ddbb2 100644 --- a/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi +++ b/arch/arm/boot/dts/imx6qdl-sabreauto.dtsi @@ -1,14 +1,7 @@ -/* - * Copyright 2012 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. #include diff --git a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi index 18b65052553d..654cf2c9b073 100644 --- a/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +++ b/arch/arm/boot/dts/imx6qdl-sabrelite.dtsi @@ -379,9 +379,6 @@ powerdown-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */ port { - #address-cells = <1>; - #size-cells = <0>; - ov5640_to_mipi_csi2: endpoint { remote-endpoint = <&mipi_csi2_in>; clock-lanes = <0>; diff --git a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi index f019f9900369..15744ad52535 100644 --- a/arch/arm/boot/dts/imx6qdl-sabresd.dtsi +++ b/arch/arm/boot/dts/imx6qdl-sabresd.dtsi @@ -1,14 +1,7 @@ -/* - * Copyright 2012 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2012 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. #include #include @@ -294,9 +287,6 @@ reset-gpios = <&gpio1 20 GPIO_ACTIVE_LOW>; port { - #address-cells = <1>; - #size-cells = <0>; - ov5640_to_mipi_csi2: endpoint { remote-endpoint = <&mipi_csi2_in>; clock-lanes = <0>; diff --git a/arch/arm/boot/dts/imx6qdl-tx6-lcd.dtsi b/arch/arm/boot/dts/imx6qdl-tx6-lcd.dtsi index 5102fc47380b..79f2354886b7 100644 --- a/arch/arm/boot/dts/imx6qdl-tx6-lcd.dtsi +++ b/arch/arm/boot/dts/imx6qdl-tx6-lcd.dtsi @@ -77,7 +77,6 @@ enable-gpios = <&gpio3 29 GPIO_ACTIVE_HIGH>; power-supply = <®_3v3>; backlight = <&backlight>; - bus-format-override = "rgb24"; port { lcd_panel_in: endpoint { diff --git a/arch/arm/boot/dts/imx6qdl-tx6-mb7.dtsi b/arch/arm/boot/dts/imx6qdl-tx6-mb7.dtsi index 4c4e2e1a931f..410972e1dca9 100644 --- a/arch/arm/boot/dts/imx6qdl-tx6-mb7.dtsi +++ b/arch/arm/boot/dts/imx6qdl-tx6-mb7.dtsi @@ -54,19 +54,16 @@ lcd-panel { compatible = "edt,et057090dhu"; - bus-format-override = "rgb24"; pixelclk-active = <0>; }; lvds0-panel { compatible = "edt,etml1010g0dka"; - bus-format-override = "spwg-18"; pixelclk-active = <0>; }; lvds1-panel { compatible = "edt,etml1010g0dka"; - bus-format-override = "spwg-18"; pixelclk-active = <0>; }; }; diff --git a/arch/arm/boot/dts/imx6qdl-tx6.dtsi b/arch/arm/boot/dts/imx6qdl-tx6.dtsi index f015e2d1cf35..a98fb2564c63 100644 --- a/arch/arm/boot/dts/imx6qdl-tx6.dtsi +++ b/arch/arm/boot/dts/imx6qdl-tx6.dtsi @@ -50,11 +50,11 @@ can0 = &can2; can1 = &can1; ethernet0 = &fec; - lcdif_23bit_pins_a = &pinctrl_disp0_1; - lcdif_24bit_pins_a = &pinctrl_disp0_2; + lcdif-23bit-pins-a = &pinctrl_disp0_1; + lcdif-24bit-pins-a = &pinctrl_disp0_2; pwm0 = &pwm1; pwm1 = &pwm2; - reg_can_xcvr = ®_can_xcvr; + reg-can-xcvr = ®_can_xcvr; stk5led = &user_led; usbotg = &usbotg; sdhc0 = &usdhc1; diff --git a/arch/arm/boot/dts/imx6qdl-udoo.dtsi b/arch/arm/boot/dts/imx6qdl-udoo.dtsi index 906387915dc5..4f27861bbb32 100644 --- a/arch/arm/boot/dts/imx6qdl-udoo.dtsi +++ b/arch/arm/boot/dts/imx6qdl-udoo.dtsi @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright 2013 Freescale Semiconductor, Inc. * * Author: Fabio Estevam - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * */ / { diff --git a/arch/arm/boot/dts/imx6qdl-wandboard-revb1.dtsi b/arch/arm/boot/dts/imx6qdl-wandboard-revb1.dtsi index a32089132263..855dc6f9df75 100644 --- a/arch/arm/boot/dts/imx6qdl-wandboard-revb1.dtsi +++ b/arch/arm/boot/dts/imx6qdl-wandboard-revb1.dtsi @@ -17,7 +17,6 @@ imx6qdl-wandboard { pinctrl_hog: hoggrp { fsl,pins = < - MX6QDL_PAD_GPIO_0__CCM_CLKO1 0x130b0 /* GPIO_0_CLKO */ MX6QDL_PAD_GPIO_2__GPIO1_IO02 0x80000000 /* uSDHC1 CD */ MX6QDL_PAD_EIM_DA9__GPIO3_IO09 0x80000000 /* uSDHC3 CD */ MX6QDL_PAD_EIM_EB1__GPIO2_IO29 0x0f0b0 /* WL_REF_ON */ diff --git a/arch/arm/boot/dts/imx6qdl-wandboard-revc1.dtsi b/arch/arm/boot/dts/imx6qdl-wandboard-revc1.dtsi index 8d893a78cdf0..49a0a557e62e 100644 --- a/arch/arm/boot/dts/imx6qdl-wandboard-revc1.dtsi +++ b/arch/arm/boot/dts/imx6qdl-wandboard-revc1.dtsi @@ -17,7 +17,6 @@ imx6qdl-wandboard { pinctrl_hog: hoggrp { fsl,pins = < - MX6QDL_PAD_GPIO_0__CCM_CLKO1 0x130b0 /* GPIO_0_CLKO */ MX6QDL_PAD_GPIO_2__GPIO1_IO02 0x80000000 /* uSDHC1 CD */ MX6QDL_PAD_EIM_DA9__GPIO3_IO09 0x80000000 /* uSDHC3 CD */ MX6QDL_PAD_CSI0_DAT14__GPIO6_IO00 0x0f0b0 /* WIFI_ON (reset, active low) */ diff --git a/arch/arm/boot/dts/imx6qdl-wandboard-revd1.dtsi b/arch/arm/boot/dts/imx6qdl-wandboard-revd1.dtsi index 3a8a4952d45e..69d9c8661439 100644 --- a/arch/arm/boot/dts/imx6qdl-wandboard-revd1.dtsi +++ b/arch/arm/boot/dts/imx6qdl-wandboard-revd1.dtsi @@ -147,7 +147,6 @@ imx6qdl-wandboard { pinctrl_hog: hoggrp { fsl,pins = < - MX6QDL_PAD_GPIO_0__CCM_CLKO1 0x130b0 MX6QDL_PAD_EIM_D22__USB_OTG_PWR 0x80000000 /* USB Power Enable */ MX6QDL_PAD_GPIO_2__GPIO1_IO02 0x80000000 /* USDHC1 CD */ MX6QDL_PAD_EIM_DA9__GPIO3_IO09 0x80000000 /* uSDHC3 CD */ diff --git a/arch/arm/boot/dts/imx6qdl-wandboard.dtsi b/arch/arm/boot/dts/imx6qdl-wandboard.dtsi index ed96d7b5feab..e1afa54404d0 100644 --- a/arch/arm/boot/dts/imx6qdl-wandboard.dtsi +++ b/arch/arm/boot/dts/imx6qdl-wandboard.dtsi @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright 2013 Freescale Semiconductor, Inc. * * Author: Fabio Estevam - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * */ #include @@ -83,6 +79,8 @@ status = "okay"; codec: sgtl5000@a { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_mclk>; compatible = "fsl,sgtl5000"; reg = <0x0a>; clocks = <&clks IMX6QDL_CLK_CKO>; @@ -142,6 +140,12 @@ >; }; + pinctrl_mclk: mclkgrp { + fsl,pins = < + MX6QDL_PAD_GPIO_0__CCM_CLKO1 0x130b0 + >; + }; + pinctrl_spdif: spdifgrp { fsl,pins = < MX6QDL_PAD_ENET_RXD0__SPDIF_OUT 0x1b0b0 diff --git a/arch/arm/boot/dts/imx6qdl-zii-rdu2.dtsi b/arch/arm/boot/dts/imx6qdl-zii-rdu2.dtsi index 911f7f0e3cea..19a075aee19e 100644 --- a/arch/arm/boot/dts/imx6qdl-zii-rdu2.dtsi +++ b/arch/arm/boot/dts/imx6qdl-zii-rdu2.dtsi @@ -263,6 +263,17 @@ }; }; +&cpu0 { + fsl,soc-operating-points = < + /* ARM kHz SOC-PU uV */ + 1200000 1300000 + 996000 1275000 + 852000 1275000 + 792000 1200000 + 396000 1200000 + >; +}; + ®_arm { vin-supply = <&sw1a_reg>; }; @@ -571,6 +582,17 @@ }; }; + touchscreen@2a { + compatible = "eeti,egalax_ts"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_ts>; + reg = <0x2a>; + interrupt-parent = <&gpio1>; + interrupts = <8 IRQ_TYPE_LEVEL_LOW>; + wakeup-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>; + status = "disabled"; + }; + hpa1: amp@60 { compatible = "ti,tpa6130a2"; pinctrl-names = "default"; @@ -666,8 +688,6 @@ compatible = "marvell,mv88e6085"; pinctrl-0 = <&pinctrl_switch_irq>; pinctrl-names = "default"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; dsa,member = <0 0>; eeprom-length = <512>; diff --git a/arch/arm/boot/dts/imx6qdl.dtsi b/arch/arm/boot/dts/imx6qdl.dtsi index c003e62bf290..911141e24681 100644 --- a/arch/arm/boot/dts/imx6qdl.dtsi +++ b/arch/arm/boot/dts/imx6qdl.dtsi @@ -1,14 +1,7 @@ -/* - * Copyright 2011 Freescale Semiconductor, Inc. - * Copyright 2011 Linaro Ltd. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2011 Freescale Semiconductor, Inc. +// Copyright 2011 Linaro Ltd. #include #include @@ -58,9 +51,6 @@ }; clocks { - #address-cells = <1>; - #size-cells = <0>; - ckil { compatible = "fsl,imx-ckil", "fixed-clock"; #clock-cells = <0>; @@ -695,11 +685,8 @@ interrupts = <0 49 IRQ_TYPE_LEVEL_HIGH>, <0 54 IRQ_TYPE_LEVEL_HIGH>, <0 127 IRQ_TYPE_LEVEL_HIGH>; - #address-cells = <1>; - #size-cells = <0>; - regulator-1p1@20c8110 { - reg = <0x20c8110>; + regulator-1p1 { compatible = "fsl,anatop-regulator"; regulator-name = "vdd1p1"; regulator-min-microvolt = <1000000>; @@ -714,8 +701,7 @@ anatop-enable-bit = <0>; }; - regulator-3p0@20c8120 { - reg = <0x20c8120>; + regulator-3p0 { compatible = "fsl,anatop-regulator"; regulator-name = "vdd3p0"; regulator-min-microvolt = <2800000>; @@ -730,8 +716,7 @@ anatop-enable-bit = <0>; }; - regulator-2p5@20c8130 { - reg = <0x20c8130>; + regulator-2p5 { compatible = "fsl,anatop-regulator"; regulator-name = "vdd2p5"; regulator-min-microvolt = <2250000>; @@ -746,8 +731,7 @@ anatop-enable-bit = <0>; }; - reg_arm: regulator-vddcore@20c8140 { - reg = <0x20c8140>; + reg_arm: regulator-vddcore { compatible = "fsl,anatop-regulator"; regulator-name = "vddarm"; regulator-min-microvolt = <725000>; @@ -764,8 +748,7 @@ anatop-max-voltage = <1450000>; }; - reg_pu: regulator-vddpu@20c8140 { - reg = <0x20c8140>; + reg_pu: regulator-vddpu { compatible = "fsl,anatop-regulator"; regulator-name = "vddpu"; regulator-min-microvolt = <725000>; @@ -782,8 +765,7 @@ anatop-max-voltage = <1450000>; }; - reg_soc: regulator-vddsoc@20c8140 { - reg = <0x20c8140>; + reg_soc: regulator-vddsoc { compatible = "fsl,anatop-regulator"; regulator-name = "vddsoc"; regulator-min-microvolt = <725000>; @@ -1187,8 +1169,6 @@ }; mipi_dsi: mipi@21e0000 { - #address-cells = <1>; - #size-cells = <0>; reg = <0x021e0000 0x4000>; status = "disabled"; @@ -1300,22 +1280,27 @@ #size-cells = <0>; reg = <2>; - ipu1_di0_disp0: disp0-endpoint { + ipu1_di0_disp0: endpoint@0 { + reg = <0>; }; - ipu1_di0_hdmi: hdmi-endpoint { + ipu1_di0_hdmi: endpoint@1 { + reg = <1>; remote-endpoint = <&hdmi_mux_0>; }; - ipu1_di0_mipi: mipi-endpoint { + ipu1_di0_mipi: endpoint@2 { + reg = <2>; remote-endpoint = <&mipi_mux_0>; }; - ipu1_di0_lvds0: lvds0-endpoint { + ipu1_di0_lvds0: endpoint@3 { + reg = <3>; remote-endpoint = <&lvds0_mux_0>; }; - ipu1_di0_lvds1: lvds1-endpoint { + ipu1_di0_lvds1: endpoint@4 { + reg = <4>; remote-endpoint = <&lvds1_mux_0>; }; }; @@ -1325,22 +1310,27 @@ #size-cells = <0>; reg = <3>; - ipu1_di1_disp1: disp1-endpoint { + ipu1_di1_disp1: endpoint@0 { + reg = <0>; }; - ipu1_di1_hdmi: hdmi-endpoint { + ipu1_di1_hdmi: endpoint@1 { + reg = <1>; remote-endpoint = <&hdmi_mux_1>; }; - ipu1_di1_mipi: mipi-endpoint { + ipu1_di1_mipi: endpoint@2 { + reg = <2>; remote-endpoint = <&mipi_mux_1>; }; - ipu1_di1_lvds0: lvds0-endpoint { + ipu1_di1_lvds0: endpoint@3 { + reg = <3>; remote-endpoint = <&lvds0_mux_1>; }; - ipu1_di1_lvds1: lvds1-endpoint { + ipu1_di1_lvds1: endpoint@4 { + reg = <4>; remote-endpoint = <&lvds1_mux_1>; }; }; diff --git a/arch/arm/boot/dts/imx6qp-sabreauto.dts b/arch/arm/boot/dts/imx6qp-sabreauto.dts index 5ce3840d83d3..d4caeeb0af70 100644 --- a/arch/arm/boot/dts/imx6qp-sabreauto.dts +++ b/arch/arm/boot/dts/imx6qp-sabreauto.dts @@ -1,44 +1,6 @@ -/* - * Copyright 2016 Freescale Semiconductor, Inc. - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation; either version 2 of the - * License, or (at your option) any later version. - * - * This file is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - */ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +// +// Copyright 2016 Freescale Semiconductor, Inc. /dts-v1/; diff --git a/arch/arm/boot/dts/imx6qp-sabresd.dts b/arch/arm/boot/dts/imx6qp-sabresd.dts index a8a5004dd9c8..f1b9cb104fdd 100644 --- a/arch/arm/boot/dts/imx6qp-sabresd.dts +++ b/arch/arm/boot/dts/imx6qp-sabresd.dts @@ -1,44 +1,6 @@ -/* - * Copyright 2016 Freescale Semiconductor, Inc. - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation; either version 2 of the - * License, or (at your option) any later version. - * - * This file is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - */ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +// +// Copyright 2016 Freescale Semiconductor, Inc. /dts-v1/; diff --git a/arch/arm/boot/dts/imx6qp-wandboard-revd1.dts b/arch/arm/boot/dts/imx6qp-wandboard-revd1.dts index 907ba0c74ba6..bcca5ac5fa51 100644 --- a/arch/arm/boot/dts/imx6qp-wandboard-revd1.dts +++ b/arch/arm/boot/dts/imx6qp-wandboard-revd1.dts @@ -1,12 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright 2013 Freescale Semiconductor, Inc. * * Author: Fabio Estevam - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * */ /dts-v1/; #include "imx6qp.dtsi" diff --git a/arch/arm/boot/dts/imx6qp-zii-rdu2.dts b/arch/arm/boot/dts/imx6qp-zii-rdu2.dts index de5b50df833c..8c293e9f36a7 100644 --- a/arch/arm/boot/dts/imx6qp-zii-rdu2.dts +++ b/arch/arm/boot/dts/imx6qp-zii-rdu2.dts @@ -53,3 +53,8 @@ reg = <0x10000000 0>; }; }; + +&gpu_3d { + assigned-clocks = <&clks IMX6QDL_CLK_GPU3D_SHADER_SEL>; + assigned-clock-parents = <&clks IMX6QDL_CLK_PLL2_PFD1_594M>; +}; diff --git a/arch/arm/boot/dts/imx6qp.dtsi b/arch/arm/boot/dts/imx6qp.dtsi index 5f4fdce715c1..5f51f8e5c1fa 100644 --- a/arch/arm/boot/dts/imx6qp.dtsi +++ b/arch/arm/boot/dts/imx6qp.dtsi @@ -1,44 +1,6 @@ -/* - * Copyright 2016 Freescale Semiconductor, Inc. - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation; either version 2 of the - * License, or (at your option) any later version. - * - * This file is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - */ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +// +// Copyright 2016 Freescale Semiconductor, Inc. #include "imx6q.dtsi" diff --git a/arch/arm/boot/dts/imx6sl-evk.dts b/arch/arm/boot/dts/imx6sl-evk.dts index 37e792fdc160..92ad01f676e3 100644 --- a/arch/arm/boot/dts/imx6sl-evk.dts +++ b/arch/arm/boot/dts/imx6sl-evk.dts @@ -1,10 +1,6 @@ -/* - * Copyright (C) 2013 Freescale Semiconductor, Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ +// SPDX-License-Identifier: GPL-2.0 +// +//Copyright (C) 2013 Freescale Semiconductor, Inc. /dts-v1/; diff --git a/arch/arm/boot/dts/imx6sl.dtsi b/arch/arm/boot/dts/imx6sl.dtsi index ab6a7e2e7e8f..994e48dc1df0 100644 --- a/arch/arm/boot/dts/imx6sl.dtsi +++ b/arch/arm/boot/dts/imx6sl.dtsi @@ -1,11 +1,6 @@ -/* - * Copyright 2013 Freescale Semiconductor, Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - */ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright 2013 Freescale Semiconductor, Inc. #include #include "imx6sl-pinfunc.h" @@ -86,9 +81,6 @@ }; clocks { - #address-cells = <1>; - #size-cells = <0>; - ckil { compatible = "fixed-clock"; #clock-cells = <0>; @@ -527,11 +519,8 @@ interrupts = <0 49 IRQ_TYPE_LEVEL_HIGH>, <0 54 IRQ_TYPE_LEVEL_HIGH>, <0 127 IRQ_TYPE_LEVEL_HIGH>; - #address-cells = <1>; - #size-cells = <0>; - regulator-1p1@20c8110 { - reg = <0x20c8110>; + regulator-1p1 { compatible = "fsl,anatop-regulator"; regulator-name = "vdd1p1"; regulator-min-microvolt = <800000>; @@ -546,8 +535,7 @@ anatop-enable-bit = <0>; }; - regulator-3p0@20c8120 { - reg = <0x20c8120>; + regulator-3p0 { compatible = "fsl,anatop-regulator"; regulator-name = "vdd3p0"; regulator-min-microvolt = <2800000>; @@ -562,8 +550,7 @@ anatop-enable-bit = <0>; }; - regulator-2p5@20c8130 { - reg = <0x20c8130>; + regulator-2p5 { compatible = "fsl,anatop-regulator"; regulator-name = "vdd2p5"; regulator-min-microvolt = <2100000>; @@ -578,8 +565,7 @@ anatop-enable-bit = <0>; }; - reg_arm: regulator-vddcore@20c8140 { - reg = <0x20c8140>; + reg_arm: regulator-vddcore { compatible = "fsl,anatop-regulator"; regulator-name = "vddarm"; regulator-min-microvolt = <725000>; @@ -596,8 +582,7 @@ anatop-max-voltage = <1450000>; }; - reg_pu: regulator-vddpu@20c8140 { - reg = <0x20c8140>; + reg_pu: regulator-vddpu { compatible = "fsl,anatop-regulator"; regulator-name = "vddpu"; regulator-min-microvolt = <725000>; @@ -614,8 +599,7 @@ anatop-max-voltage = <1450000>; }; - reg_soc: regulator-vddsoc@20c8140 { - reg = <0x20c8140>; + reg_soc: regulator-vddsoc { compatible = "fsl,anatop-regulator"; regulator-name = "vddsoc"; regulator-min-microvolt = <725000>; diff --git a/arch/arm/boot/dts/imx6sx-nitrogen6sx.dts b/arch/arm/boot/dts/imx6sx-nitrogen6sx.dts index b58f770c40d9..59e52f504922 100644 --- a/arch/arm/boot/dts/imx6sx-nitrogen6sx.dts +++ b/arch/arm/boot/dts/imx6sx-nitrogen6sx.dts @@ -48,8 +48,8 @@ compatible = "boundary,imx6sx-nitrogen6sx", "fsl,imx6sx"; aliases { - fb_lcd = &lcdif1; - t_lcd = &t_lcd; + fb-lcd = &lcdif1; + t-lcd = &t_lcd; }; memory@80000000 { diff --git a/arch/arm/boot/dts/imx6sx-sabreauto.dts b/arch/arm/boot/dts/imx6sx-sabreauto.dts index 72da5acf35a2..841a27f3198f 100644 --- a/arch/arm/boot/dts/imx6sx-sabreauto.dts +++ b/arch/arm/boot/dts/imx6sx-sabreauto.dts @@ -1,10 +1,6 @@ -/* - * Copyright (C) 2014 Freescale Semiconductor, Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright (C) 2014 Freescale Semiconductor, Inc. /dts-v1/; @@ -18,25 +14,67 @@ reg = <0x80000000 0x80000000>; }; - regulators { - compatible = "simple-bus"; + leds { + compatible = "gpio-leds"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_led>; + + user { + label = "debug"; + gpios = <&gpio1 24 GPIO_ACTIVE_HIGH>; + linux,default-trigger = "heartbeat"; + }; + }; + + vcc_sd3: regulator-vcc-sd3 { + compatible = "regulator-fixed"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_vcc_sd3>; + regulator-name = "VCC_SD3"; + regulator-min-microvolt = <3000000>; + regulator-max-microvolt = <3000000>; + gpio = <&gpio2 11 GPIO_ACTIVE_HIGH>; + enable-active-high; + }; +}; + +&anaclk2 { + clock-frequency = <24576000>; +}; + +&fec1 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_enet1>; + phy-mode = "rgmii"; + phy-handle = <ðphy1>; + fsl,magic-packet; + status = "okay"; + + mdio { #address-cells = <1>; #size-cells = <0>; - vcc_sd3: regulator@0 { - compatible = "regulator-fixed"; + ethphy0: ethernet-phy@0 { + compatible = "ethernet-phy-ieee802.3-c22"; reg = <0>; - pinctrl-names = "default"; - pinctrl-0 = <&pinctrl_vcc_sd3>; - regulator-name = "VCC_SD3"; - regulator-min-microvolt = <3000000>; - regulator-max-microvolt = <3000000>; - gpio = <&gpio2 11 GPIO_ACTIVE_HIGH>; - enable-active-high; + }; + + ethphy1: ethernet-phy@1 { + compatible = "ethernet-phy-ieee802.3-c22"; + reg = <1>; }; }; }; +&fec2 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_enet2>; + phy-mode = "rgmii"; + phy-handle = <ðphy0>; + fsl,magic-packet; + status = "okay"; +}; + &uart1 { pinctrl-names = "default"; pinctrl-0 = <&pinctrl_uart1>; @@ -69,78 +107,297 @@ }; &iomuxc { - imx6x-sabreauto { - pinctrl_uart1: uart1grp { - fsl,pins = < - MX6SX_PAD_GPIO1_IO04__UART1_TX 0x1b0b1 - MX6SX_PAD_GPIO1_IO05__UART1_RX 0x1b0b1 - >; - }; + pinctrl_egalax_int: egalax-intgrp { + fsl,pins = < + MX6SX_PAD_SD4_RESET_B__GPIO6_IO_22 0x10b0 + >; + }; - pinctrl_usdhc3: usdhc3grp { - fsl,pins = < - MX6SX_PAD_SD3_CMD__USDHC3_CMD 0x17059 - MX6SX_PAD_SD3_CLK__USDHC3_CLK 0x10059 - MX6SX_PAD_SD3_DATA0__USDHC3_DATA0 0x17059 - MX6SX_PAD_SD3_DATA1__USDHC3_DATA1 0x17059 - MX6SX_PAD_SD3_DATA2__USDHC3_DATA2 0x17059 - MX6SX_PAD_SD3_DATA3__USDHC3_DATA3 0x17059 - MX6SX_PAD_SD3_DATA4__USDHC3_DATA4 0x17059 - MX6SX_PAD_SD3_DATA5__USDHC3_DATA5 0x17059 - MX6SX_PAD_SD3_DATA6__USDHC3_DATA6 0x17059 - MX6SX_PAD_SD3_DATA7__USDHC3_DATA7 0x17059 - MX6SX_PAD_KEY_COL0__GPIO2_IO_10 0x17059 /* CD */ - MX6SX_PAD_KEY_ROW0__GPIO2_IO_15 0x17059 /* WP */ - >; - }; + pinctrl_enet1: enet1grp { + fsl,pins = < + MX6SX_PAD_ENET1_MDIO__ENET1_MDIO 0xa0b1 + MX6SX_PAD_ENET1_MDC__ENET1_MDC 0xa0b1 + MX6SX_PAD_RGMII1_TXC__ENET1_RGMII_TXC 0xa0b9 + MX6SX_PAD_RGMII1_TD0__ENET1_TX_DATA_0 0xa0b1 + MX6SX_PAD_RGMII1_TD1__ENET1_TX_DATA_1 0xa0b1 + MX6SX_PAD_RGMII1_TD2__ENET1_TX_DATA_2 0xa0b1 + MX6SX_PAD_RGMII1_TD3__ENET1_TX_DATA_3 0xa0b1 + MX6SX_PAD_RGMII1_TX_CTL__ENET1_TX_EN 0xa0b1 + MX6SX_PAD_RGMII1_RXC__ENET1_RX_CLK 0x3081 + MX6SX_PAD_RGMII1_RD0__ENET1_RX_DATA_0 0x3081 + MX6SX_PAD_RGMII1_RD1__ENET1_RX_DATA_1 0x3081 + MX6SX_PAD_RGMII1_RD2__ENET1_RX_DATA_2 0x3081 + MX6SX_PAD_RGMII1_RD3__ENET1_RX_DATA_3 0x3081 + MX6SX_PAD_RGMII1_RX_CTL__ENET1_RX_EN 0x3081 + >; + }; - pinctrl_usdhc3_100mhz: usdhc3grp-100mhz { - fsl,pins = < - MX6SX_PAD_SD3_CMD__USDHC3_CMD 0x170b9 - MX6SX_PAD_SD3_CLK__USDHC3_CLK 0x100b9 - MX6SX_PAD_SD3_DATA0__USDHC3_DATA0 0x170b9 - MX6SX_PAD_SD3_DATA1__USDHC3_DATA1 0x170b9 - MX6SX_PAD_SD3_DATA2__USDHC3_DATA2 0x170b9 - MX6SX_PAD_SD3_DATA3__USDHC3_DATA3 0x170b9 - MX6SX_PAD_SD3_DATA4__USDHC3_DATA4 0x170b9 - MX6SX_PAD_SD3_DATA5__USDHC3_DATA5 0x170b9 - MX6SX_PAD_SD3_DATA6__USDHC3_DATA6 0x170b9 - MX6SX_PAD_SD3_DATA7__USDHC3_DATA7 0x170b9 - >; - }; + pinctrl_enet2: enet2grp { + fsl,pins = < + MX6SX_PAD_RGMII2_TXC__ENET2_RGMII_TXC 0xa0b9 + MX6SX_PAD_RGMII2_TD0__ENET2_TX_DATA_0 0xa0b1 + MX6SX_PAD_RGMII2_TD1__ENET2_TX_DATA_1 0xa0b1 + MX6SX_PAD_RGMII2_TD2__ENET2_TX_DATA_2 0xa0b1 + MX6SX_PAD_RGMII2_TD3__ENET2_TX_DATA_3 0xa0b1 + MX6SX_PAD_RGMII2_TX_CTL__ENET2_TX_EN 0xa0b1 + MX6SX_PAD_RGMII2_RXC__ENET2_RX_CLK 0x3081 + MX6SX_PAD_RGMII2_RD0__ENET2_RX_DATA_0 0x3081 + MX6SX_PAD_RGMII2_RD1__ENET2_RX_DATA_1 0x3081 + MX6SX_PAD_RGMII2_RD2__ENET2_RX_DATA_2 0x3081 + MX6SX_PAD_RGMII2_RD3__ENET2_RX_DATA_3 0x3081 + MX6SX_PAD_RGMII2_RX_CTL__ENET2_RX_EN 0x3081 + >; + }; - pinctrl_usdhc3_200mhz: usdhc3grp-200mhz { - fsl,pins = < - MX6SX_PAD_SD3_CMD__USDHC3_CMD 0x170f9 - MX6SX_PAD_SD3_CLK__USDHC3_CLK 0x100f9 - MX6SX_PAD_SD3_DATA0__USDHC3_DATA0 0x170f9 - MX6SX_PAD_SD3_DATA1__USDHC3_DATA1 0x170f9 - MX6SX_PAD_SD3_DATA2__USDHC3_DATA2 0x170f9 - MX6SX_PAD_SD3_DATA3__USDHC3_DATA3 0x170f9 - MX6SX_PAD_SD3_DATA4__USDHC3_DATA4 0x170f9 - MX6SX_PAD_SD3_DATA5__USDHC3_DATA5 0x170f9 - MX6SX_PAD_SD3_DATA6__USDHC3_DATA6 0x170f9 - MX6SX_PAD_SD3_DATA7__USDHC3_DATA7 0x170f9 - >; - }; + pinctrl_i2c2: i2c2grp { + fsl,pins = < + MX6SX_PAD_GPIO1_IO03__I2C2_SDA 0x4001b8b1 + MX6SX_PAD_GPIO1_IO02__I2C2_SCL 0x4001b8b1 + >; + }; - pinctrl_usdhc4: usdhc4grp { - fsl,pins = < - MX6SX_PAD_SD4_CMD__USDHC4_CMD 0x17059 - MX6SX_PAD_SD4_CLK__USDHC4_CLK 0x10059 - MX6SX_PAD_SD4_DATA0__USDHC4_DATA0 0x17059 - MX6SX_PAD_SD4_DATA1__USDHC4_DATA1 0x17059 - MX6SX_PAD_SD4_DATA2__USDHC4_DATA2 0x17059 - MX6SX_PAD_SD4_DATA3__USDHC4_DATA3 0x17059 - MX6SX_PAD_SD4_DATA7__GPIO6_IO_21 0x17059 /* CD */ - MX6SX_PAD_SD4_DATA6__GPIO6_IO_20 0x17059 /* WP */ - >; - }; + pinctrl_i2c3: i2c3grp { + fsl,pins = < + MX6SX_PAD_KEY_ROW4__I2C3_SDA 0x4001b8b1 + MX6SX_PAD_KEY_COL4__I2C3_SCL 0x4001b8b1 + >; + }; + + pinctrl_led: ledgrp { + fsl,pins = < + MX6SX_PAD_CSI_PIXCLK__GPIO1_IO_24 0x17059 + >; + }; + + pinctrl_uart1: uart1grp { + fsl,pins = < + MX6SX_PAD_GPIO1_IO04__UART1_TX 0x1b0b1 + MX6SX_PAD_GPIO1_IO05__UART1_RX 0x1b0b1 + >; + }; + + pinctrl_usdhc3: usdhc3grp { + fsl,pins = < + MX6SX_PAD_SD3_CMD__USDHC3_CMD 0x17059 + MX6SX_PAD_SD3_CLK__USDHC3_CLK 0x10059 + MX6SX_PAD_SD3_DATA0__USDHC3_DATA0 0x17059 + MX6SX_PAD_SD3_DATA1__USDHC3_DATA1 0x17059 + MX6SX_PAD_SD3_DATA2__USDHC3_DATA2 0x17059 + MX6SX_PAD_SD3_DATA3__USDHC3_DATA3 0x17059 + MX6SX_PAD_SD3_DATA4__USDHC3_DATA4 0x17059 + MX6SX_PAD_SD3_DATA5__USDHC3_DATA5 0x17059 + MX6SX_PAD_SD3_DATA6__USDHC3_DATA6 0x17059 + MX6SX_PAD_SD3_DATA7__USDHC3_DATA7 0x17059 + MX6SX_PAD_KEY_COL0__GPIO2_IO_10 0x17059 /* CD */ + MX6SX_PAD_KEY_ROW0__GPIO2_IO_15 0x17059 /* WP */ + >; + }; + + pinctrl_usdhc3_100mhz: usdhc3grp-100mhz { + fsl,pins = < + MX6SX_PAD_SD3_CMD__USDHC3_CMD 0x170b9 + MX6SX_PAD_SD3_CLK__USDHC3_CLK 0x100b9 + MX6SX_PAD_SD3_DATA0__USDHC3_DATA0 0x170b9 + MX6SX_PAD_SD3_DATA1__USDHC3_DATA1 0x170b9 + MX6SX_PAD_SD3_DATA2__USDHC3_DATA2 0x170b9 + MX6SX_PAD_SD3_DATA3__USDHC3_DATA3 0x170b9 + MX6SX_PAD_SD3_DATA4__USDHC3_DATA4 0x170b9 + MX6SX_PAD_SD3_DATA5__USDHC3_DATA5 0x170b9 + MX6SX_PAD_SD3_DATA6__USDHC3_DATA6 0x170b9 + MX6SX_PAD_SD3_DATA7__USDHC3_DATA7 0x170b9 + >; + }; + + pinctrl_usdhc3_200mhz: usdhc3grp-200mhz { + fsl,pins = < + MX6SX_PAD_SD3_CMD__USDHC3_CMD 0x170f9 + MX6SX_PAD_SD3_CLK__USDHC3_CLK 0x100f9 + MX6SX_PAD_SD3_DATA0__USDHC3_DATA0 0x170f9 + MX6SX_PAD_SD3_DATA1__USDHC3_DATA1 0x170f9 + MX6SX_PAD_SD3_DATA2__USDHC3_DATA2 0x170f9 + MX6SX_PAD_SD3_DATA3__USDHC3_DATA3 0x170f9 + MX6SX_PAD_SD3_DATA4__USDHC3_DATA4 0x170f9 + MX6SX_PAD_SD3_DATA5__USDHC3_DATA5 0x170f9 + MX6SX_PAD_SD3_DATA6__USDHC3_DATA6 0x170f9 + MX6SX_PAD_SD3_DATA7__USDHC3_DATA7 0x170f9 + >; + }; + + pinctrl_usdhc4: usdhc4grp { + fsl,pins = < + MX6SX_PAD_SD4_CMD__USDHC4_CMD 0x17059 + MX6SX_PAD_SD4_CLK__USDHC4_CLK 0x10059 + MX6SX_PAD_SD4_DATA0__USDHC4_DATA0 0x17059 + MX6SX_PAD_SD4_DATA1__USDHC4_DATA1 0x17059 + MX6SX_PAD_SD4_DATA2__USDHC4_DATA2 0x17059 + MX6SX_PAD_SD4_DATA3__USDHC4_DATA3 0x17059 + MX6SX_PAD_SD4_DATA7__GPIO6_IO_21 0x17059 /* CD */ + MX6SX_PAD_SD4_DATA6__GPIO6_IO_20 0x17059 /* WP */ + >; + }; + + pinctrl_vcc_sd3: vccsd3grp { + fsl,pins = < + MX6SX_PAD_KEY_COL1__GPIO2_IO_11 0x17059 + >; + }; + + pinctrl_wdog: wdoggrp { + fsl,pins = < + MX6SX_PAD_GPIO1_IO13__WDOG1_WDOG_ANY 0x30b0 + >; + }; +}; + +&i2c2 { + clock-frequency = <100000>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_i2c2>; + status = "okay"; + + touchscreen@4 { + compatible = "eeti,egalax_ts"; + reg = <0x04>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_egalax_int>; + interrupt-parent = <&gpio6>; + interrupts = <22 IRQ_TYPE_EDGE_FALLING>; + wakeup-gpios = <&gpio6 22 GPIO_ACTIVE_HIGH>; + }; + + pfuze100: pmic@8 { + compatible = "fsl,pfuze100"; + reg = <0x08>; + + regulators { + sw1a_reg: sw1ab { + regulator-min-microvolt = <300000>; + regulator-max-microvolt = <1875000>; + regulator-boot-on; + regulator-always-on; + regulator-ramp-delay = <6250>; + }; + + sw1c_reg: sw1c { + regulator-min-microvolt = <300000>; + regulator-max-microvolt = <1875000>; + regulator-boot-on; + regulator-always-on; + regulator-ramp-delay = <6250>; + }; + + sw2_reg: sw2 { + regulator-min-microvolt = <800000>; + regulator-max-microvolt = <3300000>; + regulator-boot-on; + regulator-always-on; + }; + + sw3a_reg: sw3a { + regulator-min-microvolt = <400000>; + regulator-max-microvolt = <1975000>; + regulator-boot-on; + regulator-always-on; + }; + + sw3b_reg: sw3b { + regulator-min-microvolt = <400000>; + regulator-max-microvolt = <1975000>; + regulator-boot-on; + regulator-always-on; + }; + + sw4_reg: sw4 { + regulator-min-microvolt = <800000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; + + swbst_reg: swbst { + regulator-min-microvolt = <5000000>; + regulator-max-microvolt = <5150000>; + }; + + snvs_reg: vsnvs { + regulator-min-microvolt = <1000000>; + regulator-max-microvolt = <3000000>; + regulator-boot-on; + regulator-always-on; + }; - pinctrl_vcc_sd3: vccsd3grp { - fsl,pins = < - MX6SX_PAD_KEY_COL1__GPIO2_IO_11 0x17059 - >; + vref_reg: vrefddr { + regulator-boot-on; + regulator-always-on; + }; + + vgen1_reg: vgen1 { + regulator-min-microvolt = <800000>; + regulator-max-microvolt = <1550000>; + regulator-always-on; + }; + + vgen2_reg: vgen2 { + regulator-min-microvolt = <800000>; + regulator-max-microvolt = <1550000>; + }; + + vgen3_reg: vgen3 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; + + vgen4_reg: vgen4 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; + + vgen5_reg: vgen5 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; + + vgen6_reg: vgen6 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; }; }; + + max7322: gpio@68 { + compatible = "maxim,max7322"; + reg = <0x68>; + gpio-controller; + #gpio-cells = <2>; + }; +}; + +&i2c3 { + clock-frequency = <100000>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_i2c3>; + status = "okay"; + + max7310_a: gpio@30 { + compatible = "maxim,max7310"; + reg = <0x30>; + gpio-controller; + #gpio-cells = <2>; + }; + + max7310_b: gpio@32 { + compatible = "maxim,max7310"; + reg = <0x32>; + gpio-controller; + #gpio-cells = <2>; + }; +}; + +&wdog1 { + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_wdog>; + fsl,ext-reset-output; }; diff --git a/arch/arm/boot/dts/imx6sx.dtsi b/arch/arm/boot/dts/imx6sx.dtsi index 49c7205b8db8..d8b94f47498b 100644 --- a/arch/arm/boot/dts/imx6sx.dtsi +++ b/arch/arm/boot/dts/imx6sx.dtsi @@ -1,10 +1,6 @@ -/* - * Copyright 2014 Freescale Semiconductor, Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright 2014 Freescale Semiconductor, Inc. #include #include @@ -104,41 +100,46 @@ interrupt-parent = <&intc>; }; - clocks { - #address-cells = <1>; - #size-cells = <0>; + ckil: clock-ckil { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <32768>; + clock-output-names = "ckil"; + }; - ckil: clock@0 { - compatible = "fixed-clock"; - reg = <0>; - #clock-cells = <0>; - clock-frequency = <32768>; - clock-output-names = "ckil"; - }; + osc: clock-osc { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <24000000>; + clock-output-names = "osc"; + }; - osc: clock@1 { - compatible = "fixed-clock"; - reg = <1>; - #clock-cells = <0>; - clock-frequency = <24000000>; - clock-output-names = "osc"; - }; + ipp_di0: clock-ipp-di0 { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + clock-output-names = "ipp_di0"; + }; - ipp_di0: clock@2 { - compatible = "fixed-clock"; - reg = <2>; - #clock-cells = <0>; - clock-frequency = <0>; - clock-output-names = "ipp_di0"; - }; + ipp_di1: clock-ipp-di1 { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + clock-output-names = "ipp_di1"; + }; - ipp_di1: clock@3 { - compatible = "fixed-clock"; - reg = <3>; - #clock-cells = <0>; - clock-frequency = <0>; - clock-output-names = "ipp_di1"; - }; + anaclk1: clock-anaclk1 { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + clock-output-names = "anaclk1"; + }; + + anaclk2: clock-anaclk2 { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + clock-output-names = "anaclk2"; }; tempmon: tempmon { @@ -575,8 +576,8 @@ interrupts = , ; #clock-cells = <1>; - clocks = <&ckil>, <&osc>, <&ipp_di0>, <&ipp_di1>; - clock-names = "ckil", "osc", "ipp_di0", "ipp_di1"; + clocks = <&ckil>, <&osc>, <&ipp_di0>, <&ipp_di1>, <&anaclk1>, <&anaclk2>; + clock-names = "ckil", "osc", "ipp_di0", "ipp_di1", "anaclk1", "anaclk2"; }; anatop: anatop@20c8000 { @@ -586,11 +587,8 @@ interrupts = , , ; - #address-cells = <1>; - #size-cells = <0>; - regulator-1p1@20c8110 { - reg = <0x20c8110>; + regulator-1p1 { compatible = "fsl,anatop-regulator"; regulator-name = "vdd1p1"; regulator-min-microvolt = <800000>; @@ -605,8 +603,7 @@ anatop-enable-bit = <0>; }; - regulator-3p0@20c8120 { - reg = <0x20c8120>; + regulator-3p0 { compatible = "fsl,anatop-regulator"; regulator-name = "vdd3p0"; regulator-min-microvolt = <2800000>; @@ -621,8 +618,7 @@ anatop-enable-bit = <0>; }; - regulator-2p5@20c8130 { - reg = <0x20c8130>; + regulator-2p5 { compatible = "fsl,anatop-regulator"; regulator-name = "vdd2p5"; regulator-min-microvolt = <2100000>; @@ -637,8 +633,7 @@ anatop-enable-bit = <0>; }; - reg_arm: regulator-vddcore@20c8140 { - reg = <0x20c8140>; + reg_arm: regulator-vddcore { compatible = "fsl,anatop-regulator"; regulator-name = "vddarm"; regulator-min-microvolt = <725000>; @@ -655,8 +650,7 @@ anatop-max-voltage = <1450000>; }; - reg_pcie: regulator-vddpcie@20c8140 { - reg = <0x20c8140>; + reg_pcie: regulator-vddpcie { compatible = "fsl,anatop-regulator"; regulator-name = "vddpcie"; regulator-min-microvolt = <725000>; @@ -672,8 +666,7 @@ anatop-max-voltage = <1450000>; }; - reg_soc: regulator-vddsoc@20c8140 { - reg = <0x20c8140>; + reg_soc: regulator-vddsoc { compatible = "fsl,anatop-regulator"; regulator-name = "vddsoc"; regulator-min-microvolt = <725000>; diff --git a/arch/arm/boot/dts/imx6ul-14x14-evk.dts b/arch/arm/boot/dts/imx6ul-14x14-evk.dts index 6d720b20e7ed..2438669f149a 100644 --- a/arch/arm/boot/dts/imx6ul-14x14-evk.dts +++ b/arch/arm/boot/dts/imx6ul-14x14-evk.dts @@ -1,10 +1,6 @@ -/* - * Copyright (C) 2015 Freescale Semiconductor, Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright (C) 2015 Freescale Semiconductor, Inc. /dts-v1/; diff --git a/arch/arm/boot/dts/imx6ul-isiot.dtsi b/arch/arm/boot/dts/imx6ul-isiot.dtsi index 921e12c69a00..cd9928551154 100644 --- a/arch/arm/boot/dts/imx6ul-isiot.dtsi +++ b/arch/arm/boot/dts/imx6ul-isiot.dtsi @@ -153,8 +153,6 @@ stmpe811: gpio-expander@44 { compatible = "st,stmpe811"; reg = <0x44>; - #address-cells = <1>; - #size-cells = <0>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_stmpe>; interrupt-parent = <&gpio1>; diff --git a/arch/arm/boot/dts/imx6ul-tx6ul-mainboard.dts b/arch/arm/boot/dts/imx6ul-tx6ul-mainboard.dts index 2d80f7b50bc0..97686097a86e 100644 --- a/arch/arm/boot/dts/imx6ul-tx6ul-mainboard.dts +++ b/arch/arm/boot/dts/imx6ul-tx6ul-mainboard.dts @@ -48,7 +48,7 @@ compatible = "karo,imx6ul-tx6ul", "fsl,imx6ul"; aliases { - lcdif_24bit_pins_a = &pinctrl_disp0_3; + lcdif-24bit-pins-a = &pinctrl_disp0_3; mmc0 = &usdhc1; /delete-property/ mmc1; serial2 = &uart3; diff --git a/arch/arm/boot/dts/imx6ul-tx6ul.dtsi b/arch/arm/boot/dts/imx6ul-tx6ul.dtsi index f678d18ad44a..02b5ba42cd59 100644 --- a/arch/arm/boot/dts/imx6ul-tx6ul.dtsi +++ b/arch/arm/boot/dts/imx6ul-tx6ul.dtsi @@ -53,10 +53,10 @@ i2c2 = &i2c1; i2c3 = &i2c3; i2c4 = &i2c4; - lcdif_23bit_pins_a = &pinctrl_disp0_1; - lcdif_24bit_pins_a = &pinctrl_disp0_2; + lcdif-23bit-pins-a = &pinctrl_disp0_1; + lcdif-24bit-pins-a = &pinctrl_disp0_2; pwm0 = &pwm5; - reg_can_xcvr = ®_can_xcvr; + reg-can-xcvr = ®_can_xcvr; serial2 = &uart5; serial4 = &uart3; spi0 = &ecspi2; diff --git a/arch/arm/boot/dts/imx6ul.dtsi b/arch/arm/boot/dts/imx6ul.dtsi index 1241972b16ba..47a3453a4211 100644 --- a/arch/arm/boot/dts/imx6ul.dtsi +++ b/arch/arm/boot/dts/imx6ul.dtsi @@ -1,10 +1,6 @@ -/* - * Copyright 2015 Freescale Semiconductor, Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright 2015 Freescale Semiconductor, Inc. #include #include @@ -551,11 +547,8 @@ interrupts = , , ; - #address-cells = <1>; - #size-cells = <0>; - reg_3p0: regulator-3p0@20c8110 { - reg = <0x20c8110>; + reg_3p0: regulator-3p0 { compatible = "fsl,anatop-regulator"; regulator-name = "vdd3p0"; regulator-min-microvolt = <2625000>; @@ -569,8 +562,7 @@ anatop-enable-bit = <0>; }; - reg_arm: regulator-vddcore@20c8140 { - reg = <0x20c8140>; + reg_arm: regulator-vddcore { compatible = "fsl,anatop-regulator"; regulator-name = "cpu"; regulator-min-microvolt = <725000>; @@ -587,8 +579,7 @@ anatop-max-voltage = <1450000>; }; - reg_soc: regulator-vddsoc@20c8140 { - reg = <0x20c8140>; + reg_soc: regulator-vddsoc { compatible = "fsl,anatop-regulator"; regulator-name = "vddsoc"; regulator-min-microvolt = <725000>; @@ -769,6 +760,36 @@ reg = <0x02100000 0x100000>; ranges; + crypto: caam@2140000 { + compatible = "fsl,imx6ul-caam", "fsl,sec-v4.0"; + #address-cells = <1>; + #size-cells = <1>; + reg = <0x2140000 0x3c000>; + ranges = <0 0x2140000 0x3c000>; + interrupts = ; + clocks = <&clks IMX6UL_CLK_CAAM_IPG>, <&clks IMX6UL_CLK_CAAM_ACLK>, + <&clks IMX6UL_CLK_CAAM_MEM>; + clock-names = "ipg", "aclk", "mem"; + + sec_jr0: jr0@1000 { + compatible = "fsl,sec-v4.0-job-ring"; + reg = <0x1000 0x1000>; + interrupts = ; + }; + + sec_jr1: jr1@2000 { + compatible = "fsl,sec-v4.0-job-ring"; + reg = <0x2000 0x1000>; + interrupts = ; + }; + + sec_jr2: jr2@3000 { + compatible = "fsl,sec-v4.0-job-ring"; + reg = <0x3000 0x1000>; + interrupts = ; + }; + }; + usbotg1: usb@2184000 { compatible = "fsl,imx6ul-usb", "fsl,imx27-usb"; reg = <0x02184000 0x200>; diff --git a/arch/arm/boot/dts/imx6ull-pinfunc.h b/arch/arm/boot/dts/imx6ull-pinfunc.h index 118202336691..fdc46bb09cc1 100644 --- a/arch/arm/boot/dts/imx6ull-pinfunc.h +++ b/arch/arm/boot/dts/imx6ull-pinfunc.h @@ -14,6 +14,14 @@ * The pin function ID is a tuple of * */ +#define MX6ULL_PAD_UART1_TX_DATA__UART5_DTE_RX 0x0084 0x0310 0x0644 0x9 0x4 +#define MX6ULL_PAD_UART1_RX_DATA__UART5_DCE_RX 0x0088 0x0314 0x0644 0x9 0x5 +#define MX6ULL_PAD_UART1_CTS_B__UART5_DCE_RTS 0x008C 0x0318 0x0640 0x9 0x3 +#define MX6ULL_PAD_UART1_RTS_B__UART5_DTE_RTS 0x0090 0x031C 0x0640 0x9 0x4 +#define MX6ULL_PAD_UART5_TX_DATA__UART5_DTE_RX 0x00BC 0x0348 0x0644 0x0 0x6 +#define MX6ULL_PAD_UART5_RX_DATA__UART5_DCE_RX 0x00C0 0x034C 0x0644 0x0 0x7 +#define MX6ULL_PAD_ENET1_RX_EN__UART5_DCE_RTS 0x00CC 0x0358 0x0640 0x1 0x5 +#define MX6ULL_PAD_ENET1_TX_DATA0__UART5_DTE_RTS 0x00D0 0x035C 0x0640 0x1 0x6 #define MX6ULL_PAD_ENET2_RX_DATA0__EPDC_SDDO08 0x00E4 0x0370 0x0000 0x9 0x0 #define MX6ULL_PAD_ENET2_RX_DATA1__EPDC_SDDO09 0x00E8 0x0374 0x0000 0x9 0x0 #define MX6ULL_PAD_ENET2_RX_EN__EPDC_SDDO10 0x00EC 0x0378 0x0000 0x9 0x0 @@ -47,6 +55,7 @@ #define MX6ULL_PAD_CSI_DATA00__ESAI_TX_HF_CLK 0x01E4 0x0470 0x0000 0x9 0x0 #define MX6ULL_PAD_CSI_DATA01__ESAI_RX_HF_CLK 0x01E8 0x0474 0x0000 0x9 0x0 #define MX6ULL_PAD_CSI_DATA02__ESAI_RX_FS 0x01EC 0x0478 0x0000 0x9 0x0 +#define MX6ULL_PAD_CSI_DATA02__UART5_DCE_RTS 0x01EC 0x0478 0x0640 0x8 0x7 #define MX6ULL_PAD_CSI_DATA03__ESAI_RX_CLK 0x01F0 0x047C 0x0000 0x9 0x0 #define MX6ULL_PAD_CSI_DATA04__ESAI_TX_FS 0x01F4 0x0480 0x0000 0x9 0x0 #define MX6ULL_PAD_CSI_DATA05__ESAI_TX_CLK 0x01F8 0x0484 0x0000 0x9 0x0 diff --git a/arch/arm/boot/dts/imx6ull.dtsi b/arch/arm/boot/dts/imx6ull.dtsi index 571ddd71cdba..ebc25c98e5e1 100644 --- a/arch/arm/boot/dts/imx6ull.dtsi +++ b/arch/arm/boot/dts/imx6ull.dtsi @@ -45,6 +45,8 @@ /* Delete UART8 in AIPS-1 (i.MX6UL specific) */ /delete-node/ &uart8; +/* Delete CAAM node in AIPS-2 (i.MX6UL specific) */ +/delete-node/ &crypto; / { soc { diff --git a/arch/arm/boot/dts/imx7d-cl-som-imx7.dts b/arch/arm/boot/dts/imx7d-cl-som-imx7.dts index 7f645683f53b..8bf365d28cac 100644 --- a/arch/arm/boot/dts/imx7d-cl-som-imx7.dts +++ b/arch/arm/boot/dts/imx7d-cl-som-imx7.dts @@ -33,7 +33,7 @@ }; &cpu0 { - arm-supply = <&sw1a_reg>; + cpu-supply = <&sw1a_reg>; }; &fec1 { diff --git a/arch/arm/boot/dts/imx7d-nitrogen7.dts b/arch/arm/boot/dts/imx7d-nitrogen7.dts index 52167298984d..70c53e50b2fc 100644 --- a/arch/arm/boot/dts/imx7d-nitrogen7.dts +++ b/arch/arm/boot/dts/imx7d-nitrogen7.dts @@ -49,8 +49,8 @@ compatible = "boundary,imx7d-nitrogen7", "fsl,imx7d"; aliases { - fb_lcd = &lcdif; - t_lcd = &t_lcd; + fb-lcd = &lcdif; + t-lcd = &t_lcd; }; memory@80000000 { @@ -144,7 +144,7 @@ }; &cpu0 { - arm-supply = <&sw1a_reg>; + cpu-supply = <&sw1a_reg>; }; &fec1 { diff --git a/arch/arm/boot/dts/imx7d-pinfunc.h b/arch/arm/boot/dts/imx7d-pinfunc.h index f2493bc63da4..aa9dbead4b8b 100644 --- a/arch/arm/boot/dts/imx7d-pinfunc.h +++ b/arch/arm/boot/dts/imx7d-pinfunc.h @@ -592,7 +592,7 @@ #define MX7D_PAD_UART2_RX_DATA__UART2_DCE_RX 0x0130 0x03A0 0x06FC 0x0 0x2 #define MX7D_PAD_UART2_RX_DATA__UART2_DTE_TX 0x0130 0x03A0 0x0000 0x0 0x0 #define MX7D_PAD_UART2_RX_DATA__I2C2_SCL 0x0130 0x03A0 0x05DC 0x1 0x0 -#define MX7D_PAD_UART2_RX_DATA__SAI3_RX_BCLK 0x0130 0x03A0 0x0000 0x2 0x0 +#define MX7D_PAD_UART2_RX_DATA__SAI3_RX_BCLK 0x0130 0x03A0 0x06C4 0x2 0x0 #define MX7D_PAD_UART2_RX_DATA__ECSPI1_SS3 0x0130 0x03A0 0x0000 0x3 0x0 #define MX7D_PAD_UART2_RX_DATA__ENET2_1588_EVENT1_IN 0x0130 0x03A0 0x0000 0x4 0x0 #define MX7D_PAD_UART2_RX_DATA__GPIO4_IO2 0x0130 0x03A0 0x0000 0x5 0x0 @@ -1112,13 +1112,13 @@ #define MX7D_PAD_ENET1_RGMII_TD3__GPIO7_IO9 0x0250 0x04C0 0x0000 0x5 0x0 #define MX7D_PAD_ENET1_RGMII_TD3__CAAM_RNG_OSC_OBS 0x0250 0x04C0 0x0000 0x7 0x0 #define MX7D_PAD_ENET1_RGMII_TX_CTL__ENET1_RGMII_TX_CTL 0x0254 0x04C4 0x0000 0x0 0x0 -#define MX7D_PAD_ENET1_RGMII_TX_CTL__SAI1_RX_SYNC 0x0254 0x04C4 0x0000 0x2 0x0 +#define MX7D_PAD_ENET1_RGMII_TX_CTL__SAI1_RX_SYNC 0x0254 0x04C4 0x06A4 0x2 0x1 #define MX7D_PAD_ENET1_RGMII_TX_CTL__GPT2_COMPARE1 0x0254 0x04C4 0x0000 0x3 0x0 #define MX7D_PAD_ENET1_RGMII_TX_CTL__EPDC_PWR_CTRL2 0x0254 0x04C4 0x0000 0x4 0x0 #define MX7D_PAD_ENET1_RGMII_TX_CTL__GPIO7_IO10 0x0254 0x04C4 0x0000 0x5 0x0 #define MX7D_PAD_ENET1_RGMII_TXC__ENET1_RGMII_TXC 0x0258 0x04C8 0x0000 0x0 0x0 #define MX7D_PAD_ENET1_RGMII_TXC__ENET1_TX_ER 0x0258 0x04C8 0x0000 0x1 0x0 -#define MX7D_PAD_ENET1_RGMII_TXC__SAI1_RX_BCLK 0x0258 0x04C8 0x0000 0x2 0x0 +#define MX7D_PAD_ENET1_RGMII_TXC__SAI1_RX_BCLK 0x0258 0x04C8 0x069C 0x2 0x1 #define MX7D_PAD_ENET1_RGMII_TXC__GPT2_COMPARE2 0x0258 0x04C8 0x0000 0x3 0x0 #define MX7D_PAD_ENET1_RGMII_TXC__EPDC_PWR_CTRL3 0x0258 0x04C8 0x0000 0x4 0x0 #define MX7D_PAD_ENET1_RGMII_TXC__GPIO7_IO11 0x0258 0x04C8 0x0000 0x5 0x0 diff --git a/arch/arm/boot/dts/imx7d-sdb-sht11.dts b/arch/arm/boot/dts/imx7d-sdb-sht11.dts index 64a20ed1713a..996555596d40 100644 --- a/arch/arm/boot/dts/imx7d-sdb-sht11.dts +++ b/arch/arm/boot/dts/imx7d-sdb-sht11.dts @@ -1,44 +1,6 @@ -/* - * Copyright (C) 2015 Freescale Semiconductor, Inc. - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation; either version 2 of the - * License, or (at your option) any later version. - * - * This file is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - */ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +// +// Copyright (C) 2015 Freescale Semiconductor, Inc. #include "imx7d-sdb.dts" diff --git a/arch/arm/boot/dts/imx7d-sdb.dts b/arch/arm/boot/dts/imx7d-sdb.dts index 5d6a08be397f..940849163104 100644 --- a/arch/arm/boot/dts/imx7d-sdb.dts +++ b/arch/arm/boot/dts/imx7d-sdb.dts @@ -1,44 +1,6 @@ -/* - * Copyright (C) 2015 Freescale Semiconductor, Inc. - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation; either version 2 of the - * License, or (at your option) any later version. - * - * This file is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - */ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +// +// Copyright (C) 2015 Freescale Semiconductor, Inc. /dts-v1/; @@ -52,6 +14,24 @@ reg = <0x80000000 0x80000000>; }; + gpio-keys { + compatible = "gpio-keys"; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_gpio_keys>; + + volume-up { + label = "Volume Up"; + gpios = <&gpio5 11 GPIO_ACTIVE_LOW>; + linux,code = ; + }; + + volume-down { + label = "Volume Down"; + gpios = <&gpio5 10 GPIO_ACTIVE_LOW>; + linux,code = ; + }; + }; + spi4 { compatible = "spi-gpio"; pinctrl-names = "default"; @@ -161,7 +141,7 @@ }; &cpu0 { - arm-supply = <&sw1a_reg>; + cpu-supply = <&sw1a_reg>; }; &ecspi3 { @@ -519,6 +499,12 @@ >; }; + pinctrl_gpio_keys: gpio_keysgrp { + fsl,pins = < + MX7D_PAD_SD2_RESET_B__GPIO5_IO11 0x59 + MX7D_PAD_SD2_WP__GPIO5_IO10 0x59 + >; + }; pinctrl_hog: hoggrp { fsl,pins = < diff --git a/arch/arm/boot/dts/imx7d.dtsi b/arch/arm/boot/dts/imx7d.dtsi index 200714e3feea..8d3d123d0a5c 100644 --- a/arch/arm/boot/dts/imx7d.dtsi +++ b/arch/arm/boot/dts/imx7d.dtsi @@ -1,45 +1,7 @@ -/* - * Copyright 2015 Freescale Semiconductor, Inc. - * Copyright 2016 Toradex AG - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation; either version 2 of the - * License, or (at your option) any later version. - * - * This file is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - */ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +// +// Copyright 2015 Freescale Semiconductor, Inc. +// Copyright 2016 Toradex AG #include "imx7s.dtsi" #include @@ -47,12 +9,8 @@ / { cpus { cpu0: cpu@0 { - operating-points = < - /* KHz uV */ - 996000 1075000 - 792000 975000 - >; clock-frequency = <996000000>; + operating-points-v2 = <&cpu0_opp_table>; }; cpu1: cpu@1 { @@ -60,6 +18,25 @@ device_type = "cpu"; reg = <1>; clock-frequency = <996000000>; + operating-points-v2 = <&cpu0_opp_table>; + }; + }; + + cpu0_opp_table: opp-table { + compatible = "operating-points-v2"; + opp-shared; + + opp-792000000 { + opp-hz = /bits/ 64 <792000000>; + opp-microvolt = <975000>; + clock-latency-ns = <150000>; + }; + + opp-996000000 { + opp-hz = /bits/ 64 <996000000>; + opp-microvolt = <1075000>; + clock-latency-ns = <150000>; + opp-suspend; }; }; @@ -120,7 +97,7 @@ , , ; - clocks = <&clks IMX7D_ENET_AXI_ROOT_CLK>, + clocks = <&clks IMX7D_ENET2_IPG_ROOT_CLK>, <&clks IMX7D_ENET_AXI_ROOT_CLK>, <&clks IMX7D_ENET2_TIME_ROOT_CLK>, <&clks IMX7D_PLL_ENET_MAIN_125M_CLK>, diff --git a/arch/arm/boot/dts/imx7s-warp.dts b/arch/arm/boot/dts/imx7s-warp.dts index 8a30b148534d..fa390da636de 100644 --- a/arch/arm/boot/dts/imx7s-warp.dts +++ b/arch/arm/boot/dts/imx7s-warp.dts @@ -113,10 +113,6 @@ assigned-clock-rates = <884736000>; }; -&cpu0 { - arm-supply = <&sw1a_reg>; -}; - &i2c1 { pinctrl-names = "default"; pinctrl-0 = <&pinctrl_i2c1>; diff --git a/arch/arm/boot/dts/imx7s.dtsi b/arch/arm/boot/dts/imx7s.dtsi index ce85b3ca1a55..9ced589bfa96 100644 --- a/arch/arm/boot/dts/imx7s.dtsi +++ b/arch/arm/boot/dts/imx7s.dtsi @@ -1,45 +1,7 @@ -/* - * Copyright 2015 Freescale Semiconductor, Inc. - * Copyright 2016 Toradex AG - * - * This file is dual-licensed: you can use it either under the terms - * of the GPL or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This file is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation; either version 2 of the - * License, or (at your option) any later version. - * - * This file is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - */ +// SPDX-License-Identifier: GPL-2.0+ OR MIT +// +// Copyright 2015 Freescale Semiconductor, Inc. +// Copyright 2016 Toradex AG #include #include @@ -173,6 +135,17 @@ }; }; + tempmon: tempmon { + compatible = "fsl,imx7d-tempmon"; + interrupt-parent = <&gpc>; + interrupts = ; + fsl,tempmon =<&anatop>; + nvmem-cells = <&tempmon_calib>, + <&tempmon_temp_grade>; + nvmem-cell-names = "calib", "temp_grade"; + clocks = <&clks IMX7D_PLL_SYS_MAIN_CLK>; + }; + timer { compatible = "arm,armv7-timer"; interrupt-parent = <&intc>; @@ -321,7 +294,7 @@ port { tpiu_in_port: endpoint { slave-mode; - remote-endpoint = <&replicator_out_port1>; + remote-endpoint = <&replicator_out_port0>; }; }; }; @@ -540,27 +513,14 @@ }; }; - tempmon: tempmon { - compatible = "fsl,imx7d-tempmon"; - interrupts = ; - fsl,tempmon =<&anatop>; - nvmem-cells = <&tempmon_calib>, - <&tempmon_temp_grade>; - nvmem-cell-names = "calib", "temp_grade"; - clocks = <&clks IMX7D_PLL_SYS_MAIN_CLK>; - }; - anatop: anatop@30360000 { compatible = "fsl,imx7d-anatop", "fsl,imx6q-anatop", "syscon", "simple-bus"; reg = <0x30360000 0x10000>; interrupts = , ; - #address-cells = <1>; - #size-cells = <0>; - reg_1p0d: regulator-vdd1p0d@30360210 { - reg = <0x30360210>; + reg_1p0d: regulator-vdd1p0d { compatible = "fsl,anatop-regulator"; regulator-name = "vdd1p0d"; regulator-min-microvolt = <800000>; @@ -573,6 +533,20 @@ anatop-max-voltage = <1200000>; anatop-enable-bit = <0>; }; + + reg_1p2: regulator-vdd1p2 { + compatible = "fsl,anatop-regulator"; + regulator-name = "vdd1p2"; + regulator-min-microvolt = <1100000>; + regulator-max-microvolt = <1300000>; + anatop-reg-offset = <0x220>; + anatop-vol-bit-shift = <8>; + anatop-vol-bit-width = <5>; + anatop-min-bit-val = <0x14>; + anatop-min-voltage = <1100000>; + anatop-max-voltage = <1300000>; + anatop-enable-bit = <0>; + }; }; snvs: snvs@30370000 { @@ -1092,7 +1066,7 @@ , , ; - clocks = <&clks IMX7D_ENET_AXI_ROOT_CLK>, + clocks = <&clks IMX7D_ENET1_IPG_ROOT_CLK>, <&clks IMX7D_ENET_AXI_ROOT_CLK>, <&clks IMX7D_ENET1_TIME_ROOT_CLK>, <&clks IMX7D_PLL_ENET_MAIN_125M_CLK>, diff --git a/arch/arm/boot/dts/keystone-k2g-evm.dts b/arch/arm/boot/dts/keystone-k2g-evm.dts index 6a4657799b99..154fdd7a7022 100644 --- a/arch/arm/boot/dts/keystone-k2g-evm.dts +++ b/arch/arm/boot/dts/keystone-k2g-evm.dts @@ -114,6 +114,20 @@ K2G_CORE_IOPAD(0x11f0) (BUFFER_CLASS_B | PIN_PULLDOWN | MUX_MODE0) /* uart2_txd.uart2_txd */ >; }; + + dcan0_pins: pinmux_dcan0_pins { + pinctrl-single,pins = < + K2G_CORE_IOPAD(0x11fc) (BUFFER_CLASS_B | PULL_DISABLE | MUX_MODE0) /* dcan0tx.dcan0tx */ + K2G_CORE_IOPAD(0x1200) (BUFFER_CLASS_B | PIN_PULLDOWN | MUX_MODE0) /* dcan0rx.dcan0rx */ + >; + }; + + dcan1_pins: pinmux_dcan1_pins { + pinctrl-single,pins = < + K2G_CORE_IOPAD(0x1224) (BUFFER_CLASS_B | PULL_DISABLE | MUX_MODE1) /* qspicsn2.dcan1tx */ + K2G_CORE_IOPAD(0x1228) (BUFFER_CLASS_B | PIN_PULLDOWN | MUX_MODE1) /* qspicsn3.dcan1rx */ + >; + }; }; &uart0 { @@ -268,3 +282,15 @@ pinctrl-0 = <&uart2_pins>; status = "okay"; }; + +&dcan0 { + pinctrl-names = "default"; + pinctrl-0 = <&dcan0_pins>; + status = "okay"; +}; + +&dcan1 { + pinctrl-names = "default"; + pinctrl-0 = <&dcan1_pins>; + status = "okay"; +}; diff --git a/arch/arm/boot/dts/logicpd-som-lv.dtsi b/arch/arm/boot/dts/logicpd-som-lv.dtsi index 6fa7bba3e801..3bb28c03ca74 100644 --- a/arch/arm/boot/dts/logicpd-som-lv.dtsi +++ b/arch/arm/boot/dts/logicpd-som-lv.dtsi @@ -35,6 +35,13 @@ reset-gpios = <&gpio1 4 GPIO_ACTIVE_LOW>; /* gpio_4 */ #phy-cells = <0>; }; + + /* fixed 26MHz oscillator */ + hfclk_26m: oscillator { + #clock-cells = <0>; + compatible = "fixed-clock"; + clock-frequency = <26000000>; + }; }; &gpmc { @@ -79,6 +86,8 @@ reg = <0x48>; interrupts = <7>; /* SYS_NIRQ cascaded to intc */ interrupt-parent = <&intc>; + clocks = <&hfclk_26m>; + clock-names = "fck"; twl_audio: audio { compatible = "ti,twl4030-audio"; codec { @@ -98,6 +107,25 @@ pinctrl-names = "default"; pinctrl-0 = <&i2c3_pins>; clock-frequency = <400000>; + + touchscreen: tsc2004@48 { + compatible = "ti,tsc2004"; + reg = <0x48>; + vio-supply = <&vaux1>; + pinctrl-names = "default"; + pinctrl-0 = <&tsc2004_pins>; + interrupts-extended = <&gpio5 25 IRQ_TYPE_EDGE_RISING>; /* gpio 153 */ + + touchscreen-fuzz-x = <4>; + touchscreen-fuzz-y = <7>; + touchscreen-fuzz-pressure = <2>; + touchscreen-size-x = <4096>; + touchscreen-size-y = <4096>; + touchscreen-max-pressure = <2048>; + + ti,x-plate-ohms = <280>; + ti,esd-recovery-timeout-ms = <8000>; + }; }; &mmc3 { @@ -203,6 +231,12 @@ OMAP3_CORE1_IOPAD(0x20ba, PIN_OUTPUT | MUX_MODE4) /* gpmc_ncs6.gpio_57 */ >; }; + + tsc2004_pins: pinmux_tsc2004_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x2186, PIN_INPUT | MUX_MODE4) /* mcbsp4_dr.gpio_153 */ + >; + }; }; &omap3_pmx_wkup { diff --git a/arch/arm/boot/dts/logicpd-torpedo-som.dtsi b/arch/arm/boot/dts/logicpd-torpedo-som.dtsi index 3e174e474d3d..7d2302e8706c 100644 --- a/arch/arm/boot/dts/logicpd-torpedo-som.dtsi +++ b/arch/arm/boot/dts/logicpd-torpedo-som.dtsi @@ -30,6 +30,13 @@ linux,default-trigger = "none"; }; }; + + /* fixed 26MHz oscillator */ + hfclk_26m: oscillator { + #clock-cells = <0>; + compatible = "fixed-clock"; + clock-frequency = <26000000>; + }; }; &gpmc { @@ -74,6 +81,9 @@ reg = <0x48>; interrupts = <7>; /* SYS_NIRQ cascaded to intc */ interrupt-parent = <&intc>; + clocks = <&hfclk_26m>; + clock-names = "fck"; + twl_audio: audio { compatible = "ti,twl4030-audio"; codec { diff --git a/arch/arm/boot/dts/meson8.dtsi b/arch/arm/boot/dts/meson8.dtsi index dcc9292d2ffa..d77dcf890cfc 100644 --- a/arch/arm/boot/dts/meson8.dtsi +++ b/arch/arm/boot/dts/meson8.dtsi @@ -57,7 +57,7 @@ #address-cells = <1>; #size-cells = <0>; - cpu@200 { + cpu0: cpu@200 { device_type = "cpu"; compatible = "arm,cortex-a9"; next-level-cache = <&L2>; @@ -66,7 +66,7 @@ resets = <&clkc CLKC_RESET_CPU0_SOFT_RESET>; }; - cpu@201 { + cpu1: cpu@201 { device_type = "cpu"; compatible = "arm,cortex-a9"; next-level-cache = <&L2>; @@ -75,7 +75,7 @@ resets = <&clkc CLKC_RESET_CPU1_SOFT_RESET>; }; - cpu@202 { + cpu2: cpu@202 { device_type = "cpu"; compatible = "arm,cortex-a9"; next-level-cache = <&L2>; @@ -84,7 +84,7 @@ resets = <&clkc CLKC_RESET_CPU2_SOFT_RESET>; }; - cpu@203 { + cpu3: cpu@203 { device_type = "cpu"; compatible = "arm,cortex-a9"; next-level-cache = <&L2>; @@ -94,6 +94,15 @@ }; }; + pmu { + compatible = "arm,cortex-a9-pmu"; + interrupts = , + , + , + ; + interrupt-affinity = <&cpu0>, <&cpu1>, <&cpu2>, <&cpu3>; + }; + reserved-memory { #address-cells = <1>; #size-cells = <1>; @@ -272,6 +281,22 @@ function = "pwm_e"; }; }; + + uart_a1_pins: uart-a1 { + mux { + groups = "uart_tx_a1", + "uart_rx_a1"; + function = "uart_a"; + }; + }; + + uart_a1_cts_rts_pins: uart-a1-cts-rts { + mux { + groups = "uart_cts_a1", + "uart_rts_a1"; + function = "uart_a"; + }; + }; }; }; diff --git a/arch/arm/boot/dts/meson8b-odroidc1.dts b/arch/arm/boot/dts/meson8b-odroidc1.dts index 3a5603d95b70..ef3177d3da3d 100644 --- a/arch/arm/boot/dts/meson8b-odroidc1.dts +++ b/arch/arm/boot/dts/meson8b-odroidc1.dts @@ -103,10 +103,34 @@ }; }; -&uart_AO { +ðmac { status = "okay"; - pinctrl-0 = <&uart_ao_a_pins>; + + snps,reset-gpio = <&gpio GPIOH_4 GPIO_ACTIVE_HIGH>; + snps,reset-active-low; + snps,reset-delays-us = <0 10000 30000>; + + pinctrl-0 = <ð_rgmii_pins>; pinctrl-names = "default"; + + phy-mode = "rgmii"; + phy-handle = <ð_phy>; + amlogic,tx-delay-ns = <4>; + + mdio { + compatible = "snps,dwmac-mdio"; + #address-cells = <1>; + #size-cells = <0>; + + /* Realtek RTL8211F (0x001cc916) */ + eth_phy: ethernet-phy@0 { + reg = <0>; + eee-broken-1000t; + interrupt-parent = <&gpio_intc>; + /* GPIOH_3 */ + interrupts = <17 IRQ_TYPE_LEVEL_LOW>; + }; + }; }; &gpio_ao { @@ -124,12 +148,10 @@ }; }; -&usb1_phy { - status = "okay"; -}; - -&usb1 { +&ir_receiver { status = "okay"; + pinctrl-0 = <&ir_recv_pins>; + pinctrl-names = "default"; }; &sdio { @@ -158,32 +180,16 @@ }; }; -ðmac { +&uart_AO { status = "okay"; - - snps,reset-gpio = <&gpio GPIOH_4 GPIO_ACTIVE_HIGH>; - snps,reset-active-low; - snps,reset-delays-us = <0 10000 30000>; - - pinctrl-0 = <ð_rgmii_pins>; + pinctrl-0 = <&uart_ao_a_pins>; pinctrl-names = "default"; +}; - phy-mode = "rgmii"; - phy-handle = <ð_phy>; - amlogic,tx-delay-ns = <4>; - - mdio { - compatible = "snps,dwmac-mdio"; - #address-cells = <1>; - #size-cells = <0>; +&usb1_phy { + status = "okay"; +}; - /* Realtek RTL8211F (0x001cc916) */ - eth_phy: ethernet-phy@0 { - reg = <0>; - eee-broken-1000t; - interrupt-parent = <&gpio_intc>; - /* GPIOH_3 */ - interrupts = <17 IRQ_TYPE_LEVEL_LOW>; - }; - }; +&usb1 { + status = "okay"; }; diff --git a/arch/arm/boot/dts/meson8b.dtsi b/arch/arm/boot/dts/meson8b.dtsi index 553b82174604..08f7f6be7254 100644 --- a/arch/arm/boot/dts/meson8b.dtsi +++ b/arch/arm/boot/dts/meson8b.dtsi @@ -55,7 +55,7 @@ #address-cells = <1>; #size-cells = <0>; - cpu@200 { + cpu0: cpu@200 { device_type = "cpu"; compatible = "arm,cortex-a5"; next-level-cache = <&L2>; @@ -64,7 +64,7 @@ resets = <&clkc CLKC_RESET_CPU0_SOFT_RESET>; }; - cpu@201 { + cpu1: cpu@201 { device_type = "cpu"; compatible = "arm,cortex-a5"; next-level-cache = <&L2>; @@ -73,7 +73,7 @@ resets = <&clkc CLKC_RESET_CPU1_SOFT_RESET>; }; - cpu@202 { + cpu2: cpu@202 { device_type = "cpu"; compatible = "arm,cortex-a5"; next-level-cache = <&L2>; @@ -82,7 +82,7 @@ resets = <&clkc CLKC_RESET_CPU2_SOFT_RESET>; }; - cpu@203 { + cpu3: cpu@203 { device_type = "cpu"; compatible = "arm,cortex-a5"; next-level-cache = <&L2>; @@ -92,6 +92,15 @@ }; }; + pmu { + compatible = "arm,cortex-a5-pmu"; + interrupts = , + , + , + ; + interrupt-affinity = <&cpu0>, <&cpu1>, <&cpu2>, <&cpu3>; + }; + reserved-memory { #address-cells = <1>; #size-cells = <1>; @@ -139,6 +148,13 @@ function = "uart_ao"; }; }; + + ir_recv_pins: remote { + mux { + groups = "remote_input"; + function = "remote"; + }; + }; }; }; diff --git a/arch/arm/boot/dts/meson8m2-mxiii-plus.dts b/arch/arm/boot/dts/meson8m2-mxiii-plus.dts new file mode 100644 index 000000000000..f5853610b20b --- /dev/null +++ b/arch/arm/boot/dts/meson8m2-mxiii-plus.dts @@ -0,0 +1,244 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR MIT) +/* + * Copyright (c) 2018 Oleg Ivanov + * Copyright (c) 2018 Martin Blumenstingl + */ + +/dts-v1/; + +#include "meson8m2.dtsi" + +#include +#include + +/ { + model = "Tronsmart MXIII Plus"; + compatible = "tronsmart,mxiii-plus", "amlogic,meson8m2"; + + aliases { + ethernet0 = ðmac; + i2c0 = &i2c_AO; + serial0 = &uart_AO; + serial1 = &uart_A; + mmc0 = &sd_card_slot; + }; + + chosen { + stdout-path = "serial0:115200n8"; + }; + + memory { + reg = <0x40000000 0x80000000>; + }; + + adc-keys { + compatible = "adc-keys"; + io-channels = <&saradc 0>; + io-channel-names = "buttons"; + keyup-threshold-microvolt = <1710000>; + + button-function { + label = "Function"; + linux,code = ; + press-threshold-microvolt = <10000>; + }; + }; + + vcc_3v3: regulator-vcc3v3 { + compatible = "regulator-fixed"; + regulator-name = "VCC3V3"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + }; +}; + +&cpu0 { + cpu-supply = <&vcck>; +}; + +ðmac { + status = "okay"; + + pinctrl-0 = <ð_rgmii_pins>; + pinctrl-names = "default"; + + phy-handle = <ð_phy0>; + phy-mode = "rgmii"; + + amlogic,tx-delay-ns = <4>; + + snps,reset-gpio = <&gpio GPIOH_4 0>; + snps,reset-delays-us = <0 10000 1000000>; + snps,reset-active-low; + + mdio { + compatible = "snps,dwmac-mdio"; + #address-cells = <1>; + #size-cells = <0>; + + eth_phy0: ethernet-phy@0 { + /* Realtek RTL8211F (0x001cc916) */ + reg = <0>; + }; + }; +}; + +&ir_receiver { + status = "okay"; + pinctrl-0 = <&ir_recv_pins>; + pinctrl-names = "default"; +}; + +&i2c_AO { + status = "okay"; + pinctrl-0 = <&i2c_ao_pins>; + pinctrl-names = "default"; + + pmic@32 { + compatible = "ricoh,rn5t618"; + reg = <0x32>; + system-power-controller; + + regulators { + vcck: DCDC1 { + regulator-name = "VCCK"; + regulator-min-microvolt = <825000>; + regulator-max-microvolt = <1150000>; + regulator-boot-on; + regulator-always-on; + }; + + DCDC2 { + regulator-name = "VDDAO"; + regulator-min-microvolt = <950000>; + regulator-max-microvolt = <1150000>; + regulator-boot-on; + regulator-always-on; + }; + + DCDC3 { + regulator-name = "VDD_DDR"; + regulator-min-microvolt = <1500000>; + regulator-max-microvolt = <1500000>; + regulator-boot-on; + regulator-always-on; + }; + + LDO1 { + regulator-name = "VDDIO_AO28"; + regulator-min-microvolt = <2900000>; + regulator-max-microvolt = <2900000>; + regulator-boot-on; + regulator-always-on; + }; + + vddio_ao1v8: LDO2 { + regulator-name = "VDDIO_AO18"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-boot-on; + regulator-always-on; + }; + + LDO3 { + regulator-name = "VCC1V8"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-boot-on; + regulator-always-on; + }; + + LDO4 { + regulator-name = "VCC2V8"; + regulator-min-microvolt = <2850000>; + regulator-max-microvolt = <2850000>; + regulator-boot-on; + regulator-always-on; + }; + + LDO5 { + regulator-name = "AVDD1V8"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-boot-on; + regulator-always-on; + }; + + LDORTC1 { + regulator-name = "VDD_LDO"; + regulator-min-microvolt = <2700000>; + regulator-max-microvolt = <2700000>; + regulator-boot-on; + regulator-always-on; + }; + + LDORTC2 { + regulator-name = "RTC_0V9"; + regulator-min-microvolt = <900000>; + regulator-max-microvolt = <900000>; + regulator-boot-on; + regulator-always-on; + }; + }; + }; +}; + +&saradc { + status = "okay"; + vref-supply = <&vddio_ao1v8>; +}; + +&sdio { + status = "okay"; + + pinctrl-0 = <&sd_b_pins>; + pinctrl-names = "default"; + + /* SD card */ + sd_card_slot: slot@1 { + compatible = "mmc-slot"; + reg = <1>; + status = "okay"; + + bus-width = <4>; + no-sdio; + cap-mmc-highspeed; + cap-sd-highspeed; + disable-wp; + + cd-gpios = <&gpio CARD_6 GPIO_ACTIVE_HIGH>; + cd-inverted; + + vmmc-supply = <&vcc_3v3>; + }; +}; + +/* connected to the Bluetooth module */ +&uart_A { + status = "okay"; + pinctrl-0 = <&uart_a1_pins>, <&uart_a1_cts_rts_pins>; + pinctrl-names = "default"; + uart-has-rtscts; +}; + +&uart_AO { + status = "okay"; + pinctrl-0 = <&uart_ao_a_pins>; + pinctrl-names = "default"; +}; + +&usb0 { + status = "okay"; +}; + +&usb1 { + status = "okay"; +}; + +&usb0_phy { + status = "okay"; +}; + +&usb1_phy { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/meson8m2.dtsi b/arch/arm/boot/dts/meson8m2.dtsi new file mode 100644 index 000000000000..3e1f92273d7b --- /dev/null +++ b/arch/arm/boot/dts/meson8m2.dtsi @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR MIT) +/* + * Copyright (c) 2017 Martin Blumenstingl . + */ + +#include "meson8.dtsi" + +/ { + model = "Amlogic Meson8m2 SoC"; + compatible = "amlogic,meson8m2"; +}; /* end of / */ + +&clkc { + compatible = "amlogic,meson8m2-clkc", "amlogic,meson8-clkc"; +}; + +ðmac { + compatible = "amlogic,meson8m2-dwmac", "snps,dwmac"; + reg = <0xc9410000 0x10000 + 0xc1108140 0x8>; + clocks = <&clkc CLKID_ETH>, + <&clkc CLKID_MPLL2>, + <&clkc CLKID_MPLL2>; + clock-names = "stmmaceth", "clkin0", "clkin1"; + resets = <&reset RESET_ETHERNET>; + reset-names = "stmmaceth"; +}; + +&pinctrl_aobus { + compatible = "amlogic,meson8m2-aobus-pinctrl", + "amlogic,meson8-aobus-pinctrl"; +}; + +&pinctrl_cbus { + compatible = "amlogic,meson8m2-cbus-pinctrl", + "amlogic,meson8-cbus-pinctrl"; + + eth_rgmii_pins: ethernet { + mux { + groups = "eth_tx_clk_50m", "eth_tx_en", + "eth_txd3", "eth_txd2", + "eth_txd1", "eth_txd0", + "eth_rx_clk_in", "eth_rx_dv", + "eth_rxd3", "eth_rxd2", + "eth_rxd1", "eth_rxd0", + "eth_mdio", "eth_mdc"; + function = "ethernet"; + }; + }; +}; + +&wdt { + compatible = "amlogic,meson8m2-wdt", "amlogic,meson8b-wdt"; +}; diff --git a/arch/arm/boot/dts/mt2701-evb.dts b/arch/arm/boot/dts/mt2701-evb.dts index 63af4b13a36f..be0edb3dae6c 100644 --- a/arch/arm/boot/dts/mt2701-evb.dts +++ b/arch/arm/boot/dts/mt2701-evb.dts @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2015 MediaTek Inc. * Author: Erin Lo * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ /dts-v1/; diff --git a/arch/arm/boot/dts/mt2701.dtsi b/arch/arm/boot/dts/mt2701.dtsi index 05557fce0f1d..180377e56ef4 100644 --- a/arch/arm/boot/dts/mt2701.dtsi +++ b/arch/arm/boot/dts/mt2701.dtsi @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2015 MediaTek Inc. * Author: Erin.Lo * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include @@ -426,104 +419,96 @@ status = "disabled"; }; - afe: audio-controller@11220000 { - compatible = "mediatek,mt2701-audio"; - reg = <0 0x11220000 0 0x2000>, - <0 0x112a0000 0 0x20000>; - interrupts = , - ; - interrupt-names = "afe", "asys"; - power-domains = <&scpsys MT2701_POWER_DOMAIN_IFR_MSC>; - - clocks = <&infracfg CLK_INFRA_AUDIO>, - <&topckgen CLK_TOP_AUD_MUX1_SEL>, - <&topckgen CLK_TOP_AUD_MUX2_SEL>, - <&topckgen CLK_TOP_AUD_MUX1_DIV>, - <&topckgen CLK_TOP_AUD_MUX2_DIV>, - <&topckgen CLK_TOP_AUD_48K_TIMING>, - <&topckgen CLK_TOP_AUD_44K_TIMING>, - <&topckgen CLK_TOP_AUDPLL_MUX_SEL>, - <&topckgen CLK_TOP_APLL_SEL>, - <&topckgen CLK_TOP_AUD1PLL_98M>, - <&topckgen CLK_TOP_AUD2PLL_90M>, - <&topckgen CLK_TOP_HADDS2PLL_98M>, - <&topckgen CLK_TOP_HADDS2PLL_294M>, - <&topckgen CLK_TOP_AUDPLL>, - <&topckgen CLK_TOP_AUDPLL_D4>, - <&topckgen CLK_TOP_AUDPLL_D8>, - <&topckgen CLK_TOP_AUDPLL_D16>, - <&topckgen CLK_TOP_AUDPLL_D24>, - <&topckgen CLK_TOP_AUDINTBUS_SEL>, - <&clk26m>, - <&topckgen CLK_TOP_SYSPLL1_D4>, - <&topckgen CLK_TOP_AUD_K1_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K2_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K3_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K4_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K5_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K6_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K1_SRC_DIV>, - <&topckgen CLK_TOP_AUD_K2_SRC_DIV>, - <&topckgen CLK_TOP_AUD_K3_SRC_DIV>, - <&topckgen CLK_TOP_AUD_K4_SRC_DIV>, - <&topckgen CLK_TOP_AUD_K5_SRC_DIV>, - <&topckgen CLK_TOP_AUD_K6_SRC_DIV>, - <&topckgen CLK_TOP_AUD_I2S1_MCLK>, - <&topckgen CLK_TOP_AUD_I2S2_MCLK>, - <&topckgen CLK_TOP_AUD_I2S3_MCLK>, - <&topckgen CLK_TOP_AUD_I2S4_MCLK>, - <&topckgen CLK_TOP_AUD_I2S5_MCLK>, - <&topckgen CLK_TOP_AUD_I2S6_MCLK>, - <&topckgen CLK_TOP_ASM_M_SEL>, - <&topckgen CLK_TOP_ASM_H_SEL>, - <&topckgen CLK_TOP_UNIVPLL2_D4>, - <&topckgen CLK_TOP_UNIVPLL2_D2>, - <&topckgen CLK_TOP_SYSPLL_D5>; - - clock-names = "infra_sys_audio_clk", - "top_audio_mux1_sel", - "top_audio_mux2_sel", - "top_audio_mux1_div", - "top_audio_mux2_div", - "top_audio_48k_timing", - "top_audio_44k_timing", - "top_audpll_mux_sel", - "top_apll_sel", - "top_aud1_pll_98M", - "top_aud2_pll_90M", - "top_hadds2_pll_98M", - "top_hadds2_pll_294M", - "top_audpll", - "top_audpll_d4", - "top_audpll_d8", - "top_audpll_d16", - "top_audpll_d24", - "top_audintbus_sel", - "clk_26m", - "top_syspll1_d4", - "top_aud_k1_src_sel", - "top_aud_k2_src_sel", - "top_aud_k3_src_sel", - "top_aud_k4_src_sel", - "top_aud_k5_src_sel", - "top_aud_k6_src_sel", - "top_aud_k1_src_div", - "top_aud_k2_src_div", - "top_aud_k3_src_div", - "top_aud_k4_src_div", - "top_aud_k5_src_div", - "top_aud_k6_src_div", - "top_aud_i2s1_mclk", - "top_aud_i2s2_mclk", - "top_aud_i2s3_mclk", - "top_aud_i2s4_mclk", - "top_aud_i2s5_mclk", - "top_aud_i2s6_mclk", - "top_asm_m_sel", - "top_asm_h_sel", - "top_univpll2_d4", - "top_univpll2_d2", - "top_syspll_d5"; + audsys: clock-controller@11220000 { + compatible = "mediatek,mt2701-audsys", "syscon"; + reg = <0 0x11220000 0 0x2000>; + #clock-cells = <1>; + + afe: audio-controller { + compatible = "mediatek,mt2701-audio"; + interrupts = , + ; + interrupt-names = "afe", "asys"; + power-domains = <&scpsys MT2701_POWER_DOMAIN_IFR_MSC>; + + clocks = <&infracfg CLK_INFRA_AUDIO>, + <&topckgen CLK_TOP_AUD_MUX1_SEL>, + <&topckgen CLK_TOP_AUD_MUX2_SEL>, + <&topckgen CLK_TOP_AUD_48K_TIMING>, + <&topckgen CLK_TOP_AUD_44K_TIMING>, + <&topckgen CLK_TOP_AUD_K1_SRC_SEL>, + <&topckgen CLK_TOP_AUD_K2_SRC_SEL>, + <&topckgen CLK_TOP_AUD_K3_SRC_SEL>, + <&topckgen CLK_TOP_AUD_K4_SRC_SEL>, + <&topckgen CLK_TOP_AUD_K1_SRC_DIV>, + <&topckgen CLK_TOP_AUD_K2_SRC_DIV>, + <&topckgen CLK_TOP_AUD_K3_SRC_DIV>, + <&topckgen CLK_TOP_AUD_K4_SRC_DIV>, + <&topckgen CLK_TOP_AUD_I2S1_MCLK>, + <&topckgen CLK_TOP_AUD_I2S2_MCLK>, + <&topckgen CLK_TOP_AUD_I2S3_MCLK>, + <&topckgen CLK_TOP_AUD_I2S4_MCLK>, + <&audsys CLK_AUD_I2SO1>, + <&audsys CLK_AUD_I2SO2>, + <&audsys CLK_AUD_I2SO3>, + <&audsys CLK_AUD_I2SO4>, + <&audsys CLK_AUD_I2SIN1>, + <&audsys CLK_AUD_I2SIN2>, + <&audsys CLK_AUD_I2SIN3>, + <&audsys CLK_AUD_I2SIN4>, + <&audsys CLK_AUD_ASRCO1>, + <&audsys CLK_AUD_ASRCO2>, + <&audsys CLK_AUD_ASRCO3>, + <&audsys CLK_AUD_ASRCO4>, + <&audsys CLK_AUD_AFE>, + <&audsys CLK_AUD_AFE_CONN>, + <&audsys CLK_AUD_A1SYS>, + <&audsys CLK_AUD_A2SYS>, + <&audsys CLK_AUD_AFE_MRGIF>; + + clock-names = "infra_sys_audio_clk", + "top_audio_mux1_sel", + "top_audio_mux2_sel", + "top_audio_a1sys_hp", + "top_audio_a2sys_hp", + "i2s0_src_sel", + "i2s1_src_sel", + "i2s2_src_sel", + "i2s3_src_sel", + "i2s0_src_div", + "i2s1_src_div", + "i2s2_src_div", + "i2s3_src_div", + "i2s0_mclk_en", + "i2s1_mclk_en", + "i2s2_mclk_en", + "i2s3_mclk_en", + "i2so0_hop_ck", + "i2so1_hop_ck", + "i2so2_hop_ck", + "i2so3_hop_ck", + "i2si0_hop_ck", + "i2si1_hop_ck", + "i2si2_hop_ck", + "i2si3_hop_ck", + "asrc0_out_ck", + "asrc1_out_ck", + "asrc2_out_ck", + "asrc3_out_ck", + "audio_afe_pd", + "audio_afe_conn_pd", + "audio_a1sys_pd", + "audio_a2sys_pd", + "audio_mrgif_pd"; + + assigned-clocks = <&topckgen CLK_TOP_AUD_MUX1_SEL>, + <&topckgen CLK_TOP_AUD_MUX2_SEL>, + <&topckgen CLK_TOP_AUD_MUX1_DIV>, + <&topckgen CLK_TOP_AUD_MUX2_DIV>; + assigned-clock-parents = <&topckgen CLK_TOP_AUD1PLL_98M>, + <&topckgen CLK_TOP_AUD2PLL_90M>; + assigned-clock-rates = <0>, <0>, <49152000>, <45158400>; + }; }; mmsys: syscon@14000000 { diff --git a/arch/arm/boot/dts/mt6323.dtsi b/arch/arm/boot/dts/mt6323.dtsi index 7c783d6c750e..ba397407c1dd 100644 --- a/arch/arm/boot/dts/mt6323.dtsi +++ b/arch/arm/boot/dts/mt6323.dtsi @@ -1,15 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* - * Copyright (c) 2017 MediaTek Inc. + * Copyright (c) 2017-2018 MediaTek Inc. * Author: John Crispin * Sean Wang - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ &pwrap { @@ -20,6 +14,13 @@ interrupt-controller; #interrupt-cells = <2>; + mt6323_leds: leds { + compatible = "mediatek,mt6323-led"; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + mt6323regulator: mt6323regulator{ compatible = "mediatek,mt6323-regulator"; diff --git a/arch/arm/boot/dts/mt6580-evbp1.dts b/arch/arm/boot/dts/mt6580-evbp1.dts index 17daeae6bbe8..ca137897ed60 100644 --- a/arch/arm/boot/dts/mt6580-evbp1.dts +++ b/arch/arm/boot/dts/mt6580-evbp1.dts @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2015 MediaTek Inc. * Author: Mars.C * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ /dts-v1/; diff --git a/arch/arm/boot/dts/mt6580.dtsi b/arch/arm/boot/dts/mt6580.dtsi index a349dba5ff79..2bdc5ed12fca 100644 --- a/arch/arm/boot/dts/mt6580.dtsi +++ b/arch/arm/boot/dts/mt6580.dtsi @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2015 MediaTek Inc. * Author: Mars.C * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/arch/arm/boot/dts/mt6589-aquaris5.dts b/arch/arm/boot/dts/mt6589-aquaris5.dts index 594a6f3bebda..7bbaa1279a26 100644 --- a/arch/arm/boot/dts/mt6589-aquaris5.dts +++ b/arch/arm/boot/dts/mt6589-aquaris5.dts @@ -1,16 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * Copyright (c) 2014 MundoReader S.L. * Author: Matthias Brugger * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ /dts-v1/; diff --git a/arch/arm/boot/dts/mt6589.dtsi b/arch/arm/boot/dts/mt6589.dtsi index 41df742d7891..28df8495686a 100644 --- a/arch/arm/boot/dts/mt6589.dtsi +++ b/arch/arm/boot/dts/mt6589.dtsi @@ -1,17 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * Copyright (c) 2014 MundoReader S.L. * Author: Matthias Brugger * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - */ +*/ #include #include diff --git a/arch/arm/boot/dts/mt6592-evb.dts b/arch/arm/boot/dts/mt6592-evb.dts index b57237e6394a..02849f6548e3 100644 --- a/arch/arm/boot/dts/mt6592-evb.dts +++ b/arch/arm/boot/dts/mt6592-evb.dts @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2014 MediaTek Inc. * Author: Howard Chen * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ /dts-v1/; diff --git a/arch/arm/boot/dts/mt6592.dtsi b/arch/arm/boot/dts/mt6592.dtsi index c69201ffff72..8696ac891d60 100644 --- a/arch/arm/boot/dts/mt6592.dtsi +++ b/arch/arm/boot/dts/mt6592.dtsi @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2014 MediaTek Inc. * Author: Howard Chen * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/arch/arm/boot/dts/mt7623.dtsi b/arch/arm/boot/dts/mt7623.dtsi index e10c03496524..d1eb123bc73b 100644 --- a/arch/arm/boot/dts/mt7623.dtsi +++ b/arch/arm/boot/dts/mt7623.dtsi @@ -1,16 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* - * Copyright (c) 2017 MediaTek Inc. + * Copyright (c) 2017-2018 MediaTek Inc. * Author: John Crispin * Sean Wang * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include @@ -22,11 +15,12 @@ #include #include #include -#include "skeleton64.dtsi" / { compatible = "mediatek,mt7623"; interrupt-parent = <&sysirq>; + #address-cells = <2>; + #size-cells = <2>; cpu_opp_table: opp-table { compatible = "operating-points-v2"; @@ -130,14 +124,14 @@ #clock-cells = <0>; }; - rtc32k: oscillator@1 { + rtc32k: oscillator-1 { compatible = "fixed-clock"; #clock-cells = <0>; clock-frequency = <32000>; clock-output-names = "rtc32k"; }; - clk26m: oscillator@0 { + clk26m: oscillator-0 { compatible = "fixed-clock"; #clock-cells = <0>; clock-frequency = <26000000>; @@ -492,6 +486,18 @@ nvmem-cell-names = "calibration-data"; }; + btif: serial@1100c000 { + compatible = "mediatek,mt7623-btif", + "mediatek,mtk-btif"; + reg = <0 0x1100c000 0 0x1000>; + interrupts = ; + clocks = <&pericfg CLK_PERI_BTIF>; + clock-names = "main"; + reg-shift = <2>; + reg-io-width = <4>; + status = "disabled"; + }; + nandc: nfi@1100d000 { compatible = "mediatek,mt7623-nfc", "mediatek,mt2701-nfc"; @@ -517,6 +523,18 @@ status = "disabled"; }; + nor_flash: spi@11014000 { + compatible = "mediatek,mt7623-nor", + "mediatek,mt8173-nor"; + reg = <0 0x11014000 0 0x1000>; + clocks = <&pericfg CLK_PERI_FLASH>, + <&topckgen CLK_TOP_FLASH_SEL>; + clock-names = "spi", "sf"; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + spi1: spi@11016000 { compatible = "mediatek,mt7623-spi", "mediatek,mt2701-spi"; @@ -545,105 +563,99 @@ status = "disabled"; }; - afe: audio-controller@11220000 { - compatible = "mediatek,mt7623-audio", - "mediatek,mt2701-audio"; - reg = <0 0x11220000 0 0x2000>, - <0 0x112a0000 0 0x20000>; - interrupts = , - ; - interrupt-names = "afe", "asys"; - power-domains = <&scpsys MT2701_POWER_DOMAIN_IFR_MSC>; + audsys: clock-controller@11220000 { + compatible = "mediatek,mt7623-audsys", + "mediatek,mt2701-audsys", + "syscon"; + reg = <0 0x11220000 0 0x2000>; + #clock-cells = <1>; - clocks = <&infracfg CLK_INFRA_AUDIO>, - <&topckgen CLK_TOP_AUD_MUX1_SEL>, - <&topckgen CLK_TOP_AUD_MUX2_SEL>, - <&topckgen CLK_TOP_AUD_MUX1_DIV>, - <&topckgen CLK_TOP_AUD_MUX2_DIV>, - <&topckgen CLK_TOP_AUD_48K_TIMING>, - <&topckgen CLK_TOP_AUD_44K_TIMING>, - <&topckgen CLK_TOP_AUDPLL_MUX_SEL>, - <&topckgen CLK_TOP_APLL_SEL>, - <&topckgen CLK_TOP_AUD1PLL_98M>, - <&topckgen CLK_TOP_AUD2PLL_90M>, - <&topckgen CLK_TOP_HADDS2PLL_98M>, - <&topckgen CLK_TOP_HADDS2PLL_294M>, - <&topckgen CLK_TOP_AUDPLL>, - <&topckgen CLK_TOP_AUDPLL_D4>, - <&topckgen CLK_TOP_AUDPLL_D8>, - <&topckgen CLK_TOP_AUDPLL_D16>, - <&topckgen CLK_TOP_AUDPLL_D24>, - <&topckgen CLK_TOP_AUDINTBUS_SEL>, - <&clk26m>, - <&topckgen CLK_TOP_SYSPLL1_D4>, - <&topckgen CLK_TOP_AUD_K1_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K2_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K3_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K4_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K5_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K6_SRC_SEL>, - <&topckgen CLK_TOP_AUD_K1_SRC_DIV>, - <&topckgen CLK_TOP_AUD_K2_SRC_DIV>, - <&topckgen CLK_TOP_AUD_K3_SRC_DIV>, - <&topckgen CLK_TOP_AUD_K4_SRC_DIV>, - <&topckgen CLK_TOP_AUD_K5_SRC_DIV>, - <&topckgen CLK_TOP_AUD_K6_SRC_DIV>, - <&topckgen CLK_TOP_AUD_I2S1_MCLK>, - <&topckgen CLK_TOP_AUD_I2S2_MCLK>, - <&topckgen CLK_TOP_AUD_I2S3_MCLK>, - <&topckgen CLK_TOP_AUD_I2S4_MCLK>, - <&topckgen CLK_TOP_AUD_I2S5_MCLK>, - <&topckgen CLK_TOP_AUD_I2S6_MCLK>, - <&topckgen CLK_TOP_ASM_M_SEL>, - <&topckgen CLK_TOP_ASM_H_SEL>, - <&topckgen CLK_TOP_UNIVPLL2_D4>, - <&topckgen CLK_TOP_UNIVPLL2_D2>, - <&topckgen CLK_TOP_SYSPLL_D5>; - - clock-names = "infra_sys_audio_clk", - "top_audio_mux1_sel", - "top_audio_mux2_sel", - "top_audio_mux1_div", - "top_audio_mux2_div", - "top_audio_48k_timing", - "top_audio_44k_timing", - "top_audpll_mux_sel", - "top_apll_sel", - "top_aud1_pll_98M", - "top_aud2_pll_90M", - "top_hadds2_pll_98M", - "top_hadds2_pll_294M", - "top_audpll", - "top_audpll_d4", - "top_audpll_d8", - "top_audpll_d16", - "top_audpll_d24", - "top_audintbus_sel", - "clk_26m", - "top_syspll1_d4", - "top_aud_k1_src_sel", - "top_aud_k2_src_sel", - "top_aud_k3_src_sel", - "top_aud_k4_src_sel", - "top_aud_k5_src_sel", - "top_aud_k6_src_sel", - "top_aud_k1_src_div", - "top_aud_k2_src_div", - "top_aud_k3_src_div", - "top_aud_k4_src_div", - "top_aud_k5_src_div", - "top_aud_k6_src_div", - "top_aud_i2s1_mclk", - "top_aud_i2s2_mclk", - "top_aud_i2s3_mclk", - "top_aud_i2s4_mclk", - "top_aud_i2s5_mclk", - "top_aud_i2s6_mclk", - "top_asm_m_sel", - "top_asm_h_sel", - "top_univpll2_d4", - "top_univpll2_d2", - "top_syspll_d5"; + afe: audio-controller { + compatible = "mediatek,mt7623-audio", + "mediatek,mt2701-audio"; + interrupts = , + ; + interrupt-names = "afe", "asys"; + power-domains = <&scpsys MT2701_POWER_DOMAIN_IFR_MSC>; + + clocks = <&infracfg CLK_INFRA_AUDIO>, + <&topckgen CLK_TOP_AUD_MUX1_SEL>, + <&topckgen CLK_TOP_AUD_MUX2_SEL>, + <&topckgen CLK_TOP_AUD_48K_TIMING>, + <&topckgen CLK_TOP_AUD_44K_TIMING>, + <&topckgen CLK_TOP_AUD_K1_SRC_SEL>, + <&topckgen CLK_TOP_AUD_K2_SRC_SEL>, + <&topckgen CLK_TOP_AUD_K3_SRC_SEL>, + <&topckgen CLK_TOP_AUD_K4_SRC_SEL>, + <&topckgen CLK_TOP_AUD_K1_SRC_DIV>, + <&topckgen CLK_TOP_AUD_K2_SRC_DIV>, + <&topckgen CLK_TOP_AUD_K3_SRC_DIV>, + <&topckgen CLK_TOP_AUD_K4_SRC_DIV>, + <&topckgen CLK_TOP_AUD_I2S1_MCLK>, + <&topckgen CLK_TOP_AUD_I2S2_MCLK>, + <&topckgen CLK_TOP_AUD_I2S3_MCLK>, + <&topckgen CLK_TOP_AUD_I2S4_MCLK>, + <&audsys CLK_AUD_I2SO1>, + <&audsys CLK_AUD_I2SO2>, + <&audsys CLK_AUD_I2SO3>, + <&audsys CLK_AUD_I2SO4>, + <&audsys CLK_AUD_I2SIN1>, + <&audsys CLK_AUD_I2SIN2>, + <&audsys CLK_AUD_I2SIN3>, + <&audsys CLK_AUD_I2SIN4>, + <&audsys CLK_AUD_ASRCO1>, + <&audsys CLK_AUD_ASRCO2>, + <&audsys CLK_AUD_ASRCO3>, + <&audsys CLK_AUD_ASRCO4>, + <&audsys CLK_AUD_AFE>, + <&audsys CLK_AUD_AFE_CONN>, + <&audsys CLK_AUD_A1SYS>, + <&audsys CLK_AUD_A2SYS>, + <&audsys CLK_AUD_AFE_MRGIF>; + + clock-names = "infra_sys_audio_clk", + "top_audio_mux1_sel", + "top_audio_mux2_sel", + "top_audio_a1sys_hp", + "top_audio_a2sys_hp", + "i2s0_src_sel", + "i2s1_src_sel", + "i2s2_src_sel", + "i2s3_src_sel", + "i2s0_src_div", + "i2s1_src_div", + "i2s2_src_div", + "i2s3_src_div", + "i2s0_mclk_en", + "i2s1_mclk_en", + "i2s2_mclk_en", + "i2s3_mclk_en", + "i2so0_hop_ck", + "i2so1_hop_ck", + "i2so2_hop_ck", + "i2so3_hop_ck", + "i2si0_hop_ck", + "i2si1_hop_ck", + "i2si2_hop_ck", + "i2si3_hop_ck", + "asrc0_out_ck", + "asrc1_out_ck", + "asrc2_out_ck", + "asrc3_out_ck", + "audio_afe_pd", + "audio_afe_conn_pd", + "audio_a1sys_pd", + "audio_a2sys_pd", + "audio_mrgif_pd"; + + assigned-clocks = <&topckgen CLK_TOP_AUD_MUX1_SEL>, + <&topckgen CLK_TOP_AUD_MUX2_SEL>, + <&topckgen CLK_TOP_AUD_MUX1_DIV>, + <&topckgen CLK_TOP_AUD_MUX2_DIV>; + assigned-clock-parents = <&topckgen CLK_TOP_AUD1PLL_98M>, + <&topckgen CLK_TOP_AUD2PLL_90M>; + assigned-clock-rates = <0>, <0>, <49152000>, <45158400>; + }; }; mmc0: mmc@11230000 { @@ -873,6 +885,16 @@ #reset-cells = <1>; }; + hsdma: dma-controller@1b007000 { + compatible = "mediatek,mt7623-hsdma"; + reg = <0 0x1b007000 0 0x1000>; + interrupts = ; + clocks = <ðsys CLK_ETHSYS_HSDMA>; + clock-names = "hsdma"; + power-domains = <&scpsys MT2701_POWER_DOMAIN_ETH>; + #dma-cells = <1>; + }; + eth: ethernet@1b100000 { compatible = "mediatek,mt7623-eth", "mediatek,mt2701-eth", @@ -913,3 +935,298 @@ status = "disabled"; }; }; + +&pio { + cir_pins_a:cir-default { + pins-cir { + pinmux = ; + bias-disable; + }; + }; + + i2c0_pins_a: i2c0-default { + pins-i2c0 { + pinmux = , + ; + bias-disable; + }; + }; + + i2c1_pins_a: i2c1-default { + pin-i2c1 { + pinmux = , + ; + bias-disable; + }; + }; + + i2c1_pins_b: i2c1-alt { + pin-i2c1 { + pinmux = , + ; + bias-disable; + }; + }; + + i2c2_pins_a: i2c2-default { + pin-i2c2 { + pinmux = , + ; + bias-disable; + }; + }; + + i2c2_pins_b: i2c2-alt { + pin-i2c2 { + pinmux = , + ; + bias-disable; + }; + }; + + i2s0_pins_a: i2s0-default { + pin-i2s0 { + pinmux = , + , + , + , + ; + drive-strength = ; + bias-pull-down; + }; + }; + + i2s1_pins_a: i2s1-default { + pin-i2s1 { + pinmux = , + , + , + , + ; + drive-strength = ; + bias-pull-down; + }; + }; + + key_pins_a: keys-alt { + pins-keys { + pinmux = , + ; + input-enable; + }; + }; + + led_pins_a: leds-alt { + pins-leds { + pinmux = , + , + ; + }; + }; + + mmc0_pins_default: mmc0default { + pins-cmd-dat { + pinmux = , + , + , + , + , + , + , + , + ; + input-enable; + bias-pull-up; + }; + + pins-clk { + pinmux = ; + bias-pull-down; + }; + + pins-rst { + pinmux = ; + bias-pull-up; + }; + }; + + mmc0_pins_uhs: mmc0 { + pins-cmd-dat { + pinmux = , + , + , + , + , + , + , + , + ; + input-enable; + drive-strength = ; + bias-pull-up = ; + }; + + pins-clk { + pinmux = ; + drive-strength = ; + bias-pull-down = ; + }; + + pins-rst { + pinmux = ; + bias-pull-up; + }; + }; + + mmc1_pins_default: mmc1default { + pins-cmd-dat { + pinmux = , + , + , + , + ; + input-enable; + drive-strength = ; + bias-pull-up = ; + }; + + pins-clk { + pinmux = ; + bias-pull-down; + drive-strength = ; + }; + + pins-wp { + pinmux = ; + input-enable; + bias-pull-up; + }; + + pins-insert { + pinmux = ; + bias-pull-up; + }; + }; + + mmc1_pins_uhs: mmc1 { + pins-cmd-dat { + pinmux = , + , + , + , + ; + input-enable; + drive-strength = ; + bias-pull-up = ; + }; + + pins-clk { + pinmux = ; + drive-strength = ; + bias-pull-down = ; + }; + }; + + nand_pins_default: nanddefault { + pins-ale { + pinmux = ; + drive-strength = ; + bias-pull-down = ; + }; + + pins-dat { + pinmux = , + , + , + , + , + , + , + , + ; + input-enable; + drive-strength = ; + bias-pull-up; + }; + + pins-we { + pinmux = ; + drive-strength = ; + bias-pull-up = ; + }; + }; + + pcie_default: pcie_pin_default { + pins_cmd_dat { + pinmux = , + ; + bias-disable; + }; + }; + + pwm_pins_a: pwm-default { + pins-pwm { + pinmux = , + , + , + , + ; + }; + }; + + spi0_pins_a: spi0-default { + pins-spi { + pinmux = , + , + , + ; + bias-disable; + }; + }; + + spi1_pins_a: spi1-default { + pins-spi { + pinmux = , + , + , + ; + }; + }; + + spi2_pins_a: spi2-default { + pins-spi { + pinmux = , + , + , + ; + }; + }; + + uart0_pins_a: uart0-default { + pins-dat { + pinmux = , + ; + }; + }; + + uart1_pins_a: uart1-default { + pins-dat { + pinmux = , + ; + }; + }; + + uart2_pins_a: uart2-default { + pins-dat { + pinmux = , + ; + }; + }; + + uart2_pins_b: uart2-alt { + pins-dat { + pinmux = , + ; + }; + }; +}; diff --git a/arch/arm/boot/dts/mt7623a-rfb-emmc.dts b/arch/arm/boot/dts/mt7623a-rfb-emmc.dts new file mode 100644 index 000000000000..13c86936d1c8 --- /dev/null +++ b/arch/arm/boot/dts/mt7623a-rfb-emmc.dts @@ -0,0 +1,291 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2017-2018 MediaTek Inc. + * Author: Sean Wang + * + */ + +/dts-v1/; +#include +#include "mt7623a.dtsi" +#include "mt6323.dtsi" + +/ { + model = "MediaTek MT7623A with eMMC reference board"; + compatible = "mediatek,mt7623a-rfb-emmc", "mediatek,mt7623"; + + aliases { + serial2 = &uart2; + }; + + chosen { + stdout-path = "serial2:115200n8"; + }; + + cpus { + cpu@0 { + proc-supply = <&mt6323_vproc_reg>; + }; + + cpu@1 { + proc-supply = <&mt6323_vproc_reg>; + }; + + cpu@2 { + proc-supply = <&mt6323_vproc_reg>; + }; + + cpu@3 { + proc-supply = <&mt6323_vproc_reg>; + }; + }; + + gpio-keys { + compatible = "gpio-keys"; + pinctrl-names = "default"; + pinctrl-0 = <&key_pins_a>; + + factory { + label = "factory"; + linux,code = ; + gpios = <&pio 256 GPIO_ACTIVE_LOW>; + }; + + wps { + label = "wps"; + linux,code = ; + gpios = <&pio 257 GPIO_ACTIVE_HIGH>; + }; + }; + + memory@80000000 { + device_type = "memory"; + reg = <0 0x80000000 0 0x20000000>; + }; + + reg_1p8v: regulator-1p8v { + compatible = "regulator-fixed"; + regulator-name = "fixed-1.8V"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-boot-on; + regulator-always-on; + }; + + reg_3p3v: regulator-3p3v { + compatible = "regulator-fixed"; + regulator-name = "fixed-3.3V"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-boot-on; + regulator-always-on; + }; + + reg_5v: regulator-5v { + compatible = "regulator-fixed"; + regulator-name = "fixed-5V"; + regulator-min-microvolt = <5000000>; + regulator-max-microvolt = <5000000>; + regulator-boot-on; + regulator-always-on; + }; + + sound { + compatible = "mediatek,mt2701-wm8960-machine"; + mediatek,platform = <&afe>; + audio-routing = + "Headphone", "HP_L", + "Headphone", "HP_R", + "LINPUT1", "AMIC", + "RINPUT1", "AMIC"; + mediatek,audio-codec = <&wm8960>; + pinctrl-names = "default"; + pinctrl-0 = <&i2s0_pins_a>; + }; +}; + +&btif { + status = "okay"; +}; + +&crypto { + status = "okay"; +}; + +ð { + status = "okay"; + + gmac0: mac@0 { + compatible = "mediatek,eth-mac"; + reg = <0>; + phy-mode = "trgmii"; + + fixed-link { + speed = <1000>; + full-duplex; + pause; + }; + }; + + mdio-bus { + #address-cells = <1>; + #size-cells = <0>; + + switch@0 { + compatible = "mediatek,mt7530"; + reg = <0>; + mediatek,mcm; + resets = <ðsys MT2701_ETHSYS_MCM_RST>; + reset-names = "mcm"; + core-supply = <&mt6323_vpa_reg>; + io-supply = <&mt6323_vemc3v3_reg>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + label = "lan0"; + }; + + port@1 { + reg = <1>; + label = "lan1"; + }; + + port@2 { + reg = <2>; + label = "lan2"; + }; + + port@3 { + reg = <3>; + label = "lan3"; + }; + + port@4 { + reg = <4>; + label = "wan"; + }; + + port@6 { + reg = <6>; + label = "cpu"; + ethernet = <&gmac0>; + phy-mode = "trgmii"; + + fixed-link { + speed = <1000>; + full-duplex; + }; + }; + }; + }; + }; +}; + +&i2c0 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c0_pins_a>; + status = "okay"; +}; + +&i2c1 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c1_pins_b>; + status = "okay"; + + wm8960: wm8960@1a { + compatible = "wlf,wm8960"; + reg = <0x1a>; + }; +}; + +&i2c2 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c2_pins_b>; + status = "okay"; +}; + +&mmc0 { + pinctrl-names = "default", "state_uhs"; + pinctrl-0 = <&mmc0_pins_default>; + pinctrl-1 = <&mmc0_pins_uhs>; + status = "okay"; + bus-width = <8>; + max-frequency = <50000000>; + cap-mmc-highspeed; + vmmc-supply = <®_3p3v>; + vqmmc-supply = <®_1p8v>; + non-removable; +}; + +&mmc1 { + pinctrl-names = "default", "state_uhs"; + pinctrl-0 = <&mmc1_pins_default>; + pinctrl-1 = <&mmc1_pins_uhs>; + status = "okay"; + bus-width = <4>; + max-frequency = <50000000>; + cap-sd-highspeed; + cd-gpios = <&pio 261 GPIO_ACTIVE_LOW>; + vmmc-supply = <®_3p3v>; + vqmmc-supply = <®_3p3v>; +}; + +&pcie { + pinctrl-names = "default"; + pinctrl-0 = <&pcie_default>; + status = "okay"; + + pcie@0,0 { + status = "okay"; + }; + + pcie@1,0 { + status = "okay"; + }; +}; + +&pcie0_phy { + status = "okay"; +}; + +&pcie1_phy { + status = "okay"; +}; + +&pwm { + pinctrl-names = "default"; + pinctrl-0 = <&pwm_pins_a>; + status = "okay"; +}; + +&spi0 { + pinctrl-names = "default"; + pinctrl-0 = <&spi0_pins_a>; + status = "okay"; +}; + +&spi1 { + pinctrl-names = "default"; + pinctrl-0 = <&spi1_pins_a>; + status = "okay"; +}; + +&uart2 { + pinctrl-names = "default"; + pinctrl-0 = <&uart2_pins_b>; + status = "okay"; +}; + +&usb1 { + vusb33-supply = <®_3p3v>; + vbus-supply = <®_5v>; + status = "okay"; +}; + +&u3phy1 { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/mt7623a-rfb-nand.dts b/arch/arm/boot/dts/mt7623a-rfb-nand.dts new file mode 100644 index 000000000000..88d8f0b2f4c2 --- /dev/null +++ b/arch/arm/boot/dts/mt7623a-rfb-nand.dts @@ -0,0 +1,337 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2017-2018 MediaTek Inc. + * Author: Sean Wang + * + */ + +/dts-v1/; +#include +#include "mt7623a.dtsi" +#include "mt6323.dtsi" + +/ { + model = "MediaTek MT7623A with NAND reference board"; + compatible = "mediatek,mt7623a-rfb-nand", "mediatek,mt7623"; + + aliases { + serial2 = &uart2; + }; + + chosen { + stdout-path = "serial2:115200n8"; + }; + + cpus { + cpu@0 { + proc-supply = <&mt6323_vproc_reg>; + }; + + cpu@1 { + proc-supply = <&mt6323_vproc_reg>; + }; + + cpu@2 { + proc-supply = <&mt6323_vproc_reg>; + }; + + cpu@3 { + proc-supply = <&mt6323_vproc_reg>; + }; + }; + + gpio-keys { + compatible = "gpio-keys"; + pinctrl-names = "default"; + pinctrl-0 = <&key_pins_a>; + + factory { + label = "factory"; + linux,code = ; + gpios = <&pio 256 GPIO_ACTIVE_LOW>; + }; + + wps { + label = "wps"; + linux,code = ; + gpios = <&pio 257 GPIO_ACTIVE_HIGH>; + }; + }; + + memory@80000000 { + device_type = "memory"; + reg = <0 0x80000000 0 0x20000000>; + }; + + reg_1p8v: regulator-1p8v { + compatible = "regulator-fixed"; + regulator-name = "fixed-1.8V"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-boot-on; + regulator-always-on; + }; + + reg_3p3v: regulator-3p3v { + compatible = "regulator-fixed"; + regulator-name = "fixed-3.3V"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-boot-on; + regulator-always-on; + }; + + reg_5v: regulator-5v { + compatible = "regulator-fixed"; + regulator-name = "fixed-5V"; + regulator-min-microvolt = <5000000>; + regulator-max-microvolt = <5000000>; + regulator-boot-on; + regulator-always-on; + }; + + sound { + compatible = "mediatek,mt2701-wm8960-machine"; + mediatek,platform = <&afe>; + audio-routing = + "Headphone", "HP_L", + "Headphone", "HP_R", + "LINPUT1", "AMIC", + "RINPUT1", "AMIC"; + mediatek,audio-codec = <&wm8960>; + pinctrl-names = "default"; + pinctrl-0 = <&i2s0_pins_a>; + }; +}; + +&bch { + status = "okay"; +}; + +&btif { + status = "okay"; +}; + +&crypto { + status = "okay"; +}; + +ð { + status = "okay"; + + gmac0: mac@0 { + compatible = "mediatek,eth-mac"; + reg = <0>; + phy-mode = "trgmii"; + + fixed-link { + speed = <1000>; + full-duplex; + pause; + }; + }; + + mdio-bus { + #address-cells = <1>; + #size-cells = <0>; + + switch@0 { + compatible = "mediatek,mt7530"; + reg = <0>; + mediatek,mcm; + resets = <ðsys MT2701_ETHSYS_MCM_RST>; + reset-names = "mcm"; + core-supply = <&mt6323_vpa_reg>; + io-supply = <&mt6323_vemc3v3_reg>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + label = "lan0"; + }; + + port@1 { + reg = <1>; + label = "lan1"; + }; + + port@2 { + reg = <2>; + label = "lan2"; + }; + + port@3 { + reg = <3>; + label = "lan3"; + }; + + port@4 { + reg = <4>; + label = "wan"; + }; + + port@6 { + reg = <6>; + label = "cpu"; + ethernet = <&gmac0>; + phy-mode = "trgmii"; + + fixed-link { + speed = <1000>; + full-duplex; + }; + }; + }; + }; + }; +}; + +&i2c0 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c0_pins_a>; + status = "okay"; +}; + +&i2c1 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c1_pins_b>; + status = "okay"; + + wm8960: wm8960@1a { + compatible = "wlf,wm8960"; + reg = <0x1a>; + }; +}; + +&i2c2 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c2_pins_b>; + status = "okay"; +}; + +&mmc1 { + pinctrl-names = "default", "state_uhs"; + pinctrl-0 = <&mmc1_pins_default>; + pinctrl-1 = <&mmc1_pins_uhs>; + status = "okay"; + bus-width = <4>; + max-frequency = <50000000>; + cap-sd-highspeed; + cd-gpios = <&pio 261 GPIO_ACTIVE_LOW>; + vmmc-supply = <®_3p3v>; + vqmmc-supply = <®_3p3v>; +}; + +&nandc { + pinctrl-names = "default"; + pinctrl-0 = <&nand_pins_default>; + status = "okay"; + + nand@0 { + reg = <0>; + spare_per_sector = <64>; + nand-ecc-mode = "hw"; + nand-ecc-strength = <12>; + nand-ecc-step-size = <1024>; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "preloader"; + reg = <0x0 0x40000>; + }; + + partition@40000 { + label = "uboot"; + reg = <0x40000 0x80000>; + }; + + partition@c0000 { + label = "uboot-env"; + reg = <0xC0000 0x40000>; + }; + + partition@140000 { + label = "bootimg"; + reg = <0x140000 0x2000000>; + }; + + partition@2140000 { + label = "recovery"; + reg = <0x2140000 0x2000000>; + }; + + partition@4140000 { + label = "rootfs"; + reg = <0x4140000 0x1000000>; + }; + + partition@5140000 { + label = "usrdata"; + reg = <0x5140000 0x1000000>; + }; + }; + }; +}; + +&pcie { + pinctrl-names = "default"; + pinctrl-0 = <&pcie_default>; + status = "okay"; + + pcie@0,0 { + status = "okay"; + }; + + pcie@1,0 { + status = "okay"; + }; +}; + +&pcie0_phy { + status = "okay"; +}; + +&pcie1_phy { + status = "okay"; +}; + +&pwm { + pinctrl-names = "default"; + pinctrl-0 = <&pwm_pins_a>; + status = "okay"; +}; + +&spi0 { + pinctrl-names = "default"; + pinctrl-0 = <&spi0_pins_a>; + status = "okay"; +}; + +&spi1 { + pinctrl-names = "default"; + pinctrl-0 = <&spi1_pins_a>; + status = "okay"; +}; + +&uart2 { + pinctrl-names = "default"; + pinctrl-0 = <&uart2_pins_b>; + status = "okay"; +}; + +&usb1 { + vusb33-supply = <®_3p3v>; + vbus-supply = <®_5v>; + status = "okay"; +}; + +&u3phy1 { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/mt7623a.dtsi b/arch/arm/boot/dts/mt7623a.dtsi new file mode 100644 index 000000000000..0735a1fb8ad9 --- /dev/null +++ b/arch/arm/boot/dts/mt7623a.dtsi @@ -0,0 +1,44 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2017-2018 MediaTek Inc. + * Author: Sean Wang + * + */ + +/dts-v1/; +#include +#include "mt7623.dtsi" + +&afe { + power-domains = <&scpsys MT7623A_POWER_DOMAIN_IFR_MSC>; +}; + +&crypto { + power-domains = <&scpsys MT7623A_POWER_DOMAIN_ETH>; +}; + +ð { + power-domains = <&scpsys MT7623A_POWER_DOMAIN_ETH>; +}; + +&nandc { + power-domains = <&scpsys MT7623A_POWER_DOMAIN_IFR_MSC>; +}; + +&pcie { + power-domains = <&scpsys MT7623A_POWER_DOMAIN_HIF>; +}; + +&scpsys { + compatible = "mediatek,mt7623a-scpsys"; + clocks = <&topckgen CLK_TOP_ETHIF_SEL>; + clock-names = "ethif"; +}; + +&usb1 { + power-domains = <&scpsys MT7623A_POWER_DOMAIN_HIF>; +}; + +&usb2 { + power-domains = <&scpsys MT7623A_POWER_DOMAIN_HIF>; +}; diff --git a/arch/arm/boot/dts/mt7623n-bananapi-bpi-r2.dts b/arch/arm/boot/dts/mt7623n-bananapi-bpi-r2.dts index bbf56f855e46..531d905d924f 100644 --- a/arch/arm/boot/dts/mt7623n-bananapi-bpi-r2.dts +++ b/arch/arm/boot/dts/mt7623n-bananapi-bpi-r2.dts @@ -1,5 +1,5 @@ /* - * Copyright 2017 Sean Wang + * Copyright 2017-2018 Sean Wang * * SPDX-License-Identifier: (GPL-2.0+ OR MIT) */ @@ -109,10 +109,15 @@ }; memory@80000000 { - reg = <0 0x80000000 0 0x40000000>; + device_type = "memory"; + reg = <0 0x80000000 0 0x80000000>; }; }; +&btif { + status = "okay"; +}; + &cir { pinctrl-names = "default"; pinctrl-0 = <&cir_pins_a>; @@ -144,8 +149,6 @@ switch@0 { compatible = "mediatek,mt7530"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; reset-gpios = <&pio 33 0>; core-supply = <&mt6323_vpa_reg>; @@ -154,7 +157,6 @@ ports { #address-cells = <1>; #size-cells = <0>; - reg = <0>; port@0 { reg = <0>; @@ -235,6 +237,28 @@ vqmmc-supply = <®_3p3v>; }; +&mt6323_leds { + status = "okay"; + + led@0 { + reg = <0>; + label = "bpi-r2:isink:green"; + default-state = "off"; + }; + + led@1 { + reg = <1>; + label = "bpi-r2:isink:red"; + default-state = "off"; + }; + + led@2 { + reg = <2>; + label = "bpi-r2:isink:blue"; + default-state = "off"; + }; +}; + &pcie { pinctrl-names = "default"; pinctrl-0 = <&pcie_default>; @@ -257,257 +281,12 @@ status = "okay"; }; -&pio { - cir_pins_a:cir@0 { - pins-cir { - pinmux = ; - bias-disable; - }; - }; - - i2c0_pins_a: i2c@0 { - pins-i2c0 { - pinmux = , - ; - bias-disable; - }; - }; - - i2c1_pins_a: i2c@1 { - pin-i2c1 { - pinmux = , - ; - bias-disable; - }; - }; - - i2s0_pins_a: i2s@0 { - pin-i2s0 { - pinmux = , - , - , - , - ; - drive-strength = ; - bias-pull-down; - }; - }; - - i2s1_pins_a: i2s@1 { - pin-i2s1 { - pinmux = , - , - , - , - ; - drive-strength = ; - bias-pull-down; - }; - }; - - key_pins_a: keys@0 { - pins-keys { - pinmux = , - ; - input-enable; - }; - }; - - led_pins_a: leds@0 { - pins-leds { - pinmux = , - , - ; - }; - }; - - mmc0_pins_default: mmc0default { - pins-cmd-dat { - pinmux = , - , - , - , - , - , - , - , - ; - input-enable; - bias-pull-up; - }; - - pins-clk { - pinmux = ; - bias-pull-down; - }; - - pins-rst { - pinmux = ; - bias-pull-up; - }; - }; - - mmc0_pins_uhs: mmc0 { - pins-cmd-dat { - pinmux = , - , - , - , - , - , - , - , - ; - input-enable; - drive-strength = ; - bias-pull-up = ; - }; - - pins-clk { - pinmux = ; - drive-strength = ; - bias-pull-down = ; - }; - - pins-rst { - pinmux = ; - bias-pull-up; - }; - }; - - mmc1_pins_default: mmc1default { - pins-cmd-dat { - pinmux = , - , - , - , - ; - input-enable; - drive-strength = ; - bias-pull-up = ; - }; - - pins-clk { - pinmux = ; - bias-pull-down; - drive-strength = ; - }; - - pins-wp { - pinmux = ; - input-enable; - bias-pull-up; - }; - - pins-insert { - pinmux = ; - bias-pull-up; - }; - }; - - mmc1_pins_uhs: mmc1 { - pins-cmd-dat { - pinmux = , - , - , - , - ; - input-enable; - drive-strength = ; - bias-pull-up = ; - }; - - pins-clk { - pinmux = ; - drive-strength = ; - bias-pull-down = ; - }; - }; - - pcie_default: pcie_pin_default { - pins_cmd_dat { - pinmux = , - ; - bias-disable; - }; - }; - - pwm_pins_a: pwm@0 { - pins-pwm { - pinmux = , - , - , - , - ; - }; - }; - - spi0_pins_a: spi@0 { - pins-spi { - pinmux = , - , - , - ; - bias-disable; - }; - }; - - uart0_pins_a: uart@0 { - pins-dat { - pinmux = , - ; - }; - }; - - uart1_pins_a: uart@1 { - pins-dat { - pinmux = , - ; - }; - }; - - uart2_pins_a: uart@2 { - pins-dat { - pinmux = , - ; - }; - }; -}; - &pwm { pinctrl-names = "default"; pinctrl-0 = <&pwm_pins_a>; status = "okay"; }; -&pwrap { - mt6323 { - mt6323led: led { - compatible = "mediatek,mt6323-led"; - #address-cells = <1>; - #size-cells = <0>; - - led@0 { - reg = <0>; - label = "bpi-r2:isink:green"; - default-state = "off"; - }; - - led@1 { - reg = <1>; - label = "bpi-r2:isink:red"; - default-state = "off"; - }; - - led@2 { - reg = <2>; - label = "bpi-r2:isink:blue"; - default-state = "off"; - }; - }; - }; -}; - &spi0 { pinctrl-names = "default"; pinctrl-0 = <&spi0_pins_a>; diff --git a/arch/arm/boot/dts/mt7623n-rfb-emmc.dts b/arch/arm/boot/dts/mt7623n-rfb-emmc.dts new file mode 100644 index 000000000000..b7606130ade9 --- /dev/null +++ b/arch/arm/boot/dts/mt7623n-rfb-emmc.dts @@ -0,0 +1,326 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2017-2018 MediaTek Inc. + * Author: Sean Wang + * + */ + +/dts-v1/; +#include +#include "mt7623.dtsi" +#include "mt6323.dtsi" + +/ { + model = "MediaTek MT7623N with eMMC reference board"; + compatible = "mediatek,mt7623n-rfb-emmc", "mediatek,mt7623"; + + aliases { + serial0 = &uart0; + serial1 = &uart1; + serial2 = &uart2; + }; + + chosen { + stdout-path = "serial2:115200n8"; + }; + + cpus { + cpu@0 { + proc-supply = <&mt6323_vproc_reg>; + }; + + cpu@1 { + proc-supply = <&mt6323_vproc_reg>; + }; + + cpu@2 { + proc-supply = <&mt6323_vproc_reg>; + }; + + cpu@3 { + proc-supply = <&mt6323_vproc_reg>; + }; + }; + + gpio-keys { + compatible = "gpio-keys"; + pinctrl-names = "default"; + pinctrl-0 = <&key_pins_a>; + + factory { + label = "factory"; + linux,code = ; + gpios = <&pio 256 GPIO_ACTIVE_LOW>; + }; + + wps { + label = "wps"; + linux,code = ; + gpios = <&pio 257 GPIO_ACTIVE_HIGH>; + }; + }; + + memory@80000000 { + device_type = "memory"; + reg = <0 0x80000000 0 0x40000000>; + }; + + reg_1p8v: regulator-1p8v { + compatible = "regulator-fixed"; + regulator-name = "fixed-1.8V"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-boot-on; + regulator-always-on; + }; + + reg_3p3v: regulator-3p3v { + compatible = "regulator-fixed"; + regulator-name = "fixed-3.3V"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-boot-on; + regulator-always-on; + }; + + reg_5v: regulator-5v { + compatible = "regulator-fixed"; + regulator-name = "fixed-5V"; + regulator-min-microvolt = <5000000>; + regulator-max-microvolt = <5000000>; + regulator-boot-on; + regulator-always-on; + }; + + sound { + compatible = "mediatek,mt2701-wm8960-machine"; + mediatek,platform = <&afe>; + audio-routing = + "Headphone", "HP_L", + "Headphone", "HP_R", + "LINPUT1", "AMIC", + "RINPUT1", "AMIC"; + mediatek,audio-codec = <&wm8960>; + pinctrl-names = "default"; + pinctrl-0 = <&i2s0_pins_a>; + }; +}; + +&btif { + status = "okay"; +}; + +&cir { + pinctrl-names = "default"; + pinctrl-0 = <&cir_pins_a>; + status = "okay"; +}; + +&crypto { + status = "okay"; +}; + +ð { + status = "okay"; + + gmac0: mac@0 { + compatible = "mediatek,eth-mac"; + reg = <0>; + phy-mode = "trgmii"; + + fixed-link { + speed = <1000>; + full-duplex; + pause; + }; + }; + + mac@1 { + compatible = "mediatek,eth-mac"; + reg = <1>; + phy-handle = <&phy5>; + }; + + mdio-bus { + #address-cells = <1>; + #size-cells = <0>; + + phy5: ethernet-phy@5 { + reg = <5>; + phy-mode = "rgmii-rxid"; + }; + + switch@0 { + compatible = "mediatek,mt7530"; + reg = <0>; + reset-gpios = <&pio 33 0>; + core-supply = <&mt6323_vpa_reg>; + io-supply = <&mt6323_vemc3v3_reg>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + label = "lan0"; + }; + + port@1 { + reg = <1>; + label = "lan1"; + }; + + port@2 { + reg = <2>; + label = "lan2"; + }; + + port@3 { + reg = <3>; + label = "lan3"; + }; + + port@4 { + reg = <4>; + label = "wan"; + }; + + port@6 { + reg = <6>; + label = "cpu"; + ethernet = <&gmac0>; + phy-mode = "trgmii"; + + fixed-link { + speed = <1000>; + full-duplex; + }; + }; + }; + }; + }; +}; + +&i2c0 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c0_pins_a>; + status = "okay"; +}; + +&i2c1 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c1_pins_b>; + status = "okay"; + + wm8960: wm8960@1a { + compatible = "wlf,wm8960"; + reg = <0x1a>; + }; +}; + +&i2c2 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c2_pins_a>; + status = "okay"; +}; + +&mmc0 { + pinctrl-names = "default", "state_uhs"; + pinctrl-0 = <&mmc0_pins_default>; + pinctrl-1 = <&mmc0_pins_uhs>; + status = "okay"; + bus-width = <8>; + max-frequency = <50000000>; + cap-mmc-highspeed; + vmmc-supply = <®_3p3v>; + vqmmc-supply = <®_1p8v>; + non-removable; +}; + +&mmc1 { + pinctrl-names = "default", "state_uhs"; + pinctrl-0 = <&mmc1_pins_default>; + pinctrl-1 = <&mmc1_pins_uhs>; + status = "okay"; + bus-width = <4>; + max-frequency = <50000000>; + cap-sd-highspeed; + cd-gpios = <&pio 261 GPIO_ACTIVE_LOW>; + vmmc-supply = <®_3p3v>; + vqmmc-supply = <®_3p3v>; +}; + +&pcie { + pinctrl-names = "default"; + pinctrl-0 = <&pcie_default>; + status = "okay"; + + pcie@0,0 { + status = "okay"; + }; + + pcie@1,0 { + status = "okay"; + }; +}; + +&pcie0_phy { + status = "okay"; +}; + +&pcie1_phy { + status = "okay"; +}; + +&pwm { + pinctrl-names = "default"; + pinctrl-0 = <&pwm_pins_a>; + status = "okay"; +}; + +&spi0 { + pinctrl-names = "default"; + pinctrl-0 = <&spi0_pins_a>; + status = "okay"; +}; + +&spi1 { + pinctrl-names = "default"; + pinctrl-0 = <&spi1_pins_a>; + status = "okay"; +}; + +&spi2 { + pinctrl-names = "default"; + pinctrl-0 = <&spi2_pins_a>; + status = "okay"; +}; + +&uart0 { + pinctrl-names = "default"; + pinctrl-0 = <&uart0_pins_a>; + status = "okay"; +}; + +&uart1 { + pinctrl-names = "default"; + pinctrl-0 = <&uart1_pins_a>; + status = "okay"; +}; + +&uart2 { + pinctrl-names = "default"; + pinctrl-0 = <&uart2_pins_a>; + status = "okay"; +}; + +&usb1 { + vusb33-supply = <®_3p3v>; + vbus-supply = <®_5v>; + status = "okay"; +}; + +&u3phy1 { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/mt7623n-rfb-nand.dts b/arch/arm/boot/dts/mt7623n-rfb-nand.dts index f729c718aba1..96ff3c9068ae 100644 --- a/arch/arm/boot/dts/mt7623n-rfb-nand.dts +++ b/arch/arm/boot/dts/mt7623n-rfb-nand.dts @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2017 MediaTek Inc. * Author: John Crispin * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ /dts-v1/; @@ -78,34 +71,3 @@ }; }; }; - -&pio { - nand_pins_default: nanddefault { - pins-ale { - pinmux = ; - drive-strength = ; - bias-pull-down = ; - }; - - pins-dat { - pinmux = , - , - , - , - , - , - , - , - ; - input-enable; - drive-strength = ; - bias-pull-up; - }; - - pins-we { - pinmux = ; - drive-strength = ; - bias-pull-up = ; - }; - }; -}; diff --git a/arch/arm/boot/dts/mt7623n-rfb.dtsi b/arch/arm/boot/dts/mt7623n-rfb.dtsi index 256c5fd947bf..5c5cc7da5dd2 100644 --- a/arch/arm/boot/dts/mt7623n-rfb.dtsi +++ b/arch/arm/boot/dts/mt7623n-rfb.dtsi @@ -1,16 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2017 MediaTek Inc. * Author: John Crispin * Sean Wang * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ /dts-v1/; @@ -47,10 +40,11 @@ }; memory@80000000 { + device_type = "memory"; reg = <0 0x80000000 0 0x40000000>; }; - usb_p1_vbus: regulator@0 { + usb_p1_vbus: regulator-5v { compatible = "regulator-fixed"; regulator-name = "usb_vbus"; regulator-min-microvolt = <5000000>; diff --git a/arch/arm/boot/dts/mt8127-moose.dts b/arch/arm/boot/dts/mt8127-moose.dts index 073e295a1cb4..308829b2da86 100644 --- a/arch/arm/boot/dts/mt8127-moose.dts +++ b/arch/arm/boot/dts/mt8127-moose.dts @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2014 MediaTek Inc. * Author: Joe.C * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ /dts-v1/; diff --git a/arch/arm/boot/dts/mt8127.dtsi b/arch/arm/boot/dts/mt8127.dtsi index 916c095d11b9..3adfc6f7859c 100644 --- a/arch/arm/boot/dts/mt8127.dtsi +++ b/arch/arm/boot/dts/mt8127.dtsi @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2014 MediaTek Inc. * Author: Joe.C * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/arch/arm/boot/dts/mt8135-evbp1.dts b/arch/arm/boot/dts/mt8135-evbp1.dts index 460db6d05952..0ace7a40a60d 100644 --- a/arch/arm/boot/dts/mt8135-evbp1.dts +++ b/arch/arm/boot/dts/mt8135-evbp1.dts @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2014 MediaTek Inc. * Author: Joe.C * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ /dts-v1/; diff --git a/arch/arm/boot/dts/mt8135.dtsi b/arch/arm/boot/dts/mt8135.dtsi index a97b4ee4ae79..688069dc1533 100644 --- a/arch/arm/boot/dts/mt8135.dtsi +++ b/arch/arm/boot/dts/mt8135.dtsi @@ -1,15 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2014 MediaTek Inc. * Author: Joe.C * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #include diff --git a/arch/arm/boot/dts/omap2420-n810.dts b/arch/arm/boot/dts/omap2420-n810.dts index 7c485fbfa535..96b9913ecc1f 100644 --- a/arch/arm/boot/dts/omap2420-n810.dts +++ b/arch/arm/boot/dts/omap2420-n810.dts @@ -6,11 +6,70 @@ / { model = "Nokia N810"; compatible = "nokia,n810", "nokia,n8x0", "ti,omap2420", "ti,omap2"; + + vio_ape: vio_ape { + compatible = "regulator-fixed"; + regulator-name = "vio_ape"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + }; + + v28_aic: v28_aic { + compatible = "regulator-fixed"; + regulator-name = "v28_aic"; + regulator-min-microvolt = <2800000>; + regulator-max-microvolt = <2800000>; + }; +}; + +&omap2420_pmx { + mcbsp2_pins: mcbsp2_pins { + pinctrl-single,pins = < + OMAP2420_CORE_IOPAD(0x0124, PIN_INPUT | MUX_MODE1) /* eac_ac_sclk.mcbsp2_clkx */ + OMAP2420_CORE_IOPAD(0x0125, PIN_INPUT | MUX_MODE1) /* eac_ac_fs.mcbsp2_fsx */ + OMAP2420_CORE_IOPAD(0x0126, PIN_INPUT | MUX_MODE1) /* eac_ac_din.mcbsp2_dr */ + OMAP2420_CORE_IOPAD(0x0127, PIN_OUTPUT | MUX_MODE1) /* eac_ac_dout.mcbsp2_dx */ + >; + }; + + aic33_pins: aic33_pins { + pinctrl-single,pins = < + OMAP2420_CORE_IOPAD(0x0129, PIN_OUTPUT | MUX_MODE3) /* eac_ac_rst.gpio118 */ + OMAP2420_CORE_IOPAD(0x00e8, PIN_OUTPUT | MUX_MODE2) /* vlynq_tx1.sys_clkout2 */ + >; + }; }; &i2c2 { - aic3x@18 { - compatible = "tlv320aic3x"; + aic33@18 { + compatible = "ti,tlv320aic33"; reg = <0x18>; + + pinctrl-names = "default"; + pinctrl-0 = <&aic33_pins>; + + gpio-reset = <&gpio4 22 GPIO_ACTIVE_LOW>; /* gpio118 */ + + ai3x-gpio-func = < + 10 /* AIC3X_GPIO1_FUNC_DIGITAL_MIC_MODCLK */ + 5 /* AIC3X_GPIO2_FUNC_DIGITAL_MIC_INPUT */ + >; + ai3x-micbias-vg = <1>; /* 2V */ + + AVDD-supply = <&v28_aic>; + DRVDD-supply = <&v28_aic>; + IOVDD-supply = <&vio_ape>; + DVDD-supply = <&vio_ape>; + + assigned-clocks = <&sys_clkout2_src>, <&sys_clkout2>; + assigned-clock-parents = <&func_96m_ck>; + assigned-clock-rates = <0>, <12000000>; }; }; + +&mcbsp2 { + pinctrl-names = "default"; + pinctrl-0 = <&mcbsp2_pins>; + + status = "okay"; +}; diff --git a/arch/arm/boot/dts/omap3-beagle-xm.dts b/arch/arm/boot/dts/omap3-beagle-xm.dts index 0349fcc9dc26..d80587de0bbf 100644 --- a/arch/arm/boot/dts/omap3-beagle-xm.dts +++ b/arch/arm/boot/dts/omap3-beagle-xm.dts @@ -30,6 +30,13 @@ ethernet = ðernet; }; + /* fixed 26MHz oscillator */ + hfclk_26m: oscillator { + #clock-cells = <0>; + compatible = "fixed-clock"; + clock-frequency = <26000000>; + }; + leds { compatible = "gpio-leds"; @@ -274,6 +281,9 @@ interrupts = <7>; /* SYS_NIRQ cascaded to intc */ interrupt-parent = <&intc>; + clocks = <&hfclk_26m>; + clock-names = "fck"; + twl_audio: audio { compatible = "ti,twl4030-audio"; codec { diff --git a/arch/arm/boot/dts/omap3-cm-t3x.dtsi b/arch/arm/boot/dts/omap3-cm-t3x.dtsi index 9dcb18d22cde..cdb632df152a 100644 --- a/arch/arm/boot/dts/omap3-cm-t3x.dtsi +++ b/arch/arm/boot/dts/omap3-cm-t3x.dtsi @@ -60,7 +60,7 @@ regulator-max-microvolt = <3300000>; }; - tv0: connector { + tv0: svideo-connector { compatible = "svideo-connector"; label = "tv"; diff --git a/arch/arm/boot/dts/omap3-devkit8000-common.dtsi b/arch/arm/boot/dts/omap3-devkit8000-common.dtsi index 0c0bb1b01b0b..746a658e84b6 100644 --- a/arch/arm/boot/dts/omap3-devkit8000-common.dtsi +++ b/arch/arm/boot/dts/omap3-devkit8000-common.dtsi @@ -349,10 +349,17 @@ vdda_dac-supply = <&vdac>; port { - dpi_dvi_out: endpoint { + #address-cells = <1>; + #size-cells = <0>; + dpi_dvi_out: endpoint@0 { + reg = <0>; remote-endpoint = <&tfp410_in>; data-lines = <24>; }; + + endpoint@1 { + reg = <1>; + }; }; }; diff --git a/arch/arm/boot/dts/omap3-devkit8000-lcd-common.dtsi b/arch/arm/boot/dts/omap3-devkit8000-lcd-common.dtsi index 2d64bcffaaa8..1093387259e2 100644 --- a/arch/arm/boot/dts/omap3-devkit8000-lcd-common.dtsi +++ b/arch/arm/boot/dts/omap3-devkit8000-lcd-common.dtsi @@ -30,7 +30,10 @@ &dss { port { - dpi_lcd_out: endpoint { + #address-cells = <1>; + #size-cells = <0>; + dpi_lcd_out: endpoint@1 { + reg = <1>; remote-endpoint = <&lcd_in>; data-lines = <24>; }; diff --git a/arch/arm/boot/dts/omap3-gta04.dtsi b/arch/arm/boot/dts/omap3-gta04.dtsi index 4170be70460e..ac830b917776 100644 --- a/arch/arm/boot/dts/omap3-gta04.dtsi +++ b/arch/arm/boot/dts/omap3-gta04.dtsi @@ -30,6 +30,13 @@ display0 = &lcd; }; + /* fixed 26MHz oscillator */ + hfclk_26m: oscillator { + #clock-cells = <0>; + compatible = "fixed-clock"; + clock-frequency = <26000000>; + }; + gpio-keys { compatible = "gpio-keys"; @@ -312,6 +319,9 @@ interrupts = <7>; /* SYS_NIRQ cascaded to intc */ interrupt-parent = <&intc>; + clocks = <&hfclk_26m>; + clock-names = "fck"; + twl_audio: audio { compatible = "ti,twl4030-audio"; ti,enable-vibra = <1>; diff --git a/arch/arm/boot/dts/omap3-pandora-common.dtsi b/arch/arm/boot/dts/omap3-pandora-common.dtsi index f83b1029b3b7..90c98f95b2b3 100644 --- a/arch/arm/boot/dts/omap3-pandora-common.dtsi +++ b/arch/arm/boot/dts/omap3-pandora-common.dtsi @@ -27,6 +27,13 @@ display0 = &lcd; }; + /* fixed 26MHz oscillator */ + hfclk_26m: oscillator { + #clock-cells = <0>; + compatible = "fixed-clock"; + clock-frequency = <26000000>; + }; + tv: connector { compatible = "connector-analog-tv"; label = "tv"; @@ -357,6 +364,9 @@ interrupts = <7>; /* SYS_NIRQ cascaded to intc */ interrupt-parent = <&intc>; + clocks = <&hfclk_26m>; + clock-names = "fck"; + twl_power: power { compatible = "ti,twl4030-power-reset"; ti,use_poweroff; @@ -611,7 +621,7 @@ pinctrl-names = "default"; pinctrl-0 = <&penirq_pins>; interrupt-parent = <&gpio3>; - interrupts = <30 0>; /* GPIO_94 */ + interrupts = <30 IRQ_TYPE_NONE>; /* GPIO_94 */ pendown-gpio = <&gpio3 30 GPIO_ACTIVE_HIGH>; vcc-supply = <&vaux4>; diff --git a/arch/arm/boot/dts/omap3-sb-t35.dtsi b/arch/arm/boot/dts/omap3-sb-t35.dtsi index 22b4c8bdcc65..fb9842fa922c 100644 --- a/arch/arm/boot/dts/omap3-sb-t35.dtsi +++ b/arch/arm/boot/dts/omap3-sb-t35.dtsi @@ -34,7 +34,7 @@ }; }; - dvi0: connector { + dvi0: dvi-connector { compatible = "dvi-connector"; label = "dvi"; diff --git a/arch/arm/boot/dts/pxa3xx.dtsi b/arch/arm/boot/dts/pxa3xx.dtsi index 982d1a62661d..132a3b8ab148 100644 --- a/arch/arm/boot/dts/pxa3xx.dtsi +++ b/arch/arm/boot/dts/pxa3xx.dtsi @@ -8,6 +8,10 @@ (gpio <= 98) ? (0x0400 + 4 * (gpio - 27)) : \ (gpio <= 127) ? (0x0600 + 4 * (gpio - 99)) : \ 0) +#define MFP_PIN_PXA300_2(gpio) \ + ((gpio <= 1) ? (0x674 + 4 * gpio) : \ + (gpio <= 6) ? (0x2dc + 4 * gpio) : \ + 0) #define MFP_PIN_PXA310(gpio) \ ((gpio <= 2) ? (0x00b4 + 4 * gpio) : \ @@ -18,6 +22,11 @@ (gpio <= 262) ? 0 : \ (gpio <= 268) ? (0x052c + 4 * (gpio - 263)) : \ 0) +#define MFP_PIN_PXA310_2(gpio) \ + ((gpio <= 1) ? (0x674 + 4 * gpio) : \ + (gpio <= 6) ? (0x2dc + 4 * gpio) : \ + (gpio <= 10) ? (0x52c + 4 * gpio) : \ + 0) #define MFP_PIN_PXA320(gpio) \ ((gpio <= 4) ? (0x0124 + 4 * gpio) : \ @@ -30,6 +39,10 @@ (gpio <= 98) ? (0x04f0 + 4 * (gpio - 74)) : \ (gpio <= 127) ? (0x0600 + 4 * (gpio - 99)) : \ 0) +#define MFP_PIN_PXA320_2(gpio) \ + ((gpio <= 3) ? (0x674 + 4 * gpio) : \ + (gpio <= 5) ? (0x284 + 4 * gpio) : \ + 0) /* * MFP Alternate functions for pins having a gpio. @@ -148,6 +161,7 @@ compatible = "intel,pxa3xx-gpio"; reg = <0x40e00000 0x10000>; clocks = <&clks CLK_GPIO>; + gpio-ranges = <&pinctrl 0 0 128>; interrupt-names = "gpio0", "gpio1", "gpio_mux"; interrupts = <8 9 10>; gpio-controller; @@ -160,7 +174,7 @@ compatible = "marvell,pxa-mmc"; reg = <0x41100000 0x1000>; interrupts = <23>; - clocks = <&clks CLK_MMC>; + clocks = <&clks CLK_MMC1>; dmas = <&pdma 21 3 &pdma 22 3>; dma-names = "rx", "tx"; @@ -171,7 +185,7 @@ compatible = "marvell,pxa-mmc"; reg = <0x42000000 0x1000>; interrupts = <41>; - clocks = <&clks CLK_MMC1>; + clocks = <&clks CLK_MMC2>; dmas = <&pdma 93 3 &pdma 94 3>; dma-names = "rx", "tx"; @@ -182,7 +196,7 @@ compatible = "marvell,pxa-mmc"; reg = <0x42500000 0x1000>; interrupts = <55>; - clocks = <&clks CLK_MMC2>; + clocks = <&clks CLK_MMC3>; dmas = <&pdma 46 3 &pdma 47 3>; dma-names = "rx", "tx"; diff --git a/arch/arm/boot/dts/qcom-apq8064.dtsi b/arch/arm/boot/dts/qcom-apq8064.dtsi index 5341a39c0392..4a99c9255104 100644 --- a/arch/arm/boot/dts/qcom-apq8064.dtsi +++ b/arch/arm/boot/dts/qcom-apq8064.dtsi @@ -444,7 +444,7 @@ compatible = "qcom,msm-uartdm-v1.3", "qcom,msm-uartdm"; reg = <0x12450000 0x100>, <0x12400000 0x03>; - interrupts = <0 193 0x0>; + interrupts = <0 193 IRQ_TYPE_LEVEL_HIGH>; clocks = <&gcc GSBI1_UART_CLK>, <&gcc GSBI1_H_CLK>; clock-names = "core", "iface"; status = "disabled"; @@ -456,11 +456,12 @@ pinctrl-1 = <&i2c1_pins_sleep>; pinctrl-names = "default", "sleep"; reg = <0x12460000 0x1000>; - interrupts = <0 194 IRQ_TYPE_NONE>; + interrupts = <0 194 IRQ_TYPE_LEVEL_HIGH>; clocks = <&gcc GSBI1_QUP_CLK>, <&gcc GSBI1_H_CLK>; clock-names = "core", "iface"; #address-cells = <1>; #size-cells = <0>; + status = "disabled"; }; }; @@ -484,11 +485,12 @@ pinctrl-0 = <&i2c2_pins>; pinctrl-1 = <&i2c2_pins_sleep>; pinctrl-names = "default", "sleep"; - interrupts = <0 196 IRQ_TYPE_NONE>; + interrupts = <0 196 IRQ_TYPE_LEVEL_HIGH>; clocks = <&gcc GSBI2_QUP_CLK>, <&gcc GSBI2_H_CLK>; clock-names = "core", "iface"; #address-cells = <1>; #size-cells = <0>; + status = "disabled"; }; }; @@ -508,12 +510,13 @@ pinctrl-1 = <&i2c3_pins_sleep>; pinctrl-names = "default", "sleep"; reg = <0x16280000 0x1000>; - interrupts = ; + interrupts = ; clocks = <&gcc GSBI3_QUP_CLK>, <&gcc GSBI3_H_CLK>; clock-names = "core", "iface"; #address-cells = <1>; #size-cells = <0>; + status = "disabled"; }; }; @@ -534,10 +537,11 @@ pinctrl-1 = <&i2c4_pins_sleep>; pinctrl-names = "default", "sleep"; reg = <0x16380000 0x1000>; - interrupts = ; + interrupts = ; clocks = <&gcc GSBI4_QUP_CLK>, <&gcc GSBI4_H_CLK>; clock-names = "core", "iface"; + status = "disabled"; }; }; @@ -556,7 +560,7 @@ compatible = "qcom,msm-uartdm-v1.3", "qcom,msm-uartdm"; reg = <0x1a240000 0x100>, <0x1a200000 0x03>; - interrupts = <0 154 0x0>; + interrupts = <0 154 IRQ_TYPE_LEVEL_HIGH>; clocks = <&gcc GSBI5_UART_CLK>, <&gcc GSBI5_H_CLK>; clock-names = "core", "iface"; status = "disabled"; @@ -565,7 +569,7 @@ gsbi5_spi: spi@1a280000 { compatible = "qcom,spi-qup-v1.1.1"; reg = <0x1a280000 0x1000>; - interrupts = <0 155 0>; + interrupts = <0 155 IRQ_TYPE_LEVEL_HIGH>; pinctrl-0 = <&spi5_default>; pinctrl-1 = <&spi5_sleep>; pinctrl-names = "default", "sleep"; @@ -592,7 +596,7 @@ compatible = "qcom,msm-uartdm-v1.3", "qcom,msm-uartdm"; reg = <0x16540000 0x100>, <0x16500000 0x03>; - interrupts = <0 156 0x0>; + interrupts = <0 156 IRQ_TYPE_LEVEL_HIGH>; clocks = <&gcc GSBI6_UART_CLK>, <&gcc GSBI6_H_CLK>; clock-names = "core", "iface"; status = "disabled"; @@ -604,7 +608,7 @@ pinctrl-1 = <&i2c6_pins_sleep>; pinctrl-names = "default", "sleep"; reg = <0x16580000 0x1000>; - interrupts = ; + interrupts = ; clocks = <&gcc GSBI6_QUP_CLK>, <&gcc GSBI6_H_CLK>; clock-names = "core", "iface"; @@ -628,7 +632,7 @@ compatible = "qcom,msm-uartdm-v1.3", "qcom,msm-uartdm"; reg = <0x16640000 0x1000>, <0x16600000 0x1000>; - interrupts = <0 158 0x0>; + interrupts = <0 158 IRQ_TYPE_LEVEL_HIGH>; clocks = <&gcc GSBI7_UART_CLK>, <&gcc GSBI7_H_CLK>; clock-names = "core", "iface"; status = "disabled"; @@ -640,7 +644,7 @@ pinctrl-1 = <&i2c7_pins_sleep>; pinctrl-names = "default", "sleep"; reg = <0x16680000 0x1000>; - interrupts = ; + interrupts = ; clocks = <&gcc GSBI7_QUP_CLK>, <&gcc GSBI7_H_CLK>; clock-names = "core", "iface"; @@ -1056,7 +1060,7 @@ compatible = "qcom,apq8064-ahci", "generic-ahci"; status = "disabled"; reg = <0x29000000 0x180>; - interrupts = ; + interrupts = ; clocks = <&gcc SFAB_SATA_S_H_CLK>, <&gcc SATA_H_CLK>, @@ -1082,7 +1086,7 @@ sdcc1bam:dma@12402000{ compatible = "qcom,bam-v1.3.0"; reg = <0x12402000 0x8000>; - interrupts = <0 98 0>; + interrupts = <0 98 IRQ_TYPE_LEVEL_HIGH>; clocks = <&gcc SDC1_H_CLK>; clock-names = "bam_clk"; #dma-cells = <1>; @@ -1092,7 +1096,7 @@ sdcc3bam:dma@12182000{ compatible = "qcom,bam-v1.3.0"; reg = <0x12182000 0x8000>; - interrupts = <0 96 0>; + interrupts = <0 96 IRQ_TYPE_LEVEL_HIGH>; clocks = <&gcc SDC3_H_CLK>; clock-names = "bam_clk"; #dma-cells = <1>; @@ -1102,7 +1106,7 @@ sdcc4bam:dma@121c2000{ compatible = "qcom,bam-v1.3.0"; reg = <0x121c2000 0x8000>; - interrupts = <0 95 0>; + interrupts = <0 95 IRQ_TYPE_LEVEL_HIGH>; clocks = <&gcc SDC4_H_CLK>; clock-names = "bam_clk"; #dma-cells = <1>; @@ -1181,7 +1185,7 @@ compatible = "qcom,adreno-3xx"; reg = <0x04300000 0x20000>; reg-names = "kgsl_3d0_reg_memory"; - interrupts = ; + interrupts = ; interrupt-names = "kgsl_3d0_irq"; clock-names = "core_clk", @@ -1281,7 +1285,7 @@ label = "MDSS DSI CTRL->0"; #address-cells = <1>; #size-cells = <0>; - interrupts = ; + interrupts = ; reg = <0x04700000 0x200>; reg-names = "dsi_ctrl"; @@ -1350,8 +1354,8 @@ <&mmcc MDP_AXI_CLK>; reg = <0x07500000 0x100000>; interrupts = - , - ; + , + ; qcom,ncb = <2>; }; @@ -1366,8 +1370,8 @@ <&mmcc MDP_AXI_CLK>; reg = <0x07600000 0x100000>; interrupts = - , - ; + , + ; qcom,ncb = <2>; }; @@ -1382,8 +1386,8 @@ <&mmcc GFX3D_AXI_CLK>; reg = <0x07c00000 0x100000>; interrupts = - , - ; + , + ; qcom,ncb = <3>; }; @@ -1398,8 +1402,8 @@ <&mmcc GFX3D_AXI_CLK>; reg = <0x07d00000 0x100000>; interrupts = - , - ; + , + ; qcom,ncb = <3>; }; @@ -1417,8 +1421,8 @@ #address-cells = <3>; #size-cells = <2>; ranges = <0x81000000 0 0 0x0fe00000 0 0x00100000 /* I/O */ - 0x82000000 0 0 0x08000000 0 0x07e00000>; /* memory */ - interrupts = ; + 0x82000000 0 0x08000000 0x08000000 0 0x07e00000>; /* memory */ + interrupts = ; interrupt-names = "msi"; #interrupt-cells = <1>; interrupt-map-mask = <0 0 0 0x7>; diff --git a/arch/arm/boot/dts/qcom-ipq4019-ap.dk01.1.dtsi b/arch/arm/boot/dts/qcom-ipq4019-ap.dk01.1.dtsi index e413b21ee331..418f9a022336 100644 --- a/arch/arm/boot/dts/qcom-ipq4019-ap.dk01.1.dtsi +++ b/arch/arm/boot/dts/qcom-ipq4019-ap.dk01.1.dtsi @@ -20,6 +20,14 @@ model = "Qualcomm Technologies, Inc. IPQ4019/AP-DK01.1"; compatible = "qcom,ipq4019"; + aliases { + serial0 = &blsp1_uart1; + }; + + chosen { + stdout-path = "serial0:115200n8"; + }; + soc { rng@22000 { status = "ok"; @@ -61,7 +69,7 @@ status = "ok"; }; - spi_0: spi@78b5000 { + spi@78b5000 { pinctrl-0 = <&spi_0_pins>; pinctrl-names = "default"; status = "ok"; diff --git a/arch/arm/boot/dts/qcom-ipq4019-ap.dk04.1-c1.dts b/arch/arm/boot/dts/qcom-ipq4019-ap.dk04.1-c1.dts new file mode 100644 index 000000000000..7a96f300bc8d --- /dev/null +++ b/arch/arm/boot/dts/qcom-ipq4019-ap.dk04.1-c1.dts @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2018, The Linux Foundation. All rights reserved. + +#include "qcom-ipq4019-ap.dk04.1.dtsi" + +/ { + model = "Qualcomm Technologies, Inc. IPQ4019/AP-DK04.1-C1"; + compatible = "qcom,ipq4019-dk04.1-c1"; + + soc { + dma@7984000 { + status = "ok"; + }; + + qpic-nand@79b0000 { + status = "ok"; + }; + }; +}; diff --git a/arch/arm/boot/dts/qcom-ipq4019-ap.dk04.1-c3.dts b/arch/arm/boot/dts/qcom-ipq4019-ap.dk04.1-c3.dts new file mode 100644 index 000000000000..2d1c4c6e42f1 --- /dev/null +++ b/arch/arm/boot/dts/qcom-ipq4019-ap.dk04.1-c3.dts @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2018, The Linux Foundation. All rights reserved. + +#include "qcom-ipq4019-ap.dk04.1.dtsi" + +/ { + model = "Qualcomm Technologies, Inc. IPQ4019/AP-DK04.1-C3"; + compatible = "qcom,ipq4019-ap-dk04.1-c3"; +}; diff --git a/arch/arm/boot/dts/qcom-ipq4019-ap.dk04.1.dtsi b/arch/arm/boot/dts/qcom-ipq4019-ap.dk04.1.dtsi new file mode 100644 index 000000000000..7c1eb1963c67 --- /dev/null +++ b/arch/arm/boot/dts/qcom-ipq4019-ap.dk04.1.dtsi @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2018, The Linux Foundation. All rights reserved. + +#include "qcom-ipq4019.dtsi" +#include +#include + +/ { + model = "Qualcomm Technologies, Inc. IPQ4019/AP-DK04.1"; + + aliases { + serial0 = &blsp1_uart1; + serial1 = &blsp1_uart2; + }; + + chosen { + stdout-path = "serial0:115200n8"; + }; + + memory { + device_type = "memory"; + reg = <0x80000000 0x10000000>; /* 256MB */ + }; + + soc { + pinctrl@1000000 { + serial_0_pins: serial0-pinmux { + pins = "gpio16", "gpio17"; + function = "blsp_uart0"; + bias-disable; + }; + + serial_1_pins: serial1-pinmux { + pins = "gpio8", "gpio9", + "gpio10", "gpio11"; + function = "blsp_uart1"; + bias-disable; + }; + + spi_0_pins: spi-0-pinmux { + pinmux { + function = "blsp_spi0"; + pins = "gpio13", "gpio14", "gpio15"; + bias-disable; + }; + pinmux_cs { + function = "gpio"; + pins = "gpio12"; + bias-disable; + output-high; + }; + }; + + i2c_0_pins: i2c-0-pinmux { + pins = "gpio20", "gpio21"; + function = "blsp_i2c0"; + bias-disable; + }; + + nand_pins: nand-pins { + pins = "gpio53", "gpio55", "gpio56", + "gpio57", "gpio58", "gpio59", + "gpio60", "gpio62", "gpio63", + "gpio64", "gpio65", "gpio66", + "gpio67", "gpio68", "gpio69"; + function = "qpic"; + }; + }; + + serial@78af000 { + pinctrl-0 = <&serial_0_pins>; + pinctrl-names = "default"; + status = "ok"; + }; + + serial@78b0000 { + pinctrl-0 = <&serial_1_pins>; + pinctrl-names = "default"; + status = "ok"; + }; + + dma@7884000 { + status = "ok"; + }; + + spi@78b5000 { /* BLSP1 QUP1 */ + pinctrl-0 = <&spi_0_pins>; + pinctrl-names = "default"; + status = "ok"; + cs-gpios = <&tlmm 12 0>; + + m25p80@0 { + #address-cells = <1>; + #size-cells = <1>; + reg = <0>; + compatible = "n25q128a11"; + spi-max-frequency = <24000000>; + }; + }; + + pci@40000000 { + status = "ok"; + perst-gpio = <&tlmm 38 0x1>; + }; + + qpic-nand@79b0000 { + pinctrl-0 = <&nand_pins>; + pinctrl-names = "default"; + }; + }; +}; diff --git a/arch/arm/boot/dts/qcom-ipq4019-ap.dk07.1-c1.dts b/arch/arm/boot/dts/qcom-ipq4019-ap.dk07.1-c1.dts new file mode 100644 index 000000000000..8c7ef6537ae6 --- /dev/null +++ b/arch/arm/boot/dts/qcom-ipq4019-ap.dk07.1-c1.dts @@ -0,0 +1,64 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2018, The Linux Foundation. All rights reserved. + +#include "qcom-ipq4019-ap.dk07.1.dtsi" + +/ { + model = "Qualcomm Technologies, Inc. IPQ4019/AP-DK07.1-C1"; + compatible = "qcom,ipq4019-ap-dk07.1-c1"; + + soc { + pci@40000000 { + status = "ok"; + perst-gpio = <&tlmm 38 0x1>; + }; + + spi@78b6000 { + status = "ok"; + }; + + pinctrl@1000000 { + serial_1_pins: serial1-pinmux { + pins = "gpio8", "gpio9", + "gpio10", "gpio11"; + function = "blsp_uart1"; + bias-disable; + }; + + spi_0_pins: spi-0-pinmux { + pinmux { + function = "blsp_spi0"; + pins = "gpio13", "gpio14", "gpio15"; + bias-disable; + }; + pinmux_cs { + function = "gpio"; + pins = "gpio12"; + bias-disable; + output-high; + }; + }; + }; + + serial@78b0000 { + pinctrl-0 = <&serial_1_pins>; + pinctrl-names = "default"; + status = "ok"; + }; + + spi@78b5000 { + pinctrl-0 = <&spi_0_pins>; + pinctrl-names = "default"; + status = "ok"; + cs-gpios = <&tlmm 12 0>; + + m25p80@0 { + #address-cells = <1>; + #size-cells = <1>; + reg = <0>; + compatible = "n25q128a11"; + spi-max-frequency = <24000000>; + }; + }; + }; +}; diff --git a/arch/arm/boot/dts/qcom-ipq4019-ap.dk07.1-c2.dts b/arch/arm/boot/dts/qcom-ipq4019-ap.dk07.1-c2.dts new file mode 100644 index 000000000000..af7a9028d492 --- /dev/null +++ b/arch/arm/boot/dts/qcom-ipq4019-ap.dk07.1-c2.dts @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2018, The Linux Foundation. All rights reserved. + +#include "qcom-ipq4019-ap.dk07.1.dtsi" + +/ { + model = "Qualcomm Technologies, Inc. IPQ4019/AP-DK07.1-C2"; + compatible = "qcom,ipq4019-ap-dk07.1-c2"; + + soc { + pinctrl@1000000 { + serial_1_pins: serial1-pinmux { + pins = "gpio8", "gpio9"; + function = "blsp_uart1"; + bias-disable; + }; + }; + + serial@78b0000 { + pinctrl-0 = <&serial_1_pins>; + pinctrl-names = "default"; + status = "ok"; + }; + }; +}; diff --git a/arch/arm/boot/dts/qcom-ipq4019-ap.dk07.1.dtsi b/arch/arm/boot/dts/qcom-ipq4019-ap.dk07.1.dtsi new file mode 100644 index 000000000000..9f1a5a668772 --- /dev/null +++ b/arch/arm/boot/dts/qcom-ipq4019-ap.dk07.1.dtsi @@ -0,0 +1,75 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (c) 2018, The Linux Foundation. All rights reserved. + +#include "qcom-ipq4019.dtsi" +#include +#include + +/ { + model = "Qualcomm Technologies, Inc. IPQ4019/AP-DK07.1"; + + memory { + device_type = "memory"; + reg = <0x80000000 0x20000000>; /* 512MB */ + }; + + aliases { + serial0 = &blsp1_uart1; + serial1 = &blsp1_uart2; + }; + + chosen { + stdout-path = "serial0:115200n8"; + }; + + soc { + pinctrl@1000000 { + serial_0_pins: serial0-pinmux { + pins = "gpio16", "gpio17"; + function = "blsp_uart0"; + bias-disable; + }; + + i2c_0_pins: i2c-0-pinmux { + pins = "gpio20", "gpio21"; + function = "blsp_i2c0"; + bias-disable; + }; + + nand_pins: nand-pins { + pins = "gpio53", "gpio55", "gpio56", + "gpio57", "gpio58", "gpio59", + "gpio60", "gpio62", "gpio63", + "gpio64", "gpio65", "gpio66", + "gpio67", "gpio68", "gpio69"; + function = "qpic"; + }; + }; + + serial@78af000 { + pinctrl-0 = <&serial_0_pins>; + pinctrl-names = "default"; + status = "ok"; + }; + + dma@7884000 { + status = "ok"; + }; + + i2c@78b7000 { /* BLSP1 QUP2 */ + pinctrl-0 = <&i2c_0_pins>; + pinctrl-names = "default"; + status = "ok"; + }; + + dma@7984000 { + status = "ok"; + }; + + qpic-nand@79b0000 { + pinctrl-0 = <&nand_pins>; + pinctrl-names = "default"; + status = "ok"; + }; + }; +}; diff --git a/arch/arm/boot/dts/qcom-ipq4019.dtsi b/arch/arm/boot/dts/qcom-ipq4019.dtsi index 10d112a4078e..7bcd7635e723 100644 --- a/arch/arm/boot/dts/qcom-ipq4019.dtsi +++ b/arch/arm/boot/dts/qcom-ipq4019.dtsi @@ -23,9 +23,27 @@ compatible = "qcom,ipq4019"; interrupt-parent = <&intc>; + reserved-memory { + #address-cells = <0x1>; + #size-cells = <0x1>; + ranges; + + smem_region: smem@87e00000 { + reg = <0x87e00000 0x080000>; + no-map; + }; + + tz@87e80000 { + reg = <0x87e80000 0x180000>; + no-map; + }; + }; + aliases { - spi0 = &spi_0; - i2c0 = &i2c_0; + spi0 = &blsp1_spi1; + spi1 = &blsp1_spi2; + i2c0 = &blsp1_i2c3; + i2c1 = &blsp1_i2c4; }; cpus { @@ -45,7 +63,7 @@ 48000 1100000 200000 1100000 500000 1100000 - 666000 1100000 + 716000 1100000 >; clock-latency = <256000>; }; @@ -104,6 +122,12 @@ }; }; + firmware { + scm { + compatible = "qcom,scm-ipq4019"; + }; + }; + timer { compatible = "arm,armv7-timer"; interrupts = <1 2 0xf08>, @@ -149,13 +173,13 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; - interrupts = <0 208 0>; + interrupts = ; }; blsp_dma: dma@7884000 { compatible = "qcom,bam-v1.7.0"; reg = <0x07884000 0x23000>; - interrupts = ; + interrupts = ; clocks = <&gcc GCC_BLSP1_AHB_CLK>; clock-names = "bam_clk"; #dma-cells = <1>; @@ -163,7 +187,7 @@ status = "disabled"; }; - spi_0: spi@78b5000 { + blsp1_spi1: spi@78b5000 { /* BLSP1 QUP1 */ compatible = "qcom,spi-qup-v2.2.1"; reg = <0x78b5000 0x600>; interrupts = ; @@ -172,10 +196,26 @@ clock-names = "core", "iface"; #address-cells = <1>; #size-cells = <0>; + dmas = <&blsp_dma 5>, <&blsp_dma 4>; + dma-names = "rx", "tx"; + status = "disabled"; + }; + + blsp1_spi2: spi@78b6000 { /* BLSP1 QUP2 */ + compatible = "qcom,spi-qup-v2.2.1"; + reg = <0x78b6000 0x600>; + interrupts = ; + clocks = <&gcc GCC_BLSP1_QUP2_SPI_APPS_CLK>, + <&gcc GCC_BLSP1_AHB_CLK>; + clock-names = "core", "iface"; + #address-cells = <1>; + #size-cells = <0>; + dmas = <&blsp_dma 7>, <&blsp_dma 6>; + dma-names = "rx", "tx"; status = "disabled"; }; - i2c_0: i2c@78b7000 { + blsp1_i2c3: i2c@78b7000 { /* BLSP1 QUP3 */ compatible = "qcom,i2c-qup-v2.2.1"; reg = <0x78b7000 0x600>; interrupts = ; @@ -184,14 +224,29 @@ clock-names = "iface", "core"; #address-cells = <1>; #size-cells = <0>; + dmas = <&blsp_dma 9>, <&blsp_dma 8>; + dma-names = "rx", "tx"; status = "disabled"; }; + blsp1_i2c4: i2c@78b8000 { /* BLSP1 QUP4 */ + compatible = "qcom,i2c-qup-v2.2.1"; + reg = <0x78b8000 0x600>; + interrupts = ; + clocks = <&gcc GCC_BLSP1_AHB_CLK>, + <&gcc GCC_BLSP1_QUP2_I2C_APPS_CLK>; + clock-names = "iface", "core"; + #address-cells = <1>; + #size-cells = <0>; + dmas = <&blsp_dma 11>, <&blsp_dma 10>; + dma-names = "rx", "tx"; + status = "disabled"; + }; cryptobam: dma@8e04000 { compatible = "qcom,bam-v1.7.0"; reg = <0x08e04000 0x20000>; - interrupts = ; + interrupts = ; clocks = <&gcc GCC_CRYPTO_AHB_CLK>; clock-names = "bam_clk"; #dma-cells = <1>; @@ -256,10 +311,10 @@ regulator; }; - serial@78af000 { + blsp1_uart1: serial@78af000 { compatible = "qcom,msm-uartdm-v1.4", "qcom,msm-uartdm"; reg = <0x78af000 0x200>; - interrupts = <0 107 0>; + interrupts = ; status = "disabled"; clocks = <&gcc GCC_BLSP1_UART1_APPS_CLK>, <&gcc GCC_BLSP1_AHB_CLK>; @@ -268,10 +323,10 @@ dma-names = "rx", "tx"; }; - serial@78b0000 { + blsp1_uart2: serial@78b0000 { compatible = "qcom,msm-uartdm-v1.4", "qcom,msm-uartdm"; reg = <0x78b0000 0x200>; - interrupts = <0 108 0>; + interrupts = ; status = "disabled"; clocks = <&gcc GCC_BLSP1_UART2_APPS_CLK>, <&gcc GCC_BLSP1_AHB_CLK>; @@ -293,6 +348,101 @@ reg = <0x4ab000 0x4>; }; + pcie0: pci@40000000 { + compatible = "qcom,pcie-ipq4019", "snps,dw-pcie"; + reg = <0x40000000 0xf1d + 0x40000f20 0xa8 + 0x80000 0x2000 + 0x40100000 0x1000>; + reg-names = "dbi", "elbi", "parf", "config"; + device_type = "pci"; + linux,pci-domain = <0>; + bus-range = <0x00 0xff>; + num-lanes = <1>; + #address-cells = <3>; + #size-cells = <2>; + + ranges = <0x81000000 0 0x40200000 0x40200000 0 0x00100000 + 0x82000000 0 0x48000000 0x48000000 0 0x10000000>; + + interrupts = ; + interrupt-names = "msi"; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0x7>; + interrupt-map = <0 0 0 1 &intc 0 142 IRQ_TYPE_LEVEL_HIGH>, /* int_a */ + <0 0 0 2 &intc 0 143 IRQ_TYPE_LEVEL_HIGH>, /* int_b */ + <0 0 0 3 &intc 0 144 IRQ_TYPE_LEVEL_HIGH>, /* int_c */ + <0 0 0 4 &intc 0 145 IRQ_TYPE_LEVEL_HIGH>; /* int_d */ + clocks = <&gcc GCC_PCIE_AHB_CLK>, + <&gcc GCC_PCIE_AXI_M_CLK>, + <&gcc GCC_PCIE_AXI_S_CLK>; + clock-names = "aux", + "master_bus", + "slave_bus"; + + resets = <&gcc PCIE_AXI_M_ARES>, + <&gcc PCIE_AXI_S_ARES>, + <&gcc PCIE_PIPE_ARES>, + <&gcc PCIE_AXI_M_VMIDMT_ARES>, + <&gcc PCIE_AXI_S_XPU_ARES>, + <&gcc PCIE_PARF_XPU_ARES>, + <&gcc PCIE_PHY_ARES>, + <&gcc PCIE_AXI_M_STICKY_ARES>, + <&gcc PCIE_PIPE_STICKY_ARES>, + <&gcc PCIE_PWR_ARES>, + <&gcc PCIE_AHB_ARES>, + <&gcc PCIE_PHY_AHB_ARES>; + reset-names = "axi_m", + "axi_s", + "pipe", + "axi_m_vmid", + "axi_s_xpu", + "parf", + "phy", + "axi_m_sticky", + "pipe_sticky", + "pwr", + "ahb", + "phy_ahb"; + + status = "disabled"; + }; + + qpic_bam: dma@7984000 { + compatible = "qcom,bam-v1.7.0"; + reg = <0x7984000 0x1a000>; + interrupts = ; + clocks = <&gcc GCC_QPIC_CLK>; + clock-names = "bam_clk"; + #dma-cells = <1>; + qcom,ee = <0>; + status = "disabled"; + }; + + nand: qpic-nand@79b0000 { + compatible = "qcom,ipq4019-nand"; + reg = <0x79b0000 0x1000>; + #address-cells = <1>; + #size-cells = <0>; + clocks = <&gcc GCC_QPIC_CLK>, + <&gcc GCC_QPIC_AHB_CLK>; + clock-names = "core", "aon"; + + dmas = <&qpic_bam 0>, + <&qpic_bam 1>, + <&qpic_bam 2>; + dma-names = "tx", "rx", "cmd"; + status = "disabled"; + + nand@0 { + reg = <0>; + + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + nand-bus-width = <8>; + }; + }; + wifi0: wifi@a000000 { compatible = "qcom,ipq4019-wifi"; reg = <0xa000000 0x200000>; @@ -326,7 +476,7 @@ , , , - ; + ; interrupt-names = "msi0", "msi1", "msi2", "msi3", "msi4", "msi5", "msi6", "msi7", "msi8", "msi9", "msi10", "msi11", @@ -368,7 +518,7 @@ , , , - ; + ; interrupt-names = "msi0", "msi1", "msi2", "msi3", "msi4", "msi5", "msi6", "msi7", "msi8", "msi9", "msi10", "msi11", diff --git a/arch/arm/boot/dts/qcom-msm8660.dtsi b/arch/arm/boot/dts/qcom-msm8660.dtsi index 33030f9419fe..70698941f64c 100644 --- a/arch/arm/boot/dts/qcom-msm8660.dtsi +++ b/arch/arm/boot/dts/qcom-msm8660.dtsi @@ -452,7 +452,7 @@ clock-names = "ram"; rpmcc: clock-controller { - compatible = "qcom,rpmcc-apq8660", "qcom,rpmcc"; + compatible = "qcom,rpmcc-msm8660", "qcom,rpmcc"; #clock-cells = <1>; }; diff --git a/arch/arm/boot/dts/qcom-msm8974-sony-xperia-amami.dts b/arch/arm/boot/dts/qcom-msm8974-sony-xperia-amami.dts new file mode 100644 index 000000000000..5669f5f58a86 --- /dev/null +++ b/arch/arm/boot/dts/qcom-msm8974-sony-xperia-amami.dts @@ -0,0 +1,436 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "qcom-msm8974.dtsi" +#include "qcom-pm8841.dtsi" +#include "qcom-pm8941.dtsi" +#include +#include +#include + +/ { + model = "Sony Xperia Z1 Compact"; + compatible = "sony,xperia-amami", "qcom,msm8974"; + + aliases { + serial0 = &blsp1_uart2; + }; + + chosen { + stdout-path = "serial0:115200n8"; + }; + + gpio-keys { + compatible = "gpio-keys"; + input-name = "gpio-keys"; + + pinctrl-names = "default"; + pinctrl-0 = <&gpio_keys_pin_a>; + + volume-down { + label = "volume_down"; + gpios = <&pm8941_gpios 2 GPIO_ACTIVE_LOW>; + linux,input-type = <1>; + linux,code = ; + }; + + camera-snapshot { + label = "camera_snapshot"; + gpios = <&pm8941_gpios 3 GPIO_ACTIVE_LOW>; + linux,input-type = <1>; + linux,code = ; + }; + + camera-focus { + label = "camera_focus"; + gpios = <&pm8941_gpios 4 GPIO_ACTIVE_LOW>; + linux,input-type = <1>; + linux,code = ; + }; + + volume-up { + label = "volume_up"; + gpios = <&pm8941_gpios 5 GPIO_ACTIVE_LOW>; + linux,input-type = <1>; + linux,code = ; + }; + }; + + memory@0 { + reg = <0 0x40000000>, <0x40000000 0x40000000>; + device_type = "memory"; + }; + + smd { + rpm { + rpm_requests { + pm8841-regulators { + s1 { + regulator-min-microvolt = <675000>; + regulator-max-microvolt = <1050000>; + }; + + s2 { + regulator-min-microvolt = <500000>; + regulator-max-microvolt = <1050000>; + }; + + s3 { + regulator-min-microvolt = <500000>; + regulator-max-microvolt = <1050000>; + }; + + s4 { + regulator-min-microvolt = <500000>; + regulator-max-microvolt = <1050000>; + }; + }; + + pm8941-regulators { + vdd_l1_l3-supply = <&pm8941_s1>; + vdd_l2_lvs1_2_3-supply = <&pm8941_s3>; + vdd_l4_l11-supply = <&pm8941_s1>; + vdd_l5_l7-supply = <&pm8941_s2>; + vdd_l6_l12_l14_l15-supply = <&pm8941_s2>; + vdd_l9_l10_l17_l22-supply = <&vreg_boost>; + vdd_l13_l20_l23_l24-supply = <&vreg_boost>; + vdd_l21-supply = <&vreg_boost>; + + s1 { + regulator-min-microvolt = <1300000>; + regulator-max-microvolt = <1300000>; + regulator-always-on; + regulator-boot-on; + }; + + s2 { + regulator-min-microvolt = <2150000>; + regulator-max-microvolt = <2150000>; + regulator-boot-on; + }; + + s3 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-always-on; + regulator-boot-on; + }; + + s4 { + regulator-min-microvolt = <5000000>; + regulator-max-microvolt = <5000000>; + }; + + l1 { + regulator-min-microvolt = <1225000>; + regulator-max-microvolt = <1225000>; + + regulator-always-on; + regulator-boot-on; + }; + + l2 { + regulator-min-microvolt = <1200000>; + regulator-max-microvolt = <1200000>; + }; + + l3 { + regulator-min-microvolt = <1200000>; + regulator-max-microvolt = <1200000>; + }; + + l4 { + regulator-min-microvolt = <1225000>; + regulator-max-microvolt = <1225000>; + }; + + l5 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + }; + + l6 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + + regulator-boot-on; + }; + + l7 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + + regulator-boot-on; + }; + + l8 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + }; + + l9 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <2950000>; + }; + + l11 { + regulator-min-microvolt = <1300000>; + regulator-max-microvolt = <1350000>; + }; + + l12 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + + regulator-always-on; + regulator-boot-on; + }; + + l13 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <2950000>; + + regulator-boot-on; + }; + + l14 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + }; + + l15 { + regulator-min-microvolt = <2050000>; + regulator-max-microvolt = <2050000>; + }; + + l16 { + regulator-min-microvolt = <2700000>; + regulator-max-microvolt = <2700000>; + }; + + l17 { + regulator-min-microvolt = <2700000>; + regulator-max-microvolt = <2700000>; + }; + + l18 { + regulator-min-microvolt = <2850000>; + regulator-max-microvolt = <2850000>; + }; + + l19 { + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + }; + + l20 { + regulator-min-microvolt = <2950000>; + regulator-max-microvolt = <2950000>; + + regulator-allow-set-load; + regulator-boot-on; + regulator-system-load = <200000>; + }; + + l21 { + regulator-min-microvolt = <2950000>; + regulator-max-microvolt = <2950000>; + + regulator-boot-on; + }; + + l22 { + regulator-min-microvolt = <3000000>; + regulator-max-microvolt = <3000000>; + }; + + l23 { + regulator-min-microvolt = <2800000>; + regulator-max-microvolt = <2800000>; + }; + + l24 { + regulator-min-microvolt = <3075000>; + regulator-max-microvolt = <3075000>; + + regulator-boot-on; + }; + }; + }; + }; + }; +}; + +&soc { + sdhci@f9824900 { + status = "ok"; + + vmmc-supply = <&pm8941_l20>; + vqmmc-supply = <&pm8941_s3>; + + bus-width = <8>; + non-removable; + + pinctrl-names = "default"; + pinctrl-0 = <&sdhc1_pin_a>; + }; + + sdhci@f98a4900 { + status = "ok"; + + bus-width = <4>; + + vmmc-supply = <&pm8941_l21>; + vqmmc-supply = <&pm8941_l13>; + + cd-gpios = <&msmgpio 62 GPIO_ACTIVE_LOW>; + + pinctrl-names = "default"; + pinctrl-0 = <&sdhc2_pin_a>, <&sdhc2_cd_pin_a>; + }; + + serial@f991e000 { + status = "ok"; + + pinctrl-names = "default"; + pinctrl-0 = <&blsp1_uart2_pin_a>; + }; + + + pinctrl@fd510000 { + blsp1_uart2_pin_a: blsp1-uart2-pin-active { + rx { + pins = "gpio5"; + function = "blsp_uart2"; + + drive-strength = <2>; + bias-pull-up; + }; + + tx { + pins = "gpio4"; + function = "blsp_uart2"; + + drive-strength = <4>; + bias-disable; + }; + }; + + i2c2_pins: i2c2 { + mux { + pins = "gpio6", "gpio7"; + function = "blsp_i2c2"; + + drive-strength = <2>; + bias-disable; + }; + }; + + sdhc1_pin_a: sdhc1-pin-active { + clk { + pins = "sdc1_clk"; + drive-strength = <16>; + bias-disable; + }; + + cmd-data { + pins = "sdc1_cmd", "sdc1_data"; + drive-strength = <10>; + bias-pull-up; + }; + }; + + sdhc2_cd_pin_a: sdhc2-cd-pin-active { + pins = "gpio62"; + function = "gpio"; + + drive-strength = <2>; + bias-disable; + }; + + sdhc2_pin_a: sdhc2-pin-active { + clk { + pins = "sdc2_clk"; + drive-strength = <10>; + bias-disable; + }; + + cmd-data { + pins = "sdc2_cmd", "sdc2_data"; + drive-strength = <6>; + bias-pull-up; + }; + }; + }; + + dma-controller@f9944000 { + qcom,controlled-remotely; + }; + + usb@f9a55000 { + status = "ok"; + + phys = <&usb_hs1_phy>; + phy-select = <&tcsr 0xb000 0>; + extcon = <&smbb>, <&usb_id>; + vbus-supply = <&chg_otg>; + + hnp-disable; + srp-disable; + adp-disable; + + ulpi { + phy@a { + status = "ok"; + + v1p8-supply = <&pm8941_l6>; + v3p3-supply = <&pm8941_l24>; + + extcon = <&smbb>; + qcom,init-seq = /bits/ 8 <0x1 0x64>; + }; + }; + }; +}; + +&spmi_bus { + pm8941@0 { + charger@1000 { + qcom,fast-charge-safe-current = <1300000>; + qcom,fast-charge-current-limit = <1300000>; + qcom,dc-current-limit = <1300000>; + qcom,fast-charge-safe-voltage = <4400000>; + qcom,fast-charge-high-threshold-voltage = <4350000>; + qcom,fast-charge-low-threshold-voltage = <3400000>; + qcom,auto-recharge-threshold-voltage = <4200000>; + qcom,minimum-input-voltage = <4300000>; + }; + + gpios@c000 { + gpio_keys_pin_a: gpio-keys-active { + pins = "gpio2", "gpio3", "gpio4", "gpio5"; + function = "normal"; + + bias-pull-up; + power-source = ; + }; + }; + + coincell@2800 { + status = "ok"; + qcom,rset-ohms = <2100>; + qcom,vset-millivolts = <3000>; + }; + }; + + pm8941@1 { + wled@d800 { + status = "ok"; + + qcom,cs-out; + qcom,current-limit = <20>; + qcom,current-boost-limit = <805>; + qcom,switching-freq = <1600>; + qcom,ovp = <29>; + qcom,num-strings = <2>; + }; + }; +}; diff --git a/arch/arm/boot/dts/qcom-pm8941.dtsi b/arch/arm/boot/dts/qcom-pm8941.dtsi index 1d5ef55c7ee5..2515c5c217ac 100644 --- a/arch/arm/boot/dts/qcom-pm8941.dtsi +++ b/arch/arm/boot/dts/qcom-pm8941.dtsi @@ -139,6 +139,9 @@ #size-cells = <0>; #io-channel-cells = <1>; + bat_temp { + reg = ; + }; die_temp { reg = ; }; @@ -154,6 +157,9 @@ ref_vdd { reg = ; }; + vbat_sns { + reg = ; + }; }; pm8941_iadc: iadc@3600 { diff --git a/arch/arm/boot/dts/r7s72100.dtsi b/arch/arm/boot/dts/r7s72100.dtsi index ab9645a42eca..a54822e97bac 100644 --- a/arch/arm/boot/dts/r7s72100.dtsi +++ b/arch/arm/boot/dts/r7s72100.dtsi @@ -15,7 +15,6 @@ / { compatible = "renesas,r7s72100"; - interrupt-parent = <&gic>; #address-cells = <1>; #size-cells = <1>; @@ -31,61 +30,370 @@ spi4 = &spi4; }; - clocks { - ranges; + /* Fixed factor clocks */ + b_clk: b { + #clock-cells = <0>; + compatible = "fixed-factor-clock"; + clocks = <&cpg_clocks R7S72100_CLK_PLL>; + clock-mult = <1>; + clock-div = <3>; + }; + + cpus { + #address-cells = <1>; + #size-cells = <0>; + + cpu@0 { + device_type = "cpu"; + compatible = "arm,cortex-a9"; + reg = <0>; + clock-frequency = <400000000>; + clocks = <&cpg_clocks R7S72100_CLK_I>; + next-level-cache = <&L2>; + }; + }; + + /* External clocks */ + extal_clk: extal { + #clock-cells = <0>; + compatible = "fixed-clock"; + /* If clk present, value must be set by board */ + clock-frequency = <0>; + }; + + p0_clk: p0 { + #clock-cells = <0>; + compatible = "fixed-factor-clock"; + clocks = <&cpg_clocks R7S72100_CLK_PLL>; + clock-mult = <1>; + clock-div = <12>; + }; + + p1_clk: p1 { + #clock-cells = <0>; + compatible = "fixed-factor-clock"; + clocks = <&cpg_clocks R7S72100_CLK_PLL>; + clock-mult = <1>; + clock-div = <6>; + }; + + pmu { + compatible = "arm,cortex-a9-pmu"; + interrupts-extended = <&gic GIC_PPI 0 IRQ_TYPE_LEVEL_HIGH>; + }; + + rtc_x1_clk: rtc_x1 { + #clock-cells = <0>; + compatible = "fixed-clock"; + /* If clk present, value must be set by board to 32678 */ + clock-frequency = <0>; + }; + + rtc_x3_clk: rtc_x3 { + #clock-cells = <0>; + compatible = "fixed-clock"; + /* If clk present, value must be set by board to 4000000 */ + clock-frequency = <0>; + }; + + soc { + compatible = "simple-bus"; + interrupt-parent = <&gic>; + #address-cells = <1>; #size-cells = <1>; + ranges; + + L2: cache-controller@3ffff000 { + compatible = "arm,pl310-cache"; + reg = <0x3ffff000 0x1000>; + interrupts = ; + arm,early-bresp-disable; + arm,full-line-zero-disable; + cache-unified; + cache-level = <2>; + }; + + scif0: serial@e8007000 { + compatible = "renesas,scif-r7s72100", "renesas,scif"; + reg = <0xe8007000 64>; + interrupts = , + , + , + ; + clocks = <&mstp4_clks R7S72100_CLK_SCIF0>; + clock-names = "fck"; + power-domains = <&cpg_clocks>; + status = "disabled"; + }; + + scif1: serial@e8007800 { + compatible = "renesas,scif-r7s72100", "renesas,scif"; + reg = <0xe8007800 64>; + interrupts = , + , + , + ; + clocks = <&mstp4_clks R7S72100_CLK_SCIF1>; + clock-names = "fck"; + power-domains = <&cpg_clocks>; + status = "disabled"; + }; + + scif2: serial@e8008000 { + compatible = "renesas,scif-r7s72100", "renesas,scif"; + reg = <0xe8008000 64>; + interrupts = , + , + , + ; + clocks = <&mstp4_clks R7S72100_CLK_SCIF2>; + clock-names = "fck"; + power-domains = <&cpg_clocks>; + status = "disabled"; + }; + + scif3: serial@e8008800 { + compatible = "renesas,scif-r7s72100", "renesas,scif"; + reg = <0xe8008800 64>; + interrupts = , + , + , + ; + clocks = <&mstp4_clks R7S72100_CLK_SCIF3>; + clock-names = "fck"; + power-domains = <&cpg_clocks>; + status = "disabled"; + }; - /* External clocks */ - extal_clk: extal { - #clock-cells = <0>; - compatible = "fixed-clock"; - /* If clk present, value must be set by board */ - clock-frequency = <0>; + scif4: serial@e8009000 { + compatible = "renesas,scif-r7s72100", "renesas,scif"; + reg = <0xe8009000 64>; + interrupts = , + , + , + ; + clocks = <&mstp4_clks R7S72100_CLK_SCIF4>; + clock-names = "fck"; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - usb_x1_clk: usb_x1 { - #clock-cells = <0>; - compatible = "fixed-clock"; - /* If clk present, value must be set by board */ - clock-frequency = <0>; + scif5: serial@e8009800 { + compatible = "renesas,scif-r7s72100", "renesas,scif"; + reg = <0xe8009800 64>; + interrupts = , + , + , + ; + clocks = <&mstp4_clks R7S72100_CLK_SCIF5>; + clock-names = "fck"; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - rtc_x1_clk: rtc_x1 { - #clock-cells = <0>; - compatible = "fixed-clock"; - /* If clk present, value must be set by board to 32678 */ - clock-frequency = <0>; + scif6: serial@e800a000 { + compatible = "renesas,scif-r7s72100", "renesas,scif"; + reg = <0xe800a000 64>; + interrupts = , + , + , + ; + clocks = <&mstp4_clks R7S72100_CLK_SCIF6>; + clock-names = "fck"; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - rtc_x3_clk: rtc_x3 { - #clock-cells = <0>; - compatible = "fixed-clock"; - /* If clk present, value must be set by board to 4000000 */ - clock-frequency = <0>; + scif7: serial@e800a800 { + compatible = "renesas,scif-r7s72100", "renesas,scif"; + reg = <0xe800a800 64>; + interrupts = , + , + , + ; + clocks = <&mstp4_clks R7S72100_CLK_SCIF7>; + clock-names = "fck"; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - /* Fixed factor clocks */ - b_clk: b { - #clock-cells = <0>; - compatible = "fixed-factor-clock"; - clocks = <&cpg_clocks R7S72100_CLK_PLL>; - clock-mult = <1>; - clock-div = <3>; + spi0: spi@e800c800 { + compatible = "renesas,rspi-r7s72100", "renesas,rspi-rz"; + reg = <0xe800c800 0x24>; + interrupts = , + , + ; + interrupt-names = "error", "rx", "tx"; + clocks = <&mstp10_clks R7S72100_CLK_SPI0>; + power-domains = <&cpg_clocks>; + num-cs = <1>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; }; - p1_clk: p1 { - #clock-cells = <0>; - compatible = "fixed-factor-clock"; - clocks = <&cpg_clocks R7S72100_CLK_PLL>; - clock-mult = <1>; - clock-div = <6>; + + spi1: spi@e800d000 { + compatible = "renesas,rspi-r7s72100", "renesas,rspi-rz"; + reg = <0xe800d000 0x24>; + interrupts = , + , + ; + interrupt-names = "error", "rx", "tx"; + clocks = <&mstp10_clks R7S72100_CLK_SPI1>; + power-domains = <&cpg_clocks>; + num-cs = <1>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + spi2: spi@e800d800 { + compatible = "renesas,rspi-r7s72100", "renesas,rspi-rz"; + reg = <0xe800d800 0x24>; + interrupts = , + , + ; + interrupt-names = "error", "rx", "tx"; + clocks = <&mstp10_clks R7S72100_CLK_SPI2>; + power-domains = <&cpg_clocks>; + num-cs = <1>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; }; - p0_clk: p0 { - #clock-cells = <0>; - compatible = "fixed-factor-clock"; - clocks = <&cpg_clocks R7S72100_CLK_PLL>; - clock-mult = <1>; - clock-div = <12>; + + spi3: spi@e800e000 { + compatible = "renesas,rspi-r7s72100", "renesas,rspi-rz"; + reg = <0xe800e000 0x24>; + interrupts = , + , + ; + interrupt-names = "error", "rx", "tx"; + clocks = <&mstp10_clks R7S72100_CLK_SPI3>; + power-domains = <&cpg_clocks>; + num-cs = <1>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + spi4: spi@e800e800 { + compatible = "renesas,rspi-r7s72100", "renesas,rspi-rz"; + reg = <0xe800e800 0x24>; + interrupts = , + , + ; + interrupt-names = "error", "rx", "tx"; + clocks = <&mstp10_clks R7S72100_CLK_SPI4>; + power-domains = <&cpg_clocks>; + num-cs = <1>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + usbhs0: usb@e8010000 { + compatible = "renesas,usbhs-r7s72100", "renesas,rza1-usbhs"; + reg = <0xe8010000 0x1a0>; + interrupts = ; + clocks = <&mstp7_clks R7S72100_CLK_USB0>; + renesas,buswait = <4>; + power-domains = <&cpg_clocks>; + status = "disabled"; + }; + + usbhs1: usb@e8207000 { + compatible = "renesas,usbhs-r7s72100", "renesas,rza1-usbhs"; + reg = <0xe8207000 0x1a0>; + interrupts = ; + clocks = <&mstp7_clks R7S72100_CLK_USB1>; + renesas,buswait = <4>; + power-domains = <&cpg_clocks>; + status = "disabled"; + }; + + mmcif: mmc@e804c800 { + compatible = "renesas,mmcif-r7s72100", "renesas,sh-mmcif"; + reg = <0xe804c800 0x80>; + interrupts = ; + clocks = <&mstp8_clks R7S72100_CLK_MMCIF>; + power-domains = <&cpg_clocks>; + reg-io-width = <4>; + bus-width = <8>; + status = "disabled"; + }; + + sdhi0: sd@e804e000 { + compatible = "renesas,sdhi-r7s72100"; + reg = <0xe804e000 0x100>; + interrupts = ; + + clocks = <&mstp12_clks R7S72100_CLK_SDHI00>, + <&mstp12_clks R7S72100_CLK_SDHI01>; + clock-names = "core", "cd"; + power-domains = <&cpg_clocks>; + cap-sd-highspeed; + cap-sdio-irq; + status = "disabled"; + }; + + sdhi1: sd@e804e800 { + compatible = "renesas,sdhi-r7s72100"; + reg = <0xe804e800 0x100>; + interrupts = ; + + clocks = <&mstp12_clks R7S72100_CLK_SDHI10>, + <&mstp12_clks R7S72100_CLK_SDHI11>; + clock-names = "core", "cd"; + power-domains = <&cpg_clocks>; + cap-sd-highspeed; + cap-sdio-irq; + status = "disabled"; + }; + + gic: interrupt-controller@e8201000 { + compatible = "arm,pl390"; + #interrupt-cells = <3>; + #address-cells = <0>; + interrupt-controller; + reg = <0xe8201000 0x1000>, + <0xe8202000 0x1000>; + }; + + ether: ethernet@e8203000 { + compatible = "renesas,ether-r7s72100"; + reg = <0xe8203000 0x800>, + <0xe8204800 0x200>; + interrupts = ; + clocks = <&mstp7_clks R7S72100_CLK_ETHER>; + power-domains = <&cpg_clocks>; + phy-mode = "mii"; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + ceu: camera@e8210000 { + reg = <0xe8210000 0x3000>; + compatible = "renesas,r7s72100-ceu"; + interrupts = ; + clocks = <&mstp6_clks R7S72100_CLK_CEU>; + power-domains = <&cpg_clocks>; + status = "disabled"; + }; + + wdt: watchdog@fcfe0000 { + compatible = "renesas,r7s72100-wdt", "renesas,rza-wdt"; + reg = <0xfcfe0000 0x6>; + interrupts = ; + clocks = <&p0_clk>; }; /* Special CPG clocks */ @@ -135,9 +443,9 @@ #clock-cells = <1>; compatible = "renesas,r7s72100-mstp-clocks", "renesas,cpg-mstp-clocks"; reg = <0xfcfe042c 4>; - clocks = <&p0_clk>; - clock-indices = ; - clock-output-names = "rtc"; + clocks = <&b_clk>, <&p0_clk>; + clock-indices = ; + clock-output-names = "ceu", "rtc"; }; mstp7_clks: mstp7_clks@fcfe0430 { @@ -192,479 +500,209 @@ >; clock-output-names = "sdhi00", "sdhi01", "sdhi10", "sdhi11"; }; - }; - - cpus { - #address-cells = <1>; - #size-cells = <0>; - - cpu@0 { - device_type = "cpu"; - compatible = "arm,cortex-a9"; - reg = <0>; - clock-frequency = <400000000>; - clocks = <&cpg_clocks R7S72100_CLK_I>; - next-level-cache = <&L2>; - }; - }; - - pinctrl: pin-controller@fcfe3000 { - compatible = "renesas,r7s72100-ports"; - - reg = <0xfcfe3000 0x4230>; - - port0: gpio-0 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 0 6>; - }; - - port1: gpio-1 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 16 16>; - }; - port2: gpio-2 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 32 16>; + pinctrl: pin-controller@fcfe3000 { + compatible = "renesas,r7s72100-ports"; + + reg = <0xfcfe3000 0x4230>; + + port0: gpio-0 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 0 6>; + }; + + port1: gpio-1 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 16 16>; + }; + + port2: gpio-2 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 32 16>; + }; + + port3: gpio-3 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 48 16>; + }; + + port4: gpio-4 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 64 16>; + }; + + port5: gpio-5 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 80 11>; + }; + + port6: gpio-6 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 96 16>; + }; + + port7: gpio-7 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 112 16>; + }; + + port8: gpio-8 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 128 16>; + }; + + port9: gpio-9 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 144 8>; + }; + + port10: gpio-10 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 160 16>; + }; + + port11: gpio-11 { + gpio-controller; + #gpio-cells = <2>; + gpio-ranges = <&pinctrl 0 176 16>; + }; }; - port3: gpio-3 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 48 16>; + ostm0: timer@fcfec000 { + compatible = "renesas,r7s72100-ostm", "renesas,ostm"; + reg = <0xfcfec000 0x30>; + interrupts = ; + clocks = <&mstp5_clks R7S72100_CLK_OSTM0>; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - port4: gpio-4 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 64 16>; + ostm1: timer@fcfec400 { + compatible = "renesas,r7s72100-ostm", "renesas,ostm"; + reg = <0xfcfec400 0x30>; + interrupts = ; + clocks = <&mstp5_clks R7S72100_CLK_OSTM1>; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - port5: gpio-5 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 80 11>; + i2c0: i2c@fcfee000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,riic-r7s72100", "renesas,riic-rz"; + reg = <0xfcfee000 0x44>; + interrupts = , + , + , + , + , + , + , + ; + clocks = <&mstp9_clks R7S72100_CLK_I2C0>; + clock-frequency = <100000>; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - port6: gpio-6 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 96 16>; + i2c1: i2c@fcfee400 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,riic-r7s72100", "renesas,riic-rz"; + reg = <0xfcfee400 0x44>; + interrupts = , + , + , + , + , + , + , + ; + clocks = <&mstp9_clks R7S72100_CLK_I2C1>; + clock-frequency = <100000>; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - port7: gpio-7 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 112 16>; + i2c2: i2c@fcfee800 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,riic-r7s72100", "renesas,riic-rz"; + reg = <0xfcfee800 0x44>; + interrupts = , + , + , + , + , + , + , + ; + clocks = <&mstp9_clks R7S72100_CLK_I2C2>; + clock-frequency = <100000>; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - port8: gpio-8 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 128 16>; + i2c3: i2c@fcfeec00 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,riic-r7s72100", "renesas,riic-rz"; + reg = <0xfcfeec00 0x44>; + interrupts = , + , + , + , + , + , + , + ; + clocks = <&mstp9_clks R7S72100_CLK_I2C3>; + clock-frequency = <100000>; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - port9: gpio-9 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 144 8>; + mtu2: timer@fcff0000 { + compatible = "renesas,mtu2-r7s72100", "renesas,mtu2"; + reg = <0xfcff0000 0x400>; + interrupts = ; + interrupt-names = "tgi0a"; + clocks = <&mstp3_clks R7S72100_CLK_MTU2>; + clock-names = "fck"; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - port10: gpio-10 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 160 16>; + rtc: rtc@fcff1000 { + compatible = "renesas,r7s72100-rtc", "renesas,sh-rtc"; + reg = <0xfcff1000 0x2e>; + interrupts = , + , + ; + interrupt-names = "alarm", "period", "carry"; + clocks = <&mstp6_clks R7S72100_CLK_RTC>, <&rtc_x1_clk>, + <&rtc_x3_clk>, <&extal_clk>; + clock-names = "fck", "rtc_x1", "rtc_x3", "extal"; + power-domains = <&cpg_clocks>; + status = "disabled"; }; - - port11: gpio-11 { - gpio-controller; - #gpio-cells = <2>; - gpio-ranges = <&pinctrl 0 176 16>; - }; - }; - - scif0: serial@e8007000 { - compatible = "renesas,scif-r7s72100", "renesas,scif"; - reg = <0xe8007000 64>; - interrupts = , - , - , - ; - clocks = <&mstp4_clks R7S72100_CLK_SCIF0>; - clock-names = "fck"; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - scif1: serial@e8007800 { - compatible = "renesas,scif-r7s72100", "renesas,scif"; - reg = <0xe8007800 64>; - interrupts = , - , - , - ; - clocks = <&mstp4_clks R7S72100_CLK_SCIF1>; - clock-names = "fck"; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - scif2: serial@e8008000 { - compatible = "renesas,scif-r7s72100", "renesas,scif"; - reg = <0xe8008000 64>; - interrupts = , - , - , - ; - clocks = <&mstp4_clks R7S72100_CLK_SCIF2>; - clock-names = "fck"; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - scif3: serial@e8008800 { - compatible = "renesas,scif-r7s72100", "renesas,scif"; - reg = <0xe8008800 64>; - interrupts = , - , - , - ; - clocks = <&mstp4_clks R7S72100_CLK_SCIF3>; - clock-names = "fck"; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - scif4: serial@e8009000 { - compatible = "renesas,scif-r7s72100", "renesas,scif"; - reg = <0xe8009000 64>; - interrupts = , - , - , - ; - clocks = <&mstp4_clks R7S72100_CLK_SCIF4>; - clock-names = "fck"; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - scif5: serial@e8009800 { - compatible = "renesas,scif-r7s72100", "renesas,scif"; - reg = <0xe8009800 64>; - interrupts = , - , - , - ; - clocks = <&mstp4_clks R7S72100_CLK_SCIF5>; - clock-names = "fck"; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - scif6: serial@e800a000 { - compatible = "renesas,scif-r7s72100", "renesas,scif"; - reg = <0xe800a000 64>; - interrupts = , - , - , - ; - clocks = <&mstp4_clks R7S72100_CLK_SCIF6>; - clock-names = "fck"; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - scif7: serial@e800a800 { - compatible = "renesas,scif-r7s72100", "renesas,scif"; - reg = <0xe800a800 64>; - interrupts = , - , - , - ; - clocks = <&mstp4_clks R7S72100_CLK_SCIF7>; - clock-names = "fck"; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - spi0: spi@e800c800 { - compatible = "renesas,rspi-r7s72100", "renesas,rspi-rz"; - reg = <0xe800c800 0x24>; - interrupts = , - , - ; - interrupt-names = "error", "rx", "tx"; - clocks = <&mstp10_clks R7S72100_CLK_SPI0>; - power-domains = <&cpg_clocks>; - num-cs = <1>; - #address-cells = <1>; - #size-cells = <0>; - status = "disabled"; - }; - - spi1: spi@e800d000 { - compatible = "renesas,rspi-r7s72100", "renesas,rspi-rz"; - reg = <0xe800d000 0x24>; - interrupts = , - , - ; - interrupt-names = "error", "rx", "tx"; - clocks = <&mstp10_clks R7S72100_CLK_SPI1>; - power-domains = <&cpg_clocks>; - num-cs = <1>; - #address-cells = <1>; - #size-cells = <0>; - status = "disabled"; - }; - - spi2: spi@e800d800 { - compatible = "renesas,rspi-r7s72100", "renesas,rspi-rz"; - reg = <0xe800d800 0x24>; - interrupts = , - , - ; - interrupt-names = "error", "rx", "tx"; - clocks = <&mstp10_clks R7S72100_CLK_SPI2>; - power-domains = <&cpg_clocks>; - num-cs = <1>; - #address-cells = <1>; - #size-cells = <0>; - status = "disabled"; - }; - - spi3: spi@e800e000 { - compatible = "renesas,rspi-r7s72100", "renesas,rspi-rz"; - reg = <0xe800e000 0x24>; - interrupts = , - , - ; - interrupt-names = "error", "rx", "tx"; - clocks = <&mstp10_clks R7S72100_CLK_SPI3>; - power-domains = <&cpg_clocks>; - num-cs = <1>; - #address-cells = <1>; - #size-cells = <0>; - status = "disabled"; - }; - - spi4: spi@e800e800 { - compatible = "renesas,rspi-r7s72100", "renesas,rspi-rz"; - reg = <0xe800e800 0x24>; - interrupts = , - , - ; - interrupt-names = "error", "rx", "tx"; - clocks = <&mstp10_clks R7S72100_CLK_SPI4>; - power-domains = <&cpg_clocks>; - num-cs = <1>; - #address-cells = <1>; - #size-cells = <0>; - status = "disabled"; - }; - - gic: interrupt-controller@e8201000 { - compatible = "arm,pl390"; - #interrupt-cells = <3>; - #address-cells = <0>; - interrupt-controller; - reg = <0xe8201000 0x1000>, - <0xe8202000 0x1000>; - }; - - L2: cache-controller@3ffff000 { - compatible = "arm,pl310-cache"; - reg = <0x3ffff000 0x1000>; - interrupts = ; - arm,early-bresp-disable; - arm,full-line-zero-disable; - cache-unified; - cache-level = <2>; - }; - - wdt: watchdog@fcfe0000 { - compatible = "renesas,r7s72100-wdt", "renesas,rza-wdt"; - reg = <0xfcfe0000 0x6>; - interrupts = ; - clocks = <&p0_clk>; - }; - - i2c0: i2c@fcfee000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,riic-r7s72100", "renesas,riic-rz"; - reg = <0xfcfee000 0x44>; - interrupts = , - , - , - , - , - , - , - ; - clocks = <&mstp9_clks R7S72100_CLK_I2C0>; - clock-frequency = <100000>; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - i2c1: i2c@fcfee400 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,riic-r7s72100", "renesas,riic-rz"; - reg = <0xfcfee400 0x44>; - interrupts = , - , - , - , - , - , - , - ; - clocks = <&mstp9_clks R7S72100_CLK_I2C1>; - clock-frequency = <100000>; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - i2c2: i2c@fcfee800 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,riic-r7s72100", "renesas,riic-rz"; - reg = <0xfcfee800 0x44>; - interrupts = , - , - , - , - , - , - , - ; - clocks = <&mstp9_clks R7S72100_CLK_I2C2>; - clock-frequency = <100000>; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - i2c3: i2c@fcfeec00 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,riic-r7s72100", "renesas,riic-rz"; - reg = <0xfcfeec00 0x44>; - interrupts = , - , - , - , - , - , - , - ; - clocks = <&mstp9_clks R7S72100_CLK_I2C3>; - clock-frequency = <100000>; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - mtu2: timer@fcff0000 { - compatible = "renesas,mtu2-r7s72100", "renesas,mtu2"; - reg = <0xfcff0000 0x400>; - interrupts = ; - interrupt-names = "tgi0a"; - clocks = <&mstp3_clks R7S72100_CLK_MTU2>; - clock-names = "fck"; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - ether: ethernet@e8203000 { - compatible = "renesas,ether-r7s72100"; - reg = <0xe8203000 0x800>, - <0xe8204800 0x200>; - interrupts = ; - clocks = <&mstp7_clks R7S72100_CLK_ETHER>; - power-domains = <&cpg_clocks>; - phy-mode = "mii"; - #address-cells = <1>; - #size-cells = <0>; - status = "disabled"; - }; - - mmcif: mmc@e804c800 { - compatible = "renesas,mmcif-r7s72100", "renesas,sh-mmcif"; - reg = <0xe804c800 0x80>; - interrupts = ; - clocks = <&mstp8_clks R7S72100_CLK_MMCIF>; - power-domains = <&cpg_clocks>; - reg-io-width = <4>; - bus-width = <8>; - status = "disabled"; - }; - - sdhi0: sd@e804e000 { - compatible = "renesas,sdhi-r7s72100"; - reg = <0xe804e000 0x100>; - interrupts = ; - - clocks = <&mstp12_clks R7S72100_CLK_SDHI00>, - <&mstp12_clks R7S72100_CLK_SDHI01>; - clock-names = "core", "cd"; - power-domains = <&cpg_clocks>; - cap-sd-highspeed; - cap-sdio-irq; - status = "disabled"; - }; - - sdhi1: sd@e804e800 { - compatible = "renesas,sdhi-r7s72100"; - reg = <0xe804e800 0x100>; - interrupts = ; - - clocks = <&mstp12_clks R7S72100_CLK_SDHI10>, - <&mstp12_clks R7S72100_CLK_SDHI11>; - clock-names = "core", "cd"; - power-domains = <&cpg_clocks>; - cap-sd-highspeed; - cap-sdio-irq; - status = "disabled"; - }; - - ostm0: timer@fcfec000 { - compatible = "renesas,r7s72100-ostm", "renesas,ostm"; - reg = <0xfcfec000 0x30>; - interrupts = ; - clocks = <&mstp5_clks R7S72100_CLK_OSTM0>; - power-domains = <&cpg_clocks>; - status = "disabled"; - }; - - ostm1: timer@fcfec400 { - compatible = "renesas,r7s72100-ostm", "renesas,ostm"; - reg = <0xfcfec400 0x30>; - interrupts = ; - clocks = <&mstp5_clks R7S72100_CLK_OSTM1>; - power-domains = <&cpg_clocks>; - status = "disabled"; }; - rtc: rtc@fcff1000 { - compatible = "renesas,r7s72100-rtc", "renesas,sh-rtc"; - reg = <0xfcff1000 0x2e>; - interrupts = ; - interrupt-names = "alarm", "period", "carry"; - clocks = <&mstp6_clks R7S72100_CLK_RTC>, <&rtc_x1_clk>, - <&rtc_x3_clk>, <&extal_clk>; - clock-names = "fck", "rtc_x1", "rtc_x3", "extal"; - power-domains = <&cpg_clocks>; - status = "disabled"; + usb_x1_clk: usb_x1 { + #clock-cells = <0>; + compatible = "fixed-clock"; + /* If clk present, value must be set by board */ + clock-frequency = <0>; }; }; diff --git a/arch/arm/boot/dts/r8a73a4-ape6evm.dts b/arch/arm/boot/dts/r8a73a4-ape6evm.dts index ec7c86e06538..125c39c0222f 100644 --- a/arch/arm/boot/dts/r8a73a4-ape6evm.dts +++ b/arch/arm/boot/dts/r8a73a4-ape6evm.dts @@ -234,7 +234,7 @@ &sdhi0 { vmmc-supply = <&vcc_sdhi0>; bus-width = <4>; - toshiba,mmc-wrprotect-disable; + disable-wp; pinctrl-names = "default"; pinctrl-0 = <&sdhi0_pins>; status = "okay"; @@ -244,7 +244,7 @@ vmmc-supply = <&ape6evm_fixed_3v3>; bus-width = <4>; broken-cd; - toshiba,mmc-wrprotect-disable; + disable-wp; pinctrl-names = "default"; pinctrl-0 = <&sdhi1_pins>; status = "okay"; diff --git a/arch/arm/boot/dts/r8a73a4.dtsi b/arch/arm/boot/dts/r8a73a4.dtsi index 8e48090e4fdc..080d037f5733 100644 --- a/arch/arm/boot/dts/r8a73a4.dtsi +++ b/arch/arm/boot/dts/r8a73a4.dtsi @@ -57,10 +57,10 @@ timer { compatible = "arm,armv7-timer"; - interrupts = , - , - , - ; + interrupts = , + , + , + ; }; dbsc1: memory-controller@e6790000 { @@ -464,7 +464,7 @@ <0 0xf1002000 0 0x2000>, <0 0xf1004000 0 0x2000>, <0 0xf1006000 0 0x2000>; - interrupts = ; + interrupts = ; clocks = <&mstp4_clks R8A73A4_CLK_INTC_SYS>; clock-names = "clk"; power-domains = <&pd_c4>; diff --git a/arch/arm/boot/dts/r8a7740.dtsi b/arch/arm/boot/dts/r8a7740.dtsi index afd3bc5e6cf2..eb9a911deefb 100644 --- a/arch/arm/boot/dts/r8a7740.dtsi +++ b/arch/arm/boot/dts/r8a7740.dtsi @@ -67,6 +67,24 @@ power-domains = <&pd_d4>; }; + ceu0: ceu@fe910000 { + reg = <0xfe910000 0x3000>; + compatible = "renesas,r8a7740-ceu"; + interrupts = ; + clocks = <&mstp1_clks R8A7740_CLK_CEU20>; + power-domains = <&pd_a4r>; + status = "disabled"; + }; + + ceu1: ceu@fe914000 { + reg = <0xfe914000 0x3000>; + compatible = "renesas,r8a7740-ceu"; + interrupts = ; + clocks = <&mstp1_clks R8A7740_CLK_CEU21>; + power-domains = <&pd_a4r>; + status = "disabled"; + }; + cmt1: timer@e6138000 { compatible = "renesas,cmt-48-r8a7740", "renesas,cmt-48"; reg = <0xe6138000 0x170>; diff --git a/arch/arm/boot/dts/r8a7743-iwg20m.dtsi b/arch/arm/boot/dts/r8a7743-iwg20m.dtsi index 1d3e9503c5bd..d364685d9184 100644 --- a/arch/arm/boot/dts/r8a7743-iwg20m.dtsi +++ b/arch/arm/boot/dts/r8a7743-iwg20m.dtsi @@ -91,6 +91,11 @@ }; }; +&rwdt { + timeout-sec = <60>; + status = "okay"; +}; + &sdhi0 { pinctrl-0 = <&sdhi0_pins>; pinctrl-names = "default"; diff --git a/arch/arm/boot/dts/r8a7743.dtsi b/arch/arm/boot/dts/r8a7743.dtsi index 1d9073ba0ce0..142949d7066f 100644 --- a/arch/arm/boot/dts/r8a7743.dtsi +++ b/arch/arm/boot/dts/r8a7743.dtsi @@ -125,6 +125,13 @@ clock-frequency = <0>; }; + pmu { + compatible = "arm,cortex-a15-pmu"; + interrupts-extended = <&gic GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&cpu0>, <&cpu1>; + }; + /* External SCIF clock */ scif_clk: scif { compatible = "fixed-clock"; @@ -297,6 +304,16 @@ reg = <0 0xe6160000 0 0x100>; }; + rwdt: watchdog@e6020000 { + compatible = "renesas,r8a7743-wdt", + "renesas,rcar-gen2-wdt"; + reg = <0 0xe6020000 0 0x0c>; + clocks = <&cpg CPG_MOD 402>; + power-domains = <&sysc R8A7743_PD_ALWAYS_ON>; + resets = <&cpg 402>; + status = "disabled"; + }; + sysc: system-controller@e6180000 { compatible = "renesas,r8a7743-sysc"; reg = <0 0xe6180000 0 0x200>; @@ -407,7 +424,7 @@ smp-sram@0 { compatible = "renesas,smp-sram"; - reg = <0 0x10>; + reg = <0 0x100>; }; }; diff --git a/arch/arm/boot/dts/r8a7745-iwg22m.dtsi b/arch/arm/boot/dts/r8a7745-iwg22m.dtsi index 8d0a392b6811..29b6e10fdf96 100644 --- a/arch/arm/boot/dts/r8a7745-iwg22m.dtsi +++ b/arch/arm/boot/dts/r8a7745-iwg22m.dtsi @@ -91,6 +91,11 @@ }; }; +&rwdt { + timeout-sec = <60>; + status = "okay"; +}; + &sdhi1 { pinctrl-0 = <&sdhi1_pins>; pinctrl-names = "default"; diff --git a/arch/arm/boot/dts/r8a7745.dtsi b/arch/arm/boot/dts/r8a7745.dtsi index dd49a8b48f3e..1cb7a7ab0418 100644 --- a/arch/arm/boot/dts/r8a7745.dtsi +++ b/arch/arm/boot/dts/r8a7745.dtsi @@ -105,6 +105,13 @@ clock-frequency = <0>; }; + pmu { + compatible = "arm,cortex-a7-pmu"; + interrupts-extended = <&gic GIC_SPI 82 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 83 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&cpu0>, <&cpu1>; + }; + /* External SCIF clock */ scif_clk: scif { compatible = "fixed-clock"; @@ -262,6 +269,16 @@ reg = <0 0xe6160000 0 0x100>; }; + rwdt: watchdog@e6020000 { + compatible = "renesas,r8a7745-wdt", + "renesas,rcar-gen2-wdt"; + reg = <0 0xe6020000 0 0x0c>; + clocks = <&cpg CPG_MOD 402>; + power-domains = <&sysc R8A7745_PD_ALWAYS_ON>; + resets = <&cpg 402>; + status = "disabled"; + }; + sysc: system-controller@e6180000 { compatible = "renesas,r8a7745-sysc"; reg = <0 0xe6180000 0 0x200>; @@ -360,7 +377,7 @@ smp-sram@0 { compatible = "renesas,smp-sram"; - reg = <0 0x10>; + reg = <0 0x100>; }; }; diff --git a/arch/arm/boot/dts/r8a77470-iwg23s-sbc.dts b/arch/arm/boot/dts/r8a77470-iwg23s-sbc.dts new file mode 100644 index 000000000000..e3585daafdd6 --- /dev/null +++ b/arch/arm/boot/dts/r8a77470-iwg23s-sbc.dts @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Device Tree Source for the iWave-RZ/G1C single board computer + * + * Copyright (C) 2018 Renesas Electronics Corp. + */ + +/dts-v1/; +#include "r8a77470.dtsi" +/ { + model = "iWave iW-RainboW-G23S single board computer based on RZ/G1C"; + compatible = "iwave,g23s", "renesas,r8a77470"; + + aliases { + ethernet0 = &avb; + serial1 = &scif1; + }; + + chosen { + bootargs = "ignore_loglevel rw root=/dev/nfs ip=dhcp"; + stdout-path = "serial1:115200n8"; + }; + + memory@40000000 { + device_type = "memory"; + reg = <0 0x40000000 0 0x20000000>; + }; +}; + +&avb { + phy-handle = <&phy3>; + phy-mode = "gmii"; + renesas,no-ether-link; + status = "okay"; + + phy3: ethernet-phy@3 { + reg = <3>; + micrel,led-mode = <1>; + }; +}; + +&extal_clk { + clock-frequency = <20000000>; +}; + +&scif1 { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/r8a77470.dtsi b/arch/arm/boot/dts/r8a77470.dtsi new file mode 100644 index 000000000000..c85032f9605b --- /dev/null +++ b/arch/arm/boot/dts/r8a77470.dtsi @@ -0,0 +1,336 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Device Tree Source for the r8a77470 SoC + * + * Copyright (C) 2018 Renesas Electronics Corp. + */ + +#include +#include +#include +/ { + compatible = "renesas,r8a77470"; + #address-cells = <2>; + #size-cells = <2>; + + cpus { + #address-cells = <1>; + #size-cells = <0>; + + cpu0: cpu@0 { + device_type = "cpu"; + compatible = "arm,cortex-a7"; + reg = <0>; + clock-frequency = <1000000000>; + clocks = <&cpg CPG_CORE 0>; + power-domains = <&sysc 5>; + next-level-cache = <&L2_CA7>; + }; + + + L2_CA7: cache-controller-0 { + compatible = "cache"; + cache-unified; + cache-level = <2>; + power-domains = <&sysc 21>; + }; + }; + + /* External root clock */ + extal_clk: extal { + compatible = "fixed-clock"; + #clock-cells = <0>; + /* This value must be overridden by the board. */ + clock-frequency = <0>; + }; + + /* External SCIF clock */ + scif_clk: scif { + compatible = "fixed-clock"; + #clock-cells = <0>; + /* This value must be overridden by the board. */ + clock-frequency = <0>; + }; + + soc { + compatible = "simple-bus"; + interrupt-parent = <&gic>; + + #address-cells = <2>; + #size-cells = <2>; + ranges; + + cpg: clock-controller@e6150000 { + compatible = "renesas,r8a77470-cpg-mssr"; + reg = <0 0xe6150000 0 0x1000>; + clocks = <&extal_clk>, <&usb_extal_clk>; + clock-names = "extal", "usb_extal"; + #clock-cells = <2>; + #power-domain-cells = <0>; + #reset-cells = <1>; + }; + + rst: reset-controller@e6160000 { + compatible = "renesas,r8a77470-rst"; + reg = <0 0xe6160000 0 0x100>; + }; + + sysc: system-controller@e6180000 { + compatible = "renesas,r8a77470-sysc"; + reg = <0 0xe6180000 0 0x200>; + #power-domain-cells = <1>; + }; + + irqc: interrupt-controller@e61c0000 { + compatible = "renesas,irqc-r8a77470", "renesas,irqc"; + #interrupt-cells = <2>; + interrupt-controller; + reg = <0 0xe61c0000 0 0x200>; + interrupts = , + , + , + , + , + , + , + , + , + ; + clocks = <&cpg CPG_MOD 407>; + power-domains = <&sysc 32>; + resets = <&cpg 407>; + }; + + icram0: sram@e63a0000 { + compatible = "mmio-sram"; + reg = <0 0xe63a0000 0 0x12000>; + }; + + icram1: sram@e63c0000 { + compatible = "mmio-sram"; + reg = <0 0xe63c0000 0 0x1000>; + #address-cells = <1>; + #size-cells = <1>; + ranges = <0 0 0xe63c0000 0x1000>; + + smp-sram@0 { + compatible = "renesas,smp-sram"; + reg = <0 0x100>; + }; + }; + + icram2: sram@e6300000 { + compatible = "mmio-sram"; + reg = <0 0xe6300000 0 0x20000>; + }; + + dmac0: dma-controller@e6700000 { + compatible = "renesas,dmac-r8a77470", + "renesas,rcar-dmac"; + reg = <0 0xe6700000 0 0x20000>; + interrupts = ; + interrupt-names = "error", + "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14"; + clocks = <&cpg CPG_MOD 219>; + clock-names = "fck"; + power-domains = <&sysc 32>; + resets = <&cpg 219>; + #dma-cells = <1>; + dma-channels = <15>; + }; + + dmac1: dma-controller@e6720000 { + compatible = "renesas,dmac-r8a77470", + "renesas,rcar-dmac"; + reg = <0 0xe6720000 0 0x20000>; + interrupts = ; + interrupt-names = "error", + "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14"; + clocks = <&cpg CPG_MOD 218>; + clock-names = "fck"; + power-domains = <&sysc 32>; + resets = <&cpg 218>; + #dma-cells = <1>; + dma-channels = <15>; + }; + + avb: ethernet@e6800000 { + compatible = "renesas,etheravb-r8a77470", + "renesas,etheravb-rcar-gen2"; + reg = <0 0xe6800000 0 0x800>, <0 0xee0e8000 0 0x4000>; + interrupts = ; + clocks = <&cpg CPG_MOD 812>; + power-domains = <&sysc 32>; + resets = <&cpg 812>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + scif0: serial@e6e60000 { + compatible = "renesas,scif-r8a77470", + "renesas,rcar-gen2-scif", "renesas,scif"; + reg = <0 0xe6e60000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 721>, + <&cpg CPG_CORE 5>, <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac0 0x29>, <&dmac0 0x2a>, + <&dmac1 0x29>, <&dmac1 0x2a>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc 32>; + resets = <&cpg 721>; + status = "disabled"; + }; + + scif1: serial@e6e68000 { + compatible = "renesas,scif-r8a77470", + "renesas,rcar-gen2-scif", "renesas,scif"; + reg = <0 0xe6e68000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 720>, + <&cpg CPG_CORE 5>, <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac0 0x2d>, <&dmac0 0x2e>, + <&dmac1 0x2d>, <&dmac1 0x2e>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc 32>; + resets = <&cpg 720>; + status = "disabled"; + }; + + scif2: serial@e6e58000 { + compatible = "renesas,scif-r8a77470", + "renesas,rcar-gen2-scif", "renesas,scif"; + reg = <0 0xe6e58000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 719>, + <&cpg CPG_CORE 5>, <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac0 0x2b>, <&dmac0 0x2c>, + <&dmac1 0x2b>, <&dmac1 0x2c>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc 32>; + resets = <&cpg 719>; + status = "disabled"; + }; + + scif3: serial@e6ea8000 { + compatible = "renesas,scif-r8a77470", + "renesas,rcar-gen2-scif", "renesas,scif"; + reg = <0 0xe6ea8000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 718>, + <&cpg CPG_CORE 5>, <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac0 0x2f>, <&dmac0 0x30>, + <&dmac1 0x2f>, <&dmac1 0x30>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc 32>; + resets = <&cpg 718>; + status = "disabled"; + }; + + scif4: serial@e6ee0000 { + compatible = "renesas,scif-r8a77470", + "renesas,rcar-gen2-scif", "renesas,scif"; + reg = <0 0xe6ee0000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 715>, + <&cpg CPG_CORE 5>, <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac0 0xfb>, <&dmac0 0xfc>, + <&dmac1 0xfb>, <&dmac1 0xfc>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc 32>; + resets = <&cpg 715>; + status = "disabled"; + }; + + scif5: serial@e6ee8000 { + compatible = "renesas,scif-r8a77470", + "renesas,rcar-gen2-scif", "renesas,scif"; + reg = <0 0xe6ee8000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 714>, + <&cpg CPG_CORE 5>, <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac0 0xfd>, <&dmac0 0xfe>, + <&dmac1 0xfd>, <&dmac1 0xfe>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc 32>; + resets = <&cpg 714>; + status = "disabled"; + }; + + gic: interrupt-controller@f1001000 { + compatible = "arm,gic-400"; + #interrupt-cells = <3>; + #address-cells = <0>; + interrupt-controller; + reg = <0 0xf1001000 0 0x1000>, <0 0xf1002000 0 0x2000>, + <0 0xf1004000 0 0x2000>, <0 0xf1006000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 408>; + clock-names = "clk"; + power-domains = <&sysc 32>; + resets = <&cpg 408>; + }; + + prr: chipid@ff000044 { + compatible = "renesas,prr"; + reg = <0 0xff000044 0 4>; + }; + }; + + timer { + compatible = "arm,armv7-timer"; + interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>; + }; + + /* External USB clock - can be overridden by the board */ + usb_extal_clk: usb_extal { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <48000000>; + }; +}; diff --git a/arch/arm/boot/dts/r8a7790-lager.dts b/arch/arm/boot/dts/r8a7790-lager.dts index f07f9018c3e7..092610e3f953 100644 --- a/arch/arm/boot/dts/r8a7790-lager.dts +++ b/arch/arm/boot/dts/r8a7790-lager.dts @@ -902,9 +902,6 @@ status = "okay"; port { - #address-cells = <1>; - #size-cells = <0>; - vin1ep0: endpoint { remote-endpoint = <&adv7180>; bus-width = <8>; @@ -929,6 +926,11 @@ }; }; +&rwdt { + timeout-sec = <60>; + status = "okay"; +}; + &ssi1 { shared-pin; }; diff --git a/arch/arm/boot/dts/r8a7790.dtsi b/arch/arm/boot/dts/r8a7790.dtsi index 05a0fc23ac88..4d06b154bd7e 100644 --- a/arch/arm/boot/dts/r8a7790.dtsi +++ b/arch/arm/boot/dts/r8a7790.dtsi @@ -202,6 +202,24 @@ clock-frequency = <0>; }; + pmu-0 { + compatible = "arm,cortex-a15-pmu"; + interrupts-extended = <&gic GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 74 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 75 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&cpu0>, <&cpu1>, <&cpu2>, <&cpu3>; + }; + + pmu-1 { + compatible = "arm,cortex-a7-pmu"; + interrupts-extended = <&gic GIC_SPI 82 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 83 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 84 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 85 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&cpu4>, <&cpu5>, <&cpu6>, <&cpu7>; + }; + /* External SCIF clock */ scif_clk: scif { compatible = "fixed-clock"; @@ -218,6 +236,16 @@ #size-cells = <2>; ranges; + rwdt: watchdog@e6020000 { + compatible = "renesas,r8a7790-wdt", + "renesas,rcar-gen2-wdt"; + reg = <0 0xe6020000 0 0x0c>; + clocks = <&cpg CPG_MOD 402>; + power-domains = <&sysc R8A7790_PD_ALWAYS_ON>; + resets = <&cpg 402>; + status = "disabled"; + }; + gpio0: gpio@e6050000 { compatible = "renesas,gpio-r8a7790", "renesas,rcar-gen2-gpio"; @@ -443,7 +471,7 @@ smp-sram@0 { compatible = "renesas,smp-sram"; - reg = <0 0x10>; + reg = <0 0x100>; }; }; @@ -1544,7 +1572,7 @@ interrupt-controller; reg = <0 0xf1001000 0 0x1000>, <0 0xf1002000 0 0x2000>, <0 0xf1004000 0 0x2000>, <0 0xf1006000 0 0x2000>; - interrupts = ; + interrupts = ; clocks = <&cpg CPG_MOD 408>; clock-names = "clk"; power-domains = <&sysc R8A7790_PD_ALWAYS_ON>; @@ -1615,6 +1643,33 @@ resets = <&cpg 127>; }; + fdp1@fe940000 { + compatible = "renesas,fdp1"; + reg = <0 0xfe940000 0 0x2400>; + interrupts = ; + clocks = <&cpg CPG_MOD 119>; + power-domains = <&sysc R8A7790_PD_ALWAYS_ON>; + resets = <&cpg 119>; + }; + + fdp1@fe944000 { + compatible = "renesas,fdp1"; + reg = <0 0xfe944000 0 0x2400>; + interrupts = ; + clocks = <&cpg CPG_MOD 118>; + power-domains = <&sysc R8A7790_PD_ALWAYS_ON>; + resets = <&cpg 118>; + }; + + fdp1@fe948000 { + compatible = "renesas,fdp1"; + reg = <0 0xfe948000 0 0x2400>; + interrupts = ; + clocks = <&cpg CPG_MOD 117>; + power-domains = <&sysc R8A7790_PD_ALWAYS_ON>; + resets = <&cpg 117>; + }; + jpu: jpeg-codec@fe980000 { compatible = "renesas,jpu-r8a7790", "renesas,rcar-gen2-jpu"; @@ -1773,10 +1828,10 @@ timer { compatible = "arm,armv7-timer"; - interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_LOW)>; + interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>; }; /* External USB clock - can be overridden by the board */ diff --git a/arch/arm/boot/dts/r8a7791-koelsch.dts b/arch/arm/boot/dts/r8a7791-koelsch.dts index 9d7213a0b8b8..8ab793d8b2fd 100644 --- a/arch/arm/boot/dts/r8a7791-koelsch.dts +++ b/arch/arm/boot/dts/r8a7791-koelsch.dts @@ -643,6 +643,11 @@ status = "okay"; }; +&rwdt { + timeout-sec = <60>; + status = "okay"; +}; + &sata0 { status = "okay"; }; @@ -850,9 +855,6 @@ pinctrl-names = "default"; port { - #address-cells = <1>; - #size-cells = <0>; - vin0ep2: endpoint { remote-endpoint = <&adv7612_out>; bus-width = <24>; @@ -871,9 +873,6 @@ pinctrl-names = "default"; port { - #address-cells = <1>; - #size-cells = <0>; - vin1ep: endpoint { remote-endpoint = <&adv7180>; bus-width = <8>; diff --git a/arch/arm/boot/dts/r8a7791-porter.dts b/arch/arm/boot/dts/r8a7791-porter.dts index ae9ed9ff53ef..a01101b49d99 100644 --- a/arch/arm/boot/dts/r8a7791-porter.dts +++ b/arch/arm/boot/dts/r8a7791-porter.dts @@ -386,9 +386,6 @@ pinctrl-names = "default"; port { - #address-cells = <1>; - #size-cells = <0>; - vin0ep: endpoint { remote-endpoint = <&adv7180>; bus-width = <8>; @@ -481,6 +478,11 @@ }; }; +&rwdt { + timeout-sec = <60>; + status = "okay"; +}; + &ssi1 { shared-pin; }; diff --git a/arch/arm/boot/dts/r8a7791.dtsi b/arch/arm/boot/dts/r8a7791.dtsi index 506b20885413..6e1dd7ad7bd6 100644 --- a/arch/arm/boot/dts/r8a7791.dtsi +++ b/arch/arm/boot/dts/r8a7791.dtsi @@ -126,6 +126,13 @@ clock-frequency = <0>; }; + pmu { + compatible = "arm,cortex-a15-pmu"; + interrupts-extended = <&gic GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&cpu0>, <&cpu1>; + }; + /* External SCIF clock */ scif_clk: scif { compatible = "fixed-clock"; @@ -142,6 +149,16 @@ #size-cells = <2>; ranges; + rwdt: watchdog@e6020000 { + compatible = "renesas,r8a7791-wdt", + "renesas,rcar-gen2-wdt"; + reg = <0 0xe6020000 0 0x0c>; + clocks = <&cpg CPG_MOD 402>; + power-domains = <&sysc R8A7791_PD_ALWAYS_ON>; + resets = <&cpg 402>; + status = "disabled"; + }; + gpio0: gpio@e6050000 { compatible = "renesas,gpio-r8a7791", "renesas,rcar-gen2-gpio"; @@ -407,7 +424,7 @@ smp-sram@0 { compatible = "renesas,smp-sram"; - reg = <0 0x10>; + reg = <0 0x100>; }; }; @@ -1621,6 +1638,24 @@ resets = <&cpg 127>; }; + fdp1@fe940000 { + compatible = "renesas,fdp1"; + reg = <0 0xfe940000 0 0x2400>; + interrupts = ; + clocks = <&cpg CPG_MOD 119>; + power-domains = <&sysc R8A7791_PD_ALWAYS_ON>; + resets = <&cpg 119>; + }; + + fdp1@fe944000 { + compatible = "renesas,fdp1"; + reg = <0 0xfe944000 0 0x2400>; + interrupts = ; + clocks = <&cpg CPG_MOD 118>; + power-domains = <&sysc R8A7791_PD_ALWAYS_ON>; + resets = <&cpg 118>; + }; + jpu: jpeg-codec@fe980000 { compatible = "renesas,jpu-r8a7791", "renesas,rcar-gen2-jpu"; diff --git a/arch/arm/boot/dts/r8a7792-blanche.dts b/arch/arm/boot/dts/r8a7792-blanche.dts index 9b67dca6c9ef..04fb70931b3b 100644 --- a/arch/arm/boot/dts/r8a7792-blanche.dts +++ b/arch/arm/boot/dts/r8a7792-blanche.dts @@ -239,6 +239,11 @@ }; }; +&rwdt { + timeout-sec = <60>; + status = "okay"; +}; + &scif0 { pinctrl-0 = <&scif0_pins>; pinctrl-names = "default"; diff --git a/arch/arm/boot/dts/r8a7792-wheat.dts b/arch/arm/boot/dts/r8a7792-wheat.dts index b9471b67b728..db01de7a3811 100644 --- a/arch/arm/boot/dts/r8a7792-wheat.dts +++ b/arch/arm/boot/dts/r8a7792-wheat.dts @@ -168,6 +168,11 @@ }; }; +&rwdt { + timeout-sec = <60>; + status = "okay"; +}; + &scif0 { pinctrl-0 = <&scif0_pins>; pinctrl-names = "default"; @@ -240,9 +245,15 @@ status = "okay"; clock-frequency = <400000>; + /* + * The adv75xx resets its addresses to defaults during low power mode. + * Because we have two ADV7513 devices on the same bus, we must change + * both of them away from the defaults so that they do not conflict. + */ hdmi@3d { compatible = "adi,adv7513"; - reg = <0x3d>; + reg = <0x3d>, <0x2d>, <0x4d>, <0x5d>; + reg-names = "main", "cec", "edid", "packet"; adi,input-depth = <8>; adi,input-colorspace = "rgb"; @@ -272,7 +283,8 @@ hdmi@39 { compatible = "adi,adv7513"; - reg = <0x39>; + reg = <0x39>, <0x29>, <0x49>, <0x59>; + reg-names = "main", "cec", "edid", "packet"; adi,input-depth = <8>; adi,input-colorspace = "rgb"; diff --git a/arch/arm/boot/dts/r8a7792.dtsi b/arch/arm/boot/dts/r8a7792.dtsi index 268987ff0201..f44257dd86f6 100644 --- a/arch/arm/boot/dts/r8a7792.dtsi +++ b/arch/arm/boot/dts/r8a7792.dtsi @@ -85,6 +85,13 @@ clock-frequency = <0>; }; + pmu { + compatible = "arm,cortex-a15-pmu"; + interrupts-extended = <&gic GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&cpu0>, <&cpu1>; + }; + /* External SCIF clock */ scif_clk: scif { compatible = "fixed-clock"; @@ -101,6 +108,16 @@ #size-cells = <2>; ranges; + rwdt: watchdog@e6020000 { + compatible = "renesas,r8a7792-wdt", + "renesas,rcar-gen2-wdt"; + reg = <0 0xe6020000 0 0x0c>; + clocks = <&cpg CPG_MOD 402>; + power-domains = <&sysc R8A7792_PD_ALWAYS_ON>; + resets = <&cpg 402>; + status = "disabled"; + }; + gpio0: gpio@e6050000 { compatible = "renesas,gpio-r8a7792", "renesas,rcar-gen2-gpio"; @@ -341,7 +358,7 @@ smp-sram@0 { compatible = "renesas,smp-sram"; - reg = <0 0x10>; + reg = <0 0x100>; }; }; diff --git a/arch/arm/boot/dts/r8a7793-gose.dts b/arch/arm/boot/dts/r8a7793-gose.dts index 96e117d8b2cc..aa209f6e5d71 100644 --- a/arch/arm/boot/dts/r8a7793-gose.dts +++ b/arch/arm/boot/dts/r8a7793-gose.dts @@ -599,6 +599,11 @@ status = "okay"; }; +&rwdt { + timeout-sec = <60>; + status = "okay"; +}; + &scif0 { pinctrl-0 = <&scif0_pins>; pinctrl-names = "default"; @@ -758,9 +763,6 @@ pinctrl-names = "default"; port { - #address-cells = <1>; - #size-cells = <0>; - vin0ep2: endpoint { remote-endpoint = <&adv7612_out>; bus-width = <24>; @@ -780,9 +782,6 @@ status = "okay"; port { - #address-cells = <1>; - #size-cells = <0>; - vin1ep: endpoint { remote-endpoint = <&adv7180_out>; bus-width = <8>; diff --git a/arch/arm/boot/dts/r8a7793.dtsi b/arch/arm/boot/dts/r8a7793.dtsi index 4f526030dc7c..4abecfc0ca98 100644 --- a/arch/arm/boot/dts/r8a7793.dtsi +++ b/arch/arm/boot/dts/r8a7793.dtsi @@ -110,6 +110,13 @@ clock-frequency = <0>; }; + pmu { + compatible = "arm,cortex-a15-pmu"; + interrupts-extended = <&gic GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&cpu0>, <&cpu1>; + }; + /* External SCIF clock */ scif_clk: scif { compatible = "fixed-clock"; @@ -126,6 +133,16 @@ #size-cells = <2>; ranges; + rwdt: watchdog@e6020000 { + compatible = "renesas,r8a7793-wdt", + "renesas,rcar-gen2-wdt"; + reg = <0 0xe6020000 0 0x0c>; + clocks = <&cpg CPG_MOD 402>; + power-domains = <&sysc R8A7793_PD_ALWAYS_ON>; + resets = <&cpg 402>; + status = "disabled"; + }; + gpio0: gpio@e6050000 { compatible = "renesas,gpio-r8a7793", "renesas,rcar-gen2-gpio"; @@ -392,7 +409,7 @@ smp-sram@0 { compatible = "renesas,smp-sram"; - reg = <0 0x10>; + reg = <0 0x100>; }; }; @@ -1290,6 +1307,24 @@ resets = <&cpg 408>; }; + fdp1@fe940000 { + compatible = "renesas,fdp1"; + reg = <0 0xfe940000 0 0x2400>; + interrupts = ; + clocks = <&cpg CPG_MOD 119>; + power-domains = <&sysc R8A7793_PD_ALWAYS_ON>; + resets = <&cpg 119>; + }; + + fdp1@fe944000 { + compatible = "renesas,fdp1"; + reg = <0 0xfe944000 0 0x2400>; + interrupts = ; + clocks = <&cpg CPG_MOD 118>; + power-domains = <&sysc R8A7793_PD_ALWAYS_ON>; + resets = <&cpg 118>; + }; + du: display@feb00000 { compatible = "renesas,du-r8a7793"; reg = <0 0xfeb00000 0 0x40000>; diff --git a/arch/arm/boot/dts/r8a7794-alt.dts b/arch/arm/boot/dts/r8a7794-alt.dts index 26a883484ea8..e17027532941 100644 --- a/arch/arm/boot/dts/r8a7794-alt.dts +++ b/arch/arm/boot/dts/r8a7794-alt.dts @@ -181,6 +181,12 @@ }; }; }; + + eeprom@50 { + compatible = "renesas,r1ex24002", "atmel,24c02"; + reg = <0x50>; + pagesize = <16>; + }; }; /* @@ -330,6 +336,11 @@ status = "okay"; }; +&rwdt { + timeout-sec = <60>; + status = "okay"; +}; + &sdhi0 { pinctrl-0 = <&sdhi0_pins>; pinctrl-1 = <&sdhi0_pins_uhs>; @@ -375,9 +386,6 @@ pinctrl-names = "default"; port { - #address-cells = <1>; - #size-cells = <0>; - vin0ep: endpoint { remote-endpoint = <&adv7180>; bus-width = <8>; diff --git a/arch/arm/boot/dts/r8a7794-silk.dts b/arch/arm/boot/dts/r8a7794-silk.dts index 351cb3b3d966..7808aaee6644 100644 --- a/arch/arm/boot/dts/r8a7794-silk.dts +++ b/arch/arm/boot/dts/r8a7794-silk.dts @@ -475,9 +475,6 @@ pinctrl-names = "default"; port { - #address-cells = <1>; - #size-cells = <0>; - vin0ep: endpoint { remote-endpoint = <&adv7180>; bus-width = <8>; @@ -540,6 +537,11 @@ }; }; +&rwdt { + timeout-sec = <60>; + status = "okay"; +}; + &ssi1 { shared-pin; }; diff --git a/arch/arm/boot/dts/r8a7794.dtsi b/arch/arm/boot/dts/r8a7794.dtsi index d588efa6aeaa..736196903d22 100644 --- a/arch/arm/boot/dts/r8a7794.dtsi +++ b/arch/arm/boot/dts/r8a7794.dtsi @@ -103,6 +103,13 @@ clock-frequency = <0>; }; + pmu { + compatible = "arm,cortex-a7-pmu"; + interrupts-extended = <&gic GIC_SPI 82 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 83 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&cpu0>, <&cpu1>; + }; + /* External SCIF clock */ scif_clk: scif { compatible = "fixed-clock"; @@ -119,6 +126,16 @@ #size-cells = <2>; ranges; + rwdt: watchdog@e6020000 { + compatible = "renesas,r8a7794-wdt", + "renesas,rcar-gen2-wdt"; + reg = <0 0xe6020000 0 0x0c>; + clocks = <&cpg CPG_MOD 402>; + power-domains = <&sysc R8A7794_PD_ALWAYS_ON>; + resets = <&cpg 402>; + status = "disabled"; + }; + gpio0: gpio@e6050000 { compatible = "renesas,gpio-r8a7794", "renesas,rcar-gen2-gpio"; @@ -348,7 +365,7 @@ smp-sram@0 { compatible = "renesas,smp-sram"; - reg = <0 0x10>; + reg = <0 0x100>; }; }; @@ -1323,6 +1340,15 @@ resets = <&cpg 128>; }; + fdp1@fe940000 { + compatible = "renesas,fdp1"; + reg = <0 0xfe940000 0 0x2400>; + interrupts = ; + clocks = <&cpg CPG_MOD 119>; + power-domains = <&sysc R8A7794_PD_ALWAYS_ON>; + resets = <&cpg 119>; + }; + du: display@feb00000 { compatible = "renesas,du-r8a7794"; reg = <0 0xfeb00000 0 0x40000>; diff --git a/arch/arm/boot/dts/rk3036.dtsi b/arch/arm/boot/dts/rk3036.dtsi index a97458112ff6..567a6a725f9c 100644 --- a/arch/arm/boot/dts/rk3036.dtsi +++ b/arch/arm/boot/dts/rk3036.dtsi @@ -197,6 +197,8 @@ reg = <0x10118300 0x100>; interrupts = ; interrupt-names = "vop_mmu"; + clocks = <&cru ACLK_LCDC>, <&cru HCLK_LCDC>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; diff --git a/arch/arm/boot/dts/rk322x.dtsi b/arch/arm/boot/dts/rk322x.dtsi index df1e47858675..be80e9a2c9af 100644 --- a/arch/arm/boot/dts/rk322x.dtsi +++ b/arch/arm/boot/dts/rk322x.dtsi @@ -584,6 +584,8 @@ reg = <0x20020800 0x100>; interrupts = ; interrupt-names = "vpu_mmu"; + clocks = <&cru ACLK_VPU>, <&cru HCLK_VPU>; + clock-names = "aclk", "iface"; iommu-cells = <0>; status = "disabled"; }; @@ -593,6 +595,8 @@ reg = <0x20030480 0x40>, <0x200304c0 0x40>; interrupts = ; interrupt-names = "vdec_mmu"; + clocks = <&cru ACLK_RKVDEC>, <&cru HCLK_RKVDEC>; + clock-names = "aclk", "iface"; iommu-cells = <0>; status = "disabled"; }; @@ -602,6 +606,8 @@ reg = <0x20053f00 0x100>; interrupts = ; interrupt-names = "vop_mmu"; + clocks = <&cru ACLK_VOP>, <&cru HCLK_VOP>; + clock-names = "aclk", "iface"; iommu-cells = <0>; status = "disabled"; }; @@ -611,6 +617,8 @@ reg = <0x20070800 0x100>; interrupts = ; interrupt-names = "iep_mmu"; + clocks = <&cru ACLK_IEP>, <&cru HCLK_IEP>; + clock-names = "aclk", "iface"; iommu-cells = <0>; status = "disabled"; }; diff --git a/arch/arm/boot/dts/rk3288-phycore-som.dtsi b/arch/arm/boot/dts/rk3288-phycore-som.dtsi index f13bcb1cd3d9..aaab2d171ffe 100644 --- a/arch/arm/boot/dts/rk3288-phycore-som.dtsi +++ b/arch/arm/boot/dts/rk3288-phycore-som.dtsi @@ -151,6 +151,7 @@ ti,tx-internal-delay = ; ti,fifo-depth = ; enet-phy-lane-no-swap; + ti,clk-output-sel = ; }; }; }; diff --git a/arch/arm/boot/dts/rk3288-tinker.dts b/arch/arm/boot/dts/rk3288-tinker.dts index 346b0d8b474d..127488f9f174 100644 --- a/arch/arm/boot/dts/rk3288-tinker.dts +++ b/arch/arm/boot/dts/rk3288-tinker.dts @@ -49,6 +49,10 @@ model = "Rockchip RK3288 Tinker Board"; compatible = "asus,rk3288-tinker", "rockchip,rk3288"; + chosen { + stdout-path = "serial2:115200n8"; + }; + memory { reg = <0x0 0x0 0x0 0x80000000>; device_type = "memory"; diff --git a/arch/arm/boot/dts/rk3288-veyron-chromebook.dtsi b/arch/arm/boot/dts/rk3288-veyron-chromebook.dtsi index be487111d025..b16d570ff029 100644 --- a/arch/arm/boot/dts/rk3288-veyron-chromebook.dtsi +++ b/arch/arm/boot/dts/rk3288-veyron-chromebook.dtsi @@ -95,7 +95,8 @@ pinctrl-names = "default"; pinctrl-0 = <&bl_en>; pwms = <&pwm0 0 1000000 0>; - pwm-delay-us = <10000>; + post-pwm-on-delay-ms = <10>; + pwm-off-delay-ms = <10>; }; gpio-charger { diff --git a/arch/arm/boot/dts/rk3288-veyron-minnie.dts b/arch/arm/boot/dts/rk3288-veyron-minnie.dts index 544de6027aaa..4c5307e62001 100644 --- a/arch/arm/boot/dts/rk3288-veyron-minnie.dts +++ b/arch/arm/boot/dts/rk3288-veyron-minnie.dts @@ -123,6 +123,8 @@ 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255>; power-supply = <&backlight_regulator>; + post-pwm-on-delay-ms = <200>; + pwm-off-delay-ms = <200>; }; &emmc { diff --git a/arch/arm/boot/dts/rk3288.dtsi b/arch/arm/boot/dts/rk3288.dtsi index 354aff45c1af..d7e49d29ace5 100644 --- a/arch/arm/boot/dts/rk3288.dtsi +++ b/arch/arm/boot/dts/rk3288.dtsi @@ -959,6 +959,8 @@ reg = <0x0 0xff900800 0x0 0x40>; interrupts = ; interrupt-names = "iep_mmu"; + clocks = <&cru ACLK_IEP>, <&cru HCLK_IEP>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -968,6 +970,8 @@ reg = <0x0 0xff914000 0x0 0x100>, <0x0 0xff915000 0x0 0x100>; interrupts = ; interrupt-names = "isp_mmu"; + clocks = <&cru ACLK_ISP>, <&cru HCLK_ISP>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; rockchip,disable-mmu-reset; status = "disabled"; @@ -1027,6 +1031,8 @@ reg = <0x0 0xff930300 0x0 0x100>; interrupts = ; interrupt-names = "vopb_mmu"; + clocks = <&cru ACLK_VOP0>, <&cru HCLK_VOP0>; + clock-names = "aclk", "iface"; power-domains = <&power RK3288_PD_VIO>; #iommu-cells = <0>; status = "disabled"; @@ -1075,6 +1081,8 @@ reg = <0x0 0xff940300 0x0 0x100>; interrupts = ; interrupt-names = "vopl_mmu"; + clocks = <&cru ACLK_VOP1>, <&cru HCLK_VOP1>; + clock-names = "aclk", "iface"; power-domains = <&power RK3288_PD_VIO>; #iommu-cells = <0>; status = "disabled"; @@ -1206,6 +1214,8 @@ reg = <0x0 0xff9a0800 0x0 0x100>; interrupts = ; interrupt-names = "vpu_mmu"; + clocks = <&cru ACLK_VCODEC>, <&cru HCLK_VCODEC>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -1215,6 +1225,8 @@ reg = <0x0 0xff9c0440 0x0 0x40>, <0x0 0xff9c0480 0x0 0x40>; interrupts = ; interrupt-names = "hevc_mmu"; + clocks = <&cru ACLK_HEVC>, <&cru HCLK_HEVC>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -1848,16 +1860,16 @@ uart4 { uart4_xfer: uart4-xfer { - rockchip,pins = <5 12 3 &pcfg_pull_up>, - <5 13 3 &pcfg_pull_none>; + rockchip,pins = <5 15 3 &pcfg_pull_up>, + <5 14 3 &pcfg_pull_none>; }; uart4_cts: uart4-cts { - rockchip,pins = <5 14 3 &pcfg_pull_up>; + rockchip,pins = <5 12 3 &pcfg_pull_up>; }; uart4_rts: uart4-rts { - rockchip,pins = <5 15 3 &pcfg_pull_none>; + rockchip,pins = <5 13 3 &pcfg_pull_none>; }; }; diff --git a/arch/arm/boot/dts/s3c2416-smdk2416.dts b/arch/arm/boot/dts/s3c2416-smdk2416.dts index a1c9d8c695cc..5164386aff3a 100644 --- a/arch/arm/boot/dts/s3c2416-smdk2416.dts +++ b/arch/arm/boot/dts/s3c2416-smdk2416.dts @@ -12,14 +12,13 @@ model = "SMDK2416"; compatible = "samsung,s3c2416"; - memory { + memory@30000000 { + device_type = "memory"; reg = <0x30000000 0x4000000>; }; clocks { compatible = "simple-bus"; - #address-cells = <1>; - #size-cells = <1>; xti: xti { compatible = "fixed-clock"; diff --git a/arch/arm/boot/dts/s3c2416.dtsi b/arch/arm/boot/dts/s3c2416.dtsi index 3c7385cab248..6adf64ea3ff2 100644 --- a/arch/arm/boot/dts/s3c2416.dtsi +++ b/arch/arm/boot/dts/s3c2416.dtsi @@ -18,9 +18,6 @@ }; cpus { - #address-cells = <1>; - #size-cells = <0>; - cpu { compatible = "arm,arm926ej-s"; }; @@ -30,7 +27,7 @@ compatible = "samsung,s3c2416-irq"; }; - clocks: clock-controller@0x4c000000 { + clocks: clock-controller@4c000000 { compatible = "samsung,s3c2416-clock"; reg = <0x4c000000 0x40>; #clock-cells = <1>; @@ -69,7 +66,7 @@ <&clocks SCLK_UART>; }; - uart_3: serial@5000C000 { + uart_3: serial@5000c000 { compatible = "samsung,s3c2440-uart"; reg = <0x5000C000 0x4000>; interrupts = <1 18 24 4>, <1 18 25 4>; @@ -80,7 +77,7 @@ status = "disabled"; }; - sdhci_1: sdhci@4AC00000 { + sdhci_1: sdhci@4ac00000 { compatible = "samsung,s3c6410-sdhci"; reg = <0x4AC00000 0x100>; interrupts = <0 0 21 3>; @@ -91,7 +88,7 @@ status = "disabled"; }; - sdhci_0: sdhci@4A800000 { + sdhci_0: sdhci@4a800000 { compatible = "samsung,s3c6410-sdhci"; reg = <0x4A800000 0x100>; interrupts = <0 0 20 3>; diff --git a/arch/arm/boot/dts/s3c24xx.dtsi b/arch/arm/boot/dts/s3c24xx.dtsi index 34c7fe6751cf..6d8dd3cdd3c0 100644 --- a/arch/arm/boot/dts/s3c24xx.dtsi +++ b/arch/arm/boot/dts/s3c24xx.dtsi @@ -5,11 +5,11 @@ * Copyright (c) 2013 Heiko Stuebner */ -#include "skeleton.dtsi" - / { compatible = "samsung,s3c24xx"; interrupt-parent = <&intc>; + #address-cells = <1>; + #size-cells = <1>; aliases { pinctrl0 = &pinctrl_0; diff --git a/arch/arm/boot/dts/s3c6410-mini6410.dts b/arch/arm/boot/dts/s3c6410-mini6410.dts index f68601bd9c91..0e159c884f97 100644 --- a/arch/arm/boot/dts/s3c6410-mini6410.dts +++ b/arch/arm/boot/dts/s3c6410-mini6410.dts @@ -19,7 +19,8 @@ model = "FriendlyARM Mini6410 board based on S3C6410"; compatible = "friendlyarm,mini6410", "samsung,s3c6410"; - memory { + memory@50000000 { + device_type = "memory"; reg = <0x50000000 0x10000000>; }; diff --git a/arch/arm/boot/dts/s3c6410-smdk6410.dts b/arch/arm/boot/dts/s3c6410-smdk6410.dts index b6b5afcd7602..a9a5689dc462 100644 --- a/arch/arm/boot/dts/s3c6410-smdk6410.dts +++ b/arch/arm/boot/dts/s3c6410-smdk6410.dts @@ -19,7 +19,8 @@ model = "SAMSUNG SMDK6410 board based on S3C6410"; compatible = "samsung,mini6410", "samsung,s3c6410"; - memory { + memory@50000000 { + device_type = "memory"; reg = <0x50000000 0x8000000>; }; diff --git a/arch/arm/boot/dts/s3c64xx.dtsi b/arch/arm/boot/dts/s3c64xx.dtsi index e2be3fbdd3f3..2e611df37911 100644 --- a/arch/arm/boot/dts/s3c64xx.dtsi +++ b/arch/arm/boot/dts/s3c64xx.dtsi @@ -13,10 +13,12 @@ * nodes can be added to this file. */ -#include "skeleton.dtsi" #include / { + #address-cells = <1>; + #size-cells = <1>; + aliases { i2c0 = &i2c0; pinctrl0 = &pinctrl0; diff --git a/arch/arm/boot/dts/sh73a0.dtsi b/arch/arm/boot/dts/sh73a0.dtsi index 914a7c2a584f..c953648a5f41 100644 --- a/arch/arm/boot/dts/sh73a0.dtsi +++ b/arch/arm/boot/dts/sh73a0.dtsi @@ -22,7 +22,7 @@ #address-cells = <1>; #size-cells = <0>; - cpu@0 { + cpu0: cpu@0 { device_type = "cpu"; compatible = "arm,cortex-a9"; reg = <0>; @@ -31,7 +31,7 @@ power-domains = <&pd_a2sl>; next-level-cache = <&L2>; }; - cpu@1 { + cpu1: cpu@1 { device_type = "cpu"; compatible = "arm,cortex-a9"; reg = <1>; @@ -91,6 +91,7 @@ compatible = "arm,cortex-a9-pmu"; interrupts = , ; + interrupt-affinity = <&cpu0>, <&cpu1>; }; cmt1: timer@e6138000 { @@ -336,7 +337,7 @@ GIC_SPI 89 IRQ_TYPE_LEVEL_HIGH>; clocks = <&mstp3_clks SH73A0_CLK_SDHI1>; power-domains = <&pd_a3sp>; - toshiba,mmc-wrprotect-disable; + disable-wp; cap-sd-highspeed; status = "disabled"; }; @@ -348,7 +349,7 @@ GIC_SPI 105 IRQ_TYPE_LEVEL_HIGH>; clocks = <&mstp3_clks SH73A0_CLK_SDHI2>; power-domains = <&pd_a3sp>; - toshiba,mmc-wrprotect-disable; + disable-wp; cap-sd-highspeed; status = "disabled"; }; diff --git a/arch/arm/boot/dts/socfpga_cyclone5_vining_fpga.dts b/arch/arm/boot/dts/socfpga_cyclone5_vining_fpga.dts index 2459d133f1be..f50b19447de6 100644 --- a/arch/arm/boot/dts/socfpga_cyclone5_vining_fpga.dts +++ b/arch/arm/boot/dts/socfpga_cyclone5_vining_fpga.dts @@ -161,7 +161,7 @@ }; at24@50 { - compatible = "at24,24c01"; + compatible = "atmel,24c01"; pagesize = <8>; reg = <0x50>; }; @@ -213,7 +213,7 @@ #size-cells = <0>; reg = <6>; eeprom@51 { - compatible = "at,24c01"; + compatible = "atmel,24c01"; pagesize = <8>; reg = <0x51>; }; @@ -224,7 +224,7 @@ #size-cells = <0>; reg = <7>; eeprom@51 { - compatible = "at,24c01"; + compatible = "atmel,24c01"; pagesize = <8>; reg = <0x51>; }; diff --git a/arch/arm/boot/dts/ste-ccu8540-pinctrl.dtsi b/arch/arm/boot/dts/ste-ccu8540-pinctrl.dtsi deleted file mode 100644 index 52dba2e39c71..000000000000 --- a/arch/arm/boot/dts/ste-ccu8540-pinctrl.dtsi +++ /dev/null @@ -1,196 +0,0 @@ -/* - * Copyright 2012 ST-Ericsson - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ -#include "ste-nomadik-pinctrl.dtsi" - -/ { - soc { - pinctrl { - uart0 { - uart0_default_mux: uart0_mux { - default_mux { - function = "u0"; - groups = "u0_a_1"; - }; - }; - - uart0_default_mode: uart0_default { - default_cfg1 { - pins = "GPIO0", "GPIO2"; - ste,config = <&in_pu>; - }; - - default_cfg2 { - pins = "GPIO1", "GPIO3"; - ste,config = <&out_hi>; - }; - }; - - uart0_sleep_mode: uart0_sleep { - sleep_cfg1 { - pins = "GPIO0", "GPIO2"; - ste,config = <&slpm_in_pu>; - }; - - sleep_cfg2 { - pins = "GPIO1", "GPIO3"; - ste,config = <&slpm_out_hi>; - }; - }; - }; - - uart2 { - uart2_default_mode: uart2_default { - default_mux { - function = "u2"; - groups = "u2txrx_a_1"; - }; - - default_cfg1 { - pins = "GPIO120"; - ste,config = <&in_pu>; - }; - - default_cfg2 { - pins = "GPIO121"; - ste,config = <&out_hi>; - }; - }; - - uart2_sleep_mode: uart2_sleep { - sleep_cfg1 { - pins = "GPIO120"; - ste,config = <&slpm_in_pu>; - }; - - sleep_cfg2 { - pins = "GPIO121"; - ste,config = <&slpm_out_hi>; - }; - }; - }; - - i2c0 { - i2c0_default_mux: i2c_mux { - default_mux { - function = "i2c0"; - groups = "i2c0_a_1"; - }; - }; - - i2c0_default_mode: i2c_default { - default_cfg1 { - pins = "GPIO147", "GPIO148"; - ste,config = <&in_pu>; - }; - }; - - i2c0_sleep_mode: i2c_sleep { - sleep_cfg1 { - pins = "GPIO147", "GPIO148"; - ste,config = <&slpm_in_pu>; - }; - }; - }; - - i2c1 { - i2c1_default_mux: i2c_mux { - default_mux { - function = "i2c1"; - groups = "i2c1_b_2"; - }; - }; - - i2c1_default_mode: i2c_default { - default_cfg1 { - pins = "GPIO16", "GPIO17"; - ste,config = <&in_pu>; - }; - }; - - i2c1_sleep_mode: i2c_sleep { - sleep_cfg1 { - pins = "GPIO16", "GPIO17"; - ste,config = <&slpm_in_pu>; - }; - }; - }; - - i2c2 { - i2c2_default_mux: i2c_mux { - default_mux { - function = "i2c2"; - groups = "i2c2_b_2"; - }; - }; - - i2c2_default_mode: i2c_default { - default_cfg1 { - pins = "GPIO10", "GPIO11"; - ste,config = <&in_pu>; - }; - }; - - i2c2_sleep_mode: i2c_sleep { - sleep_cfg1 { - pins = "GPIO11", "GPIO11"; - ste,config = <&slpm_in_pu>; - }; - }; - }; - - i2c4 { - i2c4_default_mux: i2c_mux { - default_mux { - function = "i2c4"; - groups = "i2c4_b_2"; - }; - }; - - i2c4_default_mode: i2c_default { - default_cfg1 { - pins = "GPIO122", "GPIO123"; - ste,config = <&in_pu>; - }; - }; - - i2c4_sleep_mode: i2c_sleep { - sleep_cfg1 { - pins = "GPIO122", "GPIO123"; - ste,config = <&slpm_in_pu>; - }; - }; - }; - - i2c5 { - i2c5_default_mux: i2c_mux { - default_mux { - function = "i2c5"; - groups = "i2c5_c_2"; - }; - }; - - i2c5_default_mode: i2c_default { - default_cfg1 { - pins = "GPIO118", "GPIO119"; - ste,config = <&in_pu>; - }; - }; - - i2c5_sleep_mode: i2c_sleep { - sleep_cfg1 { - pins = "GPIO118", "GPIO119"; - ste,config = <&slpm_in_pu>; - }; - }; - }; - }; - }; -}; diff --git a/arch/arm/boot/dts/ste-ccu8540.dts b/arch/arm/boot/dts/ste-ccu8540.dts deleted file mode 100644 index 6eaaf638e52e..000000000000 --- a/arch/arm/boot/dts/ste-ccu8540.dts +++ /dev/null @@ -1,94 +0,0 @@ -/* - * Copyright 2013 ST-Ericsson AB - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ - -/dts-v1/; -#include "ste-dbx5x0.dtsi" -#include "ste-ccu8540-pinctrl.dtsi" - -/ { - model = "ST-Ericsson U8540 platform with Device Tree"; - compatible = "st-ericsson,ccu8540", "st-ericsson,u8540"; - - /* This stablilizes the serial port enumeration */ - aliases { - serial0 = &ux500_serial0; - serial1 = &ux500_serial1; - serial2 = &ux500_serial2; - }; - - memory@0 { - device_type = "memory"; - reg = <0x20000000 0x1f000000>, <0xc0000000 0x3f000000>; - }; - - soc { - pinctrl { - compatible = "stericsson,db8540-pinctrl"; - }; - - prcmu@80157000 { - reg = <0x80157000 0x2000>, <0x801b0000 0x8000>, <0x801b8000 0x3000>; - reg-names = "prcmu", "prcmu-tcpm", "prcmu-tcdm"; - }; - - uart@80120000 { - pinctrl-names = "default", "sleep"; - pinctrl-0 = <&uart0_default_mux>, <&uart0_default_mode>; - pinctrl-1 = <&uart0_sleep_mode>; - status = "okay"; - }; - - uart@80121000 { - status = "okay"; - }; - - uart@80007000 { - pinctrl-names = "default", "sleep"; - pinctrl-0 = <&uart2_default_mode>; - pinctrl-1 = <&uart2_sleep_mode>; - status = "okay"; - }; - - i2c0: i2c@80004000 { - pinctrl-names = "default","sleep"; - pinctrl-0 = <&i2c0_default_mux>, <&i2c0_default_mode>; - pinctrl-1 = <&i2c0_sleep_mode>; - }; - - i2c1: i2c@80122000 { - pinctrl-names = "default","sleep"; - pinctrl-0 = <&i2c1_default_mux>, <&i2c1_default_mode>; - pinctrl-1 = <&i2c1_sleep_mode>; - }; - - i2c2: i2c@80128000 { - pinctrl-names = "default","sleep"; - pinctrl-0 = <&i2c2_default_mux>, <&i2c2_default_mode>; - pinctrl-1 = <&i2c2_sleep_mode>; - }; - - i2c3: i2c@80110000 { - status = "disabled"; - }; - - i2c4: i2c@8012a000 { - pinctrl-names = "default","sleep"; - pinctrl-0 = <&i2c4_default_mux>, <&i2c4_default_mode>; - pinctrl-1 = <&i2c4_sleep_mode>; - }; - - i2c5: i2c@80001000 { - pinctrl-names = "default","sleep"; - pinctrl-0 = <&i2c5_default_mux>, <&i2c5_default_mode>; - pinctrl-1 = <&i2c5_sleep_mode>; - }; - }; -}; diff --git a/arch/arm/boot/dts/ste-ccu9540.dts b/arch/arm/boot/dts/ste-ccu9540.dts deleted file mode 100644 index b3b9bb8e1aa8..000000000000 --- a/arch/arm/boot/dts/ste-ccu9540.dts +++ /dev/null @@ -1,79 +0,0 @@ -/* - * Copyright 2012 ST-Ericsson AB - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ - -/dts-v1/; -#include "ste-dbx5x0.dtsi" - -/ { - model = "ST-Ericsson CCU9540 platform with Device Tree"; - compatible = "st-ericsson,ccu9540", "st-ericsson,u9540"; - - /* This stablilizes the serial port enumeration */ - aliases { - serial0 = &ux500_serial0; - serial1 = &ux500_serial1; - serial2 = &ux500_serial2; - }; - - memory { - reg = <0x00000000 0x20000000>; - }; - - soc { - uart@80120000 { - status = "okay"; - }; - - uart@80121000 { - status = "okay"; - }; - - uart@80007000 { - status = "okay"; - }; - - // External Micro SD slot - sdi0_per1@80126000 { - arm,primecell-periphid = <0x10480180>; - max-frequency = <100000000>; - bus-width = <4>; - cap-sd-highspeed; - cap-mmc-highspeed; - vmmc-supply = <&ab8500_ldo_aux3_reg>; - - cd-gpios = <&gpio7 6 GPIO_ACTIVE_HIGH>; // 230 - cd-inverted; - - status = "okay"; - }; - - - // WLAN SDIO channel - sdi1_per2@80118000 { - arm,primecell-periphid = <0x10480180>; - max-frequency = <100000000>; - bus-width = <4>; - - status = "okay"; - }; - - // On-board eMMC - sdi4_per2@80114000 { - arm,primecell-periphid = <0x10480180>; - max-frequency = <100000000>; - bus-width = <8>; - cap-mmc-highspeed; - vmmc-supply = <&ab8500_ldo_aux2_reg>; - - status = "okay"; - }; - }; -}; diff --git a/arch/arm/boot/dts/ste-snowball.dts b/arch/arm/boot/dts/ste-snowball.dts index ade1d0d4e5f4..b0b94d053098 100644 --- a/arch/arm/boot/dts/ste-snowball.dts +++ b/arch/arm/boot/dts/ste-snowball.dts @@ -46,35 +46,35 @@ #size-cells = <0>; button@1 { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = <2>; label = "userpb"; gpios = <&gpio1 0 GPIO_ACTIVE_HIGH>; }; button@2 { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = <3>; label = "extkb1"; gpios = <&gpio4 23 GPIO_ACTIVE_HIGH>; }; button@3 { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = <4>; label = "extkb2"; gpios = <&gpio4 24 GPIO_ACTIVE_HIGH>; }; button@4 { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = <5>; label = "extkb3"; gpios = <&gpio5 1 GPIO_ACTIVE_HIGH>; }; button@5 { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = <6>; label = "extkb4"; diff --git a/arch/arm/boot/dts/stih407-family.dtsi b/arch/arm/boot/dts/stih407-family.dtsi index f7362c31de29..9e29a4499938 100644 --- a/arch/arm/boot/dts/stih407-family.dtsi +++ b/arch/arm/boot/dts/stih407-family.dtsi @@ -206,19 +206,19 @@ vtg_main: sti-vtg-main@8d02800 { compatible = "st,vtg"; reg = <0x8d02800 0x200>; - interrupts = ; + interrupts = ; }; vtg_aux: sti-vtg-aux@8d00200 { compatible = "st,vtg"; reg = <0x8d00200 0x100>; - interrupts = ; + interrupts = ; }; serial@9830000 { compatible = "st,asc"; reg = <0x9830000 0x2c>; - interrupts = ; + interrupts = ; clocks = <&clk_s_c0_flexgen CLK_EXT2F_A9>; /* Pinctrl moved out to a per-board configuration */ @@ -228,7 +228,7 @@ serial@9831000 { compatible = "st,asc"; reg = <0x9831000 0x2c>; - interrupts = ; + interrupts = ; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_serial1>; clocks = <&clk_s_c0_flexgen CLK_EXT2F_A9>; @@ -239,7 +239,7 @@ serial@9832000 { compatible = "st,asc"; reg = <0x9832000 0x2c>; - interrupts = ; + interrupts = ; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_serial2>; clocks = <&clk_s_c0_flexgen CLK_EXT2F_A9>; @@ -251,7 +251,7 @@ sbc_serial0: serial@9530000 { compatible = "st,asc"; reg = <0x9530000 0x2c>; - interrupts = ; + interrupts = ; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_sbc_serial0>; clocks = <&clk_sysin>; @@ -262,7 +262,7 @@ serial@9531000 { compatible = "st,asc"; reg = <0x9531000 0x2c>; - interrupts = ; + interrupts = ; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_sbc_serial1>; clocks = <&clk_sysin>; @@ -574,7 +574,7 @@ status = "disabled"; reg = <0x09060000 0x7ff>, <0x9061008 0x20>; reg-names = "mmc", "top-mmc-delay"; - interrupts = ; + interrupts = ; interrupt-names = "mmcirq"; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_mmc0>; @@ -589,7 +589,7 @@ status = "disabled"; reg = <0x09080000 0x7ff>; reg-names = "mmc"; - interrupts = ; + interrupts = ; interrupt-names = "mmcirq"; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_sd1>; @@ -623,7 +623,7 @@ compatible = "st,ahci"; reg = <0x9b20000 0x1000>; - interrupts = ; + interrupts = ; interrupt-names = "hostc"; phys = <&phy_port0 PHY_TYPE_SATA>; @@ -646,7 +646,7 @@ compatible = "st,ahci"; reg = <0x9b28000 0x1000>; - interrupts = ; + interrupts = ; interrupt-names = "hostc"; phys = <&phy_port1 PHY_TYPE_SATA>; @@ -687,7 +687,7 @@ dwc3: dwc3@9900000 { compatible = "snps,dwc3"; reg = <0x09900000 0x100000>; - interrupts = ; + interrupts = ; dr_mode = "host"; phy-names = "usb2-phy", "usb3-phy"; phys = <&usb2_picophy0>, @@ -701,7 +701,7 @@ compatible = "st,sti-pwm"; #pwm-cells = <2>; reg = <0x9810000 0x68>; - interrupts = ; + interrupts = ; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_pwm0_chan0_default>; clock-names = "pwm"; @@ -716,7 +716,7 @@ compatible = "st,sti-pwm"; #pwm-cells = <2>; reg = <0x9510000 0x68>; - interrupts = ; + interrupts = ; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_pwm1_chan0_default &pinctrl_pwm1_chan1_default @@ -755,8 +755,8 @@ resets = <&softreset STIH407_ETH1_SOFTRESET>; reset-names = "stmmaceth"; - interrupts = , - ; + interrupts = , + ; interrupt-names = "macirq", "eth_wake_irq"; /* DMA Bus Mode */ @@ -787,7 +787,7 @@ mailbox0: mailbox@8f00000 { compatible = "st,stih407-mailbox"; reg = <0x8f00000 0x1000>; - interrupts = ; + interrupts = ; #mbox-cells = <2>; mbox-name = "a9"; status = "okay"; @@ -857,7 +857,7 @@ <&clk_s_c0_flexgen CLK_EXT2F_A9>, <&clk_s_c0_flexgen CLK_EXT2F_A9>, <&clk_s_c0_flexgen CLK_EXT2F_A9>; - interrupts = ; + interrupts = ; dma-channels = <16>; #dma-cells = <3>; }; @@ -875,7 +875,7 @@ <&clk_s_c0_flexgen CLK_TX_ICN_DMU>, <&clk_s_c0_flexgen CLK_EXT2F_A9>; - interrupts = ; + interrupts = ; dma-channels = <16>; #dma-cells = <3>; @@ -890,7 +890,7 @@ <0x8e77000 0x1000>, <0x8e78000 0x8000>; reg-names = "slimcore", "dmem", "peripherals", "imem"; - interrupts = ; + interrupts = ; dma-channels = <16>; #dma-cells = <3>; clocks = <&clk_s_c0_flexgen CLK_FDMA>, @@ -910,7 +910,7 @@ assigned-clock-parents = <0>, <&clk_s_d0_quadfs 0>; assigned-clock-rates = <50000000>; reg = <0x8d80000 0x158>; - interrupts = ; + interrupts = ; dmas = <&fdma0 2 0 1>; dma-names = "tx"; @@ -926,7 +926,7 @@ assigned-clock-parents = <0>, <&clk_s_d0_quadfs 1>; assigned-clock-rates = <50000000>; reg = <0x8d81000 0x158>; - interrupts = ; + interrupts = ; dmas = <&fdma0 3 0 1>; dma-names = "tx"; @@ -942,7 +942,7 @@ assigned-clock-parents = <0>, <&clk_s_d0_quadfs 2>; assigned-clock-rates = <50000000>; reg = <0x8d82000 0x158>; - interrupts = ; + interrupts = ; dmas = <&fdma0 4 0 1>; dma-names = "tx"; @@ -958,7 +958,7 @@ assigned-clock-parents = <0>, <&clk_s_d0_quadfs 3>; assigned-clock-rates = <50000000>; reg = <0x8d85000 0x158>; - interrupts = ; + interrupts = ; dmas = <&fdma0 7 0 1>; dma-names = "tx"; @@ -970,7 +970,7 @@ #sound-dai-cells = <0>; st,syscfg = <&syscfg_core>; reg = <0x8d83000 0x158>; - interrupts = ; + interrupts = ; dmas = <&fdma0 5 0 1>; dma-names = "rx"; @@ -982,7 +982,7 @@ #sound-dai-cells = <0>; st,syscfg = <&syscfg_core>; reg = <0x8d84000 0x158>; - interrupts = ; + interrupts = ; dmas = <&fdma0 6 0 1>; dma-names = "rx"; diff --git a/arch/arm/boot/dts/stih407-pinctrl.dtsi b/arch/arm/boot/dts/stih407-pinctrl.dtsi index 53c6888d1fc0..e393519fb84c 100644 --- a/arch/arm/boot/dts/stih407-pinctrl.dtsi +++ b/arch/arm/boot/dts/stih407-pinctrl.dtsi @@ -52,7 +52,7 @@ st,syscfg = <&syscfg_sbc>; reg = <0x0961f080 0x4>; reg-names = "irqmux"; - interrupts = ; + interrupts = ; interrupt-names = "irqmux"; ranges = <0 0x09610000 0x6000>; @@ -376,7 +376,7 @@ st,syscfg = <&syscfg_front>; reg = <0x0920f080 0x4>; reg-names = "irqmux"; - interrupts = ; + interrupts = ; interrupt-names = "irqmux"; ranges = <0 0x09200000 0x10000>; @@ -936,7 +936,7 @@ st,syscfg = <&syscfg_front>; reg = <0x0921f080 0x4>; reg-names = "irqmux"; - interrupts = ; + interrupts = ; interrupt-names = "irqmux"; ranges = <0 0x09210000 0x10000>; @@ -969,7 +969,7 @@ st,syscfg = <&syscfg_rear>; reg = <0x0922f080 0x4>; reg-names = "irqmux"; - interrupts = ; + interrupts = ; interrupt-names = "irqmux"; ranges = <0 0x09220000 0x6000>; @@ -1164,7 +1164,7 @@ st,syscfg = <&syscfg_flash>; reg = <0x0923f080 0x4>; reg-names = "irqmux"; - interrupts = ; + interrupts = ; interrupt-names = "irqmux"; ranges = <0 0x09230000 0x3000>; diff --git a/arch/arm/boot/dts/stih407.dtsi b/arch/arm/boot/dts/stih407.dtsi index 57efc87dec2b..5b7951ffc350 100644 --- a/arch/arm/boot/dts/stih407.dtsi +++ b/arch/arm/boot/dts/stih407.dtsi @@ -108,7 +108,7 @@ reg = <0x8d04000 0x1000>; reg-names = "hdmi-reg"; #sound-dai-cells = <0>; - interrupts = ; + interrupts = ; interrupt-names = "irq"; clock-names = "pix", "tmds", diff --git a/arch/arm/boot/dts/stih410.dtsi b/arch/arm/boot/dts/stih410.dtsi index 3313005ee15c..888548ea9b5c 100644 --- a/arch/arm/boot/dts/stih410.dtsi +++ b/arch/arm/boot/dts/stih410.dtsi @@ -43,7 +43,7 @@ ohci0: usb@9a03c00 { compatible = "st,st-ohci-300x"; reg = <0x9a03c00 0x100>; - interrupts = ; + interrupts = ; clocks = <&clk_s_c0_flexgen CLK_TX_ICN_DISP_0>, <&clk_s_c0_flexgen CLK_RX_ICN_DISP_0>; resets = <&powerdown STIH407_USB2_PORT0_POWERDOWN>, @@ -58,7 +58,7 @@ ehci0: usb@9a03e00 { compatible = "st,st-ehci-300x"; reg = <0x9a03e00 0x100>; - interrupts = ; + interrupts = ; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_usb0>; clocks = <&clk_s_c0_flexgen CLK_TX_ICN_DISP_0>, @@ -75,7 +75,7 @@ ohci1: usb@9a83c00 { compatible = "st,st-ohci-300x"; reg = <0x9a83c00 0x100>; - interrupts = ; + interrupts = ; clocks = <&clk_s_c0_flexgen CLK_TX_ICN_DISP_0>, <&clk_s_c0_flexgen CLK_RX_ICN_DISP_0>; resets = <&powerdown STIH407_USB2_PORT1_POWERDOWN>, @@ -90,7 +90,7 @@ ehci1: usb@9a83e00 { compatible = "st,st-ehci-300x"; reg = <0x9a83e00 0x100>; - interrupts = ; + interrupts = ; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_usb1>; clocks = <&clk_s_c0_flexgen CLK_TX_ICN_DISP_0>, @@ -202,7 +202,7 @@ reg = <0x8d04000 0x1000>; reg-names = "hdmi-reg"; #sound-dai-cells = <0>; - interrupts = ; + interrupts = ; interrupt-names = "irq"; clock-names = "pix", "tmds", @@ -254,7 +254,7 @@ bdisp0:bdisp@9f10000 { compatible = "st,stih407-bdisp"; reg = <0x9f10000 0x1000>; - interrupts = ; + interrupts = ; clock-names = "bdisp"; clocks = <&clk_s_c0_flexgen CLK_IC_BDISP_0>; }; @@ -263,8 +263,8 @@ compatible = "st,st-hva"; reg = <0x8c85000 0x400>, <0x6000000 0x40000>; reg-names = "hva_registers", "hva_esram"; - interrupts = , - ; + interrupts = , + ; clock-names = "clk_hva"; clocks = <&clk_s_c0_flexgen CLK_HVA>; }; @@ -292,7 +292,7 @@ reg = <0x94a087c 0x64>; clocks = <&clk_sysin>; clock-names = "cec-clk"; - interrupts = ; + interrupts = ; interrupt-names = "cec-irq"; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_cec0_default>; diff --git a/arch/arm/boot/dts/stihxxx-b2120.dtsi b/arch/arm/boot/dts/stihxxx-b2120.dtsi index c67edb1a8121..4dedfcb0fcb3 100644 --- a/arch/arm/boot/dts/stihxxx-b2120.dtsi +++ b/arch/arm/boot/dts/stihxxx-b2120.dtsi @@ -154,8 +154,8 @@ reg = <0x08a20000 0x10000>, <0x08a00000 0x4000>; reg-names = "c8sectpfe", "c8sectpfe-ram"; - interrupts = , - ; + interrupts = , + ; interrupt-names = "c8sectpfe-error-irq", "c8sectpfe-idle-irq"; pinctrl-0 = <&pinctrl_tsin0_serial>; diff --git a/arch/arm/boot/dts/stm32f469-disco.dts b/arch/arm/boot/dts/stm32f469-disco.dts index 2f76726bf335..3ee768cb86fc 100644 --- a/arch/arm/boot/dts/stm32f469-disco.dts +++ b/arch/arm/boot/dts/stm32f469-disco.dts @@ -46,7 +46,7 @@ */ /dts-v1/; -#include "stm32f429.dtsi" +#include "stm32f469.dtsi" #include "stm32f469-pinctrl.dtsi" #include #include @@ -112,7 +112,7 @@ vcc5v_otg: vcc5v-otg-regulator { compatible = "regulator-fixed"; enable-active-high; - gpio = <&gpiob 2 0>; + gpio = <&gpiob 2 GPIO_ACTIVE_HIGH>; regulator-name = "vcc5_host1"; regulator-always-on; }; @@ -126,6 +126,55 @@ clock-frequency = <8000000>; }; +&dsi { + #address-cells = <1>; + #size-cells = <0>; + status = "okay"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + dsi_in: endpoint { + remote-endpoint = <<dc_out_dsi>; + }; + }; + + port@1 { + reg = <1>; + dsi_out: endpoint { + remote-endpoint = <&dsi_panel_in>; + }; + }; + }; + + panel-dsi@0 { + compatible = "orisetech,otm8009a"; + reg = <0>; /* dsi virtual channel (0..3) */ + reset-gpios = <&gpioh 7 GPIO_ACTIVE_LOW>; + status = "okay"; + + port { + dsi_panel_in: endpoint { + remote-endpoint = <&dsi_out>; + }; + }; + }; +}; + +<dc { + dma-ranges; + status = "okay"; + + port { + ltdc_out_dsi: endpoint@0 { + remote-endpoint = <&dsi_in>; + }; + }; +}; + &rtc { status = "okay"; }; diff --git a/arch/arm/boot/dts/stm32f469.dtsi b/arch/arm/boot/dts/stm32f469.dtsi new file mode 100644 index 000000000000..5ae5213f68cb --- /dev/null +++ b/arch/arm/boot/dts/stm32f469.dtsi @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause) */ +/* Copyright (C) STMicroelectronics 2017 - All Rights Reserved */ + +#include "stm32f429.dtsi" + +/ { + soc { + dsi: dsi@40016c00 { + compatible = "st,stm32-dsi"; + reg = <0x40016c00 0x800>; + interrupts = <92>; + resets = <&rcc STM32F4_APB2_RESET(DSI)>; + reset-names = "apb"; + clocks = <&rcc 1 CLK_F469_DSI>, <&clk_hse>; + clock-names = "pclk", "ref"; + status = "disabled"; + }; + }; +}; diff --git a/arch/arm/boot/dts/stm32f746-disco.dts b/arch/arm/boot/dts/stm32f746-disco.dts index be94c6ad7e94..f9ad71f7c807 100644 --- a/arch/arm/boot/dts/stm32f746-disco.dts +++ b/arch/arm/boot/dts/stm32f746-disco.dts @@ -90,6 +90,14 @@ clock-frequency = <25000000>; }; +&i2c1 { + pinctrl-0 = <&i2c1_pins_b>; + pinctrl-names = "default"; + i2c-scl-rising-time-ns = <185>; + i2c-scl-falling-time-ns = <20>; + status = "okay"; +}; + &sdio1 { status = "okay"; vmmc-supply = <&mmc_vcard>; diff --git a/arch/arm/boot/dts/stm32f746.dtsi b/arch/arm/boot/dts/stm32f746.dtsi index 4be2ee575b19..1479e3eb05fa 100644 --- a/arch/arm/boot/dts/stm32f746.dtsi +++ b/arch/arm/boot/dts/stm32f746.dtsi @@ -345,6 +345,42 @@ status = "disabled"; }; + i2c2: i2c@40005800 { + compatible = "st,stm32f7-i2c"; + reg = <0x40005800 0x400>; + interrupts = <33>, + <34>; + resets = <&rcc STM32F7_APB1_RESET(I2C2)>; + clocks = <&rcc 1 CLK_I2C2>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + i2c3: i2c@40005C00 { + compatible = "st,stm32f7-i2c"; + reg = <0x40005C00 0x400>; + interrupts = <72>, + <73>; + resets = <&rcc STM32F7_APB1_RESET(I2C3)>; + clocks = <&rcc 1 CLK_I2C3>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + i2c4: i2c@40006000 { + compatible = "st,stm32f7-i2c"; + reg = <0x40006000 0x400>; + interrupts = <95>, + <96>; + resets = <&rcc STM32F7_APB1_RESET(I2C4)>; + clocks = <&rcc 1 CLK_I2C4>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + cec: cec@40006c00 { compatible = "st,stm32-cec"; reg = <0x40006C00 0x400>; diff --git a/arch/arm/boot/dts/stm32f769-disco.dts b/arch/arm/boot/dts/stm32f769-disco.dts index 2241eecdabfe..677276ba4dbe 100644 --- a/arch/arm/boot/dts/stm32f769-disco.dts +++ b/arch/arm/boot/dts/stm32f769-disco.dts @@ -111,6 +111,14 @@ clock-frequency = <25000000>; }; +&i2c1 { + pinctrl-0 = <&i2c1_pins_b>; + pinctrl-names = "default"; + i2c-scl-rising-time-ns = <185>; + i2c-scl-falling-time-ns = <20>; + status = "okay"; +}; + &rtc { status = "okay"; }; diff --git a/arch/arm/boot/dts/stm32h743-pinctrl.dtsi b/arch/arm/boot/dts/stm32h743-pinctrl.dtsi index 0f15dfb98381..24be8e63dec8 100644 --- a/arch/arm/boot/dts/stm32h743-pinctrl.dtsi +++ b/arch/arm/boot/dts/stm32h743-pinctrl.dtsi @@ -163,6 +163,16 @@ #interrupt-cells = <2>; }; + i2c1_pins_a: i2c1@0 { + pins { + pinmux = , /* I2C1_SCL */ + ; /* I2C1_SDA */ + bias-disable; + drive-open-drain; + slew-rate = <0>; + }; + }; + usart1_pins: usart1@0 { pins1 { pinmux = ; /* USART1_TX */ diff --git a/arch/arm/boot/dts/stm32h743.dtsi b/arch/arm/boot/dts/stm32h743.dtsi index 2bb103e1194d..637beffe5067 100644 --- a/arch/arm/boot/dts/stm32h743.dtsi +++ b/arch/arm/boot/dts/stm32h743.dtsi @@ -86,6 +86,7 @@ pwm { compatible = "st,stm32-pwm-lp"; + #pwm-cells = <3>; status = "disabled"; }; @@ -130,6 +131,42 @@ clocks = <&rcc USART2_CK>; }; + i2c1: i2c@40005400 { + compatible = "st,stm32f7-i2c"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x40005400 0x400>; + interrupts = <31>, + <32>; + resets = <&rcc STM32H7_APB1L_RESET(I2C1)>; + clocks = <&rcc I2C1_CK>; + status = "disabled"; + }; + + i2c2: i2c@40005800 { + compatible = "st,stm32f7-i2c"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x40005800 0x400>; + interrupts = <33>, + <34>; + resets = <&rcc STM32H7_APB1L_RESET(I2C2)>; + clocks = <&rcc I2C2_CK>; + status = "disabled"; + }; + + i2c3: i2c@40005C00 { + compatible = "st,stm32f7-i2c"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x40005C00 0x400>; + interrupts = <72>, + <73>; + resets = <&rcc STM32H7_APB1L_RESET(I2C3)>; + clocks = <&rcc I2C3_CK>; + status = "disabled"; + }; + dac: dac@40007400 { compatible = "st,stm32h7-dac-core"; reg = <0x40007400 0x400>; @@ -323,6 +360,18 @@ status = "disabled"; }; + i2c4: i2c@58001C00 { + compatible = "st,stm32f7-i2c"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x58001C00 0x400>; + interrupts = <95>, + <96>; + resets = <&rcc STM32H7_APB4_RESET(I2C4)>; + clocks = <&rcc I2C4_CK>; + status = "disabled"; + }; + lptimer2: timer@58002400 { #address-cells = <1>; #size-cells = <0>; @@ -334,6 +383,7 @@ pwm { compatible = "st,stm32-pwm-lp"; + #pwm-cells = <3>; status = "disabled"; }; @@ -360,6 +410,7 @@ pwm { compatible = "st,stm32-pwm-lp"; + #pwm-cells = <3>; status = "disabled"; }; @@ -381,6 +432,7 @@ pwm { compatible = "st,stm32-pwm-lp"; + #pwm-cells = <3>; status = "disabled"; }; }; @@ -396,6 +448,7 @@ pwm { compatible = "st,stm32-pwm-lp"; + #pwm-cells = <3>; status = "disabled"; }; }; diff --git a/arch/arm/boot/dts/stm32h743i-eval.dts b/arch/arm/boot/dts/stm32h743i-eval.dts index c7187e18ea16..3f8e0c4a998d 100644 --- a/arch/arm/boot/dts/stm32h743i-eval.dts +++ b/arch/arm/boot/dts/stm32h743i-eval.dts @@ -92,6 +92,14 @@ clock-frequency = <25000000>; }; +&i2c1 { + pinctrl-0 = <&i2c1_pins_a>; + pinctrl-names = "default"; + i2c-scl-rising-time-ns = <185>; + i2c-scl-falling-time-ns = <20>; + status = "okay"; +}; + &rtc { status = "okay"; }; diff --git a/arch/arm/boot/dts/stm32mp157-pinctrl.dtsi b/arch/arm/boot/dts/stm32mp157-pinctrl.dtsi index c0743305f31b..4839db146890 100644 --- a/arch/arm/boot/dts/stm32mp157-pinctrl.dtsi +++ b/arch/arm/boot/dts/stm32mp157-pinctrl.dtsi @@ -7,11 +7,13 @@ / { soc { - pinctrl: pin-controller { + pinctrl: pin-controller@50002000 { #address-cells = <1>; #size-cells = <1>; compatible = "st,stm32mp157-pinctrl"; ranges = <0 0x50002000 0xa400>; + interrupt-parent = <&exti>; + st,syscfg = <&exti 0x60 0xff>; pins-are-numbered; gpioa: gpio@50002000 { @@ -20,7 +22,7 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0x0 0x400>; - clocks = <&clk_pll3_p>; + clocks = <&rcc GPIOA>; st,bank-name = "GPIOA"; ngpios = <16>; gpio-ranges = <&pinctrl 0 0 16>; @@ -32,7 +34,7 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0x1000 0x400>; - clocks = <&clk_pll3_p>; + clocks = <&rcc GPIOB>; st,bank-name = "GPIOB"; ngpios = <16>; gpio-ranges = <&pinctrl 0 16 16>; @@ -44,7 +46,7 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0x2000 0x400>; - clocks = <&clk_pll3_p>; + clocks = <&rcc GPIOC>; st,bank-name = "GPIOC"; ngpios = <16>; gpio-ranges = <&pinctrl 0 32 16>; @@ -56,7 +58,7 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0x3000 0x400>; - clocks = <&clk_pll3_p>; + clocks = <&rcc GPIOD>; st,bank-name = "GPIOD"; ngpios = <16>; gpio-ranges = <&pinctrl 0 48 16>; @@ -68,7 +70,7 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0x4000 0x400>; - clocks = <&clk_pll3_p>; + clocks = <&rcc GPIOE>; st,bank-name = "GPIOE"; ngpios = <16>; gpio-ranges = <&pinctrl 0 64 16>; @@ -80,7 +82,7 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0x5000 0x400>; - clocks = <&clk_pll3_p>; + clocks = <&rcc GPIOF>; st,bank-name = "GPIOF"; ngpios = <16>; gpio-ranges = <&pinctrl 0 80 16>; @@ -92,7 +94,7 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0x6000 0x400>; - clocks = <&clk_pll3_p>; + clocks = <&rcc GPIOG>; st,bank-name = "GPIOG"; ngpios = <16>; gpio-ranges = <&pinctrl 0 96 16>; @@ -104,7 +106,7 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0x7000 0x400>; - clocks = <&clk_pll3_p>; + clocks = <&rcc GPIOH>; st,bank-name = "GPIOH"; ngpios = <16>; gpio-ranges = <&pinctrl 0 112 16>; @@ -116,7 +118,7 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0x8000 0x400>; - clocks = <&clk_pll3_p>; + clocks = <&rcc GPIOI>; st,bank-name = "GPIOI"; ngpios = <16>; gpio-ranges = <&pinctrl 0 128 16>; @@ -128,7 +130,7 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0x9000 0x400>; - clocks = <&clk_pll3_p>; + clocks = <&rcc GPIOJ>; st,bank-name = "GPIOJ"; ngpios = <16>; gpio-ranges = <&pinctrl 0 144 16>; @@ -140,13 +142,124 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0xa000 0x400>; - clocks = <&clk_pll3_p>; + clocks = <&rcc GPIOK>; st,bank-name = "GPIOK"; ngpios = <8>; gpio-ranges = <&pinctrl 0 160 8>; }; - uart4_pins_a: uart4@0 { + cec_pins_a: cec-0 { + pins { + pinmux = ; + bias-disable; + drive-open-drain; + slew-rate = <0>; + }; + }; + + i2c1_pins_a: i2c1-0 { + pins { + pinmux = , /* I2C1_SCL */ + ; /* I2C1_SDA */ + bias-disable; + drive-open-drain; + slew-rate = <0>; + }; + }; + + i2c2_pins_a: i2c2-0 { + pins { + pinmux = , /* I2C2_SCL */ + ; /* I2C2_SDA */ + bias-disable; + drive-open-drain; + slew-rate = <0>; + }; + }; + + i2c5_pins_a: i2c5-0 { + pins { + pinmux = , /* I2C5_SCL */ + ; /* I2C5_SDA */ + bias-disable; + drive-open-drain; + slew-rate = <0>; + }; + }; + + pwm2_pins_a: pwm2-0 { + pins { + pinmux = ; /* TIM2_CH4 */ + bias-pull-down; + drive-push-pull; + slew-rate = <0>; + }; + }; + + pwm8_pins_a: pwm8-0 { + pins { + pinmux = ; /* TIM8_CH4 */ + bias-pull-down; + drive-push-pull; + slew-rate = <0>; + }; + }; + + pwm12_pins_a: pwm12-0 { + pins { + pinmux = ; /* TIM12_CH1 */ + bias-pull-down; + drive-push-pull; + slew-rate = <0>; + }; + }; + + qspi_clk_pins_a: qspi-clk-0 { + pins { + pinmux = ; /* QSPI_CLK */ + bias-disable; + drive-push-pull; + slew-rate = <3>; + }; + }; + + qspi_bk1_pins_a: qspi-bk1-0 { + pins1 { + pinmux = , /* QSPI_BK1_IO0 */ + , /* QSPI_BK1_IO1 */ + , /* QSPI_BK1_IO2 */ + ; /* QSPI_BK1_IO3 */ + bias-disable; + drive-push-pull; + slew-rate = <3>; + }; + pins2 { + pinmux = ; /* QSPI_BK1_NCS */ + bias-pull-up; + drive-push-pull; + slew-rate = <3>; + }; + }; + + qspi_bk2_pins_a: qspi-bk2-0 { + pins1 { + pinmux = , /* QSPI_BK2_IO0 */ + , /* QSPI_BK2_IO1 */ + , /* QSPI_BK2_IO2 */ + ; /* QSPI_BK2_IO3 */ + bias-disable; + drive-push-pull; + slew-rate = <3>; + }; + pins2 { + pinmux = ; /* QSPI_BK2_NCS */ + bias-pull-up; + drive-push-pull; + slew-rate = <3>; + }; + }; + + uart4_pins_a: uart4-0 { pins1 { pinmux = ; /* UART4_TX */ bias-disable; @@ -160,12 +273,14 @@ }; }; - pinctrl_z: pin-controller-z { + pinctrl_z: pin-controller-z@54004000 { #address-cells = <1>; #size-cells = <1>; compatible = "st,stm32mp157-z-pinctrl"; ranges = <0 0x54004000 0x400>; pins-are-numbered; + interrupt-parent = <&exti>; + st,syscfg = <&exti 0x60 0xff>; status = "disabled"; gpioz: gpio@54004000 { @@ -174,12 +289,22 @@ interrupt-controller; #interrupt-cells = <2>; reg = <0 0x400>; - clocks = <&clk_pll2_p>; + clocks = <&rcc GPIOZ>; st,bank-name = "GPIOZ"; st,bank-ioport = <11>; ngpios = <8>; gpio-ranges = <&pinctrl_z 0 400 8>; }; + + i2c4_pins_a: i2c4-0 { + pins { + pinmux = , /* I2C4_SCL */ + ; /* I2C4_SDA */ + bias-disable; + drive-open-drain; + slew-rate = <0>; + }; + }; }; }; }; diff --git a/arch/arm/boot/dts/stm32mp157c-ed1.dts b/arch/arm/boot/dts/stm32mp157c-ed1.dts index 9f90337a22e3..ae336530b59b 100644 --- a/arch/arm/boot/dts/stm32mp157c-ed1.dts +++ b/arch/arm/boot/dts/stm32mp157c-ed1.dts @@ -16,13 +16,56 @@ stdout-path = "serial0:115200n8"; }; - memory { + memory@c0000000 { reg = <0xC0000000 0x40000000>; }; aliases { serial0 = &uart4; }; + + reg11: reg11 { + compatible = "regulator-fixed"; + regulator-name = "reg11"; + regulator-min-microvolt = <1100000>; + regulator-max-microvolt = <1100000>; + regulator-always-on; + }; + + reg18: reg18 { + compatible = "regulator-fixed"; + regulator-name = "reg18"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-always-on; + }; + + vdd_usb: vdd-usb { + compatible = "regulator-fixed"; + regulator-name = "vdd_usb"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; +}; + +&rng1 { + status = "okay"; +}; + +&timers6 { + status = "okay"; + timer@5 { + status = "okay"; + }; +}; + +&i2c4 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c4_pins_a>; + i2c-scl-rising-time-ns = <185>; + i2c-scl-falling-time-ns = <20>; + status = "okay"; }; &uart4 { @@ -30,3 +73,15 @@ pinctrl-0 = <&uart4_pins_a>; status = "okay"; }; + +&usbphyc_port0 { + phy-supply = <&vdd_usb>; + vdda1v1-supply = <®11>; + vdda1v8-supply = <®18>; +}; + +&usbphyc_port1 { + phy-supply = <&vdd_usb>; + vdda1v1-supply = <®11>; + vdda1v8-supply = <®18>; +}; diff --git a/arch/arm/boot/dts/stm32mp157c-ev1.dts b/arch/arm/boot/dts/stm32mp157c-ev1.dts index 57e6dbc52e09..9382d8063031 100644 --- a/arch/arm/boot/dts/stm32mp157c-ev1.dts +++ b/arch/arm/boot/dts/stm32mp157c-ev1.dts @@ -19,3 +19,90 @@ serial0 = &uart4; }; }; + +&cec { + pinctrl-names = "default"; + pinctrl-0 = <&cec_pins_a>; + status = "okay"; +}; + +&i2c2 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c2_pins_a>; + i2c-scl-rising-time-ns = <185>; + i2c-scl-falling-time-ns = <20>; + status = "okay"; +}; + +&i2c5 { + pinctrl-names = "default"; + pinctrl-0 = <&i2c5_pins_a>; + i2c-scl-rising-time-ns = <185>; + i2c-scl-falling-time-ns = <20>; + status = "okay"; +}; + +&qspi { + pinctrl-names = "default"; + pinctrl-0 = <&qspi_clk_pins_a &qspi_bk1_pins_a &qspi_bk2_pins_a>; + reg = <0x58003000 0x1000>, <0x70000000 0x4000000>; + #address-cells = <1>; + #size-cells = <0>; + status = "okay"; + + flash0: mx66l51235l@0 { + reg = <0>; + spi-rx-bus-width = <4>; + spi-max-frequency = <108000000>; + #address-cells = <1>; + #size-cells = <1>; + }; + + flash1: mx66l51235l@1 { + reg = <1>; + spi-rx-bus-width = <4>; + spi-max-frequency = <108000000>; + #address-cells = <1>; + #size-cells = <1>; + }; +}; + +&timers2 { + status = "disabled"; + pwm { + pinctrl-0 = <&pwm2_pins_a>; + pinctrl-names = "default"; + status = "okay"; + }; + timer@1 { + status = "okay"; + }; +}; + +&timers8 { + status = "disabled"; + pwm { + pinctrl-0 = <&pwm8_pins_a>; + pinctrl-names = "default"; + status = "okay"; + }; + timer@7 { + status = "okay"; + }; +}; + +&timers12 { + status = "disabled"; + pwm { + pinctrl-0 = <&pwm12_pins_a>; + pinctrl-names = "default"; + status = "okay"; + }; + timer@11 { + status = "okay"; + }; +}; + +&usbphyc { + status = "okay"; +}; diff --git a/arch/arm/boot/dts/stm32mp157c.dtsi b/arch/arm/boot/dts/stm32mp157c.dtsi index 9e17e42b02b2..7d1753893453 100644 --- a/arch/arm/boot/dts/stm32mp157c.dtsi +++ b/arch/arm/boot/dts/stm32mp157c.dtsi @@ -4,6 +4,8 @@ * Author: Ludovic Barre for STMicroelectronics. */ #include +#include +#include / { #address-cells = <1>; @@ -71,12 +73,6 @@ clock-frequency = <24000000>; }; - clk_pll_per: clk-pll-per { - #clock-cells = <0>; - compatible = "fixed-clock"; - clock-frequency = <64000000>; - }; - clk_hsi: clk-hsi { #clock-cells = <0>; compatible = "fixed-clock"; @@ -100,24 +96,6 @@ compatible = "fixed-clock"; clock-frequency = <4000000>; }; - - clk_pclk1: clk-pclk1 { - #clock-cells = <0>; - compatible = "fixed-clock"; - clock-frequency = <86000000>; - }; - - clk_pll3_p: clk-pll3_p { - #clock-cells = <0>; - compatible = "fixed-clock"; - clock-frequency = <172000000>; - }; - - clk_pll2_p: clk-pll2_p { - #clock-cells = <0>; - compatible = "fixed-clock"; - clock-frequency = <264000000>; - }; }; soc { @@ -127,67 +105,736 @@ interrupt-parent = <&intc>; ranges; + timers2: timer@40000000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x40000000 0x400>; + clocks = <&rcc TIM2_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + + timer@1 { + compatible = "st,stm32h7-timer-trigger"; + reg = <1>; + status = "disabled"; + }; + }; + + timers3: timer@40001000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x40001000 0x400>; + clocks = <&rcc TIM3_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + + timer@2 { + compatible = "st,stm32h7-timer-trigger"; + reg = <2>; + status = "disabled"; + }; + }; + + timers4: timer@40002000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x40002000 0x400>; + clocks = <&rcc TIM4_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + + timer@3 { + compatible = "st,stm32h7-timer-trigger"; + reg = <3>; + status = "disabled"; + }; + }; + + timers5: timer@40003000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x40003000 0x400>; + clocks = <&rcc TIM5_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + + timer@4 { + compatible = "st,stm32h7-timer-trigger"; + reg = <4>; + status = "disabled"; + }; + }; + + timers6: timer@40004000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x40004000 0x400>; + clocks = <&rcc TIM6_K>; + clock-names = "int"; + status = "disabled"; + + timer@5 { + compatible = "st,stm32h7-timer-trigger"; + reg = <5>; + status = "disabled"; + }; + }; + + timers7: timer@40005000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x40005000 0x400>; + clocks = <&rcc TIM7_K>; + clock-names = "int"; + status = "disabled"; + + timer@6 { + compatible = "st,stm32h7-timer-trigger"; + reg = <6>; + status = "disabled"; + }; + }; + + timers12: timer@40006000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x40006000 0x400>; + clocks = <&rcc TIM12_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + + timer@11 { + compatible = "st,stm32h7-timer-trigger"; + reg = <11>; + status = "disabled"; + }; + }; + + timers13: timer@40007000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x40007000 0x400>; + clocks = <&rcc TIM13_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + + timer@12 { + compatible = "st,stm32h7-timer-trigger"; + reg = <12>; + status = "disabled"; + }; + }; + + timers14: timer@40008000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x40008000 0x400>; + clocks = <&rcc TIM14_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + + timer@13 { + compatible = "st,stm32h7-timer-trigger"; + reg = <13>; + status = "disabled"; + }; + }; + + lptimer1: timer@40009000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-lptimer"; + reg = <0x40009000 0x400>; + clocks = <&rcc LPTIM1_K>; + clock-names = "mux"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm-lp"; + #pwm-cells = <3>; + status = "disabled"; + }; + + trigger@0 { + compatible = "st,stm32-lptimer-trigger"; + reg = <0>; + status = "disabled"; + }; + + counter { + compatible = "st,stm32-lptimer-counter"; + status = "disabled"; + }; + }; + usart2: serial@4000e000 { compatible = "st,stm32h7-uart"; reg = <0x4000e000 0x400>; - interrupts = ; - clocks = <&clk_pclk1>; + interrupts = ; + clocks = <&rcc USART2_K>; status = "disabled"; }; usart3: serial@4000f000 { compatible = "st,stm32h7-uart"; reg = <0x4000f000 0x400>; - interrupts = ; - clocks = <&clk_pclk1>; + interrupts = ; + clocks = <&rcc USART3_K>; status = "disabled"; }; uart4: serial@40010000 { compatible = "st,stm32h7-uart"; reg = <0x40010000 0x400>; - interrupts = ; - clocks = <&clk_pclk1>; + interrupts = ; + clocks = <&rcc UART4_K>; status = "disabled"; }; uart5: serial@40011000 { compatible = "st,stm32h7-uart"; reg = <0x40011000 0x400>; - interrupts = ; - clocks = <&clk_pclk1>; + interrupts = ; + clocks = <&rcc UART5_K>; status = "disabled"; }; + i2c1: i2c@40012000 { + compatible = "st,stm32f7-i2c"; + reg = <0x40012000 0x400>; + interrupt-names = "event", "error"; + interrupts = , + ; + clocks = <&rcc I2C1_K>; + resets = <&rcc I2C1_R>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + i2c2: i2c@40013000 { + compatible = "st,stm32f7-i2c"; + reg = <0x40013000 0x400>; + interrupt-names = "event", "error"; + interrupts = , + ; + clocks = <&rcc I2C2_K>; + resets = <&rcc I2C2_R>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + i2c3: i2c@40014000 { + compatible = "st,stm32f7-i2c"; + reg = <0x40014000 0x400>; + interrupt-names = "event", "error"; + interrupts = , + ; + clocks = <&rcc I2C3_K>; + resets = <&rcc I2C3_R>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + i2c5: i2c@40015000 { + compatible = "st,stm32f7-i2c"; + reg = <0x40015000 0x400>; + interrupt-names = "event", "error"; + interrupts = , + ; + clocks = <&rcc I2C5_K>; + resets = <&rcc I2C5_R>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + cec: cec@40016000 { + compatible = "st,stm32-cec"; + reg = <0x40016000 0x400>; + interrupts = ; + clocks = <&rcc CEC_K>, <&clk_lse>; + clock-names = "cec", "hdmi-cec"; + status = "disabled"; + }; + + dac: dac@40017000 { + compatible = "st,stm32h7-dac-core"; + reg = <0x40017000 0x400>; + clocks = <&rcc DAC12>; + clock-names = "pclk"; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + + dac1: dac@1 { + compatible = "st,stm32-dac"; + #io-channels-cells = <1>; + reg = <1>; + status = "disabled"; + }; + + dac2: dac@2 { + compatible = "st,stm32-dac"; + #io-channels-cells = <1>; + reg = <2>; + status = "disabled"; + }; + }; + uart7: serial@40018000 { compatible = "st,stm32h7-uart"; reg = <0x40018000 0x400>; - interrupts = ; - clocks = <&clk_pclk1>; + interrupts = ; + clocks = <&rcc UART7_K>; status = "disabled"; }; uart8: serial@40019000 { compatible = "st,stm32h7-uart"; reg = <0x40019000 0x400>; - interrupts = ; - clocks = <&clk_pclk1>; + interrupts = ; + clocks = <&rcc UART8_K>; status = "disabled"; }; + timers1: timer@44000000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x44000000 0x400>; + clocks = <&rcc TIM1_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + + timer@0 { + compatible = "st,stm32h7-timer-trigger"; + reg = <0>; + status = "disabled"; + }; + }; + + timers8: timer@44001000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x44001000 0x400>; + clocks = <&rcc TIM8_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + + timer@7 { + compatible = "st,stm32h7-timer-trigger"; + reg = <7>; + status = "disabled"; + }; + }; + usart6: serial@44003000 { compatible = "st,stm32h7-uart"; reg = <0x44003000 0x400>; - interrupts = ; - clocks = <&clk_pclk1>; + interrupts = ; + clocks = <&rcc USART6_K>; status = "disabled"; }; + timers15: timer@44006000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x44006000 0x400>; + clocks = <&rcc TIM15_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + + timer@14 { + compatible = "st,stm32h7-timer-trigger"; + reg = <14>; + status = "disabled"; + }; + }; + + timers16: timer@44007000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x44007000 0x400>; + clocks = <&rcc TIM16_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + timer@15 { + compatible = "st,stm32h7-timer-trigger"; + reg = <15>; + status = "disabled"; + }; + }; + + timers17: timer@44008000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-timers"; + reg = <0x44008000 0x400>; + clocks = <&rcc TIM17_K>; + clock-names = "int"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm"; + status = "disabled"; + }; + + timer@16 { + compatible = "st,stm32h7-timer-trigger"; + reg = <16>; + status = "disabled"; + }; + }; + + dma1: dma@48000000 { + compatible = "st,stm32-dma"; + reg = <0x48000000 0x400>; + interrupts = , + , + , + , + , + , + , + ; + clocks = <&rcc DMA1>; + #dma-cells = <4>; + st,mem2mem; + dma-requests = <8>; + }; + + dma2: dma@48001000 { + compatible = "st,stm32-dma"; + reg = <0x48001000 0x400>; + interrupts = , + , + , + , + , + , + , + ; + clocks = <&rcc DMA2>; + #dma-cells = <4>; + st,mem2mem; + dma-requests = <8>; + }; + + dmamux1: dma-router@48002000 { + compatible = "st,stm32h7-dmamux"; + reg = <0x48002000 0x1c>; + #dma-cells = <3>; + dma-requests = <128>; + dma-masters = <&dma1 &dma2>; + dma-channels = <16>; + clocks = <&rcc DMAMUX>; + }; + + rcc: rcc@50000000 { + compatible = "st,stm32mp1-rcc", "syscon"; + reg = <0x50000000 0x1000>; + #clock-cells = <1>; + #reset-cells = <1>; + }; + + exti: interrupt-controller@5000d000 { + compatible = "st,stm32mp1-exti", "syscon"; + interrupt-controller; + #interrupt-cells = <2>; + reg = <0x5000d000 0x400>; + }; + + lptimer2: timer@50021000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-lptimer"; + reg = <0x50021000 0x400>; + clocks = <&rcc LPTIM2_K>; + clock-names = "mux"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm-lp"; + #pwm-cells = <3>; + status = "disabled"; + }; + + trigger@1 { + compatible = "st,stm32-lptimer-trigger"; + reg = <1>; + status = "disabled"; + }; + + counter { + compatible = "st,stm32-lptimer-counter"; + status = "disabled"; + }; + }; + + lptimer3: timer@50022000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32-lptimer"; + reg = <0x50022000 0x400>; + clocks = <&rcc LPTIM3_K>; + clock-names = "mux"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm-lp"; + #pwm-cells = <3>; + status = "disabled"; + }; + + trigger@2 { + compatible = "st,stm32-lptimer-trigger"; + reg = <2>; + status = "disabled"; + }; + }; + + lptimer4: timer@50023000 { + compatible = "st,stm32-lptimer"; + reg = <0x50023000 0x400>; + clocks = <&rcc LPTIM4_K>; + clock-names = "mux"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm-lp"; + #pwm-cells = <3>; + status = "disabled"; + }; + }; + + lptimer5: timer@50024000 { + compatible = "st,stm32-lptimer"; + reg = <0x50024000 0x400>; + clocks = <&rcc LPTIM5_K>; + clock-names = "mux"; + status = "disabled"; + + pwm { + compatible = "st,stm32-pwm-lp"; + #pwm-cells = <3>; + status = "disabled"; + }; + }; + + vrefbuf: vrefbuf@50025000 { + compatible = "st,stm32-vrefbuf"; + reg = <0x50025000 0x8>; + regulator-min-microvolt = <1500000>; + regulator-max-microvolt = <2500000>; + clocks = <&rcc VREF>; + status = "disabled"; + }; + + cryp1: cryp@54001000 { + compatible = "st,stm32mp1-cryp"; + reg = <0x54001000 0x400>; + interrupts = ; + clocks = <&rcc CRYP1>; + resets = <&rcc CRYP1_R>; + status = "disabled"; + }; + + rng1: rng@54003000 { + compatible = "st,stm32-rng"; + reg = <0x54003000 0x400>; + clocks = <&rcc RNG1_K>; + resets = <&rcc RNG1_R>; + status = "disabled"; + }; + + mdma1: dma@58000000 { + compatible = "st,stm32h7-mdma"; + reg = <0x58000000 0x1000>; + interrupts = ; + clocks = <&rcc MDMA>; + #dma-cells = <5>; + dma-channels = <32>; + dma-requests = <48>; + }; + + qspi: qspi@58003000 { + compatible = "st,stm32f469-qspi"; + reg = <0x58003000 0x1000>, <0x70000000 0x10000000>; + reg-names = "qspi", "qspi_mm"; + interrupts = ; + clocks = <&rcc QSPI_K>; + resets = <&rcc QSPI_R>; + status = "disabled"; + }; + + crc1: crc@58009000 { + compatible = "st,stm32f7-crc"; + reg = <0x58009000 0x400>; + clocks = <&rcc CRC1>; + status = "disabled"; + }; + + usbh_ohci: usbh-ohci@5800c000 { + compatible = "generic-ohci"; + reg = <0x5800c000 0x1000>; + clocks = <&rcc USBH>; + resets = <&rcc USBH_R>; + interrupts = ; + status = "disabled"; + }; + + usbh_ehci: usbh-ehci@5800d000 { + compatible = "generic-ehci"; + reg = <0x5800d000 0x1000>; + clocks = <&rcc USBH>; + resets = <&rcc USBH_R>; + interrupts = ; + companion = <&usbh_ohci>; + status = "disabled"; + }; + + dsi: dsi@5a000000 { + compatible = "st,stm32-dsi"; + reg = <0x5a000000 0x800>; + clocks = <&rcc DSI_K>, <&clk_hse>, <&rcc DSI_PX>; + clock-names = "pclk", "ref", "px_clk"; + resets = <&rcc DSI_R>; + reset-names = "apb"; + status = "disabled"; + }; + + ltdc: display-controller@5a001000 { + compatible = "st,stm32-ltdc"; + reg = <0x5a001000 0x400>; + interrupts = , + ; + clocks = <&rcc LTDC_PX>; + clock-names = "lcd"; + resets = <&rcc LTDC_R>; + status = "disabled"; + }; + + usbphyc: usbphyc@5a006000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "st,stm32mp1-usbphyc"; + reg = <0x5a006000 0x1000>; + clocks = <&rcc USBPHY_K>; + resets = <&rcc USBPHY_R>; + status = "disabled"; + + usbphyc_port0: usb-phy@0 { + #phy-cells = <0>; + reg = <0>; + }; + + usbphyc_port1: usb-phy@1 { + #phy-cells = <1>; + reg = <1>; + }; + }; + usart1: serial@5c000000 { compatible = "st,stm32h7-uart"; reg = <0x5c000000 0x400>; - interrupts = ; - clocks = <&clk_pclk1>; + interrupts = ; + clocks = <&rcc USART1_K>; + status = "disabled"; + }; + + i2c4: i2c@5c002000 { + compatible = "st,stm32f7-i2c"; + reg = <0x5c002000 0x400>; + interrupt-names = "event", "error"; + interrupts = , + ; + clocks = <&rcc I2C4_K>; + resets = <&rcc I2C4_R>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + i2c6: i2c@5c009000 { + compatible = "st,stm32f7-i2c"; + reg = <0x5c009000 0x400>; + interrupt-names = "event", "error"; + interrupts = , + ; + clocks = <&rcc I2C6_K>; + resets = <&rcc I2C6_R>; + #address-cells = <1>; + #size-cells = <0>; status = "disabled"; }; }; diff --git a/arch/arm/boot/dts/sun7i-a20-olimex-som-evb-emmc.dts b/arch/arm/boot/dts/sun7i-a20-olimex-som-evb-emmc.dts new file mode 100644 index 000000000000..81ebc97b76ee --- /dev/null +++ b/arch/arm/boot/dts/sun7i-a20-olimex-som-evb-emmc.dts @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR MIT) +/* + * Device Tree Source for A20-Olimex-SOM-EVB-eMMC Board + * + * Copyright (C) 2018 Olimex Ltd. + * Author: Stefan Mavrodiev + */ + +/dts-v1/; +#include "sun7i-a20-olimex-som-evb.dts" + +/ { + + model = "Olimex A20-Olimex-SOM-EVB-eMMC"; + compatible = "olimex,a20-olimex-som-evb-emmc", "allwinner,sun7i-a20"; + + mmc2_pwrseq: mmc2_pwrseq { + compatible = "mmc-pwrseq-emmc"; + reset-gpios = <&pio 2 18 GPIO_ACTIVE_LOW>; + }; +}; + +&mmc2 { + pinctrl-names = "default"; + pinctrl-0 = <&mmc2_pins_a>; + vmmc-supply = <®_vcc3v3>; + mmc-pwrseq = <&mmc2_pwrseq>; + bus-width = <4>; + non-removable; + status = "okay"; + + emmc: emmc@0 { + reg = <0>; + compatible = "mmc-card"; + broken-hpi; + }; +}; diff --git a/arch/arm/boot/dts/sun7i-a20-olimex-som204-evb.dts b/arch/arm/boot/dts/sun7i-a20-olimex-som204-evb.dts index eae8e267b9ef..3d7b5c848fef 100644 --- a/arch/arm/boot/dts/sun7i-a20-olimex-som204-evb.dts +++ b/arch/arm/boot/dts/sun7i-a20-olimex-som204-evb.dts @@ -172,8 +172,7 @@ pinctrl-0 = <&mmc0_pins_a>; vmmc-supply = <®_vcc3v3>; bus-width = <4>; - cd-gpios = <&pio 7 1 GPIO_ACTIVE_HIGH>; - cd-inverted; + cd-gpios = <&pio 7 1 GPIO_ACTIVE_LOW>; status = "okay"; }; diff --git a/arch/arm/boot/dts/sun8i-a23-a33.dtsi b/arch/arm/boot/dts/sun8i-a23-a33.dtsi index 971f9be699a7..44f3cad3de75 100644 --- a/arch/arm/boot/dts/sun8i-a23-a33.dtsi +++ b/arch/arm/boot/dts/sun8i-a23-a33.dtsi @@ -198,6 +198,8 @@ clock-names = "ahb", "mod"; resets = <&ccu RST_BUS_NAND>; reset-names = "ahb"; + pinctrl-names = "default"; + pinctrl-0 = <&nand_pins &nand_pins_cs0 &nand_pins_rb0>; status = "disabled"; #address-cells = <1>; #size-cells = <0>; @@ -315,6 +317,37 @@ bias-pull-up; }; + nand_pins: nand-pins { + pins = "PC0", "PC1", "PC2", "PC5", + "PC8", "PC9", "PC10", "PC11", + "PC12", "PC13", "PC14", "PC15"; + function = "nand0"; + }; + + nand_pins_cs0: nand-pins-cs0 { + pins = "PC4"; + function = "nand0"; + bias-pull-up; + }; + + nand_pins_cs1: nand-pins-cs1 { + pins = "PC3"; + function = "nand0"; + bias-pull-up; + }; + + nand_pins_rb0: nand-pins-rb0 { + pins = "PC6"; + function = "nand0"; + bias-pull-up; + }; + + nand_pins_rb1: nand-pins-rb1 { + pins = "PC7"; + function = "nand0"; + bias-pull-up; + }; + pwm0_pins: pwm0 { pins = "PH0"; function = "pwm0"; diff --git a/arch/arm/boot/dts/sun8i-a33.dtsi b/arch/arm/boot/dts/sun8i-a33.dtsi index a21f2ed07a52..8d278ee001e9 100644 --- a/arch/arm/boot/dts/sun8i-a33.dtsi +++ b/arch/arm/boot/dts/sun8i-a33.dtsi @@ -236,6 +236,11 @@ #address-cells = <1>; #size-cells = <0>; reg = <1>; + + tcon0_out_dsi: endpoint@1 { + reg = <1>; + remote-endpoint = <&dsi_in_tcon0>; + }; }; }; }; @@ -280,6 +285,45 @@ #io-channel-cells = <0>; }; + dsi: dsi@1ca0000 { + compatible = "allwinner,sun6i-a31-mipi-dsi"; + reg = <0x01ca0000 0x1000>; + interrupts = ; + clocks = <&ccu CLK_BUS_MIPI_DSI>, + <&ccu CLK_DSI_SCLK>; + clock-names = "bus", "mod"; + resets = <&ccu RST_BUS_MIPI_DSI>; + phys = <&dphy>; + phy-names = "dphy"; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + #address-cells = <1>; + #size-cells = <0>; + reg = <0>; + + dsi_in_tcon0: endpoint { + remote-endpoint = <&tcon0_out_dsi>; + }; + }; + }; + }; + + dphy: d-phy@1ca1000 { + compatible = "allwinner,sun6i-a31-mipi-dphy"; + reg = <0x01ca1000 0x1000>; + clocks = <&ccu CLK_BUS_MIPI_DSI>, + <&ccu CLK_DSI_DPHY>; + clock-names = "bus", "mod"; + resets = <&ccu RST_BUS_MIPI_DSI>; + status = "disabled"; + #phy-cells = <0>; + }; + fe0: display-frontend@1e00000 { compatible = "allwinner,sun8i-a33-display-frontend"; reg = <0x01e00000 0x20000>; diff --git a/arch/arm/boot/dts/sun8i-a83t.dtsi b/arch/arm/boot/dts/sun8i-a83t.dtsi index 568307639be8..2be23d600957 100644 --- a/arch/arm/boot/dts/sun8i-a83t.dtsi +++ b/arch/arm/boot/dts/sun8i-a83t.dtsi @@ -66,6 +66,8 @@ compatible = "arm,cortex-a7"; device_type = "cpu"; operating-points-v2 = <&cpu0_opp_table>; + cci-control-port = <&cci_control0>; + enable-method = "allwinner,sun8i-a83t-smp"; reg = <0>; }; @@ -73,6 +75,8 @@ compatible = "arm,cortex-a7"; device_type = "cpu"; operating-points-v2 = <&cpu0_opp_table>; + cci-control-port = <&cci_control0>; + enable-method = "allwinner,sun8i-a83t-smp"; reg = <1>; }; @@ -80,6 +84,8 @@ compatible = "arm,cortex-a7"; device_type = "cpu"; operating-points-v2 = <&cpu0_opp_table>; + cci-control-port = <&cci_control0>; + enable-method = "allwinner,sun8i-a83t-smp"; reg = <2>; }; @@ -87,6 +93,8 @@ compatible = "arm,cortex-a7"; device_type = "cpu"; operating-points-v2 = <&cpu0_opp_table>; + cci-control-port = <&cci_control0>; + enable-method = "allwinner,sun8i-a83t-smp"; reg = <3>; }; @@ -96,6 +104,8 @@ compatible = "arm,cortex-a7"; device_type = "cpu"; operating-points-v2 = <&cpu1_opp_table>; + cci-control-port = <&cci_control1>; + enable-method = "allwinner,sun8i-a83t-smp"; reg = <0x100>; }; @@ -103,6 +113,8 @@ compatible = "arm,cortex-a7"; device_type = "cpu"; operating-points-v2 = <&cpu1_opp_table>; + cci-control-port = <&cci_control1>; + enable-method = "allwinner,sun8i-a83t-smp"; reg = <0x101>; }; @@ -110,6 +122,8 @@ compatible = "arm,cortex-a7"; device_type = "cpu"; operating-points-v2 = <&cpu1_opp_table>; + cci-control-port = <&cci_control1>; + enable-method = "allwinner,sun8i-a83t-smp"; reg = <0x102>; }; @@ -117,6 +131,8 @@ compatible = "arm,cortex-a7"; device_type = "cpu"; operating-points-v2 = <&cpu1_opp_table>; + cci-control-port = <&cci_control1>; + enable-method = "allwinner,sun8i-a83t-smp"; reg = <0x103>; }; }; @@ -349,6 +365,44 @@ }; }; + cpucfg@1700000 { + compatible = "allwinner,sun8i-a83t-cpucfg"; + reg = <0x01700000 0x400>; + }; + + cci@1790000 { + compatible = "arm,cci-400"; + #address-cells = <1>; + #size-cells = <1>; + reg = <0x01790000 0x10000>; + ranges = <0x0 0x01790000 0x10000>; + + cci_control0: slave-if@4000 { + compatible = "arm,cci-400-ctrl-if"; + interface-type = "ace"; + reg = <0x4000 0x1000>; + }; + + cci_control1: slave-if@5000 { + compatible = "arm,cci-400-ctrl-if"; + interface-type = "ace"; + reg = <0x5000 0x1000>; + }; + + pmu@9000 { + compatible = "arm,cci-400-pmu,r1"; + reg = <0x9000 0x5000>; + interrupts = , + , + , + , + , + , + , + ; + }; + }; + syscon: syscon@1c00000 { compatible = "allwinner,sun8i-a83t-system-controller", "syscon"; @@ -492,6 +546,11 @@ #size-cells = <0>; }; + sid: eeprom@1c14000 { + compatible = "allwinner,sun8i-a83t-sid"; + reg = <0x1c14000 0x400>; + }; + usb_otg: usb@1c19000 { compatible = "allwinner,sun8i-a83t-musb", "allwinner,sun8i-a33-musb"; @@ -928,6 +987,11 @@ #reset-cells = <1>; }; + r_cpucfg@1f01c00 { + compatible = "allwinner,sun8i-a83t-r-cpucfg"; + reg = <0x1f01c00 0x400>; + }; + r_pio: pinctrl@1f02c00 { compatible = "allwinner,sun8i-a83t-r-pinctrl"; reg = <0x01f02c00 0x400>; diff --git a/arch/arm/boot/dts/sun8i-h2-plus-libretech-all-h3-cc.dts b/arch/arm/boot/dts/sun8i-h2-plus-libretech-all-h3-cc.dts new file mode 100644 index 000000000000..4db0d4bb65eb --- /dev/null +++ b/arch/arm/boot/dts/sun8i-h2-plus-libretech-all-h3-cc.dts @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR MIT) +/* + * Copyright (C) 2018 Chen-Yu Tsai + */ + +/dts-v1/; +#include "sun8i-h3.dtsi" +#include "sunxi-libretech-all-h3-cc.dtsi" + +/ { + model = "Libre Computer Board ALL-H3-CC H2+"; + compatible = "libretech,all-h3-cc-h2-plus", "allwinner,sun8i-h2-plus"; +}; diff --git a/arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dts b/arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dts index 0bc031fe4c56..84cd9c061227 100644 --- a/arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dts +++ b/arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dts @@ -89,6 +89,23 @@ gpio = <&pio 0 20 GPIO_ACTIVE_HIGH>; }; + reg_vdd_cpux: vdd-cpux-regulator { + compatible = "regulator-gpio"; + regulator-name = "vdd-cpux"; + regulator-type = "voltage"; + regulator-boot-on; + regulator-always-on; + regulator-min-microvolt = <1100000>; + regulator-max-microvolt = <1300000>; + regulator-ramp-delay = <50>; /* 4ms */ + + gpios = <&r_pio 0 6 GPIO_ACTIVE_HIGH>; /* PL6 */ + enable-active-high; + gpios-states = <1>; + states = <1100000 0 + 1300000 1>; + }; + wifi_pwrseq: wifi_pwrseq { compatible = "mmc-pwrseq-simple"; reset-gpios = <&r_pio 0 7 GPIO_ACTIVE_LOW>; @@ -96,6 +113,10 @@ }; }; +&cpu0 { + cpu-supply = <®_vdd_cpux>; +}; + &ehci0 { status = "okay"; }; diff --git a/arch/arm/boot/dts/sun8i-h3-libretech-all-h3-cc.dts b/arch/arm/boot/dts/sun8i-h3-libretech-all-h3-cc.dts index b20a710da7bc..a8b2f0f1c11d 100644 --- a/arch/arm/boot/dts/sun8i-h3-libretech-all-h3-cc.dts +++ b/arch/arm/boot/dts/sun8i-h3-libretech-all-h3-cc.dts @@ -6,213 +6,9 @@ /dts-v1/; #include "sun8i-h3.dtsi" - -#include -#include +#include "sunxi-libretech-all-h3-cc.dtsi" / { model = "Libre Computer Board ALL-H3-CC H3"; compatible = "libretech,all-h3-cc-h3", "allwinner,sun8i-h3"; - - aliases { - ethernet0 = &emac; - serial0 = &uart0; - }; - - chosen { - stdout-path = "serial0:115200n8"; - }; - - connector { - compatible = "hdmi-connector"; - type = "a"; - - port { - hdmi_con_in: endpoint { - remote-endpoint = <&hdmi_out_con>; - }; - }; - }; - - leds { - compatible = "gpio-leds"; - - pwr_led { - label = "librecomputer:green:pwr"; - gpios = <&r_pio 0 10 GPIO_ACTIVE_HIGH>; /* PL10 */ - default-state = "on"; - }; - - status_led { - label = "librecomputer:blue:status"; - gpios = <&pio 0 7 GPIO_ACTIVE_HIGH>; /* PA7 */ - }; - }; - - gpio_keys { - compatible = "gpio-keys"; - - power { - label = "power"; - linux,code = ; - gpios = <&r_pio 0 2 GPIO_ACTIVE_LOW>; /* PL2 */ - }; - }; - - reg_vcc1v2: vcc1v2 { - compatible = "regulator-fixed"; - regulator-name = "vcc1v2"; - regulator-min-microvolt = <3300000>; - regulator-max-microvolt = <3300000>; - regulator-always-on; - regulator-boot-on; - vin-supply = <®_vcc5v0>; - gpio = <&r_pio 0 8 GPIO_ACTIVE_HIGH>; /* PL8 */ - enable-active-high; - }; - - reg_vcc3v3: vcc3v3 { - compatible = "regulator-fixed"; - regulator-name = "vcc3v3"; - regulator-min-microvolt = <3300000>; - regulator-max-microvolt = <3300000>; - vin-supply = <®_vcc5v0>; - }; - - /* This represents the board's 5V input */ - reg_vcc5v0: vcc5v0 { - compatible = "regulator-fixed"; - regulator-name = "vcc5v0"; - regulator-min-microvolt = <5000000>; - regulator-max-microvolt = <5000000>; - }; - - reg_vcc_dram: vcc-dram { - compatible = "regulator-fixed"; - regulator-name = "vcc-dram"; - regulator-min-microvolt = <1500000>; - regulator-max-microvolt = <1500000>; - regulator-always-on; - regulator-boot-on; - vin-supply = <®_vcc5v0>; - gpio = <&r_pio 0 9 GPIO_ACTIVE_HIGH>; /* PL9 */ - enable-active-high; - }; - - reg_vcc_io: vcc-io { - compatible = "regulator-fixed"; - regulator-name = "vcc-io"; - regulator-min-microvolt = <3300000>; - regulator-max-microvolt = <3300000>; - regulator-always-on; - regulator-boot-on; - vin-supply = <®_vcc3v3>; - gpio = <&r_pio 0 5 GPIO_ACTIVE_LOW>; /* PL5 */ - }; - - reg_vdd_cpux: vdd-cpux { - compatible = "regulator-fixed"; - regulator-name = "vdd-cpux"; - regulator-min-microvolt = <3300000>; - regulator-max-microvolt = <3300000>; - regulator-always-on; - regulator-boot-on; - vin-supply = <®_vcc5v0>; - gpio = <&r_pio 0 8 GPIO_ACTIVE_HIGH>; /* PL8 */ - enable-active-high; - }; -}; - -&codec { - allwinner,audio-routing = - "Line Out", "LINEOUT", - "MIC1", "Mic", - "Mic", "MBIAS"; - status = "okay"; -}; - -&de { - status = "okay"; -}; - -&ehci0 { - status = "okay"; -}; - -&ehci1 { - status = "okay"; -}; - -&ehci2 { - status = "okay"; -}; - -&ehci3 { - status = "okay"; -}; - -&emac { - phy-handle = <&int_mii_phy>; - phy-mode = "mii"; - allwinner,leds-active-low; - status = "okay"; -}; - -&hdmi { - status = "okay"; -}; - -&hdmi_out { - hdmi_out_con: endpoint { - remote-endpoint = <&hdmi_con_in>; - }; -}; - -&ir { - pinctrl-names = "default"; - pinctrl-0 = <&ir_pins_a>; - status = "okay"; -}; - -&mmc0 { - vmmc-supply = <®_vcc_io>; - bus-width = <4>; - cd-gpios = <&pio 5 6 GPIO_ACTIVE_LOW>; /* PF6 */ - status = "okay"; -}; - -&ohci0 { - status = "okay"; -}; - -&ohci1 { - status = "okay"; -}; - -&ohci2 { - status = "okay"; -}; - -&ohci3 { - status = "okay"; -}; - -&uart0 { - pinctrl-names = "default"; - pinctrl-0 = <&uart0_pins_a>; - status = "okay"; -}; - -&usb_otg { - dr_mode = "host"; - status = "okay"; -}; - -&usbphy { - /* VBUS on USB ports are always on */ - usb0_vbus-supply = <®_vcc5v0>; - usb1_vbus-supply = <®_vcc5v0>; - usb2_vbus-supply = <®_vcc5v0>; - usb3_vbus-supply = <®_vcc5v0>; - status = "okay"; }; diff --git a/arch/arm/boot/dts/sun8i-h3-orangepi-one.dts b/arch/arm/boot/dts/sun8i-h3-orangepi-one.dts index 232f124ce62c..245fd658defb 100644 --- a/arch/arm/boot/dts/sun8i-h3-orangepi-one.dts +++ b/arch/arm/boot/dts/sun8i-h3-orangepi-one.dts @@ -99,6 +99,27 @@ gpios = <&r_pio 0 3 GPIO_ACTIVE_LOW>; }; }; + + reg_vdd_cpux: vdd-cpux-regulator { + compatible = "regulator-gpio"; + regulator-name = "vdd-cpux"; + regulator-type = "voltage"; + regulator-boot-on; + regulator-always-on; + regulator-min-microvolt = <1100000>; + regulator-max-microvolt = <1300000>; + regulator-ramp-delay = <50>; /* 4ms */ + + gpios = <&r_pio 0 6 GPIO_ACTIVE_HIGH>; /* PL6 */ + enable-active-high; + gpios-states = <0x1>; + states = <1100000 0x0 + 1300000 0x1>; + }; +}; + +&cpu0 { + cpu-supply = <®_vdd_cpux>; }; &de { diff --git a/arch/arm/boot/dts/sun8i-h3-orangepi-pc.dts b/arch/arm/boot/dts/sun8i-h3-orangepi-pc.dts index cea4d647ecbf..46240334128f 100644 --- a/arch/arm/boot/dts/sun8i-h3-orangepi-pc.dts +++ b/arch/arm/boot/dts/sun8i-h3-orangepi-pc.dts @@ -113,6 +113,10 @@ status = "okay"; }; +&cpu0 { + cpu-supply = <®_vdd_cpux>; +}; + &ehci0 { status = "okay"; }; @@ -182,6 +186,30 @@ }; }; +&r_i2c { + status = "okay"; + + reg_vdd_cpux: regulator@65 { + compatible = "silergy,sy8106a"; + reg = <0x65>; + regulator-name = "vdd-cpux"; + silergy,fixed-microvolt = <1200000>; + /* + * The datasheet uses 1.1V as the minimum value of VDD-CPUX, + * however both the Armbian DVFS table and the official one + * have operating points with voltage under 1.1V, and both + * DVFS table are known to work properly at the lowest + * operating point. + * + * Use 1.0V as the minimum voltage instead. + */ + regulator-min-microvolt = <1000000>; + regulator-max-microvolt = <1300000>; + regulator-boot-on; + regulator-always-on; + }; +}; + &r_pio { leds_r_opc: led_pins { pins = "PL10"; diff --git a/arch/arm/boot/dts/sun8i-h3.dtsi b/arch/arm/boot/dts/sun8i-h3.dtsi index 10da8ed7db81..41d57c76f290 100644 --- a/arch/arm/boot/dts/sun8i-h3.dtsi +++ b/arch/arm/boot/dts/sun8i-h3.dtsi @@ -43,32 +43,62 @@ #include "sunxi-h3-h5.dtsi" / { + cpu0_opp_table: opp_table0 { + compatible = "operating-points-v2"; + opp-shared; + + opp@648000000 { + opp-hz = /bits/ 64 <648000000>; + opp-microvolt = <1040000 1040000 1300000>; + clock-latency-ns = <244144>; /* 8 32k periods */ + }; + + opp@816000000 { + opp-hz = /bits/ 64 <816000000>; + opp-microvolt = <1100000 1100000 1300000>; + clock-latency-ns = <244144>; /* 8 32k periods */ + }; + + opp@1008000000 { + opp-hz = /bits/ 64 <1008000000>; + opp-microvolt = <1200000 1200000 1300000>; + clock-latency-ns = <244144>; /* 8 32k periods */ + }; + }; + cpus { #address-cells = <1>; #size-cells = <0>; - cpu@0 { + cpu0: cpu@0 { compatible = "arm,cortex-a7"; device_type = "cpu"; reg = <0>; + clocks = <&ccu CLK_CPUX>; + clock-names = "cpu"; + operating-points-v2 = <&cpu0_opp_table>; + #cooling-cells = <2>; }; cpu@1 { compatible = "arm,cortex-a7"; device_type = "cpu"; reg = <1>; + operating-points-v2 = <&cpu0_opp_table>; }; cpu@2 { compatible = "arm,cortex-a7"; device_type = "cpu"; reg = <2>; + operating-points-v2 = <&cpu0_opp_table>; }; cpu@3 { compatible = "arm,cortex-a7"; device_type = "cpu"; reg = <3>; + operating-points-v2 = <&cpu0_opp_table>; }; }; diff --git a/arch/arm/boot/dts/sun8i-r16-nintendo-nes-classic.dts b/arch/arm/boot/dts/sun8i-r16-nintendo-nes-classic.dts new file mode 100644 index 000000000000..fc0658cfa319 --- /dev/null +++ b/arch/arm/boot/dts/sun8i-r16-nintendo-nes-classic.dts @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: GPL-2.0 OR X11 +/* Copyright (c) 2016 FUKAUMI Naoki */ + +/dts-v1/; +#include "sun8i-a33.dtsi" +#include "sunxi-common-regulators.dtsi" + +/ { + model = "Nintendo NES Classic Edition"; + compatible = "nintendo,nes-classic", "allwinner,sun8i-r16", + "allwinner,sun8i-a33"; + + aliases { + serial0 = &uart0; + }; + + chosen { + stdout-path = "serial0:115200n8"; + }; +}; + +&uart0 { + /* + * UART0 is available on two ports: PB and PF, both are accessible. + * PF can also be used for the SD card so PB is preferred. + */ + pinctrl-names = "default"; + pinctrl-0 = <&uart0_pins_a>; + status = "okay"; +}; + +&nfc { + status = "okay"; + + /* 2Gb Macronix MX30LF2G18AC (3V) */ + nand@0 { + #address-cells = <1>; + #size-cells = <1>; + reg = <0>; + allwinner,rb = <0>; + nand-ecc-mode = "hw"; + nand-ecc-strength = <16>; + nand-ecc-step-size = <1024>; + }; +}; + +&usb_otg { + status = "okay"; + dr_mode = "otg"; +}; + +&usbphy { + /* VBUS is always on because it is wired to the power supply */ + usb1_vbus-supply = <®_vcc5v0>; + status = "okay"; +}; diff --git a/arch/arm/boot/dts/sun8i-r16-nintendo-super-nes-classic.dts b/arch/arm/boot/dts/sun8i-r16-nintendo-super-nes-classic.dts new file mode 100644 index 000000000000..80761d7904ec --- /dev/null +++ b/arch/arm/boot/dts/sun8i-r16-nintendo-super-nes-classic.dts @@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-2.0 OR X11 +/* Copyright (c) 2018 Miquèl RAYNAL */ + +/dts-v1/; +#include "sun8i-r16-nintendo-nes-classic.dts" + +/ { + model = "Nintendo SuperNES Classic Edition"; + compatible = "nintendo,super-nes-classic", "nintendo,nes-classic", + "allwinner,sun8i-r16", "allwinner,sun8i-a33"; +}; diff --git a/arch/arm/boot/dts/sun8i-r40-bananapi-m2-ultra.dts b/arch/arm/boot/dts/sun8i-r40-bananapi-m2-ultra.dts index 27d9ccd0ef2f..25fb048c7df2 100644 --- a/arch/arm/boot/dts/sun8i-r40-bananapi-m2-ultra.dts +++ b/arch/arm/boot/dts/sun8i-r40-bananapi-m2-ultra.dts @@ -51,6 +51,7 @@ compatible = "sinovoip,bpi-m2-ultra", "allwinner,sun8i-r40"; aliases { + ethernet0 = &gmac; serial0 = &uart0; }; @@ -101,6 +102,22 @@ status = "okay"; }; +&gmac { + pinctrl-names = "default"; + pinctrl-0 = <&gmac_rgmii_pins>; + phy-handle = <&phy1>; + phy-mode = "rgmii"; + phy-supply = <®_dc1sw>; + status = "okay"; +}; + +&gmac_mdio { + phy1: ethernet-phy@1 { + compatible = "ethernet-phy-ieee802.3-c22"; + reg = <1>; + }; +}; + &i2c0 { status = "okay"; @@ -114,6 +131,48 @@ #include "axp22x.dtsi" +&mmc0 { + vmmc-supply = <®_dcdc1>; + bus-width = <4>; + cd-gpios = <&pio 7 13 GPIO_ACTIVE_HIGH>; /* PH13 */ + cd-inverted; + status = "okay"; +}; + +&mmc1 { + pinctrl-names = "default"; + pinctrl-0 = <&mmc1_pg_pins>; + vmmc-supply = <®_dldo2>; + vqmmc-supply = <®_dldo1>; + mmc-pwrseq = <&wifi_pwrseq>; + bus-width = <4>; + non-removable; + status = "okay"; +}; + +&mmc2 { + vmmc-supply = <®_dcdc1>; + vqmmc-supply = <®_dcdc1>; + bus-width = <8>; + non-removable; + status = "okay"; +}; + +&ohci1 { + status = "okay"; +}; + +&ohci2 { + status = "okay"; +}; + +®_aldo2 { + regulator-always-on; + regulator-min-microvolt = <2500000>; + regulator-max-microvolt = <2500000>; + regulator-name = "vcc-pa"; +}; + ®_aldo3 { regulator-always-on; regulator-min-microvolt = <2700000>; @@ -121,6 +180,12 @@ regulator-name = "avcc"; }; +®_dc1sw { + regulator-min-microvolt = <3000000>; + regulator-max-microvolt = <3000000>; + regulator-name = "vcc-gmac-phy"; +}; + ®_dcdc1 { regulator-always-on; regulator-min-microvolt = <3000000>; @@ -161,40 +226,6 @@ regulator-name = "vcc-wifi"; }; -&mmc0 { - vmmc-supply = <®_dcdc1>; - bus-width = <4>; - cd-gpios = <&pio 7 13 GPIO_ACTIVE_LOW>; /* PH13 */ - status = "okay"; -}; - -&mmc1 { - pinctrl-names = "default"; - pinctrl-0 = <&mmc1_pg_pins>; - vmmc-supply = <®_dldo2>; - vqmmc-supply = <®_dldo1>; - mmc-pwrseq = <&wifi_pwrseq>; - bus-width = <4>; - non-removable; - status = "okay"; -}; - -&mmc2 { - vmmc-supply = <®_dcdc1>; - vqmmc-supply = <®_dcdc1>; - bus-width = <8>; - non-removable; - status = "okay"; -}; - -&ohci1 { - status = "okay"; -}; - -&ohci2 { - status = "okay"; -}; - &uart0 { pinctrl-names = "default"; pinctrl-0 = <&uart0_pb_pins>; diff --git a/arch/arm/boot/dts/sun8i-r40.dtsi b/arch/arm/boot/dts/sun8i-r40.dtsi index 173dcc1652d2..bd97ca3dc2fa 100644 --- a/arch/arm/boot/dts/sun8i-r40.dtsi +++ b/arch/arm/boot/dts/sun8i-r40.dtsi @@ -265,6 +265,19 @@ #interrupt-cells = <3>; #gpio-cells = <3>; + gmac_rgmii_pins: gmac-rgmii-pins { + pins = "PA0", "PA1", "PA2", "PA3", + "PA4", "PA5", "PA6", "PA7", + "PA8", "PA10", "PA11", "PA12", + "PA13", "PA15", "PA16"; + function = "gmac"; + /* + * data lines in RGMII mode use DDR mode + * and need a higher signal drive strength + */ + drive-strength = <40>; + }; + i2c0_pins: i2c0-pins { pins = "PB0", "PB1"; function = "i2c0"; @@ -451,6 +464,27 @@ #size-cells = <0>; }; + gmac: ethernet@1c50000 { + compatible = "allwinner,sun8i-r40-gmac"; + syscon = <&ccu>; + reg = <0x01c50000 0x10000>; + interrupts = ; + interrupt-names = "macirq"; + resets = <&ccu RST_BUS_GMAC>; + reset-names = "stmmaceth"; + clocks = <&ccu CLK_BUS_GMAC>; + clock-names = "stmmaceth"; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + + gmac_mdio: mdio { + compatible = "snps,dwmac-mdio"; + #address-cells = <1>; + #size-cells = <0>; + }; + }; + gic: interrupt-controller@1c81000 { compatible = "arm,gic-400"; reg = <0x01c81000 0x1000>, diff --git a/arch/arm/boot/dts/sun8i-v40-bananapi-m2-berry.dts b/arch/arm/boot/dts/sun8i-v40-bananapi-m2-berry.dts index a26d72c3f9b5..35859d8f3267 100644 --- a/arch/arm/boot/dts/sun8i-v40-bananapi-m2-berry.dts +++ b/arch/arm/boot/dts/sun8i-v40-bananapi-m2-berry.dts @@ -87,6 +87,11 @@ }; }; +&ehci1 { + /* Terminus Tech FE 1.1s 4-port USB 2.0 hub here */ + status = "okay"; +}; + &i2c0 { status = "okay"; @@ -170,3 +175,8 @@ pinctrl-0 = <&uart0_pb_pins>; status = "okay"; }; + +&usbphy { + usb1_vbus-supply = <®_vcc5v0>; + status = "okay"; +}; diff --git a/arch/arm/boot/dts/sunxi-h3-h5.dtsi b/arch/arm/boot/dts/sunxi-h3-h5.dtsi index 1be1a02d6df2..c3bff1105e5d 100644 --- a/arch/arm/boot/dts/sunxi-h3-h5.dtsi +++ b/arch/arm/boot/dts/sunxi-h3-h5.dtsi @@ -822,6 +822,19 @@ status = "disabled"; }; + r_i2c: i2c@1f02400 { + compatible = "allwinner,sun6i-a31-i2c"; + reg = <0x01f02400 0x400>; + interrupts = ; + pinctrl-names = "default"; + pinctrl-0 = <&r_i2c_pins>; + clocks = <&r_ccu CLK_APB0_I2C>; + resets = <&r_ccu RST_APB0_I2C>; + status = "disabled"; + #address-cells = <1>; + #size-cells = <0>; + }; + r_pio: pinctrl@1f02c00 { compatible = "allwinner,sun8i-h3-r-pinctrl"; reg = <0x01f02c00 0x400>; @@ -837,6 +850,11 @@ pins = "PL11"; function = "s_cir_rx"; }; + + r_i2c_pins: r-i2c { + pins = "PL0", "PL1"; + function = "s_i2c"; + }; }; }; }; diff --git a/arch/arm/boot/dts/sunxi-libretech-all-h3-cc.dtsi b/arch/arm/boot/dts/sunxi-libretech-all-h3-cc.dtsi new file mode 100644 index 000000000000..f7ffdd6658a2 --- /dev/null +++ b/arch/arm/boot/dts/sunxi-libretech-all-h3-cc.dtsi @@ -0,0 +1,215 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR MIT) +/* + * Copyright (C) 2017 Chen-Yu Tsai + */ + +#include +#include + +/ { + aliases { + ethernet0 = &emac; + serial0 = &uart0; + }; + + chosen { + stdout-path = "serial0:115200n8"; + }; + + connector { + compatible = "hdmi-connector"; + type = "a"; + + port { + hdmi_con_in: endpoint { + remote-endpoint = <&hdmi_out_con>; + }; + }; + }; + + leds { + compatible = "gpio-leds"; + + pwr_led { + label = "librecomputer:green:pwr"; + gpios = <&r_pio 0 10 GPIO_ACTIVE_HIGH>; /* PL10 */ + default-state = "on"; + }; + + status_led { + label = "librecomputer:blue:status"; + gpios = <&pio 0 7 GPIO_ACTIVE_HIGH>; /* PA7 */ + }; + }; + + gpio_keys { + compatible = "gpio-keys"; + + power { + label = "power"; + linux,code = ; + gpios = <&r_pio 0 2 GPIO_ACTIVE_LOW>; /* PL2 */ + }; + }; + + reg_vcc1v2: vcc1v2 { + compatible = "regulator-fixed"; + regulator-name = "vcc1v2"; + regulator-min-microvolt = <1200000>; + regulator-max-microvolt = <1200000>; + regulator-always-on; + regulator-boot-on; + vin-supply = <®_vcc5v0>; + gpio = <&r_pio 0 8 GPIO_ACTIVE_HIGH>; /* PL8 */ + enable-active-high; + }; + + reg_vcc3v3: vcc3v3 { + compatible = "regulator-fixed"; + regulator-name = "vcc3v3"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + vin-supply = <®_vcc5v0>; + }; + + /* This represents the board's 5V input */ + reg_vcc5v0: vcc5v0 { + compatible = "regulator-fixed"; + regulator-name = "vcc5v0"; + regulator-min-microvolt = <5000000>; + regulator-max-microvolt = <5000000>; + }; + + reg_vcc_dram: vcc-dram { + compatible = "regulator-fixed"; + regulator-name = "vcc-dram"; + regulator-min-microvolt = <1500000>; + regulator-max-microvolt = <1500000>; + regulator-always-on; + regulator-boot-on; + vin-supply = <®_vcc5v0>; + gpio = <&r_pio 0 9 GPIO_ACTIVE_HIGH>; /* PL9 */ + enable-active-high; + }; + + reg_vcc_io: vcc-io { + compatible = "regulator-fixed"; + regulator-name = "vcc-io"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + regulator-boot-on; + vin-supply = <®_vcc3v3>; + gpio = <&r_pio 0 5 GPIO_ACTIVE_LOW>; /* PL5 */ + }; + + reg_vdd_cpux: vdd-cpux { + compatible = "regulator-fixed"; + regulator-name = "vdd-cpux"; + regulator-min-microvolt = <1200000>; + regulator-max-microvolt = <1200000>; + regulator-always-on; + regulator-boot-on; + vin-supply = <®_vcc5v0>; + gpio = <&r_pio 0 8 GPIO_ACTIVE_HIGH>; /* PL8 */ + enable-active-high; + }; +}; + +&codec { + allwinner,audio-routing = + "Line Out", "LINEOUT", + "MIC1", "Mic", + "Mic", "MBIAS"; + status = "okay"; +}; + +&cpu0 { + cpu-supply = <®_vdd_cpux>; +}; + +&de { + status = "okay"; +}; + +&ehci0 { + status = "okay"; +}; + +&ehci1 { + status = "okay"; +}; + +&ehci2 { + status = "okay"; +}; + +&ehci3 { + status = "okay"; +}; + +&emac { + phy-handle = <&int_mii_phy>; + phy-mode = "mii"; + allwinner,leds-active-low; + status = "okay"; +}; + +&hdmi { + status = "okay"; +}; + +&hdmi_out { + hdmi_out_con: endpoint { + remote-endpoint = <&hdmi_con_in>; + }; +}; + +&ir { + pinctrl-names = "default"; + pinctrl-0 = <&ir_pins_a>; + status = "okay"; +}; + +&mmc0 { + vmmc-supply = <®_vcc_io>; + bus-width = <4>; + cd-gpios = <&pio 5 6 GPIO_ACTIVE_LOW>; /* PF6 */ + status = "okay"; +}; + +&ohci0 { + status = "okay"; +}; + +&ohci1 { + status = "okay"; +}; + +&ohci2 { + status = "okay"; +}; + +&ohci3 { + status = "okay"; +}; + +&uart0 { + pinctrl-names = "default"; + pinctrl-0 = <&uart0_pins_a>; + status = "okay"; +}; + +&usb_otg { + dr_mode = "host"; + status = "okay"; +}; + +&usbphy { + /* VBUS on USB ports are always on */ + usb0_vbus-supply = <®_vcc5v0>; + usb1_vbus-supply = <®_vcc5v0>; + usb2_vbus-supply = <®_vcc5v0>; + usb3_vbus-supply = <®_vcc5v0>; + status = "okay"; +}; diff --git a/arch/arm/boot/dts/tegra114.dtsi b/arch/arm/boot/dts/tegra114.dtsi index 0e4a13295d8a..84c4358dacac 100644 --- a/arch/arm/boot/dts/tegra114.dtsi +++ b/arch/arm/boot/dts/tegra114.dtsi @@ -19,6 +19,7 @@ clocks = <&tegra_car TEGRA114_CLK_HOST1X>; resets = <&tegra_car 28>; reset-names = "host1x"; + iommus = <&mc TEGRA_SWGROUP_HC>; #address-cells = <1>; #size-cells = <1>; @@ -32,6 +33,8 @@ clocks = <&tegra_car TEGRA114_CLK_GR2D>; resets = <&tegra_car 21>; reset-names = "2d"; + + iommus = <&mc TEGRA_SWGROUP_G2>; }; gr3d@54180000 { @@ -40,6 +43,8 @@ clocks = <&tegra_car TEGRA114_CLK_GR3D>; resets = <&tegra_car 24>; reset-names = "3d"; + + iommus = <&mc TEGRA_SWGROUP_NV>; }; dc@54200000 { diff --git a/arch/arm/boot/dts/tegra124-apalis-v1.2.dtsi b/arch/arm/boot/dts/tegra124-apalis-v1.2.dtsi index bb67edb016c5..3455822350c5 100644 --- a/arch/arm/boot/dts/tegra124-apalis-v1.2.dtsi +++ b/arch/arm/boot/dts/tegra124-apalis-v1.2.dtsi @@ -1536,15 +1536,15 @@ }; serial@70006040 { - compatible = "nvidia,tegra124-hsuart"; + compatible = "nvidia,tegra124-hsuart", "nvidia,tegra30-hsuart"; }; serial@70006200 { - compatible = "nvidia,tegra124-hsuart"; + compatible = "nvidia,tegra124-hsuart", "nvidia,tegra30-hsuart"; }; serial@70006300 { - compatible = "nvidia,tegra124-hsuart"; + compatible = "nvidia,tegra124-hsuart", "nvidia,tegra30-hsuart"; }; hdmi_ddc: i2c@7000c700 { diff --git a/arch/arm/boot/dts/tegra124-apalis.dtsi b/arch/arm/boot/dts/tegra124-apalis.dtsi index 65a2161b9b8e..9f960c84ba10 100644 --- a/arch/arm/boot/dts/tegra124-apalis.dtsi +++ b/arch/arm/boot/dts/tegra124-apalis.dtsi @@ -1565,15 +1565,15 @@ }; serial@70006040 { - compatible = "nvidia,tegra124-hsuart"; + compatible = "nvidia,tegra124-hsuart", "nvidia,tegra30-hsuart"; }; serial@70006200 { - compatible = "nvidia,tegra124-hsuart"; + compatible = "nvidia,tegra124-hsuart", "nvidia,tegra30-hsuart"; }; serial@70006300 { - compatible = "nvidia,tegra124-hsuart"; + compatible = "nvidia,tegra124-hsuart", "nvidia,tegra30-hsuart"; }; hdmi_ddc: i2c@7000c400 { diff --git a/arch/arm/boot/dts/tegra30.dtsi b/arch/arm/boot/dts/tegra30.dtsi index a110cf84d85f..09087b9c5e26 100644 --- a/arch/arm/boot/dts/tegra30.dtsi +++ b/arch/arm/boot/dts/tegra30.dtsi @@ -112,6 +112,7 @@ clocks = <&tegra_car TEGRA30_CLK_HOST1X>; resets = <&tegra_car 28>; reset-names = "host1x"; + iommus = <&mc TEGRA_SWGROUP_HC>; #address-cells = <1>; #size-cells = <1>; @@ -125,6 +126,8 @@ clocks = <&tegra_car TEGRA30_CLK_MPE>; resets = <&tegra_car 60>; reset-names = "mpe"; + + iommus = <&mc TEGRA_SWGROUP_MPE>; }; vi@54080000 { @@ -134,6 +137,8 @@ clocks = <&tegra_car TEGRA30_CLK_VI>; resets = <&tegra_car 20>; reset-names = "vi"; + + iommus = <&mc TEGRA_SWGROUP_VI>; }; epp@540c0000 { @@ -143,6 +148,8 @@ clocks = <&tegra_car TEGRA30_CLK_EPP>; resets = <&tegra_car 19>; reset-names = "epp"; + + iommus = <&mc TEGRA_SWGROUP_EPP>; }; isp@54100000 { @@ -152,6 +159,8 @@ clocks = <&tegra_car TEGRA30_CLK_ISP>; resets = <&tegra_car 23>; reset-names = "isp"; + + iommus = <&mc TEGRA_SWGROUP_ISP>; }; gr2d@54140000 { @@ -161,6 +170,8 @@ clocks = <&tegra_car TEGRA30_CLK_GR2D>; resets = <&tegra_car 21>; reset-names = "2d"; + + iommus = <&mc TEGRA_SWGROUP_G2>; }; gr3d@54180000 { @@ -172,6 +183,9 @@ resets = <&tegra_car 24>, <&tegra_car 98>; reset-names = "3d", "3d2"; + + iommus = <&mc TEGRA_SWGROUP_NV>, + <&mc TEGRA_SWGROUP_NV2>; }; dc@54200000 { diff --git a/arch/arm/boot/dts/uniphier-pro4.dtsi b/arch/arm/boot/dts/uniphier-pro4.dtsi index 844124bc9c9c..49539f035219 100644 --- a/arch/arm/boot/dts/uniphier-pro4.dtsi +++ b/arch/arm/boot/dts/uniphier-pro4.dtsi @@ -286,7 +286,7 @@ has-transaction-translator; }; - soc-glue@5f800000 { + soc_glue: soc-glue@5f800000 { compatible = "socionext,uniphier-pro4-soc-glue", "simple-mfd", "syscon"; reg = <0x5f800000 0x2000>; @@ -371,10 +371,14 @@ interrupts = <0 66 4>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_ether_rgmii>; - clocks = <&sys_clk 6>; - resets = <&sys_rst 6>; + clock-names = "gio", "ether", "ether-gb", "ether-phy"; + clocks = <&sys_clk 12>, <&sys_clk 6>, <&sys_clk 7>, + <&sys_clk 10>; + reset-names = "gio", "ether"; + resets = <&sys_rst 12>, <&sys_rst 6>; phy-mode = "rgmii"; local-mac-address = [00 00 00 00 00 00]; + socionext,syscon-phy-mode = <&soc_glue 0>; mdio: mdio { #address-cells = <1>; diff --git a/arch/arm/boot/dts/uniphier-pxs2.dtsi b/arch/arm/boot/dts/uniphier-pxs2.dtsi index debcbd15c24b..641d96119d4f 100644 --- a/arch/arm/boot/dts/uniphier-pxs2.dtsi +++ b/arch/arm/boot/dts/uniphier-pxs2.dtsi @@ -506,10 +506,13 @@ interrupts = <0 66 4>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_ether_rgmii>; + clock-names = "ether"; clocks = <&sys_clk 6>; + reset-names = "ether"; resets = <&sys_rst 6>; phy-mode = "rgmii"; local-mac-address = [00 00 00 00 00 00]; + socionext,syscon-phy-mode = <&soc_glue 0>; mdio: mdio { #address-cells = <1>; diff --git a/arch/arm/boot/dts/vexpress-v2m-rs1.dtsi b/arch/arm/boot/dts/vexpress-v2m-rs1.dtsi index 35714ff6f467..4488c8fe213a 100644 --- a/arch/arm/boot/dts/vexpress-v2m-rs1.dtsi +++ b/arch/arm/boot/dts/vexpress-v2m-rs1.dtsi @@ -17,426 +17,436 @@ * CHANGES TO vexpress-v2m.dtsi! */ - motherboard { - model = "V2M-P1"; - arm,hbi = <0x190>; - arm,vexpress,site = <0>; - arm,v2m-memory-map = "rs1"; - compatible = "arm,vexpress,v2m-p1", "simple-bus"; - #address-cells = <2>; /* SMB chipselect number and offset */ - #size-cells = <1>; - #interrupt-cells = <1>; - ranges; - - flash@0,00000000 { - compatible = "arm,vexpress-flash", "cfi-flash"; - reg = <0 0x00000000 0x04000000>, - <4 0x00000000 0x04000000>; - bank-width = <4>; - }; +/ { + smb@8000000 { + motherboard { + model = "V2M-P1"; + arm,hbi = <0x190>; + arm,vexpress,site = <0>; + arm,v2m-memory-map = "rs1"; + compatible = "arm,vexpress,v2m-p1", "simple-bus"; + #address-cells = <2>; /* SMB chipselect number and offset */ + #size-cells = <1>; + #interrupt-cells = <1>; + ranges; - psram@1,00000000 { - compatible = "arm,vexpress-psram", "mtd-ram"; - reg = <1 0x00000000 0x02000000>; - bank-width = <4>; - }; + flash@0,00000000 { + compatible = "arm,vexpress-flash", "cfi-flash"; + reg = <0 0x00000000 0x04000000>, + <4 0x00000000 0x04000000>; + bank-width = <4>; + }; - v2m_video_ram: vram@2,00000000 { - compatible = "arm,vexpress-vram"; - reg = <2 0x00000000 0x00800000>; - }; + psram@1,00000000 { + compatible = "arm,vexpress-psram", "mtd-ram"; + reg = <1 0x00000000 0x02000000>; + bank-width = <4>; + }; - ethernet@2,02000000 { - compatible = "smsc,lan9118", "smsc,lan9115"; - reg = <2 0x02000000 0x10000>; - interrupts = <15>; - phy-mode = "mii"; - reg-io-width = <4>; - smsc,irq-active-high; - smsc,irq-push-pull; - vdd33a-supply = <&v2m_fixed_3v3>; - vddvario-supply = <&v2m_fixed_3v3>; - }; + v2m_video_ram: vram@2,00000000 { + compatible = "arm,vexpress-vram"; + reg = <2 0x00000000 0x00800000>; + }; - usb@2,03000000 { - compatible = "nxp,usb-isp1761"; - reg = <2 0x03000000 0x20000>; - interrupts = <16>; - port1-otg; - }; + ethernet@2,02000000 { + compatible = "smsc,lan9118", "smsc,lan9115"; + reg = <2 0x02000000 0x10000>; + interrupts = <15>; + phy-mode = "mii"; + reg-io-width = <4>; + smsc,irq-active-high; + smsc,irq-push-pull; + vdd33a-supply = <&v2m_fixed_3v3>; + vddvario-supply = <&v2m_fixed_3v3>; + }; - iofpga@3,00000000 { - compatible = "simple-bus"; - #address-cells = <1>; - #size-cells = <1>; - ranges = <0 3 0 0x200000>; + usb@2,03000000 { + compatible = "nxp,usb-isp1761"; + reg = <2 0x03000000 0x20000>; + interrupts = <16>; + port1-otg; + }; - v2m_sysreg: sysreg@10000 { - compatible = "arm,vexpress-sysreg"; - reg = <0x010000 0x1000>; + iofpga@3,00000000 { + compatible = "simple-bus"; + #address-cells = <1>; + #size-cells = <1>; + ranges = <0 3 0 0x200000>; + + v2m_sysreg: sysreg@10000 { + compatible = "arm,vexpress-sysreg"; + reg = <0x010000 0x1000>; + #address-cells = <1>; + #size-cells = <1>; + ranges = <0 0x10000 0x1000>; + + v2m_led_gpios: gpio@8 { + compatible = "arm,vexpress-sysreg,sys_led"; + reg = <0x008 4>; + gpio-controller; + #gpio-cells = <2>; + }; - v2m_led_gpios: sys_led { - compatible = "arm,vexpress-sysreg,sys_led"; - gpio-controller; - #gpio-cells = <2>; - }; + v2m_mmc_gpios: gpio@48 { + compatible = "arm,vexpress-sysreg,sys_mci"; + reg = <0x048 4>; + gpio-controller; + #gpio-cells = <2>; + }; - v2m_mmc_gpios: sys_mci { - compatible = "arm,vexpress-sysreg,sys_mci"; - gpio-controller; - #gpio-cells = <2>; + v2m_flash_gpios: gpio@4c { + compatible = "arm,vexpress-sysreg,sys_flash"; + reg = <0x04c 4>; + gpio-controller; + #gpio-cells = <2>; + }; }; - v2m_flash_gpios: sys_flash { - compatible = "arm,vexpress-sysreg,sys_flash"; - gpio-controller; - #gpio-cells = <2>; + v2m_sysctl: sysctl@20000 { + compatible = "arm,sp810", "arm,primecell"; + reg = <0x020000 0x1000>; + clocks = <&v2m_refclk32khz>, <&v2m_refclk1mhz>, <&smbclk>; + clock-names = "refclk", "timclk", "apb_pclk"; + #clock-cells = <1>; + clock-output-names = "timerclken0", "timerclken1", "timerclken2", "timerclken3"; + assigned-clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&v2m_sysctl 3>, <&v2m_sysctl 3>; + assigned-clock-parents = <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>; }; - }; - v2m_sysctl: sysctl@20000 { - compatible = "arm,sp810", "arm,primecell"; - reg = <0x020000 0x1000>; - clocks = <&v2m_refclk32khz>, <&v2m_refclk1mhz>, <&smbclk>; - clock-names = "refclk", "timclk", "apb_pclk"; - #clock-cells = <1>; - clock-output-names = "timerclken0", "timerclken1", "timerclken2", "timerclken3"; - assigned-clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&v2m_sysctl 3>, <&v2m_sysctl 3>; - assigned-clock-parents = <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>; - }; + /* PCI-E I2C bus */ + v2m_i2c_pcie: i2c@30000 { + compatible = "arm,versatile-i2c"; + reg = <0x030000 0x1000>; - /* PCI-E I2C bus */ - v2m_i2c_pcie: i2c@30000 { - compatible = "arm,versatile-i2c"; - reg = <0x030000 0x1000>; + #address-cells = <1>; + #size-cells = <0>; - #address-cells = <1>; - #size-cells = <0>; + pcie-switch@60 { + compatible = "idt,89hpes32h8"; + reg = <0x60>; + }; + }; - pcie-switch@60 { - compatible = "idt,89hpes32h8"; - reg = <0x60>; + aaci@40000 { + compatible = "arm,pl041", "arm,primecell"; + reg = <0x040000 0x1000>; + interrupts = <11>; + clocks = <&smbclk>; + clock-names = "apb_pclk"; }; - }; - aaci@40000 { - compatible = "arm,pl041", "arm,primecell"; - reg = <0x040000 0x1000>; - interrupts = <11>; - clocks = <&smbclk>; - clock-names = "apb_pclk"; - }; + mmci@50000 { + compatible = "arm,pl180", "arm,primecell"; + reg = <0x050000 0x1000>; + interrupts = <9 10>; + cd-gpios = <&v2m_mmc_gpios 0 0>; + wp-gpios = <&v2m_mmc_gpios 1 0>; + max-frequency = <12000000>; + vmmc-supply = <&v2m_fixed_3v3>; + clocks = <&v2m_clk24mhz>, <&smbclk>; + clock-names = "mclk", "apb_pclk"; + }; - mmci@50000 { - compatible = "arm,pl180", "arm,primecell"; - reg = <0x050000 0x1000>; - interrupts = <9 10>; - cd-gpios = <&v2m_mmc_gpios 0 0>; - wp-gpios = <&v2m_mmc_gpios 1 0>; - max-frequency = <12000000>; - vmmc-supply = <&v2m_fixed_3v3>; - clocks = <&v2m_clk24mhz>, <&smbclk>; - clock-names = "mclk", "apb_pclk"; - }; + kmi@60000 { + compatible = "arm,pl050", "arm,primecell"; + reg = <0x060000 0x1000>; + interrupts = <12>; + clocks = <&v2m_clk24mhz>, <&smbclk>; + clock-names = "KMIREFCLK", "apb_pclk"; + }; - kmi@60000 { - compatible = "arm,pl050", "arm,primecell"; - reg = <0x060000 0x1000>; - interrupts = <12>; - clocks = <&v2m_clk24mhz>, <&smbclk>; - clock-names = "KMIREFCLK", "apb_pclk"; - }; + kmi@70000 { + compatible = "arm,pl050", "arm,primecell"; + reg = <0x070000 0x1000>; + interrupts = <13>; + clocks = <&v2m_clk24mhz>, <&smbclk>; + clock-names = "KMIREFCLK", "apb_pclk"; + }; - kmi@70000 { - compatible = "arm,pl050", "arm,primecell"; - reg = <0x070000 0x1000>; - interrupts = <13>; - clocks = <&v2m_clk24mhz>, <&smbclk>; - clock-names = "KMIREFCLK", "apb_pclk"; - }; + v2m_serial0: uart@90000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x090000 0x1000>; + interrupts = <5>; + clocks = <&v2m_oscclk2>, <&smbclk>; + clock-names = "uartclk", "apb_pclk"; + }; - v2m_serial0: uart@90000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x090000 0x1000>; - interrupts = <5>; - clocks = <&v2m_oscclk2>, <&smbclk>; - clock-names = "uartclk", "apb_pclk"; - }; + v2m_serial1: uart@a0000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x0a0000 0x1000>; + interrupts = <6>; + clocks = <&v2m_oscclk2>, <&smbclk>; + clock-names = "uartclk", "apb_pclk"; + }; - v2m_serial1: uart@a0000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x0a0000 0x1000>; - interrupts = <6>; - clocks = <&v2m_oscclk2>, <&smbclk>; - clock-names = "uartclk", "apb_pclk"; - }; + v2m_serial2: uart@b0000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x0b0000 0x1000>; + interrupts = <7>; + clocks = <&v2m_oscclk2>, <&smbclk>; + clock-names = "uartclk", "apb_pclk"; + }; - v2m_serial2: uart@b0000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x0b0000 0x1000>; - interrupts = <7>; - clocks = <&v2m_oscclk2>, <&smbclk>; - clock-names = "uartclk", "apb_pclk"; - }; + v2m_serial3: uart@c0000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x0c0000 0x1000>; + interrupts = <8>; + clocks = <&v2m_oscclk2>, <&smbclk>; + clock-names = "uartclk", "apb_pclk"; + }; - v2m_serial3: uart@c0000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x0c0000 0x1000>; - interrupts = <8>; - clocks = <&v2m_oscclk2>, <&smbclk>; - clock-names = "uartclk", "apb_pclk"; - }; + wdt@f0000 { + compatible = "arm,sp805", "arm,primecell"; + reg = <0x0f0000 0x1000>; + interrupts = <0>; + clocks = <&v2m_refclk32khz>, <&smbclk>; + clock-names = "wdogclk", "apb_pclk"; + }; - wdt@f0000 { - compatible = "arm,sp805", "arm,primecell"; - reg = <0x0f0000 0x1000>; - interrupts = <0>; - clocks = <&v2m_refclk32khz>, <&smbclk>; - clock-names = "wdogclk", "apb_pclk"; - }; + v2m_timer01: timer@110000 { + compatible = "arm,sp804", "arm,primecell"; + reg = <0x110000 0x1000>; + interrupts = <2>; + clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&smbclk>; + clock-names = "timclken1", "timclken2", "apb_pclk"; + }; - v2m_timer01: timer@110000 { - compatible = "arm,sp804", "arm,primecell"; - reg = <0x110000 0x1000>; - interrupts = <2>; - clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&smbclk>; - clock-names = "timclken1", "timclken2", "apb_pclk"; - }; + v2m_timer23: timer@120000 { + compatible = "arm,sp804", "arm,primecell"; + reg = <0x120000 0x1000>; + interrupts = <3>; + clocks = <&v2m_sysctl 2>, <&v2m_sysctl 3>, <&smbclk>; + clock-names = "timclken1", "timclken2", "apb_pclk"; + }; - v2m_timer23: timer@120000 { - compatible = "arm,sp804", "arm,primecell"; - reg = <0x120000 0x1000>; - interrupts = <3>; - clocks = <&v2m_sysctl 2>, <&v2m_sysctl 3>, <&smbclk>; - clock-names = "timclken1", "timclken2", "apb_pclk"; - }; + /* DVI I2C bus */ + v2m_i2c_dvi: i2c@160000 { + compatible = "arm,versatile-i2c"; + reg = <0x160000 0x1000>; - /* DVI I2C bus */ - v2m_i2c_dvi: i2c@160000 { - compatible = "arm,versatile-i2c"; - reg = <0x160000 0x1000>; + #address-cells = <1>; + #size-cells = <0>; - #address-cells = <1>; - #size-cells = <0>; + dvi-transmitter@39 { + compatible = "sil,sii9022-tpi", "sil,sii9022"; + reg = <0x39>; + }; - dvi-transmitter@39 { - compatible = "sil,sii9022-tpi", "sil,sii9022"; - reg = <0x39>; + dvi-transmitter@60 { + compatible = "sil,sii9022-cpi", "sil,sii9022"; + reg = <0x60>; + }; }; - dvi-transmitter@60 { - compatible = "sil,sii9022-cpi", "sil,sii9022"; - reg = <0x60>; + rtc@170000 { + compatible = "arm,pl031", "arm,primecell"; + reg = <0x170000 0x1000>; + interrupts = <4>; + clocks = <&smbclk>; + clock-names = "apb_pclk"; }; - }; - rtc@170000 { - compatible = "arm,pl031", "arm,primecell"; - reg = <0x170000 0x1000>; - interrupts = <4>; - clocks = <&smbclk>; - clock-names = "apb_pclk"; - }; - - compact-flash@1a0000 { - compatible = "arm,vexpress-cf", "ata-generic"; - reg = <0x1a0000 0x100 - 0x1a0100 0xf00>; - reg-shift = <2>; - }; - - clcd@1f0000 { - compatible = "arm,pl111", "arm,primecell"; - reg = <0x1f0000 0x1000>; - interrupt-names = "combined"; - interrupts = <14>; - clocks = <&v2m_oscclk1>, <&smbclk>; - clock-names = "clcdclk", "apb_pclk"; - memory-region = <&v2m_video_ram>; - max-memory-bandwidth = <50350000>; /* 16bpp @ 25.175MHz */ - - port { - v2m_clcd_pads: endpoint { - remote-endpoint = <&v2m_clcd_panel>; - arm,pl11x,tft-r0g0b0-pads = <0 8 16>; - }; + compact-flash@1a0000 { + compatible = "arm,vexpress-cf", "ata-generic"; + reg = <0x1a0000 0x100 + 0x1a0100 0xf00>; + reg-shift = <2>; }; - panel { - compatible = "panel-dpi"; + clcd@1f0000 { + compatible = "arm,pl111", "arm,primecell"; + reg = <0x1f0000 0x1000>; + interrupt-names = "combined"; + interrupts = <14>; + clocks = <&v2m_oscclk1>, <&smbclk>; + clock-names = "clcdclk", "apb_pclk"; + memory-region = <&v2m_video_ram>; + max-memory-bandwidth = <50350000>; /* 16bpp @ 25.175MHz */ port { - v2m_clcd_panel: endpoint { - remote-endpoint = <&v2m_clcd_pads>; + v2m_clcd_pads: endpoint { + remote-endpoint = <&v2m_clcd_panel>; + arm,pl11x,tft-r0g0b0-pads = <0 8 16>; }; }; - panel-timing { - clock-frequency = <25175000>; - hactive = <640>; - hback-porch = <40>; - hfront-porch = <24>; - hsync-len = <96>; - vactive = <480>; - vback-porch = <32>; - vfront-porch = <11>; - vsync-len = <2>; + panel { + compatible = "panel-dpi"; + + port { + v2m_clcd_panel: endpoint { + remote-endpoint = <&v2m_clcd_pads>; + }; + }; + + panel-timing { + clock-frequency = <25175000>; + hactive = <640>; + hback-porch = <40>; + hfront-porch = <24>; + hsync-len = <96>; + vactive = <480>; + vback-porch = <32>; + vfront-porch = <11>; + vsync-len = <2>; + }; }; }; }; - }; - v2m_fixed_3v3: fixed-regulator-0 { - compatible = "regulator-fixed"; - regulator-name = "3V3"; - regulator-min-microvolt = <3300000>; - regulator-max-microvolt = <3300000>; - regulator-always-on; - }; + v2m_fixed_3v3: fixed-regulator-0 { + compatible = "regulator-fixed"; + regulator-name = "3V3"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; - v2m_clk24mhz: clk24mhz { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <24000000>; - clock-output-names = "v2m:clk24mhz"; - }; + v2m_clk24mhz: clk24mhz { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <24000000>; + clock-output-names = "v2m:clk24mhz"; + }; - v2m_refclk1mhz: refclk1mhz { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <1000000>; - clock-output-names = "v2m:refclk1mhz"; - }; + v2m_refclk1mhz: refclk1mhz { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <1000000>; + clock-output-names = "v2m:refclk1mhz"; + }; - v2m_refclk32khz: refclk32khz { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <32768>; - clock-output-names = "v2m:refclk32khz"; - }; + v2m_refclk32khz: refclk32khz { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <32768>; + clock-output-names = "v2m:refclk32khz"; + }; - leds { - compatible = "gpio-leds"; + leds { + compatible = "gpio-leds"; - user1 { - label = "v2m:green:user1"; - gpios = <&v2m_led_gpios 0 0>; - linux,default-trigger = "heartbeat"; - }; + user1 { + label = "v2m:green:user1"; + gpios = <&v2m_led_gpios 0 0>; + linux,default-trigger = "heartbeat"; + }; - user2 { - label = "v2m:green:user2"; - gpios = <&v2m_led_gpios 1 0>; - linux,default-trigger = "mmc0"; - }; + user2 { + label = "v2m:green:user2"; + gpios = <&v2m_led_gpios 1 0>; + linux,default-trigger = "mmc0"; + }; - user3 { - label = "v2m:green:user3"; - gpios = <&v2m_led_gpios 2 0>; - linux,default-trigger = "cpu0"; - }; + user3 { + label = "v2m:green:user3"; + gpios = <&v2m_led_gpios 2 0>; + linux,default-trigger = "cpu0"; + }; - user4 { - label = "v2m:green:user4"; - gpios = <&v2m_led_gpios 3 0>; - linux,default-trigger = "cpu1"; - }; + user4 { + label = "v2m:green:user4"; + gpios = <&v2m_led_gpios 3 0>; + linux,default-trigger = "cpu1"; + }; - user5 { - label = "v2m:green:user5"; - gpios = <&v2m_led_gpios 4 0>; - linux,default-trigger = "cpu2"; - }; + user5 { + label = "v2m:green:user5"; + gpios = <&v2m_led_gpios 4 0>; + linux,default-trigger = "cpu2"; + }; - user6 { - label = "v2m:green:user6"; - gpios = <&v2m_led_gpios 5 0>; - linux,default-trigger = "cpu3"; - }; + user6 { + label = "v2m:green:user6"; + gpios = <&v2m_led_gpios 5 0>; + linux,default-trigger = "cpu3"; + }; - user7 { - label = "v2m:green:user7"; - gpios = <&v2m_led_gpios 6 0>; - linux,default-trigger = "cpu4"; - }; + user7 { + label = "v2m:green:user7"; + gpios = <&v2m_led_gpios 6 0>; + linux,default-trigger = "cpu4"; + }; - user8 { - label = "v2m:green:user8"; - gpios = <&v2m_led_gpios 7 0>; - linux,default-trigger = "cpu5"; + user8 { + label = "v2m:green:user8"; + gpios = <&v2m_led_gpios 7 0>; + linux,default-trigger = "cpu5"; + }; }; - }; - mcc { - compatible = "arm,vexpress,config-bus"; - arm,vexpress,config-bridge = <&v2m_sysreg>; + mcc { + compatible = "arm,vexpress,config-bus"; + arm,vexpress,config-bridge = <&v2m_sysreg>; - oscclk0 { - /* MCC static memory clock */ - compatible = "arm,vexpress-osc"; - arm,vexpress-sysreg,func = <1 0>; - freq-range = <25000000 60000000>; - #clock-cells = <0>; - clock-output-names = "v2m:oscclk0"; - }; + oscclk0 { + /* MCC static memory clock */ + compatible = "arm,vexpress-osc"; + arm,vexpress-sysreg,func = <1 0>; + freq-range = <25000000 60000000>; + #clock-cells = <0>; + clock-output-names = "v2m:oscclk0"; + }; - v2m_oscclk1: oscclk1 { - /* CLCD clock */ - compatible = "arm,vexpress-osc"; - arm,vexpress-sysreg,func = <1 1>; - freq-range = <23750000 65000000>; - #clock-cells = <0>; - clock-output-names = "v2m:oscclk1"; - }; + v2m_oscclk1: oscclk1 { + /* CLCD clock */ + compatible = "arm,vexpress-osc"; + arm,vexpress-sysreg,func = <1 1>; + freq-range = <23750000 65000000>; + #clock-cells = <0>; + clock-output-names = "v2m:oscclk1"; + }; - v2m_oscclk2: oscclk2 { - /* IO FPGA peripheral clock */ - compatible = "arm,vexpress-osc"; - arm,vexpress-sysreg,func = <1 2>; - freq-range = <24000000 24000000>; - #clock-cells = <0>; - clock-output-names = "v2m:oscclk2"; - }; + v2m_oscclk2: oscclk2 { + /* IO FPGA peripheral clock */ + compatible = "arm,vexpress-osc"; + arm,vexpress-sysreg,func = <1 2>; + freq-range = <24000000 24000000>; + #clock-cells = <0>; + clock-output-names = "v2m:oscclk2"; + }; - volt-vio { - /* Logic level voltage */ - compatible = "arm,vexpress-volt"; - arm,vexpress-sysreg,func = <2 0>; - regulator-name = "VIO"; - regulator-always-on; - label = "VIO"; - }; + volt-vio { + /* Logic level voltage */ + compatible = "arm,vexpress-volt"; + arm,vexpress-sysreg,func = <2 0>; + regulator-name = "VIO"; + regulator-always-on; + label = "VIO"; + }; - temp-mcc { - /* MCC internal operating temperature */ - compatible = "arm,vexpress-temp"; - arm,vexpress-sysreg,func = <4 0>; - label = "MCC"; - }; + temp-mcc { + /* MCC internal operating temperature */ + compatible = "arm,vexpress-temp"; + arm,vexpress-sysreg,func = <4 0>; + label = "MCC"; + }; - reset { - compatible = "arm,vexpress-reset"; - arm,vexpress-sysreg,func = <5 0>; - }; + reset { + compatible = "arm,vexpress-reset"; + arm,vexpress-sysreg,func = <5 0>; + }; - muxfpga { - compatible = "arm,vexpress-muxfpga"; - arm,vexpress-sysreg,func = <7 0>; - }; + muxfpga { + compatible = "arm,vexpress-muxfpga"; + arm,vexpress-sysreg,func = <7 0>; + }; - shutdown { - compatible = "arm,vexpress-shutdown"; - arm,vexpress-sysreg,func = <8 0>; - }; + shutdown { + compatible = "arm,vexpress-shutdown"; + arm,vexpress-sysreg,func = <8 0>; + }; - reboot { - compatible = "arm,vexpress-reboot"; - arm,vexpress-sysreg,func = <9 0>; - }; + reboot { + compatible = "arm,vexpress-reboot"; + arm,vexpress-sysreg,func = <9 0>; + }; - dvimode { - compatible = "arm,vexpress-dvimode"; - arm,vexpress-sysreg,func = <11 0>; + dvimode { + compatible = "arm,vexpress-dvimode"; + arm,vexpress-sysreg,func = <11 0>; + }; }; }; }; +}; diff --git a/arch/arm/boot/dts/vexpress-v2m.dtsi b/arch/arm/boot/dts/vexpress-v2m.dtsi index b0021a816028..4db42f6326a3 100644 --- a/arch/arm/boot/dts/vexpress-v2m.dtsi +++ b/arch/arm/boot/dts/vexpress-v2m.dtsi @@ -18,425 +18,435 @@ * CHANGES TO vexpress-v2m-rs1.dtsi! */ - motherboard { - model = "V2M-P1"; - arm,hbi = <0x190>; - arm,vexpress,site = <0>; - compatible = "arm,vexpress,v2m-p1", "simple-bus"; - #address-cells = <2>; /* SMB chipselect number and offset */ - #size-cells = <1>; - #interrupt-cells = <1>; - ranges; - - flash@0,00000000 { - compatible = "arm,vexpress-flash", "cfi-flash"; - reg = <0 0x00000000 0x04000000>, - <1 0x00000000 0x04000000>; - bank-width = <4>; - }; +/ { + smb@4000000 { + motherboard { + model = "V2M-P1"; + arm,hbi = <0x190>; + arm,vexpress,site = <0>; + compatible = "arm,vexpress,v2m-p1", "simple-bus"; + #address-cells = <2>; /* SMB chipselect number and offset */ + #size-cells = <1>; + #interrupt-cells = <1>; + ranges; - psram@2,00000000 { - compatible = "arm,vexpress-psram", "mtd-ram"; - reg = <2 0x00000000 0x02000000>; - bank-width = <4>; - }; + flash@0,00000000 { + compatible = "arm,vexpress-flash", "cfi-flash"; + reg = <0 0x00000000 0x04000000>, + <1 0x00000000 0x04000000>; + bank-width = <4>; + }; - v2m_video_ram: vram@3,00000000 { - compatible = "arm,vexpress-vram"; - reg = <3 0x00000000 0x00800000>; - }; + psram@2,00000000 { + compatible = "arm,vexpress-psram", "mtd-ram"; + reg = <2 0x00000000 0x02000000>; + bank-width = <4>; + }; - ethernet@3,02000000 { - compatible = "smsc,lan9118", "smsc,lan9115"; - reg = <3 0x02000000 0x10000>; - interrupts = <15>; - phy-mode = "mii"; - reg-io-width = <4>; - smsc,irq-active-high; - smsc,irq-push-pull; - vdd33a-supply = <&v2m_fixed_3v3>; - vddvario-supply = <&v2m_fixed_3v3>; - }; + v2m_video_ram: vram@3,00000000 { + compatible = "arm,vexpress-vram"; + reg = <3 0x00000000 0x00800000>; + }; - usb@3,03000000 { - compatible = "nxp,usb-isp1761"; - reg = <3 0x03000000 0x20000>; - interrupts = <16>; - port1-otg; - }; + ethernet@3,02000000 { + compatible = "smsc,lan9118", "smsc,lan9115"; + reg = <3 0x02000000 0x10000>; + interrupts = <15>; + phy-mode = "mii"; + reg-io-width = <4>; + smsc,irq-active-high; + smsc,irq-push-pull; + vdd33a-supply = <&v2m_fixed_3v3>; + vddvario-supply = <&v2m_fixed_3v3>; + }; - iofpga@7,00000000 { - compatible = "simple-bus"; - #address-cells = <1>; - #size-cells = <1>; - ranges = <0 7 0 0x20000>; + usb@3,03000000 { + compatible = "nxp,usb-isp1761"; + reg = <3 0x03000000 0x20000>; + interrupts = <16>; + port1-otg; + }; - v2m_sysreg: sysreg@0 { - compatible = "arm,vexpress-sysreg"; - reg = <0x00000 0x1000>; + iofpga@7,00000000 { + compatible = "simple-bus"; + #address-cells = <1>; + #size-cells = <1>; + ranges = <0 7 0 0x20000>; + + v2m_sysreg: sysreg@0 { + compatible = "arm,vexpress-sysreg"; + reg = <0x00000 0x1000>; + #address-cells = <1>; + #size-cells = <1>; + ranges = <0 0 0x1000>; + + v2m_led_gpios: gpio@8 { + compatible = "arm,vexpress-sysreg,sys_led"; + reg = <0x008 4>; + gpio-controller; + #gpio-cells = <2>; + }; - v2m_led_gpios: sys_led { - compatible = "arm,vexpress-sysreg,sys_led"; - gpio-controller; - #gpio-cells = <2>; - }; + v2m_mmc_gpios: gpio@48 { + compatible = "arm,vexpress-sysreg,sys_mci"; + reg = <0x048 4>; + gpio-controller; + #gpio-cells = <2>; + }; - v2m_mmc_gpios: sys_mci { - compatible = "arm,vexpress-sysreg,sys_mci"; - gpio-controller; - #gpio-cells = <2>; + v2m_flash_gpios: gpio@4c { + compatible = "arm,vexpress-sysreg,sys_flash"; + reg = <0x04c 4>; + gpio-controller; + #gpio-cells = <2>; + }; }; - v2m_flash_gpios: sys_flash { - compatible = "arm,vexpress-sysreg,sys_flash"; - gpio-controller; - #gpio-cells = <2>; + v2m_sysctl: sysctl@1000 { + compatible = "arm,sp810", "arm,primecell"; + reg = <0x01000 0x1000>; + clocks = <&v2m_refclk32khz>, <&v2m_refclk1mhz>, <&smbclk>; + clock-names = "refclk", "timclk", "apb_pclk"; + #clock-cells = <1>; + clock-output-names = "timerclken0", "timerclken1", "timerclken2", "timerclken3"; + assigned-clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&v2m_sysctl 3>, <&v2m_sysctl 3>; + assigned-clock-parents = <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>; }; - }; - v2m_sysctl: sysctl@1000 { - compatible = "arm,sp810", "arm,primecell"; - reg = <0x01000 0x1000>; - clocks = <&v2m_refclk32khz>, <&v2m_refclk1mhz>, <&smbclk>; - clock-names = "refclk", "timclk", "apb_pclk"; - #clock-cells = <1>; - clock-output-names = "timerclken0", "timerclken1", "timerclken2", "timerclken3"; - assigned-clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&v2m_sysctl 3>, <&v2m_sysctl 3>; - assigned-clock-parents = <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>; - }; + /* PCI-E I2C bus */ + v2m_i2c_pcie: i2c@2000 { + compatible = "arm,versatile-i2c"; + reg = <0x02000 0x1000>; - /* PCI-E I2C bus */ - v2m_i2c_pcie: i2c@2000 { - compatible = "arm,versatile-i2c"; - reg = <0x02000 0x1000>; + #address-cells = <1>; + #size-cells = <0>; - #address-cells = <1>; - #size-cells = <0>; + pcie-switch@60 { + compatible = "idt,89hpes32h8"; + reg = <0x60>; + }; + }; - pcie-switch@60 { - compatible = "idt,89hpes32h8"; - reg = <0x60>; + aaci@4000 { + compatible = "arm,pl041", "arm,primecell"; + reg = <0x04000 0x1000>; + interrupts = <11>; + clocks = <&smbclk>; + clock-names = "apb_pclk"; }; - }; - aaci@4000 { - compatible = "arm,pl041", "arm,primecell"; - reg = <0x04000 0x1000>; - interrupts = <11>; - clocks = <&smbclk>; - clock-names = "apb_pclk"; - }; + mmci@5000 { + compatible = "arm,pl180", "arm,primecell"; + reg = <0x05000 0x1000>; + interrupts = <9 10>; + cd-gpios = <&v2m_mmc_gpios 0 0>; + wp-gpios = <&v2m_mmc_gpios 1 0>; + max-frequency = <12000000>; + vmmc-supply = <&v2m_fixed_3v3>; + clocks = <&v2m_clk24mhz>, <&smbclk>; + clock-names = "mclk", "apb_pclk"; + }; - mmci@5000 { - compatible = "arm,pl180", "arm,primecell"; - reg = <0x05000 0x1000>; - interrupts = <9 10>; - cd-gpios = <&v2m_mmc_gpios 0 0>; - wp-gpios = <&v2m_mmc_gpios 1 0>; - max-frequency = <12000000>; - vmmc-supply = <&v2m_fixed_3v3>; - clocks = <&v2m_clk24mhz>, <&smbclk>; - clock-names = "mclk", "apb_pclk"; - }; + kmi@6000 { + compatible = "arm,pl050", "arm,primecell"; + reg = <0x06000 0x1000>; + interrupts = <12>; + clocks = <&v2m_clk24mhz>, <&smbclk>; + clock-names = "KMIREFCLK", "apb_pclk"; + }; - kmi@6000 { - compatible = "arm,pl050", "arm,primecell"; - reg = <0x06000 0x1000>; - interrupts = <12>; - clocks = <&v2m_clk24mhz>, <&smbclk>; - clock-names = "KMIREFCLK", "apb_pclk"; - }; + kmi@7000 { + compatible = "arm,pl050", "arm,primecell"; + reg = <0x07000 0x1000>; + interrupts = <13>; + clocks = <&v2m_clk24mhz>, <&smbclk>; + clock-names = "KMIREFCLK", "apb_pclk"; + }; - kmi@7000 { - compatible = "arm,pl050", "arm,primecell"; - reg = <0x07000 0x1000>; - interrupts = <13>; - clocks = <&v2m_clk24mhz>, <&smbclk>; - clock-names = "KMIREFCLK", "apb_pclk"; - }; + v2m_serial0: uart@9000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x09000 0x1000>; + interrupts = <5>; + clocks = <&v2m_oscclk2>, <&smbclk>; + clock-names = "uartclk", "apb_pclk"; + }; - v2m_serial0: uart@9000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x09000 0x1000>; - interrupts = <5>; - clocks = <&v2m_oscclk2>, <&smbclk>; - clock-names = "uartclk", "apb_pclk"; - }; + v2m_serial1: uart@a000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x0a000 0x1000>; + interrupts = <6>; + clocks = <&v2m_oscclk2>, <&smbclk>; + clock-names = "uartclk", "apb_pclk"; + }; - v2m_serial1: uart@a000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x0a000 0x1000>; - interrupts = <6>; - clocks = <&v2m_oscclk2>, <&smbclk>; - clock-names = "uartclk", "apb_pclk"; - }; + v2m_serial2: uart@b000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x0b000 0x1000>; + interrupts = <7>; + clocks = <&v2m_oscclk2>, <&smbclk>; + clock-names = "uartclk", "apb_pclk"; + }; - v2m_serial2: uart@b000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x0b000 0x1000>; - interrupts = <7>; - clocks = <&v2m_oscclk2>, <&smbclk>; - clock-names = "uartclk", "apb_pclk"; - }; + v2m_serial3: uart@c000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x0c000 0x1000>; + interrupts = <8>; + clocks = <&v2m_oscclk2>, <&smbclk>; + clock-names = "uartclk", "apb_pclk"; + }; - v2m_serial3: uart@c000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x0c000 0x1000>; - interrupts = <8>; - clocks = <&v2m_oscclk2>, <&smbclk>; - clock-names = "uartclk", "apb_pclk"; - }; + wdt@f000 { + compatible = "arm,sp805", "arm,primecell"; + reg = <0x0f000 0x1000>; + interrupts = <0>; + clocks = <&v2m_refclk32khz>, <&smbclk>; + clock-names = "wdogclk", "apb_pclk"; + }; - wdt@f000 { - compatible = "arm,sp805", "arm,primecell"; - reg = <0x0f000 0x1000>; - interrupts = <0>; - clocks = <&v2m_refclk32khz>, <&smbclk>; - clock-names = "wdogclk", "apb_pclk"; - }; + v2m_timer01: timer@11000 { + compatible = "arm,sp804", "arm,primecell"; + reg = <0x11000 0x1000>; + interrupts = <2>; + clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&smbclk>; + clock-names = "timclken1", "timclken2", "apb_pclk"; + }; - v2m_timer01: timer@11000 { - compatible = "arm,sp804", "arm,primecell"; - reg = <0x11000 0x1000>; - interrupts = <2>; - clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&smbclk>; - clock-names = "timclken1", "timclken2", "apb_pclk"; - }; + v2m_timer23: timer@12000 { + compatible = "arm,sp804", "arm,primecell"; + reg = <0x12000 0x1000>; + interrupts = <3>; + clocks = <&v2m_sysctl 2>, <&v2m_sysctl 3>, <&smbclk>; + clock-names = "timclken1", "timclken2", "apb_pclk"; + }; - v2m_timer23: timer@12000 { - compatible = "arm,sp804", "arm,primecell"; - reg = <0x12000 0x1000>; - interrupts = <3>; - clocks = <&v2m_sysctl 2>, <&v2m_sysctl 3>, <&smbclk>; - clock-names = "timclken1", "timclken2", "apb_pclk"; - }; + /* DVI I2C bus */ + v2m_i2c_dvi: i2c@16000 { + compatible = "arm,versatile-i2c"; + reg = <0x16000 0x1000>; - /* DVI I2C bus */ - v2m_i2c_dvi: i2c@16000 { - compatible = "arm,versatile-i2c"; - reg = <0x16000 0x1000>; + #address-cells = <1>; + #size-cells = <0>; - #address-cells = <1>; - #size-cells = <0>; + dvi-transmitter@39 { + compatible = "sil,sii9022-tpi", "sil,sii9022"; + reg = <0x39>; + }; - dvi-transmitter@39 { - compatible = "sil,sii9022-tpi", "sil,sii9022"; - reg = <0x39>; + dvi-transmitter@60 { + compatible = "sil,sii9022-cpi", "sil,sii9022"; + reg = <0x60>; + }; }; - dvi-transmitter@60 { - compatible = "sil,sii9022-cpi", "sil,sii9022"; - reg = <0x60>; + rtc@17000 { + compatible = "arm,pl031", "arm,primecell"; + reg = <0x17000 0x1000>; + interrupts = <4>; + clocks = <&smbclk>; + clock-names = "apb_pclk"; }; - }; - rtc@17000 { - compatible = "arm,pl031", "arm,primecell"; - reg = <0x17000 0x1000>; - interrupts = <4>; - clocks = <&smbclk>; - clock-names = "apb_pclk"; - }; - - compact-flash@1a000 { - compatible = "arm,vexpress-cf", "ata-generic"; - reg = <0x1a000 0x100 - 0x1a100 0xf00>; - reg-shift = <2>; - }; - - clcd@1f000 { - compatible = "arm,pl111", "arm,primecell"; - reg = <0x1f000 0x1000>; - interrupt-names = "combined"; - interrupts = <14>; - clocks = <&v2m_oscclk1>, <&smbclk>; - clock-names = "clcdclk", "apb_pclk"; - memory-region = <&v2m_video_ram>; - max-memory-bandwidth = <50350000>; /* 16bpp @ 25.175MHz */ - - port { - v2m_clcd_pads: endpoint { - remote-endpoint = <&v2m_clcd_panel>; - arm,pl11x,tft-r0g0b0-pads = <0 8 16>; - }; + compact-flash@1a000 { + compatible = "arm,vexpress-cf", "ata-generic"; + reg = <0x1a000 0x100 + 0x1a100 0xf00>; + reg-shift = <2>; }; - panel { - compatible = "panel-dpi"; + clcd@1f000 { + compatible = "arm,pl111", "arm,primecell"; + reg = <0x1f000 0x1000>; + interrupt-names = "combined"; + interrupts = <14>; + clocks = <&v2m_oscclk1>, <&smbclk>; + clock-names = "clcdclk", "apb_pclk"; + memory-region = <&v2m_video_ram>; + max-memory-bandwidth = <50350000>; /* 16bpp @ 25.175MHz */ port { - v2m_clcd_panel: endpoint { - remote-endpoint = <&v2m_clcd_pads>; + v2m_clcd_pads: endpoint { + remote-endpoint = <&v2m_clcd_panel>; + arm,pl11x,tft-r0g0b0-pads = <0 8 16>; }; }; - panel-timing { - clock-frequency = <25175000>; - hactive = <640>; - hback-porch = <40>; - hfront-porch = <24>; - hsync-len = <96>; - vactive = <480>; - vback-porch = <32>; - vfront-porch = <11>; - vsync-len = <2>; + panel { + compatible = "panel-dpi"; + + port { + v2m_clcd_panel: endpoint { + remote-endpoint = <&v2m_clcd_pads>; + }; + }; + + panel-timing { + clock-frequency = <25175000>; + hactive = <640>; + hback-porch = <40>; + hfront-porch = <24>; + hsync-len = <96>; + vactive = <480>; + vback-porch = <32>; + vfront-porch = <11>; + vsync-len = <2>; + }; }; }; }; - }; - v2m_fixed_3v3: fixed-regulator-0 { - compatible = "regulator-fixed"; - regulator-name = "3V3"; - regulator-min-microvolt = <3300000>; - regulator-max-microvolt = <3300000>; - regulator-always-on; - }; + v2m_fixed_3v3: fixed-regulator-0 { + compatible = "regulator-fixed"; + regulator-name = "3V3"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; + }; - v2m_clk24mhz: clk24mhz { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <24000000>; - clock-output-names = "v2m:clk24mhz"; - }; + v2m_clk24mhz: clk24mhz { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <24000000>; + clock-output-names = "v2m:clk24mhz"; + }; - v2m_refclk1mhz: refclk1mhz { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <1000000>; - clock-output-names = "v2m:refclk1mhz"; - }; + v2m_refclk1mhz: refclk1mhz { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <1000000>; + clock-output-names = "v2m:refclk1mhz"; + }; - v2m_refclk32khz: refclk32khz { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <32768>; - clock-output-names = "v2m:refclk32khz"; - }; + v2m_refclk32khz: refclk32khz { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <32768>; + clock-output-names = "v2m:refclk32khz"; + }; - leds { - compatible = "gpio-leds"; + leds { + compatible = "gpio-leds"; - user1 { - label = "v2m:green:user1"; - gpios = <&v2m_led_gpios 0 0>; - linux,default-trigger = "heartbeat"; - }; + user1 { + label = "v2m:green:user1"; + gpios = <&v2m_led_gpios 0 0>; + linux,default-trigger = "heartbeat"; + }; - user2 { - label = "v2m:green:user2"; - gpios = <&v2m_led_gpios 1 0>; - linux,default-trigger = "mmc0"; - }; + user2 { + label = "v2m:green:user2"; + gpios = <&v2m_led_gpios 1 0>; + linux,default-trigger = "mmc0"; + }; - user3 { - label = "v2m:green:user3"; - gpios = <&v2m_led_gpios 2 0>; - linux,default-trigger = "cpu0"; - }; + user3 { + label = "v2m:green:user3"; + gpios = <&v2m_led_gpios 2 0>; + linux,default-trigger = "cpu0"; + }; - user4 { - label = "v2m:green:user4"; - gpios = <&v2m_led_gpios 3 0>; - linux,default-trigger = "cpu1"; - }; + user4 { + label = "v2m:green:user4"; + gpios = <&v2m_led_gpios 3 0>; + linux,default-trigger = "cpu1"; + }; - user5 { - label = "v2m:green:user5"; - gpios = <&v2m_led_gpios 4 0>; - linux,default-trigger = "cpu2"; - }; + user5 { + label = "v2m:green:user5"; + gpios = <&v2m_led_gpios 4 0>; + linux,default-trigger = "cpu2"; + }; - user6 { - label = "v2m:green:user6"; - gpios = <&v2m_led_gpios 5 0>; - linux,default-trigger = "cpu3"; - }; + user6 { + label = "v2m:green:user6"; + gpios = <&v2m_led_gpios 5 0>; + linux,default-trigger = "cpu3"; + }; - user7 { - label = "v2m:green:user7"; - gpios = <&v2m_led_gpios 6 0>; - linux,default-trigger = "cpu4"; - }; + user7 { + label = "v2m:green:user7"; + gpios = <&v2m_led_gpios 6 0>; + linux,default-trigger = "cpu4"; + }; - user8 { - label = "v2m:green:user8"; - gpios = <&v2m_led_gpios 7 0>; - linux,default-trigger = "cpu5"; + user8 { + label = "v2m:green:user8"; + gpios = <&v2m_led_gpios 7 0>; + linux,default-trigger = "cpu5"; + }; }; - }; - mcc { - compatible = "arm,vexpress,config-bus"; - arm,vexpress,config-bridge = <&v2m_sysreg>; + mcc { + compatible = "arm,vexpress,config-bus"; + arm,vexpress,config-bridge = <&v2m_sysreg>; - oscclk0 { - /* MCC static memory clock */ - compatible = "arm,vexpress-osc"; - arm,vexpress-sysreg,func = <1 0>; - freq-range = <25000000 60000000>; - #clock-cells = <0>; - clock-output-names = "v2m:oscclk0"; - }; + oscclk0 { + /* MCC static memory clock */ + compatible = "arm,vexpress-osc"; + arm,vexpress-sysreg,func = <1 0>; + freq-range = <25000000 60000000>; + #clock-cells = <0>; + clock-output-names = "v2m:oscclk0"; + }; - v2m_oscclk1: oscclk1 { - /* CLCD clock */ - compatible = "arm,vexpress-osc"; - arm,vexpress-sysreg,func = <1 1>; - freq-range = <23750000 65000000>; - #clock-cells = <0>; - clock-output-names = "v2m:oscclk1"; - }; + v2m_oscclk1: oscclk1 { + /* CLCD clock */ + compatible = "arm,vexpress-osc"; + arm,vexpress-sysreg,func = <1 1>; + freq-range = <23750000 65000000>; + #clock-cells = <0>; + clock-output-names = "v2m:oscclk1"; + }; - v2m_oscclk2: oscclk2 { - /* IO FPGA peripheral clock */ - compatible = "arm,vexpress-osc"; - arm,vexpress-sysreg,func = <1 2>; - freq-range = <24000000 24000000>; - #clock-cells = <0>; - clock-output-names = "v2m:oscclk2"; - }; + v2m_oscclk2: oscclk2 { + /* IO FPGA peripheral clock */ + compatible = "arm,vexpress-osc"; + arm,vexpress-sysreg,func = <1 2>; + freq-range = <24000000 24000000>; + #clock-cells = <0>; + clock-output-names = "v2m:oscclk2"; + }; - volt-vio { - /* Logic level voltage */ - compatible = "arm,vexpress-volt"; - arm,vexpress-sysreg,func = <2 0>; - regulator-name = "VIO"; - regulator-always-on; - label = "VIO"; - }; + volt-vio { + /* Logic level voltage */ + compatible = "arm,vexpress-volt"; + arm,vexpress-sysreg,func = <2 0>; + regulator-name = "VIO"; + regulator-always-on; + label = "VIO"; + }; - temp-mcc { - /* MCC internal operating temperature */ - compatible = "arm,vexpress-temp"; - arm,vexpress-sysreg,func = <4 0>; - label = "MCC"; - }; + temp-mcc { + /* MCC internal operating temperature */ + compatible = "arm,vexpress-temp"; + arm,vexpress-sysreg,func = <4 0>; + label = "MCC"; + }; - reset { - compatible = "arm,vexpress-reset"; - arm,vexpress-sysreg,func = <5 0>; - }; + reset { + compatible = "arm,vexpress-reset"; + arm,vexpress-sysreg,func = <5 0>; + }; - muxfpga { - compatible = "arm,vexpress-muxfpga"; - arm,vexpress-sysreg,func = <7 0>; - }; + muxfpga { + compatible = "arm,vexpress-muxfpga"; + arm,vexpress-sysreg,func = <7 0>; + }; - shutdown { - compatible = "arm,vexpress-shutdown"; - arm,vexpress-sysreg,func = <8 0>; - }; + shutdown { + compatible = "arm,vexpress-shutdown"; + arm,vexpress-sysreg,func = <8 0>; + }; - reboot { - compatible = "arm,vexpress-reboot"; - arm,vexpress-sysreg,func = <9 0>; - }; + reboot { + compatible = "arm,vexpress-reboot"; + arm,vexpress-sysreg,func = <9 0>; + }; - dvimode { - compatible = "arm,vexpress-dvimode"; - arm,vexpress-sysreg,func = <11 0>; + dvimode { + compatible = "arm,vexpress-dvimode"; + arm,vexpress-sysreg,func = <11 0>; + }; }; }; }; +}; \ No newline at end of file diff --git a/arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dts b/arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dts index a8ac4e2ed290..3971427a105b 100644 --- a/arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dts +++ b/arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dts @@ -9,6 +9,7 @@ */ /dts-v1/; +#include "vexpress-v2m-rs1.dtsi" / { model = "V2P-CA15"; @@ -278,8 +279,6 @@ <0 0 40 &gic 0 40 4>, <0 0 41 &gic 0 41 4>, <0 0 42 &gic 0 42 4>; - - /include/ "vexpress-v2m-rs1.dtsi" }; site2: hsb@40000000 { diff --git a/arch/arm/boot/dts/vexpress-v2p-ca15_a7.dts b/arch/arm/boot/dts/vexpress-v2p-ca15_a7.dts index a4c7713edfcd..ac6b90e9d806 100644 --- a/arch/arm/boot/dts/vexpress-v2p-ca15_a7.dts +++ b/arch/arm/boot/dts/vexpress-v2p-ca15_a7.dts @@ -9,6 +9,7 @@ */ /dts-v1/; +#include "vexpress-v2m-rs1.dtsi" / { model = "V2P-CA15_CA7"; @@ -203,7 +204,7 @@ <1 10 0xf08>; }; - pmu_a15 { + pmu-a15 { compatible = "arm,cortex-a15-pmu"; interrupts = <0 68 4>, <0 69 4>; @@ -211,7 +212,7 @@ <&cpu1>; }; - pmu_a7 { + pmu-a7 { compatible = "arm,cortex-a7-pmu"; interrupts = <0 128 4>, <0 129 4>, @@ -584,7 +585,7 @@ }; }; - smb@8000000 { + smb: smb@8000000 { compatible = "simple-bus"; #address-cells = <2>; @@ -641,8 +642,6 @@ <0 0 40 &gic 0 40 4>, <0 0 41 &gic 0 41 4>, <0 0 42 &gic 0 42 4>; - - /include/ "vexpress-v2m-rs1.dtsi" }; site2: hsb@40000000 { diff --git a/arch/arm/boot/dts/vexpress-v2p-ca5s.dts b/arch/arm/boot/dts/vexpress-v2p-ca5s.dts index 32f1906ffecf..e5b4a7570a01 100644 --- a/arch/arm/boot/dts/vexpress-v2p-ca5s.dts +++ b/arch/arm/boot/dts/vexpress-v2p-ca5s.dts @@ -9,6 +9,7 @@ */ /dts-v1/; +#include "vexpress-v2m-rs1.dtsi" / { model = "V2P-CA5s"; @@ -191,7 +192,7 @@ }; }; - smb@8000000 { + smb: smb@8000000 { compatible = "simple-bus"; #address-cells = <2>; @@ -248,8 +249,6 @@ <0 0 40 &gic 0 40 4>, <0 0 41 &gic 0 41 4>, <0 0 42 &gic 0 42 4>; - - /include/ "vexpress-v2m-rs1.dtsi" }; site2: hsb@40000000 { diff --git a/arch/arm/boot/dts/vexpress-v2p-ca9.dts b/arch/arm/boot/dts/vexpress-v2p-ca9.dts index 5814460e0549..fc43873cbdff 100644 --- a/arch/arm/boot/dts/vexpress-v2p-ca9.dts +++ b/arch/arm/boot/dts/vexpress-v2p-ca9.dts @@ -9,6 +9,7 @@ */ /dts-v1/; +#include "vexpress-v2m.dtsi" / { model = "V2P-CA9"; @@ -301,7 +302,7 @@ }; }; - smb@4000000 { + smb: smb@4000000 { compatible = "simple-bus"; #address-cells = <2>; @@ -357,8 +358,6 @@ <0 0 40 &gic 0 40 4>, <0 0 41 &gic 0 41 4>, <0 0 42 &gic 0 42 4>; - - /include/ "vexpress-v2m.dtsi" }; site2: hsb@e0000000 { diff --git a/arch/arm/boot/dts/vf-colibri-eval-v3.dtsi b/arch/arm/boot/dts/vf-colibri-eval-v3.dtsi index d8b2972527eb..e2da122a63f4 100644 --- a/arch/arm/boot/dts/vf-colibri-eval-v3.dtsi +++ b/arch/arm/boot/dts/vf-colibri-eval-v3.dtsi @@ -117,7 +117,7 @@ clocks = <&clk16m>; spi-max-frequency = <10000000>; interrupt-parent = <&gpio1>; - interrupts = <11 GPIO_ACTIVE_LOW>; + interrupts = <11 IRQ_TYPE_EDGE_RISING>; }; }; diff --git a/arch/arm/boot/dts/vf610-zii-dev-rev-b.dts b/arch/arm/boot/dts/vf610-zii-dev-rev-b.dts index 782b69a3acdf..bd79e00bf615 100644 --- a/arch/arm/boot/dts/vf610-zii-dev-rev-b.dts +++ b/arch/arm/boot/dts/vf610-zii-dev-rev-b.dts @@ -70,8 +70,6 @@ compatible = "marvell,mv88e6085"; pinctrl-0 = <&pinctrl_gpio_switch0>; pinctrl-names = "default"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; dsa,member = <0 0>; interrupt-parent = <&gpio0>; @@ -156,8 +154,6 @@ compatible = "marvell,mv88e6085"; pinctrl-0 = <&pinctrl_gpio_switch1>; pinctrl-names = "default"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; dsa,member = <0 1>; interrupt-parent = <&gpio0>; @@ -243,8 +239,6 @@ switch2: switch@0 { compatible = "marvell,mv88e6085"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; dsa,member = <0 2>; diff --git a/arch/arm/boot/dts/vf610-zii-dev-rev-c.dts b/arch/arm/boot/dts/vf610-zii-dev-rev-c.dts index c6f134c78303..0b1e94c6f25b 100644 --- a/arch/arm/boot/dts/vf610-zii-dev-rev-c.dts +++ b/arch/arm/boot/dts/vf610-zii-dev-rev-c.dts @@ -69,8 +69,6 @@ compatible = "marvell,mv88e6190"; pinctrl-0 = <&pinctrl_gpio_switch0>; pinctrl-names = "default"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; dsa,member = <0 0>; eeprom-length = <65536>; @@ -166,8 +164,6 @@ compatible = "marvell,mv88e6190"; pinctrl-0 = <&pinctrl_gpio_switch1>; pinctrl-names = "default"; - #address-cells = <1>; - #size-cells = <0>; reg = <0>; dsa,member = <0 1>; eeprom-length = <65536>; diff --git a/arch/arm/boot/dts/vf610-zii-dev.dtsi b/arch/arm/boot/dts/vf610-zii-dev.dtsi index 4890b8a5aa44..5ae5abfe1d55 100644 --- a/arch/arm/boot/dts/vf610-zii-dev.dtsi +++ b/arch/arm/boot/dts/vf610-zii-dev.dtsi @@ -222,6 +222,10 @@ status = "okay"; }; +&tempsensor { + io-channels = <&adc0 16>; +}; + &iomuxc { pinctrl_adc0_ad5: adc0ad5grp { fsl,pins = < diff --git a/arch/arm/boot/dts/vfxxx.dtsi b/arch/arm/boot/dts/vfxxx.dtsi index c3f09b737924..d392794d9c13 100644 --- a/arch/arm/boot/dts/vfxxx.dtsi +++ b/arch/arm/boot/dts/vfxxx.dtsi @@ -84,7 +84,7 @@ mask = <0x1000>; }; - iio-hwmon { + tempsensor: iio-hwmon { compatible = "iio-hwmon"; io-channels = <&adc0 16>, <&adc1 16>; }; diff --git a/arch/arm/common/Makefile b/arch/arm/common/Makefile index 70b4a14ed993..1e9f7af8f70f 100644 --- a/arch/arm/common/Makefile +++ b/arch/arm/common/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_DMABOUNCE) += dmabounce.o obj-$(CONFIG_SHARP_LOCOMO) += locomo.o obj-$(CONFIG_SHARP_PARAM) += sharpsl_param.o obj-$(CONFIG_SHARP_SCOOP) += scoop.o +obj-$(CONFIG_SMP) += secure_cntvoff.o obj-$(CONFIG_PCI_HOST_ITE8152) += it8152.o obj-$(CONFIG_MCPM) += mcpm_head.o mcpm_entry.o mcpm_platsmp.o vlock.o CFLAGS_REMOVE_mcpm_entry.o = -pg diff --git a/arch/arm/common/mcpm_entry.c b/arch/arm/common/mcpm_entry.c index 2b913f17d50f..ad574d20415c 100644 --- a/arch/arm/common/mcpm_entry.c +++ b/arch/arm/common/mcpm_entry.c @@ -9,6 +9,7 @@ * published by the Free Software Foundation. */ +#include #include #include #include @@ -174,6 +175,7 @@ bool mcpm_is_available(void) { return (platform_ops) ? true : false; } +EXPORT_SYMBOL_GPL(mcpm_is_available); /* * We can't use regular spinlocks. In the switcher case, it is possible diff --git a/arch/arm/common/secure_cntvoff.S b/arch/arm/common/secure_cntvoff.S new file mode 100644 index 000000000000..53fc7bdb6c2e --- /dev/null +++ b/arch/arm/common/secure_cntvoff.S @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2014 Renesas Electronics Corporation + * + * Initialization of CNTVOFF register from secure mode + * + */ + +#include +#include + +ENTRY(secure_cntvoff_init) + .arch armv7-a + /* + * CNTVOFF has to be initialized either from non-secure Hypervisor + * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled + * then it should be handled by the secure code. The CPU must implement + * the virtualization extensions. + */ + cps #MON_MODE + mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ + orr r0, r1, #1 + mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ + isb + mov r0, #0 + mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ + isb + mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ + isb + cps #SVC_MODE + ret lr +ENDPROC(secure_cntvoff_init) diff --git a/arch/arm/configs/bcm2835_defconfig b/arch/arm/configs/bcm2835_defconfig index 8682b15336b9..e4d188f0a4b4 100644 --- a/arch/arm/configs/bcm2835_defconfig +++ b/arch/arm/configs/bcm2835_defconfig @@ -64,6 +64,7 @@ CONFIG_BLK_DEV_SD=y CONFIG_SCSI_CONSTANTS=y CONFIG_SCSI_SCAN_ASYNC=y CONFIG_NETDEVICES=y +CONFIG_USB_LAN78XX=y CONFIG_USB_USBNET=y CONFIG_USB_NET_SMSC95XX=y CONFIG_BRCMFMAC=m @@ -127,6 +128,7 @@ CONFIG_LEDS_TRIGGER_CAMERA=y CONFIG_DMADEVICES=y CONFIG_DMA_BCM2835=y CONFIG_STAGING=y +CONFIG_BCM2835_VCHIQ=m CONFIG_MAILBOX=y CONFIG_BCM2835_MBOX=y # CONFIG_IOMMU_SUPPORT is not set diff --git a/arch/arm/configs/davinci_all_defconfig b/arch/arm/configs/davinci_all_defconfig index c302a04e8cbc..21b2d7791df4 100644 --- a/arch/arm/configs/davinci_all_defconfig +++ b/arch/arm/configs/davinci_all_defconfig @@ -56,7 +56,7 @@ CONFIG_IP_PNP_DHCP=y CONFIG_NETFILTER=y CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y -# CONFIG_FW_LOADER is not set +CONFIG_FW_LOADER=m CONFIG_DMA_CMA=y CONFIG_DA8XX_MSTPRI=y CONFIG_MTD=m @@ -212,6 +212,8 @@ CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_OMAP=m CONFIG_DMADEVICES=y CONFIG_TI_EDMA=y +CONFIG_REMOTEPROC=m +CONFIG_DA8XX_REMOTEPROC=m CONFIG_MEMORY=y CONFIG_TI_AEMIF=m CONFIG_DA8XX_DDRCTL=y diff --git a/arch/arm/configs/exynos_defconfig b/arch/arm/configs/exynos_defconfig index 629189c62fd1..85b2369d6b20 100644 --- a/arch/arm/configs/exynos_defconfig +++ b/arch/arm/configs/exynos_defconfig @@ -208,6 +208,7 @@ CONFIG_DRM_EXYNOS_DSI=y CONFIG_DRM_EXYNOS_HDMI=y CONFIG_DRM_PANEL_SIMPLE=y CONFIG_DRM_PANEL_SAMSUNG_LD9040=y +CONFIG_DRM_PANEL_SAMSUNG_S6E63J0X03=y CONFIG_DRM_PANEL_SAMSUNG_S6E8AA0=y CONFIG_DRM_NXP_PTN3460=y CONFIG_DRM_PARADE_PS8622=y diff --git a/arch/arm/configs/imx_v6_v7_defconfig b/arch/arm/configs/imx_v6_v7_defconfig index 3a308437b088..f70507ab91ee 100644 --- a/arch/arm/configs/imx_v6_v7_defconfig +++ b/arch/arm/configs/imx_v6_v7_defconfig @@ -38,6 +38,7 @@ CONFIG_SOC_IMX51=y CONFIG_SOC_IMX53=y CONFIG_SOC_IMX6Q=y CONFIG_SOC_IMX6SL=y +CONFIG_SOC_IMX6SLL=y CONFIG_SOC_IMX6SX=y CONFIG_SOC_IMX6UL=y CONFIG_SOC_IMX7D=y @@ -153,6 +154,9 @@ CONFIG_USB_RTL8152=m CONFIG_USB_USBNET=y CONFIG_USB_NET_CDC_EEM=m CONFIG_BRCMFMAC=m +CONFIG_MWIFIEX=m +CONFIG_MWIFIEX_SDIO=m +CONFIG_MWIFIEX_PCIE=m CONFIG_WL12XX=m CONFIG_WL18XX=m CONFIG_WLCORE_SDIO=m @@ -199,6 +203,7 @@ CONFIG_SPI_GPIO=y CONFIG_SPI_IMX=y CONFIG_SPI_FSL_DSPI=y CONFIG_GPIO_SYSFS=y +CONFIG_GPIO_MAX732X=y CONFIG_GPIO_MC9S08DZ60=y CONFIG_GPIO_PCA953X=y CONFIG_GPIO_STMPE=y @@ -214,11 +219,13 @@ CONFIG_CPU_THERMAL=y CONFIG_IMX_THERMAL=y CONFIG_WATCHDOG=y CONFIG_DA9062_WATCHDOG=y +CONFIG_RN5T618_WATCHDOG=y CONFIG_IMX2_WDT=y CONFIG_MFD_DA9052_I2C=y CONFIG_MFD_DA9062=y CONFIG_MFD_MC13XXX_SPI=y CONFIG_MFD_MC13XXX_I2C=y +CONFIG_MFD_RN5T618=y CONFIG_MFD_STMPE=y CONFIG_REGULATOR=y CONFIG_REGULATOR_FIXED_VOLTAGE=y @@ -229,6 +236,7 @@ CONFIG_REGULATOR_GPIO=y CONFIG_REGULATOR_MC13783=y CONFIG_REGULATOR_MC13892=y CONFIG_REGULATOR_PFUZE100=y +CONFIG_REGULATOR_RN5T618=y CONFIG_RC_CORE=y CONFIG_RC_DEVICES=y CONFIG_IR_GPIO_CIR=y @@ -374,6 +382,7 @@ CONFIG_PWM=y CONFIG_PWM_FSL_FTM=y CONFIG_PWM_IMX=y CONFIG_NVMEM_IMX_OCOTP=y +CONFIG_NVMEM_VF610_OCOTP=y CONFIG_TEE=y CONFIG_OPTEE=y CONFIG_MUX_MMIO=y diff --git a/arch/arm/configs/multi_v7_defconfig b/arch/arm/configs/multi_v7_defconfig index e6b3c96d4c09..7e1c543162c3 100644 --- a/arch/arm/configs/multi_v7_defconfig +++ b/arch/arm/configs/multi_v7_defconfig @@ -90,6 +90,7 @@ CONFIG_ARCH_R8A73A4=y CONFIG_ARCH_R8A7740=y CONFIG_ARCH_R8A7743=y CONFIG_ARCH_R8A7745=y +CONFIG_ARCH_R8A77470=y CONFIG_ARCH_R8A7778=y CONFIG_ARCH_R8A7779=y CONFIG_ARCH_R8A7790=y @@ -187,6 +188,8 @@ CONFIG_B53_MMAP_DRIVER=m CONFIG_B53_SRAB_DRIVER=m CONFIG_CAN_SUN4I=y CONFIG_BT=m +CONFIG_BT_HCIUART=m +CONFIG_BT_HCIUART_BCM=y CONFIG_BT_MRVL=m CONFIG_BT_MRVL_SDIO=m CONFIG_CFG80211=m @@ -280,6 +283,7 @@ CONFIG_FIXED_PHY=y CONFIG_ROCKCHIP_PHY=y CONFIG_USB_PEGASUS=y CONFIG_USB_RTL8152=m +CONFIG_USB_LAN78XX=m CONFIG_USB_USBNET=y CONFIG_USB_NET_SMSC75XX=y CONFIG_USB_NET_SMSC95XX=y @@ -360,10 +364,12 @@ CONFIG_SERIAL_ST_ASC=y CONFIG_SERIAL_ST_ASC_CONSOLE=y CONFIG_SERIAL_STM32=y CONFIG_SERIAL_STM32_CONSOLE=y +CONFIG_SERIAL_DEV_BUS=y CONFIG_HVC_DRIVER=y CONFIG_VIRTIO_CONSOLE=y CONFIG_I2C_CHARDEV=y CONFIG_I2C_DAVINCI=y +CONFIG_I2C_MESON=y CONFIG_I2C_MUX=y CONFIG_I2C_ARB_GPIO_CHALLENGE=m CONFIG_I2C_MUX_PCA954x=y @@ -385,6 +391,7 @@ CONFIG_I2C_S3C2410=y CONFIG_I2C_SH_MOBILE=y CONFIG_I2C_SIRF=y CONFIG_I2C_ST=y +CONFIG_I2C_STM32F7=y CONFIG_I2C_SUN6I_P2WI=y CONFIG_I2C_TEGRA=y CONFIG_I2C_UNIPHIER=y @@ -497,6 +504,7 @@ CONFIG_TEGRA_WATCHDOG=m CONFIG_MESON_WATCHDOG=y CONFIG_DW_WATCHDOG=y CONFIG_DIGICOLOR_WATCHDOG=y +CONFIG_RENESAS_WDT=m CONFIG_BCM2835_WDT=y CONFIG_BCM47XX_WDT=y CONFIG_BCM7038_WDT=m @@ -638,6 +646,7 @@ CONFIG_DRM_SUN4I=m CONFIG_DRM_FSL_DCU=m CONFIG_DRM_TEGRA=y CONFIG_DRM_PANEL_SAMSUNG_LD9040=m +CONFIG_DRM_PANEL_SAMSUNG_S6E63J0X03=m CONFIG_DRM_PANEL_SAMSUNG_S6E8AA0=m CONFIG_DRM_PANEL_SIMPLE=y CONFIG_DRM_SII9234=m @@ -650,7 +659,6 @@ CONFIG_FB_EFI=y CONFIG_FB_WM8505=y CONFIG_FB_SH_MOBILE_LCDC=y CONFIG_FB_SIMPLE=y -CONFIG_FB_SH_MOBILE_MERAM=y CONFIG_BACKLIGHT_LCD_SUPPORT=y CONFIG_BACKLIGHT_CLASS_DEVICE=y CONFIG_LCD_PLATFORM=m @@ -947,6 +955,7 @@ CONFIG_PWM_ATMEL=m CONFIG_PWM_ATMEL_HLCDC_PWM=m CONFIG_PWM_ATMEL_TCB=m CONFIG_PWM_FSL_FTM=m +CONFIG_PWM_MESON=m CONFIG_PWM_RCAR=m CONFIG_PWM_RENESAS_TPU=y CONFIG_PWM_ROCKCHIP=m @@ -972,6 +981,7 @@ CONFIG_PHY_QCOM_APQ8064_SATA=m CONFIG_PHY_MIPHY28LP=y CONFIG_PHY_RCAR_GEN2=m CONFIG_PHY_STIH407_USB=y +CONFIG_PHY_STM32_USBPHYC=y CONFIG_PHY_SUN4I_USB=y CONFIG_PHY_SUN9I_USB=y CONFIG_PHY_SAMSUNG_USB2=m diff --git a/arch/arm/configs/shmobile_defconfig b/arch/arm/configs/shmobile_defconfig index a701601fbd76..b49887e86a3d 100644 --- a/arch/arm/configs/shmobile_defconfig +++ b/arch/arm/configs/shmobile_defconfig @@ -14,6 +14,7 @@ CONFIG_ARCH_R8A73A4=y CONFIG_ARCH_R8A7740=y CONFIG_ARCH_R8A7743=y CONFIG_ARCH_R8A7745=y +CONFIG_ARCH_R8A77470=y CONFIG_ARCH_R8A7778=y CONFIG_ARCH_R8A7779=y CONFIG_ARCH_R8A7790=y @@ -127,6 +128,7 @@ CONFIG_CPU_THERMAL=y CONFIG_RCAR_THERMAL=y CONFIG_WATCHDOG=y CONFIG_DA9063_WATCHDOG=y +CONFIG_RENESAS_WDT=y CONFIG_MFD_AS3711=y CONFIG_MFD_DA9063=y CONFIG_REGULATOR_FIXED_VOLTAGE=y @@ -156,7 +158,6 @@ CONFIG_DRM_DUMB_VGA_DAC=y CONFIG_DRM_I2C_ADV7511=y CONFIG_DRM_I2C_ADV7511_AUDIO=y CONFIG_FB_SH_MOBILE_LCDC=y -CONFIG_FB_SH_MOBILE_MERAM=y # CONFIG_LCD_CLASS_DEVICE is not set # CONFIG_BACKLIGHT_GENERIC is not set CONFIG_BACKLIGHT_PWM=y diff --git a/arch/arm/crypto/sha1-armv4-large.S b/arch/arm/crypto/sha1-armv4-large.S index 99207c45ec10..f82cd8cf5a09 100644 --- a/arch/arm/crypto/sha1-armv4-large.S +++ b/arch/arm/crypto/sha1-armv4-large.S @@ -1,4 +1,14 @@ #define __ARM_ARCH__ __LINUX_ARM_ARCH__ +@ SPDX-License-Identifier: GPL-2.0 + +@ This code is taken from the OpenSSL project but the author (Andy Polyakov) +@ has relicensed it under the GPLv2. Therefore this program is free software; +@ you can redistribute it and/or modify it under the terms of the GNU General +@ Public License version 2 as published by the Free Software Foundation. +@ +@ The original headers, including the original license headers, are +@ included below for completeness. + @ ==================================================================== @ Written by Andy Polyakov for the OpenSSL @ project. The module is, however, dual licensed under OpenSSL and diff --git a/arch/arm/crypto/sha256-armv4.pl b/arch/arm/crypto/sha256-armv4.pl index fac0533ea633..b9ec44060ed3 100644 --- a/arch/arm/crypto/sha256-armv4.pl +++ b/arch/arm/crypto/sha256-armv4.pl @@ -1,12 +1,19 @@ #!/usr/bin/env perl +# SPDX-License-Identifier: GPL-2.0 + +# This code is taken from the OpenSSL project but the author (Andy Polyakov) +# has relicensed it under the GPLv2. Therefore this program is free software; +# you can redistribute it and/or modify it under the terms of the GNU General +# Public License version 2 as published by the Free Software Foundation. +# +# The original headers, including the original license headers, are +# included below for completeness. # ==================================================================== # Written by Andy Polyakov for the OpenSSL # project. The module is, however, dual licensed under OpenSSL and # CRYPTOGAMS licenses depending on where you obtain it. For further # details see http://www.openssl.org/~appro/cryptogams/. -# -# Permission to use under GPL terms is granted. # ==================================================================== # SHA256 block procedure for ARMv4. May 2007. diff --git a/arch/arm/crypto/sha256-core.S_shipped b/arch/arm/crypto/sha256-core.S_shipped index 555a1a8eec90..3b58300d611c 100644 --- a/arch/arm/crypto/sha256-core.S_shipped +++ b/arch/arm/crypto/sha256-core.S_shipped @@ -1,11 +1,18 @@ +@ SPDX-License-Identifier: GPL-2.0 + +@ This code is taken from the OpenSSL project but the author (Andy Polyakov) +@ has relicensed it under the GPLv2. Therefore this program is free software; +@ you can redistribute it and/or modify it under the terms of the GNU General +@ Public License version 2 as published by the Free Software Foundation. +@ +@ The original headers, including the original license headers, are +@ included below for completeness. @ ==================================================================== @ Written by Andy Polyakov for the OpenSSL @ project. The module is, however, dual licensed under OpenSSL and @ CRYPTOGAMS licenses depending on where you obtain it. For further @ details see http://www.openssl.org/~appro/cryptogams/. -@ -@ Permission to use under GPL terms is granted. @ ==================================================================== @ SHA256 block procedure for ARMv4. May 2007. diff --git a/arch/arm/crypto/sha512-armv4.pl b/arch/arm/crypto/sha512-armv4.pl index a2b11a844357..fb5d15048c0b 100644 --- a/arch/arm/crypto/sha512-armv4.pl +++ b/arch/arm/crypto/sha512-armv4.pl @@ -1,12 +1,19 @@ #!/usr/bin/env perl +# SPDX-License-Identifier: GPL-2.0 + +# This code is taken from the OpenSSL project but the author (Andy Polyakov) +# has relicensed it under the GPLv2. Therefore this program is free software; +# you can redistribute it and/or modify it under the terms of the GNU General +# Public License version 2 as published by the Free Software Foundation. +# +# The original headers, including the original license headers, are +# included below for completeness. # ==================================================================== # Written by Andy Polyakov for the OpenSSL # project. The module is, however, dual licensed under OpenSSL and # CRYPTOGAMS licenses depending on where you obtain it. For further # details see http://www.openssl.org/~appro/cryptogams/. -# -# Permission to use under GPL terms is granted. # ==================================================================== # SHA512 block procedure for ARMv4. September 2007. diff --git a/arch/arm/crypto/sha512-core.S_shipped b/arch/arm/crypto/sha512-core.S_shipped index 3694c4d4ca2b..b1c334a49cda 100644 --- a/arch/arm/crypto/sha512-core.S_shipped +++ b/arch/arm/crypto/sha512-core.S_shipped @@ -1,11 +1,18 @@ +@ SPDX-License-Identifier: GPL-2.0 + +@ This code is taken from the OpenSSL project but the author (Andy Polyakov) +@ has relicensed it under the GPLv2. Therefore this program is free software; +@ you can redistribute it and/or modify it under the terms of the GNU General +@ Public License version 2 as published by the Free Software Foundation. +@ +@ The original headers, including the original license headers, are +@ included below for completeness. @ ==================================================================== @ Written by Andy Polyakov for the OpenSSL @ project. The module is, however, dual licensed under OpenSSL and @ CRYPTOGAMS licenses depending on where you obtain it. For further @ details see http://www.openssl.org/~appro/cryptogams/. -@ -@ Permission to use under GPL terms is granted. @ ==================================================================== @ SHA512 block procedure for ARMv4. September 2007. diff --git a/arch/arm/include/asm/Kbuild b/arch/arm/include/asm/Kbuild index 873e3c189279..1d66db9c9db5 100644 --- a/arch/arm/include/asm/Kbuild +++ b/arch/arm/include/asm/Kbuild @@ -1,3 +1,4 @@ +generic-y += compat.h generic-y += current.h generic-y += early_ioremap.h generic-y += emergency-restart.h diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index 9342904cccca..0cd4dccbae78 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -447,6 +447,14 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) .size \name , . - \name .endm + .macro csdb +#ifdef CONFIG_THUMB2_KERNEL + .inst.w 0xf3af8014 +#else + .inst 0xe320f014 +#endif + .endm + .macro check_uaccess, addr:req, size:req, limit:req, tmp:req, bad:req #ifndef CONFIG_CPU_USE_DOMAINS adds \tmp, \addr, #\size - 1 diff --git a/arch/arm/include/asm/barrier.h b/arch/arm/include/asm/barrier.h index 40f5c410fd8c..69772e742a0a 100644 --- a/arch/arm/include/asm/barrier.h +++ b/arch/arm/include/asm/barrier.h @@ -17,6 +17,12 @@ #define isb(option) __asm__ __volatile__ ("isb " #option : : : "memory") #define dsb(option) __asm__ __volatile__ ("dsb " #option : : : "memory") #define dmb(option) __asm__ __volatile__ ("dmb " #option : : : "memory") +#ifdef CONFIG_THUMB2_KERNEL +#define CSDB ".inst.w 0xf3af8014" +#else +#define CSDB ".inst 0xe320f014" +#endif +#define csdb() __asm__ __volatile__(CSDB : : : "memory") #elif defined(CONFIG_CPU_XSC3) || __LINUX_ARM_ARCH__ == 6 #define isb(x) __asm__ __volatile__ ("mcr p15, 0, %0, c7, c5, 4" \ : : "r" (0) : "memory") @@ -37,6 +43,13 @@ #define dmb(x) __asm__ __volatile__ ("" : : : "memory") #endif +#ifndef CSDB +#define CSDB +#endif +#ifndef csdb +#define csdb() +#endif + #ifdef CONFIG_ARM_HEAVY_MB extern void (*soc_mb)(void); extern void arm_heavy_mb(void); @@ -63,6 +76,25 @@ extern void arm_heavy_mb(void); #define __smp_rmb() __smp_mb() #define __smp_wmb() dmb(ishst) +#ifdef CONFIG_CPU_SPECTRE +static inline unsigned long array_index_mask_nospec(unsigned long idx, + unsigned long sz) +{ + unsigned long mask; + + asm volatile( + "cmp %1, %2\n" + " sbc %0, %1, %1\n" + CSDB + : "=r" (mask) + : "r" (idx), "Ir" (sz) + : "cc"); + + return mask; +} +#define array_index_mask_nospec array_index_mask_nospec +#endif + #include #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm/include/asm/bugs.h b/arch/arm/include/asm/bugs.h index a97f1ea708d1..73a99c72a930 100644 --- a/arch/arm/include/asm/bugs.h +++ b/arch/arm/include/asm/bugs.h @@ -10,12 +10,14 @@ #ifndef __ASM_BUGS_H #define __ASM_BUGS_H -#ifdef CONFIG_MMU extern void check_writebuffer_bugs(void); -#define check_bugs() check_writebuffer_bugs() +#ifdef CONFIG_MMU +extern void check_bugs(void); +extern void check_other_bugs(void); #else #define check_bugs() do { } while (0) +#define check_other_bugs() do { } while (0) #endif #endif diff --git a/arch/arm/include/asm/cacheflush.h b/arch/arm/include/asm/cacheflush.h index 869080bedb89..ec1a5fd0d294 100644 --- a/arch/arm/include/asm/cacheflush.h +++ b/arch/arm/include/asm/cacheflush.h @@ -35,7 +35,7 @@ * Start addresses are inclusive and end addresses are exclusive; * start addresses should be rounded down, end addresses up. * - * See Documentation/cachetlb.txt for more information. + * See Documentation/core-api/cachetlb.rst for more information. * Please note that the implementation of these, and the required * effects are cache-type (VIVT/VIPT/PIPT) specific. * diff --git a/arch/arm/include/asm/cp15.h b/arch/arm/include/asm/cp15.h index 4c9fa72b59f5..07e27f212dc7 100644 --- a/arch/arm/include/asm/cp15.h +++ b/arch/arm/include/asm/cp15.h @@ -65,6 +65,9 @@ #define __write_sysreg(v, r, w, c, t) asm volatile(w " " c : : "r" ((t)(v))) #define write_sysreg(v, ...) __write_sysreg(v, __VA_ARGS__) +#define BPIALL __ACCESS_CP15(c7, 0, c5, 6) +#define ICIALLU __ACCESS_CP15(c7, 0, c5, 0) + extern unsigned long cr_alignment; /* defined in entry-armv.S */ static inline unsigned long get_cr(void) diff --git a/arch/arm/include/asm/cputype.h b/arch/arm/include/asm/cputype.h index cb546425da8a..0d289240b6ca 100644 --- a/arch/arm/include/asm/cputype.h +++ b/arch/arm/include/asm/cputype.h @@ -2,9 +2,6 @@ #ifndef __ASM_ARM_CPUTYPE_H #define __ASM_ARM_CPUTYPE_H -#include -#include - #define CPUID_ID 0 #define CPUID_CACHETYPE 1 #define CPUID_TCM 2 @@ -62,6 +59,7 @@ ((mpidr >> (MPIDR_LEVEL_BITS * level)) & MPIDR_LEVEL_MASK) #define ARM_CPU_IMP_ARM 0x41 +#define ARM_CPU_IMP_BRCM 0x42 #define ARM_CPU_IMP_DEC 0x44 #define ARM_CPU_IMP_INTEL 0x69 @@ -77,8 +75,17 @@ #define ARM_CPU_PART_CORTEX_A12 0x4100c0d0 #define ARM_CPU_PART_CORTEX_A17 0x4100c0e0 #define ARM_CPU_PART_CORTEX_A15 0x4100c0f0 +#define ARM_CPU_PART_CORTEX_A53 0x4100d030 +#define ARM_CPU_PART_CORTEX_A57 0x4100d070 +#define ARM_CPU_PART_CORTEX_A72 0x4100d080 +#define ARM_CPU_PART_CORTEX_A73 0x4100d090 +#define ARM_CPU_PART_CORTEX_A75 0x4100d0a0 #define ARM_CPU_PART_MASK 0xff00fff0 +/* Broadcom implemented processors */ +#define ARM_CPU_PART_BRAHMA_B15 0x420000f0 +#define ARM_CPU_PART_BRAHMA_B53 0x42001000 + /* DEC implemented cores */ #define ARM_CPU_PART_SA1100 0x4400a110 @@ -98,6 +105,11 @@ /* Qualcomm implemented cores */ #define ARM_CPU_PART_SCORPION 0x510002d0 +#ifndef __ASSEMBLY__ + +#include +#include + extern unsigned int processor_id; #ifdef CONFIG_CPU_CP15 @@ -326,4 +338,6 @@ static inline int __attribute_const__ cpuid_feature_extract_field(u32 features, #define cpuid_feature_extract(reg, field) \ cpuid_feature_extract_field(read_cpuid_ext(reg), field) +#endif /* __ASSEMBLY__ */ + #endif diff --git a/arch/arm/include/asm/kgdb.h b/arch/arm/include/asm/kgdb.h index 3b73fdcf3627..8de1100d1067 100644 --- a/arch/arm/include/asm/kgdb.h +++ b/arch/arm/include/asm/kgdb.h @@ -77,7 +77,7 @@ extern int kgdb_fault_expected; #define KGDB_MAX_NO_CPUS 1 #define BUFMAX 400 -#define NUMREGBYTES (DBG_MAX_REG_NUM << 2) +#define NUMREGBYTES (GDB_MAX_REGS << 2) #define NUMCRITREGBYTES (32 << 2) #define _R0 0 diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h index 5a953ecb0d78..231e87ad45d5 100644 --- a/arch/arm/include/asm/kvm_asm.h +++ b/arch/arm/include/asm/kvm_asm.h @@ -61,8 +61,6 @@ struct kvm_vcpu; extern char __kvm_hyp_init[]; extern char __kvm_hyp_init_end[]; -extern char __kvm_hyp_vector[]; - extern void __kvm_flush_vm_context(void); extern void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa); extern void __kvm_tlb_flush_vmid(struct kvm *kvm); diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h index c7c28c885a19..1f1fe4109b02 100644 --- a/arch/arm/include/asm/kvm_host.h +++ b/arch/arm/include/asm/kvm_host.h @@ -21,6 +21,7 @@ #include #include +#include #include #include #include @@ -280,6 +281,7 @@ void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot); struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr); +static inline bool kvm_arch_check_sve_has_vhe(void) { return true; } static inline void kvm_arch_hardware_unsetup(void) {} static inline void kvm_arch_sync_events(struct kvm *kvm) {} static inline void kvm_arch_vcpu_uninit(struct kvm_vcpu *vcpu) {} @@ -303,19 +305,49 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu, int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr); -/* All host FP/SIMD state is restored on guest exit, so nothing to save: */ -static inline void kvm_fpsimd_flush_cpu_state(void) {} +/* + * VFP/NEON switching is all done by the hyp switch code, so no need to + * coordinate with host context handling for this state: + */ +static inline void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) {} +static inline void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) {} +static inline void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) {} static inline void kvm_arm_vhe_guest_enter(void) {} static inline void kvm_arm_vhe_guest_exit(void) {} static inline bool kvm_arm_harden_branch_predictor(void) +{ + switch(read_cpuid_part()) { +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR + case ARM_CPU_PART_BRAHMA_B15: + case ARM_CPU_PART_CORTEX_A12: + case ARM_CPU_PART_CORTEX_A15: + case ARM_CPU_PART_CORTEX_A17: + return true; +#endif + default: + return false; + } +} + +#define KVM_SSBD_UNKNOWN -1 +#define KVM_SSBD_FORCE_DISABLE 0 +#define KVM_SSBD_KERNEL 1 +#define KVM_SSBD_FORCE_ENABLE 2 +#define KVM_SSBD_MITIGATED 3 + +static inline int kvm_arm_have_ssbd(void) { /* No way to detect it yet, pretend it is not there. */ - return false; + return KVM_SSBD_UNKNOWN; } static inline void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu) {} static inline void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu) {} +#define __KVM_HAVE_ARCH_VM_ALLOC +struct kvm *kvm_arch_alloc_vm(void); +void kvm_arch_free_vm(struct kvm *kvm); + #endif /* __ARM_KVM_HOST_H__ */ diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index f675162663f0..8553d68b7c8a 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -327,7 +327,28 @@ static inline int kvm_read_guest_lock(struct kvm *kvm, static inline void *kvm_get_hyp_vector(void) { - return kvm_ksym_ref(__kvm_hyp_vector); + switch(read_cpuid_part()) { +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR + case ARM_CPU_PART_CORTEX_A12: + case ARM_CPU_PART_CORTEX_A17: + { + extern char __kvm_hyp_vector_bp_inv[]; + return kvm_ksym_ref(__kvm_hyp_vector_bp_inv); + } + + case ARM_CPU_PART_BRAHMA_B15: + case ARM_CPU_PART_CORTEX_A15: + { + extern char __kvm_hyp_vector_ic_inv[]; + return kvm_ksym_ref(__kvm_hyp_vector_ic_inv); + } +#endif + default: + { + extern char __kvm_hyp_vector[]; + return kvm_ksym_ref(__kvm_hyp_vector); + } + } } static inline int kvm_map_vectors(void) @@ -335,6 +356,11 @@ static inline int kvm_map_vectors(void) return 0; } +static inline int hyp_map_aux_data(void) +{ + return 0; +} + #define kvm_phys_to_vttbr(addr) (addr) #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm/include/asm/mpu.h b/arch/arm/include/asm/mpu.h index 6d1491c8ee22..5e088c83d3d8 100644 --- a/arch/arm/include/asm/mpu.h +++ b/arch/arm/include/asm/mpu.h @@ -12,60 +12,101 @@ /* ID_MMFR0 data relevant to MPU */ #define MMFR0_PMSA (0xF << 4) #define MMFR0_PMSAv7 (3 << 4) +#define MMFR0_PMSAv8 (4 << 4) /* MPU D/I Size Register fields */ -#define MPU_RSR_SZ 1 -#define MPU_RSR_EN 0 -#define MPU_RSR_SD 8 +#define PMSAv7_RSR_SZ 1 +#define PMSAv7_RSR_EN 0 +#define PMSAv7_RSR_SD 8 /* Number of subregions (SD) */ -#define MPU_NR_SUBREGS 8 -#define MPU_MIN_SUBREG_SIZE 256 +#define PMSAv7_NR_SUBREGS 8 +#define PMSAv7_MIN_SUBREG_SIZE 256 /* The D/I RSR value for an enabled region spanning the whole of memory */ -#define MPU_RSR_ALL_MEM 63 +#define PMSAv7_RSR_ALL_MEM 63 /* Individual bits in the DR/IR ACR */ -#define MPU_ACR_XN (1 << 12) -#define MPU_ACR_SHARED (1 << 2) +#define PMSAv7_ACR_XN (1 << 12) +#define PMSAv7_ACR_SHARED (1 << 2) /* C, B and TEX[2:0] bits only have semantic meanings when grouped */ -#define MPU_RGN_CACHEABLE 0xB -#define MPU_RGN_SHARED_CACHEABLE (MPU_RGN_CACHEABLE | MPU_ACR_SHARED) -#define MPU_RGN_STRONGLY_ORDERED 0 +#define PMSAv7_RGN_CACHEABLE 0xB +#define PMSAv7_RGN_SHARED_CACHEABLE (PMSAv7_RGN_CACHEABLE | PMSAv7_ACR_SHARED) +#define PMSAv7_RGN_STRONGLY_ORDERED 0 /* Main region should only be shared for SMP */ #ifdef CONFIG_SMP -#define MPU_RGN_NORMAL (MPU_RGN_CACHEABLE | MPU_ACR_SHARED) +#define PMSAv7_RGN_NORMAL (PMSAv7_RGN_CACHEABLE | PMSAv7_ACR_SHARED) #else -#define MPU_RGN_NORMAL MPU_RGN_CACHEABLE +#define PMSAv7_RGN_NORMAL PMSAv7_RGN_CACHEABLE #endif /* Access permission bits of ACR (only define those that we use)*/ -#define MPU_AP_PL1RO_PL0NA (0x5 << 8) -#define MPU_AP_PL1RW_PL0RW (0x3 << 8) -#define MPU_AP_PL1RW_PL0R0 (0x2 << 8) -#define MPU_AP_PL1RW_PL0NA (0x1 << 8) +#define PMSAv7_AP_PL1RO_PL0NA (0x5 << 8) +#define PMSAv7_AP_PL1RW_PL0RW (0x3 << 8) +#define PMSAv7_AP_PL1RW_PL0R0 (0x2 << 8) +#define PMSAv7_AP_PL1RW_PL0NA (0x1 << 8) + +#define PMSAv8_BAR_XN 1 + +#define PMSAv8_LAR_EN 1 +#define PMSAv8_LAR_IDX(n) (((n) & 0x7) << 1) + + +#define PMSAv8_AP_PL1RW_PL0NA (0 << 1) +#define PMSAv8_AP_PL1RW_PL0RW (1 << 1) +#define PMSAv8_AP_PL1RO_PL0RO (3 << 1) + +#ifdef CONFIG_SMP +#define PMSAv8_RGN_SHARED (3 << 3) // inner sharable +#else +#define PMSAv8_RGN_SHARED (0 << 3) +#endif + +#define PMSAv8_RGN_DEVICE_nGnRnE 0 +#define PMSAv8_RGN_NORMAL 1 + +#define PMSAv8_MAIR(attr, mt) ((attr) << ((mt) * 8)) + +#ifdef CONFIG_CPU_V7M +#define PMSAv8_MINALIGN 32 +#else +#define PMSAv8_MINALIGN 64 +#endif /* For minimal static MPU region configurations */ -#define MPU_PROBE_REGION 0 -#define MPU_BG_REGION 1 -#define MPU_RAM_REGION 2 -#define MPU_ROM_REGION 3 +#define PMSAv7_PROBE_REGION 0 +#define PMSAv7_BG_REGION 1 +#define PMSAv7_RAM_REGION 2 +#define PMSAv7_ROM_REGION 3 + +/* Fixed for PMSAv8 only */ +#define PMSAv8_XIP_REGION 0 +#define PMSAv8_KERNEL_REGION 1 /* Maximum number of regions Linux is interested in */ -#define MPU_MAX_REGIONS 16 +#define MPU_MAX_REGIONS 16 -#define MPU_DATA_SIDE 0 -#define MPU_INSTR_SIDE 1 +#define PMSAv7_DATA_SIDE 0 +#define PMSAv7_INSTR_SIDE 1 #ifndef __ASSEMBLY__ struct mpu_rgn { /* Assume same attributes for d/i-side */ - u32 drbar; - u32 drsr; - u32 dracr; + union { + u32 drbar; /* PMSAv7 */ + u32 prbar; /* PMSAv8 */ + }; + union { + u32 drsr; /* PMSAv7 */ + u32 prlar; /* PMSAv8 */ + }; + union { + u32 dracr; /* PMSAv7 */ + u32 unused; /* not used in PMSAv8 */ + }; }; struct mpu_rgn_info { @@ -75,16 +116,17 @@ struct mpu_rgn_info { extern struct mpu_rgn_info mpu_rgn_info; #ifdef CONFIG_ARM_MPU +extern void __init pmsav7_adjust_lowmem_bounds(void); +extern void __init pmsav8_adjust_lowmem_bounds(void); -extern void __init adjust_lowmem_bounds_mpu(void); -extern void __init mpu_setup(void); - +extern void __init pmsav7_setup(void); +extern void __init pmsav8_setup(void); #else - -static inline void adjust_lowmem_bounds_mpu(void) {} -static inline void mpu_setup(void) {} - -#endif /* !CONFIG_ARM_MPU */ +static inline void pmsav7_adjust_lowmem_bounds(void) {}; +static inline void pmsav8_adjust_lowmem_bounds(void) {}; +static inline void pmsav7_setup(void) {}; +static inline void pmsav8_setup(void) {}; +#endif #endif /* __ASSEMBLY__ */ diff --git a/arch/arm/include/asm/pci.h b/arch/arm/include/asm/pci.h index 1f0de808d111..0abd389cf0ec 100644 --- a/arch/arm/include/asm/pci.h +++ b/arch/arm/include/asm/pci.h @@ -19,13 +19,6 @@ static inline int pci_proc_domain(struct pci_bus *bus) } #endif /* CONFIG_PCI_DOMAINS */ -/* - * The PCI address space does equal the physical memory address space. - * The networking and block device layers use this boolean for bounce - * buffer decisions. - */ -#define PCI_DMA_BUS_IS_PHYS (1) - #define HAVE_PCI_MMAP #define ARCH_GENERIC_PCI_MMAP_RESOURCE diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h index 2a4836087358..6d50a11d7793 100644 --- a/arch/arm/include/asm/pgtable-3level.h +++ b/arch/arm/include/asm/pgtable-3level.h @@ -219,7 +219,6 @@ static inline pte_t pte_mkspecial(pte_t pte) pte_val(pte) |= L_PTE_SPECIAL; return pte; } -#define __HAVE_ARCH_PTE_SPECIAL #define pmd_write(pmd) (pmd_isclear((pmd), L_PMD_SECT_RDONLY)) #define pmd_dirty(pmd) (pmd_isset((pmd), L_PMD_SECT_DIRTY)) diff --git a/arch/arm/include/asm/proc-fns.h b/arch/arm/include/asm/proc-fns.h index f2e1af45bd6f..e25f4392e1b2 100644 --- a/arch/arm/include/asm/proc-fns.h +++ b/arch/arm/include/asm/proc-fns.h @@ -36,6 +36,10 @@ extern struct processor { * Set up any processor specifics */ void (*_proc_init)(void); + /* + * Check for processor bugs + */ + void (*check_bugs)(void); /* * Disable any processor specifics */ diff --git a/arch/arm/include/asm/secure_cntvoff.h b/arch/arm/include/asm/secure_cntvoff.h new file mode 100644 index 000000000000..1f93aee1f630 --- /dev/null +++ b/arch/arm/include/asm/secure_cntvoff.h @@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASMARM_ARCH_CNTVOFF_H +#define __ASMARM_ARCH_CNTVOFF_H + +extern void secure_cntvoff_init(void); + +#endif diff --git a/arch/arm/include/asm/system_misc.h b/arch/arm/include/asm/system_misc.h index 78f6db114faf..8e76db83c498 100644 --- a/arch/arm/include/asm/system_misc.h +++ b/arch/arm/include/asm/system_misc.h @@ -8,6 +8,7 @@ #include #include #include +#include extern void cpu_init(void); @@ -15,6 +16,20 @@ void soft_restart(unsigned long); extern void (*arm_pm_restart)(enum reboot_mode reboot_mode, const char *cmd); extern void (*arm_pm_idle)(void); +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR +typedef void (*harden_branch_predictor_fn_t)(void); +DECLARE_PER_CPU(harden_branch_predictor_fn_t, harden_branch_predictor_fn); +static inline void harden_branch_predictor(void) +{ + harden_branch_predictor_fn_t fn = per_cpu(harden_branch_predictor_fn, + smp_processor_id()); + if (fn) + fn(); +} +#else +#define harden_branch_predictor() do { } while (0) +#endif + #define UDBG_UNDEFINED (1 << 0) #define UDBG_SYSCALL (1 << 1) #define UDBG_BADABORT (1 << 2) diff --git a/arch/arm/include/asm/uaccess.h b/arch/arm/include/asm/uaccess.h index 0bf2347495f1..3d614e90c19f 100644 --- a/arch/arm/include/asm/uaccess.h +++ b/arch/arm/include/asm/uaccess.h @@ -152,7 +152,7 @@ extern int __get_user_64t_4(void *); #define __get_user_check(x, p) \ ({ \ unsigned long __limit = current_thread_info()->addr_limit - 1; \ - register const typeof(*(p)) __user *__p asm("r0") = (p);\ + register typeof(*(p)) __user *__p asm("r0") = (p); \ register typeof(x) __r2 asm("r2"); \ register unsigned long __l asm("r1") = __limit; \ register int __e asm("r0"); \ diff --git a/arch/arm/include/asm/v7m.h b/arch/arm/include/asm/v7m.h index 634e77107425..187ccf6496ad 100644 --- a/arch/arm/include/asm/v7m.h +++ b/arch/arm/include/asm/v7m.h @@ -64,9 +64,17 @@ #define MPU_CTRL_ENABLE 1 #define MPU_CTRL_PRIVDEFENA (1 << 2) -#define MPU_RNR 0x98 -#define MPU_RBAR 0x9c -#define MPU_RASR 0xa0 +#define PMSAv7_RNR 0x98 +#define PMSAv7_RBAR 0x9c +#define PMSAv7_RASR 0xa0 + +#define PMSAv8_RNR 0x98 +#define PMSAv8_RBAR 0x9c +#define PMSAv8_RLAR 0xa0 +#define PMSAv8_RBAR_A(n) (PMSAv8_RBAR + 8*(n)) +#define PMSAv8_RLAR_A(n) (PMSAv8_RLAR + 8*(n)) +#define PMSAv8_MAIR0 0xc0 +#define PMSAv8_MAIR1 0xc4 /* Cache opeartions */ #define V7M_SCB_ICIALLU 0x250 /* I-cache invalidate all to PoU */ diff --git a/arch/arm/include/debug/brcmstb.S b/arch/arm/include/debug/brcmstb.S index c826f15d2f80..0f580caa81e5 100644 --- a/arch/arm/include/debug/brcmstb.S +++ b/arch/arm/include/debug/brcmstb.S @@ -11,20 +11,25 @@ * GNU General Public License for more details. */ #include +#include /* Physical register offset and virtual register offset */ #define REG_PHYS_BASE 0xf0000000 +#define REG_PHYS_BASE_V7 0x08000000 #define REG_VIRT_BASE 0xfc000000 #define REG_PHYS_ADDR(x) ((x) + REG_PHYS_BASE) +#define REG_PHYS_ADDR_V7(x) ((x) + REG_PHYS_BASE_V7) /* Product id can be read from here */ #define SUN_TOP_CTRL_BASE REG_PHYS_ADDR(0x404000) +#define SUN_TOP_CTRL_BASE_V7 REG_PHYS_ADDR_V7(0x404000) #define UARTA_3390 REG_PHYS_ADDR(0x40a900) #define UARTA_7250 REG_PHYS_ADDR(0x40b400) #define UARTA_7260 REG_PHYS_ADDR(0x40c000) #define UARTA_7268 UARTA_7260 #define UARTA_7271 UARTA_7268 +#define UARTA_7278 REG_PHYS_ADDR_V7(0x40c000) #define UARTA_7364 REG_PHYS_ADDR(0x40b000) #define UARTA_7366 UARTA_7364 #define UARTA_74371 REG_PHYS_ADDR(0x406b00) @@ -55,8 +60,21 @@ mov \rv, #0 @ yes; record init is done str \rv, [\tmp] + /* Check for V7 memory map if B53 */ + mrc p15, 0, \rv, c0, c0, 0 @ get Main ID register + ldr \rp, =ARM_CPU_PART_MASK + and \rv, \rv, \rp + ldr \rp, =ARM_CPU_PART_BRAHMA_B53 @ check for B53 CPU + cmp \rv, \rp + bne 10f + + /* if PERIPHBASE doesn't overlap REG_PHYS_BASE use V7 map */ + mrc p15, 1, \rv, c15, c3, 0 @ get PERIPHBASE from CBAR + ands \rv, \rv, #REG_PHYS_BASE + ldreq \rp, =SUN_TOP_CTRL_BASE_V7 + /* Check SUN_TOP_CTRL base */ - ldr \rp, =SUN_TOP_CTRL_BASE @ load SUN_TOP_CTRL PA +10: ldrne \rp, =SUN_TOP_CTRL_BASE @ load SUN_TOP_CTRL PA ldr \rv, [\rp, #0] @ get register contents ARM_BE8( rev \rv, \rv ) and \rv, \rv, #0xffffff00 @ strip revision bits [7:0] @@ -72,6 +90,7 @@ ARM_BE8( rev \rv, \rv ) 27: checkuart(\rp, \rv, 0x07437100, 74371) 28: checkuart(\rp, \rv, 0x74390000, 7439) 29: checkuart(\rp, \rv, 0x74450000, 7445) +30: checkuart(\rp, \rv, 0x72780000, 7278) /* No valid UART found */ 90: mov \rp, #0 diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h index caae4843cb70..16e006f708ca 100644 --- a/arch/arm/include/uapi/asm/kvm.h +++ b/arch/arm/include/uapi/asm/kvm.h @@ -91,6 +91,7 @@ struct kvm_regs { #define KVM_VGIC_V3_ADDR_TYPE_DIST 2 #define KVM_VGIC_V3_ADDR_TYPE_REDIST 3 #define KVM_VGIC_ITS_ADDR_TYPE 4 +#define KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION 5 #define KVM_VGIC_V3_DIST_SIZE SZ_64K #define KVM_VGIC_V3_REDIST_SIZE (2 * SZ_64K) diff --git a/arch/arm/kernel/Makefile b/arch/arm/kernel/Makefile index b59ac4bf82b8..8cad59465af3 100644 --- a/arch/arm/kernel/Makefile +++ b/arch/arm/kernel/Makefile @@ -31,6 +31,7 @@ else obj-y += entry-armv.o endif +obj-$(CONFIG_MMU) += bugs.o obj-$(CONFIG_CPU_IDLE) += cpuidle.o obj-$(CONFIG_ISA_DMA_API) += dma.o obj-$(CONFIG_FIQ) += fiq.o fiqasm.o diff --git a/arch/arm/kernel/asm-offsets.c b/arch/arm/kernel/asm-offsets.c index f369ece99958..974d8d7d1bcd 100644 --- a/arch/arm/kernel/asm-offsets.c +++ b/arch/arm/kernel/asm-offsets.c @@ -61,7 +61,7 @@ int main(void) { DEFINE(TSK_ACTIVE_MM, offsetof(struct task_struct, active_mm)); -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR DEFINE(TSK_STACK_CANARY, offsetof(struct task_struct, stack_canary)); #endif BLANK(); @@ -194,9 +194,11 @@ int main(void) DEFINE(MPU_RNG_INFO_USED, offsetof(struct mpu_rgn_info, used)); DEFINE(MPU_RNG_SIZE, sizeof(struct mpu_rgn)); - DEFINE(MPU_RGN_DRBAR, offsetof(struct mpu_rgn, drbar)); - DEFINE(MPU_RGN_DRSR, offsetof(struct mpu_rgn, drsr)); - DEFINE(MPU_RGN_DRACR, offsetof(struct mpu_rgn, dracr)); + DEFINE(MPU_RGN_DRBAR, offsetof(struct mpu_rgn, drbar)); + DEFINE(MPU_RGN_DRSR, offsetof(struct mpu_rgn, drsr)); + DEFINE(MPU_RGN_DRACR, offsetof(struct mpu_rgn, dracr)); + DEFINE(MPU_RGN_PRBAR, offsetof(struct mpu_rgn, prbar)); + DEFINE(MPU_RGN_PRLAR, offsetof(struct mpu_rgn, prlar)); #endif return 0; } diff --git a/arch/arm/kernel/bugs.c b/arch/arm/kernel/bugs.c new file mode 100644 index 000000000000..7be511310191 --- /dev/null +++ b/arch/arm/kernel/bugs.c @@ -0,0 +1,18 @@ +// SPDX-Identifier: GPL-2.0 +#include +#include +#include + +void check_other_bugs(void) +{ +#ifdef MULTI_CPU + if (processor.check_bugs) + processor.check_bugs(); +#endif +} + +void __init check_bugs(void) +{ + check_writebuffer_bugs(); + check_other_bugs(); +} diff --git a/arch/arm/kernel/dma.c b/arch/arm/kernel/dma.c index e651c4d0a0d9..6739d37c2bc5 100644 --- a/arch/arm/kernel/dma.c +++ b/arch/arm/kernel/dma.c @@ -276,21 +276,9 @@ static int proc_dma_show(struct seq_file *m, void *v) return 0; } -static int proc_dma_open(struct inode *inode, struct file *file) -{ - return single_open(file, proc_dma_show, NULL); -} - -static const struct file_operations proc_dma_operations = { - .open = proc_dma_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int __init proc_dma_init(void) { - proc_create("dma", 0, NULL, &proc_dma_operations); + proc_create_single("dma", 0, NULL, proc_dma_show); return 0; } diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 1752033b0070..179a9f6bd1e3 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -791,7 +791,7 @@ ENTRY(__switch_to) ldr r6, [r2, #TI_CPU_DOMAIN] #endif switch_tls r1, r4, r5, r3, r7 -#if defined(CONFIG_CC_STACKPROTECTOR) && !defined(CONFIG_SMP) +#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) ldr r7, [r2, #TI_TASK] ldr r8, =__stack_chk_guard .if (TSK_STACK_CANARY > IMM12_MASK) @@ -807,7 +807,7 @@ ENTRY(__switch_to) ldr r0, =thread_notify_head mov r1, #THREAD_NOTIFY_SWITCH bl atomic_notifier_call_chain -#if defined(CONFIG_CC_STACKPROTECTOR) && !defined(CONFIG_SMP) +#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) str r7, [r8] #endif THUMB( mov ip, r4 ) diff --git a/arch/arm/kernel/entry-common.S b/arch/arm/kernel/entry-common.S index 3c4f88701f22..106a1466518d 100644 --- a/arch/arm/kernel/entry-common.S +++ b/arch/arm/kernel/entry-common.S @@ -39,12 +39,13 @@ saved_pc .req lr .section .entry.text,"ax",%progbits .align 5 -#if !(IS_ENABLED(CONFIG_TRACE_IRQFLAGS) || IS_ENABLED(CONFIG_CONTEXT_TRACKING)) +#if !(IS_ENABLED(CONFIG_TRACE_IRQFLAGS) || IS_ENABLED(CONFIG_CONTEXT_TRACKING) || \ + IS_ENABLED(CONFIG_DEBUG_RSEQ)) /* * This is the fast syscall return path. We do as little as possible here, * such as avoiding writing r0 to the stack. We only use this path if we - * have tracing and context tracking disabled - the overheads from those - * features make this path too inefficient. + * have tracing, context tracking and rseq debug disabled - the overheads + * from those features make this path too inefficient. */ ret_fast_syscall: UNWIND(.fnstart ) @@ -71,14 +72,20 @@ fast_work_pending: /* fall through to work_pending */ #else /* - * The "replacement" ret_fast_syscall for when tracing or context tracking - * is enabled. As we will need to call out to some C functions, we save - * r0 first to avoid needing to save registers around each C function call. + * The "replacement" ret_fast_syscall for when tracing, context tracking, + * or rseq debug is enabled. As we will need to call out to some C functions, + * we save r0 first to avoid needing to save registers around each C function + * call. */ ret_fast_syscall: UNWIND(.fnstart ) UNWIND(.cantunwind ) str r0, [sp, #S_R0 + S_OFF]! @ save returned r0 +#if IS_ENABLED(CONFIG_DEBUG_RSEQ) + /* do_rseq_syscall needs interrupts enabled. */ + mov r0, sp @ 'regs' + bl do_rseq_syscall +#endif disable_irq_notrace @ disable interrupts ldr r2, [tsk, #TI_ADDR_LIMIT] cmp r2, #TASK_SIZE @@ -113,6 +120,12 @@ ENDPROC(ret_fast_syscall) */ ENTRY(ret_to_user) ret_slow_syscall: +#if IS_ENABLED(CONFIG_DEBUG_RSEQ) + /* do_rseq_syscall needs interrupts enabled. */ + enable_irq_notrace @ enable interrupts + mov r0, sp @ 'regs' + bl do_rseq_syscall +#endif disable_irq_notrace @ disable interrupts ENTRY(ret_to_user_from_irq) ldr r2, [tsk, #TI_ADDR_LIMIT] @@ -242,9 +255,7 @@ local_restart: tst r10, #_TIF_SYSCALL_WORK @ are we tracing syscalls? bne __sys_trace - cmp scno, #NR_syscalls @ check upper syscall limit - badr lr, ret_fast_syscall @ return address - ldrcc pc, [tbl, scno, lsl #2] @ call sys_* routine + invoke_syscall tbl, scno, r10, ret_fast_syscall add r1, sp, #S_OFF 2: cmp scno, #(__ARM_NR_BASE - __NR_SYSCALL_BASE) @@ -278,14 +289,8 @@ __sys_trace: mov r1, scno add r0, sp, #S_OFF bl syscall_trace_enter - - badr lr, __sys_trace_return @ return address - mov scno, r0 @ syscall number (possibly new) - add r1, sp, #S_R0 + S_OFF @ pointer to regs - cmp scno, #NR_syscalls @ check upper syscall limit - ldmccia r1, {r0 - r6} @ have to reload r0 - r6 - stmccia sp, {r4, r5} @ and update the stack args - ldrcc pc, [tbl, scno, lsl #2] @ call sys_* routine + mov scno, r0 + invoke_syscall tbl, scno, r10, __sys_trace_return, reload=1 cmp scno, #-1 @ skip the syscall? bne 2b add sp, sp, #S_OFF @ restore stack @@ -363,6 +368,10 @@ sys_syscall: bic scno, r0, #__NR_OABI_SYSCALL_BASE cmp scno, #__NR_syscall - __NR_SYSCALL_BASE cmpne scno, #NR_syscalls @ check range +#ifdef CONFIG_CPU_SPECTRE + movhs scno, #0 + csdb +#endif stmloia sp, {r5, r6} @ shuffle args movlo r0, r1 movlo r1, r2 diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index 0f07579af472..773424843d6e 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -378,6 +378,31 @@ #endif .endm + .macro invoke_syscall, table, nr, tmp, ret, reload=0 +#ifdef CONFIG_CPU_SPECTRE + mov \tmp, \nr + cmp \tmp, #NR_syscalls @ check upper syscall limit + movcs \tmp, #0 + csdb + badr lr, \ret @ return address + .if \reload + add r1, sp, #S_R0 + S_OFF @ pointer to regs + ldmccia r1, {r0 - r6} @ reload r0-r6 + stmccia sp, {r4, r5} @ update stack arguments + .endif + ldrcc pc, [\table, \tmp, lsl #2] @ call sys_* routine +#else + cmp \nr, #NR_syscalls @ check upper syscall limit + badr lr, \ret @ return address + .if \reload + add r1, sp, #S_R0 + S_OFF @ pointer to regs + ldmccia r1, {r0 - r6} @ reload r0-r6 + stmccia sp, {r4, r5} @ update stack arguments + .endif + ldrcc pc, [\table, \nr, lsl #2] @ call sys_* routine +#endif + .endm + /* * These are the registers used in the syscall handler, and allow us to * have in theory up to 7 arguments to a function - r0 to r6. diff --git a/arch/arm/kernel/head-nommu.S b/arch/arm/kernel/head-nommu.S index 2e38f85b757a..dd546d65a383 100644 --- a/arch/arm/kernel/head-nommu.S +++ b/arch/arm/kernel/head-nommu.S @@ -68,14 +68,6 @@ ENTRY(stext) beq __error_p @ yes, error 'p' #ifdef CONFIG_ARM_MPU - /* Calculate the size of a region covering just the kernel */ - ldr r5, =PLAT_PHYS_OFFSET @ Region start: PHYS_OFFSET - ldr r6, =(_end) @ Cover whole kernel - sub r6, r6, r5 @ Minimum size of region to map - clz r6, r6 @ Region size must be 2^N... - rsb r6, r6, #31 @ ...so round up region size - lsl r6, r6, #MPU_RSR_SZ @ Put size in right field - orr r6, r6, #(1 << MPU_RSR_EN) @ Set region enabled bit bl __setup_mpu #endif @@ -83,8 +75,8 @@ ENTRY(stext) ldr r12, [r10, #PROCINFO_INITFUNC] add r12, r12, r10 ret r12 -1: bl __after_proc_init - b __mmap_switched +1: ldr lr, =__mmap_switched + b __after_proc_init ENDPROC(stext) #ifdef CONFIG_SMP @@ -110,8 +102,6 @@ ENTRY(secondary_startup) ldr r7, __secondary_data #ifdef CONFIG_ARM_MPU - /* Use MPU region info supplied by __cpu_up */ - ldr r6, [r7] @ get secondary_data.mpu_rgn_info bl __secondary_setup_mpu @ Initialize the MPU #endif @@ -133,12 +123,45 @@ __secondary_data: /* * Set the Control Register and Read the process ID. */ + .text __after_proc_init: +#ifdef CONFIG_ARM_MPU +M_CLASS(movw r12, #:lower16:BASEADDR_V7M_SCB) +M_CLASS(movt r12, #:upper16:BASEADDR_V7M_SCB) +M_CLASS(ldr r3, [r12, 0x50]) +AR_CLASS(mrc p15, 0, r3, c0, c1, 4) @ Read ID_MMFR0 + and r3, r3, #(MMFR0_PMSA) @ PMSA field + teq r3, #(MMFR0_PMSAv7) @ PMSA v7 + beq 1f + teq r3, #(MMFR0_PMSAv8) @ PMSA v8 + /* + * Memory region attributes for PMSAv8: + * + * n = AttrIndx[2:0] + * n MAIR + * DEVICE_nGnRnE 000 00000000 + * NORMAL 001 11111111 + */ + ldreq r3, =PMSAv8_MAIR(0x00, PMSAv8_RGN_DEVICE_nGnRnE) | \ + PMSAv8_MAIR(0xff, PMSAv8_RGN_NORMAL) +AR_CLASS(mcreq p15, 0, r3, c10, c2, 0) @ MAIR 0 +M_CLASS(streq r3, [r12, #PMSAv8_MAIR0]) + moveq r3, #0 +AR_CLASS(mcreq p15, 0, r3, c10, c2, 1) @ MAIR 1 +M_CLASS(streq r3, [r12, #PMSAv8_MAIR1]) + +1: +#endif #ifdef CONFIG_CPU_CP15 /* * CP15 system control register value returned in r0 from * the CPU init function. */ + +#ifdef CONFIG_ARM_MPU + biceq r0, r0, #CR_BR @ Disable the 'default mem-map' + orreq r0, r0, #CR_M @ Set SCTRL.M (MPU on) +#endif #if defined(CONFIG_ALIGNMENT_TRAP) && __LINUX_ARM_ARCH__ < 6 orr r0, r0, #CR_A #else @@ -154,7 +177,15 @@ __after_proc_init: bic r0, r0, #CR_I #endif mcr p15, 0, r0, c1, c0, 0 @ write control reg + isb #elif defined (CONFIG_CPU_V7M) +#ifdef CONFIG_ARM_MPU + ldreq r3, [r12, MPU_CTRL] + biceq r3, #MPU_CTRL_PRIVDEFENA + orreq r3, #MPU_CTRL_ENABLE + streq r3, [r12, MPU_CTRL] + isb +#endif /* For V7M systems we want to modify the CCR similarly to the SCTLR */ #ifdef CONFIG_CPU_DCACHE_DISABLE bic r0, r0, #V7M_SCB_CCR_DC @@ -165,9 +196,7 @@ __after_proc_init: #ifdef CONFIG_CPU_ICACHE_DISABLE bic r0, r0, #V7M_SCB_CCR_IC #endif - movw r3, #:lower16:(BASEADDR_V7M_SCB + V7M_SCB_CCR) - movt r3, #:upper16:(BASEADDR_V7M_SCB + V7M_SCB_CCR) - str r0, [r3] + str r0, [r12, V7M_SCB_CCR] #endif /* CONFIG_CPU_CP15 elif CONFIG_CPU_V7M */ ret lr ENDPROC(__after_proc_init) @@ -184,7 +213,7 @@ ENDPROC(__after_proc_init) .endm /* Setup a single MPU region, either D or I side (D-side for unified) */ -.macro setup_region bar, acr, sr, side = MPU_DATA_SIDE, unused +.macro setup_region bar, acr, sr, side = PMSAv7_DATA_SIDE, unused mcr p15, 0, \bar, c6, c1, (0 + \side) @ I/DRBAR mcr p15, 0, \acr, c6, c1, (4 + \side) @ I/DRACR mcr p15, 0, \sr, c6, c1, (2 + \side) @ I/DRSR @@ -192,14 +221,14 @@ ENDPROC(__after_proc_init) #else .macro set_region_nr tmp, rgnr, base mov \tmp, \rgnr - str \tmp, [\base, #MPU_RNR] + str \tmp, [\base, #PMSAv7_RNR] .endm .macro setup_region bar, acr, sr, unused, base lsl \acr, \acr, #16 orr \acr, \acr, \sr - str \bar, [\base, #MPU_RBAR] - str \acr, [\base, #MPU_RASR] + str \bar, [\base, #PMSAv7_RBAR] + str \acr, [\base, #PMSAv7_RASR] .endm #endif @@ -210,8 +239,9 @@ ENDPROC(__after_proc_init) * Region 2: Normal, Shared, cacheable for RAM. From PHYS_OFFSET, size from r6 * Region 3: Normal, shared, inaccessible from PL0 to protect the vectors page * - * r6: Value to be written to DRSR (and IRSR if required) for MPU_RAM_REGION + * r6: Value to be written to DRSR (and IRSR if required) for PMSAv7_RAM_REGION */ + __HEAD ENTRY(__setup_mpu) @@ -223,7 +253,22 @@ AR_CLASS(mrc p15, 0, r0, c0, c1, 4) @ Read ID_MMFR0 M_CLASS(ldr r0, [r12, 0x50]) and r0, r0, #(MMFR0_PMSA) @ PMSA field teq r0, #(MMFR0_PMSAv7) @ PMSA v7 - bxne lr + beq __setup_pmsa_v7 + teq r0, #(MMFR0_PMSAv8) @ PMSA v8 + beq __setup_pmsa_v8 + + ret lr +ENDPROC(__setup_mpu) + +ENTRY(__setup_pmsa_v7) + /* Calculate the size of a region covering just the kernel */ + ldr r5, =PLAT_PHYS_OFFSET @ Region start: PHYS_OFFSET + ldr r6, =(_end) @ Cover whole kernel + sub r6, r6, r5 @ Minimum size of region to map + clz r6, r6 @ Region size must be 2^N... + rsb r6, r6, #31 @ ...so round up region size + lsl r6, r6, #PMSAv7_RSR_SZ @ Put size in right field + orr r6, r6, #(1 << PMSAv7_RSR_EN) @ Set region enabled bit /* Determine whether the D/I-side memory map is unified. We set the * flags here and continue to use them for the rest of this function */ @@ -234,77 +279,189 @@ M_CLASS(ldr r0, [r12, #MPU_TYPE]) tst r0, #MPUIR_nU @ MPUIR_nU = 0 for unified /* Setup second region first to free up r6 */ - set_region_nr r0, #MPU_RAM_REGION, r12 + set_region_nr r0, #PMSAv7_RAM_REGION, r12 isb /* Full access from PL0, PL1, shared for CONFIG_SMP, cacheable */ ldr r0, =PLAT_PHYS_OFFSET @ RAM starts at PHYS_OFFSET - ldr r5,=(MPU_AP_PL1RW_PL0RW | MPU_RGN_NORMAL) + ldr r5,=(PMSAv7_AP_PL1RW_PL0RW | PMSAv7_RGN_NORMAL) - setup_region r0, r5, r6, MPU_DATA_SIDE, r12 @ PHYS_OFFSET, shared, enabled + setup_region r0, r5, r6, PMSAv7_DATA_SIDE, r12 @ PHYS_OFFSET, shared, enabled beq 1f @ Memory-map not unified - setup_region r0, r5, r6, MPU_INSTR_SIDE, r12 @ PHYS_OFFSET, shared, enabled + setup_region r0, r5, r6, PMSAv7_INSTR_SIDE, r12 @ PHYS_OFFSET, shared, enabled 1: isb /* First/background region */ - set_region_nr r0, #MPU_BG_REGION, r12 + set_region_nr r0, #PMSAv7_BG_REGION, r12 isb /* Execute Never, strongly ordered, inaccessible to PL0, rw PL1 */ mov r0, #0 @ BG region starts at 0x0 - ldr r5,=(MPU_ACR_XN | MPU_RGN_STRONGLY_ORDERED | MPU_AP_PL1RW_PL0NA) - mov r6, #MPU_RSR_ALL_MEM @ 4GB region, enabled + ldr r5,=(PMSAv7_ACR_XN | PMSAv7_RGN_STRONGLY_ORDERED | PMSAv7_AP_PL1RW_PL0NA) + mov r6, #PMSAv7_RSR_ALL_MEM @ 4GB region, enabled - setup_region r0, r5, r6, MPU_DATA_SIDE, r12 @ 0x0, BG region, enabled + setup_region r0, r5, r6, PMSAv7_DATA_SIDE, r12 @ 0x0, BG region, enabled beq 2f @ Memory-map not unified - setup_region r0, r5, r6, MPU_INSTR_SIDE r12 @ 0x0, BG region, enabled + setup_region r0, r5, r6, PMSAv7_INSTR_SIDE r12 @ 0x0, BG region, enabled 2: isb #ifdef CONFIG_XIP_KERNEL - set_region_nr r0, #MPU_ROM_REGION, r12 + set_region_nr r0, #PMSAv7_ROM_REGION, r12 isb - ldr r5,=(MPU_AP_PL1RO_PL0NA | MPU_RGN_NORMAL) + ldr r5,=(PMSAv7_AP_PL1RO_PL0NA | PMSAv7_RGN_NORMAL) ldr r0, =CONFIG_XIP_PHYS_ADDR @ ROM start ldr r6, =(_exiprom) @ ROM end sub r6, r6, r0 @ Minimum size of region to map clz r6, r6 @ Region size must be 2^N... rsb r6, r6, #31 @ ...so round up region size - lsl r6, r6, #MPU_RSR_SZ @ Put size in right field - orr r6, r6, #(1 << MPU_RSR_EN) @ Set region enabled bit + lsl r6, r6, #PMSAv7_RSR_SZ @ Put size in right field + orr r6, r6, #(1 << PMSAv7_RSR_EN) @ Set region enabled bit - setup_region r0, r5, r6, MPU_DATA_SIDE, r12 @ XIP_PHYS_ADDR, shared, enabled + setup_region r0, r5, r6, PMSAv7_DATA_SIDE, r12 @ XIP_PHYS_ADDR, shared, enabled beq 3f @ Memory-map not unified - setup_region r0, r5, r6, MPU_INSTR_SIDE, r12 @ XIP_PHYS_ADDR, shared, enabled + setup_region r0, r5, r6, PMSAv7_INSTR_SIDE, r12 @ XIP_PHYS_ADDR, shared, enabled 3: isb #endif + ret lr +ENDPROC(__setup_pmsa_v7) + +ENTRY(__setup_pmsa_v8) + mov r0, #0 +AR_CLASS(mcr p15, 0, r0, c6, c2, 1) @ PRSEL +M_CLASS(str r0, [r12, #PMSAv8_RNR]) + isb + +#ifdef CONFIG_XIP_KERNEL + ldr r5, =CONFIG_XIP_PHYS_ADDR @ ROM start + ldr r6, =(_exiprom) @ ROM end + sub r6, r6, #1 + bic r6, r6, #(PMSAv8_MINALIGN - 1) + + orr r5, r5, #(PMSAv8_AP_PL1RW_PL0NA | PMSAv8_RGN_SHARED) + orr r6, r6, #(PMSAv8_LAR_IDX(PMSAv8_RGN_NORMAL) | PMSAv8_LAR_EN) + +AR_CLASS(mcr p15, 0, r5, c6, c8, 0) @ PRBAR0 +AR_CLASS(mcr p15, 0, r6, c6, c8, 1) @ PRLAR0 +M_CLASS(str r5, [r12, #PMSAv8_RBAR_A(0)]) +M_CLASS(str r6, [r12, #PMSAv8_RLAR_A(0)]) +#endif + + ldr r5, =KERNEL_START + ldr r6, =KERNEL_END + sub r6, r6, #1 + bic r6, r6, #(PMSAv8_MINALIGN - 1) + + orr r5, r5, #(PMSAv8_AP_PL1RW_PL0NA | PMSAv8_RGN_SHARED) + orr r6, r6, #(PMSAv8_LAR_IDX(PMSAv8_RGN_NORMAL) | PMSAv8_LAR_EN) + +AR_CLASS(mcr p15, 0, r5, c6, c8, 4) @ PRBAR1 +AR_CLASS(mcr p15, 0, r6, c6, c8, 5) @ PRLAR1 +M_CLASS(str r5, [r12, #PMSAv8_RBAR_A(1)]) +M_CLASS(str r6, [r12, #PMSAv8_RLAR_A(1)]) + + /* Setup Background: 0x0 - min(KERNEL_START, XIP_PHYS_ADDR) */ +#ifdef CONFIG_XIP_KERNEL + ldr r6, =KERNEL_START + ldr r5, =CONFIG_XIP_PHYS_ADDR + cmp r6, r5 + movcs r6, r5 +#else + ldr r6, =KERNEL_START +#endif + cmp r6, #0 + beq 1f + + mov r5, #0 + sub r6, r6, #1 + bic r6, r6, #(PMSAv8_MINALIGN - 1) + + orr r5, r5, #(PMSAv8_AP_PL1RW_PL0NA | PMSAv8_RGN_SHARED | PMSAv8_BAR_XN) + orr r6, r6, #(PMSAv8_LAR_IDX(PMSAv8_RGN_DEVICE_nGnRnE) | PMSAv8_LAR_EN) + +AR_CLASS(mcr p15, 0, r5, c6, c9, 0) @ PRBAR2 +AR_CLASS(mcr p15, 0, r6, c6, c9, 1) @ PRLAR2 +M_CLASS(str r5, [r12, #PMSAv8_RBAR_A(2)]) +M_CLASS(str r6, [r12, #PMSAv8_RLAR_A(2)]) + +1: + /* Setup Background: max(KERNEL_END, _exiprom) - 0xffffffff */ +#ifdef CONFIG_XIP_KERNEL + ldr r5, =KERNEL_END + ldr r6, =(_exiprom) + cmp r5, r6 + movcc r5, r6 +#else + ldr r5, =KERNEL_END +#endif + mov r6, #0xffffffff + bic r6, r6, #(PMSAv8_MINALIGN - 1) + + orr r5, r5, #(PMSAv8_AP_PL1RW_PL0NA | PMSAv8_RGN_SHARED | PMSAv8_BAR_XN) + orr r6, r6, #(PMSAv8_LAR_IDX(PMSAv8_RGN_DEVICE_nGnRnE) | PMSAv8_LAR_EN) - /* Enable the MPU */ -AR_CLASS(mrc p15, 0, r0, c1, c0, 0) @ Read SCTLR -AR_CLASS(bic r0, r0, #CR_BR) @ Disable the 'default mem-map' -AR_CLASS(orr r0, r0, #CR_M) @ Set SCTRL.M (MPU on) -AR_CLASS(mcr p15, 0, r0, c1, c0, 0) @ Enable MPU +AR_CLASS(mcr p15, 0, r5, c6, c9, 4) @ PRBAR3 +AR_CLASS(mcr p15, 0, r6, c6, c9, 5) @ PRLAR3 +M_CLASS(str r5, [r12, #PMSAv8_RBAR_A(3)]) +M_CLASS(str r6, [r12, #PMSAv8_RLAR_A(3)]) -M_CLASS(ldr r0, [r12, #MPU_CTRL]) -M_CLASS(bic r0, #MPU_CTRL_PRIVDEFENA) -M_CLASS(orr r0, #MPU_CTRL_ENABLE) -M_CLASS(str r0, [r12, #MPU_CTRL]) +#ifdef CONFIG_XIP_KERNEL + /* Setup Background: min(_exiprom, KERNEL_END) - max(KERNEL_START, XIP_PHYS_ADDR) */ + ldr r5, =(_exiprom) + ldr r6, =KERNEL_END + cmp r5, r6 + movcs r5, r6 + + ldr r6, =KERNEL_START + ldr r0, =CONFIG_XIP_PHYS_ADDR + cmp r6, r0 + movcc r6, r0 + + sub r6, r6, #1 + bic r6, r6, #(PMSAv8_MINALIGN - 1) + + orr r5, r5, #(PMSAv8_AP_PL1RW_PL0NA | PMSAv8_RGN_SHARED | PMSAv8_BAR_XN) + orr r6, r6, #(PMSAv8_LAR_IDX(PMSAv8_RGN_DEVICE_nGnRnE) | PMSAv8_LAR_EN) + +#ifdef CONFIG_CPU_V7M + /* There is no alias for n == 4 */ + mov r0, #4 + str r0, [r12, #PMSAv8_RNR] @ PRSEL isb + str r5, [r12, #PMSAv8_RBAR_A(0)] + str r6, [r12, #PMSAv8_RLAR_A(0)] +#else + mcr p15, 0, r5, c6, c10, 1 @ PRBAR4 + mcr p15, 0, r6, c6, c10, 2 @ PRLAR4 +#endif +#endif ret lr -ENDPROC(__setup_mpu) +ENDPROC(__setup_pmsa_v8) #ifdef CONFIG_SMP /* * r6: pointer at mpu_rgn_info */ + .text ENTRY(__secondary_setup_mpu) + /* Use MPU region info supplied by __cpu_up */ + ldr r6, [r7] @ get secondary_data.mpu_rgn_info + /* Probe for v7 PMSA compliance */ mrc p15, 0, r0, c0, c1, 4 @ Read ID_MMFR0 and r0, r0, #(MMFR0_PMSA) @ PMSA field teq r0, #(MMFR0_PMSAv7) @ PMSA v7 - bne __error_p + beq __secondary_setup_pmsa_v7 + teq r0, #(MMFR0_PMSAv8) @ PMSA v8 + beq __secondary_setup_pmsa_v8 + b __error_p +ENDPROC(__secondary_setup_mpu) +/* + * r6: pointer at mpu_rgn_info + */ +ENTRY(__secondary_setup_pmsa_v7) /* Determine whether the D/I-side memory map is unified. We set the * flags here and continue to use them for the rest of this function */ mrc p15, 0, r0, c0, c0, 4 @ MPUIR @@ -328,25 +485,45 @@ ENTRY(__secondary_setup_mpu) ldr r6, [r3, #MPU_RGN_DRSR] ldr r5, [r3, #MPU_RGN_DRACR] - setup_region r0, r5, r6, MPU_DATA_SIDE + setup_region r0, r5, r6, PMSAv7_DATA_SIDE beq 2f - setup_region r0, r5, r6, MPU_INSTR_SIDE + setup_region r0, r5, r6, PMSAv7_INSTR_SIDE 2: isb mrc p15, 0, r0, c0, c0, 4 @ Reevaluate the MPUIR cmp r4, #0 bgt 1b - /* Enable the MPU */ - mrc p15, 0, r0, c1, c0, 0 @ Read SCTLR - bic r0, r0, #CR_BR @ Disable the 'default mem-map' - orr r0, r0, #CR_M @ Set SCTRL.M (MPU on) - mcr p15, 0, r0, c1, c0, 0 @ Enable MPU + ret lr +ENDPROC(__secondary_setup_pmsa_v7) + +ENTRY(__secondary_setup_pmsa_v8) + ldr r4, [r6, #MPU_RNG_INFO_USED] +#ifndef CONFIG_XIP_KERNEL + add r4, r4, #1 +#endif + mov r5, #MPU_RNG_SIZE + add r3, r6, #MPU_RNG_INFO_RNGS + mla r3, r4, r5, r3 + +1: + sub r3, r3, #MPU_RNG_SIZE + sub r4, r4, #1 + + mcr p15, 0, r4, c6, c2, 1 @ PRSEL isb - ret lr -ENDPROC(__secondary_setup_mpu) + ldr r5, [r3, #MPU_RGN_PRBAR] + ldr r6, [r3, #MPU_RGN_PRLAR] + mcr p15, 0, r5, c6, c3, 0 @ PRBAR + mcr p15, 0, r6, c6, c3, 1 @ PRLAR + + cmp r4, #0 + bgt 1b + + ret lr +ENDPROC(__secondary_setup_pmsa_v8) #endif /* CONFIG_SMP */ #endif /* CONFIG_ARM_MPU */ #include "head-common.S" diff --git a/arch/arm/kernel/perf_event_v6.c b/arch/arm/kernel/perf_event_v6.c index 1d7061a38922..be42c4f66a40 100644 --- a/arch/arm/kernel/perf_event_v6.c +++ b/arch/arm/kernel/perf_event_v6.c @@ -303,12 +303,10 @@ static void armv6pmu_enable_event(struct perf_event *event) } static irqreturn_t -armv6pmu_handle_irq(int irq_num, - void *dev) +armv6pmu_handle_irq(struct arm_pmu *cpu_pmu) { unsigned long pmcr = armv6_pmcr_read(); struct perf_sample_data data; - struct arm_pmu *cpu_pmu = (struct arm_pmu *)dev; struct pmu_hw_events *cpuc = this_cpu_ptr(cpu_pmu->hw_events); struct pt_regs *regs; int idx; diff --git a/arch/arm/kernel/perf_event_v7.c b/arch/arm/kernel/perf_event_v7.c index 870b66c1e4ef..57f01e059f39 100644 --- a/arch/arm/kernel/perf_event_v7.c +++ b/arch/arm/kernel/perf_event_v7.c @@ -946,11 +946,10 @@ static void armv7pmu_disable_event(struct perf_event *event) raw_spin_unlock_irqrestore(&events->pmu_lock, flags); } -static irqreturn_t armv7pmu_handle_irq(int irq_num, void *dev) +static irqreturn_t armv7pmu_handle_irq(struct arm_pmu *cpu_pmu) { u32 pmnc; struct perf_sample_data data; - struct arm_pmu *cpu_pmu = (struct arm_pmu *)dev; struct pmu_hw_events *cpuc = this_cpu_ptr(cpu_pmu->hw_events); struct pt_regs *regs; int idx; diff --git a/arch/arm/kernel/perf_event_xscale.c b/arch/arm/kernel/perf_event_xscale.c index fcf218da660e..88d1a76f5367 100644 --- a/arch/arm/kernel/perf_event_xscale.c +++ b/arch/arm/kernel/perf_event_xscale.c @@ -142,11 +142,10 @@ xscale1_pmnc_counter_has_overflowed(unsigned long pmnc, } static irqreturn_t -xscale1pmu_handle_irq(int irq_num, void *dev) +xscale1pmu_handle_irq(struct arm_pmu *cpu_pmu) { unsigned long pmnc; struct perf_sample_data data; - struct arm_pmu *cpu_pmu = (struct arm_pmu *)dev; struct pmu_hw_events *cpuc = this_cpu_ptr(cpu_pmu->hw_events); struct pt_regs *regs; int idx; @@ -489,11 +488,10 @@ xscale2_pmnc_counter_has_overflowed(unsigned long of_flags, } static irqreturn_t -xscale2pmu_handle_irq(int irq_num, void *dev) +xscale2pmu_handle_irq(struct arm_pmu *cpu_pmu) { unsigned long pmnc, of_flags; struct perf_sample_data data; - struct arm_pmu *cpu_pmu = (struct arm_pmu *)dev; struct pmu_hw_events *cpuc = this_cpu_ptr(cpu_pmu->hw_events); struct pt_regs *regs; int idx; diff --git a/arch/arm/kernel/process.c b/arch/arm/kernel/process.c index 1523cb18b109..225d1c58d2de 100644 --- a/arch/arm/kernel/process.c +++ b/arch/arm/kernel/process.c @@ -39,7 +39,7 @@ #include #include -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR #include unsigned long __stack_chk_guard __read_mostly; EXPORT_SYMBOL(__stack_chk_guard); diff --git a/arch/arm/kernel/ptrace.c b/arch/arm/kernel/ptrace.c index 7724b0f661b3..36718a424358 100644 --- a/arch/arm/kernel/ptrace.c +++ b/arch/arm/kernel/ptrace.c @@ -205,6 +205,7 @@ void ptrace_break(struct task_struct *tsk, struct pt_regs *regs) { siginfo_t info; + clear_siginfo(&info); info.si_signo = SIGTRAP; info.si_errno = 0; info.si_code = TRAP_BRKPT; diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c index fc40a2b40595..35ca494c028c 100644 --- a/arch/arm/kernel/setup.c +++ b/arch/arm/kernel/setup.c @@ -754,7 +754,7 @@ int __init arm_add_memory(u64 start, u64 size) else size -= aligned_start - start; -#ifndef CONFIG_ARCH_PHYS_ADDR_T_64BIT +#ifndef CONFIG_PHYS_ADDR_T_64BIT if (aligned_start > ULONG_MAX) { pr_crit("Ignoring memory at 0x%08llx outside 32-bit physical address space\n", (long long)start); diff --git a/arch/arm/kernel/signal.c b/arch/arm/kernel/signal.c index bd8810d4acb3..f09e9d66d605 100644 --- a/arch/arm/kernel/signal.c +++ b/arch/arm/kernel/signal.c @@ -540,6 +540,12 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) sigset_t *oldset = sigmask_to_save(); int ret; + /* + * Increment event counter and perform fixup for the pre-signal + * frame. + */ + rseq_signal_deliver(regs); + /* * Set up the stack frame */ @@ -660,6 +666,7 @@ do_work_pending(struct pt_regs *regs, unsigned int thread_flags, int syscall) } else { clear_thread_flag(TIF_NOTIFY_RESUME); tracehook_notify_resume(regs); + rseq_handle_notify_resume(regs); } } local_irq_disable(); @@ -703,3 +710,10 @@ asmlinkage void addr_limit_check_failed(void) { addr_limit_user_check(); } + +#ifdef CONFIG_DEBUG_RSEQ +asmlinkage void do_rseq_syscall(struct pt_regs *regs) +{ + rseq_syscall(regs); +} +#endif diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index 2da087926ebe..0978282d5fc2 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -31,6 +31,7 @@ #include #include +#include #include #include #include @@ -236,8 +237,6 @@ int __cpu_disable(void) flush_cache_louis(); local_flush_tlb_all(); - clear_tasks_mm_cpumask(cpu); - return 0; } @@ -255,6 +254,7 @@ void __cpu_die(unsigned int cpu) } pr_debug("CPU%u: shutdown\n", cpu); + clear_tasks_mm_cpumask(cpu); /* * platform_cpu_kill() is generally expected to do the powering off * and/or cutting of clocks to the dying CPU. Optionally, this may @@ -405,6 +405,9 @@ asmlinkage void secondary_start_kernel(void) * before we continue - which happens after __cpu_up returns. */ set_cpu_online(cpu, true); + + check_other_bugs(); + complete(&cpu_running); local_irq_enable(); diff --git a/arch/arm/kernel/suspend.c b/arch/arm/kernel/suspend.c index a40ebb7c0896..d08099269e35 100644 --- a/arch/arm/kernel/suspend.c +++ b/arch/arm/kernel/suspend.c @@ -3,6 +3,7 @@ #include #include +#include #include #include #include @@ -36,6 +37,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) cpu_switch_mm(mm->pgd, mm); local_flush_bp_all(); local_flush_tlb_all(); + check_other_bugs(); } return ret; diff --git a/arch/arm/kernel/swp_emulate.c b/arch/arm/kernel/swp_emulate.c index 3bda08bee674..80517f293eb9 100644 --- a/arch/arm/kernel/swp_emulate.c +++ b/arch/arm/kernel/swp_emulate.c @@ -91,18 +91,6 @@ static int proc_status_show(struct seq_file *m, void *v) seq_printf(m, "Last process:\t\t%d\n", previous_pid); return 0; } - -static int proc_status_open(struct inode *inode, struct file *file) -{ - return single_open(file, proc_status_show, PDE_DATA(inode)); -} - -static const struct file_operations proc_status_fops = { - .open = proc_status_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; #endif /* @@ -112,6 +100,7 @@ static void set_segfault(struct pt_regs *regs, unsigned long addr) { siginfo_t info; + clear_siginfo(&info); down_read(¤t->mm->mmap_sem); if (find_vma(current->mm, addr) == NULL) info.si_code = SEGV_MAPERR; @@ -260,7 +249,8 @@ static int __init swp_emulation_init(void) return 0; #ifdef CONFIG_PROC_FS - if (!proc_create("cpu/swp_emulation", S_IRUGO, NULL, &proc_status_fops)) + if (!proc_create_single("cpu/swp_emulation", S_IRUGO, NULL, + proc_status_show)) return -ENOMEM; #endif /* CONFIG_PROC_FS */ diff --git a/arch/arm/kernel/sys_oabi-compat.c b/arch/arm/kernel/sys_oabi-compat.c index b9786f491873..1df21a61e379 100644 --- a/arch/arm/kernel/sys_oabi-compat.c +++ b/arch/arm/kernel/sys_oabi-compat.c @@ -286,7 +286,7 @@ asmlinkage long sys_oabi_epoll_wait(int epfd, return -EINVAL; if (!access_ok(VERIFY_WRITE, events, sizeof(*events) * maxevents)) return -EFAULT; - kbuf = kmalloc(sizeof(*kbuf) * maxevents, GFP_KERNEL); + kbuf = kmalloc_array(maxevents, sizeof(*kbuf), GFP_KERNEL); if (!kbuf) return -ENOMEM; fs = get_fs(); @@ -324,7 +324,7 @@ asmlinkage long sys_oabi_semtimedop(int semid, return -EINVAL; if (!access_ok(VERIFY_READ, tsops, sizeof(*tsops) * nsops)) return -EFAULT; - sops = kmalloc(sizeof(*sops) * nsops, GFP_KERNEL); + sops = kmalloc_array(nsops, sizeof(*sops), GFP_KERNEL); if (!sops) return -ENOMEM; err = 0; diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 2fe87109ae46..badf02ca3693 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -441,6 +441,7 @@ asmlinkage void do_undefinstr(struct pt_regs *regs) siginfo_t info; void __user *pc; + clear_siginfo(&info); pc = (void __user *)instruction_pointer(regs); if (processor_mode(regs) == SVC_MODE) { @@ -540,6 +541,7 @@ static int bad_syscall(int n, struct pt_regs *regs) { siginfo_t info; + clear_siginfo(&info); if ((current->personality & PER_MASK) != PER_LINUX) { send_sig(SIGSEGV, current, 1); return regs->ARM_r0; @@ -607,6 +609,7 @@ asmlinkage int arm_syscall(int no, struct pt_regs *regs) { siginfo_t info; + clear_siginfo(&info); if ((no >> 16) != (__ARM_NR_BASE>> 16)) return bad_syscall(no, regs); @@ -743,6 +746,8 @@ baddataabort(int code, unsigned long instr, struct pt_regs *regs) unsigned long addr = instruction_pointer(regs); siginfo_t info; + clear_siginfo(&info); + #ifdef CONFIG_DEBUG_USER if (user_debug & UDBG_BADABORT) { pr_err("[%d] %s: bad data abort: code %d instr 0x%08lx\n", diff --git a/arch/arm/kernel/vmlinux-xip.lds.S b/arch/arm/kernel/vmlinux-xip.lds.S index d32f5d35f602..3593d5c1acd2 100644 --- a/arch/arm/kernel/vmlinux-xip.lds.S +++ b/arch/arm/kernel/vmlinux-xip.lds.S @@ -13,6 +13,7 @@ #include #include #include +#include #include #include "vmlinux.lds.h" @@ -148,6 +149,9 @@ SECTIONS __init_end = .; BSS_SECTION(0, 0, 8) +#ifdef CONFIG_ARM_MPU + . = ALIGN(PMSAv8_MINALIGN); +#endif _end = .; STABS_DEBUG diff --git a/arch/arm/kernel/vmlinux.lds.S b/arch/arm/kernel/vmlinux.lds.S index b77dc675ae55..23150c0f0f4d 100644 --- a/arch/arm/kernel/vmlinux.lds.S +++ b/arch/arm/kernel/vmlinux.lds.S @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -54,6 +55,9 @@ SECTIONS . = ALIGN(1< #include #include #include @@ -71,6 +72,90 @@ __kvm_hyp_vector: W(b) hyp_irq W(b) hyp_fiq +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR + .align 5 +__kvm_hyp_vector_ic_inv: + .global __kvm_hyp_vector_ic_inv + + /* + * We encode the exception entry in the bottom 3 bits of + * SP, and we have to guarantee to be 8 bytes aligned. + */ + W(add) sp, sp, #1 /* Reset 7 */ + W(add) sp, sp, #1 /* Undef 6 */ + W(add) sp, sp, #1 /* Syscall 5 */ + W(add) sp, sp, #1 /* Prefetch abort 4 */ + W(add) sp, sp, #1 /* Data abort 3 */ + W(add) sp, sp, #1 /* HVC 2 */ + W(add) sp, sp, #1 /* IRQ 1 */ + W(nop) /* FIQ 0 */ + + mcr p15, 0, r0, c7, c5, 0 /* ICIALLU */ + isb + + b decode_vectors + + .align 5 +__kvm_hyp_vector_bp_inv: + .global __kvm_hyp_vector_bp_inv + + /* + * We encode the exception entry in the bottom 3 bits of + * SP, and we have to guarantee to be 8 bytes aligned. + */ + W(add) sp, sp, #1 /* Reset 7 */ + W(add) sp, sp, #1 /* Undef 6 */ + W(add) sp, sp, #1 /* Syscall 5 */ + W(add) sp, sp, #1 /* Prefetch abort 4 */ + W(add) sp, sp, #1 /* Data abort 3 */ + W(add) sp, sp, #1 /* HVC 2 */ + W(add) sp, sp, #1 /* IRQ 1 */ + W(nop) /* FIQ 0 */ + + mcr p15, 0, r0, c7, c5, 6 /* BPIALL */ + isb + +decode_vectors: + +#ifdef CONFIG_THUMB2_KERNEL + /* + * Yet another silly hack: Use VPIDR as a temp register. + * Thumb2 is really a pain, as SP cannot be used with most + * of the bitwise instructions. The vect_br macro ensures + * things gets cleaned-up. + */ + mcr p15, 4, r0, c0, c0, 0 /* VPIDR */ + mov r0, sp + and r0, r0, #7 + sub sp, sp, r0 + push {r1, r2} + mov r1, r0 + mrc p15, 4, r0, c0, c0, 0 /* VPIDR */ + mrc p15, 0, r2, c0, c0, 0 /* MIDR */ + mcr p15, 4, r2, c0, c0, 0 /* VPIDR */ +#endif + +.macro vect_br val, targ +ARM( eor sp, sp, #\val ) +ARM( tst sp, #7 ) +ARM( eorne sp, sp, #\val ) + +THUMB( cmp r1, #\val ) +THUMB( popeq {r1, r2} ) + + beq \targ +.endm + + vect_br 0, hyp_fiq + vect_br 1, hyp_irq + vect_br 2, hyp_hvc + vect_br 3, hyp_dabt + vect_br 4, hyp_pabt + vect_br 5, hyp_svc + vect_br 6, hyp_undef + vect_br 7, hyp_reset +#endif + .macro invalid_vector label, cause .align \label: mov r0, #\cause @@ -118,7 +203,7 @@ hyp_hvc: lsr r2, r2, #16 and r2, r2, #0xff cmp r2, #0 - bne guest_trap @ Guest called HVC + bne guest_hvc_trap @ Guest called HVC /* * Getting here means host called HVC, we shift parameters and branch @@ -149,7 +234,14 @@ hyp_hvc: bx ip 1: - push {lr} + /* + * Pushing r2 here is just a way of keeping the stack aligned to + * 8 bytes on any path that can trigger a HYP exception. Here, + * we may well be about to jump into the guest, and the guest + * exit would otherwise be badly decoded by our fancy + * "decode-exception-without-a-branch" code... + */ + push {r2, lr} mov lr, r0 mov r0, r1 @@ -159,7 +251,21 @@ hyp_hvc: THUMB( orr lr, #1) blx lr @ Call the HYP function - pop {lr} + pop {r2, lr} + eret + +guest_hvc_trap: + movw r2, #:lower16:ARM_SMCCC_ARCH_WORKAROUND_1 + movt r2, #:upper16:ARM_SMCCC_ARCH_WORKAROUND_1 + ldr r0, [sp] @ Guest's r0 + teq r0, r2 + bne guest_trap + add sp, sp, #12 + @ Returns: + @ r0 = 0 + @ r1 = HSR value (perfectly predictable) + @ r2 = ARM_SMCCC_ARCH_WORKAROUND_1 + mov r0, #0 eret guest_trap: diff --git a/arch/arm/mach-axxia/Kconfig b/arch/arm/mach-axxia/Kconfig index bb2ce1c63fd9..d3eae6037913 100644 --- a/arch/arm/mach-axxia/Kconfig +++ b/arch/arm/mach-axxia/Kconfig @@ -2,7 +2,6 @@ config ARCH_AXXIA bool "LSI Axxia platforms" depends on ARCH_MULTI_V7 && ARM_LPAE - select ARCH_DMA_ADDR_T_64BIT select ARM_AMBA select ARM_GIC select ARM_TIMER_SP804 diff --git a/arch/arm/mach-bcm/Kconfig b/arch/arm/mach-bcm/Kconfig index c2f3b0d216a4..c46a728df44e 100644 --- a/arch/arm/mach-bcm/Kconfig +++ b/arch/arm/mach-bcm/Kconfig @@ -211,7 +211,6 @@ config ARCH_BRCMSTB select BRCMSTB_L2_IRQ select BCM7120_L2_IRQ select ARCH_HAS_HOLES_MEMORYMODEL - select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE select ZONE_DMA if ARM_LPAE select SOC_BRCMSTB select SOC_BUS diff --git a/arch/arm/mach-berlin/Kconfig b/arch/arm/mach-berlin/Kconfig index 63ab1d368625..3d719cf645e3 100644 --- a/arch/arm/mach-berlin/Kconfig +++ b/arch/arm/mach-berlin/Kconfig @@ -23,8 +23,12 @@ config MACH_BERLIN_BG2 config MACH_BERLIN_BG2CD bool "Marvell Armada 1500-mini (BG2CD)" + select ARM_ERRATA_754322 + select ARM_ERRATA_775420 + select ARM_GLOBAL_TIMER select CACHE_L2X0 - select HAVE_ARM_TWD if SMP + select HAVE_ARM_SCU + select HAVE_ARM_TWD select PINCTRL_BERLIN_BG2CD config MACH_BERLIN_BG2Q diff --git a/arch/arm/mach-berlin/berlin.c b/arch/arm/mach-berlin/berlin.c index ac181c6797ee..2424ad40190c 100644 --- a/arch/arm/mach-berlin/berlin.c +++ b/arch/arm/mach-berlin/berlin.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Device Tree support for Marvell Berlin SoCs. * @@ -5,10 +6,6 @@ * * based on GPL'ed 2.6 kernel sources * (c) Marvell International Ltd. - * - * This file is licensed under the terms of the GNU General Public - * License version 2. This program is licensed "as is" without any - * warranty of any kind, whether express or implied. */ #include diff --git a/arch/arm/mach-berlin/headsmp.S b/arch/arm/mach-berlin/headsmp.S index dc82a3486b05..3057885d9772 100644 --- a/arch/arm/mach-berlin/headsmp.S +++ b/arch/arm/mach-berlin/headsmp.S @@ -1,11 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (C) 2014 Marvell Technology Group Ltd. * * Antoine Ténart - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. */ #include diff --git a/arch/arm/mach-berlin/platsmp.c b/arch/arm/mach-berlin/platsmp.c index 7586b7aec272..593fc4a69d84 100644 --- a/arch/arm/mach-berlin/platsmp.c +++ b/arch/arm/mach-berlin/platsmp.c @@ -1,11 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (C) 2014 Marvell Technology Group Ltd. * * Antoine Ténart - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. */ #include @@ -81,7 +78,6 @@ static void __init berlin_smp_prepare_cpus(unsigned int max_cpus) goto unmap_scu; scu_enable(scu_base); - flush_cache_all(); /* * Write the first instruction the CPU will execute after being reset diff --git a/arch/arm/mach-davinci/aemif.c b/arch/arm/mach-davinci/aemif.c index ff8b7e76b6e9..e4ab3f3a2a1f 100644 --- a/arch/arm/mach-davinci/aemif.c +++ b/arch/arm/mach-davinci/aemif.c @@ -189,7 +189,7 @@ int davinci_aemif_setup(struct platform_device *pdev) * Setup Async configuration register in case we did not boot * from NAND and so bootloader did not bother to set it up. */ - val = davinci_aemif_readl(base, A1CR_OFFSET + pdev->id * 4); + val = davinci_aemif_readl(base, A1CR_OFFSET + pdata->core_chipsel * 4); /* * Extended Wait is not valid and Select Strobe mode is not * used @@ -198,13 +198,13 @@ int davinci_aemif_setup(struct platform_device *pdev) if (pdata->options & NAND_BUSWIDTH_16) val |= 0x1; - davinci_aemif_writel(base, A1CR_OFFSET + pdev->id * 4, val); + davinci_aemif_writel(base, A1CR_OFFSET + pdata->core_chipsel * 4, val); clkrate = clk_get_rate(clk); if (pdata->timing) - ret = davinci_aemif_setup_timing(pdata->timing, base, pdev->id, - clkrate); + ret = davinci_aemif_setup_timing(pdata->timing, base, + pdata->core_chipsel, clkrate); if (ret < 0) dev_dbg(&pdev->dev, "NAND timing values setup fail\n"); diff --git a/arch/arm/mach-davinci/board-da830-evm.c b/arch/arm/mach-davinci/board-da830-evm.c index d1e8ce7b4bd2..14a6fc061744 100644 --- a/arch/arm/mach-davinci/board-da830-evm.c +++ b/arch/arm/mach-davinci/board-da830-evm.c @@ -315,6 +315,7 @@ static struct davinci_aemif_timing da830_evm_nandflash_timing = { }; static struct davinci_nand_pdata da830_evm_nand_pdata = { + .core_chipsel = 1, .parts = da830_evm_nand_partitions, .nr_parts = ARRAY_SIZE(da830_evm_nand_partitions), .ecc_mode = NAND_ECC_HW, diff --git a/arch/arm/mach-davinci/board-da850-evm.c b/arch/arm/mach-davinci/board-da850-evm.c index 158ed9a1483f..e22fb40e34bc 100644 --- a/arch/arm/mach-davinci/board-da850-evm.c +++ b/arch/arm/mach-davinci/board-da850-evm.c @@ -244,6 +244,7 @@ static struct davinci_aemif_timing da850_evm_nandflash_timing = { }; static struct davinci_nand_pdata da850_evm_nandflash_data = { + .core_chipsel = 1, .parts = da850_evm_nandflash_partition, .nr_parts = ARRAY_SIZE(da850_evm_nandflash_partition), .ecc_mode = NAND_ECC_HW, diff --git a/arch/arm/mach-davinci/board-dm355-evm.c b/arch/arm/mach-davinci/board-dm355-evm.c index 23ab9e8bc04c..a3377f959444 100644 --- a/arch/arm/mach-davinci/board-dm355-evm.c +++ b/arch/arm/mach-davinci/board-dm355-evm.c @@ -78,6 +78,7 @@ static struct mtd_partition davinci_nand_partitions[] = { }; static struct davinci_nand_pdata davinci_nand_data = { + .core_chipsel = 0, .mask_chipsel = BIT(14), .parts = davinci_nand_partitions, .nr_parts = ARRAY_SIZE(davinci_nand_partitions), diff --git a/arch/arm/mach-davinci/board-dm355-leopard.c b/arch/arm/mach-davinci/board-dm355-leopard.c index 59743bd76793..8249a0bf69f0 100644 --- a/arch/arm/mach-davinci/board-dm355-leopard.c +++ b/arch/arm/mach-davinci/board-dm355-leopard.c @@ -72,6 +72,7 @@ static struct mtd_partition davinci_nand_partitions[] = { }; static struct davinci_nand_pdata davinci_nand_data = { + .core_chipsel = 0, .mask_chipsel = BIT(14), .parts = davinci_nand_partitions, .nr_parts = ARRAY_SIZE(davinci_nand_partitions), diff --git a/arch/arm/mach-davinci/board-dm365-evm.c b/arch/arm/mach-davinci/board-dm365-evm.c index 0ac085b58a2b..435f7ec7d9af 100644 --- a/arch/arm/mach-davinci/board-dm365-evm.c +++ b/arch/arm/mach-davinci/board-dm365-evm.c @@ -138,6 +138,7 @@ static struct mtd_partition davinci_nand_partitions[] = { }; static struct davinci_nand_pdata davinci_nand_data = { + .core_chipsel = 0, .mask_chipsel = BIT(14), .parts = davinci_nand_partitions, .nr_parts = ARRAY_SIZE(davinci_nand_partitions), diff --git a/arch/arm/mach-davinci/board-dm644x-evm.c b/arch/arm/mach-davinci/board-dm644x-evm.c index 509e64ab1994..48436f74fd71 100644 --- a/arch/arm/mach-davinci/board-dm644x-evm.c +++ b/arch/arm/mach-davinci/board-dm644x-evm.c @@ -153,6 +153,7 @@ static struct davinci_aemif_timing davinci_evm_nandflash_timing = { }; static struct davinci_nand_pdata davinci_evm_nandflash_data = { + .core_chipsel = 0, .parts = davinci_evm_nandflash_partition, .nr_parts = ARRAY_SIZE(davinci_evm_nandflash_partition), .ecc_mode = NAND_ECC_HW, @@ -772,6 +773,8 @@ static __init void davinci_evm_init(void) struct clk *aemif_clk; struct davinci_soc_info *soc_info = &davinci_soc_info; + dm644x_init_devices(); + ret = dm644x_gpio_register(); if (ret) pr_warn("%s: GPIO init failed: %d\n", __func__, ret); diff --git a/arch/arm/mach-davinci/board-dm646x-evm.c b/arch/arm/mach-davinci/board-dm646x-evm.c index a3c0d1e87647..584064fdabf5 100644 --- a/arch/arm/mach-davinci/board-dm646x-evm.c +++ b/arch/arm/mach-davinci/board-dm646x-evm.c @@ -84,6 +84,7 @@ static struct davinci_aemif_timing dm6467tevm_nandflash_timing = { }; static struct davinci_nand_pdata davinci_nand_data = { + .core_chipsel = 0, .mask_cle = 0x80000, .mask_ale = 0x40000, .parts = davinci_nand_partitions, diff --git a/arch/arm/mach-davinci/board-mityomapl138.c b/arch/arm/mach-davinci/board-mityomapl138.c index d1c85484c2e2..37b3e48a21d1 100644 --- a/arch/arm/mach-davinci/board-mityomapl138.c +++ b/arch/arm/mach-davinci/board-mityomapl138.c @@ -400,6 +400,7 @@ static struct mtd_partition mityomapl138_nandflash_partition[] = { }; static struct davinci_nand_pdata mityomapl138_nandflash_data = { + .core_chipsel = 1, .parts = mityomapl138_nandflash_partition, .nr_parts = ARRAY_SIZE(mityomapl138_nandflash_partition), .ecc_mode = NAND_ECC_HW, diff --git a/arch/arm/mach-davinci/board-neuros-osd2.c b/arch/arm/mach-davinci/board-neuros-osd2.c index f2875770fbff..25ad9b0612be 100644 --- a/arch/arm/mach-davinci/board-neuros-osd2.c +++ b/arch/arm/mach-davinci/board-neuros-osd2.c @@ -87,6 +87,7 @@ static struct mtd_partition davinci_ntosd2_nandflash_partition[] = { }; static struct davinci_nand_pdata davinci_ntosd2_nandflash_data = { + .core_chipsel = 0, .parts = davinci_ntosd2_nandflash_partition, .nr_parts = ARRAY_SIZE(davinci_ntosd2_nandflash_partition), .ecc_mode = NAND_ECC_HW, @@ -174,6 +175,8 @@ static __init void davinci_ntosd2_init(void) struct clk *aemif_clk; struct davinci_soc_info *soc_info = &davinci_soc_info; + dm644x_init_devices(); + ret = dm644x_gpio_register(); if (ret) pr_warn("%s: GPIO init failed: %d\n", __func__, ret); diff --git a/arch/arm/mach-davinci/board-sffsdr.c b/arch/arm/mach-davinci/board-sffsdr.c index 2922da9d1684..e7c1728b0833 100644 --- a/arch/arm/mach-davinci/board-sffsdr.c +++ b/arch/arm/mach-davinci/board-sffsdr.c @@ -134,6 +134,8 @@ static __init void davinci_sffsdr_init(void) { struct davinci_soc_info *soc_info = &davinci_soc_info; + dm644x_init_devices(); + platform_add_devices(davinci_sffsdr_devices, ARRAY_SIZE(davinci_sffsdr_devices)); sffsdr_init_i2c(); diff --git a/arch/arm/mach-davinci/davinci.h b/arch/arm/mach-davinci/davinci.h index 270cef85750a..376cdd51ce9d 100644 --- a/arch/arm/mach-davinci/davinci.h +++ b/arch/arm/mach-davinci/davinci.h @@ -104,6 +104,7 @@ int dm365_gpio_register(void); /* DM644x function declarations */ void dm644x_init(void); +void dm644x_init_devices(void); void dm644x_init_time(void); void dm644x_init_asp(void); int dm644x_init_video(struct vpfe_config *, struct vpbe_config *); diff --git a/arch/arm/mach-davinci/dm644x.c b/arch/arm/mach-davinci/dm644x.c index b409801649e1..a2e8586c8a6d 100644 --- a/arch/arm/mach-davinci/dm644x.c +++ b/arch/arm/mach-davinci/dm644x.c @@ -961,19 +961,14 @@ int __init dm644x_init_video(struct vpfe_config *vpfe_cfg, return 0; } -static int __init dm644x_init_devices(void) +void __init dm644x_init_devices(void) { struct platform_device *edma_pdev; - int ret = 0; - - if (!cpu_is_davinci_dm644x()) - return 0; + int ret; edma_pdev = platform_device_register_full(&dm644x_edma_device); - if (IS_ERR(edma_pdev)) { + if (IS_ERR(edma_pdev)) pr_warn("%s: Failed to register eDMA\n", __func__); - return PTR_ERR(edma_pdev); - } platform_device_register(&dm644x_mdio_device); platform_device_register(&dm644x_emac_device); @@ -982,6 +977,4 @@ static int __init dm644x_init_devices(void) if (ret) pr_warn("%s: watchdog init failed: %d\n", __func__, ret); - return ret; } -postcore_initcall(dm644x_init_devices); diff --git a/arch/arm/mach-ep93xx/core.c b/arch/arm/mach-ep93xx/core.c index 0581ffbedddd..faf48a3b1fea 100644 --- a/arch/arm/mach-ep93xx/core.c +++ b/arch/arm/mach-ep93xx/core.c @@ -635,6 +635,7 @@ EXPORT_SYMBOL(ep93xx_keypad_release_gpio); *************************************************************************/ static struct resource ep93xx_i2s_resource[] = { DEFINE_RES_MEM(EP93XX_I2S_PHYS_BASE, 0x100), + DEFINE_RES_IRQ(IRQ_EP93XX_SAI), }; static struct platform_device ep93xx_i2s_device = { diff --git a/arch/arm/mach-exynos/Kconfig b/arch/arm/mach-exynos/Kconfig index 647c319f9f5f..b40963cf91c7 100644 --- a/arch/arm/mach-exynos/Kconfig +++ b/arch/arm/mach-exynos/Kconfig @@ -8,7 +8,6 @@ menuconfig ARCH_EXYNOS bool "Samsung EXYNOS" depends on ARCH_MULTI_V7 - select ARCH_HAS_BANDGAP select ARCH_HAS_HOLES_MEMORYMODEL select ARCH_SUPPORTS_BIG_ENDIAN select ARM_AMBA @@ -108,18 +107,6 @@ config SOC_EXYNOS5420 default y depends on ARCH_EXYNOS5 -config SOC_EXYNOS5440 - bool "SAMSUNG EXYNOS5440" - default y - depends on ARCH_EXYNOS5 - select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE - select HAVE_ARM_ARCH_TIMER - select AUTO_ZRELADDR - select PINCTRL_EXYNOS5440 - select PM_OPP - help - Enable EXYNOS5440 SoC support - config SOC_EXYNOS5800 bool "SAMSUNG EXYNOS5800" default y diff --git a/arch/arm/mach-exynos/common.h b/arch/arm/mach-exynos/common.h index 098f84a149a3..dcd21bb95e3b 100644 --- a/arch/arm/mach-exynos/common.h +++ b/arch/arm/mach-exynos/common.h @@ -21,7 +21,6 @@ #define EXYNOS5250_SOC_ID 0x43520000 #define EXYNOS5410_SOC_ID 0xE5410000 #define EXYNOS5420_SOC_ID 0xE5420000 -#define EXYNOS5440_SOC_ID 0xE5440000 #define EXYNOS5800_SOC_ID 0xE5422000 #define EXYNOS5_SOC_MASK 0xFFFFF000 @@ -39,7 +38,6 @@ IS_SAMSUNG_CPU(exynos4412, EXYNOS4412_CPU_ID, EXYNOS4_CPU_MASK) IS_SAMSUNG_CPU(exynos5250, EXYNOS5250_SOC_ID, EXYNOS5_SOC_MASK) IS_SAMSUNG_CPU(exynos5410, EXYNOS5410_SOC_ID, EXYNOS5_SOC_MASK) IS_SAMSUNG_CPU(exynos5420, EXYNOS5420_SOC_ID, EXYNOS5_SOC_MASK) -IS_SAMSUNG_CPU(exynos5440, EXYNOS5440_SOC_ID, EXYNOS5_SOC_MASK) IS_SAMSUNG_CPU(exynos5800, EXYNOS5800_SOC_ID, EXYNOS5_SOC_MASK) #if defined(CONFIG_SOC_EXYNOS3250) @@ -82,22 +80,12 @@ IS_SAMSUNG_CPU(exynos5800, EXYNOS5800_SOC_ID, EXYNOS5_SOC_MASK) # define soc_is_exynos5420() 0 #endif -#if defined(CONFIG_SOC_EXYNOS5440) -# define soc_is_exynos5440() is_samsung_exynos5440() -#else -# define soc_is_exynos5440() 0 -#endif - #if defined(CONFIG_SOC_EXYNOS5800) # define soc_is_exynos5800() is_samsung_exynos5800() #else # define soc_is_exynos5800() 0 #endif -#define soc_is_exynos4() (soc_is_exynos4210() || soc_is_exynos4412()) -#define soc_is_exynos5() (soc_is_exynos5250() || soc_is_exynos5410() || \ - soc_is_exynos5420() || soc_is_exynos5800()) - extern u32 cp15_save_diag; extern u32 cp15_save_power; @@ -149,6 +137,11 @@ extern void exynos_cpu_restore_register(void); extern void exynos_pm_central_suspend(void); extern int exynos_pm_central_resume(void); extern void exynos_enter_aftr(void); +#ifdef CONFIG_SMP +extern void exynos_scu_enable(void); +#else +static inline void exynos_scu_enable(void) { } +#endif extern struct cpuidle_exynos_data cpuidle_coupled_exynos_data; diff --git a/arch/arm/mach-exynos/exynos.c b/arch/arm/mach-exynos/exynos.c index 8c4f5e342dc1..f4b6c93a7fd0 100644 --- a/arch/arm/mach-exynos/exynos.c +++ b/arch/arm/mach-exynos/exynos.c @@ -24,15 +24,6 @@ #include "common.h" -static struct map_desc exynos4_iodesc[] __initdata = { - { - .virtual = (unsigned long)S5P_VA_COREPERI_BASE, - .pfn = __phys_to_pfn(EXYNOS4_PA_COREPERI), - .length = SZ_8K, - .type = MT_DEVICE, - }, -}; - static struct platform_device exynos_cpuidle = { .name = "exynos_cpuidle", #ifdef CONFIG_ARM_EXYNOS_CPUIDLE @@ -63,15 +54,6 @@ void __init exynos_sysram_init(void) } } -static void __init exynos_init_late(void) -{ - if (of_machine_is_compatible("samsung,exynos5440")) - /* to be supported later */ - return; - - exynos_pm_init(); -} - static int __init exynos_fdt_map_chipid(unsigned long node, const char *uname, int depth, void *data) { @@ -79,8 +61,7 @@ static int __init exynos_fdt_map_chipid(unsigned long node, const char *uname, const __be32 *reg; int len; - if (!of_flat_dt_is_compatible(node, "samsung,exynos4210-chipid") && - !of_flat_dt_is_compatible(node, "samsung,exynos5440-clock")) + if (!of_flat_dt_is_compatible(node, "samsung,exynos4210-chipid")) return 0; reg = of_get_flat_dt_prop(node, "reg", &len); @@ -95,17 +76,6 @@ static int __init exynos_fdt_map_chipid(unsigned long node, const char *uname, return 1; } -/* - * exynos_map_io - * - * register the standard cpu IO areas - */ -static void __init exynos_map_io(void) -{ - if (soc_is_exynos4()) - iotable_init(exynos4_iodesc, ARRAY_SIZE(exynos4_iodesc)); -} - static void __init exynos_init_io(void) { debug_ll_io_init(); @@ -114,8 +84,6 @@ static void __init exynos_init_io(void) /* detect cpu id and rev. */ s5p_init_cpu(S5P_VA_CHIPID); - - exynos_map_io(); } /* @@ -209,7 +177,6 @@ static char const *const exynos_dt_compat[] __initconst = { "samsung,exynos5250", "samsung,exynos5260", "samsung,exynos5420", - "samsung,exynos5440", NULL }; @@ -232,7 +199,7 @@ DT_MACHINE_START(EXYNOS_DT, "SAMSUNG EXYNOS (Flattened Device Tree)") .init_early = exynos_firmware_init, .init_irq = exynos_init_irq, .init_machine = exynos_dt_machine_init, - .init_late = exynos_init_late, + .init_late = exynos_pm_init, .dt_compat = exynos_dt_compat, .dt_fixup = exynos_dt_fixup, MACHINE_END diff --git a/arch/arm/mach-exynos/include/mach/map.h b/arch/arm/mach-exynos/include/mach/map.h index 37a5ea5e2602..22ebe3654633 100644 --- a/arch/arm/mach-exynos/include/mach/map.h +++ b/arch/arm/mach-exynos/include/mach/map.h @@ -15,6 +15,4 @@ #define EXYNOS_PA_CHIPID 0x10000000 -#define EXYNOS4_PA_COREPERI 0x10500000 - #endif /* __ASM_ARCH_MAP_H */ diff --git a/arch/arm/mach-exynos/platsmp.c b/arch/arm/mach-exynos/platsmp.c index 5156fe70e030..6a1e682371b3 100644 --- a/arch/arm/mach-exynos/platsmp.c +++ b/arch/arm/mach-exynos/platsmp.c @@ -163,6 +163,26 @@ int exynos_cluster_power_state(int cluster) S5P_CORE_LOCAL_PWR_EN); } +/** + * exynos_scu_enable : enables SCU for Cortex-A9 based system + */ +void exynos_scu_enable(void) +{ + struct device_node *np; + static void __iomem *scu_base; + + if (!scu_base) { + np = of_find_compatible_node(NULL, NULL, "arm,cortex-a9-scu"); + if (np) { + scu_base = of_iomap(np, 0); + of_node_put(np); + } else { + scu_base = ioremap(scu_a9_get_base(), SZ_4K); + } + } + scu_enable(scu_base); +} + static void __iomem *cpu_boot_reg_base(void) { if (soc_is_exynos4210() && samsung_rev() == EXYNOS4210_REV_1_1) @@ -219,11 +239,6 @@ static void write_pen_release(int val) sync_cache_w(&pen_release); } -static void __iomem *scu_base_addr(void) -{ - return (void __iomem *)(S5P_VA_SCU); -} - static DEFINE_SPINLOCK(boot_lock); static void exynos_secondary_init(unsigned int cpu) @@ -389,7 +404,7 @@ static void __init exynos_smp_prepare_cpus(unsigned int max_cpus) exynos_set_delayed_reset_assertion(true); if (read_cpuid_part() == ARM_CPU_PART_CORTEX_A9) - scu_enable(scu_base_addr()); + exynos_scu_enable(); /* * Write the address of secondary startup into the diff --git a/arch/arm/mach-exynos/pm.c b/arch/arm/mach-exynos/pm.c index a822c5073715..48e7fb38613e 100644 --- a/arch/arm/mach-exynos/pm.c +++ b/arch/arm/mach-exynos/pm.c @@ -22,8 +22,6 @@ #include #include -#include - #include "common.h" static inline void __iomem *exynos_boot_vector_addr(void) @@ -172,7 +170,7 @@ void exynos_enter_aftr(void) cpu_suspend(0, exynos_aftr_finisher); if (read_cpuid_part() == ARM_CPU_PART_CORTEX_A9) { - scu_enable(S5P_VA_SCU); + exynos_scu_enable(); if (call_firmware_op(resume) == -ENOSYS) exynos_cpu_restore_register(); } diff --git a/arch/arm/mach-exynos/suspend.c b/arch/arm/mach-exynos/suspend.c index c2ed997fedef..d3db306a5a70 100644 --- a/arch/arm/mach-exynos/suspend.c +++ b/arch/arm/mach-exynos/suspend.c @@ -30,8 +30,6 @@ #include #include -#include - #include #include "common.h" @@ -401,7 +399,7 @@ static void exynos_pm_resume(void) goto early_wakeup; if (cpuid == ARM_CPU_PART_CORTEX_A9) - scu_enable(S5P_VA_SCU); + exynos_scu_enable(); if (call_firmware_op(resume) == -ENOSYS && cpuid == ARM_CPU_PART_CORTEX_A9) diff --git a/arch/arm/mach-footbridge/dc21285.c b/arch/arm/mach-footbridge/dc21285.c index e7b350f18f5f..16d71bac0061 100644 --- a/arch/arm/mach-footbridge/dc21285.c +++ b/arch/arm/mach-footbridge/dc21285.c @@ -252,7 +252,7 @@ int __init dc21285_setup(int nr, struct pci_sys_data *sys) if (nr || !footbridge_cfn_mode()) return 0; - res = kzalloc(sizeof(struct resource) * 2, GFP_KERNEL); + res = kcalloc(2, sizeof(struct resource), GFP_KERNEL); if (!res) { printk("out of memory for root bus resources"); return 0; diff --git a/arch/arm/mach-highbank/Kconfig b/arch/arm/mach-highbank/Kconfig index 81110ec34226..5552968f07f8 100644 --- a/arch/arm/mach-highbank/Kconfig +++ b/arch/arm/mach-highbank/Kconfig @@ -1,7 +1,6 @@ config ARCH_HIGHBANK bool "Calxeda ECX-1000/2000 (Highbank/Midway)" depends on ARCH_MULTI_V7 - select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE select ARCH_HAS_HOLES_MEMORYMODEL select ARCH_SUPPORTS_BIG_ENDIAN select ARM_AMBA diff --git a/arch/arm/mach-imx/Kconfig b/arch/arm/mach-imx/Kconfig index e47fa13f4b0c..6f4232384774 100644 --- a/arch/arm/mach-imx/Kconfig +++ b/arch/arm/mach-imx/Kconfig @@ -501,6 +501,7 @@ config SOC_IMX6SL config SOC_IMX6SLL bool "i.MX6 SoloLiteLite support" + select PINCTRL_IMX6SLL select SOC_IMX6 help diff --git a/arch/arm/mach-imx/mach-mx31_3ds.c b/arch/arm/mach-imx/mach-mx31_3ds.c index 68c3f0799d5b..9d87f1dcf7bb 100644 --- a/arch/arm/mach-imx/mach-mx31_3ds.c +++ b/arch/arm/mach-imx/mach-mx31_3ds.c @@ -374,26 +374,12 @@ static struct imx_ssi_platform_data mx31_3ds_ssi_pdata = { }; /* SPI */ -static int spi0_internal_chipselect[] = { - MXC_SPI_CS(0), - MXC_SPI_CS(1), - MXC_SPI_CS(2), -}; - static const struct spi_imx_master spi0_pdata __initconst = { - .chipselect = spi0_internal_chipselect, - .num_chipselect = ARRAY_SIZE(spi0_internal_chipselect), -}; - -static int spi1_internal_chipselect[] = { - MXC_SPI_CS(0), - MXC_SPI_CS(1), - MXC_SPI_CS(2), + .num_chipselect = 3, }; static const struct spi_imx_master spi1_pdata __initconst = { - .chipselect = spi1_internal_chipselect, - .num_chipselect = ARRAY_SIZE(spi1_internal_chipselect), + .num_chipselect = 3, }; static struct spi_board_info mx31_3ds_spi_devs[] __initdata = { diff --git a/arch/arm/mach-imx/mach-mx31lilly.c b/arch/arm/mach-imx/mach-mx31lilly.c index 6fd463642954..8bf52819d4d9 100644 --- a/arch/arm/mach-imx/mach-mx31lilly.c +++ b/arch/arm/mach-imx/mach-mx31lilly.c @@ -226,20 +226,12 @@ static void __init lilly1131_usb_init(void) /* SPI */ -static int spi_internal_chipselect[] = { - MXC_SPI_CS(0), - MXC_SPI_CS(1), - MXC_SPI_CS(2), -}; - static const struct spi_imx_master spi0_pdata __initconst = { - .chipselect = spi_internal_chipselect, - .num_chipselect = ARRAY_SIZE(spi_internal_chipselect), + .num_chipselect = 3, }; static const struct spi_imx_master spi1_pdata __initconst = { - .chipselect = spi_internal_chipselect, - .num_chipselect = ARRAY_SIZE(spi_internal_chipselect), + .num_chipselect = 3, }; static struct mc13xxx_platform_data mc13783_pdata __initdata = { diff --git a/arch/arm/mach-imx/mach-mx31lite.c b/arch/arm/mach-imx/mach-mx31lite.c index a3250bc7f114..a3cbba6c955b 100644 --- a/arch/arm/mach-imx/mach-mx31lite.c +++ b/arch/arm/mach-imx/mach-mx31lite.c @@ -83,15 +83,8 @@ static const struct imxuart_platform_data uart_pdata __initconst = { }; /* SPI */ -static int spi0_internal_chipselect[] = { - MXC_SPI_CS(0), - MXC_SPI_CS(1), - MXC_SPI_CS(2), -}; - static const struct spi_imx_master spi0_pdata __initconst = { - .chipselect = spi0_internal_chipselect, - .num_chipselect = ARRAY_SIZE(spi0_internal_chipselect), + .num_chipselect = 3, }; static const struct mxc_nand_platform_data @@ -133,13 +126,8 @@ static struct platform_device smsc911x_device = { * The MC13783 is the only hard-wired SPI device on the module. */ -static int spi1_internal_chipselect[] = { - MXC_SPI_CS(0), -}; - static const struct spi_imx_master spi1_pdata __initconst = { - .chipselect = spi1_internal_chipselect, - .num_chipselect = ARRAY_SIZE(spi1_internal_chipselect), + .num_chipselect = 1, }; static struct mc13xxx_platform_data mc13783_pdata __initdata = { diff --git a/arch/arm/mach-imx/mach-mx31moboard.c b/arch/arm/mach-imx/mach-mx31moboard.c index 7716f83aecdd..643a3d749703 100644 --- a/arch/arm/mach-imx/mach-mx31moboard.c +++ b/arch/arm/mach-imx/mach-mx31moboard.c @@ -152,14 +152,8 @@ static const struct imxi2c_platform_data moboard_i2c1_data __initconst = { .bitrate = 100000, }; -static int moboard_spi1_cs[] = { - MXC_SPI_CS(0), - MXC_SPI_CS(2), -}; - static const struct spi_imx_master moboard_spi1_pdata __initconst = { - .chipselect = moboard_spi1_cs, - .num_chipselect = ARRAY_SIZE(moboard_spi1_cs), + .num_chipselect = 3, }; static struct regulator_consumer_supply sdhc_consumers[] = { @@ -296,19 +290,14 @@ static struct spi_board_info moboard_spi_board_info[] __initdata = { /* irq number is run-time assigned */ .max_speed_hz = 300000, .bus_num = 1, - .chip_select = 1, + .chip_select = 0, .platform_data = &moboard_pmic, .mode = SPI_CS_HIGH, }, }; -static int moboard_spi2_cs[] = { - MXC_SPI_CS(0), MXC_SPI_CS(1), -}; - static const struct spi_imx_master moboard_spi2_pdata __initconst = { - .chipselect = moboard_spi2_cs, - .num_chipselect = ARRAY_SIZE(moboard_spi2_cs), + .num_chipselect = 2, }; #define SDHC1_CD IOMUX_TO_GPIO(MX31_PIN_ATA_CS0) diff --git a/arch/arm/mach-imx/mach-pca100.c b/arch/arm/mach-imx/mach-pca100.c index ed675863655b..5714e2f1b106 100644 --- a/arch/arm/mach-imx/mach-pca100.c +++ b/arch/arm/mach-imx/mach-pca100.c @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include #include #include @@ -168,16 +168,15 @@ static const struct imxi2c_platform_data pca100_i2c1_data __initconst = { .bitrate = 100000, }; -static struct at24_platform_data board_eeprom = { - .byte_len = 4096, - .page_size = 32, - .flags = AT24_FLAG_ADDR16, +static const struct property_entry board_eeprom_properties[] = { + PROPERTY_ENTRY_U32("pagesize", 32), + { } }; static struct i2c_board_info pca100_i2c_devices[] = { { - I2C_BOARD_INFO("at24", 0x52), /* E0=0, E1=1, E2=0 */ - .platform_data = &board_eeprom, + I2C_BOARD_INFO("24c32", 0x52), /* E0=0, E1=1, E2=0 */ + .properties = board_eeprom_properties, }, { I2C_BOARD_INFO("pcf8563", 0x51), }, { diff --git a/arch/arm/mach-imx/mach-pcm037.c b/arch/arm/mach-imx/mach-pcm037.c index b787ba6897e4..004737c40fda 100644 --- a/arch/arm/mach-imx/mach-pcm037.c +++ b/arch/arm/mach-imx/mach-pcm037.c @@ -23,7 +23,7 @@ #include #include #include -#include +#include #include #include #include @@ -263,16 +263,15 @@ static const struct imxi2c_platform_data pcm037_i2c2_data __initconst = { .bitrate = 20000, }; -static struct at24_platform_data board_eeprom = { - .byte_len = 4096, - .page_size = 32, - .flags = AT24_FLAG_ADDR16, +static const struct property_entry board_eeprom_properties[] = { + PROPERTY_ENTRY_U32("pagesize", 32), + { } }; static struct i2c_board_info pcm037_i2c_devices[] = { { - I2C_BOARD_INFO("at24", 0x52), /* E0=0, E1=1, E2=0 */ - .platform_data = &board_eeprom, + I2C_BOARD_INFO("24c32", 0x52), /* E0=0, E1=1, E2=0 */ + .properties = board_eeprom_properties, }, { I2C_BOARD_INFO("pcf8563", 0x51), } diff --git a/arch/arm/mach-imx/mach-pcm037_eet.c b/arch/arm/mach-imx/mach-pcm037_eet.c index 95bd97710494..15bc956d466b 100644 --- a/arch/arm/mach-imx/mach-pcm037_eet.c +++ b/arch/arm/mach-imx/mach-pcm037_eet.c @@ -56,11 +56,8 @@ static struct spi_board_info pcm037_spi_dev[] = { }; /* Platform Data for MXC CSPI */ -static int pcm037_spi1_cs[] = { MXC_SPI_CS(0), MXC_SPI_CS(1), }; - static const struct spi_imx_master pcm037_spi1_pdata __initconst = { - .chipselect = pcm037_spi1_cs, - .num_chipselect = ARRAY_SIZE(pcm037_spi1_cs), + .num_chipselect = 2, }; /* GPIO-keys input device */ diff --git a/arch/arm/mach-imx/mach-pcm043.c b/arch/arm/mach-imx/mach-pcm043.c index 78e2bf8dcd96..e595e5368676 100644 --- a/arch/arm/mach-imx/mach-pcm043.c +++ b/arch/arm/mach-imx/mach-pcm043.c @@ -24,7 +24,7 @@ #include #include #include -#include +#include #include #include @@ -110,16 +110,15 @@ static const struct imxi2c_platform_data pcm043_i2c0_data __initconst = { .bitrate = 50000, }; -static struct at24_platform_data board_eeprom = { - .byte_len = 4096, - .page_size = 32, - .flags = AT24_FLAG_ADDR16, +static const struct property_entry board_eeprom_properties[] = { + PROPERTY_ENTRY_U32("pagesize", 32), + { } }; static struct i2c_board_info pcm043_i2c_devices[] = { { - I2C_BOARD_INFO("at24", 0x52), /* E0=0, E1=1, E2=0 */ - .platform_data = &board_eeprom, + I2C_BOARD_INFO("24c32", 0x52), /* E0=0, E1=1, E2=0 */ + .properties = board_eeprom_properties, }, { I2C_BOARD_INFO("pcf8563", 0x51), }, diff --git a/arch/arm/mach-imx/mach-vpr200.c b/arch/arm/mach-imx/mach-vpr200.c index 5ff154c9a086..da3336aaa4c5 100644 --- a/arch/arm/mach-imx/mach-vpr200.c +++ b/arch/arm/mach-imx/mach-vpr200.c @@ -29,7 +29,6 @@ #include #include -#include #include #include "common.h" @@ -145,15 +144,9 @@ static const struct imxi2c_platform_data vpr200_i2c0_data __initconst = { .bitrate = 50000, }; -static struct at24_platform_data vpr200_eeprom = { - .byte_len = 2048 / 8, - .page_size = 1, -}; - static struct i2c_board_info vpr200_i2c_devices[] = { { - I2C_BOARD_INFO("at24", 0x50), /* E0=0, E1=0, E2=0 */ - .platform_data = &vpr200_eeprom, + I2C_BOARD_INFO("24c02", 0x50), /* E0=0, E1=0, E2=0 */ }, { I2C_BOARD_INFO("mc13892", 0x08), .platform_data = &vpr200_pmic, diff --git a/arch/arm/mach-ixp4xx/common-pci.c b/arch/arm/mach-ixp4xx/common-pci.c index bcf3df59f71b..6835b17113e5 100644 --- a/arch/arm/mach-ixp4xx/common-pci.c +++ b/arch/arm/mach-ixp4xx/common-pci.c @@ -421,7 +421,7 @@ int ixp4xx_setup(int nr, struct pci_sys_data *sys) if (nr >= 1) return 0; - res = kzalloc(sizeof(*res) * 2, GFP_KERNEL); + res = kcalloc(2, sizeof(*res), GFP_KERNEL); if (res == NULL) { /* * If we're out of memory this early, something is wrong, diff --git a/arch/arm/mach-ks8695/board-acs5k.c b/arch/arm/mach-ks8695/board-acs5k.c index 937eb1d47e7b..ef835d82cdb9 100644 --- a/arch/arm/mach-ks8695/board-acs5k.c +++ b/arch/arm/mach-ks8695/board-acs5k.c @@ -19,7 +19,7 @@ #include #include #include -#include +#include #include #include diff --git a/arch/arm/mach-meson/Kconfig b/arch/arm/mach-meson/Kconfig index d90f61e6254f..d51cfda953d4 100644 --- a/arch/arm/mach-meson/Kconfig +++ b/arch/arm/mach-meson/Kconfig @@ -19,14 +19,7 @@ config MACH_MESON6 select MESON6_TIMER config MACH_MESON8 - bool "Amlogic Meson8 SoCs support" - default ARCH_MESON - select MESON6_TIMER - select COMMON_CLK_MESON8B - select MESON_IRQ_GPIO - -config MACH_MESON8B - bool "Amlogic Meson8b SoCs support" + bool "Amlogic Meson8, Meson8b and Meson8m2 SoCs support" default ARCH_MESON select MESON6_TIMER select COMMON_CLK_MESON8B diff --git a/arch/arm/mach-meson/meson.c b/arch/arm/mach-meson/meson.c index 4e2357178625..c8d99df32f9b 100644 --- a/arch/arm/mach-meson/meson.c +++ b/arch/arm/mach-meson/meson.c @@ -20,6 +20,7 @@ static const char * const meson_common_board_compat[] = { "amlogic,meson6", "amlogic,meson8", "amlogic,meson8b", + "amlogic,meson8m2", NULL, }; diff --git a/arch/arm/mach-omap1/board-ams-delta.c b/arch/arm/mach-omap1/board-ams-delta.c index 52e8e53ca154..80f54cb54276 100644 --- a/arch/arm/mach-omap1/board-ams-delta.c +++ b/arch/arm/mach-omap1/board-ams-delta.c @@ -12,6 +12,7 @@ * published by the Free Software Foundation. */ #include +#include #include #include #include @@ -202,7 +203,10 @@ static struct resource latch2_resources[] = { }, }; +#define LATCH2_LABEL "latch2" + static struct bgpio_pdata latch2_pdata = { + .label = LATCH2_LABEL, .base = AMS_DELTA_LATCH2_GPIO_BASE, .ngpio = AMS_DELTA_LATCH2_NGPIO, }; @@ -217,6 +221,23 @@ static struct platform_device latch2_gpio_device = { }, }; +#define LATCH2_PIN_LCD_VBLEN 0 +#define LATCH2_PIN_LCD_NDISP 1 +#define LATCH2_PIN_NAND_NCE 2 +#define LATCH2_PIN_NAND_NRE 3 +#define LATCH2_PIN_NAND_NWP 4 +#define LATCH2_PIN_NAND_NWE 5 +#define LATCH2_PIN_NAND_ALE 6 +#define LATCH2_PIN_NAND_CLE 7 +#define LATCH2_PIN_KEYBRD_PWR 8 +#define LATCH2_PIN_KEYBRD_DATAOUT 9 +#define LATCH2_PIN_SCARD_RSTIN 10 +#define LATCH2_PIN_SCARD_CMDVCC 11 +#define LATCH2_PIN_MODEM_NRESET 12 +#define LATCH2_PIN_MODEM_CODEC 13 +#define LATCH2_PIN_HOOKFLASH1 14 +#define LATCH2_PIN_HOOKFLASH2 15 + static const struct gpio latch_gpios[] __initconst = { { .gpio = LATCH1_GPIO_BASE + 6, @@ -238,11 +259,6 @@ static const struct gpio latch_gpios[] __initconst = { .flags = GPIOF_OUT_INIT_LOW, .label = "scard_cmdvcc", }, - { - .gpio = AMS_DELTA_GPIO_PIN_MODEM_CODEC, - .flags = GPIOF_OUT_INIT_LOW, - .label = "modem_codec", - }, { .gpio = AMS_DELTA_LATCH2_GPIO_BASE + 14, .flags = GPIOF_OUT_INIT_LOW, @@ -323,6 +339,22 @@ static struct platform_device ams_delta_nand_device = { .resource = ams_delta_nand_resources, }; +#define OMAP_GPIO_LABEL "gpio-0-15" + +static struct gpiod_lookup_table ams_delta_nand_gpio_table = { + .table = { + GPIO_LOOKUP(OMAP_GPIO_LABEL, AMS_DELTA_GPIO_PIN_NAND_RB, "rdy", + 0), + GPIO_LOOKUP(LATCH2_LABEL, LATCH2_PIN_NAND_NCE, "nce", 0), + GPIO_LOOKUP(LATCH2_LABEL, LATCH2_PIN_NAND_NRE, "nre", 0), + GPIO_LOOKUP(LATCH2_LABEL, LATCH2_PIN_NAND_NWP, "nwp", 0), + GPIO_LOOKUP(LATCH2_LABEL, LATCH2_PIN_NAND_NWE, "nwe", 0), + GPIO_LOOKUP(LATCH2_LABEL, LATCH2_PIN_NAND_ALE, "ale", 0), + GPIO_LOOKUP(LATCH2_LABEL, LATCH2_PIN_NAND_CLE, "cle", 0), + { }, + }, +}; + static struct resource ams_delta_kp_resources[] = { [0] = { .start = INT_KEYBOARD, @@ -358,6 +390,14 @@ static struct platform_device ams_delta_lcd_device = { .id = -1, }; +static struct gpiod_lookup_table ams_delta_lcd_gpio_table = { + .table = { + GPIO_LOOKUP(LATCH2_LABEL, LATCH2_PIN_LCD_VBLEN, "vblen", 0), + GPIO_LOOKUP(LATCH2_LABEL, LATCH2_PIN_LCD_NDISP, "ndisp", 0), + { }, + }, +}; + static const struct gpio_led gpio_leds[] __initconst = { { .name = "camera", @@ -449,11 +489,35 @@ static struct platform_device ams_delta_audio_device = { .id = -1, }; +static struct gpiod_lookup_table ams_delta_audio_gpio_table = { + .table = { + GPIO_LOOKUP(OMAP_GPIO_LABEL, AMS_DELTA_GPIO_PIN_HOOK_SWITCH, + "hook_switch", 0), + GPIO_LOOKUP(LATCH2_LABEL, LATCH2_PIN_MODEM_CODEC, + "modem_codec", 0), + { }, + }, +}; + static struct platform_device cx20442_codec_device = { .name = "cx20442-codec", .id = -1, }; +static struct gpiod_lookup_table ams_delta_serio_gpio_table = { + .table = { + GPIO_LOOKUP(OMAP_GPIO_LABEL, AMS_DELTA_GPIO_PIN_KEYBRD_DATA, + "data", 0), + GPIO_LOOKUP(OMAP_GPIO_LABEL, AMS_DELTA_GPIO_PIN_KEYBRD_CLK, + "clock", 0), + GPIO_LOOKUP(LATCH2_LABEL, LATCH2_PIN_KEYBRD_PWR, + "power", 0), + GPIO_LOOKUP(LATCH2_LABEL, LATCH2_PIN_KEYBRD_DATAOUT, + "dataout", 0), + { }, + }, +}; + static struct platform_device *ams_delta_devices[] __initdata = { &latch1_gpio_device, &latch2_gpio_device, @@ -468,6 +532,16 @@ static struct platform_device *late_devices[] __initdata = { &cx20442_codec_device, }; +static struct gpiod_lookup_table *ams_delta_gpio_tables[] __initdata = { + &ams_delta_audio_gpio_table, + &ams_delta_serio_gpio_table, +}; + +static struct gpiod_lookup_table *late_gpio_tables[] __initdata = { + &ams_delta_lcd_gpio_table, + &ams_delta_nand_gpio_table, +}; + static void __init ams_delta_init(void) { /* mux pins for uarts */ @@ -500,6 +574,20 @@ static void __init ams_delta_init(void) gpio_led_register_device(-1, &leds_pdata); platform_add_devices(ams_delta_devices, ARRAY_SIZE(ams_delta_devices)); + /* + * As soon as devices have been registered, assign their dev_names + * to respective GPIO lookup tables before they are added. + */ + ams_delta_audio_gpio_table.dev_id = + dev_name(&ams_delta_audio_device.dev); + /* + * No device name is assigned to GPIO lookup table for serio device + * as long as serio driver is not converted to platform device driver. + */ + + gpiod_add_lookup_tables(ams_delta_gpio_tables, + ARRAY_SIZE(ams_delta_gpio_tables)); + ams_delta_init_fiq(); omap_writew(omap_readw(ARM_RSTCT1) | 0x0004, ARM_RSTCT1); @@ -570,6 +658,15 @@ static int __init late_init(void) platform_add_devices(late_devices, ARRAY_SIZE(late_devices)); + /* + * As soon as devices have been registered, assign their dev_names + * to respective GPIO lookup tables before they are added. + */ + ams_delta_lcd_gpio_table.dev_id = dev_name(&ams_delta_lcd_device.dev); + ams_delta_nand_gpio_table.dev_id = dev_name(&ams_delta_nand_device.dev); + + gpiod_add_lookup_tables(late_gpio_tables, ARRAY_SIZE(late_gpio_tables)); + err = platform_device_register(&modem_nreset_device); if (err) { pr_err("Couldn't register the modem regulator device\n"); diff --git a/arch/arm/mach-omap1/board-htcherald.c b/arch/arm/mach-omap1/board-htcherald.c index 67d46690a56e..da8f3fc3180f 100644 --- a/arch/arm/mach-omap1/board-htcherald.c +++ b/arch/arm/mach-omap1/board-htcherald.c @@ -31,7 +31,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/arch/arm/mach-omap1/board-osk.c b/arch/arm/mach-omap1/board-osk.c index c66372ed29e2..9ffa8d755a59 100644 --- a/arch/arm/mach-omap1/board-osk.c +++ b/arch/arm/mach-omap1/board-osk.c @@ -303,22 +303,22 @@ static const struct omap_lcd_config osk_lcd_config __initconst = { #ifdef CONFIG_OMAP_OSK_MISTRAL #include -#include +#include #include #include #include -static struct at24_platform_data at24c04 = { - .byte_len = SZ_4K / 8, - .page_size = 16, +static const struct property_entry mistral_at24_properties[] = { + PROPERTY_ENTRY_U32("pagesize", 16), + { } }; static struct i2c_board_info __initdata mistral_i2c_board_info[] = { { /* NOTE: powered from LCD supply */ I2C_BOARD_INFO("24c04", 0x50), - .platform_data = &at24c04, + .properties = mistral_at24_properties, }, /* TODO when driver support is ready: * - optionally ov9640 camera sensor at 0x30 diff --git a/arch/arm/mach-omap1/common.h b/arch/arm/mach-omap1/common.h index d83ff257eaa8..c6537d2c2859 100644 --- a/arch/arm/mach-omap1/common.h +++ b/arch/arm/mach-omap1/common.h @@ -27,7 +27,7 @@ #define __ARCH_ARM_MACH_OMAP1_COMMON_H #include -#include +#include #include #include diff --git a/arch/arm/mach-omap1/i2c.c b/arch/arm/mach-omap1/i2c.c index 5bdf3c4190f9..9250f263ac51 100644 --- a/arch/arm/mach-omap1/i2c.c +++ b/arch/arm/mach-omap1/i2c.c @@ -20,7 +20,7 @@ */ #include -#include +#include #include #include "soc.h" diff --git a/arch/arm/mach-omap1/mcbsp.c b/arch/arm/mach-omap1/mcbsp.c index 8ed67f8d1762..27e22e702f96 100644 --- a/arch/arm/mach-omap1/mcbsp.c +++ b/arch/arm/mach-omap1/mcbsp.c @@ -389,7 +389,7 @@ static void omap_mcbsp_register_board_cfg(struct resource *res, int res_count, { int i; - omap_mcbsp_devices = kzalloc(size * sizeof(struct platform_device *), + omap_mcbsp_devices = kcalloc(size, sizeof(struct platform_device *), GFP_KERNEL); if (!omap_mcbsp_devices) { printk(KERN_ERR "Could not register McBSP devices\n"); diff --git a/arch/arm/mach-omap2/Makefile b/arch/arm/mach-omap2/Makefile index 0d9ce58bc464..01377c292db4 100644 --- a/arch/arm/mach-omap2/Makefile +++ b/arch/arm/mach-omap2/Makefile @@ -78,7 +78,6 @@ endif omap-4-5-pm-common = omap-mpuss-lowpower.o obj-$(CONFIG_ARCH_OMAP4) += $(omap-4-5-pm-common) obj-$(CONFIG_SOC_OMAP5) += $(omap-4-5-pm-common) -obj-$(CONFIG_OMAP_PM_NOOP) += omap-pm-noop.o ifeq ($(CONFIG_PM),y) obj-$(CONFIG_ARCH_OMAP2) += pm24xx.o diff --git a/arch/arm/mach-omap2/board-generic.c b/arch/arm/mach-omap2/board-generic.c index 6c61ecc62905..6b4f4975cf7a 100644 --- a/arch/arm/mach-omap2/board-generic.c +++ b/arch/arm/mach-omap2/board-generic.c @@ -31,8 +31,6 @@ static const struct of_device_id omap_dt_match_table[] __initconst = { static void __init __maybe_unused omap_generic_init(void) { pdata_quirks_init(omap_dt_match_table); - - omapdss_init_of(); omap_soc_device_init(); } diff --git a/arch/arm/mach-omap2/clockdomain.c b/arch/arm/mach-omap2/clockdomain.c index b79b1ca9aee9..6d44fe05a3fe 100644 --- a/arch/arm/mach-omap2/clockdomain.c +++ b/arch/arm/mach-omap2/clockdomain.c @@ -23,6 +23,7 @@ #include #include #include +#include #include @@ -31,6 +32,7 @@ #include "soc.h" #include "clock.h" #include "clockdomain.h" +#include "pm.h" /* clkdm_list contains all registered struct clockdomains */ static LIST_HEAD(clkdm_list); @@ -39,6 +41,8 @@ static LIST_HEAD(clkdm_list); static struct clkdm_autodep *autodeps; static struct clkdm_ops *arch_clkdm; +void clkdm_save_context(void); +void clkdm_restore_context(void); /* Private functions */ @@ -449,6 +453,22 @@ int clkdm_register_autodeps(struct clkdm_autodep *ia) return 0; } +static int cpu_notifier(struct notifier_block *nb, unsigned long cmd, void *v) +{ + switch (cmd) { + case CPU_CLUSTER_PM_ENTER: + if (enable_off_mode) + clkdm_save_context(); + break; + case CPU_CLUSTER_PM_EXIT: + if (enable_off_mode) + clkdm_restore_context(); + break; + } + + return NOTIFY_OK; +} + /** * clkdm_complete_init - set up the clockdomain layer * @@ -460,6 +480,7 @@ int clkdm_register_autodeps(struct clkdm_autodep *ia) int clkdm_complete_init(void) { struct clockdomain *clkdm; + static struct notifier_block nb; if (list_empty(&clkdm_list)) return -EACCES; @@ -474,6 +495,12 @@ int clkdm_complete_init(void) clkdm_clear_all_sleepdeps(clkdm); } + /* Only AM43XX can lose clkdm context during rtc-ddr suspend */ + if (soc_is_am43xx()) { + nb.notifier_call = cpu_notifier; + cpu_pm_register_notifier(&nb); + } + return 0; } @@ -1307,3 +1334,49 @@ int clkdm_hwmod_disable(struct clockdomain *clkdm, struct omap_hwmod *oh) return 0; } +/** + * _clkdm_save_context - save the context for the control of this clkdm + * + * Due to a suspend or hibernation operation, the state of the registers + * controlling this clkdm will be lost, save their context. + */ +static int _clkdm_save_context(struct clockdomain *clkdm, void *ununsed) +{ + if (!arch_clkdm || !arch_clkdm->clkdm_save_context) + return -EINVAL; + + return arch_clkdm->clkdm_save_context(clkdm); +} + +/** + * _clkdm_restore_context - restore context for control of this clkdm + * + * Restore the register values for this clockdomain. + */ +static int _clkdm_restore_context(struct clockdomain *clkdm, void *ununsed) +{ + if (!arch_clkdm || !arch_clkdm->clkdm_restore_context) + return -EINVAL; + + return arch_clkdm->clkdm_restore_context(clkdm); +} + +/** + * clkdm_save_context - Saves the context for each registered clkdm + * + * Save the context for each registered clockdomain. + */ +void clkdm_save_context(void) +{ + clkdm_for_each(_clkdm_save_context, NULL); +} + +/** + * clkdm_restore_context - Restores the context for each registered clkdm + * + * Restore the context for each registered clockdomain. + */ +void clkdm_restore_context(void) +{ + clkdm_for_each(_clkdm_restore_context, NULL); +} diff --git a/arch/arm/mach-omap2/clockdomain.h b/arch/arm/mach-omap2/clockdomain.h index 24667a5a9dc0..c7d0953e4aa2 100644 --- a/arch/arm/mach-omap2/clockdomain.h +++ b/arch/arm/mach-omap2/clockdomain.h @@ -141,6 +141,7 @@ struct clockdomain { int usecount; int forcewake_count; struct list_head node; + u32 context; }; /** @@ -159,6 +160,8 @@ struct clockdomain { * @clkdm_deny_idle: Disable hw supervised idle transitions for clock domain * @clkdm_clk_enable: Put the clkdm in right state for a clock enable * @clkdm_clk_disable: Put the clkdm in right state for a clock disable + * @clkdm_save_context: Save the current clkdm context + * @clkdm_restore_context: Restore the clkdm context */ struct clkdm_ops { int (*clkdm_add_wkdep)(struct clockdomain *clkdm1, struct clockdomain *clkdm2); @@ -175,6 +178,8 @@ struct clkdm_ops { void (*clkdm_deny_idle)(struct clockdomain *clkdm); int (*clkdm_clk_enable)(struct clockdomain *clkdm); int (*clkdm_clk_disable)(struct clockdomain *clkdm); + int (*clkdm_save_context)(struct clockdomain *clkdm); + int (*clkdm_restore_context)(struct clockdomain *clkdm); }; int clkdm_register_platform_funcs(struct clkdm_ops *co); @@ -214,6 +219,9 @@ int clkdm_clk_disable(struct clockdomain *clkdm, struct clk *clk); int clkdm_hwmod_enable(struct clockdomain *clkdm, struct omap_hwmod *oh); int clkdm_hwmod_disable(struct clockdomain *clkdm, struct omap_hwmod *oh); +void clkdm_save_context(void); +void clkdm_restore_context(void); + extern void __init omap242x_clockdomains_init(void); extern void __init omap243x_clockdomains_init(void); extern void __init omap3xxx_clockdomains_init(void); diff --git a/arch/arm/mach-omap2/cm33xx.c b/arch/arm/mach-omap2/cm33xx.c index 1cc0247a2cb5..084d454f6074 100644 --- a/arch/arm/mach-omap2/cm33xx.c +++ b/arch/arm/mach-omap2/cm33xx.c @@ -72,6 +72,17 @@ static inline u32 am33xx_cm_rmw_reg_bits(u32 mask, u32 bits, s16 inst, s16 idx) return v; } +static inline u32 am33xx_cm_read_reg_bits(u16 inst, s16 idx, u32 mask) +{ + u32 v; + + v = am33xx_cm_read_reg(inst, idx); + v &= mask; + v >>= __ffs(mask); + + return v; +} + /** * _clkctrl_idlest - read a CM_*_CLKCTRL register; mask & shift IDLEST bitfield * @inst: CM instance register offset (*_INST macro) @@ -338,6 +349,46 @@ static u32 am33xx_cm_xlate_clkctrl(u8 part, u16 inst, u16 offset) return cm_base.pa + inst + offset; } +/** + * am33xx_clkdm_save_context - Save the clockdomain transition context + * @clkdm: The clockdomain pointer whose context needs to be saved + * + * Save the clockdomain transition context. + */ +static int am33xx_clkdm_save_context(struct clockdomain *clkdm) +{ + clkdm->context = am33xx_cm_read_reg_bits(clkdm->cm_inst, + clkdm->clkdm_offs, + AM33XX_CLKTRCTRL_MASK); + + return 0; +} + +/** + * am33xx_restore_save_context - Restore the clockdomain transition context + * @clkdm: The clockdomain pointer whose context needs to be restored + * + * Restore the clockdomain transition context. + */ +static int am33xx_clkdm_restore_context(struct clockdomain *clkdm) +{ + switch (clkdm->context) { + case OMAP34XX_CLKSTCTRL_DISABLE_AUTO: + am33xx_clkdm_deny_idle(clkdm); + break; + case OMAP34XX_CLKSTCTRL_FORCE_SLEEP: + am33xx_clkdm_sleep(clkdm); + break; + case OMAP34XX_CLKSTCTRL_FORCE_WAKEUP: + am33xx_clkdm_wakeup(clkdm); + break; + case OMAP34XX_CLKSTCTRL_ENABLE_AUTO: + am33xx_clkdm_allow_idle(clkdm); + break; + } + return 0; +} + struct clkdm_ops am33xx_clkdm_operations = { .clkdm_sleep = am33xx_clkdm_sleep, .clkdm_wakeup = am33xx_clkdm_wakeup, @@ -345,6 +396,8 @@ struct clkdm_ops am33xx_clkdm_operations = { .clkdm_deny_idle = am33xx_clkdm_deny_idle, .clkdm_clk_enable = am33xx_clkdm_clk_enable, .clkdm_clk_disable = am33xx_clkdm_clk_disable, + .clkdm_save_context = am33xx_clkdm_save_context, + .clkdm_restore_context = am33xx_clkdm_restore_context, }; static const struct cm_ll_data am33xx_cm_ll_data = { diff --git a/arch/arm/mach-omap2/cminst44xx.c b/arch/arm/mach-omap2/cminst44xx.c index 7deefee49fc3..c11ac492b626 100644 --- a/arch/arm/mach-omap2/cminst44xx.c +++ b/arch/arm/mach-omap2/cminst44xx.c @@ -481,6 +481,47 @@ static u32 omap4_cminst_xlate_clkctrl(u8 part, u16 inst, u16 offset) return _cm_bases[part].pa + inst + offset; } +/** + * omap4_clkdm_save_context - Save the clockdomain modulemode context + * @clkdm: The clockdomain pointer whose context needs to be saved + * + * Save the clockdomain modulemode context. + */ +static int omap4_clkdm_save_context(struct clockdomain *clkdm) +{ + clkdm->context = omap4_cminst_read_inst_reg(clkdm->prcm_partition, + clkdm->cm_inst, + clkdm->clkdm_offs + + OMAP4_CM_CLKSTCTRL); + clkdm->context &= OMAP4430_MODULEMODE_MASK; + return 0; +} + +/** + * omap4_clkdm_restore_context - Restore the clockdomain modulemode context + * @clkdm: The clockdomain pointer whose context needs to be restored + * + * Restore the clockdomain modulemode context. + */ +static int omap4_clkdm_restore_context(struct clockdomain *clkdm) +{ + switch (clkdm->context) { + case OMAP34XX_CLKSTCTRL_DISABLE_AUTO: + omap4_clkdm_deny_idle(clkdm); + break; + case OMAP34XX_CLKSTCTRL_FORCE_SLEEP: + omap4_clkdm_sleep(clkdm); + break; + case OMAP34XX_CLKSTCTRL_FORCE_WAKEUP: + omap4_clkdm_wakeup(clkdm); + break; + case OMAP34XX_CLKSTCTRL_ENABLE_AUTO: + omap4_clkdm_allow_idle(clkdm); + break; + } + return 0; +} + struct clkdm_ops omap4_clkdm_operations = { .clkdm_add_wkdep = omap4_clkdm_add_wkup_sleep_dep, .clkdm_del_wkdep = omap4_clkdm_del_wkup_sleep_dep, @@ -496,6 +537,8 @@ struct clkdm_ops omap4_clkdm_operations = { .clkdm_deny_idle = omap4_clkdm_deny_idle, .clkdm_clk_enable = omap4_clkdm_clk_enable, .clkdm_clk_disable = omap4_clkdm_clk_disable, + .clkdm_save_context = omap4_clkdm_save_context, + .clkdm_restore_context = omap4_clkdm_restore_context, }; struct clkdm_ops am43xx_clkdm_operations = { diff --git a/arch/arm/mach-omap2/common.h b/arch/arm/mach-omap2/common.h index fbe0b78bf489..129455e822e4 100644 --- a/arch/arm/mach-omap2/common.h +++ b/arch/arm/mach-omap2/common.h @@ -30,7 +30,7 @@ #include #include #include -#include +#include #include #include @@ -44,6 +44,9 @@ #define OMAP_INTC_START NR_IRQS +extern int (*omap_pm_soc_init)(void); +int omap_pm_nop_init(void); + #if defined(CONFIG_PM) && defined(CONFIG_ARCH_OMAP2) int omap2_pm_init(void); #else @@ -79,9 +82,12 @@ static inline int omap4_pm_init_early(void) #if defined(CONFIG_PM) && (defined(CONFIG_SOC_AM33XX) || \ defined(CONFIG_SOC_AM43XX)) -void amx3_common_pm_init(void); +int amx3_common_pm_init(void); #else -static inline void amx3_common_pm_init(void) { } +static inline int amx3_common_pm_init(void) +{ + return 0; +} #endif extern void omap2_init_common_infrastructure(void); @@ -122,14 +128,10 @@ void am43xx_init_early(void); void am43xx_init_late(void); void omap4430_init_early(void); void omap5_init_early(void); -void omap3_init_late(void); /* Do not use this one */ +void omap3_init_late(void); void omap4430_init_late(void); void omap2420_init_late(void); void omap2430_init_late(void); -void omap3430_init_late(void); -void omap35xx_init_late(void); -void omap3630_init_late(void); -void am35xx_init_late(void); void ti81xx_init_late(void); void am33xx_init_late(void); void omap5_init_late(void); @@ -350,7 +352,5 @@ extern int omap_dss_reset(struct omap_hwmod *); /* SoC specific clock initializer */ int omap_clk_init(void); -int __init omapdss_init_of(void); - #endif /* __ASSEMBLER__ */ #endif /* __ARCH_ARM_MACH_OMAP2PLUS_COMMON_H */ diff --git a/arch/arm/mach-omap2/control.c b/arch/arm/mach-omap2/control.c index 180da403639e..0bbfb20e193f 100644 --- a/arch/arm/mach-omap2/control.c +++ b/arch/arm/mach-omap2/control.c @@ -17,6 +17,7 @@ #include #include #include +#include #include "soc.h" #include "iomap.h" @@ -621,6 +622,110 @@ void __init omap3_ctrl_init(void) } #endif /* CONFIG_ARCH_OMAP3 && CONFIG_PM */ +static unsigned long am43xx_control_reg_offsets[] = { + AM33XX_CONTROL_SYSCONFIG_OFFSET, + AM33XX_CONTROL_STATUS_OFFSET, + AM43XX_CONTROL_MPU_L2_CTRL_OFFSET, + AM33XX_CONTROL_CORE_SLDO_CTRL_OFFSET, + AM33XX_CONTROL_MPU_SLDO_CTRL_OFFSET, + AM33XX_CONTROL_CLK32KDIVRATIO_CTRL_OFFSET, + AM33XX_CONTROL_BANDGAP_CTRL_OFFSET, + AM33XX_CONTROL_BANDGAP_TRIM_OFFSET, + AM33XX_CONTROL_PLL_CLKINPULOW_CTRL_OFFSET, + AM33XX_CONTROL_MOSC_CTRL_OFFSET, + AM33XX_CONTROL_DEEPSLEEP_CTRL_OFFSET, + AM43XX_CONTROL_DISPLAY_PLL_SEL_OFFSET, + AM33XX_CONTROL_INIT_PRIORITY_0_OFFSET, + AM33XX_CONTROL_INIT_PRIORITY_1_OFFSET, + AM33XX_CONTROL_TPTC_CFG_OFFSET, + AM33XX_CONTROL_USB_CTRL0_OFFSET, + AM33XX_CONTROL_USB_CTRL1_OFFSET, + AM43XX_CONTROL_USB_CTRL2_OFFSET, + AM43XX_CONTROL_GMII_SEL_OFFSET, + AM43XX_CONTROL_MPUSS_CTRL_OFFSET, + AM43XX_CONTROL_TIMER_CASCADE_CTRL_OFFSET, + AM43XX_CONTROL_PWMSS_CTRL_OFFSET, + AM33XX_CONTROL_MREQPRIO_0_OFFSET, + AM33XX_CONTROL_MREQPRIO_1_OFFSET, + AM33XX_CONTROL_HW_EVENT_SEL_GRP1_OFFSET, + AM33XX_CONTROL_HW_EVENT_SEL_GRP2_OFFSET, + AM33XX_CONTROL_HW_EVENT_SEL_GRP3_OFFSET, + AM33XX_CONTROL_HW_EVENT_SEL_GRP4_OFFSET, + AM33XX_CONTROL_SMRT_CTRL_OFFSET, + AM33XX_CONTROL_MPUSS_HW_DEBUG_SEL_OFFSET, + AM43XX_CONTROL_CQDETECT_STS_OFFSET, + AM43XX_CONTROL_CQDETECT_STS2_OFFSET, + AM43XX_CONTROL_VTP_CTRL_OFFSET, + AM33XX_CONTROL_VREF_CTRL_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_0_3_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_4_7_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_8_11_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_12_15_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_16_19_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_20_23_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_24_27_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_28_31_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_32_35_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_36_39_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_40_43_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_44_47_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_48_51_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_52_55_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_56_59_OFFSET, + AM33XX_CONTROL_TPCC_EVT_MUX_60_63_OFFSET, + AM33XX_CONTROL_TIMER_EVT_CAPT_OFFSET, + AM33XX_CONTROL_ECAP_EVT_CAPT_OFFSET, + AM33XX_CONTROL_ADC_EVT_CAPT_OFFSET, + AM43XX_CONTROL_ADC1_EVT_CAPT_OFFSET, + AM33XX_CONTROL_RESET_ISO_OFFSET, +}; + +static u32 am33xx_control_vals[ARRAY_SIZE(am43xx_control_reg_offsets)]; + +/** + * am43xx_control_save_context - Save the wakeup domain registers + * + * Save the wkup domain registers + */ +void am43xx_control_save_context(void) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(am43xx_control_reg_offsets); i++) + am33xx_control_vals[i] = + omap_ctrl_readl(am43xx_control_reg_offsets[i]); +} + +/** + * am43xx_control_restore_context - Restore the wakeup domain registers + * + * Restore the wkup domain registers + */ +void am43xx_control_restore_context(void) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(am43xx_control_reg_offsets); i++) + omap_ctrl_writel(am33xx_control_vals[i], + am43xx_control_reg_offsets[i]); +} + +static int cpu_notifier(struct notifier_block *nb, unsigned long cmd, void *v) +{ + switch (cmd) { + case CPU_CLUSTER_PM_ENTER: + if (enable_off_mode) + am43xx_control_save_context(); + break; + case CPU_CLUSTER_PM_EXIT: + if (enable_off_mode) + am43xx_control_restore_context(); + break; + } + + return NOTIFY_OK; +} + struct control_init_data { int index; void __iomem *mem; @@ -699,6 +804,7 @@ int __init omap_control_init(void) const struct omap_prcm_init_data *data; int ret; struct regmap *syscon; + static struct notifier_block nb; for_each_matching_node_and_match(np, omap_scrm_dt_match_table, &match) { data = match->data; @@ -731,6 +837,12 @@ int __init omap_control_init(void) } } + /* Only AM43XX can lose ctrl registers context during rtc-ddr suspend */ + if (soc_is_am43xx()) { + nb.notifier_call = cpu_notifier; + cpu_pm_register_notifier(&nb); + } + return 0; } diff --git a/arch/arm/mach-omap2/control.h b/arch/arm/mach-omap2/control.h index ec406bc2c6d4..393b42110511 100644 --- a/arch/arm/mach-omap2/control.h +++ b/arch/arm/mach-omap2/control.h @@ -409,6 +409,67 @@ #define AM33XX_DEV_FEATURE 0x604 #define AM33XX_SGX_MASK BIT(29) +/* Additional AM33XX/AM43XX CONTROL registers */ +#define AM33XX_CONTROL_SYSCONFIG_OFFSET 0x0010 +#define AM33XX_CONTROL_STATUS_OFFSET 0x0040 +#define AM43XX_CONTROL_MPU_L2_CTRL_OFFSET 0x01e0 +#define AM33XX_CONTROL_CORTEX_VBBLDO_CTRL_OFFSET 0x041c +#define AM33XX_CONTROL_CORE_SLDO_CTRL_OFFSET 0x0428 +#define AM33XX_CONTROL_MPU_SLDO_CTRL_OFFSET 0x042c +#define AM33XX_CONTROL_CLK32KDIVRATIO_CTRL_OFFSET 0x0444 +#define AM33XX_CONTROL_BANDGAP_CTRL_OFFSET 0x0448 +#define AM33XX_CONTROL_BANDGAP_TRIM_OFFSET 0x044c +#define AM33XX_CONTROL_PLL_CLKINPULOW_CTRL_OFFSET 0x0458 +#define AM33XX_CONTROL_MOSC_CTRL_OFFSET 0x0468 +#define AM33XX_CONTROL_RCOSC_CTRL_OFFSET 0x046c +#define AM33XX_CONTROL_DEEPSLEEP_CTRL_OFFSET 0x0470 +#define AM43XX_CONTROL_DISPLAY_PLL_SEL_OFFSET 0x0534 +#define AM33XX_CONTROL_INIT_PRIORITY_0_OFFSET 0x0608 +#define AM33XX_CONTROL_INIT_PRIORITY_1_OFFSET 0x060c +#define AM33XX_CONTROL_MMU_CFG_OFFSET 0x0610 +#define AM33XX_CONTROL_TPTC_CFG_OFFSET 0x0614 +#define AM33XX_CONTROL_USB_CTRL0_OFFSET 0x0620 +#define AM33XX_CONTROL_USB_CTRL1_OFFSET 0x0628 +#define AM33XX_CONTROL_USB_WKUP_CTRL_OFFSET 0x0648 +#define AM43XX_CONTROL_USB_CTRL2_OFFSET 0x064c +#define AM43XX_CONTROL_GMII_SEL_OFFSET 0x0650 +#define AM43XX_CONTROL_MPUSS_CTRL_OFFSET 0x0654 +#define AM43XX_CONTROL_TIMER_CASCADE_CTRL_OFFSET 0x0658 +#define AM43XX_CONTROL_PWMSS_CTRL_OFFSET 0x0664 +#define AM33XX_CONTROL_MREQPRIO_0_OFFSET 0x0670 +#define AM33XX_CONTROL_MREQPRIO_1_OFFSET 0x0674 +#define AM33XX_CONTROL_HW_EVENT_SEL_GRP1_OFFSET 0x0690 +#define AM33XX_CONTROL_HW_EVENT_SEL_GRP2_OFFSET 0x0694 +#define AM33XX_CONTROL_HW_EVENT_SEL_GRP3_OFFSET 0x0698 +#define AM33XX_CONTROL_HW_EVENT_SEL_GRP4_OFFSET 0x069c +#define AM33XX_CONTROL_SMRT_CTRL_OFFSET 0x06a0 +#define AM33XX_CONTROL_MPUSS_HW_DEBUG_SEL_OFFSET 0x06a4 +#define AM43XX_CONTROL_CQDETECT_STS_OFFSET 0x0e00 +#define AM43XX_CONTROL_CQDETECT_STS2_OFFSET 0x0e08 +#define AM43XX_CONTROL_VTP_CTRL_OFFSET 0x0e0c +#define AM33XX_CONTROL_VREF_CTRL_OFFSET 0x0e14 +#define AM33XX_CONTROL_TPCC_EVT_MUX_0_3_OFFSET 0x0f90 +#define AM33XX_CONTROL_TPCC_EVT_MUX_4_7_OFFSET 0x0f94 +#define AM33XX_CONTROL_TPCC_EVT_MUX_8_11_OFFSET 0x0f98 +#define AM33XX_CONTROL_TPCC_EVT_MUX_12_15_OFFSET 0x0f9c +#define AM33XX_CONTROL_TPCC_EVT_MUX_16_19_OFFSET 0x0fa0 +#define AM33XX_CONTROL_TPCC_EVT_MUX_20_23_OFFSET 0x0fa4 +#define AM33XX_CONTROL_TPCC_EVT_MUX_24_27_OFFSET 0x0fa8 +#define AM33XX_CONTROL_TPCC_EVT_MUX_28_31_OFFSET 0x0fac +#define AM33XX_CONTROL_TPCC_EVT_MUX_32_35_OFFSET 0x0fb0 +#define AM33XX_CONTROL_TPCC_EVT_MUX_36_39_OFFSET 0x0fb4 +#define AM33XX_CONTROL_TPCC_EVT_MUX_40_43_OFFSET 0x0fb8 +#define AM33XX_CONTROL_TPCC_EVT_MUX_44_47_OFFSET 0x0fbc +#define AM33XX_CONTROL_TPCC_EVT_MUX_48_51_OFFSET 0x0fc0 +#define AM33XX_CONTROL_TPCC_EVT_MUX_52_55_OFFSET 0x0fc4 +#define AM33XX_CONTROL_TPCC_EVT_MUX_56_59_OFFSET 0x0fc8 +#define AM33XX_CONTROL_TPCC_EVT_MUX_60_63_OFFSET 0x0fcc +#define AM33XX_CONTROL_TIMER_EVT_CAPT_OFFSET 0x0fd0 +#define AM33XX_CONTROL_ECAP_EVT_CAPT_OFFSET 0x0fd4 +#define AM33XX_CONTROL_ADC_EVT_CAPT_OFFSET 0x0fd8 +#define AM43XX_CONTROL_ADC1_EVT_CAPT_OFFSET 0x0fdc +#define AM33XX_CONTROL_RESET_ISO_OFFSET 0x1000 + /* CONTROL OMAP STATUS register to identify OMAP3 features */ #define OMAP3_CONTROL_OMAP_STATUS 0x044c diff --git a/arch/arm/mach-omap2/display.c b/arch/arm/mach-omap2/display.c index b3f6eb5d04a2..9500b6e27380 100644 --- a/arch/arm/mach-omap2/display.c +++ b/arch/arm/mach-omap2/display.c @@ -32,7 +32,6 @@ #include #include "omap_hwmod.h" #include "omap_device.h" -#include "omap-pm.h" #include "common.h" #include "soc.h" @@ -126,11 +125,6 @@ static void omap_dsi_disable_pads(int dsi_id, unsigned lane_mask) omap4_dsi_mux_pads(dsi_id, 0); } -static int omap_dss_set_min_bus_tput(struct device *dev, unsigned long tput) -{ - return omap_pm_set_min_bus_tput(dev, OCP_INITIATOR_AGENT, tput); -} - static enum omapdss_version __init omap_display_get_version(void) { if (cpu_is_omap24xx()) @@ -169,7 +163,6 @@ static int __init omapdss_init_fbdev(void) static struct omap_dss_board_info board_data = { .dsi_enable_pads = omap_dsi_enable_pads, .dsi_disable_pads = omap_dsi_disable_pads, - .set_min_bus_tput = omap_dss_set_min_bus_tput, }; struct device_node *node; int r; @@ -392,7 +385,7 @@ static struct device_node * __init omapdss_find_dss_of_node(void) return NULL; } -int __init omapdss_init_of(void) +static int __init omapdss_init_of(void) { int r; struct device_node *node; @@ -422,3 +415,4 @@ int __init omapdss_init_of(void) return omapdss_init_fbdev(); } +omap_device_initcall(omapdss_init_of); diff --git a/arch/arm/mach-omap2/hsmmc.c b/arch/arm/mach-omap2/hsmmc.c index b064066d431c..af545193f673 100644 --- a/arch/arm/mach-omap2/hsmmc.c +++ b/arch/arm/mach-omap2/hsmmc.c @@ -18,7 +18,6 @@ #include "soc.h" #include "omap_device.h" -#include "omap-pm.h" #include "hsmmc.h" #include "control.h" @@ -35,7 +34,7 @@ static int __init omap_hsmmc_pdata_init(struct omap2_hsmmc_info *c, { char *hc_name; - hc_name = kzalloc(sizeof(char) * (HSMMC_NAME_LEN + 1), GFP_KERNEL); + hc_name = kzalloc(HSMMC_NAME_LEN + 1, GFP_KERNEL); if (!hc_name) { kfree(hc_name); return -ENOMEM; diff --git a/arch/arm/mach-omap2/i2c.c b/arch/arm/mach-omap2/i2c.c index 91a21c3923b2..37ff25ee3d89 100644 --- a/arch/arm/mach-omap2/i2c.c +++ b/arch/arm/mach-omap2/i2c.c @@ -22,7 +22,6 @@ #include "soc.h" #include "omap_hwmod.h" #include "omap_device.h" -#include "omap-pm.h" #include "prm.h" #include "common.h" diff --git a/arch/arm/mach-omap2/io.c b/arch/arm/mach-omap2/io.c index cf546dfe3b32..bb8e0bb7ef5d 100644 --- a/arch/arm/mach-omap2/io.c +++ b/arch/arm/mach-omap2/io.c @@ -37,7 +37,6 @@ #include "clock.h" #include "clock2xxx.h" #include "clock3xxx.h" -#include "omap-pm.h" #include "sdrc.h" #include "control.h" #include "serial.h" @@ -421,13 +420,6 @@ static void __init __maybe_unused omap_hwmod_init_postsetup(void) postsetup_state = _HWMOD_STATE_ENABLED; #endif omap_hwmod_for_each(_set_hwmod_postsetup_state, &postsetup_state); - - omap_pm_if_early_init(); -} - -static void __init __maybe_unused omap_common_late_init(void) -{ - omap2_common_pm_late_init(); } #ifdef CONFIG_SOC_OMAP2420 @@ -450,9 +442,7 @@ void __init omap2420_init_early(void) void __init omap2420_init_late(void) { - omap_common_late_init(); - omap2_pm_init(); - omap2_clk_enable_autoidle_all(); + omap_pm_soc_init = omap2_pm_init; } #endif @@ -476,9 +466,7 @@ void __init omap2430_init_early(void) void __init omap2430_init_late(void) { - omap_common_late_init(); - omap2_pm_init(); - omap2_clk_enable_autoidle_all(); + omap_pm_soc_init = omap2_pm_init; } #endif @@ -529,43 +517,12 @@ void __init am35xx_init_early(void) void __init omap3_init_late(void) { - omap_common_late_init(); - omap3_pm_init(); - omap2_clk_enable_autoidle_all(); -} - -void __init omap3430_init_late(void) -{ - omap_common_late_init(); - omap3_pm_init(); - omap2_clk_enable_autoidle_all(); -} - -void __init omap35xx_init_late(void) -{ - omap_common_late_init(); - omap3_pm_init(); - omap2_clk_enable_autoidle_all(); -} - -void __init omap3630_init_late(void) -{ - omap_common_late_init(); - omap3_pm_init(); - omap2_clk_enable_autoidle_all(); -} - -void __init am35xx_init_late(void) -{ - omap_common_late_init(); - omap3_pm_init(); - omap2_clk_enable_autoidle_all(); + omap_pm_soc_init = omap3_pm_init; } void __init ti81xx_init_late(void) { - omap_common_late_init(); - omap2_clk_enable_autoidle_all(); + omap_pm_soc_init = omap_pm_nop_init; } #endif @@ -621,8 +578,7 @@ void __init am33xx_init_early(void) void __init am33xx_init_late(void) { - omap_common_late_init(); - amx3_common_pm_init(); + omap_pm_soc_init = amx3_common_pm_init; } #endif @@ -645,9 +601,7 @@ void __init am43xx_init_early(void) void __init am43xx_init_late(void) { - omap_common_late_init(); - omap2_clk_enable_autoidle_all(); - amx3_common_pm_init(); + omap_pm_soc_init = amx3_common_pm_init; } #endif @@ -675,9 +629,7 @@ void __init omap4430_init_early(void) void __init omap4430_init_late(void) { - omap_common_late_init(); - omap4_pm_init(); - omap2_clk_enable_autoidle_all(); + omap_pm_soc_init = omap4_pm_init; } #endif @@ -703,9 +655,7 @@ void __init omap5_init_early(void) void __init omap5_init_late(void) { - omap_common_late_init(); - omap4_pm_init(); - omap2_clk_enable_autoidle_all(); + omap_pm_soc_init = omap4_pm_init; } #endif @@ -728,9 +678,7 @@ void __init dra7xx_init_early(void) void __init dra7xx_init_late(void) { - omap_common_late_init(); - omap4_pm_init(); - omap2_clk_enable_autoidle_all(); + omap_pm_soc_init = omap4_pm_init; } #endif diff --git a/arch/arm/mach-omap2/omap-pm-noop.c b/arch/arm/mach-omap2/omap-pm-noop.c deleted file mode 100644 index 4ead077ea4e7..000000000000 --- a/arch/arm/mach-omap2/omap-pm-noop.c +++ /dev/null @@ -1,176 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * omap-pm-noop.c - OMAP power management interface - dummy version - * - * This code implements the OMAP power management interface to - * drivers, CPUIdle, CPUFreq, and DSP Bridge. It is strictly for - * debug/demonstration use, as it does nothing but printk() whenever a - * function is called (when DEBUG is defined, below) - * - * Copyright (C) 2008-2009 Texas Instruments, Inc. - * Copyright (C) 2008-2009 Nokia Corporation - * Paul Walmsley - * - * Interface developed by (in alphabetical order): - * Karthik Dasu, Tony Lindgren, Rajendra Nayak, Sakari Poussa, Veeramanikandan - * Raju, Anand Sawant, Igor Stoppa, Paul Walmsley, Richard Woodruff - */ - -#undef DEBUG - -#include -#include -#include -#include - -#include "omap_device.h" -#include "omap-pm.h" - -static bool off_mode_enabled; -static int dummy_context_loss_counter; - -/* - * Device-driver-originated constraints (via board-*.c files) - */ - -int omap_pm_set_max_mpu_wakeup_lat(struct device *dev, long t) -{ - if (!dev || t < -1) { - WARN(1, "OMAP PM: %s: invalid parameter(s)", __func__); - return -EINVAL; - } - - if (t == -1) - pr_debug("OMAP PM: remove max MPU wakeup latency constraint: dev %s\n", - dev_name(dev)); - else - pr_debug("OMAP PM: add max MPU wakeup latency constraint: dev %s, t = %ld usec\n", - dev_name(dev), t); - - /* - * For current Linux, this needs to map the MPU to a - * powerdomain, then go through the list of current max lat - * constraints on the MPU and find the smallest. If - * the latency constraint has changed, the code should - * recompute the state to enter for the next powerdomain - * state. - * - * TI CDP code can call constraint_set here. - */ - - return 0; -} - -int omap_pm_set_min_bus_tput(struct device *dev, u8 agent_id, unsigned long r) -{ - if (!dev || (agent_id != OCP_INITIATOR_AGENT && - agent_id != OCP_TARGET_AGENT)) { - WARN(1, "OMAP PM: %s: invalid parameter(s)", __func__); - return -EINVAL; - } - - if (r == 0) - pr_debug("OMAP PM: remove min bus tput constraint: dev %s for agent_id %d\n", - dev_name(dev), agent_id); - else - pr_debug("OMAP PM: add min bus tput constraint: dev %s for agent_id %d: rate %ld KiB\n", - dev_name(dev), agent_id, r); - - /* - * This code should model the interconnect and compute the - * required clock frequency, convert that to a VDD2 OPP ID, then - * set the VDD2 OPP appropriately. - * - * TI CDP code can call constraint_set here on the VDD2 OPP. - */ - - return 0; -} - -/* - * DSP Bridge-specific constraints - */ - - -/** - * omap_pm_enable_off_mode - notify OMAP PM that off-mode is enabled - * - * Intended for use only by OMAP PM core code to notify this layer - * that off mode has been enabled. - */ -void omap_pm_enable_off_mode(void) -{ - off_mode_enabled = true; -} - -/** - * omap_pm_disable_off_mode - notify OMAP PM that off-mode is disabled - * - * Intended for use only by OMAP PM core code to notify this layer - * that off mode has been disabled. - */ -void omap_pm_disable_off_mode(void) -{ - off_mode_enabled = false; -} - -/* - * Device context loss tracking - */ - -#ifdef CONFIG_ARCH_OMAP2PLUS - -int omap_pm_get_dev_context_loss_count(struct device *dev) -{ - struct platform_device *pdev = to_platform_device(dev); - int count; - - if (WARN_ON(!dev)) - return -ENODEV; - - if (dev->pm_domain == &omap_device_pm_domain) { - count = omap_device_get_context_loss_count(pdev); - } else { - WARN_ONCE(off_mode_enabled, "omap_pm: using dummy context loss counter; device %s should be converted to omap_device", - dev_name(dev)); - - count = dummy_context_loss_counter; - - if (off_mode_enabled) { - count++; - /* - * Context loss count has to be a non-negative value. - * Clear the sign bit to get a value range from 0 to - * INT_MAX. - */ - count &= INT_MAX; - dummy_context_loss_counter = count; - } - } - - pr_debug("OMAP PM: context loss count for dev %s = %d\n", - dev_name(dev), count); - - return count; -} - -#else - -int omap_pm_get_dev_context_loss_count(struct device *dev) -{ - return dummy_context_loss_counter; -} - -#endif - -/* Should be called before clk framework init */ -int __init omap_pm_if_early_init(void) -{ - return 0; -} - -/* Must be called after clock framework is initialized */ -int __init omap_pm_if_init(void) -{ - return 0; -} diff --git a/arch/arm/mach-omap2/omap-pm.h b/arch/arm/mach-omap2/omap-pm.h deleted file mode 100644 index 5ba5df47f91b..000000000000 --- a/arch/arm/mach-omap2/omap-pm.h +++ /dev/null @@ -1,161 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* - * omap-pm.h - OMAP power management interface - * - * Copyright (C) 2008-2010 Texas Instruments, Inc. - * Copyright (C) 2008-2010 Nokia Corporation - * Paul Walmsley - * - * Interface developed by (in alphabetical order): Karthik Dasu, Jouni - * Högander, Tony Lindgren, Rajendra Nayak, Sakari Poussa, - * Veeramanikandan Raju, Anand Sawant, Igor Stoppa, Paul Walmsley, - * Richard Woodruff - */ - -#ifndef ASM_ARM_ARCH_OMAP_OMAP_PM_H -#define ASM_ARM_ARCH_OMAP_OMAP_PM_H - -#include -#include -#include -#include - -/* - * agent_id values for use with omap_pm_set_min_bus_tput(): - * - * OCP_INITIATOR_AGENT is only valid for devices that can act as - * initiators -- it represents the device's L3 interconnect - * connection. OCP_TARGET_AGENT represents the device's L4 - * interconnect connection. - */ -#define OCP_TARGET_AGENT 1 -#define OCP_INITIATOR_AGENT 2 - -/** - * omap_pm_if_early_init - OMAP PM init code called before clock fw init - * @mpu_opp_table: array ptr to struct omap_opp for MPU - * @dsp_opp_table: array ptr to struct omap_opp for DSP - * @l3_opp_table : array ptr to struct omap_opp for CORE - * - * Initialize anything that must be configured before the clock - * framework starts. The "_if_" is to avoid name collisions with the - * PM idle-loop code. - */ -int __init omap_pm_if_early_init(void); - -/** - * omap_pm_if_init - OMAP PM init code called after clock fw init - * - * The main initialization code. OPP tables are passed in here. The - * "_if_" is to avoid name collisions with the PM idle-loop code. - */ -int __init omap_pm_if_init(void); - -/* - * Device-driver-originated constraints (via board-*.c files, platform_data) - */ - - -/** - * omap_pm_set_max_mpu_wakeup_lat - set the maximum MPU wakeup latency - * @dev: struct device * requesting the constraint - * @t: maximum MPU wakeup latency in microseconds - * - * Request that the maximum interrupt latency for the MPU to be no - * greater than @t microseconds. "Interrupt latency" in this case is - * defined as the elapsed time from the occurrence of a hardware or - * timer interrupt to the time when the device driver's interrupt - * service routine has been entered by the MPU. - * - * It is intended that underlying PM code will use this information to - * determine what power state to put the MPU powerdomain into, and - * possibly the CORE powerdomain as well, since interrupt handling - * code currently runs from SDRAM. Advanced PM or board*.c code may - * also configure interrupt controller priorities, OCP bus priorities, - * CPU speed(s), etc. - * - * This function will not affect device wakeup latency, e.g., time - * elapsed from when a device driver enables a hardware device with - * clk_enable(), to when the device is ready for register access or - * other use. To control this device wakeup latency, use - * omap_pm_set_max_dev_wakeup_lat() - * - * Multiple calls to omap_pm_set_max_mpu_wakeup_lat() will replace the - * previous t value. To remove the latency target for the MPU, call - * with t = -1. - * - * XXX This constraint will be deprecated soon in favor of the more - * general omap_pm_set_max_dev_wakeup_lat() - * - * Returns -EINVAL for an invalid argument, -ERANGE if the constraint - * is not satisfiable, or 0 upon success. - */ -int omap_pm_set_max_mpu_wakeup_lat(struct device *dev, long t); - - -/** - * omap_pm_set_min_bus_tput - set minimum bus throughput needed by device - * @dev: struct device * requesting the constraint - * @tbus_id: interconnect to operate on (OCP_{INITIATOR,TARGET}_AGENT) - * @r: minimum throughput (in KiB/s) - * - * Request that the minimum data throughput on the OCP interconnect - * attached to device @dev interconnect agent @tbus_id be no less - * than @r KiB/s. - * - * It is expected that the OMAP PM or bus code will use this - * information to set the interconnect clock to run at the lowest - * possible speed that satisfies all current system users. The PM or - * bus code will adjust the estimate based on its model of the bus, so - * device driver authors should attempt to specify an accurate - * quantity for their device use case, and let the PM or bus code - * overestimate the numbers as necessary to handle request/response - * latency, other competing users on the system, etc. On OMAP2/3, if - * a driver requests a minimum L4 interconnect speed constraint, the - * code will also need to add an minimum L3 interconnect speed - * constraint, - * - * Multiple calls to omap_pm_set_min_bus_tput() will replace the - * previous rate value for this device. To remove the interconnect - * throughput restriction for this device, call with r = 0. - * - * Returns -EINVAL for an invalid argument, -ERANGE if the constraint - * is not satisfiable, or 0 upon success. - */ -int omap_pm_set_min_bus_tput(struct device *dev, u8 agent_id, unsigned long r); - - -/* - * CPUFreq-originated constraint - * - * In the future, this should be handled by custom OPP clocktype - * functions. - */ - - -/* - * Device context loss tracking - */ - -/** - * omap_pm_get_dev_context_loss_count - return count of times dev has lost ctx - * @dev: struct device * - * - * This function returns the number of times that the device @dev has - * lost its internal context. This generally occurs on a powerdomain - * transition to OFF. Drivers use this as an optimization to avoid restoring - * context if the device hasn't lost it. To use, drivers should initially - * call this in their context save functions and store the result. Early in - * the driver's context restore function, the driver should call this function - * again, and compare the result to the stored counter. If they differ, the - * driver must restore device context. If the number of context losses - * exceeds the maximum positive integer, the function will wrap to 0 and - * continue counting. Returns the number of context losses for this device, - * or negative value upon error. - */ -int omap_pm_get_dev_context_loss_count(struct device *dev); - -void omap_pm_enable_off_mode(void); -void omap_pm_disable_off_mode(void); - -#endif diff --git a/arch/arm/mach-omap2/omap_device.c b/arch/arm/mach-omap2/omap_device.c index 3b829a50d1db..41c7b905980a 100644 --- a/arch/arm/mach-omap2/omap_device.c +++ b/arch/arm/mach-omap2/omap_device.c @@ -143,7 +143,7 @@ static int omap_device_build_from_dt(struct platform_device *pdev) struct resource res; const char *oh_name; int oh_cnt, i, ret = 0; - bool device_active = false; + bool device_active = false, skip_pm_domain = false; oh_cnt = of_property_count_strings(node, "ti,hwmods"); if (oh_cnt <= 0) { @@ -151,11 +151,18 @@ static int omap_device_build_from_dt(struct platform_device *pdev) return -ENODEV; } + /* SDMA still needs special handling for omap_device_build() */ + ret = of_property_read_string_index(node, "ti,hwmods", 0, &oh_name); + if (!ret && (!strncmp("dma_system", oh_name, 10) || + !strncmp("dma", oh_name, 3))) + skip_pm_domain = true; + /* Use ti-sysc driver instead of omap_device? */ - if (!omap_hwmod_parse_module_range(NULL, node, &res)) + if (!skip_pm_domain && + !omap_hwmod_parse_module_range(NULL, node, &res)) return -ENODEV; - hwmods = kzalloc(sizeof(struct omap_hwmod *) * oh_cnt, GFP_KERNEL); + hwmods = kcalloc(oh_cnt, sizeof(struct omap_hwmod *), GFP_KERNEL); if (!hwmods) { ret = -ENOMEM; goto odbfd_exit; @@ -191,11 +198,12 @@ static int omap_device_build_from_dt(struct platform_device *pdev) r->name = dev_name(&pdev->dev); } - dev_pm_domain_set(&pdev->dev, &omap_device_pm_domain); - - if (device_active) { - omap_device_enable(pdev); - pm_runtime_set_active(&pdev->dev); + if (!skip_pm_domain) { + dev_pm_domain_set(&pdev->dev, &omap_device_pm_domain); + if (device_active) { + omap_device_enable(pdev); + pm_runtime_set_active(&pdev->dev); + } } odbfd_exit1: @@ -405,7 +413,7 @@ omap_device_copy_resources(struct omap_hwmod *oh, goto error; } - res = kzalloc(sizeof(*res) * 2, GFP_KERNEL); + res = kcalloc(2, sizeof(*res), GFP_KERNEL); if (!res) return -ENOMEM; diff --git a/arch/arm/mach-omap2/omap_hwmod.c b/arch/arm/mach-omap2/omap_hwmod.c index e7d23e200ecc..2ceffd85dd3d 100644 --- a/arch/arm/mach-omap2/omap_hwmod.c +++ b/arch/arm/mach-omap2/omap_hwmod.c @@ -481,7 +481,7 @@ static int _wait_softreset_complete(struct omap_hwmod *oh) sysc = oh->class->sysc; - if (sysc->sysc_flags & SYSS_HAS_RESET_STATUS) + if (sysc->sysc_flags & SYSS_HAS_RESET_STATUS && sysc->syss_offs > 0) omap_test_timeout((omap_hwmod_read(oh, sysc->syss_offs) & SYSS_RESETDONE_MASK), MAX_MODULE_SOFTRESET_WAIT, c); @@ -3171,19 +3171,19 @@ static int omap_hwmod_init_regbits(struct device *dev, */ int omap_hwmod_init_reg_offs(struct device *dev, const struct ti_sysc_module_data *data, - u32 *rev_offs, u32 *sysc_offs, u32 *syss_offs) + s32 *rev_offs, s32 *sysc_offs, s32 *syss_offs) { - *rev_offs = 0; + *rev_offs = -ENODEV; *sysc_offs = 0; *syss_offs = 0; - if (data->offsets[SYSC_REVISION] > 0) + if (data->offsets[SYSC_REVISION] >= 0) *rev_offs = data->offsets[SYSC_REVISION]; - if (data->offsets[SYSC_SYSCONFIG] > 0) + if (data->offsets[SYSC_SYSCONFIG] >= 0) *sysc_offs = data->offsets[SYSC_SYSCONFIG]; - if (data->offsets[SYSC_SYSSTATUS] > 0) + if (data->offsets[SYSC_SYSSTATUS] >= 0) *syss_offs = data->offsets[SYSC_SYSSTATUS]; return 0; @@ -3312,8 +3312,8 @@ static int omap_hwmod_check_module(struct device *dev, struct omap_hwmod *oh, const struct ti_sysc_module_data *data, struct sysc_regbits *sysc_fields, - u32 rev_offs, u32 sysc_offs, - u32 syss_offs, u32 sysc_flags, + s32 rev_offs, s32 sysc_offs, + s32 syss_offs, u32 sysc_flags, u32 idlemodes) { if (!oh->class->sysc) @@ -3365,7 +3365,7 @@ static int omap_hwmod_check_module(struct device *dev, int omap_hwmod_allocate_module(struct device *dev, struct omap_hwmod *oh, const struct ti_sysc_module_data *data, struct sysc_regbits *sysc_fields, - u32 rev_offs, u32 sysc_offs, u32 syss_offs, + s32 rev_offs, s32 sysc_offs, s32 syss_offs, u32 sysc_flags, u32 idlemodes) { struct omap_hwmod_class_sysconfig *sysc; @@ -3425,7 +3425,8 @@ int omap_hwmod_init_module(struct device *dev, { struct omap_hwmod *oh; struct sysc_regbits *sysc_fields; - u32 rev_offs, sysc_offs, syss_offs, sysc_flags, idlemodes; + s32 rev_offs, sysc_offs, syss_offs; + u32 sysc_flags, idlemodes; int error; if (!dev || !data) diff --git a/arch/arm/mach-omap2/omap_hwmod.h b/arch/arm/mach-omap2/omap_hwmod.h index c7122abbf977..b70cdc21f8a2 100644 --- a/arch/arm/mach-omap2/omap_hwmod.h +++ b/arch/arm/mach-omap2/omap_hwmod.h @@ -317,9 +317,9 @@ struct omap_hwmod_ocp_if { * then this field has to be populated with the correct offset structure. */ struct omap_hwmod_class_sysconfig { - u32 rev_offs; - u32 sysc_offs; - u32 syss_offs; + s32 rev_offs; + s32 sysc_offs; + s32 syss_offs; u16 sysc_flags; struct sysc_regbits *sysc_fields; u8 srst_udelay; diff --git a/arch/arm/mach-omap2/omap_hwmod_2420_data.c b/arch/arm/mach-omap2/omap_hwmod_2420_data.c index fe66cf247874..d684fac8f592 100644 --- a/arch/arm/mach-omap2/omap_hwmod_2420_data.c +++ b/arch/arm/mach-omap2/omap_hwmod_2420_data.c @@ -13,7 +13,7 @@ * XXX these should be marked initdata for multi-OMAP kernels */ -#include +#include #include #include "omap_hwmod.h" diff --git a/arch/arm/mach-omap2/omap_hwmod_2430_data.c b/arch/arm/mach-omap2/omap_hwmod_2430_data.c index 74eefd30518c..abef9f6f9bf5 100644 --- a/arch/arm/mach-omap2/omap_hwmod_2430_data.c +++ b/arch/arm/mach-omap2/omap_hwmod_2430_data.c @@ -13,7 +13,7 @@ * XXX these should be marked initdata for multi-OMAP kernels */ -#include +#include #include #include diff --git a/arch/arm/mach-omap2/omap_hwmod_33xx_43xx_ipblock_data.c b/arch/arm/mach-omap2/omap_hwmod_33xx_43xx_ipblock_data.c index 5efe91c6e95b..9ded7bf972e7 100644 --- a/arch/arm/mach-omap2/omap_hwmod_33xx_43xx_ipblock_data.c +++ b/arch/arm/mach-omap2/omap_hwmod_33xx_43xx_ipblock_data.c @@ -629,6 +629,7 @@ struct omap_hwmod am33xx_gpmc_hwmod = { /* 'i2c' class */ static struct omap_hwmod_class_sysconfig am33xx_i2c_sysc = { + .rev_offs = 0, .sysc_offs = 0x0010, .syss_offs = 0x0090, .sysc_flags = (SYSC_HAS_AUTOIDLE | SYSC_HAS_CLOCKACTIVITY | diff --git a/arch/arm/mach-omap2/omap_hwmod_33xx_data.c b/arch/arm/mach-omap2/omap_hwmod_33xx_data.c index 53e1ac3724f2..c9483bc06228 100644 --- a/arch/arm/mach-omap2/omap_hwmod_33xx_data.c +++ b/arch/arm/mach-omap2/omap_hwmod_33xx_data.c @@ -14,7 +14,7 @@ * GNU General Public License for more details. */ -#include +#include #include "omap_hwmod.h" #include "omap_hwmod_common_data.h" diff --git a/arch/arm/mach-omap2/omap_hwmod_3xxx_data.c b/arch/arm/mach-omap2/omap_hwmod_3xxx_data.c index 23336b6c7125..23e6a41a18eb 100644 --- a/arch/arm/mach-omap2/omap_hwmod_3xxx_data.c +++ b/arch/arm/mach-omap2/omap_hwmod_3xxx_data.c @@ -15,7 +15,7 @@ * XXX these should be marked initdata for multi-OMAP kernels */ -#include +#include #include #include @@ -885,6 +885,7 @@ static struct omap_hwmod omap3xxx_dma_system_hwmod = { */ static struct omap_hwmod_class_sysconfig omap3xxx_mcbsp_sysc = { + .rev_offs = -ENODEV, .sysc_offs = 0x008c, .sysc_flags = (SYSC_HAS_CLOCKACTIVITY | SYSC_HAS_ENAWAKEUP | SYSC_HAS_SIDLEMODE | SYSC_HAS_SOFTRESET), @@ -990,6 +991,7 @@ static struct omap_hwmod omap3xxx_mcbsp5_hwmod = { /* 'mcbsp sidetone' class */ static struct omap_hwmod_class_sysconfig omap3xxx_mcbsp_sidetone_sysc = { + .rev_offs = -ENODEV, .sysc_offs = 0x0010, .sysc_flags = SYSC_HAS_AUTOIDLE, .sysc_fields = &omap_hwmod_sysc_type1, @@ -1018,6 +1020,7 @@ static struct omap_hwmod omap3xxx_mcbsp3_sidetone_hwmod = { /* SR common */ static struct omap_hwmod_class_sysconfig omap34xx_sr_sysc = { + .rev_offs = -ENODEV, .sysc_offs = 0x24, .sysc_flags = (SYSC_HAS_CLOCKACTIVITY | SYSC_NO_CACHE), .sysc_fields = &omap34xx_sr_sysc_fields, @@ -1030,6 +1033,7 @@ static struct omap_hwmod_class omap34xx_smartreflex_hwmod_class = { }; static struct omap_hwmod_class_sysconfig omap36xx_sr_sysc = { + .rev_offs = -ENODEV, .sysc_offs = 0x38, .idlemodes = (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART), .sysc_flags = (SYSC_HAS_SIDLEMODE | SYSC_HAS_ENAWAKEUP | diff --git a/arch/arm/mach-omap2/omap_hwmod_43xx_data.c b/arch/arm/mach-omap2/omap_hwmod_43xx_data.c index 5f73b730d4fc..aa271ac5ebac 100644 --- a/arch/arm/mach-omap2/omap_hwmod_43xx_data.c +++ b/arch/arm/mach-omap2/omap_hwmod_43xx_data.c @@ -378,6 +378,7 @@ static struct omap_hwmod am43xx_usb_otg_ss1_hwmod = { }; static struct omap_hwmod_class_sysconfig am43xx_qspi_sysc = { + .rev_offs = 0, .sysc_offs = 0x0010, .sysc_flags = SYSC_HAS_SIDLEMODE, .idlemodes = (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART | diff --git a/arch/arm/mach-omap2/omap_hwmod_44xx_data.c b/arch/arm/mach-omap2/omap_hwmod_44xx_data.c index e4f8ae9cd637..a95dbac57a81 100644 --- a/arch/arm/mach-omap2/omap_hwmod_44xx_data.c +++ b/arch/arm/mach-omap2/omap_hwmod_44xx_data.c @@ -23,7 +23,7 @@ #include #include #include -#include +#include #include @@ -1360,6 +1360,7 @@ static struct omap_hwmod omap44xx_hsi_hwmod = { */ static struct omap_hwmod_class_sysconfig omap44xx_i2c_sysc = { + .rev_offs = 0, .sysc_offs = 0x0010, .syss_offs = 0x0090, .sysc_flags = (SYSC_HAS_AUTOIDLE | SYSC_HAS_CLOCKACTIVITY | @@ -1634,6 +1635,7 @@ static struct omap_hwmod omap44xx_mailbox_hwmod = { /* The IP is not compliant to type1 / type2 scheme */ static struct omap_hwmod_class_sysconfig omap44xx_mcasp_sysc = { + .rev_offs = 0, .sysc_offs = 0x0004, .sysc_flags = SYSC_HAS_SIDLEMODE, .idlemodes = (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART | @@ -1667,6 +1669,7 @@ static struct omap_hwmod omap44xx_mcasp_hwmod = { */ static struct omap_hwmod_class_sysconfig omap44xx_mcbsp_sysc = { + .rev_offs = -ENODEV, .sysc_offs = 0x008c, .sysc_flags = (SYSC_HAS_CLOCKACTIVITY | SYSC_HAS_ENAWAKEUP | SYSC_HAS_SIDLEMODE | SYSC_HAS_SOFTRESET), @@ -2353,6 +2356,7 @@ static struct omap_hwmod omap44xx_slimbus2_hwmod = { /* The IP is not compliant to type1 / type2 scheme */ static struct omap_hwmod_class_sysconfig omap44xx_smartreflex_sysc = { + .rev_offs = -ENODEV, .sysc_offs = 0x0038, .sysc_flags = (SYSC_HAS_ENAWAKEUP | SYSC_HAS_SIDLEMODE), .idlemodes = (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART | diff --git a/arch/arm/mach-omap2/omap_hwmod_54xx_data.c b/arch/arm/mach-omap2/omap_hwmod_54xx_data.c index c72cd84b07ec..115473d441cd 100644 --- a/arch/arm/mach-omap2/omap_hwmod_54xx_data.c +++ b/arch/arm/mach-omap2/omap_hwmod_54xx_data.c @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include @@ -804,6 +804,7 @@ static struct omap_hwmod omap54xx_gpio8_hwmod = { */ static struct omap_hwmod_class_sysconfig omap54xx_i2c_sysc = { + .rev_offs = 0, .sysc_offs = 0x0010, .syss_offs = 0x0090, .sysc_flags = (SYSC_HAS_AUTOIDLE | SYSC_HAS_CLOCKACTIVITY | @@ -974,6 +975,7 @@ static struct omap_hwmod omap54xx_mailbox_hwmod = { */ static struct omap_hwmod_class_sysconfig omap54xx_mcbsp_sysc = { + .rev_offs = -ENODEV, .sysc_offs = 0x008c, .sysc_flags = (SYSC_HAS_CLOCKACTIVITY | SYSC_HAS_ENAWAKEUP | SYSC_HAS_SIDLEMODE | SYSC_HAS_SOFTRESET), @@ -1997,6 +1999,7 @@ static struct omap_hwmod omap54xx_ocp2scp3_hwmod = { */ static struct omap_hwmod_class_sysconfig omap54xx_sata_sysc = { + .rev_offs = 0x00fc, .sysc_offs = 0x0000, .sysc_flags = (SYSC_HAS_MIDLEMODE | SYSC_HAS_SIDLEMODE), .idlemodes = (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART | diff --git a/arch/arm/mach-omap2/omap_hwmod_7xx_data.c b/arch/arm/mach-omap2/omap_hwmod_7xx_data.c index 62352d1e6361..e6c7061a8e73 100644 --- a/arch/arm/mach-omap2/omap_hwmod_7xx_data.c +++ b/arch/arm/mach-omap2/omap_hwmod_7xx_data.c @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include @@ -1070,6 +1070,7 @@ static struct omap_hwmod dra7xx_hdq1w_hwmod = { */ static struct omap_hwmod_class_sysconfig dra7xx_i2c_sysc = { + .rev_offs = 0, .sysc_offs = 0x0010, .syss_offs = 0x0090, .sysc_flags = (SYSC_HAS_AUTOIDLE | SYSC_HAS_CLOCKACTIVITY | @@ -1440,6 +1441,7 @@ static struct omap_hwmod dra7xx_mcspi4_hwmod = { * */ static struct omap_hwmod_class_sysconfig dra7xx_mcasp_sysc = { + .rev_offs = 0, .sysc_offs = 0x0004, .sysc_flags = SYSC_HAS_SIDLEMODE, .idlemodes = (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART), @@ -1898,6 +1900,7 @@ static struct omap_hwmod dra7xx_pciess2_hwmod = { */ static struct omap_hwmod_class_sysconfig dra7xx_qspi_sysc = { + .rev_offs = 0, .sysc_offs = 0x0010, .sysc_flags = SYSC_HAS_SIDLEMODE, .idlemodes = (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART | @@ -1930,6 +1933,7 @@ static struct omap_hwmod dra7xx_qspi_hwmod = { * */ static struct omap_hwmod_class_sysconfig dra7xx_rtcss_sysc = { + .rev_offs = 0x0074, .sysc_offs = 0x0078, .sysc_flags = SYSC_HAS_SIDLEMODE, .idlemodes = (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART | @@ -1965,6 +1969,7 @@ static struct omap_hwmod dra7xx_rtcss_hwmod = { */ static struct omap_hwmod_class_sysconfig dra7xx_sata_sysc = { + .rev_offs = 0x00fc, .sysc_offs = 0x0000, .sysc_flags = (SYSC_HAS_MIDLEMODE | SYSC_HAS_SIDLEMODE), .idlemodes = (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART | @@ -2003,6 +2008,7 @@ static struct omap_hwmod dra7xx_sata_hwmod = { /* The IP is not compliant to type1 / type2 scheme */ static struct omap_hwmod_class_sysconfig dra7xx_smartreflex_sysc = { + .rev_offs = -ENODEV, .sysc_offs = 0x0038, .sysc_flags = (SYSC_HAS_ENAWAKEUP | SYSC_HAS_SIDLEMODE), .idlemodes = (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART | diff --git a/arch/arm/mach-omap2/omap_hwmod_81xx_data.c b/arch/arm/mach-omap2/omap_hwmod_81xx_data.c index 686655f884c1..8e44e2728620 100644 --- a/arch/arm/mach-omap2/omap_hwmod_81xx_data.c +++ b/arch/arm/mach-omap2/omap_hwmod_81xx_data.c @@ -954,6 +954,7 @@ static struct omap_hwmod_ocp_if dm816x_l4_hs__emac1 = { }; static struct omap_hwmod_class_sysconfig dm81xx_sata_sysc = { + .rev_offs = 0x00fc, .sysc_offs = 0x1100, .sysc_flags = SYSC_HAS_SIDLEMODE, .idlemodes = SIDLE_FORCE, diff --git a/arch/arm/mach-omap2/pdata-quirks.c b/arch/arm/mach-omap2/pdata-quirks.c index 6459816c2879..7f02743edbe4 100644 --- a/arch/arm/mach-omap2/pdata-quirks.c +++ b/arch/arm/mach-omap2/pdata-quirks.c @@ -26,14 +26,12 @@ #include #include #include -#include #include #include "common.h" #include "common-board-devices.h" #include "control.h" #include "omap_device.h" -#include "omap-pm.h" #include "omap-secure.h" #include "soc.h" #include "hsmmc.h" @@ -514,18 +512,6 @@ void omap_auxdata_legacy_init(struct device *dev) dev->platform_data = &twl_gpio_auxdata; } -static struct ir_rx51_platform_data __maybe_unused rx51_ir_data = { - .set_max_mpu_wakeup_lat = omap_pm_set_max_mpu_wakeup_lat, -}; - -static struct platform_device __maybe_unused rx51_ir_device = { - .name = "ir_rx51", - .id = -1, - .dev = { - .platform_data = &rx51_ir_data, - }, -}; - #if IS_ENABLED(CONFIG_SND_OMAP_SOC_MCBSP) static struct omap_mcbsp_platform_data mcbsp_pdata; static void __init omap3_mcbsp_init(void) @@ -569,7 +555,6 @@ static struct of_dev_auxdata omap_auxdata_lookup[] = { "480c9000.smartreflex", &omap_sr_pdata[OMAP_SR_MPU]), OF_DEV_AUXDATA("ti,omap3-hsmmc", 0x4809c000, "4809c000.mmc", &mmc_pdata[0]), OF_DEV_AUXDATA("ti,omap3-hsmmc", 0x480b4000, "480b4000.mmc", &mmc_pdata[1]), - OF_DEV_AUXDATA("nokia,n900-ir", 0, "n900-ir", &rx51_ir_data), /* Only on am3517 */ OF_DEV_AUXDATA("ti,davinci_mdio", 0x5c030000, "davinci_mdio.0", NULL), OF_DEV_AUXDATA("ti,am3517-emac", 0x5c000000, "davinci_emac.0", diff --git a/arch/arm/mach-omap2/pm-debug.c b/arch/arm/mach-omap2/pm-debug.c index 5c46ea6756d7..acb698d5780f 100644 --- a/arch/arm/mach-omap2/pm-debug.c +++ b/arch/arm/mach-omap2/pm-debug.c @@ -31,7 +31,6 @@ #include "clock.h" #include "powerdomain.h" #include "clockdomain.h" -#include "omap-pm.h" #include "soc.h" #include "cm2xxx_3xxx.h" @@ -240,10 +239,6 @@ static int option_set(void *data, u64 val) *option = val; if (option == &enable_off_mode) { - if (val) - omap_pm_enable_off_mode(); - else - omap_pm_disable_off_mode(); if (cpu_is_omap34xx()) omap3_pm_off_mode_enable(val); } diff --git a/arch/arm/mach-omap2/pm.c b/arch/arm/mach-omap2/pm.c index 6f68576e5695..ca03af8fe43f 100644 --- a/arch/arm/mach-omap2/pm.c +++ b/arch/arm/mach-omap2/pm.c @@ -16,11 +16,11 @@ #include #include #include +#include #include #include -#include "omap-pm.h" #include "omap_device.h" #include "common.h" @@ -230,16 +230,20 @@ static void __init omap4_init_voltages(void) omap2_set_init_voltage("iva", "dpll_iva_m5x2_ck", "iva"); } -static int __init omap2_common_pm_init(void) +int __maybe_unused omap_pm_nop_init(void) { - omap_pm_if_init(); - return 0; } -omap_postcore_initcall(omap2_common_pm_init); + +int (*omap_pm_soc_init)(void); int __init omap2_common_pm_late_init(void) { + int error; + + if (!omap_pm_soc_init) + return 0; + /* Init the voltage layer */ omap3_twl_init(); omap4_twl_init(); @@ -252,5 +256,12 @@ int __init omap2_common_pm_late_init(void) /* Smartreflex device init */ omap_devinit_smartreflex(); + error = omap_pm_soc_init(); + if (error) + pr_warn("%s: pm soc init failed: %i\n", __func__, error); + + omap2_clk_enable_autoidle_all(); + return 0; } +omap_late_initcall(omap2_common_pm_late_init); diff --git a/arch/arm/mach-omap2/pm33xx-core.c b/arch/arm/mach-omap2/pm33xx-core.c index 93c0b5ba9f09..9b3755a2e2ec 100644 --- a/arch/arm/mach-omap2/pm33xx-core.c +++ b/arch/arm/mach-omap2/pm33xx-core.c @@ -173,7 +173,7 @@ static struct am33xx_pm_platform_data *am33xx_pm_get_pdata(void) return NULL; } -void __init amx3_common_pm_init(void) +int __init amx3_common_pm_init(void) { struct am33xx_pm_platform_data *pdata; struct platform_device_info devinfo; @@ -186,4 +186,6 @@ void __init amx3_common_pm_init(void) devinfo.size_data = sizeof(*pdata); devinfo.id = -1; platform_device_register_full(&devinfo); + + return 0; } diff --git a/arch/arm/mach-omap2/pm44xx.c b/arch/arm/mach-omap2/pm44xx.c index b3870220612e..78e1ace7d17d 100644 --- a/arch/arm/mach-omap2/pm44xx.c +++ b/arch/arm/mach-omap2/pm44xx.c @@ -131,6 +131,19 @@ static int __init pwrdms_setup(struct powerdomain *pwrdm, void *unused) return 0; } + /* + * Bootloader or kexec boot may have LOGICRETSTATE cleared + * for some domains. This is the case when kexec booting from + * Android kernels that support off mode for example. + * Make sure it's set at least for core and per, otherwise + * we currently will see lost GPIO interrupts for wlcore and + * smsc911x at least if per hits retention during idle. + */ + if (!strncmp(pwrdm->name, "core", 4) || + !strncmp(pwrdm->name, "l4per", 5) || + !strncmp(pwrdm->name, "wkup", 4)) + pwrdm_set_logic_retst(pwrdm, PWRDM_POWER_RET); + pwrst = kmalloc(sizeof(struct power_state), GFP_ATOMIC); if (!pwrst) return -ENOMEM; diff --git a/arch/arm/mach-omap2/powerdomain.c b/arch/arm/mach-omap2/powerdomain.c index 1e6a967cd2d5..1a0f69c0a376 100644 --- a/arch/arm/mach-omap2/powerdomain.c +++ b/arch/arm/mach-omap2/powerdomain.c @@ -14,6 +14,7 @@ */ #undef DEBUG +#include #include #include #include @@ -39,6 +40,9 @@ #define PWRDM_TRACE_STATES_FLAG (1<<31) +void pwrdms_save_context(void); +void pwrdms_restore_context(void); + enum { PWRDM_STATE_NOW = 0, PWRDM_STATE_PREV, @@ -333,6 +337,22 @@ int pwrdm_register_pwrdms(struct powerdomain **ps) return 0; } +static int cpu_notifier(struct notifier_block *nb, unsigned long cmd, void *v) +{ + switch (cmd) { + case CPU_CLUSTER_PM_ENTER: + if (enable_off_mode) + pwrdms_save_context(); + break; + case CPU_CLUSTER_PM_EXIT: + if (enable_off_mode) + pwrdms_restore_context(); + break; + } + + return NOTIFY_OK; +} + /** * pwrdm_complete_init - set up the powerdomain layer * @@ -347,6 +367,7 @@ int pwrdm_register_pwrdms(struct powerdomain **ps) int pwrdm_complete_init(void) { struct powerdomain *temp_p; + static struct notifier_block nb; if (list_empty(&pwrdm_list)) return -EACCES; @@ -354,6 +375,12 @@ int pwrdm_complete_init(void) list_for_each_entry(temp_p, &pwrdm_list, node) pwrdm_set_next_pwrst(temp_p, PWRDM_POWER_ON); + /* Only AM43XX can lose pwrdm context during rtc-ddr suspend */ + if (soc_is_am43xx()) { + nb.notifier_call = cpu_notifier; + cpu_pm_register_notifier(&nb); + } + return 0; } @@ -1199,3 +1226,63 @@ bool pwrdm_can_ever_lose_context(struct powerdomain *pwrdm) return 0; } + +/** + * pwrdm_save_context - save powerdomain registers + * + * Register state is going to be lost due to a suspend or hibernate + * event. Save the powerdomain registers. + */ +static int pwrdm_save_context(struct powerdomain *pwrdm, void *unused) +{ + if (arch_pwrdm && arch_pwrdm->pwrdm_save_context) + arch_pwrdm->pwrdm_save_context(pwrdm); + return 0; +} + +/** + * pwrdm_save_context - restore powerdomain registers + * + * Restore powerdomain control registers after a suspend or resume + * event. + */ +static int pwrdm_restore_context(struct powerdomain *pwrdm, void *unused) +{ + if (arch_pwrdm && arch_pwrdm->pwrdm_restore_context) + arch_pwrdm->pwrdm_restore_context(pwrdm); + return 0; +} + +static int pwrdm_lost_power(struct powerdomain *pwrdm, void *unused) +{ + int state; + + /* + * Power has been lost across all powerdomains, increment the + * counter. + */ + + state = pwrdm_read_pwrst(pwrdm); + if (state != PWRDM_POWER_OFF) { + pwrdm->state_counter[state]++; + pwrdm->state_counter[PWRDM_POWER_OFF]++; + } + pwrdm->state = state; + + return 0; +} + +void pwrdms_save_context(void) +{ + pwrdm_for_each(pwrdm_save_context, NULL); +} + +void pwrdms_restore_context(void) +{ + pwrdm_for_each(pwrdm_restore_context, NULL); +} + +void pwrdms_lost_power(void) +{ + pwrdm_for_each(pwrdm_lost_power, NULL); +} diff --git a/arch/arm/mach-omap2/powerdomain.h b/arch/arm/mach-omap2/powerdomain.h index 28a796ce07d7..9a907fb14044 100644 --- a/arch/arm/mach-omap2/powerdomain.h +++ b/arch/arm/mach-omap2/powerdomain.h @@ -144,6 +144,7 @@ struct powerdomain { s64 timer; s64 state_timer[PWRDM_MAX_PWRSTS]; #endif + u32 context; }; /** @@ -198,6 +199,8 @@ struct pwrdm_ops { int (*pwrdm_set_lowpwrstchange)(struct powerdomain *pwrdm); int (*pwrdm_wait_transition)(struct powerdomain *pwrdm); int (*pwrdm_has_voltdm)(void); + void (*pwrdm_save_context)(struct powerdomain *pwrdm); + void (*pwrdm_restore_context)(struct powerdomain *pwrdm); }; int pwrdm_register_platform_funcs(struct pwrdm_ops *custom_funcs); @@ -273,4 +276,8 @@ extern struct powerdomain gfx_omap2_pwrdm; extern void pwrdm_lock(struct powerdomain *pwrdm); extern void pwrdm_unlock(struct powerdomain *pwrdm); +extern void pwrdms_save_context(void); +extern void pwrdms_restore_context(void); + +extern void pwrdms_lost_power(void); #endif diff --git a/arch/arm/mach-omap2/prm33xx.c b/arch/arm/mach-omap2/prm33xx.c index ebaf80d72a10..d5141669c28d 100644 --- a/arch/arm/mach-omap2/prm33xx.c +++ b/arch/arm/mach-omap2/prm33xx.c @@ -342,6 +342,35 @@ static void am33xx_prm_global_warm_sw_reset(void) AM33XX_PRM_RSTCTRL_OFFSET); } +static void am33xx_pwrdm_save_context(struct powerdomain *pwrdm) +{ + pwrdm->context = am33xx_prm_read_reg(pwrdm->prcm_offs, + pwrdm->pwrstctrl_offs); + /* + * Do not save LOWPOWERSTATECHANGE, writing a 1 indicates a request, + * reading back a 1 indicates a request in progress. + */ + pwrdm->context &= ~AM33XX_LOWPOWERSTATECHANGE_MASK; +} + +static void am33xx_pwrdm_restore_context(struct powerdomain *pwrdm) +{ + int st, ctrl; + + st = am33xx_prm_read_reg(pwrdm->prcm_offs, + pwrdm->pwrstst_offs); + + am33xx_prm_write_reg(pwrdm->context, pwrdm->prcm_offs, + pwrdm->pwrstctrl_offs); + + /* Make sure we only wait for a transition if there is one */ + st &= OMAP_POWERSTATEST_MASK; + ctrl = OMAP_POWERSTATEST_MASK & pwrdm->context; + + if (st != ctrl) + am33xx_pwrdm_wait_transition(pwrdm); +} + struct pwrdm_ops am33xx_pwrdm_operations = { .pwrdm_set_next_pwrst = am33xx_pwrdm_set_next_pwrst, .pwrdm_read_next_pwrst = am33xx_pwrdm_read_next_pwrst, @@ -357,6 +386,8 @@ struct pwrdm_ops am33xx_pwrdm_operations = { .pwrdm_set_mem_retst = am33xx_pwrdm_set_mem_retst, .pwrdm_wait_transition = am33xx_pwrdm_wait_transition, .pwrdm_has_voltdm = am33xx_check_vcvp, + .pwrdm_save_context = am33xx_pwrdm_save_context, + .pwrdm_restore_context = am33xx_pwrdm_restore_context, }; static struct prm_ll_data am33xx_prm_ll_data = { diff --git a/arch/arm/mach-omap2/prm44xx.c b/arch/arm/mach-omap2/prm44xx.c index acb95936dfe7..7b95729e8359 100644 --- a/arch/arm/mach-omap2/prm44xx.c +++ b/arch/arm/mach-omap2/prm44xx.c @@ -12,6 +12,7 @@ * published by the Free Software Foundation. */ +#include #include #include #include @@ -30,6 +31,7 @@ #include "prcm44xx.h" #include "prminst44xx.h" #include "powerdomain.h" +#include "pm.h" /* Static data */ @@ -57,6 +59,13 @@ static struct omap_prcm_irq_setup omap4_prcm_irq_setup = { .reconfigure_io_chain = &omap44xx_prm_reconfigure_io_chain, }; +struct omap_prm_irq_context { + unsigned long irq_enable; + unsigned long pm_ctrl; +}; + +static struct omap_prm_irq_context omap_prm_context; + /* * omap44xx_prm_reset_src_map - map from bits in the PRM_RSTST * hardware register (which are specific to OMAP44xx SoCs) to reset @@ -667,6 +676,54 @@ static int omap4_check_vcvp(void) return 0; } +/** + * omap4_pwrdm_save_context - Saves the powerdomain state + * @pwrdm: pointer to individual powerdomain + * + * The function saves the powerdomain state control information. + * This is needed in rtc+ddr modes where we lose powerdomain context. + */ +static void omap4_pwrdm_save_context(struct powerdomain *pwrdm) +{ + pwrdm->context = omap4_prminst_read_inst_reg(pwrdm->prcm_partition, + pwrdm->prcm_offs, + pwrdm->pwrstctrl_offs); + + /* + * Do not save LOWPOWERSTATECHANGE, writing a 1 indicates a request, + * reading back a 1 indicates a request in progress. + */ + pwrdm->context &= ~OMAP4430_LOWPOWERSTATECHANGE_MASK; +} + +/** + * omap4_pwrdm_restore_context - Restores the powerdomain state + * @pwrdm: pointer to individual powerdomain + * + * The function restores the powerdomain state control information. + * This is needed in rtc+ddr modes where we lose powerdomain context. + */ +static void omap4_pwrdm_restore_context(struct powerdomain *pwrdm) +{ + int st, ctrl; + + st = omap4_prminst_read_inst_reg(pwrdm->prcm_partition, + pwrdm->prcm_offs, + pwrdm->pwrstctrl_offs); + + omap4_prminst_write_inst_reg(pwrdm->context, + pwrdm->prcm_partition, + pwrdm->prcm_offs, + pwrdm->pwrstctrl_offs); + + /* Make sure we only wait for a transition if there is one */ + st &= OMAP_POWERSTATEST_MASK; + ctrl = OMAP_POWERSTATEST_MASK & pwrdm->context; + + if (st != ctrl) + omap4_pwrdm_wait_transition(pwrdm); +} + struct pwrdm_ops omap4_pwrdm_operations = { .pwrdm_set_next_pwrst = omap4_pwrdm_set_next_pwrst, .pwrdm_read_next_pwrst = omap4_pwrdm_read_next_pwrst, @@ -685,10 +742,50 @@ struct pwrdm_ops omap4_pwrdm_operations = { .pwrdm_set_mem_retst = omap4_pwrdm_set_mem_retst, .pwrdm_wait_transition = omap4_pwrdm_wait_transition, .pwrdm_has_voltdm = omap4_check_vcvp, + .pwrdm_save_context = omap4_pwrdm_save_context, + .pwrdm_restore_context = omap4_pwrdm_restore_context, }; static int omap44xx_prm_late_init(void); +void prm_save_context(void) +{ + omap_prm_context.irq_enable = + omap4_prm_read_inst_reg(AM43XX_PRM_OCP_SOCKET_INST, + omap4_prcm_irq_setup.mask); + + omap_prm_context.pm_ctrl = + omap4_prm_read_inst_reg(AM43XX_PRM_DEVICE_INST, + omap4_prcm_irq_setup.pm_ctrl); +} + +void prm_restore_context(void) +{ + omap4_prm_write_inst_reg(omap_prm_context.irq_enable, + OMAP4430_PRM_OCP_SOCKET_INST, + omap4_prcm_irq_setup.mask); + + omap4_prm_write_inst_reg(omap_prm_context.pm_ctrl, + AM43XX_PRM_DEVICE_INST, + omap4_prcm_irq_setup.pm_ctrl); +} + +static int cpu_notifier(struct notifier_block *nb, unsigned long cmd, void *v) +{ + switch (cmd) { + case CPU_CLUSTER_PM_ENTER: + if (enable_off_mode) + prm_save_context(); + break; + case CPU_CLUSTER_PM_EXIT: + if (enable_off_mode) + prm_restore_context(); + break; + } + + return NOTIFY_OK; +} + /* * XXX document */ @@ -709,6 +806,7 @@ static const struct omap_prcm_init_data *prm_init_data; int __init omap44xx_prm_init(const struct omap_prcm_init_data *data) { + static struct notifier_block nb; omap_prm_base_init(); prm_init_data = data; @@ -730,6 +828,12 @@ int __init omap44xx_prm_init(const struct omap_prcm_init_data *data) omap4_prcm_irq_setup.mask = AM43XX_PRM_IRQENABLE_MPU_OFFSET; } + /* Only AM43XX can lose prm context during rtc-ddr suspend */ + if (soc_is_am43xx()) { + nb.notifier_call = cpu_notifier; + cpu_pm_register_notifier(&nb); + } + return prm_register(&omap44xx_prm_ll_data); } diff --git a/arch/arm/mach-omap2/prm_common.c b/arch/arm/mach-omap2/prm_common.c index 021b5a8b9c0a..058a37e6d11c 100644 --- a/arch/arm/mach-omap2/prm_common.c +++ b/arch/arm/mach-omap2/prm_common.c @@ -285,10 +285,11 @@ int omap_prcm_register_chain_handler(struct omap_prcm_irq_setup *irq_setup) prcm_irq_setup = irq_setup; - prcm_irq_chips = kzalloc(sizeof(void *) * nr_regs, GFP_KERNEL); - prcm_irq_setup->saved_mask = kzalloc(sizeof(u32) * nr_regs, GFP_KERNEL); - prcm_irq_setup->priority_mask = kzalloc(sizeof(u32) * nr_regs, - GFP_KERNEL); + prcm_irq_chips = kcalloc(nr_regs, sizeof(void *), GFP_KERNEL); + prcm_irq_setup->saved_mask = kcalloc(nr_regs, sizeof(u32), + GFP_KERNEL); + prcm_irq_setup->priority_mask = kcalloc(nr_regs, sizeof(u32), + GFP_KERNEL); if (!prcm_irq_chips || !prcm_irq_setup->saved_mask || !prcm_irq_setup->priority_mask) diff --git a/arch/arm/mach-omap2/timer.c b/arch/arm/mach-omap2/timer.c index 4fb4dc24e5e9..98ed5ac073bc 100644 --- a/arch/arm/mach-omap2/timer.c +++ b/arch/arm/mach-omap2/timer.c @@ -50,7 +50,6 @@ #include "omap_device.h" #include #include -#include "omap-pm.h" #include "soc.h" #include "common.h" @@ -71,6 +70,9 @@ static struct clock_event_device clockevent_gpt; /* Clockevent hwmod for am335x and am437x suspend */ static struct omap_hwmod *clockevent_gpt_hwmod; +/* Clockesource hwmod for am437x suspend */ +static struct omap_hwmod *clocksource_gpt_hwmod; + #ifdef CONFIG_SOC_HAS_REALTIME_COUNTER static unsigned long arch_timer_freq; @@ -168,6 +170,43 @@ static const struct of_device_id omap_timer_match[] __initconst = { { } }; +static int omap_timer_add_disabled_property(struct device_node *np) +{ + struct property *prop; + + prop = kzalloc(sizeof(*prop), GFP_KERNEL); + if (!prop) + return -ENOMEM; + + prop->name = "status"; + prop->value = "disabled"; + prop->length = strlen(prop->value); + + return of_add_property(np, prop); +} + +static int omap_timer_update_dt(struct device_node *np) +{ + int error = 0; + + if (!of_device_is_compatible(np, "ti,omap-counter32k")) { + error = omap_timer_add_disabled_property(np); + if (error) + return error; + } + + /* No parent interconnect target module configured? */ + if (of_get_property(np, "ti,hwmods", NULL)) + return error; + + /* Tag parent interconnect target module disabled */ + error = omap_timer_add_disabled_property(np->parent); + if (error) + return error; + + return 0; +} + /** * omap_get_timer_dt - get a timer using device-tree * @match - device-tree match structure for matching a device type @@ -183,6 +222,7 @@ static struct device_node * __init omap_get_timer_dt(const struct of_device_id * const char *property) { struct device_node *np; + int error; for_each_matching_node(np, match) { if (!of_device_is_available(np)) @@ -197,17 +237,9 @@ static struct device_node * __init omap_get_timer_dt(const struct of_device_id * of_get_property(np, "ti,timer-secure", NULL))) continue; - if (!of_device_is_compatible(np, "ti,omap-counter32k")) { - struct property *prop; + error = omap_timer_update_dt(np); + WARN(error, "%s: Could not update dt: %i\n", __func__, error); - prop = kzalloc(sizeof(*prop), GFP_KERNEL); - if (!prop) - return NULL; - prop->name = "status"; - prop->value = "disabled"; - prop->length = strlen(prop->value); - of_add_property(np, prop); - } return np; } @@ -266,8 +298,12 @@ static int __init omap_dm_timer_init_one(struct omap_dm_timer *timer, return -ENODEV; of_property_read_string_index(np, "ti,hwmods", 0, &oh_name); - if (!oh_name) - return -ENODEV; + if (!oh_name) { + of_property_read_string_index(np->parent, "ti,hwmods", 0, + &oh_name); + if (!oh_name) + return -ENODEV; + } timer->irq = irq_of_parse_and_map(np, 0); if (!timer->irq) @@ -419,9 +455,12 @@ static int __init __maybe_unused omap2_sync32k_clocksource_init(void) if (!np) return -ENODEV; - of_property_read_string_index(np, "ti,hwmods", 0, &oh_name); - if (!oh_name) - return -ENODEV; + of_property_read_string_index(np->parent, "ti,hwmods", 0, &oh_name); + if (!oh_name) { + of_property_read_string_index(np, "ti,hwmods", 0, &oh_name); + if (!oh_name) + return -ENODEV; + } /* * First check hwmod data is available for sync32k counter @@ -442,6 +481,26 @@ static int __init __maybe_unused omap2_sync32k_clocksource_init(void) return ret; } +static unsigned int omap2_gptimer_clksrc_load; + +static void omap2_gptimer_clksrc_suspend(struct clocksource *unused) +{ + omap2_gptimer_clksrc_load = + __omap_dm_timer_read_counter(&clksrc, OMAP_TIMER_NONPOSTED); + + omap_hwmod_idle(clocksource_gpt_hwmod); +} + +static void omap2_gptimer_clksrc_resume(struct clocksource *unused) +{ + omap_hwmod_enable(clocksource_gpt_hwmod); + + __omap_dm_timer_load_start(&clksrc, + OMAP_TIMER_CTRL_ST | OMAP_TIMER_CTRL_AR, + omap2_gptimer_clksrc_load, + OMAP_TIMER_NONPOSTED); +} + static void __init omap2_gptimer_clocksource_init(int gptimer_id, const char *fck_source, const char *property) @@ -454,6 +513,15 @@ static void __init omap2_gptimer_clocksource_init(int gptimer_id, res = omap_dm_timer_init_one(&clksrc, fck_source, property, &clocksource_gpt.name, OMAP_TIMER_NONPOSTED); + + if (soc_is_am43xx()) { + clocksource_gpt.suspend = omap2_gptimer_clksrc_suspend; + clocksource_gpt.resume = omap2_gptimer_clksrc_resume; + + clocksource_gpt_hwmod = + omap_hwmod_lookup(clocksource_gpt.name); + } + BUG_ON(res); __omap_dm_timer_load_start(&clksrc, diff --git a/arch/arm/mach-pxa/palmz72.c b/arch/arm/mach-pxa/palmz72.c index 0adb1bd6208e..4d475f6f4a77 100644 --- a/arch/arm/mach-pxa/palmz72.c +++ b/arch/arm/mach-pxa/palmz72.c @@ -30,7 +30,7 @@ #include #include #include -#include +#include #include #include diff --git a/arch/arm/mach-pxa/pxa3xx.c b/arch/arm/mach-pxa/pxa3xx.c index 4b8a0df8ea57..8c64f93b669b 100644 --- a/arch/arm/mach-pxa/pxa3xx.c +++ b/arch/arm/mach-pxa/pxa3xx.c @@ -446,6 +446,10 @@ static int __init pxa3xx_init(void) pxa3xx_init_pm(); + enable_irq_wake(IRQ_WAKEUP0); + if (cpu_is_pxa320()) + enable_irq_wake(IRQ_WAKEUP1); + register_syscore_ops(&pxa_irq_syscore_ops); register_syscore_ops(&pxa3xx_mfp_syscore_ops); diff --git a/arch/arm/mach-pxa/stargate2.c b/arch/arm/mach-pxa/stargate2.c index df62bb23dbee..bbea5fa9a140 100644 --- a/arch/arm/mach-pxa/stargate2.c +++ b/arch/arm/mach-pxa/stargate2.c @@ -27,11 +27,11 @@ #include #include -#include #include #include #include #include +#include #include #include @@ -795,9 +795,9 @@ static struct pcf857x_platform_data platform_data_pcf857x = { .context = NULL, }; -static struct at24_platform_data pca9500_eeprom_pdata = { - .byte_len = 256, - .page_size = 4, +static const struct property_entry pca9500_eeprom_properties[] = { + PROPERTY_ENTRY_U32("pagesize", 4), + { } }; /** @@ -935,7 +935,7 @@ static struct i2c_board_info __initdata stargate2_i2c_board_info[] = { }, { .type = "24c02", .addr = 0x57, - .platform_data = &pca9500_eeprom_pdata, + .properties = pca9500_eeprom_properties, }, { .type = "max1238", .addr = 0x35, diff --git a/arch/arm/mach-pxa/viper.c b/arch/arm/mach-pxa/viper.c index 207dcc2e94e7..ab2f89266bbd 100644 --- a/arch/arm/mach-pxa/viper.c +++ b/arch/arm/mach-pxa/viper.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/arch/arm/mach-rockchip/Kconfig b/arch/arm/mach-rockchip/Kconfig index a4065966881a..fafd3d7f9f8c 100644 --- a/arch/arm/mach-rockchip/Kconfig +++ b/arch/arm/mach-rockchip/Kconfig @@ -3,7 +3,6 @@ config ARCH_ROCKCHIP depends on ARCH_MULTI_V7 select PINCTRL select PINCTRL_ROCKCHIP - select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE select ARCH_HAS_RESET_CONTROLLER select ARM_AMBA select ARM_GIC diff --git a/arch/arm/mach-rpc/ecard.c b/arch/arm/mach-rpc/ecard.c index bdb5ec1cf560..39aef4876ed4 100644 --- a/arch/arm/mach-rpc/ecard.c +++ b/arch/arm/mach-rpc/ecard.c @@ -657,25 +657,13 @@ static int ecard_devices_proc_show(struct seq_file *m, void *v) return 0; } -static int ecard_devices_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, ecard_devices_proc_show, NULL); -} - -static const struct file_operations bus_ecard_proc_fops = { - .owner = THIS_MODULE, - .open = ecard_devices_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static struct proc_dir_entry *proc_bus_ecard_dir = NULL; static void ecard_proc_init(void) { proc_bus_ecard_dir = proc_mkdir("bus/ecard", NULL); - proc_create("devices", 0, proc_bus_ecard_dir, &bus_ecard_proc_fops); + proc_create_single("devices", 0, proc_bus_ecard_dir, + ecard_devices_proc_show); } #define ec_set_resource(ec,nr,st,sz) \ diff --git a/arch/arm/mach-s3c24xx/h1940-bluetooth.c b/arch/arm/mach-s3c24xx/h1940-bluetooth.c index 46ad20ea87d1..186b5321658e 100644 --- a/arch/arm/mach-s3c24xx/h1940-bluetooth.c +++ b/arch/arm/mach-s3c24xx/h1940-bluetooth.c @@ -1,4 +1,4 @@ -// SPDX-License-Identifier: GPL-1.0 +// SPDX-License-Identifier: GPL-1.0+ // // Copyright (c) Arnaud Patard // diff --git a/arch/arm/mach-s3c24xx/mach-mini2440.c b/arch/arm/mach-s3c24xx/mach-mini2440.c index 95753e0bc073..f9fc1f8d2b28 100644 --- a/arch/arm/mach-s3c24xx/mach-mini2440.c +++ b/arch/arm/mach-s3c24xx/mach-mini2440.c @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include #include #include @@ -481,15 +481,15 @@ static struct platform_device mini2440_audio = { /* * I2C devices */ -static struct at24_platform_data at24c08 = { - .byte_len = SZ_8K / 8, - .page_size = 16, +static const struct property_entry mini2440_at24_properties[] = { + PROPERTY_ENTRY_U32("pagesize", 16), + { } }; static struct i2c_board_info mini2440_i2c_devs[] __initdata = { { I2C_BOARD_INFO("24c08", 0x50), - .platform_data = &at24c08, + .properties = mini2440_at24_properties, }, }; diff --git a/arch/arm/mach-s3c64xx/mach-crag6410-module.c b/arch/arm/mach-s3c64xx/mach-crag6410-module.c index f00988705408..5aa472892465 100644 --- a/arch/arm/mach-s3c64xx/mach-crag6410-module.c +++ b/arch/arm/mach-s3c64xx/mach-crag6410-module.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include @@ -206,9 +207,6 @@ static const struct i2c_board_info wm1277_devs[] = { }; static struct arizona_pdata wm5102_reva_pdata = { - .ldo1 = { - .ldoena = S3C64XX_GPN(7), - }, .gpio_base = CODEC_GPIO_BASE, .irq_flags = IRQF_TRIGGER_HIGH, .micd_pol_gpio = CODEC_GPIO_BASE + 4, @@ -237,10 +235,16 @@ static struct spi_board_info wm5102_reva_spi_devs[] = { }, }; -static struct arizona_pdata wm5102_pdata = { - .ldo1 = { - .ldoena = S3C64XX_GPN(7), +static struct gpiod_lookup_table wm5102_reva_gpiod_table = { + .dev_id = "spi0.1", /* SPI device name */ + .table = { + GPIO_LOOKUP("GPION", 7, + "wlf,ldoena", GPIO_ACTIVE_HIGH), + { }, }, +}; + +static struct arizona_pdata wm5102_pdata = { .gpio_base = CODEC_GPIO_BASE, .irq_flags = IRQF_TRIGGER_HIGH, .micd_pol_gpio = CODEC_GPIO_BASE + 2, @@ -264,6 +268,15 @@ static struct spi_board_info wm5102_spi_devs[] = { }, }; +static struct gpiod_lookup_table wm5102_gpiod_table = { + .dev_id = "spi0.1", /* SPI device name */ + .table = { + GPIO_LOOKUP("GPION", 7, + "wlf,ldo1ena", GPIO_ACTIVE_HIGH), + { }, + }, +}; + static struct spi_board_info wm5110_spi_devs[] = { [0] = { .modalias = "wm5110", @@ -366,6 +379,9 @@ static int wlf_gf_module_probe(struct i2c_client *i2c, rev == gf_mods[i].rev)) break; + gpiod_add_lookup_table(&wm5102_reva_gpiod_table); + gpiod_add_lookup_table(&wm5102_gpiod_table); + if (i < ARRAY_SIZE(gf_mods)) { dev_info(&i2c->dev, "%s revision %d\n", gf_mods[i].name, rev + 1); diff --git a/arch/arm/mach-sa1100/simpad.c b/arch/arm/mach-sa1100/simpad.c index f45aed2519ba..406487e76a5c 100644 --- a/arch/arm/mach-sa1100/simpad.c +++ b/arch/arm/mach-sa1100/simpad.c @@ -37,7 +37,7 @@ #include #include #include -#include +#include #include "generic.h" diff --git a/arch/arm/mach-shmobile/Kconfig b/arch/arm/mach-shmobile/Kconfig index 280e7312a9e1..0b67254eabb2 100644 --- a/arch/arm/mach-shmobile/Kconfig +++ b/arch/arm/mach-shmobile/Kconfig @@ -29,7 +29,6 @@ config ARCH_RMOBILE menuconfig ARCH_RENESAS bool "Renesas ARM SoCs" depends on ARCH_MULTI_V7 && MMU - select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE select ARCH_SHMOBILE select ARM_GIC select GPIOLIB @@ -75,6 +74,10 @@ config ARCH_R8A7745 bool "RZ/G1E (R8A77450)" select ARCH_RCAR_GEN2 +config ARCH_R8A77470 + bool "RZ/G1C (R8A77470)" + select ARCH_RCAR_GEN2 + config ARCH_R8A7778 bool "R-Car M1A (R8A77781)" select ARCH_RCAR_GEN1 @@ -110,6 +113,15 @@ config ARCH_R8A7794 bool "R-Car E2 (R8A77940)" select ARCH_RCAR_GEN2 +config ARCH_R9A06G032 + bool "RZ/N1D (R9A06G032)" + select ARCH_RZN1 + +config ARCH_RZN1 + bool "RZ/N1 (R9A06G0xx) Family" + select ARM_AMBA + select CPU_V7 + config ARCH_SH73A0 bool "SH-Mobile AG5 (R8A73A00)" select ARCH_RMOBILE diff --git a/arch/arm/mach-shmobile/common.h b/arch/arm/mach-shmobile/common.h index 43c1ac696274..2109f123bdfb 100644 --- a/arch/arm/mach-shmobile/common.h +++ b/arch/arm/mach-shmobile/common.h @@ -2,7 +2,6 @@ #ifndef __ARCH_MACH_COMMON_H #define __ARCH_MACH_COMMON_H -extern void shmobile_init_cntvoff(void); extern void shmobile_init_delay(void); extern void shmobile_boot_vector(void); extern unsigned long shmobile_boot_fn; diff --git a/arch/arm/mach-shmobile/headsmp-apmu.S b/arch/arm/mach-shmobile/headsmp-apmu.S index 5672b5849401..d49ab194766a 100644 --- a/arch/arm/mach-shmobile/headsmp-apmu.S +++ b/arch/arm/mach-shmobile/headsmp-apmu.S @@ -11,29 +11,9 @@ #include #include -ENTRY(shmobile_init_cntvoff) - /* - * CNTVOFF has to be initialized either from non-secure Hypervisor - * mode or secure Monitor mode with SCR.NS==1. If TrustZone is enabled - * then it should be handled by the secure code - */ - cps #MON_MODE - mrc p15, 0, r1, c1, c1, 0 /* Get Secure Config */ - orr r0, r1, #1 - mcr p15, 0, r0, c1, c1, 0 /* Set Non Secure bit */ - instr_sync - mov r0, #0 - mcrr p15, 4, r0, r0, c14 /* CNTVOFF = 0 */ - instr_sync - mcr p15, 0, r1, c1, c1, 0 /* Set Secure bit */ - instr_sync - cps #SVC_MODE - ret lr -ENDPROC(shmobile_init_cntvoff) - #ifdef CONFIG_SMP ENTRY(shmobile_boot_apmu) - bl shmobile_init_cntvoff + bl secure_cntvoff_init b secondary_startup ENDPROC(shmobile_boot_apmu) #endif diff --git a/arch/arm/mach-shmobile/setup-rcar-gen2.c b/arch/arm/mach-shmobile/setup-rcar-gen2.c index 5561dbed7a33..88fdc1801d90 100644 --- a/arch/arm/mach-shmobile/setup-rcar-gen2.c +++ b/arch/arm/mach-shmobile/setup-rcar-gen2.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "common.h" #include "rcar-gen2.h" @@ -70,9 +71,10 @@ void __init rcar_gen2_timer_init(void) void __iomem *base; u32 freq; - shmobile_init_cntvoff(); + secure_cntvoff_init(); if (of_machine_is_compatible("renesas,r8a7745") || + of_machine_is_compatible("renesas,r8a77470") || of_machine_is_compatible("renesas,r8a7792") || of_machine_is_compatible("renesas,r8a7794")) { freq = 260000000 / 8; /* ZS / 8 */ @@ -205,6 +207,7 @@ MACHINE_END static const char * const rz_g1_boards_compat_dt[] __initconst = { "renesas,r8a7743", "renesas,r8a7745", + "renesas,r8a77470", NULL, }; diff --git a/arch/arm/mach-sunxi/Kconfig b/arch/arm/mach-sunxi/Kconfig index ce53ceaf4cc5..d9c8ecf88ec6 100644 --- a/arch/arm/mach-sunxi/Kconfig +++ b/arch/arm/mach-sunxi/Kconfig @@ -51,7 +51,7 @@ config MACH_SUN9I config ARCH_SUNXI_MC_SMP bool depends on SMP - default MACH_SUN9I + default MACH_SUN9I || MACH_SUN8I select ARM_CCI400_PORT_CTRL select ARM_CPU_SUSPEND diff --git a/arch/arm/mach-sunxi/Makefile b/arch/arm/mach-sunxi/Makefile index 7de9cc286d53..71429aa85143 100644 --- a/arch/arm/mach-sunxi/Makefile +++ b/arch/arm/mach-sunxi/Makefile @@ -1,5 +1,5 @@ CFLAGS_mc_smp.o += -march=armv7-a obj-$(CONFIG_ARCH_SUNXI) += sunxi.o -obj-$(CONFIG_ARCH_SUNXI_MC_SMP) += mc_smp.o +obj-$(CONFIG_ARCH_SUNXI_MC_SMP) += mc_smp.o headsmp.o obj-$(CONFIG_SMP) += platsmp.o diff --git a/arch/arm/mach-sunxi/headsmp.S b/arch/arm/mach-sunxi/headsmp.S new file mode 100644 index 000000000000..32d76be98541 --- /dev/null +++ b/arch/arm/mach-sunxi/headsmp.S @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: GPL-2.0 + * + * Copyright (c) 2018 Chen-Yu Tsai + * Copyright (c) 2018 Bootlin + * + * Chen-Yu Tsai + * Mylène Josserand + * + * SMP support for sunxi based systems with Cortex A7/A15 + * + */ + +#include +#include +#include + +ENTRY(sunxi_mc_smp_cluster_cache_enable) + .arch armv7-a + /* + * Enable cluster-level coherency, in preparation for turning on the MMU. + * + * Also enable regional clock gating and L2 data latency settings for + * Cortex-A15. These settings are from the vendor kernel. + */ + mrc p15, 0, r1, c0, c0, 0 + movw r2, #(ARM_CPU_PART_MASK & 0xffff) + movt r2, #(ARM_CPU_PART_MASK >> 16) + and r1, r1, r2 + movw r2, #(ARM_CPU_PART_CORTEX_A15 & 0xffff) + movt r2, #(ARM_CPU_PART_CORTEX_A15 >> 16) + cmp r1, r2 + bne not_a15 + + /* The following is Cortex-A15 specific */ + + /* ACTLR2: Enable CPU regional clock gates */ + mrc p15, 1, r1, c15, c0, 4 + orr r1, r1, #(0x1 << 31) + mcr p15, 1, r1, c15, c0, 4 + + /* L2ACTLR */ + mrc p15, 1, r1, c15, c0, 0 + /* Enable L2, GIC, and Timer regional clock gates */ + orr r1, r1, #(0x1 << 26) + /* Disable clean/evict from being pushed to external */ + orr r1, r1, #(0x1<<3) + mcr p15, 1, r1, c15, c0, 0 + + /* L2CTRL: L2 data RAM latency */ + mrc p15, 1, r1, c9, c0, 2 + bic r1, r1, #(0x7 << 0) + orr r1, r1, #(0x3 << 0) + mcr p15, 1, r1, c9, c0, 2 + + /* End of Cortex-A15 specific setup */ + not_a15: + + /* Get value of sunxi_mc_smp_first_comer */ + adr r1, first + ldr r0, [r1] + ldr r0, [r1, r0] + + /* Skip cci_enable_port_for_self if not first comer */ + cmp r0, #0 + bxeq lr + b cci_enable_port_for_self + + .align 2 + first: .word sunxi_mc_smp_first_comer - . +ENDPROC(sunxi_mc_smp_cluster_cache_enable) + +ENTRY(sunxi_mc_smp_secondary_startup) + bl sunxi_mc_smp_cluster_cache_enable + bl secure_cntvoff_init + b secondary_startup +ENDPROC(sunxi_mc_smp_secondary_startup) + +ENTRY(sunxi_mc_smp_resume) + bl sunxi_mc_smp_cluster_cache_enable + b cpu_resume +ENDPROC(sunxi_mc_smp_resume) diff --git a/arch/arm/mach-sunxi/mc_smp.c b/arch/arm/mach-sunxi/mc_smp.c index c0246ec54a0a..b4037b603897 100644 --- a/arch/arm/mach-sunxi/mc_smp.c +++ b/arch/arm/mach-sunxi/mc_smp.c @@ -55,22 +55,35 @@ #define CPUCFG_CX_RST_CTRL_L2_RST BIT(8) #define CPUCFG_CX_RST_CTRL_CX_RST(n) BIT(4 + (n)) #define CPUCFG_CX_RST_CTRL_CORE_RST(n) BIT(n) +#define CPUCFG_CX_RST_CTRL_CORE_RST_ALL (0xf << 0) #define PRCM_CPU_PO_RST_CTRL(c) (0x4 + 0x4 * (c)) #define PRCM_CPU_PO_RST_CTRL_CORE(n) BIT(n) #define PRCM_CPU_PO_RST_CTRL_CORE_ALL 0xf #define PRCM_PWROFF_GATING_REG(c) (0x100 + 0x4 * (c)) -#define PRCM_PWROFF_GATING_REG_CLUSTER BIT(4) +/* The power off register for clusters are different from a80 and a83t */ +#define PRCM_PWROFF_GATING_REG_CLUSTER_SUN8I BIT(0) +#define PRCM_PWROFF_GATING_REG_CLUSTER_SUN9I BIT(4) #define PRCM_PWROFF_GATING_REG_CORE(n) BIT(n) #define PRCM_PWR_SWITCH_REG(c, cpu) (0x140 + 0x10 * (c) + 0x4 * (cpu)) #define PRCM_CPU_SOFT_ENTRY_REG 0x164 +/* R_CPUCFG registers, specific to sun8i-a83t */ +#define R_CPUCFG_CLUSTER_PO_RST_CTRL(c) (0x30 + (c) * 0x4) +#define R_CPUCFG_CLUSTER_PO_RST_CTRL_CORE(n) BIT(n) +#define R_CPUCFG_CPU_SOFT_ENTRY_REG 0x01a4 + #define CPU0_SUPPORT_HOTPLUG_MAGIC0 0xFA50392F #define CPU0_SUPPORT_HOTPLUG_MAGIC1 0x790DCA3A static void __iomem *cpucfg_base; static void __iomem *prcm_base; static void __iomem *sram_b_smp_base; +static void __iomem *r_cpucfg_base; + +extern void sunxi_mc_smp_secondary_startup(void); +extern void sunxi_mc_smp_resume(void); +static bool is_a83t; static bool sunxi_core_is_cortex_a15(unsigned int core, unsigned int cluster) { @@ -157,6 +170,16 @@ static int sunxi_cpu_powerup(unsigned int cpu, unsigned int cluster) reg &= ~PRCM_CPU_PO_RST_CTRL_CORE(cpu); writel(reg, prcm_base + PRCM_CPU_PO_RST_CTRL(cluster)); + if (is_a83t) { + /* assert cpu power-on reset */ + reg = readl(r_cpucfg_base + + R_CPUCFG_CLUSTER_PO_RST_CTRL(cluster)); + reg &= ~(R_CPUCFG_CLUSTER_PO_RST_CTRL_CORE(cpu)); + writel(reg, r_cpucfg_base + + R_CPUCFG_CLUSTER_PO_RST_CTRL(cluster)); + udelay(10); + } + /* Cortex-A7: hold L1 reset disable signal low */ if (!sunxi_core_is_cortex_a15(cpu, cluster)) { reg = readl(cpucfg_base + CPUCFG_CX_CTRL_REG0(cluster)); @@ -180,17 +203,38 @@ static int sunxi_cpu_powerup(unsigned int cpu, unsigned int cluster) /* open power switch */ sunxi_cpu_power_switch_set(cpu, cluster, true); + /* Handle A83T bit swap */ + if (is_a83t) { + if (cpu == 0) + cpu = 4; + } + /* clear processor power gate */ reg = readl(prcm_base + PRCM_PWROFF_GATING_REG(cluster)); reg &= ~PRCM_PWROFF_GATING_REG_CORE(cpu); writel(reg, prcm_base + PRCM_PWROFF_GATING_REG(cluster)); udelay(20); + /* Handle A83T bit swap */ + if (is_a83t) { + if (cpu == 4) + cpu = 0; + } + /* de-assert processor power-on reset */ reg = readl(prcm_base + PRCM_CPU_PO_RST_CTRL(cluster)); reg |= PRCM_CPU_PO_RST_CTRL_CORE(cpu); writel(reg, prcm_base + PRCM_CPU_PO_RST_CTRL(cluster)); + if (is_a83t) { + reg = readl(r_cpucfg_base + + R_CPUCFG_CLUSTER_PO_RST_CTRL(cluster)); + reg |= R_CPUCFG_CLUSTER_PO_RST_CTRL_CORE(cpu); + writel(reg, r_cpucfg_base + + R_CPUCFG_CLUSTER_PO_RST_CTRL(cluster)); + udelay(10); + } + /* de-assert all processor resets */ reg = readl(cpucfg_base + CPUCFG_CX_RST_CTRL(cluster)); reg |= CPUCFG_CX_RST_CTRL_DBG_RST(cpu); @@ -212,6 +256,14 @@ static int sunxi_cluster_powerup(unsigned int cluster) if (cluster >= SUNXI_NR_CLUSTERS) return -EINVAL; + /* For A83T, assert cluster cores resets */ + if (is_a83t) { + reg = readl(cpucfg_base + CPUCFG_CX_RST_CTRL(cluster)); + reg &= ~CPUCFG_CX_RST_CTRL_CORE_RST_ALL; /* Core Reset */ + writel(reg, cpucfg_base + CPUCFG_CX_RST_CTRL(cluster)); + udelay(10); + } + /* assert ACINACTM */ reg = readl(cpucfg_base + CPUCFG_CX_CTRL_REG1(cluster)); reg |= CPUCFG_CX_CTRL_REG1_ACINACTM; @@ -222,6 +274,16 @@ static int sunxi_cluster_powerup(unsigned int cluster) reg &= ~PRCM_CPU_PO_RST_CTRL_CORE_ALL; writel(reg, prcm_base + PRCM_CPU_PO_RST_CTRL(cluster)); + /* assert cluster cores resets */ + if (is_a83t) { + reg = readl(r_cpucfg_base + + R_CPUCFG_CLUSTER_PO_RST_CTRL(cluster)); + reg &= ~CPUCFG_CX_RST_CTRL_CORE_RST_ALL; + writel(reg, r_cpucfg_base + + R_CPUCFG_CLUSTER_PO_RST_CTRL(cluster)); + udelay(10); + } + /* assert cluster resets */ reg = readl(cpucfg_base + CPUCFG_CX_RST_CTRL(cluster)); reg &= ~CPUCFG_CX_RST_CTRL_DBG_SOC_RST; @@ -252,7 +314,10 @@ static int sunxi_cluster_powerup(unsigned int cluster) /* clear cluster power gate */ reg = readl(prcm_base + PRCM_PWROFF_GATING_REG(cluster)); - reg &= ~PRCM_PWROFF_GATING_REG_CLUSTER; + if (is_a83t) + reg &= ~PRCM_PWROFF_GATING_REG_CLUSTER_SUN8I; + else + reg &= ~PRCM_PWROFF_GATING_REG_CLUSTER_SUN9I; writel(reg, prcm_base + PRCM_PWROFF_GATING_REG(cluster)); udelay(20); @@ -300,74 +365,7 @@ static void sunxi_cluster_cache_disable_without_axi(void) } static int sunxi_mc_smp_cpu_table[SUNXI_NR_CLUSTERS][SUNXI_CPUS_PER_CLUSTER]; -static int sunxi_mc_smp_first_comer; - -/* - * Enable cluster-level coherency, in preparation for turning on the MMU. - * - * Also enable regional clock gating and L2 data latency settings for - * Cortex-A15. These settings are from the vendor kernel. - */ -static void __naked sunxi_mc_smp_cluster_cache_enable(void) -{ - asm volatile ( - "mrc p15, 0, r1, c0, c0, 0\n" - "movw r2, #" __stringify(ARM_CPU_PART_MASK & 0xffff) "\n" - "movt r2, #" __stringify(ARM_CPU_PART_MASK >> 16) "\n" - "and r1, r1, r2\n" - "movw r2, #" __stringify(ARM_CPU_PART_CORTEX_A15 & 0xffff) "\n" - "movt r2, #" __stringify(ARM_CPU_PART_CORTEX_A15 >> 16) "\n" - "cmp r1, r2\n" - "bne not_a15\n" - - /* The following is Cortex-A15 specific */ - - /* ACTLR2: Enable CPU regional clock gates */ - "mrc p15, 1, r1, c15, c0, 4\n" - "orr r1, r1, #(0x1<<31)\n" - "mcr p15, 1, r1, c15, c0, 4\n" - - /* L2ACTLR */ - "mrc p15, 1, r1, c15, c0, 0\n" - /* Enable L2, GIC, and Timer regional clock gates */ - "orr r1, r1, #(0x1<<26)\n" - /* Disable clean/evict from being pushed to external */ - "orr r1, r1, #(0x1<<3)\n" - "mcr p15, 1, r1, c15, c0, 0\n" - - /* L2CTRL: L2 data RAM latency */ - "mrc p15, 1, r1, c9, c0, 2\n" - "bic r1, r1, #(0x7<<0)\n" - "orr r1, r1, #(0x3<<0)\n" - "mcr p15, 1, r1, c9, c0, 2\n" - - /* End of Cortex-A15 specific setup */ - "not_a15:\n" - - /* Get value of sunxi_mc_smp_first_comer */ - "adr r1, first\n" - "ldr r0, [r1]\n" - "ldr r0, [r1, r0]\n" - - /* Skip cci_enable_port_for_self if not first comer */ - "cmp r0, #0\n" - "bxeq lr\n" - "b cci_enable_port_for_self\n" - - ".align 2\n" - "first: .word sunxi_mc_smp_first_comer - .\n" - ); -} - -static void __naked sunxi_mc_smp_secondary_startup(void) -{ - asm volatile( - "bl sunxi_mc_smp_cluster_cache_enable\n" - "b secondary_startup" - /* Let compiler know about sunxi_mc_smp_cluster_cache_enable */ - :: "i" (sunxi_mc_smp_cluster_cache_enable) - ); -} +int sunxi_mc_smp_first_comer; static DEFINE_SPINLOCK(boot_lock); @@ -516,7 +514,10 @@ static int sunxi_cluster_powerdown(unsigned int cluster) /* gate cluster power */ pr_debug("%s: gate cluster power\n", __func__); reg = readl(prcm_base + PRCM_PWROFF_GATING_REG(cluster)); - reg |= PRCM_PWROFF_GATING_REG_CLUSTER; + if (is_a83t) + reg |= PRCM_PWROFF_GATING_REG_CLUSTER_SUN8I; + else + reg |= PRCM_PWROFF_GATING_REG_CLUSTER_SUN9I; writel(reg, prcm_base + PRCM_PWROFF_GATING_REG(cluster)); udelay(20); @@ -598,8 +599,12 @@ out: return !ret; } -static bool sunxi_mc_smp_cpu_can_disable(unsigned int __unused) +static bool sunxi_mc_smp_cpu_can_disable(unsigned int cpu) { + /* CPU0 hotplug not handled for sun8i-a83t */ + if (is_a83t) + if (cpu == 0) + return false; return true; } #endif @@ -637,16 +642,6 @@ static bool __init sunxi_mc_smp_cpu_table_init(void) */ typedef typeof(cpu_reset) phys_reset_t; -static void __init __naked sunxi_mc_smp_resume(void) -{ - asm volatile( - "bl sunxi_mc_smp_cluster_cache_enable\n" - "b cpu_resume" - /* Let compiler know about sunxi_mc_smp_cluster_cache_enable */ - :: "i" (sunxi_mc_smp_cluster_cache_enable) - ); -} - static int __init nocache_trampoline(unsigned long __unused) { phys_reset_t phys_reset; @@ -692,12 +687,14 @@ struct sunxi_mc_smp_nodes { struct device_node *prcm_node; struct device_node *cpucfg_node; struct device_node *sram_node; + struct device_node *r_cpucfg_node; }; /* This structure holds SoC-specific bits tied to an enable-method string. */ struct sunxi_mc_smp_data { const char *enable_method; int (*get_smp_nodes)(struct sunxi_mc_smp_nodes *nodes); + bool is_a83t; }; static void __init sunxi_mc_smp_put_nodes(struct sunxi_mc_smp_nodes *nodes) @@ -705,6 +702,7 @@ static void __init sunxi_mc_smp_put_nodes(struct sunxi_mc_smp_nodes *nodes) of_node_put(nodes->prcm_node); of_node_put(nodes->cpucfg_node); of_node_put(nodes->sram_node); + of_node_put(nodes->r_cpucfg_node); memset(nodes, 0, sizeof(*nodes)); } @@ -734,11 +732,42 @@ static int __init sun9i_a80_get_smp_nodes(struct sunxi_mc_smp_nodes *nodes) return 0; } +static int __init sun8i_a83t_get_smp_nodes(struct sunxi_mc_smp_nodes *nodes) +{ + nodes->prcm_node = of_find_compatible_node(NULL, NULL, + "allwinner,sun8i-a83t-r-ccu"); + if (!nodes->prcm_node) { + pr_err("%s: PRCM not available\n", __func__); + return -ENODEV; + } + + nodes->cpucfg_node = of_find_compatible_node(NULL, NULL, + "allwinner,sun8i-a83t-cpucfg"); + if (!nodes->cpucfg_node) { + pr_err("%s: CPUCFG not available\n", __func__); + return -ENODEV; + } + + nodes->r_cpucfg_node = of_find_compatible_node(NULL, NULL, + "allwinner,sun8i-a83t-r-cpucfg"); + if (!nodes->r_cpucfg_node) { + pr_err("%s: RCPUCFG not available\n", __func__); + return -ENODEV; + } + + return 0; +} + static const struct sunxi_mc_smp_data sunxi_mc_smp_data[] __initconst = { { .enable_method = "allwinner,sun9i-a80-smp", .get_smp_nodes = sun9i_a80_get_smp_nodes, }, + { + .enable_method = "allwinner,sun8i-a83t-smp", + .get_smp_nodes = sun8i_a83t_get_smp_nodes, + .is_a83t = true, + }, }; static int __init sunxi_mc_smp_init(void) @@ -746,6 +775,7 @@ static int __init sunxi_mc_smp_init(void) struct sunxi_mc_smp_nodes nodes = { 0 }; struct device_node *node; struct resource res; + void __iomem *addr; int i, ret; /* @@ -771,6 +801,8 @@ static int __init sunxi_mc_smp_init(void) break; } + is_a83t = sunxi_mc_smp_data[i].is_a83t; + of_node_put(node); if (ret) return -ENODEV; @@ -808,12 +840,23 @@ static int __init sunxi_mc_smp_init(void) goto err_unmap_prcm; } - sram_b_smp_base = of_io_request_and_map(nodes.sram_node, 0, - "sunxi-mc-smp"); - if (IS_ERR(sram_b_smp_base)) { - ret = PTR_ERR(sram_b_smp_base); - pr_err("%s: failed to map secure SRAM\n", __func__); - goto err_unmap_release_cpucfg; + if (is_a83t) { + r_cpucfg_base = of_io_request_and_map(nodes.r_cpucfg_node, + 0, "sunxi-mc-smp"); + if (IS_ERR(r_cpucfg_base)) { + ret = PTR_ERR(r_cpucfg_base); + pr_err("%s: failed to map R-CPUCFG registers\n", + __func__); + goto err_unmap_release_cpucfg; + } + } else { + sram_b_smp_base = of_io_request_and_map(nodes.sram_node, 0, + "sunxi-mc-smp"); + if (IS_ERR(sram_b_smp_base)) { + ret = PTR_ERR(sram_b_smp_base); + pr_err("%s: failed to map secure SRAM\n", __func__); + goto err_unmap_release_cpucfg; + } } /* Configure CCI-400 for boot cluster */ @@ -821,15 +864,18 @@ static int __init sunxi_mc_smp_init(void) if (ret) { pr_err("%s: failed to configure boot cluster: %d\n", __func__, ret); - goto err_unmap_release_secure_sram; + goto err_unmap_release_sram_rcpucfg; } /* We don't need the device nodes anymore */ sunxi_mc_smp_put_nodes(&nodes); /* Set the hardware entry point address */ - writel(__pa_symbol(sunxi_mc_smp_secondary_startup), - prcm_base + PRCM_CPU_SOFT_ENTRY_REG); + if (is_a83t) + addr = r_cpucfg_base + R_CPUCFG_CPU_SOFT_ENTRY_REG; + else + addr = prcm_base + PRCM_CPU_SOFT_ENTRY_REG; + writel(__pa_symbol(sunxi_mc_smp_secondary_startup), addr); /* Actually enable multi cluster SMP */ smp_set_ops(&sunxi_mc_smp_smp_ops); @@ -838,9 +884,14 @@ static int __init sunxi_mc_smp_init(void) return 0; -err_unmap_release_secure_sram: - iounmap(sram_b_smp_base); - of_address_to_resource(nodes.sram_node, 0, &res); +err_unmap_release_sram_rcpucfg: + if (is_a83t) { + iounmap(r_cpucfg_base); + of_address_to_resource(nodes.r_cpucfg_node, 0, &res); + } else { + iounmap(sram_b_smp_base); + of_address_to_resource(nodes.sram_node, 0, &res); + } release_mem_region(res.start, resource_size(&res)); err_unmap_release_cpucfg: iounmap(cpucfg_base); diff --git a/arch/arm/mach-sunxi/sunxi.c b/arch/arm/mach-sunxi/sunxi.c index 5e9602ce1573..de4b0e932f22 100644 --- a/arch/arm/mach-sunxi/sunxi.c +++ b/arch/arm/mach-sunxi/sunxi.c @@ -16,6 +16,7 @@ #include #include +#include static const char * const sunxi_board_dt_compat[] = { "allwinner,sun4i-a10", @@ -62,7 +63,6 @@ MACHINE_END static const char * const sun8i_board_dt_compat[] = { "allwinner,sun8i-a23", "allwinner,sun8i-a33", - "allwinner,sun8i-a83t", "allwinner,sun8i-h2-plus", "allwinner,sun8i-h3", "allwinner,sun8i-r40", @@ -75,6 +75,24 @@ DT_MACHINE_START(SUN8I_DT, "Allwinner sun8i Family") .dt_compat = sun8i_board_dt_compat, MACHINE_END +static void __init sun8i_a83t_cntvoff_init(void) +{ +#ifdef CONFIG_SMP + secure_cntvoff_init(); +#endif +} + +static const char * const sun8i_a83t_cntvoff_board_dt_compat[] = { + "allwinner,sun8i-a83t", + NULL, +}; + +DT_MACHINE_START(SUN8I_A83T_CNTVOFF_DT, "Allwinner A83t board") + .init_early = sun8i_a83t_cntvoff_init, + .init_time = sun6i_timer_init, + .dt_compat = sun8i_a83t_cntvoff_board_dt_compat, +MACHINE_END + static const char * const sun9i_board_dt_compat[] = { "allwinner,sun9i-a80", NULL, diff --git a/arch/arm/mach-tegra/Kconfig b/arch/arm/mach-tegra/Kconfig index 1e0aeb47bac6..7f3b83e0d324 100644 --- a/arch/arm/mach-tegra/Kconfig +++ b/arch/arm/mach-tegra/Kconfig @@ -15,6 +15,5 @@ menuconfig ARCH_TEGRA select RESET_CONTROLLER select SOC_BUS select ZONE_DMA if ARM_LPAE - select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE help This enables support for NVIDIA Tegra based systems. diff --git a/arch/arm/mach-tegra/tegra.c b/arch/arm/mach-tegra/tegra.c index 02e712d2ea30..f9587be48235 100644 --- a/arch/arm/mach-tegra/tegra.c +++ b/arch/arm/mach-tegra/tegra.c @@ -97,6 +97,10 @@ static void __init tegra_dt_init_late(void) if (IS_ENABLED(CONFIG_ARCH_TEGRA_2x_SOC) && of_machine_is_compatible("compal,paz00")) tegra_paz00_wifikill_init(); + + if (IS_ENABLED(CONFIG_ARCH_TEGRA_2x_SOC) && + of_machine_is_compatible("nvidia,tegra20")) + platform_device_register_simple("tegra20-cpufreq", -1, NULL, 0); } static const char * const tegra_dt_board_compat[] = { diff --git a/arch/arm/mach-ux500/Kconfig b/arch/arm/mach-ux500/Kconfig index f98332ea2ef2..c1086ebe0050 100644 --- a/arch/arm/mach-ux500/Kconfig +++ b/arch/arm/mach-ux500/Kconfig @@ -9,64 +9,33 @@ menuconfig ARCH_U8500 select ARM_ERRATA_764369 if SMP select ARM_GIC select CACHE_L2X0 + select CLKSRC_DBX500_PRCMU select CLKSRC_NOMADIK_MTU select GPIOLIB select HAVE_ARM_SCU if SMP select HAVE_ARM_TWD if SMP + select I2C + select I2C_NOMADIK + select MFD_DB8500_PRCMU select PINCTRL + select PINCTRL_AB8500 + select PINCTRL_AB8505 select PINCTRL_ABX500 + select PINCTRL_DB8500 select PINCTRL_NOMADIK select PL310_ERRATA_753970 if CACHE_L2X0 - help - Support for ST-Ericsson's Ux500 architecture - -if ARCH_U8500 - -config UX500_SOC_DB8500 - bool - select MFD_DB8500_PRCMU - select PINCTRL_DB8500 - select PINCTRL_DB8540 - select PINCTRL_AB8500 - select PINCTRL_AB8505 - select PINCTRL_AB9540 - select PINCTRL_AB8540 - select REGULATOR - select REGULATOR_DB8500_PRCMU - select CLKSRC_DBX500_PRCMU select PM_GENERIC_DOMAINS if PM - -config MACH_MOP500 - bool "U8500 Development platform, MOP500 versions" - select I2C - select I2C_NOMADIK select REGULATOR + select REGULATOR_DB8500_PRCMU select REGULATOR_FIXED_VOLTAGE select SOC_BUS - select UX500_SOC_DB8500 help - Include support for the MOP500 development platform. - -config MACH_HREFV60 - bool "U8500 Development platform, HREFv60 version" - select MACH_MOP500 - help - Include support for the HREFv60 new development platform. - Includes HREFv70, v71 etc. + Support for ST-Ericsson's Ux500 architecture -config MACH_SNOWBALL - bool "U8500 Snowball platform" - select MACH_MOP500 - help - Include support for the snowball development platform. +if ARCH_U8500 -config UX500_AUTO_PLATFORM +config UX500_SOC_DB8500 def_bool y - select MACH_MOP500 - help - At least one platform needs to be selected in order to build - a working kernel. If everything else is disabled, this - automatically enables MACH_MOP500. config UX500_DEBUG_UART int "Ux500 UART to use for low-level debug" diff --git a/arch/arm/mach-ux500/cpu-db8500.c b/arch/arm/mach-ux500/cpu-db8500.c index 36cd23c8be9b..389ecf6faa00 100644 --- a/arch/arm/mach-ux500/cpu-db8500.c +++ b/arch/arm/mach-ux500/cpu-db8500.c @@ -111,11 +111,6 @@ static void ux500_restart(enum reboot_mode mode, const char *cmd) prcmu_system_reset(0); } -static struct of_dev_auxdata u8540_auxdata_lookup[] __initdata = { - OF_DEV_AUXDATA("stericsson,db8500-prcmu", 0x80157000, "db8500-prcmu", NULL), - {}, -}; - static const struct of_device_id u8500_local_bus_nodes[] = { /* only create devices below soc node */ { .compatible = "stericsson,db8500", }, @@ -129,20 +124,13 @@ static void __init u8500_init_machine(void) /* Initialize ux500 power domains */ ux500_pm_domains_init(); - /* automatically probe child nodes of dbx5x0 devices */ - if (of_machine_is_compatible("st-ericsson,u8540")) - of_platform_populate(NULL, u8500_local_bus_nodes, - u8540_auxdata_lookup, NULL); - else - of_platform_populate(NULL, u8500_local_bus_nodes, - NULL, NULL); + of_platform_populate(NULL, u8500_local_bus_nodes, + NULL, NULL); } static const char * stericsson_dt_platform_compat[] = { "st-ericsson,u8500", - "st-ericsson,u8540", "st-ericsson,u9500", - "st-ericsson,u9540", NULL, }; diff --git a/arch/arm/mach-ux500/db8500-regs.h b/arch/arm/mach-ux500/db8500-regs.h index 27399553c841..3d6e1955119a 100644 --- a/arch/arm/mach-ux500/db8500-regs.h +++ b/arch/arm/mach-ux500/db8500-regs.h @@ -41,10 +41,6 @@ /* ASIC ID is at 0xbf4 offset within this region */ #define U8500_ASIC_ID_BASE 0x9001D000 -#define U9540_BOOT_ROM_BASE 0xFFFE0000 -/* ASIC ID is at 0xbf4 offset within this region */ -#define U9540_ASIC_ID_BASE 0xFFFFD000 - #define U8500_PER6_BASE 0xa03c0000 #define U8500_PER7_BASE 0xa03d0000 #define U8500_PER5_BASE 0xa03e0000 diff --git a/arch/arm/mach-vexpress/spc.c b/arch/arm/mach-vexpress/spc.c index 21c064267af5..0f5381d13494 100644 --- a/arch/arm/mach-vexpress/spc.c +++ b/arch/arm/mach-vexpress/spc.c @@ -403,7 +403,7 @@ static int ve_spc_populate_opps(uint32_t cluster) uint32_t data = 0, off, ret, idx; struct ve_spc_opp *opps; - opps = kzalloc(sizeof(*opps) * MAX_OPPS, GFP_KERNEL); + opps = kcalloc(MAX_OPPS, sizeof(*opps), GFP_KERNEL); if (!opps) return -ENOMEM; diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig index 7f14acf67caf..96a7b6cf459b 100644 --- a/arch/arm/mm/Kconfig +++ b/arch/arm/mm/Kconfig @@ -415,6 +415,7 @@ config CPU_V7 select CPU_CP15_MPU if !MMU select CPU_HAS_ASID if MMU select CPU_PABRT_V7 + select CPU_SPECTRE if MMU select CPU_THUMB_CAPABLE select CPU_TLB_V7 if MMU @@ -661,6 +662,7 @@ config ARM_LPAE bool "Support for the Large Physical Address Extension" depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \ !CPU_32v4 && !CPU_32v3 + select PHYS_ADDR_T_64BIT help Say Y if you have an ARMv7 processor supporting the LPAE page table format and you would like to access memory beyond the @@ -673,12 +675,6 @@ config ARM_PV_FIXUP def_bool y depends on ARM_LPAE && ARM_PATCH_PHYS_VIRT && ARCH_KEYSTONE -config ARCH_PHYS_ADDR_T_64BIT - def_bool ARM_LPAE - -config ARCH_DMA_ADDR_T_64BIT - bool - config ARM_THUMB bool "Support Thumb user binaries" if !CPU_THUMBONLY && EXPERT depends on CPU_THUMB_CAPABLE @@ -826,6 +822,28 @@ config CPU_BPREDICT_DISABLE help Say Y here to disable branch prediction. If unsure, say N. +config CPU_SPECTRE + bool + +config HARDEN_BRANCH_PREDICTOR + bool "Harden the branch predictor against aliasing attacks" if EXPERT + depends on CPU_SPECTRE + default y + help + Speculation attacks against some high-performance processors rely + on being able to manipulate the branch predictor for a victim + context by executing aliasing branches in the attacker context. + Such attacks can be partially mitigated against by clearing + internal branch predictor state and limiting the prediction + logic in some situations. + + This config option will take CPU-specific actions to harden + the branch predictor against aliasing attacks and may rely on + specific instruction sequences or control bits being set by + the system firmware. + + If unsure, say Y. + config TLS_REG_EMUL bool select NEED_KUSER_HELPERS diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile index 9dbb84923e12..7cb1699fbfc4 100644 --- a/arch/arm/mm/Makefile +++ b/arch/arm/mm/Makefile @@ -10,7 +10,7 @@ obj-$(CONFIG_MMU) += fault-armv.o flush.o idmap.o ioremap.o \ ifneq ($(CONFIG_MMU),y) obj-y += nommu.o -obj-$(CONFIG_ARM_MPU) += pmsa-v7.o +obj-$(CONFIG_ARM_MPU) += pmsa-v7.o pmsa-v8.o endif obj-$(CONFIG_ARM_PTDUMP_CORE) += dump.o @@ -97,7 +97,7 @@ obj-$(CONFIG_CPU_MOHAWK) += proc-mohawk.o obj-$(CONFIG_CPU_FEROCEON) += proc-feroceon.o obj-$(CONFIG_CPU_V6) += proc-v6.o obj-$(CONFIG_CPU_V6K) += proc-v6.o -obj-$(CONFIG_CPU_V7) += proc-v7.o +obj-$(CONFIG_CPU_V7) += proc-v7.o proc-v7-bugs.o obj-$(CONFIG_CPU_V7M) += proc-v7m.o AFLAGS_proc-v6.o :=-Wa,-march=armv6 diff --git a/arch/arm/mm/alignment.c b/arch/arm/mm/alignment.c index 2c96190e018b..bd2c739d8083 100644 --- a/arch/arm/mm/alignment.c +++ b/arch/arm/mm/alignment.c @@ -950,6 +950,7 @@ do_alignment(unsigned long addr, unsigned int fsr, struct pt_regs *regs) if (ai_usermode & UM_SIGNAL) { siginfo_t si; + clear_siginfo(&si); si.si_signo = SIGBUS; si.si_errno = 0; si.si_code = BUS_ADRALN; diff --git a/arch/arm/mm/cache-b15-rac.c b/arch/arm/mm/cache-b15-rac.c index d9586ba2e63c..c6ed14840c3c 100644 --- a/arch/arm/mm/cache-b15-rac.c +++ b/arch/arm/mm/cache-b15-rac.c @@ -33,7 +33,10 @@ extern void v7_flush_kern_cache_all(void); #define RAC_CPU_SHIFT (8) #define RACCFG_MASK (0xff) #define RAC_CONFIG1_REG (0x7c) -#define RAC_FLUSH_REG (0x80) +/* Brahma-B15 is a quad-core only design */ +#define B15_RAC_FLUSH_REG (0x80) +/* Brahma-B53 is an octo-core design */ +#define B53_RAC_FLUSH_REG (0x84) #define FLUSH_RAC (1 << 0) /* Bitmask to enable instruction and data prefetching with a 256-bytes stride */ @@ -52,6 +55,7 @@ static void __iomem *b15_rac_base; static DEFINE_SPINLOCK(rac_lock); static u32 rac_config0_reg; +static u32 rac_flush_offset; /* Initialization flag to avoid checking for b15_rac_base, and to prevent * multi-platform kernels from crashing here as well. @@ -70,14 +74,14 @@ static inline void __b15_rac_flush(void) { u32 reg; - __raw_writel(FLUSH_RAC, b15_rac_base + RAC_FLUSH_REG); + __raw_writel(FLUSH_RAC, b15_rac_base + rac_flush_offset); do { /* This dmb() is required to force the Bus Interface Unit * to clean oustanding writes, and forces an idle cycle * to be inserted. */ dmb(); - reg = __raw_readl(b15_rac_base + RAC_FLUSH_REG); + reg = __raw_readl(b15_rac_base + rac_flush_offset); } while (reg & FLUSH_RAC); } @@ -287,7 +291,7 @@ static struct syscore_ops b15_rac_syscore_ops = { static int __init b15_rac_init(void) { - struct device_node *dn; + struct device_node *dn, *cpu_dn; int ret = 0, cpu; u32 reg, en_mask = 0; @@ -305,6 +309,24 @@ static int __init b15_rac_init(void) goto out; } + cpu_dn = of_get_cpu_node(0, NULL); + if (!cpu_dn) { + ret = -ENODEV; + goto out; + } + + if (of_device_is_compatible(cpu_dn, "brcm,brahma-b15")) + rac_flush_offset = B15_RAC_FLUSH_REG; + else if (of_device_is_compatible(cpu_dn, "brcm,brahma-b53")) + rac_flush_offset = B53_RAC_FLUSH_REG; + else { + pr_err("Unsupported CPU\n"); + of_node_put(cpu_dn); + ret = -EINVAL; + goto out; + } + of_node_put(cpu_dn); + ret = register_reboot_notifier(&b15_rac_reboot_nb); if (ret) { pr_err("failed to register reboot notifier\n"); diff --git a/arch/arm/mm/dma-mapping-nommu.c b/arch/arm/mm/dma-mapping-nommu.c index 619f24a42d09..f448a0663b10 100644 --- a/arch/arm/mm/dma-mapping-nommu.c +++ b/arch/arm/mm/dma-mapping-nommu.c @@ -241,12 +241,3 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, void arch_teardown_dma_ops(struct device *dev) { } - -#define PREALLOC_DMA_DEBUG_ENTRIES 4096 - -static int __init dma_debug_do_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - return 0; -} -core_initcall(dma_debug_do_init); diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index ada8eb206a90..be0fa7e39c26 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -831,7 +831,7 @@ static int __arm_dma_mmap(struct device *dev, struct vm_area_struct *vma, unsigned long attrs) { int ret; - unsigned long nr_vma_pages = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT; + unsigned long nr_vma_pages = vma_pages(vma); unsigned long nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; unsigned long pfn = dma_to_pfn(dev, dma_addr); unsigned long off = vma->vm_pgoff; @@ -1151,15 +1151,6 @@ int arm_dma_supported(struct device *dev, u64 mask) return __dma_supported(dev, mask, false); } -#define PREALLOC_DMA_DEBUG_ENTRIES 4096 - -static int __init dma_debug_do_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - return 0; -} -core_initcall(dma_debug_do_init); - #ifdef CONFIG_ARM_DMA_USE_IOMMU static int __dma_info_to_prot(enum dma_data_direction dir, unsigned long attrs) @@ -2171,8 +2162,8 @@ arm_iommu_create_mapping(struct bus_type *bus, dma_addr_t base, u64 size) goto err; mapping->bitmap_size = bitmap_size; - mapping->bitmaps = kzalloc(extensions * sizeof(unsigned long *), - GFP_KERNEL); + mapping->bitmaps = kcalloc(extensions, sizeof(unsigned long *), + GFP_KERNEL); if (!mapping->bitmaps) goto err2; diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c index b75eada23d0a..84becc911ee3 100644 --- a/arch/arm/mm/fault.c +++ b/arch/arm/mm/fault.c @@ -163,6 +163,11 @@ __do_user_fault(struct task_struct *tsk, unsigned long addr, { struct siginfo si; + if (addr > TASK_SIZE) + harden_branch_predictor(); + + clear_siginfo(&si); + #ifdef CONFIG_DEBUG_USER if (((user_debug & UDBG_SEGV) && (sig == SIGSEGV)) || ((user_debug & UDBG_BUS) && (sig == SIGBUS))) { @@ -557,6 +562,7 @@ do_DataAbort(unsigned long addr, unsigned int fsr, struct pt_regs *regs) inf->name, fsr, addr); show_pte(current->mm, addr); + clear_siginfo(&info); info.si_signo = inf->sig; info.si_errno = 0; info.si_code = inf->code; @@ -589,6 +595,7 @@ do_PrefetchAbort(unsigned long addr, unsigned int ifsr, struct pt_regs *regs) pr_alert("Unhandled prefetch abort: %s (0x%03x) at 0x%08lx\n", inf->name, ifsr, addr); + clear_siginfo(&info); info.si_signo = inf->sig; info.si_errno = 0; info.si_code = inf->code; diff --git a/arch/arm/mm/nommu.c b/arch/arm/mm/nommu.c index 7c087961b7ce..5dd6c58d653b 100644 --- a/arch/arm/mm/nommu.c +++ b/arch/arm/mm/nommu.c @@ -99,6 +99,38 @@ void __init arm_mm_memblock_reserve(void) memblock_reserve(0, 1); } +static void __init adjust_lowmem_bounds_mpu(void) +{ + unsigned long pmsa = read_cpuid_ext(CPUID_EXT_MMFR0) & MMFR0_PMSA; + + switch (pmsa) { + case MMFR0_PMSAv7: + pmsav7_adjust_lowmem_bounds(); + break; + case MMFR0_PMSAv8: + pmsav8_adjust_lowmem_bounds(); + break; + default: + break; + } +} + +static void __init mpu_setup(void) +{ + unsigned long pmsa = read_cpuid_ext(CPUID_EXT_MMFR0) & MMFR0_PMSA; + + switch (pmsa) { + case MMFR0_PMSAv7: + pmsav7_setup(); + break; + case MMFR0_PMSAv8: + pmsav8_setup(); + break; + default: + break; + } +} + void __init adjust_lowmem_bounds(void) { phys_addr_t end; diff --git a/arch/arm/mm/pgd.c b/arch/arm/mm/pgd.c index 61e281cb29fb..a1606d950251 100644 --- a/arch/arm/mm/pgd.c +++ b/arch/arm/mm/pgd.c @@ -20,7 +20,7 @@ #include "mm.h" #ifdef CONFIG_ARM_LPAE -#define __pgd_alloc() kmalloc(PTRS_PER_PGD * sizeof(pgd_t), GFP_KERNEL) +#define __pgd_alloc() kmalloc_array(PTRS_PER_PGD, sizeof(pgd_t), GFP_KERNEL) #define __pgd_free(pgd) kfree(pgd) #else #define __pgd_alloc() (pgd_t *)__get_free_pages(GFP_KERNEL, 2) diff --git a/arch/arm/mm/pmsa-v7.c b/arch/arm/mm/pmsa-v7.c index e2853bfff74e..699fa2e88725 100644 --- a/arch/arm/mm/pmsa-v7.c +++ b/arch/arm/mm/pmsa-v7.c @@ -102,7 +102,7 @@ static inline u32 irbar_read(void) static inline void rgnr_write(u32 v) { - writel_relaxed(v, BASEADDR_V7M_SCB + MPU_RNR); + writel_relaxed(v, BASEADDR_V7M_SCB + PMSAv7_RNR); } /* Data-side / unified region attributes */ @@ -110,28 +110,28 @@ static inline void rgnr_write(u32 v) /* Region access control register */ static inline void dracr_write(u32 v) { - u32 rsr = readl_relaxed(BASEADDR_V7M_SCB + MPU_RASR) & GENMASK(15, 0); + u32 rsr = readl_relaxed(BASEADDR_V7M_SCB + PMSAv7_RASR) & GENMASK(15, 0); - writel_relaxed((v << 16) | rsr, BASEADDR_V7M_SCB + MPU_RASR); + writel_relaxed((v << 16) | rsr, BASEADDR_V7M_SCB + PMSAv7_RASR); } /* Region size register */ static inline void drsr_write(u32 v) { - u32 racr = readl_relaxed(BASEADDR_V7M_SCB + MPU_RASR) & GENMASK(31, 16); + u32 racr = readl_relaxed(BASEADDR_V7M_SCB + PMSAv7_RASR) & GENMASK(31, 16); - writel_relaxed(v | racr, BASEADDR_V7M_SCB + MPU_RASR); + writel_relaxed(v | racr, BASEADDR_V7M_SCB + PMSAv7_RASR); } /* Region base address register */ static inline void drbar_write(u32 v) { - writel_relaxed(v, BASEADDR_V7M_SCB + MPU_RBAR); + writel_relaxed(v, BASEADDR_V7M_SCB + PMSAv7_RBAR); } static inline u32 drbar_read(void) { - return readl_relaxed(BASEADDR_V7M_SCB + MPU_RBAR); + return readl_relaxed(BASEADDR_V7M_SCB + PMSAv7_RBAR); } /* ARMv7-M only supports a unified MPU, so I-side operations are nop */ @@ -143,11 +143,6 @@ static inline unsigned long irbar_read(void) {return 0;} #endif -static int __init mpu_present(void) -{ - return ((read_cpuid_ext(CPUID_EXT_MMFR0) & MMFR0_PMSA) == MMFR0_PMSAv7); -} - static bool __init try_split_region(phys_addr_t base, phys_addr_t size, struct region *region) { unsigned long subreg, bslots, sslots; @@ -161,7 +156,7 @@ static bool __init try_split_region(phys_addr_t base, phys_addr_t size, struct r bdiff = base - abase; sdiff = p2size - asize; - subreg = p2size / MPU_NR_SUBREGS; + subreg = p2size / PMSAv7_NR_SUBREGS; if ((bdiff % subreg) || (sdiff % subreg)) return false; @@ -172,17 +167,17 @@ static bool __init try_split_region(phys_addr_t base, phys_addr_t size, struct r if (bslots || sslots) { int i; - if (subreg < MPU_MIN_SUBREG_SIZE) + if (subreg < PMSAv7_MIN_SUBREG_SIZE) return false; - if (bslots + sslots > MPU_NR_SUBREGS) + if (bslots + sslots > PMSAv7_NR_SUBREGS) return false; for (i = 0; i < bslots; i++) _set_bit(i, ®ion->subreg); for (i = 1; i <= sslots; i++) - _set_bit(MPU_NR_SUBREGS - i, ®ion->subreg); + _set_bit(PMSAv7_NR_SUBREGS - i, ®ion->subreg); } region->base = abase; @@ -233,7 +228,7 @@ static int __init allocate_region(phys_addr_t base, phys_addr_t size, } /* MPU initialisation functions */ -void __init adjust_lowmem_bounds_mpu(void) +void __init pmsav7_adjust_lowmem_bounds(void) { phys_addr_t specified_mem_size = 0, total_mem_size = 0; struct memblock_region *reg; @@ -243,10 +238,7 @@ void __init adjust_lowmem_bounds_mpu(void) unsigned int mem_max_regions; int num, i; - if (!mpu_present()) - return; - - /* Free-up MPU_PROBE_REGION */ + /* Free-up PMSAv7_PROBE_REGION */ mpu_min_region_order = __mpu_min_region_order(); /* How many regions are supported */ @@ -301,12 +293,12 @@ void __init adjust_lowmem_bounds_mpu(void) num = allocate_region(mem_start, specified_mem_size, mem_max_regions, mem); for (i = 0; i < num; i++) { - unsigned long subreg = mem[i].size / MPU_NR_SUBREGS; + unsigned long subreg = mem[i].size / PMSAv7_NR_SUBREGS; total_mem_size += mem[i].size - subreg * hweight_long(mem[i].subreg); pr_debug("MPU: base %pa size %pa disable subregions: %*pbl\n", - &mem[i].base, &mem[i].size, MPU_NR_SUBREGS, &mem[i].subreg); + &mem[i].base, &mem[i].size, PMSAv7_NR_SUBREGS, &mem[i].subreg); } if (total_mem_size != specified_mem_size) { @@ -349,7 +341,7 @@ static int __init __mpu_min_region_order(void) u32 drbar_result, irbar_result; /* We've kept a region free for this probing */ - rgnr_write(MPU_PROBE_REGION); + rgnr_write(PMSAv7_PROBE_REGION); isb(); /* * As per ARM ARM, write 0xFFFFFFFC to DRBAR to find the minimum @@ -388,8 +380,8 @@ static int __init mpu_setup_region(unsigned int number, phys_addr_t start, return -ENOMEM; /* Writing N to bits 5:1 (RSR_SZ) specifies region size 2^N+1 */ - size_data = ((size_order - 1) << MPU_RSR_SZ) | 1 << MPU_RSR_EN; - size_data |= subregions << MPU_RSR_SD; + size_data = ((size_order - 1) << PMSAv7_RSR_SZ) | 1 << PMSAv7_RSR_EN; + size_data |= subregions << PMSAv7_RSR_SD; if (need_flush) flush_cache_all(); @@ -424,18 +416,15 @@ static int __init mpu_setup_region(unsigned int number, phys_addr_t start, /* * Set up default MPU regions, doing nothing if there is no MPU */ -void __init mpu_setup(void) +void __init pmsav7_setup(void) { int i, region = 0, err = 0; - if (!mpu_present()) - return; - /* Setup MPU (order is important) */ /* Background */ err |= mpu_setup_region(region++, 0, 32, - MPU_ACR_XN | MPU_RGN_STRONGLY_ORDERED | MPU_AP_PL1RW_PL0RW, + PMSAv7_ACR_XN | PMSAv7_RGN_STRONGLY_ORDERED | PMSAv7_AP_PL1RW_PL0RW, 0, false); #ifdef CONFIG_XIP_KERNEL @@ -448,13 +437,13 @@ void __init mpu_setup(void) * with BG region (which is uncachable), thus we need * to clean and invalidate cache. */ - bool need_flush = region == MPU_RAM_REGION; + bool need_flush = region == PMSAv7_RAM_REGION; if (!xip[i].size) continue; err |= mpu_setup_region(region++, xip[i].base, ilog2(xip[i].size), - MPU_AP_PL1RO_PL0NA | MPU_RGN_NORMAL, + PMSAv7_AP_PL1RO_PL0NA | PMSAv7_RGN_NORMAL, xip[i].subreg, need_flush); } #endif @@ -465,14 +454,14 @@ void __init mpu_setup(void) continue; err |= mpu_setup_region(region++, mem[i].base, ilog2(mem[i].size), - MPU_AP_PL1RW_PL0RW | MPU_RGN_NORMAL, + PMSAv7_AP_PL1RW_PL0RW | PMSAv7_RGN_NORMAL, mem[i].subreg, false); } /* Vectors */ #ifndef CONFIG_CPU_V7M err |= mpu_setup_region(region++, vectors_base, ilog2(2 * PAGE_SIZE), - MPU_AP_PL1RW_PL0NA | MPU_RGN_NORMAL, + PMSAv7_AP_PL1RW_PL0NA | PMSAv7_RGN_NORMAL, 0, false); #endif if (err) { diff --git a/arch/arm/mm/pmsa-v8.c b/arch/arm/mm/pmsa-v8.c new file mode 100644 index 000000000000..617a83def88a --- /dev/null +++ b/arch/arm/mm/pmsa-v8.c @@ -0,0 +1,307 @@ +/* + * Based on linux/arch/arm/pmsa-v7.c + * + * ARM PMSAv8 supporting functions. + */ + +#include +#include + +#include +#include +#include + +#include +#include + +#include "mm.h" + +#ifndef CONFIG_CPU_V7M + +#define PRSEL __ACCESS_CP15(c6, 0, c2, 1) +#define PRBAR __ACCESS_CP15(c6, 0, c3, 0) +#define PRLAR __ACCESS_CP15(c6, 0, c3, 1) + +static inline u32 prlar_read(void) +{ + return read_sysreg(PRLAR); +} + +static inline u32 prbar_read(void) +{ + return read_sysreg(PRBAR); +} + +static inline void prsel_write(u32 v) +{ + write_sysreg(v, PRSEL); +} + +static inline void prbar_write(u32 v) +{ + write_sysreg(v, PRBAR); +} + +static inline void prlar_write(u32 v) +{ + write_sysreg(v, PRLAR); +} +#else + +static inline u32 prlar_read(void) +{ + return readl_relaxed(BASEADDR_V7M_SCB + PMSAv8_RLAR); +} + +static inline u32 prbar_read(void) +{ + return readl_relaxed(BASEADDR_V7M_SCB + PMSAv8_RBAR); +} + +static inline void prsel_write(u32 v) +{ + writel_relaxed(v, BASEADDR_V7M_SCB + PMSAv8_RNR); +} + +static inline void prbar_write(u32 v) +{ + writel_relaxed(v, BASEADDR_V7M_SCB + PMSAv8_RBAR); +} + +static inline void prlar_write(u32 v) +{ + writel_relaxed(v, BASEADDR_V7M_SCB + PMSAv8_RLAR); +} + +#endif + +static struct range __initdata io[MPU_MAX_REGIONS]; +static struct range __initdata mem[MPU_MAX_REGIONS]; + +static unsigned int __initdata mpu_max_regions; + +static __init bool is_region_fixed(int number) +{ + switch (number) { + case PMSAv8_XIP_REGION: + case PMSAv8_KERNEL_REGION: + return true; + default: + return false; + } +} + +void __init pmsav8_adjust_lowmem_bounds(void) +{ + phys_addr_t mem_end; + struct memblock_region *reg; + bool first = true; + + for_each_memblock(memory, reg) { + if (first) { + phys_addr_t phys_offset = PHYS_OFFSET; + + /* + * Initially only use memory continuous from + * PHYS_OFFSET */ + if (reg->base != phys_offset) + panic("First memory bank must be contiguous from PHYS_OFFSET"); + mem_end = reg->base + reg->size; + first = false; + } else { + /* + * memblock auto merges contiguous blocks, remove + * all blocks afterwards in one go (we can't remove + * blocks separately while iterating) + */ + pr_notice("Ignoring RAM after %pa, memory at %pa ignored\n", + &mem_end, ®->base); + memblock_remove(reg->base, 0 - reg->base); + break; + } + } +} + +static int __init __mpu_max_regions(void) +{ + static int max_regions; + u32 mpuir; + + if (max_regions) + return max_regions; + + mpuir = read_cpuid_mputype(); + + max_regions = (mpuir & MPUIR_DREGION_SZMASK) >> MPUIR_DREGION; + + return max_regions; +} + +static int __init __pmsav8_setup_region(unsigned int number, u32 bar, u32 lar) +{ + if (number > mpu_max_regions + || number >= MPU_MAX_REGIONS) + return -ENOENT; + + dsb(); + prsel_write(number); + isb(); + prbar_write(bar); + prlar_write(lar); + + mpu_rgn_info.rgns[number].prbar = bar; + mpu_rgn_info.rgns[number].prlar = lar; + + mpu_rgn_info.used++; + + return 0; +} + +static int __init pmsav8_setup_ram(unsigned int number, phys_addr_t start,phys_addr_t end) +{ + u32 bar, lar; + + if (is_region_fixed(number)) + return -EINVAL; + + bar = start; + lar = (end - 1) & ~(PMSAv8_MINALIGN - 1);; + + bar |= PMSAv8_AP_PL1RW_PL0RW | PMSAv8_RGN_SHARED; + lar |= PMSAv8_LAR_IDX(PMSAv8_RGN_NORMAL) | PMSAv8_LAR_EN; + + return __pmsav8_setup_region(number, bar, lar); +} + +static int __init pmsav8_setup_io(unsigned int number, phys_addr_t start,phys_addr_t end) +{ + u32 bar, lar; + + if (is_region_fixed(number)) + return -EINVAL; + + bar = start; + lar = (end - 1) & ~(PMSAv8_MINALIGN - 1);; + + bar |= PMSAv8_AP_PL1RW_PL0RW | PMSAv8_RGN_SHARED | PMSAv8_BAR_XN; + lar |= PMSAv8_LAR_IDX(PMSAv8_RGN_DEVICE_nGnRnE) | PMSAv8_LAR_EN; + + return __pmsav8_setup_region(number, bar, lar); +} + +static int __init pmsav8_setup_fixed(unsigned int number, phys_addr_t start,phys_addr_t end) +{ + u32 bar, lar; + + if (!is_region_fixed(number)) + return -EINVAL; + + bar = start; + lar = (end - 1) & ~(PMSAv8_MINALIGN - 1); + + bar |= PMSAv8_AP_PL1RW_PL0NA | PMSAv8_RGN_SHARED; + lar |= PMSAv8_LAR_IDX(PMSAv8_RGN_NORMAL) | PMSAv8_LAR_EN; + + prsel_write(number); + isb(); + + if (prbar_read() != bar || prlar_read() != lar) + return -EINVAL; + + /* Reserved region was set up early, we just need a record for secondaries */ + mpu_rgn_info.rgns[number].prbar = bar; + mpu_rgn_info.rgns[number].prlar = lar; + + mpu_rgn_info.used++; + + return 0; +} + +#ifndef CONFIG_CPU_V7M +static int __init pmsav8_setup_vector(unsigned int number, phys_addr_t start,phys_addr_t end) +{ + u32 bar, lar; + + if (number == PMSAv8_KERNEL_REGION) + return -EINVAL; + + bar = start; + lar = (end - 1) & ~(PMSAv8_MINALIGN - 1); + + bar |= PMSAv8_AP_PL1RW_PL0NA | PMSAv8_RGN_SHARED; + lar |= PMSAv8_LAR_IDX(PMSAv8_RGN_NORMAL) | PMSAv8_LAR_EN; + + return __pmsav8_setup_region(number, bar, lar); +} +#endif + +void __init pmsav8_setup(void) +{ + int i, err = 0; + int region = PMSAv8_KERNEL_REGION; + + /* How many regions are supported ? */ + mpu_max_regions = __mpu_max_regions(); + + /* RAM: single chunk of memory */ + add_range(mem, ARRAY_SIZE(mem), 0, memblock.memory.regions[0].base, + memblock.memory.regions[0].base + memblock.memory.regions[0].size); + + /* IO: cover full 4G range */ + add_range(io, ARRAY_SIZE(io), 0, 0, 0xffffffff); + + /* RAM and IO: exclude kernel */ + subtract_range(mem, ARRAY_SIZE(mem), __pa(KERNEL_START), __pa(KERNEL_END)); + subtract_range(io, ARRAY_SIZE(io), __pa(KERNEL_START), __pa(KERNEL_END)); + +#ifdef CONFIG_XIP_KERNEL + /* RAM and IO: exclude xip */ + subtract_range(mem, ARRAY_SIZE(mem), CONFIG_XIP_PHYS_ADDR, __pa(_exiprom)); + subtract_range(io, ARRAY_SIZE(io), CONFIG_XIP_PHYS_ADDR, __pa(_exiprom)); +#endif + +#ifndef CONFIG_CPU_V7M + /* RAM and IO: exclude vectors */ + subtract_range(mem, ARRAY_SIZE(mem), vectors_base, vectors_base + 2 * PAGE_SIZE); + subtract_range(io, ARRAY_SIZE(io), vectors_base, vectors_base + 2 * PAGE_SIZE); +#endif + /* IO: exclude RAM */ + for (i = 0; i < ARRAY_SIZE(mem); i++) + subtract_range(io, ARRAY_SIZE(io), mem[i].start, mem[i].end); + + /* Now program MPU */ + +#ifdef CONFIG_XIP_KERNEL + /* ROM */ + err |= pmsav8_setup_fixed(PMSAv8_XIP_REGION, CONFIG_XIP_PHYS_ADDR, __pa(_exiprom)); +#endif + /* Kernel */ + err |= pmsav8_setup_fixed(region++, __pa(KERNEL_START), __pa(KERNEL_END)); + + + /* IO */ + for (i = 0; i < ARRAY_SIZE(io); i++) { + if (!io[i].end) + continue; + + err |= pmsav8_setup_io(region++, io[i].start, io[i].end); + } + + /* RAM */ + for (i = 0; i < ARRAY_SIZE(mem); i++) { + if (!mem[i].end) + continue; + + err |= pmsav8_setup_ram(region++, mem[i].start, mem[i].end); + } + + /* Vectors */ +#ifndef CONFIG_CPU_V7M + err |= pmsav8_setup_vector(region++, vectors_base, vectors_base + 2 * PAGE_SIZE); +#endif + if (err) + pr_warn("MPU region initialization failure! %d", err); + else + pr_info("Using ARM PMSAv8 Compliant MPU. Used %d of %d regions\n", + mpu_rgn_info.used, mpu_max_regions); +} diff --git a/arch/arm/mm/proc-macros.S b/arch/arm/mm/proc-macros.S index f10e31d0730a..81d0efb055c6 100644 --- a/arch/arm/mm/proc-macros.S +++ b/arch/arm/mm/proc-macros.S @@ -273,13 +273,14 @@ mcr p15, 0, ip, c7, c10, 4 @ data write barrier .endm -.macro define_processor_functions name:req, dabort:req, pabort:req, nommu=0, suspend=0 +.macro define_processor_functions name:req, dabort:req, pabort:req, nommu=0, suspend=0, bugs=0 .type \name\()_processor_functions, #object .align 2 ENTRY(\name\()_processor_functions) .word \dabort .word \pabort .word cpu_\name\()_proc_init + .word \bugs .word cpu_\name\()_proc_fin .word cpu_\name\()_reset .word cpu_\name\()_do_idle diff --git a/arch/arm/mm/proc-v7-2level.S b/arch/arm/mm/proc-v7-2level.S index c6141a5435c3..f8d45ad2a515 100644 --- a/arch/arm/mm/proc-v7-2level.S +++ b/arch/arm/mm/proc-v7-2level.S @@ -41,11 +41,6 @@ * even on Cortex-A8 revisions not affected by 430973. * If IBE is not set, the flush BTAC/BTB won't do anything. */ -ENTRY(cpu_ca8_switch_mm) -#ifdef CONFIG_MMU - mov r2, #0 - mcr p15, 0, r2, c7, c5, 6 @ flush BTAC/BTB -#endif ENTRY(cpu_v7_switch_mm) #ifdef CONFIG_MMU mmid r1, r1 @ get mm->context.id @@ -66,7 +61,6 @@ ENTRY(cpu_v7_switch_mm) #endif bx lr ENDPROC(cpu_v7_switch_mm) -ENDPROC(cpu_ca8_switch_mm) /* * cpu_v7_set_pte_ext(ptep, pte) diff --git a/arch/arm/mm/proc-v7-bugs.c b/arch/arm/mm/proc-v7-bugs.c new file mode 100644 index 000000000000..5544b82a2e7a --- /dev/null +++ b/arch/arm/mm/proc-v7-bugs.c @@ -0,0 +1,174 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +#include +#include +#include +#include + +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR +DEFINE_PER_CPU(harden_branch_predictor_fn_t, harden_branch_predictor_fn); + +extern void cpu_v7_iciallu_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm); +extern void cpu_v7_bpiall_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm); +extern void cpu_v7_smc_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm); +extern void cpu_v7_hvc_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm); + +static void harden_branch_predictor_bpiall(void) +{ + write_sysreg(0, BPIALL); +} + +static void harden_branch_predictor_iciallu(void) +{ + write_sysreg(0, ICIALLU); +} + +static void __maybe_unused call_smc_arch_workaround_1(void) +{ + arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_1, NULL); +} + +static void __maybe_unused call_hvc_arch_workaround_1(void) +{ + arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_WORKAROUND_1, NULL); +} + +static void cpu_v7_spectre_init(void) +{ + const char *spectre_v2_method = NULL; + int cpu = smp_processor_id(); + + if (per_cpu(harden_branch_predictor_fn, cpu)) + return; + + switch (read_cpuid_part()) { + case ARM_CPU_PART_CORTEX_A8: + case ARM_CPU_PART_CORTEX_A9: + case ARM_CPU_PART_CORTEX_A12: + case ARM_CPU_PART_CORTEX_A17: + case ARM_CPU_PART_CORTEX_A73: + case ARM_CPU_PART_CORTEX_A75: + if (processor.switch_mm != cpu_v7_bpiall_switch_mm) + goto bl_error; + per_cpu(harden_branch_predictor_fn, cpu) = + harden_branch_predictor_bpiall; + spectre_v2_method = "BPIALL"; + break; + + case ARM_CPU_PART_CORTEX_A15: + case ARM_CPU_PART_BRAHMA_B15: + if (processor.switch_mm != cpu_v7_iciallu_switch_mm) + goto bl_error; + per_cpu(harden_branch_predictor_fn, cpu) = + harden_branch_predictor_iciallu; + spectre_v2_method = "ICIALLU"; + break; + +#ifdef CONFIG_ARM_PSCI + default: + /* Other ARM CPUs require no workaround */ + if (read_cpuid_implementor() == ARM_CPU_IMP_ARM) + break; + /* fallthrough */ + /* Cortex A57/A72 require firmware workaround */ + case ARM_CPU_PART_CORTEX_A57: + case ARM_CPU_PART_CORTEX_A72: { + struct arm_smccc_res res; + + if (psci_ops.smccc_version == SMCCC_VERSION_1_0) + break; + + switch (psci_ops.conduit) { + case PSCI_CONDUIT_HVC: + arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, + ARM_SMCCC_ARCH_WORKAROUND_1, &res); + if ((int)res.a0 != 0) + break; + if (processor.switch_mm != cpu_v7_hvc_switch_mm && cpu) + goto bl_error; + per_cpu(harden_branch_predictor_fn, cpu) = + call_hvc_arch_workaround_1; + processor.switch_mm = cpu_v7_hvc_switch_mm; + spectre_v2_method = "hypervisor"; + break; + + case PSCI_CONDUIT_SMC: + arm_smccc_1_1_smc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, + ARM_SMCCC_ARCH_WORKAROUND_1, &res); + if ((int)res.a0 != 0) + break; + if (processor.switch_mm != cpu_v7_smc_switch_mm && cpu) + goto bl_error; + per_cpu(harden_branch_predictor_fn, cpu) = + call_smc_arch_workaround_1; + processor.switch_mm = cpu_v7_smc_switch_mm; + spectre_v2_method = "firmware"; + break; + + default: + break; + } + } +#endif + } + + if (spectre_v2_method) + pr_info("CPU%u: Spectre v2: using %s workaround\n", + smp_processor_id(), spectre_v2_method); + return; + +bl_error: + pr_err("CPU%u: Spectre v2: incorrect context switching function, system vulnerable\n", + cpu); +} +#else +static void cpu_v7_spectre_init(void) +{ +} +#endif + +static __maybe_unused bool cpu_v7_check_auxcr_set(bool *warned, + u32 mask, const char *msg) +{ + u32 aux_cr; + + asm("mrc p15, 0, %0, c1, c0, 1" : "=r" (aux_cr)); + + if ((aux_cr & mask) != mask) { + if (!*warned) + pr_err("CPU%u: %s", smp_processor_id(), msg); + *warned = true; + return false; + } + return true; +} + +static DEFINE_PER_CPU(bool, spectre_warned); + +static bool check_spectre_auxcr(bool *warned, u32 bit) +{ + return IS_ENABLED(CONFIG_HARDEN_BRANCH_PREDICTOR) && + cpu_v7_check_auxcr_set(warned, bit, + "Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable\n"); +} + +void cpu_v7_ca8_ibe(void) +{ + if (check_spectre_auxcr(this_cpu_ptr(&spectre_warned), BIT(6))) + cpu_v7_spectre_init(); +} + +void cpu_v7_ca15_ibe(void) +{ + if (check_spectre_auxcr(this_cpu_ptr(&spectre_warned), BIT(0))) + cpu_v7_spectre_init(); +} + +void cpu_v7_bugs_init(void) +{ + cpu_v7_spectre_init(); +} diff --git a/arch/arm/mm/proc-v7.S b/arch/arm/mm/proc-v7.S index b528a15f460d..6fe52819e014 100644 --- a/arch/arm/mm/proc-v7.S +++ b/arch/arm/mm/proc-v7.S @@ -9,6 +9,7 @@ * * This is the "shell" of the ARMv7 processor support. */ +#include #include #include #include @@ -93,6 +94,37 @@ ENTRY(cpu_v7_dcache_clean_area) ret lr ENDPROC(cpu_v7_dcache_clean_area) +#ifdef CONFIG_ARM_PSCI + .arch_extension sec +ENTRY(cpu_v7_smc_switch_mm) + stmfd sp!, {r0 - r3} + movw r0, #:lower16:ARM_SMCCC_ARCH_WORKAROUND_1 + movt r0, #:upper16:ARM_SMCCC_ARCH_WORKAROUND_1 + smc #0 + ldmfd sp!, {r0 - r3} + b cpu_v7_switch_mm +ENDPROC(cpu_v7_smc_switch_mm) + .arch_extension virt +ENTRY(cpu_v7_hvc_switch_mm) + stmfd sp!, {r0 - r3} + movw r0, #:lower16:ARM_SMCCC_ARCH_WORKAROUND_1 + movt r0, #:upper16:ARM_SMCCC_ARCH_WORKAROUND_1 + hvc #0 + ldmfd sp!, {r0 - r3} + b cpu_v7_switch_mm +ENDPROC(cpu_v7_smc_switch_mm) +#endif +ENTRY(cpu_v7_iciallu_switch_mm) + mov r3, #0 + mcr p15, 0, r3, c7, c5, 0 @ ICIALLU + b cpu_v7_switch_mm +ENDPROC(cpu_v7_iciallu_switch_mm) +ENTRY(cpu_v7_bpiall_switch_mm) + mov r3, #0 + mcr p15, 0, r3, c7, c5, 6 @ flush BTAC/BTB + b cpu_v7_switch_mm +ENDPROC(cpu_v7_bpiall_switch_mm) + string cpu_v7_name, "ARMv7 Processor" .align @@ -158,31 +190,6 @@ ENTRY(cpu_v7_do_resume) ENDPROC(cpu_v7_do_resume) #endif -/* - * Cortex-A8 - */ - globl_equ cpu_ca8_proc_init, cpu_v7_proc_init - globl_equ cpu_ca8_proc_fin, cpu_v7_proc_fin - globl_equ cpu_ca8_reset, cpu_v7_reset - globl_equ cpu_ca8_do_idle, cpu_v7_do_idle - globl_equ cpu_ca8_dcache_clean_area, cpu_v7_dcache_clean_area - globl_equ cpu_ca8_set_pte_ext, cpu_v7_set_pte_ext - globl_equ cpu_ca8_suspend_size, cpu_v7_suspend_size -#ifdef CONFIG_ARM_CPU_SUSPEND - globl_equ cpu_ca8_do_suspend, cpu_v7_do_suspend - globl_equ cpu_ca8_do_resume, cpu_v7_do_resume -#endif - -/* - * Cortex-A9 processor functions - */ - globl_equ cpu_ca9mp_proc_init, cpu_v7_proc_init - globl_equ cpu_ca9mp_proc_fin, cpu_v7_proc_fin - globl_equ cpu_ca9mp_reset, cpu_v7_reset - globl_equ cpu_ca9mp_do_idle, cpu_v7_do_idle - globl_equ cpu_ca9mp_dcache_clean_area, cpu_v7_dcache_clean_area - globl_equ cpu_ca9mp_switch_mm, cpu_v7_switch_mm - globl_equ cpu_ca9mp_set_pte_ext, cpu_v7_set_pte_ext .globl cpu_ca9mp_suspend_size .equ cpu_ca9mp_suspend_size, cpu_v7_suspend_size + 4 * 2 #ifdef CONFIG_ARM_CPU_SUSPEND @@ -547,12 +554,79 @@ __v7_setup_stack: __INITDATA + .weak cpu_v7_bugs_init + @ define struct processor (see and proc-macros.S) - define_processor_functions v7, dabort=v7_early_abort, pabort=v7_pabort, suspend=1 + define_processor_functions v7, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_bugs_init + +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR + @ generic v7 bpiall on context switch + globl_equ cpu_v7_bpiall_proc_init, cpu_v7_proc_init + globl_equ cpu_v7_bpiall_proc_fin, cpu_v7_proc_fin + globl_equ cpu_v7_bpiall_reset, cpu_v7_reset + globl_equ cpu_v7_bpiall_do_idle, cpu_v7_do_idle + globl_equ cpu_v7_bpiall_dcache_clean_area, cpu_v7_dcache_clean_area + globl_equ cpu_v7_bpiall_set_pte_ext, cpu_v7_set_pte_ext + globl_equ cpu_v7_bpiall_suspend_size, cpu_v7_suspend_size +#ifdef CONFIG_ARM_CPU_SUSPEND + globl_equ cpu_v7_bpiall_do_suspend, cpu_v7_do_suspend + globl_equ cpu_v7_bpiall_do_resume, cpu_v7_do_resume +#endif + define_processor_functions v7_bpiall, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_bugs_init + +#define HARDENED_BPIALL_PROCESSOR_FUNCTIONS v7_bpiall_processor_functions +#else +#define HARDENED_BPIALL_PROCESSOR_FUNCTIONS v7_processor_functions +#endif + #ifndef CONFIG_ARM_LPAE - define_processor_functions ca8, dabort=v7_early_abort, pabort=v7_pabort, suspend=1 - define_processor_functions ca9mp, dabort=v7_early_abort, pabort=v7_pabort, suspend=1 + @ Cortex-A8 - always needs bpiall switch_mm implementation + globl_equ cpu_ca8_proc_init, cpu_v7_proc_init + globl_equ cpu_ca8_proc_fin, cpu_v7_proc_fin + globl_equ cpu_ca8_reset, cpu_v7_reset + globl_equ cpu_ca8_do_idle, cpu_v7_do_idle + globl_equ cpu_ca8_dcache_clean_area, cpu_v7_dcache_clean_area + globl_equ cpu_ca8_set_pte_ext, cpu_v7_set_pte_ext + globl_equ cpu_ca8_switch_mm, cpu_v7_bpiall_switch_mm + globl_equ cpu_ca8_suspend_size, cpu_v7_suspend_size +#ifdef CONFIG_ARM_CPU_SUSPEND + globl_equ cpu_ca8_do_suspend, cpu_v7_do_suspend + globl_equ cpu_ca8_do_resume, cpu_v7_do_resume +#endif + define_processor_functions ca8, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_ca8_ibe + + @ Cortex-A9 - needs more registers preserved across suspend/resume + @ and bpiall switch_mm for hardening + globl_equ cpu_ca9mp_proc_init, cpu_v7_proc_init + globl_equ cpu_ca9mp_proc_fin, cpu_v7_proc_fin + globl_equ cpu_ca9mp_reset, cpu_v7_reset + globl_equ cpu_ca9mp_do_idle, cpu_v7_do_idle + globl_equ cpu_ca9mp_dcache_clean_area, cpu_v7_dcache_clean_area +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR + globl_equ cpu_ca9mp_switch_mm, cpu_v7_bpiall_switch_mm +#else + globl_equ cpu_ca9mp_switch_mm, cpu_v7_switch_mm +#endif + globl_equ cpu_ca9mp_set_pte_ext, cpu_v7_set_pte_ext + define_processor_functions ca9mp, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_bugs_init #endif + + @ Cortex-A15 - needs iciallu switch_mm for hardening + globl_equ cpu_ca15_proc_init, cpu_v7_proc_init + globl_equ cpu_ca15_proc_fin, cpu_v7_proc_fin + globl_equ cpu_ca15_reset, cpu_v7_reset + globl_equ cpu_ca15_do_idle, cpu_v7_do_idle + globl_equ cpu_ca15_dcache_clean_area, cpu_v7_dcache_clean_area +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR + globl_equ cpu_ca15_switch_mm, cpu_v7_iciallu_switch_mm +#else + globl_equ cpu_ca15_switch_mm, cpu_v7_switch_mm +#endif + globl_equ cpu_ca15_set_pte_ext, cpu_v7_set_pte_ext + globl_equ cpu_ca15_suspend_size, cpu_v7_suspend_size + globl_equ cpu_ca15_do_suspend, cpu_v7_do_suspend + globl_equ cpu_ca15_do_resume, cpu_v7_do_resume + define_processor_functions ca15, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_ca15_ibe #ifdef CONFIG_CPU_PJ4B define_processor_functions pj4b, dabort=v7_early_abort, pabort=v7_pabort, suspend=1 #endif @@ -669,7 +743,7 @@ __v7_ca7mp_proc_info: __v7_ca12mp_proc_info: .long 0x410fc0d0 .long 0xff0ffff0 - __v7_proc __v7_ca12mp_proc_info, __v7_ca12mp_setup + __v7_proc __v7_ca12mp_proc_info, __v7_ca12mp_setup, proc_fns = HARDENED_BPIALL_PROCESSOR_FUNCTIONS .size __v7_ca12mp_proc_info, . - __v7_ca12mp_proc_info /* @@ -679,7 +753,7 @@ __v7_ca12mp_proc_info: __v7_ca15mp_proc_info: .long 0x410fc0f0 .long 0xff0ffff0 - __v7_proc __v7_ca15mp_proc_info, __v7_ca15mp_setup + __v7_proc __v7_ca15mp_proc_info, __v7_ca15mp_setup, proc_fns = ca15_processor_functions .size __v7_ca15mp_proc_info, . - __v7_ca15mp_proc_info /* @@ -689,7 +763,7 @@ __v7_ca15mp_proc_info: __v7_b15mp_proc_info: .long 0x420f00f0 .long 0xff0ffff0 - __v7_proc __v7_b15mp_proc_info, __v7_b15mp_setup, cache_fns = b15_cache_fns + __v7_proc __v7_b15mp_proc_info, __v7_b15mp_setup, proc_fns = ca15_processor_functions, cache_fns = b15_cache_fns .size __v7_b15mp_proc_info, . - __v7_b15mp_proc_info /* @@ -699,9 +773,25 @@ __v7_b15mp_proc_info: __v7_ca17mp_proc_info: .long 0x410fc0e0 .long 0xff0ffff0 - __v7_proc __v7_ca17mp_proc_info, __v7_ca17mp_setup + __v7_proc __v7_ca17mp_proc_info, __v7_ca17mp_setup, proc_fns = HARDENED_BPIALL_PROCESSOR_FUNCTIONS .size __v7_ca17mp_proc_info, . - __v7_ca17mp_proc_info + /* ARM Ltd. Cortex A73 processor */ + .type __v7_ca73_proc_info, #object +__v7_ca73_proc_info: + .long 0x410fd090 + .long 0xff0ffff0 + __v7_proc __v7_ca73_proc_info, __v7_setup, proc_fns = HARDENED_BPIALL_PROCESSOR_FUNCTIONS + .size __v7_ca73_proc_info, . - __v7_ca73_proc_info + + /* ARM Ltd. Cortex A75 processor */ + .type __v7_ca75_proc_info, #object +__v7_ca75_proc_info: + .long 0x410fd0a0 + .long 0xff0ffff0 + __v7_proc __v7_ca75_proc_info, __v7_setup, proc_fns = HARDENED_BPIALL_PROCESSOR_FUNCTIONS + .size __v7_ca75_proc_info, . - __v7_ca75_proc_info + /* * Qualcomm Inc. Krait processors. */ diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c index b5030e1a41d8..6e8b71613039 100644 --- a/arch/arm/net/bpf_jit_32.c +++ b/arch/arm/net/bpf_jit_32.c @@ -84,7 +84,7 @@ * * 1. First argument is passed using the arm 32bit registers and rest of the * arguments are passed on stack scratch space. - * 2. First callee-saved arugument is mapped to arm 32 bit registers and rest + * 2. First callee-saved argument is mapped to arm 32 bit registers and rest * arguments are mapped to scratch space on stack. * 3. We need two 64 bit temp registers to do complex operations on eBPF * registers. @@ -234,18 +234,11 @@ static void jit_fill_hole(void *area, unsigned int size) #define SCRATCH_SIZE 80 /* total stack size used in JITed code */ -#define _STACK_SIZE \ - (ctx->prog->aux->stack_depth + \ - + SCRATCH_SIZE + \ - + 4 /* extra for skb_copy_bits buffer */) - -#define STACK_SIZE ALIGN(_STACK_SIZE, STACK_ALIGNMENT) +#define _STACK_SIZE (ctx->prog->aux->stack_depth + SCRATCH_SIZE) +#define STACK_SIZE ALIGN(_STACK_SIZE, STACK_ALIGNMENT) /* Get the offset of eBPF REGISTERs stored on scratch space. */ -#define STACK_VAR(off) (STACK_SIZE-off-4) - -/* Offset of skb_copy_bits buffer */ -#define SKB_BUFFER STACK_VAR(SCRATCH_SIZE) +#define STACK_VAR(off) (STACK_SIZE - off) #if __LINUX_ARM_ARCH__ < 7 @@ -708,7 +701,7 @@ static inline void emit_a32_arsh_r64(const u8 dst[], const u8 src[], bool dstk, } /* dst = dst >> src */ -static inline void emit_a32_lsr_r64(const u8 dst[], const u8 src[], bool dstk, +static inline void emit_a32_rsh_r64(const u8 dst[], const u8 src[], bool dstk, bool sstk, struct jit_ctx *ctx) { const u8 *tmp = bpf2a32[TMP_REG_1]; const u8 *tmp2 = bpf2a32[TMP_REG_2]; @@ -724,7 +717,7 @@ static inline void emit_a32_lsr_r64(const u8 dst[], const u8 src[], bool dstk, emit(ARM_LDR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx); } - /* Do LSH operation */ + /* Do RSH operation */ emit(ARM_RSB_I(ARM_IP, rt, 32), ctx); emit(ARM_SUBS_I(tmp2[0], rt, 32), ctx); emit(ARM_MOV_SR(ARM_LR, rd, SRTYPE_LSR, rt), ctx); @@ -774,7 +767,7 @@ static inline void emit_a32_lsh_i64(const u8 dst[], bool dstk, } /* dst = dst >> val */ -static inline void emit_a32_lsr_i64(const u8 dst[], bool dstk, +static inline void emit_a32_rsh_i64(const u8 dst[], bool dstk, const u32 val, struct jit_ctx *ctx) { const u8 *tmp = bpf2a32[TMP_REG_1]; const u8 *tmp2 = bpf2a32[TMP_REG_2]; @@ -1199,8 +1192,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) s32 jmp_offset; #define check_imm(bits, imm) do { \ - if ((((imm) > 0) && ((imm) >> (bits))) || \ - (((imm) < 0) && (~(imm) >> (bits)))) { \ + if ((imm) >= (1 << ((bits) - 1)) || \ + (imm) < -(1 << ((bits) - 1))) { \ pr_info("[%2d] imm=%d(0x%x) out of range\n", \ i, imm, imm); \ return -EINVAL; \ @@ -1330,7 +1323,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) case BPF_ALU64 | BPF_RSH | BPF_K: if (unlikely(imm > 63)) return -EINVAL; - emit_a32_lsr_i64(dst, dstk, imm, ctx); + emit_a32_rsh_i64(dst, dstk, imm, ctx); break; /* dst = dst << src */ case BPF_ALU64 | BPF_LSH | BPF_X: @@ -1338,7 +1331,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) break; /* dst = dst >> src */ case BPF_ALU64 | BPF_RSH | BPF_X: - emit_a32_lsr_r64(dst, src, dstk, sstk, ctx); + emit_a32_rsh_r64(dst, src, dstk, sstk, ctx); break; /* dst = dst >> src (signed) */ case BPF_ALU64 | BPF_ARSH | BPF_X: @@ -1452,83 +1445,6 @@ exit: emit(ARM_LDR_I(rn, ARM_SP, STACK_VAR(src_lo)), ctx); emit_ldx_r(dst, rn, dstk, off, ctx, BPF_SIZE(code)); break; - /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + imm)) */ - case BPF_LD | BPF_ABS | BPF_W: - case BPF_LD | BPF_ABS | BPF_H: - case BPF_LD | BPF_ABS | BPF_B: - /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + src + imm)) */ - case BPF_LD | BPF_IND | BPF_W: - case BPF_LD | BPF_IND | BPF_H: - case BPF_LD | BPF_IND | BPF_B: - { - const u8 r4 = bpf2a32[BPF_REG_6][1]; /* r4 = ptr to sk_buff */ - const u8 r0 = bpf2a32[BPF_REG_0][1]; /*r0: struct sk_buff *skb*/ - /* rtn value */ - const u8 r1 = bpf2a32[BPF_REG_0][0]; /* r1: int k */ - const u8 r2 = bpf2a32[BPF_REG_1][1]; /* r2: unsigned int size */ - const u8 r3 = bpf2a32[BPF_REG_1][0]; /* r3: void *buffer */ - const u8 r6 = bpf2a32[TMP_REG_1][1]; /* r6: void *(*func)(..) */ - int size; - - /* Setting up first argument */ - emit(ARM_MOV_R(r0, r4), ctx); - - /* Setting up second argument */ - emit_a32_mov_i(r1, imm, false, ctx); - if (BPF_MODE(code) == BPF_IND) - emit_a32_alu_r(r1, src_lo, false, sstk, ctx, - false, false, BPF_ADD); - - /* Setting up third argument */ - switch (BPF_SIZE(code)) { - case BPF_W: - size = 4; - break; - case BPF_H: - size = 2; - break; - case BPF_B: - size = 1; - break; - default: - return -EINVAL; - } - emit_a32_mov_i(r2, size, false, ctx); - - /* Setting up fourth argument */ - emit(ARM_ADD_I(r3, ARM_SP, imm8m(SKB_BUFFER)), ctx); - - /* Setting up function pointer to call */ - emit_a32_mov_i(r6, (unsigned int)bpf_load_pointer, false, ctx); - emit_blx_r(r6, ctx); - - emit(ARM_EOR_R(r1, r1, r1), ctx); - /* Check if return address is NULL or not. - * if NULL then jump to epilogue - * else continue to load the value from retn address - */ - emit(ARM_CMP_I(r0, 0), ctx); - jmp_offset = epilogue_offset(ctx); - check_imm24(jmp_offset); - _emit(ARM_COND_EQ, ARM_B(jmp_offset), ctx); - - /* Load value from the address */ - switch (BPF_SIZE(code)) { - case BPF_W: - emit(ARM_LDR_I(r0, r0, 0), ctx); - emit_rev32(r0, r0, ctx); - break; - case BPF_H: - emit(ARM_LDRH_I(r0, r0, 0), ctx); - emit_rev16(r0, r0, ctx); - break; - case BPF_B: - emit(ARM_LDRB_I(r0, r0, 0), ctx); - /* No need to reverse */ - break; - } - break; - } /* ST: *(size *)(dst + off) = imm */ case BPF_ST | BPF_MEM | BPF_W: case BPF_ST | BPF_MEM | BPF_H: diff --git a/arch/arm/plat-omap/Kconfig b/arch/arm/plat-omap/Kconfig index afc1a1d4f7a5..c0a242cae79a 100644 --- a/arch/arm/plat-omap/Kconfig +++ b/arch/arm/plat-omap/Kconfig @@ -115,16 +115,6 @@ config OMAP_SERIAL_WAKE to data on the serial RX line. This allows you to wake the system from serial console. -choice - prompt "OMAP PM layer selection" - depends on ARCH_OMAP - default OMAP_PM_NOOP - -config OMAP_PM_NOOP - bool "No-op/debug PM layer" - -endchoice - endmenu endif diff --git a/arch/arm/plat-samsung/adc.c b/arch/arm/plat-samsung/adc.c index 42bac8d5ab5d..2da35735fa38 100644 --- a/arch/arm/plat-samsung/adc.c +++ b/arch/arm/plat-samsung/adc.c @@ -413,8 +413,7 @@ static int s3c_adc_remove(struct platform_device *pdev) #ifdef CONFIG_PM static int s3c_adc_suspend(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct adc_device *adc = platform_get_drvdata(pdev); + struct adc_device *adc = dev_get_drvdata(dev); unsigned long flags; u32 con; diff --git a/arch/arm/plat-samsung/include/plat/map-s5p.h b/arch/arm/plat-samsung/include/plat/map-s5p.h index f5769e93544a..d69a0ca09fb5 100644 --- a/arch/arm/plat-samsung/include/plat/map-s5p.h +++ b/arch/arm/plat-samsung/include/plat/map-s5p.h @@ -11,10 +11,6 @@ #define S5P_VA_CHIPID S3C_ADDR(0x02000000) -#define S5P_VA_COREPERI_BASE S3C_ADDR(0x02800000) -#define S5P_VA_COREPERI(x) (S5P_VA_COREPERI_BASE + (x)) -#define S5P_VA_SCU S5P_VA_COREPERI(0x0) - #define VA_VIC(x) (S3C_VA_IRQ + ((x) * 0x10000)) #define VA_VIC0 VA_VIC(0) #define VA_VIC1 VA_VIC(1) diff --git a/arch/arm/probes/kprobes/test-core.c b/arch/arm/probes/kprobes/test-core.c index 9ed0129bed3c..14db14152909 100644 --- a/arch/arm/probes/kprobes/test-core.c +++ b/arch/arm/probes/kprobes/test-core.c @@ -766,8 +766,9 @@ static int coverage_start_fn(const struct decode_header *h, void *args) static int coverage_start(const union decode_item *table) { - coverage.base = kmalloc(MAX_COVERAGE_ENTRIES * - sizeof(struct coverage_entry), GFP_KERNEL); + coverage.base = kmalloc_array(MAX_COVERAGE_ENTRIES, + sizeof(struct coverage_entry), + GFP_KERNEL); coverage.num_entries = 0; coverage.nesting = 0; return table_iter(table, coverage_start_fn, &coverage); diff --git a/arch/arm/tools/syscall.tbl b/arch/arm/tools/syscall.tbl index 0bb0e9c6376c..fbc74b5fa3ed 100644 --- a/arch/arm/tools/syscall.tbl +++ b/arch/arm/tools/syscall.tbl @@ -412,3 +412,4 @@ 395 common pkey_alloc sys_pkey_alloc 396 common pkey_free sys_pkey_free 397 common statx sys_statx +398 common rseq sys_rseq diff --git a/arch/arm/vdso/Makefile b/arch/arm/vdso/Makefile index bb4118213fee..f4efff9d3afb 100644 --- a/arch/arm/vdso/Makefile +++ b/arch/arm/vdso/Makefile @@ -30,6 +30,9 @@ CFLAGS_vgettimeofday.o = -O2 # Disable gcov profiling for VDSO code GCOV_PROFILE := n +# Prevents link failures: __sanitizer_cov_trace_pc() is not linked in. +KCOV_INSTRUMENT := n + # Force dependency $(obj)/vdso.o : $(obj)/vdso.so diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c index af4ee2cef2f9..35d0f823e823 100644 --- a/arch/arm/vfp/vfpmodule.c +++ b/arch/arm/vfp/vfpmodule.c @@ -218,8 +218,7 @@ static void vfp_raise_sigfpe(unsigned int sicode, struct pt_regs *regs) { siginfo_t info; - memset(&info, 0, sizeof(info)); - + clear_siginfo(&info); info.si_signo = SIGFPE; info.si_code = sicode; info.si_addr = (void __user *)(instruction_pointer(regs) - 4); diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c index ba7f4c8f5c3e..8073625371f5 100644 --- a/arch/arm/xen/enlighten.c +++ b/arch/arm/xen/enlighten.c @@ -89,6 +89,17 @@ int xen_unmap_domain_gfn_range(struct vm_area_struct *vma, } EXPORT_SYMBOL_GPL(xen_unmap_domain_gfn_range); +/* Not used by XENFEAT_auto_translated guests. */ +int xen_remap_domain_mfn_array(struct vm_area_struct *vma, + unsigned long addr, + xen_pfn_t *mfn, int nr, + int *err_ptr, pgprot_t prot, + unsigned int domid, struct page **pages) +{ + return -ENOSYS; +} +EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_array); + static void xen_read_wallclock(struct timespec64 *ts) { u32 version; diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index eb2cf4938f6d..42c090cf0292 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -7,16 +7,19 @@ config ARM64 select ACPI_REDUCED_HARDWARE_ONLY if ACPI select ACPI_MCFG if ACPI select ACPI_SPCR_TABLE if ACPI + select ACPI_PPTT if ACPI select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_ELF_RANDOMIZE + select ARCH_HAS_FAST_MULTIPLIER select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_GIGANTIC_PAGE if (MEMORY_ISOLATION && COMPACTION) || CMA select ARCH_HAS_KCOV select ARCH_HAS_MEMBARRIER_SYNC_CORE + select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_SET_MEMORY select ARCH_HAS_SG_CHAIN select ARCH_HAS_STRICT_KERNEL_RWX @@ -43,6 +46,7 @@ config ARM64 select ARCH_USE_QUEUED_RWLOCKS select ARCH_SUPPORTS_MEMORY_FAILURE select ARCH_SUPPORTS_ATOMIC_RMW + select ARCH_SUPPORTS_INT128 if GCC_VERSION >= 50000 || CC_IS_CLANG select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_WANT_COMPAT_IPC_PARSE_VERSION select ARCH_WANT_FRAME_POINTERS @@ -99,13 +103,11 @@ config ARM64 select HAVE_ARM_SMCCC select HAVE_EBPF_JIT select HAVE_C_RECORDMCOUNT - select HAVE_CC_STACKPROTECTOR select HAVE_CMPXCHG_DOUBLE select HAVE_CMPXCHG_LOCAL select HAVE_CONTEXT_TRACKING select HAVE_DEBUG_BUGVERBOSE select HAVE_DEBUG_KMEMLEAK - select HAVE_DMA_API_DEBUG select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_EFFICIENT_UNALIGNED_ACCESS @@ -125,6 +127,7 @@ config ARM64 select HAVE_PERF_USER_STACK_DUMP select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RCU_TABLE_FREE + select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES select HAVE_KRETPROBES @@ -133,6 +136,8 @@ config ARM64 select IRQ_FORCED_THREADING select MODULES_USE_ELF_RELA select MULTI_IRQ_HANDLER + select NEED_DMA_MAP_STATE + select NEED_SG_DMA_LENGTH select NO_BOOTMEM select OF select OF_EARLY_FLATTREE @@ -142,6 +147,7 @@ config ARM64 select POWER_SUPPLY select REFCOUNT_FULL select SPARSE_IRQ + select SWIOTLB select SYSCTL_EXCEPTION_TRACE select THREAD_INFO_IN_TASK help @@ -150,9 +156,6 @@ config ARM64 config 64BIT def_bool y -config ARCH_PHYS_ADDR_T_64BIT - def_bool y - config MMU def_bool y @@ -237,24 +240,9 @@ config ZONE_DMA32 config HAVE_GENERIC_GUP def_bool y -config ARCH_DMA_ADDR_T_64BIT - def_bool y - -config NEED_DMA_MAP_STATE - def_bool y - -config NEED_SG_DMA_LENGTH - def_bool y - config SMP def_bool y -config SWIOTLB - def_bool y - -config IOMMU_HELPER - def_bool SWIOTLB - config KERNEL_MODE_NEON def_bool y @@ -938,6 +926,15 @@ config HARDEN_EL2_VECTORS If unsure, say Y. +config ARM64_SSBD + bool "Speculative Store Bypass Disable" if EXPERT + default y + help + This enables mitigation of the bypassing of previous stores + by speculative loads. + + If unsure, say Y. + menuconfig ARMV8_DEPRECATED bool "Emulate deprecated/obsolete ARMv8 instructions" depends on COMPAT @@ -1049,6 +1046,7 @@ config ARM64_PAN config ARM64_LSE_ATOMICS bool "Atomic instructions" + default y help As part of the Large System Extensions, ARMv8.1 introduces new atomic instructions that are designed specifically to scale in @@ -1057,7 +1055,8 @@ config ARM64_LSE_ATOMICS Say Y here to make use of these instructions for the in-kernel atomic routines. This incurs a small overhead on CPUs that do not support these instructions and requires the kernel to be - built with binutils >= 2.25. + built with binutils >= 2.25 in order for the new instructions + to be used. config ARM64_VHE bool "Enable support for Virtualization Host Extensions (VHE)" @@ -1130,6 +1129,7 @@ endmenu config ARM64_SVE bool "ARM Scalable Vector Extension support" default y + depends on !KVM || ARM64_VHE help The Scalable Vector Extension (SVE) is an extension to the AArch64 execution state which complements and extends the SIMD functionality @@ -1155,6 +1155,12 @@ config ARM64_SVE booting the kernel. If unsure and you are not observing these symptoms, you should assume that it is safe to say Y. + CPUs that support SVE are architecturally required to support the + Virtualization Host Extensions (VHE), so the kernel makes no + provision for supporting SVE alongside KVM without VHE enabled. + Thus, you will need to enable CONFIG_ARM64_VHE if you want to support + KVM in the same kernel image. + config ARM64_MODULE_PLTS bool select HAVE_MOD_ARCH_SPECIFIC diff --git a/arch/arm64/Kconfig.platforms b/arch/arm64/Kconfig.platforms index 2b1535cdeb7c..d5aeac351fc3 100644 --- a/arch/arm64/Kconfig.platforms +++ b/arch/arm64/Kconfig.platforms @@ -208,6 +208,12 @@ config ARCH_R8A77980 help This enables support for the Renesas R-Car V3H SoC. +config ARCH_R8A77990 + bool "Renesas R-Car E3 SoC Platform" + depends on ARCH_RENESAS + help + This enables support for the Renesas R-Car E3 SoC. + config ARCH_R8A77995 bool "Renesas R-Car D3 SoC Platform" depends on ARCH_RENESAS diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 87f7d2f9f17c..45272266dafb 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -56,12 +56,6 @@ KBUILD_AFLAGS += $(lseinstr) $(brokengasinst) KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) -ifeq ($(cc-name),clang) -KBUILD_CFLAGS += -DCONFIG_ARCH_SUPPORTS_INT128 -else -KBUILD_CFLAGS += $(call cc-ifversion, -ge, 0500, -DCONFIG_ARCH_SUPPORTS_INT128) -endif - ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__ @@ -78,7 +72,7 @@ LDFLAGS += -maarch64linux UTS_MACHINE := aarch64 endif -CHECKFLAGS += -D__aarch64__ -m64 +CHECKFLAGS += -D__aarch64__ ifeq ($(CONFIG_ARM64_MODULE_PLTS),y) KBUILD_LDFLAGS_MODULE += -T $(srctree)/arch/arm64/kernel/module.lds diff --git a/arch/arm64/boot/dts/Makefile b/arch/arm64/boot/dts/Makefile index 4aa50b9b26bc..3543bc324553 100644 --- a/arch/arm64/boot/dts/Makefile +++ b/arch/arm64/boot/dts/Makefile @@ -22,5 +22,6 @@ subdir-y += renesas subdir-y += rockchip subdir-y += socionext subdir-y += sprd +subdir-y += synaptics subdir-y += xilinx subdir-y += zte diff --git a/arch/arm64/boot/dts/allwinner/Makefile b/arch/arm64/boot/dts/allwinner/Makefile index 8bebe7da5ed9..c31f90a49481 100644 --- a/arch/arm64/boot/dts/allwinner/Makefile +++ b/arch/arm64/boot/dts/allwinner/Makefile @@ -6,10 +6,11 @@ dtb-$(CONFIG_ARCH_SUNXI) += sun50i-a64-orangepi-win.dtb dtb-$(CONFIG_ARCH_SUNXI) += sun50i-a64-pine64-plus.dtb sun50i-a64-pine64.dtb dtb-$(CONFIG_ARCH_SUNXI) += sun50i-a64-sopine-baseboard.dtb dtb-$(CONFIG_ARCH_SUNXI) += sun50i-a64-teres-i.dtb +dtb-$(CONFIG_ARCH_SUNXI) += sun50i-h5-libretech-all-h3-cc.dtb +dtb-$(CONFIG_ARCH_SUNXI) += sun50i-h5-nanopi-neo2.dtb +dtb-$(CONFIG_ARCH_SUNXI) += sun50i-h5-nanopi-neo-plus2.dtb dtb-$(CONFIG_ARCH_SUNXI) += sun50i-h5-orangepi-pc2.dtb dtb-$(CONFIG_ARCH_SUNXI) += sun50i-h5-orangepi-prime.dtb dtb-$(CONFIG_ARCH_SUNXI) += sun50i-h5-orangepi-zero-plus.dtb dtb-$(CONFIG_ARCH_SUNXI) += sun50i-h5-orangepi-zero-plus2.dtb -dtb-$(CONFIG_ARCH_SUNXI) += sun50i-h5-nanopi-neo2.dtb -dtb-$(CONFIG_ARCH_SUNXI) += sun50i-h5-nanopi-neo-plus2.dtb dtb-$(CONFIG_ARCH_SUNXI) += sun50i-h6-pine-h64.dtb diff --git a/arch/arm64/boot/dts/allwinner/axp803.dtsi b/arch/arm64/boot/dts/allwinner/axp803.dtsi index ff8af52743ff..e5eae8bafc42 100644 --- a/arch/arm64/boot/dts/allwinner/axp803.dtsi +++ b/arch/arm64/boot/dts/allwinner/axp803.dtsi @@ -146,5 +146,10 @@ regulator-max-microvolt = <3000000>; regulator-name = "rtc-ldo"; }; + + reg_drivevbus: drivevbus { + regulator-name = "drivevbus"; + status = "disabled"; + }; }; }; diff --git a/arch/arm64/boot/dts/allwinner/sun50i-a64-bananapi-m64.dts b/arch/arm64/boot/dts/allwinner/sun50i-a64-bananapi-m64.dts index 2250dec9974c..0716b1441187 100644 --- a/arch/arm64/boot/dts/allwinner/sun50i-a64-bananapi-m64.dts +++ b/arch/arm64/boot/dts/allwinner/sun50i-a64-bananapi-m64.dts @@ -86,6 +86,10 @@ }; }; +&ehci0 { + status = "okay"; +}; + &ehci1 { status = "okay"; }; @@ -155,6 +159,10 @@ status = "okay"; }; +&ohci0 { + status = "okay"; +}; + &ohci1 { status = "okay"; }; @@ -167,6 +175,7 @@ reg = <0x3a3>; interrupt-parent = <&r_intc>; interrupts = <0 IRQ_TYPE_LEVEL_LOW>; + x-powers,drive-vbus-en; /* set N_VBUSEN as output pin */ }; }; @@ -254,6 +263,11 @@ regulator-name = "vcc-wifi-io"; }; +®_drivevbus { + regulator-name = "usb0-vbus"; + status = "okay"; +}; + ®_eldo1 { regulator-min-microvolt = <1800000>; regulator-max-microvolt = <1800000>; @@ -294,6 +308,13 @@ status = "okay"; }; +&usb_otg { + dr_mode = "otg"; + status = "okay"; +}; + &usbphy { + usb0_id_det-gpios = <&pio 7 9 GPIO_ACTIVE_HIGH>; /* PH9 */ + usb0_vbus-supply = <®_drivevbus>; status = "okay"; }; diff --git a/arch/arm64/boot/dts/allwinner/sun50i-h5-libretech-all-h3-cc.dts b/arch/arm64/boot/dts/allwinner/sun50i-h5-libretech-all-h3-cc.dts new file mode 100644 index 000000000000..95e113ce8699 --- /dev/null +++ b/arch/arm64/boot/dts/allwinner/sun50i-h5-libretech-all-h3-cc.dts @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR MIT) +/* + * Copyright (C) 2018 BayLibre, SAS + * Author: Neil Armstrong + */ + +/dts-v1/; +#include "sun50i-h5.dtsi" +#include + +/ { + model = "Libre Computer Board ALL-H3-CC H5"; + compatible = "libretech,all-h3-cc-h5", "allwinner,sun50i-h5"; +}; diff --git a/arch/arm64/boot/dts/allwinner/sun50i-h5.dtsi b/arch/arm64/boot/dts/allwinner/sun50i-h5.dtsi index e237c05cfdb4..62d646baac3c 100644 --- a/arch/arm64/boot/dts/allwinner/sun50i-h5.dtsi +++ b/arch/arm64/boot/dts/allwinner/sun50i-h5.dtsi @@ -47,7 +47,7 @@ #address-cells = <1>; #size-cells = <0>; - cpu@0 { + cpu0: cpu@0 { compatible = "arm,cortex-a53", "arm,armv8"; device_type = "cpu"; reg = <0>; diff --git a/arch/arm64/boot/dts/allwinner/sun50i-h6-pine-h64.dts b/arch/arm64/boot/dts/allwinner/sun50i-h6-pine-h64.dts index d36de5eb81f3..b6f2d6b2ecae 100644 --- a/arch/arm64/boot/dts/allwinner/sun50i-h6-pine-h64.dts +++ b/arch/arm64/boot/dts/allwinner/sun50i-h6-pine-h64.dts @@ -22,6 +22,16 @@ }; }; +&r_i2c { + status = "okay"; + + pcf8563: rtc@51 { + compatible = "nxp,pcf8563"; + reg = <0x51>; + #clock-cells = <0>; + }; +}; + &uart0 { pinctrl-names = "default"; pinctrl-0 = <&uart0_ph_pins>; diff --git a/arch/arm64/boot/dts/allwinner/sun50i-h6.dtsi b/arch/arm64/boot/dts/allwinner/sun50i-h6.dtsi index 56563150d61a..c72da8cd9ef5 100644 --- a/arch/arm64/boot/dts/allwinner/sun50i-h6.dtsi +++ b/arch/arm64/boot/dts/allwinner/sun50i-h6.dtsi @@ -4,6 +4,8 @@ */ #include +#include +#include / { interrupt-parent = <&gic>; @@ -115,7 +117,7 @@ , , ; - clocks = <&ccu 26>, <&osc24M>, <&osc32k>; + clocks = <&ccu CLK_APB1>, <&osc24M>, <&osc32k>; clock-names = "apb", "hosc", "losc"; gpio-controller; #gpio-cells = <3>; @@ -134,8 +136,8 @@ interrupts = ; reg-shift = <2>; reg-io-width = <4>; - clocks = <&ccu 70>; - resets = <&ccu 21>; + clocks = <&ccu CLK_BUS_UART0>; + resets = <&ccu RST_BUS_UART0>; status = "disabled"; }; @@ -145,8 +147,8 @@ interrupts = ; reg-shift = <2>; reg-io-width = <4>; - clocks = <&ccu 71>; - resets = <&ccu 22>; + clocks = <&ccu CLK_BUS_UART1>; + resets = <&ccu RST_BUS_UART1>; status = "disabled"; }; @@ -156,8 +158,8 @@ interrupts = ; reg-shift = <2>; reg-io-width = <4>; - clocks = <&ccu 72>; - resets = <&ccu 23>; + clocks = <&ccu CLK_BUS_UART2>; + resets = <&ccu RST_BUS_UART2>; status = "disabled"; }; @@ -167,9 +169,59 @@ interrupts = ; reg-shift = <2>; reg-io-width = <4>; - clocks = <&ccu 73>; - resets = <&ccu 24>; + clocks = <&ccu CLK_BUS_UART3>; + resets = <&ccu RST_BUS_UART3>; status = "disabled"; }; + + r_ccu: clock@7010000 { + compatible = "allwinner,sun50i-h6-r-ccu"; + reg = <0x07010000 0x400>; + clocks = <&osc24M>, <&osc32k>, <&iosc>, + <&ccu CLK_PLL_PERIPH0>; + clock-names = "hosc", "losc", "iosc", "pll-periph"; + #clock-cells = <1>; + #reset-cells = <1>; + }; + + r_intc: interrupt-controller@7021000 { + compatible = "allwinner,sun50i-h6-r-intc", + "allwinner,sun6i-a31-r-intc"; + interrupt-controller; + #interrupt-cells = <2>; + reg = <0x07021000 0x400>; + interrupts = ; + }; + + r_pio: pinctrl@7022000 { + compatible = "allwinner,sun50i-h6-r-pinctrl"; + reg = <0x07022000 0x400>; + interrupts = , + ; + clocks = <&r_ccu 2>, <&osc24M>, <&osc32k>; + clock-names = "apb", "hosc", "losc"; + gpio-controller; + #gpio-cells = <3>; + interrupt-controller; + #interrupt-cells = <3>; + + r_i2c_pins: r-i2c { + pins = "PL0", "PL1"; + function = "s_i2c"; + }; + }; + + r_i2c: i2c@7081400 { + compatible = "allwinner,sun6i-a31-i2c"; + reg = <0x07081400 0x400>; + interrupts = ; + clocks = <&r_ccu 8>; + resets = <&r_ccu 4>; + pinctrl-names = "default"; + pinctrl-0 = <&r_i2c_pins>; + status = "disabled"; + #address-cells = <1>; + #size-cells = <0>; + }; }; }; diff --git a/arch/arm64/boot/dts/altera/socfpga_stratix10.dtsi b/arch/arm64/boot/dts/altera/socfpga_stratix10.dtsi index c89d0c307f8d..e6b059378dc0 100644 --- a/arch/arm64/boot/dts/altera/socfpga_stratix10.dtsi +++ b/arch/arm64/boot/dts/altera/socfpga_stratix10.dtsi @@ -17,6 +17,7 @@ /dts-v1/; #include #include +#include / { compatible = "altr,socfpga-stratix10"; @@ -92,9 +93,32 @@ interrupt-parent = <&intc>; ranges = <0 0 0 0xffffffff>; - clkmgr@ffd1000 { - compatible = "altr,clk-mgr"; + clkmgr: clock-controller@ffd10000 { + compatible = "intel,stratix10-clkmgr"; reg = <0xffd10000 0x1000>; + #clock-cells = <1>; + }; + + clocks { + cb_intosc_hs_div2_clk: cb-intosc-hs-div2-clk { + #clock-cells = <0>; + compatible = "fixed-clock"; + }; + + cb_intosc_ls_clk: cb-intosc-ls-clk { + #clock-cells = <0>; + compatible = "fixed-clock"; + }; + + f2s_free_clk: f2s-free-clk { + #clock-cells = <0>; + compatible = "fixed-clock"; + }; + + osc1: osc1 { + #clock-cells = <0>; + compatible = "fixed-clock"; + }; }; gmac0: ethernet@ff800000 { @@ -105,6 +129,8 @@ mac-address = [00 00 00 00 00 00]; resets = <&rst EMAC0_RESET>; reset-names = "stmmaceth"; + clocks = <&clkmgr STRATIX10_EMAC0_CLK>; + clock-names = "stmmaceth"; status = "disabled"; }; @@ -116,6 +142,8 @@ mac-address = [00 00 00 00 00 00]; resets = <&rst EMAC1_RESET>; reset-names = "stmmaceth"; + clocks = <&clkmgr STRATIX10_EMAC1_CLK>; + clock-names = "stmmaceth"; status = "disabled"; }; @@ -127,6 +155,8 @@ mac-address = [00 00 00 00 00 00]; resets = <&rst EMAC2_RESET>; reset-names = "stmmaceth"; + clocks = <&clkmgr STRATIX10_EMAC2_CLK>; + clock-names = "stmmaceth"; status = "disabled"; }; @@ -177,6 +207,7 @@ reg = <0xffc02800 0x100>; interrupts = <0 103 4>; resets = <&rst I2C0_RESET>; + clocks = <&clkmgr STRATIX10_L4_SP_CLK>; status = "disabled"; }; @@ -187,6 +218,7 @@ reg = <0xffc02900 0x100>; interrupts = <0 104 4>; resets = <&rst I2C1_RESET>; + clocks = <&clkmgr STRATIX10_L4_SP_CLK>; status = "disabled"; }; @@ -197,6 +229,7 @@ reg = <0xffc02a00 0x100>; interrupts = <0 105 4>; resets = <&rst I2C2_RESET>; + clocks = <&clkmgr STRATIX10_L4_SP_CLK>; status = "disabled"; }; @@ -207,6 +240,7 @@ reg = <0xffc02b00 0x100>; interrupts = <0 106 4>; resets = <&rst I2C3_RESET>; + clocks = <&clkmgr STRATIX10_L4_SP_CLK>; status = "disabled"; }; @@ -217,6 +251,7 @@ reg = <0xffc02c00 0x100>; interrupts = <0 107 4>; resets = <&rst I2C4_RESET>; + clocks = <&clkmgr STRATIX10_L4_SP_CLK>; status = "disabled"; }; @@ -229,6 +264,9 @@ fifo-depth = <0x400>; resets = <&rst SDMMC_RESET>; reset-names = "reset"; + clocks = <&clkmgr STRATIX10_L4_MP_CLK>, + <&clkmgr STRATIX10_SDMMC_CLK>; + clock-names = "biu", "ciu"; status = "disabled"; }; @@ -237,6 +275,25 @@ reg = <0xffe00000 0x100000>; }; + pdma: pdma@ffda0000 { + compatible = "arm,pl330", "arm,primecell"; + reg = <0xffda0000 0x1000>; + interrupts = <0 81 4>, + <0 82 4>, + <0 83 4>, + <0 84 4>, + <0 85 4>, + <0 86 4>, + <0 87 4>, + <0 88 4>, + <0 89 4>; + #dma-cells = <1>; + #dma-channels = <8>; + #dma-requests = <32>; + clocks = <&clkmgr STRATIX10_L4_MAIN_CLK>; + clock-names = "apb_pclk"; + }; + rst: rstmgr@ffd11000 { #reset-cells = <1>; compatible = "altr,rst-mgr"; @@ -288,24 +345,32 @@ compatible = "snps,dw-apb-timer"; interrupts = <0 113 4>; reg = <0xffc03000 0x100>; + clocks = <&clkmgr STRATIX10_L4_SP_CLK>; + clock-names = "timer"; }; timer1: timer1@ffc03100 { compatible = "snps,dw-apb-timer"; interrupts = <0 114 4>; reg = <0xffc03100 0x100>; + clocks = <&clkmgr STRATIX10_L4_SP_CLK>; + clock-names = "timer"; }; timer2: timer2@ffd00000 { compatible = "snps,dw-apb-timer"; interrupts = <0 115 4>; reg = <0xffd00000 0x100>; + clocks = <&clkmgr STRATIX10_L4_SP_CLK>; + clock-names = "timer"; }; timer3: timer3@ffd00100 { compatible = "snps,dw-apb-timer"; interrupts = <0 116 4>; reg = <0xffd00100 0x100>; + clocks = <&clkmgr STRATIX10_L4_SP_CLK>; + clock-names = "timer"; }; uart0: serial0@ffc02000 { @@ -315,6 +380,7 @@ reg-shift = <2>; reg-io-width = <4>; resets = <&rst UART0_RESET>; + clocks = <&clkmgr STRATIX10_L4_SP_CLK>; status = "disabled"; }; @@ -325,6 +391,7 @@ reg-shift = <2>; reg-io-width = <4>; resets = <&rst UART1_RESET>; + clocks = <&clkmgr STRATIX10_L4_SP_CLK>; status = "disabled"; }; @@ -387,5 +454,17 @@ resets = <&rst WATCHDOG3_RESET>; status = "disabled"; }; + + eccmgr { + compatible = "altr,socfpga-s10-ecc-manager"; + interrupts = <0 15 4>, <0 95 4>; + interrupt-controller; + #interrupt-cells = <2>; + + sdramedac { + compatible = "altr,sdram-edac-s10"; + interrupts = <16 4>, <48 4>; + }; + }; }; }; diff --git a/arch/arm64/boot/dts/altera/socfpga_stratix10_socdk.dts b/arch/arm64/boot/dts/altera/socfpga_stratix10_socdk.dts index eaf13fe29287..f9b1ef12db48 100644 --- a/arch/arm64/boot/dts/altera/socfpga_stratix10_socdk.dts +++ b/arch/arm64/boot/dts/altera/socfpga_stratix10_socdk.dts @@ -50,6 +50,21 @@ /* We expect the bootloader to fill in the reg */ reg = <0 0 0 0>; }; + + ref_033v: 033-v-ref { + compatible = "regulator-fixed"; + regulator-name = "0.33V"; + regulator-min-microvolt = <330000>; + regulator-max-microvolt = <330000>; + }; + + soc { + clocks { + osc1 { + clock-frequency = <25000000>; + }; + }; + }; }; &gpio1 { @@ -79,7 +94,7 @@ rxd2-skew-ps = <420>; /* 0ps */ rxd3-skew-ps = <420>; /* 0ps */ txen-skew-ps = <0>; /* -420ps */ - txc-skew-ps = <1860>; /* 960ps */ + txc-skew-ps = <900>; /* 0ps */ rxdv-skew-ps = <420>; /* 0ps */ rxc-skew-ps = <1680>; /* 780ps */ }; @@ -105,3 +120,30 @@ &watchdog0 { status = "okay"; }; + +&i2c1 { + status = "okay"; + clock-frequency = <100000>; + + adc@14 { + compatible = "lltc,ltc2497"; + reg = <0x14>; + vref-supply = <&ref_033v>; + }; + + temp@4c { + compatible = "maxim,max1619"; + reg = <0x4c>; + }; + + eeprom@51 { + compatible = "atmel,24c32"; + reg = <0x51>; + pagesize = <32>; + }; + + rtc@68 { + compatible = "dallas,ds1339"; + reg = <0x68>; + }; +}; diff --git a/arch/arm64/boot/dts/amlogic/meson-axg-s400.dts b/arch/arm64/boot/dts/amlogic/meson-axg-s400.dts index 57eedced5a51..4b3331fbfe39 100644 --- a/arch/arm64/boot/dts/amlogic/meson-axg-s400.dts +++ b/arch/arm64/boot/dts/amlogic/meson-axg-s400.dts @@ -15,6 +15,53 @@ serial0 = &uart_AO; serial1 = &uart_A; }; + + vddio_boot: regulator-vddio_boot { + compatible = "regulator-fixed"; + regulator-name = "VDDIO_BOOT"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + }; + + vddao_3v3: regulator-vddao_3v3 { + compatible = "regulator-fixed"; + regulator-name = "VDDAO_3V3"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + }; + + vddio_ao18: regulator-vddio_ao18 { + compatible = "regulator-fixed"; + regulator-name = "VDDIO_AO18"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + }; + + vcc_3v3: regulator-vcc_3v3 { + compatible = "regulator-fixed"; + regulator-name = "VCC_3V3"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + }; + + emmc_pwrseq: emmc-pwrseq { + compatible = "mmc-pwrseq-emmc"; + reset-gpios = <&gpio BOOT_9 GPIO_ACTIVE_LOW>; + }; + + sdio_pwrseq: sdio-pwrseq { + compatible = "mmc-pwrseq-simple"; + reset-gpios = <&gpio GPIOX_7 GPIO_ACTIVE_LOW>; + clocks = <&wifi32k>; + clock-names = "ext_clock"; + }; + + wifi32k: wifi32k { + compatible = "pwm-clock"; + #clock-cells = <0>; + clock-frequency = <32768>; + pwms = <&pwm_ab 0 30518 0>; /* PWM_A at 32.768KHz */ + }; }; ðmac { @@ -47,3 +94,62 @@ pinctrl-0 = <&i2c1_z_pins>; pinctrl-names = "default"; }; + +&i2c_AO { + status = "okay"; + pinctrl-0 = <&i2c_ao_sck_10_pins>, <&i2c_ao_sda_11_pins>; + pinctrl-names = "default"; +}; + +&pwm_ab { + status = "okay"; + pinctrl-0 = <&pwm_a_x20_pins>; + pinctrl-names = "default"; +}; + +/* emmc storage */ +&sd_emmc_c { + status = "okay"; + pinctrl-0 = <&emmc_pins>; + pinctrl-1 = <&emmc_clk_gate_pins>; + pinctrl-names = "default", "clk-gate"; + + bus-width = <8>; + cap-sd-highspeed; + cap-mmc-highspeed; + max-frequency = <180000000>; + non-removable; + disable-wp; + mmc-ddr-1_8v; + mmc-hs200-1_8v; + + vmmc-supply = <&vcc_3v3>; + vqmmc-supply = <&vddio_boot>; +}; + +/* wifi module */ +&sd_emmc_b { + status = "okay"; + #address-cells = <1>; + #size-cells = <0>; + + pinctrl-0 = <&sdio_pins>; + pinctrl-1 = <&sdio_clk_gate_pins>; + pinctrl-names = "default", "clk-gate"; + + bus-width = <4>; + cap-sd-highspeed; + max-frequency = <100000000>; + non-removable; + disable-wp; + + mmc-pwrseq = <&sdio_pwrseq>; + + vmmc-supply = <&vddao_3v3>; + vqmmc-supply = <&vddio_boot>; + + brcmf: wifi@1 { + reg = <1>; + compatible = "brcm,bcm4329-fmac"; + }; +}; diff --git a/arch/arm64/boot/dts/amlogic/meson-axg.dtsi b/arch/arm64/boot/dts/amlogic/meson-axg.dtsi index b58808eb3cc8..fee87737a201 100644 --- a/arch/arm64/boot/dts/amlogic/meson-axg.dtsi +++ b/arch/arm64/boot/dts/amlogic/meson-axg.dtsi @@ -7,6 +7,9 @@ #include #include #include +#include +#include +#include / { compatible = "amlogic,meson-axg"; @@ -107,12 +110,51 @@ #clock-cells = <0>; }; + ao_alt_xtal: ao_alt_xtal-clk { + compatible = "fixed-clock"; + clock-frequency = <32000000>; + clock-output-names = "ao_alt_xtal"; + #clock-cells = <0>; + }; + soc { compatible = "simple-bus"; #address-cells = <2>; #size-cells = <2>; ranges; + apb: apb@ffe00000 { + compatible = "simple-bus"; + reg = <0x0 0xffe00000 0x0 0x200000>; + #address-cells = <2>; + #size-cells = <2>; + ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x200000>; + + sd_emmc_b: sd@5000 { + compatible = "amlogic,meson-axg-mmc"; + reg = <0x0 0x5000 0x0 0x2000>; + interrupts = ; + status = "disabled"; + clocks = <&clkc CLKID_SD_EMMC_B>, + <&clkc CLKID_SD_EMMC_B_CLK0>, + <&clkc CLKID_FCLK_DIV2>; + clock-names = "core", "clkin0", "clkin1"; + resets = <&reset RESET_SD_EMMC_B>; + }; + + sd_emmc_c: mmc@7000 { + compatible = "amlogic,meson-axg-mmc"; + reg = <0x0 0x7000 0x0 0x2000>; + interrupts = ; + status = "disabled"; + clocks = <&clkc CLKID_SD_EMMC_C>, + <&clkc CLKID_SD_EMMC_C_CLK0>, + <&clkc CLKID_FCLK_DIV2>; + clock-names = "core", "clkin0", "clkin1"; + resets = <&reset RESET_SD_EMMC_C>; + }; + }; + cbus: bus@ffd00000 { compatible = "simple-bus"; reg = <0x0 0xffd00000 0x0 0x25000>; @@ -120,6 +162,15 @@ #size-cells = <2>; ranges = <0x0 0x0 0x0 0xffd00000 0x0 0x25000>; + gpio_intc: interrupt-controller@f080 { + compatible = "amlogic,meson-gpio-intc"; + reg = <0x0 0xf080 0x0 0x10>; + interrupt-controller; + #interrupt-cells = <2>; + amlogic,channel-interrupts = <64 65 66 67 68 69 70 71>; + status = "disabled"; + }; + pwm_ab: pwm@1b000 { compatible = "amlogic,meson-axg-ee-pwm"; reg = <0x0 0x1b000 0x0 0x20>; @@ -164,50 +215,42 @@ i2c0: i2c@1f000 { compatible = "amlogic,meson-axg-i2c"; - status = "disabled"; reg = <0x0 0x1f000 0x0 0x20>; - interrupts = , - ; + interrupts = ; + clocks = <&clkc CLKID_I2C>; #address-cells = <1>; #size-cells = <0>; - clocks = <&clkc CLKID_I2C>; - clock-names = "clk_i2c"; + status = "disabled"; }; i2c1: i2c@1e000 { compatible = "amlogic,meson-axg-i2c"; + reg = <0x0 0x1e000 0x0 0x20>; + interrupts = ; + clocks = <&clkc CLKID_I2C>; #address-cells = <1>; #size-cells = <0>; - reg = <0x0 0x1e000 0x0 0x20>; status = "disabled"; - interrupts = , - ; - clocks = <&clkc CLKID_I2C>; - clock-names = "clk_i2c"; }; i2c2: i2c@1d000 { compatible = "amlogic,meson-axg-i2c"; - status = "disabled"; reg = <0x0 0x1d000 0x0 0x20>; - interrupts = , - ; + interrupts = ; + clocks = <&clkc CLKID_I2C>; #address-cells = <1>; #size-cells = <0>; - clocks = <&clkc CLKID_I2C>; - clock-names = "clk_i2c"; + status = "disabled"; }; i2c3: i2c@1c000 { compatible = "amlogic,meson-axg-i2c"; - status = "disabled"; reg = <0x0 0x1c000 0x0 0x20>; - interrupts = , - ; + interrupts = ; + clocks = <&clkc CLKID_I2C>; #address-cells = <1>; #size-cells = <0>; - clocks = <&clkc CLKID_I2C>; - clock-names = "clk_i2c"; + status = "disabled"; }; uart_A: serial@24000 { @@ -262,10 +305,14 @@ #size-cells = <2>; ranges = <0x0 0x0 0x0 0xff63c000 0x0 0x1c00>; - clkc: clock-controller@0 { - compatible = "amlogic,axg-clkc"; - #clock-cells = <1>; - reg = <0x0 0x0 0x0 0x320>; + sysctrl: system-controller@0 { + compatible = "amlogic,meson-axg-hhi-sysctrl", "syscon", "simple-mfd"; + reg = <0 0 0 0x400>; + + clkc: clock-controller { + compatible = "amlogic,axg-clkc"; + #clock-cells = <1>; + }; }; }; @@ -309,6 +356,57 @@ gpio-ranges = <&pinctrl_periphs 0 0 86>; }; + emmc_pins: emmc { + mux { + groups = "emmc_nand_d0", + "emmc_nand_d1", + "emmc_nand_d2", + "emmc_nand_d3", + "emmc_nand_d4", + "emmc_nand_d5", + "emmc_nand_d6", + "emmc_nand_d7", + "emmc_clk", + "emmc_cmd", + "emmc_ds"; + function = "emmc"; + }; + }; + + emmc_clk_gate_pins: emmc_clk_gate { + mux { + groups = "BOOT_8"; + function = "gpio_periphs"; + }; + cfg-pull-down { + pins = "BOOT_8"; + bias-pull-down; + }; + }; + + sdio_pins: sdio { + mux { + groups = "sdio_d0", + "sdio_d1", + "sdio_d2", + "sdio_d3", + "sdio_cmd", + "sdio_clk"; + function = "sdio"; + }; + }; + + sdio_clk_gate_pins: sdio_clk_gate { + mux { + groups = "GPIOX_4"; + function = "gpio_periphs"; + }; + cfg-pull-down { + pins = "GPIOX_4"; + bias-pull-down; + }; + }; + eth_rmii_x_pins: eth-x-rmii { mux { groups = "eth_mdio_x", @@ -660,6 +758,251 @@ function = "uart_ao_b_z"; }; }; + + mclk_b_pins: mclk_b { + mux { + groups = "mclk_b"; + function = "mclk_b"; + }; + }; + + mclk_c_pins: mclk_c { + mux { + groups = "mclk_c"; + function = "mclk_c"; + }; + }; + + tdma_sclk_pins: tdma_sclk { + mux { + groups = "tdma_sclk"; + function = "tdma"; + }; + }; + + tdma_sclk_slv_pins: tdma_sclk_slv { + mux { + groups = "tdma_sclk_slv"; + function = "tdma"; + }; + }; + + tdma_fs_pins: tdma_fs { + mux { + groups = "tdma_fs"; + function = "tdma"; + }; + }; + + tdma_fs_slv_pins: tdma_fs_slv { + mux { + groups = "tdma_fs_slv"; + function = "tdma"; + }; + }; + + tdma_din0_pins: tdma_din0 { + mux { + groups = "tdma_din0"; + function = "tdma"; + }; + }; + + tdma_dout0_x14_pins: tdma_dout0_x14 { + mux { + groups = "tdma_dout0_x14"; + function = "tdma"; + }; + }; + + tdma_dout0_x15_pins: tdma_dout0_x15 { + mux { + groups = "tdma_dout0_x15"; + function = "tdma"; + }; + }; + + tdma_dout1_pins: tdma_dout1 { + mux { + groups = "tdma_dout1"; + function = "tdma"; + }; + }; + + tdma_din1_pins: tdma_din1 { + mux { + groups = "tdma_din1"; + function = "tdma"; + }; + }; + + tdmb_sclk_pins: tdmb_sclk { + mux { + groups = "tdmb_sclk"; + function = "tdmb"; + }; + }; + + tdmb_sclk_slv_pins: tdmb_sclk_slv { + mux { + groups = "tdmb_sclk_slv"; + function = "tdmb"; + }; + }; + + tdmb_fs_pins: tdmb_fs { + mux { + groups = "tdmb_fs"; + function = "tdmb"; + }; + }; + + tdmb_fs_slv_pins: tdmb_fs_slv { + mux { + groups = "tdmb_fs_slv"; + function = "tdmb"; + }; + }; + + tdmb_din0_pins: tdmb_din0 { + mux { + groups = "tdmb_din0"; + function = "tdmb"; + }; + }; + + tdmb_dout0_pins: tdmb_dout0 { + mux { + groups = "tdmb_dout0"; + function = "tdmb"; + }; + }; + + tdmb_din1_pins: tdmb_din1 { + mux { + groups = "tdmb_din1"; + function = "tdmb"; + }; + }; + + tdmb_dout1_pins: tdmb_dout1 { + mux { + groups = "tdmb_dout1"; + function = "tdmb"; + }; + }; + + tdmb_din2_pins: tdmb_din2 { + mux { + groups = "tdmb_din2"; + function = "tdmb"; + }; + }; + + tdmb_dout2_pins: tdmb_dout2 { + mux { + groups = "tdmb_dout2"; + function = "tdmb"; + }; + }; + + tdmb_din3_pins: tdmb_din3 { + mux { + groups = "tdmb_din3"; + function = "tdmb"; + }; + }; + + tdmb_dout3_pins: tdmb_dout3 { + mux { + groups = "tdmb_dout3"; + function = "tdmb"; + }; + }; + + tdmc_sclk_pins: tdmc_sclk { + mux { + groups = "tdmc_sclk"; + function = "tdmc"; + }; + }; + + tdmc_sclk_slv_pins: tdmc_sclk_slv { + mux { + groups = "tdmc_sclk_slv"; + function = "tdmc"; + }; + }; + + tdmc_fs_pins: tdmc_fs { + mux { + groups = "tdmc_fs"; + function = "tdmc"; + }; + }; + + tdmc_fs_slv_pins: tdmc_fs_slv { + mux { + groups = "tdmc_fs_slv"; + function = "tdmc"; + }; + }; + + tdmc_din0_pins: tdmc_din0 { + mux { + groups = "tdmc_din0"; + function = "tdmc"; + }; + }; + + tdmc_dout0_pins: tdmc_dout0 { + mux { + groups = "tdmc_dout0"; + function = "tdmc"; + }; + }; + + tdmc_din1_pins: tdmc_din1 { + mux { + groups = "tdmc_din1"; + function = "tdmc"; + }; + }; + + tdmc_dout1_pins: tdmc_dout1 { + mux { + groups = "tdmc_dout1"; + function = "tdmc"; + }; + }; + + tdmc_din2_pins: tdmc_din2 { + mux { + groups = "tdmc_din2"; + function = "tdmc"; + }; + }; + + tdmc_dout2_pins: tdmc_dout2 { + mux { + groups = "tdmc_dout2"; + function = "tdmc"; + }; + }; + + tdmc_din3_pins: tdmc_din3 { + mux { + groups = "tdmc_din3"; + function = "tdmc"; + }; + }; + + tdmc_dout3_pins: tdmc_dout3 { + mux { + groups = "tdmc_dout3"; + function = "tdmc"; + }; + }; }; }; @@ -688,6 +1031,17 @@ #size-cells = <2>; ranges = <0x0 0x0 0x0 0xff800000 0x0 0x100000>; + sysctrl_AO: sys-ctrl@0 { + compatible = "amlogic,meson-axg-ao-sysctrl", "syscon", "simple-mfd"; + reg = <0x0 0x0 0x0 0x100>; + + clkc_AO: clock-controller { + compatible = "amlogic,meson-axg-aoclkc"; + #clock-cells = <1>; + #reset-cells = <1>; + }; + }; + pinctrl_aobus: pinctrl@14 { compatible = "amlogic,meson-axg-aobus-pinctrl"; #address-cells = <2>; @@ -704,6 +1058,48 @@ gpio-ranges = <&pinctrl_aobus 0 0 15>; }; + i2c_ao_sck_4_pins: i2c_ao_sck_4 { + mux { + groups = "i2c_ao_sck_4"; + function = "i2c_ao"; + }; + }; + + i2c_ao_sck_8_pins: i2c_ao_sck_8 { + mux { + groups = "i2c_ao_sck_8"; + function = "i2c_ao"; + }; + }; + + i2c_ao_sck_10_pins: i2c_ao_sck_10 { + mux { + groups = "i2c_ao_sck_10"; + function = "i2c_ao"; + }; + }; + + i2c_ao_sda_5_pins: i2c_ao_sda_5 { + mux { + groups = "i2c_ao_sda_5"; + function = "i2c_ao"; + }; + }; + + i2c_ao_sda_9_pins: i2c_ao_sda_9 { + mux { + groups = "i2c_ao_sda_9"; + function = "i2c_ao"; + }; + }; + + i2c_ao_sda_11_pins: i2c_ao_sda_11 { + mux { + groups = "i2c_ao_sda_11"; + function = "i2c_ao"; + }; + }; + remote_input_ao_pins: remote_input_ao { mux { groups = "remote_input_ao"; @@ -766,20 +1162,19 @@ i2c_AO: i2c@5000 { compatible = "amlogic,meson-axg-i2c"; - status = "disabled"; reg = <0x0 0x05000 0x0 0x20>; interrupts = ; + clocks = <&clkc CLKID_AO_I2C>; #address-cells = <1>; #size-cells = <0>; - clocks = <&clkc CLKID_I2C>; - clock-names = "clk_i2c"; + status = "disabled"; }; uart_AO: serial@3000 { compatible = "amlogic,meson-gx-uart", "amlogic,meson-ao-uart"; reg = <0x0 0x3000 0x0 0x18>; interrupts = ; - clocks = <&xtal>, <&clkc CLKID_CLK81>, <&xtal>; + clocks = <&xtal>, <&clkc_AO CLKID_AO_UART1>, <&xtal>; clock-names = "xtal", "pclk", "baud"; status = "disabled"; }; @@ -788,7 +1183,7 @@ compatible = "amlogic,meson-gx-uart", "amlogic,meson-ao-uart"; reg = <0x0 0x4000 0x0 0x18>; interrupts = ; - clocks = <&xtal>, <&clkc CLKID_CLK81>, <&xtal>; + clocks = <&xtal>, <&clkc_AO CLKID_AO_UART2>, <&xtal>; clock-names = "xtal", "pclk", "baud"; status = "disabled"; }; diff --git a/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi b/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi index 562c26a0ba33..98cbba6809ca 100644 --- a/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi +++ b/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi @@ -307,11 +307,10 @@ clock-names = "isfr", "iahb", "venci"; }; -&hiubus { - clkc: clock-controller@0 { +&sysctrl { + clkc: clock-controller { compatible = "amlogic,gxbb-clkc"; #clock-cells = <1>; - reg = <0x0 0x0 0x0 0x3db>; }; }; @@ -716,6 +715,7 @@ <&clkc CLKID_SD_EMMC_A_CLK0>, <&clkc CLKID_FCLK_DIV2>; clock-names = "core", "clkin0", "clkin1"; + resets = <&reset RESET_SD_EMMC_A>; }; &sd_emmc_b { @@ -723,6 +723,7 @@ <&clkc CLKID_SD_EMMC_B_CLK0>, <&clkc CLKID_FCLK_DIV2>; clock-names = "core", "clkin0", "clkin1"; + resets = <&reset RESET_SD_EMMC_B>; }; &sd_emmc_c { @@ -730,6 +731,7 @@ <&clkc CLKID_SD_EMMC_C_CLK0>, <&clkc CLKID_FCLK_DIV2>; clock-names = "core", "clkin0", "clkin1"; + resets = <&reset RESET_SD_EMMC_C>; }; &spicc { @@ -749,12 +751,12 @@ }; &uart_AO { - clocks = <&xtal>, <&clkc CLKID_CLK81>, <&xtal>; + clocks = <&xtal>, <&clkc_AO CLKID_AO_UART1>, <&xtal>; clock-names = "xtal", "pclk", "baud"; }; &uart_AO_B { - clocks = <&xtal>, <&clkc CLKID_CLK81>, <&xtal>; + clocks = <&xtal>, <&clkc_AO CLKID_AO_UART2>, <&xtal>; clock-names = "xtal", "pclk", "baud"; }; diff --git a/arch/arm64/boot/dts/amlogic/meson-gxl.dtsi b/arch/arm64/boot/dts/amlogic/meson-gxl.dtsi index dba365ed4bd5..27538eea547b 100644 --- a/arch/arm64/boot/dts/amlogic/meson-gxl.dtsi +++ b/arch/arm64/boot/dts/amlogic/meson-gxl.dtsi @@ -267,11 +267,10 @@ clock-names = "isfr", "iahb", "venci"; }; -&hiubus { - clkc: clock-controller@0 { - compatible = "amlogic,gxl-clkc", "amlogic,gxbb-clkc"; +&sysctrl { + clkc: clock-controller { + compatible = "amlogic,gxl-clkc"; #clock-cells = <1>; - reg = <0x0 0x0 0x0 0x3db>; }; }; @@ -725,13 +724,15 @@ <&clkc CLKID_SD_EMMC_A_CLK0>, <&clkc CLKID_FCLK_DIV2>; clock-names = "core", "clkin0", "clkin1"; + resets = <&reset RESET_SD_EMMC_A>; }; &sd_emmc_b { clocks = <&clkc CLKID_SD_EMMC_B>, <&clkc CLKID_SD_EMMC_B_CLK0>, <&clkc CLKID_FCLK_DIV2>; - clock-names = "core", "clkin0", "clkin1"; + clock-names = "core", "clkin0", "clkin1"; + resets = <&reset RESET_SD_EMMC_B>; }; &sd_emmc_c { @@ -739,6 +740,7 @@ <&clkc CLKID_SD_EMMC_C_CLK0>, <&clkc CLKID_FCLK_DIV2>; clock-names = "core", "clkin0", "clkin1"; + resets = <&reset RESET_SD_EMMC_C>; }; &spicc { @@ -758,12 +760,12 @@ }; &uart_AO { - clocks = <&xtal>, <&clkc CLKID_CLK81>, <&xtal>; + clocks = <&xtal>, <&clkc_AO CLKID_AO_UART1>, <&xtal>; clock-names = "xtal", "pclk", "baud"; }; &uart_AO_B { - clocks = <&xtal>, <&clkc CLKID_CLK81>, <&xtal>; + clocks = <&xtal>, <&clkc_AO CLKID_AO_UART2>, <&xtal>; clock-names = "xtal", "pclk", "baud"; }; diff --git a/arch/arm64/boot/dts/arm/juno-base.dtsi b/arch/arm64/boot/dts/arm/juno-base.dtsi index eb749c50a736..ce56a4acda4f 100644 --- a/arch/arm64/boot/dts/arm/juno-base.dtsi +++ b/arch/arm64/boot/dts/arm/juno-base.dtsi @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #include "juno-clocks.dtsi" +#include "juno-motherboard.dtsi" / { /* @@ -572,14 +573,14 @@ thermal-sensors = <&scpi_sensors0 3>; }; - big_cluster_thermal_zone: big_cluster { + big_cluster_thermal_zone: big-cluster { polling-delay = <1000>; polling-delay-passive = <100>; thermal-sensors = <&scpi_sensors0 21>; status = "disabled"; }; - little_cluster_thermal_zone: little_cluster { + little_cluster_thermal_zone: little-cluster { polling-delay = <1000>; polling-delay-passive = <100>; thermal-sensors = <&scpi_sensors0 22>; @@ -677,7 +678,7 @@ clock-names = "pxlclk"; port { - hdlcd1_output: hdlcd1-endpoint { + hdlcd1_output: endpoint { remote-endpoint = <&tda998x_1_input>; }; }; @@ -692,7 +693,7 @@ clock-names = "pxlclk"; port { - hdlcd0_output: hdlcd0-endpoint { + hdlcd0_output: endpoint { remote-endpoint = <&tda998x_0_input>; }; }; @@ -720,7 +721,7 @@ compatible = "nxp,tda998x"; reg = <0x70>; port { - tda998x_0_input: tda998x-0-endpoint { + tda998x_0_input: endpoint { remote-endpoint = <&hdlcd0_output>; }; }; @@ -730,7 +731,7 @@ compatible = "nxp,tda998x"; reg = <0x71>; port { - tda998x_1_input: tda998x-1-endpoint { + tda998x_1_input: endpoint { remote-endpoint = <&hdlcd1_output>; }; }; @@ -795,8 +796,6 @@ <0 0 10 &gic 0 0 0 167 IRQ_TYPE_LEVEL_HIGH>, <0 0 11 &gic 0 0 0 168 IRQ_TYPE_LEVEL_HIGH>, <0 0 12 &gic 0 0 0 169 IRQ_TYPE_LEVEL_HIGH>; - - /include/ "juno-motherboard.dtsi" }; site2: tlx@60000000 { diff --git a/arch/arm64/boot/dts/arm/juno-motherboard.dtsi b/arch/arm64/boot/dts/arm/juno-motherboard.dtsi index 69804c5f1197..1792b074e9a3 100644 --- a/arch/arm64/boot/dts/arm/juno-motherboard.dtsi +++ b/arch/arm64/boot/dts/arm/juno-motherboard.dtsi @@ -7,6 +7,8 @@ * */ +/ { + smb@8000000 { mb_clk24mhz: clk24mhz { compatible = "fixed-clock"; #clock-cells = <0>; @@ -54,46 +56,46 @@ regulator-always-on; }; - gpio_keys { + gpio-keys { compatible = "gpio-keys"; power-button { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = <116>; label = "POWER"; gpios = <&iofpga_gpio0 0 0x4>; }; home-button { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = <102>; label = "HOME"; gpios = <&iofpga_gpio0 1 0x4>; }; rlock-button { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = <152>; label = "RLOCK"; gpios = <&iofpga_gpio0 2 0x4>; }; vol-up-button { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = <115>; label = "VOL+"; gpios = <&iofpga_gpio0 3 0x4>; }; vol-down-button { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = <114>; label = "VOL-"; gpios = <&iofpga_gpio0 4 0x4>; }; nmi-button { - debounce_interval = <50>; + debounce-interval = <50>; wakeup-source; linux,code = <99>; label = "NMI"; @@ -287,3 +289,5 @@ }; }; }; + }; +}; diff --git a/arch/arm64/boot/dts/arm/juno-r1.dts b/arch/arm64/boot/dts/arm/juno-r1.dts index aed6389468c4..2c5db03f226c 100644 --- a/arch/arm64/boot/dts/arm/juno-r1.dts +++ b/arch/arm64/boot/dts/arm/juno-r1.dts @@ -201,7 +201,7 @@ }; }; - pmu_a57 { + pmu-a57 { compatible = "arm,cortex-a57-pmu"; interrupts = , ; @@ -209,7 +209,7 @@ <&A57_1>; }; - pmu_a53 { + pmu-a53 { compatible = "arm,cortex-a53-pmu"; interrupts = , , @@ -278,6 +278,10 @@ remote-endpoint = <&csys2_funnel_out_port>; }; +&csys1_funnel_in_port0 { + remote-endpoint = <&stm_out_port>; +}; + &stm_out_port { remote-endpoint = <&csys1_funnel_in_port0>; }; diff --git a/arch/arm64/boot/dts/arm/juno-r2.dts b/arch/arm64/boot/dts/arm/juno-r2.dts index b39b6d6ec5aa..c51950f4a1b6 100644 --- a/arch/arm64/boot/dts/arm/juno-r2.dts +++ b/arch/arm64/boot/dts/arm/juno-r2.dts @@ -201,7 +201,7 @@ }; }; - pmu_a72 { + pmu-a72 { compatible = "arm,cortex-a72-pmu"; interrupts = , ; @@ -209,7 +209,7 @@ <&A72_1>; }; - pmu_a53 { + pmu-a53 { compatible = "arm,cortex-a53-pmu"; interrupts = , , @@ -278,6 +278,10 @@ remote-endpoint = <&csys2_funnel_out_port>; }; +&csys1_funnel_in_port0 { + remote-endpoint = <&stm_out_port>; +}; + &stm_out_port { remote-endpoint = <&csys1_funnel_in_port0>; }; diff --git a/arch/arm64/boot/dts/arm/juno.dts b/arch/arm64/boot/dts/arm/juno.dts index c9236c4b967d..2b2bf39c30ef 100644 --- a/arch/arm64/boot/dts/arm/juno.dts +++ b/arch/arm64/boot/dts/arm/juno.dts @@ -200,7 +200,7 @@ }; }; - pmu_a57 { + pmu-a57 { compatible = "arm,cortex-a57-pmu"; interrupts = , ; @@ -208,7 +208,7 @@ <&A57_1>; }; - pmu_a53 { + pmu-a53 { compatible = "arm,cortex-a53-pmu"; interrupts = , , diff --git a/arch/arm64/boot/dts/arm/rtsm_ve-aemv8a.dts b/arch/arm64/boot/dts/arm/rtsm_ve-aemv8a.dts index 06c8117e812a..602f63f72c37 100644 --- a/arch/arm64/boot/dts/arm/rtsm_ve-aemv8a.dts +++ b/arch/arm64/boot/dts/arm/rtsm_ve-aemv8a.dts @@ -12,6 +12,8 @@ /memreserve/ 0x80000000 0x00010000; +#include "rtsm_ve-motherboard.dtsi" + / { model = "RTSM_VE_AEMv8A"; compatible = "arm,rtsm_ve,aemv8a", "arm,vexpress"; @@ -162,7 +164,5 @@ <0 0 40 &gic 0 40 4>, <0 0 41 &gic 0 41 4>, <0 0 42 &gic 0 42 4>; - - /include/ "rtsm_ve-motherboard.dtsi" }; }; diff --git a/arch/arm64/boot/dts/arm/rtsm_ve-motherboard.dtsi b/arch/arm64/boot/dts/arm/rtsm_ve-motherboard.dtsi index 1134e5d8df18..d2dbc3f39263 100644 --- a/arch/arm64/boot/dts/arm/rtsm_ve-motherboard.dtsi +++ b/arch/arm64/boot/dts/arm/rtsm_ve-motherboard.dtsi @@ -7,270 +7,273 @@ * * VEMotherBoard.lisa */ - - motherboard { - arm,v2m-memory-map = "rs1"; - compatible = "arm,vexpress,v2m-p1", "simple-bus"; - #address-cells = <2>; /* SMB chipselect number and offset */ - #size-cells = <1>; - #interrupt-cells = <1>; - ranges; - - flash@0,00000000 { - compatible = "arm,vexpress-flash", "cfi-flash"; - reg = <0 0x00000000 0x04000000>, - <4 0x00000000 0x04000000>; - bank-width = <4>; - }; - - v2m_video_ram: vram@2,00000000 { - compatible = "arm,vexpress-vram"; - reg = <2 0x00000000 0x00800000>; - }; - - ethernet@2,02000000 { - compatible = "smsc,lan91c111"; - reg = <2 0x02000000 0x10000>; - interrupts = <15>; - }; - - v2m_clk24mhz: clk24mhz { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <24000000>; - clock-output-names = "v2m:clk24mhz"; - }; - - v2m_refclk1mhz: refclk1mhz { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <1000000>; - clock-output-names = "v2m:refclk1mhz"; - }; - - v2m_refclk32khz: refclk32khz { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <32768>; - clock-output-names = "v2m:refclk32khz"; - }; - - iofpga@3,00000000 { - compatible = "simple-bus"; - #address-cells = <1>; +/ { + smb@8000000 { + motherboard { + arm,v2m-memory-map = "rs1"; + compatible = "arm,vexpress,v2m-p1", "simple-bus"; + #address-cells = <2>; /* SMB chipselect number and offset */ #size-cells = <1>; - ranges = <0 3 0 0x200000>; - - v2m_sysreg: sysreg@10000 { - compatible = "arm,vexpress-sysreg"; - reg = <0x010000 0x1000>; - gpio-controller; - #gpio-cells = <2>; + #interrupt-cells = <1>; + ranges; + + flash@0,00000000 { + compatible = "arm,vexpress-flash", "cfi-flash"; + reg = <0 0x00000000 0x04000000>, + <4 0x00000000 0x04000000>; + bank-width = <4>; }; - v2m_sysctl: sysctl@20000 { - compatible = "arm,sp810", "arm,primecell"; - reg = <0x020000 0x1000>; - clocks = <&v2m_refclk32khz>, <&v2m_refclk1mhz>, <&v2m_clk24mhz>; - clock-names = "refclk", "timclk", "apb_pclk"; - #clock-cells = <1>; - clock-output-names = "timerclken0", "timerclken1", "timerclken2", "timerclken3"; - assigned-clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&v2m_sysctl 3>, <&v2m_sysctl 3>; - assigned-clock-parents = <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>; + v2m_video_ram: vram@2,00000000 { + compatible = "arm,vexpress-vram"; + reg = <2 0x00000000 0x00800000>; }; - aaci@40000 { - compatible = "arm,pl041", "arm,primecell"; - reg = <0x040000 0x1000>; - interrupts = <11>; - clocks = <&v2m_clk24mhz>; - clock-names = "apb_pclk"; + ethernet@2,02000000 { + compatible = "smsc,lan91c111"; + reg = <2 0x02000000 0x10000>; + interrupts = <15>; }; - mmci@50000 { - compatible = "arm,pl180", "arm,primecell"; - reg = <0x050000 0x1000>; - interrupts = <9 10>; - cd-gpios = <&v2m_sysreg 0 0>; - wp-gpios = <&v2m_sysreg 1 0>; - max-frequency = <12000000>; - vmmc-supply = <&v2m_fixed_3v3>; - clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; - clock-names = "mclk", "apb_pclk"; + v2m_clk24mhz: clk24mhz { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <24000000>; + clock-output-names = "v2m:clk24mhz"; }; - kmi@60000 { - compatible = "arm,pl050", "arm,primecell"; - reg = <0x060000 0x1000>; - interrupts = <12>; - clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; - clock-names = "KMIREFCLK", "apb_pclk"; + v2m_refclk1mhz: refclk1mhz { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <1000000>; + clock-output-names = "v2m:refclk1mhz"; }; - kmi@70000 { - compatible = "arm,pl050", "arm,primecell"; - reg = <0x070000 0x1000>; - interrupts = <13>; - clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; - clock-names = "KMIREFCLK", "apb_pclk"; + v2m_refclk32khz: refclk32khz { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <32768>; + clock-output-names = "v2m:refclk32khz"; }; - v2m_serial0: uart@90000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x090000 0x1000>; - interrupts = <5>; - clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; - clock-names = "uartclk", "apb_pclk"; - }; + iofpga@3,00000000 { + compatible = "simple-bus"; + #address-cells = <1>; + #size-cells = <1>; + ranges = <0 3 0 0x200000>; + + v2m_sysreg: sysreg@10000 { + compatible = "arm,vexpress-sysreg"; + reg = <0x010000 0x1000>; + gpio-controller; + #gpio-cells = <2>; + }; - v2m_serial1: uart@a0000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x0a0000 0x1000>; - interrupts = <6>; - clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; - clock-names = "uartclk", "apb_pclk"; - }; + v2m_sysctl: sysctl@20000 { + compatible = "arm,sp810", "arm,primecell"; + reg = <0x020000 0x1000>; + clocks = <&v2m_refclk32khz>, <&v2m_refclk1mhz>, <&v2m_clk24mhz>; + clock-names = "refclk", "timclk", "apb_pclk"; + #clock-cells = <1>; + clock-output-names = "timerclken0", "timerclken1", "timerclken2", "timerclken3"; + assigned-clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&v2m_sysctl 3>, <&v2m_sysctl 3>; + assigned-clock-parents = <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>, <&v2m_refclk1mhz>; + }; - v2m_serial2: uart@b0000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x0b0000 0x1000>; - interrupts = <7>; - clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; - clock-names = "uartclk", "apb_pclk"; - }; + aaci@40000 { + compatible = "arm,pl041", "arm,primecell"; + reg = <0x040000 0x1000>; + interrupts = <11>; + clocks = <&v2m_clk24mhz>; + clock-names = "apb_pclk"; + }; - v2m_serial3: uart@c0000 { - compatible = "arm,pl011", "arm,primecell"; - reg = <0x0c0000 0x1000>; - interrupts = <8>; - clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; - clock-names = "uartclk", "apb_pclk"; - }; + mmci@50000 { + compatible = "arm,pl180", "arm,primecell"; + reg = <0x050000 0x1000>; + interrupts = <9 10>; + cd-gpios = <&v2m_sysreg 0 0>; + wp-gpios = <&v2m_sysreg 1 0>; + max-frequency = <12000000>; + vmmc-supply = <&v2m_fixed_3v3>; + clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; + clock-names = "mclk", "apb_pclk"; + }; - wdt@f0000 { - compatible = "arm,sp805", "arm,primecell"; - reg = <0x0f0000 0x1000>; - interrupts = <0>; - clocks = <&v2m_refclk32khz>, <&v2m_clk24mhz>; - clock-names = "wdogclk", "apb_pclk"; - }; + kmi@60000 { + compatible = "arm,pl050", "arm,primecell"; + reg = <0x060000 0x1000>; + interrupts = <12>; + clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; + clock-names = "KMIREFCLK", "apb_pclk"; + }; - v2m_timer01: timer@110000 { - compatible = "arm,sp804", "arm,primecell"; - reg = <0x110000 0x1000>; - interrupts = <2>; - clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&v2m_clk24mhz>; - clock-names = "timclken1", "timclken2", "apb_pclk"; - }; + kmi@70000 { + compatible = "arm,pl050", "arm,primecell"; + reg = <0x070000 0x1000>; + interrupts = <13>; + clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; + clock-names = "KMIREFCLK", "apb_pclk"; + }; - v2m_timer23: timer@120000 { - compatible = "arm,sp804", "arm,primecell"; - reg = <0x120000 0x1000>; - interrupts = <3>; - clocks = <&v2m_sysctl 2>, <&v2m_sysctl 3>, <&v2m_clk24mhz>; - clock-names = "timclken1", "timclken2", "apb_pclk"; - }; + v2m_serial0: uart@90000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x090000 0x1000>; + interrupts = <5>; + clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; + clock-names = "uartclk", "apb_pclk"; + }; - rtc@170000 { - compatible = "arm,pl031", "arm,primecell"; - reg = <0x170000 0x1000>; - interrupts = <4>; - clocks = <&v2m_clk24mhz>; - clock-names = "apb_pclk"; - }; + v2m_serial1: uart@a0000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x0a0000 0x1000>; + interrupts = <6>; + clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; + clock-names = "uartclk", "apb_pclk"; + }; - clcd@1f0000 { - compatible = "arm,pl111", "arm,primecell"; - reg = <0x1f0000 0x1000>; - interrupt-names = "combined"; - interrupts = <14>; - clocks = <&v2m_oscclk1>, <&v2m_clk24mhz>; - clock-names = "clcdclk", "apb_pclk"; - arm,pl11x,framebuffer = <0x18000000 0x00180000>; - memory-region = <&v2m_video_ram>; - max-memory-bandwidth = <130000000>; /* 16bpp @ 63.5MHz */ - - port { - v2m_clcd_pads: endpoint { - remote-endpoint = <&v2m_clcd_panel>; - arm,pl11x,tft-r0g0b0-pads = <0 8 16>; - }; + v2m_serial2: uart@b0000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x0b0000 0x1000>; + interrupts = <7>; + clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; + clock-names = "uartclk", "apb_pclk"; }; - panel { - compatible = "panel-dpi"; + v2m_serial3: uart@c0000 { + compatible = "arm,pl011", "arm,primecell"; + reg = <0x0c0000 0x1000>; + interrupts = <8>; + clocks = <&v2m_clk24mhz>, <&v2m_clk24mhz>; + clock-names = "uartclk", "apb_pclk"; + }; + + wdt@f0000 { + compatible = "arm,sp805", "arm,primecell"; + reg = <0x0f0000 0x1000>; + interrupts = <0>; + clocks = <&v2m_refclk32khz>, <&v2m_clk24mhz>; + clock-names = "wdogclk", "apb_pclk"; + }; + + v2m_timer01: timer@110000 { + compatible = "arm,sp804", "arm,primecell"; + reg = <0x110000 0x1000>; + interrupts = <2>; + clocks = <&v2m_sysctl 0>, <&v2m_sysctl 1>, <&v2m_clk24mhz>; + clock-names = "timclken1", "timclken2", "apb_pclk"; + }; + + v2m_timer23: timer@120000 { + compatible = "arm,sp804", "arm,primecell"; + reg = <0x120000 0x1000>; + interrupts = <3>; + clocks = <&v2m_sysctl 2>, <&v2m_sysctl 3>, <&v2m_clk24mhz>; + clock-names = "timclken1", "timclken2", "apb_pclk"; + }; + + rtc@170000 { + compatible = "arm,pl031", "arm,primecell"; + reg = <0x170000 0x1000>; + interrupts = <4>; + clocks = <&v2m_clk24mhz>; + clock-names = "apb_pclk"; + }; + + clcd@1f0000 { + compatible = "arm,pl111", "arm,primecell"; + reg = <0x1f0000 0x1000>; + interrupt-names = "combined"; + interrupts = <14>; + clocks = <&v2m_oscclk1>, <&v2m_clk24mhz>; + clock-names = "clcdclk", "apb_pclk"; + arm,pl11x,framebuffer = <0x18000000 0x00180000>; + memory-region = <&v2m_video_ram>; + max-memory-bandwidth = <130000000>; /* 16bpp @ 63.5MHz */ port { - v2m_clcd_panel: endpoint { - remote-endpoint = <&v2m_clcd_pads>; + v2m_clcd_pads: endpoint { + remote-endpoint = <&v2m_clcd_panel>; + arm,pl11x,tft-r0g0b0-pads = <0 8 16>; }; }; - panel-timing { - clock-frequency = <63500127>; - hactive = <1024>; - hback-porch = <152>; - hfront-porch = <48>; - hsync-len = <104>; - vactive = <768>; - vback-porch = <23>; - vfront-porch = <3>; - vsync-len = <4>; + panel { + compatible = "panel-dpi"; + + port { + v2m_clcd_panel: endpoint { + remote-endpoint = <&v2m_clcd_pads>; + }; + }; + + panel-timing { + clock-frequency = <63500127>; + hactive = <1024>; + hback-porch = <152>; + hfront-porch = <48>; + hsync-len = <104>; + vactive = <768>; + vback-porch = <23>; + vfront-porch = <3>; + vsync-len = <4>; + }; }; }; - }; - virtio-block@130000 { - compatible = "virtio,mmio"; - reg = <0x130000 0x200>; - interrupts = <42>; + virtio-block@130000 { + compatible = "virtio,mmio"; + reg = <0x130000 0x200>; + interrupts = <42>; + }; }; - }; - - v2m_fixed_3v3: v2m-3v3 { - compatible = "regulator-fixed"; - regulator-name = "3V3"; - regulator-min-microvolt = <3300000>; - regulator-max-microvolt = <3300000>; - regulator-always-on; - }; - - mcc { - compatible = "arm,vexpress,config-bus"; - arm,vexpress,config-bridge = <&v2m_sysreg>; - v2m_oscclk1: oscclk1 { - /* CLCD clock */ - compatible = "arm,vexpress-osc"; - arm,vexpress-sysreg,func = <1 1>; - freq-range = <23750000 63500000>; - #clock-cells = <0>; - clock-output-names = "v2m:oscclk1"; + v2m_fixed_3v3: v2m-3v3 { + compatible = "regulator-fixed"; + regulator-name = "3V3"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-always-on; }; - reset { - compatible = "arm,vexpress-reset"; - arm,vexpress-sysreg,func = <5 0>; - }; + mcc { + compatible = "arm,vexpress,config-bus"; + arm,vexpress,config-bridge = <&v2m_sysreg>; + + v2m_oscclk1: oscclk1 { + /* CLCD clock */ + compatible = "arm,vexpress-osc"; + arm,vexpress-sysreg,func = <1 1>; + freq-range = <23750000 63500000>; + #clock-cells = <0>; + clock-output-names = "v2m:oscclk1"; + }; - muxfpga { - compatible = "arm,vexpress-muxfpga"; - arm,vexpress-sysreg,func = <7 0>; - }; + reset { + compatible = "arm,vexpress-reset"; + arm,vexpress-sysreg,func = <5 0>; + }; - shutdown { - compatible = "arm,vexpress-shutdown"; - arm,vexpress-sysreg,func = <8 0>; - }; + muxfpga { + compatible = "arm,vexpress-muxfpga"; + arm,vexpress-sysreg,func = <7 0>; + }; - reboot { - compatible = "arm,vexpress-reboot"; - arm,vexpress-sysreg,func = <9 0>; - }; + shutdown { + compatible = "arm,vexpress-shutdown"; + arm,vexpress-sysreg,func = <8 0>; + }; + + reboot { + compatible = "arm,vexpress-reboot"; + arm,vexpress-sysreg,func = <9 0>; + }; - dvimode { - compatible = "arm,vexpress-dvimode"; - arm,vexpress-sysreg,func = <11 0>; + dvimode { + compatible = "arm,vexpress-dvimode"; + arm,vexpress-sysreg,func = <11 0>; + }; }; }; }; +}; diff --git a/arch/arm64/boot/dts/arm/vexpress-v2f-1xv7-ca53x2.dts b/arch/arm64/boot/dts/arm/vexpress-v2f-1xv7-ca53x2.dts index 1c9eadc2d71e..38880380e0fa 100644 --- a/arch/arm64/boot/dts/arm/vexpress-v2f-1xv7-ca53x2.dts +++ b/arch/arm64/boot/dts/arm/vexpress-v2f-1xv7-ca53x2.dts @@ -13,6 +13,7 @@ /dts-v1/; #include +#include "vexpress-v2m-rs1.dtsi" / { model = "V2F-1XV7 Cortex-A53x2 SMM"; @@ -129,7 +130,7 @@ }; }; - smb@8000000 { + smb: smb@8000000 { compatible = "simple-bus"; #address-cells = <2>; @@ -186,7 +187,5 @@ <0 0 40 &gic GIC_SPI 40 IRQ_TYPE_LEVEL_HIGH>, <0 0 41 &gic GIC_SPI 41 IRQ_TYPE_LEVEL_HIGH>, <0 0 42 &gic GIC_SPI 42 IRQ_TYPE_LEVEL_HIGH>; - - /include/ "vexpress-v2m-rs1.dtsi" }; }; diff --git a/arch/arm64/boot/dts/broadcom/Makefile b/arch/arm64/boot/dts/broadcom/Makefile index 2a2591ef1fee..1193a9e34bbb 100644 --- a/arch/arm64/boot/dts/broadcom/Makefile +++ b/arch/arm64/boot/dts/broadcom/Makefile @@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 -dtb-$(CONFIG_ARCH_BCM2835) += bcm2837-rpi-3-b.dtb +dtb-$(CONFIG_ARCH_BCM2835) += bcm2837-rpi-3-b.dtb \ + bcm2837-rpi-3-b-plus.dtb subdir-y += northstar2 subdir-y += stingray diff --git a/arch/arm64/boot/dts/broadcom/bcm2837-rpi-3-b-plus.dts b/arch/arm64/boot/dts/broadcom/bcm2837-rpi-3-b-plus.dts new file mode 100644 index 000000000000..46ad2023cccf --- /dev/null +++ b/arch/arm64/boot/dts/broadcom/bcm2837-rpi-3-b-plus.dts @@ -0,0 +1,2 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "arm/bcm2837-rpi-3-b-plus.dts" diff --git a/arch/arm64/boot/dts/exynos/exynos5433.dtsi b/arch/arm64/boot/dts/exynos/exynos5433.dtsi index 1ad8677f6a0a..038c99792ccb 100644 --- a/arch/arm64/boot/dts/exynos/exynos5433.dtsi +++ b/arch/arm64/boot/dts/exynos/exynos5433.dtsi @@ -18,8 +18,8 @@ / { compatible = "samsung,exynos5433"; - #address-cells = <2>; - #size-cells = <2>; + #address-cells = <1>; + #size-cells = <1>; interrupt-parent = <&gic>; @@ -231,18 +231,11 @@ cpu_on = <0xC4000003>; }; - reboot: syscon-reboot { - compatible = "syscon-reboot"; - regmap = <&pmu_system_controller>; - offset = <0x400>; /* SWRESET */ - mask = <0x1>; - }; - soc: soc { compatible = "simple-bus"; #address-cells = <1>; #size-cells = <1>; - ranges = <0x0 0x0 0x0 0x18000000>; + ranges; arm_a53_pmu { compatible = "arm,cortex-a53-pmu", "arm,armv8-pmuv3"; @@ -799,6 +792,13 @@ #clock-cells = <1>; clock-names = "clkout16"; clocks = <&xxti>; + + reboot: syscon-reboot { + compatible = "syscon-reboot"; + regmap = <&pmu_system_controller>; + offset = <0x400>; /* SWRESET */ + mask = <0x1>; + }; }; gic: interrupt-controller@11001000 { @@ -829,11 +829,16 @@ <&cmu_disp CLK_ACLK_SMMU_DECON0X>, <&cmu_disp CLK_ACLK_XIU_DECON0X>, <&cmu_disp CLK_PCLK_SMMU_DECON0X>, + <&cmu_disp CLK_ACLK_SMMU_DECON1X>, + <&cmu_disp CLK_ACLK_XIU_DECON1X>, + <&cmu_disp CLK_PCLK_SMMU_DECON1X>, <&cmu_disp CLK_SCLK_DECON_VCLK>, <&cmu_disp CLK_SCLK_DECON_ECLK>; clock-names = "pclk", "aclk_decon", "aclk_smmu_decon0x", "aclk_xiu_decon0x", "pclk_smmu_decon0x", - "sclk_decon_vclk", "sclk_decon_eclk"; + "aclk_smmu_decon1x", "aclk_xiu_decon1x", + "pclk_smmu_decon1x", "sclk_decon_vclk", + "sclk_decon_eclk"; power-domains = <&pd_disp>; interrupt-names = "fifo", "vsync", "lcd_sys"; interrupts = , @@ -866,11 +871,16 @@ <&cmu_disp CLK_ACLK_SMMU_TV0X>, <&cmu_disp CLK_ACLK_XIU_TV0X>, <&cmu_disp CLK_PCLK_SMMU_TV0X>, + <&cmu_disp CLK_ACLK_SMMU_TV1X>, + <&cmu_disp CLK_ACLK_XIU_TV1X>, + <&cmu_disp CLK_PCLK_SMMU_TV1X>, <&cmu_disp CLK_SCLK_DECON_TV_VCLK>, <&cmu_disp CLK_SCLK_DECON_TV_ECLK>; clock-names = "pclk", "aclk_decon", "aclk_smmu_decon0x", "aclk_xiu_decon0x", "pclk_smmu_decon0x", - "sclk_decon_vclk", "sclk_decon_eclk"; + "aclk_smmu_decon1x", "aclk_xiu_decon1x", + "pclk_smmu_decon1x", "sclk_decon_vclk", + "sclk_decon_eclk"; samsung,disp-sysreg = <&syscon_disp>; power-domains = <&pd_disp>; interrupt-names = "fifo", "vsync", "lcd_sys"; @@ -1034,6 +1044,30 @@ power-domains = <&pd_gscl>; }; + scaler_0: scaler@15000000 { + compatible = "samsung,exynos5433-scaler"; + reg = <0x15000000 0x1294>; + interrupts = <0 402 IRQ_TYPE_LEVEL_HIGH>; + clock-names = "pclk", "aclk", "aclk_xiu"; + clocks = <&cmu_mscl CLK_PCLK_M2MSCALER0>, + <&cmu_mscl CLK_ACLK_M2MSCALER0>, + <&cmu_mscl CLK_ACLK_XIU_MSCLX>; + iommus = <&sysmmu_scaler_0>; + power-domains = <&pd_mscl>; + }; + + scaler_1: scaler@15010000 { + compatible = "samsung,exynos5433-scaler"; + reg = <0x15010000 0x1294>; + interrupts = <0 403 IRQ_TYPE_LEVEL_HIGH>; + clock-names = "pclk", "aclk", "aclk_xiu"; + clocks = <&cmu_mscl CLK_PCLK_M2MSCALER1>, + <&cmu_mscl CLK_ACLK_M2MSCALER1>, + <&cmu_mscl CLK_ACLK_XIU_MSCLX>; + iommus = <&sysmmu_scaler_1>; + power-domains = <&pd_mscl>; + }; + jpeg: codec@15020000 { compatible = "samsung,exynos5433-jpeg"; reg = <0x15020000 0x10000>; @@ -1137,6 +1171,28 @@ power-domains = <&pd_gscl>; }; + sysmmu_scaler_0: sysmmu@0x15040000 { + compatible = "samsung,exynos-sysmmu"; + reg = <0x15040000 0x1000>; + interrupts = ; + clock-names = "pclk", "aclk"; + clocks = <&cmu_mscl CLK_PCLK_SMMU_M2MSCALER0>, + <&cmu_mscl CLK_ACLK_SMMU_M2MSCALER0>; + #iommu-cells = <0>; + power-domains = <&pd_mscl>; + }; + + sysmmu_scaler_1: sysmmu@0x15050000 { + compatible = "samsung,exynos-sysmmu"; + reg = <0x15050000 0x1000>; + interrupts = ; + clock-names = "pclk", "aclk"; + clocks = <&cmu_mscl CLK_PCLK_SMMU_M2MSCALER1>, + <&cmu_mscl CLK_ACLK_SMMU_M2MSCALER1>; + #iommu-cells = <0>; + power-domains = <&pd_mscl>; + }; + sysmmu_jpeg: sysmmu@15060000 { compatible = "samsung,exynos-sysmmu"; reg = <0x15060000 0x1000>; diff --git a/arch/arm64/boot/dts/exynos/exynos7.dtsi b/arch/arm64/boot/dts/exynos/exynos7.dtsi index ad9dce6894ce..93a84338938a 100644 --- a/arch/arm64/boot/dts/exynos/exynos7.dtsi +++ b/arch/arm64/boot/dts/exynos/exynos7.dtsi @@ -12,8 +12,8 @@ / { compatible = "samsung,exynos7"; interrupt-parent = <&gic>; - #address-cells = <2>; - #size-cells = <2>; + #address-cells = <1>; + #size-cells = <1>; aliases { pinctrl0 = &pinctrl_alive; @@ -70,7 +70,7 @@ compatible = "simple-bus"; #address-cells = <1>; #size-cells = <1>; - ranges = <0 0 0 0x18000000>; + ranges; chipid@10000000 { compatible = "samsung,exynos4210-chipid"; @@ -494,13 +494,13 @@ pmu_system_controller: system-controller@105c0000 { compatible = "samsung,exynos7-pmu", "syscon"; reg = <0x105c0000 0x5000>; - }; - reboot: syscon-reboot { - compatible = "syscon-reboot"; - regmap = <&pmu_system_controller>; - offset = <0x0400>; - mask = <0x1>; + reboot: syscon-reboot { + compatible = "syscon-reboot"; + regmap = <&pmu_system_controller>; + offset = <0x0400>; + mask = <0x1>; + }; }; rtc: rtc@10590000 { diff --git a/arch/arm64/boot/dts/freescale/fsl-ls1012a.dtsi b/arch/arm64/boot/dts/freescale/fsl-ls1012a.dtsi index bb788eddf9f4..205f0f4c5df0 100644 --- a/arch/arm64/boot/dts/freescale/fsl-ls1012a.dtsi +++ b/arch/arm64/boot/dts/freescale/fsl-ls1012a.dtsi @@ -53,11 +53,11 @@ aliases { crypto = &crypto; - rtic_a = &rtic_a; - rtic_b = &rtic_b; - rtic_c = &rtic_c; - rtic_d = &rtic_d; - sec_mon = &sec_mon; + rtic-a = &rtic_a; + rtic-b = &rtic_b; + rtic-c = &rtic_c; + rtic-d = &rtic_d; + sec-mon = &sec_mon; }; cpus { diff --git a/arch/arm64/boot/dts/freescale/fsl-ls208xa-rdb.dtsi b/arch/arm64/boot/dts/freescale/fsl-ls208xa-rdb.dtsi index 5498c705ae6a..061647bd97b8 100644 --- a/arch/arm64/boot/dts/freescale/fsl-ls208xa-rdb.dtsi +++ b/arch/arm64/boot/dts/freescale/fsl-ls208xa-rdb.dtsi @@ -134,7 +134,7 @@ &dspi { status = "okay"; - dflash0: n25q512a { + dflash0: n25q512a@0 { #address-cells = <1>; #size-cells = <1>; compatible = "st,m25p80"; diff --git a/arch/arm64/boot/dts/hisilicon/hi3660.dtsi b/arch/arm64/boot/dts/hisilicon/hi3660.dtsi index ec3eb8e33a3a..8d477dcbfa58 100644 --- a/arch/arm64/boot/dts/hisilicon/hi3660.dtsi +++ b/arch/arm64/boot/dts/hisilicon/hi3660.dtsi @@ -7,6 +7,7 @@ #include #include +#include / { compatible = "hisilicon,hi3660"; @@ -62,6 +63,10 @@ next-level-cache = <&A53_L2>; cpu-idle-states = <&CPU_SLEEP &CLUSTER_SLEEP_0>; capacity-dmips-mhz = <592>; + clocks = <&stub_clock HI3660_CLK_STUB_CLUSTER0>; + operating-points-v2 = <&cluster0_opp>; + #cooling-cells = <2>; + dynamic-power-coefficient = <110>; }; cpu1: cpu@1 { @@ -72,6 +77,8 @@ next-level-cache = <&A53_L2>; cpu-idle-states = <&CPU_SLEEP &CLUSTER_SLEEP_0>; capacity-dmips-mhz = <592>; + clocks = <&stub_clock HI3660_CLK_STUB_CLUSTER0>; + operating-points-v2 = <&cluster0_opp>; }; cpu2: cpu@2 { @@ -82,6 +89,8 @@ next-level-cache = <&A53_L2>; cpu-idle-states = <&CPU_SLEEP &CLUSTER_SLEEP_0>; capacity-dmips-mhz = <592>; + clocks = <&stub_clock HI3660_CLK_STUB_CLUSTER0>; + operating-points-v2 = <&cluster0_opp>; }; cpu3: cpu@3 { @@ -92,6 +101,8 @@ next-level-cache = <&A53_L2>; cpu-idle-states = <&CPU_SLEEP &CLUSTER_SLEEP_0>; capacity-dmips-mhz = <592>; + clocks = <&stub_clock HI3660_CLK_STUB_CLUSTER0>; + operating-points-v2 = <&cluster0_opp>; }; cpu4: cpu@100 { @@ -102,6 +113,10 @@ next-level-cache = <&A73_L2>; cpu-idle-states = <&CPU_SLEEP &CLUSTER_SLEEP_1>; capacity-dmips-mhz = <1024>; + clocks = <&stub_clock HI3660_CLK_STUB_CLUSTER1>; + operating-points-v2 = <&cluster1_opp>; + #cooling-cells = <2>; + dynamic-power-coefficient = <550>; }; cpu5: cpu@101 { @@ -112,6 +127,8 @@ next-level-cache = <&A73_L2>; cpu-idle-states = <&CPU_SLEEP &CLUSTER_SLEEP_1>; capacity-dmips-mhz = <1024>; + clocks = <&stub_clock HI3660_CLK_STUB_CLUSTER1>; + operating-points-v2 = <&cluster1_opp>; }; cpu6: cpu@102 { @@ -122,6 +139,8 @@ next-level-cache = <&A73_L2>; cpu-idle-states = <&CPU_SLEEP &CLUSTER_SLEEP_1>; capacity-dmips-mhz = <1024>; + clocks = <&stub_clock HI3660_CLK_STUB_CLUSTER1>; + operating-points-v2 = <&cluster1_opp>; }; cpu7: cpu@103 { @@ -132,6 +151,8 @@ next-level-cache = <&A73_L2>; cpu-idle-states = <&CPU_SLEEP &CLUSTER_SLEEP_1>; capacity-dmips-mhz = <1024>; + clocks = <&stub_clock HI3660_CLK_STUB_CLUSTER1>; + operating-points-v2 = <&cluster1_opp>; }; idle-states { @@ -174,6 +195,76 @@ }; }; + cluster0_opp: opp_table0 { + compatible = "operating-points-v2"; + opp-shared; + + opp00 { + opp-hz = /bits/ 64 <533000000>; + opp-microvolt = <700000>; + clock-latency-ns = <300000>; + }; + + opp01 { + opp-hz = /bits/ 64 <999000000>; + opp-microvolt = <800000>; + clock-latency-ns = <300000>; + }; + + opp02 { + opp-hz = /bits/ 64 <1402000000>; + opp-microvolt = <900000>; + clock-latency-ns = <300000>; + }; + + opp03 { + opp-hz = /bits/ 64 <1709000000>; + opp-microvolt = <1000000>; + clock-latency-ns = <300000>; + }; + + opp04 { + opp-hz = /bits/ 64 <1844000000>; + opp-microvolt = <1100000>; + clock-latency-ns = <300000>; + }; + }; + + cluster1_opp: opp_table1 { + compatible = "operating-points-v2"; + opp-shared; + + opp10 { + opp-hz = /bits/ 64 <903000000>; + opp-microvolt = <700000>; + clock-latency-ns = <300000>; + }; + + opp11 { + opp-hz = /bits/ 64 <1421000000>; + opp-microvolt = <800000>; + clock-latency-ns = <300000>; + }; + + opp12 { + opp-hz = /bits/ 64 <1805000000>; + opp-microvolt = <900000>; + clock-latency-ns = <300000>; + }; + + opp13 { + opp-hz = /bits/ 64 <2112000000>; + opp-microvolt = <1000000>; + clock-latency-ns = <300000>; + }; + + opp14 { + opp-hz = /bits/ 64 <2362000000>; + opp-microvolt = <1100000>; + clock-latency-ns = <300000>; + }; + }; + gic: interrupt-controller@e82b0000 { compatible = "arm,gic-400"; reg = <0x0 0xe82b1000 0 0x1000>, /* GICD */ @@ -274,6 +365,21 @@ #reset-cells = <2>; }; + mailbox: mailbox@e896b000 { + compatible = "hisilicon,hi3660-mbox"; + reg = <0x0 0xe896b000 0x0 0x1000>; + interrupts = , + ; + #mbox-cells = <3>; + }; + + stub_clock: stub_clock@e896b500 { + compatible = "hisilicon,hi3660-stub-clk"; + reg = <0x0 0xe896b500 0x0 0x0100>; + #clock-cells = <1>; + mboxes = <&mailbox 13 3 0>; + }; + dual_timer0: timer@fff14000 { compatible = "arm,sp804", "arm,primecell"; reg = <0x0 0xfff14000 0x0 0x1000>; @@ -872,6 +978,8 @@ 0x0 0x02000000>; num-lanes = <1>; #interrupt-cells = <1>; + interrupts = <0 283 4>; + interrupt-names = "msi"; interrupt-map-mask = <0xf800 0 0 7>; interrupt-map = <0x0 0 0 1 &gic GIC_SPI 282 IRQ_TYPE_LEVEL_HIGH>, @@ -972,5 +1080,44 @@ interrupts = ; #thermal-sensor-cells = <1>; }; + + thermal-zones { + + cls0: cls0 { + polling-delay = <1000>; + polling-delay-passive = <100>; + sustainable-power = <4500>; + + /* sensor ID */ + thermal-sensors = <&tsensor 1>; + + trips { + threshold: trip-point@0 { + temperature = <65000>; + hysteresis = <1000>; + type = "passive"; + }; + + target: trip-point@1 { + temperature = <75000>; + hysteresis = <1000>; + type = "passive"; + }; + }; + + cooling-maps { + map0 { + trip = <&target>; + contribution = <1024>; + cooling-device = <&cpu0 THERMAL_NO_LIMIT THERMAL_NO_LIMIT>; + }; + map1 { + trip = <&target>; + contribution = <512>; + cooling-device = <&cpu4 THERMAL_NO_LIMIT THERMAL_NO_LIMIT>; + }; + }; + }; + }; }; }; diff --git a/arch/arm64/boot/dts/hisilicon/hi3798cv200-poplar.dts b/arch/arm64/boot/dts/hisilicon/hi3798cv200-poplar.dts index 4d5d644abb12..d30f6eb8a5ee 100644 --- a/arch/arm64/boot/dts/hisilicon/hi3798cv200-poplar.dts +++ b/arch/arm64/boot/dts/hisilicon/hi3798cv200-poplar.dts @@ -11,6 +11,7 @@ #include #include "hi3798cv200.dtsi" +#include "poplar-pinctrl.dtsi" / { model = "HiSilicon Poplar Development Board"; @@ -61,6 +62,33 @@ default-state = "off"; }; }; + + reg_pcie: regulator-pcie { + compatible = "regulator-fixed"; + regulator-name = "3V3_PCIE0"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + gpio = <&gpio6 7 0>; + enable-active-high; + }; +}; + +&ehci { + status = "okay"; +}; + +&emmc { + pinctrl-names = "default"; + pinctrl-0 = <&emmc_pins_1 &emmc_pins_2 + &emmc_pins_3 &emmc_pins_4>; + fifo-depth = <256>; + clock-frequency = <200000000>; + cap-mmc-highspeed; + mmc-ddr-1_8v; + mmc-hs200-1_8v; + non-removable; + bus-width = <8>; + status = "okay"; }; &gmac1 { @@ -146,6 +174,16 @@ status = "okay"; }; +&ohci { + status = "okay"; +}; + +&pcie { + reset-gpios = <&gpio4 4 GPIO_ACTIVE_HIGH>; + vpcie-supply = <®_pcie>; + status = "okay"; +}; + &sd0 { bus-width = <4>; cap-sd-highspeed; diff --git a/arch/arm64/boot/dts/hisilicon/hi3798cv200.dtsi b/arch/arm64/boot/dts/hisilicon/hi3798cv200.dtsi index 962bd79139e4..7c0fddd7c8cf 100644 --- a/arch/arm64/boot/dts/hisilicon/hi3798cv200.dtsi +++ b/arch/arm64/boot/dts/hisilicon/hi3798cv200.dtsi @@ -8,7 +8,9 @@ */ #include +#include #include +#include #include / { @@ -106,6 +108,113 @@ #reset-cells = <2>; }; + perictrl: peripheral-controller@8a20000 { + compatible = "hisilicon,hi3798cv200-perictrl", "syscon", + "simple-mfd"; + reg = <0x8a20000 0x1000>; + #address-cells = <1>; + #size-cells = <1>; + ranges = <0x0 0x8a20000 0x1000>; + + usb2_phy1: usb2-phy@120 { + compatible = "hisilicon,hi3798cv200-usb2-phy"; + reg = <0x120 0x4>; + clocks = <&crg HISTB_USB2_PHY1_REF_CLK>; + resets = <&crg 0xbc 4>; + #address-cells = <1>; + #size-cells = <0>; + + usb2_phy1_port0: phy@0 { + reg = <0>; + #phy-cells = <0>; + resets = <&crg 0xbc 8>; + }; + + usb2_phy1_port1: phy@1 { + reg = <1>; + #phy-cells = <0>; + resets = <&crg 0xbc 9>; + }; + }; + + usb2_phy2: usb2-phy@124 { + compatible = "hisilicon,hi3798cv200-usb2-phy"; + reg = <0x124 0x4>; + clocks = <&crg HISTB_USB2_PHY2_REF_CLK>; + resets = <&crg 0xbc 6>; + #address-cells = <1>; + #size-cells = <0>; + + usb2_phy2_port0: phy@0 { + reg = <0>; + #phy-cells = <0>; + resets = <&crg 0xbc 10>; + }; + }; + + combphy0: phy@850 { + compatible = "hisilicon,hi3798cv200-combphy"; + reg = <0x850 0x8>; + #phy-cells = <1>; + clocks = <&crg HISTB_COMBPHY0_CLK>; + resets = <&crg 0x188 4>; + assigned-clocks = <&crg HISTB_COMBPHY0_CLK>; + assigned-clock-rates = <100000000>; + hisilicon,fixed-mode = ; + }; + + combphy1: phy@858 { + compatible = "hisilicon,hi3798cv200-combphy"; + reg = <0x858 0x8>; + #phy-cells = <1>; + clocks = <&crg HISTB_COMBPHY1_CLK>; + resets = <&crg 0x188 12>; + assigned-clocks = <&crg HISTB_COMBPHY1_CLK>; + assigned-clock-rates = <100000000>; + hisilicon,mode-select-bits = <0x0008 11 (0x3 << 11)>; + }; + }; + + pmx0: pinconf@8a21000 { + compatible = "pinconf-single"; + reg = <0x8a21000 0x180>; + pinctrl-single,register-width = <32>; + pinctrl-single,function-mask = <7>; + pinctrl-single,gpio-range = < + &range 0 8 2 /* GPIO 0 */ + &range 8 1 0 /* GPIO 1 */ + &range 9 4 2 + &range 13 1 0 + &range 14 1 1 + &range 15 1 0 + &range 16 5 0 /* GPIO 2 */ + &range 21 3 1 + &range 24 4 1 /* GPIO 3 */ + &range 28 2 2 + &range 86 1 1 + &range 87 1 0 + &range 30 4 2 /* GPIO 4 */ + &range 34 3 0 + &range 37 1 2 + &range 38 3 2 /* GPIO 6 */ + &range 41 5 0 + &range 46 8 1 /* GPIO 7 */ + &range 54 8 1 /* GPIO 8 */ + &range 64 7 1 /* GPIO 9 */ + &range 71 1 0 + &range 72 6 1 /* GPIO 10 */ + &range 78 1 0 + &range 79 1 1 + &range 80 6 1 /* GPIO 11 */ + &range 70 2 1 + &range 88 8 0 /* GPIO 12 */ + >; + + range: gpio-range { + #pinctrl-single,gpio-range-cells = <3>; + }; + }; + uart0: serial@8b00000 { compatible = "arm,pl011", "arm,primecell"; reg = <0x8b00000 0x1000>; @@ -205,12 +314,17 @@ }; emmc: mmc@9830000 { - compatible = "snps,dw-mshc"; + compatible = "hisilicon,hi3798cv200-dw-mshc"; reg = <0x9830000 0x10000>; interrupts = ; clocks = <&crg HISTB_MMC_CIU_CLK>, - <&crg HISTB_MMC_BIU_CLK>; - clock-names = "ciu", "biu"; + <&crg HISTB_MMC_BIU_CLK>, + <&crg HISTB_MMC_SAMPLE_CLK>, + <&crg HISTB_MMC_DRV_CLK>; + clock-names = "ciu", "biu", "ciu-sample", "ciu-drive"; + resets = <&crg 0xa0 4>; + reset-names = "reset"; + status = "disabled"; }; gpio0: gpio@8b20000 { @@ -221,6 +335,7 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = <&pmx0 0 0 8>; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -234,6 +349,13 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = < + &pmx0 0 8 1 + &pmx0 1 9 4 + &pmx0 5 13 1 + &pmx0 6 14 1 + &pmx0 7 15 1 + >; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -247,6 +369,7 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = <&pmx0 0 16 5 &pmx0 5 21 3>; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -260,6 +383,12 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = < + &pmx0 0 24 4 + &pmx0 4 28 2 + &pmx0 6 86 1 + &pmx0 7 87 1 + >; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -273,6 +402,7 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = <&pmx0 0 30 4 &pmx0 4 34 3 &pmx0 7 37 1>; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -299,6 +429,7 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = <&pmx0 0 38 3 &pmx0 0 41 5>; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -312,6 +443,7 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = <&pmx0 0 46 8>; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -325,6 +457,7 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = <&pmx0 0 54 8>; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -338,6 +471,7 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = <&pmx0 0 64 7 &pmx0 71 1>; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -351,6 +485,7 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = <&pmx0 0 72 6 &pmx0 6 78 1 &pmx0 7 79 1>; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -364,6 +499,7 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = <&pmx0 0 80 6 &pmx0 6 70 2>; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -377,6 +513,7 @@ #gpio-cells = <2>; interrupt-controller; #interrupt-cells = <2>; + gpio-ranges = <&pmx0 0 88 8>; clocks = <&crg HISTB_APB_CLK>; clock-names = "apb_pclk"; status = "disabled"; @@ -419,5 +556,67 @@ clocks = <&sysctrl HISTB_IR_CLK>; status = "disabled"; }; + + pcie: pcie@9860000 { + compatible = "hisilicon,hi3798cv200-pcie"; + reg = <0x9860000 0x1000>, + <0x0 0x2000>, + <0x2000000 0x01000000>; + reg-names = "control", "rc-dbi", "config"; + #address-cells = <3>; + #size-cells = <2>; + device_type = "pci"; + bus-range = <0 15>; + num-lanes = <1>; + ranges = <0x81000000 0x0 0x00000000 0x4f00000 0x0 0x100000 + 0x82000000 0x0 0x3000000 0x3000000 0x0 0x01f00000>; + interrupts = ; + interrupt-names = "msi"; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0>; + interrupt-map = <0 0 0 0 &gic 0 131 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&crg HISTB_PCIE_AUX_CLK>, + <&crg HISTB_PCIE_PIPE_CLK>, + <&crg HISTB_PCIE_SYS_CLK>, + <&crg HISTB_PCIE_BUS_CLK>; + clock-names = "aux", "pipe", "sys", "bus"; + resets = <&crg 0x18c 6>, <&crg 0x18c 5>, <&crg 0x18c 4>; + reset-names = "soft", "sys", "bus"; + phys = <&combphy1 PHY_TYPE_PCIE>; + phy-names = "phy"; + status = "disabled"; + }; + + ohci: ohci@9880000 { + compatible = "generic-ohci"; + reg = <0x9880000 0x10000>; + interrupts = ; + clocks = <&crg HISTB_USB2_BUS_CLK>, + <&crg HISTB_USB2_12M_CLK>, + <&crg HISTB_USB2_48M_CLK>; + clock-names = "bus", "clk12", "clk48"; + resets = <&crg 0xb8 12>; + reset-names = "bus"; + phys = <&usb2_phy1_port0>; + phy-names = "usb"; + status = "disabled"; + }; + + ehci: ehci@9890000 { + compatible = "generic-ehci"; + reg = <0x9890000 0x10000>; + interrupts = ; + clocks = <&crg HISTB_USB2_BUS_CLK>, + <&crg HISTB_USB2_PHY_CLK>, + <&crg HISTB_USB2_UTMI_CLK>; + clock-names = "bus", "phy", "utmi"; + resets = <&crg 0xb8 12>, + <&crg 0xb8 16>, + <&crg 0xb8 13>; + reset-names = "bus", "phy", "utmi"; + phys = <&usb2_phy1_port0>; + phy-names = "usb"; + status = "disabled"; + }; }; }; diff --git a/arch/arm64/boot/dts/hisilicon/hip06-d03.dts b/arch/arm64/boot/dts/hisilicon/hip06-d03.dts index 9af633021a42..a95c6f5619bf 100644 --- a/arch/arm64/boot/dts/hisilicon/hip06-d03.dts +++ b/arch/arm64/boot/dts/hisilicon/hip06-d03.dts @@ -25,6 +25,14 @@ chosen { }; }; +&ipmi0 { + status = "ok"; +}; + +&uart0 { + status = "ok"; +}; + ð0 { status = "ok"; }; diff --git a/arch/arm64/boot/dts/hisilicon/hip06.dtsi b/arch/arm64/boot/dts/hisilicon/hip06.dtsi index 35202ebe62a7..d78a6a755d03 100644 --- a/arch/arm64/boot/dts/hisilicon/hip06.dtsi +++ b/arch/arm64/boot/dts/hisilicon/hip06.dtsi @@ -350,6 +350,27 @@ #size-cells = <2>; ranges; + isa@a01b0000 { + compatible = "hisilicon,hip06-lpc"; + #size-cells = <1>; + #address-cells = <2>; + reg = <0x0 0xa01b0000 0x0 0x1000>; + + ipmi0: bt@e4 { + compatible = "ipmi-bt"; + device_type = "ipmi"; + reg = <0x01 0xe4 0x04>; + status = "disabled"; + }; + + uart0: lpc-uart@2f8 { + compatible = "ns16550a"; + clock-frequency = <1843200>; + reg = <0x01 0x2f8 0x08>; + status = "disabled"; + }; + }; + refclk: refclk { compatible = "fixed-clock"; clock-frequency = <50000000>; diff --git a/arch/arm64/boot/dts/hisilicon/hip07-d05.dts b/arch/arm64/boot/dts/hisilicon/hip07-d05.dts index fe7c16c36025..21147e8e3f94 100644 --- a/arch/arm64/boot/dts/hisilicon/hip07-d05.dts +++ b/arch/arm64/boot/dts/hisilicon/hip07-d05.dts @@ -57,6 +57,10 @@ status = "ok"; }; +&ipmi0 { + status = "ok"; +}; + &usb_ohci { status = "ok"; }; diff --git a/arch/arm64/boot/dts/hisilicon/hip07.dtsi b/arch/arm64/boot/dts/hisilicon/hip07.dtsi index 0600a6a84ab7..9c10030a07f8 100644 --- a/arch/arm64/boot/dts/hisilicon/hip07.dtsi +++ b/arch/arm64/boot/dts/hisilicon/hip07.dtsi @@ -1114,6 +1114,20 @@ #size-cells = <2>; ranges; + isa@a01b0000 { + compatible = "hisilicon,hip07-lpc"; + #size-cells = <1>; + #address-cells = <2>; + reg = <0x0 0xa01b0000 0x0 0x1000>; + + ipmi0: bt@e4 { + compatible = "ipmi-bt"; + device_type = "ipmi"; + reg = <0x01 0xe4 0x04>; + status = "disabled"; + }; + }; + uart0: uart@602b0000 { compatible = "arm,sbsa-uart"; reg = <0x0 0x602b0000 0x0 0x1000>; diff --git a/arch/arm64/boot/dts/hisilicon/poplar-pinctrl.dtsi b/arch/arm64/boot/dts/hisilicon/poplar-pinctrl.dtsi new file mode 100644 index 000000000000..7bb19e4b084a --- /dev/null +++ b/arch/arm64/boot/dts/hisilicon/poplar-pinctrl.dtsi @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Pinctrl dts file for HiSilicon Poplar board + * + * Copyright (c) 2016-2018 HiSilicon Technologies Co., Ltd. + */ + +#include + +/* value, enable bits, disable bits, mask */ +#define PINCTRL_PULLDOWN(value, enable, disable, mask) \ + (value << 13) (enable << 13) (disable << 13) (mask << 13) +#define PINCTRL_PULLUP(value, enable, disable, mask) \ + (value << 12) (enable << 12) (disable << 12) (mask << 12) +#define PINCTRL_SLEW_RATE(value, mask) (value << 8) (mask << 8) +#define PINCTRL_DRV_STRENGTH(value, mask) (value << 4) (mask << 4) + +&pmx0 { + emmc_pins_1: emmc-pins-1 { + pinctrl-single,pins = < + 0x000 MUX_M2 + 0x004 MUX_M2 + 0x008 MUX_M2 + 0x00c MUX_M2 + 0x010 MUX_M2 + 0x014 MUX_M2 + 0x018 MUX_M2 + 0x01c MUX_M2 + 0x024 MUX_M2 + >; + pinctrl-single,bias-pulldown = < + PINCTRL_PULLDOWN(0, 1, 0, 1) + >; + pinctrl-single,bias-pullup = < + PINCTRL_PULLUP(0, 1, 0, 1) + >; + pinctrl-single,slew-rate = < + PINCTRL_SLEW_RATE(1, 1) + >; + pinctrl-single,drive-strength = < + PINCTRL_DRV_STRENGTH(0xb, 0xf) + >; + }; + + emmc_pins_2: emmc-pins-2 { + pinctrl-single,pins = < + 0x028 MUX_M2 + >; + pinctrl-single,bias-pulldown = < + PINCTRL_PULLDOWN(0, 1, 0, 1) + >; + pinctrl-single,bias-pullup = < + PINCTRL_PULLUP(0, 1, 0, 1) + >; + pinctrl-single,slew-rate = < + PINCTRL_SLEW_RATE(1, 1) + >; + pinctrl-single,drive-strength = < + PINCTRL_DRV_STRENGTH(0x9, 0xf) + >; + }; + + emmc_pins_3: emmc-pins-3 { + pinctrl-single,pins = < + 0x02c MUX_M2 + >; + pinctrl-single,bias-pulldown = < + PINCTRL_PULLDOWN(0, 1, 0, 1) + >; + pinctrl-single,bias-pullup = < + PINCTRL_PULLUP(0, 1, 0, 1) + >; + pinctrl-single,slew-rate = < + PINCTRL_SLEW_RATE(1, 1) + >; + pinctrl-single,drive-strength = < + PINCTRL_DRV_STRENGTH(3, 3) + >; + }; + + emmc_pins_4: emmc-pins-4 { + pinctrl-single,pins = < + 0x030 MUX_M2 + >; + pinctrl-single,bias-pulldown = < + PINCTRL_PULLDOWN(1, 1, 0, 1) + >; + pinctrl-single,bias-pullup = < + PINCTRL_PULLUP(0, 1, 0, 1) + >; + pinctrl-single,slew-rate = < + PINCTRL_SLEW_RATE(1, 1) + >; + pinctrl-single,drive-strength = < + PINCTRL_DRV_STRENGTH(3, 3) + >; + }; +}; diff --git a/arch/arm64/boot/dts/marvell/Makefile b/arch/arm64/boot/dts/marvell/Makefile index cb454beede55..ea9d49f2a911 100644 --- a/arch/arm64/boot/dts/marvell/Makefile +++ b/arch/arm64/boot/dts/marvell/Makefile @@ -1,8 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 -# Berlin SoC Family -dtb-$(CONFIG_ARCH_BERLIN) += berlin4ct-dmp.dtb -dtb-$(CONFIG_ARCH_BERLIN) += berlin4ct-stb.dtb - # Mvebu SoC Family dtb-$(CONFIG_ARCH_MVEBU) += armada-3720-db.dtb dtb-$(CONFIG_ARCH_MVEBU) += armada-3720-espressobin.dtb diff --git a/arch/arm64/boot/dts/marvell/armada-3720-espressobin.dts b/arch/arm64/boot/dts/marvell/armada-3720-espressobin.dts index ef7fd2ca2515..3ab25ad402b9 100644 --- a/arch/arm64/boot/dts/marvell/armada-3720-espressobin.dts +++ b/arch/arm64/boot/dts/marvell/armada-3720-espressobin.dts @@ -63,6 +63,33 @@ status = "okay"; }; +&spi0 { + status = "okay"; + + flash@0 { + reg = <0>; + compatible = "winbond,w25q32dw", "jedec,spi-flash"; + spi-max-frequency = <104000000>; + m25p,fast-read; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "uboot"; + reg = <0 0x180000>; + }; + + partition@180000 { + label = "ubootenv"; + reg = <0x180000 0x10000>; + }; + }; + }; +}; + /* Exported on the micro USB connector J5 through an FTDI */ &uart0 { pinctrl-names = "default"; diff --git a/arch/arm64/boot/dts/marvell/armada-37xx.dtsi b/arch/arm64/boot/dts/marvell/armada-37xx.dtsi index 97207a61bc79..3353252d78a0 100644 --- a/arch/arm64/boot/dts/marvell/armada-37xx.dtsi +++ b/arch/arm64/boot/dts/marvell/armada-37xx.dtsi @@ -148,10 +148,13 @@ compatible = "marvell,armada3710-nb-pinctrl", "syscon", "simple-mfd"; reg = <0x13800 0x100>, <0x13C00 0x20>; + /* MPP1[19:0] */ gpionb: gpio { #gpio-cells = <2>; gpio-ranges = <&pinctrl_nb 0 0 36>; gpio-controller; + interrupt-controller; + #interrupt-cells = <2>; interrupts = , , @@ -209,10 +212,13 @@ compatible = "marvell,armada3710-sb-pinctrl", "syscon", "simple-mfd"; reg = <0x18800 0x100>, <0x18C00 0x20>; + /* MPP2[23:0] */ gpiosb: gpio { #gpio-cells = <2>; gpio-ranges = <&pinctrl_sb 0 0 30>; gpio-controller; + interrupt-controller; + #interrupt-cells = <2>; interrupts = , , diff --git a/arch/arm64/boot/dts/marvell/armada-7040-db.dts b/arch/arm64/boot/dts/marvell/armada-7040-db.dts index d6bec058a30a..412efdb46e7c 100644 --- a/arch/arm64/boot/dts/marvell/armada-7040-db.dts +++ b/arch/arm64/boot/dts/marvell/armada-7040-db.dts @@ -242,6 +242,11 @@ phy-mode = "10gbase-kr"; /* Generic PHY, providing serdes lanes */ phys = <&cp0_comphy2 0>; + + fixed-link { + speed = <10000>; + full-duplex; + }; }; &cp0_eth1 { diff --git a/arch/arm64/boot/dts/marvell/armada-8040-db.dts b/arch/arm64/boot/dts/marvell/armada-8040-db.dts index 5689fb23bbab..1bac437369a1 100644 --- a/arch/arm64/boot/dts/marvell/armada-8040-db.dts +++ b/arch/arm64/boot/dts/marvell/armada-8040-db.dts @@ -177,6 +177,11 @@ &cp0_eth0 { status = "okay"; phy-mode = "10gbase-kr"; + + fixed-link { + speed = <10000>; + full-duplex; + }; }; &cp0_eth2 { @@ -303,6 +308,11 @@ &cp1_eth0 { status = "okay"; phy-mode = "10gbase-kr"; + + fixed-link { + speed = <10000>; + full-duplex; + }; }; &cp1_eth1 { diff --git a/arch/arm64/boot/dts/marvell/armada-8040-mcbin.dts b/arch/arm64/boot/dts/marvell/armada-8040-mcbin.dts index 81de03ef860d..a66958ff4de6 100644 --- a/arch/arm64/boot/dts/marvell/armada-8040-mcbin.dts +++ b/arch/arm64/boot/dts/marvell/armada-8040-mcbin.dts @@ -27,6 +27,7 @@ ethernet0 = &cp0_eth0; ethernet1 = &cp1_eth0; ethernet2 = &cp1_eth1; + ethernet3 = &cp1_eth2; }; /* Regulator labels correspond with schematics */ @@ -64,6 +65,42 @@ compatible = "usb-nop-xceiv"; vcc-supply = <&v_5v0_usb3_hst_vbus>; }; + + sfp_eth0: sfp-eth0 { + /* CON15,16 - CPM lane 4 */ + compatible = "sff,sfp"; + i2c-bus = <&sfpp0_i2c>; + los-gpio = <&cp1_gpio1 28 GPIO_ACTIVE_HIGH>; + mod-def0-gpio = <&cp1_gpio1 27 GPIO_ACTIVE_LOW>; + tx-disable-gpio = <&cp1_gpio1 29 GPIO_ACTIVE_HIGH>; + tx-fault-gpio = <&cp1_gpio1 26 GPIO_ACTIVE_HIGH>; + pinctrl-names = "default"; + pinctrl-0 = <&cp1_sfpp0_pins>; + }; + + sfp_eth1: sfp-eth1 { + /* CON17,18 - CPS lane 4 */ + compatible = "sff,sfp"; + i2c-bus = <&sfpp1_i2c>; + los-gpio = <&cp1_gpio1 8 GPIO_ACTIVE_HIGH>; + mod-def0-gpio = <&cp1_gpio1 11 GPIO_ACTIVE_LOW>; + tx-disable-gpio = <&cp1_gpio1 10 GPIO_ACTIVE_HIGH>; + tx-fault-gpio = <&cp0_gpio2 30 GPIO_ACTIVE_HIGH>; + pinctrl-names = "default"; + pinctrl-0 = <&cp1_sfpp1_pins &cp0_sfpp1_pins>; + }; + + sfp_eth3: sfp-eth3 { + /* CON3,4 - CPS lane 5 */ + compatible = "sff,sfp"; + i2c-bus = <&sfp_1g_i2c>; + los-gpio = <&cp0_gpio2 22 GPIO_ACTIVE_HIGH>; + mod-def0-gpio = <&cp0_gpio2 21 GPIO_ACTIVE_LOW>; + tx-disable-gpio = <&cp1_gpio1 24 GPIO_ACTIVE_HIGH>; + tx-fault-gpio = <&cp0_gpio2 19 GPIO_ACTIVE_HIGH>; + pinctrl-names = "default"; + pinctrl-0 = <&cp0_sfp_1g_pins &cp1_sfp_1g_pins>; + }; }; &uart0 { @@ -171,6 +208,10 @@ marvell,pins = "mpp47"; marvell,function = "gpio"; }; + cp0_sfp_1g_pins: sfp-1g-pins { + marvell,pins = "mpp51", "mpp53", "mpp54"; + marvell,function = "gpio"; + }; cp0_pcie_pins: pcie-pins { marvell,pins = "mpp52"; marvell,function = "gpio"; @@ -180,6 +221,10 @@ "mpp60", "mpp61"; marvell,function = "sdio"; }; + cp0_sfpp1_pins: sfpp1-pins { + marvell,pins = "mpp62"; + marvell,function = "gpio"; + }; }; &cp0_xmdio { @@ -188,11 +233,13 @@ phy0: ethernet-phy@0 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0>; + sfp = <&sfp_eth0>; }; phy8: ethernet-phy@8 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <8>; + sfp = <&sfp_eth1>; }; }; @@ -257,7 +304,22 @@ phys = <&cp1_comphy0 1>; }; +&cp1_eth2 { + /* CPS Lane 5 */ + status = "okay"; + /* Network PHY */ + phy-mode = "2500base-x"; + managed = "in-band-status"; + /* Generic PHY, providing serdes lanes */ + phys = <&cp1_comphy5 2>; + sfp = <&sfp_eth3>; +}; + &cp1_pinctrl { + cp1_sfpp1_pins: sfpp1-pins { + marvell,pins = "mpp8", "mpp10", "mpp11"; + marvell,function = "gpio"; + }; cp1_spi1_pins: spi1-pins { marvell,pins = "mpp12", "mpp13", "mpp14", "mpp15", "mpp16"; marvell,function = "spi1"; @@ -266,6 +328,14 @@ marvell,pins = "mpp6", "mpp7"; marvell,function = "uart0"; }; + cp1_sfp_1g_pins: sfp-1g-pins { + marvell,pins = "mpp24"; + marvell,function = "gpio"; + }; + cp1_sfpp0_pins: sfpp0-pins { + marvell,pins = "mpp26", "mpp27", "mpp28", "mpp29"; + marvell,function = "gpio"; + }; }; /* J27 UART header */ diff --git a/arch/arm64/boot/dts/marvell/armada-cp110.dtsi b/arch/arm64/boot/dts/marvell/armada-cp110.dtsi index ed2f1237ea1e..7dabe25f6774 100644 --- a/arch/arm64/boot/dts/marvell/armada-cp110.dtsi +++ b/arch/arm64/boot/dts/marvell/armada-cp110.dtsi @@ -236,6 +236,7 @@ compatible = "marvell,armada-8k-ahci", "generic-ahci"; reg = <0x540000 0x30000>; + dma-coherent; interrupts = ; clocks = <&CP110_LABEL(clk) 1 15>, <&CP110_LABEL(clk) 1 16>; diff --git a/arch/arm64/boot/dts/marvell/berlin4ct-dmp.dts b/arch/arm64/boot/dts/marvell/berlin4ct-dmp.dts deleted file mode 100644 index fae6c6924705..000000000000 --- a/arch/arm64/boot/dts/marvell/berlin4ct-dmp.dts +++ /dev/null @@ -1,66 +0,0 @@ -/* - * Copyright (C) 2015 Marvell Technology Group Ltd. - * - * Author: Jisheng Zhang - * - * This file is dual-licensed: you can use it either under the terms - * of the GPLv2 or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This library is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation; either version 2 of the - * License, or (at your option) any later version. - * - * This library is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - */ - -/dts-v1/; - -#include "berlin4ct.dtsi" - -/ { - model = "Marvell BG4CT DMP board"; - compatible = "marvell,berlin4ct-dmp", "marvell,berlin4ct", "marvell,berlin"; - - chosen { - stdout-path = "serial0:115200n8"; - }; - - memory@1000000 { - device_type = "memory"; - /* the first 16MB is for firmwares' usage */ - reg = <0 0x01000000 0 0x7f000000>; - }; -}; - -&uart0 { - status = "okay"; -}; diff --git a/arch/arm64/boot/dts/marvell/berlin4ct-stb.dts b/arch/arm64/boot/dts/marvell/berlin4ct-stb.dts deleted file mode 100644 index d47edad13e68..000000000000 --- a/arch/arm64/boot/dts/marvell/berlin4ct-stb.dts +++ /dev/null @@ -1,66 +0,0 @@ -/* - * Copyright (C) 2015 Marvell Technology Group Ltd. - * - * Author: Jisheng Zhang - * - * This file is dual-licensed: you can use it either under the terms - * of the GPLv2 or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This library is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation; either version 2 of the - * License, or (at your option) any later version. - * - * This library is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - */ - -/dts-v1/; - -#include "berlin4ct.dtsi" - -/ { - model = "Marvell BG4CT STB board"; - compatible = "marvell,berlin4ct-stb", "marvell,berlin4ct", "marvell,berlin"; - - chosen { - stdout-path = "serial0:115200n8"; - }; - - memory@1000000 { - device_type = "memory"; - /* the first 16MB is for firmwares' usage */ - reg = <0 0x01000000 0 0x7f000000>; - }; -}; - -&uart0 { - status = "okay"; -}; diff --git a/arch/arm64/boot/dts/mediatek/mt2712-pinfunc.h b/arch/arm64/boot/dts/mediatek/mt2712-pinfunc.h new file mode 100644 index 000000000000..1b4cb0c55744 --- /dev/null +++ b/arch/arm64/boot/dts/mediatek/mt2712-pinfunc.h @@ -0,0 +1,1123 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2018 MediaTek Inc. + * Author: Zhiyong Tao + * + */ +#ifndef __DTS_MT2712_PINFUNC_H +#define __DTS_MT2712_PINFUNC_H + +#include + +#define MT2712_PIN_0_EINT0__FUNC_GPIO0 (MTK_PIN_NO(0) | 0) +#define MT2712_PIN_0_EINT0__FUNC_EINT0 (MTK_PIN_NO(0) | 1) +#define MT2712_PIN_0_EINT0__FUNC_MBIST_DIAG_SCANOUT (MTK_PIN_NO(0) | 2) +#define MT2712_PIN_0_EINT0__FUNC_DSIA_TE (MTK_PIN_NO(0) | 3) +#define MT2712_PIN_0_EINT0__FUNC_DSIC_TE (MTK_PIN_NO(0) | 4) +#define MT2712_PIN_0_EINT0__FUNC_DIN_D3 (MTK_PIN_NO(0) | 5) +#define MT2712_PIN_0_EINT0__FUNC_PURE_HW_PROTECT (MTK_PIN_NO(0) | 6) + +#define MT2712_PIN_1_EINT1__FUNC_GPIO1 (MTK_PIN_NO(1) | 0) +#define MT2712_PIN_1_EINT1__FUNC_EINT1 (MTK_PIN_NO(1) | 1) +#define MT2712_PIN_1_EINT1__FUNC_IR_IN (MTK_PIN_NO(1) | 2) +#define MT2712_PIN_1_EINT1__FUNC_DSIB_TE (MTK_PIN_NO(1) | 3) +#define MT2712_PIN_1_EINT1__FUNC_DSID_TE (MTK_PIN_NO(1) | 4) +#define MT2712_PIN_1_EINT1__FUNC_DIN_D4 (MTK_PIN_NO(1) | 5) + +#define MT2712_PIN_2_EINT2__FUNC_GPIO2 (MTK_PIN_NO(2) | 0) +#define MT2712_PIN_2_EINT2__FUNC_EINT2 (MTK_PIN_NO(2) | 1) +#define MT2712_PIN_2_EINT2__FUNC_IR_IN (MTK_PIN_NO(2) | 2) +#define MT2712_PIN_2_EINT2__FUNC_LCM_RST1 (MTK_PIN_NO(2) | 3) +#define MT2712_PIN_2_EINT2__FUNC_DIN_D5 (MTK_PIN_NO(2) | 5) + +#define MT2712_PIN_3_EINT3__FUNC_GPIO3 (MTK_PIN_NO(3) | 0) +#define MT2712_PIN_3_EINT3__FUNC_EINT3 (MTK_PIN_NO(3) | 1) +#define MT2712_PIN_3_EINT3__FUNC_IR_IN (MTK_PIN_NO(3) | 2) +#define MT2712_PIN_3_EINT3__FUNC_LCM_RST0 (MTK_PIN_NO(3) | 3) +#define MT2712_PIN_3_EINT3__FUNC_DIN_D6 (MTK_PIN_NO(3) | 5) + +#define MT2712_PIN_4_PWM0__FUNC_GPIO4 (MTK_PIN_NO(4) | 0) +#define MT2712_PIN_4_PWM0__FUNC_PWM0 (MTK_PIN_NO(4) | 1) +#define MT2712_PIN_4_PWM0__FUNC_DISP0_PWM (MTK_PIN_NO(4) | 2) +#define MT2712_PIN_4_PWM0__FUNC_DISP1_PWM (MTK_PIN_NO(4) | 3) +#define MT2712_PIN_4_PWM0__FUNC_DIN_CLK (MTK_PIN_NO(4) | 5) + +#define MT2712_PIN_5_PWM1__FUNC_GPIO5 (MTK_PIN_NO(5) | 0) +#define MT2712_PIN_5_PWM1__FUNC_PWM1 (MTK_PIN_NO(5) | 1) +#define MT2712_PIN_5_PWM1__FUNC_DISP1_PWM (MTK_PIN_NO(5) | 2) +#define MT2712_PIN_5_PWM1__FUNC_DISP0_PWM (MTK_PIN_NO(5) | 3) +#define MT2712_PIN_5_PWM1__FUNC_DIN_VSYNC (MTK_PIN_NO(5) | 5) + +#define MT2712_PIN_6_PWM2__FUNC_GPIO6 (MTK_PIN_NO(6) | 0) +#define MT2712_PIN_6_PWM2__FUNC_PWM2 (MTK_PIN_NO(6) | 1) +#define MT2712_PIN_6_PWM2__FUNC_DISP0_PWM (MTK_PIN_NO(6) | 2) +#define MT2712_PIN_6_PWM2__FUNC_DISP1_PWM (MTK_PIN_NO(6) | 3) +#define MT2712_PIN_6_PWM2__FUNC_DISP2_PWM (MTK_PIN_NO(6) | 4) +#define MT2712_PIN_6_PWM2__FUNC_DIN_HSYNC (MTK_PIN_NO(6) | 5) + +#define MT2712_PIN_7_PWM3__FUNC_GPIO7 (MTK_PIN_NO(7) | 0) +#define MT2712_PIN_7_PWM3__FUNC_PWM3 (MTK_PIN_NO(7) | 1) +#define MT2712_PIN_7_PWM3__FUNC_DISP1_PWM (MTK_PIN_NO(7) | 2) +#define MT2712_PIN_7_PWM3__FUNC_DISP0_PWM (MTK_PIN_NO(7) | 3) +#define MT2712_PIN_7_PWM3__FUNC_LCM_RST2 (MTK_PIN_NO(7) | 4) +#define MT2712_PIN_7_PWM3__FUNC_DIN_D0 (MTK_PIN_NO(7) | 5) + +#define MT2712_PIN_8_PWM4__FUNC_GPIO8 (MTK_PIN_NO(8) | 0) +#define MT2712_PIN_8_PWM4__FUNC_PWM4 (MTK_PIN_NO(8) | 1) +#define MT2712_PIN_8_PWM4__FUNC_DISP0_PWM (MTK_PIN_NO(8) | 2) +#define MT2712_PIN_8_PWM4__FUNC_DISP1_PWM (MTK_PIN_NO(8) | 3) +#define MT2712_PIN_8_PWM4__FUNC_DSIA_TE (MTK_PIN_NO(8) | 4) +#define MT2712_PIN_8_PWM4__FUNC_DIN_D1 (MTK_PIN_NO(8) | 5) + +#define MT2712_PIN_9_PWM5__FUNC_GPIO9 (MTK_PIN_NO(9) | 0) +#define MT2712_PIN_9_PWM5__FUNC_PWM5 (MTK_PIN_NO(9) | 1) +#define MT2712_PIN_9_PWM5__FUNC_DISP1_PWM (MTK_PIN_NO(9) | 2) +#define MT2712_PIN_9_PWM5__FUNC_DISP0_PWM (MTK_PIN_NO(9) | 3) +#define MT2712_PIN_9_PWM5__FUNC_DSIB_TE (MTK_PIN_NO(9) | 4) +#define MT2712_PIN_9_PWM5__FUNC_DIN_D2 (MTK_PIN_NO(9) | 5) + +#define MT2712_PIN_10_PWM6__FUNC_GPIO10 (MTK_PIN_NO(10) | 0) +#define MT2712_PIN_10_PWM6__FUNC_PWM6 (MTK_PIN_NO(10) | 1) +#define MT2712_PIN_10_PWM6__FUNC_DISP0_PWM (MTK_PIN_NO(10) | 2) +#define MT2712_PIN_10_PWM6__FUNC_DISP1_PWM (MTK_PIN_NO(10) | 3) +#define MT2712_PIN_10_PWM6__FUNC_LCM_RST0 (MTK_PIN_NO(10) | 4) + +#define MT2712_PIN_11_PWM7__FUNC_GPIO11 (MTK_PIN_NO(11) | 0) +#define MT2712_PIN_11_PWM7__FUNC_PWM7 (MTK_PIN_NO(11) | 1) +#define MT2712_PIN_11_PWM7__FUNC_DISP1_PWM (MTK_PIN_NO(11) | 2) +#define MT2712_PIN_11_PWM7__FUNC_DISP0_PWM (MTK_PIN_NO(11) | 3) +#define MT2712_PIN_11_PWM7__FUNC_LCM_RST1 (MTK_PIN_NO(11) | 4) + +#define MT2712_PIN_12_IDDIG_P0__FUNC_GPIO12 (MTK_PIN_NO(12) | 0) +#define MT2712_PIN_12_IDDIG_P0__FUNC_IDDIG_A (MTK_PIN_NO(12) | 1) +#define MT2712_PIN_12_IDDIG_P0__FUNC_DIN_D7 (MTK_PIN_NO(12) | 5) + +#define MT2712_PIN_13_DRV_VBUS_P0__FUNC_GPIO13 (MTK_PIN_NO(13) | 0) +#define MT2712_PIN_13_DRV_VBUS_P0__FUNC_DRV_VBUS_A (MTK_PIN_NO(13) | 1) + +#define MT2712_PIN_14_IDDIG_P1__FUNC_GPIO14 (MTK_PIN_NO(14) | 0) +#define MT2712_PIN_14_IDDIG_P1__FUNC_IDDIG_B (MTK_PIN_NO(14) | 1) + +#define MT2712_PIN_15_DRV_VBUS_P1__FUNC_GPIO15 (MTK_PIN_NO(15) | 0) +#define MT2712_PIN_15_DRV_VBUS_P1__FUNC_DRV_VBUS_B (MTK_PIN_NO(15) | 1) + +#define MT2712_PIN_16_DRV_VBUS_P2__FUNC_GPIO16 (MTK_PIN_NO(16) | 0) +#define MT2712_PIN_16_DRV_VBUS_P2__FUNC_DRV_VBUS_C (MTK_PIN_NO(16) | 1) + +#define MT2712_PIN_17_DRV_VBUS_P3__FUNC_GPIO17 (MTK_PIN_NO(17) | 0) +#define MT2712_PIN_17_DRV_VBUS_P3__FUNC_DRV_VBUS_D (MTK_PIN_NO(17) | 1) + +#define MT2712_PIN_18_KPROW0__FUNC_GPIO18 (MTK_PIN_NO(18) | 0) +#define MT2712_PIN_18_KPROW0__FUNC_KROW0 (MTK_PIN_NO(18) | 1) + +#define MT2712_PIN_19_KPCOL0__FUNC_GPIO19 (MTK_PIN_NO(19) | 0) +#define MT2712_PIN_19_KPCOL0__FUNC_KCOL0 (MTK_PIN_NO(19) | 1) + +#define MT2712_PIN_20_KPROW1__FUNC_GPIO20 (MTK_PIN_NO(20) | 0) +#define MT2712_PIN_20_KPROW1__FUNC_KROW1 (MTK_PIN_NO(20) | 1) + +#define MT2712_PIN_21_KPCOL1__FUNC_GPIO21 (MTK_PIN_NO(21) | 0) +#define MT2712_PIN_21_KPCOL1__FUNC_KCOL1 (MTK_PIN_NO(21) | 1) + +#define MT2712_PIN_22_KPROW2__FUNC_GPIO22 (MTK_PIN_NO(22) | 0) +#define MT2712_PIN_22_KPROW2__FUNC_KROW2 (MTK_PIN_NO(22) | 1) +#define MT2712_PIN_22_KPROW2__FUNC_DISP1_PWM (MTK_PIN_NO(22) | 2) + +#define MT2712_PIN_23_KPCOL2__FUNC_GPIO23 (MTK_PIN_NO(23) | 0) +#define MT2712_PIN_23_KPCOL2__FUNC_KCOL2 (MTK_PIN_NO(23) | 1) +#define MT2712_PIN_23_KPCOL2__FUNC_DISP0_PWM (MTK_PIN_NO(23) | 2) + +#define MT2712_PIN_24_CMMCLK__FUNC_GPIO24 (MTK_PIN_NO(24) | 0) +#define MT2712_PIN_24_CMMCLK__FUNC_CMMCLK (MTK_PIN_NO(24) | 1) +#define MT2712_PIN_24_CMMCLK__FUNC_DBG_MON_A_1_ (MTK_PIN_NO(24) | 7) + +#define MT2712_PIN_25_CM2MCLK__FUNC_GPIO25 (MTK_PIN_NO(25) | 0) +#define MT2712_PIN_25_CM2MCLK__FUNC_CM2MCLK (MTK_PIN_NO(25) | 1) +#define MT2712_PIN_25_CM2MCLK__FUNC_DBG_MON_A_2_ (MTK_PIN_NO(25) | 7) + +#define MT2712_PIN_26_PCM_TX__FUNC_GPIO26 (MTK_PIN_NO(26) | 0) +#define MT2712_PIN_26_PCM_TX__FUNC_PCM1_DO (MTK_PIN_NO(26) | 1) +#define MT2712_PIN_26_PCM_TX__FUNC_MRG_TX (MTK_PIN_NO(26) | 2) +#define MT2712_PIN_26_PCM_TX__FUNC_DAI_TX (MTK_PIN_NO(26) | 3) +#define MT2712_PIN_26_PCM_TX__FUNC_MRG_RX (MTK_PIN_NO(26) | 4) +#define MT2712_PIN_26_PCM_TX__FUNC_DAI_RX (MTK_PIN_NO(26) | 5) +#define MT2712_PIN_26_PCM_TX__FUNC_PCM1_DI (MTK_PIN_NO(26) | 6) +#define MT2712_PIN_26_PCM_TX__FUNC_DBG_MON_A_3_ (MTK_PIN_NO(26) | 7) + +#define MT2712_PIN_27_PCM_CLK__FUNC_GPIO27 (MTK_PIN_NO(27) | 0) +#define MT2712_PIN_27_PCM_CLK__FUNC_PCM1_CLK (MTK_PIN_NO(27) | 1) +#define MT2712_PIN_27_PCM_CLK__FUNC_MRG_CLK (MTK_PIN_NO(27) | 2) +#define MT2712_PIN_27_PCM_CLK__FUNC_DAI_CLK (MTK_PIN_NO(27) | 3) +#define MT2712_PIN_27_PCM_CLK__FUNC_DBG_MON_A_4_ (MTK_PIN_NO(27) | 7) + +#define MT2712_PIN_28_PCM_RX__FUNC_GPIO28 (MTK_PIN_NO(28) | 0) +#define MT2712_PIN_28_PCM_RX__FUNC_PCM1_DI (MTK_PIN_NO(28) | 1) +#define MT2712_PIN_28_PCM_RX__FUNC_MRG_RX (MTK_PIN_NO(28) | 2) +#define MT2712_PIN_28_PCM_RX__FUNC_DAI_RX (MTK_PIN_NO(28) | 3) +#define MT2712_PIN_28_PCM_RX__FUNC_MRG_TX (MTK_PIN_NO(28) | 4) +#define MT2712_PIN_28_PCM_RX__FUNC_DAI_TX (MTK_PIN_NO(28) | 5) +#define MT2712_PIN_28_PCM_RX__FUNC_PCM1_DO (MTK_PIN_NO(28) | 6) +#define MT2712_PIN_28_PCM_RX__FUNC_DBG_MON_A_5_ (MTK_PIN_NO(28) | 7) + +#define MT2712_PIN_29_PCM_SYNC__FUNC_GPIO29 (MTK_PIN_NO(29) | 0) +#define MT2712_PIN_29_PCM_SYNC__FUNC_PCM1_SYNC (MTK_PIN_NO(29) | 1) +#define MT2712_PIN_29_PCM_SYNC__FUNC_MRG_SYNC (MTK_PIN_NO(29) | 2) +#define MT2712_PIN_29_PCM_SYNC__FUNC_DAI_SYNC (MTK_PIN_NO(29) | 3) +#define MT2712_PIN_29_PCM_SYNC__FUNC_DBG_MON_A_6_ (MTK_PIN_NO(29) | 7) + +#define MT2712_PIN_30_NCEB0__FUNC_GPIO30 (MTK_PIN_NO(30) | 0) +#define MT2712_PIN_30_NCEB0__FUNC_NCEB0 (MTK_PIN_NO(30) | 1) +#define MT2712_PIN_30_NCEB0__FUNC_USB0_FT_SDA (MTK_PIN_NO(30) | 2) +#define MT2712_PIN_30_NCEB0__FUNC_DBG_MON_A_7_ (MTK_PIN_NO(30) | 7) + +#define MT2712_PIN_31_NCEB1__FUNC_GPIO31 (MTK_PIN_NO(31) | 0) +#define MT2712_PIN_31_NCEB1__FUNC_NCEB1 (MTK_PIN_NO(31) | 1) +#define MT2712_PIN_31_NCEB1__FUNC_USB1_FT_SCL (MTK_PIN_NO(31) | 2) +#define MT2712_PIN_31_NCEB1__FUNC_DBG_MON_A_8_ (MTK_PIN_NO(31) | 7) + +#define MT2712_PIN_32_NF_DQS__FUNC_GPIO32 (MTK_PIN_NO(32) | 0) +#define MT2712_PIN_32_NF_DQS__FUNC_NF_DQS (MTK_PIN_NO(32) | 1) +#define MT2712_PIN_32_NF_DQS__FUNC_USB1_FT_SDA (MTK_PIN_NO(32) | 2) +#define MT2712_PIN_32_NF_DQS__FUNC_DBG_MON_A_9_ (MTK_PIN_NO(32) | 7) + +#define MT2712_PIN_33_NWEB__FUNC_GPIO33 (MTK_PIN_NO(33) | 0) +#define MT2712_PIN_33_NWEB__FUNC_NWEB (MTK_PIN_NO(33) | 1) +#define MT2712_PIN_33_NWEB__FUNC_USB2_FT_SCL (MTK_PIN_NO(33) | 2) +#define MT2712_PIN_33_NWEB__FUNC_DBG_MON_A_10_ (MTK_PIN_NO(33) | 7) + +#define MT2712_PIN_34_NREB__FUNC_GPIO34 (MTK_PIN_NO(34) | 0) +#define MT2712_PIN_34_NREB__FUNC_NREB (MTK_PIN_NO(34) | 1) +#define MT2712_PIN_34_NREB__FUNC_USB2_FT_SDA (MTK_PIN_NO(34) | 2) +#define MT2712_PIN_34_NREB__FUNC_DBG_MON_A_11_ (MTK_PIN_NO(34) | 7) + +#define MT2712_PIN_35_NCLE__FUNC_GPIO35 (MTK_PIN_NO(35) | 0) +#define MT2712_PIN_35_NCLE__FUNC_NCLE (MTK_PIN_NO(35) | 1) +#define MT2712_PIN_35_NCLE__FUNC_USB3_FT_SCL (MTK_PIN_NO(35) | 2) +#define MT2712_PIN_35_NCLE__FUNC_DBG_MON_A_12_ (MTK_PIN_NO(35) | 7) + +#define MT2712_PIN_36_NALE__FUNC_GPIO36 (MTK_PIN_NO(36) | 0) +#define MT2712_PIN_36_NALE__FUNC_NALE (MTK_PIN_NO(36) | 1) +#define MT2712_PIN_36_NALE__FUNC_USB3_FT_SDA (MTK_PIN_NO(36) | 2) +#define MT2712_PIN_36_NALE__FUNC_DBG_MON_A_13_ (MTK_PIN_NO(36) | 7) + +#define MT2712_PIN_37_MSDC0E_CLK__FUNC_GPIO37 (MTK_PIN_NO(37) | 0) +#define MT2712_PIN_37_MSDC0E_CLK__FUNC_MSDC0_CLK (MTK_PIN_NO(37) | 1) +#define MT2712_PIN_37_MSDC0E_CLK__FUNC_USB0_FT_SCL (MTK_PIN_NO(37) | 2) +#define MT2712_PIN_37_MSDC0E_CLK__FUNC_DBG_MON_A_0_ (MTK_PIN_NO(37) | 7) + +#define MT2712_PIN_38_MSDC0E_DAT7__FUNC_GPIO38 (MTK_PIN_NO(38) | 0) +#define MT2712_PIN_38_MSDC0E_DAT7__FUNC_MSDC0_DAT7 (MTK_PIN_NO(38) | 1) +#define MT2712_PIN_38_MSDC0E_DAT7__FUNC_NAND_ND7 (MTK_PIN_NO(38) | 2) +#define MT2712_PIN_38_MSDC0E_DAT7__FUNC_DBG_MON_A_14_ (MTK_PIN_NO(38) | 7) + +#define MT2712_PIN_39_MSDC0E_DAT6__FUNC_GPIO39 (MTK_PIN_NO(39) | 0) +#define MT2712_PIN_39_MSDC0E_DAT6__FUNC_MSDC0_DAT6 (MTK_PIN_NO(39) | 1) +#define MT2712_PIN_39_MSDC0E_DAT6__FUNC_NAND_ND6 (MTK_PIN_NO(39) | 2) +#define MT2712_PIN_39_MSDC0E_DAT6__FUNC_DBG_MON_A_15_ (MTK_PIN_NO(39) | 7) + +#define MT2712_PIN_40_MSDC0E_DAT5__FUNC_GPIO40 (MTK_PIN_NO(40) | 0) +#define MT2712_PIN_40_MSDC0E_DAT5__FUNC_MSDC0_DAT5 (MTK_PIN_NO(40) | 1) +#define MT2712_PIN_40_MSDC0E_DAT5__FUNC_NAND_ND5 (MTK_PIN_NO(40) | 2) +#define MT2712_PIN_40_MSDC0E_DAT5__FUNC_DBG_MON_A_16_ (MTK_PIN_NO(40) | 7) + +#define MT2712_PIN_41_MSDC0E_DAT4__FUNC_GPIO41 (MTK_PIN_NO(41) | 0) +#define MT2712_PIN_41_MSDC0E_DAT4__FUNC_MSDC0_DAT4 (MTK_PIN_NO(41) | 1) +#define MT2712_PIN_41_MSDC0E_DAT4__FUNC_NAND_ND4 (MTK_PIN_NO(41) | 2) +#define MT2712_PIN_41_MSDC0E_DAT4__FUNC_DBG_MON_A_17_ (MTK_PIN_NO(41) | 7) + +#define MT2712_PIN_42_MSDC0E_DAT3__FUNC_GPIO42 (MTK_PIN_NO(42) | 0) +#define MT2712_PIN_42_MSDC0E_DAT3__FUNC_MSDC0_DAT3 (MTK_PIN_NO(42) | 1) +#define MT2712_PIN_42_MSDC0E_DAT3__FUNC_NAND_ND3 (MTK_PIN_NO(42) | 2) +#define MT2712_PIN_42_MSDC0E_DAT3__FUNC_DBG_MON_A_18_ (MTK_PIN_NO(42) | 7) + +#define MT2712_PIN_43_MSDC0E_DAT2__FUNC_GPIO43 (MTK_PIN_NO(43) | 0) +#define MT2712_PIN_43_MSDC0E_DAT2__FUNC_MSDC0_DAT2 (MTK_PIN_NO(43) | 1) +#define MT2712_PIN_43_MSDC0E_DAT2__FUNC_NAND_ND2 (MTK_PIN_NO(43) | 2) +#define MT2712_PIN_43_MSDC0E_DAT2__FUNC_DBG_MON_A_19_ (MTK_PIN_NO(43) | 7) + +#define MT2712_PIN_44_MSDC0E_DAT1__FUNC_GPIO44 (MTK_PIN_NO(44) | 0) +#define MT2712_PIN_44_MSDC0E_DAT1__FUNC_MSDC0_DAT1 (MTK_PIN_NO(44) | 1) +#define MT2712_PIN_44_MSDC0E_DAT1__FUNC_NAND_ND1 (MTK_PIN_NO(44) | 2) +#define MT2712_PIN_44_MSDC0E_DAT1__FUNC_DBG_MON_A_20_ (MTK_PIN_NO(44) | 7) + +#define MT2712_PIN_45_MSDC0E_DAT0__FUNC_GPIO45 (MTK_PIN_NO(45) | 0) +#define MT2712_PIN_45_MSDC0E_DAT0__FUNC_MSDC0_DAT0 (MTK_PIN_NO(45) | 1) +#define MT2712_PIN_45_MSDC0E_DAT0__FUNC_NAND_ND0 (MTK_PIN_NO(45) | 2) +#define MT2712_PIN_45_MSDC0E_DAT0__FUNC_DBG_MON_A_21_ (MTK_PIN_NO(45) | 7) + +#define MT2712_PIN_46_MSDC0E_CMD__FUNC_GPIO46 (MTK_PIN_NO(46) | 0) +#define MT2712_PIN_46_MSDC0E_CMD__FUNC_MSDC0_CMD (MTK_PIN_NO(46) | 1) +#define MT2712_PIN_46_MSDC0E_CMD__FUNC_NAND_NRNB (MTK_PIN_NO(46) | 2) +#define MT2712_PIN_46_MSDC0E_CMD__FUNC_DBG_MON_A_22_ (MTK_PIN_NO(46) | 7) + +#define MT2712_PIN_47_MSDC0E_DSL__FUNC_GPIO47 (MTK_PIN_NO(47) | 0) +#define MT2712_PIN_47_MSDC0E_DSL__FUNC_MSDC0_DSL (MTK_PIN_NO(47) | 1) +#define MT2712_PIN_47_MSDC0E_DSL__FUNC_DBG_MON_A_23_ (MTK_PIN_NO(47) | 7) + +#define MT2712_PIN_48_MSDC0E_RSTB__FUNC_GPIO48 (MTK_PIN_NO(48) | 0) +#define MT2712_PIN_48_MSDC0E_RSTB__FUNC_MSDC0_RSTB (MTK_PIN_NO(48) | 1) +#define MT2712_PIN_48_MSDC0E_RSTB__FUNC_DBG_MON_A_24_ (MTK_PIN_NO(48) | 7) + +#define MT2712_PIN_49_MSDC3_DAT3__FUNC_GPIO49 (MTK_PIN_NO(49) | 0) +#define MT2712_PIN_49_MSDC3_DAT3__FUNC_MSDC3_DAT3 (MTK_PIN_NO(49) | 1) +#define MT2712_PIN_49_MSDC3_DAT3__FUNC_DBG_MON_A_25_ (MTK_PIN_NO(49) | 7) + +#define MT2712_PIN_50_MSDC3_DAT2__FUNC_GPIO50 (MTK_PIN_NO(50) | 0) +#define MT2712_PIN_50_MSDC3_DAT2__FUNC_MSDC3_DAT2 (MTK_PIN_NO(50) | 1) +#define MT2712_PIN_50_MSDC3_DAT2__FUNC_DBG_MON_A_26_ (MTK_PIN_NO(50) | 7) + +#define MT2712_PIN_51_MSDC3_DAT1__FUNC_GPIO51 (MTK_PIN_NO(51) | 0) +#define MT2712_PIN_51_MSDC3_DAT1__FUNC_MSDC3_DAT1 (MTK_PIN_NO(51) | 1) +#define MT2712_PIN_51_MSDC3_DAT1__FUNC_DBG_MON_A_27_ (MTK_PIN_NO(51) | 7) + +#define MT2712_PIN_52_MSDC3_DAT0__FUNC_GPIO52 (MTK_PIN_NO(52) | 0) +#define MT2712_PIN_52_MSDC3_DAT0__FUNC_MSDC3_DAT0 (MTK_PIN_NO(52) | 1) +#define MT2712_PIN_52_MSDC3_DAT0__FUNC_DBG_MON_A_28_ (MTK_PIN_NO(52) | 7) + +#define MT2712_PIN_53_MSDC3_CMD__FUNC_GPIO53 (MTK_PIN_NO(53) | 0) +#define MT2712_PIN_53_MSDC3_CMD__FUNC_MSDC3_CMD (MTK_PIN_NO(53) | 1) +#define MT2712_PIN_53_MSDC3_CMD__FUNC_DBG_MON_A_29_ (MTK_PIN_NO(53) | 7) + +#define MT2712_PIN_54_MSDC3_INS__FUNC_GPIO54 (MTK_PIN_NO(54) | 0) +#define MT2712_PIN_54_MSDC3_INS__FUNC_MSDC3_INS (MTK_PIN_NO(54) | 1) +#define MT2712_PIN_54_MSDC3_INS__FUNC_DBG_MON_A_30_ (MTK_PIN_NO(54) | 7) + +#define MT2712_PIN_55_MSDC3_DSL__FUNC_GPIO55 (MTK_PIN_NO(55) | 0) +#define MT2712_PIN_55_MSDC3_DSL__FUNC_MSDC3_DSL (MTK_PIN_NO(55) | 1) +#define MT2712_PIN_55_MSDC3_DSL__FUNC_DBG_MON_A_31_ (MTK_PIN_NO(55) | 7) + +#define MT2712_PIN_56_MSDC3_CLK__FUNC_GPIO56 (MTK_PIN_NO(56) | 0) +#define MT2712_PIN_56_MSDC3_CLK__FUNC_MSDC3_CLK (MTK_PIN_NO(56) | 1) +#define MT2712_PIN_56_MSDC3_CLK__FUNC_DBG_MON_A_32_ (MTK_PIN_NO(56) | 7) + +#define MT2712_PIN_57_NOR_CS__FUNC_GPIO57 (MTK_PIN_NO(57) | 0) +#define MT2712_PIN_57_NOR_CS__FUNC_NOR_CS (MTK_PIN_NO(57) | 1) + +#define MT2712_PIN_58_NOR_CK__FUNC_GPIO58 (MTK_PIN_NO(58) | 0) +#define MT2712_PIN_58_NOR_CK__FUNC_NOR_CK (MTK_PIN_NO(58) | 1) + +#define MT2712_PIN_59_NOR_IO0__FUNC_GPIO59 (MTK_PIN_NO(59) | 0) +#define MT2712_PIN_59_NOR_IO0__FUNC_NOR_IO0 (MTK_PIN_NO(59) | 1) + +#define MT2712_PIN_60_NOR_IO1__FUNC_GPIO60 (MTK_PIN_NO(60) | 0) +#define MT2712_PIN_60_NOR_IO1__FUNC_NOR_IO1 (MTK_PIN_NO(60) | 1) + +#define MT2712_PIN_61_NOR_IO2__FUNC_GPIO61 (MTK_PIN_NO(61) | 0) +#define MT2712_PIN_61_NOR_IO2__FUNC_NOR_IO2 (MTK_PIN_NO(61) | 1) + +#define MT2712_PIN_62_NOR_IO3__FUNC_GPIO62 (MTK_PIN_NO(62) | 0) +#define MT2712_PIN_62_NOR_IO3__FUNC_NOR_IO3 (MTK_PIN_NO(62) | 1) + +#define MT2712_PIN_63_MSDC1_CLK__FUNC_GPIO63 (MTK_PIN_NO(63) | 0) +#define MT2712_PIN_63_MSDC1_CLK__FUNC_MSDC1_CLK (MTK_PIN_NO(63) | 1) +#define MT2712_PIN_63_MSDC1_CLK__FUNC_UDI_TCK (MTK_PIN_NO(63) | 2) + +#define MT2712_PIN_64_MSDC1_DAT3__FUNC_GPIO64 (MTK_PIN_NO(64) | 0) +#define MT2712_PIN_64_MSDC1_DAT3__FUNC_MSDC1_DAT3 (MTK_PIN_NO(64) | 1) +#define MT2712_PIN_64_MSDC1_DAT3__FUNC_UDI_TDI (MTK_PIN_NO(64) | 2) + +#define MT2712_PIN_65_MSDC1_DAT1__FUNC_GPIO65 (MTK_PIN_NO(65) | 0) +#define MT2712_PIN_65_MSDC1_DAT1__FUNC_MSDC1_DAT1 (MTK_PIN_NO(65) | 1) +#define MT2712_PIN_65_MSDC1_DAT1__FUNC_UDI_TMS (MTK_PIN_NO(65) | 2) + +#define MT2712_PIN_66_MSDC1_DAT2__FUNC_GPIO66 (MTK_PIN_NO(66) | 0) +#define MT2712_PIN_66_MSDC1_DAT2__FUNC_MSDC1_DAT2 (MTK_PIN_NO(66) | 1) +#define MT2712_PIN_66_MSDC1_DAT2__FUNC_UDI_TDO (MTK_PIN_NO(66) | 2) + +#define MT2712_PIN_67_MSDC1_PSW__FUNC_GPIO67 (MTK_PIN_NO(67) | 0) +#define MT2712_PIN_67_MSDC1_PSW__FUNC_UDI_NTRST (MTK_PIN_NO(67) | 2) + +#define MT2712_PIN_68_MSDC1_DAT0__FUNC_GPIO68 (MTK_PIN_NO(68) | 0) +#define MT2712_PIN_68_MSDC1_DAT0__FUNC_MSDC1_DAT0 (MTK_PIN_NO(68) | 1) + +#define MT2712_PIN_69_MSDC1_CMD__FUNC_GPIO69 (MTK_PIN_NO(69) | 0) +#define MT2712_PIN_69_MSDC1_CMD__FUNC_MSDC1_CMD (MTK_PIN_NO(69) | 1) + +#define MT2712_PIN_70_MSDC1_INS__FUNC_GPIO70 (MTK_PIN_NO(70) | 0) + +#define MT2712_PIN_71_GBE_TXD3__FUNC_GPIO71 (MTK_PIN_NO(71) | 0) +#define MT2712_PIN_71_GBE_TXD3__FUNC_GBE_TXD3 (MTK_PIN_NO(71) | 1) +#define MT2712_PIN_71_GBE_TXD3__FUNC_DBG_MON_B_0_ (MTK_PIN_NO(71) | 7) + +#define MT2712_PIN_72_GBE_TXD2__FUNC_GPIO72 (MTK_PIN_NO(72) | 0) +#define MT2712_PIN_72_GBE_TXD2__FUNC_GBE_TXD2 (MTK_PIN_NO(72) | 1) +#define MT2712_PIN_72_GBE_TXD2__FUNC_DBG_MON_B_1_ (MTK_PIN_NO(72) | 7) + +#define MT2712_PIN_73_GBE_TXD1__FUNC_GPIO73 (MTK_PIN_NO(73) | 0) +#define MT2712_PIN_73_GBE_TXD1__FUNC_GBE_TXD1 (MTK_PIN_NO(73) | 1) +#define MT2712_PIN_73_GBE_TXD1__FUNC_DBG_MON_B_2_ (MTK_PIN_NO(73) | 7) + +#define MT2712_PIN_74_GBE_TXD0__FUNC_GPIO74 (MTK_PIN_NO(74) | 0) +#define MT2712_PIN_74_GBE_TXD0__FUNC_GBE_TXD0 (MTK_PIN_NO(74) | 1) +#define MT2712_PIN_74_GBE_TXD0__FUNC_DBG_MON_B_3_ (MTK_PIN_NO(74) | 7) + +#define MT2712_PIN_75_GBE_TXC__FUNC_GPIO75 (MTK_PIN_NO(75) | 0) +#define MT2712_PIN_75_GBE_TXC__FUNC_GBE_TXC (MTK_PIN_NO(75) | 1) +#define MT2712_PIN_75_GBE_TXC__FUNC_DBG_MON_B_4_ (MTK_PIN_NO(75) | 7) + +#define MT2712_PIN_76_GBE_TXEN__FUNC_GPIO76 (MTK_PIN_NO(76) | 0) +#define MT2712_PIN_76_GBE_TXEN__FUNC_GBE_TXEN (MTK_PIN_NO(76) | 1) +#define MT2712_PIN_76_GBE_TXEN__FUNC_DBG_MON_B_5_ (MTK_PIN_NO(76) | 7) + +#define MT2712_PIN_77_GBE_TXER__FUNC_GPIO77 (MTK_PIN_NO(77) | 0) +#define MT2712_PIN_77_GBE_TXER__FUNC_GBE_TXER (MTK_PIN_NO(77) | 1) +#define MT2712_PIN_77_GBE_TXER__FUNC_DBG_MON_B_6_ (MTK_PIN_NO(77) | 7) + +#define MT2712_PIN_78_GBE_RXD3__FUNC_GPIO78 (MTK_PIN_NO(78) | 0) +#define MT2712_PIN_78_GBE_RXD3__FUNC_GBE_RXD3 (MTK_PIN_NO(78) | 1) +#define MT2712_PIN_78_GBE_RXD3__FUNC_DBG_MON_B_7_ (MTK_PIN_NO(78) | 7) + +#define MT2712_PIN_79_GBE_RXD2__FUNC_GPIO79 (MTK_PIN_NO(79) | 0) +#define MT2712_PIN_79_GBE_RXD2__FUNC_GBE_RXD2 (MTK_PIN_NO(79) | 1) +#define MT2712_PIN_79_GBE_RXD2__FUNC_DBG_MON_B_8_ (MTK_PIN_NO(79) | 7) + +#define MT2712_PIN_80_GBE_RXD1__FUNC_GPIO80 (MTK_PIN_NO(80) | 0) +#define MT2712_PIN_80_GBE_RXD1__FUNC_GBE_RXD1 (MTK_PIN_NO(80) | 1) +#define MT2712_PIN_80_GBE_RXD1__FUNC_DBG_MON_B_9_ (MTK_PIN_NO(80) | 7) + +#define MT2712_PIN_81_GBE_RXD0__FUNC_GPIO81 (MTK_PIN_NO(81) | 0) +#define MT2712_PIN_81_GBE_RXD0__FUNC_GBE_RXD0 (MTK_PIN_NO(81) | 1) +#define MT2712_PIN_81_GBE_RXD0__FUNC_DBG_MON_B_10_ (MTK_PIN_NO(81) | 7) + +#define MT2712_PIN_82_GBE_RXDV__FUNC_GPIO82 (MTK_PIN_NO(82) | 0) +#define MT2712_PIN_82_GBE_RXDV__FUNC_GBE_RXDV (MTK_PIN_NO(82) | 1) +#define MT2712_PIN_82_GBE_RXDV__FUNC_DBG_MON_B_11_ (MTK_PIN_NO(82) | 7) + +#define MT2712_PIN_83_GBE_RXER__FUNC_GPIO83 (MTK_PIN_NO(83) | 0) +#define MT2712_PIN_83_GBE_RXER__FUNC_GBE_RXER (MTK_PIN_NO(83) | 1) +#define MT2712_PIN_83_GBE_RXER__FUNC_DBG_MON_B_12_ (MTK_PIN_NO(83) | 7) + +#define MT2712_PIN_84_GBE_RXC__FUNC_GPIO84 (MTK_PIN_NO(84) | 0) +#define MT2712_PIN_84_GBE_RXC__FUNC_GBE_RXC (MTK_PIN_NO(84) | 1) +#define MT2712_PIN_84_GBE_RXC__FUNC_DBG_MON_B_13_ (MTK_PIN_NO(84) | 7) + +#define MT2712_PIN_85_GBE_MDC__FUNC_GPIO85 (MTK_PIN_NO(85) | 0) +#define MT2712_PIN_85_GBE_MDC__FUNC_GBE_MDC (MTK_PIN_NO(85) | 1) +#define MT2712_PIN_85_GBE_MDC__FUNC_DBG_MON_B_14_ (MTK_PIN_NO(85) | 7) + +#define MT2712_PIN_86_GBE_MDIO__FUNC_GPIO86 (MTK_PIN_NO(86) | 0) +#define MT2712_PIN_86_GBE_MDIO__FUNC_GBE_MDIO (MTK_PIN_NO(86) | 1) +#define MT2712_PIN_86_GBE_MDIO__FUNC_DBG_MON_B_15_ (MTK_PIN_NO(86) | 7) + +#define MT2712_PIN_87_GBE_COL__FUNC_GPIO87 (MTK_PIN_NO(87) | 0) +#define MT2712_PIN_87_GBE_COL__FUNC_GBE_COL (MTK_PIN_NO(87) | 1) +#define MT2712_PIN_87_GBE_COL__FUNC_DBG_MON_B_16_ (MTK_PIN_NO(87) | 7) + +#define MT2712_PIN_88_GBE_INTR__FUNC_GPIO88 (MTK_PIN_NO(88) | 0) +#define MT2712_PIN_88_GBE_INTR__FUNC_GBE_INTR (MTK_PIN_NO(88) | 1) +#define MT2712_PIN_88_GBE_INTR__FUNC_GBE_CRS (MTK_PIN_NO(88) | 2) +#define MT2712_PIN_88_GBE_INTR__FUNC_DBG_MON_B_17_ (MTK_PIN_NO(88) | 7) + +#define MT2712_PIN_89_MSDC2_CLK__FUNC_GPIO89 (MTK_PIN_NO(89) | 0) +#define MT2712_PIN_89_MSDC2_CLK__FUNC_MSDC2_CLK (MTK_PIN_NO(89) | 1) +#define MT2712_PIN_89_MSDC2_CLK__FUNC_DBG_MON_B_18_ (MTK_PIN_NO(89) | 7) + +#define MT2712_PIN_90_MSDC2_DAT3__FUNC_GPIO90 (MTK_PIN_NO(90) | 0) +#define MT2712_PIN_90_MSDC2_DAT3__FUNC_MSDC2_DAT3 (MTK_PIN_NO(90) | 1) +#define MT2712_PIN_90_MSDC2_DAT3__FUNC_DBG_MON_B_19_ (MTK_PIN_NO(90) | 7) + +#define MT2712_PIN_91_MSDC2_DAT2__FUNC_GPIO91 (MTK_PIN_NO(91) | 0) +#define MT2712_PIN_91_MSDC2_DAT2__FUNC_MSDC2_DAT2 (MTK_PIN_NO(91) | 1) +#define MT2712_PIN_91_MSDC2_DAT2__FUNC_DBG_MON_B_20_ (MTK_PIN_NO(91) | 7) + +#define MT2712_PIN_92_MSDC2_DAT1__FUNC_GPIO92 (MTK_PIN_NO(92) | 0) +#define MT2712_PIN_92_MSDC2_DAT1__FUNC_MSDC2_DAT1 (MTK_PIN_NO(92) | 1) +#define MT2712_PIN_92_MSDC2_DAT1__FUNC_DBG_MON_B_21_ (MTK_PIN_NO(92) | 7) + +#define MT2712_PIN_93_MSDC2_DAT0__FUNC_GPIO93 (MTK_PIN_NO(93) | 0) +#define MT2712_PIN_93_MSDC2_DAT0__FUNC_MSDC2_DAT0 (MTK_PIN_NO(93) | 1) +#define MT2712_PIN_93_MSDC2_DAT0__FUNC_DBG_MON_B_22_ (MTK_PIN_NO(93) | 7) + +#define MT2712_PIN_94_MSDC2_INS__FUNC_GPIO94 (MTK_PIN_NO(94) | 0) +#define MT2712_PIN_94_MSDC2_INS__FUNC_DBG_MON_B_23_ (MTK_PIN_NO(94) | 7) + +#define MT2712_PIN_95_MSDC2_CMD__FUNC_GPIO95 (MTK_PIN_NO(95) | 0) +#define MT2712_PIN_95_MSDC2_CMD__FUNC_MSDC2_CMD (MTK_PIN_NO(95) | 1) +#define MT2712_PIN_95_MSDC2_CMD__FUNC_DBG_MON_B_24_ (MTK_PIN_NO(95) | 7) + +#define MT2712_PIN_96_MSDC2_PSW__FUNC_GPIO96 (MTK_PIN_NO(96) | 0) +#define MT2712_PIN_96_MSDC2_PSW__FUNC_DBG_MON_B_25_ (MTK_PIN_NO(96) | 7) + +#define MT2712_PIN_97_URXD4__FUNC_GPIO97 (MTK_PIN_NO(97) | 0) +#define MT2712_PIN_97_URXD4__FUNC_URXD4 (MTK_PIN_NO(97) | 1) +#define MT2712_PIN_97_URXD4__FUNC_UTXD4 (MTK_PIN_NO(97) | 2) +#define MT2712_PIN_97_URXD4__FUNC_MRG_CLK (MTK_PIN_NO(97) | 3) +#define MT2712_PIN_97_URXD4__FUNC_PCM1_CLK (MTK_PIN_NO(97) | 4) +#define MT2712_PIN_97_URXD4__FUNC_I2S_IQ2_SDQB (MTK_PIN_NO(97) | 5) +#define MT2712_PIN_97_URXD4__FUNC_I2SO1_WS (MTK_PIN_NO(97) | 6) +#define MT2712_PIN_97_URXD4__FUNC_DBG_MON_B_26_ (MTK_PIN_NO(97) | 7) + +#define MT2712_PIN_98_URTS4__FUNC_GPIO98 (MTK_PIN_NO(98) | 0) +#define MT2712_PIN_98_URTS4__FUNC_URTS4 (MTK_PIN_NO(98) | 1) +#define MT2712_PIN_98_URTS4__FUNC_UCTS4 (MTK_PIN_NO(98) | 2) +#define MT2712_PIN_98_URTS4__FUNC_MRG_RX (MTK_PIN_NO(98) | 3) +#define MT2712_PIN_98_URTS4__FUNC_PCM1_DI (MTK_PIN_NO(98) | 4) +#define MT2712_PIN_98_URTS4__FUNC_I2S_IQ1_SDIB (MTK_PIN_NO(98) | 5) +#define MT2712_PIN_98_URTS4__FUNC_I2SO1_MCK (MTK_PIN_NO(98) | 6) +#define MT2712_PIN_98_URTS4__FUNC_DBG_MON_B_27_ (MTK_PIN_NO(98) | 7) + +#define MT2712_PIN_99_UTXD4__FUNC_GPIO99 (MTK_PIN_NO(99) | 0) +#define MT2712_PIN_99_UTXD4__FUNC_UTXD4 (MTK_PIN_NO(99) | 1) +#define MT2712_PIN_99_UTXD4__FUNC_URXD4 (MTK_PIN_NO(99) | 2) +#define MT2712_PIN_99_UTXD4__FUNC_MRG_SYNC (MTK_PIN_NO(99) | 3) +#define MT2712_PIN_99_UTXD4__FUNC_PCM1_SYNC (MTK_PIN_NO(99) | 4) +#define MT2712_PIN_99_UTXD4__FUNC_I2S_IQ0_SDQB (MTK_PIN_NO(99) | 5) +#define MT2712_PIN_99_UTXD4__FUNC_I2SO1_BCK (MTK_PIN_NO(99) | 6) +#define MT2712_PIN_99_UTXD4__FUNC_DBG_MON_B_28_ (MTK_PIN_NO(99) | 7) + +#define MT2712_PIN_100_UCTS4__FUNC_GPIO100 (MTK_PIN_NO(100) | 0) +#define MT2712_PIN_100_UCTS4__FUNC_UCTS4 (MTK_PIN_NO(100) | 1) +#define MT2712_PIN_100_UCTS4__FUNC_URTS4 (MTK_PIN_NO(100) | 2) +#define MT2712_PIN_100_UCTS4__FUNC_MRG_TX (MTK_PIN_NO(100) | 3) +#define MT2712_PIN_100_UCTS4__FUNC_PCM1_DO (MTK_PIN_NO(100) | 4) +#define MT2712_PIN_100_UCTS4__FUNC_I2S_IQ0_SDIB (MTK_PIN_NO(100) | 5) +#define MT2712_PIN_100_UCTS4__FUNC_I2SO1_DO (MTK_PIN_NO(100) | 6) +#define MT2712_PIN_100_UCTS4__FUNC_DBG_MON_B_29_ (MTK_PIN_NO(100) | 7) + +#define MT2712_PIN_101_URXD5__FUNC_GPIO101 (MTK_PIN_NO(101) | 0) +#define MT2712_PIN_101_URXD5__FUNC_URXD5 (MTK_PIN_NO(101) | 1) +#define MT2712_PIN_101_URXD5__FUNC_UTXD5 (MTK_PIN_NO(101) | 2) +#define MT2712_PIN_101_URXD5__FUNC_I2SO3_WS (MTK_PIN_NO(101) | 3) +#define MT2712_PIN_101_URXD5__FUNC_TDMIN_LRCK (MTK_PIN_NO(101) | 4) +#define MT2712_PIN_101_URXD5__FUNC_I2SO0_WS (MTK_PIN_NO(101) | 6) +#define MT2712_PIN_101_URXD5__FUNC_DBG_MON_B_30_ (MTK_PIN_NO(101) | 7) + +#define MT2712_PIN_102_URTS5__FUNC_GPIO102 (MTK_PIN_NO(102) | 0) +#define MT2712_PIN_102_URTS5__FUNC_URTS5 (MTK_PIN_NO(102) | 1) +#define MT2712_PIN_102_URTS5__FUNC_UCTS5 (MTK_PIN_NO(102) | 2) +#define MT2712_PIN_102_URTS5__FUNC_I2SO3_MCK (MTK_PIN_NO(102) | 3) +#define MT2712_PIN_102_URTS5__FUNC_TDMIN_MCLK (MTK_PIN_NO(102) | 4) +#define MT2712_PIN_102_URTS5__FUNC_IR_IN (MTK_PIN_NO(102) | 5) +#define MT2712_PIN_102_URTS5__FUNC_I2SO0_MCK (MTK_PIN_NO(102) | 6) +#define MT2712_PIN_102_URTS5__FUNC_DBG_MON_B_31_ (MTK_PIN_NO(102) | 7) + +#define MT2712_PIN_103_UTXD5__FUNC_GPIO103 (MTK_PIN_NO(103) | 0) +#define MT2712_PIN_103_UTXD5__FUNC_UTXD5 (MTK_PIN_NO(103) | 1) +#define MT2712_PIN_103_UTXD5__FUNC_URXD5 (MTK_PIN_NO(103) | 2) +#define MT2712_PIN_103_UTXD5__FUNC_I2SO3_BCK (MTK_PIN_NO(103) | 3) +#define MT2712_PIN_103_UTXD5__FUNC_TDMIN_BCK (MTK_PIN_NO(103) | 4) +#define MT2712_PIN_103_UTXD5__FUNC_I2SO0_BCK (MTK_PIN_NO(103) | 6) +#define MT2712_PIN_103_UTXD5__FUNC_DBG_MON_B_32_ (MTK_PIN_NO(103) | 7) + +#define MT2712_PIN_104_UCTS5__FUNC_GPIO104 (MTK_PIN_NO(104) | 0) +#define MT2712_PIN_104_UCTS5__FUNC_UCTS5 (MTK_PIN_NO(104) | 1) +#define MT2712_PIN_104_UCTS5__FUNC_URTS5 (MTK_PIN_NO(104) | 2) +#define MT2712_PIN_104_UCTS5__FUNC_I2SO0_DO1 (MTK_PIN_NO(104) | 3) +#define MT2712_PIN_104_UCTS5__FUNC_TDMIN_DI (MTK_PIN_NO(104) | 4) +#define MT2712_PIN_104_UCTS5__FUNC_IR_IN (MTK_PIN_NO(104) | 5) +#define MT2712_PIN_104_UCTS5__FUNC_I2SO0_DO0 (MTK_PIN_NO(104) | 6) + +#define MT2712_PIN_105_I2C_SDA0__FUNC_GPIO105 (MTK_PIN_NO(105) | 0) +#define MT2712_PIN_105_I2C_SDA0__FUNC_SDA0 (MTK_PIN_NO(105) | 1) + +#define MT2712_PIN_106_I2C_SDA1__FUNC_GPIO106 (MTK_PIN_NO(106) | 0) +#define MT2712_PIN_106_I2C_SDA1__FUNC_SDA1 (MTK_PIN_NO(106) | 1) + +#define MT2712_PIN_107_I2C_SDA2__FUNC_GPIO107 (MTK_PIN_NO(107) | 0) +#define MT2712_PIN_107_I2C_SDA2__FUNC_SDA2 (MTK_PIN_NO(107) | 1) + +#define MT2712_PIN_108_I2C_SDA3__FUNC_GPIO108 (MTK_PIN_NO(108) | 0) +#define MT2712_PIN_108_I2C_SDA3__FUNC_SDA3 (MTK_PIN_NO(108) | 1) + +#define MT2712_PIN_109_I2C_SDA4__FUNC_GPIO109 (MTK_PIN_NO(109) | 0) +#define MT2712_PIN_109_I2C_SDA4__FUNC_SDA4 (MTK_PIN_NO(109) | 1) + +#define MT2712_PIN_110_I2C_SDA5__FUNC_GPIO110 (MTK_PIN_NO(110) | 0) +#define MT2712_PIN_110_I2C_SDA5__FUNC_SDA5 (MTK_PIN_NO(110) | 1) + +#define MT2712_PIN_111_I2C_SCL0__FUNC_GPIO111 (MTK_PIN_NO(111) | 0) +#define MT2712_PIN_111_I2C_SCL0__FUNC_SCL0 (MTK_PIN_NO(111) | 1) + +#define MT2712_PIN_112_I2C_SCL1__FUNC_GPIO112 (MTK_PIN_NO(112) | 0) +#define MT2712_PIN_112_I2C_SCL1__FUNC_SCL1 (MTK_PIN_NO(112) | 1) + +#define MT2712_PIN_113_I2C_SCL2__FUNC_GPIO113 (MTK_PIN_NO(113) | 0) +#define MT2712_PIN_113_I2C_SCL2__FUNC_SCL2 (MTK_PIN_NO(113) | 1) + +#define MT2712_PIN_114_I2C_SCL3__FUNC_GPIO114 (MTK_PIN_NO(114) | 0) +#define MT2712_PIN_114_I2C_SCL3__FUNC_SCL3 (MTK_PIN_NO(114) | 1) + +#define MT2712_PIN_115_I2C_SCL4__FUNC_GPIO115 (MTK_PIN_NO(115) | 0) +#define MT2712_PIN_115_I2C_SCL4__FUNC_SCL4 (MTK_PIN_NO(115) | 1) + +#define MT2712_PIN_116_I2C_SCL5__FUNC_GPIO116 (MTK_PIN_NO(116) | 0) +#define MT2712_PIN_116_I2C_SCL5__FUNC_SCL5 (MTK_PIN_NO(116) | 1) + +#define MT2712_PIN_117_URXD0__FUNC_GPIO117 (MTK_PIN_NO(117) | 0) +#define MT2712_PIN_117_URXD0__FUNC_URXD0 (MTK_PIN_NO(117) | 1) +#define MT2712_PIN_117_URXD0__FUNC_UTXD0 (MTK_PIN_NO(117) | 2) + +#define MT2712_PIN_118_URXD1__FUNC_GPIO118 (MTK_PIN_NO(118) | 0) +#define MT2712_PIN_118_URXD1__FUNC_URXD1 (MTK_PIN_NO(118) | 1) +#define MT2712_PIN_118_URXD1__FUNC_UTXD1 (MTK_PIN_NO(118) | 2) + +#define MT2712_PIN_119_URXD2__FUNC_GPIO119 (MTK_PIN_NO(119) | 0) +#define MT2712_PIN_119_URXD2__FUNC_URXD2 (MTK_PIN_NO(119) | 1) +#define MT2712_PIN_119_URXD2__FUNC_UTXD2 (MTK_PIN_NO(119) | 2) + +#define MT2712_PIN_120_UTXD0__FUNC_GPIO120 (MTK_PIN_NO(120) | 0) +#define MT2712_PIN_120_UTXD0__FUNC_UTXD0 (MTK_PIN_NO(120) | 1) +#define MT2712_PIN_120_UTXD0__FUNC_URXD0 (MTK_PIN_NO(120) | 2) + +#define MT2712_PIN_121_UTXD1__FUNC_GPIO121 (MTK_PIN_NO(121) | 0) +#define MT2712_PIN_121_UTXD1__FUNC_UTXD1 (MTK_PIN_NO(121) | 1) +#define MT2712_PIN_121_UTXD1__FUNC_URXD1 (MTK_PIN_NO(121) | 2) + +#define MT2712_PIN_122_UTXD2__FUNC_GPIO122 (MTK_PIN_NO(122) | 0) +#define MT2712_PIN_122_UTXD2__FUNC_UTXD2 (MTK_PIN_NO(122) | 1) +#define MT2712_PIN_122_UTXD2__FUNC_URXD2 (MTK_PIN_NO(122) | 2) + +#define MT2712_PIN_123_URXD3__FUNC_GPIO123 (MTK_PIN_NO(123) | 0) +#define MT2712_PIN_123_URXD3__FUNC_URXD3 (MTK_PIN_NO(123) | 1) +#define MT2712_PIN_123_URXD3__FUNC_UTXD3 (MTK_PIN_NO(123) | 2) +#define MT2712_PIN_123_URXD3__FUNC_PURE_HW_PROTECT (MTK_PIN_NO(123) | 3) + +#define MT2712_PIN_124_UTXD3__FUNC_GPIO124 (MTK_PIN_NO(124) | 0) +#define MT2712_PIN_124_UTXD3__FUNC_UTXD3 (MTK_PIN_NO(124) | 1) +#define MT2712_PIN_124_UTXD3__FUNC_URXD3 (MTK_PIN_NO(124) | 2) +#define MT2712_PIN_124_UTXD3__FUNC_PURE_HW_PROTECT (MTK_PIN_NO(124) | 3) + +#define MT2712_PIN_125_URTS3__FUNC_GPIO125 (MTK_PIN_NO(125) | 0) +#define MT2712_PIN_125_URTS3__FUNC_URTS3 (MTK_PIN_NO(125) | 1) +#define MT2712_PIN_125_URTS3__FUNC_UCTS3 (MTK_PIN_NO(125) | 2) +#define MT2712_PIN_125_URTS3__FUNC_WATCH_DOG (MTK_PIN_NO(125) | 3) + +#define MT2712_PIN_126_UCTS3__FUNC_GPIO126 (MTK_PIN_NO(126) | 0) +#define MT2712_PIN_126_UCTS3__FUNC_UCTS3 (MTK_PIN_NO(126) | 1) +#define MT2712_PIN_126_UCTS3__FUNC_URTS3 (MTK_PIN_NO(126) | 2) +#define MT2712_PIN_126_UCTS3__FUNC_SRCLKENA0 (MTK_PIN_NO(126) | 3) + +#define MT2712_PIN_127_SPI2_CSN__FUNC_GPIO127 (MTK_PIN_NO(127) | 0) +#define MT2712_PIN_127_SPI2_CSN__FUNC_SPI_CS_2_ (MTK_PIN_NO(127) | 1) +#define MT2712_PIN_127_SPI2_CSN__FUNC_SPI_CS_1_ (MTK_PIN_NO(127) | 2) + +#define MT2712_PIN_128_SPI2_MO__FUNC_GPIO128 (MTK_PIN_NO(128) | 0) +#define MT2712_PIN_128_SPI2_MO__FUNC_SPI_MO_2_ (MTK_PIN_NO(128) | 1) +#define MT2712_PIN_128_SPI2_MO__FUNC_SPI_SO_1_ (MTK_PIN_NO(128) | 2) + +#define MT2712_PIN_129_SPI2_MI__FUNC_GPIO129 (MTK_PIN_NO(129) | 0) +#define MT2712_PIN_129_SPI2_MI__FUNC_SPI_MI_2_ (MTK_PIN_NO(129) | 1) +#define MT2712_PIN_129_SPI2_MI__FUNC_SPI_SI_1_ (MTK_PIN_NO(129) | 2) + +#define MT2712_PIN_130_SPI2_CK__FUNC_GPIO130 (MTK_PIN_NO(130) | 0) +#define MT2712_PIN_130_SPI2_CK__FUNC_SPI_CK_2_ (MTK_PIN_NO(130) | 1) +#define MT2712_PIN_130_SPI2_CK__FUNC_SPI_CK_1_ (MTK_PIN_NO(130) | 2) + +#define MT2712_PIN_131_SPI3_CSN__FUNC_GPIO131 (MTK_PIN_NO(131) | 0) +#define MT2712_PIN_131_SPI3_CSN__FUNC_SPI_CS_3_ (MTK_PIN_NO(131) | 1) + +#define MT2712_PIN_132_SPI3_MO__FUNC_GPIO132 (MTK_PIN_NO(132) | 0) +#define MT2712_PIN_132_SPI3_MO__FUNC_SPI_MO_3_ (MTK_PIN_NO(132) | 1) + +#define MT2712_PIN_133_SPI3_MI__FUNC_GPIO133 (MTK_PIN_NO(133) | 0) +#define MT2712_PIN_133_SPI3_MI__FUNC_SPI_MI_3_ (MTK_PIN_NO(133) | 1) + +#define MT2712_PIN_134_SPI3_CK__FUNC_GPIO134 (MTK_PIN_NO(134) | 0) +#define MT2712_PIN_134_SPI3_CK__FUNC_SPI_CK_3_ (MTK_PIN_NO(134) | 1) + +#define MT2712_PIN_135_KPROW3__FUNC_GPIO135 (MTK_PIN_NO(135) | 0) +#define MT2712_PIN_135_KPROW3__FUNC_KROW3 (MTK_PIN_NO(135) | 1) +#define MT2712_PIN_135_KPROW3__FUNC_DSIC_TE (MTK_PIN_NO(135) | 2) + +#define MT2712_PIN_136_KPROW4__FUNC_GPIO136 (MTK_PIN_NO(136) | 0) +#define MT2712_PIN_136_KPROW4__FUNC_KROW4 (MTK_PIN_NO(136) | 1) +#define MT2712_PIN_136_KPROW4__FUNC_DSID_TE (MTK_PIN_NO(136) | 2) + +#define MT2712_PIN_137_KPCOL3__FUNC_GPIO137 (MTK_PIN_NO(137) | 0) +#define MT2712_PIN_137_KPCOL3__FUNC_KCOL3 (MTK_PIN_NO(137) | 1) +#define MT2712_PIN_137_KPCOL3__FUNC_DISP2_PWM (MTK_PIN_NO(137) | 2) + +#define MT2712_PIN_138_KPCOL4__FUNC_GPIO138 (MTK_PIN_NO(138) | 0) +#define MT2712_PIN_138_KPCOL4__FUNC_KCOL4 (MTK_PIN_NO(138) | 1) +#define MT2712_PIN_138_KPCOL4__FUNC_LCM_RST2 (MTK_PIN_NO(138) | 2) + +#define MT2712_PIN_139_KPCOL5__FUNC_GPIO139 (MTK_PIN_NO(139) | 0) +#define MT2712_PIN_139_KPCOL5__FUNC_KCOL5 (MTK_PIN_NO(139) | 1) +#define MT2712_PIN_139_KPCOL5__FUNC_DSIA_TE (MTK_PIN_NO(139) | 3) +#define MT2712_PIN_139_KPCOL5__FUNC_PURE_HW_PROTECT (MTK_PIN_NO(139) | 4) + +#define MT2712_PIN_140_KPCOL6__FUNC_GPIO140 (MTK_PIN_NO(140) | 0) +#define MT2712_PIN_140_KPCOL6__FUNC_KCOL6 (MTK_PIN_NO(140) | 1) +#define MT2712_PIN_140_KPCOL6__FUNC_WATCH_DOG (MTK_PIN_NO(140) | 2) +#define MT2712_PIN_140_KPCOL6__FUNC_LCM_RST1 (MTK_PIN_NO(140) | 3) + +#define MT2712_PIN_141_KPROW5__FUNC_GPIO141 (MTK_PIN_NO(141) | 0) +#define MT2712_PIN_141_KPROW5__FUNC_KROW5 (MTK_PIN_NO(141) | 1) +#define MT2712_PIN_141_KPROW5__FUNC_LCM_RST0 (MTK_PIN_NO(141) | 3) +#define MT2712_PIN_141_KPROW5__FUNC_PURE_HW_PROTECT (MTK_PIN_NO(141) | 4) + +#define MT2712_PIN_142_KPROW6__FUNC_GPIO142 (MTK_PIN_NO(142) | 0) +#define MT2712_PIN_142_KPROW6__FUNC_KROW6 (MTK_PIN_NO(142) | 1) +#define MT2712_PIN_142_KPROW6__FUNC_SRCLKENA0 (MTK_PIN_NO(142) | 2) +#define MT2712_PIN_142_KPROW6__FUNC_DSIB_TE (MTK_PIN_NO(142) | 3) + +#define MT2712_PIN_143_JTDO_ICE__FUNC_GPIO143 (MTK_PIN_NO(143) | 0) +#define MT2712_PIN_143_JTDO_ICE__FUNC_JTDO_ICE (MTK_PIN_NO(143) | 1) +#define MT2712_PIN_143_JTDO_ICE__FUNC_DFD_TDO (MTK_PIN_NO(143) | 3) + +#define MT2712_PIN_144_JTCK_ICE__FUNC_GPIO144 (MTK_PIN_NO(144) | 0) +#define MT2712_PIN_144_JTCK_ICE__FUNC_JTCK_ICE (MTK_PIN_NO(144) | 1) +#define MT2712_PIN_144_JTCK_ICE__FUNC_DFD_TCK (MTK_PIN_NO(144) | 3) + +#define MT2712_PIN_145_JTDI_ICE__FUNC_GPIO145 (MTK_PIN_NO(145) | 0) +#define MT2712_PIN_145_JTDI_ICE__FUNC_JTDI_ICE (MTK_PIN_NO(145) | 1) +#define MT2712_PIN_145_JTDI_ICE__FUNC_DFD_TDI (MTK_PIN_NO(145) | 3) + +#define MT2712_PIN_146_JTMS_ICE__FUNC_GPIO146 (MTK_PIN_NO(146) | 0) +#define MT2712_PIN_146_JTMS_ICE__FUNC_JTMS_ICE (MTK_PIN_NO(146) | 1) +#define MT2712_PIN_146_JTMS_ICE__FUNC_DFD_TMS (MTK_PIN_NO(146) | 3) + +#define MT2712_PIN_147_JTRSTB_ICE__FUNC_GPIO147 (MTK_PIN_NO(147) | 0) +#define MT2712_PIN_147_JTRSTB_ICE__FUNC_JTRST_B_ICE (MTK_PIN_NO(147) | 1) +#define MT2712_PIN_147_JTRSTB_ICE__FUNC_DFD_NTRST (MTK_PIN_NO(147) | 3) + +#define MT2712_PIN_148_GPIO148__FUNC_GPIO148 (MTK_PIN_NO(148) | 0) +#define MT2712_PIN_148_GPIO148__FUNC_JTRSTB_CM4 (MTK_PIN_NO(148) | 1) +#define MT2712_PIN_148_GPIO148__FUNC_DFD_NTRST (MTK_PIN_NO(148) | 3) + +#define MT2712_PIN_149_GPIO149__FUNC_GPIO149 (MTK_PIN_NO(149) | 0) +#define MT2712_PIN_149_GPIO149__FUNC_JTCK_CM4 (MTK_PIN_NO(149) | 1) +#define MT2712_PIN_149_GPIO149__FUNC_DFD_TCK (MTK_PIN_NO(149) | 3) + +#define MT2712_PIN_150_GPIO150__FUNC_GPIO150 (MTK_PIN_NO(150) | 0) +#define MT2712_PIN_150_GPIO150__FUNC_JTMS_CM4 (MTK_PIN_NO(150) | 1) +#define MT2712_PIN_150_GPIO150__FUNC_DFD_TMS (MTK_PIN_NO(150) | 3) + +#define MT2712_PIN_151_GPIO151__FUNC_GPIO151 (MTK_PIN_NO(151) | 0) +#define MT2712_PIN_151_GPIO151__FUNC_JTDI_CM4 (MTK_PIN_NO(151) | 1) +#define MT2712_PIN_151_GPIO151__FUNC_DFD_TDI (MTK_PIN_NO(151) | 3) + +#define MT2712_PIN_152_GPIO152__FUNC_GPIO152 (MTK_PIN_NO(152) | 0) +#define MT2712_PIN_152_GPIO152__FUNC_JTDO_CM4 (MTK_PIN_NO(152) | 1) +#define MT2712_PIN_152_GPIO152__FUNC_DFD_TDO (MTK_PIN_NO(152) | 3) + +#define MT2712_PIN_153_SPI0_CSN__FUNC_GPIO153 (MTK_PIN_NO(153) | 0) +#define MT2712_PIN_153_SPI0_CSN__FUNC_SPI_CS_0_ (MTK_PIN_NO(153) | 1) +#define MT2712_PIN_153_SPI0_CSN__FUNC_SRCLKENA0 (MTK_PIN_NO(153) | 2) +#define MT2712_PIN_153_SPI0_CSN__FUNC_UTXD0 (MTK_PIN_NO(153) | 3) +#define MT2712_PIN_153_SPI0_CSN__FUNC_I2SO0_DO1 (MTK_PIN_NO(153) | 4) +#define MT2712_PIN_153_SPI0_CSN__FUNC_TDMO0_DATA1 (MTK_PIN_NO(153) | 6) +#define MT2712_PIN_153_SPI0_CSN__FUNC_I2S_IQ2_SDQB (MTK_PIN_NO(153) | 7) + +#define MT2712_PIN_154_SPI0_MI__FUNC_GPIO154 (MTK_PIN_NO(154) | 0) +#define MT2712_PIN_154_SPI0_MI__FUNC_SPI_MI_0_ (MTK_PIN_NO(154) | 1) +#define MT2712_PIN_154_SPI0_MI__FUNC_SRCLKENA0 (MTK_PIN_NO(154) | 2) +#define MT2712_PIN_154_SPI0_MI__FUNC_URXD0 (MTK_PIN_NO(154) | 3) +#define MT2712_PIN_154_SPI0_MI__FUNC_I2SO0_DO0 (MTK_PIN_NO(154) | 4) +#define MT2712_PIN_154_SPI0_MI__FUNC_I2SO1_DO (MTK_PIN_NO(154) | 5) +#define MT2712_PIN_154_SPI0_MI__FUNC_TDMO0_DATA (MTK_PIN_NO(154) | 6) +#define MT2712_PIN_154_SPI0_MI__FUNC_I2S_IQ1_SDIB (MTK_PIN_NO(154) | 7) + +#define MT2712_PIN_155_SPI0_CK__FUNC_GPIO155 (MTK_PIN_NO(155) | 0) +#define MT2712_PIN_155_SPI0_CK__FUNC_SPI_CK_0_ (MTK_PIN_NO(155) | 1) +#define MT2712_PIN_155_SPI0_CK__FUNC_SC_APBIAS_OFF (MTK_PIN_NO(155) | 2) +#define MT2712_PIN_155_SPI0_CK__FUNC_UTXD1 (MTK_PIN_NO(155) | 3) +#define MT2712_PIN_155_SPI0_CK__FUNC_I2SO0_BCK (MTK_PIN_NO(155) | 4) +#define MT2712_PIN_155_SPI0_CK__FUNC_I2SO1_BCK (MTK_PIN_NO(155) | 5) +#define MT2712_PIN_155_SPI0_CK__FUNC_TDMO0_BCK (MTK_PIN_NO(155) | 6) +#define MT2712_PIN_155_SPI0_CK__FUNC_I2S_IQ0_SDQB (MTK_PIN_NO(155) | 7) + +#define MT2712_PIN_156_SPI0_MO__FUNC_GPIO156 (MTK_PIN_NO(156) | 0) +#define MT2712_PIN_156_SPI0_MO__FUNC_SPI_MO_0_ (MTK_PIN_NO(156) | 1) +#define MT2712_PIN_156_SPI0_MO__FUNC_SC_APBIAS_OFF (MTK_PIN_NO(156) | 2) +#define MT2712_PIN_156_SPI0_MO__FUNC_URXD1 (MTK_PIN_NO(156) | 3) +#define MT2712_PIN_156_SPI0_MO__FUNC_I2SO0_WS (MTK_PIN_NO(156) | 4) +#define MT2712_PIN_156_SPI0_MO__FUNC_I2SO1_WS (MTK_PIN_NO(156) | 5) +#define MT2712_PIN_156_SPI0_MO__FUNC_TDMO0_LRCK (MTK_PIN_NO(156) | 6) +#define MT2712_PIN_156_SPI0_MO__FUNC_I2S_IQ0_SDIB (MTK_PIN_NO(156) | 7) + +#define MT2712_PIN_157_SPI5_CSN__FUNC_GPIO157 (MTK_PIN_NO(157) | 0) +#define MT2712_PIN_157_SPI5_CSN__FUNC_SPI_CS_5_ (MTK_PIN_NO(157) | 1) +#define MT2712_PIN_157_SPI5_CSN__FUNC_LCM_RST0 (MTK_PIN_NO(157) | 2) +#define MT2712_PIN_157_SPI5_CSN__FUNC_UTXD2 (MTK_PIN_NO(157) | 3) +#define MT2712_PIN_157_SPI5_CSN__FUNC_I2SO0_MCK (MTK_PIN_NO(157) | 4) +#define MT2712_PIN_157_SPI5_CSN__FUNC_I2SO1_MCK (MTK_PIN_NO(157) | 5) +#define MT2712_PIN_157_SPI5_CSN__FUNC_TDMO0_MCLK (MTK_PIN_NO(157) | 6) + +#define MT2712_PIN_158_SPI5_MI__FUNC_GPIO158 (MTK_PIN_NO(158) | 0) +#define MT2712_PIN_158_SPI5_MI__FUNC_SPI_MI_5_ (MTK_PIN_NO(158) | 1) +#define MT2712_PIN_158_SPI5_MI__FUNC_DSIA_TE (MTK_PIN_NO(158) | 2) +#define MT2712_PIN_158_SPI5_MI__FUNC_URXD2 (MTK_PIN_NO(158) | 3) + +#define MT2712_PIN_159_SPI5_MO__FUNC_GPIO159 (MTK_PIN_NO(159) | 0) +#define MT2712_PIN_159_SPI5_MO__FUNC_SPI_MO_5_ (MTK_PIN_NO(159) | 1) +#define MT2712_PIN_159_SPI5_MO__FUNC_DSIB_TE (MTK_PIN_NO(159) | 2) +#define MT2712_PIN_159_SPI5_MO__FUNC_UTXD3 (MTK_PIN_NO(159) | 3) + +#define MT2712_PIN_160_SPI5_CK__FUNC_GPIO160 (MTK_PIN_NO(160) | 0) +#define MT2712_PIN_160_SPI5_CK__FUNC_SPI_CK_5_ (MTK_PIN_NO(160) | 1) +#define MT2712_PIN_160_SPI5_CK__FUNC_LCM_RST1 (MTK_PIN_NO(160) | 2) +#define MT2712_PIN_160_SPI5_CK__FUNC_URXD3 (MTK_PIN_NO(160) | 3) + +#define MT2712_PIN_161_SPI1_CSN__FUNC_GPIO161 (MTK_PIN_NO(161) | 0) +#define MT2712_PIN_161_SPI1_CSN__FUNC_SPI_CS_1_ (MTK_PIN_NO(161) | 1) +#define MT2712_PIN_161_SPI1_CSN__FUNC_SPI_CS_4_ (MTK_PIN_NO(161) | 2) +#define MT2712_PIN_161_SPI1_CSN__FUNC_I2S_IQ2_SDQB (MTK_PIN_NO(161) | 4) +#define MT2712_PIN_161_SPI1_CSN__FUNC_I2SO2_DO (MTK_PIN_NO(161) | 5) +#define MT2712_PIN_161_SPI1_CSN__FUNC_TDMO0_DATA1 (MTK_PIN_NO(161) | 6) +#define MT2712_PIN_161_SPI1_CSN__FUNC_I2SO0_DO1 (MTK_PIN_NO(161) | 7) + +#define MT2712_PIN_162_SPI1_SI__FUNC_GPIO162 (MTK_PIN_NO(162) | 0) +#define MT2712_PIN_162_SPI1_SI__FUNC_SPI_SI_1_ (MTK_PIN_NO(162) | 1) +#define MT2712_PIN_162_SPI1_SI__FUNC_SPI_MI_4_ (MTK_PIN_NO(162) | 2) +#define MT2712_PIN_162_SPI1_SI__FUNC_I2S_IQ1_SDIB (MTK_PIN_NO(162) | 4) +#define MT2712_PIN_162_SPI1_SI__FUNC_I2SO2_BCK (MTK_PIN_NO(162) | 5) +#define MT2712_PIN_162_SPI1_SI__FUNC_TDMO0_DATA (MTK_PIN_NO(162) | 6) +#define MT2712_PIN_162_SPI1_SI__FUNC_I2SO0_DO0 (MTK_PIN_NO(162) | 7) + +#define MT2712_PIN_163_SPI1_CK__FUNC_GPIO163 (MTK_PIN_NO(163) | 0) +#define MT2712_PIN_163_SPI1_CK__FUNC_SPI_CK_1_ (MTK_PIN_NO(163) | 1) +#define MT2712_PIN_163_SPI1_CK__FUNC_SPI_CK_4_ (MTK_PIN_NO(163) | 2) +#define MT2712_PIN_163_SPI1_CK__FUNC_I2S_IQ0_SDQB (MTK_PIN_NO(163) | 4) +#define MT2712_PIN_163_SPI1_CK__FUNC_I2SO2_WS (MTK_PIN_NO(163) | 5) +#define MT2712_PIN_163_SPI1_CK__FUNC_TDMO0_BCK (MTK_PIN_NO(163) | 6) +#define MT2712_PIN_163_SPI1_CK__FUNC_I2SO0_BCK (MTK_PIN_NO(163) | 7) + +#define MT2712_PIN_164_SPI1_SO__FUNC_GPIO164 (MTK_PIN_NO(164) | 0) +#define MT2712_PIN_164_SPI1_SO__FUNC_SPI_SO_1_ (MTK_PIN_NO(164) | 1) +#define MT2712_PIN_164_SPI1_SO__FUNC_SPI_MO_4_ (MTK_PIN_NO(164) | 2) +#define MT2712_PIN_164_SPI1_SO__FUNC_I2S_IQ0_SDIB (MTK_PIN_NO(164) | 4) +#define MT2712_PIN_164_SPI1_SO__FUNC_I2SO2_MCK (MTK_PIN_NO(164) | 5) +#define MT2712_PIN_164_SPI1_SO__FUNC_TDMO0_LRCK (MTK_PIN_NO(164) | 6) +#define MT2712_PIN_164_SPI1_SO__FUNC_I2SO0_WS (MTK_PIN_NO(164) | 7) + +#define MT2712_PIN_165_SPI4_CSN__FUNC_GPIO165 (MTK_PIN_NO(165) | 0) +#define MT2712_PIN_165_SPI4_CSN__FUNC_SPI_CS_4_ (MTK_PIN_NO(165) | 1) +#define MT2712_PIN_165_SPI4_CSN__FUNC_LCM_RST0 (MTK_PIN_NO(165) | 2) +#define MT2712_PIN_165_SPI4_CSN__FUNC_SPI_CS_1_ (MTK_PIN_NO(165) | 3) +#define MT2712_PIN_165_SPI4_CSN__FUNC_UTXD4 (MTK_PIN_NO(165) | 4) +#define MT2712_PIN_165_SPI4_CSN__FUNC_I2SO1_DO (MTK_PIN_NO(165) | 5) +#define MT2712_PIN_165_SPI4_CSN__FUNC_TDMO0_MCLK (MTK_PIN_NO(165) | 6) +#define MT2712_PIN_165_SPI4_CSN__FUNC_I2SO0_MCK (MTK_PIN_NO(165) | 7) + +#define MT2712_PIN_166_SPI4_MI__FUNC_GPIO166 (MTK_PIN_NO(166) | 0) +#define MT2712_PIN_166_SPI4_MI__FUNC_SPI_MI_4_ (MTK_PIN_NO(166) | 1) +#define MT2712_PIN_166_SPI4_MI__FUNC_DSIA_TE (MTK_PIN_NO(166) | 2) +#define MT2712_PIN_166_SPI4_MI__FUNC_SPI_SI_1_ (MTK_PIN_NO(166) | 3) +#define MT2712_PIN_166_SPI4_MI__FUNC_URXD4 (MTK_PIN_NO(166) | 4) +#define MT2712_PIN_166_SPI4_MI__FUNC_I2SO1_BCK (MTK_PIN_NO(166) | 5) + +#define MT2712_PIN_167_SPI4_MO__FUNC_GPIO167 (MTK_PIN_NO(167) | 0) +#define MT2712_PIN_167_SPI4_MO__FUNC_SPI_MO_4_ (MTK_PIN_NO(167) | 1) +#define MT2712_PIN_167_SPI4_MO__FUNC_DSIB_TE (MTK_PIN_NO(167) | 2) +#define MT2712_PIN_167_SPI4_MO__FUNC_SPI_SO_1_ (MTK_PIN_NO(167) | 3) +#define MT2712_PIN_167_SPI4_MO__FUNC_UTXD5 (MTK_PIN_NO(167) | 4) +#define MT2712_PIN_167_SPI4_MO__FUNC_I2SO1_WS (MTK_PIN_NO(167) | 5) + +#define MT2712_PIN_168_SPI4_CK__FUNC_GPIO168 (MTK_PIN_NO(168) | 0) +#define MT2712_PIN_168_SPI4_CK__FUNC_SPI_CK_4_ (MTK_PIN_NO(168) | 1) +#define MT2712_PIN_168_SPI4_CK__FUNC_LCM_RST1 (MTK_PIN_NO(168) | 2) +#define MT2712_PIN_168_SPI4_CK__FUNC_SPI_CK_1_ (MTK_PIN_NO(168) | 3) +#define MT2712_PIN_168_SPI4_CK__FUNC_URXD5 (MTK_PIN_NO(168) | 4) +#define MT2712_PIN_168_SPI4_CK__FUNC_I2SO1_MCK (MTK_PIN_NO(168) | 5) + +#define MT2712_PIN_169_I2SI0_DATA__FUNC_GPIO169 (MTK_PIN_NO(169) | 0) +#define MT2712_PIN_169_I2SI0_DATA__FUNC_I2SI0_DI (MTK_PIN_NO(169) | 1) +#define MT2712_PIN_169_I2SI0_DATA__FUNC_I2SI1_DI (MTK_PIN_NO(169) | 2) +#define MT2712_PIN_169_I2SI0_DATA__FUNC_I2SI2_DI (MTK_PIN_NO(169) | 3) +#define MT2712_PIN_169_I2SI0_DATA__FUNC_TDMIN_DI (MTK_PIN_NO(169) | 4) + +#define MT2712_PIN_170_I2SI0_LRCK__FUNC_GPIO170 (MTK_PIN_NO(170) | 0) +#define MT2712_PIN_170_I2SI0_LRCK__FUNC_I2SI0_WS (MTK_PIN_NO(170) | 1) +#define MT2712_PIN_170_I2SI0_LRCK__FUNC_I2SI1_WS (MTK_PIN_NO(170) | 2) +#define MT2712_PIN_170_I2SI0_LRCK__FUNC_I2SI2_WS (MTK_PIN_NO(170) | 3) +#define MT2712_PIN_170_I2SI0_LRCK__FUNC_TDMIN_LRCK (MTK_PIN_NO(170) | 4) +#define MT2712_PIN_170_I2SI0_LRCK__FUNC_TDMO0_DATA3 (MTK_PIN_NO(170) | 5) +#define MT2712_PIN_170_I2SI0_LRCK__FUNC_TDMO1_DATA3 (MTK_PIN_NO(170) | 6) + +#define MT2712_PIN_171_I2SI0_MCLK__FUNC_GPIO171 (MTK_PIN_NO(171) | 0) +#define MT2712_PIN_171_I2SI0_MCLK__FUNC_I2SI0_MCK (MTK_PIN_NO(171) | 1) +#define MT2712_PIN_171_I2SI0_MCLK__FUNC_I2SI1_MCK (MTK_PIN_NO(171) | 2) +#define MT2712_PIN_171_I2SI0_MCLK__FUNC_I2SI2_MCK (MTK_PIN_NO(171) | 3) +#define MT2712_PIN_171_I2SI0_MCLK__FUNC_TDMIN_MCLK (MTK_PIN_NO(171) | 4) +#define MT2712_PIN_171_I2SI0_MCLK__FUNC_TDMO0_DATA2 (MTK_PIN_NO(171) | 5) +#define MT2712_PIN_171_I2SI0_MCLK__FUNC_TDMO1_DATA2 (MTK_PIN_NO(171) | 6) + +#define MT2712_PIN_172_I2SI0_BCK__FUNC_GPIO172 (MTK_PIN_NO(172) | 0) +#define MT2712_PIN_172_I2SI0_BCK__FUNC_I2SI0_BCK (MTK_PIN_NO(172) | 1) +#define MT2712_PIN_172_I2SI0_BCK__FUNC_I2SI1_BCK (MTK_PIN_NO(172) | 2) +#define MT2712_PIN_172_I2SI0_BCK__FUNC_I2SI2_BCK (MTK_PIN_NO(172) | 3) +#define MT2712_PIN_172_I2SI0_BCK__FUNC_TDMIN_BCK (MTK_PIN_NO(172) | 4) +#define MT2712_PIN_172_I2SI0_BCK__FUNC_TDMO0_DATA1 (MTK_PIN_NO(172) | 5) +#define MT2712_PIN_172_I2SI0_BCK__FUNC_TDMO1_DATA1 (MTK_PIN_NO(172) | 6) + +#define MT2712_PIN_173_I2SI2_DATA__FUNC_GPIO173 (MTK_PIN_NO(173) | 0) +#define MT2712_PIN_173_I2SI2_DATA__FUNC_I2SI2_DI (MTK_PIN_NO(173) | 1) +#define MT2712_PIN_173_I2SI2_DATA__FUNC_I2SI0_DI (MTK_PIN_NO(173) | 2) +#define MT2712_PIN_173_I2SI2_DATA__FUNC_I2SI1_DI (MTK_PIN_NO(173) | 3) +#define MT2712_PIN_173_I2SI2_DATA__FUNC_PCM1_DI (MTK_PIN_NO(173) | 4) +#define MT2712_PIN_173_I2SI2_DATA__FUNC_TDMIN_DI (MTK_PIN_NO(173) | 5) +#define MT2712_PIN_173_I2SI2_DATA__FUNC_PCM1_DO (MTK_PIN_NO(173) | 6) + +#define MT2712_PIN_174_I2SI2_MCLK__FUNC_GPIO174 (MTK_PIN_NO(174) | 0) +#define MT2712_PIN_174_I2SI2_MCLK__FUNC_I2SI2_MCK (MTK_PIN_NO(174) | 1) +#define MT2712_PIN_174_I2SI2_MCLK__FUNC_I2SI0_MCK (MTK_PIN_NO(174) | 2) +#define MT2712_PIN_174_I2SI2_MCLK__FUNC_I2SI1_MCK (MTK_PIN_NO(174) | 3) +#define MT2712_PIN_174_I2SI2_MCLK__FUNC_PCM1_DO (MTK_PIN_NO(174) | 4) +#define MT2712_PIN_174_I2SI2_MCLK__FUNC_TDMIN_MCLK (MTK_PIN_NO(174) | 5) +#define MT2712_PIN_174_I2SI2_MCLK__FUNC_PCM1_DI (MTK_PIN_NO(174) | 6) +#define MT2712_PIN_174_I2SI2_MCLK__FUNC_I2S_IQ2_SDQB (MTK_PIN_NO(174) | 7) + +#define MT2712_PIN_175_I2SI2_BCK__FUNC_GPIO175 (MTK_PIN_NO(175) | 0) +#define MT2712_PIN_175_I2SI2_BCK__FUNC_I2SI2_BCK (MTK_PIN_NO(175) | 1) +#define MT2712_PIN_175_I2SI2_BCK__FUNC_I2SI0_BCK (MTK_PIN_NO(175) | 2) +#define MT2712_PIN_175_I2SI2_BCK__FUNC_I2SI1_BCK (MTK_PIN_NO(175) | 3) +#define MT2712_PIN_175_I2SI2_BCK__FUNC_PCM1_CLK (MTK_PIN_NO(175) | 4) +#define MT2712_PIN_175_I2SI2_BCK__FUNC_TDMIN_BCK (MTK_PIN_NO(175) | 5) + +#define MT2712_PIN_176_I2SI2_LRCK__FUNC_GPIO176 (MTK_PIN_NO(176) | 0) +#define MT2712_PIN_176_I2SI2_LRCK__FUNC_I2SI2_WS (MTK_PIN_NO(176) | 1) +#define MT2712_PIN_176_I2SI2_LRCK__FUNC_I2SI0_WS (MTK_PIN_NO(176) | 2) +#define MT2712_PIN_176_I2SI2_LRCK__FUNC_I2SI1_WS (MTK_PIN_NO(176) | 3) +#define MT2712_PIN_176_I2SI2_LRCK__FUNC_PCM1_SYNC (MTK_PIN_NO(176) | 4) +#define MT2712_PIN_176_I2SI2_LRCK__FUNC_TDMIN_LRCK (MTK_PIN_NO(176) | 5) + +#define MT2712_PIN_177_I2SI1_DATA__FUNC_GPIO177 (MTK_PIN_NO(177) | 0) +#define MT2712_PIN_177_I2SI1_DATA__FUNC_I2SI1_DI (MTK_PIN_NO(177) | 1) +#define MT2712_PIN_177_I2SI1_DATA__FUNC_I2SI0_DI (MTK_PIN_NO(177) | 2) +#define MT2712_PIN_177_I2SI1_DATA__FUNC_I2SI2_DI (MTK_PIN_NO(177) | 3) +#define MT2712_PIN_177_I2SI1_DATA__FUNC_TDMIN_DI (MTK_PIN_NO(177) | 4) + +#define MT2712_PIN_178_I2SI1_BCK__FUNC_GPIO178 (MTK_PIN_NO(178) | 0) +#define MT2712_PIN_178_I2SI1_BCK__FUNC_I2SI1_BCK (MTK_PIN_NO(178) | 1) +#define MT2712_PIN_178_I2SI1_BCK__FUNC_I2SI0_BCK (MTK_PIN_NO(178) | 2) +#define MT2712_PIN_178_I2SI1_BCK__FUNC_I2SI2_BCK (MTK_PIN_NO(178) | 3) +#define MT2712_PIN_178_I2SI1_BCK__FUNC_TDMIN_BCK (MTK_PIN_NO(178) | 4) +#define MT2712_PIN_178_I2SI1_BCK__FUNC_TDMO0_DATA3 (MTK_PIN_NO(178) | 5) +#define MT2712_PIN_178_I2SI1_BCK__FUNC_TDMO1_DATA3 (MTK_PIN_NO(178) | 6) + +#define MT2712_PIN_179_I2SI1_LRCK__FUNC_GPIO179 (MTK_PIN_NO(179) | 0) +#define MT2712_PIN_179_I2SI1_LRCK__FUNC_I2SI1_WS (MTK_PIN_NO(179) | 1) +#define MT2712_PIN_179_I2SI1_LRCK__FUNC_I2SI0_WS (MTK_PIN_NO(179) | 2) +#define MT2712_PIN_179_I2SI1_LRCK__FUNC_I2SI2_WS (MTK_PIN_NO(179) | 3) +#define MT2712_PIN_179_I2SI1_LRCK__FUNC_TDMIN_LRCK (MTK_PIN_NO(179) | 4) +#define MT2712_PIN_179_I2SI1_LRCK__FUNC_TDMO0_DATA2 (MTK_PIN_NO(179) | 5) +#define MT2712_PIN_179_I2SI1_LRCK__FUNC_TDMO1_DATA2 (MTK_PIN_NO(179) | 6) + +#define MT2712_PIN_180_I2SI1_MCLK__FUNC_GPIO180 (MTK_PIN_NO(180) | 0) +#define MT2712_PIN_180_I2SI1_MCLK__FUNC_I2SI1_MCK (MTK_PIN_NO(180) | 1) +#define MT2712_PIN_180_I2SI1_MCLK__FUNC_I2SI0_MCK (MTK_PIN_NO(180) | 2) +#define MT2712_PIN_180_I2SI1_MCLK__FUNC_I2SI2_MCK (MTK_PIN_NO(180) | 3) +#define MT2712_PIN_180_I2SI1_MCLK__FUNC_TDMIN_MCLK (MTK_PIN_NO(180) | 4) +#define MT2712_PIN_180_I2SI1_MCLK__FUNC_TDMO0_DATA1 (MTK_PIN_NO(180) | 5) +#define MT2712_PIN_180_I2SI1_MCLK__FUNC_TDMO1_DATA1 (MTK_PIN_NO(180) | 6) +#define MT2712_PIN_180_I2SI1_MCLK__FUNC_I2S_IQ2_SDIB (MTK_PIN_NO(180) | 7) + +#define MT2712_PIN_181_I2SO1_DATA0__FUNC_GPIO181 (MTK_PIN_NO(181) | 0) +#define MT2712_PIN_181_I2SO1_DATA0__FUNC_I2SO1_DO (MTK_PIN_NO(181) | 1) +#define MT2712_PIN_181_I2SO1_DATA0__FUNC_I2SO0_DO0 (MTK_PIN_NO(181) | 2) +#define MT2712_PIN_181_I2SO1_DATA0__FUNC_I2SO2_DO (MTK_PIN_NO(181) | 3) +#define MT2712_PIN_181_I2SO1_DATA0__FUNC_DAI_TX (MTK_PIN_NO(181) | 4) +#define MT2712_PIN_181_I2SO1_DATA0__FUNC_TDMIN_MCLK (MTK_PIN_NO(181) | 5) +#define MT2712_PIN_181_I2SO1_DATA0__FUNC_I2S_IQ2_SDIA (MTK_PIN_NO(181) | 7) + +#define MT2712_PIN_182_I2SO1_BCK__FUNC_GPIO182 (MTK_PIN_NO(182) | 0) +#define MT2712_PIN_182_I2SO1_BCK__FUNC_I2SO1_BCK (MTK_PIN_NO(182) | 1) +#define MT2712_PIN_182_I2SO1_BCK__FUNC_I2SO0_BCK (MTK_PIN_NO(182) | 2) +#define MT2712_PIN_182_I2SO1_BCK__FUNC_I2SO2_BCK (MTK_PIN_NO(182) | 3) +#define MT2712_PIN_182_I2SO1_BCK__FUNC_DAI_SYNC (MTK_PIN_NO(182) | 4) +#define MT2712_PIN_182_I2SO1_BCK__FUNC_TDMIN_BCK (MTK_PIN_NO(182) | 5) +#define MT2712_PIN_182_I2SO1_BCK__FUNC_TDMO0_DATA3 (MTK_PIN_NO(182) | 6) +#define MT2712_PIN_182_I2SO1_BCK__FUNC_I2S_IQ2_BCK (MTK_PIN_NO(182) | 7) + +#define MT2712_PIN_183_I2SO1_LRCK__FUNC_GPIO183 (MTK_PIN_NO(183) | 0) +#define MT2712_PIN_183_I2SO1_LRCK__FUNC_I2SO1_WS (MTK_PIN_NO(183) | 1) +#define MT2712_PIN_183_I2SO1_LRCK__FUNC_I2SO0_WS (MTK_PIN_NO(183) | 2) +#define MT2712_PIN_183_I2SO1_LRCK__FUNC_I2SO2_WS (MTK_PIN_NO(183) | 3) +#define MT2712_PIN_183_I2SO1_LRCK__FUNC_DAI_CLK (MTK_PIN_NO(183) | 4) +#define MT2712_PIN_183_I2SO1_LRCK__FUNC_TDMIN_DI (MTK_PIN_NO(183) | 5) +#define MT2712_PIN_183_I2SO1_LRCK__FUNC_TDMO0_DATA2 (MTK_PIN_NO(183) | 6) +#define MT2712_PIN_183_I2SO1_LRCK__FUNC_I2S_IQ2_WS (MTK_PIN_NO(183) | 7) + +#define MT2712_PIN_184_I2SO1_MCLK__FUNC_GPIO184 (MTK_PIN_NO(184) | 0) +#define MT2712_PIN_184_I2SO1_MCLK__FUNC_I2SO1_MCK (MTK_PIN_NO(184) | 1) +#define MT2712_PIN_184_I2SO1_MCLK__FUNC_I2SO0_MCK (MTK_PIN_NO(184) | 2) +#define MT2712_PIN_184_I2SO1_MCLK__FUNC_I2SO2_MCK (MTK_PIN_NO(184) | 3) +#define MT2712_PIN_184_I2SO1_MCLK__FUNC_DAI_RX (MTK_PIN_NO(184) | 4) +#define MT2712_PIN_184_I2SO1_MCLK__FUNC_TDMIN_LRCK (MTK_PIN_NO(184) | 5) +#define MT2712_PIN_184_I2SO1_MCLK__FUNC_TDMO0_DATA1 (MTK_PIN_NO(184) | 6) +#define MT2712_PIN_184_I2SO1_MCLK__FUNC_I2S_IQ2_SDQA (MTK_PIN_NO(184) | 7) + +#define MT2712_PIN_185_AUD_EXT_CK2__FUNC_GPIO185 (MTK_PIN_NO(185) | 0) +#define MT2712_PIN_185_AUD_EXT_CK2__FUNC_AUD_EXT_CK2 (MTK_PIN_NO(185) | 1) +#define MT2712_PIN_185_AUD_EXT_CK2__FUNC_AUD_EXT_CK1 (MTK_PIN_NO(185) | 2) +#define MT2712_PIN_185_AUD_EXT_CK2__FUNC_I2SO1_DO (MTK_PIN_NO(185) | 3) +#define MT2712_PIN_185_AUD_EXT_CK2__FUNC_I2SI2_DI (MTK_PIN_NO(185) | 4) +#define MT2712_PIN_185_AUD_EXT_CK2__FUNC_MRG_RX (MTK_PIN_NO(185) | 5) +#define MT2712_PIN_185_AUD_EXT_CK2__FUNC_PCM1_DI (MTK_PIN_NO(185) | 6) +#define MT2712_PIN_185_AUD_EXT_CK2__FUNC_I2S_IQ0_SDQB (MTK_PIN_NO(185) | 7) + +#define MT2712_PIN_186_AUD_EXT_CK1__FUNC_GPIO186 (MTK_PIN_NO(186) | 0) +#define MT2712_PIN_186_AUD_EXT_CK1__FUNC_AUD_EXT_CK1 (MTK_PIN_NO(186) | 1) +#define MT2712_PIN_186_AUD_EXT_CK1__FUNC_AUD_EXT_CK2 (MTK_PIN_NO(186) | 2) +#define MT2712_PIN_186_AUD_EXT_CK1__FUNC_I2SO0_DO1 (MTK_PIN_NO(186) | 3) +#define MT2712_PIN_186_AUD_EXT_CK1__FUNC_I2SI1_DI (MTK_PIN_NO(186) | 4) +#define MT2712_PIN_186_AUD_EXT_CK1__FUNC_MRG_TX (MTK_PIN_NO(186) | 5) +#define MT2712_PIN_186_AUD_EXT_CK1__FUNC_PCM1_DO (MTK_PIN_NO(186) | 6) +#define MT2712_PIN_186_AUD_EXT_CK1__FUNC_I2S_IQ0_SDIB (MTK_PIN_NO(186) | 7) + +#define MT2712_PIN_187_I2SO2_BCK__FUNC_GPIO187 (MTK_PIN_NO(187) | 0) +#define MT2712_PIN_187_I2SO2_BCK__FUNC_I2SO2_BCK (MTK_PIN_NO(187) | 1) +#define MT2712_PIN_187_I2SO2_BCK__FUNC_I2SO0_BCK (MTK_PIN_NO(187) | 2) +#define MT2712_PIN_187_I2SO2_BCK__FUNC_I2SO1_BCK (MTK_PIN_NO(187) | 3) +#define MT2712_PIN_187_I2SO2_BCK__FUNC_PCM1_CLK (MTK_PIN_NO(187) | 4) +#define MT2712_PIN_187_I2SO2_BCK__FUNC_MRG_SYNC (MTK_PIN_NO(187) | 5) +#define MT2712_PIN_187_I2SO2_BCK__FUNC_TDMO1_DATA3 (MTK_PIN_NO(187) | 6) +#define MT2712_PIN_187_I2SO2_BCK__FUNC_I2S_IQ0_BCK (MTK_PIN_NO(187) | 7) + +#define MT2712_PIN_188_I2SO2_LRCK__FUNC_GPIO188 (MTK_PIN_NO(188) | 0) +#define MT2712_PIN_188_I2SO2_LRCK__FUNC_I2SO2_WS (MTK_PIN_NO(188) | 1) +#define MT2712_PIN_188_I2SO2_LRCK__FUNC_I2SO0_WS (MTK_PIN_NO(188) | 2) +#define MT2712_PIN_188_I2SO2_LRCK__FUNC_I2SO1_WS (MTK_PIN_NO(188) | 3) +#define MT2712_PIN_188_I2SO2_LRCK__FUNC_PCM1_SYNC (MTK_PIN_NO(188) | 4) +#define MT2712_PIN_188_I2SO2_LRCK__FUNC_MRG_CLK (MTK_PIN_NO(188) | 5) +#define MT2712_PIN_188_I2SO2_LRCK__FUNC_TDMO1_DATA2 (MTK_PIN_NO(188) | 6) +#define MT2712_PIN_188_I2SO2_LRCK__FUNC_I2S_IQ0_WS (MTK_PIN_NO(188) | 7) + +#define MT2712_PIN_189_I2SO2_MCLK__FUNC_GPIO189 (MTK_PIN_NO(189) | 0) +#define MT2712_PIN_189_I2SO2_MCLK__FUNC_I2SO2_MCK (MTK_PIN_NO(189) | 1) +#define MT2712_PIN_189_I2SO2_MCLK__FUNC_I2SO0_MCK (MTK_PIN_NO(189) | 2) +#define MT2712_PIN_189_I2SO2_MCLK__FUNC_I2SO1_MCK (MTK_PIN_NO(189) | 3) +#define MT2712_PIN_189_I2SO2_MCLK__FUNC_PCM1_DO (MTK_PIN_NO(189) | 4) +#define MT2712_PIN_189_I2SO2_MCLK__FUNC_MRG_RX (MTK_PIN_NO(189) | 5) +#define MT2712_PIN_189_I2SO2_MCLK__FUNC_TDMO1_DATA1 (MTK_PIN_NO(189) | 6) +#define MT2712_PIN_189_I2SO2_MCLK__FUNC_I2S_IQ0_SDQA (MTK_PIN_NO(189) | 7) + +#define MT2712_PIN_190_I2SO2_DATA0__FUNC_GPIO190 (MTK_PIN_NO(190) | 0) +#define MT2712_PIN_190_I2SO2_DATA0__FUNC_I2SO2_DO (MTK_PIN_NO(190) | 1) +#define MT2712_PIN_190_I2SO2_DATA0__FUNC_I2SO0_DO0 (MTK_PIN_NO(190) | 2) +#define MT2712_PIN_190_I2SO2_DATA0__FUNC_I2SO1_DO (MTK_PIN_NO(190) | 3) +#define MT2712_PIN_190_I2SO2_DATA0__FUNC_PCM1_DI (MTK_PIN_NO(190) | 4) +#define MT2712_PIN_190_I2SO2_DATA0__FUNC_MRG_TX (MTK_PIN_NO(190) | 5) +#define MT2712_PIN_190_I2SO2_DATA0__FUNC_PCM1_DO (MTK_PIN_NO(190) | 6) +#define MT2712_PIN_190_I2SO2_DATA0__FUNC_I2S_IQ0_SDIA (MTK_PIN_NO(190) | 7) + +#define MT2712_PIN_191_I2SO0_DATA1__FUNC_GPIO191 (MTK_PIN_NO(191) | 0) +#define MT2712_PIN_191_I2SO0_DATA1__FUNC_I2SO0_DO1 (MTK_PIN_NO(191) | 1) +#define MT2712_PIN_191_I2SO0_DATA1__FUNC_I2SI0_DI (MTK_PIN_NO(191) | 2) +#define MT2712_PIN_191_I2SO0_DATA1__FUNC_I2SI1_DI (MTK_PIN_NO(191) | 3) +#define MT2712_PIN_191_I2SO0_DATA1__FUNC_I2SI2_DI (MTK_PIN_NO(191) | 4) +#define MT2712_PIN_191_I2SO0_DATA1__FUNC_DAI_TX (MTK_PIN_NO(191) | 5) +#define MT2712_PIN_191_I2SO0_DATA1__FUNC_I2S_IQ0_SDQB (MTK_PIN_NO(191) | 6) +#define MT2712_PIN_191_I2SO0_DATA1__FUNC_I2S_IQ1_SDQB (MTK_PIN_NO(191) | 7) + +#define MT2712_PIN_192_I2SO0_MCLK__FUNC_GPIO192 (MTK_PIN_NO(192) | 0) +#define MT2712_PIN_192_I2SO0_MCLK__FUNC_I2SO0_MCK (MTK_PIN_NO(192) | 1) +#define MT2712_PIN_192_I2SO0_MCLK__FUNC_I2SO1_MCK (MTK_PIN_NO(192) | 2) +#define MT2712_PIN_192_I2SO0_MCLK__FUNC_I2SO2_MCK (MTK_PIN_NO(192) | 3) +#define MT2712_PIN_192_I2SO0_MCLK__FUNC_USB4_FT_SCL (MTK_PIN_NO(192) | 4) +#define MT2712_PIN_192_I2SO0_MCLK__FUNC_TDMO1_DATA3 (MTK_PIN_NO(192) | 5) +#define MT2712_PIN_192_I2SO0_MCLK__FUNC_I2S_IQ0_SDIB (MTK_PIN_NO(192) | 6) +#define MT2712_PIN_192_I2SO0_MCLK__FUNC_I2S_IQ1_SDQA (MTK_PIN_NO(192) | 7) + +#define MT2712_PIN_193_I2SO0_DATA0__FUNC_GPIO193 (MTK_PIN_NO(193) | 0) +#define MT2712_PIN_193_I2SO0_DATA0__FUNC_I2SO0_DO0 (MTK_PIN_NO(193) | 1) +#define MT2712_PIN_193_I2SO0_DATA0__FUNC_I2SO1_DO (MTK_PIN_NO(193) | 2) +#define MT2712_PIN_193_I2SO0_DATA0__FUNC_I2SO2_DO (MTK_PIN_NO(193) | 3) +#define MT2712_PIN_193_I2SO0_DATA0__FUNC_USB4_FT_SDA (MTK_PIN_NO(193) | 4) +#define MT2712_PIN_193_I2SO0_DATA0__FUNC_I2S_IQ1_SDIA (MTK_PIN_NO(193) | 7) + +#define MT2712_PIN_194_I2SO0_LRCK__FUNC_GPIO194 (MTK_PIN_NO(194) | 0) +#define MT2712_PIN_194_I2SO0_LRCK__FUNC_I2SO0_WS (MTK_PIN_NO(194) | 1) +#define MT2712_PIN_194_I2SO0_LRCK__FUNC_I2SO1_WS (MTK_PIN_NO(194) | 2) +#define MT2712_PIN_194_I2SO0_LRCK__FUNC_I2SO2_WS (MTK_PIN_NO(194) | 3) +#define MT2712_PIN_194_I2SO0_LRCK__FUNC_USB5_FT_SCL (MTK_PIN_NO(194) | 4) +#define MT2712_PIN_194_I2SO0_LRCK__FUNC_TDMO1_DATA2 (MTK_PIN_NO(194) | 5) +#define MT2712_PIN_194_I2SO0_LRCK__FUNC_I2S_IQ1_WS (MTK_PIN_NO(194) | 7) + +#define MT2712_PIN_195_I2SO0_BCK__FUNC_GPIO195 (MTK_PIN_NO(195) | 0) +#define MT2712_PIN_195_I2SO0_BCK__FUNC_I2SO0_BCK (MTK_PIN_NO(195) | 1) +#define MT2712_PIN_195_I2SO0_BCK__FUNC_I2SO1_BCK (MTK_PIN_NO(195) | 2) +#define MT2712_PIN_195_I2SO0_BCK__FUNC_I2SO2_BCK (MTK_PIN_NO(195) | 3) +#define MT2712_PIN_195_I2SO0_BCK__FUNC_USB5_FT_SDA (MTK_PIN_NO(195) | 4) +#define MT2712_PIN_195_I2SO0_BCK__FUNC_TDMO1_DATA1 (MTK_PIN_NO(195) | 5) +#define MT2712_PIN_195_I2SO0_BCK__FUNC_I2S_IQ1_BCK (MTK_PIN_NO(195) | 7) + +#define MT2712_PIN_196_TDMO1_MCLK__FUNC_GPIO196 (MTK_PIN_NO(196) | 0) +#define MT2712_PIN_196_TDMO1_MCLK__FUNC_TDMO1_MCLK (MTK_PIN_NO(196) | 1) +#define MT2712_PIN_196_TDMO1_MCLK__FUNC_TDMO0_MCLK (MTK_PIN_NO(196) | 2) +#define MT2712_PIN_196_TDMO1_MCLK__FUNC_TDMIN_MCLK (MTK_PIN_NO(196) | 3) +#define MT2712_PIN_196_TDMO1_MCLK__FUNC_I2SO0_DO1 (MTK_PIN_NO(196) | 6) +#define MT2712_PIN_196_TDMO1_MCLK__FUNC_I2S_IQ1_SDIB (MTK_PIN_NO(196) | 7) + +#define MT2712_PIN_197_TDMO1_LRCK__FUNC_GPIO197 (MTK_PIN_NO(197) | 0) +#define MT2712_PIN_197_TDMO1_LRCK__FUNC_TDMO1_LRCK (MTK_PIN_NO(197) | 1) +#define MT2712_PIN_197_TDMO1_LRCK__FUNC_TDMO0_LRCK (MTK_PIN_NO(197) | 2) +#define MT2712_PIN_197_TDMO1_LRCK__FUNC_TDMIN_LRCK (MTK_PIN_NO(197) | 3) +#define MT2712_PIN_197_TDMO1_LRCK__FUNC_TDMO0_DATA3 (MTK_PIN_NO(197) | 4) +#define MT2712_PIN_197_TDMO1_LRCK__FUNC_TDMO1_DATA3 (MTK_PIN_NO(197) | 5) +#define MT2712_PIN_197_TDMO1_LRCK__FUNC_I2SO3_MCK (MTK_PIN_NO(197) | 6) +#define MT2712_PIN_197_TDMO1_LRCK__FUNC_TDMO1_DATA2 (MTK_PIN_NO(197) | 7) + +#define MT2712_PIN_198_TDMO1_BCK__FUNC_GPIO198 (MTK_PIN_NO(198) | 0) +#define MT2712_PIN_198_TDMO1_BCK__FUNC_TDMO1_BCK (MTK_PIN_NO(198) | 1) +#define MT2712_PIN_198_TDMO1_BCK__FUNC_TDMO0_BCK (MTK_PIN_NO(198) | 2) +#define MT2712_PIN_198_TDMO1_BCK__FUNC_TDMIN_BCK (MTK_PIN_NO(198) | 3) +#define MT2712_PIN_198_TDMO1_BCK__FUNC_TDMO0_DATA2 (MTK_PIN_NO(198) | 4) +#define MT2712_PIN_198_TDMO1_BCK__FUNC_TDMO1_DATA2 (MTK_PIN_NO(198) | 5) +#define MT2712_PIN_198_TDMO1_BCK__FUNC_I2SO3_BCK (MTK_PIN_NO(198) | 6) +#define MT2712_PIN_198_TDMO1_BCK__FUNC_TDMO1_DATA1 (MTK_PIN_NO(198) | 7) + +#define MT2712_PIN_199_TDMO1_DATA__FUNC_GPIO199 (MTK_PIN_NO(199) | 0) +#define MT2712_PIN_199_TDMO1_DATA__FUNC_TDMO1_DATA (MTK_PIN_NO(199) | 1) +#define MT2712_PIN_199_TDMO1_DATA__FUNC_TDMO0_DATA (MTK_PIN_NO(199) | 2) +#define MT2712_PIN_199_TDMO1_DATA__FUNC_TDMIN_DI (MTK_PIN_NO(199) | 3) +#define MT2712_PIN_199_TDMO1_DATA__FUNC_TDMO0_DATA1 (MTK_PIN_NO(199) | 4) +#define MT2712_PIN_199_TDMO1_DATA__FUNC_TDMO1_DATA1 (MTK_PIN_NO(199) | 5) +#define MT2712_PIN_199_TDMO1_DATA__FUNC_I2SO3_WS (MTK_PIN_NO(199) | 6) + +#define MT2712_PIN_200_TDMO0_MCLK__FUNC_GPIO200 (MTK_PIN_NO(200) | 0) +#define MT2712_PIN_200_TDMO0_MCLK__FUNC_TDMO0_MCLK0 (MTK_PIN_NO(200) | 1) +#define MT2712_PIN_200_TDMO0_MCLK__FUNC_TDMO1_MCLK0 (MTK_PIN_NO(200) | 2) +#define MT2712_PIN_200_TDMO0_MCLK__FUNC_PCM1_DI (MTK_PIN_NO(200) | 3) +#define MT2712_PIN_200_TDMO0_MCLK__FUNC_TDMO0_MCLK1 (MTK_PIN_NO(200) | 4) +#define MT2712_PIN_200_TDMO0_MCLK__FUNC_TDMO1_MCLK1 (MTK_PIN_NO(200) | 5) +#define MT2712_PIN_200_TDMO0_MCLK__FUNC_MRG_TX (MTK_PIN_NO(200) | 6) +#define MT2712_PIN_200_TDMO0_MCLK__FUNC_I2SO2_MCK (MTK_PIN_NO(200) | 7) + +#define MT2712_PIN_201_TDMO0_LRCK__FUNC_GPIO201 (MTK_PIN_NO(201) | 0) +#define MT2712_PIN_201_TDMO0_LRCK__FUNC_TDMO0_LRCK0 (MTK_PIN_NO(201) | 1) +#define MT2712_PIN_201_TDMO0_LRCK__FUNC_TDMO1_LRCK0 (MTK_PIN_NO(201) | 2) +#define MT2712_PIN_201_TDMO0_LRCK__FUNC_PCM1_SYNC (MTK_PIN_NO(201) | 3) +#define MT2712_PIN_201_TDMO0_LRCK__FUNC_TDMO0_LRCK1 (MTK_PIN_NO(201) | 4) +#define MT2712_PIN_201_TDMO0_LRCK__FUNC_TDMO1_LRCK1 (MTK_PIN_NO(201) | 5) +#define MT2712_PIN_201_TDMO0_LRCK__FUNC_MRG_RX (MTK_PIN_NO(201) | 6) +#define MT2712_PIN_201_TDMO0_LRCK__FUNC_I2SO2_WS (MTK_PIN_NO(201) | 7) + +#define MT2712_PIN_202_TDMO0_BCK__FUNC_GPIO202 (MTK_PIN_NO(202) | 0) +#define MT2712_PIN_202_TDMO0_BCK__FUNC_TDMO0_BCK0 (MTK_PIN_NO(202) | 1) +#define MT2712_PIN_202_TDMO0_BCK__FUNC_TDMO1_BCK0 (MTK_PIN_NO(202) | 2) +#define MT2712_PIN_202_TDMO0_BCK__FUNC_PCM1_CLK (MTK_PIN_NO(202) | 3) +#define MT2712_PIN_202_TDMO0_BCK__FUNC_TDMO0_BCK1 (MTK_PIN_NO(202) | 4) +#define MT2712_PIN_202_TDMO0_BCK__FUNC_TDMO1_BCK1 (MTK_PIN_NO(202) | 5) +#define MT2712_PIN_202_TDMO0_BCK__FUNC_MRG_SYNC (MTK_PIN_NO(202) | 6) +#define MT2712_PIN_202_TDMO0_BCK__FUNC_I2SO2_BCK (MTK_PIN_NO(202) | 7) + +#define MT2712_PIN_203_TDMO0_DATA__FUNC_GPIO203 (MTK_PIN_NO(203) | 0) +#define MT2712_PIN_203_TDMO0_DATA__FUNC_TDMO0_DATA0 (MTK_PIN_NO(203) | 1) +#define MT2712_PIN_203_TDMO0_DATA__FUNC_TDMO1_DATA0 (MTK_PIN_NO(203) | 2) +#define MT2712_PIN_203_TDMO0_DATA__FUNC_PCM1_DO (MTK_PIN_NO(203) | 3) +#define MT2712_PIN_203_TDMO0_DATA__FUNC_TDMO0_DATA1 (MTK_PIN_NO(203) | 4) +#define MT2712_PIN_203_TDMO0_DATA__FUNC_TDMO1_DATA1 (MTK_PIN_NO(203) | 5) +#define MT2712_PIN_203_TDMO0_DATA__FUNC_MRG_CLK (MTK_PIN_NO(203) | 6) +#define MT2712_PIN_203_TDMO0_DATA__FUNC_I2SO2_DO (MTK_PIN_NO(203) | 7) + +#define MT2712_PIN_204_PERSTB_P0__FUNC_GPIO204 (MTK_PIN_NO(204) | 0) +#define MT2712_PIN_204_PERSTB_P0__FUNC_PERST_B_P0 (MTK_PIN_NO(204) | 1) + +#define MT2712_PIN_205_CLKREQN_P0__FUNC_GPIO205 (MTK_PIN_NO(205) | 0) +#define MT2712_PIN_205_CLKREQN_P0__FUNC_CLKREQ_N_P0 (MTK_PIN_NO(205) | 1) + +#define MT2712_PIN_206_WAKEEN_P0__FUNC_GPIO206 (MTK_PIN_NO(206) | 0) +#define MT2712_PIN_206_WAKEEN_P0__FUNC_WAKE_EN_P0 (MTK_PIN_NO(206) | 1) + +#define MT2712_PIN_207_PERSTB_P1__FUNC_GPIO207 (MTK_PIN_NO(207) | 0) +#define MT2712_PIN_207_PERSTB_P1__FUNC_PERST_B_P1 (MTK_PIN_NO(207) | 1) + +#define MT2712_PIN_208_CLKREQN_P1__FUNC_GPIO208 (MTK_PIN_NO(208) | 0) +#define MT2712_PIN_208_CLKREQN_P1__FUNC_CLKREQ_N_P1 (MTK_PIN_NO(208) | 1) + +#define MT2712_PIN_209_WAKEEN_P1__FUNC_GPIO209 (MTK_PIN_NO(209) | 0) +#define MT2712_PIN_209_WAKEEN_P1__FUNC_WAKE_EN_P1 (MTK_PIN_NO(209) | 1) + +#endif /* __DTS_MT2712_PINFUNC_H */ diff --git a/arch/arm64/boot/dts/mediatek/mt2712e.dtsi b/arch/arm64/boot/dts/mediatek/mt2712e.dtsi index 9d88f41aefa0..6d8532af8346 100644 --- a/arch/arm64/boot/dts/mediatek/mt2712e.dtsi +++ b/arch/arm64/boot/dts/mediatek/mt2712e.dtsi @@ -9,6 +9,7 @@ #include #include #include +#include "mt2712-pinfunc.h" / { compatible = "mediatek,mt2712"; @@ -199,6 +200,34 @@ clock-output-names = "clkaud_ext_i_2"; }; + clki2si0_mck_i: oscillator@6 { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <30000000>; + clock-output-names = "clki2si0_mck_i"; + }; + + clki2si1_mck_i: oscillator@7 { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <30000000>; + clock-output-names = "clki2si1_mck_i"; + }; + + clki2si2_mck_i: oscillator@8 { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <30000000>; + clock-output-names = "clki2si2_mck_i"; + }; + + clktdmin_mclk_i: oscillator@9 { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <30000000>; + clock-output-names = "clktdmin_mclk_i"; + }; + timer { compatible = "arm,armv8-timer"; interrupt-parent = <&gic>; @@ -230,6 +259,23 @@ #clock-cells = <1>; }; + syscfg_pctl_a: syscfg_pctl_a@10005000 { + compatible = "mediatek,mt2712-pctl-a-syscfg", "syscon"; + reg = <0 0x10005000 0 0x1000>; + }; + + pio: pinctrl@10005000 { + compatible = "mediatek,mt2712-pinctrl"; + reg = <0 0x1000b000 0 0x1000>; + mediatek,pctl-regmap = <&syscfg_pctl_a>; + pins-are-numbered; + gpio-controller; + #gpio-cells = <2>; + interrupt-controller; + #interrupt-cells = <2>; + interrupts = ; + }; + scpsys: scpsys@10006000 { compatible = "mediatek,mt2712-scpsys", "syscon"; #power-domain-cells = <1>; diff --git a/arch/arm64/boot/dts/mediatek/mt7622-rfb1.dts b/arch/arm64/boot/dts/mediatek/mt7622-rfb1.dts index 45d8655ee423..b7837642c33a 100644 --- a/arch/arm64/boot/dts/mediatek/mt7622-rfb1.dts +++ b/arch/arm64/boot/dts/mediatek/mt7622-rfb1.dts @@ -18,7 +18,7 @@ compatible = "mediatek,mt7622-rfb1", "mediatek,mt7622"; chosen { - bootargs = "console=ttyS0,115200n1"; + bootargs = "console=ttyS0,115200n1 swiotlb=512"; }; cpus { @@ -163,10 +163,17 @@ i2s1_pins: i2s1-pins { mux { function = "i2s"; - groups = "i2s_out_bclk_ws_mclk", + groups = "i2s_out_mclk_bclk_ws", "i2s1_in_data", "i2s1_out_data"; }; + + conf { + pins = "I2S1_IN", "I2S1_OUT", "I2S_BCLK", + "I2S_WS", "I2S_MCLK"; + drive-strength = <12>; + bias-pull-down; + }; }; irrx_pins: irrx-pins { diff --git a/arch/arm64/boot/dts/mediatek/mt7622.dtsi b/arch/arm64/boot/dts/mediatek/mt7622.dtsi index e9d5130df8d1..9213c966c224 100644 --- a/arch/arm64/boot/dts/mediatek/mt7622.dtsi +++ b/arch/arm64/boot/dts/mediatek/mt7622.dtsi @@ -527,6 +527,95 @@ status = "disabled"; }; + audsys: clock-controller@11220000 { + compatible = "mediatek,mt7622-audsys", "syscon"; + reg = <0 0x11220000 0 0x2000>; + #clock-cells = <1>; + + afe: audio-controller { + compatible = "mediatek,mt7622-audio"; + interrupts = , + ; + interrupt-names = "afe", "asys"; + + clocks = <&infracfg CLK_INFRA_AUDIO_PD>, + <&topckgen CLK_TOP_AUD1_SEL>, + <&topckgen CLK_TOP_AUD2_SEL>, + <&topckgen CLK_TOP_A1SYS_HP_DIV_PD>, + <&topckgen CLK_TOP_A2SYS_HP_DIV_PD>, + <&topckgen CLK_TOP_I2S0_MCK_SEL>, + <&topckgen CLK_TOP_I2S1_MCK_SEL>, + <&topckgen CLK_TOP_I2S2_MCK_SEL>, + <&topckgen CLK_TOP_I2S3_MCK_SEL>, + <&topckgen CLK_TOP_I2S0_MCK_DIV>, + <&topckgen CLK_TOP_I2S1_MCK_DIV>, + <&topckgen CLK_TOP_I2S2_MCK_DIV>, + <&topckgen CLK_TOP_I2S3_MCK_DIV>, + <&topckgen CLK_TOP_I2S0_MCK_DIV_PD>, + <&topckgen CLK_TOP_I2S1_MCK_DIV_PD>, + <&topckgen CLK_TOP_I2S2_MCK_DIV_PD>, + <&topckgen CLK_TOP_I2S3_MCK_DIV_PD>, + <&audsys CLK_AUDIO_I2SO1>, + <&audsys CLK_AUDIO_I2SO2>, + <&audsys CLK_AUDIO_I2SO3>, + <&audsys CLK_AUDIO_I2SO4>, + <&audsys CLK_AUDIO_I2SIN1>, + <&audsys CLK_AUDIO_I2SIN2>, + <&audsys CLK_AUDIO_I2SIN3>, + <&audsys CLK_AUDIO_I2SIN4>, + <&audsys CLK_AUDIO_ASRCO1>, + <&audsys CLK_AUDIO_ASRCO2>, + <&audsys CLK_AUDIO_ASRCO3>, + <&audsys CLK_AUDIO_ASRCO4>, + <&audsys CLK_AUDIO_AFE>, + <&audsys CLK_AUDIO_AFE_CONN>, + <&audsys CLK_AUDIO_A1SYS>, + <&audsys CLK_AUDIO_A2SYS>; + + clock-names = "infra_sys_audio_clk", + "top_audio_mux1_sel", + "top_audio_mux2_sel", + "top_audio_a1sys_hp", + "top_audio_a2sys_hp", + "i2s0_src_sel", + "i2s1_src_sel", + "i2s2_src_sel", + "i2s3_src_sel", + "i2s0_src_div", + "i2s1_src_div", + "i2s2_src_div", + "i2s3_src_div", + "i2s0_mclk_en", + "i2s1_mclk_en", + "i2s2_mclk_en", + "i2s3_mclk_en", + "i2so0_hop_ck", + "i2so1_hop_ck", + "i2so2_hop_ck", + "i2so3_hop_ck", + "i2si0_hop_ck", + "i2si1_hop_ck", + "i2si2_hop_ck", + "i2si3_hop_ck", + "asrc0_out_ck", + "asrc1_out_ck", + "asrc2_out_ck", + "asrc3_out_ck", + "audio_afe_pd", + "audio_afe_conn_pd", + "audio_a1sys_pd", + "audio_a2sys_pd"; + + assigned-clocks = <&topckgen CLK_TOP_A1SYS_HP_SEL>, + <&topckgen CLK_TOP_A2SYS_HP_SEL>, + <&topckgen CLK_TOP_A1SYS_HP_DIV>, + <&topckgen CLK_TOP_A2SYS_HP_DIV>; + assigned-clock-parents = <&topckgen CLK_TOP_AUD1PLL>, + <&topckgen CLK_TOP_AUD2PLL>; + assigned-clock-rates = <0>, <0>, <49152000>, <45158400>; + }; + }; + mmc0: mmc@11230000 { compatible = "mediatek,mt7622-mmc"; reg = <0 0x11230000 0 0x1000>; @@ -735,6 +824,16 @@ #reset-cells = <1>; }; + hsdma: dma-controller@1b007000 { + compatible = "mediatek,mt7622-hsdma"; + reg = <0 0x1b007000 0 0x1000>; + interrupts = ; + clocks = <ðsys CLK_ETH_HSDMA_EN>; + clock-names = "hsdma"; + power-domains = <&scpsys MT7622_POWER_DOMAIN_ETHSYS>; + #dma-cells = <1>; + }; + eth: ethernet@1b100000 { compatible = "mediatek,mt7622-eth", "mediatek,mt2701-eth", diff --git a/arch/arm64/boot/dts/qcom/Makefile b/arch/arm64/boot/dts/qcom/Makefile index 55ec5ee7f7e8..9319e74b8906 100644 --- a/arch/arm64/boot/dts/qcom/Makefile +++ b/arch/arm64/boot/dts/qcom/Makefile @@ -6,3 +6,4 @@ dtb-$(CONFIG_ARCH_QCOM) += msm8916-mtp.dtb dtb-$(CONFIG_ARCH_QCOM) += msm8992-bullhead-rev-101.dtb dtb-$(CONFIG_ARCH_QCOM) += msm8994-angler-rev-101.dtb dtb-$(CONFIG_ARCH_QCOM) += msm8996-mtp.dtb +dtb-$(CONFIG_ARCH_QCOM) += sdm845-mtp.dtb diff --git a/arch/arm64/boot/dts/qcom/apq8096-db820c-pins.dtsi b/arch/arm64/boot/dts/qcom/apq8096-db820c-pins.dtsi index 24552f19b3fa..6a573875d45a 100644 --- a/arch/arm64/boot/dts/qcom/apq8096-db820c-pins.dtsi +++ b/arch/arm64/boot/dts/qcom/apq8096-db820c-pins.dtsi @@ -36,4 +36,30 @@ drive-strength = <2>; /* 2 MA */ }; }; + + blsp1_uart1_default: blsp1_uart1_default { + mux { + pins = "gpio41", "gpio42", "gpio43", "gpio44"; + function = "blsp_uart2"; + }; + + config { + pins = "gpio41", "gpio42", "gpio43", "gpio44"; + drive-strength = <16>; + bias-disable; + }; + }; + + blsp1_uart1_sleep: blsp1_uart1_sleep { + mux { + pins = "gpio41", "gpio42", "gpio43", "gpio44"; + function = "gpio"; + }; + + config { + pins = "gpio41", "gpio42", "gpio43", "gpio44"; + drive-strength = <2>; + bias-disable; + }; + }; }; diff --git a/arch/arm64/boot/dts/qcom/apq8096-db820c-pmic-pins.dtsi b/arch/arm64/boot/dts/qcom/apq8096-db820c-pmic-pins.dtsi index 59b29ddfb6e9..a6ad3d7fe655 100644 --- a/arch/arm64/boot/dts/qcom/apq8096-db820c-pmic-pins.dtsi +++ b/arch/arm64/boot/dts/qcom/apq8096-db820c-pmic-pins.dtsi @@ -4,7 +4,7 @@ &pm8994_gpios { pinctrl-names = "default"; - pinctrl-0 = <&ls_exp_gpio_f>; + pinctrl-0 = <&ls_exp_gpio_f &bt_en_gpios>; ls_exp_gpio_f: pm8994_gpio5 { pinconf { @@ -14,6 +14,28 @@ }; }; + bt_en_gpios: bt_en_gpios { + pinconf { + pins = "gpio19"; + function = PMIC_GPIO_FUNC_NORMAL; + output-low; + power-source = ; // 1.8V + qcom,drive-strength = ; + bias-pull-down; + }; + }; + + wlan_en_gpios: wlan_en_gpios { + pinconf { + pins = "gpio8"; + function = PMIC_GPIO_FUNC_NORMAL; + output-low; + power-source = ; // 1.8V + qcom,drive-strength = ; + bias-pull-down; + }; + }; + volume_up_gpio: pm8996_gpio2 { pinconf { pins = "gpio2"; @@ -26,6 +48,16 @@ }; }; + divclk4_pin_a: divclk4 { + pinconf { + pins = "gpio18"; + function = PMIC_GPIO_FUNC_FUNC2; + + bias-disable; + power-source = ; + }; + }; + usb3_vbus_det_gpio: pm8996_gpio22 { pinconf { pins = "gpio22"; diff --git a/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi b/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi index 1c8f1b86472d..0f829db33efe 100644 --- a/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi +++ b/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi @@ -23,6 +23,7 @@ aliases { serial0 = &blsp2_uart1; serial1 = &blsp2_uart2; + serial2 = &blsp1_uart1; i2c0 = &blsp1_i2c2; i2c1 = &blsp2_i2c1; i2c2 = &blsp2_i2c0; @@ -34,7 +35,36 @@ stdout-path = "serial0:115200n8"; }; + clocks { + divclk4: divclk4 { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <32768>; + clock-output-names = "divclk4"; + + pinctrl-names = "default"; + pinctrl-0 = <&divclk4_pin_a>; + }; + }; + soc { + serial@7570000 { + label = "BT-UART"; + status = "okay"; + pinctrl-names = "default", "sleep"; + pinctrl-0 = <&blsp1_uart1_default>; + pinctrl-1 = <&blsp1_uart1_sleep>; + + bluetooth { + compatible = "qcom,qca6174-bt"; + + /* bt_disable_n gpio */ + enable-gpios = <&pm8994_gpios 19 GPIO_ACTIVE_HIGH>; + + clocks = <&divclk4>; + }; + }; + serial@75b0000 { label = "LS-UART1"; status = "okay"; @@ -87,6 +117,16 @@ pinctrl-0 = <&sdc2_clk_on &sdc2_cmd_on &sdc2_data_on &sdc2_cd_on>; pinctrl-1 = <&sdc2_clk_off &sdc2_cmd_off &sdc2_data_off &sdc2_cd_off>; cd-gpios = <&msmgpio 38 0x1>; + vmmc-supply = <&pm8994_l21>; + vqmmc-supply = <&pm8994_l13>; + status = "okay"; + }; + + phy@627000 { + status = "okay"; + }; + + ufshc@624000 { status = "okay"; }; @@ -139,17 +179,34 @@ pinctrl-0 = <&usb2_vbus_det_gpio>; }; + wlan_en: wlan-en-1-8v { + pinctrl-names = "default"; + pinctrl-0 = <&wlan_en_gpios>; + compatible = "regulator-fixed"; + regulator-name = "wlan-en-regulator"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + + gpio = <&pm8994_gpios 8 0>; + + /* WLAN card specific delay */ + startup-delay-us = <70000>; + enable-active-high; + }; + agnoc@0 { - qcom,pcie@600000 { + pcie@600000 { + status = "okay"; perst-gpio = <&msmgpio 35 GPIO_ACTIVE_LOW>; + vddpe-3v3-supply = <&wlan_en>; }; - qcom,pcie@608000 { + pcie@608000 { status = "okay"; perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>; }; - qcom,pcie@610000 { + pcie@610000 { status = "okay"; perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>; }; diff --git a/arch/arm64/boot/dts/qcom/ipq8074-hk01.dts b/arch/arm64/boot/dts/qcom/ipq8074-hk01.dts index 6a838b5d321e..c13ddee8262b 100644 --- a/arch/arm64/boot/dts/qcom/ipq8074-hk01.dts +++ b/arch/arm64/boot/dts/qcom/ipq8074-hk01.dts @@ -21,6 +21,7 @@ aliases { serial0 = &blsp1_uart5; + serial1 = &blsp1_uart3; }; chosen { @@ -33,20 +34,61 @@ }; soc { - pinctrl@1000000 { - serial_4_pins: serial4_pinmux { - mux { - pins = "gpio23", "gpio24"; - function = "blsp4_uart1"; - bias-disable; - }; + serial@78b3000 { + status = "ok"; + }; + + spi@78b5000 { + status = "ok"; + + m25p80@0 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "jedec,spi-nor"; + reg = <0>; + spi-max-frequency = <50000000>; }; }; - serial@78b3000 { - pinctrl-0 = <&serial_4_pins>; - pinctrl-names = "default"; + serial@78b1000 { + status = "ok"; + }; + + i2c@78b6000 { + status = "ok"; + }; + + dma@7984000 { + status = "ok"; + }; + + nand@79b0000 { + status = "ok"; + + nand@0 { + reg = <0>; + nand-ecc-strength = <4>; + nand-ecc-step-size = <512>; + nand-bus-width = <8>; + }; + }; + + phy@86000 { + status = "ok"; + }; + + phy@8e000 { + status = "ok"; + }; + + pci@20000000 { + status = "ok"; + perst-gpio = <&tlmm 58 0x1>; + }; + + pci@10000000 { status = "ok"; + perst-gpio = <&tlmm 61 0x1>; }; }; }; diff --git a/arch/arm64/boot/dts/qcom/ipq8074.dtsi b/arch/arm64/boot/dts/qcom/ipq8074.dtsi index 2bc5dec5614d..18226980f7c3 100644 --- a/arch/arm64/boot/dts/qcom/ipq8074.dtsi +++ b/arch/arm64/boot/dts/qcom/ipq8074.dtsi @@ -24,7 +24,7 @@ ranges = <0 0 0 0xffffffff>; compatible = "simple-bus"; - pinctrl@1000000 { + tlmm: pinctrl@1000000 { compatible = "qcom,ipq8074-pinctrl"; reg = <0x1000000 0x300000>; interrupts = ; @@ -32,6 +32,45 @@ #gpio-cells = <0x2>; interrupt-controller; #interrupt-cells = <0x2>; + + serial_4_pins: serial4-pinmux { + pins = "gpio23", "gpio24"; + function = "blsp4_uart1"; + drive-strength = <8>; + bias-disable; + }; + + i2c_0_pins: i2c-0-pinmux { + pins = "gpio42", "gpio43"; + function = "blsp1_i2c"; + drive-strength = <8>; + bias-disable; + }; + + spi_0_pins: spi-0-pins { + pins = "gpio38", "gpio39", "gpio40", "gpio41"; + function = "blsp0_spi"; + drive-strength = <8>; + bias-disable; + }; + + hsuart_pins: hsuart-pins { + pins = "gpio46", "gpio47", "gpio48", "gpio49"; + function = "blsp2_uart"; + drive-strength = <8>; + bias-disable; + }; + + qpic_pins: qpic-pins { + pins = "gpio1", "gpio3", "gpio4", + "gpio5", "gpio6", "gpio7", + "gpio8", "gpio10", "gpio11", + "gpio12", "gpio13", "gpio14", + "gpio15", "gpio16", "gpio17"; + function = "qpic"; + drive-strength = <8>; + bias-disable; + }; }; intc: interrupt-controller@b000000 { @@ -122,6 +161,276 @@ clocks = <&gcc GCC_BLSP1_UART5_APPS_CLK>, <&gcc GCC_BLSP1_AHB_CLK>; clock-names = "core", "iface"; + pinctrl-0 = <&serial_4_pins>; + pinctrl-names = "default"; + status = "disabled"; + }; + + blsp_dma: dma@7884000 { + compatible = "qcom,bam-v1.7.0"; + reg = <0x7884000 0x2b000>; + interrupts = ; + clocks = <&gcc GCC_BLSP1_AHB_CLK>; + clock-names = "bam_clk"; + #dma-cells = <1>; + qcom,ee = <0>; + }; + + blsp1_uart1: serial@78af000 { + compatible = "qcom,msm-uartdm-v1.4", "qcom,msm-uartdm"; + reg = <0x78af000 0x200>; + interrupts = ; + clocks = <&gcc GCC_BLSP1_UART1_APPS_CLK>, + <&gcc GCC_BLSP1_AHB_CLK>; + clock-names = "core", "iface"; + status = "disabled"; + }; + + blsp1_uart3: serial@78b1000 { + compatible = "qcom,msm-uartdm-v1.4", "qcom,msm-uartdm"; + reg = <0x78b1000 0x200>; + interrupts = ; + clocks = <&gcc GCC_BLSP1_UART3_APPS_CLK>, + <&gcc GCC_BLSP1_AHB_CLK>; + clock-names = "core", "iface"; + dmas = <&blsp_dma 4>, + <&blsp_dma 5>; + dma-names = "tx", "rx"; + pinctrl-0 = <&hsuart_pins>; + pinctrl-names = "default"; + status = "disabled"; + }; + + blsp1_spi1: spi@78b5000 { + compatible = "qcom,spi-qup-v2.2.1"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x78b5000 0x600>; + interrupts = ; + spi-max-frequency = <50000000>; + clocks = <&gcc GCC_BLSP1_QUP1_SPI_APPS_CLK>, + <&gcc GCC_BLSP1_AHB_CLK>; + clock-names = "core", "iface"; + dmas = <&blsp_dma 12>, <&blsp_dma 13>; + dma-names = "tx", "rx"; + pinctrl-0 = <&spi_0_pins>; + pinctrl-names = "default"; + status = "disabled"; + }; + + blsp1_i2c2: i2c@78b6000 { + compatible = "qcom,i2c-qup-v2.2.1"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x78b6000 0x600>; + interrupts = ; + clocks = <&gcc GCC_BLSP1_AHB_CLK>, + <&gcc GCC_BLSP1_QUP2_I2C_APPS_CLK>; + clock-names = "iface", "core"; + clock-frequency = <400000>; + dmas = <&blsp_dma 15>, <&blsp_dma 14>; + dma-names = "rx", "tx"; + pinctrl-0 = <&i2c_0_pins>; + pinctrl-names = "default"; + status = "disabled"; + }; + + blsp1_i2c3: i2c@78b7000 { + compatible = "qcom,i2c-qup-v2.2.1"; + #address-cells = <1>; + #size-cells = <0>; + reg = <0x78b7000 0x600>; + interrupts = ; + clocks = <&gcc GCC_BLSP1_AHB_CLK>, + <&gcc GCC_BLSP1_QUP3_I2C_APPS_CLK>; + clock-names = "iface", "core"; + clock-frequency = <100000>; + dmas = <&blsp_dma 17>, <&blsp_dma 16>; + dma-names = "rx", "tx"; + status = "disabled"; + }; + + qpic_bam: dma@7984000 { + compatible = "qcom,bam-v1.7.0"; + reg = <0x7984000 0x1a000>; + interrupts = ; + clocks = <&gcc GCC_QPIC_AHB_CLK>; + clock-names = "bam_clk"; + #dma-cells = <1>; + qcom,ee = <0>; + status = "disabled"; + }; + + qpic_nand: nand@79b0000 { + compatible = "qcom,ipq8074-nand"; + reg = <0x79b0000 0x10000>; + #address-cells = <1>; + #size-cells = <0>; + clocks = <&gcc GCC_QPIC_CLK>, + <&gcc GCC_QPIC_AHB_CLK>; + clock-names = "core", "aon"; + + dmas = <&qpic_bam 0>, + <&qpic_bam 1>, + <&qpic_bam 2>; + dma-names = "tx", "rx", "cmd"; + pinctrl-0 = <&qpic_pins>; + pinctrl-names = "default"; + status = "disabled"; + }; + + pcie_phy0: phy@86000 { + compatible = "qcom,ipq8074-qmp-pcie-phy"; + reg = <0x86000 0x1000>; + #phy-cells = <0>; + clocks = <&gcc GCC_PCIE0_PIPE_CLK>; + clock-names = "pipe_clk"; + clock-output-names = "pcie20_phy0_pipe_clk"; + + resets = <&gcc GCC_PCIE0_PHY_BCR>, + <&gcc GCC_PCIE0PHY_PHY_BCR>; + reset-names = "phy", + "common"; + status = "disabled"; + }; + + pcie0: pci@20000000 { + compatible = "qcom,pcie-ipq8074"; + reg = <0x20000000 0xf1d + 0x20000f20 0xa8 + 0x80000 0x2000 + 0x20100000 0x1000>; + reg-names = "dbi", "elbi", "parf", "config"; + device_type = "pci"; + linux,pci-domain = <0>; + bus-range = <0x00 0xff>; + num-lanes = <1>; + #address-cells = <3>; + #size-cells = <2>; + + phys = <&pcie_phy0>; + phy-names = "pciephy"; + + ranges = <0x81000000 0 0x20200000 0x20200000 + 0 0x100000 /* downstream I/O */ + 0x82000000 0 0x20300000 0x20300000 + 0 0xd00000>; /* non-prefetchable memory */ + + interrupts = ; + interrupt-names = "msi"; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0x7>; + interrupt-map = <0 0 0 1 &intc 0 75 + IRQ_TYPE_LEVEL_HIGH>, /* int_a */ + <0 0 0 2 &intc 0 78 + IRQ_TYPE_LEVEL_HIGH>, /* int_b */ + <0 0 0 3 &intc 0 79 + IRQ_TYPE_LEVEL_HIGH>, /* int_c */ + <0 0 0 4 &intc 0 83 + IRQ_TYPE_LEVEL_HIGH>; /* int_d */ + + clocks = <&gcc GCC_SYS_NOC_PCIE0_AXI_CLK>, + <&gcc GCC_PCIE0_AXI_M_CLK>, + <&gcc GCC_PCIE0_AXI_S_CLK>, + <&gcc GCC_PCIE0_AHB_CLK>, + <&gcc GCC_PCIE0_AUX_CLK>; + + clock-names = "iface", + "axi_m", + "axi_s", + "ahb", + "aux"; + resets = <&gcc GCC_PCIE0_PIPE_ARES>, + <&gcc GCC_PCIE0_SLEEP_ARES>, + <&gcc GCC_PCIE0_CORE_STICKY_ARES>, + <&gcc GCC_PCIE0_AXI_MASTER_ARES>, + <&gcc GCC_PCIE0_AXI_SLAVE_ARES>, + <&gcc GCC_PCIE0_AHB_ARES>, + <&gcc GCC_PCIE0_AXI_MASTER_STICKY_ARES>; + reset-names = "pipe", + "sleep", + "sticky", + "axi_m", + "axi_s", + "ahb", + "axi_m_sticky"; + status = "disabled"; + }; + + pcie_phy1: phy@8e000 { + compatible = "qcom,ipq8074-qmp-pcie-phy"; + reg = <0x8e000 0x1000>; + #phy-cells = <0>; + clocks = <&gcc GCC_PCIE1_PIPE_CLK>; + clock-names = "pipe_clk"; + clock-output-names = "pcie20_phy1_pipe_clk"; + + resets = <&gcc GCC_PCIE1_PHY_BCR>, + <&gcc GCC_PCIE1PHY_PHY_BCR>; + reset-names = "phy", + "common"; + status = "disabled"; + }; + + pcie1: pci@10000000 { + compatible = "qcom,pcie-ipq8074"; + reg = <0x10000000 0xf1d + 0x10000f20 0xa8 + 0x88000 0x2000 + 0x10100000 0x1000>; + reg-names = "dbi", "elbi", "parf", "config"; + device_type = "pci"; + linux,pci-domain = <1>; + bus-range = <0x00 0xff>; + num-lanes = <1>; + #address-cells = <3>; + #size-cells = <2>; + + phys = <&pcie_phy1>; + phy-names = "pciephy"; + + ranges = <0x81000000 0 0x10200000 0x10200000 + 0 0x100000 /* downstream I/O */ + 0x82000000 0 0x10300000 0x10300000 + 0 0xd00000>; /* non-prefetchable memory */ + + interrupts = ; + interrupt-names = "msi"; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0x7>; + interrupt-map = <0 0 0 1 &intc 0 142 + IRQ_TYPE_LEVEL_HIGH>, /* int_a */ + <0 0 0 2 &intc 0 143 + IRQ_TYPE_LEVEL_HIGH>, /* int_b */ + <0 0 0 3 &intc 0 144 + IRQ_TYPE_LEVEL_HIGH>, /* int_c */ + <0 0 0 4 &intc 0 145 + IRQ_TYPE_LEVEL_HIGH>; /* int_d */ + + clocks = <&gcc GCC_SYS_NOC_PCIE1_AXI_CLK>, + <&gcc GCC_PCIE1_AXI_M_CLK>, + <&gcc GCC_PCIE1_AXI_S_CLK>, + <&gcc GCC_PCIE1_AHB_CLK>, + <&gcc GCC_PCIE1_AUX_CLK>; + clock-names = "iface", + "axi_m", + "axi_s", + "ahb", + "aux"; + resets = <&gcc GCC_PCIE1_PIPE_ARES>, + <&gcc GCC_PCIE1_SLEEP_ARES>, + <&gcc GCC_PCIE1_CORE_STICKY_ARES>, + <&gcc GCC_PCIE1_AXI_MASTER_ARES>, + <&gcc GCC_PCIE1_AXI_SLAVE_ARES>, + <&gcc GCC_PCIE1_AHB_ARES>, + <&gcc GCC_PCIE1_AXI_MASTER_STICKY_ARES>; + reset-names = "pipe", + "sleep", + "sticky", + "axi_m", + "axi_s", + "ahb", + "axi_m_sticky"; status = "disabled"; }; }; @@ -175,7 +484,7 @@ pmu { compatible = "arm,armv8-pmuv3"; - interrupts = ; + interrupts = ; }; clocks { diff --git a/arch/arm64/boot/dts/qcom/msm8916.dtsi b/arch/arm64/boot/dts/qcom/msm8916.dtsi index 66b318e1de80..650f356f69ca 100644 --- a/arch/arm64/boot/dts/qcom/msm8916.dtsi +++ b/arch/arm64/boot/dts/qcom/msm8916.dtsi @@ -179,7 +179,7 @@ pmu { compatible = "arm,cortex-a53-pmu"; - interrupts = ; + interrupts = ; }; thermal-zones { @@ -512,7 +512,7 @@ blsp_i2c2: i2c@78b6000 { compatible = "qcom,i2c-qup-v2.2.1"; reg = <0x078b6000 0x500>; - interrupts = ; + interrupts = ; clocks = <&gcc GCC_BLSP1_AHB_CLK>, <&gcc GCC_BLSP1_QUP2_I2C_APPS_CLK>; clock-names = "iface", "core"; @@ -527,7 +527,7 @@ blsp_i2c4: i2c@78b8000 { compatible = "qcom,i2c-qup-v2.2.1"; reg = <0x078b8000 0x500>; - interrupts = ; + interrupts = ; clocks = <&gcc GCC_BLSP1_AHB_CLK>, <&gcc GCC_BLSP1_QUP4_I2C_APPS_CLK>; clock-names = "iface", "core"; @@ -542,7 +542,7 @@ blsp_i2c6: i2c@78ba000 { compatible = "qcom,i2c-qup-v2.2.1"; reg = <0x078ba000 0x500>; - interrupts = ; + interrupts = ; clocks = <&gcc GCC_BLSP1_AHB_CLK>, <&gcc GCC_BLSP1_QUP6_I2C_APPS_CLK>; clock-names = "iface", "core"; @@ -574,7 +574,7 @@ "mi2s-bit-clk3"; #sound-dai-cells = <1>; - interrupts = <0 160 0>; + interrupts = <0 160 IRQ_TYPE_LEVEL_HIGH>; interrupt-names = "lpass-irq-lpaif"; reg = <0x07708000 0x10000>; reg-names = "lpass-lpaif"; @@ -594,7 +594,7 @@ reg = <0x07824900 0x11c>, <0x07824000 0x800>; reg-names = "hc_mem", "core_mem"; - interrupts = <0 123 0>, <0 138 0>; + interrupts = <0 123 IRQ_TYPE_LEVEL_HIGH>, <0 138 IRQ_TYPE_LEVEL_HIGH>; interrupt-names = "hc_irq", "pwr_irq"; clocks = <&gcc GCC_SDCC1_APPS_CLK>, <&gcc GCC_SDCC1_AHB_CLK>, @@ -611,7 +611,7 @@ reg = <0x07864900 0x11c>, <0x07864000 0x800>; reg-names = "hc_mem", "core_mem"; - interrupts = <0 125 0>, <0 221 0>; + interrupts = <0 125 IRQ_TYPE_LEVEL_HIGH>, <0 221 IRQ_TYPE_LEVEL_HIGH>; interrupt-names = "hc_irq", "pwr_irq"; clocks = <&gcc GCC_SDCC2_APPS_CLK>, <&gcc GCC_SDCC2_AHB_CLK>, @@ -818,7 +818,7 @@ iommu-ctx@2000 { compatible = "qcom,msm-iommu-v1-ns"; reg = <0x2000 0x1000>; - interrupts = ; + interrupts = ; }; }; @@ -862,7 +862,7 @@ "bus_clk", "vsync_clk"; - interrupts = <0 72 0>; + interrupts = <0 72 IRQ_TYPE_LEVEL_HIGH>; interrupt-controller; #interrupt-cells = <1>; diff --git a/arch/arm64/boot/dts/qcom/msm8992-bullhead-rev-101.dts b/arch/arm64/boot/dts/qcom/msm8992-bullhead-rev-101.dts index 454213391671..8c69516f97ed 100644 --- a/arch/arm64/boot/dts/qcom/msm8992-bullhead-rev-101.dts +++ b/arch/arm64/boot/dts/qcom/msm8992-bullhead-rev-101.dts @@ -38,4 +38,21 @@ pinctrl-1 = <&blsp1_uart2_sleep>; }; }; + + reserved-memory { + #address-cells = <2>; + #size-cells = <2>; + ranges; + + ramoops@1ff00000 { + compatible = "ramoops"; + reg = <0x0 0x1ff00000 0x0 0x40000>; + console-size = <0x10000>; + record-size = <0x10000>; + ftrace-size = <0x10000>; + pmsg-size = <0x20000>; + }; + }; }; + +#include "msm8994-smd-rpm.dtsi" diff --git a/arch/arm64/boot/dts/qcom/msm8992-pins.dtsi b/arch/arm64/boot/dts/qcom/msm8992-pins.dtsi index d2a26f0f8d73..31bc9d98e31f 100644 --- a/arch/arm64/boot/dts/qcom/msm8992-pins.dtsi +++ b/arch/arm64/boot/dts/qcom/msm8992-pins.dtsi @@ -35,4 +35,64 @@ bias-pull-down; }; }; + + /* 0-3 for sdc1 4-6 for sdc2 */ + /* Order of pins */ + /* SDC1: CLK -> 0, CMD -> 1, DATA -> 2, RCLK -> 3 */ + /* SDC2: CLK -> 4, CMD -> 5, DATA -> 6 */ + sdc1_clk_on: clk-on { + pinconf { + pins = "sdc1_clk"; + bias-disable = <0>; /* No pull */ + drive-strength = <16>; /* 16mA */ + }; + }; + + sdc1_clk_off: clk-off { + pinconf { + pins = "sdc1_clk"; + bias-disable = <0>; /* No pull */ + drive-strength = <2>; /* 2mA */ + }; + }; + + sdc1_cmd_on: cmd-on { + pinconf { + pins = "sdc1_cmd"; + bias-pull-up; + drive-strength = <8>; + }; + }; + + sdc1_cmd_off: cmd-off { + pinconf { + pins = "sdc1_cmd"; + bias-pull-up = <0x3>; /* same as 3.10 ?? */ + drive-strength = <2>; /* 2mA */ + }; + }; + + sdc1_data_on: data-on { + pinconf { + pins = "sdc1_data"; + bias-pull-up; + drive-strength = <8>; /* 8mA */ + }; + }; + + sdc1_data_off: data-off { + pinconf { + pins = "sdc1_data"; + bias-pull-up; + drive-strength = <2>; + }; + }; + + sdc1_rclk_on: rclk-on { + bias-pull-down; /* pull down */ + }; + + sdc1_rclk_off: rclk-off { + bias-pull-down; /* pull down */ + }; }; diff --git a/arch/arm64/boot/dts/qcom/msm8992.dtsi b/arch/arm64/boot/dts/qcom/msm8992.dtsi index 171578747ed0..cf5cacdd624d 100644 --- a/arch/arm64/boot/dts/qcom/msm8992.dtsi +++ b/arch/arm64/boot/dts/qcom/msm8992.dtsi @@ -202,6 +202,31 @@ reg = <0xfc400000 0x2000>; }; + sdhci1: mmc@f9824900 { + compatible = "qcom,sdhci-msm-v4"; + reg = <0xf9824900 0x1a0>, <0xf9824000 0x800>; + reg-names = "hc_mem", "core_mem"; + + interrupts = , + ; + interrupt-names = "hc_irq", "pwr_irq"; + + clocks = <&clock_gcc GCC_SDCC1_APPS_CLK>, + <&clock_gcc GCC_SDCC1_AHB_CLK>; + clock-names = "core", "iface"; + + pinctrl-names = "default", "sleep"; + pinctrl-0 = <&sdc1_clk_on &sdc1_cmd_on &sdc1_data_on + &sdc1_rclk_on>; + pinctrl-1 = <&sdc1_clk_off &sdc1_cmd_off &sdc1_data_off + &sdc1_rclk_off>; + + regulator-always-on; + bus-width = <8>; + mmc-hs400-1_8v; + status = "okay"; + }; + rpm_msg_ram: memory@fc428000 { compatible = "qcom,rpm-msg-ram"; reg = <0xfc428000 0x4000>; @@ -231,7 +256,67 @@ }; }; + smd_rpm: smd { + compatible = "qcom,smd"; + rpm { + interrupts = ; + qcom,ipc = <&apcs 8 0>; + qcom,smd-edge = <15>; + qcom,local-pid = <0>; + qcom,remote-pid = <6>; + + rpm-requests { + compatible = "qcom,rpm-msm8994"; + qcom,smd-channels = "rpm_requests"; + + pm8994-regulators { + compatible = "qcom,rpm-pm8994-regulators"; + + pm8994_s1: s1 {}; + pm8994_s2: s2 {}; + pm8994_s3: s3 {}; + pm8994_s4: s4 {}; + pm8994_s5: s5 {}; + pm8994_s6: s6 {}; + pm8994_s7: s7 {}; + + pm8994_l1: l1 {}; + pm8994_l2: l2 {}; + pm8994_l3: l3 {}; + pm8994_l4: l4 {}; + pm8994_l6: l6 {}; + pm8994_l8: l8 {}; + pm8994_l9: l9 {}; + pm8994_l10: l10 {}; + pm8994_l11: l11 {}; + pm8994_l12: l12 {}; + pm8994_l13: l13 {}; + pm8994_l14: l14 {}; + pm8994_l15: l15 {}; + pm8994_l16: l16 {}; + pm8994_l17: l17 {}; + pm8994_l18: l18 {}; + pm8994_l19: l19 {}; + pm8994_l20: l20 {}; + pm8994_l21: l21 {}; + pm8994_l22: l22 {}; + pm8994_l23: l23 {}; + pm8994_l24: l24 {}; + pm8994_l25: l25 {}; + pm8994_l26: l26 {}; + pm8994_l27: l27 {}; + pm8994_l28: l28 {}; + pm8994_l29: l29 {}; + pm8994_l30: l30 {}; + pm8994_l31: l31 {}; + pm8994_l32: l32 {}; + + pm8994_lvs1: lvs1 {}; + pm8994_lvs2: lvs2 {}; + }; + }; + }; + }; }; - #include "msm8992-pins.dtsi" diff --git a/arch/arm64/boot/dts/qcom/msm8994-smd-rpm.dtsi b/arch/arm64/boot/dts/qcom/msm8994-smd-rpm.dtsi new file mode 100644 index 000000000000..47ebd16cb680 --- /dev/null +++ b/arch/arm64/boot/dts/qcom/msm8994-smd-rpm.dtsi @@ -0,0 +1,276 @@ +/* Copyright (c) 2015, LGE Inc. All rights reserved. + * Copyright (c) 2016, The Linux Foundation. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 and + * only version 2 as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + */ + +&smd_rpm { + rpm { + rpm_requests { + pm8994-regulators { + + vdd_l1-supply = <&pm8994_s1>; + vdd_l2_26_28-supply = <&pm8994_s3>; + vdd_l3_11-supply = <&pm8994_s3>; + vdd_l4_27_31-supply = <&pm8994_s3>; + vdd_l5_7-supply = <&pm8994_s3>; + vdd_l6_12_32-supply = <&pm8994_s5>; + vdd_l8_16_30-supply = <&vreg_vph_pwr>; + vdd_l9_10_18_22-supply = <&vreg_vph_pwr>; + vdd_l13_19_23_24-supply = <&vreg_vph_pwr>; + vdd_l14_15-supply = <&pm8994_s5>; + vdd_l17_29-supply = <&vreg_vph_pwr>; + vdd_l20_21-supply = <&vreg_vph_pwr>; + vdd_l25-supply = <&pm8994_s5>; + vdd_lvs1_2 = <&pm8994_s4>; + + s1 { + regulator-min-microvolt = <800000>; + regulator-max-microvolt = <800000>; + }; + + s2 { + /* TODO */ + }; + + s3 { + regulator-min-microvolt = <1300000>; + regulator-max-microvolt = <1300000>; + }; + + s4 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-allow-set-load; + regulator-system-load = <325000>; + }; + + s5 { + regulator-min-microvolt = <2150000>; + regulator-max-microvolt = <2150000>; + }; + + s7 { + regulator-min-microvolt = <1000000>; + regulator-max-microvolt = <1000000>; + }; + + l1 { + regulator-min-microvolt = <1000000>; + regulator-max-microvolt = <1000000>; + }; + + l2 { + regulator-min-microvolt = <1250000>; + regulator-max-microvolt = <1250000>; + }; + + l3 { + regulator-min-microvolt = <1200000>; + regulator-max-microvolt = <1200000>; + }; + + l4 { + regulator-min-microvolt = <1225000>; + regulator-max-microvolt = <1225000>; + }; + + l5 { + /* TODO */ + }; + + l6 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + }; + + l7 { + /* TODO */ + }; + + l8 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + }; + + l9 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + }; + + l10 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + qcom,init-voltage = <1800000>; + }; + + l11 { + regulator-min-microvolt = <1200000>; + regulator-max-microvolt = <1200000>; + qcom,init-voltage = <1200000>; + }; + + l12 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + qcom,init-voltage = <1800000>; + proxy-supply = <&pm8994_l12>; + qcom,proxy-consumer-enable; + qcom,proxy-consumer-current = <10000>; + status = "okay"; + }; + + l13 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <2950000>; + qcom,init-voltage = <2950000>; + status = "okay"; + }; + + l14 { + regulator-min-microvolt = <1200000>; + regulator-max-microvolt = <1200000>; + qcom,init-voltage = <1200000>; + proxy-supply = <&pm8994_l14>; + qcom,proxy-consumer-enable; + qcom,proxy-consumer-current = <10000>; + status = "okay"; + }; + + l15 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + qcom,init-voltage = <1800000>; + status = "okay"; + }; + + l16 { + regulator-min-microvolt = <2700000>; + regulator-max-microvolt = <2700000>; + qcom,init-voltage = <2700000>; + status = "okay"; + }; + + l17 { + regulator-min-microvolt = <2700000>; + regulator-max-microvolt = <2700000>; + qcom,init-voltage = <2700000>; + status = "okay"; + }; + + l18 { + regulator-min-microvolt = <3000000>; + regulator-max-microvolt = <3000000>; + regulator-always-on; + qcom,init-voltage = <3000000>; + qcom,init-ldo-mode = <1>; + }; + + l19 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + qcom,init-voltage = <1800000>; + status = "okay"; + }; + + l20 { + regulator-min-microvolt = <2950000>; + regulator-max-microvolt = <2950000>; + regulator-always-on; + regulator-boot-on; + regulator-allow-set-load; + regulator-system-load = <570000>; + }; + + l21 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-always-on; + qcom,init-voltage = <1800000>; + }; + + l22 { + regulator-min-microvolt = <3100000>; + regulator-max-microvolt = <3100000>; + qcom,init-voltage = <3100000>; + }; + + l23 { + regulator-min-microvolt = <2800000>; + regulator-max-microvolt = <2800000>; + qcom,init-voltage = <2800000>; + }; + + l24 { + regulator-min-microvolt = <3075000>; + regulator-max-microvolt = <3150000>; + qcom,init-voltage = <3075000>; + }; + + l25 { + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + qcom,init-voltage = <1800000>; + }; + + l26 { + /* TODO: value from downstream + regulator-min-microvolt = <987500>; + fails to apply */ + }; + + l27 { + regulator-min-microvolt = <1050000>; + regulator-max-microvolt = <1050000>; + qcom,init-voltage = <1050000>; + }; + + l28 { + regulator-min-microvolt = <1000000>; + regulator-max-microvolt = <1000000>; + qcom,init-voltage = <1000000>; + proxy-supply = <&pm8994_l28>; + qcom,proxy-consumer-enable; + qcom,proxy-consumer-current = <10000>; + }; + + l29 { + /* TODO: Unsupported voltage range. + regulator-min-microvolt = <2800000>; + regulator-max-microvolt = <2800000>; + qcom,init-voltage = <2800000>; + */ + }; + + l30 { + /* TODO: get this verified + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + qcom,init-voltage = <1800000>; + */ + }; + + l31 { + regulator-min-microvolt = <1262500>; + regulator-max-microvolt = <1262500>; + qcom,init-voltage = <1262500>; + }; + + l32 { + /* TODO: get this verified + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + qcom,init-voltage = <1800000>; + */ + }; + }; + }; + }; +}; diff --git a/arch/arm64/boot/dts/qcom/msm8996.dtsi b/arch/arm64/boot/dts/qcom/msm8996.dtsi index 410ae787ebb4..8c7f9ca25b53 100644 --- a/arch/arm64/boot/dts/qcom/msm8996.dtsi +++ b/arch/arm64/boot/dts/qcom/msm8996.dtsi @@ -419,6 +419,16 @@ #clock-cells = <1>; }; + blsp1_uart1: serial@7570000 { + compatible = "qcom,msm-uartdm-v1.4", "qcom,msm-uartdm"; + reg = <0x07570000 0x1000>; + interrupts = ; + clocks = <&gcc GCC_BLSP1_UART2_APPS_CLK>, + <&gcc GCC_BLSP1_AHB_CLK>; + clock-names = "core", "iface"; + status = "disabled"; + }; + blsp1_spi0: spi@7575000 { compatible = "qcom,spi-qup-v2.2.1"; reg = <0x07575000 0x600>; @@ -437,7 +447,7 @@ blsp2_i2c0: i2c@75b5000 { compatible = "qcom,i2c-qup-v2.2.1"; reg = <0x075b5000 0x1000>; - interrupts = ; + interrupts = ; clocks = <&gcc GCC_BLSP2_AHB_CLK>, <&gcc GCC_BLSP2_QUP1_I2C_APPS_CLK>; clock-names = "iface", "core"; @@ -468,7 +478,7 @@ blsp2_i2c1: i2c@75b6000 { compatible = "qcom,i2c-qup-v2.2.1"; reg = <0x075b6000 0x1000>; - interrupts = ; + interrupts = ; clocks = <&gcc GCC_BLSP2_AHB_CLK>, <&gcc GCC_BLSP2_QUP2_I2C_APPS_CLK>; clock-names = "iface", "core"; @@ -493,7 +503,7 @@ blsp1_i2c2: i2c@7577000 { compatible = "qcom,i2c-qup-v2.2.1"; reg = <0x07577000 0x1000>; - interrupts = ; + interrupts = ; clocks = <&gcc GCC_BLSP1_AHB_CLK>, <&gcc GCC_BLSP1_QUP3_I2C_APPS_CLK>; clock-names = "iface", "core"; @@ -526,7 +536,8 @@ reg = <0x74a4900 0x314>, <0x74a4000 0x800>; reg-names = "hc_mem", "core_mem"; - interrupts = <0 125 0>, <0 221 0>; + interrupts = <0 125 IRQ_TYPE_LEVEL_HIGH>, + <0 221 IRQ_TYPE_LEVEL_HIGH>; interrupt-names = "hc_irq", "pwr_irq"; clock-names = "iface", "core", "xo"; @@ -623,6 +634,91 @@ #interrupt-cells = <4>; }; + ufsphy: phy@627000 { + compatible = "qcom,msm8996-ufs-phy-qmp-14nm"; + reg = <0x627000 0xda8>; + reg-names = "phy_mem"; + #phy-cells = <0>; + + vdda-phy-supply = <&pm8994_l28>; + vdda-pll-supply = <&pm8994_l12>; + + vdda-phy-max-microamp = <18380>; + vdda-pll-max-microamp = <9440>; + + vddp-ref-clk-supply = <&pm8994_l25>; + vddp-ref-clk-max-microamp = <100>; + vddp-ref-clk-always-on; + + clock-names = "ref_clk_src", "ref_clk"; + clocks = <&rpmcc RPM_SMD_LN_BB_CLK>, + <&gcc GCC_UFS_CLKREF_CLK>; + status = "disabled"; + }; + + ufshc@624000 { + compatible = "qcom,ufshc"; + reg = <0x624000 0x2500>; + interrupts = ; + + phys = <&ufsphy>; + phy-names = "ufsphy"; + + vcc-supply = <&pm8994_l20>; + vccq-supply = <&pm8994_l25>; + vccq2-supply = <&pm8994_s4>; + + vcc-max-microamp = <600000>; + vccq-max-microamp = <450000>; + vccq2-max-microamp = <450000>; + + power-domains = <&gcc UFS_GDSC>; + + clock-names = + "core_clk_src", + "core_clk", + "bus_clk", + "bus_aggr_clk", + "iface_clk", + "core_clk_unipro_src", + "core_clk_unipro", + "core_clk_ice", + "ref_clk", + "tx_lane0_sync_clk", + "rx_lane0_sync_clk"; + clocks = + <&gcc UFS_AXI_CLK_SRC>, + <&gcc GCC_UFS_AXI_CLK>, + <&gcc GCC_SYS_NOC_UFS_AXI_CLK>, + <&gcc GCC_AGGRE2_UFS_AXI_CLK>, + <&gcc GCC_UFS_AHB_CLK>, + <&gcc UFS_ICE_CORE_CLK_SRC>, + <&gcc GCC_UFS_UNIPRO_CORE_CLK>, + <&gcc GCC_UFS_ICE_CORE_CLK>, + <&rpmcc RPM_SMD_LN_BB_CLK>, + <&gcc GCC_UFS_TX_SYMBOL_0_CLK>, + <&gcc GCC_UFS_RX_SYMBOL_0_CLK>; + freq-table-hz = + <100000000 200000000>, + <0 0>, + <0 0>, + <0 0>, + <0 0>, + <150000000 300000000>, + <0 0>, + <0 0>, + <0 0>, + <0 0>, + <0 0>; + + lanes-per-direction = <1>; + status = "disabled"; + + ufs_variant { + compatible = "qcom,ufs_variant"; + }; + }; + mmcc: clock-controller@8c0000 { compatible = "qcom,mmcc-msm8996"; #clock-cells = <1>; @@ -809,7 +905,7 @@ dwc3@7600000 { compatible = "snps,dwc3"; reg = <0x7600000 0xcc00>; - interrupts = <0 138 0>; + interrupts = <0 138 IRQ_TYPE_LEVEL_HIGH>; phys = <&hsusb_phy2>; phy-names = "usb2-phy"; }; @@ -838,7 +934,7 @@ dwc3@6a00000 { compatible = "snps,dwc3"; reg = <0x6a00000 0xcc00>; - interrupts = <0 131 0>; + interrupts = <0 131 IRQ_TYPE_LEVEL_HIGH>; phys = <&hsusb_phy1>, <&ssusb_phy_0>; phy-names = "usb2-phy", "usb3-phy"; }; @@ -851,7 +947,7 @@ #size-cells = <1>; ranges; - pcie0: qcom,pcie@600000 { + pcie0: pcie@600000 { compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; status = "disabled"; power-domains = <&gcc PCIE0_GDSC>; @@ -872,7 +968,7 @@ ranges = <0x01000000 0x0 0x0c200000 0x0c200000 0x0 0x100000>, <0x02000000 0x0 0x0c300000 0x0c300000 0x0 0xd00000>; - interrupts = ; + interrupts = ; interrupt-names = "msi"; #interrupt-cells = <1>; interrupt-map-mask = <0 0 0 0x7>; @@ -904,7 +1000,7 @@ }; - pcie1: qcom,pcie@608000 { + pcie1: pcie@608000 { compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; power-domains = <&gcc PCIE1_GDSC>; bus-range = <0x00 0xff>; @@ -927,7 +1023,7 @@ ranges = <0x01000000 0x0 0x0d200000 0x0d200000 0x0 0x100000>, <0x02000000 0x0 0x0d300000 0x0d300000 0x0 0xd00000>; - interrupts = ; + interrupts = ; interrupt-names = "msi"; #interrupt-cells = <1>; interrupt-map-mask = <0 0 0 0x7>; @@ -957,7 +1053,7 @@ "bus_slave"; }; - pcie2: qcom,pcie@610000 { + pcie2: pcie@610000 { compatible = "qcom,pcie-msm8996", "snps,dw-pcie"; power-domains = <&gcc PCIE2_GDSC>; bus-range = <0x00 0xff>; @@ -980,7 +1076,7 @@ device_type = "pci"; - interrupts = ; + interrupts = ; interrupt-names = "msi"; #interrupt-cells = <1>; interrupt-map-mask = <0 0 0 0x7>; diff --git a/arch/arm64/boot/dts/qcom/sdm845-mtp.dts b/arch/arm64/boot/dts/qcom/sdm845-mtp.dts new file mode 100644 index 000000000000..979ab49913f1 --- /dev/null +++ b/arch/arm64/boot/dts/qcom/sdm845-mtp.dts @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * SDM845 MTP board device tree source + * + * Copyright (c) 2018, The Linux Foundation. All rights reserved. + */ + +/dts-v1/; + +#include "sdm845.dtsi" + +/ { + model = "Qualcomm Technologies, Inc. SDM845 MTP"; + compatible = "qcom,sdm845-mtp"; +}; diff --git a/arch/arm64/boot/dts/qcom/sdm845.dtsi b/arch/arm64/boot/dts/qcom/sdm845.dtsi new file mode 100644 index 000000000000..cdaabeb3c995 --- /dev/null +++ b/arch/arm64/boot/dts/qcom/sdm845.dtsi @@ -0,0 +1,327 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * SDM845 SoC device tree source + * + * Copyright (c) 2018, The Linux Foundation. All rights reserved. + */ + +#include + +/ { + interrupt-parent = <&intc>; + + #address-cells = <2>; + #size-cells = <2>; + + chosen { }; + + memory@80000000 { + device_type = "memory"; + /* We expect the bootloader to fill in the size */ + reg = <0 0x80000000 0 0>; + }; + + reserved-memory { + #address-cells = <2>; + #size-cells = <2>; + ranges; + + memory@85fc0000 { + reg = <0 0x85fc0000 0 0x20000>; + no-map; + }; + + memory@85fe0000 { + compatible = "qcom,cmd-db"; + reg = <0x0 0x85fe0000 0x0 0x20000>; + no-map; + }; + + smem_mem: memory@86000000 { + reg = <0x0 0x86000000 0x0 0x200000>; + no-map; + }; + + memory@86200000 { + reg = <0 0x86200000 0 0x2d00000>; + no-map; + }; + }; + + cpus { + #address-cells = <2>; + #size-cells = <0>; + + CPU0: cpu@0 { + device_type = "cpu"; + compatible = "qcom,kryo385"; + reg = <0x0 0x0>; + enable-method = "psci"; + next-level-cache = <&L2_0>; + L2_0: l2-cache { + compatible = "cache"; + next-level-cache = <&L3_0>; + L3_0: l3-cache { + compatible = "cache"; + }; + }; + }; + + CPU1: cpu@100 { + device_type = "cpu"; + compatible = "qcom,kryo385"; + reg = <0x0 0x100>; + enable-method = "psci"; + next-level-cache = <&L2_100>; + L2_100: l2-cache { + compatible = "cache"; + next-level-cache = <&L3_0>; + }; + }; + + CPU2: cpu@200 { + device_type = "cpu"; + compatible = "qcom,kryo385"; + reg = <0x0 0x200>; + enable-method = "psci"; + next-level-cache = <&L2_200>; + L2_200: l2-cache { + compatible = "cache"; + next-level-cache = <&L3_0>; + }; + }; + + CPU3: cpu@300 { + device_type = "cpu"; + compatible = "qcom,kryo385"; + reg = <0x0 0x300>; + enable-method = "psci"; + next-level-cache = <&L2_300>; + L2_300: l2-cache { + compatible = "cache"; + next-level-cache = <&L3_0>; + }; + }; + + CPU4: cpu@400 { + device_type = "cpu"; + compatible = "qcom,kryo385"; + reg = <0x0 0x400>; + enable-method = "psci"; + next-level-cache = <&L2_400>; + L2_400: l2-cache { + compatible = "cache"; + next-level-cache = <&L3_0>; + }; + }; + + CPU5: cpu@500 { + device_type = "cpu"; + compatible = "qcom,kryo385"; + reg = <0x0 0x500>; + enable-method = "psci"; + next-level-cache = <&L2_500>; + L2_500: l2-cache { + compatible = "cache"; + next-level-cache = <&L3_0>; + }; + }; + + CPU6: cpu@600 { + device_type = "cpu"; + compatible = "qcom,kryo385"; + reg = <0x0 0x600>; + enable-method = "psci"; + next-level-cache = <&L2_600>; + L2_600: l2-cache { + compatible = "cache"; + next-level-cache = <&L3_0>; + }; + }; + + CPU7: cpu@700 { + device_type = "cpu"; + compatible = "qcom,kryo385"; + reg = <0x0 0x700>; + enable-method = "psci"; + next-level-cache = <&L2_700>; + L2_700: l2-cache { + compatible = "cache"; + next-level-cache = <&L3_0>; + }; + }; + }; + + timer { + compatible = "arm,armv8-timer"; + interrupts = , + , + , + ; + }; + + clocks { + xo_board: xo-board { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <38400000>; + clock-output-names = "xo_board"; + }; + + sleep_clk: sleep-clk { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <32764>; + }; + }; + + tcsr_mutex: hwlock { + compatible = "qcom,tcsr-mutex"; + syscon = <&tcsr_mutex_regs 0 0x1000>; + #hwlock-cells = <1>; + }; + + smem { + compatible = "qcom,smem"; + memory-region = <&smem_mem>; + hwlocks = <&tcsr_mutex 3>; + }; + + psci { + compatible = "arm,psci-1.0"; + method = "smc"; + }; + + soc: soc { + #address-cells = <1>; + #size-cells = <1>; + ranges = <0 0 0 0xffffffff>; + compatible = "simple-bus"; + + gcc: clock-controller@100000 { + compatible = "qcom,gcc-sdm845"; + reg = <0x100000 0x1f0000>; + #clock-cells = <1>; + #reset-cells = <1>; + #power-domain-cells = <1>; + }; + + tcsr_mutex_regs: syscon@1f40000 { + compatible = "syscon"; + reg = <0x1f40000 0x40000>; + }; + + tlmm: pinctrl@3400000 { + compatible = "qcom,sdm845-pinctrl"; + reg = <0x03400000 0xc00000>; + interrupts = ; + gpio-controller; + #gpio-cells = <2>; + interrupt-controller; + #interrupt-cells = <2>; + }; + + spmi_bus: spmi@c440000 { + compatible = "qcom,spmi-pmic-arb"; + reg = <0xc440000 0x1100>, + <0xc600000 0x2000000>, + <0xe600000 0x100000>, + <0xe700000 0xa0000>, + <0xc40a000 0x26000>; + reg-names = "core", "chnls", "obsrvr", "intr", "cnfg"; + interrupt-names = "periph_irq"; + interrupts = ; + qcom,ee = <0>; + qcom,channel = <0>; + #address-cells = <2>; + #size-cells = <0>; + interrupt-controller; + #interrupt-cells = <4>; + cell-index = <0>; + }; + + apss_shared: mailbox@17990000 { + compatible = "qcom,sdm845-apss-shared"; + reg = <0x17990000 0x1000>; + #mbox-cells = <1>; + }; + + intc: interrupt-controller@17a00000 { + compatible = "arm,gic-v3"; + #address-cells = <1>; + #size-cells = <1>; + ranges; + #interrupt-cells = <3>; + interrupt-controller; + reg = <0x17a00000 0x10000>, /* GICD */ + <0x17a60000 0x100000>; /* GICR * 8 */ + interrupts = ; + + gic-its@17a40000 { + compatible = "arm,gic-v3-its"; + msi-controller; + #msi-cells = <1>; + reg = <0x17a40000 0x20000>; + status = "disabled"; + }; + }; + + timer@17c90000 { + #address-cells = <1>; + #size-cells = <1>; + ranges; + compatible = "arm,armv7-timer-mem"; + reg = <0x17c90000 0x1000>; + + frame@17ca0000 { + frame-number = <0>; + interrupts = , + ; + reg = <0x17ca0000 0x1000>, + <0x17cb0000 0x1000>; + }; + + frame@17cc0000 { + frame-number = <1>; + interrupts = ; + reg = <0x17cc0000 0x1000>; + status = "disabled"; + }; + + frame@17cd0000 { + frame-number = <2>; + interrupts = ; + reg = <0x17cd0000 0x1000>; + status = "disabled"; + }; + + frame@17ce0000 { + frame-number = <3>; + interrupts = ; + reg = <0x17ce0000 0x1000>; + status = "disabled"; + }; + + frame@17cf0000 { + frame-number = <4>; + interrupts = ; + reg = <0x17cf0000 0x1000>; + status = "disabled"; + }; + + frame@17d00000 { + frame-number = <5>; + interrupts = ; + reg = <0x17d00000 0x1000>; + status = "disabled"; + }; + + frame@17d10000 { + frame-number = <6>; + interrupts = ; + reg = <0x17d10000 0x1000>; + status = "disabled"; + }; + }; + }; +}; diff --git a/arch/arm64/boot/dts/renesas/Makefile b/arch/arm64/boot/dts/renesas/Makefile index 5ede06000ea4..9e2394bc3c62 100644 --- a/arch/arm64/boot/dts/renesas/Makefile +++ b/arch/arm64/boot/dts/renesas/Makefile @@ -9,5 +9,6 @@ dtb-$(CONFIG_ARCH_R8A7796) += r8a7796-m3ulcb-kf.dtb dtb-$(CONFIG_ARCH_R8A7796) += r8a7796-salvator-xs.dtb dtb-$(CONFIG_ARCH_R8A77965) += r8a77965-salvator-x.dtb r8a77965-salvator-xs.dtb dtb-$(CONFIG_ARCH_R8A77970) += r8a77970-eagle.dtb r8a77970-v3msk.dtb -dtb-$(CONFIG_ARCH_R8A77980) += r8a77980-condor.dtb +dtb-$(CONFIG_ARCH_R8A77980) += r8a77980-condor.dtb r8a77980-v3hsk.dtb +dtb-$(CONFIG_ARCH_R8A77990) += r8a77990-ebisu.dtb dtb-$(CONFIG_ARCH_R8A77995) += r8a77995-draak.dtb diff --git a/arch/arm64/boot/dts/renesas/r8a7795-es1-salvator-x.dts b/arch/arm64/boot/dts/renesas/r8a7795-es1-salvator-x.dts index 7f2a3d923f21..3f46345a4644 100644 --- a/arch/arm64/boot/dts/renesas/r8a7795-es1-salvator-x.dts +++ b/arch/arm64/boot/dts/renesas/r8a7795-es1-salvator-x.dts @@ -56,6 +56,12 @@ status = "okay"; }; +&sound_card { + dais = <&rsnd_port0 /* ak4613 */ + &rsnd_port1 /* HDMI0 */ + &rsnd_port2>; /* HDMI1 */ +}; + &hdmi0 { status = "okay"; @@ -66,6 +72,12 @@ remote-endpoint = <&hdmi0_con>; }; }; + port@2 { + reg = <2>; + dw_hdmi0_snd_in: endpoint { + remote-endpoint = <&rsnd_endpoint1>; + }; + }; }; }; @@ -83,6 +95,12 @@ remote-endpoint = <&hdmi1_con>; }; }; + port@2 { + reg = <2>; + dw_hdmi1_snd_in: endpoint { + remote-endpoint = <&rsnd_endpoint2>; + }; + }; }; }; @@ -94,6 +112,34 @@ status = "okay"; }; +&rcar_sound { + ports { + /* rsnd_port0 is on salvator-common */ + rsnd_port1: port@1 { + rsnd_endpoint1: endpoint { + remote-endpoint = <&dw_hdmi0_snd_in>; + + dai-format = "i2s"; + bitclock-master = <&rsnd_endpoint1>; + frame-master = <&rsnd_endpoint1>; + + playback = <&ssi2>; + }; + }; + rsnd_port2: port@2 { + rsnd_endpoint2: endpoint { + remote-endpoint = <&dw_hdmi1_snd_in>; + + dai-format = "i2s"; + bitclock-master = <&rsnd_endpoint2>; + frame-master = <&rsnd_endpoint2>; + + playback = <&ssi3>; + }; + }; + }; +}; + &pfc { usb2_pins: usb2 { groups = "usb2"; diff --git a/arch/arm64/boot/dts/renesas/r8a7795-es1.dtsi b/arch/arm64/boot/dts/renesas/r8a7795-es1.dtsi index f9acd125d687..e19dcd6cb767 100644 --- a/arch/arm64/boot/dts/renesas/r8a7795-es1.dtsi +++ b/arch/arm64/boot/dts/renesas/r8a7795-es1.dtsi @@ -39,7 +39,6 @@ reg = <0 0xe7730000 0 0x1000>; renesas,ipmmu-main = <&ipmmu_mm 8>; #iommu-cells = <1>; - status = "disabled"; }; /delete-node/ usb-phy@ee0e0200; @@ -108,6 +107,61 @@ resets = <&cpg 117>; renesas,fcp = <&fcpf2>; }; + + csi21: csi2@fea90000 { + compatible = "renesas,r8a7795-csi2"; + reg = <0 0xfea90000 0 0x10000>; + interrupts = ; + clocks = <&cpg CPG_MOD 713>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 713>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + csi21vin0: endpoint@0 { + reg = <0>; + remote-endpoint = <&vin0csi21>; + }; + csi21vin1: endpoint@1 { + reg = <1>; + remote-endpoint = <&vin1csi21>; + }; + csi21vin2: endpoint@2 { + reg = <2>; + remote-endpoint = <&vin2csi21>; + }; + csi21vin3: endpoint@3 { + reg = <3>; + remote-endpoint = <&vin3csi21>; + }; + csi21vin4: endpoint@4 { + reg = <4>; + remote-endpoint = <&vin4csi21>; + }; + csi21vin5: endpoint@5 { + reg = <5>; + remote-endpoint = <&vin5csi21>; + }; + csi21vin6: endpoint@6 { + reg = <6>; + remote-endpoint = <&vin6csi21>; + }; + csi21vin7: endpoint@7 { + reg = <7>; + remote-endpoint = <&vin7csi21>; + }; + }; + }; + }; }; &gpio1 { @@ -175,3 +229,91 @@ &du { vsps = <&vspd0 &vspd1 &vspd2 &vspd3>; }; + +&vin0 { + ports { + port@1 { + vin0csi21: endpoint@1 { + reg = <1>; + remote-endpoint= <&csi21vin0>; + }; + }; + }; +}; + +&vin1 { + ports { + port@1 { + vin1csi21: endpoint@1 { + reg = <1>; + remote-endpoint= <&csi21vin1>; + }; + }; + }; +}; + +&vin2 { + ports { + port@1 { + vin2csi21: endpoint@1 { + reg = <1>; + remote-endpoint= <&csi21vin2>; + }; + }; + }; +}; + +&vin3 { + ports { + port@1 { + vin3csi21: endpoint@1 { + reg = <1>; + remote-endpoint= <&csi21vin3>; + }; + }; + }; +}; + +&vin4 { + ports { + port@1 { + vin4csi21: endpoint@1 { + reg = <1>; + remote-endpoint= <&csi21vin4>; + }; + }; + }; +}; + +&vin5 { + ports { + port@1 { + vin5csi21: endpoint@1 { + reg = <1>; + remote-endpoint= <&csi21vin5>; + }; + }; + }; +}; + +&vin6 { + ports { + port@1 { + vin6csi21: endpoint@1 { + reg = <1>; + remote-endpoint= <&csi21vin6>; + }; + }; + }; +}; + +&vin7 { + ports { + port@1 { + vin7csi21: endpoint@1 { + reg = <1>; + remote-endpoint= <&csi21vin7>; + }; + }; + }; +}; diff --git a/arch/arm64/boot/dts/renesas/r8a7795-salvator-x.dts b/arch/arm64/boot/dts/renesas/r8a7795-salvator-x.dts index af467419266a..0efbef5ea9b7 100644 --- a/arch/arm64/boot/dts/renesas/r8a7795-salvator-x.dts +++ b/arch/arm64/boot/dts/renesas/r8a7795-salvator-x.dts @@ -56,6 +56,12 @@ status = "okay"; }; +&sound_card { + dais = <&rsnd_port0 /* ak4613 */ + &rsnd_port1 /* HDMI0 */ + &rsnd_port2>; /* HDMI1 */ +}; + &hdmi0 { status = "okay"; @@ -66,6 +72,12 @@ remote-endpoint = <&hdmi0_con>; }; }; + port@2 { + reg = <2>; + dw_hdmi0_snd_in: endpoint { + remote-endpoint = <&rsnd_endpoint1>; + }; + }; }; }; @@ -83,6 +95,12 @@ remote-endpoint = <&hdmi1_con>; }; }; + port@2 { + reg = <2>; + dw_hdmi1_snd_in: endpoint { + remote-endpoint = <&rsnd_endpoint2>; + }; + }; }; }; @@ -94,6 +112,34 @@ status = "okay"; }; +&rcar_sound { + ports { + /* rsnd_port0 is on salvator-common */ + rsnd_port1: port@1 { + rsnd_endpoint1: endpoint { + remote-endpoint = <&dw_hdmi0_snd_in>; + + dai-format = "i2s"; + bitclock-master = <&rsnd_endpoint1>; + frame-master = <&rsnd_endpoint1>; + + playback = <&ssi2>; + }; + }; + rsnd_port2: port@2 { + rsnd_endpoint2: endpoint { + remote-endpoint = <&dw_hdmi1_snd_in>; + + dai-format = "i2s"; + bitclock-master = <&rsnd_endpoint2>; + frame-master = <&rsnd_endpoint2>; + + playback = <&ssi3>; + }; + }; + }; +}; + &pfc { usb2_pins: usb2 { groups = "usb2"; diff --git a/arch/arm64/boot/dts/renesas/r8a7795-salvator-xs.dts b/arch/arm64/boot/dts/renesas/r8a7795-salvator-xs.dts index 8b50ceb746e8..e231b5a7cbab 100644 --- a/arch/arm64/boot/dts/renesas/r8a7795-salvator-xs.dts +++ b/arch/arm64/boot/dts/renesas/r8a7795-salvator-xs.dts @@ -56,6 +56,22 @@ status = "okay"; }; +&ehci3 { + dr_mode = "otg"; + status = "okay"; +}; + +&hsusb3 { + dr_mode = "otg"; + status = "okay"; +}; + +&sound_card { + dais = <&rsnd_port0 /* ak4613 */ + &rsnd_port1 /* HDMI0 */ + &rsnd_port2>; /* HDMI1 */ +}; + &hdmi0 { status = "okay"; @@ -66,6 +82,12 @@ remote-endpoint = <&hdmi0_con>; }; }; + port@2 { + reg = <2>; + dw_hdmi0_snd_in: endpoint { + remote-endpoint = <&rsnd_endpoint1>; + }; + }; }; }; @@ -83,6 +105,12 @@ remote-endpoint = <&hdmi1_con>; }; }; + port@2 { + reg = <2>; + dw_hdmi1_snd_in: endpoint { + remote-endpoint = <&rsnd_endpoint2>; + }; + }; }; }; @@ -94,11 +122,61 @@ status = "okay"; }; +&ohci3 { + dr_mode = "otg"; + status = "okay"; +}; + +&rcar_sound { + ports { + /* rsnd_port0 is on salvator-common */ + rsnd_port1: port@1 { + rsnd_endpoint1: endpoint { + remote-endpoint = <&dw_hdmi0_snd_in>; + + dai-format = "i2s"; + bitclock-master = <&rsnd_endpoint1>; + frame-master = <&rsnd_endpoint1>; + + playback = <&ssi2>; + }; + }; + rsnd_port2: port@2 { + rsnd_endpoint2: endpoint { + remote-endpoint = <&dw_hdmi1_snd_in>; + + dai-format = "i2s"; + bitclock-master = <&rsnd_endpoint2>; + frame-master = <&rsnd_endpoint2>; + + playback = <&ssi3>; + }; + }; + }; +}; + &pfc { usb2_pins: usb2 { groups = "usb2"; function = "usb2"; }; + + /* + * - On Salvator-X[S], GP6_3[01] are connected to ADV7482 as irq pins + * (when SW31 is the default setting on Salvator-XS). + * - If SW31 is the default setting, you cannot use USB2.0 ch3 on + * r8a7795 with Salvator-XS. + * Hence the SW31 setting must be changed like 2) below. + * 1) Default setting of SW31: ON-ON-OFF-OFF-OFF-OFF: + * - Connect GP6_3[01] to ADV7842. + * 2) Changed setting of SW31: OFF-OFF-ON-ON-ON-ON: + * - Connect GP6_3[01] to BD082065 (USB2.0 ch3's host power). + * - Connect GP6_{04,21} to ADV7842. + */ + usb2_ch3_pins: usb2_ch3 { + groups = "usb2_ch3"; + function = "usb2_ch3"; + }; }; &usb2_phy2 { @@ -107,3 +185,10 @@ status = "okay"; }; + +&usb2_phy3 { + pinctrl-0 = <&usb2_ch3_pins>; + pinctrl-names = "default"; + + status = "okay"; +}; diff --git a/arch/arm64/boot/dts/renesas/r8a7795.dtsi b/arch/arm64/boot/dts/renesas/r8a7795.dtsi index 1d5e3ac0231c..d842940b2f43 100644 --- a/arch/arm64/boot/dts/renesas/r8a7795.dtsi +++ b/arch/arm64/boot/dts/renesas/r8a7795.dtsi @@ -30,6 +30,91 @@ i2c7 = &i2c_dvfs; }; + /* + * The external audio clocks are configured as 0 Hz fixed frequency + * clocks by default. + * Boards that provide audio clocks should override them. + */ + audio_clk_a: audio_clk_a { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + }; + + audio_clk_b: audio_clk_b { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + }; + + audio_clk_c: audio_clk_c { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + }; + + /* External CAN clock - to be overridden by boards that provide it */ + can_clk: can { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + }; + + cluster0_opp: opp_table0 { + compatible = "operating-points-v2"; + opp-shared; + + opp-500000000 { + opp-hz = /bits/ 64 <500000000>; + opp-microvolt = <830000>; + clock-latency-ns = <300000>; + }; + opp-1000000000 { + opp-hz = /bits/ 64 <1000000000>; + opp-microvolt = <830000>; + clock-latency-ns = <300000>; + }; + opp-1500000000 { + opp-hz = /bits/ 64 <1500000000>; + opp-microvolt = <830000>; + clock-latency-ns = <300000>; + opp-suspend; + }; + opp-1600000000 { + opp-hz = /bits/ 64 <1600000000>; + opp-microvolt = <900000>; + clock-latency-ns = <300000>; + turbo-mode; + }; + opp-1700000000 { + opp-hz = /bits/ 64 <1700000000>; + opp-microvolt = <960000>; + clock-latency-ns = <300000>; + turbo-mode; + }; + }; + + cluster1_opp: opp_table1 { + compatible = "operating-points-v2"; + opp-shared; + + opp-800000000 { + opp-hz = /bits/ 64 <800000000>; + opp-microvolt = <820000>; + clock-latency-ns = <300000>; + }; + opp-1000000000 { + opp-hz = /bits/ 64 <1000000000>; + opp-microvolt = <820000>; + clock-latency-ns = <300000>; + }; + opp-1200000000 { + opp-hz = /bits/ 64 <1200000000>; + opp-microvolt = <820000>; + clock-latency-ns = <300000>; + }; + }; + cpus { #address-cells = <1>; #size-cells = <0>; @@ -47,7 +132,7 @@ }; a57_1: cpu@1 { - compatible = "arm,cortex-a57","arm,armv8"; + compatible = "arm,cortex-a57", "arm,armv8"; reg = <0x1>; device_type = "cpu"; power-domains = <&sysc R8A7795_PD_CA57_CPU1>; @@ -59,7 +144,7 @@ }; a57_2: cpu@2 { - compatible = "arm,cortex-a57","arm,armv8"; + compatible = "arm,cortex-a57", "arm,armv8"; reg = <0x2>; device_type = "cpu"; power-domains = <&sysc R8A7795_PD_CA57_CPU2>; @@ -71,7 +156,7 @@ }; a57_3: cpu@3 { - compatible = "arm,cortex-a57","arm,armv8"; + compatible = "arm,cortex-a57", "arm,armv8"; reg = <0x3>; device_type = "cpu"; power-domains = <&sysc R8A7795_PD_CA57_CPU3>; @@ -94,7 +179,7 @@ }; a53_1: cpu@101 { - compatible = "arm,cortex-a53","arm,armv8"; + compatible = "arm,cortex-a53", "arm,armv8"; reg = <0x101>; device_type = "cpu"; power-domains = <&sysc R8A7795_PD_CA53_CPU1>; @@ -105,7 +190,7 @@ }; a53_2: cpu@102 { - compatible = "arm,cortex-a53","arm,armv8"; + compatible = "arm,cortex-a53", "arm,armv8"; reg = <0x102>; device_type = "cpu"; power-domains = <&sysc R8A7795_PD_CA53_CPU2>; @@ -116,7 +201,7 @@ }; a53_3: cpu@103 { - compatible = "arm,cortex-a53","arm,armv8"; + compatible = "arm,cortex-a53", "arm,armv8"; reg = <0x103>; device_type = "cpu"; power-domains = <&sysc R8A7795_PD_CA53_CPU3>; @@ -155,91 +240,6 @@ clock-frequency = <0>; }; - /* - * The external audio clocks are configured as 0 Hz fixed frequency - * clocks by default. - * Boards that provide audio clocks should override them. - */ - audio_clk_a: audio_clk_a { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; - }; - - audio_clk_b: audio_clk_b { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; - }; - - audio_clk_c: audio_clk_c { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; - }; - - /* External CAN clock - to be overridden by boards that provide it */ - can_clk: can { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; - }; - - cluster0_opp: opp_table0 { - compatible = "operating-points-v2"; - opp-shared; - - opp-500000000 { - opp-hz = /bits/ 64 <500000000>; - opp-microvolt = <830000>; - clock-latency-ns = <300000>; - }; - opp-1000000000 { - opp-hz = /bits/ 64 <1000000000>; - opp-microvolt = <830000>; - clock-latency-ns = <300000>; - }; - opp-1500000000 { - opp-hz = /bits/ 64 <1500000000>; - opp-microvolt = <830000>; - clock-latency-ns = <300000>; - opp-suspend; - }; - opp-1600000000 { - opp-hz = /bits/ 64 <1600000000>; - opp-microvolt = <900000>; - clock-latency-ns = <300000>; - turbo-mode; - }; - opp-1700000000 { - opp-hz = /bits/ 64 <1700000000>; - opp-microvolt = <960000>; - clock-latency-ns = <300000>; - turbo-mode; - }; - }; - - cluster1_opp: opp_table1 { - compatible = "operating-points-v2"; - opp-shared; - - opp-800000000 { - opp-hz = /bits/ 64 <800000000>; - opp-microvolt = <820000>; - clock-latency-ns = <300000>; - }; - opp-1000000000 { - opp-hz = /bits/ 64 <1000000000>; - opp-microvolt = <820000>; - clock-latency-ns = <300000>; - }; - opp-1200000000 { - opp-hz = /bits/ 64 <1200000000>; - opp-microvolt = <820000>; - clock-latency-ns = <300000>; - }; - }; - /* External PCIe clock - can be overridden by the board */ pcie_bus_clk: pcie_bus { compatible = "fixed-clock"; @@ -247,18 +247,6 @@ clock-frequency = <0>; }; - pmu_a57 { - compatible = "arm,cortex-a57-pmu"; - interrupts-extended = <&gic GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>, - <&gic GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>, - <&gic GIC_SPI 74 IRQ_TYPE_LEVEL_HIGH>, - <&gic GIC_SPI 75 IRQ_TYPE_LEVEL_HIGH>; - interrupt-affinity = <&a57_0>, - <&a57_1>, - <&a57_2>, - <&a57_3>; - }; - pmu_a53 { compatible = "arm,cortex-a53-pmu"; interrupts-extended = <&gic GIC_SPI 84 IRQ_TYPE_LEVEL_HIGH>, @@ -271,6 +259,18 @@ <&a53_3>; }; + pmu_a57 { + compatible = "arm,cortex-a57-pmu"; + interrupts-extended = <&gic GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 74 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 75 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&a57_0>, + <&a57_1>, + <&a57_2>, + <&a57_3>; + }; + psci { compatible = "arm,psci-1.0", "arm,psci-0.2"; method = "smc"; @@ -291,23 +291,6 @@ #size-cells = <2>; ranges; - gic: interrupt-controller@f1010000 { - compatible = "arm,gic-400"; - #interrupt-cells = <3>; - #address-cells = <0>; - interrupt-controller; - reg = <0x0 0xf1010000 0 0x1000>, - <0x0 0xf1020000 0 0x20000>, - <0x0 0xf1040000 0 0x20000>, - <0x0 0xf1060000 0 0x20000>; - interrupts = ; - clocks = <&cpg CPG_MOD 408>; - clock-names = "clk"; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 408>; - }; - wdt0: watchdog@e6020000 { compatible = "renesas,r8a7795-wdt", "renesas,rcar-gen3-wdt"; reg = <0 0xe6020000 0 0x0c>; @@ -437,6 +420,11 @@ resets = <&cpg 905>; }; + pfc: pin-controller@e6060000 { + compatible = "renesas,pfc-r8a7795"; + reg = <0 0xe6060000 0 0x50c>; + }; + cpg: clock-controller@e6150000 { compatible = "renesas,r8a7795-cpg-mssr"; reg = <0 0xe6150000 0 0x1000>; @@ -452,20 +440,25 @@ reg = <0 0xe6160000 0 0x0200>; }; - prr: chipid@fff00044 { - compatible = "renesas,prr"; - reg = <0 0xfff00044 0 4>; - }; - sysc: system-controller@e6180000 { compatible = "renesas,r8a7795-sysc"; reg = <0 0xe6180000 0 0x0400>; #power-domain-cells = <1>; }; - pfc: pin-controller@e6060000 { - compatible = "renesas,pfc-r8a7795"; - reg = <0 0xe6060000 0 0x50c>; + tsc: thermal@e6198000 { + compatible = "renesas,r8a7795-thermal"; + reg = <0 0xe6198000 0 0x100>, + <0 0xe61a0000 0 0x100>, + <0 0xe61a8000 0 0x100>; + interrupts = , + , + ; + clocks = <&cpg CPG_MOD 522>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 522>; + #thermal-sensor-cells = <1>; + status = "okay"; }; intc_ex: interrupt-controller@e61c0000 { @@ -484,153 +477,326 @@ resets = <&cpg 407>; }; - ipmmu_vi0: mmu@febd0000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xfebd0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 14>; + i2c0: i2c@e6500000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a7795", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe6500000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 931>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; - }; - - ipmmu_vi1: mmu@febe0000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xfebe0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 15>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 931>; + dmas = <&dmac1 0x91>, <&dmac1 0x90>, + <&dmac2 0x91>, <&dmac2 0x90>; + dma-names = "tx", "rx", "tx", "rx"; + i2c-scl-internal-delay-ns = <110>; status = "disabled"; }; - ipmmu_vp0: mmu@fe990000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xfe990000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 16>; - power-domains = <&sysc R8A7795_PD_A3VP>; - #iommu-cells = <1>; + i2c1: i2c@e6508000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a7795", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe6508000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 930>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 930>; + dmas = <&dmac1 0x93>, <&dmac1 0x92>, + <&dmac2 0x93>, <&dmac2 0x92>; + dma-names = "tx", "rx", "tx", "rx"; + i2c-scl-internal-delay-ns = <6>; status = "disabled"; }; - ipmmu_vp1: mmu@fe980000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xfe980000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 17>; - power-domains = <&sysc R8A7795_PD_A3VP>; - #iommu-cells = <1>; + i2c2: i2c@e6510000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a7795", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe6510000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 929>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 929>; + dmas = <&dmac1 0x95>, <&dmac1 0x94>, + <&dmac2 0x95>, <&dmac2 0x94>; + dma-names = "tx", "rx", "tx", "rx"; + i2c-scl-internal-delay-ns = <6>; + status = "disabled"; }; - ipmmu_vc0: mmu@fe6b0000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xfe6b0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 12>; - power-domains = <&sysc R8A7795_PD_A3VC>; - #iommu-cells = <1>; + i2c3: i2c@e66d0000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a7795", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe66d0000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 928>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 928>; + dmas = <&dmac0 0x97>, <&dmac0 0x96>; + dma-names = "tx", "rx"; + i2c-scl-internal-delay-ns = <110>; status = "disabled"; }; - ipmmu_vc1: mmu@fe6f0000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xfe6f0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 13>; - power-domains = <&sysc R8A7795_PD_A3VC>; - #iommu-cells = <1>; + i2c4: i2c@e66d8000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a7795", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe66d8000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 927>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 927>; + dmas = <&dmac0 0x99>, <&dmac0 0x98>; + dma-names = "tx", "rx"; + i2c-scl-internal-delay-ns = <110>; status = "disabled"; }; - ipmmu_pv0: mmu@fd800000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xfd800000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 6>; + i2c5: i2c@e66e0000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a7795", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe66e0000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 919>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 919>; + dmas = <&dmac0 0x9b>, <&dmac0 0x9a>; + dma-names = "tx", "rx"; + i2c-scl-internal-delay-ns = <110>; status = "disabled"; }; - ipmmu_pv1: mmu@fd950000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xfd950000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 7>; + i2c6: i2c@e66e8000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a7795", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe66e8000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 918>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 918>; + dmas = <&dmac0 0x9d>, <&dmac0 0x9c>; + dma-names = "tx", "rx"; + i2c-scl-internal-delay-ns = <6>; status = "disabled"; }; - ipmmu_pv2: mmu@fd960000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xfd960000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 8>; + i2c_dvfs: i2c@e60b0000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,iic-r8a7795", + "renesas,rcar-gen3-iic", + "renesas,rmobile-iic"; + reg = <0 0xe60b0000 0 0x425>; + interrupts = ; + clocks = <&cpg CPG_MOD 926>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 926>; + dmas = <&dmac0 0x11>, <&dmac0 0x10>; + dma-names = "tx", "rx"; status = "disabled"; }; - ipmmu_pv3: mmu@fd970000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xfd970000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 9>; + hscif0: serial@e6540000 { + compatible = "renesas,hscif-r8a7795", + "renesas,rcar-gen3-hscif", + "renesas,hscif"; + reg = <0 0xe6540000 0 96>; + interrupts = ; + clocks = <&cpg CPG_MOD 520>, + <&cpg CPG_CORE R8A7795_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac1 0x31>, <&dmac1 0x30>, + <&dmac2 0x31>, <&dmac2 0x30>; + dma-names = "tx", "rx", "tx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 520>; status = "disabled"; }; - ipmmu_ir: mmu@ff8b0000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xff8b0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 3>; - power-domains = <&sysc R8A7795_PD_A3IR>; - #iommu-cells = <1>; + hscif1: serial@e6550000 { + compatible = "renesas,hscif-r8a7795", + "renesas,rcar-gen3-hscif", + "renesas,hscif"; + reg = <0 0xe6550000 0 96>; + interrupts = ; + clocks = <&cpg CPG_MOD 519>, + <&cpg CPG_CORE R8A7795_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac1 0x33>, <&dmac1 0x32>, + <&dmac2 0x33>, <&dmac2 0x32>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 519>; status = "disabled"; }; - ipmmu_hc: mmu@e6570000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xe6570000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 2>; + hscif2: serial@e6560000 { + compatible = "renesas,hscif-r8a7795", + "renesas,rcar-gen3-hscif", + "renesas,hscif"; + reg = <0 0xe6560000 0 96>; + interrupts = ; + clocks = <&cpg CPG_MOD 518>, + <&cpg CPG_CORE R8A7795_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac1 0x35>, <&dmac1 0x34>, + <&dmac2 0x35>, <&dmac2 0x34>; + dma-names = "tx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 518>; status = "disabled"; }; - ipmmu_rt: mmu@ffc80000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xffc80000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 10>; + hscif3: serial@e66a0000 { + compatible = "renesas,hscif-r8a7795", + "renesas,rcar-gen3-hscif", + "renesas,hscif"; + reg = <0 0xe66a0000 0 96>; + interrupts = ; + clocks = <&cpg CPG_MOD 517>, + <&cpg CPG_CORE R8A7795_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac0 0x37>, <&dmac0 0x36>; + dma-names = "tx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 517>; status = "disabled"; }; - ipmmu_mp0: mmu@ec670000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xec670000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 4>; + hscif4: serial@e66b0000 { + compatible = "renesas,hscif-r8a7795", + "renesas,rcar-gen3-hscif", + "renesas,hscif"; + reg = <0 0xe66b0000 0 96>; + interrupts = ; + clocks = <&cpg CPG_MOD 516>, + <&cpg CPG_CORE R8A7795_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac0 0x39>, <&dmac0 0x38>; + dma-names = "tx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 516>; status = "disabled"; }; - ipmmu_ds0: mmu@e6740000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xe6740000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 0>; + hsusb: usb@e6590000 { + compatible = "renesas,usbhs-r8a7795", + "renesas,rcar-gen3-usbhs"; + reg = <0 0xe6590000 0 0x100>; + interrupts = ; + clocks = <&cpg CPG_MOD 704>; + dmas = <&usb_dmac0 0>, <&usb_dmac0 1>, + <&usb_dmac1 0>, <&usb_dmac1 1>; + dma-names = "ch0", "ch1", "ch2", "ch3"; + renesas,buswait = <11>; + phys = <&usb2_phy0>; + phy-names = "usb"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 704>; + status = "disabled"; }; - ipmmu_ds1: mmu@e7740000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xe7740000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 1>; + hsusb3: usb@e659c000 { + compatible = "renesas,usbhs-r8a7795", + "renesas,rcar-gen3-usbhs"; + reg = <0 0xe659c000 0 0x100>; + interrupts = ; + clocks = <&cpg CPG_MOD 705>; + dmas = <&usb_dmac2 0>, <&usb_dmac2 1>, + <&usb_dmac3 0>, <&usb_dmac3 1>; + dma-names = "ch0", "ch1", "ch2", "ch3"; + renesas,buswait = <11>; + phys = <&usb2_phy3>; + phy-names = "usb"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 705>; + status = "disabled"; }; - ipmmu_mm: mmu@e67b0000 { - compatible = "renesas,ipmmu-r8a7795"; - reg = <0 0xe67b0000 0 0x1000>; - interrupts = , - ; + usb_dmac0: dma-controller@e65a0000 { + compatible = "renesas,r8a7795-usb-dmac", + "renesas,usb-dmac"; + reg = <0 0xe65a0000 0 0x100>; + interrupts = ; + interrupt-names = "ch0", "ch1"; + clocks = <&cpg CPG_MOD 330>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 330>; + #dma-cells = <1>; + dma-channels = <2>; + }; + + usb_dmac1: dma-controller@e65b0000 { + compatible = "renesas,r8a7795-usb-dmac", + "renesas,usb-dmac"; + reg = <0 0xe65b0000 0 0x100>; + interrupts = ; + interrupt-names = "ch0", "ch1"; + clocks = <&cpg CPG_MOD 331>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 331>; + #dma-cells = <1>; + dma-channels = <2>; + }; + + usb_dmac2: dma-controller@e6460000 { + compatible = "renesas,r8a7795-usb-dmac", + "renesas,usb-dmac"; + reg = <0 0xe6460000 0 0x100>; + interrupts = ; + interrupt-names = "ch0", "ch1"; + clocks = <&cpg CPG_MOD 326>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 326>; + #dma-cells = <1>; + dma-channels = <2>; + }; + + usb_dmac3: dma-controller@e6470000 { + compatible = "renesas,r8a7795-usb-dmac", + "renesas,usb-dmac"; + reg = <0 0xe6470000 0 0x100>; + interrupts = ; + interrupt-names = "ch0", "ch1"; + clocks = <&cpg CPG_MOD 329>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 329>; + #dma-cells = <1>; + dma-channels = <2>; + }; + + usb3_phy0: usb-phy@e65ee000 { + compatible = "renesas,r8a7795-usb3-phy", + "renesas,rcar-gen3-usb3-phy"; + reg = <0 0xe65ee000 0 0x90>; + clocks = <&cpg CPG_MOD 328>, <&usb3s0_clk>, + <&usb_extal_clk>; + clock-names = "usb3-if", "usb3s_clk", "usb_extal"; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 328>; + #phy-cells = <0>; + status = "disabled"; }; dmac0: dma-controller@e6700000 { @@ -759,155 +925,208 @@ <&ipmmu_ds1 30>, <&ipmmu_ds1 31>; }; - audma0: dma-controller@ec700000 { - compatible = "renesas,dmac-r8a7795", - "renesas,rcar-dmac"; - reg = <0 0xec700000 0 0x10000>; - interrupts = ; - interrupt-names = "error", - "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7", - "ch8", "ch9", "ch10", "ch11", - "ch12", "ch13", "ch14", "ch15"; - clocks = <&cpg CPG_MOD 502>; - clock-names = "fck"; + ipmmu_ds0: mmu@e6740000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xe6740000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 0>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 502>; - #dma-cells = <1>; - dma-channels = <16>; - iommus = <&ipmmu_mp0 0>, <&ipmmu_mp0 1>, - <&ipmmu_mp0 2>, <&ipmmu_mp0 3>, - <&ipmmu_mp0 4>, <&ipmmu_mp0 5>, - <&ipmmu_mp0 6>, <&ipmmu_mp0 7>, - <&ipmmu_mp0 8>, <&ipmmu_mp0 9>, - <&ipmmu_mp0 10>, <&ipmmu_mp0 11>, - <&ipmmu_mp0 12>, <&ipmmu_mp0 13>, - <&ipmmu_mp0 14>, <&ipmmu_mp0 15>; + #iommu-cells = <1>; }; - audma1: dma-controller@ec720000 { - compatible = "renesas,dmac-r8a7795", - "renesas,rcar-dmac"; - reg = <0 0xec720000 0 0x10000>; - interrupts = ; - interrupt-names = "error", - "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7", - "ch8", "ch9", "ch10", "ch11", - "ch12", "ch13", "ch14", "ch15"; - clocks = <&cpg CPG_MOD 501>; - clock-names = "fck"; + ipmmu_ds1: mmu@e7740000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xe7740000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 1>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 501>; - #dma-cells = <1>; - dma-channels = <16>; - iommus = <&ipmmu_mp0 16>, <&ipmmu_mp0 17>, - <&ipmmu_mp0 18>, <&ipmmu_mp0 19>, - <&ipmmu_mp0 20>, <&ipmmu_mp0 21>, - <&ipmmu_mp0 22>, <&ipmmu_mp0 23>, - <&ipmmu_mp0 24>, <&ipmmu_mp0 25>, - <&ipmmu_mp0 26>, <&ipmmu_mp0 27>, - <&ipmmu_mp0 28>, <&ipmmu_mp0 29>, - <&ipmmu_mp0 30>, <&ipmmu_mp0 31>; + #iommu-cells = <1>; }; - avb: ethernet@e6800000 { - compatible = "renesas,etheravb-r8a7795", - "renesas,etheravb-rcar-gen3"; - reg = <0 0xe6800000 0 0x800>, <0 0xe6a00000 0 0x10000>; - interrupts = , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - ; - interrupt-names = "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7", - "ch8", "ch9", "ch10", "ch11", - "ch12", "ch13", "ch14", "ch15", - "ch16", "ch17", "ch18", "ch19", - "ch20", "ch21", "ch22", "ch23", - "ch24"; - clocks = <&cpg CPG_MOD 812>; + ipmmu_hc: mmu@e6570000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xe6570000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 2>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 812>; - phy-mode = "rgmii"; - iommus = <&ipmmu_ds0 16>; - #address-cells = <1>; - #size-cells = <0>; - status = "disabled"; + #iommu-cells = <1>; }; - can0: can@e6c30000 { - compatible = "renesas,can-r8a7795", - "renesas,rcar-gen3-can"; - reg = <0 0xe6c30000 0 0x1000>; - interrupts = ; - clocks = <&cpg CPG_MOD 916>, - <&cpg CPG_CORE R8A7795_CLK_CANFD>, - <&can_clk>; - clock-names = "clkp1", "clkp2", "can_clk"; - assigned-clocks = <&cpg CPG_CORE R8A7795_CLK_CANFD>; - assigned-clock-rates = <40000000>; + ipmmu_ir: mmu@ff8b0000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xff8b0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 3>; + power-domains = <&sysc R8A7795_PD_A3IR>; + #iommu-cells = <1>; + }; + + ipmmu_mm: mmu@e67b0000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xe67b0000 0 0x1000>; + interrupts = , + ; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 916>; - status = "disabled"; + #iommu-cells = <1>; }; - can1: can@e6c38000 { - compatible = "renesas,can-r8a7795", - "renesas,rcar-gen3-can"; + ipmmu_mp0: mmu@ec670000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xec670000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 4>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_pv0: mmu@fd800000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xfd800000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 6>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_pv1: mmu@fd950000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xfd950000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 7>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_pv2: mmu@fd960000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xfd960000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 8>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_pv3: mmu@fd970000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xfd970000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 9>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_rt: mmu@ffc80000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xffc80000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 10>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_vc0: mmu@fe6b0000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xfe6b0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 12>; + power-domains = <&sysc R8A7795_PD_A3VC>; + #iommu-cells = <1>; + }; + + ipmmu_vc1: mmu@fe6f0000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xfe6f0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 13>; + power-domains = <&sysc R8A7795_PD_A3VC>; + #iommu-cells = <1>; + }; + + ipmmu_vi0: mmu@febd0000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xfebd0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 14>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_vi1: mmu@febe0000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xfebe0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 15>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_vp0: mmu@fe990000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xfe990000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 16>; + power-domains = <&sysc R8A7795_PD_A3VP>; + #iommu-cells = <1>; + }; + + ipmmu_vp1: mmu@fe980000 { + compatible = "renesas,ipmmu-r8a7795"; + reg = <0 0xfe980000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 17>; + power-domains = <&sysc R8A7795_PD_A3VP>; + #iommu-cells = <1>; + }; + + avb: ethernet@e6800000 { + compatible = "renesas,etheravb-r8a7795", + "renesas,etheravb-rcar-gen3"; + reg = <0 0xe6800000 0 0x800>, <0 0xe6a00000 0 0x10000>; + interrupts = , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + ; + interrupt-names = "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14", "ch15", + "ch16", "ch17", "ch18", "ch19", + "ch20", "ch21", "ch22", "ch23", + "ch24"; + clocks = <&cpg CPG_MOD 812>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 812>; + phy-mode = "rgmii"; + iommus = <&ipmmu_ds0 16>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + can0: can@e6c30000 { + compatible = "renesas,can-r8a7795", + "renesas,rcar-gen3-can"; + reg = <0 0xe6c30000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 916>, + <&cpg CPG_CORE R8A7795_CLK_CANFD>, + <&can_clk>; + clock-names = "clkp1", "clkp2", "can_clk"; + assigned-clocks = <&cpg CPG_CORE R8A7795_CLK_CANFD>; + assigned-clock-rates = <40000000>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 916>; + status = "disabled"; + }; + + can1: can@e6c38000 { + compatible = "renesas,can-r8a7795", + "renesas,rcar-gen3-can"; reg = <0 0xe6c38000 0 0x1000>; interrupts = ; clocks = <&cpg CPG_MOD 915>, @@ -946,211 +1165,173 @@ }; }; - drif00: rif@e6f40000 { - compatible = "renesas,r8a7795-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f40000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 515>; - clock-names = "fck"; - dmas = <&dmac1 0x20>, <&dmac2 0x20>; - dma-names = "rx", "rx"; + pwm0: pwm@e6e30000 { + compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; + reg = <0 0xe6e30000 0 0x8>; + clocks = <&cpg CPG_MOD 523>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 515>; - renesas,bonding = <&drif01>; + resets = <&cpg 523>; + #pwm-cells = <2>; status = "disabled"; }; - drif01: rif@e6f50000 { - compatible = "renesas,r8a7795-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f50000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 514>; - clock-names = "fck"; - dmas = <&dmac1 0x22>, <&dmac2 0x22>; - dma-names = "rx", "rx"; + pwm1: pwm@e6e31000 { + compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; + reg = <0 0xe6e31000 0 0x8>; + clocks = <&cpg CPG_MOD 523>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 514>; - renesas,bonding = <&drif00>; + resets = <&cpg 523>; + #pwm-cells = <2>; status = "disabled"; }; - drif10: rif@e6f60000 { - compatible = "renesas,r8a7795-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f60000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 513>; - clock-names = "fck"; - dmas = <&dmac1 0x24>, <&dmac2 0x24>; - dma-names = "rx", "rx"; + pwm2: pwm@e6e32000 { + compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; + reg = <0 0xe6e32000 0 0x8>; + clocks = <&cpg CPG_MOD 523>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 513>; - renesas,bonding = <&drif11>; + resets = <&cpg 523>; + #pwm-cells = <2>; status = "disabled"; }; - drif11: rif@e6f70000 { - compatible = "renesas,r8a7795-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f70000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 512>; - clock-names = "fck"; - dmas = <&dmac1 0x26>, <&dmac2 0x26>; - dma-names = "rx", "rx"; + pwm3: pwm@e6e33000 { + compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; + reg = <0 0xe6e33000 0 0x8>; + clocks = <&cpg CPG_MOD 523>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 512>; - renesas,bonding = <&drif10>; + resets = <&cpg 523>; + #pwm-cells = <2>; status = "disabled"; }; - drif20: rif@e6f80000 { - compatible = "renesas,r8a7795-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f80000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 511>; - clock-names = "fck"; - dmas = <&dmac1 0x28>, <&dmac2 0x28>; - dma-names = "rx", "rx"; + pwm4: pwm@e6e34000 { + compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; + reg = <0 0xe6e34000 0 0x8>; + clocks = <&cpg CPG_MOD 523>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 511>; - renesas,bonding = <&drif21>; + resets = <&cpg 523>; + #pwm-cells = <2>; status = "disabled"; }; - drif21: rif@e6f90000 { - compatible = "renesas,r8a7795-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f90000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 510>; - clock-names = "fck"; - dmas = <&dmac1 0x2a>, <&dmac2 0x2a>; - dma-names = "rx", "rx"; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 510>; - renesas,bonding = <&drif20>; + pwm5: pwm@e6e35000 { + compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; + reg = <0 0xe6e35000 0 0x8>; + clocks = <&cpg CPG_MOD 523>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 523>; + #pwm-cells = <2>; status = "disabled"; }; - drif30: rif@e6fa0000 { - compatible = "renesas,r8a7795-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6fa0000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 509>; - clock-names = "fck"; - dmas = <&dmac1 0x2c>, <&dmac2 0x2c>; - dma-names = "rx", "rx"; + pwm6: pwm@e6e36000 { + compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; + reg = <0 0xe6e36000 0 0x8>; + clocks = <&cpg CPG_MOD 523>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 509>; - renesas,bonding = <&drif31>; + resets = <&cpg 523>; + #pwm-cells = <2>; status = "disabled"; }; - drif31: rif@e6fb0000 { - compatible = "renesas,r8a7795-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6fb0000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 508>; - clock-names = "fck"; - dmas = <&dmac1 0x2e>, <&dmac2 0x2e>; - dma-names = "rx", "rx"; + scif0: serial@e6e60000 { + compatible = "renesas,scif-r8a7795", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6e60000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 207>, + <&cpg CPG_CORE R8A7795_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac1 0x51>, <&dmac1 0x50>, + <&dmac2 0x51>, <&dmac2 0x50>; + dma-names = "tx", "rx", "tx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 508>; - renesas,bonding = <&drif30>; + resets = <&cpg 207>; status = "disabled"; }; - hscif0: serial@e6540000 { - compatible = "renesas,hscif-r8a7795", - "renesas,rcar-gen3-hscif", - "renesas,hscif"; - reg = <0 0xe6540000 0 96>; - interrupts = ; - clocks = <&cpg CPG_MOD 520>, + scif1: serial@e6e68000 { + compatible = "renesas,scif-r8a7795", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6e68000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 206>, <&cpg CPG_CORE R8A7795_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x31>, <&dmac1 0x30>, - <&dmac2 0x31>, <&dmac2 0x30>; + dmas = <&dmac1 0x53>, <&dmac1 0x52>, + <&dmac2 0x53>, <&dmac2 0x52>; dma-names = "tx", "rx", "tx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 520>; + resets = <&cpg 206>; status = "disabled"; }; - hscif1: serial@e6550000 { - compatible = "renesas,hscif-r8a7795", - "renesas,rcar-gen3-hscif", - "renesas,hscif"; - reg = <0 0xe6550000 0 96>; - interrupts = ; - clocks = <&cpg CPG_MOD 519>, + scif2: serial@e6e88000 { + compatible = "renesas,scif-r8a7795", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6e88000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 310>, <&cpg CPG_CORE R8A7795_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x33>, <&dmac1 0x32>, - <&dmac2 0x33>, <&dmac2 0x32>; + dmas = <&dmac1 0x13>, <&dmac1 0x12>, + <&dmac2 0x13>, <&dmac2 0x12>; dma-names = "tx", "rx", "tx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 519>; + resets = <&cpg 310>; status = "disabled"; }; - hscif2: serial@e6560000 { - compatible = "renesas,hscif-r8a7795", - "renesas,rcar-gen3-hscif", - "renesas,hscif"; - reg = <0 0xe6560000 0 96>; - interrupts = ; - clocks = <&cpg CPG_MOD 518>, + scif3: serial@e6c50000 { + compatible = "renesas,scif-r8a7795", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6c50000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 204>, <&cpg CPG_CORE R8A7795_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x35>, <&dmac1 0x34>, - <&dmac2 0x35>, <&dmac2 0x34>; - dma-names = "tx", "rx", "tx", "rx"; + dmas = <&dmac0 0x57>, <&dmac0 0x56>; + dma-names = "tx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 518>; + resets = <&cpg 204>; status = "disabled"; }; - hscif3: serial@e66a0000 { - compatible = "renesas,hscif-r8a7795", - "renesas,rcar-gen3-hscif", - "renesas,hscif"; - reg = <0 0xe66a0000 0 96>; - interrupts = ; - clocks = <&cpg CPG_MOD 517>, + scif4: serial@e6c40000 { + compatible = "renesas,scif-r8a7795", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6c40000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 203>, <&cpg CPG_CORE R8A7795_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac0 0x37>, <&dmac0 0x36>; + dmas = <&dmac0 0x59>, <&dmac0 0x58>; dma-names = "tx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 517>; + resets = <&cpg 203>; status = "disabled"; }; - hscif4: serial@e66b0000 { - compatible = "renesas,hscif-r8a7795", - "renesas,rcar-gen3-hscif", - "renesas,hscif"; - reg = <0 0xe66b0000 0 96>; - interrupts = ; - clocks = <&cpg CPG_MOD 516>, + scif5: serial@e6f30000 { + compatible = "renesas,scif-r8a7795", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6f30000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 202>, <&cpg CPG_CORE R8A7795_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac0 0x39>, <&dmac0 0x38>; - dma-names = "tx", "rx"; + dmas = <&dmac1 0x5b>, <&dmac1 0x5a>, + <&dmac2 0x5b>, <&dmac2 0x5a>; + dma-names = "tx", "rx", "tx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 516>; + resets = <&cpg 202>; status = "disabled"; }; @@ -1216,304 +1397,379 @@ status = "disabled"; }; - scif0: serial@e6e60000 { - compatible = "renesas,scif-r8a7795", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6e60000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 207>, - <&cpg CPG_CORE R8A7795_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x51>, <&dmac1 0x50>, - <&dmac2 0x51>, <&dmac2 0x50>; - dma-names = "tx", "rx", "tx", "rx"; + vin0: video@e6ef0000 { + compatible = "renesas,vin-r8a7795"; + reg = <0 0xe6ef0000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 811>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 207>; + resets = <&cpg 811>; + renesas,id = <0>; status = "disabled"; - }; - scif1: serial@e6e68000 { - compatible = "renesas,scif-r8a7795", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6e68000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 206>, - <&cpg CPG_CORE R8A7795_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x53>, <&dmac1 0x52>, - <&dmac2 0x53>, <&dmac2 0x52>; - dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 206>; - status = "disabled"; - }; + ports { + #address-cells = <1>; + #size-cells = <0>; - scif2: serial@e6e88000 { - compatible = "renesas,scif-r8a7795", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6e88000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 310>, - <&cpg CPG_CORE R8A7795_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x13>, <&dmac1 0x12>, - <&dmac2 0x13>, <&dmac2 0x12>; - dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 310>; - status = "disabled"; - }; + port@1 { + #address-cells = <1>; + #size-cells = <0>; - scif3: serial@e6c50000 { - compatible = "renesas,scif-r8a7795", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6c50000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 204>, - <&cpg CPG_CORE R8A7795_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac0 0x57>, <&dmac0 0x56>; - dma-names = "tx", "rx"; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 204>; - status = "disabled"; - }; + reg = <1>; - scif4: serial@e6c40000 { - compatible = "renesas,scif-r8a7795", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6c40000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 203>, - <&cpg CPG_CORE R8A7795_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac0 0x59>, <&dmac0 0x58>; - dma-names = "tx", "rx"; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 203>; - status = "disabled"; + vin0csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin0>; + }; + vin0csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin0>; + }; + }; + }; }; - scif5: serial@e6f30000 { - compatible = "renesas,scif-r8a7795", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6f30000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 202>, - <&cpg CPG_CORE R8A7795_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x5b>, <&dmac1 0x5a>, - <&dmac2 0x5b>, <&dmac2 0x5a>; - dma-names = "tx", "rx", "tx", "rx"; + vin1: video@e6ef1000 { + compatible = "renesas,vin-r8a7795"; + reg = <0 0xe6ef1000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 810>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 202>; + resets = <&cpg 810>; + renesas,id = <1>; status = "disabled"; - }; - i2c_dvfs: i2c@e60b0000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,iic-r8a7795", - "renesas,rcar-gen3-iic", - "renesas,rmobile-iic"; - reg = <0 0xe60b0000 0 0x425>; - interrupts = ; - clocks = <&cpg CPG_MOD 926>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 926>; - dmas = <&dmac0 0x11>, <&dmac0 0x10>; - dma-names = "tx", "rx"; - status = "disabled"; + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin1csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin1>; + }; + vin1csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin1>; + }; + }; + }; }; - i2c0: i2c@e6500000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a7795", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe6500000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 931>; + vin2: video@e6ef2000 { + compatible = "renesas,vin-r8a7795"; + reg = <0 0xe6ef2000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 809>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 931>; - dmas = <&dmac1 0x91>, <&dmac1 0x90>, - <&dmac2 0x91>, <&dmac2 0x90>; - dma-names = "tx", "rx", "tx", "rx"; - i2c-scl-internal-delay-ns = <110>; + resets = <&cpg 809>; + renesas,id = <2>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin2csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin2>; + }; + vin2csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin2>; + }; + }; + }; }; - i2c1: i2c@e6508000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a7795", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe6508000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 930>; + vin3: video@e6ef3000 { + compatible = "renesas,vin-r8a7795"; + reg = <0 0xe6ef3000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 808>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 930>; - dmas = <&dmac1 0x93>, <&dmac1 0x92>, - <&dmac2 0x93>, <&dmac2 0x92>; - dma-names = "tx", "rx", "tx", "rx"; - i2c-scl-internal-delay-ns = <6>; + resets = <&cpg 808>; + renesas,id = <3>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin3csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin3>; + }; + vin3csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin3>; + }; + }; + }; }; - i2c2: i2c@e6510000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a7795", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe6510000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 929>; + vin4: video@e6ef4000 { + compatible = "renesas,vin-r8a7795"; + reg = <0 0xe6ef4000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 807>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 929>; - dmas = <&dmac1 0x95>, <&dmac1 0x94>, - <&dmac2 0x95>, <&dmac2 0x94>; - dma-names = "tx", "rx", "tx", "rx"; - i2c-scl-internal-delay-ns = <6>; + resets = <&cpg 807>; + renesas,id = <4>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin4csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin4>; + }; + vin4csi41: endpoint@3 { + reg = <3>; + remote-endpoint= <&csi41vin4>; + }; + }; + }; }; - i2c3: i2c@e66d0000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a7795", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe66d0000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 928>; + vin5: video@e6ef5000 { + compatible = "renesas,vin-r8a7795"; + reg = <0 0xe6ef5000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 806>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 928>; - dmas = <&dmac0 0x97>, <&dmac0 0x96>; - dma-names = "tx", "rx"; - i2c-scl-internal-delay-ns = <110>; + resets = <&cpg 806>; + renesas,id = <5>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin5csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin5>; + }; + vin5csi41: endpoint@3 { + reg = <3>; + remote-endpoint= <&csi41vin5>; + }; + }; + }; }; - i2c4: i2c@e66d8000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a7795", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe66d8000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 927>; + vin6: video@e6ef6000 { + compatible = "renesas,vin-r8a7795"; + reg = <0 0xe6ef6000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 805>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 927>; - dmas = <&dmac0 0x99>, <&dmac0 0x98>; - dma-names = "tx", "rx"; - i2c-scl-internal-delay-ns = <110>; + resets = <&cpg 805>; + renesas,id = <6>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin6csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin6>; + }; + vin6csi41: endpoint@3 { + reg = <3>; + remote-endpoint= <&csi41vin6>; + }; + }; + }; }; - i2c5: i2c@e66e0000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a7795", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe66e0000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 919>; + vin7: video@e6ef7000 { + compatible = "renesas,vin-r8a7795"; + reg = <0 0xe6ef7000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 804>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 919>; - dmas = <&dmac0 0x9b>, <&dmac0 0x9a>; - dma-names = "tx", "rx"; - i2c-scl-internal-delay-ns = <110>; + resets = <&cpg 804>; + renesas,id = <7>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin7csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin7>; + }; + vin7csi41: endpoint@3 { + reg = <3>; + remote-endpoint= <&csi41vin7>; + }; + }; + }; }; - i2c6: i2c@e66e8000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a7795", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe66e8000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 918>; + drif00: rif@e6f40000 { + compatible = "renesas,r8a7795-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f40000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 515>; + clock-names = "fck"; + dmas = <&dmac1 0x20>, <&dmac2 0x20>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 918>; - dmas = <&dmac0 0x9d>, <&dmac0 0x9c>; - dma-names = "tx", "rx"; - i2c-scl-internal-delay-ns = <6>; + resets = <&cpg 515>; + renesas,bonding = <&drif01>; status = "disabled"; }; - pwm0: pwm@e6e30000 { - compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; - reg = <0 0xe6e30000 0 0x8>; - clocks = <&cpg CPG_MOD 523>; + drif01: rif@e6f50000 { + compatible = "renesas,r8a7795-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f50000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 514>; + clock-names = "fck"; + dmas = <&dmac1 0x22>, <&dmac2 0x22>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 523>; - #pwm-cells = <2>; + resets = <&cpg 514>; + renesas,bonding = <&drif00>; status = "disabled"; }; - pwm1: pwm@e6e31000 { - compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; - reg = <0 0xe6e31000 0 0x8>; - clocks = <&cpg CPG_MOD 523>; + drif10: rif@e6f60000 { + compatible = "renesas,r8a7795-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f60000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 513>; + clock-names = "fck"; + dmas = <&dmac1 0x24>, <&dmac2 0x24>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 523>; - #pwm-cells = <2>; + resets = <&cpg 513>; + renesas,bonding = <&drif11>; status = "disabled"; }; - pwm2: pwm@e6e32000 { - compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; - reg = <0 0xe6e32000 0 0x8>; - clocks = <&cpg CPG_MOD 523>; + drif11: rif@e6f70000 { + compatible = "renesas,r8a7795-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f70000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 512>; + clock-names = "fck"; + dmas = <&dmac1 0x26>, <&dmac2 0x26>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 523>; - #pwm-cells = <2>; + resets = <&cpg 512>; + renesas,bonding = <&drif10>; status = "disabled"; }; - pwm3: pwm@e6e33000 { - compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; - reg = <0 0xe6e33000 0 0x8>; - clocks = <&cpg CPG_MOD 523>; + drif20: rif@e6f80000 { + compatible = "renesas,r8a7795-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f80000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 511>; + clock-names = "fck"; + dmas = <&dmac1 0x28>, <&dmac2 0x28>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 523>; - #pwm-cells = <2>; + resets = <&cpg 511>; + renesas,bonding = <&drif21>; status = "disabled"; }; - pwm4: pwm@e6e34000 { - compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; - reg = <0 0xe6e34000 0 0x8>; - clocks = <&cpg CPG_MOD 523>; + drif21: rif@e6f90000 { + compatible = "renesas,r8a7795-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f90000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 510>; + clock-names = "fck"; + dmas = <&dmac1 0x2a>, <&dmac2 0x2a>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 523>; - #pwm-cells = <2>; + resets = <&cpg 510>; + renesas,bonding = <&drif20>; status = "disabled"; }; - pwm5: pwm@e6e35000 { - compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; - reg = <0 0xe6e35000 0 0x8>; - clocks = <&cpg CPG_MOD 523>; + drif30: rif@e6fa0000 { + compatible = "renesas,r8a7795-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6fa0000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 509>; + clock-names = "fck"; + dmas = <&dmac1 0x2c>, <&dmac2 0x2c>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 523>; - #pwm-cells = <2>; + resets = <&cpg 509>; + renesas,bonding = <&drif31>; status = "disabled"; }; - pwm6: pwm@e6e36000 { - compatible = "renesas,pwm-r8a7795", "renesas,pwm-rcar"; - reg = <0 0xe6e36000 0 0x8>; - clocks = <&cpg CPG_MOD 523>; + drif31: rif@e6fb0000 { + compatible = "renesas,r8a7795-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6fb0000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 508>; + clock-names = "fck"; + dmas = <&dmac1 0x2e>, <&dmac2 0x2e>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 523>; - #pwm-cells = <2>; + resets = <&cpg 508>; + renesas,bonding = <&drif30>; status = "disabled"; }; @@ -1711,201 +1967,172 @@ dma-names = "rx", "tx", "rxu", "txu"; }; }; - }; - - sata: sata@ee300000 { - compatible = "renesas,sata-r8a7795", - "renesas,rcar-gen3-sata"; - reg = <0 0xee300000 0 0x200000>; - interrupts = ; - clocks = <&cpg CPG_MOD 815>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 815>; - status = "disabled"; - iommus = <&ipmmu_hc 2>; - }; - - usb3_phy0: usb-phy@e65ee000 { - compatible = "renesas,r8a7795-usb3-phy", - "renesas,rcar-gen3-usb3-phy"; - reg = <0 0xe65ee000 0 0x90>; - clocks = <&cpg CPG_MOD 328>, <&usb3s0_clk>, - <&usb_extal_clk>; - clock-names = "usb3-if", "usb3s_clk", "usb_extal"; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 328>; - #phy-cells = <0>; - status = "disabled"; - }; - - xhci0: usb@ee000000 { - compatible = "renesas,xhci-r8a7795", "renesas,rcar-gen3-xhci"; - reg = <0 0xee000000 0 0xc00>; - interrupts = ; - clocks = <&cpg CPG_MOD 328>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 328>; - status = "disabled"; - }; - - usb3_peri0: usb@ee020000 { - compatible = "renesas,r8a7795-usb3-peri", - "renesas,rcar-gen3-usb3-peri"; - reg = <0 0xee020000 0 0x400>; - interrupts = ; - clocks = <&cpg CPG_MOD 328>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 328>; - status = "disabled"; - }; - - usb_dmac0: dma-controller@e65a0000 { - compatible = "renesas,r8a7795-usb-dmac", - "renesas,usb-dmac"; - reg = <0 0xe65a0000 0 0x100>; - interrupts = ; - interrupt-names = "ch0", "ch1"; - clocks = <&cpg CPG_MOD 330>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 330>; - #dma-cells = <1>; - dma-channels = <2>; - }; - usb_dmac1: dma-controller@e65b0000 { - compatible = "renesas,r8a7795-usb-dmac", - "renesas,usb-dmac"; - reg = <0 0xe65b0000 0 0x100>; - interrupts = ; - interrupt-names = "ch0", "ch1"; - clocks = <&cpg CPG_MOD 331>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 331>; - #dma-cells = <1>; - dma-channels = <2>; + ports { + #address-cells = <1>; + #size-cells = <0>; + port@0 { + reg = <0>; + }; + port@1 { + reg = <1>; + }; + port@2 { + reg = <2>; + }; + }; }; - usb_dmac2: dma-controller@e6460000 { - compatible = "renesas,r8a7795-usb-dmac", - "renesas,usb-dmac"; - reg = <0 0xe6460000 0 0x100>; - interrupts = ; - interrupt-names = "ch0", "ch1"; - clocks = <&cpg CPG_MOD 326>; + audma0: dma-controller@ec700000 { + compatible = "renesas,dmac-r8a7795", + "renesas,rcar-dmac"; + reg = <0 0xec700000 0 0x10000>; + interrupts = ; + interrupt-names = "error", + "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14", "ch15"; + clocks = <&cpg CPG_MOD 502>; + clock-names = "fck"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 326>; + resets = <&cpg 502>; #dma-cells = <1>; - dma-channels = <2>; + dma-channels = <16>; + iommus = <&ipmmu_mp0 0>, <&ipmmu_mp0 1>, + <&ipmmu_mp0 2>, <&ipmmu_mp0 3>, + <&ipmmu_mp0 4>, <&ipmmu_mp0 5>, + <&ipmmu_mp0 6>, <&ipmmu_mp0 7>, + <&ipmmu_mp0 8>, <&ipmmu_mp0 9>, + <&ipmmu_mp0 10>, <&ipmmu_mp0 11>, + <&ipmmu_mp0 12>, <&ipmmu_mp0 13>, + <&ipmmu_mp0 14>, <&ipmmu_mp0 15>; }; - usb_dmac3: dma-controller@e6470000 { - compatible = "renesas,r8a7795-usb-dmac", - "renesas,usb-dmac"; - reg = <0 0xe6470000 0 0x100>; - interrupts = ; - interrupt-names = "ch0", "ch1"; - clocks = <&cpg CPG_MOD 329>; + audma1: dma-controller@ec720000 { + compatible = "renesas,dmac-r8a7795", + "renesas,rcar-dmac"; + reg = <0 0xec720000 0 0x10000>; + interrupts = ; + interrupt-names = "error", + "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14", "ch15"; + clocks = <&cpg CPG_MOD 501>; + clock-names = "fck"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 329>; + resets = <&cpg 501>; #dma-cells = <1>; - dma-channels = <2>; - }; - - sdhi0: sd@ee100000 { - compatible = "renesas,sdhi-r8a7795", - "renesas,rcar-gen3-sdhi"; - reg = <0 0xee100000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 314>; - max-frequency = <200000000>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 314>; - status = "disabled"; - }; - - sdhi1: sd@ee120000 { - compatible = "renesas,sdhi-r8a7795", - "renesas,rcar-gen3-sdhi"; - reg = <0 0xee120000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 313>; - max-frequency = <200000000>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 313>; - status = "disabled"; + dma-channels = <16>; + iommus = <&ipmmu_mp0 16>, <&ipmmu_mp0 17>, + <&ipmmu_mp0 18>, <&ipmmu_mp0 19>, + <&ipmmu_mp0 20>, <&ipmmu_mp0 21>, + <&ipmmu_mp0 22>, <&ipmmu_mp0 23>, + <&ipmmu_mp0 24>, <&ipmmu_mp0 25>, + <&ipmmu_mp0 26>, <&ipmmu_mp0 27>, + <&ipmmu_mp0 28>, <&ipmmu_mp0 29>, + <&ipmmu_mp0 30>, <&ipmmu_mp0 31>; }; - sdhi2: sd@ee140000 { - compatible = "renesas,sdhi-r8a7795", - "renesas,rcar-gen3-sdhi"; - reg = <0 0xee140000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 312>; - max-frequency = <200000000>; + xhci0: usb@ee000000 { + compatible = "renesas,xhci-r8a7795", "renesas,rcar-gen3-xhci"; + reg = <0 0xee000000 0 0xc00>; + interrupts = ; + clocks = <&cpg CPG_MOD 328>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 312>; + resets = <&cpg 328>; status = "disabled"; }; - sdhi3: sd@ee160000 { - compatible = "renesas,sdhi-r8a7795", - "renesas,rcar-gen3-sdhi"; - reg = <0 0xee160000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 311>; - max-frequency = <200000000>; + usb3_peri0: usb@ee020000 { + compatible = "renesas,r8a7795-usb3-peri", + "renesas,rcar-gen3-usb3-peri"; + reg = <0 0xee020000 0 0x400>; + interrupts = ; + clocks = <&cpg CPG_MOD 328>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 311>; + resets = <&cpg 328>; status = "disabled"; }; - usb2_phy0: usb-phy@ee080200 { - compatible = "renesas,usb2-phy-r8a7795", - "renesas,rcar-gen3-usb2-phy"; - reg = <0 0xee080200 0 0x700>; + ohci0: usb@ee080000 { + compatible = "generic-ohci"; + reg = <0 0xee080000 0 0x100>; interrupts = ; clocks = <&cpg CPG_MOD 703>; + phys = <&usb2_phy0>; + phy-names = "usb"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; resets = <&cpg 703>; - #phy-cells = <0>; status = "disabled"; }; - usb2_phy1: usb-phy@ee0a0200 { - compatible = "renesas,usb2-phy-r8a7795", - "renesas,rcar-gen3-usb2-phy"; - reg = <0 0xee0a0200 0 0x700>; + ohci1: usb@ee0a0000 { + compatible = "generic-ohci"; + reg = <0 0xee0a0000 0 0x100>; + interrupts = ; clocks = <&cpg CPG_MOD 702>; + phys = <&usb2_phy1>; + phy-names = "usb"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; resets = <&cpg 702>; - #phy-cells = <0>; status = "disabled"; }; - usb2_phy2: usb-phy@ee0c0200 { - compatible = "renesas,usb2-phy-r8a7795", - "renesas,rcar-gen3-usb2-phy"; - reg = <0 0xee0c0200 0 0x700>; + ohci2: usb@ee0c0000 { + compatible = "generic-ohci"; + reg = <0 0xee0c0000 0 0x100>; + interrupts = ; clocks = <&cpg CPG_MOD 701>; + phys = <&usb2_phy2>; + phy-names = "usb"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; resets = <&cpg 701>; - #phy-cells = <0>; status = "disabled"; }; - usb2_phy3: usb-phy@ee0e0200 { - compatible = "renesas,usb2-phy-r8a7795", - "renesas,rcar-gen3-usb2-phy"; - reg = <0 0xee0e0200 0 0x700>; + ohci3: usb@ee0e0000 { + compatible = "generic-ohci"; + reg = <0 0xee0e0000 0 0x100>; interrupts = ; clocks = <&cpg CPG_MOD 700>; + phys = <&usb2_phy3>; + phy-names = "usb"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; resets = <&cpg 700>; - #phy-cells = <0>; status = "disabled"; }; @@ -1961,88 +2188,129 @@ status = "disabled"; }; - ohci0: usb@ee080000 { - compatible = "generic-ohci"; - reg = <0 0xee080000 0 0x100>; + usb2_phy0: usb-phy@ee080200 { + compatible = "renesas,usb2-phy-r8a7795", + "renesas,rcar-gen3-usb2-phy"; + reg = <0 0xee080200 0 0x700>; interrupts = ; clocks = <&cpg CPG_MOD 703>; - phys = <&usb2_phy0>; - phy-names = "usb"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; resets = <&cpg 703>; + #phy-cells = <0>; status = "disabled"; }; - ohci1: usb@ee0a0000 { - compatible = "generic-ohci"; - reg = <0 0xee0a0000 0 0x100>; - interrupts = ; + usb2_phy1: usb-phy@ee0a0200 { + compatible = "renesas,usb2-phy-r8a7795", + "renesas,rcar-gen3-usb2-phy"; + reg = <0 0xee0a0200 0 0x700>; clocks = <&cpg CPG_MOD 702>; - phys = <&usb2_phy1>; - phy-names = "usb"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; resets = <&cpg 702>; + #phy-cells = <0>; status = "disabled"; }; - ohci2: usb@ee0c0000 { - compatible = "generic-ohci"; - reg = <0 0xee0c0000 0 0x100>; - interrupts = ; + usb2_phy2: usb-phy@ee0c0200 { + compatible = "renesas,usb2-phy-r8a7795", + "renesas,rcar-gen3-usb2-phy"; + reg = <0 0xee0c0200 0 0x700>; clocks = <&cpg CPG_MOD 701>; - phys = <&usb2_phy2>; - phy-names = "usb"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; resets = <&cpg 701>; + #phy-cells = <0>; status = "disabled"; }; - ohci3: usb@ee0e0000 { - compatible = "generic-ohci"; - reg = <0 0xee0e0000 0 0x100>; + usb2_phy3: usb-phy@ee0e0200 { + compatible = "renesas,usb2-phy-r8a7795", + "renesas,rcar-gen3-usb2-phy"; + reg = <0 0xee0e0200 0 0x700>; interrupts = ; clocks = <&cpg CPG_MOD 700>; - phys = <&usb2_phy3>; - phy-names = "usb"; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; resets = <&cpg 700>; + #phy-cells = <0>; status = "disabled"; }; - hsusb: usb@e6590000 { - compatible = "renesas,usbhs-r8a7795", - "renesas,rcar-gen3-usbhs"; - reg = <0 0xe6590000 0 0x100>; - interrupts = ; - clocks = <&cpg CPG_MOD 704>; - dmas = <&usb_dmac0 0>, <&usb_dmac0 1>, - <&usb_dmac1 0>, <&usb_dmac1 1>; - dma-names = "ch0", "ch1", "ch2", "ch3"; - renesas,buswait = <11>; - phys = <&usb2_phy0>; - phy-names = "usb"; + sdhi0: sd@ee100000 { + compatible = "renesas,sdhi-r8a7795", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee100000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 314>; + max-frequency = <200000000>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 704>; + resets = <&cpg 314>; status = "disabled"; }; - hsusb3: usb@e659c000 { - compatible = "renesas,usbhs-r8a7795", - "renesas,rcar-gen3-usbhs"; - reg = <0 0xe659c000 0 0x100>; - interrupts = ; - clocks = <&cpg CPG_MOD 705>; - dmas = <&usb_dmac2 0>, <&usb_dmac2 1>, - <&usb_dmac3 0>, <&usb_dmac3 1>; - dma-names = "ch0", "ch1", "ch2", "ch3"; - renesas,buswait = <11>; - phys = <&usb2_phy3>; - phy-names = "usb"; + sdhi1: sd@ee120000 { + compatible = "renesas,sdhi-r8a7795", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee120000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 313>; + max-frequency = <200000000>; power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 705>; + resets = <&cpg 313>; status = "disabled"; }; + sdhi2: sd@ee140000 { + compatible = "renesas,sdhi-r8a7795", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee140000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 312>; + max-frequency = <200000000>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 312>; + status = "disabled"; + }; + + sdhi3: sd@ee160000 { + compatible = "renesas,sdhi-r8a7795", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee160000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 311>; + max-frequency = <200000000>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 311>; + status = "disabled"; + }; + + sata: sata@ee300000 { + compatible = "renesas,sata-r8a7795", + "renesas,rcar-gen3-sata"; + reg = <0 0xee300000 0 0x200000>; + interrupts = ; + clocks = <&cpg CPG_MOD 815>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 815>; + status = "disabled"; + iommus = <&ipmmu_hc 2>; + }; + + gic: interrupt-controller@f1010000 { + compatible = "arm,gic-400"; + #interrupt-cells = <3>; + #address-cells = <0>; + interrupt-controller; + reg = <0x0 0xf1010000 0 0x1000>, + <0x0 0xf1020000 0 0x20000>, + <0x0 0xf1040000 0 0x20000>, + <0x0 0xf1060000 0 0x20000>; + interrupts = ; + clocks = <&cpg CPG_MOD 408>; + clock-names = "clk"; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 408>; + }; + pciec0: pcie@fe000000 { compatible = "renesas,pcie-r8a7795", "renesas,pcie-rcar-gen3"; @@ -2133,28 +2401,28 @@ reg = <0 0xfe890000 0 0x2000>; interrupts = ; clocks = <&cpg CPG_MOD 820>; - power-domains = <&sysc R8A7795_PD_A3VC>; - resets = <&cpg 820>; - }; - - vspbc: vsp@fe920000 { - compatible = "renesas,vsp2"; - reg = <0 0xfe920000 0 0x8000>; - interrupts = ; - clocks = <&cpg CPG_MOD 624>; - power-domains = <&sysc R8A7795_PD_A3VP>; - resets = <&cpg 624>; + power-domains = <&sysc R8A7795_PD_A3VC>; + resets = <&cpg 820>; + }; - renesas,fcp = <&fcpvb1>; + fdp1@fe940000 { + compatible = "renesas,fdp1"; + reg = <0 0xfe940000 0 0x2400>; + interrupts = ; + clocks = <&cpg CPG_MOD 119>; + power-domains = <&sysc R8A7795_PD_A3VP>; + resets = <&cpg 119>; + renesas,fcp = <&fcpf0>; }; - fcpvb1: fcp@fe92f000 { - compatible = "renesas,fcpv"; - reg = <0 0xfe92f000 0 0x200>; - clocks = <&cpg CPG_MOD 606>; + fdp1@fe944000 { + compatible = "renesas,fdp1"; + reg = <0 0xfe944000 0 0x2400>; + interrupts = ; + clocks = <&cpg CPG_MOD 118>; power-domains = <&sysc R8A7795_PD_A3VP>; - resets = <&cpg 606>; - iommus = <&ipmmu_vp1 7>; + resets = <&cpg 118>; + renesas,fcp = <&fcpf1>; }; fcpf0: fcp@fe950000 { @@ -2175,17 +2443,6 @@ iommus = <&ipmmu_vp1 1>; }; - vspbd: vsp@fe960000 { - compatible = "renesas,vsp2"; - reg = <0 0xfe960000 0 0x8000>; - interrupts = ; - clocks = <&cpg CPG_MOD 626>; - power-domains = <&sysc R8A7795_PD_A3VP>; - resets = <&cpg 626>; - - renesas,fcp = <&fcpvb0>; - }; - fcpvb0: fcp@fe96f000 { compatible = "renesas,fcpv"; reg = <0 0xfe96f000 0 0x200>; @@ -2195,15 +2452,13 @@ iommus = <&ipmmu_vp0 5>; }; - vspi0: vsp@fe9a0000 { - compatible = "renesas,vsp2"; - reg = <0 0xfe9a0000 0 0x8000>; - interrupts = ; - clocks = <&cpg CPG_MOD 631>; + fcpvb1: fcp@fe92f000 { + compatible = "renesas,fcpv"; + reg = <0 0xfe92f000 0 0x200>; + clocks = <&cpg CPG_MOD 606>; power-domains = <&sysc R8A7795_PD_A3VP>; - resets = <&cpg 631>; - - renesas,fcp = <&fcpvi0>; + resets = <&cpg 606>; + iommus = <&ipmmu_vp1 7>; }; fcpvi0: fcp@fe9af000 { @@ -2215,17 +2470,6 @@ iommus = <&ipmmu_vp0 8>; }; - vspi1: vsp@fe9b0000 { - compatible = "renesas,vsp2"; - reg = <0 0xfe9b0000 0 0x8000>; - interrupts = ; - clocks = <&cpg CPG_MOD 630>; - power-domains = <&sysc R8A7795_PD_A3VP>; - resets = <&cpg 630>; - - renesas,fcp = <&fcpvi1>; - }; - fcpvi1: fcp@fe9bf000 { compatible = "renesas,fcpv"; reg = <0 0xfe9bf000 0 0x200>; @@ -2235,6 +2479,55 @@ iommus = <&ipmmu_vp1 9>; }; + fcpvd0: fcp@fea27000 { + compatible = "renesas,fcpv"; + reg = <0 0xfea27000 0 0x200>; + clocks = <&cpg CPG_MOD 603>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 603>; + iommus = <&ipmmu_vi0 8>; + }; + + fcpvd1: fcp@fea2f000 { + compatible = "renesas,fcpv"; + reg = <0 0xfea2f000 0 0x200>; + clocks = <&cpg CPG_MOD 602>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 602>; + iommus = <&ipmmu_vi0 9>; + }; + + fcpvd2: fcp@fea37000 { + compatible = "renesas,fcpv"; + reg = <0 0xfea37000 0 0x200>; + clocks = <&cpg CPG_MOD 601>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 601>; + iommus = <&ipmmu_vi1 10>; + }; + + vspbd: vsp@fe960000 { + compatible = "renesas,vsp2"; + reg = <0 0xfe960000 0 0x8000>; + interrupts = ; + clocks = <&cpg CPG_MOD 626>; + power-domains = <&sysc R8A7795_PD_A3VP>; + resets = <&cpg 626>; + + renesas,fcp = <&fcpvb0>; + }; + + vspbc: vsp@fe920000 { + compatible = "renesas,vsp2"; + reg = <0 0xfe920000 0 0x8000>; + interrupts = ; + clocks = <&cpg CPG_MOD 624>; + power-domains = <&sysc R8A7795_PD_A3VP>; + resets = <&cpg 624>; + + renesas,fcp = <&fcpvb1>; + }; + vspd0: vsp@fea20000 { compatible = "renesas,vsp2"; reg = <0 0xfea20000 0 0x8000>; @@ -2246,15 +2539,6 @@ renesas,fcp = <&fcpvd0>; }; - fcpvd0: fcp@fea27000 { - compatible = "renesas,fcpv"; - reg = <0 0xfea27000 0 0x200>; - clocks = <&cpg CPG_MOD 603>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 603>; - iommus = <&ipmmu_vi0 8>; - }; - vspd1: vsp@fea28000 { compatible = "renesas,vsp2"; reg = <0 0xfea28000 0 0x8000>; @@ -2266,15 +2550,6 @@ renesas,fcp = <&fcpvd1>; }; - fcpvd1: fcp@fea2f000 { - compatible = "renesas,fcpv"; - reg = <0 0xfea2f000 0 0x200>; - clocks = <&cpg CPG_MOD 602>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 602>; - iommus = <&ipmmu_vi0 9>; - }; - vspd2: vsp@fea30000 { compatible = "renesas,vsp2"; reg = <0 0xfea30000 0 0x8000>; @@ -2286,33 +2561,159 @@ renesas,fcp = <&fcpvd2>; }; - fcpvd2: fcp@fea37000 { - compatible = "renesas,fcpv"; - reg = <0 0xfea37000 0 0x200>; - clocks = <&cpg CPG_MOD 601>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 601>; - iommus = <&ipmmu_vi1 10>; + vspi0: vsp@fe9a0000 { + compatible = "renesas,vsp2"; + reg = <0 0xfe9a0000 0 0x8000>; + interrupts = ; + clocks = <&cpg CPG_MOD 631>; + power-domains = <&sysc R8A7795_PD_A3VP>; + resets = <&cpg 631>; + + renesas,fcp = <&fcpvi0>; }; - fdp1@fe940000 { - compatible = "renesas,fdp1"; - reg = <0 0xfe940000 0 0x2400>; - interrupts = ; - clocks = <&cpg CPG_MOD 119>; + vspi1: vsp@fe9b0000 { + compatible = "renesas,vsp2"; + reg = <0 0xfe9b0000 0 0x8000>; + interrupts = ; + clocks = <&cpg CPG_MOD 630>; power-domains = <&sysc R8A7795_PD_A3VP>; - resets = <&cpg 119>; - renesas,fcp = <&fcpf0>; + resets = <&cpg 630>; + + renesas,fcp = <&fcpvi1>; }; - fdp1@fe944000 { - compatible = "renesas,fdp1"; - reg = <0 0xfe944000 0 0x2400>; - interrupts = ; - clocks = <&cpg CPG_MOD 118>; - power-domains = <&sysc R8A7795_PD_A3VP>; - resets = <&cpg 118>; - renesas,fcp = <&fcpf1>; + csi20: csi2@fea80000 { + compatible = "renesas,r8a7795-csi2"; + reg = <0 0xfea80000 0 0x10000>; + interrupts = ; + clocks = <&cpg CPG_MOD 714>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 714>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + csi20vin0: endpoint@0 { + reg = <0>; + remote-endpoint = <&vin0csi20>; + }; + csi20vin1: endpoint@1 { + reg = <1>; + remote-endpoint = <&vin1csi20>; + }; + csi20vin2: endpoint@2 { + reg = <2>; + remote-endpoint = <&vin2csi20>; + }; + csi20vin3: endpoint@3 { + reg = <3>; + remote-endpoint = <&vin3csi20>; + }; + csi20vin4: endpoint@4 { + reg = <4>; + remote-endpoint = <&vin4csi20>; + }; + csi20vin5: endpoint@5 { + reg = <5>; + remote-endpoint = <&vin5csi20>; + }; + csi20vin6: endpoint@6 { + reg = <6>; + remote-endpoint = <&vin6csi20>; + }; + csi20vin7: endpoint@7 { + reg = <7>; + remote-endpoint = <&vin7csi20>; + }; + }; + }; + }; + + csi40: csi2@feaa0000 { + compatible = "renesas,r8a7795-csi2"; + reg = <0 0xfeaa0000 0 0x10000>; + interrupts = ; + clocks = <&cpg CPG_MOD 716>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 716>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + csi40vin0: endpoint@0 { + reg = <0>; + remote-endpoint = <&vin0csi40>; + }; + csi40vin1: endpoint@1 { + reg = <1>; + remote-endpoint = <&vin1csi40>; + }; + csi40vin2: endpoint@2 { + reg = <2>; + remote-endpoint = <&vin2csi40>; + }; + csi40vin3: endpoint@3 { + reg = <3>; + remote-endpoint = <&vin3csi40>; + }; + }; + }; + }; + + csi41: csi2@feab0000 { + compatible = "renesas,r8a7795-csi2"; + reg = <0 0xfeab0000 0 0x10000>; + interrupts = ; + clocks = <&cpg CPG_MOD 715>; + power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; + resets = <&cpg 715>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + csi41vin4: endpoint@0 { + reg = <0>; + remote-endpoint = <&vin4csi41>; + }; + csi41vin5: endpoint@1 { + reg = <1>; + remote-endpoint = <&vin5csi41>; + }; + csi41vin6: endpoint@2 { + reg = <2>; + remote-endpoint = <&vin6csi41>; + }; + csi41vin7: endpoint@3 { + reg = <3>; + remote-endpoint = <&vin7csi41>; + }; + }; + }; }; hdmi0: hdmi@fead0000 { @@ -2337,6 +2738,10 @@ port@1 { reg = <1>; }; + port@2 { + /* HDMI sound */ + reg = <2>; + }; }; }; @@ -2362,6 +2767,10 @@ port@1 { reg = <1>; }; + port@2 { + /* HDMI sound */ + reg = <2>; + }; }; }; @@ -2412,38 +2821,12 @@ }; }; - tsc: thermal@e6198000 { - compatible = "renesas,r8a7795-thermal"; - reg = <0 0xe6198000 0 0x100>, - <0 0xe61a0000 0 0x100>, - <0 0xe61a8000 0 0x100>; - interrupts = , - , - ; - clocks = <&cpg CPG_MOD 522>; - power-domains = <&sysc R8A7795_PD_ALWAYS_ON>; - resets = <&cpg 522>; - #thermal-sensor-cells = <1>; - status = "okay"; + prr: chipid@fff00044 { + compatible = "renesas,prr"; + reg = <0 0xfff00044 0 4>; }; }; - timer { - compatible = "arm,armv8-timer"; - interrupts-extended = <&gic GIC_PPI 13 - (GIC_CPU_MASK_SIMPLE(8) | - IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 14 - (GIC_CPU_MASK_SIMPLE(8) | - IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 11 - (GIC_CPU_MASK_SIMPLE(8) | - IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 10 - (GIC_CPU_MASK_SIMPLE(8) | - IRQ_TYPE_LEVEL_LOW)>; - }; - thermal-zones { sensor_thermal1: sensor-thermal1 { polling-delay-passive = <250>; @@ -2453,12 +2836,12 @@ trips { sensor1_passive: sensor1-passive { temperature = <95000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "passive"; }; sensor1_crit: sensor1-crit { temperature = <120000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "critical"; }; }; @@ -2479,12 +2862,12 @@ trips { sensor2_passive: sensor2-passive { temperature = <95000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "passive"; }; sensor2_crit: sensor2-crit { temperature = <120000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "critical"; }; }; @@ -2505,12 +2888,12 @@ trips { sensor3_passive: sensor3-passive { temperature = <95000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "passive"; }; sensor3_crit: sensor3-crit { temperature = <120000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "critical"; }; }; @@ -2524,6 +2907,14 @@ }; }; + timer { + compatible = "arm,armv8-timer"; + interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>; + }; + /* External USB clocks - can be overridden by the board */ usb3s0_clk: usb3s0 { compatible = "fixed-clock"; diff --git a/arch/arm64/boot/dts/renesas/r8a7796-salvator-x.dts b/arch/arm64/boot/dts/renesas/r8a7796-salvator-x.dts index 498c9e807dc4..90cca09b9a5e 100644 --- a/arch/arm64/boot/dts/renesas/r8a7796-salvator-x.dts +++ b/arch/arm64/boot/dts/renesas/r8a7796-salvator-x.dts @@ -40,6 +40,11 @@ "dclkin.0", "dclkin.1", "dclkin.2"; }; +&sound_card { + dais = <&rsnd_port0 /* ak4613 */ + &rsnd_port1>; /* HDMI0 */ +}; + &hdmi0 { status = "okay"; @@ -50,9 +55,32 @@ remote-endpoint = <&hdmi0_con>; }; }; + port@2 { + reg = <2>; + dw_hdmi0_snd_in: endpoint { + remote-endpoint = <&rsnd_endpoint1>; + }; + }; }; }; &hdmi0_con { remote-endpoint = <&rcar_dw_hdmi0_out>; }; + +&rcar_sound { + ports { + /* rsnd_port0 is on salvator-common */ + rsnd_port1: port@1 { + rsnd_endpoint1: endpoint { + remote-endpoint = <&dw_hdmi0_snd_in>; + + dai-format = "i2s"; + bitclock-master = <&rsnd_endpoint1>; + frame-master = <&rsnd_endpoint1>; + + playback = <&ssi2>; + }; + }; + }; +}; diff --git a/arch/arm64/boot/dts/renesas/r8a7796-salvator-xs.dts b/arch/arm64/boot/dts/renesas/r8a7796-salvator-xs.dts index 2c37055efa94..ddf35d4cd5e5 100644 --- a/arch/arm64/boot/dts/renesas/r8a7796-salvator-xs.dts +++ b/arch/arm64/boot/dts/renesas/r8a7796-salvator-xs.dts @@ -40,6 +40,11 @@ "dclkin.0", "dclkin.1", "dclkin.2"; }; +&sound_card { + dais = <&rsnd_port0 /* ak4613 */ + &rsnd_port1>; /* HDMI0 */ +}; + &hdmi0 { status = "okay"; @@ -50,9 +55,32 @@ remote-endpoint = <&hdmi0_con>; }; }; + port@2 { + reg = <2>; + dw_hdmi0_snd_in: endpoint { + remote-endpoint = <&rsnd_endpoint1>; + }; + }; }; }; &hdmi0_con { remote-endpoint = <&rcar_dw_hdmi0_out>; }; + +&rcar_sound { + ports { + /* rsnd_port0 is on salvator-common */ + rsnd_port1: port@1 { + rsnd_endpoint1: endpoint { + remote-endpoint = <&dw_hdmi0_snd_in>; + + dai-format = "i2s"; + bitclock-master = <&rsnd_endpoint1>; + frame-master = <&rsnd_endpoint1>; + + playback = <&ssi2>; + }; + }; + }; +}; diff --git a/arch/arm64/boot/dts/renesas/r8a7796.dtsi b/arch/arm64/boot/dts/renesas/r8a7796.dtsi index 556eb8e45499..7c25be6b5af3 100644 --- a/arch/arm64/boot/dts/renesas/r8a7796.dtsi +++ b/arch/arm64/boot/dts/renesas/r8a7796.dtsi @@ -60,6 +60,72 @@ clock-frequency = <0>; }; + cluster0_opp: opp_table0 { + compatible = "operating-points-v2"; + opp-shared; + + opp-500000000 { + opp-hz = /bits/ 64 <500000000>; + opp-microvolt = <820000>; + clock-latency-ns = <300000>; + }; + opp-1000000000 { + opp-hz = /bits/ 64 <1000000000>; + opp-microvolt = <820000>; + clock-latency-ns = <300000>; + }; + opp-1500000000 { + opp-hz = /bits/ 64 <1500000000>; + opp-microvolt = <820000>; + clock-latency-ns = <300000>; + }; + opp-1600000000 { + opp-hz = /bits/ 64 <1600000000>; + opp-microvolt = <900000>; + clock-latency-ns = <300000>; + turbo-mode; + }; + opp-1700000000 { + opp-hz = /bits/ 64 <1700000000>; + opp-microvolt = <900000>; + clock-latency-ns = <300000>; + turbo-mode; + }; + opp-1800000000 { + opp-hz = /bits/ 64 <1800000000>; + opp-microvolt = <960000>; + clock-latency-ns = <300000>; + turbo-mode; + }; + }; + + cluster1_opp: opp_table1 { + compatible = "operating-points-v2"; + opp-shared; + + opp-800000000 { + opp-hz = /bits/ 64 <800000000>; + opp-microvolt = <820000>; + clock-latency-ns = <300000>; + }; + opp-1000000000 { + opp-hz = /bits/ 64 <1000000000>; + opp-microvolt = <820000>; + clock-latency-ns = <300000>; + }; + opp-1200000000 { + opp-hz = /bits/ 64 <1200000000>; + opp-microvolt = <820000>; + clock-latency-ns = <300000>; + }; + opp-1300000000 { + opp-hz = /bits/ 64 <1300000000>; + opp-microvolt = <820000>; + clock-latency-ns = <300000>; + turbo-mode; + }; + }; + cpus { #address-cells = <1>; #size-cells = <0>; @@ -77,7 +143,7 @@ }; a57_1: cpu@1 { - compatible = "arm,cortex-a57","arm,armv8"; + compatible = "arm,cortex-a57", "arm,armv8"; reg = <0x1>; device_type = "cpu"; power-domains = <&sysc R8A7796_PD_CA57_CPU1>; @@ -100,7 +166,7 @@ }; a53_1: cpu@101 { - compatible = "arm,cortex-a53","arm,armv8"; + compatible = "arm,cortex-a53", "arm,armv8"; reg = <0x101>; device_type = "cpu"; power-domains = <&sysc R8A7796_PD_CA53_CPU1>; @@ -111,7 +177,7 @@ }; a53_2: cpu@102 { - compatible = "arm,cortex-a53","arm,armv8"; + compatible = "arm,cortex-a53", "arm,armv8"; reg = <0x102>; device_type = "cpu"; power-domains = <&sysc R8A7796_PD_CA53_CPU2>; @@ -122,7 +188,7 @@ }; a53_3: cpu@103 { - compatible = "arm,cortex-a53","arm,armv8"; + compatible = "arm,cortex-a53", "arm,armv8"; reg = <0x103>; device_type = "cpu"; power-domains = <&sysc R8A7796_PD_CA53_CPU3>; @@ -161,72 +227,6 @@ clock-frequency = <0>; }; - cluster0_opp: opp_table0 { - compatible = "operating-points-v2"; - opp-shared; - - opp-500000000 { - opp-hz = /bits/ 64 <500000000>; - opp-microvolt = <820000>; - clock-latency-ns = <300000>; - }; - opp-1000000000 { - opp-hz = /bits/ 64 <1000000000>; - opp-microvolt = <820000>; - clock-latency-ns = <300000>; - }; - opp-1500000000 { - opp-hz = /bits/ 64 <1500000000>; - opp-microvolt = <820000>; - clock-latency-ns = <300000>; - }; - opp-1600000000 { - opp-hz = /bits/ 64 <1600000000>; - opp-microvolt = <900000>; - clock-latency-ns = <300000>; - turbo-mode; - }; - opp-1700000000 { - opp-hz = /bits/ 64 <1700000000>; - opp-microvolt = <900000>; - clock-latency-ns = <300000>; - turbo-mode; - }; - opp-1800000000 { - opp-hz = /bits/ 64 <1800000000>; - opp-microvolt = <960000>; - clock-latency-ns = <300000>; - turbo-mode; - }; - }; - - cluster1_opp: opp_table1 { - compatible = "operating-points-v2"; - opp-shared; - - opp-800000000 { - opp-hz = /bits/ 64 <800000000>; - opp-microvolt = <820000>; - clock-latency-ns = <300000>; - }; - opp-1000000000 { - opp-hz = /bits/ 64 <1000000000>; - opp-microvolt = <820000>; - clock-latency-ns = <300000>; - }; - opp-1200000000 { - opp-hz = /bits/ 64 <1200000000>; - opp-microvolt = <820000>; - clock-latency-ns = <300000>; - }; - opp-1300000000 { - opp-hz = /bits/ 64 <1300000000>; - opp-microvolt = <820000>; - clock-latency-ns = <300000>; - turbo-mode; - }; - }; - /* External PCIe clock - can be overridden by the board */ pcie_bus_clk: pcie_bus { compatible = "fixed-clock"; @@ -234,13 +234,6 @@ clock-frequency = <0>; }; - pmu_a57 { - compatible = "arm,cortex-a57-pmu"; - interrupts-extended = <&gic GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>, - <&gic GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>; - interrupt-affinity = <&a57_0>, <&a57_1>; - }; - pmu_a53 { compatible = "arm,cortex-a53-pmu"; interrupts-extended = <&gic GIC_SPI 84 IRQ_TYPE_LEVEL_HIGH>, @@ -250,6 +243,13 @@ interrupt-affinity = <&a53_0>, <&a53_1>, <&a53_2>, <&a53_3>; }; + pmu_a57 { + compatible = "arm,cortex-a57-pmu"; + interrupts-extended = <&gic GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&a57_0>, <&a57_1>; + }; + psci { compatible = "arm,psci-1.0", "arm,psci-0.2"; method = "smc"; @@ -269,23 +269,6 @@ #size-cells = <2>; ranges; - gic: interrupt-controller@f1010000 { - compatible = "arm,gic-400"; - #interrupt-cells = <3>; - #address-cells = <0>; - interrupt-controller; - reg = <0x0 0xf1010000 0 0x1000>, - <0x0 0xf1020000 0 0x20000>, - <0x0 0xf1040000 0 0x20000>, - <0x0 0xf1060000 0 0x20000>; - interrupts = ; - clocks = <&cpg CPG_MOD 408>; - clock-names = "clk"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 408>; - }; - wdt0: watchdog@e6020000 { compatible = "renesas,r8a7796-wdt", "renesas,rcar-gen3-wdt"; @@ -421,242 +404,72 @@ reg = <0 0xe6060000 0 0x50c>; }; - ipmmu_vi0: mmu@febd0000 { - compatible = "renesas,ipmmu-r8a7796"; - reg = <0 0xfebd0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 9>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - #iommu-cells = <1>; + cpg: clock-controller@e6150000 { + compatible = "renesas,r8a7796-cpg-mssr"; + reg = <0 0xe6150000 0 0x1000>; + clocks = <&extal_clk>, <&extalr_clk>; + clock-names = "extal", "extalr"; + #clock-cells = <2>; + #power-domain-cells = <0>; + #reset-cells = <1>; }; - ipmmu_vc0: mmu@fe6b0000 { - compatible = "renesas,ipmmu-r8a7796"; - reg = <0 0xfe6b0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 8>; - power-domains = <&sysc R8A7796_PD_A3VC>; - #iommu-cells = <1>; - status = "disabled"; + rst: reset-controller@e6160000 { + compatible = "renesas,r8a7796-rst"; + reg = <0 0xe6160000 0 0x0200>; }; - ipmmu_pv0: mmu@fd800000 { - compatible = "renesas,ipmmu-r8a7796"; - reg = <0 0xfd800000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 5>; + sysc: system-controller@e6180000 { + compatible = "renesas,r8a7796-sysc"; + reg = <0 0xe6180000 0 0x0400>; + #power-domain-cells = <1>; + }; + + tsc: thermal@e6198000 { + compatible = "renesas,r8a7796-thermal"; + reg = <0 0xe6198000 0 0x100>, + <0 0xe61a0000 0 0x100>, + <0 0xe61a8000 0 0x100>; + interrupts = , + , + ; + clocks = <&cpg CPG_MOD 522>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - #iommu-cells = <1>; + resets = <&cpg 522>; + #thermal-sensor-cells = <1>; + status = "okay"; }; - ipmmu_pv1: mmu@fd950000 { - compatible = "renesas,ipmmu-r8a7796"; - reg = <0 0xfd950000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 6>; + intc_ex: interrupt-controller@e61c0000 { + compatible = "renesas,intc-ex-r8a7796", "renesas,irqc"; + #interrupt-cells = <2>; + interrupt-controller; + reg = <0 0xe61c0000 0 0x200>; + interrupts = ; + clocks = <&cpg CPG_MOD 407>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - #iommu-cells = <1>; - status = "disabled"; + resets = <&cpg 407>; }; - ipmmu_ir: mmu@ff8b0000 { - compatible = "renesas,ipmmu-r8a7796"; - reg = <0 0xff8b0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 3>; - power-domains = <&sysc R8A7796_PD_A3IR>; - #iommu-cells = <1>; - status = "disabled"; - }; - - ipmmu_hc: mmu@e6570000 { - compatible = "renesas,ipmmu-r8a7796"; - reg = <0 0xe6570000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 2>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - #iommu-cells = <1>; - status = "disabled"; - }; - - ipmmu_rt: mmu@ffc80000 { - compatible = "renesas,ipmmu-r8a7796"; - reg = <0 0xffc80000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 7>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - #iommu-cells = <1>; - status = "disabled"; - }; - - ipmmu_mp: mmu@ec670000 { - compatible = "renesas,ipmmu-r8a7796"; - reg = <0 0xec670000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 4>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - #iommu-cells = <1>; - }; - - ipmmu_ds0: mmu@e6740000 { - compatible = "renesas,ipmmu-r8a7796"; - reg = <0 0xe6740000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 0>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - #iommu-cells = <1>; - }; - - ipmmu_ds1: mmu@e7740000 { - compatible = "renesas,ipmmu-r8a7796"; - reg = <0 0xe7740000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 1>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - #iommu-cells = <1>; - }; - - ipmmu_mm: mmu@e67b0000 { - compatible = "renesas,ipmmu-r8a7796"; - reg = <0 0xe67b0000 0 0x1000>; - interrupts = , - ; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - #iommu-cells = <1>; - }; - - cpg: clock-controller@e6150000 { - compatible = "renesas,r8a7796-cpg-mssr"; - reg = <0 0xe6150000 0 0x1000>; - clocks = <&extal_clk>, <&extalr_clk>; - clock-names = "extal", "extalr"; - #clock-cells = <2>; - #power-domain-cells = <0>; - #reset-cells = <1>; - }; - - rst: reset-controller@e6160000 { - compatible = "renesas,r8a7796-rst"; - reg = <0 0xe6160000 0 0x0200>; - }; - - prr: chipid@fff00044 { - compatible = "renesas,prr"; - reg = <0 0xfff00044 0 4>; - }; - - sysc: system-controller@e6180000 { - compatible = "renesas,r8a7796-sysc"; - reg = <0 0xe6180000 0 0x0400>; - #power-domain-cells = <1>; - }; - - intc_ex: interrupt-controller@e61c0000 { - compatible = "renesas,intc-ex-r8a7796", "renesas,irqc"; - #interrupt-cells = <2>; - interrupt-controller; - reg = <0 0xe61c0000 0 0x200>; - interrupts = ; - clocks = <&cpg CPG_MOD 407>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 407>; - }; - - i2c_dvfs: i2c@e60b0000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,iic-r8a7796", - "renesas,rcar-gen3-iic", - "renesas,rmobile-iic"; - reg = <0 0xe60b0000 0 0x425>; - interrupts = ; - clocks = <&cpg CPG_MOD 926>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 926>; - dmas = <&dmac0 0x11>, <&dmac0 0x10>; - dma-names = "tx", "rx"; - status = "disabled"; - }; - - pwm0: pwm@e6e30000 { - compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; - reg = <0 0xe6e30000 0 8>; - #pwm-cells = <2>; - clocks = <&cpg CPG_MOD 523>; - resets = <&cpg 523>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - status = "disabled"; - }; - - pwm1: pwm@e6e31000 { - compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; - reg = <0 0xe6e31000 0 8>; - #pwm-cells = <2>; - clocks = <&cpg CPG_MOD 523>; - resets = <&cpg 523>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - status = "disabled"; - }; - - pwm2: pwm@e6e32000 { - compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; - reg = <0 0xe6e32000 0 8>; - #pwm-cells = <2>; - clocks = <&cpg CPG_MOD 523>; - resets = <&cpg 523>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - status = "disabled"; - }; - - pwm3: pwm@e6e33000 { - compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; - reg = <0 0xe6e33000 0 8>; - #pwm-cells = <2>; - clocks = <&cpg CPG_MOD 523>; - resets = <&cpg 523>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - status = "disabled"; - }; - - pwm4: pwm@e6e34000 { - compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; - reg = <0 0xe6e34000 0 8>; - #pwm-cells = <2>; - clocks = <&cpg CPG_MOD 523>; - resets = <&cpg 523>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - status = "disabled"; - }; - - pwm5: pwm@e6e35000 { - compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; - reg = <0 0xe6e35000 0 8>; - #pwm-cells = <2>; - clocks = <&cpg CPG_MOD 523>; - resets = <&cpg 523>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - status = "disabled"; - }; - - pwm6: pwm@e6e36000 { - compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; - reg = <0 0xe6e36000 0 8>; - #pwm-cells = <2>; - clocks = <&cpg CPG_MOD 523>; - resets = <&cpg 523>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - status = "disabled"; - }; - - i2c0: i2c@e6500000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a7796", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe6500000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 931>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 931>; - dmas = <&dmac1 0x91>, <&dmac1 0x90>, - <&dmac2 0x91>, <&dmac2 0x90>; - dma-names = "tx", "rx", "tx", "rx"; - i2c-scl-internal-delay-ns = <110>; + i2c0: i2c@e6500000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a7796", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe6500000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 931>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 931>; + dmas = <&dmac1 0x91>, <&dmac1 0x90>, + <&dmac2 0x91>, <&dmac2 0x90>; + dma-names = "tx", "rx", "tx", "rx"; + i2c-scl-internal-delay-ns = <110>; status = "disabled"; }; @@ -758,244 +571,37 @@ status = "disabled"; }; - can0: can@e6c30000 { - compatible = "renesas,can-r8a7796", - "renesas,rcar-gen3-can"; - reg = <0 0xe6c30000 0 0x1000>; - interrupts = ; - clocks = <&cpg CPG_MOD 916>, - <&cpg CPG_CORE R8A7796_CLK_CANFD>, - <&can_clk>; - clock-names = "clkp1", "clkp2", "can_clk"; - assigned-clocks = <&cpg CPG_CORE R8A7796_CLK_CANFD>; - assigned-clock-rates = <40000000>; + i2c_dvfs: i2c@e60b0000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,iic-r8a7796", + "renesas,rcar-gen3-iic", + "renesas,rmobile-iic"; + reg = <0 0xe60b0000 0 0x425>; + interrupts = ; + clocks = <&cpg CPG_MOD 926>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 916>; + resets = <&cpg 926>; + dmas = <&dmac0 0x11>, <&dmac0 0x10>; + dma-names = "tx", "rx"; status = "disabled"; }; - can1: can@e6c38000 { - compatible = "renesas,can-r8a7796", - "renesas,rcar-gen3-can"; - reg = <0 0xe6c38000 0 0x1000>; - interrupts = ; - clocks = <&cpg CPG_MOD 915>, - <&cpg CPG_CORE R8A7796_CLK_CANFD>, - <&can_clk>; - clock-names = "clkp1", "clkp2", "can_clk"; - assigned-clocks = <&cpg CPG_CORE R8A7796_CLK_CANFD>; - assigned-clock-rates = <40000000>; + hscif0: serial@e6540000 { + compatible = "renesas,hscif-r8a7796", + "renesas,rcar-gen3-hscif", + "renesas,hscif"; + reg = <0 0xe6540000 0 0x60>; + interrupts = ; + clocks = <&cpg CPG_MOD 520>, + <&cpg CPG_CORE R8A7796_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac1 0x31>, <&dmac1 0x30>, + <&dmac2 0x31>, <&dmac2 0x30>; + dma-names = "tx", "rx", "tx", "rx"; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 915>; - status = "disabled"; - }; - - canfd: can@e66c0000 { - compatible = "renesas,r8a7796-canfd", - "renesas,rcar-gen3-canfd"; - reg = <0 0xe66c0000 0 0x8000>; - interrupts = , - ; - clocks = <&cpg CPG_MOD 914>, - <&cpg CPG_CORE R8A7796_CLK_CANFD>, - <&can_clk>; - clock-names = "fck", "canfd", "can_clk"; - assigned-clocks = <&cpg CPG_CORE R8A7796_CLK_CANFD>; - assigned-clock-rates = <40000000>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 914>; - status = "disabled"; - - channel0 { - status = "disabled"; - }; - - channel1 { - status = "disabled"; - }; - }; - - drif00: rif@e6f40000 { - compatible = "renesas,r8a7796-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f40000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 515>; - clock-names = "fck"; - dmas = <&dmac1 0x20>, <&dmac2 0x20>; - dma-names = "rx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 515>; - renesas,bonding = <&drif01>; - status = "disabled"; - }; - - drif01: rif@e6f50000 { - compatible = "renesas,r8a7796-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f50000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 514>; - clock-names = "fck"; - dmas = <&dmac1 0x22>, <&dmac2 0x22>; - dma-names = "rx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 514>; - renesas,bonding = <&drif00>; - status = "disabled"; - }; - - drif10: rif@e6f60000 { - compatible = "renesas,r8a7796-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f60000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 513>; - clock-names = "fck"; - dmas = <&dmac1 0x24>, <&dmac2 0x24>; - dma-names = "rx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 513>; - renesas,bonding = <&drif11>; - status = "disabled"; - }; - - drif11: rif@e6f70000 { - compatible = "renesas,r8a7796-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f70000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 512>; - clock-names = "fck"; - dmas = <&dmac1 0x26>, <&dmac2 0x26>; - dma-names = "rx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 512>; - renesas,bonding = <&drif10>; - status = "disabled"; - }; - - drif20: rif@e6f80000 { - compatible = "renesas,r8a7796-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f80000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 511>; - clock-names = "fck"; - dmas = <&dmac1 0x28>, <&dmac2 0x28>; - dma-names = "rx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 511>; - renesas,bonding = <&drif21>; - status = "disabled"; - }; - - drif21: rif@e6f90000 { - compatible = "renesas,r8a7796-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6f90000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 510>; - clock-names = "fck"; - dmas = <&dmac1 0x2a>, <&dmac2 0x2a>; - dma-names = "rx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 510>; - renesas,bonding = <&drif20>; - status = "disabled"; - }; - - drif30: rif@e6fa0000 { - compatible = "renesas,r8a7796-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6fa0000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 509>; - clock-names = "fck"; - dmas = <&dmac1 0x2c>, <&dmac2 0x2c>; - dma-names = "rx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 509>; - renesas,bonding = <&drif31>; - status = "disabled"; - }; - - drif31: rif@e6fb0000 { - compatible = "renesas,r8a7796-drif", - "renesas,rcar-gen3-drif"; - reg = <0 0xe6fb0000 0 0x64>; - interrupts = ; - clocks = <&cpg CPG_MOD 508>; - clock-names = "fck"; - dmas = <&dmac1 0x2e>, <&dmac2 0x2e>; - dma-names = "rx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 508>; - renesas,bonding = <&drif30>; - status = "disabled"; - }; - - avb: ethernet@e6800000 { - compatible = "renesas,etheravb-r8a7796", - "renesas,etheravb-rcar-gen3"; - reg = <0 0xe6800000 0 0x800>, <0 0xe6a00000 0 0x10000>; - interrupts = , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - ; - interrupt-names = "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7", - "ch8", "ch9", "ch10", "ch11", - "ch12", "ch13", "ch14", "ch15", - "ch16", "ch17", "ch18", "ch19", - "ch20", "ch21", "ch22", "ch23", - "ch24"; - clocks = <&cpg CPG_MOD 812>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 812>; - phy-mode = "rgmii"; - iommus = <&ipmmu_ds0 16>; - #address-cells = <1>; - #size-cells = <0>; - status = "disabled"; - }; - - hscif0: serial@e6540000 { - compatible = "renesas,hscif-r8a7796", - "renesas,rcar-gen3-hscif", - "renesas,hscif"; - reg = <0 0xe6540000 0 0x60>; - interrupts = ; - clocks = <&cpg CPG_MOD 520>, - <&cpg CPG_CORE R8A7796_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x31>, <&dmac1 0x30>, - <&dmac2 0x31>, <&dmac2 0x30>; - dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 520>; + resets = <&cpg 520>; status = "disabled"; }; @@ -1069,162 +675,61 @@ status = "disabled"; }; - scif0: serial@e6e60000 { - compatible = "renesas,scif-r8a7796", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6e60000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 207>, - <&cpg CPG_CORE R8A7796_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x51>, <&dmac1 0x50>, - <&dmac2 0x51>, <&dmac2 0x50>; - dma-names = "tx", "rx", "tx", "rx"; + hsusb: usb@e6590000 { + compatible = "renesas,usbhs-r8a7796", + "renesas,rcar-gen3-usbhs"; + reg = <0 0xe6590000 0 0x100>; + interrupts = ; + clocks = <&cpg CPG_MOD 704>; + dmas = <&usb_dmac0 0>, <&usb_dmac0 1>, + <&usb_dmac1 0>, <&usb_dmac1 1>; + dma-names = "ch0", "ch1", "ch2", "ch3"; + renesas,buswait = <11>; + phys = <&usb2_phy0>; + phy-names = "usb"; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 207>; + resets = <&cpg 704>; status = "disabled"; }; - scif1: serial@e6e68000 { - compatible = "renesas,scif-r8a7796", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6e68000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 206>, - <&cpg CPG_CORE R8A7796_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x53>, <&dmac1 0x52>, - <&dmac2 0x53>, <&dmac2 0x52>; - dma-names = "tx", "rx", "tx", "rx"; + usb_dmac0: dma-controller@e65a0000 { + compatible = "renesas,r8a7796-usb-dmac", + "renesas,usb-dmac"; + reg = <0 0xe65a0000 0 0x100>; + interrupts = ; + interrupt-names = "ch0", "ch1"; + clocks = <&cpg CPG_MOD 330>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 206>; - status = "disabled"; + resets = <&cpg 330>; + #dma-cells = <1>; + dma-channels = <2>; }; - scif2: serial@e6e88000 { - compatible = "renesas,scif-r8a7796", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6e88000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 310>, - <&cpg CPG_CORE R8A7796_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 310>; - status = "disabled"; - }; - - scif3: serial@e6c50000 { - compatible = "renesas,scif-r8a7796", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6c50000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 204>, - <&cpg CPG_CORE R8A7796_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac0 0x57>, <&dmac0 0x56>; - dma-names = "tx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 204>; - status = "disabled"; - }; - - scif4: serial@e6c40000 { - compatible = "renesas,scif-r8a7796", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6c40000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 203>, - <&cpg CPG_CORE R8A7796_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac0 0x59>, <&dmac0 0x58>; - dma-names = "tx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 203>; - status = "disabled"; - }; - - scif5: serial@e6f30000 { - compatible = "renesas,scif-r8a7796", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6f30000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 202>, - <&cpg CPG_CORE R8A7796_CLK_S3D1>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x5b>, <&dmac1 0x5a>, - <&dmac2 0x5b>, <&dmac2 0x5a>; - dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 202>; - status = "disabled"; - }; - - msiof0: spi@e6e90000 { - compatible = "renesas,msiof-r8a7796", - "renesas,rcar-gen3-msiof"; - reg = <0 0xe6e90000 0 0x0064>; - interrupts = ; - clocks = <&cpg CPG_MOD 211>; - dmas = <&dmac1 0x41>, <&dmac1 0x40>, - <&dmac2 0x41>, <&dmac2 0x40>; - dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 211>; - #address-cells = <1>; - #size-cells = <0>; - status = "disabled"; - }; - - msiof1: spi@e6ea0000 { - compatible = "renesas,msiof-r8a7796", - "renesas,rcar-gen3-msiof"; - reg = <0 0xe6ea0000 0 0x0064>; - interrupts = ; - clocks = <&cpg CPG_MOD 210>; - dmas = <&dmac1 0x43>, <&dmac1 0x42>, - <&dmac2 0x43>, <&dmac2 0x42>; - dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 210>; - #address-cells = <1>; - #size-cells = <0>; - status = "disabled"; - }; - - msiof2: spi@e6c00000 { - compatible = "renesas,msiof-r8a7796", - "renesas,rcar-gen3-msiof"; - reg = <0 0xe6c00000 0 0x0064>; - interrupts = ; - clocks = <&cpg CPG_MOD 209>; - dmas = <&dmac0 0x45>, <&dmac0 0x44>; - dma-names = "tx", "rx"; + usb_dmac1: dma-controller@e65b0000 { + compatible = "renesas,r8a7796-usb-dmac", + "renesas,usb-dmac"; + reg = <0 0xe65b0000 0 0x100>; + interrupts = ; + interrupt-names = "ch0", "ch1"; + clocks = <&cpg CPG_MOD 331>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 209>; - #address-cells = <1>; - #size-cells = <0>; - status = "disabled"; + resets = <&cpg 331>; + #dma-cells = <1>; + dma-channels = <2>; }; - msiof3: spi@e6c10000 { - compatible = "renesas,msiof-r8a7796", - "renesas,rcar-gen3-msiof"; - reg = <0 0xe6c10000 0 0x0064>; - interrupts = ; - clocks = <&cpg CPG_MOD 208>; - dmas = <&dmac0 0x47>, <&dmac0 0x46>; - dma-names = "tx", "rx"; + usb3_phy0: usb-phy@e65ee000 { + compatible = "renesas,r8a7796-usb3-phy", + "renesas,rcar-gen3-usb3-phy"; + reg = <0 0xe65ee000 0 0x90>; + clocks = <&cpg CPG_MOD 328>, <&usb3s0_clk>, + <&usb_extal_clk>; + clock-names = "usb3-if", "usb3s_clk", "usb_extal"; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 208>; - #address-cells = <1>; - #size-cells = <0>; + resets = <&cpg 328>; + #phy-cells = <0>; status = "disabled"; }; @@ -1354,306 +859,803 @@ <&ipmmu_ds1 30>, <&ipmmu_ds1 31>; }; - audma0: dma-controller@ec700000 { - compatible = "renesas,dmac-r8a7796", - "renesas,rcar-dmac"; - reg = <0 0xec700000 0 0x10000>; - interrupts = ; - interrupt-names = "error", - "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7", - "ch8", "ch9", "ch10", "ch11", - "ch12", "ch13", "ch14", "ch15"; - clocks = <&cpg CPG_MOD 502>; - clock-names = "fck"; + ipmmu_ds0: mmu@e6740000 { + compatible = "renesas,ipmmu-r8a7796"; + reg = <0 0xe6740000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 0>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 502>; - #dma-cells = <1>; - dma-channels = <16>; - iommus = <&ipmmu_mp 0>, <&ipmmu_mp 1>, - <&ipmmu_mp 2>, <&ipmmu_mp 3>, - <&ipmmu_mp 4>, <&ipmmu_mp 5>, - <&ipmmu_mp 6>, <&ipmmu_mp 7>, - <&ipmmu_mp 8>, <&ipmmu_mp 9>, - <&ipmmu_mp 10>, <&ipmmu_mp 11>, - <&ipmmu_mp 12>, <&ipmmu_mp 13>, - <&ipmmu_mp 14>, <&ipmmu_mp 15>; + #iommu-cells = <1>; }; - audma1: dma-controller@ec720000 { - compatible = "renesas,dmac-r8a7796", - "renesas,rcar-dmac"; - reg = <0 0xec720000 0 0x10000>; - interrupts = ; - interrupt-names = "error", - "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7", - "ch8", "ch9", "ch10", "ch11", - "ch12", "ch13", "ch14", "ch15"; - clocks = <&cpg CPG_MOD 501>; - clock-names = "fck"; + ipmmu_ds1: mmu@e7740000 { + compatible = "renesas,ipmmu-r8a7796"; + reg = <0 0xe7740000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 1>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 501>; - #dma-cells = <1>; - dma-channels = <16>; - iommus = <&ipmmu_mp 16>, <&ipmmu_mp 17>, - <&ipmmu_mp 18>, <&ipmmu_mp 19>, - <&ipmmu_mp 20>, <&ipmmu_mp 21>, - <&ipmmu_mp 22>, <&ipmmu_mp 23>, - <&ipmmu_mp 24>, <&ipmmu_mp 25>, - <&ipmmu_mp 26>, <&ipmmu_mp 27>, - <&ipmmu_mp 28>, <&ipmmu_mp 29>, - <&ipmmu_mp 30>, <&ipmmu_mp 31>; + #iommu-cells = <1>; }; - usb_dmac0: dma-controller@e65a0000 { - compatible = "renesas,r8a7796-usb-dmac", - "renesas,usb-dmac"; - reg = <0 0xe65a0000 0 0x100>; - interrupts = ; - interrupt-names = "ch0", "ch1"; - clocks = <&cpg CPG_MOD 330>; + ipmmu_hc: mmu@e6570000 { + compatible = "renesas,ipmmu-r8a7796"; + reg = <0 0xe6570000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 2>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 330>; - #dma-cells = <1>; - dma-channels = <2>; + #iommu-cells = <1>; }; - usb_dmac1: dma-controller@e65b0000 { - compatible = "renesas,r8a7796-usb-dmac", - "renesas,usb-dmac"; - reg = <0 0xe65b0000 0 0x100>; - interrupts = ; - interrupt-names = "ch0", "ch1"; - clocks = <&cpg CPG_MOD 331>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 331>; - #dma-cells = <1>; - dma-channels = <2>; + ipmmu_ir: mmu@ff8b0000 { + compatible = "renesas,ipmmu-r8a7796"; + reg = <0 0xff8b0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 3>; + power-domains = <&sysc R8A7796_PD_A3IR>; + #iommu-cells = <1>; }; - hsusb: usb@e6590000 { - compatible = "renesas,usbhs-r8a7796", - "renesas,rcar-gen3-usbhs"; - reg = <0 0xe6590000 0 0x100>; - interrupts = ; - clocks = <&cpg CPG_MOD 704>; - dmas = <&usb_dmac0 0>, <&usb_dmac0 1>, - <&usb_dmac1 0>, <&usb_dmac1 1>; - dma-names = "ch0", "ch1", "ch2", "ch3"; - renesas,buswait = <11>; - phys = <&usb2_phy0>; - phy-names = "usb"; + ipmmu_mm: mmu@e67b0000 { + compatible = "renesas,ipmmu-r8a7796"; + reg = <0 0xe67b0000 0 0x1000>; + interrupts = , + ; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_mp: mmu@ec670000 { + compatible = "renesas,ipmmu-r8a7796"; + reg = <0 0xec670000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 4>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_pv0: mmu@fd800000 { + compatible = "renesas,ipmmu-r8a7796"; + reg = <0 0xfd800000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 5>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_pv1: mmu@fd950000 { + compatible = "renesas,ipmmu-r8a7796"; + reg = <0 0xfd950000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 6>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_rt: mmu@ffc80000 { + compatible = "renesas,ipmmu-r8a7796"; + reg = <0 0xffc80000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 7>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_vc0: mmu@fe6b0000 { + compatible = "renesas,ipmmu-r8a7796"; + reg = <0 0xfe6b0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 8>; + power-domains = <&sysc R8A7796_PD_A3VC>; + #iommu-cells = <1>; + }; + + ipmmu_vi0: mmu@febd0000 { + compatible = "renesas,ipmmu-r8a7796"; + reg = <0 0xfebd0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 9>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + avb: ethernet@e6800000 { + compatible = "renesas,etheravb-r8a7796", + "renesas,etheravb-rcar-gen3"; + reg = <0 0xe6800000 0 0x800>, <0 0xe6a00000 0 0x10000>; + interrupts = , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + ; + interrupt-names = "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14", "ch15", + "ch16", "ch17", "ch18", "ch19", + "ch20", "ch21", "ch22", "ch23", + "ch24"; + clocks = <&cpg CPG_MOD 812>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 812>; + phy-mode = "rgmii"; + iommus = <&ipmmu_ds0 16>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + can0: can@e6c30000 { + compatible = "renesas,can-r8a7796", + "renesas,rcar-gen3-can"; + reg = <0 0xe6c30000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 916>, + <&cpg CPG_CORE R8A7796_CLK_CANFD>, + <&can_clk>; + clock-names = "clkp1", "clkp2", "can_clk"; + assigned-clocks = <&cpg CPG_CORE R8A7796_CLK_CANFD>; + assigned-clock-rates = <40000000>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 916>; + status = "disabled"; + }; + + can1: can@e6c38000 { + compatible = "renesas,can-r8a7796", + "renesas,rcar-gen3-can"; + reg = <0 0xe6c38000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 915>, + <&cpg CPG_CORE R8A7796_CLK_CANFD>, + <&can_clk>; + clock-names = "clkp1", "clkp2", "can_clk"; + assigned-clocks = <&cpg CPG_CORE R8A7796_CLK_CANFD>; + assigned-clock-rates = <40000000>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 915>; + status = "disabled"; + }; + + canfd: can@e66c0000 { + compatible = "renesas,r8a7796-canfd", + "renesas,rcar-gen3-canfd"; + reg = <0 0xe66c0000 0 0x8000>; + interrupts = , + ; + clocks = <&cpg CPG_MOD 914>, + <&cpg CPG_CORE R8A7796_CLK_CANFD>, + <&can_clk>; + clock-names = "fck", "canfd", "can_clk"; + assigned-clocks = <&cpg CPG_CORE R8A7796_CLK_CANFD>; + assigned-clock-rates = <40000000>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 914>; + status = "disabled"; + + channel0 { + status = "disabled"; + }; + + channel1 { + status = "disabled"; + }; + }; + + pwm0: pwm@e6e30000 { + compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; + reg = <0 0xe6e30000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm1: pwm@e6e31000 { + compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; + reg = <0 0xe6e31000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm2: pwm@e6e32000 { + compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; + reg = <0 0xe6e32000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm3: pwm@e6e33000 { + compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; + reg = <0 0xe6e33000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm4: pwm@e6e34000 { + compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; + reg = <0 0xe6e34000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm5: pwm@e6e35000 { + compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; + reg = <0 0xe6e35000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm6: pwm@e6e36000 { + compatible = "renesas,pwm-r8a7796", "renesas,pwm-rcar"; + reg = <0 0xe6e36000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + status = "disabled"; + }; + + scif0: serial@e6e60000 { + compatible = "renesas,scif-r8a7796", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6e60000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 207>, + <&cpg CPG_CORE R8A7796_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac1 0x51>, <&dmac1 0x50>, + <&dmac2 0x51>, <&dmac2 0x50>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 207>; + status = "disabled"; + }; + + scif1: serial@e6e68000 { + compatible = "renesas,scif-r8a7796", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6e68000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 206>, + <&cpg CPG_CORE R8A7796_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac1 0x53>, <&dmac1 0x52>, + <&dmac2 0x53>, <&dmac2 0x52>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 206>; + status = "disabled"; + }; + + scif2: serial@e6e88000 { + compatible = "renesas,scif-r8a7796", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6e88000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 310>, + <&cpg CPG_CORE R8A7796_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 310>; + status = "disabled"; + }; + + scif3: serial@e6c50000 { + compatible = "renesas,scif-r8a7796", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6c50000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 204>, + <&cpg CPG_CORE R8A7796_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac0 0x57>, <&dmac0 0x56>; + dma-names = "tx", "rx"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 204>; + status = "disabled"; + }; + + scif4: serial@e6c40000 { + compatible = "renesas,scif-r8a7796", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6c40000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 203>, + <&cpg CPG_CORE R8A7796_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac0 0x59>, <&dmac0 0x58>; + dma-names = "tx", "rx"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 203>; + status = "disabled"; + }; + + scif5: serial@e6f30000 { + compatible = "renesas,scif-r8a7796", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6f30000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 202>, + <&cpg CPG_CORE R8A7796_CLK_S3D1>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac1 0x5b>, <&dmac1 0x5a>, + <&dmac2 0x5b>, <&dmac2 0x5a>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 202>; + status = "disabled"; + }; + + msiof0: spi@e6e90000 { + compatible = "renesas,msiof-r8a7796", + "renesas,rcar-gen3-msiof"; + reg = <0 0xe6e90000 0 0x0064>; + interrupts = ; + clocks = <&cpg CPG_MOD 211>; + dmas = <&dmac1 0x41>, <&dmac1 0x40>, + <&dmac2 0x41>, <&dmac2 0x40>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 211>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + msiof1: spi@e6ea0000 { + compatible = "renesas,msiof-r8a7796", + "renesas,rcar-gen3-msiof"; + reg = <0 0xe6ea0000 0 0x0064>; + interrupts = ; + clocks = <&cpg CPG_MOD 210>; + dmas = <&dmac1 0x43>, <&dmac1 0x42>, + <&dmac2 0x43>, <&dmac2 0x42>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 210>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + msiof2: spi@e6c00000 { + compatible = "renesas,msiof-r8a7796", + "renesas,rcar-gen3-msiof"; + reg = <0 0xe6c00000 0 0x0064>; + interrupts = ; + clocks = <&cpg CPG_MOD 209>; + dmas = <&dmac0 0x45>, <&dmac0 0x44>; + dma-names = "tx", "rx"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 209>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + msiof3: spi@e6c10000 { + compatible = "renesas,msiof-r8a7796", + "renesas,rcar-gen3-msiof"; + reg = <0 0xe6c10000 0 0x0064>; + interrupts = ; + clocks = <&cpg CPG_MOD 208>; + dmas = <&dmac0 0x47>, <&dmac0 0x46>; + dma-names = "tx", "rx"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 208>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + vin0: video@e6ef0000 { + compatible = "renesas,vin-r8a7796"; + reg = <0 0xe6ef0000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 811>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 811>; + renesas,id = <0>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin0csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin0>; + }; + vin0csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin0>; + }; + }; + }; + }; + + vin1: video@e6ef1000 { + compatible = "renesas,vin-r8a7796"; + reg = <0 0xe6ef1000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 810>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 704>; + resets = <&cpg 810>; + renesas,id = <1>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin1csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin1>; + }; + vin1csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin1>; + }; + }; + }; }; - usb3_phy0: usb-phy@e65ee000 { - compatible = "renesas,r8a7796-usb3-phy", - "renesas,rcar-gen3-usb3-phy"; - reg = <0 0xe65ee000 0 0x90>; - clocks = <&cpg CPG_MOD 328>, <&usb3s0_clk>, - <&usb_extal_clk>; - clock-names = "usb3-if", "usb3s_clk", "usb_extal"; + vin2: video@e6ef2000 { + compatible = "renesas,vin-r8a7796"; + reg = <0 0xe6ef2000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 809>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 328>; - #phy-cells = <0>; + resets = <&cpg 809>; + renesas,id = <2>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin2csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin2>; + }; + vin2csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin2>; + }; + }; + }; }; - xhci0: usb@ee000000 { - compatible = "renesas,xhci-r8a7796", - "renesas,rcar-gen3-xhci"; - reg = <0 0xee000000 0 0xc00>; - interrupts = ; - clocks = <&cpg CPG_MOD 328>; + vin3: video@e6ef3000 { + compatible = "renesas,vin-r8a7796"; + reg = <0 0xe6ef3000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 808>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 328>; + resets = <&cpg 808>; + renesas,id = <3>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin3csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin3>; + }; + vin3csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin3>; + }; + }; + }; }; - usb3_peri0: usb@ee020000 { - compatible = "renesas,r8a7796-usb3-peri", - "renesas,rcar-gen3-usb3-peri"; - reg = <0 0xee020000 0 0x400>; - interrupts = ; - clocks = <&cpg CPG_MOD 328>; + vin4: video@e6ef4000 { + compatible = "renesas,vin-r8a7796"; + reg = <0 0xe6ef4000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 807>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 328>; + resets = <&cpg 807>; + renesas,id = <4>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin4csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin4>; + }; + vin4csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin4>; + }; + }; + }; }; - ohci0: usb@ee080000 { - compatible = "generic-ohci"; - reg = <0 0xee080000 0 0x100>; - interrupts = ; - clocks = <&cpg CPG_MOD 703>; - phys = <&usb2_phy0>; - phy-names = "usb"; + vin5: video@e6ef5000 { + compatible = "renesas,vin-r8a7796"; + reg = <0 0xe6ef5000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 806>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 703>; + resets = <&cpg 806>; + renesas,id = <5>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin5csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin5>; + }; + vin5csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin5>; + }; + }; + }; }; - ehci0: usb@ee080100 { - compatible = "generic-ehci"; - reg = <0 0xee080100 0 0x100>; - interrupts = ; - clocks = <&cpg CPG_MOD 703>; - phys = <&usb2_phy0>; - phy-names = "usb"; - companion= <&ohci0>; + vin6: video@e6ef6000 { + compatible = "renesas,vin-r8a7796"; + reg = <0 0xe6ef6000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 805>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 703>; + resets = <&cpg 805>; + renesas,id = <6>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin6csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin6>; + }; + vin6csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin6>; + }; + }; + }; + }; + + vin7: video@e6ef7000 { + compatible = "renesas,vin-r8a7796"; + reg = <0 0xe6ef7000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 804>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 804>; + renesas,id = <7>; status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin7csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin7>; + }; + vin7csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin7>; + }; + }; + }; }; - usb2_phy0: usb-phy@ee080200 { - compatible = "renesas,usb2-phy-r8a7796", - "renesas,rcar-gen3-usb2-phy"; - reg = <0 0xee080200 0 0x700>; - interrupts = ; - clocks = <&cpg CPG_MOD 703>; + drif00: rif@e6f40000 { + compatible = "renesas,r8a7796-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f40000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 515>; + clock-names = "fck"; + dmas = <&dmac1 0x20>, <&dmac2 0x20>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 703>; - #phy-cells = <0>; + resets = <&cpg 515>; + renesas,bonding = <&drif01>; status = "disabled"; }; - ohci1: usb@ee0a0000 { - compatible = "generic-ohci"; - reg = <0 0xee0a0000 0 0x100>; - interrupts = ; - clocks = <&cpg CPG_MOD 702>; - phys = <&usb2_phy1>; - phy-names = "usb"; + drif01: rif@e6f50000 { + compatible = "renesas,r8a7796-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f50000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 514>; + clock-names = "fck"; + dmas = <&dmac1 0x22>, <&dmac2 0x22>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 702>; + resets = <&cpg 514>; + renesas,bonding = <&drif00>; status = "disabled"; }; - ehci1: usb@ee0a0100 { - compatible = "generic-ehci"; - reg = <0 0xee0a0100 0 0x100>; - interrupts = ; - clocks = <&cpg CPG_MOD 702>; - phys = <&usb2_phy1>; - phy-names = "usb"; - companion= <&ohci1>; + drif10: rif@e6f60000 { + compatible = "renesas,r8a7796-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f60000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 513>; + clock-names = "fck"; + dmas = <&dmac1 0x24>, <&dmac2 0x24>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 702>; + resets = <&cpg 513>; + renesas,bonding = <&drif11>; status = "disabled"; }; - usb2_phy1: usb-phy@ee0a0200 { - compatible = "renesas,usb2-phy-r8a7796", - "renesas,rcar-gen3-usb2-phy"; - reg = <0 0xee0a0200 0 0x700>; - clocks = <&cpg CPG_MOD 702>; + drif11: rif@e6f70000 { + compatible = "renesas,r8a7796-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f70000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 512>; + clock-names = "fck"; + dmas = <&dmac1 0x26>, <&dmac2 0x26>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 702>; - #phy-cells = <0>; + resets = <&cpg 512>; + renesas,bonding = <&drif10>; status = "disabled"; }; - sdhi0: sd@ee100000 { - compatible = "renesas,sdhi-r8a7796", - "renesas,rcar-gen3-sdhi"; - reg = <0 0xee100000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 314>; - max-frequency = <200000000>; + drif20: rif@e6f80000 { + compatible = "renesas,r8a7796-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f80000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 511>; + clock-names = "fck"; + dmas = <&dmac1 0x28>, <&dmac2 0x28>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 314>; + resets = <&cpg 511>; + renesas,bonding = <&drif21>; status = "disabled"; }; - sdhi1: sd@ee120000 { - compatible = "renesas,sdhi-r8a7796", - "renesas,rcar-gen3-sdhi"; - reg = <0 0xee120000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 313>; - max-frequency = <200000000>; + drif21: rif@e6f90000 { + compatible = "renesas,r8a7796-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6f90000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 510>; + clock-names = "fck"; + dmas = <&dmac1 0x2a>, <&dmac2 0x2a>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 313>; + resets = <&cpg 510>; + renesas,bonding = <&drif20>; status = "disabled"; }; - sdhi2: sd@ee140000 { - compatible = "renesas,sdhi-r8a7796", - "renesas,rcar-gen3-sdhi"; - reg = <0 0xee140000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 312>; - max-frequency = <200000000>; + drif30: rif@e6fa0000 { + compatible = "renesas,r8a7796-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6fa0000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 509>; + clock-names = "fck"; + dmas = <&dmac1 0x2c>, <&dmac2 0x2c>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 312>; + resets = <&cpg 509>; + renesas,bonding = <&drif31>; status = "disabled"; }; - sdhi3: sd@ee160000 { - compatible = "renesas,sdhi-r8a7796", - "renesas,rcar-gen3-sdhi"; - reg = <0 0xee160000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 311>; - max-frequency = <200000000>; + drif31: rif@e6fb0000 { + compatible = "renesas,r8a7796-drif", + "renesas,rcar-gen3-drif"; + reg = <0 0xe6fb0000 0 0x64>; + interrupts = ; + clocks = <&cpg CPG_MOD 508>; + clock-names = "fck"; + dmas = <&dmac1 0x2e>, <&dmac2 0x2e>; + dma-names = "rx", "rx"; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 311>; + resets = <&cpg 508>; + renesas,bonding = <&drif30>; status = "disabled"; }; - tsc: thermal@e6198000 { - compatible = "renesas,r8a7796-thermal"; - reg = <0 0xe6198000 0 0x100>, - <0 0xe61a0000 0 0x100>, - <0 0xe61a8000 0 0x100>; - interrupts = , - , - ; - clocks = <&cpg CPG_MOD 522>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 522>; - #thermal-sensor-cells = <1>; - status = "okay"; - }; - rcar_sound: sound@ec500000 { /* * #sound-dai-cells is required @@ -1848,6 +1850,261 @@ dma-names = "rx", "tx", "rxu", "txu"; }; }; + + ports { + #address-cells = <1>; + #size-cells = <0>; + port@0 { + reg = <0>; + }; + port@1 { + reg = <1>; + }; + }; + }; + + audma0: dma-controller@ec700000 { + compatible = "renesas,dmac-r8a7796", + "renesas,rcar-dmac"; + reg = <0 0xec700000 0 0x10000>; + interrupts = ; + interrupt-names = "error", + "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14", "ch15"; + clocks = <&cpg CPG_MOD 502>; + clock-names = "fck"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 502>; + #dma-cells = <1>; + dma-channels = <16>; + iommus = <&ipmmu_mp 0>, <&ipmmu_mp 1>, + <&ipmmu_mp 2>, <&ipmmu_mp 3>, + <&ipmmu_mp 4>, <&ipmmu_mp 5>, + <&ipmmu_mp 6>, <&ipmmu_mp 7>, + <&ipmmu_mp 8>, <&ipmmu_mp 9>, + <&ipmmu_mp 10>, <&ipmmu_mp 11>, + <&ipmmu_mp 12>, <&ipmmu_mp 13>, + <&ipmmu_mp 14>, <&ipmmu_mp 15>; + }; + + audma1: dma-controller@ec720000 { + compatible = "renesas,dmac-r8a7796", + "renesas,rcar-dmac"; + reg = <0 0xec720000 0 0x10000>; + interrupts = ; + interrupt-names = "error", + "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14", "ch15"; + clocks = <&cpg CPG_MOD 501>; + clock-names = "fck"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 501>; + #dma-cells = <1>; + dma-channels = <16>; + iommus = <&ipmmu_mp 16>, <&ipmmu_mp 17>, + <&ipmmu_mp 18>, <&ipmmu_mp 19>, + <&ipmmu_mp 20>, <&ipmmu_mp 21>, + <&ipmmu_mp 22>, <&ipmmu_mp 23>, + <&ipmmu_mp 24>, <&ipmmu_mp 25>, + <&ipmmu_mp 26>, <&ipmmu_mp 27>, + <&ipmmu_mp 28>, <&ipmmu_mp 29>, + <&ipmmu_mp 30>, <&ipmmu_mp 31>; + }; + + xhci0: usb@ee000000 { + compatible = "renesas,xhci-r8a7796", + "renesas,rcar-gen3-xhci"; + reg = <0 0xee000000 0 0xc00>; + interrupts = ; + clocks = <&cpg CPG_MOD 328>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 328>; + status = "disabled"; + }; + + usb3_peri0: usb@ee020000 { + compatible = "renesas,r8a7796-usb3-peri", + "renesas,rcar-gen3-usb3-peri"; + reg = <0 0xee020000 0 0x400>; + interrupts = ; + clocks = <&cpg CPG_MOD 328>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 328>; + status = "disabled"; + }; + + ohci0: usb@ee080000 { + compatible = "generic-ohci"; + reg = <0 0xee080000 0 0x100>; + interrupts = ; + clocks = <&cpg CPG_MOD 703>; + phys = <&usb2_phy0>; + phy-names = "usb"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 703>; + status = "disabled"; + }; + + ohci1: usb@ee0a0000 { + compatible = "generic-ohci"; + reg = <0 0xee0a0000 0 0x100>; + interrupts = ; + clocks = <&cpg CPG_MOD 702>; + phys = <&usb2_phy1>; + phy-names = "usb"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 702>; + status = "disabled"; + }; + + ehci0: usb@ee080100 { + compatible = "generic-ehci"; + reg = <0 0xee080100 0 0x100>; + interrupts = ; + clocks = <&cpg CPG_MOD 703>; + phys = <&usb2_phy0>; + phy-names = "usb"; + companion= <&ohci0>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 703>; + status = "disabled"; + }; + + ehci1: usb@ee0a0100 { + compatible = "generic-ehci"; + reg = <0 0xee0a0100 0 0x100>; + interrupts = ; + clocks = <&cpg CPG_MOD 702>; + phys = <&usb2_phy1>; + phy-names = "usb"; + companion= <&ohci1>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 702>; + status = "disabled"; + }; + + usb2_phy0: usb-phy@ee080200 { + compatible = "renesas,usb2-phy-r8a7796", + "renesas,rcar-gen3-usb2-phy"; + reg = <0 0xee080200 0 0x700>; + interrupts = ; + clocks = <&cpg CPG_MOD 703>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 703>; + #phy-cells = <0>; + status = "disabled"; + }; + + usb2_phy1: usb-phy@ee0a0200 { + compatible = "renesas,usb2-phy-r8a7796", + "renesas,rcar-gen3-usb2-phy"; + reg = <0 0xee0a0200 0 0x700>; + clocks = <&cpg CPG_MOD 702>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 702>; + #phy-cells = <0>; + status = "disabled"; + }; + + sdhi0: sd@ee100000 { + compatible = "renesas,sdhi-r8a7796", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee100000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 314>; + max-frequency = <200000000>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 314>; + status = "disabled"; + }; + + sdhi1: sd@ee120000 { + compatible = "renesas,sdhi-r8a7796", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee120000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 313>; + max-frequency = <200000000>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 313>; + status = "disabled"; + }; + + sdhi2: sd@ee140000 { + compatible = "renesas,sdhi-r8a7796", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee140000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 312>; + max-frequency = <200000000>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 312>; + status = "disabled"; + }; + + sdhi3: sd@ee160000 { + compatible = "renesas,sdhi-r8a7796", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee160000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 311>; + max-frequency = <200000000>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 311>; + status = "disabled"; + }; + + gic: interrupt-controller@f1010000 { + compatible = "arm,gic-400"; + #interrupt-cells = <3>; + #address-cells = <0>; + interrupt-controller; + reg = <0x0 0xf1010000 0 0x1000>, + <0x0 0xf1020000 0 0x20000>, + <0x0 0xf1040000 0 0x20000>, + <0x0 0xf1060000 0 0x20000>; + interrupts = ; + clocks = <&cpg CPG_MOD 408>; + clock-names = "clk"; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 408>; }; pciec0: pcie@fe000000 { @@ -1860,6 +2117,26 @@ /* placeholder */ }; + imr-lx4@fe860000 { + compatible = "renesas,r8a7796-imr-lx4", + "renesas,imr-lx4"; + reg = <0 0xfe860000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 823>; + power-domains = <&sysc R8A7796_PD_A3VC>; + resets = <&cpg 823>; + }; + + imr-lx4@fe870000 { + compatible = "renesas,r8a7796-imr-lx4", + "renesas,imr-lx4"; + reg = <0 0xfe870000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 822>; + power-domains = <&sysc R8A7796_PD_A3VC>; + resets = <&cpg 822>; + }; + fdp1@fe940000 { compatible = "renesas,fdp1"; reg = <0 0xfe940000 0 0x2400>; @@ -1878,17 +2155,6 @@ resets = <&cpg 615>; }; - vspb: vsp@fe960000 { - compatible = "renesas,vsp2"; - reg = <0 0xfe960000 0 0x8000>; - interrupts = ; - clocks = <&cpg CPG_MOD 626>; - power-domains = <&sysc R8A7796_PD_A3VC>; - resets = <&cpg 626>; - - renesas,fcp = <&fcpvb0>; - }; - fcpvb0: fcp@fe96f000 { compatible = "renesas,fcpv"; reg = <0 0xfe96f000 0 0x200>; @@ -1897,17 +2163,6 @@ resets = <&cpg 607>; }; - vspi0: vsp@fe9a0000 { - compatible = "renesas,vsp2"; - reg = <0 0xfe9a0000 0 0x8000>; - interrupts = ; - clocks = <&cpg CPG_MOD 631>; - power-domains = <&sysc R8A7796_PD_A3VC>; - resets = <&cpg 631>; - - renesas,fcp = <&fcpvi0>; - }; - fcpvi0: fcp@fe9af000 { compatible = "renesas,fcpv"; reg = <0 0xfe9af000 0 0x200>; @@ -1917,6 +2172,44 @@ iommus = <&ipmmu_vc0 19>; }; + fcpvd0: fcp@fea27000 { + compatible = "renesas,fcpv"; + reg = <0 0xfea27000 0 0x200>; + clocks = <&cpg CPG_MOD 603>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 603>; + iommus = <&ipmmu_vi0 8>; + }; + + fcpvd1: fcp@fea2f000 { + compatible = "renesas,fcpv"; + reg = <0 0xfea2f000 0 0x200>; + clocks = <&cpg CPG_MOD 602>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 602>; + iommus = <&ipmmu_vi0 9>; + }; + + fcpvd2: fcp@fea37000 { + compatible = "renesas,fcpv"; + reg = <0 0xfea37000 0 0x200>; + clocks = <&cpg CPG_MOD 601>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 601>; + iommus = <&ipmmu_vi0 10>; + }; + + vspb: vsp@fe960000 { + compatible = "renesas,vsp2"; + reg = <0 0xfe960000 0 0x8000>; + interrupts = ; + clocks = <&cpg CPG_MOD 626>; + power-domains = <&sysc R8A7796_PD_A3VC>; + resets = <&cpg 626>; + + renesas,fcp = <&fcpvb0>; + }; + vspd0: vsp@fea20000 { compatible = "renesas,vsp2"; reg = <0 0xfea20000 0 0x8000>; @@ -1928,15 +2221,6 @@ renesas,fcp = <&fcpvd0>; }; - fcpvd0: fcp@fea27000 { - compatible = "renesas,fcpv"; - reg = <0 0xfea27000 0 0x200>; - clocks = <&cpg CPG_MOD 603>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 603>; - iommus = <&ipmmu_vi0 8>; - }; - vspd1: vsp@fea28000 { compatible = "renesas,vsp2"; reg = <0 0xfea28000 0 0x8000>; @@ -1948,15 +2232,6 @@ renesas,fcp = <&fcpvd1>; }; - fcpvd1: fcp@fea2f000 { - compatible = "renesas,fcpv"; - reg = <0 0xfea2f000 0 0x200>; - clocks = <&cpg CPG_MOD 602>; - power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 602>; - iommus = <&ipmmu_vi0 9>; - }; - vspd2: vsp@fea30000 { compatible = "renesas,vsp2"; reg = <0 0xfea30000 0 0x8000>; @@ -1968,13 +2243,126 @@ renesas,fcp = <&fcpvd2>; }; - fcpvd2: fcp@fea37000 { - compatible = "renesas,fcpv"; - reg = <0 0xfea37000 0 0x200>; - clocks = <&cpg CPG_MOD 601>; + vspi0: vsp@fe9a0000 { + compatible = "renesas,vsp2"; + reg = <0 0xfe9a0000 0 0x8000>; + interrupts = ; + clocks = <&cpg CPG_MOD 631>; + power-domains = <&sysc R8A7796_PD_A3VC>; + resets = <&cpg 631>; + + renesas,fcp = <&fcpvi0>; + }; + + csi20: csi2@fea80000 { + compatible = "renesas,r8a7796-csi2"; + reg = <0 0xfea80000 0 0x10000>; + interrupts = ; + clocks = <&cpg CPG_MOD 714>; power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; - resets = <&cpg 601>; - iommus = <&ipmmu_vi0 10>; + resets = <&cpg 714>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + csi20vin0: endpoint@0 { + reg = <0>; + remote-endpoint = <&vin0csi20>; + }; + csi20vin1: endpoint@1 { + reg = <1>; + remote-endpoint = <&vin1csi20>; + }; + csi20vin2: endpoint@2 { + reg = <2>; + remote-endpoint = <&vin2csi20>; + }; + csi20vin3: endpoint@3 { + reg = <3>; + remote-endpoint = <&vin3csi20>; + }; + csi20vin4: endpoint@4 { + reg = <4>; + remote-endpoint = <&vin4csi20>; + }; + csi20vin5: endpoint@5 { + reg = <5>; + remote-endpoint = <&vin5csi20>; + }; + csi20vin6: endpoint@6 { + reg = <6>; + remote-endpoint = <&vin6csi20>; + }; + csi20vin7: endpoint@7 { + reg = <7>; + remote-endpoint = <&vin7csi20>; + }; + }; + }; + }; + + csi40: csi2@feaa0000 { + compatible = "renesas,r8a7796-csi2"; + reg = <0 0xfeaa0000 0 0x10000>; + interrupts = ; + clocks = <&cpg CPG_MOD 716>; + power-domains = <&sysc R8A7796_PD_ALWAYS_ON>; + resets = <&cpg 716>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + csi40vin0: endpoint@0 { + reg = <0>; + remote-endpoint = <&vin0csi40>; + }; + csi40vin1: endpoint@1 { + reg = <1>; + remote-endpoint = <&vin1csi40>; + }; + csi40vin2: endpoint@2 { + reg = <2>; + remote-endpoint = <&vin2csi40>; + }; + csi40vin3: endpoint@3 { + reg = <3>; + remote-endpoint = <&vin3csi40>; + }; + csi40vin4: endpoint@4 { + reg = <4>; + remote-endpoint = <&vin4csi40>; + }; + csi40vin5: endpoint@5 { + reg = <5>; + remote-endpoint = <&vin5csi40>; + }; + csi40vin6: endpoint@6 { + reg = <6>; + remote-endpoint = <&vin6csi40>; + }; + csi40vin7: endpoint@7 { + reg = <7>; + remote-endpoint = <&vin7csi40>; + }; + }; + + }; }; hdmi0: hdmi@fead0000 { @@ -1999,6 +2387,10 @@ port@1 { reg = <1>; }; + port@2 { + /* HDMI sound */ + reg = <2>; + }; }; }; @@ -2042,35 +2434,12 @@ }; }; - imr-lx4@fe860000 { - compatible = "renesas,r8a7796-imr-lx4", - "renesas,imr-lx4"; - reg = <0 0xfe860000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 823>; - power-domains = <&sysc R8A7796_PD_A3VC>; - resets = <&cpg 823>; - }; - - imr-lx4@fe870000 { - compatible = "renesas,r8a7796-imr-lx4", - "renesas,imr-lx4"; - reg = <0 0xfe870000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 822>; - power-domains = <&sysc R8A7796_PD_A3VC>; - resets = <&cpg 822>; + prr: chipid@fff00044 { + compatible = "renesas,prr"; + reg = <0 0xfff00044 0 4>; }; }; - timer { - compatible = "arm,armv8-timer"; - interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(6) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(6) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(6) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(6) | IRQ_TYPE_LEVEL_LOW)>; - }; - thermal-zones { sensor_thermal1: sensor-thermal1 { polling-delay-passive = <250>; @@ -2080,12 +2449,12 @@ trips { sensor1_passive: sensor1-passive { temperature = <95000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "passive"; }; sensor1_crit: sensor1-crit { temperature = <120000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "critical"; }; }; @@ -2106,12 +2475,12 @@ trips { sensor2_passive: sensor2-passive { temperature = <95000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "passive"; }; sensor2_crit: sensor2-crit { temperature = <120000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "critical"; }; }; @@ -2132,12 +2501,12 @@ trips { sensor3_passive: sensor3-passive { temperature = <95000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "passive"; }; sensor3_crit: sensor3-crit { temperature = <120000>; - hysteresis = <2000>; + hysteresis = <1000>; type = "critical"; }; }; @@ -2151,6 +2520,14 @@ }; }; + timer { + compatible = "arm,armv8-timer"; + interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(6) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(6) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(6) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(6) | IRQ_TYPE_LEVEL_LOW)>; + }; + /* External USB clocks - can be overridden by the board */ usb3s0_clk: usb3s0 { compatible = "fixed-clock"; diff --git a/arch/arm64/boot/dts/renesas/r8a77965-salvator-x.dts b/arch/arm64/boot/dts/renesas/r8a77965-salvator-x.dts index 75d890d91df9..340a3c72b65a 100644 --- a/arch/arm64/boot/dts/renesas/r8a77965-salvator-x.dts +++ b/arch/arm64/boot/dts/renesas/r8a77965-salvator-x.dts @@ -19,3 +19,31 @@ reg = <0x0 0x48000000 0x0 0x78000000>; }; }; + +&du { + clocks = <&cpg CPG_MOD 724>, + <&cpg CPG_MOD 723>, + <&cpg CPG_MOD 721>, + <&versaclock5 1>, + <&x21_clk>, + <&versaclock5 2>; + clock-names = "du.0", "du.1", "du.3", + "dclkin.0", "dclkin.1", "dclkin.3"; +}; + +&hdmi0 { + status = "okay"; + + ports { + port@1 { + reg = <1>; + rcar_dw_hdmi0_out: endpoint { + remote-endpoint = <&hdmi0_con>; + }; + }; + }; +}; + +&hdmi0_con { + remote-endpoint = <&rcar_dw_hdmi0_out>; +}; diff --git a/arch/arm64/boot/dts/renesas/r8a77965-salvator-xs.dts b/arch/arm64/boot/dts/renesas/r8a77965-salvator-xs.dts index a83a00deed9e..9de4e3db1621 100644 --- a/arch/arm64/boot/dts/renesas/r8a77965-salvator-xs.dts +++ b/arch/arm64/boot/dts/renesas/r8a77965-salvator-xs.dts @@ -19,3 +19,31 @@ reg = <0x0 0x48000000 0x0 0x78000000>; }; }; + +&du { + clocks = <&cpg CPG_MOD 724>, + <&cpg CPG_MOD 723>, + <&cpg CPG_MOD 721>, + <&versaclock6 1>, + <&x21_clk>, + <&versaclock6 2>; + clock-names = "du.0", "du.1", "du.3", + "dclkin.0", "dclkin.1", "dclkin.3"; +}; + +&hdmi0 { + status = "okay"; + + ports { + port@1 { + reg = <1>; + rcar_dw_hdmi0_out: endpoint { + remote-endpoint = <&hdmi0_con>; + }; + }; + }; +}; + +&hdmi0_con { + remote-endpoint = <&rcar_dw_hdmi0_out>; +}; diff --git a/arch/arm64/boot/dts/renesas/r8a77965.dtsi b/arch/arm64/boot/dts/renesas/r8a77965.dtsi index f0871fcdd984..486aecacb22a 100644 --- a/arch/arm64/boot/dts/renesas/r8a77965.dtsi +++ b/arch/arm64/boot/dts/renesas/r8a77965.dtsi @@ -8,8 +8,9 @@ * Copyright (C) 2016 Renesas Electronics Corp. */ -#include +#include #include +#include #define CPG_AUDIO_CLK_I 10 @@ -19,12 +20,44 @@ #size-cells = <2>; aliases { + i2c0 = &i2c0; + i2c1 = &i2c1; + i2c2 = &i2c2; + i2c3 = &i2c3; + i2c4 = &i2c4; + i2c5 = &i2c5; + i2c6 = &i2c6; i2c7 = &i2c_dvfs; }; - psci { - compatible = "arm,psci-1.0", "arm,psci-0.2"; - method = "smc"; + /* + * The external audio clocks are configured as 0 Hz fixed frequency + * clocks by default. + * Boards that provide audio clocks should override them. + */ + audio_clk_a: audio_clk_a { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + }; + + audio_clk_b: audio_clk_b { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + }; + + audio_clk_c: audio_clk_c { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + }; + + /* External CAN clock - to be overridden by boards that provide it */ + can_clk: can { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; }; cpus { @@ -35,23 +68,23 @@ compatible = "arm,cortex-a57", "arm,armv8"; reg = <0x0>; device_type = "cpu"; - power-domains = <&sysc 0>; + power-domains = <&sysc R8A77965_PD_CA57_CPU0>; next-level-cache = <&L2_CA57>; enable-method = "psci"; }; a57_1: cpu@1 { - compatible = "arm,cortex-a57","arm,armv8"; + compatible = "arm,cortex-a57", "arm,armv8"; reg = <0x1>; device_type = "cpu"; - power-domains = <&sysc 1>; + power-domains = <&sysc R8A77965_PD_CA57_CPU1>; next-level-cache = <&L2_CA57>; enable-method = "psci"; }; L2_CA57: cache-controller-0 { compatible = "cache"; - power-domains = <&sysc 12>; + power-domains = <&sysc R8A77965_PD_CA57_SCU>; cache-unified; cache-level = <2>; }; @@ -71,34 +104,24 @@ clock-frequency = <0>; }; - /* - * The external audio clocks are configured as 0 Hz fixed frequency - * clocks by default. - * Boards that provide audio clocks should override them. - */ - audio_clk_a: audio_clk_a { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; - }; - - audio_clk_b: audio_clk_b { + /* External PCIe clock - can be overridden by the board */ + pcie_bus_clk: pcie_bus { compatible = "fixed-clock"; #clock-cells = <0>; clock-frequency = <0>; }; - audio_clk_c: audio_clk_c { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; + pmu_a57 { + compatible = "arm,cortex-a57-pmu"; + interrupts-extended = <&gic GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&a57_0>, + <&a57_1>; }; - /* External CAN clock - to be overridden by boards that provide it */ - can_clk: can { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; + psci { + compatible = "arm,psci-1.0", "arm,psci-0.2"; + method = "smc"; }; /* External SCIF clock - to be overridden by boards that provide it */ @@ -108,42 +131,6 @@ clock-frequency = <0>; }; - /* External PCIe clock - can be overridden by the board */ - pcie_bus_clk: pcie_bus { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; - }; - - /* External USB clocks - can be overridden by the board */ - usb3s0_clk: usb3s0 { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; - }; - - usb_extal_clk: usb_extal { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; - }; - - timer { - compatible = "arm,armv8-timer"; - interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>; - }; - - pmu_a57 { - compatible = "arm,cortex-a57-pmu"; - interrupts-extended = <&gic GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>, - <&gic GIC_SPI 73 IRQ_TYPE_LEVEL_HIGH>; - interrupt-affinity = <&a57_0>, - <&a57_1>; - }; - soc { compatible = "simple-bus"; interrupt-parent = <&gic>; @@ -151,52 +138,9 @@ #size-cells = <2>; ranges; - gic: interrupt-controller@f1010000 { - compatible = "arm,gic-400"; - #interrupt-cells = <3>; - #address-cells = <0>; - interrupt-controller; - reg = <0x0 0xf1010000 0 0x1000>, - <0x0 0xf1020000 0 0x20000>, - <0x0 0xf1040000 0 0x20000>, - <0x0 0xf1060000 0 0x20000>; - interrupts = ; - clocks = <&cpg CPG_MOD 408>; - clock-names = "clk"; - power-domains = <&sysc 32>; - resets = <&cpg 408>; - }; - - pfc: pin-controller@e6060000 { - compatible = "renesas,pfc-r8a77965"; - reg = <0 0xe6060000 0 0x50c>; - }; - - cpg: clock-controller@e6150000 { - compatible = "renesas,r8a77965-cpg-mssr"; - reg = <0 0xe6150000 0 0x1000>; - clocks = <&extal_clk>, <&extalr_clk>; - clock-names = "extal", "extalr"; - #clock-cells = <2>; - #power-domain-cells = <0>; - #reset-cells = <1>; - }; - - rst: reset-controller@e6160000 { - compatible = "renesas,r8a77965-rst"; - reg = <0 0xe6160000 0 0x0200>; - }; - - prr: chipid@fff00044 { - compatible = "renesas,prr"; - reg = <0 0xfff00044 0 4>; - }; - - sysc: system-controller@e6180000 { - compatible = "renesas,r8a77965-sysc"; - reg = <0 0xe6180000 0 0x0400>; - #power-domain-cells = <1>; + wdt0: watchdog@e6020000 { + reg = <0 0xe6020000 0 0x0c>; + /* placeholder */ }; gpio0: gpio@e6050000 { @@ -210,7 +154,7 @@ #interrupt-cells = <2>; interrupt-controller; clocks = <&cpg CPG_MOD 912>; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 912>; }; @@ -225,7 +169,7 @@ #interrupt-cells = <2>; interrupt-controller; clocks = <&cpg CPG_MOD 911>; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 911>; }; @@ -240,7 +184,7 @@ #interrupt-cells = <2>; interrupt-controller; clocks = <&cpg CPG_MOD 910>; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 910>; }; @@ -255,7 +199,7 @@ #interrupt-cells = <2>; interrupt-controller; clocks = <&cpg CPG_MOD 909>; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 909>; }; @@ -270,7 +214,7 @@ #interrupt-cells = <2>; interrupt-controller; clocks = <&cpg CPG_MOD 908>; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 908>; }; @@ -285,7 +229,7 @@ #interrupt-cells = <2>; interrupt-controller; clocks = <&cpg CPG_MOD 907>; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 907>; }; @@ -300,7 +244,7 @@ #interrupt-cells = <2>; interrupt-controller; clocks = <&cpg CPG_MOD 906>; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 906>; }; @@ -315,10 +259,51 @@ #interrupt-cells = <2>; interrupt-controller; clocks = <&cpg CPG_MOD 905>; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 905>; }; + pfc: pin-controller@e6060000 { + compatible = "renesas,pfc-r8a77965"; + reg = <0 0xe6060000 0 0x50c>; + }; + + cpg: clock-controller@e6150000 { + compatible = "renesas,r8a77965-cpg-mssr"; + reg = <0 0xe6150000 0 0x1000>; + clocks = <&extal_clk>, <&extalr_clk>; + clock-names = "extal", "extalr"; + #clock-cells = <2>; + #power-domain-cells = <0>; + #reset-cells = <1>; + }; + + rst: reset-controller@e6160000 { + compatible = "renesas,r8a77965-rst"; + reg = <0 0xe6160000 0 0x0200>; + }; + + sysc: system-controller@e6180000 { + compatible = "renesas,r8a77965-sysc"; + reg = <0 0xe6180000 0 0x0400>; + #power-domain-cells = <1>; + }; + + tsc: thermal@e6198000 { + compatible = "renesas,r8a77965-thermal"; + reg = <0 0xe6198000 0 0x100>, + <0 0xe61a0000 0 0x100>, + <0 0xe61a8000 0 0x100>; + interrupts = , + , + ; + clocks = <&cpg CPG_MOD 522>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 522>; + #thermal-sensor-cells = <1>; + status = "okay"; + }; + intc_ex: interrupt-controller@e61c0000 { compatible = "renesas,intc-ex-r8a77965", "renesas,irqc"; #interrupt-cells = <2>; @@ -331,84 +316,273 @@ GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH GIC_SPI 161 IRQ_TYPE_LEVEL_HIGH>; clocks = <&cpg CPG_MOD 407>; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 407>; }; - dmac0: dma-controller@e6700000 { - compatible = "renesas,dmac-r8a77965", - "renesas,rcar-dmac"; - reg = <0 0xe6700000 0 0x10000>; - interrupts = ; - interrupt-names = "error", - "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7", - "ch8", "ch9", "ch10", "ch11", - "ch12", "ch13", "ch14", "ch15"; - clocks = <&cpg CPG_MOD 219>; - clock-names = "fck"; - power-domains = <&sysc 32>; - resets = <&cpg 219>; - #dma-cells = <1>; - dma-channels = <16>; + i2c0: i2c@e6500000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a77965", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe6500000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 931>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 931>; + dmas = <&dmac1 0x91>, <&dmac1 0x90>, + <&dmac2 0x91>, <&dmac2 0x90>; + dma-names = "tx", "rx", "tx", "rx"; + i2c-scl-internal-delay-ns = <110>; + status = "disabled"; }; - dmac1: dma-controller@e7300000 { - compatible = "renesas,dmac-r8a77965", - "renesas,rcar-dmac"; - reg = <0 0xe7300000 0 0x10000>; - interrupts = ; - interrupt-names = "error", - "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7", - "ch8", "ch9", "ch10", "ch11", - "ch12", "ch13", "ch14", "ch15"; - clocks = <&cpg CPG_MOD 218>; - clock-names = "fck"; - power-domains = <&sysc 32>; - resets = <&cpg 218>; - #dma-cells = <1>; - dma-channels = <16>; + i2c1: i2c@e6508000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a77965", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe6508000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 930>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 930>; + dmas = <&dmac1 0x93>, <&dmac1 0x92>, + <&dmac2 0x93>, <&dmac2 0x92>; + dma-names = "tx", "rx", "tx", "rx"; + i2c-scl-internal-delay-ns = <6>; + status = "disabled"; }; - dmac2: dma-controller@e7310000 { - compatible = "renesas,dmac-r8a77965", - "renesas,rcar-dmac"; - reg = <0 0xe7310000 0 0x10000>; - interrupts = ; + #size-cells = <0>; + compatible = "renesas,i2c-r8a77965", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe6510000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 929>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 929>; + dmas = <&dmac1 0x95>, <&dmac1 0x94>, + <&dmac2 0x95>, <&dmac2 0x94>; + dma-names = "tx", "rx", "tx", "rx"; + i2c-scl-internal-delay-ns = <6>; + status = "disabled"; + }; + + i2c3: i2c@e66d0000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a77965", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe66d0000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 928>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 928>; + dmas = <&dmac0 0x97>, <&dmac0 0x96>; + dma-names = "tx", "rx"; + i2c-scl-internal-delay-ns = <110>; + status = "disabled"; + }; + + i2c4: i2c@e66d8000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a77965", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe66d8000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 927>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 927>; + dmas = <&dmac0 0x99>, <&dmac0 0x98>; + dma-names = "tx", "rx"; + i2c-scl-internal-delay-ns = <110>; + status = "disabled"; + }; + + i2c5: i2c@e66e0000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a77965", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe66e0000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 919>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 919>; + dmas = <&dmac0 0x9b>, <&dmac0 0x9a>; + dma-names = "tx", "rx"; + i2c-scl-internal-delay-ns = <110>; + status = "disabled"; + }; + + i2c6: i2c@e66e8000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a77965", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe66e8000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 918>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 918>; + dmas = <&dmac0 0x9d>, <&dmac0 0x9c>; + dma-names = "tx", "rx"; + i2c-scl-internal-delay-ns = <6>; + status = "disabled"; + }; + + i2c_dvfs: i2c@e60b0000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,iic-r8a77965", + "renesas,rcar-gen3-iic", + "renesas,rmobile-iic"; + reg = <0 0xe60b0000 0 0x425>; + interrupts = ; + clocks = <&cpg CPG_MOD 926>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 926>; + dmas = <&dmac0 0x11>, <&dmac0 0x10>; + dma-names = "tx", "rx"; + status = "disabled"; + }; + + hsusb: usb@e6590000 { + compatible = "renesas,usbhs-r8a7796", + "renesas,rcar-gen3-usbhs"; + reg = <0 0xe6590000 0 0x100>; + interrupts = ; + clocks = <&cpg CPG_MOD 704>; + dmas = <&usb_dmac0 0>, <&usb_dmac0 1>, + <&usb_dmac1 0>, <&usb_dmac1 1>; + dma-names = "ch0", "ch1", "ch2", "ch3"; + renesas,buswait = <11>; + phys = <&usb2_phy0>; + phy-names = "usb"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 704>; + status = "disabled"; + }; + + usb_dmac0: dma-controller@e65a0000 { + compatible = "renesas,r8a77965-usb-dmac", + "renesas,usb-dmac"; + reg = <0 0xe65a0000 0 0x100>; + interrupts = ; + interrupt-names = "ch0", "ch1"; + clocks = <&cpg CPG_MOD 330>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 330>; + #dma-cells = <1>; + dma-channels = <2>; + }; + + usb_dmac1: dma-controller@e65b0000 { + compatible = "renesas,r8a77965-usb-dmac", + "renesas,usb-dmac"; + reg = <0 0xe65b0000 0 0x100>; + interrupts = ; + interrupt-names = "ch0", "ch1"; + clocks = <&cpg CPG_MOD 331>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 331>; + #dma-cells = <1>; + dma-channels = <2>; + }; + + usb3_phy0: usb-phy@e65ee000 { + compatible = "renesas,r8a77965-usb3-phy", + "renesas,rcar-gen3-usb3-phy"; + reg = <0 0xe65ee000 0 0x90>; + clocks = <&cpg CPG_MOD 328>, <&usb3s0_clk>, + <&usb_extal_clk>; + clock-names = "usb3-if", "usb3s_clk", "usb_extal"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 328>; + #phy-cells = <0>; + status = "disabled"; + }; + + dmac0: dma-controller@e6700000 { + compatible = "renesas,dmac-r8a77965", + "renesas,rcar-dmac"; + reg = <0 0xe6700000 0 0x10000>; + interrupts = ; + interrupt-names = "error", + "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14", "ch15"; + clocks = <&cpg CPG_MOD 219>; + clock-names = "fck"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 219>; + #dma-cells = <1>; + dma-channels = <16>; + }; + + dmac1: dma-controller@e7300000 { + compatible = "renesas,dmac-r8a77965", + "renesas,rcar-dmac"; + reg = <0 0xe7300000 0 0x10000>; + interrupts = ; + interrupt-names = "error", + "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14", "ch15"; + clocks = <&cpg CPG_MOD 218>; + clock-names = "fck"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 218>; + #dma-cells = <1>; + dma-channels = <16>; + }; + + dmac2: dma-controller@e7310000 { + compatible = "renesas,dmac-r8a77965", + "renesas,rcar-dmac"; + reg = <0 0xe7310000 0 0x10000>; + interrupts = ; clock-names = "fck"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 217>; #dma-cells = <1>; dma-channels = <16>; }; + avb: ethernet@e6800000 { + compatible = "renesas,etheravb-r8a77965", + "renesas,etheravb-rcar-gen3"; + reg = <0 0xe6800000 0 0x800>, <0 0xe6a00000 0 0x10000>; + interrupts = , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + ; + interrupt-names = "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14", "ch15", + "ch16", "ch17", "ch18", "ch19", + "ch20", "ch21", "ch22", "ch23", + "ch24"; + clocks = <&cpg CPG_MOD 812>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 812>; + phy-mode = "rgmii"; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + pwm0: pwm@e6e30000 { + compatible = "renesas,pwm-r8a77965", "renesas,pwm-rcar"; + reg = <0 0xe6e30000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm1: pwm@e6e31000 { + compatible = "renesas,pwm-r8a77965", "renesas,pwm-rcar"; + reg = <0 0xe6e31000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm2: pwm@e6e32000 { + compatible = "renesas,pwm-r8a77965", "renesas,pwm-rcar"; + reg = <0 0xe6e32000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm3: pwm@e6e33000 { + compatible = "renesas,pwm-r8a77965", "renesas,pwm-rcar"; + reg = <0 0xe6e33000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm4: pwm@e6e34000 { + compatible = "renesas,pwm-r8a77965", "renesas,pwm-rcar"; + reg = <0 0xe6e34000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm5: pwm@e6e35000 { + compatible = "renesas,pwm-r8a77965", "renesas,pwm-rcar"; + reg = <0 0xe6e35000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + status = "disabled"; + }; + + pwm6: pwm@e6e36000 { + compatible = "renesas,pwm-r8a77965", "renesas,pwm-rcar"; + reg = <0 0xe6e36000 0 8>; + #pwm-cells = <2>; + clocks = <&cpg CPG_MOD 523>; + resets = <&cpg 523>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + status = "disabled"; + }; + scif0: serial@e6e60000 { compatible = "renesas,scif-r8a77965", "renesas,rcar-gen3-scif", "renesas,scif"; @@ -449,7 +738,7 @@ dmas = <&dmac1 0x51>, <&dmac1 0x50>, <&dmac2 0x51>, <&dmac2 0x50>; dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 207>; status = "disabled"; }; @@ -466,7 +755,7 @@ dmas = <&dmac1 0x53>, <&dmac1 0x52>, <&dmac2 0x53>, <&dmac2 0x52>; dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 206>; status = "disabled"; }; @@ -480,7 +769,7 @@ <&cpg CPG_CORE 20>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 310>; status = "disabled"; }; @@ -496,7 +785,7 @@ clock-names = "fck", "brg_int", "scif_clk"; dmas = <&dmac0 0x57>, <&dmac0 0x56>; dma-names = "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 204>; status = "disabled"; }; @@ -512,7 +801,7 @@ clock-names = "fck", "brg_int", "scif_clk"; dmas = <&dmac0 0x59>, <&dmac0 0x58>; dma-names = "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 203>; status = "disabled"; }; @@ -529,243 +818,772 @@ dmas = <&dmac1 0x5b>, <&dmac1 0x5a>, <&dmac2 0x5b>, <&dmac2 0x5a>; dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; resets = <&cpg 202>; status = "disabled"; }; - avb: ethernet@e6800000 { - compatible = "renesas,etheravb-r8a77965", - "renesas,etheravb-rcar-gen3"; - reg = <0 0xe6800000 0 0x800>, <0 0xe6a00000 0 0x10000>; - interrupts = , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - ; - interrupt-names = "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7", - "ch8", "ch9", "ch10", "ch11", - "ch12", "ch13", "ch14", "ch15", - "ch16", "ch17", "ch18", "ch19", - "ch20", "ch21", "ch22", "ch23", - "ch24"; - clocks = <&cpg CPG_MOD 812>; - power-domains = <&sysc 32>; - resets = <&cpg 812>; - phy-mode = "rgmii"; + msiof0: spi@e6e90000 { + compatible = "renesas,msiof-r8a77965", + "renesas,rcar-gen3-msiof"; + reg = <0 0xe6e90000 0 0x0064>; + interrupts = ; + clocks = <&cpg CPG_MOD 211>; + dmas = <&dmac1 0x41>, <&dmac1 0x40>, + <&dmac2 0x41>, <&dmac2 0x40>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 211>; #address-cells = <1>; #size-cells = <0>; status = "disabled"; }; - csi20: csi2@fea80000 { - reg = <0 0xfea80000 0 0x10000>; - /* placeholder */ + msiof1: spi@e6ea0000 { + compatible = "renesas,msiof-r8a77965", + "renesas,rcar-gen3-msiof"; + reg = <0 0xe6ea0000 0 0x0064>; + interrupts = ; + clocks = <&cpg CPG_MOD 210>; + dmas = <&dmac1 0x43>, <&dmac1 0x42>, + <&dmac2 0x43>, <&dmac2 0x42>; + dma-names = "tx", "rx", "tx", "rx"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 210>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + msiof2: spi@e6c00000 { + compatible = "renesas,msiof-r8a77965", + "renesas,rcar-gen3-msiof"; + reg = <0 0xe6c00000 0 0x0064>; + interrupts = ; + clocks = <&cpg CPG_MOD 209>; + dmas = <&dmac0 0x45>, <&dmac0 0x44>; + dma-names = "tx", "rx"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 209>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + msiof3: spi@e6c10000 { + compatible = "renesas,msiof-r8a77965", + "renesas,rcar-gen3-msiof"; + reg = <0 0xe6c10000 0 0x0064>; + interrupts = ; + clocks = <&cpg CPG_MOD 208>; + dmas = <&dmac0 0x47>, <&dmac0 0x46>; + dma-names = "tx", "rx"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 208>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + vin0: video@e6ef0000 { + compatible = "renesas,vin-r8a77965"; + reg = <0 0xe6ef0000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 811>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 811>; + renesas,id = <0>; + status = "disabled"; ports { #address-cells = <1>; #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin0csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin0>; + }; + vin0csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin0>; + }; + }; }; }; - csi40: csi2@feaa0000 { - reg = <0 0xfeaa0000 0 0x10000>; - /* placeholder */ + vin1: video@e6ef1000 { + compatible = "renesas,vin-r8a77965"; + reg = <0 0xe6ef1000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 810>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 810>; + renesas,id = <1>; + status = "disabled"; ports { #address-cells = <1>; #size-cells = <0>; - }; - }; - vin0: video@e6ef0000 { - reg = <0 0xe6ef0000 0 0x1000>; - /* placeholder */ - }; + port@1 { + #address-cells = <1>; + #size-cells = <0>; - vin1: video@e6ef1000 { - reg = <0 0xe6ef1000 0 0x1000>; - /* placeholder */ + reg = <1>; + + vin1csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin1>; + }; + vin1csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin1>; + }; + }; + }; }; vin2: video@e6ef2000 { + compatible = "renesas,vin-r8a77965"; reg = <0 0xe6ef2000 0 0x1000>; - /* placeholder */ + interrupts = ; + clocks = <&cpg CPG_MOD 809>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 809>; + renesas,id = <2>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin2csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin2>; + }; + vin2csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin2>; + }; + }; + }; }; vin3: video@e6ef3000 { + compatible = "renesas,vin-r8a77965"; reg = <0 0xe6ef3000 0 0x1000>; - /* placeholder */ + interrupts = ; + clocks = <&cpg CPG_MOD 808>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 808>; + renesas,id = <3>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin3csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin3>; + }; + vin3csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin3>; + }; + }; + }; }; vin4: video@e6ef4000 { + compatible = "renesas,vin-r8a77965"; reg = <0 0xe6ef4000 0 0x1000>; - /* placeholder */ + interrupts = ; + clocks = <&cpg CPG_MOD 807>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 807>; + renesas,id = <4>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin4csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin4>; + }; + vin4csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin4>; + }; + }; + }; }; vin5: video@e6ef5000 { + compatible = "renesas,vin-r8a77965"; reg = <0 0xe6ef5000 0 0x1000>; - /* placeholder */ + interrupts = ; + clocks = <&cpg CPG_MOD 806>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 806>; + renesas,id = <5>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin5csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin5>; + }; + vin5csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin5>; + }; + }; + }; }; vin6: video@e6ef6000 { + compatible = "renesas,vin-r8a77965"; reg = <0 0xe6ef6000 0 0x1000>; - /* placeholder */ + interrupts = ; + clocks = <&cpg CPG_MOD 805>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 805>; + renesas,id = <6>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin6csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin6>; + }; + vin6csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin6>; + }; + }; + }; }; vin7: video@e6ef7000 { + compatible = "renesas,vin-r8a77965"; reg = <0 0xe6ef7000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 804>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 804>; + renesas,id = <7>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin7csi20: endpoint@0 { + reg = <0>; + remote-endpoint= <&csi20vin7>; + }; + vin7csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin7>; + }; + }; + }; + }; + + rcar_sound: sound@ec500000 { + reg = <0 0xec500000 0 0x1000>, /* SCU */ + <0 0xec5a0000 0 0x100>, /* ADG */ + <0 0xec540000 0 0x1000>, /* SSIU */ + <0 0xec541000 0 0x280>, /* SSI */ + <0 0xec740000 0 0x200>; /* Audio DMAC peri peri*/ /* placeholder */ + + rcar_sound,dvc { + dvc0: dvc-0 { + }; + dvc1: dvc-1 { + }; + }; + + rcar_sound,src { + src0: src-0 { + }; + src1: src-1 { + }; + }; + + rcar_sound,ssi { + ssi0: ssi-0 { + }; + ssi1: ssi-1 { + }; + }; + + ports { + #address-cells = <1>; + #size-cells = <0>; + port@0 { + reg = <0>; + }; + }; + }; + + xhci0: usb@ee000000 { + compatible = "renesas,xhci-r8a77965", + "renesas,rcar-gen3-xhci"; + reg = <0 0xee000000 0 0xc00>; + interrupts = ; + clocks = <&cpg CPG_MOD 328>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 328>; + status = "disabled"; + }; + + usb3_peri0: usb@ee020000 { + compatible = "renesas,r8a77965-usb3-peri", + "renesas,rcar-gen3-usb3-peri"; + reg = <0 0xee020000 0 0x400>; + interrupts = ; + clocks = <&cpg CPG_MOD 328>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 328>; + status = "disabled"; }; ohci0: usb@ee080000 { + compatible = "generic-ohci"; reg = <0 0xee080000 0 0x100>; - /* placeholder */ + interrupts = ; + clocks = <&cpg CPG_MOD 703>; + phys = <&usb2_phy0>; + phy-names = "usb"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 703>; + status = "disabled"; + }; + + ohci1: usb@ee0a0000 { + compatible = "generic-ohci"; + reg = <0 0xee0a0000 0 0x100>; + interrupts = ; + clocks = <&cpg CPG_MOD 702>; + phys = <&usb2_phy1>; + phy-names = "usb"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 702>; + status = "disabled"; }; ehci0: usb@ee080100 { + compatible = "generic-ehci"; reg = <0 0xee080100 0 0x100>; - /* placeholder */ + interrupts = ; + clocks = <&cpg CPG_MOD 703>; + phys = <&usb2_phy0>; + phy-names = "usb"; + companion = <&ohci0>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 703>; + status = "disabled"; + }; + + ehci1: usb@ee0a0100 { + compatible = "generic-ehci"; + reg = <0 0xee0a0100 0 0x100>; + interrupts = ; + clocks = <&cpg CPG_MOD 702>; + phys = <&usb2_phy1>; + phy-names = "usb"; + companion = <&ohci1>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 702>; + status = "disabled"; }; usb2_phy0: usb-phy@ee080200 { + compatible = "renesas,usb2-phy-r8a77965", + "renesas,rcar-gen3-usb2-phy"; reg = <0 0xee080200 0 0x700>; - /* placeholder */ + interrupts = ; + clocks = <&cpg CPG_MOD 703>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 703>; + #phy-cells = <0>; + status = "disabled"; }; usb2_phy1: usb-phy@ee0a0200 { + compatible = "renesas,usb2-phy-r8a77965", + "renesas,rcar-gen3-usb2-phy"; reg = <0 0xee0a0200 0 0x700>; - /* placeholder */ + clocks = <&cpg CPG_MOD 703>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 703>; + #phy-cells = <0>; + status = "disabled"; }; - ohci1: usb@ee0a0000 { - reg = <0 0xee0a0000 0 0x100>; - /* placeholder */ + sdhi0: sd@ee100000 { + compatible = "renesas,sdhi-r8a77965", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee100000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 314>; + max-frequency = <200000000>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 314>; + status = "disabled"; }; - ehci1: usb@ee0a0100 { - reg = <0 0xee0a0100 0 0x100>; - /* placeholder */ + sdhi1: sd@ee120000 { + compatible = "renesas,sdhi-r8a77965", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee120000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 313>; + max-frequency = <200000000>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 313>; + status = "disabled"; }; - i2c0: i2c@e6500000 { - reg = <0 0xe6500000 0 0x40>; - /* placeholder */ + sdhi2: sd@ee140000 { + compatible = "renesas,sdhi-r8a77965", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee140000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 312>; + max-frequency = <200000000>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 312>; + status = "disabled"; }; - i2c1: i2c@e6508000 { - reg = <0 0xe6508000 0 0x40>; - /* placeholder */ + sdhi3: sd@ee160000 { + compatible = "renesas,sdhi-r8a77965", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee160000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 311>; + max-frequency = <200000000>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 311>; + status = "disabled"; }; - i2c2: i2c@e6510000 { - #address-cells = <1>; - #size-cells = <0>; + gic: interrupt-controller@f1010000 { + compatible = "arm,gic-400"; + #interrupt-cells = <3>; + #address-cells = <0>; + interrupt-controller; + reg = <0x0 0xf1010000 0 0x1000>, + <0x0 0xf1020000 0 0x20000>, + <0x0 0xf1040000 0 0x20000>, + <0x0 0xf1060000 0 0x20000>; + interrupts = ; + clocks = <&cpg CPG_MOD 408>; + clock-names = "clk"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 408>; + }; - reg = <0 0xe6510000 0 0x40>; + pciec0: pcie@fe000000 { + reg = <0 0xfe000000 0 0x80000>; /* placeholder */ }; - i2c3: i2c@e66d0000 { - reg = <0 0xe66d0000 0 0x40>; + pciec1: pcie@ee800000 { + reg = <0 0xee800000 0 0x80000>; /* placeholder */ }; - i2c4: i2c@e66d8000 { - #address-cells = <1>; - #size-cells = <0>; + fcpf0: fcp@fe950000 { + compatible = "renesas,fcpf"; + reg = <0 0xfe950000 0 0x200>; + clocks = <&cpg CPG_MOD 615>; + power-domains = <&sysc R8A77965_PD_A3VP>; + resets = <&cpg 615>; + }; - reg = <0 0xe66d8000 0 0x40>; - /* placeholder */ + vspb: vsp@fe960000 { + compatible = "renesas,vsp2"; + reg = <0 0xfe960000 0 0x8000>; + interrupts = ; + clocks = <&cpg CPG_MOD 626>; + power-domains = <&sysc R8A77965_PD_A3VP>; + resets = <&cpg 626>; + + renesas,fcp = <&fcpvb0>; }; - i2c5: i2c@e66e0000 { - reg = <0 0xe66e0000 0 0x40>; - /* placeholder */ + fcpvb0: fcp@fe96f000 { + compatible = "renesas,fcpv"; + reg = <0 0xfe96f000 0 0x200>; + clocks = <&cpg CPG_MOD 607>; + power-domains = <&sysc R8A77965_PD_A3VP>; + resets = <&cpg 607>; }; - i2c6: i2c@e66e8000 { - reg = <0 0xe66e8000 0 0x40>; - /* placeholder */ + vspi0: vsp@fe9a0000 { + compatible = "renesas,vsp2"; + reg = <0 0xfe9a0000 0 0x8000>; + interrupts = ; + clocks = <&cpg CPG_MOD 631>; + power-domains = <&sysc R8A77965_PD_A3VP>; + resets = <&cpg 631>; + + renesas,fcp = <&fcpvi0>; }; - i2c_dvfs: i2c@e60b0000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,iic-r8a77965", - "renesas,rcar-gen3-iic", - "renesas,rmobile-iic"; - reg = <0 0xe60b0000 0 0x425>; - interrupts = ; - clocks = <&cpg CPG_MOD 926>; - power-domains = <&sysc 32>; - resets = <&cpg 926>; - dmas = <&dmac0 0x11>, <&dmac0 0x10>; - dma-names = "tx", "rx"; - status = "disabled"; + fcpvi0: fcp@fe9af000 { + compatible = "renesas,fcpv"; + reg = <0 0xfe9af000 0 0x200>; + clocks = <&cpg CPG_MOD 611>; + power-domains = <&sysc R8A77965_PD_A3VP>; + resets = <&cpg 611>; }; - pwm0: pwm@e6e30000 { - reg = <0 0xe6e30000 0 8>; - /* placeholder */ + vspd0: vsp@fea20000 { + compatible = "renesas,vsp2"; + reg = <0 0xfea20000 0 0x8000>; + interrupts = ; + clocks = <&cpg CPG_MOD 623>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 623>; + + renesas,fcp = <&fcpvd0>; }; - pwm1: pwm@e6e31000 { - reg = <0 0xe6e31000 0 8>; - #pwm-cells = <2>; - /* placeholder */ + fcpvd0: fcp@fea27000 { + compatible = "renesas,fcpv"; + reg = <0 0xfea27000 0 0x200>; + clocks = <&cpg CPG_MOD 603>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 603>; }; - pwm2: pwm@e6e32000 { - reg = <0 0xe6e32000 0 8>; - /* placeholder */ + vspd1: vsp@fea28000 { + compatible = "renesas,vsp2"; + reg = <0 0xfea28000 0 0x8000>; + interrupts = ; + clocks = <&cpg CPG_MOD 622>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 622>; + + renesas,fcp = <&fcpvd1>; }; - pwm3: pwm@e6e33000 { - reg = <0 0xe6e33000 0 8>; - /* placeholder */ + fcpvd1: fcp@fea2f000 { + compatible = "renesas,fcpv"; + reg = <0 0xfea2f000 0 0x200>; + clocks = <&cpg CPG_MOD 602>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 602>; }; - pwm4: pwm@e6e34000 { - reg = <0 0xe6e34000 0 8>; - /* placeholder */ + csi20: csi2@fea80000 { + compatible = "renesas,r8a77965-csi2"; + reg = <0 0xfea80000 0 0x10000>; + interrupts = ; + clocks = <&cpg CPG_MOD 714>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 714>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + csi20vin0: endpoint@0 { + reg = <0>; + remote-endpoint = <&vin0csi20>; + }; + csi20vin1: endpoint@1 { + reg = <1>; + remote-endpoint = <&vin1csi20>; + }; + csi20vin2: endpoint@2 { + reg = <2>; + remote-endpoint = <&vin2csi20>; + }; + csi20vin3: endpoint@3 { + reg = <3>; + remote-endpoint = <&vin3csi20>; + }; + csi20vin4: endpoint@4 { + reg = <4>; + remote-endpoint = <&vin4csi20>; + }; + csi20vin5: endpoint@5 { + reg = <5>; + remote-endpoint = <&vin5csi20>; + }; + csi20vin6: endpoint@6 { + reg = <6>; + remote-endpoint = <&vin6csi20>; + }; + csi20vin7: endpoint@7 { + reg = <7>; + remote-endpoint = <&vin7csi20>; + }; + }; + }; }; - pwm5: pwm@e6e35000 { - reg = <0 0xe6e35000 0 8>; - /* placeholder */ + csi40: csi2@feaa0000 { + compatible = "renesas,r8a77965-csi2"; + reg = <0 0xfeaa0000 0 0x10000>; + interrupts = ; + clocks = <&cpg CPG_MOD 716>; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 716>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + csi40vin0: endpoint@0 { + reg = <0>; + remote-endpoint = <&vin0csi40>; + }; + csi40vin1: endpoint@1 { + reg = <1>; + remote-endpoint = <&vin1csi40>; + }; + csi40vin2: endpoint@2 { + reg = <2>; + remote-endpoint = <&vin2csi40>; + }; + csi40vin3: endpoint@3 { + reg = <3>; + remote-endpoint = <&vin3csi40>; + }; + csi40vin4: endpoint@4 { + reg = <4>; + remote-endpoint = <&vin4csi40>; + }; + csi40vin5: endpoint@5 { + reg = <5>; + remote-endpoint = <&vin5csi40>; + }; + csi40vin6: endpoint@6 { + reg = <6>; + remote-endpoint = <&vin6csi40>; + }; + csi40vin7: endpoint@7 { + reg = <7>; + remote-endpoint = <&vin7csi40>; + }; + }; + }; }; - pwm6: pwm@e6e36000 { - reg = <0 0xe6e36000 0 8>; - /* placeholder */ + hdmi0: hdmi@fead0000 { + compatible = "renesas,r8a77965-hdmi", + "renesas,rcar-gen3-hdmi"; + reg = <0 0xfead0000 0 0x10000>; + interrupts = ; + clocks = <&cpg CPG_MOD 729>, + <&cpg CPG_CORE R8A77965_CLK_HDMI>; + clock-names = "iahb", "isfr"; + power-domains = <&sysc R8A77965_PD_ALWAYS_ON>; + resets = <&cpg 729>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + port@0 { + reg = <0>; + dw_hdmi0_in: endpoint { + remote-endpoint = <&du_out_hdmi0>; + }; + }; + port@1 { + reg = <1>; + }; + }; }; du: display@feb00000 { - reg = <0 0xfeb00000 0 0x80000>, - <0 0xfeb90000 0 0x14>; - /* placeholder */ + compatible = "renesas,du-r8a77965"; + reg = <0 0xfeb00000 0 0x80000>; + reg-names = "du"; + interrupts = , + , + ; + clocks = <&cpg CPG_MOD 724>, + <&cpg CPG_MOD 723>, + <&cpg CPG_MOD 721>; + clock-names = "du.0", "du.1", "du.3"; + status = "disabled"; + + vsps = <&vspd0 0 &vspd1 0 &vspd0 1>; ports { #address-cells = <1>; @@ -779,6 +1597,7 @@ port@1 { reg = <1>; du_out_hdmi0: endpoint { + remote-endpoint = <&dw_hdmi0_in>; }; }; port@2 { @@ -789,90 +1608,74 @@ }; }; - hsusb: usb@e6590000 { - reg = <0 0xe6590000 0 0x100>; - /* placeholder */ - }; - - pciec0: pcie@fe000000 { - reg = <0 0xfe000000 0 0x80000>; - /* placeholder */ - }; - - pciec1: pcie@ee800000 { - reg = <0 0xee800000 0 0x80000>; - /* placeholder */ + prr: chipid@fff00044 { + compatible = "renesas,prr"; + reg = <0 0xfff00044 0 4>; }; + }; - rcar_sound: sound@ec500000 { - reg = <0 0xec500000 0 0x1000>, /* SCU */ - <0 0xec5a0000 0 0x100>, /* ADG */ - <0 0xec540000 0 0x1000>, /* SSIU */ - <0 0xec541000 0 0x280>, /* SSI */ - <0 0xec740000 0 0x200>; /* Audio DMAC peri peri*/ - /* placeholder */ + timer { + compatible = "arm,armv8-timer"; + interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>; + }; - rcar_sound,dvc { - dvc0: dvc-0 { - }; - dvc1: dvc-1 { + thermal-zones { + sensor_thermal1: sensor-thermal1 { + polling-delay-passive = <250>; + polling-delay = <1000>; + thermal-sensors = <&tsc 0>; + + trips { + sensor1_crit: sensor1-crit { + temperature = <120000>; + hysteresis = <1000>; + type = "critical"; }; }; + }; - rcar_sound,src { - src0: src-0 { - }; - src1: src-1 { - }; - }; + sensor_thermal2: sensor-thermal2 { + polling-delay-passive = <250>; + polling-delay = <1000>; + thermal-sensors = <&tsc 1>; - rcar_sound,ssi { - ssi0: ssi-0 { - }; - ssi1: ssi-1 { + trips { + sensor2_crit: sensor2-crit { + temperature = <120000>; + hysteresis = <1000>; + type = "critical"; }; }; }; - sdhi0: sd@ee100000 { - reg = <0 0xee100000 0 0x2000>; - /* placeholder */ - }; - - sdhi1: sd@ee120000 { - reg = <0 0xee120000 0 0x2000>; - /* placeholder */ - }; + sensor_thermal3: sensor-thermal3 { + polling-delay-passive = <250>; + polling-delay = <1000>; + thermal-sensors = <&tsc 2>; - sdhi2: sd@ee140000 { - reg = <0 0xee140000 0 0x2000>; - /* placeholder */ - }; - - sdhi3: sd@ee160000 { - reg = <0 0xee160000 0 0x2000>; - /* placeholder */ - }; - - usb3_phy0: usb-phy@e65ee000 { - reg = <0 0xe65ee000 0 0x90>; - #phy-cells = <0>; - /* placeholder */ - }; - - usb3_peri0: usb@ee020000 { - reg = <0 0xee020000 0 0x400>; - /* placeholder */ + trips { + sensor3_crit: sensor3-crit { + temperature = <120000>; + hysteresis = <1000>; + type = "critical"; + }; + }; }; + }; - xhci0: usb@ee000000 { - reg = <0 0xee000000 0 0xc00>; - /* placeholder */ - }; + /* External USB clocks - can be overridden by the board */ + usb3s0_clk: usb3s0 { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + }; - wdt0: watchdog@e6020000 { - reg = <0 0xe6020000 0 0x0c>; - /* placeholder */ - }; + usb_extal_clk: usb_extal { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; }; }; diff --git a/arch/arm64/boot/dts/renesas/r8a77970-eagle.dts b/arch/arm64/boot/dts/renesas/r8a77970-eagle.dts index 3c5f598c9766..21f9cf5c6e84 100644 --- a/arch/arm64/boot/dts/renesas/r8a77970-eagle.dts +++ b/arch/arm64/boot/dts/renesas/r8a77970-eagle.dts @@ -31,9 +31,57 @@ /* first 128MB is reserved for secure area. */ reg = <0x0 0x48000000 0x0 0x38000000>; }; + + hdmi-out { + compatible = "hdmi-connector"; + type = "a"; + + port { + hdmi_con_out: endpoint { + remote-endpoint = <&adv7511_out>; + }; + }; + }; + + d3p3: regulator-fixed { + compatible = "regulator-fixed"; + regulator-name = "fixed-3.3V"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-boot-on; + regulator-always-on; + }; + + lvds-decoder { + compatible = "thine,thc63lvd1024"; + + vcc-supply = <&d3p3>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + thc63lvd1024_in: endpoint { + remote-endpoint = <&lvds0_out>; + }; + }; + + port@2 { + reg = <2>; + thc63lvd1024_out: endpoint { + remote-endpoint = <&adv7511_in>; + }; + }; + }; + }; }; &avb { + pinctrl-0 = <&avb_pins>; + pinctrl-names = "default"; + renesas,no-ether-link; phy-handle = <&phy0>; phy-mode = "rgmii-id"; @@ -47,6 +95,16 @@ }; }; +&canfd { + pinctrl-0 = <&canfd0_pins>; + pinctrl-names = "default"; + status = "okay"; + + channel0 { + status = "okay"; + }; +}; + &extal_clk { clock-frequency = <16666666>; }; @@ -68,9 +126,51 @@ gpio-controller; #gpio-cells = <2>; }; + + hdmi@39 { + compatible = "adi,adv7511w"; + reg = <0x39>; + interrupt-parent = <&gpio1>; + interrupts = <20 IRQ_TYPE_LEVEL_LOW>; + + adi,input-depth = <8>; + adi,input-colorspace = "rgb"; + adi,input-clock = "1x"; + adi,input-style = <1>; + adi,input-justification = "evenly"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + adv7511_in: endpoint { + remote-endpoint = <&thc63lvd1024_out>; + }; + }; + + port@1 { + reg = <1>; + adv7511_out: endpoint { + remote-endpoint = <&hdmi_con_out>; + }; + }; + }; + }; }; &pfc { + avb_pins: avb0 { + groups = "avb0_mdio", "avb0_rgmii", "avb0_txcrefclk"; + function = "avb0"; + }; + + canfd0_pins: canfd0 { + groups = "canfd0_data_a"; + function = "canfd0"; + }; + i2c0_pins: i2c0 { groups = "i2c0"; function = "i2c0"; @@ -93,3 +193,19 @@ status = "okay"; }; + +&du { + status = "okay"; +}; + +&lvds0 { + status = "okay"; + + ports { + port@1 { + lvds0_out: endpoint { + remote-endpoint = <&thc63lvd1024_in>; + }; + }; + }; +}; diff --git a/arch/arm64/boot/dts/renesas/r8a77970-v3msk.dts b/arch/arm64/boot/dts/renesas/r8a77970-v3msk.dts index a8ceeac77992..9fce031a596f 100644 --- a/arch/arm64/boot/dts/renesas/r8a77970-v3msk.dts +++ b/arch/arm64/boot/dts/renesas/r8a77970-v3msk.dts @@ -29,9 +29,71 @@ /* first 128MB is reserved for secure area. */ reg = <0x0 0x48000000 0x0 0x38000000>; }; + + osc5_clk: osc5-clock { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <148500000>; + }; + + vcc_d1_8v: regulator-0 { + compatible = "regulator-fixed"; + regulator-name = "VCC_D1.8V"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-boot-on; + regulator-always-on; + }; + + vcc_d3_3v: regulator-1 { + compatible = "regulator-fixed"; + regulator-name = "VCC_D3.3V"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-boot-on; + regulator-always-on; + }; + + lvds-decoder { + compatible = "thine,thc63lvd1024"; + vcc-supply = <&vcc_d3_3v>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + thc63lvd1024_in: endpoint { + remote-endpoint = <&lvds0_out>; + }; + }; + + port@2 { + reg = <2>; + thc63lvd1024_out: endpoint { + remote-endpoint = <&adv7511_in>; + }; + }; + }; + }; + + hdmi-out { + compatible = "hdmi-connector"; + type = "a"; + + port { + hdmi_con: endpoint { + remote-endpoint = <&adv7511_out>; + }; + }; + }; }; &avb { + pinctrl-0 = <&avb_pins>; + pinctrl-names = "default"; + renesas,no-ether-link; phy-handle = <&phy0>; phy-mode = "rgmii-id"; @@ -43,6 +105,13 @@ }; }; +&du { + clocks = <&cpg CPG_MOD 724>, + <&osc5_clk>; + clock-names = "du.0", "dclkin.0"; + status = "okay"; +}; + &extal_clk { clock-frequency = <16666666>; }; @@ -52,12 +121,80 @@ }; &pfc { + avb_pins: avb0 { + groups = "avb0_mdio", "avb0_rgmii", "avb0_txcrefclk"; + function = "avb0"; + }; + + i2c0_pins: i2c0 { + groups = "i2c0"; + function = "i2c0"; + }; + scif0_pins: scif0 { groups = "scif0_data"; function = "scif0"; }; }; +&i2c0 { + pinctrl-0 = <&i2c0_pins>; + pinctrl-names = "default"; + + status = "okay"; + clock-frequency = <400000>; + + hdmi@39{ + compatible = "adi,adv7511w"; + #sound-dai-cells = <0>; + reg = <0x39>; + interrupt-parent = <&gpio1>; + interrupts = <20 IRQ_TYPE_LEVEL_LOW>; + avdd-supply = <&vcc_d1_8v>; + dvdd-supply = <&vcc_d1_8v>; + pvdd-supply = <&vcc_d1_8v>; + bgvdd-supply = <&vcc_d1_8v>; + dvdd-3v-supply = <&vcc_d3_3v>; + + adi,input-depth = <8>; + adi,input-colorspace = "rgb"; + adi,input-clock = "1x"; + adi,input-style = <1>; + adi,input-justification = "evenly"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + adv7511_in: endpoint { + remote-endpoint = <&thc63lvd1024_out>; + }; + }; + + port@1 { + reg = <1>; + adv7511_out: endpoint { + remote-endpoint = <&hdmi_con>; + }; + }; + }; + }; +}; + +&lvds0 { + status = "okay"; + + ports { + port@1 { + lvds0_out: endpoint { + remote-endpoint = <&thc63lvd1024_in>; + }; + }; + }; +}; + &scif0 { pinctrl-0 = <&scif0_pins>; pinctrl-names = "default"; diff --git a/arch/arm64/boot/dts/renesas/r8a77970.dtsi b/arch/arm64/boot/dts/renesas/r8a77970.dtsi index c6db8ea43906..98a2317a16c4 100644 --- a/arch/arm64/boot/dts/renesas/r8a77970.dtsi +++ b/arch/arm64/boot/dts/renesas/r8a77970.dtsi @@ -41,6 +41,16 @@ enable-method = "psci"; }; + a53_1: cpu@1 { + device_type = "cpu"; + compatible = "arm,cortex-a53", "arm,armv8"; + reg = <1>; + clocks = <&cpg CPG_CORE R8A77970_CLK_Z2>; + power-domains = <&sysc R8A77970_PD_CA53_CPU1>; + next-level-cache = <&L2_CA53>; + enable-method = "psci"; + }; + L2_CA53: cache-controller { compatible = "cache"; power-domains = <&sysc R8A77970_PD_CA53_SCU>; @@ -63,11 +73,25 @@ clock-frequency = <0>; }; + pmu_a53 { + compatible = "arm,cortex-a53-pmu"; + interrupts-extended = <&gic GIC_SPI 84 IRQ_TYPE_LEVEL_HIGH>, + <&gic GIC_SPI 85 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&a53_0>, <&a53_1>; + }; + psci { compatible = "arm,psci-1.0", "arm,psci-0.2"; method = "smc"; }; + /* External CAN clock - to be overridden by boards that provide it */ + can_clk: can { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + }; + /* External SCIF clock - to be overridden by boards that provide it */ scif_clk: scif { compatible = "fixed-clock"; @@ -83,23 +107,6 @@ #size-cells = <2>; ranges; - gic: interrupt-controller@f1010000 { - compatible = "arm,gic-400"; - #interrupt-cells = <3>; - #address-cells = <0>; - interrupt-controller; - reg = <0 0xf1010000 0 0x1000>, - <0 0xf1020000 0 0x20000>, - <0 0xf1040000 0 0x20000>, - <0 0xf1060000 0 0x20000>; - interrupts = ; - clocks = <&cpg CPG_MOD 408>; - clock-names = "clk"; - power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; - resets = <&cpg 408>; - }; - rwdt: watchdog@e6020000 { compatible = "renesas,r8a77970-wdt", "renesas,rcar-gen3-wdt"; @@ -110,75 +117,6 @@ status = "disabled"; }; - cpg: clock-controller@e6150000 { - compatible = "renesas,r8a77970-cpg-mssr"; - reg = <0 0xe6150000 0 0x1000>; - clocks = <&extal_clk>, <&extalr_clk>; - clock-names = "extal", "extalr"; - #clock-cells = <2>; - #power-domain-cells = <0>; - #reset-cells = <1>; - }; - - rst: reset-controller@e6160000 { - compatible = "renesas,r8a77970-rst"; - reg = <0 0xe6160000 0 0x200>; - }; - - sysc: system-controller@e6180000 { - compatible = "renesas,r8a77970-sysc"; - reg = <0 0xe6180000 0 0x440>; - #power-domain-cells = <1>; - }; - - ipmmu_vi0: mmu@febd0000 { - compatible = "renesas,ipmmu-r8a77970"; - reg = <0 0xfebd0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 9>; - power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; - #iommu-cells = <1>; - status = "disabled"; - }; - - ipmmu_ir: mmu@ff8b0000 { - compatible = "renesas,ipmmu-r8a77970"; - reg = <0 0xff8b0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 3>; - power-domains = <&sysc R8A77970_PD_A3IR>; - #iommu-cells = <1>; - status = "disabled"; - }; - - ipmmu_rt: mmu@ffc80000 { - compatible = "renesas,ipmmu-r8a77970"; - reg = <0 0xffc80000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 7>; - power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; - #iommu-cells = <1>; - }; - - ipmmu_ds1: mmu@e7740000 { - compatible = "renesas,ipmmu-r8a77970"; - reg = <0 0xe7740000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 1>; - power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; - #iommu-cells = <1>; - }; - - ipmmu_mm: mmu@e67b0000 { - compatible = "renesas,ipmmu-r8a77970"; - reg = <0 0xe67b0000 0 0x1000>; - interrupts = , - ; - power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; - #iommu-cells = <1>; - }; - - pfc: pin-controller@e6060000 { - compatible = "renesas,pfc-r8a77970"; - reg = <0 0xe6060000 0 0x504>; - }; - gpio0: gpio@e6050000 { compatible = "renesas,gpio-r8a77970", "renesas,rcar-gen3-gpio"; @@ -269,6 +207,32 @@ resets = <&cpg 907>; }; + pfc: pin-controller@e6060000 { + compatible = "renesas,pfc-r8a77970"; + reg = <0 0xe6060000 0 0x504>; + }; + + cpg: clock-controller@e6150000 { + compatible = "renesas,r8a77970-cpg-mssr"; + reg = <0 0xe6150000 0 0x1000>; + clocks = <&extal_clk>, <&extalr_clk>; + clock-names = "extal", "extalr"; + #clock-cells = <2>; + #power-domain-cells = <0>; + #reset-cells = <1>; + }; + + rst: reset-controller@e6160000 { + compatible = "renesas,r8a77970-rst"; + reg = <0 0xe6160000 0 0x200>; + }; + + sysc: system-controller@e6180000 { + compatible = "renesas,r8a77970-sysc"; + reg = <0 0xe6180000 0 0x440>; + #power-domain-cells = <1>; + }; + intc_ex: interrupt-controller@e61c0000 { compatible = "renesas,intc-ex-r8a77970", "renesas,irqc"; #interrupt-cells = <2>; @@ -285,67 +249,6 @@ resets = <&cpg 407>; }; - prr: chipid@fff00044 { - compatible = "renesas,prr"; - reg = <0 0xfff00044 0 4>; - }; - - dmac1: dma-controller@e7300000 { - compatible = "renesas,dmac-r8a77970", - "renesas,rcar-dmac"; - reg = <0 0xe7300000 0 0x10000>; - interrupts = ; - interrupt-names = "error", - "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7"; - clocks = <&cpg CPG_MOD 218>; - clock-names = "fck"; - power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; - resets = <&cpg 218>; - #dma-cells = <1>; - dma-channels = <8>; - iommus = <&ipmmu_ds1 0>, <&ipmmu_ds1 1>, - <&ipmmu_ds1 2>, <&ipmmu_ds1 3>, - <&ipmmu_ds1 4>, <&ipmmu_ds1 5>, - <&ipmmu_ds1 6>, <&ipmmu_ds1 7>; - }; - - dmac2: dma-controller@e7310000 { - compatible = "renesas,dmac-r8a77970", - "renesas,rcar-dmac"; - reg = <0 0xe7310000 0 0x10000>; - interrupts = ; - interrupt-names = "error", - "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7"; - clocks = <&cpg CPG_MOD 217>; - clock-names = "fck"; - power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; - resets = <&cpg 217>; - #dma-cells = <1>; - dma-channels = <8>; - iommus = <&ipmmu_ds1 16>, <&ipmmu_ds1 17>, - <&ipmmu_ds1 18>, <&ipmmu_ds1 19>, - <&ipmmu_ds1 20>, <&ipmmu_ds1 21>, - <&ipmmu_ds1 22>, <&ipmmu_ds1 23>; - }; - i2c0: i2c@e6500000 { compatible = "renesas,i2c-r8a77970", "renesas,rcar-gen3-i2c"; @@ -502,6 +405,77 @@ status = "disabled"; }; + canfd: can@e66c0000 { + compatible = "renesas,r8a77970-canfd", + "renesas,rcar-gen3-canfd"; + reg = <0 0xe66c0000 0 0x8000>; + interrupts = , + ; + clocks = <&cpg CPG_MOD 914>, + <&cpg CPG_CORE R8A77970_CLK_CANFD>, + <&can_clk>; + clock-names = "fck", "canfd", "can_clk"; + assigned-clocks = <&cpg CPG_CORE R8A77970_CLK_CANFD>; + assigned-clock-rates = <40000000>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 914>; + status = "disabled"; + + channel0 { + status = "disabled"; + }; + + channel1 { + status = "disabled"; + }; + }; + + avb: ethernet@e6800000 { + compatible = "renesas,etheravb-r8a77970", + "renesas,etheravb-rcar-gen3"; + reg = <0 0xe6800000 0 0x800>; + interrupts = , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + ; + interrupt-names = "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14", "ch15", + "ch16", "ch17", "ch18", "ch19", + "ch20", "ch21", "ch22", "ch23", + "ch24"; + clocks = <&cpg CPG_MOD 812>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 812>; + phy-mode = "rgmii"; + iommus = <&ipmmu_rt 3>; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + scif0: serial@e6e60000 { compatible = "renesas,scif-r8a77970", "renesas,rcar-gen3-scif", @@ -573,57 +547,358 @@ status = "disabled"; }; - avb: ethernet@e6800000 { - compatible = "renesas,etheravb-r8a77970", - "renesas,etheravb-rcar-gen3"; - reg = <0 0xe6800000 0 0x800>; - interrupts = , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - , - ; - interrupt-names = "ch0", "ch1", "ch2", "ch3", - "ch4", "ch5", "ch6", "ch7", - "ch8", "ch9", "ch10", "ch11", - "ch12", "ch13", "ch14", "ch15", - "ch16", "ch17", "ch18", "ch19", - "ch20", "ch21", "ch22", "ch23", - "ch24"; - clocks = <&cpg CPG_MOD 812>; + + vin0: video@e6ef0000 { + compatible = "renesas,vin-r8a77970"; + reg = <0 0xe6ef0000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 811>; power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; - resets = <&cpg 812>; - phy-mode = "rgmii"; - iommus = <&ipmmu_rt 3>; - #address-cells = <1>; - #size-cells = <0>; + resets = <&cpg 811>; + renesas,id = <0>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin0csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin0>; + }; + }; + }; + }; + + vin1: video@e6ef1000 { + compatible = "renesas,vin-r8a77970"; + reg = <0 0xe6ef1000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 810>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 810>; + renesas,id = <1>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin1csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin1>; + }; + }; + }; + }; + + vin2: video@e6ef2000 { + compatible = "renesas,vin-r8a77970"; + reg = <0 0xe6ef2000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 809>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 809>; + renesas,id = <2>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin2csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin2>; + }; + }; + }; + }; + + vin3: video@e6ef3000 { + compatible = "renesas,vin-r8a77970"; + reg = <0 0xe6ef3000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 808>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 808>; + renesas,id = <3>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + vin3csi40: endpoint@2 { + reg = <2>; + remote-endpoint= <&csi40vin3>; + }; + }; + }; + }; + + dmac1: dma-controller@e7300000 { + compatible = "renesas,dmac-r8a77970", + "renesas,rcar-dmac"; + reg = <0 0xe7300000 0 0x10000>; + interrupts = ; + interrupt-names = "error", + "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7"; + clocks = <&cpg CPG_MOD 218>; + clock-names = "fck"; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 218>; + #dma-cells = <1>; + dma-channels = <8>; + iommus = <&ipmmu_ds1 0>, <&ipmmu_ds1 1>, + <&ipmmu_ds1 2>, <&ipmmu_ds1 3>, + <&ipmmu_ds1 4>, <&ipmmu_ds1 5>, + <&ipmmu_ds1 6>, <&ipmmu_ds1 7>; + }; + + dmac2: dma-controller@e7310000 { + compatible = "renesas,dmac-r8a77970", + "renesas,rcar-dmac"; + reg = <0 0xe7310000 0 0x10000>; + interrupts = ; + interrupt-names = "error", + "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7"; + clocks = <&cpg CPG_MOD 217>; + clock-names = "fck"; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 217>; + #dma-cells = <1>; + dma-channels = <8>; + iommus = <&ipmmu_ds1 16>, <&ipmmu_ds1 17>, + <&ipmmu_ds1 18>, <&ipmmu_ds1 19>, + <&ipmmu_ds1 20>, <&ipmmu_ds1 21>, + <&ipmmu_ds1 22>, <&ipmmu_ds1 23>; + }; + + ipmmu_ds1: mmu@e7740000 { + compatible = "renesas,ipmmu-r8a77970"; + reg = <0 0xe7740000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 0>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_ir: mmu@ff8b0000 { + compatible = "renesas,ipmmu-r8a77970"; + reg = <0 0xff8b0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 3>; + power-domains = <&sysc R8A77970_PD_A3IR>; + #iommu-cells = <1>; + }; + + ipmmu_mm: mmu@e67b0000 { + compatible = "renesas,ipmmu-r8a77970"; + reg = <0 0xe67b0000 0 0x1000>; + interrupts = , + ; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_rt: mmu@ffc80000 { + compatible = "renesas,ipmmu-r8a77970"; + reg = <0 0xffc80000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 7>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + ipmmu_vi0: mmu@febd0000 { + compatible = "renesas,ipmmu-r8a77970"; + reg = <0 0xfebd0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 9>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + #iommu-cells = <1>; + }; + + gic: interrupt-controller@f1010000 { + compatible = "arm,gic-400"; + #interrupt-cells = <3>; + #address-cells = <0>; + interrupt-controller; + reg = <0 0xf1010000 0 0x1000>, + <0 0xf1020000 0 0x20000>, + <0 0xf1040000 0 0x20000>, + <0 0xf1060000 0 0x20000>; + interrupts = ; + clocks = <&cpg CPG_MOD 408>; + clock-names = "clk"; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 408>; + }; + + vspd0: vsp@fea20000 { + compatible = "renesas,vsp2"; + reg = <0 0xfea20000 0 0x8000>; + interrupts = ; + clocks = <&cpg CPG_MOD 623>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 623>; + renesas,fcp = <&fcpvd0>; + }; + + fcpvd0: fcp@fea27000 { + compatible = "renesas,fcpv"; + reg = <0 0xfea27000 0 0x200>; + clocks = <&cpg CPG_MOD 603>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 603>; + }; + + csi40: csi2@feaa0000 { + compatible = "renesas,r8a77970-csi2"; + reg = <0 0xfeaa0000 0 0x10000>; + interrupts = ; + clocks = <&cpg CPG_MOD 716>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 716>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@1 { + #address-cells = <1>; + #size-cells = <0>; + + reg = <1>; + + csi40vin0: endpoint@0 { + reg = <0>; + remote-endpoint = <&vin0csi40>; + }; + csi40vin1: endpoint@1 { + reg = <1>; + remote-endpoint = <&vin1csi40>; + }; + csi40vin2: endpoint@2 { + reg = <2>; + remote-endpoint = <&vin2csi40>; + }; + csi40vin3: endpoint@3 { + reg = <3>; + remote-endpoint = <&vin3csi40>; + }; + }; + }; + }; + + du: display@feb00000 { + compatible = "renesas,du-r8a77970"; + reg = <0 0xfeb00000 0 0x80000>; + interrupts = ; + clocks = <&cpg CPG_MOD 724>; + clock-names = "du.0"; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 724>; + vsps = <&vspd0>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + du_out_rgb: endpoint { + }; + }; + + port@1 { + reg = <1>; + du_out_lvds0: endpoint { + remote-endpoint = <&lvds0_in>; + }; + }; + }; + }; + + lvds0: lvds-encoder@feb90000 { + compatible = "renesas,r8a77970-lvds"; + reg = <0 0xfeb90000 0 0x14>; + clocks = <&cpg CPG_MOD 727>; + power-domains = <&sysc R8A77970_PD_ALWAYS_ON>; + resets = <&cpg 727>; + status = "disabled"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + lvds0_in: endpoint { + remote-endpoint = + <&du_out_lvds0>; + }; + }; + port@1 { + reg = <1>; + lvds0_out: endpoint { + }; + }; + }; + }; + + prr: chipid@fff00044 { + compatible = "renesas,prr"; + reg = <0 0xfff00044 0 4>; }; }; timer { compatible = "arm,armv8-timer"; - interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(1) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(1) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(1) | IRQ_TYPE_LEVEL_LOW)>, - <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(1) | IRQ_TYPE_LEVEL_LOW)>; + interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(2) | IRQ_TYPE_LEVEL_LOW)>; }; }; diff --git a/arch/arm64/boot/dts/renesas/r8a77980-condor.dts b/arch/arm64/boot/dts/renesas/r8a77980-condor.dts index 06cf6845765a..0b93a7d76585 100644 --- a/arch/arm64/boot/dts/renesas/r8a77980-condor.dts +++ b/arch/arm64/boot/dts/renesas/r8a77980-condor.dts @@ -27,9 +27,30 @@ /* first 128MB is reserved for secure area. */ reg = <0 0x48000000 0 0x78000000>; }; + + d3_3v: regulator-0 { + compatible = "regulator-fixed"; + regulator-name = "D3.3V"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + regulator-boot-on; + regulator-always-on; + }; + + vddq_vin01: regulator-1 { + compatible = "regulator-fixed"; + regulator-name = "VDDQ_VIN01"; + regulator-min-microvolt = <1800000>; + regulator-max-microvolt = <1800000>; + regulator-boot-on; + regulator-always-on; + }; }; &avb { + pinctrl-0 = <&avb_pins>; + pinctrl-names = "default"; + phy-mode = "rgmii-id"; phy-handle = <&phy0>; renesas,no-ether-link; @@ -41,6 +62,16 @@ }; }; +&canfd { + pinctrl-0 = <&canfd0_pins>; + pinctrl-names = "default"; + status = "okay"; + + channel0 { + status = "okay"; + }; +}; + &extal_clk { clock-frequency = <16666666>; }; @@ -49,7 +80,57 @@ clock-frequency = <32768>; }; +&mmc0 { + pinctrl-0 = <&mmc_pins>; + pinctrl-1 = <&mmc_pins_uhs>; + pinctrl-names = "default", "state_uhs"; + + vmmc-supply = <&d3_3v>; + vqmmc-supply = <&vddq_vin01>; + mmc-hs200-1_8v; + bus-width = <8>; + non-removable; + status = "okay"; +}; + +&pfc { + avb_pins: avb { + groups = "avb_mdio", "avb_rgmii"; + function = "avb"; + }; + + canfd0_pins: canfd0 { + groups = "canfd0_data_a"; + function = "canfd0"; + }; + + mmc_pins: mmc { + groups = "mmc_data8", "mmc_ctrl", "mmc_ds"; + function = "mmc"; + power-source = <3300>; + }; + + mmc_pins_uhs: mmc_uhs { + groups = "mmc_data8", "mmc_ctrl", "mmc_ds"; + function = "mmc"; + power-source = <1800>; + }; + + scif0_pins: scif0 { + groups = "scif0_data"; + function = "scif0"; + }; + + scif_clk_pins: scif_clk { + groups = "scif_clk_b"; + function = "scif_clk"; + }; +}; + &scif0 { + pinctrl-0 = <&scif0_pins>, <&scif_clk_pins>; + pinctrl-names = "default"; + status = "okay"; }; diff --git a/arch/arm64/boot/dts/renesas/r8a77980-v3hsk.dts b/arch/arm64/boot/dts/renesas/r8a77980-v3hsk.dts new file mode 100644 index 000000000000..c9680994555d --- /dev/null +++ b/arch/arm64/boot/dts/renesas/r8a77980-v3hsk.dts @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Device Tree Source for the V3H Starter Kit board + * + * Copyright (C) 2018 Renesas Electronics Corp. + * Copyright (C) 2018 Cogent Embedded, Inc. + */ + +/dts-v1/; +#include "r8a77980.dtsi" + +/ { + model = "Renesas V3H Starter Kit board"; + compatible = "renesas,v3hsk", "renesas,r8a77980"; + + aliases { + serial0 = &scif0; + }; + + chosen { + stdout-path = "serial0:115200n8"; + }; + + memory@48000000 { + device_type = "memory"; + /* first 128MB is reserved for secure area. */ + reg = <0 0x48000000 0 0x78000000>; + }; +}; + +&extal_clk { + clock-frequency = <16666666>; +}; + +&extalr_clk { + clock-frequency = <32768>; +}; + +&pfc { + scif0_pins: scif0 { + groups = "scif0_data"; + function = "scif0"; + }; + + scif_clk_pins: scif_clk { + groups = "scif_clk_b"; + function = "scif_clk"; + }; +}; + +&scif0 { + pinctrl-0 = <&scif0_pins>, <&scif_clk_pins>; + pinctrl-names = "default"; + + status = "okay"; +}; + +&scif_clk { + clock-frequency = <14745600>; +}; diff --git a/arch/arm64/boot/dts/renesas/r8a77980.dtsi b/arch/arm64/boot/dts/renesas/r8a77980.dtsi index 03845fd74996..4c40f9f0ebc9 100644 --- a/arch/arm64/boot/dts/renesas/r8a77980.dtsi +++ b/arch/arm64/boot/dts/renesas/r8a77980.dtsi @@ -6,9 +6,10 @@ * Copyright (C) 2018 Cogent Embedded, Inc. */ +#include #include #include -#include +#include / { compatible = "renesas,r8a77980"; @@ -23,20 +24,27 @@ device_type = "cpu"; compatible = "arm,cortex-a53", "arm,armv8"; reg = <0>; - clocks = <&cpg CPG_CORE 0>; - power-domains = <&sysc 5>; + clocks = <&cpg CPG_CORE R8A77980_CLK_Z2>; + power-domains = <&sysc R8A77980_PD_CA53_CPU0>; next-level-cache = <&L2_CA53>; enable-method = "psci"; }; L2_CA53: cache-controller { compatible = "cache"; - power-domains = <&sysc 21>; + power-domains = <&sysc R8A77980_PD_CA53_SCU>; cache-unified; cache-level = <2>; }; }; + /* External CAN clock - to be overridden by boards that provide it */ + can_clk: can { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; + }; + extal_clk: extal { compatible = "fixed-clock"; #clock-cells = <0>; @@ -71,6 +79,11 @@ #size-cells = <2>; ranges; + pfc: pin-controller@e6060000 { + compatible = "renesas,pfc-r8a77980"; + reg = <0 0xe6060000 0 0x50c>; + }; + cpg: clock-controller@e6150000 { compatible = "renesas,r8a77980-cpg-mssr"; reg = <0 0xe6150000 0 0x1000>; @@ -99,13 +112,13 @@ reg = <0 0xe6540000 0 0x60>; interrupts = ; clocks = <&cpg CPG_MOD 520>, - <&cpg CPG_CORE 19>, + <&cpg CPG_CORE R8A77980_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; dmas = <&dmac1 0x31>, <&dmac1 0x30>, <&dmac2 0x31>, <&dmac2 0x30>; dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 520>; status = "disabled"; }; @@ -117,13 +130,13 @@ reg = <0 0xe6550000 0 0x60>; interrupts = ; clocks = <&cpg CPG_MOD 519>, - <&cpg CPG_CORE 19>, + <&cpg CPG_CORE R8A77980_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; dmas = <&dmac1 0x33>, <&dmac1 0x32>, <&dmac2 0x33>, <&dmac2 0x32>; dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 519>; status = "disabled"; }; @@ -135,13 +148,13 @@ reg = <0 0xe6560000 0 0x60>; interrupts = ; clocks = <&cpg CPG_MOD 518>, - <&cpg CPG_CORE 19>, + <&cpg CPG_CORE R8A77980_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; dmas = <&dmac1 0x35>, <&dmac1 0x34>, <&dmac2 0x35>, <&dmac2 0x34>; dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 518>; status = "disabled"; }; @@ -153,17 +166,42 @@ reg = <0 0xe66a0000 0 0x60>; interrupts = ; clocks = <&cpg CPG_MOD 517>, - <&cpg CPG_CORE 19>, + <&cpg CPG_CORE R8A77980_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; dmas = <&dmac1 0x37>, <&dmac1 0x36>, <&dmac2 0x37>, <&dmac2 0x36>; dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 517>; status = "disabled"; }; + canfd: can@e66c0000 { + compatible = "renesas,r8a77980-canfd", + "renesas,rcar-gen3-canfd"; + reg = <0 0xe66c0000 0 0x8000>; + interrupts = , + ; + clocks = <&cpg CPG_MOD 914>, + <&cpg CPG_CORE R8A77980_CLK_CANFD>, + <&can_clk>; + clock-names = "fck", "canfd", "can_clk"; + assigned-clocks = <&cpg CPG_CORE R8A77980_CLK_CANFD>; + assigned-clock-rates = <40000000>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; + resets = <&cpg 914>; + status = "disabled"; + + channel0 { + status = "disabled"; + }; + + channel1 { + status = "disabled"; + }; + }; + avb: ethernet@e6800000 { compatible = "renesas,etheravb-r8a77980", "renesas,etheravb-rcar-gen3"; @@ -201,11 +239,12 @@ "ch20", "ch21", "ch22", "ch23", "ch24"; clocks = <&cpg CPG_MOD 812>; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 812>; phy-mode = "rgmii"; #address-cells = <1>; #size-cells = <0>; + status = "disabled"; }; scif0: serial@e6e60000 { @@ -215,13 +254,13 @@ reg = <0 0xe6e60000 0 0x40>; interrupts = ; clocks = <&cpg CPG_MOD 207>, - <&cpg CPG_CORE 19>, + <&cpg CPG_CORE R8A77980_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; dmas = <&dmac1 0x51>, <&dmac1 0x50>, <&dmac2 0x51>, <&dmac2 0x50>; dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 207>; status = "disabled"; }; @@ -233,13 +272,13 @@ reg = <0 0xe6e68000 0 0x40>; interrupts = ; clocks = <&cpg CPG_MOD 206>, - <&cpg CPG_CORE 19>, + <&cpg CPG_CORE R8A77980_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; dmas = <&dmac1 0x53>, <&dmac1 0x52>, <&dmac2 0x53>, <&dmac2 0x52>; dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 206>; status = "disabled"; }; @@ -251,13 +290,13 @@ reg = <0 0xe6c50000 0 0x40>; interrupts = ; clocks = <&cpg CPG_MOD 204>, - <&cpg CPG_CORE 19>, + <&cpg CPG_CORE R8A77980_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; dmas = <&dmac1 0x57>, <&dmac1 0x56>, <&dmac2 0x57>, <&dmac2 0x56>; dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 204>; status = "disabled"; }; @@ -269,13 +308,13 @@ reg = <0 0xe6c40000 0 0x40>; interrupts = ; clocks = <&cpg CPG_MOD 203>, - <&cpg CPG_CORE 19>, + <&cpg CPG_CORE R8A77980_CLK_S3D1>, <&scif_clk>; clock-names = "fck", "brg_int", "scif_clk"; dmas = <&dmac1 0x59>, <&dmac1 0x58>, <&dmac2 0x59>, <&dmac2 0x58>; dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 203>; status = "disabled"; }; @@ -308,7 +347,7 @@ "ch12", "ch13", "ch14", "ch15"; clocks = <&cpg CPG_MOD 218>; clock-names = "fck"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 218>; #dma-cells = <1>; dma-channels = <16>; @@ -342,12 +381,24 @@ "ch12", "ch13", "ch14", "ch15"; clocks = <&cpg CPG_MOD 217>; clock-names = "fck"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 217>; #dma-cells = <1>; dma-channels = <16>; }; + mmc0: mmc@ee140000 { + compatible = "renesas,sdhi-r8a77980", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee140000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 314>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; + resets = <&cpg 314>; + max-frequency = <200000000>; + status = "disabled"; + }; + gic: interrupt-controller@f1010000 { compatible = "arm,gic-400"; #interrupt-cells = <3>; @@ -361,7 +412,7 @@ IRQ_TYPE_LEVEL_HIGH)>; clocks = <&cpg CPG_MOD 408>; clock-names = "clk"; - power-domains = <&sysc 32>; + power-domains = <&sysc R8A77980_PD_ALWAYS_ON>; resets = <&cpg 408>; }; diff --git a/arch/arm64/boot/dts/renesas/r8a77990-ebisu.dts b/arch/arm64/boot/dts/renesas/r8a77990-ebisu.dts new file mode 100644 index 000000000000..7a09d0524f9b --- /dev/null +++ b/arch/arm64/boot/dts/renesas/r8a77990-ebisu.dts @@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Device Tree Source for the ebisu board + * + * Copyright (C) 2018 Renesas Electronics Corp. + */ + +/dts-v1/; +#include "r8a77990.dtsi" +#include + +/ { + model = "Renesas Ebisu board based on r8a77990"; + compatible = "renesas,ebisu", "renesas,r8a77990"; + + aliases { + serial0 = &scif2; + ethernet0 = &avb; + }; + + chosen { + bootargs = "ignore_loglevel"; + stdout-path = "serial0:115200n8"; + }; + + memory@48000000 { + device_type = "memory"; + /* first 128MB is reserved for secure area. */ + reg = <0x0 0x48000000 0x0 0x38000000>; + }; +}; + +&avb { + pinctrl-0 = <&avb_pins>; + pinctrl-names = "default"; + renesas,no-ether-link; + phy-handle = <&phy0>; + phy-mode = "rgmii-txid"; + status = "okay"; + + phy0: ethernet-phy@0 { + rxc-skew-ps = <1500>; + reg = <0>; + interrupt-parent = <&gpio2>; + interrupts = <21 IRQ_TYPE_LEVEL_LOW>; + reset-gpios = <&gpio1 20 GPIO_ACTIVE_LOW>; + }; +}; + +&extal_clk { + clock-frequency = <48000000>; +}; + +&pfc { + avb_pins: avb { + mux { + groups = "avb_link", "avb_mii"; + function = "avb"; + }; + }; +}; + +&scif2 { + status = "okay"; +}; diff --git a/arch/arm64/boot/dts/renesas/r8a77990.dtsi b/arch/arm64/boot/dts/renesas/r8a77990.dtsi new file mode 100644 index 000000000000..be4f519711a1 --- /dev/null +++ b/arch/arm64/boot/dts/renesas/r8a77990.dtsi @@ -0,0 +1,281 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Device Tree Source for the r8a77990 SoC + * + * Copyright (C) 2018 Renesas Electronics Corp. + */ + +#include +#include + +/ { + compatible = "renesas,r8a77990"; + #address-cells = <2>; + #size-cells = <2>; + + cpus { + #address-cells = <1>; + #size-cells = <0>; + + /* 1 core only at this point */ + a53_0: cpu@0 { + compatible = "arm,cortex-a53", "arm,armv8"; + reg = <0x0>; + device_type = "cpu"; + power-domains = <&sysc 5>; + next-level-cache = <&L2_CA53>; + enable-method = "psci"; + }; + + L2_CA53: cache-controller-0 { + compatible = "cache"; + power-domains = <&sysc 21>; + cache-unified; + cache-level = <2>; + }; + }; + + extal_clk: extal { + compatible = "fixed-clock"; + #clock-cells = <0>; + /* This value must be overridden by the board */ + clock-frequency = <0>; + }; + + pmu_a53 { + compatible = "arm,cortex-a53-pmu"; + interrupts-extended = <&gic GIC_SPI 84 IRQ_TYPE_LEVEL_HIGH>; + interrupt-affinity = <&a53_0>; + }; + + psci { + compatible = "arm,psci-1.0", "arm,psci-0.2"; + method = "smc"; + }; + + soc: soc { + compatible = "simple-bus"; + interrupt-parent = <&gic>; + #address-cells = <2>; + #size-cells = <2>; + ranges; + + gpio0: gpio@e6050000 { + compatible = "renesas,gpio-r8a77990", + "renesas,rcar-gen3-gpio"; + reg = <0 0xe6050000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 0 18>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 912>; + power-domains = <&sysc 32>; + resets = <&cpg 912>; + }; + + gpio1: gpio@e6051000 { + compatible = "renesas,gpio-r8a77990", + "renesas,rcar-gen3-gpio"; + reg = <0 0xe6051000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 32 23>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 911>; + power-domains = <&sysc 32>; + resets = <&cpg 911>; + }; + + gpio2: gpio@e6052000 { + compatible = "renesas,gpio-r8a77990", + "renesas,rcar-gen3-gpio"; + reg = <0 0xe6052000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 64 26>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 910>; + power-domains = <&sysc 32>; + resets = <&cpg 910>; + }; + + gpio3: gpio@e6053000 { + compatible = "renesas,gpio-r8a77990", + "renesas,rcar-gen3-gpio"; + reg = <0 0xe6053000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 96 16>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 909>; + power-domains = <&sysc 32>; + resets = <&cpg 909>; + }; + + gpio4: gpio@e6054000 { + compatible = "renesas,gpio-r8a77990", + "renesas,rcar-gen3-gpio"; + reg = <0 0xe6054000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 128 11>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 908>; + power-domains = <&sysc 32>; + resets = <&cpg 908>; + }; + + gpio5: gpio@e6055000 { + compatible = "renesas,gpio-r8a77990", + "renesas,rcar-gen3-gpio"; + reg = <0 0xe6055000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 160 20>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 907>; + power-domains = <&sysc 32>; + resets = <&cpg 907>; + }; + + gpio6: gpio@e6055400 { + compatible = "renesas,gpio-r8a77990", + "renesas,rcar-gen3-gpio"; + reg = <0 0xe6055400 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 192 18>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 906>; + power-domains = <&sysc 32>; + resets = <&cpg 906>; + }; + + pfc: pin-controller@e6060000 { + compatible = "renesas,pfc-r8a77990"; + reg = <0 0xe6060000 0 0x508>; + }; + + cpg: clock-controller@e6150000 { + compatible = "renesas,r8a77990-cpg-mssr"; + reg = <0 0xe6150000 0 0x1000>; + clocks = <&extal_clk>; + clock-names = "extal"; + #clock-cells = <2>; + #power-domain-cells = <0>; + #reset-cells = <1>; + }; + + rst: reset-controller@e6160000 { + compatible = "renesas,r8a77990-rst"; + reg = <0 0xe6160000 0 0x0200>; + }; + + sysc: system-controller@e6180000 { + compatible = "renesas,r8a77990-sysc"; + reg = <0 0xe6180000 0 0x0400>; + #power-domain-cells = <1>; + }; + + avb: ethernet@e6800000 { + compatible = "renesas,etheravb-r8a77990", + "renesas,etheravb-rcar-gen3"; + reg = <0 0xe6800000 0 0x800>, <0 0xe6a00000 0 0x10000>; + interrupts = , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + , + ; + interrupt-names = "ch0", "ch1", "ch2", "ch3", + "ch4", "ch5", "ch6", "ch7", + "ch8", "ch9", "ch10", "ch11", + "ch12", "ch13", "ch14", "ch15", + "ch16", "ch17", "ch18", "ch19", + "ch20", "ch21", "ch22", "ch23", + "ch24"; + clocks = <&cpg CPG_MOD 812>; + power-domains = <&sysc 32>; + resets = <&cpg 812>; + phy-mode = "rgmii"; + #address-cells = <1>; + #size-cells = <0>; + status = "disabled"; + }; + + scif2: serial@e6e88000 { + compatible = "renesas,scif-r8a77990", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6e88000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 310>; + clock-names = "fck"; + power-domains = <&sysc 32>; + resets = <&cpg 310>; + status = "disabled"; + }; + + gic: interrupt-controller@f1010000 { + compatible = "arm,gic-400"; + #interrupt-cells = <3>; + #address-cells = <0>; + interrupt-controller; + reg = <0x0 0xf1010000 0 0x1000>, + <0x0 0xf1020000 0 0x20000>, + <0x0 0xf1040000 0 0x20000>, + <0x0 0xf1060000 0 0x20000>; + interrupts = ; + clocks = <&cpg CPG_MOD 408>; + clock-names = "clk"; + power-domains = <&sysc 32>; + resets = <&cpg 408>; + }; + + prr: chipid@fff00044 { + compatible = "renesas,prr"; + reg = <0 0xfff00044 0 4>; + }; + }; + + timer { + compatible = "arm,armv8-timer"; + interrupts-extended = <&gic GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(1) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(1) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(1) | IRQ_TYPE_LEVEL_LOW)>, + <&gic GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(1) | IRQ_TYPE_LEVEL_LOW)>; + }; +}; diff --git a/arch/arm64/boot/dts/renesas/r8a77995-draak.dts b/arch/arm64/boot/dts/renesas/r8a77995-draak.dts index d03f19414028..9d73de8bc94d 100644 --- a/arch/arm64/boot/dts/renesas/r8a77995-draak.dts +++ b/arch/arm64/boot/dts/renesas/r8a77995-draak.dts @@ -91,7 +91,7 @@ &pfc { avb0_pins: avb { mux { - groups = "avb0_link", "avb0_mdc", "avb0_mii"; + groups = "avb0_link", "avb0_mdio", "avb0_mii"; function = "avb0"; }; }; diff --git a/arch/arm64/boot/dts/renesas/r8a77995.dtsi b/arch/arm64/boot/dts/renesas/r8a77995.dtsi index 82aed7ee984c..2506f46293e8 100644 --- a/arch/arm64/boot/dts/renesas/r8a77995.dtsi +++ b/arch/arm64/boot/dts/renesas/r8a77995.dtsi @@ -18,9 +18,11 @@ #address-cells = <2>; #size-cells = <2>; - psci { - compatible = "arm,psci-1.0", "arm,psci-0.2"; - method = "smc"; + /* External CAN clock - to be overridden by boards that provide it */ + can_clk: can { + compatible = "fixed-clock"; + #clock-cells = <0>; + clock-frequency = <0>; }; cpus { @@ -51,18 +53,16 @@ clock-frequency = <0>; }; - /* External CAN clock - to be overridden by boards that provide it */ - can_clk: can { - compatible = "fixed-clock"; - #clock-cells = <0>; - clock-frequency = <0>; - }; - pmu_a53 { compatible = "arm,cortex-a53-pmu"; interrupts-extended = <&gic GIC_SPI 84 IRQ_TYPE_LEVEL_HIGH>; }; + psci { + compatible = "arm,psci-1.0", "arm,psci-0.2"; + method = "smc"; + }; + scif_clk: scif { compatible = "fixed-clock"; #clock-cells = <0>; @@ -76,23 +76,6 @@ #size-cells = <2>; ranges; - gic: interrupt-controller@f1010000 { - compatible = "arm,gic-400"; - #interrupt-cells = <3>; - #address-cells = <0>; - interrupt-controller; - reg = <0x0 0xf1010000 0 0x1000>, - <0x0 0xf1020000 0 0x20000>, - <0x0 0xf1040000 0 0x20000>, - <0x0 0xf1060000 0 0x20000>; - interrupts = ; - clocks = <&cpg CPG_MOD 408>; - clock-names = "clk"; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 408>; - }; - rwdt: watchdog@e6020000 { compatible = "renesas,r8a77995-wdt", "renesas,rcar-gen3-wdt"; @@ -103,88 +86,123 @@ status = "disabled"; }; - ipmmu_vi0: mmu@febd0000 { - compatible = "renesas,ipmmu-r8a77995"; - reg = <0 0xfebd0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 14>; - #iommu-cells = <1>; - status = "disabled"; - }; - - ipmmu_vp0: mmu@fe990000 { - compatible = "renesas,ipmmu-r8a77995"; - reg = <0 0xfe990000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 16>; - #iommu-cells = <1>; - status = "disabled"; - }; - - ipmmu_vc0: mmu@fe6b0000 { - compatible = "renesas,ipmmu-r8a77995"; - reg = <0 0xfe6b0000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 12>; - #iommu-cells = <1>; - status = "disabled"; + gpio0: gpio@e6050000 { + compatible = "renesas,gpio-r8a77995", + "renesas,rcar-gen3-gpio", + "renesas,gpio-rcar"; + reg = <0 0xe6050000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 0 9>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 912>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 912>; }; - ipmmu_pv0: mmu@fd800000 { - compatible = "renesas,ipmmu-r8a77995"; - reg = <0 0xfd800000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 6>; - #iommu-cells = <1>; - status = "disabled"; + gpio1: gpio@e6051000 { + compatible = "renesas,gpio-r8a77995", + "renesas,rcar-gen3-gpio", + "renesas,gpio-rcar"; + reg = <0 0xe6051000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 32 32>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 911>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 911>; }; - ipmmu_hc: mmu@e6570000 { - compatible = "renesas,ipmmu-r8a77995"; - reg = <0 0xe6570000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 2>; - #iommu-cells = <1>; - status = "disabled"; + gpio2: gpio@e6052000 { + compatible = "renesas,gpio-r8a77995", + "renesas,rcar-gen3-gpio", + "renesas,gpio-rcar"; + reg = <0 0xe6052000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 64 32>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 910>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 910>; }; - ipmmu_rt: mmu@ffc80000 { - compatible = "renesas,ipmmu-r8a77995"; - reg = <0 0xffc80000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 10>; - #iommu-cells = <1>; - status = "disabled"; + gpio3: gpio@e6053000 { + compatible = "renesas,gpio-r8a77995", + "renesas,rcar-gen3-gpio", + "renesas,gpio-rcar"; + reg = <0 0xe6053000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 96 10>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 909>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 909>; }; - ipmmu_mp: mmu@ec670000 { - compatible = "renesas,ipmmu-r8a77995"; - reg = <0 0xec670000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 4>; - #iommu-cells = <1>; - status = "disabled"; + gpio4: gpio@e6054000 { + compatible = "renesas,gpio-r8a77995", + "renesas,rcar-gen3-gpio", + "renesas,gpio-rcar"; + reg = <0 0xe6054000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 128 32>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 908>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 908>; }; - ipmmu_ds0: mmu@e6740000 { - compatible = "renesas,ipmmu-r8a77995"; - reg = <0 0xe6740000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 0>; - #iommu-cells = <1>; - status = "disabled"; + gpio5: gpio@e6055000 { + compatible = "renesas,gpio-r8a77995", + "renesas,rcar-gen3-gpio", + "renesas,gpio-rcar"; + reg = <0 0xe6055000 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 160 21>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 907>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 907>; }; - ipmmu_ds1: mmu@e7740000 { - compatible = "renesas,ipmmu-r8a77995"; - reg = <0 0xe7740000 0 0x1000>; - renesas,ipmmu-main = <&ipmmu_mm 1>; - #iommu-cells = <1>; - status = "disabled"; + gpio6: gpio@e6055400 { + compatible = "renesas,gpio-r8a77995", + "renesas,rcar-gen3-gpio", + "renesas,gpio-rcar"; + reg = <0 0xe6055400 0 0x50>; + interrupts = ; + #gpio-cells = <2>; + gpio-controller; + gpio-ranges = <&pfc 0 192 14>; + #interrupt-cells = <2>; + interrupt-controller; + clocks = <&cpg CPG_MOD 906>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 906>; }; - ipmmu_mm: mmu@e67b0000 { - compatible = "renesas,ipmmu-r8a77995"; - reg = <0 0xe67b0000 0 0x1000>; - interrupts = , - ; - #iommu-cells = <1>; - status = "disabled"; + pfc: pin-controller@e6060000 { + compatible = "renesas,pfc-r8a77995"; + reg = <0 0xe6060000 0 0x508>; }; - cpg: clock-controller@e6150000 { compatible = "renesas,r8a77995-cpg-mssr"; reg = <0 0xe6150000 0 0x1000>; @@ -200,16 +218,6 @@ reg = <0 0xe6160000 0 0x0200>; }; - pfc: pin-controller@e6060000 { - compatible = "renesas,pfc-r8a77995"; - reg = <0 0xe6060000 0 0x508>; - }; - - prr: chipid@fff00044 { - compatible = "renesas,prr"; - reg = <0 0xfff00044 0 4>; - }; - sysc: system-controller@e6180000 { compatible = "renesas,r8a77995-sysc"; reg = <0 0xe6180000 0 0x0400>; @@ -232,6 +240,98 @@ resets = <&cpg 407>; }; + i2c0: i2c@e6500000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a77995", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe6500000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 931>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 931>; + dmas = <&dmac1 0x91>, <&dmac1 0x90>, + <&dmac2 0x91>, <&dmac2 0x90>; + dma-names = "tx", "rx", "tx", "rx"; + i2c-scl-internal-delay-ns = <6>; + status = "disabled"; + }; + + i2c1: i2c@e6508000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a77995", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe6508000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 930>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 930>; + dmas = <&dmac1 0x93>, <&dmac1 0x92>, + <&dmac2 0x93>, <&dmac2 0x92>; + dma-names = "tx", "rx", "tx", "rx"; + i2c-scl-internal-delay-ns = <6>; + status = "disabled"; + }; + + i2c2: i2c@e6510000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a77995", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe6510000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 929>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 929>; + dmas = <&dmac1 0x95>, <&dmac1 0x94>, + <&dmac2 0x95>, <&dmac2 0x94>; + dma-names = "tx", "rx", "tx", "rx"; + i2c-scl-internal-delay-ns = <6>; + status = "disabled"; + }; + + i2c3: i2c@e66d0000 { + #address-cells = <1>; + #size-cells = <0>; + compatible = "renesas,i2c-r8a77995", + "renesas,rcar-gen3-i2c"; + reg = <0 0xe66d0000 0 0x40>; + interrupts = ; + clocks = <&cpg CPG_MOD 928>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 928>; + dmas = <&dmac0 0x97>, <&dmac0 0x96>; + dma-names = "tx", "rx"; + i2c-scl-internal-delay-ns = <6>; + status = "disabled"; + }; + + canfd: can@e66c0000 { + compatible = "renesas,r8a77995-canfd", + "renesas,rcar-gen3-canfd"; + reg = <0 0xe66c0000 0 0x8000>; + interrupts = , + ; + clocks = <&cpg CPG_MOD 914>, + <&cpg CPG_CORE R8A77995_CLK_CANFD>, + <&can_clk>; + clock-names = "fck", "canfd", "can_clk"; + assigned-clocks = <&cpg CPG_CORE R8A77995_CLK_CANFD>; + assigned-clock-rates = <40000000>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 914>; + status = "disabled"; + + channel0 { + status = "disabled"; + }; + + channel1 { + status = "disabled"; + }; + }; + dmac0: dma-controller@e6700000 { compatible = "renesas,dmac-r8a77995", "renesas,rcar-dmac"; @@ -304,173 +404,75 @@ dma-channels = <8>; }; - gpio0: gpio@e6050000 { - compatible = "renesas,gpio-r8a77995", - "renesas,rcar-gen3-gpio", - "renesas,gpio-rcar"; - reg = <0 0xe6050000 0 0x50>; - interrupts = ; - #gpio-cells = <2>; - gpio-controller; - gpio-ranges = <&pfc 0 0 9>; - #interrupt-cells = <2>; - interrupt-controller; - clocks = <&cpg CPG_MOD 912>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 912>; - }; - - gpio1: gpio@e6051000 { - compatible = "renesas,gpio-r8a77995", - "renesas,rcar-gen3-gpio", - "renesas,gpio-rcar"; - reg = <0 0xe6051000 0 0x50>; - interrupts = ; - #gpio-cells = <2>; - gpio-controller; - gpio-ranges = <&pfc 0 32 32>; - #interrupt-cells = <2>; - interrupt-controller; - clocks = <&cpg CPG_MOD 911>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 911>; - }; - - gpio2: gpio@e6052000 { - compatible = "renesas,gpio-r8a77995", - "renesas,rcar-gen3-gpio", - "renesas,gpio-rcar"; - reg = <0 0xe6052000 0 0x50>; - interrupts = ; - #gpio-cells = <2>; - gpio-controller; - gpio-ranges = <&pfc 0 64 32>; - #interrupt-cells = <2>; - interrupt-controller; - clocks = <&cpg CPG_MOD 910>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 910>; - }; - - gpio3: gpio@e6053000 { - compatible = "renesas,gpio-r8a77995", - "renesas,rcar-gen3-gpio", - "renesas,gpio-rcar"; - reg = <0 0xe6053000 0 0x50>; - interrupts = ; - #gpio-cells = <2>; - gpio-controller; - gpio-ranges = <&pfc 0 96 10>; - #interrupt-cells = <2>; - interrupt-controller; - clocks = <&cpg CPG_MOD 909>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 909>; + ipmmu_ds0: mmu@e6740000 { + compatible = "renesas,ipmmu-r8a77995"; + reg = <0 0xe6740000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 0>; + #iommu-cells = <1>; }; - gpio4: gpio@e6054000 { - compatible = "renesas,gpio-r8a77995", - "renesas,rcar-gen3-gpio", - "renesas,gpio-rcar"; - reg = <0 0xe6054000 0 0x50>; - interrupts = ; - #gpio-cells = <2>; - gpio-controller; - gpio-ranges = <&pfc 0 128 32>; - #interrupt-cells = <2>; - interrupt-controller; - clocks = <&cpg CPG_MOD 908>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 908>; + ipmmu_ds1: mmu@e7740000 { + compatible = "renesas,ipmmu-r8a77995"; + reg = <0 0xe7740000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 1>; + #iommu-cells = <1>; }; - gpio5: gpio@e6055000 { - compatible = "renesas,gpio-r8a77995", - "renesas,rcar-gen3-gpio", - "renesas,gpio-rcar"; - reg = <0 0xe6055000 0 0x50>; - interrupts = ; - #gpio-cells = <2>; - gpio-controller; - gpio-ranges = <&pfc 0 160 21>; - #interrupt-cells = <2>; - interrupt-controller; - clocks = <&cpg CPG_MOD 907>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 907>; + ipmmu_hc: mmu@e6570000 { + compatible = "renesas,ipmmu-r8a77995"; + reg = <0 0xe6570000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 2>; + #iommu-cells = <1>; }; - gpio6: gpio@e6055400 { - compatible = "renesas,gpio-r8a77995", - "renesas,rcar-gen3-gpio", - "renesas,gpio-rcar"; - reg = <0 0xe6055400 0 0x50>; - interrupts = ; - #gpio-cells = <2>; - gpio-controller; - gpio-ranges = <&pfc 0 192 14>; - #interrupt-cells = <2>; - interrupt-controller; - clocks = <&cpg CPG_MOD 906>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 906>; + ipmmu_mm: mmu@e67b0000 { + compatible = "renesas,ipmmu-r8a77995"; + reg = <0 0xe67b0000 0 0x1000>; + interrupts = , + ; + #iommu-cells = <1>; }; - can0: can@e6c30000 { - compatible = "renesas,can-r8a77995", - "renesas,rcar-gen3-can"; - reg = <0 0xe6c30000 0 0x1000>; - interrupts = ; - clocks = <&cpg CPG_MOD 916>, - <&cpg CPG_CORE R8A77995_CLK_CANFD>, - <&can_clk>; - clock-names = "clkp1", "clkp2", "can_clk"; - assigned-clocks = <&cpg CPG_CORE R8A77995_CLK_CANFD>; - assigned-clock-rates = <40000000>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 916>; - status = "disabled"; + ipmmu_mp: mmu@ec670000 { + compatible = "renesas,ipmmu-r8a77995"; + reg = <0 0xec670000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 4>; + #iommu-cells = <1>; }; - can1: can@e6c38000 { - compatible = "renesas,can-r8a77995", - "renesas,rcar-gen3-can"; - reg = <0 0xe6c38000 0 0x1000>; - interrupts = ; - clocks = <&cpg CPG_MOD 915>, - <&cpg CPG_CORE R8A77995_CLK_CANFD>, - <&can_clk>; - clock-names = "clkp1", "clkp2", "can_clk"; - assigned-clocks = <&cpg CPG_CORE R8A77995_CLK_CANFD>; - assigned-clock-rates = <40000000>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 915>; - status = "disabled"; + ipmmu_pv0: mmu@fd800000 { + compatible = "renesas,ipmmu-r8a77995"; + reg = <0 0xfd800000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 6>; + #iommu-cells = <1>; }; - canfd: can@e66c0000 { - compatible = "renesas,r8a77995-canfd", - "renesas,rcar-gen3-canfd"; - reg = <0 0xe66c0000 0 0x8000>; - interrupts = , - ; - clocks = <&cpg CPG_MOD 914>, - <&cpg CPG_CORE R8A77995_CLK_CANFD>, - <&can_clk>; - clock-names = "fck", "canfd", "can_clk"; - assigned-clocks = <&cpg CPG_CORE R8A77995_CLK_CANFD>; - assigned-clock-rates = <40000000>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 914>; - status = "disabled"; + ipmmu_rt: mmu@ffc80000 { + compatible = "renesas,ipmmu-r8a77995"; + reg = <0 0xffc80000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 10>; + #iommu-cells = <1>; + }; - channel0 { - status = "disabled"; - }; + ipmmu_vc0: mmu@fe6b0000 { + compatible = "renesas,ipmmu-r8a77995"; + reg = <0 0xfe6b0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 12>; + #iommu-cells = <1>; + }; - channel1 { - status = "disabled"; - }; + ipmmu_vi0: mmu@febd0000 { + compatible = "renesas,ipmmu-r8a77995"; + reg = <0 0xfebd0000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 14>; + #iommu-cells = <1>; + }; + + ipmmu_vp0: mmu@fe990000 { + compatible = "renesas,ipmmu-r8a77995"; + reg = <0 0xfe990000 0 0x1000>; + renesas,ipmmu-main = <&ipmmu_mm 16>; + #iommu-cells = <1>; }; avb: ethernet@e6800000 { @@ -519,87 +521,35 @@ status = "disabled"; }; - scif2: serial@e6e88000 { - compatible = "renesas,scif-r8a77995", - "renesas,rcar-gen3-scif", "renesas,scif"; - reg = <0 0xe6e88000 0 64>; - interrupts = ; - clocks = <&cpg CPG_MOD 310>, - <&cpg CPG_CORE R8A77995_CLK_S3D1C>, - <&scif_clk>; - clock-names = "fck", "brg_int", "scif_clk"; - dmas = <&dmac1 0x13>, <&dmac1 0x12>, - <&dmac2 0x13>, <&dmac2 0x12>; - dma-names = "tx", "rx", "tx", "rx"; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 310>; - status = "disabled"; - }; - - i2c0: i2c@e6500000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a77995", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe6500000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 931>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 931>; - dmas = <&dmac1 0x91>, <&dmac1 0x90>, - <&dmac2 0x91>, <&dmac2 0x90>; - dma-names = "tx", "rx", "tx", "rx"; - i2c-scl-internal-delay-ns = <6>; - status = "disabled"; - }; - - i2c1: i2c@e6508000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a77995", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe6508000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 930>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 930>; - dmas = <&dmac1 0x93>, <&dmac1 0x92>, - <&dmac2 0x93>, <&dmac2 0x92>; - dma-names = "tx", "rx", "tx", "rx"; - i2c-scl-internal-delay-ns = <6>; - status = "disabled"; - }; - - i2c2: i2c@e6510000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a77995", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe6510000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 929>; + can0: can@e6c30000 { + compatible = "renesas,can-r8a77995", + "renesas,rcar-gen3-can"; + reg = <0 0xe6c30000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 916>, + <&cpg CPG_CORE R8A77995_CLK_CANFD>, + <&can_clk>; + clock-names = "clkp1", "clkp2", "can_clk"; + assigned-clocks = <&cpg CPG_CORE R8A77995_CLK_CANFD>; + assigned-clock-rates = <40000000>; power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 929>; - dmas = <&dmac1 0x95>, <&dmac1 0x94>, - <&dmac2 0x95>, <&dmac2 0x94>; - dma-names = "tx", "rx", "tx", "rx"; - i2c-scl-internal-delay-ns = <6>; + resets = <&cpg 916>; status = "disabled"; }; - i2c3: i2c@e66d0000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "renesas,i2c-r8a77995", - "renesas,rcar-gen3-i2c"; - reg = <0 0xe66d0000 0 0x40>; - interrupts = ; - clocks = <&cpg CPG_MOD 928>; + can1: can@e6c38000 { + compatible = "renesas,can-r8a77995", + "renesas,rcar-gen3-can"; + reg = <0 0xe6c38000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 915>, + <&cpg CPG_CORE R8A77995_CLK_CANFD>, + <&can_clk>; + clock-names = "clkp1", "clkp2", "can_clk"; + assigned-clocks = <&cpg CPG_CORE R8A77995_CLK_CANFD>; + assigned-clock-rates = <40000000>; power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 928>; - dmas = <&dmac0 0x97>, <&dmac0 0x96>; - dma-names = "tx", "rx"; - i2c-scl-internal-delay-ns = <6>; + resets = <&cpg 915>; status = "disabled"; }; @@ -643,38 +593,54 @@ status = "disabled"; }; - sdhi2: sd@ee140000 { - compatible = "renesas,sdhi-r8a77995", - "renesas,rcar-gen3-sdhi"; - reg = <0 0xee140000 0 0x2000>; - interrupts = ; - clocks = <&cpg CPG_MOD 312>; - max-frequency = <200000000>; + scif2: serial@e6e88000 { + compatible = "renesas,scif-r8a77995", + "renesas,rcar-gen3-scif", "renesas,scif"; + reg = <0 0xe6e88000 0 64>; + interrupts = ; + clocks = <&cpg CPG_MOD 310>, + <&cpg CPG_CORE R8A77995_CLK_S3D1C>, + <&scif_clk>; + clock-names = "fck", "brg_int", "scif_clk"; + dmas = <&dmac1 0x13>, <&dmac1 0x12>, + <&dmac2 0x13>, <&dmac2 0x12>; + dma-names = "tx", "rx", "tx", "rx"; power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 312>; + resets = <&cpg 310>; status = "disabled"; }; - ehci0: usb@ee080100 { - compatible = "generic-ehci"; - reg = <0 0xee080100 0 0x100>; + vin4: video@e6ef4000 { + compatible = "renesas,vin-r8a77995"; + reg = <0 0xe6ef4000 0 0x1000>; + interrupts = ; + clocks = <&cpg CPG_MOD 807>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 807>; + renesas,id = <4>; + status = "disabled"; + }; + + ohci0: usb@ee080000 { + compatible = "generic-ohci"; + reg = <0 0xee080000 0 0x100>; interrupts = ; clocks = <&cpg CPG_MOD 703>; phys = <&usb2_phy0>; phy-names = "usb"; - companion = <&ohci0>; power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; resets = <&cpg 703>; status = "disabled"; }; - ohci0: usb@ee080000 { - compatible = "generic-ohci"; - reg = <0 0xee080000 0 0x100>; + ehci0: usb@ee080100 { + compatible = "generic-ehci"; + reg = <0 0xee080100 0 0x100>; interrupts = ; clocks = <&cpg CPG_MOD 703>; phys = <&usb2_phy0>; phy-names = "usb"; + companion = <&ohci0>; power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; resets = <&cpg 703>; status = "disabled"; @@ -692,6 +658,35 @@ status = "disabled"; }; + sdhi2: sd@ee140000 { + compatible = "renesas,sdhi-r8a77995", + "renesas,rcar-gen3-sdhi"; + reg = <0 0xee140000 0 0x2000>; + interrupts = ; + clocks = <&cpg CPG_MOD 312>; + max-frequency = <200000000>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 312>; + status = "disabled"; + }; + + gic: interrupt-controller@f1010000 { + compatible = "arm,gic-400"; + #interrupt-cells = <3>; + #address-cells = <0>; + interrupt-controller; + reg = <0x0 0xf1010000 0 0x1000>, + <0x0 0xf1020000 0 0x20000>, + <0x0 0xf1040000 0 0x20000>, + <0x0 0xf1060000 0 0x20000>; + interrupts = ; + clocks = <&cpg CPG_MOD 408>; + clock-names = "clk"; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 408>; + }; + vspbs: vsp@fe960000 { compatible = "renesas,vsp2"; reg = <0 0xfe960000 0 0x8000>; @@ -702,15 +697,6 @@ renesas,fcp = <&fcpvb0>; }; - fcpvb0: fcp@fe96f000 { - compatible = "renesas,fcpv"; - reg = <0 0xfe96f000 0 0x200>; - clocks = <&cpg CPG_MOD 607>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 607>; - iommus = <&ipmmu_vp0 5>; - }; - vspd0: vsp@fea20000 { compatible = "renesas,vsp2"; reg = <0 0xfea20000 0 0x8000>; @@ -721,15 +707,6 @@ renesas,fcp = <&fcpvd0>; }; - fcpvd0: fcp@fea27000 { - compatible = "renesas,fcpv"; - reg = <0 0xfea27000 0 0x200>; - clocks = <&cpg CPG_MOD 603>; - power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; - resets = <&cpg 603>; - iommus = <&ipmmu_vi0 8>; - }; - vspd1: vsp@fea28000 { compatible = "renesas,vsp2"; reg = <0 0xfea28000 0 0x8000>; @@ -740,6 +717,24 @@ renesas,fcp = <&fcpvd1>; }; + fcpvb0: fcp@fe96f000 { + compatible = "renesas,fcpv"; + reg = <0 0xfe96f000 0 0x200>; + clocks = <&cpg CPG_MOD 607>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 607>; + iommus = <&ipmmu_vp0 5>; + }; + + fcpvd0: fcp@fea27000 { + compatible = "renesas,fcpv"; + reg = <0 0xfea27000 0 0x200>; + clocks = <&cpg CPG_MOD 603>; + power-domains = <&sysc R8A77995_PD_ALWAYS_ON>; + resets = <&cpg 603>; + iommus = <&ipmmu_vi0 8>; + }; + fcpvd1: fcp@fea2f000 { compatible = "renesas,fcpv"; reg = <0 0xfea2f000 0 0x200>; @@ -783,6 +778,11 @@ }; }; }; + + prr: chipid@fff00044 { + compatible = "renesas,prr"; + reg = <0 0xfff00044 0 4>; + }; }; timer { diff --git a/arch/arm64/boot/dts/renesas/salvator-common.dtsi b/arch/arm64/boot/dts/renesas/salvator-common.dtsi index 2a7f36abd2dd..9256fbaaab7f 100644 --- a/arch/arm64/boot/dts/renesas/salvator-common.dtsi +++ b/arch/arm64/boot/dts/renesas/salvator-common.dtsi @@ -66,6 +66,29 @@ enable-gpios = <&gpio6 7 GPIO_ACTIVE_HIGH>; }; + cvbs-in { + compatible = "composite-video-connector"; + label = "CVBS IN"; + + port { + cvbs_con: endpoint { + remote-endpoint = <&adv7482_ain7>; + }; + }; + }; + + hdmi-in { + compatible = "hdmi-connector"; + label = "HDMI IN"; + type = "a"; + + port { + hdmi_in_con: endpoint { + remote-endpoint = <&adv7482_hdmi>; + }; + }; + }; + reg_1p8v: regulator0 { compatible = "regulator-fixed"; regulator-name = "fixed-1.8V"; @@ -93,20 +116,12 @@ regulator-always-on; }; - rsnd_ak4613: sound { - compatible = "simple-audio-card"; - - simple-audio-card,format = "left_j"; - simple-audio-card,bitclock-master = <&sndcpu>; - simple-audio-card,frame-master = <&sndcpu>; + sound_card: sound { + compatible = "audio-graph-card"; - sndcpu: simple-audio-card,cpu { - sound-dai = <&rcar_sound>; - }; + label = "rcar-sound"; - sndcodec: simple-audio-card,codec { - sound-dai = <&ak4613>; - }; + dais = <&rsnd_port0>; }; vbus0_usb2: regulator-vbus0-usb2 { @@ -268,6 +283,37 @@ }; }; +&csi20 { + status = "okay"; + + ports { + port@0 { + reg = <0>; + csi20_in: endpoint { + clock-lanes = <0>; + data-lanes = <1>; + remote-endpoint = <&adv7482_txb>; + }; + }; + }; +}; + +&csi40 { + status = "okay"; + + ports { + port@0 { + reg = <0>; + + csi40_in: endpoint { + clock-lanes = <0>; + data-lanes = <1 2 3 4>; + remote-endpoint = <&adv7482_txa>; + }; + }; + }; +}; + &du { pinctrl-0 = <&du_pins>; pinctrl-names = "default"; @@ -322,6 +368,12 @@ asahi-kasei,out4-single-end; asahi-kasei,out5-single-end; asahi-kasei,out6-single-end; + + port { + ak4613_endpoint: endpoint { + remote-endpoint = <&rsnd_endpoint0>; + }; + }; }; cs2000: clk_multiplier@4f { @@ -359,6 +411,55 @@ shunt-resistor-micro-ohms = <5000>; }; + + video-receiver@70 { + compatible = "adi,adv7482"; + reg = <0x70>; + + #address-cells = <1>; + #size-cells = <0>; + + interrupt-parent = <&gpio6>; + interrupt-names = "intrq1", "intrq2"; + interrupts = <30 IRQ_TYPE_LEVEL_LOW>, + <31 IRQ_TYPE_LEVEL_LOW>; + + port@7 { + reg = <7>; + + adv7482_ain7: endpoint { + remote-endpoint = <&cvbs_con>; + }; + }; + + port@8 { + reg = <8>; + + adv7482_hdmi: endpoint { + remote-endpoint = <&hdmi_in_con>; + }; + }; + + port@10 { + reg = <10>; + + adv7482_txa: endpoint { + clock-lanes = <0>; + data-lanes = <1 2 3 4>; + remote-endpoint = <&csi40_in>; + }; + }; + + port@11 { + reg = <11>; + + adv7482_txb: endpoint { + clock-lanes = <0>; + data-lanes = <1>; + remote-endpoint = <&csi20_in>; + }; + }; + }; }; &i2c_dvfs { @@ -376,6 +477,8 @@ #interrupt-cells = <2>; gpio-controller; #gpio-cells = <2>; + rohm,ddr-backup-power = <0xf>; + rohm,rstbmode-level; regulators { dvfs: dvfs { @@ -387,6 +490,12 @@ }; }; }; + + eeprom@50 { + compatible = "rohm,br24t01", "atmel,24c01"; + reg = <0x50>; + pagesize = <8>; + }; }; &ohci0 { @@ -416,12 +525,12 @@ avb_pins: avb { mux { - groups = "avb_link", "avb_mdc", "avb_mii"; + groups = "avb_link", "avb_mdio", "avb_mii"; function = "avb"; }; - pins_mdc { - groups = "avb_mdc"; + pins_mdio { + groups = "avb_mdio"; drive-strength = <24>; }; @@ -581,10 +690,18 @@ <&audio_clk_c>, <&cpg CPG_CORE CPG_AUDIO_CLK_I>; - rcar_sound,dai { - dai0 { - playback = <&ssi0 &src0 &dvc0>; - capture = <&ssi1 &src1 &dvc1>; + ports { + rsnd_port0: port@0 { + rsnd_endpoint0: endpoint { + remote-endpoint = <&ak4613_endpoint>; + + dai-format = "left_j"; + bitclock-master = <&rsnd_endpoint0>; + frame-master = <&rsnd_endpoint0>; + + playback = <&ssi0 &src0 &dvc0>; + capture = <&ssi1 &src1 &dvc1>; + }; }; }; }; @@ -689,6 +806,38 @@ clock-frequency = <100000000>; }; +&vin0 { + status = "okay"; +}; + +&vin1 { + status = "okay"; +}; + +&vin2 { + status = "okay"; +}; + +&vin3 { + status = "okay"; +}; + +&vin4 { + status = "okay"; +}; + +&vin5 { + status = "okay"; +}; + +&vin6 { + status = "okay"; +}; + +&vin7 { + status = "okay"; +}; + &wdt0 { timeout-sec = <60>; status = "okay"; diff --git a/arch/arm64/boot/dts/renesas/ulcb.dtsi b/arch/arm64/boot/dts/renesas/ulcb.dtsi index 6f814845f8b6..0edb16e6b372 100644 --- a/arch/arm64/boot/dts/renesas/ulcb.dtsi +++ b/arch/arm64/boot/dts/renesas/ulcb.dtsi @@ -243,6 +243,32 @@ &i2c_dvfs { status = "okay"; + + pmic: pmic@30 { + pinctrl-0 = <&irq0_pins>; + pinctrl-names = "default"; + + compatible = "rohm,bd9571mwv"; + reg = <0x30>; + interrupt-parent = <&intc_ex>; + interrupts = <0 IRQ_TYPE_LEVEL_LOW>; + interrupt-controller; + #interrupt-cells = <2>; + gpio-controller; + #gpio-cells = <2>; + rohm,ddr-backup-power = <0xf>; + rohm,rstbmode-pulse; + + regulators { + dvfs: dvfs { + regulator-name = "dvfs"; + regulator-min-microvolt = <750000>; + regulator-max-microvolt = <1030000>; + regulator-boot-on; + regulator-always-on; + }; + }; + }; }; &ohci1 { @@ -255,12 +281,12 @@ avb_pins: avb { mux { - groups = "avb_link", "avb_mdc", "avb_mii"; + groups = "avb_link", "avb_mdio", "avb_mii"; function = "avb"; }; - pins_mdc { - groups = "avb_mdc"; + pins_mdio { + groups = "avb_mdio"; drive-strength = <24>; }; @@ -276,6 +302,11 @@ function = "i2c2"; }; + irq0_pins: irq0 { + groups = "intc_ex_irq0"; + function = "intc_ex"; + }; + scif2_pins: scif2 { groups = "scif2_data_a"; function = "scif2"; diff --git a/arch/arm64/boot/dts/rockchip/rk3328.dtsi b/arch/arm64/boot/dts/rockchip/rk3328.dtsi index be2bfbc6b483..b8e9da15e00c 100644 --- a/arch/arm64/boot/dts/rockchip/rk3328.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3328.dtsi @@ -595,6 +595,8 @@ reg = <0x0 0xff330200 0 0x100>; interrupts = ; interrupt-names = "h265e_mmu"; + clocks = <&cru ACLK_H265>, <&cru PCLK_H265>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -604,6 +606,8 @@ reg = <0x0 0xff340800 0x0 0x40>; interrupts = ; interrupt-names = "vepu_mmu"; + clocks = <&cru ACLK_VPU>, <&cru HCLK_VPU>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -613,6 +617,8 @@ reg = <0x0 0xff350800 0x0 0x40>; interrupts = ; interrupt-names = "vpu_mmu"; + clocks = <&cru ACLK_VPU>, <&cru HCLK_VPU>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -622,6 +628,8 @@ reg = <0x0 0xff360480 0x0 0x40>, <0x0 0xff3604c0 0x0 0x40>; interrupts = ; interrupt-names = "rkvdec_mmu"; + clocks = <&cru ACLK_RKVDEC>, <&cru HCLK_RKVDEC>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -631,6 +639,8 @@ reg = <0x0 0xff373f00 0x0 0x100>; interrupts = ; interrupt-names = "vop_mmu"; + clocks = <&cru ACLK_VOP>, <&cru HCLK_VOP>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; diff --git a/arch/arm64/boot/dts/rockchip/rk3368.dtsi b/arch/arm64/boot/dts/rockchip/rk3368.dtsi index 03458ac44201..ad91ced78649 100644 --- a/arch/arm64/boot/dts/rockchip/rk3368.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3368.dtsi @@ -742,6 +742,8 @@ reg = <0x0 0xff900800 0x0 0x100>; interrupts = ; interrupt-names = "iep_mmu"; + clocks = <&cru ACLK_IEP>, <&cru HCLK_IEP>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -752,6 +754,8 @@ <0x0 0xff915000 0x0 0x100>; interrupts = ; interrupt-names = "isp_mmu"; + clocks = <&cru ACLK_ISP>, <&cru HCLK_ISP>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; rockchip,disable-mmu-reset; status = "disabled"; @@ -762,6 +766,8 @@ reg = <0x0 0xff930300 0x0 0x100>; interrupts = ; interrupt-names = "vop_mmu"; + clocks = <&cru ACLK_VOP>, <&cru HCLK_VOP>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -772,6 +778,8 @@ <0x0 0xff9a0480 0x0 0x40>; interrupts = ; interrupt-names = "hevc_mmu"; + clocks = <&cru ACLK_VIDEO>, <&cru HCLK_VIDEO>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -782,6 +790,8 @@ interrupts = , ; interrupt-names = "vepu_mmu", "vdpu_mmu"; + clocks = <&cru ACLK_VIDEO>, <&cru HCLK_VIDEO>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; diff --git a/arch/arm64/boot/dts/rockchip/rk3399-firefly.dts b/arch/arm64/boot/dts/rockchip/rk3399-firefly.dts index 4f28628aa091..2a352763c848 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-firefly.dts +++ b/arch/arm64/boot/dts/rockchip/rk3399-firefly.dts @@ -662,6 +662,14 @@ status = "okay"; }; +&tcphy0 { + status = "okay"; +}; + +&tcphy1 { + status = "okay"; +}; + &tsadc { /* tshut mode 0:CRU 1:GPIO */ rockchip,hw-tshut-mode = <1>; diff --git a/arch/arm64/boot/dts/rockchip/rk3399-gru-kevin.dts b/arch/arm64/boot/dts/rockchip/rk3399-gru-kevin.dts index 191a6bcb1704..82179125bfb7 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-gru-kevin.dts +++ b/arch/arm64/boot/dts/rockchip/rk3399-gru-kevin.dts @@ -255,7 +255,7 @@ ap_i2c_dig: &i2c2 { &ap_i2c_tp { trackpad@4a { - compatible = "atmel,atmel_mxt_tp"; + compatible = "atmel,maxtouch"; reg = <0x4a>; pinctrl-names = "default"; pinctrl-0 = <&trackpad_int_l>; @@ -271,7 +271,7 @@ ap_i2c_dig: &i2c2 { &ap_i2c_ts { touchscreen@4b { - compatible = "atmel,atmel_mxt_ts"; + compatible = "atmel,maxtouch"; reg = <0x4b>; pinctrl-names = "default"; pinctrl-0 = <&touch_int_l>; diff --git a/arch/arm64/boot/dts/rockchip/rk3399-gru.dtsi b/arch/arm64/boot/dts/rockchip/rk3399-gru.dtsi index 18f546f2dfd1..f49bfab75dd0 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-gru.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3399-gru.dtsi @@ -588,7 +588,9 @@ <&cru ACLK_PERILP0>, <&cru HCLK_PERILP0>, <&cru PCLK_PERILP0>, <&cru ACLK_CCI>, <&cru HCLK_PERILP1>, <&cru PCLK_PERILP1>, - <&cru ACLK_VIO>; + <&cru ACLK_VIO>, <&cru ACLK_HDCP>, + <&cru ACLK_GIC_PRE>, + <&cru PCLK_DDR>; assigned-clock-rates = <600000000>, <800000000>, <1000000000>, @@ -597,7 +599,9 @@ <100000000>, <100000000>, <50000000>, <800000000>, <100000000>, <50000000>, - <400000000>; + <400000000>, <400000000>, + <200000000>, + <200000000>; }; &emmc_phy { diff --git a/arch/arm64/boot/dts/rockchip/rk3399-puma-haikou.dts b/arch/arm64/boot/dts/rockchip/rk3399-puma-haikou.dts index 7d3e8bfd51dd..e0afdd8b62bd 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-puma-haikou.dts +++ b/arch/arm64/boot/dts/rockchip/rk3399-puma-haikou.dts @@ -143,6 +143,11 @@ }; }; +&hdmi { + ddc-i2c-bus = <&i2c3>; + status = "okay"; +}; + &i2c1 { status = "okay"; clock-frequency = <400000>; @@ -246,6 +251,10 @@ status = "okay"; }; +&tcphy0 { + status = "okay"; +}; + &u2phy0 { status = "okay"; }; @@ -281,3 +290,19 @@ &usb_host0_ohci { status = "okay"; }; + +&vopb { + status = "okay"; +}; + +&vopb_mmu { + status = "okay"; +}; + +&vopl { + status = "okay"; +}; + +&vopl_mmu { + status = "okay"; +}; diff --git a/arch/arm64/boot/dts/rockchip/rk3399-puma.dtsi b/arch/arm64/boot/dts/rockchip/rk3399-puma.dtsi index 4a2d06abe9c1..14a0f1998639 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-puma.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3399-puma.dtsi @@ -527,6 +527,10 @@ }; }; +&tcphy1 { + status = "okay"; +}; + &tsadc { rockchip,hw-tshut-mode = <1>; rockchip,hw-tshut-polarity = <1>; diff --git a/arch/arm64/boot/dts/rockchip/rk3399-sapphire-excavator.dts b/arch/arm64/boot/dts/rockchip/rk3399-sapphire-excavator.dts index 56952d1a3fb8..ad7548d3b93d 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-sapphire-excavator.dts +++ b/arch/arm64/boot/dts/rockchip/rk3399-sapphire-excavator.dts @@ -190,6 +190,18 @@ status = "okay"; }; +&pcie_phy { + status = "okay"; +}; + +&pcie0 { + ep-gpios = <&gpio2 RK_PA4 GPIO_ACTIVE_HIGH>; + num-lanes = <4>; + pinctrl-names = "default"; + pinctrl-0 = <&pcie_clkreqn_cpm>; + status = "okay"; +}; + &pinctrl { sdio-pwrseq { wifi_enable_h: wifi-enable-h { diff --git a/arch/arm64/boot/dts/rockchip/rk3399-sapphire.dtsi b/arch/arm64/boot/dts/rockchip/rk3399-sapphire.dtsi index e5daed7d2026..941b627094d7 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-sapphire.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3399-sapphire.dtsi @@ -471,21 +471,6 @@ gpio1830-supply = <&vcc_3v0>; }; -&pcie_phy { - status = "okay"; -}; - -&pcie0 { - assigned-clocks = <&cru SCLK_PCIEPHY_REF>; - assigned-clock-parents = <&cru SCLK_PCIEPHY_REF100M>; - assigned-clock-rates = <100000000>; - ep-gpios = <&gpio2 RK_PA4 GPIO_ACTIVE_HIGH>; - num-lanes = <4>; - pinctrl-names = "default"; - pinctrl-0 = <&pcie_clkreqn_cpm>; - status = "okay"; -}; - &pmu_io_domains { pmu1830-supply = <&vcc_3v0>; status = "okay"; @@ -560,6 +545,14 @@ status = "okay"; }; +&tcphy0 { + status = "okay"; +}; + +&tcphy1 { + status = "okay"; +}; + &tsadc { /* tshut mode 0:CRU 1:GPIO */ rockchip,hw-tshut-mode = <1>; diff --git a/arch/arm64/boot/dts/rockchip/rk3399.dtsi b/arch/arm64/boot/dts/rockchip/rk3399.dtsi index 4550c0f82be9..e0040b648f43 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3399.dtsi @@ -312,6 +312,8 @@ reg = <0x0 0xfe320000 0x0 0x4000>; interrupts = ; max-frequency = <150000000>; + assigned-clocks = <&cru HCLK_SD>; + assigned-clock-rates = <200000000>; clocks = <&cru HCLK_SDMMC>, <&cru SCLK_SDMMC>, <&cru SCLK_SDMMC_DRV>, <&cru SCLK_SDMMC_SAMPLE>; clock-names = "biu", "ciu", "ciu-drive", "ciu-sample"; @@ -411,8 +413,8 @@ reg = <0x0 0xfe800000 0x0 0x100000>; interrupts = ; dr_mode = "otg"; - phys = <&u2phy0_otg>; - phy-names = "usb2-phy"; + phys = <&u2phy0_otg>, <&tcphy0_usb3>; + phy-names = "usb2-phy", "usb3-phy"; phy_type = "utmi_wide"; snps,dis_enblslpm_quirk; snps,dis-u2-freeclk-exists-quirk; @@ -444,8 +446,8 @@ reg = <0x0 0xfe900000 0x0 0x100000>; interrupts = ; dr_mode = "otg"; - phys = <&u2phy1_otg>; - phy-names = "usb2-phy"; + phys = <&u2phy1_otg>, <&tcphy1_usb3>; + phy-names = "usb2-phy", "usb3-phy"; phy_type = "utmi_wide"; snps,dis_enblslpm_quirk; snps,dis-u2-freeclk-exists-quirk; @@ -461,8 +463,8 @@ compatible = "rockchip,rk3399-cdn-dp"; reg = <0x0 0xfec00000 0x0 0x100000>; interrupts = ; - assigned-clocks = <&cru SCLK_DP_CORE>; - assigned-clock-rates = <100000000>; + assigned-clocks = <&cru SCLK_DP_CORE>, <&cru SCLK_SPDIF_REC_DPTX>; + assigned-clock-rates = <100000000>, <200000000>; clocks = <&cru SCLK_DP_CORE>, <&cru PCLK_DP_CTRL>, <&cru SCLK_SPDIF_REC_DPTX>, <&cru PCLK_VIO_GRF>; clock-names = "core-clk", "pclk", "spdif", "grf"; @@ -1234,6 +1236,8 @@ reg = <0x0 0xff650800 0x0 0x40>; interrupts = ; interrupt-names = "vpu_mmu"; + clocks = <&cru ACLK_VCODEC>, <&cru HCLK_VCODEC>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -1243,6 +1247,8 @@ reg = <0x0 0xff660480 0x0 0x40>, <0x0 0xff6604c0 0x0 0x40>; interrupts = ; interrupt-names = "vdec_mmu"; + clocks = <&cru ACLK_VDU>, <&cru HCLK_VDU>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -1252,6 +1258,8 @@ reg = <0x0 0xff670800 0x0 0x40>; interrupts = ; interrupt-names = "iep_mmu"; + clocks = <&cru ACLK_IEP>, <&cru HCLK_IEP>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; status = "disabled"; }; @@ -1323,7 +1331,9 @@ <&cru ACLK_PERILP0>, <&cru HCLK_PERILP0>, <&cru PCLK_PERILP0>, <&cru ACLK_CCI>, <&cru HCLK_PERILP1>, <&cru PCLK_PERILP1>, - <&cru ACLK_VIO>; + <&cru ACLK_VIO>, <&cru ACLK_HDCP>, + <&cru ACLK_GIC_PRE>, + <&cru PCLK_DDR>; assigned-clock-rates = <594000000>, <800000000>, <1000000000>, @@ -1332,7 +1342,9 @@ <100000000>, <100000000>, <50000000>, <600000000>, <100000000>, <50000000>, - <400000000>; + <400000000>, <400000000>, + <200000000>, + <200000000>; }; grf: syscon@ff770000 { @@ -1599,7 +1611,7 @@ interrupts = ; interrupt-names = "vopl_mmu"; clocks = <&cru ACLK_VOP1>, <&cru HCLK_VOP1>; - clock-names = "aclk", "hclk"; + clock-names = "aclk", "iface"; power-domains = <&power RK3399_PD_VOPL>; #iommu-cells = <0>; status = "disabled"; @@ -1656,7 +1668,7 @@ interrupts = ; interrupt-names = "vopb_mmu"; clocks = <&cru ACLK_VOP0>, <&cru HCLK_VOP0>; - clock-names = "aclk", "hclk"; + clock-names = "aclk", "iface"; power-domains = <&power RK3399_PD_VOPB>; #iommu-cells = <0>; status = "disabled"; @@ -1667,6 +1679,8 @@ reg = <0x0 0xff914000 0x0 0x100>, <0x0 0xff915000 0x0 0x100>; interrupts = ; interrupt-names = "isp0_mmu"; + clocks = <&cru ACLK_ISP0_NOC>, <&cru HCLK_ISP0_NOC>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; rockchip,disable-mmu-reset; status = "disabled"; @@ -1677,6 +1691,8 @@ reg = <0x0 0xff924000 0x0 0x100>, <0x0 0xff925000 0x0 0x100>; interrupts = ; interrupt-names = "isp1_mmu"; + clocks = <&cru ACLK_ISP1_NOC>, <&cru HCLK_ISP1_NOC>; + clock-names = "aclk", "iface"; #iommu-cells = <0>; rockchip,disable-mmu-reset; status = "disabled"; diff --git a/arch/arm64/boot/dts/socionext/uniphier-ld11.dtsi b/arch/arm64/boot/dts/socionext/uniphier-ld11.dtsi index c32dd3419c87..d63b56e944de 100644 --- a/arch/arm64/boot/dts/socionext/uniphier-ld11.dtsi +++ b/arch/arm64/boot/dts/socionext/uniphier-ld11.dtsi @@ -549,10 +549,13 @@ status = "disabled"; reg = <0x65000000 0x8500>; interrupts = <0 66 4>; + clock-names = "ether"; clocks = <&sys_clk 6>; + reset-names = "ether"; resets = <&sys_rst 6>; - phy-mode = "rmii"; + phy-mode = "internal"; local-mac-address = [00 00 00 00 00 00]; + socionext,syscon-phy-mode = <&soc_glue 0>; mdio: mdio { #address-cells = <1>; diff --git a/arch/arm64/boot/dts/socionext/uniphier-ld20.dtsi b/arch/arm64/boot/dts/socionext/uniphier-ld20.dtsi index 3a5ed789c056..0298bd0d0e1a 100644 --- a/arch/arm64/boot/dts/socionext/uniphier-ld20.dtsi +++ b/arch/arm64/boot/dts/socionext/uniphier-ld20.dtsi @@ -604,10 +604,13 @@ interrupts = <0 66 4>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_ether_rgmii>; + clock-names = "ether"; clocks = <&sys_clk 6>; + reset-names = "ether"; resets = <&sys_rst 6>; phy-mode = "rgmii"; local-mac-address = [00 00 00 00 00 00]; + socionext,syscon-phy-mode = <&soc_glue 0>; mdio: mdio { #address-cells = <1>; diff --git a/arch/arm64/boot/dts/socionext/uniphier-pxs3.dtsi b/arch/arm64/boot/dts/socionext/uniphier-pxs3.dtsi index e85d6ddea3c2..2a4cf427f5d3 100644 --- a/arch/arm64/boot/dts/socionext/uniphier-pxs3.dtsi +++ b/arch/arm64/boot/dts/socionext/uniphier-pxs3.dtsi @@ -341,7 +341,7 @@ cdns,phy-dll-delay-sdclk-hsmmc = <21>; }; - soc-glue@5f800000 { + soc_glue: soc-glue@5f800000 { compatible = "socionext,uniphier-pxs3-soc-glue", "simple-mfd", "syscon"; reg = <0x5f800000 0x2000>; @@ -412,10 +412,13 @@ interrupts = <0 66 4>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_ether_rgmii>; + clock-names = "ether"; clocks = <&sys_clk 6>; + reset-names = "ether"; resets = <&sys_rst 6>; phy-mode = "rgmii"; local-mac-address = [00 00 00 00 00 00]; + socionext,syscon-phy-mode = <&soc_glue 0>; mdio0: mdio { #address-cells = <1>; @@ -430,10 +433,13 @@ interrupts = <0 67 4>; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_ether1_rgmii>; + clock-names = "ether"; clocks = <&sys_clk 7>; + reset-names = "ether"; resets = <&sys_rst 7>; phy-mode = "rgmii"; local-mac-address = [00 00 00 00 00 00]; + socionext,syscon-phy-mode = <&soc_glue 1>; mdio1: mdio { #address-cells = <1>; diff --git a/arch/arm64/boot/dts/sprd/sc2731.dtsi b/arch/arm64/boot/dts/sprd/sc2731.dtsi index 4331006185bf..98d3b4fdb9ad 100644 --- a/arch/arm64/boot/dts/sprd/sc2731.dtsi +++ b/arch/arm64/boot/dts/sprd/sc2731.dtsi @@ -24,6 +24,17 @@ interrupts = <2 IRQ_TYPE_LEVEL_HIGH>; }; + pmic_eic: gpio@300 { + compatible = "sprd,sc27xx-eic"; + reg = <0x300>; + interrupt-parent = <&sc2731_pmic>; + interrupts = <5 IRQ_TYPE_LEVEL_HIGH>; + gpio-controller; + #gpio-cells = <2>; + interrupt-controller; + #interrupt-cells = <2>; + }; + regulators { compatible = "sprd,sc27xx-regulator"; diff --git a/arch/arm64/boot/dts/sprd/sc9860.dtsi b/arch/arm64/boot/dts/sprd/sc9860.dtsi index 5dbfb796d9f9..3f5160d2f130 100644 --- a/arch/arm64/boot/dts/sprd/sc9860.dtsi +++ b/arch/arm64/boot/dts/sprd/sc9860.dtsi @@ -7,6 +7,8 @@ */ #include +#include +#include #include "whale2.dtsi" / { @@ -326,7 +328,7 @@ reg = <4>; soc_funnel_in_port1: endpoint { slave-mode; - remote-endpioint = + remote-endpoint = <&stm_out_port>; }; }; @@ -679,5 +681,33 @@ }; }; }; + + gpio-keys { + compatible = "gpio-keys"; + + key-volumedown { + label = "Volume Down Key"; + linux,code = ; + gpios = <&eic_debounce 2 GPIO_ACTIVE_LOW>; + debounce-interval = <2>; + wakeup-source; + }; + + key-volumeup { + label = "Volume Up Key"; + linux,code = ; + gpios = <&pmic_eic 10 GPIO_ACTIVE_HIGH>; + debounce-interval = <2>; + wakeup-source; + }; + + key-power { + label = "Power Key"; + linux,code = ; + gpios = <&pmic_eic 1 GPIO_ACTIVE_HIGH>; + debounce-interval = <2>; + wakeup-source; + }; + }; }; }; diff --git a/arch/arm64/boot/dts/sprd/whale2.dtsi b/arch/arm64/boot/dts/sprd/whale2.dtsi index 66a881e6da92..e9db9108f3c0 100644 --- a/arch/arm64/boot/dts/sprd/whale2.dtsi +++ b/arch/arm64/boot/dts/sprd/whale2.dtsi @@ -154,6 +154,56 @@ clocks = <&aon_gate CLK_SPLK_EB>; }; + eic_debounce: gpio@40210000 { + compatible = "sprd,sc9860-eic-debounce"; + reg = <0 0x40210000 0 0x80>; + gpio-controller; + #gpio-cells = <2>; + interrupt-controller; + #interrupt-cells = <2>; + interrupts = ; + }; + + eic_latch: gpio@40210080 { + compatible = "sprd,sc9860-eic-latch"; + reg = <0 0x40210080 0 0x20>; + gpio-controller; + #gpio-cells = <2>; + interrupt-controller; + #interrupt-cells = <2>; + interrupts = ; + }; + + eic_async: gpio@402100a0 { + compatible = "sprd,sc9860-eic-async"; + reg = <0 0x402100a0 0 0x20>; + gpio-controller; + #gpio-cells = <2>; + interrupt-controller; + #interrupt-cells = <2>; + interrupts = ; + }; + + eic_sync: gpio@402100c0 { + compatible = "sprd,sc9860-eic-sync"; + reg = <0 0x402100c0 0 0x20>; + gpio-controller; + #gpio-cells = <2>; + interrupt-controller; + #interrupt-cells = <2>; + interrupts = ; + }; + + ap_gpio: gpio@40280000 { + compatible = "sprd,sc9860-gpio"; + reg = <0 0x40280000 0 0x1000>; + gpio-controller; + #gpio-cells = <2>; + interrupt-controller; + #interrupt-cells = <2>; + interrupts = ; + }; + pin_controller: pinctrl@402a0000 { compatible = "sprd,sc9860-pinctrl"; reg = <0 0x402a0000 0 0x10000>; @@ -164,8 +214,9 @@ reg = <0 0x40310000 0 0x1000>; interrupts = ; timeout-sec = <12>; - clock-names = "enable"; - clocks = <&aon_gate CLK_APCPU_WDG_EB>; + clock-names = "enable", "rtc_enable"; + clocks = <&aon_gate CLK_APCPU_WDG_EB>, + <&aon_gate CLK_AP_WDG_RTC_EB>; }; }; diff --git a/arch/arm64/boot/dts/synaptics/Makefile b/arch/arm64/boot/dts/synaptics/Makefile new file mode 100644 index 000000000000..de71ddda6835 --- /dev/null +++ b/arch/arm64/boot/dts/synaptics/Makefile @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0 +# Berlin SoC Family +dtb-$(CONFIG_ARCH_BERLIN) += berlin4ct-dmp.dtb +dtb-$(CONFIG_ARCH_BERLIN) += berlin4ct-stb.dtb diff --git a/arch/arm64/boot/dts/synaptics/berlin4ct-dmp.dts b/arch/arm64/boot/dts/synaptics/berlin4ct-dmp.dts new file mode 100644 index 000000000000..c64a179ebbb7 --- /dev/null +++ b/arch/arm64/boot/dts/synaptics/berlin4ct-dmp.dts @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) +/* + * Copyright (C) 2015 Marvell Technology Group Ltd. + * + * Author: Jisheng Zhang + */ + +/dts-v1/; + +#include "berlin4ct.dtsi" + +/ { + model = "Marvell BG4CT DMP board"; + compatible = "marvell,berlin4ct-dmp", "marvell,berlin4ct", "marvell,berlin"; + + chosen { + stdout-path = "serial0:115200n8"; + }; + + memory@1000000 { + device_type = "memory"; + /* the first 16MB is for firmwares' usage */ + reg = <0 0x01000000 0 0x7f000000>; + }; +}; + +&uart0 { + status = "okay"; +}; diff --git a/arch/arm64/boot/dts/synaptics/berlin4ct-stb.dts b/arch/arm64/boot/dts/synaptics/berlin4ct-stb.dts new file mode 100644 index 000000000000..277dccfa05cb --- /dev/null +++ b/arch/arm64/boot/dts/synaptics/berlin4ct-stb.dts @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) +/* + * Copyright (C) 2015 Marvell Technology Group Ltd. + * + * Author: Jisheng Zhang + */ + +/dts-v1/; + +#include "berlin4ct.dtsi" + +/ { + model = "Marvell BG4CT STB board"; + compatible = "marvell,berlin4ct-stb", "marvell,berlin4ct", "marvell,berlin"; + + chosen { + stdout-path = "serial0:115200n8"; + }; + + memory@1000000 { + device_type = "memory"; + /* the first 16MB is for firmwares' usage */ + reg = <0 0x01000000 0 0x7f000000>; + }; +}; + +&uart0 { + status = "okay"; +}; diff --git a/arch/arm64/boot/dts/marvell/berlin4ct.dtsi b/arch/arm64/boot/dts/synaptics/berlin4ct.dtsi similarity index 78% rename from arch/arm64/boot/dts/marvell/berlin4ct.dtsi rename to arch/arm64/boot/dts/synaptics/berlin4ct.dtsi index d2f88b92d8e2..216767e2edf6 100644 --- a/arch/arm64/boot/dts/marvell/berlin4ct.dtsi +++ b/arch/arm64/boot/dts/synaptics/berlin4ct.dtsi @@ -1,45 +1,8 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) /* * Copyright (C) 2015 Marvell Technology Group Ltd. * * Author: Jisheng Zhang - * - * This file is dual-licensed: you can use it either under the terms - * of the GPLv2 or the X11 license, at your option. Note that this dual - * licensing only applies to this file, and not this project as a - * whole. - * - * a) This library is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License as - * published by the Free Software Foundation; either version 2 of the - * License, or (at your option) any later version. - * - * This library is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * Or, alternatively, - * - * b) Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or - * sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. */ #include diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index ecf613761e78..3cfa8ca26738 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -53,6 +53,7 @@ CONFIG_ARCH_R8A7796=y CONFIG_ARCH_R8A77965=y CONFIG_ARCH_R8A77970=y CONFIG_ARCH_R8A77980=y +CONFIG_ARCH_R8A77990=y CONFIG_ARCH_R8A77995=y CONFIG_ARCH_STRATIX10=y CONFIG_ARCH_TEGRA=y @@ -75,10 +76,12 @@ CONFIG_PCI_HISI=y CONFIG_PCIE_QCOM=y CONFIG_PCIE_KIRIN=y CONFIG_PCIE_ARMADA_8K=y +CONFIG_PCIE_HISI_STB=y CONFIG_PCI_AARDVARK=y CONFIG_PCI_TEGRA=y CONFIG_PCIE_RCAR=y -CONFIG_PCIE_ROCKCHIP=m +CONFIG_PCIE_ROCKCHIP=y +CONFIG_PCIE_ROCKCHIP_HOST=m CONFIG_PCI_HOST_GENERIC=y CONFIG_PCI_XGENE=y CONFIG_PCI_HOST_THUNDER_PEM=y @@ -157,8 +160,10 @@ CONFIG_BT_HIDP=m # CONFIG_BT_LE is not set CONFIG_BT_LEDS=y # CONFIG_BT_DEBUGFS is not set +CONFIG_BT_HCIBTUSB=m CONFIG_BT_HCIUART=m CONFIG_BT_HCIUART_LL=y +CONFIG_BT_HCIUART_BCM=y CONFIG_CFG80211=m CONFIG_MAC80211=m CONFIG_MAC80211_LEDS=y @@ -169,6 +174,9 @@ CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug" CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y CONFIG_DMA_CMA=y +CONFIG_CMA_SIZE_MBYTES=32 +CONFIG_HISILICON_LPC=y +CONFIG_SIMPLE_PM_BUS=y CONFIG_MTD=y CONFIG_MTD_BLOCK=y CONFIG_MTD_M25P80=y @@ -187,6 +195,9 @@ CONFIG_BLK_DEV_SD=y CONFIG_SCSI_SAS_ATA=y CONFIG_SCSI_HISI_SAS=y CONFIG_SCSI_HISI_SAS_PCI=y +CONFIG_SCSI_UFSHCD=m +CONFIG_SCSI_UFSHCD_PLATFORM=m +CONFIG_SCSI_UFS_QCOM=m CONFIG_ATA=y CONFIG_SATA_AHCI=y CONFIG_SATA_AHCI_PLATFORM=y @@ -206,8 +217,10 @@ CONFIG_VETH=m CONFIG_VIRTIO_NET=y CONFIG_AMD_XGBE=y CONFIG_NET_XGENE=y +CONFIG_ATL1C=m CONFIG_MACB=y CONFIG_THUNDER_NIC_PF=y +CONFIG_HIX5HD2_GMAC=y CONFIG_HNS_DSAF=y CONFIG_HNS_ENET=y CONFIG_E1000E=y @@ -239,6 +252,7 @@ CONFIG_ROCKCHIP_PHY=y CONFIG_USB_PEGASUS=m CONFIG_USB_RTL8150=m CONFIG_USB_RTL8152=m +CONFIG_USB_LAN78XX=m CONFIG_USB_USBNET=m CONFIG_USB_NET_DM9601=m CONFIG_USB_NET_SR9800=m @@ -246,13 +260,19 @@ CONFIG_USB_NET_SMSC75XX=m CONFIG_USB_NET_SMSC95XX=m CONFIG_USB_NET_PLUSB=m CONFIG_USB_NET_MCS7830=m +CONFIG_ATH10K=m +CONFIG_ATH10K_PCI=m CONFIG_BRCMFMAC=m +CONFIG_MWIFIEX=m +CONFIG_MWIFIEX_PCIE=m CONFIG_WL18XX=m CONFIG_WLCORE_SDIO=m CONFIG_INPUT_EVDEV=y CONFIG_KEYBOARD_ADC=m CONFIG_KEYBOARD_CROS_EC=y CONFIG_KEYBOARD_GPIO=y +CONFIG_INPUT_TOUCHSCREEN=y +CONFIG_TOUCHSCREEN_ATMEL_MXT=m CONFIG_INPUT_MISC=y CONFIG_INPUT_PM8941_PWRKEY=y CONFIG_INPUT_HISI_POWERKEY=y @@ -286,6 +306,7 @@ CONFIG_SERIAL_MVEBU_UART=y CONFIG_SERIAL_DEV_BUS=y CONFIG_SERIAL_DEV_CTRL_TTYPORT=y CONFIG_VIRTIO_CONSOLE=y +CONFIG_I2C_HID=m CONFIG_I2C_CHARDEV=y CONFIG_I2C_MUX=y CONFIG_I2C_MUX_PCA954x=y @@ -303,6 +324,7 @@ CONFIG_I2C_UNIPHIER_F=y CONFIG_I2C_RCAR=y CONFIG_I2C_CROS_EC_TUNNEL=y CONFIG_SPI=y +CONFIG_SPI_ARMADA_3700=y CONFIG_SPI_MESON_SPICC=m CONFIG_SPI_MESON_SPIFC=m CONFIG_SPI_BCM2835=m @@ -320,6 +342,7 @@ CONFIG_PINCTRL_MAX77620=y CONFIG_PINCTRL_MSM8916=y CONFIG_PINCTRL_MSM8994=y CONFIG_PINCTRL_MSM8996=y +CONFIG_PINCTRL_MT7622=y CONFIG_PINCTRL_QDF2XXX=y CONFIG_PINCTRL_QCOM_SPMI_PMIC=y CONFIG_GPIO_DWAPB=y @@ -332,6 +355,8 @@ CONFIG_GPIO_XGENE_SB=y CONFIG_GPIO_PCA953X=y CONFIG_GPIO_PCA953X_IRQ=y CONFIG_GPIO_MAX77620=y +CONFIG_POWER_AVS=y +CONFIG_ROCKCHIP_IODOMAIN=y CONFIG_POWER_RESET_MSM=y CONFIG_POWER_RESET_XGENE=y CONFIG_POWER_RESET_SYSCON=y @@ -343,6 +368,7 @@ CONFIG_SENSORS_INA2XX=m CONFIG_THERMAL_GOV_POWER_ALLOCATOR=y CONFIG_CPU_THERMAL=y CONFIG_THERMAL_EMULATION=y +CONFIG_ARMADA_THERMAL=y CONFIG_BRCMSTB_THERMAL=m CONFIG_EXYNOS_THERMAL=y CONFIG_RCAR_GEN3_THERMAL=y @@ -361,6 +387,7 @@ CONFIG_MFD_AXP20X_RSB=y CONFIG_MFD_CROS_EC=y CONFIG_MFD_CROS_EC_I2C=y CONFIG_MFD_CROS_EC_SPI=y +CONFIG_MFD_CROS_EC_CHARDEV=m CONFIG_MFD_EXYNOS_LPASS=m CONFIG_MFD_HI6421_PMIC=y CONFIG_MFD_HI655X_PMIC=y @@ -439,7 +466,8 @@ CONFIG_SND_BCM2835_SOC_I2S=m CONFIG_SND_SOC_SAMSUNG=y CONFIG_SND_SOC_RCAR=m CONFIG_SND_SOC_AK4613=m -CONFIG_SND_SIMPLE_CARD=y +CONFIG_SND_SIMPLE_CARD=m +CONFIG_SND_AUDIO_GRAPH_CARD=m CONFIG_USB=y CONFIG_USB_OTG=y CONFIG_USB_XHCI_HCD=y @@ -485,6 +513,7 @@ CONFIG_MMC_SPI=y CONFIG_MMC_SDHI=y CONFIG_MMC_DW=y CONFIG_MMC_DW_EXYNOS=y +CONFIG_MMC_DW_HI3798CV200=y CONFIG_MMC_DW_K3=y CONFIG_MMC_DW_ROCKCHIP=y CONFIG_MMC_SUNXI=y @@ -514,6 +543,7 @@ CONFIG_RTC_DRV_SUN6I=y CONFIG_RTC_DRV_ARMADA38X=y CONFIG_RTC_DRV_TEGRA=y CONFIG_RTC_DRV_XGENE=y +CONFIG_RTC_DRV_CROS_EC=y CONFIG_DMADEVICES=y CONFIG_DMA_BCM2835=m CONFIG_K3_DMA=y @@ -556,6 +586,7 @@ CONFIG_TEGRA_IOMMU_SMMU=y CONFIG_ARM_SMMU=y CONFIG_ARM_SMMU_V3=y CONFIG_QCOM_IOMMU=y +CONFIG_RPMSG_QCOM_GLINK_RPM=y CONFIG_RPMSG_QCOM_SMD=y CONFIG_RASPBERRYPI_POWER=y CONFIG_QCOM_SMEM=y @@ -567,12 +598,18 @@ CONFIG_ARCH_TEGRA_132_SOC=y CONFIG_ARCH_TEGRA_210_SOC=y CONFIG_ARCH_TEGRA_186_SOC=y CONFIG_ARCH_TEGRA_194_SOC=y +CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y CONFIG_EXTCON_USB_GPIO=y +CONFIG_EXTCON_USBC_CROS_EC=y CONFIG_MEMORY=y CONFIG_TEGRA_MC=y CONFIG_IIO=y CONFIG_EXYNOS_ADC=y CONFIG_ROCKCHIP_SARADC=m +CONFIG_IIO_CROS_EC_SENSORS_CORE=m +CONFIG_IIO_CROS_EC_SENSORS=m +CONFIG_IIO_CROS_EC_LIGHT_PROX=m +CONFIG_IIO_CROS_EC_BARO=m CONFIG_PWM=y CONFIG_PWM_BCM2835=m CONFIG_PWM_CROS_EC=m @@ -581,21 +618,26 @@ CONFIG_PWM_RCAR=m CONFIG_PWM_ROCKCHIP=y CONFIG_PWM_SAMSUNG=y CONFIG_PWM_TEGRA=m +CONFIG_PHY_HISTB_COMBPHY=y +CONFIG_PHY_HISI_INNO_USB2=y CONFIG_PHY_RCAR_GEN3_USB2=y CONFIG_PHY_RCAR_GEN3_USB3=m CONFIG_PHY_HI6220_USB=y CONFIG_PHY_QCOM_USB_HS=y CONFIG_PHY_SUN4I_USB=y CONFIG_PHY_MVEBU_CP110_COMPHY=y +CONFIG_PHY_QCOM_QMP=m CONFIG_PHY_ROCKCHIP_INNO_USB2=y CONFIG_PHY_ROCKCHIP_EMMC=y CONFIG_PHY_ROCKCHIP_PCIE=m +CONFIG_PHY_ROCKCHIP_TYPEC=y CONFIG_PHY_XGENE=y CONFIG_PHY_TEGRA_XUSB=y CONFIG_QCOM_L2_PMU=y CONFIG_QCOM_L3_PMU=y CONFIG_MESON_EFUSE=m CONFIG_QCOM_QFPROM=y +CONFIG_ROCKCHIP_EFUSE=y CONFIG_UNIPHIER_EFUSE=y CONFIG_TEE=y CONFIG_OPTEE=y diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig index cb5a243110c4..e3fdb0fd6f70 100644 --- a/arch/arm64/crypto/Kconfig +++ b/arch/arm64/crypto/Kconfig @@ -47,6 +47,12 @@ config CRYPTO_SM3_ARM64_CE select CRYPTO_HASH select CRYPTO_SM3 +config CRYPTO_SM4_ARM64_CE + tristate "SM4 symmetric cipher (ARMv8.2 Crypto Extensions)" + depends on KERNEL_MODE_NEON + select CRYPTO_ALGAPI + select CRYPTO_SM4 + config CRYPTO_GHASH_ARM64_CE tristate "GHASH/AES-GCM using ARMv8 Crypto Extensions" depends on KERNEL_MODE_NEON diff --git a/arch/arm64/crypto/Makefile b/arch/arm64/crypto/Makefile index f35ac684b1c0..bcafd016618e 100644 --- a/arch/arm64/crypto/Makefile +++ b/arch/arm64/crypto/Makefile @@ -23,6 +23,9 @@ sha3-ce-y := sha3-ce-glue.o sha3-ce-core.o obj-$(CONFIG_CRYPTO_SM3_ARM64_CE) += sm3-ce.o sm3-ce-y := sm3-ce-glue.o sm3-ce-core.o +obj-$(CONFIG_CRYPTO_SM4_ARM64_CE) += sm4-ce.o +sm4-ce-y := sm4-ce-glue.o sm4-ce-core.o + obj-$(CONFIG_CRYPTO_GHASH_ARM64_CE) += ghash-ce.o ghash-ce-y := ghash-ce-glue.o ghash-ce-core.o diff --git a/arch/arm64/crypto/aes-ce-ccm-core.S b/arch/arm64/crypto/aes-ce-ccm-core.S index e3a375c4cb83..88f5aef7934c 100644 --- a/arch/arm64/crypto/aes-ce-ccm-core.S +++ b/arch/arm64/crypto/aes-ce-ccm-core.S @@ -19,24 +19,33 @@ * u32 *macp, u8 const rk[], u32 rounds); */ ENTRY(ce_aes_ccm_auth_data) - ldr w8, [x3] /* leftover from prev round? */ + frame_push 7 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x5 + + ldr w25, [x22] /* leftover from prev round? */ ld1 {v0.16b}, [x0] /* load mac */ - cbz w8, 1f - sub w8, w8, #16 + cbz w25, 1f + sub w25, w25, #16 eor v1.16b, v1.16b, v1.16b -0: ldrb w7, [x1], #1 /* get 1 byte of input */ - subs w2, w2, #1 - add w8, w8, #1 +0: ldrb w7, [x20], #1 /* get 1 byte of input */ + subs w21, w21, #1 + add w25, w25, #1 ins v1.b[0], w7 ext v1.16b, v1.16b, v1.16b, #1 /* rotate in the input bytes */ beq 8f /* out of input? */ - cbnz w8, 0b + cbnz w25, 0b eor v0.16b, v0.16b, v1.16b -1: ld1 {v3.4s}, [x4] /* load first round key */ - prfm pldl1strm, [x1] - cmp w5, #12 /* which key size? */ - add x6, x4, #16 - sub w7, w5, #2 /* modified # of rounds */ +1: ld1 {v3.4s}, [x23] /* load first round key */ + prfm pldl1strm, [x20] + cmp w24, #12 /* which key size? */ + add x6, x23, #16 + sub w7, w24, #2 /* modified # of rounds */ bmi 2f bne 5f mov v5.16b, v3.16b @@ -55,33 +64,43 @@ ENTRY(ce_aes_ccm_auth_data) ld1 {v5.4s}, [x6], #16 /* load next round key */ bpl 3b aese v0.16b, v4.16b - subs w2, w2, #16 /* last data? */ + subs w21, w21, #16 /* last data? */ eor v0.16b, v0.16b, v5.16b /* final round */ bmi 6f - ld1 {v1.16b}, [x1], #16 /* load next input block */ + ld1 {v1.16b}, [x20], #16 /* load next input block */ eor v0.16b, v0.16b, v1.16b /* xor with mac */ - bne 1b -6: st1 {v0.16b}, [x0] /* store mac */ + beq 6f + + if_will_cond_yield_neon + st1 {v0.16b}, [x19] /* store mac */ + do_cond_yield_neon + ld1 {v0.16b}, [x19] /* reload mac */ + endif_yield_neon + + b 1b +6: st1 {v0.16b}, [x19] /* store mac */ beq 10f - adds w2, w2, #16 + adds w21, w21, #16 beq 10f - mov w8, w2 -7: ldrb w7, [x1], #1 + mov w25, w21 +7: ldrb w7, [x20], #1 umov w6, v0.b[0] eor w6, w6, w7 - strb w6, [x0], #1 - subs w2, w2, #1 + strb w6, [x19], #1 + subs w21, w21, #1 beq 10f ext v0.16b, v0.16b, v0.16b, #1 /* rotate out the mac bytes */ b 7b -8: mov w7, w8 - add w8, w8, #16 +8: mov w7, w25 + add w25, w25, #16 9: ext v1.16b, v1.16b, v1.16b, #1 adds w7, w7, #1 bne 9b eor v0.16b, v0.16b, v1.16b - st1 {v0.16b}, [x0] -10: str w8, [x3] + st1 {v0.16b}, [x19] +10: str w25, [x22] + + frame_pop ret ENDPROC(ce_aes_ccm_auth_data) @@ -126,19 +145,29 @@ ENTRY(ce_aes_ccm_final) ENDPROC(ce_aes_ccm_final) .macro aes_ccm_do_crypt,enc - ldr x8, [x6, #8] /* load lower ctr */ - ld1 {v0.16b}, [x5] /* load mac */ -CPU_LE( rev x8, x8 ) /* keep swabbed ctr in reg */ + frame_push 8 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x5 + mov x25, x6 + + ldr x26, [x25, #8] /* load lower ctr */ + ld1 {v0.16b}, [x24] /* load mac */ +CPU_LE( rev x26, x26 ) /* keep swabbed ctr in reg */ 0: /* outer loop */ - ld1 {v1.8b}, [x6] /* load upper ctr */ - prfm pldl1strm, [x1] - add x8, x8, #1 - rev x9, x8 - cmp w4, #12 /* which key size? */ - sub w7, w4, #2 /* get modified # of rounds */ + ld1 {v1.8b}, [x25] /* load upper ctr */ + prfm pldl1strm, [x20] + add x26, x26, #1 + rev x9, x26 + cmp w23, #12 /* which key size? */ + sub w7, w23, #2 /* get modified # of rounds */ ins v1.d[1], x9 /* no carry in lower ctr */ - ld1 {v3.4s}, [x3] /* load first round key */ - add x10, x3, #16 + ld1 {v3.4s}, [x22] /* load first round key */ + add x10, x22, #16 bmi 1f bne 4f mov v5.16b, v3.16b @@ -165,9 +194,9 @@ CPU_LE( rev x8, x8 ) /* keep swabbed ctr in reg */ bpl 2b aese v0.16b, v4.16b aese v1.16b, v4.16b - subs w2, w2, #16 - bmi 6f /* partial block? */ - ld1 {v2.16b}, [x1], #16 /* load next input block */ + subs w21, w21, #16 + bmi 7f /* partial block? */ + ld1 {v2.16b}, [x20], #16 /* load next input block */ .if \enc == 1 eor v2.16b, v2.16b, v5.16b /* final round enc+mac */ eor v1.16b, v1.16b, v2.16b /* xor with crypted ctr */ @@ -176,18 +205,29 @@ CPU_LE( rev x8, x8 ) /* keep swabbed ctr in reg */ eor v1.16b, v2.16b, v5.16b /* final round enc */ .endif eor v0.16b, v0.16b, v2.16b /* xor mac with pt ^ rk[last] */ - st1 {v1.16b}, [x0], #16 /* write output block */ - bne 0b -CPU_LE( rev x8, x8 ) - st1 {v0.16b}, [x5] /* store mac */ - str x8, [x6, #8] /* store lsb end of ctr (BE) */ -5: ret - -6: eor v0.16b, v0.16b, v5.16b /* final round mac */ + st1 {v1.16b}, [x19], #16 /* write output block */ + beq 5f + + if_will_cond_yield_neon + st1 {v0.16b}, [x24] /* store mac */ + do_cond_yield_neon + ld1 {v0.16b}, [x24] /* reload mac */ + endif_yield_neon + + b 0b +5: +CPU_LE( rev x26, x26 ) + st1 {v0.16b}, [x24] /* store mac */ + str x26, [x25, #8] /* store lsb end of ctr (BE) */ + +6: frame_pop + ret + +7: eor v0.16b, v0.16b, v5.16b /* final round mac */ eor v1.16b, v1.16b, v5.16b /* final round enc */ - st1 {v0.16b}, [x5] /* store mac */ - add w2, w2, #16 /* process partial tail block */ -7: ldrb w9, [x1], #1 /* get 1 byte of input */ + st1 {v0.16b}, [x24] /* store mac */ + add w21, w21, #16 /* process partial tail block */ +8: ldrb w9, [x20], #1 /* get 1 byte of input */ umov w6, v1.b[0] /* get top crypted ctr byte */ umov w7, v0.b[0] /* get top mac byte */ .if \enc == 1 @@ -197,13 +237,13 @@ CPU_LE( rev x8, x8 ) eor w9, w9, w6 eor w7, w7, w9 .endif - strb w9, [x0], #1 /* store out byte */ - strb w7, [x5], #1 /* store mac byte */ - subs w2, w2, #1 - beq 5b + strb w9, [x19], #1 /* store out byte */ + strb w7, [x24], #1 /* store mac byte */ + subs w21, w21, #1 + beq 6b ext v0.16b, v0.16b, v0.16b, #1 /* shift out mac byte */ ext v1.16b, v1.16b, v1.16b, #1 /* shift out ctr byte */ - b 7b + b 8b .endm /* diff --git a/arch/arm64/crypto/aes-ce.S b/arch/arm64/crypto/aes-ce.S index 50330f5c3adc..623e74ed1c67 100644 --- a/arch/arm64/crypto/aes-ce.S +++ b/arch/arm64/crypto/aes-ce.S @@ -30,18 +30,21 @@ .endm /* prepare for encryption with key in rk[] */ - .macro enc_prepare, rounds, rk, ignore - load_round_keys \rounds, \rk + .macro enc_prepare, rounds, rk, temp + mov \temp, \rk + load_round_keys \rounds, \temp .endm /* prepare for encryption (again) but with new key in rk[] */ - .macro enc_switch_key, rounds, rk, ignore - load_round_keys \rounds, \rk + .macro enc_switch_key, rounds, rk, temp + mov \temp, \rk + load_round_keys \rounds, \temp .endm /* prepare for decryption with key in rk[] */ - .macro dec_prepare, rounds, rk, ignore - load_round_keys \rounds, \rk + .macro dec_prepare, rounds, rk, temp + mov \temp, \rk + load_round_keys \rounds, \temp .endm .macro do_enc_Nx, de, mc, k, i0, i1, i2, i3 diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S index a68412e1e3a4..483a7130cf0e 100644 --- a/arch/arm64/crypto/aes-modes.S +++ b/arch/arm64/crypto/aes-modes.S @@ -14,12 +14,12 @@ .align 4 aes_encrypt_block4x: - encrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7 + encrypt_block4x v0, v1, v2, v3, w22, x21, x8, w7 ret ENDPROC(aes_encrypt_block4x) aes_decrypt_block4x: - decrypt_block4x v0, v1, v2, v3, w3, x2, x8, w7 + decrypt_block4x v0, v1, v2, v3, w22, x21, x8, w7 ret ENDPROC(aes_decrypt_block4x) @@ -31,57 +31,71 @@ ENDPROC(aes_decrypt_block4x) */ AES_ENTRY(aes_ecb_encrypt) - stp x29, x30, [sp, #-16]! - mov x29, sp + frame_push 5 - enc_prepare w3, x2, x5 + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + +.Lecbencrestart: + enc_prepare w22, x21, x5 .LecbencloopNx: - subs w4, w4, #4 + subs w23, w23, #4 bmi .Lecbenc1x - ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ + ld1 {v0.16b-v3.16b}, [x20], #64 /* get 4 pt blocks */ bl aes_encrypt_block4x - st1 {v0.16b-v3.16b}, [x0], #64 + st1 {v0.16b-v3.16b}, [x19], #64 + cond_yield_neon .Lecbencrestart b .LecbencloopNx .Lecbenc1x: - adds w4, w4, #4 + adds w23, w23, #4 beq .Lecbencout .Lecbencloop: - ld1 {v0.16b}, [x1], #16 /* get next pt block */ - encrypt_block v0, w3, x2, x5, w6 - st1 {v0.16b}, [x0], #16 - subs w4, w4, #1 + ld1 {v0.16b}, [x20], #16 /* get next pt block */ + encrypt_block v0, w22, x21, x5, w6 + st1 {v0.16b}, [x19], #16 + subs w23, w23, #1 bne .Lecbencloop .Lecbencout: - ldp x29, x30, [sp], #16 + frame_pop ret AES_ENDPROC(aes_ecb_encrypt) AES_ENTRY(aes_ecb_decrypt) - stp x29, x30, [sp, #-16]! - mov x29, sp + frame_push 5 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 - dec_prepare w3, x2, x5 +.Lecbdecrestart: + dec_prepare w22, x21, x5 .LecbdecloopNx: - subs w4, w4, #4 + subs w23, w23, #4 bmi .Lecbdec1x - ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 ct blocks */ + ld1 {v0.16b-v3.16b}, [x20], #64 /* get 4 ct blocks */ bl aes_decrypt_block4x - st1 {v0.16b-v3.16b}, [x0], #64 + st1 {v0.16b-v3.16b}, [x19], #64 + cond_yield_neon .Lecbdecrestart b .LecbdecloopNx .Lecbdec1x: - adds w4, w4, #4 + adds w23, w23, #4 beq .Lecbdecout .Lecbdecloop: - ld1 {v0.16b}, [x1], #16 /* get next ct block */ - decrypt_block v0, w3, x2, x5, w6 - st1 {v0.16b}, [x0], #16 - subs w4, w4, #1 + ld1 {v0.16b}, [x20], #16 /* get next ct block */ + decrypt_block v0, w22, x21, x5, w6 + st1 {v0.16b}, [x19], #16 + subs w23, w23, #1 bne .Lecbdecloop .Lecbdecout: - ldp x29, x30, [sp], #16 + frame_pop ret AES_ENDPROC(aes_ecb_decrypt) @@ -94,78 +108,100 @@ AES_ENDPROC(aes_ecb_decrypt) */ AES_ENTRY(aes_cbc_encrypt) - ld1 {v4.16b}, [x5] /* get iv */ - enc_prepare w3, x2, x6 + frame_push 6 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x5 + +.Lcbcencrestart: + ld1 {v4.16b}, [x24] /* get iv */ + enc_prepare w22, x21, x6 .Lcbcencloop4x: - subs w4, w4, #4 + subs w23, w23, #4 bmi .Lcbcenc1x - ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ + ld1 {v0.16b-v3.16b}, [x20], #64 /* get 4 pt blocks */ eor v0.16b, v0.16b, v4.16b /* ..and xor with iv */ - encrypt_block v0, w3, x2, x6, w7 + encrypt_block v0, w22, x21, x6, w7 eor v1.16b, v1.16b, v0.16b - encrypt_block v1, w3, x2, x6, w7 + encrypt_block v1, w22, x21, x6, w7 eor v2.16b, v2.16b, v1.16b - encrypt_block v2, w3, x2, x6, w7 + encrypt_block v2, w22, x21, x6, w7 eor v3.16b, v3.16b, v2.16b - encrypt_block v3, w3, x2, x6, w7 - st1 {v0.16b-v3.16b}, [x0], #64 + encrypt_block v3, w22, x21, x6, w7 + st1 {v0.16b-v3.16b}, [x19], #64 mov v4.16b, v3.16b + st1 {v4.16b}, [x24] /* return iv */ + cond_yield_neon .Lcbcencrestart b .Lcbcencloop4x .Lcbcenc1x: - adds w4, w4, #4 + adds w23, w23, #4 beq .Lcbcencout .Lcbcencloop: - ld1 {v0.16b}, [x1], #16 /* get next pt block */ + ld1 {v0.16b}, [x20], #16 /* get next pt block */ eor v4.16b, v4.16b, v0.16b /* ..and xor with iv */ - encrypt_block v4, w3, x2, x6, w7 - st1 {v4.16b}, [x0], #16 - subs w4, w4, #1 + encrypt_block v4, w22, x21, x6, w7 + st1 {v4.16b}, [x19], #16 + subs w23, w23, #1 bne .Lcbcencloop .Lcbcencout: - st1 {v4.16b}, [x5] /* return iv */ + st1 {v4.16b}, [x24] /* return iv */ + frame_pop ret AES_ENDPROC(aes_cbc_encrypt) AES_ENTRY(aes_cbc_decrypt) - stp x29, x30, [sp, #-16]! - mov x29, sp + frame_push 6 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x5 - ld1 {v7.16b}, [x5] /* get iv */ - dec_prepare w3, x2, x6 +.Lcbcdecrestart: + ld1 {v7.16b}, [x24] /* get iv */ + dec_prepare w22, x21, x6 .LcbcdecloopNx: - subs w4, w4, #4 + subs w23, w23, #4 bmi .Lcbcdec1x - ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 ct blocks */ + ld1 {v0.16b-v3.16b}, [x20], #64 /* get 4 ct blocks */ mov v4.16b, v0.16b mov v5.16b, v1.16b mov v6.16b, v2.16b bl aes_decrypt_block4x - sub x1, x1, #16 + sub x20, x20, #16 eor v0.16b, v0.16b, v7.16b eor v1.16b, v1.16b, v4.16b - ld1 {v7.16b}, [x1], #16 /* reload 1 ct block */ + ld1 {v7.16b}, [x20], #16 /* reload 1 ct block */ eor v2.16b, v2.16b, v5.16b eor v3.16b, v3.16b, v6.16b - st1 {v0.16b-v3.16b}, [x0], #64 + st1 {v0.16b-v3.16b}, [x19], #64 + st1 {v7.16b}, [x24] /* return iv */ + cond_yield_neon .Lcbcdecrestart b .LcbcdecloopNx .Lcbcdec1x: - adds w4, w4, #4 + adds w23, w23, #4 beq .Lcbcdecout .Lcbcdecloop: - ld1 {v1.16b}, [x1], #16 /* get next ct block */ + ld1 {v1.16b}, [x20], #16 /* get next ct block */ mov v0.16b, v1.16b /* ...and copy to v0 */ - decrypt_block v0, w3, x2, x6, w7 + decrypt_block v0, w22, x21, x6, w7 eor v0.16b, v0.16b, v7.16b /* xor with iv => pt */ mov v7.16b, v1.16b /* ct is next iv */ - st1 {v0.16b}, [x0], #16 - subs w4, w4, #1 + st1 {v0.16b}, [x19], #16 + subs w23, w23, #1 bne .Lcbcdecloop .Lcbcdecout: - st1 {v7.16b}, [x5] /* return iv */ - ldp x29, x30, [sp], #16 + st1 {v7.16b}, [x24] /* return iv */ + frame_pop ret AES_ENDPROC(aes_cbc_decrypt) @@ -176,19 +212,26 @@ AES_ENDPROC(aes_cbc_decrypt) */ AES_ENTRY(aes_ctr_encrypt) - stp x29, x30, [sp, #-16]! - mov x29, sp + frame_push 6 - enc_prepare w3, x2, x6 - ld1 {v4.16b}, [x5] + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x5 + +.Lctrrestart: + enc_prepare w22, x21, x6 + ld1 {v4.16b}, [x24] umov x6, v4.d[1] /* keep swabbed ctr in reg */ rev x6, x6 - cmn w6, w4 /* 32 bit overflow? */ - bcs .Lctrloop .LctrloopNx: - subs w4, w4, #4 + subs w23, w23, #4 bmi .Lctr1x + cmn w6, #4 /* 32 bit overflow? */ + bcs .Lctr1x ldr q8, =0x30000000200000001 /* addends 1,2,3[,0] */ dup v7.4s, w6 mov v0.16b, v4.16b @@ -200,25 +243,27 @@ AES_ENTRY(aes_ctr_encrypt) mov v1.s[3], v8.s[0] mov v2.s[3], v8.s[1] mov v3.s[3], v8.s[2] - ld1 {v5.16b-v7.16b}, [x1], #48 /* get 3 input blocks */ + ld1 {v5.16b-v7.16b}, [x20], #48 /* get 3 input blocks */ bl aes_encrypt_block4x eor v0.16b, v5.16b, v0.16b - ld1 {v5.16b}, [x1], #16 /* get 1 input block */ + ld1 {v5.16b}, [x20], #16 /* get 1 input block */ eor v1.16b, v6.16b, v1.16b eor v2.16b, v7.16b, v2.16b eor v3.16b, v5.16b, v3.16b - st1 {v0.16b-v3.16b}, [x0], #64 + st1 {v0.16b-v3.16b}, [x19], #64 add x6, x6, #4 rev x7, x6 ins v4.d[1], x7 - cbz w4, .Lctrout + cbz w23, .Lctrout + st1 {v4.16b}, [x24] /* return next CTR value */ + cond_yield_neon .Lctrrestart b .LctrloopNx .Lctr1x: - adds w4, w4, #4 + adds w23, w23, #4 beq .Lctrout .Lctrloop: mov v0.16b, v4.16b - encrypt_block v0, w3, x2, x8, w7 + encrypt_block v0, w22, x21, x8, w7 adds x6, x6, #1 /* increment BE ctr */ rev x7, x6 @@ -226,22 +271,22 @@ AES_ENTRY(aes_ctr_encrypt) bcs .Lctrcarry /* overflow? */ .Lctrcarrydone: - subs w4, w4, #1 + subs w23, w23, #1 bmi .Lctrtailblock /* blocks <0 means tail block */ - ld1 {v3.16b}, [x1], #16 + ld1 {v3.16b}, [x20], #16 eor v3.16b, v0.16b, v3.16b - st1 {v3.16b}, [x0], #16 + st1 {v3.16b}, [x19], #16 bne .Lctrloop .Lctrout: - st1 {v4.16b}, [x5] /* return next CTR value */ - ldp x29, x30, [sp], #16 + st1 {v4.16b}, [x24] /* return next CTR value */ +.Lctrret: + frame_pop ret .Lctrtailblock: - st1 {v0.16b}, [x0] - ldp x29, x30, [sp], #16 - ret + st1 {v0.16b}, [x19] + b .Lctrret .Lctrcarry: umov x7, v4.d[0] /* load upper word of ctr */ @@ -274,10 +319,16 @@ CPU_LE( .quad 1, 0x87 ) CPU_BE( .quad 0x87, 1 ) AES_ENTRY(aes_xts_encrypt) - stp x29, x30, [sp, #-16]! - mov x29, sp + frame_push 6 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x6 - ld1 {v4.16b}, [x6] + ld1 {v4.16b}, [x24] cbz w7, .Lxtsencnotfirst enc_prepare w3, x5, x8 @@ -286,15 +337,17 @@ AES_ENTRY(aes_xts_encrypt) ldr q7, .Lxts_mul_x b .LxtsencNx +.Lxtsencrestart: + ld1 {v4.16b}, [x24] .Lxtsencnotfirst: - enc_prepare w3, x2, x8 + enc_prepare w22, x21, x8 .LxtsencloopNx: ldr q7, .Lxts_mul_x next_tweak v4, v4, v7, v8 .LxtsencNx: - subs w4, w4, #4 + subs w23, w23, #4 bmi .Lxtsenc1x - ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 pt blocks */ + ld1 {v0.16b-v3.16b}, [x20], #64 /* get 4 pt blocks */ next_tweak v5, v4, v7, v8 eor v0.16b, v0.16b, v4.16b next_tweak v6, v5, v7, v8 @@ -307,35 +360,43 @@ AES_ENTRY(aes_xts_encrypt) eor v0.16b, v0.16b, v4.16b eor v1.16b, v1.16b, v5.16b eor v2.16b, v2.16b, v6.16b - st1 {v0.16b-v3.16b}, [x0], #64 + st1 {v0.16b-v3.16b}, [x19], #64 mov v4.16b, v7.16b - cbz w4, .Lxtsencout + cbz w23, .Lxtsencout + st1 {v4.16b}, [x24] + cond_yield_neon .Lxtsencrestart b .LxtsencloopNx .Lxtsenc1x: - adds w4, w4, #4 + adds w23, w23, #4 beq .Lxtsencout .Lxtsencloop: - ld1 {v1.16b}, [x1], #16 + ld1 {v1.16b}, [x20], #16 eor v0.16b, v1.16b, v4.16b - encrypt_block v0, w3, x2, x8, w7 + encrypt_block v0, w22, x21, x8, w7 eor v0.16b, v0.16b, v4.16b - st1 {v0.16b}, [x0], #16 - subs w4, w4, #1 + st1 {v0.16b}, [x19], #16 + subs w23, w23, #1 beq .Lxtsencout next_tweak v4, v4, v7, v8 b .Lxtsencloop .Lxtsencout: - st1 {v4.16b}, [x6] - ldp x29, x30, [sp], #16 + st1 {v4.16b}, [x24] + frame_pop ret AES_ENDPROC(aes_xts_encrypt) AES_ENTRY(aes_xts_decrypt) - stp x29, x30, [sp, #-16]! - mov x29, sp + frame_push 6 - ld1 {v4.16b}, [x6] + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x6 + + ld1 {v4.16b}, [x24] cbz w7, .Lxtsdecnotfirst enc_prepare w3, x5, x8 @@ -344,15 +405,17 @@ AES_ENTRY(aes_xts_decrypt) ldr q7, .Lxts_mul_x b .LxtsdecNx +.Lxtsdecrestart: + ld1 {v4.16b}, [x24] .Lxtsdecnotfirst: - dec_prepare w3, x2, x8 + dec_prepare w22, x21, x8 .LxtsdecloopNx: ldr q7, .Lxts_mul_x next_tweak v4, v4, v7, v8 .LxtsdecNx: - subs w4, w4, #4 + subs w23, w23, #4 bmi .Lxtsdec1x - ld1 {v0.16b-v3.16b}, [x1], #64 /* get 4 ct blocks */ + ld1 {v0.16b-v3.16b}, [x20], #64 /* get 4 ct blocks */ next_tweak v5, v4, v7, v8 eor v0.16b, v0.16b, v4.16b next_tweak v6, v5, v7, v8 @@ -365,26 +428,28 @@ AES_ENTRY(aes_xts_decrypt) eor v0.16b, v0.16b, v4.16b eor v1.16b, v1.16b, v5.16b eor v2.16b, v2.16b, v6.16b - st1 {v0.16b-v3.16b}, [x0], #64 + st1 {v0.16b-v3.16b}, [x19], #64 mov v4.16b, v7.16b - cbz w4, .Lxtsdecout + cbz w23, .Lxtsdecout + st1 {v4.16b}, [x24] + cond_yield_neon .Lxtsdecrestart b .LxtsdecloopNx .Lxtsdec1x: - adds w4, w4, #4 + adds w23, w23, #4 beq .Lxtsdecout .Lxtsdecloop: - ld1 {v1.16b}, [x1], #16 + ld1 {v1.16b}, [x20], #16 eor v0.16b, v1.16b, v4.16b - decrypt_block v0, w3, x2, x8, w7 + decrypt_block v0, w22, x21, x8, w7 eor v0.16b, v0.16b, v4.16b - st1 {v0.16b}, [x0], #16 - subs w4, w4, #1 + st1 {v0.16b}, [x19], #16 + subs w23, w23, #1 beq .Lxtsdecout next_tweak v4, v4, v7, v8 b .Lxtsdecloop .Lxtsdecout: - st1 {v4.16b}, [x6] - ldp x29, x30, [sp], #16 + st1 {v4.16b}, [x24] + frame_pop ret AES_ENDPROC(aes_xts_decrypt) @@ -393,43 +458,61 @@ AES_ENDPROC(aes_xts_decrypt) * int blocks, u8 dg[], int enc_before, int enc_after) */ AES_ENTRY(aes_mac_update) - ld1 {v0.16b}, [x4] /* get dg */ + frame_push 6 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x6 + + ld1 {v0.16b}, [x23] /* get dg */ enc_prepare w2, x1, x7 cbz w5, .Lmacloop4x encrypt_block v0, w2, x1, x7, w8 .Lmacloop4x: - subs w3, w3, #4 + subs w22, w22, #4 bmi .Lmac1x - ld1 {v1.16b-v4.16b}, [x0], #64 /* get next pt block */ + ld1 {v1.16b-v4.16b}, [x19], #64 /* get next pt block */ eor v0.16b, v0.16b, v1.16b /* ..and xor with dg */ - encrypt_block v0, w2, x1, x7, w8 + encrypt_block v0, w21, x20, x7, w8 eor v0.16b, v0.16b, v2.16b - encrypt_block v0, w2, x1, x7, w8 + encrypt_block v0, w21, x20, x7, w8 eor v0.16b, v0.16b, v3.16b - encrypt_block v0, w2, x1, x7, w8 + encrypt_block v0, w21, x20, x7, w8 eor v0.16b, v0.16b, v4.16b - cmp w3, wzr - csinv x5, x6, xzr, eq + cmp w22, wzr + csinv x5, x24, xzr, eq cbz w5, .Lmacout - encrypt_block v0, w2, x1, x7, w8 + encrypt_block v0, w21, x20, x7, w8 + st1 {v0.16b}, [x23] /* return dg */ + cond_yield_neon .Lmacrestart b .Lmacloop4x .Lmac1x: - add w3, w3, #4 + add w22, w22, #4 .Lmacloop: - cbz w3, .Lmacout - ld1 {v1.16b}, [x0], #16 /* get next pt block */ + cbz w22, .Lmacout + ld1 {v1.16b}, [x19], #16 /* get next pt block */ eor v0.16b, v0.16b, v1.16b /* ..and xor with dg */ - subs w3, w3, #1 - csinv x5, x6, xzr, eq + subs w22, w22, #1 + csinv x5, x24, xzr, eq cbz w5, .Lmacout - encrypt_block v0, w2, x1, x7, w8 +.Lmacenc: + encrypt_block v0, w21, x20, x7, w8 b .Lmacloop .Lmacout: - st1 {v0.16b}, [x4] /* return dg */ + st1 {v0.16b}, [x23] /* return dg */ + frame_pop ret + +.Lmacrestart: + ld1 {v0.16b}, [x23] /* get dg */ + enc_prepare w21, x20, x0 + b .Lmacloop4x AES_ENDPROC(aes_mac_update) diff --git a/arch/arm64/crypto/aes-neonbs-core.S b/arch/arm64/crypto/aes-neonbs-core.S index ca0472500433..e613a87f8b53 100644 --- a/arch/arm64/crypto/aes-neonbs-core.S +++ b/arch/arm64/crypto/aes-neonbs-core.S @@ -565,54 +565,61 @@ ENDPROC(aesbs_decrypt8) * int blocks) */ .macro __ecb_crypt, do8, o0, o1, o2, o3, o4, o5, o6, o7 - stp x29, x30, [sp, #-16]! - mov x29, sp + frame_push 5 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 99: mov x5, #1 - lsl x5, x5, x4 - subs w4, w4, #8 - csel x4, x4, xzr, pl + lsl x5, x5, x23 + subs w23, w23, #8 + csel x23, x23, xzr, pl csel x5, x5, xzr, mi - ld1 {v0.16b}, [x1], #16 + ld1 {v0.16b}, [x20], #16 tbnz x5, #1, 0f - ld1 {v1.16b}, [x1], #16 + ld1 {v1.16b}, [x20], #16 tbnz x5, #2, 0f - ld1 {v2.16b}, [x1], #16 + ld1 {v2.16b}, [x20], #16 tbnz x5, #3, 0f - ld1 {v3.16b}, [x1], #16 + ld1 {v3.16b}, [x20], #16 tbnz x5, #4, 0f - ld1 {v4.16b}, [x1], #16 + ld1 {v4.16b}, [x20], #16 tbnz x5, #5, 0f - ld1 {v5.16b}, [x1], #16 + ld1 {v5.16b}, [x20], #16 tbnz x5, #6, 0f - ld1 {v6.16b}, [x1], #16 + ld1 {v6.16b}, [x20], #16 tbnz x5, #7, 0f - ld1 {v7.16b}, [x1], #16 + ld1 {v7.16b}, [x20], #16 -0: mov bskey, x2 - mov rounds, x3 +0: mov bskey, x21 + mov rounds, x22 bl \do8 - st1 {\o0\().16b}, [x0], #16 + st1 {\o0\().16b}, [x19], #16 tbnz x5, #1, 1f - st1 {\o1\().16b}, [x0], #16 + st1 {\o1\().16b}, [x19], #16 tbnz x5, #2, 1f - st1 {\o2\().16b}, [x0], #16 + st1 {\o2\().16b}, [x19], #16 tbnz x5, #3, 1f - st1 {\o3\().16b}, [x0], #16 + st1 {\o3\().16b}, [x19], #16 tbnz x5, #4, 1f - st1 {\o4\().16b}, [x0], #16 + st1 {\o4\().16b}, [x19], #16 tbnz x5, #5, 1f - st1 {\o5\().16b}, [x0], #16 + st1 {\o5\().16b}, [x19], #16 tbnz x5, #6, 1f - st1 {\o6\().16b}, [x0], #16 + st1 {\o6\().16b}, [x19], #16 tbnz x5, #7, 1f - st1 {\o7\().16b}, [x0], #16 + st1 {\o7\().16b}, [x19], #16 - cbnz x4, 99b + cbz x23, 1f + cond_yield_neon + b 99b -1: ldp x29, x30, [sp], #16 +1: frame_pop ret .endm @@ -632,43 +639,49 @@ ENDPROC(aesbs_ecb_decrypt) */ .align 4 ENTRY(aesbs_cbc_decrypt) - stp x29, x30, [sp, #-16]! - mov x29, sp + frame_push 6 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x5 99: mov x6, #1 - lsl x6, x6, x4 - subs w4, w4, #8 - csel x4, x4, xzr, pl + lsl x6, x6, x23 + subs w23, w23, #8 + csel x23, x23, xzr, pl csel x6, x6, xzr, mi - ld1 {v0.16b}, [x1], #16 + ld1 {v0.16b}, [x20], #16 mov v25.16b, v0.16b tbnz x6, #1, 0f - ld1 {v1.16b}, [x1], #16 + ld1 {v1.16b}, [x20], #16 mov v26.16b, v1.16b tbnz x6, #2, 0f - ld1 {v2.16b}, [x1], #16 + ld1 {v2.16b}, [x20], #16 mov v27.16b, v2.16b tbnz x6, #3, 0f - ld1 {v3.16b}, [x1], #16 + ld1 {v3.16b}, [x20], #16 mov v28.16b, v3.16b tbnz x6, #4, 0f - ld1 {v4.16b}, [x1], #16 + ld1 {v4.16b}, [x20], #16 mov v29.16b, v4.16b tbnz x6, #5, 0f - ld1 {v5.16b}, [x1], #16 + ld1 {v5.16b}, [x20], #16 mov v30.16b, v5.16b tbnz x6, #6, 0f - ld1 {v6.16b}, [x1], #16 + ld1 {v6.16b}, [x20], #16 mov v31.16b, v6.16b tbnz x6, #7, 0f - ld1 {v7.16b}, [x1] + ld1 {v7.16b}, [x20] -0: mov bskey, x2 - mov rounds, x3 +0: mov bskey, x21 + mov rounds, x22 bl aesbs_decrypt8 - ld1 {v24.16b}, [x5] // load IV + ld1 {v24.16b}, [x24] // load IV eor v1.16b, v1.16b, v25.16b eor v6.16b, v6.16b, v26.16b @@ -679,34 +692,36 @@ ENTRY(aesbs_cbc_decrypt) eor v3.16b, v3.16b, v30.16b eor v5.16b, v5.16b, v31.16b - st1 {v0.16b}, [x0], #16 + st1 {v0.16b}, [x19], #16 mov v24.16b, v25.16b tbnz x6, #1, 1f - st1 {v1.16b}, [x0], #16 + st1 {v1.16b}, [x19], #16 mov v24.16b, v26.16b tbnz x6, #2, 1f - st1 {v6.16b}, [x0], #16 + st1 {v6.16b}, [x19], #16 mov v24.16b, v27.16b tbnz x6, #3, 1f - st1 {v4.16b}, [x0], #16 + st1 {v4.16b}, [x19], #16 mov v24.16b, v28.16b tbnz x6, #4, 1f - st1 {v2.16b}, [x0], #16 + st1 {v2.16b}, [x19], #16 mov v24.16b, v29.16b tbnz x6, #5, 1f - st1 {v7.16b}, [x0], #16 + st1 {v7.16b}, [x19], #16 mov v24.16b, v30.16b tbnz x6, #6, 1f - st1 {v3.16b}, [x0], #16 + st1 {v3.16b}, [x19], #16 mov v24.16b, v31.16b tbnz x6, #7, 1f - ld1 {v24.16b}, [x1], #16 - st1 {v5.16b}, [x0], #16 -1: st1 {v24.16b}, [x5] // store IV + ld1 {v24.16b}, [x20], #16 + st1 {v5.16b}, [x19], #16 +1: st1 {v24.16b}, [x24] // store IV - cbnz x4, 99b + cbz x23, 2f + cond_yield_neon + b 99b - ldp x29, x30, [sp], #16 +2: frame_pop ret ENDPROC(aesbs_cbc_decrypt) @@ -731,87 +746,93 @@ CPU_BE( .quad 0x87, 1 ) */ __xts_crypt8: mov x6, #1 - lsl x6, x6, x4 - subs w4, w4, #8 - csel x4, x4, xzr, pl + lsl x6, x6, x23 + subs w23, w23, #8 + csel x23, x23, xzr, pl csel x6, x6, xzr, mi - ld1 {v0.16b}, [x1], #16 + ld1 {v0.16b}, [x20], #16 next_tweak v26, v25, v30, v31 eor v0.16b, v0.16b, v25.16b tbnz x6, #1, 0f - ld1 {v1.16b}, [x1], #16 + ld1 {v1.16b}, [x20], #16 next_tweak v27, v26, v30, v31 eor v1.16b, v1.16b, v26.16b tbnz x6, #2, 0f - ld1 {v2.16b}, [x1], #16 + ld1 {v2.16b}, [x20], #16 next_tweak v28, v27, v30, v31 eor v2.16b, v2.16b, v27.16b tbnz x6, #3, 0f - ld1 {v3.16b}, [x1], #16 + ld1 {v3.16b}, [x20], #16 next_tweak v29, v28, v30, v31 eor v3.16b, v3.16b, v28.16b tbnz x6, #4, 0f - ld1 {v4.16b}, [x1], #16 - str q29, [sp, #16] + ld1 {v4.16b}, [x20], #16 + str q29, [sp, #.Lframe_local_offset] eor v4.16b, v4.16b, v29.16b next_tweak v29, v29, v30, v31 tbnz x6, #5, 0f - ld1 {v5.16b}, [x1], #16 - str q29, [sp, #32] + ld1 {v5.16b}, [x20], #16 + str q29, [sp, #.Lframe_local_offset + 16] eor v5.16b, v5.16b, v29.16b next_tweak v29, v29, v30, v31 tbnz x6, #6, 0f - ld1 {v6.16b}, [x1], #16 - str q29, [sp, #48] + ld1 {v6.16b}, [x20], #16 + str q29, [sp, #.Lframe_local_offset + 32] eor v6.16b, v6.16b, v29.16b next_tweak v29, v29, v30, v31 tbnz x6, #7, 0f - ld1 {v7.16b}, [x1], #16 - str q29, [sp, #64] + ld1 {v7.16b}, [x20], #16 + str q29, [sp, #.Lframe_local_offset + 48] eor v7.16b, v7.16b, v29.16b next_tweak v29, v29, v30, v31 -0: mov bskey, x2 - mov rounds, x3 +0: mov bskey, x21 + mov rounds, x22 br x7 ENDPROC(__xts_crypt8) .macro __xts_crypt, do8, o0, o1, o2, o3, o4, o5, o6, o7 - stp x29, x30, [sp, #-80]! - mov x29, sp + frame_push 6, 64 - ldr q30, .Lxts_mul_x - ld1 {v25.16b}, [x5] + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x5 + +0: ldr q30, .Lxts_mul_x + ld1 {v25.16b}, [x24] 99: adr x7, \do8 bl __xts_crypt8 - ldp q16, q17, [sp, #16] - ldp q18, q19, [sp, #48] + ldp q16, q17, [sp, #.Lframe_local_offset] + ldp q18, q19, [sp, #.Lframe_local_offset + 32] eor \o0\().16b, \o0\().16b, v25.16b eor \o1\().16b, \o1\().16b, v26.16b eor \o2\().16b, \o2\().16b, v27.16b eor \o3\().16b, \o3\().16b, v28.16b - st1 {\o0\().16b}, [x0], #16 + st1 {\o0\().16b}, [x19], #16 mov v25.16b, v26.16b tbnz x6, #1, 1f - st1 {\o1\().16b}, [x0], #16 + st1 {\o1\().16b}, [x19], #16 mov v25.16b, v27.16b tbnz x6, #2, 1f - st1 {\o2\().16b}, [x0], #16 + st1 {\o2\().16b}, [x19], #16 mov v25.16b, v28.16b tbnz x6, #3, 1f - st1 {\o3\().16b}, [x0], #16 + st1 {\o3\().16b}, [x19], #16 mov v25.16b, v29.16b tbnz x6, #4, 1f @@ -820,18 +841,22 @@ ENDPROC(__xts_crypt8) eor \o6\().16b, \o6\().16b, v18.16b eor \o7\().16b, \o7\().16b, v19.16b - st1 {\o4\().16b}, [x0], #16 + st1 {\o4\().16b}, [x19], #16 tbnz x6, #5, 1f - st1 {\o5\().16b}, [x0], #16 + st1 {\o5\().16b}, [x19], #16 tbnz x6, #6, 1f - st1 {\o6\().16b}, [x0], #16 + st1 {\o6\().16b}, [x19], #16 tbnz x6, #7, 1f - st1 {\o7\().16b}, [x0], #16 + st1 {\o7\().16b}, [x19], #16 + + cbz x23, 1f + st1 {v25.16b}, [x24] - cbnz x4, 99b + cond_yield_neon 0b + b 99b -1: st1 {v25.16b}, [x5] - ldp x29, x30, [sp], #80 +1: st1 {v25.16b}, [x24] + frame_pop ret .endm @@ -856,24 +881,31 @@ ENDPROC(aesbs_xts_decrypt) * int rounds, int blocks, u8 iv[], u8 final[]) */ ENTRY(aesbs_ctr_encrypt) - stp x29, x30, [sp, #-16]! - mov x29, sp - - cmp x6, #0 - cset x10, ne - add x4, x4, x10 // do one extra block if final - - ldp x7, x8, [x5] - ld1 {v0.16b}, [x5] + frame_push 8 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x5 + mov x25, x6 + + cmp x25, #0 + cset x26, ne + add x23, x23, x26 // do one extra block if final + +98: ldp x7, x8, [x24] + ld1 {v0.16b}, [x24] CPU_LE( rev x7, x7 ) CPU_LE( rev x8, x8 ) adds x8, x8, #1 adc x7, x7, xzr 99: mov x9, #1 - lsl x9, x9, x4 - subs w4, w4, #8 - csel x4, x4, xzr, pl + lsl x9, x9, x23 + subs w23, w23, #8 + csel x23, x23, xzr, pl csel x9, x9, xzr, le tbnz x9, #1, 0f @@ -891,82 +923,85 @@ CPU_LE( rev x8, x8 ) tbnz x9, #7, 0f next_ctr v7 -0: mov bskey, x2 - mov rounds, x3 +0: mov bskey, x21 + mov rounds, x22 bl aesbs_encrypt8 - lsr x9, x9, x10 // disregard the extra block + lsr x9, x9, x26 // disregard the extra block tbnz x9, #0, 0f - ld1 {v8.16b}, [x1], #16 + ld1 {v8.16b}, [x20], #16 eor v0.16b, v0.16b, v8.16b - st1 {v0.16b}, [x0], #16 + st1 {v0.16b}, [x19], #16 tbnz x9, #1, 1f - ld1 {v9.16b}, [x1], #16 + ld1 {v9.16b}, [x20], #16 eor v1.16b, v1.16b, v9.16b - st1 {v1.16b}, [x0], #16 + st1 {v1.16b}, [x19], #16 tbnz x9, #2, 2f - ld1 {v10.16b}, [x1], #16 + ld1 {v10.16b}, [x20], #16 eor v4.16b, v4.16b, v10.16b - st1 {v4.16b}, [x0], #16 + st1 {v4.16b}, [x19], #16 tbnz x9, #3, 3f - ld1 {v11.16b}, [x1], #16 + ld1 {v11.16b}, [x20], #16 eor v6.16b, v6.16b, v11.16b - st1 {v6.16b}, [x0], #16 + st1 {v6.16b}, [x19], #16 tbnz x9, #4, 4f - ld1 {v12.16b}, [x1], #16 + ld1 {v12.16b}, [x20], #16 eor v3.16b, v3.16b, v12.16b - st1 {v3.16b}, [x0], #16 + st1 {v3.16b}, [x19], #16 tbnz x9, #5, 5f - ld1 {v13.16b}, [x1], #16 + ld1 {v13.16b}, [x20], #16 eor v7.16b, v7.16b, v13.16b - st1 {v7.16b}, [x0], #16 + st1 {v7.16b}, [x19], #16 tbnz x9, #6, 6f - ld1 {v14.16b}, [x1], #16 + ld1 {v14.16b}, [x20], #16 eor v2.16b, v2.16b, v14.16b - st1 {v2.16b}, [x0], #16 + st1 {v2.16b}, [x19], #16 tbnz x9, #7, 7f - ld1 {v15.16b}, [x1], #16 + ld1 {v15.16b}, [x20], #16 eor v5.16b, v5.16b, v15.16b - st1 {v5.16b}, [x0], #16 + st1 {v5.16b}, [x19], #16 8: next_ctr v0 - cbnz x4, 99b + st1 {v0.16b}, [x24] + cbz x23, 0f + + cond_yield_neon 98b + b 99b -0: st1 {v0.16b}, [x5] - ldp x29, x30, [sp], #16 +0: frame_pop ret /* * If we are handling the tail of the input (x6 != NULL), return the * final keystream block back to the caller. */ -1: cbz x6, 8b - st1 {v1.16b}, [x6] +1: cbz x25, 8b + st1 {v1.16b}, [x25] b 8b -2: cbz x6, 8b - st1 {v4.16b}, [x6] +2: cbz x25, 8b + st1 {v4.16b}, [x25] b 8b -3: cbz x6, 8b - st1 {v6.16b}, [x6] +3: cbz x25, 8b + st1 {v6.16b}, [x25] b 8b -4: cbz x6, 8b - st1 {v3.16b}, [x6] +4: cbz x25, 8b + st1 {v3.16b}, [x25] b 8b -5: cbz x6, 8b - st1 {v7.16b}, [x6] +5: cbz x25, 8b + st1 {v7.16b}, [x25] b 8b -6: cbz x6, 8b - st1 {v2.16b}, [x6] +6: cbz x25, 8b + st1 {v2.16b}, [x25] b 8b -7: cbz x6, 8b - st1 {v5.16b}, [x6] +7: cbz x25, 8b + st1 {v5.16b}, [x25] b 8b ENDPROC(aesbs_ctr_encrypt) diff --git a/arch/arm64/crypto/crc32-ce-core.S b/arch/arm64/crypto/crc32-ce-core.S index 16ed3c7ebd37..8061bf0f9c66 100644 --- a/arch/arm64/crypto/crc32-ce-core.S +++ b/arch/arm64/crypto/crc32-ce-core.S @@ -100,9 +100,10 @@ dCONSTANT .req d0 qCONSTANT .req q0 - BUF .req x0 - LEN .req x1 - CRC .req x2 + BUF .req x19 + LEN .req x20 + CRC .req x21 + CONST .req x22 vzr .req v9 @@ -123,7 +124,14 @@ ENTRY(crc32_pmull_le) ENTRY(crc32c_pmull_le) adr_l x3, .Lcrc32c_constants -0: bic LEN, LEN, #15 +0: frame_push 4, 64 + + mov BUF, x0 + mov LEN, x1 + mov CRC, x2 + mov CONST, x3 + + bic LEN, LEN, #15 ld1 {v1.16b-v4.16b}, [BUF], #0x40 movi vzr.16b, #0 fmov dCONSTANT, CRC @@ -132,7 +140,7 @@ ENTRY(crc32c_pmull_le) cmp LEN, #0x40 b.lt less_64 - ldr qCONSTANT, [x3] + ldr qCONSTANT, [CONST] loop_64: /* 64 bytes Full cache line folding */ sub LEN, LEN, #0x40 @@ -162,10 +170,21 @@ loop_64: /* 64 bytes Full cache line folding */ eor v4.16b, v4.16b, v8.16b cmp LEN, #0x40 - b.ge loop_64 + b.lt less_64 + + if_will_cond_yield_neon + stp q1, q2, [sp, #.Lframe_local_offset] + stp q3, q4, [sp, #.Lframe_local_offset + 32] + do_cond_yield_neon + ldp q1, q2, [sp, #.Lframe_local_offset] + ldp q3, q4, [sp, #.Lframe_local_offset + 32] + ldr qCONSTANT, [CONST] + movi vzr.16b, #0 + endif_yield_neon + b loop_64 less_64: /* Folding cache line into 128bit */ - ldr qCONSTANT, [x3, #16] + ldr qCONSTANT, [CONST, #16] pmull2 v5.1q, v1.2d, vCONSTANT.2d pmull v1.1q, v1.1d, vCONSTANT.1d @@ -204,8 +223,8 @@ fold_64: eor v1.16b, v1.16b, v2.16b /* final 32-bit fold */ - ldr dCONSTANT, [x3, #32] - ldr d3, [x3, #40] + ldr dCONSTANT, [CONST, #32] + ldr d3, [CONST, #40] ext v2.16b, v1.16b, vzr.16b, #4 and v1.16b, v1.16b, v3.16b @@ -213,7 +232,7 @@ fold_64: eor v1.16b, v1.16b, v2.16b /* Finish up with the bit-reversed barrett reduction 64 ==> 32 bits */ - ldr qCONSTANT, [x3, #48] + ldr qCONSTANT, [CONST, #48] and v2.16b, v1.16b, v3.16b ext v2.16b, vzr.16b, v2.16b, #8 @@ -223,6 +242,7 @@ fold_64: eor v1.16b, v1.16b, v2.16b mov w0, v1.s[1] + frame_pop ret ENDPROC(crc32_pmull_le) ENDPROC(crc32c_pmull_le) diff --git a/arch/arm64/crypto/crct10dif-ce-core.S b/arch/arm64/crypto/crct10dif-ce-core.S index f179c01bd55c..663ea71cdb38 100644 --- a/arch/arm64/crypto/crct10dif-ce-core.S +++ b/arch/arm64/crypto/crct10dif-ce-core.S @@ -74,13 +74,19 @@ .text .cpu generic+crypto - arg1_low32 .req w0 - arg2 .req x1 - arg3 .req x2 + arg1_low32 .req w19 + arg2 .req x20 + arg3 .req x21 vzr .req v13 ENTRY(crc_t10dif_pmull) + frame_push 3, 128 + + mov arg1_low32, w0 + mov arg2, x1 + mov arg3, x2 + movi vzr.16b, #0 // init zero register // adjust the 16-bit initial_crc value, scale it to 32 bits @@ -175,8 +181,25 @@ CPU_LE( ext v12.16b, v12.16b, v12.16b, #8 ) subs arg3, arg3, #128 // check if there is another 64B in the buffer to be able to fold - b.ge _fold_64_B_loop + b.lt _fold_64_B_end + + if_will_cond_yield_neon + stp q0, q1, [sp, #.Lframe_local_offset] + stp q2, q3, [sp, #.Lframe_local_offset + 32] + stp q4, q5, [sp, #.Lframe_local_offset + 64] + stp q6, q7, [sp, #.Lframe_local_offset + 96] + do_cond_yield_neon + ldp q0, q1, [sp, #.Lframe_local_offset] + ldp q2, q3, [sp, #.Lframe_local_offset + 32] + ldp q4, q5, [sp, #.Lframe_local_offset + 64] + ldp q6, q7, [sp, #.Lframe_local_offset + 96] + ldr_l q10, rk3, x8 + movi vzr.16b, #0 // init zero register + endif_yield_neon + + b _fold_64_B_loop +_fold_64_B_end: // at this point, the buffer pointer is pointing at the last y Bytes // of the buffer the 64B of folded data is in 4 of the vector // registers: v0, v1, v2, v3 @@ -304,6 +327,7 @@ _barrett: _cleanup: // scale the result back to 16 bits lsr x0, x0, #16 + frame_pop ret _less_than_128: diff --git a/arch/arm64/crypto/ghash-ce-core.S b/arch/arm64/crypto/ghash-ce-core.S index 11ebf1ae248a..dcffb9e77589 100644 --- a/arch/arm64/crypto/ghash-ce-core.S +++ b/arch/arm64/crypto/ghash-ce-core.S @@ -213,22 +213,31 @@ .endm .macro __pmull_ghash, pn - ld1 {SHASH.2d}, [x3] - ld1 {XL.2d}, [x1] + frame_push 5 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + +0: ld1 {SHASH.2d}, [x22] + ld1 {XL.2d}, [x20] ext SHASH2.16b, SHASH.16b, SHASH.16b, #8 eor SHASH2.16b, SHASH2.16b, SHASH.16b __pmull_pre_\pn /* do the head block first, if supplied */ - cbz x4, 0f - ld1 {T1.2d}, [x4] - b 1f + cbz x23, 1f + ld1 {T1.2d}, [x23] + mov x23, xzr + b 2f -0: ld1 {T1.2d}, [x2], #16 - sub w0, w0, #1 +1: ld1 {T1.2d}, [x21], #16 + sub w19, w19, #1 -1: /* multiply XL by SHASH in GF(2^128) */ +2: /* multiply XL by SHASH in GF(2^128) */ CPU_LE( rev64 T1.16b, T1.16b ) ext T2.16b, XL.16b, XL.16b, #8 @@ -250,9 +259,18 @@ CPU_LE( rev64 T1.16b, T1.16b ) eor T2.16b, T2.16b, XH.16b eor XL.16b, XL.16b, T2.16b - cbnz w0, 0b + cbz w19, 3f + + if_will_cond_yield_neon + st1 {XL.2d}, [x20] + do_cond_yield_neon + b 0b + endif_yield_neon + + b 1b - st1 {XL.2d}, [x1] +3: st1 {XL.2d}, [x20] + frame_pop ret .endm @@ -304,38 +322,55 @@ ENDPROC(pmull_ghash_update_p8) .endm .macro pmull_gcm_do_crypt, enc - ld1 {SHASH.2d}, [x4] - ld1 {XL.2d}, [x1] - ldr x8, [x5, #8] // load lower counter + frame_push 10 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + mov x23, x4 + mov x24, x5 + mov x25, x6 + mov x26, x7 + .if \enc == 1 + ldr x27, [sp, #96] // first stacked arg + .endif + + ldr x28, [x24, #8] // load lower counter +CPU_LE( rev x28, x28 ) + +0: mov x0, x25 + load_round_keys w26, x0 + ld1 {SHASH.2d}, [x23] + ld1 {XL.2d}, [x20] movi MASK.16b, #0xe1 ext SHASH2.16b, SHASH.16b, SHASH.16b, #8 -CPU_LE( rev x8, x8 ) shl MASK.2d, MASK.2d, #57 eor SHASH2.16b, SHASH2.16b, SHASH.16b .if \enc == 1 - ld1 {KS.16b}, [x7] + ld1 {KS.16b}, [x27] .endif -0: ld1 {CTR.8b}, [x5] // load upper counter - ld1 {INP.16b}, [x3], #16 - rev x9, x8 - add x8, x8, #1 - sub w0, w0, #1 +1: ld1 {CTR.8b}, [x24] // load upper counter + ld1 {INP.16b}, [x22], #16 + rev x9, x28 + add x28, x28, #1 + sub w19, w19, #1 ins CTR.d[1], x9 // set lower counter .if \enc == 1 eor INP.16b, INP.16b, KS.16b // encrypt input - st1 {INP.16b}, [x2], #16 + st1 {INP.16b}, [x21], #16 .endif rev64 T1.16b, INP.16b - cmp w6, #12 - b.ge 2f // AES-192/256? + cmp w26, #12 + b.ge 4f // AES-192/256? -1: enc_round CTR, v21 +2: enc_round CTR, v21 ext T2.16b, XL.16b, XL.16b, #8 ext IN1.16b, T1.16b, T1.16b, #8 @@ -390,27 +425,39 @@ CPU_LE( rev x8, x8 ) .if \enc == 0 eor INP.16b, INP.16b, KS.16b - st1 {INP.16b}, [x2], #16 + st1 {INP.16b}, [x21], #16 .endif - cbnz w0, 0b + cbz w19, 3f -CPU_LE( rev x8, x8 ) - st1 {XL.2d}, [x1] - str x8, [x5, #8] // store lower counter + if_will_cond_yield_neon + st1 {XL.2d}, [x20] + .if \enc == 1 + st1 {KS.16b}, [x27] + .endif + do_cond_yield_neon + b 0b + endif_yield_neon + b 1b + +3: st1 {XL.2d}, [x20] .if \enc == 1 - st1 {KS.16b}, [x7] + st1 {KS.16b}, [x27] .endif +CPU_LE( rev x28, x28 ) + str x28, [x24, #8] // store lower counter + + frame_pop ret -2: b.eq 3f // AES-192? +4: b.eq 5f // AES-192? enc_round CTR, v17 enc_round CTR, v18 -3: enc_round CTR, v19 +5: enc_round CTR, v19 enc_round CTR, v20 - b 1b + b 2b .endm /* diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c index cfc9c92814fd..7cf0b1aa6ea8 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -63,11 +63,12 @@ static void (*pmull_ghash_update)(int blocks, u64 dg[], const char *src, asmlinkage void pmull_gcm_encrypt(int blocks, u64 dg[], u8 dst[], const u8 src[], struct ghash_key const *k, - u8 ctr[], int rounds, u8 ks[]); + u8 ctr[], u32 const rk[], int rounds, + u8 ks[]); asmlinkage void pmull_gcm_decrypt(int blocks, u64 dg[], u8 dst[], const u8 src[], struct ghash_key const *k, - u8 ctr[], int rounds); + u8 ctr[], u32 const rk[], int rounds); asmlinkage void pmull_gcm_encrypt_block(u8 dst[], u8 const src[], u32 const rk[], int rounds); @@ -368,26 +369,29 @@ static int gcm_encrypt(struct aead_request *req) pmull_gcm_encrypt_block(ks, iv, NULL, num_rounds(&ctx->aes_key)); put_unaligned_be32(3, iv + GCM_IV_SIZE); + kernel_neon_end(); - err = skcipher_walk_aead_encrypt(&walk, req, true); + err = skcipher_walk_aead_encrypt(&walk, req, false); while (walk.nbytes >= AES_BLOCK_SIZE) { int blocks = walk.nbytes / AES_BLOCK_SIZE; + kernel_neon_begin(); pmull_gcm_encrypt(blocks, dg, walk.dst.virt.addr, walk.src.virt.addr, &ctx->ghash_key, - iv, num_rounds(&ctx->aes_key), ks); + iv, ctx->aes_key.key_enc, + num_rounds(&ctx->aes_key), ks); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } - kernel_neon_end(); } else { __aes_arm64_encrypt(ctx->aes_key.key_enc, tag, iv, num_rounds(&ctx->aes_key)); put_unaligned_be32(2, iv + GCM_IV_SIZE); - err = skcipher_walk_aead_encrypt(&walk, req, true); + err = skcipher_walk_aead_encrypt(&walk, req, false); while (walk.nbytes >= AES_BLOCK_SIZE) { int blocks = walk.nbytes / AES_BLOCK_SIZE; @@ -467,15 +471,19 @@ static int gcm_decrypt(struct aead_request *req) pmull_gcm_encrypt_block(tag, iv, ctx->aes_key.key_enc, num_rounds(&ctx->aes_key)); put_unaligned_be32(2, iv + GCM_IV_SIZE); + kernel_neon_end(); - err = skcipher_walk_aead_decrypt(&walk, req, true); + err = skcipher_walk_aead_decrypt(&walk, req, false); while (walk.nbytes >= AES_BLOCK_SIZE) { int blocks = walk.nbytes / AES_BLOCK_SIZE; + kernel_neon_begin(); pmull_gcm_decrypt(blocks, dg, walk.dst.virt.addr, walk.src.virt.addr, &ctx->ghash_key, - iv, num_rounds(&ctx->aes_key)); + iv, ctx->aes_key.key_enc, + num_rounds(&ctx->aes_key)); + kernel_neon_end(); err = skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); @@ -483,14 +491,12 @@ static int gcm_decrypt(struct aead_request *req) if (walk.nbytes) pmull_gcm_encrypt_block(iv, iv, NULL, num_rounds(&ctx->aes_key)); - - kernel_neon_end(); } else { __aes_arm64_encrypt(ctx->aes_key.key_enc, tag, iv, num_rounds(&ctx->aes_key)); put_unaligned_be32(2, iv + GCM_IV_SIZE); - err = skcipher_walk_aead_decrypt(&walk, req, true); + err = skcipher_walk_aead_decrypt(&walk, req, false); while (walk.nbytes >= AES_BLOCK_SIZE) { int blocks = walk.nbytes / AES_BLOCK_SIZE; diff --git a/arch/arm64/crypto/sha1-ce-core.S b/arch/arm64/crypto/sha1-ce-core.S index 46049850727d..78eb35fb5056 100644 --- a/arch/arm64/crypto/sha1-ce-core.S +++ b/arch/arm64/crypto/sha1-ce-core.S @@ -69,30 +69,36 @@ * int blocks) */ ENTRY(sha1_ce_transform) + frame_push 3 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + /* load round constants */ - loadrc k0.4s, 0x5a827999, w6 +0: loadrc k0.4s, 0x5a827999, w6 loadrc k1.4s, 0x6ed9eba1, w6 loadrc k2.4s, 0x8f1bbcdc, w6 loadrc k3.4s, 0xca62c1d6, w6 /* load state */ - ld1 {dgav.4s}, [x0] - ldr dgb, [x0, #16] + ld1 {dgav.4s}, [x19] + ldr dgb, [x19, #16] /* load sha1_ce_state::finalize */ ldr_l w4, sha1_ce_offsetof_finalize, x4 - ldr w4, [x0, x4] + ldr w4, [x19, x4] /* load input */ -0: ld1 {v8.4s-v11.4s}, [x1], #64 - sub w2, w2, #1 +1: ld1 {v8.4s-v11.4s}, [x20], #64 + sub w21, w21, #1 CPU_LE( rev32 v8.16b, v8.16b ) CPU_LE( rev32 v9.16b, v9.16b ) CPU_LE( rev32 v10.16b, v10.16b ) CPU_LE( rev32 v11.16b, v11.16b ) -1: add t0.4s, v8.4s, k0.4s +2: add t0.4s, v8.4s, k0.4s mov dg0v.16b, dgav.16b add_update c, ev, k0, 8, 9, 10, 11, dgb @@ -123,16 +129,25 @@ CPU_LE( rev32 v11.16b, v11.16b ) add dgbv.2s, dgbv.2s, dg1v.2s add dgav.4s, dgav.4s, dg0v.4s - cbnz w2, 0b + cbz w21, 3f + + if_will_cond_yield_neon + st1 {dgav.4s}, [x19] + str dgb, [x19, #16] + do_cond_yield_neon + b 0b + endif_yield_neon + + b 1b /* * Final block: add padding and total bit count. * Skip if the input size was not a round multiple of the block size, * the padding is handled by the C code in that case. */ - cbz x4, 3f +3: cbz x4, 4f ldr_l w4, sha1_ce_offsetof_count, x4 - ldr x4, [x0, x4] + ldr x4, [x19, x4] movi v9.2d, #0 mov x8, #0x80000000 movi v10.2d, #0 @@ -141,10 +156,11 @@ CPU_LE( rev32 v11.16b, v11.16b ) mov x4, #0 mov v11.d[0], xzr mov v11.d[1], x7 - b 1b + b 2b /* store new state */ -3: st1 {dgav.4s}, [x0] - str dgb, [x0, #16] +4: st1 {dgav.4s}, [x19] + str dgb, [x19, #16] + frame_pop ret ENDPROC(sha1_ce_transform) diff --git a/arch/arm64/crypto/sha2-ce-core.S b/arch/arm64/crypto/sha2-ce-core.S index 4c3c89b812ce..cd8b36412469 100644 --- a/arch/arm64/crypto/sha2-ce-core.S +++ b/arch/arm64/crypto/sha2-ce-core.S @@ -79,30 +79,36 @@ */ .text ENTRY(sha2_ce_transform) + frame_push 3 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + /* load round constants */ - adr_l x8, .Lsha2_rcon +0: adr_l x8, .Lsha2_rcon ld1 { v0.4s- v3.4s}, [x8], #64 ld1 { v4.4s- v7.4s}, [x8], #64 ld1 { v8.4s-v11.4s}, [x8], #64 ld1 {v12.4s-v15.4s}, [x8] /* load state */ - ld1 {dgav.4s, dgbv.4s}, [x0] + ld1 {dgav.4s, dgbv.4s}, [x19] /* load sha256_ce_state::finalize */ ldr_l w4, sha256_ce_offsetof_finalize, x4 - ldr w4, [x0, x4] + ldr w4, [x19, x4] /* load input */ -0: ld1 {v16.4s-v19.4s}, [x1], #64 - sub w2, w2, #1 +1: ld1 {v16.4s-v19.4s}, [x20], #64 + sub w21, w21, #1 CPU_LE( rev32 v16.16b, v16.16b ) CPU_LE( rev32 v17.16b, v17.16b ) CPU_LE( rev32 v18.16b, v18.16b ) CPU_LE( rev32 v19.16b, v19.16b ) -1: add t0.4s, v16.4s, v0.4s +2: add t0.4s, v16.4s, v0.4s mov dg0v.16b, dgav.16b mov dg1v.16b, dgbv.16b @@ -131,16 +137,24 @@ CPU_LE( rev32 v19.16b, v19.16b ) add dgbv.4s, dgbv.4s, dg1v.4s /* handled all input blocks? */ - cbnz w2, 0b + cbz w21, 3f + + if_will_cond_yield_neon + st1 {dgav.4s, dgbv.4s}, [x19] + do_cond_yield_neon + b 0b + endif_yield_neon + + b 1b /* * Final block: add padding and total bit count. * Skip if the input size was not a round multiple of the block size, * the padding is handled by the C code in that case. */ - cbz x4, 3f +3: cbz x4, 4f ldr_l w4, sha256_ce_offsetof_count, x4 - ldr x4, [x0, x4] + ldr x4, [x19, x4] movi v17.2d, #0 mov x8, #0x80000000 movi v18.2d, #0 @@ -149,9 +163,10 @@ CPU_LE( rev32 v19.16b, v19.16b ) mov x4, #0 mov v19.d[0], xzr mov v19.d[1], x7 - b 1b + b 2b /* store new state */ -3: st1 {dgav.4s, dgbv.4s}, [x0] +4: st1 {dgav.4s, dgbv.4s}, [x19] + frame_pop ret ENDPROC(sha2_ce_transform) diff --git a/arch/arm64/crypto/sha256-core.S_shipped b/arch/arm64/crypto/sha256-core.S_shipped index 3ce82cc860bc..7c7ce2e3bad6 100644 --- a/arch/arm64/crypto/sha256-core.S_shipped +++ b/arch/arm64/crypto/sha256-core.S_shipped @@ -1,3 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0 + +// This code is taken from the OpenSSL project but the author (Andy Polyakov) +// has relicensed it under the GPLv2. Therefore this program is free software; +// you can redistribute it and/or modify it under the terms of the GNU General +// Public License version 2 as published by the Free Software Foundation. +// +// The original headers, including the original license headers, are +// included below for completeness. + // Copyright 2014-2016 The OpenSSL Project Authors. All Rights Reserved. // // Licensed under the OpenSSL license (the "License"). You may not use @@ -10,8 +20,6 @@ // project. The module is, however, dual licensed under OpenSSL and // CRYPTOGAMS licenses depending on where you obtain it. For further // details see http://www.openssl.org/~appro/cryptogams/. -// -// Permission to use under GPLv2 terms is granted. // ==================================================================== // // SHA256/512 for ARMv8. diff --git a/arch/arm64/crypto/sha3-ce-core.S b/arch/arm64/crypto/sha3-ce-core.S index 332ad7530690..a7d587fa54f6 100644 --- a/arch/arm64/crypto/sha3-ce-core.S +++ b/arch/arm64/crypto/sha3-ce-core.S @@ -41,9 +41,16 @@ */ .text ENTRY(sha3_ce_transform) - /* load state */ - add x8, x0, #32 - ld1 { v0.1d- v3.1d}, [x0] + frame_push 4 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + mov x22, x3 + +0: /* load state */ + add x8, x19, #32 + ld1 { v0.1d- v3.1d}, [x19] ld1 { v4.1d- v7.1d}, [x8], #32 ld1 { v8.1d-v11.1d}, [x8], #32 ld1 {v12.1d-v15.1d}, [x8], #32 @@ -51,13 +58,13 @@ ENTRY(sha3_ce_transform) ld1 {v20.1d-v23.1d}, [x8], #32 ld1 {v24.1d}, [x8] -0: sub w2, w2, #1 +1: sub w21, w21, #1 mov w8, #24 adr_l x9, .Lsha3_rcon /* load input */ - ld1 {v25.8b-v28.8b}, [x1], #32 - ld1 {v29.8b-v31.8b}, [x1], #24 + ld1 {v25.8b-v28.8b}, [x20], #32 + ld1 {v29.8b-v31.8b}, [x20], #24 eor v0.8b, v0.8b, v25.8b eor v1.8b, v1.8b, v26.8b eor v2.8b, v2.8b, v27.8b @@ -66,10 +73,10 @@ ENTRY(sha3_ce_transform) eor v5.8b, v5.8b, v30.8b eor v6.8b, v6.8b, v31.8b - tbnz x3, #6, 2f // SHA3-512 + tbnz x22, #6, 3f // SHA3-512 - ld1 {v25.8b-v28.8b}, [x1], #32 - ld1 {v29.8b-v30.8b}, [x1], #16 + ld1 {v25.8b-v28.8b}, [x20], #32 + ld1 {v29.8b-v30.8b}, [x20], #16 eor v7.8b, v7.8b, v25.8b eor v8.8b, v8.8b, v26.8b eor v9.8b, v9.8b, v27.8b @@ -77,34 +84,34 @@ ENTRY(sha3_ce_transform) eor v11.8b, v11.8b, v29.8b eor v12.8b, v12.8b, v30.8b - tbnz x3, #4, 1f // SHA3-384 or SHA3-224 + tbnz x22, #4, 2f // SHA3-384 or SHA3-224 // SHA3-256 - ld1 {v25.8b-v28.8b}, [x1], #32 + ld1 {v25.8b-v28.8b}, [x20], #32 eor v13.8b, v13.8b, v25.8b eor v14.8b, v14.8b, v26.8b eor v15.8b, v15.8b, v27.8b eor v16.8b, v16.8b, v28.8b - b 3f + b 4f -1: tbz x3, #2, 3f // bit 2 cleared? SHA-384 +2: tbz x22, #2, 4f // bit 2 cleared? SHA-384 // SHA3-224 - ld1 {v25.8b-v28.8b}, [x1], #32 - ld1 {v29.8b}, [x1], #8 + ld1 {v25.8b-v28.8b}, [x20], #32 + ld1 {v29.8b}, [x20], #8 eor v13.8b, v13.8b, v25.8b eor v14.8b, v14.8b, v26.8b eor v15.8b, v15.8b, v27.8b eor v16.8b, v16.8b, v28.8b eor v17.8b, v17.8b, v29.8b - b 3f + b 4f // SHA3-512 -2: ld1 {v25.8b-v26.8b}, [x1], #16 +3: ld1 {v25.8b-v26.8b}, [x20], #16 eor v7.8b, v7.8b, v25.8b eor v8.8b, v8.8b, v26.8b -3: sub w8, w8, #1 +4: sub w8, w8, #1 eor3 v29.16b, v4.16b, v9.16b, v14.16b eor3 v26.16b, v1.16b, v6.16b, v11.16b @@ -183,17 +190,33 @@ ENTRY(sha3_ce_transform) eor v0.16b, v0.16b, v31.16b - cbnz w8, 3b - cbnz w2, 0b + cbnz w8, 4b + cbz w21, 5f + + if_will_cond_yield_neon + add x8, x19, #32 + st1 { v0.1d- v3.1d}, [x19] + st1 { v4.1d- v7.1d}, [x8], #32 + st1 { v8.1d-v11.1d}, [x8], #32 + st1 {v12.1d-v15.1d}, [x8], #32 + st1 {v16.1d-v19.1d}, [x8], #32 + st1 {v20.1d-v23.1d}, [x8], #32 + st1 {v24.1d}, [x8] + do_cond_yield_neon + b 0b + endif_yield_neon + + b 1b /* save state */ - st1 { v0.1d- v3.1d}, [x0], #32 - st1 { v4.1d- v7.1d}, [x0], #32 - st1 { v8.1d-v11.1d}, [x0], #32 - st1 {v12.1d-v15.1d}, [x0], #32 - st1 {v16.1d-v19.1d}, [x0], #32 - st1 {v20.1d-v23.1d}, [x0], #32 - st1 {v24.1d}, [x0] +5: st1 { v0.1d- v3.1d}, [x19], #32 + st1 { v4.1d- v7.1d}, [x19], #32 + st1 { v8.1d-v11.1d}, [x19], #32 + st1 {v12.1d-v15.1d}, [x19], #32 + st1 {v16.1d-v19.1d}, [x19], #32 + st1 {v20.1d-v23.1d}, [x19], #32 + st1 {v24.1d}, [x19] + frame_pop ret ENDPROC(sha3_ce_transform) diff --git a/arch/arm64/crypto/sha512-armv8.pl b/arch/arm64/crypto/sha512-armv8.pl index c55efb308544..2d8655d5b1af 100644 --- a/arch/arm64/crypto/sha512-armv8.pl +++ b/arch/arm64/crypto/sha512-armv8.pl @@ -1,4 +1,14 @@ #! /usr/bin/env perl +# SPDX-License-Identifier: GPL-2.0 + +# This code is taken from the OpenSSL project but the author (Andy Polyakov) +# has relicensed it under the GPLv2. Therefore this program is free software; +# you can redistribute it and/or modify it under the terms of the GNU General +# Public License version 2 as published by the Free Software Foundation. +# +# The original headers, including the original license headers, are +# included below for completeness. + # Copyright 2014-2016 The OpenSSL Project Authors. All Rights Reserved. # # Licensed under the OpenSSL license (the "License"). You may not use @@ -11,8 +21,6 @@ # project. The module is, however, dual licensed under OpenSSL and # CRYPTOGAMS licenses depending on where you obtain it. For further # details see http://www.openssl.org/~appro/cryptogams/. -# -# Permission to use under GPLv2 terms is granted. # ==================================================================== # # SHA256/512 for ARMv8. diff --git a/arch/arm64/crypto/sha512-ce-core.S b/arch/arm64/crypto/sha512-ce-core.S index 7f3bca5c59a2..ce65e3abe4f2 100644 --- a/arch/arm64/crypto/sha512-ce-core.S +++ b/arch/arm64/crypto/sha512-ce-core.S @@ -107,17 +107,23 @@ */ .text ENTRY(sha512_ce_transform) + frame_push 3 + + mov x19, x0 + mov x20, x1 + mov x21, x2 + /* load state */ - ld1 {v8.2d-v11.2d}, [x0] +0: ld1 {v8.2d-v11.2d}, [x19] /* load first 4 round constants */ adr_l x3, .Lsha512_rcon ld1 {v20.2d-v23.2d}, [x3], #64 /* load input */ -0: ld1 {v12.2d-v15.2d}, [x1], #64 - ld1 {v16.2d-v19.2d}, [x1], #64 - sub w2, w2, #1 +1: ld1 {v12.2d-v15.2d}, [x20], #64 + ld1 {v16.2d-v19.2d}, [x20], #64 + sub w21, w21, #1 CPU_LE( rev64 v12.16b, v12.16b ) CPU_LE( rev64 v13.16b, v13.16b ) @@ -196,9 +202,18 @@ CPU_LE( rev64 v19.16b, v19.16b ) add v11.2d, v11.2d, v3.2d /* handled all input blocks? */ - cbnz w2, 0b + cbz w21, 3f + + if_will_cond_yield_neon + st1 {v8.2d-v11.2d}, [x19] + do_cond_yield_neon + b 0b + endif_yield_neon + + b 1b /* store new state */ -3: st1 {v8.2d-v11.2d}, [x0] +3: st1 {v8.2d-v11.2d}, [x19] + frame_pop ret ENDPROC(sha512_ce_transform) diff --git a/arch/arm64/crypto/sha512-core.S_shipped b/arch/arm64/crypto/sha512-core.S_shipped index bd0f59f06c9d..e063a6106720 100644 --- a/arch/arm64/crypto/sha512-core.S_shipped +++ b/arch/arm64/crypto/sha512-core.S_shipped @@ -1,3 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0 + +// This code is taken from the OpenSSL project but the author (Andy Polyakov) +// has relicensed it under the GPLv2. Therefore this program is free software; +// you can redistribute it and/or modify it under the terms of the GNU General +// Public License version 2 as published by the Free Software Foundation. +// +// The original headers, including the original license headers, are +// included below for completeness. + // Copyright 2014-2016 The OpenSSL Project Authors. All Rights Reserved. // // Licensed under the OpenSSL license (the "License"). You may not use @@ -10,8 +20,6 @@ // project. The module is, however, dual licensed under OpenSSL and // CRYPTOGAMS licenses depending on where you obtain it. For further // details see http://www.openssl.org/~appro/cryptogams/. -// -// Permission to use under GPLv2 terms is granted. // ==================================================================== // // SHA256/512 for ARMv8. diff --git a/arch/arm64/crypto/sm4-ce-core.S b/arch/arm64/crypto/sm4-ce-core.S new file mode 100644 index 000000000000..af3bfbc3f4d4 --- /dev/null +++ b/arch/arm64/crypto/sm4-ce-core.S @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include + + .irp b, 0, 1, 2, 3, 4, 5, 6, 7, 8 + .set .Lv\b\().4s, \b + .endr + + .macro sm4e, rd, rn + .inst 0xcec08400 | .L\rd | (.L\rn << 5) + .endm + + /* + * void sm4_ce_do_crypt(const u32 *rk, u32 *out, const u32 *in); + */ + .text +ENTRY(sm4_ce_do_crypt) + ld1 {v8.4s}, [x2] + ld1 {v0.4s-v3.4s}, [x0], #64 +CPU_LE( rev32 v8.16b, v8.16b ) + ld1 {v4.4s-v7.4s}, [x0] + sm4e v8.4s, v0.4s + sm4e v8.4s, v1.4s + sm4e v8.4s, v2.4s + sm4e v8.4s, v3.4s + sm4e v8.4s, v4.4s + sm4e v8.4s, v5.4s + sm4e v8.4s, v6.4s + sm4e v8.4s, v7.4s + rev64 v8.4s, v8.4s + ext v8.16b, v8.16b, v8.16b, #8 +CPU_LE( rev32 v8.16b, v8.16b ) + st1 {v8.4s}, [x1] + ret +ENDPROC(sm4_ce_do_crypt) diff --git a/arch/arm64/crypto/sm4-ce-glue.c b/arch/arm64/crypto/sm4-ce-glue.c new file mode 100644 index 000000000000..b7fb5274b250 --- /dev/null +++ b/arch/arm64/crypto/sm4-ce-glue.c @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include +#include +#include + +MODULE_ALIAS_CRYPTO("sm4"); +MODULE_ALIAS_CRYPTO("sm4-ce"); +MODULE_DESCRIPTION("SM4 symmetric cipher using ARMv8 Crypto Extensions"); +MODULE_AUTHOR("Ard Biesheuvel "); +MODULE_LICENSE("GPL v2"); + +asmlinkage void sm4_ce_do_crypt(const u32 *rk, void *out, const void *in); + +static void sm4_ce_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +{ + const struct crypto_sm4_ctx *ctx = crypto_tfm_ctx(tfm); + + if (!may_use_simd()) { + crypto_sm4_encrypt(tfm, out, in); + } else { + kernel_neon_begin(); + sm4_ce_do_crypt(ctx->rkey_enc, out, in); + kernel_neon_end(); + } +} + +static void sm4_ce_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +{ + const struct crypto_sm4_ctx *ctx = crypto_tfm_ctx(tfm); + + if (!may_use_simd()) { + crypto_sm4_decrypt(tfm, out, in); + } else { + kernel_neon_begin(); + sm4_ce_do_crypt(ctx->rkey_dec, out, in); + kernel_neon_end(); + } +} + +static struct crypto_alg sm4_ce_alg = { + .cra_name = "sm4", + .cra_driver_name = "sm4-ce", + .cra_priority = 200, + .cra_flags = CRYPTO_ALG_TYPE_CIPHER, + .cra_blocksize = SM4_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct crypto_sm4_ctx), + .cra_module = THIS_MODULE, + .cra_u.cipher = { + .cia_min_keysize = SM4_KEY_SIZE, + .cia_max_keysize = SM4_KEY_SIZE, + .cia_setkey = crypto_sm4_set_key, + .cia_encrypt = sm4_ce_encrypt, + .cia_decrypt = sm4_ce_decrypt + } +}; + +static int __init sm4_ce_mod_init(void) +{ + return crypto_register_alg(&sm4_ce_alg); +} + +static void __exit sm4_ce_mod_fini(void) +{ + crypto_unregister_alg(&sm4_ce_alg); +} + +module_cpu_feature_match(SM3, sm4_ce_mod_init); +module_exit(sm4_ce_mod_fini); diff --git a/arch/arm64/include/asm/acpi.h b/arch/arm64/include/asm/acpi.h index 32f465a80e4e..0db62a4cbce2 100644 --- a/arch/arm64/include/asm/acpi.h +++ b/arch/arm64/include/asm/acpi.h @@ -86,6 +86,10 @@ static inline bool acpi_has_cpu_in_madt(void) } struct acpi_madt_generic_interrupt *acpi_cpu_get_madt_gicc(int cpu); +static inline u32 get_acpi_id_for_cpu(unsigned int cpu) +{ + return acpi_cpu_get_madt_gicc(cpu)->uid; +} static inline void arch_fix_phys_package_id(int num, u32 slot) { } void __init acpi_init_cpus(void); diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h index 9bbffc7a301f..5df5cfe1c143 100644 --- a/arch/arm64/include/asm/cache.h +++ b/arch/arm64/include/asm/cache.h @@ -33,7 +33,7 @@ #define ICACHE_POLICY_VIPT 2 #define ICACHE_POLICY_PIPT 3 -#define L1_CACHE_SHIFT 7 +#define L1_CACHE_SHIFT (6) #define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) /* @@ -43,7 +43,7 @@ * cache before the transfer is done, causing old data to be seen by * the CPU. */ -#define ARCH_DMA_MINALIGN L1_CACHE_BYTES +#define ARCH_DMA_MINALIGN (128) #ifndef __ASSEMBLY__ @@ -77,7 +77,7 @@ static inline u32 cache_type_cwg(void) static inline int cache_line_size(void) { u32 cwg = cache_type_cwg(); - return cwg ? 4 << cwg : L1_CACHE_BYTES; + return cwg ? 4 << cwg : ARCH_DMA_MINALIGN; } #endif /* __ASSEMBLY__ */ diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/cacheflush.h index 0094c6653b06..d264a7274811 100644 --- a/arch/arm64/include/asm/cacheflush.h +++ b/arch/arm64/include/asm/cacheflush.h @@ -36,7 +36,7 @@ * Start addresses are inclusive and end addresses are exclusive; start * addresses should be rounded down, end addresses up. * - * See Documentation/cachetlb.txt for more information. Please note that + * See Documentation/core-api/cachetlb.rst for more information. Please note that * the implementation assumes non-aliasing VIPT D-cache and (aliasing) * VIPT I-cache. * diff --git a/arch/arm64/include/asm/cmpxchg.h b/arch/arm64/include/asm/cmpxchg.h index 4f5fd2a36e6e..3b0938281541 100644 --- a/arch/arm64/include/asm/cmpxchg.h +++ b/arch/arm64/include/asm/cmpxchg.h @@ -204,7 +204,9 @@ static inline void __cmpwait_case_##name(volatile void *ptr, \ unsigned long tmp; \ \ asm volatile( \ - " ldxr" #sz "\t%" #w "[tmp], %[v]\n" \ + " sevl\n" \ + " wfe\n" \ + " ldxr" #sz "\t%" #w "[tmp], %[v]\n" \ " eor %" #w "[tmp], %" #w "[tmp], %" #w "[val]\n" \ " cbnz %" #w "[tmp], 1f\n" \ " wfe\n" \ diff --git a/arch/arm64/include/asm/compat.h b/arch/arm64/include/asm/compat.h index c00c62e1a4a3..1a037b94eba1 100644 --- a/arch/arm64/include/asm/compat.h +++ b/arch/arm64/include/asm/compat.h @@ -34,7 +34,6 @@ typedef u32 compat_size_t; typedef s32 compat_ssize_t; -typedef s32 compat_time_t; typedef s32 compat_clock_t; typedef s32 compat_pid_t; typedef u16 __compat_uid_t; @@ -66,16 +65,6 @@ typedef u32 compat_ulong_t; typedef u64 compat_u64; typedef u32 compat_uptr_t; -struct compat_timespec { - compat_time_t tv_sec; - s32 tv_nsec; -}; - -struct compat_timeval { - compat_time_t tv_sec; - s32 tv_usec; -}; - struct compat_stat { #ifdef __AARCH64EB__ short st_dev; @@ -192,10 +181,10 @@ struct compat_ipc64_perm { struct compat_semid64_ds { struct compat_ipc64_perm sem_perm; - compat_time_t sem_otime; - compat_ulong_t __unused1; - compat_time_t sem_ctime; - compat_ulong_t __unused2; + compat_ulong_t sem_otime; + compat_ulong_t sem_otime_high; + compat_ulong_t sem_ctime; + compat_ulong_t sem_ctime_high; compat_ulong_t sem_nsems; compat_ulong_t __unused3; compat_ulong_t __unused4; @@ -203,12 +192,12 @@ struct compat_semid64_ds { struct compat_msqid64_ds { struct compat_ipc64_perm msg_perm; - compat_time_t msg_stime; - compat_ulong_t __unused1; - compat_time_t msg_rtime; - compat_ulong_t __unused2; - compat_time_t msg_ctime; - compat_ulong_t __unused3; + compat_ulong_t msg_stime; + compat_ulong_t msg_stime_high; + compat_ulong_t msg_rtime; + compat_ulong_t msg_rtime_high; + compat_ulong_t msg_ctime; + compat_ulong_t msg_ctime_high; compat_ulong_t msg_cbytes; compat_ulong_t msg_qnum; compat_ulong_t msg_qbytes; @@ -221,12 +210,12 @@ struct compat_msqid64_ds { struct compat_shmid64_ds { struct compat_ipc64_perm shm_perm; compat_size_t shm_segsz; - compat_time_t shm_atime; - compat_ulong_t __unused1; - compat_time_t shm_dtime; - compat_ulong_t __unused2; - compat_time_t shm_ctime; - compat_ulong_t __unused3; + compat_ulong_t shm_atime; + compat_ulong_t shm_atime_high; + compat_ulong_t shm_dtime; + compat_ulong_t shm_dtime_high; + compat_ulong_t shm_ctime; + compat_ulong_t shm_ctime_high; compat_pid_t shm_cpid; compat_pid_t shm_lpid; compat_ulong_t shm_nattch; diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index bc51b72fafd4..8a699c708fc9 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -48,7 +48,8 @@ #define ARM64_HAS_CACHE_IDC 27 #define ARM64_HAS_CACHE_DIC 28 #define ARM64_HW_DBM 29 +#define ARM64_SSBD 30 -#define ARM64_NCAPS 30 +#define ARM64_NCAPS 31 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 09b0f2a80c8f..1717ba1db35d 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -11,9 +11,7 @@ #include #include -#include #include -#include #include /* @@ -510,33 +508,28 @@ static inline bool system_supports_sve(void) cpus_have_const_cap(ARM64_SVE); } -/* - * Read the pseudo-ZCR used by cpufeatures to identify the supported SVE - * vector length. - * - * Use only if SVE is present. - * This function clobbers the SVE vector length. - */ -static inline u64 read_zcr_features(void) -{ - u64 zcr; - unsigned int vq_max; - - /* - * Set the maximum possible VL, and write zeroes to all other - * bits to see if they stick. - */ - sve_kernel_enable(NULL); - write_sysreg_s(ZCR_ELx_LEN_MASK, SYS_ZCR_EL1); - - zcr = read_sysreg_s(SYS_ZCR_EL1); - zcr &= ~(u64)ZCR_ELx_LEN_MASK; /* find sticky 1s outside LEN field */ - vq_max = sve_vq_from_vl(sve_get_vl()); - zcr |= vq_max - 1; /* set LEN field to maximum effective value */ +#define ARM64_SSBD_UNKNOWN -1 +#define ARM64_SSBD_FORCE_DISABLE 0 +#define ARM64_SSBD_KERNEL 1 +#define ARM64_SSBD_FORCE_ENABLE 2 +#define ARM64_SSBD_MITIGATED 3 - return zcr; +static inline int arm64_get_ssbd_state(void) +{ +#ifdef CONFIG_ARM64_SSBD + extern int ssbd_state; + return ssbd_state; +#else + return ARM64_SSBD_UNKNOWN; +#endif } +#ifdef CONFIG_ARM64_SSBD +void arm64_set_ssbd_mitigation(bool state); +#else +static inline void arm64_set_ssbd_mitigation(bool state) {} +#endif + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/elf.h b/arch/arm64/include/asm/elf.h index fac1c4de7898..433b9554c6a1 100644 --- a/arch/arm64/include/asm/elf.h +++ b/arch/arm64/include/asm/elf.h @@ -121,6 +121,9 @@ #ifndef __ASSEMBLY__ +#include +#include /* for signal_minsigstksz, used by ARCH_DLINFO */ + typedef unsigned long elf_greg_t; #define ELF_NGREG (sizeof(struct user_pt_regs) / sizeof(elf_greg_t)) @@ -148,6 +151,16 @@ typedef struct user_fpsimd_state elf_fpregset_t; do { \ NEW_AUX_ENT(AT_SYSINFO_EHDR, \ (elf_addr_t)current->mm->context.vdso); \ + \ + /* \ + * Should always be nonzero unless there's a kernel bug. \ + * If we haven't determined a sensible value to give to \ + * userspace, omit the entry: \ + */ \ + if (likely(signal_minsigstksz)) \ + NEW_AUX_ENT(AT_MINSIGSTKSZ, signal_minsigstksz); \ + else \ + NEW_AUX_ENT(AT_IGNORE, 0); \ } while (0) #define ARCH_HAS_SETUP_ADDITIONAL_PAGES diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index aa7162ae93e3..fa92747a49c8 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -18,6 +18,8 @@ #include #include +#include +#include #ifndef __ASSEMBLY__ @@ -41,6 +43,8 @@ struct task_struct; extern void fpsimd_save_state(struct user_fpsimd_state *state); extern void fpsimd_load_state(struct user_fpsimd_state *state); +extern void fpsimd_save(void); + extern void fpsimd_thread_switch(struct task_struct *next); extern void fpsimd_flush_thread(void); @@ -49,12 +53,27 @@ extern void fpsimd_preserve_current_state(void); extern void fpsimd_restore_current_state(void); extern void fpsimd_update_current_state(struct user_fpsimd_state const *state); +extern void fpsimd_bind_task_to_cpu(void); +extern void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *state); + extern void fpsimd_flush_task_state(struct task_struct *target); +extern void fpsimd_flush_cpu_state(void); extern void sve_flush_cpu_state(void); /* Maximum VL that SVE VL-agnostic software can transparently support */ #define SVE_VL_ARCH_MAX 0x100 +/* Offset of FFR in the SVE register dump */ +static inline size_t sve_ffr_offset(int vl) +{ + return SVE_SIG_FFR_OFFSET(sve_vq_from_vl(vl)) - SVE_SIG_REGS_OFFSET; +} + +static inline void *sve_pffr(struct thread_struct *thread) +{ + return (char *)thread->sve_state + sve_ffr_offset(thread->sve_vl); +} + extern void sve_save_state(void *state, u32 *pfpsr); extern void sve_load_state(void const *state, u32 const *pfpsr, unsigned long vq_minus_1); @@ -63,6 +82,8 @@ extern unsigned int sve_get_vl(void); struct arm64_cpu_capabilities; extern void sve_kernel_enable(const struct arm64_cpu_capabilities *__unused); +extern u64 read_zcr_features(void); + extern int __ro_after_init sve_max_vl; #ifdef CONFIG_ARM64_SVE diff --git a/arch/arm64/include/asm/fpsimdmacros.h b/arch/arm64/include/asm/fpsimdmacros.h index e050d765ca9e..46843515d77b 100644 --- a/arch/arm64/include/asm/fpsimdmacros.h +++ b/arch/arm64/include/asm/fpsimdmacros.h @@ -207,12 +207,14 @@ str w\nxtmp, [\xpfpsr, #4] .endm -.macro sve_load nxbase, xpfpsr, xvqminus1, nxtmp +.macro sve_load nxbase, xpfpsr, xvqminus1, nxtmp, xtmp2 mrs_s x\nxtmp, SYS_ZCR_EL1 - bic x\nxtmp, x\nxtmp, ZCR_ELx_LEN_MASK - orr x\nxtmp, x\nxtmp, \xvqminus1 - msr_s SYS_ZCR_EL1, x\nxtmp // self-synchronising - + bic \xtmp2, x\nxtmp, ZCR_ELx_LEN_MASK + orr \xtmp2, \xtmp2, \xvqminus1 + cmp \xtmp2, x\nxtmp + b.eq 921f + msr_s SYS_ZCR_EL1, \xtmp2 // self-synchronising +921: _for n, 0, 31, _sve_ldr_v \n, \nxbase, \n - 34 _sve_ldr_p 0, \nxbase _sve_wrffr 0 diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index f6648a3e4152..102b5a5c47b6 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -20,6 +20,9 @@ #include +#define VCPU_WORKAROUND_2_FLAG_SHIFT 0 +#define VCPU_WORKAROUND_2_FLAG (_AC(1, UL) << VCPU_WORKAROUND_2_FLAG_SHIFT) + #define ARM_EXIT_WITH_SERROR_BIT 31 #define ARM_EXCEPTION_CODE(x) ((x) & ~(1U << ARM_EXIT_WITH_SERROR_BIT)) #define ARM_SERROR_PENDING(x) !!((x) & (1U << ARM_EXIT_WITH_SERROR_BIT)) @@ -30,19 +33,19 @@ /* The hyp-stub will return this for any kvm_call_hyp() call */ #define ARM_EXCEPTION_HYP_GONE HVC_STUB_ERR -#define KVM_ARM64_DEBUG_DIRTY_SHIFT 0 -#define KVM_ARM64_DEBUG_DIRTY (1 << KVM_ARM64_DEBUG_DIRTY_SHIFT) +#ifndef __ASSEMBLY__ + +#include /* Translate a kernel address of @sym into its equivalent linear mapping */ #define kvm_ksym_ref(sym) \ ({ \ void *val = &sym; \ if (!is_kernel_in_hyp_mode()) \ - val = phys_to_virt((u64)&sym - kimage_voffset); \ + val = lm_alias(&sym); \ val; \ }) -#ifndef __ASSEMBLY__ struct kvm; struct kvm_vcpu; @@ -71,14 +74,37 @@ extern u32 __kvm_get_mdcr_el2(void); extern u32 __init_stage2_translation(void); +/* Home-grown __this_cpu_{ptr,read} variants that always work at HYP */ +#define __hyp_this_cpu_ptr(sym) \ + ({ \ + void *__ptr = hyp_symbol_addr(sym); \ + __ptr += read_sysreg(tpidr_el2); \ + (typeof(&sym))__ptr; \ + }) + +#define __hyp_this_cpu_read(sym) \ + ({ \ + *__hyp_this_cpu_ptr(sym); \ + }) + #else /* __ASSEMBLY__ */ -.macro get_host_ctxt reg, tmp - adr_l \reg, kvm_host_cpu_state +.macro hyp_adr_this_cpu reg, sym, tmp + adr_l \reg, \sym mrs \tmp, tpidr_el2 add \reg, \reg, \tmp .endm +.macro hyp_ldr_this_cpu reg, sym, tmp + adr_l \reg, \sym + mrs \tmp, tpidr_el2 + ldr \reg, [\reg, \tmp] +.endm + +.macro get_host_ctxt reg, tmp + hyp_adr_this_cpu \reg, kvm_host_cpu_state, \tmp +.endm + .macro get_vcpu_ptr vcpu, ctxt get_host_ctxt \ctxt, \vcpu ldr \vcpu, [\ctxt, #HOST_CONTEXT_VCPU] diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 469de8acd06f..fda9a8ca48be 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -30,6 +30,7 @@ #include #include #include +#include #define __KVM_HAVE_ARCH_INTC_INITIALIZED @@ -216,8 +217,11 @@ struct kvm_vcpu_arch { /* Exception Information */ struct kvm_vcpu_fault_info fault; - /* Guest debug state */ - u64 debug_flags; + /* State of various workarounds, see kvm_asm.h for bit assignment */ + u64 workaround_flags; + + /* Miscellaneous vcpu state flags */ + u64 flags; /* * We maintain more than a single set of debug registers to support @@ -238,6 +242,10 @@ struct kvm_vcpu_arch { /* Pointer to host CPU context */ kvm_cpu_context_t *host_cpu_context; + + struct thread_info *host_thread_info; /* hyp VA */ + struct user_fpsimd_state *host_fpsimd_state; /* hyp VA */ + struct { /* {Break,watch}point registers */ struct kvm_guest_debug_arch regs; @@ -293,6 +301,12 @@ struct kvm_vcpu_arch { bool sysregs_loaded_on_cpu; }; +/* vcpu_arch flags field values: */ +#define KVM_ARM64_DEBUG_DIRTY (1 << 0) +#define KVM_ARM64_FP_ENABLED (1 << 1) /* guest FP regs loaded */ +#define KVM_ARM64_FP_HOST (1 << 2) /* host FP regs loaded */ +#define KVM_ARM64_HOST_SVE_IN_USE (1 << 3) /* backup for host TIF_SVE */ + #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs) /* @@ -394,6 +408,19 @@ static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr, kvm_call_hyp(__kvm_set_tpidr_el2, tpidr_el2); } +static inline bool kvm_arch_check_sve_has_vhe(void) +{ + /* + * The Arm architecture specifies that implementation of SVE + * requires VHE also to be implemented. The KVM code for arm64 + * relies on this when SVE is present: + */ + if (system_supports_sve()) + return has_vhe(); + else + return true; +} + static inline void kvm_arch_hardware_unsetup(void) {} static inline void kvm_arch_sync_events(struct kvm *kvm) {} static inline void kvm_arch_vcpu_uninit(struct kvm_vcpu *vcpu) {} @@ -420,15 +447,18 @@ static inline void __cpu_init_stage2(void) "PARange is %d bits, unsupported configuration!", parange); } -/* - * All host FP/SIMD state is restored on guest exit, so nothing needs - * doing here except in the SVE case: -*/ -static inline void kvm_fpsimd_flush_cpu_state(void) +/* Guest/host FPSIMD coordination helpers */ +int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu); +void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu); +void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu); +void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu); + +#ifdef CONFIG_KVM /* Avoid conflicts with core headers if CONFIG_KVM=n */ +static inline int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu) { - if (system_supports_sve()) - sve_flush_cpu_state(); + return kvm_arch_vcpu_run_map_fp(vcpu); } +#endif static inline void kvm_arm_vhe_guest_enter(void) { @@ -452,7 +482,34 @@ static inline bool kvm_arm_harden_branch_predictor(void) return cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR); } +#define KVM_SSBD_UNKNOWN -1 +#define KVM_SSBD_FORCE_DISABLE 0 +#define KVM_SSBD_KERNEL 1 +#define KVM_SSBD_FORCE_ENABLE 2 +#define KVM_SSBD_MITIGATED 3 + +static inline int kvm_arm_have_ssbd(void) +{ + switch (arm64_get_ssbd_state()) { + case ARM64_SSBD_FORCE_DISABLE: + return KVM_SSBD_FORCE_DISABLE; + case ARM64_SSBD_KERNEL: + return KVM_SSBD_KERNEL; + case ARM64_SSBD_FORCE_ENABLE: + return KVM_SSBD_FORCE_ENABLE; + case ARM64_SSBD_MITIGATED: + return KVM_SSBD_MITIGATED; + case ARM64_SSBD_UNKNOWN: + default: + return KVM_SSBD_UNKNOWN; + } +} + void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu); void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu); +#define __KVM_HAVE_ARCH_VM_ALLOC +struct kvm *kvm_arch_alloc_vm(void); +void kvm_arch_free_vm(struct kvm *kvm); + #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 6128992c2ded..fb9a7127bb75 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -72,7 +72,6 @@ #ifdef __ASSEMBLY__ #include -#include /* * Convert a kernel VA into a HYP VA. @@ -473,6 +472,30 @@ static inline int kvm_map_vectors(void) } #endif +#ifdef CONFIG_ARM64_SSBD +DECLARE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required); + +static inline int hyp_map_aux_data(void) +{ + int cpu, err; + + for_each_possible_cpu(cpu) { + u64 *ptr; + + ptr = per_cpu_ptr(&arm64_ssbd_callback_required, cpu); + err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP); + if (err) + return err; + } + return 0; +} +#else +static inline int hyp_map_aux_data(void) +{ + return 0; +} +#endif + #define kvm_phys_to_vttbr(addr) phys_to_ttbr(addr) #endif /* __ASSEMBLY__ */ diff --git a/arch/arm64/include/asm/pci.h b/arch/arm64/include/asm/pci.h index 8747f7c5e0e7..9e690686e8aa 100644 --- a/arch/arm64/include/asm/pci.h +++ b/arch/arm64/include/asm/pci.h @@ -18,11 +18,6 @@ #define pcibios_assign_all_busses() \ (pci_has_flag(PCI_REASSIGN_ALL_BUS)) -/* - * PCI address space differs from physical memory address space - */ -#define PCI_DMA_BUS_IS_PHYS (0) - #define ARCH_GENERIC_PCI_MMAP_RESOURCE 1 extern int isa_dma_bridge_buggy; diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7c4c8f318ba9..9f82d6b53851 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -306,8 +306,6 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b) #define HPAGE_MASK (~(HPAGE_SIZE - 1)) #define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT) -#define __HAVE_ARCH_PTE_SPECIAL - static inline pte_t pgd_pte(pgd_t pgd) { return __pte(pgd_val(pgd)); diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 767598932549..a73ae1e49200 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -35,6 +35,8 @@ #ifdef __KERNEL__ #include +#include +#include #include #include @@ -156,7 +158,9 @@ static inline void arch_thread_struct_whitelist(unsigned long *offset, /* Sync TPIDR_EL0 back to thread_struct for current */ void tls_preserve_current_state(void); -#define INIT_THREAD { } +#define INIT_THREAD { \ + .fpsimd_cpu = NR_CPUS, \ +} static inline void start_thread_common(struct pt_regs *regs, unsigned long pc) { @@ -244,6 +248,20 @@ void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused); void cpu_enable_cache_maint_trap(const struct arm64_cpu_capabilities *__unused); void cpu_clear_disr(const struct arm64_cpu_capabilities *__unused); +extern unsigned long __ro_after_init signal_minsigstksz; /* sigframe size */ +extern void __init minsigstksz_setup(void); + +/* + * Not at the top of the file due to a direct #include cycle between + * and . Deferring this #include + * ensures that contents of processor.h are visible to fpsimd.h even if + * processor.h is included first. + * + * These prctl helpers are the only things in this file that require + * fpsimd.h. The core code expects them to be in this header. + */ +#include + /* Userspace interface for PR_SVE_{SET,GET}_VL prctl()s: */ #define SVE_SET_VL(arg) sve_set_current_vl(arg) #define SVE_GET_VL() sve_get_current_vl() diff --git a/arch/arm64/include/asm/spinlock.h b/arch/arm64/include/asm/spinlock.h index ebdae15d665d..26c5bd7d88d8 100644 --- a/arch/arm64/include/asm/spinlock.h +++ b/arch/arm64/include/asm/spinlock.h @@ -122,11 +122,6 @@ static inline int arch_spin_value_unlocked(arch_spinlock_t lock) static inline int arch_spin_is_locked(arch_spinlock_t *lock) { - /* - * Ensure prior spin_lock operations to other locks have completed - * on this CPU before we test whether "lock" is locked. - */ - smp_mb(); /* ^^^ */ return !arch_spin_value_unlocked(READ_ONCE(*lock)); } diff --git a/arch/arm64/include/asm/stat.h b/arch/arm64/include/asm/stat.h index 15e35598ac40..eab738019707 100644 --- a/arch/arm64/include/asm/stat.h +++ b/arch/arm64/include/asm/stat.h @@ -20,6 +20,7 @@ #ifdef CONFIG_COMPAT +#include #include /* diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index 740aa03c5f0d..cb2c10a8f0a8 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -45,12 +45,6 @@ struct thread_info { int preempt_count; /* 0 => preemptable, <0 => bug */ }; -#define INIT_THREAD_INFO(tsk) \ -{ \ - .preempt_count = INIT_PREEMPT_COUNT, \ - .addr_limit = KERNEL_DS, \ -} - #define thread_saved_pc(tsk) \ ((unsigned long)(tsk->thread.cpu_context.pc)) #define thread_saved_sp(tsk) \ @@ -94,6 +88,7 @@ void arch_release_task_struct(struct task_struct *tsk); #define TIF_32BIT 22 /* 32bit process */ #define TIF_SVE 23 /* Scalable Vector Extension in use */ #define TIF_SVE_VL_INHERIT 24 /* Inherit sve_vl_onexec across exec */ +#define TIF_SSBD 25 /* Wants SSB mitigation */ #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) @@ -117,5 +112,12 @@ void arch_release_task_struct(struct task_struct *tsk); _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \ _TIF_NOHZ) +#define INIT_THREAD_INFO(tsk) \ +{ \ + .flags = _TIF_FOREIGN_FPSTATE, \ + .preempt_count = INIT_PREEMPT_COUNT, \ + .addr_limit = KERNEL_DS, \ +} + #endif /* __KERNEL__ */ #endif /* __ASM_THREAD_INFO_H */ diff --git a/arch/arm64/include/asm/topology.h b/arch/arm64/include/asm/topology.h index c4f2d50491eb..df48212f767b 100644 --- a/arch/arm64/include/asm/topology.h +++ b/arch/arm64/include/asm/topology.h @@ -7,14 +7,16 @@ struct cpu_topology { int thread_id; int core_id; - int cluster_id; + int package_id; + int llc_id; cpumask_t thread_sibling; cpumask_t core_sibling; + cpumask_t llc_siblings; }; extern struct cpu_topology cpu_topology[NR_CPUS]; -#define topology_physical_package_id(cpu) (cpu_topology[cpu].cluster_id) +#define topology_physical_package_id(cpu) (cpu_topology[cpu].package_id) #define topology_core_id(cpu) (cpu_topology[cpu].core_id) #define topology_core_cpumask(cpu) (&cpu_topology[cpu].core_sibling) #define topology_sibling_cpumask(cpu) (&cpu_topology[cpu].thread_sibling) diff --git a/arch/arm64/include/uapi/asm/auxvec.h b/arch/arm64/include/uapi/asm/auxvec.h index ec0a86d484e1..743c0b84fd30 100644 --- a/arch/arm64/include/uapi/asm/auxvec.h +++ b/arch/arm64/include/uapi/asm/auxvec.h @@ -19,7 +19,8 @@ /* vDSO location */ #define AT_SYSINFO_EHDR 33 +#define AT_MINSIGSTKSZ 51 /* stack needed for signal delivery */ -#define AT_VECTOR_SIZE_ARCH 1 /* entries in ARCH_DLINFO */ +#define AT_VECTOR_SIZE_ARCH 2 /* entries in ARCH_DLINFO */ #endif diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index 04b3256f8e6d..4e76630dd655 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -91,6 +91,7 @@ struct kvm_regs { #define KVM_VGIC_V3_ADDR_TYPE_DIST 2 #define KVM_VGIC_V3_ADDR_TYPE_REDIST 3 #define KVM_VGIC_ITS_ADDR_TYPE 4 +#define KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION 5 #define KVM_VGIC_V3_DIST_SIZE SZ_64K #define KVM_VGIC_V3_REDIST_SIZE (2 * SZ_64K) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index bf825f38d206..0025f8691046 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -54,6 +54,7 @@ arm64-obj-$(CONFIG_ARM64_RELOC_TEST) += arm64-reloc-test.o arm64-reloc-test-y := reloc_test_core.o reloc_test_syms.o arm64-obj-$(CONFIG_CRASH_DUMP) += crash_dump.o arm64-obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o +arm64-obj-$(CONFIG_ARM64_SSBD) += ssbd.o obj-y += $(arm64-obj-y) vdso/ probes/ obj-m += $(arm64-obj-m) diff --git a/arch/arm64/kernel/armv8_deprecated.c b/arch/arm64/kernel/armv8_deprecated.c index 6e47fc3ab549..d4707abb2f16 100644 --- a/arch/arm64/kernel/armv8_deprecated.c +++ b/arch/arm64/kernel/armv8_deprecated.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include @@ -20,8 +21,6 @@ #include #include #include -#include -#include #define CREATE_TRACE_POINTS #include "trace-events-emulation.h" @@ -235,8 +234,8 @@ static void __init register_insn_emulation_sysctl(void) struct insn_emulation *insn; struct ctl_table *insns_sysctl, *sysctl; - insns_sysctl = kzalloc(sizeof(*sysctl) * (nr_insn_emulated + 1), - GFP_KERNEL); + insns_sysctl = kcalloc(nr_insn_emulated + 1, sizeof(*sysctl), + GFP_KERNEL); raw_spin_lock_irqsave(&insn_emulation_lock, flags); list_for_each_entry(insn, &insn_emulation, node) { diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 5bdda651bd05..323aeb5f2fe6 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -136,6 +136,7 @@ int main(void) #ifdef CONFIG_KVM_ARM_HOST DEFINE(VCPU_CONTEXT, offsetof(struct kvm_vcpu, arch.ctxt)); DEFINE(VCPU_FAULT_DISR, offsetof(struct kvm_vcpu, arch.fault.disr_el1)); + DEFINE(VCPU_WORKAROUND_FLAGS, offsetof(struct kvm_vcpu, arch.workaround_flags)); DEFINE(CPU_GP_REGS, offsetof(struct kvm_cpu_context, gp_regs)); DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_regs, regs)); DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs)); diff --git a/arch/arm64/kernel/cacheinfo.c b/arch/arm64/kernel/cacheinfo.c index 380f2e2fbed5..0bf0a835122f 100644 --- a/arch/arm64/kernel/cacheinfo.c +++ b/arch/arm64/kernel/cacheinfo.c @@ -17,6 +17,7 @@ * along with this program. If not, see . */ +#include #include #include @@ -46,7 +47,7 @@ static void ci_leaf_init(struct cacheinfo *this_leaf, static int __init_cache_level(unsigned int cpu) { - unsigned int ctype, level, leaves, of_level; + unsigned int ctype, level, leaves, fw_level; struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); for (level = 1, leaves = 0; level <= MAX_CACHE_LEVEL; level++) { @@ -59,15 +60,19 @@ static int __init_cache_level(unsigned int cpu) leaves += (ctype == CACHE_TYPE_SEPARATE) ? 2 : 1; } - of_level = of_find_last_cache_level(cpu); - if (level < of_level) { + if (acpi_disabled) + fw_level = of_find_last_cache_level(cpu); + else + fw_level = acpi_find_last_cache_level(cpu); + + if (level < fw_level) { /* * some external caches not specified in CLIDR_EL1 * the information may be available in the device tree * only unified external caches are considered here */ - leaves += (of_level - level); - level = of_level; + leaves += (fw_level - level); + level = fw_level; } this_cpu_ci->num_levels = level; diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index e4a1182deff7..1d2b6d768efe 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -16,6 +16,8 @@ * along with this program. If not, see . */ +#include +#include #include #include #include @@ -232,6 +234,178 @@ enable_smccc_arch_workaround_1(const struct arm64_cpu_capabilities *entry) } #endif /* CONFIG_HARDEN_BRANCH_PREDICTOR */ +#ifdef CONFIG_ARM64_SSBD +DEFINE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required); + +int ssbd_state __read_mostly = ARM64_SSBD_KERNEL; + +static const struct ssbd_options { + const char *str; + int state; +} ssbd_options[] = { + { "force-on", ARM64_SSBD_FORCE_ENABLE, }, + { "force-off", ARM64_SSBD_FORCE_DISABLE, }, + { "kernel", ARM64_SSBD_KERNEL, }, +}; + +static int __init ssbd_cfg(char *buf) +{ + int i; + + if (!buf || !buf[0]) + return -EINVAL; + + for (i = 0; i < ARRAY_SIZE(ssbd_options); i++) { + int len = strlen(ssbd_options[i].str); + + if (strncmp(buf, ssbd_options[i].str, len)) + continue; + + ssbd_state = ssbd_options[i].state; + return 0; + } + + return -EINVAL; +} +early_param("ssbd", ssbd_cfg); + +void __init arm64_update_smccc_conduit(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, + int nr_inst) +{ + u32 insn; + + BUG_ON(nr_inst != 1); + + switch (psci_ops.conduit) { + case PSCI_CONDUIT_HVC: + insn = aarch64_insn_get_hvc_value(); + break; + case PSCI_CONDUIT_SMC: + insn = aarch64_insn_get_smc_value(); + break; + default: + return; + } + + *updptr = cpu_to_le32(insn); +} + +void __init arm64_enable_wa2_handling(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, + int nr_inst) +{ + BUG_ON(nr_inst != 1); + /* + * Only allow mitigation on EL1 entry/exit and guest + * ARCH_WORKAROUND_2 handling if the SSBD state allows it to + * be flipped. + */ + if (arm64_get_ssbd_state() == ARM64_SSBD_KERNEL) + *updptr = cpu_to_le32(aarch64_insn_gen_nop()); +} + +void arm64_set_ssbd_mitigation(bool state) +{ + switch (psci_ops.conduit) { + case PSCI_CONDUIT_HVC: + arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_WORKAROUND_2, state, NULL); + break; + + case PSCI_CONDUIT_SMC: + arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, state, NULL); + break; + + default: + WARN_ON_ONCE(1); + break; + } +} + +static bool has_ssbd_mitigation(const struct arm64_cpu_capabilities *entry, + int scope) +{ + struct arm_smccc_res res; + bool required = true; + s32 val; + + WARN_ON(scope != SCOPE_LOCAL_CPU || preemptible()); + + if (psci_ops.smccc_version == SMCCC_VERSION_1_0) { + ssbd_state = ARM64_SSBD_UNKNOWN; + return false; + } + + switch (psci_ops.conduit) { + case PSCI_CONDUIT_HVC: + arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, + ARM_SMCCC_ARCH_WORKAROUND_2, &res); + break; + + case PSCI_CONDUIT_SMC: + arm_smccc_1_1_smc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, + ARM_SMCCC_ARCH_WORKAROUND_2, &res); + break; + + default: + ssbd_state = ARM64_SSBD_UNKNOWN; + return false; + } + + val = (s32)res.a0; + + switch (val) { + case SMCCC_RET_NOT_SUPPORTED: + ssbd_state = ARM64_SSBD_UNKNOWN; + return false; + + case SMCCC_RET_NOT_REQUIRED: + pr_info_once("%s mitigation not required\n", entry->desc); + ssbd_state = ARM64_SSBD_MITIGATED; + return false; + + case SMCCC_RET_SUCCESS: + required = true; + break; + + case 1: /* Mitigation not required on this CPU */ + required = false; + break; + + default: + WARN_ON(1); + return false; + } + + switch (ssbd_state) { + case ARM64_SSBD_FORCE_DISABLE: + pr_info_once("%s disabled from command-line\n", entry->desc); + arm64_set_ssbd_mitigation(false); + required = false; + break; + + case ARM64_SSBD_KERNEL: + if (required) { + __this_cpu_write(arm64_ssbd_callback_required, 1); + arm64_set_ssbd_mitigation(true); + } + break; + + case ARM64_SSBD_FORCE_ENABLE: + pr_info_once("%s forced from command-line\n", entry->desc); + arm64_set_ssbd_mitigation(true); + required = true; + break; + + default: + WARN_ON(1); + break; + } + + return required; +} +#endif /* CONFIG_ARM64_SSBD */ + #define CAP_MIDR_RANGE(model, v_min, r_min, v_max, r_max) \ .matches = is_affected_midr_range, \ .midr_range = MIDR_RANGE(model, v_min, r_min, v_max, r_max) @@ -487,6 +661,14 @@ const struct arm64_cpu_capabilities arm64_errata[] = { .type = ARM64_CPUCAP_LOCAL_CPU_ERRATUM, ERRATA_MIDR_RANGE_LIST(arm64_harden_el2_vectors), }, +#endif +#ifdef CONFIG_ARM64_SSBD + { + .desc = "Speculative Store Bypass Disable", + .capability = ARM64_SSBD, + .type = ARM64_CPUCAP_LOCAL_CPU_ERRATUM, + .matches = has_ssbd_mitigation, + }, #endif { } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 9d1b06d67c53..d2856b129097 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1606,7 +1606,6 @@ static void __init setup_system_capabilities(void) void __init setup_cpu_features(void) { u32 cwg; - int cls; setup_system_capabilities(); mark_const_caps_ready(); @@ -1619,6 +1618,7 @@ void __init setup_cpu_features(void) pr_info("emulated: Privileged Access Never (PAN) using TTBR0_EL1 switching\n"); sve_setup(); + minsigstksz_setup(); /* Advertise that we have computed the system capabilities */ set_sys_caps_initialised(); @@ -1627,13 +1627,9 @@ void __init setup_cpu_features(void) * Check for sane CTR_EL0.CWG value. */ cwg = cache_type_cwg(); - cls = cache_line_size(); if (!cwg) - pr_warn("No Cache Writeback Granule information, assuming cache line size %d\n", - cls); - if (L1_CACHE_BYTES < cls) - pr_warn("L1_CACHE_BYTES smaller than the Cache Writeback Granule (%d < %d)\n", - L1_CACHE_BYTES, cls); + pr_warn("No Cache Writeback Granule information, assuming %d\n", + ARCH_DMA_MINALIGN); } static bool __maybe_unused diff --git a/arch/arm64/kernel/entry-fpsimd.S b/arch/arm64/kernel/entry-fpsimd.S index 73f17bffcd23..12d4958e6429 100644 --- a/arch/arm64/kernel/entry-fpsimd.S +++ b/arch/arm64/kernel/entry-fpsimd.S @@ -49,7 +49,7 @@ ENTRY(sve_save_state) ENDPROC(sve_save_state) ENTRY(sve_load_state) - sve_load 0, x1, x2, 3 + sve_load 0, x1, x2, 3, x4 ret ENDPROC(sve_load_state) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index ec2ee720e33e..28ad8799406f 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -18,6 +18,7 @@ * along with this program. If not, see . */ +#include #include #include @@ -137,6 +138,25 @@ alternative_else_nop_endif add \dst, \dst, #(\sym - .entry.tramp.text) .endm + // This macro corrupts x0-x3. It is the caller's duty + // to save/restore them if required. + .macro apply_ssbd, state, targ, tmp1, tmp2 +#ifdef CONFIG_ARM64_SSBD +alternative_cb arm64_enable_wa2_handling + b \targ +alternative_cb_end + ldr_this_cpu \tmp2, arm64_ssbd_callback_required, \tmp1 + cbz \tmp2, \targ + ldr \tmp2, [tsk, #TSK_TI_FLAGS] + tbnz \tmp2, #TIF_SSBD, \targ + mov w0, #ARM_SMCCC_ARCH_WORKAROUND_2 + mov w1, #\state +alternative_cb arm64_update_smccc_conduit + nop // Patched to SMC/HVC #0 +alternative_cb_end +#endif + .endm + .macro kernel_entry, el, regsize = 64 .if \regsize == 32 mov w0, w0 // zero upper 32 bits of x0 @@ -163,6 +183,14 @@ alternative_else_nop_endif ldr x19, [tsk, #TSK_TI_FLAGS] // since we can unmask debug disable_step_tsk x19, x20 // exceptions when scheduling. + apply_ssbd 1, 1f, x22, x23 + +#ifdef CONFIG_ARM64_SSBD + ldp x0, x1, [sp, #16 * 0] + ldp x2, x3, [sp, #16 * 1] +#endif +1: + mov x29, xzr // fp pointed to user-space .else add x21, sp, #S_FRAME_SIZE @@ -303,6 +331,8 @@ alternative_if ARM64_WORKAROUND_845719 alternative_else_nop_endif #endif 3: + apply_ssbd 0, 5f, x0, x1 +5: .endif msr elr_el1, x21 // set up the return data diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 87a35364e750..84c68b14f1b2 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -31,18 +31,19 @@ #include #include #include -#include #include #include #include #include #include +#include #include #include #include #include #include +#include #include #include #include @@ -118,7 +119,6 @@ */ struct fpsimd_last_state_struct { struct user_fpsimd_state *st; - bool sve_in_use; }; static DEFINE_PER_CPU(struct fpsimd_last_state_struct, fpsimd_last_state); @@ -129,7 +129,7 @@ static int sve_default_vl = -1; #ifdef CONFIG_ARM64_SVE /* Maximum supported vector length across all CPUs (initially poisoned) */ -int __ro_after_init sve_max_vl = -1; +int __ro_after_init sve_max_vl = SVE_VL_MIN; /* Set of available vector lengths, as vq_to_bit(vq): */ static __ro_after_init DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); static void __percpu *efi_sve_state; @@ -159,19 +159,6 @@ static void sve_free(struct task_struct *task) __sve_free(task); } - -/* Offset of FFR in the SVE register dump */ -static size_t sve_ffr_offset(int vl) -{ - return SVE_SIG_FFR_OFFSET(sve_vq_from_vl(vl)) - SVE_SIG_REGS_OFFSET; -} - -static void *sve_pffr(struct task_struct *task) -{ - return (char *)task->thread.sve_state + - sve_ffr_offset(task->thread.sve_vl); -} - static void change_cpacr(u64 val, u64 mask) { u64 cpacr = read_sysreg(CPACR_EL1); @@ -252,31 +239,24 @@ static void task_fpsimd_load(void) WARN_ON(!in_softirq() && !irqs_disabled()); if (system_supports_sve() && test_thread_flag(TIF_SVE)) - sve_load_state(sve_pffr(current), + sve_load_state(sve_pffr(¤t->thread), ¤t->thread.uw.fpsimd_state.fpsr, sve_vq_from_vl(current->thread.sve_vl) - 1); else fpsimd_load_state(¤t->thread.uw.fpsimd_state); - - if (system_supports_sve()) { - /* Toggle SVE trapping for userspace if needed */ - if (test_thread_flag(TIF_SVE)) - sve_user_enable(); - else - sve_user_disable(); - - /* Serialised by exception return to user */ - } } /* - * Ensure current's FPSIMD/SVE storage in thread_struct is up to date - * with respect to the CPU registers. + * Ensure FPSIMD/SVE storage in memory for the loaded context is up to + * date with respect to the CPU registers. * * Softirqs (and preemption) must be disabled. */ -static void task_fpsimd_save(void) +void fpsimd_save(void) { + struct user_fpsimd_state *st = __this_cpu_read(fpsimd_last_state.st); + /* set by fpsimd_bind_task_to_cpu() or fpsimd_bind_state_to_cpu() */ + WARN_ON(!in_softirq() && !irqs_disabled()); if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) { @@ -291,10 +271,9 @@ static void task_fpsimd_save(void) return; } - sve_save_state(sve_pffr(current), - ¤t->thread.uw.fpsimd_state.fpsr); + sve_save_state(sve_pffr(¤t->thread), &st->fpsr); } else - fpsimd_save_state(¤t->thread.uw.fpsimd_state); + fpsimd_save_state(st); } } @@ -360,22 +339,13 @@ static int sve_proc_do_default_vl(struct ctl_table *table, int write, return ret; /* Writing -1 has the special meaning "set to max": */ - if (vl == -1) { - /* Fail safe if sve_max_vl wasn't initialised */ - if (WARN_ON(!sve_vl_valid(sve_max_vl))) - vl = SVE_VL_MIN; - else - vl = sve_max_vl; - - goto chosen; - } + if (vl == -1) + vl = sve_max_vl; if (!sve_vl_valid(vl)) return -EINVAL; - vl = find_supported_vector_length(vl); -chosen: - sve_default_vl = vl; + sve_default_vl = find_supported_vector_length(vl); return 0; } @@ -598,7 +568,7 @@ int sve_set_vector_length(struct task_struct *task, if (task == current) { local_bh_disable(); - task_fpsimd_save(); + fpsimd_save(); set_thread_flag(TIF_FOREIGN_FPSTATE); } @@ -618,10 +588,8 @@ int sve_set_vector_length(struct task_struct *task, task->thread.sve_vl = vl; out: - if (flags & PR_SVE_VL_INHERIT) - set_tsk_thread_flag(task, TIF_SVE_VL_INHERIT); - else - clear_tsk_thread_flag(task, TIF_SVE_VL_INHERIT); + update_tsk_thread_flag(task, TIF_SVE_VL_INHERIT, + flags & PR_SVE_VL_INHERIT); return 0; } @@ -765,6 +733,33 @@ void sve_kernel_enable(const struct arm64_cpu_capabilities *__always_unused p) isb(); } +/* + * Read the pseudo-ZCR used by cpufeatures to identify the supported SVE + * vector length. + * + * Use only if SVE is present. + * This function clobbers the SVE vector length. + */ +u64 read_zcr_features(void) +{ + u64 zcr; + unsigned int vq_max; + + /* + * Set the maximum possible VL, and write zeroes to all other + * bits to see if they stick. + */ + sve_kernel_enable(NULL); + write_sysreg_s(ZCR_ELx_LEN_MASK, SYS_ZCR_EL1); + + zcr = read_sysreg_s(SYS_ZCR_EL1); + zcr &= ~(u64)ZCR_ELx_LEN_MASK; /* find sticky 1s outside LEN field */ + vq_max = sve_vq_from_vl(sve_get_vl()); + zcr |= vq_max - 1; /* set LEN field to maximum effective value */ + + return zcr; +} + void __init sve_setup(void) { u64 zcr; @@ -839,7 +834,7 @@ asmlinkage void do_sve_acc(unsigned int esr, struct pt_regs *regs) local_bh_disable(); - task_fpsimd_save(); + fpsimd_save(); fpsimd_to_sve(current); /* Force ret_to_user to reload the registers: */ @@ -882,7 +877,7 @@ asmlinkage void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs) si_code = FPE_FLTRES; } - memset(&info, 0, sizeof(info)); + clear_siginfo(&info); info.si_signo = SIGFPE; info.si_code = si_code; info.si_addr = (void __user *)instruction_pointer(regs); @@ -892,31 +887,25 @@ asmlinkage void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs) void fpsimd_thread_switch(struct task_struct *next) { + bool wrong_task, wrong_cpu; + if (!system_supports_fpsimd()) return; + + /* Save unsaved fpsimd state, if any: */ + fpsimd_save(); + /* - * Save the current FPSIMD state to memory, but only if whatever is in - * the registers is in fact the most recent userland FPSIMD state of - * 'current'. + * Fix up TIF_FOREIGN_FPSTATE to correctly describe next's + * state. For kernel threads, FPSIMD registers are never loaded + * and wrong_task and wrong_cpu will always be true. */ - if (current->mm) - task_fpsimd_save(); + wrong_task = __this_cpu_read(fpsimd_last_state.st) != + &next->thread.uw.fpsimd_state; + wrong_cpu = next->thread.fpsimd_cpu != smp_processor_id(); - if (next->mm) { - /* - * If we are switching to a task whose most recent userland - * FPSIMD state is already in the registers of *this* cpu, - * we can skip loading the state from memory. Otherwise, set - * the TIF_FOREIGN_FPSTATE flag so the state will be loaded - * upon the next return to userland. - */ - if (__this_cpu_read(fpsimd_last_state.st) == - &next->thread.uw.fpsimd_state - && next->thread.fpsimd_cpu == smp_processor_id()) - clear_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE); - else - set_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE); - } + update_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE, + wrong_task || wrong_cpu); } void fpsimd_flush_thread(void) @@ -982,7 +971,7 @@ void fpsimd_preserve_current_state(void) return; local_bh_disable(); - task_fpsimd_save(); + fpsimd_save(); local_bh_enable(); } @@ -1002,14 +991,33 @@ void fpsimd_signal_preserve_current_state(void) * Associate current's FPSIMD context with this cpu * Preemption must be disabled when calling this function. */ -static void fpsimd_bind_to_cpu(void) +void fpsimd_bind_task_to_cpu(void) { struct fpsimd_last_state_struct *last = this_cpu_ptr(&fpsimd_last_state); last->st = ¤t->thread.uw.fpsimd_state; - last->sve_in_use = test_thread_flag(TIF_SVE); current->thread.fpsimd_cpu = smp_processor_id(); + + if (system_supports_sve()) { + /* Toggle SVE trapping for userspace if needed */ + if (test_thread_flag(TIF_SVE)) + sve_user_enable(); + else + sve_user_disable(); + + /* Serialised by exception return to user */ + } +} + +void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *st) +{ + struct fpsimd_last_state_struct *last = + this_cpu_ptr(&fpsimd_last_state); + + WARN_ON(!in_softirq() && !irqs_disabled()); + + last->st = st; } /* @@ -1026,7 +1034,7 @@ void fpsimd_restore_current_state(void) if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) { task_fpsimd_load(); - fpsimd_bind_to_cpu(); + fpsimd_bind_task_to_cpu(); } local_bh_enable(); @@ -1049,9 +1057,9 @@ void fpsimd_update_current_state(struct user_fpsimd_state const *state) fpsimd_to_sve(current); task_fpsimd_load(); + fpsimd_bind_task_to_cpu(); - if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) - fpsimd_bind_to_cpu(); + clear_thread_flag(TIF_FOREIGN_FPSTATE); local_bh_enable(); } @@ -1064,29 +1072,12 @@ void fpsimd_flush_task_state(struct task_struct *t) t->thread.fpsimd_cpu = NR_CPUS; } -static inline void fpsimd_flush_cpu_state(void) +void fpsimd_flush_cpu_state(void) { __this_cpu_write(fpsimd_last_state.st, NULL); + set_thread_flag(TIF_FOREIGN_FPSTATE); } -/* - * Invalidate any task SVE state currently held in this CPU's regs. - * - * This is used to prevent the kernel from trying to reuse SVE register data - * that is detroyed by KVM guest enter/exit. This function should go away when - * KVM SVE support is implemented. Don't use it for anything else. - */ -#ifdef CONFIG_ARM64_SVE -void sve_flush_cpu_state(void) -{ - struct fpsimd_last_state_struct const *last = - this_cpu_ptr(&fpsimd_last_state); - - if (last->st && last->sve_in_use) - fpsimd_flush_cpu_state(); -} -#endif /* CONFIG_ARM64_SVE */ - #ifdef CONFIG_KERNEL_MODE_NEON DEFINE_PER_CPU(bool, kernel_neon_busy); @@ -1120,11 +1111,8 @@ void kernel_neon_begin(void) __this_cpu_write(kernel_neon_busy, true); - /* Save unsaved task fpsimd state, if any: */ - if (current->mm) { - task_fpsimd_save(); - set_thread_flag(TIF_FOREIGN_FPSTATE); - } + /* Save unsaved fpsimd state, if any: */ + fpsimd_save(); /* Invalidate any task state remaining in the fpsimd regs: */ fpsimd_flush_cpu_state(); @@ -1246,13 +1234,10 @@ static int fpsimd_cpu_pm_notifier(struct notifier_block *self, { switch (cmd) { case CPU_PM_ENTER: - if (current->mm) - task_fpsimd_save(); + fpsimd_save(); fpsimd_flush_cpu_state(); break; case CPU_PM_EXIT: - if (current->mm) - set_thread_flag(TIF_FOREIGN_FPSTATE); break; case CPU_PM_ENTER_FAILED: default: diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 1ec5f28c39fc..6b2686d54411 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -313,6 +313,17 @@ int swsusp_arch_suspend(void) sleep_cpu = -EINVAL; __cpu_suspend_exit(); + + /* + * Just in case the boot kernel did turn the SSBD + * mitigation off behind our back, let's set the state + * to what we expect it to be. + */ + switch (arm64_get_ssbd_state()) { + case ARM64_SSBD_FORCE_ENABLE: + case ARM64_SSBD_KERNEL: + arm64_set_ssbd_mitigation(true); + } } local_daif_restore(flags); diff --git a/arch/arm64/kernel/hw_breakpoint.c b/arch/arm64/kernel/hw_breakpoint.c index 74bb56f656ef..413dbe530da8 100644 --- a/arch/arm64/kernel/hw_breakpoint.c +++ b/arch/arm64/kernel/hw_breakpoint.c @@ -30,7 +30,6 @@ #include #include -#include #include #include #include diff --git a/arch/arm64/kernel/perf_event.c b/arch/arm64/kernel/perf_event.c index 85a251b6dfa8..33147aacdafd 100644 --- a/arch/arm64/kernel/perf_event.c +++ b/arch/arm64/kernel/perf_event.c @@ -670,11 +670,10 @@ static void armv8pmu_disable_event(struct perf_event *event) raw_spin_unlock_irqrestore(&events->pmu_lock, flags); } -static irqreturn_t armv8pmu_handle_irq(int irq_num, void *dev) +static irqreturn_t armv8pmu_handle_irq(struct arm_pmu *cpu_pmu) { u32 pmovsr; struct perf_sample_data data; - struct arm_pmu *cpu_pmu = (struct arm_pmu *)dev; struct pmu_hw_events *cpuc = this_cpu_ptr(cpu_pmu->hw_events); struct pt_regs *regs; int idx; diff --git a/arch/arm64/kernel/perf_regs.c b/arch/arm64/kernel/perf_regs.c index 1d091d048d04..0bbac612146e 100644 --- a/arch/arm64/kernel/perf_regs.c +++ b/arch/arm64/kernel/perf_regs.c @@ -1,11 +1,11 @@ // SPDX-License-Identifier: GPL-2.0 +#include #include #include #include #include #include -#include #include #include diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index f08a2ed9db0d..e10bc363f533 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -59,7 +59,7 @@ #include #include -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR #include unsigned long __stack_chk_guard __read_mostly; EXPORT_SYMBOL(__stack_chk_guard); diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c index 7ff81fed46e1..5c338ce5a7fa 100644 --- a/arch/arm64/kernel/ptrace.c +++ b/arch/arm64/kernel/ptrace.c @@ -44,6 +44,7 @@ #include #include #include +#include #include #include #include @@ -766,9 +767,6 @@ static void sve_init_header_from_task(struct user_sve_header *header, vq = sve_vq_from_vl(header->vl); header->max_vl = sve_max_vl; - if (WARN_ON(!sve_vl_valid(sve_max_vl))) - header->max_vl = header->vl; - header->size = SVE_PT_SIZE(vq, header->flags); header->max_size = SVE_PT_SIZE(sve_vq_from_vl(header->max_vl), SVE_PT_REGS_SVE); @@ -1046,8 +1044,6 @@ static const struct user_regset_view user_aarch64_view = { }; #ifdef CONFIG_COMPAT -#include - enum compat_regset { REGSET_COMPAT_GPR, REGSET_COMPAT_VFP, diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c index 154b7d30145d..511af13e8d8f 100644 --- a/arch/arm64/kernel/signal.c +++ b/arch/arm64/kernel/signal.c @@ -17,6 +17,7 @@ * along with this program. If not, see . */ +#include #include #include #include @@ -570,8 +571,15 @@ badframe: return 0; } -/* Determine the layout of optional records in the signal frame */ -static int setup_sigframe_layout(struct rt_sigframe_user_layout *user) +/* + * Determine the layout of optional records in the signal frame + * + * add_all: if true, lays out the biggest possible signal frame for + * this task; otherwise, generates a layout for the current state + * of the task. + */ +static int setup_sigframe_layout(struct rt_sigframe_user_layout *user, + bool add_all) { int err; @@ -581,7 +589,7 @@ static int setup_sigframe_layout(struct rt_sigframe_user_layout *user) return err; /* fault information, if valid */ - if (current->thread.fault_code) { + if (add_all || current->thread.fault_code) { err = sigframe_alloc(user, &user->esr_offset, sizeof(struct esr_context)); if (err) @@ -591,8 +599,14 @@ static int setup_sigframe_layout(struct rt_sigframe_user_layout *user) if (system_supports_sve()) { unsigned int vq = 0; - if (test_thread_flag(TIF_SVE)) - vq = sve_vq_from_vl(current->thread.sve_vl); + if (add_all || test_thread_flag(TIF_SVE)) { + int vl = sve_max_vl; + + if (!add_all) + vl = current->thread.sve_vl; + + vq = sve_vq_from_vl(vl); + } err = sigframe_alloc(user, &user->sve_offset, SVE_SIG_CONTEXT_SIZE(vq)); @@ -603,7 +617,6 @@ static int setup_sigframe_layout(struct rt_sigframe_user_layout *user) return sigframe_alloc_end(user); } - static int setup_sigframe(struct rt_sigframe_user_layout *user, struct pt_regs *regs, sigset_t *set) { @@ -701,7 +714,7 @@ static int get_sigframe(struct rt_sigframe_user_layout *user, int err; init_user_layout(user); - err = setup_sigframe_layout(user); + err = setup_sigframe_layout(user, false); if (err) return err; @@ -830,11 +843,12 @@ static void do_signal(struct pt_regs *regs) unsigned long continue_addr = 0, restart_addr = 0; int retval = 0; struct ksignal ksig; + bool syscall = in_syscall(regs); /* * If we were from a system call, check for system call restarting... */ - if (in_syscall(regs)) { + if (syscall) { continue_addr = regs->pc; restart_addr = continue_addr - (compat_thumb_mode(regs) ? 2 : 4); retval = regs->regs[0]; @@ -886,7 +900,7 @@ static void do_signal(struct pt_regs *regs) * Handle restarting a different system call. As above, if a debugger * has chosen to restart at a different PC, ignore the restart. */ - if (in_syscall(regs) && regs->pc == restart_addr) { + if (syscall && regs->pc == restart_addr) { if (retval == -ERESTART_RESTARTBLOCK) setup_restart_syscall(regs); user_rewind_single_step(current); @@ -936,3 +950,28 @@ asmlinkage void do_notify_resume(struct pt_regs *regs, thread_flags = READ_ONCE(current_thread_info()->flags); } while (thread_flags & _TIF_WORK_MASK); } + +unsigned long __ro_after_init signal_minsigstksz; + +/* + * Determine the stack space required for guaranteed signal devliery. + * This function is used to populate AT_MINSIGSTKSZ at process startup. + * cpufeatures setup is assumed to be complete. + */ +void __init minsigstksz_setup(void) +{ + struct rt_sigframe_user_layout user; + + init_user_layout(&user); + + /* + * If this fails, SIGFRAME_MAXSZ needs to be enlarged. It won't + * be big enough, but it's our best guess: + */ + if (WARN_ON(setup_sigframe_layout(&user, true))) + return; + + signal_minsigstksz = sigframe_size(&user) + + round_up(sizeof(struct frame_record), 16) + + 16; /* max alignment padding */ +} diff --git a/arch/arm64/kernel/ssbd.c b/arch/arm64/kernel/ssbd.c new file mode 100644 index 000000000000..3432e5ef9f41 --- /dev/null +++ b/arch/arm64/kernel/ssbd.c @@ -0,0 +1,110 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2018 ARM Ltd, All Rights Reserved. + */ + +#include +#include +#include + +#include + +/* + * prctl interface for SSBD + * FIXME: Drop the below ifdefery once merged in 4.18. + */ +#ifdef PR_SPEC_STORE_BYPASS +static int ssbd_prctl_set(struct task_struct *task, unsigned long ctrl) +{ + int state = arm64_get_ssbd_state(); + + /* Unsupported */ + if (state == ARM64_SSBD_UNKNOWN) + return -EINVAL; + + /* Treat the unaffected/mitigated state separately */ + if (state == ARM64_SSBD_MITIGATED) { + switch (ctrl) { + case PR_SPEC_ENABLE: + return -EPERM; + case PR_SPEC_DISABLE: + case PR_SPEC_FORCE_DISABLE: + return 0; + } + } + + /* + * Things are a bit backward here: the arm64 internal API + * *enables the mitigation* when the userspace API *disables + * speculation*. So much fun. + */ + switch (ctrl) { + case PR_SPEC_ENABLE: + /* If speculation is force disabled, enable is not allowed */ + if (state == ARM64_SSBD_FORCE_ENABLE || + task_spec_ssb_force_disable(task)) + return -EPERM; + task_clear_spec_ssb_disable(task); + clear_tsk_thread_flag(task, TIF_SSBD); + break; + case PR_SPEC_DISABLE: + if (state == ARM64_SSBD_FORCE_DISABLE) + return -EPERM; + task_set_spec_ssb_disable(task); + set_tsk_thread_flag(task, TIF_SSBD); + break; + case PR_SPEC_FORCE_DISABLE: + if (state == ARM64_SSBD_FORCE_DISABLE) + return -EPERM; + task_set_spec_ssb_disable(task); + task_set_spec_ssb_force_disable(task); + set_tsk_thread_flag(task, TIF_SSBD); + break; + default: + return -ERANGE; + } + + return 0; +} + +int arch_prctl_spec_ctrl_set(struct task_struct *task, unsigned long which, + unsigned long ctrl) +{ + switch (which) { + case PR_SPEC_STORE_BYPASS: + return ssbd_prctl_set(task, ctrl); + default: + return -ENODEV; + } +} + +static int ssbd_prctl_get(struct task_struct *task) +{ + switch (arm64_get_ssbd_state()) { + case ARM64_SSBD_UNKNOWN: + return -EINVAL; + case ARM64_SSBD_FORCE_ENABLE: + return PR_SPEC_DISABLE; + case ARM64_SSBD_KERNEL: + if (task_spec_ssb_force_disable(task)) + return PR_SPEC_PRCTL | PR_SPEC_FORCE_DISABLE; + if (task_spec_ssb_disable(task)) + return PR_SPEC_PRCTL | PR_SPEC_DISABLE; + return PR_SPEC_PRCTL | PR_SPEC_ENABLE; + case ARM64_SSBD_FORCE_DISABLE: + return PR_SPEC_ENABLE; + default: + return PR_SPEC_NOT_AFFECTED; + } +} + +int arch_prctl_spec_ctrl_get(struct task_struct *task, unsigned long which) +{ + switch (which) { + case PR_SPEC_STORE_BYPASS: + return ssbd_prctl_get(task); + default: + return -ENODEV; + } +} +#endif /* PR_SPEC_STORE_BYPASS */ diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c index a307b9e13392..70c283368b64 100644 --- a/arch/arm64/kernel/suspend.c +++ b/arch/arm64/kernel/suspend.c @@ -62,6 +62,14 @@ void notrace __cpu_suspend_exit(void) */ if (hw_breakpoint_restore) hw_breakpoint_restore(cpu); + + /* + * On resume, firmware implementing dynamic mitigation will + * have turned the mitigation on. If the user has forcefully + * disabled it, make sure their wishes are obeyed. + */ + if (arm64_get_ssbd_state() == ARM64_SSBD_FORCE_DISABLE) + arm64_set_ssbd_mitigation(false); } /* diff --git a/arch/arm64/kernel/sys_compat.c b/arch/arm64/kernel/sys_compat.c index 93ab57dcfc14..a6109825eeb9 100644 --- a/arch/arm64/kernel/sys_compat.c +++ b/arch/arm64/kernel/sys_compat.c @@ -112,6 +112,7 @@ long compat_arm_syscall(struct pt_regs *regs) break; } + clear_siginfo(&info); info.si_signo = SIGILL; info.si_errno = 0; info.si_code = ILL_ILLTRP; diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index 21868530018e..f845a8617812 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -11,7 +11,9 @@ * for more details. */ +#include #include +#include #include #include #include @@ -22,6 +24,7 @@ #include #include #include +#include #include #include @@ -47,7 +50,7 @@ static int __init get_cpu_for_node(struct device_node *node) return cpu; } -static int __init parse_core(struct device_node *core, int cluster_id, +static int __init parse_core(struct device_node *core, int package_id, int core_id) { char name[10]; @@ -63,7 +66,7 @@ static int __init parse_core(struct device_node *core, int cluster_id, leaf = false; cpu = get_cpu_for_node(t); if (cpu >= 0) { - cpu_topology[cpu].cluster_id = cluster_id; + cpu_topology[cpu].package_id = package_id; cpu_topology[cpu].core_id = core_id; cpu_topology[cpu].thread_id = i; } else { @@ -85,7 +88,7 @@ static int __init parse_core(struct device_node *core, int cluster_id, return -EINVAL; } - cpu_topology[cpu].cluster_id = cluster_id; + cpu_topology[cpu].package_id = package_id; cpu_topology[cpu].core_id = core_id; } else if (leaf) { pr_err("%pOF: Can't get CPU for leaf core\n", core); @@ -101,7 +104,7 @@ static int __init parse_cluster(struct device_node *cluster, int depth) bool leaf = true; bool has_cores = false; struct device_node *c; - static int cluster_id __initdata; + static int package_id __initdata; int core_id = 0; int i, ret; @@ -140,7 +143,7 @@ static int __init parse_cluster(struct device_node *cluster, int depth) } if (leaf) { - ret = parse_core(c, cluster_id, core_id++); + ret = parse_core(c, package_id, core_id++); } else { pr_err("%pOF: Non-leaf cluster with core %s\n", cluster, name); @@ -158,7 +161,7 @@ static int __init parse_cluster(struct device_node *cluster, int depth) pr_warn("%pOF: empty cluster\n", cluster); if (leaf) - cluster_id++; + package_id++; return 0; } @@ -194,7 +197,7 @@ static int __init parse_dt_topology(void) * only mark cores described in the DT as possible. */ for_each_possible_cpu(cpu) - if (cpu_topology[cpu].cluster_id == -1) + if (cpu_topology[cpu].package_id == -1) ret = -EINVAL; out_map: @@ -212,7 +215,14 @@ EXPORT_SYMBOL_GPL(cpu_topology); const struct cpumask *cpu_coregroup_mask(int cpu) { - return &cpu_topology[cpu].core_sibling; + const cpumask_t *core_mask = &cpu_topology[cpu].core_sibling; + + if (cpu_topology[cpu].llc_id != -1) { + if (cpumask_subset(&cpu_topology[cpu].llc_siblings, core_mask)) + core_mask = &cpu_topology[cpu].llc_siblings; + } + + return core_mask; } static void update_siblings_masks(unsigned int cpuid) @@ -224,7 +234,12 @@ static void update_siblings_masks(unsigned int cpuid) for_each_possible_cpu(cpu) { cpu_topo = &cpu_topology[cpu]; - if (cpuid_topo->cluster_id != cpu_topo->cluster_id) + if (cpuid_topo->llc_id == cpu_topo->llc_id) { + cpumask_set_cpu(cpu, &cpuid_topo->llc_siblings); + cpumask_set_cpu(cpuid, &cpu_topo->llc_siblings); + } + + if (cpuid_topo->package_id != cpu_topo->package_id) continue; cpumask_set_cpu(cpuid, &cpu_topo->core_sibling); @@ -245,7 +260,7 @@ void store_cpu_topology(unsigned int cpuid) struct cpu_topology *cpuid_topo = &cpu_topology[cpuid]; u64 mpidr; - if (cpuid_topo->cluster_id != -1) + if (cpuid_topo->package_id != -1) goto topology_populated; mpidr = read_cpuid_mpidr(); @@ -259,19 +274,19 @@ void store_cpu_topology(unsigned int cpuid) /* Multiprocessor system : Multi-threads per core */ cpuid_topo->thread_id = MPIDR_AFFINITY_LEVEL(mpidr, 0); cpuid_topo->core_id = MPIDR_AFFINITY_LEVEL(mpidr, 1); - cpuid_topo->cluster_id = MPIDR_AFFINITY_LEVEL(mpidr, 2) | + cpuid_topo->package_id = MPIDR_AFFINITY_LEVEL(mpidr, 2) | MPIDR_AFFINITY_LEVEL(mpidr, 3) << 8; } else { /* Multiprocessor system : Single-thread per core */ cpuid_topo->thread_id = -1; cpuid_topo->core_id = MPIDR_AFFINITY_LEVEL(mpidr, 0); - cpuid_topo->cluster_id = MPIDR_AFFINITY_LEVEL(mpidr, 1) | + cpuid_topo->package_id = MPIDR_AFFINITY_LEVEL(mpidr, 1) | MPIDR_AFFINITY_LEVEL(mpidr, 2) << 8 | MPIDR_AFFINITY_LEVEL(mpidr, 3) << 16; } pr_debug("CPU%u: cluster %d core %d thread %d mpidr %#016llx\n", - cpuid, cpuid_topo->cluster_id, cpuid_topo->core_id, + cpuid, cpuid_topo->package_id, cpuid_topo->core_id, cpuid_topo->thread_id, mpidr); topology_populated: @@ -287,7 +302,11 @@ static void __init reset_cpu_topology(void) cpu_topo->thread_id = -1; cpu_topo->core_id = 0; - cpu_topo->cluster_id = -1; + cpu_topo->package_id = -1; + + cpu_topo->llc_id = -1; + cpumask_clear(&cpu_topo->llc_siblings); + cpumask_set_cpu(cpu, &cpu_topo->llc_siblings); cpumask_clear(&cpu_topo->core_sibling); cpumask_set_cpu(cpu, &cpu_topo->core_sibling); @@ -296,6 +315,59 @@ static void __init reset_cpu_topology(void) } } +#ifdef CONFIG_ACPI +/* + * Propagate the topology information of the processor_topology_node tree to the + * cpu_topology array. + */ +static int __init parse_acpi_topology(void) +{ + bool is_threaded; + int cpu, topology_id; + + is_threaded = read_cpuid_mpidr() & MPIDR_MT_BITMASK; + + for_each_possible_cpu(cpu) { + int i, cache_id; + + topology_id = find_acpi_cpu_topology(cpu, 0); + if (topology_id < 0) + return topology_id; + + if (is_threaded) { + cpu_topology[cpu].thread_id = topology_id; + topology_id = find_acpi_cpu_topology(cpu, 1); + cpu_topology[cpu].core_id = topology_id; + } else { + cpu_topology[cpu].thread_id = -1; + cpu_topology[cpu].core_id = topology_id; + } + topology_id = find_acpi_cpu_topology_package(cpu); + cpu_topology[cpu].package_id = topology_id; + + i = acpi_find_last_cache_level(cpu); + + if (i > 0) { + /* + * this is the only part of cpu_topology that has + * a direct relationship with the cache topology + */ + cache_id = find_acpi_cpu_cache_topology(cpu, i); + if (cache_id > 0) + cpu_topology[cpu].llc_id = cache_id; + } + } + + return 0; +} + +#else +static inline int __init parse_acpi_topology(void) +{ + return -EINVAL; +} +#endif + void __init init_cpu_topology(void) { reset_cpu_topology(); @@ -304,6 +376,8 @@ void __init init_cpu_topology(void) * Discard anything that was parsed if we hit an error so we * don't use partial information. */ - if (of_have_populated_dt() && parse_dt_topology()) + if (!acpi_disabled && parse_acpi_topology()) + reset_cpu_topology(); + else if (of_have_populated_dt() && parse_dt_topology()) reset_cpu_topology(); } diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 8bbdc17e49df..d399d459397b 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -635,6 +635,7 @@ asmlinkage void bad_el0_sync(struct pt_regs *regs, int reason, unsigned int esr) siginfo_t info; void __user *pc = (void __user *)instruction_pointer(regs); + clear_siginfo(&info); info.si_signo = SIGILL; info.si_errno = 0; info.si_code = ILL_ILLOPC; diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 0221aca6493d..605d1b60469c 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -34,25 +34,25 @@ jiffies = jiffies_64; * 4 KB (see related ASSERT() below) \ */ \ . = ALIGN(SZ_4K); \ - VMLINUX_SYMBOL(__hyp_idmap_text_start) = .; \ + __hyp_idmap_text_start = .; \ *(.hyp.idmap.text) \ - VMLINUX_SYMBOL(__hyp_idmap_text_end) = .; \ - VMLINUX_SYMBOL(__hyp_text_start) = .; \ + __hyp_idmap_text_end = .; \ + __hyp_text_start = .; \ *(.hyp.text) \ - VMLINUX_SYMBOL(__hyp_text_end) = .; + __hyp_text_end = .; #define IDMAP_TEXT \ . = ALIGN(SZ_4K); \ - VMLINUX_SYMBOL(__idmap_text_start) = .; \ + __idmap_text_start = .; \ *(.idmap.text) \ - VMLINUX_SYMBOL(__idmap_text_end) = .; + __idmap_text_end = .; #ifdef CONFIG_HIBERNATION #define HIBERNATE_TEXT \ . = ALIGN(SZ_4K); \ - VMLINUX_SYMBOL(__hibernate_exit_text_start) = .;\ + __hibernate_exit_text_start = .; \ *(.hibernate_exit.text) \ - VMLINUX_SYMBOL(__hibernate_exit_text_end) = .; + __hibernate_exit_text_end = .; #else #define HIBERNATE_TEXT #endif @@ -60,10 +60,10 @@ jiffies = jiffies_64; #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 #define TRAMP_TEXT \ . = ALIGN(PAGE_SIZE); \ - VMLINUX_SYMBOL(__entry_tramp_text_start) = .; \ + __entry_tramp_text_start = .; \ *(.entry.tramp.text) \ . = ALIGN(PAGE_SIZE); \ - VMLINUX_SYMBOL(__entry_tramp_text_end) = .; + __entry_tramp_text_end = .; #else #define TRAMP_TEXT #endif diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index a2e3a5af1113..47b23bf617c7 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -39,6 +39,7 @@ config KVM select HAVE_KVM_IRQ_ROUTING select IRQ_BYPASS_MANAGER select HAVE_KVM_IRQ_BYPASS + select HAVE_KVM_VCPU_RUN_PID_CHANGE ---help--- Support hosting virtualized guest machines. We don't support KVM with 16K page tables yet, due to the multiple diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 93afff91cb7c..0f2a135ba15b 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -19,7 +19,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/psci.o $(KVM)/arm/perf.o kvm-$(CONFIG_KVM_ARM_HOST) += inject_fault.o regmap.o va_layout.o kvm-$(CONFIG_KVM_ARM_HOST) += hyp.o hyp-init.o handle_exit.o kvm-$(CONFIG_KVM_ARM_HOST) += guest.o debug.o reset.o sys_regs.o sys_regs_generic_v8.o -kvm-$(CONFIG_KVM_ARM_HOST) += vgic-sys-reg-v3.o +kvm-$(CONFIG_KVM_ARM_HOST) += vgic-sys-reg-v3.o fpsimd.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/aarch32.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic.o diff --git a/arch/arm64/kvm/debug.c b/arch/arm64/kvm/debug.c index a1f4ebdfe6d3..00d422336a45 100644 --- a/arch/arm64/kvm/debug.c +++ b/arch/arm64/kvm/debug.c @@ -103,7 +103,7 @@ void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu) * * Additionally, KVM only traps guest accesses to the debug registers if * the guest is not actively using them (see the KVM_ARM64_DEBUG_DIRTY - * flag on vcpu->arch.debug_flags). Since the guest must not interfere + * flag on vcpu->arch.flags). Since the guest must not interfere * with the hardware state when debugging the guest, we must ensure that * trapping is enabled whenever we are debugging the guest using the * debug registers. @@ -111,7 +111,7 @@ void kvm_arm_reset_debug_ptr(struct kvm_vcpu *vcpu) void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) { - bool trap_debug = !(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY); + bool trap_debug = !(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY); unsigned long mdscr; trace_kvm_arm_setup_debug(vcpu, vcpu->guest_debug); @@ -184,7 +184,7 @@ void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) vcpu_write_sys_reg(vcpu, mdscr, MDSCR_EL1); vcpu->arch.debug_ptr = &vcpu->arch.external_debug_state; - vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY; + vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; trap_debug = true; trace_kvm_arm_set_regset("BKPTS", get_num_brps(), @@ -206,7 +206,7 @@ void kvm_arm_setup_debug(struct kvm_vcpu *vcpu) /* If KDE or MDE are set, perform a full save/restore cycle. */ if (vcpu_read_sys_reg(vcpu, MDSCR_EL1) & (DBG_MDSCR_KDE | DBG_MDSCR_MDE)) - vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY; + vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; trace_kvm_arm_set_dreg32("MDCR_EL2", vcpu->arch.mdcr_el2); trace_kvm_arm_set_dreg32("MDSCR_EL1", vcpu_read_sys_reg(vcpu, MDSCR_EL1)); diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c new file mode 100644 index 000000000000..dc6ecfa5a2d2 --- /dev/null +++ b/arch/arm64/kvm/fpsimd.c @@ -0,0 +1,110 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * arch/arm64/kvm/fpsimd.c: Guest/host FPSIMD context coordination helpers + * + * Copyright 2018 Arm Limited + * Author: Dave Martin + */ +#include +#include +#include +#include +#include +#include +#include + +/* + * Called on entry to KVM_RUN unless this vcpu previously ran at least + * once and the most recent prior KVM_RUN for this vcpu was called from + * the same task as current (highly likely). + * + * This is guaranteed to execute before kvm_arch_vcpu_load_fp(vcpu), + * such that on entering hyp the relevant parts of current are already + * mapped. + */ +int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) +{ + int ret; + + struct thread_info *ti = ¤t->thread_info; + struct user_fpsimd_state *fpsimd = ¤t->thread.uw.fpsimd_state; + + /* + * Make sure the host task thread flags and fpsimd state are + * visible to hyp: + */ + ret = create_hyp_mappings(ti, ti + 1, PAGE_HYP); + if (ret) + goto error; + + ret = create_hyp_mappings(fpsimd, fpsimd + 1, PAGE_HYP); + if (ret) + goto error; + + vcpu->arch.host_thread_info = kern_hyp_va(ti); + vcpu->arch.host_fpsimd_state = kern_hyp_va(fpsimd); +error: + return ret; +} + +/* + * Prepare vcpu for saving the host's FPSIMD state and loading the guest's. + * The actual loading is done by the FPSIMD access trap taken to hyp. + * + * Here, we just set the correct metadata to indicate that the FPSIMD + * state in the cpu regs (if any) belongs to current on the host. + * + * TIF_SVE is backed up here, since it may get clobbered with guest state. + * This flag is restored by kvm_arch_vcpu_put_fp(vcpu). + */ +void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) +{ + BUG_ON(!current->mm); + + vcpu->arch.flags &= ~(KVM_ARM64_FP_ENABLED | KVM_ARM64_HOST_SVE_IN_USE); + vcpu->arch.flags |= KVM_ARM64_FP_HOST; + if (test_thread_flag(TIF_SVE)) + vcpu->arch.flags |= KVM_ARM64_HOST_SVE_IN_USE; +} + +/* + * If the guest FPSIMD state was loaded, update the host's context + * tracking data mark the CPU FPSIMD regs as dirty and belonging to vcpu + * so that they will be written back if the kernel clobbers them due to + * kernel-mode NEON before re-entry into the guest. + */ +void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) +{ + WARN_ON_ONCE(!irqs_disabled()); + + if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { + fpsimd_bind_state_to_cpu(&vcpu->arch.ctxt.gp_regs.fp_regs); + clear_thread_flag(TIF_FOREIGN_FPSTATE); + clear_thread_flag(TIF_SVE); + } +} + +/* + * Write back the vcpu FPSIMD regs if they are dirty, and invalidate the + * cpu FPSIMD regs so that they can't be spuriously reused if this vcpu + * disappears and another task or vcpu appears that recycles the same + * struct fpsimd_state. + */ +void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) +{ + local_bh_disable(); + + update_thread_flag(TIF_SVE, + vcpu->arch.flags & KVM_ARM64_HOST_SVE_IN_USE); + + if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { + /* Clean guest FP state to memory and invalidate cpu view */ + fpsimd_save(); + fpsimd_flush_cpu_state(); + } else if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) { + /* Ensure user trap controls are correctly restored */ + fpsimd_bind_task_to_cpu(); + } + + local_bh_enable(); +} diff --git a/arch/arm64/kvm/hyp/debug-sr.c b/arch/arm64/kvm/hyp/debug-sr.c index 3e717f66f011..50009766e5e5 100644 --- a/arch/arm64/kvm/hyp/debug-sr.c +++ b/arch/arm64/kvm/hyp/debug-sr.c @@ -163,7 +163,7 @@ void __hyp_text __debug_switch_to_guest(struct kvm_vcpu *vcpu) if (!has_vhe()) __debug_save_spe_nvhe(&vcpu->arch.host_debug_state.pmscr_el1); - if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)) + if (!(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY)) return; host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context); @@ -185,7 +185,7 @@ void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu) if (!has_vhe()) __debug_restore_spe_nvhe(vcpu->arch.host_debug_state.pmscr_el1); - if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)) + if (!(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY)) return; host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context); @@ -196,7 +196,7 @@ void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu) __debug_save_state(vcpu, guest_dbg, guest_ctxt); __debug_restore_state(vcpu, host_dbg, host_ctxt); - vcpu->arch.debug_flags &= ~KVM_ARM64_DEBUG_DIRTY; + vcpu->arch.flags &= ~KVM_ARM64_DEBUG_DIRTY; } u32 __hyp_text __kvm_get_mdcr_el2(void) diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index e41a161d313a..fad1e164fe48 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -166,46 +166,3 @@ abort_guest_exit_end: orr x0, x0, x5 1: ret ENDPROC(__guest_exit) - -ENTRY(__fpsimd_guest_restore) - // x0: esr - // x1: vcpu - // x2-x29,lr: vcpu regs - // vcpu x0-x1 on the stack - stp x2, x3, [sp, #-16]! - stp x4, lr, [sp, #-16]! - -alternative_if_not ARM64_HAS_VIRT_HOST_EXTN - mrs x2, cptr_el2 - bic x2, x2, #CPTR_EL2_TFP - msr cptr_el2, x2 -alternative_else - mrs x2, cpacr_el1 - orr x2, x2, #CPACR_EL1_FPEN - msr cpacr_el1, x2 -alternative_endif - isb - - mov x3, x1 - - ldr x0, [x3, #VCPU_HOST_CONTEXT] - kern_hyp_va x0 - add x0, x0, #CPU_GP_REG_OFFSET(CPU_FP_REGS) - bl __fpsimd_save_state - - add x2, x3, #VCPU_CONTEXT - add x0, x2, #CPU_GP_REG_OFFSET(CPU_FP_REGS) - bl __fpsimd_restore_state - - // Skip restoring fpexc32 for AArch64 guests - mrs x1, hcr_el2 - tbnz x1, #HCR_RW_SHIFT, 1f - ldr x4, [x3, #VCPU_FPEXC32_EL2] - msr fpexc32_el2, x4 -1: - ldp x4, lr, [sp], #16 - ldp x2, x3, [sp], #16 - ldp x0, x1, [sp], #16 - - eret -ENDPROC(__fpsimd_guest_restore) diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S index bffece27b5c1..24b4fbafe3e4 100644 --- a/arch/arm64/kvm/hyp/hyp-entry.S +++ b/arch/arm64/kvm/hyp/hyp-entry.S @@ -106,32 +106,49 @@ el1_hvc_guest: */ ldr x1, [sp] // Guest's x0 eor w1, w1, #ARM_SMCCC_ARCH_WORKAROUND_1 + cbz w1, wa_epilogue + + /* ARM_SMCCC_ARCH_WORKAROUND_2 handling */ + eor w1, w1, #(ARM_SMCCC_ARCH_WORKAROUND_1 ^ \ + ARM_SMCCC_ARCH_WORKAROUND_2) cbnz w1, el1_trap - mov x0, x1 + +#ifdef CONFIG_ARM64_SSBD +alternative_cb arm64_enable_wa2_handling + b wa2_end +alternative_cb_end + get_vcpu_ptr x2, x0 + ldr x0, [x2, #VCPU_WORKAROUND_FLAGS] + + // Sanitize the argument and update the guest flags + ldr x1, [sp, #8] // Guest's x1 + clz w1, w1 // Murphy's device: + lsr w1, w1, #5 // w1 = !!w1 without using + eor w1, w1, #1 // the flags... + bfi x0, x1, #VCPU_WORKAROUND_2_FLAG_SHIFT, #1 + str x0, [x2, #VCPU_WORKAROUND_FLAGS] + + /* Check that we actually need to perform the call */ + hyp_ldr_this_cpu x0, arm64_ssbd_callback_required, x2 + cbz x0, wa2_end + + mov w0, #ARM_SMCCC_ARCH_WORKAROUND_2 + smc #0 + + /* Don't leak data from the SMC call */ + mov x3, xzr +wa2_end: + mov x2, xzr + mov x1, xzr +#endif + +wa_epilogue: + mov x0, xzr add sp, sp, #16 eret el1_trap: get_vcpu_ptr x1, x0 - - mrs x0, esr_el2 - lsr x0, x0, #ESR_ELx_EC_SHIFT - /* - * x0: ESR_EC - * x1: vcpu pointer - */ - - /* - * We trap the first access to the FP/SIMD to save the host context - * and restore the guest context lazily. - * If FP/SIMD is not implemented, handle the trap and inject an - * undefined instruction exception to the guest. - */ -alternative_if_not ARM64_HAS_NO_FPSIMD - cmp x0, #ESR_ELx_EC_FP_ASIMD - b.eq __fpsimd_guest_restore -alternative_else_nop_endif - mov x0, #ARM_EXCEPTION_TRAP b __guest_exit diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index d9645236e474..d496ef579859 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -15,27 +15,32 @@ * along with this program. If not, see . */ +#include #include #include #include #include +#include #include #include +#include #include #include #include #include +#include +#include -static bool __hyp_text __fpsimd_enabled_nvhe(void) +/* Check whether the FP regs were dirtied while in the host-side run loop: */ +static bool __hyp_text update_fp_enabled(struct kvm_vcpu *vcpu) { - return !(read_sysreg(cptr_el2) & CPTR_EL2_TFP); -} + if (vcpu->arch.host_thread_info->flags & _TIF_FOREIGN_FPSTATE) + vcpu->arch.flags &= ~(KVM_ARM64_FP_ENABLED | + KVM_ARM64_FP_HOST); -static bool fpsimd_enabled_vhe(void) -{ - return !!(read_sysreg(cpacr_el1) & CPACR_EL1_FPEN); + return !!(vcpu->arch.flags & KVM_ARM64_FP_ENABLED); } /* Save the 32-bit only FPSIMD system register state */ @@ -92,7 +97,10 @@ static void activate_traps_vhe(struct kvm_vcpu *vcpu) val = read_sysreg(cpacr_el1); val |= CPACR_EL1_TTA; - val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN); + val &= ~CPACR_EL1_ZEN; + if (!update_fp_enabled(vcpu)) + val &= ~CPACR_EL1_FPEN; + write_sysreg(val, cpacr_el1); write_sysreg(kvm_get_hyp_vector(), vbar_el1); @@ -105,7 +113,10 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu) __activate_traps_common(vcpu); val = CPTR_EL2_DEFAULT; - val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ; + val |= CPTR_EL2_TTA | CPTR_EL2_TZ; + if (!update_fp_enabled(vcpu)) + val |= CPTR_EL2_TFP; + write_sysreg(val, cptr_el2); } @@ -318,6 +329,50 @@ static bool __hyp_text __skip_instr(struct kvm_vcpu *vcpu) } } +static bool __hyp_text __hyp_switch_fpsimd(struct kvm_vcpu *vcpu) +{ + struct user_fpsimd_state *host_fpsimd = vcpu->arch.host_fpsimd_state; + + if (has_vhe()) + write_sysreg(read_sysreg(cpacr_el1) | CPACR_EL1_FPEN, + cpacr_el1); + else + write_sysreg(read_sysreg(cptr_el2) & ~(u64)CPTR_EL2_TFP, + cptr_el2); + + isb(); + + if (vcpu->arch.flags & KVM_ARM64_FP_HOST) { + /* + * In the SVE case, VHE is assumed: it is enforced by + * Kconfig and kvm_arch_init(). + */ + if (system_supports_sve() && + (vcpu->arch.flags & KVM_ARM64_HOST_SVE_IN_USE)) { + struct thread_struct *thread = container_of( + host_fpsimd, + struct thread_struct, uw.fpsimd_state); + + sve_save_state(sve_pffr(thread), &host_fpsimd->fpsr); + } else { + __fpsimd_save_state(host_fpsimd); + } + + vcpu->arch.flags &= ~KVM_ARM64_FP_HOST; + } + + __fpsimd_restore_state(&vcpu->arch.ctxt.gp_regs.fp_regs); + + /* Skip restoring fpexc32 for AArch64 guests */ + if (!(read_sysreg(hcr_el2) & HCR_RW)) + write_sysreg(vcpu->arch.ctxt.sys_regs[FPEXC32_EL2], + fpexc32_el2); + + vcpu->arch.flags |= KVM_ARM64_FP_ENABLED; + + return true; +} + /* * Return true when we were able to fixup the guest exit and should return to * the guest, false when we should restore the host state and return to the @@ -334,11 +389,23 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) * same PC once the SError has been injected, and replay the * trapping instruction. */ - if (*exit_code == ARM_EXCEPTION_TRAP && !__populate_fault_info(vcpu)) + if (*exit_code != ARM_EXCEPTION_TRAP) + goto exit; + + /* + * We trap the first access to the FP/SIMD to save the host context + * and restore the guest context lazily. + * If FP/SIMD is not implemented, handle the trap and inject an + * undefined instruction exception to the guest. + */ + if (system_supports_fpsimd() && + kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_FP_ASIMD) + return __hyp_switch_fpsimd(vcpu); + + if (!__populate_fault_info(vcpu)) return true; - if (static_branch_unlikely(&vgic_v2_cpuif_trap) && - *exit_code == ARM_EXCEPTION_TRAP) { + if (static_branch_unlikely(&vgic_v2_cpuif_trap)) { bool valid; valid = kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_DABT_LOW && @@ -350,12 +417,8 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) if (valid) { int ret = __vgic_v2_perform_cpuif_access(vcpu); - if (ret == 1) { - if (__skip_instr(vcpu)) - return true; - else - *exit_code = ARM_EXCEPTION_TRAP; - } + if (ret == 1 && __skip_instr(vcpu)) + return true; if (ret == -1) { /* Promote an illegal access to an @@ -368,33 +431,63 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) *vcpu_cpsr(vcpu) &= ~DBG_SPSR_SS; *exit_code = ARM_EXCEPTION_EL1_SERROR; } + + goto exit; } } if (static_branch_unlikely(&vgic_v3_cpuif_trap) && - *exit_code == ARM_EXCEPTION_TRAP && (kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_SYS64 || kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_CP15_32)) { int ret = __vgic_v3_perform_cpuif_access(vcpu); - if (ret == 1) { - if (__skip_instr(vcpu)) - return true; - else - *exit_code = ARM_EXCEPTION_TRAP; - } + if (ret == 1 && __skip_instr(vcpu)) + return true; } +exit: /* Return to the host kernel and handle the exit */ return false; } +static inline bool __hyp_text __needs_ssbd_off(struct kvm_vcpu *vcpu) +{ + if (!cpus_have_const_cap(ARM64_SSBD)) + return false; + + return !(vcpu->arch.workaround_flags & VCPU_WORKAROUND_2_FLAG); +} + +static void __hyp_text __set_guest_arch_workaround_state(struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_ARM64_SSBD + /* + * The host runs with the workaround always present. If the + * guest wants it disabled, so be it... + */ + if (__needs_ssbd_off(vcpu) && + __hyp_this_cpu_read(arm64_ssbd_callback_required)) + arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 0, NULL); +#endif +} + +static void __hyp_text __set_host_arch_workaround_state(struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_ARM64_SSBD + /* + * If the guest has disabled the workaround, bring it back on. + */ + if (__needs_ssbd_off(vcpu) && + __hyp_this_cpu_read(arm64_ssbd_callback_required)) + arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 1, NULL); +#endif +} + /* Switch to the guest for VHE systems running in EL2 */ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) { struct kvm_cpu_context *host_ctxt; struct kvm_cpu_context *guest_ctxt; - bool fp_enabled; u64 exit_code; host_ctxt = vcpu->arch.host_cpu_context; @@ -409,6 +502,8 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) sysreg_restore_guest_state_vhe(guest_ctxt); __debug_switch_to_guest(vcpu); + __set_guest_arch_workaround_state(vcpu); + do { /* Jump in the fire! */ exit_code = __guest_enter(vcpu, host_ctxt); @@ -416,7 +511,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) /* And we're baaack! */ } while (fixup_guest_exit(vcpu, &exit_code)); - fp_enabled = fpsimd_enabled_vhe(); + __set_host_arch_workaround_state(vcpu); sysreg_save_guest_state_vhe(guest_ctxt); @@ -424,11 +519,8 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) sysreg_restore_host_state_vhe(host_ctxt); - if (fp_enabled) { - __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs); - __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs); + if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) __fpsimd_save_fpexc32(vcpu); - } __debug_switch_to_host(vcpu); @@ -440,7 +532,6 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu) { struct kvm_cpu_context *host_ctxt; struct kvm_cpu_context *guest_ctxt; - bool fp_enabled; u64 exit_code; vcpu = kern_hyp_va(vcpu); @@ -465,6 +556,8 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu) __sysreg_restore_state_nvhe(guest_ctxt); __debug_switch_to_guest(vcpu); + __set_guest_arch_workaround_state(vcpu); + do { /* Jump in the fire! */ exit_code = __guest_enter(vcpu, host_ctxt); @@ -472,7 +565,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu) /* And we're baaack! */ } while (fixup_guest_exit(vcpu, &exit_code)); - fp_enabled = __fpsimd_enabled_nvhe(); + __set_host_arch_workaround_state(vcpu); __sysreg_save_state_nvhe(guest_ctxt); __sysreg32_save_state(vcpu); @@ -484,11 +577,8 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu) __sysreg_restore_state_nvhe(host_ctxt); - if (fp_enabled) { - __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs); - __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs); + if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) __fpsimd_save_fpexc32(vcpu); - } /* * This must come after restoring the host sysregs, since a non-VHE diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index b3894df6bf1a..35bc16832efe 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -196,7 +196,7 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu) sysreg[DACR32_EL2] = read_sysreg(dacr32_el2); sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2); - if (has_vhe() || vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY) + if (has_vhe() || vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY) sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2); } @@ -218,7 +218,7 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu) write_sysreg(sysreg[DACR32_EL2], dacr32_el2); write_sysreg(sysreg[IFSR32_EL2], ifsr32_el2); - if (has_vhe() || vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY) + if (has_vhe() || vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY) write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2); } diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 3256b9228e75..a74311beda35 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -122,6 +122,10 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) /* Reset PMU */ kvm_pmu_vcpu_reset(vcpu); + /* Default workaround setup is enabled (if supported) */ + if (kvm_arm_have_ssbd() == KVM_SSBD_KERNEL) + vcpu->arch.workaround_flags |= VCPU_WORKAROUND_2_FLAG; + /* Reset timer */ return kvm_timer_vcpu_reset(vcpu); } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 6e3b969391fd..a4363735d3f8 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -31,7 +31,6 @@ #include #include #include -#include #include #include #include @@ -338,7 +337,7 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu, { if (p->is_write) { vcpu_write_sys_reg(vcpu, p->regval, r->reg); - vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY; + vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; } else { p->regval = vcpu_read_sys_reg(vcpu, r->reg); } @@ -369,7 +368,7 @@ static void reg_to_dbg(struct kvm_vcpu *vcpu, } *dbg_reg = val; - vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY; + vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; } static void dbg_to_reg(struct kvm_vcpu *vcpu, @@ -1441,7 +1440,7 @@ static bool trap_debug32(struct kvm_vcpu *vcpu, { if (p->is_write) { vcpu_cp14(vcpu, r->reg) = p->regval; - vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY; + vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; } else { p->regval = vcpu_cp14(vcpu, r->reg); } @@ -1473,7 +1472,7 @@ static bool trap_xvr(struct kvm_vcpu *vcpu, val |= p->regval << 32; *dbg_reg = val; - vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY; + vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; } else { p->regval = *dbg_reg >> 32; } diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index 301417ae2ba8..c127f94da8e2 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -263,7 +263,7 @@ static int asids_init(void) */ WARN_ON(NUM_USER_ASIDS - 1 <= num_possible_cpus()); atomic64_set(&asid_generation, ASID_FIRST_VERSION); - asid_map = kzalloc(BITS_TO_LONGS(NUM_USER_ASIDS) * sizeof(*asid_map), + asid_map = kcalloc(BITS_TO_LONGS(NUM_USER_ASIDS), sizeof(*asid_map), GFP_KERNEL); if (!asid_map) panic("Failed to allocate bitmap for %lu ASIDs\n", diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c index a96ec0181818..49e217ac7e1e 100644 --- a/arch/arm64/mm/dma-mapping.c +++ b/arch/arm64/mm/dma-mapping.c @@ -504,20 +504,15 @@ static int __init arm64_dma_init(void) max_pfn > (arm64_dma_phys_limit >> PAGE_SHIFT)) swiotlb = 1; + WARN_TAINT(ARCH_DMA_MINALIGN < cache_line_size(), + TAINT_CPU_OUT_OF_SPEC, + "ARCH_DMA_MINALIGN smaller than CTR_EL0.CWG (%d < %d)", + ARCH_DMA_MINALIGN, cache_line_size()); + return atomic_pool_init(); } arch_initcall(arm64_dma_init); -#define PREALLOC_DMA_DEBUG_ENTRIES 4096 - -static int __init dma_debug_do_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - return 0; -} -fs_initcall(dma_debug_do_init); - - #ifdef CONFIG_IOMMU_DMA #include #include diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 2af3dd89bcdb..b8eecc7b9531 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -235,8 +235,9 @@ static bool is_el1_instruction_abort(unsigned int esr) return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_CUR; } -static inline bool is_permission_fault(unsigned int esr, struct pt_regs *regs, - unsigned long addr) +static inline bool is_el1_permission_fault(unsigned int esr, + struct pt_regs *regs, + unsigned long addr) { unsigned int ec = ESR_ELx_EC(esr); unsigned int fsc_type = esr & ESR_ELx_FSC_TYPE; @@ -254,6 +255,22 @@ static inline bool is_permission_fault(unsigned int esr, struct pt_regs *regs, return false; } +static void die_kernel_fault(const char *msg, unsigned long addr, + unsigned int esr, struct pt_regs *regs) +{ + bust_spinlocks(1); + + pr_alert("Unable to handle kernel %s at virtual address %016lx\n", msg, + addr); + + mem_abort_decode(esr); + + show_pte(addr); + die("Oops", regs, esr); + bust_spinlocks(0); + do_exit(SIGKILL); +} + static void __do_kernel_fault(unsigned long addr, unsigned int esr, struct pt_regs *regs) { @@ -266,9 +283,7 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr, if (!is_el1_instruction_abort(esr) && fixup_exception(regs)) return; - bust_spinlocks(1); - - if (is_permission_fault(esr, regs, addr)) { + if (is_el1_permission_fault(esr, regs, addr)) { if (esr & ESR_ELx_WNR) msg = "write to read-only memory"; else @@ -279,15 +294,7 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr, msg = "paging request"; } - pr_alert("Unable to handle kernel %s at virtual address %08lx\n", msg, - addr); - - mem_abort_decode(esr); - - show_pte(addr); - die("Oops", regs, esr); - bust_spinlocks(0); - do_exit(SIGKILL); + die_kernel_fault(msg, addr, esr, regs); } static void __do_user_fault(struct siginfo *info, unsigned int esr) @@ -356,11 +363,12 @@ static void do_bad_area(unsigned long addr, unsigned int esr, struct pt_regs *re */ if (user_mode(regs)) { const struct fault_info *inf = esr_to_fault_info(esr); - struct siginfo si = { - .si_signo = inf->sig, - .si_code = inf->code, - .si_addr = (void __user *)addr, - }; + struct siginfo si; + + clear_siginfo(&si); + si.si_signo = inf->sig; + si.si_code = inf->code; + si.si_addr = (void __user *)addr; __do_user_fault(&si, esr); } else { @@ -446,16 +454,19 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, mm_flags |= FAULT_FLAG_WRITE; } - if (addr < TASK_SIZE && is_permission_fault(esr, regs, addr)) { + if (addr < TASK_SIZE && is_el1_permission_fault(esr, regs, addr)) { /* regs->orig_addr_limit may be 0 if we entered from EL0 */ if (regs->orig_addr_limit == KERNEL_DS) - die("Accessing user space memory with fs=KERNEL_DS", regs, esr); + die_kernel_fault("access to user memory with fs=KERNEL_DS", + addr, esr, regs); if (is_el1_instruction_abort(esr)) - die("Attempting to execute userspace memory", regs, esr); + die_kernel_fault("execution of user memory", + addr, esr, regs); if (!search_exception_tables(regs->pc)) - die("Accessing user space memory outside uaccess.h routines", regs, esr); + die_kernel_fault("access to user memory outside uaccess routines", + addr, esr, regs); } perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); @@ -634,6 +645,7 @@ static int do_sea(unsigned long addr, unsigned int esr, struct pt_regs *regs) nmi_exit(); } + clear_siginfo(&info); info.si_signo = inf->sig; info.si_errno = 0; info.si_code = inf->code; @@ -738,6 +750,7 @@ asmlinkage void __exception do_mem_abort(unsigned long addr, unsigned int esr, show_pte(addr); } + clear_siginfo(&info); info.si_signo = inf->sig; info.si_errno = 0; info.si_code = inf->code; @@ -780,6 +793,7 @@ asmlinkage void __exception do_sp_pc_abort(unsigned long addr, local_irq_enable(); } + clear_siginfo(&info); info.si_signo = SIGBUS; info.si_errno = 0; info.si_code = BUS_ADRALN; @@ -823,7 +837,6 @@ asmlinkage int __exception do_debug_exception(unsigned long addr, struct pt_regs *regs) { const struct fault_info *inf = debug_fault_info + DBG_ESR_EVT(esr); - struct siginfo info; int rv; /* @@ -839,6 +852,9 @@ asmlinkage int __exception do_debug_exception(unsigned long addr, if (!inf->fn(addr, esr, regs)) { rv = 1; } else { + struct siginfo info; + + clear_siginfo(&info); info.si_signo = inf->sig; info.si_errno = 0; info.si_code = inf->code; diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 1b18b4722420..325cfb3b858a 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -310,7 +310,7 @@ static void __init arm64_memory_present(void) } #endif -static phys_addr_t memory_limit = (phys_addr_t)ULLONG_MAX; +static phys_addr_t memory_limit = PHYS_ADDR_MAX; /* * Limit the memory size that was specified via FDT. @@ -401,7 +401,7 @@ void __init arm64_memblock_init(void) * high up in memory, add back the kernel region that must be accessible * via the linear mapping. */ - if (memory_limit != (phys_addr_t)ULLONG_MAX) { + if (memory_limit != PHYS_ADDR_MAX) { memblock_mem_limit_remove_map(memory_limit); memblock_add(__pa_symbol(_text), (u64)(_end - _text)); } @@ -666,7 +666,7 @@ __setup("keepinitrd", keepinitrd_setup); */ static int dump_mem_limit(struct notifier_block *self, unsigned long v, void *p) { - if (memory_limit != (phys_addr_t)ULLONG_MAX) { + if (memory_limit != PHYS_ADDR_MAX) { pr_emerg("Memory Limit: %llu MB\n", memory_limit >> 20); } else { pr_emerg("Memory Limit: none\n"); diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index a93350451e8e..a6fdaea07c63 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -21,7 +21,6 @@ #include #include #include -#include #include #include @@ -80,23 +79,66 @@ static inline void emit(const u32 insn, struct jit_ctx *ctx) ctx->idx++; } +static inline void emit_a64_mov_i(const int is64, const int reg, + const s32 val, struct jit_ctx *ctx) +{ + u16 hi = val >> 16; + u16 lo = val & 0xffff; + + if (hi & 0x8000) { + if (hi == 0xffff) { + emit(A64_MOVN(is64, reg, (u16)~lo, 0), ctx); + } else { + emit(A64_MOVN(is64, reg, (u16)~hi, 16), ctx); + if (lo != 0xffff) + emit(A64_MOVK(is64, reg, lo, 0), ctx); + } + } else { + emit(A64_MOVZ(is64, reg, lo, 0), ctx); + if (hi) + emit(A64_MOVK(is64, reg, hi, 16), ctx); + } +} + +static int i64_i16_blocks(const u64 val, bool inverse) +{ + return (((val >> 0) & 0xffff) != (inverse ? 0xffff : 0x0000)) + + (((val >> 16) & 0xffff) != (inverse ? 0xffff : 0x0000)) + + (((val >> 32) & 0xffff) != (inverse ? 0xffff : 0x0000)) + + (((val >> 48) & 0xffff) != (inverse ? 0xffff : 0x0000)); +} + static inline void emit_a64_mov_i64(const int reg, const u64 val, struct jit_ctx *ctx) { - u64 tmp = val; - int shift = 0; - - emit(A64_MOVZ(1, reg, tmp & 0xffff, shift), ctx); - tmp >>= 16; - shift += 16; - while (tmp) { - if (tmp & 0xffff) - emit(A64_MOVK(1, reg, tmp & 0xffff, shift), ctx); - tmp >>= 16; - shift += 16; + u64 nrm_tmp = val, rev_tmp = ~val; + bool inverse; + int shift; + + if (!(nrm_tmp >> 32)) + return emit_a64_mov_i(0, reg, (u32)val, ctx); + + inverse = i64_i16_blocks(nrm_tmp, true) < i64_i16_blocks(nrm_tmp, false); + shift = max(round_down((inverse ? (fls64(rev_tmp) - 1) : + (fls64(nrm_tmp) - 1)), 16), 0); + if (inverse) + emit(A64_MOVN(1, reg, (rev_tmp >> shift) & 0xffff, shift), ctx); + else + emit(A64_MOVZ(1, reg, (nrm_tmp >> shift) & 0xffff, shift), ctx); + shift -= 16; + while (shift >= 0) { + if (((nrm_tmp >> shift) & 0xffff) != (inverse ? 0xffff : 0x0000)) + emit(A64_MOVK(1, reg, (nrm_tmp >> shift) & 0xffff, shift), ctx); + shift -= 16; } } +/* + * This is an unoptimized 64 immediate emission used for BPF to BPF call + * addresses. It will always do a full 64 bit decomposition as otherwise + * more complexity in the last extra pass is required since we previously + * reserved 4 instructions for the address. + */ static inline void emit_addr_mov_i64(const int reg, const u64 val, struct jit_ctx *ctx) { @@ -111,26 +153,6 @@ static inline void emit_addr_mov_i64(const int reg, const u64 val, } } -static inline void emit_a64_mov_i(const int is64, const int reg, - const s32 val, struct jit_ctx *ctx) -{ - u16 hi = val >> 16; - u16 lo = val & 0xffff; - - if (hi & 0x8000) { - if (hi == 0xffff) { - emit(A64_MOVN(is64, reg, (u16)~lo, 0), ctx); - } else { - emit(A64_MOVN(is64, reg, (u16)~hi, 16), ctx); - emit(A64_MOVK(is64, reg, lo, 0), ctx); - } - } else { - emit(A64_MOVZ(is64, reg, lo, 0), ctx); - if (hi) - emit(A64_MOVK(is64, reg, hi, 16), ctx); - } -} - static inline int bpf2a64_offset(int bpf_to, int bpf_from, const struct jit_ctx *ctx) { @@ -163,7 +185,7 @@ static inline int epilogue_offset(const struct jit_ctx *ctx) /* Tail call offset to jump into */ #define PROLOGUE_OFFSET 7 -static int build_prologue(struct jit_ctx *ctx) +static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) { const struct bpf_prog *prog = ctx->prog; const u8 r6 = bpf2a64[BPF_REG_6]; @@ -188,7 +210,7 @@ static int build_prologue(struct jit_ctx *ctx) * | ... | BPF prog stack * | | * +-----+ <= (BPF_FP - prog->aux->stack_depth) - * |RSVD | JIT scratchpad + * |RSVD | padding * current A64_SP => +-----+ <= (BPF_FP - ctx->stack_size) * | | * | ... | Function call stack @@ -210,19 +232,19 @@ static int build_prologue(struct jit_ctx *ctx) /* Set up BPF prog stack base register */ emit(A64_MOV(1, fp, A64_SP), ctx); - /* Initialize tail_call_cnt */ - emit(A64_MOVZ(1, tcc, 0, 0), ctx); + if (!ebpf_from_cbpf) { + /* Initialize tail_call_cnt */ + emit(A64_MOVZ(1, tcc, 0, 0), ctx); - cur_offset = ctx->idx - idx0; - if (cur_offset != PROLOGUE_OFFSET) { - pr_err_once("PROLOGUE_OFFSET = %d, expected %d!\n", - cur_offset, PROLOGUE_OFFSET); - return -1; + cur_offset = ctx->idx - idx0; + if (cur_offset != PROLOGUE_OFFSET) { + pr_err_once("PROLOGUE_OFFSET = %d, expected %d!\n", + cur_offset, PROLOGUE_OFFSET); + return -1; + } } - /* 4 byte extra for skb_copy_bits buffer */ - ctx->stack_size = prog->aux->stack_depth + 4; - ctx->stack_size = STACK_ALIGN(ctx->stack_size); + ctx->stack_size = STACK_ALIGN(prog->aux->stack_depth); /* Set up function call stack */ emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); @@ -723,71 +745,6 @@ emit_cond_jmp: emit(A64_CBNZ(0, tmp3, jmp_offset), ctx); break; - /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + imm)) */ - case BPF_LD | BPF_ABS | BPF_W: - case BPF_LD | BPF_ABS | BPF_H: - case BPF_LD | BPF_ABS | BPF_B: - /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + src + imm)) */ - case BPF_LD | BPF_IND | BPF_W: - case BPF_LD | BPF_IND | BPF_H: - case BPF_LD | BPF_IND | BPF_B: - { - const u8 r0 = bpf2a64[BPF_REG_0]; /* r0 = return value */ - const u8 r6 = bpf2a64[BPF_REG_6]; /* r6 = pointer to sk_buff */ - const u8 fp = bpf2a64[BPF_REG_FP]; - const u8 r1 = bpf2a64[BPF_REG_1]; /* r1: struct sk_buff *skb */ - const u8 r2 = bpf2a64[BPF_REG_2]; /* r2: int k */ - const u8 r3 = bpf2a64[BPF_REG_3]; /* r3: unsigned int size */ - const u8 r4 = bpf2a64[BPF_REG_4]; /* r4: void *buffer */ - const u8 r5 = bpf2a64[BPF_REG_5]; /* r5: void *(*func)(...) */ - int size; - - emit(A64_MOV(1, r1, r6), ctx); - emit_a64_mov_i(0, r2, imm, ctx); - if (BPF_MODE(code) == BPF_IND) - emit(A64_ADD(0, r2, r2, src), ctx); - switch (BPF_SIZE(code)) { - case BPF_W: - size = 4; - break; - case BPF_H: - size = 2; - break; - case BPF_B: - size = 1; - break; - default: - return -EINVAL; - } - emit_a64_mov_i64(r3, size, ctx); - emit(A64_SUB_I(1, r4, fp, ctx->stack_size), ctx); - emit_a64_mov_i64(r5, (unsigned long)bpf_load_pointer, ctx); - emit(A64_BLR(r5), ctx); - emit(A64_MOV(1, r0, A64_R(0)), ctx); - - jmp_offset = epilogue_offset(ctx); - check_imm19(jmp_offset); - emit(A64_CBZ(1, r0, jmp_offset), ctx); - emit(A64_MOV(1, r5, r0), ctx); - switch (BPF_SIZE(code)) { - case BPF_W: - emit(A64_LDR32(r0, r5, A64_ZR), ctx); -#ifndef CONFIG_CPU_BIG_ENDIAN - emit(A64_REV32(0, r0, r0), ctx); -#endif - break; - case BPF_H: - emit(A64_LDRH(r0, r5, A64_ZR), ctx); -#ifndef CONFIG_CPU_BIG_ENDIAN - emit(A64_REV16(0, r0, r0), ctx); -#endif - break; - case BPF_B: - emit(A64_LDRB(r0, r5, A64_ZR), ctx); - break; - } - break; - } default: pr_err_once("unknown opcode %02x\n", code); return -EINVAL; @@ -851,6 +808,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) struct bpf_prog *tmp, *orig_prog = prog; struct bpf_binary_header *header; struct arm64_jit_data *jit_data; + bool was_classic = bpf_prog_was_classic(prog); bool tmp_blinded = false; bool extra_pass = false; struct jit_ctx ctx; @@ -905,7 +863,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) goto out_off; } - if (build_prologue(&ctx)) { + if (build_prologue(&ctx, was_classic)) { prog = orig_prog; goto out_off; } @@ -928,7 +886,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) skip_init_ctx: ctx.idx = 0; - build_prologue(&ctx); + build_prologue(&ctx, was_classic); if (build_body(&ctx)) { bpf_jit_binary_free(header); diff --git a/arch/c6x/Kconfig b/arch/c6x/Kconfig index c6b4dd1418b4..bf59855628ac 100644 --- a/arch/c6x/Kconfig +++ b/arch/c6x/Kconfig @@ -6,11 +6,13 @@ config C6X def_bool y + select ARCH_HAS_SYNC_DMA_FOR_CPU + select ARCH_HAS_SYNC_DMA_FOR_DEVICE select CLKDEV_LOOKUP + select DMA_NONCOHERENT_OPS select GENERIC_ATOMIC64 select GENERIC_IRQ_SHOW select HAVE_ARCH_TRACEHOOK - select HAVE_DMA_API_DEBUG select HAVE_MEMBLOCK select SPARSE_IRQ select IRQ_DOMAIN diff --git a/arch/c6x/include/asm/Kbuild b/arch/c6x/include/asm/Kbuild index fd4c840de837..33a2c94fed0d 100644 --- a/arch/c6x/include/asm/Kbuild +++ b/arch/c6x/include/asm/Kbuild @@ -1,10 +1,12 @@ generic-y += atomic.h generic-y += barrier.h generic-y += bugs.h +generic-y += compat.h generic-y += current.h generic-y += device.h generic-y += div64.h generic-y += dma.h +generic-y += dma-mapping.h generic-y += emergency-restart.h generic-y += exec.h generic-y += extable.h diff --git a/arch/c6x/include/asm/dma-mapping.h b/arch/c6x/include/asm/dma-mapping.h deleted file mode 100644 index 05daf1038111..000000000000 --- a/arch/c6x/include/asm/dma-mapping.h +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Port on Texas Instruments TMS320C6x architecture - * - * Copyright (C) 2004, 2009, 2010, 2011 Texas Instruments Incorporated - * Author: Aurelien Jacquiot - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - */ -#ifndef _ASM_C6X_DMA_MAPPING_H -#define _ASM_C6X_DMA_MAPPING_H - -extern const struct dma_map_ops c6x_dma_ops; - -static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus) -{ - return &c6x_dma_ops; -} - -extern void coherent_mem_init(u32 start, u32 size); -void *c6x_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle, - gfp_t gfp, unsigned long attrs); -void c6x_dma_free(struct device *dev, size_t size, void *vaddr, - dma_addr_t dma_handle, unsigned long attrs); - -#endif /* _ASM_C6X_DMA_MAPPING_H */ diff --git a/arch/c6x/include/asm/setup.h b/arch/c6x/include/asm/setup.h index 852afb209afb..350f34debb19 100644 --- a/arch/c6x/include/asm/setup.h +++ b/arch/c6x/include/asm/setup.h @@ -28,5 +28,7 @@ extern unsigned char c6x_fuse_mac[6]; extern void machine_init(unsigned long dt_ptr); extern void time_init(void); +extern void coherent_mem_init(u32 start, u32 size); + #endif /* !__ASSEMBLY__ */ #endif /* _ASM_C6X_SETUP_H */ diff --git a/arch/c6x/kernel/Makefile b/arch/c6x/kernel/Makefile index 02f340d7b8fe..fbe74174de87 100644 --- a/arch/c6x/kernel/Makefile +++ b/arch/c6x/kernel/Makefile @@ -8,6 +8,6 @@ extra-y := head.o vmlinux.lds obj-y := process.o traps.o irq.o signal.o ptrace.o obj-y += setup.o sys_c6x.o time.o devicetree.o obj-y += switch_to.o entry.o vectors.o c6x_ksyms.o -obj-y += soc.o dma.o +obj-y += soc.o obj-$(CONFIG_MODULES) += module.o diff --git a/arch/c6x/kernel/dma.c b/arch/c6x/kernel/dma.c deleted file mode 100644 index 9fff8be75f58..000000000000 --- a/arch/c6x/kernel/dma.c +++ /dev/null @@ -1,149 +0,0 @@ -/* - * Copyright (C) 2011 Texas Instruments Incorporated - * Author: Mark Salter - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ -#include -#include -#include -#include -#include - -#include - -static void c6x_dma_sync(dma_addr_t handle, size_t size, - enum dma_data_direction dir) -{ - unsigned long paddr = handle; - - BUG_ON(!valid_dma_direction(dir)); - - switch (dir) { - case DMA_FROM_DEVICE: - L2_cache_block_invalidate(paddr, paddr + size); - break; - case DMA_TO_DEVICE: - L2_cache_block_writeback(paddr, paddr + size); - break; - case DMA_BIDIRECTIONAL: - L2_cache_block_writeback_invalidate(paddr, paddr + size); - break; - default: - break; - } -} - -static dma_addr_t c6x_dma_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - dma_addr_t handle = virt_to_phys(page_address(page) + offset); - - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - c6x_dma_sync(handle, size, dir); - - return handle; -} - -static void c6x_dma_unmap_page(struct device *dev, dma_addr_t handle, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - c6x_dma_sync(handle, size, dir); -} - -static int c6x_dma_map_sg(struct device *dev, struct scatterlist *sglist, - int nents, enum dma_data_direction dir, unsigned long attrs) -{ - struct scatterlist *sg; - int i; - - for_each_sg(sglist, sg, nents, i) { - sg->dma_address = sg_phys(sg); - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - c6x_dma_sync(sg->dma_address, sg->length, dir); - } - - return nents; -} - -static void c6x_dma_unmap_sg(struct device *dev, struct scatterlist *sglist, - int nents, enum dma_data_direction dir, unsigned long attrs) -{ - struct scatterlist *sg; - int i; - - if (attrs & DMA_ATTR_SKIP_CPU_SYNC) - return; - - for_each_sg(sglist, sg, nents, i) - c6x_dma_sync(sg_dma_address(sg), sg->length, dir); -} - -static void c6x_dma_sync_single_for_cpu(struct device *dev, dma_addr_t handle, - size_t size, enum dma_data_direction dir) -{ - c6x_dma_sync(handle, size, dir); - -} - -static void c6x_dma_sync_single_for_device(struct device *dev, - dma_addr_t handle, size_t size, enum dma_data_direction dir) -{ - c6x_dma_sync(handle, size, dir); - -} - -static void c6x_dma_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sglist, int nents, - enum dma_data_direction dir) -{ - struct scatterlist *sg; - int i; - - for_each_sg(sglist, sg, nents, i) - c6x_dma_sync_single_for_cpu(dev, sg_dma_address(sg), - sg->length, dir); - -} - -static void c6x_dma_sync_sg_for_device(struct device *dev, - struct scatterlist *sglist, int nents, - enum dma_data_direction dir) -{ - struct scatterlist *sg; - int i; - - for_each_sg(sglist, sg, nents, i) - c6x_dma_sync_single_for_device(dev, sg_dma_address(sg), - sg->length, dir); - -} - -const struct dma_map_ops c6x_dma_ops = { - .alloc = c6x_dma_alloc, - .free = c6x_dma_free, - .map_page = c6x_dma_map_page, - .unmap_page = c6x_dma_unmap_page, - .map_sg = c6x_dma_map_sg, - .unmap_sg = c6x_dma_unmap_sg, - .sync_single_for_device = c6x_dma_sync_single_for_device, - .sync_single_for_cpu = c6x_dma_sync_single_for_cpu, - .sync_sg_for_device = c6x_dma_sync_sg_for_device, - .sync_sg_for_cpu = c6x_dma_sync_sg_for_cpu, -}; -EXPORT_SYMBOL(c6x_dma_ops); - -/* Number of entries preallocated for DMA-API debugging */ -#define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16) - -static int __init dma_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - - return 0; -} -fs_initcall(dma_init); diff --git a/arch/c6x/kernel/traps.c b/arch/c6x/kernel/traps.c index 4c1d4b84dd2b..5c60aea3b75a 100644 --- a/arch/c6x/kernel/traps.c +++ b/arch/c6x/kernel/traps.c @@ -244,7 +244,6 @@ static struct exception_info eexcept_table[128] = { static void do_trap(struct exception_info *except_info, struct pt_regs *regs) { unsigned long addr = instruction_pointer(regs); - siginfo_t info; if (except_info->code != TRAP_BRKPT) pr_err("TRAP: %s PC[0x%lx] signo[%d] code[%d]\n", @@ -253,12 +252,8 @@ static void do_trap(struct exception_info *except_info, struct pt_regs *regs) die_if_kernel(except_info->kernel_str, regs, addr); - info.si_signo = except_info->signo; - info.si_errno = 0; - info.si_code = except_info->code; - info.si_addr = (void __user *)addr; - - force_sig_info(except_info->signo, &info, current); + force_sig_fault(except_info->signo, except_info->code, + (void __user *)addr, current); } /* diff --git a/arch/c6x/mm/dma-coherent.c b/arch/c6x/mm/dma-coherent.c index 95e38ad27c69..d0a8e0c4b27e 100644 --- a/arch/c6x/mm/dma-coherent.c +++ b/arch/c6x/mm/dma-coherent.c @@ -19,10 +19,12 @@ #include #include #include -#include +#include #include +#include #include +#include /* * DMA coherent memory management, can be redefined using the memdma= @@ -73,7 +75,7 @@ static void __free_dma_pages(u32 addr, int order) * Allocate DMA coherent memory space and return both the kernel * virtual and DMA address for that space. */ -void *c6x_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle, +void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle, gfp_t gfp, unsigned long attrs) { u32 paddr; @@ -98,7 +100,7 @@ void *c6x_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle, /* * Free DMA coherent memory as defined by the above mapping. */ -void c6x_dma_free(struct device *dev, size_t size, void *vaddr, +void arch_dma_free(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_handle, unsigned long attrs) { int order; @@ -139,3 +141,35 @@ void __init coherent_mem_init(phys_addr_t start, u32 size) dma_bitmap = phys_to_virt(bitmap_phys); memset(dma_bitmap, 0, dma_pages * PAGE_SIZE); } + +static void c6x_dma_sync(struct device *dev, phys_addr_t paddr, size_t size, + enum dma_data_direction dir) +{ + BUG_ON(!valid_dma_direction(dir)); + + switch (dir) { + case DMA_FROM_DEVICE: + L2_cache_block_invalidate(paddr, paddr + size); + break; + case DMA_TO_DEVICE: + L2_cache_block_writeback(paddr, paddr + size); + break; + case DMA_BIDIRECTIONAL: + L2_cache_block_writeback_invalidate(paddr, paddr + size); + break; + default: + break; + } +} + +void arch_sync_dma_for_device(struct device *dev, phys_addr_t paddr, + size_t size, enum dma_data_direction dir) +{ + return c6x_dma_sync(dev, paddr, size, dir); +} + +void arch_sync_dma_for_cpu(struct device *dev, phys_addr_t paddr, + size_t size, enum dma_data_direction dir) +{ + return c6x_dma_sync(dev, paddr, size, dir); +} diff --git a/arch/h8300/include/asm/Kbuild b/arch/h8300/include/asm/Kbuild index 14bac06b7116..a5d0b2991f47 100644 --- a/arch/h8300/include/asm/Kbuild +++ b/arch/h8300/include/asm/Kbuild @@ -3,6 +3,7 @@ generic-y += barrier.h generic-y += bugs.h generic-y += cacheflush.h generic-y += checksum.h +generic-y += compat.h generic-y += current.h generic-y += delay.h generic-y += device.h diff --git a/arch/h8300/include/asm/pci.h b/arch/h8300/include/asm/pci.h index 7c9e55d62215..d4d345a52092 100644 --- a/arch/h8300/include/asm/pci.h +++ b/arch/h8300/include/asm/pci.h @@ -15,6 +15,4 @@ static inline void pcibios_penalize_isa_irq(int irq, int active) /* We don't do dynamic PCI IRQ allocation */ } -#define PCI_DMA_BUS_IS_PHYS (1) - #endif /* _ASM_H8300_PCI_H */ diff --git a/arch/hexagon/Kconfig b/arch/hexagon/Kconfig index 76d2f20d525e..37adb2003033 100644 --- a/arch/hexagon/Kconfig +++ b/arch/hexagon/Kconfig @@ -19,6 +19,7 @@ config HEXAGON select GENERIC_IRQ_SHOW select HAVE_ARCH_KGDB select HAVE_ARCH_TRACEHOOK + select NEED_SG_DMA_LENGTH select NO_IOPORT_MAP select GENERIC_IOMAP select GENERIC_SMP_IDLE_THREAD @@ -63,9 +64,6 @@ config GENERIC_CSUM config GENERIC_IRQ_PROBE def_bool y -config NEED_SG_DMA_LENGTH - def_bool y - config RWSEM_GENERIC_SPINLOCK def_bool n diff --git a/arch/hexagon/include/asm/Kbuild b/arch/hexagon/include/asm/Kbuild index e9743f689fb8..dd2fd9c0d292 100644 --- a/arch/hexagon/include/asm/Kbuild +++ b/arch/hexagon/include/asm/Kbuild @@ -2,6 +2,7 @@ generic-y += barrier.h generic-y += bug.h generic-y += bugs.h +generic-y += compat.h generic-y += current.h generic-y += device.h generic-y += div64.h diff --git a/arch/hexagon/include/asm/pgtable.h b/arch/hexagon/include/asm/pgtable.h index aef02f7ca8aa..65125d0b02dd 100644 --- a/arch/hexagon/include/asm/pgtable.h +++ b/arch/hexagon/include/asm/pgtable.h @@ -30,7 +30,6 @@ /* A handy thing to have if one has the RAM. Declared in head.S */ extern unsigned long empty_zero_page; -extern unsigned long zero_page_mask; /* * The PTE model described here is that of the Hexagon Virtual Machine, diff --git a/arch/hexagon/kernel/dma.c b/arch/hexagon/kernel/dma.c index ad8347c29dcf..77459df34e2e 100644 --- a/arch/hexagon/kernel/dma.c +++ b/arch/hexagon/kernel/dma.c @@ -208,7 +208,6 @@ const struct dma_map_ops hexagon_dma_ops = { .sync_single_for_cpu = hexagon_sync_single_for_cpu, .sync_single_for_device = hexagon_sync_single_for_device, .mapping_error = hexagon_mapping_error, - .is_phys = 1, }; void __init hexagon_dma_init(void) diff --git a/arch/hexagon/kernel/setup.c b/arch/hexagon/kernel/setup.c index 6981949f5df3..dc8c7e75b5d1 100644 --- a/arch/hexagon/kernel/setup.c +++ b/arch/hexagon/kernel/setup.c @@ -66,7 +66,7 @@ void __init setup_arch(char **cmdline_p) */ __vmsetvec(_K_VM_event_vector); - printk(KERN_INFO "PHYS_OFFSET=0x%08x\n", PHYS_OFFSET); + printk(KERN_INFO "PHYS_OFFSET=0x%08lx\n", PHYS_OFFSET); /* * Simulator has a few differences from the hardware. diff --git a/arch/hexagon/kernel/traps.c b/arch/hexagon/kernel/traps.c index 2942a9204a9a..91ee04842c22 100644 --- a/arch/hexagon/kernel/traps.c +++ b/arch/hexagon/kernel/traps.c @@ -412,10 +412,6 @@ void do_trap0(struct pt_regs *regs) case TRAP_DEBUG: /* Trap0 0xdb is debug breakpoint */ if (user_mode(regs)) { - struct siginfo info; - - info.si_signo = SIGTRAP; - info.si_errno = 0; /* * Some architecures add some per-thread state * to distinguish between breakpoint traps and @@ -423,9 +419,8 @@ void do_trap0(struct pt_regs *regs) * set the si_code value appropriately, or we * may want to use a different trap0 flavor. */ - info.si_code = TRAP_BRKPT; - info.si_addr = (void __user *) pt_elr(regs); - force_sig_info(SIGTRAP, &info, current); + force_sig_fault(SIGTRAP, TRAP_BRKPT, + (void __user *) pt_elr(regs), current); } else { #ifdef CONFIG_KGDB kgdb_handle_exception(pt_cause(regs), SIGTRAP, diff --git a/arch/hexagon/mm/init.c b/arch/hexagon/mm/init.c index 192584d5ac2f..1495d45e472d 100644 --- a/arch/hexagon/mm/init.c +++ b/arch/hexagon/mm/init.c @@ -39,9 +39,6 @@ unsigned long __phys_offset; /* physical kernel offset >> 12 */ /* Set as variable to limit PMD copies */ int max_kernel_seg = 0x303; -/* think this should be (page_size-1) the way it's used...*/ -unsigned long zero_page_mask; - /* indicate pfn's of high memory */ unsigned long highstart_pfn, highend_pfn; diff --git a/arch/hexagon/mm/vm_fault.c b/arch/hexagon/mm/vm_fault.c index 3eec33c5cfd7..933bbcef5363 100644 --- a/arch/hexagon/mm/vm_fault.c +++ b/arch/hexagon/mm/vm_fault.c @@ -50,7 +50,7 @@ void do_page_fault(unsigned long address, long cause, struct pt_regs *regs) { struct vm_area_struct *vma; struct mm_struct *mm = current->mm; - siginfo_t info; + int si_signo; int si_code = SEGV_MAPERR; int fault; const struct exception_table_entry *fixup; @@ -140,28 +140,22 @@ good_area: * unable to fix up the page fault. */ if (fault & VM_FAULT_SIGBUS) { - info.si_signo = SIGBUS; - info.si_code = BUS_ADRERR; + si_signo = SIGBUS; + si_code = BUS_ADRERR; } /* Address is not in the memory map */ else { - info.si_signo = SIGSEGV; - info.si_code = SEGV_ACCERR; + si_signo = SIGSEGV; + si_code = SEGV_ACCERR; } - info.si_errno = 0; - info.si_addr = (void __user *)address; - force_sig_info(info.si_signo, &info, current); + force_sig_fault(si_signo, si_code, (void __user *)address, current); return; bad_area: up_read(&mm->mmap_sem); if (user_mode(regs)) { - info.si_signo = SIGSEGV; - info.si_errno = 0; - info.si_code = si_code; - info.si_addr = (void *)address; - force_sig_info(info.si_signo, &info, current); + force_sig_fault(SIGSEGV, si_code, (void __user *)address, current); return; } /* Kernel-mode fault falls through */ diff --git a/arch/ia64/Kconfig b/arch/ia64/Kconfig index bbe12a038d21..ff861420b8f5 100644 --- a/arch/ia64/Kconfig +++ b/arch/ia64/Kconfig @@ -29,7 +29,6 @@ config IA64 select HAVE_FUNCTION_TRACER select TTY select HAVE_ARCH_TRACEHOOK - select HAVE_DMA_API_DEBUG select HAVE_MEMBLOCK select HAVE_MEMBLOCK_NODE_MAP select HAVE_VIRT_CPU_ACCOUNTING @@ -54,6 +53,8 @@ config IA64 select MODULES_USE_ELF_RELA select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_AUDITSYSCALL + select NEED_DMA_MAP_STATE + select NEED_SG_DMA_LENGTH default y help The Itanium Processor Family is Intel's 64-bit successor to @@ -78,18 +79,6 @@ config MMU bool default y -config ARCH_DMA_ADDR_T_64BIT - def_bool y - -config NEED_DMA_MAP_STATE - def_bool y - -config NEED_SG_DMA_LENGTH - def_bool y - -config SWIOTLB - bool - config STACKTRACE_SUPPORT def_bool y @@ -146,7 +135,6 @@ config IA64_GENERIC bool "generic" select NUMA select ACPI_NUMA - select DMA_DIRECT_OPS select SWIOTLB select PCI_MSI help @@ -167,7 +155,6 @@ config IA64_GENERIC config IA64_DIG bool "DIG-compliant" - select DMA_DIRECT_OPS select SWIOTLB config IA64_DIG_VTD @@ -183,7 +170,6 @@ config IA64_HP_ZX1 config IA64_HP_ZX1_SWIOTLB bool "HP-zx1/sx1000 with software I/O TLB" - select DMA_DIRECT_OPS select SWIOTLB help Build a kernel that runs on HP zx1 and sx1000 systems even when they @@ -207,7 +193,6 @@ config IA64_SGI_UV bool "SGI-UV" select NUMA select ACPI_NUMA - select DMA_DIRECT_OPS select SWIOTLB help Selecting this option will optimize the kernel for use on UV based @@ -218,7 +203,6 @@ config IA64_SGI_UV config IA64_HP_SIM bool "Ski-simulator" - select DMA_DIRECT_OPS select SWIOTLB depends on !PM @@ -397,7 +381,7 @@ config ARCH_DISCONTIGMEM_ENABLE Say Y to support efficient handling of discontiguous physical memory, for architectures which are either NUMA (Non-Uniform Memory Access) or have huge holes in the physical address space for other reasons. - See for more. + See for more. config ARCH_FLATMEM_ENABLE def_bool y @@ -471,6 +455,7 @@ config IA64_MCA_RECOVERY config PERFMON bool "Performance monitor support" + depends on BROKEN help Selects whether support for the IA-64 performance monitor hardware is included in the kernel. This makes some kernel data-structures a @@ -613,6 +598,3 @@ source "security/Kconfig" source "crypto/Kconfig" source "lib/Kconfig" - -config IOMMU_HELPER - def_bool (IA64_HP_ZX1 || IA64_HP_ZX1_SWIOTLB || IA64_GENERIC || SWIOTLB) diff --git a/arch/ia64/Makefile b/arch/ia64/Makefile index 2dd7f519ad0b..45f59808b842 100644 --- a/arch/ia64/Makefile +++ b/arch/ia64/Makefile @@ -18,7 +18,7 @@ READELF := $(CROSS_COMPILE)readelf export AWK -CHECKFLAGS += -m64 -D__ia64=1 -D__ia64__=1 -D_LP64 -D__LP64__ +CHECKFLAGS += -D__ia64=1 -D__ia64__=1 -D_LP64 -D__LP64__ OBJCOPYFLAGS := --strip-all LDFLAGS_vmlinux := -static diff --git a/arch/ia64/hp/common/sba_iommu.c b/arch/ia64/hp/common/sba_iommu.c index aec4a3354abe..ee5b652d320a 100644 --- a/arch/ia64/hp/common/sba_iommu.c +++ b/arch/ia64/hp/common/sba_iommu.c @@ -1845,9 +1845,6 @@ static void ioc_init(unsigned long hpa, struct ioc *ioc) ioc_resource_init(ioc); ioc_sac_init(ioc); - if ((long) ~iovp_mask > (long) ia64_max_iommu_merge_mask) - ia64_max_iommu_merge_mask = ~iovp_mask; - printk(KERN_INFO PFX "%s %d.%d HPA 0x%lx IOVA space %dMb at 0x%lx\n", ioc->name, (ioc->rev >> 4) & 0xF, ioc->rev & 0xF, @@ -1942,19 +1939,6 @@ static const struct seq_operations ioc_seq_ops = { .show = ioc_show }; -static int -ioc_open(struct inode *inode, struct file *file) -{ - return seq_open(file, &ioc_seq_ops); -} - -static const struct file_operations ioc_fops = { - .open = ioc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = seq_release -}; - static void __init ioc_proc_init(void) { @@ -1964,7 +1948,7 @@ ioc_proc_init(void) if (!dir) return; - proc_create(ioc_list->name, 0, dir, &ioc_fops); + proc_create_seq(ioc_list->name, 0, dir, &ioc_seq_ops); } #endif diff --git a/arch/ia64/hp/sim/simserial.c b/arch/ia64/hp/sim/simserial.c index a419ccf33cde..663388a73d4e 100644 --- a/arch/ia64/hp/sim/simserial.c +++ b/arch/ia64/hp/sim/simserial.c @@ -435,19 +435,6 @@ static int rs_proc_show(struct seq_file *m, void *v) return 0; } -static int rs_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, rs_proc_show, NULL); -} - -static const struct file_operations rs_proc_fops = { - .owner = THIS_MODULE, - .open = rs_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static const struct tty_operations hp_ops = { .open = rs_open, .close = rs_close, @@ -462,7 +449,7 @@ static const struct tty_operations hp_ops = { .unthrottle = rs_unthrottle, .send_xchar = rs_send_xchar, .hangup = rs_hangup, - .proc_fops = &rs_proc_fops, + .proc_show = rs_proc_show, }; static const struct tty_port_operations hp_port_ops = { diff --git a/arch/ia64/include/asm/Kbuild b/arch/ia64/include/asm/Kbuild index 6dd867873364..557bbc8ba9f5 100644 --- a/arch/ia64/include/asm/Kbuild +++ b/arch/ia64/include/asm/Kbuild @@ -1,3 +1,4 @@ +generic-y += compat.h generic-y += exec.h generic-y += irq_work.h generic-y += mcs_spinlock.h diff --git a/arch/ia64/include/asm/hardirq.h b/arch/ia64/include/asm/hardirq.h index bdc4669c71c3..ccde7c2ba00f 100644 --- a/arch/ia64/include/asm/hardirq.h +++ b/arch/ia64/include/asm/hardirq.h @@ -13,7 +13,7 @@ #define __ARCH_IRQ_STAT 1 -#define local_softirq_pending() (local_cpu_data->softirq_pending) +#define local_softirq_pending_ref ia64_cpu_info.softirq_pending #include #include diff --git a/arch/ia64/include/asm/pci.h b/arch/ia64/include/asm/pci.h index b1d04e8bafc8..780e8744ba85 100644 --- a/arch/ia64/include/asm/pci.h +++ b/arch/ia64/include/asm/pci.h @@ -30,23 +30,6 @@ struct pci_vector_struct { #define PCIBIOS_MIN_IO 0x1000 #define PCIBIOS_MIN_MEM 0x10000000 -/* - * PCI_DMA_BUS_IS_PHYS should be set to 1 if there is _necessarily_ a direct - * correspondence between device bus addresses and CPU physical addresses. - * Platforms with a hardware I/O MMU _must_ turn this off to suppress the - * bounce buffer handling code in the block and network device layers. - * Platforms with separate bus address spaces _must_ turn this off and provide - * a device DMA mapping implementation that takes care of the necessary - * address translation. - * - * For now, the ia64 platforms which may have separate/multiple bus address - * spaces all have I/O MMUs which support the merging of physically - * discontiguous buffers, so we can use that as the sole factor to determine - * the setting of PCI_DMA_BUS_IS_PHYS. - */ -extern unsigned long ia64_max_iommu_merge_mask; -#define PCI_DMA_BUS_IS_PHYS (ia64_max_iommu_merge_mask == ~0UL) - #define HAVE_PCI_MMAP #define ARCH_GENERIC_PCI_MMAP_RESOURCE #define arch_can_pci_mmap_wc() 1 diff --git a/arch/ia64/include/uapi/asm/Kbuild b/arch/ia64/include/uapi/asm/Kbuild index c0527cfc48f0..3982e673e967 100644 --- a/arch/ia64/include/uapi/asm/Kbuild +++ b/arch/ia64/include/uapi/asm/Kbuild @@ -2,5 +2,9 @@ include include/uapi/asm-generic/Kbuild.asm generic-y += bpf_perf_event.h +generic-y += ipcbuf.h generic-y += kvm_para.h +generic-y += msgbuf.h generic-y += poll.h +generic-y += sembuf.h +generic-y += shmbuf.h diff --git a/arch/ia64/include/uapi/asm/ipcbuf.h b/arch/ia64/include/uapi/asm/ipcbuf.h deleted file mode 100644 index 90d6445a14df..000000000000 --- a/arch/ia64/include/uapi/asm/ipcbuf.h +++ /dev/null @@ -1,2 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#include diff --git a/arch/ia64/include/uapi/asm/msgbuf.h b/arch/ia64/include/uapi/asm/msgbuf.h deleted file mode 100644 index aa25df92d9dc..000000000000 --- a/arch/ia64/include/uapi/asm/msgbuf.h +++ /dev/null @@ -1,28 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _ASM_IA64_MSGBUF_H -#define _ASM_IA64_MSGBUF_H - -/* - * The msqid64_ds structure for IA-64 architecture. - * Note extra padding because this structure is passed back and forth - * between kernel and user space. - * - * Pad space is left for: - * - 2 miscellaneous 64-bit values - */ - -struct msqid64_ds { - struct ipc64_perm msg_perm; - __kernel_time_t msg_stime; /* last msgsnd time */ - __kernel_time_t msg_rtime; /* last msgrcv time */ - __kernel_time_t msg_ctime; /* last change time */ - unsigned long msg_cbytes; /* current number of bytes on queue */ - unsigned long msg_qnum; /* number of messages in queue */ - unsigned long msg_qbytes; /* max number of bytes on queue */ - __kernel_pid_t msg_lspid; /* pid of last msgsnd */ - __kernel_pid_t msg_lrpid; /* last receive pid */ - unsigned long __unused1; - unsigned long __unused2; -}; - -#endif /* _ASM_IA64_MSGBUF_H */ diff --git a/arch/ia64/include/uapi/asm/sembuf.h b/arch/ia64/include/uapi/asm/sembuf.h deleted file mode 100644 index 6ed058760afc..000000000000 --- a/arch/ia64/include/uapi/asm/sembuf.h +++ /dev/null @@ -1,23 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _ASM_IA64_SEMBUF_H -#define _ASM_IA64_SEMBUF_H - -/* - * The semid64_ds structure for IA-64 architecture. - * Note extra padding because this structure is passed back and forth - * between kernel and user space. - * - * Pad space is left for: - * - 2 miscellaneous 64-bit values - */ - -struct semid64_ds { - struct ipc64_perm sem_perm; /* permissions .. see ipc.h */ - __kernel_time_t sem_otime; /* last semop time */ - __kernel_time_t sem_ctime; /* last change time */ - unsigned long sem_nsems; /* no. of semaphores in array */ - unsigned long __unused1; - unsigned long __unused2; -}; - -#endif /* _ASM_IA64_SEMBUF_H */ diff --git a/arch/ia64/include/uapi/asm/shmbuf.h b/arch/ia64/include/uapi/asm/shmbuf.h deleted file mode 100644 index 6ef57cb70dee..000000000000 --- a/arch/ia64/include/uapi/asm/shmbuf.h +++ /dev/null @@ -1,39 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _ASM_IA64_SHMBUF_H -#define _ASM_IA64_SHMBUF_H - -/* - * The shmid64_ds structure for IA-64 architecture. - * Note extra padding because this structure is passed back and forth - * between kernel and user space. - * - * Pad space is left for: - * - 2 miscellaneous 64-bit values - */ - -struct shmid64_ds { - struct ipc64_perm shm_perm; /* operation perms */ - size_t shm_segsz; /* size of segment (bytes) */ - __kernel_time_t shm_atime; /* last attach time */ - __kernel_time_t shm_dtime; /* last detach time */ - __kernel_time_t shm_ctime; /* last change time */ - __kernel_pid_t shm_cpid; /* pid of creator */ - __kernel_pid_t shm_lpid; /* pid of last operator */ - unsigned long shm_nattch; /* no. of current attaches */ - unsigned long __unused1; - unsigned long __unused2; -}; - -struct shminfo64 { - unsigned long shmmax; - unsigned long shmmin; - unsigned long shmmni; - unsigned long shmseg; - unsigned long shmall; - unsigned long __unused1; - unsigned long __unused2; - unsigned long __unused3; - unsigned long __unused4; -}; - -#endif /* _ASM_IA64_SHMBUF_H */ diff --git a/arch/ia64/include/uapi/asm/siginfo.h b/arch/ia64/include/uapi/asm/siginfo.h index 5aa454ed89db..52b5af424511 100644 --- a/arch/ia64/include/uapi/asm/siginfo.h +++ b/arch/ia64/include/uapi/asm/siginfo.h @@ -27,11 +27,4 @@ #define __ISR_VALID_BIT 0 #define __ISR_VALID (1 << __ISR_VALID_BIT) -/* - * SIGFPE si_codes - */ -#ifdef __KERNEL__ -#define FPE_FIXME 0 /* Broken dup of SI_USER */ -#endif /* __KERNEL__ */ - #endif /* _UAPI_ASM_IA64_SIGINFO_H */ diff --git a/arch/ia64/kernel/brl_emu.c b/arch/ia64/kernel/brl_emu.c index 9bcc908bc85e..a61f6c6a36f8 100644 --- a/arch/ia64/kernel/brl_emu.c +++ b/arch/ia64/kernel/brl_emu.c @@ -62,6 +62,7 @@ ia64_emulate_brl (struct pt_regs *regs, unsigned long ar_ec) struct illegal_op_return rv; long tmp_taken, unimplemented_address; + clear_siginfo(&siginfo); rv.fkt = (unsigned long) -1; /* diff --git a/arch/ia64/kernel/dma-mapping.c b/arch/ia64/kernel/dma-mapping.c index f2d57e66fd86..7a471d8d67d4 100644 --- a/arch/ia64/kernel/dma-mapping.c +++ b/arch/ia64/kernel/dma-mapping.c @@ -9,16 +9,6 @@ int iommu_detected __read_mostly; const struct dma_map_ops *dma_ops; EXPORT_SYMBOL(dma_ops); -#define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16) - -static int __init dma_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - - return 0; -} -fs_initcall(dma_init); - const struct dma_map_ops *dma_get_ops(struct device *dev) { return dma_ops; diff --git a/arch/ia64/kernel/mca_drv.c b/arch/ia64/kernel/mca_drv.c index 94f8bf777afa..dfe40cbdf3b3 100644 --- a/arch/ia64/kernel/mca_drv.c +++ b/arch/ia64/kernel/mca_drv.c @@ -350,7 +350,8 @@ init_record_index_pools(void) /* - 3 - */ slidx_pool.max_idx = (rec_max_size/sect_min_size) * 2 + 1; slidx_pool.buffer = - kmalloc(slidx_pool.max_idx * sizeof(slidx_list_t), GFP_KERNEL); + kmalloc_array(slidx_pool.max_idx, sizeof(slidx_list_t), + GFP_KERNEL); return slidx_pool.buffer ? 0 : -ENOMEM; } diff --git a/arch/ia64/kernel/palinfo.c b/arch/ia64/kernel/palinfo.c index b6e597860888..f4a94241265c 100644 --- a/arch/ia64/kernel/palinfo.c +++ b/arch/ia64/kernel/palinfo.c @@ -920,18 +920,6 @@ static int proc_palinfo_show(struct seq_file *m, void *v) return 0; } -static int proc_palinfo_open(struct inode *inode, struct file *file) -{ - return single_open(file, proc_palinfo_show, PDE_DATA(inode)); -} - -static const struct file_operations proc_palinfo_fops = { - .open = proc_palinfo_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int palinfo_add_proc(unsigned int cpu) { pal_func_cpu_u_t f; @@ -948,8 +936,8 @@ static int palinfo_add_proc(unsigned int cpu) for (j=0; j < NR_PALINFO_ENTRIES; j++) { f.func_id = j; - proc_create_data(palinfo_entries[j].name, 0, cpu_dir, - &proc_palinfo_fops, (void *)f.value); + proc_create_single_data(palinfo_entries[j].name, 0, cpu_dir, + proc_palinfo_show, (void *)f.value); } return 0; } diff --git a/arch/ia64/kernel/perfmon.c b/arch/ia64/kernel/perfmon.c index 8fb280e33114..3b38c717008a 100644 --- a/arch/ia64/kernel/perfmon.c +++ b/arch/ia64/kernel/perfmon.c @@ -5708,13 +5708,6 @@ const struct seq_operations pfm_seq_ops = { .show = pfm_proc_show }; -static int -pfm_proc_open(struct inode *inode, struct file *file) -{ - return seq_open(file, &pfm_seq_ops); -} - - /* * we come here as soon as local_cpu_data->pfm_syst_wide is set. this happens * during pfm_enable() hence before pfm_start(). We cannot assume monitoring @@ -6537,13 +6530,6 @@ found: return 0; } -static const struct file_operations pfm_proc_fops = { - .open = pfm_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = seq_release, -}; - int __init pfm_init(void) { @@ -6615,7 +6601,7 @@ pfm_init(void) /* * create /proc/perfmon (mostly for debugging purposes) */ - perfmon_dir = proc_create("perfmon", S_IRUGO, NULL, &pfm_proc_fops); + perfmon_dir = proc_create_seq("perfmon", S_IRUGO, NULL, &pfm_seq_ops); if (perfmon_dir == NULL) { printk(KERN_ERR "perfmon: cannot create /proc entry, perfmon disabled\n"); pmu_conf = NULL; diff --git a/arch/ia64/kernel/salinfo.c b/arch/ia64/kernel/salinfo.c index 52c404b08904..aba1f463a8dd 100644 --- a/arch/ia64/kernel/salinfo.c +++ b/arch/ia64/kernel/salinfo.c @@ -54,8 +54,6 @@ MODULE_AUTHOR("Jesse Barnes "); MODULE_DESCRIPTION("/proc interface to IA-64 SAL features"); MODULE_LICENSE("GPL"); -static const struct file_operations proc_salinfo_fops; - typedef struct { const char *name; /* name of the proc entry */ unsigned long feature; /* feature bit */ @@ -578,6 +576,17 @@ static int salinfo_cpu_pre_down(unsigned int cpu) return 0; } +/* + * 'data' contains an integer that corresponds to the feature we're + * testing + */ +static int proc_salinfo_show(struct seq_file *m, void *v) +{ + unsigned long data = (unsigned long)v; + seq_puts(m, (sal_platform_features & data) ? "1\n" : "0\n"); + return 0; +} + static int __init salinfo_init(void) { @@ -593,9 +602,9 @@ salinfo_init(void) for (i=0; i < NR_SALINFO_ENTRIES; i++) { /* pass the feature bit in question as misc data */ - *sdir++ = proc_create_data(salinfo_entries[i].name, 0, salinfo_dir, - &proc_salinfo_fops, - (void *)salinfo_entries[i].feature); + *sdir++ = proc_create_single_data(salinfo_entries[i].name, 0, + salinfo_dir, proc_salinfo_show, + (void *)salinfo_entries[i].feature); } for (i = 0; i < ARRAY_SIZE(salinfo_log_name); i++) { @@ -633,27 +642,4 @@ salinfo_init(void) return 0; } -/* - * 'data' contains an integer that corresponds to the feature we're - * testing - */ -static int proc_salinfo_show(struct seq_file *m, void *v) -{ - unsigned long data = (unsigned long)v; - seq_puts(m, (sal_platform_features & data) ? "1\n" : "0\n"); - return 0; -} - -static int proc_salinfo_open(struct inode *inode, struct file *file) -{ - return single_open(file, proc_salinfo_show, PDE_DATA(inode)); -} - -static const struct file_operations proc_salinfo_fops = { - .open = proc_salinfo_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - module_init(salinfo_init); diff --git a/arch/ia64/kernel/setup.c b/arch/ia64/kernel/setup.c index dee56bcb993d..ad43cbf70628 100644 --- a/arch/ia64/kernel/setup.c +++ b/arch/ia64/kernel/setup.c @@ -123,18 +123,6 @@ unsigned long ia64_i_cache_stride_shift = ~0; #define CACHE_STRIDE_SHIFT 5 unsigned long ia64_cache_stride_shift = ~0; -/* - * The merge_mask variable needs to be set to (max(iommu_page_size(iommu)) - 1). This - * mask specifies a mask of address bits that must be 0 in order for two buffers to be - * mergeable by the I/O MMU (i.e., the end address of the first buffer and the start - * address of the second buffer must be aligned to (merge_mask+1) in order to be - * mergeable). By default, we assume there is no I/O MMU which can merge physically - * discontiguous buffers, so we set the merge_mask to ~0UL, which corresponds to a iommu - * page-size of 2^64. - */ -unsigned long ia64_max_iommu_merge_mask = ~0UL; -EXPORT_SYMBOL(ia64_max_iommu_merge_mask); - /* * We use a special marker for the end of memory and it uses the extra (+1) slot */ diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c index 54547c7cf8a2..d1234a5ba4c5 100644 --- a/arch/ia64/kernel/signal.c +++ b/arch/ia64/kernel/signal.c @@ -153,6 +153,7 @@ ia64_rt_sigreturn (struct sigscratch *scr) return retval; give_sigsegv: + clear_siginfo(&si); si.si_signo = SIGSEGV; si.si_errno = 0; si.si_code = SI_KERNEL; @@ -236,6 +237,7 @@ force_sigsegv_info (int sig, void __user *addr) unsigned long flags; struct siginfo si; + clear_siginfo(&si); if (sig == SIGSEGV) { /* * Acquiring siglock around the sa_handler-update is almost diff --git a/arch/ia64/kernel/topology.c b/arch/ia64/kernel/topology.c index d76529cbff20..9b820f7a6a98 100644 --- a/arch/ia64/kernel/topology.c +++ b/arch/ia64/kernel/topology.c @@ -85,7 +85,7 @@ static int __init topology_init(void) } #endif - sysfs_cpus = kzalloc(sizeof(struct ia64_cpu) * NR_CPUS, GFP_KERNEL); + sysfs_cpus = kcalloc(NR_CPUS, sizeof(struct ia64_cpu), GFP_KERNEL); if (!sysfs_cpus) panic("kzalloc in topology_init failed - NR_CPUS too big?"); @@ -319,8 +319,8 @@ static int cpu_cache_sysfs_init(unsigned int cpu) return -1; } - this_cache=kzalloc(sizeof(struct cache_info)*unique_caches, - GFP_KERNEL); + this_cache=kcalloc(unique_caches, sizeof(struct cache_info), + GFP_KERNEL); if (this_cache == NULL) return -ENOMEM; diff --git a/arch/ia64/kernel/traps.c b/arch/ia64/kernel/traps.c index 6d4e76a4267f..c6f4932073a1 100644 --- a/arch/ia64/kernel/traps.c +++ b/arch/ia64/kernel/traps.c @@ -104,6 +104,7 @@ __kprobes ia64_bad_break (unsigned long break_num, struct pt_regs *regs) int sig, code; /* SIGILL, SIGFPE, SIGSEGV, and SIGBUS want these field initialized: */ + clear_siginfo(&siginfo); siginfo.si_addr = (void __user *) (regs->cr_iip + ia64_psr(regs)->ri); siginfo.si_imm = break_num; siginfo.si_flags = 0; /* clear __ISR_VALID */ @@ -293,7 +294,6 @@ handle_fpu_swa (int fp_fault, struct pt_regs *regs, unsigned long isr) { long exception, bundle[2]; unsigned long fault_ip; - struct siginfo siginfo; fault_ip = regs->cr_iip; if (!fp_fault && (ia64_psr(regs)->ri == 0)) @@ -344,13 +344,16 @@ handle_fpu_swa (int fp_fault, struct pt_regs *regs, unsigned long isr) printk(KERN_ERR "handle_fpu_swa: fp_emulate() returned -1\n"); return -1; } else { + struct siginfo siginfo; + /* is next instruction a trap? */ if (exception & 2) { ia64_increment_ip(regs); } + clear_siginfo(&siginfo); siginfo.si_signo = SIGFPE; siginfo.si_errno = 0; - siginfo.si_code = FPE_FIXME; /* default code */ + siginfo.si_code = FPE_FLTUNK; /* default code */ siginfo.si_addr = (void __user *) (regs->cr_iip + ia64_psr(regs)->ri); if (isr & 0x11) { siginfo.si_code = FPE_FLTINV; @@ -372,9 +375,12 @@ handle_fpu_swa (int fp_fault, struct pt_regs *regs, unsigned long isr) return -1; } else if (exception != 0) { /* raise exception */ + struct siginfo siginfo; + + clear_siginfo(&siginfo); siginfo.si_signo = SIGFPE; siginfo.si_errno = 0; - siginfo.si_code = FPE_FIXME; /* default code */ + siginfo.si_code = FPE_FLTUNK; /* default code */ siginfo.si_addr = (void __user *) (regs->cr_iip + ia64_psr(regs)->ri); if (isr & 0x880) { siginfo.si_code = FPE_FLTOVF; @@ -420,7 +426,7 @@ ia64_illegal_op_fault (unsigned long ec, long arg1, long arg2, long arg3, if (die_if_kernel(buf, ®s, 0)) return rv; - memset(&si, 0, sizeof(si)); + clear_siginfo(&si); si.si_signo = SIGILL; si.si_code = ILL_ILLOPC; si.si_addr = (void __user *) (regs.cr_iip + ia64_psr(®s)->ri); @@ -434,7 +440,6 @@ ia64_fault (unsigned long vector, unsigned long isr, unsigned long ifa, long arg7, struct pt_regs regs) { unsigned long code, error = isr, iip; - struct siginfo siginfo; char buf[128]; int result, sig; static const char *reason[] = { @@ -485,6 +490,7 @@ ia64_fault (unsigned long vector, unsigned long isr, unsigned long ifa, case 26: /* NaT Consumption */ if (user_mode(®s)) { + struct siginfo siginfo; void __user *addr; if (((isr >> 4) & 0xf) == 2) { @@ -499,6 +505,7 @@ ia64_fault (unsigned long vector, unsigned long isr, unsigned long ifa, addr = (void __user *) (regs.cr_iip + ia64_psr(®s)->ri); } + clear_siginfo(&siginfo); siginfo.si_signo = sig; siginfo.si_code = code; siginfo.si_errno = 0; @@ -515,6 +522,9 @@ ia64_fault (unsigned long vector, unsigned long isr, unsigned long ifa, case 31: /* Unsupported Data Reference */ if (user_mode(®s)) { + struct siginfo siginfo; + + clear_siginfo(&siginfo); siginfo.si_signo = SIGILL; siginfo.si_code = ILL_ILLOPN; siginfo.si_errno = 0; @@ -531,6 +541,10 @@ ia64_fault (unsigned long vector, unsigned long isr, unsigned long ifa, case 29: /* Debug */ case 35: /* Taken Branch Trap */ case 36: /* Single Step Trap */ + { + struct siginfo siginfo; + + clear_siginfo(&siginfo); if (fsys_mode(current, ®s)) { extern char __kernel_syscall_via_break[]; /* @@ -578,11 +592,15 @@ ia64_fault (unsigned long vector, unsigned long isr, unsigned long ifa, siginfo.si_isr = isr; force_sig_info(SIGTRAP, &siginfo, current); return; + } case 32: /* fp fault */ case 33: /* fp trap */ result = handle_fpu_swa((vector == 32) ? 1 : 0, ®s, isr); if ((result < 0) || (current->thread.flags & IA64_THREAD_FPEMU_SIGFPE)) { + struct siginfo siginfo; + + clear_siginfo(&siginfo); siginfo.si_signo = SIGFPE; siginfo.si_errno = 0; siginfo.si_code = FPE_FLTINV; @@ -616,6 +634,9 @@ ia64_fault (unsigned long vector, unsigned long isr, unsigned long ifa, } else { /* Unimplemented Instr. Address Trap */ if (user_mode(®s)) { + struct siginfo siginfo; + + clear_siginfo(&siginfo); siginfo.si_signo = SIGILL; siginfo.si_code = ILL_BADIADDR; siginfo.si_errno = 0; diff --git a/arch/ia64/kernel/unaligned.c b/arch/ia64/kernel/unaligned.c index 72e9b4242564..e309f9859acc 100644 --- a/arch/ia64/kernel/unaligned.c +++ b/arch/ia64/kernel/unaligned.c @@ -1537,6 +1537,7 @@ ia64_handle_unaligned (unsigned long ifa, struct pt_regs *regs) /* NOT_REACHED */ } force_sigbus: + clear_siginfo(&si); si.si_signo = SIGBUS; si.si_errno = 0; si.si_code = BUS_ADRALN; diff --git a/arch/ia64/mm/fault.c b/arch/ia64/mm/fault.c index dfdc152d6737..817fa120645f 100644 --- a/arch/ia64/mm/fault.c +++ b/arch/ia64/mm/fault.c @@ -85,7 +85,6 @@ ia64_do_page_fault (unsigned long address, unsigned long isr, struct pt_regs *re int signal = SIGSEGV, code = SEGV_MAPERR; struct vm_area_struct *vma, *prev_vma; struct mm_struct *mm = current->mm; - struct siginfo si; unsigned long mask; int fault; unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; @@ -249,6 +248,9 @@ retry: return; } if (user_mode(regs)) { + struct siginfo si; + + clear_siginfo(&si); si.si_signo = signal; si.si_errno = 0; si.si_code = code; diff --git a/arch/ia64/mm/tlb.c b/arch/ia64/mm/tlb.c index 46ecc5d948aa..acf10eb9da15 100644 --- a/arch/ia64/mm/tlb.c +++ b/arch/ia64/mm/tlb.c @@ -430,8 +430,9 @@ int ia64_itr_entry(u64 target_mask, u64 va, u64 pte, u64 log_size) int cpu = smp_processor_id(); if (!ia64_idtrs[cpu]) { - ia64_idtrs[cpu] = kmalloc(2 * IA64_TR_ALLOC_MAX * - sizeof (struct ia64_tr_entry), GFP_KERNEL); + ia64_idtrs[cpu] = kmalloc_array(2 * IA64_TR_ALLOC_MAX, + sizeof(struct ia64_tr_entry), + GFP_KERNEL); if (!ia64_idtrs[cpu]) return -ENOMEM; } diff --git a/arch/ia64/sn/kernel/io_common.c b/arch/ia64/sn/kernel/io_common.c index 11f2275570fb..102aabad6d20 100644 --- a/arch/ia64/sn/kernel/io_common.c +++ b/arch/ia64/sn/kernel/io_common.c @@ -132,7 +132,7 @@ static s64 sn_device_fixup_war(u64 nasid, u64 widget, int device, printk_once(KERN_WARNING "PROM version < 4.50 -- implementing old PROM flush WAR\n"); - war_list = kzalloc(DEV_PER_WIDGET * sizeof(*war_list), GFP_KERNEL); + war_list = kcalloc(DEV_PER_WIDGET, sizeof(*war_list), GFP_KERNEL); BUG_ON(!war_list); SAL_CALL_NOLOCK(isrv, SN_SAL_IOIF_GET_WIDGET_DMAFLUSH_LIST, @@ -480,11 +480,6 @@ sn_io_early_init(void) tioca_init_provider(); tioce_init_provider(); - /* - * This is needed to avoid bounce limit checks in the blk layer - */ - ia64_max_iommu_merge_mask = ~PAGE_MASK; - sn_irq_lh_init(); INIT_LIST_HEAD(&sn_sysdata_list); sn_init_cpei_timer(); diff --git a/arch/ia64/sn/kernel/irq.c b/arch/ia64/sn/kernel/irq.c index 85d095154902..d9b576df4f82 100644 --- a/arch/ia64/sn/kernel/irq.c +++ b/arch/ia64/sn/kernel/irq.c @@ -474,7 +474,8 @@ void __init sn_irq_lh_init(void) { int i; - sn_irq_lh = kmalloc(sizeof(struct list_head *) * NR_IRQS, GFP_KERNEL); + sn_irq_lh = kmalloc_array(NR_IRQS, sizeof(struct list_head *), + GFP_KERNEL); if (!sn_irq_lh) panic("SN PCI INIT: Failed to allocate memory for PCI init\n"); diff --git a/arch/ia64/sn/kernel/sn2/prominfo_proc.c b/arch/ia64/sn/kernel/sn2/prominfo_proc.c index ec4de2b09653..e15457bf21ac 100644 --- a/arch/ia64/sn/kernel/sn2/prominfo_proc.c +++ b/arch/ia64/sn/kernel/sn2/prominfo_proc.c @@ -140,18 +140,6 @@ static int proc_fit_show(struct seq_file *m, void *v) return 0; } -static int proc_fit_open(struct inode *inode, struct file *file) -{ - return single_open(file, proc_fit_show, PDE_DATA(inode)); -} - -static const struct file_operations proc_fit_fops = { - .open = proc_fit_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int proc_version_show(struct seq_file *m, void *v) { unsigned long nasid = (unsigned long)m->private; @@ -174,18 +162,6 @@ static int proc_version_show(struct seq_file *m, void *v) return 0; } -static int proc_version_open(struct inode *inode, struct file *file) -{ - return single_open(file, proc_version_show, PDE_DATA(inode)); -} - -static const struct file_operations proc_version_fops = { - .open = proc_version_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - /* module entry points */ int __init prominfo_init(void); void __exit prominfo_exit(void); @@ -217,10 +193,10 @@ int __init prominfo_init(void) if (!dir) continue; nasid = cnodeid_to_nasid(cnodeid); - proc_create_data("fit", 0, dir, - &proc_fit_fops, (void *)nasid); - proc_create_data("version", 0, dir, - &proc_version_fops, (void *)nasid); + proc_create_single_data("fit", 0, dir, proc_fit_show, + (void *)nasid); + proc_create_single_data("version", 0, dir, proc_version_show, + (void *)nasid); } return 0; } diff --git a/arch/ia64/sn/kernel/sn2/sn_proc_fs.c b/arch/ia64/sn/kernel/sn2/sn_proc_fs.c index 29cf8f8c08e9..c2a4d84297b0 100644 --- a/arch/ia64/sn/kernel/sn2/sn_proc_fs.c +++ b/arch/ia64/sn/kernel/sn2/sn_proc_fs.c @@ -18,33 +18,18 @@ static int partition_id_show(struct seq_file *s, void *p) return 0; } -static int partition_id_open(struct inode *inode, struct file *file) -{ - return single_open(file, partition_id_show, NULL); -} - static int system_serial_number_show(struct seq_file *s, void *p) { seq_printf(s, "%s\n", sn_system_serial_number()); return 0; } -static int system_serial_number_open(struct inode *inode, struct file *file) -{ - return single_open(file, system_serial_number_show, NULL); -} - static int licenseID_show(struct seq_file *s, void *p) { seq_printf(s, "0x%llx\n", sn_partition_serial_number_val()); return 0; } -static int licenseID_open(struct inode *inode, struct file *file) -{ - return single_open(file, licenseID_show, NULL); -} - static int coherence_id_show(struct seq_file *s, void *p) { seq_printf(s, "%d\n", partition_coherence_id()); @@ -52,43 +37,10 @@ static int coherence_id_show(struct seq_file *s, void *p) return 0; } -static int coherence_id_open(struct inode *inode, struct file *file) -{ - return single_open(file, coherence_id_show, NULL); -} - /* /proc/sgi_sn/sn_topology uses seq_file, see sn_hwperf.c */ extern int sn_topology_open(struct inode *, struct file *); extern int sn_topology_release(struct inode *, struct file *); -static const struct file_operations proc_partition_id_fops = { - .open = partition_id_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - -static const struct file_operations proc_system_sn_fops = { - .open = system_serial_number_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - -static const struct file_operations proc_license_id_fops = { - .open = licenseID_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - -static const struct file_operations proc_coherence_id_fops = { - .open = coherence_id_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static const struct file_operations proc_sn_topo_fops = { .open = sn_topology_open, .read = seq_read, @@ -104,13 +56,13 @@ void register_sn_procfs(void) if (!(sgi_proc_dir = proc_mkdir("sgi_sn", NULL))) return; - proc_create("partition_id", 0444, sgi_proc_dir, - &proc_partition_id_fops); - proc_create("system_serial_number", 0444, sgi_proc_dir, - &proc_system_sn_fops); - proc_create("licenseID", 0444, sgi_proc_dir, &proc_license_id_fops); - proc_create("coherence_id", 0444, sgi_proc_dir, - &proc_coherence_id_fops); + proc_create_single("partition_id", 0444, sgi_proc_dir, + partition_id_show); + proc_create_single("system_serial_number", 0444, sgi_proc_dir, + system_serial_number_show); + proc_create_single("licenseID", 0444, sgi_proc_dir, licenseID_show); + proc_create_single("coherence_id", 0444, sgi_proc_dir, + coherence_id_show); proc_create("sn_topology", 0444, sgi_proc_dir, &proc_sn_topo_fops); } diff --git a/arch/ia64/sn/pci/pcibr/pcibr_provider.c b/arch/ia64/sn/pci/pcibr/pcibr_provider.c index 8dbbef4a4f47..7195df1da121 100644 --- a/arch/ia64/sn/pci/pcibr/pcibr_provider.c +++ b/arch/ia64/sn/pci/pcibr/pcibr_provider.c @@ -184,7 +184,7 @@ pcibr_bus_fixup(struct pcibus_bussoft *prom_bussoft, struct pci_controller *cont /* Setup the PMU ATE map */ soft->pbi_int_ate_resource.lowest_free_index = 0; soft->pbi_int_ate_resource.ate = - kzalloc(soft->pbi_int_ate_size * sizeof(u64), GFP_KERNEL); + kcalloc(soft->pbi_int_ate_size, sizeof(u64), GFP_KERNEL); if (!soft->pbi_int_ate_resource.ate) { kfree(soft); diff --git a/arch/m68k/68000/timers.c b/arch/m68k/68000/timers.c index 252455bce144..71ddb4c98726 100644 --- a/arch/m68k/68000/timers.c +++ b/arch/m68k/68000/timers.c @@ -125,7 +125,9 @@ int m68328_hwclk(int set, struct rtc_time *t) { if (!set) { long now = RTCTIME; - t->tm_year = t->tm_mon = t->tm_mday = 1; + t->tm_year = 1; + t->tm_mon = 0; + t->tm_mday = 1; t->tm_hour = (now >> 24) % 24; t->tm_min = (now >> 16) % 60; t->tm_sec = now % 60; diff --git a/arch/m68k/Kconfig.bus b/arch/m68k/Kconfig.bus index d5e66ec136db..aef698fa50e5 100644 --- a/arch/m68k/Kconfig.bus +++ b/arch/m68k/Kconfig.bus @@ -59,6 +59,10 @@ config ATARI_ROM_ISA config GENERIC_ISA_DMA def_bool ISA +source "drivers/zorro/Kconfig" + +endif + config PCI bool "PCI support" depends on M54xx @@ -66,10 +70,8 @@ config PCI Enable the PCI bus. Support for the PCI bus hardware built into the ColdFire 547x and 548x processors. +if PCI source "drivers/pci/Kconfig" - -source "drivers/zorro/Kconfig" - endif if !MMU diff --git a/arch/m68k/apollo/config.c b/arch/m68k/apollo/config.c index 0d27706f14d4..b2a6bc63f8cd 100644 --- a/arch/m68k/apollo/config.c +++ b/arch/m68k/apollo/config.c @@ -221,8 +221,10 @@ int dn_dummy_hwclk(int op, struct rtc_time *t) { t->tm_hour=rtc->hours; t->tm_mday=rtc->day_of_month; t->tm_wday=rtc->day_of_week; - t->tm_mon=rtc->month; + t->tm_mon = rtc->month - 1; t->tm_year=rtc->year; + if (t->tm_year < 70) + t->tm_year += 100; } else { rtc->second=t->tm_sec; rtc->minute=t->tm_min; @@ -230,8 +232,8 @@ int dn_dummy_hwclk(int op, struct rtc_time *t) { rtc->day_of_month=t->tm_mday; if(t->tm_wday!=-1) rtc->day_of_week=t->tm_wday; - rtc->month=t->tm_mon; - rtc->year=t->tm_year; + rtc->month = t->tm_mon + 1; + rtc->year = t->tm_year % 100; } return 0; diff --git a/arch/m68k/coldfire/pci.c b/arch/m68k/coldfire/pci.c index 3097fa2ca746..62b0eb6cf69a 100644 --- a/arch/m68k/coldfire/pci.c +++ b/arch/m68k/coldfire/pci.c @@ -23,20 +23,10 @@ /* * Memory and IO mappings. We use a 1:1 mapping for local host memory to - * PCI bus memory (no reason not to really). IO space doesn't matter, we - * always use access functions for that. The device configuration space is - * mapped over the IO map space when we enable it in the PCICAR register. + * PCI bus memory (no reason not to really). IO space is mapped in its own + * separate address region. The device configuration space is mapped over + * the IO map space when we enable it in the PCICAR register. */ -#define PCI_MEM_PA 0xf0000000 /* Host physical address */ -#define PCI_MEM_BA 0xf0000000 /* Bus physical address */ -#define PCI_MEM_SIZE 0x08000000 /* 128 MB */ -#define PCI_MEM_MASK (PCI_MEM_SIZE - 1) - -#define PCI_IO_PA 0xf8000000 /* Host physical address */ -#define PCI_IO_BA 0x00000000 /* Bus physical address */ -#define PCI_IO_SIZE 0x00010000 /* 64k */ -#define PCI_IO_MASK (PCI_IO_SIZE - 1) - static struct pci_bus *rootbus; static unsigned long iospace; @@ -56,13 +46,6 @@ static unsigned char mcf_host_irq[] = { 0, 69, 69, 71, 71, }; - -static inline void syncio(void) -{ - /* The ColdFire "nop" instruction waits for all bus IO to complete */ - __asm__ __volatile__ ("nop"); -} - /* * Configuration space access functions. Configuration space access is * through the IO mapping window, enabling it via the PCICAR register. @@ -84,9 +67,9 @@ static int mcf_pci_readconfig(struct pci_bus *bus, unsigned int devfn, return PCIBIOS_SUCCESSFUL; } - syncio(); addr = mcf_mk_pcicar(bus->number, devfn, where); __raw_writel(PCICAR_E | addr, PCICAR); + __raw_readl(PCICAR); addr = iospace + (where & 0x3); switch (size) { @@ -101,8 +84,8 @@ static int mcf_pci_readconfig(struct pci_bus *bus, unsigned int devfn, break; } - syncio(); __raw_writel(0, PCICAR); + __raw_readl(PCICAR); return PCIBIOS_SUCCESSFUL; } @@ -116,9 +99,9 @@ static int mcf_pci_writeconfig(struct pci_bus *bus, unsigned int devfn, return PCIBIOS_SUCCESSFUL; } - syncio(); addr = mcf_mk_pcicar(bus->number, devfn, where); __raw_writel(PCICAR_E | addr, PCICAR); + __raw_readl(PCICAR); addr = iospace + (where & 0x3); switch (size) { @@ -133,8 +116,8 @@ static int mcf_pci_writeconfig(struct pci_bus *bus, unsigned int devfn, break; } - syncio(); __raw_writel(0, PCICAR); + __raw_readl(PCICAR); return PCIBIOS_SUCCESSFUL; } @@ -143,89 +126,6 @@ static struct pci_ops mcf_pci_ops = { .write = mcf_pci_writeconfig, }; -/* - * IO address space access functions. Pretty strait forward, these are - * directly mapped in to the IO mapping window. And that is mapped into - * virtual address space. - */ -u8 mcf_pci_inb(u32 addr) -{ - return __raw_readb(iospace + (addr & PCI_IO_MASK)); -} -EXPORT_SYMBOL(mcf_pci_inb); - -u16 mcf_pci_inw(u32 addr) -{ - return le16_to_cpu(__raw_readw(iospace + (addr & PCI_IO_MASK))); -} -EXPORT_SYMBOL(mcf_pci_inw); - -u32 mcf_pci_inl(u32 addr) -{ - return le32_to_cpu(__raw_readl(iospace + (addr & PCI_IO_MASK))); -} -EXPORT_SYMBOL(mcf_pci_inl); - -void mcf_pci_insb(u32 addr, u8 *buf, u32 len) -{ - for (; len; len--) - *buf++ = mcf_pci_inb(addr); -} -EXPORT_SYMBOL(mcf_pci_insb); - -void mcf_pci_insw(u32 addr, u16 *buf, u32 len) -{ - for (; len; len--) - *buf++ = mcf_pci_inw(addr); -} -EXPORT_SYMBOL(mcf_pci_insw); - -void mcf_pci_insl(u32 addr, u32 *buf, u32 len) -{ - for (; len; len--) - *buf++ = mcf_pci_inl(addr); -} -EXPORT_SYMBOL(mcf_pci_insl); - -void mcf_pci_outb(u8 v, u32 addr) -{ - __raw_writeb(v, iospace + (addr & PCI_IO_MASK)); -} -EXPORT_SYMBOL(mcf_pci_outb); - -void mcf_pci_outw(u16 v, u32 addr) -{ - __raw_writew(cpu_to_le16(v), iospace + (addr & PCI_IO_MASK)); -} -EXPORT_SYMBOL(mcf_pci_outw); - -void mcf_pci_outl(u32 v, u32 addr) -{ - __raw_writel(cpu_to_le32(v), iospace + (addr & PCI_IO_MASK)); -} -EXPORT_SYMBOL(mcf_pci_outl); - -void mcf_pci_outsb(u32 addr, const u8 *buf, u32 len) -{ - for (; len; len--) - mcf_pci_outb(*buf++, addr); -} -EXPORT_SYMBOL(mcf_pci_outsb); - -void mcf_pci_outsw(u32 addr, const u16 *buf, u32 len) -{ - for (; len; len--) - mcf_pci_outw(*buf++, addr); -} -EXPORT_SYMBOL(mcf_pci_outsw); - -void mcf_pci_outsl(u32 addr, const u32 *buf, u32 len) -{ - for (; len; len--) - mcf_pci_outl(*buf++, addr); -} -EXPORT_SYMBOL(mcf_pci_outsl); - /* * Initialize the PCI bus registers, and scan the bus. */ diff --git a/arch/m68k/configs/amiga_defconfig b/arch/m68k/configs/amiga_defconfig index 37a8e5ab8728..a874e54404d1 100644 --- a/arch/m68k/configs/amiga_defconfig +++ b/arch/m68k/configs/amiga_defconfig @@ -98,8 +98,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -204,7 +204,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -233,12 +233,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -259,7 +259,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -310,7 +310,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -414,6 +413,7 @@ CONFIG_ARIADNE=y # CONFIG_NET_VENDOR_MARVELL is not set # CONFIG_NET_VENDOR_MICREL is not set # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set CONFIG_HYDRA=y CONFIG_APNE=y CONFIG_ZORRO8390=y @@ -485,6 +485,7 @@ CONFIG_RTC_DRV_MSM6242=m CONFIG_RTC_DRV_RP5C01=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_HEARTBEAT=y CONFIG_PROC_HARDWARE=y CONFIG_AMIGA_BUILTIN_SERIAL=y @@ -621,6 +622,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -647,6 +649,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/configs/apollo_defconfig b/arch/m68k/configs/apollo_defconfig index 6a466266b852..8ce39e23aa42 100644 --- a/arch/m68k/configs/apollo_defconfig +++ b/arch/m68k/configs/apollo_defconfig @@ -96,8 +96,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -202,7 +202,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -231,12 +231,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -257,7 +257,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -308,7 +308,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -392,6 +391,7 @@ CONFIG_VETH=m # CONFIG_NET_VENDOR_MICREL is not set # CONFIG_NET_VENDOR_NATSEMI is not set # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set # CONFIG_NET_VENDOR_QUALCOMM is not set # CONFIG_NET_VENDOR_RENESAS is not set # CONFIG_NET_VENDOR_ROCKER is not set @@ -446,6 +446,7 @@ CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_GENERIC=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_HEARTBEAT=y CONFIG_PROC_HARDWARE=y CONFIG_EXT4_FS=y @@ -580,6 +581,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -606,6 +608,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/configs/atari_defconfig b/arch/m68k/configs/atari_defconfig index b0691a7a3345..346c4e75edf8 100644 --- a/arch/m68k/configs/atari_defconfig +++ b/arch/m68k/configs/atari_defconfig @@ -96,8 +96,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -202,7 +202,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -231,12 +231,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -257,7 +257,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -308,7 +308,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -401,6 +400,7 @@ CONFIG_ATARILANCE=y # CONFIG_NET_VENDOR_MARVELL is not set # CONFIG_NET_VENDOR_MICREL is not set # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set CONFIG_NE2000=y # CONFIG_NET_VENDOR_QUALCOMM is not set # CONFIG_NET_VENDOR_RENESAS is not set @@ -461,6 +461,7 @@ CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_GENERIC=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_HEARTBEAT=y CONFIG_PROC_HARDWARE=y CONFIG_NATFEAT=y @@ -602,6 +603,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -628,6 +630,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/configs/bvme6000_defconfig b/arch/m68k/configs/bvme6000_defconfig index 6f6470fa9a50..fca9c7aa71a3 100644 --- a/arch/m68k/configs/bvme6000_defconfig +++ b/arch/m68k/configs/bvme6000_defconfig @@ -94,8 +94,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -200,7 +200,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -229,12 +229,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -255,7 +255,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -306,7 +306,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -391,6 +390,7 @@ CONFIG_BVME6000_NET=y # CONFIG_NET_VENDOR_MICREL is not set # CONFIG_NET_VENDOR_NATSEMI is not set # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set # CONFIG_NET_VENDOR_QUALCOMM is not set # CONFIG_NET_VENDOR_RENESAS is not set # CONFIG_NET_VENDOR_ROCKER is not set @@ -439,6 +439,7 @@ CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_GENERIC=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_PROC_HARDWARE=y CONFIG_EXT4_FS=y CONFIG_REISERFS_FS=m @@ -572,6 +573,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -598,6 +600,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/configs/hp300_defconfig b/arch/m68k/configs/hp300_defconfig index 31a1a2b5e860..f9eab174915c 100644 --- a/arch/m68k/configs/hp300_defconfig +++ b/arch/m68k/configs/hp300_defconfig @@ -96,8 +96,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -202,7 +202,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -231,12 +231,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -257,7 +257,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -308,7 +308,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -393,6 +392,7 @@ CONFIG_HPLANCE=y # CONFIG_NET_VENDOR_MICREL is not set # CONFIG_NET_VENDOR_NATSEMI is not set # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set # CONFIG_NET_VENDOR_QUALCOMM is not set # CONFIG_NET_VENDOR_RENESAS is not set # CONFIG_NET_VENDOR_ROCKER is not set @@ -449,6 +449,7 @@ CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_GENERIC=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_PROC_HARDWARE=y CONFIG_EXT4_FS=y CONFIG_REISERFS_FS=m @@ -582,6 +583,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -608,6 +610,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/configs/mac_defconfig b/arch/m68k/configs/mac_defconfig index 390d4a87441c..b52e597899eb 100644 --- a/arch/m68k/configs/mac_defconfig +++ b/arch/m68k/configs/mac_defconfig @@ -95,8 +95,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -201,7 +201,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -230,12 +230,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -256,7 +256,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -310,7 +310,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -410,6 +409,7 @@ CONFIG_MAC89x0=y # CONFIG_NET_VENDOR_MICREL is not set CONFIG_MACSONIC=y # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set CONFIG_MAC8390=y # CONFIG_NET_VENDOR_QUALCOMM is not set # CONFIG_NET_VENDOR_RENESAS is not set @@ -471,6 +471,7 @@ CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_GENERIC=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_PROC_HARDWARE=y CONFIG_EXT4_FS=y CONFIG_REISERFS_FS=m @@ -604,6 +605,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -630,6 +632,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/configs/multi_defconfig b/arch/m68k/configs/multi_defconfig index 77be97d82dc3..2a84eeec5b02 100644 --- a/arch/m68k/configs/multi_defconfig +++ b/arch/m68k/configs/multi_defconfig @@ -105,8 +105,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -211,7 +211,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -240,12 +240,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -266,7 +266,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -320,7 +320,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -452,6 +451,7 @@ CONFIG_MVME16x_NET=y # CONFIG_NET_VENDOR_MICREL is not set CONFIG_MACSONIC=y # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set CONFIG_HYDRA=y CONFIG_MAC8390=y CONFIG_NE2000=y @@ -541,6 +541,7 @@ CONFIG_RTC_DRV_RP5C01=m CONFIG_RTC_DRV_GENERIC=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_HEARTBEAT=y CONFIG_PROC_HARDWARE=y CONFIG_NATFEAT=y @@ -684,6 +685,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -710,6 +712,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/configs/mvme147_defconfig b/arch/m68k/configs/mvme147_defconfig index 2ca140757b0f..476e69994340 100644 --- a/arch/m68k/configs/mvme147_defconfig +++ b/arch/m68k/configs/mvme147_defconfig @@ -93,8 +93,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -199,7 +199,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -228,12 +228,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -254,7 +254,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -305,7 +305,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -391,6 +390,7 @@ CONFIG_MVME147_NET=y # CONFIG_NET_VENDOR_MICREL is not set # CONFIG_NET_VENDOR_NATSEMI is not set # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set # CONFIG_NET_VENDOR_QUALCOMM is not set # CONFIG_NET_VENDOR_RENESAS is not set # CONFIG_NET_VENDOR_ROCKER is not set @@ -439,6 +439,7 @@ CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_GENERIC=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_PROC_HARDWARE=y CONFIG_EXT4_FS=y CONFIG_REISERFS_FS=m @@ -572,6 +573,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -598,6 +600,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/configs/mvme16x_defconfig b/arch/m68k/configs/mvme16x_defconfig index 6a3b4dcc5aab..1477cda9146e 100644 --- a/arch/m68k/configs/mvme16x_defconfig +++ b/arch/m68k/configs/mvme16x_defconfig @@ -94,8 +94,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -200,7 +200,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -229,12 +229,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -255,7 +255,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -306,7 +306,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -391,6 +390,7 @@ CONFIG_MVME16x_NET=y # CONFIG_NET_VENDOR_MICREL is not set # CONFIG_NET_VENDOR_NATSEMI is not set # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set # CONFIG_NET_VENDOR_QUALCOMM is not set # CONFIG_NET_VENDOR_RENESAS is not set # CONFIG_NET_VENDOR_ROCKER is not set @@ -439,6 +439,7 @@ CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_GENERIC=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_PROC_HARDWARE=y CONFIG_EXT4_FS=y CONFIG_REISERFS_FS=m @@ -572,6 +573,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -598,6 +600,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/configs/q40_defconfig b/arch/m68k/configs/q40_defconfig index 2a3e29c97652..b3a543dc48a0 100644 --- a/arch/m68k/configs/q40_defconfig +++ b/arch/m68k/configs/q40_defconfig @@ -94,8 +94,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -200,7 +200,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -229,12 +229,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -255,7 +255,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -306,7 +306,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -400,6 +399,7 @@ CONFIG_VETH=m # CONFIG_NET_VENDOR_MARVELL is not set # CONFIG_NET_VENDOR_MICREL is not set # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set CONFIG_NE2000=y # CONFIG_NET_VENDOR_QUALCOMM is not set # CONFIG_NET_VENDOR_RENESAS is not set @@ -461,6 +461,7 @@ CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_GENERIC=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_HEARTBEAT=y CONFIG_PROC_HARDWARE=y CONFIG_EXT4_FS=y @@ -595,6 +596,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -621,6 +623,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/configs/sun3_defconfig b/arch/m68k/configs/sun3_defconfig index cba2494c99b2..d543ed5dfa96 100644 --- a/arch/m68k/configs/sun3_defconfig +++ b/arch/m68k/configs/sun3_defconfig @@ -91,8 +91,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -197,7 +197,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -226,12 +226,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -252,7 +252,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -303,7 +303,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -388,6 +387,7 @@ CONFIG_SUN3_82586=y # CONFIG_NET_VENDOR_MICREL is not set # CONFIG_NET_VENDOR_NATSEMI is not set # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set # CONFIG_NET_VENDOR_QUALCOMM is not set # CONFIG_NET_VENDOR_RENESAS is not set # CONFIG_NET_VENDOR_ROCKER is not set @@ -441,6 +441,7 @@ CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_GENERIC=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_PROC_HARDWARE=y CONFIG_EXT4_FS=y CONFIG_REISERFS_FS=m @@ -573,6 +574,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -599,6 +601,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/configs/sun3x_defconfig b/arch/m68k/configs/sun3x_defconfig index d911561137fd..a67e54246023 100644 --- a/arch/m68k/configs/sun3x_defconfig +++ b/arch/m68k/configs/sun3x_defconfig @@ -91,8 +91,8 @@ CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NF_TABLES=m -CONFIG_NF_TABLES_INET=m -CONFIG_NF_TABLES_NETDEV=m +CONFIG_NF_TABLES_INET=y +CONFIG_NF_TABLES_NETDEV=y CONFIG_NFT_EXTHDR=m CONFIG_NFT_META=m CONFIG_NFT_RT=m @@ -197,7 +197,7 @@ CONFIG_NF_SOCKET_IPV4=m CONFIG_NFT_CHAIN_ROUTE_IPV4=m CONFIG_NFT_DUP_IPV4=m CONFIG_NFT_FIB_IPV4=m -CONFIG_NF_TABLES_ARP=m +CONFIG_NF_TABLES_ARP=y CONFIG_NF_FLOW_TABLE_IPV4=m CONFIG_NF_LOG_ARP=m CONFIG_NFT_CHAIN_NAT_IPV4=m @@ -226,12 +226,12 @@ CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_NF_SOCKET_IPV6=m CONFIG_NFT_CHAIN_ROUTE_IPV6=m -CONFIG_NFT_DUP_IPV6=m -CONFIG_NFT_FIB_IPV6=m -CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_NFT_CHAIN_NAT_IPV6=m CONFIG_NFT_MASQ_IPV6=m CONFIG_NFT_REDIR_IPV6=m +CONFIG_NFT_DUP_IPV6=m +CONFIG_NFT_FIB_IPV6=m +CONFIG_NF_FLOW_TABLE_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_IP6_NF_MATCH_AH=m CONFIG_IP6_NF_MATCH_EUI64=m @@ -252,7 +252,7 @@ CONFIG_IP6_NF_RAW=m CONFIG_IP6_NF_NAT=m CONFIG_IP6_NF_TARGET_MASQUERADE=m CONFIG_IP6_NF_TARGET_NPT=m -CONFIG_NF_TABLES_BRIDGE=m +CONFIG_NF_TABLES_BRIDGE=y CONFIG_NFT_BRIDGE_META=m CONFIG_NFT_BRIDGE_REJECT=m CONFIG_NF_LOG_BRIDGE=m @@ -303,7 +303,6 @@ CONFIG_NET_MPLS_GSO=m CONFIG_MPLS_ROUTING=m CONFIG_MPLS_IPTUNNEL=m CONFIG_NET_NSH=m -CONFIG_NET_L3_MASTER_DEV=y CONFIG_AF_KCM=m # CONFIG_WIRELESS is not set CONFIG_PSAMPLE=m @@ -389,6 +388,7 @@ CONFIG_SUN3LANCE=y # CONFIG_NET_VENDOR_MICREL is not set # CONFIG_NET_VENDOR_NATSEMI is not set # CONFIG_NET_VENDOR_NETRONOME is not set +# CONFIG_NET_VENDOR_NI is not set # CONFIG_NET_VENDOR_QUALCOMM is not set # CONFIG_NET_VENDOR_RENESAS is not set # CONFIG_NET_VENDOR_ROCKER is not set @@ -441,6 +441,7 @@ CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_GENERIC=m # CONFIG_VIRTIO_MENU is not set # CONFIG_IOMMU_SUPPORT is not set +CONFIG_DAX=m CONFIG_PROC_HARDWARE=y CONFIG_EXT4_FS=y CONFIG_REISERFS_FS=m @@ -574,6 +575,7 @@ CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_MCRYPTD=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CHACHA20POLY1305=m +CONFIG_CRYPTO_CFB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_KEYWRAP=m @@ -600,6 +602,8 @@ CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m +CONFIG_CRYPTO_SM4=m +CONFIG_CRYPTO_SPECK=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_LZO=m diff --git a/arch/m68k/include/asm/Kbuild b/arch/m68k/include/asm/Kbuild index 88a9d27df1ac..4d8d68c4e3dd 100644 --- a/arch/m68k/include/asm/Kbuild +++ b/arch/m68k/include/asm/Kbuild @@ -1,4 +1,5 @@ generic-y += barrier.h +generic-y += compat.h generic-y += device.h generic-y += emergency-restart.h generic-y += exec.h diff --git a/arch/m68k/include/asm/atarihw.h b/arch/m68k/include/asm/atarihw.h index 972c8f33f055..9000b249d225 100644 --- a/arch/m68k/include/asm/atarihw.h +++ b/arch/m68k/include/asm/atarihw.h @@ -23,6 +23,7 @@ #include #include #include +#include extern u_long atari_mch_cookie; extern u_long atari_mch_type; diff --git a/arch/m68k/include/asm/delay.h b/arch/m68k/include/asm/delay.h index 7f474121e4ca..751712f8beea 100644 --- a/arch/m68k/include/asm/delay.h +++ b/arch/m68k/include/asm/delay.h @@ -49,8 +49,6 @@ extern void __bad_udelay(void); * The simpler m68k and ColdFire processors do not have a 32*32->64 * multiply instruction. So we need to handle them a little differently. * We use a bit of shifting and a single 32*32->32 multiply to get close. - * This is a macro so that the const version can factor out the first - * multiply and shift. */ #define HZSCALE (268435456 / (1000000 / HZ)) @@ -115,6 +113,13 @@ static inline void __udelay(unsigned long usecs) */ #define HZSCALE (268435456 / (1000000 / HZ)) -#define ndelay(n) __delay(DIV_ROUND_UP((n) * ((((HZSCALE) >> 11) * (loops_per_jiffy >> 11)) >> 6), 1000)) +static inline void ndelay(unsigned long nsec) +{ + __delay(DIV_ROUND_UP(nsec * + ((((HZSCALE) >> 11) * + (loops_per_jiffy >> 11)) >> 6), + 1000)); +} +#define ndelay(n) ndelay(n) #endif /* defined(_M68K_DELAY_H) */ diff --git a/arch/m68k/include/asm/io.h b/arch/m68k/include/asm/io.h index 756089cc019c..ca2849afb087 100644 --- a/arch/m68k/include/asm/io.h +++ b/arch/m68k/include/asm/io.h @@ -1,14 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0 */ -#ifdef __uClinux__ +#if defined(__uClinux__) || defined(CONFIG_COLDFIRE) #include #else #include #endif - -#define readb_relaxed(addr) readb(addr) -#define readw_relaxed(addr) readw(addr) -#define readl_relaxed(addr) readl(addr) - -#define writeb_relaxed(b, addr) writeb(b, addr) -#define writew_relaxed(b, addr) writew(b, addr) -#define writel_relaxed(b, addr) writel(b, addr) diff --git a/arch/m68k/include/asm/io_mm.h b/arch/m68k/include/asm/io_mm.h index ed5333e87879..fe485f4f5fac 100644 --- a/arch/m68k/include/asm/io_mm.h +++ b/arch/m68k/include/asm/io_mm.h @@ -26,6 +26,7 @@ #include #include #include +#include #include @@ -85,53 +86,7 @@ #endif /* ATARI_ROM_ISA */ -#if defined(CONFIG_PCI) && defined(CONFIG_COLDFIRE) - -#define HAVE_ARCH_PIO_SIZE -#define PIO_OFFSET 0 -#define PIO_MASK 0xffff -#define PIO_RESERVED 0x10000 - -u8 mcf_pci_inb(u32 addr); -u16 mcf_pci_inw(u32 addr); -u32 mcf_pci_inl(u32 addr); -void mcf_pci_insb(u32 addr, u8 *buf, u32 len); -void mcf_pci_insw(u32 addr, u16 *buf, u32 len); -void mcf_pci_insl(u32 addr, u32 *buf, u32 len); - -void mcf_pci_outb(u8 v, u32 addr); -void mcf_pci_outw(u16 v, u32 addr); -void mcf_pci_outl(u32 v, u32 addr); -void mcf_pci_outsb(u32 addr, const u8 *buf, u32 len); -void mcf_pci_outsw(u32 addr, const u16 *buf, u32 len); -void mcf_pci_outsl(u32 addr, const u32 *buf, u32 len); - -#define inb mcf_pci_inb -#define inb_p mcf_pci_inb -#define inw mcf_pci_inw -#define inw_p mcf_pci_inw -#define inl mcf_pci_inl -#define inl_p mcf_pci_inl -#define insb mcf_pci_insb -#define insw mcf_pci_insw -#define insl mcf_pci_insl - -#define outb mcf_pci_outb -#define outb_p mcf_pci_outb -#define outw mcf_pci_outw -#define outw_p mcf_pci_outw -#define outl mcf_pci_outl -#define outl_p mcf_pci_outl -#define outsb mcf_pci_outsb -#define outsw mcf_pci_outsw -#define outsl mcf_pci_outsl - -#define readb(addr) in_8(addr) -#define writeb(v, addr) out_8((addr), (v)) -#define readw(addr) in_le16(addr) -#define writew(v, addr) out_le16((addr), (v)) - -#elif defined(CONFIG_ISA) || defined(CONFIG_ATARI_ROM_ISA) +#if defined(CONFIG_ISA) || defined(CONFIG_ATARI_ROM_ISA) #if MULTI_ISA == 0 #undef MULTI_ISA @@ -414,8 +369,7 @@ static inline void isa_delay(void) #define writew(val, addr) out_le16((addr), (val)) #endif /* CONFIG_ATARI_ROM_ISA */ -#if !defined(CONFIG_ISA) && !defined(CONFIG_ATARI_ROM_ISA) && \ - !(defined(CONFIG_PCI) && defined(CONFIG_COLDFIRE)) +#if !defined(CONFIG_ISA) && !defined(CONFIG_ATARI_ROM_ISA) /* * We need to define dummy functions for GENERIC_IOMAP support. */ @@ -461,39 +415,6 @@ static inline void isa_delay(void) #define mmiowb() -static inline void __iomem *ioremap(unsigned long physaddr, unsigned long size) -{ - return __ioremap(physaddr, size, IOMAP_NOCACHE_SER); -} -static inline void __iomem *ioremap_nocache(unsigned long physaddr, unsigned long size) -{ - return __ioremap(physaddr, size, IOMAP_NOCACHE_SER); -} -#define ioremap_uc ioremap_nocache -static inline void __iomem *ioremap_wt(unsigned long physaddr, - unsigned long size) -{ - return __ioremap(physaddr, size, IOMAP_WRITETHROUGH); -} -static inline void __iomem *ioremap_fullcache(unsigned long physaddr, - unsigned long size) -{ - return __ioremap(physaddr, size, IOMAP_FULL_CACHING); -} - -static inline void memset_io(volatile void __iomem *addr, unsigned char val, int count) -{ - __builtin_memset((void __force *) addr, val, count); -} -static inline void memcpy_fromio(void *dst, const volatile void __iomem *src, int count) -{ - __builtin_memcpy(dst, (void __force *) src, count); -} -static inline void memcpy_toio(volatile void __iomem *dst, const void *src, int count) -{ - __builtin_memcpy((void __force *) dst, src, count); -} - #ifndef CONFIG_SUN3 #define IO_SPACE_LIMIT 0xffff #else @@ -515,13 +436,12 @@ static inline void memcpy_toio(volatile void __iomem *dst, const void *src, int */ #define xlate_dev_kmem_ptr(p) p -static inline void __iomem *ioport_map(unsigned long port, unsigned int nr) -{ - return (void __iomem *) port; -} +#define readb_relaxed(addr) readb(addr) +#define readw_relaxed(addr) readw(addr) +#define readl_relaxed(addr) readl(addr) -static inline void ioport_unmap(void __iomem *p) -{ -} +#define writeb_relaxed(b, addr) writeb(b, addr) +#define writew_relaxed(b, addr) writew(b, addr) +#define writel_relaxed(b, addr) writel(b, addr) #endif /* _IO_H */ diff --git a/arch/m68k/include/asm/io_no.h b/arch/m68k/include/asm/io_no.h index 86f45b403bcc..83a0a6d449f4 100644 --- a/arch/m68k/include/asm/io_no.h +++ b/arch/m68k/include/asm/io_no.h @@ -2,191 +2,148 @@ #ifndef _M68KNOMMU_IO_H #define _M68KNOMMU_IO_H -#ifdef __KERNEL__ - -#define ARCH_HAS_IOREMAP_WT - -#include -#include - /* - * These are for ISA/PCI shared memory _only_ and should never be used - * on any other type of memory, including Zorro memory. They are meant to - * access the bus in the bus byte order which is little-endian!. - * - * readX/writeX() are used to access memory mapped devices. On some - * architectures the memory mapped IO stuff needs to be accessed - * differently. On the m68k architecture, we just read/write the - * memory location directly. - */ -/* ++roman: The assignments to temp. vars avoid that gcc sometimes generates - * two accesses to memory, which may be undesirable for some devices. + * Convert a physical memory address into a IO memory address. + * For us this is trivially a type cast. */ +#define iomem(a) ((void __iomem *) (a)) /* - * swap functions are sometimes needed to interface little-endian hardware + * The non-MMU m68k and ColdFire IO and memory mapped hardware access + * functions have always worked in CPU native endian. We need to define + * that behavior here first before we include asm-generic/io.h. */ -static inline unsigned short _swapw(volatile unsigned short v) -{ - return ((v << 8) | (v >> 8)); -} - -static inline unsigned int _swapl(volatile unsigned long v) -{ - return ((v << 24) | ((v & 0xff00) << 8) | ((v & 0xff0000) >> 8) | (v >> 24)); -} - -#define readb(addr) \ +#define __raw_readb(addr) \ ({ unsigned char __v = (*(volatile unsigned char *) (addr)); __v; }) -#define readw(addr) \ +#define __raw_readw(addr) \ ({ unsigned short __v = (*(volatile unsigned short *) (addr)); __v; }) -#define readl(addr) \ +#define __raw_readl(addr) \ ({ unsigned int __v = (*(volatile unsigned int *) (addr)); __v; }) -#define writeb(b,addr) (void)((*(volatile unsigned char *) (addr)) = (b)) -#define writew(b,addr) (void)((*(volatile unsigned short *) (addr)) = (b)) -#define writel(b,addr) (void)((*(volatile unsigned int *) (addr)) = (b)) +#define __raw_writeb(b, addr) (void)((*(volatile unsigned char *) (addr)) = (b)) +#define __raw_writew(b, addr) (void)((*(volatile unsigned short *) (addr)) = (b)) +#define __raw_writel(b, addr) (void)((*(volatile unsigned int *) (addr)) = (b)) -#define __raw_readb readb -#define __raw_readw readw -#define __raw_readl readl -#define __raw_writeb writeb -#define __raw_writew writew -#define __raw_writel writel +#if defined(CONFIG_COLDFIRE) +/* + * For ColdFire platforms we may need to do some extra checks for what + * type of address range we are accessing. Include the ColdFire platform + * definitions so we can figure out if need to do something special. + */ +#include +#include +#include +#endif /* CONFIG_COLDFIRE */ -static inline void io_outsb(unsigned int addr, const void *buf, int len) +#if defined(IOMEMBASE) +/* + * The ColdFire SoC internal peripherals are mapped into virtual address + * space using the ACR registers of the cache control unit. This means we + * are using a 1:1 physical:virtual mapping for them. We can quickly + * determine if we are accessing an internal peripheral device given the + * physical or vitrual address using the same range check. This check logic + * applies just the same of there is no MMU but something like a PCI bus + * is present. + */ +static int __cf_internalio(unsigned long addr) { - volatile unsigned char *ap = (volatile unsigned char *) addr; - unsigned char *bp = (unsigned char *) buf; - while (len--) - *ap = *bp++; + return (addr >= IOMEMBASE) && (addr <= IOMEMBASE + IOMEMSIZE - 1); } -static inline void io_outsw(unsigned int addr, const void *buf, int len) +static int cf_internalio(const volatile void __iomem *addr) { - volatile unsigned short *ap = (volatile unsigned short *) addr; - unsigned short *bp = (unsigned short *) buf; - while (len--) - *ap = _swapw(*bp++); + return __cf_internalio((unsigned long) addr); } -static inline void io_outsl(unsigned int addr, const void *buf, int len) +/* + * We need to treat built-in peripherals and bus based address ranges + * differently. Local built-in peripherals (and the ColdFire SoC parts + * have quite a lot of them) are always native endian - which is big + * endian on m68k/ColdFire. Bus based address ranges, like the PCI bus, + * are accessed little endian - so we need to byte swap those. + */ +#define readw readw +static inline u16 readw(const volatile void __iomem *addr) { - volatile unsigned int *ap = (volatile unsigned int *) addr; - unsigned int *bp = (unsigned int *) buf; - while (len--) - *ap = _swapl(*bp++); + if (cf_internalio(addr)) + return __raw_readw(addr); + return __le16_to_cpu(__raw_readw(addr)); } -static inline void io_insb(unsigned int addr, void *buf, int len) +#define readl readl +static inline u32 readl(const volatile void __iomem *addr) { - volatile unsigned char *ap = (volatile unsigned char *) addr; - unsigned char *bp = (unsigned char *) buf; - while (len--) - *bp++ = *ap; + if (cf_internalio(addr)) + return __raw_readl(addr); + return __le32_to_cpu(__raw_readl(addr)); } -static inline void io_insw(unsigned int addr, void *buf, int len) +#define writew writew +static inline void writew(u16 value, volatile void __iomem *addr) { - volatile unsigned short *ap = (volatile unsigned short *) addr; - unsigned short *bp = (unsigned short *) buf; - while (len--) - *bp++ = _swapw(*ap); + if (cf_internalio(addr)) + __raw_writew(value, addr); + else + __raw_writew(__cpu_to_le16(value), addr); } -static inline void io_insl(unsigned int addr, void *buf, int len) +#define writel writel +static inline void writel(u32 value, volatile void __iomem *addr) { - volatile unsigned int *ap = (volatile unsigned int *) addr; - unsigned int *bp = (unsigned int *) buf; - while (len--) - *bp++ = _swapl(*ap); + if (cf_internalio(addr)) + __raw_writel(value, addr); + else + __raw_writel(__cpu_to_le32(value), addr); } -#define mmiowb() - -/* - * make the short names macros so specific devices - * can override them as required - */ - -#define memset_io(a,b,c) memset((void *)(a),(b),(c)) -#define memcpy_fromio(a,b,c) memcpy((a),(void *)(b),(c)) -#define memcpy_toio(a,b,c) memcpy((void *)(a),(b),(c)) - -#define inb(addr) readb(addr) -#define inw(addr) readw(addr) -#define inl(addr) readl(addr) -#define outb(x,addr) ((void) writeb(x,addr)) -#define outw(x,addr) ((void) writew(x,addr)) -#define outl(x,addr) ((void) writel(x,addr)) - -#define inb_p(addr) inb(addr) -#define inw_p(addr) inw(addr) -#define inl_p(addr) inl(addr) -#define outb_p(x,addr) outb(x,addr) -#define outw_p(x,addr) outw(x,addr) -#define outl_p(x,addr) outl(x,addr) +#else -#define outsb(a,b,l) io_outsb(a,b,l) -#define outsw(a,b,l) io_outsw(a,b,l) -#define outsl(a,b,l) io_outsl(a,b,l) +#define readb __raw_readb +#define readw __raw_readw +#define readl __raw_readl +#define writeb __raw_writeb +#define writew __raw_writew +#define writel __raw_writel -#define insb(a,b,l) io_insb(a,b,l) -#define insw(a,b,l) io_insw(a,b,l) -#define insl(a,b,l) io_insl(a,b,l) - -#define IO_SPACE_LIMIT 0xffffffff - - -/* Values for nocacheflag and cmode */ -#define IOMAP_FULL_CACHING 0 -#define IOMAP_NOCACHE_SER 1 -#define IOMAP_NOCACHE_NONSER 2 -#define IOMAP_WRITETHROUGH 3 - -static inline void *__ioremap(unsigned long physaddr, unsigned long size, int cacheflag) -{ - return (void *) physaddr; -} -static inline void *ioremap(unsigned long physaddr, unsigned long size) -{ - return __ioremap(physaddr, size, IOMAP_NOCACHE_SER); -} -static inline void *ioremap_nocache(unsigned long physaddr, unsigned long size) -{ - return __ioremap(physaddr, size, IOMAP_NOCACHE_SER); -} -static inline void *ioremap_wt(unsigned long physaddr, unsigned long size) -{ - return __ioremap(physaddr, size, IOMAP_WRITETHROUGH); -} -static inline void *ioremap_fullcache(unsigned long physaddr, unsigned long size) -{ - return __ioremap(physaddr, size, IOMAP_FULL_CACHING); -} - -#define iounmap(addr) do { } while(0) +#endif /* IOMEMBASE */ +#if defined(CONFIG_PCI) /* - * Convert a physical pointer to a virtual kernel pointer for /dev/mem - * access + * Support for PCI bus access uses the asm-generic access functions. + * We need to supply the base address and masks for the normal memory + * and IO address space mappings. */ -#define xlate_dev_mem_ptr(p) __va(p) +#define PCI_MEM_PA 0xf0000000 /* Host physical address */ +#define PCI_MEM_BA 0xf0000000 /* Bus physical address */ +#define PCI_MEM_SIZE 0x08000000 /* 128 MB */ +#define PCI_MEM_MASK (PCI_MEM_SIZE - 1) + +#define PCI_IO_PA 0xf8000000 /* Host physical address */ +#define PCI_IO_BA 0x00000000 /* Bus physical address */ +#define PCI_IO_SIZE 0x00010000 /* 64k */ +#define PCI_IO_MASK (PCI_IO_SIZE - 1) + +#define HAVE_ARCH_PIO_SIZE +#define PIO_OFFSET 0 +#define PIO_MASK 0xffff +#define PIO_RESERVED 0x10000 +#define PCI_IOBASE ((void __iomem *) PCI_IO_PA) +#define PCI_SPACE_LIMIT PCI_IO_MASK +#endif /* CONFIG_PCI */ /* - * Convert a virtual cached pointer to an uncached pointer + * These are defined in kmap.h as static inline functions. To maintain + * previous behavior we put these define guards here so io_mm.h doesn't + * see them. */ -#define xlate_dev_kmem_ptr(p) p +#ifdef CONFIG_MMU +#define memset_io memset_io +#define memcpy_fromio memcpy_fromio +#define memcpy_toio memcpy_toio +#endif -static inline void __iomem *ioport_map(unsigned long port, unsigned int nr) -{ - return (void __iomem *) port; -} - -static inline void ioport_unmap(void __iomem *p) -{ -} - -#endif /* __KERNEL__ */ +#include +#include +#include #endif /* _M68KNOMMU_IO_H */ diff --git a/arch/m68k/include/asm/kmap.h b/arch/m68k/include/asm/kmap.h new file mode 100644 index 000000000000..84b8333db8ad --- /dev/null +++ b/arch/m68k/include/asm/kmap.h @@ -0,0 +1,80 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _KMAP_H +#define _KMAP_H + +#ifdef CONFIG_MMU + +/* Values for nocacheflag and cmode */ +#define IOMAP_FULL_CACHING 0 +#define IOMAP_NOCACHE_SER 1 +#define IOMAP_NOCACHE_NONSER 2 +#define IOMAP_WRITETHROUGH 3 + +/* + * These functions exported by arch/m68k/mm/kmap.c. + * Only needed on MMU enabled systems. + */ +extern void __iomem *__ioremap(unsigned long physaddr, unsigned long size, + int cacheflag); +extern void iounmap(void __iomem *addr); +extern void __iounmap(void *addr, unsigned long size); + +#define ioremap ioremap +static inline void __iomem *ioremap(unsigned long physaddr, unsigned long size) +{ + return __ioremap(physaddr, size, IOMAP_NOCACHE_SER); +} + +#define ioremap_nocache ioremap_nocache +static inline void __iomem *ioremap_nocache(unsigned long physaddr, + unsigned long size) +{ + return __ioremap(physaddr, size, IOMAP_NOCACHE_SER); +} + +#define ioremap_uc ioremap_nocache +static inline void __iomem *ioremap_wt(unsigned long physaddr, + unsigned long size) +{ + return __ioremap(physaddr, size, IOMAP_WRITETHROUGH); +} + +#define ioremap_fillcache ioremap_fullcache +static inline void __iomem *ioremap_fullcache(unsigned long physaddr, + unsigned long size) +{ + return __ioremap(physaddr, size, IOMAP_FULL_CACHING); +} + +static inline void memset_io(volatile void __iomem *addr, unsigned char val, + int count) +{ + __builtin_memset((void __force *) addr, val, count); +} + +static inline void memcpy_fromio(void *dst, const volatile void __iomem *src, + int count) +{ + __builtin_memcpy(dst, (void __force *) src, count); +} + +static inline void memcpy_toio(volatile void __iomem *dst, const void *src, + int count) +{ + __builtin_memcpy((void __force *) dst, src, count); +} + +#endif /* CONFIG_MMU */ + +#define ioport_map ioport_map +static inline void __iomem *ioport_map(unsigned long port, unsigned int nr) +{ + return (void __iomem *) port; +} + +#define ioport_unmap ioport_unmap +static inline void ioport_unmap(void __iomem *p) +{ +} + +#endif /* _KMAP_H */ diff --git a/arch/m68k/include/asm/nubus.h b/arch/m68k/include/asm/nubus.h index d0d2039e434e..c2281da6c51a 100644 --- a/arch/m68k/include/asm/nubus.h +++ b/arch/m68k/include/asm/nubus.h @@ -3,6 +3,7 @@ #define _ASM_M68K_NUBUS_H #include +#include #define nubus_readb raw_inb #define nubus_readw raw_inw diff --git a/arch/m68k/include/asm/pci.h b/arch/m68k/include/asm/pci.h index ef26fae8cf0b..5a4bc223743b 100644 --- a/arch/m68k/include/asm/pci.h +++ b/arch/m68k/include/asm/pci.h @@ -4,12 +4,6 @@ #include -/* The PCI address space does equal the physical memory - * address space. The networking and block device layers use - * this boolean for bounce buffer decisions. - */ -#define PCI_DMA_BUS_IS_PHYS (1) - #define pcibios_assign_all_busses() 1 #define PCIBIOS_MIN_IO 0x00000100 diff --git a/arch/m68k/include/asm/q40_master.h b/arch/m68k/include/asm/q40_master.h index 3a89c088800c..9b00fb8079e6 100644 --- a/arch/m68k/include/asm/q40_master.h +++ b/arch/m68k/include/asm/q40_master.h @@ -8,7 +8,7 @@ #define _Q40_MASTER_H #include - +#include #define q40_master_addr 0xff000000 diff --git a/arch/m68k/include/asm/raw_io.h b/arch/m68k/include/asm/raw_io.h index 05e940c29b54..85761255dde5 100644 --- a/arch/m68k/include/asm/raw_io.h +++ b/arch/m68k/include/asm/raw_io.h @@ -13,20 +13,6 @@ #include - -/* Values for nocacheflag and cmode */ -#define IOMAP_FULL_CACHING 0 -#define IOMAP_NOCACHE_SER 1 -#define IOMAP_NOCACHE_NONSER 2 -#define IOMAP_WRITETHROUGH 3 - -extern void iounmap(void __iomem *addr); - -extern void __iomem *__ioremap(unsigned long physaddr, unsigned long size, - int cacheflag); -extern void __iounmap(void *addr, unsigned long size); - - /* ++roman: The assignments to temp. vars avoid that gcc sometimes generates * two accesses to memory, which may be undesirable for some devices. */ diff --git a/arch/m68k/include/asm/uaccess_mm.h b/arch/m68k/include/asm/uaccess_mm.h index 75c172e909ac..c4cb889660aa 100644 --- a/arch/m68k/include/asm/uaccess_mm.h +++ b/arch/m68k/include/asm/uaccess_mm.h @@ -141,10 +141,12 @@ asm volatile ("\n" \ case 4: \ __get_user_asm(__gu_err, x, ptr, u32, l, r, -EFAULT); \ break; \ -/* case 8: disabled because gcc-4.1 has a broken typeof \ - { \ - const void *__gu_ptr = (ptr); \ - u64 __gu_val; \ + case 8: { \ + const void *__gu_ptr = (ptr); \ + union { \ + u64 l; \ + __typeof__(*(ptr)) t; \ + } __gu_val; \ asm volatile ("\n" \ "1: "MOVES".l (%2)+,%1\n" \ "2: "MOVES".l (%2),%R1\n" \ @@ -162,13 +164,13 @@ asm volatile ("\n" \ " .long 1b,10b\n" \ " .long 2b,10b\n" \ " .previous" \ - : "+d" (__gu_err), "=&r" (__gu_val), \ + : "+d" (__gu_err), "=&r" (__gu_val.l), \ "+a" (__gu_ptr) \ : "i" (-EFAULT) \ : "memory"); \ - (x) = (__force typeof(*(ptr)))__gu_val; \ + (x) = __gu_val.t; \ break; \ - } */ \ + } \ default: \ __gu_err = __get_user_bad(); \ break; \ diff --git a/arch/m68k/include/asm/vga.h b/arch/m68k/include/asm/vga.h index 010a624d1b39..4742e6bc3ab8 100644 --- a/arch/m68k/include/asm/vga.h +++ b/arch/m68k/include/asm/vga.h @@ -2,7 +2,15 @@ #ifndef _ASM_M68K_VGA_H #define _ASM_M68K_VGA_H +/* + * Some ColdFire platforms do in fact have a PCI bus. So for those we want + * to use the real IO access functions, don't fake them out or redirect them + * for that case. + */ +#ifndef CONFIG_PCI + #include +#include /* * FIXME @@ -25,4 +33,5 @@ #define writeb raw_outb #define writew raw_outw +#endif /* CONFIG_PCI */ #endif /* _ASM_M68K_VGA_H */ diff --git a/arch/m68k/include/asm/virtconvert.h b/arch/m68k/include/asm/virtconvert.h index 4aea6be7b220..dfe43083b579 100644 --- a/arch/m68k/include/asm/virtconvert.h +++ b/arch/m68k/include/asm/virtconvert.h @@ -16,11 +16,13 @@ /* * Change virtual addresses to physical addresses and vv. */ +#define virt_to_phys virt_to_phys static inline unsigned long virt_to_phys(void *address) { return __pa(address); } +#define phys_to_virt phys_to_virt static inline void *phys_to_virt(unsigned long address) { return __va(address); diff --git a/arch/m68k/include/asm/zorro.h b/arch/m68k/include/asm/zorro.h index 96f64bf7bcaa..60fc4b6f294d 100644 --- a/arch/m68k/include/asm/zorro.h +++ b/arch/m68k/include/asm/zorro.h @@ -3,6 +3,7 @@ #define _ASM_M68K_ZORRO_H #include +#include #define z_readb raw_inb #define z_readw raw_inw diff --git a/arch/m68k/kernel/dma.c b/arch/m68k/kernel/dma.c index c01b9b8f97bf..463572c4943f 100644 --- a/arch/m68k/kernel/dma.c +++ b/arch/m68k/kernel/dma.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -165,3 +166,12 @@ const struct dma_map_ops m68k_dma_ops = { .sync_sg_for_device = m68k_dma_sync_sg_for_device, }; EXPORT_SYMBOL(m68k_dma_ops); + +void arch_setup_pdev_archdata(struct platform_device *pdev) +{ + if (pdev->dev.coherent_dma_mask == DMA_MASK_NONE && + pdev->dev.dma_mask == NULL) { + pdev->dev.coherent_dma_mask = DMA_BIT_MASK(32); + pdev->dev.dma_mask = &pdev->dev.coherent_dma_mask; + } +} diff --git a/arch/m68k/kernel/setup_mm.c b/arch/m68k/kernel/setup_mm.c index dd25bfc22fb4..f35e3ebd6331 100644 --- a/arch/m68k/kernel/setup_mm.c +++ b/arch/m68k/kernel/setup_mm.c @@ -527,21 +527,9 @@ static int hardware_proc_show(struct seq_file *m, void *v) return 0; } -static int hardware_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, hardware_proc_show, NULL); -} - -static const struct file_operations hardware_proc_fops = { - .open = hardware_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int __init proc_hardware_init(void) { - proc_create("hardware", 0, NULL, &hardware_proc_fops); + proc_create_single("hardware", 0, NULL, hardware_proc_show); return 0; } module_init(proc_hardware_init); diff --git a/arch/m68k/kernel/signal.c b/arch/m68k/kernel/signal.c index f7cd5ecfacd3..72850b85ecf8 100644 --- a/arch/m68k/kernel/signal.c +++ b/arch/m68k/kernel/signal.c @@ -576,41 +576,42 @@ static inline int rt_save_fpu_state(struct ucontext __user *uc, struct pt_regs * static inline void siginfo_build_tests(void) { - /* This needs to be tested on m68k as it has a lesser - * alignment requirment than x86 and that can cause surprises. + /* + * This needs to be tested on m68k as it has a lesser + * alignment requirement than x86 and that can cause surprises. */ /* This is part of the ABI and can never change in size: */ BUILD_BUG_ON(sizeof(siginfo_t) != 128); - /* Ensure the know fields never change in location */ + /* Ensure the known fields never change in location */ BUILD_BUG_ON(offsetof(siginfo_t, si_signo) != 0); BUILD_BUG_ON(offsetof(siginfo_t, si_errno) != 4); BUILD_BUG_ON(offsetof(siginfo_t, si_code) != 8); /* _kill */ - BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x0C); + BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x0c); BUILD_BUG_ON(offsetof(siginfo_t, si_uid) != 0x10); /* _timer */ - BUILD_BUG_ON(offsetof(siginfo_t, si_tid) != 0x0C); + BUILD_BUG_ON(offsetof(siginfo_t, si_tid) != 0x0c); BUILD_BUG_ON(offsetof(siginfo_t, si_overrun) != 0x10); BUILD_BUG_ON(offsetof(siginfo_t, si_value) != 0x14); /* _rt */ - BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x0C); + BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x0c); BUILD_BUG_ON(offsetof(siginfo_t, si_uid) != 0x10); BUILD_BUG_ON(offsetof(siginfo_t, si_value) != 0x14); /* _sigchld */ - BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x0C); + BUILD_BUG_ON(offsetof(siginfo_t, si_pid) != 0x0c); BUILD_BUG_ON(offsetof(siginfo_t, si_uid) != 0x10); BUILD_BUG_ON(offsetof(siginfo_t, si_status) != 0x14); BUILD_BUG_ON(offsetof(siginfo_t, si_utime) != 0x18); - BUILD_BUG_ON(offsetof(siginfo_t, si_stime) != 0x1C); + BUILD_BUG_ON(offsetof(siginfo_t, si_stime) != 0x1c); /* _sigfault */ - BUILD_BUG_ON(offsetof(siginfo_t, si_addr) != 0x0C); + BUILD_BUG_ON(offsetof(siginfo_t, si_addr) != 0x0c); /* _sigfault._mcerr */ BUILD_BUG_ON(offsetof(siginfo_t, si_addr_lsb) != 0x10); @@ -623,11 +624,11 @@ static inline void siginfo_build_tests(void) BUILD_BUG_ON(offsetof(siginfo_t, si_pkey) != 0x12); /* _sigpoll */ - BUILD_BUG_ON(offsetof(siginfo_t, si_band) != 0x0C); + BUILD_BUG_ON(offsetof(siginfo_t, si_band) != 0x0c); BUILD_BUG_ON(offsetof(siginfo_t, si_fd) != 0x10); /* _sigsys */ - BUILD_BUG_ON(offsetof(siginfo_t, si_call_addr) != 0x0C); + BUILD_BUG_ON(offsetof(siginfo_t, si_call_addr) != 0x0c); BUILD_BUG_ON(offsetof(siginfo_t, si_syscall) != 0x10); BUILD_BUG_ON(offsetof(siginfo_t, si_arch) != 0x14); diff --git a/arch/m68k/kernel/time.c b/arch/m68k/kernel/time.c index 97dd4e26f234..3a8b47f8f97b 100644 --- a/arch/m68k/kernel/time.c +++ b/arch/m68k/kernel/time.c @@ -71,23 +71,26 @@ static irqreturn_t timer_interrupt(int irq, void *dummy) return IRQ_HANDLED; } -void read_persistent_clock(struct timespec *ts) +#ifdef CONFIG_M68KCLASSIC +#if !IS_BUILTIN(CONFIG_RTC_DRV_GENERIC) +void read_persistent_clock64(struct timespec64 *ts) { struct rtc_time time; + ts->tv_sec = 0; ts->tv_nsec = 0; - if (mach_hwclk) { - mach_hwclk(0, &time); + if (!mach_hwclk) + return; - if ((time.tm_year += 1900) < 1970) - time.tm_year += 100; - ts->tv_sec = mktime(time.tm_year, time.tm_mon, time.tm_mday, - time.tm_hour, time.tm_min, time.tm_sec); - } + mach_hwclk(0, &time); + + ts->tv_sec = mktime64(time.tm_year + 1900, time.tm_mon + 1, time.tm_mday, + time.tm_hour, time.tm_min, time.tm_sec); } +#endif -#if defined(CONFIG_ARCH_USES_GETTIMEOFFSET) && IS_ENABLED(CONFIG_RTC_DRV_GENERIC) +#if IS_ENABLED(CONFIG_RTC_DRV_GENERIC) static int rtc_generic_get_time(struct device *dev, struct rtc_time *tm) { mach_hwclk(0, tm); @@ -145,8 +148,8 @@ static int __init rtc_init(void) } module_init(rtc_init); - -#endif /* CONFIG_ARCH_USES_GETTIMEOFFSET */ +#endif /* CONFIG_RTC_DRV_GENERIC */ +#endif /* CONFIG M68KCLASSIC */ void __init time_init(void) { diff --git a/arch/m68k/kernel/traps.c b/arch/m68k/kernel/traps.c index c1cc4e99aa94..b2fd000b9285 100644 --- a/arch/m68k/kernel/traps.c +++ b/arch/m68k/kernel/traps.c @@ -1007,9 +1007,9 @@ void bad_super_trap (struct frame *fp) asmlinkage void trap_c(struct frame *fp) { - int sig; + int sig, si_code; + void __user *addr; int vector = (fp->ptregs.vector >> 2) & 0xff; - siginfo_t info; if (fp->ptregs.sr & PS_S) { if (vector == VEC_TRACE) { @@ -1029,21 +1029,21 @@ asmlinkage void trap_c(struct frame *fp) /* send the appropriate signal to the user program */ switch (vector) { case VEC_ADDRERR: - info.si_code = BUS_ADRALN; + si_code = BUS_ADRALN; sig = SIGBUS; break; case VEC_ILLEGAL: case VEC_LINE10: case VEC_LINE11: - info.si_code = ILL_ILLOPC; + si_code = ILL_ILLOPC; sig = SIGILL; break; case VEC_PRIV: - info.si_code = ILL_PRVOPC; + si_code = ILL_PRVOPC; sig = SIGILL; break; case VEC_COPROC: - info.si_code = ILL_COPROC; + si_code = ILL_COPROC; sig = SIGILL; break; case VEC_TRAP1: @@ -1060,76 +1060,74 @@ asmlinkage void trap_c(struct frame *fp) case VEC_TRAP12: case VEC_TRAP13: case VEC_TRAP14: - info.si_code = ILL_ILLTRP; + si_code = ILL_ILLTRP; sig = SIGILL; break; case VEC_FPBRUC: case VEC_FPOE: case VEC_FPNAN: - info.si_code = FPE_FLTINV; + si_code = FPE_FLTINV; sig = SIGFPE; break; case VEC_FPIR: - info.si_code = FPE_FLTRES; + si_code = FPE_FLTRES; sig = SIGFPE; break; case VEC_FPDIVZ: - info.si_code = FPE_FLTDIV; + si_code = FPE_FLTDIV; sig = SIGFPE; break; case VEC_FPUNDER: - info.si_code = FPE_FLTUND; + si_code = FPE_FLTUND; sig = SIGFPE; break; case VEC_FPOVER: - info.si_code = FPE_FLTOVF; + si_code = FPE_FLTOVF; sig = SIGFPE; break; case VEC_ZERODIV: - info.si_code = FPE_INTDIV; + si_code = FPE_INTDIV; sig = SIGFPE; break; case VEC_CHK: case VEC_TRAP: - info.si_code = FPE_INTOVF; + si_code = FPE_INTOVF; sig = SIGFPE; break; case VEC_TRACE: /* ptrace single step */ - info.si_code = TRAP_TRACE; + si_code = TRAP_TRACE; sig = SIGTRAP; break; case VEC_TRAP15: /* breakpoint */ - info.si_code = TRAP_BRKPT; + si_code = TRAP_BRKPT; sig = SIGTRAP; break; default: - info.si_code = ILL_ILLOPC; + si_code = ILL_ILLOPC; sig = SIGILL; break; } - info.si_signo = sig; - info.si_errno = 0; switch (fp->ptregs.format) { default: - info.si_addr = (void *) fp->ptregs.pc; + addr = (void __user *) fp->ptregs.pc; break; case 2: - info.si_addr = (void *) fp->un.fmt2.iaddr; + addr = (void __user *) fp->un.fmt2.iaddr; break; case 7: - info.si_addr = (void *) fp->un.fmt7.effaddr; + addr = (void __user *) fp->un.fmt7.effaddr; break; case 9: - info.si_addr = (void *) fp->un.fmt9.iaddr; + addr = (void __user *) fp->un.fmt9.iaddr; break; case 10: - info.si_addr = (void *) fp->un.fmta.daddr; + addr = (void __user *) fp->un.fmta.daddr; break; case 11: - info.si_addr = (void *) fp->un.fmtb.daddr; + addr = (void __user*) fp->un.fmtb.daddr; break; } - force_sig_info (sig, &info, current); + force_sig_fault(sig, si_code, addr, current); } void die_if_kernel (char *str, struct pt_regs *fp, int nr) @@ -1161,12 +1159,6 @@ asmlinkage void fpsp040_die(void) #ifdef CONFIG_M68KFPU_EMU asmlinkage void fpemu_signal(int signal, int code, void *addr) { - siginfo_t info; - - info.si_signo = signal; - info.si_errno = 0; - info.si_code = code; - info.si_addr = addr; - force_sig_info(signal, &info, current); + force_sig_fault(signal, code, addr, current); } #endif diff --git a/arch/m68k/mac/config.c b/arch/m68k/mac/config.c index 0c3275aa0197..e522307db47c 100644 --- a/arch/m68k/mac/config.c +++ b/arch/m68k/mac/config.c @@ -1005,7 +1005,7 @@ int __init mac_platform_init(void) struct resource swim_rsrc = { .flags = IORESOURCE_MEM, .start = (resource_size_t)swim_base, - .end = (resource_size_t)swim_base + 0x2000, + .end = (resource_size_t)swim_base + 0x1FFF, }; platform_device_register_simple("swim", -1, &swim_rsrc, 1); diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c index 03253c4f8e6a..f2ff3779875a 100644 --- a/arch/m68k/mm/fault.c +++ b/arch/m68k/mm/fault.c @@ -21,35 +21,32 @@ extern void die_if_kernel(char *, struct pt_regs *, long); int send_fault_sig(struct pt_regs *regs) { - siginfo_t siginfo; + int signo, si_code; + void __user *addr; - clear_siginfo(&siginfo); - siginfo.si_signo = current->thread.signo; - siginfo.si_code = current->thread.code; - siginfo.si_addr = (void *)current->thread.faddr; - pr_debug("send_fault_sig: %p,%d,%d\n", siginfo.si_addr, - siginfo.si_signo, siginfo.si_code); + signo = current->thread.signo; + si_code = current->thread.code; + addr = (void __user *)current->thread.faddr; + pr_debug("send_fault_sig: %p,%d,%d\n", addr, signo, si_code); if (user_mode(regs)) { - force_sig_info(siginfo.si_signo, - &siginfo, current); + force_sig_fault(signo, si_code, addr, current); } else { if (fixup_exception(regs)) return -1; - //if (siginfo.si_signo == SIGBUS) - // force_sig_info(siginfo.si_signo, - // &siginfo, current); + //if (signo == SIGBUS) + // force_sig_fault(si_signo, si_code, addr, current); /* * Oops. The kernel tried to access some bad page. We'll have to * terminate things with extreme prejudice. */ - if ((unsigned long)siginfo.si_addr < PAGE_SIZE) + if ((unsigned long)addr < PAGE_SIZE) pr_alert("Unable to handle kernel NULL pointer dereference"); else pr_alert("Unable to handle kernel access"); - pr_cont(" at virtual address %p\n", siginfo.si_addr); + pr_cont(" at virtual address %p\n", addr); die_if_kernel("Oops", regs, 0 /*error_code*/); do_exit(SIGKILL); } diff --git a/arch/m68k/mm/kmap.c b/arch/m68k/mm/kmap.c index c2a38321c96d..40a3b327da07 100644 --- a/arch/m68k/mm/kmap.c +++ b/arch/m68k/mm/kmap.c @@ -89,7 +89,8 @@ static inline void free_io_area(void *addr) for (p = &iolist ; (tmp = *p) ; p = &tmp->next) { if (tmp->addr == addr) { *p = tmp->next; - __iounmap(tmp->addr, tmp->size); + /* remove gap added in get_io_area() */ + __iounmap(tmp->addr, tmp->size - IO_SIZE); kfree(tmp); return; } @@ -125,6 +126,10 @@ void __iomem *__ioremap(unsigned long physaddr, unsigned long size, int cachefla return (void __iomem *)physaddr; } #endif +#ifdef CONFIG_COLDFIRE + if (__cf_internalio(physaddr)) + return (void __iomem *) physaddr; +#endif #ifdef DEBUG printk("ioremap: 0x%lx,0x%lx(%d) - ", physaddr, size, cacheflag); @@ -235,6 +240,10 @@ void iounmap(void __iomem *addr) ((unsigned long)addr > 0x60000000))) free_io_area((__force void *)addr); #else +#ifdef CONFIG_COLDFIRE + if (cf_internalio(addr)) + return; +#endif free_io_area((__force void *)addr); #endif } diff --git a/arch/m68k/mvme147/config.c b/arch/m68k/mvme147/config.c index 8778612d1f31..f8a710fd84cd 100644 --- a/arch/m68k/mvme147/config.c +++ b/arch/m68k/mvme147/config.c @@ -153,12 +153,14 @@ int mvme147_hwclk(int op, struct rtc_time *t) if (!op) { m147_rtc->ctrl = RTC_READ; t->tm_year = bcd2int (m147_rtc->bcd_year); - t->tm_mon = bcd2int (m147_rtc->bcd_mth); + t->tm_mon = bcd2int(m147_rtc->bcd_mth) - 1; t->tm_mday = bcd2int (m147_rtc->bcd_dom); t->tm_hour = bcd2int (m147_rtc->bcd_hr); t->tm_min = bcd2int (m147_rtc->bcd_min); t->tm_sec = bcd2int (m147_rtc->bcd_sec); m147_rtc->ctrl = 0; + if (t->tm_year < 70) + t->tm_year += 100; } return 0; } diff --git a/arch/m68k/mvme16x/config.c b/arch/m68k/mvme16x/config.c index 6fa06d4d16bf..4ffd9ef98de4 100644 --- a/arch/m68k/mvme16x/config.c +++ b/arch/m68k/mvme16x/config.c @@ -400,12 +400,14 @@ int mvme16x_hwclk(int op, struct rtc_time *t) if (!op) { rtc->ctrl = RTC_READ; t->tm_year = bcd2int (rtc->bcd_year); - t->tm_mon = bcd2int (rtc->bcd_mth); + t->tm_mon = bcd2int(rtc->bcd_mth) - 1; t->tm_mday = bcd2int (rtc->bcd_dom); t->tm_hour = bcd2int (rtc->bcd_hr); t->tm_min = bcd2int (rtc->bcd_min); t->tm_sec = bcd2int (rtc->bcd_sec); rtc->ctrl = 0; + if (t->tm_year < 70) + t->tm_year += 100; } return 0; } diff --git a/arch/m68k/sun3/intersil.c b/arch/m68k/sun3/intersil.c index 2cd0bcbe6f30..d911070af02a 100644 --- a/arch/m68k/sun3/intersil.c +++ b/arch/m68k/sun3/intersil.c @@ -48,9 +48,9 @@ int sun3_hwclk(int set, struct rtc_time *t) todintersil->hour = t->tm_hour; todintersil->minute = t->tm_min; todintersil->second = t->tm_sec; - todintersil->month = t->tm_mon; + todintersil->month = t->tm_mon + 1; todintersil->day = t->tm_mday; - todintersil->year = t->tm_year - 68; + todintersil->year = (t->tm_year - 68) % 100; todintersil->weekday = t->tm_wday; } else { /* read clock */ @@ -58,10 +58,12 @@ int sun3_hwclk(int set, struct rtc_time *t) t->tm_hour = todintersil->hour; t->tm_min = todintersil->minute; t->tm_sec = todintersil->second; - t->tm_mon = todintersil->month; + t->tm_mon = todintersil->month - 1; t->tm_mday = todintersil->day; t->tm_year = todintersil->year + 68; t->tm_wday = todintersil->weekday; + if (t->tm_year < 70) + t->tm_year += 100; } intersil_clock->cmd_reg = START_VAL; diff --git a/arch/m68k/sun3x/time.c b/arch/m68k/sun3x/time.c index 7a2c53d9f779..047e2bcee3d7 100644 --- a/arch/m68k/sun3x/time.c +++ b/arch/m68k/sun3x/time.c @@ -52,8 +52,8 @@ int sun3x_hwclk(int set, struct rtc_time *t) h->hour = bin2bcd(t->tm_hour); h->wday = bin2bcd(t->tm_wday); h->mday = bin2bcd(t->tm_mday); - h->month = bin2bcd(t->tm_mon); - h->year = bin2bcd(t->tm_year); + h->month = bin2bcd(t->tm_mon + 1); + h->year = bin2bcd(t->tm_year % 100); h->csr &= ~C_WRITE; } else { h->csr |= C_READ; @@ -62,9 +62,11 @@ int sun3x_hwclk(int set, struct rtc_time *t) t->tm_hour = bcd2bin(h->hour); t->tm_wday = bcd2bin(h->wday); t->tm_mday = bcd2bin(h->mday); - t->tm_mon = bcd2bin(h->month); + t->tm_mon = bcd2bin(h->month) - 1; t->tm_year = bcd2bin(h->year); h->csr &= ~C_READ; + if (t->tm_year < 70) + t->tm_year += 100; } local_irq_restore(flags); diff --git a/arch/microblaze/Kconfig b/arch/microblaze/Kconfig index 3817a3e2146c..d14782100088 100644 --- a/arch/microblaze/Kconfig +++ b/arch/microblaze/Kconfig @@ -19,7 +19,6 @@ config MICROBLAZE select HAVE_ARCH_HASH select HAVE_ARCH_KGDB select HAVE_DEBUG_KMEMLEAK - select HAVE_DMA_API_DEBUG select HAVE_DYNAMIC_FTRACE select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_GRAPH_TRACER diff --git a/arch/microblaze/Kconfig.debug b/arch/microblaze/Kconfig.debug index 012e377330cd..331a3bb66297 100644 --- a/arch/microblaze/Kconfig.debug +++ b/arch/microblaze/Kconfig.debug @@ -8,14 +8,6 @@ config TRACE_IRQFLAGS_SUPPORT source "lib/Kconfig.debug" -config EARLY_PRINTK - bool "Early printk function for kernel" - depends on SERIAL_UARTLITE_CONSOLE || SERIAL_8250_CONSOLE - default n - help - This option turns on/off early printk messages to console. - First Uartlite node is taken. - config HEART_BEAT bool "Heart beat function for kernel" default n diff --git a/arch/microblaze/boot/Makefile b/arch/microblaze/boot/Makefile index fd46385a4c97..600e5a198bd2 100644 --- a/arch/microblaze/boot/Makefile +++ b/arch/microblaze/boot/Makefile @@ -22,17 +22,19 @@ $(obj)/linux.bin.gz: $(obj)/linux.bin FORCE quiet_cmd_cp = CP $< $@$2 cmd_cp = cat $< >$@$2 || (rm -f $@ && echo false) -quiet_cmd_strip = STRIP $@ +quiet_cmd_strip = STRIP $< $@$2 cmd_strip = $(STRIP) -K microblaze_start -K _end -K __log_buf \ - -K _fdt_start vmlinux -o $@ + -K _fdt_start $< -o $@$2 UIMAGE_LOADADDR = $(CONFIG_KERNEL_BASE_ADDR) +UIMAGE_IN = $@ +UIMAGE_OUT = $@.ub $(obj)/simpleImage.%: vmlinux FORCE $(call if_changed,cp,.unstrip) $(call if_changed,objcopy) $(call if_changed,uimage) - $(call if_changed,strip) - @echo 'Kernel: $@ is ready' ' (#'`cat .version`')' + $(call if_changed,strip,.strip) + @echo 'Kernel: $(UIMAGE_OUT) is ready' ' (#'`cat .version`')' clean-files += simpleImage.*.unstrip linux.bin.ub diff --git a/arch/microblaze/boot/dts/system.dts b/arch/microblaze/boot/dts/system.dts index b620da23febb..8a420c6702eb 100644 --- a/arch/microblaze/boot/dts/system.dts +++ b/arch/microblaze/boot/dts/system.dts @@ -44,7 +44,7 @@ } ; chosen { bootargs = "console=ttyUL0,115200 highres=on"; - linux,stdout-path = "/plb@0/serial@84000000"; + stdout-path = "/plb@0/serial@84000000"; } ; cpus { #address-cells = <1>; diff --git a/arch/microblaze/include/asm/Kbuild b/arch/microblaze/include/asm/Kbuild index 3c80a5a308ed..fe6a6c6e5003 100644 --- a/arch/microblaze/include/asm/Kbuild +++ b/arch/microblaze/include/asm/Kbuild @@ -2,6 +2,7 @@ generic-y += barrier.h generic-y += bitops.h generic-y += bug.h generic-y += bugs.h +generic-y += compat.h generic-y += device.h generic-y += div64.h generic-y += emergency-restart.h diff --git a/arch/microblaze/include/asm/cacheflush.h b/arch/microblaze/include/asm/cacheflush.h index ffea82a16d2c..b091de77b15b 100644 --- a/arch/microblaze/include/asm/cacheflush.h +++ b/arch/microblaze/include/asm/cacheflush.h @@ -19,7 +19,7 @@ #include #include -/* Look at Documentation/cachetlb.txt */ +/* Look at Documentation/core-api/cachetlb.rst */ /* * Cache handling functions. diff --git a/arch/microblaze/include/asm/cpuinfo.h b/arch/microblaze/include/asm/cpuinfo.h index 3337417fcdca..8f4996730552 100644 --- a/arch/microblaze/include/asm/cpuinfo.h +++ b/arch/microblaze/include/asm/cpuinfo.h @@ -13,7 +13,7 @@ #ifndef _ASM_MICROBLAZE_CPUINFO_H #define _ASM_MICROBLAZE_CPUINFO_H -#include +#include /* CPU Version and FPGA Family code conversion table type */ struct cpu_ver_key { diff --git a/arch/microblaze/include/asm/pci.h b/arch/microblaze/include/asm/pci.h index 5de871eb4a59..859c19828dd4 100644 --- a/arch/microblaze/include/asm/pci.h +++ b/arch/microblaze/include/asm/pci.h @@ -19,7 +19,6 @@ #include #include -#include #include #define PCIBIOS_MIN_IO 0x1000 @@ -62,16 +61,6 @@ extern int pci_mmap_legacy_page_range(struct pci_bus *bus, #define HAVE_PCI_LEGACY 1 -/* The PCI address space does equal the physical memory - * address space (no IOMMU). The IDE and SCSI device layers use - * this boolean for bounce buffer decisions. - */ -#define PCI_DMA_BUS_IS_PHYS (1) - -extern void pcibios_claim_one_bus(struct pci_bus *b); - -extern void pcibios_finish_adding_to_bus(struct pci_bus *bus); - extern void pcibios_resource_survey(void); struct file; diff --git a/arch/microblaze/include/asm/prom.h b/arch/microblaze/include/asm/prom.h deleted file mode 100644 index 2f03ac815851..000000000000 --- a/arch/microblaze/include/asm/prom.h +++ /dev/null @@ -1,27 +0,0 @@ -/* - * Definitions for talking to the Open Firmware PROM on - * Power Macintosh computers. - * - * Copyright (C) 1996-2005 Paul Mackerras. - * - * Updates for PPC64 by Peter Bergner & David Engebretsen, IBM Corp. - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. - */ -#ifndef _ASM_MICROBLAZE_PROM_H -#define _ASM_MICROBLAZE_PROM_H - -#include - -/* Other Prototypes */ -enum early_consoles { - UARTLITE = 1, - UART16550 = 2, -}; - -extern int of_early_console(void *version); - -#endif /* _ASM_MICROBLAZE_PROM_H */ diff --git a/arch/microblaze/include/asm/setup.h b/arch/microblaze/include/asm/setup.h index 7c968c1d1729..d5384f6f36f7 100644 --- a/arch/microblaze/include/asm/setup.h +++ b/arch/microblaze/include/asm/setup.h @@ -19,16 +19,11 @@ extern char cmd_line[COMMAND_LINE_SIZE]; extern char *klimit; -int setup_early_printk(char *opt); -void remap_early_printk(void); -void disable_early_printk(void); - void microblaze_heartbeat(void); void microblaze_setup_heartbeat(void); # ifdef CONFIG_MMU extern void mmu_reset(void); -extern void early_console_reg_tlb_alloc(unsigned int addr); # endif /* CONFIG_MMU */ extern void of_platform_reset_gpio_probe(void); diff --git a/arch/microblaze/kernel/Makefile b/arch/microblaze/kernel/Makefile index 0da76fa1ab17..7e99cf6984a1 100644 --- a/arch/microblaze/kernel/Makefile +++ b/arch/microblaze/kernel/Makefile @@ -22,7 +22,6 @@ obj-y += dma.o exceptions.o \ obj-y += cpu/ -obj-$(CONFIG_EARLY_PRINTK) += early_printk.o obj-$(CONFIG_HEART_BEAT) += heartbeat.o obj-$(CONFIG_MODULES) += microblaze_ksyms.o module.o obj-$(CONFIG_MMU) += misc.o diff --git a/arch/microblaze/kernel/dma.c b/arch/microblaze/kernel/dma.c index c91e8cef98dd..3145e7dc8ab1 100644 --- a/arch/microblaze/kernel/dma.c +++ b/arch/microblaze/kernel/dma.c @@ -184,14 +184,3 @@ const struct dma_map_ops dma_nommu_ops = { .sync_sg_for_device = dma_nommu_sync_sg_for_device, }; EXPORT_SYMBOL(dma_nommu_ops); - -/* Number of entries preallocated for DMA-API debugging */ -#define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16) - -static int __init dma_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - - return 0; -} -fs_initcall(dma_init); diff --git a/arch/microblaze/kernel/early_printk.c b/arch/microblaze/kernel/early_printk.c deleted file mode 100644 index 365f2d53f1b2..000000000000 --- a/arch/microblaze/kernel/early_printk.c +++ /dev/null @@ -1,184 +0,0 @@ -/* - * Early printk support for Microblaze. - * - * Copyright (C) 2007-2009 Michal Simek - * Copyright (C) 2007-2009 PetaLogix - * Copyright (C) 2003-2006 Yasushi SHOJI - * - * This file is subject to the terms and conditions of the GNU General Public - * License. See the file "COPYING" in the main directory of this archive - * for more details. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -static u32 base_addr; - -#ifdef CONFIG_SERIAL_UARTLITE_CONSOLE -static void early_printk_uartlite_putc(char c) -{ - /* - * Limit how many times we'll spin waiting for TX FIFO status. - * This will prevent lockups if the base address is incorrectly - * set, or any other issue on the UARTLITE. - * This limit is pretty arbitrary, unless we are at about 10 baud - * we'll never timeout on a working UART. - */ - - unsigned retries = 1000000; - /* read status bit - 0x8 offset */ - while (--retries && (in_be32(base_addr + 8) & (1 << 3))) - ; - - /* Only attempt the iowrite if we didn't timeout */ - /* write to TX_FIFO - 0x4 offset */ - if (retries) - out_be32(base_addr + 4, c & 0xff); -} - -static void early_printk_uartlite_write(struct console *unused, - const char *s, unsigned n) -{ - while (*s && n-- > 0) { - if (*s == '\n') - early_printk_uartlite_putc('\r'); - early_printk_uartlite_putc(*s); - s++; - } -} - -static struct console early_serial_uartlite_console = { - .name = "earlyser", - .write = early_printk_uartlite_write, - .flags = CON_PRINTBUFFER | CON_BOOT, - .index = -1, -}; -#endif /* CONFIG_SERIAL_UARTLITE_CONSOLE */ - -#ifdef CONFIG_SERIAL_8250_CONSOLE -static void early_printk_uart16550_putc(char c) -{ - /* - * Limit how many times we'll spin waiting for TX FIFO status. - * This will prevent lockups if the base address is incorrectly - * set, or any other issue on the UARTLITE. - * This limit is pretty arbitrary, unless we are at about 10 baud - * we'll never timeout on a working UART. - */ - - #define UART_LSR_TEMT 0x40 /* Transmitter empty */ - #define UART_LSR_THRE 0x20 /* Transmit-hold-register empty */ - #define BOTH_EMPTY (UART_LSR_TEMT | UART_LSR_THRE) - - unsigned retries = 10000; - - while (--retries && - !((in_be32(base_addr + 0x14) & BOTH_EMPTY) == BOTH_EMPTY)) - ; - - if (retries) - out_be32(base_addr, c & 0xff); -} - -static void early_printk_uart16550_write(struct console *unused, - const char *s, unsigned n) -{ - while (*s && n-- > 0) { - if (*s == '\n') - early_printk_uart16550_putc('\r'); - early_printk_uart16550_putc(*s); - s++; - } -} - -static struct console early_serial_uart16550_console = { - .name = "earlyser", - .write = early_printk_uart16550_write, - .flags = CON_PRINTBUFFER | CON_BOOT, - .index = -1, -}; -#endif /* CONFIG_SERIAL_8250_CONSOLE */ - -int __init setup_early_printk(char *opt) -{ - int version = 0; - - if (early_console) - return 1; - - base_addr = of_early_console(&version); - if (base_addr) { -#ifdef CONFIG_MMU - early_console_reg_tlb_alloc(base_addr); -#endif - switch (version) { -#ifdef CONFIG_SERIAL_UARTLITE_CONSOLE - case UARTLITE: - pr_info("Early console on uartlite at 0x%08x\n", - base_addr); - early_console = &early_serial_uartlite_console; - break; -#endif -#ifdef CONFIG_SERIAL_8250_CONSOLE - case UART16550: - pr_info("Early console on uart16650 at 0x%08x\n", - base_addr); - early_console = &early_serial_uart16550_console; - break; -#endif - default: - pr_info("Unsupported early console %d\n", - version); - return 1; - } - - register_console(early_console); - return 0; - } - return 1; -} - -/* Remap early console to virtual address and do not allocate one TLB - * only for early console because of performance degression */ -void __init remap_early_printk(void) -{ - if (!early_console) - return; - pr_info("early_printk_console remapping from 0x%x to ", base_addr); - base_addr = (u32) ioremap(base_addr, PAGE_SIZE); - pr_cont("0x%x\n", base_addr); - -#ifdef CONFIG_MMU - /* - * Early console is on the top of skipped TLB entries - * decrease tlb_skip size ensure that hardcoded TLB entry will be - * used by generic algorithm - * FIXME check if early console mapping is on the top by rereading - * TLB entry and compare baseaddr - * mts rtlbx, (tlb_skip - 1) - * nop - * mfs rX, rtlblo - * nop - * cmp rX, orig_base_addr - */ - tlb_skip -= 1; -#endif -} - -void __init disable_early_printk(void) -{ - if (!early_console) - return; - pr_warn("disabling early console\n"); - unregister_console(early_console); - early_console = NULL; -} diff --git a/arch/microblaze/kernel/exceptions.c b/arch/microblaze/kernel/exceptions.c index e6f338d0496b..eafff21fcb0e 100644 --- a/arch/microblaze/kernel/exceptions.c +++ b/arch/microblaze/kernel/exceptions.c @@ -60,16 +60,10 @@ asmlinkage void sw_exception(struct pt_regs *regs) void _exception(int signr, struct pt_regs *regs, int code, unsigned long addr) { - siginfo_t info; - if (kernel_mode(regs)) die("Exception in kernel mode", regs, signr); - info.si_signo = signr; - info.si_errno = 0; - info.si_code = code; - info.si_addr = (void __user *) addr; - force_sig_info(signr, &info, current); + force_sig_fault(signr, code, (void __user *)addr, current); } asmlinkage void full_exception(struct pt_regs *regs, unsigned int type, diff --git a/arch/microblaze/kernel/misc.S b/arch/microblaze/kernel/misc.S index 1dafddeb8a0b..6759af688451 100644 --- a/arch/microblaze/kernel/misc.S +++ b/arch/microblaze/kernel/misc.S @@ -63,38 +63,3 @@ _tlbie_1: nop .size _tlbie, . - _tlbie - -/* - * Allocate TLB entry for early console - */ -.globl early_console_reg_tlb_alloc; -.type early_console_reg_tlb_alloc, @function -.align 4; -early_console_reg_tlb_alloc: - /* - * Load a TLB entry for the UART, so that microblaze_progress() can use - * the UARTs nice and early. We use a 4k real==virtual mapping. - */ - lwi r4, r0, tlb_skip - mts rtlbx, r4 /* TLB slot 63 */ - - or r4,r5,r0 - andi r4,r4,0xfffff000 - ori r4,r4,(TLB_WR|TLB_I|TLB_M|TLB_G) - - andi r5,r5,0xfffff000 - ori r5,r5,(TLB_VALID | TLB_PAGESZ(PAGESZ_4K)) - - mts rtlblo,r4 /* Load the data portion of the entry */ - nop - mts rtlbhi,r5 /* Load the tag portion of the entry */ - nop - - lwi r5, r0, tlb_skip - addik r5, r5, 1 - swi r5, r0, tlb_skip - - rtsd r15, 8 - nop - - .size early_console_reg_tlb_alloc, . - early_console_reg_tlb_alloc diff --git a/arch/microblaze/kernel/platform.c b/arch/microblaze/kernel/platform.c index b9529caa507a..2540d60610d9 100644 --- a/arch/microblaze/kernel/platform.c +++ b/arch/microblaze/kernel/platform.c @@ -12,7 +12,6 @@ #include #include -#include #include static struct of_device_id xilinx_of_bus_ids[] __initdata = { diff --git a/arch/microblaze/kernel/prom.c b/arch/microblaze/kernel/prom.c index 68f099960ebc..c76c93b90b79 100644 --- a/arch/microblaze/kernel/prom.c +++ b/arch/microblaze/kernel/prom.c @@ -13,91 +13,11 @@ * 2 of the License, or (at your option) any later version. */ -#include -#include #include #include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include #include #include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#ifdef CONFIG_EARLY_PRINTK -static const char *stdout; - -static int __init early_init_dt_scan_chosen_serial(unsigned long node, - const char *uname, int depth, void *data) -{ - int l; - const char *p; - - pr_debug("%s: depth: %d, uname: %s\n", __func__, depth, uname); - - if (depth == 1 && (strcmp(uname, "chosen") == 0 || - strcmp(uname, "chosen@0") == 0)) { - p = of_get_flat_dt_prop(node, "linux,stdout-path", &l); - if (p != NULL && l > 0) - stdout = p; /* store pointer to stdout-path */ - } - - if (stdout && strstr(stdout, uname)) { - p = of_get_flat_dt_prop(node, "compatible", &l); - pr_debug("Compatible string: %s\n", p); - - if ((strncmp(p, "xlnx,xps-uart16550", 18) == 0) || - (strncmp(p, "xlnx,axi-uart16550", 18) == 0)) { - unsigned int addr; - - *(u32 *)data = UART16550; - - addr = *(u32 *)of_get_flat_dt_prop(node, "reg", &l); - addr += *(u32 *)of_get_flat_dt_prop(node, - "reg-offset", &l); - /* clear register offset */ - return be32_to_cpu(addr) & ~3; - } - if ((strncmp(p, "xlnx,xps-uartlite", 17) == 0) || - (strncmp(p, "xlnx,opb-uartlite", 17) == 0) || - (strncmp(p, "xlnx,axi-uartlite", 17) == 0) || - (strncmp(p, "xlnx,mdm", 8) == 0)) { - const unsigned int *addrp; - - *(u32 *)data = UARTLITE; - - addrp = of_get_flat_dt_prop(node, "reg", &l); - return be32_to_cpup(addrp); /* return address */ - } - } - return 0; -} - -/* this function is looking for early console - Microblaze specific */ -int __init of_early_console(void *version) -{ - return of_scan_flat_dt(early_init_dt_scan_chosen_serial, version); -} -#endif - void __init early_init_devtree(void *params) { pr_debug(" -> early_init_devtree(%p)\n", params); @@ -106,8 +26,6 @@ void __init early_init_devtree(void *params) if (!strlen(boot_command_line)) strlcpy(boot_command_line, cmd_line, COMMAND_LINE_SIZE); - parse_early_param(); - memblock_allow_resize(); pr_debug("Phys. mem: %lx\n", (unsigned long) memblock_phys_mem_size()); diff --git a/arch/microblaze/kernel/setup.c b/arch/microblaze/kernel/setup.c index be98ffe28ca8..bbd6968ce55b 100644 --- a/arch/microblaze/kernel/setup.c +++ b/arch/microblaze/kernel/setup.c @@ -27,13 +27,12 @@ #include #include #include -#include +#include #include #include #include #include -#include #include DEFINE_PER_CPU(unsigned int, KSP); /* Saved kernel stack pointer */ @@ -54,6 +53,9 @@ void __init setup_arch(char **cmdline_p) { *cmdline_p = boot_command_line; + setup_memory(); + parse_early_param(); + console_verbose(); unflatten_device_tree(); @@ -62,13 +64,6 @@ void __init setup_arch(char **cmdline_p) microblaze_cache_init(); - setup_memory(); - -#ifdef CONFIG_EARLY_PRINTK - /* remap early console to virtual address */ - remap_early_printk(); -#endif - xilinx_pci_init(); #if defined(CONFIG_DUMMY_CONSOLE) @@ -133,10 +128,6 @@ void __init machine_early_init(const char *cmdline, unsigned int ram, /* initialize device tree for usage in early_printk */ early_init_devtree(_fdt_start); -#ifdef CONFIG_EARLY_PRINTK - setup_early_printk(NULL); -#endif - /* setup kernel_tlb after BSS cleaning * Maybe worth to move to asm code */ kernel_tlb = tlb0 + tlb1; diff --git a/arch/microblaze/mm/fault.c b/arch/microblaze/mm/fault.c index f91b30f8aaa8..af607447c683 100644 --- a/arch/microblaze/mm/fault.c +++ b/arch/microblaze/mm/fault.c @@ -88,7 +88,6 @@ void do_page_fault(struct pt_regs *regs, unsigned long address, { struct vm_area_struct *vma; struct mm_struct *mm = current->mm; - siginfo_t info; int code = SEGV_MAPERR; int is_write = error_code & ESR_S; int fault; @@ -269,11 +268,6 @@ bad_area_nosemaphore: /* User mode accesses cause a SIGSEGV */ if (user_mode(regs)) { _exception(SIGSEGV, regs, code, address); -/* info.si_signo = SIGSEGV; - info.si_errno = 0; - info.si_code = code; - info.si_addr = (void *) address; - force_sig_info(SIGSEGV, &info, current);*/ return; } @@ -295,11 +289,7 @@ out_of_memory: do_sigbus: up_read(&mm->mmap_sem); if (user_mode(regs)) { - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRERR; - info.si_addr = (void __user *)address; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGBUS, BUS_ADRERR, (void __user *)address, current); return; } bad_page_fault(regs, address, SIGBUS); diff --git a/arch/microblaze/pci/indirect_pci.c b/arch/microblaze/pci/indirect_pci.c index ae4fca46c9f6..24030837a425 100644 --- a/arch/microblaze/pci/indirect_pci.c +++ b/arch/microblaze/pci/indirect_pci.c @@ -16,7 +16,6 @@ #include #include -#include #include static int diff --git a/arch/microblaze/pci/pci-common.c b/arch/microblaze/pci/pci-common.c index 161f9758c631..f34346d56095 100644 --- a/arch/microblaze/pci/pci-common.c +++ b/arch/microblaze/pci/pci-common.c @@ -915,67 +915,6 @@ void __init pcibios_resource_survey(void) pci_assign_unassigned_resources(); } -/* This is used by the PCI hotplug driver to allocate resource - * of newly plugged busses. We can try to consolidate with the - * rest of the code later, for now, keep it as-is as our main - * resource allocation function doesn't deal with sub-trees yet. - */ -void pcibios_claim_one_bus(struct pci_bus *bus) -{ - struct pci_dev *dev; - struct pci_bus *child_bus; - - list_for_each_entry(dev, &bus->devices, bus_list) { - int i; - - for (i = 0; i < PCI_NUM_RESOURCES; i++) { - struct resource *r = &dev->resource[i]; - - if (r->parent || !r->start || !r->flags) - continue; - - pr_debug("PCI: Claiming %s: ", pci_name(dev)); - pr_debug("Resource %d: %016llx..%016llx [%x]\n", - i, (unsigned long long)r->start, - (unsigned long long)r->end, - (unsigned int)r->flags); - - if (pci_claim_resource(dev, i) == 0) - continue; - - pci_claim_bridge_resource(dev, i); - } - } - - list_for_each_entry(child_bus, &bus->children, node) - pcibios_claim_one_bus(child_bus); -} -EXPORT_SYMBOL_GPL(pcibios_claim_one_bus); - - -/* pcibios_finish_adding_to_bus - * - * This is to be called by the hotplug code after devices have been - * added to a bus, this include calling it for a PHB that is just - * being added - */ -void pcibios_finish_adding_to_bus(struct pci_bus *bus) -{ - pr_debug("PCI: Finishing adding to hotplug bus %04x:%02x\n", - pci_domain_nr(bus), bus->number); - - /* Allocate bus and devices resources */ - pcibios_allocate_bus_resources(bus); - pcibios_claim_one_bus(bus); - - /* Add new devices to global lists. Register in proc, sysfs. */ - pci_bus_add_devices(bus); - - /* Fixup EEH */ - /* eeh_add_device_tree_late(bus); */ -} -EXPORT_SYMBOL_GPL(pcibios_finish_adding_to_bus); - static void pcibios_setup_phb_resources(struct pci_controller *hose, struct list_head *resources) { diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index 225c95da23ce..3f9deec70b92 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -22,6 +22,11 @@ config MIPS select GENERIC_CPU_AUTOPROBE select GENERIC_IRQ_PROBE select GENERIC_IRQ_SHOW + select GENERIC_LIB_ASHLDI3 + select GENERIC_LIB_ASHRDI3 + select GENERIC_LIB_CMPDI2 + select GENERIC_LIB_LSHRDI3 + select GENERIC_LIB_UCMPDI2 select GENERIC_PCI_IOMAP select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC select GENERIC_SMP_IDLE_THREAD @@ -36,13 +41,11 @@ config MIPS select HAVE_ARCH_TRANSPARENT_HUGEPAGE if CPU_SUPPORTS_HUGEPAGES && 64BIT select HAVE_CBPF_JIT if (!64BIT && !CPU_MICROMIPS) select HAVE_EBPF_JIT if (64BIT && !CPU_MICROMIPS) - select HAVE_CC_STACKPROTECTOR select HAVE_CONTEXT_TRACKING select HAVE_COPY_THREAD_TLS select HAVE_C_RECORDMCOUNT select HAVE_DEBUG_KMEMLEAK select HAVE_DEBUG_STACKOVERFLOW - select HAVE_DMA_API_DEBUG select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_EXIT_THREAD @@ -62,6 +65,7 @@ config MIPS select HAVE_OPROFILE select HAVE_PERF_EVENTS select HAVE_REGS_AND_STACK_ACCESS_API + select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING_GEN if 64BIT || !SMP select IRQ_FORCED_THREADING @@ -132,7 +136,7 @@ config MIPS_GENERIC config MIPS_ALCHEMY bool "Alchemy processor based machines" - select ARCH_PHYS_ADDR_T_64BIT + select PHYS_ADDR_T_64BIT select CEVT_R4K select CSRC_R4K select IRQ_MIPS_CPU @@ -890,7 +894,7 @@ config CAVIUM_OCTEON_SOC bool "Cavium Networks Octeon SoC based boards" select CEVT_R4K select ARCH_HAS_PHYS_TO_DMA - select ARCH_PHYS_ADDR_T_64BIT + select PHYS_ADDR_T_64BIT select DMA_COHERENT select SYS_SUPPORTS_64BIT_KERNEL select SYS_SUPPORTS_BIG_ENDIAN @@ -912,6 +916,7 @@ config CAVIUM_OCTEON_SOC select MIPS_NR_CPU_NR_MAP_1024 select BUILTIN_DTB select MTD_COMPLEX_MAPPINGS + select SWIOTLB select SYS_SUPPORTS_RELOCATABLE help This option supports all of the Octeon reference boards from Cavium @@ -936,7 +941,7 @@ config NLM_XLR_BOARD select SWAP_IO_SPACE select SYS_SUPPORTS_32BIT_KERNEL select SYS_SUPPORTS_64BIT_KERNEL - select ARCH_PHYS_ADDR_T_64BIT + select PHYS_ADDR_T_64BIT select SYS_SUPPORTS_BIG_ENDIAN select SYS_SUPPORTS_HIGHMEM select DMA_COHERENT @@ -962,7 +967,7 @@ config NLM_XLP_BOARD select HW_HAS_PCI select SYS_SUPPORTS_32BIT_KERNEL select SYS_SUPPORTS_64BIT_KERNEL - select ARCH_PHYS_ADDR_T_64BIT + select PHYS_ADDR_T_64BIT select GPIOLIB select SYS_SUPPORTS_BIG_ENDIAN select SYS_SUPPORTS_LITTLE_ENDIAN @@ -1101,9 +1106,6 @@ config GPIO_TXX9 config FW_CFE bool -config ARCH_DMA_ADDR_T_64BIT - def_bool (HIGHMEM && ARCH_PHYS_ADDR_T_64BIT) || 64BIT - config ARCH_SUPPORTS_UPROBES bool @@ -1122,9 +1124,6 @@ config DMA_NONCOHERENT bool select NEED_DMA_MAP_STATE -config NEED_DMA_MAP_STATE - bool - config SYS_HAS_EARLY_PRINTK bool @@ -1373,6 +1372,7 @@ config CPU_LOONGSON3 select MIPS_PGD_C0_CONTEXT select MIPS_L1_CACHE_SHIFT_6 select GPIOLIB + select SWIOTLB help The Loongson 3 processor implements the MIPS64R2 instruction set with many extensions. @@ -1770,7 +1770,7 @@ config CPU_MIPS32_R5_XPA depends on SYS_SUPPORTS_HIGHMEM select XPA select HIGHMEM - select ARCH_PHYS_ADDR_T_64BIT + select PHYS_ADDR_T_64BIT default n help Choose this option if you want to enable the Extended Physical @@ -2402,9 +2402,6 @@ config SB1_PASS_2_1_WORKAROUNDS default y -config ARCH_PHYS_ADDR_T_64BIT - bool - choice prompt "SmartMIPS or microMIPS ASE support" @@ -2556,7 +2553,7 @@ config ARCH_DISCONTIGMEM_ENABLE Say Y to support efficient handling of discontiguous physical memory, for architectures which are either NUMA (Non-Uniform Memory Access) or have huge holes in the physical address space for other reasons. - See for more. + See for more. config ARCH_SPARSEMEM_ENABLE bool diff --git a/arch/mips/Makefile b/arch/mips/Makefile index 5e9fce076ab6..e2122cca4ae2 100644 --- a/arch/mips/Makefile +++ b/arch/mips/Makefile @@ -309,9 +309,6 @@ ifdef CONFIG_MIPS CHECKFLAGS += $(shell $(CC) $(KBUILD_CFLAGS) -dM -E -x c /dev/null | \ egrep -vw '__GNUC_(|MINOR_|PATCHLEVEL_)_' | \ sed -e "s/^\#define /-D'/" -e "s/ /'='/" -e "s/$$/'/" -e 's/\$$/&&/g') -ifdef CONFIG_64BIT -CHECKFLAGS += -m64 -endif endif OBJCOPYFLAGS += --remove-section=.reginfo diff --git a/arch/mips/alchemy/board-gpr.c b/arch/mips/alchemy/board-gpr.c index 4e79dbd54a33..fa75d75b5ba9 100644 --- a/arch/mips/alchemy/board-gpr.c +++ b/arch/mips/alchemy/board-gpr.c @@ -29,7 +29,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/arch/mips/alchemy/common/clock.c b/arch/mips/alchemy/common/clock.c index 6b6f6851df92..d129475fd40d 100644 --- a/arch/mips/alchemy/common/clock.c +++ b/arch/mips/alchemy/common/clock.c @@ -985,7 +985,7 @@ static int __init alchemy_clk_setup_imux(int ctype) return -ENODEV; } - a = kzalloc((sizeof(*a)) * 6, GFP_KERNEL); + a = kcalloc(6, sizeof(*a), GFP_KERNEL); if (!a) return -ENOMEM; diff --git a/arch/mips/alchemy/common/dbdma.c b/arch/mips/alchemy/common/dbdma.c index fc482d900ddd..4ca2c28878e0 100644 --- a/arch/mips/alchemy/common/dbdma.c +++ b/arch/mips/alchemy/common/dbdma.c @@ -411,8 +411,8 @@ u32 au1xxx_dbdma_ring_alloc(u32 chanid, int entries) * and if we try that first we are likely to not waste larger * slabs of memory. */ - desc_base = (u32)kmalloc(entries * sizeof(au1x_ddma_desc_t), - GFP_KERNEL|GFP_DMA); + desc_base = (u32)kmalloc_array(entries, sizeof(au1x_ddma_desc_t), + GFP_KERNEL|GFP_DMA); if (desc_base == 0) return 0; @@ -1050,7 +1050,7 @@ static int __init dbdma_setup(unsigned int irq, dbdev_tab_t *idtable) { int ret; - dbdev_tab = kzalloc(sizeof(dbdev_tab_t) * DBDEV_TAB_SIZE, GFP_KERNEL); + dbdev_tab = kcalloc(DBDEV_TAB_SIZE, sizeof(dbdev_tab_t), GFP_KERNEL); if (!dbdev_tab) return -ENOMEM; diff --git a/arch/mips/alchemy/common/platform.c b/arch/mips/alchemy/common/platform.c index d77a64f4c78b..1454d9f6ab2d 100644 --- a/arch/mips/alchemy/common/platform.c +++ b/arch/mips/alchemy/common/platform.c @@ -115,7 +115,7 @@ static void __init alchemy_setup_uarts(int ctype) uartclk = clk_get_rate(clk); clk_put(clk); - ports = kzalloc(s * (c + 1), GFP_KERNEL); + ports = kcalloc(s, (c + 1), GFP_KERNEL); if (!ports) { printk(KERN_INFO "Alchemy: no memory for UART data\n"); return; @@ -198,7 +198,7 @@ static unsigned long alchemy_ehci_data[][2] __initdata = { static int __init _new_usbres(struct resource **r, struct platform_device **d) { - *r = kzalloc(sizeof(struct resource) * 2, GFP_KERNEL); + *r = kcalloc(2, sizeof(struct resource), GFP_KERNEL); if (!*r) return -ENOMEM; *d = kzalloc(sizeof(struct platform_device), GFP_KERNEL); diff --git a/arch/mips/alchemy/devboards/platform.c b/arch/mips/alchemy/devboards/platform.c index 4640edab207c..203854ddd1bb 100644 --- a/arch/mips/alchemy/devboards/platform.c +++ b/arch/mips/alchemy/devboards/platform.c @@ -103,7 +103,7 @@ int __init db1x_register_pcmcia_socket(phys_addr_t pcmcia_attr_start, if (stschg_irq) cnt++; - sr = kzalloc(sizeof(struct resource) * cnt, GFP_KERNEL); + sr = kcalloc(cnt, sizeof(struct resource), GFP_KERNEL); if (!sr) return -ENOMEM; @@ -178,7 +178,7 @@ int __init db1x_register_norflash(unsigned long size, int width, return -EINVAL; ret = -ENOMEM; - parts = kzalloc(sizeof(struct mtd_partition) * 5, GFP_KERNEL); + parts = kcalloc(5, sizeof(struct mtd_partition), GFP_KERNEL); if (!parts) goto out; diff --git a/arch/mips/bcm47xx/board.c b/arch/mips/bcm47xx/board.c index edfaef0d73a4..a80910d2738c 100644 --- a/arch/mips/bcm47xx/board.c +++ b/arch/mips/bcm47xx/board.c @@ -172,6 +172,8 @@ struct bcm47xx_board_type_list1 bcm47xx_board_list_board_id[] __initconst = { {{BCM47XX_BOARD_NETGEAR_WNDR4000, "Netgear WNDR4000"}, "U12H181T00_NETGEAR"}, {{BCM47XX_BOARD_NETGEAR_WNDR4500V1, "Netgear WNDR4500 V1"}, "U12H189T00_NETGEAR"}, {{BCM47XX_BOARD_NETGEAR_WNDR4500V2, "Netgear WNDR4500 V2"}, "U12H224T00_NETGEAR"}, + {{BCM47XX_BOARD_NETGEAR_WNR1000_V3, "Netgear WNR1000 V3"}, "U12H139T00_NETGEAR"}, + {{BCM47XX_BOARD_NETGEAR_WNR1000_V3, "Netgear WNR1000 V3"}, "U12H139T50_NETGEAR"}, {{BCM47XX_BOARD_NETGEAR_WNR2000, "Netgear WNR2000"}, "U12H114T00_NETGEAR"}, {{BCM47XX_BOARD_NETGEAR_WNR3500L, "Netgear WNR3500L"}, "U12H136T99_NETGEAR"}, {{BCM47XX_BOARD_NETGEAR_WNR3500U, "Netgear WNR3500U"}, "U12H136T00_NETGEAR"}, diff --git a/arch/mips/bcm47xx/buttons.c b/arch/mips/bcm47xx/buttons.c index 88d400d256c4..977990a609ba 100644 --- a/arch/mips/bcm47xx/buttons.c +++ b/arch/mips/bcm47xx/buttons.c @@ -411,6 +411,12 @@ bcm47xx_buttons_netgear_wndr4500v1[] __initconst = { BCM47XX_GPIO_KEY(6, KEY_RESTART), }; +static const struct gpio_keys_button +bcm47xx_buttons_netgear_wnr1000_v3[] __initconst = { + BCM47XX_GPIO_KEY(2, KEY_WPS_BUTTON), + BCM47XX_GPIO_KEY(3, KEY_RESTART), +}; + static const struct gpio_keys_button bcm47xx_buttons_netgear_wnr3500lv1[] __initconst = { BCM47XX_GPIO_KEY(4, KEY_RESTART), @@ -670,6 +676,9 @@ int __init bcm47xx_buttons_register(void) case BCM47XX_BOARD_NETGEAR_WNDR4500V1: err = bcm47xx_copy_bdata(bcm47xx_buttons_netgear_wndr4500v1); break; + case BCM47XX_BOARD_NETGEAR_WNR1000_V3: + err = bcm47xx_copy_bdata(bcm47xx_buttons_netgear_wnr1000_v3); + break; case BCM47XX_BOARD_NETGEAR_WNR3500L: err = bcm47xx_copy_bdata(bcm47xx_buttons_netgear_wnr3500lv1); break; diff --git a/arch/mips/bcm47xx/leds.c b/arch/mips/bcm47xx/leds.c index 34a7b3fbdfd9..d85fcdac8bf0 100644 --- a/arch/mips/bcm47xx/leds.c +++ b/arch/mips/bcm47xx/leds.c @@ -497,6 +497,12 @@ bcm47xx_leds_netgear_wndr4500v1[] __initconst = { BCM47XX_GPIO_LED(14, "green", "usb2", 1, LEDS_GPIO_DEFSTATE_OFF), }; +static const struct gpio_led +bcm47xx_leds_netgear_wnr1000_v3[] __initconst = { + BCM47XX_GPIO_LED(0, "blue", "wlan", 0, LEDS_GPIO_DEFSTATE_OFF), + BCM47XX_GPIO_LED(1, "green", "wps", 0, LEDS_GPIO_DEFSTATE_OFF), +}; + static const struct gpio_led bcm47xx_leds_netgear_wnr3500lv1[] __initconst = { BCM47XX_GPIO_LED(0, "blue", "wlan", 1, LEDS_GPIO_DEFSTATE_OFF), @@ -532,7 +538,7 @@ bcm47xx_leds_simpletech_simpleshare[] __initconst = { * Init **************************************************/ -static struct gpio_led_platform_data bcm47xx_leds_pdata; +static struct gpio_led_platform_data bcm47xx_leds_pdata __initdata; #define bcm47xx_set_pdata(dev_leds) do { \ bcm47xx_leds_pdata.leds = dev_leds; \ @@ -758,6 +764,9 @@ void __init bcm47xx_leds_register(void) case BCM47XX_BOARD_NETGEAR_WNDR4500V1: bcm47xx_set_pdata(bcm47xx_leds_netgear_wndr4500v1); break; + case BCM47XX_BOARD_NETGEAR_WNR1000_V3: + bcm47xx_set_pdata(bcm47xx_leds_netgear_wnr1000_v3); + break; case BCM47XX_BOARD_NETGEAR_WNR3500L: bcm47xx_set_pdata(bcm47xx_leds_netgear_wnr3500lv1); break; diff --git a/arch/mips/bmips/dma.c b/arch/mips/bmips/dma.c index 04790f4e1805..6dec30842b2f 100644 --- a/arch/mips/bmips/dma.c +++ b/arch/mips/bmips/dma.c @@ -94,7 +94,7 @@ static int __init bmips_init_dma_ranges(void) goto out_bad; /* add a dummy (zero) entry at the end as a sentinel */ - bmips_dma_ranges = kzalloc(sizeof(struct bmips_dma_range) * (len + 1), + bmips_dma_ranges = kcalloc(len + 1, sizeof(struct bmips_dma_range), GFP_KERNEL); if (!bmips_dma_ranges) goto out_bad; diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile index adce180f3ee4..abe77add8789 100644 --- a/arch/mips/boot/compressed/Makefile +++ b/arch/mips/boot/compressed/Makefile @@ -46,10 +46,13 @@ $(obj)/uart-ath79.c: $(srctree)/arch/mips/ath79/early_printk.c vmlinuzobjs-$(CONFIG_KERNEL_XZ) += $(obj)/ashldi3.o $(obj)/bswapsi.o -extra-y += ashldi3.c bswapsi.c -$(obj)/ashldi3.o $(obj)/bswapsi.o: KBUILD_CFLAGS += -I$(srctree)/arch/mips/lib -$(obj)/ashldi3.c $(obj)/bswapsi.c: $(obj)/%.c: $(srctree)/arch/mips/lib/%.c - $(call cmd,shipped) +extra-y += ashldi3.c +$(obj)/ashldi3.c: $(obj)/%.c: $(srctree)/lib/%.c FORCE + $(call if_changed,shipped) + +extra-y += bswapsi.c +$(obj)/bswapsi.c: $(obj)/%.c: $(srctree)/arch/mips/lib/%.c FORCE + $(call if_changed,shipped) targets := $(notdir $(vmlinuzobjs-y)) diff --git a/arch/mips/boot/dts/brcm/Makefile b/arch/mips/boot/dts/brcm/Makefile index d8787c9a499e..d85f446cc0ce 100644 --- a/arch/mips/boot/dts/brcm/Makefile +++ b/arch/mips/boot/dts/brcm/Makefile @@ -34,4 +34,4 @@ dtb-$(CONFIG_DT_NONE) += \ bcm97425svmb.dtb \ bcm97435svmb.dtb -obj-y += $(patsubst %.dtb, %.dtb.o, $(dtb-y)) +obj-$(CONFIG_BUILTIN_DTB) += $(addsuffix .o, $(dtb-y)) diff --git a/arch/mips/boot/dts/cavium-octeon/Makefile b/arch/mips/boot/dts/cavium-octeon/Makefile index 24a8efcd7b03..17aef35f311b 100644 --- a/arch/mips/boot/dts/cavium-octeon/Makefile +++ b/arch/mips/boot/dts/cavium-octeon/Makefile @@ -1,4 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 dtb-$(CONFIG_CAVIUM_OCTEON_SOC) += octeon_3xxx.dtb octeon_68xx.dtb -obj-y += $(patsubst %.dtb, %.dtb.o, $(dtb-y)) +obj-$(CONFIG_BUILTIN_DTB) += $(addsuffix .o, $(dtb-y)) diff --git a/arch/mips/boot/dts/ingenic/Makefile b/arch/mips/boot/dts/ingenic/Makefile index 5b1361a89e02..9cc48441eb71 100644 --- a/arch/mips/boot/dts/ingenic/Makefile +++ b/arch/mips/boot/dts/ingenic/Makefile @@ -3,4 +3,4 @@ dtb-$(CONFIG_JZ4740_QI_LB60) += qi_lb60.dtb dtb-$(CONFIG_JZ4770_GCW0) += gcw0.dtb dtb-$(CONFIG_JZ4780_CI20) += ci20.dtb -obj-y += $(patsubst %.dtb, %.dtb.o, $(dtb-y)) +obj-$(CONFIG_BUILTIN_DTB) += $(addsuffix .o, $(dtb-y)) diff --git a/arch/mips/boot/dts/ingenic/ci20.dts b/arch/mips/boot/dts/ingenic/ci20.dts index 38078594cf97..50cff3cbcc6d 100644 --- a/arch/mips/boot/dts/ingenic/ci20.dts +++ b/arch/mips/boot/dts/ingenic/ci20.dts @@ -36,6 +36,28 @@ clock-frequency = <48000000>; }; +&mmc0 { + status = "okay"; + + bus-width = <4>; + max-frequency = <50000000>; + + pinctrl-names = "default"; + pinctrl-0 = <&pins_mmc0>; + + cd-gpios = <&gpf 20 GPIO_ACTIVE_LOW>; +}; + +&mmc1 { + status = "okay"; + + bus-width = <4>; + max-frequency = <50000000>; + + pinctrl-names = "default"; + pinctrl-0 = <&pins_mmc1>; +}; + &uart0 { status = "okay"; @@ -203,4 +225,16 @@ groups = "nemc-cs6"; bias-disable; }; + + pins_mmc0: mmc0 { + function = "mmc0"; + groups = "mmc0-1bit-e", "mmc0-4bit-e"; + bias-disable; + }; + + pins_mmc1: mmc1 { + function = "mmc1"; + groups = "mmc1-1bit-d", "mmc1-4bit-d"; + bias-disable; + }; }; diff --git a/arch/mips/boot/dts/ingenic/jz4740.dtsi b/arch/mips/boot/dts/ingenic/jz4740.dtsi index cd5185bb90ae..26c6b561d6f7 100644 --- a/arch/mips/boot/dts/ingenic/jz4740.dtsi +++ b/arch/mips/boot/dts/ingenic/jz4740.dtsi @@ -45,6 +45,14 @@ #clock-cells = <1>; }; + watchdog: watchdog@10002000 { + compatible = "ingenic,jz4740-watchdog"; + reg = <0x10002000 0x10>; + + clocks = <&cgu JZ4740_CLK_RTC>; + clock-names = "rtc"; + }; + rtc_dev: rtc@10003000 { compatible = "ingenic,jz4740-rtc"; reg = <0x10003000 0x40>; diff --git a/arch/mips/boot/dts/ingenic/jz4780.dtsi b/arch/mips/boot/dts/ingenic/jz4780.dtsi index 9b5794667aee..aa4e8f75ff5d 100644 --- a/arch/mips/boot/dts/ingenic/jz4780.dtsi +++ b/arch/mips/boot/dts/ingenic/jz4780.dtsi @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include / { #address-cells = <1>; @@ -221,7 +222,10 @@ watchdog: watchdog@10002000 { compatible = "ingenic,jz4780-watchdog"; - reg = <0x10002000 0x100>; + reg = <0x10002000 0x10>; + + clocks = <&cgu JZ4780_CLK_RTCLK>; + clock-names = "rtc"; }; nemc: nemc@13410000 { @@ -241,6 +245,57 @@ status = "disabled"; }; + dma: dma@13420000 { + compatible = "ingenic,jz4780-dma"; + reg = <0x13420000 0x10000>; + #dma-cells = <2>; + + interrupt-parent = <&intc>; + interrupts = <10>; + + clocks = <&cgu JZ4780_CLK_PDMA>; + }; + + mmc0: mmc@13450000 { + compatible = "ingenic,jz4780-mmc"; + reg = <0x13450000 0x1000>; + + interrupt-parent = <&intc>; + interrupts = <37>; + + clocks = <&cgu JZ4780_CLK_MSC0>; + clock-names = "mmc"; + + cap-sd-highspeed; + cap-mmc-highspeed; + cap-sdio-irq; + dmas = <&dma JZ4780_DMA_MSC0_RX 0xffffffff>, + <&dma JZ4780_DMA_MSC0_TX 0xffffffff>; + dma-names = "rx", "tx"; + + status = "disabled"; + }; + + mmc1: mmc@13460000 { + compatible = "ingenic,jz4780-mmc"; + reg = <0x13460000 0x1000>; + + interrupt-parent = <&intc>; + interrupts = <36>; + + clocks = <&cgu JZ4780_CLK_MSC1>; + clock-names = "mmc"; + + cap-sd-highspeed; + cap-mmc-highspeed; + cap-sdio-irq; + dmas = <&dma JZ4780_DMA_MSC1_RX 0xffffffff>, + <&dma JZ4780_DMA_MSC1_TX 0xffffffff>; + dma-names = "rx", "tx"; + + status = "disabled"; + }; + bch: bch@134d0000 { compatible = "ingenic,jz4780-bch"; reg = <0x134d0000 0x10000>; diff --git a/arch/mips/boot/dts/lantiq/Makefile b/arch/mips/boot/dts/lantiq/Makefile index 51ab9c1dff42..f5dfc06242b9 100644 --- a/arch/mips/boot/dts/lantiq/Makefile +++ b/arch/mips/boot/dts/lantiq/Makefile @@ -1,4 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 dtb-$(CONFIG_DT_EASY50712) += easy50712.dtb -obj-y += $(patsubst %.dtb, %.dtb.o, $(dtb-y)) +obj-$(CONFIG_BUILTIN_DTB) += $(addsuffix .o, $(dtb-y)) diff --git a/arch/mips/boot/dts/mscc/Makefile b/arch/mips/boot/dts/mscc/Makefile index c51164537c02..3c6aed9f5439 100644 --- a/arch/mips/boot/dts/mscc/Makefile +++ b/arch/mips/boot/dts/mscc/Makefile @@ -1,3 +1,3 @@ dtb-$(CONFIG_LEGACY_BOARD_OCELOT) += ocelot_pcb123.dtb -obj-y += $(patsubst %.dtb, %.dtb.o, $(dtb-y)) +obj-$(CONFIG_BUILTIN_DTB) += $(addsuffix .o, $(dtb-y)) diff --git a/arch/mips/boot/dts/mscc/ocelot.dtsi b/arch/mips/boot/dts/mscc/ocelot.dtsi index dd239cab2f9d..4f33dbc67348 100644 --- a/arch/mips/boot/dts/mscc/ocelot.dtsi +++ b/arch/mips/boot/dts/mscc/ocelot.dtsi @@ -91,6 +91,72 @@ status = "disabled"; }; + switch@1010000 { + compatible = "mscc,vsc7514-switch"; + reg = <0x1010000 0x10000>, + <0x1030000 0x10000>, + <0x1080000 0x100>, + <0x10d0000 0x10000>, + <0x11e0000 0x100>, + <0x11f0000 0x100>, + <0x1200000 0x100>, + <0x1210000 0x100>, + <0x1220000 0x100>, + <0x1230000 0x100>, + <0x1240000 0x100>, + <0x1250000 0x100>, + <0x1260000 0x100>, + <0x1270000 0x100>, + <0x1280000 0x100>, + <0x1800000 0x80000>, + <0x1880000 0x10000>; + reg-names = "sys", "rew", "qs", "hsio", "port0", + "port1", "port2", "port3", "port4", "port5", + "port6", "port7", "port8", "port9", "port10", + "qsys", "ana"; + interrupts = <21 22>; + interrupt-names = "xtr", "inj"; + + ethernet-ports { + #address-cells = <1>; + #size-cells = <0>; + + port0: port@0 { + reg = <0>; + }; + port1: port@1 { + reg = <1>; + }; + port2: port@2 { + reg = <2>; + }; + port3: port@3 { + reg = <3>; + }; + port4: port@4 { + reg = <4>; + }; + port5: port@5 { + reg = <5>; + }; + port6: port@6 { + reg = <6>; + }; + port7: port@7 { + reg = <7>; + }; + port8: port@8 { + reg = <8>; + }; + port9: port@9 { + reg = <9>; + }; + port10: port@10 { + reg = <10>; + }; + }; + }; + reset@1070008 { compatible = "mscc,ocelot-chip-reset"; reg = <0x1070008 0x4>; @@ -113,5 +179,27 @@ function = "uart2"; }; }; + + mdio0: mdio@107009c { + #address-cells = <1>; + #size-cells = <0>; + compatible = "mscc,ocelot-miim"; + reg = <0x107009c 0x36>, <0x10700f0 0x8>; + interrupts = <14>; + status = "disabled"; + + phy0: ethernet-phy@0 { + reg = <0>; + }; + phy1: ethernet-phy@1 { + reg = <1>; + }; + phy2: ethernet-phy@2 { + reg = <2>; + }; + phy3: ethernet-phy@3 { + reg = <3>; + }; + }; }; }; diff --git a/arch/mips/boot/dts/mscc/ocelot_pcb123.dts b/arch/mips/boot/dts/mscc/ocelot_pcb123.dts index 29d6414f8886..4ccd65379059 100644 --- a/arch/mips/boot/dts/mscc/ocelot_pcb123.dts +++ b/arch/mips/boot/dts/mscc/ocelot_pcb123.dts @@ -25,3 +25,23 @@ &uart2 { status = "okay"; }; + +&mdio0 { + status = "okay"; +}; + +&port0 { + phy-handle = <&phy0>; +}; + +&port1 { + phy-handle = <&phy1>; +}; + +&port2 { + phy-handle = <&phy2>; +}; + +&port3 { + phy-handle = <&phy3>; +}; diff --git a/arch/mips/boot/dts/mti/Makefile b/arch/mips/boot/dts/mti/Makefile index 3508720cb6d9..b5f7426998b1 100644 --- a/arch/mips/boot/dts/mti/Makefile +++ b/arch/mips/boot/dts/mti/Makefile @@ -2,4 +2,4 @@ dtb-$(CONFIG_MIPS_MALTA) += malta.dtb dtb-$(CONFIG_LEGACY_BOARD_SEAD3) += sead3.dtb -obj-y += $(patsubst %.dtb, %.dtb.o, $(dtb-y)) +obj-$(CONFIG_BUILTIN_DTB) += $(addsuffix .o, $(dtb-y)) diff --git a/arch/mips/boot/dts/netlogic/Makefile b/arch/mips/boot/dts/netlogic/Makefile index d630b27950f0..45af4224494f 100644 --- a/arch/mips/boot/dts/netlogic/Makefile +++ b/arch/mips/boot/dts/netlogic/Makefile @@ -5,4 +5,4 @@ dtb-$(CONFIG_DT_XLP_FVP) += xlp_fvp.dtb dtb-$(CONFIG_DT_XLP_GVP) += xlp_gvp.dtb dtb-$(CONFIG_DT_XLP_RVP) += xlp_rvp.dtb -obj-y += $(patsubst %.dtb, %.dtb.o, $(dtb-y)) +obj-$(CONFIG_BUILTIN_DTB) += $(addsuffix .o, $(dtb-y)) diff --git a/arch/mips/boot/dts/pic32/Makefile b/arch/mips/boot/dts/pic32/Makefile index ba9bcef8fde9..fb57f36324db 100644 --- a/arch/mips/boot/dts/pic32/Makefile +++ b/arch/mips/boot/dts/pic32/Makefile @@ -4,4 +4,4 @@ dtb-$(CONFIG_DTB_PIC32_MZDA_SK) += pic32mzda_sk.dtb dtb-$(CONFIG_DTB_PIC32_NONE) += \ pic32mzda_sk.dtb -obj-y += $(patsubst %.dtb, %.dtb.o, $(dtb-y)) +obj-$(CONFIG_BUILTIN_DTB) += $(addsuffix .o, $(dtb-y)) diff --git a/arch/mips/boot/dts/ralink/Makefile b/arch/mips/boot/dts/ralink/Makefile index 94bee5b38b53..6c26dfa0a903 100644 --- a/arch/mips/boot/dts/ralink/Makefile +++ b/arch/mips/boot/dts/ralink/Makefile @@ -6,4 +6,4 @@ dtb-$(CONFIG_DTB_MT7620A_EVAL) += mt7620a_eval.dtb dtb-$(CONFIG_DTB_OMEGA2P) += omega2p.dtb dtb-$(CONFIG_DTB_VOCORE2) += vocore2.dtb -obj-y += $(patsubst %.dtb, %.dtb.o, $(dtb-y)) +obj-$(CONFIG_BUILTIN_DTB) += $(addsuffix .o, $(dtb-y)) diff --git a/arch/mips/cavium-octeon/Kconfig b/arch/mips/cavium-octeon/Kconfig index b5eee1a57d6c..4984e462be30 100644 --- a/arch/mips/cavium-octeon/Kconfig +++ b/arch/mips/cavium-octeon/Kconfig @@ -67,18 +67,6 @@ config CAVIUM_OCTEON_LOCK_L2_MEMCPY help Lock the kernel's implementation of memcpy() into L2. -config IOMMU_HELPER - bool - -config NEED_SG_DMA_LENGTH - bool - -config SWIOTLB - def_bool y - select DMA_DIRECT_OPS - select IOMMU_HELPER - select NEED_SG_DMA_LENGTH - config OCTEON_ILM tristate "Module to measure interrupt latency using Octeon CIU Timer" help diff --git a/arch/mips/configs/ci20_defconfig b/arch/mips/configs/ci20_defconfig index b5f4ad8f2c45..be23fd25eeaa 100644 --- a/arch/mips/configs/ci20_defconfig +++ b/arch/mips/configs/ci20_defconfig @@ -104,10 +104,14 @@ CONFIG_REGULATOR_FIXED_VOLTAGE=y # CONFIG_HID is not set # CONFIG_USB_SUPPORT is not set CONFIG_MMC=y +CONFIG_MMC_JZ4740=y CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_JZ4740=y +CONFIG_DMADEVICES=y +CONFIG_DMA_JZ4780=y # CONFIG_IOMMU_SUPPORT is not set CONFIG_MEMORY=y +CONFIG_EXT4_FS=y # CONFIG_DNOTIFY is not set CONFIG_PROC_KCORE=y # CONFIG_PROC_PAGE_MONITOR is not set diff --git a/arch/mips/configs/qi_lb60_defconfig b/arch/mips/configs/qi_lb60_defconfig index 3b02ff9a7c64..d8b7211a7b0f 100644 --- a/arch/mips/configs/qi_lb60_defconfig +++ b/arch/mips/configs/qi_lb60_defconfig @@ -72,6 +72,8 @@ CONFIG_POWER_SUPPLY=y CONFIG_BATTERY_JZ4740=y CONFIG_CHARGER_GPIO=y # CONFIG_HWMON is not set +CONFIG_WATCHDOG=y +CONFIG_JZ4740_WDT=y CONFIG_MFD_JZ4740_ADC=y CONFIG_REGULATOR=y CONFIG_REGULATOR_FIXED_VOLTAGE=y diff --git a/arch/mips/dec/time.c b/arch/mips/dec/time.c index a2a150e4fbc2..c38686f89a18 100644 --- a/arch/mips/dec/time.c +++ b/arch/mips/dec/time.c @@ -19,7 +19,7 @@ #include #include -void read_persistent_clock(struct timespec *ts) +void read_persistent_clock64(struct timespec64 *ts) { unsigned int year, mon, day, hour, min, sec, real_year; unsigned long flags; @@ -54,19 +54,20 @@ void read_persistent_clock(struct timespec *ts) year += real_year - 72 + 2000; - ts->tv_sec = mktime(year, mon, day, hour, min, sec); + ts->tv_sec = mktime64(year, mon, day, hour, min, sec); ts->tv_nsec = 0; } /* - * In order to set the CMOS clock precisely, rtc_mips_set_mmss has to + * In order to set the CMOS clock precisely, update_persistent_clock64 has to * be called 500 ms after the second nowtime has started, because when * nowtime is written into the registers of the CMOS clock, it will * jump to the next second precisely 500 ms later. Check the Dallas * DS1287 data sheet for details. */ -int rtc_mips_set_mmss(unsigned long nowtime) +int update_persistent_clock64(struct timespec64 now) { + time64_t nowtime = now.tv_sec; int retval = 0; int real_seconds, real_minutes, cmos_minutes; unsigned char save_control, save_freq_select; @@ -91,8 +92,7 @@ int rtc_mips_set_mmss(unsigned long nowtime) * messing with unknown time zones but requires your * RTC not to be off by more than 15 minutes */ - real_seconds = nowtime % 60; - real_minutes = nowtime / 60; + real_minutes = div_s64_rem(nowtime, 60, &real_seconds); if (((abs(real_minutes - cmos_minutes) + 15) / 30) & 1) real_minutes += 30; /* correct for half hour time zone */ real_minutes %= 60; diff --git a/arch/mips/include/asm/compat.h b/arch/mips/include/asm/compat.h index 9a0fa66b81ac..78675f19440f 100644 --- a/arch/mips/include/asm/compat.h +++ b/arch/mips/include/asm/compat.h @@ -14,7 +14,6 @@ typedef u32 compat_size_t; typedef s32 compat_ssize_t; -typedef s32 compat_time_t; typedef s32 compat_clock_t; typedef s32 compat_suseconds_t; @@ -38,24 +37,16 @@ typedef struct { typedef s32 compat_timer_t; typedef s32 compat_key_t; +typedef s16 compat_short_t; typedef s32 compat_int_t; typedef s32 compat_long_t; typedef s64 compat_s64; +typedef u16 compat_ushort_t; typedef u32 compat_uint_t; typedef u32 compat_ulong_t; typedef u64 compat_u64; typedef u32 compat_uptr_t; -struct compat_timespec { - compat_time_t tv_sec; - s32 tv_nsec; -}; - -struct compat_timeval { - compat_time_t tv_sec; - s32 tv_usec; -}; - struct compat_stat { compat_dev_t st_dev; s32 st_pad1[3]; @@ -168,35 +159,35 @@ struct compat_ipc64_perm { struct compat_semid64_ds { struct compat_ipc64_perm sem_perm; - compat_time_t sem_otime; - compat_time_t sem_ctime; + compat_ulong_t sem_otime; + compat_ulong_t sem_ctime; compat_ulong_t sem_nsems; - compat_ulong_t __unused1; - compat_ulong_t __unused2; + compat_ulong_t sem_otime_high; + compat_ulong_t sem_ctime_high; }; struct compat_msqid64_ds { struct compat_ipc64_perm msg_perm; #ifndef CONFIG_CPU_LITTLE_ENDIAN - compat_ulong_t __unused1; + compat_ulong_t msg_stime_high; #endif - compat_time_t msg_stime; + compat_ulong_t msg_stime; #ifdef CONFIG_CPU_LITTLE_ENDIAN - compat_ulong_t __unused1; + compat_ulong_t msg_stime_high; #endif #ifndef CONFIG_CPU_LITTLE_ENDIAN - compat_ulong_t __unused2; + compat_ulong_t msg_rtime_high; #endif - compat_time_t msg_rtime; + compat_ulong_t msg_rtime; #ifdef CONFIG_CPU_LITTLE_ENDIAN - compat_ulong_t __unused2; + compat_ulong_t msg_rtime_high; #endif #ifndef CONFIG_CPU_LITTLE_ENDIAN - compat_ulong_t __unused3; + compat_ulong_t msg_ctime_high; #endif - compat_time_t msg_ctime; + compat_ulong_t msg_ctime; #ifdef CONFIG_CPU_LITTLE_ENDIAN - compat_ulong_t __unused3; + compat_ulong_t msg_ctime_high; #endif compat_ulong_t msg_cbytes; compat_ulong_t msg_qnum; @@ -210,14 +201,16 @@ struct compat_msqid64_ds { struct compat_shmid64_ds { struct compat_ipc64_perm shm_perm; compat_size_t shm_segsz; - compat_time_t shm_atime; - compat_time_t shm_dtime; - compat_time_t shm_ctime; + compat_ulong_t shm_atime; + compat_ulong_t shm_dtime; + compat_ulong_t shm_ctime; compat_pid_t shm_cpid; compat_pid_t shm_lpid; compat_ulong_t shm_nattch; - compat_ulong_t __unused1; - compat_ulong_t __unused2; + compat_ushort_t shm_atime_high; + compat_ushort_t shm_dtime_high; + compat_ushort_t shm_ctime_high; + compat_ushort_t __unused2; }; /* MIPS has unusual order of fields in stack_t */ diff --git a/arch/mips/include/asm/cpu-features.h b/arch/mips/include/asm/cpu-features.h index 5f74590e0bea..9cdb4e4ce258 100644 --- a/arch/mips/include/asm/cpu-features.h +++ b/arch/mips/include/asm/cpu-features.h @@ -535,6 +535,13 @@ # define cpu_has_shared_ftlb_entries 0 #endif +#ifdef CONFIG_MIPS_MT_SMP +# define cpu_has_mipsmt_pertccounters \ + (cpu_data[0].options & MIPS_CPU_MT_PER_TC_PERF_COUNTERS) +#else +# define cpu_has_mipsmt_pertccounters 0 +#endif /* CONFIG_MIPS_MT_SMP */ + /* * Guest capabilities */ diff --git a/arch/mips/include/asm/cpu.h b/arch/mips/include/asm/cpu.h index d39324c4adf1..5b9d02ef4f60 100644 --- a/arch/mips/include/asm/cpu.h +++ b/arch/mips/include/asm/cpu.h @@ -418,6 +418,8 @@ enum cpu_type_enum { MBIT_ULL(54) /* CPU shares FTLB RAM with another */ #define MIPS_CPU_SHARED_FTLB_ENTRIES \ MBIT_ULL(55) /* CPU shares FTLB entries with another */ +#define MIPS_CPU_MT_PER_TC_PERF_COUNTERS \ + MBIT_ULL(56) /* CPU has perf counters implemented per TC (MIPSMT ASE) */ /* * CPU ASE encodings diff --git a/arch/mips/include/asm/mach-bcm47xx/bcm47xx_board.h b/arch/mips/include/asm/mach-bcm47xx/bcm47xx_board.h index cbf9da7f2f94..0ef8893e07f8 100644 --- a/arch/mips/include/asm/mach-bcm47xx/bcm47xx_board.h +++ b/arch/mips/include/asm/mach-bcm47xx/bcm47xx_board.h @@ -110,6 +110,7 @@ enum bcm47xx_board { BCM47XX_BOARD_NETGEAR_WNDR4000, BCM47XX_BOARD_NETGEAR_WNDR4500V1, BCM47XX_BOARD_NETGEAR_WNDR4500V2, + BCM47XX_BOARD_NETGEAR_WNR1000_V3, BCM47XX_BOARD_NETGEAR_WNR2000, BCM47XX_BOARD_NETGEAR_WNR3500L, BCM47XX_BOARD_NETGEAR_WNR3500U, diff --git a/arch/mips/include/asm/mach-jz4740/platform.h b/arch/mips/include/asm/mach-jz4740/platform.h index 3645974b7f65..c0c932ac72a7 100644 --- a/arch/mips/include/asm/mach-jz4740/platform.h +++ b/arch/mips/include/asm/mach-jz4740/platform.h @@ -29,7 +29,6 @@ extern struct platform_device jz4740_i2s_device; extern struct platform_device jz4740_pcm_device; extern struct platform_device jz4740_codec_device; extern struct platform_device jz4740_adc_device; -extern struct platform_device jz4740_wdt_device; extern struct platform_device jz4740_pwm_device; extern struct platform_device jz4740_dma_device; diff --git a/arch/mips/include/asm/mc146818-time.h b/arch/mips/include/asm/mc146818-time.h index 9e1ad26abdc0..cbf5cec345f1 100644 --- a/arch/mips/include/asm/mc146818-time.h +++ b/arch/mips/include/asm/mc146818-time.h @@ -86,7 +86,7 @@ static inline int mc146818_set_rtc_mmss(unsigned long nowtime) return retval; } -static inline unsigned long mc146818_get_cmos_time(void) +static inline time64_t mc146818_get_cmos_time(void) { unsigned int year, mon, day, hour, min, sec; unsigned long flags; @@ -113,7 +113,7 @@ static inline unsigned long mc146818_get_cmos_time(void) spin_unlock_irqrestore(&rtc_lock, flags); year = mc146818_decode_year(year); - return mktime(year, mon, day, hour, min, sec); + return mktime64(year, mon, day, hour, min, sec); } #endif /* __ASM_MC146818_TIME_H */ diff --git a/arch/mips/include/asm/mipsregs.h b/arch/mips/include/asm/mipsregs.h index f65859784a4c..ae461d91cd1f 100644 --- a/arch/mips/include/asm/mipsregs.h +++ b/arch/mips/include/asm/mipsregs.h @@ -685,6 +685,11 @@ #define MIPS_CONF7_IAR (_ULCAST_(1) << 10) #define MIPS_CONF7_AR (_ULCAST_(1) << 16) +/* Config7 Bits specific to MIPS Technologies. */ + +/* Performance counters implemented Per TC */ +#define MTI_CONF7_PTC (_ULCAST_(1) << 19) + /* WatchLo* register definitions */ #define MIPS_WATCHLO_IRW (_ULCAST_(0x7) << 0) diff --git a/arch/mips/include/asm/pci.h b/arch/mips/include/asm/pci.h index 2339f42f047a..436099883022 100644 --- a/arch/mips/include/asm/pci.h +++ b/arch/mips/include/asm/pci.h @@ -121,13 +121,6 @@ extern unsigned long PCIBIOS_MIN_MEM; #include #include -/* - * The PCI address space does equal the physical memory address space. - * The networking and block device layers use this boolean for bounce - * buffer decisions. - */ -#define PCI_DMA_BUS_IS_PHYS (1) - #ifdef CONFIG_PCI_DOMAINS_GENERIC static inline int pci_proc_domain(struct pci_bus *bus) { diff --git a/arch/mips/include/asm/time.h b/arch/mips/include/asm/time.h index 17d4cd20f18c..b85ec64ee7e9 100644 --- a/arch/mips/include/asm/time.h +++ b/arch/mips/include/asm/time.h @@ -21,15 +21,6 @@ extern spinlock_t rtc_lock; -/* - * RTC ops. By default, they point to weak no-op RTC functions. - * rtc_mips_set_time - reverse the above translation and set time to RTC. - * rtc_mips_set_mmss - similar to rtc_set_time, but only min and sec need - * to be set. Used by RTC sync-up. - */ -extern int rtc_mips_set_time(unsigned long); -extern int rtc_mips_set_mmss(unsigned long); - /* * board specific routines required by time_init(). */ diff --git a/arch/mips/include/uapi/asm/msgbuf.h b/arch/mips/include/uapi/asm/msgbuf.h index eb4d0f9d7364..46aa15b13e4e 100644 --- a/arch/mips/include/uapi/asm/msgbuf.h +++ b/arch/mips/include/uapi/asm/msgbuf.h @@ -9,33 +9,15 @@ * between kernel and user space. * * Pad space is left for: - * - extension of time_t to 64-bit on 32-bitsystem to solve the y2038 problem * - 2 miscellaneous unsigned long values */ +#if defined(__mips64) struct msqid64_ds { struct ipc64_perm msg_perm; -#if !defined(__mips64) && defined(__MIPSEB__) - unsigned long __unused1; -#endif __kernel_time_t msg_stime; /* last msgsnd time */ -#if !defined(__mips64) && defined(__MIPSEL__) - unsigned long __unused1; -#endif -#if !defined(__mips64) && defined(__MIPSEB__) - unsigned long __unused2; -#endif __kernel_time_t msg_rtime; /* last msgrcv time */ -#if !defined(__mips64) && defined(__MIPSEL__) - unsigned long __unused2; -#endif -#if !defined(__mips64) && defined(__MIPSEB__) - unsigned long __unused3; -#endif __kernel_time_t msg_ctime; /* last change time */ -#if !defined(__mips64) && defined(__MIPSEL__) - unsigned long __unused3; -#endif unsigned long msg_cbytes; /* current number of bytes on queue */ unsigned long msg_qnum; /* number of messages in queue */ unsigned long msg_qbytes; /* max number of bytes on queue */ @@ -44,5 +26,42 @@ struct msqid64_ds { unsigned long __unused4; unsigned long __unused5; }; +#elif defined (__MIPSEB__) +struct msqid64_ds { + struct ipc64_perm msg_perm; + unsigned long msg_stime_high; + unsigned long msg_stime; /* last msgsnd time */ + unsigned long msg_rtime_high; + unsigned long msg_rtime; /* last msgrcv time */ + unsigned long msg_ctime_high; + unsigned long msg_ctime; /* last change time */ + unsigned long msg_cbytes; /* current number of bytes on queue */ + unsigned long msg_qnum; /* number of messages in queue */ + unsigned long msg_qbytes; /* max number of bytes on queue */ + __kernel_pid_t msg_lspid; /* pid of last msgsnd */ + __kernel_pid_t msg_lrpid; /* last receive pid */ + unsigned long __unused4; + unsigned long __unused5; +}; +#elif defined (__MIPSEL__) +struct msqid64_ds { + struct ipc64_perm msg_perm; + unsigned long msg_stime; /* last msgsnd time */ + unsigned long msg_stime_high; + unsigned long msg_rtime; /* last msgrcv time */ + unsigned long msg_rtime_high; + unsigned long msg_ctime; /* last change time */ + unsigned long msg_ctime_high; + unsigned long msg_cbytes; /* current number of bytes on queue */ + unsigned long msg_qnum; /* number of messages in queue */ + unsigned long msg_qbytes; /* max number of bytes on queue */ + __kernel_pid_t msg_lspid; /* pid of last msgsnd */ + __kernel_pid_t msg_lrpid; /* last receive pid */ + unsigned long __unused4; + unsigned long __unused5; +}; +#else +#warning no endianess set +#endif #endif /* _ASM_MSGBUF_H */ diff --git a/arch/mips/include/uapi/asm/sembuf.h b/arch/mips/include/uapi/asm/sembuf.h index 2c0f507ab7d1..60c89e6cb25b 100644 --- a/arch/mips/include/uapi/asm/sembuf.h +++ b/arch/mips/include/uapi/asm/sembuf.h @@ -7,10 +7,11 @@ * Note extra padding because this structure is passed back and forth * between kernel and user space. * - * Pad space is left for: - * - 2 miscellaneous 64-bit values + * Pad space is left for 2 miscellaneous 64-bit values on mips64, + * but used for the upper 32 bit of the time values on mips32. */ +#ifdef __mips64 struct semid64_ds { struct ipc64_perm sem_perm; /* permissions .. see ipc.h */ __kernel_time_t sem_otime; /* last semop time */ @@ -19,5 +20,15 @@ struct semid64_ds { unsigned long __unused1; unsigned long __unused2; }; +#else +struct semid64_ds { + struct ipc64_perm sem_perm; /* permissions .. see ipc.h */ + unsigned long sem_otime; /* last semop time */ + unsigned long sem_ctime; /* last change time */ + unsigned long sem_nsems; /* no. of semaphores in array */ + unsigned long sem_otime_high; + unsigned long sem_ctime_high; +}; +#endif #endif /* _ASM_SEMBUF_H */ diff --git a/arch/mips/include/uapi/asm/shmbuf.h b/arch/mips/include/uapi/asm/shmbuf.h index 379e6bca518b..9b9bba3401f2 100644 --- a/arch/mips/include/uapi/asm/shmbuf.h +++ b/arch/mips/include/uapi/asm/shmbuf.h @@ -7,10 +7,13 @@ * Note extra padding because this structure is passed back and forth * between kernel and user space. * - * Pad space is left for: - * - 2 miscellaneous 32-bit rsp. 64-bit values + * As MIPS was lacking proper padding after shm_?time, we use 48 bits + * of the padding at the end to store a few additional bits of the time. + * libc implementations need to take care to convert this into a proper + * data structure when moving to 64-bit time_t. */ +#ifdef __mips64 struct shmid64_ds { struct ipc64_perm shm_perm; /* operation perms */ size_t shm_segsz; /* size of segment (bytes) */ @@ -23,6 +26,22 @@ struct shmid64_ds { unsigned long __unused1; unsigned long __unused2; }; +#else +struct shmid64_ds { + struct ipc64_perm shm_perm; /* operation perms */ + size_t shm_segsz; /* size of segment (bytes) */ + unsigned long shm_atime; /* last attach time */ + unsigned long shm_dtime; /* last detach time */ + unsigned long shm_ctime; /* last change time */ + __kernel_pid_t shm_cpid; /* pid of creator */ + __kernel_pid_t shm_lpid; /* pid of last operator */ + unsigned long shm_nattch; /* no. of current attaches */ + unsigned short shm_atime_high; + unsigned short shm_dtime_high; + unsigned short shm_ctime_high; + unsigned short __unused1; +}; +#endif struct shminfo64 { unsigned long shmmax; diff --git a/arch/mips/jz4740/platform.c b/arch/mips/jz4740/platform.c index 5b7cdd67a9d9..cbc5f8e87230 100644 --- a/arch/mips/jz4740/platform.c +++ b/arch/mips/jz4740/platform.c @@ -233,22 +233,6 @@ struct platform_device jz4740_adc_device = { .resource = jz4740_adc_resources, }; -/* Watchdog */ -static struct resource jz4740_wdt_resources[] = { - { - .start = JZ4740_WDT_BASE_ADDR, - .end = JZ4740_WDT_BASE_ADDR + 0x10 - 1, - .flags = IORESOURCE_MEM, - }, -}; - -struct platform_device jz4740_wdt_device = { - .name = "jz4740-wdt", - .id = -1, - .num_resources = ARRAY_SIZE(jz4740_wdt_resources), - .resource = jz4740_wdt_resources, -}; - /* PWM */ struct platform_device jz4740_pwm_device = { .name = "jz4740-pwm", diff --git a/arch/mips/jz4740/reset.c b/arch/mips/jz4740/reset.c index 67780c4b6573..5bf0cf44b55f 100644 --- a/arch/mips/jz4740/reset.c +++ b/arch/mips/jz4740/reset.c @@ -12,18 +12,9 @@ * */ -#include -#include -#include -#include - #include -#include -#include - #include "reset.h" -#include "clock.h" static void jz4740_halt(void) { @@ -36,29 +27,7 @@ static void jz4740_halt(void) } } -#define JZ_REG_WDT_DATA 0x00 -#define JZ_REG_WDT_COUNTER_ENABLE 0x04 -#define JZ_REG_WDT_COUNTER 0x08 -#define JZ_REG_WDT_CTRL 0x0c - -static void jz4740_restart(char *command) -{ - void __iomem *wdt_base = ioremap(JZ4740_WDT_BASE_ADDR, 0x0f); - - jz4740_timer_enable_watchdog(); - - writeb(0, wdt_base + JZ_REG_WDT_COUNTER_ENABLE); - - writew(0, wdt_base + JZ_REG_WDT_COUNTER); - writew(0, wdt_base + JZ_REG_WDT_DATA); - writew(BIT(2), wdt_base + JZ_REG_WDT_CTRL); - - writeb(1, wdt_base + JZ_REG_WDT_COUNTER_ENABLE); - jz4740_halt(); -} - void jz4740_reset_init(void) { - _machine_restart = jz4740_restart; _machine_halt = jz4740_halt; } diff --git a/arch/mips/kernel/asm-offsets.c b/arch/mips/kernel/asm-offsets.c index c1cd41456d42..cbe4742d2fff 100644 --- a/arch/mips/kernel/asm-offsets.c +++ b/arch/mips/kernel/asm-offsets.c @@ -83,7 +83,7 @@ void output_task_defines(void) OFFSET(TASK_FLAGS, task_struct, flags); OFFSET(TASK_MM, task_struct, mm); OFFSET(TASK_PID, task_struct, pid); -#if defined(CONFIG_CC_STACKPROTECTOR) +#if defined(CONFIG_STACKPROTECTOR) OFFSET(TASK_STACK_CANARY, task_struct, stack_canary); #endif DEFINE(TASK_STRUCT_SIZE, sizeof(struct task_struct)); diff --git a/arch/mips/kernel/cpu-probe.c b/arch/mips/kernel/cpu-probe.c index 6b07b739f914..b2509c19cfb5 100644 --- a/arch/mips/kernel/cpu-probe.c +++ b/arch/mips/kernel/cpu-probe.c @@ -414,6 +414,14 @@ static int __init ftlb_disable(char *s) __setup("noftlb", ftlb_disable); +/* + * Check if the CPU has per tc perf counters + */ +static inline void cpu_set_mt_per_tc_perf(struct cpuinfo_mips *c) +{ + if (read_c0_config7() & MTI_CONF7_PTC) + c->options |= MIPS_CPU_MT_PER_TC_PERF_COUNTERS; +} static inline void check_errata(void) { @@ -1572,6 +1580,7 @@ static inline void cpu_probe_mips(struct cpuinfo_mips *c, unsigned int cpu) c->cputype = CPU_34K; c->writecombine = _CACHE_UNCACHED; __cpu_name[cpu] = "MIPS 34Kc"; + cpu_set_mt_per_tc_perf(c); break; case PRID_IMP_74K: c->cputype = CPU_74K; @@ -1592,6 +1601,7 @@ static inline void cpu_probe_mips(struct cpuinfo_mips *c, unsigned int cpu) c->cputype = CPU_1004K; c->writecombine = _CACHE_UNCACHED; __cpu_name[cpu] = "MIPS 1004Kc"; + cpu_set_mt_per_tc_perf(c); break; case PRID_IMP_1074K: c->cputype = CPU_1074K; @@ -1601,10 +1611,12 @@ static inline void cpu_probe_mips(struct cpuinfo_mips *c, unsigned int cpu) case PRID_IMP_INTERAPTIV_UP: c->cputype = CPU_INTERAPTIV; __cpu_name[cpu] = "MIPS interAptiv"; + cpu_set_mt_per_tc_perf(c); break; case PRID_IMP_INTERAPTIV_MP: c->cputype = CPU_INTERAPTIV; __cpu_name[cpu] = "MIPS interAptiv (multi)"; + cpu_set_mt_per_tc_perf(c); break; case PRID_IMP_PROAPTIV_UP: c->cputype = CPU_PROAPTIV; diff --git a/arch/mips/kernel/octeon_switch.S b/arch/mips/kernel/octeon_switch.S index e42113fe2762..896080b445c2 100644 --- a/arch/mips/kernel/octeon_switch.S +++ b/arch/mips/kernel/octeon_switch.S @@ -61,7 +61,7 @@ #endif 3: -#if defined(CONFIG_CC_STACKPROTECTOR) && !defined(CONFIG_SMP) +#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) PTR_LA t8, __stack_chk_guard LONG_L t9, TASK_STACK_CANARY(a1) LONG_S t9, 0(t8) diff --git a/arch/mips/kernel/perf_event_mipsxx.c b/arch/mips/kernel/perf_event_mipsxx.c index ee73550f0b9a..413863508f6f 100644 --- a/arch/mips/kernel/perf_event_mipsxx.c +++ b/arch/mips/kernel/perf_event_mipsxx.c @@ -129,20 +129,14 @@ static struct mips_pmu mipspmu; #ifdef CONFIG_MIPS_PERF_SHARED_TC_COUNTERS -static int cpu_has_mipsmt_pertccounters; - static DEFINE_RWLOCK(pmuint_rwlock); #if defined(CONFIG_CPU_BMIPS5000) #define vpe_id() (cpu_has_mipsmt_pertccounters ? \ 0 : (smp_processor_id() & MIPS_CPUID_TO_COUNTER_MASK)) #else -/* - * FIXME: For VSMP, vpe_id() is redefined for Perf-events, because - * cpu_data[cpuid].vpe_id reports 0 for _both_ CPUs. - */ #define vpe_id() (cpu_has_mipsmt_pertccounters ? \ - 0 : smp_processor_id()) + 0 : cpu_vpe_id(¤t_cpu_data)) #endif /* Copied from op_model_mipsxx.c */ @@ -329,7 +323,11 @@ static int mipsxx_pmu_alloc_counter(struct cpu_hw_events *cpuc, static void mipsxx_pmu_enable_event(struct hw_perf_event *evt, int idx) { + struct perf_event *event = container_of(evt, struct perf_event, hw); struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); +#ifdef CONFIG_MIPS_MT_SMP + unsigned int range = evt->event_base >> 24; +#endif /* CONFIG_MIPS_MT_SMP */ WARN_ON(idx < 0 || idx >= mipspmu.num_counters); @@ -337,11 +335,37 @@ static void mipsxx_pmu_enable_event(struct hw_perf_event *evt, int idx) (evt->config_base & M_PERFCTL_CONFIG_MASK) | /* Make sure interrupt enabled. */ MIPS_PERFCTRL_IE; - if (IS_ENABLED(CONFIG_CPU_BMIPS5000)) + +#ifdef CONFIG_CPU_BMIPS5000 + { /* enable the counter for the calling thread */ cpuc->saved_ctrl[idx] |= (1 << (12 + vpe_id())) | BRCM_PERFCTRL_TC; + } +#else +#ifdef CONFIG_MIPS_MT_SMP + if (range > V) { + /* The counter is processor wide. Set it up to count all TCs. */ + pr_debug("Enabling perf counter for all TCs\n"); + cpuc->saved_ctrl[idx] |= M_TC_EN_ALL; + } else +#endif /* CONFIG_MIPS_MT_SMP */ + { + unsigned int cpu, ctrl; + + /* + * Set up the counter for a particular CPU when event->cpu is + * a valid CPU number. Otherwise set up the counter for the CPU + * scheduling this thread. + */ + cpu = (event->cpu >= 0) ? event->cpu : smp_processor_id(); + ctrl = M_PERFCTL_VPEID(cpu_vpe_id(&cpu_data[cpu])); + ctrl |= M_TC_EN_VPE; + cpuc->saved_ctrl[idx] |= ctrl; + pr_debug("Enabling perf counter for CPU%d\n", cpu); + } +#endif /* CONFIG_CPU_BMIPS5000 */ /* * We do not actually let the counter run. Leave it until start(). */ @@ -655,13 +679,14 @@ static unsigned int mipspmu_perf_event_encode(const struct mips_perf_event *pev) * event_id. */ #ifdef CONFIG_MIPS_MT_SMP - return ((unsigned int)pev->range << 24) | - (pev->cntr_mask & 0xffff00) | - (pev->event_id & 0xff); -#else - return (pev->cntr_mask & 0xffff00) | - (pev->event_id & 0xff); -#endif + if (num_possible_cpus() > 1) + return ((unsigned int)pev->range << 24) | + (pev->cntr_mask & 0xffff00) | + (pev->event_id & 0xff); + else +#endif /* CONFIG_MIPS_MT_SMP */ + return ((pev->cntr_mask & 0xffff00) | + (pev->event_id & 0xff)); } static const struct mips_perf_event *mipspmu_map_general_event(int idx) @@ -1265,37 +1290,6 @@ static const struct mips_perf_event xlp_cache_map }, }; -#ifdef CONFIG_MIPS_MT_SMP -static void check_and_calc_range(struct perf_event *event, - const struct mips_perf_event *pev) -{ - struct hw_perf_event *hwc = &event->hw; - - if (event->cpu >= 0) { - if (pev->range > V) { - /* - * The user selected an event that is processor - * wide, while expecting it to be VPE wide. - */ - hwc->config_base |= M_TC_EN_ALL; - } else { - /* - * FIXME: cpu_data[event->cpu].vpe_id reports 0 - * for both CPUs. - */ - hwc->config_base |= M_PERFCTL_VPEID(event->cpu); - hwc->config_base |= M_TC_EN_VPE; - } - } else - hwc->config_base |= M_TC_EN_ALL; -} -#else -static void check_and_calc_range(struct perf_event *event, - const struct mips_perf_event *pev) -{ -} -#endif - static int __hw_perf_event_init(struct perf_event *event) { struct perf_event_attr *attr = &event->attr; @@ -1331,10 +1325,6 @@ static int __hw_perf_event_init(struct perf_event *event) */ hwc->config_base = MIPS_PERFCTRL_IE; - /* Calculate range bits and validate it. */ - if (num_possible_cpus() > 1) - check_and_calc_range(event, pev); - hwc->event_base = mipspmu_perf_event_encode(pev); if (PERF_TYPE_RAW == event->attr.type) mutex_unlock(&raw_event_mutex); @@ -1723,7 +1713,6 @@ init_hw_perf_events(void) } #ifdef CONFIG_MIPS_PERF_SHARED_TC_COUNTERS - cpu_has_mipsmt_pertccounters = read_c0_config7() & (1<<19); if (!cpu_has_mipsmt_pertccounters) counters = counters_total_to_per_cpu(counters); #endif diff --git a/arch/mips/kernel/process.c b/arch/mips/kernel/process.c index b9e9bf628849..8d85046adcc8 100644 --- a/arch/mips/kernel/process.c +++ b/arch/mips/kernel/process.c @@ -180,7 +180,7 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long usp, return 0; } -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR #include unsigned long __stack_chk_guard __read_mostly; EXPORT_SYMBOL(__stack_chk_guard); @@ -721,6 +721,10 @@ int mips_set_process_fp_mode(struct task_struct *task, unsigned int value) if (value & ~known_bits) return -EOPNOTSUPP; + /* Setting FRE without FR is not supported. */ + if ((value & (PR_FP_MODE_FR | PR_FP_MODE_FRE)) == PR_FP_MODE_FRE) + return -EOPNOTSUPP; + /* Avoid inadvertently triggering emulation */ if ((value & PR_FP_MODE_FR) && raw_cpu_has_fpu && !(raw_current_cpu_data.fpu_id & MIPS_FPIR_F64)) diff --git a/arch/mips/kernel/ptrace.c b/arch/mips/kernel/ptrace.c index 8d098b9f395c..9f6c3f2aa2e2 100644 --- a/arch/mips/kernel/ptrace.c +++ b/arch/mips/kernel/ptrace.c @@ -811,14 +811,14 @@ long arch_ptrace(struct task_struct *child, long request, /* * The odd registers are actually the high * order bits of the values stored in the even - * registers - unless we're using r2k_switch.S. + * registers. */ tmp = get_fpr32(&fregs[(addr & ~1) - FPR_BASE], addr & 1); break; } #endif - tmp = get_fpr32(&fregs[addr - FPR_BASE], 0); + tmp = get_fpr64(&fregs[addr - FPR_BASE], 0); break; case PC: tmp = regs->cp0_epc; @@ -906,7 +906,7 @@ long arch_ptrace(struct task_struct *child, long request, /* * The odd registers are actually the high * order bits of the values stored in the even - * registers - unless we're using r2k_switch.S. + * registers. */ set_fpr32(&fregs[(addr & ~1) - FPR_BASE], addr & 1, data); diff --git a/arch/mips/kernel/ptrace32.c b/arch/mips/kernel/ptrace32.c index 656a137c1fe2..7edc629304c8 100644 --- a/arch/mips/kernel/ptrace32.c +++ b/arch/mips/kernel/ptrace32.c @@ -103,13 +103,13 @@ long compat_arch_ptrace(struct task_struct *child, compat_long_t request, /* * The odd registers are actually the high * order bits of the values stored in the even - * registers - unless we're using r2k_switch.S. + * registers. */ tmp = get_fpr32(&fregs[(addr & ~1) - FPR_BASE], addr & 1); break; } - tmp = get_fpr32(&fregs[addr - FPR_BASE], 0); + tmp = get_fpr64(&fregs[addr - FPR_BASE], 0); break; case PC: tmp = regs->cp0_epc; @@ -216,7 +216,7 @@ long compat_arch_ptrace(struct task_struct *child, compat_long_t request, /* * The odd registers are actually the high * order bits of the values stored in the even - * registers - unless we're using r2k_switch.S. + * registers. */ set_fpr32(&fregs[(addr & ~1) - FPR_BASE], addr & 1, data); diff --git a/arch/mips/kernel/r2300_switch.S b/arch/mips/kernel/r2300_switch.S index 665897139f30..71b1aafae1bb 100644 --- a/arch/mips/kernel/r2300_switch.S +++ b/arch/mips/kernel/r2300_switch.S @@ -36,7 +36,7 @@ LEAF(resume) cpu_save_nonscratch a0 sw ra, THREAD_REG31(a0) -#if defined(CONFIG_CC_STACKPROTECTOR) && !defined(CONFIG_SMP) +#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) PTR_LA t8, __stack_chk_guard LONG_L t9, TASK_STACK_CANARY(a1) LONG_S t9, 0(t8) diff --git a/arch/mips/kernel/r4k_switch.S b/arch/mips/kernel/r4k_switch.S index 17cf9341c1cf..58232ae6cfae 100644 --- a/arch/mips/kernel/r4k_switch.S +++ b/arch/mips/kernel/r4k_switch.S @@ -31,7 +31,7 @@ cpu_save_nonscratch a0 LONG_S ra, THREAD_REG31(a0) -#if defined(CONFIG_CC_STACKPROTECTOR) && !defined(CONFIG_SMP) +#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) PTR_LA t8, __stack_chk_guard LONG_L t9, TASK_STACK_CANARY(a1) LONG_S t9, 0(t8) diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c index 563188ac6fa2..2c96c0c68116 100644 --- a/arch/mips/kernel/setup.c +++ b/arch/mips/kernel/setup.c @@ -93,7 +93,7 @@ void __init add_memory_region(phys_addr_t start, phys_addr_t size, long type) * If the region reaches the top of the physical address space, adjust * the size slightly so that (start + size) doesn't overflow */ - if (start + size - 1 == (phys_addr_t)ULLONG_MAX) + if (start + size - 1 == PHYS_ADDR_MAX) --size; /* Sanity check */ @@ -376,7 +376,7 @@ static void __init bootmem_init(void) unsigned long reserved_end; unsigned long mapstart = ~0UL; unsigned long bootmap_size; - phys_addr_t ramstart = (phys_addr_t)ULLONG_MAX; + phys_addr_t ramstart = PHYS_ADDR_MAX; bool bootmap_valid = false; int i; diff --git a/arch/mips/kernel/signal32.c b/arch/mips/kernel/signal32.c index c4db910a8794..b5d9e1784aff 100644 --- a/arch/mips/kernel/signal32.c +++ b/arch/mips/kernel/signal32.c @@ -8,13 +8,13 @@ * Copyright (C) 1999, 2000 Silicon Graphics, Inc. * Copyright (C) 2016, Imagination Technologies Ltd. */ +#include #include #include #include #include #include -#include #include #include #include diff --git a/arch/mips/kernel/time.c b/arch/mips/kernel/time.c index a6ebc8135112..bfe02ded25d1 100644 --- a/arch/mips/kernel/time.c +++ b/arch/mips/kernel/time.c @@ -34,21 +34,6 @@ DEFINE_SPINLOCK(rtc_lock); EXPORT_SYMBOL(rtc_lock); -int __weak rtc_mips_set_time(unsigned long sec) -{ - return -ENODEV; -} - -int __weak rtc_mips_set_mmss(unsigned long nowtime) -{ - return rtc_mips_set_time(nowtime); -} - -int update_persistent_clock(struct timespec now) -{ - return rtc_mips_set_mmss(now.tv_sec); -} - static int null_perf_irq(void) { return 0; diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c index 967e9e4e795e..d67fa74622ee 100644 --- a/arch/mips/kernel/traps.c +++ b/arch/mips/kernel/traps.c @@ -699,17 +699,11 @@ static int simulate_sync(struct pt_regs *regs, unsigned int opcode) asmlinkage void do_ov(struct pt_regs *regs) { enum ctx_state prev_state; - siginfo_t info; - - clear_siginfo(&info); - info.si_signo = SIGFPE; - info.si_code = FPE_INTOVF; - info.si_addr = (void __user *)regs->cp0_epc; prev_state = exception_enter(); die_if_kernel("Integer overflow", regs); - force_sig_info(SIGFPE, &info, current); + force_sig_fault(SIGFPE, FPE_INTOVF, (void __user *)regs->cp0_epc, current); exception_exit(prev_state); } @@ -722,32 +716,27 @@ asmlinkage void do_ov(struct pt_regs *regs) void force_fcr31_sig(unsigned long fcr31, void __user *fault_addr, struct task_struct *tsk) { - struct siginfo si; - - clear_siginfo(&si); - si.si_addr = fault_addr; - si.si_signo = SIGFPE; + int si_code = FPE_FLTUNK; if (fcr31 & FPU_CSR_INV_X) - si.si_code = FPE_FLTINV; + si_code = FPE_FLTINV; else if (fcr31 & FPU_CSR_DIV_X) - si.si_code = FPE_FLTDIV; + si_code = FPE_FLTDIV; else if (fcr31 & FPU_CSR_OVF_X) - si.si_code = FPE_FLTOVF; + si_code = FPE_FLTOVF; else if (fcr31 & FPU_CSR_UDF_X) - si.si_code = FPE_FLTUND; + si_code = FPE_FLTUND; else if (fcr31 & FPU_CSR_INE_X) - si.si_code = FPE_FLTRES; + si_code = FPE_FLTRES; - force_sig_info(SIGFPE, &si, tsk); + force_sig_fault(SIGFPE, si_code, fault_addr, tsk); } int process_fpemu_return(int sig, void __user *fault_addr, unsigned long fcr31) { - struct siginfo si; + int si_code; struct vm_area_struct *vma; - clear_siginfo(&si); switch (sig) { case 0: return 0; @@ -757,23 +746,18 @@ int process_fpemu_return(int sig, void __user *fault_addr, unsigned long fcr31) return 1; case SIGBUS: - si.si_addr = fault_addr; - si.si_signo = sig; - si.si_code = BUS_ADRERR; - force_sig_info(sig, &si, current); + force_sig_fault(SIGBUS, BUS_ADRERR, fault_addr, current); return 1; case SIGSEGV: - si.si_addr = fault_addr; - si.si_signo = sig; down_read(¤t->mm->mmap_sem); vma = find_vma(current->mm, (unsigned long)fault_addr); if (vma && (vma->vm_start <= (unsigned long)fault_addr)) - si.si_code = SEGV_ACCERR; + si_code = SEGV_ACCERR; else - si.si_code = SEGV_MAPERR; + si_code = SEGV_MAPERR; up_read(¤t->mm->mmap_sem); - force_sig_info(sig, &si, current); + force_sig_fault(SIGSEGV, si_code, fault_addr, current); return 1; default: @@ -896,10 +880,8 @@ out: void do_trap_or_bp(struct pt_regs *regs, unsigned int code, int si_code, const char *str) { - siginfo_t info; char b[40]; - clear_siginfo(&info); #ifdef CONFIG_KGDB_LOW_LEVEL_TRAP if (kgdb_ll_trap(DIE_TRAP, str, regs, code, current->thread.trap_nr, SIGTRAP) == NOTIFY_STOP) @@ -921,13 +903,9 @@ void do_trap_or_bp(struct pt_regs *regs, unsigned int code, int si_code, case BRK_DIVZERO: scnprintf(b, sizeof(b), "%s instruction in kernel code", str); die_if_kernel(b, regs); - if (code == BRK_DIVZERO) - info.si_code = FPE_INTDIV; - else - info.si_code = FPE_INTOVF; - info.si_signo = SIGFPE; - info.si_addr = (void __user *) regs->cp0_epc; - force_sig_info(SIGFPE, &info, current); + force_sig_fault(SIGFPE, + code == BRK_DIVZERO ? FPE_INTDIV : FPE_INTOVF, + (void __user *) regs->cp0_epc, current); break; case BRK_BUG: die_if_kernel("Kernel bug detected", regs); @@ -952,9 +930,7 @@ void do_trap_or_bp(struct pt_regs *regs, unsigned int code, int si_code, scnprintf(b, sizeof(b), "%s instruction in kernel code", str); die_if_kernel(b, regs); if (si_code) { - info.si_signo = SIGTRAP; - info.si_code = si_code; - force_sig_info(SIGTRAP, &info, current); + force_sig_fault(SIGTRAP, si_code, NULL, current); } else { force_sig(SIGTRAP, current); } @@ -1506,13 +1482,8 @@ asmlinkage void do_mdmx(struct pt_regs *regs) */ asmlinkage void do_watch(struct pt_regs *regs) { - siginfo_t info; enum ctx_state prev_state; - clear_siginfo(&info); - info.si_signo = SIGTRAP; - info.si_code = TRAP_HWBKPT; - prev_state = exception_enter(); /* * Clear WP (bit 22) bit of cause register so we don't loop @@ -1528,7 +1499,7 @@ asmlinkage void do_watch(struct pt_regs *regs) if (test_tsk_thread_flag(current, TIF_LOAD_WATCH)) { mips_read_watch_registers(); local_irq_enable(); - force_sig_info(SIGTRAP, &info, current); + force_sig_fault(SIGTRAP, TRAP_HWBKPT, NULL, current); } else { mips_clear_watch_registers(); local_irq_enable(); diff --git a/arch/mips/kernel/vpe.c b/arch/mips/kernel/vpe.c index 544ea21bfef9..0bef238d2c0c 100644 --- a/arch/mips/kernel/vpe.c +++ b/arch/mips/kernel/vpe.c @@ -872,7 +872,7 @@ static ssize_t vpe_write(struct file *file, const char __user *buffer, return -ENODEV; if ((count + v->len) > v->plen) { - pr_warn("VPE loader: elf size too big. Perhaps strip uneeded symbols\n"); + pr_warn("VPE loader: elf size too big. Perhaps strip unneeded symbols\n"); return -ENOMEM; } diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c index 0f725e9cee8f..7cd76f93a438 100644 --- a/arch/mips/kvm/mips.c +++ b/arch/mips/kvm/mips.c @@ -1076,7 +1076,7 @@ int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu) return -ENOIOCTLCMD; } -int kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf) +vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf) { return VM_FAULT_SIGBUS; } diff --git a/arch/mips/lasat/ds1603.c b/arch/mips/lasat/ds1603.c index 8bd5cf820eed..e6ce39fefa78 100644 --- a/arch/mips/lasat/ds1603.c +++ b/arch/mips/lasat/ds1603.c @@ -136,7 +136,7 @@ static void rtc_end_op(void) lasat_ndelay(1000); } -void read_persistent_clock(struct timespec *ts) +void read_persistent_clock64(struct timespec64 *ts) { unsigned long word; unsigned long flags; @@ -152,14 +152,19 @@ void read_persistent_clock(struct timespec *ts) ts->tv_nsec = 0; } -int rtc_mips_set_mmss(unsigned long time) +int update_persistent_clock64(struct timespec64 now) { + time64_t time = now.tv_sec; unsigned long flags; spin_lock_irqsave(&rtc_lock, flags); rtc_init_op(); rtc_write_byte(SET_TIME_CMD); - rtc_write_word(time); + /* + * Due to the hardware limitation, we cast to 'unsigned long' type, + * so it will overflow in year 2106 on 32-bit machine. + */ + rtc_write_word((unsigned long)time); rtc_end_op(); spin_unlock_irqrestore(&rtc_lock, flags); diff --git a/arch/mips/lasat/sysctl.c b/arch/mips/lasat/sysctl.c index 6f7422400f32..ead07c243c6a 100644 --- a/arch/mips/lasat/sysctl.c +++ b/arch/mips/lasat/sysctl.c @@ -73,8 +73,16 @@ int proc_dolasatrtc(struct ctl_table *table, int write, if (r) return r; - if (write) - rtc_mips_set_mmss(rtctmp); + if (write) { + /* + * Due to the RTC hardware limitation, we can not actually + * use the full 64-bit range here. + */ + ts.tv_sec = rtctmp; + ts.tv_nsec = 0; + + update_persistent_clock64(ts); + } return 0; } diff --git a/arch/mips/lib/Makefile b/arch/mips/lib/Makefile index e84e12655fa8..6537e022ef62 100644 --- a/arch/mips/lib/Makefile +++ b/arch/mips/lib/Makefile @@ -16,5 +16,4 @@ obj-$(CONFIG_CPU_R3000) += r3k_dump_tlb.o obj-$(CONFIG_CPU_TX39XX) += r3k_dump_tlb.o # libgcc-style stuff needed in the kernel -obj-y += ashldi3.o ashrdi3.o bswapsi.o bswapdi.o cmpdi2.o lshrdi3.o multi3.o \ - ucmpdi2.o +obj-y += bswapsi.o bswapdi.o multi3.o diff --git a/arch/mips/lib/ashldi3.c b/arch/mips/lib/ashldi3.c deleted file mode 100644 index 24cd6903e797..000000000000 --- a/arch/mips/lib/ashldi3.c +++ /dev/null @@ -1,30 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include - -#include "libgcc.h" - -long long notrace __ashldi3(long long u, word_type b) -{ - DWunion uu, w; - word_type bm; - - if (b == 0) - return u; - - uu.ll = u; - bm = 32 - b; - - if (bm <= 0) { - w.s.low = 0; - w.s.high = (unsigned int) uu.s.low << -bm; - } else { - const unsigned int carries = (unsigned int) uu.s.low >> bm; - - w.s.low = (unsigned int) uu.s.low << b; - w.s.high = ((unsigned int) uu.s.high << b) | carries; - } - - return w.ll; -} - -EXPORT_SYMBOL(__ashldi3); diff --git a/arch/mips/lib/ashrdi3.c b/arch/mips/lib/ashrdi3.c deleted file mode 100644 index 23f5295af51e..000000000000 --- a/arch/mips/lib/ashrdi3.c +++ /dev/null @@ -1,32 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include - -#include "libgcc.h" - -long long notrace __ashrdi3(long long u, word_type b) -{ - DWunion uu, w; - word_type bm; - - if (b == 0) - return u; - - uu.ll = u; - bm = 32 - b; - - if (bm <= 0) { - /* w.s.high = 1..1 or 0..0 */ - w.s.high = - uu.s.high >> 31; - w.s.low = uu.s.high >> -bm; - } else { - const unsigned int carries = (unsigned int) uu.s.high << bm; - - w.s.high = uu.s.high >> b; - w.s.low = ((unsigned int) uu.s.low >> b) | carries; - } - - return w.ll; -} - -EXPORT_SYMBOL(__ashrdi3); diff --git a/arch/mips/lib/cmpdi2.c b/arch/mips/lib/cmpdi2.c deleted file mode 100644 index 93cfc785927d..000000000000 --- a/arch/mips/lib/cmpdi2.c +++ /dev/null @@ -1,28 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include - -#include "libgcc.h" - -word_type notrace __cmpdi2(long long a, long long b) -{ - const DWunion au = { - .ll = a - }; - const DWunion bu = { - .ll = b - }; - - if (au.s.high < bu.s.high) - return 0; - else if (au.s.high > bu.s.high) - return 2; - - if ((unsigned int) au.s.low < (unsigned int) bu.s.low) - return 0; - else if ((unsigned int) au.s.low > (unsigned int) bu.s.low) - return 2; - - return 1; -} - -EXPORT_SYMBOL(__cmpdi2); diff --git a/arch/mips/lib/lshrdi3.c b/arch/mips/lib/lshrdi3.c deleted file mode 100644 index 914b971aca3b..000000000000 --- a/arch/mips/lib/lshrdi3.c +++ /dev/null @@ -1,30 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include - -#include "libgcc.h" - -long long notrace __lshrdi3(long long u, word_type b) -{ - DWunion uu, w; - word_type bm; - - if (b == 0) - return u; - - uu.ll = u; - bm = 32 - b; - - if (bm <= 0) { - w.s.high = 0; - w.s.low = (unsigned int) uu.s.high >> -bm; - } else { - const unsigned int carries = (unsigned int) uu.s.high << bm; - - w.s.high = (unsigned int) uu.s.high >> b; - w.s.low = ((unsigned int) uu.s.low >> b) | carries; - } - - return w.ll; -} - -EXPORT_SYMBOL(__lshrdi3); diff --git a/arch/mips/lib/memset.S b/arch/mips/lib/memset.S index f7327979a8f8..1cc306520a55 100644 --- a/arch/mips/lib/memset.S +++ b/arch/mips/lib/memset.S @@ -95,7 +95,7 @@ sltiu t0, a2, STORSIZE /* very small region? */ bnez t0, .Lsmall_memset\@ - andi t0, a0, STORMASK /* aligned? */ + andi t0, a0, STORMASK /* aligned? */ #ifdef CONFIG_CPU_MICROMIPS move t8, a1 /* used by 'swp' instruction */ @@ -103,12 +103,12 @@ #endif #ifndef CONFIG_CPU_DADDI_WORKAROUNDS beqz t0, 1f - PTR_SUBU t0, STORSIZE /* alignment in bytes */ + PTR_SUBU t0, STORSIZE /* alignment in bytes */ #else .set noat li AT, STORSIZE beqz t0, 1f - PTR_SUBU t0, AT /* alignment in bytes */ + PTR_SUBU t0, AT /* alignment in bytes */ .set at #endif @@ -149,7 +149,7 @@ 1: ori t1, a2, 0x3f /* # of full blocks */ xori t1, 0x3f beqz t1, .Lmemset_partial\@ /* no block to fill */ - andi t0, a2, 0x40-STORSIZE + andi t0, a2, 0x40-STORSIZE PTR_ADDU t1, a0 /* end address */ .set reorder @@ -174,7 +174,7 @@ .set at #endif jr t1 - PTR_ADDU a0, t0 /* dest ptr */ + PTR_ADDU a0, t0 /* dest ptr */ .set push .set noreorder @@ -186,7 +186,7 @@ beqz a2, 1f #ifndef CONFIG_CPU_MIPSR6 - PTR_ADDU a0, a2 /* What's left */ + PTR_ADDU a0, a2 /* What's left */ R10KCBARRIER(0(ra)) #ifdef __MIPSEB__ EX(LONG_S_R, a1, -1(a0), .Llast_fixup\@) @@ -194,7 +194,7 @@ EX(LONG_S_L, a1, -1(a0), .Llast_fixup\@) #endif #else - PTR_SUBU t0, $0, a2 + PTR_SUBU t0, $0, a2 PTR_ADDIU t0, 1 STORE_BYTE(0) STORE_BYTE(1) @@ -210,11 +210,11 @@ 0: #endif 1: jr ra - move a2, zero + move a2, zero .Lsmall_memset\@: beqz a2, 2f - PTR_ADDU t1, a0, a2 + PTR_ADDU t1, a0, a2 1: PTR_ADDIU a0, 1 /* fill bytewise */ R10KCBARRIER(0(ra)) @@ -222,7 +222,7 @@ EX(sb, a1, -1(a0), .Lsmall_fixup\@) 2: jr ra /* done */ - move a2, zero + move a2, zero .if __memset == 1 END(memset) .set __memset, 0 @@ -238,7 +238,7 @@ .Lfirst_fixup\@: jr ra - nop + nop .Lfwd_fixup\@: PTR_L t0, TI_TASK($28) @@ -246,7 +246,7 @@ LONG_L t0, THREAD_BUADDR(t0) LONG_ADDU a2, t1 jr ra - LONG_SUBU a2, t0 + LONG_SUBU a2, t0 .Lpartial_fixup\@: PTR_L t0, TI_TASK($28) @@ -254,7 +254,7 @@ LONG_L t0, THREAD_BUADDR(t0) LONG_ADDU a2, a0 jr ra - LONG_SUBU a2, t0 + LONG_SUBU a2, t0 .Llast_fixup\@: jr ra @@ -278,7 +278,7 @@ LEAF(memset) EXPORT_SYMBOL(memset) beqz a1, 1f - move v0, a0 /* result */ + move v0, a0 /* result */ andi a1, 0xff /* spread fillword */ LONG_SLL t1, a1, 8 diff --git a/arch/mips/lib/ucmpdi2.c b/arch/mips/lib/ucmpdi2.c deleted file mode 100644 index c31c78ca4175..000000000000 --- a/arch/mips/lib/ucmpdi2.c +++ /dev/null @@ -1,22 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include - -#include "libgcc.h" - -word_type notrace __ucmpdi2(unsigned long long a, unsigned long long b) -{ - const DWunion au = {.ll = a}; - const DWunion bu = {.ll = b}; - - if ((unsigned int) au.s.high < (unsigned int) bu.s.high) - return 0; - else if ((unsigned int) au.s.high > (unsigned int) bu.s.high) - return 2; - if ((unsigned int) au.s.low < (unsigned int) bu.s.low) - return 0; - else if ((unsigned int) au.s.low > (unsigned int) bu.s.low) - return 2; - return 1; -} - -EXPORT_SYMBOL(__ucmpdi2); diff --git a/arch/mips/loongson64/Kconfig b/arch/mips/loongson64/Kconfig index 72af0c183969..c79e6a565572 100644 --- a/arch/mips/loongson64/Kconfig +++ b/arch/mips/loongson64/Kconfig @@ -130,21 +130,6 @@ config LOONGSON_UART_BASE default y depends on EARLY_PRINTK || SERIAL_8250 -config IOMMU_HELPER - bool - -config NEED_SG_DMA_LENGTH - bool - -config SWIOTLB - bool "Soft IOMMU Support for All-Memory DMA" - default y - depends on CPU_LOONGSON3 - select DMA_DIRECT_OPS - select IOMMU_HELPER - select NEED_SG_DMA_LENGTH - select NEED_DMA_MAP_STATE - config PHYS48_TO_HT40 bool default y if CPU_LOONGSON3 diff --git a/arch/mips/loongson64/common/time.c b/arch/mips/loongson64/common/time.c index e1a5382ad47e..0ba53c55ff33 100644 --- a/arch/mips/loongson64/common/time.c +++ b/arch/mips/loongson64/common/time.c @@ -29,7 +29,7 @@ void __init plat_time_init(void) #endif } -void read_persistent_clock(struct timespec *ts) +void read_persistent_clock64(struct timespec64 *ts) { ts->tv_sec = mc146818_get_cmos_time(); ts->tv_nsec = 0; diff --git a/arch/mips/mm/dma-default.c b/arch/mips/mm/dma-default.c index dcafa43613b6..f9fef0028ca2 100644 --- a/arch/mips/mm/dma-default.c +++ b/arch/mips/mm/dma-default.c @@ -402,13 +402,3 @@ static const struct dma_map_ops mips_default_dma_map_ops = { const struct dma_map_ops *mips_dma_map_ops = &mips_default_dma_map_ops; EXPORT_SYMBOL(mips_dma_map_ops); - -#define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16) - -static int __init mips_dma_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - - return 0; -} -fs_initcall(mips_dma_init); diff --git a/arch/mips/mm/fault.c b/arch/mips/mm/fault.c index 4f8f5bf46977..5f71f2b903b7 100644 --- a/arch/mips/mm/fault.c +++ b/arch/mips/mm/fault.c @@ -42,7 +42,7 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write, struct task_struct *tsk = current; struct mm_struct *mm = tsk->mm; const int field = sizeof(unsigned long) * 2; - siginfo_t info; + int si_code; int fault; unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; @@ -63,7 +63,7 @@ static void __kprobes __do_page_fault(struct pt_regs *regs, unsigned long write, return; #endif - info.si_code = SEGV_MAPERR; + si_code = SEGV_MAPERR; /* * We fault-in kernel-space virtual memory on-demand. The @@ -112,7 +112,7 @@ retry: * we can handle it.. */ good_area: - info.si_code = SEGV_ACCERR; + si_code = SEGV_ACCERR; if (write) { if (!(vma->vm_flags & VM_WRITE)) @@ -223,11 +223,7 @@ bad_area_nosemaphore: pr_cont("\n"); } current->thread.trap_nr = (regs->cp0_cause >> 2) & 0x1f; - info.si_signo = SIGSEGV; - info.si_errno = 0; - /* info.si_code has been set above */ - info.si_addr = (void __user *) address; - force_sig_info(SIGSEGV, &info, tsk); + force_sig_fault(SIGSEGV, si_code, (void __user *)address, tsk); return; } @@ -283,11 +279,7 @@ do_sigbus: #endif current->thread.trap_nr = (regs->cp0_cause >> 2) & 0x1f; tsk->thread.cp0_badvaddr = address; - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRERR; - info.si_addr = (void __user *) address; - force_sig_info(SIGBUS, &info, tsk); + force_sig_fault(SIGBUS, BUS_ADRERR, (void __user *)address, tsk); return; #ifndef CONFIG_64BIT diff --git a/arch/mips/mm/sc-debugfs.c b/arch/mips/mm/sc-debugfs.c index 2e2132d3f5c7..2a116084216f 100644 --- a/arch/mips/mm/sc-debugfs.c +++ b/arch/mips/mm/sc-debugfs.c @@ -31,17 +31,10 @@ static ssize_t sc_prefetch_write(struct file *file, const char __user *user_buf, size_t count, loff_t *ppos) { - char buf[32]; - ssize_t buf_size; bool enabled; int err; - buf_size = min(count, sizeof(buf) - 1); - if (copy_from_user(buf, user_buf, buf_size)) - return -EFAULT; - - buf[buf_size] = '\0'; - err = strtobool(buf, &enabled); + err = kstrtobool_from_user(user_buf, count, &enabled); if (err) return err; diff --git a/arch/mips/mti-malta/malta-time.c b/arch/mips/mti-malta/malta-time.c index 66c866740ff2..d22b7edc3886 100644 --- a/arch/mips/mti-malta/malta-time.c +++ b/arch/mips/mti-malta/malta-time.c @@ -134,7 +134,7 @@ static void __init estimate_frequencies(void) } } -void read_persistent_clock(struct timespec *ts) +void read_persistent_clock64(struct timespec64 *ts) { ts->tv_sec = mc146818_get_cmos_time(); ts->tv_nsec = 0; diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index 3e2798bfea4f..aeb7b1b0f202 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -95,7 +95,6 @@ enum reg_val_type { * struct jit_ctx - JIT context * @skf: The sk_filter * @stack_size: eBPF stack size - * @tmp_offset: eBPF $sp offset to 8-byte temporary memory * @idx: Instruction index * @flags: JIT flags * @offsets: Instruction offsets @@ -105,7 +104,6 @@ enum reg_val_type { struct jit_ctx { const struct bpf_prog *skf; int stack_size; - int tmp_offset; u32 idx; u32 flags; u32 *offsets; @@ -293,7 +291,6 @@ static int gen_int_prologue(struct jit_ctx *ctx) locals_size = (ctx->flags & EBPF_SEEN_FP) ? MAX_BPF_STACK : 0; stack_adjust += locals_size; - ctx->tmp_offset = locals_size; ctx->stack_size = stack_adjust; @@ -399,7 +396,6 @@ static void gen_imm_to_reg(const struct bpf_insn *insn, int reg, emit_instr(ctx, lui, reg, upper >> 16); emit_instr(ctx, addiu, reg, reg, lower); } - } static int gen_imm_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, @@ -547,28 +543,6 @@ static int gen_imm_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, return 0; } -static void * __must_check -ool_skb_header_pointer(const struct sk_buff *skb, int offset, - int len, void *buffer) -{ - return skb_header_pointer(skb, offset, len, buffer); -} - -static int size_to_len(const struct bpf_insn *insn) -{ - switch (BPF_SIZE(insn->code)) { - case BPF_B: - return 1; - case BPF_H: - return 2; - case BPF_W: - return 4; - case BPF_DW: - return 8; - } - return 0; -} - static void emit_const_to_reg(struct jit_ctx *ctx, int dst, u64 value) { if (value >= 0xffffffffffff8000ull || value < 0x8000ull) { @@ -1267,110 +1241,6 @@ jeq_common: return -EINVAL; break; - case BPF_LD | BPF_B | BPF_ABS: - case BPF_LD | BPF_H | BPF_ABS: - case BPF_LD | BPF_W | BPF_ABS: - case BPF_LD | BPF_DW | BPF_ABS: - ctx->flags |= EBPF_SAVE_RA; - - gen_imm_to_reg(insn, MIPS_R_A1, ctx); - emit_instr(ctx, addiu, MIPS_R_A2, MIPS_R_ZERO, size_to_len(insn)); - - if (insn->imm < 0) { - emit_const_to_reg(ctx, MIPS_R_T9, (u64)bpf_internal_load_pointer_neg_helper); - } else { - emit_const_to_reg(ctx, MIPS_R_T9, (u64)ool_skb_header_pointer); - emit_instr(ctx, daddiu, MIPS_R_A3, MIPS_R_SP, ctx->tmp_offset); - } - goto ld_skb_common; - - case BPF_LD | BPF_B | BPF_IND: - case BPF_LD | BPF_H | BPF_IND: - case BPF_LD | BPF_W | BPF_IND: - case BPF_LD | BPF_DW | BPF_IND: - ctx->flags |= EBPF_SAVE_RA; - src = ebpf_to_mips_reg(ctx, insn, src_reg_no_fp); - if (src < 0) - return src; - ts = get_reg_val_type(ctx, this_idx, insn->src_reg); - if (ts == REG_32BIT_ZERO_EX) { - /* sign extend */ - emit_instr(ctx, sll, MIPS_R_A1, src, 0); - src = MIPS_R_A1; - } - if (insn->imm >= S16_MIN && insn->imm <= S16_MAX) { - emit_instr(ctx, daddiu, MIPS_R_A1, src, insn->imm); - } else { - gen_imm_to_reg(insn, MIPS_R_AT, ctx); - emit_instr(ctx, daddu, MIPS_R_A1, MIPS_R_AT, src); - } - /* truncate to 32-bit int */ - emit_instr(ctx, sll, MIPS_R_A1, MIPS_R_A1, 0); - emit_instr(ctx, daddiu, MIPS_R_A3, MIPS_R_SP, ctx->tmp_offset); - emit_instr(ctx, slt, MIPS_R_AT, MIPS_R_A1, MIPS_R_ZERO); - - emit_const_to_reg(ctx, MIPS_R_T8, (u64)bpf_internal_load_pointer_neg_helper); - emit_const_to_reg(ctx, MIPS_R_T9, (u64)ool_skb_header_pointer); - emit_instr(ctx, addiu, MIPS_R_A2, MIPS_R_ZERO, size_to_len(insn)); - emit_instr(ctx, movn, MIPS_R_T9, MIPS_R_T8, MIPS_R_AT); - -ld_skb_common: - emit_instr(ctx, jalr, MIPS_R_RA, MIPS_R_T9); - /* delay slot move */ - emit_instr(ctx, daddu, MIPS_R_A0, MIPS_R_S0, MIPS_R_ZERO); - - /* Check the error value */ - b_off = b_imm(exit_idx, ctx); - if (is_bad_offset(b_off)) { - target = j_target(ctx, exit_idx); - if (target == (unsigned int)-1) - return -E2BIG; - - if (!(ctx->offsets[this_idx] & OFFSETS_B_CONV)) { - ctx->offsets[this_idx] |= OFFSETS_B_CONV; - ctx->long_b_conversion = 1; - } - emit_instr(ctx, bne, MIPS_R_V0, MIPS_R_ZERO, 4 * 3); - emit_instr(ctx, nop); - emit_instr(ctx, j, target); - emit_instr(ctx, nop); - } else { - emit_instr(ctx, beq, MIPS_R_V0, MIPS_R_ZERO, b_off); - emit_instr(ctx, nop); - } - -#ifdef __BIG_ENDIAN - need_swap = false; -#else - need_swap = true; -#endif - dst = MIPS_R_V0; - switch (BPF_SIZE(insn->code)) { - case BPF_B: - emit_instr(ctx, lbu, dst, 0, MIPS_R_V0); - break; - case BPF_H: - emit_instr(ctx, lhu, dst, 0, MIPS_R_V0); - if (need_swap) - emit_instr(ctx, wsbh, dst, dst); - break; - case BPF_W: - emit_instr(ctx, lw, dst, 0, MIPS_R_V0); - if (need_swap) { - emit_instr(ctx, wsbh, dst, dst); - emit_instr(ctx, rotr, dst, dst, 16); - } - break; - case BPF_DW: - emit_instr(ctx, ld, dst, 0, MIPS_R_V0); - if (need_swap) { - emit_instr(ctx, dsbh, dst, dst); - emit_instr(ctx, dshd, dst, dst); - } - break; - } - - break; case BPF_ALU | BPF_END | BPF_FROM_BE: case BPF_ALU | BPF_END | BPF_FROM_LE: dst = ebpf_to_mips_reg(ctx, insn, dst_reg); diff --git a/arch/mips/netlogic/Kconfig b/arch/mips/netlogic/Kconfig index 7fcfc7fe9f14..412351c5acc6 100644 --- a/arch/mips/netlogic/Kconfig +++ b/arch/mips/netlogic/Kconfig @@ -83,10 +83,4 @@ endif config NLM_COMMON bool -config IOMMU_HELPER - bool - -config NEED_SG_DMA_LENGTH - bool - endif diff --git a/arch/mips/oprofile/op_model_mipsxx.c b/arch/mips/oprofile/op_model_mipsxx.c index c3e4c18ef8d4..7c04b17f4a48 100644 --- a/arch/mips/oprofile/op_model_mipsxx.c +++ b/arch/mips/oprofile/op_model_mipsxx.c @@ -36,7 +36,6 @@ static int perfcount_irq; #endif #ifdef CONFIG_MIPS_MT_SMP -static int cpu_has_mipsmt_pertccounters; #define WHAT (MIPS_PERFCTRL_MT_EN_VPE | \ M_PERFCTL_VPEID(cpu_vpe_id(¤t_cpu_data))) #define vpe_id() (cpu_has_mipsmt_pertccounters ? \ @@ -326,7 +325,6 @@ static int __init mipsxx_init(void) } #ifdef CONFIG_MIPS_MT_SMP - cpu_has_mipsmt_pertccounters = read_c0_config7() & (1<<19); if (!cpu_has_mipsmt_pertccounters) counters = counters_total_to_per_cpu(counters); #endif diff --git a/arch/mips/pci/ops-pmcmsp.c b/arch/mips/pci/ops-pmcmsp.c index dd2d9f7e9412..7649372103af 100644 --- a/arch/mips/pci/ops-pmcmsp.c +++ b/arch/mips/pci/ops-pmcmsp.c @@ -83,18 +83,6 @@ static int show_msp_pci_counts(struct seq_file *m, void *v) return 0; } -static int msp_pci_rd_cnt_open(struct inode *inode, struct file *file) -{ - return single_open(file, show_msp_pci_counts, NULL); -} - -static const struct file_operations msp_pci_rd_cnt_fops = { - .open = msp_pci_rd_cnt_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - /***************************************************************************** * * FUNCTION: gen_pci_cfg_wr_show @@ -160,18 +148,6 @@ static int gen_pci_cfg_wr_show(struct seq_file *m, void *v) return 0; } -static int gen_pci_cfg_wr_open(struct inode *inode, struct file *file) -{ - return single_open(file, gen_pci_cfg_wr_show, NULL); -} - -static const struct file_operations gen_pci_cfg_wr_fops = { - .open = gen_pci_cfg_wr_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - /***************************************************************************** * * FUNCTION: pci_proc_init @@ -188,8 +164,8 @@ static const struct file_operations gen_pci_cfg_wr_fops = { ****************************************************************************/ static void pci_proc_init(void) { - proc_create("pmc_msp_pci_rd_cnt", 0, NULL, &msp_pci_rd_cnt_fops); - proc_create("pmc_msp_pci_cfg_wr", 0, NULL, &gen_pci_cfg_wr_fops); + proc_create_single("pmc_msp_pci_rd_cnt", 0, NULL, show_msp_pci_counts); + proc_create_single("pmc_msp_pci_cfg_wr", 0, NULL, gen_pci_cfg_wr_show); } #endif /* CONFIG_PROC_FS && PCI_COUNTERS */ diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c index 0c65c38e05d6..f1e92bf743c2 100644 --- a/arch/mips/pci/pci-legacy.c +++ b/arch/mips/pci/pci-legacy.c @@ -263,9 +263,8 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask) (!(r->flags & IORESOURCE_ROM_ENABLE))) continue; if (!r->start && r->end) { - printk(KERN_ERR "PCI: Device %s not available " - "because of resource collisions\n", - pci_name(dev)); + pci_err(dev, + "can't enable device: resource collisions\n"); return -EINVAL; } if (r->flags & IORESOURCE_IO) @@ -274,8 +273,7 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask) cmd |= PCI_COMMAND_MEMORY; } if (cmd != old_cmd) { - printk("PCI: Enabling device %s (%04x -> %04x)\n", - pci_name(dev), old_cmd, cmd); + pci_info(dev, "enabling device (%04x -> %04x)\n", old_cmd, cmd); pci_write_config_word(dev, PCI_COMMAND, cmd); } return 0; diff --git a/arch/mips/sibyte/common/bus_watcher.c b/arch/mips/sibyte/common/bus_watcher.c index a4e55999ecb4..4bb85de9229b 100644 --- a/arch/mips/sibyte/common/bus_watcher.c +++ b/arch/mips/sibyte/common/bus_watcher.c @@ -142,24 +142,12 @@ static int bw_proc_show(struct seq_file *m, void *v) return 0; } -static int bw_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, bw_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations bw_proc_fops = { - .open = bw_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static void create_proc_decoder(struct bw_stats_struct *stats) { struct proc_dir_entry *ent; - ent = proc_create_data("bus_watcher", S_IWUSR | S_IRUGO, NULL, - &bw_proc_fops, stats); + ent = proc_create_single_data("bus_watcher", S_IWUSR | S_IRUGO, NULL, + bw_proc_show, stats); if (!ent) { printk(KERN_INFO "Unable to initialize bus_watcher /proc entry\n"); return; diff --git a/arch/mips/sibyte/swarm/rtc_m41t81.c b/arch/mips/sibyte/swarm/rtc_m41t81.c index e62466445f08..4ac8ccdf56bb 100644 --- a/arch/mips/sibyte/swarm/rtc_m41t81.c +++ b/arch/mips/sibyte/swarm/rtc_m41t81.c @@ -141,13 +141,13 @@ static int m41t81_write(uint8_t addr, int b) return 0; } -int m41t81_set_time(unsigned long t) +int m41t81_set_time(time64_t t) { struct rtc_time tm; unsigned long flags; /* Note we don't care about the century */ - rtc_time_to_tm(t, &tm); + rtc_time64_to_tm(t, &tm); /* * Note the write order matters as it ensures the correctness. @@ -188,7 +188,7 @@ int m41t81_set_time(unsigned long t) return 0; } -unsigned long m41t81_get_time(void) +time64_t m41t81_get_time(void) { unsigned int year, mon, day, hour, min, sec; unsigned long flags; @@ -218,7 +218,7 @@ unsigned long m41t81_get_time(void) year += 2000; - return mktime(year, mon, day, hour, min, sec); + return mktime64(year, mon, day, hour, min, sec); } int m41t81_probe(void) diff --git a/arch/mips/sibyte/swarm/rtc_xicor1241.c b/arch/mips/sibyte/swarm/rtc_xicor1241.c index 50a82c495427..2dcaaa7e3bfa 100644 --- a/arch/mips/sibyte/swarm/rtc_xicor1241.c +++ b/arch/mips/sibyte/swarm/rtc_xicor1241.c @@ -109,13 +109,13 @@ static int xicor_write(uint8_t addr, int b) } } -int xicor_set_time(unsigned long t) +int xicor_set_time(time64_t t) { struct rtc_time tm; int tmp; unsigned long flags; - rtc_time_to_tm(t, &tm); + rtc_time64_to_tm(t, &tm); tm.tm_year += 1900; spin_lock_irqsave(&rtc_lock, flags); @@ -168,7 +168,7 @@ int xicor_set_time(unsigned long t) return 0; } -unsigned long xicor_get_time(void) +time64_t xicor_get_time(void) { unsigned int year, mon, day, hour, min, sec, y2k; unsigned long flags; @@ -201,7 +201,7 @@ unsigned long xicor_get_time(void) year += (y2k * 100); - return mktime(year, mon, day, hour, min, sec); + return mktime64(year, mon, day, hour, min, sec); } int xicor_probe(void) diff --git a/arch/mips/sibyte/swarm/setup.c b/arch/mips/sibyte/swarm/setup.c index 494fb0a475ac..152ca71cc2d7 100644 --- a/arch/mips/sibyte/swarm/setup.c +++ b/arch/mips/sibyte/swarm/setup.c @@ -57,12 +57,12 @@ extern void sb1250_setup(void); #endif extern int xicor_probe(void); -extern int xicor_set_time(unsigned long); -extern unsigned long xicor_get_time(void); +extern int xicor_set_time(time64_t); +extern time64_t xicor_get_time(void); extern int m41t81_probe(void); -extern int m41t81_set_time(unsigned long); -extern unsigned long m41t81_get_time(void); +extern int m41t81_set_time(time64_t); +extern time64_t m41t81_get_time(void); const char *get_system_type(void) { @@ -87,9 +87,9 @@ enum swarm_rtc_type { enum swarm_rtc_type swarm_rtc_type; -void read_persistent_clock(struct timespec *ts) +void read_persistent_clock64(struct timespec64 *ts) { - unsigned long sec; + time64_t sec; switch (swarm_rtc_type) { case RTC_XICOR: @@ -102,15 +102,17 @@ void read_persistent_clock(struct timespec *ts) case RTC_NONE: default: - sec = mktime(2000, 1, 1, 0, 0, 0); + sec = mktime64(2000, 1, 1, 0, 0, 0); break; } ts->tv_sec = sec; ts->tv_nsec = 0; } -int rtc_mips_set_time(unsigned long sec) +int update_persistent_clock64(struct timespec64 now) { + time64_t sec = now.tv_sec; + switch (swarm_rtc_type) { case RTC_XICOR: return xicor_set_time(sec); diff --git a/arch/mips/sni/time.c b/arch/mips/sni/time.c index 0eb7d1e8821b..dbace1f3e1a9 100644 --- a/arch/mips/sni/time.c +++ b/arch/mips/sni/time.c @@ -171,9 +171,3 @@ void __init plat_time_init(void) } setup_pit_timer(); } - -void read_persistent_clock(struct timespec *ts) -{ - ts->tv_sec = -1; - ts->tv_nsec = 0; -} diff --git a/arch/mips/txx9/rbtx4939/setup.c b/arch/mips/txx9/rbtx4939/setup.c index fd26fadc8617..ef29a9c2ffd6 100644 --- a/arch/mips/txx9/rbtx4939/setup.c +++ b/arch/mips/txx9/rbtx4939/setup.c @@ -219,7 +219,7 @@ static int __init rbtx4939_led_probe(struct platform_device *pdev) "nand-disk", }; - leds_data = kzalloc(sizeof(*leds_data) * RBTX4939_MAX_7SEGLEDS, + leds_data = kcalloc(RBTX4939_MAX_7SEGLEDS, sizeof(*leds_data), GFP_KERNEL); if (!leds_data) return -ENOMEM; diff --git a/arch/nds32/Kconfig b/arch/nds32/Kconfig index b7404f2dcf5b..6aed974276d8 100644 --- a/arch/nds32/Kconfig +++ b/arch/nds32/Kconfig @@ -5,10 +5,13 @@ config NDS32 def_bool y + select ARCH_HAS_SYNC_DMA_FOR_CPU + select ARCH_HAS_SYNC_DMA_FOR_DEVICE select ARCH_WANT_FRAME_POINTERS if FTRACE select CLKSRC_MMIO select CLONE_BACKWARDS select COMMON_CLK + select DMA_NONCOHERENT_OPS select GENERIC_ASHLDI3 select GENERIC_ASHRDI3 select GENERIC_LSHRDI3 diff --git a/arch/nds32/include/asm/Kbuild b/arch/nds32/include/asm/Kbuild index 142e612aa639..dbc4e5422550 100644 --- a/arch/nds32/include/asm/Kbuild +++ b/arch/nds32/include/asm/Kbuild @@ -9,10 +9,12 @@ generic-y += checksum.h generic-y += clkdev.h generic-y += cmpxchg.h generic-y += cmpxchg-local.h +generic-y += compat.h generic-y += cputime.h generic-y += device.h generic-y += div64.h generic-y += dma.h +generic-y += dma-mapping.h generic-y += emergency-restart.h generic-y += errno.h generic-y += exec.h diff --git a/arch/nds32/include/asm/dma-mapping.h b/arch/nds32/include/asm/dma-mapping.h deleted file mode 100644 index 2dd47d245c25..000000000000 --- a/arch/nds32/include/asm/dma-mapping.h +++ /dev/null @@ -1,14 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -// Copyright (C) 2005-2017 Andes Technology Corporation - -#ifndef ASMNDS32_DMA_MAPPING_H -#define ASMNDS32_DMA_MAPPING_H - -extern struct dma_map_ops nds32_dma_ops; - -static inline struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus) -{ - return &nds32_dma_ops; -} - -#endif diff --git a/arch/nds32/kernel/dma.c b/arch/nds32/kernel/dma.c index d291800fc621..d0dbd4fe9645 100644 --- a/arch/nds32/kernel/dma.c +++ b/arch/nds32/kernel/dma.c @@ -3,17 +3,14 @@ #include #include -#include #include -#include -#include +#include #include #include #include #include #include #include -#include #include /* @@ -22,11 +19,6 @@ static pte_t *consistent_pte; static DEFINE_RAW_SPINLOCK(consistent_lock); -enum master_type { - FOR_CPU = 0, - FOR_DEVICE = 1, -}; - /* * VM region handling support. * @@ -124,10 +116,8 @@ out: return c; } -/* FIXME: attrs is not used. */ -static void *nds32_dma_alloc_coherent(struct device *dev, size_t size, - dma_addr_t * handle, gfp_t gfp, - unsigned long attrs) +void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle, + gfp_t gfp, unsigned long attrs) { struct page *page; struct arch_vm_region *c; @@ -232,8 +222,8 @@ no_page: return NULL; } -static void nds32_dma_free(struct device *dev, size_t size, void *cpu_addr, - dma_addr_t handle, unsigned long attrs) +void arch_dma_free(struct device *dev, size_t size, void *cpu_addr, + dma_addr_t handle, unsigned long attrs) { struct arch_vm_region *c; unsigned long flags, addr; @@ -333,145 +323,69 @@ static int __init consistent_init(void) } core_initcall(consistent_init); -static void consistent_sync(void *vaddr, size_t size, int direction, int master_type); -static dma_addr_t nds32_dma_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, - enum dma_data_direction dir, - unsigned long attrs) -{ - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - consistent_sync((void *)(page_address(page) + offset), size, dir, FOR_DEVICE); - return page_to_phys(page) + offset; -} - -static void nds32_dma_unmap_page(struct device *dev, dma_addr_t handle, - size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - consistent_sync(phys_to_virt(handle), size, dir, FOR_CPU); -} - -/* - * Make an area consistent for devices. - */ -static void consistent_sync(void *vaddr, size_t size, int direction, int master_type) -{ - unsigned long start = (unsigned long)vaddr; - unsigned long end = start + size; - - if (master_type == FOR_CPU) { - switch (direction) { - case DMA_TO_DEVICE: - break; - case DMA_FROM_DEVICE: - case DMA_BIDIRECTIONAL: - cpu_dma_inval_range(start, end); - break; - default: - BUG(); - } - } else { - /* FOR_DEVICE */ - switch (direction) { - case DMA_FROM_DEVICE: - break; - case DMA_TO_DEVICE: - case DMA_BIDIRECTIONAL: - cpu_dma_wb_range(start, end); - break; - default: - BUG(); - } - } -} -static int nds32_dma_map_sg(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, - unsigned long attrs) +static inline void cache_op(phys_addr_t paddr, size_t size, + void (*fn)(unsigned long start, unsigned long end)) { - int i; + struct page *page = pfn_to_page(paddr >> PAGE_SHIFT); + unsigned offset = paddr & ~PAGE_MASK; + size_t left = size; + unsigned long start; - for (i = 0; i < nents; i++, sg++) { - void *virt; - unsigned long pfn; - struct page *page = sg_page(sg); + do { + size_t len = left; - sg->dma_address = sg_phys(sg); - pfn = page_to_pfn(page) + sg->offset / PAGE_SIZE; - page = pfn_to_page(pfn); if (PageHighMem(page)) { - virt = kmap_atomic(page); - consistent_sync(virt, sg->length, dir, FOR_CPU); - kunmap_atomic(virt); + void *addr; + + if (offset + len > PAGE_SIZE) { + if (offset >= PAGE_SIZE) { + page += offset >> PAGE_SHIFT; + offset &= ~PAGE_MASK; + } + len = PAGE_SIZE - offset; + } + + addr = kmap_atomic(page); + start = (unsigned long)(addr + offset); + fn(start, start + len); + kunmap_atomic(addr); } else { - if (sg->offset > PAGE_SIZE) - panic("sg->offset:%08x > PAGE_SIZE\n", - sg->offset); - virt = page_address(page) + sg->offset; - consistent_sync(virt, sg->length, dir, FOR_CPU); + start = (unsigned long)phys_to_virt(paddr); + fn(start, start + size); } - } - return nents; + offset = 0; + page++; + left -= len; + } while (left); } -static void nds32_dma_unmap_sg(struct device *dev, struct scatterlist *sg, - int nhwentries, enum dma_data_direction dir, - unsigned long attrs) +void arch_sync_dma_for_device(struct device *dev, phys_addr_t paddr, + size_t size, enum dma_data_direction dir) { -} - -static void -nds32_dma_sync_single_for_cpu(struct device *dev, dma_addr_t handle, - size_t size, enum dma_data_direction dir) -{ - consistent_sync((void *)phys_to_virt(handle), size, dir, FOR_CPU); -} - -static void -nds32_dma_sync_single_for_device(struct device *dev, dma_addr_t handle, - size_t size, enum dma_data_direction dir) -{ - consistent_sync((void *)phys_to_virt(handle), size, dir, FOR_DEVICE); -} - -static void -nds32_dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, int nents, - enum dma_data_direction dir) -{ - int i; - - for (i = 0; i < nents; i++, sg++) { - char *virt = - page_address((struct page *)sg->page_link) + sg->offset; - consistent_sync(virt, sg->length, dir, FOR_CPU); + switch (dir) { + case DMA_FROM_DEVICE: + break; + case DMA_TO_DEVICE: + case DMA_BIDIRECTIONAL: + cache_op(paddr, size, cpu_dma_wb_range); + break; + default: + BUG(); } } -static void -nds32_dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir) +void arch_sync_dma_for_cpu(struct device *dev, phys_addr_t paddr, + size_t size, enum dma_data_direction dir) { - int i; - - for (i = 0; i < nents; i++, sg++) { - char *virt = - page_address((struct page *)sg->page_link) + sg->offset; - consistent_sync(virt, sg->length, dir, FOR_DEVICE); + switch (dir) { + case DMA_TO_DEVICE: + break; + case DMA_FROM_DEVICE: + case DMA_BIDIRECTIONAL: + cache_op(paddr, size, cpu_dma_inval_range); + break; + default: + BUG(); } } - -struct dma_map_ops nds32_dma_ops = { - .alloc = nds32_dma_alloc_coherent, - .free = nds32_dma_free, - .map_page = nds32_dma_map_page, - .unmap_page = nds32_dma_unmap_page, - .map_sg = nds32_dma_map_sg, - .unmap_sg = nds32_dma_unmap_sg, - .sync_single_for_device = nds32_dma_sync_single_for_device, - .sync_single_for_cpu = nds32_dma_sync_single_for_cpu, - .sync_sg_for_cpu = nds32_dma_sync_sg_for_cpu, - .sync_sg_for_device = nds32_dma_sync_sg_for_device, -}; - -EXPORT_SYMBOL(nds32_dma_ops); diff --git a/arch/nds32/kernel/traps.c b/arch/nds32/kernel/traps.c index 6e34eb9824a4..a6205fd4db52 100644 --- a/arch/nds32/kernel/traps.c +++ b/arch/nds32/kernel/traps.c @@ -222,19 +222,13 @@ void die_if_kernel(const char *str, struct pt_regs *regs, int err) int bad_syscall(int n, struct pt_regs *regs) { - siginfo_t info; - if (current->personality != PER_LINUX) { send_sig(SIGSEGV, current, 1); return regs->uregs[0]; } - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_ILLTRP; - info.si_addr = (void __user *)instruction_pointer(regs) - 4; - - force_sig_info(SIGILL, &info, current); + force_sig_fault(SIGILL, ILL_ILLTRP, + (void __user *)instruction_pointer(regs) - 4, current); die_if_kernel("Oops - bad syscall", regs, n); return regs->uregs[0]; } @@ -287,16 +281,11 @@ void __init early_trap_init(void) void send_sigtrap(struct task_struct *tsk, struct pt_regs *regs, int error_code, int si_code) { - struct siginfo info; - tsk->thread.trap_no = ENTRY_DEBUG_RELATED; tsk->thread.error_code = error_code; - memset(&info, 0, sizeof(info)); - info.si_signo = SIGTRAP; - info.si_code = si_code; - info.si_addr = (void __user *)instruction_pointer(regs); - force_sig_info(SIGTRAP, &info, tsk); + force_sig_fault(SIGTRAP, si_code, + (void __user *)instruction_pointer(regs), tsk); } void do_debug_trap(unsigned long entry, unsigned long addr, @@ -318,29 +307,22 @@ void do_debug_trap(unsigned long entry, unsigned long addr, void unhandled_interruption(struct pt_regs *regs) { - siginfo_t si; pr_emerg("unhandled_interruption\n"); show_regs(regs); if (!user_mode(regs)) do_exit(SIGKILL); - si.si_signo = SIGKILL; - si.si_errno = 0; - force_sig_info(SIGKILL, &si, current); + force_sig(SIGKILL, current); } void unhandled_exceptions(unsigned long entry, unsigned long addr, unsigned long type, struct pt_regs *regs) { - siginfo_t si; pr_emerg("Unhandled Exception: entry: %lx addr:%lx itype:%lx\n", entry, addr, type); show_regs(regs); if (!user_mode(regs)) do_exit(SIGKILL); - si.si_signo = SIGKILL; - si.si_errno = 0; - si.si_addr = (void *)addr; - force_sig_info(SIGKILL, &si, current); + force_sig(SIGKILL, current); } extern int do_page_fault(unsigned long entry, unsigned long addr, @@ -363,14 +345,11 @@ void do_dispatch_tlb_misc(unsigned long entry, unsigned long addr, void do_revinsn(struct pt_regs *regs) { - siginfo_t si; pr_emerg("Reserved Instruction\n"); show_regs(regs); if (!user_mode(regs)) do_exit(SIGILL); - si.si_signo = SIGILL; - si.si_errno = 0; - force_sig_info(SIGILL, &si, current); + force_sig(SIGILL, current); } #ifdef CONFIG_ALIGNMENT_TRAP diff --git a/arch/nds32/mm/fault.c b/arch/nds32/mm/fault.c index 3a246fb8098c..9bdb7c3ecbb6 100644 --- a/arch/nds32/mm/fault.c +++ b/arch/nds32/mm/fault.c @@ -72,7 +72,7 @@ void do_page_fault(unsigned long entry, unsigned long addr, struct task_struct *tsk; struct mm_struct *mm; struct vm_area_struct *vma; - siginfo_t info; + int si_code; int fault; unsigned int mask = VM_READ | VM_WRITE | VM_EXEC; unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; @@ -80,7 +80,7 @@ void do_page_fault(unsigned long entry, unsigned long addr, error_code = error_code & (ITYPE_mskINST | ITYPE_mskETYPE); tsk = current; mm = tsk->mm; - info.si_code = SEGV_MAPERR; + si_code = SEGV_MAPERR; /* * We fault-in kernel-space virtual memory on-demand. The * 'reference' page table is init_mm.pgd. @@ -161,7 +161,7 @@ retry: */ good_area: - info.si_code = SEGV_ACCERR; + si_code = SEGV_ACCERR; /* first do some preliminary protection checks */ if (entry == ENTRY_PTE_NOT_PRESENT) { @@ -266,11 +266,7 @@ bad_area_nosemaphore: tsk->thread.address = addr; tsk->thread.error_code = error_code; tsk->thread.trap_no = entry; - info.si_signo = SIGSEGV; - info.si_errno = 0; - /* info.si_code has been set above */ - info.si_addr = (void *)addr; - force_sig_info(SIGSEGV, &info, tsk); + force_sig_fault(SIGSEGV, si_code, (void __user *)addr, tsk); return; } @@ -339,11 +335,7 @@ do_sigbus: tsk->thread.address = addr; tsk->thread.error_code = error_code; tsk->thread.trap_no = entry; - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRERR; - info.si_addr = (void *)addr; - force_sig_info(SIGBUS, &info, tsk); + force_sig_fault(SIGBUS, BUS_ADRERR, (void __user *)addr, tsk); return; diff --git a/arch/nios2/include/asm/Kbuild b/arch/nios2/include/asm/Kbuild index d232da2cbb38..64ed3d656956 100644 --- a/arch/nios2/include/asm/Kbuild +++ b/arch/nios2/include/asm/Kbuild @@ -4,6 +4,7 @@ generic-y += bitops.h generic-y += bug.h generic-y += bugs.h generic-y += cmpxchg.h +generic-y += compat.h generic-y += current.h generic-y += device.h generic-y += div64.h diff --git a/arch/nios2/kernel/traps.c b/arch/nios2/kernel/traps.c index 8184e7d6b385..3bc3cd22b750 100644 --- a/arch/nios2/kernel/traps.c +++ b/arch/nios2/kernel/traps.c @@ -26,13 +26,7 @@ static DEFINE_SPINLOCK(die_lock); static void _send_sig(int signo, int code, unsigned long addr) { - siginfo_t info; - - info.si_signo = signo; - info.si_errno = 0; - info.si_code = code; - info.si_addr = (void __user *) addr; - force_sig_info(signo, &info, current); + force_sig_fault(signo, code, (void __user *) addr, current); } void die(const char *str, struct pt_regs *regs, long err) diff --git a/arch/openrisc/Makefile b/arch/openrisc/Makefile index cf8802962864..89076a66eee2 100644 --- a/arch/openrisc/Makefile +++ b/arch/openrisc/Makefile @@ -25,7 +25,6 @@ LDFLAGS_vmlinux := LIBGCC := $(shell $(CC) $(KBUILD_CFLAGS) -print-libgcc-file-name) KBUILD_CFLAGS += -pipe -ffixed-r10 -D__linux__ -CHECKFLAGS += -mbig-endian ifeq ($(CONFIG_OPENRISC_HAVE_INST_MUL),y) KBUILD_CFLAGS += $(call cc-option,-mhard-mul) diff --git a/arch/openrisc/include/asm/Kbuild b/arch/openrisc/include/asm/Kbuild index f05c722a21f8..65964d390b10 100644 --- a/arch/openrisc/include/asm/Kbuild +++ b/arch/openrisc/include/asm/Kbuild @@ -2,6 +2,7 @@ generic-y += barrier.h generic-y += bug.h generic-y += bugs.h generic-y += checksum.h +generic-y += compat.h generic-y += current.h generic-y += device.h generic-y += div64.h diff --git a/arch/openrisc/kernel/dma.c b/arch/openrisc/kernel/dma.c index a945f00011b4..ec7fd45704d2 100644 --- a/arch/openrisc/kernel/dma.c +++ b/arch/openrisc/kernel/dma.c @@ -247,14 +247,3 @@ const struct dma_map_ops or1k_dma_map_ops = { .sync_single_for_device = or1k_sync_single_for_device, }; EXPORT_SYMBOL(or1k_dma_map_ops); - -/* Number of entries preallocated for DMA-API debugging */ -#define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16) - -static int __init dma_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - - return 0; -} -fs_initcall(dma_init); diff --git a/arch/openrisc/kernel/traps.c b/arch/openrisc/kernel/traps.c index 113c175fe469..fac246e6f37a 100644 --- a/arch/openrisc/kernel/traps.c +++ b/arch/openrisc/kernel/traps.c @@ -250,27 +250,16 @@ void __init trap_init(void) asmlinkage void do_trap(struct pt_regs *regs, unsigned long address) { - siginfo_t info; - memset(&info, 0, sizeof(info)); - info.si_signo = SIGTRAP; - info.si_code = TRAP_TRACE; - info.si_addr = (void *)address; - force_sig_info(SIGTRAP, &info, current); + force_sig_fault(SIGTRAP, TRAP_TRACE, (void __user *)address, current); regs->pc += 4; } asmlinkage void do_unaligned_access(struct pt_regs *regs, unsigned long address) { - siginfo_t info; - if (user_mode(regs)) { /* Send a SIGBUS */ - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRALN; - info.si_addr = (void __user *)address; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGBUS, BUS_ADRALN, (void __user *)address, current); } else { printk("KERNEL: Unaligned Access 0x%.8lx\n", address); show_registers(regs); @@ -281,15 +270,9 @@ asmlinkage void do_unaligned_access(struct pt_regs *regs, unsigned long address) asmlinkage void do_bus_fault(struct pt_regs *regs, unsigned long address) { - siginfo_t info; - if (user_mode(regs)) { /* Send a SIGBUS */ - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRERR; - info.si_addr = (void *)address; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGBUS, BUS_ADRERR, (void __user *)address, current); } else { /* Kernel mode */ printk("KERNEL: Bus error (SIGBUS) 0x%.8lx\n", address); show_registers(regs); @@ -464,7 +447,6 @@ static inline void simulate_swa(struct pt_regs *regs, unsigned long address, asmlinkage void do_illegal_instruction(struct pt_regs *regs, unsigned long address) { - siginfo_t info; unsigned int op; unsigned int insn = *((unsigned int *)address); @@ -485,11 +467,7 @@ asmlinkage void do_illegal_instruction(struct pt_regs *regs, if (user_mode(regs)) { /* Send a SIGILL */ - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_ILLOPC; - info.si_addr = (void *)address; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGILL, ILL_ILLOPC, (void __user *)address, current); } else { /* Kernel mode */ printk("KERNEL: Illegal instruction (SIGILL) 0x%.8lx\n", address); diff --git a/arch/openrisc/mm/fault.c b/arch/openrisc/mm/fault.c index d0021dfae20a..9f011d16cc46 100644 --- a/arch/openrisc/mm/fault.c +++ b/arch/openrisc/mm/fault.c @@ -52,7 +52,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address, struct task_struct *tsk; struct mm_struct *mm; struct vm_area_struct *vma; - siginfo_t info; + int si_code; int fault; unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; @@ -97,7 +97,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address, } mm = tsk->mm; - info.si_code = SEGV_MAPERR; + si_code = SEGV_MAPERR; /* * If we're in an interrupt or have no user @@ -139,7 +139,7 @@ retry: */ good_area: - info.si_code = SEGV_ACCERR; + si_code = SEGV_ACCERR; /* first do some preliminary protection checks */ @@ -213,11 +213,7 @@ bad_area_nosemaphore: /* User mode accesses just cause a SIGSEGV */ if (user_mode(regs)) { - info.si_signo = SIGSEGV; - info.si_errno = 0; - /* info.si_code has been set above */ - info.si_addr = (void *)address; - force_sig_info(SIGSEGV, &info, tsk); + force_sig_fault(SIGSEGV, si_code, (void __user *)address, tsk); return; } @@ -282,11 +278,7 @@ do_sigbus: * Send a sigbus, regardless of whether we were in kernel * or user mode. */ - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRERR; - info.si_addr = (void *)address; - force_sig_info(SIGBUS, &info, tsk); + force_sig_fault(SIGBUS, BUS_ADRERR, (void __user *)address, tsk); /* Kernel mode? Handle exceptions or die */ if (!user_mode(regs)) diff --git a/arch/parisc/Kconfig b/arch/parisc/Kconfig index fc5a574c3482..c480770fabcd 100644 --- a/arch/parisc/Kconfig +++ b/arch/parisc/Kconfig @@ -51,6 +51,8 @@ config PARISC select GENERIC_CLOCKEVENTS select ARCH_NO_COHERENT_DMA_MMAP select CPU_NO_EFFICIENT_FFS + select NEED_DMA_MAP_STATE + select NEED_SG_DMA_LENGTH help The PA-RISC microprocessor is designed by Hewlett-Packard and used @@ -111,12 +113,6 @@ config PM config STACKTRACE_SUPPORT def_bool y -config NEED_DMA_MAP_STATE - def_bool y - -config NEED_SG_DMA_LENGTH - def_bool y - config ISA_DMA_API bool @@ -279,7 +275,7 @@ config SMP machines, but will use only one CPU of a multiprocessor machine. On a uniprocessor machine, the kernel will run faster if you say N. - See also and the SMP-HOWTO + See also and the SMP-HOWTO available at . If you don't know what to do here, say N. diff --git a/arch/parisc/Makefile b/arch/parisc/Makefile index 34ac503e28ad..714284ea6cc2 100644 --- a/arch/parisc/Makefile +++ b/arch/parisc/Makefile @@ -22,13 +22,13 @@ KBUILD_IMAGE := vmlinuz KBUILD_DEFCONFIG := default_defconfig NM = sh $(srctree)/arch/parisc/nm -CHECKFLAGS += -D__hppa__=1 -mbig-endian +CHECKFLAGS += -D__hppa__=1 LIBGCC = $(shell $(CC) $(KBUILD_CFLAGS) -print-libgcc-file-name) export LIBGCC ifdef CONFIG_64BIT UTS_MACHINE := parisc64 -CHECKFLAGS += -D__LP64__=1 -m64 +CHECKFLAGS += -D__LP64__=1 CC_ARCHES = hppa64 LD_BFD := elf64-hppa-linux else # 32-bit diff --git a/arch/parisc/include/asm/compat.h b/arch/parisc/include/asm/compat.h index 57b8b2a2fd4e..ab8a54771507 100644 --- a/arch/parisc/include/asm/compat.h +++ b/arch/parisc/include/asm/compat.h @@ -13,7 +13,6 @@ typedef u32 compat_size_t; typedef s32 compat_ssize_t; -typedef s32 compat_time_t; typedef s32 compat_clock_t; typedef s32 compat_pid_t; typedef u32 __compat_uid_t; @@ -40,16 +39,6 @@ typedef u32 compat_ulong_t; typedef u64 compat_u64; typedef u32 compat_uptr_t; -struct compat_timespec { - compat_time_t tv_sec; - s32 tv_nsec; -}; - -struct compat_timeval { - compat_time_t tv_sec; - s32 tv_usec; -}; - struct compat_stat { compat_dev_t st_dev; /* dev_t is 32 bits on parisc */ compat_ino_t st_ino; /* 32 bits */ @@ -149,10 +138,10 @@ struct compat_ipc64_perm { struct compat_semid64_ds { struct compat_ipc64_perm sem_perm; - unsigned int __unused1; - compat_time_t sem_otime; - unsigned int __unused2; - compat_time_t sem_ctime; + unsigned int sem_otime_high; + unsigned int sem_otime; + unsigned int sem_ctime_high; + unsigned int sem_ctime; compat_ulong_t sem_nsems; compat_ulong_t __unused3; compat_ulong_t __unused4; @@ -160,12 +149,12 @@ struct compat_semid64_ds { struct compat_msqid64_ds { struct compat_ipc64_perm msg_perm; - unsigned int __unused1; - compat_time_t msg_stime; - unsigned int __unused2; - compat_time_t msg_rtime; - unsigned int __unused3; - compat_time_t msg_ctime; + unsigned int msg_stime_high; + unsigned int msg_stime; + unsigned int msg_rtime_high; + unsigned int msg_rtime; + unsigned int msg_ctime_high; + unsigned int msg_ctime; compat_ulong_t msg_cbytes; compat_ulong_t msg_qnum; compat_ulong_t msg_qbytes; @@ -177,12 +166,12 @@ struct compat_msqid64_ds { struct compat_shmid64_ds { struct compat_ipc64_perm shm_perm; - unsigned int __unused1; - compat_time_t shm_atime; - unsigned int __unused2; - compat_time_t shm_dtime; - unsigned int __unused3; - compat_time_t shm_ctime; + unsigned int shm_atime_high; + unsigned int shm_atime; + unsigned int shm_dtime_high; + unsigned int shm_dtime; + unsigned int shm_ctime_high; + unsigned int shm_ctime; unsigned int __unused4; compat_size_t shm_segsz; compat_pid_t shm_cpid; diff --git a/arch/parisc/include/asm/hardirq.h b/arch/parisc/include/asm/hardirq.h index 077815169258..1a1235a9d533 100644 --- a/arch/parisc/include/asm/hardirq.h +++ b/arch/parisc/include/asm/hardirq.h @@ -34,14 +34,6 @@ DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat); #define __IRQ_STAT(cpu, member) (irq_stat[cpu].member) #define inc_irq_stat(member) this_cpu_inc(irq_stat.member) #define __inc_irq_stat(member) __this_cpu_inc(irq_stat.member) -#define local_softirq_pending() this_cpu_read(irq_stat.__softirq_pending) - -#define __ARCH_SET_SOFTIRQ_PENDING - -#define set_softirq_pending(x) \ - this_cpu_write(irq_stat.__softirq_pending, (x)) -#define or_softirq_pending(x) this_cpu_or(irq_stat.__softirq_pending, (x)) - #define ack_bad_irq(irq) WARN(1, "unexpected IRQ trap at vector %02x\n", irq) #endif /* _PARISC_HARDIRQ_H */ diff --git a/arch/parisc/include/asm/pci.h b/arch/parisc/include/asm/pci.h index 96b7deec512d..3328fd17c19d 100644 --- a/arch/parisc/include/asm/pci.h +++ b/arch/parisc/include/asm/pci.h @@ -87,29 +87,6 @@ struct pci_hba_data { #define PCI_F_EXTEND 0UL #endif /* !CONFIG_64BIT */ -/* - * If the PCI device's view of memory is the same as the CPU's view of memory, - * PCI_DMA_BUS_IS_PHYS is true. The networking and block device layers use - * this boolean for bounce buffer decisions. - */ -#ifdef CONFIG_PA20 -/* All PA-2.0 machines have an IOMMU. */ -#define PCI_DMA_BUS_IS_PHYS 0 -#define parisc_has_iommu() do { } while (0) -#else - -#if defined(CONFIG_IOMMU_CCIO) || defined(CONFIG_IOMMU_SBA) -extern int parisc_bus_is_phys; /* in arch/parisc/kernel/setup.c */ -#define PCI_DMA_BUS_IS_PHYS parisc_bus_is_phys -#define parisc_has_iommu() do { parisc_bus_is_phys = 0; } while (0) -#else -#define PCI_DMA_BUS_IS_PHYS 1 -#define parisc_has_iommu() do { } while (0) -#endif - -#endif /* !CONFIG_PA20 */ - - /* ** Most PCI devices (eg Tulip, NCR720) also export the same registers ** to both MMIO and I/O port space. Due to poor performance of I/O Port diff --git a/arch/parisc/include/uapi/asm/msgbuf.h b/arch/parisc/include/uapi/asm/msgbuf.h index b48b810e626b..6a2e9ab2ef8d 100644 --- a/arch/parisc/include/uapi/asm/msgbuf.h +++ b/arch/parisc/include/uapi/asm/msgbuf.h @@ -10,31 +10,30 @@ * between kernel and user space. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem * - 2 miscellaneous 32-bit values */ struct msqid64_ds { struct ipc64_perm msg_perm; -#if __BITS_PER_LONG != 64 - unsigned int __pad1; -#endif +#if __BITS_PER_LONG == 64 __kernel_time_t msg_stime; /* last msgsnd time */ -#if __BITS_PER_LONG != 64 - unsigned int __pad2; -#endif __kernel_time_t msg_rtime; /* last msgrcv time */ -#if __BITS_PER_LONG != 64 - unsigned int __pad3; -#endif __kernel_time_t msg_ctime; /* last change time */ - unsigned long msg_cbytes; /* current number of bytes on queue */ - unsigned long msg_qnum; /* number of messages in queue */ - unsigned long msg_qbytes; /* max number of bytes on queue */ - __kernel_pid_t msg_lspid; /* pid of last msgsnd */ - __kernel_pid_t msg_lrpid; /* last receive pid */ - unsigned long __unused1; - unsigned long __unused2; +#else + unsigned long msg_stime_high; + unsigned long msg_stime; /* last msgsnd time */ + unsigned long msg_rtime_high; + unsigned long msg_rtime; /* last msgrcv time */ + unsigned long msg_ctime_high; + unsigned long msg_ctime; /* last change time */ +#endif + unsigned long msg_cbytes; /* current number of bytes on queue */ + unsigned long msg_qnum; /* number of messages in queue */ + unsigned long msg_qbytes; /* max number of bytes on queue */ + __kernel_pid_t msg_lspid; /* pid of last msgsnd */ + __kernel_pid_t msg_lrpid; /* last receive pid */ + unsigned long __unused1; + unsigned long __unused2; }; #endif /* _PARISC_MSGBUF_H */ diff --git a/arch/parisc/include/uapi/asm/sembuf.h b/arch/parisc/include/uapi/asm/sembuf.h index 746c5d86a9b1..3c31163b1241 100644 --- a/arch/parisc/include/uapi/asm/sembuf.h +++ b/arch/parisc/include/uapi/asm/sembuf.h @@ -10,21 +10,21 @@ * between kernel and user space. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem * - 2 miscellaneous 32-bit values */ struct semid64_ds { struct ipc64_perm sem_perm; /* permissions .. see ipc.h */ -#if __BITS_PER_LONG != 64 - unsigned int __pad1; -#endif +#if __BITS_PER_LONG == 64 __kernel_time_t sem_otime; /* last semop time */ -#if __BITS_PER_LONG != 64 - unsigned int __pad2; -#endif __kernel_time_t sem_ctime; /* last change time */ - unsigned long sem_nsems; /* no. of semaphores in array */ +#else + unsigned long sem_otime_high; + unsigned long sem_otime; /* last semop time */ + unsigned long sem_ctime_high; + unsigned long sem_ctime; /* last change time */ +#endif + unsigned long sem_nsems; /* no. of semaphores in array */ unsigned long __unused1; unsigned long __unused2; }; diff --git a/arch/parisc/include/uapi/asm/shmbuf.h b/arch/parisc/include/uapi/asm/shmbuf.h index cd4dbce55d0b..c89b3dd8db21 100644 --- a/arch/parisc/include/uapi/asm/shmbuf.h +++ b/arch/parisc/include/uapi/asm/shmbuf.h @@ -10,25 +10,22 @@ * between kernel and user space. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem * - 2 miscellaneous 32-bit values */ struct shmid64_ds { struct ipc64_perm shm_perm; /* operation perms */ -#if __BITS_PER_LONG != 64 - unsigned int __pad1; -#endif +#if __BITS_PER_LONG == 64 __kernel_time_t shm_atime; /* last attach time */ -#if __BITS_PER_LONG != 64 - unsigned int __pad2; -#endif __kernel_time_t shm_dtime; /* last detach time */ -#if __BITS_PER_LONG != 64 - unsigned int __pad3; -#endif __kernel_time_t shm_ctime; /* last change time */ -#if __BITS_PER_LONG != 64 +#else + unsigned long shm_atime_high; + unsigned long shm_atime; /* last attach time */ + unsigned long shm_dtime_high; + unsigned long shm_dtime; /* last detach time */ + unsigned long shm_ctime_high; + unsigned long shm_ctime; /* last change time */ unsigned int __pad4; #endif __kernel_size_t shm_segsz; /* size of segment (bytes) */ diff --git a/arch/parisc/kernel/pci-dma.c b/arch/parisc/kernel/pci-dma.c index 91bc0cac03a1..6df07ce4f3c2 100644 --- a/arch/parisc/kernel/pci-dma.c +++ b/arch/parisc/kernel/pci-dma.c @@ -367,19 +367,6 @@ static int proc_pcxl_dma_show(struct seq_file *m, void *v) return 0; } -static int proc_pcxl_dma_open(struct inode *inode, struct file *file) -{ - return single_open(file, proc_pcxl_dma_show, NULL); -} - -static const struct file_operations proc_pcxl_dma_ops = { - .owner = THIS_MODULE, - .open = proc_pcxl_dma_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int __init pcxl_dma_init(void) { @@ -397,8 +384,8 @@ pcxl_dma_init(void) "pcxl_dma_init: Unable to create gsc /proc dir entry\n"); else { struct proc_dir_entry* ent; - ent = proc_create("pcxl_dma", 0, proc_gsc_root, - &proc_pcxl_dma_ops); + ent = proc_create_single("pcxl_dma", 0, proc_gsc_root, + proc_pcxl_dma_show); if (!ent) printk(KERN_WARNING "pci-dma.c: Unable to create pcxl_dma /proc entry.\n"); diff --git a/arch/parisc/kernel/pdc_chassis.c b/arch/parisc/kernel/pdc_chassis.c index 3e04242de5a7..28e07482b0f1 100644 --- a/arch/parisc/kernel/pdc_chassis.c +++ b/arch/parisc/kernel/pdc_chassis.c @@ -266,18 +266,6 @@ static int pdc_chassis_warn_show(struct seq_file *m, void *v) return 0; } -static int pdc_chassis_warn_open(struct inode *inode, struct file *file) -{ - return single_open(file, pdc_chassis_warn_show, NULL); -} - -static const struct file_operations pdc_chassis_warn_fops = { - .open = pdc_chassis_warn_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int __init pdc_chassis_create_procfs(void) { unsigned long test; @@ -292,7 +280,7 @@ static int __init pdc_chassis_create_procfs(void) printk(KERN_INFO "Enabling PDC chassis warnings support v%s\n", PDC_CHASSIS_VER); - proc_create("chassis", 0400, NULL, &pdc_chassis_warn_fops); + proc_create_single("chassis", 0400, NULL, pdc_chassis_warn_show); return 0; } diff --git a/arch/parisc/kernel/ptrace.c b/arch/parisc/kernel/ptrace.c index 1a2be6e639b5..7aa1d4d0d444 100644 --- a/arch/parisc/kernel/ptrace.c +++ b/arch/parisc/kernel/ptrace.c @@ -76,8 +76,6 @@ void user_enable_single_step(struct task_struct *task) set_tsk_thread_flag(task, TIF_SINGLESTEP); if (pa_psw(task)->n) { - struct siginfo si; - /* Nullified, just crank over the queue. */ task_regs(task)->iaoq[0] = task_regs(task)->iaoq[1]; task_regs(task)->iasq[0] = task_regs(task)->iasq[1]; @@ -90,11 +88,9 @@ void user_enable_single_step(struct task_struct *task) ptrace_disable(task); /* Don't wake up the task, but let the parent know something happened. */ - si.si_code = TRAP_TRACE; - si.si_addr = (void __user *) (task_regs(task)->iaoq[0] & ~3); - si.si_signo = SIGTRAP; - si.si_errno = 0; - force_sig_info(SIGTRAP, &si, task); + force_sig_fault(SIGTRAP, TRAP_TRACE, + (void __user *) (task_regs(task)->iaoq[0] & ~3), + task); /* notify_parent(task, SIGCHLD); */ return; } diff --git a/arch/parisc/kernel/setup.c b/arch/parisc/kernel/setup.c index 0e9675f857a5..8d3a7b80ac42 100644 --- a/arch/parisc/kernel/setup.c +++ b/arch/parisc/kernel/setup.c @@ -58,11 +58,6 @@ struct proc_dir_entry * proc_runway_root __read_mostly = NULL; struct proc_dir_entry * proc_gsc_root __read_mostly = NULL; struct proc_dir_entry * proc_mckinley_root __read_mostly = NULL; -#if !defined(CONFIG_PA20) && (defined(CONFIG_IOMMU_CCIO) || defined(CONFIG_IOMMU_SBA)) -int parisc_bus_is_phys __read_mostly = 1; /* Assume no IOMMU is present */ -EXPORT_SYMBOL(parisc_bus_is_phys); -#endif - void __init setup_cmdline(char **cmdline_p) { extern unsigned int boot_args[]; diff --git a/arch/parisc/kernel/traps.c b/arch/parisc/kernel/traps.c index 71d31274d782..4309ad31a874 100644 --- a/arch/parisc/kernel/traps.c +++ b/arch/parisc/kernel/traps.c @@ -297,13 +297,8 @@ void die_if_kernel(char *str, struct pt_regs *regs, long err) #define GDB_BREAK_INSN 0x10004 static void handle_gdb_break(struct pt_regs *regs, int wot) { - struct siginfo si; - - si.si_signo = SIGTRAP; - si.si_errno = 0; - si.si_code = wot; - si.si_addr = (void __user *) (regs->iaoq[0] & ~3); - force_sig_info(SIGTRAP, &si, current); + force_sig_fault(SIGTRAP, wot, + (void __user *) (regs->iaoq[0] & ~3), current); } static void handle_break(struct pt_regs *regs) @@ -487,7 +482,7 @@ void notrace handle_interruption(int code, struct pt_regs *regs) { unsigned long fault_address = 0; unsigned long fault_space = 0; - struct siginfo si; + int si_code; if (code == 1) pdc_console_restart(); /* switch back to pdc if HPMC */ @@ -571,7 +566,7 @@ void notrace handle_interruption(int code, struct pt_regs *regs) case 8: /* Illegal instruction trap */ die_if_kernel("Illegal instruction", regs, code); - si.si_code = ILL_ILLOPC; + si_code = ILL_ILLOPC; goto give_sigill; case 9: @@ -582,7 +577,7 @@ void notrace handle_interruption(int code, struct pt_regs *regs) case 10: /* Privileged operation trap */ die_if_kernel("Privileged operation", regs, code); - si.si_code = ILL_PRVOPC; + si_code = ILL_PRVOPC; goto give_sigill; case 11: @@ -605,20 +600,16 @@ void notrace handle_interruption(int code, struct pt_regs *regs) } die_if_kernel("Privileged register usage", regs, code); - si.si_code = ILL_PRVREG; + si_code = ILL_PRVREG; give_sigill: - si.si_signo = SIGILL; - si.si_errno = 0; - si.si_addr = (void __user *) regs->iaoq[0]; - force_sig_info(SIGILL, &si, current); + force_sig_fault(SIGILL, si_code, + (void __user *) regs->iaoq[0], current); return; case 12: /* Overflow Trap, let the userland signal handler do the cleanup */ - si.si_signo = SIGFPE; - si.si_code = FPE_INTOVF; - si.si_addr = (void __user *) regs->iaoq[0]; - force_sig_info(SIGFPE, &si, current); + force_sig_fault(SIGFPE, FPE_INTOVF, + (void __user *) regs->iaoq[0], current); return; case 13: @@ -626,13 +617,11 @@ void notrace handle_interruption(int code, struct pt_regs *regs) The condition succeeds in an instruction which traps on condition */ if(user_mode(regs)){ - si.si_signo = SIGFPE; /* Let userspace app figure it out from the insn pointed * to by si_addr. */ - si.si_code = FPE_CONDTRAP; - si.si_addr = (void __user *) regs->iaoq[0]; - force_sig_info(SIGFPE, &si, current); + force_sig_fault(SIGFPE, FPE_CONDTRAP, + (void __user *) regs->iaoq[0], current); return; } /* The kernel doesn't want to handle condition codes */ @@ -741,14 +730,10 @@ void notrace handle_interruption(int code, struct pt_regs *regs) return; die_if_kernel("Protection id trap", regs, code); - si.si_code = SEGV_MAPERR; - si.si_signo = SIGSEGV; - si.si_errno = 0; - if (code == 7) - si.si_addr = (void __user *) regs->iaoq[0]; - else - si.si_addr = (void __user *) regs->ior; - force_sig_info(SIGSEGV, &si, current); + force_sig_fault(SIGSEGV, SEGV_MAPERR, + (code == 7)? + ((void __user *) regs->iaoq[0]) : + ((void __user *) regs->ior), current); return; case 28: @@ -762,11 +747,8 @@ void notrace handle_interruption(int code, struct pt_regs *regs) "handle_interruption() pid=%d command='%s'\n", task_pid_nr(current), current->comm); /* SIGBUS, for lack of a better one. */ - si.si_signo = SIGBUS; - si.si_code = BUS_OBJERR; - si.si_errno = 0; - si.si_addr = (void __user *) regs->ior; - force_sig_info(SIGBUS, &si, current); + force_sig_fault(SIGBUS, BUS_OBJERR, + (void __user *)regs->ior, current); return; } pdc_chassis_send_status(PDC_CHASSIS_DIRECT_PANIC); @@ -781,11 +763,8 @@ void notrace handle_interruption(int code, struct pt_regs *regs) "User fault %d on space 0x%08lx, pid=%d command='%s'\n", code, fault_space, task_pid_nr(current), current->comm); - si.si_signo = SIGSEGV; - si.si_errno = 0; - si.si_code = SEGV_MAPERR; - si.si_addr = (void __user *) regs->ior; - force_sig_info(SIGSEGV, &si, current); + force_sig_fault(SIGSEGV, SEGV_MAPERR, + (void __user *)regs->ior, current); return; } } diff --git a/arch/parisc/kernel/unaligned.c b/arch/parisc/kernel/unaligned.c index e36f7b75ab07..932bfc0b7cd8 100644 --- a/arch/parisc/kernel/unaligned.c +++ b/arch/parisc/kernel/unaligned.c @@ -452,7 +452,6 @@ void handle_unaligned(struct pt_regs *regs) unsigned long newbase = R1(regs->iir)?regs->gr[R1(regs->iir)]:0; int modify = 0; int ret = ERR_NOTHANDLED; - struct siginfo si; register int flop=0; /* true if this is a flop */ __inc_irq_stat(irq_unaligned_count); @@ -690,21 +689,15 @@ void handle_unaligned(struct pt_regs *regs) if (ret == ERR_PAGEFAULT) { - si.si_signo = SIGSEGV; - si.si_errno = 0; - si.si_code = SEGV_MAPERR; - si.si_addr = (void __user *)regs->ior; - force_sig_info(SIGSEGV, &si, current); + force_sig_fault(SIGSEGV, SEGV_MAPERR, + (void __user *)regs->ior, current); } else { force_sigbus: /* couldn't handle it ... */ - si.si_signo = SIGBUS; - si.si_errno = 0; - si.si_code = BUS_ADRALN; - si.si_addr = (void __user *)regs->ior; - force_sig_info(SIGBUS, &si, current); + force_sig_fault(SIGBUS, BUS_ADRALN, + (void __user *)regs->ior, current); } return; diff --git a/arch/parisc/math-emu/driver.c b/arch/parisc/math-emu/driver.c index 2fb59d2e2b29..0590e05571d1 100644 --- a/arch/parisc/math-emu/driver.c +++ b/arch/parisc/math-emu/driver.c @@ -81,7 +81,6 @@ int handle_fpe(struct pt_regs *regs) { extern void printbinary(unsigned long x, int nbits); - struct siginfo si; unsigned int orig_sw, sw; int signalcode; /* need an intermediate copy of float regs because FPU emulation @@ -117,11 +116,8 @@ handle_fpe(struct pt_regs *regs) memcpy(regs->fr, frcopy, sizeof regs->fr); if (signalcode != 0) { - si.si_signo = signalcode >> 24; - si.si_errno = 0; - si.si_code = signalcode & 0xffffff; - si.si_addr = (void __user *) regs->iaoq[0]; - force_sig_info(si.si_signo, &si, current); + force_sig_fault(signalcode >> 24, signalcode & 0xffffff, + (void __user *) regs->iaoq[0], current); return -1; } diff --git a/arch/parisc/mm/fault.c b/arch/parisc/mm/fault.c index e247edbca68e..a80117980fc2 100644 --- a/arch/parisc/mm/fault.c +++ b/arch/parisc/mm/fault.c @@ -353,23 +353,22 @@ bad_area: up_read(&mm->mmap_sem); if (user_mode(regs)) { - struct siginfo si; - unsigned int lsb = 0; + int signo, si_code; switch (code) { case 15: /* Data TLB miss fault/Data page fault */ /* send SIGSEGV when outside of vma */ if (!vma || address < vma->vm_start || address >= vma->vm_end) { - si.si_signo = SIGSEGV; - si.si_code = SEGV_MAPERR; + signo = SIGSEGV; + si_code = SEGV_MAPERR; break; } /* send SIGSEGV for wrong permissions */ if ((vma->vm_flags & acc_type) != acc_type) { - si.si_signo = SIGSEGV; - si.si_code = SEGV_ACCERR; + signo = SIGSEGV; + si_code = SEGV_ACCERR; break; } @@ -377,43 +376,40 @@ bad_area: /* fall through */ case 17: /* NA data TLB miss / page fault */ case 18: /* Unaligned access - PCXS only */ - si.si_signo = SIGBUS; - si.si_code = (code == 18) ? BUS_ADRALN : BUS_ADRERR; + signo = SIGBUS; + si_code = (code == 18) ? BUS_ADRALN : BUS_ADRERR; break; case 16: /* Non-access instruction TLB miss fault */ case 26: /* PCXL: Data memory access rights trap */ default: - si.si_signo = SIGSEGV; - si.si_code = (code == 26) ? SEGV_ACCERR : SEGV_MAPERR; + signo = SIGSEGV; + si_code = (code == 26) ? SEGV_ACCERR : SEGV_MAPERR; break; } - #ifdef CONFIG_MEMORY_FAILURE if (fault & (VM_FAULT_HWPOISON|VM_FAULT_HWPOISON_LARGE)) { + unsigned int lsb = 0; printk(KERN_ERR "MCE: Killing %s:%d due to hardware memory corruption fault at %08lx\n", tsk->comm, tsk->pid, address); - si.si_signo = SIGBUS; - si.si_code = BUS_MCEERR_AR; + /* + * Either small page or large page may be poisoned. + * In other words, VM_FAULT_HWPOISON_LARGE and + * VM_FAULT_HWPOISON are mutually exclusive. + */ + if (fault & VM_FAULT_HWPOISON_LARGE) + lsb = hstate_index_to_shift(VM_FAULT_GET_HINDEX(fault)); + else if (fault & VM_FAULT_HWPOISON) + lsb = PAGE_SHIFT; + + force_sig_mceerr(BUS_MCEERR_AR, (void __user *) address, + lsb, current); + return; } #endif + show_signal_msg(regs, code, address, tsk, vma); - /* - * Either small page or large page may be poisoned. - * In other words, VM_FAULT_HWPOISON_LARGE and - * VM_FAULT_HWPOISON are mutually exclusive. - */ - if (fault & VM_FAULT_HWPOISON_LARGE) - lsb = hstate_index_to_shift(VM_FAULT_GET_HINDEX(fault)); - else if (fault & VM_FAULT_HWPOISON) - lsb = PAGE_SHIFT; - else - show_signal_msg(regs, code, address, tsk, vma); - si.si_addr_lsb = lsb; - - si.si_errno = 0; - si.si_addr = (void __user *) address; - force_sig_info(si.si_signo, &si, current); + force_sig_fault(signo, si_code, (void __user *) address, current); return; } diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index c32a181a7cbb..9f2b75fe2c2d 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -13,12 +13,6 @@ config 64BIT bool default y if PPC64 -config ARCH_PHYS_ADDR_T_64BIT - def_bool PPC64 || PHYS_64BIT - -config ARCH_DMA_ADDR_T_64BIT - def_bool ARCH_PHYS_ADDR_T_64BIT - config MMU bool default y @@ -141,6 +135,7 @@ config PPC select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_PHYS_TO_DMA select ARCH_HAS_PMEM_API if PPC64 + select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_MEMBARRIER_CALLBACKS select ARCH_HAS_SCALED_CPUTIME if VIRT_CPU_ACCOUNTING_NATIVE select ARCH_HAS_SG_CHAIN @@ -162,6 +157,7 @@ config PPC select BUILDTIME_EXTABLE_SORT select CLONE_BACKWARDS select DCACHE_WORD_ACCESS if PPC64 && CPU_LITTLE_ENDIAN + select DYNAMIC_FTRACE if FUNCTION_TRACER select EDAC_ATOMIC_SCRUB select EDAC_SUPPORT select GENERIC_ATOMIC64 if PPC32 @@ -187,7 +183,6 @@ config PPC select HAVE_CONTEXT_TRACKING if PPC64 select HAVE_DEBUG_KMEMLEAK select HAVE_DEBUG_STACKOVERFLOW - select HAVE_DMA_API_DEBUG select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS if MPROFILE_KERNEL select HAVE_EBPF_JIT if PPC64 @@ -195,7 +190,7 @@ config PPC select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER - select HAVE_GCC_PLUGINS + select HAVE_GCC_PLUGINS if GCC_VERSION >= 50200 # plugin support on gcc <= 5.1 is buggy on PPC select HAVE_GENERIC_GUP select HAVE_HW_BREAKPOINT if PERF_EVENTS && (PPC_BOOK3S || PPC_8xx) select HAVE_IDE @@ -205,6 +200,7 @@ config PPC select HAVE_KPROBES select HAVE_KPROBES_ON_FTRACE select HAVE_KRETPROBES + select HAVE_LD_DEAD_CODE_DATA_ELIMINATION select HAVE_LIVEPATCH if HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_MEMBLOCK select HAVE_MEMBLOCK_NODE_MAP @@ -220,18 +216,23 @@ config PPC select HAVE_PERF_USER_STACK_DUMP select HAVE_RCU_TABLE_FREE if SMP select HAVE_REGS_AND_STACK_ACCESS_API + select HAVE_RELIABLE_STACKTRACE if PPC64 && CPU_LITTLE_ENDIAN select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING select HAVE_IRQ_TIME_ACCOUNTING + select HAVE_RSEQ + select IOMMU_HELPER if PPC64 select IRQ_DOMAIN select IRQ_FORCED_THREADING select MODULES_USE_ELF_RELA + select NEED_SG_DMA_LENGTH select NO_BOOTMEM select OF select OF_EARLY_FLATTREE select OF_RESERVED_MEM select OLD_SIGACTION if PPC32 select OLD_SIGSUSPEND + select RTC_LIB select SPARSE_IRQ select SYSCTL_EXCEPTION_TRACE select VIRT_TO_BUS if !PPC64 @@ -460,36 +461,9 @@ config LD_HEAD_STUB_CATCH If unsure, say "N". -config DISABLE_MPROFILE_KERNEL - bool "Disable use of mprofile-kernel for kernel tracing" - depends on PPC64 && CPU_LITTLE_ENDIAN - default y - help - Selecting this options disables use of the mprofile-kernel ABI for - kernel tracing. That will cause options such as live patching - (CONFIG_LIVEPATCH) which depend on CONFIG_DYNAMIC_FTRACE_WITH_REGS to - be disabled also. - - If you have a toolchain which supports mprofile-kernel, then you can - disable this. Otherwise leave it enabled. If you're not sure, say - "Y". - config MPROFILE_KERNEL depends on PPC64 && CPU_LITTLE_ENDIAN - def_bool !DISABLE_MPROFILE_KERNEL - -config IOMMU_HELPER - def_bool PPC64 - -config SWIOTLB - bool "SWIOTLB support" - default n - select IOMMU_HELPER - ---help--- - Support for IO bounce buffering for systems without an IOMMU. - This allows us to DMA to the full physical address space on - platforms where the size of a physical address is larger - than the bus address. Not all platforms support this. + def_bool $(success,$(srctree)/arch/powerpc/tools/gcc-check-mprofile-kernel.sh $(CC) -I$(srctree)/include -D__KERNEL__) config HOTPLUG_CPU bool "Support for enabling/disabling CPUs" @@ -883,7 +857,7 @@ config PPC_MEM_KEYS page-based protections, but without requiring modification of the page tables when an application changes protection domains. - For details, see Documentation/vm/protection-keys.txt + For details, see Documentation/vm/protection-keys.rst If unsure, say y. @@ -913,9 +887,6 @@ config ZONE_DMA config NEED_DMA_MAP_STATE def_bool (PPC64 || NOT_COHERENT_CACHE) -config NEED_SG_DMA_LENGTH - def_bool y - config GENERIC_ISA_DMA bool depends on ISA_DMA_API diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile index 95813df90801..bd06a3ccda31 100644 --- a/arch/powerpc/Makefile +++ b/arch/powerpc/Makefile @@ -17,17 +17,18 @@ HAS_BIARCH := $(call cc-option-yn, -m32) # Set default 32 bits cross compilers for vdso and boot wrapper CROSS32_COMPILE ?= -CROSS32CC := $(CROSS32_COMPILE)gcc -CROSS32AR := $(CROSS32_COMPILE)ar - ifeq ($(HAS_BIARCH),y) ifeq ($(CROSS32_COMPILE),) -CROSS32CC := $(CC) -m32 -KBUILD_ARFLAGS += --target=elf32-powerpc +ifdef CONFIG_PPC32 +# These options will be overridden by any -mcpu option that the CPU +# or platform code sets later on the command line, but they are needed +# to set a sane 32-bit cpu target for the 64-bit cross compiler which +# may default to the wrong ISA. +KBUILD_CFLAGS += -mcpu=powerpc +KBUILD_AFLAGS += -mcpu=powerpc +endif endif endif - -export CROSS32CC CROSS32AR ifeq ($(CROSS_COMPILE),) KBUILD_DEFCONFIG := $(shell uname -m)_defconfig @@ -74,13 +75,15 @@ endif endif ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y) -override LD += -EL +KBUILD_CFLAGS += -mlittle-endian +LDFLAGS += -EL LDEMULATION := lppc GNUTARGET := powerpcle MULTIPLEWORD := -mno-multiple KBUILD_CFLAGS_MODULE += $(call cc-option,-mno-save-toc-indirect) else -override LD += -EB +KBUILD_CFLAGS += $(call cc-option,-mbig-endian) +LDFLAGS += -EB LDEMULATION := ppc GNUTARGET := powerpc MULTIPLEWORD := -mmultiple @@ -93,19 +96,19 @@ aflags-$(CONFIG_CPU_BIG_ENDIAN) += $(call cc-option,-mabi=elfv1) aflags-$(CONFIG_CPU_LITTLE_ENDIAN) += -mabi=elfv2 endif -cflags-$(CONFIG_CPU_LITTLE_ENDIAN) += -mlittle-endian -cflags-$(CONFIG_CPU_BIG_ENDIAN) += $(call cc-option,-mbig-endian) ifneq ($(cc-name),clang) cflags-$(CONFIG_CPU_LITTLE_ENDIAN) += -mno-strict-align endif +cflags-$(CONFIG_CPU_BIG_ENDIAN) += $(call cc-option,-mbig-endian) +cflags-$(CONFIG_CPU_LITTLE_ENDIAN) += -mlittle-endian aflags-$(CONFIG_CPU_BIG_ENDIAN) += $(call cc-option,-mbig-endian) aflags-$(CONFIG_CPU_LITTLE_ENDIAN) += -mlittle-endian ifeq ($(HAS_BIARCH),y) -override AS += -a$(BITS) -override LD += -m elf$(BITS)$(LDEMULATION) -override CC += -m$(BITS) +KBUILD_CFLAGS += -m$(BITS) +KBUILD_AFLAGS += -m$(BITS) -Wl,-a$(BITS) +LDFLAGS += -m elf$(BITS)$(LDEMULATION) KBUILD_ARFLAGS += --target=elf$(BITS)-$(GNUTARGET) endif @@ -158,18 +161,7 @@ CFLAGS-$(CONFIG_GENERIC_CPU) += -mcpu=powerpc64 endif ifdef CONFIG_MPROFILE_KERNEL - ifeq ($(shell $(srctree)/arch/powerpc/tools/gcc-check-mprofile-kernel.sh $(CC) -I$(srctree)/include -D__KERNEL__),OK) - CC_FLAGS_FTRACE := -pg -mprofile-kernel - KBUILD_CPPFLAGS += -DCC_USING_MPROFILE_KERNEL - else - # If the user asked for mprofile-kernel but the toolchain doesn't - # support it, emit a warning and deliberately break the build later - # with mprofile-kernel-not-supported. We would prefer to make this an - # error right here, but then the user would never be able to run - # oldconfig to change their configuration. - $(warning Compiler does not support mprofile-kernel, set CONFIG_DISABLE_MPROFILE_KERNEL) - CC_FLAGS_FTRACE := -mprofile-kernel-not-supported - endif + CC_FLAGS_FTRACE := -pg -mprofile-kernel endif CFLAGS-$(CONFIG_CELL_CPU) += $(call cc-option,-mcpu=cell) @@ -178,6 +170,7 @@ CFLAGS-$(CONFIG_POWER6_CPU) += $(call cc-option,-mcpu=power6) CFLAGS-$(CONFIG_POWER7_CPU) += $(call cc-option,-mcpu=power7) CFLAGS-$(CONFIG_POWER8_CPU) += $(call cc-option,-mcpu=power8) CFLAGS-$(CONFIG_POWER9_CPU) += $(call cc-option,-mcpu=power9) +CFLAGS-$(CONFIG_PPC_8xx) += $(call cc-option,-mcpu=860) # Altivec option not allowed with e500mc64 in GCC. ifeq ($(CONFIG_ALTIVEC),y) diff --git a/arch/powerpc/boot/Makefile b/arch/powerpc/boot/Makefile index 26d5d2a5b8e9..deea20c334df 100644 --- a/arch/powerpc/boot/Makefile +++ b/arch/powerpc/boot/Makefile @@ -23,19 +23,23 @@ all: $(obj)/zImage compress-$(CONFIG_KERNEL_GZIP) := CONFIG_KERNEL_GZIP compress-$(CONFIG_KERNEL_XZ) := CONFIG_KERNEL_XZ +ifdef CROSS32_COMPILE + BOOTCC := $(CROSS32_COMPILE)gcc + BOOTAR := $(CROSS32_COMPILE)ar +else + BOOTCC := $(CC) + BOOTAR := $(AR) +endif + BOOTCFLAGS := -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs \ -fno-strict-aliasing -Os -msoft-float -pipe \ -fomit-frame-pointer -fno-builtin -fPIC -nostdinc \ -D$(compress-y) -BOOTCC := $(CC) ifdef CONFIG_PPC64_BOOT_WRAPPER BOOTCFLAGS += -m64 else BOOTCFLAGS += -m32 -ifdef CROSS32_COMPILE - BOOTCC := $(CROSS32_COMPILE)gcc -endif endif BOOTCFLAGS += -isystem $(shell $(BOOTCC) -print-file-name=include) @@ -49,6 +53,8 @@ endif BOOTAFLAGS := -D__ASSEMBLY__ $(BOOTCFLAGS) -traditional -nostdinc +BOOTARFLAGS := -cr$(KBUILD_ARFLAGS) + ifdef CONFIG_DEBUG_INFO BOOTCFLAGS += -g endif @@ -120,7 +126,7 @@ src-wlib-$(CONFIG_40x) += 4xx.c planetcore.c src-wlib-$(CONFIG_44x) += 4xx.c ebony.c bamboo.c src-wlib-$(CONFIG_PPC_8xx) += mpc8xx.c planetcore.c fsl-soc.c src-wlib-$(CONFIG_PPC_82xx) += pq2.c fsl-soc.c planetcore.c -src-wlib-$(CONFIG_EMBEDDED6xx) += mpsc.c mv64x60.c mv64x60_i2c.c ugecon.c fsl-soc.c +src-wlib-$(CONFIG_EMBEDDED6xx) += ugecon.c fsl-soc.c src-wlib-$(CONFIG_XILINX_VIRTEX) += uartlite.c src-wlib-$(CONFIG_CPM) += cpm-serial.c @@ -143,8 +149,8 @@ src-plat-$(CONFIG_PPC_82xx) += cuboot-pq2.c fixed-head.S ep8248e.c cuboot-824x.c src-plat-$(CONFIG_PPC_83xx) += cuboot-83xx.c fixed-head.S redboot-83xx.c src-plat-$(CONFIG_FSL_SOC_BOOKE) += cuboot-85xx.c cuboot-85xx-cpm2.c src-plat-$(CONFIG_EMBEDDED6xx) += cuboot-pq2.c cuboot-mpc7448hpc2.c \ - cuboot-c2k.c gamecube-head.S \ - gamecube.c wii-head.S wii.c holly.c \ + gamecube-head.S gamecube.c \ + wii-head.S wii.c holly.c \ fixed-head.S mvme5100.c src-plat-$(CONFIG_AMIGAONE) += cuboot-amigaone.c src-plat-$(CONFIG_PPC_PS3) += ps3-head.S ps3-hvcall.S ps3.c @@ -202,7 +208,7 @@ quiet_cmd_bootas = BOOTAS $@ cmd_bootas = $(BOOTCC) -Wp,-MD,$(depfile) $(BOOTAFLAGS) -c -o $@ $< quiet_cmd_bootar = BOOTAR $@ - cmd_bootar = $(CROSS32AR) -cr$(KBUILD_ARFLAGS) $@.$$$$ $(filter-out FORCE,$^); mv $@.$$$$ $@ + cmd_bootar = $(BOOTAR) $(BOOTARFLAGS) $@.$$$$ $(filter-out FORCE,$^); mv $@.$$$$ $@ $(obj-libfdt): $(obj)/%.o: $(srctree)/scripts/dtc/libfdt/%.c FORCE $(call if_changed_dep,bootcc) @@ -339,7 +345,6 @@ image-$(CONFIG_MVME7100) += dtbImage.mvme7100 # Board ports in arch/powerpc/platform/embedded6xx/Kconfig image-$(CONFIG_STORCENTER) += cuImage.storcenter image-$(CONFIG_MPC7448HPC2) += cuImage.mpc7448hpc2 -image-$(CONFIG_PPC_C2K) += cuImage.c2k image-$(CONFIG_GAMECUBE) += dtbImage.gamecube image-$(CONFIG_WII) += dtbImage.wii image-$(CONFIG_MVME5100) += dtbImage.mvme5100 diff --git a/arch/powerpc/boot/cuboot-c2k.c b/arch/powerpc/boot/cuboot-c2k.c deleted file mode 100644 index 9309c51f1d65..000000000000 --- a/arch/powerpc/boot/cuboot-c2k.c +++ /dev/null @@ -1,189 +0,0 @@ -/* - * GEFanuc C2K platform code. - * - * Author: Remi Machet - * - * Originated from prpmc2800.c - * - * 2008 (c) Stanford University - * 2007 (c) MontaVista, Software, Inc. - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License version 2 as published - * by the Free Software Foundation. - */ - -#include "types.h" -#include "stdio.h" -#include "io.h" -#include "ops.h" -#include "elf.h" -#include "mv64x60.h" -#include "cuboot.h" -#include "ppcboot.h" - -static u8 *bridge_base; - -static void c2k_bridge_setup(u32 mem_size) -{ - u32 i, v[30], enables, acc_bits; - u32 pci_base_hi, pci_base_lo, size, buf[2]; - unsigned long cpu_base; - int rc; - void *devp, *mv64x60_devp; - u8 *bridge_pbase, is_coherent; - struct mv64x60_cpu2pci_win *tbl; - int bus; - - bridge_pbase = mv64x60_get_bridge_pbase(); - is_coherent = mv64x60_is_coherent(); - - if (is_coherent) - acc_bits = MV64x60_PCI_ACC_CNTL_SNOOP_WB - | MV64x60_PCI_ACC_CNTL_SWAP_NONE - | MV64x60_PCI_ACC_CNTL_MBURST_32_BYTES - | MV64x60_PCI_ACC_CNTL_RDSIZE_32_BYTES; - else - acc_bits = MV64x60_PCI_ACC_CNTL_SNOOP_NONE - | MV64x60_PCI_ACC_CNTL_SWAP_NONE - | MV64x60_PCI_ACC_CNTL_MBURST_128_BYTES - | MV64x60_PCI_ACC_CNTL_RDSIZE_256_BYTES; - - mv64x60_config_ctlr_windows(bridge_base, bridge_pbase, is_coherent); - mv64x60_devp = find_node_by_compatible(NULL, "marvell,mv64360"); - if (mv64x60_devp == NULL) - fatal("Error: Missing marvell,mv64360 device tree node\n\r"); - - enables = in_le32((u32 *)(bridge_base + MV64x60_CPU_BAR_ENABLE)); - enables |= 0x007ffe00; /* Disable all cpu->pci windows */ - out_le32((u32 *)(bridge_base + MV64x60_CPU_BAR_ENABLE), enables); - - /* Get the cpu -> pci i/o & mem mappings from the device tree */ - devp = NULL; - for (bus = 0; ; bus++) { - char name[] = "pci "; - - name[strlen(name)-1] = bus+'0'; - - devp = find_node_by_alias(name); - if (devp == NULL) - break; - - if (bus >= 2) - fatal("Error: Only 2 PCI controllers are supported at" \ - " this time.\n"); - - mv64x60_config_pci_windows(bridge_base, bridge_pbase, bus, 0, - mem_size, acc_bits); - - rc = getprop(devp, "ranges", v, sizeof(v)); - if (rc == 0) - fatal("Error: Can't find marvell,mv64360-pci ranges" - " property\n\r"); - - /* Get the cpu -> pci i/o & mem mappings from the device tree */ - - for (i = 0; i < rc; i += 6) { - switch (v[i] & 0xff000000) { - case 0x01000000: /* PCI I/O Space */ - tbl = mv64x60_cpu2pci_io; - break; - case 0x02000000: /* PCI MEM Space */ - tbl = mv64x60_cpu2pci_mem; - break; - default: - continue; - } - - pci_base_hi = v[i+1]; - pci_base_lo = v[i+2]; - cpu_base = v[i+3]; - size = v[i+5]; - - buf[0] = cpu_base; - buf[1] = size; - - if (!dt_xlate_addr(devp, buf, sizeof(buf), &cpu_base)) - fatal("Error: Can't translate PCI address " \ - "0x%x\n\r", (u32)cpu_base); - - mv64x60_config_cpu2pci_window(bridge_base, bus, - pci_base_hi, pci_base_lo, cpu_base, size, tbl); - } - - enables &= ~(3<<(9+bus*5)); /* Enable cpu->pci i/o, - cpu->pci mem0 */ - out_le32((u32 *)(bridge_base + MV64x60_CPU_BAR_ENABLE), - enables); - }; -} - -static void c2k_fixups(void) -{ - u32 mem_size; - - mem_size = mv64x60_get_mem_size(bridge_base); - c2k_bridge_setup(mem_size); /* Do necessary bridge setup */ -} - -#define MV64x60_MPP_CNTL_0 0xf000 -#define MV64x60_MPP_CNTL_2 0xf008 -#define MV64x60_GPP_IO_CNTL 0xf100 -#define MV64x60_GPP_LEVEL_CNTL 0xf110 -#define MV64x60_GPP_VALUE_SET 0xf118 - -static void c2k_reset(void) -{ - u32 temp; - - udelay(5000000); - - if (bridge_base != 0) { - temp = in_le32((u32 *)(bridge_base + MV64x60_MPP_CNTL_0)); - temp &= 0xFFFF0FFF; - out_le32((u32 *)(bridge_base + MV64x60_MPP_CNTL_0), temp); - - temp = in_le32((u32 *)(bridge_base + MV64x60_GPP_LEVEL_CNTL)); - temp |= 0x00000004; - out_le32((u32 *)(bridge_base + MV64x60_GPP_LEVEL_CNTL), temp); - - temp = in_le32((u32 *)(bridge_base + MV64x60_GPP_IO_CNTL)); - temp |= 0x00000004; - out_le32((u32 *)(bridge_base + MV64x60_GPP_IO_CNTL), temp); - - temp = in_le32((u32 *)(bridge_base + MV64x60_MPP_CNTL_2)); - temp &= 0xFFFF0FFF; - out_le32((u32 *)(bridge_base + MV64x60_MPP_CNTL_2), temp); - - temp = in_le32((u32 *)(bridge_base + MV64x60_GPP_LEVEL_CNTL)); - temp |= 0x00080000; - out_le32((u32 *)(bridge_base + MV64x60_GPP_LEVEL_CNTL), temp); - - temp = in_le32((u32 *)(bridge_base + MV64x60_GPP_IO_CNTL)); - temp |= 0x00080000; - out_le32((u32 *)(bridge_base + MV64x60_GPP_IO_CNTL), temp); - - out_le32((u32 *)(bridge_base + MV64x60_GPP_VALUE_SET), - 0x00080004); - } - - for (;;); -} - -static bd_t bd; - -void platform_init(unsigned long r3, unsigned long r4, unsigned long r5, - unsigned long r6, unsigned long r7) -{ - CUBOOT_INIT(); - - fdt_init(_dtb_start); - - bridge_base = mv64x60_get_bridge_base(); - - platform_ops.fixups = c2k_fixups; - platform_ops.exit = c2k_reset; - - if (serial_console_init() < 0) - exit(); -} diff --git a/arch/powerpc/boot/dts/c2k.dts b/arch/powerpc/boot/dts/c2k.dts deleted file mode 100644 index c5beb72d18b7..000000000000 --- a/arch/powerpc/boot/dts/c2k.dts +++ /dev/null @@ -1,366 +0,0 @@ -/* Device Tree Source for GEFanuc C2K - * - * Author: Remi Machet - * - * Originated from prpmc2800.dts - * - * 2008 (c) Stanford University - * 2007 (c) MontaVista, Software, Inc. - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License version 2 as published - * by the Free Software Foundation. - */ - -/dts-v1/; - -/ { - #address-cells = <1>; - #size-cells = <1>; - model = "C2K"; - compatible = "GEFanuc,C2K"; - coherency-off; - - aliases { - pci0 = &PCI0; - pci1 = &PCI1; - }; - - cpus { - #address-cells = <1>; - #size-cells = <0>; - - cpu@0 { - device_type = "cpu"; - compatible = "PowerPC,7447"; - reg = <0>; - clock-frequency = <996000000>; /* 996 MHz */ - bus-frequency = <166666667>; /* 166.6666 MHz */ - timebase-frequency = <41666667>; /* 166.6666/4 MHz */ - i-cache-line-size = <32>; - d-cache-line-size = <32>; - i-cache-size = <32768>; - d-cache-size = <32768>; - }; - }; - - memory { - device_type = "memory"; - reg = <0x00000000 0x40000000>; /* 1GB */ - }; - - system-controller@d8000000 { /* Marvell Discovery */ - #address-cells = <1>; - #size-cells = <1>; - model = "mv64460"; - compatible = "marvell,mv64360"; - clock-frequency = <166666667>; /* 166.66... MHz */ - reg = <0xd8000000 0x00010000>; - virtual-reg = <0xd8000000>; - ranges = <0xd4000000 0xd4000000 0x01000000 /* PCI 0 I/O Space */ - 0x80000000 0x80000000 0x08000000 /* PCI 0 MEM Space */ - 0xd0000000 0xd0000000 0x01000000 /* PCI 1 I/O Space */ - 0xa0000000 0xa0000000 0x08000000 /* PCI 1 MEM Space */ - 0xd8100000 0xd8100000 0x00010000 /* FPGA */ - 0xd8110000 0xd8110000 0x00010000 /* FPGA USARTs */ - 0xf8000000 0xf8000000 0x08000000 /* User FLASH */ - 0x00000000 0xd8000000 0x00010000 /* Bridge's regs */ - 0xd8140000 0xd8140000 0x00040000>; /* Integrated SRAM */ - - mdio@2000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "marvell,mv64360-mdio"; - reg = <0x2000 4>; - PHY0: ethernet-phy@0 { - interrupts = <76>; /* GPP 12 */ - interrupt-parent = <&PIC>; - reg = <0>; - }; - PHY1: ethernet-phy@1 { - interrupts = <76>; /* GPP 12 */ - interrupt-parent = <&PIC>; - reg = <1>; - }; - PHY2: ethernet-phy@2 { - interrupts = <76>; /* GPP 12 */ - interrupt-parent = <&PIC>; - reg = <2>; - }; - }; - - ethernet-group@2000 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "marvell,mv64360-eth-group"; - reg = <0x2000 0x2000>; - ethernet@0 { - device_type = "network"; - compatible = "marvell,mv64360-eth"; - reg = <0>; - interrupts = <32>; - interrupt-parent = <&PIC>; - phy = <&PHY0>; - local-mac-address = [ 00 00 00 00 00 00 ]; - }; - ethernet@1 { - device_type = "network"; - compatible = "marvell,mv64360-eth"; - reg = <1>; - interrupts = <33>; - interrupt-parent = <&PIC>; - phy = <&PHY1>; - local-mac-address = [ 00 00 00 00 00 00 ]; - }; - ethernet@2 { - device_type = "network"; - compatible = "marvell,mv64360-eth"; - reg = <2>; - interrupts = <34>; - interrupt-parent = <&PIC>; - phy = <&PHY2>; - local-mac-address = [ 00 00 00 00 00 00 ]; - }; - }; - - SDMA0: sdma@4000 { - compatible = "marvell,mv64360-sdma"; - reg = <0x4000 0xc18>; - virtual-reg = <0xd8004000>; - interrupt-base = <0>; - interrupts = <36>; - interrupt-parent = <&PIC>; - }; - - SDMA1: sdma@6000 { - compatible = "marvell,mv64360-sdma"; - reg = <0x6000 0xc18>; - virtual-reg = <0xd8006000>; - interrupt-base = <0>; - interrupts = <38>; - interrupt-parent = <&PIC>; - }; - - BRG0: brg@b200 { - compatible = "marvell,mv64360-brg"; - reg = <0xb200 0x8>; - clock-src = <8>; - clock-frequency = <133333333>; - current-speed = <115200>; - }; - - BRG1: brg@b208 { - compatible = "marvell,mv64360-brg"; - reg = <0xb208 0x8>; - clock-src = <8>; - clock-frequency = <133333333>; - current-speed = <115200>; - }; - - CUNIT: cunit@f200 { - reg = <0xf200 0x200>; - }; - - MPSCROUTING: mpscrouting@b400 { - reg = <0xb400 0xc>; - }; - - MPSCINTR: mpscintr@b800 { - reg = <0xb800 0x100>; - virtual-reg = <0xd800b800>; - }; - - MPSC0: mpsc@8000 { - compatible = "marvell,mv64360-mpsc"; - reg = <0x8000 0x38>; - virtual-reg = <0xd8008000>; - sdma = <&SDMA0>; - brg = <&BRG0>; - cunit = <&CUNIT>; - mpscrouting = <&MPSCROUTING>; - mpscintr = <&MPSCINTR>; - cell-index = <0>; - interrupts = <40>; - interrupt-parent = <&PIC>; - }; - - MPSC1: mpsc@9000 { - compatible = "marvell,mv64360-mpsc"; - reg = <0x9000 0x38>; - virtual-reg = <0xd8009000>; - sdma = <&SDMA1>; - brg = <&BRG1>; - cunit = <&CUNIT>; - mpscrouting = <&MPSCROUTING>; - mpscintr = <&MPSCINTR>; - cell-index = <1>; - interrupts = <42>; - interrupt-parent = <&PIC>; - }; - - wdt@b410 { /* watchdog timer */ - compatible = "marvell,mv64360-wdt"; - reg = <0xb410 0x8>; - }; - - i2c@c000 { - compatible = "marvell,mv64360-i2c"; - reg = <0xc000 0x20>; - virtual-reg = <0xd800c000>; - interrupts = <37>; - interrupt-parent = <&PIC>; - }; - - PIC: pic { - #interrupt-cells = <1>; - #address-cells = <0>; - compatible = "marvell,mv64360-pic"; - reg = <0x0000 0x88>; - interrupt-controller; - }; - - mpp@f000 { - compatible = "marvell,mv64360-mpp"; - reg = <0xf000 0x10>; - }; - - gpp@f100 { - compatible = "marvell,mv64360-gpp"; - reg = <0xf100 0x20>; - }; - - PCI0: pci@80000000 { - #address-cells = <3>; - #size-cells = <2>; - #interrupt-cells = <1>; - device_type = "pci"; - compatible = "marvell,mv64360-pci"; - reg = <0x0cf8 0x8>; - ranges = <0x01000000 0x0 0x00000000 0xd4000000 0x0 0x01000000 - 0x02000000 0x0 0x80000000 0x80000000 0x0 0x08000000>; - bus-range = <0 255>; - clock-frequency = <66000000>; - interrupt-pci-iack = <0x0c34>; - interrupt-parent = <&PIC>; - interrupt-map-mask = <0x0000 0x0 0x0 0x7>; - interrupt-map = < - /* Only one interrupt line for PMC0 slot (INTA) */ - 0x0000 0 0 1 &PIC 88 - >; - }; - - - PCI1: pci@a0000000 { - #address-cells = <3>; - #size-cells = <2>; - #interrupt-cells = <1>; - device_type = "pci"; - compatible = "marvell,mv64360-pci"; - reg = <0x0c78 0x8>; - ranges = <0x01000000 0x0 0x00000000 0xd0000000 0x0 0x01000000 - 0x02000000 0x0 0x80000000 0xa0000000 0x0 0x08000000>; - bus-range = <0 255>; - clock-frequency = <66000000>; - interrupt-pci-iack = <0x0cb4>; - interrupt-parent = <&PIC>; - interrupt-map-mask = <0xf800 0x00 0x00 0x7>; - interrupt-map = < - /* IDSEL 0x01: PMC1 ? */ - 0x0800 0 0 1 &PIC 88 - /* IDSEL 0x02: cPCI bridge */ - 0x1000 0 0 1 &PIC 88 - /* IDSEL 0x03: USB controller */ - 0x1800 0 0 1 &PIC 91 - /* IDSEL 0x04: SATA controller */ - 0x2000 0 0 1 &PIC 95 - >; - }; - - cpu-error@70 { - compatible = "marvell,mv64360-cpu-error"; - reg = <0x0070 0x10 0x0128 0x28>; - interrupts = <3>; - interrupt-parent = <&PIC>; - }; - - sram-ctrl@380 { - compatible = "marvell,mv64360-sram-ctrl"; - reg = <0x0380 0x80>; - interrupts = <13>; - interrupt-parent = <&PIC>; - }; - - pci-error@1d40 { - compatible = "marvell,mv64360-pci-error"; - reg = <0x1d40 0x40 0x0c28 0x4>; - interrupts = <12>; - interrupt-parent = <&PIC>; - }; - - pci-error@1dc0 { - compatible = "marvell,mv64360-pci-error"; - reg = <0x1dc0 0x40 0x0ca8 0x4>; - interrupts = <16>; - interrupt-parent = <&PIC>; - }; - - mem-ctrl@1400 { - compatible = "marvell,mv64360-mem-ctrl"; - reg = <0x1400 0x60>; - interrupts = <17>; - interrupt-parent = <&PIC>; - }; - /* Devices attached to the device controller */ - devicebus@45c { - #address-cells = <2>; - #size-cells = <1>; - compatible = "marvell,mv64306-devctrl"; - reg = <0x45C 0x88>; - interrupts = <1>; - interrupt-parent = <&PIC>; - ranges = <0 0 0xd8100000 0x10000 - 2 0 0xd8110000 0x10000 - 4 0 0xf8000000 0x8000000>; - fpga@0,0 { - compatible = "sbs,fpga-c2k"; - reg = <0 0 0x10000>; - }; - fpga_usart@2,0 { - compatible = "sbs,fpga_usart-c2k"; - reg = <2 0 0x10000>; - }; - nor_flash@4,0 { - compatible = "cfi-flash"; - reg = <4 0 0x8000000>; /* 128MB */ - bank-width = <4>; - device-width = <1>; - #address-cells = <1>; - #size-cells = <1>; - partition@0 { - label = "boot"; - reg = <0x00000000 0x00080000>; - }; - partition@40000 { - label = "kernel"; - reg = <0x00080000 0x00400000>; - }; - partition@440000 { - label = "initrd"; - reg = <0x00480000 0x00B80000>; - }; - partition@1000000 { - label = "rootfs"; - reg = <0x01000000 0x06800000>; - }; - partition@7800000 { - label = "recovery"; - reg = <0x07800000 0x00800000>; - read-only; - }; - }; - }; - }; - chosen { - stdout-path = &MPSC0; - }; -}; diff --git a/arch/powerpc/boot/dts/fsl/t104xqds.dtsi b/arch/powerpc/boot/dts/fsl/t104xqds.dtsi index 2fd4cbe7098f..615479732252 100644 --- a/arch/powerpc/boot/dts/fsl/t104xqds.dtsi +++ b/arch/powerpc/boot/dts/fsl/t104xqds.dtsi @@ -269,7 +269,7 @@ i2c@118000 { pca9547@77 { - compatible = "philips,pca9547"; + compatible = "nxp,pca9547"; reg = <0x77>; }; rtc@68 { diff --git a/arch/powerpc/boot/dts/sbc8349.dts b/arch/powerpc/boot/dts/sbc8349.dts deleted file mode 100644 index fc89e00b765c..000000000000 --- a/arch/powerpc/boot/dts/sbc8349.dts +++ /dev/null @@ -1,331 +0,0 @@ -/* - * SBC8349E Device Tree Source - * - * Copyright 2007 Wind River Inc. - * - * Paul Gortmaker (see MAINTAINERS for contact information) - * - * -based largely on the Freescale MPC834x_MDS dts. - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the - * Free Software Foundation; either version 2 of the License, or (at your - * option) any later version. - */ - -/dts-v1/; - -/ { - model = "SBC8349E"; - compatible = "SBC834xE"; - #address-cells = <1>; - #size-cells = <1>; - - aliases { - ethernet0 = &enet0; - ethernet1 = &enet1; - serial0 = &serial0; - serial1 = &serial1; - pci0 = &pci0; - }; - - cpus { - #address-cells = <1>; - #size-cells = <0>; - - PowerPC,8349@0 { - device_type = "cpu"; - reg = <0x0>; - d-cache-line-size = <32>; - i-cache-line-size = <32>; - d-cache-size = <32768>; - i-cache-size = <32768>; - timebase-frequency = <0>; // from bootloader - bus-frequency = <0>; // from bootloader - clock-frequency = <0>; // from bootloader - }; - }; - - memory { - device_type = "memory"; - reg = <0x00000000 0x10000000>; // 256MB at 0 - }; - - soc8349@e0000000 { - #address-cells = <1>; - #size-cells = <1>; - device_type = "soc"; - ranges = <0x0 0xe0000000 0x00100000>; - reg = <0xe0000000 0x00000200>; - bus-frequency = <0>; - - wdt@200 { - compatible = "mpc83xx_wdt"; - reg = <0x200 0x100>; - }; - - i2c@3000 { - #address-cells = <1>; - #size-cells = <0>; - cell-index = <0>; - compatible = "fsl-i2c"; - reg = <0x3000 0x100>; - interrupts = <14 0x8>; - interrupt-parent = <&ipic>; - dfsrr; - }; - - i2c@3100 { - #address-cells = <1>; - #size-cells = <0>; - cell-index = <1>; - compatible = "fsl-i2c"; - reg = <0x3100 0x100>; - interrupts = <15 0x8>; - interrupt-parent = <&ipic>; - dfsrr; - }; - - spi@7000 { - cell-index = <0>; - compatible = "fsl,spi"; - reg = <0x7000 0x1000>; - interrupts = <16 0x8>; - interrupt-parent = <&ipic>; - mode = "cpu"; - }; - - dma@82a8 { - #address-cells = <1>; - #size-cells = <1>; - compatible = "fsl,mpc8349-dma", "fsl,elo-dma"; - reg = <0x82a8 4>; - ranges = <0 0x8100 0x1a8>; - interrupt-parent = <&ipic>; - interrupts = <71 8>; - cell-index = <0>; - dma-channel@0 { - compatible = "fsl,mpc8349-dma-channel", "fsl,elo-dma-channel"; - reg = <0 0x80>; - cell-index = <0>; - interrupt-parent = <&ipic>; - interrupts = <71 8>; - }; - dma-channel@80 { - compatible = "fsl,mpc8349-dma-channel", "fsl,elo-dma-channel"; - reg = <0x80 0x80>; - cell-index = <1>; - interrupt-parent = <&ipic>; - interrupts = <71 8>; - }; - dma-channel@100 { - compatible = "fsl,mpc8349-dma-channel", "fsl,elo-dma-channel"; - reg = <0x100 0x80>; - cell-index = <2>; - interrupt-parent = <&ipic>; - interrupts = <71 8>; - }; - dma-channel@180 { - compatible = "fsl,mpc8349-dma-channel", "fsl,elo-dma-channel"; - reg = <0x180 0x28>; - cell-index = <3>; - interrupt-parent = <&ipic>; - interrupts = <71 8>; - }; - }; - - /* phy type (ULPI or SERIAL) are only types supported for MPH */ - /* port = 0 or 1 */ - usb@22000 { - compatible = "fsl-usb2-mph"; - reg = <0x22000 0x1000>; - #address-cells = <1>; - #size-cells = <0>; - interrupt-parent = <&ipic>; - interrupts = <39 0x8>; - phy_type = "ulpi"; - port0; - }; - - enet0: ethernet@24000 { - #address-cells = <1>; - #size-cells = <1>; - cell-index = <0>; - device_type = "network"; - model = "TSEC"; - compatible = "gianfar"; - reg = <0x24000 0x1000>; - ranges = <0x0 0x24000 0x1000>; - local-mac-address = [ 00 00 00 00 00 00 ]; - interrupts = <32 0x8 33 0x8 34 0x8>; - interrupt-parent = <&ipic>; - tbi-handle = <&tbi0>; - phy-handle = <&phy0>; - linux,network-index = <0>; - - mdio@520 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "fsl,gianfar-mdio"; - reg = <0x520 0x20>; - - phy0: ethernet-phy@19 { - interrupt-parent = <&ipic>; - interrupts = <20 0x8>; - reg = <0x19>; - }; - - phy1: ethernet-phy@1a { - interrupt-parent = <&ipic>; - interrupts = <21 0x8>; - reg = <0x1a>; - }; - - tbi0: tbi-phy@11 { - reg = <0x11>; - device_type = "tbi-phy"; - }; - }; - }; - - enet1: ethernet@25000 { - #address-cells = <1>; - #size-cells = <1>; - cell-index = <1>; - device_type = "network"; - model = "TSEC"; - compatible = "gianfar"; - reg = <0x25000 0x1000>; - ranges = <0x0 0x25000 0x1000>; - local-mac-address = [ 00 00 00 00 00 00 ]; - interrupts = <35 0x8 36 0x8 37 0x8>; - interrupt-parent = <&ipic>; - tbi-handle = <&tbi1>; - phy-handle = <&phy1>; - linux,network-index = <1>; - - mdio@520 { - #address-cells = <1>; - #size-cells = <0>; - compatible = "fsl,gianfar-tbi"; - reg = <0x520 0x20>; - - tbi1: tbi-phy@11 { - reg = <0x11>; - device_type = "tbi-phy"; - }; - }; - }; - - serial0: serial@4500 { - cell-index = <0>; - device_type = "serial"; - compatible = "fsl,ns16550", "ns16550"; - reg = <0x4500 0x100>; - clock-frequency = <0>; - interrupts = <9 0x8>; - interrupt-parent = <&ipic>; - }; - - serial1: serial@4600 { - cell-index = <1>; - device_type = "serial"; - compatible = "fsl,ns16550", "ns16550"; - reg = <0x4600 0x100>; - clock-frequency = <0>; - interrupts = <10 0x8>; - interrupt-parent = <&ipic>; - }; - - crypto@30000 { - compatible = "fsl,sec2.0"; - reg = <0x30000 0x10000>; - interrupts = <11 0x8>; - interrupt-parent = <&ipic>; - fsl,num-channels = <4>; - fsl,channel-fifo-len = <24>; - fsl,exec-units-mask = <0x7e>; - fsl,descriptor-types-mask = <0x01010ebf>; - }; - - /* IPIC - * interrupts cell = - * sense values match linux IORESOURCE_IRQ_* defines: - * sense == 8: Level, low assertion - * sense == 2: Edge, high-to-low change - */ - ipic: pic@700 { - interrupt-controller; - #address-cells = <0>; - #interrupt-cells = <2>; - reg = <0x700 0x100>; - device_type = "ipic"; - }; - }; - - localbus@e0005000 { - #address-cells = <2>; - #size-cells = <1>; - compatible = "fsl,mpc8349-localbus", "simple-bus"; - reg = <0xe0005000 0x1000>; - interrupts = <77 0x8>; - interrupt-parent = <&ipic>; - ranges = <0x0 0x0 0xff800000 0x00800000 /* 8MB Flash */ - 0x1 0x0 0xf8000000 0x00002000 /* 8KB EEPROM */ - 0x2 0x0 0x10000000 0x04000000 /* 64MB SDRAM */ - 0x3 0x0 0x10000000 0x04000000>; /* 64MB SDRAM */ - - flash@0,0 { - #address-cells = <1>; - #size-cells = <1>; - compatible = "intel,28F640J3A", "cfi-flash"; - reg = <0x0 0x0 0x800000>; - bank-width = <2>; - device-width = <1>; - - partition@0 { - label = "u-boot"; - reg = <0x00000000 0x00040000>; - read-only; - }; - - partition@40000 { - label = "user"; - reg = <0x00040000 0x006c0000>; - }; - - partition@700000 { - label = "legacy u-boot"; - reg = <0x00700000 0x00100000>; - read-only; - }; - - }; - }; - - pci0: pci@e0008500 { - interrupt-map-mask = <0xf800 0x0 0x0 0x7>; - interrupt-map = < - - /* IDSEL 0x11 */ - 0x8800 0x0 0x0 0x1 &ipic 48 0x8 - 0x8800 0x0 0x0 0x2 &ipic 17 0x8 - 0x8800 0x0 0x0 0x3 &ipic 18 0x8 - 0x8800 0x0 0x0 0x4 &ipic 19 0x8>; - - interrupt-parent = <&ipic>; - interrupts = <0x42 0x8>; - bus-range = <0 0>; - ranges = <0x02000000 0x0 0x90000000 0x90000000 0x0 0x10000000 - 0x42000000 0x0 0x80000000 0x80000000 0x0 0x10000000 - 0x01000000 0x0 0x00000000 0xe2000000 0x0 0x00100000>; - clock-frequency = <66666666>; - #interrupt-cells = <1>; - #size-cells = <2>; - #address-cells = <3>; - reg = <0xe0008500 0x100 /* internal registers */ - 0xe0008300 0x8>; /* config space access registers */ - compatible = "fsl,mpc8349-pci"; - device_type = "pci"; - }; -}; diff --git a/arch/powerpc/boot/mpc8xx.c b/arch/powerpc/boot/mpc8xx.c index add55a7f184f..c9bd9285c548 100644 --- a/arch/powerpc/boot/mpc8xx.c +++ b/arch/powerpc/boot/mpc8xx.c @@ -24,7 +24,7 @@ u32 mpc885_get_clock(u32 crystal) { u32 *immr; u32 plprcr; - int mfi, mfn, mfd, pdf, div; + int mfi, mfn, mfd, pdf; u32 ret; immr = fsl_get_immr(); @@ -43,7 +43,6 @@ u32 mpc885_get_clock(u32 crystal) } pdf = (plprcr >> 1) & 0xf; - div = (plprcr >> 20) & 3; mfd = (plprcr >> 22) & 0x1f; mfn = (plprcr >> 27) & 0x1f; diff --git a/arch/powerpc/boot/mpsc.c b/arch/powerpc/boot/mpsc.c deleted file mode 100644 index 425ad88cce8d..000000000000 --- a/arch/powerpc/boot/mpsc.c +++ /dev/null @@ -1,169 +0,0 @@ -/* - * MPSC/UART driver for the Marvell mv64360, mv64460, ... - * - * Author: Mark A. Greer - * - * 2007 (c) MontaVista Software, Inc. This file is licensed under - * the terms of the GNU General Public License version 2. This program - * is licensed "as is" without any warranty of any kind, whether express - * or implied. - */ - -#include -#include -#include "types.h" -#include "string.h" -#include "stdio.h" -#include "io.h" -#include "ops.h" - - -#define MPSC_CHR_1 0x000c - -#define MPSC_CHR_2 0x0010 -#define MPSC_CHR_2_TA (1<<7) -#define MPSC_CHR_2_TCS (1<<9) -#define MPSC_CHR_2_RA (1<<23) -#define MPSC_CHR_2_CRD (1<<25) -#define MPSC_CHR_2_EH (1<<31) - -#define MPSC_CHR_4 0x0018 -#define MPSC_CHR_4_Z (1<<29) - -#define MPSC_CHR_5 0x001c -#define MPSC_CHR_5_CTL1_INTR (1<<12) -#define MPSC_CHR_5_CTL1_VALID (1<<15) - -#define MPSC_CHR_10 0x0030 - -#define MPSC_INTR_CAUSE 0x0000 -#define MPSC_INTR_CAUSE_RCC (1<<6) -#define MPSC_INTR_MASK 0x0080 - -#define SDMA_SDCM 0x0008 -#define SDMA_SDCM_AR (1<<15) -#define SDMA_SDCM_AT (1<<31) - -static volatile char *mpsc_base; -static volatile char *mpscintr_base; -static u32 chr1, chr2; - -static int mpsc_open(void) -{ - chr1 = in_le32((u32 *)(mpsc_base + MPSC_CHR_1)) & 0x00ff0000; - chr2 = in_le32((u32 *)(mpsc_base + MPSC_CHR_2)) & ~(MPSC_CHR_2_TA - | MPSC_CHR_2_TCS | MPSC_CHR_2_RA | MPSC_CHR_2_CRD - | MPSC_CHR_2_EH); - out_le32((u32 *)(mpsc_base + MPSC_CHR_4), MPSC_CHR_4_Z); - out_le32((u32 *)(mpsc_base + MPSC_CHR_5), - MPSC_CHR_5_CTL1_INTR | MPSC_CHR_5_CTL1_VALID); - out_le32((u32 *)(mpsc_base + MPSC_CHR_2), chr2 | MPSC_CHR_2_EH); - return 0; -} - -static void mpsc_putc(unsigned char c) -{ - while (in_le32((u32 *)(mpsc_base + MPSC_CHR_2)) & MPSC_CHR_2_TCS); - - out_le32((u32 *)(mpsc_base + MPSC_CHR_1), chr1 | c); - out_le32((u32 *)(mpsc_base + MPSC_CHR_2), chr2 | MPSC_CHR_2_TCS); -} - -static unsigned char mpsc_getc(void) -{ - u32 cause = 0; - unsigned char c; - - while (!(cause & MPSC_INTR_CAUSE_RCC)) - cause = in_le32((u32 *)(mpscintr_base + MPSC_INTR_CAUSE)); - - c = in_8((u8 *)(mpsc_base + MPSC_CHR_10 + 2)); - out_8((u8 *)(mpsc_base + MPSC_CHR_10 + 2), c); - out_le32((u32 *)(mpscintr_base + MPSC_INTR_CAUSE), - cause & ~MPSC_INTR_CAUSE_RCC); - - return c; -} - -static u8 mpsc_tstc(void) -{ - return (u8)((in_le32((u32 *)(mpscintr_base + MPSC_INTR_CAUSE)) - & MPSC_INTR_CAUSE_RCC) != 0); -} - -static void mpsc_stop_dma(volatile char *sdma_base) -{ - out_le32((u32 *)(mpsc_base + MPSC_CHR_2),MPSC_CHR_2_TA | MPSC_CHR_2_RA); - out_le32((u32 *)(sdma_base + SDMA_SDCM), SDMA_SDCM_AR | SDMA_SDCM_AT); - - while ((in_le32((u32 *)(sdma_base + SDMA_SDCM)) - & (SDMA_SDCM_AR | SDMA_SDCM_AT)) != 0) - udelay(100); -} - -static volatile char *mpsc_get_virtreg_of_phandle(void *devp, char *prop) -{ - void *v; - int n; - - n = getprop(devp, prop, &v, sizeof(v)); - if (n != sizeof(v)) - goto err_out; - - devp = find_node_by_linuxphandle((u32)v); - if (devp == NULL) - goto err_out; - - n = getprop(devp, "virtual-reg", &v, sizeof(v)); - if (n == sizeof(v)) - return v; - -err_out: - return NULL; -} - -int mpsc_console_init(void *devp, struct serial_console_data *scdp) -{ - void *v; - int n, reg_set; - volatile char *sdma_base; - - n = getprop(devp, "virtual-reg", &v, sizeof(v)); - if (n != sizeof(v)) - goto err_out; - mpsc_base = v; - - sdma_base = mpsc_get_virtreg_of_phandle(devp, "sdma"); - if (sdma_base == NULL) - goto err_out; - - mpscintr_base = mpsc_get_virtreg_of_phandle(devp, "mpscintr"); - if (mpscintr_base == NULL) - goto err_out; - - n = getprop(devp, "cell-index", &v, sizeof(v)); - if (n != sizeof(v)) - goto err_out; - reg_set = (int)v; - - mpscintr_base += (reg_set == 0) ? 0x4 : 0xc; - - /* Make sure the mpsc ctlrs are shutdown */ - out_le32((u32 *)(mpscintr_base + MPSC_INTR_CAUSE), 0); - out_le32((u32 *)(mpscintr_base + MPSC_INTR_CAUSE), 0); - out_le32((u32 *)(mpscintr_base + MPSC_INTR_MASK), 0); - out_le32((u32 *)(mpscintr_base + MPSC_INTR_MASK), 0); - - mpsc_stop_dma(sdma_base); - - scdp->open = mpsc_open; - scdp->putc = mpsc_putc; - scdp->getc = mpsc_getc; - scdp->tstc = mpsc_tstc; - scdp->close = NULL; - - return 0; - -err_out: - return -1; -} diff --git a/arch/powerpc/boot/mv64x60.c b/arch/powerpc/boot/mv64x60.c deleted file mode 100644 index d9bb302b91d2..000000000000 --- a/arch/powerpc/boot/mv64x60.c +++ /dev/null @@ -1,581 +0,0 @@ -/* - * Marvell hostbridge routines - * - * Author: Mark A. Greer - * - * 2004, 2005, 2007 (c) MontaVista Software, Inc. This file is licensed under - * the terms of the GNU General Public License version 2. This program - * is licensed "as is" without any warranty of any kind, whether express - * or implied. - */ - -#include -#include -#include "types.h" -#include "elf.h" -#include "page.h" -#include "string.h" -#include "stdio.h" -#include "io.h" -#include "ops.h" -#include "mv64x60.h" - -#define PCI_DEVFN(slot,func) ((((slot) & 0x1f) << 3) | ((func) & 0x07)) - -#define MV64x60_CPU2MEM_WINDOWS 4 -#define MV64x60_CPU2MEM_0_BASE 0x0008 -#define MV64x60_CPU2MEM_0_SIZE 0x0010 -#define MV64x60_CPU2MEM_1_BASE 0x0208 -#define MV64x60_CPU2MEM_1_SIZE 0x0210 -#define MV64x60_CPU2MEM_2_BASE 0x0018 -#define MV64x60_CPU2MEM_2_SIZE 0x0020 -#define MV64x60_CPU2MEM_3_BASE 0x0218 -#define MV64x60_CPU2MEM_3_SIZE 0x0220 - -#define MV64x60_ENET2MEM_BAR_ENABLE 0x2290 -#define MV64x60_ENET2MEM_0_BASE 0x2200 -#define MV64x60_ENET2MEM_0_SIZE 0x2204 -#define MV64x60_ENET2MEM_1_BASE 0x2208 -#define MV64x60_ENET2MEM_1_SIZE 0x220c -#define MV64x60_ENET2MEM_2_BASE 0x2210 -#define MV64x60_ENET2MEM_2_SIZE 0x2214 -#define MV64x60_ENET2MEM_3_BASE 0x2218 -#define MV64x60_ENET2MEM_3_SIZE 0x221c -#define MV64x60_ENET2MEM_4_BASE 0x2220 -#define MV64x60_ENET2MEM_4_SIZE 0x2224 -#define MV64x60_ENET2MEM_5_BASE 0x2228 -#define MV64x60_ENET2MEM_5_SIZE 0x222c -#define MV64x60_ENET2MEM_ACC_PROT_0 0x2294 -#define MV64x60_ENET2MEM_ACC_PROT_1 0x2298 -#define MV64x60_ENET2MEM_ACC_PROT_2 0x229c - -#define MV64x60_MPSC2MEM_BAR_ENABLE 0xf250 -#define MV64x60_MPSC2MEM_0_BASE 0xf200 -#define MV64x60_MPSC2MEM_0_SIZE 0xf204 -#define MV64x60_MPSC2MEM_1_BASE 0xf208 -#define MV64x60_MPSC2MEM_1_SIZE 0xf20c -#define MV64x60_MPSC2MEM_2_BASE 0xf210 -#define MV64x60_MPSC2MEM_2_SIZE 0xf214 -#define MV64x60_MPSC2MEM_3_BASE 0xf218 -#define MV64x60_MPSC2MEM_3_SIZE 0xf21c -#define MV64x60_MPSC_0_REMAP 0xf240 -#define MV64x60_MPSC_1_REMAP 0xf244 -#define MV64x60_MPSC2MEM_ACC_PROT_0 0xf254 -#define MV64x60_MPSC2MEM_ACC_PROT_1 0xf258 -#define MV64x60_MPSC2REGS_BASE 0xf25c - -#define MV64x60_IDMA2MEM_BAR_ENABLE 0x0a80 -#define MV64x60_IDMA2MEM_0_BASE 0x0a00 -#define MV64x60_IDMA2MEM_0_SIZE 0x0a04 -#define MV64x60_IDMA2MEM_1_BASE 0x0a08 -#define MV64x60_IDMA2MEM_1_SIZE 0x0a0c -#define MV64x60_IDMA2MEM_2_BASE 0x0a10 -#define MV64x60_IDMA2MEM_2_SIZE 0x0a14 -#define MV64x60_IDMA2MEM_3_BASE 0x0a18 -#define MV64x60_IDMA2MEM_3_SIZE 0x0a1c -#define MV64x60_IDMA2MEM_4_BASE 0x0a20 -#define MV64x60_IDMA2MEM_4_SIZE 0x0a24 -#define MV64x60_IDMA2MEM_5_BASE 0x0a28 -#define MV64x60_IDMA2MEM_5_SIZE 0x0a2c -#define MV64x60_IDMA2MEM_6_BASE 0x0a30 -#define MV64x60_IDMA2MEM_6_SIZE 0x0a34 -#define MV64x60_IDMA2MEM_7_BASE 0x0a38 -#define MV64x60_IDMA2MEM_7_SIZE 0x0a3c -#define MV64x60_IDMA2MEM_ACC_PROT_0 0x0a70 -#define MV64x60_IDMA2MEM_ACC_PROT_1 0x0a74 -#define MV64x60_IDMA2MEM_ACC_PROT_2 0x0a78 -#define MV64x60_IDMA2MEM_ACC_PROT_3 0x0a7c - -#define MV64x60_PCI_ACC_CNTL_WINDOWS 6 -#define MV64x60_PCI0_PCI_DECODE_CNTL 0x0d3c -#define MV64x60_PCI1_PCI_DECODE_CNTL 0x0dbc - -#define MV64x60_PCI0_BAR_ENABLE 0x0c3c -#define MV64x60_PCI02MEM_0_SIZE 0x0c08 -#define MV64x60_PCI0_ACC_CNTL_0_BASE_LO 0x1e00 -#define MV64x60_PCI0_ACC_CNTL_0_BASE_HI 0x1e04 -#define MV64x60_PCI0_ACC_CNTL_0_SIZE 0x1e08 -#define MV64x60_PCI0_ACC_CNTL_1_BASE_LO 0x1e10 -#define MV64x60_PCI0_ACC_CNTL_1_BASE_HI 0x1e14 -#define MV64x60_PCI0_ACC_CNTL_1_SIZE 0x1e18 -#define MV64x60_PCI0_ACC_CNTL_2_BASE_LO 0x1e20 -#define MV64x60_PCI0_ACC_CNTL_2_BASE_HI 0x1e24 -#define MV64x60_PCI0_ACC_CNTL_2_SIZE 0x1e28 -#define MV64x60_PCI0_ACC_CNTL_3_BASE_LO 0x1e30 -#define MV64x60_PCI0_ACC_CNTL_3_BASE_HI 0x1e34 -#define MV64x60_PCI0_ACC_CNTL_3_SIZE 0x1e38 -#define MV64x60_PCI0_ACC_CNTL_4_BASE_LO 0x1e40 -#define MV64x60_PCI0_ACC_CNTL_4_BASE_HI 0x1e44 -#define MV64x60_PCI0_ACC_CNTL_4_SIZE 0x1e48 -#define MV64x60_PCI0_ACC_CNTL_5_BASE_LO 0x1e50 -#define MV64x60_PCI0_ACC_CNTL_5_BASE_HI 0x1e54 -#define MV64x60_PCI0_ACC_CNTL_5_SIZE 0x1e58 - -#define MV64x60_PCI1_BAR_ENABLE 0x0cbc -#define MV64x60_PCI12MEM_0_SIZE 0x0c88 -#define MV64x60_PCI1_ACC_CNTL_0_BASE_LO 0x1e80 -#define MV64x60_PCI1_ACC_CNTL_0_BASE_HI 0x1e84 -#define MV64x60_PCI1_ACC_CNTL_0_SIZE 0x1e88 -#define MV64x60_PCI1_ACC_CNTL_1_BASE_LO 0x1e90 -#define MV64x60_PCI1_ACC_CNTL_1_BASE_HI 0x1e94 -#define MV64x60_PCI1_ACC_CNTL_1_SIZE 0x1e98 -#define MV64x60_PCI1_ACC_CNTL_2_BASE_LO 0x1ea0 -#define MV64x60_PCI1_ACC_CNTL_2_BASE_HI 0x1ea4 -#define MV64x60_PCI1_ACC_CNTL_2_SIZE 0x1ea8 -#define MV64x60_PCI1_ACC_CNTL_3_BASE_LO 0x1eb0 -#define MV64x60_PCI1_ACC_CNTL_3_BASE_HI 0x1eb4 -#define MV64x60_PCI1_ACC_CNTL_3_SIZE 0x1eb8 -#define MV64x60_PCI1_ACC_CNTL_4_BASE_LO 0x1ec0 -#define MV64x60_PCI1_ACC_CNTL_4_BASE_HI 0x1ec4 -#define MV64x60_PCI1_ACC_CNTL_4_SIZE 0x1ec8 -#define MV64x60_PCI1_ACC_CNTL_5_BASE_LO 0x1ed0 -#define MV64x60_PCI1_ACC_CNTL_5_BASE_HI 0x1ed4 -#define MV64x60_PCI1_ACC_CNTL_5_SIZE 0x1ed8 - -#define MV64x60_CPU2PCI_SWAP_NONE 0x01000000 - -#define MV64x60_CPU2PCI0_IO_BASE 0x0048 -#define MV64x60_CPU2PCI0_IO_SIZE 0x0050 -#define MV64x60_CPU2PCI0_IO_REMAP 0x00f0 -#define MV64x60_CPU2PCI0_MEM_0_BASE 0x0058 -#define MV64x60_CPU2PCI0_MEM_0_SIZE 0x0060 -#define MV64x60_CPU2PCI0_MEM_0_REMAP_LO 0x00f8 -#define MV64x60_CPU2PCI0_MEM_0_REMAP_HI 0x0320 - -#define MV64x60_CPU2PCI1_IO_BASE 0x0090 -#define MV64x60_CPU2PCI1_IO_SIZE 0x0098 -#define MV64x60_CPU2PCI1_IO_REMAP 0x0108 -#define MV64x60_CPU2PCI1_MEM_0_BASE 0x00a0 -#define MV64x60_CPU2PCI1_MEM_0_SIZE 0x00a8 -#define MV64x60_CPU2PCI1_MEM_0_REMAP_LO 0x0110 -#define MV64x60_CPU2PCI1_MEM_0_REMAP_HI 0x0340 - -struct mv64x60_mem_win { - u32 hi; - u32 lo; - u32 size; -}; - -struct mv64x60_pci_win { - u32 fcn; - u32 hi; - u32 lo; - u32 size; -}; - -/* PCI config access routines */ -struct { - u32 addr; - u32 data; -} static mv64x60_pci_cfgio[2] = { - { /* hose 0 */ - .addr = 0xcf8, - .data = 0xcfc, - }, - { /* hose 1 */ - .addr = 0xc78, - .data = 0xc7c, - } -}; - -u32 mv64x60_cfg_read(u8 *bridge_base, u8 hose, u8 bus, u8 devfn, u8 offset) -{ - out_le32((u32 *)(bridge_base + mv64x60_pci_cfgio[hose].addr), - (1 << 31) | (bus << 16) | (devfn << 8) | offset); - return in_le32((u32 *)(bridge_base + mv64x60_pci_cfgio[hose].data)); -} - -void mv64x60_cfg_write(u8 *bridge_base, u8 hose, u8 bus, u8 devfn, u8 offset, - u32 val) -{ - out_le32((u32 *)(bridge_base + mv64x60_pci_cfgio[hose].addr), - (1 << 31) | (bus << 16) | (devfn << 8) | offset); - out_le32((u32 *)(bridge_base + mv64x60_pci_cfgio[hose].data), val); -} - -/* I/O ctlr -> system memory setup */ -static struct mv64x60_mem_win mv64x60_cpu2mem[MV64x60_CPU2MEM_WINDOWS] = { - { - .lo = MV64x60_CPU2MEM_0_BASE, - .size = MV64x60_CPU2MEM_0_SIZE, - }, - { - .lo = MV64x60_CPU2MEM_1_BASE, - .size = MV64x60_CPU2MEM_1_SIZE, - }, - { - .lo = MV64x60_CPU2MEM_2_BASE, - .size = MV64x60_CPU2MEM_2_SIZE, - }, - { - .lo = MV64x60_CPU2MEM_3_BASE, - .size = MV64x60_CPU2MEM_3_SIZE, - }, -}; - -static struct mv64x60_mem_win mv64x60_enet2mem[MV64x60_CPU2MEM_WINDOWS] = { - { - .lo = MV64x60_ENET2MEM_0_BASE, - .size = MV64x60_ENET2MEM_0_SIZE, - }, - { - .lo = MV64x60_ENET2MEM_1_BASE, - .size = MV64x60_ENET2MEM_1_SIZE, - }, - { - .lo = MV64x60_ENET2MEM_2_BASE, - .size = MV64x60_ENET2MEM_2_SIZE, - }, - { - .lo = MV64x60_ENET2MEM_3_BASE, - .size = MV64x60_ENET2MEM_3_SIZE, - }, -}; - -static struct mv64x60_mem_win mv64x60_mpsc2mem[MV64x60_CPU2MEM_WINDOWS] = { - { - .lo = MV64x60_MPSC2MEM_0_BASE, - .size = MV64x60_MPSC2MEM_0_SIZE, - }, - { - .lo = MV64x60_MPSC2MEM_1_BASE, - .size = MV64x60_MPSC2MEM_1_SIZE, - }, - { - .lo = MV64x60_MPSC2MEM_2_BASE, - .size = MV64x60_MPSC2MEM_2_SIZE, - }, - { - .lo = MV64x60_MPSC2MEM_3_BASE, - .size = MV64x60_MPSC2MEM_3_SIZE, - }, -}; - -static struct mv64x60_mem_win mv64x60_idma2mem[MV64x60_CPU2MEM_WINDOWS] = { - { - .lo = MV64x60_IDMA2MEM_0_BASE, - .size = MV64x60_IDMA2MEM_0_SIZE, - }, - { - .lo = MV64x60_IDMA2MEM_1_BASE, - .size = MV64x60_IDMA2MEM_1_SIZE, - }, - { - .lo = MV64x60_IDMA2MEM_2_BASE, - .size = MV64x60_IDMA2MEM_2_SIZE, - }, - { - .lo = MV64x60_IDMA2MEM_3_BASE, - .size = MV64x60_IDMA2MEM_3_SIZE, - }, -}; - -static u32 mv64x60_dram_selects[MV64x60_CPU2MEM_WINDOWS] = {0xe,0xd,0xb,0x7}; - -/* - * ENET, MPSC, and IDMA ctlrs on the MV64x60 have separate windows that - * must be set up so that the respective ctlr can access system memory. - * Configure them to be same as cpu->memory windows. - */ -void mv64x60_config_ctlr_windows(u8 *bridge_base, u8 *bridge_pbase, - u8 is_coherent) -{ - u32 i, base, size, enables, prot = 0, snoop_bits = 0; - - /* Disable ctlr->mem windows */ - out_le32((u32 *)(bridge_base + MV64x60_ENET2MEM_BAR_ENABLE), 0x3f); - out_le32((u32 *)(bridge_base + MV64x60_MPSC2MEM_BAR_ENABLE), 0xf); - out_le32((u32 *)(bridge_base + MV64x60_ENET2MEM_BAR_ENABLE), 0xff); - - if (is_coherent) - snoop_bits = 0x2 << 12; /* Writeback */ - - enables = in_le32((u32 *)(bridge_base + MV64x60_CPU_BAR_ENABLE)) & 0xf; - - for (i=0; ibridge's reg window to the bridge's internal registers. */ - out_le32((u32 *)(bridge_base + MV64x60_MPSC2REGS_BASE), - (u32)bridge_pbase); - - out_le32((u32 *)(bridge_base + MV64x60_ENET2MEM_BAR_ENABLE), enables); - out_le32((u32 *)(bridge_base + MV64x60_MPSC2MEM_BAR_ENABLE), enables); - out_le32((u32 *)(bridge_base + MV64x60_IDMA2MEM_BAR_ENABLE), enables); -} - -/* PCI MEM -> system memory, et. al. setup */ -static struct mv64x60_pci_win mv64x60_pci2mem[2] = { - { /* hose 0 */ - .fcn = 0, - .hi = 0x14, - .lo = 0x10, - .size = MV64x60_PCI02MEM_0_SIZE, - }, - { /* hose 1 */ - .fcn = 0, - .hi = 0x94, - .lo = 0x90, - .size = MV64x60_PCI12MEM_0_SIZE, - }, -}; - -static struct -mv64x60_mem_win mv64x60_pci_acc[2][MV64x60_PCI_ACC_CNTL_WINDOWS] = { - { /* hose 0 */ - { - .hi = MV64x60_PCI0_ACC_CNTL_0_BASE_HI, - .lo = MV64x60_PCI0_ACC_CNTL_0_BASE_LO, - .size = MV64x60_PCI0_ACC_CNTL_0_SIZE, - }, - { - .hi = MV64x60_PCI0_ACC_CNTL_1_BASE_HI, - .lo = MV64x60_PCI0_ACC_CNTL_1_BASE_LO, - .size = MV64x60_PCI0_ACC_CNTL_1_SIZE, - }, - { - .hi = MV64x60_PCI0_ACC_CNTL_2_BASE_HI, - .lo = MV64x60_PCI0_ACC_CNTL_2_BASE_LO, - .size = MV64x60_PCI0_ACC_CNTL_2_SIZE, - }, - { - .hi = MV64x60_PCI0_ACC_CNTL_3_BASE_HI, - .lo = MV64x60_PCI0_ACC_CNTL_3_BASE_LO, - .size = MV64x60_PCI0_ACC_CNTL_3_SIZE, - }, - }, - { /* hose 1 */ - { - .hi = MV64x60_PCI1_ACC_CNTL_0_BASE_HI, - .lo = MV64x60_PCI1_ACC_CNTL_0_BASE_LO, - .size = MV64x60_PCI1_ACC_CNTL_0_SIZE, - }, - { - .hi = MV64x60_PCI1_ACC_CNTL_1_BASE_HI, - .lo = MV64x60_PCI1_ACC_CNTL_1_BASE_LO, - .size = MV64x60_PCI1_ACC_CNTL_1_SIZE, - }, - { - .hi = MV64x60_PCI1_ACC_CNTL_2_BASE_HI, - .lo = MV64x60_PCI1_ACC_CNTL_2_BASE_LO, - .size = MV64x60_PCI1_ACC_CNTL_2_SIZE, - }, - { - .hi = MV64x60_PCI1_ACC_CNTL_3_BASE_HI, - .lo = MV64x60_PCI1_ACC_CNTL_3_BASE_LO, - .size = MV64x60_PCI1_ACC_CNTL_3_SIZE, - }, - }, -}; - -static struct mv64x60_mem_win mv64x60_pci2reg[2] = { - { - .hi = 0x24, - .lo = 0x20, - .size = 0, - }, - { - .hi = 0xa4, - .lo = 0xa0, - .size = 0, - }, -}; - -/* Only need to use 1 window (per hose) to get access to all of system memory */ -void mv64x60_config_pci_windows(u8 *bridge_base, u8 *bridge_pbase, u8 hose, - u8 bus, u32 mem_size, u32 acc_bits) -{ - u32 i, offset, bar_enable, enables; - - /* Disable all windows but PCI MEM -> Bridge's regs window */ - enables = ~(1 << 9); - bar_enable = hose ? MV64x60_PCI1_BAR_ENABLE : MV64x60_PCI0_BAR_ENABLE; - out_le32((u32 *)(bridge_base + bar_enable), enables); - - for (i=0; i System Mem addr 0 */ - mv64x60_cfg_write(bridge_base, hose, bus, - PCI_DEVFN(0, mv64x60_pci2mem[hose].fcn), - mv64x60_pci2mem[hose].hi, 0); - mv64x60_cfg_write(bridge_base, hose, bus, - PCI_DEVFN(0, mv64x60_pci2mem[hose].fcn), - mv64x60_pci2mem[hose].lo, 0); - out_le32((u32 *)(bridge_base + mv64x60_pci2mem[hose].size),mem_size); - - acc_bits |= MV64x60_PCI_ACC_CNTL_ENABLE; - out_le32((u32 *)(bridge_base + mv64x60_pci_acc[hose][0].hi), 0); - out_le32((u32 *)(bridge_base + mv64x60_pci_acc[hose][0].lo), acc_bits); - out_le32((u32 *)(bridge_base + mv64x60_pci_acc[hose][0].size),mem_size); - - /* Set PCI MEM->bridge's reg window to where they are in CPU mem map */ - i = (u32)bridge_base; - i &= 0xffff0000; - i |= (0x2 << 1); - mv64x60_cfg_write(bridge_base, hose, bus, PCI_DEVFN(0,0), - mv64x60_pci2reg[hose].hi, 0); - mv64x60_cfg_write(bridge_base, hose, bus, PCI_DEVFN(0,0), - mv64x60_pci2reg[hose].lo, i); - - enables &= ~0x1; /* Enable PCI MEM -> System Mem window 0 */ - out_le32((u32 *)(bridge_base + bar_enable), enables); -} - -/* CPU -> PCI I/O & MEM setup */ -struct mv64x60_cpu2pci_win mv64x60_cpu2pci_io[2] = { - { /* hose 0 */ - .lo = MV64x60_CPU2PCI0_IO_BASE, - .size = MV64x60_CPU2PCI0_IO_SIZE, - .remap_hi = 0, - .remap_lo = MV64x60_CPU2PCI0_IO_REMAP, - }, - { /* hose 1 */ - .lo = MV64x60_CPU2PCI1_IO_BASE, - .size = MV64x60_CPU2PCI1_IO_SIZE, - .remap_hi = 0, - .remap_lo = MV64x60_CPU2PCI1_IO_REMAP, - }, -}; - -struct mv64x60_cpu2pci_win mv64x60_cpu2pci_mem[2] = { - { /* hose 0 */ - .lo = MV64x60_CPU2PCI0_MEM_0_BASE, - .size = MV64x60_CPU2PCI0_MEM_0_SIZE, - .remap_hi = MV64x60_CPU2PCI0_MEM_0_REMAP_HI, - .remap_lo = MV64x60_CPU2PCI0_MEM_0_REMAP_LO, - }, - { /* hose 1 */ - .lo = MV64x60_CPU2PCI1_MEM_0_BASE, - .size = MV64x60_CPU2PCI1_MEM_0_SIZE, - .remap_hi = MV64x60_CPU2PCI1_MEM_0_REMAP_HI, - .remap_lo = MV64x60_CPU2PCI1_MEM_0_REMAP_LO, - }, -}; - -/* Only need to set up 1 window to pci mem space */ -void mv64x60_config_cpu2pci_window(u8 *bridge_base, u8 hose, u32 pci_base_hi, - u32 pci_base_lo, u32 cpu_base, u32 size, - struct mv64x60_cpu2pci_win *offset_tbl) -{ - cpu_base >>= 16; - cpu_base |= MV64x60_CPU2PCI_SWAP_NONE; - out_le32((u32 *)(bridge_base + offset_tbl[hose].lo), cpu_base); - - if (offset_tbl[hose].remap_hi != 0) - out_le32((u32 *)(bridge_base + offset_tbl[hose].remap_hi), - pci_base_hi); - out_le32((u32 *)(bridge_base + offset_tbl[hose].remap_lo), - pci_base_lo >> 16); - - size = (size - 1) >> 16; - out_le32((u32 *)(bridge_base + offset_tbl[hose].size), size); -} - -/* Read mem ctlr to get the amount of mem in system */ -u32 mv64x60_get_mem_size(u8 *bridge_base) -{ - u32 enables, i, v; - u32 mem = 0; - - enables = in_le32((u32 *)(bridge_base + MV64x60_CPU_BAR_ENABLE)) & 0xf; - - for (i=0; i - * - * 2007 (c) MontaVista Software, Inc. This file is licensed under - * the terms of the GNU General Public License version 2. This program - * is licensed "as is" without any warranty of any kind, whether express - * or implied. - */ - -#ifndef _PPC_BOOT_MV64x60_H_ -#define _PPC_BOOT_MV64x60_H_ - -#define MV64x60_CPU_BAR_ENABLE 0x0278 - -#define MV64x60_PCI_ACC_CNTL_ENABLE (1<<0) -#define MV64x60_PCI_ACC_CNTL_REQ64 (1<<1) -#define MV64x60_PCI_ACC_CNTL_SNOOP_NONE 0x00000000 -#define MV64x60_PCI_ACC_CNTL_SNOOP_WT 0x00000004 -#define MV64x60_PCI_ACC_CNTL_SNOOP_WB 0x00000008 -#define MV64x60_PCI_ACC_CNTL_SNOOP_MASK 0x0000000c -#define MV64x60_PCI_ACC_CNTL_ACCPROT (1<<4) -#define MV64x60_PCI_ACC_CNTL_WRPROT (1<<5) -#define MV64x60_PCI_ACC_CNTL_SWAP_BYTE 0x00000000 -#define MV64x60_PCI_ACC_CNTL_SWAP_NONE 0x00000040 -#define MV64x60_PCI_ACC_CNTL_SWAP_BYTE_WORD 0x00000080 -#define MV64x60_PCI_ACC_CNTL_SWAP_WORD 0x000000c0 -#define MV64x60_PCI_ACC_CNTL_SWAP_MASK 0x000000c0 -#define MV64x60_PCI_ACC_CNTL_MBURST_32_BYTES 0x00000000 -#define MV64x60_PCI_ACC_CNTL_MBURST_64_BYTES 0x00000100 -#define MV64x60_PCI_ACC_CNTL_MBURST_128_BYTES 0x00000200 -#define MV64x60_PCI_ACC_CNTL_MBURST_MASK 0x00000300 -#define MV64x60_PCI_ACC_CNTL_RDSIZE_32_BYTES 0x00000000 -#define MV64x60_PCI_ACC_CNTL_RDSIZE_64_BYTES 0x00000400 -#define MV64x60_PCI_ACC_CNTL_RDSIZE_128_BYTES 0x00000800 -#define MV64x60_PCI_ACC_CNTL_RDSIZE_256_BYTES 0x00000c00 -#define MV64x60_PCI_ACC_CNTL_RDSIZE_MASK 0x00000c00 - -struct mv64x60_cpu2pci_win { - u32 lo; - u32 size; - u32 remap_hi; - u32 remap_lo; -}; - -extern struct mv64x60_cpu2pci_win mv64x60_cpu2pci_io[2]; -extern struct mv64x60_cpu2pci_win mv64x60_cpu2pci_mem[2]; - -u32 mv64x60_cfg_read(u8 *bridge_base, u8 hose, u8 bus, u8 devfn, - u8 offset); -void mv64x60_cfg_write(u8 *bridge_base, u8 hose, u8 bus, u8 devfn, - u8 offset, u32 val); - -void mv64x60_config_ctlr_windows(u8 *bridge_base, u8 *bridge_pbase, - u8 is_coherent); -void mv64x60_config_pci_windows(u8 *bridge_base, u8 *bridge_pbase, u8 hose, - u8 bus, u32 mem_size, u32 acc_bits); -void mv64x60_config_cpu2pci_window(u8 *bridge_base, u8 hose, u32 pci_base_hi, - u32 pci_base_lo, u32 cpu_base, u32 size, - struct mv64x60_cpu2pci_win *offset_tbl); -u32 mv64x60_get_mem_size(u8 *bridge_base); -u8 *mv64x60_get_bridge_pbase(void); -u8 *mv64x60_get_bridge_base(void); -u8 mv64x60_is_coherent(void); - -int mv64x60_i2c_open(void); -int mv64x60_i2c_read(u32 devaddr, u8 *buf, u32 offset, u32 offset_size, - u32 count); -void mv64x60_i2c_close(void); - -#endif /* _PPC_BOOT_MV64x60_H_ */ diff --git a/arch/powerpc/boot/mv64x60_i2c.c b/arch/powerpc/boot/mv64x60_i2c.c deleted file mode 100644 index 52a3212b6638..000000000000 --- a/arch/powerpc/boot/mv64x60_i2c.c +++ /dev/null @@ -1,204 +0,0 @@ -/* - * Bootloader version of the i2c driver for the MV64x60. - * - * Author: Dale Farnsworth - * Maintained by: Mark A. Greer - * - * 2003, 2007 (c) MontaVista, Software, Inc. This file is licensed under - * the terms of the GNU General Public License version 2. This program is - * licensed "as is" without any warranty of any kind, whether express or - * implied. - */ - -#include -#include -#include "types.h" -#include "elf.h" -#include "page.h" -#include "string.h" -#include "stdio.h" -#include "io.h" -#include "ops.h" -#include "mv64x60.h" - -/* Register defines */ -#define MV64x60_I2C_REG_SLAVE_ADDR 0x00 -#define MV64x60_I2C_REG_DATA 0x04 -#define MV64x60_I2C_REG_CONTROL 0x08 -#define MV64x60_I2C_REG_STATUS 0x0c -#define MV64x60_I2C_REG_BAUD 0x0c -#define MV64x60_I2C_REG_EXT_SLAVE_ADDR 0x10 -#define MV64x60_I2C_REG_SOFT_RESET 0x1c - -#define MV64x60_I2C_CONTROL_ACK 0x04 -#define MV64x60_I2C_CONTROL_IFLG 0x08 -#define MV64x60_I2C_CONTROL_STOP 0x10 -#define MV64x60_I2C_CONTROL_START 0x20 -#define MV64x60_I2C_CONTROL_TWSIEN 0x40 -#define MV64x60_I2C_CONTROL_INTEN 0x80 - -#define MV64x60_I2C_STATUS_BUS_ERR 0x00 -#define MV64x60_I2C_STATUS_MAST_START 0x08 -#define MV64x60_I2C_STATUS_MAST_REPEAT_START 0x10 -#define MV64x60_I2C_STATUS_MAST_WR_ADDR_ACK 0x18 -#define MV64x60_I2C_STATUS_MAST_WR_ADDR_NO_ACK 0x20 -#define MV64x60_I2C_STATUS_MAST_WR_ACK 0x28 -#define MV64x60_I2C_STATUS_MAST_WR_NO_ACK 0x30 -#define MV64x60_I2C_STATUS_MAST_LOST_ARB 0x38 -#define MV64x60_I2C_STATUS_MAST_RD_ADDR_ACK 0x40 -#define MV64x60_I2C_STATUS_MAST_RD_ADDR_NO_ACK 0x48 -#define MV64x60_I2C_STATUS_MAST_RD_DATA_ACK 0x50 -#define MV64x60_I2C_STATUS_MAST_RD_DATA_NO_ACK 0x58 -#define MV64x60_I2C_STATUS_MAST_WR_ADDR_2_ACK 0xd0 -#define MV64x60_I2C_STATUS_MAST_WR_ADDR_2_NO_ACK 0xd8 -#define MV64x60_I2C_STATUS_MAST_RD_ADDR_2_ACK 0xe0 -#define MV64x60_I2C_STATUS_MAST_RD_ADDR_2_NO_ACK 0xe8 -#define MV64x60_I2C_STATUS_NO_STATUS 0xf8 - -static u8 *ctlr_base; - -static int mv64x60_i2c_wait_for_status(int wanted) -{ - int i; - int status; - - for (i=0; i<1000; i++) { - udelay(10); - status = in_le32((u32 *)(ctlr_base + MV64x60_I2C_REG_STATUS)) - & 0xff; - if (status == wanted) - return status; - } - return -status; -} - -static int mv64x60_i2c_control(int control, int status) -{ - out_le32((u32 *)(ctlr_base + MV64x60_I2C_REG_CONTROL), control & 0xff); - return mv64x60_i2c_wait_for_status(status); -} - -static int mv64x60_i2c_read_byte(int control, int status) -{ - out_le32((u32 *)(ctlr_base + MV64x60_I2C_REG_CONTROL), control & 0xff); - if (mv64x60_i2c_wait_for_status(status) < 0) - return -1; - return in_le32((u32 *)(ctlr_base + MV64x60_I2C_REG_DATA)) & 0xff; -} - -static int mv64x60_i2c_write_byte(int data, int control, int status) -{ - out_le32((u32 *)(ctlr_base + MV64x60_I2C_REG_DATA), data & 0xff); - out_le32((u32 *)(ctlr_base + MV64x60_I2C_REG_CONTROL), control & 0xff); - return mv64x60_i2c_wait_for_status(status); -} - -int mv64x60_i2c_read(u32 devaddr, u8 *buf, u32 offset, u32 offset_size, - u32 count) -{ - int i; - int data; - int control; - int status; - - if (ctlr_base == NULL) - return -1; - - /* send reset */ - out_le32((u32 *)(ctlr_base + MV64x60_I2C_REG_SOFT_RESET), 0); - out_le32((u32 *)(ctlr_base + MV64x60_I2C_REG_SLAVE_ADDR), 0); - out_le32((u32 *)(ctlr_base + MV64x60_I2C_REG_EXT_SLAVE_ADDR), 0); - out_le32((u32 *)(ctlr_base + MV64x60_I2C_REG_BAUD), (4 << 3) | 0x4); - - if (mv64x60_i2c_control(MV64x60_I2C_CONTROL_TWSIEN, - MV64x60_I2C_STATUS_NO_STATUS) < 0) - return -1; - - /* send start */ - control = MV64x60_I2C_CONTROL_START | MV64x60_I2C_CONTROL_TWSIEN; - status = MV64x60_I2C_STATUS_MAST_START; - if (mv64x60_i2c_control(control, status) < 0) - return -1; - - /* select device for writing */ - data = devaddr & ~0x1; - control = MV64x60_I2C_CONTROL_TWSIEN; - status = MV64x60_I2C_STATUS_MAST_WR_ADDR_ACK; - if (mv64x60_i2c_write_byte(data, control, status) < 0) - return -1; - - /* send offset of data */ - control = MV64x60_I2C_CONTROL_TWSIEN; - status = MV64x60_I2C_STATUS_MAST_WR_ACK; - if (offset_size > 1) { - if (mv64x60_i2c_write_byte(offset >> 8, control, status) < 0) - return -1; - } - if (mv64x60_i2c_write_byte(offset, control, status) < 0) - return -1; - - /* resend start */ - control = MV64x60_I2C_CONTROL_START | MV64x60_I2C_CONTROL_TWSIEN; - status = MV64x60_I2C_STATUS_MAST_REPEAT_START; - if (mv64x60_i2c_control(control, status) < 0) - return -1; - - /* select device for reading */ - data = devaddr | 0x1; - control = MV64x60_I2C_CONTROL_TWSIEN; - status = MV64x60_I2C_STATUS_MAST_RD_ADDR_ACK; - if (mv64x60_i2c_write_byte(data, control, status) < 0) - return -1; - - /* read all but last byte of data */ - control = MV64x60_I2C_CONTROL_ACK | MV64x60_I2C_CONTROL_TWSIEN; - status = MV64x60_I2C_STATUS_MAST_RD_DATA_ACK; - - for (i=1; i #endif /* _ASM_POWERPC_BARRIER_H */ diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h index 5073cc75f1c8..6a6673907e45 100644 --- a/arch/powerpc/include/asm/book3s/32/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h @@ -99,6 +99,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage) static inline void pgtable_free(void *table, unsigned index_size) { if (!index_size) { + pgtable_page_dtor(virt_to_page(table)); free_page((unsigned long)table); } else { BUG_ON(index_size > MAX_PGTABLE_INDEX_SIZE); diff --git a/arch/powerpc/include/asm/book3s/32/pgtable.h b/arch/powerpc/include/asm/book3s/32/pgtable.h index c615abdce119..02f5acd7ccc4 100644 --- a/arch/powerpc/include/asm/book3s/32/pgtable.h +++ b/arch/powerpc/include/asm/book3s/32/pgtable.h @@ -235,15 +235,18 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm, } -static inline void __ptep_set_access_flags(struct mm_struct *mm, +static inline void __ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, pte_t entry, - unsigned long address) + unsigned long address, + int psize) { unsigned long set = pte_val(entry) & (_PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_RW | _PAGE_EXEC); unsigned long clr = ~pte_val(entry) & _PAGE_RO; pte_update(ptep, clr, set); + + flush_tlb_page(vma, address); } #define __HAVE_ARCH_PTE_SAME diff --git a/arch/powerpc/include/asm/book3s/64/hash-4k.h b/arch/powerpc/include/asm/book3s/64/hash-4k.h index 4b5423030d4b..9a3798660cef 100644 --- a/arch/powerpc/include/asm/book3s/64/hash-4k.h +++ b/arch/powerpc/include/asm/book3s/64/hash-4k.h @@ -38,8 +38,12 @@ #define H_PAGE_4K_PFN 0x0 #define H_PAGE_THP_HUGE 0x0 #define H_PAGE_COMBO 0x0 -#define H_PTE_FRAG_NR 0 -#define H_PTE_FRAG_SIZE_SHIFT 0 + +/* 8 bytes per each pte entry */ +#define H_PTE_FRAG_SIZE_SHIFT (H_PTE_INDEX_SIZE + 3) +#define H_PTE_FRAG_NR (PAGE_SIZE >> H_PTE_FRAG_SIZE_SHIFT) +#define H_PMD_FRAG_SIZE_SHIFT (H_PMD_INDEX_SIZE + 3) +#define H_PMD_FRAG_NR (PAGE_SIZE >> H_PMD_FRAG_SIZE_SHIFT) /* memory key bits, only 8 keys supported */ #define H_PTE_PKEY_BIT0 0 diff --git a/arch/powerpc/include/asm/book3s/64/hash-64k.h b/arch/powerpc/include/asm/book3s/64/hash-64k.h index cc82745355b3..c81793d47af9 100644 --- a/arch/powerpc/include/asm/book3s/64/hash-64k.h +++ b/arch/powerpc/include/asm/book3s/64/hash-64k.h @@ -46,6 +46,13 @@ #define H_PTE_FRAG_SIZE_SHIFT (H_PTE_INDEX_SIZE + 3 + 1) #define H_PTE_FRAG_NR (PAGE_SIZE >> H_PTE_FRAG_SIZE_SHIFT) +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLB_PAGE) +#define H_PMD_FRAG_SIZE_SHIFT (H_PMD_INDEX_SIZE + 3 + 1) +#else +#define H_PMD_FRAG_SIZE_SHIFT (H_PMD_INDEX_SIZE + 3) +#endif +#define H_PMD_FRAG_NR (PAGE_SIZE >> H_PMD_FRAG_SIZE_SHIFT) + #ifndef __ASSEMBLY__ #include diff --git a/arch/powerpc/include/asm/book3s/64/hash.h b/arch/powerpc/include/asm/book3s/64/hash.h index cc8cd656ccfe..0387b155f13d 100644 --- a/arch/powerpc/include/asm/book3s/64/hash.h +++ b/arch/powerpc/include/asm/book3s/64/hash.h @@ -23,16 +23,6 @@ H_PUD_INDEX_SIZE + H_PGD_INDEX_SIZE + PAGE_SHIFT) #define H_PGTABLE_RANGE (ASM_CONST(1) << H_PGTABLE_EADDR_SIZE) -#if (defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLB_PAGE)) && \ - defined(CONFIG_PPC_64K_PAGES) -/* - * only with hash 64k we need to use the second half of pmd page table - * to store pointer to deposited pgtable_t - */ -#define H_PMD_CACHE_INDEX (H_PMD_INDEX_SIZE + 1) -#else -#define H_PMD_CACHE_INDEX H_PMD_INDEX_SIZE -#endif /* * We store the slot details in the second half of page table. * Increase the pud level table so that hugetlb ptes can be stored diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h index 5094696eecd6..9c8c669a6b6a 100644 --- a/arch/powerpc/include/asm/book3s/64/mmu.h +++ b/arch/powerpc/include/asm/book3s/64/mmu.h @@ -134,10 +134,11 @@ typedef struct { #ifdef CONFIG_PPC_SUBPAGE_PROT struct subpage_prot_table spt; #endif /* CONFIG_PPC_SUBPAGE_PROT */ -#ifdef CONFIG_PPC_64K_PAGES - /* for 4K PTE fragment support */ + /* + * pagetable fragment support + */ void *pte_frag; -#endif + void *pmd_frag; #ifdef CONFIG_SPAPR_TCE_IOMMU struct list_head iommu_group_mem_list; #endif diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h index 558a159600ad..01ee40f11f3a 100644 --- a/arch/powerpc/include/asm/book3s/64/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h @@ -42,7 +42,9 @@ extern struct kmem_cache *pgtable_cache[]; }) extern pte_t *pte_fragment_alloc(struct mm_struct *, unsigned long, int); +extern pmd_t *pmd_fragment_alloc(struct mm_struct *, unsigned long); extern void pte_fragment_free(unsigned long *, int); +extern void pmd_fragment_free(unsigned long *); extern void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift); #ifdef CONFIG_SMP extern void __tlb_remove_table(void *_table); @@ -88,8 +90,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) * need to do this for 4k. */ #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_PPC_64K_PAGES) && \ - ((H_PGD_INDEX_SIZE == H_PUD_CACHE_INDEX) || \ - (H_PGD_INDEX_SIZE == H_PMD_CACHE_INDEX)) + (H_PGD_INDEX_SIZE == H_PUD_CACHE_INDEX) memset(pgd, 0, PGD_TABLE_SIZE); #endif return pgd; @@ -124,36 +125,35 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) } static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, - unsigned long address) + unsigned long address) { /* * By now all the pud entries should be none entries. So go * ahead and flush the page walk cache */ flush_tlb_pgtable(tlb, address); - pgtable_free_tlb(tlb, pud, PUD_CACHE_INDEX); + pgtable_free_tlb(tlb, pud, PUD_INDEX); } static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX), - pgtable_gfp_flags(mm, GFP_KERNEL)); + return pmd_fragment_alloc(mm, addr); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { - kmem_cache_free(PGT_CACHE(PMD_CACHE_INDEX), pmd); + pmd_fragment_free((unsigned long *)pmd); } static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, - unsigned long address) + unsigned long address) { /* * By now all the pud entries should be none entries. So go * ahead and flush the page walk cache */ flush_tlb_pgtable(tlb, address); - return pgtable_free_tlb(tlb, pmd, PMD_CACHE_INDEX); + return pgtable_free_tlb(tlb, pmd, PMD_INDEX); } static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, @@ -173,31 +173,6 @@ static inline pgtable_t pmd_pgtable(pmd_t pmd) return (pgtable_t)pmd_page_vaddr(pmd); } -#ifdef CONFIG_PPC_4K_PAGES -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) -{ - return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); -} - -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) -{ - struct page *page; - pte_t *pte; - - pte = (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT); - if (!pte) - return NULL; - page = virt_to_page(pte); - if (!pgtable_page_ctor(page)) { - __free_page(page); - return NULL; - } - return pte; -} -#else /* if CONFIG_PPC_64K_PAGES */ - static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) { @@ -209,7 +184,6 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm, { return (pgtable_t)pte_fragment_alloc(mm, address, 0); } -#endif static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { @@ -229,7 +203,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table, * ahead and flush the page walk cache */ flush_tlb_pgtable(tlb, address); - pgtable_free_tlb(tlb, table, 0); + pgtable_free_tlb(tlb, table, PTE_INDEX); } #define check_pgt_cache() do { } while (0) diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 47b5ffc8715d..63cee159022b 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -212,13 +212,13 @@ extern unsigned long __pte_index_size; extern unsigned long __pmd_index_size; extern unsigned long __pud_index_size; extern unsigned long __pgd_index_size; -extern unsigned long __pmd_cache_index; extern unsigned long __pud_cache_index; #define PTE_INDEX_SIZE __pte_index_size #define PMD_INDEX_SIZE __pmd_index_size #define PUD_INDEX_SIZE __pud_index_size #define PGD_INDEX_SIZE __pgd_index_size -#define PMD_CACHE_INDEX __pmd_cache_index +/* pmd table use page table fragments */ +#define PMD_CACHE_INDEX 0 #define PUD_CACHE_INDEX __pud_cache_index /* * Because of use of pte fragments and THP, size of page table @@ -246,6 +246,12 @@ extern unsigned long __pte_frag_size_shift; #define PTE_FRAG_SIZE_SHIFT __pte_frag_size_shift #define PTE_FRAG_SIZE (1UL << PTE_FRAG_SIZE_SHIFT) +extern unsigned long __pmd_frag_nr; +#define PMD_FRAG_NR __pmd_frag_nr +extern unsigned long __pmd_frag_size_shift; +#define PMD_FRAG_SIZE_SHIFT __pmd_frag_size_shift +#define PMD_FRAG_SIZE (1UL << PMD_FRAG_SIZE_SHIFT) + #define PTRS_PER_PTE (1 << PTE_INDEX_SIZE) #define PTRS_PER_PMD (1 << PMD_INDEX_SIZE) #define PTRS_PER_PUD (1 << PUD_INDEX_SIZE) @@ -273,6 +279,16 @@ extern unsigned long __pte_frag_size_shift; /* Bits to mask out from a PGD to get to the PUD page */ #define PGD_MASKED_BITS 0xc0000000000000ffUL +/* + * Used as an indicator for rcu callback functions + */ +enum pgtable_index { + PTE_INDEX = 0, + PMD_INDEX, + PUD_INDEX, + PGD_INDEX, +}; + extern unsigned long __vmalloc_start; extern unsigned long __vmalloc_end; #define VMALLOC_START __vmalloc_start @@ -319,9 +335,6 @@ extern unsigned long pci_io_base; /* Advertise special mapping type for AGP */ #define HAVE_PAGE_AGP -/* Advertise support for _PAGE_SPECIAL */ -#define __HAVE_ARCH_PTE_SPECIAL - #ifndef __ASSEMBLY__ /* @@ -751,12 +764,14 @@ static inline bool check_pte_access(unsigned long access, unsigned long ptev) * Generic functions with hash/radix callbacks */ -static inline void __ptep_set_access_flags(struct mm_struct *mm, +static inline void __ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, pte_t entry, - unsigned long address) + unsigned long address, + int psize) { if (radix_enabled()) - return radix__ptep_set_access_flags(mm, ptep, entry, address); + return radix__ptep_set_access_flags(vma, ptep, entry, + address, psize); return hash__ptep_set_access_flags(ptep, entry); } diff --git a/arch/powerpc/include/asm/book3s/64/radix-4k.h b/arch/powerpc/include/asm/book3s/64/radix-4k.h index ca366ec86310..863c3e8286fb 100644 --- a/arch/powerpc/include/asm/book3s/64/radix-4k.h +++ b/arch/powerpc/include/asm/book3s/64/radix-4k.h @@ -15,4 +15,7 @@ #define RADIX_PTE_FRAG_SIZE_SHIFT (RADIX_PTE_INDEX_SIZE + 3) #define RADIX_PTE_FRAG_NR (PAGE_SIZE >> RADIX_PTE_FRAG_SIZE_SHIFT) +#define RADIX_PMD_FRAG_SIZE_SHIFT (RADIX_PMD_INDEX_SIZE + 3) +#define RADIX_PMD_FRAG_NR (PAGE_SIZE >> RADIX_PMD_FRAG_SIZE_SHIFT) + #endif /* _ASM_POWERPC_PGTABLE_RADIX_4K_H */ diff --git a/arch/powerpc/include/asm/book3s/64/radix-64k.h b/arch/powerpc/include/asm/book3s/64/radix-64k.h index 830082496876..ccb78ca9d0c5 100644 --- a/arch/powerpc/include/asm/book3s/64/radix-64k.h +++ b/arch/powerpc/include/asm/book3s/64/radix-64k.h @@ -16,4 +16,8 @@ */ #define RADIX_PTE_FRAG_SIZE_SHIFT (RADIX_PTE_INDEX_SIZE + 3) #define RADIX_PTE_FRAG_NR (PAGE_SIZE >> RADIX_PTE_FRAG_SIZE_SHIFT) + +#define RADIX_PMD_FRAG_SIZE_SHIFT (RADIX_PMD_INDEX_SIZE + 3) +#define RADIX_PMD_FRAG_NR (PAGE_SIZE >> RADIX_PMD_FRAG_SIZE_SHIFT) + #endif /* _ASM_POWERPC_PGTABLE_RADIX_64K_H */ diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 705193e7192f..ef9f96742ce1 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -124,23 +124,28 @@ extern void radix__mark_rodata_ro(void); extern void radix__mark_initmem_nx(void); #endif +extern void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, + pte_t entry, unsigned long address, + int psize); + static inline unsigned long __radix_pte_update(pte_t *ptep, unsigned long clr, unsigned long set) { - pte_t pte; - unsigned long old_pte, new_pte; - - do { - pte = READ_ONCE(*ptep); - old_pte = pte_val(pte); - new_pte = (old_pte | set) & ~clr; - - } while (!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); - - return old_pte; + __be64 old_be, tmp_be; + + __asm__ __volatile__( + "1: ldarx %0,0,%3 # pte_update\n" + " andc %1,%0,%5 \n" + " or %1,%1,%4 \n" + " stdcx. %1,0,%3 \n" + " bne- 1b" + : "=&r" (old_be), "=&r" (tmp_be), "=m" (*ptep) + : "r" (ptep), "r" (cpu_to_be64(set)), "r" (cpu_to_be64(clr)) + : "cc" ); + + return be64_to_cpu(old_be); } - static inline unsigned long radix__pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long clr, @@ -176,48 +181,14 @@ static inline pte_t radix__ptep_get_and_clear_full(struct mm_struct *mm, unsigned long old_pte; if (full) { - /* - * If we are trying to clear the pte, we can skip - * the DD1 pte update sequence and batch the tlb flush. The - * tlb flush batching is done by mmu gather code. We - * still keep the cmp_xchg update to make sure we get - * correct R/C bit which might be updated via Nest MMU. - */ - old_pte = __radix_pte_update(ptep, ~0ul, 0); + old_pte = pte_val(*ptep); + *ptep = __pte(0); } else old_pte = radix__pte_update(mm, addr, ptep, ~0ul, 0, 0); return __pte(old_pte); } -/* - * Set the dirty and/or accessed bits atomically in a linux PTE, this - * function doesn't need to invalidate tlb. - */ -static inline void radix__ptep_set_access_flags(struct mm_struct *mm, - pte_t *ptep, pte_t entry, - unsigned long address) -{ - - unsigned long set = pte_val(entry) & (_PAGE_DIRTY | _PAGE_ACCESSED | - _PAGE_RW | _PAGE_EXEC); - - if (cpu_has_feature(CPU_FTR_POWER9_DD1)) { - - unsigned long old_pte, new_pte; - - old_pte = __radix_pte_update(ptep, ~0, 0); - /* - * new value of pte - */ - new_pte = old_pte | set; - radix__flush_tlb_pte_p9_dd1(old_pte, mm, address); - __radix_pte_update(ptep, 0, new_pte); - } else - __radix_pte_update(ptep, 0, set); - asm volatile("ptesync" : : : "memory"); -} - static inline int radix__pte_same(pte_t pte_a, pte_t pte_b) { return ((pte_raw(pte_a) ^ pte_raw(pte_b)) == 0); @@ -232,7 +203,24 @@ static inline void radix__set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, int percpu) { *ptep = pte; - asm volatile("ptesync" : : : "memory"); + + /* + * The architecture suggests a ptesync after setting the pte, which + * orders the store that updates the pte with subsequent page table + * walk accesses which may load the pte. Without this it may be + * possible for a subsequent access to result in spurious fault. + * + * This is not necessary for correctness, because a spurious fault + * is tolerated by the page fault handler, and this store will + * eventually be seen. In testing, there was no noticable increase + * in user faults on POWER9. Avoiding ptesync here is a significant + * win for things like fork. If a future microarchitecture benefits + * from ptesync, it should probably go into update_mmu_cache, rather + * than set_pte_at (which is used to set ptes unrelated to faults). + * + * Spurious faults to vmalloc region are not tolerated, so there is + * a ptesync in flush_cache_vmap. + */ } static inline int radix__pmd_bad(pmd_t pmd) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h index 19b45ba6caf9..ef5c3f2994c9 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h @@ -51,4 +51,11 @@ extern void radix__flush_tlb_all(void); extern void radix__flush_tlb_pte_p9_dd1(unsigned long old_pte, struct mm_struct *mm, unsigned long address); +extern void radix__flush_tlb_lpid_page(unsigned int lpid, + unsigned long addr, + unsigned long page_size); +extern void radix__flush_pwc_lpid(unsigned int lpid); +extern void radix__local_flush_tlb_lpid(unsigned int lpid); +extern void radix__local_flush_tlb_lpid_guest(unsigned int lpid); + #endif diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index 0cac17253513..ebf572ea621e 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -4,7 +4,7 @@ #define MMU_NO_CONTEXT ~0UL - +#include #include #include @@ -137,6 +137,16 @@ static inline void flush_all_mm(struct mm_struct *mm) #define flush_tlb_page(vma, addr) local_flush_tlb_page(vma, addr) #define flush_all_mm(mm) local_flush_all_mm(mm) #endif /* CONFIG_SMP */ + +#define flush_tlb_fix_spurious_fault flush_tlb_fix_spurious_fault +static inline void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, + unsigned long address) +{ + /* See ptep_set_access_flags comment */ + if (atomic_read(&vma->vm_mm->context.copros) > 0) + flush_tlb_page(vma, address); +} + /* * flush the page walk cache for the address */ diff --git a/arch/powerpc/include/asm/cache.h b/arch/powerpc/include/asm/cache.h index c1d257aa4c2d..66298461b640 100644 --- a/arch/powerpc/include/asm/cache.h +++ b/arch/powerpc/include/asm/cache.h @@ -9,11 +9,14 @@ #if defined(CONFIG_PPC_8xx) || defined(CONFIG_403GCX) #define L1_CACHE_SHIFT 4 #define MAX_COPY_PREFETCH 1 +#define IFETCH_ALIGN_SHIFT 2 #elif defined(CONFIG_PPC_E500MC) #define L1_CACHE_SHIFT 6 #define MAX_COPY_PREFETCH 4 +#define IFETCH_ALIGN_SHIFT 3 #elif defined(CONFIG_PPC32) #define MAX_COPY_PREFETCH 4 +#define IFETCH_ALIGN_SHIFT 3 /* 603 fetches 2 insn at a time */ #if defined(CONFIG_PPC_47x) #define L1_CACHE_SHIFT 7 #else diff --git a/arch/powerpc/include/asm/cacheflush.h b/arch/powerpc/include/asm/cacheflush.h index 11843e37d9cf..0d72ec75da63 100644 --- a/arch/powerpc/include/asm/cacheflush.h +++ b/arch/powerpc/include/asm/cacheflush.h @@ -23,9 +23,21 @@ #define flush_cache_range(vma, start, end) do { } while (0) #define flush_cache_page(vma, vmaddr, pfn) do { } while (0) #define flush_icache_page(vma, page) do { } while (0) -#define flush_cache_vmap(start, end) do { } while (0) #define flush_cache_vunmap(start, end) do { } while (0) +#ifdef CONFIG_PPC_BOOK3S_64 +/* + * Book3s has no ptesync after setting a pte, so without this ptesync it's + * possible for a kernel virtual mapping access to return a spurious fault + * if it's accessed right after the pte is set. The page fault handler does + * not expect this type of fault. flush_cache_vmap is not exactly the right + * place to put this, but it seems to work well enough. + */ +#define flush_cache_vmap(start, end) do { asm volatile("ptesync" ::: "memory"); } while (0) +#else +#define flush_cache_vmap(start, end) do { } while (0) +#endif + #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 extern void flush_dcache_page(struct page *page); #define flush_dcache_mmap_lock(mapping) do { } while (0) diff --git a/arch/powerpc/include/asm/checksum.h b/arch/powerpc/include/asm/checksum.h index 842124b199b5..a78a57e5058d 100644 --- a/arch/powerpc/include/asm/checksum.h +++ b/arch/powerpc/include/asm/checksum.h @@ -12,6 +12,8 @@ #ifdef CONFIG_GENERIC_CSUM #include #else +#include +#include /* * Computes the checksum of a memory block at src, length len, * and adds in "sum" (32-bit), while copying the block to dst. @@ -55,11 +57,7 @@ static inline __sum16 csum_fold(__wsum sum) static inline u32 from64to32(u64 x) { - /* add up 32-bit and 32-bit for 32+c bit */ - x = (x & 0xffffffff) + (x >> 32); - /* add up carry.. */ - x = (x & 0xffffffff) + (x >> 32); - return (u32)x; + return (x + ror64(x, 32)) >> 32; } static inline __wsum csum_tcpudp_nofold(__be32 saddr, __be32 daddr, __u32 len, @@ -112,7 +110,7 @@ static inline __wsum csum_add(__wsum csum, __wsum addend) #ifdef __powerpc64__ res += (__force u64)addend; - return (__force __wsum) from64to32(res); + return (__force __wsum)((u32)res + (res >> 32)); #else asm("addc %0,%0,%1;" "addze %0,%0;" @@ -214,6 +212,11 @@ static inline __sum16 ip_compute_csum(const void *buff, int len) return csum_fold(csum_partial(buff, len, 0)); } +#define _HAVE_ARCH_IPV6_CSUM +__sum16 csum_ipv6_magic(const struct in6_addr *saddr, + const struct in6_addr *daddr, + __u32 len, __u8 proto, __wsum sum); + #endif #endif /* __KERNEL__ */ #endif diff --git a/arch/powerpc/include/asm/compat.h b/arch/powerpc/include/asm/compat.h index 62168e1158f1..85c8af2bb272 100644 --- a/arch/powerpc/include/asm/compat.h +++ b/arch/powerpc/include/asm/compat.h @@ -17,7 +17,6 @@ typedef u32 compat_size_t; typedef s32 compat_ssize_t; -typedef s32 compat_time_t; typedef s32 compat_clock_t; typedef s32 compat_pid_t; typedef u32 __compat_uid_t; @@ -45,16 +44,6 @@ typedef u32 compat_ulong_t; typedef u64 compat_u64; typedef u32 compat_uptr_t; -struct compat_timespec { - compat_time_t tv_sec; - s32 tv_nsec; -}; - -struct compat_timeval { - compat_time_t tv_sec; - s32 tv_usec; -}; - struct compat_stat { compat_dev_t st_dev; compat_ino_t st_ino; @@ -173,10 +162,10 @@ struct compat_ipc64_perm { struct compat_semid64_ds { struct compat_ipc64_perm sem_perm; - unsigned int __unused1; - compat_time_t sem_otime; - unsigned int __unused2; - compat_time_t sem_ctime; + unsigned int sem_otime_high; + unsigned int sem_otime; + unsigned int sem_ctime_high; + unsigned int sem_ctime; compat_ulong_t sem_nsems; compat_ulong_t __unused3; compat_ulong_t __unused4; @@ -184,12 +173,12 @@ struct compat_semid64_ds { struct compat_msqid64_ds { struct compat_ipc64_perm msg_perm; - unsigned int __unused1; - compat_time_t msg_stime; - unsigned int __unused2; - compat_time_t msg_rtime; - unsigned int __unused3; - compat_time_t msg_ctime; + unsigned int msg_stime_high; + unsigned int msg_stime; + unsigned int msg_rtime_high; + unsigned int msg_rtime; + unsigned int msg_ctime_high; + unsigned int msg_ctime; compat_ulong_t msg_cbytes; compat_ulong_t msg_qnum; compat_ulong_t msg_qbytes; @@ -201,12 +190,12 @@ struct compat_msqid64_ds { struct compat_shmid64_ds { struct compat_ipc64_perm shm_perm; - unsigned int __unused1; - compat_time_t shm_atime; - unsigned int __unused2; - compat_time_t shm_dtime; - unsigned int __unused3; - compat_time_t shm_ctime; + unsigned int shm_atime_high; + unsigned int shm_atime; + unsigned int shm_dtime_high; + unsigned int shm_dtime; + unsigned int shm_ctime_high; + unsigned int shm_ctime; unsigned int __unused4; compat_size_t shm_segsz; compat_pid_t shm_cpid; diff --git a/arch/powerpc/include/asm/cputable.h b/arch/powerpc/include/asm/cputable.h index 66fcab13c8b4..9c0a3083571b 100644 --- a/arch/powerpc/include/asm/cputable.h +++ b/arch/powerpc/include/asm/cputable.h @@ -215,6 +215,7 @@ static inline void cpu_feature_keys_init(void) { } #define CPU_FTR_P9_TM_HV_ASSIST LONG_ASM_CONST(0x0000100000000000) #define CPU_FTR_P9_TM_XER_SO_BUG LONG_ASM_CONST(0x0000200000000000) #define CPU_FTR_P9_TLBIE_BUG LONG_ASM_CONST(0x0000400000000000) +#define CPU_FTR_P9_TIDR LONG_ASM_CONST(0x0000800000000000) #ifndef __ASSEMBLY__ @@ -462,7 +463,7 @@ static inline void cpu_feature_keys_init(void) { } CPU_FTR_CFAR | CPU_FTR_HVMODE | CPU_FTR_VMX_COPY | \ CPU_FTR_DBELL | CPU_FTR_HAS_PPR | CPU_FTR_ARCH_207S | \ CPU_FTR_TM_COMP | CPU_FTR_ARCH_300 | CPU_FTR_PKEY | \ - CPU_FTR_P9_TLBIE_BUG) + CPU_FTR_P9_TLBIE_BUG | CPU_FTR_P9_TIDR) #define CPU_FTRS_POWER9_DD1 ((CPU_FTRS_POWER9 | CPU_FTR_POWER9_DD1) & \ (~CPU_FTR_SAO)) #define CPU_FTRS_POWER9_DD2_0 CPU_FTRS_POWER9 diff --git a/arch/powerpc/include/asm/cputime.h b/arch/powerpc/include/asm/cputime.h index 99b541865d8d..bc4903badb3f 100644 --- a/arch/powerpc/include/asm/cputime.h +++ b/arch/powerpc/include/asm/cputime.h @@ -47,9 +47,23 @@ static inline unsigned long cputime_to_usecs(const cputime_t ct) * has to be populated in the new task */ #ifdef CONFIG_PPC64 +#define get_accounting(tsk) (&get_paca()->accounting) static inline void arch_vtime_task_switch(struct task_struct *tsk) { } #else -void arch_vtime_task_switch(struct task_struct *tsk); +#define get_accounting(tsk) (&task_thread_info(tsk)->accounting) +/* + * Called from the context switch with interrupts disabled, to charge all + * accumulated times to the current process, and to prepare accounting on + * the next process. + */ +static inline void arch_vtime_task_switch(struct task_struct *prev) +{ + struct cpu_accounting_data *acct = get_accounting(current); + struct cpu_accounting_data *acct0 = get_accounting(prev); + + acct->starttime = acct0->starttime; + acct->startspurr = acct0->startspurr; +} #endif #endif /* __KERNEL__ */ diff --git a/arch/powerpc/include/asm/eeh.h b/arch/powerpc/include/asm/eeh.h index c2266ca61853..677102baf3cd 100644 --- a/arch/powerpc/include/asm/eeh.h +++ b/arch/powerpc/include/asm/eeh.h @@ -106,6 +106,9 @@ struct eeh_pe { #define eeh_pe_for_each_dev(pe, edev, tmp) \ list_for_each_entry_safe(edev, tmp, &pe->edevs, list) +#define eeh_for_each_pe(root, pe) \ + for (pe = root; pe; pe = eeh_pe_next(pe, root)) + static inline bool eeh_pe_passed(struct eeh_pe *pe) { return pe ? !!atomic_read(&pe->pass_dev_cnt) : false; @@ -262,19 +265,21 @@ static inline bool eeh_state_active(int state) == (EEH_STATE_MMIO_ACTIVE | EEH_STATE_DMA_ACTIVE); } -typedef void *(*eeh_traverse_func)(void *data, void *flag); +typedef void *(*eeh_edev_traverse_func)(struct eeh_dev *edev, void *flag); +typedef void *(*eeh_pe_traverse_func)(struct eeh_pe *pe, void *flag); void eeh_set_pe_aux_size(int size); int eeh_phb_pe_create(struct pci_controller *phb); struct eeh_pe *eeh_phb_pe_get(struct pci_controller *phb); +struct eeh_pe *eeh_pe_next(struct eeh_pe *pe, struct eeh_pe *root); struct eeh_pe *eeh_pe_get(struct pci_controller *phb, int pe_no, int config_addr); int eeh_add_to_parent_pe(struct eeh_dev *edev); int eeh_rmv_from_parent_pe(struct eeh_dev *edev); void eeh_pe_update_time_stamp(struct eeh_pe *pe); void *eeh_pe_traverse(struct eeh_pe *root, - eeh_traverse_func fn, void *flag); + eeh_pe_traverse_func fn, void *flag); void *eeh_pe_dev_traverse(struct eeh_pe *root, - eeh_traverse_func fn, void *flag); + eeh_edev_traverse_func fn, void *flag); void eeh_pe_restore_bars(struct eeh_pe *pe); const char *eeh_pe_loc_get(struct eeh_pe *pe); struct pci_bus *eeh_pe_bus_get(struct eeh_pe *pe); diff --git a/arch/powerpc/include/asm/feature-fixups.h b/arch/powerpc/include/asm/feature-fixups.h index a9b64df34e2a..fcfd05672b1b 100644 --- a/arch/powerpc/include/asm/feature-fixups.h +++ b/arch/powerpc/include/asm/feature-fixups.h @@ -211,6 +211,14 @@ label##3: \ FTR_ENTRY_OFFSET 951b-952b; \ .popsection; +#define NOSPEC_BARRIER_FIXUP_SECTION \ +953: \ + .pushsection __barrier_nospec_fixup,"a"; \ + .align 2; \ +954: \ + FTR_ENTRY_OFFSET 953b-954b; \ + .popsection; + #ifndef __ASSEMBLY__ #include @@ -219,6 +227,7 @@ extern long stf_barrier_fallback; extern long __start___stf_entry_barrier_fixup, __stop___stf_entry_barrier_fixup; extern long __start___stf_exit_barrier_fixup, __stop___stf_exit_barrier_fixup; extern long __start___rfi_flush_fixup, __stop___rfi_flush_fixup; +extern long __start___barrier_nospec_fixup, __stop___barrier_nospec_fixup; void apply_feature_fixups(void); void setup_feature_keys(void); diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index b2dabd06659d..3dfb80b86561 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -48,9 +48,6 @@ #else /* !__ASSEMBLY__ */ extern void _mcount(void); -#ifdef CONFIG_DYNAMIC_FTRACE -# define FTRACE_ADDR ((unsigned long)ftrace_caller) -# define FTRACE_REGS_ADDR FTRACE_ADDR static inline unsigned long ftrace_call_adjust(unsigned long addr) { /* reloction of mcount call site is the same as the address */ @@ -60,15 +57,15 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr) struct dyn_arch_ftrace { struct module *mod; }; -#endif /* CONFIG_DYNAMIC_FTRACE */ #endif /* __ASSEMBLY__ */ #ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS #define ARCH_SUPPORTS_FTRACE_OPS 1 #endif -#endif +#endif /* CONFIG_FUNCTION_TRACER */ -#if defined(CONFIG_FTRACE_SYSCALLS) && !defined(__ASSEMBLY__) +#ifndef __ASSEMBLY__ +#ifdef CONFIG_FTRACE_SYSCALLS /* * Some syscall entry functions on powerpc start with "ppc_" (fork and clone, * for instance) or ppc32_/ppc64_. We should also match the sys_ variant with @@ -94,7 +91,25 @@ static inline bool arch_syscall_match_sym_name(const char *sym, const char *name (!strncmp(sym, "ppc32_", 6) && !strcmp(sym + 6, name + 4)) || (!strncmp(sym, "ppc64_", 6) && !strcmp(sym + 6, name + 4)); } -#endif -#endif /* CONFIG_FTRACE_SYSCALLS && !__ASSEMBLY__ */ +#endif /* PPC64_ELF_ABI_v1 */ +#endif /* CONFIG_FTRACE_SYSCALLS */ + +#ifdef CONFIG_PPC64 +#include + +static inline void this_cpu_disable_ftrace(void) +{ + get_paca()->ftrace_enabled = 0; +} + +static inline void this_cpu_enable_ftrace(void) +{ + get_paca()->ftrace_enabled = 1; +} +#else /* CONFIG_PPC64 */ +static inline void this_cpu_disable_ftrace(void) { } +static inline void this_cpu_enable_ftrace(void) { } +#endif /* CONFIG_PPC64 */ +#endif /* !__ASSEMBLY__ */ #endif /* _ASM_POWERPC_FTRACE */ diff --git a/arch/powerpc/include/asm/hardirq.h b/arch/powerpc/include/asm/hardirq.h index 5986d473722b..f1e9067bd5ac 100644 --- a/arch/powerpc/include/asm/hardirq.h +++ b/arch/powerpc/include/asm/hardirq.h @@ -8,6 +8,7 @@ typedef struct { unsigned int __softirq_pending; unsigned int timer_irqs_event; + unsigned int broadcast_irqs_event; unsigned int timer_irqs_others; unsigned int pmu_irqs; unsigned int mce_exceptions; @@ -25,15 +26,8 @@ typedef struct { DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat); #define __ARCH_IRQ_STAT - -#define local_softirq_pending() __this_cpu_read(irq_stat.__softirq_pending) - -#define __ARCH_SET_SOFTIRQ_PENDING #define __ARCH_IRQ_EXIT_IRQS_DISABLED -#define set_softirq_pending(x) __this_cpu_write(irq_stat.__softirq_pending, (x)) -#define or_softirq_pending(x) __this_cpu_or(irq_stat.__softirq_pending, (x)) - static inline void ack_bad_irq(unsigned int irq) { printk(KERN_CRIT "unexpected IRQ trap at vector %02x\n", irq); diff --git a/arch/powerpc/include/asm/hugetlb.h b/arch/powerpc/include/asm/hugetlb.h index 78540c074d70..3225eb6402cc 100644 --- a/arch/powerpc/include/asm/hugetlb.h +++ b/arch/powerpc/include/asm/hugetlb.h @@ -166,22 +166,9 @@ static inline pte_t huge_pte_wrprotect(pte_t pte) return pte_wrprotect(pte); } -static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t pte, int dirty) -{ -#ifdef HUGETLB_NEED_PRELOAD - /* - * The "return 1" forces a call of update_mmu_cache, which will write a - * TLB entry. Without this, platforms that don't do a write of the TLB - * entry in the TLB miss handler asm will fault ad infinitum. - */ - ptep_set_access_flags(vma, addr, ptep, pte, dirty); - return 1; -#else - return ptep_set_access_flags(vma, addr, ptep, pte, dirty); -#endif -} +extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, int dirty); static inline pte_t huge_ptep_get(pte_t *ptep) { @@ -202,7 +189,7 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma, static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr, unsigned pdshift) { - return 0; + return NULL; } #endif /* CONFIG_HUGETLB_PAGE */ diff --git a/arch/powerpc/include/asm/hvcall.h b/arch/powerpc/include/asm/hvcall.h index 2e2dddab5d65..662c8347d699 100644 --- a/arch/powerpc/include/asm/hvcall.h +++ b/arch/powerpc/include/asm/hvcall.h @@ -279,6 +279,7 @@ #define H_GET_MPP_X 0x314 #define H_SET_MODE 0x31C #define H_CLEAR_HPT 0x358 +#define H_REQUEST_VMC 0x360 #define H_RESIZE_HPT_PREPARE 0x36C #define H_RESIZE_HPT_COMMIT 0x370 #define H_REGISTER_PROC_TBL 0x37C diff --git a/arch/powerpc/include/asm/hw_irq.h b/arch/powerpc/include/asm/hw_irq.h index 855e17d158b1..e151774cb577 100644 --- a/arch/powerpc/include/asm/hw_irq.h +++ b/arch/powerpc/include/asm/hw_irq.h @@ -55,6 +55,7 @@ extern void replay_system_reset(void); extern void __replay_interrupt(unsigned int vector); extern void timer_interrupt(struct pt_regs *); +extern void timer_broadcast_interrupt(void); extern void performance_monitor_exception(struct pt_regs *regs); extern void WatchdogException(struct pt_regs *regs); extern void unknown_exception(struct pt_regs *regs); @@ -228,8 +229,8 @@ static inline bool arch_irqs_disabled(void) #define __hard_irq_enable() asm volatile("wrteei 1" : : : "memory") #define __hard_irq_disable() asm volatile("wrteei 0" : : : "memory") #else -#define __hard_irq_enable() __mtmsrd(local_paca->kernel_msr | MSR_EE, 1) -#define __hard_irq_disable() __mtmsrd(local_paca->kernel_msr, 1) +#define __hard_irq_enable() __mtmsrd(MSR_EE|MSR_RI, 1) +#define __hard_irq_disable() __mtmsrd(MSR_RI, 1) #endif #define hard_irq_disable() do { \ @@ -237,8 +238,12 @@ static inline bool arch_irqs_disabled(void) __hard_irq_disable(); \ flags = irq_soft_mask_set_return(IRQS_ALL_DISABLED); \ local_paca->irq_happened |= PACA_IRQ_HARD_DIS; \ - if (!arch_irqs_disabled_flags(flags)) \ + if (!arch_irqs_disabled_flags(flags)) { \ + asm ("stdx %%r1, 0, %1 ;" \ + : "=m" (local_paca->saved_r1) \ + : "b" (&local_paca->saved_r1)); \ trace_hardirqs_off(); \ + } \ } while(0) static inline bool lazy_irq_pending(void) diff --git a/arch/powerpc/include/asm/imc-pmu.h b/arch/powerpc/include/asm/imc-pmu.h index d76cb11be3e3..69f516ecb2fd 100644 --- a/arch/powerpc/include/asm/imc-pmu.h +++ b/arch/powerpc/include/asm/imc-pmu.h @@ -128,4 +128,5 @@ extern int init_imc_pmu(struct device_node *parent, struct imc_pmu *pmu_ptr, int pmu_id); extern void thread_imc_disable(void); extern int get_max_nest_dev(void); +extern void unregister_thread_imc(void); #endif /* __ASM_POWERPC_IMC_PMU_H */ diff --git a/arch/powerpc/include/asm/io.h b/arch/powerpc/include/asm/io.h index af074923d598..e0331e754568 100644 --- a/arch/powerpc/include/asm/io.h +++ b/arch/powerpc/include/asm/io.h @@ -367,6 +367,11 @@ static inline void __raw_writeq(unsigned long v, volatile void __iomem *addr) *(volatile unsigned long __force *)PCI_FIX_ADDR(addr) = v; } +static inline void __raw_writeq_be(unsigned long v, volatile void __iomem *addr) +{ + __raw_writeq((__force unsigned long)cpu_to_be64(v), addr); +} + /* * Real mode versions of the above. Those instructions are only supposed * to be used in hypervisor real mode as per the architecture spec. @@ -395,6 +400,11 @@ static inline void __raw_rm_writeq(u64 val, volatile void __iomem *paddr) : : "r" (val), "r" (paddr) : "memory"); } +static inline void __raw_rm_writeq_be(u64 val, volatile void __iomem *paddr) +{ + __raw_rm_writeq((__force u64)cpu_to_be64(val), paddr); +} + static inline u8 __raw_rm_readb(volatile void __iomem *paddr) { u8 ret; diff --git a/arch/powerpc/include/asm/kvm_book3s.h b/arch/powerpc/include/asm/kvm_book3s.h index e7377b73cfec..1f345a0b6ba2 100644 --- a/arch/powerpc/include/asm/kvm_book3s.h +++ b/arch/powerpc/include/asm/kvm_book3s.h @@ -104,6 +104,7 @@ struct kvmppc_vcore { ulong vtb; /* virtual timebase */ ulong conferring_threads; unsigned int halt_poll_ns; + atomic_t online_count; }; struct kvmppc_vcpu_book3s { @@ -209,6 +210,7 @@ extern void kvmppc_book3s_queue_irqprio(struct kvm_vcpu *vcpu, unsigned int vec) extern void kvmppc_book3s_dequeue_irqprio(struct kvm_vcpu *vcpu, unsigned int vec); extern void kvmppc_inject_interrupt(struct kvm_vcpu *vcpu, int vec, u64 flags); +extern void kvmppc_trigger_fac_interrupt(struct kvm_vcpu *vcpu, ulong fac); extern void kvmppc_set_bat(struct kvm_vcpu *vcpu, struct kvmppc_bat *bat, bool upper, u32 val); extern void kvmppc_giveup_ext(struct kvm_vcpu *vcpu, ulong msr); @@ -256,6 +258,21 @@ extern int kvmppc_hcall_impl_pr(unsigned long cmd); extern int kvmppc_hcall_impl_hv_realmode(unsigned long cmd); extern void kvmppc_copy_to_svcpu(struct kvm_vcpu *vcpu); extern void kvmppc_copy_from_svcpu(struct kvm_vcpu *vcpu); + +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM +void kvmppc_save_tm_pr(struct kvm_vcpu *vcpu); +void kvmppc_restore_tm_pr(struct kvm_vcpu *vcpu); +void kvmppc_save_tm_sprs(struct kvm_vcpu *vcpu); +void kvmppc_restore_tm_sprs(struct kvm_vcpu *vcpu); +#else +static inline void kvmppc_save_tm_pr(struct kvm_vcpu *vcpu) {} +static inline void kvmppc_restore_tm_pr(struct kvm_vcpu *vcpu) {} +static inline void kvmppc_save_tm_sprs(struct kvm_vcpu *vcpu) {} +static inline void kvmppc_restore_tm_sprs(struct kvm_vcpu *vcpu) {} +#endif + +void kvmppc_giveup_fac(struct kvm_vcpu *vcpu, ulong fac); + extern int kvm_irq_bypass; static inline struct kvmppc_vcpu_book3s *to_book3s(struct kvm_vcpu *vcpu) @@ -274,12 +291,12 @@ static inline struct kvmppc_vcpu_book3s *to_book3s(struct kvm_vcpu *vcpu) static inline void kvmppc_set_gpr(struct kvm_vcpu *vcpu, int num, ulong val) { - vcpu->arch.gpr[num] = val; + vcpu->arch.regs.gpr[num] = val; } static inline ulong kvmppc_get_gpr(struct kvm_vcpu *vcpu, int num) { - return vcpu->arch.gpr[num]; + return vcpu->arch.regs.gpr[num]; } static inline void kvmppc_set_cr(struct kvm_vcpu *vcpu, u32 val) @@ -294,42 +311,42 @@ static inline u32 kvmppc_get_cr(struct kvm_vcpu *vcpu) static inline void kvmppc_set_xer(struct kvm_vcpu *vcpu, ulong val) { - vcpu->arch.xer = val; + vcpu->arch.regs.xer = val; } static inline ulong kvmppc_get_xer(struct kvm_vcpu *vcpu) { - return vcpu->arch.xer; + return vcpu->arch.regs.xer; } static inline void kvmppc_set_ctr(struct kvm_vcpu *vcpu, ulong val) { - vcpu->arch.ctr = val; + vcpu->arch.regs.ctr = val; } static inline ulong kvmppc_get_ctr(struct kvm_vcpu *vcpu) { - return vcpu->arch.ctr; + return vcpu->arch.regs.ctr; } static inline void kvmppc_set_lr(struct kvm_vcpu *vcpu, ulong val) { - vcpu->arch.lr = val; + vcpu->arch.regs.link = val; } static inline ulong kvmppc_get_lr(struct kvm_vcpu *vcpu) { - return vcpu->arch.lr; + return vcpu->arch.regs.link; } static inline void kvmppc_set_pc(struct kvm_vcpu *vcpu, ulong val) { - vcpu->arch.pc = val; + vcpu->arch.regs.nip = val; } static inline ulong kvmppc_get_pc(struct kvm_vcpu *vcpu) { - return vcpu->arch.pc; + return vcpu->arch.regs.nip; } static inline u64 kvmppc_get_msr(struct kvm_vcpu *vcpu); diff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h index c424e44f4c00..dc435a5af7d6 100644 --- a/arch/powerpc/include/asm/kvm_book3s_64.h +++ b/arch/powerpc/include/asm/kvm_book3s_64.h @@ -483,15 +483,15 @@ static inline u64 sanitize_msr(u64 msr) static inline void copy_from_checkpoint(struct kvm_vcpu *vcpu) { vcpu->arch.cr = vcpu->arch.cr_tm; - vcpu->arch.xer = vcpu->arch.xer_tm; - vcpu->arch.lr = vcpu->arch.lr_tm; - vcpu->arch.ctr = vcpu->arch.ctr_tm; + vcpu->arch.regs.xer = vcpu->arch.xer_tm; + vcpu->arch.regs.link = vcpu->arch.lr_tm; + vcpu->arch.regs.ctr = vcpu->arch.ctr_tm; vcpu->arch.amr = vcpu->arch.amr_tm; vcpu->arch.ppr = vcpu->arch.ppr_tm; vcpu->arch.dscr = vcpu->arch.dscr_tm; vcpu->arch.tar = vcpu->arch.tar_tm; - memcpy(vcpu->arch.gpr, vcpu->arch.gpr_tm, - sizeof(vcpu->arch.gpr)); + memcpy(vcpu->arch.regs.gpr, vcpu->arch.gpr_tm, + sizeof(vcpu->arch.regs.gpr)); vcpu->arch.fp = vcpu->arch.fp_tm; vcpu->arch.vr = vcpu->arch.vr_tm; vcpu->arch.vrsave = vcpu->arch.vrsave_tm; @@ -500,15 +500,15 @@ static inline void copy_from_checkpoint(struct kvm_vcpu *vcpu) static inline void copy_to_checkpoint(struct kvm_vcpu *vcpu) { vcpu->arch.cr_tm = vcpu->arch.cr; - vcpu->arch.xer_tm = vcpu->arch.xer; - vcpu->arch.lr_tm = vcpu->arch.lr; - vcpu->arch.ctr_tm = vcpu->arch.ctr; + vcpu->arch.xer_tm = vcpu->arch.regs.xer; + vcpu->arch.lr_tm = vcpu->arch.regs.link; + vcpu->arch.ctr_tm = vcpu->arch.regs.ctr; vcpu->arch.amr_tm = vcpu->arch.amr; vcpu->arch.ppr_tm = vcpu->arch.ppr; vcpu->arch.dscr_tm = vcpu->arch.dscr; vcpu->arch.tar_tm = vcpu->arch.tar; - memcpy(vcpu->arch.gpr_tm, vcpu->arch.gpr, - sizeof(vcpu->arch.gpr)); + memcpy(vcpu->arch.gpr_tm, vcpu->arch.regs.gpr, + sizeof(vcpu->arch.regs.gpr)); vcpu->arch.fp_tm = vcpu->arch.fp; vcpu->arch.vr_tm = vcpu->arch.vr; vcpu->arch.vrsave_tm = vcpu->arch.vrsave; diff --git a/arch/powerpc/include/asm/kvm_booke.h b/arch/powerpc/include/asm/kvm_booke.h index bc6e29e4dfd4..d513e3ed1c65 100644 --- a/arch/powerpc/include/asm/kvm_booke.h +++ b/arch/powerpc/include/asm/kvm_booke.h @@ -36,12 +36,12 @@ static inline void kvmppc_set_gpr(struct kvm_vcpu *vcpu, int num, ulong val) { - vcpu->arch.gpr[num] = val; + vcpu->arch.regs.gpr[num] = val; } static inline ulong kvmppc_get_gpr(struct kvm_vcpu *vcpu, int num) { - return vcpu->arch.gpr[num]; + return vcpu->arch.regs.gpr[num]; } static inline void kvmppc_set_cr(struct kvm_vcpu *vcpu, u32 val) @@ -56,12 +56,12 @@ static inline u32 kvmppc_get_cr(struct kvm_vcpu *vcpu) static inline void kvmppc_set_xer(struct kvm_vcpu *vcpu, ulong val) { - vcpu->arch.xer = val; + vcpu->arch.regs.xer = val; } static inline ulong kvmppc_get_xer(struct kvm_vcpu *vcpu) { - return vcpu->arch.xer; + return vcpu->arch.regs.xer; } static inline bool kvmppc_need_byteswap(struct kvm_vcpu *vcpu) @@ -72,32 +72,32 @@ static inline bool kvmppc_need_byteswap(struct kvm_vcpu *vcpu) static inline void kvmppc_set_ctr(struct kvm_vcpu *vcpu, ulong val) { - vcpu->arch.ctr = val; + vcpu->arch.regs.ctr = val; } static inline ulong kvmppc_get_ctr(struct kvm_vcpu *vcpu) { - return vcpu->arch.ctr; + return vcpu->arch.regs.ctr; } static inline void kvmppc_set_lr(struct kvm_vcpu *vcpu, ulong val) { - vcpu->arch.lr = val; + vcpu->arch.regs.link = val; } static inline ulong kvmppc_get_lr(struct kvm_vcpu *vcpu) { - return vcpu->arch.lr; + return vcpu->arch.regs.link; } static inline void kvmppc_set_pc(struct kvm_vcpu *vcpu, ulong val) { - vcpu->arch.pc = val; + vcpu->arch.regs.nip = val; } static inline ulong kvmppc_get_pc(struct kvm_vcpu *vcpu) { - return vcpu->arch.pc; + return vcpu->arch.regs.nip; } static inline ulong kvmppc_get_fault_dar(struct kvm_vcpu *vcpu) diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h index 17498e9a26e4..fa4efa7e88f7 100644 --- a/arch/powerpc/include/asm/kvm_host.h +++ b/arch/powerpc/include/asm/kvm_host.h @@ -269,7 +269,6 @@ struct kvm_arch { unsigned long host_lpcr; unsigned long sdr1; unsigned long host_sdr1; - int tlbie_lock; unsigned long lpcr; unsigned long vrma_slb_v; int mmu_ready; @@ -454,6 +453,12 @@ struct mmio_hpte_cache { #define KVMPPC_VSX_COPY_WORD 1 #define KVMPPC_VSX_COPY_DWORD 2 #define KVMPPC_VSX_COPY_DWORD_LOAD_DUMP 3 +#define KVMPPC_VSX_COPY_WORD_LOAD_DUMP 4 + +#define KVMPPC_VMX_COPY_BYTE 8 +#define KVMPPC_VMX_COPY_HWORD 9 +#define KVMPPC_VMX_COPY_WORD 10 +#define KVMPPC_VMX_COPY_DWORD 11 struct openpic; @@ -486,7 +491,7 @@ struct kvm_vcpu_arch { struct kvmppc_book3s_shadow_vcpu *shadow_vcpu; #endif - ulong gpr[32]; + struct pt_regs regs; struct thread_fp_state fp; @@ -521,14 +526,10 @@ struct kvm_vcpu_arch { u32 qpr[32]; #endif - ulong pc; - ulong ctr; - ulong lr; #ifdef CONFIG_PPC_BOOK3S ulong tar; #endif - ulong xer; u32 cr; #ifdef CONFIG_PPC_BOOK3S @@ -626,7 +627,6 @@ struct kvm_vcpu_arch { struct thread_vr_state vr_tm; u32 vrsave_tm; /* also USPRG0 */ - #endif #ifdef CONFIG_KVM_EXIT_TIMING @@ -681,16 +681,17 @@ struct kvm_vcpu_arch { * Number of simulations for vsx. * If we use 2*8bytes to simulate 1*16bytes, * then the number should be 2 and - * mmio_vsx_copy_type=KVMPPC_VSX_COPY_DWORD. + * mmio_copy_type=KVMPPC_VSX_COPY_DWORD. * If we use 4*4bytes to simulate 1*16bytes, * the number should be 4 and * mmio_vsx_copy_type=KVMPPC_VSX_COPY_WORD. */ u8 mmio_vsx_copy_nums; u8 mmio_vsx_offset; - u8 mmio_vsx_copy_type; u8 mmio_vsx_tx_sx_enabled; u8 mmio_vmx_copy_nums; + u8 mmio_vmx_offset; + u8 mmio_copy_type; u8 osi_needed; u8 osi_enabled; u8 papr_enabled; @@ -772,6 +773,8 @@ struct kvm_vcpu_arch { u64 busy_preempt; u32 emul_inst; + + u32 online; #endif #ifdef CONFIG_KVM_BOOK3S_HV_EXIT_TIMING diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h index abe7032cdb54..e991821dd7fa 100644 --- a/arch/powerpc/include/asm/kvm_ppc.h +++ b/arch/powerpc/include/asm/kvm_ppc.h @@ -52,7 +52,7 @@ enum emulation_result { EMULATE_EXIT_USER, /* emulation requires exit to user-space */ }; -enum instruction_type { +enum instruction_fetch_type { INST_GENERIC, INST_SC, /* system call */ }; @@ -81,10 +81,10 @@ extern int kvmppc_handle_loads(struct kvm_run *run, struct kvm_vcpu *vcpu, extern int kvmppc_handle_vsx_load(struct kvm_run *run, struct kvm_vcpu *vcpu, unsigned int rt, unsigned int bytes, int is_default_endian, int mmio_sign_extend); -extern int kvmppc_handle_load128_by2x64(struct kvm_run *run, - struct kvm_vcpu *vcpu, unsigned int rt, int is_default_endian); -extern int kvmppc_handle_store128_by2x64(struct kvm_run *run, - struct kvm_vcpu *vcpu, unsigned int rs, int is_default_endian); +extern int kvmppc_handle_vmx_load(struct kvm_run *run, struct kvm_vcpu *vcpu, + unsigned int rt, unsigned int bytes, int is_default_endian); +extern int kvmppc_handle_vmx_store(struct kvm_run *run, struct kvm_vcpu *vcpu, + unsigned int rs, unsigned int bytes, int is_default_endian); extern int kvmppc_handle_store(struct kvm_run *run, struct kvm_vcpu *vcpu, u64 val, unsigned int bytes, int is_default_endian); @@ -93,7 +93,7 @@ extern int kvmppc_handle_vsx_store(struct kvm_run *run, struct kvm_vcpu *vcpu, int is_default_endian); extern int kvmppc_load_last_inst(struct kvm_vcpu *vcpu, - enum instruction_type type, u32 *inst); + enum instruction_fetch_type type, u32 *inst); extern int kvmppc_ld(struct kvm_vcpu *vcpu, ulong *eaddr, int size, void *ptr, bool data); @@ -265,6 +265,8 @@ union kvmppc_one_reg { vector128 vval; u64 vsxval[2]; u32 vsx32val[4]; + u16 vsx16val[8]; + u8 vsx8val[16]; struct { u64 addr; u64 length; @@ -324,13 +326,14 @@ struct kvmppc_ops { int (*get_rmmu_info)(struct kvm *kvm, struct kvm_ppc_rmmu_info *info); int (*set_smt_mode)(struct kvm *kvm, unsigned long mode, unsigned long flags); + void (*giveup_ext)(struct kvm_vcpu *vcpu, ulong msr); }; extern struct kvmppc_ops *kvmppc_hv_ops; extern struct kvmppc_ops *kvmppc_pr_ops; static inline int kvmppc_get_last_inst(struct kvm_vcpu *vcpu, - enum instruction_type type, u32 *inst) + enum instruction_fetch_type type, u32 *inst) { int ret = EMULATE_DONE; u32 fetched_inst; diff --git a/arch/powerpc/include/asm/machdep.h b/arch/powerpc/include/asm/machdep.h index ffe7c71e1132..a47de82fb8e2 100644 --- a/arch/powerpc/include/asm/machdep.h +++ b/arch/powerpc/include/asm/machdep.h @@ -83,7 +83,7 @@ struct machdep_calls { int (*set_rtc_time)(struct rtc_time *); void (*get_rtc_time)(struct rtc_time *); - unsigned long (*get_boot_time)(void); + time64_t (*get_boot_time)(void); unsigned char (*rtc_read_val)(int addr); void (*rtc_write_val)(int addr, unsigned char val); diff --git a/arch/powerpc/include/asm/mmu-book3e.h b/arch/powerpc/include/asm/mmu-book3e.h index cda94a0f5146..e20072972e35 100644 --- a/arch/powerpc/include/asm/mmu-book3e.h +++ b/arch/powerpc/include/asm/mmu-book3e.h @@ -230,10 +230,6 @@ typedef struct { unsigned int id; unsigned int active; unsigned long vdso_base; -#ifdef CONFIG_PPC_64K_PAGES - /* for 4K PTE fragment support */ - void *pte_frag; -#endif } mm_context_t; /* Page size definitions, common between 32 and 64-bit @@ -275,8 +271,6 @@ static inline unsigned int mmu_psize_to_shift(unsigned int mmu_psize) */ #if defined(CONFIG_PPC_4K_PAGES) #define mmu_virtual_psize MMU_PAGE_4K -#elif defined(CONFIG_PPC_64K_PAGES) -#define mmu_virtual_psize MMU_PAGE_64K #else #error Unsupported page size #endif diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h index 1835ca1505d6..896efa559996 100644 --- a/arch/powerpc/include/asm/mmu_context.h +++ b/arch/powerpc/include/asm/mmu_context.h @@ -250,11 +250,6 @@ static inline bool arch_vma_access_permitted(struct vm_area_struct *vma, #define thread_pkey_regs_restore(new_thread, old_thread) #define thread_pkey_regs_init(thread) -static inline int vma_pkey(struct vm_area_struct *vma) -{ - return 0; -} - static inline u64 pte_to_hpte_pkey_bits(u64 pteflags) { return 0x0UL; diff --git a/arch/powerpc/include/asm/module.h b/arch/powerpc/include/asm/module.h index 4f6573934792..d61b0818e267 100644 --- a/arch/powerpc/include/asm/module.h +++ b/arch/powerpc/include/asm/module.h @@ -14,7 +14,7 @@ #include -#ifdef CC_USING_MPROFILE_KERNEL +#ifdef CONFIG_MPROFILE_KERNEL #define MODULE_ARCH_VERMAGIC_FTRACE "mprofile-kernel " #else #define MODULE_ARCH_VERMAGIC_FTRACE "" @@ -50,10 +50,6 @@ struct mod_arch_specific { unsigned int stubs_section; /* Index of stubs section in module */ unsigned int toc_section; /* What section is the TOC? */ bool toc_fixed; /* Have we fixed up .TOC.? */ -#ifdef CONFIG_DYNAMIC_FTRACE - unsigned long toc; - unsigned long tramp; -#endif /* For module function descriptor dereference */ unsigned long start_opd; @@ -62,10 +58,14 @@ struct mod_arch_specific { /* Indices of PLT sections within module. */ unsigned int core_plt_section; unsigned int init_plt_section; +#endif /* powerpc64 */ + #ifdef CONFIG_DYNAMIC_FTRACE unsigned long tramp; +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS + unsigned long tramp_regs; +#endif #endif -#endif /* powerpc64 */ /* List of BUG addresses, source line numbers and filenames */ struct list_head bug_list; diff --git a/arch/powerpc/include/asm/mpc52xx.h b/arch/powerpc/include/asm/mpc52xx.h index e94cede14522..ce1e0aabaa64 100644 --- a/arch/powerpc/include/asm/mpc52xx.h +++ b/arch/powerpc/include/asm/mpc52xx.h @@ -350,14 +350,14 @@ extern struct mpc52xx_suspend mpc52xx_suspend; extern int __init mpc52xx_pm_init(void); extern int mpc52xx_set_wakeup_gpio(u8 pin, u8 level); -#ifdef CONFIG_PPC_LITE5200 -extern int __init lite5200_pm_init(void); - /* lite5200 calls mpc5200 suspend functions, so here they are */ extern int mpc52xx_pm_prepare(void); extern int mpc52xx_pm_enter(suspend_state_t); extern void mpc52xx_pm_finish(void); extern char saved_sram[0x4000]; /* reuse buffer from mpc52xx suspend */ + +#ifdef CONFIG_PPC_LITE5200 +int __init lite5200_pm_init(void); #endif #endif /* CONFIG_PM */ diff --git a/arch/powerpc/include/asm/nmi.h b/arch/powerpc/include/asm/nmi.h index 9c80939b4d14..0f571e0ebca1 100644 --- a/arch/powerpc/include/asm/nmi.h +++ b/arch/powerpc/include/asm/nmi.h @@ -8,4 +8,10 @@ extern void arch_touch_nmi_watchdog(void); static inline void arch_touch_nmi_watchdog(void) {} #endif +#if defined(CONFIG_PPC_BOOK3S_64) && defined(CONFIG_STACKTRACE) +extern void arch_trigger_cpumask_backtrace(const cpumask_t *mask, + bool exclude_self); +#define arch_trigger_cpumask_backtrace arch_trigger_cpumask_backtrace +#endif + #endif /* _ASM_NMI_H */ diff --git a/arch/powerpc/include/asm/nohash/32/pgalloc.h b/arch/powerpc/include/asm/nohash/32/pgalloc.h index 29d37bd1f3b3..1707781d2f20 100644 --- a/arch/powerpc/include/asm/nohash/32/pgalloc.h +++ b/arch/powerpc/include/asm/nohash/32/pgalloc.h @@ -100,6 +100,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage) static inline void pgtable_free(void *table, unsigned index_size) { if (!index_size) { + pgtable_page_dtor(virt_to_page(table)); free_page((unsigned long)table); } else { BUG_ON(index_size > MAX_PGTABLE_INDEX_SIZE); diff --git a/arch/powerpc/include/asm/nohash/32/pgtable.h b/arch/powerpc/include/asm/nohash/32/pgtable.h index 03bbd1149530..7c46a98cc7f4 100644 --- a/arch/powerpc/include/asm/nohash/32/pgtable.h +++ b/arch/powerpc/include/asm/nohash/32/pgtable.h @@ -133,7 +133,7 @@ extern int icache_44x_need_flush; #ifndef __ASSEMBLY__ #define pte_clear(mm, addr, ptep) \ - do { pte_update(ptep, ~_PAGE_HASHPTE, 0); } while (0) + do { pte_update(ptep, ~0, 0); } while (0) #define pmd_none(pmd) (!pmd_val(pmd)) #define pmd_bad(pmd) (pmd_val(pmd) & _PMD_BAD) @@ -145,21 +145,6 @@ static inline void pmd_clear(pmd_t *pmdp) -/* - * When flushing the tlb entry for a page, we also need to flush the hash - * table entry. flush_hash_pages is assembler (for speed) in hashtable.S. - */ -extern int flush_hash_pages(unsigned context, unsigned long va, - unsigned long pmdval, int count); - -/* Add an HPTE to the hash table */ -extern void add_hash_page(unsigned context, unsigned long va, - unsigned long pmdval); - -/* Flush an entry from the TLB/hash table */ -extern void flush_hash_entry(struct mm_struct *mm, pte_t *ptep, - unsigned long address); - /* * PTE updates. This function is called whenever an existing * valid PTE is updated. This does -not- include set_pte_at() @@ -246,12 +231,6 @@ static inline int __ptep_test_and_clear_young(unsigned int context, unsigned lon { unsigned long old; old = pte_update(ptep, _PAGE_ACCESSED, 0); -#if _PAGE_HASHPTE != 0 - if (old & _PAGE_HASHPTE) { - unsigned long ptephys = __pa(ptep) & PAGE_MASK; - flush_hash_pages(context, addr, ptephys, 1); - } -#endif return (old & _PAGE_ACCESSED) != 0; } #define ptep_test_and_clear_young(__vma, __addr, __ptep) \ @@ -261,7 +240,7 @@ static inline int __ptep_test_and_clear_young(unsigned int context, unsigned lon static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - return __pte(pte_update(ptep, ~_PAGE_HASHPTE, 0)); + return __pte(pte_update(ptep, ~0, 0)); } #define __HAVE_ARCH_PTEP_SET_WRPROTECT @@ -277,19 +256,27 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm, } -static inline void __ptep_set_access_flags(struct mm_struct *mm, +static inline void __ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, pte_t entry, - unsigned long address) + unsigned long address, + int psize) { unsigned long set = pte_val(entry) & (_PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_RW | _PAGE_EXEC); unsigned long clr = ~pte_val(entry) & (_PAGE_RO | _PAGE_NA); pte_update(ptep, clr, set); + + flush_tlb_page(vma, address); +} + +static inline int pte_young(pte_t pte) +{ + return pte_val(pte) & _PAGE_ACCESSED; } #define __HAVE_ARCH_PTE_SAME -#define pte_same(A,B) (((pte_val(A) ^ pte_val(B)) & ~_PAGE_HASHPTE) == 0) +#define pte_same(A,B) ((pte_val(A) ^ pte_val(B)) == 0) /* * Note that on Book E processors, the pmd contains the kernel virtual @@ -330,7 +317,7 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm, /* * Encode and decode a swap entry. * Note that the bits we use in a PTE for representing a swap entry - * must not include the _PAGE_PRESENT bit or the _PAGE_HASHPTE bit (if used). + * must not include the _PAGE_PRESENT bit. * -- paulus */ #define __swp_type(entry) ((entry).val & 0x1f) diff --git a/arch/powerpc/include/asm/nohash/32/pte-40x.h b/arch/powerpc/include/asm/nohash/32/pte-40x.h index 124f9ac23a1e..bb4b3a4b92a0 100644 --- a/arch/powerpc/include/asm/nohash/32/pte-40x.h +++ b/arch/powerpc/include/asm/nohash/32/pte-40x.h @@ -52,12 +52,9 @@ #define _PMD_PRESENT 0x400 /* PMD points to page of PTEs */ #define _PMD_BAD 0x802 -#define _PMD_SIZE 0x0e0 /* size field, != 0 for large-page PMD entry */ #define _PMD_SIZE_4M 0x0c0 #define _PMD_SIZE_16M 0x0e0 -#define PMD_PAGE_SIZE(pmdval) (1024 << (((pmdval) & _PMD_SIZE) >> 4)) - /* Until my rework is finished, 40x still needs atomic PTE updates */ #define PTE_ATOMIC_UPDATES 1 diff --git a/arch/powerpc/include/asm/nohash/64/pgalloc.h b/arch/powerpc/include/asm/nohash/64/pgalloc.h index 9721c7867b9c..0e693f322cb2 100644 --- a/arch/powerpc/include/asm/nohash/64/pgalloc.h +++ b/arch/powerpc/include/asm/nohash/64/pgalloc.h @@ -52,8 +52,6 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) kmem_cache_free(PGT_CACHE(PGD_INDEX_SIZE), pgd); } -#ifndef CONFIG_PPC_64K_PAGES - #define pgd_populate(MM, PGD, PUD) pgd_set(PGD, (unsigned long)PUD) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) @@ -86,6 +84,18 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, #define pmd_pgtable(pmd) pmd_page(pmd) +static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) +{ + return kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX), + pgtable_gfp_flags(mm, GFP_KERNEL)); +} + +static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) +{ + kmem_cache_free(PGT_CACHE(PMD_CACHE_INDEX), pmd); +} + + static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) { @@ -120,84 +130,47 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage) __free_page(ptepage); } -extern void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift); -#ifdef CONFIG_SMP -extern void __tlb_remove_table(void *_table); -#endif -static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table, - unsigned long address) +static inline void pgtable_free(void *table, int shift) { - tlb_flush_pgtable(tlb, address); - pgtable_free_tlb(tlb, page_address(table), 0); + if (!shift) { + pgtable_page_dtor(virt_to_page(table)); + free_page((unsigned long)table); + } else { + BUG_ON(shift > MAX_PGTABLE_INDEX_SIZE); + kmem_cache_free(PGT_CACHE(shift), table); + } } -#else /* if CONFIG_PPC_64K_PAGES */ - -extern pte_t *pte_fragment_alloc(struct mm_struct *, unsigned long, int); -extern void pte_fragment_free(unsigned long *, int); -extern void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift); #ifdef CONFIG_SMP -extern void __tlb_remove_table(void *_table); -#endif - -#define pud_populate(mm, pud, pmd) pud_set(pud, (unsigned long)pmd) - -static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, - pte_t *pte) +static inline void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift) { - pmd_set(pmd, (unsigned long)pte); -} + unsigned long pgf = (unsigned long)table; -static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, - pgtable_t pte_page) -{ - pmd_set(pmd, (unsigned long)pte_page); + BUG_ON(shift > MAX_PGTABLE_INDEX_SIZE); + pgf |= shift; + tlb_remove_table(tlb, (void *)pgf); } -static inline pgtable_t pmd_pgtable(pmd_t pmd) +static inline void __tlb_remove_table(void *_table) { - return (pgtable_t)(pmd_val(pmd) & ~PMD_MASKED_BITS); -} + void *table = (void *)((unsigned long)_table & ~MAX_PGTABLE_INDEX_SIZE); + unsigned shift = (unsigned long)_table & MAX_PGTABLE_INDEX_SIZE; -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) -{ - return (pte_t *)pte_fragment_alloc(mm, address, 1); + pgtable_free(table, shift); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) -{ - return (pgtable_t)pte_fragment_alloc(mm, address, 0); -} - -static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) -{ - pte_fragment_free((unsigned long *)pte, 1); -} - -static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage) +#else +static inline void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift) { - pte_fragment_free((unsigned long *)ptepage, 0); + pgtable_free(table, shift); } +#endif static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table, unsigned long address) { tlb_flush_pgtable(tlb, address); - pgtable_free_tlb(tlb, table, 0); -} -#endif /* CONFIG_PPC_64K_PAGES */ - -static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - return kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX), - pgtable_gfp_flags(mm, GFP_KERNEL)); -} - -static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) -{ - kmem_cache_free(PGT_CACHE(PMD_CACHE_INDEX), pmd); + pgtable_free_tlb(tlb, page_address(table), 0); } #define __pmd_free_tlb(tlb, pmd, addr) \ diff --git a/arch/powerpc/include/asm/nohash/64/pgtable-64k.h b/arch/powerpc/include/asm/nohash/64/pgtable-64k.h deleted file mode 100644 index 7210c2818e41..000000000000 --- a/arch/powerpc/include/asm/nohash/64/pgtable-64k.h +++ /dev/null @@ -1,57 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_POWERPC_NOHASH_64_PGTABLE_64K_H -#define _ASM_POWERPC_NOHASH_64_PGTABLE_64K_H - -#define __ARCH_USE_5LEVEL_HACK -#include - - -#define PTE_INDEX_SIZE 8 -#define PMD_INDEX_SIZE 10 -#define PUD_INDEX_SIZE 0 -#define PGD_INDEX_SIZE 12 - -/* - * we support 32 fragments per PTE page of 64K size - */ -#define PTE_FRAG_NR 32 -/* - * We use a 2K PTE page fragment and another 2K for storing - * real_pte_t hash index - */ -#define PTE_FRAG_SIZE_SHIFT 11 -#define PTE_FRAG_SIZE (1UL << PTE_FRAG_SIZE_SHIFT) - -#ifndef __ASSEMBLY__ -#define PTE_TABLE_SIZE PTE_FRAG_SIZE -#define PMD_TABLE_SIZE (sizeof(pmd_t) << PMD_INDEX_SIZE) -#define PUD_TABLE_SIZE (0) -#define PGD_TABLE_SIZE (sizeof(pgd_t) << PGD_INDEX_SIZE) -#endif /* __ASSEMBLY__ */ - -#define PTRS_PER_PTE (1 << PTE_INDEX_SIZE) -#define PTRS_PER_PMD (1 << PMD_INDEX_SIZE) -#define PTRS_PER_PGD (1 << PGD_INDEX_SIZE) - -/* PMD_SHIFT determines what a second-level page table entry can map */ -#define PMD_SHIFT (PAGE_SHIFT + PTE_INDEX_SIZE) -#define PMD_SIZE (1UL << PMD_SHIFT) -#define PMD_MASK (~(PMD_SIZE-1)) - -/* PGDIR_SHIFT determines what a third-level page table entry can map */ -#define PGDIR_SHIFT (PMD_SHIFT + PMD_INDEX_SIZE) -#define PGDIR_SIZE (1UL << PGDIR_SHIFT) -#define PGDIR_MASK (~(PGDIR_SIZE-1)) - -/* - * Bits to mask out from a PMD to get to the PTE page - * PMDs point to PTE table fragments which are PTE_FRAG_SIZE aligned. - */ -#define PMD_MASKED_BITS (PTE_FRAG_SIZE - 1) -/* Bits to mask out from a PGD/PUD to get to the PMD page */ -#define PUD_MASKED_BITS 0x1ff - -#define pgd_pte(pgd) (pud_pte(((pud_t){ pgd }))) -#define pte_pgd(pte) ((pgd_t)pte_pud(pte)) - -#endif /* _ASM_POWERPC_NOHASH_64_PGTABLE_64K_H */ diff --git a/arch/powerpc/include/asm/nohash/64/pgtable.h b/arch/powerpc/include/asm/nohash/64/pgtable.h index 5c5f75d005ad..dd0c7236208f 100644 --- a/arch/powerpc/include/asm/nohash/64/pgtable.h +++ b/arch/powerpc/include/asm/nohash/64/pgtable.h @@ -6,13 +6,13 @@ * the ppc64 hashed page table. */ -#ifdef CONFIG_PPC_64K_PAGES -#include -#else #include -#endif #include +#ifdef CONFIG_PPC_64K_PAGES +#error "Page size not supported" +#endif + #define FIRST_USER_ADDRESS 0UL /* @@ -173,8 +173,6 @@ static inline void pgd_set(pgd_t *pgdp, unsigned long val) /* to find an entry in a kernel page-table-directory */ /* This now only contains the vmalloc pages */ #define pgd_offset_k(address) pgd_offset(&init_mm, address) -extern void hpte_need_flush(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned long pte, int huge); /* Atomic PTE updates */ static inline unsigned long pte_update(struct mm_struct *mm, @@ -188,14 +186,12 @@ static inline unsigned long pte_update(struct mm_struct *mm, __asm__ __volatile__( "1: ldarx %0,0,%3 # pte_update\n\ - andi. %1,%0,%6\n\ - bne- 1b \n\ andc %1,%0,%4 \n\ - or %1,%1,%7\n\ + or %1,%1,%6\n\ stdcx. %1,0,%3 \n\ bne- 1b" : "=&r" (old), "=&r" (tmp), "=m" (*ptep) - : "r" (ptep), "r" (clr), "m" (*ptep), "i" (_PAGE_BUSY), "r" (set) + : "r" (ptep), "r" (clr), "m" (*ptep), "r" (set) : "cc" ); #else unsigned long old = pte_val(*ptep); @@ -205,20 +201,20 @@ static inline unsigned long pte_update(struct mm_struct *mm, if (!huge) assert_pte_locked(mm, addr); -#ifdef CONFIG_PPC_BOOK3S_64 - if (old & _PAGE_HASHPTE) - hpte_need_flush(mm, addr, ptep, old, huge); -#endif - return old; } +static inline int pte_young(pte_t pte) +{ + return pte_val(pte) & _PAGE_ACCESSED; +} + static inline int __ptep_test_and_clear_young(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { unsigned long old; - if ((pte_val(*ptep) & (_PAGE_ACCESSED | _PAGE_HASHPTE)) == 0) + if (pte_young(*ptep)) return 0; old = pte_update(mm, addr, ptep, _PAGE_ACCESSED, 0, 0); return (old & _PAGE_ACCESSED) != 0; @@ -285,9 +281,10 @@ static inline void pte_clear(struct mm_struct *mm, unsigned long addr, /* Set the dirty and/or accessed bits atomically in a linux PTE, this * function doesn't need to flush the hash entry */ -static inline void __ptep_set_access_flags(struct mm_struct *mm, +static inline void __ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, pte_t entry, - unsigned long address) + unsigned long address, + int psize) { unsigned long bits = pte_val(entry) & (_PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_RW | _PAGE_EXEC); @@ -297,22 +294,22 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm, __asm__ __volatile__( "1: ldarx %0,0,%4\n\ - andi. %1,%0,%6\n\ - bne- 1b \n\ or %0,%3,%0\n\ stdcx. %0,0,%4\n\ bne- 1b" :"=&r" (old), "=&r" (tmp), "=m" (*ptep) - :"r" (bits), "r" (ptep), "m" (*ptep), "i" (_PAGE_BUSY) + :"r" (bits), "r" (ptep), "m" (*ptep) :"cc"); #else unsigned long old = pte_val(*ptep); *ptep = __pte(old | bits); #endif + + flush_tlb_page(vma, address); } #define __HAVE_ARCH_PTE_SAME -#define pte_same(A,B) (((pte_val(A) ^ pte_val(B)) & ~_PAGE_HPTEFLAGS) == 0) +#define pte_same(A,B) ((pte_val(A) ^ pte_val(B)) == 0) #define pte_ERROR(e) \ pr_err("%s:%d: bad pte %08lx.\n", __FILE__, __LINE__, pte_val(e)) @@ -324,11 +321,6 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm, /* Encode and de-code a swap entry */ #define MAX_SWAPFILES_CHECK() do { \ BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > SWP_TYPE_BITS); \ - /* \ - * Don't have overlapping bits with _PAGE_HPTEFLAGS \ - * We filter HPTEFLAGS on set_pte. \ - */ \ - BUILD_BUG_ON(_PAGE_HPTEFLAGS & (0x1f << _PAGE_BIT_SWAP_TYPE)); \ } while (0) /* * on pte we don't need handle RADIX_TREE_EXCEPTIONAL_SHIFT; diff --git a/arch/powerpc/include/asm/nohash/pgtable.h b/arch/powerpc/include/asm/nohash/pgtable.h index c56de1e8026f..2160be2e4339 100644 --- a/arch/powerpc/include/asm/nohash/pgtable.h +++ b/arch/powerpc/include/asm/nohash/pgtable.h @@ -17,7 +17,6 @@ static inline int pte_write(pte_t pte) } static inline int pte_read(pte_t pte) { return 1; } static inline int pte_dirty(pte_t pte) { return pte_val(pte) & _PAGE_DIRTY; } -static inline int pte_young(pte_t pte) { return pte_val(pte) & _PAGE_ACCESSED; } static inline int pte_special(pte_t pte) { return pte_val(pte) & _PAGE_SPECIAL; } static inline int pte_none(pte_t pte) { return (pte_val(pte) & ~_PTE_NONE_MASK) == 0; } static inline pgprot_t pte_pgprot(pte_t pte) { return __pgprot(pte_val(pte) & PAGE_PROT_BITS); } @@ -148,70 +147,33 @@ extern void set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, int percpu) { -#if defined(CONFIG_PPC_STD_MMU_32) && defined(CONFIG_SMP) && !defined(CONFIG_PTE_64BIT) - /* First case is 32-bit Hash MMU in SMP mode with 32-bit PTEs. We use the - * helper pte_update() which does an atomic update. We need to do that - * because a concurrent invalidation can clear _PAGE_HASHPTE. If it's a - * per-CPU PTE such as a kmap_atomic, we do a simple update preserving - * the hash bits instead (ie, same as the non-SMP case) - */ - if (percpu) - *ptep = __pte((pte_val(*ptep) & _PAGE_HASHPTE) - | (pte_val(pte) & ~_PAGE_HASHPTE)); - else - pte_update(ptep, ~_PAGE_HASHPTE, pte_val(pte)); - -#elif defined(CONFIG_PPC32) && defined(CONFIG_PTE_64BIT) /* Second case is 32-bit with 64-bit PTE. In this case, we * can just store as long as we do the two halves in the right order - * with a barrier in between. This is possible because we take care, - * in the hash code, to pre-invalidate if the PTE was already hashed, - * which synchronizes us with any concurrent invalidation. - * In the percpu case, we also fallback to the simple update preserving - * the hash bits + * with a barrier in between. + * In the percpu case, we also fallback to the simple update */ - if (percpu) { - *ptep = __pte((pte_val(*ptep) & _PAGE_HASHPTE) - | (pte_val(pte) & ~_PAGE_HASHPTE)); + if (IS_ENABLED(CONFIG_PPC32) && IS_ENABLED(CONFIG_PTE_64BIT) && !percpu) { + __asm__ __volatile__("\ + stw%U0%X0 %2,%0\n\ + eieio\n\ + stw%U0%X0 %L2,%1" + : "=m" (*ptep), "=m" (*((unsigned char *)ptep+4)) + : "r" (pte) : "memory"); return; } -#if _PAGE_HASHPTE != 0 - if (pte_val(*ptep) & _PAGE_HASHPTE) - flush_hash_entry(mm, ptep, addr); -#endif - __asm__ __volatile__("\ - stw%U0%X0 %2,%0\n\ - eieio\n\ - stw%U0%X0 %L2,%1" - : "=m" (*ptep), "=m" (*((unsigned char *)ptep+4)) - : "r" (pte) : "memory"); - -#elif defined(CONFIG_PPC_STD_MMU_32) - /* Third case is 32-bit hash table in UP mode, we need to preserve - * the _PAGE_HASHPTE bit since we may not have invalidated the previous - * translation in the hash yet (done in a subsequent flush_tlb_xxx()) - * and see we need to keep track that this PTE needs invalidating - */ - *ptep = __pte((pte_val(*ptep) & _PAGE_HASHPTE) - | (pte_val(pte) & ~_PAGE_HASHPTE)); - -#else /* Anything else just stores the PTE normally. That covers all 64-bit * cases, and 32-bit non-hash with 32-bit PTEs. */ *ptep = pte; -#ifdef CONFIG_PPC_BOOK3E_64 /* * With hardware tablewalk, a sync is needed to ensure that * subsequent accesses see the PTE we just wrote. Unlike userspace * mappings, we can't tolerate spurious faults, so make sure * the new PTE will be seen the first time. */ - if (is_kernel_addr(addr)) + if (IS_ENABLED(CONFIG_PPC_BOOK3E_64) && is_kernel_addr(addr)) mb(); -#endif -#endif } diff --git a/arch/powerpc/include/asm/nohash/pte-book3e.h b/arch/powerpc/include/asm/nohash/pte-book3e.h index ccee8eb509bb..12730b81cd98 100644 --- a/arch/powerpc/include/asm/nohash/pte-book3e.h +++ b/arch/powerpc/include/asm/nohash/pte-book3e.h @@ -57,14 +57,8 @@ #define _PAGE_USER (_PAGE_BAP_UR | _PAGE_BAP_SR) /* Can be read */ #define _PAGE_PRIVILEGED (_PAGE_BAP_SR) -#define _PAGE_HASHPTE 0 -#define _PAGE_BUSY 0 - #define _PAGE_SPECIAL _PAGE_SW0 -/* Flags to be preserved on PTE modifications */ -#define _PAGE_HPTEFLAGS _PAGE_BUSY - /* Base page size */ #ifdef CONFIG_PPC_64K_PAGES #define _PAGE_PSIZE _PAGE_PSIZE_64K diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h index d886a5b7ff21..3bab299eda49 100644 --- a/arch/powerpc/include/asm/opal-api.h +++ b/arch/powerpc/include/asm/opal-api.h @@ -201,13 +201,21 @@ #define OPAL_SET_POWER_SHIFT_RATIO 155 #define OPAL_SENSOR_GROUP_CLEAR 156 #define OPAL_PCI_SET_P2P 157 +#define OPAL_QUIESCE 158 #define OPAL_NPU_SPA_SETUP 159 #define OPAL_NPU_SPA_CLEAR_CACHE 160 #define OPAL_NPU_TL_SET 161 +#define OPAL_SENSOR_READ_U64 162 #define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164 #define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165 #define OPAL_LAST 165 +#define QUIESCE_HOLD 1 /* Spin all calls at entry */ +#define QUIESCE_REJECT 2 /* Fail all calls with OPAL_BUSY */ +#define QUIESCE_LOCK_BREAK 3 /* Set to ignore locks. */ +#define QUIESCE_RESUME 4 /* Un-quiesce */ +#define QUIESCE_RESUME_FAST_REBOOT 5 /* Un-quiesce, fast reboot */ + /* Device tree flags */ /* diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h index 03e1a920491e..e1b2910c6e81 100644 --- a/arch/powerpc/include/asm/opal.h +++ b/arch/powerpc/include/asm/opal.h @@ -201,6 +201,7 @@ int64_t opal_get_param(uint64_t token, uint32_t param_id, uint64_t buffer, int64_t opal_set_param(uint64_t token, uint32_t param_id, uint64_t buffer, uint64_t length); int64_t opal_sensor_read(uint32_t sensor_hndl, int token, __be32 *sensor_data); +int64_t opal_sensor_read_u64(u32 sensor_hndl, int token, __be64 *sensor_data); int64_t opal_handle_hmi(void); int64_t opal_register_dump_region(uint32_t id, uint64_t start, uint64_t end); int64_t opal_unregister_dump_region(uint32_t id); @@ -293,6 +294,7 @@ int opal_set_power_shift_ratio(u32 handle, int token, u32 psr); int opal_sensor_group_clear(u32 group_hndl, int token); s64 opal_signal_system_reset(s32 cpu); +s64 opal_quiesce(u64 shutdown_type, s32 cpu); /* Internal functions */ extern int early_init_dt_scan_opal(unsigned long node, const char *uname, @@ -323,9 +325,10 @@ extern int opal_async_wait_response(uint64_t token, struct opal_msg *msg); extern int opal_async_wait_response_interruptible(uint64_t token, struct opal_msg *msg); extern int opal_get_sensor_data(u32 sensor_hndl, u32 *sensor_data); +extern int opal_get_sensor_data_u64(u32 sensor_hndl, u64 *sensor_data); struct rtc_time; -extern unsigned long opal_get_boot_time(void); +extern time64_t opal_get_boot_time(void); extern void opal_nvram_init(void); extern void opal_flash_update_init(void); extern void opal_flash_update_print_message(void); diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h index 3f109a3e3edb..6d34bd71139d 100644 --- a/arch/powerpc/include/asm/paca.h +++ b/arch/powerpc/include/asm/paca.h @@ -161,7 +161,7 @@ struct paca_struct { struct task_struct *__current; /* Pointer to current */ u64 kstack; /* Saved Kernel stack addr */ u64 stab_rr; /* stab/slb round-robin counter */ - u64 saved_r1; /* r1 save for RTAS calls or PM */ + u64 saved_r1; /* r1 save for RTAS calls or PM or EE=0 */ u64 saved_msr; /* MSR saved here by enter_rtas */ u16 trap_save; /* Used when bad stack is encountered */ u8 irq_soft_mask; /* mask for irq soft masking */ @@ -222,6 +222,7 @@ struct paca_struct { u8 hmi_event_available; /* HMI event is available */ u8 hmi_p9_special_emu; /* HMI P9 special emulation */ #endif + u8 ftrace_enabled; /* Hard disable ftrace */ /* Stuff for accurate time accounting */ struct cpu_accounting_data accounting; diff --git a/arch/powerpc/include/asm/page.h b/arch/powerpc/include/asm/page.h index dec9ce5ba8af..db7be0779d55 100644 --- a/arch/powerpc/include/asm/page.h +++ b/arch/powerpc/include/asm/page.h @@ -39,6 +39,7 @@ #ifndef __ASSEMBLY__ #ifdef CONFIG_HUGETLB_PAGE +extern bool hugetlb_disabled; extern unsigned int HPAGE_SHIFT; #else #define HPAGE_SHIFT PAGE_SHIFT diff --git a/arch/powerpc/include/asm/pci.h b/arch/powerpc/include/asm/pci.h index 401c62aad5e4..2af9ded80540 100644 --- a/arch/powerpc/include/asm/pci.h +++ b/arch/powerpc/include/asm/pci.h @@ -92,24 +92,6 @@ extern int pci_mmap_legacy_page_range(struct pci_bus *bus, #define HAVE_PCI_LEGACY 1 -#ifdef CONFIG_PPC64 - -/* The PCI address space does not equal the physical memory address - * space (we have an IOMMU). The IDE and SCSI device layers use - * this boolean for bounce buffer decisions. - */ -#define PCI_DMA_BUS_IS_PHYS (0) - -#else /* 32-bit */ - -/* The PCI address space does equal the physical memory - * address space (no IOMMU). The IDE and SCSI device layers use - * this boolean for bounce buffer decisions. - */ -#define PCI_DMA_BUS_IS_PHYS (1) - -#endif /* CONFIG_PPC64 */ - extern void pcibios_claim_one_bus(struct pci_bus *b); extern void pcibios_finish_adding_to_bus(struct pci_bus *bus); diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h index ab7d2d996be4..14c79a7dc855 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -8,6 +8,7 @@ #include /* For TASK_SIZE */ #include #include +#include struct mm_struct; diff --git a/arch/powerpc/include/asm/pkeys.h b/arch/powerpc/include/asm/pkeys.h index 0409c80c32c0..5ba80cffb505 100644 --- a/arch/powerpc/include/asm/pkeys.h +++ b/arch/powerpc/include/asm/pkeys.h @@ -15,19 +15,6 @@ DECLARE_STATIC_KEY_TRUE(pkey_disabled); extern int pkeys_total; /* total pkeys as per device tree */ extern u32 initial_allocation_mask; /* bits set for reserved keys */ -/* - * Define these here temporarily so we're not dependent on patching linux/mm.h. - * Once it's updated we can drop these. - */ -#ifndef VM_PKEY_BIT0 -# define VM_PKEY_SHIFT VM_HIGH_ARCH_BIT_0 -# define VM_PKEY_BIT0 VM_HIGH_ARCH_0 -# define VM_PKEY_BIT1 VM_HIGH_ARCH_1 -# define VM_PKEY_BIT2 VM_HIGH_ARCH_2 -# define VM_PKEY_BIT3 VM_HIGH_ARCH_3 -# define VM_PKEY_BIT4 VM_HIGH_ARCH_4 -#endif - #define ARCH_VM_PKEY_FLAGS (VM_PKEY_BIT0 | VM_PKEY_BIT1 | VM_PKEY_BIT2 | \ VM_PKEY_BIT3 | VM_PKEY_BIT4) diff --git a/arch/powerpc/include/asm/plpar_wrappers.h b/arch/powerpc/include/asm/plpar_wrappers.h index 96c1a46acbd0..cff5a411e595 100644 --- a/arch/powerpc/include/asm/plpar_wrappers.h +++ b/arch/powerpc/include/asm/plpar_wrappers.h @@ -39,10 +39,10 @@ static inline long extended_cede_processor(unsigned long latency_hint) set_cede_latency_hint(latency_hint); rc = cede_processor(); -#ifdef CONFIG_TRACE_IRQFLAGS - /* Ensure that H_CEDE returns with IRQs on */ - if (WARN_ON(!(mfmsr() & MSR_EE))) - __hard_irq_enable(); +#ifdef CONFIG_PPC_IRQ_SOFT_MASK_DEBUG + /* Ensure that H_CEDE returns with IRQs on */ + if (WARN_ON(!(mfmsr() & MSR_EE))) + __hard_irq_enable(); #endif set_cede_latency_hint(old_latency_hint); diff --git a/arch/powerpc/include/asm/pmac_pfunc.h b/arch/powerpc/include/asm/pmac_pfunc.h index 73bd8f28f2a8..cee4e9f5b8cf 100644 --- a/arch/powerpc/include/asm/pmac_pfunc.h +++ b/arch/powerpc/include/asm/pmac_pfunc.h @@ -245,6 +245,7 @@ extern void pmf_put_function(struct pmf_function *func); extern int pmf_call_one(struct pmf_function *func, struct pmf_args *args); +int pmac_pfunc_base_install(void); /* Suspend/resume code called by via-pmu directly for now */ extern void pmac_pfunc_base_suspend(void); diff --git a/arch/powerpc/include/asm/pnv-ocxl.h b/arch/powerpc/include/asm/pnv-ocxl.h index f6945d3bc971..208b5503f4ed 100644 --- a/arch/powerpc/include/asm/pnv-ocxl.h +++ b/arch/powerpc/include/asm/pnv-ocxl.h @@ -28,7 +28,7 @@ extern int pnv_ocxl_map_xsl_regs(struct pci_dev *dev, void __iomem **dsisr, extern int pnv_ocxl_spa_setup(struct pci_dev *dev, void *spa_mem, int PE_mask, void **platform_data); extern void pnv_ocxl_spa_release(void *platform_data); -extern int pnv_ocxl_spa_remove_pe(void *platform_data, int pe_handle); +extern int pnv_ocxl_spa_remove_pe_from_cache(void *platform_data, int pe_handle); extern int pnv_ocxl_alloc_xive_irq(u32 *irq, u64 *trigger_addr); extern void pnv_ocxl_free_xive_irq(u32 irq); diff --git a/arch/powerpc/include/asm/ppc-opcode.h b/arch/powerpc/include/asm/ppc-opcode.h index 18883b8a6dac..4436887bc415 100644 --- a/arch/powerpc/include/asm/ppc-opcode.h +++ b/arch/powerpc/include/asm/ppc-opcode.h @@ -162,6 +162,7 @@ /* VMX Vector Store Instructions */ #define OP_31_XOP_STVX 231 +#define OP_31 31 #define OP_LWZ 32 #define OP_STFS 52 #define OP_STFSU 53 diff --git a/arch/powerpc/include/asm/ppc_asm.h b/arch/powerpc/include/asm/ppc_asm.h index 13f7f4c0e1ea..75ece56dcd62 100644 --- a/arch/powerpc/include/asm/ppc_asm.h +++ b/arch/powerpc/include/asm/ppc_asm.h @@ -80,10 +80,8 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_SPLPAR) #else #define SAVE_GPR(n, base) stw n,GPR0+4*(n)(base) #define REST_GPR(n, base) lwz n,GPR0+4*(n)(base) -#define SAVE_NVGPRS(base) SAVE_GPR(13, base); SAVE_8GPRS(14, base); \ - SAVE_10GPRS(22, base) -#define REST_NVGPRS(base) REST_GPR(13, base); REST_8GPRS(14, base); \ - REST_10GPRS(22, base) +#define SAVE_NVGPRS(base) stmw 13, GPR0+4*13(base) +#define REST_NVGPRS(base) lmw 13, GPR0+4*13(base) #endif #define SAVE_2GPRS(n, base) SAVE_GPR(n, base); SAVE_GPR(n+1, base) diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h index c4b36a494a63..5debe337ea9d 100644 --- a/arch/powerpc/include/asm/processor.h +++ b/arch/powerpc/include/asm/processor.h @@ -249,7 +249,7 @@ struct thread_struct { unsigned long ksp_vsid; #endif struct pt_regs *regs; /* Pointer to saved register state */ - mm_segment_t fs; /* for get_fs() validation */ + mm_segment_t addr_limit; /* for get_fs() validation */ #ifdef CONFIG_BOOKE /* BookE base exception scratch space; align on cacheline */ unsigned long normsave[8] ____cacheline_aligned; @@ -264,10 +264,6 @@ struct thread_struct { struct thread_fp_state *fp_save_area; int fpexc_mode; /* floating-point exception mode */ unsigned int align_ctl; /* alignment handling control */ -#ifdef CONFIG_PPC64 - unsigned long start_tb; /* Start purr when proc switched in */ - unsigned long accum_tb; /* Total accumulated purr for process */ -#endif #ifdef CONFIG_HAVE_HW_BREAKPOINT struct perf_event *ptrace_bps[HBP_NUM]; /* @@ -383,7 +379,7 @@ struct thread_struct { #define INIT_THREAD { \ .ksp = INIT_SP, \ .ksp_limit = INIT_SP_LIMIT, \ - .fs = KERNEL_DS, \ + .addr_limit = KERNEL_DS, \ .pgdir = swapper_pg_dir, \ .fpexc_mode = MSR_FE0 | MSR_FE1, \ SPEFSCR_INIT \ @@ -392,7 +388,7 @@ struct thread_struct { #define INIT_THREAD { \ .ksp = INIT_SP, \ .regs = (struct pt_regs *)INIT_SP - 1, /* XXX bogus, I think */ \ - .fs = KERNEL_DS, \ + .addr_limit = KERNEL_DS, \ .fpexc_mode = 0, \ .ppr = INIT_PPR, \ .fscr = FSCR_TAR | FSCR_EBB \ diff --git a/arch/powerpc/include/asm/pte-common.h b/arch/powerpc/include/asm/pte-common.h index c4a72c7a8c83..bef56141a549 100644 --- a/arch/powerpc/include/asm/pte-common.h +++ b/arch/powerpc/include/asm/pte-common.h @@ -60,10 +60,6 @@ #ifndef _PMD_PRESENT_MASK #define _PMD_PRESENT_MASK _PMD_PRESENT #endif -#ifndef _PMD_SIZE -#define _PMD_SIZE 0 -#define PMD_PAGE_SIZE(pmd) bad_call_to_PMD_PAGE_SIZE() -#endif #ifndef _PMD_USER #define _PMD_USER 0 #endif @@ -88,11 +84,7 @@ #define _PTE_NONE_MASK _PAGE_HPTEFLAGS #endif -/* Make sure we get a link error if PMD_PAGE_SIZE is ever called on a - * kernel without large page PMD support - */ #ifndef __ASSEMBLY__ -extern unsigned long bad_call_to_PMD_PAGE_SIZE(void); /* * Don't just check for any non zero bits in __PAGE_USER, since for book3e @@ -216,9 +208,6 @@ static inline bool pte_user(pte_t pte) #define PAGE_AGP (PAGE_KERNEL_NC) #define HAVE_PAGE_AGP -/* Advertise support for _PAGE_SPECIAL */ -#define __HAVE_ARCH_PTE_SPECIAL - #ifndef _PAGE_READ /* if not defined, we should not find _PAGE_WRITE too */ #define _PAGE_READ 0 diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h index cb0f272ce123..562568414cf4 100644 --- a/arch/powerpc/include/asm/reg.h +++ b/arch/powerpc/include/asm/reg.h @@ -146,6 +146,12 @@ #define MSR_64BIT 0 #endif +/* Condition Register related */ +#define CR0_SHIFT 28 +#define CR0_MASK 0xF +#define CR0_TBEGIN_FAILURE (0x2 << 28) /* 0b0010 */ + + /* Power Management - Processor Stop Status and Control Register Fields */ #define PSSCR_RL_MASK 0x0000000F /* Requested Level */ #define PSSCR_MTL_MASK 0x000000F0 /* Maximum Transition Level */ @@ -239,13 +245,27 @@ #define SPRN_TFIAR 0x81 /* Transaction Failure Inst Addr */ #define SPRN_TEXASR 0x82 /* Transaction EXception & Summary */ #define SPRN_TEXASRU 0x83 /* '' '' '' Upper 32 */ -#define TEXASR_ABORT __MASK(63-31) /* terminated by tabort or treclaim */ -#define TEXASR_SUSP __MASK(63-32) /* tx failed in suspended state */ -#define TEXASR_HV __MASK(63-34) /* MSR[HV] when failure occurred */ -#define TEXASR_PR __MASK(63-35) /* MSR[PR] when failure occurred */ -#define TEXASR_FS __MASK(63-36) /* TEXASR Failure Summary */ -#define TEXASR_EXACT __MASK(63-37) /* TFIAR value is exact */ + +#define TEXASR_FC_LG (63 - 7) /* Failure Code */ +#define TEXASR_AB_LG (63 - 31) /* Abort */ +#define TEXASR_SU_LG (63 - 32) /* Suspend */ +#define TEXASR_HV_LG (63 - 34) /* Hypervisor state*/ +#define TEXASR_PR_LG (63 - 35) /* Privilege level */ +#define TEXASR_FS_LG (63 - 36) /* failure summary */ +#define TEXASR_EX_LG (63 - 37) /* TFIAR exact bit */ +#define TEXASR_ROT_LG (63 - 38) /* ROT bit */ + +#define TEXASR_ABORT __MASK(TEXASR_AB_LG) /* terminated by tabort or treclaim */ +#define TEXASR_SUSP __MASK(TEXASR_SU_LG) /* tx failed in suspended state */ +#define TEXASR_HV __MASK(TEXASR_HV_LG) /* MSR[HV] when failure occurred */ +#define TEXASR_PR __MASK(TEXASR_PR_LG) /* MSR[PR] when failure occurred */ +#define TEXASR_FS __MASK(TEXASR_FS_LG) /* TEXASR Failure Summary */ +#define TEXASR_EXACT __MASK(TEXASR_EX_LG) /* TFIAR value is exact */ +#define TEXASR_ROT __MASK(TEXASR_ROT_LG) +#define TEXASR_FC (ASM_CONST(0xFF) << TEXASR_FC_LG) + #define SPRN_TFHAR 0x80 /* Transaction Failure Handler Addr */ + #define SPRN_TIDR 144 /* Thread ID register */ #define SPRN_CTRLF 0x088 #define SPRN_CTRLT 0x098 @@ -365,6 +385,7 @@ #define SPRN_PSSCR 0x357 /* Processor Stop Status and Control Register (ISA 3.0) */ #define SPRN_PSSCR_PR 0x337 /* PSSCR ISA 3.0, privileged mode access */ #define SPRN_PMCR 0x374 /* Power Management Control Register */ +#define SPRN_RWMR 0x375 /* Region-Weighting Mode Register */ /* HFSCR and FSCR bit numbers are the same */ #define FSCR_SCV_LG 12 /* Enable System Call Vectored */ diff --git a/arch/powerpc/include/asm/rheap.h b/arch/powerpc/include/asm/rheap.h index 172381769cfc..8e83703d6736 100644 --- a/arch/powerpc/include/asm/rheap.h +++ b/arch/powerpc/include/asm/rheap.h @@ -83,6 +83,9 @@ extern int rh_get_stats(rh_info_t * info, int what, int max_stats, /* Simple dump of remote heap info */ extern void rh_dump(rh_info_t * info); +/* Simple dump of remote info block */ +void rh_dump_blk(rh_info_t *info, rh_block_t *blk); + /* Set owner of taken block */ extern int rh_set_owner(rh_info_t * info, unsigned long start, const char *owner); diff --git a/arch/powerpc/include/asm/rtas.h b/arch/powerpc/include/asm/rtas.h index ec9dd79398ee..71e393c46a49 100644 --- a/arch/powerpc/include/asm/rtas.h +++ b/arch/powerpc/include/asm/rtas.h @@ -361,7 +361,7 @@ extern int rtas_offline_cpus_mask(cpumask_var_t cpus); extern int rtas_ibm_suspend_me(u64 handle); struct rtc_time; -extern unsigned long rtas_get_boot_time(void); +extern time64_t rtas_get_boot_time(void); extern void rtas_get_rtc_time(struct rtc_time *rtc_time); extern int rtas_set_rtc_time(struct rtc_time *rtc_time); diff --git a/arch/powerpc/include/asm/setup.h b/arch/powerpc/include/asm/setup.h index 27fa52ed6d00..8721fd004291 100644 --- a/arch/powerpc/include/asm/setup.h +++ b/arch/powerpc/include/asm/setup.h @@ -52,6 +52,15 @@ enum l1d_flush_type { void setup_rfi_flush(enum l1d_flush_type, bool enable); void do_rfi_flush_fixups(enum l1d_flush_type types); +void setup_barrier_nospec(void); +void do_barrier_nospec_fixups(bool enable); +extern bool barrier_nospec_enabled; + +#ifdef CONFIG_PPC_BOOK3S_64 +void do_barrier_nospec_fixups_range(bool enable, void *start, void *end); +#else +static inline void do_barrier_nospec_fixups_range(bool enable, void *start, void *end) { }; +#endif #endif /* !__ASSEMBLY__ */ diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h index cfecfee1194b..29ffaabdf75b 100644 --- a/arch/powerpc/include/asm/smp.h +++ b/arch/powerpc/include/asm/smp.h @@ -58,6 +58,7 @@ struct smp_ops_t { extern void smp_flush_nmi_ipi(u64 delay_us); extern int smp_send_nmi_ipi(int cpu, void (*fn)(struct pt_regs *), u64 delay_us); +extern int smp_send_safe_nmi_ipi(int cpu, void (*fn)(struct pt_regs *), u64 delay_us); extern void smp_send_debugger_break(void); extern void start_secondary_resume(void); extern void smp_generic_give_timebase(void); diff --git a/arch/powerpc/include/asm/sstep.h b/arch/powerpc/include/asm/sstep.h index ab9d849644d0..4547891a684b 100644 --- a/arch/powerpc/include/asm/sstep.h +++ b/arch/powerpc/include/asm/sstep.h @@ -97,6 +97,8 @@ enum instruction_type { #define SIZE(n) ((n) << 12) #define GETSIZE(w) ((w) >> 12) +#define GETTYPE(t) ((t) & INSTR_TYPE_MASK) + #define MKOP(t, f, s) ((t) | (f) | SIZE(s)) struct instruction_op { diff --git a/arch/powerpc/include/asm/switch_to.h b/arch/powerpc/include/asm/switch_to.h index be8c9fa23983..5b03d8a82409 100644 --- a/arch/powerpc/include/asm/switch_to.h +++ b/arch/powerpc/include/asm/switch_to.h @@ -94,6 +94,5 @@ static inline void clear_task_ebb(struct task_struct *t) extern int set_thread_uses_vas(void); extern int set_thread_tidr(struct task_struct *t); -extern void clear_thread_tidr(struct task_struct *t); #endif /* _ASM_POWERPC_SWITCH_TO_H */ diff --git a/arch/powerpc/include/asm/syscalls.h b/arch/powerpc/include/asm/syscalls.h index 1b90a3516a35..398171fdcd9f 100644 --- a/arch/powerpc/include/asm/syscalls.h +++ b/arch/powerpc/include/asm/syscalls.h @@ -16,7 +16,7 @@ asmlinkage long sys_mmap2(unsigned long addr, size_t len, unsigned long prot, unsigned long flags, unsigned long fd, unsigned long pgoff); asmlinkage long ppc64_personality(unsigned long personality); -asmlinkage int ppc_rtas(struct rtas_args __user *uargs); +asmlinkage long sys_rtas(struct rtas_args __user *uargs); #endif /* __KERNEL__ */ #endif /* __ASM_POWERPC_SYSCALLS_H */ diff --git a/arch/powerpc/include/asm/systbl.h b/arch/powerpc/include/asm/systbl.h index d61f9c96d916..cfcf6a874cfa 100644 --- a/arch/powerpc/include/asm/systbl.h +++ b/arch/powerpc/include/asm/systbl.h @@ -147,7 +147,7 @@ SYSCALL_SPU(setfsuid) SYSCALL_SPU(setfsgid) SYSCALL_SPU(llseek) COMPAT_SYS_SPU(getdents) -SYSX_SPU(sys_select,ppc32_select,sys_select) +COMPAT_SPU_NEW(select) SYSCALL_SPU(flock) SYSCALL_SPU(msync) COMPAT_SYS_SPU(readv) @@ -245,7 +245,7 @@ SYSCALL_SPU(epoll_create) SYSCALL_SPU(epoll_ctl) SYSCALL_SPU(epoll_wait) SYSCALL_SPU(remap_file_pages) -SYSX_SPU(sys_timer_create,compat_sys_timer_create,sys_timer_create) +COMPAT_SYS_SPU(timer_create) COMPAT_SYS_SPU(timer_settime) COMPAT_SYS_SPU(timer_gettime) SYSCALL_SPU(timer_getoverrun) @@ -260,7 +260,7 @@ COMPAT_SYS_SPU(utimes) COMPAT_SYS_SPU(statfs64) COMPAT_SYS_SPU(fstatfs64) SYSX(sys_ni_syscall,ppc_fadvise64_64,ppc_fadvise64_64) -PPC_SYS_SPU(rtas) +SYSCALL_SPU(rtas) OLDSYS(debug_setcontext) SYSCALL(ni_syscall) COMPAT_SYS(migrate_pages) @@ -392,3 +392,4 @@ SYSCALL(statx) SYSCALL(pkey_alloc) SYSCALL(pkey_free) SYSCALL(pkey_mprotect) +SYSCALL(rseq) diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/asm/thread_info.h index 5964145db03d..f308dfeb2746 100644 --- a/arch/powerpc/include/asm/thread_info.h +++ b/arch/powerpc/include/asm/thread_info.h @@ -79,8 +79,7 @@ extern int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src #define TIF_SYSCALL_TRACE 0 /* syscall trace active */ #define TIF_SIGPENDING 1 /* signal pending */ #define TIF_NEED_RESCHED 2 /* rescheduling necessary */ -#define TIF_POLLING_NRFLAG 3 /* true if poll_idle() is polling - TIF_NEED_RESCHED */ +#define TIF_FSCHECK 3 /* Check FS is USER_DS on return */ #define TIF_32BIT 4 /* 32 bit binary */ #define TIF_RESTORE_TM 5 /* need to restore TM FP/VEC/VSX */ #define TIF_PATCH_PENDING 6 /* pending live patching update */ @@ -99,6 +98,7 @@ extern int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src #if defined(CONFIG_PPC64) #define TIF_ELF2ABI 18 /* function descriptors must die! */ #endif +#define TIF_POLLING_NRFLAG 19 /* true if poll_idle() is polling TIF_NEED_RESCHED */ /* as above, but as bit values */ #define _TIF_SYSCALL_TRACE (1<context.copros) > 0); + /* + * It's possible for mm_access to take a reference on mm_users to + * access the remote mm from another thread, but it's not allowed + * to set mm_cpumask, so mm_users may be > 1 here. + */ + WARN_ON(current->mm != mm); + atomic_set(&mm->context.active_cpus, 1); + cpumask_clear(mm_cpumask(mm)); + cpumask_set_cpu(smp_processor_id(), mm_cpumask(mm)); +} #else /* CONFIG_PPC_BOOK3S_64 */ static inline int mm_is_thread_local(struct mm_struct *mm) { diff --git a/arch/powerpc/include/asm/tm.h b/arch/powerpc/include/asm/tm.h index b1658c97047c..e94f6db5e367 100644 --- a/arch/powerpc/include/asm/tm.h +++ b/arch/powerpc/include/asm/tm.h @@ -10,12 +10,10 @@ #ifndef __ASSEMBLY__ -extern void tm_enable(void); extern void tm_reclaim(struct thread_struct *thread, uint8_t cause); extern void tm_reclaim_current(uint8_t cause); extern void tm_recheckpoint(struct thread_struct *thread); -extern void tm_abort(uint8_t cause); extern void tm_save_sprs(struct thread_struct *thread); extern void tm_restore_sprs(struct thread_struct *thread); diff --git a/arch/powerpc/include/asm/trace.h b/arch/powerpc/include/asm/trace.h index 33f3b479138b..d018e8602694 100644 --- a/arch/powerpc/include/asm/trace.h +++ b/arch/powerpc/include/asm/trace.h @@ -81,8 +81,7 @@ TRACE_EVENT_FN_COND(hcall_entry, TRACE_EVENT_FN_COND(hcall_exit, - TP_PROTO(unsigned long opcode, unsigned long retval, - unsigned long *retbuf), + TP_PROTO(unsigned long opcode, long retval, unsigned long *retbuf), TP_ARGS(opcode, retval, retbuf), @@ -90,7 +89,7 @@ TRACE_EVENT_FN_COND(hcall_exit, TP_STRUCT__entry( __field(unsigned long, opcode) - __field(unsigned long, retval) + __field(long, retval) ), TP_fast_assign( @@ -98,7 +97,7 @@ TRACE_EVENT_FN_COND(hcall_exit, __entry->retval = retval; ), - TP_printk("opcode=%lu retval=%lu", __entry->opcode, __entry->retval), + TP_printk("opcode=%lu retval=%ld", __entry->opcode, __entry->retval), hcall_tracepoint_regfunc, hcall_tracepoint_unregfunc ); diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/uaccess.h index a62ee663b2c8..468653ce844c 100644 --- a/arch/powerpc/include/asm/uaccess.h +++ b/arch/powerpc/include/asm/uaccess.h @@ -30,8 +30,14 @@ #endif #define get_ds() (KERNEL_DS) -#define get_fs() (current->thread.fs) -#define set_fs(val) (current->thread.fs = (val)) +#define get_fs() (current->thread.addr_limit) + +static inline void set_fs(mm_segment_t fs) +{ + current->thread.addr_limit = fs; + /* On user-mode return check addr_limit (fs) is correct */ + set_thread_flag(TIF_FSCHECK); +} #define segment_eq(a, b) ((a).seg == (b).seg) @@ -252,6 +258,7 @@ do { \ __chk_user_ptr(ptr); \ if (!is_kernel_addr((unsigned long)__gu_addr)) \ might_fault(); \ + barrier_nospec(); \ __get_user_size(__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__typeof__(*(ptr)))__gu_val; \ __gu_err; \ @@ -263,8 +270,10 @@ do { \ unsigned long __gu_val = 0; \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \ might_fault(); \ - if (access_ok(VERIFY_READ, __gu_addr, (size))) \ + if (access_ok(VERIFY_READ, __gu_addr, (size))) { \ + barrier_nospec(); \ __get_user_size(__gu_val, __gu_addr, (size), __gu_err); \ + } \ (x) = (__force __typeof__(*(ptr)))__gu_val; \ __gu_err; \ }) @@ -275,6 +284,7 @@ do { \ unsigned long __gu_val; \ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \ __chk_user_ptr(ptr); \ + barrier_nospec(); \ __get_user_size(__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__force __typeof__(*(ptr)))__gu_val; \ __gu_err; \ @@ -302,15 +312,19 @@ static inline unsigned long raw_copy_from_user(void *to, switch (n) { case 1: + barrier_nospec(); __get_user_size(*(u8 *)to, from, 1, ret); break; case 2: + barrier_nospec(); __get_user_size(*(u16 *)to, from, 2, ret); break; case 4: + barrier_nospec(); __get_user_size(*(u32 *)to, from, 4, ret); break; case 8: + barrier_nospec(); __get_user_size(*(u64 *)to, from, 8, ret); break; } @@ -318,6 +332,7 @@ static inline unsigned long raw_copy_from_user(void *to, return 0; } + barrier_nospec(); return __copy_tofrom_user((__force void __user *)to, from, n); } diff --git a/arch/powerpc/include/asm/unistd.h b/arch/powerpc/include/asm/unistd.h index daf1ba97a00c..1e9708632dce 100644 --- a/arch/powerpc/include/asm/unistd.h +++ b/arch/powerpc/include/asm/unistd.h @@ -12,7 +12,7 @@ #include -#define NR_syscalls 387 +#define NR_syscalls 388 #define __NR__exit __NR_exit diff --git a/arch/powerpc/include/asm/xive-regs.h b/arch/powerpc/include/asm/xive-regs.h index fa4288822b68..6de989f8defd 100644 --- a/arch/powerpc/include/asm/xive-regs.h +++ b/arch/powerpc/include/asm/xive-regs.h @@ -123,10 +123,4 @@ #define TM_QW3_NSR_I PPC_BIT8(2) #define TM_QW3_NSR_GRP_LVL PPC_BIT8(3,7) -/* Utilities to manipulate these (originaly from OPAL) */ -#define MASK_TO_LSH(m) (__builtin_ffsl(m) - 1) -#define GETFIELD(m, v) (((v) & (m)) >> MASK_TO_LSH(m)) -#define SETFIELD(m, v, val) \ - (((v) & ~(m)) | ((((typeof(v))(val)) << MASK_TO_LSH(m)) & (m))) - #endif /* _ASM_POWERPC_XIVE_REGS_H */ diff --git a/arch/powerpc/include/asm/xmon.h b/arch/powerpc/include/asm/xmon.h index eb42a0c6e1d9..30ff69bd8f43 100644 --- a/arch/powerpc/include/asm/xmon.h +++ b/arch/powerpc/include/asm/xmon.h @@ -29,7 +29,7 @@ static inline void xmon_register_spus(struct list_head *list) { }; extern int cpus_are_in_xmon(void); #endif -extern void xmon_printf(const char *format, ...); +extern __printf(1, 2) void xmon_printf(const char *format, ...); #endif /* __KERNEL __ */ #endif /* __ASM_POWERPC_XMON_H */ diff --git a/arch/powerpc/include/asm/xor.h b/arch/powerpc/include/asm/xor.h index a36c2069d8ed..7d6dc503349d 100644 --- a/arch/powerpc/include/asm/xor.h +++ b/arch/powerpc/include/asm/xor.h @@ -24,17 +24,7 @@ #include #include - -void xor_altivec_2(unsigned long bytes, unsigned long *v1_in, - unsigned long *v2_in); -void xor_altivec_3(unsigned long bytes, unsigned long *v1_in, - unsigned long *v2_in, unsigned long *v3_in); -void xor_altivec_4(unsigned long bytes, unsigned long *v1_in, - unsigned long *v2_in, unsigned long *v3_in, - unsigned long *v4_in); -void xor_altivec_5(unsigned long bytes, unsigned long *v1_in, - unsigned long *v2_in, unsigned long *v3_in, - unsigned long *v4_in, unsigned long *v5_in); +#include static struct xor_block_template xor_block_altivec = { .name = "altivec", diff --git a/arch/powerpc/include/asm/xor_altivec.h b/arch/powerpc/include/asm/xor_altivec.h new file mode 100644 index 000000000000..6ca923510b59 --- /dev/null +++ b/arch/powerpc/include/asm/xor_altivec.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_POWERPC_XOR_ALTIVEC_H +#define _ASM_POWERPC_XOR_ALTIVEC_H + +#ifdef CONFIG_ALTIVEC + +void xor_altivec_2(unsigned long bytes, unsigned long *v1_in, + unsigned long *v2_in); +void xor_altivec_3(unsigned long bytes, unsigned long *v1_in, + unsigned long *v2_in, unsigned long *v3_in); +void xor_altivec_4(unsigned long bytes, unsigned long *v1_in, + unsigned long *v2_in, unsigned long *v3_in, + unsigned long *v4_in); +void xor_altivec_5(unsigned long bytes, unsigned long *v1_in, + unsigned long *v2_in, unsigned long *v3_in, + unsigned long *v4_in, unsigned long *v5_in); + +#endif +#endif /* _ASM_POWERPC_XOR_ALTIVEC_H */ diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h index 833ed9a16adf..1b32b56a03d3 100644 --- a/arch/powerpc/include/uapi/asm/kvm.h +++ b/arch/powerpc/include/uapi/asm/kvm.h @@ -633,6 +633,7 @@ struct kvm_ppc_cpu_char { #define KVM_REG_PPC_PSSCR (KVM_REG_PPC | KVM_REG_SIZE_U64 | 0xbd) #define KVM_REG_PPC_DEC_EXPIRY (KVM_REG_PPC | KVM_REG_SIZE_U64 | 0xbe) +#define KVM_REG_PPC_ONLINE (KVM_REG_PPC | KVM_REG_SIZE_U32 | 0xbf) /* Transactional Memory checkpointed state: * This is all GPRs, all VSX regs and a subset of SPRs diff --git a/arch/powerpc/include/uapi/asm/msgbuf.h b/arch/powerpc/include/uapi/asm/msgbuf.h index 65beb0942500..2b1b37797a47 100644 --- a/arch/powerpc/include/uapi/asm/msgbuf.h +++ b/arch/powerpc/include/uapi/asm/msgbuf.h @@ -10,18 +10,18 @@ struct msqid64_ds { struct ipc64_perm msg_perm; -#ifndef __powerpc64__ - unsigned int __unused1; -#endif +#ifdef __powerpc64__ __kernel_time_t msg_stime; /* last msgsnd time */ -#ifndef __powerpc64__ - unsigned int __unused2; -#endif __kernel_time_t msg_rtime; /* last msgrcv time */ -#ifndef __powerpc64__ - unsigned int __unused3; -#endif __kernel_time_t msg_ctime; /* last change time */ +#else + unsigned long msg_stime_high; + unsigned long msg_stime; /* last msgsnd time */ + unsigned long msg_rtime_high; + unsigned long msg_rtime; /* last msgrcv time */ + unsigned long msg_ctime_high; + unsigned long msg_ctime; /* last change time */ +#endif unsigned long msg_cbytes; /* current number of bytes on queue */ unsigned long msg_qnum; /* number of messages in queue */ unsigned long msg_qbytes; /* max number of bytes on queue */ diff --git a/arch/powerpc/include/uapi/asm/sembuf.h b/arch/powerpc/include/uapi/asm/sembuf.h index 8f393d60f02d..3f60946f77e3 100644 --- a/arch/powerpc/include/uapi/asm/sembuf.h +++ b/arch/powerpc/include/uapi/asm/sembuf.h @@ -15,20 +15,20 @@ * between kernel and user space. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem - * - 2 miscellaneous 32-bit values + * - 2 miscellaneous 32/64-bit values */ struct semid64_ds { struct ipc64_perm sem_perm; /* permissions .. see ipc.h */ #ifndef __powerpc64__ - unsigned long __unused1; -#endif + unsigned long sem_otime_high; + unsigned long sem_otime; /* last semop time */ + unsigned long sem_ctime_high; + unsigned long sem_ctime; /* last change time */ +#else __kernel_time_t sem_otime; /* last semop time */ -#ifndef __powerpc64__ - unsigned long __unused2; -#endif __kernel_time_t sem_ctime; /* last change time */ +#endif unsigned long sem_nsems; /* no. of semaphores in array */ unsigned long __unused3; unsigned long __unused4; diff --git a/arch/powerpc/include/uapi/asm/shmbuf.h b/arch/powerpc/include/uapi/asm/shmbuf.h index deb1c3e503d3..b591c4d7e4c5 100644 --- a/arch/powerpc/include/uapi/asm/shmbuf.h +++ b/arch/powerpc/include/uapi/asm/shmbuf.h @@ -16,25 +16,22 @@ * between kernel and user space. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem * - 2 miscellaneous 32-bit values */ struct shmid64_ds { struct ipc64_perm shm_perm; /* operation perms */ -#ifndef __powerpc64__ - unsigned long __unused1; -#endif +#ifdef __powerpc64__ __kernel_time_t shm_atime; /* last attach time */ -#ifndef __powerpc64__ - unsigned long __unused2; -#endif __kernel_time_t shm_dtime; /* last detach time */ -#ifndef __powerpc64__ - unsigned long __unused3; -#endif __kernel_time_t shm_ctime; /* last change time */ -#ifndef __powerpc64__ +#else + unsigned long shm_atime_high; + unsigned long shm_atime; /* last attach time */ + unsigned long shm_dtime_high; + unsigned long shm_dtime; /* last detach time */ + unsigned long shm_ctime_high; + unsigned long shm_ctime; /* last change time */ unsigned long __unused4; #endif size_t shm_segsz; /* size of segment (bytes) */ diff --git a/arch/powerpc/include/uapi/asm/siginfo.h b/arch/powerpc/include/uapi/asm/siginfo.h index 9f142451a01f..1d51d9b88221 100644 --- a/arch/powerpc/include/uapi/asm/siginfo.h +++ b/arch/powerpc/include/uapi/asm/siginfo.h @@ -15,19 +15,4 @@ #include -/* - * SIGFPE si_codes - */ -#ifdef __KERNEL__ -#define FPE_FIXME 0 /* Broken dup of SI_USER */ -#endif /* __KERNEL__ */ - -/* - * SIGTRAP si_codes - */ -#ifdef __KERNEL__ -#define TRAP_FIXME 0 /* Broken dup of SI_USER */ -#endif /* __KERNEL__ */ - - #endif /* _ASM_POWERPC_SIGINFO_H */ diff --git a/arch/powerpc/include/uapi/asm/unistd.h b/arch/powerpc/include/uapi/asm/unistd.h index 389c36fd8299..ac5ba55066dd 100644 --- a/arch/powerpc/include/uapi/asm/unistd.h +++ b/arch/powerpc/include/uapi/asm/unistd.h @@ -398,5 +398,6 @@ #define __NR_pkey_alloc 384 #define __NR_pkey_free 385 #define __NR_pkey_mprotect 386 +#define __NR_rseq 387 #endif /* _UAPI_ASM_POWERPC_UNISTD_H_ */ diff --git a/arch/powerpc/kernel/align.c b/arch/powerpc/kernel/align.c index 3e6c0744c174..11550a3d1ac2 100644 --- a/arch/powerpc/kernel/align.c +++ b/arch/powerpc/kernel/align.c @@ -339,7 +339,7 @@ int fix_alignment(struct pt_regs *regs) if (r < 0) return -EINVAL; - type = op.type & INSTR_TYPE_MASK; + type = GETTYPE(op.type); if (!OP_IS_LOAD_STORE(type)) { if (op.type != CACHEOP + DCBZ) return -EINVAL; diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c index 373dc1d6ef44..0a0544335950 100644 --- a/arch/powerpc/kernel/asm-offsets.c +++ b/arch/powerpc/kernel/asm-offsets.c @@ -13,6 +13,7 @@ * 2 of the License, or (at your option) any later version. */ +#include #include #include #include @@ -42,7 +43,6 @@ #include #include #include -#include #include #include #include @@ -180,6 +180,7 @@ int main(void) OFFSET(PACAKMSR, paca_struct, kernel_msr); OFFSET(PACAIRQSOFTMASK, paca_struct, irq_soft_mask); OFFSET(PACAIRQHAPPENED, paca_struct, irq_happened); + OFFSET(PACA_FTRACE_ENABLED, paca_struct, ftrace_enabled); #ifdef CONFIG_PPC_BOOK3S OFFSET(PACACONTEXTID, paca_struct, mm_ctx_id); #ifdef CONFIG_PPC_MM_SLICES @@ -425,20 +426,20 @@ int main(void) OFFSET(VCPU_HOST_STACK, kvm_vcpu, arch.host_stack); OFFSET(VCPU_HOST_PID, kvm_vcpu, arch.host_pid); OFFSET(VCPU_GUEST_PID, kvm_vcpu, arch.pid); - OFFSET(VCPU_GPRS, kvm_vcpu, arch.gpr); + OFFSET(VCPU_GPRS, kvm_vcpu, arch.regs.gpr); OFFSET(VCPU_VRSAVE, kvm_vcpu, arch.vrsave); OFFSET(VCPU_FPRS, kvm_vcpu, arch.fp.fpr); #ifdef CONFIG_ALTIVEC OFFSET(VCPU_VRS, kvm_vcpu, arch.vr.vr); #endif - OFFSET(VCPU_XER, kvm_vcpu, arch.xer); - OFFSET(VCPU_CTR, kvm_vcpu, arch.ctr); - OFFSET(VCPU_LR, kvm_vcpu, arch.lr); + OFFSET(VCPU_XER, kvm_vcpu, arch.regs.xer); + OFFSET(VCPU_CTR, kvm_vcpu, arch.regs.ctr); + OFFSET(VCPU_LR, kvm_vcpu, arch.regs.link); #ifdef CONFIG_PPC_BOOK3S OFFSET(VCPU_TAR, kvm_vcpu, arch.tar); #endif OFFSET(VCPU_CR, kvm_vcpu, arch.cr); - OFFSET(VCPU_PC, kvm_vcpu, arch.pc); + OFFSET(VCPU_PC, kvm_vcpu, arch.regs.nip); #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE OFFSET(VCPU_MSR, kvm_vcpu, arch.shregs.msr); OFFSET(VCPU_SRR0, kvm_vcpu, arch.shregs.srr0); @@ -695,10 +696,10 @@ int main(void) #else /* CONFIG_PPC_BOOK3S */ OFFSET(VCPU_CR, kvm_vcpu, arch.cr); - OFFSET(VCPU_XER, kvm_vcpu, arch.xer); - OFFSET(VCPU_LR, kvm_vcpu, arch.lr); - OFFSET(VCPU_CTR, kvm_vcpu, arch.ctr); - OFFSET(VCPU_PC, kvm_vcpu, arch.pc); + OFFSET(VCPU_XER, kvm_vcpu, arch.regs.xer); + OFFSET(VCPU_LR, kvm_vcpu, arch.regs.link); + OFFSET(VCPU_CTR, kvm_vcpu, arch.regs.ctr); + OFFSET(VCPU_PC, kvm_vcpu, arch.regs.nip); OFFSET(VCPU_SPRG9, kvm_vcpu, arch.sprg9); OFFSET(VCPU_LAST_INST, kvm_vcpu, arch.last_inst); OFFSET(VCPU_FAULT_DEAR, kvm_vcpu, arch.fault_dear); diff --git a/arch/powerpc/kernel/btext.c b/arch/powerpc/kernel/btext.c index 6537cba1a758..b2072d5bbf2b 100644 --- a/arch/powerpc/kernel/btext.c +++ b/arch/powerpc/kernel/btext.c @@ -157,20 +157,20 @@ void btext_map(void) /* By default, we are no longer mapped */ boot_text_mapped = 0; - if (dispDeviceBase == 0) + if (!dispDeviceBase) return; base = ((unsigned long) dispDeviceBase) & 0xFFFFF000UL; offset = ((unsigned long) dispDeviceBase) - base; size = dispDeviceRowBytes * dispDeviceRect[3] + offset + dispDeviceRect[0]; vbase = __ioremap(base, size, pgprot_val(pgprot_noncached_wc(__pgprot(0)))); - if (vbase == 0) + if (!vbase) return; logicalDisplayBase = vbase + offset; boot_text_mapped = 1; } -int btext_initialize(struct device_node *np) +static int btext_initialize(struct device_node *np) { unsigned int width, height, depth, pitch; unsigned long address = 0; @@ -270,7 +270,7 @@ static unsigned char * calc_base(int x, int y) unsigned char *base; base = logicalDisplayBase; - if (base == 0) + if (!base) base = dispDeviceBase; base += (x + dispDeviceRect[0]) * (dispDeviceDepth >> 3); base += (y + dispDeviceRect[1]) * dispDeviceRowBytes; @@ -281,7 +281,7 @@ static unsigned char * calc_base(int x, int y) void btext_update_display(unsigned long phys, int width, int height, int depth, int pitch) { - if (dispDeviceBase == 0) + if (!dispDeviceBase) return; /* check it's the same frame buffer (within 256MB) */ diff --git a/arch/powerpc/kernel/dma.c b/arch/powerpc/kernel/dma.c index da20569de9d4..155170d70324 100644 --- a/arch/powerpc/kernel/dma.c +++ b/arch/powerpc/kernel/dma.c @@ -235,8 +235,6 @@ static inline dma_addr_t dma_nommu_map_page(struct device *dev, enum dma_data_direction dir, unsigned long attrs) { - BUG_ON(dir == DMA_NONE); - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) __dma_sync_page(page, offset, size, dir); @@ -309,8 +307,6 @@ int dma_set_coherent_mask(struct device *dev, u64 mask) } EXPORT_SYMBOL(dma_set_coherent_mask); -#define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16) - int dma_set_mask(struct device *dev, u64 dma_mask) { if (ppc_md.dma_set_mask) @@ -361,7 +357,6 @@ EXPORT_SYMBOL_GPL(dma_get_required_mask); static int __init dma_init(void) { - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); #ifdef CONFIG_PCI dma_debug_add_bus(&pci_bus_type); #endif diff --git a/arch/powerpc/kernel/dt_cpu_ftrs.c b/arch/powerpc/kernel/dt_cpu_ftrs.c index c904477abaf3..4be1c0de9406 100644 --- a/arch/powerpc/kernel/dt_cpu_ftrs.c +++ b/arch/powerpc/kernel/dt_cpu_ftrs.c @@ -717,6 +717,7 @@ static __init void cpufeatures_cpu_quirks(void) if ((version & 0xffff0000) == 0x004e0000) { cur_cpu_spec->cpu_features &= ~(CPU_FTR_DAWR); cur_cpu_spec->cpu_features |= CPU_FTR_P9_TLBIE_BUG; + cur_cpu_spec->cpu_features |= CPU_FTR_P9_TIDR; } /* diff --git a/arch/powerpc/kernel/eeh.c b/arch/powerpc/kernel/eeh.c index bc640e4c5ca5..5746809cfaad 100644 --- a/arch/powerpc/kernel/eeh.c +++ b/arch/powerpc/kernel/eeh.c @@ -263,9 +263,8 @@ static size_t eeh_dump_dev_log(struct eeh_dev *edev, char *buf, size_t len) return n; } -static void *eeh_dump_pe_log(void *data, void *flag) +static void *eeh_dump_pe_log(struct eeh_pe *pe, void *flag) { - struct eeh_pe *pe = data; struct eeh_dev *edev, *tmp; size_t *plen = flag; @@ -542,8 +541,12 @@ int eeh_dev_check_failure(struct eeh_dev *edev) /* Frozen parent PE ? */ ret = eeh_ops->get_state(parent_pe, NULL); - if (ret > 0 && !eeh_state_active(ret)) + if (ret > 0 && !eeh_state_active(ret)) { pe = parent_pe; + pr_err("EEH: Failure of PHB#%x-PE#%x will be handled at parent PHB#%x-PE#%x.\n", + pe->phb->global_number, pe->addr, + pe->phb->global_number, parent_pe->addr); + } /* Next parent level */ parent_pe = parent_pe->parent; @@ -686,9 +689,9 @@ int eeh_pci_enable(struct eeh_pe *pe, int function) return rc; } -static void *eeh_disable_and_save_dev_state(void *data, void *userdata) +static void *eeh_disable_and_save_dev_state(struct eeh_dev *edev, + void *userdata) { - struct eeh_dev *edev = data; struct pci_dev *pdev = eeh_dev_to_pci_dev(edev); struct pci_dev *dev = userdata; @@ -714,9 +717,8 @@ static void *eeh_disable_and_save_dev_state(void *data, void *userdata) return NULL; } -static void *eeh_restore_dev_state(void *data, void *userdata) +static void *eeh_restore_dev_state(struct eeh_dev *edev, void *userdata) { - struct eeh_dev *edev = data; struct pci_dn *pdn = eeh_dev_to_pdn(edev); struct pci_dev *pdev = eeh_dev_to_pci_dev(edev); struct pci_dev *dev = userdata; @@ -856,11 +858,10 @@ int pcibios_set_pcie_reset_state(struct pci_dev *dev, enum pcie_reset_state stat * the indicated device and its children so that the bunch of the * devices could be reset properly. */ -static void *eeh_set_dev_freset(void *data, void *flag) +static void *eeh_set_dev_freset(struct eeh_dev *edev, void *flag) { struct pci_dev *dev; unsigned int *freset = (unsigned int *)flag; - struct eeh_dev *edev = (struct eeh_dev *)data; dev = eeh_dev_to_pci_dev(edev); if (dev) @@ -1775,18 +1776,6 @@ static int proc_eeh_show(struct seq_file *m, void *v) return 0; } -static int proc_eeh_open(struct inode *inode, struct file *file) -{ - return single_open(file, proc_eeh_show, NULL); -} - -static const struct file_operations proc_eeh_operations = { - .open = proc_eeh_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - #ifdef CONFIG_DEBUG_FS static int eeh_enable_dbgfs_set(void *data, u64 val) { @@ -1828,7 +1817,7 @@ DEFINE_SIMPLE_ATTRIBUTE(eeh_freeze_dbgfs_ops, eeh_freeze_dbgfs_get, static int __init eeh_init_proc(void) { if (machine_is(pseries) || machine_is(powernv)) { - proc_create("powerpc/eeh", 0, NULL, &proc_eeh_operations); + proc_create_single("powerpc/eeh", 0, NULL, proc_eeh_show); #ifdef CONFIG_DEBUG_FS debugfs_create_file("eeh_enable", 0600, powerpc_debugfs_root, NULL, diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_driver.c index b8a329f04814..67619b4b3f96 100644 --- a/arch/powerpc/kernel/eeh_driver.c +++ b/arch/powerpc/kernel/eeh_driver.c @@ -39,18 +39,82 @@ struct eeh_rmv_data { int removed; }; -/** - * eeh_pcid_name - Retrieve name of PCI device driver - * @pdev: PCI device - * - * This routine is used to retrieve the name of PCI device driver - * if that's valid. - */ -static inline const char *eeh_pcid_name(struct pci_dev *pdev) +static int eeh_result_priority(enum pci_ers_result result) +{ + switch (result) { + case PCI_ERS_RESULT_NONE: + return 1; + case PCI_ERS_RESULT_NO_AER_DRIVER: + return 2; + case PCI_ERS_RESULT_RECOVERED: + return 3; + case PCI_ERS_RESULT_CAN_RECOVER: + return 4; + case PCI_ERS_RESULT_DISCONNECT: + return 5; + case PCI_ERS_RESULT_NEED_RESET: + return 6; + default: + WARN_ONCE(1, "Unknown pci_ers_result value: %d\n", (int)result); + return 0; + } +}; + +const char *pci_ers_result_name(enum pci_ers_result result) +{ + switch (result) { + case PCI_ERS_RESULT_NONE: + return "none"; + case PCI_ERS_RESULT_CAN_RECOVER: + return "can recover"; + case PCI_ERS_RESULT_NEED_RESET: + return "need reset"; + case PCI_ERS_RESULT_DISCONNECT: + return "disconnect"; + case PCI_ERS_RESULT_RECOVERED: + return "recovered"; + case PCI_ERS_RESULT_NO_AER_DRIVER: + return "no AER driver"; + default: + WARN_ONCE(1, "Unknown result type: %d\n", (int)result); + return "unknown"; + } +}; + +static __printf(2, 3) void eeh_edev_info(const struct eeh_dev *edev, + const char *fmt, ...) +{ + struct va_format vaf; + va_list args; + + va_start(args, fmt); + + vaf.fmt = fmt; + vaf.va = &args; + + printk(KERN_INFO "EEH: PE#%x (PCI %s): %pV\n", edev->pe_config_addr, + edev->pdev ? dev_name(&edev->pdev->dev) : "none", &vaf); + + va_end(args); +} + +static enum pci_ers_result pci_ers_merge_result(enum pci_ers_result old, + enum pci_ers_result new) +{ + if (eeh_result_priority(new) > eeh_result_priority(old)) + return new; + return old; +} + +static bool eeh_dev_removed(struct eeh_dev *edev) { - if (pdev && pdev->dev.driver) - return pdev->dev.driver->name; - return ""; + return !edev || (edev->mode & EEH_DEV_REMOVED); +} + +static bool eeh_edev_actionable(struct eeh_dev *edev) +{ + return (edev->pdev && !eeh_dev_removed(edev) && + !eeh_pe_passed(edev->pe)); } /** @@ -98,22 +162,20 @@ static inline void eeh_pcid_put(struct pci_dev *pdev) * do real work because EEH should freeze DMA transfers for those PCI * devices encountering EEH errors, which includes MSI or MSI-X. */ -static void eeh_disable_irq(struct pci_dev *dev) +static void eeh_disable_irq(struct eeh_dev *edev) { - struct eeh_dev *edev = pci_dev_to_eeh_dev(dev); - /* Don't disable MSI and MSI-X interrupts. They are * effectively disabled by the DMA Stopped state * when an EEH error occurs. */ - if (dev->msi_enabled || dev->msix_enabled) + if (edev->pdev->msi_enabled || edev->pdev->msix_enabled) return; - if (!irq_has_action(dev->irq)) + if (!irq_has_action(edev->pdev->irq)) return; edev->mode |= EEH_DEV_IRQ_DISABLED; - disable_irq_nosync(dev->irq); + disable_irq_nosync(edev->pdev->irq); } /** @@ -123,10 +185,8 @@ static void eeh_disable_irq(struct pci_dev *dev) * This routine must be called to enable interrupt while failed * device could be resumed. */ -static void eeh_enable_irq(struct pci_dev *dev) +static void eeh_enable_irq(struct eeh_dev *edev) { - struct eeh_dev *edev = pci_dev_to_eeh_dev(dev); - if ((edev->mode) & EEH_DEV_IRQ_DISABLED) { edev->mode &= ~EEH_DEV_IRQ_DISABLED; /* @@ -149,23 +209,13 @@ static void eeh_enable_irq(struct pci_dev *dev) * * tglx */ - if (irqd_irq_disabled(irq_get_irq_data(dev->irq))) - enable_irq(dev->irq); + if (irqd_irq_disabled(irq_get_irq_data(edev->pdev->irq))) + enable_irq(edev->pdev->irq); } } -static bool eeh_dev_removed(struct eeh_dev *edev) -{ - /* EEH device removed ? */ - if (!edev || (edev->mode & EEH_DEV_REMOVED)) - return true; - - return false; -} - -static void *eeh_dev_save_state(void *data, void *userdata) +static void *eeh_dev_save_state(struct eeh_dev *edev, void *userdata) { - struct eeh_dev *edev = data; struct pci_dev *pdev; if (!edev) @@ -189,144 +239,155 @@ static void *eeh_dev_save_state(void *data, void *userdata) return NULL; } -/** - * eeh_report_error - Report pci error to each device driver - * @data: eeh device - * @userdata: return value - * - * Report an EEH error to each device driver, collect up and - * merge the device driver responses. Cumulative response - * passed back in "userdata". - */ -static void *eeh_report_error(void *data, void *userdata) +static void eeh_set_channel_state(struct eeh_pe *root, enum pci_channel_state s) { - struct eeh_dev *edev = (struct eeh_dev *)data; - struct pci_dev *dev = eeh_dev_to_pci_dev(edev); - enum pci_ers_result rc, *res = userdata; - struct pci_driver *driver; + struct eeh_pe *pe; + struct eeh_dev *edev, *tmp; - if (!dev || eeh_dev_removed(edev) || eeh_pe_passed(edev->pe)) - return NULL; + eeh_for_each_pe(root, pe) + eeh_pe_for_each_dev(pe, edev, tmp) + if (eeh_edev_actionable(edev)) + edev->pdev->error_state = s; +} - device_lock(&dev->dev); - dev->error_state = pci_channel_io_frozen; +static void eeh_set_irq_state(struct eeh_pe *root, bool enable) +{ + struct eeh_pe *pe; + struct eeh_dev *edev, *tmp; - driver = eeh_pcid_get(dev); - if (!driver) goto out_no_dev; + eeh_for_each_pe(root, pe) { + eeh_pe_for_each_dev(pe, edev, tmp) { + if (!eeh_edev_actionable(edev)) + continue; - eeh_disable_irq(dev); + if (!eeh_pcid_get(edev->pdev)) + continue; - if (!driver->err_handler || - !driver->err_handler->error_detected) - goto out; + if (enable) + eeh_enable_irq(edev); + else + eeh_disable_irq(edev); - rc = driver->err_handler->error_detected(dev, pci_channel_io_frozen); + eeh_pcid_put(edev->pdev); + } + } +} - /* A driver that needs a reset trumps all others */ - if (rc == PCI_ERS_RESULT_NEED_RESET) *res = rc; - if (*res == PCI_ERS_RESULT_NONE) *res = rc; +typedef enum pci_ers_result (*eeh_report_fn)(struct eeh_dev *, + struct pci_driver *); +static void eeh_pe_report_edev(struct eeh_dev *edev, eeh_report_fn fn, + enum pci_ers_result *result) +{ + struct pci_driver *driver; + enum pci_ers_result new_result; + + device_lock(&edev->pdev->dev); + if (eeh_edev_actionable(edev)) { + driver = eeh_pcid_get(edev->pdev); + + if (!driver) + eeh_edev_info(edev, "no driver"); + else if (!driver->err_handler) + eeh_edev_info(edev, "driver not EEH aware"); + else if (edev->mode & EEH_DEV_NO_HANDLER) + eeh_edev_info(edev, "driver bound too late"); + else { + new_result = fn(edev, driver); + eeh_edev_info(edev, "%s driver reports: '%s'", + driver->name, + pci_ers_result_name(new_result)); + if (result) + *result = pci_ers_merge_result(*result, + new_result); + } + if (driver) + eeh_pcid_put(edev->pdev); + } else { + eeh_edev_info(edev, "not actionable (%d,%d,%d)", !!edev->pdev, + !eeh_dev_removed(edev), !eeh_pe_passed(edev->pe)); + } + device_unlock(&edev->pdev->dev); +} - edev->in_error = true; - pci_uevent_ers(dev, PCI_ERS_RESULT_NONE); +static void eeh_pe_report(const char *name, struct eeh_pe *root, + eeh_report_fn fn, enum pci_ers_result *result) +{ + struct eeh_pe *pe; + struct eeh_dev *edev, *tmp; -out: - eeh_pcid_put(dev); -out_no_dev: - device_unlock(&dev->dev); - return NULL; + pr_info("EEH: Beginning: '%s'\n", name); + eeh_for_each_pe(root, pe) eeh_pe_for_each_dev(pe, edev, tmp) + eeh_pe_report_edev(edev, fn, result); + if (result) + pr_info("EEH: Finished:'%s' with aggregate recovery state:'%s'\n", + name, pci_ers_result_name(*result)); + else + pr_info("EEH: Finished:'%s'", name); } /** - * eeh_report_mmio_enabled - Tell drivers that MMIO has been enabled - * @data: eeh device - * @userdata: return value + * eeh_report_error - Report pci error to each device driver + * @edev: eeh device + * @driver: device's PCI driver * - * Tells each device driver that IO ports, MMIO and config space I/O - * are now enabled. Collects up and merges the device driver responses. - * Cumulative response passed back in "userdata". + * Report an EEH error to each device driver. */ -static void *eeh_report_mmio_enabled(void *data, void *userdata) +static enum pci_ers_result eeh_report_error(struct eeh_dev *edev, + struct pci_driver *driver) { - struct eeh_dev *edev = (struct eeh_dev *)data; - struct pci_dev *dev = eeh_dev_to_pci_dev(edev); - enum pci_ers_result rc, *res = userdata; - struct pci_driver *driver; - - if (!dev || eeh_dev_removed(edev) || eeh_pe_passed(edev->pe)) - return NULL; - - device_lock(&dev->dev); - driver = eeh_pcid_get(dev); - if (!driver) goto out_no_dev; + enum pci_ers_result rc; + struct pci_dev *dev = edev->pdev; - if (!driver->err_handler || - !driver->err_handler->mmio_enabled || - (edev->mode & EEH_DEV_NO_HANDLER)) - goto out; + if (!driver->err_handler->error_detected) + return PCI_ERS_RESULT_NONE; - rc = driver->err_handler->mmio_enabled(dev); + eeh_edev_info(edev, "Invoking %s->error_detected(IO frozen)", + driver->name); + rc = driver->err_handler->error_detected(dev, pci_channel_io_frozen); - /* A driver that needs a reset trumps all others */ - if (rc == PCI_ERS_RESULT_NEED_RESET) *res = rc; - if (*res == PCI_ERS_RESULT_NONE) *res = rc; + edev->in_error = true; + pci_uevent_ers(dev, PCI_ERS_RESULT_NONE); + return rc; +} -out: - eeh_pcid_put(dev); -out_no_dev: - device_unlock(&dev->dev); - return NULL; +/** + * eeh_report_mmio_enabled - Tell drivers that MMIO has been enabled + * @edev: eeh device + * @driver: device's PCI driver + * + * Tells each device driver that IO ports, MMIO and config space I/O + * are now enabled. + */ +static enum pci_ers_result eeh_report_mmio_enabled(struct eeh_dev *edev, + struct pci_driver *driver) +{ + if (!driver->err_handler->mmio_enabled) + return PCI_ERS_RESULT_NONE; + eeh_edev_info(edev, "Invoking %s->mmio_enabled()", driver->name); + return driver->err_handler->mmio_enabled(edev->pdev); } /** * eeh_report_reset - Tell device that slot has been reset - * @data: eeh device - * @userdata: return value + * @edev: eeh device + * @driver: device's PCI driver * * This routine must be called while EEH tries to reset particular * PCI device so that the associated PCI device driver could take * some actions, usually to save data the driver needs so that the * driver can work again while the device is recovered. */ -static void *eeh_report_reset(void *data, void *userdata) +static enum pci_ers_result eeh_report_reset(struct eeh_dev *edev, + struct pci_driver *driver) { - struct eeh_dev *edev = (struct eeh_dev *)data; - struct pci_dev *dev = eeh_dev_to_pci_dev(edev); - enum pci_ers_result rc, *res = userdata; - struct pci_driver *driver; - - if (!dev || eeh_dev_removed(edev) || eeh_pe_passed(edev->pe)) - return NULL; - - device_lock(&dev->dev); - dev->error_state = pci_channel_io_normal; - - driver = eeh_pcid_get(dev); - if (!driver) goto out_no_dev; - - eeh_enable_irq(dev); - - if (!driver->err_handler || - !driver->err_handler->slot_reset || - (edev->mode & EEH_DEV_NO_HANDLER) || - (!edev->in_error)) - goto out; - - rc = driver->err_handler->slot_reset(dev); - if ((*res == PCI_ERS_RESULT_NONE) || - (*res == PCI_ERS_RESULT_RECOVERED)) *res = rc; - if (*res == PCI_ERS_RESULT_DISCONNECT && - rc == PCI_ERS_RESULT_NEED_RESET) *res = rc; - -out: - eeh_pcid_put(dev); -out_no_dev: - device_unlock(&dev->dev); - return NULL; + if (!driver->err_handler->slot_reset || !edev->in_error) + return PCI_ERS_RESULT_NONE; + eeh_edev_info(edev, "Invoking %s->slot_reset()", driver->name); + return driver->err_handler->slot_reset(edev->pdev); } -static void *eeh_dev_restore_state(void *data, void *userdata) +static void *eeh_dev_restore_state(struct eeh_dev *edev, void *userdata) { - struct eeh_dev *edev = data; struct pci_dev *pdev; if (!edev) @@ -355,91 +416,53 @@ static void *eeh_dev_restore_state(void *data, void *userdata) /** * eeh_report_resume - Tell device to resume normal operations - * @data: eeh device - * @userdata: return value + * @edev: eeh device + * @driver: device's PCI driver * * This routine must be called to notify the device driver that it * could resume so that the device driver can do some initialization * to make the recovered device work again. */ -static void *eeh_report_resume(void *data, void *userdata) +static enum pci_ers_result eeh_report_resume(struct eeh_dev *edev, + struct pci_driver *driver) { - struct eeh_dev *edev = (struct eeh_dev *)data; - struct pci_dev *dev = eeh_dev_to_pci_dev(edev); - bool was_in_error; - struct pci_driver *driver; + if (!driver->err_handler->resume || !edev->in_error) + return PCI_ERS_RESULT_NONE; - if (!dev || eeh_dev_removed(edev) || eeh_pe_passed(edev->pe)) - return NULL; - - device_lock(&dev->dev); - dev->error_state = pci_channel_io_normal; - - driver = eeh_pcid_get(dev); - if (!driver) goto out_no_dev; - - was_in_error = edev->in_error; - edev->in_error = false; - eeh_enable_irq(dev); - - if (!driver->err_handler || - !driver->err_handler->resume || - (edev->mode & EEH_DEV_NO_HANDLER) || !was_in_error) { - edev->mode &= ~EEH_DEV_NO_HANDLER; - goto out; - } - - driver->err_handler->resume(dev); + eeh_edev_info(edev, "Invoking %s->resume()", driver->name); + driver->err_handler->resume(edev->pdev); - pci_uevent_ers(dev, PCI_ERS_RESULT_RECOVERED); -out: - eeh_pcid_put(dev); + pci_uevent_ers(edev->pdev, PCI_ERS_RESULT_RECOVERED); #ifdef CONFIG_PCI_IOV if (eeh_ops->notify_resume && eeh_dev_to_pdn(edev)) eeh_ops->notify_resume(eeh_dev_to_pdn(edev)); #endif -out_no_dev: - device_unlock(&dev->dev); - return NULL; + return PCI_ERS_RESULT_NONE; } /** * eeh_report_failure - Tell device driver that device is dead. - * @data: eeh device - * @userdata: return value + * @edev: eeh device + * @driver: device's PCI driver * * This informs the device driver that the device is permanently * dead, and that no further recovery attempts will be made on it. */ -static void *eeh_report_failure(void *data, void *userdata) +static enum pci_ers_result eeh_report_failure(struct eeh_dev *edev, + struct pci_driver *driver) { - struct eeh_dev *edev = (struct eeh_dev *)data; - struct pci_dev *dev = eeh_dev_to_pci_dev(edev); - struct pci_driver *driver; + enum pci_ers_result rc; - if (!dev || eeh_dev_removed(edev) || eeh_pe_passed(edev->pe)) - return NULL; - - device_lock(&dev->dev); - dev->error_state = pci_channel_io_perm_failure; - - driver = eeh_pcid_get(dev); - if (!driver) goto out_no_dev; + if (!driver->err_handler->error_detected) + return PCI_ERS_RESULT_NONE; - eeh_disable_irq(dev); + eeh_edev_info(edev, "Invoking %s->error_detected(permanent failure)", + driver->name); + rc = driver->err_handler->error_detected(edev->pdev, + pci_channel_io_perm_failure); - if (!driver->err_handler || - !driver->err_handler->error_detected) - goto out; - - driver->err_handler->error_detected(dev, pci_channel_io_perm_failure); - - pci_uevent_ers(dev, PCI_ERS_RESULT_DISCONNECT); -out: - eeh_pcid_put(dev); -out_no_dev: - device_unlock(&dev->dev); - return NULL; + pci_uevent_ers(edev->pdev, PCI_ERS_RESULT_DISCONNECT); + return rc; } static void *eeh_add_virt_device(void *data, void *userdata) @@ -458,9 +481,11 @@ static void *eeh_add_virt_device(void *data, void *userdata) driver = eeh_pcid_get(dev); if (driver) { - eeh_pcid_put(dev); - if (driver->err_handler) + if (driver->err_handler) { + eeh_pcid_put(dev); return NULL; + } + eeh_pcid_put(dev); } #ifdef CONFIG_PCI_IOV @@ -469,10 +494,9 @@ static void *eeh_add_virt_device(void *data, void *userdata) return NULL; } -static void *eeh_rmv_device(void *data, void *userdata) +static void *eeh_rmv_device(struct eeh_dev *edev, void *userdata) { struct pci_driver *driver; - struct eeh_dev *edev = (struct eeh_dev *)data; struct pci_dev *dev = eeh_dev_to_pci_dev(edev); struct eeh_rmv_data *rmv_data = (struct eeh_rmv_data *)userdata; int *removed = rmv_data ? &rmv_data->removed : NULL; @@ -497,17 +521,19 @@ static void *eeh_rmv_device(void *data, void *userdata) if (eeh_dev_removed(edev)) return NULL; - driver = eeh_pcid_get(dev); - if (driver) { - eeh_pcid_put(dev); - if (removed && - eeh_pe_passed(edev->pe)) - return NULL; - if (removed && - driver->err_handler && - driver->err_handler->error_detected && - driver->err_handler->slot_reset) + if (removed) { + if (eeh_pe_passed(edev->pe)) return NULL; + driver = eeh_pcid_get(dev); + if (driver) { + if (driver->err_handler && + driver->err_handler->error_detected && + driver->err_handler->slot_reset) { + eeh_pcid_put(dev); + return NULL; + } + eeh_pcid_put(dev); + } } /* Remove it from PCI subsystem */ @@ -542,9 +568,8 @@ static void *eeh_rmv_device(void *data, void *userdata) return NULL; } -static void *eeh_pe_detach_dev(void *data, void *userdata) +static void *eeh_pe_detach_dev(struct eeh_pe *pe, void *userdata) { - struct eeh_pe *pe = (struct eeh_pe *)data; struct eeh_dev *edev, *tmp; eeh_pe_for_each_dev(pe, edev, tmp) { @@ -565,9 +590,8 @@ static void *eeh_pe_detach_dev(void *data, void *userdata) * PE reset (for 3 times), we try to clear the frozen state * for 3 times as well. */ -static void *__eeh_clear_pe_frozen_state(void *data, void *flag) +static void *__eeh_clear_pe_frozen_state(struct eeh_pe *pe, void *flag) { - struct eeh_pe *pe = (struct eeh_pe *)data; bool clear_sw_state = *(bool *)flag; int i, rc = 1; @@ -762,6 +786,7 @@ void eeh_handle_normal_event(struct eeh_pe *pe) { struct pci_bus *bus; struct eeh_dev *edev, *tmp; + struct eeh_pe *tmp_pe; int rc = 0; enum pci_ers_result result = PCI_ERS_RESULT_NONE; struct eeh_rmv_data rmv_data = {LIST_HEAD_INIT(rmv_data.edev_list), 0}; @@ -778,14 +803,13 @@ void eeh_handle_normal_event(struct eeh_pe *pe) eeh_pe_update_time_stamp(pe); pe->freeze_count++; if (pe->freeze_count > eeh_max_freezes) { - pr_err("EEH: PHB#%x-PE#%x has failed %d times in the\n" - "last hour and has been permanently disabled.\n", + pr_err("EEH: PHB#%x-PE#%x has failed %d times in the last hour and has been permanently disabled.\n", pe->phb->global_number, pe->addr, pe->freeze_count); goto hard_fail; } - pr_warn("EEH: This PCI device has failed %d times in the last hour\n", - pe->freeze_count); + pr_warn("EEH: This PCI device has failed %d times in the last hour and will be permanently disabled after %d failures.\n", + pe->freeze_count, eeh_max_freezes); /* Walk the various device drivers attached to this slot through * a reset sequence, giving each an opportunity to do what it needs @@ -798,7 +822,10 @@ void eeh_handle_normal_event(struct eeh_pe *pe) * hotplug for this case. */ pr_info("EEH: Notify device drivers to shutdown\n"); - eeh_pe_dev_traverse(pe, eeh_report_error, &result); + eeh_set_channel_state(pe, pci_channel_io_frozen); + eeh_set_irq_state(pe, false); + eeh_pe_report("error_detected(IO frozen)", pe, eeh_report_error, + &result); if ((pe->type & EEH_PE_PHB) && result != PCI_ERS_RESULT_NONE && result != PCI_ERS_RESULT_NEED_RESET) @@ -845,7 +872,8 @@ void eeh_handle_normal_event(struct eeh_pe *pe) result = PCI_ERS_RESULT_NEED_RESET; } else { pr_info("EEH: Notify device drivers to resume I/O\n"); - eeh_pe_dev_traverse(pe, eeh_report_mmio_enabled, &result); + eeh_pe_report("mmio_enabled", pe, + eeh_report_mmio_enabled, &result); } } @@ -888,7 +916,9 @@ void eeh_handle_normal_event(struct eeh_pe *pe) pr_info("EEH: Notify device drivers " "the completion of reset\n"); result = PCI_ERS_RESULT_NONE; - eeh_pe_dev_traverse(pe, eeh_report_reset, &result); + eeh_set_channel_state(pe, pci_channel_io_normal); + eeh_set_irq_state(pe, true); + eeh_pe_report("slot_reset", pe, eeh_report_reset, &result); } /* All devices should claim they have recovered by now. */ @@ -909,8 +939,17 @@ void eeh_handle_normal_event(struct eeh_pe *pe) /* Tell all device drivers that they can resume operations */ pr_info("EEH: Notify device driver to resume\n"); - eeh_pe_dev_traverse(pe, eeh_report_resume, NULL); + eeh_set_channel_state(pe, pci_channel_io_normal); + eeh_set_irq_state(pe, true); + eeh_pe_report("resume", pe, eeh_report_resume, NULL); + eeh_for_each_pe(pe, tmp_pe) { + eeh_pe_for_each_dev(tmp_pe, edev, tmp) { + edev->mode &= ~EEH_DEV_NO_HANDLER; + edev->in_error = false; + } + } + pr_info("EEH: Recovery successful.\n"); goto final; hard_fail: @@ -926,7 +965,10 @@ hard_fail: eeh_slot_error_detail(pe, EEH_LOG_PERM); /* Notify all devices that they're about to go down. */ - eeh_pe_dev_traverse(pe, eeh_report_failure, NULL); + eeh_set_channel_state(pe, pci_channel_io_perm_failure); + eeh_set_irq_state(pe, false); + eeh_pe_report("error_detected(permanent failure)", pe, + eeh_report_failure, NULL); /* Mark the PE to be removed permanently */ eeh_pe_state_mark(pe, EEH_PE_REMOVED); @@ -1035,7 +1077,9 @@ void eeh_handle_special_event(void) /* Notify all devices to be down */ eeh_pe_state_clear(pe, EEH_PE_PRI_BUS); - eeh_pe_dev_traverse(pe, + eeh_set_channel_state(pe, pci_channel_io_perm_failure); + eeh_pe_report( + "error_detected(permanent failure)", pe, eeh_report_failure, NULL); bus = eeh_pe_bus_get(phb_pe); if (!bus) { diff --git a/arch/powerpc/kernel/eeh_pe.c b/arch/powerpc/kernel/eeh_pe.c index ee5a67d57aab..1b238ecc553e 100644 --- a/arch/powerpc/kernel/eeh_pe.c +++ b/arch/powerpc/kernel/eeh_pe.c @@ -142,8 +142,7 @@ struct eeh_pe *eeh_phb_pe_get(struct pci_controller *phb) * The function is used to retrieve the next PE in the * hierarchy PE tree. */ -static struct eeh_pe *eeh_pe_next(struct eeh_pe *pe, - struct eeh_pe *root) +struct eeh_pe *eeh_pe_next(struct eeh_pe *pe, struct eeh_pe *root) { struct list_head *next = pe->child_list.next; @@ -173,12 +172,12 @@ static struct eeh_pe *eeh_pe_next(struct eeh_pe *pe, * to be traversed. */ void *eeh_pe_traverse(struct eeh_pe *root, - eeh_traverse_func fn, void *flag) + eeh_pe_traverse_func fn, void *flag) { struct eeh_pe *pe; void *ret; - for (pe = root; pe; pe = eeh_pe_next(pe, root)) { + eeh_for_each_pe(root, pe) { ret = fn(pe, flag); if (ret) return ret; } @@ -196,7 +195,7 @@ void *eeh_pe_traverse(struct eeh_pe *root, * PE and its child PEs. */ void *eeh_pe_dev_traverse(struct eeh_pe *root, - eeh_traverse_func fn, void *flag) + eeh_edev_traverse_func fn, void *flag) { struct eeh_pe *pe; struct eeh_dev *edev, *tmp; @@ -209,7 +208,7 @@ void *eeh_pe_dev_traverse(struct eeh_pe *root, } /* Traverse root PE */ - for (pe = root; pe; pe = eeh_pe_next(pe, root)) { + eeh_for_each_pe(root, pe) { eeh_pe_for_each_dev(pe, edev, tmp) { ret = fn(edev, flag); if (ret) @@ -235,9 +234,8 @@ struct eeh_pe_get_flag { int config_addr; }; -static void *__eeh_pe_get(void *data, void *flag) +static void *__eeh_pe_get(struct eeh_pe *pe, void *flag) { - struct eeh_pe *pe = (struct eeh_pe *)data; struct eeh_pe_get_flag *tmp = (struct eeh_pe_get_flag *) flag; /* Unexpected PHB PE */ @@ -551,9 +549,8 @@ void eeh_pe_update_time_stamp(struct eeh_pe *pe) * PE. Also, the associated PCI devices will be put into IO frozen * state as well. */ -static void *__eeh_pe_state_mark(void *data, void *flag) +static void *__eeh_pe_state_mark(struct eeh_pe *pe, void *flag) { - struct eeh_pe *pe = (struct eeh_pe *)data; int state = *((int *)flag); struct eeh_dev *edev, *tmp; struct pci_dev *pdev; @@ -595,9 +592,8 @@ void eeh_pe_state_mark(struct eeh_pe *pe, int state) } EXPORT_SYMBOL_GPL(eeh_pe_state_mark); -static void *__eeh_pe_dev_mode_mark(void *data, void *flag) +static void *__eeh_pe_dev_mode_mark(struct eeh_dev *edev, void *flag) { - struct eeh_dev *edev = data; int mode = *((int *)flag); edev->mode |= mode; @@ -625,9 +621,8 @@ void eeh_pe_dev_mode_mark(struct eeh_pe *pe, int mode) * given PE. Besides, we also clear the check count of the PE * as well. */ -static void *__eeh_pe_state_clear(void *data, void *flag) +static void *__eeh_pe_state_clear(struct eeh_pe *pe, void *flag) { - struct eeh_pe *pe = (struct eeh_pe *)data; int state = *((int *)flag); struct eeh_dev *edev, *tmp; struct pci_dev *pdev; @@ -858,9 +853,8 @@ static void eeh_restore_device_bars(struct eeh_dev *edev) * the expansion ROM base address, the latency timer, and etc. * from the saved values in the device node. */ -static void *eeh_restore_one_device_bars(void *data, void *flag) +static void *eeh_restore_one_device_bars(struct eeh_dev *edev, void *flag) { - struct eeh_dev *edev = (struct eeh_dev *)data; struct pci_dn *pdn = eeh_dev_to_pdn(edev); /* Do special restore for bridges */ diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S index eb8d01bae8c6..973577f2141c 100644 --- a/arch/powerpc/kernel/entry_32.S +++ b/arch/powerpc/kernel/entry_32.S @@ -365,6 +365,13 @@ syscall_dotrace_cont: blrl /* Call handler */ .globl ret_from_syscall ret_from_syscall: +#ifdef CONFIG_DEBUG_RSEQ + /* Check whether the syscall is issued inside a restartable sequence */ + stw r3,GPR3(r1) + addi r3,r1,STACK_FRAME_OVERHEAD + bl rseq_syscall + lwz r3,GPR3(r1) +#endif mr r6,r3 CURRENT_THREAD_INFO(r12, r1) /* disable interrupts so current_thread_info()->flags can't change */ diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S index 51695608c68b..729e9ef4d3bb 100644 --- a/arch/powerpc/kernel/entry_64.S +++ b/arch/powerpc/kernel/entry_64.S @@ -36,6 +36,7 @@ #include #include #include +#include #include #ifdef CONFIG_PPC_BOOK3S #include @@ -178,12 +179,29 @@ system_call: /* label this so stack traces look sane */ clrldi r8,r8,32 15: slwi r0,r0,4 + + barrier_nospec_asm + /* + * Prevent the load of the handler below (based on the user-passed + * system call number) being speculatively executed until the test + * against NR_syscalls and branch to .Lsyscall_enosys above has + * committed. + */ + ldx r12,r11,r0 /* Fetch system call handler [ptr] */ mtctr r12 bctrl /* Call handler */ .Lsyscall_exit: std r3,RESULT(r1) + +#ifdef CONFIG_DEBUG_RSEQ + /* Check whether the syscall is issued inside a restartable sequence */ + addi r3,r1,STACK_FRAME_OVERHEAD + bl rseq_syscall + ld r3,RESULT(r1) +#endif + CURRENT_THREAD_INFO(r12, r1) ld r8,_MSR(r1) @@ -596,6 +614,7 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_1T_SEGMENT) * actually hit this code path. */ + isync slbie r6 slbie r6 /* Workaround POWER5 < DD2.1 issue */ slbmte r7,r0 diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index f283958129f2..285c6465324a 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -1501,6 +1501,7 @@ masked_##_H##interrupt: \ xori r10,r10,MSR_EE; /* clear MSR_EE */ \ mtspr SPRN_##_H##SRR1,r10; \ 2: mtcrf 0x80,r9; \ + std r1,PACAR1(r13); \ ld r9,PACA_EXGEN+EX_R9(r13); \ ld r10,PACA_EXGEN+EX_R10(r13); \ ld r11,PACA_EXGEN+EX_R11(r13); \ diff --git a/arch/powerpc/kernel/fadump.c b/arch/powerpc/kernel/fadump.c index 3c2c2688918f..07e8396d472b 100644 --- a/arch/powerpc/kernel/fadump.c +++ b/arch/powerpc/kernel/fadump.c @@ -335,6 +335,26 @@ static unsigned long get_fadump_area_size(void) return size; } +static void __init fadump_reserve_crash_area(unsigned long base, + unsigned long size) +{ + struct memblock_region *reg; + unsigned long mstart, mend, msize; + + for_each_memblock(memory, reg) { + mstart = max_t(unsigned long, base, reg->base); + mend = reg->base + reg->size; + mend = min(base + size, mend); + + if (mstart < mend) { + msize = mend - mstart; + memblock_reserve(mstart, msize); + pr_info("Reserved %ldMB of memory at %#016lx for saving crash dump\n", + (msize >> 20), mstart); + } + } +} + int __init fadump_reserve_mem(void) { unsigned long base, size, memory_boundary; @@ -380,7 +400,16 @@ int __init fadump_reserve_mem(void) memory_boundary = memblock_end_of_DRAM(); if (fw_dump.dump_active) { - printk(KERN_INFO "Firmware-assisted dump is active.\n"); + pr_info("Firmware-assisted dump is active.\n"); + +#ifdef CONFIG_HUGETLB_PAGE + /* + * FADump capture kernel doesn't care much about hugepages. + * In fact, handling hugepages in capture kernel is asking for + * trouble. So, disable HugeTLB support when fadump is active. + */ + hugetlb_disabled = true; +#endif /* * If last boot has crashed then reserve all the memory * above boot_memory_size so that we don't touch it until @@ -389,11 +418,7 @@ int __init fadump_reserve_mem(void) */ base = fw_dump.boot_memory_size; size = memory_boundary - base; - memblock_reserve(base, size); - printk(KERN_INFO "Reserved %ldMB of memory at %ldMB " - "for saving crash dump\n", - (unsigned long)(size >> 20), - (unsigned long)(base >> 20)); + fadump_reserve_crash_area(base, size); fw_dump.fadumphdr_addr = be64_to_cpu(fdm_active->rmr_region.destination_address) + @@ -1155,6 +1180,9 @@ void fadump_cleanup(void) init_fadump_mem_struct(&fdm, be64_to_cpu(fdm_active->cpu_state_data.destination_address)); fadump_invalidate_dump(&fdm); + } else if (fw_dump.dump_registered) { + /* Un-register Firmware-assisted dump if it was registered. */ + fadump_unregister_dump(&fdm); } } diff --git a/arch/powerpc/kernel/head_8xx.S b/arch/powerpc/kernel/head_8xx.S index d8670a37d70c..6cab07e76732 100644 --- a/arch/powerpc/kernel/head_8xx.S +++ b/arch/powerpc/kernel/head_8xx.S @@ -913,7 +913,7 @@ start_here: tovirt(r6,r6) lis r5, abatron_pteptrs@h ori r5, r5, abatron_pteptrs@l - stw r5, 0xf0(r0) /* Must match your Abatron config file */ + stw r5, 0xf0(0) /* Must match your Abatron config file */ tophys(r5,r5) stw r6, 0(r5) diff --git a/arch/powerpc/kernel/hw_breakpoint.c b/arch/powerpc/kernel/hw_breakpoint.c index 4c1012b80d3b..80547dad37da 100644 --- a/arch/powerpc/kernel/hw_breakpoint.c +++ b/arch/powerpc/kernel/hw_breakpoint.c @@ -178,8 +178,8 @@ int arch_validate_hwbkpt_settings(struct perf_event *bp) if (cpu_has_feature(CPU_FTR_DAWR)) { length_max = 512 ; /* 64 doublewords */ /* DAWR region can't cross 512 boundary */ - if ((bp->attr.bp_addr >> 10) != - ((bp->attr.bp_addr + bp->attr.bp_len - 1) >> 10)) + if ((bp->attr.bp_addr >> 9) != + ((bp->attr.bp_addr + bp->attr.bp_len - 1) >> 9)) return -EINVAL; } if (info->len > diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c index 061aa0f47bb1..0682fef1f385 100644 --- a/arch/powerpc/kernel/irq.c +++ b/arch/powerpc/kernel/irq.c @@ -89,7 +89,7 @@ atomic_t ppc_n_lost_interrupts; #ifdef CONFIG_TAU_INT extern int tau_initialized; -extern int tau_interrupts(int); +u32 tau_interrupts(unsigned long cpu); #endif #endif /* CONFIG_PPC32 */ @@ -508,6 +508,11 @@ int arch_show_interrupts(struct seq_file *p, int prec) seq_printf(p, "%10u ", per_cpu(irq_stat, j).timer_irqs_event); seq_printf(p, " Local timer interrupts for timer event device\n"); + seq_printf(p, "%*s: ", prec, "BCT"); + for_each_online_cpu(j) + seq_printf(p, "%10u ", per_cpu(irq_stat, j).broadcast_irqs_event); + seq_printf(p, " Broadcast timer interrupts for timer event device\n"); + seq_printf(p, "%*s: ", prec, "LOC"); for_each_online_cpu(j) seq_printf(p, "%10u ", per_cpu(irq_stat, j).timer_irqs_others); @@ -567,6 +572,7 @@ u64 arch_irq_stat_cpu(unsigned int cpu) { u64 sum = per_cpu(irq_stat, cpu).timer_irqs_event; + sum += per_cpu(irq_stat, cpu).broadcast_irqs_event; sum += per_cpu(irq_stat, cpu).pmu_irqs; sum += per_cpu(irq_stat, cpu).mce_exceptions; sum += per_cpu(irq_stat, cpu).spurious_irqs; diff --git a/arch/powerpc/kernel/kvm.c b/arch/powerpc/kernel/kvm.c index 9ad37f827a97..683b5b3805bd 100644 --- a/arch/powerpc/kernel/kvm.c +++ b/arch/powerpc/kernel/kvm.c @@ -25,6 +25,7 @@ #include #include #include +#include #include #include @@ -672,14 +673,13 @@ static void kvm_use_magic_page(void) { u32 *p; u32 *start, *end; - u32 tmp; u32 features; /* Tell the host to map the magic page to -4096 on all CPUs */ on_each_cpu(kvm_map_magic_page, &features, 1); /* Quick self-test to see if the mapping works */ - if (__get_user(tmp, (u32*)KVM_MAGIC_PAGE)) { + if (!fault_in_pages_readable((const char *)KVM_MAGIC_PAGE, sizeof(u32))) { kvm_patching_worked = false; return; } diff --git a/arch/powerpc/kernel/machine_kexec.c b/arch/powerpc/kernel/machine_kexec.c index 2694d078741d..936c7e2d421e 100644 --- a/arch/powerpc/kernel/machine_kexec.c +++ b/arch/powerpc/kernel/machine_kexec.c @@ -98,12 +98,14 @@ void machine_kexec(struct kimage *image) int save_ftrace_enabled; save_ftrace_enabled = __ftrace_enabled_save(); + this_cpu_disable_ftrace(); if (ppc_md.machine_kexec) ppc_md.machine_kexec(image); else default_machine_kexec(image); + this_cpu_enable_ftrace(); __ftrace_enabled_restore(save_ftrace_enabled); /* Fall back to normal restart if we're still alive. */ diff --git a/arch/powerpc/kernel/machine_kexec_64.c b/arch/powerpc/kernel/machine_kexec_64.c index 1044bf15d5ed..a0f6f45005bd 100644 --- a/arch/powerpc/kernel/machine_kexec_64.c +++ b/arch/powerpc/kernel/machine_kexec_64.c @@ -231,16 +231,16 @@ static void kexec_prepare_cpus(void) /* we are sure every CPU has IRQs off at this point */ kexec_all_irq_disabled = 1; - /* after we tell the others to go down */ - if (ppc_md.kexec_cpu_down) - ppc_md.kexec_cpu_down(0, 0); - /* * Before removing MMU mappings make sure all CPUs have entered real * mode: */ kexec_prepare_cpus_wait(KEXEC_STATE_REAL_MODE); + /* after we tell the others to go down */ + if (ppc_md.kexec_cpu_down) + ppc_md.kexec_cpu_down(0, 0); + put_cpu(); } diff --git a/arch/powerpc/kernel/misc.S b/arch/powerpc/kernel/misc.S index 384357cb8bc0..0b196cdcd15d 100644 --- a/arch/powerpc/kernel/misc.S +++ b/arch/powerpc/kernel/misc.S @@ -25,23 +25,12 @@ /* * Returns (address we are running at) - (address we were linked at) * for use before the text and data are mapped to KERNELBASE. - */ - -_GLOBAL(reloc_offset) - mflr r0 - bl 1f -1: mflr r3 - PPC_LL r4,(2f-1b)(r3) - subf r3,r4,r3 - mtlr r0 - blr - .align 3 -2: PPC_LONG 1b - -/* * add_reloc_offset(x) returns x + reloc_offset(). */ + +_GLOBAL(reloc_offset) + li r3, 0 _GLOBAL(add_reloc_offset) mflr r0 bl 1f @@ -60,6 +49,10 @@ _GLOBAL(setjmp) PPC_STL r0,0(r3) PPC_STL r1,SZL(r3) PPC_STL r2,2*SZL(r3) +#ifdef CONFIG_PPC32 + mfcr r12 + stmw r12, 3*SZL(r3) +#else mfcr r0 PPC_STL r0,3*SZL(r3) PPC_STL r13,4*SZL(r3) @@ -81,14 +74,16 @@ _GLOBAL(setjmp) PPC_STL r29,20*SZL(r3) PPC_STL r30,21*SZL(r3) PPC_STL r31,22*SZL(r3) +#endif li r3,0 blr _GLOBAL(longjmp) - PPC_LCMPI r4,0 - bne 1f - li r4,1 -1: PPC_LL r13,4*SZL(r3) +#ifdef CONFIG_PPC32 + lmw r12, 3*SZL(r3) + mtcrf 0x38, r12 +#else + PPC_LL r13,4*SZL(r3) PPC_LL r14,5*SZL(r3) PPC_LL r15,6*SZL(r3) PPC_LL r16,7*SZL(r3) @@ -109,11 +104,14 @@ _GLOBAL(longjmp) PPC_LL r31,22*SZL(r3) PPC_LL r0,3*SZL(r3) mtcrf 0x38,r0 +#endif PPC_LL r0,0(r3) PPC_LL r1,SZL(r3) PPC_LL r2,2*SZL(r3) mtlr r0 - mr r3,r4 + mr. r3, r4 + bnelr + li r3, 1 blr _GLOBAL(current_stack_pointer) diff --git a/arch/powerpc/kernel/module.c b/arch/powerpc/kernel/module.c index 3f7ba0f5bf29..1b3c6835e730 100644 --- a/arch/powerpc/kernel/module.c +++ b/arch/powerpc/kernel/module.c @@ -72,6 +72,12 @@ int module_finalize(const Elf_Ehdr *hdr, do_feature_fixups(powerpc_firmware_features, (void *)sect->sh_addr, (void *)sect->sh_addr + sect->sh_size); + + sect = find_section(hdr, sechdrs, "__spec_barrier_fixup"); + if (sect != NULL) + do_barrier_nospec_fixups_range(barrier_nospec_enabled, + (void *)sect->sh_addr, + (void *)sect->sh_addr + sect->sh_size); #endif sect = find_section(hdr, sechdrs, "__lwsync_fixup"); diff --git a/arch/powerpc/kernel/module_32.c b/arch/powerpc/kernel/module_32.c index 5a7a78f12562..88d83771f462 100644 --- a/arch/powerpc/kernel/module_32.c +++ b/arch/powerpc/kernel/module_32.c @@ -109,12 +109,12 @@ static unsigned long get_plt_size(const Elf32_Ehdr *hdr, for (i = 1; i < hdr->e_shnum; i++) { /* If it's called *.init*, and we're not init, we're not interested */ - if ((strstr(secstrings + sechdrs[i].sh_name, ".init") != 0) + if ((strstr(secstrings + sechdrs[i].sh_name, ".init") != NULL) != is_init) continue; /* We don't want to look at debug sections. */ - if (strstr(secstrings + sechdrs[i].sh_name, ".debug") != 0) + if (strstr(secstrings + sechdrs[i].sh_name, ".debug")) continue; if (sechdrs[i].sh_type == SHT_RELA) { diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c index a2636c250b7b..b8d61e019d06 100644 --- a/arch/powerpc/kernel/module_64.c +++ b/arch/powerpc/kernel/module_64.c @@ -280,6 +280,10 @@ static unsigned long get_stubs_size(const Elf64_Ehdr *hdr, #ifdef CONFIG_DYNAMIC_FTRACE /* make the trampoline to the ftrace_caller */ relocs++; +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS + /* an additional one for ftrace_regs_caller */ + relocs++; +#endif #endif pr_debug("Looks like a total of %lu stubs, max\n", relocs); @@ -462,9 +466,12 @@ static unsigned long stub_for_addr(const Elf64_Shdr *sechdrs, return (unsigned long)&stubs[i]; } -#ifdef CC_USING_MPROFILE_KERNEL -static bool is_early_mcount_callsite(u32 *instruction) +#ifdef CONFIG_MPROFILE_KERNEL +static bool is_mprofile_mcount_callsite(const char *name, u32 *instruction) { + if (strcmp("_mcount", name)) + return false; + /* * Check if this is one of the -mprofile-kernel sequences. */ @@ -496,8 +503,7 @@ static void squash_toc_save_inst(const char *name, unsigned long addr) #else static void squash_toc_save_inst(const char *name, unsigned long addr) { } -/* without -mprofile-kernel, mcount calls are never early */ -static bool is_early_mcount_callsite(u32 *instruction) +static bool is_mprofile_mcount_callsite(const char *name, u32 *instruction) { return false; } @@ -505,11 +511,11 @@ static bool is_early_mcount_callsite(u32 *instruction) /* We expect a noop next: if it is, replace it with instruction to restore r2. */ -static int restore_r2(u32 *instruction, struct module *me) +static int restore_r2(const char *name, u32 *instruction, struct module *me) { u32 *prev_insn = instruction - 1; - if (is_early_mcount_callsite(prev_insn)) + if (is_mprofile_mcount_callsite(name, prev_insn)) return 1; /* @@ -650,7 +656,8 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, value = stub_for_addr(sechdrs, value, me); if (!value) return -ENOENT; - if (!restore_r2((u32 *)location + 1, me)) + if (!restore_r2(strtab + sym->st_name, + (u32 *)location + 1, me)) return -ENOEXEC; squash_toc_save_inst(strtab + sym->st_name, value); @@ -746,7 +753,7 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, #ifdef CONFIG_DYNAMIC_FTRACE -#ifdef CC_USING_MPROFILE_KERNEL +#ifdef CONFIG_MPROFILE_KERNEL #define PACATOC offsetof(struct paca_struct, kernel_toc) @@ -762,7 +769,8 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, * via the paca (in r13). The target (ftrace_caller()) is responsible for * saving and restoring the toc before returning. */ -static unsigned long create_ftrace_stub(const Elf64_Shdr *sechdrs, struct module *me) +static unsigned long create_ftrace_stub(const Elf64_Shdr *sechdrs, + struct module *me, unsigned long addr) { struct ppc64_stub_entry *entry; unsigned int i, num_stubs; @@ -789,9 +797,10 @@ static unsigned long create_ftrace_stub(const Elf64_Shdr *sechdrs, struct module memcpy(entry->jump, stub_insns, sizeof(stub_insns)); /* Stub uses address relative to kernel toc (from the paca) */ - reladdr = (unsigned long)ftrace_caller - kernel_toc_addr(); + reladdr = addr - kernel_toc_addr(); if (reladdr > 0x7FFFFFFF || reladdr < -(0x80000000L)) { - pr_err("%s: Address of ftrace_caller out of range of kernel_toc.\n", me->name); + pr_err("%s: Address of %ps out of range of kernel_toc.\n", + me->name, (void *)addr); return 0; } @@ -799,22 +808,29 @@ static unsigned long create_ftrace_stub(const Elf64_Shdr *sechdrs, struct module entry->jump[2] |= PPC_LO(reladdr); /* Eventhough we don't use funcdata in the stub, it's needed elsewhere. */ - entry->funcdata = func_desc((unsigned long)ftrace_caller); + entry->funcdata = func_desc(addr); entry->magic = STUB_MAGIC; return (unsigned long)entry; } #else -static unsigned long create_ftrace_stub(const Elf64_Shdr *sechdrs, struct module *me) +static unsigned long create_ftrace_stub(const Elf64_Shdr *sechdrs, + struct module *me, unsigned long addr) { - return stub_for_addr(sechdrs, (unsigned long)ftrace_caller, me); + return stub_for_addr(sechdrs, addr, me); } #endif int module_finalize_ftrace(struct module *mod, const Elf_Shdr *sechdrs) { - mod->arch.toc = my_r2(sechdrs, mod); - mod->arch.tramp = create_ftrace_stub(sechdrs, mod); + mod->arch.tramp = create_ftrace_stub(sechdrs, mod, + (unsigned long)ftrace_caller); +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS + mod->arch.tramp_regs = create_ftrace_stub(sechdrs, mod, + (unsigned long)ftrace_regs_caller); + if (!mod->arch.tramp_regs) + return -ENOENT; +#endif if (!mod->arch.tramp) return -ENOENT; diff --git a/arch/powerpc/kernel/nvram_64.c b/arch/powerpc/kernel/nvram_64.c index ba681dac7b46..22e9d281324d 100644 --- a/arch/powerpc/kernel/nvram_64.c +++ b/arch/powerpc/kernel/nvram_64.c @@ -1030,7 +1030,7 @@ loff_t __init nvram_create_partition(const char *name, int sig, return -ENOSPC; /* Create our OS partition */ - new_part = kmalloc(sizeof(*new_part), GFP_KERNEL); + new_part = kzalloc(sizeof(*new_part), GFP_KERNEL); if (!new_part) { pr_err("%s: kmalloc failed\n", __func__); return -ENOMEM; @@ -1039,7 +1039,7 @@ loff_t __init nvram_create_partition(const char *name, int sig, new_part->index = free_part->index; new_part->header.signature = sig; new_part->header.length = size; - strncpy(new_part->header.name, name, 12); + memcpy(new_part->header.name, name, strnlen(name, sizeof(new_part->header.name))); new_part->header.checksum = nvram_checksum(&new_part->header); rc = nvram_write_header(new_part); diff --git a/arch/powerpc/kernel/pci_32.c b/arch/powerpc/kernel/pci_32.c index 85ad2f78b889..4f861055a852 100644 --- a/arch/powerpc/kernel/pci_32.c +++ b/arch/powerpc/kernel/pci_32.c @@ -11,11 +11,13 @@ #include #include #include +#include #include #include #include #include #include +#include #include #include @@ -283,7 +285,11 @@ pci_bus_to_hose(int bus) * Note that the returned IO or memory base is a physical address */ -long sys_pciconfig_iobase(long which, unsigned long bus, unsigned long devfn) +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpragmas" +#pragma GCC diagnostic ignored "-Wattribute-alias" +SYSCALL_DEFINE3(pciconfig_iobase, long, which, + unsigned long, bus, unsigned long, devfn) { struct pci_controller* hose; long result = -EOPNOTSUPP; @@ -307,5 +313,4 @@ long sys_pciconfig_iobase(long which, unsigned long bus, unsigned long devfn) return result; } - - +#pragma GCC diagnostic pop diff --git a/arch/powerpc/kernel/pci_64.c b/arch/powerpc/kernel/pci_64.c index 15ce0306b092..812171c09f42 100644 --- a/arch/powerpc/kernel/pci_64.c +++ b/arch/powerpc/kernel/pci_64.c @@ -203,8 +203,11 @@ void pcibios_setup_phb_io_space(struct pci_controller *hose) #define IOBASE_ISA_IO 3 #define IOBASE_ISA_MEM 4 -long sys_pciconfig_iobase(long which, unsigned long in_bus, - unsigned long in_devfn) +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpragmas" +#pragma GCC diagnostic ignored "-Wattribute-alias" +SYSCALL_DEFINE3(pciconfig_iobase, long, which, unsigned long, in_bus, + unsigned long, in_devfn) { struct pci_controller* hose; struct pci_bus *tmp_bus, *bus = NULL; @@ -256,6 +259,7 @@ long sys_pciconfig_iobase(long which, unsigned long in_bus, return -EOPNOTSUPP; } +#pragma GCC diagnostic pop #ifdef CONFIG_NUMA int pcibus_to_node(struct pci_bus *bus) diff --git a/arch/powerpc/kernel/ppc_save_regs.S b/arch/powerpc/kernel/ppc_save_regs.S index 1b1787d52896..8afbe213d729 100644 --- a/arch/powerpc/kernel/ppc_save_regs.S +++ b/arch/powerpc/kernel/ppc_save_regs.S @@ -25,6 +25,9 @@ */ _GLOBAL(ppc_save_regs) PPC_STL r0,0*SZL(r3) +#ifdef CONFIG_PPC32 + stmw r2, 2*SZL(r3) +#else PPC_STL r2,2*SZL(r3) PPC_STL r3,3*SZL(r3) PPC_STL r4,4*SZL(r3) @@ -55,6 +58,7 @@ _GLOBAL(ppc_save_regs) PPC_STL r29,29*SZL(r3) PPC_STL r30,30*SZL(r3) PPC_STL r31,31*SZL(r3) +#endif /* go up one stack frame for SP */ PPC_LL r4,0(r1) PPC_STL r4,1*SZL(r3) diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c index 1237f13fed51..9ef4aea9fffe 100644 --- a/arch/powerpc/kernel/process.c +++ b/arch/powerpc/kernel/process.c @@ -154,6 +154,7 @@ unsigned long msr_check_and_set(unsigned long bits) return newmsr; } +EXPORT_SYMBOL_GPL(msr_check_and_set); void __msr_check_and_clear(unsigned long bits) { @@ -632,6 +633,7 @@ void do_break (struct pt_regs *regs, unsigned long address, hw_breakpoint_disable(); /* Deliver the signal to userspace */ + clear_siginfo(&info); info.si_signo = SIGTRAP; info.si_errno = 0; info.si_code = TRAP_HWBKPT; @@ -845,10 +847,6 @@ bool ppc_breakpoint_available(void) } EXPORT_SYMBOL_GPL(ppc_breakpoint_available); -#ifdef CONFIG_PPC64 -DEFINE_PER_CPU(struct cpu_usage, cpu_usage_array); -#endif - static inline bool hw_brk_match(struct arch_hw_breakpoint *a, struct arch_hw_breakpoint *b) { @@ -1154,7 +1152,7 @@ static inline void restore_sprs(struct thread_struct *old_thread, mtspr(SPRN_TAR, new_thread->tar); } - if (cpu_has_feature(CPU_FTR_ARCH_300) && + if (cpu_has_feature(CPU_FTR_P9_TIDR) && old_thread->tidr != new_thread->tidr) mtspr(SPRN_TIDR, new_thread->tidr); #endif @@ -1181,20 +1179,6 @@ struct task_struct *__switch_to(struct task_struct *prev, WARN_ON(!irqs_disabled()); -#ifdef CONFIG_PPC64 - /* - * Collect processor utilization data per process - */ - if (firmware_has_feature(FW_FEATURE_SPLPAR)) { - struct cpu_usage *cu = this_cpu_ptr(&cpu_usage_array); - long unsigned start_tb, current_tb; - start_tb = old_thread->start_tb; - cu->current_tb = current_tb = mfspr(SPRN_PURR); - old_thread->accum_tb += (current_tb - start_tb); - new_thread->start_tb = current_tb; - } -#endif /* CONFIG_PPC64 */ - #ifdef CONFIG_PPC_BOOK3S_64 batch = this_cpu_ptr(&ppc64_tlb_batch); if (batch->active) { @@ -1437,7 +1421,7 @@ void show_regs(struct pt_regs * regs) pr_cont("DAR: "REG" DSISR: %08lx ", regs->dar, regs->dsisr); #endif #ifdef CONFIG_PPC64 - pr_cont("SOFTE: %ld ", regs->softe); + pr_cont("IRQMASK: %lx ", regs->softe); #endif #ifdef CONFIG_PPC_TRANSACTIONAL_MEM if (MSR_TM_ACTIVE(regs->msr)) @@ -1496,104 +1480,42 @@ int set_thread_uses_vas(void) } #ifdef CONFIG_PPC64 -static DEFINE_SPINLOCK(vas_thread_id_lock); -static DEFINE_IDA(vas_thread_ida); - -/* - * We need to assign a unique thread id to each thread in a process. +/** + * Assign a TIDR (thread ID) for task @t and set it in the thread + * structure. For now, we only support setting TIDR for 'current' task. * - * This thread id, referred to as TIDR, and separate from the Linux's tgid, - * is intended to be used to direct an ASB_Notify from the hardware to the - * thread, when a suitable event occurs in the system. + * Since the TID value is a truncated form of it PID, it is possible + * (but unlikely) for 2 threads to have the same TID. In the unlikely event + * that 2 threads share the same TID and are waiting, one of the following + * cases will happen: * - * One such event is a "paste" instruction in the context of Fast Thread - * Wakeup (aka Core-to-core wake up in the Virtual Accelerator Switchboard - * (VAS) in POWER9. + * 1. The correct thread is running, the wrong thread is not + * In this situation, the correct thread is woken and proceeds to pass it's + * condition check. * - * To get a unique TIDR per process we could simply reuse task_pid_nr() but - * the problem is that task_pid_nr() is not yet available copy_thread() is - * called. Fixing that would require changing more intrusive arch-neutral - * code in code path in copy_process()?. + * 2. Neither threads are running + * In this situation, neither thread will be woken. When scheduled, the waiting + * threads will execute either a wait, which will return immediately, followed + * by a condition check, which will pass for the correct thread and fail + * for the wrong thread, or they will execute the condition check immediately. * - * Further, to assign unique TIDRs within each process, we need an atomic - * field (or an IDR) in task_struct, which again intrudes into the arch- - * neutral code. So try to assign globally unique TIDRs for now. + * 3. The wrong thread is running, the correct thread is not + * The wrong thread will be woken, but will fail it's condition check and + * re-execute wait. The correct thread, when scheduled, will execute either + * it's condition check (which will pass), or wait, which returns immediately + * when called the first time after the thread is scheduled, followed by it's + * condition check (which will pass). * - * NOTE: TIDR 0 indicates that the thread does not need a TIDR value. - * For now, only threads that expect to be notified by the VAS - * hardware need a TIDR value and we assign values > 0 for those. - */ -#define MAX_THREAD_CONTEXT ((1 << 16) - 1) -static int assign_thread_tidr(void) -{ - int index; - int err; - unsigned long flags; - -again: - if (!ida_pre_get(&vas_thread_ida, GFP_KERNEL)) - return -ENOMEM; - - spin_lock_irqsave(&vas_thread_id_lock, flags); - err = ida_get_new_above(&vas_thread_ida, 1, &index); - spin_unlock_irqrestore(&vas_thread_id_lock, flags); - - if (err == -EAGAIN) - goto again; - else if (err) - return err; - - if (index > MAX_THREAD_CONTEXT) { - spin_lock_irqsave(&vas_thread_id_lock, flags); - ida_remove(&vas_thread_ida, index); - spin_unlock_irqrestore(&vas_thread_id_lock, flags); - return -ENOMEM; - } - - return index; -} - -static void free_thread_tidr(int id) -{ - unsigned long flags; - - spin_lock_irqsave(&vas_thread_id_lock, flags); - ida_remove(&vas_thread_ida, id); - spin_unlock_irqrestore(&vas_thread_id_lock, flags); -} - -/* - * Clear any TIDR value assigned to this thread. - */ -void clear_thread_tidr(struct task_struct *t) -{ - if (!t->thread.tidr) - return; - - if (!cpu_has_feature(CPU_FTR_ARCH_300)) { - WARN_ON_ONCE(1); - return; - } - - mtspr(SPRN_TIDR, 0); - free_thread_tidr(t->thread.tidr); - t->thread.tidr = 0; -} - -void arch_release_task_struct(struct task_struct *t) -{ - clear_thread_tidr(t); -} - -/* - * Assign a unique TIDR (thread id) for task @t and set it in the thread - * structure. For now, we only support setting TIDR for 'current' task. + * 4. Both threads are running + * Both threads will be woken. The wrong thread will fail it's condition check + * and execute another wait, while the correct thread will pass it's condition + * check. + * + * @t: the task to set the thread ID for */ int set_thread_tidr(struct task_struct *t) { - int rc; - - if (!cpu_has_feature(CPU_FTR_ARCH_300)) + if (!cpu_has_feature(CPU_FTR_P9_TIDR)) return -EINVAL; if (t != current) @@ -1602,11 +1524,7 @@ int set_thread_tidr(struct task_struct *t) if (t->thread.tidr) return 0; - rc = assign_thread_tidr(); - if (rc < 0) - return rc; - - t->thread.tidr = rc; + t->thread.tidr = (u16)task_pid_nr(t); mtspr(SPRN_TIDR, t->thread.tidr); return 0; diff --git a/arch/powerpc/kernel/prom.c b/arch/powerpc/kernel/prom.c index 9dbed488aba1..05e7fb47a7a4 100644 --- a/arch/powerpc/kernel/prom.c +++ b/arch/powerpc/kernel/prom.c @@ -332,25 +332,10 @@ static int __init early_init_dt_scan_cpus(unsigned long node, * NOTE: This must match the parsing done in smp_setup_cpu_maps. */ for (i = 0; i < nthreads; i++) { - /* - * version 2 of the kexec param format adds the phys cpuid of - * booted proc. - */ - if (fdt_version(initial_boot_params) >= 2) { - if (be32_to_cpu(intserv[i]) == - fdt_boot_cpuid_phys(initial_boot_params)) { - found = boot_cpu_count; - found_thread = i; - } - } else { - /* - * Check if it's the boot-cpu, set it's hw index now, - * unfortunately this format did not support booting - * off secondary threads. - */ - if (of_get_flat_dt_prop(node, - "linux,boot-cpu", NULL) != NULL) - found = boot_cpu_count; + if (be32_to_cpu(intserv[i]) == + fdt_boot_cpuid_phys(initial_boot_params)) { + found = boot_cpu_count; + found_thread = i; } #ifdef CONFIG_SMP /* logical cpu id is always 0 on UP kernels */ diff --git a/arch/powerpc/kernel/prom_init.c b/arch/powerpc/kernel/prom_init.c index f9d6befb55a6..5425dd3d6a9f 100644 --- a/arch/powerpc/kernel/prom_init.c +++ b/arch/powerpc/kernel/prom_init.c @@ -103,7 +103,7 @@ int of_workarounds; #ifdef DEBUG_PROM #define prom_debug(x...) prom_printf(x) #else -#define prom_debug(x...) +#define prom_debug(x...) do { } while (0) #endif @@ -301,6 +301,10 @@ static void __init prom_print(const char *msg) } +/* + * Both prom_print_hex & prom_print_dec takes an unsigned long as input so that + * we do not need __udivdi3 or __umoddi3 on 32bits. + */ static void __init prom_print_hex(unsigned long val) { int i, nibbles = sizeof(val)*2; @@ -334,12 +338,14 @@ static void __init prom_print_dec(unsigned long val) call_prom("write", 3, 1, prom.stdout, buf+i, size); } +__printf(1, 2) static void __init prom_printf(const char *format, ...) { const char *p, *q, *s; va_list args; unsigned long v; long vs; + int n = 0; va_start(args, format); for (p = format; *p != 0; p = q) { @@ -358,6 +364,10 @@ static void __init prom_printf(const char *format, ...) ++q; if (*q == 0) break; + while (*q == 'l') { + ++q; + ++n; + } switch (*q) { case 's': ++q; @@ -366,39 +376,55 @@ static void __init prom_printf(const char *format, ...) break; case 'x': ++q; - v = va_arg(args, unsigned long); + switch (n) { + case 0: + v = va_arg(args, unsigned int); + break; + case 1: + v = va_arg(args, unsigned long); + break; + case 2: + default: + v = va_arg(args, unsigned long long); + break; + } prom_print_hex(v); break; - case 'd': + case 'u': ++q; - vs = va_arg(args, int); - if (vs < 0) { - prom_print("-"); - vs = -vs; + switch (n) { + case 0: + v = va_arg(args, unsigned int); + break; + case 1: + v = va_arg(args, unsigned long); + break; + case 2: + default: + v = va_arg(args, unsigned long long); + break; } - prom_print_dec(vs); + prom_print_dec(v); break; - case 'l': + case 'd': ++q; - if (*q == 0) + switch (n) { + case 0: + vs = va_arg(args, int); break; - else if (*q == 'x') { - ++q; - v = va_arg(args, unsigned long); - prom_print_hex(v); - } else if (*q == 'u') { /* '%lu' */ - ++q; - v = va_arg(args, unsigned long); - prom_print_dec(v); - } else if (*q == 'd') { /* %ld */ - ++q; + case 1: vs = va_arg(args, long); - if (vs < 0) { - prom_print("-"); - vs = -vs; - } - prom_print_dec(vs); + break; + case 2: + default: + vs = va_arg(args, long long); + break; } + if (vs < 0) { + prom_print("-"); + vs = -vs; + } + prom_print_dec(vs); break; } } @@ -1160,7 +1186,7 @@ static void __init prom_send_capabilities(void) */ cores = DIV_ROUND_UP(NR_CPUS, prom_count_smt_threads()); - prom_printf("Max number of cores passed to firmware: %lu (NR_CPUS = %lu)\n", + prom_printf("Max number of cores passed to firmware: %u (NR_CPUS = %d)\n", cores, NR_CPUS); ibm_architecture_vec.vec5.max_cpus = cpu_to_be32(cores); @@ -1242,7 +1268,7 @@ static unsigned long __init alloc_up(unsigned long size, unsigned long align) if (align) base = _ALIGN_UP(base, align); - prom_debug("alloc_up(%x, %x)\n", size, align); + prom_debug("%s(%lx, %lx)\n", __func__, size, align); if (ram_top == 0) prom_panic("alloc_up() called with mem not initialized\n"); @@ -1253,7 +1279,7 @@ static unsigned long __init alloc_up(unsigned long size, unsigned long align) for(; (base + size) <= alloc_top; base = _ALIGN_UP(base + 0x100000, align)) { - prom_debug(" trying: 0x%x\n\r", base); + prom_debug(" trying: 0x%lx\n\r", base); addr = (unsigned long)prom_claim(base, size, 0); if (addr != PROM_ERROR && addr != 0) break; @@ -1265,12 +1291,12 @@ static unsigned long __init alloc_up(unsigned long size, unsigned long align) return 0; alloc_bottom = addr + size; - prom_debug(" -> %x\n", addr); - prom_debug(" alloc_bottom : %x\n", alloc_bottom); - prom_debug(" alloc_top : %x\n", alloc_top); - prom_debug(" alloc_top_hi : %x\n", alloc_top_high); - prom_debug(" rmo_top : %x\n", rmo_top); - prom_debug(" ram_top : %x\n", ram_top); + prom_debug(" -> %lx\n", addr); + prom_debug(" alloc_bottom : %lx\n", alloc_bottom); + prom_debug(" alloc_top : %lx\n", alloc_top); + prom_debug(" alloc_top_hi : %lx\n", alloc_top_high); + prom_debug(" rmo_top : %lx\n", rmo_top); + prom_debug(" ram_top : %lx\n", ram_top); return addr; } @@ -1285,7 +1311,7 @@ static unsigned long __init alloc_down(unsigned long size, unsigned long align, { unsigned long base, addr = 0; - prom_debug("alloc_down(%x, %x, %s)\n", size, align, + prom_debug("%s(%lx, %lx, %s)\n", __func__, size, align, highmem ? "(high)" : "(low)"); if (ram_top == 0) prom_panic("alloc_down() called with mem not initialized\n"); @@ -1313,7 +1339,7 @@ static unsigned long __init alloc_down(unsigned long size, unsigned long align, base = _ALIGN_DOWN(alloc_top - size, align); for (; base > alloc_bottom; base = _ALIGN_DOWN(base - 0x100000, align)) { - prom_debug(" trying: 0x%x\n\r", base); + prom_debug(" trying: 0x%lx\n\r", base); addr = (unsigned long)prom_claim(base, size, 0); if (addr != PROM_ERROR && addr != 0) break; @@ -1324,12 +1350,12 @@ static unsigned long __init alloc_down(unsigned long size, unsigned long align, alloc_top = addr; bail: - prom_debug(" -> %x\n", addr); - prom_debug(" alloc_bottom : %x\n", alloc_bottom); - prom_debug(" alloc_top : %x\n", alloc_top); - prom_debug(" alloc_top_hi : %x\n", alloc_top_high); - prom_debug(" rmo_top : %x\n", rmo_top); - prom_debug(" ram_top : %x\n", ram_top); + prom_debug(" -> %lx\n", addr); + prom_debug(" alloc_bottom : %lx\n", alloc_bottom); + prom_debug(" alloc_top : %lx\n", alloc_top); + prom_debug(" alloc_top_hi : %lx\n", alloc_top_high); + prom_debug(" rmo_top : %lx\n", rmo_top); + prom_debug(" ram_top : %lx\n", ram_top); return addr; } @@ -1455,7 +1481,7 @@ static void __init prom_init_mem(void) if (size == 0) continue; - prom_debug(" %x %x\n", base, size); + prom_debug(" %lx %lx\n", base, size); if (base == 0 && (of_platform & PLATFORM_LPAR)) rmo_top = size; if ((base + size) > ram_top) @@ -1475,12 +1501,12 @@ static void __init prom_init_mem(void) if (prom_memory_limit) { if (prom_memory_limit <= alloc_bottom) { - prom_printf("Ignoring mem=%x <= alloc_bottom.\n", - prom_memory_limit); + prom_printf("Ignoring mem=%lx <= alloc_bottom.\n", + prom_memory_limit); prom_memory_limit = 0; } else if (prom_memory_limit >= ram_top) { - prom_printf("Ignoring mem=%x >= ram_top.\n", - prom_memory_limit); + prom_printf("Ignoring mem=%lx >= ram_top.\n", + prom_memory_limit); prom_memory_limit = 0; } else { ram_top = prom_memory_limit; @@ -1512,12 +1538,13 @@ static void __init prom_init_mem(void) alloc_bottom = PAGE_ALIGN(prom_initrd_end); prom_printf("memory layout at init:\n"); - prom_printf(" memory_limit : %x (16 MB aligned)\n", prom_memory_limit); - prom_printf(" alloc_bottom : %x\n", alloc_bottom); - prom_printf(" alloc_top : %x\n", alloc_top); - prom_printf(" alloc_top_hi : %x\n", alloc_top_high); - prom_printf(" rmo_top : %x\n", rmo_top); - prom_printf(" ram_top : %x\n", ram_top); + prom_printf(" memory_limit : %lx (16 MB aligned)\n", + prom_memory_limit); + prom_printf(" alloc_bottom : %lx\n", alloc_bottom); + prom_printf(" alloc_top : %lx\n", alloc_top); + prom_printf(" alloc_top_hi : %lx\n", alloc_top_high); + prom_printf(" rmo_top : %lx\n", rmo_top); + prom_printf(" ram_top : %lx\n", ram_top); } static void __init prom_close_stdin(void) @@ -1578,7 +1605,7 @@ static void __init prom_instantiate_opal(void) return; } - prom_printf("instantiating opal at 0x%x...", base); + prom_printf("instantiating opal at 0x%llx...", base); if (call_prom_ret("call-method", 4, 3, rets, ADDR("load-opal-runtime"), @@ -1594,10 +1621,10 @@ static void __init prom_instantiate_opal(void) reserve_mem(base, size); - prom_debug("opal base = 0x%x\n", base); - prom_debug("opal align = 0x%x\n", align); - prom_debug("opal entry = 0x%x\n", entry); - prom_debug("opal size = 0x%x\n", (long)size); + prom_debug("opal base = 0x%llx\n", base); + prom_debug("opal align = 0x%llx\n", align); + prom_debug("opal entry = 0x%llx\n", entry); + prom_debug("opal size = 0x%llx\n", size); prom_setprop(opal_node, "/ibm,opal", "opal-base-address", &base, sizeof(base)); @@ -1674,7 +1701,7 @@ static void __init prom_instantiate_rtas(void) prom_debug("rtas base = 0x%x\n", base); prom_debug("rtas entry = 0x%x\n", entry); - prom_debug("rtas size = 0x%x\n", (long)size); + prom_debug("rtas size = 0x%x\n", size); prom_debug("prom_instantiate_rtas: end...\n"); } @@ -1732,7 +1759,7 @@ static void __init prom_instantiate_sml(void) if (base == 0) prom_panic("Could not allocate memory for sml\n"); - prom_printf("instantiating sml at 0x%x...", base); + prom_printf("instantiating sml at 0x%llx...", base); memset((void *)base, 0, size); @@ -1751,8 +1778,8 @@ static void __init prom_instantiate_sml(void) prom_setprop(ibmvtpm_node, "/vdevice/vtpm", "linux,sml-size", &size, sizeof(size)); - prom_debug("sml base = 0x%x\n", base); - prom_debug("sml size = 0x%x\n", (long)size); + prom_debug("sml base = 0x%llx\n", base); + prom_debug("sml size = 0x%x\n", size); prom_debug("prom_instantiate_sml: end...\n"); } @@ -1845,7 +1872,7 @@ static void __init prom_initialize_tce_table(void) prom_debug("TCE table: %s\n", path); prom_debug("\tnode = 0x%x\n", node); - prom_debug("\tbase = 0x%x\n", base); + prom_debug("\tbase = 0x%llx\n", base); prom_debug("\tsize = 0x%x\n", minsize); /* Initialize the table to have a one-to-one mapping @@ -1932,12 +1959,12 @@ static void __init prom_hold_cpus(void) } prom_debug("prom_hold_cpus: start...\n"); - prom_debug(" 1) spinloop = 0x%x\n", (unsigned long)spinloop); - prom_debug(" 1) *spinloop = 0x%x\n", *spinloop); - prom_debug(" 1) acknowledge = 0x%x\n", + prom_debug(" 1) spinloop = 0x%lx\n", (unsigned long)spinloop); + prom_debug(" 1) *spinloop = 0x%lx\n", *spinloop); + prom_debug(" 1) acknowledge = 0x%lx\n", (unsigned long)acknowledge); - prom_debug(" 1) *acknowledge = 0x%x\n", *acknowledge); - prom_debug(" 1) secondary_hold = 0x%x\n", secondary_hold); + prom_debug(" 1) *acknowledge = 0x%lx\n", *acknowledge); + prom_debug(" 1) secondary_hold = 0x%lx\n", secondary_hold); /* Set the common spinloop variable, so all of the secondary cpus * will block when they are awakened from their OF spinloop. @@ -1965,7 +1992,7 @@ static void __init prom_hold_cpus(void) prom_getprop(node, "reg", ®, sizeof(reg)); cpu_no = be32_to_cpu(reg); - prom_debug("cpu hw idx = %lu\n", cpu_no); + prom_debug("cpu hw idx = %u\n", cpu_no); /* Init the acknowledge var which will be reset by * the secondary cpu when it awakens from its OF @@ -1975,7 +2002,7 @@ static void __init prom_hold_cpus(void) if (cpu_no != prom.cpu) { /* Primary Thread of non-boot cpu or any thread */ - prom_printf("starting cpu hw idx %lu... ", cpu_no); + prom_printf("starting cpu hw idx %u... ", cpu_no); call_prom("start-cpu", 3, 0, node, secondary_hold, cpu_no); @@ -1986,11 +2013,11 @@ static void __init prom_hold_cpus(void) if (*acknowledge == cpu_no) prom_printf("done\n"); else - prom_printf("failed: %x\n", *acknowledge); + prom_printf("failed: %lx\n", *acknowledge); } #ifdef CONFIG_SMP else - prom_printf("boot cpu hw idx %lu\n", cpu_no); + prom_printf("boot cpu hw idx %u\n", cpu_no); #endif /* CONFIG_SMP */ } @@ -2268,7 +2295,7 @@ static void __init *make_room(unsigned long *mem_start, unsigned long *mem_end, while ((*mem_start + needed) > *mem_end) { unsigned long room, chunk; - prom_debug("Chunk exhausted, claiming more at %x...\n", + prom_debug("Chunk exhausted, claiming more at %lx...\n", alloc_bottom); room = alloc_top - alloc_bottom; if (room > DEVTREE_CHUNK_SIZE) @@ -2494,7 +2521,7 @@ static void __init flatten_device_tree(void) room = alloc_top - alloc_bottom - 0x4000; if (room > DEVTREE_CHUNK_SIZE) room = DEVTREE_CHUNK_SIZE; - prom_debug("starting device tree allocs at %x\n", alloc_bottom); + prom_debug("starting device tree allocs at %lx\n", alloc_bottom); /* Now try to claim that */ mem_start = (unsigned long)alloc_up(room, PAGE_SIZE); @@ -2557,7 +2584,7 @@ static void __init flatten_device_tree(void) int i; prom_printf("reserved memory map:\n"); for (i = 0; i < mem_reserve_cnt; i++) - prom_printf(" %x - %x\n", + prom_printf(" %llx - %llx\n", be64_to_cpu(mem_reserve_map[i].base), be64_to_cpu(mem_reserve_map[i].size)); } @@ -2567,9 +2594,9 @@ static void __init flatten_device_tree(void) */ mem_reserve_cnt = MEM_RESERVE_MAP_SIZE; - prom_printf("Device tree strings 0x%x -> 0x%x\n", + prom_printf("Device tree strings 0x%lx -> 0x%lx\n", dt_string_start, dt_string_end); - prom_printf("Device tree struct 0x%x -> 0x%x\n", + prom_printf("Device tree struct 0x%lx -> 0x%lx\n", dt_struct_start, dt_struct_end); } @@ -3001,7 +3028,7 @@ static void __init prom_find_boot_cpu(void) prom_getprop(cpu_pkg, "reg", &rval, sizeof(rval)); prom.cpu = be32_to_cpu(rval); - prom_debug("Booting CPU hw index = %lu\n", prom.cpu); + prom_debug("Booting CPU hw index = %d\n", prom.cpu); } static void __init prom_check_initrd(unsigned long r3, unsigned long r4) @@ -3023,8 +3050,8 @@ static void __init prom_check_initrd(unsigned long r3, unsigned long r4) reserve_mem(prom_initrd_start, prom_initrd_end - prom_initrd_start); - prom_debug("initrd_start=0x%x\n", prom_initrd_start); - prom_debug("initrd_end=0x%x\n", prom_initrd_end); + prom_debug("initrd_start=0x%lx\n", prom_initrd_start); + prom_debug("initrd_end=0x%lx\n", prom_initrd_end); } #endif /* CONFIG_BLK_DEV_INITRD */ } @@ -3277,7 +3304,7 @@ unsigned long __init prom_init(unsigned long r3, unsigned long r4, /* Don't print anything after quiesce under OPAL, it crashes OFW */ if (of_platform != PLATFORM_OPAL) { prom_printf("Booting Linux via __start() @ 0x%lx ...\n", kbase); - prom_debug("->dt_header_start=0x%x\n", hdr); + prom_debug("->dt_header_start=0x%lx\n", hdr); } #ifdef CONFIG_PPC32 diff --git a/arch/powerpc/kernel/ptrace.c b/arch/powerpc/kernel/ptrace.c index d23cf632edf0..9667666eb18e 100644 --- a/arch/powerpc/kernel/ptrace.c +++ b/arch/powerpc/kernel/ptrace.c @@ -2443,6 +2443,7 @@ static int ptrace_set_debugreg(struct task_struct *task, unsigned long addr, /* Create a new breakpoint request if one doesn't exist already */ hw_breakpoint_init(&attr); attr.bp_addr = hw_brk.address; + attr.bp_len = 8; arch_bp_generic_fields(hw_brk.type, &attr.bp_type); @@ -3081,27 +3082,19 @@ long arch_ptrace(struct task_struct *child, long request, #endif /* CONFIG_HAVE_HW_BREAKPOINT */ #endif /* CONFIG_PPC_ADV_DEBUG_REGS */ - if (!access_ok(VERIFY_WRITE, datavp, - sizeof(struct ppc_debug_info))) + if (copy_to_user(datavp, &dbginfo, + sizeof(struct ppc_debug_info))) return -EFAULT; - ret = __copy_to_user(datavp, &dbginfo, - sizeof(struct ppc_debug_info)) ? - -EFAULT : 0; - break; + return 0; } case PPC_PTRACE_SETHWDEBUG: { struct ppc_hw_breakpoint bp_info; - if (!access_ok(VERIFY_READ, datavp, - sizeof(struct ppc_hw_breakpoint))) + if (copy_from_user(&bp_info, datavp, + sizeof(struct ppc_hw_breakpoint))) return -EFAULT; - ret = __copy_from_user(&bp_info, datavp, - sizeof(struct ppc_hw_breakpoint)) ? - -EFAULT : 0; - if (!ret) - ret = ppc_set_hwdebug(child, &bp_info); - break; + return ppc_set_hwdebug(child, &bp_info); } case PPC_PTRACE_DELHWDEBUG: { diff --git a/arch/powerpc/kernel/rtas-proc.c b/arch/powerpc/kernel/rtas-proc.c index fb070d8cad07..487dcd8da4de 100644 --- a/arch/powerpc/kernel/rtas-proc.c +++ b/arch/powerpc/kernel/rtas-proc.c @@ -154,18 +154,6 @@ static ssize_t ppc_rtas_tone_volume_write(struct file *file, static int ppc_rtas_tone_volume_show(struct seq_file *m, void *v); static int ppc_rtas_rmo_buf_show(struct seq_file *m, void *v); -static int sensors_open(struct inode *inode, struct file *file) -{ - return single_open(file, ppc_rtas_sensors_show, NULL); -} - -static const struct file_operations ppc_rtas_sensors_operations = { - .open = sensors_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int poweron_open(struct inode *inode, struct file *file) { return single_open(file, ppc_rtas_poweron_show, NULL); @@ -231,18 +219,6 @@ static const struct file_operations ppc_rtas_tone_volume_operations = { .release = single_release, }; -static int rmo_buf_open(struct inode *inode, struct file *file) -{ - return single_open(file, ppc_rtas_rmo_buf_show, NULL); -} - -static const struct file_operations ppc_rtas_rmo_buf_ops = { - .open = rmo_buf_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int ppc_rtas_find_all_sensors(void); static void ppc_rtas_process_sensor(struct seq_file *m, struct individual_sensor *s, int state, int error, const char *loc); @@ -267,20 +243,20 @@ static int __init proc_rtas_init(void) &ppc_rtas_clock_operations); proc_create("powerpc/rtas/poweron", 0644, NULL, &ppc_rtas_poweron_operations); - proc_create("powerpc/rtas/sensors", 0444, NULL, - &ppc_rtas_sensors_operations); + proc_create_single("powerpc/rtas/sensors", 0444, NULL, + ppc_rtas_sensors_show); proc_create("powerpc/rtas/frequency", 0644, NULL, &ppc_rtas_tone_freq_operations); proc_create("powerpc/rtas/volume", 0644, NULL, &ppc_rtas_tone_volume_operations); - proc_create("powerpc/rtas/rmo_buffer", 0400, NULL, - &ppc_rtas_rmo_buf_ops); + proc_create_single("powerpc/rtas/rmo_buffer", 0400, NULL, + ppc_rtas_rmo_buf_show); return 0; } __initcall(proc_rtas_init); -static int parse_number(const char __user *p, size_t count, unsigned long *val) +static int parse_number(const char __user *p, size_t count, u64 *val) { char buf[40]; char *end; @@ -293,7 +269,7 @@ static int parse_number(const char __user *p, size_t count, unsigned long *val) buf[count] = 0; - *val = simple_strtoul(buf, &end, 10); + *val = simple_strtoull(buf, &end, 10); if (*end && *end != '\n') return -EINVAL; @@ -307,17 +283,17 @@ static ssize_t ppc_rtas_poweron_write(struct file *file, const char __user *buf, size_t count, loff_t *ppos) { struct rtc_time tm; - unsigned long nowtime; + time64_t nowtime; int error = parse_number(buf, count, &nowtime); if (error) return error; power_on_time = nowtime; /* save the time */ - to_tm(nowtime, &tm); + rtc_time64_to_tm(nowtime, &tm); error = rtas_call(rtas_token("set-time-for-power-on"), 7, 1, NULL, - tm.tm_year, tm.tm_mon, tm.tm_mday, + tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec, 0 /* nano */); if (error) printk(KERN_WARNING "error: setting poweron time returned: %s\n", @@ -373,14 +349,14 @@ static ssize_t ppc_rtas_clock_write(struct file *file, const char __user *buf, size_t count, loff_t *ppos) { struct rtc_time tm; - unsigned long nowtime; + time64_t nowtime; int error = parse_number(buf, count, &nowtime); if (error) return error; - to_tm(nowtime, &tm); + rtc_time64_to_tm(nowtime, &tm); error = rtas_call(rtas_token("set-time-of-day"), 7, 1, NULL, - tm.tm_year, tm.tm_mon, tm.tm_mday, + tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec, 0); if (error) printk(KERN_WARNING "error: setting the clock returned: %s\n", @@ -401,8 +377,8 @@ static int ppc_rtas_clock_show(struct seq_file *m, void *v) unsigned int year, mon, day, hour, min, sec; year = ret[0]; mon = ret[1]; day = ret[2]; hour = ret[3]; min = ret[4]; sec = ret[5]; - seq_printf(m, "%lu\n", - mktime(year, mon, day, hour, min, sec)); + seq_printf(m, "%lld\n", + mktime64(year, mon, day, hour, min, sec)); } return 0; } @@ -528,7 +504,7 @@ static void ppc_rtas_process_sensor(struct seq_file *m, "EPOW power off" }; const char * battery_cyclestate[] = { "None", "In progress", "Requested" }; - const char * battery_charging[] = { "Charging", "Discharching", + const char * battery_charging[] = { "Charging", "Discharging", "No current flow" }; const char * ibm_drconnector[] = { "Empty", "Present", "Unusable", "Exchange" }; @@ -731,7 +707,7 @@ static void get_location_code(struct seq_file *m, struct individual_sensor *s, static ssize_t ppc_rtas_tone_freq_write(struct file *file, const char __user *buf, size_t count, loff_t *ppos) { - unsigned long freq; + u64 freq; int error = parse_number(buf, count, &freq); if (error) return error; @@ -756,7 +732,7 @@ static int ppc_rtas_tone_freq_show(struct seq_file *m, void *v) static ssize_t ppc_rtas_tone_volume_write(struct file *file, const char __user *buf, size_t count, loff_t *ppos) { - unsigned long volume; + u64 volume; int error = parse_number(buf, count, &volume); if (error) return error; diff --git a/arch/powerpc/kernel/rtas-rtc.c b/arch/powerpc/kernel/rtas-rtc.c index 49600985c7ef..a28239b8b0c0 100644 --- a/arch/powerpc/kernel/rtas-rtc.c +++ b/arch/powerpc/kernel/rtas-rtc.c @@ -13,7 +13,7 @@ #define MAX_RTC_WAIT 5000 /* 5 sec */ #define RTAS_CLOCK_BUSY (-2) -unsigned long __init rtas_get_boot_time(void) +time64_t __init rtas_get_boot_time(void) { int ret[8]; int error; @@ -38,7 +38,7 @@ unsigned long __init rtas_get_boot_time(void) return 0; } - return mktime(ret[0], ret[1], ret[2], ret[3], ret[4], ret[5]); + return mktime64(ret[0], ret[1], ret[2], ret[3], ret[4], ret[5]); } /* NOTE: get_rtc_time will get an error if executed in interrupt context diff --git a/arch/powerpc/kernel/rtas.c b/arch/powerpc/kernel/rtas.c index 3f1c4fcbe0aa..7fb9f83dcde8 100644 --- a/arch/powerpc/kernel/rtas.c +++ b/arch/powerpc/kernel/rtas.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include @@ -1050,7 +1051,10 @@ struct pseries_errorlog *get_pseries_errorlog(struct rtas_error_log *log, } /* We assume to be passed big endian arguments */ -asmlinkage int ppc_rtas(struct rtas_args __user *uargs) +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpragmas" +#pragma GCC diagnostic ignored "-Wattribute-alias" +SYSCALL_DEFINE1(rtas, struct rtas_args __user *, uargs) { struct rtas_args args; unsigned long flags; @@ -1136,6 +1140,7 @@ asmlinkage int ppc_rtas(struct rtas_args __user *uargs) return 0; } +#pragma GCC diagnostic pop /* * Call early during boot, before mem init, to retrieve the RTAS diff --git a/arch/powerpc/kernel/rtasd.c b/arch/powerpc/kernel/rtasd.c index f915db93cd42..44d66c33d59d 100644 --- a/arch/powerpc/kernel/rtasd.c +++ b/arch/powerpc/kernel/rtasd.c @@ -559,7 +559,8 @@ static int __init rtas_event_scan_init(void) rtas_error_log_max = rtas_get_error_log_max(); rtas_error_log_buffer_max = rtas_error_log_max + sizeof(int); - rtas_log_buf = vmalloc(rtas_error_log_buffer_max*LOG_NUMBER); + rtas_log_buf = vmalloc(array_size(LOG_NUMBER, + rtas_error_log_buffer_max)); if (!rtas_log_buf) { printk(KERN_ERR "rtasd: no memory\n"); return -ENOMEM; diff --git a/arch/powerpc/kernel/security.c b/arch/powerpc/kernel/security.c index b98a722da915..a8b277362931 100644 --- a/arch/powerpc/kernel/security.c +++ b/arch/powerpc/kernel/security.c @@ -10,10 +10,78 @@ #include #include +#include unsigned long powerpc_security_features __read_mostly = SEC_FTR_DEFAULT; +bool barrier_nospec_enabled; + +static void enable_barrier_nospec(bool enable) +{ + barrier_nospec_enabled = enable; + do_barrier_nospec_fixups(enable); +} + +void setup_barrier_nospec(void) +{ + bool enable; + + /* + * It would make sense to check SEC_FTR_SPEC_BAR_ORI31 below as well. + * But there's a good reason not to. The two flags we check below are + * both are enabled by default in the kernel, so if the hcall is not + * functional they will be enabled. + * On a system where the host firmware has been updated (so the ori + * functions as a barrier), but on which the hypervisor (KVM/Qemu) has + * not been updated, we would like to enable the barrier. Dropping the + * check for SEC_FTR_SPEC_BAR_ORI31 achieves that. The only downside is + * we potentially enable the barrier on systems where the host firmware + * is not updated, but that's harmless as it's a no-op. + */ + enable = security_ftr_enabled(SEC_FTR_FAVOUR_SECURITY) && + security_ftr_enabled(SEC_FTR_BNDS_CHK_SPEC_BAR); + + enable_barrier_nospec(enable); +} + +#ifdef CONFIG_DEBUG_FS +static int barrier_nospec_set(void *data, u64 val) +{ + switch (val) { + case 0: + case 1: + break; + default: + return -EINVAL; + } + + if (!!val == !!barrier_nospec_enabled) + return 0; + + enable_barrier_nospec(!!val); + + return 0; +} + +static int barrier_nospec_get(void *data, u64 *val) +{ + *val = barrier_nospec_enabled ? 1 : 0; + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(fops_barrier_nospec, + barrier_nospec_get, barrier_nospec_set, "%llu\n"); + +static __init int barrier_nospec_debugfs_init(void) +{ + debugfs_create_file("barrier_nospec", 0600, powerpc_debugfs_root, NULL, + &fops_barrier_nospec); + return 0; +} +device_initcall(barrier_nospec_debugfs_init); +#endif /* CONFIG_DEBUG_FS */ + ssize_t cpu_show_meltdown(struct device *dev, struct device_attribute *attr, char *buf) { bool thread_priv; @@ -52,6 +120,9 @@ ssize_t cpu_show_spectre_v1(struct device *dev, struct device_attribute *attr, c if (!security_ftr_enabled(SEC_FTR_BNDS_CHK_SPEC_BAR)) return sprintf(buf, "Not affected\n"); + if (barrier_nospec_enabled) + return sprintf(buf, "Mitigation: __user pointer sanitization\n"); + return sprintf(buf, "Vulnerable\n"); } diff --git a/arch/powerpc/kernel/setup-common.c b/arch/powerpc/kernel/setup-common.c index 0af5c11b9e78..62b1a40d8957 100644 --- a/arch/powerpc/kernel/setup-common.c +++ b/arch/powerpc/kernel/setup-common.c @@ -192,12 +192,6 @@ void machine_halt(void) machine_hang(); } - -#ifdef CONFIG_TAU -extern u32 cpu_temp(unsigned long cpu); -extern u32 cpu_temp_both(unsigned long cpu); -#endif /* CONFIG_TAU */ - #ifdef CONFIG_SMP DEFINE_PER_CPU(unsigned int, cpu_pvr); #endif diff --git a/arch/powerpc/kernel/setup.h b/arch/powerpc/kernel/setup.h index d144df54ad40..c6a592b67386 100644 --- a/arch/powerpc/kernel/setup.h +++ b/arch/powerpc/kernel/setup.h @@ -62,4 +62,10 @@ void kvm_cma_reserve(void); static inline void kvm_cma_reserve(void) { }; #endif +#ifdef CONFIG_TAU +u32 cpu_temp(unsigned long cpu); +u32 cpu_temp_both(unsigned long cpu); +u32 tau_interrupts(unsigned long cpu); +#endif /* CONFIG_TAU */ + #endif /* __ARCH_POWERPC_KERNEL_SETUP_H */ diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c index b78f142a4148..7a7ce8ad455e 100644 --- a/arch/powerpc/kernel/setup_64.c +++ b/arch/powerpc/kernel/setup_64.c @@ -346,6 +346,13 @@ void __init early_setup(unsigned long dt_ptr) */ cpu_ready_for_interrupts(); + /* + * We enable ftrace here, but since we only support DYNAMIC_FTRACE, it + * will only actually get enabled on the boot cpu much later once + * ftrace itself has been initialized. + */ + this_cpu_enable_ftrace(); + DBG(" <- early_setup()\n"); #ifdef CONFIG_PPC_EARLY_DEBUG_BOOTX diff --git a/arch/powerpc/kernel/signal.c b/arch/powerpc/kernel/signal.c index 61db86ecd318..17fe4339ba59 100644 --- a/arch/powerpc/kernel/signal.c +++ b/arch/powerpc/kernel/signal.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -133,6 +134,8 @@ static void do_signal(struct task_struct *tsk) /* Re-enable the breakpoints for the signal stack */ thread_change_pc(tsk, tsk->thread.regs); + rseq_signal_deliver(tsk->thread.regs); + if (is32) { if (ksig.ka.sa.sa_flags & SA_SIGINFO) ret = handle_rt_signal32(&ksig, oldset, tsk); @@ -150,6 +153,9 @@ void do_notify_resume(struct pt_regs *regs, unsigned long thread_info_flags) { user_exit(); + /* Check valid addr_limit, TIF check is done there */ + addr_limit_user_check(); + if (thread_info_flags & _TIF_UPROBE) uprobe_notify_resume(regs); @@ -164,6 +170,7 @@ void do_notify_resume(struct pt_regs *regs, unsigned long thread_info_flags) if (thread_info_flags & _TIF_NOTIFY_RESUME) { clear_thread_flag(TIF_NOTIFY_RESUME); tracehook_notify_resume(regs); + rseq_handle_notify_resume(regs); } user_enter(); diff --git a/arch/powerpc/kernel/signal.h b/arch/powerpc/kernel/signal.h index a6467f843acf..800433685888 100644 --- a/arch/powerpc/kernel/signal.h +++ b/arch/powerpc/kernel/signal.h @@ -49,10 +49,8 @@ extern int handle_rt_signal64(struct ksignal *ksig, sigset_t *set, #else /* CONFIG_PPC64 */ -extern long sys_rt_sigreturn(int r3, int r4, int r5, int r6, int r7, int r8, - struct pt_regs *regs); -extern long sys_sigreturn(int r3, int r4, int r5, int r6, int r7, int r8, - struct pt_regs *regs); +extern long sys_rt_sigreturn(void); +extern long sys_sigreturn(void); static inline int handle_rt_signal64(struct ksignal *ksig, sigset_t *set, struct task_struct *tsk) diff --git a/arch/powerpc/kernel/signal_32.c b/arch/powerpc/kernel/signal_32.c index 492f03451877..5eedbb282d42 100644 --- a/arch/powerpc/kernel/signal_32.c +++ b/arch/powerpc/kernel/signal_32.c @@ -25,9 +25,10 @@ #include #include #include +#include #include -#ifdef CONFIG_PPC64 #include +#ifdef CONFIG_PPC64 #include #else #include @@ -57,10 +58,6 @@ #ifdef CONFIG_PPC64 -#define sys_rt_sigreturn compat_sys_rt_sigreturn -#define sys_swapcontext compat_sys_swapcontext -#define sys_sigreturn compat_sys_sigreturn - #define old_sigaction old_sigaction32 #define sigcontext sigcontext32 #define mcontext mcontext32 @@ -1041,11 +1038,18 @@ static int do_setcontext_tm(struct ucontext __user *ucp, } #endif -long sys_swapcontext(struct ucontext __user *old_ctx, - struct ucontext __user *new_ctx, - int ctx_size, int r6, int r7, int r8, struct pt_regs *regs) +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpragmas" +#pragma GCC diagnostic ignored "-Wattribute-alias" +#ifdef CONFIG_PPC64 +COMPAT_SYSCALL_DEFINE3(swapcontext, struct ucontext __user *, old_ctx, + struct ucontext __user *, new_ctx, int, ctx_size) +#else +SYSCALL_DEFINE3(swapcontext, struct ucontext __user *, old_ctx, + struct ucontext __user *, new_ctx, long, ctx_size) +#endif { - unsigned char tmp __maybe_unused; + struct pt_regs *regs = current_pt_regs(); int ctx_has_vsx_region = 0; #ifdef CONFIG_PPC64 @@ -1109,9 +1113,8 @@ long sys_swapcontext(struct ucontext __user *old_ctx, } if (new_ctx == NULL) return 0; - if (!access_ok(VERIFY_READ, new_ctx, ctx_size) - || __get_user(tmp, (u8 __user *) new_ctx) - || __get_user(tmp, (u8 __user *) new_ctx + ctx_size - 1)) + if (!access_ok(VERIFY_READ, new_ctx, ctx_size) || + fault_in_pages_readable((u8 __user *)new_ctx, ctx_size)) return -EFAULT; /* @@ -1131,11 +1134,16 @@ long sys_swapcontext(struct ucontext __user *old_ctx, set_thread_flag(TIF_RESTOREALL); return 0; } +#pragma GCC diagnostic pop -long sys_rt_sigreturn(int r3, int r4, int r5, int r6, int r7, int r8, - struct pt_regs *regs) +#ifdef CONFIG_PPC64 +COMPAT_SYSCALL_DEFINE0(rt_sigreturn) +#else +SYSCALL_DEFINE0(rt_sigreturn) +#endif { struct rt_sigframe __user *rt_sf; + struct pt_regs *regs = current_pt_regs(); #ifdef CONFIG_PPC_TRANSACTIONAL_MEM struct ucontext __user *uc_transact; unsigned long msr_hi; @@ -1223,15 +1231,16 @@ long sys_rt_sigreturn(int r3, int r4, int r5, int r6, int r7, int r8, return 0; } +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpragmas" +#pragma GCC diagnostic ignored "-Wattribute-alias" #ifdef CONFIG_PPC32 -int sys_debug_setcontext(struct ucontext __user *ctx, - int ndbg, struct sig_dbg_op __user *dbg, - int r6, int r7, int r8, - struct pt_regs *regs) +SYSCALL_DEFINE3(debug_setcontext, struct ucontext __user *, ctx, + int, ndbg, struct sig_dbg_op __user *, dbg) { + struct pt_regs *regs = current_pt_regs(); struct sig_dbg_op op; int i; - unsigned char tmp __maybe_unused; unsigned long new_msr = regs->msr; #ifdef CONFIG_PPC_ADV_DEBUG_REGS unsigned long new_dbcr0 = current->thread.debug.dbcr0; @@ -1287,9 +1296,8 @@ int sys_debug_setcontext(struct ucontext __user *ctx, current->thread.debug.dbcr0 = new_dbcr0; #endif - if (!access_ok(VERIFY_READ, ctx, sizeof(*ctx)) - || __get_user(tmp, (u8 __user *) ctx) - || __get_user(tmp, (u8 __user *) (ctx + 1) - 1)) + if (!access_ok(VERIFY_READ, ctx, sizeof(*ctx)) || + fault_in_pages_readable((u8 __user *)ctx, sizeof(*ctx))) return -EFAULT; /* @@ -1329,6 +1337,7 @@ int sys_debug_setcontext(struct ucontext __user *ctx, return 0; } #endif +#pragma GCC diagnostic pop /* * OK, we're invoking a handler @@ -1419,9 +1428,13 @@ badframe: /* * Do a signal return; undo the signal stack. */ -long sys_sigreturn(int r3, int r4, int r5, int r6, int r7, int r8, - struct pt_regs *regs) +#ifdef CONFIG_PPC64 +COMPAT_SYSCALL_DEFINE0(sigreturn) +#else +SYSCALL_DEFINE0(sigreturn) +#endif { + struct pt_regs *regs = current_pt_regs(); struct sigframe __user *sf; struct sigcontext __user *sc; struct sigcontext sigctx; diff --git a/arch/powerpc/kernel/signal_64.c b/arch/powerpc/kernel/signal_64.c index 720117690822..d42b60020389 100644 --- a/arch/powerpc/kernel/signal_64.c +++ b/arch/powerpc/kernel/signal_64.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include @@ -624,17 +625,17 @@ static long setup_trampoline(unsigned int syscall, unsigned int __user *tramp) /* * Handle {get,set,swap}_context operations */ -int sys_swapcontext(struct ucontext __user *old_ctx, - struct ucontext __user *new_ctx, - long ctx_size, long r6, long r7, long r8, struct pt_regs *regs) +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpragmas" +#pragma GCC diagnostic ignored "-Wattribute-alias" +SYSCALL_DEFINE3(swapcontext, struct ucontext __user *, old_ctx, + struct ucontext __user *, new_ctx, long, ctx_size) { unsigned char tmp; sigset_t set; unsigned long new_msr = 0; int ctx_has_vsx_region = 0; - BUG_ON(regs != current->thread.regs); - if (new_ctx && get_user(new_msr, &new_ctx->uc_mcontext.gp_regs[PT_MSR])) return -EFAULT; @@ -692,24 +693,22 @@ int sys_swapcontext(struct ucontext __user *old_ctx, set_thread_flag(TIF_RESTOREALL); return 0; } +#pragma GCC diagnostic pop /* * Do a signal return; undo the signal stack. */ -int sys_rt_sigreturn(unsigned long r3, unsigned long r4, unsigned long r5, - unsigned long r6, unsigned long r7, unsigned long r8, - struct pt_regs *regs) +SYSCALL_DEFINE0(rt_sigreturn) { + struct pt_regs *regs = current_pt_regs(); struct ucontext __user *uc = (struct ucontext __user *)regs->gpr[1]; sigset_t set; #ifdef CONFIG_PPC_TRANSACTIONAL_MEM unsigned long msr; #endif - BUG_ON(current->thread.regs != regs); - /* Always make any pending restarted system calls return -EINTR */ current->restart_block.fn = do_no_restart_syscall; diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index 9ca7148b5881..5eadfffabe35 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -59,6 +59,7 @@ #include #include #include +#include #ifdef DEBUG #include @@ -155,11 +156,13 @@ static irqreturn_t reschedule_action(int irq, void *data) return IRQ_HANDLED; } +#ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST static irqreturn_t tick_broadcast_ipi_action(int irq, void *data) { - tick_broadcast_ipi_handler(); + timer_broadcast_interrupt(); return IRQ_HANDLED; } +#endif #ifdef CONFIG_NMI_IPI static irqreturn_t nmi_ipi_action(int irq, void *data) @@ -172,7 +175,9 @@ static irqreturn_t nmi_ipi_action(int irq, void *data) static irq_handler_t smp_ipi_action[] = { [PPC_MSG_CALL_FUNCTION] = call_function_action, [PPC_MSG_RESCHEDULE] = reschedule_action, +#ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST [PPC_MSG_TICK_BROADCAST] = tick_broadcast_ipi_action, +#endif #ifdef CONFIG_NMI_IPI [PPC_MSG_NMI_IPI] = nmi_ipi_action, #endif @@ -186,8 +191,12 @@ static irq_handler_t smp_ipi_action[] = { const char *smp_ipi_name[] = { [PPC_MSG_CALL_FUNCTION] = "ipi call function", [PPC_MSG_RESCHEDULE] = "ipi reschedule", +#ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST [PPC_MSG_TICK_BROADCAST] = "ipi tick-broadcast", +#endif +#ifdef CONFIG_NMI_IPI [PPC_MSG_NMI_IPI] = "nmi ipi", +#endif }; /* optional function to request ipi, for controllers with >= 4 ipis */ @@ -277,8 +286,10 @@ irqreturn_t smp_ipi_demux_relaxed(void) generic_smp_call_function_interrupt(); if (all & IPI_MESSAGE(PPC_MSG_RESCHEDULE)) scheduler_ipi(); +#ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST if (all & IPI_MESSAGE(PPC_MSG_TICK_BROADCAST)) - tick_broadcast_ipi_handler(); + timer_broadcast_interrupt(); +#endif #ifdef CONFIG_NMI_IPI if (all & IPI_MESSAGE(PPC_MSG_NMI_IPI)) nmi_ipi_action(0, NULL); @@ -419,9 +430,9 @@ out: return ret; } -static void do_smp_send_nmi_ipi(int cpu) +static void do_smp_send_nmi_ipi(int cpu, bool safe) { - if (smp_ops->cause_nmi_ipi && smp_ops->cause_nmi_ipi(cpu)) + if (!safe && smp_ops->cause_nmi_ipi && smp_ops->cause_nmi_ipi(cpu)) return; if (cpu >= 0) { @@ -461,7 +472,7 @@ void smp_flush_nmi_ipi(u64 delay_us) * - delay_us > 0 is the delay before giving up waiting for targets to * enter the handler, == 0 specifies indefinite delay. */ -int smp_send_nmi_ipi(int cpu, void (*fn)(struct pt_regs *), u64 delay_us) +int __smp_send_nmi_ipi(int cpu, void (*fn)(struct pt_regs *), u64 delay_us, bool safe) { unsigned long flags; int me = raw_smp_processor_id(); @@ -494,7 +505,7 @@ int smp_send_nmi_ipi(int cpu, void (*fn)(struct pt_regs *), u64 delay_us) nmi_ipi_busy_count++; nmi_ipi_unlock(); - do_smp_send_nmi_ipi(cpu); + do_smp_send_nmi_ipi(cpu, safe); while (!cpumask_empty(&nmi_ipi_pending_mask)) { udelay(1); @@ -516,6 +527,16 @@ int smp_send_nmi_ipi(int cpu, void (*fn)(struct pt_regs *), u64 delay_us) return ret; } + +int smp_send_nmi_ipi(int cpu, void (*fn)(struct pt_regs *), u64 delay_us) +{ + return __smp_send_nmi_ipi(cpu, fn, delay_us, false); +} + +int smp_send_safe_nmi_ipi(int cpu, void (*fn)(struct pt_regs *), u64 delay_us) +{ + return __smp_send_nmi_ipi(cpu, fn, delay_us, true); +} #endif /* CONFIG_NMI_IPI */ #ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST @@ -559,7 +580,7 @@ void crash_send_ipi(void (*crash_ipi_callback)(struct pt_regs *)) * entire NMI dance and waiting for * cpus to clear pending mask, etc. */ - do_smp_send_nmi_ipi(cpu); + do_smp_send_nmi_ipi(cpu, false); } } } @@ -1066,6 +1087,9 @@ void start_secondary(void *unused) local_irq_enable(); + /* We can enable ftrace for secondary cpus now */ + this_cpu_enable_ftrace(); + cpu_startup_entry(CPUHP_AP_ONLINE_IDLE); BUG(); @@ -1162,6 +1186,8 @@ int __cpu_disable(void) if (!smp_ops->cpu_disable) return -ENOSYS; + this_cpu_disable_ftrace(); + err = smp_ops->cpu_disable(); if (err) return err; @@ -1180,6 +1206,12 @@ void __cpu_die(unsigned int cpu) void cpu_die(void) { + /* + * Disable on the down path. This will be re-enabled by + * start_secondary() via start_secondary_resume() below + */ + this_cpu_disable_ftrace(); + if (ppc_md.cpu_die) ppc_md.cpu_die(); diff --git a/arch/powerpc/kernel/stacktrace.c b/arch/powerpc/kernel/stacktrace.c index d534ed901538..07e97f289c52 100644 --- a/arch/powerpc/kernel/stacktrace.c +++ b/arch/powerpc/kernel/stacktrace.c @@ -1,21 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0 + /* - * Stack trace utility + * Stack trace utility functions etc. * * Copyright 2008 Christoph Hellwig, IBM Corp. - * - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. + * Copyright 2018 SUSE Linux GmbH + * Copyright 2018 Nick Piggin, Michael Ellerman, IBM Corp. */ #include +#include +#include +#include #include #include +#include #include #include #include +#include +#include + +#include /* * Save stack-backtrace addresses into a stack_trace buffer. @@ -76,3 +82,164 @@ save_stack_trace_regs(struct pt_regs *regs, struct stack_trace *trace) save_context_stack(trace, regs->gpr[1], current, 0); } EXPORT_SYMBOL_GPL(save_stack_trace_regs); + +#ifdef CONFIG_HAVE_RELIABLE_STACKTRACE +int +save_stack_trace_tsk_reliable(struct task_struct *tsk, + struct stack_trace *trace) +{ + unsigned long sp; + unsigned long stack_page = (unsigned long)task_stack_page(tsk); + unsigned long stack_end; + int graph_idx = 0; + + /* + * The last frame (unwinding first) may not yet have saved + * its LR onto the stack. + */ + int firstframe = 1; + + if (tsk == current) + sp = current_stack_pointer(); + else + sp = tsk->thread.ksp; + + stack_end = stack_page + THREAD_SIZE; + if (!is_idle_task(tsk)) { + /* + * For user tasks, this is the SP value loaded on + * kernel entry, see "PACAKSAVE(r13)" in _switch() and + * system_call_common()/EXCEPTION_PROLOG_COMMON(). + * + * Likewise for non-swapper kernel threads, + * this also happens to be the top of the stack + * as setup by copy_thread(). + * + * Note that stack backlinks are not properly setup by + * copy_thread() and thus, a forked task() will have + * an unreliable stack trace until it's been + * _switch()'ed to for the first time. + */ + stack_end -= STACK_FRAME_OVERHEAD + sizeof(struct pt_regs); + } else { + /* + * idle tasks have a custom stack layout, + * c.f. cpu_idle_thread_init(). + */ + stack_end -= STACK_FRAME_OVERHEAD; + } + + if (sp < stack_page + sizeof(struct thread_struct) || + sp > stack_end - STACK_FRAME_MIN_SIZE) { + return 1; + } + + for (;;) { + unsigned long *stack = (unsigned long *) sp; + unsigned long newsp, ip; + + /* sanity check: ABI requires SP to be aligned 16 bytes. */ + if (sp & 0xF) + return 1; + + /* Mark stacktraces with exception frames as unreliable. */ + if (sp <= stack_end - STACK_INT_FRAME_SIZE && + stack[STACK_FRAME_MARKER] == STACK_FRAME_REGS_MARKER) { + return 1; + } + + newsp = stack[0]; + /* Stack grows downwards; unwinder may only go up. */ + if (newsp <= sp) + return 1; + + if (newsp != stack_end && + newsp > stack_end - STACK_FRAME_MIN_SIZE) { + return 1; /* invalid backlink, too far up. */ + } + + /* Examine the saved LR: it must point into kernel code. */ + ip = stack[STACK_FRAME_LR_SAVE]; + if (!firstframe && !__kernel_text_address(ip)) + return 1; + firstframe = 0; + + /* + * FIXME: IMHO these tests do not belong in + * arch-dependent code, they are generic. + */ + ip = ftrace_graph_ret_addr(tsk, &graph_idx, ip, NULL); +#ifdef CONFIG_KPROBES + /* + * Mark stacktraces with kretprobed functions on them + * as unreliable. + */ + if (ip == (unsigned long)kretprobe_trampoline) + return 1; +#endif + + if (!trace->skip) + trace->entries[trace->nr_entries++] = ip; + else + trace->skip--; + + if (newsp == stack_end) + break; + + if (trace->nr_entries >= trace->max_entries) + return -E2BIG; + + sp = newsp; + } + return 0; +} +EXPORT_SYMBOL_GPL(save_stack_trace_tsk_reliable); +#endif /* CONFIG_HAVE_RELIABLE_STACKTRACE */ + +#ifdef CONFIG_PPC_BOOK3S_64 +static void handle_backtrace_ipi(struct pt_regs *regs) +{ + nmi_cpu_backtrace(regs); +} + +static void raise_backtrace_ipi(cpumask_t *mask) +{ + unsigned int cpu; + + for_each_cpu(cpu, mask) { + if (cpu == smp_processor_id()) + handle_backtrace_ipi(NULL); + else + smp_send_safe_nmi_ipi(cpu, handle_backtrace_ipi, 5 * USEC_PER_SEC); + } + + for_each_cpu(cpu, mask) { + struct paca_struct *p = paca_ptrs[cpu]; + + cpumask_clear_cpu(cpu, mask); + + pr_warn("CPU %d didn't respond to backtrace IPI, inspecting paca.\n", cpu); + if (!virt_addr_valid(p)) { + pr_warn("paca pointer appears corrupt? (%px)\n", p); + continue; + } + + pr_warn("irq_soft_mask: 0x%02x in_mce: %d in_nmi: %d", + p->irq_soft_mask, p->in_mce, p->in_nmi); + + if (virt_addr_valid(p->__current)) + pr_cont(" current: %d (%s)\n", p->__current->pid, + p->__current->comm); + else + pr_cont(" current pointer corrupt? (%px)\n", p->__current); + + pr_warn("Back trace of paca->saved_r1 (0x%016llx) (possibly stale):\n", p->saved_r1); + show_stack(p->__current, (unsigned long *)p->saved_r1); + } +} + +void arch_trigger_cpumask_backtrace(const cpumask_t *mask, bool exclude_self) +{ + nmi_trigger_cpumask_backtrace(mask, exclude_self, raise_backtrace_ipi); +} +#endif /* CONFIG_PPC64 */ diff --git a/arch/powerpc/kernel/sys_ppc32.c b/arch/powerpc/kernel/sys_ppc32.c index c11c73373691..bdf58ba1a94b 100644 --- a/arch/powerpc/kernel/sys_ppc32.c +++ b/arch/powerpc/kernel/sys_ppc32.c @@ -52,15 +52,6 @@ #include #include - -asmlinkage long ppc32_select(u32 n, compat_ulong_t __user *inp, - compat_ulong_t __user *outp, compat_ulong_t __user *exp, - compat_uptr_t tvp_x) -{ - /* sign extend n */ - return compat_sys_select((int)n, inp, outp, exp, compat_ptr(tvp_x)); -} - unsigned long compat_sys_mmap2(unsigned long addr, size_t len, unsigned long prot, unsigned long flags, unsigned long fd, unsigned long pgoff) diff --git a/arch/powerpc/kernel/syscalls.c b/arch/powerpc/kernel/syscalls.c index 466216506eb2..083fa06962fd 100644 --- a/arch/powerpc/kernel/syscalls.c +++ b/arch/powerpc/kernel/syscalls.c @@ -62,6 +62,9 @@ out: return ret; } +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpragmas" +#pragma GCC diagnostic ignored "-Wattribute-alias" SYSCALL_DEFINE6(mmap2, unsigned long, addr, size_t, len, unsigned long, prot, unsigned long, flags, unsigned long, fd, unsigned long, pgoff) @@ -75,6 +78,7 @@ SYSCALL_DEFINE6(mmap, unsigned long, addr, size_t, len, { return do_mmap2(addr, len, prot, flags, fd, offset, PAGE_SHIFT); } +#pragma GCC diagnostic pop #ifdef CONFIG_PPC32 /* diff --git a/arch/powerpc/kernel/systbl.S b/arch/powerpc/kernel/systbl.S index 7ccb7f81f8db..919a32746ede 100644 --- a/arch/powerpc/kernel/systbl.S +++ b/arch/powerpc/kernel/systbl.S @@ -35,7 +35,7 @@ #endif #define SYSCALL_SPU(func) SYSCALL(func) #define COMPAT_SYS_SPU(func) COMPAT_SYS(func) -#define PPC_SYS_SPU(func) PPC_SYS(func) +#define COMPAT_SPU_NEW(func) COMPAT_SYS(func) #define SYSX_SPU(f, f3264, f32) SYSX(f, f3264, f32) .section .rodata,"a" diff --git a/arch/powerpc/kernel/systbl_chk.c b/arch/powerpc/kernel/systbl_chk.c index 55323a620cfe..4653258722ac 100644 --- a/arch/powerpc/kernel/systbl_chk.c +++ b/arch/powerpc/kernel/systbl_chk.c @@ -31,7 +31,7 @@ #define SYSCALL_SPU(func) SYSCALL(func) #define COMPAT_SYS_SPU(func) COMPAT_SYS(func) -#define PPC_SYS_SPU(func) PPC_SYS(func) +#define COMPAT_SPU_NEW(func) COMPAT_SYS(_new##func) #define SYSX_SPU(f, f3264, f32) SYSX(f, f3264, f32) /* Just insert a marker for ni_syscalls */ diff --git a/arch/powerpc/kernel/systbl_chk.sh b/arch/powerpc/kernel/systbl_chk.sh index 31b6e7c358ca..f2e356c2a345 100644 --- a/arch/powerpc/kernel/systbl_chk.sh +++ b/arch/powerpc/kernel/systbl_chk.sh @@ -16,7 +16,7 @@ awk 'BEGIN { num = -1; } # Ignore the beginning of the file /^START_TABLE/ { num = 0; next; } /^END_TABLE/ { if (num != $2) { - printf "NR_syscalls (%s) is not one more than the last syscall (%s)\n", + printf "Error: NR_syscalls (%s) is not one more than the last syscall (%s)\n", $2, num - 1; exit(1); } @@ -25,7 +25,7 @@ awk 'BEGIN { num = -1; } # Ignore the beginning of the file { if (num == -1) next; if (($1 != -1) && ($1 != num)) { - printf "Syscall %s out of order (expected %s)\n", + printf "Error: Syscall %s out of order (expected %s)\n", $1, num; exit(1); }; diff --git a/arch/powerpc/kernel/tau_6xx.c b/arch/powerpc/kernel/tau_6xx.c index 8cdd852aedd1..e2ab8a111b69 100644 --- a/arch/powerpc/kernel/tau_6xx.c +++ b/arch/powerpc/kernel/tau_6xx.c @@ -27,6 +27,9 @@ #include #include #include +#include + +#include "setup.h" static struct tau_temp { @@ -50,7 +53,7 @@ struct timer_list tau_timer; #define shrink_timer 2*HZ /* period between shrinking the window */ #define min_window 2 /* minimum window size, degrees C */ -void set_thresholds(unsigned long cpu) +static void set_thresholds(unsigned long cpu) { #ifdef CONFIG_TAU_INT /* @@ -70,7 +73,7 @@ void set_thresholds(unsigned long cpu) #endif } -void TAUupdate(int cpu) +static void TAUupdate(int cpu) { unsigned thrm; @@ -205,7 +208,7 @@ static void tau_timeout_smp(struct timer_list *unused) int tau_initialized = 0; -void __init TAU_init_smp(void * info) +static void __init TAU_init_smp(void *info) { unsigned long cpu = smp_processor_id(); @@ -217,7 +220,7 @@ void __init TAU_init_smp(void * info) set_thresholds(cpu); } -int __init TAU_init(void) +static int __init TAU_init(void) { /* We assume in SMP that if one CPU has TAU support, they * all have it --BenH @@ -259,12 +262,12 @@ u32 cpu_temp_both(unsigned long cpu) return ((tau[cpu].high << 16) | tau[cpu].low); } -int cpu_temp(unsigned long cpu) +u32 cpu_temp(unsigned long cpu) { return ((tau[cpu].high + tau[cpu].low) / 2); } -int tau_interrupts(unsigned long cpu) +u32 tau_interrupts(unsigned long cpu) { return (tau[cpu].interrupts); } diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c index 360e71d455cc..70f145e02487 100644 --- a/arch/powerpc/kernel/time.c +++ b/arch/powerpc/kernel/time.c @@ -163,12 +163,6 @@ EXPORT_SYMBOL(__cputime_usec_factor); void (*dtl_consumer)(struct dtl_entry *, u64); #endif -#ifdef CONFIG_PPC64 -#define get_accounting(tsk) (&get_paca()->accounting) -#else -#define get_accounting(tsk) (&task_thread_info(tsk)->accounting) -#endif - static void calc_cputime_factors(void) { struct div_result res; @@ -421,21 +415,6 @@ void vtime_flush(struct task_struct *tsk) acct->softirq_time = 0; } -#ifdef CONFIG_PPC32 -/* - * Called from the context switch with interrupts disabled, to charge all - * accumulated times to the current process, and to prepare accounting on - * the next process. - */ -void arch_vtime_task_switch(struct task_struct *prev) -{ - struct cpu_accounting_data *acct = get_accounting(current); - - acct->starttime = get_accounting(prev)->starttime; - acct->startspurr = get_accounting(prev)->startspurr; -} -#endif /* CONFIG_PPC32 */ - #else /* ! CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ #define calc_cputime_factors() #endif @@ -513,6 +492,35 @@ static inline void clear_irq_work_pending(void) "i" (offsetof(struct paca_struct, irq_work_pending))); } +void arch_irq_work_raise(void) +{ + preempt_disable(); + set_irq_work_pending_flag(); + /* + * Non-nmi code running with interrupts disabled will replay + * irq_happened before it re-enables interrupts, so setthe + * decrementer there instead of causing a hardware exception + * which would immediately hit the masked interrupt handler + * and have the net effect of setting the decrementer in + * irq_happened. + * + * NMI interrupts can not check this when they return, so the + * decrementer hardware exception is raised, which will fire + * when interrupts are next enabled. + * + * BookE does not support this yet, it must audit all NMI + * interrupt handlers to ensure they call nmi_enter() so this + * check would be correct. + */ + if (IS_ENABLED(CONFIG_BOOKE) || !irqs_disabled() || in_nmi()) { + set_dec(1); + } else { + hard_irq_disable(); + local_paca->irq_happened |= PACA_IRQ_DEC; + } + preempt_enable(); +} + #else /* 32-bit */ DEFINE_PER_CPU(u8, irq_work_pending); @@ -521,8 +529,6 @@ DEFINE_PER_CPU(u8, irq_work_pending); #define test_irq_work_pending() __this_cpu_read(irq_work_pending) #define clear_irq_work_pending() __this_cpu_write(irq_work_pending, 0) -#endif /* 32 vs 64 bit */ - void arch_irq_work_raise(void) { preempt_disable(); @@ -531,6 +537,8 @@ void arch_irq_work_raise(void) preempt_enable(); } +#endif /* 32 vs 64 bit */ + #else /* CONFIG_IRQ_WORK */ #define test_irq_work_pending() 0 @@ -538,60 +546,16 @@ void arch_irq_work_raise(void) #endif /* CONFIG_IRQ_WORK */ -static void __timer_interrupt(void) -{ - struct pt_regs *regs = get_irq_regs(); - u64 *next_tb = this_cpu_ptr(&decrementers_next_tb); - struct clock_event_device *evt = this_cpu_ptr(&decrementers); - u64 now; - - trace_timer_interrupt_entry(regs); - - if (test_irq_work_pending()) { - clear_irq_work_pending(); - irq_work_run(); - } - - now = get_tb_or_rtc(); - if (now >= *next_tb) { - *next_tb = ~(u64)0; - if (evt->event_handler) - evt->event_handler(evt); - __this_cpu_inc(irq_stat.timer_irqs_event); - } else { - now = *next_tb - now; - if (now <= decrementer_max) - set_dec(now); - /* We may have raced with new irq work */ - if (test_irq_work_pending()) - set_dec(1); - __this_cpu_inc(irq_stat.timer_irqs_others); - } - -#ifdef CONFIG_PPC64 - /* collect purr register values often, for accurate calculations */ - if (firmware_has_feature(FW_FEATURE_SPLPAR)) { - struct cpu_usage *cu = this_cpu_ptr(&cpu_usage_array); - cu->current_tb = mfspr(SPRN_PURR); - } -#endif - - trace_timer_interrupt_exit(regs); -} - /* * timer_interrupt - gets called when the decrementer overflows, * with interrupts disabled. */ -void timer_interrupt(struct pt_regs * regs) +void timer_interrupt(struct pt_regs *regs) { - struct pt_regs *old_regs; + struct clock_event_device *evt = this_cpu_ptr(&decrementers); u64 *next_tb = this_cpu_ptr(&decrementers_next_tb); - - /* Ensure a positive value is written to the decrementer, or else - * some CPUs will continue to take decrementer exceptions. - */ - set_dec(decrementer_max); + struct pt_regs *old_regs; + u64 now; /* Some implementations of hotplug will get timer interrupts while * offline, just ignore these and we also need to set @@ -599,11 +563,23 @@ void timer_interrupt(struct pt_regs * regs) * don't replay timer interrupt when return, otherwise we'll trap * here infinitely :( */ - if (!cpu_online(smp_processor_id())) { + if (unlikely(!cpu_online(smp_processor_id()))) { *next_tb = ~(u64)0; + set_dec(decrementer_max); return; } + /* Ensure a positive value is written to the decrementer, or else + * some CPUs will continue to take decrementer exceptions. When the + * PPC_WATCHDOG (decrementer based) is configured, keep this at most + * 31 bits, which is about 4 seconds on most systems, which gives + * the watchdog a chance of catching timer interrupt hard lockups. + */ + if (IS_ENABLED(CONFIG_PPC_WATCHDOG)) + set_dec(0x7fffffff); + else + set_dec(decrementer_max); + /* Conditionally hard-enable interrupts now that the DEC has been * bumped to its maximum value */ @@ -617,13 +593,46 @@ void timer_interrupt(struct pt_regs * regs) old_regs = set_irq_regs(regs); irq_enter(); + trace_timer_interrupt_entry(regs); + + if (test_irq_work_pending()) { + clear_irq_work_pending(); + irq_work_run(); + } + + now = get_tb_or_rtc(); + if (now >= *next_tb) { + *next_tb = ~(u64)0; + if (evt->event_handler) + evt->event_handler(evt); + __this_cpu_inc(irq_stat.timer_irqs_event); + } else { + now = *next_tb - now; + if (now <= decrementer_max) + set_dec(now); + /* We may have raced with new irq work */ + if (test_irq_work_pending()) + set_dec(1); + __this_cpu_inc(irq_stat.timer_irqs_others); + } - __timer_interrupt(); + trace_timer_interrupt_exit(regs); irq_exit(); set_irq_regs(old_regs); } EXPORT_SYMBOL(timer_interrupt); +#ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST +void timer_broadcast_interrupt(void) +{ + u64 *next_tb = this_cpu_ptr(&decrementers_next_tb); + + *next_tb = ~(u64)0; + tick_receive_broadcast(); + __this_cpu_inc(irq_stat.broadcast_irqs_event); +} +#endif + /* * Hypervisor decrementer interrupts shouldn't occur but are sometimes * left pending on exit from a KVM guest. We don't need to do anything @@ -781,21 +790,19 @@ void __init generic_calibrate_decr(void) } } -int update_persistent_clock(struct timespec now) +int update_persistent_clock64(struct timespec64 now) { struct rtc_time tm; if (!ppc_md.set_rtc_time) return -ENODEV; - to_tm(now.tv_sec + 1 + timezone_offset, &tm); - tm.tm_year -= 1900; - tm.tm_mon -= 1; + rtc_time64_to_tm(now.tv_sec + 1 + timezone_offset, &tm); return ppc_md.set_rtc_time(&tm); } -static void __read_persistent_clock(struct timespec *ts) +static void __read_persistent_clock(struct timespec64 *ts) { struct rtc_time tm; static int first = 1; @@ -819,11 +826,10 @@ static void __read_persistent_clock(struct timespec *ts) } ppc_md.get_rtc_time(&tm); - ts->tv_sec = mktime(tm.tm_year+1900, tm.tm_mon+1, tm.tm_mday, - tm.tm_hour, tm.tm_min, tm.tm_sec); + ts->tv_sec = rtc_tm_to_time64(&tm); } -void read_persistent_clock(struct timespec *ts) +void read_persistent_clock64(struct timespec64 *ts) { __read_persistent_clock(ts); @@ -971,15 +977,6 @@ static int decrementer_shutdown(struct clock_event_device *dev) return 0; } -/* Interrupt handler for the timer broadcast IPI */ -void tick_broadcast_ipi_handler(void) -{ - u64 *next_tb = this_cpu_ptr(&decrementers_next_tb); - - *next_tb = get_tb_or_rtc(); - __timer_interrupt(); -} - static void register_decrementer_clockevent(int cpu) { struct clock_event_device *dec = &per_cpu(decrementers, cpu); @@ -1141,56 +1138,6 @@ void __init time_init(void) #endif } - -#define FEBRUARY 2 -#define STARTOFTIME 1970 -#define SECDAY 86400L -#define SECYR (SECDAY * 365) -#define leapyear(year) ((year) % 4 == 0 && \ - ((year) % 100 != 0 || (year) % 400 == 0)) -#define days_in_year(a) (leapyear(a) ? 366 : 365) -#define days_in_month(a) (month_days[(a) - 1]) - -static int month_days[12] = { - 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 -}; - -void to_tm(int tim, struct rtc_time * tm) -{ - register int i; - register long hms, day; - - day = tim / SECDAY; - hms = tim % SECDAY; - - /* Hours, minutes, seconds are easy */ - tm->tm_hour = hms / 3600; - tm->tm_min = (hms % 3600) / 60; - tm->tm_sec = (hms % 3600) % 60; - - /* Number of years in days */ - for (i = STARTOFTIME; day >= days_in_year(i); i++) - day -= days_in_year(i); - tm->tm_year = i; - - /* Number of months in days left */ - if (leapyear(tm->tm_year)) - days_in_month(FEBRUARY) = 29; - for (i = 1; day >= days_in_month(i); i++) - day -= days_in_month(i); - days_in_month(FEBRUARY) = 28; - tm->tm_mon = i; - - /* Days are what is left over (+1) from all that. */ - tm->tm_mday = day + 1; - - /* - * No-one uses the day of the week. - */ - tm->tm_wday = -1; -} -EXPORT_SYMBOL(to_tm); - /* * Divide a 128-bit dividend by a 32-bit divisor, leaving a 128 bit * result. diff --git a/arch/powerpc/kernel/tm.S b/arch/powerpc/kernel/tm.S index b92ac8e711db..ff12f47a96b6 100644 --- a/arch/powerpc/kernel/tm.S +++ b/arch/powerpc/kernel/tm.S @@ -12,6 +12,7 @@ #include #include #include +#include #ifdef CONFIG_VSX /* See fpu.S, this is borrowed from there */ @@ -55,6 +56,16 @@ _GLOBAL(tm_enable) or r4, r4, r3 mtmsrd r4 1: blr +EXPORT_SYMBOL_GPL(tm_enable); + +_GLOBAL(tm_disable) + mfmsr r4 + li r3, MSR_TM >> 32 + sldi r3, r3, 32 + andc r4, r4, r3 + mtmsrd r4 + blr +EXPORT_SYMBOL_GPL(tm_disable); _GLOBAL(tm_save_sprs) mfspr r0, SPRN_TFHAR @@ -78,6 +89,7 @@ _GLOBAL(tm_restore_sprs) _GLOBAL(tm_abort) TABORT(R3) blr +EXPORT_SYMBOL_GPL(tm_abort); /* void tm_reclaim(struct thread_struct *thread, * uint8_t cause) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 4741fe112f05..4bfbb54dee51 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -144,7 +144,7 @@ __ftrace_make_nop(struct module *mod, return -EINVAL; } -#ifdef CC_USING_MPROFILE_KERNEL +#ifdef CONFIG_MPROFILE_KERNEL /* When using -mkernel_profile there is no load to jump over */ pop = PPC_INST_NOP; @@ -188,7 +188,7 @@ __ftrace_make_nop(struct module *mod, pr_err("Expected %08x found %08x\n", PPC_INST_LD_TOC, op); return -EINVAL; } -#endif /* CC_USING_MPROFILE_KERNEL */ +#endif /* CONFIG_MPROFILE_KERNEL */ if (patch_instruction((unsigned int *)ip, pop)) { pr_err("Patching NOP failed.\n"); @@ -324,7 +324,7 @@ int ftrace_make_nop(struct module *mod, * They should effectively be a NOP, and follow formal constraints, * depending on the ABI. Return false if they don't. */ -#ifndef CC_USING_MPROFILE_KERNEL +#ifndef CONFIG_MPROFILE_KERNEL static int expected_nop_sequence(void *ip, unsigned int op0, unsigned int op1) { @@ -357,6 +357,8 @@ __ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) { unsigned int op[2]; void *ip = (void *)rec->ip; + unsigned long entry, ptr, tramp; + struct module *mod = rec->arch.mod; /* read where this goes */ if (probe_kernel_read(op, ip, sizeof(op))) @@ -368,19 +370,44 @@ __ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) return -EINVAL; } - /* If we never set up a trampoline to ftrace_caller, then bail */ - if (!rec->arch.mod->arch.tramp) { + /* If we never set up ftrace trampoline(s), then bail */ +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS + if (!mod->arch.tramp || !mod->arch.tramp_regs) { +#else + if (!mod->arch.tramp) { +#endif pr_err("No ftrace trampoline\n"); return -EINVAL; } +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS + if (rec->flags & FTRACE_FL_REGS) + tramp = mod->arch.tramp_regs; + else +#endif + tramp = mod->arch.tramp; + + if (module_trampoline_target(mod, tramp, &ptr)) { + pr_err("Failed to get trampoline target\n"); + return -EFAULT; + } + + pr_devel("trampoline target %lx", ptr); + + entry = ppc_global_function_entry((void *)addr); + /* This should match what was called */ + if (ptr != entry) { + pr_err("addr %lx does not match expected %lx\n", ptr, entry); + return -EINVAL; + } + /* Ensure branch is within 24 bits */ - if (!create_branch(ip, rec->arch.mod->arch.tramp, BRANCH_SET_LINK)) { + if (!create_branch(ip, tramp, BRANCH_SET_LINK)) { pr_err("Branch out of range\n"); return -EINVAL; } - if (patch_branch(ip, rec->arch.mod->arch.tramp, BRANCH_SET_LINK)) { + if (patch_branch(ip, tramp, BRANCH_SET_LINK)) { pr_err("REL24 out of range!\n"); return -EINVAL; } @@ -388,14 +415,6 @@ __ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) return 0; } -#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS -int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, - unsigned long addr) -{ - return ftrace_make_call(rec, addr); -} -#endif - #else /* !CONFIG_PPC64: */ static int __ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) @@ -472,53 +491,158 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) #endif /* CONFIG_MODULES */ } -int ftrace_update_ftrace_func(ftrace_func_t func) +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS +#ifdef CONFIG_MODULES +static int +__ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, + unsigned long addr) { - unsigned long ip = (unsigned long)(&ftrace_call); - unsigned int old, new; - int ret; + unsigned int op; + unsigned long ip = rec->ip; + unsigned long entry, ptr, tramp; + struct module *mod = rec->arch.mod; - old = *(unsigned int *)&ftrace_call; - new = ftrace_call_replace(ip, (unsigned long)func, 1); - ret = ftrace_modify_code(ip, old, new); + /* If we never set up ftrace trampolines, then bail */ + if (!mod->arch.tramp || !mod->arch.tramp_regs) { + pr_err("No ftrace trampoline\n"); + return -EINVAL; + } - return ret; -} + /* read where this goes */ + if (probe_kernel_read(&op, (void *)ip, sizeof(int))) { + pr_err("Fetching opcode failed.\n"); + return -EFAULT; + } -static int __ftrace_replace_code(struct dyn_ftrace *rec, int enable) -{ - unsigned long ftrace_addr = (unsigned long)FTRACE_ADDR; - int ret; + /* Make sure that that this is still a 24bit jump */ + if (!is_bl_op(op)) { + pr_err("Not expected bl: opcode is %x\n", op); + return -EINVAL; + } + + /* lets find where the pointer goes */ + tramp = find_bl_target(ip, op); + entry = ppc_global_function_entry((void *)old_addr); + + pr_devel("ip:%lx jumps to %lx", ip, tramp); - ret = ftrace_update_record(rec, enable); + if (tramp != entry) { + /* old_addr is not within range, so we must have used a trampoline */ + if (module_trampoline_target(mod, tramp, &ptr)) { + pr_err("Failed to get trampoline target\n"); + return -EFAULT; + } + + pr_devel("trampoline target %lx", ptr); + + /* This should match what was called */ + if (ptr != entry) { + pr_err("addr %lx does not match expected %lx\n", ptr, entry); + return -EINVAL; + } + } + + /* The new target may be within range */ + if (test_24bit_addr(ip, addr)) { + /* within range */ + if (patch_branch((unsigned int *)ip, addr, BRANCH_SET_LINK)) { + pr_err("REL24 out of range!\n"); + return -EINVAL; + } - switch (ret) { - case FTRACE_UPDATE_IGNORE: return 0; - case FTRACE_UPDATE_MAKE_CALL: - return ftrace_make_call(rec, ftrace_addr); - case FTRACE_UPDATE_MAKE_NOP: - return ftrace_make_nop(NULL, rec, ftrace_addr); + } + + if (rec->flags & FTRACE_FL_REGS) + tramp = mod->arch.tramp_regs; + else + tramp = mod->arch.tramp; + + if (module_trampoline_target(mod, tramp, &ptr)) { + pr_err("Failed to get trampoline target\n"); + return -EFAULT; + } + + pr_devel("trampoline target %lx", ptr); + + entry = ppc_global_function_entry((void *)addr); + /* This should match what was called */ + if (ptr != entry) { + pr_err("addr %lx does not match expected %lx\n", ptr, entry); + return -EINVAL; + } + + /* Ensure branch is within 24 bits */ + if (!create_branch((unsigned int *)ip, tramp, BRANCH_SET_LINK)) { + pr_err("Branch out of range\n"); + return -EINVAL; + } + + if (patch_branch((unsigned int *)ip, tramp, BRANCH_SET_LINK)) { + pr_err("REL24 out of range!\n"); + return -EINVAL; } return 0; } +#endif -void ftrace_replace_code(int enable) +int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, + unsigned long addr) { - struct ftrace_rec_iter *iter; - struct dyn_ftrace *rec; + unsigned long ip = rec->ip; + unsigned int old, new; + + /* + * If the calling address is more that 24 bits away, + * then we had to use a trampoline to make the call. + * Otherwise just update the call site. + */ + if (test_24bit_addr(ip, addr) && test_24bit_addr(ip, old_addr)) { + /* within range */ + old = ftrace_call_replace(ip, old_addr, 1); + new = ftrace_call_replace(ip, addr, 1); + return ftrace_modify_code(ip, old, new); + } + +#ifdef CONFIG_MODULES + /* + * Out of range jumps are called from modules. + */ + if (!rec->arch.mod) { + pr_err("No module loaded\n"); + return -EINVAL; + } + + return __ftrace_modify_call(rec, old_addr, addr); +#else + /* We should not get here without modules */ + return -EINVAL; +#endif /* CONFIG_MODULES */ +} +#endif + +int ftrace_update_ftrace_func(ftrace_func_t func) +{ + unsigned long ip = (unsigned long)(&ftrace_call); + unsigned int old, new; int ret; - for (iter = ftrace_rec_iter_start(); iter; - iter = ftrace_rec_iter_next(iter)) { - rec = ftrace_rec_iter_record(iter); - ret = __ftrace_replace_code(rec, enable); - if (ret) { - ftrace_bug(ret, rec); - return; - } + old = *(unsigned int *)&ftrace_call; + new = ftrace_call_replace(ip, (unsigned long)func, 1); + ret = ftrace_modify_code(ip, old, new); + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS + /* Also update the regs callback function */ + if (!ret) { + ip = (unsigned long)(&ftrace_regs_call); + old = *(unsigned int *)&ftrace_regs_call; + new = ftrace_call_replace(ip, (unsigned long)func, 1); + ret = ftrace_modify_code(ip, old, new); } +#endif + + return ret; } /* @@ -538,7 +662,6 @@ int __init ftrace_dyn_arch_init(void) #ifdef CONFIG_FUNCTION_GRAPH_TRACER -#ifdef CONFIG_DYNAMIC_FTRACE extern void ftrace_graph_call(void); extern void ftrace_graph_stub(void); @@ -567,7 +690,6 @@ int ftrace_disable_ftrace_graph_caller(void) return ftrace_modify_code(ip, old, new); } -#endif /* CONFIG_DYNAMIC_FTRACE */ /* * Hook the return address and push it in the stack of return addrs diff --git a/arch/powerpc/kernel/trace/ftrace_32.S b/arch/powerpc/kernel/trace/ftrace_32.S index afef2c076282..2c29098f630f 100644 --- a/arch/powerpc/kernel/trace/ftrace_32.S +++ b/arch/powerpc/kernel/trace/ftrace_32.S @@ -14,7 +14,6 @@ #include #include -#ifdef CONFIG_DYNAMIC_FTRACE _GLOBAL(mcount) _GLOBAL(_mcount) /* @@ -47,26 +46,7 @@ _GLOBAL(ftrace_graph_stub) MCOUNT_RESTORE_FRAME /* old link register ends up in ctr reg */ bctr -#else -_GLOBAL(mcount) -_GLOBAL(_mcount) - - MCOUNT_SAVE_FRAME - subi r3, r3, MCOUNT_INSN_SIZE - LOAD_REG_ADDR(r5, ftrace_trace_function) - lwz r5,0(r5) - - mtctr r5 - bctrl - nop - -#ifdef CONFIG_FUNCTION_GRAPH_TRACER - b ftrace_graph_caller -#endif - MCOUNT_RESTORE_FRAME - bctr -#endif EXPORT_SYMBOL(_mcount) _GLOBAL(ftrace_stub) diff --git a/arch/powerpc/kernel/trace/ftrace_64.S b/arch/powerpc/kernel/trace/ftrace_64.S index e5ccea19821e..e25f77c10a72 100644 --- a/arch/powerpc/kernel/trace/ftrace_64.S +++ b/arch/powerpc/kernel/trace/ftrace_64.S @@ -14,7 +14,6 @@ #include #include -#ifdef CONFIG_DYNAMIC_FTRACE _GLOBAL(mcount) _GLOBAL(_mcount) EXPORT_SYMBOL(_mcount) @@ -23,34 +22,6 @@ EXPORT_SYMBOL(_mcount) mtlr r0 bctr -#else /* CONFIG_DYNAMIC_FTRACE */ -_GLOBAL_TOC(_mcount) -EXPORT_SYMBOL(_mcount) - /* Taken from output of objdump from lib64/glibc */ - mflr r3 - ld r11, 0(r1) - stdu r1, -112(r1) - std r3, 128(r1) - ld r4, 16(r11) - - subi r3, r3, MCOUNT_INSN_SIZE - LOAD_REG_ADDR(r5,ftrace_trace_function) - ld r5,0(r5) - ld r5,0(r5) - mtctr r5 - bctrl - nop - -#ifdef CONFIG_FUNCTION_GRAPH_TRACER - b ftrace_graph_caller -#endif - ld r0, 128(r1) - mtlr r0 - addi r1, r1, 112 -_GLOBAL(ftrace_stub) - blr -#endif /* CONFIG_DYNAMIC_FTRACE */ - #ifdef CONFIG_FUNCTION_GRAPH_TRACER _GLOBAL(return_to_handler) /* need to save return values */ diff --git a/arch/powerpc/kernel/trace/ftrace_64_mprofile.S b/arch/powerpc/kernel/trace/ftrace_64_mprofile.S index 3f3e81852422..9a5b5a513604 100644 --- a/arch/powerpc/kernel/trace/ftrace_64_mprofile.S +++ b/arch/powerpc/kernel/trace/ftrace_64_mprofile.S @@ -17,11 +17,10 @@ #include #include -#ifdef CONFIG_DYNAMIC_FTRACE /* * - * ftrace_caller() is the function that replaces _mcount() when ftrace is - * active. + * ftrace_caller()/ftrace_regs_caller() is the function that replaces _mcount() + * when ftrace is active. * * We arrive here after a function A calls function B, and we are the trace * function for B. When we enter r1 points to A's stack frame, B has not yet @@ -37,7 +36,7 @@ * Our job is to save the register state into a struct pt_regs (on the stack) * and then arrange for the ftrace function to be called. */ -_GLOBAL(ftrace_caller) +_GLOBAL(ftrace_regs_caller) /* Save the original return address in A's stack frame */ std r0,LRSAVE(r1) @@ -47,6 +46,12 @@ _GLOBAL(ftrace_caller) /* Save all gprs to pt_regs */ SAVE_GPR(0, r1) SAVE_10GPRS(2, r1) + + /* Ok to continue? */ + lbz r3, PACA_FTRACE_ENABLED(r13) + cmpdi r3, 0 + beq ftrace_no_trace + SAVE_10GPRS(12, r1) SAVE_10GPRS(22, r1) @@ -94,8 +99,8 @@ _GLOBAL(ftrace_caller) addi r6, r1 ,STACK_FRAME_OVERHEAD /* ftrace_call(r3, r4, r5, r6) */ -.globl ftrace_call -ftrace_call: +.globl ftrace_regs_call +ftrace_regs_call: bl ftrace_stub nop @@ -156,6 +161,7 @@ ftrace_call: bne- livepatch_handler #endif +ftrace_caller_common: #ifdef CONFIG_FUNCTION_GRAPH_TRACER .globl ftrace_graph_call ftrace_graph_call: @@ -168,6 +174,74 @@ _GLOBAL(ftrace_graph_stub) _GLOBAL(ftrace_stub) blr +ftrace_no_trace: + mflr r3 + mtctr r3 + REST_GPR(3, r1) + addi r1, r1, SWITCH_FRAME_SIZE + mtlr r0 + bctr + +_GLOBAL(ftrace_caller) + /* Save the original return address in A's stack frame */ + std r0, LRSAVE(r1) + + /* Create our stack frame + pt_regs */ + stdu r1, -SWITCH_FRAME_SIZE(r1) + + /* Save all gprs to pt_regs */ + SAVE_8GPRS(3, r1) + + lbz r3, PACA_FTRACE_ENABLED(r13) + cmpdi r3, 0 + beq ftrace_no_trace + + /* Get the _mcount() call site out of LR */ + mflr r7 + std r7, _NIP(r1) + + /* Save callee's TOC in the ABI compliant location */ + std r2, 24(r1) + ld r2, PACATOC(r13) /* get kernel TOC in r2 */ + + addis r3, r2, function_trace_op@toc@ha + addi r3, r3, function_trace_op@toc@l + ld r5, 0(r3) + + /* Calculate ip from nip-4 into r3 for call below */ + subi r3, r7, MCOUNT_INSN_SIZE + + /* Put the original return address in r4 as parent_ip */ + mr r4, r0 + + /* Set pt_regs to NULL */ + li r6, 0 + + /* ftrace_call(r3, r4, r5, r6) */ +.globl ftrace_call +ftrace_call: + bl ftrace_stub + nop + + ld r3, _NIP(r1) + mtctr r3 + + /* Restore gprs */ + REST_8GPRS(3,r1) + + /* Restore callee's TOC */ + ld r2, 24(r1) + + /* Pop our stack frame */ + addi r1, r1, SWITCH_FRAME_SIZE + + /* Reload original LR */ + ld r0, LRSAVE(r1) + mtlr r0 + + /* Handle function_graph or go back */ + b ftrace_caller_common + #ifdef CONFIG_LIVEPATCH /* * This function runs in the mcount context, between two functions. As @@ -236,8 +310,6 @@ livepatch_handler: blr #endif /* CONFIG_LIVEPATCH */ -#endif /* CONFIG_DYNAMIC_FTRACE */ - #ifdef CONFIG_FUNCTION_GRAPH_TRACER _GLOBAL(ftrace_graph_caller) stdu r1, -112(r1) diff --git a/arch/powerpc/kernel/trace/ftrace_64_pg.S b/arch/powerpc/kernel/trace/ftrace_64_pg.S index f095358da96e..4c515c4023de 100644 --- a/arch/powerpc/kernel/trace/ftrace_64_pg.S +++ b/arch/powerpc/kernel/trace/ftrace_64_pg.S @@ -14,8 +14,11 @@ #include #include -#ifdef CONFIG_DYNAMIC_FTRACE _GLOBAL_TOC(ftrace_caller) + lbz r3, PACA_FTRACE_ENABLED(r13) + cmpdi r3, 0 + beqlr + /* Taken from output of objdump from lib64/glibc */ mflr r3 ld r11, 0(r1) @@ -39,7 +42,6 @@ _GLOBAL(ftrace_graph_stub) _GLOBAL(ftrace_stub) blr -#endif /* CONFIG_DYNAMIC_FTRACE */ #ifdef CONFIG_FUNCTION_GRAPH_TRACER _GLOBAL(ftrace_graph_caller) diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c index 0904492e7032..0e17dcb48720 100644 --- a/arch/powerpc/kernel/traps.c +++ b/arch/powerpc/kernel/traps.c @@ -296,7 +296,6 @@ NOKPROBE_SYMBOL(die); void user_single_step_siginfo(struct task_struct *tsk, struct pt_regs *regs, siginfo_t *info) { - memset(info, 0, sizeof(*info)); info->si_signo = SIGTRAP; info->si_code = TRAP_TRACE; info->si_addr = (void __user *)regs->nip; @@ -334,7 +333,7 @@ void _exception_pkey(int signr, struct pt_regs *regs, int code, */ thread_pkey_regs_save(¤t->thread); - memset(&info, 0, sizeof(info)); + clear_siginfo(&info); info.si_signo = signr; info.si_code = code; info.si_addr = (void __user *) addr; @@ -970,7 +969,7 @@ void unknown_exception(struct pt_regs *regs) printk("Bad trap at PC: %lx, SR: %lx, vector=%lx\n", regs->nip, regs->msr, regs->trap); - _exception(SIGTRAP, regs, TRAP_FIXME, 0); + _exception(SIGTRAP, regs, TRAP_UNK, 0); exception_exit(prev_state); } @@ -992,7 +991,7 @@ bail: void RunModeException(struct pt_regs *regs) { - _exception(SIGTRAP, regs, TRAP_FIXME, 0); + _exception(SIGTRAP, regs, TRAP_UNK, 0); } void single_step_exception(struct pt_regs *regs) @@ -1032,7 +1031,7 @@ static void emulate_single_step(struct pt_regs *regs) static inline int __parse_fpscr(unsigned long fpscr) { - int ret = FPE_FIXME; + int ret = FPE_FLTUNK; /* Invalid operation */ if ((fpscr & FPSCR_VE) && (fpscr & FPSCR_VX)) @@ -1973,7 +1972,7 @@ void SPEFloatingPointException(struct pt_regs *regs) extern int do_spe_mathemu(struct pt_regs *regs); unsigned long spefscr; int fpexc_mode; - int code = FPE_FIXME; + int code = FPE_FLTUNK; int err; flush_spe_to_thread(current); @@ -2042,7 +2041,7 @@ void SPEFloatingPointRoundException(struct pt_regs *regs) printk(KERN_ERR "unrecognized spe instruction " "in %s at %lx\n", current->comm, regs->nip); } else { - _exception(SIGFPE, regs, FPE_FIXME, regs->nip); + _exception(SIGFPE, regs, FPE_FLTUNK, regs->nip); return; } } diff --git a/arch/powerpc/kernel/vdso.c b/arch/powerpc/kernel/vdso.c index b44ec104a5a1..d2205b97628c 100644 --- a/arch/powerpc/kernel/vdso.c +++ b/arch/powerpc/kernel/vdso.c @@ -791,7 +791,7 @@ static int __init vdso_init(void) #ifdef CONFIG_VDSO32 /* Make sure pages are in the correct state */ - vdso32_pagelist = kzalloc(sizeof(struct page *) * (vdso32_pages + 2), + vdso32_pagelist = kcalloc(vdso32_pages + 2, sizeof(struct page *), GFP_KERNEL); BUG_ON(vdso32_pagelist == NULL); for (i = 0; i < vdso32_pages; i++) { @@ -805,7 +805,7 @@ static int __init vdso_init(void) #endif #ifdef CONFIG_PPC64 - vdso64_pagelist = kzalloc(sizeof(struct page *) * (vdso64_pages + 2), + vdso64_pagelist = kcalloc(vdso64_pages + 2, sizeof(struct page *), GFP_KERNEL); BUG_ON(vdso64_pagelist == NULL); for (i = 0; i < vdso64_pages; i++) { diff --git a/arch/powerpc/kernel/vdso32/Makefile b/arch/powerpc/kernel/vdso32/Makefile index b8c434d1d459..50112d4473bb 100644 --- a/arch/powerpc/kernel/vdso32/Makefile +++ b/arch/powerpc/kernel/vdso32/Makefile @@ -8,8 +8,15 @@ obj-vdso32 = sigtramp.o gettimeofday.o datapage.o cacheflush.o note.o \ # Build rules -ifeq ($(CONFIG_PPC32),y) -CROSS32CC := $(CC) +ifdef CROSS32_COMPILE + VDSOCC := $(CROSS32_COMPILE)gcc +else + VDSOCC := $(CC) +endif + +CC32FLAGS := +ifdef CONFIG_PPC64 +CC32FLAGS += -m32 endif targets := $(obj-vdso32) vdso32.so vdso32.so.dbg @@ -45,9 +52,9 @@ $(obj-vdso32): %.o: %.S FORCE # actual build commands quiet_cmd_vdso32ld = VDSO32L $@ - cmd_vdso32ld = $(CROSS32CC) $(c_flags) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^) + cmd_vdso32ld = $(VDSOCC) $(c_flags) $(CC32FLAGS) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^) quiet_cmd_vdso32as = VDSO32A $@ - cmd_vdso32as = $(CROSS32CC) $(a_flags) -c -o $@ $< + cmd_vdso32as = $(VDSOCC) $(a_flags) $(CC32FLAGS) -c -o $@ $< # install commands for the unstripped file quiet_cmd_vdso_install = INSTALL $@ diff --git a/arch/powerpc/kernel/vecemu.c b/arch/powerpc/kernel/vecemu.c index 8812085883fd..4acd3fb2b38e 100644 --- a/arch/powerpc/kernel/vecemu.c +++ b/arch/powerpc/kernel/vecemu.c @@ -8,6 +8,7 @@ #include #include #include +#include #include /* Functions in vector.S */ diff --git a/arch/powerpc/kernel/vmlinux.lds.S b/arch/powerpc/kernel/vmlinux.lds.S index b8d82678f8b4..5baac79df97e 100644 --- a/arch/powerpc/kernel/vmlinux.lds.S +++ b/arch/powerpc/kernel/vmlinux.lds.S @@ -89,7 +89,7 @@ SECTIONS */ .text BLOCK(0) : AT(ADDR(.text) - LOAD_OFFSET) { #ifdef CONFIG_LD_HEAD_STUB_CATCH - *(.linker_stub_catch); + KEEP(*(.linker_stub_catch)); . = . ; #endif @@ -98,7 +98,7 @@ SECTIONS ALIGN_FUNCTION(); #endif /* careful! __ftr_alt_* sections need to be close to .text */ - *(.text.hot .text .text.fixup .text.unlikely .fixup __ftr_alt_* .ref.text); + *(.text.hot TEXT_MAIN .text.fixup .text.unlikely .fixup __ftr_alt_* .ref.text); SCHED_TEXT CPUIDLE_TEXT LOCK_TEXT @@ -153,6 +153,13 @@ SECTIONS *(__rfi_flush_fixup) __stop___rfi_flush_fixup = .; } + + . = ALIGN(8); + __spec_barrier_fixup : AT(ADDR(__spec_barrier_fixup) - LOAD_OFFSET) { + __start___barrier_nospec_fixup = .; + *(__barrier_nospec_fixup) + __stop___barrier_nospec_fixup = .; + } #endif EXCEPTION_TABLE(0) @@ -184,10 +191,10 @@ SECTIONS .init.data : AT(ADDR(.init.data) - LOAD_OFFSET) { INIT_DATA __vtop_table_begin = .; - *(.vtop_fixup); + KEEP(*(.vtop_fixup)); __vtop_table_end = .; __ptov_table_begin = .; - *(.ptov_fixup); + KEEP(*(.ptov_fixup)); __ptov_table_end = .; } @@ -208,26 +215,26 @@ SECTIONS . = ALIGN(8); __ftr_fixup : AT(ADDR(__ftr_fixup) - LOAD_OFFSET) { __start___ftr_fixup = .; - *(__ftr_fixup) + KEEP(*(__ftr_fixup)) __stop___ftr_fixup = .; } . = ALIGN(8); __mmu_ftr_fixup : AT(ADDR(__mmu_ftr_fixup) - LOAD_OFFSET) { __start___mmu_ftr_fixup = .; - *(__mmu_ftr_fixup) + KEEP(*(__mmu_ftr_fixup)) __stop___mmu_ftr_fixup = .; } . = ALIGN(8); __lwsync_fixup : AT(ADDR(__lwsync_fixup) - LOAD_OFFSET) { __start___lwsync_fixup = .; - *(__lwsync_fixup) + KEEP(*(__lwsync_fixup)) __stop___lwsync_fixup = .; } #ifdef CONFIG_PPC64 . = ALIGN(8); __fw_ftr_fixup : AT(ADDR(__fw_ftr_fixup) - LOAD_OFFSET) { __start___fw_ftr_fixup = .; - *(__fw_ftr_fixup) + KEEP(*(__fw_ftr_fixup)) __stop___fw_ftr_fixup = .; } #endif @@ -240,7 +247,7 @@ SECTIONS . = ALIGN(8); .machine.desc : AT(ADDR(.machine.desc) - LOAD_OFFSET) { __machine_desc_start = . ; - *(.machine.desc) + KEEP(*(.machine.desc)) __machine_desc_end = . ; } #ifdef CONFIG_RELOCATABLE @@ -288,7 +295,7 @@ SECTIONS .data : AT(ADDR(.data) - LOAD_OFFSET) { DATA_DATA *(.data.rel*) - *(.sdata) + *(SDATA_MAIN) *(.sdata2) *(.got.plt) *(.got) *(.plt) @@ -303,7 +310,7 @@ SECTIONS .opd : AT(ADDR(.opd) - LOAD_OFFSET) { __start_opd = .; - *(.opd) + KEEP(*(.opd)) __end_opd = .; } diff --git a/arch/powerpc/kernel/watchdog.c b/arch/powerpc/kernel/watchdog.c index 6256dc3b0087..1d82274f7e9f 100644 --- a/arch/powerpc/kernel/watchdog.c +++ b/arch/powerpc/kernel/watchdog.c @@ -64,7 +64,7 @@ * means the CPU(s) with their bit still set in the pending mask have had * their heartbeat stop, and action is taken. * - * Some platforms implement true NMI IPIs, which can by used by the SMP + * Some platforms implement true NMI IPIs, which can be used by the SMP * watchdog to detect an unresponsive CPU and pull it out of its stuck * state with the NMI IPI, to get crash/debug data from it. This way the * SMP watchdog can detect hardware interrupts off lockups. @@ -111,7 +111,13 @@ static inline void wd_smp_unlock(unsigned long *flags) static void wd_lockup_ipi(struct pt_regs *regs) { - pr_emerg("CPU %d Hard LOCKUP\n", raw_smp_processor_id()); + int cpu = raw_smp_processor_id(); + u64 tb = get_tb(); + + pr_emerg("CPU %d Hard LOCKUP\n", cpu); + pr_emerg("CPU %d TB:%lld, last heartbeat TB:%lld (%lldms ago)\n", + cpu, tb, per_cpu(wd_timer_tb, cpu), + tb_to_ns(tb - per_cpu(wd_timer_tb, cpu)) / 1000000); print_modules(); print_irqtrace_events(current); if (regs) @@ -154,6 +160,9 @@ static void watchdog_smp_panic(int cpu, u64 tb) pr_emerg("CPU %d detected hard LOCKUP on other CPUs %*pbl\n", cpu, cpumask_pr_args(&wd_smp_cpus_pending)); + pr_emerg("CPU %d TB:%lld, last SMP heartbeat TB:%lld (%lldms ago)\n", + cpu, tb, wd_smp_last_reset_tb, + tb_to_ns(tb - wd_smp_last_reset_tb) / 1000000); if (!sysctl_hardlockup_all_cpu_backtrace) { /* @@ -194,10 +203,19 @@ static void wd_smp_clear_cpu_pending(int cpu, u64 tb) { if (!cpumask_test_cpu(cpu, &wd_smp_cpus_pending)) { if (unlikely(cpumask_test_cpu(cpu, &wd_smp_cpus_stuck))) { + struct pt_regs *regs = get_irq_regs(); unsigned long flags; - pr_emerg("CPU %d became unstuck\n", cpu); wd_smp_lock(&flags); + + pr_emerg("CPU %d became unstuck TB:%lld\n", + cpu, tb); + print_irqtrace_events(current); + if (regs) + show_regs(regs); + else + dump_stack(); + cpumask_clear_cpu(cpu, &wd_smp_cpus_stuck); wd_smp_unlock(&flags); } @@ -245,8 +263,6 @@ void soft_nmi_interrupt(struct pt_regs *regs) tb = get_tb(); if (tb - per_cpu(wd_timer_tb, cpu) >= wd_panic_timeout_tb) { - per_cpu(wd_timer_tb, cpu) = tb; - wd_smp_lock(&flags); if (cpumask_test_cpu(cpu, &wd_smp_cpus_stuck)) { wd_smp_unlock(&flags); @@ -254,7 +270,11 @@ void soft_nmi_interrupt(struct pt_regs *regs) } set_cpu_stuck(cpu, tb); - pr_emerg("CPU %d self-detected hard LOCKUP @ %pS\n", cpu, (void *)regs->nip); + pr_emerg("CPU %d self-detected hard LOCKUP @ %pS\n", + cpu, (void *)regs->nip); + pr_emerg("CPU %d TB:%lld, last heartbeat TB:%lld (%lldms ago)\n", + cpu, tb, per_cpu(wd_timer_tb, cpu), + tb_to_ns(tb - per_cpu(wd_timer_tb, cpu)) / 1000000); print_modules(); print_irqtrace_events(current); show_regs(regs); diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile index 4b19da8c87ae..f872c04bb5b1 100644 --- a/arch/powerpc/kvm/Makefile +++ b/arch/powerpc/kvm/Makefile @@ -63,6 +63,9 @@ kvm-pr-y := \ book3s_64_mmu.o \ book3s_32_mmu.o +kvm-book3s_64-builtin-objs-$(CONFIG_KVM_BOOK3S_64_HANDLER) += \ + tm.o + ifdef CONFIG_KVM_BOOK3S_PR_POSSIBLE kvm-book3s_64-builtin-objs-$(CONFIG_KVM_BOOK3S_64_HANDLER) += \ book3s_rmhandlers.o diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c index 97d4a112648f..edaf4720d156 100644 --- a/arch/powerpc/kvm/book3s.c +++ b/arch/powerpc/kvm/book3s.c @@ -134,7 +134,7 @@ void kvmppc_inject_interrupt(struct kvm_vcpu *vcpu, int vec, u64 flags) { kvmppc_unfixup_split_real(vcpu); kvmppc_set_srr0(vcpu, kvmppc_get_pc(vcpu)); - kvmppc_set_srr1(vcpu, kvmppc_get_msr(vcpu) | flags); + kvmppc_set_srr1(vcpu, (kvmppc_get_msr(vcpu) & ~0x783f0000ul) | flags); kvmppc_set_pc(vcpu, kvmppc_interrupt_offset(vcpu) + vec); vcpu->arch.mmu.reset_msr(vcpu); } @@ -256,18 +256,15 @@ void kvmppc_core_queue_data_storage(struct kvm_vcpu *vcpu, ulong dar, { kvmppc_set_dar(vcpu, dar); kvmppc_set_dsisr(vcpu, flags); - kvmppc_book3s_queue_irqprio(vcpu, BOOK3S_INTERRUPT_DATA_STORAGE); + kvmppc_inject_interrupt(vcpu, BOOK3S_INTERRUPT_DATA_STORAGE, 0); } -EXPORT_SYMBOL_GPL(kvmppc_core_queue_data_storage); /* used by kvm_hv */ +EXPORT_SYMBOL_GPL(kvmppc_core_queue_data_storage); void kvmppc_core_queue_inst_storage(struct kvm_vcpu *vcpu, ulong flags) { - u64 msr = kvmppc_get_msr(vcpu); - msr &= ~(SRR1_ISI_NOPT | SRR1_ISI_N_OR_G | SRR1_ISI_PROT); - msr |= flags & (SRR1_ISI_NOPT | SRR1_ISI_N_OR_G | SRR1_ISI_PROT); - kvmppc_set_msr_fast(vcpu, msr); - kvmppc_book3s_queue_irqprio(vcpu, BOOK3S_INTERRUPT_INST_STORAGE); + kvmppc_inject_interrupt(vcpu, BOOK3S_INTERRUPT_INST_STORAGE, flags); } +EXPORT_SYMBOL_GPL(kvmppc_core_queue_inst_storage); static int kvmppc_book3s_irqprio_deliver(struct kvm_vcpu *vcpu, unsigned int priority) @@ -450,8 +447,8 @@ int kvmppc_xlate(struct kvm_vcpu *vcpu, ulong eaddr, enum xlate_instdata xlid, return r; } -int kvmppc_load_last_inst(struct kvm_vcpu *vcpu, enum instruction_type type, - u32 *inst) +int kvmppc_load_last_inst(struct kvm_vcpu *vcpu, + enum instruction_fetch_type type, u32 *inst) { ulong pc = kvmppc_get_pc(vcpu); int r; @@ -509,8 +506,6 @@ int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) { int i; - vcpu_load(vcpu); - regs->pc = kvmppc_get_pc(vcpu); regs->cr = kvmppc_get_cr(vcpu); regs->ctr = kvmppc_get_ctr(vcpu); @@ -532,7 +527,6 @@ int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) for (i = 0; i < ARRAY_SIZE(regs->gpr); i++) regs->gpr[i] = kvmppc_get_gpr(vcpu, i); - vcpu_put(vcpu); return 0; } @@ -540,8 +534,6 @@ int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) { int i; - vcpu_load(vcpu); - kvmppc_set_pc(vcpu, regs->pc); kvmppc_set_cr(vcpu, regs->cr); kvmppc_set_ctr(vcpu, regs->ctr); @@ -562,7 +554,6 @@ int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) for (i = 0; i < ARRAY_SIZE(regs->gpr); i++) kvmppc_set_gpr(vcpu, i, regs->gpr[i]); - vcpu_put(vcpu); return 0; } diff --git a/arch/powerpc/kvm/book3s.h b/arch/powerpc/kvm/book3s.h index 4ad5e287b8bc..14ef03501d21 100644 --- a/arch/powerpc/kvm/book3s.h +++ b/arch/powerpc/kvm/book3s.h @@ -31,4 +31,10 @@ extern int kvmppc_core_emulate_mfspr_pr(struct kvm_vcpu *vcpu, extern int kvmppc_book3s_init_pr(void); extern void kvmppc_book3s_exit_pr(void); +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM +extern void kvmppc_emulate_tabort(struct kvm_vcpu *vcpu, int ra_val); +#else +static inline void kvmppc_emulate_tabort(struct kvm_vcpu *vcpu, int ra_val) {} +#endif + #endif diff --git a/arch/powerpc/kvm/book3s_32_mmu.c b/arch/powerpc/kvm/book3s_32_mmu.c index 1992676c7a94..45c8ea4a0487 100644 --- a/arch/powerpc/kvm/book3s_32_mmu.c +++ b/arch/powerpc/kvm/book3s_32_mmu.c @@ -52,7 +52,7 @@ static inline bool check_debug_ip(struct kvm_vcpu *vcpu) { #ifdef DEBUG_MMU_PTE_IP - return vcpu->arch.pc == DEBUG_MMU_PTE_IP; + return vcpu->arch.regs.nip == DEBUG_MMU_PTE_IP; #else return true; #endif diff --git a/arch/powerpc/kvm/book3s_64_mmu.c b/arch/powerpc/kvm/book3s_64_mmu.c index a93d719edc90..cf9d686e8162 100644 --- a/arch/powerpc/kvm/book3s_64_mmu.c +++ b/arch/powerpc/kvm/book3s_64_mmu.c @@ -38,7 +38,16 @@ static void kvmppc_mmu_book3s_64_reset_msr(struct kvm_vcpu *vcpu) { - kvmppc_set_msr(vcpu, vcpu->arch.intr_msr); + unsigned long msr = vcpu->arch.intr_msr; + unsigned long cur_msr = kvmppc_get_msr(vcpu); + + /* If transactional, change to suspend mode on IRQ delivery */ + if (MSR_TM_TRANSACTIONAL(cur_msr)) + msr |= MSR_TS_S; + else + msr |= cur_msr & MSR_TS_MASK; + + kvmppc_set_msr(vcpu, msr); } static struct kvmppc_slb *kvmppc_mmu_book3s_64_find_slbe( diff --git a/arch/powerpc/kvm/book3s_64_mmu_hv.c b/arch/powerpc/kvm/book3s_64_mmu_hv.c index a670fa5fbe50..7f3a8cf5d66f 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_hv.c +++ b/arch/powerpc/kvm/book3s_64_mmu_hv.c @@ -108,7 +108,7 @@ int kvmppc_allocate_hpt(struct kvm_hpt_info *info, u32 order) npte = 1ul << (order - 4); /* Allocate reverse map array */ - rev = vmalloc(sizeof(struct revmap_entry) * npte); + rev = vmalloc(array_size(npte, sizeof(struct revmap_entry))); if (!rev) { if (cma) kvm_free_hpt_cma(page, 1 << (order - PAGE_SHIFT)); @@ -272,6 +272,9 @@ int kvmppc_mmu_hv_init(void) if (!cpu_has_feature(CPU_FTR_HVMODE)) return -EINVAL; + if (!mmu_has_feature(MMU_FTR_LOCKLESS_TLBIE)) + return -EINVAL; + /* POWER7 has 10-bit LPIDs (12-bit in POWER8) */ host_lpid = mfspr(SPRN_LPID); rsvd_lpid = LPID_RSVD; diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c index 361f42c8c73e..176f911ee983 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c @@ -139,44 +139,24 @@ int kvmppc_mmu_radix_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, return 0; } -#ifdef CONFIG_PPC_64K_PAGES -#define MMU_BASE_PSIZE MMU_PAGE_64K -#else -#define MMU_BASE_PSIZE MMU_PAGE_4K -#endif - static void kvmppc_radix_tlbie_page(struct kvm *kvm, unsigned long addr, unsigned int pshift) { - int psize = MMU_BASE_PSIZE; - - if (pshift >= PUD_SHIFT) - psize = MMU_PAGE_1G; - else if (pshift >= PMD_SHIFT) - psize = MMU_PAGE_2M; - addr &= ~0xfffUL; - addr |= mmu_psize_defs[psize].ap << 5; - asm volatile("ptesync": : :"memory"); - asm volatile(PPC_TLBIE_5(%0, %1, 0, 0, 1) - : : "r" (addr), "r" (kvm->arch.lpid) : "memory"); - if (cpu_has_feature(CPU_FTR_P9_TLBIE_BUG)) - asm volatile(PPC_TLBIE_5(%0, %1, 0, 0, 1) - : : "r" (addr), "r" (kvm->arch.lpid) : "memory"); - asm volatile("eieio ; tlbsync ; ptesync": : :"memory"); + unsigned long psize = PAGE_SIZE; + + if (pshift) + psize = 1UL << pshift; + + addr &= ~(psize - 1); + radix__flush_tlb_lpid_page(kvm->arch.lpid, addr, psize); } -static void kvmppc_radix_flush_pwc(struct kvm *kvm, unsigned long addr) +static void kvmppc_radix_flush_pwc(struct kvm *kvm) { - unsigned long rb = 0x2 << PPC_BITLSHIFT(53); /* IS = 2 */ - - asm volatile("ptesync": : :"memory"); - /* RIC=1 PRS=0 R=1 IS=2 */ - asm volatile(PPC_TLBIE_5(%0, %1, 1, 0, 1) - : : "r" (rb), "r" (kvm->arch.lpid) : "memory"); - asm volatile("eieio ; tlbsync ; ptesync": : :"memory"); + radix__flush_pwc_lpid(kvm->arch.lpid); } -unsigned long kvmppc_radix_update_pte(struct kvm *kvm, pte_t *ptep, +static unsigned long kvmppc_radix_update_pte(struct kvm *kvm, pte_t *ptep, unsigned long clr, unsigned long set, unsigned long addr, unsigned int shift) { @@ -200,6 +180,7 @@ void kvmppc_radix_set_pte_at(struct kvm *kvm, unsigned long addr, } static struct kmem_cache *kvm_pte_cache; +static struct kmem_cache *kvm_pmd_cache; static pte_t *kvmppc_pte_alloc(void) { @@ -217,6 +198,177 @@ static inline int pmd_is_leaf(pmd_t pmd) return !!(pmd_val(pmd) & _PAGE_PTE); } +static pmd_t *kvmppc_pmd_alloc(void) +{ + return kmem_cache_alloc(kvm_pmd_cache, GFP_KERNEL); +} + +static void kvmppc_pmd_free(pmd_t *pmdp) +{ + kmem_cache_free(kvm_pmd_cache, pmdp); +} + +static void kvmppc_unmap_pte(struct kvm *kvm, pte_t *pte, + unsigned long gpa, unsigned int shift) + +{ + unsigned long page_size = 1ul << shift; + unsigned long old; + + old = kvmppc_radix_update_pte(kvm, pte, ~0UL, 0, gpa, shift); + kvmppc_radix_tlbie_page(kvm, gpa, shift); + if (old & _PAGE_DIRTY) { + unsigned long gfn = gpa >> PAGE_SHIFT; + struct kvm_memory_slot *memslot; + + memslot = gfn_to_memslot(kvm, gfn); + if (memslot && memslot->dirty_bitmap) + kvmppc_update_dirty_map(memslot, gfn, page_size); + } +} + +/* + * kvmppc_free_p?d are used to free existing page tables, and recursively + * descend and clear and free children. + * Callers are responsible for flushing the PWC. + * + * When page tables are being unmapped/freed as part of page fault path + * (full == false), ptes are not expected. There is code to unmap them + * and emit a warning if encountered, but there may already be data + * corruption due to the unexpected mappings. + */ +static void kvmppc_unmap_free_pte(struct kvm *kvm, pte_t *pte, bool full) +{ + if (full) { + memset(pte, 0, sizeof(long) << PTE_INDEX_SIZE); + } else { + pte_t *p = pte; + unsigned long it; + + for (it = 0; it < PTRS_PER_PTE; ++it, ++p) { + if (pte_val(*p) == 0) + continue; + WARN_ON_ONCE(1); + kvmppc_unmap_pte(kvm, p, + pte_pfn(*p) << PAGE_SHIFT, + PAGE_SHIFT); + } + } + + kvmppc_pte_free(pte); +} + +static void kvmppc_unmap_free_pmd(struct kvm *kvm, pmd_t *pmd, bool full) +{ + unsigned long im; + pmd_t *p = pmd; + + for (im = 0; im < PTRS_PER_PMD; ++im, ++p) { + if (!pmd_present(*p)) + continue; + if (pmd_is_leaf(*p)) { + if (full) { + pmd_clear(p); + } else { + WARN_ON_ONCE(1); + kvmppc_unmap_pte(kvm, (pte_t *)p, + pte_pfn(*(pte_t *)p) << PAGE_SHIFT, + PMD_SHIFT); + } + } else { + pte_t *pte; + + pte = pte_offset_map(p, 0); + kvmppc_unmap_free_pte(kvm, pte, full); + pmd_clear(p); + } + } + kvmppc_pmd_free(pmd); +} + +static void kvmppc_unmap_free_pud(struct kvm *kvm, pud_t *pud) +{ + unsigned long iu; + pud_t *p = pud; + + for (iu = 0; iu < PTRS_PER_PUD; ++iu, ++p) { + if (!pud_present(*p)) + continue; + if (pud_huge(*p)) { + pud_clear(p); + } else { + pmd_t *pmd; + + pmd = pmd_offset(p, 0); + kvmppc_unmap_free_pmd(kvm, pmd, true); + pud_clear(p); + } + } + pud_free(kvm->mm, pud); +} + +void kvmppc_free_radix(struct kvm *kvm) +{ + unsigned long ig; + pgd_t *pgd; + + if (!kvm->arch.pgtable) + return; + pgd = kvm->arch.pgtable; + for (ig = 0; ig < PTRS_PER_PGD; ++ig, ++pgd) { + pud_t *pud; + + if (!pgd_present(*pgd)) + continue; + pud = pud_offset(pgd, 0); + kvmppc_unmap_free_pud(kvm, pud); + pgd_clear(pgd); + } + pgd_free(kvm->mm, kvm->arch.pgtable); + kvm->arch.pgtable = NULL; +} + +static void kvmppc_unmap_free_pmd_entry_table(struct kvm *kvm, pmd_t *pmd, + unsigned long gpa) +{ + pte_t *pte = pte_offset_kernel(pmd, 0); + + /* + * Clearing the pmd entry then flushing the PWC ensures that the pte + * page no longer be cached by the MMU, so can be freed without + * flushing the PWC again. + */ + pmd_clear(pmd); + kvmppc_radix_flush_pwc(kvm); + + kvmppc_unmap_free_pte(kvm, pte, false); +} + +static void kvmppc_unmap_free_pud_entry_table(struct kvm *kvm, pud_t *pud, + unsigned long gpa) +{ + pmd_t *pmd = pmd_offset(pud, 0); + + /* + * Clearing the pud entry then flushing the PWC ensures that the pmd + * page and any children pte pages will no longer be cached by the MMU, + * so can be freed without flushing the PWC again. + */ + pud_clear(pud); + kvmppc_radix_flush_pwc(kvm); + + kvmppc_unmap_free_pmd(kvm, pmd, false); +} + +/* + * There are a number of bits which may differ between different faults to + * the same partition scope entry. RC bits, in the course of cleaning and + * aging. And the write bit can change, either the access could have been + * upgraded, or a read fault could happen concurrently with a write fault + * that sets those bits first. + */ +#define PTE_BITS_MUST_MATCH (~(_PAGE_WRITE | _PAGE_DIRTY | _PAGE_ACCESSED)) + static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, unsigned int level, unsigned long mmu_seq) { @@ -224,7 +376,6 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, pud_t *pud, *new_pud = NULL; pmd_t *pmd, *new_pmd = NULL; pte_t *ptep, *new_ptep = NULL; - unsigned long old; int ret; /* Traverse the guest's 2nd-level tree, allocate new levels needed */ @@ -239,7 +390,7 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, if (pud && pud_present(*pud) && !pud_huge(*pud)) pmd = pmd_offset(pud, gpa); else if (level <= 1) - new_pmd = pmd_alloc_one(kvm->mm, gpa); + new_pmd = kvmppc_pmd_alloc(); if (level == 0 && !(pmd && pmd_present(*pmd) && !pmd_is_leaf(*pmd))) new_ptep = kvmppc_pte_alloc(); @@ -262,42 +413,39 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, if (pud_huge(*pud)) { unsigned long hgpa = gpa & PUD_MASK; + /* Check if we raced and someone else has set the same thing */ + if (level == 2) { + if (pud_raw(*pud) == pte_raw(pte)) { + ret = 0; + goto out_unlock; + } + /* Valid 1GB page here already, add our extra bits */ + WARN_ON_ONCE((pud_val(*pud) ^ pte_val(pte)) & + PTE_BITS_MUST_MATCH); + kvmppc_radix_update_pte(kvm, (pte_t *)pud, + 0, pte_val(pte), hgpa, PUD_SHIFT); + ret = 0; + goto out_unlock; + } /* * If we raced with another CPU which has just put * a 1GB pte in after we saw a pmd page, try again. */ - if (level <= 1 && !new_pmd) { + if (!new_pmd) { ret = -EAGAIN; goto out_unlock; } - /* Check if we raced and someone else has set the same thing */ - if (level == 2 && pud_raw(*pud) == pte_raw(pte)) { - ret = 0; - goto out_unlock; - } /* Valid 1GB page here already, remove it */ - old = kvmppc_radix_update_pte(kvm, (pte_t *)pud, - ~0UL, 0, hgpa, PUD_SHIFT); - kvmppc_radix_tlbie_page(kvm, hgpa, PUD_SHIFT); - if (old & _PAGE_DIRTY) { - unsigned long gfn = hgpa >> PAGE_SHIFT; - struct kvm_memory_slot *memslot; - memslot = gfn_to_memslot(kvm, gfn); - if (memslot && memslot->dirty_bitmap) - kvmppc_update_dirty_map(memslot, - gfn, PUD_SIZE); - } + kvmppc_unmap_pte(kvm, (pte_t *)pud, hgpa, PUD_SHIFT); } if (level == 2) { if (!pud_none(*pud)) { /* * There's a page table page here, but we wanted to * install a large page, so remove and free the page - * table page. new_pmd will be NULL since level == 2. + * table page. */ - new_pmd = pmd_offset(pud, 0); - pud_clear(pud); - kvmppc_radix_flush_pwc(kvm, gpa); + kvmppc_unmap_free_pud_entry_table(kvm, pud, gpa); } kvmppc_radix_set_pte_at(kvm, gpa, (pte_t *)pud, pte); ret = 0; @@ -313,42 +461,40 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, if (pmd_is_leaf(*pmd)) { unsigned long lgpa = gpa & PMD_MASK; + /* Check if we raced and someone else has set the same thing */ + if (level == 1) { + if (pmd_raw(*pmd) == pte_raw(pte)) { + ret = 0; + goto out_unlock; + } + /* Valid 2MB page here already, add our extra bits */ + WARN_ON_ONCE((pmd_val(*pmd) ^ pte_val(pte)) & + PTE_BITS_MUST_MATCH); + kvmppc_radix_update_pte(kvm, pmdp_ptep(pmd), + 0, pte_val(pte), lgpa, PMD_SHIFT); + ret = 0; + goto out_unlock; + } + /* * If we raced with another CPU which has just put * a 2MB pte in after we saw a pte page, try again. */ - if (level == 0 && !new_ptep) { + if (!new_ptep) { ret = -EAGAIN; goto out_unlock; } - /* Check if we raced and someone else has set the same thing */ - if (level == 1 && pmd_raw(*pmd) == pte_raw(pte)) { - ret = 0; - goto out_unlock; - } /* Valid 2MB page here already, remove it */ - old = kvmppc_radix_update_pte(kvm, pmdp_ptep(pmd), - ~0UL, 0, lgpa, PMD_SHIFT); - kvmppc_radix_tlbie_page(kvm, lgpa, PMD_SHIFT); - if (old & _PAGE_DIRTY) { - unsigned long gfn = lgpa >> PAGE_SHIFT; - struct kvm_memory_slot *memslot; - memslot = gfn_to_memslot(kvm, gfn); - if (memslot && memslot->dirty_bitmap) - kvmppc_update_dirty_map(memslot, - gfn, PMD_SIZE); - } + kvmppc_unmap_pte(kvm, pmdp_ptep(pmd), lgpa, PMD_SHIFT); } if (level == 1) { if (!pmd_none(*pmd)) { /* * There's a page table page here, but we wanted to * install a large page, so remove and free the page - * table page. new_ptep will be NULL since level == 1. + * table page. */ - new_ptep = pte_offset_kernel(pmd, 0); - pmd_clear(pmd); - kvmppc_radix_flush_pwc(kvm, gpa); + kvmppc_unmap_free_pmd_entry_table(kvm, pmd, gpa); } kvmppc_radix_set_pte_at(kvm, gpa, pmdp_ptep(pmd), pte); ret = 0; @@ -367,12 +513,12 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, ret = 0; goto out_unlock; } - /* PTE was previously valid, so invalidate it */ - old = kvmppc_radix_update_pte(kvm, ptep, _PAGE_PRESENT, - 0, gpa, 0); - kvmppc_radix_tlbie_page(kvm, gpa, 0); - if (old & _PAGE_DIRTY) - mark_page_dirty(kvm, gpa >> PAGE_SHIFT); + /* Valid page here already, add our extra bits */ + WARN_ON_ONCE((pte_val(*ptep) ^ pte_val(pte)) & + PTE_BITS_MUST_MATCH); + kvmppc_radix_update_pte(kvm, ptep, 0, pte_val(pte), gpa, 0); + ret = 0; + goto out_unlock; } kvmppc_radix_set_pte_at(kvm, gpa, ptep, pte); ret = 0; @@ -382,7 +528,7 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, if (new_pud) pud_free(kvm->mm, new_pud); if (new_pmd) - pmd_free(kvm->mm, new_pmd); + kvmppc_pmd_free(new_pmd); if (new_ptep) kvmppc_pte_free(new_ptep); return ret; @@ -554,9 +700,13 @@ int kvmppc_book3s_radix_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu, unsigned long mask = (1ul << shift) - PAGE_SIZE; pte = __pte(pte_val(pte) | (hva & mask)); } - if (!(writing || upgrade_write)) - pte = __pte(pte_val(pte) & ~ _PAGE_WRITE); - pte = __pte(pte_val(pte) | _PAGE_EXEC); + pte = __pte(pte_val(pte) | _PAGE_EXEC | _PAGE_ACCESSED); + if (writing || upgrade_write) { + if (pte_val(pte) & _PAGE_WRITE) + pte = __pte(pte_val(pte) | _PAGE_DIRTY); + } else { + pte = __pte(pte_val(pte) & ~(_PAGE_WRITE | _PAGE_DIRTY)); + } } /* Allocate space in the tree and write the PTE */ @@ -723,67 +873,37 @@ int kvmppc_init_vm_radix(struct kvm *kvm) return 0; } -void kvmppc_free_radix(struct kvm *kvm) +static void pte_ctor(void *addr) { - unsigned long ig, iu, im; - pte_t *pte; - pmd_t *pmd; - pud_t *pud; - pgd_t *pgd; - - if (!kvm->arch.pgtable) - return; - pgd = kvm->arch.pgtable; - for (ig = 0; ig < PTRS_PER_PGD; ++ig, ++pgd) { - if (!pgd_present(*pgd)) - continue; - pud = pud_offset(pgd, 0); - for (iu = 0; iu < PTRS_PER_PUD; ++iu, ++pud) { - if (!pud_present(*pud)) - continue; - if (pud_huge(*pud)) { - pud_clear(pud); - continue; - } - pmd = pmd_offset(pud, 0); - for (im = 0; im < PTRS_PER_PMD; ++im, ++pmd) { - if (pmd_is_leaf(*pmd)) { - pmd_clear(pmd); - continue; - } - if (!pmd_present(*pmd)) - continue; - pte = pte_offset_map(pmd, 0); - memset(pte, 0, sizeof(long) << PTE_INDEX_SIZE); - kvmppc_pte_free(pte); - pmd_clear(pmd); - } - pmd_free(kvm->mm, pmd_offset(pud, 0)); - pud_clear(pud); - } - pud_free(kvm->mm, pud_offset(pgd, 0)); - pgd_clear(pgd); - } - pgd_free(kvm->mm, kvm->arch.pgtable); - kvm->arch.pgtable = NULL; + memset(addr, 0, RADIX_PTE_TABLE_SIZE); } -static void pte_ctor(void *addr) +static void pmd_ctor(void *addr) { - memset(addr, 0, PTE_TABLE_SIZE); + memset(addr, 0, RADIX_PMD_TABLE_SIZE); } int kvmppc_radix_init(void) { - unsigned long size = sizeof(void *) << PTE_INDEX_SIZE; + unsigned long size = sizeof(void *) << RADIX_PTE_INDEX_SIZE; kvm_pte_cache = kmem_cache_create("kvm-pte", size, size, 0, pte_ctor); if (!kvm_pte_cache) return -ENOMEM; + + size = sizeof(void *) << RADIX_PMD_INDEX_SIZE; + + kvm_pmd_cache = kmem_cache_create("kvm-pmd", size, size, 0, pmd_ctor); + if (!kvm_pmd_cache) { + kmem_cache_destroy(kvm_pte_cache); + return -ENOMEM; + } + return 0; } void kvmppc_radix_exit(void) { kmem_cache_destroy(kvm_pte_cache); + kmem_cache_destroy(kvm_pmd_cache); } diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c index 4dffa611376d..d066e37551ec 100644 --- a/arch/powerpc/kvm/book3s_64_vio.c +++ b/arch/powerpc/kvm/book3s_64_vio.c @@ -176,14 +176,12 @@ extern long kvm_spapr_tce_attach_iommu_group(struct kvm *kvm, int tablefd, if (!tbltmp) continue; - /* - * Make sure hardware table parameters are exactly the same; - * this is used in the TCE handlers where boundary checks - * use only the first attached table. - */ - if ((tbltmp->it_page_shift == stt->page_shift) && - (tbltmp->it_offset == stt->offset) && - (tbltmp->it_size == stt->size)) { + /* Make sure hardware table parameters are compatible */ + if ((tbltmp->it_page_shift <= stt->page_shift) && + (tbltmp->it_offset << tbltmp->it_page_shift == + stt->offset << stt->page_shift) && + (tbltmp->it_size << tbltmp->it_page_shift == + stt->size << stt->page_shift)) { /* * Reference the table to avoid races with * add/remove DMA windows. @@ -237,7 +235,7 @@ static void release_spapr_tce_table(struct rcu_head *head) kfree(stt); } -static int kvm_spapr_tce_fault(struct vm_fault *vmf) +static vm_fault_t kvm_spapr_tce_fault(struct vm_fault *vmf) { struct kvmppc_spapr_tce_table *stt = vmf->vma->vm_file->private_data; struct page *page; @@ -302,7 +300,8 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm, int ret = -ENOMEM; int i; - if (!args->size) + if (!args->size || args->page_shift < 12 || args->page_shift > 34 || + (args->offset + args->size > (ULLONG_MAX >> args->page_shift))) return -EINVAL; size = _ALIGN_UP(args->size, PAGE_SIZE >> 3); @@ -396,7 +395,7 @@ static long kvmppc_tce_iommu_mapped_dec(struct kvm *kvm, return H_SUCCESS; } -static long kvmppc_tce_iommu_unmap(struct kvm *kvm, +static long kvmppc_tce_iommu_do_unmap(struct kvm *kvm, struct iommu_table *tbl, unsigned long entry) { enum dma_data_direction dir = DMA_NONE; @@ -416,7 +415,24 @@ static long kvmppc_tce_iommu_unmap(struct kvm *kvm, return ret; } -long kvmppc_tce_iommu_map(struct kvm *kvm, struct iommu_table *tbl, +static long kvmppc_tce_iommu_unmap(struct kvm *kvm, + struct kvmppc_spapr_tce_table *stt, struct iommu_table *tbl, + unsigned long entry) +{ + unsigned long i, ret = H_SUCCESS; + unsigned long subpages = 1ULL << (stt->page_shift - tbl->it_page_shift); + unsigned long io_entry = entry * subpages; + + for (i = 0; i < subpages; ++i) { + ret = kvmppc_tce_iommu_do_unmap(kvm, tbl, io_entry + i); + if (ret != H_SUCCESS) + break; + } + + return ret; +} + +long kvmppc_tce_iommu_do_map(struct kvm *kvm, struct iommu_table *tbl, unsigned long entry, unsigned long ua, enum dma_data_direction dir) { @@ -453,6 +469,27 @@ long kvmppc_tce_iommu_map(struct kvm *kvm, struct iommu_table *tbl, return 0; } +static long kvmppc_tce_iommu_map(struct kvm *kvm, + struct kvmppc_spapr_tce_table *stt, struct iommu_table *tbl, + unsigned long entry, unsigned long ua, + enum dma_data_direction dir) +{ + unsigned long i, pgoff, ret = H_SUCCESS; + unsigned long subpages = 1ULL << (stt->page_shift - tbl->it_page_shift); + unsigned long io_entry = entry * subpages; + + for (i = 0, pgoff = 0; i < subpages; + ++i, pgoff += IOMMU_PAGE_SIZE(tbl)) { + + ret = kvmppc_tce_iommu_do_map(kvm, tbl, + io_entry + i, ua + pgoff, dir); + if (ret != H_SUCCESS) + break; + } + + return ret; +} + long kvmppc_h_put_tce(struct kvm_vcpu *vcpu, unsigned long liobn, unsigned long ioba, unsigned long tce) { @@ -491,10 +528,10 @@ long kvmppc_h_put_tce(struct kvm_vcpu *vcpu, unsigned long liobn, list_for_each_entry_lockless(stit, &stt->iommu_tables, next) { if (dir == DMA_NONE) - ret = kvmppc_tce_iommu_unmap(vcpu->kvm, + ret = kvmppc_tce_iommu_unmap(vcpu->kvm, stt, stit->tbl, entry); else - ret = kvmppc_tce_iommu_map(vcpu->kvm, stit->tbl, + ret = kvmppc_tce_iommu_map(vcpu->kvm, stt, stit->tbl, entry, ua, dir); if (ret == H_SUCCESS) @@ -570,7 +607,7 @@ long kvmppc_h_put_tce_indirect(struct kvm_vcpu *vcpu, return H_PARAMETER; list_for_each_entry_lockless(stit, &stt->iommu_tables, next) { - ret = kvmppc_tce_iommu_map(vcpu->kvm, + ret = kvmppc_tce_iommu_map(vcpu->kvm, stt, stit->tbl, entry + i, ua, iommu_tce_direction(tce)); @@ -615,10 +652,10 @@ long kvmppc_h_stuff_tce(struct kvm_vcpu *vcpu, return H_PARAMETER; list_for_each_entry_lockless(stit, &stt->iommu_tables, next) { - unsigned long entry = ioba >> stit->tbl->it_page_shift; + unsigned long entry = ioba >> stt->page_shift; for (i = 0; i < npages; ++i) { - ret = kvmppc_tce_iommu_unmap(vcpu->kvm, + ret = kvmppc_tce_iommu_unmap(vcpu->kvm, stt, stit->tbl, entry + i); if (ret == H_SUCCESS) diff --git a/arch/powerpc/kvm/book3s_64_vio_hv.c b/arch/powerpc/kvm/book3s_64_vio_hv.c index 6651f736a0b1..925fc316a104 100644 --- a/arch/powerpc/kvm/book3s_64_vio_hv.c +++ b/arch/powerpc/kvm/book3s_64_vio_hv.c @@ -221,7 +221,7 @@ static long kvmppc_rm_tce_iommu_mapped_dec(struct kvm *kvm, return H_SUCCESS; } -static long kvmppc_rm_tce_iommu_unmap(struct kvm *kvm, +static long kvmppc_rm_tce_iommu_do_unmap(struct kvm *kvm, struct iommu_table *tbl, unsigned long entry) { enum dma_data_direction dir = DMA_NONE; @@ -245,7 +245,24 @@ static long kvmppc_rm_tce_iommu_unmap(struct kvm *kvm, return ret; } -static long kvmppc_rm_tce_iommu_map(struct kvm *kvm, struct iommu_table *tbl, +static long kvmppc_rm_tce_iommu_unmap(struct kvm *kvm, + struct kvmppc_spapr_tce_table *stt, struct iommu_table *tbl, + unsigned long entry) +{ + unsigned long i, ret = H_SUCCESS; + unsigned long subpages = 1ULL << (stt->page_shift - tbl->it_page_shift); + unsigned long io_entry = entry * subpages; + + for (i = 0; i < subpages; ++i) { + ret = kvmppc_rm_tce_iommu_do_unmap(kvm, tbl, io_entry + i); + if (ret != H_SUCCESS) + break; + } + + return ret; +} + +static long kvmppc_rm_tce_iommu_do_map(struct kvm *kvm, struct iommu_table *tbl, unsigned long entry, unsigned long ua, enum dma_data_direction dir) { @@ -290,6 +307,27 @@ static long kvmppc_rm_tce_iommu_map(struct kvm *kvm, struct iommu_table *tbl, return 0; } +static long kvmppc_rm_tce_iommu_map(struct kvm *kvm, + struct kvmppc_spapr_tce_table *stt, struct iommu_table *tbl, + unsigned long entry, unsigned long ua, + enum dma_data_direction dir) +{ + unsigned long i, pgoff, ret = H_SUCCESS; + unsigned long subpages = 1ULL << (stt->page_shift - tbl->it_page_shift); + unsigned long io_entry = entry * subpages; + + for (i = 0, pgoff = 0; i < subpages; + ++i, pgoff += IOMMU_PAGE_SIZE(tbl)) { + + ret = kvmppc_rm_tce_iommu_do_map(kvm, tbl, + io_entry + i, ua + pgoff, dir); + if (ret != H_SUCCESS) + break; + } + + return ret; +} + long kvmppc_rm_h_put_tce(struct kvm_vcpu *vcpu, unsigned long liobn, unsigned long ioba, unsigned long tce) { @@ -327,10 +365,10 @@ long kvmppc_rm_h_put_tce(struct kvm_vcpu *vcpu, unsigned long liobn, list_for_each_entry_lockless(stit, &stt->iommu_tables, next) { if (dir == DMA_NONE) - ret = kvmppc_rm_tce_iommu_unmap(vcpu->kvm, + ret = kvmppc_rm_tce_iommu_unmap(vcpu->kvm, stt, stit->tbl, entry); else - ret = kvmppc_rm_tce_iommu_map(vcpu->kvm, + ret = kvmppc_rm_tce_iommu_map(vcpu->kvm, stt, stit->tbl, entry, ua, dir); if (ret == H_SUCCESS) @@ -477,7 +515,7 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu, return H_PARAMETER; list_for_each_entry_lockless(stit, &stt->iommu_tables, next) { - ret = kvmppc_rm_tce_iommu_map(vcpu->kvm, + ret = kvmppc_rm_tce_iommu_map(vcpu->kvm, stt, stit->tbl, entry + i, ua, iommu_tce_direction(tce)); @@ -526,10 +564,10 @@ long kvmppc_rm_h_stuff_tce(struct kvm_vcpu *vcpu, return H_PARAMETER; list_for_each_entry_lockless(stit, &stt->iommu_tables, next) { - unsigned long entry = ioba >> stit->tbl->it_page_shift; + unsigned long entry = ioba >> stt->page_shift; for (i = 0; i < npages; ++i) { - ret = kvmppc_rm_tce_iommu_unmap(vcpu->kvm, + ret = kvmppc_rm_tce_iommu_unmap(vcpu->kvm, stt, stit->tbl, entry + i); if (ret == H_SUCCESS) @@ -571,7 +609,7 @@ long kvmppc_h_get_tce(struct kvm_vcpu *vcpu, unsigned long liobn, page = stt->pages[idx / TCES_PER_PAGE]; tbl = (u64 *)page_address(page); - vcpu->arch.gpr[4] = tbl[idx % TCES_PER_PAGE]; + vcpu->arch.regs.gpr[4] = tbl[idx % TCES_PER_PAGE]; return H_SUCCESS; } diff --git a/arch/powerpc/kvm/book3s_emulate.c b/arch/powerpc/kvm/book3s_emulate.c index 68d68983948e..36b11c5a0dbb 100644 --- a/arch/powerpc/kvm/book3s_emulate.c +++ b/arch/powerpc/kvm/book3s_emulate.c @@ -23,7 +23,9 @@ #include #include #include +#include #include "book3s.h" +#include #define OP_19_XOP_RFID 18 #define OP_19_XOP_RFI 50 @@ -47,6 +49,12 @@ #define OP_31_XOP_EIOIO 854 #define OP_31_XOP_SLBMFEE 915 +#define OP_31_XOP_TBEGIN 654 +#define OP_31_XOP_TABORT 910 + +#define OP_31_XOP_TRECLAIM 942 +#define OP_31_XOP_TRCHKPT 1006 + /* DCBZ is actually 1014, but we patch it to 1010 so we get a trap */ #define OP_31_XOP_DCBZ 1010 @@ -87,6 +95,157 @@ static bool spr_allowed(struct kvm_vcpu *vcpu, enum priv_level level) return true; } +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM +static inline void kvmppc_copyto_vcpu_tm(struct kvm_vcpu *vcpu) +{ + memcpy(&vcpu->arch.gpr_tm[0], &vcpu->arch.regs.gpr[0], + sizeof(vcpu->arch.gpr_tm)); + memcpy(&vcpu->arch.fp_tm, &vcpu->arch.fp, + sizeof(struct thread_fp_state)); + memcpy(&vcpu->arch.vr_tm, &vcpu->arch.vr, + sizeof(struct thread_vr_state)); + vcpu->arch.ppr_tm = vcpu->arch.ppr; + vcpu->arch.dscr_tm = vcpu->arch.dscr; + vcpu->arch.amr_tm = vcpu->arch.amr; + vcpu->arch.ctr_tm = vcpu->arch.regs.ctr; + vcpu->arch.tar_tm = vcpu->arch.tar; + vcpu->arch.lr_tm = vcpu->arch.regs.link; + vcpu->arch.cr_tm = vcpu->arch.cr; + vcpu->arch.xer_tm = vcpu->arch.regs.xer; + vcpu->arch.vrsave_tm = vcpu->arch.vrsave; +} + +static inline void kvmppc_copyfrom_vcpu_tm(struct kvm_vcpu *vcpu) +{ + memcpy(&vcpu->arch.regs.gpr[0], &vcpu->arch.gpr_tm[0], + sizeof(vcpu->arch.regs.gpr)); + memcpy(&vcpu->arch.fp, &vcpu->arch.fp_tm, + sizeof(struct thread_fp_state)); + memcpy(&vcpu->arch.vr, &vcpu->arch.vr_tm, + sizeof(struct thread_vr_state)); + vcpu->arch.ppr = vcpu->arch.ppr_tm; + vcpu->arch.dscr = vcpu->arch.dscr_tm; + vcpu->arch.amr = vcpu->arch.amr_tm; + vcpu->arch.regs.ctr = vcpu->arch.ctr_tm; + vcpu->arch.tar = vcpu->arch.tar_tm; + vcpu->arch.regs.link = vcpu->arch.lr_tm; + vcpu->arch.cr = vcpu->arch.cr_tm; + vcpu->arch.regs.xer = vcpu->arch.xer_tm; + vcpu->arch.vrsave = vcpu->arch.vrsave_tm; +} + +static void kvmppc_emulate_treclaim(struct kvm_vcpu *vcpu, int ra_val) +{ + unsigned long guest_msr = kvmppc_get_msr(vcpu); + int fc_val = ra_val ? ra_val : 1; + uint64_t texasr; + + /* CR0 = 0 | MSR[TS] | 0 */ + vcpu->arch.cr = (vcpu->arch.cr & ~(CR0_MASK << CR0_SHIFT)) | + (((guest_msr & MSR_TS_MASK) >> (MSR_TS_S_LG - 1)) + << CR0_SHIFT); + + preempt_disable(); + tm_enable(); + texasr = mfspr(SPRN_TEXASR); + kvmppc_save_tm_pr(vcpu); + kvmppc_copyfrom_vcpu_tm(vcpu); + + /* failure recording depends on Failure Summary bit */ + if (!(texasr & TEXASR_FS)) { + texasr &= ~TEXASR_FC; + texasr |= ((u64)fc_val << TEXASR_FC_LG) | TEXASR_FS; + + texasr &= ~(TEXASR_PR | TEXASR_HV); + if (kvmppc_get_msr(vcpu) & MSR_PR) + texasr |= TEXASR_PR; + + if (kvmppc_get_msr(vcpu) & MSR_HV) + texasr |= TEXASR_HV; + + vcpu->arch.texasr = texasr; + vcpu->arch.tfiar = kvmppc_get_pc(vcpu); + mtspr(SPRN_TEXASR, texasr); + mtspr(SPRN_TFIAR, vcpu->arch.tfiar); + } + tm_disable(); + /* + * treclaim need quit to non-transactional state. + */ + guest_msr &= ~(MSR_TS_MASK); + kvmppc_set_msr(vcpu, guest_msr); + preempt_enable(); + + if (vcpu->arch.shadow_fscr & FSCR_TAR) + mtspr(SPRN_TAR, vcpu->arch.tar); +} + +static void kvmppc_emulate_trchkpt(struct kvm_vcpu *vcpu) +{ + unsigned long guest_msr = kvmppc_get_msr(vcpu); + + preempt_disable(); + /* + * need flush FP/VEC/VSX to vcpu save area before + * copy. + */ + kvmppc_giveup_ext(vcpu, MSR_VSX); + kvmppc_giveup_fac(vcpu, FSCR_TAR_LG); + kvmppc_copyto_vcpu_tm(vcpu); + kvmppc_save_tm_sprs(vcpu); + + /* + * as a result of trecheckpoint. set TS to suspended. + */ + guest_msr &= ~(MSR_TS_MASK); + guest_msr |= MSR_TS_S; + kvmppc_set_msr(vcpu, guest_msr); + kvmppc_restore_tm_pr(vcpu); + preempt_enable(); +} + +/* emulate tabort. at guest privilege state */ +void kvmppc_emulate_tabort(struct kvm_vcpu *vcpu, int ra_val) +{ + /* currently we only emulate tabort. but no emulation of other + * tabort variants since there is no kernel usage of them at + * present. + */ + unsigned long guest_msr = kvmppc_get_msr(vcpu); + uint64_t org_texasr; + + preempt_disable(); + tm_enable(); + org_texasr = mfspr(SPRN_TEXASR); + tm_abort(ra_val); + + /* CR0 = 0 | MSR[TS] | 0 */ + vcpu->arch.cr = (vcpu->arch.cr & ~(CR0_MASK << CR0_SHIFT)) | + (((guest_msr & MSR_TS_MASK) >> (MSR_TS_S_LG - 1)) + << CR0_SHIFT); + + vcpu->arch.texasr = mfspr(SPRN_TEXASR); + /* failure recording depends on Failure Summary bit, + * and tabort will be treated as nops in non-transactional + * state. + */ + if (!(org_texasr & TEXASR_FS) && + MSR_TM_ACTIVE(guest_msr)) { + vcpu->arch.texasr &= ~(TEXASR_PR | TEXASR_HV); + if (guest_msr & MSR_PR) + vcpu->arch.texasr |= TEXASR_PR; + + if (guest_msr & MSR_HV) + vcpu->arch.texasr |= TEXASR_HV; + + vcpu->arch.tfiar = kvmppc_get_pc(vcpu); + } + tm_disable(); + preempt_enable(); +} + +#endif + int kvmppc_core_emulate_op_pr(struct kvm_run *run, struct kvm_vcpu *vcpu, unsigned int inst, int *advance) { @@ -117,11 +276,28 @@ int kvmppc_core_emulate_op_pr(struct kvm_run *run, struct kvm_vcpu *vcpu, case 19: switch (get_xop(inst)) { case OP_19_XOP_RFID: - case OP_19_XOP_RFI: + case OP_19_XOP_RFI: { + unsigned long srr1 = kvmppc_get_srr1(vcpu); +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + unsigned long cur_msr = kvmppc_get_msr(vcpu); + + /* + * add rules to fit in ISA specification regarding TM + * state transistion in TM disable/Suspended state, + * and target TM state is TM inactive(00) state. (the + * change should be suppressed). + */ + if (((cur_msr & MSR_TM) == 0) && + ((srr1 & MSR_TM) == 0) && + MSR_TM_SUSPENDED(cur_msr) && + !MSR_TM_ACTIVE(srr1)) + srr1 |= MSR_TS_S; +#endif kvmppc_set_pc(vcpu, kvmppc_get_srr0(vcpu)); - kvmppc_set_msr(vcpu, kvmppc_get_srr1(vcpu)); + kvmppc_set_msr(vcpu, srr1); *advance = 0; break; + } default: emulated = EMULATE_FAIL; @@ -304,6 +480,140 @@ int kvmppc_core_emulate_op_pr(struct kvm_run *run, struct kvm_vcpu *vcpu, break; } +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + case OP_31_XOP_TBEGIN: + { + if (!cpu_has_feature(CPU_FTR_TM)) + break; + + if (!(kvmppc_get_msr(vcpu) & MSR_TM)) { + kvmppc_trigger_fac_interrupt(vcpu, FSCR_TM_LG); + emulated = EMULATE_AGAIN; + break; + } + + if (!(kvmppc_get_msr(vcpu) & MSR_PR)) { + preempt_disable(); + vcpu->arch.cr = (CR0_TBEGIN_FAILURE | + (vcpu->arch.cr & ~(CR0_MASK << CR0_SHIFT))); + + vcpu->arch.texasr = (TEXASR_FS | TEXASR_EXACT | + (((u64)(TM_CAUSE_EMULATE | TM_CAUSE_PERSISTENT)) + << TEXASR_FC_LG)); + + if ((inst >> 21) & 0x1) + vcpu->arch.texasr |= TEXASR_ROT; + + if (kvmppc_get_msr(vcpu) & MSR_HV) + vcpu->arch.texasr |= TEXASR_HV; + + vcpu->arch.tfhar = kvmppc_get_pc(vcpu) + 4; + vcpu->arch.tfiar = kvmppc_get_pc(vcpu); + + kvmppc_restore_tm_sprs(vcpu); + preempt_enable(); + } else + emulated = EMULATE_FAIL; + break; + } + case OP_31_XOP_TABORT: + { + ulong guest_msr = kvmppc_get_msr(vcpu); + unsigned long ra_val = 0; + + if (!cpu_has_feature(CPU_FTR_TM)) + break; + + if (!(kvmppc_get_msr(vcpu) & MSR_TM)) { + kvmppc_trigger_fac_interrupt(vcpu, FSCR_TM_LG); + emulated = EMULATE_AGAIN; + break; + } + + /* only emulate for privilege guest, since problem state + * guest can run with TM enabled and we don't expect to + * trap at here for that case. + */ + WARN_ON(guest_msr & MSR_PR); + + if (ra) + ra_val = kvmppc_get_gpr(vcpu, ra); + + kvmppc_emulate_tabort(vcpu, ra_val); + break; + } + case OP_31_XOP_TRECLAIM: + { + ulong guest_msr = kvmppc_get_msr(vcpu); + unsigned long ra_val = 0; + + if (!cpu_has_feature(CPU_FTR_TM)) + break; + + if (!(kvmppc_get_msr(vcpu) & MSR_TM)) { + kvmppc_trigger_fac_interrupt(vcpu, FSCR_TM_LG); + emulated = EMULATE_AGAIN; + break; + } + + /* generate interrupts based on priorities */ + if (guest_msr & MSR_PR) { + /* Privileged Instruction type Program Interrupt */ + kvmppc_core_queue_program(vcpu, SRR1_PROGPRIV); + emulated = EMULATE_AGAIN; + break; + } + + if (!MSR_TM_ACTIVE(guest_msr)) { + /* TM bad thing interrupt */ + kvmppc_core_queue_program(vcpu, SRR1_PROGTM); + emulated = EMULATE_AGAIN; + break; + } + + if (ra) + ra_val = kvmppc_get_gpr(vcpu, ra); + kvmppc_emulate_treclaim(vcpu, ra_val); + break; + } + case OP_31_XOP_TRCHKPT: + { + ulong guest_msr = kvmppc_get_msr(vcpu); + unsigned long texasr; + + if (!cpu_has_feature(CPU_FTR_TM)) + break; + + if (!(kvmppc_get_msr(vcpu) & MSR_TM)) { + kvmppc_trigger_fac_interrupt(vcpu, FSCR_TM_LG); + emulated = EMULATE_AGAIN; + break; + } + + /* generate interrupt based on priorities */ + if (guest_msr & MSR_PR) { + /* Privileged Instruction type Program Intr */ + kvmppc_core_queue_program(vcpu, SRR1_PROGPRIV); + emulated = EMULATE_AGAIN; + break; + } + + tm_enable(); + texasr = mfspr(SPRN_TEXASR); + tm_disable(); + + if (MSR_TM_ACTIVE(guest_msr) || + !(texasr & (TEXASR_FS))) { + /* TM bad thing interrupt */ + kvmppc_core_queue_program(vcpu, SRR1_PROGTM); + emulated = EMULATE_AGAIN; + break; + } + + kvmppc_emulate_trchkpt(vcpu); + break; + } +#endif default: emulated = EMULATE_FAIL; } @@ -465,13 +775,38 @@ int kvmppc_core_emulate_mtspr_pr(struct kvm_vcpu *vcpu, int sprn, ulong spr_val) break; #ifdef CONFIG_PPC_TRANSACTIONAL_MEM case SPRN_TFHAR: - vcpu->arch.tfhar = spr_val; - break; case SPRN_TEXASR: - vcpu->arch.texasr = spr_val; - break; case SPRN_TFIAR: - vcpu->arch.tfiar = spr_val; + if (!cpu_has_feature(CPU_FTR_TM)) + break; + + if (!(kvmppc_get_msr(vcpu) & MSR_TM)) { + kvmppc_trigger_fac_interrupt(vcpu, FSCR_TM_LG); + emulated = EMULATE_AGAIN; + break; + } + + if (MSR_TM_ACTIVE(kvmppc_get_msr(vcpu)) && + !((MSR_TM_SUSPENDED(kvmppc_get_msr(vcpu))) && + (sprn == SPRN_TFHAR))) { + /* it is illegal to mtspr() TM regs in + * other than non-transactional state, with + * the exception of TFHAR in suspend state. + */ + kvmppc_core_queue_program(vcpu, SRR1_PROGTM); + emulated = EMULATE_AGAIN; + break; + } + + tm_enable(); + if (sprn == SPRN_TFHAR) + mtspr(SPRN_TFHAR, spr_val); + else if (sprn == SPRN_TEXASR) + mtspr(SPRN_TEXASR, spr_val); + else + mtspr(SPRN_TFIAR, spr_val); + tm_disable(); + break; #endif #endif @@ -618,13 +953,25 @@ int kvmppc_core_emulate_mfspr_pr(struct kvm_vcpu *vcpu, int sprn, ulong *spr_val break; #ifdef CONFIG_PPC_TRANSACTIONAL_MEM case SPRN_TFHAR: - *spr_val = vcpu->arch.tfhar; - break; case SPRN_TEXASR: - *spr_val = vcpu->arch.texasr; - break; case SPRN_TFIAR: - *spr_val = vcpu->arch.tfiar; + if (!cpu_has_feature(CPU_FTR_TM)) + break; + + if (!(kvmppc_get_msr(vcpu) & MSR_TM)) { + kvmppc_trigger_fac_interrupt(vcpu, FSCR_TM_LG); + emulated = EMULATE_AGAIN; + break; + } + + tm_enable(); + if (sprn == SPRN_TFHAR) + *spr_val = mfspr(SPRN_TFHAR); + else if (sprn == SPRN_TEXASR) + *spr_val = mfspr(SPRN_TEXASR); + else if (sprn == SPRN_TFIAR) + *spr_val = mfspr(SPRN_TFIAR); + tm_disable(); break; #endif #endif diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c index 9963f65c212b..de686b340f4a 100644 --- a/arch/powerpc/kvm/book3s_hv.c +++ b/arch/powerpc/kvm/book3s_hv.c @@ -123,6 +123,32 @@ static bool no_mixing_hpt_and_radix; static void kvmppc_end_cede(struct kvm_vcpu *vcpu); static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu); +/* + * RWMR values for POWER8. These control the rate at which PURR + * and SPURR count and should be set according to the number of + * online threads in the vcore being run. + */ +#define RWMR_RPA_P8_1THREAD 0x164520C62609AECA +#define RWMR_RPA_P8_2THREAD 0x7FFF2908450D8DA9 +#define RWMR_RPA_P8_3THREAD 0x164520C62609AECA +#define RWMR_RPA_P8_4THREAD 0x199A421245058DA9 +#define RWMR_RPA_P8_5THREAD 0x164520C62609AECA +#define RWMR_RPA_P8_6THREAD 0x164520C62609AECA +#define RWMR_RPA_P8_7THREAD 0x164520C62609AECA +#define RWMR_RPA_P8_8THREAD 0x164520C62609AECA + +static unsigned long p8_rwmr_values[MAX_SMT_THREADS + 1] = { + RWMR_RPA_P8_1THREAD, + RWMR_RPA_P8_1THREAD, + RWMR_RPA_P8_2THREAD, + RWMR_RPA_P8_3THREAD, + RWMR_RPA_P8_4THREAD, + RWMR_RPA_P8_5THREAD, + RWMR_RPA_P8_6THREAD, + RWMR_RPA_P8_7THREAD, + RWMR_RPA_P8_8THREAD, +}; + static inline struct kvm_vcpu *next_runnable_thread(struct kvmppc_vcore *vc, int *ip) { @@ -371,13 +397,13 @@ static void kvmppc_dump_regs(struct kvm_vcpu *vcpu) pr_err("vcpu %p (%d):\n", vcpu, vcpu->vcpu_id); pr_err("pc = %.16lx msr = %.16llx trap = %x\n", - vcpu->arch.pc, vcpu->arch.shregs.msr, vcpu->arch.trap); + vcpu->arch.regs.nip, vcpu->arch.shregs.msr, vcpu->arch.trap); for (r = 0; r < 16; ++r) pr_err("r%2d = %.16lx r%d = %.16lx\n", r, kvmppc_get_gpr(vcpu, r), r+16, kvmppc_get_gpr(vcpu, r+16)); pr_err("ctr = %.16lx lr = %.16lx\n", - vcpu->arch.ctr, vcpu->arch.lr); + vcpu->arch.regs.ctr, vcpu->arch.regs.link); pr_err("srr0 = %.16llx srr1 = %.16llx\n", vcpu->arch.shregs.srr0, vcpu->arch.shregs.srr1); pr_err("sprg0 = %.16llx sprg1 = %.16llx\n", @@ -385,7 +411,7 @@ static void kvmppc_dump_regs(struct kvm_vcpu *vcpu) pr_err("sprg2 = %.16llx sprg3 = %.16llx\n", vcpu->arch.shregs.sprg2, vcpu->arch.shregs.sprg3); pr_err("cr = %.8x xer = %.16lx dsisr = %.8x\n", - vcpu->arch.cr, vcpu->arch.xer, vcpu->arch.shregs.dsisr); + vcpu->arch.cr, vcpu->arch.regs.xer, vcpu->arch.shregs.dsisr); pr_err("dar = %.16llx\n", vcpu->arch.shregs.dar); pr_err("fault dar = %.16lx dsisr = %.8x\n", vcpu->arch.fault_dar, vcpu->arch.fault_dsisr); @@ -1526,6 +1552,9 @@ static int kvmppc_get_one_reg_hv(struct kvm_vcpu *vcpu, u64 id, *val = get_reg_val(id, vcpu->arch.dec_expires + vcpu->arch.vcore->tb_offset); break; + case KVM_REG_PPC_ONLINE: + *val = get_reg_val(id, vcpu->arch.online); + break; default: r = -EINVAL; break; @@ -1757,6 +1786,14 @@ static int kvmppc_set_one_reg_hv(struct kvm_vcpu *vcpu, u64 id, vcpu->arch.dec_expires = set_reg_val(id, *val) - vcpu->arch.vcore->tb_offset; break; + case KVM_REG_PPC_ONLINE: + i = set_reg_val(id, *val); + if (i && !vcpu->arch.online) + atomic_inc(&vcpu->arch.vcore->online_count); + else if (!i && vcpu->arch.online) + atomic_dec(&vcpu->arch.vcore->online_count); + vcpu->arch.online = i; + break; default: r = -EINVAL; break; @@ -2850,6 +2887,25 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) } } + /* + * On POWER8, set RWMR register. + * Since it only affects PURR and SPURR, it doesn't affect + * the host, so we don't save/restore the host value. + */ + if (is_power8) { + unsigned long rwmr_val = RWMR_RPA_P8_8THREAD; + int n_online = atomic_read(&vc->online_count); + + /* + * Use the 8-thread value if we're doing split-core + * or if the vcore's online count looks bogus. + */ + if (split == 1 && threads_per_subcore == MAX_SMT_THREADS && + n_online >= 1 && n_online <= MAX_SMT_THREADS) + rwmr_val = p8_rwmr_values[n_online]; + mtspr(SPRN_RWMR, rwmr_val); + } + /* Start all the threads */ active = 0; for (sub = 0; sub < core_info.n_subcores; ++sub) { @@ -2902,6 +2958,32 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) for (sub = 0; sub < core_info.n_subcores; ++sub) spin_unlock(&core_info.vc[sub]->lock); + if (kvm_is_radix(vc->kvm)) { + int tmp = pcpu; + + /* + * Do we need to flush the process scoped TLB for the LPAR? + * + * On POWER9, individual threads can come in here, but the + * TLB is shared between the 4 threads in a core, hence + * invalidating on one thread invalidates for all. + * Thus we make all 4 threads use the same bit here. + * + * Hash must be flushed in realmode in order to use tlbiel. + */ + mtspr(SPRN_LPID, vc->kvm->arch.lpid); + isync(); + + if (cpu_has_feature(CPU_FTR_ARCH_300)) + tmp &= ~0x3UL; + + if (cpumask_test_cpu(tmp, &vc->kvm->arch.need_tlb_flush)) { + radix__local_flush_tlb_lpid_guest(vc->kvm->arch.lpid); + /* Clear the bit after the TLB flush */ + cpumask_clear_cpu(tmp, &vc->kvm->arch.need_tlb_flush); + } + } + /* * Interrupts will be enabled once we get into the guest, * so tell lockdep that we're about to enable interrupts. @@ -2912,8 +2994,12 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) srcu_idx = srcu_read_lock(&vc->kvm->srcu); + this_cpu_disable_ftrace(); + trap = __kvmppc_vcore_entry(); + this_cpu_enable_ftrace(); + srcu_read_unlock(&vc->kvm->srcu, srcu_idx); trace_hardirqs_off(); @@ -3352,6 +3438,15 @@ static int kvmppc_vcpu_run_hv(struct kvm_run *run, struct kvm_vcpu *vcpu) } #endif + /* + * Force online to 1 for the sake of old userspace which doesn't + * set it. + */ + if (!vcpu->arch.online) { + atomic_inc(&vcpu->arch.vcore->online_count); + vcpu->arch.online = 1; + } + kvmppc_core_prepare_to_enter(vcpu); /* No need to go into the guest when all we'll do is come back out */ @@ -3544,7 +3639,7 @@ static void kvmppc_core_free_memslot_hv(struct kvm_memory_slot *free, static int kvmppc_core_create_memslot_hv(struct kvm_memory_slot *slot, unsigned long npages) { - slot->arch.rmap = vzalloc(npages * sizeof(*slot->arch.rmap)); + slot->arch.rmap = vzalloc(array_size(npages, sizeof(*slot->arch.rmap))); if (!slot->arch.rmap) return -ENOMEM; @@ -3951,8 +4046,7 @@ static int kvmppc_core_init_vm_hv(struct kvm *kvm) */ snprintf(buf, sizeof(buf), "vm%d", current->pid); kvm->arch.debugfs_dir = debugfs_create_dir(buf, kvm_debugfs_dir); - if (!IS_ERR_OR_NULL(kvm->arch.debugfs_dir)) - kvmppc_mmu_debugfs_init(kvm); + kvmppc_mmu_debugfs_init(kvm); return 0; } diff --git a/arch/powerpc/kvm/book3s_hv_builtin.c b/arch/powerpc/kvm/book3s_hv_builtin.c index de18299f92b7..d4a3f4da409b 100644 --- a/arch/powerpc/kvm/book3s_hv_builtin.c +++ b/arch/powerpc/kvm/book3s_hv_builtin.c @@ -18,6 +18,7 @@ #include #include +#include #include #include #include @@ -211,9 +212,9 @@ long kvmppc_h_random(struct kvm_vcpu *vcpu) /* Only need to do the expensive mfmsr() on radix */ if (kvm_is_radix(vcpu->kvm) && (mfmsr() & MSR_IR)) - r = powernv_get_random_long(&vcpu->arch.gpr[4]); + r = powernv_get_random_long(&vcpu->arch.regs.gpr[4]); else - r = powernv_get_random_real_mode(&vcpu->arch.gpr[4]); + r = powernv_get_random_real_mode(&vcpu->arch.regs.gpr[4]); if (r) return H_SUCCESS; @@ -562,7 +563,7 @@ unsigned long kvmppc_rm_h_xirr_x(struct kvm_vcpu *vcpu) { if (!kvmppc_xics_enabled(vcpu)) return H_TOO_HARD; - vcpu->arch.gpr[5] = get_tb(); + vcpu->arch.regs.gpr[5] = get_tb(); if (xive_enabled()) { if (is_rm()) return xive_rm_h_xirr(vcpu); @@ -633,7 +634,19 @@ int kvmppc_rm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr) void kvmppc_bad_interrupt(struct pt_regs *regs) { - die("Bad interrupt in KVM entry/exit code", regs, SIGABRT); + /* + * 100 could happen at any time, 200 can happen due to invalid real + * address access for example (or any time due to a hardware problem). + */ + if (TRAP(regs) == 0x100) { + get_paca()->in_nmi++; + system_reset_exception(regs); + get_paca()->in_nmi--; + } else if (TRAP(regs) == 0x200) { + machine_check_exception(regs); + } else { + die("Bad interrupt in KVM entry/exit code", regs, SIGABRT); + } panic("Bad KVM trap"); } diff --git a/arch/powerpc/kvm/book3s_hv_interrupts.S b/arch/powerpc/kvm/book3s_hv_interrupts.S index 0e8493033288..82f2ff9410b6 100644 --- a/arch/powerpc/kvm/book3s_hv_interrupts.S +++ b/arch/powerpc/kvm/book3s_hv_interrupts.S @@ -137,7 +137,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300) /* * We return here in virtual mode after the guest exits * with something that we can't handle in real mode. - * Interrupts are enabled again at this point. + * Interrupts are still hard-disabled. */ /* diff --git a/arch/powerpc/kvm/book3s_hv_rm_mmu.c b/arch/powerpc/kvm/book3s_hv_rm_mmu.c index 78e6a392330f..1f22d9e977d4 100644 --- a/arch/powerpc/kvm/book3s_hv_rm_mmu.c +++ b/arch/powerpc/kvm/book3s_hv_rm_mmu.c @@ -418,7 +418,8 @@ long kvmppc_h_enter(struct kvm_vcpu *vcpu, unsigned long flags, long pte_index, unsigned long pteh, unsigned long ptel) { return kvmppc_do_h_enter(vcpu->kvm, flags, pte_index, pteh, ptel, - vcpu->arch.pgdir, true, &vcpu->arch.gpr[4]); + vcpu->arch.pgdir, true, + &vcpu->arch.regs.gpr[4]); } #ifdef __BIG_ENDIAN__ @@ -434,24 +435,6 @@ static inline int is_mmio_hpte(unsigned long v, unsigned long r) (HPTE_R_KEY_HI | HPTE_R_KEY_LO)); } -static inline int try_lock_tlbie(unsigned int *lock) -{ - unsigned int tmp, old; - unsigned int token = LOCK_TOKEN; - - asm volatile("1:lwarx %1,0,%2\n" - " cmpwi cr0,%1,0\n" - " bne 2f\n" - " stwcx. %3,0,%2\n" - " bne- 1b\n" - " isync\n" - "2:" - : "=&r" (tmp), "=&r" (old) - : "r" (lock), "r" (token) - : "cc", "memory"); - return old == 0; -} - static void do_tlbies(struct kvm *kvm, unsigned long *rbvalues, long npages, int global, bool need_sync) { @@ -463,8 +446,6 @@ static void do_tlbies(struct kvm *kvm, unsigned long *rbvalues, * the RS field, this is backwards-compatible with P7 and P8. */ if (global) { - while (!try_lock_tlbie(&kvm->arch.tlbie_lock)) - cpu_relax(); if (need_sync) asm volatile("ptesync" : : : "memory"); for (i = 0; i < npages; ++i) { @@ -483,7 +464,6 @@ static void do_tlbies(struct kvm *kvm, unsigned long *rbvalues, } asm volatile("eieio; tlbsync; ptesync" : : : "memory"); - kvm->arch.tlbie_lock = 0; } else { if (need_sync) asm volatile("ptesync" : : : "memory"); @@ -561,13 +541,13 @@ long kvmppc_h_remove(struct kvm_vcpu *vcpu, unsigned long flags, unsigned long pte_index, unsigned long avpn) { return kvmppc_do_h_remove(vcpu->kvm, flags, pte_index, avpn, - &vcpu->arch.gpr[4]); + &vcpu->arch.regs.gpr[4]); } long kvmppc_h_bulk_remove(struct kvm_vcpu *vcpu) { struct kvm *kvm = vcpu->kvm; - unsigned long *args = &vcpu->arch.gpr[4]; + unsigned long *args = &vcpu->arch.regs.gpr[4]; __be64 *hp, *hptes[4]; unsigned long tlbrb[4]; long int i, j, k, n, found, indexes[4]; @@ -787,8 +767,8 @@ long kvmppc_h_read(struct kvm_vcpu *vcpu, unsigned long flags, r = rev[i].guest_rpte | (r & (HPTE_R_R | HPTE_R_C)); r &= ~HPTE_GR_RESERVED; } - vcpu->arch.gpr[4 + i * 2] = v; - vcpu->arch.gpr[5 + i * 2] = r; + vcpu->arch.regs.gpr[4 + i * 2] = v; + vcpu->arch.regs.gpr[5 + i * 2] = r; } return H_SUCCESS; } @@ -834,7 +814,7 @@ long kvmppc_h_clear_ref(struct kvm_vcpu *vcpu, unsigned long flags, } } } - vcpu->arch.gpr[4] = gr; + vcpu->arch.regs.gpr[4] = gr; ret = H_SUCCESS; out: unlock_hpte(hpte, v & ~HPTE_V_HVLOCK); @@ -881,7 +861,7 @@ long kvmppc_h_clear_mod(struct kvm_vcpu *vcpu, unsigned long flags, kvmppc_set_dirty_from_hpte(kvm, v, gr); } } - vcpu->arch.gpr[4] = gr; + vcpu->arch.regs.gpr[4] = gr; ret = H_SUCCESS; out: unlock_hpte(hpte, v & ~HPTE_V_HVLOCK); diff --git a/arch/powerpc/kvm/book3s_hv_rm_xics.c b/arch/powerpc/kvm/book3s_hv_rm_xics.c index 2a862618f072..758d1d23215e 100644 --- a/arch/powerpc/kvm/book3s_hv_rm_xics.c +++ b/arch/powerpc/kvm/book3s_hv_rm_xics.c @@ -517,7 +517,7 @@ unsigned long xics_rm_h_xirr(struct kvm_vcpu *vcpu) } while (!icp_rm_try_update(icp, old_state, new_state)); /* Return the result in GPR4 */ - vcpu->arch.gpr[4] = xirr; + vcpu->arch.regs.gpr[4] = xirr; return check_too_hard(xics, icp); } diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S index 07ca1b2a7966..153988d878e8 100644 --- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S +++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S @@ -39,8 +39,6 @@ BEGIN_FTR_SECTION; \ extsw reg, reg; \ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_300) -#define VCPU_GPRS_TM(reg) (((reg) * ULONG_SIZE) + VCPU_GPR_TM) - /* Values in HSTATE_NAPPING(r13) */ #define NAPPING_CEDE 1 #define NAPPING_NOVCPU 2 @@ -342,6 +340,9 @@ kvm_start_guest: ld r2,PACATOC(r13) + li r0,0 + stb r0,PACA_FTRACE_ENABLED(r13) + li r0,KVM_HWTHREAD_IN_KVM stb r0,HSTATE_HWTHREAD_STATE(r13) @@ -636,6 +637,10 @@ kvmppc_hv_entry: /* Primary thread switches to guest partition. */ cmpwi r6,0 bne 10f + + /* Radix has already switched LPID and flushed core TLB */ + bne cr7, 22f + lwz r7,KVM_LPID(r9) BEGIN_FTR_SECTION ld r6,KVM_SDR1(r9) @@ -647,7 +652,7 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_300) mtspr SPRN_LPID,r7 isync - /* See if we need to flush the TLB */ + /* See if we need to flush the TLB. Hash has to be done in RM */ lhz r6,PACAPACAINDEX(r13) /* test_bit(cpu, need_tlb_flush) */ BEGIN_FTR_SECTION /* @@ -674,15 +679,10 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300) li r7,0x800 /* IS field = 0b10 */ ptesync li r0,0 /* RS for P9 version of tlbiel */ - bne cr7, 29f 28: tlbiel r7 /* On P9, rs=0, RIC=0, PRS=0, R=0 */ addi r7,r7,0x1000 bdnz 28b - b 30f -29: PPC_TLBIEL(7,0,2,1,1) /* for radix, RIC=2, PRS=1, R=1 */ - addi r7,r7,0x1000 - bdnz 29b -30: ptesync + ptesync 23: ldarx r7,0,r6 /* clear the bit after TLB flushed */ andc r7,r7,r8 stdcx. r7,0,r6 @@ -796,7 +796,10 @@ END_FTR_SECTION(CPU_FTR_TM | CPU_FTR_P9_TM_HV_ASSIST, 0) /* * NOTE THAT THIS TRASHES ALL NON-VOLATILE REGISTERS INCLUDING CR */ - bl kvmppc_restore_tm + mr r3, r4 + ld r4, VCPU_MSR(r3) + bl kvmppc_restore_tm_hv + ld r4, HSTATE_KVM_VCPU(r13) 91: #endif @@ -1780,7 +1783,10 @@ END_FTR_SECTION(CPU_FTR_TM | CPU_FTR_P9_TM_HV_ASSIST, 0) /* * NOTE THAT THIS TRASHES ALL NON-VOLATILE REGISTERS INCLUDING CR */ - bl kvmppc_save_tm + mr r3, r9 + ld r4, VCPU_MSR(r3) + bl kvmppc_save_tm_hv + ld r9, HSTATE_KVM_VCPU(r13) 91: #endif @@ -2683,8 +2689,9 @@ END_FTR_SECTION(CPU_FTR_TM | CPU_FTR_P9_TM_HV_ASSIST, 0) /* * NOTE THAT THIS TRASHES ALL NON-VOLATILE REGISTERS INCLUDING CR */ - ld r9, HSTATE_KVM_VCPU(r13) - bl kvmppc_save_tm + ld r3, HSTATE_KVM_VCPU(r13) + ld r4, VCPU_MSR(r3) + bl kvmppc_save_tm_hv 91: #endif @@ -2802,7 +2809,10 @@ END_FTR_SECTION(CPU_FTR_TM | CPU_FTR_P9_TM_HV_ASSIST, 0) /* * NOTE THAT THIS TRASHES ALL NON-VOLATILE REGISTERS INCLUDING CR */ - bl kvmppc_restore_tm + mr r3, r4 + ld r4, VCPU_MSR(r3) + bl kvmppc_restore_tm_hv + ld r4, HSTATE_KVM_VCPU(r13) 91: #endif @@ -3123,11 +3133,22 @@ END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) #ifdef CONFIG_PPC_TRANSACTIONAL_MEM /* * Save transactional state and TM-related registers. - * Called with r9 pointing to the vcpu struct. + * Called with r3 pointing to the vcpu struct and r4 containing + * the guest MSR value. * This can modify all checkpointed registers, but - * restores r1, r2 and r9 (vcpu pointer) before exit. + * restores r1 and r2 before exit. */ -kvmppc_save_tm: +kvmppc_save_tm_hv: + /* See if we need to handle fake suspend mode */ +BEGIN_FTR_SECTION + b __kvmppc_save_tm +END_FTR_SECTION_IFCLR(CPU_FTR_P9_TM_HV_ASSIST) + + lbz r0, HSTATE_FAKE_SUSPEND(r13) /* Were we fake suspended? */ + cmpwi r0, 0 + beq __kvmppc_save_tm + + /* The following code handles the fake_suspend = 1 case */ mflr r0 std r0, PPC_LR_STKOFF(r1) stdu r1, -PPC_MIN_STKFRM(r1) @@ -3138,59 +3159,37 @@ kvmppc_save_tm: rldimi r8, r0, MSR_TM_LG, 63-MSR_TM_LG mtmsrd r8 - ld r5, VCPU_MSR(r9) - rldicl. r5, r5, 64 - MSR_TS_S_LG, 62 - beq 1f /* TM not active in guest. */ - - std r1, HSTATE_HOST_R1(r13) - li r3, TM_CAUSE_KVM_RESCHED - -BEGIN_FTR_SECTION - lbz r0, HSTATE_FAKE_SUSPEND(r13) /* Were we fake suspended? */ - cmpwi r0, 0 - beq 3f rldicl. r8, r8, 64 - MSR_TS_S_LG, 62 /* Did we actually hrfid? */ beq 4f -BEGIN_FTR_SECTION_NESTED(96) +BEGIN_FTR_SECTION bl pnv_power9_force_smt4_catch -END_FTR_SECTION_NESTED(CPU_FTR_P9_TM_XER_SO_BUG, CPU_FTR_P9_TM_XER_SO_BUG, 96) +END_FTR_SECTION_IFSET(CPU_FTR_P9_TM_XER_SO_BUG) nop - b 6f -3: - /* Emulation of the treclaim instruction needs TEXASR before treclaim */ - mfspr r6, SPRN_TEXASR - std r6, VCPU_ORIG_TEXASR(r9) -6: -END_FTR_SECTION_IFSET(CPU_FTR_P9_TM_HV_ASSIST) - /* Clear the MSR RI since r1, r13 are all going to be foobar. */ + std r1, HSTATE_HOST_R1(r13) + + /* Clear the MSR RI since r1, r13 may be foobar. */ li r5, 0 mtmsrd r5, 1 - /* All GPRs are volatile at this point. */ + /* We have to treclaim here because that's the only way to do S->N */ + li r3, TM_CAUSE_KVM_RESCHED TRECLAIM(R3) - /* Temporarily store r13 and r9 so we have some regs to play with */ - SET_SCRATCH0(r13) - GET_PACA(r13) - std r9, PACATMSCRATCH(r13) - - /* If doing TM emulation on POWER9 DD2.2, check for fake suspend mode */ -BEGIN_FTR_SECTION - lbz r9, HSTATE_FAKE_SUSPEND(r13) - cmpwi r9, 0 - beq 2f /* * We were in fake suspend, so we are not going to save the * register state as the guest checkpointed state (since * we already have it), therefore we can now use any volatile GPR. */ - /* Reload stack pointer and TOC. */ + /* Reload PACA pointer, stack pointer and TOC. */ + GET_PACA(r13) ld r1, HSTATE_HOST_R1(r13) ld r2, PACATOC(r13) + /* Set MSR RI now we have r1 and r13 back. */ li r5, MSR_RI mtmsrd r5, 1 + HMT_MEDIUM ld r6, HSTATE_DSCR(r13) mtspr SPRN_DSCR, r6 @@ -3205,85 +3204,9 @@ END_FTR_SECTION_NESTED(CPU_FTR_P9_TM_XER_SO_BUG, CPU_FTR_P9_TM_XER_SO_BUG, 96) li r0, PSSCR_FAKE_SUSPEND andc r3, r3, r0 mtspr SPRN_PSSCR, r3 - ld r9, HSTATE_KVM_VCPU(r13) - /* Don't save TEXASR, use value from last exit in real suspend state */ - b 11f -2: -END_FTR_SECTION_IFSET(CPU_FTR_P9_TM_HV_ASSIST) + /* Don't save TEXASR, use value from last exit in real suspend state */ ld r9, HSTATE_KVM_VCPU(r13) - - /* Get a few more GPRs free. */ - std r29, VCPU_GPRS_TM(29)(r9) - std r30, VCPU_GPRS_TM(30)(r9) - std r31, VCPU_GPRS_TM(31)(r9) - - /* Save away PPR and DSCR soon so don't run with user values. */ - mfspr r31, SPRN_PPR - HMT_MEDIUM - mfspr r30, SPRN_DSCR - ld r29, HSTATE_DSCR(r13) - mtspr SPRN_DSCR, r29 - - /* Save all but r9, r13 & r29-r31 */ - reg = 0 - .rept 29 - .if (reg != 9) && (reg != 13) - std reg, VCPU_GPRS_TM(reg)(r9) - .endif - reg = reg + 1 - .endr - /* ... now save r13 */ - GET_SCRATCH0(r4) - std r4, VCPU_GPRS_TM(13)(r9) - /* ... and save r9 */ - ld r4, PACATMSCRATCH(r13) - std r4, VCPU_GPRS_TM(9)(r9) - - /* Reload stack pointer and TOC. */ - ld r1, HSTATE_HOST_R1(r13) - ld r2, PACATOC(r13) - - /* Set MSR RI now we have r1 and r13 back. */ - li r5, MSR_RI - mtmsrd r5, 1 - - /* Save away checkpinted SPRs. */ - std r31, VCPU_PPR_TM(r9) - std r30, VCPU_DSCR_TM(r9) - mflr r5 - mfcr r6 - mfctr r7 - mfspr r8, SPRN_AMR - mfspr r10, SPRN_TAR - mfxer r11 - std r5, VCPU_LR_TM(r9) - stw r6, VCPU_CR_TM(r9) - std r7, VCPU_CTR_TM(r9) - std r8, VCPU_AMR_TM(r9) - std r10, VCPU_TAR_TM(r9) - std r11, VCPU_XER_TM(r9) - - /* Restore r12 as trap number. */ - lwz r12, VCPU_TRAP(r9) - - /* Save FP/VSX. */ - addi r3, r9, VCPU_FPRS_TM - bl store_fp_state - addi r3, r9, VCPU_VRS_TM - bl store_vr_state - mfspr r6, SPRN_VRSAVE - stw r6, VCPU_VRSAVE_TM(r9) -1: - /* - * We need to save these SPRs after the treclaim so that the software - * error code is recorded correctly in the TEXASR. Also the user may - * change these outside of a transaction, so they must always be - * context switched. - */ - mfspr r7, SPRN_TEXASR - std r7, VCPU_TEXASR(r9) -11: mfspr r5, SPRN_TFHAR mfspr r6, SPRN_TFIAR std r5, VCPU_TFHAR(r9) @@ -3296,149 +3219,63 @@ END_FTR_SECTION_IFSET(CPU_FTR_P9_TM_HV_ASSIST) /* * Restore transactional state and TM-related registers. - * Called with r4 pointing to the vcpu struct. + * Called with r3 pointing to the vcpu struct + * and r4 containing the guest MSR value. * This potentially modifies all checkpointed registers. - * It restores r1, r2, r4 from the PACA. + * It restores r1 and r2 from the PACA. */ -kvmppc_restore_tm: +kvmppc_restore_tm_hv: + /* + * If we are doing TM emulation for the guest on a POWER9 DD2, + * then we don't actually do a trechkpt -- we either set up + * fake-suspend mode, or emulate a TM rollback. + */ +BEGIN_FTR_SECTION + b __kvmppc_restore_tm +END_FTR_SECTION_IFCLR(CPU_FTR_P9_TM_HV_ASSIST) mflr r0 std r0, PPC_LR_STKOFF(r1) - /* Turn on TM/FP/VSX/VMX so we can restore them. */ + li r0, 0 + stb r0, HSTATE_FAKE_SUSPEND(r13) + + /* Turn on TM so we can restore TM SPRs */ mfmsr r5 - li r6, MSR_TM >> 32 - sldi r6, r6, 32 - or r5, r5, r6 - ori r5, r5, MSR_FP - oris r5, r5, (MSR_VEC | MSR_VSX)@h + li r0, 1 + rldimi r5, r0, MSR_TM_LG, 63-MSR_TM_LG mtmsrd r5 /* * The user may change these outside of a transaction, so they must * always be context switched. */ - ld r5, VCPU_TFHAR(r4) - ld r6, VCPU_TFIAR(r4) - ld r7, VCPU_TEXASR(r4) + ld r5, VCPU_TFHAR(r3) + ld r6, VCPU_TFIAR(r3) + ld r7, VCPU_TEXASR(r3) mtspr SPRN_TFHAR, r5 mtspr SPRN_TFIAR, r6 mtspr SPRN_TEXASR, r7 - li r0, 0 - stb r0, HSTATE_FAKE_SUSPEND(r13) - ld r5, VCPU_MSR(r4) - rldicl. r5, r5, 64 - MSR_TS_S_LG, 62 + rldicl. r5, r4, 64 - MSR_TS_S_LG, 62 beqlr /* TM not active in guest */ - std r1, HSTATE_HOST_R1(r13) - /* Make sure the failure summary is set, otherwise we'll program check - * when we trechkpt. It's possible that this might have been not set - * on a kvmppc_set_one_reg() call but we shouldn't let this crash the - * host. - */ + /* Make sure the failure summary is set */ oris r7, r7, (TEXASR_FS)@h mtspr SPRN_TEXASR, r7 - /* - * If we are doing TM emulation for the guest on a POWER9 DD2, - * then we don't actually do a trechkpt -- we either set up - * fake-suspend mode, or emulate a TM rollback. - */ -BEGIN_FTR_SECTION - b .Ldo_tm_fake_load -END_FTR_SECTION_IFSET(CPU_FTR_P9_TM_HV_ASSIST) - - /* - * We need to load up the checkpointed state for the guest. - * We need to do this early as it will blow away any GPRs, VSRs and - * some SPRs. - */ - - mr r31, r4 - addi r3, r31, VCPU_FPRS_TM - bl load_fp_state - addi r3, r31, VCPU_VRS_TM - bl load_vr_state - mr r4, r31 - lwz r7, VCPU_VRSAVE_TM(r4) - mtspr SPRN_VRSAVE, r7 - - ld r5, VCPU_LR_TM(r4) - lwz r6, VCPU_CR_TM(r4) - ld r7, VCPU_CTR_TM(r4) - ld r8, VCPU_AMR_TM(r4) - ld r9, VCPU_TAR_TM(r4) - ld r10, VCPU_XER_TM(r4) - mtlr r5 - mtcr r6 - mtctr r7 - mtspr SPRN_AMR, r8 - mtspr SPRN_TAR, r9 - mtxer r10 - - /* - * Load up PPR and DSCR values but don't put them in the actual SPRs - * till the last moment to avoid running with userspace PPR and DSCR for - * too long. - */ - ld r29, VCPU_DSCR_TM(r4) - ld r30, VCPU_PPR_TM(r4) - - std r2, PACATMSCRATCH(r13) /* Save TOC */ - - /* Clear the MSR RI since r1, r13 are all going to be foobar. */ - li r5, 0 - mtmsrd r5, 1 - - /* Load GPRs r0-r28 */ - reg = 0 - .rept 29 - ld reg, VCPU_GPRS_TM(reg)(r31) - reg = reg + 1 - .endr - - mtspr SPRN_DSCR, r29 - mtspr SPRN_PPR, r30 - - /* Load final GPRs */ - ld 29, VCPU_GPRS_TM(29)(r31) - ld 30, VCPU_GPRS_TM(30)(r31) - ld 31, VCPU_GPRS_TM(31)(r31) - - /* TM checkpointed state is now setup. All GPRs are now volatile. */ - TRECHKPT - - /* Now let's get back the state we need. */ - HMT_MEDIUM - GET_PACA(r13) - ld r29, HSTATE_DSCR(r13) - mtspr SPRN_DSCR, r29 - ld r4, HSTATE_KVM_VCPU(r13) - ld r1, HSTATE_HOST_R1(r13) - ld r2, PACATMSCRATCH(r13) - - /* Set the MSR RI since we have our registers back. */ - li r5, MSR_RI - mtmsrd r5, 1 -9: - ld r0, PPC_LR_STKOFF(r1) - mtlr r0 - blr - -.Ldo_tm_fake_load: cmpwi r5, 1 /* check for suspended state */ bgt 10f stb r5, HSTATE_FAKE_SUSPEND(r13) - b 9b /* and return */ + b 9f /* and return */ 10: stdu r1, -PPC_MIN_STKFRM(r1) /* guest is in transactional state, so simulate rollback */ - mr r3, r4 bl kvmhv_emulate_tm_rollback nop - ld r4, HSTATE_KVM_VCPU(r13) /* our vcpu pointer has been trashed */ addi r1, r1, PPC_MIN_STKFRM - b 9b -#endif +9: ld r0, PPC_LR_STKOFF(r1) + mtlr r0 + blr +#endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ /* * We come here if we get any exception or interrupt while we are @@ -3569,6 +3406,8 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX) bcl 20, 31, .+4 5: mflr r3 addi r3, r3, 9f - 5b + li r4, -1 + rldimi r3, r4, 62, 0 /* ensure 0xc000000000000000 bits are set */ ld r4, PACAKMSR(r13) mtspr SPRN_SRR0, r3 mtspr SPRN_SRR1, r4 diff --git a/arch/powerpc/kvm/book3s_hv_tm.c b/arch/powerpc/kvm/book3s_hv_tm.c index bf710ad3a6d7..008285058f9b 100644 --- a/arch/powerpc/kvm/book3s_hv_tm.c +++ b/arch/powerpc/kvm/book3s_hv_tm.c @@ -19,7 +19,7 @@ static void emulate_tx_failure(struct kvm_vcpu *vcpu, u64 failure_cause) u64 texasr, tfiar; u64 msr = vcpu->arch.shregs.msr; - tfiar = vcpu->arch.pc & ~0x3ull; + tfiar = vcpu->arch.regs.nip & ~0x3ull; texasr = (failure_cause << 56) | TEXASR_ABORT | TEXASR_FS | TEXASR_EXACT; if (MSR_TM_SUSPENDED(vcpu->arch.shregs.msr)) texasr |= TEXASR_SUSP; @@ -57,8 +57,8 @@ int kvmhv_p9_tm_emulation(struct kvm_vcpu *vcpu) (newmsr & MSR_TM))); newmsr = sanitize_msr(newmsr); vcpu->arch.shregs.msr = newmsr; - vcpu->arch.cfar = vcpu->arch.pc - 4; - vcpu->arch.pc = vcpu->arch.shregs.srr0; + vcpu->arch.cfar = vcpu->arch.regs.nip - 4; + vcpu->arch.regs.nip = vcpu->arch.shregs.srr0; return RESUME_GUEST; case PPC_INST_RFEBB: @@ -90,8 +90,8 @@ int kvmhv_p9_tm_emulation(struct kvm_vcpu *vcpu) vcpu->arch.bescr = bescr; msr = (msr & ~MSR_TS_MASK) | MSR_TS_T; vcpu->arch.shregs.msr = msr; - vcpu->arch.cfar = vcpu->arch.pc - 4; - vcpu->arch.pc = vcpu->arch.ebbrr; + vcpu->arch.cfar = vcpu->arch.regs.nip - 4; + vcpu->arch.regs.nip = vcpu->arch.ebbrr; return RESUME_GUEST; case PPC_INST_MTMSRD: diff --git a/arch/powerpc/kvm/book3s_hv_tm_builtin.c b/arch/powerpc/kvm/book3s_hv_tm_builtin.c index d98ccfd2b88c..b2c7c6fca4f9 100644 --- a/arch/powerpc/kvm/book3s_hv_tm_builtin.c +++ b/arch/powerpc/kvm/book3s_hv_tm_builtin.c @@ -35,8 +35,8 @@ int kvmhv_p9_tm_emulation_early(struct kvm_vcpu *vcpu) return 0; newmsr = sanitize_msr(newmsr); vcpu->arch.shregs.msr = newmsr; - vcpu->arch.cfar = vcpu->arch.pc - 4; - vcpu->arch.pc = vcpu->arch.shregs.srr0; + vcpu->arch.cfar = vcpu->arch.regs.nip - 4; + vcpu->arch.regs.nip = vcpu->arch.shregs.srr0; return 1; case PPC_INST_RFEBB: @@ -58,8 +58,8 @@ int kvmhv_p9_tm_emulation_early(struct kvm_vcpu *vcpu) mtspr(SPRN_BESCR, bescr); msr = (msr & ~MSR_TS_MASK) | MSR_TS_T; vcpu->arch.shregs.msr = msr; - vcpu->arch.cfar = vcpu->arch.pc - 4; - vcpu->arch.pc = mfspr(SPRN_EBBRR); + vcpu->arch.cfar = vcpu->arch.regs.nip - 4; + vcpu->arch.regs.nip = mfspr(SPRN_EBBRR); return 1; case PPC_INST_MTMSRD: @@ -103,7 +103,7 @@ int kvmhv_p9_tm_emulation_early(struct kvm_vcpu *vcpu) void kvmhv_emulate_tm_rollback(struct kvm_vcpu *vcpu) { vcpu->arch.shregs.msr &= ~MSR_TS_MASK; /* go to N state */ - vcpu->arch.pc = vcpu->arch.tfhar; + vcpu->arch.regs.nip = vcpu->arch.tfhar; copy_from_checkpoint(vcpu); vcpu->arch.cr = (vcpu->arch.cr & 0x0fffffff) | 0xa0000000; } diff --git a/arch/powerpc/kvm/book3s_pr.c b/arch/powerpc/kvm/book3s_pr.c index d3f304d06adf..c3b8006f0eac 100644 --- a/arch/powerpc/kvm/book3s_pr.c +++ b/arch/powerpc/kvm/book3s_pr.c @@ -42,6 +42,8 @@ #include #include #include +#include +#include #include "book3s.h" @@ -53,7 +55,9 @@ static int kvmppc_handle_ext(struct kvm_vcpu *vcpu, unsigned int exit_nr, ulong msr); -static void kvmppc_giveup_fac(struct kvm_vcpu *vcpu, ulong fac); +#ifdef CONFIG_PPC_BOOK3S_64 +static int kvmppc_handle_fac(struct kvm_vcpu *vcpu, ulong fac); +#endif /* Some compatibility defines */ #ifdef CONFIG_PPC_BOOK3S_32 @@ -114,6 +118,8 @@ static void kvmppc_core_vcpu_load_pr(struct kvm_vcpu *vcpu, int cpu) if (kvmppc_is_split_real(vcpu)) kvmppc_fixup_split_real(vcpu); + + kvmppc_restore_tm_pr(vcpu); } static void kvmppc_core_vcpu_put_pr(struct kvm_vcpu *vcpu) @@ -133,6 +139,7 @@ static void kvmppc_core_vcpu_put_pr(struct kvm_vcpu *vcpu) kvmppc_giveup_ext(vcpu, MSR_FP | MSR_VEC | MSR_VSX); kvmppc_giveup_fac(vcpu, FSCR_TAR_LG); + kvmppc_save_tm_pr(vcpu); /* Enable AIL if supported */ if (cpu_has_feature(CPU_FTR_HVMODE) && @@ -147,25 +154,25 @@ void kvmppc_copy_to_svcpu(struct kvm_vcpu *vcpu) { struct kvmppc_book3s_shadow_vcpu *svcpu = svcpu_get(vcpu); - svcpu->gpr[0] = vcpu->arch.gpr[0]; - svcpu->gpr[1] = vcpu->arch.gpr[1]; - svcpu->gpr[2] = vcpu->arch.gpr[2]; - svcpu->gpr[3] = vcpu->arch.gpr[3]; - svcpu->gpr[4] = vcpu->arch.gpr[4]; - svcpu->gpr[5] = vcpu->arch.gpr[5]; - svcpu->gpr[6] = vcpu->arch.gpr[6]; - svcpu->gpr[7] = vcpu->arch.gpr[7]; - svcpu->gpr[8] = vcpu->arch.gpr[8]; - svcpu->gpr[9] = vcpu->arch.gpr[9]; - svcpu->gpr[10] = vcpu->arch.gpr[10]; - svcpu->gpr[11] = vcpu->arch.gpr[11]; - svcpu->gpr[12] = vcpu->arch.gpr[12]; - svcpu->gpr[13] = vcpu->arch.gpr[13]; + svcpu->gpr[0] = vcpu->arch.regs.gpr[0]; + svcpu->gpr[1] = vcpu->arch.regs.gpr[1]; + svcpu->gpr[2] = vcpu->arch.regs.gpr[2]; + svcpu->gpr[3] = vcpu->arch.regs.gpr[3]; + svcpu->gpr[4] = vcpu->arch.regs.gpr[4]; + svcpu->gpr[5] = vcpu->arch.regs.gpr[5]; + svcpu->gpr[6] = vcpu->arch.regs.gpr[6]; + svcpu->gpr[7] = vcpu->arch.regs.gpr[7]; + svcpu->gpr[8] = vcpu->arch.regs.gpr[8]; + svcpu->gpr[9] = vcpu->arch.regs.gpr[9]; + svcpu->gpr[10] = vcpu->arch.regs.gpr[10]; + svcpu->gpr[11] = vcpu->arch.regs.gpr[11]; + svcpu->gpr[12] = vcpu->arch.regs.gpr[12]; + svcpu->gpr[13] = vcpu->arch.regs.gpr[13]; svcpu->cr = vcpu->arch.cr; - svcpu->xer = vcpu->arch.xer; - svcpu->ctr = vcpu->arch.ctr; - svcpu->lr = vcpu->arch.lr; - svcpu->pc = vcpu->arch.pc; + svcpu->xer = vcpu->arch.regs.xer; + svcpu->ctr = vcpu->arch.regs.ctr; + svcpu->lr = vcpu->arch.regs.link; + svcpu->pc = vcpu->arch.regs.nip; #ifdef CONFIG_PPC_BOOK3S_64 svcpu->shadow_fscr = vcpu->arch.shadow_fscr; #endif @@ -182,10 +189,45 @@ void kvmppc_copy_to_svcpu(struct kvm_vcpu *vcpu) svcpu_put(svcpu); } +static void kvmppc_recalc_shadow_msr(struct kvm_vcpu *vcpu) +{ + ulong guest_msr = kvmppc_get_msr(vcpu); + ulong smsr = guest_msr; + + /* Guest MSR values */ +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + smsr &= MSR_FE0 | MSR_FE1 | MSR_SF | MSR_SE | MSR_BE | MSR_LE | + MSR_TM | MSR_TS_MASK; +#else + smsr &= MSR_FE0 | MSR_FE1 | MSR_SF | MSR_SE | MSR_BE | MSR_LE; +#endif + /* Process MSR values */ + smsr |= MSR_ME | MSR_RI | MSR_IR | MSR_DR | MSR_PR | MSR_EE; + /* External providers the guest reserved */ + smsr |= (guest_msr & vcpu->arch.guest_owned_ext); + /* 64-bit Process MSR values */ +#ifdef CONFIG_PPC_BOOK3S_64 + smsr |= MSR_ISF | MSR_HV; +#endif +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + /* + * in guest privileged state, we want to fail all TM transactions. + * So disable MSR TM bit so that all tbegin. will be able to be + * trapped into host. + */ + if (!(guest_msr & MSR_PR)) + smsr &= ~MSR_TM; +#endif + vcpu->arch.shadow_msr = smsr; +} + /* Copy data touched by real-mode code from shadow vcpu back to vcpu */ void kvmppc_copy_from_svcpu(struct kvm_vcpu *vcpu) { struct kvmppc_book3s_shadow_vcpu *svcpu = svcpu_get(vcpu); +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + ulong old_msr; +#endif /* * Maybe we were already preempted and synced the svcpu from @@ -194,25 +236,25 @@ void kvmppc_copy_from_svcpu(struct kvm_vcpu *vcpu) if (!svcpu->in_use) goto out; - vcpu->arch.gpr[0] = svcpu->gpr[0]; - vcpu->arch.gpr[1] = svcpu->gpr[1]; - vcpu->arch.gpr[2] = svcpu->gpr[2]; - vcpu->arch.gpr[3] = svcpu->gpr[3]; - vcpu->arch.gpr[4] = svcpu->gpr[4]; - vcpu->arch.gpr[5] = svcpu->gpr[5]; - vcpu->arch.gpr[6] = svcpu->gpr[6]; - vcpu->arch.gpr[7] = svcpu->gpr[7]; - vcpu->arch.gpr[8] = svcpu->gpr[8]; - vcpu->arch.gpr[9] = svcpu->gpr[9]; - vcpu->arch.gpr[10] = svcpu->gpr[10]; - vcpu->arch.gpr[11] = svcpu->gpr[11]; - vcpu->arch.gpr[12] = svcpu->gpr[12]; - vcpu->arch.gpr[13] = svcpu->gpr[13]; + vcpu->arch.regs.gpr[0] = svcpu->gpr[0]; + vcpu->arch.regs.gpr[1] = svcpu->gpr[1]; + vcpu->arch.regs.gpr[2] = svcpu->gpr[2]; + vcpu->arch.regs.gpr[3] = svcpu->gpr[3]; + vcpu->arch.regs.gpr[4] = svcpu->gpr[4]; + vcpu->arch.regs.gpr[5] = svcpu->gpr[5]; + vcpu->arch.regs.gpr[6] = svcpu->gpr[6]; + vcpu->arch.regs.gpr[7] = svcpu->gpr[7]; + vcpu->arch.regs.gpr[8] = svcpu->gpr[8]; + vcpu->arch.regs.gpr[9] = svcpu->gpr[9]; + vcpu->arch.regs.gpr[10] = svcpu->gpr[10]; + vcpu->arch.regs.gpr[11] = svcpu->gpr[11]; + vcpu->arch.regs.gpr[12] = svcpu->gpr[12]; + vcpu->arch.regs.gpr[13] = svcpu->gpr[13]; vcpu->arch.cr = svcpu->cr; - vcpu->arch.xer = svcpu->xer; - vcpu->arch.ctr = svcpu->ctr; - vcpu->arch.lr = svcpu->lr; - vcpu->arch.pc = svcpu->pc; + vcpu->arch.regs.xer = svcpu->xer; + vcpu->arch.regs.ctr = svcpu->ctr; + vcpu->arch.regs.link = svcpu->lr; + vcpu->arch.regs.nip = svcpu->pc; vcpu->arch.shadow_srr1 = svcpu->shadow_srr1; vcpu->arch.fault_dar = svcpu->fault_dar; vcpu->arch.fault_dsisr = svcpu->fault_dsisr; @@ -228,12 +270,116 @@ void kvmppc_copy_from_svcpu(struct kvm_vcpu *vcpu) to_book3s(vcpu)->vtb += get_vtb() - vcpu->arch.entry_vtb; if (cpu_has_feature(CPU_FTR_ARCH_207S)) vcpu->arch.ic += mfspr(SPRN_IC) - vcpu->arch.entry_ic; + +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + /* + * Unlike other MSR bits, MSR[TS]bits can be changed at guest without + * notifying host: + * modified by unprivileged instructions like "tbegin"/"tend"/ + * "tresume"/"tsuspend" in PR KVM guest. + * + * It is necessary to sync here to calculate a correct shadow_msr. + * + * privileged guest's tbegin will be failed at present. So we + * only take care of problem state guest. + */ + old_msr = kvmppc_get_msr(vcpu); + if (unlikely((old_msr & MSR_PR) && + (vcpu->arch.shadow_srr1 & (MSR_TS_MASK)) != + (old_msr & (MSR_TS_MASK)))) { + old_msr &= ~(MSR_TS_MASK); + old_msr |= (vcpu->arch.shadow_srr1 & (MSR_TS_MASK)); + kvmppc_set_msr_fast(vcpu, old_msr); + kvmppc_recalc_shadow_msr(vcpu); + } +#endif + svcpu->in_use = false; out: svcpu_put(svcpu); } +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM +void kvmppc_save_tm_sprs(struct kvm_vcpu *vcpu) +{ + tm_enable(); + vcpu->arch.tfhar = mfspr(SPRN_TFHAR); + vcpu->arch.texasr = mfspr(SPRN_TEXASR); + vcpu->arch.tfiar = mfspr(SPRN_TFIAR); + tm_disable(); +} + +void kvmppc_restore_tm_sprs(struct kvm_vcpu *vcpu) +{ + tm_enable(); + mtspr(SPRN_TFHAR, vcpu->arch.tfhar); + mtspr(SPRN_TEXASR, vcpu->arch.texasr); + mtspr(SPRN_TFIAR, vcpu->arch.tfiar); + tm_disable(); +} + +/* loadup math bits which is enabled at kvmppc_get_msr() but not enabled at + * hardware. + */ +static void kvmppc_handle_lost_math_exts(struct kvm_vcpu *vcpu) +{ + ulong exit_nr; + ulong ext_diff = (kvmppc_get_msr(vcpu) & ~vcpu->arch.guest_owned_ext) & + (MSR_FP | MSR_VEC | MSR_VSX); + + if (!ext_diff) + return; + + if (ext_diff == MSR_FP) + exit_nr = BOOK3S_INTERRUPT_FP_UNAVAIL; + else if (ext_diff == MSR_VEC) + exit_nr = BOOK3S_INTERRUPT_ALTIVEC; + else + exit_nr = BOOK3S_INTERRUPT_VSX; + + kvmppc_handle_ext(vcpu, exit_nr, ext_diff); +} + +void kvmppc_save_tm_pr(struct kvm_vcpu *vcpu) +{ + if (!(MSR_TM_ACTIVE(kvmppc_get_msr(vcpu)))) { + kvmppc_save_tm_sprs(vcpu); + return; + } + + kvmppc_giveup_fac(vcpu, FSCR_TAR_LG); + kvmppc_giveup_ext(vcpu, MSR_VSX); + + preempt_disable(); + _kvmppc_save_tm_pr(vcpu, mfmsr()); + preempt_enable(); +} + +void kvmppc_restore_tm_pr(struct kvm_vcpu *vcpu) +{ + if (!MSR_TM_ACTIVE(kvmppc_get_msr(vcpu))) { + kvmppc_restore_tm_sprs(vcpu); + if (kvmppc_get_msr(vcpu) & MSR_TM) { + kvmppc_handle_lost_math_exts(vcpu); + if (vcpu->arch.fscr & FSCR_TAR) + kvmppc_handle_fac(vcpu, FSCR_TAR_LG); + } + return; + } + + preempt_disable(); + _kvmppc_restore_tm_pr(vcpu, kvmppc_get_msr(vcpu)); + preempt_enable(); + + if (kvmppc_get_msr(vcpu) & MSR_TM) { + kvmppc_handle_lost_math_exts(vcpu); + if (vcpu->arch.fscr & FSCR_TAR) + kvmppc_handle_fac(vcpu, FSCR_TAR_LG); + } +} +#endif + static int kvmppc_core_check_requests_pr(struct kvm_vcpu *vcpu) { int r = 1; /* Indicate we want to get back into the guest */ @@ -306,32 +452,29 @@ static void kvm_set_spte_hva_pr(struct kvm *kvm, unsigned long hva, pte_t pte) /*****************************************/ -static void kvmppc_recalc_shadow_msr(struct kvm_vcpu *vcpu) -{ - ulong guest_msr = kvmppc_get_msr(vcpu); - ulong smsr = guest_msr; - - /* Guest MSR values */ - smsr &= MSR_FE0 | MSR_FE1 | MSR_SF | MSR_SE | MSR_BE | MSR_LE; - /* Process MSR values */ - smsr |= MSR_ME | MSR_RI | MSR_IR | MSR_DR | MSR_PR | MSR_EE; - /* External providers the guest reserved */ - smsr |= (guest_msr & vcpu->arch.guest_owned_ext); - /* 64-bit Process MSR values */ -#ifdef CONFIG_PPC_BOOK3S_64 - smsr |= MSR_ISF | MSR_HV; -#endif - vcpu->arch.shadow_msr = smsr; -} - static void kvmppc_set_msr_pr(struct kvm_vcpu *vcpu, u64 msr) { - ulong old_msr = kvmppc_get_msr(vcpu); + ulong old_msr; + + /* For PAPR guest, make sure MSR reflects guest mode */ + if (vcpu->arch.papr_enabled) + msr = (msr & ~MSR_HV) | MSR_ME; #ifdef EXIT_DEBUG printk(KERN_INFO "KVM: Set MSR to 0x%llx\n", msr); #endif +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + /* We should never target guest MSR to TS=10 && PR=0, + * since we always fail transaction for guest privilege + * state. + */ + if (!(msr & MSR_PR) && MSR_TM_TRANSACTIONAL(msr)) + kvmppc_emulate_tabort(vcpu, + TM_CAUSE_KVM_FAC_UNAV | TM_CAUSE_PERSISTENT); +#endif + + old_msr = kvmppc_get_msr(vcpu); msr &= to_book3s(vcpu)->msr_mask; kvmppc_set_msr_fast(vcpu, msr); kvmppc_recalc_shadow_msr(vcpu); @@ -387,6 +530,11 @@ static void kvmppc_set_msr_pr(struct kvm_vcpu *vcpu, u64 msr) /* Preload FPU if it's enabled */ if (kvmppc_get_msr(vcpu) & MSR_FP) kvmppc_handle_ext(vcpu, BOOK3S_INTERRUPT_FP_UNAVAIL, MSR_FP); + +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + if (kvmppc_get_msr(vcpu) & MSR_TM) + kvmppc_handle_lost_math_exts(vcpu); +#endif } void kvmppc_set_pvr_pr(struct kvm_vcpu *vcpu, u32 pvr) @@ -584,24 +732,20 @@ int kvmppc_handle_pagefault(struct kvm_run *run, struct kvm_vcpu *vcpu, pte.may_execute = !data; } - if (page_found == -ENOENT) { - /* Page not found in guest PTE entries */ - u64 ssrr1 = vcpu->arch.shadow_srr1; - u64 msr = kvmppc_get_msr(vcpu); - kvmppc_set_dar(vcpu, kvmppc_get_fault_dar(vcpu)); - kvmppc_set_dsisr(vcpu, vcpu->arch.fault_dsisr); - kvmppc_set_msr_fast(vcpu, msr | (ssrr1 & 0xf8000000ULL)); - kvmppc_book3s_queue_irqprio(vcpu, vec); - } else if (page_found == -EPERM) { - /* Storage protection */ - u32 dsisr = vcpu->arch.fault_dsisr; - u64 ssrr1 = vcpu->arch.shadow_srr1; - u64 msr = kvmppc_get_msr(vcpu); - kvmppc_set_dar(vcpu, kvmppc_get_fault_dar(vcpu)); - dsisr = (dsisr & ~DSISR_NOHPTE) | DSISR_PROTFAULT; - kvmppc_set_dsisr(vcpu, dsisr); - kvmppc_set_msr_fast(vcpu, msr | (ssrr1 & 0xf8000000ULL)); - kvmppc_book3s_queue_irqprio(vcpu, vec); + if (page_found == -ENOENT || page_found == -EPERM) { + /* Page not found in guest PTE entries, or protection fault */ + u64 flags; + + if (page_found == -EPERM) + flags = DSISR_PROTFAULT; + else + flags = DSISR_NOHPTE; + if (data) { + flags |= vcpu->arch.fault_dsisr & DSISR_ISSTORE; + kvmppc_core_queue_data_storage(vcpu, eaddr, flags); + } else { + kvmppc_core_queue_inst_storage(vcpu, flags); + } } else if (page_found == -EINVAL) { /* Page not found in guest SLB */ kvmppc_set_dar(vcpu, kvmppc_get_fault_dar(vcpu)); @@ -683,7 +827,7 @@ void kvmppc_giveup_ext(struct kvm_vcpu *vcpu, ulong msr) } /* Give up facility (TAR / EBB / DSCR) */ -static void kvmppc_giveup_fac(struct kvm_vcpu *vcpu, ulong fac) +void kvmppc_giveup_fac(struct kvm_vcpu *vcpu, ulong fac) { #ifdef CONFIG_PPC_BOOK3S_64 if (!(vcpu->arch.shadow_fscr & (1ULL << fac))) { @@ -802,7 +946,7 @@ static void kvmppc_handle_lost_ext(struct kvm_vcpu *vcpu) #ifdef CONFIG_PPC_BOOK3S_64 -static void kvmppc_trigger_fac_interrupt(struct kvm_vcpu *vcpu, ulong fac) +void kvmppc_trigger_fac_interrupt(struct kvm_vcpu *vcpu, ulong fac) { /* Inject the Interrupt Cause field and trigger a guest interrupt */ vcpu->arch.fscr &= ~(0xffULL << 56); @@ -864,6 +1008,18 @@ static int kvmppc_handle_fac(struct kvm_vcpu *vcpu, ulong fac) break; } +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + /* Since we disabled MSR_TM at privilege state, the mfspr instruction + * for TM spr can trigger TM fac unavailable. In this case, the + * emulation is handled by kvmppc_emulate_fac(), which invokes + * kvmppc_emulate_mfspr() finally. But note the mfspr can include + * RT for NV registers. So it need to restore those NV reg to reflect + * the update. + */ + if ((fac == FSCR_TM_LG) && !(kvmppc_get_msr(vcpu) & MSR_PR)) + return RESUME_GUEST_NV; +#endif + return RESUME_GUEST; } @@ -872,7 +1028,12 @@ void kvmppc_set_fscr(struct kvm_vcpu *vcpu, u64 fscr) if ((vcpu->arch.fscr & FSCR_TAR) && !(fscr & FSCR_TAR)) { /* TAR got dropped, drop it in shadow too */ kvmppc_giveup_fac(vcpu, FSCR_TAR_LG); + } else if (!(vcpu->arch.fscr & FSCR_TAR) && (fscr & FSCR_TAR)) { + vcpu->arch.fscr = fscr; + kvmppc_handle_fac(vcpu, FSCR_TAR_LG); + return; } + vcpu->arch.fscr = fscr; } #endif @@ -1017,10 +1178,8 @@ int kvmppc_handle_exit_pr(struct kvm_run *run, struct kvm_vcpu *vcpu, kvmppc_mmu_pte_flush(vcpu, kvmppc_get_pc(vcpu), ~0xFFFUL); r = RESUME_GUEST; } else { - u64 msr = kvmppc_get_msr(vcpu); - msr |= shadow_srr1 & 0x58000000; - kvmppc_set_msr_fast(vcpu, msr); - kvmppc_book3s_queue_irqprio(vcpu, exit_nr); + kvmppc_core_queue_inst_storage(vcpu, + shadow_srr1 & 0x58000000); r = RESUME_GUEST; } break; @@ -1059,9 +1218,7 @@ int kvmppc_handle_exit_pr(struct kvm_run *run, struct kvm_vcpu *vcpu, r = kvmppc_handle_pagefault(run, vcpu, dar, exit_nr); srcu_read_unlock(&vcpu->kvm->srcu, idx); } else { - kvmppc_set_dar(vcpu, dar); - kvmppc_set_dsisr(vcpu, fault_dsisr); - kvmppc_book3s_queue_irqprio(vcpu, exit_nr); + kvmppc_core_queue_data_storage(vcpu, dar, fault_dsisr); r = RESUME_GUEST; } break; @@ -1092,10 +1249,13 @@ int kvmppc_handle_exit_pr(struct kvm_run *run, struct kvm_vcpu *vcpu, case BOOK3S_INTERRUPT_EXTERNAL: case BOOK3S_INTERRUPT_EXTERNAL_LEVEL: case BOOK3S_INTERRUPT_EXTERNAL_HV: + case BOOK3S_INTERRUPT_H_VIRT: vcpu->stat.ext_intr_exits++; r = RESUME_GUEST; break; + case BOOK3S_INTERRUPT_HMI: case BOOK3S_INTERRUPT_PERFMON: + case BOOK3S_INTERRUPT_SYSTEM_RESET: r = RESUME_GUEST; break; case BOOK3S_INTERRUPT_PROGRAM: @@ -1225,8 +1385,7 @@ int kvmppc_handle_exit_pr(struct kvm_run *run, struct kvm_vcpu *vcpu, } #ifdef CONFIG_PPC_BOOK3S_64 case BOOK3S_INTERRUPT_FAC_UNAVAIL: - kvmppc_handle_fac(vcpu, vcpu->arch.shadow_fscr >> 56); - r = RESUME_GUEST; + r = kvmppc_handle_fac(vcpu, vcpu->arch.shadow_fscr >> 56); break; #endif case BOOK3S_INTERRUPT_MACHINE_CHECK: @@ -1379,6 +1538,73 @@ static int kvmppc_get_one_reg_pr(struct kvm_vcpu *vcpu, u64 id, else *val = get_reg_val(id, 0); break; +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + case KVM_REG_PPC_TFHAR: + *val = get_reg_val(id, vcpu->arch.tfhar); + break; + case KVM_REG_PPC_TFIAR: + *val = get_reg_val(id, vcpu->arch.tfiar); + break; + case KVM_REG_PPC_TEXASR: + *val = get_reg_val(id, vcpu->arch.texasr); + break; + case KVM_REG_PPC_TM_GPR0 ... KVM_REG_PPC_TM_GPR31: + *val = get_reg_val(id, + vcpu->arch.gpr_tm[id-KVM_REG_PPC_TM_GPR0]); + break; + case KVM_REG_PPC_TM_VSR0 ... KVM_REG_PPC_TM_VSR63: + { + int i, j; + + i = id - KVM_REG_PPC_TM_VSR0; + if (i < 32) + for (j = 0; j < TS_FPRWIDTH; j++) + val->vsxval[j] = vcpu->arch.fp_tm.fpr[i][j]; + else { + if (cpu_has_feature(CPU_FTR_ALTIVEC)) + val->vval = vcpu->arch.vr_tm.vr[i-32]; + else + r = -ENXIO; + } + break; + } + case KVM_REG_PPC_TM_CR: + *val = get_reg_val(id, vcpu->arch.cr_tm); + break; + case KVM_REG_PPC_TM_XER: + *val = get_reg_val(id, vcpu->arch.xer_tm); + break; + case KVM_REG_PPC_TM_LR: + *val = get_reg_val(id, vcpu->arch.lr_tm); + break; + case KVM_REG_PPC_TM_CTR: + *val = get_reg_val(id, vcpu->arch.ctr_tm); + break; + case KVM_REG_PPC_TM_FPSCR: + *val = get_reg_val(id, vcpu->arch.fp_tm.fpscr); + break; + case KVM_REG_PPC_TM_AMR: + *val = get_reg_val(id, vcpu->arch.amr_tm); + break; + case KVM_REG_PPC_TM_PPR: + *val = get_reg_val(id, vcpu->arch.ppr_tm); + break; + case KVM_REG_PPC_TM_VRSAVE: + *val = get_reg_val(id, vcpu->arch.vrsave_tm); + break; + case KVM_REG_PPC_TM_VSCR: + if (cpu_has_feature(CPU_FTR_ALTIVEC)) + *val = get_reg_val(id, vcpu->arch.vr_tm.vscr.u[3]); + else + r = -ENXIO; + break; + case KVM_REG_PPC_TM_DSCR: + *val = get_reg_val(id, vcpu->arch.dscr_tm); + break; + case KVM_REG_PPC_TM_TAR: + *val = get_reg_val(id, vcpu->arch.tar_tm); + break; +#endif default: r = -EINVAL; break; @@ -1412,6 +1638,72 @@ static int kvmppc_set_one_reg_pr(struct kvm_vcpu *vcpu, u64 id, case KVM_REG_PPC_LPCR_64: kvmppc_set_lpcr_pr(vcpu, set_reg_val(id, *val)); break; +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + case KVM_REG_PPC_TFHAR: + vcpu->arch.tfhar = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TFIAR: + vcpu->arch.tfiar = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TEXASR: + vcpu->arch.texasr = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TM_GPR0 ... KVM_REG_PPC_TM_GPR31: + vcpu->arch.gpr_tm[id - KVM_REG_PPC_TM_GPR0] = + set_reg_val(id, *val); + break; + case KVM_REG_PPC_TM_VSR0 ... KVM_REG_PPC_TM_VSR63: + { + int i, j; + + i = id - KVM_REG_PPC_TM_VSR0; + if (i < 32) + for (j = 0; j < TS_FPRWIDTH; j++) + vcpu->arch.fp_tm.fpr[i][j] = val->vsxval[j]; + else + if (cpu_has_feature(CPU_FTR_ALTIVEC)) + vcpu->arch.vr_tm.vr[i-32] = val->vval; + else + r = -ENXIO; + break; + } + case KVM_REG_PPC_TM_CR: + vcpu->arch.cr_tm = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TM_XER: + vcpu->arch.xer_tm = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TM_LR: + vcpu->arch.lr_tm = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TM_CTR: + vcpu->arch.ctr_tm = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TM_FPSCR: + vcpu->arch.fp_tm.fpscr = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TM_AMR: + vcpu->arch.amr_tm = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TM_PPR: + vcpu->arch.ppr_tm = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TM_VRSAVE: + vcpu->arch.vrsave_tm = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TM_VSCR: + if (cpu_has_feature(CPU_FTR_ALTIVEC)) + vcpu->arch.vr.vscr.u[3] = set_reg_val(id, *val); + else + r = -ENXIO; + break; + case KVM_REG_PPC_TM_DSCR: + vcpu->arch.dscr_tm = set_reg_val(id, *val); + break; + case KVM_REG_PPC_TM_TAR: + vcpu->arch.tar_tm = set_reg_val(id, *val); + break; +#endif default: r = -EINVAL; break; @@ -1687,6 +1979,17 @@ static int kvm_vm_ioctl_get_smmu_info_pr(struct kvm *kvm, return 0; } + +static int kvm_configure_mmu_pr(struct kvm *kvm, struct kvm_ppc_mmuv3_cfg *cfg) +{ + if (!cpu_has_feature(CPU_FTR_ARCH_300)) + return -ENODEV; + /* Require flags and process table base and size to all be zero. */ + if (cfg->flags || cfg->process_table) + return -EINVAL; + return 0; +} + #else static int kvm_vm_ioctl_get_smmu_info_pr(struct kvm *kvm, struct kvm_ppc_smmu_info *info) @@ -1735,9 +2038,12 @@ static void kvmppc_core_destroy_vm_pr(struct kvm *kvm) static int kvmppc_core_check_processor_compat_pr(void) { /* - * Disable KVM for Power9 untill the required bits merged. + * PR KVM can work on POWER9 inside a guest partition + * running in HPT mode. It can't work if we are using + * radix translation (because radix provides no way for + * a process to have unique translations in quadrant 3). */ - if (cpu_has_feature(CPU_FTR_ARCH_300)) + if (cpu_has_feature(CPU_FTR_ARCH_300) && radix_enabled()) return -EIO; return 0; } @@ -1781,7 +2087,9 @@ static struct kvmppc_ops kvm_ops_pr = { .arch_vm_ioctl = kvm_arch_vm_ioctl_pr, #ifdef CONFIG_PPC_BOOK3S_64 .hcall_implemented = kvmppc_hcall_impl_pr, + .configure_mmu = kvm_configure_mmu_pr, #endif + .giveup_ext = kvmppc_giveup_ext, }; diff --git a/arch/powerpc/kvm/book3s_segment.S b/arch/powerpc/kvm/book3s_segment.S index 93a180ceefad..98ccc7ec5d48 100644 --- a/arch/powerpc/kvm/book3s_segment.S +++ b/arch/powerpc/kvm/book3s_segment.S @@ -383,6 +383,19 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S) */ PPC_LL r6, HSTATE_HOST_MSR(r13) +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM + /* + * We don't want to change MSR[TS] bits via rfi here. + * The actual TM handling logic will be in host with + * recovered DR/IR bits after HSTATE_VMHANDLER. + * And MSR_TM can be enabled in HOST_MSR so rfid may + * not suppress this change and can lead to exception. + * Manually set MSR to prevent TS state change here. + */ + mfmsr r7 + rldicl r7, r7, 64 - MSR_TS_S_LG, 62 + rldimi r6, r7, MSR_TS_S_LG, 63 - MSR_TS_T_LG +#endif PPC_LL r8, HSTATE_VMHANDLER(r13) #ifdef CONFIG_PPC64 diff --git a/arch/powerpc/kvm/book3s_xive_template.c b/arch/powerpc/kvm/book3s_xive_template.c index 99c3620b40d9..6e41ba7ec8f4 100644 --- a/arch/powerpc/kvm/book3s_xive_template.c +++ b/arch/powerpc/kvm/book3s_xive_template.c @@ -334,7 +334,7 @@ X_STATIC unsigned long GLUE(X_PFX,h_xirr)(struct kvm_vcpu *vcpu) */ /* Return interrupt and old CPPR in GPR4 */ - vcpu->arch.gpr[4] = hirq | (old_cppr << 24); + vcpu->arch.regs.gpr[4] = hirq | (old_cppr << 24); return H_SUCCESS; } @@ -369,7 +369,7 @@ X_STATIC unsigned long GLUE(X_PFX,h_ipoll)(struct kvm_vcpu *vcpu, unsigned long hirq = GLUE(X_PFX,scan_interrupts)(xc, pending, scan_poll); /* Return interrupt and old CPPR in GPR4 */ - vcpu->arch.gpr[4] = hirq | (xc->cppr << 24); + vcpu->arch.regs.gpr[4] = hirq | (xc->cppr << 24); return H_SUCCESS; } diff --git a/arch/powerpc/kvm/booke.c b/arch/powerpc/kvm/booke.c index 876d4f294fdd..a9ca016da670 100644 --- a/arch/powerpc/kvm/booke.c +++ b/arch/powerpc/kvm/booke.c @@ -77,8 +77,10 @@ void kvmppc_dump_vcpu(struct kvm_vcpu *vcpu) { int i; - printk("pc: %08lx msr: %08llx\n", vcpu->arch.pc, vcpu->arch.shared->msr); - printk("lr: %08lx ctr: %08lx\n", vcpu->arch.lr, vcpu->arch.ctr); + printk("pc: %08lx msr: %08llx\n", vcpu->arch.regs.nip, + vcpu->arch.shared->msr); + printk("lr: %08lx ctr: %08lx\n", vcpu->arch.regs.link, + vcpu->arch.regs.ctr); printk("srr0: %08llx srr1: %08llx\n", vcpu->arch.shared->srr0, vcpu->arch.shared->srr1); @@ -491,24 +493,25 @@ static int kvmppc_booke_irqprio_deliver(struct kvm_vcpu *vcpu, if (allowed) { switch (int_class) { case INT_CLASS_NONCRIT: - set_guest_srr(vcpu, vcpu->arch.pc, + set_guest_srr(vcpu, vcpu->arch.regs.nip, vcpu->arch.shared->msr); break; case INT_CLASS_CRIT: - set_guest_csrr(vcpu, vcpu->arch.pc, + set_guest_csrr(vcpu, vcpu->arch.regs.nip, vcpu->arch.shared->msr); break; case INT_CLASS_DBG: - set_guest_dsrr(vcpu, vcpu->arch.pc, + set_guest_dsrr(vcpu, vcpu->arch.regs.nip, vcpu->arch.shared->msr); break; case INT_CLASS_MC: - set_guest_mcsrr(vcpu, vcpu->arch.pc, + set_guest_mcsrr(vcpu, vcpu->arch.regs.nip, vcpu->arch.shared->msr); break; } - vcpu->arch.pc = vcpu->arch.ivpr | vcpu->arch.ivor[priority]; + vcpu->arch.regs.nip = vcpu->arch.ivpr | + vcpu->arch.ivor[priority]; if (update_esr == true) kvmppc_set_esr(vcpu, vcpu->arch.queued_esr); if (update_dear == true) @@ -826,7 +829,7 @@ static int emulation_exit(struct kvm_run *run, struct kvm_vcpu *vcpu) case EMULATE_FAIL: printk(KERN_CRIT "%s: emulation at %lx failed (%08x)\n", - __func__, vcpu->arch.pc, vcpu->arch.last_inst); + __func__, vcpu->arch.regs.nip, vcpu->arch.last_inst); /* For debugging, encode the failing instruction and * report it to userspace. */ run->hw.hardware_exit_reason = ~0ULL << 32; @@ -875,7 +878,7 @@ static int kvmppc_handle_debug(struct kvm_run *run, struct kvm_vcpu *vcpu) */ vcpu->arch.dbsr = 0; run->debug.arch.status = 0; - run->debug.arch.address = vcpu->arch.pc; + run->debug.arch.address = vcpu->arch.regs.nip; if (dbsr & (DBSR_IAC1 | DBSR_IAC2 | DBSR_IAC3 | DBSR_IAC4)) { run->debug.arch.status |= KVMPPC_DEBUG_BREAKPOINT; @@ -971,7 +974,7 @@ static int kvmppc_resume_inst_load(struct kvm_run *run, struct kvm_vcpu *vcpu, case EMULATE_FAIL: pr_debug("%s: load instruction from guest address %lx failed\n", - __func__, vcpu->arch.pc); + __func__, vcpu->arch.regs.nip); /* For debugging, encode the failing instruction and * report it to userspace. */ run->hw.hardware_exit_reason = ~0ULL << 32; @@ -1169,7 +1172,7 @@ int kvmppc_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu, case BOOKE_INTERRUPT_SPE_FP_DATA: case BOOKE_INTERRUPT_SPE_FP_ROUND: printk(KERN_CRIT "%s: unexpected SPE interrupt %u at %08lx\n", - __func__, exit_nr, vcpu->arch.pc); + __func__, exit_nr, vcpu->arch.regs.nip); run->hw.hardware_exit_reason = exit_nr; r = RESUME_HOST; break; @@ -1299,7 +1302,7 @@ int kvmppc_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu, } case BOOKE_INTERRUPT_ITLB_MISS: { - unsigned long eaddr = vcpu->arch.pc; + unsigned long eaddr = vcpu->arch.regs.nip; gpa_t gpaddr; gfn_t gfn; int gtlb_index; @@ -1391,7 +1394,7 @@ int kvm_arch_vcpu_setup(struct kvm_vcpu *vcpu) int i; int r; - vcpu->arch.pc = 0; + vcpu->arch.regs.nip = 0; vcpu->arch.shared->pir = vcpu->vcpu_id; kvmppc_set_gpr(vcpu, 1, (16<<20) - 8); /* -8 for the callee-save LR slot */ kvmppc_set_msr(vcpu, 0); @@ -1440,10 +1443,10 @@ int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) vcpu_load(vcpu); - regs->pc = vcpu->arch.pc; + regs->pc = vcpu->arch.regs.nip; regs->cr = kvmppc_get_cr(vcpu); - regs->ctr = vcpu->arch.ctr; - regs->lr = vcpu->arch.lr; + regs->ctr = vcpu->arch.regs.ctr; + regs->lr = vcpu->arch.regs.link; regs->xer = kvmppc_get_xer(vcpu); regs->msr = vcpu->arch.shared->msr; regs->srr0 = kvmppc_get_srr0(vcpu); @@ -1471,10 +1474,10 @@ int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) vcpu_load(vcpu); - vcpu->arch.pc = regs->pc; + vcpu->arch.regs.nip = regs->pc; kvmppc_set_cr(vcpu, regs->cr); - vcpu->arch.ctr = regs->ctr; - vcpu->arch.lr = regs->lr; + vcpu->arch.regs.ctr = regs->ctr; + vcpu->arch.regs.link = regs->lr; kvmppc_set_xer(vcpu, regs->xer); kvmppc_set_msr(vcpu, regs->msr); kvmppc_set_srr0(vcpu, regs->srr0); diff --git a/arch/powerpc/kvm/booke_emulate.c b/arch/powerpc/kvm/booke_emulate.c index a82f64502de1..d23e582f0fee 100644 --- a/arch/powerpc/kvm/booke_emulate.c +++ b/arch/powerpc/kvm/booke_emulate.c @@ -34,19 +34,19 @@ static void kvmppc_emul_rfi(struct kvm_vcpu *vcpu) { - vcpu->arch.pc = vcpu->arch.shared->srr0; + vcpu->arch.regs.nip = vcpu->arch.shared->srr0; kvmppc_set_msr(vcpu, vcpu->arch.shared->srr1); } static void kvmppc_emul_rfdi(struct kvm_vcpu *vcpu) { - vcpu->arch.pc = vcpu->arch.dsrr0; + vcpu->arch.regs.nip = vcpu->arch.dsrr0; kvmppc_set_msr(vcpu, vcpu->arch.dsrr1); } static void kvmppc_emul_rfci(struct kvm_vcpu *vcpu) { - vcpu->arch.pc = vcpu->arch.csrr0; + vcpu->arch.regs.nip = vcpu->arch.csrr0; kvmppc_set_msr(vcpu, vcpu->arch.csrr1); } diff --git a/arch/powerpc/kvm/e500_emulate.c b/arch/powerpc/kvm/e500_emulate.c index 990db69a1d0b..3f8189eb56ed 100644 --- a/arch/powerpc/kvm/e500_emulate.c +++ b/arch/powerpc/kvm/e500_emulate.c @@ -53,7 +53,7 @@ static int dbell2prio(ulong param) static int kvmppc_e500_emul_msgclr(struct kvm_vcpu *vcpu, int rb) { - ulong param = vcpu->arch.gpr[rb]; + ulong param = vcpu->arch.regs.gpr[rb]; int prio = dbell2prio(param); if (prio < 0) @@ -65,7 +65,7 @@ static int kvmppc_e500_emul_msgclr(struct kvm_vcpu *vcpu, int rb) static int kvmppc_e500_emul_msgsnd(struct kvm_vcpu *vcpu, int rb) { - ulong param = vcpu->arch.gpr[rb]; + ulong param = vcpu->arch.regs.gpr[rb]; int prio = dbell2prio(rb); int pir = param & PPC_DBELL_PIR_MASK; int i; @@ -94,7 +94,7 @@ static int kvmppc_e500_emul_ehpriv(struct kvm_run *run, struct kvm_vcpu *vcpu, switch (get_oc(inst)) { case EHPRIV_OC_DEBUG: run->exit_reason = KVM_EXIT_DEBUG; - run->debug.arch.address = vcpu->arch.pc; + run->debug.arch.address = vcpu->arch.regs.nip; run->debug.arch.status = 0; kvmppc_account_exit(vcpu, DEBUG_EXITS); emulated = EMULATE_EXIT_USER; diff --git a/arch/powerpc/kvm/e500_mmu.c b/arch/powerpc/kvm/e500_mmu.c index ddbf8f0284c0..24296f4cadc6 100644 --- a/arch/powerpc/kvm/e500_mmu.c +++ b/arch/powerpc/kvm/e500_mmu.c @@ -513,7 +513,7 @@ void kvmppc_mmu_itlb_miss(struct kvm_vcpu *vcpu) { unsigned int as = !!(vcpu->arch.shared->msr & MSR_IS); - kvmppc_e500_deliver_tlb_miss(vcpu, vcpu->arch.pc, as); + kvmppc_e500_deliver_tlb_miss(vcpu, vcpu->arch.regs.nip, as); } void kvmppc_mmu_dtlb_miss(struct kvm_vcpu *vcpu) diff --git a/arch/powerpc/kvm/e500_mmu_host.c b/arch/powerpc/kvm/e500_mmu_host.c index c878b4ffb86f..8f2985e46f6f 100644 --- a/arch/powerpc/kvm/e500_mmu_host.c +++ b/arch/powerpc/kvm/e500_mmu_host.c @@ -625,8 +625,8 @@ void kvmppc_mmu_map(struct kvm_vcpu *vcpu, u64 eaddr, gpa_t gpaddr, } #ifdef CONFIG_KVM_BOOKE_HV -int kvmppc_load_last_inst(struct kvm_vcpu *vcpu, enum instruction_type type, - u32 *instr) +int kvmppc_load_last_inst(struct kvm_vcpu *vcpu, + enum instruction_fetch_type type, u32 *instr) { gva_t geaddr; hpa_t addr; @@ -715,8 +715,8 @@ int kvmppc_load_last_inst(struct kvm_vcpu *vcpu, enum instruction_type type, return EMULATE_DONE; } #else -int kvmppc_load_last_inst(struct kvm_vcpu *vcpu, enum instruction_type type, - u32 *instr) +int kvmppc_load_last_inst(struct kvm_vcpu *vcpu, + enum instruction_fetch_type type, u32 *instr) { return EMULATE_AGAIN; } diff --git a/arch/powerpc/kvm/emulate_loadstore.c b/arch/powerpc/kvm/emulate_loadstore.c index a382e15135e6..afde788be141 100644 --- a/arch/powerpc/kvm/emulate_loadstore.c +++ b/arch/powerpc/kvm/emulate_loadstore.c @@ -31,6 +31,7 @@ #include #include #include +#include #include "timing.h" #include "trace.h" @@ -84,8 +85,9 @@ int kvmppc_emulate_loadstore(struct kvm_vcpu *vcpu) struct kvm_run *run = vcpu->run; u32 inst; int ra, rs, rt; - enum emulation_result emulated; + enum emulation_result emulated = EMULATE_FAIL; int advance = 1; + struct instruction_op op; /* this default type might be overwritten by subcategories */ kvmppc_set_exit_type(vcpu, EMULATED_INST_EXITS); @@ -107,580 +109,276 @@ int kvmppc_emulate_loadstore(struct kvm_vcpu *vcpu) vcpu->arch.mmio_vsx_tx_sx_enabled = get_tx_or_sx(inst); vcpu->arch.mmio_vsx_copy_nums = 0; vcpu->arch.mmio_vsx_offset = 0; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_NONE; + vcpu->arch.mmio_copy_type = KVMPPC_VSX_COPY_NONE; vcpu->arch.mmio_sp64_extend = 0; vcpu->arch.mmio_sign_extend = 0; vcpu->arch.mmio_vmx_copy_nums = 0; + vcpu->arch.mmio_vmx_offset = 0; + vcpu->arch.mmio_host_swabbed = 0; - switch (get_op(inst)) { - case 31: - switch (get_xop(inst)) { - case OP_31_XOP_LWZX: - emulated = kvmppc_handle_load(run, vcpu, rt, 4, 1); - break; - - case OP_31_XOP_LWZUX: - emulated = kvmppc_handle_load(run, vcpu, rt, 4, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; + emulated = EMULATE_FAIL; + vcpu->arch.regs.msr = vcpu->arch.shared->msr; + vcpu->arch.regs.ccr = vcpu->arch.cr; + if (analyse_instr(&op, &vcpu->arch.regs, inst) == 0) { + int type = op.type & INSTR_TYPE_MASK; + int size = GETSIZE(op.type); - case OP_31_XOP_LBZX: - emulated = kvmppc_handle_load(run, vcpu, rt, 1, 1); - break; - - case OP_31_XOP_LBZUX: - emulated = kvmppc_handle_load(run, vcpu, rt, 1, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; + switch (type) { + case LOAD: { + int instr_byte_swap = op.type & BYTEREV; - case OP_31_XOP_STDX: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 8, 1); - break; + if (op.type & SIGNEXT) + emulated = kvmppc_handle_loads(run, vcpu, + op.reg, size, !instr_byte_swap); + else + emulated = kvmppc_handle_load(run, vcpu, + op.reg, size, !instr_byte_swap); - case OP_31_XOP_STDUX: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 8, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; + if ((op.type & UPDATE) && (emulated != EMULATE_FAIL)) + kvmppc_set_gpr(vcpu, op.update_reg, op.ea); - case OP_31_XOP_STWX: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 4, 1); - break; - - case OP_31_XOP_STWUX: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 4, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_31_XOP_STBX: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 1, 1); - break; - - case OP_31_XOP_STBUX: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 1, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_31_XOP_LHAX: - emulated = kvmppc_handle_loads(run, vcpu, rt, 2, 1); - break; - - case OP_31_XOP_LHAUX: - emulated = kvmppc_handle_loads(run, vcpu, rt, 2, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_31_XOP_LHZX: - emulated = kvmppc_handle_load(run, vcpu, rt, 2, 1); - break; - - case OP_31_XOP_LHZUX: - emulated = kvmppc_handle_load(run, vcpu, rt, 2, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_31_XOP_STHX: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 2, 1); - break; - - case OP_31_XOP_STHUX: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 2, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_31_XOP_DCBST: - case OP_31_XOP_DCBF: - case OP_31_XOP_DCBI: - /* Do nothing. The guest is performing dcbi because - * hardware DMA is not snooped by the dcache, but - * emulated DMA either goes through the dcache as - * normal writes, or the host kernel has handled dcache - * coherence. */ - break; - - case OP_31_XOP_LWBRX: - emulated = kvmppc_handle_load(run, vcpu, rt, 4, 0); - break; - - case OP_31_XOP_STWBRX: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 4, 0); break; - - case OP_31_XOP_LHBRX: - emulated = kvmppc_handle_load(run, vcpu, rt, 2, 0); - break; - - case OP_31_XOP_STHBRX: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 2, 0); - break; - - case OP_31_XOP_LDBRX: - emulated = kvmppc_handle_load(run, vcpu, rt, 8, 0); - break; - - case OP_31_XOP_STDBRX: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 8, 0); - break; - - case OP_31_XOP_LDX: - emulated = kvmppc_handle_load(run, vcpu, rt, 8, 1); - break; - - case OP_31_XOP_LDUX: - emulated = kvmppc_handle_load(run, vcpu, rt, 8, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_31_XOP_LWAX: - emulated = kvmppc_handle_loads(run, vcpu, rt, 4, 1); - break; - - case OP_31_XOP_LWAUX: - emulated = kvmppc_handle_loads(run, vcpu, rt, 4, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - + } #ifdef CONFIG_PPC_FPU - case OP_31_XOP_LFSX: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_sp64_extend = 1; - emulated = kvmppc_handle_load(run, vcpu, - KVM_MMIO_REG_FPR|rt, 4, 1); - break; - - case OP_31_XOP_LFSUX: + case LOAD_FP: if (kvmppc_check_fp_disabled(vcpu)) return EMULATE_DONE; - vcpu->arch.mmio_sp64_extend = 1; - emulated = kvmppc_handle_load(run, vcpu, - KVM_MMIO_REG_FPR|rt, 4, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - case OP_31_XOP_LFDX: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - emulated = kvmppc_handle_load(run, vcpu, - KVM_MMIO_REG_FPR|rt, 8, 1); - break; + if (op.type & FPCONV) + vcpu->arch.mmio_sp64_extend = 1; - case OP_31_XOP_LFDUX: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - emulated = kvmppc_handle_load(run, vcpu, - KVM_MMIO_REG_FPR|rt, 8, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_31_XOP_LFIWAX: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - emulated = kvmppc_handle_loads(run, vcpu, - KVM_MMIO_REG_FPR|rt, 4, 1); - break; + if (op.type & SIGNEXT) + emulated = kvmppc_handle_loads(run, vcpu, + KVM_MMIO_REG_FPR|op.reg, size, 1); + else + emulated = kvmppc_handle_load(run, vcpu, + KVM_MMIO_REG_FPR|op.reg, size, 1); - case OP_31_XOP_LFIWZX: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - emulated = kvmppc_handle_load(run, vcpu, - KVM_MMIO_REG_FPR|rt, 4, 1); - break; + if ((op.type & UPDATE) && (emulated != EMULATE_FAIL)) + kvmppc_set_gpr(vcpu, op.update_reg, op.ea); - case OP_31_XOP_STFSX: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_sp64_extend = 1; - emulated = kvmppc_handle_store(run, vcpu, - VCPU_FPR(vcpu, rs), 4, 1); break; - - case OP_31_XOP_STFSUX: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_sp64_extend = 1; - emulated = kvmppc_handle_store(run, vcpu, - VCPU_FPR(vcpu, rs), 4, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_31_XOP_STFDX: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - emulated = kvmppc_handle_store(run, vcpu, - VCPU_FPR(vcpu, rs), 8, 1); - break; - - case OP_31_XOP_STFDUX: - if (kvmppc_check_fp_disabled(vcpu)) +#endif +#ifdef CONFIG_ALTIVEC + case LOAD_VMX: + if (kvmppc_check_altivec_disabled(vcpu)) return EMULATE_DONE; - emulated = kvmppc_handle_store(run, vcpu, - VCPU_FPR(vcpu, rs), 8, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - case OP_31_XOP_STFIWX: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - emulated = kvmppc_handle_store(run, vcpu, - VCPU_FPR(vcpu, rs), 4, 1); + /* Hardware enforces alignment of VMX accesses */ + vcpu->arch.vaddr_accessed &= ~((unsigned long)size - 1); + vcpu->arch.paddr_accessed &= ~((unsigned long)size - 1); + + if (size == 16) { /* lvx */ + vcpu->arch.mmio_copy_type = + KVMPPC_VMX_COPY_DWORD; + } else if (size == 4) { /* lvewx */ + vcpu->arch.mmio_copy_type = + KVMPPC_VMX_COPY_WORD; + } else if (size == 2) { /* lvehx */ + vcpu->arch.mmio_copy_type = + KVMPPC_VMX_COPY_HWORD; + } else if (size == 1) { /* lvebx */ + vcpu->arch.mmio_copy_type = + KVMPPC_VMX_COPY_BYTE; + } else + break; + + vcpu->arch.mmio_vmx_offset = + (vcpu->arch.vaddr_accessed & 0xf)/size; + + if (size == 16) { + vcpu->arch.mmio_vmx_copy_nums = 2; + emulated = kvmppc_handle_vmx_load(run, + vcpu, KVM_MMIO_REG_VMX|op.reg, + 8, 1); + } else { + vcpu->arch.mmio_vmx_copy_nums = 1; + emulated = kvmppc_handle_vmx_load(run, vcpu, + KVM_MMIO_REG_VMX|op.reg, + size, 1); + } break; #endif - #ifdef CONFIG_VSX - case OP_31_XOP_LXSDX: - if (kvmppc_check_vsx_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_vsx_copy_nums = 1; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_DWORD; - emulated = kvmppc_handle_vsx_load(run, vcpu, - KVM_MMIO_REG_VSX|rt, 8, 1, 0); - break; - - case OP_31_XOP_LXSSPX: - if (kvmppc_check_vsx_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_vsx_copy_nums = 1; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_DWORD; - vcpu->arch.mmio_sp64_extend = 1; - emulated = kvmppc_handle_vsx_load(run, vcpu, - KVM_MMIO_REG_VSX|rt, 4, 1, 0); - break; + case LOAD_VSX: { + int io_size_each; + + if (op.vsx_flags & VSX_CHECK_VEC) { + if (kvmppc_check_altivec_disabled(vcpu)) + return EMULATE_DONE; + } else { + if (kvmppc_check_vsx_disabled(vcpu)) + return EMULATE_DONE; + } + + if (op.vsx_flags & VSX_FPCONV) + vcpu->arch.mmio_sp64_extend = 1; + + if (op.element_size == 8) { + if (op.vsx_flags & VSX_SPLAT) + vcpu->arch.mmio_copy_type = + KVMPPC_VSX_COPY_DWORD_LOAD_DUMP; + else + vcpu->arch.mmio_copy_type = + KVMPPC_VSX_COPY_DWORD; + } else if (op.element_size == 4) { + if (op.vsx_flags & VSX_SPLAT) + vcpu->arch.mmio_copy_type = + KVMPPC_VSX_COPY_WORD_LOAD_DUMP; + else + vcpu->arch.mmio_copy_type = + KVMPPC_VSX_COPY_WORD; + } else + break; + + if (size < op.element_size) { + /* precision convert case: lxsspx, etc */ + vcpu->arch.mmio_vsx_copy_nums = 1; + io_size_each = size; + } else { /* lxvw4x, lxvd2x, etc */ + vcpu->arch.mmio_vsx_copy_nums = + size/op.element_size; + io_size_each = op.element_size; + } - case OP_31_XOP_LXSIWAX: - if (kvmppc_check_vsx_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_vsx_copy_nums = 1; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_DWORD; emulated = kvmppc_handle_vsx_load(run, vcpu, - KVM_MMIO_REG_VSX|rt, 4, 1, 1); - break; - - case OP_31_XOP_LXSIWZX: - if (kvmppc_check_vsx_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_vsx_copy_nums = 1; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_DWORD; - emulated = kvmppc_handle_vsx_load(run, vcpu, - KVM_MMIO_REG_VSX|rt, 4, 1, 0); + KVM_MMIO_REG_VSX | (op.reg & 0x1f), + io_size_each, 1, op.type & SIGNEXT); break; + } +#endif + case STORE: + /* if need byte reverse, op.val has been reversed by + * analyse_instr(). + */ + emulated = kvmppc_handle_store(run, vcpu, op.val, + size, 1); - case OP_31_XOP_LXVD2X: - /* - * In this case, the official load/store process is like this: - * Step1, exit from vm by page fault isr, then kvm save vsr. - * Please see guest_exit_cont->store_fp_state->SAVE_32VSRS - * as reference. - * - * Step2, copy data between memory and VCPU - * Notice: for LXVD2X/STXVD2X/LXVW4X/STXVW4X, we use - * 2copies*8bytes or 4copies*4bytes - * to simulate one copy of 16bytes. - * Also there is an endian issue here, we should notice the - * layout of memory. - * Please see MARCO of LXVD2X_ROT/STXVD2X_ROT as more reference. - * If host is little-endian, kvm will call XXSWAPD for - * LXVD2X_ROT/STXVD2X_ROT. - * So, if host is little-endian, - * the postion of memeory should be swapped. - * - * Step3, return to guest, kvm reset register. - * Please see kvmppc_hv_entry->load_fp_state->REST_32VSRS - * as reference. - */ - if (kvmppc_check_vsx_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_vsx_copy_nums = 2; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_DWORD; - emulated = kvmppc_handle_vsx_load(run, vcpu, - KVM_MMIO_REG_VSX|rt, 8, 1, 0); - break; + if ((op.type & UPDATE) && (emulated != EMULATE_FAIL)) + kvmppc_set_gpr(vcpu, op.update_reg, op.ea); - case OP_31_XOP_LXVW4X: - if (kvmppc_check_vsx_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_vsx_copy_nums = 4; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_WORD; - emulated = kvmppc_handle_vsx_load(run, vcpu, - KVM_MMIO_REG_VSX|rt, 4, 1, 0); break; - - case OP_31_XOP_LXVDSX: - if (kvmppc_check_vsx_disabled(vcpu)) +#ifdef CONFIG_PPC_FPU + case STORE_FP: + if (kvmppc_check_fp_disabled(vcpu)) return EMULATE_DONE; - vcpu->arch.mmio_vsx_copy_nums = 1; - vcpu->arch.mmio_vsx_copy_type = - KVMPPC_VSX_COPY_DWORD_LOAD_DUMP; - emulated = kvmppc_handle_vsx_load(run, vcpu, - KVM_MMIO_REG_VSX|rt, 8, 1, 0); - break; - case OP_31_XOP_STXSDX: - if (kvmppc_check_vsx_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_vsx_copy_nums = 1; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_DWORD; - emulated = kvmppc_handle_vsx_store(run, vcpu, - rs, 8, 1); - break; + /* The FP registers need to be flushed so that + * kvmppc_handle_store() can read actual FP vals + * from vcpu->arch. + */ + if (vcpu->kvm->arch.kvm_ops->giveup_ext) + vcpu->kvm->arch.kvm_ops->giveup_ext(vcpu, + MSR_FP); - case OP_31_XOP_STXSSPX: - if (kvmppc_check_vsx_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_vsx_copy_nums = 1; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_DWORD; - vcpu->arch.mmio_sp64_extend = 1; - emulated = kvmppc_handle_vsx_store(run, vcpu, - rs, 4, 1); - break; + if (op.type & FPCONV) + vcpu->arch.mmio_sp64_extend = 1; - case OP_31_XOP_STXSIWX: - if (kvmppc_check_vsx_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_vsx_offset = 1; - vcpu->arch.mmio_vsx_copy_nums = 1; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_WORD; - emulated = kvmppc_handle_vsx_store(run, vcpu, - rs, 4, 1); - break; + emulated = kvmppc_handle_store(run, vcpu, + VCPU_FPR(vcpu, op.reg), size, 1); - case OP_31_XOP_STXVD2X: - if (kvmppc_check_vsx_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_vsx_copy_nums = 2; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_DWORD; - emulated = kvmppc_handle_vsx_store(run, vcpu, - rs, 8, 1); - break; + if ((op.type & UPDATE) && (emulated != EMULATE_FAIL)) + kvmppc_set_gpr(vcpu, op.update_reg, op.ea); - case OP_31_XOP_STXVW4X: - if (kvmppc_check_vsx_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_vsx_copy_nums = 4; - vcpu->arch.mmio_vsx_copy_type = KVMPPC_VSX_COPY_WORD; - emulated = kvmppc_handle_vsx_store(run, vcpu, - rs, 4, 1); break; -#endif /* CONFIG_VSX */ - +#endif #ifdef CONFIG_ALTIVEC - case OP_31_XOP_LVX: + case STORE_VMX: if (kvmppc_check_altivec_disabled(vcpu)) return EMULATE_DONE; - vcpu->arch.vaddr_accessed &= ~0xFULL; - vcpu->arch.paddr_accessed &= ~0xFULL; - vcpu->arch.mmio_vmx_copy_nums = 2; - emulated = kvmppc_handle_load128_by2x64(run, vcpu, - KVM_MMIO_REG_VMX|rt, 1); - break; - case OP_31_XOP_STVX: - if (kvmppc_check_altivec_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.vaddr_accessed &= ~0xFULL; - vcpu->arch.paddr_accessed &= ~0xFULL; - vcpu->arch.mmio_vmx_copy_nums = 2; - emulated = kvmppc_handle_store128_by2x64(run, vcpu, - rs, 1); - break; -#endif /* CONFIG_ALTIVEC */ + /* Hardware enforces alignment of VMX accesses. */ + vcpu->arch.vaddr_accessed &= ~((unsigned long)size - 1); + vcpu->arch.paddr_accessed &= ~((unsigned long)size - 1); + + if (vcpu->kvm->arch.kvm_ops->giveup_ext) + vcpu->kvm->arch.kvm_ops->giveup_ext(vcpu, + MSR_VEC); + if (size == 16) { /* stvx */ + vcpu->arch.mmio_copy_type = + KVMPPC_VMX_COPY_DWORD; + } else if (size == 4) { /* stvewx */ + vcpu->arch.mmio_copy_type = + KVMPPC_VMX_COPY_WORD; + } else if (size == 2) { /* stvehx */ + vcpu->arch.mmio_copy_type = + KVMPPC_VMX_COPY_HWORD; + } else if (size == 1) { /* stvebx */ + vcpu->arch.mmio_copy_type = + KVMPPC_VMX_COPY_BYTE; + } else + break; + + vcpu->arch.mmio_vmx_offset = + (vcpu->arch.vaddr_accessed & 0xf)/size; + + if (size == 16) { + vcpu->arch.mmio_vmx_copy_nums = 2; + emulated = kvmppc_handle_vmx_store(run, + vcpu, op.reg, 8, 1); + } else { + vcpu->arch.mmio_vmx_copy_nums = 1; + emulated = kvmppc_handle_vmx_store(run, + vcpu, op.reg, size, 1); + } - default: - emulated = EMULATE_FAIL; break; - } - break; - - case OP_LWZ: - emulated = kvmppc_handle_load(run, vcpu, rt, 4, 1); - break; - -#ifdef CONFIG_PPC_FPU - case OP_STFS: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_sp64_extend = 1; - emulated = kvmppc_handle_store(run, vcpu, - VCPU_FPR(vcpu, rs), - 4, 1); - break; - - case OP_STFSU: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_sp64_extend = 1; - emulated = kvmppc_handle_store(run, vcpu, - VCPU_FPR(vcpu, rs), - 4, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_STFD: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - emulated = kvmppc_handle_store(run, vcpu, - VCPU_FPR(vcpu, rs), - 8, 1); - break; - - case OP_STFDU: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - emulated = kvmppc_handle_store(run, vcpu, - VCPU_FPR(vcpu, rs), - 8, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; #endif +#ifdef CONFIG_VSX + case STORE_VSX: { + int io_size_each; + + if (op.vsx_flags & VSX_CHECK_VEC) { + if (kvmppc_check_altivec_disabled(vcpu)) + return EMULATE_DONE; + } else { + if (kvmppc_check_vsx_disabled(vcpu)) + return EMULATE_DONE; + } + + if (vcpu->kvm->arch.kvm_ops->giveup_ext) + vcpu->kvm->arch.kvm_ops->giveup_ext(vcpu, + MSR_VSX); + + if (op.vsx_flags & VSX_FPCONV) + vcpu->arch.mmio_sp64_extend = 1; + + if (op.element_size == 8) + vcpu->arch.mmio_copy_type = + KVMPPC_VSX_COPY_DWORD; + else if (op.element_size == 4) + vcpu->arch.mmio_copy_type = + KVMPPC_VSX_COPY_WORD; + else + break; + + if (size < op.element_size) { + /* precise conversion case, like stxsspx */ + vcpu->arch.mmio_vsx_copy_nums = 1; + io_size_each = size; + } else { /* stxvw4x, stxvd2x, etc */ + vcpu->arch.mmio_vsx_copy_nums = + size/op.element_size; + io_size_each = op.element_size; + } - case OP_LD: - rt = get_rt(inst); - switch (inst & 3) { - case 0: /* ld */ - emulated = kvmppc_handle_load(run, vcpu, rt, 8, 1); - break; - case 1: /* ldu */ - emulated = kvmppc_handle_load(run, vcpu, rt, 8, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - case 2: /* lwa */ - emulated = kvmppc_handle_loads(run, vcpu, rt, 4, 1); + emulated = kvmppc_handle_vsx_store(run, vcpu, + op.reg & 0x1f, io_size_each, 1); break; - default: - emulated = EMULATE_FAIL; } - break; - - case OP_LWZU: - emulated = kvmppc_handle_load(run, vcpu, rt, 4, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_LBZ: - emulated = kvmppc_handle_load(run, vcpu, rt, 1, 1); - break; - - case OP_LBZU: - emulated = kvmppc_handle_load(run, vcpu, rt, 1, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_STW: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), - 4, 1); - break; - - case OP_STD: - rs = get_rs(inst); - switch (inst & 3) { - case 0: /* std */ - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 8, 1); - break; - case 1: /* stdu */ - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 8, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); +#endif + case CACHEOP: + /* Do nothing. The guest is performing dcbi because + * hardware DMA is not snooped by the dcache, but + * emulated DMA either goes through the dcache as + * normal writes, or the host kernel has handled dcache + * coherence. + */ + emulated = EMULATE_DONE; break; default: - emulated = EMULATE_FAIL; + break; } - break; - - case OP_STWU: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 4, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_STB: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 1, 1); - break; - - case OP_STBU: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 1, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_LHZ: - emulated = kvmppc_handle_load(run, vcpu, rt, 2, 1); - break; - - case OP_LHZU: - emulated = kvmppc_handle_load(run, vcpu, rt, 2, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_LHA: - emulated = kvmppc_handle_loads(run, vcpu, rt, 2, 1); - break; - - case OP_LHAU: - emulated = kvmppc_handle_loads(run, vcpu, rt, 2, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_STH: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 2, 1); - break; - - case OP_STHU: - emulated = kvmppc_handle_store(run, vcpu, - kvmppc_get_gpr(vcpu, rs), 2, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - -#ifdef CONFIG_PPC_FPU - case OP_LFS: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_sp64_extend = 1; - emulated = kvmppc_handle_load(run, vcpu, - KVM_MMIO_REG_FPR|rt, 4, 1); - break; - - case OP_LFSU: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - vcpu->arch.mmio_sp64_extend = 1; - emulated = kvmppc_handle_load(run, vcpu, - KVM_MMIO_REG_FPR|rt, 4, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; - - case OP_LFD: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - emulated = kvmppc_handle_load(run, vcpu, - KVM_MMIO_REG_FPR|rt, 8, 1); - break; - - case OP_LFDU: - if (kvmppc_check_fp_disabled(vcpu)) - return EMULATE_DONE; - emulated = kvmppc_handle_load(run, vcpu, - KVM_MMIO_REG_FPR|rt, 8, 1); - kvmppc_set_gpr(vcpu, ra, vcpu->arch.vaddr_accessed); - break; -#endif - - default: - emulated = EMULATE_FAIL; - break; } if (emulated == EMULATE_FAIL) { diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c index 4e387647b5af..0e8c20c5eaac 100644 --- a/arch/powerpc/kvm/powerpc.c +++ b/arch/powerpc/kvm/powerpc.c @@ -648,9 +648,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) #endif #ifdef CONFIG_PPC_TRANSACTIONAL_MEM case KVM_CAP_PPC_HTM: - r = hv_enabled && - (!!(cur_cpu_spec->cpu_user_features2 & PPC_FEATURE2_HTM) || - cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST)); + r = !!(cur_cpu_spec->cpu_user_features2 & PPC_FEATURE2_HTM) || + (hv_enabled && cpu_has_feature(CPU_FTR_P9_TM_HV_ASSIST)); break; #endif default: @@ -907,6 +906,26 @@ static inline void kvmppc_set_vsr_dword_dump(struct kvm_vcpu *vcpu, } } +static inline void kvmppc_set_vsr_word_dump(struct kvm_vcpu *vcpu, + u32 gpr) +{ + union kvmppc_one_reg val; + int index = vcpu->arch.io_gpr & KVM_MMIO_REG_MASK; + + if (vcpu->arch.mmio_vsx_tx_sx_enabled) { + val.vsx32val[0] = gpr; + val.vsx32val[1] = gpr; + val.vsx32val[2] = gpr; + val.vsx32val[3] = gpr; + VCPU_VSX_VR(vcpu, index) = val.vval; + } else { + val.vsx32val[0] = gpr; + val.vsx32val[1] = gpr; + VCPU_VSX_FPR(vcpu, index, 0) = val.vsxval[0]; + VCPU_VSX_FPR(vcpu, index, 1) = val.vsxval[0]; + } +} + static inline void kvmppc_set_vsr_word(struct kvm_vcpu *vcpu, u32 gpr32) { @@ -933,30 +952,110 @@ static inline void kvmppc_set_vsr_word(struct kvm_vcpu *vcpu, #endif /* CONFIG_VSX */ #ifdef CONFIG_ALTIVEC +static inline int kvmppc_get_vmx_offset_generic(struct kvm_vcpu *vcpu, + int index, int element_size) +{ + int offset; + int elts = sizeof(vector128)/element_size; + + if ((index < 0) || (index >= elts)) + return -1; + + if (kvmppc_need_byteswap(vcpu)) + offset = elts - index - 1; + else + offset = index; + + return offset; +} + +static inline int kvmppc_get_vmx_dword_offset(struct kvm_vcpu *vcpu, + int index) +{ + return kvmppc_get_vmx_offset_generic(vcpu, index, 8); +} + +static inline int kvmppc_get_vmx_word_offset(struct kvm_vcpu *vcpu, + int index) +{ + return kvmppc_get_vmx_offset_generic(vcpu, index, 4); +} + +static inline int kvmppc_get_vmx_hword_offset(struct kvm_vcpu *vcpu, + int index) +{ + return kvmppc_get_vmx_offset_generic(vcpu, index, 2); +} + +static inline int kvmppc_get_vmx_byte_offset(struct kvm_vcpu *vcpu, + int index) +{ + return kvmppc_get_vmx_offset_generic(vcpu, index, 1); +} + + static inline void kvmppc_set_vmx_dword(struct kvm_vcpu *vcpu, - u64 gpr) + u64 gpr) { + union kvmppc_one_reg val; + int offset = kvmppc_get_vmx_dword_offset(vcpu, + vcpu->arch.mmio_vmx_offset); int index = vcpu->arch.io_gpr & KVM_MMIO_REG_MASK; - u32 hi, lo; - u32 di; -#ifdef __BIG_ENDIAN - hi = gpr >> 32; - lo = gpr & 0xffffffff; -#else - lo = gpr >> 32; - hi = gpr & 0xffffffff; -#endif + if (offset == -1) + return; + + val.vval = VCPU_VSX_VR(vcpu, index); + val.vsxval[offset] = gpr; + VCPU_VSX_VR(vcpu, index) = val.vval; +} + +static inline void kvmppc_set_vmx_word(struct kvm_vcpu *vcpu, + u32 gpr32) +{ + union kvmppc_one_reg val; + int offset = kvmppc_get_vmx_word_offset(vcpu, + vcpu->arch.mmio_vmx_offset); + int index = vcpu->arch.io_gpr & KVM_MMIO_REG_MASK; - di = 2 - vcpu->arch.mmio_vmx_copy_nums; /* doubleword index */ - if (di > 1) + if (offset == -1) return; - if (vcpu->arch.mmio_host_swabbed) - di = 1 - di; + val.vval = VCPU_VSX_VR(vcpu, index); + val.vsx32val[offset] = gpr32; + VCPU_VSX_VR(vcpu, index) = val.vval; +} + +static inline void kvmppc_set_vmx_hword(struct kvm_vcpu *vcpu, + u16 gpr16) +{ + union kvmppc_one_reg val; + int offset = kvmppc_get_vmx_hword_offset(vcpu, + vcpu->arch.mmio_vmx_offset); + int index = vcpu->arch.io_gpr & KVM_MMIO_REG_MASK; + + if (offset == -1) + return; + + val.vval = VCPU_VSX_VR(vcpu, index); + val.vsx16val[offset] = gpr16; + VCPU_VSX_VR(vcpu, index) = val.vval; +} + +static inline void kvmppc_set_vmx_byte(struct kvm_vcpu *vcpu, + u8 gpr8) +{ + union kvmppc_one_reg val; + int offset = kvmppc_get_vmx_byte_offset(vcpu, + vcpu->arch.mmio_vmx_offset); + int index = vcpu->arch.io_gpr & KVM_MMIO_REG_MASK; + + if (offset == -1) + return; - VCPU_VSX_VR(vcpu, index).u[di * 2] = hi; - VCPU_VSX_VR(vcpu, index).u[di * 2 + 1] = lo; + val.vval = VCPU_VSX_VR(vcpu, index); + val.vsx8val[offset] = gpr8; + VCPU_VSX_VR(vcpu, index) = val.vval; } #endif /* CONFIG_ALTIVEC */ @@ -1041,6 +1140,9 @@ static void kvmppc_complete_mmio_load(struct kvm_vcpu *vcpu, kvmppc_set_gpr(vcpu, vcpu->arch.io_gpr, gpr); break; case KVM_MMIO_REG_FPR: + if (vcpu->kvm->arch.kvm_ops->giveup_ext) + vcpu->kvm->arch.kvm_ops->giveup_ext(vcpu, MSR_FP); + VCPU_FPR(vcpu, vcpu->arch.io_gpr & KVM_MMIO_REG_MASK) = gpr; break; #ifdef CONFIG_PPC_BOOK3S @@ -1054,18 +1156,36 @@ static void kvmppc_complete_mmio_load(struct kvm_vcpu *vcpu, #endif #ifdef CONFIG_VSX case KVM_MMIO_REG_VSX: - if (vcpu->arch.mmio_vsx_copy_type == KVMPPC_VSX_COPY_DWORD) + if (vcpu->kvm->arch.kvm_ops->giveup_ext) + vcpu->kvm->arch.kvm_ops->giveup_ext(vcpu, MSR_VSX); + + if (vcpu->arch.mmio_copy_type == KVMPPC_VSX_COPY_DWORD) kvmppc_set_vsr_dword(vcpu, gpr); - else if (vcpu->arch.mmio_vsx_copy_type == KVMPPC_VSX_COPY_WORD) + else if (vcpu->arch.mmio_copy_type == KVMPPC_VSX_COPY_WORD) kvmppc_set_vsr_word(vcpu, gpr); - else if (vcpu->arch.mmio_vsx_copy_type == + else if (vcpu->arch.mmio_copy_type == KVMPPC_VSX_COPY_DWORD_LOAD_DUMP) kvmppc_set_vsr_dword_dump(vcpu, gpr); + else if (vcpu->arch.mmio_copy_type == + KVMPPC_VSX_COPY_WORD_LOAD_DUMP) + kvmppc_set_vsr_word_dump(vcpu, gpr); break; #endif #ifdef CONFIG_ALTIVEC case KVM_MMIO_REG_VMX: - kvmppc_set_vmx_dword(vcpu, gpr); + if (vcpu->kvm->arch.kvm_ops->giveup_ext) + vcpu->kvm->arch.kvm_ops->giveup_ext(vcpu, MSR_VEC); + + if (vcpu->arch.mmio_copy_type == KVMPPC_VMX_COPY_DWORD) + kvmppc_set_vmx_dword(vcpu, gpr); + else if (vcpu->arch.mmio_copy_type == KVMPPC_VMX_COPY_WORD) + kvmppc_set_vmx_word(vcpu, gpr); + else if (vcpu->arch.mmio_copy_type == + KVMPPC_VMX_COPY_HWORD) + kvmppc_set_vmx_hword(vcpu, gpr); + else if (vcpu->arch.mmio_copy_type == + KVMPPC_VMX_COPY_BYTE) + kvmppc_set_vmx_byte(vcpu, gpr); break; #endif default: @@ -1228,7 +1348,7 @@ static inline int kvmppc_get_vsr_data(struct kvm_vcpu *vcpu, int rs, u64 *val) u32 dword_offset, word_offset; union kvmppc_one_reg reg; int vsx_offset = 0; - int copy_type = vcpu->arch.mmio_vsx_copy_type; + int copy_type = vcpu->arch.mmio_copy_type; int result = 0; switch (copy_type) { @@ -1344,14 +1464,16 @@ static int kvmppc_emulate_mmio_vsx_loadstore(struct kvm_vcpu *vcpu, #endif /* CONFIG_VSX */ #ifdef CONFIG_ALTIVEC -/* handle quadword load access in two halves */ -int kvmppc_handle_load128_by2x64(struct kvm_run *run, struct kvm_vcpu *vcpu, - unsigned int rt, int is_default_endian) +int kvmppc_handle_vmx_load(struct kvm_run *run, struct kvm_vcpu *vcpu, + unsigned int rt, unsigned int bytes, int is_default_endian) { enum emulation_result emulated = EMULATE_DONE; + if (vcpu->arch.mmio_vsx_copy_nums > 2) + return EMULATE_FAIL; + while (vcpu->arch.mmio_vmx_copy_nums) { - emulated = __kvmppc_handle_load(run, vcpu, rt, 8, + emulated = __kvmppc_handle_load(run, vcpu, rt, bytes, is_default_endian, 0); if (emulated != EMULATE_DONE) @@ -1359,55 +1481,127 @@ int kvmppc_handle_load128_by2x64(struct kvm_run *run, struct kvm_vcpu *vcpu, vcpu->arch.paddr_accessed += run->mmio.len; vcpu->arch.mmio_vmx_copy_nums--; + vcpu->arch.mmio_vmx_offset++; } return emulated; } -static inline int kvmppc_get_vmx_data(struct kvm_vcpu *vcpu, int rs, u64 *val) +int kvmppc_get_vmx_dword(struct kvm_vcpu *vcpu, int index, u64 *val) { - vector128 vrs = VCPU_VSX_VR(vcpu, rs); - u32 di; - u64 w0, w1; + union kvmppc_one_reg reg; + int vmx_offset = 0; + int result = 0; - di = 2 - vcpu->arch.mmio_vmx_copy_nums; /* doubleword index */ - if (di > 1) + vmx_offset = + kvmppc_get_vmx_dword_offset(vcpu, vcpu->arch.mmio_vmx_offset); + + if (vmx_offset == -1) return -1; - if (vcpu->arch.mmio_host_swabbed) - di = 1 - di; + reg.vval = VCPU_VSX_VR(vcpu, index); + *val = reg.vsxval[vmx_offset]; - w0 = vrs.u[di * 2]; - w1 = vrs.u[di * 2 + 1]; + return result; +} -#ifdef __BIG_ENDIAN - *val = (w0 << 32) | w1; -#else - *val = (w1 << 32) | w0; -#endif - return 0; +int kvmppc_get_vmx_word(struct kvm_vcpu *vcpu, int index, u64 *val) +{ + union kvmppc_one_reg reg; + int vmx_offset = 0; + int result = 0; + + vmx_offset = + kvmppc_get_vmx_word_offset(vcpu, vcpu->arch.mmio_vmx_offset); + + if (vmx_offset == -1) + return -1; + + reg.vval = VCPU_VSX_VR(vcpu, index); + *val = reg.vsx32val[vmx_offset]; + + return result; +} + +int kvmppc_get_vmx_hword(struct kvm_vcpu *vcpu, int index, u64 *val) +{ + union kvmppc_one_reg reg; + int vmx_offset = 0; + int result = 0; + + vmx_offset = + kvmppc_get_vmx_hword_offset(vcpu, vcpu->arch.mmio_vmx_offset); + + if (vmx_offset == -1) + return -1; + + reg.vval = VCPU_VSX_VR(vcpu, index); + *val = reg.vsx16val[vmx_offset]; + + return result; } -/* handle quadword store in two halves */ -int kvmppc_handle_store128_by2x64(struct kvm_run *run, struct kvm_vcpu *vcpu, - unsigned int rs, int is_default_endian) +int kvmppc_get_vmx_byte(struct kvm_vcpu *vcpu, int index, u64 *val) +{ + union kvmppc_one_reg reg; + int vmx_offset = 0; + int result = 0; + + vmx_offset = + kvmppc_get_vmx_byte_offset(vcpu, vcpu->arch.mmio_vmx_offset); + + if (vmx_offset == -1) + return -1; + + reg.vval = VCPU_VSX_VR(vcpu, index); + *val = reg.vsx8val[vmx_offset]; + + return result; +} + +int kvmppc_handle_vmx_store(struct kvm_run *run, struct kvm_vcpu *vcpu, + unsigned int rs, unsigned int bytes, int is_default_endian) { u64 val = 0; + unsigned int index = rs & KVM_MMIO_REG_MASK; enum emulation_result emulated = EMULATE_DONE; + if (vcpu->arch.mmio_vsx_copy_nums > 2) + return EMULATE_FAIL; + vcpu->arch.io_gpr = rs; while (vcpu->arch.mmio_vmx_copy_nums) { - if (kvmppc_get_vmx_data(vcpu, rs, &val) == -1) + switch (vcpu->arch.mmio_copy_type) { + case KVMPPC_VMX_COPY_DWORD: + if (kvmppc_get_vmx_dword(vcpu, index, &val) == -1) + return EMULATE_FAIL; + + break; + case KVMPPC_VMX_COPY_WORD: + if (kvmppc_get_vmx_word(vcpu, index, &val) == -1) + return EMULATE_FAIL; + break; + case KVMPPC_VMX_COPY_HWORD: + if (kvmppc_get_vmx_hword(vcpu, index, &val) == -1) + return EMULATE_FAIL; + break; + case KVMPPC_VMX_COPY_BYTE: + if (kvmppc_get_vmx_byte(vcpu, index, &val) == -1) + return EMULATE_FAIL; + break; + default: return EMULATE_FAIL; + } - emulated = kvmppc_handle_store(run, vcpu, val, 8, + emulated = kvmppc_handle_store(run, vcpu, val, bytes, is_default_endian); if (emulated != EMULATE_DONE) break; vcpu->arch.paddr_accessed += run->mmio.len; vcpu->arch.mmio_vmx_copy_nums--; + vcpu->arch.mmio_vmx_offset++; } return emulated; @@ -1422,11 +1616,11 @@ static int kvmppc_emulate_mmio_vmx_loadstore(struct kvm_vcpu *vcpu, vcpu->arch.paddr_accessed += run->mmio.len; if (!vcpu->mmio_is_write) { - emulated = kvmppc_handle_load128_by2x64(run, vcpu, - vcpu->arch.io_gpr, 1); + emulated = kvmppc_handle_vmx_load(run, vcpu, + vcpu->arch.io_gpr, run->mmio.len, 1); } else { - emulated = kvmppc_handle_store128_by2x64(run, vcpu, - vcpu->arch.io_gpr, 1); + emulated = kvmppc_handle_vmx_store(run, vcpu, + vcpu->arch.io_gpr, run->mmio.len, 1); } switch (emulated) { @@ -1570,8 +1764,10 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) } #endif #ifdef CONFIG_ALTIVEC - if (vcpu->arch.mmio_vmx_copy_nums > 0) + if (vcpu->arch.mmio_vmx_copy_nums > 0) { vcpu->arch.mmio_vmx_copy_nums--; + vcpu->arch.mmio_vmx_offset++; + } if (vcpu->arch.mmio_vmx_copy_nums > 0) { r = kvmppc_emulate_mmio_vmx_loadstore(vcpu, run); @@ -1784,16 +1980,16 @@ long kvm_arch_vcpu_ioctl(struct file *filp, void __user *argp = (void __user *)arg; long r; - vcpu_load(vcpu); - switch (ioctl) { case KVM_ENABLE_CAP: { struct kvm_enable_cap cap; r = -EFAULT; + vcpu_load(vcpu); if (copy_from_user(&cap, argp, sizeof(cap))) goto out; r = kvm_vcpu_ioctl_enable_cap(vcpu, &cap); + vcpu_put(vcpu); break; } @@ -1815,9 +2011,11 @@ long kvm_arch_vcpu_ioctl(struct file *filp, case KVM_DIRTY_TLB: { struct kvm_dirty_tlb dirty; r = -EFAULT; + vcpu_load(vcpu); if (copy_from_user(&dirty, argp, sizeof(dirty))) goto out; r = kvm_vcpu_ioctl_dirty_tlb(vcpu, &dirty); + vcpu_put(vcpu); break; } #endif @@ -1826,11 +2024,10 @@ long kvm_arch_vcpu_ioctl(struct file *filp, } out: - vcpu_put(vcpu); return r; } -int kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf) +vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf) { return VM_FAULT_SIGBUS; } diff --git a/arch/powerpc/kvm/tm.S b/arch/powerpc/kvm/tm.S new file mode 100644 index 000000000000..90e330f21356 --- /dev/null +++ b/arch/powerpc/kvm/tm.S @@ -0,0 +1,384 @@ +/* + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License, version 2, as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * Derived from book3s_hv_rmhandlers.S, which is: + * + * Copyright 2011 Paul Mackerras, IBM Corp. + * + */ + +#include +#include +#include +#include +#include +#include + +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM +#define VCPU_GPRS_TM(reg) (((reg) * ULONG_SIZE) + VCPU_GPR_TM) + +/* + * Save transactional state and TM-related registers. + * Called with: + * - r3 pointing to the vcpu struct + * - r4 points to the MSR with current TS bits: + * (For HV KVM, it is VCPU_MSR ; For PR KVM, it is host MSR). + * This can modify all checkpointed registers, but + * restores r1, r2 before exit. + */ +_GLOBAL(__kvmppc_save_tm) + mflr r0 + std r0, PPC_LR_STKOFF(r1) + + /* Turn on TM. */ + mfmsr r8 + li r0, 1 + rldimi r8, r0, MSR_TM_LG, 63-MSR_TM_LG + ori r8, r8, MSR_FP + oris r8, r8, (MSR_VEC | MSR_VSX)@h + mtmsrd r8 + + rldicl. r4, r4, 64 - MSR_TS_S_LG, 62 + beq 1f /* TM not active in guest. */ + + std r1, HSTATE_SCRATCH2(r13) + std r3, HSTATE_SCRATCH1(r13) + +#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE +BEGIN_FTR_SECTION + /* Emulation of the treclaim instruction needs TEXASR before treclaim */ + mfspr r6, SPRN_TEXASR + std r6, VCPU_ORIG_TEXASR(r3) +END_FTR_SECTION_IFSET(CPU_FTR_P9_TM_HV_ASSIST) +#endif + + /* Clear the MSR RI since r1, r13 are all going to be foobar. */ + li r5, 0 + mtmsrd r5, 1 + + li r3, TM_CAUSE_KVM_RESCHED + + /* All GPRs are volatile at this point. */ + TRECLAIM(R3) + + /* Temporarily store r13 and r9 so we have some regs to play with */ + SET_SCRATCH0(r13) + GET_PACA(r13) + std r9, PACATMSCRATCH(r13) + ld r9, HSTATE_SCRATCH1(r13) + + /* Get a few more GPRs free. */ + std r29, VCPU_GPRS_TM(29)(r9) + std r30, VCPU_GPRS_TM(30)(r9) + std r31, VCPU_GPRS_TM(31)(r9) + + /* Save away PPR and DSCR soon so don't run with user values. */ + mfspr r31, SPRN_PPR + HMT_MEDIUM + mfspr r30, SPRN_DSCR +#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE + ld r29, HSTATE_DSCR(r13) + mtspr SPRN_DSCR, r29 +#endif + + /* Save all but r9, r13 & r29-r31 */ + reg = 0 + .rept 29 + .if (reg != 9) && (reg != 13) + std reg, VCPU_GPRS_TM(reg)(r9) + .endif + reg = reg + 1 + .endr + /* ... now save r13 */ + GET_SCRATCH0(r4) + std r4, VCPU_GPRS_TM(13)(r9) + /* ... and save r9 */ + ld r4, PACATMSCRATCH(r13) + std r4, VCPU_GPRS_TM(9)(r9) + + /* Reload stack pointer and TOC. */ + ld r1, HSTATE_SCRATCH2(r13) + ld r2, PACATOC(r13) + + /* Set MSR RI now we have r1 and r13 back. */ + li r5, MSR_RI + mtmsrd r5, 1 + + /* Save away checkpinted SPRs. */ + std r31, VCPU_PPR_TM(r9) + std r30, VCPU_DSCR_TM(r9) + mflr r5 + mfcr r6 + mfctr r7 + mfspr r8, SPRN_AMR + mfspr r10, SPRN_TAR + mfxer r11 + std r5, VCPU_LR_TM(r9) + stw r6, VCPU_CR_TM(r9) + std r7, VCPU_CTR_TM(r9) + std r8, VCPU_AMR_TM(r9) + std r10, VCPU_TAR_TM(r9) + std r11, VCPU_XER_TM(r9) + + /* Restore r12 as trap number. */ + lwz r12, VCPU_TRAP(r9) + + /* Save FP/VSX. */ + addi r3, r9, VCPU_FPRS_TM + bl store_fp_state + addi r3, r9, VCPU_VRS_TM + bl store_vr_state + mfspr r6, SPRN_VRSAVE + stw r6, VCPU_VRSAVE_TM(r9) +1: + /* + * We need to save these SPRs after the treclaim so that the software + * error code is recorded correctly in the TEXASR. Also the user may + * change these outside of a transaction, so they must always be + * context switched. + */ + mfspr r7, SPRN_TEXASR + std r7, VCPU_TEXASR(r9) +11: + mfspr r5, SPRN_TFHAR + mfspr r6, SPRN_TFIAR + std r5, VCPU_TFHAR(r9) + std r6, VCPU_TFIAR(r9) + + ld r0, PPC_LR_STKOFF(r1) + mtlr r0 + blr + +/* + * _kvmppc_save_tm_pr() is a wrapper around __kvmppc_save_tm(), so that it can + * be invoked from C function by PR KVM only. + */ +_GLOBAL(_kvmppc_save_tm_pr) + mflr r5 + std r5, PPC_LR_STKOFF(r1) + stdu r1, -SWITCH_FRAME_SIZE(r1) + SAVE_NVGPRS(r1) + + /* save MSR since TM/math bits might be impacted + * by __kvmppc_save_tm(). + */ + mfmsr r5 + SAVE_GPR(5, r1) + + /* also save DSCR/CR/TAR so that it can be recovered later */ + mfspr r6, SPRN_DSCR + SAVE_GPR(6, r1) + + mfcr r7 + stw r7, _CCR(r1) + + mfspr r8, SPRN_TAR + SAVE_GPR(8, r1) + + bl __kvmppc_save_tm + + REST_GPR(8, r1) + mtspr SPRN_TAR, r8 + + ld r7, _CCR(r1) + mtcr r7 + + REST_GPR(6, r1) + mtspr SPRN_DSCR, r6 + + /* need preserve current MSR's MSR_TS bits */ + REST_GPR(5, r1) + mfmsr r6 + rldicl r6, r6, 64 - MSR_TS_S_LG, 62 + rldimi r5, r6, MSR_TS_S_LG, 63 - MSR_TS_T_LG + mtmsrd r5 + + REST_NVGPRS(r1) + addi r1, r1, SWITCH_FRAME_SIZE + ld r5, PPC_LR_STKOFF(r1) + mtlr r5 + blr + +EXPORT_SYMBOL_GPL(_kvmppc_save_tm_pr); + +/* + * Restore transactional state and TM-related registers. + * Called with: + * - r3 pointing to the vcpu struct. + * - r4 is the guest MSR with desired TS bits: + * For HV KVM, it is VCPU_MSR + * For PR KVM, it is provided by caller + * This potentially modifies all checkpointed registers. + * It restores r1, r2 from the PACA. + */ +_GLOBAL(__kvmppc_restore_tm) + mflr r0 + std r0, PPC_LR_STKOFF(r1) + + /* Turn on TM/FP/VSX/VMX so we can restore them. */ + mfmsr r5 + li r6, MSR_TM >> 32 + sldi r6, r6, 32 + or r5, r5, r6 + ori r5, r5, MSR_FP + oris r5, r5, (MSR_VEC | MSR_VSX)@h + mtmsrd r5 + + /* + * The user may change these outside of a transaction, so they must + * always be context switched. + */ + ld r5, VCPU_TFHAR(r3) + ld r6, VCPU_TFIAR(r3) + ld r7, VCPU_TEXASR(r3) + mtspr SPRN_TFHAR, r5 + mtspr SPRN_TFIAR, r6 + mtspr SPRN_TEXASR, r7 + + mr r5, r4 + rldicl. r5, r5, 64 - MSR_TS_S_LG, 62 + beqlr /* TM not active in guest */ + std r1, HSTATE_SCRATCH2(r13) + + /* Make sure the failure summary is set, otherwise we'll program check + * when we trechkpt. It's possible that this might have been not set + * on a kvmppc_set_one_reg() call but we shouldn't let this crash the + * host. + */ + oris r7, r7, (TEXASR_FS)@h + mtspr SPRN_TEXASR, r7 + + /* + * We need to load up the checkpointed state for the guest. + * We need to do this early as it will blow away any GPRs, VSRs and + * some SPRs. + */ + + mr r31, r3 + addi r3, r31, VCPU_FPRS_TM + bl load_fp_state + addi r3, r31, VCPU_VRS_TM + bl load_vr_state + mr r3, r31 + lwz r7, VCPU_VRSAVE_TM(r3) + mtspr SPRN_VRSAVE, r7 + + ld r5, VCPU_LR_TM(r3) + lwz r6, VCPU_CR_TM(r3) + ld r7, VCPU_CTR_TM(r3) + ld r8, VCPU_AMR_TM(r3) + ld r9, VCPU_TAR_TM(r3) + ld r10, VCPU_XER_TM(r3) + mtlr r5 + mtcr r6 + mtctr r7 + mtspr SPRN_AMR, r8 + mtspr SPRN_TAR, r9 + mtxer r10 + + /* + * Load up PPR and DSCR values but don't put them in the actual SPRs + * till the last moment to avoid running with userspace PPR and DSCR for + * too long. + */ + ld r29, VCPU_DSCR_TM(r3) + ld r30, VCPU_PPR_TM(r3) + + std r2, PACATMSCRATCH(r13) /* Save TOC */ + + /* Clear the MSR RI since r1, r13 are all going to be foobar. */ + li r5, 0 + mtmsrd r5, 1 + + /* Load GPRs r0-r28 */ + reg = 0 + .rept 29 + ld reg, VCPU_GPRS_TM(reg)(r31) + reg = reg + 1 + .endr + + mtspr SPRN_DSCR, r29 + mtspr SPRN_PPR, r30 + + /* Load final GPRs */ + ld 29, VCPU_GPRS_TM(29)(r31) + ld 30, VCPU_GPRS_TM(30)(r31) + ld 31, VCPU_GPRS_TM(31)(r31) + + /* TM checkpointed state is now setup. All GPRs are now volatile. */ + TRECHKPT + + /* Now let's get back the state we need. */ + HMT_MEDIUM + GET_PACA(r13) +#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE + ld r29, HSTATE_DSCR(r13) + mtspr SPRN_DSCR, r29 +#endif + ld r1, HSTATE_SCRATCH2(r13) + ld r2, PACATMSCRATCH(r13) + + /* Set the MSR RI since we have our registers back. */ + li r5, MSR_RI + mtmsrd r5, 1 + ld r0, PPC_LR_STKOFF(r1) + mtlr r0 + blr + +/* + * _kvmppc_restore_tm_pr() is a wrapper around __kvmppc_restore_tm(), so that it + * can be invoked from C function by PR KVM only. + */ +_GLOBAL(_kvmppc_restore_tm_pr) + mflr r5 + std r5, PPC_LR_STKOFF(r1) + stdu r1, -SWITCH_FRAME_SIZE(r1) + SAVE_NVGPRS(r1) + + /* save MSR to avoid TM/math bits change */ + mfmsr r5 + SAVE_GPR(5, r1) + + /* also save DSCR/CR/TAR so that it can be recovered later */ + mfspr r6, SPRN_DSCR + SAVE_GPR(6, r1) + + mfcr r7 + stw r7, _CCR(r1) + + mfspr r8, SPRN_TAR + SAVE_GPR(8, r1) + + bl __kvmppc_restore_tm + + REST_GPR(8, r1) + mtspr SPRN_TAR, r8 + + ld r7, _CCR(r1) + mtcr r7 + + REST_GPR(6, r1) + mtspr SPRN_DSCR, r6 + + /* need preserve current MSR's MSR_TS bits */ + REST_GPR(5, r1) + mfmsr r6 + rldicl r6, r6, 64 - MSR_TS_S_LG, 62 + rldimi r5, r6, MSR_TS_S_LG, 63 - MSR_TS_T_LG + mtmsrd r5 + + REST_NVGPRS(r1) + addi r1, r1, SWITCH_FRAME_SIZE + ld r5, PPC_LR_STKOFF(r1) + mtlr r5 + blr + +EXPORT_SYMBOL_GPL(_kvmppc_restore_tm_pr); +#endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile index 653901042ad7..d0ca13ad8231 100644 --- a/arch/powerpc/lib/Makefile +++ b/arch/powerpc/lib/Makefile @@ -26,13 +26,14 @@ obj-$(CONFIG_PPC_BOOK3S_64) += copyuser_power7.o copypage_power7.o \ memcpy_power7.o obj64-y += copypage_64.o copyuser_64.o mem_64.o hweight_64.o \ - string_64.o memcpy_64.o memcmp_64.o pmem.o + memcpy_64.o pmem.o obj64-$(CONFIG_SMP) += locks.o obj64-$(CONFIG_ALTIVEC) += vmx-helper.o obj64-$(CONFIG_KPROBES_SANITY_TEST) += test_emulate_step.o -obj-y += checksum_$(BITS).o checksum_wrappers.o +obj-y += checksum_$(BITS).o checksum_wrappers.o \ + string_$(BITS).o memcmp_$(BITS).o obj-y += sstep.o ldstfp.o quad.o obj64-y += quad.o diff --git a/arch/powerpc/lib/checksum_32.S b/arch/powerpc/lib/checksum_32.S index 9a671c774b22..aa224069f93a 100644 --- a/arch/powerpc/lib/checksum_32.S +++ b/arch/powerpc/lib/checksum_32.S @@ -47,16 +47,25 @@ _GLOBAL(__csum_partial) bdnz 2b 21: srwi. r6,r4,4 /* # blocks of 4 words to do */ beq 3f + lwz r0,4(r3) mtctr r6 -22: lwz r0,4(r3) lwz r6,8(r3) + adde r5,r5,r0 lwz r7,12(r3) + adde r5,r5,r6 lwzu r8,16(r3) + adde r5,r5,r7 + bdz 23f +22: lwz r0,4(r3) + adde r5,r5,r8 + lwz r6,8(r3) adde r5,r5,r0 + lwz r7,12(r3) adde r5,r5,r6 + lwzu r8,16(r3) adde r5,r5,r7 - adde r5,r5,r8 bdnz 22b +23: adde r5,r5,r8 3: andi. r0,r4,2 beq+ 4f lhz r0,4(r3) @@ -293,3 +302,36 @@ dst_error: EX_TABLE(51b, dst_error); EXPORT_SYMBOL(csum_partial_copy_generic) + +/* + * __sum16 csum_ipv6_magic(const struct in6_addr *saddr, + * const struct in6_addr *daddr, + * __u32 len, __u8 proto, __wsum sum) + */ + +_GLOBAL(csum_ipv6_magic) + lwz r8, 0(r3) + lwz r9, 4(r3) + addc r0, r7, r8 + lwz r10, 8(r3) + adde r0, r0, r9 + lwz r11, 12(r3) + adde r0, r0, r10 + lwz r8, 0(r4) + adde r0, r0, r11 + lwz r9, 4(r4) + adde r0, r0, r8 + lwz r10, 8(r4) + adde r0, r0, r9 + lwz r11, 12(r4) + adde r0, r0, r10 + add r5, r5, r6 /* assumption: len + proto doesn't carry */ + adde r0, r0, r11 + adde r0, r0, r5 + addze r0, r0 + rotlwi r3, r0, 16 + add r3, r0, r3 + not r3, r3 + rlwinm r3, r3, 16, 16, 31 + blr +EXPORT_SYMBOL(csum_ipv6_magic) diff --git a/arch/powerpc/lib/checksum_64.S b/arch/powerpc/lib/checksum_64.S index d7f1a966136e..886ed94b9c13 100644 --- a/arch/powerpc/lib/checksum_64.S +++ b/arch/powerpc/lib/checksum_64.S @@ -429,3 +429,31 @@ dstnr; stb r6,0(r4) stw r6,0(r8) blr EXPORT_SYMBOL(csum_partial_copy_generic) + +/* + * __sum16 csum_ipv6_magic(const struct in6_addr *saddr, + * const struct in6_addr *daddr, + * __u32 len, __u8 proto, __wsum sum) + */ + +_GLOBAL(csum_ipv6_magic) + ld r8, 0(r3) + ld r9, 8(r3) + add r5, r5, r6 + addc r0, r8, r9 + ld r10, 0(r4) + ld r11, 8(r4) + adde r0, r0, r10 + add r5, r5, r7 + adde r0, r0, r11 + adde r0, r0, r5 + addze r0, r0 + rotldi r3, r0, 32 /* fold two 32 bit halves together */ + add r3, r0, r3 + srdi r0, r3, 32 + rotlwi r3, r0, 16 /* fold two 16 bit halves together */ + add r3, r0, r3 + not r3, r3 + rlwinm r3, r3, 16, 16, 31 + blr +EXPORT_SYMBOL(csum_ipv6_magic) diff --git a/arch/powerpc/lib/feature-fixups-test.S b/arch/powerpc/lib/feature-fixups-test.S index f4613118132e..f16cec989506 100644 --- a/arch/powerpc/lib/feature-fixups-test.S +++ b/arch/powerpc/lib/feature-fixups-test.S @@ -167,16 +167,52 @@ globl(ftr_fixup_test6_expected) blt 2b b 3f b 1b -2: or 1,1,1 +3: or 1,1,1 + or 2,2,2 + or 3,3,3 + +globl(ftr_fixup_test7) + or 1,1,1 +BEGIN_FTR_SECTION + or 2,2,2 + or 2,2,2 or 2,2,2 -3: or 3,3,3 + or 2,2,2 + or 2,2,2 + or 2,2,2 + or 2,2,2 +FTR_SECTION_ELSE +2: b 3f +3: or 5,5,5 + beq 3b + b 1f + or 6,6,6 + b 2b + bdnz 3b +1: +ALT_FTR_SECTION_END(0, 1) + or 1,1,1 + or 1,1,1 + +globl(end_ftr_fixup_test7) + nop +globl(ftr_fixup_test7_expected) + or 1,1,1 +2: b 3f +3: or 5,5,5 + beq 3b + b 1f + or 6,6,6 + b 2b + bdnz 3b +1: or 1,1,1 #if 0 /* Test that if we have a larger else case the assembler spots it and * reports an error. #if 0'ed so as not to break the build normally. */ -ftr_fixup_test7: +ftr_fixup_test_too_big: or 1,1,1 BEGIN_FTR_SECTION or 2,2,2 diff --git a/arch/powerpc/lib/feature-fixups.c b/arch/powerpc/lib/feature-fixups.c index e1bcdc32a851..8b69f868298c 100644 --- a/arch/powerpc/lib/feature-fixups.c +++ b/arch/powerpc/lib/feature-fixups.c @@ -277,6 +277,43 @@ void do_rfi_flush_fixups(enum l1d_flush_type types) (types & L1D_FLUSH_MTTRIG) ? "mttrig type" : "unknown"); } + +void do_barrier_nospec_fixups_range(bool enable, void *fixup_start, void *fixup_end) +{ + unsigned int instr, *dest; + long *start, *end; + int i; + + start = fixup_start; + end = fixup_end; + + instr = 0x60000000; /* nop */ + + if (enable) { + pr_info("barrier-nospec: using ORI speculation barrier\n"); + instr = 0x63ff0000; /* ori 31,31,0 speculation barrier */ + } + + for (i = 0; start < end; start++, i++) { + dest = (void *)start + *start; + + pr_devel("patching dest %lx\n", (unsigned long)dest); + patch_instruction(dest, instr); + } + + printk(KERN_DEBUG "barrier-nospec: patched %d locations\n", i); +} + +void do_barrier_nospec_fixups(bool enable) +{ + void *start, *end; + + start = PTRRELOC(&__start___barrier_nospec_fixup), + end = PTRRELOC(&__stop___barrier_nospec_fixup); + + do_barrier_nospec_fixups_range(enable, start, end); +} + #endif /* CONFIG_PPC_BOOK3S_64 */ void do_lwsync_fixups(unsigned long value, void *fixup_start, void *fixup_end) @@ -400,7 +437,7 @@ static void test_basic_patching(void) extern unsigned int end_ftr_fixup_test1[]; extern unsigned int ftr_fixup_test1_orig[]; extern unsigned int ftr_fixup_test1_expected[]; - int size = end_ftr_fixup_test1 - ftr_fixup_test1; + int size = 4 * (end_ftr_fixup_test1 - ftr_fixup_test1); fixup.value = fixup.mask = 8; fixup.start_off = calc_offset(&fixup, ftr_fixup_test1 + 1); @@ -432,7 +469,7 @@ static void test_alternative_patching(void) extern unsigned int ftr_fixup_test2_orig[]; extern unsigned int ftr_fixup_test2_alt[]; extern unsigned int ftr_fixup_test2_expected[]; - int size = end_ftr_fixup_test2 - ftr_fixup_test2; + int size = 4 * (end_ftr_fixup_test2 - ftr_fixup_test2); fixup.value = fixup.mask = 0xF; fixup.start_off = calc_offset(&fixup, ftr_fixup_test2 + 1); @@ -464,7 +501,7 @@ static void test_alternative_case_too_big(void) extern unsigned int end_ftr_fixup_test3[]; extern unsigned int ftr_fixup_test3_orig[]; extern unsigned int ftr_fixup_test3_alt[]; - int size = end_ftr_fixup_test3 - ftr_fixup_test3; + int size = 4 * (end_ftr_fixup_test3 - ftr_fixup_test3); fixup.value = fixup.mask = 0xC; fixup.start_off = calc_offset(&fixup, ftr_fixup_test3 + 1); @@ -491,7 +528,7 @@ static void test_alternative_case_too_small(void) extern unsigned int ftr_fixup_test4_orig[]; extern unsigned int ftr_fixup_test4_alt[]; extern unsigned int ftr_fixup_test4_expected[]; - int size = end_ftr_fixup_test4 - ftr_fixup_test4; + int size = 4 * (end_ftr_fixup_test4 - ftr_fixup_test4); unsigned long flag; /* Check a high-bit flag */ @@ -525,7 +562,7 @@ static void test_alternative_case_with_branch(void) extern unsigned int ftr_fixup_test5[]; extern unsigned int end_ftr_fixup_test5[]; extern unsigned int ftr_fixup_test5_expected[]; - int size = end_ftr_fixup_test5 - ftr_fixup_test5; + int size = 4 * (end_ftr_fixup_test5 - ftr_fixup_test5); check(memcmp(ftr_fixup_test5, ftr_fixup_test5_expected, size) == 0); } @@ -535,11 +572,21 @@ static void test_alternative_case_with_external_branch(void) extern unsigned int ftr_fixup_test6[]; extern unsigned int end_ftr_fixup_test6[]; extern unsigned int ftr_fixup_test6_expected[]; - int size = end_ftr_fixup_test6 - ftr_fixup_test6; + int size = 4 * (end_ftr_fixup_test6 - ftr_fixup_test6); check(memcmp(ftr_fixup_test6, ftr_fixup_test6_expected, size) == 0); } +static void test_alternative_case_with_branch_to_end(void) +{ + extern unsigned int ftr_fixup_test7[]; + extern unsigned int end_ftr_fixup_test7[]; + extern unsigned int ftr_fixup_test7_expected[]; + int size = 4 * (end_ftr_fixup_test7 - ftr_fixup_test7); + + check(memcmp(ftr_fixup_test7, ftr_fixup_test7_expected, size) == 0); +} + static void test_cpu_macros(void) { extern u8 ftr_fixup_test_FTR_macros[]; @@ -595,6 +642,7 @@ static int __init test_feature_fixups(void) test_alternative_case_too_small(); test_alternative_case_with_branch(); test_alternative_case_with_external_branch(); + test_alternative_case_with_branch_to_end(); test_cpu_macros(); test_fw_macros(); test_lwsync_macros(); diff --git a/arch/powerpc/lib/memcmp_32.S b/arch/powerpc/lib/memcmp_32.S new file mode 100644 index 000000000000..5010e376f7b8 --- /dev/null +++ b/arch/powerpc/lib/memcmp_32.S @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * memcmp for PowerPC32 + * + * Copyright (C) 1996 Paul Mackerras. + * + */ + +#include +#include + + .text + +_GLOBAL(memcmp) + srawi. r7, r5, 2 /* Divide len by 4 */ + mr r6, r3 + beq- 3f + mtctr r7 + li r7, 0 +1: lwzx r3, r6, r7 + lwzx r0, r4, r7 + addi r7, r7, 4 + cmplw cr0, r3, r0 + bdnzt eq, 1b + bne 5f +3: andi. r3, r5, 3 + beqlr + cmplwi cr1, r3, 2 + blt- cr1, 4f + lhzx r3, r6, r7 + lhzx r0, r4, r7 + addi r7, r7, 2 + subf. r3, r0, r3 + beqlr cr1 + bnelr +4: lbzx r3, r6, r7 + lbzx r0, r4, r7 + subf. r3, r0, r3 + blr +5: li r3, 1 + bgtlr + li r3, -1 + blr +EXPORT_SYMBOL(memcmp) diff --git a/arch/powerpc/lib/rheap.c b/arch/powerpc/lib/rheap.c index 94058c21a482..6aa774aa5b16 100644 --- a/arch/powerpc/lib/rheap.c +++ b/arch/powerpc/lib/rheap.c @@ -54,7 +54,7 @@ static int grow(rh_info_t * info, int max_blocks) new_blocks = max_blocks - info->max_blocks; - block = kmalloc(sizeof(rh_block_t) * max_blocks, GFP_ATOMIC); + block = kmalloc_array(max_blocks, sizeof(rh_block_t), GFP_ATOMIC); if (block == NULL) return -ENOMEM; diff --git a/arch/powerpc/lib/sstep.c b/arch/powerpc/lib/sstep.c index 34d68f1b1b40..d81568f783e5 100644 --- a/arch/powerpc/lib/sstep.c +++ b/arch/powerpc/lib/sstep.c @@ -1065,9 +1065,10 @@ static nokprobe_inline void do_popcnt(const struct pt_regs *regs, { unsigned long long out = v1; - out -= (out >> 1) & 0x5555555555555555; - out = (0x3333333333333333 & out) + (0x3333333333333333 & (out >> 2)); - out = (out + (out >> 4)) & 0x0f0f0f0f0f0f0f0f; + out -= (out >> 1) & 0x5555555555555555ULL; + out = (0x3333333333333333ULL & out) + + (0x3333333333333333ULL & (out >> 2)); + out = (out + (out >> 4)) & 0x0f0f0f0f0f0f0f0fULL; if (size == 8) { /* popcntb */ op->val = out; @@ -1076,7 +1077,7 @@ static nokprobe_inline void do_popcnt(const struct pt_regs *regs, out += out >> 8; out += out >> 16; if (size == 32) { /* popcntw */ - op->val = out & 0x0000003f0000003f; + op->val = out & 0x0000003f0000003fULL; return; } @@ -1114,7 +1115,7 @@ static nokprobe_inline void do_prty(const struct pt_regs *regs, res ^= res >> 16; if (size == 32) { /* prtyw */ - op->val = res & 0x0000000100000001; + op->val = res & 0x0000000100000001ULL; return; } @@ -2544,6 +2545,15 @@ int analyse_instr(struct instruction_op *op, const struct pt_regs *regs, #endif /* __powerpc64__ */ } + +#ifdef CONFIG_VSX + if ((GETTYPE(op->type) == LOAD_VSX || + GETTYPE(op->type) == STORE_VSX) && + !cpu_has_feature(CPU_FTR_VSX)) { + return -1; + } +#endif /* CONFIG_VSX */ + return 0; logical_done: @@ -2641,7 +2651,7 @@ void emulate_update_regs(struct pt_regs *regs, struct instruction_op *op) unsigned long next_pc; next_pc = truncate_if_32bit(regs->msr, regs->nip + 4); - switch (op->type & INSTR_TYPE_MASK) { + switch (GETTYPE(op->type)) { case COMPUTE: if (op->type & SETREG) regs->gpr[op->reg] = op->val; @@ -2739,7 +2749,7 @@ int emulate_loadstore(struct pt_regs *regs, struct instruction_op *op) err = 0; size = GETSIZE(op->type); - type = op->type & INSTR_TYPE_MASK; + type = GETTYPE(op->type); cross_endian = (regs->msr & MSR_LE) != (MSR_KERNEL & MSR_LE); ea = truncate_if_32bit(regs->msr, op->ea); @@ -3001,7 +3011,7 @@ int emulate_step(struct pt_regs *regs, unsigned int instr) } err = 0; - type = op.type & INSTR_TYPE_MASK; + type = GETTYPE(op.type); if (OP_IS_LOAD_STORE(type)) { err = emulate_loadstore(regs, &op); diff --git a/arch/powerpc/lib/string.S b/arch/powerpc/lib/string.S index a787776822d8..4b41970e9ed8 100644 --- a/arch/powerpc/lib/string.S +++ b/arch/powerpc/lib/string.S @@ -8,10 +8,9 @@ * as published by the Free Software Foundation; either version * 2 of the License, or (at your option) any later version. */ -#include -#include #include #include +#include .text @@ -23,7 +22,7 @@ _GLOBAL(strncpy) mtctr r5 addi r6,r3,-1 addi r4,r4,-1 - .balign 16 + .balign IFETCH_ALIGN_BYTES 1: lbzu r0,1(r4) cmpwi 0,r0,0 stbu r0,1(r6) @@ -43,7 +42,7 @@ _GLOBAL(strncmp) mtctr r5 addi r5,r3,-1 addi r4,r4,-1 - .balign 16 + .balign IFETCH_ALIGN_BYTES 1: lbzu r3,1(r5) cmpwi 1,r3,0 lbzu r0,1(r4) @@ -55,29 +54,12 @@ _GLOBAL(strncmp) blr EXPORT_SYMBOL(strncmp) -#ifdef CONFIG_PPC32 -_GLOBAL(memcmp) - PPC_LCMPI 0,r5,0 - beq- 2f - mtctr r5 - addi r6,r3,-1 - addi r4,r4,-1 -1: lbzu r3,1(r6) - lbzu r0,1(r4) - subf. r3,r0,r3 - bdnzt 2,1b - blr -2: li r3,0 - blr -EXPORT_SYMBOL(memcmp) -#endif - _GLOBAL(memchr) PPC_LCMPI 0,r5,0 beq- 2f mtctr r5 addi r3,r3,-1 - .balign 16 + .balign IFETCH_ALIGN_BYTES 1: lbzu r0,1(r3) cmpw 0,r0,r4 bdnzf 2,1b @@ -85,47 +67,3 @@ _GLOBAL(memchr) 2: li r3,0 blr EXPORT_SYMBOL(memchr) - -#ifdef CONFIG_PPC32 -_GLOBAL(__clear_user) - addi r6,r3,-4 - li r3,0 - li r5,0 - cmplwi 0,r4,4 - blt 7f - /* clear a single word */ -11: stwu r5,4(r6) - beqlr - /* clear word sized chunks */ - andi. r0,r6,3 - add r4,r0,r4 - subf r6,r0,r6 - srwi r0,r4,2 - andi. r4,r4,3 - mtctr r0 - bdz 7f -1: stwu r5,4(r6) - bdnz 1b - /* clear byte sized chunks */ -7: cmpwi 0,r4,0 - beqlr - mtctr r4 - addi r6,r6,3 -8: stbu r5,1(r6) - bdnz 8b - blr -90: mr r3,r4 - blr -91: mfctr r3 - slwi r3,r3,2 - add r3,r3,r4 - blr -92: mfctr r3 - blr - - EX_TABLE(11b, 90b) - EX_TABLE(1b, 91b) - EX_TABLE(8b, 92b) - -EXPORT_SYMBOL(__clear_user) -#endif diff --git a/arch/powerpc/lib/string_32.S b/arch/powerpc/lib/string_32.S new file mode 100644 index 000000000000..f69a6aab7bfb --- /dev/null +++ b/arch/powerpc/lib/string_32.S @@ -0,0 +1,90 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * String handling functions for PowerPC32 + * + * Copyright (C) 1996 Paul Mackerras. + * + */ + +#include +#include +#include + + .text + +CACHELINE_BYTES = L1_CACHE_BYTES +LG_CACHELINE_BYTES = L1_CACHE_SHIFT +CACHELINE_MASK = (L1_CACHE_BYTES-1) + +_GLOBAL(__clear_user) +/* + * Use dcbz on the complete cache lines in the destination + * to set them to zero. This requires that the destination + * area is cacheable. + */ + cmplwi cr0, r4, 4 + mr r10, r3 + li r3, 0 + blt 7f + +11: stw r3, 0(r10) + beqlr + andi. r0, r10, 3 + add r11, r0, r4 + subf r6, r0, r10 + + clrlwi r7, r6, 32 - LG_CACHELINE_BYTES + add r8, r7, r11 + srwi r9, r8, LG_CACHELINE_BYTES + addic. r9, r9, -1 /* total number of complete cachelines */ + ble 2f + xori r0, r7, CACHELINE_MASK & ~3 + srwi. r0, r0, 2 + beq 3f + mtctr r0 +4: stwu r3, 4(r6) + bdnz 4b +3: mtctr r9 + li r7, 4 +10: dcbz r7, r6 + addi r6, r6, CACHELINE_BYTES + bdnz 10b + clrlwi r11, r8, 32 - LG_CACHELINE_BYTES + addi r11, r11, 4 + +2: srwi r0 ,r11 ,2 + mtctr r0 + bdz 6f +1: stwu r3, 4(r6) + bdnz 1b +6: andi. r11, r11, 3 + beqlr + mtctr r11 + addi r6, r6, 3 +8: stbu r3, 1(r6) + bdnz 8b + blr + +7: cmpwi cr0, r4, 0 + beqlr + mtctr r4 + addi r6, r10, -1 +9: stbu r3, 1(r6) + bdnz 9b + blr + +90: mr r3, r4 + blr +91: add r3, r10, r4 + subf r3, r6, r3 + blr + + EX_TABLE(11b, 90b) + EX_TABLE(4b, 91b) + EX_TABLE(10b, 91b) + EX_TABLE(1b, 91b) + EX_TABLE(8b, 91b) + EX_TABLE(9b, 91b) + +EXPORT_SYMBOL(__clear_user) diff --git a/arch/powerpc/lib/test_emulate_step.c b/arch/powerpc/lib/test_emulate_step.c index 2534c1447554..6c47daa61614 100644 --- a/arch/powerpc/lib/test_emulate_step.c +++ b/arch/powerpc/lib/test_emulate_step.c @@ -387,10 +387,14 @@ static void __init test_lxvd2x_stxvd2x(void) /* lxvd2x vsr39, r3, r4 */ stepped = emulate_step(®s, TEST_LXVD2X(39, 3, 4)); - if (stepped == 1) + if (stepped == 1 && cpu_has_feature(CPU_FTR_VSX)) { show_result("lxvd2x", "PASS"); - else - show_result("lxvd2x", "FAIL"); + } else { + if (!cpu_has_feature(CPU_FTR_VSX)) + show_result("lxvd2x", "PASS (!CPU_FTR_VSX)"); + else + show_result("lxvd2x", "FAIL"); + } /*** stxvd2x ***/ @@ -404,10 +408,15 @@ static void __init test_lxvd2x_stxvd2x(void) stepped = emulate_step(®s, TEST_STXVD2X(39, 3, 4)); if (stepped == 1 && cached_b[0] == c.b[0] && cached_b[1] == c.b[1] && - cached_b[2] == c.b[2] && cached_b[3] == c.b[3]) + cached_b[2] == c.b[2] && cached_b[3] == c.b[3] && + cpu_has_feature(CPU_FTR_VSX)) { show_result("stxvd2x", "PASS"); - else - show_result("stxvd2x", "FAIL"); + } else { + if (!cpu_has_feature(CPU_FTR_VSX)) + show_result("stxvd2x", "PASS (!CPU_FTR_VSX)"); + else + show_result("stxvd2x", "FAIL"); + } } #else static void __init test_lxvd2x_stxvd2x(void) diff --git a/arch/powerpc/lib/xor_vmx_glue.c b/arch/powerpc/lib/xor_vmx_glue.c index 6521fe5e8cef..dab2b6bfcf36 100644 --- a/arch/powerpc/lib/xor_vmx_glue.c +++ b/arch/powerpc/lib/xor_vmx_glue.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "xor_vmx.h" void xor_altivec_2(unsigned long bytes, unsigned long *v1_in, diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index c01d627e687a..b1ca7a0974e3 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -66,37 +67,33 @@ static inline bool notify_page_fault(struct pt_regs *regs) } /* - * Check whether the instruction at regs->nip is a store using + * Check whether the instruction inst is a store using * an update addressing form which will update r1. */ -static bool store_updates_sp(struct pt_regs *regs) +static bool store_updates_sp(unsigned int inst) { - unsigned int inst; - - if (get_user(inst, (unsigned int __user *)regs->nip)) - return false; /* check for 1 in the rA field */ if (((inst >> 16) & 0x1f) != 1) return false; /* check major opcode */ switch (inst >> 26) { - case 37: /* stwu */ - case 39: /* stbu */ - case 45: /* sthu */ - case 53: /* stfsu */ - case 55: /* stfdu */ + case OP_STWU: + case OP_STBU: + case OP_STHU: + case OP_STFSU: + case OP_STFDU: return true; - case 62: /* std or stdu */ + case OP_STD: /* std or stdu */ return (inst & 3) == 1; - case 31: + case OP_31: /* check minor opcode */ switch ((inst >> 1) & 0x3ff) { - case 181: /* stdux */ - case 183: /* stwux */ - case 247: /* stbux */ - case 439: /* sthux */ - case 695: /* stfsux */ - case 759: /* stfdux */ + case OP_31_XOP_STDUX: + case OP_31_XOP_STWUX: + case OP_31_XOP_STBUX: + case OP_31_XOP_STHUX: + case OP_31_XOP_STFSUX: + case OP_31_XOP_STFDUX: return true; } } @@ -168,6 +165,7 @@ static int do_sigbus(struct pt_regs *regs, unsigned long address, return SIGBUS; current->thread.trap_nr = BUS_ADRERR; + clear_siginfo(&info); info.si_signo = SIGBUS; info.si_errno = 0; info.si_code = BUS_ADRERR; @@ -234,8 +232,8 @@ static bool bad_kernel_fault(bool is_exec, unsigned long error_code, } static bool bad_stack_expansion(struct pt_regs *regs, unsigned long address, - struct vm_area_struct *vma, - bool store_update_sp) + struct vm_area_struct *vma, unsigned int flags, + bool *must_retry) { /* * N.B. The POWER/Open ABI allows programs to access up to @@ -247,6 +245,7 @@ static bool bad_stack_expansion(struct pt_regs *regs, unsigned long address, * expand to 1MB without further checks. */ if (address + 0x100000 < vma->vm_end) { + unsigned int __user *nip = (unsigned int __user *)regs->nip; /* get user regs even if this fault is in kernel mode */ struct pt_regs *uregs = current->thread.regs; if (uregs == NULL) @@ -264,8 +263,22 @@ static bool bad_stack_expansion(struct pt_regs *regs, unsigned long address, * between the last mapped region and the stack will * expand the stack rather than segfaulting. */ - if (address + 2048 < uregs->gpr[1] && !store_update_sp) - return true; + if (address + 2048 >= uregs->gpr[1]) + return false; + + if ((flags & FAULT_FLAG_WRITE) && (flags & FAULT_FLAG_USER) && + access_ok(VERIFY_READ, nip, sizeof(*nip))) { + unsigned int inst; + int res; + + pagefault_disable(); + res = __get_user_inatomic(inst, nip); + pagefault_enable(); + if (!res) + return !store_updates_sp(inst); + *must_retry = true; + } + return true; } return false; } @@ -403,7 +416,7 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address, int is_user = user_mode(regs); int is_write = page_fault_is_write(error_code); int fault, major = 0; - bool store_update_sp = false; + bool must_retry = false; if (notify_page_fault(regs)) return 0; @@ -454,9 +467,6 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address, * can result in fault, which will cause a deadlock when called with * mmap_sem held */ - if (is_write && is_user) - store_update_sp = store_updates_sp(regs); - if (is_user) flags |= FAULT_FLAG_USER; if (is_write) @@ -503,8 +513,17 @@ retry: return bad_area(regs, address); /* The stack is being expanded, check if it's valid */ - if (unlikely(bad_stack_expansion(regs, address, vma, store_update_sp))) - return bad_area(regs, address); + if (unlikely(bad_stack_expansion(regs, address, vma, flags, + &must_retry))) { + if (!must_retry) + return bad_area(regs, address); + + up_read(&mm->mmap_sem); + if (fault_in_pages_readable((const char __user *)regs->nip, + sizeof(unsigned int))) + return bad_area_nosemaphore(regs, address); + goto retry; + } /* Try to expand it */ if (unlikely(expand_stack(vma, address))) diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c index 0bd3790d35df..8318716e5075 100644 --- a/arch/powerpc/mm/hash_utils_64.c +++ b/arch/powerpc/mm/hash_utils_64.c @@ -64,6 +64,7 @@ #include #include #include +#include #ifdef DEBUG #define DBG(fmt...) udbg_printf(fmt) @@ -572,8 +573,10 @@ static void __init htab_scan_page_sizes(void) } #ifdef CONFIG_HUGETLB_PAGE - /* Reserve 16G huge page memory sections for huge pages */ - of_scan_flat_dt(htab_dt_scan_hugepage_blocks, NULL); + if (!hugetlb_disabled) { + /* Reserve 16G huge page memory sections for huge pages */ + of_scan_flat_dt(htab_dt_scan_hugepage_blocks, NULL); + } #endif /* CONFIG_HUGETLB_PAGE */ } @@ -1010,13 +1013,14 @@ void __init hash__early_init_mmu(void) */ __pte_frag_nr = H_PTE_FRAG_NR; __pte_frag_size_shift = H_PTE_FRAG_SIZE_SHIFT; + __pmd_frag_nr = H_PMD_FRAG_NR; + __pmd_frag_size_shift = H_PMD_FRAG_SIZE_SHIFT; __pte_index_size = H_PTE_INDEX_SIZE; __pmd_index_size = H_PMD_INDEX_SIZE; __pud_index_size = H_PUD_INDEX_SIZE; __pgd_index_size = H_PGD_INDEX_SIZE; __pud_cache_index = H_PUD_CACHE_INDEX; - __pmd_cache_index = H_PMD_CACHE_INDEX; __pte_table_size = H_PTE_TABLE_SIZE; __pmd_table_size = H_PMD_TABLE_SIZE; __pud_table_size = H_PUD_TABLE_SIZE; diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index f1153f8254e3..7c5f479c5c00 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -35,6 +35,8 @@ #define PAGE_SHIFT_16M 24 #define PAGE_SHIFT_16G 34 +bool hugetlb_disabled = false; + unsigned int HPAGE_SHIFT; EXPORT_SYMBOL(HPAGE_SHIFT); @@ -50,7 +52,8 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long s } static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp, - unsigned long address, unsigned pdshift, unsigned pshift) + unsigned long address, unsigned int pdshift, + unsigned int pshift, spinlock_t *ptl) { struct kmem_cache *cachep; pte_t *new; @@ -80,8 +83,7 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp, */ smp_wmb(); - spin_lock(&mm->page_table_lock); - + spin_lock(ptl); /* * We have multiple higher-level entries that point to the same * actual pte location. Fill in each as we go and backtrack on error. @@ -111,7 +113,7 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp, *hpdp = __hugepd(0); kmem_cache_free(cachep, new); } - spin_unlock(&mm->page_table_lock); + spin_unlock(ptl); return 0; } @@ -136,6 +138,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz hugepd_t *hpdp = NULL; unsigned pshift = __ffs(sz); unsigned pdshift = PGDIR_SHIFT; + spinlock_t *ptl; addr &= ~(sz-1); pg = pgd_offset(mm, addr); @@ -144,39 +147,46 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz if (pshift == PGDIR_SHIFT) /* 16GB huge page */ return (pte_t *) pg; - else if (pshift > PUD_SHIFT) + else if (pshift > PUD_SHIFT) { /* * We need to use hugepd table */ + ptl = &mm->page_table_lock; hpdp = (hugepd_t *)pg; - else { + } else { pdshift = PUD_SHIFT; pu = pud_alloc(mm, pg, addr); if (pshift == PUD_SHIFT) return (pte_t *)pu; - else if (pshift > PMD_SHIFT) + else if (pshift > PMD_SHIFT) { + ptl = pud_lockptr(mm, pu); hpdp = (hugepd_t *)pu; - else { + } else { pdshift = PMD_SHIFT; pm = pmd_alloc(mm, pu, addr); if (pshift == PMD_SHIFT) /* 16MB hugepage */ return (pte_t *)pm; - else + else { + ptl = pmd_lockptr(mm, pm); hpdp = (hugepd_t *)pm; + } } } #else if (pshift >= HUGEPD_PGD_SHIFT) { + ptl = &mm->page_table_lock; hpdp = (hugepd_t *)pg; } else { pdshift = PUD_SHIFT; pu = pud_alloc(mm, pg, addr); if (pshift >= HUGEPD_PUD_SHIFT) { + ptl = pud_lockptr(mm, pu); hpdp = (hugepd_t *)pu; } else { pdshift = PMD_SHIFT; pm = pmd_alloc(mm, pu, addr); + ptl = pmd_lockptr(mm, pm); hpdp = (hugepd_t *)pm; } } @@ -186,7 +196,8 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz BUG_ON(!hugepd_none(*hpdp) && !hugepd_ok(*hpdp)); - if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr, pdshift, pshift)) + if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr, + pdshift, pshift, ptl)) return NULL; return hugepte_offset(*hpdp, addr, pdshift); @@ -497,6 +508,10 @@ struct page *follow_huge_pd(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; retry: + /* + * hugepage directory entries are protected by mm->page_table_lock + * Use this instead of huge_pte_lockptr + */ ptl = &mm->page_table_lock; spin_lock(ptl); @@ -651,6 +666,11 @@ static int __init hugetlbpage_init(void) { int psize; + if (hugetlb_disabled) { + pr_info("HugeTLB support is disabled!\n"); + return 0; + } + #if !defined(CONFIG_PPC_FSL_BOOK3E) && !defined(CONFIG_PPC_8xx) if (!radix_enabled() && !mmu_has_feature(MMU_FTR_16M_PAGE)) return -ENODEV; diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c index c3c39b02b2ba..5c8530d0c611 100644 --- a/arch/powerpc/mm/mem.c +++ b/arch/powerpc/mm/mem.c @@ -215,7 +215,7 @@ void __init mem_topology_setup(void) /* Place all memblock_regions in the same node and merge contiguous * memblock_regions */ - memblock_set_node(0, (phys_addr_t)ULLONG_MAX, &memblock.memory, 0); + memblock_set_node(0, PHYS_ADDR_MAX, &memblock.memory, 0); } void __init initmem_init(void) @@ -509,8 +509,10 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, */ unsigned long access, trap; - if (radix_enabled()) + if (radix_enabled()) { + prefetch((void *)address); return; + } /* We only want HPTEs for linux PTEs that have _PAGE_ACCESSED set */ if (!pte_young(*ptep) || address >= TASK_SIZE) diff --git a/arch/powerpc/mm/mmu_context.c b/arch/powerpc/mm/mmu_context.c index 0ab297c4cfad..f84e14f23e50 100644 --- a/arch/powerpc/mm/mmu_context.c +++ b/arch/powerpc/mm/mmu_context.c @@ -57,8 +57,10 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * in switch_slb(), and/or the store of paca->mm_ctx_id in * copy_mm_to_paca(). * - * On the read side the barrier is in pte_xchg(), which orders - * the store to the PTE vs the load of mm_cpumask. + * On the other side, the barrier is in mm/tlb-radix.c for + * radix which orders earlier stores to clear the PTEs vs + * the load of mm_cpumask. And pte_xchg which does the same + * thing for hash. * * This full barrier is needed by membarrier when switching * between processes after store to rq->curr, before user-space diff --git a/arch/powerpc/mm/mmu_context_book3s64.c b/arch/powerpc/mm/mmu_context_book3s64.c index b75194dff64c..f3d4b4a0e561 100644 --- a/arch/powerpc/mm/mmu_context_book3s64.c +++ b/arch/powerpc/mm/mmu_context_book3s64.c @@ -159,9 +159,8 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm) mm->context.id = index; -#ifdef CONFIG_PPC_64K_PAGES mm->context.pte_frag = NULL; -#endif + mm->context.pmd_frag = NULL; #ifdef CONFIG_SPAPR_TCE_IOMMU mm_iommu_init(mm); #endif @@ -192,17 +191,11 @@ static void destroy_contexts(mm_context_t *ctx) spin_unlock(&mmu_context_lock); } -#ifdef CONFIG_PPC_64K_PAGES -static void destroy_pagetable_page(struct mm_struct *mm) +static void pte_frag_destroy(void *pte_frag) { int count; - void *pte_frag; struct page *page; - pte_frag = mm->context.pte_frag; - if (!pte_frag) - return; - page = virt_to_page(pte_frag); /* drop all the pending references */ count = ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; @@ -213,12 +206,34 @@ static void destroy_pagetable_page(struct mm_struct *mm) } } -#else -static inline void destroy_pagetable_page(struct mm_struct *mm) +static void pmd_frag_destroy(void *pmd_frag) { + int count; + struct page *page; + + page = virt_to_page(pmd_frag); + /* drop all the pending references */ + count = ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT; + /* We allow PTE_FRAG_NR fragments from a PTE page */ + if (page_ref_sub_and_test(page, PMD_FRAG_NR - count)) { + pgtable_pmd_page_dtor(page); + free_unref_page(page); + } +} + +static void destroy_pagetable_page(struct mm_struct *mm) +{ + void *frag; + + frag = mm->context.pte_frag; + if (frag) + pte_frag_destroy(frag); + + frag = mm->context.pmd_frag; + if (frag) + pmd_frag_destroy(frag); return; } -#endif void destroy_context(struct mm_struct *mm) { diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c index 4c615fcb0cf0..abb43646927a 100644 --- a/arch/powerpc/mm/mmu_context_iommu.c +++ b/arch/powerpc/mm/mmu_context_iommu.c @@ -159,7 +159,7 @@ long mm_iommu_get(struct mm_struct *mm, unsigned long ua, unsigned long entries, goto unlock_exit; } - mem->hpas = vzalloc(entries * sizeof(mem->hpas[0])); + mem->hpas = vzalloc(array_size(entries, sizeof(mem->hpas[0]))); if (!mem->hpas) { kfree(mem); ret = -ENOMEM; diff --git a/arch/powerpc/mm/mmu_context_nohash.c b/arch/powerpc/mm/mmu_context_nohash.c index be8f5c9d4d08..4d80239ef83c 100644 --- a/arch/powerpc/mm/mmu_context_nohash.c +++ b/arch/powerpc/mm/mmu_context_nohash.c @@ -54,16 +54,44 @@ #include "mmu_decl.h" -static unsigned int first_context, last_context; +/* + * The MPC8xx has only 16 contexts. We rotate through them on each task switch. + * A better way would be to keep track of tasks that own contexts, and implement + * an LRU usage. That way very active tasks don't always have to pay the TLB + * reload overhead. The kernel pages are mapped shared, so the kernel can run on + * behalf of any task that makes a kernel entry. Shared does not mean they are + * not protected, just that the ASID comparison is not performed. -- Dan + * + * The IBM4xx has 256 contexts, so we can just rotate through these as a way of + * "switching" contexts. If the TID of the TLB is zero, the PID/TID comparison + * is disabled, so we can use a TID of zero to represent all kernel pages as + * shared among all contexts. -- Dan + * + * The IBM 47x core supports 16-bit PIDs, thus 65535 contexts. We should + * normally never have to steal though the facility is present if needed. + * -- BenH + */ +#define FIRST_CONTEXT 1 +#ifdef DEBUG_CLAMP_LAST_CONTEXT +#define LAST_CONTEXT DEBUG_CLAMP_LAST_CONTEXT +#elif defined(CONFIG_PPC_8xx) +#define LAST_CONTEXT 16 +#elif defined(CONFIG_PPC_47x) +#define LAST_CONTEXT 65535 +#else +#define LAST_CONTEXT 255 +#endif + static unsigned int next_context, nr_free_contexts; static unsigned long *context_map; +#ifdef CONFIG_SMP static unsigned long *stale_map[NR_CPUS]; +#endif static struct mm_struct **context_mm; static DEFINE_RAW_SPINLOCK(context_lock); -static bool no_selective_tlbil; #define CTX_MAP_SIZE \ - (sizeof(unsigned long) * (last_context / BITS_PER_LONG + 1)) + (sizeof(unsigned long) * (LAST_CONTEXT / BITS_PER_LONG + 1)) /* Steal a context from a task that has one at the moment. @@ -87,7 +115,7 @@ static unsigned int steal_context_smp(unsigned int id) struct mm_struct *mm; unsigned int cpu, max, i; - max = last_context - first_context; + max = LAST_CONTEXT - FIRST_CONTEXT; /* Attempt to free next_context first and then loop until we manage */ while (max--) { @@ -99,8 +127,8 @@ static unsigned int steal_context_smp(unsigned int id) */ if (mm->context.active) { id++; - if (id > last_context) - id = first_context; + if (id > LAST_CONTEXT) + id = FIRST_CONTEXT; continue; } pr_hardcont(" | steal %d from 0x%p", id, mm); @@ -139,10 +167,12 @@ static unsigned int steal_context_smp(unsigned int id) static unsigned int steal_all_contexts(void) { struct mm_struct *mm; +#ifdef CONFIG_SMP int cpu = smp_processor_id(); +#endif unsigned int id; - for (id = first_context; id <= last_context; id++) { + for (id = FIRST_CONTEXT; id <= LAST_CONTEXT; id++) { /* Pick up the victim mm */ mm = context_mm[id]; @@ -150,22 +180,24 @@ static unsigned int steal_all_contexts(void) /* Mark this mm as having no context anymore */ mm->context.id = MMU_NO_CONTEXT; - if (id != first_context) { + if (id != FIRST_CONTEXT) { context_mm[id] = NULL; __clear_bit(id, context_map); #ifdef DEBUG_MAP_CONSISTENCY mm->context.active = 0; #endif } +#ifdef CONFIG_SMP __clear_bit(id, stale_map[cpu]); +#endif } /* Flush the TLB for all contexts (not to be used on SMP) */ _tlbil_all(); - nr_free_contexts = last_context - first_context; + nr_free_contexts = LAST_CONTEXT - FIRST_CONTEXT; - return first_context; + return FIRST_CONTEXT; } /* Note that this will also be called on SMP if all other CPUs are @@ -176,7 +208,9 @@ static unsigned int steal_all_contexts(void) static unsigned int steal_context_up(unsigned int id) { struct mm_struct *mm; +#ifdef CONFIG_SMP int cpu = smp_processor_id(); +#endif /* Pick up the victim mm */ mm = context_mm[id]; @@ -190,7 +224,9 @@ static unsigned int steal_context_up(unsigned int id) mm->context.id = MMU_NO_CONTEXT; /* XXX This clear should ultimately be part of local_flush_tlb_mm */ +#ifdef CONFIG_SMP __clear_bit(id, stale_map[cpu]); +#endif return id; } @@ -201,7 +237,7 @@ static void context_check_map(void) unsigned int id, nrf, nact; nrf = nact = 0; - for (id = first_context; id <= last_context; id++) { + for (id = FIRST_CONTEXT; id <= LAST_CONTEXT; id++) { int used = test_bit(id, context_map); if (!used) nrf++; @@ -219,7 +255,7 @@ static void context_check_map(void) if (nact > num_online_cpus()) pr_err("MMU: More active contexts than CPUs ! (%d vs %d)\n", nact, num_online_cpus()); - if (first_context > 0 && !test_bit(0, context_map)) + if (FIRST_CONTEXT > 0 && !test_bit(0, context_map)) pr_err("MMU: Context 0 has been freed !!!\n"); } #else @@ -229,7 +265,10 @@ static void context_check_map(void) { } void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next, struct task_struct *tsk) { - unsigned int i, id, cpu = smp_processor_id(); + unsigned int id; +#ifdef CONFIG_SMP + unsigned int i, cpu = smp_processor_id(); +#endif unsigned long *map; /* No lockless fast path .. yet */ @@ -263,8 +302,8 @@ void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next, /* We really don't have a context, let's try to acquire one */ id = next_context; - if (id > last_context) - id = first_context; + if (id > LAST_CONTEXT) + id = FIRST_CONTEXT; map = context_map; /* No more free contexts, let's try to steal one */ @@ -277,7 +316,7 @@ void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next, goto stolen; } #endif /* CONFIG_SMP */ - if (no_selective_tlbil) + if (IS_ENABLED(CONFIG_PPC_8xx)) id = steal_all_contexts(); else id = steal_context_up(id); @@ -287,9 +326,9 @@ void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next, /* We know there's at least one free context, try to find it */ while (__test_and_set_bit(id, map)) { - id = find_next_zero_bit(map, last_context+1, id); - if (id > last_context) - id = first_context; + id = find_next_zero_bit(map, LAST_CONTEXT+1, id); + if (id > LAST_CONTEXT) + id = FIRST_CONTEXT; } stolen: next_context = id + 1; @@ -303,6 +342,7 @@ void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next, /* If that context got marked stale on this CPU, then flush the * local TLB for it and unmark it before we use it */ +#ifdef CONFIG_SMP if (test_bit(id, stale_map[cpu])) { pr_hardcont(" | stale flush %d [%d..%d]", id, cpu_first_thread_sibling(cpu), @@ -317,6 +357,7 @@ void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next, __clear_bit(id, stale_map[i]); } } +#endif /* Flick the MMU and release lock */ pr_hardcont(" -> %d\n", id); @@ -417,52 +458,12 @@ void __init mmu_context_init(void) */ init_mm.context.active = NR_CPUS; - /* - * The MPC8xx has only 16 contexts. We rotate through them on each - * task switch. A better way would be to keep track of tasks that - * own contexts, and implement an LRU usage. That way very active - * tasks don't always have to pay the TLB reload overhead. The - * kernel pages are mapped shared, so the kernel can run on behalf - * of any task that makes a kernel entry. Shared does not mean they - * are not protected, just that the ASID comparison is not performed. - * -- Dan - * - * The IBM4xx has 256 contexts, so we can just rotate through these - * as a way of "switching" contexts. If the TID of the TLB is zero, - * the PID/TID comparison is disabled, so we can use a TID of zero - * to represent all kernel pages as shared among all contexts. - * -- Dan - * - * The IBM 47x core supports 16-bit PIDs, thus 65535 contexts. We - * should normally never have to steal though the facility is - * present if needed. - * -- BenH - */ - if (mmu_has_feature(MMU_FTR_TYPE_8xx)) { - first_context = 1; - last_context = 16; - no_selective_tlbil = true; - } else if (mmu_has_feature(MMU_FTR_TYPE_47x)) { - first_context = 1; - last_context = 65535; - no_selective_tlbil = false; - } else { - first_context = 1; - last_context = 255; - no_selective_tlbil = false; - } - -#ifdef DEBUG_CLAMP_LAST_CONTEXT - last_context = DEBUG_CLAMP_LAST_CONTEXT; -#endif /* * Allocate the maps used by context management */ context_map = memblock_virt_alloc(CTX_MAP_SIZE, 0); - context_mm = memblock_virt_alloc(sizeof(void *) * (last_context + 1), 0); -#ifndef CONFIG_SMP - stale_map[0] = memblock_virt_alloc(CTX_MAP_SIZE, 0); -#else + context_mm = memblock_virt_alloc(sizeof(void *) * (LAST_CONTEXT + 1), 0); +#ifdef CONFIG_SMP stale_map[boot_cpuid] = memblock_virt_alloc(CTX_MAP_SIZE, 0); cpuhp_setup_state_nocalls(CPUHP_POWERPC_MMU_CTX_PREPARE, @@ -472,17 +473,17 @@ void __init mmu_context_init(void) printk(KERN_INFO "MMU: Allocated %zu bytes of context maps for %d contexts\n", - 2 * CTX_MAP_SIZE + (sizeof(void *) * (last_context + 1)), - last_context - first_context + 1); + 2 * CTX_MAP_SIZE + (sizeof(void *) * (LAST_CONTEXT + 1)), + LAST_CONTEXT - FIRST_CONTEXT + 1); /* * Some processors have too few contexts to reserve one for * init_mm, and require using context 0 for a normal task. * Other processors reserve the use of context zero for the kernel. - * This code assumes first_context < 32. + * This code assumes FIRST_CONTEXT < 32. */ - context_map[0] = (1 << first_context) - 1; - next_context = first_context; - nr_free_contexts = last_context - first_context + 1; + context_map[0] = (1 << FIRST_CONTEXT) - 1; + next_context = FIRST_CONTEXT; + nr_free_contexts = LAST_CONTEXT - FIRST_CONTEXT + 1; } diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c index 57a5029b4521..0c7e05d89244 100644 --- a/arch/powerpc/mm/numa.c +++ b/arch/powerpc/mm/numa.c @@ -1316,7 +1316,7 @@ int numa_update_cpu_topology(bool cpus_locked) if (!weight) return 0; - updates = kzalloc(weight * (sizeof(*updates)), GFP_KERNEL); + updates = kcalloc(weight, sizeof(*updates), GFP_KERNEL); if (!updates) return 0; diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c index 518518fb7c45..c1f4ca45c93a 100644 --- a/arch/powerpc/mm/pgtable-book3s64.c +++ b/arch/powerpc/mm/pgtable-book3s64.c @@ -9,14 +9,22 @@ #include #include +#include #include #include #include +#include +#include #include "mmu_decl.h" #include +unsigned long __pmd_frag_nr; +EXPORT_SYMBOL(__pmd_frag_nr); +unsigned long __pmd_frag_size_shift; +EXPORT_SYMBOL(__pmd_frag_size_shift); + int (*register_process_table)(unsigned long base, unsigned long page_size, unsigned long tbl_size); @@ -34,13 +42,16 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, int changed; #ifdef CONFIG_DEBUG_VM WARN_ON(!pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp)); - assert_spin_locked(&vma->vm_mm->page_table_lock); + assert_spin_locked(pmd_lockptr(vma->vm_mm, pmdp)); #endif changed = !pmd_same(*(pmdp), entry); if (changed) { - __ptep_set_access_flags(vma->vm_mm, pmdp_ptep(pmdp), - pmd_pte(entry), address); - flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE); + /* + * We can use MMU_PAGE_2M here, because only radix + * path look at the psize. + */ + __ptep_set_access_flags(vma, pmdp_ptep(pmdp), + pmd_pte(entry), address, MMU_PAGE_2M); } return changed; } @@ -59,7 +70,7 @@ void set_pmd_at(struct mm_struct *mm, unsigned long addr, { #ifdef CONFIG_DEBUG_VM WARN_ON(pte_present(pmd_pte(*pmdp)) && !pte_protnone(pmd_pte(*pmdp))); - assert_spin_locked(&mm->page_table_lock); + assert_spin_locked(pmd_lockptr(mm, pmdp)); WARN_ON(!(pmd_trans_huge(pmd) || pmd_devmap(pmd))); #endif trace_hugepage_set_pmd(addr, pmd_val(pmd)); @@ -141,7 +152,8 @@ pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd) { - return; + if (radix_enabled()) + prefetch((void *)addr); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ @@ -171,3 +183,258 @@ int __meminit remove_section_mapping(unsigned long start, unsigned long end) return hash__remove_section_mapping(start, end); } #endif /* CONFIG_MEMORY_HOTPLUG */ + +void __init mmu_partition_table_init(void) +{ + unsigned long patb_size = 1UL << PATB_SIZE_SHIFT; + unsigned long ptcr; + + BUILD_BUG_ON_MSG((PATB_SIZE_SHIFT > 36), "Partition table size too large."); + partition_tb = __va(memblock_alloc_base(patb_size, patb_size, + MEMBLOCK_ALLOC_ANYWHERE)); + + /* Initialize the Partition Table with no entries */ + memset((void *)partition_tb, 0, patb_size); + + /* + * update partition table control register, + * 64 K size. + */ + ptcr = __pa(partition_tb) | (PATB_SIZE_SHIFT - 12); + mtspr(SPRN_PTCR, ptcr); + powernv_set_nmmu_ptcr(ptcr); +} + +void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0, + unsigned long dw1) +{ + unsigned long old = be64_to_cpu(partition_tb[lpid].patb0); + + partition_tb[lpid].patb0 = cpu_to_be64(dw0); + partition_tb[lpid].patb1 = cpu_to_be64(dw1); + + /* + * Global flush of TLBs and partition table caches for this lpid. + * The type of flush (hash or radix) depends on what the previous + * use of this partition ID was, not the new use. + */ + asm volatile("ptesync" : : : "memory"); + if (old & PATB_HR) { + asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : : + "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); + asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : : + "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); + trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1); + } else { + asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : : + "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); + trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0); + } + /* do we need fixup here ?*/ + asm volatile("eieio; tlbsync; ptesync" : : : "memory"); +} +EXPORT_SYMBOL_GPL(mmu_partition_table_set_entry); + +static pmd_t *get_pmd_from_cache(struct mm_struct *mm) +{ + void *pmd_frag, *ret; + + spin_lock(&mm->page_table_lock); + ret = mm->context.pmd_frag; + if (ret) { + pmd_frag = ret + PMD_FRAG_SIZE; + /* + * If we have taken up all the fragments mark PTE page NULL + */ + if (((unsigned long)pmd_frag & ~PAGE_MASK) == 0) + pmd_frag = NULL; + mm->context.pmd_frag = pmd_frag; + } + spin_unlock(&mm->page_table_lock); + return (pmd_t *)ret; +} + +static pmd_t *__alloc_for_pmdcache(struct mm_struct *mm) +{ + void *ret = NULL; + struct page *page; + gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO; + + if (mm == &init_mm) + gfp &= ~__GFP_ACCOUNT; + page = alloc_page(gfp); + if (!page) + return NULL; + if (!pgtable_pmd_page_ctor(page)) { + __free_pages(page, 0); + return NULL; + } + + ret = page_address(page); + /* + * if we support only one fragment just return the + * allocated page. + */ + if (PMD_FRAG_NR == 1) + return ret; + + spin_lock(&mm->page_table_lock); + /* + * If we find pgtable_page set, we return + * the allocated page with single fragement + * count. + */ + if (likely(!mm->context.pmd_frag)) { + set_page_count(page, PMD_FRAG_NR); + mm->context.pmd_frag = ret + PMD_FRAG_SIZE; + } + spin_unlock(&mm->page_table_lock); + + return (pmd_t *)ret; +} + +pmd_t *pmd_fragment_alloc(struct mm_struct *mm, unsigned long vmaddr) +{ + pmd_t *pmd; + + pmd = get_pmd_from_cache(mm); + if (pmd) + return pmd; + + return __alloc_for_pmdcache(mm); +} + +void pmd_fragment_free(unsigned long *pmd) +{ + struct page *page = virt_to_page(pmd); + + if (put_page_testzero(page)) { + pgtable_pmd_page_dtor(page); + free_unref_page(page); + } +} + +static pte_t *get_pte_from_cache(struct mm_struct *mm) +{ + void *pte_frag, *ret; + + spin_lock(&mm->page_table_lock); + ret = mm->context.pte_frag; + if (ret) { + pte_frag = ret + PTE_FRAG_SIZE; + /* + * If we have taken up all the fragments mark PTE page NULL + */ + if (((unsigned long)pte_frag & ~PAGE_MASK) == 0) + pte_frag = NULL; + mm->context.pte_frag = pte_frag; + } + spin_unlock(&mm->page_table_lock); + return (pte_t *)ret; +} + +static pte_t *__alloc_for_ptecache(struct mm_struct *mm, int kernel) +{ + void *ret = NULL; + struct page *page; + + if (!kernel) { + page = alloc_page(PGALLOC_GFP | __GFP_ACCOUNT); + if (!page) + return NULL; + if (!pgtable_page_ctor(page)) { + __free_page(page); + return NULL; + } + } else { + page = alloc_page(PGALLOC_GFP); + if (!page) + return NULL; + } + + + ret = page_address(page); + /* + * if we support only one fragment just return the + * allocated page. + */ + if (PTE_FRAG_NR == 1) + return ret; + spin_lock(&mm->page_table_lock); + /* + * If we find pgtable_page set, we return + * the allocated page with single fragement + * count. + */ + if (likely(!mm->context.pte_frag)) { + set_page_count(page, PTE_FRAG_NR); + mm->context.pte_frag = ret + PTE_FRAG_SIZE; + } + spin_unlock(&mm->page_table_lock); + + return (pte_t *)ret; +} + +pte_t *pte_fragment_alloc(struct mm_struct *mm, unsigned long vmaddr, int kernel) +{ + pte_t *pte; + + pte = get_pte_from_cache(mm); + if (pte) + return pte; + + return __alloc_for_ptecache(mm, kernel); +} + +void pte_fragment_free(unsigned long *table, int kernel) +{ + struct page *page = virt_to_page(table); + + if (put_page_testzero(page)) { + if (!kernel) + pgtable_page_dtor(page); + free_unref_page(page); + } +} + +static inline void pgtable_free(void *table, int index) +{ + switch (index) { + case PTE_INDEX: + pte_fragment_free(table, 0); + break; + case PMD_INDEX: + pmd_fragment_free(table); + break; + case PUD_INDEX: + kmem_cache_free(PGT_CACHE(PUD_CACHE_INDEX), table); + break; + /* We don't free pgd table via RCU callback */ + default: + BUG(); + } +} + +#ifdef CONFIG_SMP +void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int index) +{ + unsigned long pgf = (unsigned long)table; + + BUG_ON(index > MAX_PGTABLE_INDEX_SIZE); + pgf |= index; + tlb_remove_table(tlb, (void *)pgf); +} + +void __tlb_remove_table(void *_table) +{ + void *table = (void *)((unsigned long)_table & ~MAX_PGTABLE_INDEX_SIZE); + unsigned int index = (unsigned long)_table & MAX_PGTABLE_INDEX_SIZE; + + return pgtable_free(table, index); +} +#else +void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int index) +{ + return pgtable_free(table, index); +} +#endif diff --git a/arch/powerpc/mm/pgtable-hash64.c b/arch/powerpc/mm/pgtable-hash64.c index 199bfda5f0d9..692bfc9e372c 100644 --- a/arch/powerpc/mm/pgtable-hash64.c +++ b/arch/powerpc/mm/pgtable-hash64.c @@ -193,7 +193,7 @@ unsigned long hash__pmd_hugepage_update(struct mm_struct *mm, unsigned long addr #ifdef CONFIG_DEBUG_VM WARN_ON(!hash__pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp)); - assert_spin_locked(&mm->page_table_lock); + assert_spin_locked(pmd_lockptr(mm, pmdp)); #endif __asm__ __volatile__( @@ -265,7 +265,8 @@ void hash__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, pgtable_t pgtable) { pgtable_t *pgtable_slot; - assert_spin_locked(&mm->page_table_lock); + + assert_spin_locked(pmd_lockptr(mm, pmdp)); /* * we store the pgtable in the second half of PMD */ @@ -285,7 +286,8 @@ pgtable_t hash__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) pgtable_t pgtable; pgtable_t *pgtable_slot; - assert_spin_locked(&mm->page_table_lock); + assert_spin_locked(pmd_lockptr(mm, pmdp)); + pgtable_slot = (pgtable_t *)pmdp + PTRS_PER_PMD; pgtable = *pgtable_slot; /* diff --git a/arch/powerpc/mm/pgtable-radix.c b/arch/powerpc/mm/pgtable-radix.c index f1891e215e39..96f68c5aa1f5 100644 --- a/arch/powerpc/mm/pgtable-radix.c +++ b/arch/powerpc/mm/pgtable-radix.c @@ -617,7 +617,6 @@ void __init radix__early_init_mmu(void) __pud_index_size = RADIX_PUD_INDEX_SIZE; __pgd_index_size = RADIX_PGD_INDEX_SIZE; __pud_cache_index = RADIX_PUD_INDEX_SIZE; - __pmd_cache_index = RADIX_PMD_INDEX_SIZE; __pte_table_size = RADIX_PTE_TABLE_SIZE; __pmd_table_size = RADIX_PMD_TABLE_SIZE; __pud_table_size = RADIX_PUD_TABLE_SIZE; @@ -640,6 +639,8 @@ void __init radix__early_init_mmu(void) #endif __pte_frag_nr = RADIX_PTE_FRAG_NR; __pte_frag_size_shift = RADIX_PTE_FRAG_SIZE_SHIFT; + __pmd_frag_nr = RADIX_PMD_FRAG_NR; + __pmd_frag_size_shift = RADIX_PMD_FRAG_SIZE_SHIFT; if (!firmware_has_feature(FW_FEATURE_LPAR)) { radix_init_native(); @@ -975,7 +976,7 @@ unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long add #ifdef CONFIG_DEBUG_VM WARN_ON(!radix__pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp)); - assert_spin_locked(&mm->page_table_lock); + assert_spin_locked(pmd_lockptr(mm, pmdp)); #endif old = radix__pte_update(mm, addr, (pte_t *)pmdp, clr, set, 1); @@ -1083,3 +1084,36 @@ int radix__has_transparent_hugepage(void) return 0; } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, + pte_t entry, unsigned long address, int psize) +{ + struct mm_struct *mm = vma->vm_mm; + unsigned long set = pte_val(entry) & (_PAGE_DIRTY | _PAGE_ACCESSED | + _PAGE_RW | _PAGE_EXEC); + /* + * To avoid NMMU hang while relaxing access, we need mark + * the pte invalid in between. + */ + if (cpu_has_feature(CPU_FTR_POWER9_DD1) || + atomic_read(&mm->context.copros) > 0) { + unsigned long old_pte, new_pte; + + old_pte = __radix_pte_update(ptep, ~0, 0); + /* + * new value of pte + */ + new_pte = old_pte | set; + radix__flush_tlb_page_psize(mm, address, psize); + __radix_pte_update(ptep, 0, new_pte); + } else { + __radix_pte_update(ptep, 0, set); + /* + * Book3S does not require a TLB flush when relaxing access + * restrictions when the address space is not attached to a + * NMMU, because the core MMU will reload the pte after taking + * an access fault, which is defined by the architectue. + */ + } + /* See ptesync comment in radix__set_pte_at */ +} diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 9f361ae571e9..d71c7777669c 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -221,14 +221,55 @@ int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, entry = set_access_flags_filter(entry, vma, dirty); changed = !pte_same(*(ptep), entry); if (changed) { - if (!is_vm_hugetlb_page(vma)) - assert_pte_locked(vma->vm_mm, address); - __ptep_set_access_flags(vma->vm_mm, ptep, entry, address); - flush_tlb_page(vma, address); + assert_pte_locked(vma->vm_mm, address); + __ptep_set_access_flags(vma, ptep, entry, + address, mmu_virtual_psize); } return changed; } +#ifdef CONFIG_HUGETLB_PAGE +extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, int dirty) +{ +#ifdef HUGETLB_NEED_PRELOAD + /* + * The "return 1" forces a call of update_mmu_cache, which will write a + * TLB entry. Without this, platforms that don't do a write of the TLB + * entry in the TLB miss handler asm will fault ad infinitum. + */ + ptep_set_access_flags(vma, addr, ptep, pte, dirty); + return 1; +#else + int changed, psize; + + pte = set_access_flags_filter(pte, vma, dirty); + changed = !pte_same(*(ptep), pte); + if (changed) { + +#ifdef CONFIG_PPC_BOOK3S_64 + struct hstate *h = hstate_vma(vma); + + psize = hstate_get_psize(h); +#ifdef CONFIG_DEBUG_VM + assert_spin_locked(huge_pte_lockptr(h, vma->vm_mm, ptep)); +#endif + +#else + /* + * Not used on non book3s64 platforms. But 8xx + * can possibly use tsize derived from hstate. + */ + psize = 0; +#endif + __ptep_set_access_flags(vma, ptep, pte, addr, psize); + } + return changed; +#endif +} +#endif /* CONFIG_HUGETLB_PAGE */ + #ifdef CONFIG_DEBUG_VM void assert_pte_locked(struct mm_struct *mm, unsigned long addr) { diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c index 9bf659d5078c..53e9eeecd5d4 100644 --- a/arch/powerpc/mm/pgtable_64.c +++ b/arch/powerpc/mm/pgtable_64.c @@ -33,7 +33,6 @@ #include #include #include -#include #include #include @@ -47,13 +46,11 @@ #include #include #include -#include #include #include #include #include #include -#include #include "mmu_decl.h" @@ -75,8 +72,6 @@ unsigned long __pud_index_size; EXPORT_SYMBOL(__pud_index_size); unsigned long __pgd_index_size; EXPORT_SYMBOL(__pgd_index_size); -unsigned long __pmd_cache_index; -EXPORT_SYMBOL(__pmd_cache_index); unsigned long __pud_cache_index; EXPORT_SYMBOL(__pud_cache_index); unsigned long __pte_table_size; @@ -316,172 +311,6 @@ struct page *pmd_page(pmd_t pmd) return virt_to_page(pmd_page_vaddr(pmd)); } -#ifdef CONFIG_PPC_64K_PAGES -static pte_t *get_from_cache(struct mm_struct *mm) -{ - void *pte_frag, *ret; - - spin_lock(&mm->page_table_lock); - ret = mm->context.pte_frag; - if (ret) { - pte_frag = ret + PTE_FRAG_SIZE; - /* - * If we have taken up all the fragments mark PTE page NULL - */ - if (((unsigned long)pte_frag & ~PAGE_MASK) == 0) - pte_frag = NULL; - mm->context.pte_frag = pte_frag; - } - spin_unlock(&mm->page_table_lock); - return (pte_t *)ret; -} - -static pte_t *__alloc_for_cache(struct mm_struct *mm, int kernel) -{ - void *ret = NULL; - struct page *page; - - if (!kernel) { - page = alloc_page(PGALLOC_GFP | __GFP_ACCOUNT); - if (!page) - return NULL; - if (!pgtable_page_ctor(page)) { - __free_page(page); - return NULL; - } - } else { - page = alloc_page(PGALLOC_GFP); - if (!page) - return NULL; - } - - ret = page_address(page); - spin_lock(&mm->page_table_lock); - /* - * If we find pgtable_page set, we return - * the allocated page with single fragement - * count. - */ - if (likely(!mm->context.pte_frag)) { - set_page_count(page, PTE_FRAG_NR); - mm->context.pte_frag = ret + PTE_FRAG_SIZE; - } - spin_unlock(&mm->page_table_lock); - - return (pte_t *)ret; -} - -pte_t *pte_fragment_alloc(struct mm_struct *mm, unsigned long vmaddr, int kernel) -{ - pte_t *pte; - - pte = get_from_cache(mm); - if (pte) - return pte; - - return __alloc_for_cache(mm, kernel); -} -#endif /* CONFIG_PPC_64K_PAGES */ - -void pte_fragment_free(unsigned long *table, int kernel) -{ - struct page *page = virt_to_page(table); - if (put_page_testzero(page)) { - if (!kernel) - pgtable_page_dtor(page); - free_unref_page(page); - } -} - -#ifdef CONFIG_SMP -void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift) -{ - unsigned long pgf = (unsigned long)table; - - BUG_ON(shift > MAX_PGTABLE_INDEX_SIZE); - pgf |= shift; - tlb_remove_table(tlb, (void *)pgf); -} - -void __tlb_remove_table(void *_table) -{ - void *table = (void *)((unsigned long)_table & ~MAX_PGTABLE_INDEX_SIZE); - unsigned shift = (unsigned long)_table & MAX_PGTABLE_INDEX_SIZE; - - if (!shift) - /* PTE page needs special handling */ - pte_fragment_free(table, 0); - else { - BUG_ON(shift > MAX_PGTABLE_INDEX_SIZE); - kmem_cache_free(PGT_CACHE(shift), table); - } -} -#else -void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift) -{ - if (!shift) { - /* PTE page needs special handling */ - pte_fragment_free(table, 0); - } else { - BUG_ON(shift > MAX_PGTABLE_INDEX_SIZE); - kmem_cache_free(PGT_CACHE(shift), table); - } -} -#endif - -#ifdef CONFIG_PPC_BOOK3S_64 -void __init mmu_partition_table_init(void) -{ - unsigned long patb_size = 1UL << PATB_SIZE_SHIFT; - unsigned long ptcr; - - BUILD_BUG_ON_MSG((PATB_SIZE_SHIFT > 36), "Partition table size too large."); - partition_tb = __va(memblock_alloc_base(patb_size, patb_size, - MEMBLOCK_ALLOC_ANYWHERE)); - - /* Initialize the Partition Table with no entries */ - memset((void *)partition_tb, 0, patb_size); - - /* - * update partition table control register, - * 64 K size. - */ - ptcr = __pa(partition_tb) | (PATB_SIZE_SHIFT - 12); - mtspr(SPRN_PTCR, ptcr); - powernv_set_nmmu_ptcr(ptcr); -} - -void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0, - unsigned long dw1) -{ - unsigned long old = be64_to_cpu(partition_tb[lpid].patb0); - - partition_tb[lpid].patb0 = cpu_to_be64(dw0); - partition_tb[lpid].patb1 = cpu_to_be64(dw1); - - /* - * Global flush of TLBs and partition table caches for this lpid. - * The type of flush (hash or radix) depends on what the previous - * use of this partition ID was, not the new use. - */ - asm volatile("ptesync" : : : "memory"); - if (old & PATB_HR) { - asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : : - "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); - asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : : - "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); - trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1); - } else { - asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : : - "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid)); - trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0); - } - /* do we need fixup here ?*/ - asm volatile("eieio; tlbsync; ptesync" : : : "memory"); -} -EXPORT_SYMBOL_GPL(mmu_partition_table_set_entry); -#endif /* CONFIG_PPC_BOOK3S_64 */ - #ifdef CONFIG_STRICT_KERNEL_RWX void mark_rodata_ro(void) { diff --git a/arch/powerpc/mm/pkeys.c b/arch/powerpc/mm/pkeys.c index 0eafdf01edc7..e6f500fabf5e 100644 --- a/arch/powerpc/mm/pkeys.c +++ b/arch/powerpc/mm/pkeys.c @@ -383,9 +383,9 @@ int __arch_override_mprotect_pkey(struct vm_area_struct *vma, int prot, { /* * If the currently associated pkey is execute-only, but the requested - * protection requires read or write, move it back to the default pkey. + * protection is not execute-only, move it back to the default pkey. */ - if (vma_is_pkey_exec_only(vma) && (prot & (PROT_READ | PROT_WRITE))) + if (vma_is_pkey_exec_only(vma) && (prot != PROT_EXEC)) return 0; /* diff --git a/arch/powerpc/mm/ppc_mmu_32.c b/arch/powerpc/mm/ppc_mmu_32.c index 2a049fb8523d..bea6c544e38f 100644 --- a/arch/powerpc/mm/ppc_mmu_32.c +++ b/arch/powerpc/mm/ppc_mmu_32.c @@ -167,7 +167,7 @@ void hash_preload(struct mm_struct *mm, unsigned long ea, { pmd_t *pmd; - if (Hash == 0) + if (!Hash) return; pmd = pmd_offset(pud_offset(pgd_offset(mm, ea), ea), ea); if (!pmd_none(*pmd)) diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c index 66577cc66dc9..cb796724a6fc 100644 --- a/arch/powerpc/mm/slb.c +++ b/arch/powerpc/mm/slb.c @@ -63,14 +63,14 @@ static inline void slb_shadow_update(unsigned long ea, int ssize, * updating it. No write barriers are needed here, provided * we only update the current CPU's SLB shadow buffer. */ - p->save_area[index].esid = 0; - p->save_area[index].vsid = cpu_to_be64(mk_vsid_data(ea, ssize, flags)); - p->save_area[index].esid = cpu_to_be64(mk_esid_data(ea, ssize, index)); + WRITE_ONCE(p->save_area[index].esid, 0); + WRITE_ONCE(p->save_area[index].vsid, cpu_to_be64(mk_vsid_data(ea, ssize, flags))); + WRITE_ONCE(p->save_area[index].esid, cpu_to_be64(mk_esid_data(ea, ssize, index))); } static inline void slb_shadow_clear(enum slb_index index) { - get_slb_shadow()->save_area[index].esid = 0; + WRITE_ONCE(get_slb_shadow()->save_area[index].esid, 0); } static inline void create_shadowed_slbe(unsigned long ea, int ssize, @@ -352,6 +352,14 @@ static void insert_slb_entry(unsigned long vsid, unsigned long ea, /* * We are irq disabled, hence should be safe to access PACA. */ + VM_WARN_ON(!irqs_disabled()); + + /* + * We can't take a PMU exception in the following code, so hard + * disable interrupts. + */ + hard_irq_disable(); + index = get_paca()->stab_rr; /* @@ -369,6 +377,11 @@ static void insert_slb_entry(unsigned long vsid, unsigned long ea, ((unsigned long) ssize << SLB_VSID_SSIZE_SHIFT); esid_data = mk_esid_data(ea, ssize, index); + /* + * No need for an isync before or after this slbmte. The exception + * we enter with and the rfid we exit with are context synchronizing. + * Also we only handle user segments here. + */ asm volatile("slbmte %0, %1" : : "r" (vsid_data), "r" (esid_data) : "memory"); diff --git a/arch/powerpc/mm/subpage-prot.c b/arch/powerpc/mm/subpage-prot.c index f14a07c2fb90..75cb646a79c3 100644 --- a/arch/powerpc/mm/subpage-prot.c +++ b/arch/powerpc/mm/subpage-prot.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include @@ -185,7 +186,11 @@ static void subpage_mark_vma_nohuge(struct mm_struct *mm, unsigned long addr, * in a 2-bit field won't allow writes to a page that is otherwise * write-protected. */ -long sys_subpage_prot(unsigned long addr, unsigned long len, u32 __user *map) +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpragmas" +#pragma GCC diagnostic ignored "-Wattribute-alias" +SYSCALL_DEFINE3(subpage_prot, unsigned long, addr, + unsigned long, len, u32 __user *, map) { struct mm_struct *mm = current->mm; struct subpage_prot_table *spt = &mm->context.spt; @@ -267,3 +272,4 @@ long sys_subpage_prot(unsigned long addr, unsigned long len, u32 __user *map) up_write(&mm->mmap_sem); return err; } +#pragma GCC diagnostic pop diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index a5d7309c2d05..67a6e86d3e7e 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -12,6 +12,8 @@ #include #include #include +#include +#include #include #include @@ -118,6 +120,53 @@ static inline void __tlbie_pid(unsigned long pid, unsigned long ric) trace_tlbie(0, 0, rb, rs, ric, prs, r); } +static inline void __tlbiel_lpid(unsigned long lpid, int set, + unsigned long ric) +{ + unsigned long rb,rs,prs,r; + + rb = PPC_BIT(52); /* IS = 2 */ + rb |= set << PPC_BITLSHIFT(51); + rs = 0; /* LPID comes from LPIDR */ + prs = 0; /* partition scoped */ + r = 1; /* radix format */ + + asm volatile(PPC_TLBIEL(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); + trace_tlbie(lpid, 1, rb, rs, ric, prs, r); +} + +static inline void __tlbie_lpid(unsigned long lpid, unsigned long ric) +{ + unsigned long rb,rs,prs,r; + + rb = PPC_BIT(52); /* IS = 2 */ + rs = lpid; + prs = 0; /* partition scoped */ + r = 1; /* radix format */ + + asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); + trace_tlbie(lpid, 0, rb, rs, ric, prs, r); +} + +static inline void __tlbiel_lpid_guest(unsigned long lpid, int set, + unsigned long ric) +{ + unsigned long rb,rs,prs,r; + + rb = PPC_BIT(52); /* IS = 2 */ + rb |= set << PPC_BITLSHIFT(51); + rs = 0; /* LPID comes from LPIDR */ + prs = 1; /* process scoped */ + r = 1; /* radix format */ + + asm volatile(PPC_TLBIEL(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); + trace_tlbie(lpid, 1, rb, rs, ric, prs, r); +} + + static inline void __tlbiel_va(unsigned long va, unsigned long pid, unsigned long ap, unsigned long ric) { @@ -150,6 +199,22 @@ static inline void __tlbie_va(unsigned long va, unsigned long pid, trace_tlbie(0, 0, rb, rs, ric, prs, r); } +static inline void __tlbie_lpid_va(unsigned long va, unsigned long lpid, + unsigned long ap, unsigned long ric) +{ + unsigned long rb,rs,prs,r; + + rb = va & ~(PPC_BITMASK(52, 63)); + rb |= ap << PPC_BITLSHIFT(58); + rs = lpid; + prs = 0; /* partition scoped */ + r = 1; /* radix format */ + + asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); + trace_tlbie(lpid, 0, rb, rs, ric, prs, r); +} + static inline void fixup_tlbie(void) { unsigned long pid = 0; @@ -161,6 +226,16 @@ static inline void fixup_tlbie(void) } } +static inline void fixup_tlbie_lpid(unsigned long lpid) +{ + unsigned long va = ((1UL << 52) - 1); + + if (cpu_has_feature(CPU_FTR_P9_TLBIE_BUG)) { + asm volatile("ptesync": : :"memory"); + __tlbie_lpid_va(va, lpid, mmu_get_ap(MMU_PAGE_64K), RIC_FLUSH_TLB); + } +} + /* * We use 128 set in radix mode and 256 set in hpt mode. */ @@ -214,6 +289,86 @@ static inline void _tlbie_pid(unsigned long pid, unsigned long ric) asm volatile("eieio; tlbsync; ptesync": : :"memory"); } +static inline void _tlbiel_lpid(unsigned long lpid, unsigned long ric) +{ + int set; + + VM_BUG_ON(mfspr(SPRN_LPID) != lpid); + + asm volatile("ptesync": : :"memory"); + + /* + * Flush the first set of the TLB, and if we're doing a RIC_FLUSH_ALL, + * also flush the entire Page Walk Cache. + */ + __tlbiel_lpid(lpid, 0, ric); + + /* For PWC, only one flush is needed */ + if (ric == RIC_FLUSH_PWC) { + asm volatile("ptesync": : :"memory"); + return; + } + + /* For the remaining sets, just flush the TLB */ + for (set = 1; set < POWER9_TLB_SETS_RADIX ; set++) + __tlbiel_lpid(lpid, set, RIC_FLUSH_TLB); + + asm volatile("ptesync": : :"memory"); + asm volatile(PPC_INVALIDATE_ERAT "; isync" : : :"memory"); +} + +static inline void _tlbie_lpid(unsigned long lpid, unsigned long ric) +{ + asm volatile("ptesync": : :"memory"); + + /* + * Workaround the fact that the "ric" argument to __tlbie_pid + * must be a compile-time contraint to match the "i" constraint + * in the asm statement. + */ + switch (ric) { + case RIC_FLUSH_TLB: + __tlbie_lpid(lpid, RIC_FLUSH_TLB); + break; + case RIC_FLUSH_PWC: + __tlbie_lpid(lpid, RIC_FLUSH_PWC); + break; + case RIC_FLUSH_ALL: + default: + __tlbie_lpid(lpid, RIC_FLUSH_ALL); + } + fixup_tlbie_lpid(lpid); + asm volatile("eieio; tlbsync; ptesync": : :"memory"); +} + +static inline void _tlbiel_lpid_guest(unsigned long lpid, unsigned long ric) +{ + int set; + + VM_BUG_ON(mfspr(SPRN_LPID) != lpid); + + asm volatile("ptesync": : :"memory"); + + /* + * Flush the first set of the TLB, and if we're doing a RIC_FLUSH_ALL, + * also flush the entire Page Walk Cache. + */ + __tlbiel_lpid_guest(lpid, 0, ric); + + /* For PWC, only one flush is needed */ + if (ric == RIC_FLUSH_PWC) { + asm volatile("ptesync": : :"memory"); + return; + } + + /* For the remaining sets, just flush the TLB */ + for (set = 1; set < POWER9_TLB_SETS_RADIX ; set++) + __tlbiel_lpid_guest(lpid, set, RIC_FLUSH_TLB); + + asm volatile("ptesync": : :"memory"); +} + + static inline void __tlbiel_va_range(unsigned long start, unsigned long end, unsigned long pid, unsigned long page_size, unsigned long psize) @@ -268,6 +423,17 @@ static inline void _tlbie_va(unsigned long va, unsigned long pid, asm volatile("eieio; tlbsync; ptesync": : :"memory"); } +static inline void _tlbie_lpid_va(unsigned long va, unsigned long lpid, + unsigned long psize, unsigned long ric) +{ + unsigned long ap = mmu_get_ap(psize); + + asm volatile("ptesync": : :"memory"); + __tlbie_lpid_va(va, lpid, ap, ric); + fixup_tlbie_lpid(lpid); + asm volatile("eieio; tlbsync; ptesync": : :"memory"); +} + static inline void _tlbie_va_range(unsigned long start, unsigned long end, unsigned long pid, unsigned long page_size, unsigned long psize, bool also_pwc) @@ -340,6 +506,15 @@ void radix__local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmadd } EXPORT_SYMBOL(radix__local_flush_tlb_page); +static bool mm_is_singlethreaded(struct mm_struct *mm) +{ + if (atomic_read(&mm->context.copros) > 0) + return false; + if (atomic_read(&mm->mm_users) <= 1 && current->mm == mm) + return true; + return false; +} + static bool mm_needs_flush_escalation(struct mm_struct *mm) { /* @@ -347,10 +522,47 @@ static bool mm_needs_flush_escalation(struct mm_struct *mm) * caching PTEs and not flushing them properly when * RIC = 0 for a PID/LPID invalidate */ - return atomic_read(&mm->context.copros) != 0; + if (atomic_read(&mm->context.copros) > 0) + return true; + return false; } #ifdef CONFIG_SMP +static void do_exit_flush_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + unsigned long pid = mm->context.id; + + if (current->mm == mm) + return; /* Local CPU */ + + if (current->active_mm == mm) { + /* + * Must be a kernel thread because sender is single-threaded. + */ + BUG_ON(current->mm); + mmgrab(&init_mm); + switch_mm(mm, &init_mm, current); + current->active_mm = &init_mm; + mmdrop(mm); + } + _tlbiel_pid(pid, RIC_FLUSH_ALL); +} + +static void exit_flush_lazy_tlbs(struct mm_struct *mm) +{ + /* + * Would be nice if this was async so it could be run in + * parallel with our local flush, but generic code does not + * give a good API for it. Could extend the generic code or + * make a special powerpc IPI for flushing TLBs. + * For now it's not too performance critical. + */ + smp_call_function_many(mm_cpumask(mm), do_exit_flush_lazy_tlb, + (void *)mm, 1); + mm_reset_thread_local(mm); +} + void radix__flush_tlb_mm(struct mm_struct *mm) { unsigned long pid; @@ -360,18 +572,30 @@ void radix__flush_tlb_mm(struct mm_struct *mm) return; preempt_disable(); + /* + * Order loads of mm_cpumask vs previous stores to clear ptes before + * the invalidate. See barrier in switch_mm_irqs_off + */ + smp_mb(); if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + exit_flush_lazy_tlbs(mm); + goto local; + } + if (mm_needs_flush_escalation(mm)) _tlbie_pid(pid, RIC_FLUSH_ALL); else _tlbie_pid(pid, RIC_FLUSH_TLB); - } else + } else { +local: _tlbiel_pid(pid, RIC_FLUSH_TLB); + } preempt_enable(); } EXPORT_SYMBOL(radix__flush_tlb_mm); -void radix__flush_all_mm(struct mm_struct *mm) +static void __flush_all_mm(struct mm_struct *mm, bool fullmm) { unsigned long pid; @@ -380,12 +604,25 @@ void radix__flush_all_mm(struct mm_struct *mm) return; preempt_disable(); - if (!mm_is_thread_local(mm)) + smp_mb(); /* see radix__flush_tlb_mm */ + if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + if (!fullmm) { + exit_flush_lazy_tlbs(mm); + goto local; + } + } _tlbie_pid(pid, RIC_FLUSH_ALL); - else + } else { +local: _tlbiel_pid(pid, RIC_FLUSH_ALL); + } preempt_enable(); } +void radix__flush_all_mm(struct mm_struct *mm) +{ + __flush_all_mm(mm, false); +} EXPORT_SYMBOL(radix__flush_all_mm); void radix__flush_tlb_pwc(struct mmu_gather *tlb, unsigned long addr) @@ -404,10 +641,17 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, return; preempt_disable(); - if (!mm_is_thread_local(mm)) + smp_mb(); /* see radix__flush_tlb_mm */ + if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + exit_flush_lazy_tlbs(mm); + goto local; + } _tlbie_va(vmaddr, pid, psize, RIC_FLUSH_TLB); - else + } else { +local: _tlbiel_va(vmaddr, pid, psize, RIC_FLUSH_TLB); + } preempt_enable(); } @@ -466,14 +710,22 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, return; preempt_disable(); - if (mm_is_thread_local(mm)) { - local = true; - full = (end == TLB_FLUSH_ALL || - nr_pages > tlb_local_single_page_flush_ceiling); - } else { + smp_mb(); /* see radix__flush_tlb_mm */ + if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + if (end != TLB_FLUSH_ALL) { + exit_flush_lazy_tlbs(mm); + goto is_local; + } + } local = false; full = (end == TLB_FLUSH_ALL || nr_pages > tlb_single_page_flush_ceiling); + } else { +is_local: + local = true; + full = (end == TLB_FLUSH_ALL || + nr_pages > tlb_local_single_page_flush_ceiling); } if (full) { @@ -534,6 +786,49 @@ static int radix_get_mmu_psize(int page_size) return psize; } +/* + * Flush partition scoped LPID address translation for all CPUs. + */ +void radix__flush_tlb_lpid_page(unsigned int lpid, + unsigned long addr, + unsigned long page_size) +{ + int psize = radix_get_mmu_psize(page_size); + + _tlbie_lpid_va(addr, lpid, psize, RIC_FLUSH_TLB); +} +EXPORT_SYMBOL_GPL(radix__flush_tlb_lpid_page); + +/* + * Flush partition scoped PWC from LPID for all CPUs. + */ +void radix__flush_pwc_lpid(unsigned int lpid) +{ + _tlbie_lpid(lpid, RIC_FLUSH_PWC); +} +EXPORT_SYMBOL_GPL(radix__flush_pwc_lpid); + +/* + * Flush partition scoped translations from LPID (=LPIDR) + */ +void radix__local_flush_tlb_lpid(unsigned int lpid) +{ + _tlbiel_lpid(lpid, RIC_FLUSH_ALL); +} +EXPORT_SYMBOL_GPL(radix__local_flush_tlb_lpid); + +/* + * Flush process scoped translations from LPID (=LPIDR). + * Important difference, the guest normally manages its own translations, + * but some cases e.g., vCPU CPU migration require KVM to flush. + */ +void radix__local_flush_tlb_lpid_guest(unsigned int lpid) +{ + _tlbiel_lpid_guest(lpid, RIC_FLUSH_ALL); +} +EXPORT_SYMBOL_GPL(radix__local_flush_tlb_lpid_guest); + + static void radix__flush_tlb_pwc_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize); @@ -551,7 +846,7 @@ void radix__tlb_flush(struct mmu_gather *tlb) * See the comment for radix in arch_exit_mmap(). */ if (tlb->fullmm) { - radix__flush_all_mm(mm); + __flush_all_mm(mm, true); } else if ( (psize = radix_get_mmu_psize(page_size)) == -1) { if (!tlb->need_flush_all) radix__flush_tlb_mm(mm); @@ -584,24 +879,33 @@ static inline void __radix__flush_tlb_range_psize(struct mm_struct *mm, return; preempt_disable(); - if (mm_is_thread_local(mm)) { - local = true; - full = (end == TLB_FLUSH_ALL || - nr_pages > tlb_local_single_page_flush_ceiling); - } else { + smp_mb(); /* see radix__flush_tlb_mm */ + if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + if (end != TLB_FLUSH_ALL) { + exit_flush_lazy_tlbs(mm); + goto is_local; + } + } local = false; full = (end == TLB_FLUSH_ALL || nr_pages > tlb_single_page_flush_ceiling); + } else { +is_local: + local = true; + full = (end == TLB_FLUSH_ALL || + nr_pages > tlb_local_single_page_flush_ceiling); } if (full) { - if (!local && mm_needs_flush_escalation(mm)) - also_pwc = true; - - if (local) + if (local) { _tlbiel_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); - else - _tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL: RIC_FLUSH_TLB); + } else { + if (mm_needs_flush_escalation(mm)) + also_pwc = true; + + _tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); + } } else { if (local) _tlbiel_va_range(start, end, pid, page_size, psize, also_pwc); @@ -642,11 +946,17 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) /* Otherwise first do the PWC, then iterate the pages. */ preempt_disable(); - - if (mm_is_thread_local(mm)) { - _tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); - } else { + smp_mb(); /* see radix__flush_tlb_mm */ + if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + exit_flush_lazy_tlbs(mm); + goto local; + } _tlbie_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); + goto local; + } else { +local: + _tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); } preempt_enable(); diff --git a/arch/powerpc/mm/tlb_hash32.c b/arch/powerpc/mm/tlb_hash32.c index 702d7689d714..cf8472cf3d59 100644 --- a/arch/powerpc/mm/tlb_hash32.c +++ b/arch/powerpc/mm/tlb_hash32.c @@ -41,7 +41,7 @@ void flush_hash_entry(struct mm_struct *mm, pte_t *ptep, unsigned long addr) { unsigned long ptephys; - if (Hash != 0) { + if (Hash) { ptephys = __pa(ptep) & PAGE_MASK; flush_hash_pages(mm->context.id, addr, ptephys, 1); } @@ -54,7 +54,7 @@ EXPORT_SYMBOL(flush_hash_entry); */ void tlb_flush(struct mmu_gather *tlb) { - if (Hash == 0) { + if (!Hash) { /* * 603 needs to flush the whole TLB here since * it doesn't use a hash table. @@ -84,7 +84,7 @@ static void flush_range(struct mm_struct *mm, unsigned long start, int count; unsigned int ctx = mm->context.id; - if (Hash == 0) { + if (!Hash) { _tlbia(); return; } @@ -124,7 +124,7 @@ void flush_tlb_mm(struct mm_struct *mm) { struct vm_area_struct *mp; - if (Hash == 0) { + if (!Hash) { _tlbia(); return; } @@ -145,7 +145,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr) struct mm_struct *mm; pmd_t *pmd; - if (Hash == 0) { + if (!Hash) { _tlbie(vmaddr); return; } diff --git a/arch/powerpc/net/Makefile b/arch/powerpc/net/Makefile index 02d369ca6a53..809f019d3cba 100644 --- a/arch/powerpc/net/Makefile +++ b/arch/powerpc/net/Makefile @@ -3,7 +3,7 @@ # Arch-specific network modules # ifeq ($(CONFIG_PPC64),y) -obj-$(CONFIG_BPF_JIT) += bpf_jit_asm64.o bpf_jit_comp64.o +obj-$(CONFIG_BPF_JIT) += bpf_jit_comp64.o else obj-$(CONFIG_BPF_JIT) += bpf_jit_asm.o bpf_jit_comp.o endif diff --git a/arch/powerpc/net/bpf_jit64.h b/arch/powerpc/net/bpf_jit64.h index 8bdef7ed28a8..3609be4692b3 100644 --- a/arch/powerpc/net/bpf_jit64.h +++ b/arch/powerpc/net/bpf_jit64.h @@ -20,7 +20,7 @@ * with our redzone usage. * * [ prev sp ] <------------- - * [ nv gpr save area ] 8*8 | + * [ nv gpr save area ] 6*8 | * [ tail_call_cnt ] 8 | * [ local_tmp_var ] 8 | * fp (r31) --> [ ebpf stack space ] upto 512 | @@ -28,8 +28,8 @@ * sp (r1) ---> [ stack pointer ] -------------- */ -/* for gpr non volatile registers BPG_REG_6 to 10, plus skb cache registers */ -#define BPF_PPC_STACK_SAVE (8*8) +/* for gpr non volatile registers BPG_REG_6 to 10 */ +#define BPF_PPC_STACK_SAVE (6*8) /* for bpf JIT code internal usage */ #define BPF_PPC_STACK_LOCALS 16 /* stack frame excluding BPF stack, ensure this is quadword aligned */ @@ -39,10 +39,8 @@ #ifndef __ASSEMBLY__ /* BPF register usage */ -#define SKB_HLEN_REG (MAX_BPF_JIT_REG + 0) -#define SKB_DATA_REG (MAX_BPF_JIT_REG + 1) -#define TMP_REG_1 (MAX_BPF_JIT_REG + 2) -#define TMP_REG_2 (MAX_BPF_JIT_REG + 3) +#define TMP_REG_1 (MAX_BPF_JIT_REG + 0) +#define TMP_REG_2 (MAX_BPF_JIT_REG + 1) /* BPF to ppc register mappings */ static const int b2p[] = { @@ -63,40 +61,23 @@ static const int b2p[] = { [BPF_REG_FP] = 31, /* eBPF jit internal registers */ [BPF_REG_AX] = 2, - [SKB_HLEN_REG] = 25, - [SKB_DATA_REG] = 26, [TMP_REG_1] = 9, [TMP_REG_2] = 10 }; -/* PPC NVR range -- update this if we ever use NVRs below r24 */ -#define BPF_PPC_NVR_MIN 24 - -/* Assembly helpers */ -#define DECLARE_LOAD_FUNC(func) u64 func(u64 r3, u64 r4); \ - u64 func##_negative_offset(u64 r3, u64 r4); \ - u64 func##_positive_offset(u64 r3, u64 r4); - -DECLARE_LOAD_FUNC(sk_load_word); -DECLARE_LOAD_FUNC(sk_load_half); -DECLARE_LOAD_FUNC(sk_load_byte); - -#define CHOOSE_LOAD_FUNC(imm, func) \ - (imm < 0 ? \ - (imm >= SKF_LL_OFF ? func##_negative_offset : func) : \ - func##_positive_offset) +/* PPC NVR range -- update this if we ever use NVRs below r27 */ +#define BPF_PPC_NVR_MIN 27 #define SEEN_FUNC 0x1000 /* might call external helpers */ #define SEEN_STACK 0x2000 /* uses BPF stack */ -#define SEEN_SKB 0x4000 /* uses sk_buff */ -#define SEEN_TAILCALL 0x8000 /* uses tail calls */ +#define SEEN_TAILCALL 0x4000 /* uses tail calls */ struct codegen_context { /* * This is used to track register usage as well * as calls to external helpers. * - register usage is tracked with corresponding - * bits (r3-r10 and r25-r31) + * bits (r3-r10 and r27-r31) * - rest of the bits can be used to track other * things -- for now, we use bits 16 to 23 * encoded in SEEN_* macros above diff --git a/arch/powerpc/net/bpf_jit_asm64.S b/arch/powerpc/net/bpf_jit_asm64.S deleted file mode 100644 index 7e4c51430b84..000000000000 --- a/arch/powerpc/net/bpf_jit_asm64.S +++ /dev/null @@ -1,180 +0,0 @@ -/* - * bpf_jit_asm64.S: Packet/header access helper functions - * for PPC64 BPF compiler. - * - * Copyright 2016, Naveen N. Rao - * IBM Corporation - * - * Based on bpf_jit_asm.S by Matt Evans - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; version 2 - * of the License. - */ - -#include -#include -#include "bpf_jit64.h" - -/* - * All of these routines are called directly from generated code, - * with the below register usage: - * r27 skb pointer (ctx) - * r25 skb header length - * r26 skb->data pointer - * r4 offset - * - * Result is passed back in: - * r8 data read in host endian format (accumulator) - * - * r9 is used as a temporary register - */ - -#define r_skb r27 -#define r_hlen r25 -#define r_data r26 -#define r_off r4 -#define r_val r8 -#define r_tmp r9 - -_GLOBAL_TOC(sk_load_word) - cmpdi r_off, 0 - blt bpf_slow_path_word_neg - b sk_load_word_positive_offset - -_GLOBAL_TOC(sk_load_word_positive_offset) - /* Are we accessing past headlen? */ - subi r_tmp, r_hlen, 4 - cmpd r_tmp, r_off - blt bpf_slow_path_word - /* Nope, just hitting the header. cr0 here is eq or gt! */ - LWZX_BE r_val, r_data, r_off - blr /* Return success, cr0 != LT */ - -_GLOBAL_TOC(sk_load_half) - cmpdi r_off, 0 - blt bpf_slow_path_half_neg - b sk_load_half_positive_offset - -_GLOBAL_TOC(sk_load_half_positive_offset) - subi r_tmp, r_hlen, 2 - cmpd r_tmp, r_off - blt bpf_slow_path_half - LHZX_BE r_val, r_data, r_off - blr - -_GLOBAL_TOC(sk_load_byte) - cmpdi r_off, 0 - blt bpf_slow_path_byte_neg - b sk_load_byte_positive_offset - -_GLOBAL_TOC(sk_load_byte_positive_offset) - cmpd r_hlen, r_off - ble bpf_slow_path_byte - lbzx r_val, r_data, r_off - blr - -/* - * Call out to skb_copy_bits: - * Allocate a new stack frame here to remain ABI-compliant in - * stashing LR. - */ -#define bpf_slow_path_common(SIZE) \ - mflr r0; \ - std r0, PPC_LR_STKOFF(r1); \ - stdu r1, -(STACK_FRAME_MIN_SIZE + BPF_PPC_STACK_LOCALS)(r1); \ - mr r3, r_skb; \ - /* r4 = r_off as passed */ \ - addi r5, r1, STACK_FRAME_MIN_SIZE; \ - li r6, SIZE; \ - bl skb_copy_bits; \ - nop; \ - /* save r5 */ \ - addi r5, r1, STACK_FRAME_MIN_SIZE; \ - /* r3 = 0 on success */ \ - addi r1, r1, STACK_FRAME_MIN_SIZE + BPF_PPC_STACK_LOCALS; \ - ld r0, PPC_LR_STKOFF(r1); \ - mtlr r0; \ - cmpdi r3, 0; \ - blt bpf_error; /* cr0 = LT */ - -bpf_slow_path_word: - bpf_slow_path_common(4) - /* Data value is on stack, and cr0 != LT */ - LWZX_BE r_val, 0, r5 - blr - -bpf_slow_path_half: - bpf_slow_path_common(2) - LHZX_BE r_val, 0, r5 - blr - -bpf_slow_path_byte: - bpf_slow_path_common(1) - lbzx r_val, 0, r5 - blr - -/* - * Call out to bpf_internal_load_pointer_neg_helper - */ -#define sk_negative_common(SIZE) \ - mflr r0; \ - std r0, PPC_LR_STKOFF(r1); \ - stdu r1, -STACK_FRAME_MIN_SIZE(r1); \ - mr r3, r_skb; \ - /* r4 = r_off, as passed */ \ - li r5, SIZE; \ - bl bpf_internal_load_pointer_neg_helper; \ - nop; \ - addi r1, r1, STACK_FRAME_MIN_SIZE; \ - ld r0, PPC_LR_STKOFF(r1); \ - mtlr r0; \ - /* R3 != 0 on success */ \ - cmpldi r3, 0; \ - beq bpf_error_slow; /* cr0 = EQ */ - -bpf_slow_path_word_neg: - lis r_tmp, -32 /* SKF_LL_OFF */ - cmpd r_off, r_tmp /* addr < SKF_* */ - blt bpf_error /* cr0 = LT */ - b sk_load_word_negative_offset - -_GLOBAL_TOC(sk_load_word_negative_offset) - sk_negative_common(4) - LWZX_BE r_val, 0, r3 - blr - -bpf_slow_path_half_neg: - lis r_tmp, -32 /* SKF_LL_OFF */ - cmpd r_off, r_tmp /* addr < SKF_* */ - blt bpf_error /* cr0 = LT */ - b sk_load_half_negative_offset - -_GLOBAL_TOC(sk_load_half_negative_offset) - sk_negative_common(2) - LHZX_BE r_val, 0, r3 - blr - -bpf_slow_path_byte_neg: - lis r_tmp, -32 /* SKF_LL_OFF */ - cmpd r_off, r_tmp /* addr < SKF_* */ - blt bpf_error /* cr0 = LT */ - b sk_load_byte_negative_offset - -_GLOBAL_TOC(sk_load_byte_negative_offset) - sk_negative_common(1) - lbzx r_val, 0, r3 - blr - -bpf_error_slow: - /* fabricate a cr0 = lt */ - li r_tmp, -1 - cmpdi r_tmp, 0 -bpf_error: - /* - * Entered with cr0 = lt - * Generated code will 'blt epilogue', returning 0. - */ - li r_val, 0 - blr diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c index a9636d8cba15..5b061fc81df3 100644 --- a/arch/powerpc/net/bpf_jit_comp.c +++ b/arch/powerpc/net/bpf_jit_comp.c @@ -566,7 +566,7 @@ void bpf_jit_compile(struct bpf_prog *fp) if (!bpf_jit_enable) return; - addrs = kzalloc((flen+1) * sizeof(*addrs), GFP_KERNEL); + addrs = kcalloc(flen + 1, sizeof(*addrs), GFP_KERNEL); if (addrs == NULL) return; diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c index 0ef3d9580e98..380cbf9a40d9 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -59,7 +59,7 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx) * [ prev sp ] <------------- * [ ... ] | * sp (r1) ---> [ stack pointer ] -------------- - * [ nv gpr save area ] 8*8 + * [ nv gpr save area ] 6*8 * [ tail_call_cnt ] 8 * [ local_tmp_var ] 8 * [ unused red zone ] 208 bytes protected @@ -88,21 +88,6 @@ static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg) BUG(); } -static void bpf_jit_emit_skb_loads(u32 *image, struct codegen_context *ctx) -{ - /* - * Load skb->len and skb->data_len - * r3 points to skb - */ - PPC_LWZ(b2p[SKB_HLEN_REG], 3, offsetof(struct sk_buff, len)); - PPC_LWZ(b2p[TMP_REG_1], 3, offsetof(struct sk_buff, data_len)); - /* header_len = len - data_len */ - PPC_SUB(b2p[SKB_HLEN_REG], b2p[SKB_HLEN_REG], b2p[TMP_REG_1]); - - /* skb->data pointer */ - PPC_BPF_LL(b2p[SKB_DATA_REG], 3, offsetof(struct sk_buff, data)); -} - static void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) { int i; @@ -145,18 +130,6 @@ static void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx) if (bpf_is_seen_register(ctx, i)) PPC_BPF_STL(b2p[i], 1, bpf_jit_stack_offsetof(ctx, b2p[i])); - /* - * Save additional non-volatile regs if we cache skb - * Also, setup skb data - */ - if (ctx->seen & SEEN_SKB) { - PPC_BPF_STL(b2p[SKB_HLEN_REG], 1, - bpf_jit_stack_offsetof(ctx, b2p[SKB_HLEN_REG])); - PPC_BPF_STL(b2p[SKB_DATA_REG], 1, - bpf_jit_stack_offsetof(ctx, b2p[SKB_DATA_REG])); - bpf_jit_emit_skb_loads(image, ctx); - } - /* Setup frame pointer to point to the bpf stack area */ if (bpf_is_seen_register(ctx, BPF_REG_FP)) PPC_ADDI(b2p[BPF_REG_FP], 1, @@ -172,14 +145,6 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx if (bpf_is_seen_register(ctx, i)) PPC_BPF_LL(b2p[i], 1, bpf_jit_stack_offsetof(ctx, b2p[i])); - /* Restore non-volatile registers used for skb cache */ - if (ctx->seen & SEEN_SKB) { - PPC_BPF_LL(b2p[SKB_HLEN_REG], 1, - bpf_jit_stack_offsetof(ctx, b2p[SKB_HLEN_REG])); - PPC_BPF_LL(b2p[SKB_DATA_REG], 1, - bpf_jit_stack_offsetof(ctx, b2p[SKB_DATA_REG])); - } - /* Tear down our stack frame */ if (bpf_has_stack_frame(ctx)) { PPC_ADDI(1, 1, BPF_PPC_STACKFRAME + ctx->stack_size); @@ -202,25 +167,37 @@ static void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx) static void bpf_jit_emit_func_call(u32 *image, struct codegen_context *ctx, u64 func) { + unsigned int i, ctx_idx = ctx->idx; + + /* Load function address into r12 */ + PPC_LI64(12, func); + + /* For bpf-to-bpf function calls, the callee's address is unknown + * until the last extra pass. As seen above, we use PPC_LI64() to + * load the callee's address, but this may optimize the number of + * instructions required based on the nature of the address. + * + * Since we don't want the number of instructions emitted to change, + * we pad the optimized PPC_LI64() call with NOPs to guarantee that + * we always have a five-instruction sequence, which is the maximum + * that PPC_LI64() can emit. + */ + for (i = ctx->idx - ctx_idx; i < 5; i++) + PPC_NOP(); + #ifdef PPC64_ELF_ABI_v1 - /* func points to the function descriptor */ - PPC_LI64(b2p[TMP_REG_2], func); - /* Load actual entry point from function descriptor */ - PPC_BPF_LL(b2p[TMP_REG_1], b2p[TMP_REG_2], 0); - /* ... and move it to LR */ - PPC_MTLR(b2p[TMP_REG_1]); /* * Load TOC from function descriptor at offset 8. * We can clobber r2 since we get called through a * function pointer (so caller will save/restore r2) * and since we don't use a TOC ourself. */ - PPC_BPF_LL(2, b2p[TMP_REG_2], 8); -#else - /* We can clobber r12 */ - PPC_FUNC_ADDR(12, func); - PPC_MTLR(12); + PPC_BPF_LL(2, 12, 8); + /* Load actual entry point from function descriptor */ + PPC_BPF_LL(12, 12, 0); #endif + + PPC_MTLR(12); PPC_BLRL(); } @@ -291,7 +268,7 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 /* Assemble the body code between the prologue & epilogue */ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx, - u32 *addrs) + u32 *addrs, bool extra_pass) { const struct bpf_insn *insn = fp->insnsi; int flen = fp->len; @@ -747,29 +724,30 @@ emit_clear: break; /* - * Call kernel helper + * Call kernel helper or bpf function */ case BPF_JMP | BPF_CALL: ctx->seen |= SEEN_FUNC; - func = (u8 *) __bpf_call_base + imm; - /* Save skb pointer if we need to re-cache skb data */ - if ((ctx->seen & SEEN_SKB) && - bpf_helper_changes_pkt_data(func)) - PPC_BPF_STL(3, 1, bpf_jit_stack_local(ctx)); + /* bpf function call */ + if (insn[i].src_reg == BPF_PSEUDO_CALL) + if (!extra_pass) + func = NULL; + else if (fp->aux->func && off < fp->aux->func_cnt) + /* use the subprog id from the off + * field to lookup the callee address + */ + func = (u8 *) fp->aux->func[off]->bpf_func; + else + return -EINVAL; + /* kernel helper call */ + else + func = (u8 *) __bpf_call_base + imm; bpf_jit_emit_func_call(image, ctx, (u64)func); /* move return value from r3 to BPF_REG_0 */ PPC_MR(b2p[BPF_REG_0], 3); - - /* refresh skb cache */ - if ((ctx->seen & SEEN_SKB) && - bpf_helper_changes_pkt_data(func)) { - /* reload skb pointer to r3 */ - PPC_BPF_LL(3, 1, bpf_jit_stack_local(ctx)); - bpf_jit_emit_skb_loads(image, ctx); - } break; /* @@ -886,65 +864,6 @@ cond_branch: PPC_BCC(true_cond, addrs[i + 1 + off]); break; - /* - * Loads from packet header/data - * Assume 32-bit input value in imm and X (src_reg) - */ - - /* Absolute loads */ - case BPF_LD | BPF_W | BPF_ABS: - func = (u8 *)CHOOSE_LOAD_FUNC(imm, sk_load_word); - goto common_load_abs; - case BPF_LD | BPF_H | BPF_ABS: - func = (u8 *)CHOOSE_LOAD_FUNC(imm, sk_load_half); - goto common_load_abs; - case BPF_LD | BPF_B | BPF_ABS: - func = (u8 *)CHOOSE_LOAD_FUNC(imm, sk_load_byte); -common_load_abs: - /* - * Load from [imm] - * Load into r4, which can just be passed onto - * skb load helpers as the second parameter - */ - PPC_LI32(4, imm); - goto common_load; - - /* Indirect loads */ - case BPF_LD | BPF_W | BPF_IND: - func = (u8 *)sk_load_word; - goto common_load_ind; - case BPF_LD | BPF_H | BPF_IND: - func = (u8 *)sk_load_half; - goto common_load_ind; - case BPF_LD | BPF_B | BPF_IND: - func = (u8 *)sk_load_byte; -common_load_ind: - /* - * Load from [src_reg + imm] - * Treat src_reg as a 32-bit value - */ - PPC_EXTSW(4, src_reg); - if (imm) { - if (imm >= -32768 && imm < 32768) - PPC_ADDI(4, 4, IMM_L(imm)); - else { - PPC_LI32(b2p[TMP_REG_1], imm); - PPC_ADD(4, 4, b2p[TMP_REG_1]); - } - } - -common_load: - ctx->seen |= SEEN_SKB; - ctx->seen |= SEEN_FUNC; - bpf_jit_emit_func_call(image, ctx, (u64)func); - - /* - * Helper returns 'lt' condition on error, and an - * appropriate return value in BPF_REG_0 - */ - PPC_BCC(COND_LT, exit_addr); - break; - /* * Tail call */ @@ -971,6 +890,14 @@ common_load: return 0; } +struct powerpc64_jit_data { + struct bpf_binary_header *header; + u32 *addrs; + u8 *image; + u32 proglen; + struct codegen_context ctx; +}; + struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) { u32 proglen; @@ -978,6 +905,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) u8 *image = NULL; u32 *code_base; u32 *addrs; + struct powerpc64_jit_data *jit_data; struct codegen_context cgctx; int pass; int flen; @@ -985,6 +913,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) struct bpf_prog *org_fp = fp; struct bpf_prog *tmp_fp; bool bpf_blinded = false; + bool extra_pass = false; if (!fp->jit_requested) return org_fp; @@ -998,11 +927,32 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) fp = tmp_fp; } + jit_data = fp->aux->jit_data; + if (!jit_data) { + jit_data = kzalloc(sizeof(*jit_data), GFP_KERNEL); + if (!jit_data) { + fp = org_fp; + goto out; + } + fp->aux->jit_data = jit_data; + } + flen = fp->len; - addrs = kzalloc((flen+1) * sizeof(*addrs), GFP_KERNEL); + addrs = jit_data->addrs; + if (addrs) { + cgctx = jit_data->ctx; + image = jit_data->image; + bpf_hdr = jit_data->header; + proglen = jit_data->proglen; + alloclen = proglen + FUNCTION_DESCR_SIZE; + extra_pass = true; + goto skip_init_ctx; + } + + addrs = kcalloc(flen + 1, sizeof(*addrs), GFP_KERNEL); if (addrs == NULL) { fp = org_fp; - goto out; + goto out_addrs; } memset(&cgctx, 0, sizeof(struct codegen_context)); @@ -1011,10 +961,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) cgctx.stack_size = round_up(fp->aux->stack_depth, 16); /* Scouting faux-generate pass 0 */ - if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) { + if (bpf_jit_build_body(fp, 0, &cgctx, addrs, false)) { /* We hit something illegal or unsupported. */ fp = org_fp; - goto out; + goto out_addrs; } /* @@ -1032,9 +982,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) bpf_jit_fill_ill_insns); if (!bpf_hdr) { fp = org_fp; - goto out; + goto out_addrs; } +skip_init_ctx: code_base = (u32 *)(image + FUNCTION_DESCR_SIZE); /* Code generation passes 1-2 */ @@ -1042,7 +993,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) /* Now build the prologue, body code & epilogue for real. */ cgctx.idx = 0; bpf_jit_build_prologue(code_base, &cgctx); - bpf_jit_build_body(fp, code_base, &cgctx, addrs); + bpf_jit_build_body(fp, code_base, &cgctx, addrs, extra_pass); bpf_jit_build_epilogue(code_base, &cgctx); if (bpf_jit_enable > 1) @@ -1068,10 +1019,20 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) fp->jited_len = alloclen; bpf_flush_icache(bpf_hdr, (u8 *)bpf_hdr + (bpf_hdr->pages * PAGE_SIZE)); + if (!fp->is_func || extra_pass) { +out_addrs: + kfree(addrs); + kfree(jit_data); + fp->aux->jit_data = NULL; + } else { + jit_data->addrs = addrs; + jit_data->ctx = cgctx; + jit_data->proglen = proglen; + jit_data->image = image; + jit_data->header = bpf_hdr; + } out: - kfree(addrs); - if (bpf_blinded) bpf_jit_prog_release_other(fp, fp == org_fp ? tmp_fp : org_fp); diff --git a/arch/powerpc/oprofile/backtrace.c b/arch/powerpc/oprofile/backtrace.c index ecc66d5f02c9..ad054dd0d666 100644 --- a/arch/powerpc/oprofile/backtrace.c +++ b/arch/powerpc/oprofile/backtrace.c @@ -7,6 +7,7 @@ * 2 of the License, or (at your option) any later version. **/ +#include #include #include #include diff --git a/arch/powerpc/oprofile/cell/spu_profiler.c b/arch/powerpc/oprofile/cell/spu_profiler.c index 5182f2936af2..4e099e556645 100644 --- a/arch/powerpc/oprofile/cell/spu_profiler.c +++ b/arch/powerpc/oprofile/cell/spu_profiler.c @@ -210,8 +210,8 @@ int start_spu_profiling_cycles(unsigned int cycles_reset) timer.function = profile_spus; /* Allocate arrays for collecting SPU PC samples */ - samples = kzalloc(SPUS_PER_NODE * - TRACE_ARRAY_SIZE * sizeof(u32), GFP_KERNEL); + samples = kcalloc(SPUS_PER_NODE * TRACE_ARRAY_SIZE, sizeof(u32), + GFP_KERNEL); if (!samples) return -ENOMEM; diff --git a/arch/powerpc/perf/core-fsl-emb.c b/arch/powerpc/perf/core-fsl-emb.c index 85f1d18e5fd3..ba485844d506 100644 --- a/arch/powerpc/perf/core-fsl-emb.c +++ b/arch/powerpc/perf/core-fsl-emb.c @@ -42,7 +42,7 @@ static DEFINE_MUTEX(pmc_reserve_mutex); static inline int perf_intr_is_nmi(struct pt_regs *regs) { #ifdef __powerpc64__ - return !regs->softe; + return (regs->softe & IRQS_DISABLED); #else return 0; #endif diff --git a/arch/powerpc/perf/imc-pmu.c b/arch/powerpc/perf/imc-pmu.c index d7532e7b9ab5..d1977b61f827 100644 --- a/arch/powerpc/perf/imc-pmu.c +++ b/arch/powerpc/perf/imc-pmu.c @@ -40,6 +40,7 @@ static struct imc_pmu *core_imc_pmu; /* Thread IMC data structures and variables */ static DEFINE_PER_CPU(u64 *, thread_imc_mem); +static struct imc_pmu *thread_imc_pmu; static int thread_imc_mem_size; struct imc_pmu *imc_event_to_pmu(struct perf_event *event) @@ -1146,14 +1147,14 @@ static int init_nest_pmu_ref(void) static void cleanup_all_core_imc_memory(void) { - int i, nr_cores = DIV_ROUND_UP(num_present_cpus(), threads_per_core); + int i, nr_cores = DIV_ROUND_UP(num_possible_cpus(), threads_per_core); struct imc_mem_info *ptr = core_imc_pmu->mem_info; int size = core_imc_pmu->counter_mem_size; /* mem_info will never be NULL */ for (i = 0; i < nr_cores; i++) { if (ptr[i].vbase) - free_pages((u64)ptr->vbase, get_order(size)); + free_pages((u64)ptr[i].vbase, get_order(size)); } kfree(ptr); @@ -1191,7 +1192,6 @@ static void imc_common_mem_free(struct imc_pmu *pmu_ptr) if (pmu_ptr->attr_groups[IMC_EVENT_ATTR]) kfree(pmu_ptr->attr_groups[IMC_EVENT_ATTR]->attrs); kfree(pmu_ptr->attr_groups[IMC_EVENT_ATTR]); - kfree(pmu_ptr); } /* @@ -1208,6 +1208,7 @@ static void imc_common_cpuhp_mem_free(struct imc_pmu *pmu_ptr) cpuhp_remove_state(CPUHP_AP_PERF_POWERPC_NEST_IMC_ONLINE); kfree(nest_imc_refc); kfree(per_nest_pmu_arr); + per_nest_pmu_arr = NULL; } if (nest_pmus > 0) @@ -1228,6 +1229,16 @@ static void imc_common_cpuhp_mem_free(struct imc_pmu *pmu_ptr) } } +/* + * Function to unregister thread-imc if core-imc + * is not registered. + */ +void unregister_thread_imc(void) +{ + imc_common_cpuhp_mem_free(thread_imc_pmu); + imc_common_mem_free(thread_imc_pmu); + perf_pmu_unregister(&thread_imc_pmu->pmu); +} /* * imc_mem_init : Function to support memory allocation for core imc. @@ -1236,7 +1247,7 @@ static int imc_mem_init(struct imc_pmu *pmu_ptr, struct device_node *parent, int pmu_index) { const char *s; - int nr_cores, cpu, res; + int nr_cores, cpu, res = -ENOMEM; if (of_property_read_string(parent, "name", &s)) return -ENODEV; @@ -1246,7 +1257,7 @@ static int imc_mem_init(struct imc_pmu *pmu_ptr, struct device_node *parent, /* Update the pmu name */ pmu_ptr->pmu.name = kasprintf(GFP_KERNEL, "%s%s_imc", "nest_", s); if (!pmu_ptr->pmu.name) - return -ENOMEM; + goto err; /* Needed for hotplug/migration */ if (!per_nest_pmu_arr) { @@ -1254,7 +1265,7 @@ static int imc_mem_init(struct imc_pmu *pmu_ptr, struct device_node *parent, sizeof(struct imc_pmu *), GFP_KERNEL); if (!per_nest_pmu_arr) - return -ENOMEM; + goto err; } per_nest_pmu_arr[pmu_index] = pmu_ptr; break; @@ -1262,21 +1273,21 @@ static int imc_mem_init(struct imc_pmu *pmu_ptr, struct device_node *parent, /* Update the pmu name */ pmu_ptr->pmu.name = kasprintf(GFP_KERNEL, "%s%s", s, "_imc"); if (!pmu_ptr->pmu.name) - return -ENOMEM; + goto err; - nr_cores = DIV_ROUND_UP(num_present_cpus(), threads_per_core); + nr_cores = DIV_ROUND_UP(num_possible_cpus(), threads_per_core); pmu_ptr->mem_info = kcalloc(nr_cores, sizeof(struct imc_mem_info), GFP_KERNEL); if (!pmu_ptr->mem_info) - return -ENOMEM; + goto err; core_imc_refc = kcalloc(nr_cores, sizeof(struct imc_pmu_ref), GFP_KERNEL); if (!core_imc_refc) { kfree(pmu_ptr->mem_info); - return -ENOMEM; + goto err; } core_imc_pmu = pmu_ptr; @@ -1285,23 +1296,26 @@ static int imc_mem_init(struct imc_pmu *pmu_ptr, struct device_node *parent, /* Update the pmu name */ pmu_ptr->pmu.name = kasprintf(GFP_KERNEL, "%s%s", s, "_imc"); if (!pmu_ptr->pmu.name) - return -ENOMEM; + goto err; thread_imc_mem_size = pmu_ptr->counter_mem_size; for_each_online_cpu(cpu) { res = thread_imc_mem_alloc(cpu, pmu_ptr->counter_mem_size); if (res) { cleanup_all_thread_imc_memory(); - return res; + goto err; } } + thread_imc_pmu = pmu_ptr; break; default: return -EINVAL; } return 0; +err: + return res; } /* @@ -1319,10 +1333,8 @@ int init_imc_pmu(struct device_node *parent, struct imc_pmu *pmu_ptr, int pmu_id int ret; ret = imc_mem_init(pmu_ptr, parent, pmu_idx); - if (ret) { - imc_common_mem_free(pmu_ptr); - return ret; - } + if (ret) + goto err_free_mem; switch (pmu_ptr->domain) { case IMC_DOMAIN_NEST: @@ -1337,7 +1349,9 @@ int init_imc_pmu(struct device_node *parent, struct imc_pmu *pmu_ptr, int pmu_id ret = init_nest_pmu_ref(); if (ret) { mutex_unlock(&nest_init_lock); - goto err_free; + kfree(per_nest_pmu_arr); + per_nest_pmu_arr = NULL; + goto err_free_mem; } /* Register for cpu hotplug notification. */ ret = nest_pmu_cpumask_init(); @@ -1345,7 +1359,8 @@ int init_imc_pmu(struct device_node *parent, struct imc_pmu *pmu_ptr, int pmu_id mutex_unlock(&nest_init_lock); kfree(nest_imc_refc); kfree(per_nest_pmu_arr); - goto err_free; + per_nest_pmu_arr = NULL; + goto err_free_mem; } } nest_pmus++; @@ -1355,7 +1370,7 @@ int init_imc_pmu(struct device_node *parent, struct imc_pmu *pmu_ptr, int pmu_id ret = core_imc_pmu_cpumask_init(); if (ret) { cleanup_all_core_imc_memory(); - return ret; + goto err_free_mem; } break; @@ -1363,33 +1378,34 @@ int init_imc_pmu(struct device_node *parent, struct imc_pmu *pmu_ptr, int pmu_id ret = thread_imc_cpu_init(); if (ret) { cleanup_all_thread_imc_memory(); - return ret; + goto err_free_mem; } break; default: - return -1; /* Unknown domain */ + return -EINVAL; /* Unknown domain */ } ret = update_events_in_group(parent, pmu_ptr); if (ret) - goto err_free; + goto err_free_cpuhp_mem; ret = update_pmu_ops(pmu_ptr); if (ret) - goto err_free; + goto err_free_cpuhp_mem; ret = perf_pmu_register(&pmu_ptr->pmu, pmu_ptr->pmu.name, -1); if (ret) - goto err_free; + goto err_free_cpuhp_mem; pr_info("%s performance monitor hardware support registered\n", pmu_ptr->pmu.name); return 0; -err_free: - imc_common_mem_free(pmu_ptr); +err_free_cpuhp_mem: imc_common_cpuhp_mem_free(pmu_ptr); +err_free_mem: + imc_common_mem_free(pmu_ptr); return ret; } diff --git a/arch/powerpc/perf/isa207-common.h b/arch/powerpc/perf/isa207-common.h index 6c737d675792..6a0b586c935a 100644 --- a/arch/powerpc/perf/isa207-common.h +++ b/arch/powerpc/perf/isa207-common.h @@ -17,70 +17,6 @@ #include #include -/* - * Raw event encoding for PowerISA v2.07: - * - * 60 56 52 48 44 40 36 32 - * | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | - * | | [ ] [ thresh_cmp ] [ thresh_ctl ] - * | | | | - * | | *- IFM (Linux) thresh start/stop OR FAB match -* - * | *- BHRB (Linux) - * *- EBB (Linux) - * - * 28 24 20 16 12 8 4 0 - * | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | - * [ ] [ sample ] [cache] [ pmc ] [unit ] c m [ pmcxsel ] - * | | | | | - * | | | | *- mark - * | | *- L1/L2/L3 cache_sel | - * | | | - * | *- sampling mode for marked events *- combine - * | - * *- thresh_sel - * - * Below uses IBM bit numbering. - * - * MMCR1[x:y] = unit (PMCxUNIT) - * MMCR1[x] = combine (PMCxCOMB) - * - * if pmc == 3 and unit == 0 and pmcxsel[0:6] == 0b0101011 - * # PM_MRK_FAB_RSP_MATCH - * MMCR1[20:27] = thresh_ctl (FAB_CRESP_MATCH / FAB_TYPE_MATCH) - * else if pmc == 4 and unit == 0xf and pmcxsel[0:6] == 0b0101001 - * # PM_MRK_FAB_RSP_MATCH_CYC - * MMCR1[20:27] = thresh_ctl (FAB_CRESP_MATCH / FAB_TYPE_MATCH) - * else - * MMCRA[48:55] = thresh_ctl (THRESH START/END) - * - * if thresh_sel: - * MMCRA[45:47] = thresh_sel - * - * if thresh_cmp: - * MMCRA[22:24] = thresh_cmp[0:2] - * MMCRA[25:31] = thresh_cmp[3:9] - * - * if unit == 6 or unit == 7 - * MMCRC[53:55] = cache_sel[1:3] (L2EVENT_SEL) - * else if unit == 8 or unit == 9: - * if cache_sel[0] == 0: # L3 bank - * MMCRC[47:49] = cache_sel[1:3] (L3EVENT_SEL0) - * else if cache_sel[0] == 1: - * MMCRC[50:51] = cache_sel[2:3] (L3EVENT_SEL1) - * else if cache_sel[1]: # L1 event - * MMCR1[16] = cache_sel[2] - * MMCR1[17] = cache_sel[3] - * - * if mark: - * MMCRA[63] = 1 (SAMPLE_ENABLE) - * MMCRA[57:59] = sample[0:2] (RAND_SAMP_ELIG) - * MMCRA[61:62] = sample[3:4] (RAND_SAMP_MODE) - * - * if EBB and BHRB: - * MMCRA[32:33] = IFM - * - */ - #define EVENT_EBB_MASK 1ull #define EVENT_EBB_SHIFT PERF_EVENT_CONFIG_EBB_SHIFT #define EVENT_BHRB_MASK 1ull diff --git a/arch/powerpc/perf/power8-pmu.c b/arch/powerpc/perf/power8-pmu.c index c9356955cab4..d12a2db26353 100644 --- a/arch/powerpc/perf/power8-pmu.c +++ b/arch/powerpc/perf/power8-pmu.c @@ -30,6 +30,70 @@ enum { #define POWER8_MMCRA_IFM2 0x0000000080000000UL #define POWER8_MMCRA_IFM3 0x00000000C0000000UL +/* + * Raw event encoding for PowerISA v2.07 (Power8): + * + * 60 56 52 48 44 40 36 32 + * | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | + * | | [ ] [ thresh_cmp ] [ thresh_ctl ] + * | | | | + * | | *- IFM (Linux) thresh start/stop OR FAB match -* + * | *- BHRB (Linux) + * *- EBB (Linux) + * + * 28 24 20 16 12 8 4 0 + * | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | - - - - | + * [ ] [ sample ] [cache] [ pmc ] [unit ] c m [ pmcxsel ] + * | | | | | + * | | | | *- mark + * | | *- L1/L2/L3 cache_sel | + * | | | + * | *- sampling mode for marked events *- combine + * | + * *- thresh_sel + * + * Below uses IBM bit numbering. + * + * MMCR1[x:y] = unit (PMCxUNIT) + * MMCR1[x] = combine (PMCxCOMB) + * + * if pmc == 3 and unit == 0 and pmcxsel[0:6] == 0b0101011 + * # PM_MRK_FAB_RSP_MATCH + * MMCR1[20:27] = thresh_ctl (FAB_CRESP_MATCH / FAB_TYPE_MATCH) + * else if pmc == 4 and unit == 0xf and pmcxsel[0:6] == 0b0101001 + * # PM_MRK_FAB_RSP_MATCH_CYC + * MMCR1[20:27] = thresh_ctl (FAB_CRESP_MATCH / FAB_TYPE_MATCH) + * else + * MMCRA[48:55] = thresh_ctl (THRESH START/END) + * + * if thresh_sel: + * MMCRA[45:47] = thresh_sel + * + * if thresh_cmp: + * MMCRA[22:24] = thresh_cmp[0:2] + * MMCRA[25:31] = thresh_cmp[3:9] + * + * if unit == 6 or unit == 7 + * MMCRC[53:55] = cache_sel[1:3] (L2EVENT_SEL) + * else if unit == 8 or unit == 9: + * if cache_sel[0] == 0: # L3 bank + * MMCRC[47:49] = cache_sel[1:3] (L3EVENT_SEL0) + * else if cache_sel[0] == 1: + * MMCRC[50:51] = cache_sel[2:3] (L3EVENT_SEL1) + * else if cache_sel[1]: # L1 event + * MMCR1[16] = cache_sel[2] + * MMCR1[17] = cache_sel[3] + * + * if mark: + * MMCRA[63] = 1 (SAMPLE_ENABLE) + * MMCRA[57:59] = sample[0:2] (RAND_SAMP_ELIG) + * MMCRA[61:62] = sample[3:4] (RAND_SAMP_MODE) + * + * if EBB and BHRB: + * MMCRA[32:33] = IFM + * + */ + /* PowerISA v2.07 format attribute structure*/ extern struct attribute_group isa207_pmu_format_group; diff --git a/arch/powerpc/platforms/4xx/hsta_msi.c b/arch/powerpc/platforms/4xx/hsta_msi.c index 9926ad67af76..1c18f2955f7d 100644 --- a/arch/powerpc/platforms/4xx/hsta_msi.c +++ b/arch/powerpc/platforms/4xx/hsta_msi.c @@ -156,7 +156,8 @@ static int hsta_msi_probe(struct platform_device *pdev) if (ret) goto out; - ppc4xx_hsta_msi.irq_map = kmalloc(sizeof(int) * irq_count, GFP_KERNEL); + ppc4xx_hsta_msi.irq_map = kmalloc_array(irq_count, sizeof(int), + GFP_KERNEL); if (!ppc4xx_hsta_msi.irq_map) { ret = -ENOMEM; goto out1; diff --git a/arch/powerpc/platforms/4xx/msi.c b/arch/powerpc/platforms/4xx/msi.c index 96aaae678928..81b2cbce7df8 100644 --- a/arch/powerpc/platforms/4xx/msi.c +++ b/arch/powerpc/platforms/4xx/msi.c @@ -89,7 +89,7 @@ static int ppc4xx_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) if (type == PCI_CAP_ID_MSIX) pr_debug("ppc4xx msi: MSI-X untested, trying anyway.\n"); - msi_data->msi_virqs = kmalloc((msi_irqs) * sizeof(int), GFP_KERNEL); + msi_data->msi_virqs = kmalloc_array(msi_irqs, sizeof(int), GFP_KERNEL); if (!msi_data->msi_virqs) return -ENOMEM; diff --git a/arch/powerpc/platforms/4xx/pci.c b/arch/powerpc/platforms/4xx/pci.c index 73e6b36bcd51..5aca523551ae 100644 --- a/arch/powerpc/platforms/4xx/pci.c +++ b/arch/powerpc/platforms/4xx/pci.c @@ -1449,7 +1449,7 @@ static int __init ppc4xx_pciex_check_core_init(struct device_node *np) count = ppc4xx_pciex_hwops->core_init(np); if (count > 0) { ppc4xx_pciex_ports = - kzalloc(count * sizeof(struct ppc4xx_pciex_port), + kcalloc(count, sizeof(struct ppc4xx_pciex_port), GFP_KERNEL); if (ppc4xx_pciex_ports) { ppc4xx_pciex_port_count = count; diff --git a/arch/powerpc/platforms/83xx/Kconfig b/arch/powerpc/platforms/83xx/Kconfig index 7e38b7b71a5a..071f53b0c0a0 100644 --- a/arch/powerpc/platforms/83xx/Kconfig +++ b/arch/powerpc/platforms/83xx/Kconfig @@ -90,13 +90,6 @@ config MPC837x_RDB help This option enables support for the MPC837x RDB and WLAN Boards. -config SBC834x - bool "Wind River SBC834x" - select DEFAULT_UIMAGE - select PPC_MPC834x - help - This option enables support for the Wind River SBC834x board. - config ASP834x bool "Analogue & Micro ASP 834x" select PPC_MPC834x diff --git a/arch/powerpc/platforms/83xx/Makefile b/arch/powerpc/platforms/83xx/Makefile index bb4720897f6a..41cb5f842eff 100644 --- a/arch/powerpc/platforms/83xx/Makefile +++ b/arch/powerpc/platforms/83xx/Makefile @@ -14,7 +14,6 @@ obj-$(CONFIG_MPC836x_MDS) += mpc836x_mds.o obj-$(CONFIG_MPC836x_RDK) += mpc836x_rdk.o obj-$(CONFIG_MPC832x_MDS) += mpc832x_mds.o obj-$(CONFIG_MPC837x_MDS) += mpc837x_mds.o -obj-$(CONFIG_SBC834x) += sbc834x.o obj-$(CONFIG_MPC837x_RDB) += mpc837x_rdb.o obj-$(CONFIG_ASP834x) += asp834x.o obj-$(CONFIG_KMETER1) += km83xx.o diff --git a/arch/powerpc/platforms/83xx/sbc834x.c b/arch/powerpc/platforms/83xx/sbc834x.c deleted file mode 100644 index cb4bdabfdf1c..000000000000 --- a/arch/powerpc/platforms/83xx/sbc834x.c +++ /dev/null @@ -1,73 +0,0 @@ -/* - * arch/powerpc/platforms/83xx/sbc834x.c - * - * Wind River SBC834x board specific routines - * - * By Paul Gortmaker (see MAINTAINERS for contact information) - * - * Based largely on the mpc834x_mds.c support by Kumar Gala. - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the - * Free Software Foundation; either version 2 of the License, or (at your - * option) any later version. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "mpc83xx.h" - -/* ************************************************************************ - * - * Setup the architecture - * - */ -static void __init sbc834x_setup_arch(void) -{ - mpc83xx_setup_arch(); -} - -machine_device_initcall(sbc834x, mpc83xx_declare_of_platform_devices); - -/* - * Called very early, MMU is off, device-tree isn't unflattened - */ -static int __init sbc834x_probe(void) -{ - return of_machine_is_compatible("SBC834xE"); -} - -define_machine(sbc834x) { - .name = "SBC834xE", - .probe = sbc834x_probe, - .setup_arch = sbc834x_setup_arch, - .init_IRQ = mpc83xx_ipic_init_IRQ, - .get_irq = ipic_get_irq, - .restart = mpc83xx_restart, - .time_init = mpc83xx_time_init, - .calibrate_decr = generic_calibrate_decr, - .progress = udbg_progress, -}; diff --git a/arch/powerpc/platforms/8xx/adder875.c b/arch/powerpc/platforms/8xx/adder875.c index 333dece79394..bcef9f66191e 100644 --- a/arch/powerpc/platforms/8xx/adder875.c +++ b/arch/powerpc/platforms/8xx/adder875.c @@ -111,7 +111,5 @@ define_machine(adder875) { .get_irq = mpc8xx_get_irq, .restart = mpc8xx_restart, .calibrate_decr = generic_calibrate_decr, - .set_rtc_time = mpc8xx_set_rtc_time, - .get_rtc_time = mpc8xx_get_rtc_time, .progress = udbg_progress, }; diff --git a/arch/powerpc/platforms/8xx/ep88xc.c b/arch/powerpc/platforms/8xx/ep88xc.c index cd0d90f1fb1c..ebcf34a14789 100644 --- a/arch/powerpc/platforms/8xx/ep88xc.c +++ b/arch/powerpc/platforms/8xx/ep88xc.c @@ -170,7 +170,5 @@ define_machine(ep88xc) { .get_irq = mpc8xx_get_irq, .restart = mpc8xx_restart, .calibrate_decr = mpc8xx_calibrate_decr, - .set_rtc_time = mpc8xx_set_rtc_time, - .get_rtc_time = mpc8xx_get_rtc_time, .progress = udbg_progress, }; diff --git a/arch/powerpc/platforms/8xx/m8xx_setup.c b/arch/powerpc/platforms/8xx/m8xx_setup.c index 2188d691a40f..027c42d8966c 100644 --- a/arch/powerpc/platforms/8xx/m8xx_setup.c +++ b/arch/powerpc/platforms/8xx/m8xx_setup.c @@ -169,15 +169,14 @@ int mpc8xx_set_rtc_time(struct rtc_time *tm) { sitk8xx_t __iomem *sys_tmr1; sit8xx_t __iomem *sys_tmr2; - int time; + time64_t time; sys_tmr1 = immr_map(im_sitk); sys_tmr2 = immr_map(im_sit); - time = mktime(tm->tm_year+1900, tm->tm_mon+1, tm->tm_mday, - tm->tm_hour, tm->tm_min, tm->tm_sec); + time = rtc_tm_to_time64(tm); out_be32(&sys_tmr1->sitk_rtck, KAPWR_KEY); - out_be32(&sys_tmr2->sit_rtc, time); + out_be32(&sys_tmr2->sit_rtc, (u32)time); out_be32(&sys_tmr1->sitk_rtck, ~KAPWR_KEY); immr_unmap(sys_tmr2); @@ -192,9 +191,7 @@ void mpc8xx_get_rtc_time(struct rtc_time *tm) /* Get time from the RTC. */ data = in_be32(&sys_tmr->sit_rtc); - to_tm(data, tm); - tm->tm_year -= 1900; - tm->tm_mon -= 1; + rtc_time64_to_tm(data, tm); immr_unmap(sys_tmr); return; } diff --git a/arch/powerpc/platforms/8xx/mpc885ads_setup.c b/arch/powerpc/platforms/8xx/mpc885ads_setup.c index e821a42d5816..a0c83c1905c6 100644 --- a/arch/powerpc/platforms/8xx/mpc885ads_setup.c +++ b/arch/powerpc/platforms/8xx/mpc885ads_setup.c @@ -220,7 +220,5 @@ define_machine(mpc885_ads) { .get_irq = mpc8xx_get_irq, .restart = mpc8xx_restart, .calibrate_decr = mpc8xx_calibrate_decr, - .set_rtc_time = mpc8xx_set_rtc_time, - .get_rtc_time = mpc8xx_get_rtc_time, .progress = udbg_progress, }; diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype index 67d3125d0610..e6a1de521319 100644 --- a/arch/powerpc/platforms/Kconfig.cputype +++ b/arch/powerpc/platforms/Kconfig.cputype @@ -222,6 +222,7 @@ config PTE_64BIT config PHYS_64BIT bool 'Large physical address support' if E500 || PPC_86xx depends on (44x || E500 || PPC_86xx) && !PPC_83xx && !PPC_82xx + select PHYS_ADDR_T_64BIT ---help--- This option enables kernel support for larger than 32-bit physical addresses. This feature may not be available on all cores. @@ -292,6 +293,10 @@ config PPC_STD_MMU_32 def_bool y depends on PPC_STD_MMU && PPC32 +config ARCH_ENABLE_SPLIT_PMD_PTLOCK + def_bool y + depends on PPC_BOOK3S_64 + config PPC_RADIX_MMU bool "Radix MMU Support" depends on PPC_BOOK3S_64 diff --git a/arch/powerpc/platforms/cell/spu_callbacks.c b/arch/powerpc/platforms/cell/spu_callbacks.c index a494028b2cdf..8ae86200ef6c 100644 --- a/arch/powerpc/platforms/cell/spu_callbacks.c +++ b/arch/powerpc/platforms/cell/spu_callbacks.c @@ -44,7 +44,7 @@ static void *spu_syscall_table[] = { #define SYSCALL_SPU(func) sys_##func, #define COMPAT_SYS_SPU(func) sys_##func, -#define PPC_SYS_SPU(func) ppc_##func, +#define COMPAT_SPU_NEW(func) sys_##func, #define SYSX_SPU(f, f3264, f32) f, #include diff --git a/arch/powerpc/platforms/cell/spu_syscalls.c b/arch/powerpc/platforms/cell/spu_syscalls.c index 5e6e0bad6db6..263413a34823 100644 --- a/arch/powerpc/platforms/cell/spu_syscalls.c +++ b/arch/powerpc/platforms/cell/spu_syscalls.c @@ -26,6 +26,7 @@ #include #include #include +#include #include @@ -90,7 +91,7 @@ SYSCALL_DEFINE4(spu_create, const char __user *, name, unsigned int, flags, return ret; } -asmlinkage long sys_spu_run(int fd, __u32 __user *unpc, __u32 __user *ustatus) +SYSCALL_DEFINE3(spu_run,int, fd, __u32 __user *, unpc, __u32 __user *, ustatus) { long ret; struct fd arg; diff --git a/arch/powerpc/platforms/cell/spufs/fault.c b/arch/powerpc/platforms/cell/spufs/fault.c index 870c0a82d560..1e002e94d0f6 100644 --- a/arch/powerpc/platforms/cell/spufs/fault.c +++ b/arch/powerpc/platforms/cell/spufs/fault.c @@ -44,7 +44,7 @@ static void spufs_handle_event(struct spu_context *ctx, return; } - memset(&info, 0, sizeof(info)); + clear_siginfo(&info); switch (type) { case SPE_EVENT_INVALID_DMA: diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c index 469bdd0b748f..43e7b93f27c7 100644 --- a/arch/powerpc/platforms/cell/spufs/file.c +++ b/arch/powerpc/platforms/cell/spufs/file.c @@ -232,12 +232,13 @@ spufs_mem_write(struct file *file, const char __user *buffer, return size; } -static int +static vm_fault_t spufs_mem_mmap_fault(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; struct spu_context *ctx = vma->vm_file->private_data; unsigned long pfn, offset; + vm_fault_t ret; offset = vmf->pgoff << PAGE_SHIFT; if (offset >= LS_SIZE) @@ -256,11 +257,11 @@ spufs_mem_mmap_fault(struct vm_fault *vmf) vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot); pfn = (ctx->spu->local_store_phys + offset) >> PAGE_SHIFT; } - vm_insert_pfn(vma, vmf->address, pfn); + ret = vmf_insert_pfn(vma, vmf->address, pfn); spu_release(ctx); - return VM_FAULT_NOPAGE; + return ret; } static int spufs_mem_mmap_access(struct vm_area_struct *vma, @@ -312,13 +313,14 @@ static const struct file_operations spufs_mem_fops = { .mmap = spufs_mem_mmap, }; -static int spufs_ps_fault(struct vm_fault *vmf, +static vm_fault_t spufs_ps_fault(struct vm_fault *vmf, unsigned long ps_offs, unsigned long ps_size) { struct spu_context *ctx = vmf->vma->vm_file->private_data; unsigned long area, offset = vmf->pgoff << PAGE_SHIFT; - int ret = 0; + int err = 0; + vm_fault_t ret = VM_FAULT_NOPAGE; spu_context_nospu_trace(spufs_ps_fault__enter, ctx); @@ -349,25 +351,26 @@ static int spufs_ps_fault(struct vm_fault *vmf, if (ctx->state == SPU_STATE_SAVED) { up_read(¤t->mm->mmap_sem); spu_context_nospu_trace(spufs_ps_fault__sleep, ctx); - ret = spufs_wait(ctx->run_wq, ctx->state == SPU_STATE_RUNNABLE); + err = spufs_wait(ctx->run_wq, ctx->state == SPU_STATE_RUNNABLE); spu_context_trace(spufs_ps_fault__wake, ctx, ctx->spu); down_read(¤t->mm->mmap_sem); } else { area = ctx->spu->problem_phys + ps_offs; - vm_insert_pfn(vmf->vma, vmf->address, (area + offset) >> PAGE_SHIFT); + ret = vmf_insert_pfn(vmf->vma, vmf->address, + (area + offset) >> PAGE_SHIFT); spu_context_trace(spufs_ps_fault__insert, ctx, ctx->spu); } - if (!ret) + if (!err) spu_release(ctx); refault: put_spu_context(ctx); - return VM_FAULT_NOPAGE; + return ret; } #if SPUFS_MMAP_4K -static int spufs_cntl_mmap_fault(struct vm_fault *vmf) +static vm_fault_t spufs_cntl_mmap_fault(struct vm_fault *vmf) { return spufs_ps_fault(vmf, 0x4000, SPUFS_CNTL_MAP_SIZE); } @@ -1040,7 +1043,7 @@ static ssize_t spufs_signal1_write(struct file *file, const char __user *buf, return 4; } -static int +static vm_fault_t spufs_signal1_mmap_fault(struct vm_fault *vmf) { #if SPUFS_SIGNAL_MAP_SIZE == 0x1000 @@ -1178,7 +1181,7 @@ static ssize_t spufs_signal2_write(struct file *file, const char __user *buf, } #if SPUFS_MMAP_4K -static int +static vm_fault_t spufs_signal2_mmap_fault(struct vm_fault *vmf) { #if SPUFS_SIGNAL_MAP_SIZE == 0x1000 @@ -1307,7 +1310,7 @@ DEFINE_SPUFS_ATTRIBUTE(spufs_signal2_type, spufs_signal2_type_get, spufs_signal2_type_set, "%llu\n", SPU_ATTR_ACQUIRE); #if SPUFS_MMAP_4K -static int +static vm_fault_t spufs_mss_mmap_fault(struct vm_fault *vmf) { return spufs_ps_fault(vmf, 0x0000, SPUFS_MSS_MAP_SIZE); @@ -1369,7 +1372,7 @@ static const struct file_operations spufs_mss_fops = { .llseek = no_llseek, }; -static int +static vm_fault_t spufs_psmap_mmap_fault(struct vm_fault *vmf) { return spufs_ps_fault(vmf, 0x0000, SPUFS_PS_MAP_SIZE); @@ -1429,7 +1432,7 @@ static const struct file_operations spufs_psmap_fops = { #if SPUFS_MMAP_4K -static int +static vm_fault_t spufs_mfc_mmap_fault(struct vm_fault *vmf) { return spufs_ps_fault(vmf, 0x3000, SPUFS_MFC_MAP_SIZE); diff --git a/arch/powerpc/platforms/cell/spufs/sched.c b/arch/powerpc/platforms/cell/spufs/sched.c index ccc421503363..c9ef3c532169 100644 --- a/arch/powerpc/platforms/cell/spufs/sched.c +++ b/arch/powerpc/platforms/cell/spufs/sched.c @@ -1095,18 +1095,6 @@ static int show_spu_loadavg(struct seq_file *s, void *private) atomic_read(&nr_spu_contexts), idr_get_cursor(&task_active_pid_ns(current)->idr) - 1); return 0; -} - -static int spu_loadavg_open(struct inode *inode, struct file *file) -{ - return single_open(file, show_spu_loadavg, NULL); -} - -static const struct file_operations spu_loadavg_fops = { - .open = spu_loadavg_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, }; int __init spu_sched_init(void) @@ -1135,7 +1123,7 @@ int __init spu_sched_init(void) mod_timer(&spuloadavg_timer, 0); - entry = proc_create("spu_loadavg", 0, NULL, &spu_loadavg_fops); + entry = proc_create_single("spu_loadavg", 0, NULL, show_spu_loadavg); if (!entry) goto out_stop_kthread; diff --git a/arch/powerpc/platforms/chrp/pci.c b/arch/powerpc/platforms/chrp/pci.c index 0f512d35f7c5..5ddb57b82921 100644 --- a/arch/powerpc/platforms/chrp/pci.c +++ b/arch/powerpc/platforms/chrp/pci.c @@ -31,7 +31,7 @@ void __iomem *gg2_pci_config_base; * limit the bus number to 3 bits */ -int gg2_read_config(struct pci_bus *bus, unsigned int devfn, int off, +static int gg2_read_config(struct pci_bus *bus, unsigned int devfn, int off, int len, u32 *val) { volatile void __iomem *cfg_data; @@ -58,7 +58,7 @@ int gg2_read_config(struct pci_bus *bus, unsigned int devfn, int off, return PCIBIOS_SUCCESSFUL; } -int gg2_write_config(struct pci_bus *bus, unsigned int devfn, int off, +static int gg2_write_config(struct pci_bus *bus, unsigned int devfn, int off, int len, u32 val) { volatile void __iomem *cfg_data; @@ -94,8 +94,8 @@ static struct pci_ops gg2_pci_ops = /* * Access functions for PCI config space using RTAS calls. */ -int rtas_read_config(struct pci_bus *bus, unsigned int devfn, int offset, - int len, u32 *val) +static int rtas_read_config(struct pci_bus *bus, unsigned int devfn, int offset, + int len, u32 *val) { struct pci_controller *hose = pci_bus_to_host(bus); unsigned long addr = (offset & 0xff) | ((devfn & 0xff) << 8) @@ -109,8 +109,8 @@ int rtas_read_config(struct pci_bus *bus, unsigned int devfn, int offset, return rval? PCIBIOS_DEVICE_NOT_FOUND: PCIBIOS_SUCCESSFUL; } -int rtas_write_config(struct pci_bus *bus, unsigned int devfn, int offset, - int len, u32 val) +static int rtas_write_config(struct pci_bus *bus, unsigned int devfn, int offset, + int len, u32 val) { struct pci_controller *hose = pci_bus_to_host(bus); unsigned long addr = (offset & 0xff) | ((devfn & 0xff) << 8) diff --git a/arch/powerpc/platforms/chrp/setup.c b/arch/powerpc/platforms/chrp/setup.c index 481ed133e04b..d6d8ffc0271e 100644 --- a/arch/powerpc/platforms/chrp/setup.c +++ b/arch/powerpc/platforms/chrp/setup.c @@ -94,7 +94,7 @@ static const char *chrp_names[] = { "Total Impact Briq" }; -void chrp_show_cpuinfo(struct seq_file *m) +static void chrp_show_cpuinfo(struct seq_file *m) { int i, sdramen; unsigned int t; @@ -299,7 +299,7 @@ out_put: of_node_put(node); } -void __init chrp_setup_arch(void) +static void __init chrp_setup_arch(void) { struct device_node *root = of_find_node_by_path("/"); const char *machine = NULL; @@ -382,7 +382,7 @@ static void __init chrp_find_openpic(void) { struct device_node *np, *root; int len, i, j; - int isu_size, idu_size; + int isu_size; const unsigned int *iranges, *opprop = NULL; int oplen = 0; unsigned long opaddr; @@ -427,11 +427,9 @@ static void __init chrp_find_openpic(void) } isu_size = 0; - idu_size = 0; if (len > 0 && iranges[1] != 0) { printk(KERN_INFO "OpenPIC irqs %d..%d in IDU\n", iranges[0], iranges[0] + iranges[1] - 1); - idu_size = iranges[1]; } if (len > 1) isu_size = iranges[3]; @@ -523,7 +521,7 @@ static void __init chrp_find_8259(void) } } -void __init chrp_init_IRQ(void) +static void __init chrp_init_IRQ(void) { #if defined(CONFIG_VT) && defined(CONFIG_INPUT_ADBHID) && defined(CONFIG_XMON) struct device_node *kbd; @@ -555,7 +553,7 @@ void __init chrp_init_IRQ(void) #endif } -void __init +static void __init chrp_init2(void) { #ifdef CONFIG_NVRAM diff --git a/arch/powerpc/platforms/chrp/time.c b/arch/powerpc/platforms/chrp/time.c index 03d115aaa191..acde7bbe0716 100644 --- a/arch/powerpc/platforms/chrp/time.c +++ b/arch/powerpc/platforms/chrp/time.c @@ -28,6 +28,8 @@ #include #include +#include + extern spinlock_t rtc_lock; #define NVRAM_AS0 0x74 @@ -63,7 +65,7 @@ long __init chrp_time_init(void) return 0; } -int chrp_cmos_clock_read(int addr) +static int chrp_cmos_clock_read(int addr) { if (nvram_as1 != 0) outb(addr>>8, nvram_as1); @@ -71,7 +73,7 @@ int chrp_cmos_clock_read(int addr) return (inb(nvram_data)); } -void chrp_cmos_clock_write(unsigned long val, int addr) +static void chrp_cmos_clock_write(unsigned long val, int addr) { if (nvram_as1 != 0) outb(addr>>8, nvram_as1); diff --git a/arch/powerpc/platforms/embedded6xx/Kconfig b/arch/powerpc/platforms/embedded6xx/Kconfig index 9fb2d5912c5a..8ea16db5ff48 100644 --- a/arch/powerpc/platforms/embedded6xx/Kconfig +++ b/arch/powerpc/platforms/embedded6xx/Kconfig @@ -48,16 +48,6 @@ config PPC_HOLLY Select PPC_HOLLY if configuring for an IBM 750GX/CL Eval Board with TSI108/9 bridge (Hickory/Holly) -config PPC_C2K - bool "SBS/GEFanuc C2K board" - depends on EMBEDDED6xx - select MV64X60 - select NOT_COHERENT_CACHE - select MTD_CFI_I4 - help - This option enables support for the GE Fanuc C2K board (formerly - an SBS board). - config MVME5100 bool "Motorola/Emerson MVME5100" depends on EMBEDDED6xx diff --git a/arch/powerpc/platforms/embedded6xx/Makefile b/arch/powerpc/platforms/embedded6xx/Makefile index 12154e3257ad..e656ae9f23c6 100644 --- a/arch/powerpc/platforms/embedded6xx/Makefile +++ b/arch/powerpc/platforms/embedded6xx/Makefile @@ -6,7 +6,6 @@ obj-$(CONFIG_MPC7448HPC2) += mpc7448_hpc2.o obj-$(CONFIG_LINKSTATION) += linkstation.o ls_uart.o obj-$(CONFIG_STORCENTER) += storcenter.o obj-$(CONFIG_PPC_HOLLY) += holly.o -obj-$(CONFIG_PPC_C2K) += c2k.o obj-$(CONFIG_USBGECKO_UDBG) += usbgecko_udbg.o obj-$(CONFIG_GAMECUBE_COMMON) += flipper-pic.o obj-$(CONFIG_GAMECUBE) += gamecube.o diff --git a/arch/powerpc/platforms/embedded6xx/c2k.c b/arch/powerpc/platforms/embedded6xx/c2k.c deleted file mode 100644 index d19e4e759597..000000000000 --- a/arch/powerpc/platforms/embedded6xx/c2k.c +++ /dev/null @@ -1,148 +0,0 @@ -/* - * Board setup routines for the GEFanuc C2K board - * - * Author: Remi Machet - * - * Originated from prpmc2800.c - * - * 2008 (c) Stanford University - * 2007 (c) MontaVista, Software, Inc. - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License version 2 as published - * by the Free Software Foundation. - */ - -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include - -#include - -#include - -#define MV64x60_MPP_CNTL_0 0x0000 -#define MV64x60_MPP_CNTL_2 0x0008 - -#define MV64x60_GPP_IO_CNTL 0x0000 -#define MV64x60_GPP_LEVEL_CNTL 0x0010 -#define MV64x60_GPP_VALUE_SET 0x0018 - -static void __iomem *mv64x60_mpp_reg_base; -static void __iomem *mv64x60_gpp_reg_base; - -static void __init c2k_setup_arch(void) -{ - struct device_node *np; - phys_addr_t paddr; - const unsigned int *reg; - - /* - * ioremap mpp and gpp registers in case they are later - * needed by c2k_reset_board(). - */ - np = of_find_compatible_node(NULL, NULL, "marvell,mv64360-mpp"); - reg = of_get_property(np, "reg", NULL); - paddr = of_translate_address(np, reg); - of_node_put(np); - mv64x60_mpp_reg_base = ioremap(paddr, reg[1]); - - np = of_find_compatible_node(NULL, NULL, "marvell,mv64360-gpp"); - reg = of_get_property(np, "reg", NULL); - paddr = of_translate_address(np, reg); - of_node_put(np); - mv64x60_gpp_reg_base = ioremap(paddr, reg[1]); - -#ifdef CONFIG_PCI - mv64x60_pci_init(); -#endif -} - -static void c2k_reset_board(void) -{ - u32 temp; - - local_irq_disable(); - - temp = in_le32(mv64x60_mpp_reg_base + MV64x60_MPP_CNTL_0); - temp &= 0xFFFF0FFF; - out_le32(mv64x60_mpp_reg_base + MV64x60_MPP_CNTL_0, temp); - - temp = in_le32(mv64x60_gpp_reg_base + MV64x60_GPP_LEVEL_CNTL); - temp |= 0x00000004; - out_le32(mv64x60_gpp_reg_base + MV64x60_GPP_LEVEL_CNTL, temp); - - temp = in_le32(mv64x60_gpp_reg_base + MV64x60_GPP_IO_CNTL); - temp |= 0x00000004; - out_le32(mv64x60_gpp_reg_base + MV64x60_GPP_IO_CNTL, temp); - - temp = in_le32(mv64x60_mpp_reg_base + MV64x60_MPP_CNTL_2); - temp &= 0xFFFF0FFF; - out_le32(mv64x60_mpp_reg_base + MV64x60_MPP_CNTL_2, temp); - - temp = in_le32(mv64x60_gpp_reg_base + MV64x60_GPP_LEVEL_CNTL); - temp |= 0x00080000; - out_le32(mv64x60_gpp_reg_base + MV64x60_GPP_LEVEL_CNTL, temp); - - temp = in_le32(mv64x60_gpp_reg_base + MV64x60_GPP_IO_CNTL); - temp |= 0x00080000; - out_le32(mv64x60_gpp_reg_base + MV64x60_GPP_IO_CNTL, temp); - - out_le32(mv64x60_gpp_reg_base + MV64x60_GPP_VALUE_SET, 0x00080004); -} - -static void __noreturn c2k_restart(char *cmd) -{ - c2k_reset_board(); - msleep(100); - panic("restart failed\n"); -} - -#ifdef CONFIG_NOT_COHERENT_CACHE -#define COHERENCY_SETTING "off" -#else -#define COHERENCY_SETTING "on" -#endif - -void c2k_show_cpuinfo(struct seq_file *m) -{ - seq_printf(m, "Vendor\t\t: GEFanuc\n"); - seq_printf(m, "coherency\t: %s\n", COHERENCY_SETTING); -} - -/* - * Called very early, device-tree isn't unflattened - */ -static int __init c2k_probe(void) -{ - if (!of_machine_is_compatible("GEFanuc,C2K")) - return 0; - - printk(KERN_INFO "Detected a GEFanuc C2K board\n"); - - _set_L2CR(0); - _set_L2CR(L2CR_L2E | L2CR_L2PE | L2CR_L2I); - - mv64x60_init_early(); - - return 1; -} - -define_machine(c2k) { - .name = "C2K", - .probe = c2k_probe, - .setup_arch = c2k_setup_arch, - .show_cpuinfo = c2k_show_cpuinfo, - .init_IRQ = mv64x60_init_irq, - .get_irq = mv64x60_get_irq, - .restart = c2k_restart, - .calibrate_decr = generic_calibrate_decr, -}; diff --git a/arch/powerpc/platforms/embedded6xx/flipper-pic.c b/arch/powerpc/platforms/embedded6xx/flipper-pic.c index 7206f3f573d4..db0be007fd06 100644 --- a/arch/powerpc/platforms/embedded6xx/flipper-pic.c +++ b/arch/powerpc/platforms/embedded6xx/flipper-pic.c @@ -108,16 +108,8 @@ static int flipper_pic_map(struct irq_domain *h, unsigned int virq, return 0; } -static int flipper_pic_match(struct irq_domain *h, struct device_node *np, - enum irq_domain_bus_token bus_token) -{ - return 1; -} - - static const struct irq_domain_ops flipper_irq_domain_ops = { .map = flipper_pic_map, - .match = flipper_pic_match, }; /* diff --git a/arch/powerpc/platforms/embedded6xx/hlwd-pic.c b/arch/powerpc/platforms/embedded6xx/hlwd-pic.c index 89c54de88b7a..8112b39879d6 100644 --- a/arch/powerpc/platforms/embedded6xx/hlwd-pic.c +++ b/arch/powerpc/platforms/embedded6xx/hlwd-pic.c @@ -35,6 +35,8 @@ */ #define HW_BROADWAY_ICR 0x00 #define HW_BROADWAY_IMR 0x04 +#define HW_STARLET_ICR 0x08 +#define HW_STARLET_IMR 0x0c /* @@ -74,6 +76,9 @@ static void hlwd_pic_unmask(struct irq_data *d) void __iomem *io_base = irq_data_get_irq_chip_data(d); setbits32(io_base + HW_BROADWAY_IMR, 1 << irq); + + /* Make sure the ARM (aka. Starlet) doesn't handle this interrupt. */ + clrbits32(io_base + HW_STARLET_IMR, 1 << irq); } @@ -155,7 +160,7 @@ static void __hlwd_quiesce(void __iomem *io_base) out_be32(io_base + HW_BROADWAY_ICR, 0xffffffff); } -struct irq_domain *hlwd_pic_init(struct device_node *np) +static struct irq_domain *hlwd_pic_init(struct device_node *np) { struct irq_domain *irq_domain; struct resource res; diff --git a/arch/powerpc/platforms/maple/maple.h b/arch/powerpc/platforms/maple/maple.h index d10f4af3a42e..4f358b55c341 100644 --- a/arch/powerpc/platforms/maple/maple.h +++ b/arch/powerpc/platforms/maple/maple.h @@ -6,7 +6,7 @@ */ extern int maple_set_rtc_time(struct rtc_time *tm); extern void maple_get_rtc_time(struct rtc_time *tm); -extern unsigned long maple_get_boot_time(void); +extern time64_t maple_get_boot_time(void); extern void maple_calibrate_decr(void); extern void maple_pci_init(void); extern void maple_pci_irq_fixup(struct pci_dev *dev); diff --git a/arch/powerpc/platforms/maple/time.c b/arch/powerpc/platforms/maple/time.c index cfddc87f81bf..becf2ebf7df5 100644 --- a/arch/powerpc/platforms/maple/time.c +++ b/arch/powerpc/platforms/maple/time.c @@ -137,7 +137,7 @@ static struct resource rtc_iores = { .flags = IORESOURCE_IO | IORESOURCE_BUSY, }; -unsigned long __init maple_get_boot_time(void) +time64_t __init maple_get_boot_time(void) { struct rtc_time tm; struct device_node *rtcs; @@ -170,7 +170,6 @@ unsigned long __init maple_get_boot_time(void) request_resource(&ioport_resource, &rtc_iores); maple_get_rtc_time(&tm); - return mktime(tm.tm_year+1900, tm.tm_mon+1, tm.tm_mday, - tm.tm_hour, tm.tm_min, tm.tm_sec); + return rtc_tm_to_time64(&tm); } diff --git a/arch/powerpc/platforms/pasemi/pasemi.h b/arch/powerpc/platforms/pasemi/pasemi.h index 329d2a619254..70b56048ed1b 100644 --- a/arch/powerpc/platforms/pasemi/pasemi.h +++ b/arch/powerpc/platforms/pasemi/pasemi.h @@ -2,7 +2,7 @@ #ifndef _PASEMI_PASEMI_H #define _PASEMI_PASEMI_H -extern unsigned long pas_get_boot_time(void); +extern time64_t pas_get_boot_time(void); extern void pas_pci_init(void); extern void pas_pci_irq_fixup(struct pci_dev *dev); extern void pas_pci_dma_dev_setup(struct pci_dev *dev); diff --git a/arch/powerpc/platforms/pasemi/pci.c b/arch/powerpc/platforms/pasemi/pci.c index 5ff6108f19e9..aea9ff2c8e6d 100644 --- a/arch/powerpc/platforms/pasemi/pci.c +++ b/arch/powerpc/platforms/pasemi/pci.c @@ -224,6 +224,8 @@ void __init pas_pci_init(void) return; } + pci_set_flags(PCI_SCAN_ALL_PCIE_DEVS); + for (np = NULL; (np = of_get_next_child(root, np)) != NULL;) if (np->name && !strcmp(np->name, "pxp") && !pas_add_bridge(np)) of_node_get(np); diff --git a/arch/powerpc/platforms/pasemi/time.c b/arch/powerpc/platforms/pasemi/time.c index fa54351ac268..ea815254ee7b 100644 --- a/arch/powerpc/platforms/pasemi/time.c +++ b/arch/powerpc/platforms/pasemi/time.c @@ -21,8 +21,8 @@ #include -unsigned long __init pas_get_boot_time(void) +time64_t __init pas_get_boot_time(void) { /* Let's just return a fake date right now */ - return mktime(2006, 1, 1, 12, 0, 0); + return mktime64(2006, 1, 1, 12, 0, 0); } diff --git a/arch/powerpc/platforms/powermac/bootx_init.c b/arch/powerpc/platforms/powermac/bootx_init.c index c3c9bbb3573a..3b3b0b9b3577 100644 --- a/arch/powerpc/platforms/powermac/bootx_init.c +++ b/arch/powerpc/platforms/powermac/bootx_init.c @@ -468,7 +468,7 @@ void __init bootx_init(unsigned long r3, unsigned long r4) boot_infos_t *bi = (boot_infos_t *) r4; unsigned long hdr; unsigned long space; - unsigned long ptr, x; + unsigned long ptr; char *model; unsigned long offset = reloc_offset(); @@ -519,7 +519,7 @@ void __init bootx_init(unsigned long r3, unsigned long r4) ; } if (bi->architecture != BOOT_ARCH_PCI) { - bootx_printf(" !!! WARNING - Usupported machine" + bootx_printf(" !!! WARNING - Unsupported machine" " architecture !\n"); for (;;) ; @@ -562,6 +562,8 @@ void __init bootx_init(unsigned long r3, unsigned long r4) * MMU switched OFF, so this should not be useful anymore. */ if (bi->version < 4) { + unsigned long x __maybe_unused; + bootx_printf("Touching pages...\n"); /* diff --git a/arch/powerpc/platforms/powermac/pci.c b/arch/powerpc/platforms/powermac/pci.c index 0b8174a79993..df762bb3c735 100644 --- a/arch/powerpc/platforms/powermac/pci.c +++ b/arch/powerpc/platforms/powermac/pci.c @@ -62,7 +62,7 @@ struct device_node *k2_skiplist[2]; static int __init fixup_one_level_bus_range(struct device_node *node, int higher) { - for (; node != 0;node = node->sibling) { + for (; node; node = node->sibling) { const int * bus_range; const unsigned int *class_code; int len; @@ -1219,7 +1219,7 @@ static void fixup_u4_pcie(struct pci_dev* dev) region = r; } /* Nothing found, bail */ - if (region == 0) + if (!region) return; /* Print things out */ diff --git a/arch/powerpc/platforms/powermac/pmac.h b/arch/powerpc/platforms/powermac/pmac.h index 6f15b8804e9b..16a52afdb76e 100644 --- a/arch/powerpc/platforms/powermac/pmac.h +++ b/arch/powerpc/platforms/powermac/pmac.h @@ -15,7 +15,7 @@ struct rtc_time; extern int pmac_newworld; extern long pmac_time_init(void); -extern unsigned long pmac_get_boot_time(void); +extern time64_t pmac_get_boot_time(void); extern void pmac_get_rtc_time(struct rtc_time *); extern int pmac_set_rtc_time(struct rtc_time *); extern void pmac_read_rtc_time(void); diff --git a/arch/powerpc/platforms/powermac/setup.c b/arch/powerpc/platforms/powermac/setup.c index ab668cb72263..3a529fcdae97 100644 --- a/arch/powerpc/platforms/powermac/setup.c +++ b/arch/powerpc/platforms/powermac/setup.c @@ -152,7 +152,7 @@ static void pmac_show_cpuinfo(struct seq_file *m) of_get_property(np, "d-cache-size", NULL); seq_printf(m, "L2 cache\t:"); has_l2cache = 1; - if (of_get_property(np, "cache-unified", NULL) != 0 && dc) { + if (of_get_property(np, "cache-unified", NULL) && dc) { seq_printf(m, " %dK unified", *dc / 1024); } else { if (ic) @@ -244,12 +244,12 @@ static void __init l2cr_init(void) /* Checks "l2cr-value" property in the registry */ if (cpu_has_feature(CPU_FTR_L2CR)) { struct device_node *np = of_find_node_by_name(NULL, "cpus"); - if (np == 0) + if (!np) np = of_find_node_by_type(NULL, "cpu"); - if (np != 0) { + if (np) { const unsigned int *l2cr = of_get_property(np, "l2cr-value", NULL); - if (l2cr != 0) { + if (l2cr) { ppc_override_l2cr = 1; ppc_override_l2cr_value = *l2cr; _set_L2CR(0); @@ -352,6 +352,7 @@ static int pmac_late_init(void) } machine_late_initcall(powermac, pmac_late_init); +void note_bootable_part(dev_t dev, int part, int goodness); /* * This is __ref because we check for "initializing" before * touching any of the __init sensitive things and "initializing" diff --git a/arch/powerpc/platforms/powermac/smp.c b/arch/powerpc/platforms/powermac/smp.c index 95275e0e2efa..447da6db450a 100644 --- a/arch/powerpc/platforms/powermac/smp.c +++ b/arch/powerpc/platforms/powermac/smp.c @@ -65,7 +65,6 @@ #endif extern void __secondary_start_pmac_0(void); -extern int pmac_pfunc_base_install(void); static void (*pmac_tb_freeze)(int freeze); static u64 timebase; diff --git a/arch/powerpc/platforms/powermac/time.c b/arch/powerpc/platforms/powermac/time.c index 274af6fa388e..7c968e46736f 100644 --- a/arch/powerpc/platforms/powermac/time.c +++ b/arch/powerpc/platforms/powermac/time.c @@ -84,29 +84,11 @@ long __init pmac_time_init(void) return delta; } -#if defined(CONFIG_ADB_CUDA) || defined(CONFIG_ADB_PMU) -static void to_rtc_time(unsigned long now, struct rtc_time *tm) -{ - to_tm(now, tm); - tm->tm_year -= 1900; - tm->tm_mon -= 1; -} -#endif - -#if defined(CONFIG_ADB_CUDA) || defined(CONFIG_ADB_PMU) || \ - defined(CONFIG_PMAC_SMU) -static unsigned long from_rtc_time(struct rtc_time *tm) -{ - return mktime(tm->tm_year+1900, tm->tm_mon+1, tm->tm_mday, - tm->tm_hour, tm->tm_min, tm->tm_sec); -} -#endif - #ifdef CONFIG_ADB_CUDA -static unsigned long cuda_get_time(void) +static time64_t cuda_get_time(void) { struct adb_request req; - unsigned int now; + time64_t now; if (cuda_request(&req, NULL, 2, CUDA_PACKET, CUDA_GET_TIME) < 0) return 0; @@ -117,17 +99,17 @@ static unsigned long cuda_get_time(void) req.reply_len); now = (req.reply[3] << 24) + (req.reply[4] << 16) + (req.reply[5] << 8) + req.reply[6]; - return ((unsigned long)now) - RTC_OFFSET; + return now - RTC_OFFSET; } -#define cuda_get_rtc_time(tm) to_rtc_time(cuda_get_time(), (tm)) +#define cuda_get_rtc_time(tm) rtc_time64_to_tm(cuda_get_time(), (tm)) static int cuda_set_rtc_time(struct rtc_time *tm) { - unsigned int nowtime; + time64_t nowtime; struct adb_request req; - nowtime = from_rtc_time(tm) + RTC_OFFSET; + nowtime = rtc_tm_to_time64(tm) + RTC_OFFSET; if (cuda_request(&req, NULL, 6, CUDA_PACKET, CUDA_SET_TIME, nowtime >> 24, nowtime >> 16, nowtime >> 8, nowtime) < 0) @@ -147,10 +129,10 @@ static int cuda_set_rtc_time(struct rtc_time *tm) #endif #ifdef CONFIG_ADB_PMU -static unsigned long pmu_get_time(void) +static time64_t pmu_get_time(void) { struct adb_request req; - unsigned int now; + time64_t now; if (pmu_request(&req, NULL, 1, PMU_READ_RTC) < 0) return 0; @@ -160,17 +142,17 @@ static unsigned long pmu_get_time(void) req.reply_len); now = (req.reply[0] << 24) + (req.reply[1] << 16) + (req.reply[2] << 8) + req.reply[3]; - return ((unsigned long)now) - RTC_OFFSET; + return now - RTC_OFFSET; } -#define pmu_get_rtc_time(tm) to_rtc_time(pmu_get_time(), (tm)) +#define pmu_get_rtc_time(tm) rtc_time64_to_tm(pmu_get_time(), (tm)) static int pmu_set_rtc_time(struct rtc_time *tm) { - unsigned int nowtime; + time64_t nowtime; struct adb_request req; - nowtime = from_rtc_time(tm) + RTC_OFFSET; + nowtime = rtc_tm_to_time64(tm) + RTC_OFFSET; if (pmu_request(&req, NULL, 5, PMU_SET_RTC, nowtime >> 24, nowtime >> 16, nowtime >> 8, nowtime) < 0) return -ENXIO; @@ -188,13 +170,13 @@ static int pmu_set_rtc_time(struct rtc_time *tm) #endif #ifdef CONFIG_PMAC_SMU -static unsigned long smu_get_time(void) +static time64_t smu_get_time(void) { struct rtc_time tm; if (smu_get_rtc_time(&tm, 1)) return 0; - return from_rtc_time(&tm); + return rtc_tm_to_time64(&tm); } #else @@ -204,7 +186,7 @@ static unsigned long smu_get_time(void) #endif /* Can't be __init, it's called when suspending and resuming */ -unsigned long pmac_get_boot_time(void) +time64_t pmac_get_boot_time(void) { /* Get the time from the RTC, used only at boot time */ switch (sys_ctrler) { diff --git a/arch/powerpc/platforms/powernv/copy-paste.h b/arch/powerpc/platforms/powernv/copy-paste.h index c9a503623431..cb36f9fbcef3 100644 --- a/arch/powerpc/platforms/powernv/copy-paste.h +++ b/arch/powerpc/platforms/powernv/copy-paste.h @@ -7,9 +7,8 @@ * 2 of the License, or (at your option) any later version. */ #include +#include -#define CR0_SHIFT 28 -#define CR0_MASK 0xF /* * Copy/paste instructions: * @@ -42,5 +41,6 @@ static inline int vas_paste(void *paste_address, int offset) : "b" (offset), "b" (paste_address) : "memory", "cr0"); - return (cr >> CR0_SHIFT) & CR0_MASK; + /* We mask with 0xE to ignore SO */ + return (cr >> CR0_SHIFT) & 0xE; } diff --git a/arch/powerpc/platforms/powernv/idle.c b/arch/powerpc/platforms/powernv/idle.c index 1f12ab1e6030..1c5d0675b43c 100644 --- a/arch/powerpc/platforms/powernv/idle.c +++ b/arch/powerpc/platforms/powernv/idle.c @@ -79,7 +79,7 @@ static int pnv_save_sprs_for_deep_states(void) uint64_t msr_val = MSR_IDLE; uint64_t psscr_val = pnv_deepest_stop_psscr_val; - for_each_possible_cpu(cpu) { + for_each_present_cpu(cpu) { uint64_t pir = get_hard_smp_processor_id(cpu); uint64_t hsprg0_val = (uint64_t)paca_ptrs[cpu]; @@ -814,7 +814,7 @@ static int __init pnv_init_idle_states(void) int cpu; pr_info("powernv: idle: Saving PACA pointers of all CPUs in their thread sibling PACA\n"); - for_each_possible_cpu(cpu) { + for_each_present_cpu(cpu) { int base_cpu = cpu_first_thread_sibling(cpu); int idx = cpu_thread_in_core(cpu); int i; diff --git a/arch/powerpc/platforms/powernv/memtrace.c b/arch/powerpc/platforms/powernv/memtrace.c index fc222a0c2ac4..b99283df8584 100644 --- a/arch/powerpc/platforms/powernv/memtrace.c +++ b/arch/powerpc/platforms/powernv/memtrace.c @@ -131,7 +131,7 @@ static u64 memtrace_alloc_node(u32 nid, u64 size) u64 start_pfn, end_pfn, nr_pages; u64 base_pfn; - if (!NODE_DATA(nid) || !node_spanned_pages(nid)) + if (!node_spanned_pages(nid)) return 0; start_pfn = node_start_pfn(nid); diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c index 525e966dce34..8cdf91f5d3a4 100644 --- a/arch/powerpc/platforms/powernv/npu-dma.c +++ b/arch/powerpc/platforms/powernv/npu-dma.c @@ -459,10 +459,9 @@ static void mmio_launch_invalidate(struct mmio_atsd_reg *mmio_atsd_reg, struct npu *npu = mmio_atsd_reg->npu; int reg = mmio_atsd_reg->reg; - __raw_writeq(cpu_to_be64(va), - npu->mmio_atsd_regs[reg] + XTS_ATSD_AVA); + __raw_writeq_be(va, npu->mmio_atsd_regs[reg] + XTS_ATSD_AVA); eieio(); - __raw_writeq(cpu_to_be64(launch), npu->mmio_atsd_regs[reg]); + __raw_writeq_be(launch, npu->mmio_atsd_regs[reg]); } static void mmio_invalidate_pid(struct mmio_atsd_reg mmio_atsd_reg[NV_MAX_NPUS], diff --git a/arch/powerpc/platforms/powernv/ocxl.c b/arch/powerpc/platforms/powernv/ocxl.c index fa9b53af3c7b..8c65aacda9c8 100644 --- a/arch/powerpc/platforms/powernv/ocxl.c +++ b/arch/powerpc/platforms/powernv/ocxl.c @@ -475,7 +475,7 @@ void pnv_ocxl_spa_release(void *platform_data) } EXPORT_SYMBOL_GPL(pnv_ocxl_spa_release); -int pnv_ocxl_spa_remove_pe(void *platform_data, int pe_handle) +int pnv_ocxl_spa_remove_pe_from_cache(void *platform_data, int pe_handle) { struct spa_data *data = (struct spa_data *) platform_data; int rc; @@ -483,7 +483,7 @@ int pnv_ocxl_spa_remove_pe(void *platform_data, int pe_handle) rc = opal_npu_spa_clear_cache(data->phb_opal_id, data->bdfn, pe_handle); return rc; } -EXPORT_SYMBOL_GPL(pnv_ocxl_spa_remove_pe); +EXPORT_SYMBOL_GPL(pnv_ocxl_spa_remove_pe_from_cache); int pnv_ocxl_alloc_xive_irq(u32 *irq, u64 *trigger_addr) { diff --git a/arch/powerpc/platforms/powernv/opal-hmi.c b/arch/powerpc/platforms/powernv/opal-hmi.c index 4efc95b4c7d4..586ec71a4e17 100644 --- a/arch/powerpc/platforms/powernv/opal-hmi.c +++ b/arch/powerpc/platforms/powernv/opal-hmi.c @@ -177,7 +177,7 @@ static void print_hmi_event_info(struct OpalHMIEvent *hmi_evt) "Processor recovery occurred for masked error", "Timer facility experienced an error", "TFMR SPR is corrupted", - "UPS (Uniterrupted Power System) Overflow indication", + "UPS (Uninterrupted Power System) Overflow indication", "An XSCOM operation failure", "An XSCOM operation completed", "SCOM has set a reserved FIR bit to cause recovery", diff --git a/arch/powerpc/platforms/powernv/opal-imc.c b/arch/powerpc/platforms/powernv/opal-imc.c index 2a14fda5ea26..58a07948c76e 100644 --- a/arch/powerpc/platforms/powernv/opal-imc.c +++ b/arch/powerpc/platforms/powernv/opal-imc.c @@ -115,8 +115,10 @@ static int imc_get_mem_addr_nest(struct device_node *node, return -ENOMEM; chipid_arr = kcalloc(nr_chips, sizeof(*chipid_arr), GFP_KERNEL); - if (!chipid_arr) + if (!chipid_arr) { + kfree(base_addr_arr); return -ENOMEM; + } if (of_property_read_u32_array(node, "chip-id", chipid_arr, nr_chips)) goto error; @@ -143,7 +145,6 @@ static int imc_get_mem_addr_nest(struct device_node *node, return 0; error: - kfree(pmu_ptr->mem_info); kfree(base_addr_arr); kfree(chipid_arr); return -1; @@ -183,8 +184,14 @@ static int imc_pmu_create(struct device_node *parent, int pmu_index, int domain) /* Function to register IMC pmu */ ret = init_imc_pmu(parent, pmu_ptr, pmu_index); - if (ret) + if (ret) { pr_err("IMC PMU %s Register failed\n", pmu_ptr->pmu.name); + kfree(pmu_ptr->pmu.name); + if (pmu_ptr->domain == IMC_DOMAIN_NEST) + kfree(pmu_ptr->mem_info); + kfree(pmu_ptr); + return ret; + } return 0; @@ -248,6 +255,7 @@ static int opal_imc_counters_probe(struct platform_device *pdev) { struct device_node *imc_dev = pdev->dev.of_node; int pmu_count = 0, domain; + bool core_imc_reg = false, thread_imc_reg = false; u32 type; /* @@ -285,6 +293,10 @@ static int opal_imc_counters_probe(struct platform_device *pdev) if (!imc_pmu_create(imc_dev, pmu_count, domain)) { if (domain == IMC_DOMAIN_NEST) pmu_count++; + if (domain == IMC_DOMAIN_CORE) + core_imc_reg = true; + if (domain == IMC_DOMAIN_THREAD) + thread_imc_reg = true; } } @@ -292,6 +304,10 @@ static int opal_imc_counters_probe(struct platform_device *pdev) if (pmu_count == 0) debugfs_remove_recursive(imc_debugfs_parent); + /* If core imc is not registered, unregister thread-imc */ + if (!core_imc_reg && thread_imc_reg) + unregister_thread_imc(); + return 0; } diff --git a/arch/powerpc/platforms/powernv/opal-irqchip.c b/arch/powerpc/platforms/powernv/opal-irqchip.c index 9d1b8c0aaf93..605c7e5d52c2 100644 --- a/arch/powerpc/platforms/powernv/opal-irqchip.c +++ b/arch/powerpc/platforms/powernv/opal-irqchip.c @@ -22,7 +22,6 @@ #include #include #include -#include #include #include @@ -38,37 +37,47 @@ struct opal_event_irqchip { unsigned long mask; }; static struct opal_event_irqchip opal_event_irqchip; - +static u64 last_outstanding_events; static unsigned int opal_irq_count; static unsigned int *opal_irqs; -static void opal_handle_irq_work(struct irq_work *work); -static u64 last_outstanding_events; -static struct irq_work opal_event_irq_work = { - .func = opal_handle_irq_work, -}; - -void opal_handle_events(uint64_t events) +void opal_handle_events(void) { - int virq, hwirq = 0; - u64 mask = opal_event_irqchip.mask; + __be64 events = 0; + u64 e; + + e = READ_ONCE(last_outstanding_events) & opal_event_irqchip.mask; +again: + while (e) { + int virq, hwirq; + + hwirq = fls64(e) - 1; + e &= ~BIT_ULL(hwirq); + + local_irq_disable(); + virq = irq_find_mapping(opal_event_irqchip.domain, hwirq); + if (virq) { + irq_enter(); + generic_handle_irq(virq); + irq_exit(); + } + local_irq_enable(); - if (!in_irq() && (events & mask)) { - last_outstanding_events = events; - irq_work_queue(&opal_event_irq_work); - return; + cond_resched(); } + last_outstanding_events = 0; + if (opal_poll_events(&events) != OPAL_SUCCESS) + return; + e = be64_to_cpu(events) & opal_event_irqchip.mask; + if (e) + goto again; +} - while (events & mask) { - hwirq = fls64(events) - 1; - if (BIT_ULL(hwirq) & mask) { - virq = irq_find_mapping(opal_event_irqchip.domain, - hwirq); - if (virq) - generic_handle_irq(virq); - } - events &= ~BIT_ULL(hwirq); - } +bool opal_have_pending_events(void) +{ + if (last_outstanding_events & opal_event_irqchip.mask) + return true; + return false; } static void opal_event_mask(struct irq_data *d) @@ -78,24 +87,9 @@ static void opal_event_mask(struct irq_data *d) static void opal_event_unmask(struct irq_data *d) { - __be64 events; - set_bit(d->hwirq, &opal_event_irqchip.mask); - - opal_poll_events(&events); - last_outstanding_events = be64_to_cpu(events); - - /* - * We can't just handle the events now with opal_handle_events(). - * If we did we would deadlock when opal_event_unmask() is called from - * handle_level_irq() with the irq descriptor lock held, because - * calling opal_handle_events() would call generic_handle_irq() and - * then handle_level_irq() which would try to take the descriptor lock - * again. Instead queue the events for later. - */ - if (last_outstanding_events & opal_event_irqchip.mask) - /* Need to retrigger the interrupt */ - irq_work_queue(&opal_event_irq_work); + if (opal_have_pending_events()) + opal_wake_poller(); } static int opal_event_set_type(struct irq_data *d, unsigned int flow_type) @@ -136,16 +130,13 @@ static irqreturn_t opal_interrupt(int irq, void *data) __be64 events; opal_handle_interrupt(virq_to_hw(irq), &events); - opal_handle_events(be64_to_cpu(events)); + last_outstanding_events = be64_to_cpu(events); + if (opal_have_pending_events()) + opal_wake_poller(); return IRQ_HANDLED; } -static void opal_handle_irq_work(struct irq_work *work) -{ - opal_handle_events(last_outstanding_events); -} - static int opal_event_match(struct irq_domain *h, struct device_node *node, enum irq_domain_bus_token bus_token) { @@ -177,7 +168,7 @@ void opal_event_shutdown(void) if (!opal_irqs[i]) continue; - if (in_interrupt()) + if (in_interrupt() || irqs_disabled()) disable_irq_nosync(opal_irqs[i]); else free_irq(opal_irqs[i], NULL); diff --git a/arch/powerpc/platforms/powernv/opal-rtc.c b/arch/powerpc/platforms/powernv/opal-rtc.c index aa2a5139462e..42ec642a3eba 100644 --- a/arch/powerpc/platforms/powernv/opal-rtc.c +++ b/arch/powerpc/platforms/powernv/opal-rtc.c @@ -34,7 +34,7 @@ static void opal_to_tm(u32 y_m_d, u64 h_m_s_ms, struct rtc_time *tm) tm->tm_wday = -1; } -unsigned long __init opal_get_boot_time(void) +time64_t __init opal_get_boot_time(void) { struct rtc_time tm; u32 y_m_d; @@ -61,8 +61,7 @@ unsigned long __init opal_get_boot_time(void) y_m_d = be32_to_cpu(__y_m_d); h_m_s_ms = be64_to_cpu(__h_m_s_ms); opal_to_tm(y_m_d, h_m_s_ms, &tm); - return mktime(tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday, - tm.tm_hour, tm.tm_min, tm.tm_sec); + return rtc_tm_to_time64(&tm); } static __init int opal_time_init(void) diff --git a/arch/powerpc/platforms/powernv/opal-sensor.c b/arch/powerpc/platforms/powernv/opal-sensor.c index 0a7074bb91dc..35a5f4b9aeb5 100644 --- a/arch/powerpc/platforms/powernv/opal-sensor.c +++ b/arch/powerpc/platforms/powernv/opal-sensor.c @@ -72,6 +72,59 @@ out: } EXPORT_SYMBOL_GPL(opal_get_sensor_data); +int opal_get_sensor_data_u64(u32 sensor_hndl, u64 *sensor_data) +{ + int ret, token; + struct opal_msg msg; + __be64 data; + + if (!opal_check_token(OPAL_SENSOR_READ_U64)) { + u32 sdata; + + ret = opal_get_sensor_data(sensor_hndl, &sdata); + if (!ret) + *sensor_data = sdata; + return ret; + } + + token = opal_async_get_token_interruptible(); + if (token < 0) + return token; + + ret = opal_sensor_read_u64(sensor_hndl, token, &data); + switch (ret) { + case OPAL_ASYNC_COMPLETION: + ret = opal_async_wait_response(token, &msg); + if (ret) { + pr_err("%s: Failed to wait for the async response, %d\n", + __func__, ret); + goto out_token; + } + + ret = opal_error_code(opal_get_async_rc(msg)); + *sensor_data = be64_to_cpu(data); + break; + + case OPAL_SUCCESS: + ret = 0; + *sensor_data = be64_to_cpu(data); + break; + + case OPAL_WRONG_STATE: + ret = -EIO; + break; + + default: + ret = opal_error_code(ret); + break; + } + +out_token: + opal_async_release_token(token); + return ret; +} +EXPORT_SYMBOL_GPL(opal_get_sensor_data_u64); + int __init opal_sensor_init(void) { struct platform_device *pdev; diff --git a/arch/powerpc/platforms/powernv/opal-sysparam.c b/arch/powerpc/platforms/powernv/opal-sysparam.c index 6fd4092798d5..9aa87df114fd 100644 --- a/arch/powerpc/platforms/powernv/opal-sysparam.c +++ b/arch/powerpc/platforms/powernv/opal-sysparam.c @@ -198,21 +198,21 @@ void __init opal_sys_param_init(void) goto out_param_buf; } - id = kzalloc(sizeof(*id) * count, GFP_KERNEL); + id = kcalloc(count, sizeof(*id), GFP_KERNEL); if (!id) { pr_err("SYSPARAM: Failed to allocate memory to read parameter " "id\n"); goto out_param_buf; } - size = kzalloc(sizeof(*size) * count, GFP_KERNEL); + size = kcalloc(count, sizeof(*size), GFP_KERNEL); if (!size) { pr_err("SYSPARAM: Failed to allocate memory to read parameter " "size\n"); goto out_free_id; } - perm = kzalloc(sizeof(*perm) * count, GFP_KERNEL); + perm = kcalloc(count, sizeof(*perm), GFP_KERNEL); if (!perm) { pr_err("SYSPARAM: Failed to allocate memory to read supported " "action on the parameter"); @@ -235,7 +235,7 @@ void __init opal_sys_param_init(void) goto out_free_perm; } - attr = kzalloc(sizeof(*attr) * count, GFP_KERNEL); + attr = kcalloc(count, sizeof(*attr), GFP_KERNEL); if (!attr) { pr_err("SYSPARAM: Failed to allocate memory for parameter " "attributes\n"); diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S index 3da30c2f26b4..a8d9b4089c31 100644 --- a/arch/powerpc/platforms/powernv/opal-wrappers.S +++ b/arch/powerpc/platforms/powernv/opal-wrappers.S @@ -320,8 +320,10 @@ OPAL_CALL(opal_set_powercap, OPAL_SET_POWERCAP); OPAL_CALL(opal_get_power_shift_ratio, OPAL_GET_POWER_SHIFT_RATIO); OPAL_CALL(opal_set_power_shift_ratio, OPAL_SET_POWER_SHIFT_RATIO); OPAL_CALL(opal_sensor_group_clear, OPAL_SENSOR_GROUP_CLEAR); +OPAL_CALL(opal_quiesce, OPAL_QUIESCE); OPAL_CALL(opal_npu_spa_setup, OPAL_NPU_SPA_SETUP); OPAL_CALL(opal_npu_spa_clear_cache, OPAL_NPU_SPA_CLEAR_CACHE); OPAL_CALL(opal_npu_tl_set, OPAL_NPU_TL_SET); OPAL_CALL(opal_pci_get_pbcq_tunnel_bar, OPAL_PCI_GET_PBCQ_TUNNEL_BAR); OPAL_CALL(opal_pci_set_pbcq_tunnel_bar, OPAL_PCI_SET_PBCQ_TUNNEL_BAR); +OPAL_CALL(opal_sensor_read_u64, OPAL_SENSOR_READ_U64); diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c index 48fbb41af5d1..0d539c661748 100644 --- a/arch/powerpc/platforms/powernv/opal.c +++ b/arch/powerpc/platforms/powernv/opal.c @@ -540,21 +540,15 @@ int opal_hmi_exception_early(struct pt_regs *regs) /* HMI exception handler called in virtual mode during check_irq_replay. */ int opal_handle_hmi_exception(struct pt_regs *regs) { - s64 rc; - __be64 evt = 0; - /* * Check if HMI event is available. - * if Yes, then call opal_poll_events to pull opal messages and - * process them. + * if Yes, then wake kopald to process them. */ if (!local_paca->hmi_event_available) return 0; local_paca->hmi_event_available = 0; - rc = opal_poll_events(&evt); - if (rc == OPAL_SUCCESS && evt) - opal_handle_events(be64_to_cpu(evt)); + opal_wake_poller(); return 1; } @@ -757,14 +751,19 @@ static void __init opal_imc_init_dev(void) static int kopald(void *unused) { unsigned long timeout = msecs_to_jiffies(opal_heartbeat) + 1; - __be64 events; set_freezable(); do { try_to_freeze(); - opal_poll_events(&events); - opal_handle_events(be64_to_cpu(events)); - schedule_timeout_interruptible(timeout); + + opal_handle_events(); + + set_current_state(TASK_INTERRUPTIBLE); + if (opal_have_pending_events()) + __set_current_state(TASK_RUNNING); + else + schedule_timeout(timeout); + } while (!kthread_should_stop()); return 0; diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c index 3f9c69d7623a..5bd0eb6681bc 100644 --- a/arch/powerpc/platforms/powernv/pci-ioda.c +++ b/arch/powerpc/platforms/powernv/pci-ioda.c @@ -1976,9 +1976,10 @@ static void pnv_pci_p7ioc_tce_invalidate(struct iommu_table *tbl, mb(); /* Ensure above stores are visible */ while (start <= end) { if (rm) - __raw_rm_writeq(cpu_to_be64(start), invalidate); + __raw_rm_writeq_be(start, invalidate); else - __raw_writeq(cpu_to_be64(start), invalidate); + __raw_writeq_be(start, invalidate); + start += inc; } @@ -2055,9 +2056,9 @@ static void pnv_pci_phb3_tce_invalidate_entire(struct pnv_phb *phb, bool rm) mb(); /* Ensure previous TCE table stores are visible */ if (rm) - __raw_rm_writeq(cpu_to_be64(val), invalidate); + __raw_rm_writeq_be(val, invalidate); else - __raw_writeq(cpu_to_be64(val), invalidate); + __raw_writeq_be(val, invalidate); } static inline void pnv_pci_phb3_tce_invalidate_pe(struct pnv_ioda_pe *pe) @@ -2067,7 +2068,7 @@ static inline void pnv_pci_phb3_tce_invalidate_pe(struct pnv_ioda_pe *pe) unsigned long val = PHB3_TCE_KILL_INVAL_PE | (pe->pe_number & 0xFF); mb(); /* Ensure above stores are visible */ - __raw_writeq(cpu_to_be64(val), invalidate); + __raw_writeq_be(val, invalidate); } static void pnv_pci_phb3_tce_invalidate(struct pnv_ioda_pe *pe, bool rm, @@ -2090,9 +2091,9 @@ static void pnv_pci_phb3_tce_invalidate(struct pnv_ioda_pe *pe, bool rm, while (start <= end) { if (rm) - __raw_rm_writeq(cpu_to_be64(start), invalidate); + __raw_rm_writeq_be(start, invalidate); else - __raw_writeq(cpu_to_be64(start), invalidate); + __raw_writeq_be(start, invalidate); start += inc; } } @@ -2910,6 +2911,34 @@ static void pnv_pci_ioda2_table_free_pages(struct iommu_table *tbl) tbl->it_indirect_levels); } +static unsigned long pnv_ioda_parse_tce_sizes(struct pnv_phb *phb) +{ + struct pci_controller *hose = phb->hose; + struct device_node *dn = hose->dn; + unsigned long mask = 0; + int i, rc, count; + u32 val; + + count = of_property_count_u32_elems(dn, "ibm,supported-tce-sizes"); + if (count <= 0) { + mask = SZ_4K | SZ_64K; + /* Add 16M for POWER8 by default */ + if (cpu_has_feature(CPU_FTR_ARCH_207S) && + !cpu_has_feature(CPU_FTR_ARCH_300)) + mask |= SZ_16M; + return mask; + } + + for (i = 0; i < count; i++) { + rc = of_property_read_u32_index(dn, "ibm,supported-tce-sizes", + i, &val); + if (rc == 0) + mask |= 1ULL << val; + } + + return mask; +} + static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb, struct pnv_ioda_pe *pe) { @@ -2934,7 +2963,7 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb, pe->table_group.max_dynamic_windows_supported = IOMMU_TABLE_GROUP_MAX_TABLES; pe->table_group.max_levels = POWERNV_IOMMU_MAX_LEVELS; - pe->table_group.pgsizes = SZ_4K | SZ_64K | SZ_16M; + pe->table_group.pgsizes = pnv_ioda_parse_tce_sizes(phb); #ifdef CONFIG_IOMMU_API pe->table_group.ops = &pnv_pci_ioda2_ops; #endif @@ -3642,7 +3671,6 @@ static void pnv_pci_ioda2_release_pe_dma(struct pnv_ioda_pe *pe) WARN_ON(pe->table_group.group); } - pnv_pci_ioda2_table_free_pages(tbl); iommu_tce_table_put(tbl); } diff --git a/arch/powerpc/platforms/powernv/powernv.h b/arch/powerpc/platforms/powernv/powernv.h index 94f17ab1374b..fd4a1c5a6369 100644 --- a/arch/powerpc/platforms/powernv/powernv.h +++ b/arch/powerpc/platforms/powernv/powernv.h @@ -24,7 +24,8 @@ extern u32 pnv_get_supported_cpuidle_states(void); extern void pnv_lpc_init(void); -extern void opal_handle_events(uint64_t events); +extern void opal_handle_events(void); +extern bool opal_have_pending_events(void); extern void opal_event_shutdown(void); bool cpu_core_split_required(void); diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c index a6648ec99ca7..f96df0a25d05 100644 --- a/arch/powerpc/platforms/powernv/setup.c +++ b/arch/powerpc/platforms/powernv/setup.c @@ -124,6 +124,7 @@ static void pnv_setup_rfi_flush(void) security_ftr_enabled(SEC_FTR_L1D_FLUSH_HV)); setup_rfi_flush(type, enable); + setup_barrier_nospec(); } static void __init pnv_setup_arch(void) @@ -357,15 +358,7 @@ static void pnv_kexec_cpu_down(int crash_shutdown, int secondary) #ifdef CONFIG_MEMORY_HOTPLUG_SPARSE static unsigned long pnv_memory_block_size(void) { - /* - * We map the kernel linear region with 1GB large pages on radix. For - * memory hot unplug to work our memory block size must be at least - * this size. - */ - if (radix_enabled()) - return 1UL * 1024 * 1024 * 1024; - else - return 256UL * 1024 * 1024; + return 256UL * 1024 * 1024; } #endif diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c index 19af6de6b6f0..b80909957792 100644 --- a/arch/powerpc/platforms/powernv/smp.c +++ b/arch/powerpc/platforms/powernv/smp.c @@ -334,7 +334,16 @@ static int pnv_cause_nmi_ipi(int cpu) int64_t rc; if (cpu >= 0) { - rc = opal_signal_system_reset(get_hard_smp_processor_id(cpu)); + int h = get_hard_smp_processor_id(cpu); + + if (opal_check_token(OPAL_QUIESCE)) + opal_quiesce(QUIESCE_HOLD, h); + + rc = opal_signal_system_reset(h); + + if (opal_check_token(OPAL_QUIESCE)) + opal_quiesce(QUIESCE_RESUME, h); + if (rc != OPAL_SUCCESS) return 0; return 1; @@ -343,6 +352,8 @@ static int pnv_cause_nmi_ipi(int cpu) bool success = true; int c; + if (opal_check_token(OPAL_QUIESCE)) + opal_quiesce(QUIESCE_HOLD, -1); /* * We do not use broadcasts (yet), because it's not clear @@ -358,6 +369,10 @@ static int pnv_cause_nmi_ipi(int cpu) if (rc != OPAL_SUCCESS) success = false; } + + if (opal_check_token(OPAL_QUIESCE)) + opal_quiesce(QUIESCE_RESUME, -1); + if (success) return 1; diff --git a/arch/powerpc/platforms/ps3/platform.h b/arch/powerpc/platforms/ps3/platform.h index 1809cfc562ee..9bc68f913466 100644 --- a/arch/powerpc/platforms/ps3/platform.h +++ b/arch/powerpc/platforms/ps3/platform.h @@ -57,7 +57,7 @@ static inline void ps3_smp_cleanup_cpu(int cpu) { } /* time */ void __init ps3_calibrate_decr(void); -unsigned long __init ps3_get_boot_time(void); +time64_t __init ps3_get_boot_time(void); void ps3_get_rtc_time(struct rtc_time *time); int ps3_set_rtc_time(struct rtc_time *time); diff --git a/arch/powerpc/platforms/ps3/repository.c b/arch/powerpc/platforms/ps3/repository.c index 50dbaf24b1ee..e49c887787c4 100644 --- a/arch/powerpc/platforms/ps3/repository.c +++ b/arch/powerpc/platforms/ps3/repository.c @@ -101,9 +101,9 @@ static u64 make_first_field(const char *text, u64 index) static u64 make_field(const char *text, u64 index) { - u64 n; + u64 n = 0; - strncpy((char *)&n, text, 8); + memcpy((char *)&n, text, strnlen(text, sizeof(n))); return n + index; } diff --git a/arch/powerpc/platforms/ps3/time.c b/arch/powerpc/platforms/ps3/time.c index 11b45b58c81b..08ca76e23d09 100644 --- a/arch/powerpc/platforms/ps3/time.c +++ b/arch/powerpc/platforms/ps3/time.c @@ -28,30 +28,6 @@ #include "platform.h" -#define dump_tm(_a) _dump_tm(_a, __func__, __LINE__) -static void _dump_tm(const struct rtc_time *tm, const char* func, int line) -{ - pr_debug("%s:%d tm_sec %d\n", func, line, tm->tm_sec); - pr_debug("%s:%d tm_min %d\n", func, line, tm->tm_min); - pr_debug("%s:%d tm_hour %d\n", func, line, tm->tm_hour); - pr_debug("%s:%d tm_mday %d\n", func, line, tm->tm_mday); - pr_debug("%s:%d tm_mon %d\n", func, line, tm->tm_mon); - pr_debug("%s:%d tm_year %d\n", func, line, tm->tm_year); - pr_debug("%s:%d tm_wday %d\n", func, line, tm->tm_wday); -} - -#define dump_time(_a) _dump_time(_a, __func__, __LINE__) -static void __maybe_unused _dump_time(int time, const char *func, - int line) -{ - struct rtc_time tm; - - to_tm(time, &tm); - - pr_debug("%s:%d time %d\n", func, line, time); - _dump_tm(&tm, func, line); -} - void __init ps3_calibrate_decr(void) { int result; @@ -76,7 +52,7 @@ static u64 read_rtc(void) return rtc_val; } -unsigned long __init ps3_get_boot_time(void) +time64_t __init ps3_get_boot_time(void) { return read_rtc() + ps3_os_area_get_rtc_diff(); } diff --git a/arch/powerpc/platforms/pseries/hvCall_inst.c b/arch/powerpc/platforms/pseries/hvCall_inst.c index 89b7ce807e70..6da320c786cd 100644 --- a/arch/powerpc/platforms/pseries/hvCall_inst.c +++ b/arch/powerpc/platforms/pseries/hvCall_inst.c @@ -125,7 +125,7 @@ static void probe_hcall_entry(void *ignored, unsigned long opcode, unsigned long h->purr_start = mfspr(SPRN_PURR); } -static void probe_hcall_exit(void *ignored, unsigned long opcode, unsigned long retval, +static void probe_hcall_exit(void *ignored, unsigned long opcode, long retval, unsigned long *retbuf) { struct hcall_stats *h; diff --git a/arch/powerpc/platforms/pseries/kexec.c b/arch/powerpc/platforms/pseries/kexec.c index 3fe126796975..46fbaef69a59 100644 --- a/arch/powerpc/platforms/pseries/kexec.c +++ b/arch/powerpc/platforms/pseries/kexec.c @@ -57,8 +57,11 @@ void pseries_kexec_cpu_down(int crash_shutdown, int secondary) } } - if (xive_enabled()) + if (xive_enabled()) { xive_kexec_teardown_cpu(secondary); - else + + if (!secondary) + xive_shutdown(); + } else xics_kexec_teardown_cpu(secondary); } diff --git a/arch/powerpc/platforms/pseries/lpar.c b/arch/powerpc/platforms/pseries/lpar.c index adb996ed51e1..5a392e40f3d2 100644 --- a/arch/powerpc/platforms/pseries/lpar.c +++ b/arch/powerpc/platforms/pseries/lpar.c @@ -902,8 +902,7 @@ out: local_irq_restore(flags); } -void __trace_hcall_exit(long opcode, unsigned long retval, - unsigned long *retbuf) +void __trace_hcall_exit(long opcode, long retval, unsigned long *retbuf) { unsigned long flags; unsigned int *depth; diff --git a/arch/powerpc/platforms/pseries/lparcfg.c b/arch/powerpc/platforms/pseries/lparcfg.c index c508c938dc71..7c872dc01bdb 100644 --- a/arch/powerpc/platforms/pseries/lparcfg.c +++ b/arch/powerpc/platforms/pseries/lparcfg.c @@ -52,18 +52,20 @@ * Track sum of all purrs across all processors. This is used to further * calculate usage values by different applications */ +static void cpu_get_purr(void *arg) +{ + atomic64_t *sum = arg; + + atomic64_add(mfspr(SPRN_PURR), sum); +} + static unsigned long get_purr(void) { - unsigned long sum_purr = 0; - int cpu; + atomic64_t purr = ATOMIC64_INIT(0); - for_each_possible_cpu(cpu) { - struct cpu_usage *cu; + on_each_cpu(cpu_get_purr, &purr, 1); - cu = &per_cpu(cpu_usage_array, cpu); - sum_purr += cu->current_tb; - } - return sum_purr; + return atomic64_read(&purr); } /* diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c index fdb32e056ef4..139f0af6c3d9 100644 --- a/arch/powerpc/platforms/pseries/setup.c +++ b/arch/powerpc/platforms/pseries/setup.c @@ -534,6 +534,7 @@ void pseries_setup_rfi_flush(void) security_ftr_enabled(SEC_FTR_L1D_FLUSH_PR); setup_rfi_flush(types, enable); + setup_barrier_nospec(); } #ifdef CONFIG_PCI_IOV diff --git a/arch/powerpc/sysdev/Makefile b/arch/powerpc/sysdev/Makefile index 9861407d644a..ea2f595b5133 100644 --- a/arch/powerpc/sysdev/Makefile +++ b/arch/powerpc/sysdev/Makefile @@ -28,9 +28,6 @@ obj-$(CONFIG_FSL_85XX_CACHE_SRAM) += fsl_85xx_l2ctlr.o fsl_85xx_cache_sram.o obj-$(CONFIG_SIMPLE_GPIO) += simple_gpio.o obj-$(CONFIG_FSL_RIO) += fsl_rio.o fsl_rmu.o obj-$(CONFIG_TSI108_BRIDGE) += tsi108_pci.o tsi108_dev.o -mv64x60-$(CONFIG_PCI) += mv64x60_pci.o -obj-$(CONFIG_MV64X60) += $(mv64x60-y) mv64x60_pic.o mv64x60_dev.o \ - mv64x60_udbg.o obj-$(CONFIG_RTC_DRV_CMOS) += rtc_cmos_setup.o obj-$(CONFIG_PPC_INDIRECT_PCI) += indirect_pci.o diff --git a/arch/powerpc/sysdev/cpm_gpio.c b/arch/powerpc/sysdev/cpm_gpio.c index 0badc90be666..0695d26bd301 100644 --- a/arch/powerpc/sysdev/cpm_gpio.c +++ b/arch/powerpc/sysdev/cpm_gpio.c @@ -63,7 +63,6 @@ static struct platform_driver cpm_gpio_driver = { .probe = cpm_gpio_probe, .driver = { .name = "cpm-gpio", - .owner = THIS_MODULE, .of_match_table = cpm_gpio_match, }, }; diff --git a/arch/powerpc/sysdev/mpic.c b/arch/powerpc/sysdev/mpic.c index 1d4e0ef658d3..353b43972bbf 100644 --- a/arch/powerpc/sysdev/mpic.c +++ b/arch/powerpc/sysdev/mpic.c @@ -544,7 +544,7 @@ static void __init mpic_scan_ht_pics(struct mpic *mpic) printk(KERN_INFO "mpic: Setting up HT PICs workarounds for U3/U4\n"); /* Allocate fixups array */ - mpic->fixups = kzalloc(128 * sizeof(*mpic->fixups), GFP_KERNEL); + mpic->fixups = kcalloc(128, sizeof(*mpic->fixups), GFP_KERNEL); BUG_ON(mpic->fixups == NULL); /* Init spinlock */ @@ -1324,7 +1324,7 @@ struct mpic * __init mpic_alloc(struct device_node *node, if (psrc) { /* Allocate a bitmap with one bit per interrupt */ unsigned int mapsize = BITS_TO_LONGS(intvec_top + 1); - mpic->protected = kzalloc(mapsize*sizeof(long), GFP_KERNEL); + mpic->protected = kcalloc(mapsize, sizeof(long), GFP_KERNEL); BUG_ON(mpic->protected == NULL); for (i = 0; i < psize/sizeof(u32); i++) { if (psrc[i] > intvec_top) @@ -1639,8 +1639,9 @@ void __init mpic_init(struct mpic *mpic) #ifdef CONFIG_PM /* allocate memory to save mpic state */ - mpic->save_data = kmalloc(mpic->num_sources * sizeof(*mpic->save_data), - GFP_KERNEL); + mpic->save_data = kmalloc_array(mpic->num_sources, + sizeof(*mpic->save_data), + GFP_KERNEL); BUG_ON(mpic->save_data == NULL); #endif diff --git a/arch/powerpc/sysdev/mv64x60.h b/arch/powerpc/sysdev/mv64x60.h deleted file mode 100644 index 60cfcb90d1fa..000000000000 --- a/arch/powerpc/sysdev/mv64x60.h +++ /dev/null @@ -1,13 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef __MV64X60_H__ -#define __MV64X60_H__ - -#include - -extern void __init mv64x60_init_irq(void); -extern unsigned int mv64x60_get_irq(void); - -extern void __init mv64x60_pci_init(void); -extern void __init mv64x60_init_early(void); - -#endif /* __MV64X60_H__ */ diff --git a/arch/powerpc/sysdev/mv64x60_dev.c b/arch/powerpc/sysdev/mv64x60_dev.c deleted file mode 100644 index 185a67e742a6..000000000000 --- a/arch/powerpc/sysdev/mv64x60_dev.c +++ /dev/null @@ -1,535 +0,0 @@ -/* - * Platform device setup for Marvell mv64360/mv64460 host bridges (Discovery) - * - * Author: Dale Farnsworth - * - * 2007 (c) MontaVista, Software, Inc. This file is licensed under - * the terms of the GNU General Public License version 2. This program - * is licensed "as is" without any warranty of any kind, whether express - * or implied. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include - -/* These functions provide the necessary setup for the mv64x60 drivers. */ - -static const struct of_device_id of_mv64x60_devices[] __initconst = { - { .compatible = "marvell,mv64306-devctrl", }, - {} -}; - -/* - * Create MPSC platform devices - */ -static int __init mv64x60_mpsc_register_shared_pdev(struct device_node *np) -{ - struct platform_device *pdev; - struct resource r[2]; - struct mpsc_shared_pdata pdata; - const phandle *ph; - struct device_node *mpscrouting, *mpscintr; - int err; - - ph = of_get_property(np, "mpscrouting", NULL); - mpscrouting = of_find_node_by_phandle(*ph); - if (!mpscrouting) - return -ENODEV; - - err = of_address_to_resource(mpscrouting, 0, &r[0]); - of_node_put(mpscrouting); - if (err) - return err; - - ph = of_get_property(np, "mpscintr", NULL); - mpscintr = of_find_node_by_phandle(*ph); - if (!mpscintr) - return -ENODEV; - - err = of_address_to_resource(mpscintr, 0, &r[1]); - of_node_put(mpscintr); - if (err) - return err; - - memset(&pdata, 0, sizeof(pdata)); - - pdev = platform_device_alloc(MPSC_SHARED_NAME, 0); - if (!pdev) - return -ENOMEM; - - err = platform_device_add_resources(pdev, r, 2); - if (err) - goto error; - - err = platform_device_add_data(pdev, &pdata, sizeof(pdata)); - if (err) - goto error; - - err = platform_device_add(pdev); - if (err) - goto error; - - return 0; - -error: - platform_device_put(pdev); - return err; -} - - -static int __init mv64x60_mpsc_device_setup(struct device_node *np, int id) -{ - struct resource r[5]; - struct mpsc_pdata pdata; - struct platform_device *pdev; - const unsigned int *prop; - const phandle *ph; - struct device_node *sdma, *brg; - int err; - int port_number; - - /* only register the shared platform device the first time through */ - if (id == 0 && (err = mv64x60_mpsc_register_shared_pdev(np))) - return err; - - memset(r, 0, sizeof(r)); - - err = of_address_to_resource(np, 0, &r[0]); - if (err) - return err; - - of_irq_to_resource(np, 0, &r[4]); - - ph = of_get_property(np, "sdma", NULL); - sdma = of_find_node_by_phandle(*ph); - if (!sdma) - return -ENODEV; - - of_irq_to_resource(sdma, 0, &r[3]); - err = of_address_to_resource(sdma, 0, &r[1]); - of_node_put(sdma); - if (err) - return err; - - ph = of_get_property(np, "brg", NULL); - brg = of_find_node_by_phandle(*ph); - if (!brg) - return -ENODEV; - - err = of_address_to_resource(brg, 0, &r[2]); - of_node_put(brg); - if (err) - return err; - - prop = of_get_property(np, "cell-index", NULL); - if (!prop) - return -ENODEV; - port_number = *(int *)prop; - - memset(&pdata, 0, sizeof(pdata)); - - pdata.cache_mgmt = 1; /* All current revs need this set */ - - pdata.max_idle = 40; /* default */ - prop = of_get_property(np, "max_idle", NULL); - if (prop) - pdata.max_idle = *prop; - - prop = of_get_property(brg, "current-speed", NULL); - if (prop) - pdata.default_baud = *prop; - - /* Default is 8 bits, no parity, no flow control */ - pdata.default_bits = 8; - pdata.default_parity = 'n'; - pdata.default_flow = 'n'; - - prop = of_get_property(np, "chr_1", NULL); - if (prop) - pdata.chr_1_val = *prop; - - prop = of_get_property(np, "chr_2", NULL); - if (prop) - pdata.chr_2_val = *prop; - - prop = of_get_property(np, "chr_10", NULL); - if (prop) - pdata.chr_10_val = *prop; - - prop = of_get_property(np, "mpcr", NULL); - if (prop) - pdata.mpcr_val = *prop; - - prop = of_get_property(brg, "bcr", NULL); - if (prop) - pdata.bcr_val = *prop; - - pdata.brg_can_tune = 1; /* All current revs need this set */ - - prop = of_get_property(brg, "clock-src", NULL); - if (prop) - pdata.brg_clk_src = *prop; - - prop = of_get_property(brg, "clock-frequency", NULL); - if (prop) - pdata.brg_clk_freq = *prop; - - pdev = platform_device_alloc(MPSC_CTLR_NAME, port_number); - if (!pdev) - return -ENOMEM; - pdev->dev.coherent_dma_mask = DMA_BIT_MASK(32); - - err = platform_device_add_resources(pdev, r, 5); - if (err) - goto error; - - err = platform_device_add_data(pdev, &pdata, sizeof(pdata)); - if (err) - goto error; - - err = platform_device_add(pdev); - if (err) - goto error; - - return 0; - -error: - platform_device_put(pdev); - return err; -} - -/* - * Create mv64x60_eth platform devices - */ -static struct platform_device * __init mv64x60_eth_register_shared_pdev( - struct device_node *np, int id) -{ - struct platform_device *pdev; - struct resource r[2]; - int err; - - err = of_address_to_resource(np, 0, &r[0]); - if (err) - return ERR_PTR(err); - - /* register an orion mdio bus driver */ - r[1].start = r[0].start + 0x4; - r[1].end = r[0].start + 0x84 - 1; - r[1].flags = IORESOURCE_MEM; - - if (id == 0) { - pdev = platform_device_register_simple("orion-mdio", -1, &r[1], 1); - if (IS_ERR(pdev)) - return pdev; - } - - pdev = platform_device_register_simple(MV643XX_ETH_SHARED_NAME, id, - &r[0], 1); - - return pdev; -} - -static int __init mv64x60_eth_device_setup(struct device_node *np, int id, - struct platform_device *shared_pdev) -{ - struct resource r[1]; - struct mv643xx_eth_platform_data pdata; - struct platform_device *pdev; - struct device_node *phy; - const u8 *mac_addr; - const int *prop; - const phandle *ph; - int err; - - memset(r, 0, sizeof(r)); - of_irq_to_resource(np, 0, &r[0]); - - memset(&pdata, 0, sizeof(pdata)); - - pdata.shared = shared_pdev; - - prop = of_get_property(np, "reg", NULL); - if (!prop) - return -ENODEV; - pdata.port_number = *prop; - - mac_addr = of_get_mac_address(np); - if (mac_addr) - memcpy(pdata.mac_addr, mac_addr, 6); - - prop = of_get_property(np, "speed", NULL); - if (prop) - pdata.speed = *prop; - - prop = of_get_property(np, "tx_queue_size", NULL); - if (prop) - pdata.tx_queue_size = *prop; - - prop = of_get_property(np, "rx_queue_size", NULL); - if (prop) - pdata.rx_queue_size = *prop; - - prop = of_get_property(np, "tx_sram_addr", NULL); - if (prop) - pdata.tx_sram_addr = *prop; - - prop = of_get_property(np, "tx_sram_size", NULL); - if (prop) - pdata.tx_sram_size = *prop; - - prop = of_get_property(np, "rx_sram_addr", NULL); - if (prop) - pdata.rx_sram_addr = *prop; - - prop = of_get_property(np, "rx_sram_size", NULL); - if (prop) - pdata.rx_sram_size = *prop; - - ph = of_get_property(np, "phy", NULL); - if (!ph) - return -ENODEV; - - phy = of_find_node_by_phandle(*ph); - if (phy == NULL) - return -ENODEV; - - prop = of_get_property(phy, "reg", NULL); - if (prop) - pdata.phy_addr = MV643XX_ETH_PHY_ADDR(*prop); - - of_node_put(phy); - - pdev = platform_device_alloc(MV643XX_ETH_NAME, id); - if (!pdev) - return -ENOMEM; - - pdev->dev.coherent_dma_mask = DMA_BIT_MASK(32); - err = platform_device_add_resources(pdev, r, 1); - if (err) - goto error; - - err = platform_device_add_data(pdev, &pdata, sizeof(pdata)); - if (err) - goto error; - - err = platform_device_add(pdev); - if (err) - goto error; - - return 0; - -error: - platform_device_put(pdev); - return err; -} - -/* - * Create mv64x60_i2c platform devices - */ -static int __init mv64x60_i2c_device_setup(struct device_node *np, int id) -{ - struct resource r[2]; - struct platform_device *pdev; - struct mv64xxx_i2c_pdata pdata; - const unsigned int *prop; - int err; - - memset(r, 0, sizeof(r)); - - err = of_address_to_resource(np, 0, &r[0]); - if (err) - return err; - - of_irq_to_resource(np, 0, &r[1]); - - memset(&pdata, 0, sizeof(pdata)); - - pdata.freq_m = 8; /* default */ - prop = of_get_property(np, "freq_m", NULL); - if (prop) - pdata.freq_m = *prop; - - pdata.freq_n = 3; /* default */ - prop = of_get_property(np, "freq_n", NULL); - if (prop) - pdata.freq_n = *prop; - - pdata.timeout = 1000; /* default: 1 second */ - - pdev = platform_device_alloc(MV64XXX_I2C_CTLR_NAME, id); - if (!pdev) - return -ENOMEM; - - err = platform_device_add_resources(pdev, r, 2); - if (err) - goto error; - - err = platform_device_add_data(pdev, &pdata, sizeof(pdata)); - if (err) - goto error; - - err = platform_device_add(pdev); - if (err) - goto error; - - return 0; - -error: - platform_device_put(pdev); - return err; -} - -/* - * Create mv64x60_wdt platform devices - */ -static int __init mv64x60_wdt_device_setup(struct device_node *np, int id) -{ - struct resource r; - struct platform_device *pdev; - struct mv64x60_wdt_pdata pdata; - const unsigned int *prop; - int err; - - err = of_address_to_resource(np, 0, &r); - if (err) - return err; - - memset(&pdata, 0, sizeof(pdata)); - - pdata.timeout = 10; /* Default: 10 seconds */ - - np = of_get_parent(np); - if (!np) - return -ENODEV; - - prop = of_get_property(np, "clock-frequency", NULL); - of_node_put(np); - if (!prop) - return -ENODEV; - pdata.bus_clk = *prop / 1000000; /* wdt driver wants freq in MHz */ - - pdev = platform_device_alloc(MV64x60_WDT_NAME, id); - if (!pdev) - return -ENOMEM; - - err = platform_device_add_resources(pdev, &r, 1); - if (err) - goto error; - - err = platform_device_add_data(pdev, &pdata, sizeof(pdata)); - if (err) - goto error; - - err = platform_device_add(pdev); - if (err) - goto error; - - return 0; - -error: - platform_device_put(pdev); - return err; -} - -static int __init mv64x60_device_setup(void) -{ - struct device_node *np, *np2; - struct platform_device *pdev; - int id, id2; - int err; - - id = 0; - for_each_compatible_node(np, NULL, "marvell,mv64360-mpsc") { - err = mv64x60_mpsc_device_setup(np, id++); - if (err) - printk(KERN_ERR "Failed to initialize MV64x60 " - "serial device %pOF: error %d.\n", - np, err); - } - - id = 0; - id2 = 0; - for_each_compatible_node(np, NULL, "marvell,mv64360-eth-group") { - pdev = mv64x60_eth_register_shared_pdev(np, id++); - if (IS_ERR(pdev)) { - err = PTR_ERR(pdev); - printk(KERN_ERR "Failed to initialize MV64x60 " - "network block %pOF: error %d.\n", - np, err); - continue; - } - for_each_child_of_node(np, np2) { - if (!of_device_is_compatible(np2, - "marvell,mv64360-eth")) - continue; - err = mv64x60_eth_device_setup(np2, id2++, pdev); - if (err) - printk(KERN_ERR "Failed to initialize " - "MV64x60 network device %pOF: " - "error %d.\n", - np2, err); - } - } - - id = 0; - for_each_compatible_node(np, "i2c", "marvell,mv64360-i2c") { - err = mv64x60_i2c_device_setup(np, id++); - if (err) - printk(KERN_ERR "Failed to initialize MV64x60 I2C " - "bus %pOF: error %d.\n", - np, err); - } - - /* support up to one watchdog timer */ - np = of_find_compatible_node(np, NULL, "marvell,mv64360-wdt"); - if (np) { - if ((err = mv64x60_wdt_device_setup(np, id))) - printk(KERN_ERR "Failed to initialize MV64x60 " - "Watchdog %pOF: error %d.\n", - np, err); - of_node_put(np); - } - - /* Now add every node that is on the device bus */ - for_each_compatible_node(np, NULL, "marvell,mv64360") - of_platform_bus_probe(np, of_mv64x60_devices, NULL); - - return 0; -} -arch_initcall(mv64x60_device_setup); - -static int __init mv64x60_add_mpsc_console(void) -{ - struct device_node *np = NULL; - const char *prop; - - prop = of_get_property(of_chosen, "linux,stdout-path", NULL); - if (prop == NULL) - goto not_mpsc; - - np = of_find_node_by_path(prop); - if (!np) - goto not_mpsc; - - if (!of_device_is_compatible(np, "marvell,mv64360-mpsc")) - goto not_mpsc; - - prop = of_get_property(np, "cell-index", NULL); - if (!prop) - goto not_mpsc; - - add_preferred_console("ttyMM", *(int *)prop, NULL); - -not_mpsc: - return 0; -} -console_initcall(mv64x60_add_mpsc_console); diff --git a/arch/powerpc/sysdev/mv64x60_pci.c b/arch/powerpc/sysdev/mv64x60_pci.c deleted file mode 100644 index 1afcdb428e51..000000000000 --- a/arch/powerpc/sysdev/mv64x60_pci.c +++ /dev/null @@ -1,171 +0,0 @@ -/* - * PCI bus setup for Marvell mv64360/mv64460 host bridges (Discovery) - * - * Author: Dale Farnsworth - * - * 2007 (c) MontaVista, Software, Inc. This file is licensed under - * the terms of the GNU General Public License version 2. This program - * is licensed "as is" without any warranty of any kind, whether express - * or implied. - */ - -#include -#include -#include -#include -#include - -#include -#include - -#define PCI_HEADER_TYPE_INVALID 0x7f /* Invalid PCI header type */ - -#ifdef CONFIG_SYSFS -/* 32-bit hex or dec stringified number + '\n' */ -#define MV64X60_VAL_LEN_MAX 11 -#define MV64X60_PCICFG_CPCI_HOTSWAP 0x68 - -static ssize_t mv64x60_hs_reg_read(struct file *filp, struct kobject *kobj, - struct bin_attribute *attr, char *buf, - loff_t off, size_t count) -{ - struct pci_dev *phb; - u32 v; - - if (off > 0) - return 0; - if (count < MV64X60_VAL_LEN_MAX) - return -EINVAL; - - phb = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0)); - if (!phb) - return -ENODEV; - pci_read_config_dword(phb, MV64X60_PCICFG_CPCI_HOTSWAP, &v); - pci_dev_put(phb); - - return sprintf(buf, "0x%08x\n", v); -} - -static ssize_t mv64x60_hs_reg_write(struct file *filp, struct kobject *kobj, - struct bin_attribute *attr, char *buf, - loff_t off, size_t count) -{ - struct pci_dev *phb; - u32 v; - - if (off > 0) - return 0; - if (count <= 0) - return -EINVAL; - - if (sscanf(buf, "%i", &v) != 1) - return -EINVAL; - - phb = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(0, 0)); - if (!phb) - return -ENODEV; - pci_write_config_dword(phb, MV64X60_PCICFG_CPCI_HOTSWAP, v); - pci_dev_put(phb); - - return count; -} - -static const struct bin_attribute mv64x60_hs_reg_attr = { /* Hotswap register */ - .attr = { - .name = "hs_reg", - .mode = 0644, - }, - .size = MV64X60_VAL_LEN_MAX, - .read = mv64x60_hs_reg_read, - .write = mv64x60_hs_reg_write, -}; - -static int __init mv64x60_sysfs_init(void) -{ - struct device_node *np; - struct platform_device *pdev; - const unsigned int *prop; - - np = of_find_compatible_node(NULL, NULL, "marvell,mv64360"); - if (!np) - return 0; - - prop = of_get_property(np, "hs_reg_valid", NULL); - of_node_put(np); - - pdev = platform_device_register_simple("marvell,mv64360", 0, NULL, 0); - if (IS_ERR(pdev)) - return PTR_ERR(pdev); - - return sysfs_create_bin_file(&pdev->dev.kobj, &mv64x60_hs_reg_attr); -} - -subsys_initcall(mv64x60_sysfs_init); - -#endif /* CONFIG_SYSFS */ - -static void mv64x60_pci_fixup_early(struct pci_dev *dev) -{ - /* - * Set the host bridge hdr_type to an invalid value so that - * pci_setup_device() will ignore the host bridge. - */ - dev->hdr_type = PCI_HEADER_TYPE_INVALID; -} -DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_MARVELL, PCI_DEVICE_ID_MARVELL_MV64360, - mv64x60_pci_fixup_early); -DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_MARVELL, PCI_DEVICE_ID_MARVELL_MV64460, - mv64x60_pci_fixup_early); - -static int __init mv64x60_add_bridge(struct device_node *dev) -{ - int len; - struct pci_controller *hose; - struct resource rsrc; - const int *bus_range; - int primary; - - memset(&rsrc, 0, sizeof(rsrc)); - - /* Fetch host bridge registers address */ - if (of_address_to_resource(dev, 0, &rsrc)) { - printk(KERN_ERR "No PCI reg property in device tree\n"); - return -ENODEV; - } - - /* Get bus range if any */ - bus_range = of_get_property(dev, "bus-range", &len); - if (bus_range == NULL || len < 2 * sizeof(int)) - printk(KERN_WARNING "Can't get bus-range for %pOF, assume" - " bus 0\n", dev); - - hose = pcibios_alloc_controller(dev); - if (!hose) - return -ENOMEM; - - hose->first_busno = bus_range ? bus_range[0] : 0; - hose->last_busno = bus_range ? bus_range[1] : 0xff; - - setup_indirect_pci(hose, rsrc.start, rsrc.start + 4, 0); - hose->self_busno = hose->first_busno; - - printk(KERN_INFO "Found MV64x60 PCI host bridge at 0x%016llx. " - "Firmware bus number: %d->%d\n", - (unsigned long long)rsrc.start, hose->first_busno, - hose->last_busno); - - /* Interpret the "ranges" property */ - /* This also maps the I/O region and sets isa_io/mem_base */ - primary = (hose->first_busno == 0); - pci_process_bridge_OF_ranges(hose, dev, primary); - - return 0; -} - -void __init mv64x60_pci_init(void) -{ - struct device_node *np; - - for_each_compatible_node(np, "pci", "marvell,mv64360-pci") - mv64x60_add_bridge(np); -} diff --git a/arch/powerpc/sysdev/mv64x60_pic.c b/arch/powerpc/sysdev/mv64x60_pic.c deleted file mode 100644 index a79953deb489..000000000000 --- a/arch/powerpc/sysdev/mv64x60_pic.c +++ /dev/null @@ -1,297 +0,0 @@ -/* - * Interrupt handling for Marvell mv64360/mv64460 host bridges (Discovery) - * - * Author: Dale Farnsworth - * - * 2007 (c) MontaVista, Software, Inc. This file is licensed under - * the terms of the GNU General Public License version 2. This program - * is licensed "as is" without any warranty of any kind, whether express - * or implied. - */ - -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include - -#include "mv64x60.h" - -/* Interrupt Controller Interface Registers */ -#define MV64X60_IC_MAIN_CAUSE_LO 0x0004 -#define MV64X60_IC_MAIN_CAUSE_HI 0x000c -#define MV64X60_IC_CPU0_INTR_MASK_LO 0x0014 -#define MV64X60_IC_CPU0_INTR_MASK_HI 0x001c -#define MV64X60_IC_CPU0_SELECT_CAUSE 0x0024 - -#define MV64X60_HIGH_GPP_GROUPS 0x0f000000 -#define MV64X60_SELECT_CAUSE_HIGH 0x40000000 - -/* General Purpose Pins Controller Interface Registers */ -#define MV64x60_GPP_INTR_CAUSE 0x0008 -#define MV64x60_GPP_INTR_MASK 0x000c - -#define MV64x60_LEVEL1_LOW 0 -#define MV64x60_LEVEL1_HIGH 1 -#define MV64x60_LEVEL1_GPP 2 - -#define MV64x60_LEVEL1_MASK 0x00000060 -#define MV64x60_LEVEL1_OFFSET 5 - -#define MV64x60_LEVEL2_MASK 0x0000001f - -#define MV64x60_NUM_IRQS 96 - -static DEFINE_SPINLOCK(mv64x60_lock); - -static void __iomem *mv64x60_irq_reg_base; -static void __iomem *mv64x60_gpp_reg_base; - -/* - * Interrupt Controller Handling - * - * The interrupt controller handles three groups of interrupts: - * main low: IRQ0-IRQ31 - * main high: IRQ32-IRQ63 - * gpp: IRQ64-IRQ95 - * - * This code handles interrupts in two levels. Level 1 selects the - * interrupt group, and level 2 selects an IRQ within that group. - * Each group has its own irq_chip structure. - */ - -static u32 mv64x60_cached_low_mask; -static u32 mv64x60_cached_high_mask = MV64X60_HIGH_GPP_GROUPS; -static u32 mv64x60_cached_gpp_mask; - -static struct irq_domain *mv64x60_irq_host; - -/* - * mv64x60_chip_low functions - */ - -static void mv64x60_mask_low(struct irq_data *d) -{ - int level2 = irqd_to_hwirq(d) & MV64x60_LEVEL2_MASK; - unsigned long flags; - - spin_lock_irqsave(&mv64x60_lock, flags); - mv64x60_cached_low_mask &= ~(1 << level2); - out_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_INTR_MASK_LO, - mv64x60_cached_low_mask); - spin_unlock_irqrestore(&mv64x60_lock, flags); - (void)in_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_INTR_MASK_LO); -} - -static void mv64x60_unmask_low(struct irq_data *d) -{ - int level2 = irqd_to_hwirq(d) & MV64x60_LEVEL2_MASK; - unsigned long flags; - - spin_lock_irqsave(&mv64x60_lock, flags); - mv64x60_cached_low_mask |= 1 << level2; - out_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_INTR_MASK_LO, - mv64x60_cached_low_mask); - spin_unlock_irqrestore(&mv64x60_lock, flags); - (void)in_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_INTR_MASK_LO); -} - -static struct irq_chip mv64x60_chip_low = { - .name = "mv64x60_low", - .irq_mask = mv64x60_mask_low, - .irq_mask_ack = mv64x60_mask_low, - .irq_unmask = mv64x60_unmask_low, -}; - -/* - * mv64x60_chip_high functions - */ - -static void mv64x60_mask_high(struct irq_data *d) -{ - int level2 = irqd_to_hwirq(d) & MV64x60_LEVEL2_MASK; - unsigned long flags; - - spin_lock_irqsave(&mv64x60_lock, flags); - mv64x60_cached_high_mask &= ~(1 << level2); - out_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_INTR_MASK_HI, - mv64x60_cached_high_mask); - spin_unlock_irqrestore(&mv64x60_lock, flags); - (void)in_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_INTR_MASK_HI); -} - -static void mv64x60_unmask_high(struct irq_data *d) -{ - int level2 = irqd_to_hwirq(d) & MV64x60_LEVEL2_MASK; - unsigned long flags; - - spin_lock_irqsave(&mv64x60_lock, flags); - mv64x60_cached_high_mask |= 1 << level2; - out_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_INTR_MASK_HI, - mv64x60_cached_high_mask); - spin_unlock_irqrestore(&mv64x60_lock, flags); - (void)in_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_INTR_MASK_HI); -} - -static struct irq_chip mv64x60_chip_high = { - .name = "mv64x60_high", - .irq_mask = mv64x60_mask_high, - .irq_mask_ack = mv64x60_mask_high, - .irq_unmask = mv64x60_unmask_high, -}; - -/* - * mv64x60_chip_gpp functions - */ - -static void mv64x60_mask_gpp(struct irq_data *d) -{ - int level2 = irqd_to_hwirq(d) & MV64x60_LEVEL2_MASK; - unsigned long flags; - - spin_lock_irqsave(&mv64x60_lock, flags); - mv64x60_cached_gpp_mask &= ~(1 << level2); - out_le32(mv64x60_gpp_reg_base + MV64x60_GPP_INTR_MASK, - mv64x60_cached_gpp_mask); - spin_unlock_irqrestore(&mv64x60_lock, flags); - (void)in_le32(mv64x60_gpp_reg_base + MV64x60_GPP_INTR_MASK); -} - -static void mv64x60_mask_ack_gpp(struct irq_data *d) -{ - int level2 = irqd_to_hwirq(d) & MV64x60_LEVEL2_MASK; - unsigned long flags; - - spin_lock_irqsave(&mv64x60_lock, flags); - mv64x60_cached_gpp_mask &= ~(1 << level2); - out_le32(mv64x60_gpp_reg_base + MV64x60_GPP_INTR_MASK, - mv64x60_cached_gpp_mask); - out_le32(mv64x60_gpp_reg_base + MV64x60_GPP_INTR_CAUSE, - ~(1 << level2)); - spin_unlock_irqrestore(&mv64x60_lock, flags); - (void)in_le32(mv64x60_gpp_reg_base + MV64x60_GPP_INTR_CAUSE); -} - -static void mv64x60_unmask_gpp(struct irq_data *d) -{ - int level2 = irqd_to_hwirq(d) & MV64x60_LEVEL2_MASK; - unsigned long flags; - - spin_lock_irqsave(&mv64x60_lock, flags); - mv64x60_cached_gpp_mask |= 1 << level2; - out_le32(mv64x60_gpp_reg_base + MV64x60_GPP_INTR_MASK, - mv64x60_cached_gpp_mask); - spin_unlock_irqrestore(&mv64x60_lock, flags); - (void)in_le32(mv64x60_gpp_reg_base + MV64x60_GPP_INTR_MASK); -} - -static struct irq_chip mv64x60_chip_gpp = { - .name = "mv64x60_gpp", - .irq_mask = mv64x60_mask_gpp, - .irq_mask_ack = mv64x60_mask_ack_gpp, - .irq_unmask = mv64x60_unmask_gpp, -}; - -/* - * mv64x60_host_ops functions - */ - -static struct irq_chip *mv64x60_chips[] = { - [MV64x60_LEVEL1_LOW] = &mv64x60_chip_low, - [MV64x60_LEVEL1_HIGH] = &mv64x60_chip_high, - [MV64x60_LEVEL1_GPP] = &mv64x60_chip_gpp, -}; - -static int mv64x60_host_map(struct irq_domain *h, unsigned int virq, - irq_hw_number_t hwirq) -{ - int level1; - - irq_set_status_flags(virq, IRQ_LEVEL); - - level1 = (hwirq & MV64x60_LEVEL1_MASK) >> MV64x60_LEVEL1_OFFSET; - BUG_ON(level1 > MV64x60_LEVEL1_GPP); - irq_set_chip_and_handler(virq, mv64x60_chips[level1], - handle_level_irq); - - return 0; -} - -static const struct irq_domain_ops mv64x60_host_ops = { - .map = mv64x60_host_map, -}; - -/* - * Global functions - */ - -void __init mv64x60_init_irq(void) -{ - struct device_node *np; - phys_addr_t paddr; - unsigned int size; - const unsigned int *reg; - unsigned long flags; - - np = of_find_compatible_node(NULL, NULL, "marvell,mv64360-gpp"); - reg = of_get_property(np, "reg", &size); - paddr = of_translate_address(np, reg); - mv64x60_gpp_reg_base = ioremap(paddr, reg[1]); - of_node_put(np); - - np = of_find_compatible_node(NULL, NULL, "marvell,mv64360-pic"); - reg = of_get_property(np, "reg", &size); - paddr = of_translate_address(np, reg); - mv64x60_irq_reg_base = ioremap(paddr, reg[1]); - - mv64x60_irq_host = irq_domain_add_linear(np, MV64x60_NUM_IRQS, - &mv64x60_host_ops, NULL); - - spin_lock_irqsave(&mv64x60_lock, flags); - out_le32(mv64x60_gpp_reg_base + MV64x60_GPP_INTR_MASK, - mv64x60_cached_gpp_mask); - out_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_INTR_MASK_LO, - mv64x60_cached_low_mask); - out_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_INTR_MASK_HI, - mv64x60_cached_high_mask); - - out_le32(mv64x60_gpp_reg_base + MV64x60_GPP_INTR_CAUSE, 0); - out_le32(mv64x60_irq_reg_base + MV64X60_IC_MAIN_CAUSE_LO, 0); - out_le32(mv64x60_irq_reg_base + MV64X60_IC_MAIN_CAUSE_HI, 0); - spin_unlock_irqrestore(&mv64x60_lock, flags); -} - -unsigned int mv64x60_get_irq(void) -{ - u32 cause; - int level1; - irq_hw_number_t hwirq; - int virq = 0; - - cause = in_le32(mv64x60_irq_reg_base + MV64X60_IC_CPU0_SELECT_CAUSE); - if (cause & MV64X60_SELECT_CAUSE_HIGH) { - cause &= mv64x60_cached_high_mask; - level1 = MV64x60_LEVEL1_HIGH; - if (cause & MV64X60_HIGH_GPP_GROUPS) { - cause = in_le32(mv64x60_gpp_reg_base + - MV64x60_GPP_INTR_CAUSE); - cause &= mv64x60_cached_gpp_mask; - level1 = MV64x60_LEVEL1_GPP; - } - } else { - cause &= mv64x60_cached_low_mask; - level1 = MV64x60_LEVEL1_LOW; - } - if (cause) { - hwirq = (level1 << MV64x60_LEVEL1_OFFSET) | __ilog2(cause); - virq = irq_linear_revmap(mv64x60_irq_host, hwirq); - } - - return virq; -} diff --git a/arch/powerpc/sysdev/mv64x60_udbg.c b/arch/powerpc/sysdev/mv64x60_udbg.c deleted file mode 100644 index 3b8734b870e9..000000000000 --- a/arch/powerpc/sysdev/mv64x60_udbg.c +++ /dev/null @@ -1,152 +0,0 @@ -/* - * udbg serial input/output routines for the Marvell MV64x60 (Discovery). - * - * Author: Dale Farnsworth - * - * 2007 (c) MontaVista Software, Inc. This file is licensed under - * the terms of the GNU General Public License version 2. This program - * is licensed "as is" without any warranty of any kind, whether express - * or implied. - */ - -#include -#include -#include - -#include - -#define MPSC_0_CR1_OFFSET 0x000c - -#define MPSC_0_CR2_OFFSET 0x0010 -#define MPSC_CHR_2_TCS (1 << 9) - -#define MPSC_0_CHR_10_OFFSET 0x0030 - -#define MPSC_INTR_CAUSE_OFF_0 0x0004 -#define MPSC_INTR_CAUSE_OFF_1 0x000c -#define MPSC_INTR_CAUSE_RCC (1<<6) - -static void __iomem *mpsc_base; -static void __iomem *mpsc_intr_cause; - -static void mv64x60_udbg_putc(char c) -{ - if (c == '\n') - mv64x60_udbg_putc('\r'); - - while(in_le32(mpsc_base + MPSC_0_CR2_OFFSET) & MPSC_CHR_2_TCS) - ; - out_le32(mpsc_base + MPSC_0_CR1_OFFSET, c); - out_le32(mpsc_base + MPSC_0_CR2_OFFSET, MPSC_CHR_2_TCS); -} - -static int mv64x60_udbg_testc(void) -{ - return (in_le32(mpsc_intr_cause) & MPSC_INTR_CAUSE_RCC) != 0; -} - -static int mv64x60_udbg_getc(void) -{ - int cause = 0; - int c; - - while (!mv64x60_udbg_testc()) - ; - - c = in_8(mpsc_base + MPSC_0_CHR_10_OFFSET + 2); - out_8(mpsc_base + MPSC_0_CHR_10_OFFSET + 2, c); - out_le32(mpsc_intr_cause, cause & ~MPSC_INTR_CAUSE_RCC); - return c; -} - -static int mv64x60_udbg_getc_poll(void) -{ - if (!mv64x60_udbg_testc()) - return -1; - - return mv64x60_udbg_getc(); -} - -static void mv64x60_udbg_init(void) -{ - struct device_node *np, *mpscintr, *stdout = NULL; - const char *path; - const phandle *ph; - struct resource r[2]; - const int *block_index; - int intr_cause_offset; - int err; - - path = of_get_property(of_chosen, "linux,stdout-path", NULL); - if (!path) - return; - - stdout = of_find_node_by_path(path); - if (!stdout) - return; - - for_each_compatible_node(np, NULL, "marvell,mv64360-mpsc") { - if (np == stdout) - break; - } - - of_node_put(stdout); - if (!np) - return; - - block_index = of_get_property(np, "cell-index", NULL); - if (!block_index) - goto error; - - switch (*block_index) { - case 0: - intr_cause_offset = MPSC_INTR_CAUSE_OFF_0; - break; - case 1: - intr_cause_offset = MPSC_INTR_CAUSE_OFF_1; - break; - default: - goto error; - } - - err = of_address_to_resource(np, 0, &r[0]); - if (err) - goto error; - - ph = of_get_property(np, "mpscintr", NULL); - mpscintr = of_find_node_by_phandle(*ph); - if (!mpscintr) - goto error; - - err = of_address_to_resource(mpscintr, 0, &r[1]); - of_node_put(mpscintr); - if (err) - goto error; - - of_node_put(np); - - mpsc_base = ioremap(r[0].start, resource_size(&r[0])); - if (!mpsc_base) - return; - - mpsc_intr_cause = ioremap(r[1].start, resource_size(&r[1])); - if (!mpsc_intr_cause) { - iounmap(mpsc_base); - return; - } - mpsc_intr_cause += intr_cause_offset; - - udbg_putc = mv64x60_udbg_putc; - udbg_getc = mv64x60_udbg_getc; - udbg_getc_poll = mv64x60_udbg_getc_poll; - - return; - -error: - of_node_put(np); -} - -void mv64x60_init_early(void) -{ - mv64x60_udbg_init(); -} diff --git a/arch/powerpc/sysdev/xics/xics-common.c b/arch/powerpc/sysdev/xics/xics-common.c index 77e864d5506d..f87470319d71 100644 --- a/arch/powerpc/sysdev/xics/xics-common.c +++ b/arch/powerpc/sysdev/xics/xics-common.c @@ -446,10 +446,11 @@ static void __init xics_get_server_size(void) np = of_find_compatible_node(NULL, NULL, "ibm,ppc-xics"); if (!np) return; + isize = of_get_property(np, "ibm,interrupt-server#-size", NULL); - if (!isize) - return; - xics_interrupt_server_size = be32_to_cpu(*isize); + if (isize) + xics_interrupt_server_size = be32_to_cpu(*isize); + of_node_put(np); } diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c index b48454be5b98..311185b9960a 100644 --- a/arch/powerpc/sysdev/xive/native.c +++ b/arch/powerpc/sysdev/xive/native.c @@ -341,7 +341,7 @@ static void xive_native_update_pending(struct xive_cpu *xc) * of the hypervisor interrupt (if any) */ cppr = ack & 0xff; - he = GETFIELD(TM_QW3_NSR_HE, (ack >> 8)); + he = (ack >> 8) >> 6; switch(he) { case TM_QW3_NSR_HE_NONE: /* Nothing to see here */ break; @@ -489,7 +489,7 @@ static bool xive_parse_provisioning(struct device_node *np) if (rc == 0) return true; - xive_provision_chips = kzalloc(4 * xive_provision_chip_count, + xive_provision_chips = kcalloc(4, xive_provision_chip_count, GFP_KERNEL); if (WARN_ON(!xive_provision_chips)) return false; diff --git a/arch/powerpc/sysdev/xive/spapr.c b/arch/powerpc/sysdev/xive/spapr.c index 091f1d0d0af1..575db3b06a6b 100644 --- a/arch/powerpc/sysdev/xive/spapr.c +++ b/arch/powerpc/sysdev/xive/spapr.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include @@ -108,6 +109,51 @@ static void xive_irq_bitmap_free(int irq) } } + +/* Based on the similar routines in RTAS */ +static unsigned int plpar_busy_delay_time(long rc) +{ + unsigned int ms = 0; + + if (H_IS_LONG_BUSY(rc)) { + ms = get_longbusy_msecs(rc); + } else if (rc == H_BUSY) { + ms = 10; /* seems appropriate for XIVE hcalls */ + } + + return ms; +} + +static unsigned int plpar_busy_delay(int rc) +{ + unsigned int ms; + + ms = plpar_busy_delay_time(rc); + if (ms) + mdelay(ms); + + return ms; +} + +/* + * Note: this call has a partition wide scope and can take a while to + * complete. If it returns H_LONG_BUSY_* it should be retried + * periodically. + */ +static long plpar_int_reset(unsigned long flags) +{ + long rc; + + do { + rc = plpar_hcall_norets(H_INT_RESET, flags); + } while (plpar_busy_delay(rc)); + + if (rc) + pr_err("H_INT_RESET failed %ld\n", rc); + + return rc; +} + static long plpar_int_get_source_info(unsigned long flags, unsigned long lisn, unsigned long *src_flags, @@ -118,7 +164,10 @@ static long plpar_int_get_source_info(unsigned long flags, unsigned long retbuf[PLPAR_HCALL_BUFSIZE]; long rc; - rc = plpar_hcall(H_INT_GET_SOURCE_INFO, retbuf, flags, lisn); + do { + rc = plpar_hcall(H_INT_GET_SOURCE_INFO, retbuf, flags, lisn); + } while (plpar_busy_delay(rc)); + if (rc) { pr_err("H_INT_GET_SOURCE_INFO lisn=%ld failed %ld\n", lisn, rc); return rc; @@ -151,8 +200,11 @@ static long plpar_int_set_source_config(unsigned long flags, flags, lisn, target, prio, sw_irq); - rc = plpar_hcall_norets(H_INT_SET_SOURCE_CONFIG, flags, lisn, - target, prio, sw_irq); + do { + rc = plpar_hcall_norets(H_INT_SET_SOURCE_CONFIG, flags, lisn, + target, prio, sw_irq); + } while (plpar_busy_delay(rc)); + if (rc) { pr_err("H_INT_SET_SOURCE_CONFIG lisn=%ld target=%lx prio=%lx failed %ld\n", lisn, target, prio, rc); @@ -171,7 +223,11 @@ static long plpar_int_get_queue_info(unsigned long flags, unsigned long retbuf[PLPAR_HCALL_BUFSIZE]; long rc; - rc = plpar_hcall(H_INT_GET_QUEUE_INFO, retbuf, flags, target, priority); + do { + rc = plpar_hcall(H_INT_GET_QUEUE_INFO, retbuf, flags, target, + priority); + } while (plpar_busy_delay(rc)); + if (rc) { pr_err("H_INT_GET_QUEUE_INFO cpu=%ld prio=%ld failed %ld\n", target, priority, rc); @@ -200,8 +256,11 @@ static long plpar_int_set_queue_config(unsigned long flags, pr_devel("H_INT_SET_QUEUE_CONFIG flags=%lx target=%lx priority=%lx qpage=%lx qsize=%lx\n", flags, target, priority, qpage, qsize); - rc = plpar_hcall_norets(H_INT_SET_QUEUE_CONFIG, flags, target, - priority, qpage, qsize); + do { + rc = plpar_hcall_norets(H_INT_SET_QUEUE_CONFIG, flags, target, + priority, qpage, qsize); + } while (plpar_busy_delay(rc)); + if (rc) { pr_err("H_INT_SET_QUEUE_CONFIG cpu=%ld prio=%ld qpage=%lx returned %ld\n", target, priority, qpage, rc); @@ -215,7 +274,10 @@ static long plpar_int_sync(unsigned long flags, unsigned long lisn) { long rc; - rc = plpar_hcall_norets(H_INT_SYNC, flags, lisn); + do { + rc = plpar_hcall_norets(H_INT_SYNC, flags, lisn); + } while (plpar_busy_delay(rc)); + if (rc) { pr_err("H_INT_SYNC lisn=%ld returned %ld\n", lisn, rc); return rc; @@ -238,7 +300,11 @@ static long plpar_int_esb(unsigned long flags, pr_devel("H_INT_ESB flags=%lx lisn=%lx offset=%lx in=%lx\n", flags, lisn, offset, in_data); - rc = plpar_hcall(H_INT_ESB, retbuf, flags, lisn, offset, in_data); + do { + rc = plpar_hcall(H_INT_ESB, retbuf, flags, lisn, offset, + in_data); + } while (plpar_busy_delay(rc)); + if (rc) { pr_err("H_INT_ESB lisn=%ld offset=%ld returned %ld\n", lisn, offset, rc); @@ -445,11 +511,7 @@ static void xive_spapr_put_ipi(unsigned int cpu, struct xive_cpu *xc) static void xive_spapr_shutdown(void) { - long rc; - - rc = plpar_hcall_norets(H_INT_RESET, 0); - if (rc) - pr_err("H_INT_RESET failed %ld\n", rc); + plpar_int_reset(0); } /* diff --git a/arch/powerpc/tools/gcc-check-mprofile-kernel.sh b/arch/powerpc/tools/gcc-check-mprofile-kernel.sh index 061f8035bdbe..137f3376ac2b 100755 --- a/arch/powerpc/tools/gcc-check-mprofile-kernel.sh +++ b/arch/powerpc/tools/gcc-check-mprofile-kernel.sh @@ -7,18 +7,21 @@ set -o pipefail # To debug, uncomment the following line # set -x +# -mprofile-kernel is only supported on 64le, so this should not be invoked +# for other targets. Therefore we can pass in -m64 and -mlittle-endian +# explicitly, to take care of toolchains defaulting to other targets. + # Test whether the compile option -mprofile-kernel exists and generates # profiling code (ie. a call to _mcount()). echo "int func() { return 0; }" | \ - $* -S -x c -O2 -p -mprofile-kernel - -o - 2> /dev/null | \ - grep -q "_mcount" + $* -m64 -mlittle-endian -S -x c -O2 -p -mprofile-kernel - -o - \ + 2> /dev/null | grep -q "_mcount" # Test whether the notrace attribute correctly suppresses calls to _mcount(). echo -e "#include \nnotrace int func() { return 0; }" | \ - $* -S -x c -O2 -p -mprofile-kernel - -o - 2> /dev/null | \ - grep -q "_mcount" && \ + $* -m64 -mlittle-endian -S -x c -O2 -p -mprofile-kernel - -o - \ + 2> /dev/null | grep -q "_mcount" && \ exit 1 -echo "OK" exit 0 diff --git a/arch/powerpc/xmon/nonstdio.h b/arch/powerpc/xmon/nonstdio.h index 2202ec61972c..e8deac6c84e2 100644 --- a/arch/powerpc/xmon/nonstdio.h +++ b/arch/powerpc/xmon/nonstdio.h @@ -1,13 +1,13 @@ /* SPDX-License-Identifier: GPL-2.0 */ #define EOF (-1) -#define printf xmon_printf -#define putchar xmon_putchar - extern void xmon_set_pagination_lpp(unsigned long lpp); extern void xmon_start_pagination(void); extern void xmon_end_pagination(void); extern int xmon_putchar(int c); extern void xmon_puts(const char *); extern char *xmon_gets(char *, int); -extern void xmon_printf(const char *, ...); +extern __printf(1, 2) void xmon_printf(const char *fmt, ...); + +#define printf xmon_printf +#define putchar xmon_putchar diff --git a/arch/powerpc/xmon/spu-dis.c b/arch/powerpc/xmon/spu-dis.c index e5f89837c82e..4cbc7da88524 100644 --- a/arch/powerpc/xmon/spu-dis.c +++ b/arch/powerpc/xmon/spu-dis.c @@ -102,7 +102,7 @@ print_insn_spu (unsigned long insn, unsigned long memaddr) if (index == 0) { - printf(".long 0x%x", insn); + printf(".long 0x%lx", insn); } else { @@ -134,27 +134,27 @@ print_insn_spu (unsigned long insn, unsigned long memaddr) switch (arg) { case A_T: - printf("$%d", + printf("$%lu", DECODE_INSN_RT (insn)); break; case A_A: - printf("$%d", + printf("$%lu", DECODE_INSN_RA (insn)); break; case A_B: - printf("$%d", + printf("$%lu", DECODE_INSN_RB (insn)); break; case A_C: - printf("$%d", + printf("$%lu", DECODE_INSN_RC (insn)); break; case A_S: - printf("$sp%d", + printf("$sp%lu", DECODE_INSN_RA (insn)); break; case A_H: - printf("$ch%d", + printf("$ch%lu", DECODE_INSN_RA (insn)); break; case A_P: @@ -162,11 +162,11 @@ print_insn_spu (unsigned long insn, unsigned long memaddr) printf("("); break; case A_U7A: - printf("%d", + printf("%lu", 173 - DECODE_INSN_U8 (insn)); break; case A_U7B: - printf("%d", + printf("%lu", 155 - DECODE_INSN_U8 (insn)); break; case A_S3: diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c index a0842f1ff72c..47166ad2a669 100644 --- a/arch/powerpc/xmon/xmon.c +++ b/arch/powerpc/xmon/xmon.c @@ -515,7 +515,7 @@ static int xmon_core(struct pt_regs *regs, int fromipi) get_output_lock(); excprint(regs); if (bp) { - printf("cpu 0x%x stopped at breakpoint 0x%lx (", + printf("cpu 0x%x stopped at breakpoint 0x%tx (", cpu, BP_NUM(bp)); xmon_print_symbol(regs->nip, " ", ")\n"); } @@ -622,7 +622,7 @@ static int xmon_core(struct pt_regs *regs, int fromipi) excprint(regs); bp = at_breakpoint(regs->nip); if (bp) { - printf("Stopped at breakpoint %lx (", BP_NUM(bp)); + printf("Stopped at breakpoint %tx (", BP_NUM(bp)); xmon_print_symbol(regs->nip, " ", ")\n"); } if (unrecoverable_excp(regs)) @@ -778,6 +778,16 @@ static int xmon_fault_handler(struct pt_regs *regs) return 0; } +/* Force enable xmon if not already enabled */ +static inline void force_enable_xmon(void) +{ + /* Enable xmon hooks if needed */ + if (!xmon_on) { + printf("xmon: Enabling debugger hooks\n"); + xmon_on = 1; + } +} + static struct bpt *at_breakpoint(unsigned long pc) { int i; @@ -1094,6 +1104,7 @@ static int do_step(struct pt_regs *regs) unsigned int instr; int stepped; + force_enable_xmon(); /* check we are in 64-bit kernel mode, translation enabled */ if ((regs->msr & (MSR_64BIT|MSR_PR|MSR_IR)) == (MSR_64BIT|MSR_IR)) { if (mread(regs->nip, &instr, 4) == 4) { @@ -1160,7 +1171,11 @@ static int cpu_cmd(void) } /* try to switch to cpu specified */ if (!cpumask_test_cpu(cpu, &cpus_in_xmon)) { - printf("cpu 0x%x isn't in xmon\n", cpu); + printf("cpu 0x%lx isn't in xmon\n", cpu); +#ifdef CONFIG_PPC64 + printf("backtrace of paca[0x%lx].saved_r1 (possibly stale):\n", cpu); + xmon_show_stack(paca_ptrs[cpu]->saved_r1, 0, 0); +#endif return 0; } xmon_taken = 0; @@ -1174,7 +1189,7 @@ static int cpu_cmd(void) /* take control back */ mb(); xmon_owner = smp_processor_id(); - printf("cpu 0x%x didn't take control\n", cpu); + printf("cpu 0x%lx didn't take control\n", cpu); return 0; } barrier(); @@ -1268,16 +1283,6 @@ static long check_bp_loc(unsigned long addr) return 1; } -/* Force enable xmon if not already enabled */ -static inline void force_enable_xmon(void) -{ - /* Enable xmon hooks if needed */ - if (!xmon_on) { - printf("xmon: Enabling debugger hooks\n"); - xmon_on = 1; - } -} - static char *breakpoint_help_string = "Breakpoint command usage:\n" "b show breakpoints\n" @@ -1374,7 +1379,7 @@ bpt_cmds(void) } } - printf("Cleared breakpoint %lx (", BP_NUM(bp)); + printf("Cleared breakpoint %tx (", BP_NUM(bp)); xmon_print_symbol(bp->address, " ", ")\n"); bp->enabled = 0; break; @@ -1401,7 +1406,7 @@ bpt_cmds(void) for (bp = bpts; bp < &bpts[NBPTS]; ++bp) { if (!bp->enabled) continue; - printf("%2x %s ", BP_NUM(bp), + printf("%tx %s ", BP_NUM(bp), (bp->enabled & BP_CIABR) ? "inst": "trap"); xmon_print_symbol(bp->address, " ", "\n"); } @@ -1618,11 +1623,11 @@ static void excprint(struct pt_regs *fp) #endif /* CONFIG_SMP */ trap = TRAP(fp); - printf("Vector: %lx %s at [%lx]\n", fp->trap, getvecname(trap), fp); + printf("Vector: %lx %s at [%px]\n", fp->trap, getvecname(trap), fp); printf(" pc: "); xmon_print_symbol(fp->nip, ": ", "\n"); - printf(" lr: ", fp->link); + printf(" lr: "); xmon_print_symbol(fp->link, ": ", "\n"); printf(" sp: %lx\n", fp->gpr[1]); @@ -1634,13 +1639,13 @@ static void excprint(struct pt_regs *fp) printf(" dsisr: %lx\n", fp->dsisr); } - printf(" current = 0x%lx\n", current); + printf(" current = 0x%px\n", current); #ifdef CONFIG_PPC64 - printf(" paca = 0x%lx\t softe: %d\t irq_happened: 0x%02x\n", + printf(" paca = 0x%px\t irqmask: 0x%02x\t irq_happened: 0x%02x\n", local_paca, local_paca->irq_soft_mask, local_paca->irq_happened); #endif if (current) { - printf(" pid = %ld, comm = %s\n", + printf(" pid = %d, comm = %s\n", current->pid, current->comm); } @@ -1676,16 +1681,16 @@ static void prregs(struct pt_regs *fp) #ifdef CONFIG_PPC64 if (FULL_REGS(fp)) { for (n = 0; n < 16; ++n) - printf("R%.2ld = "REG" R%.2ld = "REG"\n", + printf("R%.2d = "REG" R%.2d = "REG"\n", n, fp->gpr[n], n+16, fp->gpr[n+16]); } else { for (n = 0; n < 7; ++n) - printf("R%.2ld = "REG" R%.2ld = "REG"\n", + printf("R%.2d = "REG" R%.2d = "REG"\n", n, fp->gpr[n], n+7, fp->gpr[n+7]); } #else for (n = 0; n < 32; ++n) { - printf("R%.2d = %.8x%s", n, fp->gpr[n], + printf("R%.2d = %.8lx%s", n, fp->gpr[n], (n & 3) == 3? "\n": " "); if (n == 12 && !FULL_REGS(fp)) { printf("\n"); @@ -1789,9 +1794,9 @@ static void dump_206_sprs(void) /* Actually some of these pre-date 2.06, but whatevs */ - printf("srr0 = %.16lx srr1 = %.16lx dsisr = %.8x\n", + printf("srr0 = %.16lx srr1 = %.16lx dsisr = %.8lx\n", mfspr(SPRN_SRR0), mfspr(SPRN_SRR1), mfspr(SPRN_DSISR)); - printf("dscr = %.16lx ppr = %.16lx pir = %.8x\n", + printf("dscr = %.16lx ppr = %.16lx pir = %.8lx\n", mfspr(SPRN_DSCR), mfspr(SPRN_PPR), mfspr(SPRN_PIR)); printf("amr = %.16lx uamor = %.16lx\n", mfspr(SPRN_AMR), mfspr(SPRN_UAMOR)); @@ -1799,11 +1804,11 @@ static void dump_206_sprs(void) if (!(mfmsr() & MSR_HV)) return; - printf("sdr1 = %.16lx hdar = %.16lx hdsisr = %.8x\n", + printf("sdr1 = %.16lx hdar = %.16lx hdsisr = %.8lx\n", mfspr(SPRN_SDR1), mfspr(SPRN_HDAR), mfspr(SPRN_HDSISR)); printf("hsrr0 = %.16lx hsrr1 = %.16lx hdec = %.16lx\n", mfspr(SPRN_HSRR0), mfspr(SPRN_HSRR1), mfspr(SPRN_HDEC)); - printf("lpcr = %.16lx pcr = %.16lx lpidr = %.8x\n", + printf("lpcr = %.16lx pcr = %.16lx lpidr = %.8lx\n", mfspr(SPRN_LPCR), mfspr(SPRN_PCR), mfspr(SPRN_LPID)); printf("hsprg0 = %.16lx hsprg1 = %.16lx amor = %.16lx\n", mfspr(SPRN_HSPRG0), mfspr(SPRN_HSPRG1), mfspr(SPRN_AMOR)); @@ -1820,10 +1825,10 @@ static void dump_207_sprs(void) if (!cpu_has_feature(CPU_FTR_ARCH_207S)) return; - printf("dpdes = %.16lx tir = %.16lx cir = %.8x\n", + printf("dpdes = %.16lx tir = %.16lx cir = %.8lx\n", mfspr(SPRN_DPDES), mfspr(SPRN_TIR), mfspr(SPRN_CIR)); - printf("fscr = %.16lx tar = %.16lx pspb = %.8x\n", + printf("fscr = %.16lx tar = %.16lx pspb = %.8lx\n", mfspr(SPRN_FSCR), mfspr(SPRN_TAR), mfspr(SPRN_PSPB)); msr = mfmsr(); @@ -1836,12 +1841,12 @@ static void dump_207_sprs(void) printf("mmcr0 = %.16lx mmcr1 = %.16lx mmcr2 = %.16lx\n", mfspr(SPRN_MMCR0), mfspr(SPRN_MMCR1), mfspr(SPRN_MMCR2)); - printf("pmc1 = %.8x pmc2 = %.8x pmc3 = %.8x pmc4 = %.8x\n", + printf("pmc1 = %.8lx pmc2 = %.8lx pmc3 = %.8lx pmc4 = %.8lx\n", mfspr(SPRN_PMC1), mfspr(SPRN_PMC2), mfspr(SPRN_PMC3), mfspr(SPRN_PMC4)); - printf("mmcra = %.16lx siar = %.16lx pmc5 = %.8x\n", + printf("mmcra = %.16lx siar = %.16lx pmc5 = %.8lx\n", mfspr(SPRN_MMCRA), mfspr(SPRN_SIAR), mfspr(SPRN_PMC5)); - printf("sdar = %.16lx sier = %.16lx pmc6 = %.8x\n", + printf("sdar = %.16lx sier = %.16lx pmc6 = %.8lx\n", mfspr(SPRN_SDAR), mfspr(SPRN_SIER), mfspr(SPRN_PMC6)); printf("ebbhr = %.16lx ebbrr = %.16lx bescr = %.16lx\n", mfspr(SPRN_EBBHR), mfspr(SPRN_EBBRR), mfspr(SPRN_BESCR)); @@ -2345,31 +2350,31 @@ static void dump_one_paca(int cpu) printf("paca for cpu 0x%x @ %px:\n", cpu, p); - printf(" %-*s = %s\n", 20, "possible", cpu_possible(cpu) ? "yes" : "no"); - printf(" %-*s = %s\n", 20, "present", cpu_present(cpu) ? "yes" : "no"); - printf(" %-*s = %s\n", 20, "online", cpu_online(cpu) ? "yes" : "no"); + printf(" %-*s = %s\n", 25, "possible", cpu_possible(cpu) ? "yes" : "no"); + printf(" %-*s = %s\n", 25, "present", cpu_present(cpu) ? "yes" : "no"); + printf(" %-*s = %s\n", 25, "online", cpu_online(cpu) ? "yes" : "no"); -#define DUMP(paca, name, format) \ - printf(" %-*s = %#-*"format"\t(0x%lx)\n", 20, #name, 18, paca->name, \ +#define DUMP(paca, name, format) \ + printf(" %-*s = "format"\t(0x%lx)\n", 25, #name, 18, paca->name, \ offsetof(struct paca_struct, name)); - DUMP(p, lock_token, "x"); - DUMP(p, paca_index, "x"); - DUMP(p, kernel_toc, "lx"); - DUMP(p, kernelbase, "lx"); - DUMP(p, kernel_msr, "lx"); - DUMP(p, emergency_sp, "px"); + DUMP(p, lock_token, "%#-*x"); + DUMP(p, paca_index, "%#-*x"); + DUMP(p, kernel_toc, "%#-*llx"); + DUMP(p, kernelbase, "%#-*llx"); + DUMP(p, kernel_msr, "%#-*llx"); + DUMP(p, emergency_sp, "%-*px"); #ifdef CONFIG_PPC_BOOK3S_64 - DUMP(p, nmi_emergency_sp, "px"); - DUMP(p, mc_emergency_sp, "px"); - DUMP(p, in_nmi, "x"); - DUMP(p, in_mce, "x"); - DUMP(p, hmi_event_available, "x"); + DUMP(p, nmi_emergency_sp, "%-*px"); + DUMP(p, mc_emergency_sp, "%-*px"); + DUMP(p, in_nmi, "%#-*x"); + DUMP(p, in_mce, "%#-*x"); + DUMP(p, hmi_event_available, "%#-*x"); #endif - DUMP(p, data_offset, "lx"); - DUMP(p, hw_cpu_id, "x"); - DUMP(p, cpu_start, "x"); - DUMP(p, kexec_state, "x"); + DUMP(p, data_offset, "%#-*llx"); + DUMP(p, hw_cpu_id, "%#-*x"); + DUMP(p, cpu_start, "%#-*x"); + DUMP(p, kexec_state, "%#-*x"); #ifdef CONFIG_PPC_BOOK3S_64 for (i = 0; i < SLB_NUM_BOLTED; i++) { u64 esid, vsid; @@ -2381,58 +2386,69 @@ static void dump_one_paca(int cpu) vsid = be64_to_cpu(p->slb_shadow_ptr->save_area[i].vsid); if (esid || vsid) { - printf(" slb_shadow[%d]: = 0x%016lx 0x%016lx\n", - i, esid, vsid); + printf(" %-*s[%d] = 0x%016llx 0x%016llx\n", + 22, "slb_shadow", i, esid, vsid); } } - DUMP(p, vmalloc_sllp, "x"); - DUMP(p, slb_cache_ptr, "x"); + DUMP(p, vmalloc_sllp, "%#-*x"); + DUMP(p, slb_cache_ptr, "%#-*x"); for (i = 0; i < SLB_CACHE_ENTRIES; i++) - printf(" slb_cache[%d]: = 0x%016lx\n", i, p->slb_cache[i]); + printf(" %-*s[%d] = 0x%016x\n", + 22, "slb_cache", i, p->slb_cache[i]); - DUMP(p, rfi_flush_fallback_area, "px"); + DUMP(p, rfi_flush_fallback_area, "%-*px"); #endif - DUMP(p, dscr_default, "llx"); + DUMP(p, dscr_default, "%#-*llx"); #ifdef CONFIG_PPC_BOOK3E - DUMP(p, pgd, "px"); - DUMP(p, kernel_pgd, "px"); - DUMP(p, tcd_ptr, "px"); - DUMP(p, mc_kstack, "px"); - DUMP(p, crit_kstack, "px"); - DUMP(p, dbg_kstack, "px"); + DUMP(p, pgd, "%-*px"); + DUMP(p, kernel_pgd, "%-*px"); + DUMP(p, tcd_ptr, "%-*px"); + DUMP(p, mc_kstack, "%-*px"); + DUMP(p, crit_kstack, "%-*px"); + DUMP(p, dbg_kstack, "%-*px"); #endif - DUMP(p, __current, "px"); - DUMP(p, kstack, "lx"); - printf(" kstack_base = 0x%016lx\n", p->kstack & ~(THREAD_SIZE - 1)); - DUMP(p, stab_rr, "lx"); - DUMP(p, saved_r1, "lx"); - DUMP(p, trap_save, "x"); - DUMP(p, irq_soft_mask, "x"); - DUMP(p, irq_happened, "x"); - DUMP(p, io_sync, "x"); - DUMP(p, irq_work_pending, "x"); - DUMP(p, nap_state_lost, "x"); - DUMP(p, sprg_vdso, "llx"); + DUMP(p, __current, "%-*px"); + DUMP(p, kstack, "%#-*llx"); + printf(" %-*s = 0x%016llx\n", 25, "kstack_base", p->kstack & ~(THREAD_SIZE - 1)); + DUMP(p, stab_rr, "%#-*llx"); + DUMP(p, saved_r1, "%#-*llx"); + DUMP(p, trap_save, "%#-*x"); + DUMP(p, irq_soft_mask, "%#-*x"); + DUMP(p, irq_happened, "%#-*x"); + DUMP(p, io_sync, "%#-*x"); + DUMP(p, irq_work_pending, "%#-*x"); + DUMP(p, nap_state_lost, "%#-*x"); + DUMP(p, sprg_vdso, "%#-*llx"); #ifdef CONFIG_PPC_TRANSACTIONAL_MEM - DUMP(p, tm_scratch, "llx"); + DUMP(p, tm_scratch, "%#-*llx"); #endif #ifdef CONFIG_PPC_POWERNV - DUMP(p, core_idle_state_ptr, "px"); - DUMP(p, thread_idle_state, "x"); - DUMP(p, thread_mask, "x"); - DUMP(p, subcore_sibling_mask, "x"); + DUMP(p, core_idle_state_ptr, "%-*px"); + DUMP(p, thread_idle_state, "%#-*x"); + DUMP(p, thread_mask, "%#-*x"); + DUMP(p, subcore_sibling_mask, "%#-*x"); + DUMP(p, thread_sibling_pacas, "%-*px"); + DUMP(p, requested_psscr, "%#-*llx"); + DUMP(p, stop_sprs.pid, "%#-*llx"); + DUMP(p, stop_sprs.ldbar, "%#-*llx"); + DUMP(p, stop_sprs.fscr, "%#-*llx"); + DUMP(p, stop_sprs.hfscr, "%#-*llx"); + DUMP(p, stop_sprs.mmcr1, "%#-*llx"); + DUMP(p, stop_sprs.mmcr2, "%#-*llx"); + DUMP(p, stop_sprs.mmcra, "%#-*llx"); + DUMP(p, dont_stop.counter, "%#-*x"); #endif - DUMP(p, accounting.utime, "llx"); - DUMP(p, accounting.stime, "llx"); - DUMP(p, accounting.utime_scaled, "llx"); - DUMP(p, accounting.starttime, "llx"); - DUMP(p, accounting.starttime_user, "llx"); - DUMP(p, accounting.startspurr, "llx"); - DUMP(p, accounting.utime_sspurr, "llx"); - DUMP(p, accounting.steal_time, "llx"); + DUMP(p, accounting.utime, "%#-*lx"); + DUMP(p, accounting.stime, "%#-*lx"); + DUMP(p, accounting.utime_scaled, "%#-*lx"); + DUMP(p, accounting.starttime, "%#-*lx"); + DUMP(p, accounting.starttime_user, "%#-*lx"); + DUMP(p, accounting.startspurr, "%#-*lx"); + DUMP(p, accounting.utime_sspurr, "%#-*lx"); + DUMP(p, accounting.steal_time, "%#-*lx"); #undef DUMP catch_memory_errors = 0; @@ -2578,7 +2594,7 @@ static void dump_by_size(unsigned long addr, long count, int size) default: val = 0; } - printf("%0*lx", size * 2, val); + printf("%0*llx", size * 2, val); } printf("\n"); } @@ -2742,7 +2758,7 @@ generic_inst_dump(unsigned long adr, long count, int praddr, dotted = 0; last_inst = inst; if (praddr) - printf(REG" %.8x", adr, inst); + printf(REG" %.8lx", adr, inst); printf("\t"); dump_func(inst, adr); printf("\n"); @@ -2874,7 +2890,7 @@ memdiffs(unsigned char *p1, unsigned char *p2, unsigned nb, unsigned maxpr) for( n = nb; n > 0; --n ) if( *p1++ != *p2++ ) if( ++prt <= maxpr ) - printf("%.16x %.2x # %.16x %.2x\n", p1 - 1, + printf("%px %.2x # %px %.2x\n", p1 - 1, p1[-1], p2 - 1, p2[-1]); if( prt > maxpr ) printf("Total of %d differences\n", prt); @@ -2934,13 +2950,13 @@ memzcan(void) if (ok && !ook) { printf("%.8x .. ", a); } else if (!ok && ook) - printf("%.8x\n", a - mskip); + printf("%.8lx\n", a - mskip); ook = ok; if (a + mskip < a) break; } if (ook) - printf("%.8x\n", a - mskip); + printf("%.8lx\n", a - mskip); } static void show_task(struct task_struct *tsk) @@ -3024,13 +3040,13 @@ static void show_pte(unsigned long addr) return; } - printf("pgd @ 0x%016lx\n", pgdir); + printf("pgd @ 0x%px\n", pgdir); if (pgd_huge(*pgdp)) { format_pte(pgdp, pgd_val(*pgdp)); return; } - printf("pgdp @ 0x%016lx = 0x%016lx\n", pgdp, pgd_val(*pgdp)); + printf("pgdp @ 0x%px = 0x%016lx\n", pgdp, pgd_val(*pgdp)); pudp = pud_offset(pgdp, addr); @@ -3044,7 +3060,7 @@ static void show_pte(unsigned long addr) return; } - printf("pudp @ 0x%016lx = 0x%016lx\n", pudp, pud_val(*pudp)); + printf("pudp @ 0x%px = 0x%016lx\n", pudp, pud_val(*pudp)); pmdp = pmd_offset(pudp, addr); @@ -3057,7 +3073,7 @@ static void show_pte(unsigned long addr) format_pte(pmdp, pmd_val(*pmdp)); return; } - printf("pmdp @ 0x%016lx = 0x%016lx\n", pmdp, pmd_val(*pmdp)); + printf("pmdp @ 0x%px = 0x%016lx\n", pmdp, pmd_val(*pmdp)); ptep = pte_offset_map(pmdp, addr); if (pte_none(*ptep)) { @@ -3161,7 +3177,7 @@ skipbl(void) } #define N_PTREGS 44 -static char *regnames[N_PTREGS] = { +static const char *regnames[N_PTREGS] = { "r0", "r1", "r2", "r3", "r4", "r5", "r6", "r7", "r8", "r9", "r10", "r11", "r12", "r13", "r14", "r15", "r16", "r17", "r18", "r19", "r20", "r21", "r22", "r23", @@ -3196,18 +3212,17 @@ scanhex(unsigned long *vp) regname[i] = c; } regname[i] = 0; - for (i = 0; i < N_PTREGS; ++i) { - if (strcmp(regnames[i], regname) == 0) { - if (xmon_regs == NULL) { - printf("regs not available\n"); - return 0; - } - *vp = ((unsigned long *)xmon_regs)[i]; - return 1; - } + i = match_string(regnames, N_PTREGS, regname); + if (i < 0) { + printf("invalid register name '%%%s'\n", regname); + return 0; } - printf("invalid register name '%%%s'\n", regname); - return 0; + if (xmon_regs == NULL) { + printf("regs not available\n"); + return 0; + } + *vp = ((unsigned long *)xmon_regs)[i]; + return 1; } /* skip leading "0x" if any */ @@ -3456,9 +3471,9 @@ static void dump_tlb_44x(void) asm volatile("tlbre %0,%1,0" : "=r" (w0) : "r" (i)); asm volatile("tlbre %0,%1,1" : "=r" (w1) : "r" (i)); asm volatile("tlbre %0,%1,2" : "=r" (w2) : "r" (i)); - printf("[%02x] %08x %08x %08x ", i, w0, w1, w2); + printf("[%02x] %08lx %08lx %08lx ", i, w0, w1, w2); if (w0 & PPC44x_TLB_VALID) { - printf("V %08x -> %01x%08x %c%c%c%c%c", + printf("V %08lx -> %01lx%08lx %c%c%c%c%c", w0 & PPC44x_TLB_EPN_MASK, w1 & PPC44x_TLB_ERPN_MASK, w1 & PPC44x_TLB_RPN_MASK, @@ -3882,19 +3897,19 @@ static void dump_spu_fields(struct spu *spu) DUMP_FIELD(spu, "0x%lx", ls_size); DUMP_FIELD(spu, "0x%x", node); DUMP_FIELD(spu, "0x%lx", flags); - DUMP_FIELD(spu, "%d", class_0_pending); - DUMP_FIELD(spu, "0x%lx", class_0_dar); - DUMP_FIELD(spu, "0x%lx", class_1_dar); - DUMP_FIELD(spu, "0x%lx", class_1_dsisr); - DUMP_FIELD(spu, "0x%lx", irqs[0]); - DUMP_FIELD(spu, "0x%lx", irqs[1]); - DUMP_FIELD(spu, "0x%lx", irqs[2]); + DUMP_FIELD(spu, "%llu", class_0_pending); + DUMP_FIELD(spu, "0x%llx", class_0_dar); + DUMP_FIELD(spu, "0x%llx", class_1_dar); + DUMP_FIELD(spu, "0x%llx", class_1_dsisr); + DUMP_FIELD(spu, "0x%x", irqs[0]); + DUMP_FIELD(spu, "0x%x", irqs[1]); + DUMP_FIELD(spu, "0x%x", irqs[2]); DUMP_FIELD(spu, "0x%x", slb_replace); DUMP_FIELD(spu, "%d", pid); DUMP_FIELD(spu, "0x%p", mm); DUMP_FIELD(spu, "0x%p", ctx); DUMP_FIELD(spu, "0x%p", rq); - DUMP_FIELD(spu, "0x%p", timestamp); + DUMP_FIELD(spu, "0x%llx", timestamp); DUMP_FIELD(spu, "0x%lx", problem_phys); DUMP_FIELD(spu, "0x%p", problem); DUMP_VALUE("0x%x", problem->spu_runcntl_RW, @@ -3925,7 +3940,7 @@ static void dump_spu_ls(unsigned long num, int subcmd) __delay(200); } else { catch_memory_errors = 0; - printf("*** Error: accessing spu info for spu %d\n", num); + printf("*** Error: accessing spu info for spu %ld\n", num); return; } catch_memory_errors = 0; diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index cd4fd85fde84..f12680c9b947 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -3,8 +3,16 @@ # see Documentation/kbuild/kconfig-language.txt. # +config 64BIT + bool + +config 32BIT + bool + config RISCV def_bool y + # even on 32-bit, physical (and DMA) addresses are > 32-bits + select PHYS_ADDR_T_64BIT select OF select OF_EARLY_FLATTREE select OF_IRQ @@ -22,9 +30,9 @@ config RISCV select GENERIC_ATOMIC64 if !64BIT || !RISCV_ISA_A select HAVE_MEMBLOCK select HAVE_MEMBLOCK_NODE_MAP - select HAVE_DMA_API_DEBUG select HAVE_DMA_CONTIGUOUS select HAVE_GENERIC_DMA_COHERENT + select HAVE_PERF_EVENTS select IRQ_DOMAIN select NO_BOOTMEM select RISCV_ISA_A if SMP @@ -35,20 +43,14 @@ config RISCV select THREAD_INFO_IN_TASK select RISCV_TIMER select GENERIC_IRQ_MULTI_HANDLER + select ARCH_HAS_PTE_SPECIAL config MMU def_bool y -# even on 32-bit, physical (and DMA) addresses are > 32-bits -config ARCH_PHYS_ADDR_T_64BIT - def_bool y - config ZONE_DMA32 bool - default y - -config ARCH_DMA_ADDR_T_64BIT - def_bool y + default y if 64BIT config PAGE_OFFSET hex @@ -101,21 +103,20 @@ choice config ARCH_RV32I bool "RV32I" - select CPU_SUPPORTS_32BIT_KERNEL select 32BIT - select GENERIC_ASHLDI3 - select GENERIC_ASHRDI3 - select GENERIC_LSHRDI3 + select GENERIC_LIB_ASHLDI3 + select GENERIC_LIB_ASHRDI3 + select GENERIC_LIB_LSHRDI3 config ARCH_RV64I bool "RV64I" - select CPU_SUPPORTS_64BIT_KERNEL select 64BIT select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FTRACE_MCOUNT_RECORD select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS + select SWIOTLB endchoice @@ -171,11 +172,6 @@ config NR_CPUS depends on SMP default "8" -config CPU_SUPPORTS_32BIT_KERNEL - bool -config CPU_SUPPORTS_64BIT_KERNEL - bool - choice prompt "CPU Tuning" default TUNE_GENERIC @@ -198,27 +194,22 @@ config RISCV_ISA_C config RISCV_ISA_A def_bool y -endmenu - -menu "Kernel type" +menu "supported PMU type" + depends on PERF_EVENTS -choice - prompt "Kernel code model" - default 64BIT - -config 32BIT - bool "32-bit kernel" - depends on CPU_SUPPORTS_32BIT_KERNEL +config RISCV_BASE_PMU + bool "Base Performance Monitoring Unit" + def_bool y help - Select this option to build a 32-bit kernel. + A base PMU that serves as a reference implementation and has limited + feature of perf. It can run on any RISC-V machines so serves as the + fallback, but this option can also be disable to reduce kernel size. -config 64BIT - bool "64-bit kernel" - depends on CPU_SUPPORTS_64BIT_KERNEL - help - Select this option to build a 64-bit kernel. +endmenu -endchoice +endmenu + +menu "Kernel type" source "mm/Kconfig" diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 76e958a5414a..6d4a5f6c3f4f 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -71,6 +71,9 @@ KBUILD_CFLAGS_MODULE += $(call cc-option,-mno-relax) # architectures. It's faster to have GCC emit only aligned accesses. KBUILD_CFLAGS += $(call cc-option,-mstrict-align) +# arch specific predefines for sparse +CHECKFLAGS += -D__riscv -D__riscv_xlen=$(BITS) + head-y := arch/riscv/kernel/head.o core-y += arch/riscv/kernel/ arch/riscv/mm/ diff --git a/arch/riscv/configs/defconfig b/arch/riscv/configs/defconfig index bca0eee733b0..07326466871b 100644 --- a/arch/riscv/configs/defconfig +++ b/arch/riscv/configs/defconfig @@ -44,6 +44,7 @@ CONFIG_INPUT_MOUSEDEV=y CONFIG_SERIAL_8250=y CONFIG_SERIAL_8250_CONSOLE=y CONFIG_SERIAL_OF_PLATFORM=y +CONFIG_HVC_RISCV_SBI=y # CONFIG_PTP_1588_CLOCK is not set CONFIG_DRM=y CONFIG_DRM_RADEON=y diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 4286a5f83876..576ffdca06ba 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -25,6 +25,7 @@ generic-y += kdebug.h generic-y += kmap_types.h generic-y += kvm_para.h generic-y += local.h +generic-y += local64.h generic-y += mm-arch-hooks.h generic-y += mman.h generic-y += module.h diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/cacheflush.h index efd89a88d2d0..8f13074413a7 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -47,7 +47,7 @@ static inline void flush_dcache_page(struct page *page) #else /* CONFIG_SMP */ -#define flush_icache_all() sbi_remote_fence_i(0) +#define flush_icache_all() sbi_remote_fence_i(NULL) void flush_icache_mm(struct mm_struct *mm, bool local); #endif /* CONFIG_SMP */ diff --git a/arch/riscv/include/asm/dma-mapping.h b/arch/riscv/include/asm/dma-mapping.h new file mode 100644 index 000000000000..8facc1c8fa05 --- /dev/null +++ b/arch/riscv/include/asm/dma-mapping.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 +#ifndef _RISCV_ASM_DMA_MAPPING_H +#define _RISCV_ASM_DMA_MAPPING_H 1 + +#ifdef CONFIG_SWIOTLB +#include +static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus) +{ + return &swiotlb_dma_ops; +} +#else +#include +#endif /* CONFIG_SWIOTLB */ + +#endif /* _RISCV_ASM_DMA_MAPPING_H */ diff --git a/arch/riscv/include/asm/pci.h b/arch/riscv/include/asm/pci.h index 0f2fc9ef20fc..b3638c505728 100644 --- a/arch/riscv/include/asm/pci.h +++ b/arch/riscv/include/asm/pci.h @@ -26,9 +26,6 @@ /* RISC-V shim does not initialize PCI bus */ #define pcibios_assign_all_busses() 1 -/* We do not have an IOMMU */ -#define PCI_DMA_BUS_IS_PHYS 1 - extern int isa_dma_bridge_buggy; #ifdef CONFIG_PCI diff --git a/arch/riscv/include/asm/perf_event.h b/arch/riscv/include/asm/perf_event.h new file mode 100644 index 000000000000..0e638a0c3feb --- /dev/null +++ b/arch/riscv/include/asm/perf_event.h @@ -0,0 +1,84 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2018 SiFive + * Copyright (C) 2018 Andes Technology Corporation + * + */ + +#ifndef _ASM_RISCV_PERF_EVENT_H +#define _ASM_RISCV_PERF_EVENT_H + +#include +#include + +#define RISCV_BASE_COUNTERS 2 + +/* + * The RISCV_MAX_COUNTERS parameter should be specified. + */ + +#ifdef CONFIG_RISCV_BASE_PMU +#define RISCV_MAX_COUNTERS 2 +#endif + +#ifndef RISCV_MAX_COUNTERS +#error "Please provide a valid RISCV_MAX_COUNTERS for the PMU." +#endif + +/* + * These are the indexes of bits in counteren register *minus* 1, + * except for cycle. It would be coherent if it can directly mapped + * to counteren bit definition, but there is a *time* register at + * counteren[1]. Per-cpu structure is scarce resource here. + * + * According to the spec, an implementation can support counter up to + * mhpmcounter31, but many high-end processors has at most 6 general + * PMCs, we give the definition to MHPMCOUNTER8 here. + */ +#define RISCV_PMU_CYCLE 0 +#define RISCV_PMU_INSTRET 1 +#define RISCV_PMU_MHPMCOUNTER3 2 +#define RISCV_PMU_MHPMCOUNTER4 3 +#define RISCV_PMU_MHPMCOUNTER5 4 +#define RISCV_PMU_MHPMCOUNTER6 5 +#define RISCV_PMU_MHPMCOUNTER7 6 +#define RISCV_PMU_MHPMCOUNTER8 7 + +#define RISCV_OP_UNSUPP (-EOPNOTSUPP) + +struct cpu_hw_events { + /* # currently enabled events*/ + int n_events; + /* currently enabled events */ + struct perf_event *events[RISCV_MAX_COUNTERS]; + /* vendor-defined PMU data */ + void *platform; +}; + +struct riscv_pmu { + struct pmu *pmu; + + /* generic hw/cache events table */ + const int *hw_events; + const int (*cache_events)[PERF_COUNT_HW_CACHE_MAX] + [PERF_COUNT_HW_CACHE_OP_MAX] + [PERF_COUNT_HW_CACHE_RESULT_MAX]; + /* method used to map hw/cache events */ + int (*map_hw_event)(u64 config); + int (*map_cache_event)(u64 config); + + /* max generic hw events in map */ + int max_events; + /* number total counters, 2(base) + x(general) */ + int num_counters; + /* the width of the counter */ + int counter_width; + + /* vendor-defined PMU features */ + void *platform; + + irqreturn_t (*handle_irq)(int irq_num, void *dev); + int irq; +}; + +#endif /* _ASM_RISCV_PERF_EVENT_H */ diff --git a/arch/riscv/include/asm/pgtable-bits.h b/arch/riscv/include/asm/pgtable-bits.h index 997ddbb1d370..2fa2942be221 100644 --- a/arch/riscv/include/asm/pgtable-bits.h +++ b/arch/riscv/include/asm/pgtable-bits.h @@ -42,7 +42,4 @@ _PAGE_WRITE | _PAGE_EXEC | \ _PAGE_USER | _PAGE_GLOBAL)) -/* Advertise support for _PAGE_SPECIAL */ -#define __HAVE_ARCH_PTE_SPECIAL - #endif /* _ASM_RISCV_PGTABLE_BITS_H */ diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 7b209aec355d..85c2d8bae957 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -49,7 +49,7 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, #include -#define flush_tlb_all() sbi_remote_sfence_vma(0, 0, -1) +#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1) #define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0) #define flush_tlb_range(vma, start, end) \ sbi_remote_sfence_vma(mm_cpumask((vma)->vm_mm)->bits, \ diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index 14b0b22fb578..473cfc84e412 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -392,19 +392,21 @@ do { \ }) -extern unsigned long __must_check __copy_user(void __user *to, +extern unsigned long __must_check __asm_copy_to_user(void __user *to, + const void *from, unsigned long n); +extern unsigned long __must_check __asm_copy_from_user(void *to, const void __user *from, unsigned long n); static inline unsigned long raw_copy_from_user(void *to, const void __user *from, unsigned long n) { - return __copy_user(to, from, n); + return __asm_copy_to_user(to, from, n); } static inline unsigned long raw_copy_to_user(void __user *to, const void *from, unsigned long n) { - return __copy_user(to, from, n); + return __asm_copy_from_user(to, from, n); } extern long strncpy_from_user(char *dest, const char __user *src, long count); diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 8586dd96c2f0..e1274fc03af4 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -39,4 +39,6 @@ obj-$(CONFIG_MODULE_SECTIONS) += module-sections.o obj-$(CONFIG_FUNCTION_TRACER) += mcount.o ftrace.o obj-$(CONFIG_DYNAMIC_FTRACE) += mcount-dyn.o +obj-$(CONFIG_PERF_EVENTS) += perf_event.o + clean: diff --git a/arch/riscv/kernel/cacheinfo.c b/arch/riscv/kernel/cacheinfo.c index 10ed2749e246..0bc86e5f8f3f 100644 --- a/arch/riscv/kernel/cacheinfo.c +++ b/arch/riscv/kernel/cacheinfo.c @@ -20,7 +20,6 @@ static void ci_leaf_init(struct cacheinfo *this_leaf, struct device_node *node, enum cache_type type, unsigned int level) { - this_leaf->of_node = node; this_leaf->level = level; this_leaf->type = type; /* not a sector cache */ diff --git a/arch/riscv/kernel/mcount.S b/arch/riscv/kernel/mcount.S index ce9bdc57a2a1..5721624886a1 100644 --- a/arch/riscv/kernel/mcount.S +++ b/arch/riscv/kernel/mcount.S @@ -126,5 +126,5 @@ do_trace: RESTORE_ABI_STATE ret ENDPROC(_mcount) -EXPORT_SYMBOL(_mcount) #endif +EXPORT_SYMBOL(_mcount) diff --git a/arch/riscv/kernel/module.c b/arch/riscv/kernel/module.c index 5dddba301d0a..1d5e9b934b8c 100644 --- a/arch/riscv/kernel/module.c +++ b/arch/riscv/kernel/module.c @@ -17,6 +17,17 @@ #include #include +static int apply_r_riscv_32_rela(struct module *me, u32 *location, Elf_Addr v) +{ + if (v != (u32)v) { + pr_err("%s: value %016llx out of range for 32-bit field\n", + me->name, v); + return -EINVAL; + } + *location = v; + return 0; +} + static int apply_r_riscv_64_rela(struct module *me, u32 *location, Elf_Addr v) { *(u64 *)location = v; @@ -265,6 +276,7 @@ static int apply_r_riscv_sub32_rela(struct module *me, u32 *location, static int (*reloc_handlers_rela[]) (struct module *me, u32 *location, Elf_Addr v) = { + [R_RISCV_32] = apply_r_riscv_32_rela, [R_RISCV_64] = apply_r_riscv_64_rela, [R_RISCV_BRANCH] = apply_r_riscv_branch_rela, [R_RISCV_JAL] = apply_r_riscv_jal_rela, diff --git a/arch/riscv/kernel/perf_event.c b/arch/riscv/kernel/perf_event.c new file mode 100644 index 000000000000..b0e10c4e9f77 --- /dev/null +++ b/arch/riscv/kernel/perf_event.c @@ -0,0 +1,485 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2008 Thomas Gleixner + * Copyright (C) 2008-2009 Red Hat, Inc., Ingo Molnar + * Copyright (C) 2009 Jaswinder Singh Rajput + * Copyright (C) 2009 Advanced Micro Devices, Inc., Robert Richter + * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra + * Copyright (C) 2009 Intel Corporation, + * Copyright (C) 2009 Google, Inc., Stephane Eranian + * Copyright 2014 Tilera Corporation. All Rights Reserved. + * Copyright (C) 2018 Andes Technology Corporation + * + * Perf_events support for RISC-V platforms. + * + * Since the spec. (as of now, Priv-Spec 1.10) does not provide enough + * functionality for perf event to fully work, this file provides + * the very basic framework only. + * + * For platform portings, please check Documentations/riscv/pmu.txt. + * + * The Copyright line includes x86 and tile ones. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static const struct riscv_pmu *riscv_pmu __read_mostly; +static DEFINE_PER_CPU(struct cpu_hw_events, cpu_hw_events); + +/* + * Hardware & cache maps and their methods + */ + +static const int riscv_hw_event_map[] = { + [PERF_COUNT_HW_CPU_CYCLES] = RISCV_PMU_CYCLE, + [PERF_COUNT_HW_INSTRUCTIONS] = RISCV_PMU_INSTRET, + [PERF_COUNT_HW_CACHE_REFERENCES] = RISCV_OP_UNSUPP, + [PERF_COUNT_HW_CACHE_MISSES] = RISCV_OP_UNSUPP, + [PERF_COUNT_HW_BRANCH_INSTRUCTIONS] = RISCV_OP_UNSUPP, + [PERF_COUNT_HW_BRANCH_MISSES] = RISCV_OP_UNSUPP, + [PERF_COUNT_HW_BUS_CYCLES] = RISCV_OP_UNSUPP, +}; + +#define C(x) PERF_COUNT_HW_CACHE_##x +static const int riscv_cache_event_map[PERF_COUNT_HW_CACHE_MAX] +[PERF_COUNT_HW_CACHE_OP_MAX] +[PERF_COUNT_HW_CACHE_RESULT_MAX] = { + [C(L1D)] = { + [C(OP_READ)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_WRITE)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_PREFETCH)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + }, + [C(L1I)] = { + [C(OP_READ)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_WRITE)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_PREFETCH)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + }, + [C(LL)] = { + [C(OP_READ)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_WRITE)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_PREFETCH)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + }, + [C(DTLB)] = { + [C(OP_READ)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_WRITE)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_PREFETCH)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + }, + [C(ITLB)] = { + [C(OP_READ)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_WRITE)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_PREFETCH)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + }, + [C(BPU)] = { + [C(OP_READ)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_WRITE)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + [C(OP_PREFETCH)] = { + [C(RESULT_ACCESS)] = RISCV_OP_UNSUPP, + [C(RESULT_MISS)] = RISCV_OP_UNSUPP, + }, + }, +}; + +static int riscv_map_hw_event(u64 config) +{ + if (config >= riscv_pmu->max_events) + return -EINVAL; + + return riscv_pmu->hw_events[config]; +} + +int riscv_map_cache_decode(u64 config, unsigned int *type, + unsigned int *op, unsigned int *result) +{ + return -ENOENT; +} + +static int riscv_map_cache_event(u64 config) +{ + unsigned int type, op, result; + int err = -ENOENT; + int code; + + err = riscv_map_cache_decode(config, &type, &op, &result); + if (!riscv_pmu->cache_events || err) + return err; + + if (type >= PERF_COUNT_HW_CACHE_MAX || + op >= PERF_COUNT_HW_CACHE_OP_MAX || + result >= PERF_COUNT_HW_CACHE_RESULT_MAX) + return -EINVAL; + + code = (*riscv_pmu->cache_events)[type][op][result]; + if (code == RISCV_OP_UNSUPP) + return -EINVAL; + + return code; +} + +/* + * Low-level functions: reading/writing counters + */ + +static inline u64 read_counter(int idx) +{ + u64 val = 0; + + switch (idx) { + case RISCV_PMU_CYCLE: + val = csr_read(cycle); + break; + case RISCV_PMU_INSTRET: + val = csr_read(instret); + break; + default: + WARN_ON_ONCE(idx < 0 || idx > RISCV_MAX_COUNTERS); + return -EINVAL; + } + + return val; +} + +static inline void write_counter(int idx, u64 value) +{ + /* currently not supported */ + WARN_ON_ONCE(1); +} + +/* + * pmu->read: read and update the counter + * + * Other architectures' implementation often have a xxx_perf_event_update + * routine, which can return counter values when called in the IRQ, but + * return void when being called by the pmu->read method. + */ +static void riscv_pmu_read(struct perf_event *event) +{ + struct hw_perf_event *hwc = &event->hw; + u64 prev_raw_count, new_raw_count; + u64 oldval; + int idx = hwc->idx; + u64 delta; + + do { + prev_raw_count = local64_read(&hwc->prev_count); + new_raw_count = read_counter(idx); + + oldval = local64_cmpxchg(&hwc->prev_count, prev_raw_count, + new_raw_count); + } while (oldval != prev_raw_count); + + /* + * delta is the value to update the counter we maintain in the kernel. + */ + delta = (new_raw_count - prev_raw_count) & + ((1ULL << riscv_pmu->counter_width) - 1); + local64_add(delta, &event->count); + /* + * Something like local64_sub(delta, &hwc->period_left) here is + * needed if there is an interrupt for perf. + */ +} + +/* + * State transition functions: + * + * stop()/start() & add()/del() + */ + +/* + * pmu->stop: stop the counter + */ +static void riscv_pmu_stop(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc = &event->hw; + + WARN_ON_ONCE(hwc->state & PERF_HES_STOPPED); + hwc->state |= PERF_HES_STOPPED; + + if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE)) { + riscv_pmu->pmu->read(event); + hwc->state |= PERF_HES_UPTODATE; + } +} + +/* + * pmu->start: start the event. + */ +static void riscv_pmu_start(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc = &event->hw; + + if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED))) + return; + + if (flags & PERF_EF_RELOAD) { + WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE)); + + /* + * Set the counter to the period to the next interrupt here, + * if you have any. + */ + } + + hwc->state = 0; + perf_event_update_userpage(event); + + /* + * Since we cannot write to counters, this serves as an initialization + * to the delta-mechanism in pmu->read(); otherwise, the delta would be + * wrong when pmu->read is called for the first time. + */ + local64_set(&hwc->prev_count, read_counter(hwc->idx)); +} + +/* + * pmu->add: add the event to PMU. + */ +static int riscv_pmu_add(struct perf_event *event, int flags) +{ + struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); + struct hw_perf_event *hwc = &event->hw; + + if (cpuc->n_events == riscv_pmu->num_counters) + return -ENOSPC; + + /* + * We don't have general conunters, so no binding-event-to-counter + * process here. + * + * Indexing using hwc->config generally not works, since config may + * contain extra information, but here the only info we have in + * hwc->config is the event index. + */ + hwc->idx = hwc->config; + cpuc->events[hwc->idx] = event; + cpuc->n_events++; + + hwc->state = PERF_HES_UPTODATE | PERF_HES_STOPPED; + + if (flags & PERF_EF_START) + riscv_pmu->pmu->start(event, PERF_EF_RELOAD); + + return 0; +} + +/* + * pmu->del: delete the event from PMU. + */ +static void riscv_pmu_del(struct perf_event *event, int flags) +{ + struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); + struct hw_perf_event *hwc = &event->hw; + + cpuc->events[hwc->idx] = NULL; + cpuc->n_events--; + riscv_pmu->pmu->stop(event, PERF_EF_UPDATE); + perf_event_update_userpage(event); +} + +/* + * Interrupt: a skeletion for reference. + */ + +static DEFINE_MUTEX(pmc_reserve_mutex); + +irqreturn_t riscv_base_pmu_handle_irq(int irq_num, void *dev) +{ + return IRQ_NONE; +} + +static int reserve_pmc_hardware(void) +{ + int err = 0; + + mutex_lock(&pmc_reserve_mutex); + if (riscv_pmu->irq >= 0 && riscv_pmu->handle_irq) { + err = request_irq(riscv_pmu->irq, riscv_pmu->handle_irq, + IRQF_PERCPU, "riscv-base-perf", NULL); + } + mutex_unlock(&pmc_reserve_mutex); + + return err; +} + +void release_pmc_hardware(void) +{ + mutex_lock(&pmc_reserve_mutex); + if (riscv_pmu->irq >= 0) + free_irq(riscv_pmu->irq, NULL); + mutex_unlock(&pmc_reserve_mutex); +} + +/* + * Event Initialization/Finalization + */ + +static atomic_t riscv_active_events = ATOMIC_INIT(0); + +static void riscv_event_destroy(struct perf_event *event) +{ + if (atomic_dec_return(&riscv_active_events) == 0) + release_pmc_hardware(); +} + +static int riscv_event_init(struct perf_event *event) +{ + struct perf_event_attr *attr = &event->attr; + struct hw_perf_event *hwc = &event->hw; + int err; + int code; + + if (atomic_inc_return(&riscv_active_events) == 1) { + err = reserve_pmc_hardware(); + + if (err) { + pr_warn("PMC hardware not available\n"); + atomic_dec(&riscv_active_events); + return -EBUSY; + } + } + + switch (event->attr.type) { + case PERF_TYPE_HARDWARE: + code = riscv_pmu->map_hw_event(attr->config); + break; + case PERF_TYPE_HW_CACHE: + code = riscv_pmu->map_cache_event(attr->config); + break; + case PERF_TYPE_RAW: + return -EOPNOTSUPP; + default: + return -ENOENT; + } + + event->destroy = riscv_event_destroy; + if (code < 0) { + event->destroy(event); + return code; + } + + /* + * idx is set to -1 because the index of a general event should not be + * decided until binding to some counter in pmu->add(). + * + * But since we don't have such support, later in pmu->add(), we just + * use hwc->config as the index instead. + */ + hwc->config = code; + hwc->idx = -1; + + return 0; +} + +/* + * Initialization + */ + +static struct pmu min_pmu = { + .name = "riscv-base", + .event_init = riscv_event_init, + .add = riscv_pmu_add, + .del = riscv_pmu_del, + .start = riscv_pmu_start, + .stop = riscv_pmu_stop, + .read = riscv_pmu_read, +}; + +static const struct riscv_pmu riscv_base_pmu = { + .pmu = &min_pmu, + .max_events = ARRAY_SIZE(riscv_hw_event_map), + .map_hw_event = riscv_map_hw_event, + .hw_events = riscv_hw_event_map, + .map_cache_event = riscv_map_cache_event, + .cache_events = &riscv_cache_event_map, + .counter_width = 63, + .num_counters = RISCV_BASE_COUNTERS + 0, + .handle_irq = &riscv_base_pmu_handle_irq, + + /* This means this PMU has no IRQ. */ + .irq = -1, +}; + +static const struct of_device_id riscv_pmu_of_ids[] = { + {.compatible = "riscv,base-pmu", .data = &riscv_base_pmu}, + { /* sentinel value */ } +}; + +int __init init_hw_perf_events(void) +{ + struct device_node *node = of_find_node_by_type(NULL, "pmu"); + const struct of_device_id *of_id; + + riscv_pmu = &riscv_base_pmu; + + if (node) { + of_id = of_match_node(riscv_pmu_of_ids, node); + + if (of_id) + riscv_pmu = of_id->data; + } + + perf_pmu_register(riscv_pmu->pmu, "cpu", PERF_TYPE_RAW); + return 0; +} +arch_initcall(init_hw_perf_events); diff --git a/arch/riscv/kernel/riscv_ksyms.c b/arch/riscv/kernel/riscv_ksyms.c index 551734248748..f247d6d2137c 100644 --- a/arch/riscv/kernel/riscv_ksyms.c +++ b/arch/riscv/kernel/riscv_ksyms.c @@ -13,6 +13,7 @@ * Assembly functions that may be used (directly or indirectly) by modules */ EXPORT_SYMBOL(__clear_user); -EXPORT_SYMBOL(__copy_user); +EXPORT_SYMBOL(__asm_copy_to_user); +EXPORT_SYMBOL(__asm_copy_from_user); EXPORT_SYMBOL(memset); EXPORT_SYMBOL(memcpy); diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index c11f40c1b2a8..ee44a48faf79 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include @@ -206,6 +207,7 @@ void __init setup_arch(char **cmdline_p) setup_bootmem(); paging_init(); unflatten_device_tree(); + swiotlb_init(1); #ifdef CONFIG_SMP setup_smp(); diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index 93132cb59184..81a1952015a6 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -63,18 +63,6 @@ void die(struct pt_regs *regs, const char *str) do_exit(SIGSEGV); } -static inline void do_trap_siginfo(int signo, int code, - unsigned long addr, struct task_struct *tsk) -{ - siginfo_t info; - - info.si_signo = signo; - info.si_errno = 0; - info.si_code = code; - info.si_addr = (void __user *)addr; - force_sig_info(signo, &info, tsk); -} - void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr, struct task_struct *tsk) { @@ -87,7 +75,7 @@ void do_trap(struct pt_regs *regs, int signo, int code, show_regs(regs); } - do_trap_siginfo(signo, code, addr, tsk); + force_sig_fault(signo, code, (void __user *)addr, tsk); } static void do_trap_error(struct pt_regs *regs, int signo, int code, @@ -149,7 +137,7 @@ asmlinkage void do_trap_break(struct pt_regs *regs) } #endif /* CONFIG_GENERIC_BUG */ - do_trap_siginfo(SIGTRAP, TRAP_BRKPT, regs->sepc, current); + force_sig_fault(SIGTRAP, TRAP_BRKPT, (void __user *)(regs->sepc), current); regs->sepc += 0x4; } @@ -160,7 +148,7 @@ int is_valid_bugaddr(unsigned long pc) if (pc < PAGE_OFFSET) return 0; - if (probe_kernel_address((bug_insn_t __user *)pc, insn)) + if (probe_kernel_address((bug_insn_t *)pc, insn)) return 0; return (insn == __BUG_INSN); } diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S index 58fb2877c865..399e6f0c2d98 100644 --- a/arch/riscv/lib/uaccess.S +++ b/arch/riscv/lib/uaccess.S @@ -13,7 +13,8 @@ _epc: .previous .endm -ENTRY(__copy_user) +ENTRY(__asm_copy_to_user) +ENTRY(__asm_copy_from_user) /* Enable access to user memory */ li t6, SR_SUM @@ -63,7 +64,8 @@ ENTRY(__copy_user) addi a0, a0, 1 bltu a1, a3, 5b j 3b -ENDPROC(__copy_user) +ENDPROC(__asm_copy_to_user) +ENDPROC(__asm_copy_from_user) ENTRY(__clear_user) @@ -84,7 +86,7 @@ ENTRY(__clear_user) bgeu t0, t1, 2f bltu a0, t0, 4f 1: - fixup REG_S, zero, (a0), 10f + fixup REG_S, zero, (a0), 11f addi a0, a0, SZREG bltu a0, t1, 1b 2: @@ -96,12 +98,12 @@ ENTRY(__clear_user) li a0, 0 ret 4: /* Edge case: unalignment */ - fixup sb, zero, (a0), 10f + fixup sb, zero, (a0), 11f addi a0, a0, 1 bltu a0, t0, 4b j 1b 5: /* Edge case: remainder */ - fixup sb, zero, (a0), 10f + fixup sb, zero, (a0), 11f addi a0, a0, 1 bltu a0, a3, 5b j 3b @@ -109,9 +111,14 @@ ENDPROC(__clear_user) .section .fixup,"ax" .balign 4 + /* Fixup code for __copy_user(10) and __clear_user(11) */ 10: /* Disable access to user memory */ csrs sstatus, t6 - sub a0, a3, a0 + mv a0, a2 + ret +11: + csrs sstatus, t6 + mv a0, a1 ret .previous diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 199ac3e4da1d..baed39772c84 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -35,9 +35,6 @@ config GENERIC_BUG config GENERIC_BUG_RELATIVE_POINTERS def_bool y -config ARCH_DMA_ADDR_T_64BIT - def_bool y - config GENERIC_LOCKBREAK def_bool y if SMP && PREEMPT @@ -68,6 +65,7 @@ config S390 select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_GIGANTIC_PAGE if (MEMORY_ISOLATION && COMPACTION) || CMA select ARCH_HAS_KCOV + select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_SET_MEMORY select ARCH_HAS_SG_CHAIN select ARCH_HAS_STRICT_KERNEL_RWX @@ -133,7 +131,6 @@ config S390 select HAVE_CMPXCHG_LOCAL select HAVE_COPY_THREAD_TLS select HAVE_DEBUG_KMEMLEAK - select HAVE_DMA_API_DEBUG select HAVE_DMA_CONTIGUOUS select DMA_DIRECT_OPS select HAVE_DYNAMIC_FTRACE @@ -709,7 +706,11 @@ config QDIO menuconfig PCI bool "PCI support" select PCI_MSI + select IOMMU_HELPER select IOMMU_SUPPORT + select NEED_DMA_MAP_STATE + select NEED_SG_DMA_LENGTH + help Enable PCI support. @@ -733,15 +734,6 @@ config PCI_DOMAINS config HAS_IOMEM def_bool PCI -config IOMMU_HELPER - def_bool PCI - -config NEED_SG_DMA_LENGTH - def_bool PCI - -config NEED_DMA_MAP_STATE - def_bool PCI - config CHSC_SCH def_tristate m prompt "Support for CHSC subchannels" @@ -847,6 +839,10 @@ config CCW source "drivers/Kconfig" +config HAVE_PNETID + tristate + default (SMC || CCWGROUP) + source "fs/Kconfig" source "arch/s390/Kconfig.debug" diff --git a/arch/s390/Makefile b/arch/s390/Makefile index c79936d02f7b..68a690442be0 100644 --- a/arch/s390/Makefile +++ b/arch/s390/Makefile @@ -18,7 +18,7 @@ KBUILD_CFLAGS += -m64 KBUILD_AFLAGS += -m64 UTS_MACHINE := s390x STACK_SIZE := 16384 -CHECKFLAGS += -D__s390__ -D__s390x__ -mbig-endian +CHECKFLAGS += -D__s390__ -D__s390x__ export LD_BFD diff --git a/arch/s390/appldata/appldata_base.c b/arch/s390/appldata/appldata_base.c index cb6e8066b1ad..ee6a9c387c87 100644 --- a/arch/s390/appldata/appldata_base.c +++ b/arch/s390/appldata/appldata_base.c @@ -391,7 +391,7 @@ int appldata_register_ops(struct appldata_ops *ops) if (ops->size > APPLDATA_MAX_REC_SIZE) return -EINVAL; - ops->ctl_table = kzalloc(4 * sizeof(struct ctl_table), GFP_KERNEL); + ops->ctl_table = kcalloc(4, sizeof(struct ctl_table), GFP_KERNEL); if (!ops->ctl_table) return -ENOMEM; diff --git a/arch/s390/crypto/arch_random.c b/arch/s390/crypto/arch_random.c index 8720e9203ecf..dd95cdbd22ce 100644 --- a/arch/s390/crypto/arch_random.c +++ b/arch/s390/crypto/arch_random.c @@ -2,14 +2,37 @@ /* * s390 arch random implementation. * - * Copyright IBM Corp. 2017 - * Author(s): Harald Freudenberger + * Copyright IBM Corp. 2017, 2018 + * Author(s): Harald Freudenberger + * + * The s390_arch_random_generate() function may be called from random.c + * in interrupt context. So this implementation does the best to be very + * fast. There is a buffer of random data which is asynchronously checked + * and filled by a workqueue thread. + * If there are enough bytes in the buffer the s390_arch_random_generate() + * just delivers these bytes. Otherwise false is returned until the + * worker thread refills the buffer. + * The worker fills the rng buffer by pulling fresh entropy from the + * high quality (but slow) true hardware random generator. This entropy + * is then spread over the buffer with an pseudo random generator PRNG. + * As the arch_get_random_seed_long() fetches 8 bytes and the calling + * function add_interrupt_randomness() counts this as 1 bit entropy the + * distribution needs to make sure there is in fact 1 bit entropy contained + * in 8 bytes of the buffer. The current values pull 32 byte entropy + * and scatter this into a 2048 byte buffer. So 8 byte in the buffer + * will contain 1 bit of entropy. + * The worker thread is rescheduled based on the charge level of the + * buffer but at least with 500 ms delay to avoid too much CPU consumption. + * So the max. amount of rng data delivered via arch_get_random_seed is + * limited to 4k bytes per second. */ #include #include #include +#include #include +#include #include DEFINE_STATIC_KEY_FALSE(s390_arch_random_available); @@ -17,11 +40,83 @@ DEFINE_STATIC_KEY_FALSE(s390_arch_random_available); atomic64_t s390_arch_random_counter = ATOMIC64_INIT(0); EXPORT_SYMBOL(s390_arch_random_counter); +#define ARCH_REFILL_TICKS (HZ/2) +#define ARCH_PRNG_SEED_SIZE 32 +#define ARCH_RNG_BUF_SIZE 2048 + +static DEFINE_SPINLOCK(arch_rng_lock); +static u8 *arch_rng_buf; +static unsigned int arch_rng_buf_idx; + +static void arch_rng_refill_buffer(struct work_struct *); +static DECLARE_DELAYED_WORK(arch_rng_work, arch_rng_refill_buffer); + +bool s390_arch_random_generate(u8 *buf, unsigned int nbytes) +{ + /* lock rng buffer */ + if (!spin_trylock(&arch_rng_lock)) + return false; + + /* try to resolve the requested amount of bytes from the buffer */ + arch_rng_buf_idx -= nbytes; + if (arch_rng_buf_idx < ARCH_RNG_BUF_SIZE) { + memcpy(buf, arch_rng_buf + arch_rng_buf_idx, nbytes); + atomic64_add(nbytes, &s390_arch_random_counter); + spin_unlock(&arch_rng_lock); + return true; + } + + /* not enough bytes in rng buffer, refill is done asynchronously */ + spin_unlock(&arch_rng_lock); + + return false; +} +EXPORT_SYMBOL(s390_arch_random_generate); + +static void arch_rng_refill_buffer(struct work_struct *unused) +{ + unsigned int delay = ARCH_REFILL_TICKS; + + spin_lock(&arch_rng_lock); + if (arch_rng_buf_idx > ARCH_RNG_BUF_SIZE) { + /* buffer is exhausted and needs refill */ + u8 seed[ARCH_PRNG_SEED_SIZE]; + u8 prng_wa[240]; + /* fetch ARCH_PRNG_SEED_SIZE bytes of entropy */ + cpacf_trng(NULL, 0, seed, sizeof(seed)); + /* blow this entropy up to ARCH_RNG_BUF_SIZE with PRNG */ + memset(prng_wa, 0, sizeof(prng_wa)); + cpacf_prno(CPACF_PRNO_SHA512_DRNG_SEED, + &prng_wa, NULL, 0, seed, sizeof(seed)); + cpacf_prno(CPACF_PRNO_SHA512_DRNG_GEN, + &prng_wa, arch_rng_buf, ARCH_RNG_BUF_SIZE, NULL, 0); + arch_rng_buf_idx = ARCH_RNG_BUF_SIZE; + } + delay += (ARCH_REFILL_TICKS * arch_rng_buf_idx) / ARCH_RNG_BUF_SIZE; + spin_unlock(&arch_rng_lock); + + /* kick next check */ + queue_delayed_work(system_long_wq, &arch_rng_work, delay); +} + static int __init s390_arch_random_init(void) { - /* check if subfunction CPACF_PRNO_TRNG is available */ - if (cpacf_query_func(CPACF_PRNO, CPACF_PRNO_TRNG)) + /* all the needed PRNO subfunctions available ? */ + if (cpacf_query_func(CPACF_PRNO, CPACF_PRNO_TRNG) && + cpacf_query_func(CPACF_PRNO, CPACF_PRNO_SHA512_DRNG_GEN)) { + + /* alloc arch random working buffer */ + arch_rng_buf = kmalloc(ARCH_RNG_BUF_SIZE, GFP_KERNEL); + if (!arch_rng_buf) + return -ENOMEM; + + /* kick worker queue job to fill the random buffer */ + queue_delayed_work(system_long_wq, + &arch_rng_work, ARCH_REFILL_TICKS); + + /* enable arch random to the outside world */ static_branch_enable(&s390_arch_random_available); + } return 0; } diff --git a/arch/s390/hypfs/hypfs_diag.c b/arch/s390/hypfs/hypfs_diag.c index be8cc53204b5..a2945b289a29 100644 --- a/arch/s390/hypfs/hypfs_diag.c +++ b/arch/s390/hypfs/hypfs_diag.c @@ -239,7 +239,7 @@ static void *page_align_ptr(void *ptr) static void *diag204_alloc_vbuf(int pages) { /* The buffer has to be page aligned! */ - diag204_buf_vmalloc = vmalloc(PAGE_SIZE * (pages + 1)); + diag204_buf_vmalloc = vmalloc(array_size(PAGE_SIZE, (pages + 1))); if (!diag204_buf_vmalloc) return ERR_PTR(-ENOMEM); diag204_buf = page_align_ptr(diag204_buf_vmalloc); diff --git a/arch/s390/hypfs/hypfs_diag0c.c b/arch/s390/hypfs/hypfs_diag0c.c index dce87f1bec94..cebf05150cc1 100644 --- a/arch/s390/hypfs/hypfs_diag0c.c +++ b/arch/s390/hypfs/hypfs_diag0c.c @@ -49,7 +49,8 @@ static void *diag0c_store(unsigned int *count) get_online_cpus(); cpu_count = num_online_cpus(); - cpu_vec = kmalloc(sizeof(*cpu_vec) * num_possible_cpus(), GFP_KERNEL); + cpu_vec = kmalloc_array(num_possible_cpus(), sizeof(*cpu_vec), + GFP_KERNEL); if (!cpu_vec) goto fail_put_online_cpus; /* Note: Diag 0c needs 8 byte alignment and real storage */ diff --git a/arch/s390/hypfs/hypfs_sprp.c b/arch/s390/hypfs/hypfs_sprp.c index ae0ed8dd5f1b..5d85a039391c 100644 --- a/arch/s390/hypfs/hypfs_sprp.c +++ b/arch/s390/hypfs/hypfs_sprp.c @@ -13,7 +13,6 @@ #include #include #include -#include #include #include #include "hypfs.h" diff --git a/arch/s390/include/asm/archrandom.h b/arch/s390/include/asm/archrandom.h index 09aed1095336..c67b82dfa558 100644 --- a/arch/s390/include/asm/archrandom.h +++ b/arch/s390/include/asm/archrandom.h @@ -15,16 +15,11 @@ #include #include -#include DECLARE_STATIC_KEY_FALSE(s390_arch_random_available); extern atomic64_t s390_arch_random_counter; -static void s390_arch_random_generate(u8 *buf, unsigned int nbytes) -{ - cpacf_trng(NULL, 0, buf, nbytes); - atomic64_add(nbytes, &s390_arch_random_counter); -} +bool s390_arch_random_generate(u8 *buf, unsigned int nbytes); static inline bool arch_has_random(void) { @@ -51,8 +46,7 @@ static inline bool arch_get_random_int(unsigned int *v) static inline bool arch_get_random_seed_long(unsigned long *v) { if (static_branch_likely(&s390_arch_random_available)) { - s390_arch_random_generate((u8 *)v, sizeof(*v)); - return true; + return s390_arch_random_generate((u8 *)v, sizeof(*v)); } return false; } @@ -60,8 +54,7 @@ static inline bool arch_get_random_seed_long(unsigned long *v) static inline bool arch_get_random_seed_int(unsigned int *v) { if (static_branch_likely(&s390_arch_random_available)) { - s390_arch_random_generate((u8 *)v, sizeof(*v)); - return true; + return s390_arch_random_generate((u8 *)v, sizeof(*v)); } return false; } diff --git a/arch/s390/include/asm/ccwdev.h b/arch/s390/include/asm/ccwdev.h index 20bce136b2e5..a29dd430fb40 100644 --- a/arch/s390/include/asm/ccwdev.h +++ b/arch/s390/include/asm/ccwdev.h @@ -231,4 +231,5 @@ int ccw_device_siosl(struct ccw_device *); extern void ccw_device_get_schid(struct ccw_device *, struct subchannel_id *); struct channel_path_desc_fmt0 *ccw_device_get_chp_desc(struct ccw_device *, int); +u8 *ccw_device_get_util_str(struct ccw_device *cdev, int chp_idx); #endif /* _S390_CCWDEV_H_ */ diff --git a/arch/s390/include/asm/ccwgroup.h b/arch/s390/include/asm/ccwgroup.h index 99aa817dad32..860cab7479c3 100644 --- a/arch/s390/include/asm/ccwgroup.h +++ b/arch/s390/include/asm/ccwgroup.h @@ -73,4 +73,14 @@ extern void ccwgroup_remove_ccwdev(struct ccw_device *cdev); #define to_ccwgroupdev(x) container_of((x), struct ccwgroup_device, dev) #define to_ccwgroupdrv(x) container_of((x), struct ccwgroup_driver, driver) + +#if IS_ENABLED(CONFIG_CCWGROUP) +bool dev_is_ccwgroup(struct device *dev); +#else /* CONFIG_CCWGROUP */ +static inline bool dev_is_ccwgroup(struct device *dev) +{ + return false; +} +#endif /* CONFIG_CCWGROUP */ + #endif diff --git a/arch/s390/include/asm/compat.h b/arch/s390/include/asm/compat.h index 9830fb6b076e..97db2fba546a 100644 --- a/arch/s390/include/asm/compat.h +++ b/arch/s390/include/asm/compat.h @@ -53,7 +53,6 @@ typedef u32 compat_size_t; typedef s32 compat_ssize_t; -typedef s32 compat_time_t; typedef s32 compat_clock_t; typedef s32 compat_pid_t; typedef u16 __compat_uid_t; @@ -97,16 +96,6 @@ typedef struct { u32 gprs_high[NUM_GPRS]; } s390_compat_regs_high; -struct compat_timespec { - compat_time_t tv_sec; - s32 tv_nsec; -}; - -struct compat_timeval { - compat_time_t tv_sec; - s32 tv_usec; -}; - struct compat_stat { compat_dev_t st_dev; u16 __pad1; @@ -243,10 +232,10 @@ struct compat_ipc64_perm { struct compat_semid64_ds { struct compat_ipc64_perm sem_perm; - compat_time_t sem_otime; - compat_ulong_t __pad1; - compat_time_t sem_ctime; - compat_ulong_t __pad2; + compat_ulong_t sem_otime; + compat_ulong_t sem_otime_high; + compat_ulong_t sem_ctime; + compat_ulong_t sem_ctime_high; compat_ulong_t sem_nsems; compat_ulong_t __unused1; compat_ulong_t __unused2; @@ -254,12 +243,12 @@ struct compat_semid64_ds { struct compat_msqid64_ds { struct compat_ipc64_perm msg_perm; - compat_time_t msg_stime; - compat_ulong_t __pad1; - compat_time_t msg_rtime; - compat_ulong_t __pad2; - compat_time_t msg_ctime; - compat_ulong_t __pad3; + compat_ulong_t msg_stime; + compat_ulong_t msg_stime_high; + compat_ulong_t msg_rtime; + compat_ulong_t msg_rtime_high; + compat_ulong_t msg_ctime; + compat_ulong_t msg_ctime_high; compat_ulong_t msg_cbytes; compat_ulong_t msg_qnum; compat_ulong_t msg_qbytes; @@ -272,12 +261,12 @@ struct compat_msqid64_ds { struct compat_shmid64_ds { struct compat_ipc64_perm shm_perm; compat_size_t shm_segsz; - compat_time_t shm_atime; - compat_ulong_t __pad1; - compat_time_t shm_dtime; - compat_ulong_t __pad2; - compat_time_t shm_ctime; - compat_ulong_t __pad3; + compat_ulong_t shm_atime; + compat_ulong_t shm_atime_high; + compat_ulong_t shm_dtime; + compat_ulong_t shm_dtime_high; + compat_ulong_t shm_ctime; + compat_ulong_t shm_ctime_high; compat_pid_t shm_cpid; compat_pid_t shm_lpid; compat_ulong_t shm_nattch; diff --git a/arch/s390/include/asm/cpu_mf.h b/arch/s390/include/asm/cpu_mf.h index f58d17e9dd65..de023a9a88ca 100644 --- a/arch/s390/include/asm/cpu_mf.h +++ b/arch/s390/include/asm/cpu_mf.h @@ -113,7 +113,7 @@ struct hws_basic_entry { struct hws_diag_entry { unsigned int def:16; /* 0-15 Data Entry Format */ - unsigned int R:14; /* 16-19 and 20-30 reserved */ + unsigned int R:15; /* 16-19 and 20-30 reserved */ unsigned int I:1; /* 31 entry valid or invalid */ u8 data[]; /* Machine-dependent sample data */ } __packed; @@ -129,7 +129,9 @@ struct hws_trailer_entry { unsigned int f:1; /* 0 - Block Full Indicator */ unsigned int a:1; /* 1 - Alert request control */ unsigned int t:1; /* 2 - Timestamp format */ - unsigned long long:61; /* 3 - 63: Reserved */ + unsigned int :29; /* 3 - 31: Reserved */ + unsigned int bsdes:16; /* 32-47: size of basic SDE */ + unsigned int dsdes:16; /* 48-63: size of diagnostic SDE */ }; unsigned long long flags; /* 0 - 63: All indicators */ }; diff --git a/arch/s390/include/asm/css_chars.h b/arch/s390/include/asm/css_chars.h index 0563fd3e8458..480bb02ccacd 100644 --- a/arch/s390/include/asm/css_chars.h +++ b/arch/s390/include/asm/css_chars.h @@ -6,36 +6,38 @@ struct css_general_char { u64 : 12; - u32 dynio : 1; /* bit 12 */ - u32 : 4; - u32 eadm : 1; /* bit 17 */ - u32 : 23; - u32 aif : 1; /* bit 41 */ - u32 : 3; - u32 mcss : 1; /* bit 45 */ - u32 fcs : 1; /* bit 46 */ - u32 : 1; - u32 ext_mb : 1; /* bit 48 */ - u32 : 7; - u32 aif_tdd : 1; /* bit 56 */ - u32 : 1; - u32 qebsm : 1; /* bit 58 */ - u32 : 2; - u32 aiv : 1; /* bit 61 */ - u32 : 5; - u32 aif_osa : 1; /* bit 67 */ - u32 : 12; - u32 eadm_rf : 1; /* bit 80 */ - u32 : 1; - u32 cib : 1; /* bit 82 */ - u32 : 5; - u32 fcx : 1; /* bit 88 */ - u32 : 19; - u32 alt_ssi : 1; /* bit 108 */ - u32 : 1; - u32 narf : 1; /* bit 110 */ - u32 : 12; - u32 util_str : 1;/* bit 123 */ + u64 dynio : 1; /* bit 12 */ + u64 : 4; + u64 eadm : 1; /* bit 17 */ + u64 : 23; + u64 aif : 1; /* bit 41 */ + u64 : 3; + u64 mcss : 1; /* bit 45 */ + u64 fcs : 1; /* bit 46 */ + u64 : 1; + u64 ext_mb : 1; /* bit 48 */ + u64 : 7; + u64 aif_tdd : 1; /* bit 56 */ + u64 : 1; + u64 qebsm : 1; /* bit 58 */ + u64 : 2; + u64 aiv : 1; /* bit 61 */ + u64 : 2; + + u64 : 3; + u64 aif_osa : 1; /* bit 67 */ + u64 : 12; + u64 eadm_rf : 1; /* bit 80 */ + u64 : 1; + u64 cib : 1; /* bit 82 */ + u64 : 5; + u64 fcx : 1; /* bit 88 */ + u64 : 19; + u64 alt_ssi : 1; /* bit 108 */ + u64 : 1; + u64 narf : 1; /* bit 110 */ + u64 : 12; + u64 util_str : 1;/* bit 123 */ } __packed; extern struct css_general_char css_general_characteristics; diff --git a/arch/s390/include/asm/ctl_reg.h b/arch/s390/include/asm/ctl_reg.h index 99c93d0346f9..4600453536c2 100644 --- a/arch/s390/include/asm/ctl_reg.h +++ b/arch/s390/include/asm/ctl_reg.h @@ -10,8 +10,20 @@ #include +#define CR0_CLOCK_COMPARATOR_SIGN _BITUL(63 - 10) +#define CR0_EMERGENCY_SIGNAL_SUBMASK _BITUL(63 - 49) +#define CR0_EXTERNAL_CALL_SUBMASK _BITUL(63 - 50) +#define CR0_CLOCK_COMPARATOR_SUBMASK _BITUL(63 - 52) +#define CR0_CPU_TIMER_SUBMASK _BITUL(63 - 53) +#define CR0_SERVICE_SIGNAL_SUBMASK _BITUL(63 - 54) +#define CR0_UNUSED_56 _BITUL(63 - 56) +#define CR0_INTERRUPT_KEY_SUBMASK _BITUL(63 - 57) +#define CR0_MEASUREMENT_ALERT_SUBMASK _BITUL(63 - 58) + #define CR2_GUARDED_STORAGE _BITUL(63 - 59) +#define CR14_UNUSED_32 _BITUL(63 - 32) +#define CR14_UNUSED_33 _BITUL(63 - 33) #define CR14_CHANNEL_REPORT_SUBMASK _BITUL(63 - 35) #define CR14_RECOVERY_SUBMASK _BITUL(63 - 36) #define CR14_DEGRADATION_SUBMASK _BITUL(63 - 37) diff --git a/arch/s390/include/asm/elf.h b/arch/s390/include/asm/elf.h index 1a61b1b997f2..7d22a474a040 100644 --- a/arch/s390/include/asm/elf.h +++ b/arch/s390/include/asm/elf.h @@ -125,8 +125,9 @@ * ELF register definitions.. */ +#include + #include -#include #include #include @@ -136,7 +137,6 @@ typedef s390_regs elf_gregset_t; typedef s390_fp_regs compat_elf_fpregset_t; typedef s390_compat_regs compat_elf_gregset_t; -#include #include /* for task_struct */ #include diff --git a/arch/s390/include/asm/hardirq.h b/arch/s390/include/asm/hardirq.h index a296c6acfd07..dfbc3c6c0674 100644 --- a/arch/s390/include/asm/hardirq.h +++ b/arch/s390/include/asm/hardirq.h @@ -14,6 +14,8 @@ #include #define local_softirq_pending() (S390_lowcore.softirq_pending) +#define set_softirq_pending(x) (S390_lowcore.softirq_pending = (x)) +#define or_softirq_pending(x) (S390_lowcore.softirq_pending |= (x)) #define __ARCH_IRQ_STAT #define __ARCH_HAS_DO_SOFTIRQ diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h index 81cdb6b55118..a2188e309bd6 100644 --- a/arch/s390/include/asm/kvm_host.h +++ b/arch/s390/include/asm/kvm_host.h @@ -812,6 +812,7 @@ struct kvm_arch{ int use_irqchip; int use_cmma; int use_pfmfi; + int use_skf; int user_cpu_state_ctrl; int user_sigp; int user_stsi; diff --git a/arch/s390/include/asm/mmu.h b/arch/s390/include/asm/mmu.h index c639c95850e4..f5ff9dbad8ac 100644 --- a/arch/s390/include/asm/mmu.h +++ b/arch/s390/include/asm/mmu.h @@ -21,7 +21,7 @@ typedef struct { /* The mmu context uses extended page tables. */ unsigned int has_pgste:1; /* The mmu context uses storage keys. */ - unsigned int use_skey:1; + unsigned int uses_skeys:1; /* The mmu context uses CMM. */ unsigned int uses_cmm:1; } mm_context_t; diff --git a/arch/s390/include/asm/mmu_context.h b/arch/s390/include/asm/mmu_context.h index 324f6f452982..d16bc79c30bb 100644 --- a/arch/s390/include/asm/mmu_context.h +++ b/arch/s390/include/asm/mmu_context.h @@ -30,7 +30,7 @@ static inline int init_new_context(struct task_struct *tsk, test_thread_flag(TIF_PGSTE) || (current->mm && current->mm->context.alloc_pgste); mm->context.has_pgste = 0; - mm->context.use_skey = 0; + mm->context.uses_skeys = 0; mm->context.uses_cmm = 0; #endif switch (mm->context.asce_limit) { diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h index 12fe3591034f..94f8db468c9b 100644 --- a/arch/s390/include/asm/pci.h +++ b/arch/s390/include/asm/pci.h @@ -2,8 +2,6 @@ #ifndef __ASM_S390_PCI_H #define __ASM_S390_PCI_H -/* must be set before including asm-generic/pci.h */ -#define PCI_DMA_BUS_IS_PHYS (0) /* must be set before including pci_clp.h */ #define PCI_BAR_COUNT 6 diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 2d24d33bf188..5ab636089c60 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -171,7 +171,6 @@ static inline int is_module_addr(void *addr) #define _PAGE_WRITE 0x020 /* SW pte write bit */ #define _PAGE_SPECIAL 0x040 /* SW associated with special page */ #define _PAGE_UNUSED 0x080 /* SW bit for pgste usage state */ -#define __HAVE_ARCH_PTE_SPECIAL #ifdef CONFIG_MEM_SOFT_DIRTY #define _PAGE_SOFT_DIRTY 0x002 /* SW pte soft dirty bit */ @@ -507,10 +506,10 @@ static inline int mm_alloc_pgste(struct mm_struct *mm) * faults should no longer be backed by zero pages */ #define mm_forbids_zeropage mm_has_pgste -static inline int mm_use_skey(struct mm_struct *mm) +static inline int mm_uses_skeys(struct mm_struct *mm) { #ifdef CONFIG_PGSTE - if (mm->context.use_skey) + if (mm->context.uses_skeys) return 1; #endif return 0; diff --git a/arch/s390/include/asm/pnet.h b/arch/s390/include/asm/pnet.h new file mode 100644 index 000000000000..6e278584f8f1 --- /dev/null +++ b/arch/s390/include/asm/pnet.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * IBM System z PNET ID Support + * + * Copyright IBM Corp. 2018 + */ + +#ifndef _ASM_S390_PNET_H +#define _ASM_S390_PNET_H + +#include +#include + +#define PNETIDS_LEN 64 /* Total utility string length in bytes + * to cover up to 4 PNETIDs of 16 bytes + * for up to 4 device ports + */ +#define MAX_PNETID_LEN 16 /* Max.length of a single port PNETID */ +#define MAX_PNETID_PORTS (PNETIDS_LEN / MAX_PNETID_LEN) + /* Max. # of ports with a PNETID */ + +int pnet_id_by_dev_port(struct device *dev, unsigned short port, u8 *pnetid); +#endif /* _ASM_S390_PNET_H */ diff --git a/arch/s390/include/uapi/asm/Kbuild b/arch/s390/include/uapi/asm/Kbuild index faef3f7e8353..e364873e0d10 100644 --- a/arch/s390/include/uapi/asm/Kbuild +++ b/arch/s390/include/uapi/asm/Kbuild @@ -9,9 +9,12 @@ generic-y += errno.h generic-y += fcntl.h generic-y += ioctl.h generic-y += mman.h +generic-y += msgbuf.h generic-y += param.h generic-y += poll.h generic-y += resource.h +generic-y += sembuf.h +generic-y += shmbuf.h generic-y += sockios.h generic-y += swab.h generic-y += termbits.h diff --git a/arch/s390/include/uapi/asm/msgbuf.h b/arch/s390/include/uapi/asm/msgbuf.h deleted file mode 100644 index 604f847cd68c..000000000000 --- a/arch/s390/include/uapi/asm/msgbuf.h +++ /dev/null @@ -1,38 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _S390_MSGBUF_H -#define _S390_MSGBUF_H - -/* - * The msqid64_ds structure for S/390 architecture. - * Note extra padding because this structure is passed back and forth - * between kernel and user space. - * - * Pad space is left for: - * - 64-bit time_t to solve y2038 problem - * - 2 miscellaneous 32-bit values - */ - -struct msqid64_ds { - struct ipc64_perm msg_perm; - __kernel_time_t msg_stime; /* last msgsnd time */ -#ifndef __s390x__ - unsigned long __unused1; -#endif /* ! __s390x__ */ - __kernel_time_t msg_rtime; /* last msgrcv time */ -#ifndef __s390x__ - unsigned long __unused2; -#endif /* ! __s390x__ */ - __kernel_time_t msg_ctime; /* last change time */ -#ifndef __s390x__ - unsigned long __unused3; -#endif /* ! __s390x__ */ - unsigned long msg_cbytes; /* current number of bytes on queue */ - unsigned long msg_qnum; /* number of messages in queue */ - unsigned long msg_qbytes; /* max number of bytes on queue */ - __kernel_pid_t msg_lspid; /* pid of last msgsnd */ - __kernel_pid_t msg_lrpid; /* last receive pid */ - unsigned long __unused4; - unsigned long __unused5; -}; - -#endif /* _S390_MSGBUF_H */ diff --git a/arch/s390/include/uapi/asm/sembuf.h b/arch/s390/include/uapi/asm/sembuf.h deleted file mode 100644 index 3e917697b668..000000000000 --- a/arch/s390/include/uapi/asm/sembuf.h +++ /dev/null @@ -1,30 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _S390_SEMBUF_H -#define _S390_SEMBUF_H - -/* - * The semid64_ds structure for S/390 architecture. - * Note extra padding because this structure is passed back and forth - * between kernel and user space. - * - * Pad space is left for: - * - 64-bit time_t to solve y2038 problem (for !__s390x__) - * - 2 miscellaneous 32-bit values - */ - -struct semid64_ds { - struct ipc64_perm sem_perm; /* permissions .. see ipc.h */ - __kernel_time_t sem_otime; /* last semop time */ -#ifndef __s390x__ - unsigned long __unused1; -#endif /* ! __s390x__ */ - __kernel_time_t sem_ctime; /* last change time */ -#ifndef __s390x__ - unsigned long __unused2; -#endif /* ! __s390x__ */ - unsigned long sem_nsems; /* no. of semaphores in array */ - unsigned long __unused3; - unsigned long __unused4; -}; - -#endif /* _S390_SEMBUF_H */ diff --git a/arch/s390/include/uapi/asm/shmbuf.h b/arch/s390/include/uapi/asm/shmbuf.h deleted file mode 100644 index 9cdce8d7ce60..000000000000 --- a/arch/s390/include/uapi/asm/shmbuf.h +++ /dev/null @@ -1,49 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _S390_SHMBUF_H -#define _S390_SHMBUF_H - -/* - * The shmid64_ds structure for S/390 architecture. - * Note extra padding because this structure is passed back and forth - * between kernel and user space. - * - * Pad space is left for: - * - 64-bit time_t to solve y2038 problem (for !__s390x__) - * - 2 miscellaneous 32-bit values - */ - -struct shmid64_ds { - struct ipc64_perm shm_perm; /* operation perms */ - size_t shm_segsz; /* size of segment (bytes) */ - __kernel_time_t shm_atime; /* last attach time */ -#ifndef __s390x__ - unsigned long __unused1; -#endif /* ! __s390x__ */ - __kernel_time_t shm_dtime; /* last detach time */ -#ifndef __s390x__ - unsigned long __unused2; -#endif /* ! __s390x__ */ - __kernel_time_t shm_ctime; /* last change time */ -#ifndef __s390x__ - unsigned long __unused3; -#endif /* ! __s390x__ */ - __kernel_pid_t shm_cpid; /* pid of creator */ - __kernel_pid_t shm_lpid; /* pid of last operator */ - unsigned long shm_nattch; /* no. of current attaches */ - unsigned long __unused4; - unsigned long __unused5; -}; - -struct shminfo64 { - unsigned long shmmax; - unsigned long shmmin; - unsigned long shmmni; - unsigned long shmseg; - unsigned long shmall; - unsigned long __unused1; - unsigned long __unused2; - unsigned long __unused3; - unsigned long __unused4; -}; - -#endif /* _S390_SHMBUF_H */ diff --git a/arch/s390/kernel/Makefile b/arch/s390/kernel/Makefile index f92dd8ed3884..2fed39b26b42 100644 --- a/arch/s390/kernel/Makefile +++ b/arch/s390/kernel/Makefile @@ -6,22 +6,26 @@ ifdef CONFIG_FUNCTION_TRACER # Do not trace tracer code -CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE) # Do not trace early setup code -CFLAGS_REMOVE_als.o = $(CC_FLAGS_FTRACE) -CFLAGS_REMOVE_early.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_als.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_early.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_early_nobss.o = $(CC_FLAGS_FTRACE) endif -GCOV_PROFILE_als.o := n -GCOV_PROFILE_early.o := n +GCOV_PROFILE_als.o := n +GCOV_PROFILE_early.o := n +GCOV_PROFILE_early_nobss.o := n -KCOV_INSTRUMENT_als.o := n -KCOV_INSTRUMENT_early.o := n +KCOV_INSTRUMENT_als.o := n +KCOV_INSTRUMENT_early.o := n +KCOV_INSTRUMENT_early_nobss.o := n -UBSAN_SANITIZE_als.o := n -UBSAN_SANITIZE_early.o := n +UBSAN_SANITIZE_als.o := n +UBSAN_SANITIZE_early.o := n +UBSAN_SANITIZE_early_nobss.o := n # # Use -march=z900 for als.c to be able to print an error @@ -57,7 +61,7 @@ CFLAGS_ptrace.o += -DUTS_MACHINE='"$(UTS_MACHINE)"' obj-y := traps.o time.o process.o base.o early.o setup.o idle.o vtime.o obj-y += processor.o sys_s390.o ptrace.o signal.o cpcmd.o ebcdic.o nmi.o -obj-y += debug.o irq.o ipl.o dis.o diag.o vdso.o als.o +obj-y += debug.o irq.o ipl.o dis.o diag.o vdso.o als.o early_nobss.o obj-y += sysinfo.o jump_label.o lgr.o os_info.o machine_kexec.o pgm_check.o obj-y += runtime_instr.o cache.o fpu.o dumpstack.o guarded_storage.o sthyi.o obj-y += entry.o reipl.o relocate_kernel.o kdebugfs.o alternative.o @@ -94,3 +98,6 @@ obj-$(CONFIG_TRACEPOINTS) += trace.o # vdso obj-y += vdso64/ obj-$(CONFIG_COMPAT) += vdso32/ + +chkbss := head.o head64.o als.o early_nobss.o +include $(srctree)/arch/s390/scripts/Makefile.chkbss diff --git a/arch/s390/kernel/debug.c b/arch/s390/kernel/debug.c index 80e974adb9e8..d374f9b218b4 100644 --- a/arch/s390/kernel/debug.c +++ b/arch/s390/kernel/debug.c @@ -194,11 +194,13 @@ static debug_entry_t ***debug_areas_alloc(int pages_per_area, int nr_areas) debug_entry_t ***areas; int i, j; - areas = kmalloc(nr_areas * sizeof(debug_entry_t **), GFP_KERNEL); + areas = kmalloc_array(nr_areas, sizeof(debug_entry_t **), GFP_KERNEL); if (!areas) goto fail_malloc_areas; for (i = 0; i < nr_areas; i++) { - areas[i] = kmalloc(pages_per_area * sizeof(debug_entry_t *), GFP_KERNEL); + areas[i] = kmalloc_array(pages_per_area, + sizeof(debug_entry_t *), + GFP_KERNEL); if (!areas[i]) goto fail_malloc_areas2; for (j = 0; j < pages_per_area; j++) { diff --git a/arch/s390/kernel/early.c b/arch/s390/kernel/early.c index 32daa0f84325..827699eb48fa 100644 --- a/arch/s390/kernel/early.c +++ b/arch/s390/kernel/early.c @@ -33,32 +33,6 @@ static void __init setup_boot_command_line(void); -/* - * Get the TOD clock running. - */ -static void __init reset_tod_clock(void) -{ - u64 time; - - if (store_tod_clock(&time) == 0) - return; - /* TOD clock not running. Set the clock to Unix Epoch. */ - if (set_tod_clock(TOD_UNIX_EPOCH) != 0 || store_tod_clock(&time) != 0) - disabled_wait(0); - - memset(tod_clock_base, 0, 16); - *(__u64 *) &tod_clock_base[1] = TOD_UNIX_EPOCH; - S390_lowcore.last_update_clock = TOD_UNIX_EPOCH; -} - -/* - * Clear bss memory - */ -static noinline __init void clear_bss_section(void) -{ - memset(__bss_start, 0, __bss_stop - __bss_start); -} - /* * Initialize storage key for kernel pages */ @@ -310,57 +284,6 @@ static int __init cad_setup(char *str) } early_param("cad", cad_setup); -static __init void memmove_early(void *dst, const void *src, size_t n) -{ - unsigned long addr; - long incr; - psw_t old; - - if (!n) - return; - incr = 1; - if (dst > src) { - incr = -incr; - dst += n - 1; - src += n - 1; - } - old = S390_lowcore.program_new_psw; - S390_lowcore.program_new_psw.mask = __extract_psw(); - asm volatile( - " larl %[addr],1f\n" - " stg %[addr],%[psw_pgm_addr]\n" - "0: mvc 0(1,%[dst]),0(%[src])\n" - " agr %[dst],%[incr]\n" - " agr %[src],%[incr]\n" - " brctg %[n],0b\n" - "1:\n" - : [addr] "=&d" (addr), - [psw_pgm_addr] "=Q" (S390_lowcore.program_new_psw.addr), - [dst] "+&a" (dst), [src] "+&a" (src), [n] "+d" (n) - : [incr] "d" (incr) - : "cc", "memory"); - S390_lowcore.program_new_psw = old; -} - -static __init noinline void rescue_initrd(void) -{ -#ifdef CONFIG_BLK_DEV_INITRD - unsigned long min_initrd_addr = (unsigned long) _end + (4UL << 20); - /* - * Just like in case of IPL from VM reader we make sure there is a - * gap of 4MB between end of kernel and start of initrd. - * That way we can also be sure that saving an NSS will succeed, - * which however only requires different segments. - */ - if (!INITRD_START || !INITRD_SIZE) - return; - if (INITRD_START >= min_initrd_addr) - return; - memmove_early((void *) min_initrd_addr, (void *) INITRD_START, INITRD_SIZE); - INITRD_START = min_initrd_addr; -#endif -} - /* Set up boot command line */ static void __init append_to_cmdline(size_t (*ipl_data)(char *, size_t)) { @@ -410,9 +333,6 @@ static void __init setup_boot_command_line(void) void __init startup_init(void) { - reset_tod_clock(); - rescue_initrd(); - clear_bss_section(); time_early_init(); init_kernel_storage_key(); lockdep_off(); diff --git a/arch/s390/kernel/early_nobss.c b/arch/s390/kernel/early_nobss.c new file mode 100644 index 000000000000..2d84fc48df3a --- /dev/null +++ b/arch/s390/kernel/early_nobss.c @@ -0,0 +1,65 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright IBM Corp. 2007, 2018 + */ + +/* + * Early setup functions which may not rely on an initialized bss + * section. The last thing that is supposed to happen here is + * initialization of the bss section. + */ + +#include +#include +#include +#include +#include +#include +#include "entry.h" + +static void __init reset_tod_clock(void) +{ + u64 time; + + if (store_tod_clock(&time) == 0) + return; + /* TOD clock not running. Set the clock to Unix Epoch. */ + if (set_tod_clock(TOD_UNIX_EPOCH) != 0 || store_tod_clock(&time) != 0) + disabled_wait(0); + + memset(tod_clock_base, 0, 16); + *(__u64 *) &tod_clock_base[1] = TOD_UNIX_EPOCH; + S390_lowcore.last_update_clock = TOD_UNIX_EPOCH; +} + +static void __init rescue_initrd(void) +{ + unsigned long min_initrd_addr = (unsigned long) _end + (4UL << 20); + + /* + * Just like in case of IPL from VM reader we make sure there is a + * gap of 4MB between end of kernel and start of initrd. + * That way we can also be sure that saving an NSS will succeed, + * which however only requires different segments. + */ + if (!IS_ENABLED(CONFIG_BLK_DEV_INITRD)) + return; + if (!INITRD_START || !INITRD_SIZE) + return; + if (INITRD_START >= min_initrd_addr) + return; + memmove((void *) min_initrd_addr, (void *) INITRD_START, INITRD_SIZE); + INITRD_START = min_initrd_addr; +} + +static void __init clear_bss_section(void) +{ + memset(__bss_start, 0, __bss_stop - __bss_start); +} + +void __init startup_init_nobss(void) +{ + reset_tod_clock(); + rescue_initrd(); + clear_bss_section(); +} diff --git a/arch/s390/kernel/entry.h b/arch/s390/kernel/entry.h index e87758f8fbdc..961abfac2c5f 100644 --- a/arch/s390/kernel/entry.h +++ b/arch/s390/kernel/entry.h @@ -58,6 +58,7 @@ void do_notify_resume(struct pt_regs *regs); void __init init_IRQ(void); void do_IRQ(struct pt_regs *regs, int irq); void do_restart(void); +void __init startup_init_nobss(void); void __init startup_init(void); void die(struct pt_regs *regs, const char *str); int setup_profiling_timer(unsigned int multiplier); diff --git a/arch/s390/kernel/head64.S b/arch/s390/kernel/head64.S index 38a973ccf501..791cb9000e86 100644 --- a/arch/s390/kernel/head64.S +++ b/arch/s390/kernel/head64.S @@ -40,8 +40,12 @@ ENTRY(startup_continue) stg %r15,__LC_KERNEL_STACK # set end of kernel stack aghi %r15,-160 # -# Save ipl parameters, clear bss memory, initialize storage key for kernel pages, -# and create a kernel NSS if the SAVESYS= parm is defined +# Early setup functions that may not rely on an initialized bss section, +# like moving the initrd. Returns with an initialized bss section. +# + brasl %r14,startup_init_nobss +# +# Early machine initialization and detection functions. # brasl %r14,startup_init lpswe .Lentry-.LPG1(13) # jump to _stext in primary-space, diff --git a/arch/s390/kernel/module.c b/arch/s390/kernel/module.c index 0dc8ac8548ee..d298d3cb46d0 100644 --- a/arch/s390/kernel/module.c +++ b/arch/s390/kernel/module.c @@ -123,8 +123,8 @@ int module_frob_arch_sections(Elf_Ehdr *hdr, Elf_Shdr *sechdrs, /* Allocate one syminfo structure per symbol. */ me->arch.nsyms = symtab->sh_size / sizeof(Elf_Sym); - me->arch.syminfo = vmalloc(me->arch.nsyms * - sizeof(struct mod_arch_syminfo)); + me->arch.syminfo = vmalloc(array_size(sizeof(struct mod_arch_syminfo), + me->arch.nsyms)); if (!me->arch.syminfo) return -ENOMEM; symbols = (void *) hdr + symtab->sh_offset; diff --git a/arch/s390/kernel/nospec-branch.c b/arch/s390/kernel/nospec-branch.c index 8ad6a7128b3a..18ae7b9c71d6 100644 --- a/arch/s390/kernel/nospec-branch.c +++ b/arch/s390/kernel/nospec-branch.c @@ -36,9 +36,9 @@ early_param("nospec", nospec_setup_early); static int __init nospec_report(void) { if (IS_ENABLED(CC_USING_EXPOLINE) && !nospec_disable) - pr_info("Spectre V2 mitigation: execute trampolines.\n"); + pr_info("Spectre V2 mitigation: execute trampolines\n"); if (__test_facility(82, S390_lowcore.alt_stfle_fac_list)) - pr_info("Spectre V2 mitigation: limited branch prediction.\n"); + pr_info("Spectre V2 mitigation: limited branch prediction\n"); return 0; } arch_initcall(nospec_report); diff --git a/arch/s390/kernel/perf_cpum_cf_events.c b/arch/s390/kernel/perf_cpum_cf_events.c index feebb2944882..d63fb3c56b8a 100644 --- a/arch/s390/kernel/perf_cpum_cf_events.c +++ b/arch/s390/kernel/perf_cpum_cf_events.c @@ -527,7 +527,7 @@ static __init struct attribute **merge_attr(struct attribute **a, j++; j++; - new = kmalloc(sizeof(struct attribute *) * j, GFP_KERNEL); + new = kmalloc_array(j, sizeof(struct attribute *), GFP_KERNEL); if (!new) return NULL; j = 0; diff --git a/arch/s390/kernel/sthyi.c b/arch/s390/kernel/sthyi.c index 80b862e9c53c..0859cde36f75 100644 --- a/arch/s390/kernel/sthyi.c +++ b/arch/s390/kernel/sthyi.c @@ -315,7 +315,7 @@ static void fill_diag(struct sthyi_sctns *sctns) if (pages <= 0) return; - diag204_buf = vmalloc(PAGE_SIZE * pages); + diag204_buf = vmalloc(array_size(pages, PAGE_SIZE)); if (!diag204_buf) return; diff --git a/arch/s390/kernel/sysinfo.c b/arch/s390/kernel/sysinfo.c index fc7e04c2195b..54f5496913fa 100644 --- a/arch/s390/kernel/sysinfo.c +++ b/arch/s390/kernel/sysinfo.c @@ -294,21 +294,9 @@ static int sysinfo_show(struct seq_file *m, void *v) return 0; } -static int sysinfo_open(struct inode *inode, struct file *file) -{ - return single_open(file, sysinfo_show, NULL); -} - -static const struct file_operations sysinfo_fops = { - .open = sysinfo_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int __init sysinfo_create_proc(void) { - proc_create("sysinfo", 0444, NULL, &sysinfo_fops); + proc_create_single("sysinfo", 0444, NULL, sysinfo_show); return 0; } device_initcall(sysinfo_create_proc); @@ -386,18 +374,6 @@ static const struct seq_operations service_level_seq_ops = { .show = service_level_show }; -static int service_level_open(struct inode *inode, struct file *file) -{ - return seq_open(file, &service_level_seq_ops); -} - -static const struct file_operations service_level_ops = { - .open = service_level_open, - .read = seq_read, - .llseek = seq_lseek, - .release = seq_release -}; - static void service_level_vm_print(struct seq_file *m, struct service_level *slr) { @@ -420,7 +396,7 @@ static struct service_level service_level_vm = { static __init int create_proc_service_level(void) { - proc_create("service_levels", 0, NULL, &service_level_ops); + proc_create_seq("service_levels", 0, NULL, &service_level_seq_ops); if (MACHINE_IS_VM) register_service_level(&service_level_vm); return 0; diff --git a/arch/s390/kernel/traps.c b/arch/s390/kernel/traps.c index a5297a22bc1e..8003b38c1688 100644 --- a/arch/s390/kernel/traps.c +++ b/arch/s390/kernel/traps.c @@ -44,14 +44,8 @@ int is_valid_bugaddr(unsigned long addr) void do_report_trap(struct pt_regs *regs, int si_signo, int si_code, char *str) { - siginfo_t info; - if (user_mode(regs)) { - info.si_signo = si_signo; - info.si_errno = 0; - info.si_code = si_code; - info.si_addr = get_trap_ip(regs); - force_sig_info(si_signo, &info, current); + force_sig_fault(si_signo, si_code, get_trap_ip(regs), current); report_user_fault(regs, si_signo, 0); } else { const struct exception_table_entry *fixup; @@ -80,18 +74,12 @@ NOKPROBE_SYMBOL(do_trap); void do_per_trap(struct pt_regs *regs) { - siginfo_t info; - if (notify_die(DIE_SSTEP, "sstep", regs, 0, 0, SIGTRAP) == NOTIFY_STOP) return; if (!current->ptrace) return; - info.si_signo = SIGTRAP; - info.si_errno = 0; - info.si_code = TRAP_HWBKPT; - info.si_addr = - (void __force __user *) current->thread.per_event.address; - force_sig_info(SIGTRAP, &info, current); + force_sig_fault(SIGTRAP, TRAP_HWBKPT, + (void __force __user *) current->thread.per_event.address, current); } NOKPROBE_SYMBOL(do_per_trap); @@ -165,7 +153,6 @@ void translation_exception(struct pt_regs *regs) void illegal_op(struct pt_regs *regs) { - siginfo_t info; __u8 opcode[6]; __u16 __user *location; int is_uprobe_insn = 0; @@ -177,13 +164,9 @@ void illegal_op(struct pt_regs *regs) if (get_user(*((__u16 *) opcode), (__u16 __user *) location)) return; if (*((__u16 *) opcode) == S390_BREAKPOINT_U16) { - if (current->ptrace) { - info.si_signo = SIGTRAP; - info.si_errno = 0; - info.si_code = TRAP_BRKPT; - info.si_addr = location; - force_sig_info(SIGTRAP, &info, current); - } else + if (current->ptrace) + force_sig_fault(SIGTRAP, TRAP_BRKPT, location, current); + else signal = SIGILL; #ifdef CONFIG_UPROBES } else if (*((__u16 *) opcode) == UPROBE_SWBP_INSN) { diff --git a/arch/s390/kernel/vdso.c b/arch/s390/kernel/vdso.c index f3a1c7c6824e..09abae40f917 100644 --- a/arch/s390/kernel/vdso.c +++ b/arch/s390/kernel/vdso.c @@ -285,7 +285,7 @@ static int __init vdso_init(void) + PAGE_SIZE - 1) >> PAGE_SHIFT) + 1; /* Make sure pages are in the correct state */ - vdso32_pagelist = kzalloc(sizeof(struct page *) * (vdso32_pages + 1), + vdso32_pagelist = kcalloc(vdso32_pages + 1, sizeof(struct page *), GFP_KERNEL); BUG_ON(vdso32_pagelist == NULL); for (i = 0; i < vdso32_pages - 1; i++) { @@ -303,7 +303,7 @@ static int __init vdso_init(void) + PAGE_SIZE - 1) >> PAGE_SHIFT) + 1; /* Make sure pages are in the correct state */ - vdso64_pagelist = kzalloc(sizeof(struct page *) * (vdso64_pages + 1), + vdso64_pagelist = kcalloc(vdso64_pages + 1, sizeof(struct page *), GFP_KERNEL); BUG_ON(vdso64_pagelist == NULL); for (i = 0; i < vdso64_pages - 1; i++) { diff --git a/arch/s390/kernel/vmlinux.lds.S b/arch/s390/kernel/vmlinux.lds.S index 08d12cfaf091..f0414f52817b 100644 --- a/arch/s390/kernel/vmlinux.lds.S +++ b/arch/s390/kernel/vmlinux.lds.S @@ -82,10 +82,10 @@ SECTIONS . = ALIGN(PAGE_SIZE); .init.text : AT(ADDR(.init.text) - LOAD_OFFSET) { - VMLINUX_SYMBOL(_sinittext) = . ; + _sinittext = .; INIT_TEXT . = ALIGN(PAGE_SIZE); - VMLINUX_SYMBOL(_einittext) = . ; + _einittext = .; } /* diff --git a/arch/s390/kvm/gaccess.c b/arch/s390/kvm/gaccess.c index 8e2b8647ee12..07d30ffcfa41 100644 --- a/arch/s390/kvm/gaccess.c +++ b/arch/s390/kvm/gaccess.c @@ -847,7 +847,7 @@ int access_guest(struct kvm_vcpu *vcpu, unsigned long ga, u8 ar, void *data, nr_pages = (((ga & ~PAGE_MASK) + len - 1) >> PAGE_SHIFT) + 1; pages = pages_array; if (nr_pages > ARRAY_SIZE(pages_array)) - pages = vmalloc(nr_pages * sizeof(unsigned long)); + pages = vmalloc(array_size(nr_pages, sizeof(unsigned long))); if (!pages) return -ENOMEM; need_ipte_lock = psw_bits(*psw).dat && !asce.r; diff --git a/arch/s390/kvm/guestdbg.c b/arch/s390/kvm/guestdbg.c index b5f3e82006d0..394a5f53805b 100644 --- a/arch/s390/kvm/guestdbg.c +++ b/arch/s390/kvm/guestdbg.c @@ -153,7 +153,7 @@ void kvm_s390_patch_guest_per_regs(struct kvm_vcpu *vcpu) if (guestdbg_sstep_enabled(vcpu)) { /* disable timer (clock-comparator) interrupts */ - vcpu->arch.sie_block->gcr[0] &= ~0x800ul; + vcpu->arch.sie_block->gcr[0] &= ~CR0_CLOCK_COMPARATOR_SUBMASK; vcpu->arch.sie_block->gcr[9] |= PER_EVENT_IFETCH; vcpu->arch.sie_block->gcr[10] = 0; vcpu->arch.sie_block->gcr[11] = -1UL; diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c index 37d06e022238..daa09f89ca2d 100644 --- a/arch/s390/kvm/interrupt.c +++ b/arch/s390/kvm/interrupt.c @@ -159,7 +159,7 @@ static int psw_interrupts_disabled(struct kvm_vcpu *vcpu) static int ckc_interrupts_enabled(struct kvm_vcpu *vcpu) { if (psw_extint_disabled(vcpu) || - !(vcpu->arch.sie_block->gcr[0] & 0x800ul)) + !(vcpu->arch.sie_block->gcr[0] & CR0_CLOCK_COMPARATOR_SUBMASK)) return 0; if (guestdbg_enabled(vcpu) && guestdbg_sstep_enabled(vcpu)) /* No timer interrupts when single stepping */ @@ -172,7 +172,7 @@ static int ckc_irq_pending(struct kvm_vcpu *vcpu) const u64 now = kvm_s390_get_tod_clock_fast(vcpu->kvm); const u64 ckc = vcpu->arch.sie_block->ckc; - if (vcpu->arch.sie_block->gcr[0] & 0x0020000000000000ul) { + if (vcpu->arch.sie_block->gcr[0] & CR0_CLOCK_COMPARATOR_SIGN) { if ((s64)ckc >= (s64)now) return 0; } else if (ckc >= now) { @@ -184,7 +184,7 @@ static int ckc_irq_pending(struct kvm_vcpu *vcpu) static int cpu_timer_interrupts_enabled(struct kvm_vcpu *vcpu) { return !psw_extint_disabled(vcpu) && - (vcpu->arch.sie_block->gcr[0] & 0x400ul); + (vcpu->arch.sie_block->gcr[0] & CR0_CPU_TIMER_SUBMASK); } static int cpu_timer_irq_pending(struct kvm_vcpu *vcpu) @@ -285,15 +285,15 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu) active_mask &= ~IRQ_PEND_IO_MASK; else active_mask = disable_iscs(vcpu, active_mask); - if (!(vcpu->arch.sie_block->gcr[0] & 0x2000ul)) + if (!(vcpu->arch.sie_block->gcr[0] & CR0_EXTERNAL_CALL_SUBMASK)) __clear_bit(IRQ_PEND_EXT_EXTERNAL, &active_mask); - if (!(vcpu->arch.sie_block->gcr[0] & 0x4000ul)) + if (!(vcpu->arch.sie_block->gcr[0] & CR0_EMERGENCY_SIGNAL_SUBMASK)) __clear_bit(IRQ_PEND_EXT_EMERGENCY, &active_mask); - if (!(vcpu->arch.sie_block->gcr[0] & 0x800ul)) + if (!(vcpu->arch.sie_block->gcr[0] & CR0_CLOCK_COMPARATOR_SUBMASK)) __clear_bit(IRQ_PEND_EXT_CLOCK_COMP, &active_mask); - if (!(vcpu->arch.sie_block->gcr[0] & 0x400ul)) + if (!(vcpu->arch.sie_block->gcr[0] & CR0_CPU_TIMER_SUBMASK)) __clear_bit(IRQ_PEND_EXT_CPU_TIMER, &active_mask); - if (!(vcpu->arch.sie_block->gcr[0] & 0x200ul)) + if (!(vcpu->arch.sie_block->gcr[0] & CR0_SERVICE_SIGNAL_SUBMASK)) __clear_bit(IRQ_PEND_EXT_SERVICE, &active_mask); if (psw_mchk_disabled(vcpu)) active_mask &= ~IRQ_PEND_MCHK_MASK; @@ -1042,7 +1042,7 @@ int kvm_s390_vcpu_has_irq(struct kvm_vcpu *vcpu, int exclude_stop) /* external call pending and deliverable */ if (kvm_s390_ext_call_pending(vcpu) && !psw_extint_disabled(vcpu) && - (vcpu->arch.sie_block->gcr[0] & 0x2000ul)) + (vcpu->arch.sie_block->gcr[0] & CR0_EXTERNAL_CALL_SUBMASK)) return 1; if (!exclude_stop && kvm_s390_is_stop_irq_pending(vcpu)) @@ -1062,7 +1062,7 @@ static u64 __calculate_sltime(struct kvm_vcpu *vcpu) u64 cputm, sltime = 0; if (ckc_interrupts_enabled(vcpu)) { - if (vcpu->arch.sie_block->gcr[0] & 0x0020000000000000ul) { + if (vcpu->arch.sie_block->gcr[0] & CR0_CLOCK_COMPARATOR_SIGN) { if ((s64)now < (s64)ckc) sltime = tod_to_ns((s64)ckc - (s64)now); } else if (now < ckc) { diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index 64c986243018..3b7a5151b6a5 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -791,11 +791,21 @@ static int kvm_s390_set_mem_control(struct kvm *kvm, struct kvm_device_attr *att static void kvm_s390_vcpu_crypto_setup(struct kvm_vcpu *vcpu); -static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr) +void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm) { struct kvm_vcpu *vcpu; int i; + kvm_s390_vcpu_block_all(kvm); + + kvm_for_each_vcpu(i, vcpu, kvm) + kvm_s390_vcpu_crypto_setup(vcpu); + + kvm_s390_vcpu_unblock_all(kvm); +} + +static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr) +{ if (!test_kvm_facility(kvm, 76)) return -EINVAL; @@ -832,10 +842,7 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr) return -ENXIO; } - kvm_for_each_vcpu(i, vcpu, kvm) { - kvm_s390_vcpu_crypto_setup(vcpu); - exit_sie(vcpu); - } + kvm_s390_vcpu_crypto_reset_all(kvm); mutex_unlock(&kvm->lock); return 0; } @@ -1033,8 +1040,8 @@ static int kvm_s390_set_tod(struct kvm *kvm, struct kvm_device_attr *attr) return ret; } -static void kvm_s390_get_tod_clock_ext(struct kvm *kvm, - struct kvm_s390_vm_tod_clock *gtod) +static void kvm_s390_get_tod_clock(struct kvm *kvm, + struct kvm_s390_vm_tod_clock *gtod) { struct kvm_s390_tod_clock_ext htod; @@ -1043,10 +1050,12 @@ static void kvm_s390_get_tod_clock_ext(struct kvm *kvm, get_tod_clock_ext((char *)&htod); gtod->tod = htod.tod + kvm->arch.epoch; - gtod->epoch_idx = htod.epoch_idx + kvm->arch.epdx; - - if (gtod->tod < htod.tod) - gtod->epoch_idx += 1; + gtod->epoch_idx = 0; + if (test_kvm_facility(kvm, 139)) { + gtod->epoch_idx = htod.epoch_idx + kvm->arch.epdx; + if (gtod->tod < htod.tod) + gtod->epoch_idx += 1; + } preempt_enable(); } @@ -1056,12 +1065,7 @@ static int kvm_s390_get_tod_ext(struct kvm *kvm, struct kvm_device_attr *attr) struct kvm_s390_vm_tod_clock gtod; memset(>od, 0, sizeof(gtod)); - - if (test_kvm_facility(kvm, 139)) - kvm_s390_get_tod_clock_ext(kvm, >od); - else - gtod.tod = kvm_s390_get_tod_clock_fast(kvm); - + kvm_s390_get_tod_clock(kvm, >od); if (copy_to_user((void __user *)attr->addr, >od, sizeof(gtod))) return -EFAULT; @@ -1493,7 +1497,7 @@ static long kvm_s390_get_skeys(struct kvm *kvm, struct kvm_s390_skeys *args) return -EINVAL; /* Is this guest using storage keys? */ - if (!mm_use_skey(current->mm)) + if (!mm_uses_skeys(current->mm)) return KVM_S390_GET_SKEYS_NONE; /* Enforce sane limit on memory allocation */ @@ -1725,7 +1729,7 @@ static int kvm_s390_set_cmma_bits(struct kvm *kvm, if (args->count == 0) return 0; - bits = vmalloc(sizeof(*bits) * args->count); + bits = vmalloc(array_size(sizeof(*bits), args->count)); if (!bits) return -ENOMEM; @@ -1982,10 +1986,10 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) rc = -ENOMEM; - kvm->arch.use_esca = 0; /* start with basic SCA */ if (!sclp.has_64bscao) alloc_flags |= GFP_DMA; rwlock_init(&kvm->arch.sca_lock); + /* start with basic SCA */ kvm->arch.sca = (struct bsca_block *) get_zeroed_page(alloc_flags); if (!kvm->arch.sca) goto out_err; @@ -2036,8 +2040,6 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) kvm_s390_crypto_init(kvm); mutex_init(&kvm->arch.float_int.ais_lock); - kvm->arch.float_int.simm = 0; - kvm->arch.float_int.nimm = 0; spin_lock_init(&kvm->arch.float_int.lock); for (i = 0; i < FIRQ_LIST_COUNT; i++) INIT_LIST_HEAD(&kvm->arch.float_int.lists[i]); @@ -2063,11 +2065,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) kvm->arch.gmap->pfault_enabled = 0; } - kvm->arch.css_support = 0; - kvm->arch.use_irqchip = 0; kvm->arch.use_pfmfi = sclp.has_pfmfi; - kvm->arch.epoch = 0; - + kvm->arch.use_skf = sclp.has_skey; spin_lock_init(&kvm->arch.start_stop_lock); kvm_s390_vsie_init(kvm); kvm_s390_gisa_init(kvm); @@ -2433,8 +2432,12 @@ static void kvm_s390_vcpu_initial_reset(struct kvm_vcpu *vcpu) vcpu->arch.sie_block->ckc = 0UL; vcpu->arch.sie_block->todpr = 0; memset(vcpu->arch.sie_block->gcr, 0, 16 * sizeof(__u64)); - vcpu->arch.sie_block->gcr[0] = 0xE0UL; - vcpu->arch.sie_block->gcr[14] = 0xC2000000UL; + vcpu->arch.sie_block->gcr[0] = CR0_UNUSED_56 | + CR0_INTERRUPT_KEY_SUBMASK | + CR0_MEASUREMENT_ALERT_SUBMASK; + vcpu->arch.sie_block->gcr[14] = CR14_UNUSED_32 | + CR14_UNUSED_33 | + CR14_EXTERNAL_DAMAGE_SUBMASK; /* make sure the new fpc will be lazily loaded */ save_fpu_regs(); current->thread.fpu.fpc = 0; @@ -3192,7 +3195,7 @@ static int kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu) return 0; if (kvm_s390_vcpu_has_irq(vcpu, 0)) return 0; - if (!(vcpu->arch.sie_block->gcr[0] & 0x200ul)) + if (!(vcpu->arch.sie_block->gcr[0] & CR0_SERVICE_SIGNAL_SUBMASK)) return 0; if (!vcpu->arch.gmap->pfault_enabled) return 0; @@ -3990,7 +3993,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, return r; } -int kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf) +vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf) { #ifdef CONFIG_KVM_S390_UCONTROL if ((vmf->pgoff == KVM_S390_SIE_PAGE_OFFSET) diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h index 1b5621f4fe5b..981e3ba97461 100644 --- a/arch/s390/kvm/kvm-s390.h +++ b/arch/s390/kvm/kvm-s390.h @@ -410,4 +410,17 @@ static inline int kvm_s390_use_sca_entries(void) } void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu, struct mcck_volatile_info *mcck_info); + +/** + * kvm_s390_vcpu_crypto_reset_all + * + * Reset the crypto attributes for each vcpu. This can be done while the vcpus + * are running as each vcpu will be removed from SIE before resetting the crypt + * attributes and restored to SIE afterward. + * + * Note: The kvm->lock must be held while calling this function + * + * @kvm: the KVM guest + */ +void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm); #endif diff --git a/arch/s390/kvm/priv.c b/arch/s390/kvm/priv.c index ebfa0442e569..eb0eb60c7be6 100644 --- a/arch/s390/kvm/priv.c +++ b/arch/s390/kvm/priv.c @@ -26,7 +26,6 @@ #include #include #include -#include #include #include "gaccess.h" #include "kvm-s390.h" @@ -205,24 +204,28 @@ static int handle_store_cpu_address(struct kvm_vcpu *vcpu) int kvm_s390_skey_check_enable(struct kvm_vcpu *vcpu) { - int rc = 0; + int rc; struct kvm_s390_sie_block *sie_block = vcpu->arch.sie_block; trace_kvm_s390_skey_related_inst(vcpu); - if (!(sie_block->ictl & (ICTL_ISKE | ICTL_SSKE | ICTL_RRBE)) && + /* Already enabled? */ + if (vcpu->kvm->arch.use_skf && + !(sie_block->ictl & (ICTL_ISKE | ICTL_SSKE | ICTL_RRBE)) && !kvm_s390_test_cpuflags(vcpu, CPUSTAT_KSS)) - return rc; + return 0; rc = s390_enable_skey(); VCPU_EVENT(vcpu, 3, "enabling storage keys for guest: %d", rc); - if (!rc) { - if (kvm_s390_test_cpuflags(vcpu, CPUSTAT_KSS)) - kvm_s390_clear_cpuflags(vcpu, CPUSTAT_KSS); - else - sie_block->ictl &= ~(ICTL_ISKE | ICTL_SSKE | - ICTL_RRBE); - } - return rc; + if (rc) + return rc; + + if (kvm_s390_test_cpuflags(vcpu, CPUSTAT_KSS)) + kvm_s390_clear_cpuflags(vcpu, CPUSTAT_KSS); + if (!vcpu->kvm->arch.use_skf) + sie_block->ictl |= ICTL_ISKE | ICTL_SSKE | ICTL_RRBE; + else + sie_block->ictl &= ~(ICTL_ISKE | ICTL_SSKE | ICTL_RRBE); + return 0; } static int try_handle_skey(struct kvm_vcpu *vcpu) @@ -232,7 +235,7 @@ static int try_handle_skey(struct kvm_vcpu *vcpu) rc = kvm_s390_skey_check_enable(vcpu); if (rc) return rc; - if (sclp.has_skey) { + if (vcpu->kvm->arch.use_skf) { /* with storage-key facility, SIE interprets it for us */ kvm_s390_retry_instr(vcpu); VCPU_EVENT(vcpu, 4, "%s", "retrying storage key operation"); diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c index 969882b54266..84c89cb9636f 100644 --- a/arch/s390/kvm/vsie.c +++ b/arch/s390/kvm/vsie.c @@ -557,7 +557,7 @@ static int pin_blocks(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) if (test_kvm_cpu_feat(vcpu->kvm, KVM_S390_VM_CPU_FEAT_64BSCAO)) gpa |= (u64) READ_ONCE(scb_o->scaoh) << 32; if (gpa) { - if (!(gpa & ~0x1fffUL)) + if (gpa < 2 * PAGE_SIZE) rc = set_validity_icpt(scb_s, 0x0038U); else if ((gpa & ~0x1fffUL) == kvm_s390_get_prefix(vcpu)) rc = set_validity_icpt(scb_s, 0x0011U); @@ -578,7 +578,7 @@ static int pin_blocks(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) gpa = READ_ONCE(scb_o->itdba) & ~0xffUL; if (gpa && (scb_s->ecb & ECB_TE)) { - if (!(gpa & ~0x1fffUL)) { + if (gpa < 2 * PAGE_SIZE) { rc = set_validity_icpt(scb_s, 0x0080U); goto unpin; } @@ -594,7 +594,7 @@ static int pin_blocks(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) gpa = READ_ONCE(scb_o->gvrd) & ~0x1ffUL; if (gpa && (scb_s->eca & ECA_VX) && !(scb_s->ecd & ECD_HOSTREGMGMT)) { - if (!(gpa & ~0x1fffUL)) { + if (gpa < 2 * PAGE_SIZE) { rc = set_validity_icpt(scb_s, 0x1310U); goto unpin; } @@ -613,7 +613,7 @@ static int pin_blocks(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) gpa = READ_ONCE(scb_o->riccbd) & ~0x3fUL; if (gpa && (scb_s->ecb3 & ECB3_RI)) { - if (!(gpa & ~0x1fffUL)) { + if (gpa < 2 * PAGE_SIZE) { rc = set_validity_icpt(scb_s, 0x0043U); goto unpin; } @@ -632,7 +632,7 @@ static int pin_blocks(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) gpa = READ_ONCE(scb_o->sdnxo) & ~0xfUL; sdnxc = READ_ONCE(scb_o->sdnxo) & 0xfUL; - if (!gpa || !(gpa & ~0x1fffUL)) { + if (!gpa || gpa < 2 * PAGE_SIZE) { rc = set_validity_icpt(scb_s, 0x10b0U); goto unpin; } diff --git a/arch/s390/lib/Makefile b/arch/s390/lib/Makefile index 9bfe0802684b..57ab40188d4b 100644 --- a/arch/s390/lib/Makefile +++ b/arch/s390/lib/Makefile @@ -8,3 +8,6 @@ obj-y += mem.o xor.o lib-$(CONFIG_SMP) += spinlock.o lib-$(CONFIG_KPROBES) += probes.o lib-$(CONFIG_UPROBES) += probes.o + +chkbss := mem.o +include $(srctree)/arch/s390/scripts/Makefile.chkbss diff --git a/arch/s390/mm/extmem.c b/arch/s390/mm/extmem.c index 920d40894535..6ad15d3fab81 100644 --- a/arch/s390/mm/extmem.c +++ b/arch/s390/mm/extmem.c @@ -103,7 +103,7 @@ static int scode_set; static int dcss_set_subcodes(void) { - char *name = kmalloc(8 * sizeof(char), GFP_KERNEL | GFP_DMA); + char *name = kmalloc(8, GFP_KERNEL | GFP_DMA); unsigned long rx, ry; int rc; diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c index 93faeca52284..e074480d3598 100644 --- a/arch/s390/mm/fault.c +++ b/arch/s390/mm/fault.c @@ -265,14 +265,10 @@ void report_user_fault(struct pt_regs *regs, long signr, int is_mm_fault) */ static noinline void do_sigsegv(struct pt_regs *regs, int si_code) { - struct siginfo si; - report_user_fault(regs, SIGSEGV, 1); - si.si_signo = SIGSEGV; - si.si_errno = 0; - si.si_code = si_code; - si.si_addr = (void __user *)(regs->int_parm_long & __FAIL_ADDR_MASK); - force_sig_info(SIGSEGV, &si, current); + force_sig_fault(SIGSEGV, si_code, + (void __user *)(regs->int_parm_long & __FAIL_ADDR_MASK), + current); } static noinline void do_no_context(struct pt_regs *regs) @@ -316,18 +312,13 @@ static noinline void do_low_address(struct pt_regs *regs) static noinline void do_sigbus(struct pt_regs *regs) { - struct task_struct *tsk = current; - struct siginfo si; - /* * Send a sigbus, regardless of whether we were in kernel * or user mode. */ - si.si_signo = SIGBUS; - si.si_errno = 0; - si.si_code = BUS_ADRERR; - si.si_addr = (void __user *)(regs->int_parm_long & __FAIL_ADDR_MASK); - force_sig_info(SIGBUS, &si, tsk); + force_sig_fault(SIGBUS, BUS_ADRERR, + (void __user *)(regs->int_parm_long & __FAIL_ADDR_MASK), + current); } static noinline int signal_return(struct pt_regs *regs) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 2c55a2b9d6c6..bc56ec8abcf7 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -2184,14 +2184,14 @@ int s390_enable_skey(void) int rc = 0; down_write(&mm->mmap_sem); - if (mm_use_skey(mm)) + if (mm_uses_skeys(mm)) goto out_up; - mm->context.use_skey = 1; + mm->context.uses_skeys = 1; for (vma = mm->mmap; vma; vma = vma->vm_next) { if (ksm_madvise(vma, vma->vm_start, vma->vm_end, MADV_UNMERGEABLE, &vma->vm_flags)) { - mm->context.use_skey = 0; + mm->context.uses_skeys = 0; rc = -ENOMEM; goto out_up; } diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 562f72955956..84bd6329a88d 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -190,14 +190,15 @@ unsigned long *page_table_alloc(struct mm_struct *mm) if (!list_empty(&mm->context.pgtable_list)) { page = list_first_entry(&mm->context.pgtable_list, struct page, lru); - mask = atomic_read(&page->_mapcount); + mask = atomic_read(&page->_refcount) >> 24; mask = (mask | (mask >> 4)) & 3; if (mask != 3) { table = (unsigned long *) page_to_phys(page); bit = mask & 1; /* =1 -> second 2K */ if (bit) table += PTRS_PER_PTE; - atomic_xor_bits(&page->_mapcount, 1U << bit); + atomic_xor_bits(&page->_refcount, + 1U << (bit + 24)); list_del(&page->lru); } } @@ -218,12 +219,12 @@ unsigned long *page_table_alloc(struct mm_struct *mm) table = (unsigned long *) page_to_phys(page); if (mm_alloc_pgste(mm)) { /* Return 4K page table with PGSTEs */ - atomic_set(&page->_mapcount, 3); + atomic_xor_bits(&page->_refcount, 3 << 24); memset64((u64 *)table, _PAGE_INVALID, PTRS_PER_PTE); memset64((u64 *)table + PTRS_PER_PTE, 0, PTRS_PER_PTE); } else { /* Return the first 2K fragment of the page */ - atomic_set(&page->_mapcount, 1); + atomic_xor_bits(&page->_refcount, 1 << 24); memset64((u64 *)table, _PAGE_INVALID, 2 * PTRS_PER_PTE); spin_lock_bh(&mm->context.lock); list_add(&page->lru, &mm->context.pgtable_list); @@ -242,7 +243,8 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) /* Free 2K page table fragment of a 4K page */ bit = (__pa(table) & ~PAGE_MASK)/(PTRS_PER_PTE*sizeof(pte_t)); spin_lock_bh(&mm->context.lock); - mask = atomic_xor_bits(&page->_mapcount, 1U << bit); + mask = atomic_xor_bits(&page->_refcount, 1U << (bit + 24)); + mask >>= 24; if (mask & 3) list_add(&page->lru, &mm->context.pgtable_list); else @@ -253,7 +255,6 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) } pgtable_page_dtor(page); - atomic_set(&page->_mapcount, -1); __free_page(page); } @@ -274,7 +275,8 @@ void page_table_free_rcu(struct mmu_gather *tlb, unsigned long *table, } bit = (__pa(table) & ~PAGE_MASK) / (PTRS_PER_PTE*sizeof(pte_t)); spin_lock_bh(&mm->context.lock); - mask = atomic_xor_bits(&page->_mapcount, 0x11U << bit); + mask = atomic_xor_bits(&page->_refcount, 0x11U << (bit + 24)); + mask >>= 24; if (mask & 3) list_add_tail(&page->lru, &mm->context.pgtable_list); else @@ -296,12 +298,13 @@ static void __tlb_remove_table(void *_table) break; case 1: /* lower 2K of a 4K page table */ case 2: /* higher 2K of a 4K page table */ - if (atomic_xor_bits(&page->_mapcount, mask << 4) != 0) + mask = atomic_xor_bits(&page->_refcount, mask << (4 + 24)); + mask >>= 24; + if (mask != 0) break; /* fallthrough */ case 3: /* 4K page table with pgstes */ pgtable_page_dtor(page); - atomic_set(&page->_mapcount, -1); __free_page(page); break; } diff --git a/arch/s390/mm/pgtable.c b/arch/s390/mm/pgtable.c index 4f2b65d01a70..301e466e4263 100644 --- a/arch/s390/mm/pgtable.c +++ b/arch/s390/mm/pgtable.c @@ -158,7 +158,7 @@ static inline pgste_t pgste_update_all(pte_t pte, pgste_t pgste, #ifdef CONFIG_PGSTE unsigned long address, bits, skey; - if (!mm_use_skey(mm) || pte_val(pte) & _PAGE_INVALID) + if (!mm_uses_skeys(mm) || pte_val(pte) & _PAGE_INVALID) return pgste; address = pte_val(pte) & PAGE_MASK; skey = (unsigned long) page_get_storage_key(address); @@ -180,7 +180,7 @@ static inline void pgste_set_key(pte_t *ptep, pgste_t pgste, pte_t entry, unsigned long address; unsigned long nkey; - if (!mm_use_skey(mm) || pte_val(entry) & _PAGE_INVALID) + if (!mm_uses_skeys(mm) || pte_val(entry) & _PAGE_INVALID) return; VM_BUG_ON(!(pte_val(*ptep) & _PAGE_INVALID)); address = pte_val(entry) & PAGE_MASK; diff --git a/arch/s390/net/Makefile b/arch/s390/net/Makefile index e0d5f245e42b..8cab6deb0403 100644 --- a/arch/s390/net/Makefile +++ b/arch/s390/net/Makefile @@ -2,4 +2,5 @@ # # Arch-specific network modules # -obj-$(CONFIG_BPF_JIT) += bpf_jit.o bpf_jit_comp.o +obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o +obj-$(CONFIG_HAVE_PNETID) += pnet.o diff --git a/arch/s390/net/bpf_jit.S b/arch/s390/net/bpf_jit.S deleted file mode 100644 index 9f794869c1b0..000000000000 --- a/arch/s390/net/bpf_jit.S +++ /dev/null @@ -1,120 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* - * BPF Jit compiler for s390, help functions. - * - * Copyright IBM Corp. 2012,2015 - * - * Author(s): Martin Schwidefsky - * Michael Holzheu - */ - -#include -#include -#include "bpf_jit.h" - -/* - * Calling convention: - * registers %r7-%r10, %r11,%r13, and %r15 are call saved - * - * Input (64 bit): - * %r3 (%b2) = offset into skb data - * %r6 (%b5) = return address - * %r7 (%b6) = skb pointer - * %r12 = skb data pointer - * - * Output: - * %r14= %b0 = return value (read skb value) - * - * Work registers: %r2,%r4,%r5,%r14 - * - * skb_copy_bits takes 4 parameters: - * %r2 = skb pointer - * %r3 = offset into skb data - * %r4 = pointer to temp buffer - * %r5 = length to copy - * Return value in %r2: 0 = ok - * - * bpf_internal_load_pointer_neg_helper takes 3 parameters: - * %r2 = skb pointer - * %r3 = offset into data - * %r4 = length to copy - * Return value in %r2: Pointer to data - */ - -#define SKF_MAX_NEG_OFF -0x200000 /* SKF_LL_OFF from filter.h */ - -/* - * Load SIZE bytes from SKB - */ -#define sk_load_common(NAME, SIZE, LOAD) \ -ENTRY(sk_load_##NAME); \ - ltgr %r3,%r3; /* Is offset negative? */ \ - jl sk_load_##NAME##_slow_neg; \ -ENTRY(sk_load_##NAME##_pos); \ - aghi %r3,SIZE; /* Offset + SIZE */ \ - clg %r3,STK_OFF_HLEN(%r15); /* Offset + SIZE > hlen? */ \ - jh sk_load_##NAME##_slow; \ - LOAD %r14,-SIZE(%r3,%r12); /* Get data from skb */ \ - B_EX OFF_OK,%r6; /* Return */ \ - \ -sk_load_##NAME##_slow:; \ - lgr %r2,%r7; /* Arg1 = skb pointer */ \ - aghi %r3,-SIZE; /* Arg2 = offset */ \ - la %r4,STK_OFF_TMP(%r15); /* Arg3 = temp bufffer */ \ - lghi %r5,SIZE; /* Arg4 = size */ \ - brasl %r14,skb_copy_bits; /* Get data from skb */ \ - LOAD %r14,STK_OFF_TMP(%r15); /* Load from temp bufffer */ \ - ltgr %r2,%r2; /* Set cc to (%r2 != 0) */ \ - BR_EX %r6; /* Return */ - -sk_load_common(word, 4, llgf) /* r14 = *(u32 *) (skb->data+offset) */ -sk_load_common(half, 2, llgh) /* r14 = *(u16 *) (skb->data+offset) */ - - GEN_BR_THUNK %r6 - GEN_B_THUNK OFF_OK,%r6 - -/* - * Load 1 byte from SKB (optimized version) - */ - /* r14 = *(u8 *) (skb->data+offset) */ -ENTRY(sk_load_byte) - ltgr %r3,%r3 # Is offset negative? - jl sk_load_byte_slow_neg -ENTRY(sk_load_byte_pos) - clg %r3,STK_OFF_HLEN(%r15) # Offset >= hlen? - jnl sk_load_byte_slow - llgc %r14,0(%r3,%r12) # Get byte from skb - B_EX OFF_OK,%r6 # Return OK - -sk_load_byte_slow: - lgr %r2,%r7 # Arg1 = skb pointer - # Arg2 = offset - la %r4,STK_OFF_TMP(%r15) # Arg3 = pointer to temp buffer - lghi %r5,1 # Arg4 = size (1 byte) - brasl %r14,skb_copy_bits # Get data from skb - llgc %r14,STK_OFF_TMP(%r15) # Load result from temp buffer - ltgr %r2,%r2 # Set cc to (%r2 != 0) - BR_EX %r6 # Return cc - -#define sk_negative_common(NAME, SIZE, LOAD) \ -sk_load_##NAME##_slow_neg:; \ - cgfi %r3,SKF_MAX_NEG_OFF; \ - jl bpf_error; \ - lgr %r2,%r7; /* Arg1 = skb pointer */ \ - /* Arg2 = offset */ \ - lghi %r4,SIZE; /* Arg3 = size */ \ - brasl %r14,bpf_internal_load_pointer_neg_helper; \ - ltgr %r2,%r2; \ - jz bpf_error; \ - LOAD %r14,0(%r2); /* Get data from pointer */ \ - xr %r3,%r3; /* Set cc to zero */ \ - BR_EX %r6; /* Return cc */ - -sk_negative_common(word, 4, llgf) -sk_negative_common(half, 2, llgh) -sk_negative_common(byte, 1, llgc) - -bpf_error: -# force a return 0 from jit handler - ltgr %r15,%r15 # Set condition code - BR_EX %r6 diff --git a/arch/s390/net/bpf_jit.h b/arch/s390/net/bpf_jit.h index 5e1e5133132d..7822ea92e54a 100644 --- a/arch/s390/net/bpf_jit.h +++ b/arch/s390/net/bpf_jit.h @@ -16,9 +16,6 @@ #include #include -extern u8 sk_load_word_pos[], sk_load_half_pos[], sk_load_byte_pos[]; -extern u8 sk_load_word[], sk_load_half[], sk_load_byte[]; - #endif /* __ASSEMBLY__ */ /* @@ -36,15 +33,6 @@ extern u8 sk_load_word[], sk_load_half[], sk_load_byte[]; * | | | * | BPF stack | | * | | | - * +---------------+ | - * | 8 byte skbp | | - * R15+176 -> +---------------+ | - * | 8 byte hlen | | - * R15+168 -> +---------------+ | - * | 4 byte align | | - * +---------------+ | - * | 4 byte temp | | - * | for bpf_jit.S | | * R15+160 -> +---------------+ | * | new backchain | | * R15+152 -> +---------------+ | @@ -57,17 +45,11 @@ extern u8 sk_load_word[], sk_load_half[], sk_load_byte[]; * The stack size used by the BPF program ("BPF stack" above) is passed * via "aux->stack_depth". */ -#define STK_SPACE_ADD (8 + 8 + 4 + 4 + 160) +#define STK_SPACE_ADD (160) #define STK_160_UNUSED (160 - 12 * 8) #define STK_OFF (STK_SPACE_ADD - STK_160_UNUSED) -#define STK_OFF_TMP 160 /* Offset of tmp buffer on stack */ -#define STK_OFF_HLEN 168 /* Offset of SKB header length on stack */ -#define STK_OFF_SKBP 176 /* Offset of SKB pointer on stack */ #define STK_OFF_R6 (160 - 11 * 8) /* Offset of r6 on stack */ #define STK_OFF_TCCNT (160 - 12 * 8) /* Offset of tail_call_cnt on stack */ -/* Offset to skip condition code check */ -#define OFF_OK 4 - #endif /* __ARCH_S390_NET_BPF_JIT_H */ diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index dd2bcf0e7d00..d2db8acb1a55 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -51,23 +51,21 @@ struct bpf_jit { #define BPF_SIZE_MAX 0xffff /* Max size for program (16 bit branches) */ -#define SEEN_SKB 1 /* skb access */ -#define SEEN_MEM 2 /* use mem[] for temporary storage */ -#define SEEN_RET0 4 /* ret0_ip points to a valid return 0 */ -#define SEEN_LITERAL 8 /* code uses literals */ -#define SEEN_FUNC 16 /* calls C functions */ -#define SEEN_TAIL_CALL 32 /* code uses tail calls */ -#define SEEN_REG_AX 64 /* code uses constant blinding */ -#define SEEN_STACK (SEEN_FUNC | SEEN_MEM | SEEN_SKB) +#define SEEN_MEM (1 << 0) /* use mem[] for temporary storage */ +#define SEEN_RET0 (1 << 1) /* ret0_ip points to a valid return 0 */ +#define SEEN_LITERAL (1 << 2) /* code uses literals */ +#define SEEN_FUNC (1 << 3) /* calls C functions */ +#define SEEN_TAIL_CALL (1 << 4) /* code uses tail calls */ +#define SEEN_REG_AX (1 << 5) /* code uses constant blinding */ +#define SEEN_STACK (SEEN_FUNC | SEEN_MEM) /* * s390 registers */ #define REG_W0 (MAX_BPF_JIT_REG + 0) /* Work register 1 (even) */ #define REG_W1 (MAX_BPF_JIT_REG + 1) /* Work register 2 (odd) */ -#define REG_SKB_DATA (MAX_BPF_JIT_REG + 2) /* SKB data register */ -#define REG_L (MAX_BPF_JIT_REG + 3) /* Literal pool register */ -#define REG_15 (MAX_BPF_JIT_REG + 4) /* Register 15 */ +#define REG_L (MAX_BPF_JIT_REG + 2) /* Literal pool register */ +#define REG_15 (MAX_BPF_JIT_REG + 3) /* Register 15 */ #define REG_0 REG_W0 /* Register 0 */ #define REG_1 REG_W1 /* Register 1 */ #define REG_2 BPF_REG_1 /* Register 2 */ @@ -92,10 +90,8 @@ static const int reg2hex[] = { [BPF_REG_9] = 10, /* BPF stack pointer */ [BPF_REG_FP] = 13, - /* Register for blinding (shared with REG_SKB_DATA) */ + /* Register for blinding */ [BPF_REG_AX] = 12, - /* SKB data pointer */ - [REG_SKB_DATA] = 12, /* Work registers for s390x backend */ [REG_W0] = 0, [REG_W1] = 1, @@ -401,27 +397,6 @@ static void save_restore_regs(struct bpf_jit *jit, int op, u32 stack_depth) } while (re <= 15); } -/* - * For SKB access %b1 contains the SKB pointer. For "bpf_jit.S" - * we store the SKB header length on the stack and the SKB data - * pointer in REG_SKB_DATA if BPF_REG_AX is not used. - */ -static void emit_load_skb_data_hlen(struct bpf_jit *jit) -{ - /* Header length: llgf %w1,(%b1) */ - EMIT6_DISP_LH(0xe3000000, 0x0016, REG_W1, REG_0, BPF_REG_1, - offsetof(struct sk_buff, len)); - /* s %w1,(%b1) */ - EMIT4_DISP(0x5b000000, REG_W1, BPF_REG_1, - offsetof(struct sk_buff, data_len)); - /* stg %w1,ST_OFF_HLEN(%r0,%r15) */ - EMIT6_DISP_LH(0xe3000000, 0x0024, REG_W1, REG_0, REG_15, STK_OFF_HLEN); - if (!(jit->seen & SEEN_REG_AX)) - /* lg %skb_data,data_off(%b1) */ - EMIT6_DISP_LH(0xe3000000, 0x0004, REG_SKB_DATA, REG_0, - BPF_REG_1, offsetof(struct sk_buff, data)); -} - /* * Emit function prologue * @@ -462,12 +437,6 @@ static void bpf_jit_prologue(struct bpf_jit *jit, u32 stack_depth) EMIT6_DISP_LH(0xe3000000, 0x0024, REG_W1, REG_0, REG_15, 152); } - if (jit->seen & SEEN_SKB) { - emit_load_skb_data_hlen(jit); - /* stg %b1,ST_OFF_SKBP(%r0,%r15) */ - EMIT6_DISP_LH(0xe3000000, 0x0024, BPF_REG_1, REG_0, REG_15, - STK_OFF_SKBP); - } } /* @@ -537,12 +506,12 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, int i { struct bpf_insn *insn = &fp->insnsi[i]; int jmp_off, last, insn_count = 1; - unsigned int func_addr, mask; u32 dst_reg = insn->dst_reg; u32 src_reg = insn->src_reg; u32 *addrs = jit->addrs; s32 imm = insn->imm; s16 off = insn->off; + unsigned int mask; if (dst_reg == BPF_REG_AX || src_reg == BPF_REG_AX) jit->seen |= SEEN_REG_AX; @@ -1029,13 +998,6 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, int i } /* lgr %b0,%r2: load return value into %b0 */ EMIT4(0xb9040000, BPF_REG_0, REG_2); - if ((jit->seen & SEEN_SKB) && - bpf_helper_changes_pkt_data((void *)func)) { - /* lg %b1,ST_OFF_SKBP(%r15) */ - EMIT6_DISP_LH(0xe3000000, 0x0004, BPF_REG_1, REG_0, - REG_15, STK_OFF_SKBP); - emit_load_skb_data_hlen(jit); - } break; } case BPF_JMP | BPF_TAIL_CALL: @@ -1235,73 +1197,6 @@ branch_oc: jmp_off = addrs[i + off + 1] - (addrs[i + 1] - 4); EMIT4_PCREL(0xa7040000 | mask << 8, jmp_off); break; - /* - * BPF_LD - */ - case BPF_LD | BPF_ABS | BPF_B: /* b0 = *(u8 *) (skb->data+imm) */ - case BPF_LD | BPF_IND | BPF_B: /* b0 = *(u8 *) (skb->data+imm+src) */ - if ((BPF_MODE(insn->code) == BPF_ABS) && (imm >= 0)) - func_addr = __pa(sk_load_byte_pos); - else - func_addr = __pa(sk_load_byte); - goto call_fn; - case BPF_LD | BPF_ABS | BPF_H: /* b0 = *(u16 *) (skb->data+imm) */ - case BPF_LD | BPF_IND | BPF_H: /* b0 = *(u16 *) (skb->data+imm+src) */ - if ((BPF_MODE(insn->code) == BPF_ABS) && (imm >= 0)) - func_addr = __pa(sk_load_half_pos); - else - func_addr = __pa(sk_load_half); - goto call_fn; - case BPF_LD | BPF_ABS | BPF_W: /* b0 = *(u32 *) (skb->data+imm) */ - case BPF_LD | BPF_IND | BPF_W: /* b0 = *(u32 *) (skb->data+imm+src) */ - if ((BPF_MODE(insn->code) == BPF_ABS) && (imm >= 0)) - func_addr = __pa(sk_load_word_pos); - else - func_addr = __pa(sk_load_word); - goto call_fn; -call_fn: - jit->seen |= SEEN_SKB | SEEN_RET0 | SEEN_FUNC; - REG_SET_SEEN(REG_14); /* Return address of possible func call */ - - /* - * Implicit input: - * BPF_REG_6 (R7) : skb pointer - * REG_SKB_DATA (R12): skb data pointer (if no BPF_REG_AX) - * - * Calculated input: - * BPF_REG_2 (R3) : offset of byte(s) to fetch in skb - * BPF_REG_5 (R6) : return address - * - * Output: - * BPF_REG_0 (R14): data read from skb - * - * Scratch registers (BPF_REG_1-5) - */ - - /* Call function: llilf %w1,func_addr */ - EMIT6_IMM(0xc00f0000, REG_W1, func_addr); - - /* Offset: lgfi %b2,imm */ - EMIT6_IMM(0xc0010000, BPF_REG_2, imm); - if (BPF_MODE(insn->code) == BPF_IND) - /* agfr %b2,%src (%src is s32 here) */ - EMIT4(0xb9180000, BPF_REG_2, src_reg); - - /* Reload REG_SKB_DATA if BPF_REG_AX is used */ - if (jit->seen & SEEN_REG_AX) - /* lg %skb_data,data_off(%b6) */ - EMIT6_DISP_LH(0xe3000000, 0x0004, REG_SKB_DATA, REG_0, - BPF_REG_6, offsetof(struct sk_buff, data)); - /* basr %b5,%w1 (%b5 is call saved) */ - EMIT2(0x0d00, BPF_REG_5, REG_W1); - - /* - * Note: For fast access we jump directly after the - * jnz instruction from bpf_jit.S - */ - /* jnz */ - EMIT4_PCREL(0xa7740000, jit->ret0_ip - jit->prg); - break; default: /* too complex, give up */ pr_err("Unknown opcode %02x\n", insn->code); return -1; diff --git a/arch/s390/net/pnet.c b/arch/s390/net/pnet.c new file mode 100644 index 000000000000..e22f1b10a6c7 --- /dev/null +++ b/arch/s390/net/pnet.c @@ -0,0 +1,80 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * IBM System z PNET ID Support + * + * Copyright IBM Corp. 2018 + */ + +#include +#include +#include +#include +#include +#include +#include + +/* + * Get the PNETIDs from a device. + * s390 hardware supports the definition of a so-called Physical Network + * Identifier (short PNETID) per network device port. These PNETIDs can be + * used to identify network devices that are attached to the same physical + * network (broadcast domain). + * + * The device can be + * - a ccwgroup device with all bundled subchannels having the same PNETID + * - a PCI attached network device + * + * Returns: + * 0: PNETIDs extracted from device. + * -ENOMEM: No memory to extract utility string. + * -EOPNOTSUPP: Device type without utility string support + */ +static int pnet_ids_by_device(struct device *dev, u8 *pnetids) +{ + memset(pnetids, 0, PNETIDS_LEN); + if (dev_is_ccwgroup(dev)) { + struct ccwgroup_device *gdev = to_ccwgroupdev(dev); + u8 *util_str; + + util_str = ccw_device_get_util_str(gdev->cdev[0], 0); + if (!util_str) + return -ENOMEM; + memcpy(pnetids, util_str, PNETIDS_LEN); + kfree(util_str); + return 0; + } + if (dev_is_pci(dev)) { + struct zpci_dev *zdev = to_zpci(to_pci_dev(dev)); + + memcpy(pnetids, zdev->util_str, sizeof(zdev->util_str)); + return 0; + } + return -EOPNOTSUPP; +} + +/* + * Extract the pnetid for a device port. + * + * Return 0 if a pnetid is found and -ENOENT otherwise. + */ +int pnet_id_by_dev_port(struct device *dev, unsigned short port, u8 *pnetid) +{ + u8 pnetids[MAX_PNETID_PORTS][MAX_PNETID_LEN]; + static const u8 zero[MAX_PNETID_LEN] = { 0 }; + int rc = 0; + + if (!dev || port >= MAX_PNETID_PORTS) + return -ENOENT; + + if (!pnet_ids_by_device(dev, (u8 *)pnetids) && + memcmp(pnetids[port], zero, MAX_PNETID_LEN)) + memcpy(pnetid, pnetids[port], MAX_PNETID_LEN); + else + rc = -ENOENT; + + return rc; +} +EXPORT_SYMBOL_GPL(pnet_id_by_dev_port); + +MODULE_DESCRIPTION("pnetid determination from utility strings"); +MODULE_LICENSE("GPL"); diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c index 93cd0f1ca12b..19b2d2a9b43d 100644 --- a/arch/s390/pci/pci_clp.c +++ b/arch/s390/pci/pci_clp.c @@ -19,7 +19,6 @@ #include #include #include -#include #include #include diff --git a/arch/s390/pci/pci_dma.c b/arch/s390/pci/pci_dma.c index 2d15d84c20ed..d387a0fbdd7e 100644 --- a/arch/s390/pci/pci_dma.c +++ b/arch/s390/pci/pci_dma.c @@ -668,15 +668,6 @@ void zpci_dma_exit(void) kmem_cache_destroy(dma_region_table_cache); } -#define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16) - -static int __init dma_debug_do_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - return 0; -} -fs_initcall(dma_debug_do_init); - const struct dma_map_ops s390_pci_dma_ops = { .alloc = s390_dma_alloc, .free = s390_dma_free, @@ -685,8 +676,6 @@ const struct dma_map_ops s390_pci_dma_ops = { .map_page = s390_dma_map_pages, .unmap_page = s390_dma_unmap_pages, .mapping_error = s390_mapping_error, - /* if we support direct DMA this must be conditional */ - .is_phys = 0, /* dma_supported is unconditionally true without a callback */ }; EXPORT_SYMBOL_GPL(s390_pci_dma_ops); diff --git a/arch/s390/purgatory/Makefile b/arch/s390/purgatory/Makefile index e9525bc1b4a6..1ace023cbdce 100644 --- a/arch/s390/purgatory/Makefile +++ b/arch/s390/purgatory/Makefile @@ -21,7 +21,7 @@ LDFLAGS_purgatory.ro += -z nodefaultlib KBUILD_CFLAGS := -fno-strict-aliasing -Wall -Wstrict-prototypes KBUILD_CFLAGS += -Wno-pointer-sign -Wno-sign-compare KBUILD_CFLAGS += -fno-zero-initialized-in-bss -fno-builtin -ffreestanding -KBUILD_CFLAGS += -c -MD -Os -m64 +KBUILD_CFLAGS += -c -MD -Os -m64 -msoft-float KBUILD_CFLAGS += $(call cc-option,-fno-PIE) $(obj)/purgatory.ro: $(PURGATORY_OBJS) FORCE diff --git a/arch/s390/scripts/Makefile.chkbss b/arch/s390/scripts/Makefile.chkbss new file mode 100644 index 000000000000..d92f2d94a5d9 --- /dev/null +++ b/arch/s390/scripts/Makefile.chkbss @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0 + +quiet_cmd_chkbss = CHKBSS $< +define cmd_chkbss + if ! $(OBJDUMP) -j .bss -w -h $< | awk 'END { if ($$3) exit 1 }'; then \ + echo "error: $< .bss section is not empty" >&2; exit 1; \ + fi; \ + touch $@; +endef + +$(obj)/built-in.a: $(patsubst %, $(obj)/%.chkbss, $(chkbss)) + +%.o.chkbss: %.o + $(call cmd,chkbss) diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig index 1851eaeee131..dd4f3d3e644f 100644 --- a/arch/sh/Kconfig +++ b/arch/sh/Kconfig @@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 config SUPERH def_bool y + select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_MIGHT_HAVE_PC_PARPORT select ARCH_NO_COHERENT_DMA_MMAP if !MMU @@ -14,7 +15,6 @@ config SUPERH select HAVE_OPROFILE select HAVE_GENERIC_DMA_COHERENT select HAVE_ARCH_TRACEHOOK - select HAVE_DMA_API_DEBUG select HAVE_PERF_EVENTS select HAVE_DEBUG_BUGVERBOSE select ARCH_HAVE_CUSTOM_GPIO_H @@ -51,6 +51,9 @@ config SUPERH select HAVE_ARCH_AUDITSYSCALL select HAVE_FUTEX_CMPXCHG if FUTEX select HAVE_NMI + select NEED_DMA_MAP_STATE + select NEED_SG_DMA_LENGTH + help The SuperH is a RISC processor targeted for use in embedded systems and consumer electronics; it was also used in the Sega Dreamcast @@ -58,7 +61,7 @@ config SUPERH . config SUPERH32 - def_bool ARCH = "sh" + def_bool "$(ARCH)" = "sh" select HAVE_KPROBES select HAVE_KRETPROBES select HAVE_IOREMAP_PROT if MMU && !X2TLB @@ -74,10 +77,10 @@ config SUPERH32 select PERF_EVENTS select ARCH_HIBERNATION_POSSIBLE if MMU select SPARSE_IRQ - select HAVE_CC_STACKPROTECTOR + select HAVE_STACKPROTECTOR config SUPERH64 - def_bool ARCH = "sh64" + def_bool "$(ARCH)" = "sh64" select HAVE_EXIT_THREAD select KALLSYMS @@ -161,12 +164,6 @@ config DMA_COHERENT config DMA_NONCOHERENT def_bool !DMA_COHERENT -config NEED_DMA_MAP_STATE - def_bool DMA_NONCOHERENT - -config NEED_SG_DMA_LENGTH - def_bool y - config PGTABLE_LEVELS default 3 if X2TLB default 2 @@ -690,7 +687,7 @@ config SMP People using multiprocessor machines who say Y here should also say Y to "Enhanced Real Time Clock Support", below. - See also and the SMP-HOWTO + See also and the SMP-HOWTO available at . If you don't know what to do here, say N. diff --git a/arch/sh/boards/board-sh7785lcr.c b/arch/sh/boards/board-sh7785lcr.c index d7d232dea33e..3cba60ff7aab 100644 --- a/arch/sh/boards/board-sh7785lcr.c +++ b/arch/sh/boards/board-sh7785lcr.c @@ -17,7 +17,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/arch/sh/boards/mach-migor/setup.c b/arch/sh/boards/mach-migor/setup.c index 271dfc260e82..3d7d0046cf49 100644 --- a/arch/sh/boards/mach-migor/setup.c +++ b/arch/sh/boards/mach-migor/setup.c @@ -359,7 +359,7 @@ static struct gpiod_lookup_table ov7725_gpios = { static struct gpiod_lookup_table tw9910_gpios = { .dev_id = "0-0045", .table = { - GPIO_LOOKUP("sh7722_pfc", GPIO_PTT2, "pdn", GPIO_ACTIVE_HIGH), + GPIO_LOOKUP("sh7722_pfc", GPIO_PTT2, "pdn", GPIO_ACTIVE_LOW), GPIO_LOOKUP("sh7722_pfc", GPIO_PTT3, "rstb", GPIO_ACTIVE_LOW), }, }; diff --git a/arch/sh/drivers/dma/dma-api.c b/arch/sh/drivers/dma/dma-api.c index c0eec08d8f95..b05be597b19f 100644 --- a/arch/sh/drivers/dma/dma-api.c +++ b/arch/sh/drivers/dma/dma-api.c @@ -339,18 +339,6 @@ static int dma_proc_show(struct seq_file *m, void *v) return 0; } -static int dma_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, dma_proc_show, NULL); -} - -static const struct file_operations dma_proc_fops = { - .open = dma_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - int register_dmac(struct dma_info *info) { unsigned int total_channels, i; @@ -423,7 +411,7 @@ EXPORT_SYMBOL(unregister_dmac); static int __init dma_api_init(void) { printk(KERN_NOTICE "DMA: Registering DMA API.\n"); - return proc_create("dma", 0, NULL, &dma_proc_fops) ? 0 : -ENOMEM; + return proc_create_single("dma", 0, NULL, dma_proc_show) ? 0 : -ENOMEM; } subsys_initcall(dma_api_init); diff --git a/arch/sh/drivers/dma/dmabrg.c b/arch/sh/drivers/dma/dmabrg.c index c0dd904483c7..e5a57a109d6c 100644 --- a/arch/sh/drivers/dma/dmabrg.c +++ b/arch/sh/drivers/dma/dmabrg.c @@ -154,7 +154,7 @@ static int __init dmabrg_init(void) unsigned long or; int ret; - dmabrg_handlers = kzalloc(10 * sizeof(struct dmabrg_handler), + dmabrg_handlers = kcalloc(10, sizeof(struct dmabrg_handler), GFP_KERNEL); if (!dmabrg_handlers) return -ENOMEM; diff --git a/arch/sh/drivers/pci/pcie-sh7786.c b/arch/sh/drivers/pci/pcie-sh7786.c index 382e7ecf4c82..3d81a8b80942 100644 --- a/arch/sh/drivers/pci/pcie-sh7786.c +++ b/arch/sh/drivers/pci/pcie-sh7786.c @@ -561,7 +561,7 @@ static int __init sh7786_pcie_init(void) if (unlikely(nr_ports == 0)) return -ENODEV; - sh7786_pcie_ports = kzalloc(nr_ports * sizeof(struct sh7786_pcie_port), + sh7786_pcie_ports = kcalloc(nr_ports, sizeof(struct sh7786_pcie_port), GFP_KERNEL); if (unlikely(!sh7786_pcie_ports)) return -ENOMEM; diff --git a/arch/sh/include/asm/Kbuild b/arch/sh/include/asm/Kbuild index 1efcce74997b..46dd82ab2c29 100644 --- a/arch/sh/include/asm/Kbuild +++ b/arch/sh/include/asm/Kbuild @@ -1,3 +1,4 @@ +generic-y += compat.h generic-y += current.h generic-y += delay.h generic-y += div64.h diff --git a/arch/sh/include/asm/pci.h b/arch/sh/include/asm/pci.h index 0033f0df2b3b..10a36b1cf2ea 100644 --- a/arch/sh/include/asm/pci.h +++ b/arch/sh/include/asm/pci.h @@ -71,12 +71,6 @@ extern unsigned long PCIBIOS_MIN_IO, PCIBIOS_MIN_MEM; * SuperH has everything mapped statically like x86. */ -/* The PCI address space does equal the physical memory - * address space. The networking and block device layers use - * this boolean for bounce buffer decisions. - */ -#define PCI_DMA_BUS_IS_PHYS (dma_ops->is_phys) - #ifdef CONFIG_PCI /* * None of the SH PCI controllers support MWI, it is always treated as a diff --git a/arch/sh/include/asm/pgtable.h b/arch/sh/include/asm/pgtable.h index 89c513a982fc..f6abfe2bca93 100644 --- a/arch/sh/include/asm/pgtable.h +++ b/arch/sh/include/asm/pgtable.h @@ -156,8 +156,6 @@ extern void page_table_range_init(unsigned long start, unsigned long end, #define HAVE_ARCH_UNMAPPED_AREA #define HAVE_ARCH_UNMAPPED_AREA_TOPDOWN -#define __HAVE_ARCH_PTE_SPECIAL - #include #endif /* __ASM_SH_PGTABLE_H */ diff --git a/arch/sh/include/asm/vmlinux.lds.h b/arch/sh/include/asm/vmlinux.lds.h index f312813f39d8..992955685874 100644 --- a/arch/sh/include/asm/vmlinux.lds.h +++ b/arch/sh/include/asm/vmlinux.lds.h @@ -7,9 +7,9 @@ #ifdef CONFIG_DWARF_UNWINDER #define DWARF_EH_FRAME \ .eh_frame : AT(ADDR(.eh_frame) - LOAD_OFFSET) { \ - VMLINUX_SYMBOL(__start_eh_frame) = .; \ + __start_eh_frame = .; \ *(.eh_frame) \ - VMLINUX_SYMBOL(__stop_eh_frame) = .; \ + __stop_eh_frame = .; \ } #else #define DWARF_EH_FRAME diff --git a/arch/sh/kernel/dma-nommu.c b/arch/sh/kernel/dma-nommu.c index 178457d7620c..3e3a32fc676e 100644 --- a/arch/sh/kernel/dma-nommu.c +++ b/arch/sh/kernel/dma-nommu.c @@ -78,7 +78,6 @@ const struct dma_map_ops nommu_dma_ops = { .sync_single_for_device = nommu_sync_single_for_device, .sync_sg_for_device = nommu_sync_sg_for_device, #endif - .is_phys = 1, }; void __init no_iommu_init(void) diff --git a/arch/sh/kernel/hw_breakpoint.c b/arch/sh/kernel/hw_breakpoint.c index afe965712a69..8648ed05ccf0 100644 --- a/arch/sh/kernel/hw_breakpoint.c +++ b/arch/sh/kernel/hw_breakpoint.c @@ -347,13 +347,8 @@ static int __kprobes hw_breakpoint_handler(struct die_args *args) /* Deliver the signal to userspace */ if (!arch_check_bp_in_kernelspace(bp)) { - siginfo_t info; - - info.si_signo = args->signr; - info.si_errno = notifier_to_errno(rc); - info.si_code = TRAP_HWBKPT; - - force_sig_info(args->signr, &info, current); + force_sig_fault(SIGTRAP, TRAP_HWBKPT, + (void __user *)NULL, current); } rcu_read_unlock(); diff --git a/arch/sh/kernel/irq.c b/arch/sh/kernel/irq.c index 245dbeb20afe..5717c7cbdd97 100644 --- a/arch/sh/kernel/irq.c +++ b/arch/sh/kernel/irq.c @@ -44,7 +44,7 @@ int arch_show_interrupts(struct seq_file *p, int prec) seq_printf(p, "%*s: ", prec, "NMI"); for_each_online_cpu(j) - seq_printf(p, "%10u ", irq_stat[j].__nmi_count); + seq_printf(p, "%10u ", nmi_count(j)); seq_printf(p, " Non-maskable interrupts\n"); seq_printf(p, "%*s: %10u\n", prec, "ERR", atomic_read(&irq_err_count)); diff --git a/arch/sh/kernel/process.c b/arch/sh/kernel/process.c index 68b1a67533ce..4d1bfc848dd3 100644 --- a/arch/sh/kernel/process.c +++ b/arch/sh/kernel/process.c @@ -12,7 +12,7 @@ struct kmem_cache *task_xstate_cachep = NULL; unsigned int xstate_size; -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR unsigned long __stack_chk_guard __read_mostly; EXPORT_SYMBOL(__stack_chk_guard); #endif diff --git a/arch/sh/kernel/process_32.c b/arch/sh/kernel/process_32.c index 93522069cb15..27fddb56b3e1 100644 --- a/arch/sh/kernel/process_32.c +++ b/arch/sh/kernel/process_32.c @@ -177,7 +177,7 @@ __switch_to(struct task_struct *prev, struct task_struct *next) { struct thread_struct *next_t = &next->thread; -#if defined(CONFIG_CC_STACKPROTECTOR) && !defined(CONFIG_SMP) +#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) __stack_chk_guard = next->stack_canary; #endif diff --git a/arch/sh/kernel/traps_32.c b/arch/sh/kernel/traps_32.c index b3770bb26211..60709ad17fc7 100644 --- a/arch/sh/kernel/traps_32.c +++ b/arch/sh/kernel/traps_32.c @@ -477,7 +477,6 @@ asmlinkage void do_address_error(struct pt_regs *regs, { unsigned long error_code = 0; mm_segment_t oldfs; - siginfo_t info; insn_size_t instruction; int tmp; @@ -537,11 +536,7 @@ uspace_segv: "access (PC %lx PR %lx)\n", current->comm, regs->pc, regs->pr); - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = si_code; - info.si_addr = (void __user *)address; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGBUS, si_code, (void __user *)address, current); } else { inc_unaligned_kernel_access(); @@ -598,19 +593,20 @@ int is_dsp_inst(struct pt_regs *regs) #ifdef CONFIG_CPU_SH2A asmlinkage void do_divide_error(unsigned long r4) { - siginfo_t info; + int code; switch (r4) { case TRAP_DIVZERO_ERROR: - info.si_code = FPE_INTDIV; + code = FPE_INTDIV; break; case TRAP_DIVOVF_ERROR: - info.si_code = FPE_INTOVF; + code = FPE_INTOVF; break; + default: + /* Let gcc know unhandled cases don't make it past here */ + return; } - - info.si_signo = SIGFPE; - force_sig_info(info.si_signo, &info, current); + force_sig_fault(SIGFPE, code, NULL, current); } #endif diff --git a/arch/sh/math-emu/math.c b/arch/sh/math-emu/math.c index c86f4360c6ce..a0fa8fc88739 100644 --- a/arch/sh/math-emu/math.c +++ b/arch/sh/math-emu/math.c @@ -507,7 +507,6 @@ static int ieee_fpe_handler(struct pt_regs *regs) unsigned short insn = *(unsigned short *)regs->pc; unsigned short finsn; unsigned long nextpc; - siginfo_t info; int nib[4] = { (insn >> 12) & 0xf, (insn >> 8) & 0xf, @@ -560,11 +559,8 @@ static int ieee_fpe_handler(struct pt_regs *regs) ~(FPSCR_CAUSE_MASK | FPSCR_FLAG_MASK); task_thread_info(tsk)->status |= TS_USEDFPU; } else { - info.si_signo = SIGFPE; - info.si_errno = 0; - info.si_code = FPE_FLTINV; - info.si_addr = (void __user *)regs->pc; - force_sig_info(SIGFPE, &info, tsk); + force_sig_fault(SIGFPE, FPE_FLTINV, + (void __user *)regs->pc, tsk); } regs->pc = nextpc; diff --git a/arch/sh/mm/consistent.c b/arch/sh/mm/consistent.c index f1b44697ad68..fceb2adfcac7 100644 --- a/arch/sh/mm/consistent.c +++ b/arch/sh/mm/consistent.c @@ -20,18 +20,9 @@ #include #include -#define PREALLOC_DMA_DEBUG_ENTRIES 4096 - const struct dma_map_ops *dma_ops; EXPORT_SYMBOL(dma_ops); -static int __init dma_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - return 0; -} -fs_initcall(dma_init); - void *dma_generic_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) diff --git a/arch/sh/mm/fault.c b/arch/sh/mm/fault.c index 6fd1bf7481c7..b8e7bb84b6b1 100644 --- a/arch/sh/mm/fault.c +++ b/arch/sh/mm/fault.c @@ -42,14 +42,7 @@ static void force_sig_info_fault(int si_signo, int si_code, unsigned long address, struct task_struct *tsk) { - siginfo_t info; - - info.si_signo = si_signo; - info.si_errno = 0; - info.si_code = si_code; - info.si_addr = (void __user *)address; - - force_sig_info(si_signo, &info, tsk); + force_sig_fault(si_signo, si_code, (void __user *)address, tsk); } /* diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig index 8767e45f1b2b..0f535debf802 100644 --- a/arch/sparc/Kconfig +++ b/arch/sparc/Kconfig @@ -1,6 +1,6 @@ config 64BIT - bool "64-bit kernel" if ARCH = "sparc" - default ARCH = "sparc64" + bool "64-bit kernel" if "$(ARCH)" = "sparc" + default "$(ARCH)" = "sparc64" help SPARC is a family of RISC microprocessors designed and marketed by Sun Microsystems, incorporated. They are very widely found in Sun @@ -25,7 +25,6 @@ config SPARC select RTC_CLASS select RTC_DRV_M48T59 select RTC_SYSTOHC - select HAVE_DMA_API_DEBUG select HAVE_ARCH_JUMP_LABEL if SPARC64 select GENERIC_IRQ_SHOW select ARCH_WANT_IPC_PARSE_VERSION @@ -44,6 +43,8 @@ config SPARC select ARCH_HAS_SG_CHAIN select CPU_NO_EFFICIENT_FFS select LOCKDEP_SMALL if LOCKDEP + select NEED_DMA_MAP_STATE + select NEED_SG_DMA_LENGTH config SPARC32 def_bool !64BIT @@ -67,6 +68,7 @@ config SPARC64 select HAVE_SYSCALL_TRACEPOINTS select HAVE_CONTEXT_TRACKING select HAVE_DEBUG_KMEMLEAK + select IOMMU_HELPER select SPARSE_IRQ select RTC_DRV_CMOS select RTC_DRV_BQ4802 @@ -86,6 +88,7 @@ config SPARC64 select ARCH_USE_QUEUED_SPINLOCKS select GENERIC_TIME_VSYSCALL select ARCH_CLOCKSOURCE_DATA + select ARCH_HAS_PTE_SPECIAL config ARCH_DEFCONFIG string @@ -102,14 +105,6 @@ config ARCH_ATU bool default y if SPARC64 -config ARCH_DMA_ADDR_T_64BIT - bool - default y if ARCH_ATU - -config IOMMU_HELPER - bool - default y if SPARC64 - config STACKTRACE_SUPPORT bool default y if SPARC64 @@ -146,12 +141,6 @@ config ZONE_DMA bool default y if SPARC32 -config NEED_DMA_MAP_STATE - def_bool y - -config NEED_SG_DMA_LENGTH - def_bool y - config GENERIC_ISA_DMA bool default y if SPARC32 @@ -189,7 +178,7 @@ config SMP Y to "Enhanced Real Time Clock Support", below. The "Advanced Power Management" code will be disabled if you say Y here. - See also and the SMP-HOWTO + See also and the SMP-HOWTO available at . If you don't know what to do here, say N. diff --git a/arch/sparc/Makefile b/arch/sparc/Makefile index edac927e4952..966a13d2b127 100644 --- a/arch/sparc/Makefile +++ b/arch/sparc/Makefile @@ -39,7 +39,7 @@ else # sparc64 # -CHECKFLAGS += -D__sparc__ -D__sparc_v9__ -D__arch64__ -m64 +CHECKFLAGS += -D__sparc__ -D__sparc_v9__ -D__arch64__ LDFLAGS := -m elf64_sparc export BITS := 64 UTS_MACHINE := sparc64 diff --git a/arch/sparc/include/asm/compat.h b/arch/sparc/include/asm/compat.h index 615283e16f22..4eb51d2dae98 100644 --- a/arch/sparc/include/asm/compat.h +++ b/arch/sparc/include/asm/compat.h @@ -11,7 +11,6 @@ typedef u32 compat_size_t; typedef s32 compat_ssize_t; -typedef s32 compat_time_t; typedef s32 compat_clock_t; typedef s32 compat_pid_t; typedef u16 __compat_uid_t; @@ -39,16 +38,6 @@ typedef u32 compat_ulong_t; typedef u64 compat_u64; typedef u32 compat_uptr_t; -struct compat_timespec { - compat_time_t tv_sec; - s32 tv_nsec; -}; - -struct compat_timeval { - compat_time_t tv_sec; - s32 tv_usec; -}; - struct compat_stat { compat_dev_t st_dev; compat_ino_t st_ino; @@ -168,6 +157,7 @@ static inline compat_uptr_t ptr_to_compat(void __user *uptr) return (u32)(unsigned long)uptr; } +#ifdef CONFIG_COMPAT static inline void __user *arch_compat_alloc_user_space(long len) { struct pt_regs *regs = current_thread_info()->kregs; @@ -184,6 +174,7 @@ static inline void __user *arch_compat_alloc_user_space(long len) return (void __user *) usp; } +#endif struct compat_ipc64_perm { compat_key_t key; @@ -201,10 +192,10 @@ struct compat_ipc64_perm { struct compat_semid64_ds { struct compat_ipc64_perm sem_perm; - unsigned int __pad1; - compat_time_t sem_otime; - unsigned int __pad2; - compat_time_t sem_ctime; + unsigned int sem_otime_high; + unsigned int sem_otime; + unsigned int sem_ctime_high; + unsigned int sem_ctime; u32 sem_nsems; u32 __unused1; u32 __unused2; @@ -212,12 +203,12 @@ struct compat_semid64_ds { struct compat_msqid64_ds { struct compat_ipc64_perm msg_perm; - unsigned int __pad1; - compat_time_t msg_stime; - unsigned int __pad2; - compat_time_t msg_rtime; - unsigned int __pad3; - compat_time_t msg_ctime; + unsigned int msg_stime_high; + unsigned int msg_stime; + unsigned int msg_rtime_high; + unsigned int msg_rtime; + unsigned int msg_ctime_high; + unsigned int msg_ctime; unsigned int msg_cbytes; unsigned int msg_qnum; unsigned int msg_qbytes; @@ -229,12 +220,12 @@ struct compat_msqid64_ds { struct compat_shmid64_ds { struct compat_ipc64_perm shm_perm; - unsigned int __pad1; - compat_time_t shm_atime; - unsigned int __pad2; - compat_time_t shm_dtime; - unsigned int __pad3; - compat_time_t shm_ctime; + unsigned int shm_atime_high; + unsigned int shm_atime; + unsigned int shm_dtime_high; + unsigned int shm_dtime; + unsigned int shm_ctime_high; + unsigned int shm_ctime; compat_size_t shm_segsz; compat_pid_t shm_cpid; compat_pid_t shm_lpid; @@ -243,6 +234,7 @@ struct compat_shmid64_ds { unsigned int __unused2; }; +#ifdef CONFIG_COMPAT static inline int is_compat_task(void) { return test_thread_flag(TIF_32BIT); @@ -254,5 +246,6 @@ static inline bool in_compat_syscall(void) return pt_regs_trap_type(current_pt_regs()) == 0x110; } #define in_compat_syscall in_compat_syscall +#endif #endif /* _ASM_SPARC64_COMPAT_H */ diff --git a/arch/sparc/include/asm/hardirq_64.h b/arch/sparc/include/asm/hardirq_64.h index f56540271993..75b92bfe04b5 100644 --- a/arch/sparc/include/asm/hardirq_64.h +++ b/arch/sparc/include/asm/hardirq_64.h @@ -10,8 +10,9 @@ #include #define __ARCH_IRQ_STAT -#define local_softirq_pending() \ - (local_cpu_data().__softirq_pending) + +#define local_softirq_pending_ref \ + __cpu_data.__softirq_pending void ack_bad_irq(unsigned int irq); diff --git a/include/linux/iommu-common.h b/arch/sparc/include/asm/iommu-common.h similarity index 100% rename from include/linux/iommu-common.h rename to arch/sparc/include/asm/iommu-common.h diff --git a/arch/sparc/include/asm/iommu_64.h b/arch/sparc/include/asm/iommu_64.h index 9ed6b54caa4b..0ef6dedf747e 100644 --- a/arch/sparc/include/asm/iommu_64.h +++ b/arch/sparc/include/asm/iommu_64.h @@ -17,7 +17,7 @@ #define IOPTE_WRITE 0x0000000000000002UL #define IOMMU_NUM_CTXS 4096 -#include +#include struct iommu_arena { unsigned long *map; diff --git a/arch/sparc/include/asm/pci_32.h b/arch/sparc/include/asm/pci_32.h index 98917e48727d..cfc0ee9476c6 100644 --- a/arch/sparc/include/asm/pci_32.h +++ b/arch/sparc/include/asm/pci_32.h @@ -17,10 +17,6 @@ #define PCI_IRQ_NONE 0xffffffff -/* Dynamic DMA mapping stuff. - */ -#define PCI_DMA_BUS_IS_PHYS (0) - #endif /* __KERNEL__ */ #ifndef CONFIG_LEON_PCI diff --git a/arch/sparc/include/asm/pci_64.h b/arch/sparc/include/asm/pci_64.h index 671274e36cfa..fac77813402c 100644 --- a/arch/sparc/include/asm/pci_64.h +++ b/arch/sparc/include/asm/pci_64.h @@ -17,12 +17,6 @@ #define PCI_IRQ_NONE 0xffffffff -/* The PCI address space does not equal the physical memory - * address space. The networking and block device layers use - * this boolean for bounce buffer decisions. - */ -#define PCI_DMA_BUS_IS_PHYS (0) - /* PCI IOMMU mapping bypass support. */ /* PCI 64-bit addressing works for all slots on all controller diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 44d6ac47e035..1393a8ac596b 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -117,9 +117,6 @@ bool kern_addr_valid(unsigned long addr); #define _PAGE_PMD_HUGE _AC(0x0100000000000000,UL) /* Huge page */ #define _PAGE_PUD_HUGE _PAGE_PMD_HUGE -/* Advertise support for _PAGE_SPECIAL */ -#define __HAVE_ARCH_PTE_SPECIAL - /* SUN4U pte bits... */ #define _PAGE_SZ4MB_4U _AC(0x6000000000000000,UL) /* 4MB Page */ #define _PAGE_SZ512K_4U _AC(0x4000000000000000,UL) /* 512K Page */ diff --git a/arch/sparc/include/uapi/asm/jsflash.h b/arch/sparc/include/uapi/asm/jsflash.h deleted file mode 100644 index 68c98a54281a..000000000000 --- a/arch/sparc/include/uapi/asm/jsflash.h +++ /dev/null @@ -1,40 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -/* - * jsflash.h: OS Flash SIMM support for JavaStations. - * - * Copyright (C) 1999 Pete Zaitcev - */ - -#ifndef _SPARC_JSFLASH_H -#define _SPARC_JSFLASH_H - -#ifndef _SPARC_TYPES_H -#include -#endif - -/* - * Semantics of the offset is a full address. - * Hardcode it or get it from probe ioctl. - * - * We use full bus address, so that we would be - * automatically compatible with possible future systems. - */ - -#define JSFLASH_IDENT (('F'<<8)|54) -struct jsflash_ident_arg { - __u64 off; /* 0x20000000 is included */ - __u32 size; - char name[32]; /* With trailing zero */ -}; - -#define JSFLASH_ERASE (('F'<<8)|55) -/* Put 0 as argument, may be flags or sector number... */ - -#define JSFLASH_PROGRAM (('F'<<8)|56) -struct jsflash_program_arg { - __u64 data; /* char* for sparc and sparc64 */ - __u64 off; - __u32 size; -}; - -#endif /* _SPARC_JSFLASH_H */ diff --git a/arch/sparc/include/uapi/asm/msgbuf.h b/arch/sparc/include/uapi/asm/msgbuf.h index b601c4f4d956..ffc46c211d6d 100644 --- a/arch/sparc/include/uapi/asm/msgbuf.h +++ b/arch/sparc/include/uapi/asm/msgbuf.h @@ -8,25 +8,22 @@ * between kernel and user space. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem * - 2 miscellaneous 32-bit values */ - -#if defined(__sparc__) && defined(__arch64__) -# define PADDING(x) -#else -# define PADDING(x) unsigned int x; -#endif - - struct msqid64_ds { struct ipc64_perm msg_perm; - PADDING(__pad1) +#if defined(__sparc__) && defined(__arch64__) __kernel_time_t msg_stime; /* last msgsnd time */ - PADDING(__pad2) __kernel_time_t msg_rtime; /* last msgrcv time */ - PADDING(__pad3) __kernel_time_t msg_ctime; /* last change time */ +#else + unsigned long msg_stime_high; + unsigned long msg_stime; /* last msgsnd time */ + unsigned long msg_rtime_high; + unsigned long msg_rtime; /* last msgrcv time */ + unsigned long msg_ctime_high; + unsigned long msg_ctime; /* last change time */ +#endif unsigned long msg_cbytes; /* current number of bytes on queue */ unsigned long msg_qnum; /* number of messages in queue */ unsigned long msg_qbytes; /* max number of bytes on queue */ @@ -35,5 +32,4 @@ struct msqid64_ds { unsigned long __unused1; unsigned long __unused2; }; -#undef PADDING #endif /* _SPARC_MSGBUF_H */ diff --git a/arch/sparc/include/uapi/asm/sembuf.h b/arch/sparc/include/uapi/asm/sembuf.h index f49b0ffa0ab8..f3d309c2e1cd 100644 --- a/arch/sparc/include/uapi/asm/sembuf.h +++ b/arch/sparc/include/uapi/asm/sembuf.h @@ -8,25 +8,23 @@ * between kernel and user space. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem * - 2 miscellaneous 32-bit values */ -#if defined(__sparc__) && defined(__arch64__) -# define PADDING(x) -#else -# define PADDING(x) unsigned int x; -#endif struct semid64_ds { struct ipc64_perm sem_perm; /* permissions .. see ipc.h */ - PADDING(__pad1) +#if defined(__sparc__) && defined(__arch64__) __kernel_time_t sem_otime; /* last semop time */ - PADDING(__pad2) __kernel_time_t sem_ctime; /* last change time */ +#else + unsigned long sem_otime_high; + unsigned long sem_otime; /* last semop time */ + unsigned long sem_ctime_high; + unsigned long sem_ctime; /* last change time */ +#endif unsigned long sem_nsems; /* no. of semaphores in array */ unsigned long __unused1; unsigned long __unused2; }; -#undef PADDING #endif /* _SPARC64_SEMBUF_H */ diff --git a/arch/sparc/include/uapi/asm/shmbuf.h b/arch/sparc/include/uapi/asm/shmbuf.h index 286631db705c..06618b84822d 100644 --- a/arch/sparc/include/uapi/asm/shmbuf.h +++ b/arch/sparc/include/uapi/asm/shmbuf.h @@ -8,24 +8,23 @@ * between kernel and user space. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem * - 2 miscellaneous 32-bit values */ -#if defined(__sparc__) && defined(__arch64__) -# define PADDING(x) -#else -# define PADDING(x) unsigned int x; -#endif - struct shmid64_ds { struct ipc64_perm shm_perm; /* operation perms */ - PADDING(__pad1) +#if defined(__sparc__) && defined(__arch64__) __kernel_time_t shm_atime; /* last attach time */ - PADDING(__pad2) __kernel_time_t shm_dtime; /* last detach time */ - PADDING(__pad3) __kernel_time_t shm_ctime; /* last change time */ +#else + unsigned long shm_atime_high; + unsigned long shm_atime; /* last attach time */ + unsigned long shm_dtime_high; + unsigned long shm_dtime; /* last detach time */ + unsigned long shm_ctime_high; + unsigned long shm_ctime; /* last change time */ +#endif size_t shm_segsz; /* size of segment (bytes) */ __kernel_pid_t shm_cpid; /* pid of creator */ __kernel_pid_t shm_lpid; /* pid of last operator */ @@ -46,6 +45,4 @@ struct shminfo64 { unsigned long __unused4; }; -#undef PADDING - #endif /* _SPARC_SHMBUF_H */ diff --git a/arch/sparc/include/uapi/asm/siginfo.h b/arch/sparc/include/uapi/asm/siginfo.h index 896ce447d16a..e7049550ac82 100644 --- a/arch/sparc/include/uapi/asm/siginfo.h +++ b/arch/sparc/include/uapi/asm/siginfo.h @@ -17,13 +17,6 @@ #define SI_NOINFO 32767 /* no information in siginfo_t */ -/* - * SIGFPE si_codes - */ -#ifdef __KERNEL__ -#define FPE_FIXME 0 /* Broken dup of SI_USER */ -#endif /* __KERNEL__ */ - /* * SIGEMT si_codes */ diff --git a/arch/sparc/kernel/Makefile b/arch/sparc/kernel/Makefile index 76cb57750dda..cf8640841b7a 100644 --- a/arch/sparc/kernel/Makefile +++ b/arch/sparc/kernel/Makefile @@ -59,7 +59,7 @@ obj-$(CONFIG_SPARC32) += leon_pmc.o obj-$(CONFIG_SPARC64) += reboot.o obj-$(CONFIG_SPARC64) += sysfs.o -obj-$(CONFIG_SPARC64) += iommu.o +obj-$(CONFIG_SPARC64) += iommu.o iommu-common.o obj-$(CONFIG_SPARC64) += central.o obj-$(CONFIG_SPARC64) += starfire.o obj-$(CONFIG_SPARC64) += power.o @@ -74,8 +74,6 @@ obj-$(CONFIG_SPARC64) += pcr.o obj-$(CONFIG_SPARC64) += nmi.o obj-$(CONFIG_SPARC64_SMP) += cpumap.o -obj-y += dma.o - obj-$(CONFIG_PCIC_PCI) += pcic.o obj-$(CONFIG_LEON_PCI) += leon_pci.o obj-$(CONFIG_SPARC_GRPCI2)+= leon_pci_grpci2.o diff --git a/arch/sparc/kernel/dma.c b/arch/sparc/kernel/dma.c deleted file mode 100644 index f73e7597c971..000000000000 --- a/arch/sparc/kernel/dma.c +++ /dev/null @@ -1,13 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -#include -#include -#include - -#define PREALLOC_DMA_DEBUG_ENTRIES (1 << 15) - -static int __init dma_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - return 0; -} -fs_initcall(dma_init); diff --git a/lib/iommu-common.c b/arch/sparc/kernel/iommu-common.c similarity index 98% rename from lib/iommu-common.c rename to arch/sparc/kernel/iommu-common.c index 55b00de106b5..59cb16691322 100644 --- a/lib/iommu-common.c +++ b/arch/sparc/kernel/iommu-common.c @@ -8,9 +8,9 @@ #include #include #include -#include #include #include +#include static unsigned long iommu_large_alloc = 15; @@ -93,7 +93,6 @@ void iommu_tbl_pool_init(struct iommu_map_table *iommu, p->hint = p->start; p->end = num_entries; } -EXPORT_SYMBOL(iommu_tbl_pool_init); unsigned long iommu_tbl_range_alloc(struct device *dev, struct iommu_map_table *iommu, @@ -224,7 +223,6 @@ bail: return n; } -EXPORT_SYMBOL(iommu_tbl_range_alloc); static struct iommu_pool *get_pool(struct iommu_map_table *tbl, unsigned long entry) @@ -264,4 +262,3 @@ void iommu_tbl_range_free(struct iommu_map_table *iommu, u64 dma_addr, bitmap_clear(iommu->map, entry, npages); spin_unlock_irqrestore(&(pool->lock), flags); } -EXPORT_SYMBOL(iommu_tbl_range_free); diff --git a/arch/sparc/kernel/iommu.c b/arch/sparc/kernel/iommu.c index b08dc3416f06..40d008b0bd3e 100644 --- a/arch/sparc/kernel/iommu.c +++ b/arch/sparc/kernel/iommu.c @@ -14,7 +14,7 @@ #include #include #include -#include +#include #ifdef CONFIG_PCI #include diff --git a/arch/sparc/kernel/ioport.c b/arch/sparc/kernel/ioport.c index 3bcef9ce74df..cca9134cfa7d 100644 --- a/arch/sparc/kernel/ioport.c +++ b/arch/sparc/kernel/ioport.c @@ -678,25 +678,14 @@ static int sparc_io_proc_show(struct seq_file *m, void *v) return 0; } - -static int sparc_io_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, sparc_io_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations sparc_io_proc_fops = { - .owner = THIS_MODULE, - .open = sparc_io_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; #endif /* CONFIG_PROC_FS */ static void register_proc_sparc_ioport(void) { #ifdef CONFIG_PROC_FS - proc_create_data("io_map", 0, NULL, &sparc_io_proc_fops, &sparc_iomap); - proc_create_data("dvma_map", 0, NULL, &sparc_io_proc_fops, &_sparc_dvma); + proc_create_single_data("io_map", 0, NULL, sparc_io_proc_show, + &sparc_iomap); + proc_create_single_data("dvma_map", 0, NULL, sparc_io_proc_show, + &_sparc_dvma); #endif } diff --git a/arch/sparc/kernel/ldc.c b/arch/sparc/kernel/ldc.c index 86b625f9d8dc..c0fa3ef6cf01 100644 --- a/arch/sparc/kernel/ldc.c +++ b/arch/sparc/kernel/ldc.c @@ -16,7 +16,7 @@ #include #include #include -#include +#include #include #include diff --git a/arch/sparc/kernel/leon_pci.c b/arch/sparc/kernel/leon_pci.c index 15b59169c535..e5e5ff6b9a5c 100644 --- a/arch/sparc/kernel/leon_pci.c +++ b/arch/sparc/kernel/leon_pci.c @@ -60,50 +60,30 @@ void leon_pci_init(struct platform_device *ofdev, struct leon_pci_info *info) pci_bus_add_devices(root_bus); } -void pcibios_fixup_bus(struct pci_bus *pbus) +int pcibios_enable_device(struct pci_dev *dev, int mask) { - struct pci_dev *dev; - int i, has_io, has_mem; - u16 cmd; + u16 cmd, oldcmd; + int i; - list_for_each_entry(dev, &pbus->devices, bus_list) { - /* - * We can not rely on that the bootloader has enabled I/O - * or memory access to PCI devices. Instead we enable it here - * if the device has BARs of respective type. - */ - has_io = has_mem = 0; - for (i = 0; i < PCI_ROM_RESOURCE; i++) { - unsigned long f = dev->resource[i].flags; - if (f & IORESOURCE_IO) - has_io = 1; - else if (f & IORESOURCE_MEM) - has_mem = 1; - } - /* ROM BARs are mapped into 32-bit memory space */ - if (dev->resource[PCI_ROM_RESOURCE].end != 0) { - dev->resource[PCI_ROM_RESOURCE].flags |= - IORESOURCE_ROM_ENABLE; - has_mem = 1; - } - pci_bus_read_config_word(pbus, dev->devfn, PCI_COMMAND, &cmd); - if (has_io && !(cmd & PCI_COMMAND_IO)) { -#ifdef CONFIG_PCI_DEBUG - printk(KERN_INFO "LEONPCI: Enabling I/O for dev %s\n", - pci_name(dev)); -#endif + pci_read_config_word(dev, PCI_COMMAND, &cmd); + oldcmd = cmd; + + for (i = 0; i < PCI_NUM_RESOURCES; i++) { + struct resource *res = &dev->resource[i]; + + /* Only set up the requested stuff */ + if (!(mask & (1<flags & IORESOURCE_IO) cmd |= PCI_COMMAND_IO; - pci_bus_write_config_word(pbus, dev->devfn, PCI_COMMAND, - cmd); - } - if (has_mem && !(cmd & PCI_COMMAND_MEMORY)) { -#ifdef CONFIG_PCI_DEBUG - printk(KERN_INFO "LEONPCI: Enabling MEMORY for dev" - "%s\n", pci_name(dev)); -#endif + if (res->flags & IORESOURCE_MEM) cmd |= PCI_COMMAND_MEMORY; - pci_bus_write_config_word(pbus, dev->devfn, PCI_COMMAND, - cmd); - } } + + if (cmd != oldcmd) { + pci_info(dev, "enabling device (%04x -> %04x)\n", oldcmd, cmd); + pci_write_config_word(dev, PCI_COMMAND, cmd); + } + return 0; } diff --git a/arch/sparc/kernel/nmi.c b/arch/sparc/kernel/nmi.c index 048ad783ea3f..8babbeb30adf 100644 --- a/arch/sparc/kernel/nmi.c +++ b/arch/sparc/kernel/nmi.c @@ -166,7 +166,8 @@ static int __init check_nmi_watchdog(void) if (!atomic_read(&nmi_active)) return 0; - prev_nmi_count = kmalloc(nr_cpu_ids * sizeof(unsigned int), GFP_KERNEL); + prev_nmi_count = kmalloc_array(nr_cpu_ids, sizeof(unsigned int), + GFP_KERNEL); if (!prev_nmi_count) { err = -ENOMEM; goto error; diff --git a/arch/sparc/kernel/pci.c b/arch/sparc/kernel/pci.c index 41b20edb427d..17ea16a1337c 100644 --- a/arch/sparc/kernel/pci.c +++ b/arch/sparc/kernel/pci.c @@ -214,8 +214,8 @@ static void pci_parse_of_addrs(struct platform_device *op, if (!addrs) return; if (ofpci_verbose) - printk(" parse addresses (%d bytes) @ %p\n", - proplen, addrs); + pci_info(dev, " parse addresses (%d bytes) @ %p\n", + proplen, addrs); op_res = &op->resource[0]; for (; proplen >= 20; proplen -= 20, addrs += 5, op_res++) { struct resource *res; @@ -227,8 +227,8 @@ static void pci_parse_of_addrs(struct platform_device *op, continue; i = addrs[0] & 0xff; if (ofpci_verbose) - printk(" start: %llx, end: %llx, i: %x\n", - op_res->start, op_res->end, i); + pci_info(dev, " start: %llx, end: %llx, i: %x\n", + op_res->start, op_res->end, i); if (PCI_BASE_ADDRESS_0 <= i && i <= PCI_BASE_ADDRESS_5) { res = &dev->resource[(i - PCI_BASE_ADDRESS_0) >> 2]; @@ -236,13 +236,15 @@ static void pci_parse_of_addrs(struct platform_device *op, res = &dev->resource[PCI_ROM_RESOURCE]; flags |= IORESOURCE_READONLY | IORESOURCE_SIZEALIGN; } else { - printk(KERN_ERR "PCI: bad cfg reg num 0x%x\n", i); + pci_err(dev, "bad cfg reg num 0x%x\n", i); continue; } res->start = op_res->start; res->end = op_res->end; res->flags = flags; res->name = pci_name(dev); + + pci_info(dev, "reg 0x%x: %pR\n", i, res); } } @@ -289,8 +291,8 @@ static struct pci_dev *of_create_pci_dev(struct pci_pbm_info *pbm, type = ""; if (ofpci_verbose) - printk(" create device, devfn: %x, type: %s\n", - devfn, type); + pci_info(bus," create device, devfn: %x, type: %s\n", + devfn, type); dev->sysdata = node; dev->dev.parent = bus->bridge; @@ -323,10 +325,6 @@ static struct pci_dev *of_create_pci_dev(struct pci_pbm_info *pbm, dev_set_name(&dev->dev, "%04x:%02x:%02x.%d", pci_domain_nr(bus), dev->bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn)); - if (ofpci_verbose) - printk(" class: 0x%x device name: %s\n", - dev->class, pci_name(dev)); - /* I have seen IDE devices which will not respond to * the bmdma simplex check reads if bus mastering is * disabled. @@ -353,10 +351,13 @@ static struct pci_dev *of_create_pci_dev(struct pci_pbm_info *pbm, dev->irq = PCI_IRQ_NONE; } + pci_info(dev, "[%04x:%04x] type %02x class %#08x\n", + dev->vendor, dev->device, dev->hdr_type, dev->class); + pci_parse_of_addrs(sd->op, node, dev); if (ofpci_verbose) - printk(" adding to system ...\n"); + pci_info(dev, " adding to system ...\n"); pci_device_add(dev, bus); @@ -430,19 +431,19 @@ static void of_scan_pci_bridge(struct pci_pbm_info *pbm, u64 size; if (ofpci_verbose) - printk("of_scan_pci_bridge(%s)\n", node->full_name); + pci_info(dev, "of_scan_pci_bridge(%s)\n", node->full_name); /* parse bus-range property */ busrange = of_get_property(node, "bus-range", &len); if (busrange == NULL || len != 8) { - printk(KERN_DEBUG "Can't get bus-range for PCI-PCI bridge %s\n", + pci_info(dev, "Can't get bus-range for PCI-PCI bridge %s\n", node->full_name); return; } if (ofpci_verbose) - printk(" Bridge bus range [%u --> %u]\n", - busrange[0], busrange[1]); + pci_info(dev, " Bridge bus range [%u --> %u]\n", + busrange[0], busrange[1]); ranges = of_get_property(node, "ranges", &len); simba = 0; @@ -454,8 +455,8 @@ static void of_scan_pci_bridge(struct pci_pbm_info *pbm, bus = pci_add_new_bus(dev->bus, dev, busrange[0]); if (!bus) { - printk(KERN_ERR "Failed to create pci bus for %s\n", - node->full_name); + pci_err(dev, "Failed to create pci bus for %s\n", + node->full_name); return; } @@ -464,8 +465,8 @@ static void of_scan_pci_bridge(struct pci_pbm_info *pbm, bus->bridge_ctl = 0; if (ofpci_verbose) - printk(" Bridge ranges[%p] simba[%d]\n", - ranges, simba); + pci_info(dev, " Bridge ranges[%p] simba[%d]\n", + ranges, simba); /* parse ranges property, or cook one up by hand for Simba */ /* PCI #address-cells == 3 and #size-cells == 2 always */ @@ -487,10 +488,10 @@ static void of_scan_pci_bridge(struct pci_pbm_info *pbm, u64 start; if (ofpci_verbose) - printk(" RAW Range[%08x:%08x:%08x:%08x:%08x:%08x:" - "%08x:%08x]\n", - ranges[0], ranges[1], ranges[2], ranges[3], - ranges[4], ranges[5], ranges[6], ranges[7]); + pci_info(dev, " RAW Range[%08x:%08x:%08x:%08x:%08x:%08x:" + "%08x:%08x]\n", + ranges[0], ranges[1], ranges[2], ranges[3], + ranges[4], ranges[5], ranges[6], ranges[7]); flags = pci_parse_of_flags(ranges[0]); size = GET_64BIT(ranges, 6); @@ -510,14 +511,14 @@ static void of_scan_pci_bridge(struct pci_pbm_info *pbm, if (flags & IORESOURCE_IO) { res = bus->resource[0]; if (res->flags) { - printk(KERN_ERR "PCI: ignoring extra I/O range" - " for bridge %s\n", node->full_name); + pci_err(dev, "ignoring extra I/O range" + " for bridge %s\n", node->full_name); continue; } } else { if (i >= PCI_NUM_RESOURCES - PCI_BRIDGE_RESOURCES) { - printk(KERN_ERR "PCI: too many memory ranges" - " for bridge %s\n", node->full_name); + pci_err(dev, "too many memory ranges" + " for bridge %s\n", node->full_name); continue; } res = bus->resource[i]; @@ -529,8 +530,8 @@ static void of_scan_pci_bridge(struct pci_pbm_info *pbm, region.end = region.start + size - 1; if (ofpci_verbose) - printk(" Using flags[%08x] start[%016llx] size[%016llx]\n", - flags, start, size); + pci_info(dev, " Using flags[%08x] start[%016llx] size[%016llx]\n", + flags, start, size); pcibios_bus_to_resource(dev->bus, res, ®ion); } @@ -538,7 +539,7 @@ after_ranges: sprintf(bus->name, "PCI Bus %04x:%02x", pci_domain_nr(bus), bus->number); if (ofpci_verbose) - printk(" bus name: %s\n", bus->name); + pci_info(dev, " bus name: %s\n", bus->name); pci_of_scan_bus(pbm, node, bus); } @@ -553,14 +554,14 @@ static void pci_of_scan_bus(struct pci_pbm_info *pbm, struct pci_dev *dev; if (ofpci_verbose) - printk("PCI: scan_bus[%s] bus no %d\n", - node->full_name, bus->number); + pci_info(bus, "scan_bus[%s] bus no %d\n", + node->full_name, bus->number); child = NULL; prev_devfn = -1; while ((child = of_get_next_child(node, child)) != NULL) { if (ofpci_verbose) - printk(" * %s\n", child->full_name); + pci_info(bus, " * %s\n", child->full_name); reg = of_get_property(child, "reg", ®len); if (reg == NULL || reglen < 20) continue; @@ -581,8 +582,7 @@ static void pci_of_scan_bus(struct pci_pbm_info *pbm, if (!dev) continue; if (ofpci_verbose) - printk("PCI: dev header type: %x\n", - dev->hdr_type); + pci_info(dev, "dev header type: %x\n", dev->hdr_type); if (pci_is_bridge(dev)) of_scan_pci_bridge(pbm, child, dev); @@ -624,6 +624,45 @@ static void pci_bus_register_of_sysfs(struct pci_bus *bus) pci_bus_register_of_sysfs(child_bus); } +static void pci_claim_legacy_resources(struct pci_dev *dev) +{ + struct pci_bus_region region; + struct resource *p, *root, *conflict; + + if ((dev->class >> 8) != PCI_CLASS_DISPLAY_VGA) + return; + + p = kzalloc(sizeof(*p), GFP_KERNEL); + if (!p) + return; + + p->name = "Video RAM area"; + p->flags = IORESOURCE_MEM | IORESOURCE_BUSY; + + region.start = 0xa0000UL; + region.end = region.start + 0x1ffffUL; + pcibios_bus_to_resource(dev->bus, p, ®ion); + + root = pci_find_parent_resource(dev, p); + if (!root) { + pci_info(dev, "can't claim VGA legacy %pR: no compatible bridge window\n", p); + goto err; + } + + conflict = request_resource_conflict(root, p); + if (conflict) { + pci_info(dev, "can't claim VGA legacy %pR: address conflict with %s %pR\n", + p, conflict->name, conflict); + goto err; + } + + pci_info(dev, "VGA legacy framebuffer %pR\n", p); + return; + +err: + kfree(p); +} + static void pci_claim_bus_resources(struct pci_bus *bus) { struct pci_bus *child_bus; @@ -639,15 +678,13 @@ static void pci_claim_bus_resources(struct pci_bus *bus) continue; if (ofpci_verbose) - printk("PCI: Claiming %s: " - "Resource %d: %016llx..%016llx [%x]\n", - pci_name(dev), i, - (unsigned long long)r->start, - (unsigned long long)r->end, - (unsigned int)r->flags); + pci_info(dev, "Claiming Resource %d: %pR\n", + i, r); pci_claim_resource(dev, i); } + + pci_claim_legacy_resources(dev); } list_for_each_entry(child_bus, &bus->children, node) @@ -687,6 +724,7 @@ struct pci_bus *pci_scan_one_pbm(struct pci_pbm_info *pbm, pci_bus_register_of_sysfs(bus); pci_claim_bus_resources(bus); + pci_bus_add_devices(bus); return bus; } @@ -713,9 +751,7 @@ int pcibios_enable_device(struct pci_dev *dev, int mask) } if (cmd != oldcmd) { - printk(KERN_DEBUG "PCI: Enabling device: (%s), cmd %x\n", - pci_name(dev), cmd); - /* Enable the appropriate bits in the PCI command register. */ + pci_info(dev, "enabling device (%04x -> %04x)\n", oldcmd, cmd); pci_write_config_word(dev, PCI_COMMAND, cmd); } return 0; @@ -1075,8 +1111,8 @@ static void pci_bus_slot_names(struct device_node *node, struct pci_bus *bus) sp = prop->names; if (ofpci_verbose) - printk("PCI: Making slots for [%s] mask[0x%02x]\n", - node->full_name, mask); + pci_info(bus, "Making slots for [%s] mask[0x%02x]\n", + node->full_name, mask); i = 0; while (mask) { @@ -1089,12 +1125,12 @@ static void pci_bus_slot_names(struct device_node *node, struct pci_bus *bus) } if (ofpci_verbose) - printk("PCI: Making slot [%s]\n", sp); + pci_info(bus, "Making slot [%s]\n", sp); pci_slot = pci_create_slot(bus, i, sp, NULL); if (IS_ERR(pci_slot)) - printk(KERN_ERR "PCI: pci_create_slot returned %ld\n", - PTR_ERR(pci_slot)); + pci_err(bus, "pci_create_slot returned %ld\n", + PTR_ERR(pci_slot)); sp += strlen(sp) + 1; mask &= ~this_bit; diff --git a/arch/sparc/kernel/pci_common.c b/arch/sparc/kernel/pci_common.c index 38d46bcc8634..4759ccd542fe 100644 --- a/arch/sparc/kernel/pci_common.c +++ b/arch/sparc/kernel/pci_common.c @@ -329,23 +329,6 @@ void pci_get_pbm_props(struct pci_pbm_info *pbm) } } -static void pci_register_legacy_regions(struct resource *io_res, - struct resource *mem_res) -{ - struct resource *p; - - /* VGA Video RAM. */ - p = kzalloc(sizeof(*p), GFP_KERNEL); - if (!p) - return; - - p->name = "Video RAM area"; - p->start = mem_res->start + 0xa0000UL; - p->end = p->start + 0x1ffffUL; - p->flags = IORESOURCE_BUSY; - request_resource(mem_res, p); -} - static void pci_register_iommu_region(struct pci_pbm_info *pbm) { const u32 *vdma = of_get_property(pbm->op->dev.of_node, "virtual-dma", @@ -487,8 +470,6 @@ void pci_determine_mem_io_space(struct pci_pbm_info *pbm) if (pbm->mem64_space.flags) request_resource(&iomem_resource, &pbm->mem64_space); - pci_register_legacy_regions(&pbm->io_space, - &pbm->mem_space); pci_register_iommu_region(pbm); } @@ -508,8 +489,8 @@ void pci_scan_for_target_abort(struct pci_pbm_info *pbm, PCI_STATUS_REC_TARGET_ABORT)); if (error_bits) { pci_write_config_word(pdev, PCI_STATUS, error_bits); - printk("%s: Device %s saw Target Abort [%016x]\n", - pbm->name, pci_name(pdev), status); + pci_info(pdev, "%s: Device saw Target Abort [%016x]\n", + pbm->name, status); } } @@ -531,8 +512,8 @@ void pci_scan_for_master_abort(struct pci_pbm_info *pbm, (status & (PCI_STATUS_REC_MASTER_ABORT)); if (error_bits) { pci_write_config_word(pdev, PCI_STATUS, error_bits); - printk("%s: Device %s received Master Abort [%016x]\n", - pbm->name, pci_name(pdev), status); + pci_info(pdev, "%s: Device received Master Abort " + "[%016x]\n", pbm->name, status); } } @@ -555,8 +536,8 @@ void pci_scan_for_parity_error(struct pci_pbm_info *pbm, PCI_STATUS_DETECTED_PARITY)); if (error_bits) { pci_write_config_word(pdev, PCI_STATUS, error_bits); - printk("%s: Device %s saw Parity Error [%016x]\n", - pbm->name, pci_name(pdev), status); + pci_info(pdev, "%s: Device saw Parity Error [%016x]\n", + pbm->name, status); } } diff --git a/arch/sparc/kernel/pci_msi.c b/arch/sparc/kernel/pci_msi.c index 1994d7638406..fb5899cbfa51 100644 --- a/arch/sparc/kernel/pci_msi.c +++ b/arch/sparc/kernel/pci_msi.c @@ -191,8 +191,8 @@ static void sparc64_teardown_msi_irq(unsigned int irq, break; } if (i >= pbm->msi_num) { - printk(KERN_ERR "%s: teardown: No MSI for irq %u\n", - pbm->name, irq); + pci_err(pdev, "%s: teardown: No MSI for irq %u\n", pbm->name, + irq); return; } @@ -201,9 +201,9 @@ static void sparc64_teardown_msi_irq(unsigned int irq, err = ops->msi_teardown(pbm, msi_num); if (err) { - printk(KERN_ERR "%s: teardown: ops->teardown() on MSI %u, " - "irq %u, gives error %d\n", - pbm->name, msi_num, irq, err); + pci_err(pdev, "%s: teardown: ops->teardown() on MSI %u, " + "irq %u, gives error %d\n", pbm->name, msi_num, irq, + err); return; } diff --git a/arch/sparc/kernel/pci_sun4v.c b/arch/sparc/kernel/pci_sun4v.c index 249367228c33..565d9ac883d0 100644 --- a/arch/sparc/kernel/pci_sun4v.c +++ b/arch/sparc/kernel/pci_sun4v.c @@ -16,7 +16,7 @@ #include #include #include -#include +#include #include #include diff --git a/arch/sparc/kernel/pcic.c b/arch/sparc/kernel/pcic.c index 22f8774977d5..ee4c9a9a171c 100644 --- a/arch/sparc/kernel/pcic.c +++ b/arch/sparc/kernel/pcic.c @@ -518,10 +518,10 @@ static void pcic_map_pci_device(struct linux_pcic *pcic, * board in a PCI slot. We must remap it * under 64K but it is not done yet. XXX */ - printk("PCIC: Skipping I/O space at 0x%lx, " - "this will Oops if a driver attaches " - "device '%s' at %02x:%02x)\n", address, - namebuf, dev->bus->number, dev->devfn); + pci_info(dev, "PCIC: Skipping I/O space at " + "0x%lx, this will Oops if a driver " + "attaches device '%s'\n", address, + namebuf); } } } @@ -551,8 +551,8 @@ pcic_fill_irq(struct linux_pcic *pcic, struct pci_dev *dev, int node) p++; } if (i >= pcic->pcic_imdim) { - printk("PCIC: device %s devfn %02x:%02x not found in %d\n", - namebuf, dev->bus->number, dev->devfn, pcic->pcic_imdim); + pci_info(dev, "PCIC: device %s not found in %d\n", namebuf, + pcic->pcic_imdim); dev->irq = 0; return; } @@ -565,7 +565,7 @@ pcic_fill_irq(struct linux_pcic *pcic, struct pci_dev *dev, int node) ivec = readw(pcic->pcic_regs+PCI_INT_SELECT_HI); real_irq = ivec >> ((i-4) << 2) & 0xF; } else { /* Corrupted map */ - printk("PCIC: BAD PIN %d\n", i); for (;;) {} + pci_info(dev, "PCIC: BAD PIN %d\n", i); for (;;) {} } /* P3 */ /* printk("PCIC: device %s pin %d ivec 0x%x irq %x\n", namebuf, i, ivec, dev->irq); */ @@ -574,10 +574,10 @@ pcic_fill_irq(struct linux_pcic *pcic, struct pci_dev *dev, int node) */ if (real_irq == 0 || p->force) { if (p->irq == 0 || p->irq >= 15) { /* Corrupted map */ - printk("PCIC: BAD IRQ %d\n", p->irq); for (;;) {} + pci_info(dev, "PCIC: BAD IRQ %d\n", p->irq); for (;;) {} } - printk("PCIC: setting irq %d at pin %d for device %02x:%02x\n", - p->irq, p->pin, dev->bus->number, dev->devfn); + pci_info(dev, "PCIC: setting irq %d at pin %d\n", p->irq, + p->pin); real_irq = p->irq; i = p->pin; @@ -602,15 +602,13 @@ pcic_fill_irq(struct linux_pcic *pcic, struct pci_dev *dev, int node) void pcibios_fixup_bus(struct pci_bus *bus) { struct pci_dev *dev; - int i, has_io, has_mem; - unsigned int cmd = 0; struct linux_pcic *pcic; /* struct linux_pbm_info* pbm = &pcic->pbm; */ int node; struct pcidev_cookie *pcp; if (!pcic0_up) { - printk("pcibios_fixup_bus: no PCIC\n"); + pci_info(bus, "pcibios_fixup_bus: no PCIC\n"); return; } pcic = &pcic0; @@ -619,44 +617,12 @@ void pcibios_fixup_bus(struct pci_bus *bus) * Next crud is an equivalent of pbm = pcic_bus_to_pbm(bus); */ if (bus->number != 0) { - printk("pcibios_fixup_bus: nonzero bus 0x%x\n", bus->number); + pci_info(bus, "pcibios_fixup_bus: nonzero bus 0x%x\n", + bus->number); return; } list_for_each_entry(dev, &bus->devices, bus_list) { - - /* - * Comment from i386 branch: - * There are buggy BIOSes that forget to enable I/O and memory - * access to PCI devices. We try to fix this, but we need to - * be sure that the BIOS didn't forget to assign an address - * to the device. [mj] - * OBP is a case of such BIOS :-) - */ - has_io = has_mem = 0; - for(i=0; i<6; i++) { - unsigned long f = dev->resource[i].flags; - if (f & IORESOURCE_IO) { - has_io = 1; - } else if (f & IORESOURCE_MEM) - has_mem = 1; - } - pcic_read_config(dev->bus, dev->devfn, PCI_COMMAND, 2, &cmd); - if (has_io && !(cmd & PCI_COMMAND_IO)) { - printk("PCIC: Enabling I/O for device %02x:%02x\n", - dev->bus->number, dev->devfn); - cmd |= PCI_COMMAND_IO; - pcic_write_config(dev->bus, dev->devfn, - PCI_COMMAND, 2, cmd); - } - if (has_mem && !(cmd & PCI_COMMAND_MEMORY)) { - printk("PCIC: Enabling memory for device %02x:%02x\n", - dev->bus->number, dev->devfn); - cmd |= PCI_COMMAND_MEMORY; - pcic_write_config(dev->bus, dev->devfn, - PCI_COMMAND, 2, cmd); - } - node = pdev_to_pnode(&pcic->pbm, dev); if(node == 0) node = -1; @@ -675,6 +641,34 @@ void pcibios_fixup_bus(struct pci_bus *bus) } } +int pcibios_enable_device(struct pci_dev *dev, int mask) +{ + u16 cmd, oldcmd; + int i; + + pci_read_config_word(dev, PCI_COMMAND, &cmd); + oldcmd = cmd; + + for (i = 0; i < PCI_NUM_RESOURCES; i++) { + struct resource *res = &dev->resource[i]; + + /* Only set up the requested stuff */ + if (!(mask & (1<flags & IORESOURCE_IO) + cmd |= PCI_COMMAND_IO; + if (res->flags & IORESOURCE_MEM) + cmd |= PCI_COMMAND_MEMORY; + } + + if (cmd != oldcmd) { + pci_info(dev, "enabling device (%04x -> %04x)\n", oldcmd, cmd); + pci_write_config_word(dev, PCI_COMMAND, cmd); + } + return 0; +} + /* Makes compiler happy */ static volatile int pcic_timer_dummy; @@ -747,17 +741,11 @@ static void watchdog_reset() { } #endif -int pcibios_enable_device(struct pci_dev *pdev, int mask) -{ - return 0; -} - /* * NMI */ void pcic_nmi(unsigned int pend, struct pt_regs *regs) { - pend = swab32(pend); if (!pcic_speculative || (pend & PCI_SYS_INT_PENDING_PIO) == 0) { diff --git a/arch/sparc/kernel/process_64.c b/arch/sparc/kernel/process_64.c index 454a8af28f13..6c086086ca8f 100644 --- a/arch/sparc/kernel/process_64.c +++ b/arch/sparc/kernel/process_64.c @@ -518,14 +518,7 @@ void synchronize_user_stack(void) static void stack_unaligned(unsigned long sp) { - siginfo_t info; - - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRALN; - info.si_addr = (void __user *) sp; - info.si_trapno = 0; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGBUS, BUS_ADRALN, (void __user *) sp, 0, current); } void fault_in_user_windows(void) diff --git a/arch/sparc/kernel/sys_sparc_32.c b/arch/sparc/kernel/sys_sparc_32.c index e8c3cb6b6d08..7f3d9c59719a 100644 --- a/arch/sparc/kernel/sys_sparc_32.c +++ b/arch/sparc/kernel/sys_sparc_32.c @@ -147,17 +147,11 @@ SYSCALL_DEFINE0(nis_syscall) asmlinkage void sparc_breakpoint (struct pt_regs *regs) { - siginfo_t info; #ifdef DEBUG_SPARC_BREAKPOINT printk ("TRAP: Entering kernel PC=%x, nPC=%x\n", regs->pc, regs->npc); #endif - info.si_signo = SIGTRAP; - info.si_errno = 0; - info.si_code = TRAP_BRKPT; - info.si_addr = (void __user *)regs->pc; - info.si_trapno = 0; - force_sig_info(SIGTRAP, &info, current); + force_sig_fault(SIGTRAP, TRAP_BRKPT, (void __user *)regs->pc, 0, current); #ifdef DEBUG_SPARC_BREAKPOINT printk ("TRAP: Returning to space: PC=%x nPC=%x\n", regs->pc, regs->npc); diff --git a/arch/sparc/kernel/sys_sparc_64.c b/arch/sparc/kernel/sys_sparc_64.c index 9ef8de63f28b..63baa8aa9414 100644 --- a/arch/sparc/kernel/sys_sparc_64.c +++ b/arch/sparc/kernel/sys_sparc_64.c @@ -502,7 +502,6 @@ SYSCALL_DEFINE0(nis_syscall) asmlinkage void sparc_breakpoint(struct pt_regs *regs) { enum ctx_state prev_state = exception_enter(); - siginfo_t info; if (test_thread_flag(TIF_32BIT)) { regs->tpc &= 0xffffffff; @@ -511,12 +510,7 @@ asmlinkage void sparc_breakpoint(struct pt_regs *regs) #ifdef DEBUG_SPARC_BREAKPOINT printk ("TRAP: Entering kernel PC=%lx, nPC=%lx\n", regs->tpc, regs->tnpc); #endif - info.si_signo = SIGTRAP; - info.si_errno = 0; - info.si_code = TRAP_BRKPT; - info.si_addr = (void __user *)regs->tpc; - info.si_trapno = 0; - force_sig_info(SIGTRAP, &info, current); + force_sig_fault(SIGTRAP, TRAP_BRKPT, (void __user *)regs->tpc, 0, current); #ifdef DEBUG_SPARC_BREAKPOINT printk ("TRAP: Returning to space: PC=%lx nPC=%lx\n", regs->tpc, regs->tnpc); #endif @@ -571,7 +565,8 @@ SYSCALL_DEFINE5(utrap_install, utrap_entry_t, type, } if (!current_thread_info()->utraps) { current_thread_info()->utraps = - kzalloc((UT_TRAP_INSTRUCTION_31+1)*sizeof(long), GFP_KERNEL); + kcalloc(UT_TRAP_INSTRUCTION_31 + 1, sizeof(long), + GFP_KERNEL); if (!current_thread_info()->utraps) return -ENOMEM; current_thread_info()->utraps[0] = 1; @@ -581,8 +576,9 @@ SYSCALL_DEFINE5(utrap_install, utrap_entry_t, type, unsigned long *p = current_thread_info()->utraps; current_thread_info()->utraps = - kmalloc((UT_TRAP_INSTRUCTION_31+1)*sizeof(long), - GFP_KERNEL); + kmalloc_array(UT_TRAP_INSTRUCTION_31 + 1, + sizeof(long), + GFP_KERNEL); if (!current_thread_info()->utraps) { current_thread_info()->utraps = p; return -ENOMEM; diff --git a/arch/sparc/kernel/traps_32.c b/arch/sparc/kernel/traps_32.c index b1ed763e4787..bcdfc6168dd5 100644 --- a/arch/sparc/kernel/traps_32.c +++ b/arch/sparc/kernel/traps_32.c @@ -93,8 +93,6 @@ void __noreturn die_if_kernel(char *str, struct pt_regs *regs) void do_hw_interrupt(struct pt_regs *regs, unsigned long type) { - siginfo_t info; - if(type < 0x80) { /* Sun OS's puke from bad traps, Linux survives! */ printk("Unimplemented Sparc TRAP, type = %02lx\n", type); @@ -104,19 +102,13 @@ void do_hw_interrupt(struct pt_regs *regs, unsigned long type) if(regs->psr & PSR_PS) die_if_kernel("Kernel bad trap", regs); - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_ILLTRP; - info.si_addr = (void __user *)regs->pc; - info.si_trapno = type - 0x80; - force_sig_info(SIGILL, &info, current); + force_sig_fault(SIGILL, ILL_ILLTRP, + (void __user *)regs->pc, type - 0x80, current); } void do_illegal_instruction(struct pt_regs *regs, unsigned long pc, unsigned long npc, unsigned long psr) { - siginfo_t info; - if(psr & PSR_PS) die_if_kernel("Kernel illegal instruction", regs); #ifdef TRAP_DEBUG @@ -124,27 +116,15 @@ void do_illegal_instruction(struct pt_regs *regs, unsigned long pc, unsigned lon regs->pc, *(unsigned long *)regs->pc); #endif - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_ILLOPC; - info.si_addr = (void __user *)pc; - info.si_trapno = 0; - send_sig_info(SIGILL, &info, current); + send_sig_fault(SIGILL, ILL_ILLOPC, (void __user *)pc, 0, current); } void do_priv_instruction(struct pt_regs *regs, unsigned long pc, unsigned long npc, unsigned long psr) { - siginfo_t info; - if(psr & PSR_PS) die_if_kernel("Penguin instruction from Penguin mode??!?!", regs); - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_PRVOPC; - info.si_addr = (void __user *)pc; - info.si_trapno = 0; - send_sig_info(SIGILL, &info, current); + send_sig_fault(SIGILL, ILL_PRVOPC, (void __user *)pc, 0, current); } /* XXX User may want to be allowed to do this. XXX */ @@ -152,8 +132,6 @@ void do_priv_instruction(struct pt_regs *regs, unsigned long pc, unsigned long n void do_memaccess_unaligned(struct pt_regs *regs, unsigned long pc, unsigned long npc, unsigned long psr) { - siginfo_t info; - if(regs->psr & PSR_PS) { printk("KERNEL MNA at pc %08lx npc %08lx called by %08lx\n", pc, npc, regs->u_regs[UREG_RETPC]); @@ -165,12 +143,9 @@ void do_memaccess_unaligned(struct pt_regs *regs, unsigned long pc, unsigned lon instruction_dump ((unsigned long *) regs->pc); printk ("do_MNA!\n"); #endif - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRALN; - info.si_addr = /* FIXME: Should dig out mna address */ (void *)0; - info.si_trapno = 0; - send_sig_info(SIGBUS, &info, current); + send_sig_fault(SIGBUS, BUS_ADRALN, + /* FIXME: Should dig out mna address */ (void *)0, + 0, current); } static unsigned long init_fsr = 0x0UL; @@ -226,9 +201,9 @@ void do_fpe_trap(struct pt_regs *regs, unsigned long pc, unsigned long npc, unsigned long psr) { static int calls; - siginfo_t info; unsigned long fsr; int ret = 0; + int code; #ifndef CONFIG_SMP struct task_struct *fpt = last_task_used_math; #else @@ -303,24 +278,20 @@ void do_fpe_trap(struct pt_regs *regs, unsigned long pc, unsigned long npc, } fsr = fpt->thread.fsr; - info.si_signo = SIGFPE; - info.si_errno = 0; - info.si_addr = (void __user *)pc; - info.si_trapno = 0; - info.si_code = FPE_FIXME; + code = FPE_FLTUNK; if ((fsr & 0x1c000) == (1 << 14)) { if (fsr & 0x10) - info.si_code = FPE_FLTINV; + code = FPE_FLTINV; else if (fsr & 0x08) - info.si_code = FPE_FLTOVF; + code = FPE_FLTOVF; else if (fsr & 0x04) - info.si_code = FPE_FLTUND; + code = FPE_FLTUND; else if (fsr & 0x02) - info.si_code = FPE_FLTDIV; + code = FPE_FLTDIV; else if (fsr & 0x01) - info.si_code = FPE_FLTRES; + code = FPE_FLTRES; } - send_sig_info(SIGFPE, &info, fpt); + send_sig_fault(SIGFPE, code, (void __user *)pc, 0, fpt); #ifndef CONFIG_SMP last_task_used_math = NULL; #endif @@ -332,16 +303,9 @@ void do_fpe_trap(struct pt_regs *regs, unsigned long pc, unsigned long npc, void handle_tag_overflow(struct pt_regs *regs, unsigned long pc, unsigned long npc, unsigned long psr) { - siginfo_t info; - if(psr & PSR_PS) die_if_kernel("Penguin overflow trap from kernel mode", regs); - info.si_signo = SIGEMT; - info.si_errno = 0; - info.si_code = EMT_TAGOVF; - info.si_addr = (void __user *)pc; - info.si_trapno = 0; - send_sig_info(SIGEMT, &info, current); + send_sig_fault(SIGEMT, EMT_TAGOVF, (void __user *)pc, 0, current); } void handle_watchpoint(struct pt_regs *regs, unsigned long pc, unsigned long npc, @@ -359,61 +323,33 @@ void handle_watchpoint(struct pt_regs *regs, unsigned long pc, unsigned long npc void handle_reg_access(struct pt_regs *regs, unsigned long pc, unsigned long npc, unsigned long psr) { - siginfo_t info; - #ifdef TRAP_DEBUG printk("Register Access Exception at PC %08lx NPC %08lx PSR %08lx\n", pc, npc, psr); #endif - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_OBJERR; - info.si_addr = (void __user *)pc; - info.si_trapno = 0; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGBUS, BUS_OBJERR, (void __user *)pc, 0, current); } void handle_cp_disabled(struct pt_regs *regs, unsigned long pc, unsigned long npc, unsigned long psr) { - siginfo_t info; - - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_COPROC; - info.si_addr = (void __user *)pc; - info.si_trapno = 0; - send_sig_info(SIGILL, &info, current); + send_sig_fault(SIGILL, ILL_COPROC, (void __user *)pc, 0, current); } void handle_cp_exception(struct pt_regs *regs, unsigned long pc, unsigned long npc, unsigned long psr) { - siginfo_t info; - #ifdef TRAP_DEBUG printk("Co-Processor Exception at PC %08lx NPC %08lx PSR %08lx\n", pc, npc, psr); #endif - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_COPROC; - info.si_addr = (void __user *)pc; - info.si_trapno = 0; - send_sig_info(SIGILL, &info, current); + send_sig_fault(SIGILL, ILL_COPROC, (void __user *)pc, 0, current); } void handle_hw_divzero(struct pt_regs *regs, unsigned long pc, unsigned long npc, unsigned long psr) { - siginfo_t info; - - info.si_signo = SIGFPE; - info.si_errno = 0; - info.si_code = FPE_INTDIV; - info.si_addr = (void __user *)pc; - info.si_trapno = 0; - send_sig_info(SIGFPE, &info, current); + send_sig_fault(SIGFPE, FPE_INTDIV, (void __user *)pc, 0, current); } #ifdef CONFIG_DEBUG_BUGVERBOSE diff --git a/arch/sparc/kernel/traps_64.c b/arch/sparc/kernel/traps_64.c index 462a21abd105..aa624ed79db1 100644 --- a/arch/sparc/kernel/traps_64.c +++ b/arch/sparc/kernel/traps_64.c @@ -87,7 +87,6 @@ static void dump_tl1_traplog(struct tl1_traplog *p) void bad_trap(struct pt_regs *regs, long lvl) { char buffer[36]; - siginfo_t info; if (notify_die(DIE_TRAP, "bad trap", regs, 0, lvl, SIGTRAP) == NOTIFY_STOP) @@ -107,12 +106,8 @@ void bad_trap(struct pt_regs *regs, long lvl) regs->tpc &= 0xffffffff; regs->tnpc &= 0xffffffff; } - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_ILLTRP; - info.si_addr = (void __user *)regs->tpc; - info.si_trapno = lvl; - force_sig_info(SIGILL, &info, current); + force_sig_fault(SIGILL, ILL_ILLTRP, + (void __user *)regs->tpc, lvl, current); } void bad_trap_tl1(struct pt_regs *regs, long lvl) @@ -191,7 +186,6 @@ EXPORT_SYMBOL_GPL(unregister_dimm_printer); void spitfire_insn_access_exception(struct pt_regs *regs, unsigned long sfsr, unsigned long sfar) { enum ctx_state prev_state = exception_enter(); - siginfo_t info; if (notify_die(DIE_TRAP, "instruction access exception", regs, 0, 0x8, SIGTRAP) == NOTIFY_STOP) @@ -206,12 +200,8 @@ void spitfire_insn_access_exception(struct pt_regs *regs, unsigned long sfsr, un regs->tpc &= 0xffffffff; regs->tnpc &= 0xffffffff; } - info.si_signo = SIGSEGV; - info.si_errno = 0; - info.si_code = SEGV_MAPERR; - info.si_addr = (void __user *)regs->tpc; - info.si_trapno = 0; - force_sig_info(SIGSEGV, &info, current); + force_sig_fault(SIGSEGV, SEGV_MAPERR, + (void __user *)regs->tpc, 0, current); out: exception_exit(prev_state); } @@ -230,7 +220,6 @@ void sun4v_insn_access_exception(struct pt_regs *regs, unsigned long addr, unsig { unsigned short type = (type_ctx >> 16); unsigned short ctx = (type_ctx & 0xffff); - siginfo_t info; if (notify_die(DIE_TRAP, "instruction access exception", regs, 0, 0x8, SIGTRAP) == NOTIFY_STOP) @@ -247,12 +236,7 @@ void sun4v_insn_access_exception(struct pt_regs *regs, unsigned long addr, unsig regs->tpc &= 0xffffffff; regs->tnpc &= 0xffffffff; } - info.si_signo = SIGSEGV; - info.si_errno = 0; - info.si_code = SEGV_MAPERR; - info.si_addr = (void __user *) addr; - info.si_trapno = 0; - force_sig_info(SIGSEGV, &info, current); + force_sig_fault(SIGSEGV, SEGV_MAPERR, (void __user *) addr, 0, current); } void sun4v_insn_access_exception_tl1(struct pt_regs *regs, unsigned long addr, unsigned long type_ctx) @@ -307,7 +291,6 @@ bool is_no_fault_exception(struct pt_regs *regs) void spitfire_data_access_exception(struct pt_regs *regs, unsigned long sfsr, unsigned long sfar) { enum ctx_state prev_state = exception_enter(); - siginfo_t info; if (notify_die(DIE_TRAP, "data access exception", regs, 0, 0x30, SIGTRAP) == NOTIFY_STOP) @@ -338,12 +321,7 @@ void spitfire_data_access_exception(struct pt_regs *regs, unsigned long sfsr, un if (is_no_fault_exception(regs)) return; - info.si_signo = SIGSEGV; - info.si_errno = 0; - info.si_code = SEGV_MAPERR; - info.si_addr = (void __user *)sfar; - info.si_trapno = 0; - force_sig_info(SIGSEGV, &info, current); + force_sig_fault(SIGSEGV, SEGV_MAPERR, (void __user *)sfar, 0, current); out: exception_exit(prev_state); } @@ -559,8 +537,6 @@ static void spitfire_cee_log(unsigned long afsr, unsigned long afar, unsigned lo static void spitfire_ue_log(unsigned long afsr, unsigned long afar, unsigned long udbh, unsigned long udbl, unsigned long tt, int tl1, struct pt_regs *regs) { - siginfo_t info; - printk(KERN_WARNING "CPU[%d]: Uncorrectable Error AFSR[%lx] " "AFAR[%lx] UDBL[%lx] UDBH[%ld] TT[%lx] TL>1[%d]\n", smp_processor_id(), afsr, afar, udbl, udbh, tt, tl1); @@ -595,12 +571,7 @@ static void spitfire_ue_log(unsigned long afsr, unsigned long afar, unsigned lon regs->tpc &= 0xffffffff; regs->tnpc &= 0xffffffff; } - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_OBJERR; - info.si_addr = (void *)0; - info.si_trapno = 0; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGBUS, BUS_OBJERR, (void *)0, 0, current); } void spitfire_access_error(struct pt_regs *regs, unsigned long status_encoded, unsigned long afar) @@ -2190,7 +2161,6 @@ bool sun4v_nonresum_error_user_handled(struct pt_regs *regs, if (attrs & SUN4V_ERR_ATTRS_MEMORY) { unsigned long addr = ent->err_raddr; - siginfo_t info; if (addr == ~(u64)0) { /* This seems highly unlikely to ever occur */ @@ -2211,21 +2181,13 @@ bool sun4v_nonresum_error_user_handled(struct pt_regs *regs, addr += PAGE_SIZE; } } - info.si_signo = SIGKILL; - info.si_errno = 0; - info.si_trapno = 0; - force_sig_info(info.si_signo, &info, current); + force_sig(SIGKILL, current); return true; } if (attrs & SUN4V_ERR_ATTRS_PIO) { - siginfo_t info; - - info.si_signo = SIGBUS; - info.si_code = BUS_ADRERR; - info.si_addr = (void __user *)sun4v_get_vaddr(regs); - force_sig_info(info.si_signo, &info, current); - + force_sig_fault(SIGBUS, BUS_ADRERR, + (void __user *)sun4v_get_vaddr(regs), 0, current); return true; } @@ -2362,30 +2324,27 @@ static void do_fpe_common(struct pt_regs *regs) regs->tnpc += 4; } else { unsigned long fsr = current_thread_info()->xfsr[0]; - siginfo_t info; + int code; if (test_thread_flag(TIF_32BIT)) { regs->tpc &= 0xffffffff; regs->tnpc &= 0xffffffff; } - info.si_signo = SIGFPE; - info.si_errno = 0; - info.si_addr = (void __user *)regs->tpc; - info.si_trapno = 0; - info.si_code = FPE_FIXME; + code = FPE_FLTUNK; if ((fsr & 0x1c000) == (1 << 14)) { if (fsr & 0x10) - info.si_code = FPE_FLTINV; + code = FPE_FLTINV; else if (fsr & 0x08) - info.si_code = FPE_FLTOVF; + code = FPE_FLTOVF; else if (fsr & 0x04) - info.si_code = FPE_FLTUND; + code = FPE_FLTUND; else if (fsr & 0x02) - info.si_code = FPE_FLTDIV; + code = FPE_FLTDIV; else if (fsr & 0x01) - info.si_code = FPE_FLTRES; + code = FPE_FLTRES; } - force_sig_info(SIGFPE, &info, current); + force_sig_fault(SIGFPE, code, + (void __user *)regs->tpc, 0, current); } } @@ -2428,7 +2387,6 @@ out: void do_tof(struct pt_regs *regs) { enum ctx_state prev_state = exception_enter(); - siginfo_t info; if (notify_die(DIE_TRAP, "tagged arithmetic overflow", regs, 0, 0x26, SIGEMT) == NOTIFY_STOP) @@ -2440,12 +2398,8 @@ void do_tof(struct pt_regs *regs) regs->tpc &= 0xffffffff; regs->tnpc &= 0xffffffff; } - info.si_signo = SIGEMT; - info.si_errno = 0; - info.si_code = EMT_TAGOVF; - info.si_addr = (void __user *)regs->tpc; - info.si_trapno = 0; - force_sig_info(SIGEMT, &info, current); + force_sig_fault(SIGEMT, EMT_TAGOVF, + (void __user *)regs->tpc, 0, current); out: exception_exit(prev_state); } @@ -2453,7 +2407,6 @@ out: void do_div0(struct pt_regs *regs) { enum ctx_state prev_state = exception_enter(); - siginfo_t info; if (notify_die(DIE_TRAP, "integer division by zero", regs, 0, 0x28, SIGFPE) == NOTIFY_STOP) @@ -2465,12 +2418,8 @@ void do_div0(struct pt_regs *regs) regs->tpc &= 0xffffffff; regs->tnpc &= 0xffffffff; } - info.si_signo = SIGFPE; - info.si_errno = 0; - info.si_code = FPE_INTDIV; - info.si_addr = (void __user *)regs->tpc; - info.si_trapno = 0; - force_sig_info(SIGFPE, &info, current); + force_sig_fault(SIGFPE, FPE_INTDIV, + (void __user *)regs->tpc, 0, current); out: exception_exit(prev_state); } @@ -2632,7 +2581,6 @@ void do_illegal_instruction(struct pt_regs *regs) unsigned long pc = regs->tpc; unsigned long tstate = regs->tstate; u32 insn; - siginfo_t info; if (notify_die(DIE_TRAP, "illegal instruction", regs, 0, 0x10, SIGILL) == NOTIFY_STOP) @@ -2666,12 +2614,7 @@ void do_illegal_instruction(struct pt_regs *regs) } } } - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_ILLOPC; - info.si_addr = (void __user *)pc; - info.si_trapno = 0; - force_sig_info(SIGILL, &info, current); + force_sig_fault(SIGILL, ILL_ILLOPC, (void __user *)pc, 0, current); out: exception_exit(prev_state); } @@ -2679,7 +2622,6 @@ out: void mem_address_unaligned(struct pt_regs *regs, unsigned long sfar, unsigned long sfsr) { enum ctx_state prev_state = exception_enter(); - siginfo_t info; if (notify_die(DIE_TRAP, "memory address unaligned", regs, 0, 0x34, SIGSEGV) == NOTIFY_STOP) @@ -2692,20 +2634,13 @@ void mem_address_unaligned(struct pt_regs *regs, unsigned long sfar, unsigned lo if (is_no_fault_exception(regs)) return; - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRALN; - info.si_addr = (void __user *)sfar; - info.si_trapno = 0; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGBUS, BUS_ADRALN, (void __user *)sfar, 0, current); out: exception_exit(prev_state); } void sun4v_do_mna(struct pt_regs *regs, unsigned long addr, unsigned long type_ctx) { - siginfo_t info; - if (notify_die(DIE_TRAP, "memory address unaligned", regs, 0, 0x34, SIGSEGV) == NOTIFY_STOP) return; @@ -2717,12 +2652,7 @@ void sun4v_do_mna(struct pt_regs *regs, unsigned long addr, unsigned long type_c if (is_no_fault_exception(regs)) return; - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRALN; - info.si_addr = (void __user *) addr; - info.si_trapno = 0; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGBUS, BUS_ADRALN, (void __user *) addr, 0, current); } /* sun4v_mem_corrupt_detect_precise() - Handle precise exception on an ADI @@ -2775,7 +2705,6 @@ void sun4v_mem_corrupt_detect_precise(struct pt_regs *regs, unsigned long addr, void do_privop(struct pt_regs *regs) { enum ctx_state prev_state = exception_enter(); - siginfo_t info; if (notify_die(DIE_TRAP, "privileged operation", regs, 0, 0x11, SIGILL) == NOTIFY_STOP) @@ -2785,12 +2714,8 @@ void do_privop(struct pt_regs *regs) regs->tpc &= 0xffffffff; regs->tnpc &= 0xffffffff; } - info.si_signo = SIGILL; - info.si_errno = 0; - info.si_code = ILL_PRVOPC; - info.si_addr = (void __user *)regs->tpc; - info.si_trapno = 0; - force_sig_info(SIGILL, &info, current); + force_sig_fault(SIGILL, ILL_PRVOPC, + (void __user *)regs->tpc, 0, current); out: exception_exit(prev_state); } diff --git a/arch/sparc/kernel/unaligned_32.c b/arch/sparc/kernel/unaligned_32.c index 7642d7e4f0d9..64ac8c0c1429 100644 --- a/arch/sparc/kernel/unaligned_32.c +++ b/arch/sparc/kernel/unaligned_32.c @@ -311,14 +311,9 @@ static inline int ok_for_user(struct pt_regs *regs, unsigned int insn, static void user_mna_trap_fault(struct pt_regs *regs, unsigned int insn) { - siginfo_t info; - - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRALN; - info.si_addr = (void __user *)safe_compute_effective_address(regs, insn); - info.si_trapno = 0; - send_sig_info(SIGBUS, &info, current); + send_sig_fault(SIGBUS, BUS_ADRALN, + (void __user *)safe_compute_effective_address(regs, insn), + 0, current); } asmlinkage void user_unaligned_trap(struct pt_regs *regs, unsigned int insn) diff --git a/arch/sparc/mm/fault_32.c b/arch/sparc/mm/fault_32.c index a8103a84b4ac..9f75b6444bf1 100644 --- a/arch/sparc/mm/fault_32.c +++ b/arch/sparc/mm/fault_32.c @@ -127,19 +127,11 @@ show_signal_msg(struct pt_regs *regs, int sig, int code, static void __do_fault_siginfo(int code, int sig, struct pt_regs *regs, unsigned long addr) { - siginfo_t info; - - info.si_signo = sig; - info.si_code = code; - info.si_errno = 0; - info.si_addr = (void __user *) addr; - info.si_trapno = 0; - if (unlikely(show_unhandled_signals)) - show_signal_msg(regs, sig, info.si_code, + show_signal_msg(regs, sig, code, addr, current); - force_sig_info (sig, &info, current); + force_sig_fault(sig, code, (void __user *) addr, 0, current); } static unsigned long compute_si_addr(struct pt_regs *regs, int text_fault) diff --git a/arch/sparc/mm/fault_64.c b/arch/sparc/mm/fault_64.c index 41363f46797b..63166fcf9e25 100644 --- a/arch/sparc/mm/fault_64.c +++ b/arch/sparc/mm/fault_64.c @@ -170,11 +170,7 @@ static void do_fault_siginfo(int code, int sig, struct pt_regs *regs, int fault_code) { unsigned long addr; - siginfo_t info; - info.si_code = code; - info.si_signo = sig; - info.si_errno = 0; if (fault_code & FAULT_CODE_ITLB) { addr = regs->tpc; } else { @@ -187,13 +183,11 @@ static void do_fault_siginfo(int code, int sig, struct pt_regs *regs, else addr = fault_addr; } - info.si_addr = (void __user *) addr; - info.si_trapno = 0; if (unlikely(show_unhandled_signals)) show_signal_msg(regs, sig, code, addr, current); - force_sig_info(sig, &info, current); + force_sig_fault(sig, code, (void __user *) addr, 0, current); } static unsigned int get_fault_insn(struct pt_regs *regs, unsigned int insn) diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 8aeb1aabe76e..f396048a0d68 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -1620,7 +1620,7 @@ static void __init bootmem_init_nonnuma(void) (top_of_ram - total_ram) >> 20); init_node_masks_nonnuma(); - memblock_set_node(0, (phys_addr_t)ULLONG_MAX, &memblock.memory, 0); + memblock_set_node(0, PHYS_ADDR_MAX, &memblock.memory, 0); allocate_node_data(0); node_set_online(0); } diff --git a/arch/sparc/net/Makefile b/arch/sparc/net/Makefile index 76fa8e95b721..d32aac3a25b8 100644 --- a/arch/sparc/net/Makefile +++ b/arch/sparc/net/Makefile @@ -1,4 +1,7 @@ # # Arch-specific network modules # -obj-$(CONFIG_BPF_JIT) += bpf_jit_asm_$(BITS).o bpf_jit_comp_$(BITS).o +obj-$(CONFIG_BPF_JIT) += bpf_jit_comp_$(BITS).o +ifeq ($(BITS),32) +obj-$(CONFIG_BPF_JIT) += bpf_jit_asm_32.o +endif diff --git a/arch/sparc/net/bpf_jit_64.h b/arch/sparc/net/bpf_jit_64.h index 428f7fd19175..fbc836f1c51c 100644 --- a/arch/sparc/net/bpf_jit_64.h +++ b/arch/sparc/net/bpf_jit_64.h @@ -33,35 +33,6 @@ #define I5 0x1d #define FP 0x1e #define I7 0x1f - -#define r_SKB L0 -#define r_HEADLEN L4 -#define r_SKB_DATA L5 -#define r_TMP G1 -#define r_TMP2 G3 - -/* assembly code in arch/sparc/net/bpf_jit_asm_64.S */ -extern u32 bpf_jit_load_word[]; -extern u32 bpf_jit_load_half[]; -extern u32 bpf_jit_load_byte[]; -extern u32 bpf_jit_load_byte_msh[]; -extern u32 bpf_jit_load_word_positive_offset[]; -extern u32 bpf_jit_load_half_positive_offset[]; -extern u32 bpf_jit_load_byte_positive_offset[]; -extern u32 bpf_jit_load_byte_msh_positive_offset[]; -extern u32 bpf_jit_load_word_negative_offset[]; -extern u32 bpf_jit_load_half_negative_offset[]; -extern u32 bpf_jit_load_byte_negative_offset[]; -extern u32 bpf_jit_load_byte_msh_negative_offset[]; - -#else -#define r_RESULT %o0 -#define r_SKB %o0 -#define r_OFF %o1 -#define r_HEADLEN %l4 -#define r_SKB_DATA %l5 -#define r_TMP %g1 -#define r_TMP2 %g3 #endif #endif /* _BPF_JIT_H */ diff --git a/arch/sparc/net/bpf_jit_asm_64.S b/arch/sparc/net/bpf_jit_asm_64.S deleted file mode 100644 index 7177867052a1..000000000000 --- a/arch/sparc/net/bpf_jit_asm_64.S +++ /dev/null @@ -1,162 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#include - -#include "bpf_jit_64.h" - -#define SAVE_SZ 176 -#define SCRATCH_OFF STACK_BIAS + 128 -#define BE_PTR(label) be,pn %xcc, label -#define SIGN_EXTEND(reg) sra reg, 0, reg - -#define SKF_MAX_NEG_OFF (-0x200000) /* SKF_LL_OFF from filter.h */ - - .text - .globl bpf_jit_load_word -bpf_jit_load_word: - cmp r_OFF, 0 - bl bpf_slow_path_word_neg - nop - .globl bpf_jit_load_word_positive_offset -bpf_jit_load_word_positive_offset: - sub r_HEADLEN, r_OFF, r_TMP - cmp r_TMP, 3 - ble bpf_slow_path_word - add r_SKB_DATA, r_OFF, r_TMP - andcc r_TMP, 3, %g0 - bne load_word_unaligned - nop - retl - ld [r_TMP], r_RESULT -load_word_unaligned: - ldub [r_TMP + 0x0], r_OFF - ldub [r_TMP + 0x1], r_TMP2 - sll r_OFF, 8, r_OFF - or r_OFF, r_TMP2, r_OFF - ldub [r_TMP + 0x2], r_TMP2 - sll r_OFF, 8, r_OFF - or r_OFF, r_TMP2, r_OFF - ldub [r_TMP + 0x3], r_TMP2 - sll r_OFF, 8, r_OFF - retl - or r_OFF, r_TMP2, r_RESULT - - .globl bpf_jit_load_half -bpf_jit_load_half: - cmp r_OFF, 0 - bl bpf_slow_path_half_neg - nop - .globl bpf_jit_load_half_positive_offset -bpf_jit_load_half_positive_offset: - sub r_HEADLEN, r_OFF, r_TMP - cmp r_TMP, 1 - ble bpf_slow_path_half - add r_SKB_DATA, r_OFF, r_TMP - andcc r_TMP, 1, %g0 - bne load_half_unaligned - nop - retl - lduh [r_TMP], r_RESULT -load_half_unaligned: - ldub [r_TMP + 0x0], r_OFF - ldub [r_TMP + 0x1], r_TMP2 - sll r_OFF, 8, r_OFF - retl - or r_OFF, r_TMP2, r_RESULT - - .globl bpf_jit_load_byte -bpf_jit_load_byte: - cmp r_OFF, 0 - bl bpf_slow_path_byte_neg - nop - .globl bpf_jit_load_byte_positive_offset -bpf_jit_load_byte_positive_offset: - cmp r_OFF, r_HEADLEN - bge bpf_slow_path_byte - nop - retl - ldub [r_SKB_DATA + r_OFF], r_RESULT - -#define bpf_slow_path_common(LEN) \ - save %sp, -SAVE_SZ, %sp; \ - mov %i0, %o0; \ - mov %i1, %o1; \ - add %fp, SCRATCH_OFF, %o2; \ - call skb_copy_bits; \ - mov (LEN), %o3; \ - cmp %o0, 0; \ - restore; - -bpf_slow_path_word: - bpf_slow_path_common(4) - bl bpf_error - ld [%sp + SCRATCH_OFF], r_RESULT - retl - nop -bpf_slow_path_half: - bpf_slow_path_common(2) - bl bpf_error - lduh [%sp + SCRATCH_OFF], r_RESULT - retl - nop -bpf_slow_path_byte: - bpf_slow_path_common(1) - bl bpf_error - ldub [%sp + SCRATCH_OFF], r_RESULT - retl - nop - -#define bpf_negative_common(LEN) \ - save %sp, -SAVE_SZ, %sp; \ - mov %i0, %o0; \ - mov %i1, %o1; \ - SIGN_EXTEND(%o1); \ - call bpf_internal_load_pointer_neg_helper; \ - mov (LEN), %o2; \ - mov %o0, r_TMP; \ - cmp %o0, 0; \ - BE_PTR(bpf_error); \ - restore; - -bpf_slow_path_word_neg: - sethi %hi(SKF_MAX_NEG_OFF), r_TMP - cmp r_OFF, r_TMP - bl bpf_error - nop - .globl bpf_jit_load_word_negative_offset -bpf_jit_load_word_negative_offset: - bpf_negative_common(4) - andcc r_TMP, 3, %g0 - bne load_word_unaligned - nop - retl - ld [r_TMP], r_RESULT - -bpf_slow_path_half_neg: - sethi %hi(SKF_MAX_NEG_OFF), r_TMP - cmp r_OFF, r_TMP - bl bpf_error - nop - .globl bpf_jit_load_half_negative_offset -bpf_jit_load_half_negative_offset: - bpf_negative_common(2) - andcc r_TMP, 1, %g0 - bne load_half_unaligned - nop - retl - lduh [r_TMP], r_RESULT - -bpf_slow_path_byte_neg: - sethi %hi(SKF_MAX_NEG_OFF), r_TMP - cmp r_OFF, r_TMP - bl bpf_error - nop - .globl bpf_jit_load_byte_negative_offset -bpf_jit_load_byte_negative_offset: - bpf_negative_common(1) - retl - ldub [r_TMP], r_RESULT - -bpf_error: - /* Make the JIT program itself return zero. */ - ret - restore %g0, %g0, %o0 diff --git a/arch/sparc/net/bpf_jit_comp_32.c b/arch/sparc/net/bpf_jit_comp_32.c index 3bd8ca95e521..a5ff88643d5c 100644 --- a/arch/sparc/net/bpf_jit_comp_32.c +++ b/arch/sparc/net/bpf_jit_comp_32.c @@ -335,7 +335,7 @@ void bpf_jit_compile(struct bpf_prog *fp) if (!bpf_jit_enable) return; - addrs = kmalloc(flen * sizeof(*addrs), GFP_KERNEL); + addrs = kmalloc_array(flen, sizeof(*addrs), GFP_KERNEL); if (addrs == NULL) return; diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c index 48a25869349b..222785af550b 100644 --- a/arch/sparc/net/bpf_jit_comp_64.c +++ b/arch/sparc/net/bpf_jit_comp_64.c @@ -48,10 +48,6 @@ static void bpf_flush_icache(void *start_, void *end_) } } -#define SEEN_DATAREF 1 /* might call external helpers */ -#define SEEN_XREG 2 /* ebx is used */ -#define SEEN_MEM 4 /* use mem[] for temporary storage */ - #define S13(X) ((X) & 0x1fff) #define S5(X) ((X) & 0x1f) #define IMMED 0x00002000 @@ -198,7 +194,6 @@ struct jit_ctx { bool tmp_1_used; bool tmp_2_used; bool tmp_3_used; - bool saw_ld_abs_ind; bool saw_frame_pointer; bool saw_call; bool saw_tail_call; @@ -207,9 +202,7 @@ struct jit_ctx { #define TMP_REG_1 (MAX_BPF_JIT_REG + 0) #define TMP_REG_2 (MAX_BPF_JIT_REG + 1) -#define SKB_HLEN_REG (MAX_BPF_JIT_REG + 2) -#define SKB_DATA_REG (MAX_BPF_JIT_REG + 3) -#define TMP_REG_3 (MAX_BPF_JIT_REG + 4) +#define TMP_REG_3 (MAX_BPF_JIT_REG + 2) /* Map BPF registers to SPARC registers */ static const int bpf2sparc[] = { @@ -238,9 +231,6 @@ static const int bpf2sparc[] = { [TMP_REG_1] = G1, [TMP_REG_2] = G2, [TMP_REG_3] = G3, - - [SKB_HLEN_REG] = L4, - [SKB_DATA_REG] = L5, }; static void emit(const u32 insn, struct jit_ctx *ctx) @@ -800,25 +790,6 @@ static int emit_compare_and_branch(const u8 code, const u8 dst, u8 src, return 0; } -static void load_skb_regs(struct jit_ctx *ctx, u8 r_skb) -{ - const u8 r_headlen = bpf2sparc[SKB_HLEN_REG]; - const u8 r_data = bpf2sparc[SKB_DATA_REG]; - const u8 r_tmp = bpf2sparc[TMP_REG_1]; - unsigned int off; - - off = offsetof(struct sk_buff, len); - emit(LD32I | RS1(r_skb) | S13(off) | RD(r_headlen), ctx); - - off = offsetof(struct sk_buff, data_len); - emit(LD32I | RS1(r_skb) | S13(off) | RD(r_tmp), ctx); - - emit(SUB | RS1(r_headlen) | RS2(r_tmp) | RD(r_headlen), ctx); - - off = offsetof(struct sk_buff, data); - emit(LDPTRI | RS1(r_skb) | S13(off) | RD(r_data), ctx); -} - /* Just skip the save instruction and the ctx register move. */ #define BPF_TAILCALL_PROLOGUE_SKIP 16 #define BPF_TAILCALL_CNT_SP_OFF (STACK_BIAS + 128) @@ -857,9 +828,6 @@ static void build_prologue(struct jit_ctx *ctx) emit_reg_move(I0, O0, ctx); /* If you add anything here, adjust BPF_TAILCALL_PROLOGUE_SKIP above. */ - - if (ctx->saw_ld_abs_ind) - load_skb_regs(ctx, bpf2sparc[BPF_REG_1]); } static void build_epilogue(struct jit_ctx *ctx) @@ -926,7 +894,6 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) const int i = insn - ctx->prog->insnsi; const s16 off = insn->off; const s32 imm = insn->imm; - u32 *func; if (insn->src_reg == BPF_REG_FP) ctx->saw_frame_pointer = true; @@ -1225,16 +1192,11 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) u8 *func = ((u8 *)__bpf_call_base) + imm; ctx->saw_call = true; - if (ctx->saw_ld_abs_ind && bpf_helper_changes_pkt_data(func)) - emit_reg_move(bpf2sparc[BPF_REG_1], L7, ctx); emit_call((u32 *)func, ctx); emit_nop(ctx); emit_reg_move(O0, bpf2sparc[BPF_REG_0], ctx); - - if (ctx->saw_ld_abs_ind && bpf_helper_changes_pkt_data(func)) - load_skb_regs(ctx, L7); break; } @@ -1412,43 +1374,6 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) emit_nop(ctx); break; } -#define CHOOSE_LOAD_FUNC(K, func) \ - ((int)K < 0 ? ((int)K >= SKF_LL_OFF ? func##_negative_offset : func) : func##_positive_offset) - - /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + imm)) */ - case BPF_LD | BPF_ABS | BPF_W: - func = CHOOSE_LOAD_FUNC(imm, bpf_jit_load_word); - goto common_load; - case BPF_LD | BPF_ABS | BPF_H: - func = CHOOSE_LOAD_FUNC(imm, bpf_jit_load_half); - goto common_load; - case BPF_LD | BPF_ABS | BPF_B: - func = CHOOSE_LOAD_FUNC(imm, bpf_jit_load_byte); - goto common_load; - /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + src + imm)) */ - case BPF_LD | BPF_IND | BPF_W: - func = bpf_jit_load_word; - goto common_load; - case BPF_LD | BPF_IND | BPF_H: - func = bpf_jit_load_half; - goto common_load; - - case BPF_LD | BPF_IND | BPF_B: - func = bpf_jit_load_byte; - common_load: - ctx->saw_ld_abs_ind = true; - - emit_reg_move(bpf2sparc[BPF_REG_6], O0, ctx); - emit_loadimm(imm, O1, ctx); - - if (BPF_MODE(code) == BPF_IND) - emit_alu(ADD, src, O1, ctx); - - emit_call(func, ctx); - emit_alu_K(SRA, O1, 0, ctx); - - emit_reg_move(O0, bpf2sparc[BPF_REG_0], ctx); - break; default: pr_err_once("unknown opcode %02x\n", code); @@ -1583,12 +1508,11 @@ skip_init_ctx: build_epilogue(&ctx); if (bpf_jit_enable > 1) - pr_info("Pass %d: shrink = %d, seen = [%c%c%c%c%c%c%c]\n", pass, + pr_info("Pass %d: shrink = %d, seen = [%c%c%c%c%c%c]\n", pass, image_size - (ctx.idx * 4), ctx.tmp_1_used ? '1' : ' ', ctx.tmp_2_used ? '2' : ' ', ctx.tmp_3_used ? '3' : ' ', - ctx.saw_ld_abs_ind ? 'L' : ' ', ctx.saw_frame_pointer ? 'F' : ' ', ctx.saw_call ? 'C' : ' ', ctx.saw_tail_call ? 'T' : ' '); diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common index c68add8df3ae..07f84c842cc3 100644 --- a/arch/um/Kconfig.common +++ b/arch/um/Kconfig.common @@ -54,10 +54,6 @@ config HZ int default 100 -config SUBARCH - string - option env="SUBARCH" - config NR_CPUS int range 1 1 diff --git a/arch/um/Kconfig.um b/arch/um/Kconfig.um index 3e7f228b22e1..20da5a8ca949 100644 --- a/arch/um/Kconfig.um +++ b/arch/um/Kconfig.um @@ -80,7 +80,7 @@ config MAGIC_SYSRQ On UML, this is accomplished by sending a "sysrq" command with mconsole, followed by the letter for the requested command. - The keys are documented in . Don't say Y + The keys are documented in . Don't say Y unless you really know what this hack does. config KERNEL_STACK_ORDER diff --git a/arch/um/drivers/ubd_kern.c b/arch/um/drivers/ubd_kern.c index d4e8c497ae86..83c470364dfb 100644 --- a/arch/um/drivers/ubd_kern.c +++ b/arch/um/drivers/ubd_kern.c @@ -208,19 +208,6 @@ static int fake_ide_media_proc_show(struct seq_file *m, void *v) return 0; } -static int fake_ide_media_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, fake_ide_media_proc_show, NULL); -} - -static const struct file_operations fake_ide_media_proc_fops = { - .owner = THIS_MODULE, - .open = fake_ide_media_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static void make_ide_entries(const char *dev_name) { struct proc_dir_entry *dir, *ent; @@ -231,7 +218,8 @@ static void make_ide_entries(const char *dev_name) dir = proc_mkdir(dev_name, proc_ide); if(!dir) return; - ent = proc_create("media", S_IRUGO, dir, &fake_ide_media_proc_fops); + ent = proc_create_single("media", S_IRUGO, dir, + fake_ide_media_proc_show); if(!ent) return; snprintf(name, sizeof(name), "ide0/%s", dev_name); proc_symlink(dev_name, proc_ide_root, name); @@ -1139,9 +1127,9 @@ static int __init ubd_init(void) return -1; } - irq_req_buffer = kmalloc( - sizeof(struct io_thread_req *) * UBD_REQ_BUFFER_SIZE, - GFP_KERNEL + irq_req_buffer = kmalloc_array(UBD_REQ_BUFFER_SIZE, + sizeof(struct io_thread_req *), + GFP_KERNEL ); irq_remainder = 0; @@ -1149,9 +1137,9 @@ static int __init ubd_init(void) printk(KERN_ERR "Failed to initialize ubd buffering\n"); return -1; } - io_req_buffer = kmalloc( - sizeof(struct io_thread_req *) * UBD_REQ_BUFFER_SIZE, - GFP_KERNEL + io_req_buffer = kmalloc_array(UBD_REQ_BUFFER_SIZE, + sizeof(struct io_thread_req *), + GFP_KERNEL ); io_remainder = 0; diff --git a/arch/um/drivers/vector_kern.c b/arch/um/drivers/vector_kern.c index 02168fe25105..50ee3bb5a63a 100644 --- a/arch/um/drivers/vector_kern.c +++ b/arch/um/drivers/vector_kern.c @@ -188,7 +188,7 @@ static int get_transport_options(struct arglist *def) if (strncmp(transport, TRANS_TAP, TRANS_TAP_LEN) == 0) return (vec_rx | VECTOR_BPF); if (strncmp(transport, TRANS_RAW, TRANS_RAW_LEN) == 0) - return (vec_rx | vec_tx); + return (vec_rx | vec_tx | VECTOR_QDISC_BYPASS); return (vec_rx | vec_tx); } @@ -504,15 +504,19 @@ static struct vector_queue *create_queue( result = kmalloc(sizeof(struct vector_queue), GFP_KERNEL); if (result == NULL) - goto out_fail; + return NULL; result->max_depth = max_size; result->dev = vp->dev; result->mmsg_vector = kmalloc( (sizeof(struct mmsghdr) * max_size), GFP_KERNEL); + if (result->mmsg_vector == NULL) + goto out_mmsg_fail; result->skbuff_vector = kmalloc( (sizeof(void *) * max_size), GFP_KERNEL); - if (result->mmsg_vector == NULL || result->skbuff_vector == NULL) - goto out_fail; + if (result->skbuff_vector == NULL) + goto out_skb_fail; + + /* further failures can be handled safely by destroy_queue*/ mmsg_vector = result->mmsg_vector; for (i = 0; i < max_size; i++) { @@ -527,14 +531,14 @@ static struct vector_queue *create_queue( result->max_iov_frags = num_extra_frags; for (i = 0; i < max_size; i++) { if (vp->header_size > 0) - iov = kmalloc( - sizeof(struct iovec) * (3 + num_extra_frags), - GFP_KERNEL + iov = kmalloc_array(3 + num_extra_frags, + sizeof(struct iovec), + GFP_KERNEL ); else - iov = kmalloc( - sizeof(struct iovec) * (2 + num_extra_frags), - GFP_KERNEL + iov = kmalloc_array(2 + num_extra_frags, + sizeof(struct iovec), + GFP_KERNEL ); if (iov == NULL) goto out_fail; @@ -563,6 +567,11 @@ static struct vector_queue *create_queue( result->head = 0; result->tail = 0; return result; +out_skb_fail: + kfree(result->mmsg_vector); +out_mmsg_fail: + kfree(result); + return NULL; out_fail: destroy_queue(result); return NULL; @@ -1232,9 +1241,8 @@ static int vector_net_open(struct net_device *dev) if ((vp->options & VECTOR_QDISC_BYPASS) != 0) { if (!uml_raw_enable_qdisc_bypass(vp->fds->rx_fd)) - vp->options = vp->options | VECTOR_BPF; + vp->options |= VECTOR_BPF; } - if ((vp->options & VECTOR_BPF) != 0) vp->bpf = uml_vector_default_bpf(vp->fds->rx_fd, dev->dev_addr); diff --git a/arch/um/drivers/vector_transports.c b/arch/um/drivers/vector_transports.c index 9065047f844b..77e4ebc206ae 100644 --- a/arch/um/drivers/vector_transports.c +++ b/arch/um/drivers/vector_transports.c @@ -120,7 +120,8 @@ static int raw_form_header(uint8_t *header, skb, vheader, virtio_legacy_is_little_endian(), - false + false, + 0 ); return 0; diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild index bb5a196c3061..b10dde6cb793 100644 --- a/arch/um/include/asm/Kbuild +++ b/arch/um/include/asm/Kbuild @@ -1,6 +1,7 @@ generic-y += barrier.h generic-y += bpf_perf_event.h generic-y += bug.h +generic-y += compat.h generic-y += current.h generic-y += delay.h generic-y += device.h diff --git a/arch/um/include/asm/common.lds.S b/arch/um/include/asm/common.lds.S index b30d73ca29d0..7adb4e6b658a 100644 --- a/arch/um/include/asm/common.lds.S +++ b/arch/um/include/asm/common.lds.S @@ -53,12 +53,6 @@ CON_INITCALL } - .uml.initcall.init : { - __uml_initcall_start = .; - *(.uml.initcall.init) - __uml_initcall_end = .; - } - SECURITY_INIT .exitcall : { diff --git a/arch/um/include/shared/init.h b/arch/um/include/shared/init.h index b3f5865a92c9..c66de434a983 100644 --- a/arch/um/include/shared/init.h +++ b/arch/um/include/shared/init.h @@ -64,14 +64,10 @@ struct uml_param { int (*setup_func)(char *, int *); }; -extern initcall_t __uml_initcall_start, __uml_initcall_end; extern initcall_t __uml_postsetup_start, __uml_postsetup_end; extern const char *__uml_help_start, *__uml_help_end; #endif -#define __uml_initcall(fn) \ - static initcall_t __uml_initcall_##fn __uml_init_call = fn - #define __uml_exitcall(fn) \ static exitcall_t __uml_exitcall_##fn __uml_exit_call = fn @@ -108,7 +104,6 @@ extern struct uml_param __uml_setup_start, __uml_setup_end; */ #define __uml_init_setup __used __section(.uml.setup.init) #define __uml_setup_help __used __section(.uml.help.init) -#define __uml_init_call __used __section(.uml.initcall.init) #define __uml_postsetup_call __used __section(.uml.postsetup.init) #define __uml_exit_call __used __section(.uml.exitcall.exit) diff --git a/arch/um/kernel/ptrace.c b/arch/um/kernel/ptrace.c index bc2a516c190f..1a1d88a4d940 100644 --- a/arch/um/kernel/ptrace.c +++ b/arch/um/kernel/ptrace.c @@ -115,17 +115,10 @@ long arch_ptrace(struct task_struct *child, long request, static void send_sigtrap(struct task_struct *tsk, struct uml_pt_regs *regs, int error_code) { - struct siginfo info; - - memset(&info, 0, sizeof(info)); - info.si_signo = SIGTRAP; - info.si_code = TRAP_BRKPT; - - /* User-mode eip? */ - info.si_addr = UPT_IS_USER(regs) ? (void __user *) UPT_IP(regs) : NULL; - /* Send us the fake SIGTRAP */ - force_sig_info(SIGTRAP, &info, tsk); + force_sig_fault(SIGTRAP, TRAP_BRKPT, + /* User-mode eip? */ + UPT_IS_USER(regs) ? (void __user *) UPT_IP(regs) : NULL, tsk); } /* diff --git a/arch/um/kernel/trap.c b/arch/um/kernel/trap.c index b2b02df9896e..ec9a42c14c56 100644 --- a/arch/um/kernel/trap.c +++ b/arch/um/kernel/trap.c @@ -162,13 +162,9 @@ static void show_segv_info(struct uml_pt_regs *regs) static void bad_segv(struct faultinfo fi, unsigned long ip) { - struct siginfo si; - - si.si_signo = SIGSEGV; - si.si_code = SEGV_ACCERR; - si.si_addr = (void __user *) FAULT_ADDRESS(fi); current->thread.arch.faultinfo = fi; - force_sig_info(SIGSEGV, &si, current); + force_sig_fault(SIGSEGV, SEGV_ACCERR, (void __user *) FAULT_ADDRESS(fi), + current); } void fatal_sigsegv(void) @@ -214,8 +210,8 @@ void segv_handler(int sig, struct siginfo *unused_si, struct uml_pt_regs *regs) unsigned long segv(struct faultinfo fi, unsigned long ip, int is_user, struct uml_pt_regs *regs) { - struct siginfo si; jmp_buf *catcher; + int si_code; int err; int is_write = FAULT_WRITE(fi); unsigned long address = FAULT_ADDRESS(fi); @@ -239,7 +235,7 @@ unsigned long segv(struct faultinfo fi, unsigned long ip, int is_user, if (SEGV_IS_FIXABLE(&fi)) err = handle_page_fault(address, ip, is_write, is_user, - &si.si_code); + &si_code); else { err = -EFAULT; /* @@ -271,18 +267,14 @@ unsigned long segv(struct faultinfo fi, unsigned long ip, int is_user, show_segv_info(regs); if (err == -EACCES) { - si.si_signo = SIGBUS; - si.si_errno = 0; - si.si_code = BUS_ADRERR; - si.si_addr = (void __user *)address; current->thread.arch.faultinfo = fi; - force_sig_info(SIGBUS, &si, current); + force_sig_fault(SIGBUS, BUS_ADRERR, (void __user *)address, + current); } else { BUG_ON(err != -EFAULT); - si.si_signo = SIGSEGV; - si.si_addr = (void __user *) address; current->thread.arch.faultinfo = fi; - force_sig_info(SIGSEGV, &si, current); + force_sig_fault(SIGSEGV, si_code, (void __user *) address, + current); } out: @@ -294,9 +286,7 @@ out: void relay_signal(int sig, struct siginfo *si, struct uml_pt_regs *regs) { - struct faultinfo *fi; - struct siginfo clean_si; - + int code, err; if (!UPT_IS_USER(regs)) { if (sig == SIGBUS) printk(KERN_ERR "Bus error - the host /dev/shm or /tmp " @@ -306,29 +296,21 @@ void relay_signal(int sig, struct siginfo *si, struct uml_pt_regs *regs) arch_examine_signal(sig, regs); - clear_siginfo(&clean_si); - clean_si.si_signo = si->si_signo; - clean_si.si_errno = si->si_errno; - clean_si.si_code = si->si_code; - switch (sig) { - case SIGILL: - case SIGFPE: - case SIGSEGV: - case SIGBUS: - case SIGTRAP: - fi = UPT_FAULTINFO(regs); - clean_si.si_addr = (void __user *) FAULT_ADDRESS(*fi); + /* Is the signal layout for the signal known? + * Signal data must be scrubbed to prevent information leaks. + */ + code = si->si_code; + err = si->si_errno; + if ((err == 0) && (siginfo_layout(sig, code) == SIL_FAULT)) { + struct faultinfo *fi = UPT_FAULTINFO(regs); current->thread.arch.faultinfo = *fi; -#ifdef __ARCH_SI_TRAPNO - clean_si.si_trapno = si->si_trapno; -#endif - break; - default: - printk(KERN_ERR "Attempted to relay unknown signal %d (si_code = %d)\n", - sig, si->si_code); + force_sig_fault(sig, code, (void __user *)FAULT_ADDRESS(*fi), + current); + } else { + printk(KERN_ERR "Attempted to relay unknown signal %d (si_code = %d) with errno %d\n", + sig, code, err); + force_sig(sig, current); } - - force_sig_info(sig, &clean_si, current); } void bus_handler(int sig, struct siginfo *si, struct uml_pt_regs *regs) diff --git a/arch/um/os-Linux/main.c b/arch/um/os-Linux/main.c index 5f970ece5ac3..f1fee2b91239 100644 --- a/arch/um/os-Linux/main.c +++ b/arch/um/os-Linux/main.c @@ -40,17 +40,6 @@ static void set_stklim(void) } } -static __init void do_uml_initcalls(void) -{ - initcall_t *call; - - call = &__uml_initcall_start; - while (call < &__uml_initcall_end) { - (*call)(); - call++; - } -} - static void last_ditch_exit(int sig) { uml_cleanup(); @@ -151,7 +140,6 @@ int __init main(int argc, char **argv, char **envp) scan_elf_aux(envp); #endif - do_uml_initcalls(); change_sig(SIGPIPE, 0); ret = linux_main(argc, argv); diff --git a/arch/unicore32/Kconfig b/arch/unicore32/Kconfig index 462e59a7ae78..03f991e44288 100644 --- a/arch/unicore32/Kconfig +++ b/arch/unicore32/Kconfig @@ -19,6 +19,8 @@ config UNICORE32 select ARCH_WANT_FRAME_POINTERS select GENERIC_IOMAP select MODULES_USE_ELF_REL + select NEED_DMA_MAP_STATE + select SWIOTLB help UniCore-32 is 32-bit Instruction Set Architecture, including a series of low-power-consumption RISC chip @@ -61,9 +63,6 @@ config ARCH_MAY_HAVE_PC_FDC config ZONE_DMA def_bool y -config NEED_DMA_MAP_STATE - def_bool y - source "init/Kconfig" source "kernel/Kconfig.freezer" diff --git a/arch/unicore32/include/asm/Kbuild b/arch/unicore32/include/asm/Kbuild index 6f70c76c81fc..bfc7abe77905 100644 --- a/arch/unicore32/include/asm/Kbuild +++ b/arch/unicore32/include/asm/Kbuild @@ -1,5 +1,6 @@ generic-y += atomic.h generic-y += bugs.h +generic-y += compat.h generic-y += current.h generic-y += device.h generic-y += div64.h diff --git a/arch/unicore32/include/asm/cacheflush.h b/arch/unicore32/include/asm/cacheflush.h index 1d9132b66039..1c8b9f13a9e1 100644 --- a/arch/unicore32/include/asm/cacheflush.h +++ b/arch/unicore32/include/asm/cacheflush.h @@ -33,7 +33,7 @@ * Start addresses are inclusive and end addresses are exclusive; * start addresses should be rounded down, end addresses up. * - * See Documentation/cachetlb.txt for more information. + * See Documentation/core-api/cachetlb.rst for more information. * Please note that the implementation of these, and the required * effects are cache-type (VIVT/VIPT/PIPT) specific. * diff --git a/arch/unicore32/kernel/fpu-ucf64.c b/arch/unicore32/kernel/fpu-ucf64.c index 12c8c9527b8e..8594b168f25e 100644 --- a/arch/unicore32/kernel/fpu-ucf64.c +++ b/arch/unicore32/kernel/fpu-ucf64.c @@ -52,14 +52,14 @@ * Raise a SIGFPE for the current process. * sicode describes the signal being raised. */ -void ucf64_raise_sigfpe(unsigned int sicode, struct pt_regs *regs) +void ucf64_raise_sigfpe(struct pt_regs *regs) { siginfo_t info; - memset(&info, 0, sizeof(info)); + clear_siginfo(&info); info.si_signo = SIGFPE; - info.si_code = sicode; + info.si_code = FPE_FLTUNK; info.si_addr = (void __user *)(instruction_pointer(regs) - 4); /* @@ -94,7 +94,7 @@ void ucf64_exchandler(u32 inst, u32 fpexc, struct pt_regs *regs) pr_debug("UniCore-F64 FPSCR 0x%08x INST 0x%08x\n", cff(FPSCR), inst); - ucf64_raise_sigfpe(0, regs); + ucf64_raise_sigfpe(regs); return; } diff --git a/arch/unicore32/kernel/pm.c b/arch/unicore32/kernel/pm.c index 784bc2db3b28..6f8164d91dc2 100644 --- a/arch/unicore32/kernel/pm.c +++ b/arch/unicore32/kernel/pm.c @@ -109,8 +109,9 @@ static int __init puv3_pm_init(void) return -EINVAL; } - sleep_save = kmalloc(puv3_cpu_pm_fns->save_count - * sizeof(unsigned long), GFP_KERNEL); + sleep_save = kmalloc_array(puv3_cpu_pm_fns->save_count, + sizeof(unsigned long), + GFP_KERNEL); if (!sleep_save) { printk(KERN_ERR "failed to alloc memory for pm save\n"); return -ENOMEM; diff --git a/arch/unicore32/mm/Kconfig b/arch/unicore32/mm/Kconfig index e9154a59d561..82759b6aba67 100644 --- a/arch/unicore32/mm/Kconfig +++ b/arch/unicore32/mm/Kconfig @@ -39,14 +39,3 @@ config CPU_TLB_SINGLE_ENTRY_DISABLE default y help Say Y here to disable the TLB single entry operations. - -config SWIOTLB - def_bool y - select DMA_DIRECT_OPS - -config IOMMU_HELPER - def_bool SWIOTLB - -config NEED_SG_DMA_LENGTH - def_bool SWIOTLB - diff --git a/arch/unicore32/mm/fault.c b/arch/unicore32/mm/fault.c index bbefcc46a45e..381473412937 100644 --- a/arch/unicore32/mm/fault.c +++ b/arch/unicore32/mm/fault.c @@ -125,6 +125,7 @@ static void __do_user_fault(struct task_struct *tsk, unsigned long addr, tsk->thread.address = addr; tsk->thread.error_code = fsr; tsk->thread.trap_no = 14; + clear_siginfo(&si); si.si_signo = sig; si.si_errno = 0; si.si_code = code; @@ -472,6 +473,7 @@ asmlinkage void do_DataAbort(unsigned long addr, unsigned int fsr, printk(KERN_ALERT "Unhandled fault: %s (0x%03x) at 0x%08lx\n", inf->name, fsr, addr); + clear_siginfo(&info); info.si_signo = inf->sig; info.si_errno = 0; info.si_code = inf->code; @@ -491,6 +493,7 @@ asmlinkage void do_PrefetchAbort(unsigned long addr, printk(KERN_ALERT "Unhandled prefetch abort: %s (0x%03x) at 0x%08lx\n", inf->name, ifsr, addr); + clear_siginfo(&info); info.si_signo = inf->sig; info.si_errno = 0; info.si_code = inf->code; diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index c07f492b871a..f1dbb4ee19d7 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1,8 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 # Select 32 or 64 bit config 64BIT - bool "64-bit kernel" if ARCH = "x86" - default ARCH != "i386" + bool "64-bit kernel" if "$(ARCH)" = "x86" + default "$(ARCH)" != "i386" ---help--- Say yes to build a 64-bit kernel - formerly known as x86_64 Say no to build a 32-bit kernel - formerly known as i386 @@ -28,6 +28,8 @@ config X86_64 select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA + select NEED_DMA_MAP_STATE + select SWIOTLB select X86_DEV_DMA_OPS select ARCH_HAS_SYSCALL_WRAPPER @@ -58,8 +60,10 @@ config X86 select ARCH_HAS_KCOV if X86_64 select ARCH_HAS_MEMBARRIER_SYNC_CORE select ARCH_HAS_PMEM_API if X86_64 + select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_REFCOUNT select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 + select ARCH_HAS_UACCESS_MCSAFE if X86_64 select ARCH_HAS_SET_MEMORY select ARCH_HAS_SG_CHAIN select ARCH_HAS_STRICT_KERNEL_RWX @@ -126,7 +130,6 @@ config X86 select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64 select HAVE_ARCH_VMAP_STACK if X86_64 select HAVE_ARCH_WITHIN_STACK_FRAMES - select HAVE_CC_STACKPROTECTOR select HAVE_CMPXCHG_DOUBLE select HAVE_CMPXCHG_LOCAL select HAVE_CONTEXT_TRACKING if X86_64 @@ -134,11 +137,10 @@ config X86 select HAVE_C_RECORDMCOUNT select HAVE_DEBUG_KMEMLEAK select HAVE_DEBUG_STACKOVERFLOW - select HAVE_DMA_API_DEBUG select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS - select HAVE_EBPF_JIT if X86_64 + select HAVE_EBPF_JIT select HAVE_EFFICIENT_UNALIGNED_ACCESS select HAVE_EXIT_THREAD select HAVE_FENTRY if X86_64 || DYNAMIC_FTRACE @@ -179,11 +181,14 @@ config X86 select HAVE_RCU_TABLE_FREE select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if X86_64 && UNWINDER_FRAME_POINTER && STACK_VALIDATION + select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR select HAVE_STACK_VALIDATION if X86_64 + select HAVE_RSEQ select HAVE_SYSCALL_TRACEPOINTS select HAVE_UNSTABLE_SCHED_CLOCK select HAVE_USER_RETURN_NOTIFIER select IRQ_FORCED_THREADING + select NEED_SG_DMA_LENGTH select PCI_LOCKLESS_CONFIG select PERF_EVENTS select RTC_LIB @@ -236,13 +241,6 @@ config ARCH_MMAP_RND_COMPAT_BITS_MAX config SBUS bool -config NEED_DMA_MAP_STATE - def_bool y - depends on X86_64 || INTEL_IOMMU || DMA_API_DEBUG || SWIOTLB - -config NEED_SG_DMA_LENGTH - def_bool y - config GENERIC_ISA_DMA def_bool y depends on ISA_DMA_API @@ -329,7 +327,7 @@ config X86_64_SMP config X86_32_LAZY_GS def_bool y - depends on X86_32 && CC_STACKPROTECTOR_NONE + depends on X86_32 && !STACKPROTECTOR config ARCH_SUPPORTS_UPROBES def_bool y @@ -337,6 +335,9 @@ config ARCH_SUPPORTS_UPROBES config FIX_EARLYCON_MEM def_bool y +config DYNAMIC_PHYSICAL_MASK + bool + config PGTABLE_LEVELS int default 5 if X86_5LEVEL @@ -345,6 +346,15 @@ config PGTABLE_LEVELS default 2 source "init/Kconfig" + +config CC_HAS_SANE_STACKPROTECTOR + bool + default $(success,$(srctree)/scripts/gcc-x86_64-has-stack-protector.sh $(CC)) if 64BIT + default $(success,$(srctree)/scripts/gcc-x86_32-has-stack-protector.sh $(CC)) + help + We have to make sure stack protector is unconditionally disabled if + the compiler produces broken code. + source "kernel/Kconfig.freezer" menu "Processor type and features" @@ -875,6 +885,7 @@ config DMI config GART_IOMMU bool "Old AMD GART IOMMU support" + select IOMMU_HELPER select SWIOTLB depends on X86_64 && PCI && AMD_NB ---help--- @@ -896,6 +907,7 @@ config GART_IOMMU config CALGARY_IOMMU bool "IBM Calgary IOMMU support" + select IOMMU_HELPER select SWIOTLB depends on X86_64 && PCI ---help--- @@ -923,20 +935,6 @@ config CALGARY_IOMMU_ENABLED_BY_DEFAULT Calgary anyway, pass 'iommu=calgary' on the kernel command line. If unsure, say Y. -# need this always selected by IOMMU for the VIA workaround -config SWIOTLB - def_bool y if X86_64 - ---help--- - Support for software bounce buffers used on x86-64 systems - which don't have a hardware IOMMU. Using this PCI devices - which can only access 32-bits of memory can be used on systems - with more than 3 GB of memory. - If unsure, say Y. - -config IOMMU_HELPER - def_bool y - depends on CALGARY_IOMMU || GART_IOMMU || SWIOTLB || AMD_IOMMU - config MAXSMP bool "Enable Maximum number of SMP Processors and NUMA Nodes" depends on X86_64 && SMP && DEBUG_KERNEL @@ -1458,6 +1456,7 @@ config HIGHMEM config X86_PAE bool "PAE (Physical Address Extension) Support" depends on X86_32 && !HIGHMEM4G + select PHYS_ADDR_T_64BIT select SWIOTLB ---help--- PAE is required for NX support, and furthermore enables @@ -1485,14 +1484,6 @@ config X86_5LEVEL Say N if unsure. -config ARCH_PHYS_ADDR_T_64BIT - def_bool y - depends on X86_64 || X86_PAE - -config ARCH_DMA_ADDR_T_64BIT - def_bool y - depends on X86_64 || HIGHMEM64G - config X86_DIRECT_GBPAGES def_bool y depends on X86_64 && !DEBUG_PAGEALLOC @@ -1508,6 +1499,7 @@ config ARCH_HAS_MEM_ENCRYPT config AMD_MEM_ENCRYPT bool "AMD Secure Memory Encryption (SME) support" depends on X86_64 && CPU_SUP_AMD + select DYNAMIC_PHYSICAL_MASK ---help--- Say yes to enable support for the encryption of system memory. This requires an AMD processor that supports Secure Memory diff --git a/arch/x86/Kconfig.debug b/arch/x86/Kconfig.debug index 192e4d2f9efc..c6dd1d980081 100644 --- a/arch/x86/Kconfig.debug +++ b/arch/x86/Kconfig.debug @@ -72,6 +72,9 @@ config EARLY_PRINTK_USB_XDBC You should normally say N here, unless you want to debug early crashes or need a very simple printk logging facility. +config MCSAFE_TEST + def_bool n + config X86_PTDUMP_CORE def_bool n diff --git a/arch/x86/Makefile b/arch/x86/Makefile index 60135cbd905c..f0a6ea22429d 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -94,7 +94,7 @@ ifeq ($(CONFIG_X86_32),y) else BITS := 64 UTS_MACHINE := x86_64 - CHECKFLAGS += -D__x86_64__ -m64 + CHECKFLAGS += -D__x86_64__ biarch := -m64 KBUILD_AFLAGS += -m64 diff --git a/arch/x86/boot/compressed/cmdline.c b/arch/x86/boot/compressed/cmdline.c index 0cb325734cfb..af6cda0b7900 100644 --- a/arch/x86/boot/compressed/cmdline.c +++ b/arch/x86/boot/compressed/cmdline.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include "misc.h" -#if CONFIG_EARLY_PRINTK || CONFIG_RANDOMIZE_BASE +#if CONFIG_EARLY_PRINTK || CONFIG_RANDOMIZE_BASE || CONFIG_X86_5LEVEL static unsigned long fs; static inline void set_fs(unsigned long seg) diff --git a/arch/x86/boot/compressed/eboot.c b/arch/x86/boot/compressed/eboot.c index 09f36c0d9d4f..a8a8642d2b0b 100644 --- a/arch/x86/boot/compressed/eboot.c +++ b/arch/x86/boot/compressed/eboot.c @@ -109,23 +109,34 @@ void efi_char16_printk(efi_system_table_t *table, efi_char16_t *str) } static efi_status_t -__setup_efi_pci32(efi_pci_io_protocol_32 *pci, struct pci_setup_rom **__rom) +__setup_efi_pci(efi_pci_io_protocol_t *pci, struct pci_setup_rom **__rom) { struct pci_setup_rom *rom = NULL; efi_status_t status; unsigned long size; - uint64_t attributes; + uint64_t attributes, romsize; + void *romimage; - status = efi_early->call(pci->attributes, pci, - EfiPciIoAttributeOperationGet, 0, 0, - &attributes); + status = efi_call_proto(efi_pci_io_protocol, attributes, pci, + EfiPciIoAttributeOperationGet, 0, 0, + &attributes); if (status != EFI_SUCCESS) return status; - if (!pci->romimage || !pci->romsize) + /* + * Some firmware images contain EFI function pointers at the place where the + * romimage and romsize fields are supposed to be. Typically the EFI + * code is mapped at high addresses, translating to an unrealistically + * large romsize. The UEFI spec limits the size of option ROMs to 16 + * MiB so we reject any ROMs over 16 MiB in size to catch this. + */ + romimage = (void *)(unsigned long)efi_table_attr(efi_pci_io_protocol, + romimage, pci); + romsize = efi_table_attr(efi_pci_io_protocol, romsize, pci); + if (!romimage || !romsize || romsize > SZ_16M) return EFI_INVALID_PARAMETER; - size = pci->romsize + sizeof(*rom); + size = romsize + sizeof(*rom); status = efi_call_early(allocate_pool, EFI_LOADER_DATA, size, &rom); if (status != EFI_SUCCESS) { @@ -141,30 +152,32 @@ __setup_efi_pci32(efi_pci_io_protocol_32 *pci, struct pci_setup_rom **__rom) rom->pcilen = pci->romsize; *__rom = rom; - status = efi_early->call(pci->pci.read, pci, EfiPciIoWidthUint16, - PCI_VENDOR_ID, 1, &(rom->vendor)); + status = efi_call_proto(efi_pci_io_protocol, pci.read, pci, + EfiPciIoWidthUint16, PCI_VENDOR_ID, 1, + &rom->vendor); if (status != EFI_SUCCESS) { efi_printk(sys_table, "Failed to read rom->vendor\n"); goto free_struct; } - status = efi_early->call(pci->pci.read, pci, EfiPciIoWidthUint16, - PCI_DEVICE_ID, 1, &(rom->devid)); + status = efi_call_proto(efi_pci_io_protocol, pci.read, pci, + EfiPciIoWidthUint16, PCI_DEVICE_ID, 1, + &rom->devid); if (status != EFI_SUCCESS) { efi_printk(sys_table, "Failed to read rom->devid\n"); goto free_struct; } - status = efi_early->call(pci->get_location, pci, &(rom->segment), - &(rom->bus), &(rom->device), &(rom->function)); + status = efi_call_proto(efi_pci_io_protocol, get_location, pci, + &rom->segment, &rom->bus, &rom->device, + &rom->function); if (status != EFI_SUCCESS) goto free_struct; - memcpy(rom->romdata, (void *)(unsigned long)pci->romimage, - pci->romsize); + memcpy(rom->romdata, romimage, romsize); return status; free_struct: @@ -176,7 +189,7 @@ static void setup_efi_pci32(struct boot_params *params, void **pci_handle, unsigned long size) { - efi_pci_io_protocol_32 *pci = NULL; + efi_pci_io_protocol_t *pci = NULL; efi_guid_t pci_proto = EFI_PCI_IO_PROTOCOL_GUID; u32 *handles = (u32 *)(unsigned long)pci_handle; efi_status_t status; @@ -203,7 +216,7 @@ setup_efi_pci32(struct boot_params *params, void **pci_handle, if (!pci) continue; - status = __setup_efi_pci32(pci, &rom); + status = __setup_efi_pci(pci, &rom); if (status != EFI_SUCCESS) continue; @@ -217,74 +230,11 @@ setup_efi_pci32(struct boot_params *params, void **pci_handle, } } -static efi_status_t -__setup_efi_pci64(efi_pci_io_protocol_64 *pci, struct pci_setup_rom **__rom) -{ - struct pci_setup_rom *rom; - efi_status_t status; - unsigned long size; - uint64_t attributes; - - status = efi_early->call(pci->attributes, pci, - EfiPciIoAttributeOperationGet, 0, - &attributes); - if (status != EFI_SUCCESS) - return status; - - if (!pci->romimage || !pci->romsize) - return EFI_INVALID_PARAMETER; - - size = pci->romsize + sizeof(*rom); - - status = efi_call_early(allocate_pool, EFI_LOADER_DATA, size, &rom); - if (status != EFI_SUCCESS) { - efi_printk(sys_table, "Failed to alloc mem for rom\n"); - return status; - } - - rom->data.type = SETUP_PCI; - rom->data.len = size - sizeof(struct setup_data); - rom->data.next = 0; - rom->pcilen = pci->romsize; - *__rom = rom; - - status = efi_early->call(pci->pci.read, pci, EfiPciIoWidthUint16, - PCI_VENDOR_ID, 1, &(rom->vendor)); - - if (status != EFI_SUCCESS) { - efi_printk(sys_table, "Failed to read rom->vendor\n"); - goto free_struct; - } - - status = efi_early->call(pci->pci.read, pci, EfiPciIoWidthUint16, - PCI_DEVICE_ID, 1, &(rom->devid)); - - if (status != EFI_SUCCESS) { - efi_printk(sys_table, "Failed to read rom->devid\n"); - goto free_struct; - } - - status = efi_early->call(pci->get_location, pci, &(rom->segment), - &(rom->bus), &(rom->device), &(rom->function)); - - if (status != EFI_SUCCESS) - goto free_struct; - - memcpy(rom->romdata, (void *)(unsigned long)pci->romimage, - pci->romsize); - return status; - -free_struct: - efi_call_early(free_pool, rom); - return status; - -} - static void setup_efi_pci64(struct boot_params *params, void **pci_handle, unsigned long size) { - efi_pci_io_protocol_64 *pci = NULL; + efi_pci_io_protocol_t *pci = NULL; efi_guid_t pci_proto = EFI_PCI_IO_PROTOCOL_GUID; u64 *handles = (u64 *)(unsigned long)pci_handle; efi_status_t status; @@ -311,7 +261,7 @@ setup_efi_pci64(struct boot_params *params, void **pci_handle, if (!pci) continue; - status = __setup_efi_pci64(pci, &rom); + status = __setup_efi_pci(pci, &rom); if (status != EFI_SUCCESS) continue; diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S index 8169e8b7a4dc..64037895b085 100644 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -365,6 +365,7 @@ ENTRY(startup_64) * this function call. */ pushq %rsi + movq %rsi, %rdi /* real mode address */ call paging_prepare popq %rsi diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c index a0a50b91ecef..b87a7582853d 100644 --- a/arch/x86/boot/compressed/kaslr.c +++ b/arch/x86/boot/compressed/kaslr.c @@ -47,7 +47,7 @@ #include #ifdef CONFIG_X86_5LEVEL -unsigned int pgtable_l5_enabled __ro_after_init; +unsigned int __pgtable_l5_enabled; unsigned int pgdir_shift __ro_after_init = 39; unsigned int ptrs_per_p4d __ro_after_init = 1; #endif @@ -734,7 +734,7 @@ void choose_random_location(unsigned long input, #ifdef CONFIG_X86_5LEVEL if (__read_cr4() & X86_CR4_LA57) { - pgtable_l5_enabled = 1; + __pgtable_l5_enabled = 1; pgdir_shift = 48; ptrs_per_p4d = 512; } diff --git a/arch/x86/boot/compressed/kaslr_64.c b/arch/x86/boot/compressed/kaslr_64.c index 522d11431433..748456c365f4 100644 --- a/arch/x86/boot/compressed/kaslr_64.c +++ b/arch/x86/boot/compressed/kaslr_64.c @@ -69,6 +69,8 @@ static struct alloc_pgt_data pgt_data; /* The top level page table entry pointer. */ static unsigned long top_level_pgt; +phys_addr_t physical_mask = (1ULL << __PHYSICAL_MASK_SHIFT) - 1; + /* * Mapping information structure passed to kernel_ident_mapping_init(). * Due to relocation, pointers must be assigned at run time not build time. @@ -81,6 +83,9 @@ void initialize_identity_maps(void) /* If running as an SEV guest, the encryption mask is required. */ set_sev_encryption_mask(); + /* Exclude the encryption mask from __PHYSICAL_MASK */ + physical_mask &= ~sme_me_mask; + /* Init mapping_info with run-time function/buffer pointers. */ mapping_info.alloc_pgt_page = alloc_pgt_page; mapping_info.context = &pgt_data; diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index 9e11be4cae19..a423bdb42686 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -12,10 +12,8 @@ #undef CONFIG_PARAVIRT_SPINLOCKS #undef CONFIG_KASAN -#ifdef CONFIG_X86_5LEVEL -/* cpu_feature_enabled() cannot be used that early */ -#define pgtable_l5_enabled __pgtable_l5_enabled -#endif +/* cpu_feature_enabled() cannot be used this early */ +#define USE_EARLY_PGTABLE_L5 #include #include diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index a362fa0b849c..8c5107545251 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -31,16 +31,23 @@ static char trampoline_save[TRAMPOLINE_32BIT_SIZE]; */ unsigned long *trampoline_32bit __section(.data); -struct paging_config paging_prepare(void) +extern struct boot_params *boot_params; +int cmdline_find_option_bool(const char *option); + +struct paging_config paging_prepare(void *rmode) { struct paging_config paging_config = {}; unsigned long bios_start, ebda_start; + /* Initialize boot_params. Required for cmdline_find_option_bool(). */ + boot_params = rmode; + /* * Check if LA57 is desired and supported. * - * There are two parts to the check: + * There are several parts to the check: * - if the kernel supports 5-level paging: CONFIG_X86_5LEVEL=y + * - if user asked to disable 5-level paging: no5lvl in cmdline * - if the machine supports 5-level paging: * + CPUID leaf 7 is supported * + the leaf has the feature bit set @@ -48,6 +55,7 @@ struct paging_config paging_prepare(void) * That's substitute for boot_cpu_has() in early boot code. */ if (IS_ENABLED(CONFIG_X86_5LEVEL) && + !cmdline_find_option_bool("no5lvl") && native_cpuid_eax(0) >= 7 && (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { paging_config.l5_required = 1; @@ -130,7 +138,7 @@ void cleanup_trampoline(void *pgtable) { void *trampoline_pgtable; - trampoline_pgtable = trampoline_32bit + TRAMPOLINE_32BIT_PGTABLE_OFFSET; + trampoline_pgtable = trampoline_32bit + TRAMPOLINE_32BIT_PGTABLE_OFFSET / sizeof(unsigned long); /* * Move the top level page table out of trampoline memory, diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile index 5f07333bb224..a450ad573dcb 100644 --- a/arch/x86/crypto/Makefile +++ b/arch/x86/crypto/Makefile @@ -15,7 +15,6 @@ obj-$(CONFIG_CRYPTO_GLUE_HELPER_X86) += glue_helper.o obj-$(CONFIG_CRYPTO_AES_586) += aes-i586.o obj-$(CONFIG_CRYPTO_TWOFISH_586) += twofish-i586.o -obj-$(CONFIG_CRYPTO_SALSA20_586) += salsa20-i586.o obj-$(CONFIG_CRYPTO_SERPENT_SSE2_586) += serpent-sse2-i586.o obj-$(CONFIG_CRYPTO_AES_X86_64) += aes-x86_64.o @@ -24,7 +23,6 @@ obj-$(CONFIG_CRYPTO_CAMELLIA_X86_64) += camellia-x86_64.o obj-$(CONFIG_CRYPTO_BLOWFISH_X86_64) += blowfish-x86_64.o obj-$(CONFIG_CRYPTO_TWOFISH_X86_64) += twofish-x86_64.o obj-$(CONFIG_CRYPTO_TWOFISH_X86_64_3WAY) += twofish-x86_64-3way.o -obj-$(CONFIG_CRYPTO_SALSA20_X86_64) += salsa20-x86_64.o obj-$(CONFIG_CRYPTO_CHACHA20_X86_64) += chacha20-x86_64.o obj-$(CONFIG_CRYPTO_SERPENT_SSE2_X86_64) += serpent-sse2-x86_64.o obj-$(CONFIG_CRYPTO_AES_NI_INTEL) += aesni-intel.o @@ -38,6 +36,16 @@ obj-$(CONFIG_CRYPTO_SHA512_SSSE3) += sha512-ssse3.o obj-$(CONFIG_CRYPTO_CRCT10DIF_PCLMUL) += crct10dif-pclmul.o obj-$(CONFIG_CRYPTO_POLY1305_X86_64) += poly1305-x86_64.o +obj-$(CONFIG_CRYPTO_AEGIS128_AESNI_SSE2) += aegis128-aesni.o +obj-$(CONFIG_CRYPTO_AEGIS128L_AESNI_SSE2) += aegis128l-aesni.o +obj-$(CONFIG_CRYPTO_AEGIS256_AESNI_SSE2) += aegis256-aesni.o + +obj-$(CONFIG_CRYPTO_MORUS640_GLUE) += morus640_glue.o +obj-$(CONFIG_CRYPTO_MORUS1280_GLUE) += morus1280_glue.o + +obj-$(CONFIG_CRYPTO_MORUS640_SSE2) += morus640-sse2.o +obj-$(CONFIG_CRYPTO_MORUS1280_SSE2) += morus1280-sse2.o + # These modules require assembler to support AVX. ifeq ($(avx_supported),yes) obj-$(CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64) += \ @@ -55,11 +63,12 @@ ifeq ($(avx2_supported),yes) obj-$(CONFIG_CRYPTO_SHA1_MB) += sha1-mb/ obj-$(CONFIG_CRYPTO_SHA256_MB) += sha256-mb/ obj-$(CONFIG_CRYPTO_SHA512_MB) += sha512-mb/ + + obj-$(CONFIG_CRYPTO_MORUS1280_AVX2) += morus1280-avx2.o endif aes-i586-y := aes-i586-asm_32.o aes_glue.o twofish-i586-y := twofish-i586-asm_32.o twofish_glue.o -salsa20-i586-y := salsa20-i586-asm_32.o salsa20_glue.o serpent-sse2-i586-y := serpent-sse2-i586-asm_32.o serpent_sse2_glue.o aes-x86_64-y := aes-x86_64-asm_64.o aes_glue.o @@ -68,10 +77,16 @@ camellia-x86_64-y := camellia-x86_64-asm_64.o camellia_glue.o blowfish-x86_64-y := blowfish-x86_64-asm_64.o blowfish_glue.o twofish-x86_64-y := twofish-x86_64-asm_64.o twofish_glue.o twofish-x86_64-3way-y := twofish-x86_64-asm_64-3way.o twofish_glue_3way.o -salsa20-x86_64-y := salsa20-x86_64-asm_64.o salsa20_glue.o chacha20-x86_64-y := chacha20-ssse3-x86_64.o chacha20_glue.o serpent-sse2-x86_64-y := serpent-sse2-x86_64-asm_64.o serpent_sse2_glue.o +aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o +aegis128l-aesni-y := aegis128l-aesni-asm.o aegis128l-aesni-glue.o +aegis256-aesni-y := aegis256-aesni-asm.o aegis256-aesni-glue.o + +morus640-sse2-y := morus640-sse2-asm.o morus640-sse2-glue.o +morus1280-sse2-y := morus1280-sse2-asm.o morus1280-sse2-glue.o + ifeq ($(avx_supported),yes) camellia-aesni-avx-x86_64-y := camellia-aesni-avx-asm_64.o \ camellia_aesni_avx_glue.o @@ -87,6 +102,8 @@ ifeq ($(avx2_supported),yes) camellia-aesni-avx2-y := camellia-aesni-avx2-asm_64.o camellia_aesni_avx2_glue.o chacha20-x86_64-y += chacha20-avx2-x86_64.o serpent-avx2-y := serpent-avx2-asm_64.o serpent_avx2_glue.o + + morus1280-avx2-y := morus1280-avx2-asm.o morus1280-avx2-glue.o endif aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o fpu.o diff --git a/arch/x86/crypto/aegis128-aesni-asm.S b/arch/x86/crypto/aegis128-aesni-asm.S new file mode 100644 index 000000000000..9254e0b6cc06 --- /dev/null +++ b/arch/x86/crypto/aegis128-aesni-asm.S @@ -0,0 +1,749 @@ +/* + * AES-NI + SSE2 implementation of AEGIS-128 + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation. + */ + +#include +#include + +#define STATE0 %xmm0 +#define STATE1 %xmm1 +#define STATE2 %xmm2 +#define STATE3 %xmm3 +#define STATE4 %xmm4 +#define KEY %xmm5 +#define MSG %xmm5 +#define T0 %xmm6 +#define T1 %xmm7 + +#define STATEP %rdi +#define LEN %rsi +#define SRC %rdx +#define DST %rcx + +.section .rodata.cst16.aegis128_const, "aM", @progbits, 32 +.align 16 +.Laegis128_const_0: + .byte 0x00, 0x01, 0x01, 0x02, 0x03, 0x05, 0x08, 0x0d + .byte 0x15, 0x22, 0x37, 0x59, 0x90, 0xe9, 0x79, 0x62 +.Laegis128_const_1: + .byte 0xdb, 0x3d, 0x18, 0x55, 0x6d, 0xc2, 0x2f, 0xf1 + .byte 0x20, 0x11, 0x31, 0x42, 0x73, 0xb5, 0x28, 0xdd + +.section .rodata.cst16.aegis128_counter, "aM", @progbits, 16 +.align 16 +.Laegis128_counter: + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f + +.text + +/* + * aegis128_update + * input: + * STATE[0-4] - input state + * output: + * STATE[0-4] - output state (shifted positions) + * changed: + * T0 + */ +.macro aegis128_update + movdqa STATE4, T0 + aesenc STATE0, STATE4 + aesenc STATE1, STATE0 + aesenc STATE2, STATE1 + aesenc STATE3, STATE2 + aesenc T0, STATE3 +.endm + +/* + * __load_partial: internal ABI + * input: + * LEN - bytes + * SRC - src + * output: + * MSG - message block + * changed: + * T0 + * %r8 + * %r9 + */ +__load_partial: + xor %r9, %r9 + pxor MSG, MSG + + mov LEN, %r8 + and $0x1, %r8 + jz .Lld_partial_1 + + mov LEN, %r8 + and $0x1E, %r8 + add SRC, %r8 + mov (%r8), %r9b + +.Lld_partial_1: + mov LEN, %r8 + and $0x2, %r8 + jz .Lld_partial_2 + + mov LEN, %r8 + and $0x1C, %r8 + add SRC, %r8 + shl $0x10, %r9 + mov (%r8), %r9w + +.Lld_partial_2: + mov LEN, %r8 + and $0x4, %r8 + jz .Lld_partial_4 + + mov LEN, %r8 + and $0x18, %r8 + add SRC, %r8 + shl $32, %r9 + mov (%r8), %r8d + xor %r8, %r9 + +.Lld_partial_4: + movq %r9, MSG + + mov LEN, %r8 + and $0x8, %r8 + jz .Lld_partial_8 + + mov LEN, %r8 + and $0x10, %r8 + add SRC, %r8 + pslldq $8, MSG + movq (%r8), T0 + pxor T0, MSG + +.Lld_partial_8: + ret +ENDPROC(__load_partial) + +/* + * __store_partial: internal ABI + * input: + * LEN - bytes + * DST - dst + * output: + * T0 - message block + * changed: + * %r8 + * %r9 + * %r10 + */ +__store_partial: + mov LEN, %r8 + mov DST, %r9 + + movq T0, %r10 + + cmp $8, %r8 + jl .Lst_partial_8 + + mov %r10, (%r9) + psrldq $8, T0 + movq T0, %r10 + + sub $8, %r8 + add $8, %r9 + +.Lst_partial_8: + cmp $4, %r8 + jl .Lst_partial_4 + + mov %r10d, (%r9) + shr $32, %r10 + + sub $4, %r8 + add $4, %r9 + +.Lst_partial_4: + cmp $2, %r8 + jl .Lst_partial_2 + + mov %r10w, (%r9) + shr $0x10, %r10 + + sub $2, %r8 + add $2, %r9 + +.Lst_partial_2: + cmp $1, %r8 + jl .Lst_partial_1 + + mov %r10b, (%r9) + +.Lst_partial_1: + ret +ENDPROC(__store_partial) + +/* + * void crypto_aegis128_aesni_init(void *state, const void *key, const void *iv); + */ +ENTRY(crypto_aegis128_aesni_init) + FRAME_BEGIN + + /* load IV: */ + movdqu (%rdx), T1 + + /* load key: */ + movdqa (%rsi), KEY + pxor KEY, T1 + movdqa T1, STATE0 + movdqa KEY, STATE3 + movdqa KEY, STATE4 + + /* load the constants: */ + movdqa .Laegis128_const_0, STATE2 + movdqa .Laegis128_const_1, STATE1 + pxor STATE2, STATE3 + pxor STATE1, STATE4 + + /* update 10 times with KEY / KEY xor IV: */ + aegis128_update; pxor KEY, STATE4 + aegis128_update; pxor T1, STATE3 + aegis128_update; pxor KEY, STATE2 + aegis128_update; pxor T1, STATE1 + aegis128_update; pxor KEY, STATE0 + aegis128_update; pxor T1, STATE4 + aegis128_update; pxor KEY, STATE3 + aegis128_update; pxor T1, STATE2 + aegis128_update; pxor KEY, STATE1 + aegis128_update; pxor T1, STATE0 + + /* store the state: */ + movdqu STATE0, 0x00(STATEP) + movdqu STATE1, 0x10(STATEP) + movdqu STATE2, 0x20(STATEP) + movdqu STATE3, 0x30(STATEP) + movdqu STATE4, 0x40(STATEP) + + FRAME_END + ret +ENDPROC(crypto_aegis128_aesni_init) + +/* + * void crypto_aegis128_aesni_ad(void *state, unsigned int length, + * const void *data); + */ +ENTRY(crypto_aegis128_aesni_ad) + FRAME_BEGIN + + cmp $0x10, LEN + jb .Lad_out + + /* load the state: */ + movdqu 0x00(STATEP), STATE0 + movdqu 0x10(STATEP), STATE1 + movdqu 0x20(STATEP), STATE2 + movdqu 0x30(STATEP), STATE3 + movdqu 0x40(STATEP), STATE4 + + mov SRC, %r8 + and $0xF, %r8 + jnz .Lad_u_loop + +.align 8 +.Lad_a_loop: + movdqa 0x00(SRC), MSG + aegis128_update + pxor MSG, STATE4 + sub $0x10, LEN + cmp $0x10, LEN + jl .Lad_out_1 + + movdqa 0x10(SRC), MSG + aegis128_update + pxor MSG, STATE3 + sub $0x10, LEN + cmp $0x10, LEN + jl .Lad_out_2 + + movdqa 0x20(SRC), MSG + aegis128_update + pxor MSG, STATE2 + sub $0x10, LEN + cmp $0x10, LEN + jl .Lad_out_3 + + movdqa 0x30(SRC), MSG + aegis128_update + pxor MSG, STATE1 + sub $0x10, LEN + cmp $0x10, LEN + jl .Lad_out_4 + + movdqa 0x40(SRC), MSG + aegis128_update + pxor MSG, STATE0 + sub $0x10, LEN + cmp $0x10, LEN + jl .Lad_out_0 + + add $0x50, SRC + jmp .Lad_a_loop + +.align 8 +.Lad_u_loop: + movdqu 0x00(SRC), MSG + aegis128_update + pxor MSG, STATE4 + sub $0x10, LEN + cmp $0x10, LEN + jl .Lad_out_1 + + movdqu 0x10(SRC), MSG + aegis128_update + pxor MSG, STATE3 + sub $0x10, LEN + cmp $0x10, LEN + jl .Lad_out_2 + + movdqu 0x20(SRC), MSG + aegis128_update + pxor MSG, STATE2 + sub $0x10, LEN + cmp $0x10, LEN + jl .Lad_out_3 + + movdqu 0x30(SRC), MSG + aegis128_update + pxor MSG, STATE1 + sub $0x10, LEN + cmp $0x10, LEN + jl .Lad_out_4 + + movdqu 0x40(SRC), MSG + aegis128_update + pxor MSG, STATE0 + sub $0x10, LEN + cmp $0x10, LEN + jl .Lad_out_0 + + add $0x50, SRC + jmp .Lad_u_loop + + /* store the state: */ +.Lad_out_0: + movdqu STATE0, 0x00(STATEP) + movdqu STATE1, 0x10(STATEP) + movdqu STATE2, 0x20(STATEP) + movdqu STATE3, 0x30(STATEP) + movdqu STATE4, 0x40(STATEP) + FRAME_END + ret + +.Lad_out_1: + movdqu STATE4, 0x00(STATEP) + movdqu STATE0, 0x10(STATEP) + movdqu STATE1, 0x20(STATEP) + movdqu STATE2, 0x30(STATEP) + movdqu STATE3, 0x40(STATEP) + FRAME_END + ret + +.Lad_out_2: + movdqu STATE3, 0x00(STATEP) + movdqu STATE4, 0x10(STATEP) + movdqu STATE0, 0x20(STATEP) + movdqu STATE1, 0x30(STATEP) + movdqu STATE2, 0x40(STATEP) + FRAME_END + ret + +.Lad_out_3: + movdqu STATE2, 0x00(STATEP) + movdqu STATE3, 0x10(STATEP) + movdqu STATE4, 0x20(STATEP) + movdqu STATE0, 0x30(STATEP) + movdqu STATE1, 0x40(STATEP) + FRAME_END + ret + +.Lad_out_4: + movdqu STATE1, 0x00(STATEP) + movdqu STATE2, 0x10(STATEP) + movdqu STATE3, 0x20(STATEP) + movdqu STATE4, 0x30(STATEP) + movdqu STATE0, 0x40(STATEP) + FRAME_END + ret + +.Lad_out: + FRAME_END + ret +ENDPROC(crypto_aegis128_aesni_ad) + +.macro encrypt_block a s0 s1 s2 s3 s4 i + movdq\a (\i * 0x10)(SRC), MSG + movdqa MSG, T0 + pxor \s1, T0 + pxor \s4, T0 + movdqa \s2, T1 + pand \s3, T1 + pxor T1, T0 + movdq\a T0, (\i * 0x10)(DST) + + aegis128_update + pxor MSG, \s4 + + sub $0x10, LEN + cmp $0x10, LEN + jl .Lenc_out_\i +.endm + +/* + * void crypto_aegis128_aesni_enc(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis128_aesni_enc) + FRAME_BEGIN + + cmp $0x10, LEN + jb .Lenc_out + + /* load the state: */ + movdqu 0x00(STATEP), STATE0 + movdqu 0x10(STATEP), STATE1 + movdqu 0x20(STATEP), STATE2 + movdqu 0x30(STATEP), STATE3 + movdqu 0x40(STATEP), STATE4 + + mov SRC, %r8 + or DST, %r8 + and $0xF, %r8 + jnz .Lenc_u_loop + +.align 8 +.Lenc_a_loop: + encrypt_block a STATE0 STATE1 STATE2 STATE3 STATE4 0 + encrypt_block a STATE4 STATE0 STATE1 STATE2 STATE3 1 + encrypt_block a STATE3 STATE4 STATE0 STATE1 STATE2 2 + encrypt_block a STATE2 STATE3 STATE4 STATE0 STATE1 3 + encrypt_block a STATE1 STATE2 STATE3 STATE4 STATE0 4 + + add $0x50, SRC + add $0x50, DST + jmp .Lenc_a_loop + +.align 8 +.Lenc_u_loop: + encrypt_block u STATE0 STATE1 STATE2 STATE3 STATE4 0 + encrypt_block u STATE4 STATE0 STATE1 STATE2 STATE3 1 + encrypt_block u STATE3 STATE4 STATE0 STATE1 STATE2 2 + encrypt_block u STATE2 STATE3 STATE4 STATE0 STATE1 3 + encrypt_block u STATE1 STATE2 STATE3 STATE4 STATE0 4 + + add $0x50, SRC + add $0x50, DST + jmp .Lenc_u_loop + + /* store the state: */ +.Lenc_out_0: + movdqu STATE4, 0x00(STATEP) + movdqu STATE0, 0x10(STATEP) + movdqu STATE1, 0x20(STATEP) + movdqu STATE2, 0x30(STATEP) + movdqu STATE3, 0x40(STATEP) + FRAME_END + ret + +.Lenc_out_1: + movdqu STATE3, 0x00(STATEP) + movdqu STATE4, 0x10(STATEP) + movdqu STATE0, 0x20(STATEP) + movdqu STATE1, 0x30(STATEP) + movdqu STATE2, 0x40(STATEP) + FRAME_END + ret + +.Lenc_out_2: + movdqu STATE2, 0x00(STATEP) + movdqu STATE3, 0x10(STATEP) + movdqu STATE4, 0x20(STATEP) + movdqu STATE0, 0x30(STATEP) + movdqu STATE1, 0x40(STATEP) + FRAME_END + ret + +.Lenc_out_3: + movdqu STATE1, 0x00(STATEP) + movdqu STATE2, 0x10(STATEP) + movdqu STATE3, 0x20(STATEP) + movdqu STATE4, 0x30(STATEP) + movdqu STATE0, 0x40(STATEP) + FRAME_END + ret + +.Lenc_out_4: + movdqu STATE0, 0x00(STATEP) + movdqu STATE1, 0x10(STATEP) + movdqu STATE2, 0x20(STATEP) + movdqu STATE3, 0x30(STATEP) + movdqu STATE4, 0x40(STATEP) + FRAME_END + ret + +.Lenc_out: + FRAME_END + ret +ENDPROC(crypto_aegis128_aesni_enc) + +/* + * void crypto_aegis128_aesni_enc_tail(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis128_aesni_enc_tail) + FRAME_BEGIN + + /* load the state: */ + movdqu 0x00(STATEP), STATE0 + movdqu 0x10(STATEP), STATE1 + movdqu 0x20(STATEP), STATE2 + movdqu 0x30(STATEP), STATE3 + movdqu 0x40(STATEP), STATE4 + + /* encrypt message: */ + call __load_partial + + movdqa MSG, T0 + pxor STATE1, T0 + pxor STATE4, T0 + movdqa STATE2, T1 + pand STATE3, T1 + pxor T1, T0 + + call __store_partial + + aegis128_update + pxor MSG, STATE4 + + /* store the state: */ + movdqu STATE4, 0x00(STATEP) + movdqu STATE0, 0x10(STATEP) + movdqu STATE1, 0x20(STATEP) + movdqu STATE2, 0x30(STATEP) + movdqu STATE3, 0x40(STATEP) + + FRAME_END +ENDPROC(crypto_aegis128_aesni_enc_tail) + +.macro decrypt_block a s0 s1 s2 s3 s4 i + movdq\a (\i * 0x10)(SRC), MSG + pxor \s1, MSG + pxor \s4, MSG + movdqa \s2, T1 + pand \s3, T1 + pxor T1, MSG + movdq\a MSG, (\i * 0x10)(DST) + + aegis128_update + pxor MSG, \s4 + + sub $0x10, LEN + cmp $0x10, LEN + jl .Ldec_out_\i +.endm + +/* + * void crypto_aegis128_aesni_dec(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis128_aesni_dec) + FRAME_BEGIN + + cmp $0x10, LEN + jb .Ldec_out + + /* load the state: */ + movdqu 0x00(STATEP), STATE0 + movdqu 0x10(STATEP), STATE1 + movdqu 0x20(STATEP), STATE2 + movdqu 0x30(STATEP), STATE3 + movdqu 0x40(STATEP), STATE4 + + mov SRC, %r8 + or DST, %r8 + and $0xF, %r8 + jnz .Ldec_u_loop + +.align 8 +.Ldec_a_loop: + decrypt_block a STATE0 STATE1 STATE2 STATE3 STATE4 0 + decrypt_block a STATE4 STATE0 STATE1 STATE2 STATE3 1 + decrypt_block a STATE3 STATE4 STATE0 STATE1 STATE2 2 + decrypt_block a STATE2 STATE3 STATE4 STATE0 STATE1 3 + decrypt_block a STATE1 STATE2 STATE3 STATE4 STATE0 4 + + add $0x50, SRC + add $0x50, DST + jmp .Ldec_a_loop + +.align 8 +.Ldec_u_loop: + decrypt_block u STATE0 STATE1 STATE2 STATE3 STATE4 0 + decrypt_block u STATE4 STATE0 STATE1 STATE2 STATE3 1 + decrypt_block u STATE3 STATE4 STATE0 STATE1 STATE2 2 + decrypt_block u STATE2 STATE3 STATE4 STATE0 STATE1 3 + decrypt_block u STATE1 STATE2 STATE3 STATE4 STATE0 4 + + add $0x50, SRC + add $0x50, DST + jmp .Ldec_u_loop + + /* store the state: */ +.Ldec_out_0: + movdqu STATE4, 0x00(STATEP) + movdqu STATE0, 0x10(STATEP) + movdqu STATE1, 0x20(STATEP) + movdqu STATE2, 0x30(STATEP) + movdqu STATE3, 0x40(STATEP) + FRAME_END + ret + +.Ldec_out_1: + movdqu STATE3, 0x00(STATEP) + movdqu STATE4, 0x10(STATEP) + movdqu STATE0, 0x20(STATEP) + movdqu STATE1, 0x30(STATEP) + movdqu STATE2, 0x40(STATEP) + FRAME_END + ret + +.Ldec_out_2: + movdqu STATE2, 0x00(STATEP) + movdqu STATE3, 0x10(STATEP) + movdqu STATE4, 0x20(STATEP) + movdqu STATE0, 0x30(STATEP) + movdqu STATE1, 0x40(STATEP) + FRAME_END + ret + +.Ldec_out_3: + movdqu STATE1, 0x00(STATEP) + movdqu STATE2, 0x10(STATEP) + movdqu STATE3, 0x20(STATEP) + movdqu STATE4, 0x30(STATEP) + movdqu STATE0, 0x40(STATEP) + FRAME_END + ret + +.Ldec_out_4: + movdqu STATE0, 0x00(STATEP) + movdqu STATE1, 0x10(STATEP) + movdqu STATE2, 0x20(STATEP) + movdqu STATE3, 0x30(STATEP) + movdqu STATE4, 0x40(STATEP) + FRAME_END + ret + +.Ldec_out: + FRAME_END + ret +ENDPROC(crypto_aegis128_aesni_dec) + +/* + * void crypto_aegis128_aesni_dec_tail(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis128_aesni_dec_tail) + FRAME_BEGIN + + /* load the state: */ + movdqu 0x00(STATEP), STATE0 + movdqu 0x10(STATEP), STATE1 + movdqu 0x20(STATEP), STATE2 + movdqu 0x30(STATEP), STATE3 + movdqu 0x40(STATEP), STATE4 + + /* decrypt message: */ + call __load_partial + + pxor STATE1, MSG + pxor STATE4, MSG + movdqa STATE2, T1 + pand STATE3, T1 + pxor T1, MSG + + movdqa MSG, T0 + call __store_partial + + /* mask with byte count: */ + movq LEN, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + movdqa .Laegis128_counter, T1 + pcmpgtb T1, T0 + pand T0, MSG + + aegis128_update + pxor MSG, STATE4 + + /* store the state: */ + movdqu STATE4, 0x00(STATEP) + movdqu STATE0, 0x10(STATEP) + movdqu STATE1, 0x20(STATEP) + movdqu STATE2, 0x30(STATEP) + movdqu STATE3, 0x40(STATEP) + + FRAME_END + ret +ENDPROC(crypto_aegis128_aesni_dec_tail) + +/* + * void crypto_aegis128_aesni_final(void *state, void *tag_xor, + * u64 assoclen, u64 cryptlen); + */ +ENTRY(crypto_aegis128_aesni_final) + FRAME_BEGIN + + /* load the state: */ + movdqu 0x00(STATEP), STATE0 + movdqu 0x10(STATEP), STATE1 + movdqu 0x20(STATEP), STATE2 + movdqu 0x30(STATEP), STATE3 + movdqu 0x40(STATEP), STATE4 + + /* prepare length block: */ + movq %rdx, MSG + movq %rcx, T0 + pslldq $8, T0 + pxor T0, MSG + psllq $3, MSG /* multiply by 8 (to get bit count) */ + + pxor STATE3, MSG + + /* update state: */ + aegis128_update; pxor MSG, STATE4 + aegis128_update; pxor MSG, STATE3 + aegis128_update; pxor MSG, STATE2 + aegis128_update; pxor MSG, STATE1 + aegis128_update; pxor MSG, STATE0 + aegis128_update; pxor MSG, STATE4 + aegis128_update; pxor MSG, STATE3 + + /* xor tag: */ + movdqu (%rsi), MSG + + pxor STATE0, MSG + pxor STATE1, MSG + pxor STATE2, MSG + pxor STATE3, MSG + pxor STATE4, MSG + + movdqu MSG, (%rsi) + + FRAME_END + ret +ENDPROC(crypto_aegis128_aesni_final) diff --git a/arch/x86/crypto/aegis128-aesni-glue.c b/arch/x86/crypto/aegis128-aesni-glue.c new file mode 100644 index 000000000000..5de7c0d46edf --- /dev/null +++ b/arch/x86/crypto/aegis128-aesni-glue.c @@ -0,0 +1,407 @@ +/* + * The AEGIS-128 Authenticated-Encryption Algorithm + * Glue for AES-NI + SSE2 implementation + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include +#include +#include + +#define AEGIS128_BLOCK_ALIGN 16 +#define AEGIS128_BLOCK_SIZE 16 +#define AEGIS128_NONCE_SIZE 16 +#define AEGIS128_STATE_BLOCKS 5 +#define AEGIS128_KEY_SIZE 16 +#define AEGIS128_MIN_AUTH_SIZE 8 +#define AEGIS128_MAX_AUTH_SIZE 16 + +asmlinkage void crypto_aegis128_aesni_init(void *state, void *key, void *iv); + +asmlinkage void crypto_aegis128_aesni_ad( + void *state, unsigned int length, const void *data); + +asmlinkage void crypto_aegis128_aesni_enc( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis128_aesni_dec( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis128_aesni_enc_tail( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis128_aesni_dec_tail( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis128_aesni_final( + void *state, void *tag_xor, unsigned int cryptlen, + unsigned int assoclen); + +struct aegis_block { + u8 bytes[AEGIS128_BLOCK_SIZE] __aligned(AEGIS128_BLOCK_ALIGN); +}; + +struct aegis_state { + struct aegis_block blocks[AEGIS128_STATE_BLOCKS]; +}; + +struct aegis_ctx { + struct aegis_block key; +}; + +struct aegis_crypt_ops { + int (*skcipher_walk_init)(struct skcipher_walk *walk, + struct aead_request *req, bool atomic); + + void (*crypt_blocks)(void *state, unsigned int length, const void *src, + void *dst); + void (*crypt_tail)(void *state, unsigned int length, const void *src, + void *dst); +}; + +static void crypto_aegis128_aesni_process_ad( + struct aegis_state *state, struct scatterlist *sg_src, + unsigned int assoclen) +{ + struct scatter_walk walk; + struct aegis_block buf; + unsigned int pos = 0; + + scatterwalk_start(&walk, sg_src); + while (assoclen != 0) { + unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int left = size; + void *mapped = scatterwalk_map(&walk); + const u8 *src = (const u8 *)mapped; + + if (pos + size >= AEGIS128_BLOCK_SIZE) { + if (pos > 0) { + unsigned int fill = AEGIS128_BLOCK_SIZE - pos; + memcpy(buf.bytes + pos, src, fill); + crypto_aegis128_aesni_ad(state, + AEGIS128_BLOCK_SIZE, + buf.bytes); + pos = 0; + left -= fill; + src += fill; + } + + crypto_aegis128_aesni_ad(state, left, src); + + src += left & ~(AEGIS128_BLOCK_SIZE - 1); + left &= AEGIS128_BLOCK_SIZE - 1; + } + + memcpy(buf.bytes + pos, src, left); + pos += left; + assoclen -= size; + + scatterwalk_unmap(mapped); + scatterwalk_advance(&walk, size); + scatterwalk_done(&walk, 0, assoclen); + } + + if (pos > 0) { + memset(buf.bytes + pos, 0, AEGIS128_BLOCK_SIZE - pos); + crypto_aegis128_aesni_ad(state, AEGIS128_BLOCK_SIZE, buf.bytes); + } +} + +static void crypto_aegis128_aesni_process_crypt( + struct aegis_state *state, struct aead_request *req, + const struct aegis_crypt_ops *ops) +{ + struct skcipher_walk walk; + u8 *src, *dst; + unsigned int chunksize, base; + + ops->skcipher_walk_init(&walk, req, false); + + while (walk.nbytes) { + src = walk.src.virt.addr; + dst = walk.dst.virt.addr; + chunksize = walk.nbytes; + + ops->crypt_blocks(state, chunksize, src, dst); + + base = chunksize & ~(AEGIS128_BLOCK_SIZE - 1); + src += base; + dst += base; + chunksize &= AEGIS128_BLOCK_SIZE - 1; + + if (chunksize > 0) + ops->crypt_tail(state, chunksize, src, dst); + + skcipher_walk_done(&walk, 0); + } +} + +static struct aegis_ctx *crypto_aegis128_aesni_ctx(struct crypto_aead *aead) +{ + u8 *ctx = crypto_aead_ctx(aead); + ctx = PTR_ALIGN(ctx, __alignof__(struct aegis_ctx)); + return (void *)ctx; +} + +static int crypto_aegis128_aesni_setkey(struct crypto_aead *aead, const u8 *key, + unsigned int keylen) +{ + struct aegis_ctx *ctx = crypto_aegis128_aesni_ctx(aead); + + if (keylen != AEGIS128_KEY_SIZE) { + crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; + } + + memcpy(ctx->key.bytes, key, AEGIS128_KEY_SIZE); + + return 0; +} + +static int crypto_aegis128_aesni_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + if (authsize > AEGIS128_MAX_AUTH_SIZE) + return -EINVAL; + if (authsize < AEGIS128_MIN_AUTH_SIZE) + return -EINVAL; + return 0; +} + +static void crypto_aegis128_aesni_crypt(struct aead_request *req, + struct aegis_block *tag_xor, + unsigned int cryptlen, + const struct aegis_crypt_ops *ops) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_ctx *ctx = crypto_aegis128_aesni_ctx(tfm); + struct aegis_state state; + + kernel_fpu_begin(); + + crypto_aegis128_aesni_init(&state, ctx->key.bytes, req->iv); + crypto_aegis128_aesni_process_ad(&state, req->src, req->assoclen); + crypto_aegis128_aesni_process_crypt(&state, req, ops); + crypto_aegis128_aesni_final(&state, tag_xor, req->assoclen, cryptlen); + + kernel_fpu_end(); +} + +static int crypto_aegis128_aesni_encrypt(struct aead_request *req) +{ + static const struct aegis_crypt_ops OPS = { + .skcipher_walk_init = skcipher_walk_aead_encrypt, + .crypt_blocks = crypto_aegis128_aesni_enc, + .crypt_tail = crypto_aegis128_aesni_enc_tail, + }; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_block tag = {}; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen; + + crypto_aegis128_aesni_crypt(req, &tag, cryptlen, &OPS); + + scatterwalk_map_and_copy(tag.bytes, req->dst, + req->assoclen + cryptlen, authsize, 1); + return 0; +} + +static int crypto_aegis128_aesni_decrypt(struct aead_request *req) +{ + static const struct aegis_block zeros = {}; + + static const struct aegis_crypt_ops OPS = { + .skcipher_walk_init = skcipher_walk_aead_decrypt, + .crypt_blocks = crypto_aegis128_aesni_dec, + .crypt_tail = crypto_aegis128_aesni_dec_tail, + }; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_block tag; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen - authsize; + + scatterwalk_map_and_copy(tag.bytes, req->src, + req->assoclen + cryptlen, authsize, 0); + + crypto_aegis128_aesni_crypt(req, &tag, cryptlen, &OPS); + + return crypto_memneq(tag.bytes, zeros.bytes, authsize) ? -EBADMSG : 0; +} + +static int crypto_aegis128_aesni_init_tfm(struct crypto_aead *aead) +{ + return 0; +} + +static void crypto_aegis128_aesni_exit_tfm(struct crypto_aead *aead) +{ +} + +static int cryptd_aegis128_aesni_setkey(struct crypto_aead *aead, + const u8 *key, unsigned int keylen) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + return crypto_aead_setkey(&cryptd_tfm->base, key, keylen); +} + +static int cryptd_aegis128_aesni_setauthsize(struct crypto_aead *aead, + unsigned int authsize) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + return crypto_aead_setauthsize(&cryptd_tfm->base, authsize); +} + +static int cryptd_aegis128_aesni_encrypt(struct aead_request *req) +{ + struct crypto_aead *aead = crypto_aead_reqtfm(req); + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + aead = &cryptd_tfm->base; + if (irq_fpu_usable() && (!in_atomic() || + !cryptd_aead_queued(cryptd_tfm))) + aead = cryptd_aead_child(cryptd_tfm); + + aead_request_set_tfm(req, aead); + + return crypto_aead_encrypt(req); +} + +static int cryptd_aegis128_aesni_decrypt(struct aead_request *req) +{ + struct crypto_aead *aead = crypto_aead_reqtfm(req); + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + aead = &cryptd_tfm->base; + if (irq_fpu_usable() && (!in_atomic() || + !cryptd_aead_queued(cryptd_tfm))) + aead = cryptd_aead_child(cryptd_tfm); + + aead_request_set_tfm(req, aead); + + return crypto_aead_decrypt(req); +} + +static int cryptd_aegis128_aesni_init_tfm(struct crypto_aead *aead) +{ + struct cryptd_aead *cryptd_tfm; + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + + cryptd_tfm = cryptd_alloc_aead("__aegis128-aesni", CRYPTO_ALG_INTERNAL, + CRYPTO_ALG_INTERNAL); + if (IS_ERR(cryptd_tfm)) + return PTR_ERR(cryptd_tfm); + + *ctx = cryptd_tfm; + crypto_aead_set_reqsize(aead, crypto_aead_reqsize(&cryptd_tfm->base)); + return 0; +} + +static void cryptd_aegis128_aesni_exit_tfm(struct crypto_aead *aead) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + + cryptd_free_aead(*ctx); +} + +static struct aead_alg crypto_aegis128_aesni_alg[] = { + { + .setkey = crypto_aegis128_aesni_setkey, + .setauthsize = crypto_aegis128_aesni_setauthsize, + .encrypt = crypto_aegis128_aesni_encrypt, + .decrypt = crypto_aegis128_aesni_decrypt, + .init = crypto_aegis128_aesni_init_tfm, + .exit = crypto_aegis128_aesni_exit_tfm, + + .ivsize = AEGIS128_NONCE_SIZE, + .maxauthsize = AEGIS128_MAX_AUTH_SIZE, + .chunksize = AEGIS128_BLOCK_SIZE, + + .base = { + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct aegis_ctx) + + __alignof__(struct aegis_ctx), + .cra_alignmask = 0, + + .cra_name = "__aegis128", + .cra_driver_name = "__aegis128-aesni", + + .cra_module = THIS_MODULE, + } + }, { + .setkey = cryptd_aegis128_aesni_setkey, + .setauthsize = cryptd_aegis128_aesni_setauthsize, + .encrypt = cryptd_aegis128_aesni_encrypt, + .decrypt = cryptd_aegis128_aesni_decrypt, + .init = cryptd_aegis128_aesni_init_tfm, + .exit = cryptd_aegis128_aesni_exit_tfm, + + .ivsize = AEGIS128_NONCE_SIZE, + .maxauthsize = AEGIS128_MAX_AUTH_SIZE, + .chunksize = AEGIS128_BLOCK_SIZE, + + .base = { + .cra_flags = CRYPTO_ALG_ASYNC, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct cryptd_aead *), + .cra_alignmask = 0, + + .cra_priority = 400, + + .cra_name = "aegis128", + .cra_driver_name = "aegis128-aesni", + + .cra_module = THIS_MODULE, + } + } +}; + +static const struct x86_cpu_id aesni_cpu_id[] = { + X86_FEATURE_MATCH(X86_FEATURE_AES), + X86_FEATURE_MATCH(X86_FEATURE_XMM2), + {} +}; +MODULE_DEVICE_TABLE(x86cpu, aesni_cpu_id); + +static int __init crypto_aegis128_aesni_module_init(void) +{ + if (!x86_match_cpu(aesni_cpu_id)) + return -ENODEV; + + return crypto_register_aeads(crypto_aegis128_aesni_alg, + ARRAY_SIZE(crypto_aegis128_aesni_alg)); +} + +static void __exit crypto_aegis128_aesni_module_exit(void) +{ + crypto_unregister_aeads(crypto_aegis128_aesni_alg, + ARRAY_SIZE(crypto_aegis128_aesni_alg)); +} + +module_init(crypto_aegis128_aesni_module_init); +module_exit(crypto_aegis128_aesni_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("AEGIS-128 AEAD algorithm -- AESNI+SSE2 implementation"); +MODULE_ALIAS_CRYPTO("aegis128"); +MODULE_ALIAS_CRYPTO("aegis128-aesni"); diff --git a/arch/x86/crypto/aegis128l-aesni-asm.S b/arch/x86/crypto/aegis128l-aesni-asm.S new file mode 100644 index 000000000000..9263c344f2c7 --- /dev/null +++ b/arch/x86/crypto/aegis128l-aesni-asm.S @@ -0,0 +1,825 @@ +/* + * AES-NI + SSE2 implementation of AEGIS-128L + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation. + */ + +#include +#include + +#define STATE0 %xmm0 +#define STATE1 %xmm1 +#define STATE2 %xmm2 +#define STATE3 %xmm3 +#define STATE4 %xmm4 +#define STATE5 %xmm5 +#define STATE6 %xmm6 +#define STATE7 %xmm7 +#define MSG0 %xmm8 +#define MSG1 %xmm9 +#define T0 %xmm10 +#define T1 %xmm11 +#define T2 %xmm12 +#define T3 %xmm13 + +#define STATEP %rdi +#define LEN %rsi +#define SRC %rdx +#define DST %rcx + +.section .rodata.cst16.aegis128l_const, "aM", @progbits, 32 +.align 16 +.Laegis128l_const_0: + .byte 0x00, 0x01, 0x01, 0x02, 0x03, 0x05, 0x08, 0x0d + .byte 0x15, 0x22, 0x37, 0x59, 0x90, 0xe9, 0x79, 0x62 +.Laegis128l_const_1: + .byte 0xdb, 0x3d, 0x18, 0x55, 0x6d, 0xc2, 0x2f, 0xf1 + .byte 0x20, 0x11, 0x31, 0x42, 0x73, 0xb5, 0x28, 0xdd + +.section .rodata.cst16.aegis128l_counter, "aM", @progbits, 16 +.align 16 +.Laegis128l_counter0: + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f +.Laegis128l_counter1: + .byte 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17 + .byte 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f + +.text + +/* + * __load_partial: internal ABI + * input: + * LEN - bytes + * SRC - src + * output: + * MSG0 - first message block + * MSG1 - second message block + * changed: + * T0 + * %r8 + * %r9 + */ +__load_partial: + xor %r9, %r9 + pxor MSG0, MSG0 + pxor MSG1, MSG1 + + mov LEN, %r8 + and $0x1, %r8 + jz .Lld_partial_1 + + mov LEN, %r8 + and $0x1E, %r8 + add SRC, %r8 + mov (%r8), %r9b + +.Lld_partial_1: + mov LEN, %r8 + and $0x2, %r8 + jz .Lld_partial_2 + + mov LEN, %r8 + and $0x1C, %r8 + add SRC, %r8 + shl $0x10, %r9 + mov (%r8), %r9w + +.Lld_partial_2: + mov LEN, %r8 + and $0x4, %r8 + jz .Lld_partial_4 + + mov LEN, %r8 + and $0x18, %r8 + add SRC, %r8 + shl $32, %r9 + mov (%r8), %r8d + xor %r8, %r9 + +.Lld_partial_4: + movq %r9, MSG0 + + mov LEN, %r8 + and $0x8, %r8 + jz .Lld_partial_8 + + mov LEN, %r8 + and $0x10, %r8 + add SRC, %r8 + pslldq $8, MSG0 + movq (%r8), T0 + pxor T0, MSG0 + +.Lld_partial_8: + mov LEN, %r8 + and $0x10, %r8 + jz .Lld_partial_16 + + movdqa MSG0, MSG1 + movdqu (SRC), MSG0 + +.Lld_partial_16: + ret +ENDPROC(__load_partial) + +/* + * __store_partial: internal ABI + * input: + * LEN - bytes + * DST - dst + * output: + * T0 - first message block + * T1 - second message block + * changed: + * %r8 + * %r9 + * %r10 + */ +__store_partial: + mov LEN, %r8 + mov DST, %r9 + + cmp $16, %r8 + jl .Lst_partial_16 + + movdqu T0, (%r9) + movdqa T1, T0 + + sub $16, %r8 + add $16, %r9 + +.Lst_partial_16: + movq T0, %r10 + + cmp $8, %r8 + jl .Lst_partial_8 + + mov %r10, (%r9) + psrldq $8, T0 + movq T0, %r10 + + sub $8, %r8 + add $8, %r9 + +.Lst_partial_8: + cmp $4, %r8 + jl .Lst_partial_4 + + mov %r10d, (%r9) + shr $32, %r10 + + sub $4, %r8 + add $4, %r9 + +.Lst_partial_4: + cmp $2, %r8 + jl .Lst_partial_2 + + mov %r10w, (%r9) + shr $0x10, %r10 + + sub $2, %r8 + add $2, %r9 + +.Lst_partial_2: + cmp $1, %r8 + jl .Lst_partial_1 + + mov %r10b, (%r9) + +.Lst_partial_1: + ret +ENDPROC(__store_partial) + +.macro update + movdqa STATE7, T0 + aesenc STATE0, STATE7 + aesenc STATE1, STATE0 + aesenc STATE2, STATE1 + aesenc STATE3, STATE2 + aesenc STATE4, STATE3 + aesenc STATE5, STATE4 + aesenc STATE6, STATE5 + aesenc T0, STATE6 +.endm + +.macro update0 + update + pxor MSG0, STATE7 + pxor MSG1, STATE3 +.endm + +.macro update1 + update + pxor MSG0, STATE6 + pxor MSG1, STATE2 +.endm + +.macro update2 + update + pxor MSG0, STATE5 + pxor MSG1, STATE1 +.endm + +.macro update3 + update + pxor MSG0, STATE4 + pxor MSG1, STATE0 +.endm + +.macro update4 + update + pxor MSG0, STATE3 + pxor MSG1, STATE7 +.endm + +.macro update5 + update + pxor MSG0, STATE2 + pxor MSG1, STATE6 +.endm + +.macro update6 + update + pxor MSG0, STATE1 + pxor MSG1, STATE5 +.endm + +.macro update7 + update + pxor MSG0, STATE0 + pxor MSG1, STATE4 +.endm + +.macro state_load + movdqu 0x00(STATEP), STATE0 + movdqu 0x10(STATEP), STATE1 + movdqu 0x20(STATEP), STATE2 + movdqu 0x30(STATEP), STATE3 + movdqu 0x40(STATEP), STATE4 + movdqu 0x50(STATEP), STATE5 + movdqu 0x60(STATEP), STATE6 + movdqu 0x70(STATEP), STATE7 +.endm + +.macro state_store s0 s1 s2 s3 s4 s5 s6 s7 + movdqu \s7, 0x00(STATEP) + movdqu \s0, 0x10(STATEP) + movdqu \s1, 0x20(STATEP) + movdqu \s2, 0x30(STATEP) + movdqu \s3, 0x40(STATEP) + movdqu \s4, 0x50(STATEP) + movdqu \s5, 0x60(STATEP) + movdqu \s6, 0x70(STATEP) +.endm + +.macro state_store0 + state_store STATE0 STATE1 STATE2 STATE3 STATE4 STATE5 STATE6 STATE7 +.endm + +.macro state_store1 + state_store STATE7 STATE0 STATE1 STATE2 STATE3 STATE4 STATE5 STATE6 +.endm + +.macro state_store2 + state_store STATE6 STATE7 STATE0 STATE1 STATE2 STATE3 STATE4 STATE5 +.endm + +.macro state_store3 + state_store STATE5 STATE6 STATE7 STATE0 STATE1 STATE2 STATE3 STATE4 +.endm + +.macro state_store4 + state_store STATE4 STATE5 STATE6 STATE7 STATE0 STATE1 STATE2 STATE3 +.endm + +.macro state_store5 + state_store STATE3 STATE4 STATE5 STATE6 STATE7 STATE0 STATE1 STATE2 +.endm + +.macro state_store6 + state_store STATE2 STATE3 STATE4 STATE5 STATE6 STATE7 STATE0 STATE1 +.endm + +.macro state_store7 + state_store STATE1 STATE2 STATE3 STATE4 STATE5 STATE6 STATE7 STATE0 +.endm + +/* + * void crypto_aegis128l_aesni_init(void *state, const void *key, const void *iv); + */ +ENTRY(crypto_aegis128l_aesni_init) + FRAME_BEGIN + + /* load key: */ + movdqa (%rsi), MSG1 + movdqa MSG1, STATE0 + movdqa MSG1, STATE4 + movdqa MSG1, STATE5 + movdqa MSG1, STATE6 + movdqa MSG1, STATE7 + + /* load IV: */ + movdqu (%rdx), MSG0 + pxor MSG0, STATE0 + pxor MSG0, STATE4 + + /* load the constants: */ + movdqa .Laegis128l_const_0, STATE2 + movdqa .Laegis128l_const_1, STATE1 + movdqa STATE1, STATE3 + pxor STATE2, STATE5 + pxor STATE1, STATE6 + pxor STATE2, STATE7 + + /* update 10 times with IV and KEY: */ + update0 + update1 + update2 + update3 + update4 + update5 + update6 + update7 + update0 + update1 + + state_store1 + + FRAME_END + ret +ENDPROC(crypto_aegis128l_aesni_init) + +.macro ad_block a i + movdq\a (\i * 0x20 + 0x00)(SRC), MSG0 + movdq\a (\i * 0x20 + 0x10)(SRC), MSG1 + update\i + sub $0x20, LEN + cmp $0x20, LEN + jl .Lad_out_\i +.endm + +/* + * void crypto_aegis128l_aesni_ad(void *state, unsigned int length, + * const void *data); + */ +ENTRY(crypto_aegis128l_aesni_ad) + FRAME_BEGIN + + cmp $0x20, LEN + jb .Lad_out + + state_load + + mov SRC, %r8 + and $0xf, %r8 + jnz .Lad_u_loop + +.align 8 +.Lad_a_loop: + ad_block a 0 + ad_block a 1 + ad_block a 2 + ad_block a 3 + ad_block a 4 + ad_block a 5 + ad_block a 6 + ad_block a 7 + + add $0x100, SRC + jmp .Lad_a_loop + +.align 8 +.Lad_u_loop: + ad_block u 0 + ad_block u 1 + ad_block u 2 + ad_block u 3 + ad_block u 4 + ad_block u 5 + ad_block u 6 + ad_block u 7 + + add $0x100, SRC + jmp .Lad_u_loop + +.Lad_out_0: + state_store0 + FRAME_END + ret + +.Lad_out_1: + state_store1 + FRAME_END + ret + +.Lad_out_2: + state_store2 + FRAME_END + ret + +.Lad_out_3: + state_store3 + FRAME_END + ret + +.Lad_out_4: + state_store4 + FRAME_END + ret + +.Lad_out_5: + state_store5 + FRAME_END + ret + +.Lad_out_6: + state_store6 + FRAME_END + ret + +.Lad_out_7: + state_store7 + FRAME_END + ret + +.Lad_out: + FRAME_END + ret +ENDPROC(crypto_aegis128l_aesni_ad) + +.macro crypt m0 m1 s0 s1 s2 s3 s4 s5 s6 s7 + pxor \s1, \m0 + pxor \s6, \m0 + movdqa \s2, T3 + pand \s3, T3 + pxor T3, \m0 + + pxor \s2, \m1 + pxor \s5, \m1 + movdqa \s6, T3 + pand \s7, T3 + pxor T3, \m1 +.endm + +.macro crypt0 m0 m1 + crypt \m0 \m1 STATE0 STATE1 STATE2 STATE3 STATE4 STATE5 STATE6 STATE7 +.endm + +.macro crypt1 m0 m1 + crypt \m0 \m1 STATE7 STATE0 STATE1 STATE2 STATE3 STATE4 STATE5 STATE6 +.endm + +.macro crypt2 m0 m1 + crypt \m0 \m1 STATE6 STATE7 STATE0 STATE1 STATE2 STATE3 STATE4 STATE5 +.endm + +.macro crypt3 m0 m1 + crypt \m0 \m1 STATE5 STATE6 STATE7 STATE0 STATE1 STATE2 STATE3 STATE4 +.endm + +.macro crypt4 m0 m1 + crypt \m0 \m1 STATE4 STATE5 STATE6 STATE7 STATE0 STATE1 STATE2 STATE3 +.endm + +.macro crypt5 m0 m1 + crypt \m0 \m1 STATE3 STATE4 STATE5 STATE6 STATE7 STATE0 STATE1 STATE2 +.endm + +.macro crypt6 m0 m1 + crypt \m0 \m1 STATE2 STATE3 STATE4 STATE5 STATE6 STATE7 STATE0 STATE1 +.endm + +.macro crypt7 m0 m1 + crypt \m0 \m1 STATE1 STATE2 STATE3 STATE4 STATE5 STATE6 STATE7 STATE0 +.endm + +.macro encrypt_block a i + movdq\a (\i * 0x20 + 0x00)(SRC), MSG0 + movdq\a (\i * 0x20 + 0x10)(SRC), MSG1 + movdqa MSG0, T0 + movdqa MSG1, T1 + crypt\i T0, T1 + movdq\a T0, (\i * 0x20 + 0x00)(DST) + movdq\a T1, (\i * 0x20 + 0x10)(DST) + + update\i + + sub $0x20, LEN + cmp $0x20, LEN + jl .Lenc_out_\i +.endm + +.macro decrypt_block a i + movdq\a (\i * 0x20 + 0x00)(SRC), MSG0 + movdq\a (\i * 0x20 + 0x10)(SRC), MSG1 + crypt\i MSG0, MSG1 + movdq\a MSG0, (\i * 0x20 + 0x00)(DST) + movdq\a MSG1, (\i * 0x20 + 0x10)(DST) + + update\i + + sub $0x20, LEN + cmp $0x20, LEN + jl .Ldec_out_\i +.endm + +/* + * void crypto_aegis128l_aesni_enc(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis128l_aesni_enc) + FRAME_BEGIN + + cmp $0x20, LEN + jb .Lenc_out + + state_load + + mov SRC, %r8 + or DST, %r8 + and $0xf, %r8 + jnz .Lenc_u_loop + +.align 8 +.Lenc_a_loop: + encrypt_block a 0 + encrypt_block a 1 + encrypt_block a 2 + encrypt_block a 3 + encrypt_block a 4 + encrypt_block a 5 + encrypt_block a 6 + encrypt_block a 7 + + add $0x100, SRC + add $0x100, DST + jmp .Lenc_a_loop + +.align 8 +.Lenc_u_loop: + encrypt_block u 0 + encrypt_block u 1 + encrypt_block u 2 + encrypt_block u 3 + encrypt_block u 4 + encrypt_block u 5 + encrypt_block u 6 + encrypt_block u 7 + + add $0x100, SRC + add $0x100, DST + jmp .Lenc_u_loop + +.Lenc_out_0: + state_store0 + FRAME_END + ret + +.Lenc_out_1: + state_store1 + FRAME_END + ret + +.Lenc_out_2: + state_store2 + FRAME_END + ret + +.Lenc_out_3: + state_store3 + FRAME_END + ret + +.Lenc_out_4: + state_store4 + FRAME_END + ret + +.Lenc_out_5: + state_store5 + FRAME_END + ret + +.Lenc_out_6: + state_store6 + FRAME_END + ret + +.Lenc_out_7: + state_store7 + FRAME_END + ret + +.Lenc_out: + FRAME_END + ret +ENDPROC(crypto_aegis128l_aesni_enc) + +/* + * void crypto_aegis128l_aesni_enc_tail(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis128l_aesni_enc_tail) + FRAME_BEGIN + + state_load + + /* encrypt message: */ + call __load_partial + + movdqa MSG0, T0 + movdqa MSG1, T1 + crypt0 T0, T1 + + call __store_partial + + update0 + + state_store0 + + FRAME_END +ENDPROC(crypto_aegis128l_aesni_enc_tail) + +/* + * void crypto_aegis128l_aesni_dec(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis128l_aesni_dec) + FRAME_BEGIN + + cmp $0x20, LEN + jb .Ldec_out + + state_load + + mov SRC, %r8 + or DST, %r8 + and $0xF, %r8 + jnz .Ldec_u_loop + +.align 8 +.Ldec_a_loop: + decrypt_block a 0 + decrypt_block a 1 + decrypt_block a 2 + decrypt_block a 3 + decrypt_block a 4 + decrypt_block a 5 + decrypt_block a 6 + decrypt_block a 7 + + add $0x100, SRC + add $0x100, DST + jmp .Ldec_a_loop + +.align 8 +.Ldec_u_loop: + decrypt_block u 0 + decrypt_block u 1 + decrypt_block u 2 + decrypt_block u 3 + decrypt_block u 4 + decrypt_block u 5 + decrypt_block u 6 + decrypt_block u 7 + + add $0x100, SRC + add $0x100, DST + jmp .Ldec_u_loop + +.Ldec_out_0: + state_store0 + FRAME_END + ret + +.Ldec_out_1: + state_store1 + FRAME_END + ret + +.Ldec_out_2: + state_store2 + FRAME_END + ret + +.Ldec_out_3: + state_store3 + FRAME_END + ret + +.Ldec_out_4: + state_store4 + FRAME_END + ret + +.Ldec_out_5: + state_store5 + FRAME_END + ret + +.Ldec_out_6: + state_store6 + FRAME_END + ret + +.Ldec_out_7: + state_store7 + FRAME_END + ret + +.Ldec_out: + FRAME_END + ret +ENDPROC(crypto_aegis128l_aesni_dec) + +/* + * void crypto_aegis128l_aesni_dec_tail(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis128l_aesni_dec_tail) + FRAME_BEGIN + + state_load + + /* decrypt message: */ + call __load_partial + + crypt0 MSG0, MSG1 + + movdqa MSG0, T0 + movdqa MSG1, T1 + call __store_partial + + /* mask with byte count: */ + movq LEN, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + movdqa T0, T1 + movdqa .Laegis128l_counter0, T2 + movdqa .Laegis128l_counter1, T3 + pcmpgtb T2, T0 + pcmpgtb T3, T1 + pand T0, MSG0 + pand T1, MSG1 + + update0 + + state_store0 + + FRAME_END + ret +ENDPROC(crypto_aegis128l_aesni_dec_tail) + +/* + * void crypto_aegis128l_aesni_final(void *state, void *tag_xor, + * u64 assoclen, u64 cryptlen); + */ +ENTRY(crypto_aegis128l_aesni_final) + FRAME_BEGIN + + state_load + + /* prepare length block: */ + movq %rdx, MSG0 + movq %rcx, T0 + pslldq $8, T0 + pxor T0, MSG0 + psllq $3, MSG0 /* multiply by 8 (to get bit count) */ + + pxor STATE2, MSG0 + movdqa MSG0, MSG1 + + /* update state: */ + update0 + update1 + update2 + update3 + update4 + update5 + update6 + + /* xor tag: */ + movdqu (%rsi), T0 + + pxor STATE1, T0 + pxor STATE2, T0 + pxor STATE3, T0 + pxor STATE4, T0 + pxor STATE5, T0 + pxor STATE6, T0 + pxor STATE7, T0 + + movdqu T0, (%rsi) + + FRAME_END + ret +ENDPROC(crypto_aegis128l_aesni_final) diff --git a/arch/x86/crypto/aegis128l-aesni-glue.c b/arch/x86/crypto/aegis128l-aesni-glue.c new file mode 100644 index 000000000000..876e4866e633 --- /dev/null +++ b/arch/x86/crypto/aegis128l-aesni-glue.c @@ -0,0 +1,407 @@ +/* + * The AEGIS-128L Authenticated-Encryption Algorithm + * Glue for AES-NI + SSE2 implementation + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include +#include +#include + +#define AEGIS128L_BLOCK_ALIGN 16 +#define AEGIS128L_BLOCK_SIZE 32 +#define AEGIS128L_NONCE_SIZE 16 +#define AEGIS128L_STATE_BLOCKS 8 +#define AEGIS128L_KEY_SIZE 16 +#define AEGIS128L_MIN_AUTH_SIZE 8 +#define AEGIS128L_MAX_AUTH_SIZE 16 + +asmlinkage void crypto_aegis128l_aesni_init(void *state, void *key, void *iv); + +asmlinkage void crypto_aegis128l_aesni_ad( + void *state, unsigned int length, const void *data); + +asmlinkage void crypto_aegis128l_aesni_enc( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis128l_aesni_dec( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis128l_aesni_enc_tail( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis128l_aesni_dec_tail( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis128l_aesni_final( + void *state, void *tag_xor, unsigned int cryptlen, + unsigned int assoclen); + +struct aegis_block { + u8 bytes[AEGIS128L_BLOCK_SIZE] __aligned(AEGIS128L_BLOCK_ALIGN); +}; + +struct aegis_state { + struct aegis_block blocks[AEGIS128L_STATE_BLOCKS]; +}; + +struct aegis_ctx { + struct aegis_block key; +}; + +struct aegis_crypt_ops { + int (*skcipher_walk_init)(struct skcipher_walk *walk, + struct aead_request *req, bool atomic); + + void (*crypt_blocks)(void *state, unsigned int length, const void *src, + void *dst); + void (*crypt_tail)(void *state, unsigned int length, const void *src, + void *dst); +}; + +static void crypto_aegis128l_aesni_process_ad( + struct aegis_state *state, struct scatterlist *sg_src, + unsigned int assoclen) +{ + struct scatter_walk walk; + struct aegis_block buf; + unsigned int pos = 0; + + scatterwalk_start(&walk, sg_src); + while (assoclen != 0) { + unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int left = size; + void *mapped = scatterwalk_map(&walk); + const u8 *src = (const u8 *)mapped; + + if (pos + size >= AEGIS128L_BLOCK_SIZE) { + if (pos > 0) { + unsigned int fill = AEGIS128L_BLOCK_SIZE - pos; + memcpy(buf.bytes + pos, src, fill); + crypto_aegis128l_aesni_ad(state, + AEGIS128L_BLOCK_SIZE, + buf.bytes); + pos = 0; + left -= fill; + src += fill; + } + + crypto_aegis128l_aesni_ad(state, left, src); + + src += left & ~(AEGIS128L_BLOCK_SIZE - 1); + left &= AEGIS128L_BLOCK_SIZE - 1; + } + + memcpy(buf.bytes + pos, src, left); + pos += left; + assoclen -= size; + + scatterwalk_unmap(mapped); + scatterwalk_advance(&walk, size); + scatterwalk_done(&walk, 0, assoclen); + } + + if (pos > 0) { + memset(buf.bytes + pos, 0, AEGIS128L_BLOCK_SIZE - pos); + crypto_aegis128l_aesni_ad(state, AEGIS128L_BLOCK_SIZE, buf.bytes); + } +} + +static void crypto_aegis128l_aesni_process_crypt( + struct aegis_state *state, struct aead_request *req, + const struct aegis_crypt_ops *ops) +{ + struct skcipher_walk walk; + u8 *src, *dst; + unsigned int chunksize, base; + + ops->skcipher_walk_init(&walk, req, false); + + while (walk.nbytes) { + src = walk.src.virt.addr; + dst = walk.dst.virt.addr; + chunksize = walk.nbytes; + + ops->crypt_blocks(state, chunksize, src, dst); + + base = chunksize & ~(AEGIS128L_BLOCK_SIZE - 1); + src += base; + dst += base; + chunksize &= AEGIS128L_BLOCK_SIZE - 1; + + if (chunksize > 0) + ops->crypt_tail(state, chunksize, src, dst); + + skcipher_walk_done(&walk, 0); + } +} + +static struct aegis_ctx *crypto_aegis128l_aesni_ctx(struct crypto_aead *aead) +{ + u8 *ctx = crypto_aead_ctx(aead); + ctx = PTR_ALIGN(ctx, __alignof__(struct aegis_ctx)); + return (void *)ctx; +} + +static int crypto_aegis128l_aesni_setkey(struct crypto_aead *aead, + const u8 *key, unsigned int keylen) +{ + struct aegis_ctx *ctx = crypto_aegis128l_aesni_ctx(aead); + + if (keylen != AEGIS128L_KEY_SIZE) { + crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; + } + + memcpy(ctx->key.bytes, key, AEGIS128L_KEY_SIZE); + + return 0; +} + +static int crypto_aegis128l_aesni_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + if (authsize > AEGIS128L_MAX_AUTH_SIZE) + return -EINVAL; + if (authsize < AEGIS128L_MIN_AUTH_SIZE) + return -EINVAL; + return 0; +} + +static void crypto_aegis128l_aesni_crypt(struct aead_request *req, + struct aegis_block *tag_xor, + unsigned int cryptlen, + const struct aegis_crypt_ops *ops) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_ctx *ctx = crypto_aegis128l_aesni_ctx(tfm); + struct aegis_state state; + + kernel_fpu_begin(); + + crypto_aegis128l_aesni_init(&state, ctx->key.bytes, req->iv); + crypto_aegis128l_aesni_process_ad(&state, req->src, req->assoclen); + crypto_aegis128l_aesni_process_crypt(&state, req, ops); + crypto_aegis128l_aesni_final(&state, tag_xor, req->assoclen, cryptlen); + + kernel_fpu_end(); +} + +static int crypto_aegis128l_aesni_encrypt(struct aead_request *req) +{ + static const struct aegis_crypt_ops OPS = { + .skcipher_walk_init = skcipher_walk_aead_encrypt, + .crypt_blocks = crypto_aegis128l_aesni_enc, + .crypt_tail = crypto_aegis128l_aesni_enc_tail, + }; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_block tag = {}; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen; + + crypto_aegis128l_aesni_crypt(req, &tag, cryptlen, &OPS); + + scatterwalk_map_and_copy(tag.bytes, req->dst, + req->assoclen + cryptlen, authsize, 1); + return 0; +} + +static int crypto_aegis128l_aesni_decrypt(struct aead_request *req) +{ + static const struct aegis_block zeros = {}; + + static const struct aegis_crypt_ops OPS = { + .skcipher_walk_init = skcipher_walk_aead_decrypt, + .crypt_blocks = crypto_aegis128l_aesni_dec, + .crypt_tail = crypto_aegis128l_aesni_dec_tail, + }; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_block tag; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen - authsize; + + scatterwalk_map_and_copy(tag.bytes, req->src, + req->assoclen + cryptlen, authsize, 0); + + crypto_aegis128l_aesni_crypt(req, &tag, cryptlen, &OPS); + + return crypto_memneq(tag.bytes, zeros.bytes, authsize) ? -EBADMSG : 0; +} + +static int crypto_aegis128l_aesni_init_tfm(struct crypto_aead *aead) +{ + return 0; +} + +static void crypto_aegis128l_aesni_exit_tfm(struct crypto_aead *aead) +{ +} + +static int cryptd_aegis128l_aesni_setkey(struct crypto_aead *aead, + const u8 *key, unsigned int keylen) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + return crypto_aead_setkey(&cryptd_tfm->base, key, keylen); +} + +static int cryptd_aegis128l_aesni_setauthsize(struct crypto_aead *aead, + unsigned int authsize) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + return crypto_aead_setauthsize(&cryptd_tfm->base, authsize); +} + +static int cryptd_aegis128l_aesni_encrypt(struct aead_request *req) +{ + struct crypto_aead *aead = crypto_aead_reqtfm(req); + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + aead = &cryptd_tfm->base; + if (irq_fpu_usable() && (!in_atomic() || + !cryptd_aead_queued(cryptd_tfm))) + aead = cryptd_aead_child(cryptd_tfm); + + aead_request_set_tfm(req, aead); + + return crypto_aead_encrypt(req); +} + +static int cryptd_aegis128l_aesni_decrypt(struct aead_request *req) +{ + struct crypto_aead *aead = crypto_aead_reqtfm(req); + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + aead = &cryptd_tfm->base; + if (irq_fpu_usable() && (!in_atomic() || + !cryptd_aead_queued(cryptd_tfm))) + aead = cryptd_aead_child(cryptd_tfm); + + aead_request_set_tfm(req, aead); + + return crypto_aead_decrypt(req); +} + +static int cryptd_aegis128l_aesni_init_tfm(struct crypto_aead *aead) +{ + struct cryptd_aead *cryptd_tfm; + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + + cryptd_tfm = cryptd_alloc_aead("__aegis128l-aesni", CRYPTO_ALG_INTERNAL, + CRYPTO_ALG_INTERNAL); + if (IS_ERR(cryptd_tfm)) + return PTR_ERR(cryptd_tfm); + + *ctx = cryptd_tfm; + crypto_aead_set_reqsize(aead, crypto_aead_reqsize(&cryptd_tfm->base)); + return 0; +} + +static void cryptd_aegis128l_aesni_exit_tfm(struct crypto_aead *aead) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + + cryptd_free_aead(*ctx); +} + +static struct aead_alg crypto_aegis128l_aesni_alg[] = { + { + .setkey = crypto_aegis128l_aesni_setkey, + .setauthsize = crypto_aegis128l_aesni_setauthsize, + .encrypt = crypto_aegis128l_aesni_encrypt, + .decrypt = crypto_aegis128l_aesni_decrypt, + .init = crypto_aegis128l_aesni_init_tfm, + .exit = crypto_aegis128l_aesni_exit_tfm, + + .ivsize = AEGIS128L_NONCE_SIZE, + .maxauthsize = AEGIS128L_MAX_AUTH_SIZE, + .chunksize = AEGIS128L_BLOCK_SIZE, + + .base = { + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct aegis_ctx) + + __alignof__(struct aegis_ctx), + .cra_alignmask = 0, + + .cra_name = "__aegis128l", + .cra_driver_name = "__aegis128l-aesni", + + .cra_module = THIS_MODULE, + } + }, { + .setkey = cryptd_aegis128l_aesni_setkey, + .setauthsize = cryptd_aegis128l_aesni_setauthsize, + .encrypt = cryptd_aegis128l_aesni_encrypt, + .decrypt = cryptd_aegis128l_aesni_decrypt, + .init = cryptd_aegis128l_aesni_init_tfm, + .exit = cryptd_aegis128l_aesni_exit_tfm, + + .ivsize = AEGIS128L_NONCE_SIZE, + .maxauthsize = AEGIS128L_MAX_AUTH_SIZE, + .chunksize = AEGIS128L_BLOCK_SIZE, + + .base = { + .cra_flags = CRYPTO_ALG_ASYNC, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct cryptd_aead *), + .cra_alignmask = 0, + + .cra_priority = 400, + + .cra_name = "aegis128l", + .cra_driver_name = "aegis128l-aesni", + + .cra_module = THIS_MODULE, + } + } +}; + +static const struct x86_cpu_id aesni_cpu_id[] = { + X86_FEATURE_MATCH(X86_FEATURE_AES), + X86_FEATURE_MATCH(X86_FEATURE_XMM2), + {} +}; +MODULE_DEVICE_TABLE(x86cpu, aesni_cpu_id); + +static int __init crypto_aegis128l_aesni_module_init(void) +{ + if (!x86_match_cpu(aesni_cpu_id)) + return -ENODEV; + + return crypto_register_aeads(crypto_aegis128l_aesni_alg, + ARRAY_SIZE(crypto_aegis128l_aesni_alg)); +} + +static void __exit crypto_aegis128l_aesni_module_exit(void) +{ + crypto_unregister_aeads(crypto_aegis128l_aesni_alg, + ARRAY_SIZE(crypto_aegis128l_aesni_alg)); +} + +module_init(crypto_aegis128l_aesni_module_init); +module_exit(crypto_aegis128l_aesni_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("AEGIS-128L AEAD algorithm -- AESNI+SSE2 implementation"); +MODULE_ALIAS_CRYPTO("aegis128l"); +MODULE_ALIAS_CRYPTO("aegis128l-aesni"); diff --git a/arch/x86/crypto/aegis256-aesni-asm.S b/arch/x86/crypto/aegis256-aesni-asm.S new file mode 100644 index 000000000000..1d977d515bf9 --- /dev/null +++ b/arch/x86/crypto/aegis256-aesni-asm.S @@ -0,0 +1,702 @@ +/* + * AES-NI + SSE2 implementation of AEGIS-128L + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation. + */ + +#include +#include + +#define STATE0 %xmm0 +#define STATE1 %xmm1 +#define STATE2 %xmm2 +#define STATE3 %xmm3 +#define STATE4 %xmm4 +#define STATE5 %xmm5 +#define MSG %xmm6 +#define T0 %xmm7 +#define T1 %xmm8 +#define T2 %xmm9 +#define T3 %xmm10 + +#define STATEP %rdi +#define LEN %rsi +#define SRC %rdx +#define DST %rcx + +.section .rodata.cst16.aegis256_const, "aM", @progbits, 32 +.align 16 +.Laegis256_const_0: + .byte 0x00, 0x01, 0x01, 0x02, 0x03, 0x05, 0x08, 0x0d + .byte 0x15, 0x22, 0x37, 0x59, 0x90, 0xe9, 0x79, 0x62 +.Laegis256_const_1: + .byte 0xdb, 0x3d, 0x18, 0x55, 0x6d, 0xc2, 0x2f, 0xf1 + .byte 0x20, 0x11, 0x31, 0x42, 0x73, 0xb5, 0x28, 0xdd + +.section .rodata.cst16.aegis256_counter, "aM", @progbits, 16 +.align 16 +.Laegis256_counter: + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f + +.text + +/* + * __load_partial: internal ABI + * input: + * LEN - bytes + * SRC - src + * output: + * MSG - message block + * changed: + * T0 + * %r8 + * %r9 + */ +__load_partial: + xor %r9, %r9 + pxor MSG, MSG + + mov LEN, %r8 + and $0x1, %r8 + jz .Lld_partial_1 + + mov LEN, %r8 + and $0x1E, %r8 + add SRC, %r8 + mov (%r8), %r9b + +.Lld_partial_1: + mov LEN, %r8 + and $0x2, %r8 + jz .Lld_partial_2 + + mov LEN, %r8 + and $0x1C, %r8 + add SRC, %r8 + shl $0x10, %r9 + mov (%r8), %r9w + +.Lld_partial_2: + mov LEN, %r8 + and $0x4, %r8 + jz .Lld_partial_4 + + mov LEN, %r8 + and $0x18, %r8 + add SRC, %r8 + shl $32, %r9 + mov (%r8), %r8d + xor %r8, %r9 + +.Lld_partial_4: + movq %r9, MSG + + mov LEN, %r8 + and $0x8, %r8 + jz .Lld_partial_8 + + mov LEN, %r8 + and $0x10, %r8 + add SRC, %r8 + pslldq $8, MSG + movq (%r8), T0 + pxor T0, MSG + +.Lld_partial_8: + ret +ENDPROC(__load_partial) + +/* + * __store_partial: internal ABI + * input: + * LEN - bytes + * DST - dst + * output: + * T0 - message block + * changed: + * %r8 + * %r9 + * %r10 + */ +__store_partial: + mov LEN, %r8 + mov DST, %r9 + + movq T0, %r10 + + cmp $8, %r8 + jl .Lst_partial_8 + + mov %r10, (%r9) + psrldq $8, T0 + movq T0, %r10 + + sub $8, %r8 + add $8, %r9 + +.Lst_partial_8: + cmp $4, %r8 + jl .Lst_partial_4 + + mov %r10d, (%r9) + shr $32, %r10 + + sub $4, %r8 + add $4, %r9 + +.Lst_partial_4: + cmp $2, %r8 + jl .Lst_partial_2 + + mov %r10w, (%r9) + shr $0x10, %r10 + + sub $2, %r8 + add $2, %r9 + +.Lst_partial_2: + cmp $1, %r8 + jl .Lst_partial_1 + + mov %r10b, (%r9) + +.Lst_partial_1: + ret +ENDPROC(__store_partial) + +.macro update + movdqa STATE5, T0 + aesenc STATE0, STATE5 + aesenc STATE1, STATE0 + aesenc STATE2, STATE1 + aesenc STATE3, STATE2 + aesenc STATE4, STATE3 + aesenc T0, STATE4 +.endm + +.macro update0 m + update + pxor \m, STATE5 +.endm + +.macro update1 m + update + pxor \m, STATE4 +.endm + +.macro update2 m + update + pxor \m, STATE3 +.endm + +.macro update3 m + update + pxor \m, STATE2 +.endm + +.macro update4 m + update + pxor \m, STATE1 +.endm + +.macro update5 m + update + pxor \m, STATE0 +.endm + +.macro state_load + movdqu 0x00(STATEP), STATE0 + movdqu 0x10(STATEP), STATE1 + movdqu 0x20(STATEP), STATE2 + movdqu 0x30(STATEP), STATE3 + movdqu 0x40(STATEP), STATE4 + movdqu 0x50(STATEP), STATE5 +.endm + +.macro state_store s0 s1 s2 s3 s4 s5 + movdqu \s5, 0x00(STATEP) + movdqu \s0, 0x10(STATEP) + movdqu \s1, 0x20(STATEP) + movdqu \s2, 0x30(STATEP) + movdqu \s3, 0x40(STATEP) + movdqu \s4, 0x50(STATEP) +.endm + +.macro state_store0 + state_store STATE0 STATE1 STATE2 STATE3 STATE4 STATE5 +.endm + +.macro state_store1 + state_store STATE5 STATE0 STATE1 STATE2 STATE3 STATE4 +.endm + +.macro state_store2 + state_store STATE4 STATE5 STATE0 STATE1 STATE2 STATE3 +.endm + +.macro state_store3 + state_store STATE3 STATE4 STATE5 STATE0 STATE1 STATE2 +.endm + +.macro state_store4 + state_store STATE2 STATE3 STATE4 STATE5 STATE0 STATE1 +.endm + +.macro state_store5 + state_store STATE1 STATE2 STATE3 STATE4 STATE5 STATE0 +.endm + +/* + * void crypto_aegis256_aesni_init(void *state, const void *key, const void *iv); + */ +ENTRY(crypto_aegis256_aesni_init) + FRAME_BEGIN + + /* load key: */ + movdqa 0x00(%rsi), MSG + movdqa 0x10(%rsi), T1 + movdqa MSG, STATE4 + movdqa T1, STATE5 + + /* load IV: */ + movdqu 0x00(%rdx), T2 + movdqu 0x10(%rdx), T3 + pxor MSG, T2 + pxor T1, T3 + movdqa T2, STATE0 + movdqa T3, STATE1 + + /* load the constants: */ + movdqa .Laegis256_const_0, STATE3 + movdqa .Laegis256_const_1, STATE2 + pxor STATE3, STATE4 + pxor STATE2, STATE5 + + /* update 10 times with IV and KEY: */ + update0 MSG + update1 T1 + update2 T2 + update3 T3 + update4 MSG + update5 T1 + update0 T2 + update1 T3 + update2 MSG + update3 T1 + update4 T2 + update5 T3 + update0 MSG + update1 T1 + update2 T2 + update3 T3 + + state_store3 + + FRAME_END + ret +ENDPROC(crypto_aegis256_aesni_init) + +.macro ad_block a i + movdq\a (\i * 0x10)(SRC), MSG + update\i MSG + sub $0x10, LEN + cmp $0x10, LEN + jl .Lad_out_\i +.endm + +/* + * void crypto_aegis256_aesni_ad(void *state, unsigned int length, + * const void *data); + */ +ENTRY(crypto_aegis256_aesni_ad) + FRAME_BEGIN + + cmp $0x10, LEN + jb .Lad_out + + state_load + + mov SRC, %r8 + and $0xf, %r8 + jnz .Lad_u_loop + +.align 8 +.Lad_a_loop: + ad_block a 0 + ad_block a 1 + ad_block a 2 + ad_block a 3 + ad_block a 4 + ad_block a 5 + + add $0x60, SRC + jmp .Lad_a_loop + +.align 8 +.Lad_u_loop: + ad_block u 0 + ad_block u 1 + ad_block u 2 + ad_block u 3 + ad_block u 4 + ad_block u 5 + + add $0x60, SRC + jmp .Lad_u_loop + +.Lad_out_0: + state_store0 + FRAME_END + ret + +.Lad_out_1: + state_store1 + FRAME_END + ret + +.Lad_out_2: + state_store2 + FRAME_END + ret + +.Lad_out_3: + state_store3 + FRAME_END + ret + +.Lad_out_4: + state_store4 + FRAME_END + ret + +.Lad_out_5: + state_store5 + FRAME_END + ret + +.Lad_out: + FRAME_END + ret +ENDPROC(crypto_aegis256_aesni_ad) + +.macro crypt m s0 s1 s2 s3 s4 s5 + pxor \s1, \m + pxor \s4, \m + pxor \s5, \m + movdqa \s2, T3 + pand \s3, T3 + pxor T3, \m +.endm + +.macro crypt0 m + crypt \m STATE0 STATE1 STATE2 STATE3 STATE4 STATE5 +.endm + +.macro crypt1 m + crypt \m STATE5 STATE0 STATE1 STATE2 STATE3 STATE4 +.endm + +.macro crypt2 m + crypt \m STATE4 STATE5 STATE0 STATE1 STATE2 STATE3 +.endm + +.macro crypt3 m + crypt \m STATE3 STATE4 STATE5 STATE0 STATE1 STATE2 +.endm + +.macro crypt4 m + crypt \m STATE2 STATE3 STATE4 STATE5 STATE0 STATE1 +.endm + +.macro crypt5 m + crypt \m STATE1 STATE2 STATE3 STATE4 STATE5 STATE0 +.endm + +.macro encrypt_block a i + movdq\a (\i * 0x10)(SRC), MSG + movdqa MSG, T0 + crypt\i T0 + movdq\a T0, (\i * 0x10)(DST) + + update\i MSG + + sub $0x10, LEN + cmp $0x10, LEN + jl .Lenc_out_\i +.endm + +.macro decrypt_block a i + movdq\a (\i * 0x10)(SRC), MSG + crypt\i MSG + movdq\a MSG, (\i * 0x10)(DST) + + update\i MSG + + sub $0x10, LEN + cmp $0x10, LEN + jl .Ldec_out_\i +.endm + +/* + * void crypto_aegis256_aesni_enc(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis256_aesni_enc) + FRAME_BEGIN + + cmp $0x10, LEN + jb .Lenc_out + + state_load + + mov SRC, %r8 + or DST, %r8 + and $0xf, %r8 + jnz .Lenc_u_loop + +.align 8 +.Lenc_a_loop: + encrypt_block a 0 + encrypt_block a 1 + encrypt_block a 2 + encrypt_block a 3 + encrypt_block a 4 + encrypt_block a 5 + + add $0x60, SRC + add $0x60, DST + jmp .Lenc_a_loop + +.align 8 +.Lenc_u_loop: + encrypt_block u 0 + encrypt_block u 1 + encrypt_block u 2 + encrypt_block u 3 + encrypt_block u 4 + encrypt_block u 5 + + add $0x60, SRC + add $0x60, DST + jmp .Lenc_u_loop + +.Lenc_out_0: + state_store0 + FRAME_END + ret + +.Lenc_out_1: + state_store1 + FRAME_END + ret + +.Lenc_out_2: + state_store2 + FRAME_END + ret + +.Lenc_out_3: + state_store3 + FRAME_END + ret + +.Lenc_out_4: + state_store4 + FRAME_END + ret + +.Lenc_out_5: + state_store5 + FRAME_END + ret + +.Lenc_out: + FRAME_END + ret +ENDPROC(crypto_aegis256_aesni_enc) + +/* + * void crypto_aegis256_aesni_enc_tail(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis256_aesni_enc_tail) + FRAME_BEGIN + + state_load + + /* encrypt message: */ + call __load_partial + + movdqa MSG, T0 + crypt0 T0 + + call __store_partial + + update0 MSG + + state_store0 + + FRAME_END +ENDPROC(crypto_aegis256_aesni_enc_tail) + +/* + * void crypto_aegis256_aesni_dec(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis256_aesni_dec) + FRAME_BEGIN + + cmp $0x10, LEN + jb .Ldec_out + + state_load + + mov SRC, %r8 + or DST, %r8 + and $0xF, %r8 + jnz .Ldec_u_loop + +.align 8 +.Ldec_a_loop: + decrypt_block a 0 + decrypt_block a 1 + decrypt_block a 2 + decrypt_block a 3 + decrypt_block a 4 + decrypt_block a 5 + + add $0x60, SRC + add $0x60, DST + jmp .Ldec_a_loop + +.align 8 +.Ldec_u_loop: + decrypt_block u 0 + decrypt_block u 1 + decrypt_block u 2 + decrypt_block u 3 + decrypt_block u 4 + decrypt_block u 5 + + add $0x60, SRC + add $0x60, DST + jmp .Ldec_u_loop + +.Ldec_out_0: + state_store0 + FRAME_END + ret + +.Ldec_out_1: + state_store1 + FRAME_END + ret + +.Ldec_out_2: + state_store2 + FRAME_END + ret + +.Ldec_out_3: + state_store3 + FRAME_END + ret + +.Ldec_out_4: + state_store4 + FRAME_END + ret + +.Ldec_out_5: + state_store5 + FRAME_END + ret + +.Ldec_out: + FRAME_END + ret +ENDPROC(crypto_aegis256_aesni_dec) + +/* + * void crypto_aegis256_aesni_dec_tail(void *state, unsigned int length, + * const void *src, void *dst); + */ +ENTRY(crypto_aegis256_aesni_dec_tail) + FRAME_BEGIN + + state_load + + /* decrypt message: */ + call __load_partial + + crypt0 MSG + + movdqa MSG, T0 + call __store_partial + + /* mask with byte count: */ + movq LEN, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + movdqa .Laegis256_counter, T1 + pcmpgtb T1, T0 + pand T0, MSG + + update0 MSG + + state_store0 + + FRAME_END + ret +ENDPROC(crypto_aegis256_aesni_dec_tail) + +/* + * void crypto_aegis256_aesni_final(void *state, void *tag_xor, + * u64 assoclen, u64 cryptlen); + */ +ENTRY(crypto_aegis256_aesni_final) + FRAME_BEGIN + + state_load + + /* prepare length block: */ + movq %rdx, MSG + movq %rcx, T0 + pslldq $8, T0 + pxor T0, MSG + psllq $3, MSG /* multiply by 8 (to get bit count) */ + + pxor STATE3, MSG + + /* update state: */ + update0 MSG + update1 MSG + update2 MSG + update3 MSG + update4 MSG + update5 MSG + update0 MSG + + /* xor tag: */ + movdqu (%rsi), MSG + + pxor STATE0, MSG + pxor STATE1, MSG + pxor STATE2, MSG + pxor STATE3, MSG + pxor STATE4, MSG + pxor STATE5, MSG + + movdqu MSG, (%rsi) + + FRAME_END + ret +ENDPROC(crypto_aegis256_aesni_final) diff --git a/arch/x86/crypto/aegis256-aesni-glue.c b/arch/x86/crypto/aegis256-aesni-glue.c new file mode 100644 index 000000000000..2b5dd3af8f4d --- /dev/null +++ b/arch/x86/crypto/aegis256-aesni-glue.c @@ -0,0 +1,407 @@ +/* + * The AEGIS-256 Authenticated-Encryption Algorithm + * Glue for AES-NI + SSE2 implementation + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include +#include +#include + +#define AEGIS256_BLOCK_ALIGN 16 +#define AEGIS256_BLOCK_SIZE 16 +#define AEGIS256_NONCE_SIZE 32 +#define AEGIS256_STATE_BLOCKS 6 +#define AEGIS256_KEY_SIZE 32 +#define AEGIS256_MIN_AUTH_SIZE 8 +#define AEGIS256_MAX_AUTH_SIZE 16 + +asmlinkage void crypto_aegis256_aesni_init(void *state, void *key, void *iv); + +asmlinkage void crypto_aegis256_aesni_ad( + void *state, unsigned int length, const void *data); + +asmlinkage void crypto_aegis256_aesni_enc( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis256_aesni_dec( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis256_aesni_enc_tail( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis256_aesni_dec_tail( + void *state, unsigned int length, const void *src, void *dst); + +asmlinkage void crypto_aegis256_aesni_final( + void *state, void *tag_xor, unsigned int cryptlen, + unsigned int assoclen); + +struct aegis_block { + u8 bytes[AEGIS256_BLOCK_SIZE] __aligned(AEGIS256_BLOCK_ALIGN); +}; + +struct aegis_state { + struct aegis_block blocks[AEGIS256_STATE_BLOCKS]; +}; + +struct aegis_ctx { + struct aegis_block key[AEGIS256_KEY_SIZE / AEGIS256_BLOCK_SIZE]; +}; + +struct aegis_crypt_ops { + int (*skcipher_walk_init)(struct skcipher_walk *walk, + struct aead_request *req, bool atomic); + + void (*crypt_blocks)(void *state, unsigned int length, const void *src, + void *dst); + void (*crypt_tail)(void *state, unsigned int length, const void *src, + void *dst); +}; + +static void crypto_aegis256_aesni_process_ad( + struct aegis_state *state, struct scatterlist *sg_src, + unsigned int assoclen) +{ + struct scatter_walk walk; + struct aegis_block buf; + unsigned int pos = 0; + + scatterwalk_start(&walk, sg_src); + while (assoclen != 0) { + unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int left = size; + void *mapped = scatterwalk_map(&walk); + const u8 *src = (const u8 *)mapped; + + if (pos + size >= AEGIS256_BLOCK_SIZE) { + if (pos > 0) { + unsigned int fill = AEGIS256_BLOCK_SIZE - pos; + memcpy(buf.bytes + pos, src, fill); + crypto_aegis256_aesni_ad(state, + AEGIS256_BLOCK_SIZE, + buf.bytes); + pos = 0; + left -= fill; + src += fill; + } + + crypto_aegis256_aesni_ad(state, left, src); + + src += left & ~(AEGIS256_BLOCK_SIZE - 1); + left &= AEGIS256_BLOCK_SIZE - 1; + } + + memcpy(buf.bytes + pos, src, left); + pos += left; + assoclen -= size; + + scatterwalk_unmap(mapped); + scatterwalk_advance(&walk, size); + scatterwalk_done(&walk, 0, assoclen); + } + + if (pos > 0) { + memset(buf.bytes + pos, 0, AEGIS256_BLOCK_SIZE - pos); + crypto_aegis256_aesni_ad(state, AEGIS256_BLOCK_SIZE, buf.bytes); + } +} + +static void crypto_aegis256_aesni_process_crypt( + struct aegis_state *state, struct aead_request *req, + const struct aegis_crypt_ops *ops) +{ + struct skcipher_walk walk; + u8 *src, *dst; + unsigned int chunksize, base; + + ops->skcipher_walk_init(&walk, req, false); + + while (walk.nbytes) { + src = walk.src.virt.addr; + dst = walk.dst.virt.addr; + chunksize = walk.nbytes; + + ops->crypt_blocks(state, chunksize, src, dst); + + base = chunksize & ~(AEGIS256_BLOCK_SIZE - 1); + src += base; + dst += base; + chunksize &= AEGIS256_BLOCK_SIZE - 1; + + if (chunksize > 0) + ops->crypt_tail(state, chunksize, src, dst); + + skcipher_walk_done(&walk, 0); + } +} + +static struct aegis_ctx *crypto_aegis256_aesni_ctx(struct crypto_aead *aead) +{ + u8 *ctx = crypto_aead_ctx(aead); + ctx = PTR_ALIGN(ctx, __alignof__(struct aegis_ctx)); + return (void *)ctx; +} + +static int crypto_aegis256_aesni_setkey(struct crypto_aead *aead, const u8 *key, + unsigned int keylen) +{ + struct aegis_ctx *ctx = crypto_aegis256_aesni_ctx(aead); + + if (keylen != AEGIS256_KEY_SIZE) { + crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; + } + + memcpy(ctx->key, key, AEGIS256_KEY_SIZE); + + return 0; +} + +static int crypto_aegis256_aesni_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + if (authsize > AEGIS256_MAX_AUTH_SIZE) + return -EINVAL; + if (authsize < AEGIS256_MIN_AUTH_SIZE) + return -EINVAL; + return 0; +} + +static void crypto_aegis256_aesni_crypt(struct aead_request *req, + struct aegis_block *tag_xor, + unsigned int cryptlen, + const struct aegis_crypt_ops *ops) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_ctx *ctx = crypto_aegis256_aesni_ctx(tfm); + struct aegis_state state; + + kernel_fpu_begin(); + + crypto_aegis256_aesni_init(&state, ctx->key, req->iv); + crypto_aegis256_aesni_process_ad(&state, req->src, req->assoclen); + crypto_aegis256_aesni_process_crypt(&state, req, ops); + crypto_aegis256_aesni_final(&state, tag_xor, req->assoclen, cryptlen); + + kernel_fpu_end(); +} + +static int crypto_aegis256_aesni_encrypt(struct aead_request *req) +{ + static const struct aegis_crypt_ops OPS = { + .skcipher_walk_init = skcipher_walk_aead_encrypt, + .crypt_blocks = crypto_aegis256_aesni_enc, + .crypt_tail = crypto_aegis256_aesni_enc_tail, + }; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_block tag = {}; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen; + + crypto_aegis256_aesni_crypt(req, &tag, cryptlen, &OPS); + + scatterwalk_map_and_copy(tag.bytes, req->dst, + req->assoclen + cryptlen, authsize, 1); + return 0; +} + +static int crypto_aegis256_aesni_decrypt(struct aead_request *req) +{ + static const struct aegis_block zeros = {}; + + static const struct aegis_crypt_ops OPS = { + .skcipher_walk_init = skcipher_walk_aead_decrypt, + .crypt_blocks = crypto_aegis256_aesni_dec, + .crypt_tail = crypto_aegis256_aesni_dec_tail, + }; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_block tag; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen - authsize; + + scatterwalk_map_and_copy(tag.bytes, req->src, + req->assoclen + cryptlen, authsize, 0); + + crypto_aegis256_aesni_crypt(req, &tag, cryptlen, &OPS); + + return crypto_memneq(tag.bytes, zeros.bytes, authsize) ? -EBADMSG : 0; +} + +static int crypto_aegis256_aesni_init_tfm(struct crypto_aead *aead) +{ + return 0; +} + +static void crypto_aegis256_aesni_exit_tfm(struct crypto_aead *aead) +{ +} + +static int cryptd_aegis256_aesni_setkey(struct crypto_aead *aead, + const u8 *key, unsigned int keylen) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + return crypto_aead_setkey(&cryptd_tfm->base, key, keylen); +} + +static int cryptd_aegis256_aesni_setauthsize(struct crypto_aead *aead, + unsigned int authsize) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + return crypto_aead_setauthsize(&cryptd_tfm->base, authsize); +} + +static int cryptd_aegis256_aesni_encrypt(struct aead_request *req) +{ + struct crypto_aead *aead = crypto_aead_reqtfm(req); + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + aead = &cryptd_tfm->base; + if (irq_fpu_usable() && (!in_atomic() || + !cryptd_aead_queued(cryptd_tfm))) + aead = cryptd_aead_child(cryptd_tfm); + + aead_request_set_tfm(req, aead); + + return crypto_aead_encrypt(req); +} + +static int cryptd_aegis256_aesni_decrypt(struct aead_request *req) +{ + struct crypto_aead *aead = crypto_aead_reqtfm(req); + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + aead = &cryptd_tfm->base; + if (irq_fpu_usable() && (!in_atomic() || + !cryptd_aead_queued(cryptd_tfm))) + aead = cryptd_aead_child(cryptd_tfm); + + aead_request_set_tfm(req, aead); + + return crypto_aead_decrypt(req); +} + +static int cryptd_aegis256_aesni_init_tfm(struct crypto_aead *aead) +{ + struct cryptd_aead *cryptd_tfm; + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + + cryptd_tfm = cryptd_alloc_aead("__aegis256-aesni", CRYPTO_ALG_INTERNAL, + CRYPTO_ALG_INTERNAL); + if (IS_ERR(cryptd_tfm)) + return PTR_ERR(cryptd_tfm); + + *ctx = cryptd_tfm; + crypto_aead_set_reqsize(aead, crypto_aead_reqsize(&cryptd_tfm->base)); + return 0; +} + +static void cryptd_aegis256_aesni_exit_tfm(struct crypto_aead *aead) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + + cryptd_free_aead(*ctx); +} + +static struct aead_alg crypto_aegis256_aesni_alg[] = { + { + .setkey = crypto_aegis256_aesni_setkey, + .setauthsize = crypto_aegis256_aesni_setauthsize, + .encrypt = crypto_aegis256_aesni_encrypt, + .decrypt = crypto_aegis256_aesni_decrypt, + .init = crypto_aegis256_aesni_init_tfm, + .exit = crypto_aegis256_aesni_exit_tfm, + + .ivsize = AEGIS256_NONCE_SIZE, + .maxauthsize = AEGIS256_MAX_AUTH_SIZE, + .chunksize = AEGIS256_BLOCK_SIZE, + + .base = { + .cra_flags = CRYPTO_ALG_INTERNAL, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct aegis_ctx) + + __alignof__(struct aegis_ctx), + .cra_alignmask = 0, + + .cra_name = "__aegis256", + .cra_driver_name = "__aegis256-aesni", + + .cra_module = THIS_MODULE, + } + }, { + .setkey = cryptd_aegis256_aesni_setkey, + .setauthsize = cryptd_aegis256_aesni_setauthsize, + .encrypt = cryptd_aegis256_aesni_encrypt, + .decrypt = cryptd_aegis256_aesni_decrypt, + .init = cryptd_aegis256_aesni_init_tfm, + .exit = cryptd_aegis256_aesni_exit_tfm, + + .ivsize = AEGIS256_NONCE_SIZE, + .maxauthsize = AEGIS256_MAX_AUTH_SIZE, + .chunksize = AEGIS256_BLOCK_SIZE, + + .base = { + .cra_flags = CRYPTO_ALG_ASYNC, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct cryptd_aead *), + .cra_alignmask = 0, + + .cra_priority = 400, + + .cra_name = "aegis256", + .cra_driver_name = "aegis256-aesni", + + .cra_module = THIS_MODULE, + } + } +}; + +static const struct x86_cpu_id aesni_cpu_id[] = { + X86_FEATURE_MATCH(X86_FEATURE_AES), + X86_FEATURE_MATCH(X86_FEATURE_XMM2), + {} +}; +MODULE_DEVICE_TABLE(x86cpu, aesni_cpu_id); + +static int __init crypto_aegis256_aesni_module_init(void) +{ + if (!x86_match_cpu(aesni_cpu_id)) + return -ENODEV; + + return crypto_register_aeads(crypto_aegis256_aesni_alg, + ARRAY_SIZE(crypto_aegis256_aesni_alg)); +} + +static void __exit crypto_aegis256_aesni_module_exit(void) +{ + crypto_unregister_aeads(crypto_aegis256_aesni_alg, + ARRAY_SIZE(crypto_aegis256_aesni_alg)); +} + +module_init(crypto_aegis256_aesni_module_init); +module_exit(crypto_aegis256_aesni_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("AEGIS-256 AEAD algorithm -- AESNI+SSE2 implementation"); +MODULE_ALIAS_CRYPTO("aegis256"); +MODULE_ALIAS_CRYPTO("aegis256-aesni"); diff --git a/arch/x86/crypto/ghash-clmulni-intel_glue.c b/arch/x86/crypto/ghash-clmulni-intel_glue.c index 0420bab19efb..2ddbe3a1868b 100644 --- a/arch/x86/crypto/ghash-clmulni-intel_glue.c +++ b/arch/x86/crypto/ghash-clmulni-intel_glue.c @@ -364,5 +364,5 @@ module_exit(ghash_pclmulqdqni_mod_exit); MODULE_LICENSE("GPL"); MODULE_DESCRIPTION("GHASH Message Digest Algorithm, " - "acclerated by PCLMULQDQ-NI"); + "accelerated by PCLMULQDQ-NI"); MODULE_ALIAS_CRYPTO("ghash"); diff --git a/arch/x86/crypto/morus1280-avx2-asm.S b/arch/x86/crypto/morus1280-avx2-asm.S new file mode 100644 index 000000000000..37d422e77931 --- /dev/null +++ b/arch/x86/crypto/morus1280-avx2-asm.S @@ -0,0 +1,621 @@ +/* + * AVX2 implementation of MORUS-1280 + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation. + */ + +#include +#include + +#define SHUFFLE_MASK(i0, i1, i2, i3) \ + (i0 | (i1 << 2) | (i2 << 4) | (i3 << 6)) + +#define MASK1 SHUFFLE_MASK(3, 0, 1, 2) +#define MASK2 SHUFFLE_MASK(2, 3, 0, 1) +#define MASK3 SHUFFLE_MASK(1, 2, 3, 0) + +#define STATE0 %ymm0 +#define STATE0_LOW %xmm0 +#define STATE1 %ymm1 +#define STATE2 %ymm2 +#define STATE3 %ymm3 +#define STATE4 %ymm4 +#define KEY %ymm5 +#define MSG %ymm5 +#define MSG_LOW %xmm5 +#define T0 %ymm6 +#define T0_LOW %xmm6 +#define T1 %ymm7 + +.section .rodata.cst32.morus1280_const, "aM", @progbits, 32 +.align 32 +.Lmorus1280_const: + .byte 0x00, 0x01, 0x01, 0x02, 0x03, 0x05, 0x08, 0x0d + .byte 0x15, 0x22, 0x37, 0x59, 0x90, 0xe9, 0x79, 0x62 + .byte 0xdb, 0x3d, 0x18, 0x55, 0x6d, 0xc2, 0x2f, 0xf1 + .byte 0x20, 0x11, 0x31, 0x42, 0x73, 0xb5, 0x28, 0xdd + +.section .rodata.cst32.morus1280_counter, "aM", @progbits, 32 +.align 32 +.Lmorus1280_counter: + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f + .byte 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17 + .byte 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f + +.text + +.macro morus1280_round s0, s1, s2, s3, s4, b, w + vpand \s1, \s2, T0 + vpxor T0, \s0, \s0 + vpxor \s3, \s0, \s0 + vpsllq $\b, \s0, T0 + vpsrlq $(64 - \b), \s0, \s0 + vpxor T0, \s0, \s0 + vpermq $\w, \s3, \s3 +.endm + +/* + * __morus1280_update: internal ABI + * input: + * STATE[0-4] - input state + * MSG - message block + * output: + * STATE[0-4] - output state + * changed: + * T0 + */ +__morus1280_update: + morus1280_round STATE0, STATE1, STATE2, STATE3, STATE4, 13, MASK1 + vpxor MSG, STATE1, STATE1 + morus1280_round STATE1, STATE2, STATE3, STATE4, STATE0, 46, MASK2 + vpxor MSG, STATE2, STATE2 + morus1280_round STATE2, STATE3, STATE4, STATE0, STATE1, 38, MASK3 + vpxor MSG, STATE3, STATE3 + morus1280_round STATE3, STATE4, STATE0, STATE1, STATE2, 7, MASK2 + vpxor MSG, STATE4, STATE4 + morus1280_round STATE4, STATE0, STATE1, STATE2, STATE3, 4, MASK1 + ret +ENDPROC(__morus1280_update) + +/* + * __morus1280_update_zero: internal ABI + * input: + * STATE[0-4] - input state + * output: + * STATE[0-4] - output state + * changed: + * T0 + */ +__morus1280_update_zero: + morus1280_round STATE0, STATE1, STATE2, STATE3, STATE4, 13, MASK1 + morus1280_round STATE1, STATE2, STATE3, STATE4, STATE0, 46, MASK2 + morus1280_round STATE2, STATE3, STATE4, STATE0, STATE1, 38, MASK3 + morus1280_round STATE3, STATE4, STATE0, STATE1, STATE2, 7, MASK2 + morus1280_round STATE4, STATE0, STATE1, STATE2, STATE3, 4, MASK1 + ret +ENDPROC(__morus1280_update_zero) + +/* + * __load_partial: internal ABI + * input: + * %rsi - src + * %rcx - bytes + * output: + * MSG - message block + * changed: + * %r8 + * %r9 + */ +__load_partial: + xor %r9, %r9 + vpxor MSG, MSG, MSG + + mov %rcx, %r8 + and $0x1, %r8 + jz .Lld_partial_1 + + mov %rcx, %r8 + and $0x1E, %r8 + add %rsi, %r8 + mov (%r8), %r9b + +.Lld_partial_1: + mov %rcx, %r8 + and $0x2, %r8 + jz .Lld_partial_2 + + mov %rcx, %r8 + and $0x1C, %r8 + add %rsi, %r8 + shl $16, %r9 + mov (%r8), %r9w + +.Lld_partial_2: + mov %rcx, %r8 + and $0x4, %r8 + jz .Lld_partial_4 + + mov %rcx, %r8 + and $0x18, %r8 + add %rsi, %r8 + shl $32, %r9 + mov (%r8), %r8d + xor %r8, %r9 + +.Lld_partial_4: + movq %r9, MSG_LOW + + mov %rcx, %r8 + and $0x8, %r8 + jz .Lld_partial_8 + + mov %rcx, %r8 + and $0x10, %r8 + add %rsi, %r8 + pshufd $MASK2, MSG_LOW, MSG_LOW + pinsrq $0, (%r8), MSG_LOW + +.Lld_partial_8: + mov %rcx, %r8 + and $0x10, %r8 + jz .Lld_partial_16 + + vpermq $MASK2, MSG, MSG + movdqu (%rsi), MSG_LOW + +.Lld_partial_16: + ret +ENDPROC(__load_partial) + +/* + * __store_partial: internal ABI + * input: + * %rdx - dst + * %rcx - bytes + * output: + * T0 - message block + * changed: + * %r8 + * %r9 + * %r10 + */ +__store_partial: + mov %rcx, %r8 + mov %rdx, %r9 + + cmp $16, %r8 + jl .Lst_partial_16 + + movdqu T0_LOW, (%r9) + vpermq $MASK2, T0, T0 + + sub $16, %r8 + add $16, %r9 + +.Lst_partial_16: + movq T0_LOW, %r10 + + cmp $8, %r8 + jl .Lst_partial_8 + + mov %r10, (%r9) + pextrq $1, T0_LOW, %r10 + + sub $8, %r8 + add $8, %r9 + +.Lst_partial_8: + cmp $4, %r8 + jl .Lst_partial_4 + + mov %r10d, (%r9) + shr $32, %r10 + + sub $4, %r8 + add $4, %r9 + +.Lst_partial_4: + cmp $2, %r8 + jl .Lst_partial_2 + + mov %r10w, (%r9) + shr $16, %r10 + + sub $2, %r8 + add $2, %r9 + +.Lst_partial_2: + cmp $1, %r8 + jl .Lst_partial_1 + + mov %r10b, (%r9) + +.Lst_partial_1: + ret +ENDPROC(__store_partial) + +/* + * void crypto_morus1280_avx2_init(void *state, const void *key, + * const void *iv); + */ +ENTRY(crypto_morus1280_avx2_init) + FRAME_BEGIN + + /* load IV: */ + vpxor STATE0, STATE0, STATE0 + movdqu (%rdx), STATE0_LOW + /* load key: */ + vmovdqu (%rsi), KEY + vmovdqa KEY, STATE1 + /* load all ones: */ + vpcmpeqd STATE2, STATE2, STATE2 + /* load all zeros: */ + vpxor STATE3, STATE3, STATE3 + /* load the constant: */ + vmovdqa .Lmorus1280_const, STATE4 + + /* update 16 times with zero: */ + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + + /* xor-in the key again after updates: */ + vpxor KEY, STATE1, STATE1 + + /* store the state: */ + vmovdqu STATE0, (0 * 32)(%rdi) + vmovdqu STATE1, (1 * 32)(%rdi) + vmovdqu STATE2, (2 * 32)(%rdi) + vmovdqu STATE3, (3 * 32)(%rdi) + vmovdqu STATE4, (4 * 32)(%rdi) + + FRAME_END + ret +ENDPROC(crypto_morus1280_avx2_init) + +/* + * void crypto_morus1280_avx2_ad(void *state, const void *data, + * unsigned int length); + */ +ENTRY(crypto_morus1280_avx2_ad) + FRAME_BEGIN + + cmp $32, %rdx + jb .Lad_out + + /* load the state: */ + vmovdqu (0 * 32)(%rdi), STATE0 + vmovdqu (1 * 32)(%rdi), STATE1 + vmovdqu (2 * 32)(%rdi), STATE2 + vmovdqu (3 * 32)(%rdi), STATE3 + vmovdqu (4 * 32)(%rdi), STATE4 + + mov %rsi, %r8 + and $0x1F, %r8 + jnz .Lad_u_loop + +.align 4 +.Lad_a_loop: + vmovdqa (%rsi), MSG + call __morus1280_update + sub $32, %rdx + add $32, %rsi + cmp $32, %rdx + jge .Lad_a_loop + + jmp .Lad_cont +.align 4 +.Lad_u_loop: + vmovdqu (%rsi), MSG + call __morus1280_update + sub $32, %rdx + add $32, %rsi + cmp $32, %rdx + jge .Lad_u_loop + +.Lad_cont: + /* store the state: */ + vmovdqu STATE0, (0 * 32)(%rdi) + vmovdqu STATE1, (1 * 32)(%rdi) + vmovdqu STATE2, (2 * 32)(%rdi) + vmovdqu STATE3, (3 * 32)(%rdi) + vmovdqu STATE4, (4 * 32)(%rdi) + +.Lad_out: + FRAME_END + ret +ENDPROC(crypto_morus1280_avx2_ad) + +/* + * void crypto_morus1280_avx2_enc(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus1280_avx2_enc) + FRAME_BEGIN + + cmp $32, %rcx + jb .Lenc_out + + /* load the state: */ + vmovdqu (0 * 32)(%rdi), STATE0 + vmovdqu (1 * 32)(%rdi), STATE1 + vmovdqu (2 * 32)(%rdi), STATE2 + vmovdqu (3 * 32)(%rdi), STATE3 + vmovdqu (4 * 32)(%rdi), STATE4 + + mov %rsi, %r8 + or %rdx, %r8 + and $0x1F, %r8 + jnz .Lenc_u_loop + +.align 4 +.Lenc_a_loop: + vmovdqa (%rsi), MSG + vmovdqa MSG, T0 + vpxor STATE0, T0, T0 + vpermq $MASK3, STATE1, T1 + vpxor T1, T0, T0 + vpand STATE2, STATE3, T1 + vpxor T1, T0, T0 + vmovdqa T0, (%rdx) + + call __morus1280_update + sub $32, %rcx + add $32, %rsi + add $32, %rdx + cmp $32, %rcx + jge .Lenc_a_loop + + jmp .Lenc_cont +.align 4 +.Lenc_u_loop: + vmovdqu (%rsi), MSG + vmovdqa MSG, T0 + vpxor STATE0, T0, T0 + vpermq $MASK3, STATE1, T1 + vpxor T1, T0, T0 + vpand STATE2, STATE3, T1 + vpxor T1, T0, T0 + vmovdqu T0, (%rdx) + + call __morus1280_update + sub $32, %rcx + add $32, %rsi + add $32, %rdx + cmp $32, %rcx + jge .Lenc_u_loop + +.Lenc_cont: + /* store the state: */ + vmovdqu STATE0, (0 * 32)(%rdi) + vmovdqu STATE1, (1 * 32)(%rdi) + vmovdqu STATE2, (2 * 32)(%rdi) + vmovdqu STATE3, (3 * 32)(%rdi) + vmovdqu STATE4, (4 * 32)(%rdi) + +.Lenc_out: + FRAME_END + ret +ENDPROC(crypto_morus1280_avx2_enc) + +/* + * void crypto_morus1280_avx2_enc_tail(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus1280_avx2_enc_tail) + FRAME_BEGIN + + /* load the state: */ + vmovdqu (0 * 32)(%rdi), STATE0 + vmovdqu (1 * 32)(%rdi), STATE1 + vmovdqu (2 * 32)(%rdi), STATE2 + vmovdqu (3 * 32)(%rdi), STATE3 + vmovdqu (4 * 32)(%rdi), STATE4 + + /* encrypt message: */ + call __load_partial + + vmovdqa MSG, T0 + vpxor STATE0, T0, T0 + vpermq $MASK3, STATE1, T1 + vpxor T1, T0, T0 + vpand STATE2, STATE3, T1 + vpxor T1, T0, T0 + + call __store_partial + + call __morus1280_update + + /* store the state: */ + vmovdqu STATE0, (0 * 32)(%rdi) + vmovdqu STATE1, (1 * 32)(%rdi) + vmovdqu STATE2, (2 * 32)(%rdi) + vmovdqu STATE3, (3 * 32)(%rdi) + vmovdqu STATE4, (4 * 32)(%rdi) + + FRAME_END +ENDPROC(crypto_morus1280_avx2_enc_tail) + +/* + * void crypto_morus1280_avx2_dec(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus1280_avx2_dec) + FRAME_BEGIN + + cmp $32, %rcx + jb .Ldec_out + + /* load the state: */ + vmovdqu (0 * 32)(%rdi), STATE0 + vmovdqu (1 * 32)(%rdi), STATE1 + vmovdqu (2 * 32)(%rdi), STATE2 + vmovdqu (3 * 32)(%rdi), STATE3 + vmovdqu (4 * 32)(%rdi), STATE4 + + mov %rsi, %r8 + or %rdx, %r8 + and $0x1F, %r8 + jnz .Ldec_u_loop + +.align 4 +.Ldec_a_loop: + vmovdqa (%rsi), MSG + vpxor STATE0, MSG, MSG + vpermq $MASK3, STATE1, T0 + vpxor T0, MSG, MSG + vpand STATE2, STATE3, T0 + vpxor T0, MSG, MSG + vmovdqa MSG, (%rdx) + + call __morus1280_update + sub $32, %rcx + add $32, %rsi + add $32, %rdx + cmp $32, %rcx + jge .Ldec_a_loop + + jmp .Ldec_cont +.align 4 +.Ldec_u_loop: + vmovdqu (%rsi), MSG + vpxor STATE0, MSG, MSG + vpermq $MASK3, STATE1, T0 + vpxor T0, MSG, MSG + vpand STATE2, STATE3, T0 + vpxor T0, MSG, MSG + vmovdqu MSG, (%rdx) + + call __morus1280_update + sub $32, %rcx + add $32, %rsi + add $32, %rdx + cmp $32, %rcx + jge .Ldec_u_loop + +.Ldec_cont: + /* store the state: */ + vmovdqu STATE0, (0 * 32)(%rdi) + vmovdqu STATE1, (1 * 32)(%rdi) + vmovdqu STATE2, (2 * 32)(%rdi) + vmovdqu STATE3, (3 * 32)(%rdi) + vmovdqu STATE4, (4 * 32)(%rdi) + +.Ldec_out: + FRAME_END + ret +ENDPROC(crypto_morus1280_avx2_dec) + +/* + * void crypto_morus1280_avx2_dec_tail(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus1280_avx2_dec_tail) + FRAME_BEGIN + + /* load the state: */ + vmovdqu (0 * 32)(%rdi), STATE0 + vmovdqu (1 * 32)(%rdi), STATE1 + vmovdqu (2 * 32)(%rdi), STATE2 + vmovdqu (3 * 32)(%rdi), STATE3 + vmovdqu (4 * 32)(%rdi), STATE4 + + /* decrypt message: */ + call __load_partial + + vpxor STATE0, MSG, MSG + vpermq $MASK3, STATE1, T0 + vpxor T0, MSG, MSG + vpand STATE2, STATE3, T0 + vpxor T0, MSG, MSG + vmovdqa MSG, T0 + + call __store_partial + + /* mask with byte count: */ + movq %rcx, T0_LOW + vpbroadcastb T0_LOW, T0 + vmovdqa .Lmorus1280_counter, T1 + vpcmpgtb T1, T0, T0 + vpand T0, MSG, MSG + + call __morus1280_update + + /* store the state: */ + vmovdqu STATE0, (0 * 32)(%rdi) + vmovdqu STATE1, (1 * 32)(%rdi) + vmovdqu STATE2, (2 * 32)(%rdi) + vmovdqu STATE3, (3 * 32)(%rdi) + vmovdqu STATE4, (4 * 32)(%rdi) + + FRAME_END + ret +ENDPROC(crypto_morus1280_avx2_dec_tail) + +/* + * void crypto_morus1280_avx2_final(void *state, void *tag_xor, + * u64 assoclen, u64 cryptlen); + */ +ENTRY(crypto_morus1280_avx2_final) + FRAME_BEGIN + + /* load the state: */ + vmovdqu (0 * 32)(%rdi), STATE0 + vmovdqu (1 * 32)(%rdi), STATE1 + vmovdqu (2 * 32)(%rdi), STATE2 + vmovdqu (3 * 32)(%rdi), STATE3 + vmovdqu (4 * 32)(%rdi), STATE4 + + /* xor state[0] into state[4]: */ + vpxor STATE0, STATE4, STATE4 + + /* prepare length block: */ + vpxor MSG, MSG, MSG + vpinsrq $0, %rdx, MSG_LOW, MSG_LOW + vpinsrq $1, %rcx, MSG_LOW, MSG_LOW + vpsllq $3, MSG, MSG /* multiply by 8 (to get bit count) */ + + /* update state: */ + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + + /* xor tag: */ + vmovdqu (%rsi), MSG + + vpxor STATE0, MSG, MSG + vpermq $MASK3, STATE1, T0 + vpxor T0, MSG, MSG + vpand STATE2, STATE3, T0 + vpxor T0, MSG, MSG + vmovdqu MSG, (%rsi) + + FRAME_END + ret +ENDPROC(crypto_morus1280_avx2_final) diff --git a/arch/x86/crypto/morus1280-avx2-glue.c b/arch/x86/crypto/morus1280-avx2-glue.c new file mode 100644 index 000000000000..f111f36d26dc --- /dev/null +++ b/arch/x86/crypto/morus1280-avx2-glue.c @@ -0,0 +1,68 @@ +/* + * The MORUS-1280 Authenticated-Encryption Algorithm + * Glue for AVX2 implementation + * + * Copyright (c) 2016-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include + +asmlinkage void crypto_morus1280_avx2_init(void *state, const void *key, + const void *iv); +asmlinkage void crypto_morus1280_avx2_ad(void *state, const void *data, + unsigned int length); + +asmlinkage void crypto_morus1280_avx2_enc(void *state, const void *src, + void *dst, unsigned int length); +asmlinkage void crypto_morus1280_avx2_dec(void *state, const void *src, + void *dst, unsigned int length); + +asmlinkage void crypto_morus1280_avx2_enc_tail(void *state, const void *src, + void *dst, unsigned int length); +asmlinkage void crypto_morus1280_avx2_dec_tail(void *state, const void *src, + void *dst, unsigned int length); + +asmlinkage void crypto_morus1280_avx2_final(void *state, void *tag_xor, + u64 assoclen, u64 cryptlen); + +MORUS1280_DECLARE_ALGS(avx2, "morus1280-avx2", 400); + +static const struct x86_cpu_id avx2_cpu_id[] = { + X86_FEATURE_MATCH(X86_FEATURE_AVX2), + {} +}; +MODULE_DEVICE_TABLE(x86cpu, avx2_cpu_id); + +static int __init crypto_morus1280_avx2_module_init(void) +{ + if (!x86_match_cpu(avx2_cpu_id)) + return -ENODEV; + + return crypto_register_aeads(crypto_morus1280_avx2_algs, + ARRAY_SIZE(crypto_morus1280_avx2_algs)); +} + +static void __exit crypto_morus1280_avx2_module_exit(void) +{ + crypto_unregister_aeads(crypto_morus1280_avx2_algs, + ARRAY_SIZE(crypto_morus1280_avx2_algs)); +} + +module_init(crypto_morus1280_avx2_module_init); +module_exit(crypto_morus1280_avx2_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("MORUS-1280 AEAD algorithm -- AVX2 implementation"); +MODULE_ALIAS_CRYPTO("morus1280"); +MODULE_ALIAS_CRYPTO("morus1280-avx2"); diff --git a/arch/x86/crypto/morus1280-sse2-asm.S b/arch/x86/crypto/morus1280-sse2-asm.S new file mode 100644 index 000000000000..1fe637c7be9d --- /dev/null +++ b/arch/x86/crypto/morus1280-sse2-asm.S @@ -0,0 +1,895 @@ +/* + * SSE2 implementation of MORUS-1280 + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation. + */ + +#include +#include + +#define SHUFFLE_MASK(i0, i1, i2, i3) \ + (i0 | (i1 << 2) | (i2 << 4) | (i3 << 6)) + +#define MASK2 SHUFFLE_MASK(2, 3, 0, 1) + +#define STATE0_LO %xmm0 +#define STATE0_HI %xmm1 +#define STATE1_LO %xmm2 +#define STATE1_HI %xmm3 +#define STATE2_LO %xmm4 +#define STATE2_HI %xmm5 +#define STATE3_LO %xmm6 +#define STATE3_HI %xmm7 +#define STATE4_LO %xmm8 +#define STATE4_HI %xmm9 +#define KEY_LO %xmm10 +#define KEY_HI %xmm11 +#define MSG_LO %xmm10 +#define MSG_HI %xmm11 +#define T0_LO %xmm12 +#define T0_HI %xmm13 +#define T1_LO %xmm14 +#define T1_HI %xmm15 + +.section .rodata.cst16.morus640_const, "aM", @progbits, 16 +.align 16 +.Lmorus640_const_0: + .byte 0x00, 0x01, 0x01, 0x02, 0x03, 0x05, 0x08, 0x0d + .byte 0x15, 0x22, 0x37, 0x59, 0x90, 0xe9, 0x79, 0x62 +.Lmorus640_const_1: + .byte 0xdb, 0x3d, 0x18, 0x55, 0x6d, 0xc2, 0x2f, 0xf1 + .byte 0x20, 0x11, 0x31, 0x42, 0x73, 0xb5, 0x28, 0xdd + +.section .rodata.cst16.morus640_counter, "aM", @progbits, 16 +.align 16 +.Lmorus640_counter_0: + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f +.Lmorus640_counter_1: + .byte 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17 + .byte 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f + +.text + +.macro rol1 hi, lo + /* + * HI_1 | HI_0 || LO_1 | LO_0 + * ==> + * HI_0 | HI_1 || LO_1 | LO_0 + * ==> + * HI_0 | LO_1 || LO_0 | HI_1 + */ + pshufd $MASK2, \hi, \hi + movdqa \hi, T0_LO + punpcklqdq \lo, T0_LO + punpckhqdq \hi, \lo + movdqa \lo, \hi + movdqa T0_LO, \lo +.endm + +.macro rol2 hi, lo + movdqa \lo, T0_LO + movdqa \hi, \lo + movdqa T0_LO, \hi +.endm + +.macro rol3 hi, lo + /* + * HI_1 | HI_0 || LO_1 | LO_0 + * ==> + * HI_0 | HI_1 || LO_1 | LO_0 + * ==> + * LO_0 | HI_1 || HI_0 | LO_1 + */ + pshufd $MASK2, \hi, \hi + movdqa \lo, T0_LO + punpckhqdq \hi, T0_LO + punpcklqdq \lo, \hi + movdqa T0_LO, \lo +.endm + +.macro morus1280_round s0_l, s0_h, s1_l, s1_h, s2_l, s2_h, s3_l, s3_h, s4_l, s4_h, b, w + movdqa \s1_l, T0_LO + pand \s2_l, T0_LO + pxor T0_LO, \s0_l + + movdqa \s1_h, T0_LO + pand \s2_h, T0_LO + pxor T0_LO, \s0_h + + pxor \s3_l, \s0_l + pxor \s3_h, \s0_h + + movdqa \s0_l, T0_LO + psllq $\b, T0_LO + psrlq $(64 - \b), \s0_l + pxor T0_LO, \s0_l + + movdqa \s0_h, T0_LO + psllq $\b, T0_LO + psrlq $(64 - \b), \s0_h + pxor T0_LO, \s0_h + + \w \s3_h, \s3_l +.endm + +/* + * __morus1280_update: internal ABI + * input: + * STATE[0-4] - input state + * MSG - message block + * output: + * STATE[0-4] - output state + * changed: + * T0 + */ +__morus1280_update: + morus1280_round \ + STATE0_LO, STATE0_HI, \ + STATE1_LO, STATE1_HI, \ + STATE2_LO, STATE2_HI, \ + STATE3_LO, STATE3_HI, \ + STATE4_LO, STATE4_HI, \ + 13, rol1 + pxor MSG_LO, STATE1_LO + pxor MSG_HI, STATE1_HI + morus1280_round \ + STATE1_LO, STATE1_HI, \ + STATE2_LO, STATE2_HI, \ + STATE3_LO, STATE3_HI, \ + STATE4_LO, STATE4_HI, \ + STATE0_LO, STATE0_HI, \ + 46, rol2 + pxor MSG_LO, STATE2_LO + pxor MSG_HI, STATE2_HI + morus1280_round \ + STATE2_LO, STATE2_HI, \ + STATE3_LO, STATE3_HI, \ + STATE4_LO, STATE4_HI, \ + STATE0_LO, STATE0_HI, \ + STATE1_LO, STATE1_HI, \ + 38, rol3 + pxor MSG_LO, STATE3_LO + pxor MSG_HI, STATE3_HI + morus1280_round \ + STATE3_LO, STATE3_HI, \ + STATE4_LO, STATE4_HI, \ + STATE0_LO, STATE0_HI, \ + STATE1_LO, STATE1_HI, \ + STATE2_LO, STATE2_HI, \ + 7, rol2 + pxor MSG_LO, STATE4_LO + pxor MSG_HI, STATE4_HI + morus1280_round \ + STATE4_LO, STATE4_HI, \ + STATE0_LO, STATE0_HI, \ + STATE1_LO, STATE1_HI, \ + STATE2_LO, STATE2_HI, \ + STATE3_LO, STATE3_HI, \ + 4, rol1 + ret +ENDPROC(__morus1280_update) + +/* + * __morus1280_update_zero: internal ABI + * input: + * STATE[0-4] - input state + * output: + * STATE[0-4] - output state + * changed: + * T0 + */ +__morus1280_update_zero: + morus1280_round \ + STATE0_LO, STATE0_HI, \ + STATE1_LO, STATE1_HI, \ + STATE2_LO, STATE2_HI, \ + STATE3_LO, STATE3_HI, \ + STATE4_LO, STATE4_HI, \ + 13, rol1 + morus1280_round \ + STATE1_LO, STATE1_HI, \ + STATE2_LO, STATE2_HI, \ + STATE3_LO, STATE3_HI, \ + STATE4_LO, STATE4_HI, \ + STATE0_LO, STATE0_HI, \ + 46, rol2 + morus1280_round \ + STATE2_LO, STATE2_HI, \ + STATE3_LO, STATE3_HI, \ + STATE4_LO, STATE4_HI, \ + STATE0_LO, STATE0_HI, \ + STATE1_LO, STATE1_HI, \ + 38, rol3 + morus1280_round \ + STATE3_LO, STATE3_HI, \ + STATE4_LO, STATE4_HI, \ + STATE0_LO, STATE0_HI, \ + STATE1_LO, STATE1_HI, \ + STATE2_LO, STATE2_HI, \ + 7, rol2 + morus1280_round \ + STATE4_LO, STATE4_HI, \ + STATE0_LO, STATE0_HI, \ + STATE1_LO, STATE1_HI, \ + STATE2_LO, STATE2_HI, \ + STATE3_LO, STATE3_HI, \ + 4, rol1 + ret +ENDPROC(__morus1280_update_zero) + +/* + * __load_partial: internal ABI + * input: + * %rsi - src + * %rcx - bytes + * output: + * MSG - message block + * changed: + * %r8 + * %r9 + */ +__load_partial: + xor %r9, %r9 + pxor MSG_LO, MSG_LO + pxor MSG_HI, MSG_HI + + mov %rcx, %r8 + and $0x1, %r8 + jz .Lld_partial_1 + + mov %rcx, %r8 + and $0x1E, %r8 + add %rsi, %r8 + mov (%r8), %r9b + +.Lld_partial_1: + mov %rcx, %r8 + and $0x2, %r8 + jz .Lld_partial_2 + + mov %rcx, %r8 + and $0x1C, %r8 + add %rsi, %r8 + shl $16, %r9 + mov (%r8), %r9w + +.Lld_partial_2: + mov %rcx, %r8 + and $0x4, %r8 + jz .Lld_partial_4 + + mov %rcx, %r8 + and $0x18, %r8 + add %rsi, %r8 + shl $32, %r9 + mov (%r8), %r8d + xor %r8, %r9 + +.Lld_partial_4: + movq %r9, MSG_LO + + mov %rcx, %r8 + and $0x8, %r8 + jz .Lld_partial_8 + + mov %rcx, %r8 + and $0x10, %r8 + add %rsi, %r8 + pslldq $8, MSG_LO + movq (%r8), T0_LO + pxor T0_LO, MSG_LO + +.Lld_partial_8: + mov %rcx, %r8 + and $0x10, %r8 + jz .Lld_partial_16 + + movdqa MSG_LO, MSG_HI + movdqu (%rsi), MSG_LO + +.Lld_partial_16: + ret +ENDPROC(__load_partial) + +/* + * __store_partial: internal ABI + * input: + * %rdx - dst + * %rcx - bytes + * output: + * T0 - message block + * changed: + * %r8 + * %r9 + * %r10 + */ +__store_partial: + mov %rcx, %r8 + mov %rdx, %r9 + + cmp $16, %r8 + jl .Lst_partial_16 + + movdqu T0_LO, (%r9) + movdqa T0_HI, T0_LO + + sub $16, %r8 + add $16, %r9 + +.Lst_partial_16: + movq T0_LO, %r10 + + cmp $8, %r8 + jl .Lst_partial_8 + + mov %r10, (%r9) + psrldq $8, T0_LO + movq T0_LO, %r10 + + sub $8, %r8 + add $8, %r9 + +.Lst_partial_8: + cmp $4, %r8 + jl .Lst_partial_4 + + mov %r10d, (%r9) + shr $32, %r10 + + sub $4, %r8 + add $4, %r9 + +.Lst_partial_4: + cmp $2, %r8 + jl .Lst_partial_2 + + mov %r10w, (%r9) + shr $16, %r10 + + sub $2, %r8 + add $2, %r9 + +.Lst_partial_2: + cmp $1, %r8 + jl .Lst_partial_1 + + mov %r10b, (%r9) + +.Lst_partial_1: + ret +ENDPROC(__store_partial) + +/* + * void crypto_morus1280_sse2_init(void *state, const void *key, + * const void *iv); + */ +ENTRY(crypto_morus1280_sse2_init) + FRAME_BEGIN + + /* load IV: */ + pxor STATE0_HI, STATE0_HI + movdqu (%rdx), STATE0_LO + /* load key: */ + movdqu 0(%rsi), KEY_LO + movdqu 16(%rsi), KEY_HI + movdqa KEY_LO, STATE1_LO + movdqa KEY_HI, STATE1_HI + /* load all ones: */ + pcmpeqd STATE2_LO, STATE2_LO + pcmpeqd STATE2_HI, STATE2_HI + /* load all zeros: */ + pxor STATE3_LO, STATE3_LO + pxor STATE3_HI, STATE3_HI + /* load the constant: */ + movdqa .Lmorus640_const_0, STATE4_LO + movdqa .Lmorus640_const_1, STATE4_HI + + /* update 16 times with zero: */ + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + call __morus1280_update_zero + + /* xor-in the key again after updates: */ + pxor KEY_LO, STATE1_LO + pxor KEY_HI, STATE1_HI + + /* store the state: */ + movdqu STATE0_LO, (0 * 16)(%rdi) + movdqu STATE0_HI, (1 * 16)(%rdi) + movdqu STATE1_LO, (2 * 16)(%rdi) + movdqu STATE1_HI, (3 * 16)(%rdi) + movdqu STATE2_LO, (4 * 16)(%rdi) + movdqu STATE2_HI, (5 * 16)(%rdi) + movdqu STATE3_LO, (6 * 16)(%rdi) + movdqu STATE3_HI, (7 * 16)(%rdi) + movdqu STATE4_LO, (8 * 16)(%rdi) + movdqu STATE4_HI, (9 * 16)(%rdi) + + FRAME_END + ret +ENDPROC(crypto_morus1280_sse2_init) + +/* + * void crypto_morus1280_sse2_ad(void *state, const void *data, + * unsigned int length); + */ +ENTRY(crypto_morus1280_sse2_ad) + FRAME_BEGIN + + cmp $32, %rdx + jb .Lad_out + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0_LO + movdqu (1 * 16)(%rdi), STATE0_HI + movdqu (2 * 16)(%rdi), STATE1_LO + movdqu (3 * 16)(%rdi), STATE1_HI + movdqu (4 * 16)(%rdi), STATE2_LO + movdqu (5 * 16)(%rdi), STATE2_HI + movdqu (6 * 16)(%rdi), STATE3_LO + movdqu (7 * 16)(%rdi), STATE3_HI + movdqu (8 * 16)(%rdi), STATE4_LO + movdqu (9 * 16)(%rdi), STATE4_HI + + mov %rsi, %r8 + and $0xF, %r8 + jnz .Lad_u_loop + +.align 4 +.Lad_a_loop: + movdqa 0(%rsi), MSG_LO + movdqa 16(%rsi), MSG_HI + call __morus1280_update + sub $32, %rdx + add $32, %rsi + cmp $32, %rdx + jge .Lad_a_loop + + jmp .Lad_cont +.align 4 +.Lad_u_loop: + movdqu 0(%rsi), MSG_LO + movdqu 16(%rsi), MSG_HI + call __morus1280_update + sub $32, %rdx + add $32, %rsi + cmp $32, %rdx + jge .Lad_u_loop + +.Lad_cont: + /* store the state: */ + movdqu STATE0_LO, (0 * 16)(%rdi) + movdqu STATE0_HI, (1 * 16)(%rdi) + movdqu STATE1_LO, (2 * 16)(%rdi) + movdqu STATE1_HI, (3 * 16)(%rdi) + movdqu STATE2_LO, (4 * 16)(%rdi) + movdqu STATE2_HI, (5 * 16)(%rdi) + movdqu STATE3_LO, (6 * 16)(%rdi) + movdqu STATE3_HI, (7 * 16)(%rdi) + movdqu STATE4_LO, (8 * 16)(%rdi) + movdqu STATE4_HI, (9 * 16)(%rdi) + +.Lad_out: + FRAME_END + ret +ENDPROC(crypto_morus1280_sse2_ad) + +/* + * void crypto_morus1280_sse2_enc(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus1280_sse2_enc) + FRAME_BEGIN + + cmp $32, %rcx + jb .Lenc_out + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0_LO + movdqu (1 * 16)(%rdi), STATE0_HI + movdqu (2 * 16)(%rdi), STATE1_LO + movdqu (3 * 16)(%rdi), STATE1_HI + movdqu (4 * 16)(%rdi), STATE2_LO + movdqu (5 * 16)(%rdi), STATE2_HI + movdqu (6 * 16)(%rdi), STATE3_LO + movdqu (7 * 16)(%rdi), STATE3_HI + movdqu (8 * 16)(%rdi), STATE4_LO + movdqu (9 * 16)(%rdi), STATE4_HI + + mov %rsi, %r8 + or %rdx, %r8 + and $0xF, %r8 + jnz .Lenc_u_loop + +.align 4 +.Lenc_a_loop: + movdqa 0(%rsi), MSG_LO + movdqa 16(%rsi), MSG_HI + movdqa STATE1_LO, T1_LO + movdqa STATE1_HI, T1_HI + rol3 T1_HI, T1_LO + movdqa MSG_LO, T0_LO + movdqa MSG_HI, T0_HI + pxor T1_LO, T0_LO + pxor T1_HI, T0_HI + pxor STATE0_LO, T0_LO + pxor STATE0_HI, T0_HI + movdqa STATE2_LO, T1_LO + movdqa STATE2_HI, T1_HI + pand STATE3_LO, T1_LO + pand STATE3_HI, T1_HI + pxor T1_LO, T0_LO + pxor T1_HI, T0_HI + movdqa T0_LO, 0(%rdx) + movdqa T0_HI, 16(%rdx) + + call __morus1280_update + sub $32, %rcx + add $32, %rsi + add $32, %rdx + cmp $32, %rcx + jge .Lenc_a_loop + + jmp .Lenc_cont +.align 4 +.Lenc_u_loop: + movdqu 0(%rsi), MSG_LO + movdqu 16(%rsi), MSG_HI + movdqa STATE1_LO, T1_LO + movdqa STATE1_HI, T1_HI + rol3 T1_HI, T1_LO + movdqa MSG_LO, T0_LO + movdqa MSG_HI, T0_HI + pxor T1_LO, T0_LO + pxor T1_HI, T0_HI + pxor STATE0_LO, T0_LO + pxor STATE0_HI, T0_HI + movdqa STATE2_LO, T1_LO + movdqa STATE2_HI, T1_HI + pand STATE3_LO, T1_LO + pand STATE3_HI, T1_HI + pxor T1_LO, T0_LO + pxor T1_HI, T0_HI + movdqu T0_LO, 0(%rdx) + movdqu T0_HI, 16(%rdx) + + call __morus1280_update + sub $32, %rcx + add $32, %rsi + add $32, %rdx + cmp $32, %rcx + jge .Lenc_u_loop + +.Lenc_cont: + /* store the state: */ + movdqu STATE0_LO, (0 * 16)(%rdi) + movdqu STATE0_HI, (1 * 16)(%rdi) + movdqu STATE1_LO, (2 * 16)(%rdi) + movdqu STATE1_HI, (3 * 16)(%rdi) + movdqu STATE2_LO, (4 * 16)(%rdi) + movdqu STATE2_HI, (5 * 16)(%rdi) + movdqu STATE3_LO, (6 * 16)(%rdi) + movdqu STATE3_HI, (7 * 16)(%rdi) + movdqu STATE4_LO, (8 * 16)(%rdi) + movdqu STATE4_HI, (9 * 16)(%rdi) + +.Lenc_out: + FRAME_END + ret +ENDPROC(crypto_morus1280_sse2_enc) + +/* + * void crypto_morus1280_sse2_enc_tail(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus1280_sse2_enc_tail) + FRAME_BEGIN + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0_LO + movdqu (1 * 16)(%rdi), STATE0_HI + movdqu (2 * 16)(%rdi), STATE1_LO + movdqu (3 * 16)(%rdi), STATE1_HI + movdqu (4 * 16)(%rdi), STATE2_LO + movdqu (5 * 16)(%rdi), STATE2_HI + movdqu (6 * 16)(%rdi), STATE3_LO + movdqu (7 * 16)(%rdi), STATE3_HI + movdqu (8 * 16)(%rdi), STATE4_LO + movdqu (9 * 16)(%rdi), STATE4_HI + + /* encrypt message: */ + call __load_partial + + movdqa STATE1_LO, T1_LO + movdqa STATE1_HI, T1_HI + rol3 T1_HI, T1_LO + movdqa MSG_LO, T0_LO + movdqa MSG_HI, T0_HI + pxor T1_LO, T0_LO + pxor T1_HI, T0_HI + pxor STATE0_LO, T0_LO + pxor STATE0_HI, T0_HI + movdqa STATE2_LO, T1_LO + movdqa STATE2_HI, T1_HI + pand STATE3_LO, T1_LO + pand STATE3_HI, T1_HI + pxor T1_LO, T0_LO + pxor T1_HI, T0_HI + + call __store_partial + + call __morus1280_update + + /* store the state: */ + movdqu STATE0_LO, (0 * 16)(%rdi) + movdqu STATE0_HI, (1 * 16)(%rdi) + movdqu STATE1_LO, (2 * 16)(%rdi) + movdqu STATE1_HI, (3 * 16)(%rdi) + movdqu STATE2_LO, (4 * 16)(%rdi) + movdqu STATE2_HI, (5 * 16)(%rdi) + movdqu STATE3_LO, (6 * 16)(%rdi) + movdqu STATE3_HI, (7 * 16)(%rdi) + movdqu STATE4_LO, (8 * 16)(%rdi) + movdqu STATE4_HI, (9 * 16)(%rdi) + + FRAME_END +ENDPROC(crypto_morus1280_sse2_enc_tail) + +/* + * void crypto_morus1280_sse2_dec(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus1280_sse2_dec) + FRAME_BEGIN + + cmp $32, %rcx + jb .Ldec_out + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0_LO + movdqu (1 * 16)(%rdi), STATE0_HI + movdqu (2 * 16)(%rdi), STATE1_LO + movdqu (3 * 16)(%rdi), STATE1_HI + movdqu (4 * 16)(%rdi), STATE2_LO + movdqu (5 * 16)(%rdi), STATE2_HI + movdqu (6 * 16)(%rdi), STATE3_LO + movdqu (7 * 16)(%rdi), STATE3_HI + movdqu (8 * 16)(%rdi), STATE4_LO + movdqu (9 * 16)(%rdi), STATE4_HI + + mov %rsi, %r8 + or %rdx, %r8 + and $0xF, %r8 + jnz .Ldec_u_loop + +.align 4 +.Ldec_a_loop: + movdqa 0(%rsi), MSG_LO + movdqa 16(%rsi), MSG_HI + pxor STATE0_LO, MSG_LO + pxor STATE0_HI, MSG_HI + movdqa STATE1_LO, T1_LO + movdqa STATE1_HI, T1_HI + rol3 T1_HI, T1_LO + pxor T1_LO, MSG_LO + pxor T1_HI, MSG_HI + movdqa STATE2_LO, T1_LO + movdqa STATE2_HI, T1_HI + pand STATE3_LO, T1_LO + pand STATE3_HI, T1_HI + pxor T1_LO, MSG_LO + pxor T1_HI, MSG_HI + movdqa MSG_LO, 0(%rdx) + movdqa MSG_HI, 16(%rdx) + + call __morus1280_update + sub $32, %rcx + add $32, %rsi + add $32, %rdx + cmp $32, %rcx + jge .Ldec_a_loop + + jmp .Ldec_cont +.align 4 +.Ldec_u_loop: + movdqu 0(%rsi), MSG_LO + movdqu 16(%rsi), MSG_HI + pxor STATE0_LO, MSG_LO + pxor STATE0_HI, MSG_HI + movdqa STATE1_LO, T1_LO + movdqa STATE1_HI, T1_HI + rol3 T1_HI, T1_LO + pxor T1_LO, MSG_LO + pxor T1_HI, MSG_HI + movdqa STATE2_LO, T1_LO + movdqa STATE2_HI, T1_HI + pand STATE3_LO, T1_LO + pand STATE3_HI, T1_HI + pxor T1_LO, MSG_LO + pxor T1_HI, MSG_HI + movdqu MSG_LO, 0(%rdx) + movdqu MSG_HI, 16(%rdx) + + call __morus1280_update + sub $32, %rcx + add $32, %rsi + add $32, %rdx + cmp $32, %rcx + jge .Ldec_u_loop + +.Ldec_cont: + /* store the state: */ + movdqu STATE0_LO, (0 * 16)(%rdi) + movdqu STATE0_HI, (1 * 16)(%rdi) + movdqu STATE1_LO, (2 * 16)(%rdi) + movdqu STATE1_HI, (3 * 16)(%rdi) + movdqu STATE2_LO, (4 * 16)(%rdi) + movdqu STATE2_HI, (5 * 16)(%rdi) + movdqu STATE3_LO, (6 * 16)(%rdi) + movdqu STATE3_HI, (7 * 16)(%rdi) + movdqu STATE4_LO, (8 * 16)(%rdi) + movdqu STATE4_HI, (9 * 16)(%rdi) + +.Ldec_out: + FRAME_END + ret +ENDPROC(crypto_morus1280_sse2_dec) + +/* + * void crypto_morus1280_sse2_dec_tail(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus1280_sse2_dec_tail) + FRAME_BEGIN + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0_LO + movdqu (1 * 16)(%rdi), STATE0_HI + movdqu (2 * 16)(%rdi), STATE1_LO + movdqu (3 * 16)(%rdi), STATE1_HI + movdqu (4 * 16)(%rdi), STATE2_LO + movdqu (5 * 16)(%rdi), STATE2_HI + movdqu (6 * 16)(%rdi), STATE3_LO + movdqu (7 * 16)(%rdi), STATE3_HI + movdqu (8 * 16)(%rdi), STATE4_LO + movdqu (9 * 16)(%rdi), STATE4_HI + + /* decrypt message: */ + call __load_partial + + pxor STATE0_LO, MSG_LO + pxor STATE0_HI, MSG_HI + movdqa STATE1_LO, T1_LO + movdqa STATE1_HI, T1_HI + rol3 T1_HI, T1_LO + pxor T1_LO, MSG_LO + pxor T1_HI, MSG_HI + movdqa STATE2_LO, T1_LO + movdqa STATE2_HI, T1_HI + pand STATE3_LO, T1_LO + pand STATE3_HI, T1_HI + pxor T1_LO, MSG_LO + pxor T1_HI, MSG_HI + movdqa MSG_LO, T0_LO + movdqa MSG_HI, T0_HI + + call __store_partial + + /* mask with byte count: */ + movq %rcx, T0_LO + punpcklbw T0_LO, T0_LO + punpcklbw T0_LO, T0_LO + punpcklbw T0_LO, T0_LO + punpcklbw T0_LO, T0_LO + movdqa T0_LO, T0_HI + movdqa .Lmorus640_counter_0, T1_LO + movdqa .Lmorus640_counter_1, T1_HI + pcmpgtb T1_LO, T0_LO + pcmpgtb T1_HI, T0_HI + pand T0_LO, MSG_LO + pand T0_HI, MSG_HI + + call __morus1280_update + + /* store the state: */ + movdqu STATE0_LO, (0 * 16)(%rdi) + movdqu STATE0_HI, (1 * 16)(%rdi) + movdqu STATE1_LO, (2 * 16)(%rdi) + movdqu STATE1_HI, (3 * 16)(%rdi) + movdqu STATE2_LO, (4 * 16)(%rdi) + movdqu STATE2_HI, (5 * 16)(%rdi) + movdqu STATE3_LO, (6 * 16)(%rdi) + movdqu STATE3_HI, (7 * 16)(%rdi) + movdqu STATE4_LO, (8 * 16)(%rdi) + movdqu STATE4_HI, (9 * 16)(%rdi) + + FRAME_END + ret +ENDPROC(crypto_morus1280_sse2_dec_tail) + +/* + * void crypto_morus1280_sse2_final(void *state, void *tag_xor, + * u64 assoclen, u64 cryptlen); + */ +ENTRY(crypto_morus1280_sse2_final) + FRAME_BEGIN + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0_LO + movdqu (1 * 16)(%rdi), STATE0_HI + movdqu (2 * 16)(%rdi), STATE1_LO + movdqu (3 * 16)(%rdi), STATE1_HI + movdqu (4 * 16)(%rdi), STATE2_LO + movdqu (5 * 16)(%rdi), STATE2_HI + movdqu (6 * 16)(%rdi), STATE3_LO + movdqu (7 * 16)(%rdi), STATE3_HI + movdqu (8 * 16)(%rdi), STATE4_LO + movdqu (9 * 16)(%rdi), STATE4_HI + + /* xor state[0] into state[4]: */ + pxor STATE0_LO, STATE4_LO + pxor STATE0_HI, STATE4_HI + + /* prepare length block: */ + movq %rdx, MSG_LO + movq %rcx, T0_LO + pslldq $8, T0_LO + pxor T0_LO, MSG_LO + psllq $3, MSG_LO /* multiply by 8 (to get bit count) */ + pxor MSG_HI, MSG_HI + + /* update state: */ + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + call __morus1280_update + + /* xor tag: */ + movdqu 0(%rsi), MSG_LO + movdqu 16(%rsi), MSG_HI + + pxor STATE0_LO, MSG_LO + pxor STATE0_HI, MSG_HI + movdqa STATE1_LO, T0_LO + movdqa STATE1_HI, T0_HI + rol3 T0_HI, T0_LO + pxor T0_LO, MSG_LO + pxor T0_HI, MSG_HI + movdqa STATE2_LO, T0_LO + movdqa STATE2_HI, T0_HI + pand STATE3_LO, T0_LO + pand STATE3_HI, T0_HI + pxor T0_LO, MSG_LO + pxor T0_HI, MSG_HI + + movdqu MSG_LO, 0(%rsi) + movdqu MSG_HI, 16(%rsi) + + FRAME_END + ret +ENDPROC(crypto_morus1280_sse2_final) diff --git a/arch/x86/crypto/morus1280-sse2-glue.c b/arch/x86/crypto/morus1280-sse2-glue.c new file mode 100644 index 000000000000..839270aa713c --- /dev/null +++ b/arch/x86/crypto/morus1280-sse2-glue.c @@ -0,0 +1,68 @@ +/* + * The MORUS-1280 Authenticated-Encryption Algorithm + * Glue for SSE2 implementation + * + * Copyright (c) 2016-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include + +asmlinkage void crypto_morus1280_sse2_init(void *state, const void *key, + const void *iv); +asmlinkage void crypto_morus1280_sse2_ad(void *state, const void *data, + unsigned int length); + +asmlinkage void crypto_morus1280_sse2_enc(void *state, const void *src, + void *dst, unsigned int length); +asmlinkage void crypto_morus1280_sse2_dec(void *state, const void *src, + void *dst, unsigned int length); + +asmlinkage void crypto_morus1280_sse2_enc_tail(void *state, const void *src, + void *dst, unsigned int length); +asmlinkage void crypto_morus1280_sse2_dec_tail(void *state, const void *src, + void *dst, unsigned int length); + +asmlinkage void crypto_morus1280_sse2_final(void *state, void *tag_xor, + u64 assoclen, u64 cryptlen); + +MORUS1280_DECLARE_ALGS(sse2, "morus1280-sse2", 350); + +static const struct x86_cpu_id sse2_cpu_id[] = { + X86_FEATURE_MATCH(X86_FEATURE_XMM2), + {} +}; +MODULE_DEVICE_TABLE(x86cpu, sse2_cpu_id); + +static int __init crypto_morus1280_sse2_module_init(void) +{ + if (!x86_match_cpu(sse2_cpu_id)) + return -ENODEV; + + return crypto_register_aeads(crypto_morus1280_sse2_algs, + ARRAY_SIZE(crypto_morus1280_sse2_algs)); +} + +static void __exit crypto_morus1280_sse2_module_exit(void) +{ + crypto_unregister_aeads(crypto_morus1280_sse2_algs, + ARRAY_SIZE(crypto_morus1280_sse2_algs)); +} + +module_init(crypto_morus1280_sse2_module_init); +module_exit(crypto_morus1280_sse2_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("MORUS-1280 AEAD algorithm -- SSE2 implementation"); +MODULE_ALIAS_CRYPTO("morus1280"); +MODULE_ALIAS_CRYPTO("morus1280-sse2"); diff --git a/arch/x86/crypto/morus1280_glue.c b/arch/x86/crypto/morus1280_glue.c new file mode 100644 index 000000000000..0dccdda1eb3a --- /dev/null +++ b/arch/x86/crypto/morus1280_glue.c @@ -0,0 +1,302 @@ +/* + * The MORUS-1280 Authenticated-Encryption Algorithm + * Common x86 SIMD glue skeleton + * + * Copyright (c) 2016-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +struct morus1280_state { + struct morus1280_block s[MORUS_STATE_BLOCKS]; +}; + +struct morus1280_ops { + int (*skcipher_walk_init)(struct skcipher_walk *walk, + struct aead_request *req, bool atomic); + + void (*crypt_blocks)(void *state, const void *src, void *dst, + unsigned int length); + void (*crypt_tail)(void *state, const void *src, void *dst, + unsigned int length); +}; + +static void crypto_morus1280_glue_process_ad( + struct morus1280_state *state, + const struct morus1280_glue_ops *ops, + struct scatterlist *sg_src, unsigned int assoclen) +{ + struct scatter_walk walk; + struct morus1280_block buf; + unsigned int pos = 0; + + scatterwalk_start(&walk, sg_src); + while (assoclen != 0) { + unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int left = size; + void *mapped = scatterwalk_map(&walk); + const u8 *src = (const u8 *)mapped; + + if (pos + size >= MORUS1280_BLOCK_SIZE) { + if (pos > 0) { + unsigned int fill = MORUS1280_BLOCK_SIZE - pos; + memcpy(buf.bytes + pos, src, fill); + ops->ad(state, buf.bytes, MORUS1280_BLOCK_SIZE); + pos = 0; + left -= fill; + src += fill; + } + + ops->ad(state, src, left); + src += left & ~(MORUS1280_BLOCK_SIZE - 1); + left &= MORUS1280_BLOCK_SIZE - 1; + } + + memcpy(buf.bytes + pos, src, left); + + pos += left; + assoclen -= size; + scatterwalk_unmap(mapped); + scatterwalk_advance(&walk, size); + scatterwalk_done(&walk, 0, assoclen); + } + + if (pos > 0) { + memset(buf.bytes + pos, 0, MORUS1280_BLOCK_SIZE - pos); + ops->ad(state, buf.bytes, MORUS1280_BLOCK_SIZE); + } +} + +static void crypto_morus1280_glue_process_crypt(struct morus1280_state *state, + struct morus1280_ops ops, + struct aead_request *req) +{ + struct skcipher_walk walk; + u8 *cursor_src, *cursor_dst; + unsigned int chunksize, base; + + ops.skcipher_walk_init(&walk, req, false); + + while (walk.nbytes) { + cursor_src = walk.src.virt.addr; + cursor_dst = walk.dst.virt.addr; + chunksize = walk.nbytes; + + ops.crypt_blocks(state, cursor_src, cursor_dst, chunksize); + + base = chunksize & ~(MORUS1280_BLOCK_SIZE - 1); + cursor_src += base; + cursor_dst += base; + chunksize &= MORUS1280_BLOCK_SIZE - 1; + + if (chunksize > 0) + ops.crypt_tail(state, cursor_src, cursor_dst, + chunksize); + + skcipher_walk_done(&walk, 0); + } +} + +int crypto_morus1280_glue_setkey(struct crypto_aead *aead, const u8 *key, + unsigned int keylen) +{ + struct morus1280_ctx *ctx = crypto_aead_ctx(aead); + + if (keylen == MORUS1280_BLOCK_SIZE) { + memcpy(ctx->key.bytes, key, MORUS1280_BLOCK_SIZE); + } else if (keylen == MORUS1280_BLOCK_SIZE / 2) { + memcpy(ctx->key.bytes, key, keylen); + memcpy(ctx->key.bytes + keylen, key, keylen); + } else { + crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; + } + + return 0; +} +EXPORT_SYMBOL_GPL(crypto_morus1280_glue_setkey); + +int crypto_morus1280_glue_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + return (authsize <= MORUS_MAX_AUTH_SIZE) ? 0 : -EINVAL; +} +EXPORT_SYMBOL_GPL(crypto_morus1280_glue_setauthsize); + +static void crypto_morus1280_glue_crypt(struct aead_request *req, + struct morus1280_ops ops, + unsigned int cryptlen, + struct morus1280_block *tag_xor) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct morus1280_ctx *ctx = crypto_aead_ctx(tfm); + struct morus1280_state state; + + kernel_fpu_begin(); + + ctx->ops->init(&state, &ctx->key, req->iv); + crypto_morus1280_glue_process_ad(&state, ctx->ops, req->src, req->assoclen); + crypto_morus1280_glue_process_crypt(&state, ops, req); + ctx->ops->final(&state, tag_xor, req->assoclen, cryptlen); + + kernel_fpu_end(); +} + +int crypto_morus1280_glue_encrypt(struct aead_request *req) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct morus1280_ctx *ctx = crypto_aead_ctx(tfm); + struct morus1280_ops OPS = { + .skcipher_walk_init = skcipher_walk_aead_encrypt, + .crypt_blocks = ctx->ops->enc, + .crypt_tail = ctx->ops->enc_tail, + }; + + struct morus1280_block tag = {}; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen; + + crypto_morus1280_glue_crypt(req, OPS, cryptlen, &tag); + + scatterwalk_map_and_copy(tag.bytes, req->dst, + req->assoclen + cryptlen, authsize, 1); + return 0; +} +EXPORT_SYMBOL_GPL(crypto_morus1280_glue_encrypt); + +int crypto_morus1280_glue_decrypt(struct aead_request *req) +{ + static const u8 zeros[MORUS1280_BLOCK_SIZE] = {}; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct morus1280_ctx *ctx = crypto_aead_ctx(tfm); + struct morus1280_ops OPS = { + .skcipher_walk_init = skcipher_walk_aead_decrypt, + .crypt_blocks = ctx->ops->dec, + .crypt_tail = ctx->ops->dec_tail, + }; + + struct morus1280_block tag; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen - authsize; + + scatterwalk_map_and_copy(tag.bytes, req->src, + req->assoclen + cryptlen, authsize, 0); + + crypto_morus1280_glue_crypt(req, OPS, cryptlen, &tag); + + return crypto_memneq(tag.bytes, zeros, authsize) ? -EBADMSG : 0; +} +EXPORT_SYMBOL_GPL(crypto_morus1280_glue_decrypt); + +void crypto_morus1280_glue_init_ops(struct crypto_aead *aead, + const struct morus1280_glue_ops *ops) +{ + struct morus1280_ctx *ctx = crypto_aead_ctx(aead); + ctx->ops = ops; +} +EXPORT_SYMBOL_GPL(crypto_morus1280_glue_init_ops); + +int cryptd_morus1280_glue_setkey(struct crypto_aead *aead, const u8 *key, + unsigned int keylen) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + return crypto_aead_setkey(&cryptd_tfm->base, key, keylen); +} +EXPORT_SYMBOL_GPL(cryptd_morus1280_glue_setkey); + +int cryptd_morus1280_glue_setauthsize(struct crypto_aead *aead, + unsigned int authsize) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + return crypto_aead_setauthsize(&cryptd_tfm->base, authsize); +} +EXPORT_SYMBOL_GPL(cryptd_morus1280_glue_setauthsize); + +int cryptd_morus1280_glue_encrypt(struct aead_request *req) +{ + struct crypto_aead *aead = crypto_aead_reqtfm(req); + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + aead = &cryptd_tfm->base; + if (irq_fpu_usable() && (!in_atomic() || + !cryptd_aead_queued(cryptd_tfm))) + aead = cryptd_aead_child(cryptd_tfm); + + aead_request_set_tfm(req, aead); + + return crypto_aead_encrypt(req); +} +EXPORT_SYMBOL_GPL(cryptd_morus1280_glue_encrypt); + +int cryptd_morus1280_glue_decrypt(struct aead_request *req) +{ + struct crypto_aead *aead = crypto_aead_reqtfm(req); + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + aead = &cryptd_tfm->base; + if (irq_fpu_usable() && (!in_atomic() || + !cryptd_aead_queued(cryptd_tfm))) + aead = cryptd_aead_child(cryptd_tfm); + + aead_request_set_tfm(req, aead); + + return crypto_aead_decrypt(req); +} +EXPORT_SYMBOL_GPL(cryptd_morus1280_glue_decrypt); + +int cryptd_morus1280_glue_init_tfm(struct crypto_aead *aead) +{ + struct cryptd_aead *cryptd_tfm; + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + const char *name = crypto_aead_alg(aead)->base.cra_driver_name; + char internal_name[CRYPTO_MAX_ALG_NAME]; + + if (snprintf(internal_name, CRYPTO_MAX_ALG_NAME, "__%s", name) + >= CRYPTO_MAX_ALG_NAME) + return -ENAMETOOLONG; + + cryptd_tfm = cryptd_alloc_aead(internal_name, CRYPTO_ALG_INTERNAL, + CRYPTO_ALG_INTERNAL); + if (IS_ERR(cryptd_tfm)) + return PTR_ERR(cryptd_tfm); + + *ctx = cryptd_tfm; + crypto_aead_set_reqsize(aead, crypto_aead_reqsize(&cryptd_tfm->base)); + return 0; +} +EXPORT_SYMBOL_GPL(cryptd_morus1280_glue_init_tfm); + +void cryptd_morus1280_glue_exit_tfm(struct crypto_aead *aead) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + + cryptd_free_aead(*ctx); +} +EXPORT_SYMBOL_GPL(cryptd_morus1280_glue_exit_tfm); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("MORUS-1280 AEAD mode -- glue for x86 optimizations"); diff --git a/arch/x86/crypto/morus640-sse2-asm.S b/arch/x86/crypto/morus640-sse2-asm.S new file mode 100644 index 000000000000..71c72a0a0862 --- /dev/null +++ b/arch/x86/crypto/morus640-sse2-asm.S @@ -0,0 +1,614 @@ +/* + * SSE2 implementation of MORUS-640 + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation. + */ + +#include +#include + +#define SHUFFLE_MASK(i0, i1, i2, i3) \ + (i0 | (i1 << 2) | (i2 << 4) | (i3 << 6)) + +#define MASK1 SHUFFLE_MASK(3, 0, 1, 2) +#define MASK2 SHUFFLE_MASK(2, 3, 0, 1) +#define MASK3 SHUFFLE_MASK(1, 2, 3, 0) + +#define STATE0 %xmm0 +#define STATE1 %xmm1 +#define STATE2 %xmm2 +#define STATE3 %xmm3 +#define STATE4 %xmm4 +#define KEY %xmm5 +#define MSG %xmm5 +#define T0 %xmm6 +#define T1 %xmm7 + +.section .rodata.cst16.morus640_const, "aM", @progbits, 32 +.align 16 +.Lmorus640_const_0: + .byte 0x00, 0x01, 0x01, 0x02, 0x03, 0x05, 0x08, 0x0d + .byte 0x15, 0x22, 0x37, 0x59, 0x90, 0xe9, 0x79, 0x62 +.Lmorus640_const_1: + .byte 0xdb, 0x3d, 0x18, 0x55, 0x6d, 0xc2, 0x2f, 0xf1 + .byte 0x20, 0x11, 0x31, 0x42, 0x73, 0xb5, 0x28, 0xdd + +.section .rodata.cst16.morus640_counter, "aM", @progbits, 16 +.align 16 +.Lmorus640_counter: + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f + +.text + +.macro morus640_round s0, s1, s2, s3, s4, b, w + movdqa \s1, T0 + pand \s2, T0 + pxor T0, \s0 + pxor \s3, \s0 + movdqa \s0, T0 + pslld $\b, T0 + psrld $(32 - \b), \s0 + pxor T0, \s0 + pshufd $\w, \s3, \s3 +.endm + +/* + * __morus640_update: internal ABI + * input: + * STATE[0-4] - input state + * MSG - message block + * output: + * STATE[0-4] - output state + * changed: + * T0 + */ +__morus640_update: + morus640_round STATE0, STATE1, STATE2, STATE3, STATE4, 5, MASK1 + pxor MSG, STATE1 + morus640_round STATE1, STATE2, STATE3, STATE4, STATE0, 31, MASK2 + pxor MSG, STATE2 + morus640_round STATE2, STATE3, STATE4, STATE0, STATE1, 7, MASK3 + pxor MSG, STATE3 + morus640_round STATE3, STATE4, STATE0, STATE1, STATE2, 22, MASK2 + pxor MSG, STATE4 + morus640_round STATE4, STATE0, STATE1, STATE2, STATE3, 13, MASK1 + ret +ENDPROC(__morus640_update) + + +/* + * __morus640_update_zero: internal ABI + * input: + * STATE[0-4] - input state + * output: + * STATE[0-4] - output state + * changed: + * T0 + */ +__morus640_update_zero: + morus640_round STATE0, STATE1, STATE2, STATE3, STATE4, 5, MASK1 + morus640_round STATE1, STATE2, STATE3, STATE4, STATE0, 31, MASK2 + morus640_round STATE2, STATE3, STATE4, STATE0, STATE1, 7, MASK3 + morus640_round STATE3, STATE4, STATE0, STATE1, STATE2, 22, MASK2 + morus640_round STATE4, STATE0, STATE1, STATE2, STATE3, 13, MASK1 + ret +ENDPROC(__morus640_update_zero) + +/* + * __load_partial: internal ABI + * input: + * %rsi - src + * %rcx - bytes + * output: + * MSG - message block + * changed: + * T0 + * %r8 + * %r9 + */ +__load_partial: + xor %r9, %r9 + pxor MSG, MSG + + mov %rcx, %r8 + and $0x1, %r8 + jz .Lld_partial_1 + + mov %rcx, %r8 + and $0x1E, %r8 + add %rsi, %r8 + mov (%r8), %r9b + +.Lld_partial_1: + mov %rcx, %r8 + and $0x2, %r8 + jz .Lld_partial_2 + + mov %rcx, %r8 + and $0x1C, %r8 + add %rsi, %r8 + shl $16, %r9 + mov (%r8), %r9w + +.Lld_partial_2: + mov %rcx, %r8 + and $0x4, %r8 + jz .Lld_partial_4 + + mov %rcx, %r8 + and $0x18, %r8 + add %rsi, %r8 + shl $32, %r9 + mov (%r8), %r8d + xor %r8, %r9 + +.Lld_partial_4: + movq %r9, MSG + + mov %rcx, %r8 + and $0x8, %r8 + jz .Lld_partial_8 + + mov %rcx, %r8 + and $0x10, %r8 + add %rsi, %r8 + pslldq $8, MSG + movq (%r8), T0 + pxor T0, MSG + +.Lld_partial_8: + ret +ENDPROC(__load_partial) + +/* + * __store_partial: internal ABI + * input: + * %rdx - dst + * %rcx - bytes + * output: + * T0 - message block + * changed: + * %r8 + * %r9 + * %r10 + */ +__store_partial: + mov %rcx, %r8 + mov %rdx, %r9 + + movq T0, %r10 + + cmp $8, %r8 + jl .Lst_partial_8 + + mov %r10, (%r9) + psrldq $8, T0 + movq T0, %r10 + + sub $8, %r8 + add $8, %r9 + +.Lst_partial_8: + cmp $4, %r8 + jl .Lst_partial_4 + + mov %r10d, (%r9) + shr $32, %r10 + + sub $4, %r8 + add $4, %r9 + +.Lst_partial_4: + cmp $2, %r8 + jl .Lst_partial_2 + + mov %r10w, (%r9) + shr $16, %r10 + + sub $2, %r8 + add $2, %r9 + +.Lst_partial_2: + cmp $1, %r8 + jl .Lst_partial_1 + + mov %r10b, (%r9) + +.Lst_partial_1: + ret +ENDPROC(__store_partial) + +/* + * void crypto_morus640_sse2_init(void *state, const void *key, const void *iv); + */ +ENTRY(crypto_morus640_sse2_init) + FRAME_BEGIN + + /* load IV: */ + movdqu (%rdx), STATE0 + /* load key: */ + movdqu (%rsi), KEY + movdqa KEY, STATE1 + /* load all ones: */ + pcmpeqd STATE2, STATE2 + /* load the constants: */ + movdqa .Lmorus640_const_0, STATE3 + movdqa .Lmorus640_const_1, STATE4 + + /* update 16 times with zero: */ + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + call __morus640_update_zero + + /* xor-in the key again after updates: */ + pxor KEY, STATE1 + + /* store the state: */ + movdqu STATE0, (0 * 16)(%rdi) + movdqu STATE1, (1 * 16)(%rdi) + movdqu STATE2, (2 * 16)(%rdi) + movdqu STATE3, (3 * 16)(%rdi) + movdqu STATE4, (4 * 16)(%rdi) + + FRAME_END + ret +ENDPROC(crypto_morus640_sse2_init) + +/* + * void crypto_morus640_sse2_ad(void *state, const void *data, + * unsigned int length); + */ +ENTRY(crypto_morus640_sse2_ad) + FRAME_BEGIN + + cmp $16, %rdx + jb .Lad_out + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0 + movdqu (1 * 16)(%rdi), STATE1 + movdqu (2 * 16)(%rdi), STATE2 + movdqu (3 * 16)(%rdi), STATE3 + movdqu (4 * 16)(%rdi), STATE4 + + mov %rsi, %r8 + and $0xF, %r8 + jnz .Lad_u_loop + +.align 4 +.Lad_a_loop: + movdqa (%rsi), MSG + call __morus640_update + sub $16, %rdx + add $16, %rsi + cmp $16, %rdx + jge .Lad_a_loop + + jmp .Lad_cont +.align 4 +.Lad_u_loop: + movdqu (%rsi), MSG + call __morus640_update + sub $16, %rdx + add $16, %rsi + cmp $16, %rdx + jge .Lad_u_loop + +.Lad_cont: + /* store the state: */ + movdqu STATE0, (0 * 16)(%rdi) + movdqu STATE1, (1 * 16)(%rdi) + movdqu STATE2, (2 * 16)(%rdi) + movdqu STATE3, (3 * 16)(%rdi) + movdqu STATE4, (4 * 16)(%rdi) + +.Lad_out: + FRAME_END + ret +ENDPROC(crypto_morus640_sse2_ad) + +/* + * void crypto_morus640_sse2_enc(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus640_sse2_enc) + FRAME_BEGIN + + cmp $16, %rcx + jb .Lenc_out + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0 + movdqu (1 * 16)(%rdi), STATE1 + movdqu (2 * 16)(%rdi), STATE2 + movdqu (3 * 16)(%rdi), STATE3 + movdqu (4 * 16)(%rdi), STATE4 + + mov %rsi, %r8 + or %rdx, %r8 + and $0xF, %r8 + jnz .Lenc_u_loop + +.align 4 +.Lenc_a_loop: + movdqa (%rsi), MSG + movdqa MSG, T0 + pxor STATE0, T0 + pshufd $MASK3, STATE1, T1 + pxor T1, T0 + movdqa STATE2, T1 + pand STATE3, T1 + pxor T1, T0 + movdqa T0, (%rdx) + + call __morus640_update + sub $16, %rcx + add $16, %rsi + add $16, %rdx + cmp $16, %rcx + jge .Lenc_a_loop + + jmp .Lenc_cont +.align 4 +.Lenc_u_loop: + movdqu (%rsi), MSG + movdqa MSG, T0 + pxor STATE0, T0 + pshufd $MASK3, STATE1, T1 + pxor T1, T0 + movdqa STATE2, T1 + pand STATE3, T1 + pxor T1, T0 + movdqu T0, (%rdx) + + call __morus640_update + sub $16, %rcx + add $16, %rsi + add $16, %rdx + cmp $16, %rcx + jge .Lenc_u_loop + +.Lenc_cont: + /* store the state: */ + movdqu STATE0, (0 * 16)(%rdi) + movdqu STATE1, (1 * 16)(%rdi) + movdqu STATE2, (2 * 16)(%rdi) + movdqu STATE3, (3 * 16)(%rdi) + movdqu STATE4, (4 * 16)(%rdi) + +.Lenc_out: + FRAME_END + ret +ENDPROC(crypto_morus640_sse2_enc) + +/* + * void crypto_morus640_sse2_enc_tail(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus640_sse2_enc_tail) + FRAME_BEGIN + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0 + movdqu (1 * 16)(%rdi), STATE1 + movdqu (2 * 16)(%rdi), STATE2 + movdqu (3 * 16)(%rdi), STATE3 + movdqu (4 * 16)(%rdi), STATE4 + + /* encrypt message: */ + call __load_partial + + movdqa MSG, T0 + pxor STATE0, T0 + pshufd $MASK3, STATE1, T1 + pxor T1, T0 + movdqa STATE2, T1 + pand STATE3, T1 + pxor T1, T0 + + call __store_partial + + call __morus640_update + + /* store the state: */ + movdqu STATE0, (0 * 16)(%rdi) + movdqu STATE1, (1 * 16)(%rdi) + movdqu STATE2, (2 * 16)(%rdi) + movdqu STATE3, (3 * 16)(%rdi) + movdqu STATE4, (4 * 16)(%rdi) + + FRAME_END +ENDPROC(crypto_morus640_sse2_enc_tail) + +/* + * void crypto_morus640_sse2_dec(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus640_sse2_dec) + FRAME_BEGIN + + cmp $16, %rcx + jb .Ldec_out + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0 + movdqu (1 * 16)(%rdi), STATE1 + movdqu (2 * 16)(%rdi), STATE2 + movdqu (3 * 16)(%rdi), STATE3 + movdqu (4 * 16)(%rdi), STATE4 + + mov %rsi, %r8 + or %rdx, %r8 + and $0xF, %r8 + jnz .Ldec_u_loop + +.align 4 +.Ldec_a_loop: + movdqa (%rsi), MSG + pxor STATE0, MSG + pshufd $MASK3, STATE1, T0 + pxor T0, MSG + movdqa STATE2, T0 + pand STATE3, T0 + pxor T0, MSG + movdqa MSG, (%rdx) + + call __morus640_update + sub $16, %rcx + add $16, %rsi + add $16, %rdx + cmp $16, %rcx + jge .Ldec_a_loop + + jmp .Ldec_cont +.align 4 +.Ldec_u_loop: + movdqu (%rsi), MSG + pxor STATE0, MSG + pshufd $MASK3, STATE1, T0 + pxor T0, MSG + movdqa STATE2, T0 + pand STATE3, T0 + pxor T0, MSG + movdqu MSG, (%rdx) + + call __morus640_update + sub $16, %rcx + add $16, %rsi + add $16, %rdx + cmp $16, %rcx + jge .Ldec_u_loop + +.Ldec_cont: + /* store the state: */ + movdqu STATE0, (0 * 16)(%rdi) + movdqu STATE1, (1 * 16)(%rdi) + movdqu STATE2, (2 * 16)(%rdi) + movdqu STATE3, (3 * 16)(%rdi) + movdqu STATE4, (4 * 16)(%rdi) + +.Ldec_out: + FRAME_END + ret +ENDPROC(crypto_morus640_sse2_dec) + +/* + * void crypto_morus640_sse2_dec_tail(void *state, const void *src, void *dst, + * unsigned int length); + */ +ENTRY(crypto_morus640_sse2_dec_tail) + FRAME_BEGIN + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0 + movdqu (1 * 16)(%rdi), STATE1 + movdqu (2 * 16)(%rdi), STATE2 + movdqu (3 * 16)(%rdi), STATE3 + movdqu (4 * 16)(%rdi), STATE4 + + /* decrypt message: */ + call __load_partial + + pxor STATE0, MSG + pshufd $MASK3, STATE1, T0 + pxor T0, MSG + movdqa STATE2, T0 + pand STATE3, T0 + pxor T0, MSG + movdqa MSG, T0 + + call __store_partial + + /* mask with byte count: */ + movq %rcx, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + punpcklbw T0, T0 + movdqa .Lmorus640_counter, T1 + pcmpgtb T1, T0 + pand T0, MSG + + call __morus640_update + + /* store the state: */ + movdqu STATE0, (0 * 16)(%rdi) + movdqu STATE1, (1 * 16)(%rdi) + movdqu STATE2, (2 * 16)(%rdi) + movdqu STATE3, (3 * 16)(%rdi) + movdqu STATE4, (4 * 16)(%rdi) + + FRAME_END + ret +ENDPROC(crypto_morus640_sse2_dec_tail) + +/* + * void crypto_morus640_sse2_final(void *state, void *tag_xor, + * u64 assoclen, u64 cryptlen); + */ +ENTRY(crypto_morus640_sse2_final) + FRAME_BEGIN + + /* load the state: */ + movdqu (0 * 16)(%rdi), STATE0 + movdqu (1 * 16)(%rdi), STATE1 + movdqu (2 * 16)(%rdi), STATE2 + movdqu (3 * 16)(%rdi), STATE3 + movdqu (4 * 16)(%rdi), STATE4 + + /* xor state[0] into state[4]: */ + pxor STATE0, STATE4 + + /* prepare length block: */ + movq %rdx, MSG + movq %rcx, T0 + pslldq $8, T0 + pxor T0, MSG + psllq $3, MSG /* multiply by 8 (to get bit count) */ + + /* update state: */ + call __morus640_update + call __morus640_update + call __morus640_update + call __morus640_update + call __morus640_update + call __morus640_update + call __morus640_update + call __morus640_update + call __morus640_update + call __morus640_update + + /* xor tag: */ + movdqu (%rsi), MSG + + pxor STATE0, MSG + pshufd $MASK3, STATE1, T0 + pxor T0, MSG + movdqa STATE2, T0 + pand STATE3, T0 + pxor T0, MSG + + movdqu MSG, (%rsi) + + FRAME_END + ret +ENDPROC(crypto_morus640_sse2_final) diff --git a/arch/x86/crypto/morus640-sse2-glue.c b/arch/x86/crypto/morus640-sse2-glue.c new file mode 100644 index 000000000000..26b47e2db8d2 --- /dev/null +++ b/arch/x86/crypto/morus640-sse2-glue.c @@ -0,0 +1,68 @@ +/* + * The MORUS-640 Authenticated-Encryption Algorithm + * Glue for SSE2 implementation + * + * Copyright (c) 2016-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include + +asmlinkage void crypto_morus640_sse2_init(void *state, const void *key, + const void *iv); +asmlinkage void crypto_morus640_sse2_ad(void *state, const void *data, + unsigned int length); + +asmlinkage void crypto_morus640_sse2_enc(void *state, const void *src, + void *dst, unsigned int length); +asmlinkage void crypto_morus640_sse2_dec(void *state, const void *src, + void *dst, unsigned int length); + +asmlinkage void crypto_morus640_sse2_enc_tail(void *state, const void *src, + void *dst, unsigned int length); +asmlinkage void crypto_morus640_sse2_dec_tail(void *state, const void *src, + void *dst, unsigned int length); + +asmlinkage void crypto_morus640_sse2_final(void *state, void *tag_xor, + u64 assoclen, u64 cryptlen); + +MORUS640_DECLARE_ALGS(sse2, "morus640-sse2", 400); + +static const struct x86_cpu_id sse2_cpu_id[] = { + X86_FEATURE_MATCH(X86_FEATURE_XMM2), + {} +}; +MODULE_DEVICE_TABLE(x86cpu, sse2_cpu_id); + +static int __init crypto_morus640_sse2_module_init(void) +{ + if (!x86_match_cpu(sse2_cpu_id)) + return -ENODEV; + + return crypto_register_aeads(crypto_morus640_sse2_algs, + ARRAY_SIZE(crypto_morus640_sse2_algs)); +} + +static void __exit crypto_morus640_sse2_module_exit(void) +{ + crypto_unregister_aeads(crypto_morus640_sse2_algs, + ARRAY_SIZE(crypto_morus640_sse2_algs)); +} + +module_init(crypto_morus640_sse2_module_init); +module_exit(crypto_morus640_sse2_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("MORUS-640 AEAD algorithm -- SSE2 implementation"); +MODULE_ALIAS_CRYPTO("morus640"); +MODULE_ALIAS_CRYPTO("morus640-sse2"); diff --git a/arch/x86/crypto/morus640_glue.c b/arch/x86/crypto/morus640_glue.c new file mode 100644 index 000000000000..7b58fe4d9bd1 --- /dev/null +++ b/arch/x86/crypto/morus640_glue.c @@ -0,0 +1,298 @@ +/* + * The MORUS-640 Authenticated-Encryption Algorithm + * Common x86 SIMD glue skeleton + * + * Copyright (c) 2016-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +struct morus640_state { + struct morus640_block s[MORUS_STATE_BLOCKS]; +}; + +struct morus640_ops { + int (*skcipher_walk_init)(struct skcipher_walk *walk, + struct aead_request *req, bool atomic); + + void (*crypt_blocks)(void *state, const void *src, void *dst, + unsigned int length); + void (*crypt_tail)(void *state, const void *src, void *dst, + unsigned int length); +}; + +static void crypto_morus640_glue_process_ad( + struct morus640_state *state, + const struct morus640_glue_ops *ops, + struct scatterlist *sg_src, unsigned int assoclen) +{ + struct scatter_walk walk; + struct morus640_block buf; + unsigned int pos = 0; + + scatterwalk_start(&walk, sg_src); + while (assoclen != 0) { + unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int left = size; + void *mapped = scatterwalk_map(&walk); + const u8 *src = (const u8 *)mapped; + + if (pos + size >= MORUS640_BLOCK_SIZE) { + if (pos > 0) { + unsigned int fill = MORUS640_BLOCK_SIZE - pos; + memcpy(buf.bytes + pos, src, fill); + ops->ad(state, buf.bytes, MORUS640_BLOCK_SIZE); + pos = 0; + left -= fill; + src += fill; + } + + ops->ad(state, src, left); + src += left & ~(MORUS640_BLOCK_SIZE - 1); + left &= MORUS640_BLOCK_SIZE - 1; + } + + memcpy(buf.bytes + pos, src, left); + + pos += left; + assoclen -= size; + scatterwalk_unmap(mapped); + scatterwalk_advance(&walk, size); + scatterwalk_done(&walk, 0, assoclen); + } + + if (pos > 0) { + memset(buf.bytes + pos, 0, MORUS640_BLOCK_SIZE - pos); + ops->ad(state, buf.bytes, MORUS640_BLOCK_SIZE); + } +} + +static void crypto_morus640_glue_process_crypt(struct morus640_state *state, + struct morus640_ops ops, + struct aead_request *req) +{ + struct skcipher_walk walk; + u8 *cursor_src, *cursor_dst; + unsigned int chunksize, base; + + ops.skcipher_walk_init(&walk, req, false); + + while (walk.nbytes) { + cursor_src = walk.src.virt.addr; + cursor_dst = walk.dst.virt.addr; + chunksize = walk.nbytes; + + ops.crypt_blocks(state, cursor_src, cursor_dst, chunksize); + + base = chunksize & ~(MORUS640_BLOCK_SIZE - 1); + cursor_src += base; + cursor_dst += base; + chunksize &= MORUS640_BLOCK_SIZE - 1; + + if (chunksize > 0) + ops.crypt_tail(state, cursor_src, cursor_dst, + chunksize); + + skcipher_walk_done(&walk, 0); + } +} + +int crypto_morus640_glue_setkey(struct crypto_aead *aead, const u8 *key, + unsigned int keylen) +{ + struct morus640_ctx *ctx = crypto_aead_ctx(aead); + + if (keylen != MORUS640_BLOCK_SIZE) { + crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; + } + + memcpy(ctx->key.bytes, key, MORUS640_BLOCK_SIZE); + return 0; +} +EXPORT_SYMBOL_GPL(crypto_morus640_glue_setkey); + +int crypto_morus640_glue_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + return (authsize <= MORUS_MAX_AUTH_SIZE) ? 0 : -EINVAL; +} +EXPORT_SYMBOL_GPL(crypto_morus640_glue_setauthsize); + +static void crypto_morus640_glue_crypt(struct aead_request *req, + struct morus640_ops ops, + unsigned int cryptlen, + struct morus640_block *tag_xor) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct morus640_ctx *ctx = crypto_aead_ctx(tfm); + struct morus640_state state; + + kernel_fpu_begin(); + + ctx->ops->init(&state, &ctx->key, req->iv); + crypto_morus640_glue_process_ad(&state, ctx->ops, req->src, req->assoclen); + crypto_morus640_glue_process_crypt(&state, ops, req); + ctx->ops->final(&state, tag_xor, req->assoclen, cryptlen); + + kernel_fpu_end(); +} + +int crypto_morus640_glue_encrypt(struct aead_request *req) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct morus640_ctx *ctx = crypto_aead_ctx(tfm); + struct morus640_ops OPS = { + .skcipher_walk_init = skcipher_walk_aead_encrypt, + .crypt_blocks = ctx->ops->enc, + .crypt_tail = ctx->ops->enc_tail, + }; + + struct morus640_block tag = {}; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen; + + crypto_morus640_glue_crypt(req, OPS, cryptlen, &tag); + + scatterwalk_map_and_copy(tag.bytes, req->dst, + req->assoclen + cryptlen, authsize, 1); + return 0; +} +EXPORT_SYMBOL_GPL(crypto_morus640_glue_encrypt); + +int crypto_morus640_glue_decrypt(struct aead_request *req) +{ + static const u8 zeros[MORUS640_BLOCK_SIZE] = {}; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct morus640_ctx *ctx = crypto_aead_ctx(tfm); + struct morus640_ops OPS = { + .skcipher_walk_init = skcipher_walk_aead_decrypt, + .crypt_blocks = ctx->ops->dec, + .crypt_tail = ctx->ops->dec_tail, + }; + + struct morus640_block tag; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen - authsize; + + scatterwalk_map_and_copy(tag.bytes, req->src, + req->assoclen + cryptlen, authsize, 0); + + crypto_morus640_glue_crypt(req, OPS, cryptlen, &tag); + + return crypto_memneq(tag.bytes, zeros, authsize) ? -EBADMSG : 0; +} +EXPORT_SYMBOL_GPL(crypto_morus640_glue_decrypt); + +void crypto_morus640_glue_init_ops(struct crypto_aead *aead, + const struct morus640_glue_ops *ops) +{ + struct morus640_ctx *ctx = crypto_aead_ctx(aead); + ctx->ops = ops; +} +EXPORT_SYMBOL_GPL(crypto_morus640_glue_init_ops); + +int cryptd_morus640_glue_setkey(struct crypto_aead *aead, const u8 *key, + unsigned int keylen) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + return crypto_aead_setkey(&cryptd_tfm->base, key, keylen); +} +EXPORT_SYMBOL_GPL(cryptd_morus640_glue_setkey); + +int cryptd_morus640_glue_setauthsize(struct crypto_aead *aead, + unsigned int authsize) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + return crypto_aead_setauthsize(&cryptd_tfm->base, authsize); +} +EXPORT_SYMBOL_GPL(cryptd_morus640_glue_setauthsize); + +int cryptd_morus640_glue_encrypt(struct aead_request *req) +{ + struct crypto_aead *aead = crypto_aead_reqtfm(req); + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + aead = &cryptd_tfm->base; + if (irq_fpu_usable() && (!in_atomic() || + !cryptd_aead_queued(cryptd_tfm))) + aead = cryptd_aead_child(cryptd_tfm); + + aead_request_set_tfm(req, aead); + + return crypto_aead_encrypt(req); +} +EXPORT_SYMBOL_GPL(cryptd_morus640_glue_encrypt); + +int cryptd_morus640_glue_decrypt(struct aead_request *req) +{ + struct crypto_aead *aead = crypto_aead_reqtfm(req); + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + struct cryptd_aead *cryptd_tfm = *ctx; + + aead = &cryptd_tfm->base; + if (irq_fpu_usable() && (!in_atomic() || + !cryptd_aead_queued(cryptd_tfm))) + aead = cryptd_aead_child(cryptd_tfm); + + aead_request_set_tfm(req, aead); + + return crypto_aead_decrypt(req); +} +EXPORT_SYMBOL_GPL(cryptd_morus640_glue_decrypt); + +int cryptd_morus640_glue_init_tfm(struct crypto_aead *aead) +{ + struct cryptd_aead *cryptd_tfm; + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + const char *name = crypto_aead_alg(aead)->base.cra_driver_name; + char internal_name[CRYPTO_MAX_ALG_NAME]; + + if (snprintf(internal_name, CRYPTO_MAX_ALG_NAME, "__%s", name) + >= CRYPTO_MAX_ALG_NAME) + return -ENAMETOOLONG; + + cryptd_tfm = cryptd_alloc_aead(internal_name, CRYPTO_ALG_INTERNAL, + CRYPTO_ALG_INTERNAL); + if (IS_ERR(cryptd_tfm)) + return PTR_ERR(cryptd_tfm); + + *ctx = cryptd_tfm; + crypto_aead_set_reqsize(aead, crypto_aead_reqsize(&cryptd_tfm->base)); + return 0; +} +EXPORT_SYMBOL_GPL(cryptd_morus640_glue_init_tfm); + +void cryptd_morus640_glue_exit_tfm(struct crypto_aead *aead) +{ + struct cryptd_aead **ctx = crypto_aead_ctx(aead); + + cryptd_free_aead(*ctx); +} +EXPORT_SYMBOL_GPL(cryptd_morus640_glue_exit_tfm); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("MORUS-640 AEAD mode -- glue for x86 optimizations"); diff --git a/arch/x86/crypto/salsa20-i586-asm_32.S b/arch/x86/crypto/salsa20-i586-asm_32.S deleted file mode 100644 index 6014b7b9e52a..000000000000 --- a/arch/x86/crypto/salsa20-i586-asm_32.S +++ /dev/null @@ -1,938 +0,0 @@ -# Derived from: -# salsa20_pm.s version 20051229 -# D. J. Bernstein -# Public domain. - -#include - -.text - -# enter salsa20_encrypt_bytes -ENTRY(salsa20_encrypt_bytes) - mov %esp,%eax - and $31,%eax - add $256,%eax - sub %eax,%esp - # eax_stack = eax - movl %eax,80(%esp) - # ebx_stack = ebx - movl %ebx,84(%esp) - # esi_stack = esi - movl %esi,88(%esp) - # edi_stack = edi - movl %edi,92(%esp) - # ebp_stack = ebp - movl %ebp,96(%esp) - # x = arg1 - movl 4(%esp,%eax),%edx - # m = arg2 - movl 8(%esp,%eax),%esi - # out = arg3 - movl 12(%esp,%eax),%edi - # bytes = arg4 - movl 16(%esp,%eax),%ebx - # bytes -= 0 - sub $0,%ebx - # goto done if unsigned<= - jbe ._done -._start: - # in0 = *(uint32 *) (x + 0) - movl 0(%edx),%eax - # in1 = *(uint32 *) (x + 4) - movl 4(%edx),%ecx - # in2 = *(uint32 *) (x + 8) - movl 8(%edx),%ebp - # j0 = in0 - movl %eax,164(%esp) - # in3 = *(uint32 *) (x + 12) - movl 12(%edx),%eax - # j1 = in1 - movl %ecx,168(%esp) - # in4 = *(uint32 *) (x + 16) - movl 16(%edx),%ecx - # j2 = in2 - movl %ebp,172(%esp) - # in5 = *(uint32 *) (x + 20) - movl 20(%edx),%ebp - # j3 = in3 - movl %eax,176(%esp) - # in6 = *(uint32 *) (x + 24) - movl 24(%edx),%eax - # j4 = in4 - movl %ecx,180(%esp) - # in7 = *(uint32 *) (x + 28) - movl 28(%edx),%ecx - # j5 = in5 - movl %ebp,184(%esp) - # in8 = *(uint32 *) (x + 32) - movl 32(%edx),%ebp - # j6 = in6 - movl %eax,188(%esp) - # in9 = *(uint32 *) (x + 36) - movl 36(%edx),%eax - # j7 = in7 - movl %ecx,192(%esp) - # in10 = *(uint32 *) (x + 40) - movl 40(%edx),%ecx - # j8 = in8 - movl %ebp,196(%esp) - # in11 = *(uint32 *) (x + 44) - movl 44(%edx),%ebp - # j9 = in9 - movl %eax,200(%esp) - # in12 = *(uint32 *) (x + 48) - movl 48(%edx),%eax - # j10 = in10 - movl %ecx,204(%esp) - # in13 = *(uint32 *) (x + 52) - movl 52(%edx),%ecx - # j11 = in11 - movl %ebp,208(%esp) - # in14 = *(uint32 *) (x + 56) - movl 56(%edx),%ebp - # j12 = in12 - movl %eax,212(%esp) - # in15 = *(uint32 *) (x + 60) - movl 60(%edx),%eax - # j13 = in13 - movl %ecx,216(%esp) - # j14 = in14 - movl %ebp,220(%esp) - # j15 = in15 - movl %eax,224(%esp) - # x_backup = x - movl %edx,64(%esp) -._bytesatleast1: - # bytes - 64 - cmp $64,%ebx - # goto nocopy if unsigned>= - jae ._nocopy - # ctarget = out - movl %edi,228(%esp) - # out = &tmp - leal 0(%esp),%edi - # i = bytes - mov %ebx,%ecx - # while (i) { *out++ = *m++; --i } - rep movsb - # out = &tmp - leal 0(%esp),%edi - # m = &tmp - leal 0(%esp),%esi -._nocopy: - # out_backup = out - movl %edi,72(%esp) - # m_backup = m - movl %esi,68(%esp) - # bytes_backup = bytes - movl %ebx,76(%esp) - # in0 = j0 - movl 164(%esp),%eax - # in1 = j1 - movl 168(%esp),%ecx - # in2 = j2 - movl 172(%esp),%edx - # in3 = j3 - movl 176(%esp),%ebx - # x0 = in0 - movl %eax,100(%esp) - # x1 = in1 - movl %ecx,104(%esp) - # x2 = in2 - movl %edx,108(%esp) - # x3 = in3 - movl %ebx,112(%esp) - # in4 = j4 - movl 180(%esp),%eax - # in5 = j5 - movl 184(%esp),%ecx - # in6 = j6 - movl 188(%esp),%edx - # in7 = j7 - movl 192(%esp),%ebx - # x4 = in4 - movl %eax,116(%esp) - # x5 = in5 - movl %ecx,120(%esp) - # x6 = in6 - movl %edx,124(%esp) - # x7 = in7 - movl %ebx,128(%esp) - # in8 = j8 - movl 196(%esp),%eax - # in9 = j9 - movl 200(%esp),%ecx - # in10 = j10 - movl 204(%esp),%edx - # in11 = j11 - movl 208(%esp),%ebx - # x8 = in8 - movl %eax,132(%esp) - # x9 = in9 - movl %ecx,136(%esp) - # x10 = in10 - movl %edx,140(%esp) - # x11 = in11 - movl %ebx,144(%esp) - # in12 = j12 - movl 212(%esp),%eax - # in13 = j13 - movl 216(%esp),%ecx - # in14 = j14 - movl 220(%esp),%edx - # in15 = j15 - movl 224(%esp),%ebx - # x12 = in12 - movl %eax,148(%esp) - # x13 = in13 - movl %ecx,152(%esp) - # x14 = in14 - movl %edx,156(%esp) - # x15 = in15 - movl %ebx,160(%esp) - # i = 20 - mov $20,%ebp - # p = x0 - movl 100(%esp),%eax - # s = x5 - movl 120(%esp),%ecx - # t = x10 - movl 140(%esp),%edx - # w = x15 - movl 160(%esp),%ebx -._mainloop: - # x0 = p - movl %eax,100(%esp) - # x10 = t - movl %edx,140(%esp) - # p += x12 - addl 148(%esp),%eax - # x5 = s - movl %ecx,120(%esp) - # t += x6 - addl 124(%esp),%edx - # x15 = w - movl %ebx,160(%esp) - # r = x1 - movl 104(%esp),%esi - # r += s - add %ecx,%esi - # v = x11 - movl 144(%esp),%edi - # v += w - add %ebx,%edi - # p <<<= 7 - rol $7,%eax - # p ^= x4 - xorl 116(%esp),%eax - # t <<<= 7 - rol $7,%edx - # t ^= x14 - xorl 156(%esp),%edx - # r <<<= 7 - rol $7,%esi - # r ^= x9 - xorl 136(%esp),%esi - # v <<<= 7 - rol $7,%edi - # v ^= x3 - xorl 112(%esp),%edi - # x4 = p - movl %eax,116(%esp) - # x14 = t - movl %edx,156(%esp) - # p += x0 - addl 100(%esp),%eax - # x9 = r - movl %esi,136(%esp) - # t += x10 - addl 140(%esp),%edx - # x3 = v - movl %edi,112(%esp) - # p <<<= 9 - rol $9,%eax - # p ^= x8 - xorl 132(%esp),%eax - # t <<<= 9 - rol $9,%edx - # t ^= x2 - xorl 108(%esp),%edx - # s += r - add %esi,%ecx - # s <<<= 9 - rol $9,%ecx - # s ^= x13 - xorl 152(%esp),%ecx - # w += v - add %edi,%ebx - # w <<<= 9 - rol $9,%ebx - # w ^= x7 - xorl 128(%esp),%ebx - # x8 = p - movl %eax,132(%esp) - # x2 = t - movl %edx,108(%esp) - # p += x4 - addl 116(%esp),%eax - # x13 = s - movl %ecx,152(%esp) - # t += x14 - addl 156(%esp),%edx - # x7 = w - movl %ebx,128(%esp) - # p <<<= 13 - rol $13,%eax - # p ^= x12 - xorl 148(%esp),%eax - # t <<<= 13 - rol $13,%edx - # t ^= x6 - xorl 124(%esp),%edx - # r += s - add %ecx,%esi - # r <<<= 13 - rol $13,%esi - # r ^= x1 - xorl 104(%esp),%esi - # v += w - add %ebx,%edi - # v <<<= 13 - rol $13,%edi - # v ^= x11 - xorl 144(%esp),%edi - # x12 = p - movl %eax,148(%esp) - # x6 = t - movl %edx,124(%esp) - # p += x8 - addl 132(%esp),%eax - # x1 = r - movl %esi,104(%esp) - # t += x2 - addl 108(%esp),%edx - # x11 = v - movl %edi,144(%esp) - # p <<<= 18 - rol $18,%eax - # p ^= x0 - xorl 100(%esp),%eax - # t <<<= 18 - rol $18,%edx - # t ^= x10 - xorl 140(%esp),%edx - # s += r - add %esi,%ecx - # s <<<= 18 - rol $18,%ecx - # s ^= x5 - xorl 120(%esp),%ecx - # w += v - add %edi,%ebx - # w <<<= 18 - rol $18,%ebx - # w ^= x15 - xorl 160(%esp),%ebx - # x0 = p - movl %eax,100(%esp) - # x10 = t - movl %edx,140(%esp) - # p += x3 - addl 112(%esp),%eax - # p <<<= 7 - rol $7,%eax - # x5 = s - movl %ecx,120(%esp) - # t += x9 - addl 136(%esp),%edx - # x15 = w - movl %ebx,160(%esp) - # r = x4 - movl 116(%esp),%esi - # r += s - add %ecx,%esi - # v = x14 - movl 156(%esp),%edi - # v += w - add %ebx,%edi - # p ^= x1 - xorl 104(%esp),%eax - # t <<<= 7 - rol $7,%edx - # t ^= x11 - xorl 144(%esp),%edx - # r <<<= 7 - rol $7,%esi - # r ^= x6 - xorl 124(%esp),%esi - # v <<<= 7 - rol $7,%edi - # v ^= x12 - xorl 148(%esp),%edi - # x1 = p - movl %eax,104(%esp) - # x11 = t - movl %edx,144(%esp) - # p += x0 - addl 100(%esp),%eax - # x6 = r - movl %esi,124(%esp) - # t += x10 - addl 140(%esp),%edx - # x12 = v - movl %edi,148(%esp) - # p <<<= 9 - rol $9,%eax - # p ^= x2 - xorl 108(%esp),%eax - # t <<<= 9 - rol $9,%edx - # t ^= x8 - xorl 132(%esp),%edx - # s += r - add %esi,%ecx - # s <<<= 9 - rol $9,%ecx - # s ^= x7 - xorl 128(%esp),%ecx - # w += v - add %edi,%ebx - # w <<<= 9 - rol $9,%ebx - # w ^= x13 - xorl 152(%esp),%ebx - # x2 = p - movl %eax,108(%esp) - # x8 = t - movl %edx,132(%esp) - # p += x1 - addl 104(%esp),%eax - # x7 = s - movl %ecx,128(%esp) - # t += x11 - addl 144(%esp),%edx - # x13 = w - movl %ebx,152(%esp) - # p <<<= 13 - rol $13,%eax - # p ^= x3 - xorl 112(%esp),%eax - # t <<<= 13 - rol $13,%edx - # t ^= x9 - xorl 136(%esp),%edx - # r += s - add %ecx,%esi - # r <<<= 13 - rol $13,%esi - # r ^= x4 - xorl 116(%esp),%esi - # v += w - add %ebx,%edi - # v <<<= 13 - rol $13,%edi - # v ^= x14 - xorl 156(%esp),%edi - # x3 = p - movl %eax,112(%esp) - # x9 = t - movl %edx,136(%esp) - # p += x2 - addl 108(%esp),%eax - # x4 = r - movl %esi,116(%esp) - # t += x8 - addl 132(%esp),%edx - # x14 = v - movl %edi,156(%esp) - # p <<<= 18 - rol $18,%eax - # p ^= x0 - xorl 100(%esp),%eax - # t <<<= 18 - rol $18,%edx - # t ^= x10 - xorl 140(%esp),%edx - # s += r - add %esi,%ecx - # s <<<= 18 - rol $18,%ecx - # s ^= x5 - xorl 120(%esp),%ecx - # w += v - add %edi,%ebx - # w <<<= 18 - rol $18,%ebx - # w ^= x15 - xorl 160(%esp),%ebx - # x0 = p - movl %eax,100(%esp) - # x10 = t - movl %edx,140(%esp) - # p += x12 - addl 148(%esp),%eax - # x5 = s - movl %ecx,120(%esp) - # t += x6 - addl 124(%esp),%edx - # x15 = w - movl %ebx,160(%esp) - # r = x1 - movl 104(%esp),%esi - # r += s - add %ecx,%esi - # v = x11 - movl 144(%esp),%edi - # v += w - add %ebx,%edi - # p <<<= 7 - rol $7,%eax - # p ^= x4 - xorl 116(%esp),%eax - # t <<<= 7 - rol $7,%edx - # t ^= x14 - xorl 156(%esp),%edx - # r <<<= 7 - rol $7,%esi - # r ^= x9 - xorl 136(%esp),%esi - # v <<<= 7 - rol $7,%edi - # v ^= x3 - xorl 112(%esp),%edi - # x4 = p - movl %eax,116(%esp) - # x14 = t - movl %edx,156(%esp) - # p += x0 - addl 100(%esp),%eax - # x9 = r - movl %esi,136(%esp) - # t += x10 - addl 140(%esp),%edx - # x3 = v - movl %edi,112(%esp) - # p <<<= 9 - rol $9,%eax - # p ^= x8 - xorl 132(%esp),%eax - # t <<<= 9 - rol $9,%edx - # t ^= x2 - xorl 108(%esp),%edx - # s += r - add %esi,%ecx - # s <<<= 9 - rol $9,%ecx - # s ^= x13 - xorl 152(%esp),%ecx - # w += v - add %edi,%ebx - # w <<<= 9 - rol $9,%ebx - # w ^= x7 - xorl 128(%esp),%ebx - # x8 = p - movl %eax,132(%esp) - # x2 = t - movl %edx,108(%esp) - # p += x4 - addl 116(%esp),%eax - # x13 = s - movl %ecx,152(%esp) - # t += x14 - addl 156(%esp),%edx - # x7 = w - movl %ebx,128(%esp) - # p <<<= 13 - rol $13,%eax - # p ^= x12 - xorl 148(%esp),%eax - # t <<<= 13 - rol $13,%edx - # t ^= x6 - xorl 124(%esp),%edx - # r += s - add %ecx,%esi - # r <<<= 13 - rol $13,%esi - # r ^= x1 - xorl 104(%esp),%esi - # v += w - add %ebx,%edi - # v <<<= 13 - rol $13,%edi - # v ^= x11 - xorl 144(%esp),%edi - # x12 = p - movl %eax,148(%esp) - # x6 = t - movl %edx,124(%esp) - # p += x8 - addl 132(%esp),%eax - # x1 = r - movl %esi,104(%esp) - # t += x2 - addl 108(%esp),%edx - # x11 = v - movl %edi,144(%esp) - # p <<<= 18 - rol $18,%eax - # p ^= x0 - xorl 100(%esp),%eax - # t <<<= 18 - rol $18,%edx - # t ^= x10 - xorl 140(%esp),%edx - # s += r - add %esi,%ecx - # s <<<= 18 - rol $18,%ecx - # s ^= x5 - xorl 120(%esp),%ecx - # w += v - add %edi,%ebx - # w <<<= 18 - rol $18,%ebx - # w ^= x15 - xorl 160(%esp),%ebx - # x0 = p - movl %eax,100(%esp) - # x10 = t - movl %edx,140(%esp) - # p += x3 - addl 112(%esp),%eax - # p <<<= 7 - rol $7,%eax - # x5 = s - movl %ecx,120(%esp) - # t += x9 - addl 136(%esp),%edx - # x15 = w - movl %ebx,160(%esp) - # r = x4 - movl 116(%esp),%esi - # r += s - add %ecx,%esi - # v = x14 - movl 156(%esp),%edi - # v += w - add %ebx,%edi - # p ^= x1 - xorl 104(%esp),%eax - # t <<<= 7 - rol $7,%edx - # t ^= x11 - xorl 144(%esp),%edx - # r <<<= 7 - rol $7,%esi - # r ^= x6 - xorl 124(%esp),%esi - # v <<<= 7 - rol $7,%edi - # v ^= x12 - xorl 148(%esp),%edi - # x1 = p - movl %eax,104(%esp) - # x11 = t - movl %edx,144(%esp) - # p += x0 - addl 100(%esp),%eax - # x6 = r - movl %esi,124(%esp) - # t += x10 - addl 140(%esp),%edx - # x12 = v - movl %edi,148(%esp) - # p <<<= 9 - rol $9,%eax - # p ^= x2 - xorl 108(%esp),%eax - # t <<<= 9 - rol $9,%edx - # t ^= x8 - xorl 132(%esp),%edx - # s += r - add %esi,%ecx - # s <<<= 9 - rol $9,%ecx - # s ^= x7 - xorl 128(%esp),%ecx - # w += v - add %edi,%ebx - # w <<<= 9 - rol $9,%ebx - # w ^= x13 - xorl 152(%esp),%ebx - # x2 = p - movl %eax,108(%esp) - # x8 = t - movl %edx,132(%esp) - # p += x1 - addl 104(%esp),%eax - # x7 = s - movl %ecx,128(%esp) - # t += x11 - addl 144(%esp),%edx - # x13 = w - movl %ebx,152(%esp) - # p <<<= 13 - rol $13,%eax - # p ^= x3 - xorl 112(%esp),%eax - # t <<<= 13 - rol $13,%edx - # t ^= x9 - xorl 136(%esp),%edx - # r += s - add %ecx,%esi - # r <<<= 13 - rol $13,%esi - # r ^= x4 - xorl 116(%esp),%esi - # v += w - add %ebx,%edi - # v <<<= 13 - rol $13,%edi - # v ^= x14 - xorl 156(%esp),%edi - # x3 = p - movl %eax,112(%esp) - # x9 = t - movl %edx,136(%esp) - # p += x2 - addl 108(%esp),%eax - # x4 = r - movl %esi,116(%esp) - # t += x8 - addl 132(%esp),%edx - # x14 = v - movl %edi,156(%esp) - # p <<<= 18 - rol $18,%eax - # p ^= x0 - xorl 100(%esp),%eax - # t <<<= 18 - rol $18,%edx - # t ^= x10 - xorl 140(%esp),%edx - # s += r - add %esi,%ecx - # s <<<= 18 - rol $18,%ecx - # s ^= x5 - xorl 120(%esp),%ecx - # w += v - add %edi,%ebx - # w <<<= 18 - rol $18,%ebx - # w ^= x15 - xorl 160(%esp),%ebx - # i -= 4 - sub $4,%ebp - # goto mainloop if unsigned > - ja ._mainloop - # x0 = p - movl %eax,100(%esp) - # x5 = s - movl %ecx,120(%esp) - # x10 = t - movl %edx,140(%esp) - # x15 = w - movl %ebx,160(%esp) - # out = out_backup - movl 72(%esp),%edi - # m = m_backup - movl 68(%esp),%esi - # in0 = x0 - movl 100(%esp),%eax - # in1 = x1 - movl 104(%esp),%ecx - # in0 += j0 - addl 164(%esp),%eax - # in1 += j1 - addl 168(%esp),%ecx - # in0 ^= *(uint32 *) (m + 0) - xorl 0(%esi),%eax - # in1 ^= *(uint32 *) (m + 4) - xorl 4(%esi),%ecx - # *(uint32 *) (out + 0) = in0 - movl %eax,0(%edi) - # *(uint32 *) (out + 4) = in1 - movl %ecx,4(%edi) - # in2 = x2 - movl 108(%esp),%eax - # in3 = x3 - movl 112(%esp),%ecx - # in2 += j2 - addl 172(%esp),%eax - # in3 += j3 - addl 176(%esp),%ecx - # in2 ^= *(uint32 *) (m + 8) - xorl 8(%esi),%eax - # in3 ^= *(uint32 *) (m + 12) - xorl 12(%esi),%ecx - # *(uint32 *) (out + 8) = in2 - movl %eax,8(%edi) - # *(uint32 *) (out + 12) = in3 - movl %ecx,12(%edi) - # in4 = x4 - movl 116(%esp),%eax - # in5 = x5 - movl 120(%esp),%ecx - # in4 += j4 - addl 180(%esp),%eax - # in5 += j5 - addl 184(%esp),%ecx - # in4 ^= *(uint32 *) (m + 16) - xorl 16(%esi),%eax - # in5 ^= *(uint32 *) (m + 20) - xorl 20(%esi),%ecx - # *(uint32 *) (out + 16) = in4 - movl %eax,16(%edi) - # *(uint32 *) (out + 20) = in5 - movl %ecx,20(%edi) - # in6 = x6 - movl 124(%esp),%eax - # in7 = x7 - movl 128(%esp),%ecx - # in6 += j6 - addl 188(%esp),%eax - # in7 += j7 - addl 192(%esp),%ecx - # in6 ^= *(uint32 *) (m + 24) - xorl 24(%esi),%eax - # in7 ^= *(uint32 *) (m + 28) - xorl 28(%esi),%ecx - # *(uint32 *) (out + 24) = in6 - movl %eax,24(%edi) - # *(uint32 *) (out + 28) = in7 - movl %ecx,28(%edi) - # in8 = x8 - movl 132(%esp),%eax - # in9 = x9 - movl 136(%esp),%ecx - # in8 += j8 - addl 196(%esp),%eax - # in9 += j9 - addl 200(%esp),%ecx - # in8 ^= *(uint32 *) (m + 32) - xorl 32(%esi),%eax - # in9 ^= *(uint32 *) (m + 36) - xorl 36(%esi),%ecx - # *(uint32 *) (out + 32) = in8 - movl %eax,32(%edi) - # *(uint32 *) (out + 36) = in9 - movl %ecx,36(%edi) - # in10 = x10 - movl 140(%esp),%eax - # in11 = x11 - movl 144(%esp),%ecx - # in10 += j10 - addl 204(%esp),%eax - # in11 += j11 - addl 208(%esp),%ecx - # in10 ^= *(uint32 *) (m + 40) - xorl 40(%esi),%eax - # in11 ^= *(uint32 *) (m + 44) - xorl 44(%esi),%ecx - # *(uint32 *) (out + 40) = in10 - movl %eax,40(%edi) - # *(uint32 *) (out + 44) = in11 - movl %ecx,44(%edi) - # in12 = x12 - movl 148(%esp),%eax - # in13 = x13 - movl 152(%esp),%ecx - # in12 += j12 - addl 212(%esp),%eax - # in13 += j13 - addl 216(%esp),%ecx - # in12 ^= *(uint32 *) (m + 48) - xorl 48(%esi),%eax - # in13 ^= *(uint32 *) (m + 52) - xorl 52(%esi),%ecx - # *(uint32 *) (out + 48) = in12 - movl %eax,48(%edi) - # *(uint32 *) (out + 52) = in13 - movl %ecx,52(%edi) - # in14 = x14 - movl 156(%esp),%eax - # in15 = x15 - movl 160(%esp),%ecx - # in14 += j14 - addl 220(%esp),%eax - # in15 += j15 - addl 224(%esp),%ecx - # in14 ^= *(uint32 *) (m + 56) - xorl 56(%esi),%eax - # in15 ^= *(uint32 *) (m + 60) - xorl 60(%esi),%ecx - # *(uint32 *) (out + 56) = in14 - movl %eax,56(%edi) - # *(uint32 *) (out + 60) = in15 - movl %ecx,60(%edi) - # bytes = bytes_backup - movl 76(%esp),%ebx - # in8 = j8 - movl 196(%esp),%eax - # in9 = j9 - movl 200(%esp),%ecx - # in8 += 1 - add $1,%eax - # in9 += 0 + carry - adc $0,%ecx - # j8 = in8 - movl %eax,196(%esp) - # j9 = in9 - movl %ecx,200(%esp) - # bytes - 64 - cmp $64,%ebx - # goto bytesatleast65 if unsigned> - ja ._bytesatleast65 - # goto bytesatleast64 if unsigned>= - jae ._bytesatleast64 - # m = out - mov %edi,%esi - # out = ctarget - movl 228(%esp),%edi - # i = bytes - mov %ebx,%ecx - # while (i) { *out++ = *m++; --i } - rep movsb -._bytesatleast64: - # x = x_backup - movl 64(%esp),%eax - # in8 = j8 - movl 196(%esp),%ecx - # in9 = j9 - movl 200(%esp),%edx - # *(uint32 *) (x + 32) = in8 - movl %ecx,32(%eax) - # *(uint32 *) (x + 36) = in9 - movl %edx,36(%eax) -._done: - # eax = eax_stack - movl 80(%esp),%eax - # ebx = ebx_stack - movl 84(%esp),%ebx - # esi = esi_stack - movl 88(%esp),%esi - # edi = edi_stack - movl 92(%esp),%edi - # ebp = ebp_stack - movl 96(%esp),%ebp - # leave - add %eax,%esp - ret -._bytesatleast65: - # bytes -= 64 - sub $64,%ebx - # out += 64 - add $64,%edi - # m += 64 - add $64,%esi - # goto bytesatleast1 - jmp ._bytesatleast1 -ENDPROC(salsa20_encrypt_bytes) diff --git a/arch/x86/crypto/salsa20-x86_64-asm_64.S b/arch/x86/crypto/salsa20-x86_64-asm_64.S deleted file mode 100644 index 03a4918f41ee..000000000000 --- a/arch/x86/crypto/salsa20-x86_64-asm_64.S +++ /dev/null @@ -1,805 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#include - -# enter salsa20_encrypt_bytes -ENTRY(salsa20_encrypt_bytes) - mov %rsp,%r11 - and $31,%r11 - add $256,%r11 - sub %r11,%rsp - # x = arg1 - mov %rdi,%r8 - # m = arg2 - mov %rsi,%rsi - # out = arg3 - mov %rdx,%rdi - # bytes = arg4 - mov %rcx,%rdx - # unsigned>? bytes - 0 - cmp $0,%rdx - # comment:fp stack unchanged by jump - # goto done if !unsigned> - jbe ._done - # comment:fp stack unchanged by fallthrough -# start: -._start: - # r11_stack = r11 - movq %r11,0(%rsp) - # r12_stack = r12 - movq %r12,8(%rsp) - # r13_stack = r13 - movq %r13,16(%rsp) - # r14_stack = r14 - movq %r14,24(%rsp) - # r15_stack = r15 - movq %r15,32(%rsp) - # rbx_stack = rbx - movq %rbx,40(%rsp) - # rbp_stack = rbp - movq %rbp,48(%rsp) - # in0 = *(uint64 *) (x + 0) - movq 0(%r8),%rcx - # in2 = *(uint64 *) (x + 8) - movq 8(%r8),%r9 - # in4 = *(uint64 *) (x + 16) - movq 16(%r8),%rax - # in6 = *(uint64 *) (x + 24) - movq 24(%r8),%r10 - # in8 = *(uint64 *) (x + 32) - movq 32(%r8),%r11 - # in10 = *(uint64 *) (x + 40) - movq 40(%r8),%r12 - # in12 = *(uint64 *) (x + 48) - movq 48(%r8),%r13 - # in14 = *(uint64 *) (x + 56) - movq 56(%r8),%r14 - # j0 = in0 - movq %rcx,56(%rsp) - # j2 = in2 - movq %r9,64(%rsp) - # j4 = in4 - movq %rax,72(%rsp) - # j6 = in6 - movq %r10,80(%rsp) - # j8 = in8 - movq %r11,88(%rsp) - # j10 = in10 - movq %r12,96(%rsp) - # j12 = in12 - movq %r13,104(%rsp) - # j14 = in14 - movq %r14,112(%rsp) - # x_backup = x - movq %r8,120(%rsp) -# bytesatleast1: -._bytesatleast1: - # unsigned>= 32 - shr $32,%rdi - # x3 = j2 - movq 64(%rsp),%rsi - # x2 = x3 - mov %rsi,%rcx - # (uint64) x3 >>= 32 - shr $32,%rsi - # x5 = j4 - movq 72(%rsp),%r8 - # x4 = x5 - mov %r8,%r9 - # (uint64) x5 >>= 32 - shr $32,%r8 - # x5_stack = x5 - movq %r8,160(%rsp) - # x7 = j6 - movq 80(%rsp),%r8 - # x6 = x7 - mov %r8,%rax - # (uint64) x7 >>= 32 - shr $32,%r8 - # x9 = j8 - movq 88(%rsp),%r10 - # x8 = x9 - mov %r10,%r11 - # (uint64) x9 >>= 32 - shr $32,%r10 - # x11 = j10 - movq 96(%rsp),%r12 - # x10 = x11 - mov %r12,%r13 - # x10_stack = x10 - movq %r13,168(%rsp) - # (uint64) x11 >>= 32 - shr $32,%r12 - # x13 = j12 - movq 104(%rsp),%r13 - # x12 = x13 - mov %r13,%r14 - # (uint64) x13 >>= 32 - shr $32,%r13 - # x15 = j14 - movq 112(%rsp),%r15 - # x14 = x15 - mov %r15,%rbx - # (uint64) x15 >>= 32 - shr $32,%r15 - # x15_stack = x15 - movq %r15,176(%rsp) - # i = 20 - mov $20,%r15 -# mainloop: -._mainloop: - # i_backup = i - movq %r15,184(%rsp) - # x5 = x5_stack - movq 160(%rsp),%r15 - # a = x12 + x0 - lea (%r14,%rdx),%rbp - # (uint32) a <<<= 7 - rol $7,%ebp - # x4 ^= a - xor %rbp,%r9 - # b = x1 + x5 - lea (%rdi,%r15),%rbp - # (uint32) b <<<= 7 - rol $7,%ebp - # x9 ^= b - xor %rbp,%r10 - # a = x0 + x4 - lea (%rdx,%r9),%rbp - # (uint32) a <<<= 9 - rol $9,%ebp - # x8 ^= a - xor %rbp,%r11 - # b = x5 + x9 - lea (%r15,%r10),%rbp - # (uint32) b <<<= 9 - rol $9,%ebp - # x13 ^= b - xor %rbp,%r13 - # a = x4 + x8 - lea (%r9,%r11),%rbp - # (uint32) a <<<= 13 - rol $13,%ebp - # x12 ^= a - xor %rbp,%r14 - # b = x9 + x13 - lea (%r10,%r13),%rbp - # (uint32) b <<<= 13 - rol $13,%ebp - # x1 ^= b - xor %rbp,%rdi - # a = x8 + x12 - lea (%r11,%r14),%rbp - # (uint32) a <<<= 18 - rol $18,%ebp - # x0 ^= a - xor %rbp,%rdx - # b = x13 + x1 - lea (%r13,%rdi),%rbp - # (uint32) b <<<= 18 - rol $18,%ebp - # x5 ^= b - xor %rbp,%r15 - # x10 = x10_stack - movq 168(%rsp),%rbp - # x5_stack = x5 - movq %r15,160(%rsp) - # c = x6 + x10 - lea (%rax,%rbp),%r15 - # (uint32) c <<<= 7 - rol $7,%r15d - # x14 ^= c - xor %r15,%rbx - # c = x10 + x14 - lea (%rbp,%rbx),%r15 - # (uint32) c <<<= 9 - rol $9,%r15d - # x2 ^= c - xor %r15,%rcx - # c = x14 + x2 - lea (%rbx,%rcx),%r15 - # (uint32) c <<<= 13 - rol $13,%r15d - # x6 ^= c - xor %r15,%rax - # c = x2 + x6 - lea (%rcx,%rax),%r15 - # (uint32) c <<<= 18 - rol $18,%r15d - # x10 ^= c - xor %r15,%rbp - # x15 = x15_stack - movq 176(%rsp),%r15 - # x10_stack = x10 - movq %rbp,168(%rsp) - # d = x11 + x15 - lea (%r12,%r15),%rbp - # (uint32) d <<<= 7 - rol $7,%ebp - # x3 ^= d - xor %rbp,%rsi - # d = x15 + x3 - lea (%r15,%rsi),%rbp - # (uint32) d <<<= 9 - rol $9,%ebp - # x7 ^= d - xor %rbp,%r8 - # d = x3 + x7 - lea (%rsi,%r8),%rbp - # (uint32) d <<<= 13 - rol $13,%ebp - # x11 ^= d - xor %rbp,%r12 - # d = x7 + x11 - lea (%r8,%r12),%rbp - # (uint32) d <<<= 18 - rol $18,%ebp - # x15 ^= d - xor %rbp,%r15 - # x15_stack = x15 - movq %r15,176(%rsp) - # x5 = x5_stack - movq 160(%rsp),%r15 - # a = x3 + x0 - lea (%rsi,%rdx),%rbp - # (uint32) a <<<= 7 - rol $7,%ebp - # x1 ^= a - xor %rbp,%rdi - # b = x4 + x5 - lea (%r9,%r15),%rbp - # (uint32) b <<<= 7 - rol $7,%ebp - # x6 ^= b - xor %rbp,%rax - # a = x0 + x1 - lea (%rdx,%rdi),%rbp - # (uint32) a <<<= 9 - rol $9,%ebp - # x2 ^= a - xor %rbp,%rcx - # b = x5 + x6 - lea (%r15,%rax),%rbp - # (uint32) b <<<= 9 - rol $9,%ebp - # x7 ^= b - xor %rbp,%r8 - # a = x1 + x2 - lea (%rdi,%rcx),%rbp - # (uint32) a <<<= 13 - rol $13,%ebp - # x3 ^= a - xor %rbp,%rsi - # b = x6 + x7 - lea (%rax,%r8),%rbp - # (uint32) b <<<= 13 - rol $13,%ebp - # x4 ^= b - xor %rbp,%r9 - # a = x2 + x3 - lea (%rcx,%rsi),%rbp - # (uint32) a <<<= 18 - rol $18,%ebp - # x0 ^= a - xor %rbp,%rdx - # b = x7 + x4 - lea (%r8,%r9),%rbp - # (uint32) b <<<= 18 - rol $18,%ebp - # x5 ^= b - xor %rbp,%r15 - # x10 = x10_stack - movq 168(%rsp),%rbp - # x5_stack = x5 - movq %r15,160(%rsp) - # c = x9 + x10 - lea (%r10,%rbp),%r15 - # (uint32) c <<<= 7 - rol $7,%r15d - # x11 ^= c - xor %r15,%r12 - # c = x10 + x11 - lea (%rbp,%r12),%r15 - # (uint32) c <<<= 9 - rol $9,%r15d - # x8 ^= c - xor %r15,%r11 - # c = x11 + x8 - lea (%r12,%r11),%r15 - # (uint32) c <<<= 13 - rol $13,%r15d - # x9 ^= c - xor %r15,%r10 - # c = x8 + x9 - lea (%r11,%r10),%r15 - # (uint32) c <<<= 18 - rol $18,%r15d - # x10 ^= c - xor %r15,%rbp - # x15 = x15_stack - movq 176(%rsp),%r15 - # x10_stack = x10 - movq %rbp,168(%rsp) - # d = x14 + x15 - lea (%rbx,%r15),%rbp - # (uint32) d <<<= 7 - rol $7,%ebp - # x12 ^= d - xor %rbp,%r14 - # d = x15 + x12 - lea (%r15,%r14),%rbp - # (uint32) d <<<= 9 - rol $9,%ebp - # x13 ^= d - xor %rbp,%r13 - # d = x12 + x13 - lea (%r14,%r13),%rbp - # (uint32) d <<<= 13 - rol $13,%ebp - # x14 ^= d - xor %rbp,%rbx - # d = x13 + x14 - lea (%r13,%rbx),%rbp - # (uint32) d <<<= 18 - rol $18,%ebp - # x15 ^= d - xor %rbp,%r15 - # x15_stack = x15 - movq %r15,176(%rsp) - # x5 = x5_stack - movq 160(%rsp),%r15 - # a = x12 + x0 - lea (%r14,%rdx),%rbp - # (uint32) a <<<= 7 - rol $7,%ebp - # x4 ^= a - xor %rbp,%r9 - # b = x1 + x5 - lea (%rdi,%r15),%rbp - # (uint32) b <<<= 7 - rol $7,%ebp - # x9 ^= b - xor %rbp,%r10 - # a = x0 + x4 - lea (%rdx,%r9),%rbp - # (uint32) a <<<= 9 - rol $9,%ebp - # x8 ^= a - xor %rbp,%r11 - # b = x5 + x9 - lea (%r15,%r10),%rbp - # (uint32) b <<<= 9 - rol $9,%ebp - # x13 ^= b - xor %rbp,%r13 - # a = x4 + x8 - lea (%r9,%r11),%rbp - # (uint32) a <<<= 13 - rol $13,%ebp - # x12 ^= a - xor %rbp,%r14 - # b = x9 + x13 - lea (%r10,%r13),%rbp - # (uint32) b <<<= 13 - rol $13,%ebp - # x1 ^= b - xor %rbp,%rdi - # a = x8 + x12 - lea (%r11,%r14),%rbp - # (uint32) a <<<= 18 - rol $18,%ebp - # x0 ^= a - xor %rbp,%rdx - # b = x13 + x1 - lea (%r13,%rdi),%rbp - # (uint32) b <<<= 18 - rol $18,%ebp - # x5 ^= b - xor %rbp,%r15 - # x10 = x10_stack - movq 168(%rsp),%rbp - # x5_stack = x5 - movq %r15,160(%rsp) - # c = x6 + x10 - lea (%rax,%rbp),%r15 - # (uint32) c <<<= 7 - rol $7,%r15d - # x14 ^= c - xor %r15,%rbx - # c = x10 + x14 - lea (%rbp,%rbx),%r15 - # (uint32) c <<<= 9 - rol $9,%r15d - # x2 ^= c - xor %r15,%rcx - # c = x14 + x2 - lea (%rbx,%rcx),%r15 - # (uint32) c <<<= 13 - rol $13,%r15d - # x6 ^= c - xor %r15,%rax - # c = x2 + x6 - lea (%rcx,%rax),%r15 - # (uint32) c <<<= 18 - rol $18,%r15d - # x10 ^= c - xor %r15,%rbp - # x15 = x15_stack - movq 176(%rsp),%r15 - # x10_stack = x10 - movq %rbp,168(%rsp) - # d = x11 + x15 - lea (%r12,%r15),%rbp - # (uint32) d <<<= 7 - rol $7,%ebp - # x3 ^= d - xor %rbp,%rsi - # d = x15 + x3 - lea (%r15,%rsi),%rbp - # (uint32) d <<<= 9 - rol $9,%ebp - # x7 ^= d - xor %rbp,%r8 - # d = x3 + x7 - lea (%rsi,%r8),%rbp - # (uint32) d <<<= 13 - rol $13,%ebp - # x11 ^= d - xor %rbp,%r12 - # d = x7 + x11 - lea (%r8,%r12),%rbp - # (uint32) d <<<= 18 - rol $18,%ebp - # x15 ^= d - xor %rbp,%r15 - # x15_stack = x15 - movq %r15,176(%rsp) - # x5 = x5_stack - movq 160(%rsp),%r15 - # a = x3 + x0 - lea (%rsi,%rdx),%rbp - # (uint32) a <<<= 7 - rol $7,%ebp - # x1 ^= a - xor %rbp,%rdi - # b = x4 + x5 - lea (%r9,%r15),%rbp - # (uint32) b <<<= 7 - rol $7,%ebp - # x6 ^= b - xor %rbp,%rax - # a = x0 + x1 - lea (%rdx,%rdi),%rbp - # (uint32) a <<<= 9 - rol $9,%ebp - # x2 ^= a - xor %rbp,%rcx - # b = x5 + x6 - lea (%r15,%rax),%rbp - # (uint32) b <<<= 9 - rol $9,%ebp - # x7 ^= b - xor %rbp,%r8 - # a = x1 + x2 - lea (%rdi,%rcx),%rbp - # (uint32) a <<<= 13 - rol $13,%ebp - # x3 ^= a - xor %rbp,%rsi - # b = x6 + x7 - lea (%rax,%r8),%rbp - # (uint32) b <<<= 13 - rol $13,%ebp - # x4 ^= b - xor %rbp,%r9 - # a = x2 + x3 - lea (%rcx,%rsi),%rbp - # (uint32) a <<<= 18 - rol $18,%ebp - # x0 ^= a - xor %rbp,%rdx - # b = x7 + x4 - lea (%r8,%r9),%rbp - # (uint32) b <<<= 18 - rol $18,%ebp - # x5 ^= b - xor %rbp,%r15 - # x10 = x10_stack - movq 168(%rsp),%rbp - # x5_stack = x5 - movq %r15,160(%rsp) - # c = x9 + x10 - lea (%r10,%rbp),%r15 - # (uint32) c <<<= 7 - rol $7,%r15d - # x11 ^= c - xor %r15,%r12 - # c = x10 + x11 - lea (%rbp,%r12),%r15 - # (uint32) c <<<= 9 - rol $9,%r15d - # x8 ^= c - xor %r15,%r11 - # c = x11 + x8 - lea (%r12,%r11),%r15 - # (uint32) c <<<= 13 - rol $13,%r15d - # x9 ^= c - xor %r15,%r10 - # c = x8 + x9 - lea (%r11,%r10),%r15 - # (uint32) c <<<= 18 - rol $18,%r15d - # x10 ^= c - xor %r15,%rbp - # x15 = x15_stack - movq 176(%rsp),%r15 - # x10_stack = x10 - movq %rbp,168(%rsp) - # d = x14 + x15 - lea (%rbx,%r15),%rbp - # (uint32) d <<<= 7 - rol $7,%ebp - # x12 ^= d - xor %rbp,%r14 - # d = x15 + x12 - lea (%r15,%r14),%rbp - # (uint32) d <<<= 9 - rol $9,%ebp - # x13 ^= d - xor %rbp,%r13 - # d = x12 + x13 - lea (%r14,%r13),%rbp - # (uint32) d <<<= 13 - rol $13,%ebp - # x14 ^= d - xor %rbp,%rbx - # d = x13 + x14 - lea (%r13,%rbx),%rbp - # (uint32) d <<<= 18 - rol $18,%ebp - # x15 ^= d - xor %rbp,%r15 - # x15_stack = x15 - movq %r15,176(%rsp) - # i = i_backup - movq 184(%rsp),%r15 - # unsigned>? i -= 4 - sub $4,%r15 - # comment:fp stack unchanged by jump - # goto mainloop if unsigned> - ja ._mainloop - # (uint32) x2 += j2 - addl 64(%rsp),%ecx - # x3 <<= 32 - shl $32,%rsi - # x3 += j2 - addq 64(%rsp),%rsi - # (uint64) x3 >>= 32 - shr $32,%rsi - # x3 <<= 32 - shl $32,%rsi - # x2 += x3 - add %rsi,%rcx - # (uint32) x6 += j6 - addl 80(%rsp),%eax - # x7 <<= 32 - shl $32,%r8 - # x7 += j6 - addq 80(%rsp),%r8 - # (uint64) x7 >>= 32 - shr $32,%r8 - # x7 <<= 32 - shl $32,%r8 - # x6 += x7 - add %r8,%rax - # (uint32) x8 += j8 - addl 88(%rsp),%r11d - # x9 <<= 32 - shl $32,%r10 - # x9 += j8 - addq 88(%rsp),%r10 - # (uint64) x9 >>= 32 - shr $32,%r10 - # x9 <<= 32 - shl $32,%r10 - # x8 += x9 - add %r10,%r11 - # (uint32) x12 += j12 - addl 104(%rsp),%r14d - # x13 <<= 32 - shl $32,%r13 - # x13 += j12 - addq 104(%rsp),%r13 - # (uint64) x13 >>= 32 - shr $32,%r13 - # x13 <<= 32 - shl $32,%r13 - # x12 += x13 - add %r13,%r14 - # (uint32) x0 += j0 - addl 56(%rsp),%edx - # x1 <<= 32 - shl $32,%rdi - # x1 += j0 - addq 56(%rsp),%rdi - # (uint64) x1 >>= 32 - shr $32,%rdi - # x1 <<= 32 - shl $32,%rdi - # x0 += x1 - add %rdi,%rdx - # x5 = x5_stack - movq 160(%rsp),%rdi - # (uint32) x4 += j4 - addl 72(%rsp),%r9d - # x5 <<= 32 - shl $32,%rdi - # x5 += j4 - addq 72(%rsp),%rdi - # (uint64) x5 >>= 32 - shr $32,%rdi - # x5 <<= 32 - shl $32,%rdi - # x4 += x5 - add %rdi,%r9 - # x10 = x10_stack - movq 168(%rsp),%r8 - # (uint32) x10 += j10 - addl 96(%rsp),%r8d - # x11 <<= 32 - shl $32,%r12 - # x11 += j10 - addq 96(%rsp),%r12 - # (uint64) x11 >>= 32 - shr $32,%r12 - # x11 <<= 32 - shl $32,%r12 - # x10 += x11 - add %r12,%r8 - # x15 = x15_stack - movq 176(%rsp),%rdi - # (uint32) x14 += j14 - addl 112(%rsp),%ebx - # x15 <<= 32 - shl $32,%rdi - # x15 += j14 - addq 112(%rsp),%rdi - # (uint64) x15 >>= 32 - shr $32,%rdi - # x15 <<= 32 - shl $32,%rdi - # x14 += x15 - add %rdi,%rbx - # out = out_backup - movq 136(%rsp),%rdi - # m = m_backup - movq 144(%rsp),%rsi - # x0 ^= *(uint64 *) (m + 0) - xorq 0(%rsi),%rdx - # *(uint64 *) (out + 0) = x0 - movq %rdx,0(%rdi) - # x2 ^= *(uint64 *) (m + 8) - xorq 8(%rsi),%rcx - # *(uint64 *) (out + 8) = x2 - movq %rcx,8(%rdi) - # x4 ^= *(uint64 *) (m + 16) - xorq 16(%rsi),%r9 - # *(uint64 *) (out + 16) = x4 - movq %r9,16(%rdi) - # x6 ^= *(uint64 *) (m + 24) - xorq 24(%rsi),%rax - # *(uint64 *) (out + 24) = x6 - movq %rax,24(%rdi) - # x8 ^= *(uint64 *) (m + 32) - xorq 32(%rsi),%r11 - # *(uint64 *) (out + 32) = x8 - movq %r11,32(%rdi) - # x10 ^= *(uint64 *) (m + 40) - xorq 40(%rsi),%r8 - # *(uint64 *) (out + 40) = x10 - movq %r8,40(%rdi) - # x12 ^= *(uint64 *) (m + 48) - xorq 48(%rsi),%r14 - # *(uint64 *) (out + 48) = x12 - movq %r14,48(%rdi) - # x14 ^= *(uint64 *) (m + 56) - xorq 56(%rsi),%rbx - # *(uint64 *) (out + 56) = x14 - movq %rbx,56(%rdi) - # bytes = bytes_backup - movq 152(%rsp),%rdx - # in8 = j8 - movq 88(%rsp),%rcx - # in8 += 1 - add $1,%rcx - # j8 = in8 - movq %rcx,88(%rsp) - # unsigned>? unsigned - ja ._bytesatleast65 - # comment:fp stack unchanged by jump - # goto bytesatleast64 if !unsigned< - jae ._bytesatleast64 - # m = out - mov %rdi,%rsi - # out = ctarget - movq 128(%rsp),%rdi - # i = bytes - mov %rdx,%rcx - # while (i) { *out++ = *m++; --i } - rep movsb - # comment:fp stack unchanged by fallthrough -# bytesatleast64: -._bytesatleast64: - # x = x_backup - movq 120(%rsp),%rdi - # in8 = j8 - movq 88(%rsp),%rsi - # *(uint64 *) (x + 32) = in8 - movq %rsi,32(%rdi) - # r11 = r11_stack - movq 0(%rsp),%r11 - # r12 = r12_stack - movq 8(%rsp),%r12 - # r13 = r13_stack - movq 16(%rsp),%r13 - # r14 = r14_stack - movq 24(%rsp),%r14 - # r15 = r15_stack - movq 32(%rsp),%r15 - # rbx = rbx_stack - movq 40(%rsp),%rbx - # rbp = rbp_stack - movq 48(%rsp),%rbp - # comment:fp stack unchanged by fallthrough -# done: -._done: - # leave - add %r11,%rsp - mov %rdi,%rax - mov %rsi,%rdx - ret -# bytesatleast65: -._bytesatleast65: - # bytes -= 64 - sub $64,%rdx - # out += 64 - add $64,%rdi - # m += 64 - add $64,%rsi - # comment:fp stack unchanged by jump - # goto bytesatleast1 - jmp ._bytesatleast1 -ENDPROC(salsa20_encrypt_bytes) diff --git a/arch/x86/crypto/salsa20_glue.c b/arch/x86/crypto/salsa20_glue.c deleted file mode 100644 index b07d7d959806..000000000000 --- a/arch/x86/crypto/salsa20_glue.c +++ /dev/null @@ -1,91 +0,0 @@ -/* - * Glue code for optimized assembly version of Salsa20. - * - * Copyright (c) 2007 Tan Swee Heng - * - * The assembly codes are public domain assembly codes written by Daniel. J. - * Bernstein . The codes are modified to include indentation - * and to remove extraneous comments and functions that are not needed. - * - i586 version, renamed as salsa20-i586-asm_32.S - * available from - * - x86-64 version, renamed as salsa20-x86_64-asm_64.S - * available from - * - * Also modified to set up the initial state using the generic C code rather - * than in assembly. - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the Free - * Software Foundation; either version 2 of the License, or (at your option) - * any later version. - * - */ - -#include -#include -#include -#include - -asmlinkage void salsa20_encrypt_bytes(u32 state[16], const u8 *src, u8 *dst, - u32 bytes); - -static int salsa20_asm_crypt(struct skcipher_request *req) -{ - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - const struct salsa20_ctx *ctx = crypto_skcipher_ctx(tfm); - struct skcipher_walk walk; - u32 state[16]; - int err; - - err = skcipher_walk_virt(&walk, req, true); - - crypto_salsa20_init(state, ctx, walk.iv); - - while (walk.nbytes > 0) { - unsigned int nbytes = walk.nbytes; - - if (nbytes < walk.total) - nbytes = round_down(nbytes, walk.stride); - - salsa20_encrypt_bytes(state, walk.src.virt.addr, - walk.dst.virt.addr, nbytes); - err = skcipher_walk_done(&walk, walk.nbytes - nbytes); - } - - return err; -} - -static struct skcipher_alg alg = { - .base.cra_name = "salsa20", - .base.cra_driver_name = "salsa20-asm", - .base.cra_priority = 200, - .base.cra_blocksize = 1, - .base.cra_ctxsize = sizeof(struct salsa20_ctx), - .base.cra_module = THIS_MODULE, - - .min_keysize = SALSA20_MIN_KEY_SIZE, - .max_keysize = SALSA20_MAX_KEY_SIZE, - .ivsize = SALSA20_IV_SIZE, - .chunksize = SALSA20_BLOCK_SIZE, - .setkey = crypto_salsa20_setkey, - .encrypt = salsa20_asm_crypt, - .decrypt = salsa20_asm_crypt, -}; - -static int __init init(void) -{ - return crypto_register_skcipher(&alg); -} - -static void __exit fini(void) -{ - crypto_unregister_skcipher(&alg); -} - -module_init(init); -module_exit(fini); - -MODULE_LICENSE("GPL"); -MODULE_DESCRIPTION ("Salsa20 stream cipher algorithm (optimized assembly version)"); -MODULE_ALIAS_CRYPTO("salsa20"); -MODULE_ALIAS_CRYPTO("salsa20-asm"); diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index fbf6a6c3fd2d..92190879b228 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -164,6 +164,7 @@ static void exit_to_usermode_loop(struct pt_regs *regs, u32 cached_flags) if (cached_flags & _TIF_NOTIFY_RESUME) { clear_thread_flag(TIF_NOTIFY_RESUME); tracehook_notify_resume(regs); + rseq_handle_notify_resume(regs); } if (cached_flags & _TIF_USER_RETURN_NOTIFY) @@ -254,6 +255,8 @@ __visible inline void syscall_return_slowpath(struct pt_regs *regs) WARN(irqs_disabled(), "syscall %ld left IRQs disabled", regs->orig_ax)) local_irq_enable(); + rseq_syscall(regs); + /* * First do one-time work. If these work items are enabled, we * want to run them exactly once per syscall exit with IRQs on. diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index bef8e2b202a8..2582881d19ce 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -239,7 +239,7 @@ ENTRY(__switch_to_asm) movl %esp, TASK_threadsp(%eax) movl TASK_threadsp(%edx), %esp -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR movl TASK_stack_canary(%edx), %ebx movl %ebx, PER_CPU_VAR(stack_canary)+stack_canary_offset #endif diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 3166b9674429..73a522d53b53 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -357,7 +357,7 @@ ENTRY(__switch_to_asm) movq %rsp, TASK_threadsp(%rdi) movq TASK_threadsp(%rsi), %rsp -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR movq TASK_stack_canary(%rsi), %rbx movq %rbx, PER_CPU_VAR(irq_stack_union)+stack_canary_offset #endif diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl index d6b27dab1b30..3cf7b533b3d1 100644 --- a/arch/x86/entry/syscalls/syscall_32.tbl +++ b/arch/x86/entry/syscalls/syscall_32.tbl @@ -396,3 +396,5 @@ 382 i386 pkey_free sys_pkey_free __ia32_sys_pkey_free 383 i386 statx sys_statx __ia32_sys_statx 384 i386 arch_prctl sys_arch_prctl __ia32_compat_sys_arch_prctl +385 i386 io_pgetevents sys_io_pgetevents __ia32_compat_sys_io_pgetevents +386 i386 rseq sys_rseq __ia32_sys_rseq diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl index 4dfe42666d0c..f0b1709a5ffb 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -341,6 +341,8 @@ 330 common pkey_alloc __x64_sys_pkey_alloc 331 common pkey_free __x64_sys_pkey_free 332 common statx __x64_sys_statx +333 common io_pgetevents __x64_sys_io_pgetevents +334 common rseq __x64_sys_rseq # # x32-specific system call numbers start at 512 to avoid cache impact diff --git a/arch/x86/entry/vdso/Makefile b/arch/x86/entry/vdso/Makefile index d998a487c9b1..261802b1cc50 100644 --- a/arch/x86/entry/vdso/Makefile +++ b/arch/x86/entry/vdso/Makefile @@ -44,14 +44,14 @@ obj-y += $(vdso_img_objs) targets += $(vdso_img_cfiles) targets += $(vdso_img_sodbg) $(vdso_img-y:%=vdso%.so) -export CPPFLAGS_vdso.lds += -P -C +CPPFLAGS_vdso.lds += -P -C VDSO_LDFLAGS_vdso.lds = -m64 -Wl,-soname=linux-vdso.so.1 \ -Wl,--no-undefined \ -Wl,-z,max-page-size=4096 -Wl,-z,common-page-size=4096 \ $(DISABLE_LTO) -$(obj)/vdso64.so.dbg: $(src)/vdso.lds $(vobjs) FORCE +$(obj)/vdso64.so.dbg: $(obj)/vdso.lds $(vobjs) FORCE $(call if_changed,vdso) HOST_EXTRACFLAGS += -I$(srctree)/tools/include -I$(srctree)/include/uapi -I$(srctree)/arch/$(SUBARCH)/include/uapi @@ -100,11 +100,8 @@ VDSO_LDFLAGS_vdsox32.lds = -Wl,-m,elf32_x86_64 \ -Wl,-z,max-page-size=4096 \ -Wl,-z,common-page-size=4096 -# 64-bit objects to re-brand as x32 -vobjs64-for-x32 := $(filter-out $(vobjs-nox32),$(vobjs-y)) - # x32-rebranded versions -vobjx32s-y := $(vobjs64-for-x32:.o=-x32.o) +vobjx32s-y := $(vobjs-y:.o=-x32.o) # same thing, but in the output directory vobjx32s := $(foreach F,$(vobjx32s-y),$(obj)/$F) @@ -122,7 +119,7 @@ $(obj)/%.so: OBJCOPYFLAGS := -S $(obj)/%.so: $(obj)/%.so.dbg $(call if_changed,objcopy) -$(obj)/vdsox32.so.dbg: $(src)/vdsox32.lds $(vobjx32s) FORCE +$(obj)/vdsox32.so.dbg: $(obj)/vdsox32.lds $(vobjx32s) FORCE $(call if_changed,vdso) CPPFLAGS_vdso32.lds = $(CPPFLAGS_vdso.lds) diff --git a/arch/x86/entry/vsyscall/vsyscall_64.c b/arch/x86/entry/vsyscall/vsyscall_64.c index 70b7845434cb..82ed001e8909 100644 --- a/arch/x86/entry/vsyscall/vsyscall_64.c +++ b/arch/x86/entry/vsyscall/vsyscall_64.c @@ -107,7 +107,7 @@ static bool write_ok_or_segv(unsigned long ptr, size_t size) thread->cr2 = ptr; thread->trap_nr = X86_TRAP_PF; - memset(&info, 0, sizeof(info)); + clear_siginfo(&info); info.si_signo = SIGSEGV; info.si_errno = 0; info.si_code = SEGV_MAPERR; @@ -201,7 +201,7 @@ bool emulate_vsyscall(struct pt_regs *regs, unsigned long address) /* * Handle seccomp. regs->ip must be the original value. - * See seccomp_send_sigsys and Documentation/prctl/seccomp_filter.txt. + * See seccomp_send_sigsys and Documentation/userspace-api/seccomp_filter.rst. * * We could optimize the seccomp disabled case, but performance * here doesn't matter. diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c index 786fd875de92..4b98101209a1 100644 --- a/arch/x86/events/amd/ibs.c +++ b/arch/x86/events/amd/ibs.c @@ -889,7 +889,7 @@ static void force_ibs_eilvt_setup(void) if (!ibs_eilvt_valid()) goto out; - pr_info("IBS: LVT offset %d assigned\n", offset); + pr_info("LVT offset %d assigned\n", offset); return; out: diff --git a/arch/x86/events/amd/iommu.c b/arch/x86/events/amd/iommu.c index 38b5d41b0c37..3210fee27e7f 100644 --- a/arch/x86/events/amd/iommu.c +++ b/arch/x86/events/amd/iommu.c @@ -387,7 +387,7 @@ static __init int _init_events_attrs(void) while (amd_iommu_v2_event_descs[i].attr.attr.name) i++; - attrs = kzalloc(sizeof(struct attribute **) * (i + 1), GFP_KERNEL); + attrs = kcalloc(i + 1, sizeof(struct attribute **), GFP_KERNEL); if (!attrs) return -ENOMEM; diff --git a/arch/x86/events/amd/uncore.c b/arch/x86/events/amd/uncore.c index f5cbbba99283..981ba5e8241b 100644 --- a/arch/x86/events/amd/uncore.c +++ b/arch/x86/events/amd/uncore.c @@ -19,6 +19,7 @@ #include #include #include +#include #define NUM_COUNTERS_NB 4 #define NUM_COUNTERS_L2 4 @@ -399,26 +400,8 @@ static int amd_uncore_cpu_starting(unsigned int cpu) } if (amd_uncore_llc) { - unsigned int apicid = cpu_data(cpu).apicid; - unsigned int nshared, subleaf, prev_eax = 0; - uncore = *per_cpu_ptr(amd_uncore_llc, cpu); - /* - * Iterate over Cache Topology Definition leaves until no - * more cache descriptions are available. - */ - for (subleaf = 0; subleaf < 5; subleaf++) { - cpuid_count(0x8000001d, subleaf, &eax, &ebx, &ecx, &edx); - - /* EAX[0:4] gives type of cache */ - if (!(eax & 0x1f)) - break; - - prev_eax = eax; - } - nshared = ((prev_eax >> 14) & 0xfff) + 1; - - uncore->id = apicid - (apicid % nshared); + uncore->id = per_cpu(cpu_llc_id, cpu); uncore = amd_uncore_find_online_sibling(uncore, amd_uncore_llc); *per_cpu_ptr(amd_uncore_llc, cpu) = uncore; diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c index 45b2b1c93d04..5f4829f10129 100644 --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -1637,7 +1637,7 @@ __init struct attribute **merge_attr(struct attribute **a, struct attribute **b) j++; j++; - new = kmalloc(sizeof(struct attribute *) * j, GFP_KERNEL); + new = kmalloc_array(j, sizeof(struct attribute *), GFP_KERNEL); if (!new) return NULL; @@ -2397,7 +2397,7 @@ static unsigned long get_segment_base(unsigned int segment) #ifdef CONFIG_IA32_EMULATION -#include +#include static inline int perf_callchain_user32(struct pt_regs *regs, struct perf_callchain_entry_ctx *entry) diff --git a/arch/x86/events/intel/pt.c b/arch/x86/events/intel/pt.c index 3b993942a0e4..8d016ce5b80d 100644 --- a/arch/x86/events/intel/pt.c +++ b/arch/x86/events/intel/pt.c @@ -1194,7 +1194,7 @@ static int pt_event_addr_filters_validate(struct list_head *filters) filter->action == PERF_ADDR_FILTER_ACTION_START) return -EOPNOTSUPP; - if (!filter->inode) { + if (!filter->path.dentry) { if (!valid_kernel_ip(filter->offset)) return -EINVAL; @@ -1221,7 +1221,7 @@ static void pt_event_addr_filters_sync(struct perf_event *event) return; list_for_each_entry(filter, &head->list, entry) { - if (filter->inode && !offs[range]) { + if (filter->path.dentry && !offs[range]) { msr_a = msr_b = 0; } else { /* apply the offset */ diff --git a/arch/x86/events/intel/uncore.c b/arch/x86/events/intel/uncore.c index a7956fc7ca1d..27a461414b30 100644 --- a/arch/x86/events/intel/uncore.c +++ b/arch/x86/events/intel/uncore.c @@ -203,7 +203,7 @@ static void uncore_assign_hw_event(struct intel_uncore_box *box, hwc->idx = idx; hwc->last_tag = ++box->tags[idx]; - if (hwc->idx == UNCORE_PMC_IDX_FIXED) { + if (uncore_pmc_fixed(hwc->idx)) { hwc->event_base = uncore_fixed_ctr(box); hwc->config_base = uncore_fixed_ctl(box); return; @@ -218,7 +218,9 @@ void uncore_perf_event_update(struct intel_uncore_box *box, struct perf_event *e u64 prev_count, new_count, delta; int shift; - if (event->hw.idx >= UNCORE_PMC_IDX_FIXED) + if (uncore_pmc_freerunning(event->hw.idx)) + shift = 64 - uncore_freerunning_bits(box, event); + else if (uncore_pmc_fixed(event->hw.idx)) shift = 64 - uncore_fixed_ctr_bits(box); else shift = 64 - uncore_perf_ctr_bits(box); @@ -449,15 +451,30 @@ static int uncore_assign_events(struct intel_uncore_box *box, int assign[], int return ret ? -EINVAL : 0; } -static void uncore_pmu_event_start(struct perf_event *event, int flags) +void uncore_pmu_event_start(struct perf_event *event, int flags) { struct intel_uncore_box *box = uncore_event_to_box(event); int idx = event->hw.idx; - if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED))) + if (WARN_ON_ONCE(idx == -1 || idx >= UNCORE_PMC_IDX_MAX)) return; - if (WARN_ON_ONCE(idx == -1 || idx >= UNCORE_PMC_IDX_MAX)) + /* + * Free running counter is read-only and always active. + * Use the current counter value as start point. + * There is no overflow interrupt for free running counter. + * Use hrtimer to periodically poll the counter to avoid overflow. + */ + if (uncore_pmc_freerunning(event->hw.idx)) { + list_add_tail(&event->active_entry, &box->active_list); + local64_set(&event->hw.prev_count, + uncore_read_counter(box, event)); + if (box->n_active++ == 0) + uncore_pmu_start_hrtimer(box); + return; + } + + if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED))) return; event->hw.state = 0; @@ -474,11 +491,20 @@ static void uncore_pmu_event_start(struct perf_event *event, int flags) } } -static void uncore_pmu_event_stop(struct perf_event *event, int flags) +void uncore_pmu_event_stop(struct perf_event *event, int flags) { struct intel_uncore_box *box = uncore_event_to_box(event); struct hw_perf_event *hwc = &event->hw; + /* Cannot disable free running counter which is read-only */ + if (uncore_pmc_freerunning(hwc->idx)) { + list_del(&event->active_entry); + if (--box->n_active == 0) + uncore_pmu_cancel_hrtimer(box); + uncore_perf_event_update(box, event); + return; + } + if (__test_and_clear_bit(hwc->idx, box->active_mask)) { uncore_disable_event(box, event); box->n_active--; @@ -502,7 +528,7 @@ static void uncore_pmu_event_stop(struct perf_event *event, int flags) } } -static int uncore_pmu_event_add(struct perf_event *event, int flags) +int uncore_pmu_event_add(struct perf_event *event, int flags) { struct intel_uncore_box *box = uncore_event_to_box(event); struct hw_perf_event *hwc = &event->hw; @@ -512,6 +538,17 @@ static int uncore_pmu_event_add(struct perf_event *event, int flags) if (!box) return -ENODEV; + /* + * The free funning counter is assigned in event_init(). + * The free running counter event and free running counter + * are 1:1 mapped. It doesn't need to be tracked in event_list. + */ + if (uncore_pmc_freerunning(hwc->idx)) { + if (flags & PERF_EF_START) + uncore_pmu_event_start(event, 0); + return 0; + } + ret = n = uncore_collect_events(box, event, false); if (ret < 0) return ret; @@ -563,13 +600,21 @@ static int uncore_pmu_event_add(struct perf_event *event, int flags) return 0; } -static void uncore_pmu_event_del(struct perf_event *event, int flags) +void uncore_pmu_event_del(struct perf_event *event, int flags) { struct intel_uncore_box *box = uncore_event_to_box(event); int i; uncore_pmu_event_stop(event, PERF_EF_UPDATE); + /* + * The event for free running counter is not tracked by event_list. + * It doesn't need to force event->hw.idx = -1 to reassign the counter. + * Because the event and the free running counter are 1:1 mapped. + */ + if (uncore_pmc_freerunning(event->hw.idx)) + return; + for (i = 0; i < box->n_events; i++) { if (event == box->event_list[i]) { uncore_put_event_constraint(box, event); @@ -603,6 +648,10 @@ static int uncore_validate_group(struct intel_uncore_pmu *pmu, struct intel_uncore_box *fake_box; int ret = -EINVAL, n; + /* The free running counter is always active. */ + if (uncore_pmc_freerunning(event->hw.idx)) + return 0; + fake_box = uncore_alloc_box(pmu->type, NUMA_NO_NODE); if (!fake_box) return -ENOMEM; @@ -690,6 +739,17 @@ static int uncore_pmu_event_init(struct perf_event *event) /* fixed counters have event field hardcoded to zero */ hwc->config = 0ULL; + } else if (is_freerunning_event(event)) { + if (!check_valid_freerunning_event(box, event)) + return -EINVAL; + event->hw.idx = UNCORE_PMC_IDX_FREERUNNING; + /* + * The free running counter event and free running counter + * are always 1:1 mapped. + * The free running counter is always active. + * Assign the free running counter here. + */ + event->hw.event_base = uncore_freerunning_counter(box, event); } else { hwc->config = event->attr.config & (pmu->type->event_mask | ((u64)pmu->type->event_mask_ext << 32)); @@ -805,12 +865,10 @@ static void uncore_types_exit(struct intel_uncore_type **types) static int __init uncore_type_init(struct intel_uncore_type *type, bool setid) { struct intel_uncore_pmu *pmus; - struct attribute_group *attr_group; - struct attribute **attrs; size_t size; int i, j; - pmus = kzalloc(sizeof(*pmus) * type->num_boxes, GFP_KERNEL); + pmus = kcalloc(type->num_boxes, sizeof(*pmus), GFP_KERNEL); if (!pmus) return -ENOMEM; @@ -831,21 +889,24 @@ static int __init uncore_type_init(struct intel_uncore_type *type, bool setid) 0, type->num_counters, 0, 0); if (type->event_descs) { + struct { + struct attribute_group group; + struct attribute *attrs[]; + } *attr_group; for (i = 0; type->event_descs[i].attr.attr.name; i++); - attr_group = kzalloc(sizeof(struct attribute *) * (i + 1) + - sizeof(*attr_group), GFP_KERNEL); + attr_group = kzalloc(struct_size(attr_group, attrs, i + 1), + GFP_KERNEL); if (!attr_group) goto err; - attrs = (struct attribute **)(attr_group + 1); - attr_group->name = "events"; - attr_group->attrs = attrs; + attr_group->group.name = "events"; + attr_group->group.attrs = attr_group->attrs; for (j = 0; j < i; j++) - attrs[j] = &type->event_descs[j].attr.attr; + attr_group->attrs[j] = &type->event_descs[j].attr.attr; - type->events_group = attr_group; + type->events_group = &attr_group->group; } type->pmu_group = &uncore_pmu_attr_group; diff --git a/arch/x86/events/intel/uncore.h b/arch/x86/events/intel/uncore.h index 414dc7e7c950..c9e1e0bef3c3 100644 --- a/arch/x86/events/intel/uncore.h +++ b/arch/x86/events/intel/uncore.h @@ -12,8 +12,13 @@ #define UNCORE_FIXED_EVENT 0xff #define UNCORE_PMC_IDX_MAX_GENERIC 8 +#define UNCORE_PMC_IDX_MAX_FIXED 1 +#define UNCORE_PMC_IDX_MAX_FREERUNNING 1 #define UNCORE_PMC_IDX_FIXED UNCORE_PMC_IDX_MAX_GENERIC -#define UNCORE_PMC_IDX_MAX (UNCORE_PMC_IDX_FIXED + 1) +#define UNCORE_PMC_IDX_FREERUNNING (UNCORE_PMC_IDX_FIXED + \ + UNCORE_PMC_IDX_MAX_FIXED) +#define UNCORE_PMC_IDX_MAX (UNCORE_PMC_IDX_FREERUNNING + \ + UNCORE_PMC_IDX_MAX_FREERUNNING) #define UNCORE_PCI_DEV_FULL_DATA(dev, func, type, idx) \ ((dev << 24) | (func << 16) | (type << 8) | idx) @@ -35,6 +40,7 @@ struct intel_uncore_ops; struct intel_uncore_pmu; struct intel_uncore_box; struct uncore_event_desc; +struct freerunning_counters; struct intel_uncore_type { const char *name; @@ -42,6 +48,7 @@ struct intel_uncore_type { int num_boxes; int perf_ctr_bits; int fixed_ctr_bits; + int num_freerunning_types; unsigned perf_ctr; unsigned event_ctl; unsigned event_mask; @@ -59,6 +66,7 @@ struct intel_uncore_type { struct intel_uncore_pmu *pmus; struct intel_uncore_ops *ops; struct uncore_event_desc *event_descs; + struct freerunning_counters *freerunning; const struct attribute_group *attr_groups[4]; struct pmu *pmu; /* for custom pmu ops */ }; @@ -129,6 +137,14 @@ struct uncore_event_desc { const char *config; }; +struct freerunning_counters { + unsigned int counter_base; + unsigned int counter_offset; + unsigned int box_offset; + unsigned int num_counters; + unsigned int bits; +}; + struct pci2phy_map { struct list_head list; int segment; @@ -157,6 +173,16 @@ static ssize_t __uncore_##_var##_show(struct kobject *kobj, \ static struct kobj_attribute format_attr_##_var = \ __ATTR(_name, 0444, __uncore_##_var##_show, NULL) +static inline bool uncore_pmc_fixed(int idx) +{ + return idx == UNCORE_PMC_IDX_FIXED; +} + +static inline bool uncore_pmc_freerunning(int idx) +{ + return idx == UNCORE_PMC_IDX_FREERUNNING; +} + static inline unsigned uncore_pci_box_ctl(struct intel_uncore_box *box) { return box->pmu->type->box_ctl; @@ -214,6 +240,60 @@ static inline unsigned uncore_msr_fixed_ctr(struct intel_uncore_box *box) return box->pmu->type->fixed_ctr + uncore_msr_box_offset(box); } + +/* + * In the uncore document, there is no event-code assigned to free running + * counters. Some events need to be defined to indicate the free running + * counters. The events are encoded as event-code + umask-code. + * + * The event-code for all free running counters is 0xff, which is the same as + * the fixed counters. + * + * The umask-code is used to distinguish a fixed counter and a free running + * counter, and different types of free running counters. + * - For fixed counters, the umask-code is 0x0X. + * X indicates the index of the fixed counter, which starts from 0. + * - For free running counters, the umask-code uses the rest of the space. + * It would bare the format of 0xXY. + * X stands for the type of free running counters, which starts from 1. + * Y stands for the index of free running counters of same type, which + * starts from 0. + * + * For example, there are three types of IIO free running counters on Skylake + * server, IO CLOCKS counters, BANDWIDTH counters and UTILIZATION counters. + * The event-code for all the free running counters is 0xff. + * 'ioclk' is the first counter of IO CLOCKS. IO CLOCKS is the first type, + * which umask-code starts from 0x10. + * So 'ioclk' is encoded as event=0xff,umask=0x10 + * 'bw_in_port2' is the third counter of BANDWIDTH counters. BANDWIDTH is + * the second type, which umask-code starts from 0x20. + * So 'bw_in_port2' is encoded as event=0xff,umask=0x22 + */ +static inline unsigned int uncore_freerunning_idx(u64 config) +{ + return ((config >> 8) & 0xf); +} + +#define UNCORE_FREERUNNING_UMASK_START 0x10 + +static inline unsigned int uncore_freerunning_type(u64 config) +{ + return ((((config >> 8) - UNCORE_FREERUNNING_UMASK_START) >> 4) & 0xf); +} + +static inline +unsigned int uncore_freerunning_counter(struct intel_uncore_box *box, + struct perf_event *event) +{ + unsigned int type = uncore_freerunning_type(event->attr.config); + unsigned int idx = uncore_freerunning_idx(event->attr.config); + struct intel_uncore_pmu *pmu = box->pmu; + + return pmu->type->freerunning[type].counter_base + + pmu->type->freerunning[type].counter_offset * idx + + pmu->type->freerunning[type].box_offset * pmu->pmu_idx; +} + static inline unsigned uncore_msr_event_ctl(struct intel_uncore_box *box, int idx) { @@ -276,11 +356,52 @@ static inline int uncore_fixed_ctr_bits(struct intel_uncore_box *box) return box->pmu->type->fixed_ctr_bits; } +static inline +unsigned int uncore_freerunning_bits(struct intel_uncore_box *box, + struct perf_event *event) +{ + unsigned int type = uncore_freerunning_type(event->attr.config); + + return box->pmu->type->freerunning[type].bits; +} + +static inline int uncore_num_freerunning(struct intel_uncore_box *box, + struct perf_event *event) +{ + unsigned int type = uncore_freerunning_type(event->attr.config); + + return box->pmu->type->freerunning[type].num_counters; +} + +static inline int uncore_num_freerunning_types(struct intel_uncore_box *box, + struct perf_event *event) +{ + return box->pmu->type->num_freerunning_types; +} + +static inline bool check_valid_freerunning_event(struct intel_uncore_box *box, + struct perf_event *event) +{ + unsigned int type = uncore_freerunning_type(event->attr.config); + unsigned int idx = uncore_freerunning_idx(event->attr.config); + + return (type < uncore_num_freerunning_types(box, event)) && + (idx < uncore_num_freerunning(box, event)); +} + static inline int uncore_num_counters(struct intel_uncore_box *box) { return box->pmu->type->num_counters; } +static inline bool is_freerunning_event(struct perf_event *event) +{ + u64 cfg = event->attr.config; + + return ((cfg & UNCORE_FIXED_EVENT) == UNCORE_FIXED_EVENT) && + (((cfg >> 8) & 0xff) >= UNCORE_FREERUNNING_UMASK_START); +} + static inline void uncore_disable_box(struct intel_uncore_box *box) { if (box->pmu->type->ops->disable_box) @@ -346,6 +467,10 @@ struct intel_uncore_box *uncore_pmu_to_box(struct intel_uncore_pmu *pmu, int cpu u64 uncore_msr_read_counter(struct intel_uncore_box *box, struct perf_event *event); void uncore_pmu_start_hrtimer(struct intel_uncore_box *box); void uncore_pmu_cancel_hrtimer(struct intel_uncore_box *box); +void uncore_pmu_event_start(struct perf_event *event, int flags); +void uncore_pmu_event_stop(struct perf_event *event, int flags); +int uncore_pmu_event_add(struct perf_event *event, int flags); +void uncore_pmu_event_del(struct perf_event *event, int flags); void uncore_pmu_event_read(struct perf_event *event); void uncore_perf_event_update(struct intel_uncore_box *box, struct perf_event *event); struct event_constraint * diff --git a/arch/x86/events/intel/uncore_nhmex.c b/arch/x86/events/intel/uncore_nhmex.c index 93e7a8397cde..173e2674be6e 100644 --- a/arch/x86/events/intel/uncore_nhmex.c +++ b/arch/x86/events/intel/uncore_nhmex.c @@ -246,7 +246,7 @@ static void nhmex_uncore_msr_enable_event(struct intel_uncore_box *box, struct p { struct hw_perf_event *hwc = &event->hw; - if (hwc->idx >= UNCORE_PMC_IDX_FIXED) + if (hwc->idx == UNCORE_PMC_IDX_FIXED) wrmsrl(hwc->config_base, NHMEX_PMON_CTL_EN_BIT0); else if (box->pmu->type->event_mask & NHMEX_PMON_CTL_EN_BIT0) wrmsrl(hwc->config_base, hwc->config | NHMEX_PMON_CTL_EN_BIT22); diff --git a/arch/x86/events/intel/uncore_snb.c b/arch/x86/events/intel/uncore_snb.c index aee5e8496be4..8527c3e1038b 100644 --- a/arch/x86/events/intel/uncore_snb.c +++ b/arch/x86/events/intel/uncore_snb.c @@ -285,6 +285,15 @@ static struct uncore_event_desc snb_uncore_imc_events[] = { #define SNB_UNCORE_PCI_IMC_DATA_WRITES_BASE 0x5054 #define SNB_UNCORE_PCI_IMC_CTR_BASE SNB_UNCORE_PCI_IMC_DATA_READS_BASE +enum perf_snb_uncore_imc_freerunning_types { + SNB_PCI_UNCORE_IMC_DATA = 0, + SNB_PCI_UNCORE_IMC_FREERUNNING_TYPE_MAX, +}; + +static struct freerunning_counters snb_uncore_imc_freerunning[] = { + [SNB_PCI_UNCORE_IMC_DATA] = { SNB_UNCORE_PCI_IMC_DATA_READS_BASE, 0x4, 0x0, 2, 32 }, +}; + static struct attribute *snb_uncore_imc_formats_attr[] = { &format_attr_event.attr, NULL, @@ -341,9 +350,8 @@ static u64 snb_uncore_imc_read_counter(struct intel_uncore_box *box, struct perf } /* - * custom event_init() function because we define our own fixed, free - * running counters, so we do not want to conflict with generic uncore - * logic. Also simplifies processing + * Keep the custom event_init() function compatible with old event + * encoding for free running counters. */ static int snb_uncore_imc_event_init(struct perf_event *event) { @@ -405,11 +413,11 @@ static int snb_uncore_imc_event_init(struct perf_event *event) switch (cfg) { case SNB_UNCORE_PCI_IMC_DATA_READS: base = SNB_UNCORE_PCI_IMC_DATA_READS_BASE; - idx = UNCORE_PMC_IDX_FIXED; + idx = UNCORE_PMC_IDX_FREERUNNING; break; case SNB_UNCORE_PCI_IMC_DATA_WRITES: base = SNB_UNCORE_PCI_IMC_DATA_WRITES_BASE; - idx = UNCORE_PMC_IDX_FIXED + 1; + idx = UNCORE_PMC_IDX_FREERUNNING; break; default: return -EINVAL; @@ -430,75 +438,6 @@ static int snb_uncore_imc_hw_config(struct intel_uncore_box *box, struct perf_ev return 0; } -static void snb_uncore_imc_event_start(struct perf_event *event, int flags) -{ - struct intel_uncore_box *box = uncore_event_to_box(event); - u64 count; - - if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED))) - return; - - event->hw.state = 0; - box->n_active++; - - list_add_tail(&event->active_entry, &box->active_list); - - count = snb_uncore_imc_read_counter(box, event); - local64_set(&event->hw.prev_count, count); - - if (box->n_active == 1) - uncore_pmu_start_hrtimer(box); -} - -static void snb_uncore_imc_event_stop(struct perf_event *event, int flags) -{ - struct intel_uncore_box *box = uncore_event_to_box(event); - struct hw_perf_event *hwc = &event->hw; - - if (!(hwc->state & PERF_HES_STOPPED)) { - box->n_active--; - - WARN_ON_ONCE(hwc->state & PERF_HES_STOPPED); - hwc->state |= PERF_HES_STOPPED; - - list_del(&event->active_entry); - - if (box->n_active == 0) - uncore_pmu_cancel_hrtimer(box); - } - - if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE)) { - /* - * Drain the remaining delta count out of a event - * that we are disabling: - */ - uncore_perf_event_update(box, event); - hwc->state |= PERF_HES_UPTODATE; - } -} - -static int snb_uncore_imc_event_add(struct perf_event *event, int flags) -{ - struct intel_uncore_box *box = uncore_event_to_box(event); - struct hw_perf_event *hwc = &event->hw; - - if (!box) - return -ENODEV; - - hwc->state = PERF_HES_UPTODATE | PERF_HES_STOPPED; - if (!(flags & PERF_EF_START)) - hwc->state |= PERF_HES_ARCH; - - snb_uncore_imc_event_start(event, 0); - - return 0; -} - -static void snb_uncore_imc_event_del(struct perf_event *event, int flags) -{ - snb_uncore_imc_event_stop(event, PERF_EF_UPDATE); -} - int snb_pci2phy_map_init(int devid) { struct pci_dev *dev = NULL; @@ -530,10 +469,10 @@ int snb_pci2phy_map_init(int devid) static struct pmu snb_uncore_imc_pmu = { .task_ctx_nr = perf_invalid_context, .event_init = snb_uncore_imc_event_init, - .add = snb_uncore_imc_event_add, - .del = snb_uncore_imc_event_del, - .start = snb_uncore_imc_event_start, - .stop = snb_uncore_imc_event_stop, + .add = uncore_pmu_event_add, + .del = uncore_pmu_event_del, + .start = uncore_pmu_event_start, + .stop = uncore_pmu_event_stop, .read = uncore_pmu_event_read, }; @@ -552,12 +491,10 @@ static struct intel_uncore_type snb_uncore_imc = { .name = "imc", .num_counters = 2, .num_boxes = 1, - .fixed_ctr_bits = 32, - .fixed_ctr = SNB_UNCORE_PCI_IMC_CTR_BASE, + .num_freerunning_types = SNB_PCI_UNCORE_IMC_FREERUNNING_TYPE_MAX, + .freerunning = snb_uncore_imc_freerunning, .event_descs = snb_uncore_imc_events, .format_group = &snb_uncore_imc_format_group, - .perf_ctr = SNB_UNCORE_PCI_IMC_DATA_READS_BASE, - .event_mask = SNB_UNCORE_PCI_IMC_EVENT_MASK, .ops = &snb_uncore_imc_ops, .pmu = &snb_uncore_imc_pmu, }; diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c index 77076a102e34..87dc0263a2e1 100644 --- a/arch/x86/events/intel/uncore_snbep.c +++ b/arch/x86/events/intel/uncore_snbep.c @@ -3522,6 +3522,87 @@ static struct intel_uncore_type skx_uncore_iio = { .format_group = &skx_uncore_iio_format_group, }; +enum perf_uncore_iio_freerunning_type_id { + SKX_IIO_MSR_IOCLK = 0, + SKX_IIO_MSR_BW = 1, + SKX_IIO_MSR_UTIL = 2, + + SKX_IIO_FREERUNNING_TYPE_MAX, +}; + + +static struct freerunning_counters skx_iio_freerunning[] = { + [SKX_IIO_MSR_IOCLK] = { 0xa45, 0x1, 0x20, 1, 36 }, + [SKX_IIO_MSR_BW] = { 0xb00, 0x1, 0x10, 8, 36 }, + [SKX_IIO_MSR_UTIL] = { 0xb08, 0x1, 0x10, 8, 36 }, +}; + +static struct uncore_event_desc skx_uncore_iio_freerunning_events[] = { + /* Free-Running IO CLOCKS Counter */ + INTEL_UNCORE_EVENT_DESC(ioclk, "event=0xff,umask=0x10"), + /* Free-Running IIO BANDWIDTH Counters */ + INTEL_UNCORE_EVENT_DESC(bw_in_port0, "event=0xff,umask=0x20"), + INTEL_UNCORE_EVENT_DESC(bw_in_port0.scale, "3.814697266e-6"), + INTEL_UNCORE_EVENT_DESC(bw_in_port0.unit, "MiB"), + INTEL_UNCORE_EVENT_DESC(bw_in_port1, "event=0xff,umask=0x21"), + INTEL_UNCORE_EVENT_DESC(bw_in_port1.scale, "3.814697266e-6"), + INTEL_UNCORE_EVENT_DESC(bw_in_port1.unit, "MiB"), + INTEL_UNCORE_EVENT_DESC(bw_in_port2, "event=0xff,umask=0x22"), + INTEL_UNCORE_EVENT_DESC(bw_in_port2.scale, "3.814697266e-6"), + INTEL_UNCORE_EVENT_DESC(bw_in_port2.unit, "MiB"), + INTEL_UNCORE_EVENT_DESC(bw_in_port3, "event=0xff,umask=0x23"), + INTEL_UNCORE_EVENT_DESC(bw_in_port3.scale, "3.814697266e-6"), + INTEL_UNCORE_EVENT_DESC(bw_in_port3.unit, "MiB"), + INTEL_UNCORE_EVENT_DESC(bw_out_port0, "event=0xff,umask=0x24"), + INTEL_UNCORE_EVENT_DESC(bw_out_port0.scale, "3.814697266e-6"), + INTEL_UNCORE_EVENT_DESC(bw_out_port0.unit, "MiB"), + INTEL_UNCORE_EVENT_DESC(bw_out_port1, "event=0xff,umask=0x25"), + INTEL_UNCORE_EVENT_DESC(bw_out_port1.scale, "3.814697266e-6"), + INTEL_UNCORE_EVENT_DESC(bw_out_port1.unit, "MiB"), + INTEL_UNCORE_EVENT_DESC(bw_out_port2, "event=0xff,umask=0x26"), + INTEL_UNCORE_EVENT_DESC(bw_out_port2.scale, "3.814697266e-6"), + INTEL_UNCORE_EVENT_DESC(bw_out_port2.unit, "MiB"), + INTEL_UNCORE_EVENT_DESC(bw_out_port3, "event=0xff,umask=0x27"), + INTEL_UNCORE_EVENT_DESC(bw_out_port3.scale, "3.814697266e-6"), + INTEL_UNCORE_EVENT_DESC(bw_out_port3.unit, "MiB"), + /* Free-running IIO UTILIZATION Counters */ + INTEL_UNCORE_EVENT_DESC(util_in_port0, "event=0xff,umask=0x30"), + INTEL_UNCORE_EVENT_DESC(util_out_port0, "event=0xff,umask=0x31"), + INTEL_UNCORE_EVENT_DESC(util_in_port1, "event=0xff,umask=0x32"), + INTEL_UNCORE_EVENT_DESC(util_out_port1, "event=0xff,umask=0x33"), + INTEL_UNCORE_EVENT_DESC(util_in_port2, "event=0xff,umask=0x34"), + INTEL_UNCORE_EVENT_DESC(util_out_port2, "event=0xff,umask=0x35"), + INTEL_UNCORE_EVENT_DESC(util_in_port3, "event=0xff,umask=0x36"), + INTEL_UNCORE_EVENT_DESC(util_out_port3, "event=0xff,umask=0x37"), + { /* end: all zeroes */ }, +}; + +static struct intel_uncore_ops skx_uncore_iio_freerunning_ops = { + .read_counter = uncore_msr_read_counter, +}; + +static struct attribute *skx_uncore_iio_freerunning_formats_attr[] = { + &format_attr_event.attr, + &format_attr_umask.attr, + NULL, +}; + +static const struct attribute_group skx_uncore_iio_freerunning_format_group = { + .name = "format", + .attrs = skx_uncore_iio_freerunning_formats_attr, +}; + +static struct intel_uncore_type skx_uncore_iio_free_running = { + .name = "iio_free_running", + .num_counters = 17, + .num_boxes = 6, + .num_freerunning_types = SKX_IIO_FREERUNNING_TYPE_MAX, + .freerunning = skx_iio_freerunning, + .ops = &skx_uncore_iio_freerunning_ops, + .event_descs = skx_uncore_iio_freerunning_events, + .format_group = &skx_uncore_iio_freerunning_format_group, +}; + static struct attribute *skx_uncore_formats_attr[] = { &format_attr_event.attr, &format_attr_umask.attr, @@ -3595,6 +3676,7 @@ static struct intel_uncore_type *skx_msr_uncores[] = { &skx_uncore_ubox, &skx_uncore_chabox, &skx_uncore_iio, + &skx_uncore_iio_free_running, &skx_uncore_irp, &skx_uncore_pcu, NULL, diff --git a/arch/x86/hyperv/Makefile b/arch/x86/hyperv/Makefile index 367a8203cfcf..b173d404e3df 100644 --- a/arch/x86/hyperv/Makefile +++ b/arch/x86/hyperv/Makefile @@ -1 +1,2 @@ -obj-y := hv_init.o mmu.o +obj-y := hv_init.o mmu.o +obj-$(CONFIG_X86_64) += hv_apic.o diff --git a/arch/x86/hyperv/hv_apic.c b/arch/x86/hyperv/hv_apic.c new file mode 100644 index 000000000000..f68855499391 --- /dev/null +++ b/arch/x86/hyperv/hv_apic.c @@ -0,0 +1,256 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* + * Hyper-V specific APIC code. + * + * Copyright (C) 2018, Microsoft, Inc. + * + * Author : K. Y. Srinivasan + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, GOOD TITLE or + * NON INFRINGEMENT. See the GNU General Public License for more + * details. + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static struct apic orig_apic; + +static u64 hv_apic_icr_read(void) +{ + u64 reg_val; + + rdmsrl(HV_X64_MSR_ICR, reg_val); + return reg_val; +} + +static void hv_apic_icr_write(u32 low, u32 id) +{ + u64 reg_val; + + reg_val = SET_APIC_DEST_FIELD(id); + reg_val = reg_val << 32; + reg_val |= low; + + wrmsrl(HV_X64_MSR_ICR, reg_val); +} + +static u32 hv_apic_read(u32 reg) +{ + u32 reg_val, hi; + + switch (reg) { + case APIC_EOI: + rdmsr(HV_X64_MSR_EOI, reg_val, hi); + return reg_val; + case APIC_TASKPRI: + rdmsr(HV_X64_MSR_TPR, reg_val, hi); + return reg_val; + + default: + return native_apic_mem_read(reg); + } +} + +static void hv_apic_write(u32 reg, u32 val) +{ + switch (reg) { + case APIC_EOI: + wrmsr(HV_X64_MSR_EOI, val, 0); + break; + case APIC_TASKPRI: + wrmsr(HV_X64_MSR_TPR, val, 0); + break; + default: + native_apic_mem_write(reg, val); + } +} + +static void hv_apic_eoi_write(u32 reg, u32 val) +{ + wrmsr(HV_X64_MSR_EOI, val, 0); +} + +/* + * IPI implementation on Hyper-V. + */ +static bool __send_ipi_mask_ex(const struct cpumask *mask, int vector) +{ + struct ipi_arg_ex **arg; + struct ipi_arg_ex *ipi_arg; + unsigned long flags; + int nr_bank = 0; + int ret = 1; + + local_irq_save(flags); + arg = (struct ipi_arg_ex **)this_cpu_ptr(hyperv_pcpu_input_arg); + + ipi_arg = *arg; + if (unlikely(!ipi_arg)) + goto ipi_mask_ex_done; + + ipi_arg->vector = vector; + ipi_arg->reserved = 0; + ipi_arg->vp_set.valid_bank_mask = 0; + + if (!cpumask_equal(mask, cpu_present_mask)) { + ipi_arg->vp_set.format = HV_GENERIC_SET_SPARSE_4K; + nr_bank = cpumask_to_vpset(&(ipi_arg->vp_set), mask); + } + if (!nr_bank) + ipi_arg->vp_set.format = HV_GENERIC_SET_ALL; + + ret = hv_do_rep_hypercall(HVCALL_SEND_IPI_EX, 0, nr_bank, + ipi_arg, NULL); + +ipi_mask_ex_done: + local_irq_restore(flags); + return ((ret == 0) ? true : false); +} + +static bool __send_ipi_mask(const struct cpumask *mask, int vector) +{ + int cur_cpu, vcpu; + struct ipi_arg_non_ex **arg; + struct ipi_arg_non_ex *ipi_arg; + int ret = 1; + unsigned long flags; + + if (cpumask_empty(mask)) + return true; + + if (!hv_hypercall_pg) + return false; + + if ((vector < HV_IPI_LOW_VECTOR) || (vector > HV_IPI_HIGH_VECTOR)) + return false; + + if ((ms_hyperv.hints & HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED)) + return __send_ipi_mask_ex(mask, vector); + + local_irq_save(flags); + arg = (struct ipi_arg_non_ex **)this_cpu_ptr(hyperv_pcpu_input_arg); + + ipi_arg = *arg; + if (unlikely(!ipi_arg)) + goto ipi_mask_done; + + ipi_arg->vector = vector; + ipi_arg->reserved = 0; + ipi_arg->cpu_mask = 0; + + for_each_cpu(cur_cpu, mask) { + vcpu = hv_cpu_number_to_vp_number(cur_cpu); + /* + * This particular version of the IPI hypercall can + * only target upto 64 CPUs. + */ + if (vcpu >= 64) + goto ipi_mask_done; + + __set_bit(vcpu, (unsigned long *)&ipi_arg->cpu_mask); + } + + ret = hv_do_hypercall(HVCALL_SEND_IPI, ipi_arg, NULL); + +ipi_mask_done: + local_irq_restore(flags); + return ((ret == 0) ? true : false); +} + +static bool __send_ipi_one(int cpu, int vector) +{ + struct cpumask mask = CPU_MASK_NONE; + + cpumask_set_cpu(cpu, &mask); + return __send_ipi_mask(&mask, vector); +} + +static void hv_send_ipi(int cpu, int vector) +{ + if (!__send_ipi_one(cpu, vector)) + orig_apic.send_IPI(cpu, vector); +} + +static void hv_send_ipi_mask(const struct cpumask *mask, int vector) +{ + if (!__send_ipi_mask(mask, vector)) + orig_apic.send_IPI_mask(mask, vector); +} + +static void hv_send_ipi_mask_allbutself(const struct cpumask *mask, int vector) +{ + unsigned int this_cpu = smp_processor_id(); + struct cpumask new_mask; + const struct cpumask *local_mask; + + cpumask_copy(&new_mask, mask); + cpumask_clear_cpu(this_cpu, &new_mask); + local_mask = &new_mask; + if (!__send_ipi_mask(local_mask, vector)) + orig_apic.send_IPI_mask_allbutself(mask, vector); +} + +static void hv_send_ipi_allbutself(int vector) +{ + hv_send_ipi_mask_allbutself(cpu_online_mask, vector); +} + +static void hv_send_ipi_all(int vector) +{ + if (!__send_ipi_mask(cpu_online_mask, vector)) + orig_apic.send_IPI_all(vector); +} + +static void hv_send_ipi_self(int vector) +{ + if (!__send_ipi_one(smp_processor_id(), vector)) + orig_apic.send_IPI_self(vector); +} + +void __init hv_apic_init(void) +{ + if (ms_hyperv.hints & HV_X64_CLUSTER_IPI_RECOMMENDED) { + if ((ms_hyperv.hints & HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED)) + pr_info("Hyper-V: Using ext hypercalls for IPI\n"); + else + pr_info("Hyper-V: Using IPI hypercalls\n"); + /* + * Set the IPI entry points. + */ + orig_apic = *apic; + + apic->send_IPI = hv_send_ipi; + apic->send_IPI_mask = hv_send_ipi_mask; + apic->send_IPI_mask_allbutself = hv_send_ipi_mask_allbutself; + apic->send_IPI_allbutself = hv_send_ipi_allbutself; + apic->send_IPI_all = hv_send_ipi_all; + apic->send_IPI_self = hv_send_ipi_self; + } + + if (ms_hyperv.hints & HV_X64_APIC_ACCESS_RECOMMENDED) { + pr_info("Hyper-V: Using MSR based APIC access\n"); + apic_set_eoi_write(hv_apic_eoi_write); + apic->read = hv_apic_read; + apic->write = hv_apic_write; + apic->icr_write = hv_apic_icr_write; + apic->icr_read = hv_apic_icr_read; + } +} diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c index cfecc2272f2d..4c431e1c1eff 100644 --- a/arch/x86/hyperv/hv_init.c +++ b/arch/x86/hyperv/hv_init.c @@ -91,12 +91,19 @@ EXPORT_SYMBOL_GPL(hv_vp_index); struct hv_vp_assist_page **hv_vp_assist_page; EXPORT_SYMBOL_GPL(hv_vp_assist_page); +void __percpu **hyperv_pcpu_input_arg; +EXPORT_SYMBOL_GPL(hyperv_pcpu_input_arg); + u32 hv_max_vp_index; static int hv_cpu_init(unsigned int cpu) { u64 msr_vp_index; struct hv_vp_assist_page **hvp = &hv_vp_assist_page[smp_processor_id()]; + void **input_arg; + + input_arg = (void **)this_cpu_ptr(hyperv_pcpu_input_arg); + *input_arg = page_address(alloc_page(GFP_KERNEL)); hv_get_vp_index(msr_vp_index); @@ -217,6 +224,16 @@ static int hv_cpu_die(unsigned int cpu) { struct hv_reenlightenment_control re_ctrl; unsigned int new_cpu; + unsigned long flags; + void **input_arg; + void *input_pg = NULL; + + local_irq_save(flags); + input_arg = (void **)this_cpu_ptr(hyperv_pcpu_input_arg); + input_pg = *input_arg; + *input_arg = NULL; + local_irq_restore(flags); + free_page((unsigned long)input_pg); if (hv_vp_assist_page && hv_vp_assist_page[cpu]) wrmsrl(HV_X64_MSR_VP_ASSIST_PAGE, 0); @@ -242,8 +259,9 @@ static int hv_cpu_die(unsigned int cpu) * * 1. Setup the hypercall page. * 2. Register Hyper-V specific clocksource. + * 3. Setup Hyper-V specific APIC entry points. */ -void hyperv_init(void) +void __init hyperv_init(void) { u64 guest_id, required_msrs; union hv_x64_msr_hypercall_contents hypercall_msr; @@ -259,6 +277,16 @@ void hyperv_init(void) if ((ms_hyperv.features & required_msrs) != required_msrs) return; + /* + * Allocate the per-CPU state for the hypercall input arg. + * If this allocation fails, we will not be able to setup + * (per-CPU) hypercall input page and thus this failure is + * fatal on Hyper-V. + */ + hyperv_pcpu_input_arg = alloc_percpu(void *); + + BUG_ON(hyperv_pcpu_input_arg == NULL); + /* Allocate percpu VP index */ hv_vp_index = kmalloc_array(num_possible_cpus(), sizeof(*hv_vp_index), GFP_KERNEL); @@ -296,7 +324,7 @@ void hyperv_init(void) hypercall_msr.guest_physical_address = vmalloc_to_pfn(hv_hypercall_pg); wrmsrl(HV_X64_MSR_HYPERCALL, hypercall_msr.as_uint64); - hyper_alloc_mmu(); + hv_apic_init(); /* * Register Hyper-V specific clocksource. diff --git a/arch/x86/hyperv/mmu.c b/arch/x86/hyperv/mmu.c index 56c9ebac946f..de27615c51ea 100644 --- a/arch/x86/hyperv/mmu.c +++ b/arch/x86/hyperv/mmu.c @@ -13,32 +13,9 @@ #define CREATE_TRACE_POINTS #include -/* HvFlushVirtualAddressSpace, HvFlushVirtualAddressList hypercalls */ -struct hv_flush_pcpu { - u64 address_space; - u64 flags; - u64 processor_mask; - u64 gva_list[]; -}; - -/* HvFlushVirtualAddressSpaceEx, HvFlushVirtualAddressListEx hypercalls */ -struct hv_flush_pcpu_ex { - u64 address_space; - u64 flags; - struct { - u64 format; - u64 valid_bank_mask; - u64 bank_contents[]; - } hv_vp_set; - u64 gva_list[]; -}; - /* Each gva in gva_list encodes up to 4096 pages to flush */ #define HV_TLB_FLUSH_UNIT (4096 * PAGE_SIZE) -static struct hv_flush_pcpu __percpu **pcpu_flush; - -static struct hv_flush_pcpu_ex __percpu **pcpu_flush_ex; /* * Fills in gva_list starting from offset. Returns the number of items added. @@ -70,53 +47,18 @@ static inline int fill_gva_list(u64 gva_list[], int offset, return gva_n - offset; } -/* Return the number of banks in the resulting vp_set */ -static inline int cpumask_to_vp_set(struct hv_flush_pcpu_ex *flush, - const struct cpumask *cpus) -{ - int cpu, vcpu, vcpu_bank, vcpu_offset, nr_bank = 1; - - /* valid_bank_mask can represent up to 64 banks */ - if (hv_max_vp_index / 64 >= 64) - return 0; - - /* - * Clear all banks up to the maximum possible bank as hv_flush_pcpu_ex - * structs are not cleared between calls, we risk flushing unneeded - * vCPUs otherwise. - */ - for (vcpu_bank = 0; vcpu_bank <= hv_max_vp_index / 64; vcpu_bank++) - flush->hv_vp_set.bank_contents[vcpu_bank] = 0; - - /* - * Some banks may end up being empty but this is acceptable. - */ - for_each_cpu(cpu, cpus) { - vcpu = hv_cpu_number_to_vp_number(cpu); - vcpu_bank = vcpu / 64; - vcpu_offset = vcpu % 64; - __set_bit(vcpu_offset, (unsigned long *) - &flush->hv_vp_set.bank_contents[vcpu_bank]); - if (vcpu_bank >= nr_bank) - nr_bank = vcpu_bank + 1; - } - flush->hv_vp_set.valid_bank_mask = GENMASK_ULL(nr_bank - 1, 0); - - return nr_bank; -} - static void hyperv_flush_tlb_others(const struct cpumask *cpus, const struct flush_tlb_info *info) { int cpu, vcpu, gva_n, max_gvas; - struct hv_flush_pcpu **flush_pcpu; - struct hv_flush_pcpu *flush; + struct hv_tlb_flush **flush_pcpu; + struct hv_tlb_flush *flush; u64 status = U64_MAX; unsigned long flags; trace_hyperv_mmu_flush_tlb_others(cpus, info); - if (!pcpu_flush || !hv_hypercall_pg) + if (!hv_hypercall_pg) goto do_native; if (cpumask_empty(cpus)) @@ -124,10 +66,8 @@ static void hyperv_flush_tlb_others(const struct cpumask *cpus, local_irq_save(flags); - flush_pcpu = this_cpu_ptr(pcpu_flush); - - if (unlikely(!*flush_pcpu)) - *flush_pcpu = page_address(alloc_page(GFP_ATOMIC)); + flush_pcpu = (struct hv_tlb_flush **) + this_cpu_ptr(hyperv_pcpu_input_arg); flush = *flush_pcpu; @@ -196,14 +136,14 @@ static void hyperv_flush_tlb_others_ex(const struct cpumask *cpus, const struct flush_tlb_info *info) { int nr_bank = 0, max_gvas, gva_n; - struct hv_flush_pcpu_ex **flush_pcpu; - struct hv_flush_pcpu_ex *flush; + struct hv_tlb_flush_ex **flush_pcpu; + struct hv_tlb_flush_ex *flush; u64 status = U64_MAX; unsigned long flags; trace_hyperv_mmu_flush_tlb_others(cpus, info); - if (!pcpu_flush_ex || !hv_hypercall_pg) + if (!hv_hypercall_pg) goto do_native; if (cpumask_empty(cpus)) @@ -211,10 +151,8 @@ static void hyperv_flush_tlb_others_ex(const struct cpumask *cpus, local_irq_save(flags); - flush_pcpu = this_cpu_ptr(pcpu_flush_ex); - - if (unlikely(!*flush_pcpu)) - *flush_pcpu = page_address(alloc_page(GFP_ATOMIC)); + flush_pcpu = (struct hv_tlb_flush_ex **) + this_cpu_ptr(hyperv_pcpu_input_arg); flush = *flush_pcpu; @@ -239,8 +177,8 @@ static void hyperv_flush_tlb_others_ex(const struct cpumask *cpus, flush->hv_vp_set.valid_bank_mask = 0; if (!cpumask_equal(cpus, cpu_present_mask)) { - flush->hv_vp_set.format = HV_GENERIC_SET_SPARCE_4K; - nr_bank = cpumask_to_vp_set(flush, cpus); + flush->hv_vp_set.format = HV_GENERIC_SET_SPARSE_4K; + nr_bank = cpumask_to_vpset(&(flush->hv_vp_set), cpus); } if (!nr_bank) { @@ -296,14 +234,3 @@ void hyperv_setup_mmu_ops(void) pv_mmu_ops.flush_tlb_others = hyperv_flush_tlb_others_ex; } } - -void hyper_alloc_mmu(void) -{ - if (!(ms_hyperv.hints & HV_X64_REMOTE_TLB_FLUSH_RECOMMENDED)) - return; - - if (!(ms_hyperv.hints & HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED)) - pcpu_flush = alloc_percpu(struct hv_flush_pcpu *); - else - pcpu_flush_ex = alloc_percpu(struct hv_flush_pcpu_ex *); -} diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h index 08acd954f00e..74a9e06b6cfd 100644 --- a/arch/x86/include/asm/apic.h +++ b/arch/x86/include/asm/apic.h @@ -436,6 +436,8 @@ static inline void apic_set_eoi_write(void (*eoi_write)(u32 reg, u32 v)) {} #endif /* CONFIG_X86_LOCAL_APIC */ +extern void apic_ack_irq(struct irq_data *data); + static inline void ack_APIC_irq(void) { /* diff --git a/arch/x86/include/asm/cacheinfo.h b/arch/x86/include/asm/cacheinfo.h new file mode 100644 index 000000000000..e958e28f7ab5 --- /dev/null +++ b/arch/x86/include/asm/cacheinfo.h @@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_CACHEINFO_H +#define _ASM_X86_CACHEINFO_H + +void cacheinfo_amd_init_llc_id(struct cpuinfo_x86 *c, int cpu, u8 node_id); + +#endif /* _ASM_X86_CACHEINFO_H */ diff --git a/arch/x86/include/asm/compat.h b/arch/x86/include/asm/compat.h index e1c8dab86670..fb97cf7c4137 100644 --- a/arch/x86/include/asm/compat.h +++ b/arch/x86/include/asm/compat.h @@ -17,7 +17,6 @@ typedef u32 compat_size_t; typedef s32 compat_ssize_t; -typedef s32 compat_time_t; typedef s32 compat_clock_t; typedef s32 compat_pid_t; typedef u16 __compat_uid_t; @@ -46,16 +45,6 @@ typedef u32 compat_u32; typedef u64 __attribute__((aligned(4))) compat_u64; typedef u32 compat_uptr_t; -struct compat_timespec { - compat_time_t tv_sec; - s32 tv_nsec; -}; - -struct compat_timeval { - compat_time_t tv_sec; - s32 tv_usec; -}; - struct compat_stat { compat_dev_t st_dev; u16 __pad1; @@ -145,10 +134,10 @@ struct compat_ipc64_perm { struct compat_semid64_ds { struct compat_ipc64_perm sem_perm; - compat_time_t sem_otime; - compat_ulong_t __unused1; - compat_time_t sem_ctime; - compat_ulong_t __unused2; + compat_ulong_t sem_otime; + compat_ulong_t sem_otime_high; + compat_ulong_t sem_ctime; + compat_ulong_t sem_ctime_high; compat_ulong_t sem_nsems; compat_ulong_t __unused3; compat_ulong_t __unused4; @@ -156,12 +145,12 @@ struct compat_semid64_ds { struct compat_msqid64_ds { struct compat_ipc64_perm msg_perm; - compat_time_t msg_stime; - compat_ulong_t __unused1; - compat_time_t msg_rtime; - compat_ulong_t __unused2; - compat_time_t msg_ctime; - compat_ulong_t __unused3; + compat_ulong_t msg_stime; + compat_ulong_t msg_stime_high; + compat_ulong_t msg_rtime; + compat_ulong_t msg_rtime_high; + compat_ulong_t msg_ctime; + compat_ulong_t msg_ctime_high; compat_ulong_t msg_cbytes; compat_ulong_t msg_qnum; compat_ulong_t msg_qbytes; @@ -174,12 +163,12 @@ struct compat_msqid64_ds { struct compat_shmid64_ds { struct compat_ipc64_perm shm_perm; compat_size_t shm_segsz; - compat_time_t shm_atime; - compat_ulong_t __unused1; - compat_time_t shm_dtime; - compat_ulong_t __unused2; - compat_time_t shm_ctime; - compat_ulong_t __unused3; + compat_ulong_t shm_atime; + compat_ulong_t shm_atime_high; + compat_ulong_t shm_dtime; + compat_ulong_t shm_dtime_high; + compat_ulong_t shm_ctime; + compat_ulong_t shm_ctime_high; compat_pid_t shm_cpid; compat_pid_t shm_lpid; compat_ulong_t shm_nattch; diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index fb00a2fca990..5701f5cecd31 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -282,7 +282,9 @@ #define X86_FEATURE_AMD_IBPB (13*32+12) /* "" Indirect Branch Prediction Barrier */ #define X86_FEATURE_AMD_IBRS (13*32+14) /* "" Indirect Branch Restricted Speculation */ #define X86_FEATURE_AMD_STIBP (13*32+15) /* "" Single Thread Indirect Branch Predictors */ +#define X86_FEATURE_AMD_SSBD (13*32+24) /* "" Speculative Store Bypass Disable */ #define X86_FEATURE_VIRT_SSBD (13*32+25) /* Virtualized Speculative Store Bypass Disable */ +#define X86_FEATURE_AMD_SSB_NO (13*32+26) /* "" Speculative Store Bypass is fixed in hardware. */ /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 14 */ #define X86_FEATURE_DTHERM (14*32+ 0) /* Digital Thermal Sensor */ diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h index 89ce4bfd241f..ce4d176b3d13 100644 --- a/arch/x86/include/asm/dma-mapping.h +++ b/arch/x86/include/asm/dma-mapping.h @@ -30,10 +30,7 @@ static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus) return dma_ops; } -int arch_dma_supported(struct device *dev, u64 mask); -#define arch_dma_supported arch_dma_supported - -bool arch_dma_alloc_attrs(struct device **dev, gfp_t *gfp); +bool arch_dma_alloc_attrs(struct device **dev); #define arch_dma_alloc_attrs arch_dma_alloc_attrs #endif diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index cc8f8fcf9b4a..c18ed65287d5 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -63,7 +63,7 @@ static inline bool arch_syscall_match_sym_name(const char *sym, const char *name #ifndef COMPILE_OFFSETS #if defined(CONFIG_FTRACE_SYSCALLS) && defined(CONFIG_IA32_EMULATION) -#include +#include /* * Because ia32 syscalls do not map to x86_64 syscall numbers diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index 5ea2afd4c871..740a428acf1e 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -50,14 +50,6 @@ DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat); #define inc_irq_stat(member) this_cpu_inc(irq_stat.member) -#define local_softirq_pending() this_cpu_read(irq_stat.__softirq_pending) - -#define __ARCH_SET_SOFTIRQ_PENDING - -#define set_softirq_pending(x) \ - this_cpu_write(irq_stat.__softirq_pending, (x)) -#define or_softirq_pending(x) this_cpu_or(irq_stat.__softirq_pending, (x)) - extern void ack_bad_irq(unsigned int irq); extern u64 arch_irq_stat_cpu(unsigned int cpu); diff --git a/arch/x86/include/asm/hyperv-tlfs.h b/arch/x86/include/asm/hyperv-tlfs.h index 416cb0e0c496..b8c89265baf0 100644 --- a/arch/x86/include/asm/hyperv-tlfs.h +++ b/arch/x86/include/asm/hyperv-tlfs.h @@ -164,6 +164,11 @@ */ #define HV_X64_DEPRECATING_AEOI_RECOMMENDED (1 << 9) +/* + * Recommend using cluster IPI hypercalls. + */ +#define HV_X64_CLUSTER_IPI_RECOMMENDED (1 << 10) + /* Recommend using the newer ExProcessorMasks interface */ #define HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED (1 << 11) @@ -303,6 +308,9 @@ struct ms_hyperv_tsc_page { /* TSC emulation after migration */ #define HV_X64_MSR_REENLIGHTENMENT_CONTROL 0x40000106 +/* Nested features (CPUID 0x4000000A) EAX */ +#define HV_X64_NESTED_MSR_BITMAP BIT(19) + struct hv_reenlightenment_control { __u64 vector:8; __u64 reserved1:8; @@ -329,12 +337,17 @@ struct hv_tsc_emulation_status { #define HV_X64_MSR_HYPERCALL_PAGE_ADDRESS_MASK \ (~((1ull << HV_X64_MSR_HYPERCALL_PAGE_ADDRESS_SHIFT) - 1)) +#define HV_IPI_LOW_VECTOR 0x10 +#define HV_IPI_HIGH_VECTOR 0xff + /* Declare the various hypercall operations. */ #define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE 0x0002 #define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST 0x0003 #define HVCALL_NOTIFY_LONG_SPIN_WAIT 0x0008 +#define HVCALL_SEND_IPI 0x000b #define HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX 0x0013 #define HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX 0x0014 +#define HVCALL_SEND_IPI_EX 0x0015 #define HVCALL_POST_MESSAGE 0x005c #define HVCALL_SIGNAL_EVENT 0x005d @@ -360,7 +373,7 @@ struct hv_tsc_emulation_status { #define HV_FLUSH_USE_EXTENDED_RANGE_FORMAT BIT(3) enum HV_GENERIC_SET_FORMAT { - HV_GENERIC_SET_SPARCE_4K, + HV_GENERIC_SET_SPARSE_4K, HV_GENERIC_SET_ALL, }; @@ -668,7 +681,11 @@ struct hv_enlightened_vmcs { u32 hv_clean_fields; u32 hv_padding_32; u32 hv_synthetic_controls; - u32 hv_enlightenments_control; + struct { + u32 nested_flush_hypercall:1; + u32 msr_bitmap:1; + u32 reserved:30; + } hv_enlightenments_control; u32 hv_vp_id; u64 hv_vm_id; @@ -706,4 +723,38 @@ struct hv_enlightened_vmcs { #define HV_STIMER_AUTOENABLE (1ULL << 3) #define HV_STIMER_SINT(config) (__u8)(((config) >> 16) & 0x0F) +struct ipi_arg_non_ex { + u32 vector; + u32 reserved; + u64 cpu_mask; +}; + +struct hv_vpset { + u64 format; + u64 valid_bank_mask; + u64 bank_contents[]; +}; + +struct ipi_arg_ex { + u32 vector; + u32 reserved; + struct hv_vpset vp_set; +}; + +/* HvFlushVirtualAddressSpace, HvFlushVirtualAddressList hypercalls */ +struct hv_tlb_flush { + u64 address_space; + u64 flags; + u64 processor_mask; + u64 gva_list[]; +}; + +/* HvFlushVirtualAddressSpaceEx, HvFlushVirtualAddressListEx hypercalls */ +struct hv_tlb_flush_ex { + u64 address_space; + u64 flags; + struct hv_vpset hv_vp_set; + u64 gva_list[]; +}; + #endif diff --git a/arch/x86/include/asm/intel_mid_vrtc.h b/arch/x86/include/asm/intel_mid_vrtc.h index 35555016b1be..0b44b1abe4d9 100644 --- a/arch/x86/include/asm/intel_mid_vrtc.h +++ b/arch/x86/include/asm/intel_mid_vrtc.h @@ -4,7 +4,7 @@ extern unsigned char vrtc_cmos_read(unsigned char reg); extern void vrtc_cmos_write(unsigned char val, unsigned char reg); -extern void vrtc_get_time(struct timespec *now); -extern int vrtc_set_mmss(const struct timespec *now); +extern void vrtc_get_time(struct timespec64 *now); +extern int vrtc_set_mmss(const struct timespec64 *now); #endif diff --git a/arch/x86/include/asm/io.h b/arch/x86/include/asm/io.h index f6e5b9375d8c..6de64840dd22 100644 --- a/arch/x86/include/asm/io.h +++ b/arch/x86/include/asm/io.h @@ -94,10 +94,10 @@ build_mmio_write(__writel, "l", unsigned int, "r", ) #ifdef CONFIG_X86_64 -build_mmio_read(readq, "q", unsigned long, "=r", :"memory") -build_mmio_read(__readq, "q", unsigned long, "=r", ) -build_mmio_write(writeq, "q", unsigned long, "r", :"memory") -build_mmio_write(__writeq, "q", unsigned long, "r", ) +build_mmio_read(readq, "q", u64, "=r", :"memory") +build_mmio_read(__readq, "q", u64, "=r", ) +build_mmio_write(writeq, "q", u64, "r", :"memory") +build_mmio_write(__writeq, "q", u64, "r", ) #define readq_relaxed(a) __readq(a) #define writeq_relaxed(v, a) __writeq(v, a) diff --git a/arch/x86/include/asm/kvm_emulate.h b/arch/x86/include/asm/kvm_emulate.h index b24b1c8b3979..0f82cd91cd3c 100644 --- a/arch/x86/include/asm/kvm_emulate.h +++ b/arch/x86/include/asm/kvm_emulate.h @@ -107,11 +107,12 @@ struct x86_emulate_ops { * @addr: [IN ] Linear address from which to read. * @val: [OUT] Value read from memory, zero-extended to 'u_long'. * @bytes: [IN ] Number of bytes to read from memory. + * @system:[IN ] Whether the access is forced to be at CPL0. */ int (*read_std)(struct x86_emulate_ctxt *ctxt, unsigned long addr, void *val, unsigned int bytes, - struct x86_exception *fault); + struct x86_exception *fault, bool system); /* * read_phys: Read bytes of standard (non-emulated/special) memory. @@ -129,10 +130,11 @@ struct x86_emulate_ops { * @addr: [IN ] Linear address to which to write. * @val: [OUT] Value write to memory, zero-extended to 'u_long'. * @bytes: [IN ] Number of bytes to write to memory. + * @system:[IN ] Whether the access is forced to be at CPL0. */ int (*write_std)(struct x86_emulate_ctxt *ctxt, unsigned long addr, void *val, unsigned int bytes, - struct x86_exception *fault); + struct x86_exception *fault, bool system); /* * fetch: Read bytes of standard (non-emulated/special) memory. * Used for instruction fetch. diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index f4b2588865e9..c13cd28d9d1b 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -258,7 +258,8 @@ union kvm_mmu_page_role { unsigned smep_andnot_wp:1; unsigned smap_andnot_wp:1; unsigned ad_disabled:1; - unsigned :7; + unsigned guest_mode:1; + unsigned :6; /* * This is left at the top of the word so that @@ -476,6 +477,7 @@ struct kvm_vcpu_hv { struct kvm_hyperv_exit exit; struct kvm_vcpu_hv_stimer stimer[HV_SYNIC_STIMER_COUNT]; DECLARE_BITMAP(stimer_pending_bitmap, HV_SYNIC_STIMER_COUNT); + cpumask_t tlb_lush; }; struct kvm_vcpu_arch { @@ -995,7 +997,7 @@ struct kvm_x86_ops { void (*hwapic_irr_update)(struct kvm_vcpu *vcpu, int max_irr); void (*hwapic_isr_update)(struct kvm_vcpu *vcpu, int isr); void (*load_eoi_exitmap)(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitmap); - void (*set_virtual_x2apic_mode)(struct kvm_vcpu *vcpu, bool set); + void (*set_virtual_apic_mode)(struct kvm_vcpu *vcpu); void (*set_apic_access_page_addr)(struct kvm_vcpu *vcpu, hpa_t hpa); void (*deliver_posted_interrupt)(struct kvm_vcpu *vcpu, int vector); int (*sync_pir_to_irr)(struct kvm_vcpu *vcpu); @@ -1277,6 +1279,7 @@ void __kvm_mmu_free_some_pages(struct kvm_vcpu *vcpu); int kvm_mmu_load(struct kvm_vcpu *vcpu); void kvm_mmu_unload(struct kvm_vcpu *vcpu); void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu); +void kvm_mmu_free_roots(struct kvm_vcpu *vcpu); gpa_t translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, u32 access, struct x86_exception *exception); gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, diff --git a/arch/x86/include/asm/mc146818rtc.h b/arch/x86/include/asm/mc146818rtc.h index 1775a32f7ea6..97198001e567 100644 --- a/arch/x86/include/asm/mc146818rtc.h +++ b/arch/x86/include/asm/mc146818rtc.h @@ -95,8 +95,8 @@ static inline unsigned char current_lock_cmos_reg(void) unsigned char rtc_cmos_read(unsigned char addr); void rtc_cmos_write(unsigned char val, unsigned char addr); -extern int mach_set_rtc_mmss(const struct timespec *now); -extern void mach_get_cmos_time(struct timespec *now); +extern int mach_set_rtc_mmss(const struct timespec64 *now); +extern void mach_get_cmos_time(struct timespec64 *now); #define RTC_IRQ 8 diff --git a/arch/x86/include/asm/mcsafe_test.h b/arch/x86/include/asm/mcsafe_test.h new file mode 100644 index 000000000000..eb59804b6201 --- /dev/null +++ b/arch/x86/include/asm/mcsafe_test.h @@ -0,0 +1,75 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _MCSAFE_TEST_H_ +#define _MCSAFE_TEST_H_ + +#ifndef __ASSEMBLY__ +#ifdef CONFIG_MCSAFE_TEST +extern unsigned long mcsafe_test_src; +extern unsigned long mcsafe_test_dst; + +static inline void mcsafe_inject_src(void *addr) +{ + if (addr) + mcsafe_test_src = (unsigned long) addr; + else + mcsafe_test_src = ~0UL; +} + +static inline void mcsafe_inject_dst(void *addr) +{ + if (addr) + mcsafe_test_dst = (unsigned long) addr; + else + mcsafe_test_dst = ~0UL; +} +#else /* CONFIG_MCSAFE_TEST */ +static inline void mcsafe_inject_src(void *addr) +{ +} + +static inline void mcsafe_inject_dst(void *addr) +{ +} +#endif /* CONFIG_MCSAFE_TEST */ + +#else /* __ASSEMBLY__ */ +#include + +#ifdef CONFIG_MCSAFE_TEST +.macro MCSAFE_TEST_CTL + .pushsection .data + .align 8 + .globl mcsafe_test_src + mcsafe_test_src: + .quad 0 + EXPORT_SYMBOL_GPL(mcsafe_test_src) + .globl mcsafe_test_dst + mcsafe_test_dst: + .quad 0 + EXPORT_SYMBOL_GPL(mcsafe_test_dst) + .popsection +.endm + +.macro MCSAFE_TEST_SRC reg count target + leaq \count(\reg), %r9 + cmp mcsafe_test_src, %r9 + ja \target +.endm + +.macro MCSAFE_TEST_DST reg count target + leaq \count(\reg), %r9 + cmp mcsafe_test_dst, %r9 + ja \target +.endm +#else +.macro MCSAFE_TEST_CTL +.endm + +.macro MCSAFE_TEST_SRC reg count target +.endm + +.macro MCSAFE_TEST_DST reg count target +.endm +#endif /* CONFIG_MCSAFE_TEST */ +#endif /* __ASSEMBLY__ */ +#endif /* _MCSAFE_TEST_H_ */ diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index cf9911b5a53c..bbc796eb0a3b 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -288,21 +288,6 @@ static inline void arch_unmap(struct mm_struct *mm, struct vm_area_struct *vma, mpx_notify_unmap(mm, vma, start, end); } -#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS -static inline int vma_pkey(struct vm_area_struct *vma) -{ - unsigned long vma_pkey_mask = VM_PKEY_BIT0 | VM_PKEY_BIT1 | - VM_PKEY_BIT2 | VM_PKEY_BIT3; - - return (vma->vm_flags & vma_pkey_mask) >> VM_PKEY_SHIFT; -} -#else -static inline int vma_pkey(struct vm_area_struct *vma) -{ - return 0; -} -#endif - /* * We only want to enforce protection keys on the current process * because we effectively have no access to PKRU for other diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h index b90e79610cf7..3cd14311edfa 100644 --- a/arch/x86/include/asm/mshyperv.h +++ b/arch/x86/include/asm/mshyperv.h @@ -122,6 +122,7 @@ static inline void hv_disable_stimer0_percpu_irq(int irq) {} #if IS_ENABLED(CONFIG_HYPERV) extern struct clocksource *hyperv_cs; extern void *hv_hypercall_pg; +extern void __percpu **hyperv_pcpu_input_arg; static inline u64 hv_do_hypercall(u64 control, void *input, void *output) { @@ -258,9 +259,41 @@ static inline int hv_cpu_number_to_vp_number(int cpu_number) return hv_vp_index[cpu_number]; } -void hyperv_init(void); +static inline int cpumask_to_vpset(struct hv_vpset *vpset, + const struct cpumask *cpus) +{ + int cpu, vcpu, vcpu_bank, vcpu_offset, nr_bank = 1; + + /* valid_bank_mask can represent up to 64 banks */ + if (hv_max_vp_index / 64 >= 64) + return 0; + + /* + * Clear all banks up to the maximum possible bank as hv_tlb_flush_ex + * structs are not cleared between calls, we risk flushing unneeded + * vCPUs otherwise. + */ + for (vcpu_bank = 0; vcpu_bank <= hv_max_vp_index / 64; vcpu_bank++) + vpset->bank_contents[vcpu_bank] = 0; + + /* + * Some banks may end up being empty but this is acceptable. + */ + for_each_cpu(cpu, cpus) { + vcpu = hv_cpu_number_to_vp_number(cpu); + vcpu_bank = vcpu / 64; + vcpu_offset = vcpu % 64; + __set_bit(vcpu_offset, (unsigned long *) + &vpset->bank_contents[vcpu_bank]); + if (vcpu_bank >= nr_bank) + nr_bank = vcpu_bank + 1; + } + vpset->valid_bank_mask = GENMASK_ULL(nr_bank - 1, 0); + return nr_bank; +} + +void __init hyperv_init(void); void hyperv_setup_mmu_ops(void); -void hyper_alloc_mmu(void); void hyperv_report_panic(struct pt_regs *regs, long err); bool hv_is_hyperv_initialized(void); void hyperv_cleanup(void); @@ -269,6 +302,13 @@ void hyperv_reenlightenment_intr(struct pt_regs *regs); void set_hv_tscchange_cb(void (*cb)(void)); void clear_hv_tscchange_cb(void); void hyperv_stop_tsc_emulation(void); + +#ifdef CONFIG_X86_64 +void hv_apic_init(void); +#else +static inline void hv_apic_init(void) {} +#endif + #else /* CONFIG_HYPERV */ static inline void hyperv_init(void) {} static inline bool hv_is_hyperv_initialized(void) { return false; } diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index fda2114197b3..68b2c3150de1 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -62,8 +62,8 @@ #define NHM_C3_AUTO_DEMOTE (1UL << 25) #define NHM_C1_AUTO_DEMOTE (1UL << 26) #define ATM_LNC_C6_AUTO_DEMOTE (1UL << 25) -#define SNB_C1_AUTO_UNDEMOTE (1UL << 27) -#define SNB_C3_AUTO_UNDEMOTE (1UL << 28) +#define SNB_C3_AUTO_UNDEMOTE (1UL << 27) +#define SNB_C1_AUTO_UNDEMOTE (1UL << 28) #define MSR_MTRRcap 0x000000fe diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index 8b38df98548e..f6f6c63da62f 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -308,16 +308,20 @@ do { \ * lfence * jmp spec_trap * do_rop: - * mov %rax,(%rsp) + * mov %rax,(%rsp) for x86_64 + * mov %edx,(%esp) for x86_32 * retq * * Without retpolines configured: * - * jmp *%rax + * jmp *%rax for x86_64 + * jmp *%edx for x86_32 */ #ifdef CONFIG_RETPOLINE -# define RETPOLINE_RAX_BPF_JIT_SIZE 17 -# define RETPOLINE_RAX_BPF_JIT() \ +# ifdef CONFIG_X86_64 +# define RETPOLINE_RAX_BPF_JIT_SIZE 17 +# define RETPOLINE_RAX_BPF_JIT() \ +do { \ EMIT1_off32(0xE8, 7); /* callq do_rop */ \ /* spec_trap: */ \ EMIT2(0xF3, 0x90); /* pause */ \ @@ -325,11 +329,30 @@ do { \ EMIT2(0xEB, 0xF9); /* jmp spec_trap */ \ /* do_rop: */ \ EMIT4(0x48, 0x89, 0x04, 0x24); /* mov %rax,(%rsp) */ \ - EMIT1(0xC3); /* retq */ -#else -# define RETPOLINE_RAX_BPF_JIT_SIZE 2 -# define RETPOLINE_RAX_BPF_JIT() \ - EMIT2(0xFF, 0xE0); /* jmp *%rax */ + EMIT1(0xC3); /* retq */ \ +} while (0) +# else /* !CONFIG_X86_64 */ +# define RETPOLINE_EDX_BPF_JIT() \ +do { \ + EMIT1_off32(0xE8, 7); /* call do_rop */ \ + /* spec_trap: */ \ + EMIT2(0xF3, 0x90); /* pause */ \ + EMIT3(0x0F, 0xAE, 0xE8); /* lfence */ \ + EMIT2(0xEB, 0xF9); /* jmp spec_trap */ \ + /* do_rop: */ \ + EMIT3(0x89, 0x14, 0x24); /* mov %edx,(%esp) */ \ + EMIT1(0xC3); /* ret */ \ +} while (0) +# endif +#else /* !CONFIG_RETPOLINE */ +# ifdef CONFIG_X86_64 +# define RETPOLINE_RAX_BPF_JIT_SIZE 2 +# define RETPOLINE_RAX_BPF_JIT() \ + EMIT2(0xFF, 0xE0); /* jmp *%rax */ +# else /* !CONFIG_X86_64 */ +# define RETPOLINE_EDX_BPF_JIT() \ + EMIT2(0xFF, 0xE2) /* jmp *%edx */ +# endif #endif #endif /* _ASM_X86_NOSPEC_BRANCH_H_ */ diff --git a/arch/x86/include/asm/page_64_types.h b/arch/x86/include/asm/page_64_types.h index 2c5a966dc222..6afac386a434 100644 --- a/arch/x86/include/asm/page_64_types.h +++ b/arch/x86/include/asm/page_64_types.h @@ -53,7 +53,7 @@ #define __PHYSICAL_MASK_SHIFT 52 #ifdef CONFIG_X86_5LEVEL -#define __VIRTUAL_MASK_SHIFT (pgtable_l5_enabled ? 56 : 47) +#define __VIRTUAL_MASK_SHIFT (pgtable_l5_enabled() ? 56 : 47) #else #define __VIRTUAL_MASK_SHIFT 47 #endif diff --git a/arch/x86/include/asm/page_types.h b/arch/x86/include/asm/page_types.h index 1e53560a84bb..c85e15010f48 100644 --- a/arch/x86/include/asm/page_types.h +++ b/arch/x86/include/asm/page_types.h @@ -17,7 +17,6 @@ #define PUD_PAGE_SIZE (_AC(1, UL) << PUD_SHIFT) #define PUD_PAGE_MASK (~(PUD_PAGE_SIZE-1)) -#define __PHYSICAL_MASK ((phys_addr_t)(__sme_clr((1ULL << __PHYSICAL_MASK_SHIFT) - 1))) #define __VIRTUAL_MASK ((1UL << __VIRTUAL_MASK_SHIFT) - 1) /* Cast *PAGE_MASK to a signed type so that it is sign-extended if @@ -55,6 +54,13 @@ #ifndef __ASSEMBLY__ +#ifdef CONFIG_DYNAMIC_PHYSICAL_MASK +extern phys_addr_t physical_mask; +#define __PHYSICAL_MASK physical_mask +#else +#define __PHYSICAL_MASK ((phys_addr_t)((1ULL << __PHYSICAL_MASK_SHIFT) - 1)) +#endif + extern int devmem_is_allowed(unsigned long pagenr); extern unsigned long max_low_pfn_mapped; diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index 9be2bf13825b..d49bbf4bb5c8 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -574,14 +574,14 @@ static inline void __set_pgd(pgd_t *pgdp, pgd_t pgd) } #define set_pgd(pgdp, pgdval) do { \ - if (pgtable_l5_enabled) \ + if (pgtable_l5_enabled()) \ __set_pgd(pgdp, pgdval); \ else \ set_p4d((p4d_t *)(pgdp), (p4d_t) { (pgdval).pgd }); \ } while (0) #define pgd_clear(pgdp) do { \ - if (pgtable_l5_enabled) \ + if (pgtable_l5_enabled()) \ set_pgd(pgdp, __pgd(0)); \ } while (0) diff --git a/arch/x86/include/asm/pci.h b/arch/x86/include/asm/pci.h index d32175e30259..662963681ea6 100644 --- a/arch/x86/include/asm/pci.h +++ b/arch/x86/include/asm/pci.h @@ -117,9 +117,6 @@ void native_restore_msi_irqs(struct pci_dev *dev); #define native_setup_msi_irqs NULL #define native_teardown_msi_irq NULL #endif - -#define PCI_DMA_BUS_IS_PHYS (dma_ops->is_phys) - #endif /* __KERNEL__ */ #ifdef CONFIG_X86_64 diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index 263c142a6a6c..ada6410fd2ec 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -167,7 +167,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, #if CONFIG_PGTABLE_LEVELS > 4 static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4d) { - if (!pgtable_l5_enabled) + if (!pgtable_l5_enabled()) return; paravirt_alloc_p4d(mm, __pa(p4d) >> PAGE_SHIFT); set_pgd(pgd, __pgd(_PAGE_TABLE | __pa(p4d))); @@ -193,7 +193,7 @@ extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long address) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled()) ___p4d_free_tlb(tlb, p4d); } diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index f1633de5a675..99ecde23c3ec 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -65,7 +65,7 @@ extern pmdval_t early_pmd_flags; #ifndef __PAGETABLE_P4D_FOLDED #define set_pgd(pgdp, pgd) native_set_pgd(pgdp, pgd) -#define pgd_clear(pgd) (pgtable_l5_enabled ? native_pgd_clear(pgd) : 0) +#define pgd_clear(pgd) (pgtable_l5_enabled() ? native_pgd_clear(pgd) : 0) #endif #ifndef set_p4d @@ -881,7 +881,7 @@ static inline unsigned long p4d_index(unsigned long address) #if CONFIG_PGTABLE_LEVELS > 4 static inline int pgd_present(pgd_t pgd) { - if (!pgtable_l5_enabled) + if (!pgtable_l5_enabled()) return 1; return pgd_flags(pgd) & _PAGE_PRESENT; } @@ -898,9 +898,9 @@ static inline unsigned long pgd_page_vaddr(pgd_t pgd) #define pgd_page(pgd) pfn_to_page(pgd_pfn(pgd)) /* to find an entry in a page-table-directory. */ -static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) +static __always_inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) { - if (!pgtable_l5_enabled) + if (!pgtable_l5_enabled()) return (p4d_t *)pgd; return (p4d_t *)pgd_page_vaddr(*pgd) + p4d_index(address); } @@ -909,7 +909,7 @@ static inline int pgd_bad(pgd_t pgd) { unsigned long ignore_flags = _PAGE_USER; - if (!pgtable_l5_enabled) + if (!pgtable_l5_enabled()) return 0; if (IS_ENABLED(CONFIG_PAGE_TABLE_ISOLATION)) @@ -920,7 +920,7 @@ static inline int pgd_bad(pgd_t pgd) static inline int pgd_none(pgd_t pgd) { - if (!pgtable_l5_enabled) + if (!pgtable_l5_enabled()) return 0; /* * There is no need to do a workaround for the KNL stray diff --git a/arch/x86/include/asm/pgtable_32_types.h b/arch/x86/include/asm/pgtable_32_types.h index e3225e83db7d..d9a001a4a872 100644 --- a/arch/x86/include/asm/pgtable_32_types.h +++ b/arch/x86/include/asm/pgtable_32_types.h @@ -15,7 +15,7 @@ # include #endif -#define pgtable_l5_enabled 0 +#define pgtable_l5_enabled() 0 #define PGDIR_SIZE (1UL << PGDIR_SHIFT) #define PGDIR_MASK (~(PGDIR_SIZE - 1)) diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtable_64.h index 877bc27718ae..0fdcd21dadbd 100644 --- a/arch/x86/include/asm/pgtable_64.h +++ b/arch/x86/include/asm/pgtable_64.h @@ -216,11 +216,11 @@ static inline pgd_t pti_set_user_pgd(pgd_t *pgdp, pgd_t pgd) } #endif -static inline void native_set_p4d(p4d_t *p4dp, p4d_t p4d) +static __always_inline void native_set_p4d(p4d_t *p4dp, p4d_t p4d) { pgd_t pgd; - if (pgtable_l5_enabled || !IS_ENABLED(CONFIG_PAGE_TABLE_ISOLATION)) { + if (pgtable_l5_enabled() || !IS_ENABLED(CONFIG_PAGE_TABLE_ISOLATION)) { *p4dp = p4d; return; } @@ -230,7 +230,7 @@ static inline void native_set_p4d(p4d_t *p4dp, p4d_t p4d) *p4dp = native_make_p4d(native_pgd_val(pgd)); } -static inline void native_p4d_clear(p4d_t *p4d) +static __always_inline void native_p4d_clear(p4d_t *p4d) { native_set_p4d(p4d, native_make_p4d(0)); } diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index adb47552e6bb..054765ab2da2 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -22,12 +22,23 @@ typedef struct { pteval_t pte; } pte_t; #ifdef CONFIG_X86_5LEVEL extern unsigned int __pgtable_l5_enabled; -#ifndef pgtable_l5_enabled -#define pgtable_l5_enabled cpu_feature_enabled(X86_FEATURE_LA57) -#endif + +#ifdef USE_EARLY_PGTABLE_L5 +/* + * cpu_feature_enabled() is not available in early boot code. + * Use variable instead. + */ +static inline bool pgtable_l5_enabled(void) +{ + return __pgtable_l5_enabled; +} #else -#define pgtable_l5_enabled 0 -#endif +#define pgtable_l5_enabled() cpu_feature_enabled(X86_FEATURE_LA57) +#endif /* USE_EARLY_PGTABLE_L5 */ + +#else +#define pgtable_l5_enabled() 0 +#endif /* CONFIG_X86_5LEVEL */ extern unsigned int pgdir_shift; extern unsigned int ptrs_per_p4d; @@ -102,7 +113,7 @@ extern unsigned int ptrs_per_p4d; #define LDT_PGD_ENTRY_L4 -3UL #define LDT_PGD_ENTRY_L5 -112UL -#define LDT_PGD_ENTRY (pgtable_l5_enabled ? LDT_PGD_ENTRY_L5 : LDT_PGD_ENTRY_L4) +#define LDT_PGD_ENTRY (pgtable_l5_enabled() ? LDT_PGD_ENTRY_L5 : LDT_PGD_ENTRY_L4) #define LDT_BASE_ADDR (LDT_PGD_ENTRY << PGDIR_SHIFT) #define __VMALLOC_BASE_L4 0xffffc90000000000UL @@ -116,7 +127,7 @@ extern unsigned int ptrs_per_p4d; #ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT # define VMALLOC_START vmalloc_base -# define VMALLOC_SIZE_TB (pgtable_l5_enabled ? VMALLOC_SIZE_TB_L5 : VMALLOC_SIZE_TB_L4) +# define VMALLOC_SIZE_TB (pgtable_l5_enabled() ? VMALLOC_SIZE_TB_L5 : VMALLOC_SIZE_TB_L4) # define VMEMMAP_START vmemmap_base #else # define VMALLOC_START __VMALLOC_BASE_L4 diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 1e5a40673953..99fff853c944 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -65,7 +65,6 @@ #define _PAGE_PKEY_BIT2 (_AT(pteval_t, 0)) #define _PAGE_PKEY_BIT3 (_AT(pteval_t, 0)) #endif -#define __HAVE_ARCH_PTE_SPECIAL #define _PAGE_PKEY_MASK (_PAGE_PKEY_BIT0 | \ _PAGE_PKEY_BIT1 | \ diff --git a/arch/x86/include/asm/pkeys.h b/arch/x86/include/asm/pkeys.h index 851c04b7a092..19b137f1b3be 100644 --- a/arch/x86/include/asm/pkeys.h +++ b/arch/x86/include/asm/pkeys.h @@ -9,6 +9,11 @@ extern int arch_set_user_pkey_access(struct task_struct *tsk, int pkey, unsigned long init_val); +static inline bool arch_pkeys_enabled(void) +{ + return boot_cpu_has(X86_FEATURE_OSPKE); +} + /* * Try to dedicate one of the protection keys to be used as an * execute-only protection key. @@ -116,4 +121,12 @@ extern int __arch_set_user_pkey_access(struct task_struct *tsk, int pkey, unsigned long init_val); extern void copy_init_pkru_to_fpregs(void); +static inline int vma_pkey(struct vm_area_struct *vma) +{ + unsigned long vma_pkey_mask = VM_PKEY_BIT0 | VM_PKEY_BIT1 | + VM_PKEY_BIT2 | VM_PKEY_BIT3; + + return (vma->vm_flags & vma_pkey_mask) >> VM_PKEY_SHIFT; +} + #endif /*_ASM_X86_PKEYS_H */ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 21a114914ba4..cfd29ee8c3da 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -186,15 +186,6 @@ extern void identify_boot_cpu(void); extern void identify_secondary_cpu(struct cpuinfo_x86 *); extern void print_cpu_info(struct cpuinfo_x86 *); void print_cpu_msr(struct cpuinfo_x86 *); -extern void init_scattered_cpuid_features(struct cpuinfo_x86 *c); -extern u32 get_scattered_cpuid_leaf(unsigned int level, - unsigned int sub_leaf, - enum cpuid_regs_idx reg); -extern unsigned int init_intel_cacheinfo(struct cpuinfo_x86 *c); -extern void init_amd_cacheinfo(struct cpuinfo_x86 *c); - -extern void detect_extended_topology(struct cpuinfo_x86 *c); -extern void detect_ht(struct cpuinfo_x86 *c); #ifdef CONFIG_X86_32 extern int have_cpuid_p(void); @@ -421,7 +412,7 @@ extern asmlinkage void ignore_sysret(void); void save_fsgs_for_kvm(void); #endif #else /* X86_64 */ -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR /* * Make sure stack canary segment base is cached-aligned: * "For Intel Atom processors, avoid non zero segment base address diff --git a/arch/x86/include/asm/pvclock.h b/arch/x86/include/asm/pvclock.h index a7471dcd2205..b6033680d458 100644 --- a/arch/x86/include/asm/pvclock.h +++ b/arch/x86/include/asm/pvclock.h @@ -12,7 +12,7 @@ void pvclock_set_flags(u8 flags); unsigned long pvclock_tsc_khz(struct pvclock_vcpu_time_info *src); void pvclock_read_wallclock(struct pvclock_wall_clock *wall, struct pvclock_vcpu_time_info *vcpu, - struct timespec *ts); + struct timespec64 *ts); void pvclock_resume(void); void pvclock_touch_watchdogs(void); diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index 5e16b5d40d32..3e70bed8a978 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -7,6 +7,14 @@ #include #include +#define _Q_PENDING_LOOPS (1 << 9) + +#ifdef CONFIG_PARAVIRT_SPINLOCKS +extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +extern void __pv_init_lock_hash(void); +extern void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +extern void __raw_callee_save___pv_queued_spin_unlock(struct qspinlock *lock); + #define queued_spin_unlock queued_spin_unlock /** * queued_spin_unlock - release a queued spinlock @@ -16,15 +24,9 @@ */ static inline void native_queued_spin_unlock(struct qspinlock *lock) { - smp_store_release((u8 *)lock, 0); + smp_store_release(&lock->locked, 0); } -#ifdef CONFIG_PARAVIRT_SPINLOCKS -extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); -extern void __pv_init_lock_hash(void); -extern void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); -extern void __raw_callee_save___pv_queued_spin_unlock(struct qspinlock *lock); - static inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) { pv_queued_spin_lock_slowpath(lock, val); @@ -40,11 +42,6 @@ static inline bool vcpu_is_preempted(long cpu) { return pv_vcpu_is_preempted(cpu); } -#else -static inline void queued_spin_unlock(struct qspinlock *lock) -{ - native_queued_spin_unlock(lock); -} #endif #ifdef CONFIG_PARAVIRT diff --git a/arch/x86/include/asm/qspinlock_paravirt.h b/arch/x86/include/asm/qspinlock_paravirt.h index 923307ea11c7..9ef5ee03d2d7 100644 --- a/arch/x86/include/asm/qspinlock_paravirt.h +++ b/arch/x86/include/asm/qspinlock_paravirt.h @@ -22,8 +22,7 @@ PV_CALLEE_SAVE_REGS_THUNK(__pv_queued_spin_unlock_slowpath); * * void __pv_queued_spin_unlock(struct qspinlock *lock) * { - * struct __qspinlock *l = (void *)lock; - * u8 lockval = cmpxchg(&l->locked, _Q_LOCKED_VAL, 0); + * u8 lockval = cmpxchg(&lock->locked, _Q_LOCKED_VAL, 0); * * if (likely(lockval == _Q_LOCKED_VAL)) * return; diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h index 8f09012b92e7..e293c122d0d5 100644 --- a/arch/x86/include/asm/segment.h +++ b/arch/x86/include/asm/segment.h @@ -146,7 +146,7 @@ # define __KERNEL_PERCPU 0 #endif -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR # define __KERNEL_STACK_CANARY (GDT_ENTRY_STACK_CANARY*8) #else # define __KERNEL_STACK_CANARY 0 diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index f75bff8f9d82..547c4fe50711 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -171,7 +171,6 @@ static inline int wbinvd_on_all_cpus(void) wbinvd(); return 0; } -#define smp_num_siblings 1 #endif /* CONFIG_SMP */ extern unsigned disabled_cpus; diff --git a/arch/x86/include/asm/sparsemem.h b/arch/x86/include/asm/sparsemem.h index 4617a2bf123c..199218719a86 100644 --- a/arch/x86/include/asm/sparsemem.h +++ b/arch/x86/include/asm/sparsemem.h @@ -27,8 +27,8 @@ # endif #else /* CONFIG_X86_32 */ # define SECTION_SIZE_BITS 27 /* matt - 128 is convenient right now */ -# define MAX_PHYSADDR_BITS (pgtable_l5_enabled ? 52 : 44) -# define MAX_PHYSMEM_BITS (pgtable_l5_enabled ? 52 : 46) +# define MAX_PHYSADDR_BITS (pgtable_l5_enabled() ? 52 : 44) +# define MAX_PHYSMEM_BITS (pgtable_l5_enabled() ? 52 : 46) #endif #endif /* CONFIG_SPARSEMEM */ diff --git a/arch/x86/include/asm/stackprotector.h b/arch/x86/include/asm/stackprotector.h index 371b3a4af000..8ec97a62c245 100644 --- a/arch/x86/include/asm/stackprotector.h +++ b/arch/x86/include/asm/stackprotector.h @@ -34,7 +34,7 @@ #ifndef _ASM_STACKPROTECTOR_H #define _ASM_STACKPROTECTOR_H 1 -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR #include #include @@ -105,7 +105,7 @@ static inline void load_stack_canary_segment(void) #endif } -#else /* CC_STACKPROTECTOR */ +#else /* STACKPROTECTOR */ #define GDT_STACK_CANARY_INIT @@ -121,5 +121,5 @@ static inline void load_stack_canary_segment(void) #endif } -#endif /* CC_STACKPROTECTOR */ +#endif /* STACKPROTECTOR */ #endif /* _ASM_STACKPROTECTOR_H */ diff --git a/arch/x86/include/asm/stacktrace.h b/arch/x86/include/asm/stacktrace.h index 133d9425fced..b6dc698f992a 100644 --- a/arch/x86/include/asm/stacktrace.h +++ b/arch/x86/include/asm/stacktrace.h @@ -111,4 +111,6 @@ static inline unsigned long caller_frame_pointer(void) return (unsigned long)frame; } +void show_opcodes(u8 *rip, const char *loglvl); +void show_ip(struct pt_regs *regs, const char *loglvl); #endif /* _ASM_X86_STACKTRACE_H */ diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h index 533f74c300c2..d33f92b9fa22 100644 --- a/arch/x86/include/asm/string_64.h +++ b/arch/x86/include/asm/string_64.h @@ -116,7 +116,8 @@ int strcmp(const char *cs, const char *ct); #endif #define __HAVE_ARCH_MEMCPY_MCSAFE 1 -__must_check int memcpy_mcsafe_unrolled(void *dst, const void *src, size_t cnt); +__must_check unsigned long __memcpy_mcsafe(void *dst, const void *src, + size_t cnt); DECLARE_STATIC_KEY_FALSE(mcsafe_key); /** @@ -131,14 +132,15 @@ DECLARE_STATIC_KEY_FALSE(mcsafe_key); * actually do machine check recovery. Everyone else can just * use memcpy(). * - * Return 0 for success, -EFAULT for fail + * Return 0 for success, or number of bytes not copied if there was an + * exception. */ -static __always_inline __must_check int +static __always_inline __must_check unsigned long memcpy_mcsafe(void *dst, const void *src, size_t cnt) { #ifdef CONFIG_X86_MCE if (static_branch_unlikely(&mcsafe_key)) - return memcpy_mcsafe_unrolled(dst, src, cnt); + return __memcpy_mcsafe(dst, src, cnt); else #endif memcpy(dst, src, cnt); diff --git a/arch/x86/include/asm/trace/irq_vectors.h b/arch/x86/include/asm/trace/irq_vectors.h index 22647a642e98..0af81b590a0c 100644 --- a/arch/x86/include/asm/trace/irq_vectors.h +++ b/arch/x86/include/asm/trace/irq_vectors.h @@ -236,7 +236,7 @@ TRACE_EVENT(vector_alloc, TP_PROTO(unsigned int irq, unsigned int vector, bool reserved, int ret), - TP_ARGS(irq, vector, ret, reserved), + TP_ARGS(irq, vector, reserved, ret), TP_STRUCT__entry( __field( unsigned int, irq ) diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h index 62546b3a398e..62acb613114b 100644 --- a/arch/x86/include/asm/uaccess_64.h +++ b/arch/x86/include/asm/uaccess_64.h @@ -46,6 +46,17 @@ copy_user_generic(void *to, const void *from, unsigned len) return ret; } +static __always_inline __must_check unsigned long +copy_to_user_mcsafe(void *to, const void *from, unsigned len) +{ + unsigned long ret; + + __uaccess_begin(); + ret = memcpy_mcsafe(to, from, len); + __uaccess_end(); + return ret; +} + static __always_inline __must_check unsigned long raw_copy_from_user(void *dst, const void __user *src, unsigned long size) { @@ -194,4 +205,7 @@ __copy_from_user_flushcache(void *dst, const void __user *src, unsigned size) unsigned long copy_user_handle_tail(char *to, char *from, unsigned len); +unsigned long +mcsafe_handle_tail(char *to, char *from, unsigned len); + #endif /* _ASM_X86_UACCESS_64_H */ diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h index 5db8b0b10766..425e6b8b9547 100644 --- a/arch/x86/include/asm/vmx.h +++ b/arch/x86/include/asm/vmx.h @@ -207,7 +207,9 @@ enum vmcs_field { EPTP_LIST_ADDRESS = 0x00002024, EPTP_LIST_ADDRESS_HIGH = 0x00002025, VMREAD_BITMAP = 0x00002026, + VMREAD_BITMAP_HIGH = 0x00002027, VMWRITE_BITMAP = 0x00002028, + VMWRITE_BITMAP_HIGH = 0x00002029, XSS_EXIT_BITMAP = 0x0000202C, XSS_EXIT_BITMAP_HIGH = 0x0000202D, TSC_MULTIPLIER = 0x00002032, diff --git a/arch/x86/include/asm/x86_init.h b/arch/x86/include/asm/x86_init.h index ce8b4da07e35..b85a7c54c6a1 100644 --- a/arch/x86/include/asm/x86_init.h +++ b/arch/x86/include/asm/x86_init.h @@ -170,7 +170,7 @@ struct x86_cpuinit_ops { void (*fixup_cpu_id)(struct cpuinfo_x86 *c, int node); }; -struct timespec; +struct timespec64; /** * struct x86_legacy_devices - legacy x86 devices @@ -264,8 +264,8 @@ struct x86_hyper_runtime { struct x86_platform_ops { unsigned long (*calibrate_cpu)(void); unsigned long (*calibrate_tsc)(void); - void (*get_wallclock)(struct timespec *ts); - int (*set_wallclock)(const struct timespec *ts); + void (*get_wallclock)(struct timespec64 *ts); + int (*set_wallclock)(const struct timespec64 *ts); void (*iommu_shutdown)(void); bool (*is_untracked_pat_range)(u64 start, u64 end); void (*nmi_init)(void); @@ -301,5 +301,6 @@ extern struct x86_apic_ops x86_apic_ops; extern void x86_early_init_platform_quirks(void); extern void x86_init_noop(void); extern void x86_init_uint_noop(unsigned int unused); +extern bool x86_pnpbios_disabled(void); #endif diff --git a/arch/x86/include/uapi/asm/sembuf.h b/arch/x86/include/uapi/asm/sembuf.h index cabd7476bd6c..89de6cd9f0a7 100644 --- a/arch/x86/include/uapi/asm/sembuf.h +++ b/arch/x86/include/uapi/asm/sembuf.h @@ -8,15 +8,24 @@ * between kernel and user space. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem * - 2 miscellaneous 32-bit values + * + * x86_64 and x32 incorrectly added padding here, so the structures + * are still incompatible with the padding on x86. */ struct semid64_ds { struct ipc64_perm sem_perm; /* permissions .. see ipc.h */ +#ifdef __i386__ + unsigned long sem_otime; /* last semop time */ + unsigned long sem_otime_high; + unsigned long sem_ctime; /* last change time */ + unsigned long sem_ctime_high; +#else __kernel_time_t sem_otime; /* last semop time */ __kernel_ulong_t __unused1; __kernel_time_t sem_ctime; /* last change time */ __kernel_ulong_t __unused2; +#endif __kernel_ulong_t sem_nsems; /* no. of semaphores in array */ __kernel_ulong_t __unused3; __kernel_ulong_t __unused4; diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c index 7553819c74c3..3982f79d2377 100644 --- a/arch/x86/kernel/apic/io_apic.c +++ b/arch/x86/kernel/apic/io_apic.c @@ -1851,7 +1851,7 @@ static void ioapic_ir_ack_level(struct irq_data *irq_data) * intr-remapping table entry. Hence for the io-apic * EOI we use the pin number. */ - ack_APIC_irq(); + apic_ack_irq(irq_data); eoi_ioapic_pin(data->entry.vector, data); } diff --git a/arch/x86/kernel/apic/vector.c b/arch/x86/kernel/apic/vector.c index bb6f7a2148d7..35aaee4fc028 100644 --- a/arch/x86/kernel/apic/vector.c +++ b/arch/x86/kernel/apic/vector.c @@ -235,6 +235,15 @@ static int allocate_vector(struct irq_data *irqd, const struct cpumask *dest) if (vector && cpu_online(cpu) && cpumask_test_cpu(cpu, dest)) return 0; + /* + * Careful here. @apicd might either have move_in_progress set or + * be enqueued for cleanup. Assigning a new vector would either + * leave a stale vector on some CPU around or in case of a pending + * cleanup corrupt the hlist. + */ + if (apicd->move_in_progress || !hlist_unhashed(&apicd->clist)) + return -EBUSY; + vector = irq_matrix_alloc(vector_matrix, dest, resvd, &cpu); if (vector > 0) apic_update_vector(irqd, vector, cpu); @@ -579,8 +588,7 @@ error: static void x86_vector_debug_show(struct seq_file *m, struct irq_domain *d, struct irq_data *irqd, int ind) { - unsigned int cpu, vector, prev_cpu, prev_vector; - struct apic_chip_data *apicd; + struct apic_chip_data apicd; unsigned long flags; int irq; @@ -596,24 +604,26 @@ static void x86_vector_debug_show(struct seq_file *m, struct irq_domain *d, return; } - apicd = irqd->chip_data; - if (!apicd) { + if (!irqd->chip_data) { seq_printf(m, "%*sVector: Not assigned\n", ind, ""); return; } raw_spin_lock_irqsave(&vector_lock, flags); - cpu = apicd->cpu; - vector = apicd->vector; - prev_cpu = apicd->prev_cpu; - prev_vector = apicd->prev_vector; + memcpy(&apicd, irqd->chip_data, sizeof(apicd)); raw_spin_unlock_irqrestore(&vector_lock, flags); - seq_printf(m, "%*sVector: %5u\n", ind, "", vector); - seq_printf(m, "%*sTarget: %5u\n", ind, "", cpu); - if (prev_vector) { - seq_printf(m, "%*sPrevious vector: %5u\n", ind, "", prev_vector); - seq_printf(m, "%*sPrevious target: %5u\n", ind, "", prev_cpu); + + seq_printf(m, "%*sVector: %5u\n", ind, "", apicd.vector); + seq_printf(m, "%*sTarget: %5u\n", ind, "", apicd.cpu); + if (apicd.prev_vector) { + seq_printf(m, "%*sPrevious vector: %5u\n", ind, "", apicd.prev_vector); + seq_printf(m, "%*sPrevious target: %5u\n", ind, "", apicd.prev_cpu); } + seq_printf(m, "%*smove_in_progress: %u\n", ind, "", apicd.move_in_progress ? 1 : 0); + seq_printf(m, "%*sis_managed: %u\n", ind, "", apicd.is_managed ? 1 : 0); + seq_printf(m, "%*scan_reserve: %u\n", ind, "", apicd.can_reserve ? 1 : 0); + seq_printf(m, "%*shas_reserved: %u\n", ind, "", apicd.has_reserved ? 1 : 0); + seq_printf(m, "%*scleanup_pending: %u\n", ind, "", !hlist_unhashed(&apicd.clist)); } #endif @@ -800,13 +810,18 @@ static int apic_retrigger_irq(struct irq_data *irqd) return 1; } -void apic_ack_edge(struct irq_data *irqd) +void apic_ack_irq(struct irq_data *irqd) { - irq_complete_move(irqd_cfg(irqd)); irq_move_irq(irqd); ack_APIC_irq(); } +void apic_ack_edge(struct irq_data *irqd) +{ + irq_complete_move(irqd_cfg(irqd)); + apic_ack_irq(irqd); +} + static struct irq_chip lapic_controller = { .name = "APIC", .irq_ack = apic_ack_edge, diff --git a/arch/x86/kernel/apm_32.c b/arch/x86/kernel/apm_32.c index dfcbe6924eaf..5d0de79fdab0 100644 --- a/arch/x86/kernel/apm_32.c +++ b/arch/x86/kernel/apm_32.c @@ -1715,19 +1715,6 @@ static int proc_apm_show(struct seq_file *m, void *v) return 0; } -static int proc_apm_open(struct inode *inode, struct file *file) -{ - return single_open(file, proc_apm_show, NULL); -} - -static const struct file_operations apm_file_ops = { - .owner = THIS_MODULE, - .open = proc_apm_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int apm(void *unused) { unsigned short bx; @@ -2360,7 +2347,7 @@ static int __init apm_init(void) set_desc_base(&gdt[APM_DS >> 3], (unsigned long)__va((unsigned long)apm_info.bios.dseg << 4)); - proc_create("apm", 0, NULL, &apm_file_ops); + proc_create_single("apm", 0, NULL, proc_apm_show); kapmd_task = kthread_create(apm, NULL, "kapmd"); if (IS_ERR(kapmd_task)) { @@ -2446,7 +2433,7 @@ MODULE_PARM_DESC(idle_threshold, "System idle percentage above which to make APM BIOS idle calls"); module_param(idle_period, int, 0444); MODULE_PARM_DESC(idle_period, - "Period (in sec/100) over which to caculate the idle percentage"); + "Period (in sec/100) over which to calculate the idle percentage"); module_param(smp, bool, 0444); MODULE_PARM_DESC(smp, "Set this to enable APM use on an SMP platform. Use with caution on older systems"); diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 76417a9aab73..dcb008c320fe 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -32,7 +32,7 @@ void common(void) { BLANK(); OFFSET(TASK_threadsp, task_struct, thread.sp); -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR OFFSET(TASK_stack_canary, task_struct, stack_canary); #endif diff --git a/arch/x86/kernel/asm-offsets_32.c b/arch/x86/kernel/asm-offsets_32.c index f91ba53e06c8..a4a3be399f4b 100644 --- a/arch/x86/kernel/asm-offsets_32.c +++ b/arch/x86/kernel/asm-offsets_32.c @@ -50,7 +50,7 @@ void foo(void) DEFINE(TSS_sysenter_sp0, offsetof(struct cpu_entry_area, tss.x86_tss.sp0) - offsetofend(struct cpu_entry_area, entry_stack_page.stack)); -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR BLANK(); OFFSET(stack_canary_offset, stack_canary, canary); #endif diff --git a/arch/x86/kernel/asm-offsets_64.c b/arch/x86/kernel/asm-offsets_64.c index bf51e51d808d..b2dcd161f514 100644 --- a/arch/x86/kernel/asm-offsets_64.c +++ b/arch/x86/kernel/asm-offsets_64.c @@ -69,7 +69,7 @@ int main(void) OFFSET(TSS_sp1, tss_struct, x86_tss.sp1); BLANK(); -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR DEFINE(stack_canary_offset, offsetof(union irq_stack_union, stack_canary)); BLANK(); #endif diff --git a/arch/x86/kernel/cpu/Makefile b/arch/x86/kernel/cpu/Makefile index a66229f51b12..7a40196967cb 100644 --- a/arch/x86/kernel/cpu/Makefile +++ b/arch/x86/kernel/cpu/Makefile @@ -17,7 +17,7 @@ KCOV_INSTRUMENT_perf_event.o := n nostackp := $(call cc-option, -fno-stack-protector) CFLAGS_common.o := $(nostackp) -obj-y := intel_cacheinfo.o scattered.o topology.o +obj-y := cacheinfo.o scattered.o topology.o obj-y += common.o obj-y += rdrand.o obj-y += match.o diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c index 1b18be3f35a8..082d7875cef8 100644 --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -298,7 +299,6 @@ static int nearby_node(int apicid) } #endif -#ifdef CONFIG_SMP /* * Fix up cpu_core_id for pre-F17h systems to be in the * [0 .. cores_per_node - 1] range. Not really needed but @@ -328,6 +328,7 @@ static void amd_get_topology(struct cpuinfo_x86 *c) /* get information required for multi-node processors */ if (boot_cpu_has(X86_FEATURE_TOPOEXT)) { + int err; u32 eax, ebx, ecx, edx; cpuid(0x8000001e, &eax, &ebx, &ecx, &edx); @@ -346,21 +347,15 @@ static void amd_get_topology(struct cpuinfo_x86 *c) } /* - * We may have multiple LLCs if L3 caches exist, so check if we - * have an L3 cache by looking at the L3 cache CPUID leaf. + * In case leaf B is available, use it to derive + * topology information. */ - if (cpuid_edx(0x80000006)) { - if (c->x86 == 0x17) { - /* - * LLC is at the core complex level. - * Core complex id is ApicId[3]. - */ - per_cpu(cpu_llc_id, cpu) = c->apicid >> 3; - } else { - /* LLC is at the node level. */ - per_cpu(cpu_llc_id, cpu) = node_id; - } - } + err = detect_extended_topology(c); + if (!err) + c->x86_coreid_bits = get_count_order(c->x86_max_cores); + + cacheinfo_amd_init_llc_id(c, cpu, node_id); + } else if (cpu_has(c, X86_FEATURE_NODEID_MSR)) { u64 value; @@ -376,7 +371,6 @@ static void amd_get_topology(struct cpuinfo_x86 *c) legacy_fixup_core_id(c); } } -#endif /* * On a AMD dual core setup the lower bits of the APIC id distinguish the cores. @@ -384,7 +378,6 @@ static void amd_get_topology(struct cpuinfo_x86 *c) */ static void amd_detect_cmp(struct cpuinfo_x86 *c) { -#ifdef CONFIG_SMP unsigned bits; int cpu = smp_processor_id(); @@ -395,17 +388,11 @@ static void amd_detect_cmp(struct cpuinfo_x86 *c) c->phys_proc_id = c->initial_apicid >> bits; /* use socket ID also for last level cache */ per_cpu(cpu_llc_id, cpu) = c->phys_proc_id; - amd_get_topology(c); -#endif } u16 amd_get_nb_id(int cpu) { - u16 id = 0; -#ifdef CONFIG_SMP - id = per_cpu(cpu_llc_id, cpu); -#endif - return id; + return per_cpu(cpu_llc_id, cpu); } EXPORT_SYMBOL_GPL(amd_get_nb_id); @@ -864,6 +851,7 @@ static void init_amd(struct cpuinfo_x86 *c) /* Multi core CPU? */ if (c->extended_cpuid_level >= 0x80000008) { amd_detect_cmp(c); + amd_get_topology(c); srat_detect_node(c); } diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index 7416fc206b4a..cd0fda1fff6d 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -529,18 +529,15 @@ static enum ssb_mitigation __init __ssb_select_mitigation(void) if (mode == SPEC_STORE_BYPASS_DISABLE) { setup_force_cpu_cap(X86_FEATURE_SPEC_STORE_BYPASS_DISABLE); /* - * Intel uses the SPEC CTRL MSR Bit(2) for this, while AMD uses - * a completely different MSR and bit dependent on family. + * Intel uses the SPEC CTRL MSR Bit(2) for this, while AMD may + * use a completely different MSR and bit dependent on family. */ - switch (boot_cpu_data.x86_vendor) { - case X86_VENDOR_INTEL: + if (!static_cpu_has(X86_FEATURE_MSR_SPEC_CTRL)) + x86_amd_ssb_disable(); + else { x86_spec_ctrl_base |= SPEC_CTRL_SSBD; x86_spec_ctrl_mask |= SPEC_CTRL_SSBD; wrmsrl(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base); - break; - case X86_VENDOR_AMD: - x86_amd_ssb_disable(); - break; } } diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinfo.c new file mode 100644 index 000000000000..38354c66df81 --- /dev/null +++ b/arch/x86/kernel/cpu/cacheinfo.c @@ -0,0 +1,1010 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Routines to identify caches on Intel CPU. + * + * Changes: + * Venkatesh Pallipadi : Adding cache identification through cpuid(4) + * Ashok Raj : Work with CPU hotplug infrastructure. + * Andi Kleen / Andreas Herrmann : CPUID4 emulation on AMD. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include "cpu.h" + +#define LVL_1_INST 1 +#define LVL_1_DATA 2 +#define LVL_2 3 +#define LVL_3 4 +#define LVL_TRACE 5 + +struct _cache_table { + unsigned char descriptor; + char cache_type; + short size; +}; + +#define MB(x) ((x) * 1024) + +/* All the cache descriptor types we care about (no TLB or + trace cache entries) */ + +static const struct _cache_table cache_table[] = +{ + { 0x06, LVL_1_INST, 8 }, /* 4-way set assoc, 32 byte line size */ + { 0x08, LVL_1_INST, 16 }, /* 4-way set assoc, 32 byte line size */ + { 0x09, LVL_1_INST, 32 }, /* 4-way set assoc, 64 byte line size */ + { 0x0a, LVL_1_DATA, 8 }, /* 2 way set assoc, 32 byte line size */ + { 0x0c, LVL_1_DATA, 16 }, /* 4-way set assoc, 32 byte line size */ + { 0x0d, LVL_1_DATA, 16 }, /* 4-way set assoc, 64 byte line size */ + { 0x0e, LVL_1_DATA, 24 }, /* 6-way set assoc, 64 byte line size */ + { 0x21, LVL_2, 256 }, /* 8-way set assoc, 64 byte line size */ + { 0x22, LVL_3, 512 }, /* 4-way set assoc, sectored cache, 64 byte line size */ + { 0x23, LVL_3, MB(1) }, /* 8-way set assoc, sectored cache, 64 byte line size */ + { 0x25, LVL_3, MB(2) }, /* 8-way set assoc, sectored cache, 64 byte line size */ + { 0x29, LVL_3, MB(4) }, /* 8-way set assoc, sectored cache, 64 byte line size */ + { 0x2c, LVL_1_DATA, 32 }, /* 8-way set assoc, 64 byte line size */ + { 0x30, LVL_1_INST, 32 }, /* 8-way set assoc, 64 byte line size */ + { 0x39, LVL_2, 128 }, /* 4-way set assoc, sectored cache, 64 byte line size */ + { 0x3a, LVL_2, 192 }, /* 6-way set assoc, sectored cache, 64 byte line size */ + { 0x3b, LVL_2, 128 }, /* 2-way set assoc, sectored cache, 64 byte line size */ + { 0x3c, LVL_2, 256 }, /* 4-way set assoc, sectored cache, 64 byte line size */ + { 0x3d, LVL_2, 384 }, /* 6-way set assoc, sectored cache, 64 byte line size */ + { 0x3e, LVL_2, 512 }, /* 4-way set assoc, sectored cache, 64 byte line size */ + { 0x3f, LVL_2, 256 }, /* 2-way set assoc, 64 byte line size */ + { 0x41, LVL_2, 128 }, /* 4-way set assoc, 32 byte line size */ + { 0x42, LVL_2, 256 }, /* 4-way set assoc, 32 byte line size */ + { 0x43, LVL_2, 512 }, /* 4-way set assoc, 32 byte line size */ + { 0x44, LVL_2, MB(1) }, /* 4-way set assoc, 32 byte line size */ + { 0x45, LVL_2, MB(2) }, /* 4-way set assoc, 32 byte line size */ + { 0x46, LVL_3, MB(4) }, /* 4-way set assoc, 64 byte line size */ + { 0x47, LVL_3, MB(8) }, /* 8-way set assoc, 64 byte line size */ + { 0x48, LVL_2, MB(3) }, /* 12-way set assoc, 64 byte line size */ + { 0x49, LVL_3, MB(4) }, /* 16-way set assoc, 64 byte line size */ + { 0x4a, LVL_3, MB(6) }, /* 12-way set assoc, 64 byte line size */ + { 0x4b, LVL_3, MB(8) }, /* 16-way set assoc, 64 byte line size */ + { 0x4c, LVL_3, MB(12) }, /* 12-way set assoc, 64 byte line size */ + { 0x4d, LVL_3, MB(16) }, /* 16-way set assoc, 64 byte line size */ + { 0x4e, LVL_2, MB(6) }, /* 24-way set assoc, 64 byte line size */ + { 0x60, LVL_1_DATA, 16 }, /* 8-way set assoc, sectored cache, 64 byte line size */ + { 0x66, LVL_1_DATA, 8 }, /* 4-way set assoc, sectored cache, 64 byte line size */ + { 0x67, LVL_1_DATA, 16 }, /* 4-way set assoc, sectored cache, 64 byte line size */ + { 0x68, LVL_1_DATA, 32 }, /* 4-way set assoc, sectored cache, 64 byte line size */ + { 0x70, LVL_TRACE, 12 }, /* 8-way set assoc */ + { 0x71, LVL_TRACE, 16 }, /* 8-way set assoc */ + { 0x72, LVL_TRACE, 32 }, /* 8-way set assoc */ + { 0x73, LVL_TRACE, 64 }, /* 8-way set assoc */ + { 0x78, LVL_2, MB(1) }, /* 4-way set assoc, 64 byte line size */ + { 0x79, LVL_2, 128 }, /* 8-way set assoc, sectored cache, 64 byte line size */ + { 0x7a, LVL_2, 256 }, /* 8-way set assoc, sectored cache, 64 byte line size */ + { 0x7b, LVL_2, 512 }, /* 8-way set assoc, sectored cache, 64 byte line size */ + { 0x7c, LVL_2, MB(1) }, /* 8-way set assoc, sectored cache, 64 byte line size */ + { 0x7d, LVL_2, MB(2) }, /* 8-way set assoc, 64 byte line size */ + { 0x7f, LVL_2, 512 }, /* 2-way set assoc, 64 byte line size */ + { 0x80, LVL_2, 512 }, /* 8-way set assoc, 64 byte line size */ + { 0x82, LVL_2, 256 }, /* 8-way set assoc, 32 byte line size */ + { 0x83, LVL_2, 512 }, /* 8-way set assoc, 32 byte line size */ + { 0x84, LVL_2, MB(1) }, /* 8-way set assoc, 32 byte line size */ + { 0x85, LVL_2, MB(2) }, /* 8-way set assoc, 32 byte line size */ + { 0x86, LVL_2, 512 }, /* 4-way set assoc, 64 byte line size */ + { 0x87, LVL_2, MB(1) }, /* 8-way set assoc, 64 byte line size */ + { 0xd0, LVL_3, 512 }, /* 4-way set assoc, 64 byte line size */ + { 0xd1, LVL_3, MB(1) }, /* 4-way set assoc, 64 byte line size */ + { 0xd2, LVL_3, MB(2) }, /* 4-way set assoc, 64 byte line size */ + { 0xd6, LVL_3, MB(1) }, /* 8-way set assoc, 64 byte line size */ + { 0xd7, LVL_3, MB(2) }, /* 8-way set assoc, 64 byte line size */ + { 0xd8, LVL_3, MB(4) }, /* 12-way set assoc, 64 byte line size */ + { 0xdc, LVL_3, MB(2) }, /* 12-way set assoc, 64 byte line size */ + { 0xdd, LVL_3, MB(4) }, /* 12-way set assoc, 64 byte line size */ + { 0xde, LVL_3, MB(8) }, /* 12-way set assoc, 64 byte line size */ + { 0xe2, LVL_3, MB(2) }, /* 16-way set assoc, 64 byte line size */ + { 0xe3, LVL_3, MB(4) }, /* 16-way set assoc, 64 byte line size */ + { 0xe4, LVL_3, MB(8) }, /* 16-way set assoc, 64 byte line size */ + { 0xea, LVL_3, MB(12) }, /* 24-way set assoc, 64 byte line size */ + { 0xeb, LVL_3, MB(18) }, /* 24-way set assoc, 64 byte line size */ + { 0xec, LVL_3, MB(24) }, /* 24-way set assoc, 64 byte line size */ + { 0x00, 0, 0} +}; + + +enum _cache_type { + CTYPE_NULL = 0, + CTYPE_DATA = 1, + CTYPE_INST = 2, + CTYPE_UNIFIED = 3 +}; + +union _cpuid4_leaf_eax { + struct { + enum _cache_type type:5; + unsigned int level:3; + unsigned int is_self_initializing:1; + unsigned int is_fully_associative:1; + unsigned int reserved:4; + unsigned int num_threads_sharing:12; + unsigned int num_cores_on_die:6; + } split; + u32 full; +}; + +union _cpuid4_leaf_ebx { + struct { + unsigned int coherency_line_size:12; + unsigned int physical_line_partition:10; + unsigned int ways_of_associativity:10; + } split; + u32 full; +}; + +union _cpuid4_leaf_ecx { + struct { + unsigned int number_of_sets:32; + } split; + u32 full; +}; + +struct _cpuid4_info_regs { + union _cpuid4_leaf_eax eax; + union _cpuid4_leaf_ebx ebx; + union _cpuid4_leaf_ecx ecx; + unsigned int id; + unsigned long size; + struct amd_northbridge *nb; +}; + +static unsigned short num_cache_leaves; + +/* AMD doesn't have CPUID4. Emulate it here to report the same + information to the user. This makes some assumptions about the machine: + L2 not shared, no SMT etc. that is currently true on AMD CPUs. + + In theory the TLBs could be reported as fake type (they are in "dummy"). + Maybe later */ +union l1_cache { + struct { + unsigned line_size:8; + unsigned lines_per_tag:8; + unsigned assoc:8; + unsigned size_in_kb:8; + }; + unsigned val; +}; + +union l2_cache { + struct { + unsigned line_size:8; + unsigned lines_per_tag:4; + unsigned assoc:4; + unsigned size_in_kb:16; + }; + unsigned val; +}; + +union l3_cache { + struct { + unsigned line_size:8; + unsigned lines_per_tag:4; + unsigned assoc:4; + unsigned res:2; + unsigned size_encoded:14; + }; + unsigned val; +}; + +static const unsigned short assocs[] = { + [1] = 1, + [2] = 2, + [4] = 4, + [6] = 8, + [8] = 16, + [0xa] = 32, + [0xb] = 48, + [0xc] = 64, + [0xd] = 96, + [0xe] = 128, + [0xf] = 0xffff /* fully associative - no way to show this currently */ +}; + +static const unsigned char levels[] = { 1, 1, 2, 3 }; +static const unsigned char types[] = { 1, 2, 3, 3 }; + +static const enum cache_type cache_type_map[] = { + [CTYPE_NULL] = CACHE_TYPE_NOCACHE, + [CTYPE_DATA] = CACHE_TYPE_DATA, + [CTYPE_INST] = CACHE_TYPE_INST, + [CTYPE_UNIFIED] = CACHE_TYPE_UNIFIED, +}; + +static void +amd_cpuid4(int leaf, union _cpuid4_leaf_eax *eax, + union _cpuid4_leaf_ebx *ebx, + union _cpuid4_leaf_ecx *ecx) +{ + unsigned dummy; + unsigned line_size, lines_per_tag, assoc, size_in_kb; + union l1_cache l1i, l1d; + union l2_cache l2; + union l3_cache l3; + union l1_cache *l1 = &l1d; + + eax->full = 0; + ebx->full = 0; + ecx->full = 0; + + cpuid(0x80000005, &dummy, &dummy, &l1d.val, &l1i.val); + cpuid(0x80000006, &dummy, &dummy, &l2.val, &l3.val); + + switch (leaf) { + case 1: + l1 = &l1i; + case 0: + if (!l1->val) + return; + assoc = assocs[l1->assoc]; + line_size = l1->line_size; + lines_per_tag = l1->lines_per_tag; + size_in_kb = l1->size_in_kb; + break; + case 2: + if (!l2.val) + return; + assoc = assocs[l2.assoc]; + line_size = l2.line_size; + lines_per_tag = l2.lines_per_tag; + /* cpu_data has errata corrections for K7 applied */ + size_in_kb = __this_cpu_read(cpu_info.x86_cache_size); + break; + case 3: + if (!l3.val) + return; + assoc = assocs[l3.assoc]; + line_size = l3.line_size; + lines_per_tag = l3.lines_per_tag; + size_in_kb = l3.size_encoded * 512; + if (boot_cpu_has(X86_FEATURE_AMD_DCM)) { + size_in_kb = size_in_kb >> 1; + assoc = assoc >> 1; + } + break; + default: + return; + } + + eax->split.is_self_initializing = 1; + eax->split.type = types[leaf]; + eax->split.level = levels[leaf]; + eax->split.num_threads_sharing = 0; + eax->split.num_cores_on_die = __this_cpu_read(cpu_info.x86_max_cores) - 1; + + + if (assoc == 0xffff) + eax->split.is_fully_associative = 1; + ebx->split.coherency_line_size = line_size - 1; + ebx->split.ways_of_associativity = assoc - 1; + ebx->split.physical_line_partition = lines_per_tag - 1; + ecx->split.number_of_sets = (size_in_kb * 1024) / line_size / + (ebx->split.ways_of_associativity + 1) - 1; +} + +#if defined(CONFIG_AMD_NB) && defined(CONFIG_SYSFS) + +/* + * L3 cache descriptors + */ +static void amd_calc_l3_indices(struct amd_northbridge *nb) +{ + struct amd_l3_cache *l3 = &nb->l3_cache; + unsigned int sc0, sc1, sc2, sc3; + u32 val = 0; + + pci_read_config_dword(nb->misc, 0x1C4, &val); + + /* calculate subcache sizes */ + l3->subcaches[0] = sc0 = !(val & BIT(0)); + l3->subcaches[1] = sc1 = !(val & BIT(4)); + + if (boot_cpu_data.x86 == 0x15) { + l3->subcaches[0] = sc0 += !(val & BIT(1)); + l3->subcaches[1] = sc1 += !(val & BIT(5)); + } + + l3->subcaches[2] = sc2 = !(val & BIT(8)) + !(val & BIT(9)); + l3->subcaches[3] = sc3 = !(val & BIT(12)) + !(val & BIT(13)); + + l3->indices = (max(max3(sc0, sc1, sc2), sc3) << 10) - 1; +} + +/* + * check whether a slot used for disabling an L3 index is occupied. + * @l3: L3 cache descriptor + * @slot: slot number (0..1) + * + * @returns: the disabled index if used or negative value if slot free. + */ +static int amd_get_l3_disable_slot(struct amd_northbridge *nb, unsigned slot) +{ + unsigned int reg = 0; + + pci_read_config_dword(nb->misc, 0x1BC + slot * 4, ®); + + /* check whether this slot is activated already */ + if (reg & (3UL << 30)) + return reg & 0xfff; + + return -1; +} + +static ssize_t show_cache_disable(struct cacheinfo *this_leaf, char *buf, + unsigned int slot) +{ + int index; + struct amd_northbridge *nb = this_leaf->priv; + + index = amd_get_l3_disable_slot(nb, slot); + if (index >= 0) + return sprintf(buf, "%d\n", index); + + return sprintf(buf, "FREE\n"); +} + +#define SHOW_CACHE_DISABLE(slot) \ +static ssize_t \ +cache_disable_##slot##_show(struct device *dev, \ + struct device_attribute *attr, char *buf) \ +{ \ + struct cacheinfo *this_leaf = dev_get_drvdata(dev); \ + return show_cache_disable(this_leaf, buf, slot); \ +} +SHOW_CACHE_DISABLE(0) +SHOW_CACHE_DISABLE(1) + +static void amd_l3_disable_index(struct amd_northbridge *nb, int cpu, + unsigned slot, unsigned long idx) +{ + int i; + + idx |= BIT(30); + + /* + * disable index in all 4 subcaches + */ + for (i = 0; i < 4; i++) { + u32 reg = idx | (i << 20); + + if (!nb->l3_cache.subcaches[i]) + continue; + + pci_write_config_dword(nb->misc, 0x1BC + slot * 4, reg); + + /* + * We need to WBINVD on a core on the node containing the L3 + * cache which indices we disable therefore a simple wbinvd() + * is not sufficient. + */ + wbinvd_on_cpu(cpu); + + reg |= BIT(31); + pci_write_config_dword(nb->misc, 0x1BC + slot * 4, reg); + } +} + +/* + * disable a L3 cache index by using a disable-slot + * + * @l3: L3 cache descriptor + * @cpu: A CPU on the node containing the L3 cache + * @slot: slot number (0..1) + * @index: index to disable + * + * @return: 0 on success, error status on failure + */ +static int amd_set_l3_disable_slot(struct amd_northbridge *nb, int cpu, + unsigned slot, unsigned long index) +{ + int ret = 0; + + /* check if @slot is already used or the index is already disabled */ + ret = amd_get_l3_disable_slot(nb, slot); + if (ret >= 0) + return -EEXIST; + + if (index > nb->l3_cache.indices) + return -EINVAL; + + /* check whether the other slot has disabled the same index already */ + if (index == amd_get_l3_disable_slot(nb, !slot)) + return -EEXIST; + + amd_l3_disable_index(nb, cpu, slot, index); + + return 0; +} + +static ssize_t store_cache_disable(struct cacheinfo *this_leaf, + const char *buf, size_t count, + unsigned int slot) +{ + unsigned long val = 0; + int cpu, err = 0; + struct amd_northbridge *nb = this_leaf->priv; + + if (!capable(CAP_SYS_ADMIN)) + return -EPERM; + + cpu = cpumask_first(&this_leaf->shared_cpu_map); + + if (kstrtoul(buf, 10, &val) < 0) + return -EINVAL; + + err = amd_set_l3_disable_slot(nb, cpu, slot, val); + if (err) { + if (err == -EEXIST) + pr_warn("L3 slot %d in use/index already disabled!\n", + slot); + return err; + } + return count; +} + +#define STORE_CACHE_DISABLE(slot) \ +static ssize_t \ +cache_disable_##slot##_store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t count) \ +{ \ + struct cacheinfo *this_leaf = dev_get_drvdata(dev); \ + return store_cache_disable(this_leaf, buf, count, slot); \ +} +STORE_CACHE_DISABLE(0) +STORE_CACHE_DISABLE(1) + +static ssize_t subcaches_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cacheinfo *this_leaf = dev_get_drvdata(dev); + int cpu = cpumask_first(&this_leaf->shared_cpu_map); + + return sprintf(buf, "%x\n", amd_get_subcaches(cpu)); +} + +static ssize_t subcaches_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct cacheinfo *this_leaf = dev_get_drvdata(dev); + int cpu = cpumask_first(&this_leaf->shared_cpu_map); + unsigned long val; + + if (!capable(CAP_SYS_ADMIN)) + return -EPERM; + + if (kstrtoul(buf, 16, &val) < 0) + return -EINVAL; + + if (amd_set_subcaches(cpu, val)) + return -EINVAL; + + return count; +} + +static DEVICE_ATTR_RW(cache_disable_0); +static DEVICE_ATTR_RW(cache_disable_1); +static DEVICE_ATTR_RW(subcaches); + +static umode_t +cache_private_attrs_is_visible(struct kobject *kobj, + struct attribute *attr, int unused) +{ + struct device *dev = kobj_to_dev(kobj); + struct cacheinfo *this_leaf = dev_get_drvdata(dev); + umode_t mode = attr->mode; + + if (!this_leaf->priv) + return 0; + + if ((attr == &dev_attr_subcaches.attr) && + amd_nb_has_feature(AMD_NB_L3_PARTITIONING)) + return mode; + + if ((attr == &dev_attr_cache_disable_0.attr || + attr == &dev_attr_cache_disable_1.attr) && + amd_nb_has_feature(AMD_NB_L3_INDEX_DISABLE)) + return mode; + + return 0; +} + +static struct attribute_group cache_private_group = { + .is_visible = cache_private_attrs_is_visible, +}; + +static void init_amd_l3_attrs(void) +{ + int n = 1; + static struct attribute **amd_l3_attrs; + + if (amd_l3_attrs) /* already initialized */ + return; + + if (amd_nb_has_feature(AMD_NB_L3_INDEX_DISABLE)) + n += 2; + if (amd_nb_has_feature(AMD_NB_L3_PARTITIONING)) + n += 1; + + amd_l3_attrs = kcalloc(n, sizeof(*amd_l3_attrs), GFP_KERNEL); + if (!amd_l3_attrs) + return; + + n = 0; + if (amd_nb_has_feature(AMD_NB_L3_INDEX_DISABLE)) { + amd_l3_attrs[n++] = &dev_attr_cache_disable_0.attr; + amd_l3_attrs[n++] = &dev_attr_cache_disable_1.attr; + } + if (amd_nb_has_feature(AMD_NB_L3_PARTITIONING)) + amd_l3_attrs[n++] = &dev_attr_subcaches.attr; + + cache_private_group.attrs = amd_l3_attrs; +} + +const struct attribute_group * +cache_get_priv_group(struct cacheinfo *this_leaf) +{ + struct amd_northbridge *nb = this_leaf->priv; + + if (this_leaf->level < 3 || !nb) + return NULL; + + if (nb && nb->l3_cache.indices) + init_amd_l3_attrs(); + + return &cache_private_group; +} + +static void amd_init_l3_cache(struct _cpuid4_info_regs *this_leaf, int index) +{ + int node; + + /* only for L3, and not in virtualized environments */ + if (index < 3) + return; + + node = amd_get_nb_id(smp_processor_id()); + this_leaf->nb = node_to_amd_nb(node); + if (this_leaf->nb && !this_leaf->nb->l3_cache.indices) + amd_calc_l3_indices(this_leaf->nb); +} +#else +#define amd_init_l3_cache(x, y) +#endif /* CONFIG_AMD_NB && CONFIG_SYSFS */ + +static int +cpuid4_cache_lookup_regs(int index, struct _cpuid4_info_regs *this_leaf) +{ + union _cpuid4_leaf_eax eax; + union _cpuid4_leaf_ebx ebx; + union _cpuid4_leaf_ecx ecx; + unsigned edx; + + if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD) { + if (boot_cpu_has(X86_FEATURE_TOPOEXT)) + cpuid_count(0x8000001d, index, &eax.full, + &ebx.full, &ecx.full, &edx); + else + amd_cpuid4(index, &eax, &ebx, &ecx); + amd_init_l3_cache(this_leaf, index); + } else { + cpuid_count(4, index, &eax.full, &ebx.full, &ecx.full, &edx); + } + + if (eax.split.type == CTYPE_NULL) + return -EIO; /* better error ? */ + + this_leaf->eax = eax; + this_leaf->ebx = ebx; + this_leaf->ecx = ecx; + this_leaf->size = (ecx.split.number_of_sets + 1) * + (ebx.split.coherency_line_size + 1) * + (ebx.split.physical_line_partition + 1) * + (ebx.split.ways_of_associativity + 1); + return 0; +} + +static int find_num_cache_leaves(struct cpuinfo_x86 *c) +{ + unsigned int eax, ebx, ecx, edx, op; + union _cpuid4_leaf_eax cache_eax; + int i = -1; + + if (c->x86_vendor == X86_VENDOR_AMD) + op = 0x8000001d; + else + op = 4; + + do { + ++i; + /* Do cpuid(op) loop to find out num_cache_leaves */ + cpuid_count(op, i, &eax, &ebx, &ecx, &edx); + cache_eax.full = eax; + } while (cache_eax.split.type != CTYPE_NULL); + return i; +} + +void cacheinfo_amd_init_llc_id(struct cpuinfo_x86 *c, int cpu, u8 node_id) +{ + /* + * We may have multiple LLCs if L3 caches exist, so check if we + * have an L3 cache by looking at the L3 cache CPUID leaf. + */ + if (!cpuid_edx(0x80000006)) + return; + + if (c->x86 < 0x17) { + /* LLC is at the node level. */ + per_cpu(cpu_llc_id, cpu) = node_id; + } else if (c->x86 == 0x17 && + c->x86_model >= 0 && c->x86_model <= 0x1F) { + /* + * LLC is at the core complex level. + * Core complex ID is ApicId[3] for these processors. + */ + per_cpu(cpu_llc_id, cpu) = c->apicid >> 3; + } else { + /* + * LLC ID is calculated from the number of threads sharing the + * cache. + * */ + u32 eax, ebx, ecx, edx, num_sharing_cache = 0; + u32 llc_index = find_num_cache_leaves(c) - 1; + + cpuid_count(0x8000001d, llc_index, &eax, &ebx, &ecx, &edx); + if (eax) + num_sharing_cache = ((eax >> 14) & 0xfff) + 1; + + if (num_sharing_cache) { + int bits = get_count_order(num_sharing_cache) - 1; + + per_cpu(cpu_llc_id, cpu) = c->apicid >> bits; + } + } +} + +void init_amd_cacheinfo(struct cpuinfo_x86 *c) +{ + + if (boot_cpu_has(X86_FEATURE_TOPOEXT)) { + num_cache_leaves = find_num_cache_leaves(c); + } else if (c->extended_cpuid_level >= 0x80000006) { + if (cpuid_edx(0x80000006) & 0xf000) + num_cache_leaves = 4; + else + num_cache_leaves = 3; + } +} + +void init_intel_cacheinfo(struct cpuinfo_x86 *c) +{ + /* Cache sizes */ + unsigned int trace = 0, l1i = 0, l1d = 0, l2 = 0, l3 = 0; + unsigned int new_l1d = 0, new_l1i = 0; /* Cache sizes from cpuid(4) */ + unsigned int new_l2 = 0, new_l3 = 0, i; /* Cache sizes from cpuid(4) */ + unsigned int l2_id = 0, l3_id = 0, num_threads_sharing, index_msb; +#ifdef CONFIG_SMP + unsigned int cpu = c->cpu_index; +#endif + + if (c->cpuid_level > 3) { + static int is_initialized; + + if (is_initialized == 0) { + /* Init num_cache_leaves from boot CPU */ + num_cache_leaves = find_num_cache_leaves(c); + is_initialized++; + } + + /* + * Whenever possible use cpuid(4), deterministic cache + * parameters cpuid leaf to find the cache details + */ + for (i = 0; i < num_cache_leaves; i++) { + struct _cpuid4_info_regs this_leaf = {}; + int retval; + + retval = cpuid4_cache_lookup_regs(i, &this_leaf); + if (retval < 0) + continue; + + switch (this_leaf.eax.split.level) { + case 1: + if (this_leaf.eax.split.type == CTYPE_DATA) + new_l1d = this_leaf.size/1024; + else if (this_leaf.eax.split.type == CTYPE_INST) + new_l1i = this_leaf.size/1024; + break; + case 2: + new_l2 = this_leaf.size/1024; + num_threads_sharing = 1 + this_leaf.eax.split.num_threads_sharing; + index_msb = get_count_order(num_threads_sharing); + l2_id = c->apicid & ~((1 << index_msb) - 1); + break; + case 3: + new_l3 = this_leaf.size/1024; + num_threads_sharing = 1 + this_leaf.eax.split.num_threads_sharing; + index_msb = get_count_order(num_threads_sharing); + l3_id = c->apicid & ~((1 << index_msb) - 1); + break; + default: + break; + } + } + } + /* + * Don't use cpuid2 if cpuid4 is supported. For P4, we use cpuid2 for + * trace cache + */ + if ((num_cache_leaves == 0 || c->x86 == 15) && c->cpuid_level > 1) { + /* supports eax=2 call */ + int j, n; + unsigned int regs[4]; + unsigned char *dp = (unsigned char *)regs; + int only_trace = 0; + + if (num_cache_leaves != 0 && c->x86 == 15) + only_trace = 1; + + /* Number of times to iterate */ + n = cpuid_eax(2) & 0xFF; + + for (i = 0 ; i < n ; i++) { + cpuid(2, ®s[0], ®s[1], ®s[2], ®s[3]); + + /* If bit 31 is set, this is an unknown format */ + for (j = 0 ; j < 3 ; j++) + if (regs[j] & (1 << 31)) + regs[j] = 0; + + /* Byte 0 is level count, not a descriptor */ + for (j = 1 ; j < 16 ; j++) { + unsigned char des = dp[j]; + unsigned char k = 0; + + /* look up this descriptor in the table */ + while (cache_table[k].descriptor != 0) { + if (cache_table[k].descriptor == des) { + if (only_trace && cache_table[k].cache_type != LVL_TRACE) + break; + switch (cache_table[k].cache_type) { + case LVL_1_INST: + l1i += cache_table[k].size; + break; + case LVL_1_DATA: + l1d += cache_table[k].size; + break; + case LVL_2: + l2 += cache_table[k].size; + break; + case LVL_3: + l3 += cache_table[k].size; + break; + case LVL_TRACE: + trace += cache_table[k].size; + break; + } + + break; + } + + k++; + } + } + } + } + + if (new_l1d) + l1d = new_l1d; + + if (new_l1i) + l1i = new_l1i; + + if (new_l2) { + l2 = new_l2; +#ifdef CONFIG_SMP + per_cpu(cpu_llc_id, cpu) = l2_id; +#endif + } + + if (new_l3) { + l3 = new_l3; +#ifdef CONFIG_SMP + per_cpu(cpu_llc_id, cpu) = l3_id; +#endif + } + +#ifdef CONFIG_SMP + /* + * If cpu_llc_id is not yet set, this means cpuid_level < 4 which in + * turns means that the only possibility is SMT (as indicated in + * cpuid1). Since cpuid2 doesn't specify shared caches, and we know + * that SMT shares all caches, we can unconditionally set cpu_llc_id to + * c->phys_proc_id. + */ + if (per_cpu(cpu_llc_id, cpu) == BAD_APICID) + per_cpu(cpu_llc_id, cpu) = c->phys_proc_id; +#endif + + c->x86_cache_size = l3 ? l3 : (l2 ? l2 : (l1i+l1d)); + + if (!l2) + cpu_detect_cache_sizes(c); +} + +static int __cache_amd_cpumap_setup(unsigned int cpu, int index, + struct _cpuid4_info_regs *base) +{ + struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); + struct cacheinfo *this_leaf; + int i, sibling; + + /* + * For L3, always use the pre-calculated cpu_llc_shared_mask + * to derive shared_cpu_map. + */ + if (index == 3) { + for_each_cpu(i, cpu_llc_shared_mask(cpu)) { + this_cpu_ci = get_cpu_cacheinfo(i); + if (!this_cpu_ci->info_list) + continue; + this_leaf = this_cpu_ci->info_list + index; + for_each_cpu(sibling, cpu_llc_shared_mask(cpu)) { + if (!cpu_online(sibling)) + continue; + cpumask_set_cpu(sibling, + &this_leaf->shared_cpu_map); + } + } + } else if (boot_cpu_has(X86_FEATURE_TOPOEXT)) { + unsigned int apicid, nshared, first, last; + + nshared = base->eax.split.num_threads_sharing + 1; + apicid = cpu_data(cpu).apicid; + first = apicid - (apicid % nshared); + last = first + nshared - 1; + + for_each_online_cpu(i) { + this_cpu_ci = get_cpu_cacheinfo(i); + if (!this_cpu_ci->info_list) + continue; + + apicid = cpu_data(i).apicid; + if ((apicid < first) || (apicid > last)) + continue; + + this_leaf = this_cpu_ci->info_list + index; + + for_each_online_cpu(sibling) { + apicid = cpu_data(sibling).apicid; + if ((apicid < first) || (apicid > last)) + continue; + cpumask_set_cpu(sibling, + &this_leaf->shared_cpu_map); + } + } + } else + return 0; + + return 1; +} + +static void __cache_cpumap_setup(unsigned int cpu, int index, + struct _cpuid4_info_regs *base) +{ + struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); + struct cacheinfo *this_leaf, *sibling_leaf; + unsigned long num_threads_sharing; + int index_msb, i; + struct cpuinfo_x86 *c = &cpu_data(cpu); + + if (c->x86_vendor == X86_VENDOR_AMD) { + if (__cache_amd_cpumap_setup(cpu, index, base)) + return; + } + + this_leaf = this_cpu_ci->info_list + index; + num_threads_sharing = 1 + base->eax.split.num_threads_sharing; + + cpumask_set_cpu(cpu, &this_leaf->shared_cpu_map); + if (num_threads_sharing == 1) + return; + + index_msb = get_count_order(num_threads_sharing); + + for_each_online_cpu(i) + if (cpu_data(i).apicid >> index_msb == c->apicid >> index_msb) { + struct cpu_cacheinfo *sib_cpu_ci = get_cpu_cacheinfo(i); + + if (i == cpu || !sib_cpu_ci->info_list) + continue;/* skip if itself or no cacheinfo */ + sibling_leaf = sib_cpu_ci->info_list + index; + cpumask_set_cpu(i, &this_leaf->shared_cpu_map); + cpumask_set_cpu(cpu, &sibling_leaf->shared_cpu_map); + } +} + +static void ci_leaf_init(struct cacheinfo *this_leaf, + struct _cpuid4_info_regs *base) +{ + this_leaf->id = base->id; + this_leaf->attributes = CACHE_ID; + this_leaf->level = base->eax.split.level; + this_leaf->type = cache_type_map[base->eax.split.type]; + this_leaf->coherency_line_size = + base->ebx.split.coherency_line_size + 1; + this_leaf->ways_of_associativity = + base->ebx.split.ways_of_associativity + 1; + this_leaf->size = base->size; + this_leaf->number_of_sets = base->ecx.split.number_of_sets + 1; + this_leaf->physical_line_partition = + base->ebx.split.physical_line_partition + 1; + this_leaf->priv = base->nb; +} + +static int __init_cache_level(unsigned int cpu) +{ + struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); + + if (!num_cache_leaves) + return -ENOENT; + if (!this_cpu_ci) + return -EINVAL; + this_cpu_ci->num_levels = 3; + this_cpu_ci->num_leaves = num_cache_leaves; + return 0; +} + +/* + * The max shared threads number comes from CPUID.4:EAX[25-14] with input + * ECX as cache index. Then right shift apicid by the number's order to get + * cache id for this cache node. + */ +static void get_cache_id(int cpu, struct _cpuid4_info_regs *id4_regs) +{ + struct cpuinfo_x86 *c = &cpu_data(cpu); + unsigned long num_threads_sharing; + int index_msb; + + num_threads_sharing = 1 + id4_regs->eax.split.num_threads_sharing; + index_msb = get_count_order(num_threads_sharing); + id4_regs->id = c->apicid >> index_msb; +} + +static int __populate_cache_leaves(unsigned int cpu) +{ + unsigned int idx, ret; + struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); + struct cacheinfo *this_leaf = this_cpu_ci->info_list; + struct _cpuid4_info_regs id4_regs = {}; + + for (idx = 0; idx < this_cpu_ci->num_leaves; idx++) { + ret = cpuid4_cache_lookup_regs(idx, &id4_regs); + if (ret) + return ret; + get_cache_id(cpu, &id4_regs); + ci_leaf_init(this_leaf++, &id4_regs); + __cache_cpumap_setup(cpu, idx, &id4_regs); + } + this_cpu_ci->cpu_map_populated = true; + + return 0; +} + +DEFINE_SMP_CALL_CACHE_FUNCTION(init_cache_level) +DEFINE_SMP_CALL_CACHE_FUNCTION(populate_cache_leaves) diff --git a/arch/x86/kernel/cpu/centaur.c b/arch/x86/kernel/cpu/centaur.c index e5ec0f11c0de..14433ff5b828 100644 --- a/arch/x86/kernel/cpu/centaur.c +++ b/arch/x86/kernel/cpu/centaur.c @@ -18,6 +18,13 @@ #define RNG_ENABLED (1 << 3) #define RNG_ENABLE (1 << 6) /* MSR_VIA_RNG */ +#define X86_VMX_FEATURE_PROC_CTLS_TPR_SHADOW 0x00200000 +#define X86_VMX_FEATURE_PROC_CTLS_VNMI 0x00400000 +#define X86_VMX_FEATURE_PROC_CTLS_2ND_CTLS 0x80000000 +#define X86_VMX_FEATURE_PROC_CTLS2_VIRT_APIC 0x00000001 +#define X86_VMX_FEATURE_PROC_CTLS2_EPT 0x00000002 +#define X86_VMX_FEATURE_PROC_CTLS2_VPID 0x00000020 + static void init_c3(struct cpuinfo_x86 *c) { u32 lo, hi; @@ -112,6 +119,31 @@ static void early_init_centaur(struct cpuinfo_x86 *c) } } +static void centaur_detect_vmx_virtcap(struct cpuinfo_x86 *c) +{ + u32 vmx_msr_low, vmx_msr_high, msr_ctl, msr_ctl2; + + rdmsr(MSR_IA32_VMX_PROCBASED_CTLS, vmx_msr_low, vmx_msr_high); + msr_ctl = vmx_msr_high | vmx_msr_low; + + if (msr_ctl & X86_VMX_FEATURE_PROC_CTLS_TPR_SHADOW) + set_cpu_cap(c, X86_FEATURE_TPR_SHADOW); + if (msr_ctl & X86_VMX_FEATURE_PROC_CTLS_VNMI) + set_cpu_cap(c, X86_FEATURE_VNMI); + if (msr_ctl & X86_VMX_FEATURE_PROC_CTLS_2ND_CTLS) { + rdmsr(MSR_IA32_VMX_PROCBASED_CTLS2, + vmx_msr_low, vmx_msr_high); + msr_ctl2 = vmx_msr_high | vmx_msr_low; + if ((msr_ctl2 & X86_VMX_FEATURE_PROC_CTLS2_VIRT_APIC) && + (msr_ctl & X86_VMX_FEATURE_PROC_CTLS_TPR_SHADOW)) + set_cpu_cap(c, X86_FEATURE_FLEXPRIORITY); + if (msr_ctl2 & X86_VMX_FEATURE_PROC_CTLS2_EPT) + set_cpu_cap(c, X86_FEATURE_EPT); + if (msr_ctl2 & X86_VMX_FEATURE_PROC_CTLS2_VPID) + set_cpu_cap(c, X86_FEATURE_VPID); + } +} + static void init_centaur(struct cpuinfo_x86 *c) { #ifdef CONFIG_X86_32 @@ -128,6 +160,24 @@ static void init_centaur(struct cpuinfo_x86 *c) clear_cpu_cap(c, 0*32+31); #endif early_init_centaur(c); + init_intel_cacheinfo(c); + detect_num_cpu_cores(c); +#ifdef CONFIG_X86_32 + detect_ht(c); +#endif + + if (c->cpuid_level > 9) { + unsigned int eax = cpuid_eax(10); + + /* + * Check for version and the number of counters + * Version(eax[7:0]) can't be 0; + * Counters(eax[15:8]) should be greater than 1; + */ + if ((eax & 0xff) && (((eax >> 8) & 0xff) > 1)) + set_cpu_cap(c, X86_FEATURE_ARCH_PERFMON); + } + switch (c->x86) { #ifdef CONFIG_X86_32 case 5: @@ -199,6 +249,9 @@ static void init_centaur(struct cpuinfo_x86 *c) #ifdef CONFIG_X86_64 set_cpu_cap(c, X86_FEATURE_LFENCE_RDTSC); #endif + + if (cpu_has(c, X86_FEATURE_VMX)) + centaur_detect_vmx_virtcap(c); } #ifdef CONFIG_X86_32 diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 38276f58d3bf..0df7151cfef4 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -66,6 +66,13 @@ cpumask_var_t cpu_callin_mask; /* representing cpus for which sibling maps can be computed */ cpumask_var_t cpu_sibling_setup_mask; +/* Number of siblings per CPU package */ +int smp_num_siblings = 1; +EXPORT_SYMBOL(smp_num_siblings); + +/* Last level cache ID of each logical CPU */ +DEFINE_PER_CPU_READ_MOSTLY(u16, cpu_llc_id) = BAD_APICID; + /* correctly size the local cpu masks */ void __init setup_cpu_local_masks(void) { @@ -577,6 +584,19 @@ static void get_model_name(struct cpuinfo_x86 *c) *(s + 1) = '\0'; } +void detect_num_cpu_cores(struct cpuinfo_x86 *c) +{ + unsigned int eax, ebx, ecx, edx; + + c->x86_max_cores = 1; + if (!IS_ENABLED(CONFIG_SMP) || c->cpuid_level < 4) + return; + + cpuid_count(4, 0, &eax, &ebx, &ecx, &edx); + if (eax & 0x1f) + c->x86_max_cores = (eax >> 26) + 1; +} + void cpu_detect_cache_sizes(struct cpuinfo_x86 *c) { unsigned int n, dummy, ebx, ecx, edx, l2size; @@ -783,6 +803,12 @@ static void init_speculation_control(struct cpuinfo_x86 *c) set_cpu_cap(c, X86_FEATURE_STIBP); set_cpu_cap(c, X86_FEATURE_MSR_SPEC_CTRL); } + + if (cpu_has(c, X86_FEATURE_AMD_SSBD)) { + set_cpu_cap(c, X86_FEATURE_SSBD); + set_cpu_cap(c, X86_FEATURE_MSR_SPEC_CTRL); + clear_cpu_cap(c, X86_FEATURE_VIRT_SSBD); + } } void get_cpu_cap(struct cpuinfo_x86 *c) @@ -972,7 +998,8 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c) rdmsrl(MSR_IA32_ARCH_CAPABILITIES, ia32_cap); if (!x86_match_cpu(cpu_no_spec_store_bypass) && - !(ia32_cap & ARCH_CAP_SSB_NO)) + !(ia32_cap & ARCH_CAP_SSB_NO) && + !cpu_has(c, X86_FEATURE_AMD_SSB_NO)) setup_force_cpu_bug(X86_BUG_SPEC_STORE_BYPASS); if (x86_match_cpu(cpu_no_meltdown)) @@ -1044,6 +1071,21 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c) */ setup_clear_cpu_cap(X86_FEATURE_PCID); #endif + + /* + * Later in the boot process pgtable_l5_enabled() relies on + * cpu_feature_enabled(X86_FEATURE_LA57). If 5-level paging is not + * enabled by this point we need to clear the feature bit to avoid + * false-positives at the later stage. + * + * pgtable_l5_enabled() can be false here for several reasons: + * - 5-level paging is disabled compile-time; + * - it's 32-bit kernel; + * - machine doesn't support 5-level paging; + * - user specified 'no5lvl' in kernel command line. + */ + if (!pgtable_l5_enabled()) + setup_clear_cpu_cap(X86_FEATURE_LA57); } void __init early_cpu_init(void) @@ -1557,7 +1599,7 @@ DEFINE_PER_CPU(unsigned long, cpu_current_top_of_stack) = (unsigned long)&init_thread_union + THREAD_SIZE; EXPORT_PER_CPU_SYMBOL(cpu_current_top_of_stack); -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR DEFINE_PER_CPU_ALIGNED(struct stack_canary, stack_canary); #endif diff --git a/arch/x86/kernel/cpu/cpu.h b/arch/x86/kernel/cpu/cpu.h index 37672d299e35..38216f678fc3 100644 --- a/arch/x86/kernel/cpu/cpu.h +++ b/arch/x86/kernel/cpu/cpu.h @@ -47,6 +47,16 @@ extern const struct cpu_dev *const __x86_cpu_dev_start[], extern void get_cpu_cap(struct cpuinfo_x86 *c); extern void cpu_detect_cache_sizes(struct cpuinfo_x86 *c); +extern void init_scattered_cpuid_features(struct cpuinfo_x86 *c); +extern u32 get_scattered_cpuid_leaf(unsigned int level, + unsigned int sub_leaf, + enum cpuid_regs_idx reg); +extern void init_intel_cacheinfo(struct cpuinfo_x86 *c); +extern void init_amd_cacheinfo(struct cpuinfo_x86 *c); + +extern void detect_num_cpu_cores(struct cpuinfo_x86 *c); +extern int detect_extended_topology(struct cpuinfo_x86 *c); +extern void detect_ht(struct cpuinfo_x86 *c); unsigned int aperfmperf_get_khz(int cpu); diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c index 577e7f7ae273..eb75564f2d25 100644 --- a/arch/x86/kernel/cpu/intel.c +++ b/arch/x86/kernel/cpu/intel.c @@ -456,24 +456,6 @@ static void srat_detect_node(struct cpuinfo_x86 *c) #endif } -/* - * find out the number of processor cores on the die - */ -static int intel_num_cpu_cores(struct cpuinfo_x86 *c) -{ - unsigned int eax, ebx, ecx, edx; - - if (!IS_ENABLED(CONFIG_SMP) || c->cpuid_level < 4) - return 1; - - /* Intel has a non-standard dependency on %ecx for this CPUID level. */ - cpuid_count(4, 0, &eax, &ebx, &ecx, &edx); - if (eax & 0x1f) - return (eax >> 26) + 1; - else - return 1; -} - static void detect_vmx_virtcap(struct cpuinfo_x86 *c) { /* Intel VMX MSR indicated features */ @@ -656,8 +638,6 @@ static void init_intel_misc_features(struct cpuinfo_x86 *c) static void init_intel(struct cpuinfo_x86 *c) { - unsigned int l2 = 0; - early_init_intel(c); intel_workarounds(c); @@ -674,19 +654,13 @@ static void init_intel(struct cpuinfo_x86 *c) * let's use the legacy cpuid vector 0x1 and 0x4 for topology * detection. */ - c->x86_max_cores = intel_num_cpu_cores(c); + detect_num_cpu_cores(c); #ifdef CONFIG_X86_32 detect_ht(c); #endif } - l2 = init_intel_cacheinfo(c); - - /* Detect legacy cache sizes if init_intel_cacheinfo did not */ - if (l2 == 0) { - cpu_detect_cache_sizes(c); - l2 = c->x86_cache_size; - } + init_intel_cacheinfo(c); if (c->cpuid_level > 9) { unsigned eax = cpuid_eax(10); @@ -699,7 +673,8 @@ static void init_intel(struct cpuinfo_x86 *c) set_cpu_cap(c, X86_FEATURE_LFENCE_RDTSC); if (boot_cpu_has(X86_FEATURE_DS)) { - unsigned int l1; + unsigned int l1, l2; + rdmsr(MSR_IA32_MISC_ENABLE, l1, l2); if (!(l1 & (1<<11))) set_cpu_cap(c, X86_FEATURE_BTS); @@ -727,6 +702,7 @@ static void init_intel(struct cpuinfo_x86 *c) * Dixon is NOT a Celeron. */ if (c->x86 == 6) { + unsigned int l2 = c->x86_cache_size; char *p = NULL; switch (c->x86_model) { diff --git a/arch/x86/kernel/cpu/intel_cacheinfo.c b/arch/x86/kernel/cpu/intel_cacheinfo.c deleted file mode 100644 index 54d04d574148..000000000000 --- a/arch/x86/kernel/cpu/intel_cacheinfo.c +++ /dev/null @@ -1,968 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * Routines to identify caches on Intel CPU. - * - * Changes: - * Venkatesh Pallipadi : Adding cache identification through cpuid(4) - * Ashok Raj : Work with CPU hotplug infrastructure. - * Andi Kleen / Andreas Herrmann : CPUID4 emulation on AMD. - */ - -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include - -#define LVL_1_INST 1 -#define LVL_1_DATA 2 -#define LVL_2 3 -#define LVL_3 4 -#define LVL_TRACE 5 - -struct _cache_table { - unsigned char descriptor; - char cache_type; - short size; -}; - -#define MB(x) ((x) * 1024) - -/* All the cache descriptor types we care about (no TLB or - trace cache entries) */ - -static const struct _cache_table cache_table[] = -{ - { 0x06, LVL_1_INST, 8 }, /* 4-way set assoc, 32 byte line size */ - { 0x08, LVL_1_INST, 16 }, /* 4-way set assoc, 32 byte line size */ - { 0x09, LVL_1_INST, 32 }, /* 4-way set assoc, 64 byte line size */ - { 0x0a, LVL_1_DATA, 8 }, /* 2 way set assoc, 32 byte line size */ - { 0x0c, LVL_1_DATA, 16 }, /* 4-way set assoc, 32 byte line size */ - { 0x0d, LVL_1_DATA, 16 }, /* 4-way set assoc, 64 byte line size */ - { 0x0e, LVL_1_DATA, 24 }, /* 6-way set assoc, 64 byte line size */ - { 0x21, LVL_2, 256 }, /* 8-way set assoc, 64 byte line size */ - { 0x22, LVL_3, 512 }, /* 4-way set assoc, sectored cache, 64 byte line size */ - { 0x23, LVL_3, MB(1) }, /* 8-way set assoc, sectored cache, 64 byte line size */ - { 0x25, LVL_3, MB(2) }, /* 8-way set assoc, sectored cache, 64 byte line size */ - { 0x29, LVL_3, MB(4) }, /* 8-way set assoc, sectored cache, 64 byte line size */ - { 0x2c, LVL_1_DATA, 32 }, /* 8-way set assoc, 64 byte line size */ - { 0x30, LVL_1_INST, 32 }, /* 8-way set assoc, 64 byte line size */ - { 0x39, LVL_2, 128 }, /* 4-way set assoc, sectored cache, 64 byte line size */ - { 0x3a, LVL_2, 192 }, /* 6-way set assoc, sectored cache, 64 byte line size */ - { 0x3b, LVL_2, 128 }, /* 2-way set assoc, sectored cache, 64 byte line size */ - { 0x3c, LVL_2, 256 }, /* 4-way set assoc, sectored cache, 64 byte line size */ - { 0x3d, LVL_2, 384 }, /* 6-way set assoc, sectored cache, 64 byte line size */ - { 0x3e, LVL_2, 512 }, /* 4-way set assoc, sectored cache, 64 byte line size */ - { 0x3f, LVL_2, 256 }, /* 2-way set assoc, 64 byte line size */ - { 0x41, LVL_2, 128 }, /* 4-way set assoc, 32 byte line size */ - { 0x42, LVL_2, 256 }, /* 4-way set assoc, 32 byte line size */ - { 0x43, LVL_2, 512 }, /* 4-way set assoc, 32 byte line size */ - { 0x44, LVL_2, MB(1) }, /* 4-way set assoc, 32 byte line size */ - { 0x45, LVL_2, MB(2) }, /* 4-way set assoc, 32 byte line size */ - { 0x46, LVL_3, MB(4) }, /* 4-way set assoc, 64 byte line size */ - { 0x47, LVL_3, MB(8) }, /* 8-way set assoc, 64 byte line size */ - { 0x48, LVL_2, MB(3) }, /* 12-way set assoc, 64 byte line size */ - { 0x49, LVL_3, MB(4) }, /* 16-way set assoc, 64 byte line size */ - { 0x4a, LVL_3, MB(6) }, /* 12-way set assoc, 64 byte line size */ - { 0x4b, LVL_3, MB(8) }, /* 16-way set assoc, 64 byte line size */ - { 0x4c, LVL_3, MB(12) }, /* 12-way set assoc, 64 byte line size */ - { 0x4d, LVL_3, MB(16) }, /* 16-way set assoc, 64 byte line size */ - { 0x4e, LVL_2, MB(6) }, /* 24-way set assoc, 64 byte line size */ - { 0x60, LVL_1_DATA, 16 }, /* 8-way set assoc, sectored cache, 64 byte line size */ - { 0x66, LVL_1_DATA, 8 }, /* 4-way set assoc, sectored cache, 64 byte line size */ - { 0x67, LVL_1_DATA, 16 }, /* 4-way set assoc, sectored cache, 64 byte line size */ - { 0x68, LVL_1_DATA, 32 }, /* 4-way set assoc, sectored cache, 64 byte line size */ - { 0x70, LVL_TRACE, 12 }, /* 8-way set assoc */ - { 0x71, LVL_TRACE, 16 }, /* 8-way set assoc */ - { 0x72, LVL_TRACE, 32 }, /* 8-way set assoc */ - { 0x73, LVL_TRACE, 64 }, /* 8-way set assoc */ - { 0x78, LVL_2, MB(1) }, /* 4-way set assoc, 64 byte line size */ - { 0x79, LVL_2, 128 }, /* 8-way set assoc, sectored cache, 64 byte line size */ - { 0x7a, LVL_2, 256 }, /* 8-way set assoc, sectored cache, 64 byte line size */ - { 0x7b, LVL_2, 512 }, /* 8-way set assoc, sectored cache, 64 byte line size */ - { 0x7c, LVL_2, MB(1) }, /* 8-way set assoc, sectored cache, 64 byte line size */ - { 0x7d, LVL_2, MB(2) }, /* 8-way set assoc, 64 byte line size */ - { 0x7f, LVL_2, 512 }, /* 2-way set assoc, 64 byte line size */ - { 0x80, LVL_2, 512 }, /* 8-way set assoc, 64 byte line size */ - { 0x82, LVL_2, 256 }, /* 8-way set assoc, 32 byte line size */ - { 0x83, LVL_2, 512 }, /* 8-way set assoc, 32 byte line size */ - { 0x84, LVL_2, MB(1) }, /* 8-way set assoc, 32 byte line size */ - { 0x85, LVL_2, MB(2) }, /* 8-way set assoc, 32 byte line size */ - { 0x86, LVL_2, 512 }, /* 4-way set assoc, 64 byte line size */ - { 0x87, LVL_2, MB(1) }, /* 8-way set assoc, 64 byte line size */ - { 0xd0, LVL_3, 512 }, /* 4-way set assoc, 64 byte line size */ - { 0xd1, LVL_3, MB(1) }, /* 4-way set assoc, 64 byte line size */ - { 0xd2, LVL_3, MB(2) }, /* 4-way set assoc, 64 byte line size */ - { 0xd6, LVL_3, MB(1) }, /* 8-way set assoc, 64 byte line size */ - { 0xd7, LVL_3, MB(2) }, /* 8-way set assoc, 64 byte line size */ - { 0xd8, LVL_3, MB(4) }, /* 12-way set assoc, 64 byte line size */ - { 0xdc, LVL_3, MB(2) }, /* 12-way set assoc, 64 byte line size */ - { 0xdd, LVL_3, MB(4) }, /* 12-way set assoc, 64 byte line size */ - { 0xde, LVL_3, MB(8) }, /* 12-way set assoc, 64 byte line size */ - { 0xe2, LVL_3, MB(2) }, /* 16-way set assoc, 64 byte line size */ - { 0xe3, LVL_3, MB(4) }, /* 16-way set assoc, 64 byte line size */ - { 0xe4, LVL_3, MB(8) }, /* 16-way set assoc, 64 byte line size */ - { 0xea, LVL_3, MB(12) }, /* 24-way set assoc, 64 byte line size */ - { 0xeb, LVL_3, MB(18) }, /* 24-way set assoc, 64 byte line size */ - { 0xec, LVL_3, MB(24) }, /* 24-way set assoc, 64 byte line size */ - { 0x00, 0, 0} -}; - - -enum _cache_type { - CTYPE_NULL = 0, - CTYPE_DATA = 1, - CTYPE_INST = 2, - CTYPE_UNIFIED = 3 -}; - -union _cpuid4_leaf_eax { - struct { - enum _cache_type type:5; - unsigned int level:3; - unsigned int is_self_initializing:1; - unsigned int is_fully_associative:1; - unsigned int reserved:4; - unsigned int num_threads_sharing:12; - unsigned int num_cores_on_die:6; - } split; - u32 full; -}; - -union _cpuid4_leaf_ebx { - struct { - unsigned int coherency_line_size:12; - unsigned int physical_line_partition:10; - unsigned int ways_of_associativity:10; - } split; - u32 full; -}; - -union _cpuid4_leaf_ecx { - struct { - unsigned int number_of_sets:32; - } split; - u32 full; -}; - -struct _cpuid4_info_regs { - union _cpuid4_leaf_eax eax; - union _cpuid4_leaf_ebx ebx; - union _cpuid4_leaf_ecx ecx; - unsigned int id; - unsigned long size; - struct amd_northbridge *nb; -}; - -static unsigned short num_cache_leaves; - -/* AMD doesn't have CPUID4. Emulate it here to report the same - information to the user. This makes some assumptions about the machine: - L2 not shared, no SMT etc. that is currently true on AMD CPUs. - - In theory the TLBs could be reported as fake type (they are in "dummy"). - Maybe later */ -union l1_cache { - struct { - unsigned line_size:8; - unsigned lines_per_tag:8; - unsigned assoc:8; - unsigned size_in_kb:8; - }; - unsigned val; -}; - -union l2_cache { - struct { - unsigned line_size:8; - unsigned lines_per_tag:4; - unsigned assoc:4; - unsigned size_in_kb:16; - }; - unsigned val; -}; - -union l3_cache { - struct { - unsigned line_size:8; - unsigned lines_per_tag:4; - unsigned assoc:4; - unsigned res:2; - unsigned size_encoded:14; - }; - unsigned val; -}; - -static const unsigned short assocs[] = { - [1] = 1, - [2] = 2, - [4] = 4, - [6] = 8, - [8] = 16, - [0xa] = 32, - [0xb] = 48, - [0xc] = 64, - [0xd] = 96, - [0xe] = 128, - [0xf] = 0xffff /* fully associative - no way to show this currently */ -}; - -static const unsigned char levels[] = { 1, 1, 2, 3 }; -static const unsigned char types[] = { 1, 2, 3, 3 }; - -static const enum cache_type cache_type_map[] = { - [CTYPE_NULL] = CACHE_TYPE_NOCACHE, - [CTYPE_DATA] = CACHE_TYPE_DATA, - [CTYPE_INST] = CACHE_TYPE_INST, - [CTYPE_UNIFIED] = CACHE_TYPE_UNIFIED, -}; - -static void -amd_cpuid4(int leaf, union _cpuid4_leaf_eax *eax, - union _cpuid4_leaf_ebx *ebx, - union _cpuid4_leaf_ecx *ecx) -{ - unsigned dummy; - unsigned line_size, lines_per_tag, assoc, size_in_kb; - union l1_cache l1i, l1d; - union l2_cache l2; - union l3_cache l3; - union l1_cache *l1 = &l1d; - - eax->full = 0; - ebx->full = 0; - ecx->full = 0; - - cpuid(0x80000005, &dummy, &dummy, &l1d.val, &l1i.val); - cpuid(0x80000006, &dummy, &dummy, &l2.val, &l3.val); - - switch (leaf) { - case 1: - l1 = &l1i; - case 0: - if (!l1->val) - return; - assoc = assocs[l1->assoc]; - line_size = l1->line_size; - lines_per_tag = l1->lines_per_tag; - size_in_kb = l1->size_in_kb; - break; - case 2: - if (!l2.val) - return; - assoc = assocs[l2.assoc]; - line_size = l2.line_size; - lines_per_tag = l2.lines_per_tag; - /* cpu_data has errata corrections for K7 applied */ - size_in_kb = __this_cpu_read(cpu_info.x86_cache_size); - break; - case 3: - if (!l3.val) - return; - assoc = assocs[l3.assoc]; - line_size = l3.line_size; - lines_per_tag = l3.lines_per_tag; - size_in_kb = l3.size_encoded * 512; - if (boot_cpu_has(X86_FEATURE_AMD_DCM)) { - size_in_kb = size_in_kb >> 1; - assoc = assoc >> 1; - } - break; - default: - return; - } - - eax->split.is_self_initializing = 1; - eax->split.type = types[leaf]; - eax->split.level = levels[leaf]; - eax->split.num_threads_sharing = 0; - eax->split.num_cores_on_die = __this_cpu_read(cpu_info.x86_max_cores) - 1; - - - if (assoc == 0xffff) - eax->split.is_fully_associative = 1; - ebx->split.coherency_line_size = line_size - 1; - ebx->split.ways_of_associativity = assoc - 1; - ebx->split.physical_line_partition = lines_per_tag - 1; - ecx->split.number_of_sets = (size_in_kb * 1024) / line_size / - (ebx->split.ways_of_associativity + 1) - 1; -} - -#if defined(CONFIG_AMD_NB) && defined(CONFIG_SYSFS) - -/* - * L3 cache descriptors - */ -static void amd_calc_l3_indices(struct amd_northbridge *nb) -{ - struct amd_l3_cache *l3 = &nb->l3_cache; - unsigned int sc0, sc1, sc2, sc3; - u32 val = 0; - - pci_read_config_dword(nb->misc, 0x1C4, &val); - - /* calculate subcache sizes */ - l3->subcaches[0] = sc0 = !(val & BIT(0)); - l3->subcaches[1] = sc1 = !(val & BIT(4)); - - if (boot_cpu_data.x86 == 0x15) { - l3->subcaches[0] = sc0 += !(val & BIT(1)); - l3->subcaches[1] = sc1 += !(val & BIT(5)); - } - - l3->subcaches[2] = sc2 = !(val & BIT(8)) + !(val & BIT(9)); - l3->subcaches[3] = sc3 = !(val & BIT(12)) + !(val & BIT(13)); - - l3->indices = (max(max3(sc0, sc1, sc2), sc3) << 10) - 1; -} - -/* - * check whether a slot used for disabling an L3 index is occupied. - * @l3: L3 cache descriptor - * @slot: slot number (0..1) - * - * @returns: the disabled index if used or negative value if slot free. - */ -static int amd_get_l3_disable_slot(struct amd_northbridge *nb, unsigned slot) -{ - unsigned int reg = 0; - - pci_read_config_dword(nb->misc, 0x1BC + slot * 4, ®); - - /* check whether this slot is activated already */ - if (reg & (3UL << 30)) - return reg & 0xfff; - - return -1; -} - -static ssize_t show_cache_disable(struct cacheinfo *this_leaf, char *buf, - unsigned int slot) -{ - int index; - struct amd_northbridge *nb = this_leaf->priv; - - index = amd_get_l3_disable_slot(nb, slot); - if (index >= 0) - return sprintf(buf, "%d\n", index); - - return sprintf(buf, "FREE\n"); -} - -#define SHOW_CACHE_DISABLE(slot) \ -static ssize_t \ -cache_disable_##slot##_show(struct device *dev, \ - struct device_attribute *attr, char *buf) \ -{ \ - struct cacheinfo *this_leaf = dev_get_drvdata(dev); \ - return show_cache_disable(this_leaf, buf, slot); \ -} -SHOW_CACHE_DISABLE(0) -SHOW_CACHE_DISABLE(1) - -static void amd_l3_disable_index(struct amd_northbridge *nb, int cpu, - unsigned slot, unsigned long idx) -{ - int i; - - idx |= BIT(30); - - /* - * disable index in all 4 subcaches - */ - for (i = 0; i < 4; i++) { - u32 reg = idx | (i << 20); - - if (!nb->l3_cache.subcaches[i]) - continue; - - pci_write_config_dword(nb->misc, 0x1BC + slot * 4, reg); - - /* - * We need to WBINVD on a core on the node containing the L3 - * cache which indices we disable therefore a simple wbinvd() - * is not sufficient. - */ - wbinvd_on_cpu(cpu); - - reg |= BIT(31); - pci_write_config_dword(nb->misc, 0x1BC + slot * 4, reg); - } -} - -/* - * disable a L3 cache index by using a disable-slot - * - * @l3: L3 cache descriptor - * @cpu: A CPU on the node containing the L3 cache - * @slot: slot number (0..1) - * @index: index to disable - * - * @return: 0 on success, error status on failure - */ -static int amd_set_l3_disable_slot(struct amd_northbridge *nb, int cpu, - unsigned slot, unsigned long index) -{ - int ret = 0; - - /* check if @slot is already used or the index is already disabled */ - ret = amd_get_l3_disable_slot(nb, slot); - if (ret >= 0) - return -EEXIST; - - if (index > nb->l3_cache.indices) - return -EINVAL; - - /* check whether the other slot has disabled the same index already */ - if (index == amd_get_l3_disable_slot(nb, !slot)) - return -EEXIST; - - amd_l3_disable_index(nb, cpu, slot, index); - - return 0; -} - -static ssize_t store_cache_disable(struct cacheinfo *this_leaf, - const char *buf, size_t count, - unsigned int slot) -{ - unsigned long val = 0; - int cpu, err = 0; - struct amd_northbridge *nb = this_leaf->priv; - - if (!capable(CAP_SYS_ADMIN)) - return -EPERM; - - cpu = cpumask_first(&this_leaf->shared_cpu_map); - - if (kstrtoul(buf, 10, &val) < 0) - return -EINVAL; - - err = amd_set_l3_disable_slot(nb, cpu, slot, val); - if (err) { - if (err == -EEXIST) - pr_warn("L3 slot %d in use/index already disabled!\n", - slot); - return err; - } - return count; -} - -#define STORE_CACHE_DISABLE(slot) \ -static ssize_t \ -cache_disable_##slot##_store(struct device *dev, \ - struct device_attribute *attr, \ - const char *buf, size_t count) \ -{ \ - struct cacheinfo *this_leaf = dev_get_drvdata(dev); \ - return store_cache_disable(this_leaf, buf, count, slot); \ -} -STORE_CACHE_DISABLE(0) -STORE_CACHE_DISABLE(1) - -static ssize_t subcaches_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - struct cacheinfo *this_leaf = dev_get_drvdata(dev); - int cpu = cpumask_first(&this_leaf->shared_cpu_map); - - return sprintf(buf, "%x\n", amd_get_subcaches(cpu)); -} - -static ssize_t subcaches_store(struct device *dev, - struct device_attribute *attr, - const char *buf, size_t count) -{ - struct cacheinfo *this_leaf = dev_get_drvdata(dev); - int cpu = cpumask_first(&this_leaf->shared_cpu_map); - unsigned long val; - - if (!capable(CAP_SYS_ADMIN)) - return -EPERM; - - if (kstrtoul(buf, 16, &val) < 0) - return -EINVAL; - - if (amd_set_subcaches(cpu, val)) - return -EINVAL; - - return count; -} - -static DEVICE_ATTR_RW(cache_disable_0); -static DEVICE_ATTR_RW(cache_disable_1); -static DEVICE_ATTR_RW(subcaches); - -static umode_t -cache_private_attrs_is_visible(struct kobject *kobj, - struct attribute *attr, int unused) -{ - struct device *dev = kobj_to_dev(kobj); - struct cacheinfo *this_leaf = dev_get_drvdata(dev); - umode_t mode = attr->mode; - - if (!this_leaf->priv) - return 0; - - if ((attr == &dev_attr_subcaches.attr) && - amd_nb_has_feature(AMD_NB_L3_PARTITIONING)) - return mode; - - if ((attr == &dev_attr_cache_disable_0.attr || - attr == &dev_attr_cache_disable_1.attr) && - amd_nb_has_feature(AMD_NB_L3_INDEX_DISABLE)) - return mode; - - return 0; -} - -static struct attribute_group cache_private_group = { - .is_visible = cache_private_attrs_is_visible, -}; - -static void init_amd_l3_attrs(void) -{ - int n = 1; - static struct attribute **amd_l3_attrs; - - if (amd_l3_attrs) /* already initialized */ - return; - - if (amd_nb_has_feature(AMD_NB_L3_INDEX_DISABLE)) - n += 2; - if (amd_nb_has_feature(AMD_NB_L3_PARTITIONING)) - n += 1; - - amd_l3_attrs = kcalloc(n, sizeof(*amd_l3_attrs), GFP_KERNEL); - if (!amd_l3_attrs) - return; - - n = 0; - if (amd_nb_has_feature(AMD_NB_L3_INDEX_DISABLE)) { - amd_l3_attrs[n++] = &dev_attr_cache_disable_0.attr; - amd_l3_attrs[n++] = &dev_attr_cache_disable_1.attr; - } - if (amd_nb_has_feature(AMD_NB_L3_PARTITIONING)) - amd_l3_attrs[n++] = &dev_attr_subcaches.attr; - - cache_private_group.attrs = amd_l3_attrs; -} - -const struct attribute_group * -cache_get_priv_group(struct cacheinfo *this_leaf) -{ - struct amd_northbridge *nb = this_leaf->priv; - - if (this_leaf->level < 3 || !nb) - return NULL; - - if (nb && nb->l3_cache.indices) - init_amd_l3_attrs(); - - return &cache_private_group; -} - -static void amd_init_l3_cache(struct _cpuid4_info_regs *this_leaf, int index) -{ - int node; - - /* only for L3, and not in virtualized environments */ - if (index < 3) - return; - - node = amd_get_nb_id(smp_processor_id()); - this_leaf->nb = node_to_amd_nb(node); - if (this_leaf->nb && !this_leaf->nb->l3_cache.indices) - amd_calc_l3_indices(this_leaf->nb); -} -#else -#define amd_init_l3_cache(x, y) -#endif /* CONFIG_AMD_NB && CONFIG_SYSFS */ - -static int -cpuid4_cache_lookup_regs(int index, struct _cpuid4_info_regs *this_leaf) -{ - union _cpuid4_leaf_eax eax; - union _cpuid4_leaf_ebx ebx; - union _cpuid4_leaf_ecx ecx; - unsigned edx; - - if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD) { - if (boot_cpu_has(X86_FEATURE_TOPOEXT)) - cpuid_count(0x8000001d, index, &eax.full, - &ebx.full, &ecx.full, &edx); - else - amd_cpuid4(index, &eax, &ebx, &ecx); - amd_init_l3_cache(this_leaf, index); - } else { - cpuid_count(4, index, &eax.full, &ebx.full, &ecx.full, &edx); - } - - if (eax.split.type == CTYPE_NULL) - return -EIO; /* better error ? */ - - this_leaf->eax = eax; - this_leaf->ebx = ebx; - this_leaf->ecx = ecx; - this_leaf->size = (ecx.split.number_of_sets + 1) * - (ebx.split.coherency_line_size + 1) * - (ebx.split.physical_line_partition + 1) * - (ebx.split.ways_of_associativity + 1); - return 0; -} - -static int find_num_cache_leaves(struct cpuinfo_x86 *c) -{ - unsigned int eax, ebx, ecx, edx, op; - union _cpuid4_leaf_eax cache_eax; - int i = -1; - - if (c->x86_vendor == X86_VENDOR_AMD) - op = 0x8000001d; - else - op = 4; - - do { - ++i; - /* Do cpuid(op) loop to find out num_cache_leaves */ - cpuid_count(op, i, &eax, &ebx, &ecx, &edx); - cache_eax.full = eax; - } while (cache_eax.split.type != CTYPE_NULL); - return i; -} - -void init_amd_cacheinfo(struct cpuinfo_x86 *c) -{ - - if (boot_cpu_has(X86_FEATURE_TOPOEXT)) { - num_cache_leaves = find_num_cache_leaves(c); - } else if (c->extended_cpuid_level >= 0x80000006) { - if (cpuid_edx(0x80000006) & 0xf000) - num_cache_leaves = 4; - else - num_cache_leaves = 3; - } -} - -unsigned int init_intel_cacheinfo(struct cpuinfo_x86 *c) -{ - /* Cache sizes */ - unsigned int trace = 0, l1i = 0, l1d = 0, l2 = 0, l3 = 0; - unsigned int new_l1d = 0, new_l1i = 0; /* Cache sizes from cpuid(4) */ - unsigned int new_l2 = 0, new_l3 = 0, i; /* Cache sizes from cpuid(4) */ - unsigned int l2_id = 0, l3_id = 0, num_threads_sharing, index_msb; -#ifdef CONFIG_SMP - unsigned int cpu = c->cpu_index; -#endif - - if (c->cpuid_level > 3) { - static int is_initialized; - - if (is_initialized == 0) { - /* Init num_cache_leaves from boot CPU */ - num_cache_leaves = find_num_cache_leaves(c); - is_initialized++; - } - - /* - * Whenever possible use cpuid(4), deterministic cache - * parameters cpuid leaf to find the cache details - */ - for (i = 0; i < num_cache_leaves; i++) { - struct _cpuid4_info_regs this_leaf = {}; - int retval; - - retval = cpuid4_cache_lookup_regs(i, &this_leaf); - if (retval < 0) - continue; - - switch (this_leaf.eax.split.level) { - case 1: - if (this_leaf.eax.split.type == CTYPE_DATA) - new_l1d = this_leaf.size/1024; - else if (this_leaf.eax.split.type == CTYPE_INST) - new_l1i = this_leaf.size/1024; - break; - case 2: - new_l2 = this_leaf.size/1024; - num_threads_sharing = 1 + this_leaf.eax.split.num_threads_sharing; - index_msb = get_count_order(num_threads_sharing); - l2_id = c->apicid & ~((1 << index_msb) - 1); - break; - case 3: - new_l3 = this_leaf.size/1024; - num_threads_sharing = 1 + this_leaf.eax.split.num_threads_sharing; - index_msb = get_count_order(num_threads_sharing); - l3_id = c->apicid & ~((1 << index_msb) - 1); - break; - default: - break; - } - } - } - /* - * Don't use cpuid2 if cpuid4 is supported. For P4, we use cpuid2 for - * trace cache - */ - if ((num_cache_leaves == 0 || c->x86 == 15) && c->cpuid_level > 1) { - /* supports eax=2 call */ - int j, n; - unsigned int regs[4]; - unsigned char *dp = (unsigned char *)regs; - int only_trace = 0; - - if (num_cache_leaves != 0 && c->x86 == 15) - only_trace = 1; - - /* Number of times to iterate */ - n = cpuid_eax(2) & 0xFF; - - for (i = 0 ; i < n ; i++) { - cpuid(2, ®s[0], ®s[1], ®s[2], ®s[3]); - - /* If bit 31 is set, this is an unknown format */ - for (j = 0 ; j < 3 ; j++) - if (regs[j] & (1 << 31)) - regs[j] = 0; - - /* Byte 0 is level count, not a descriptor */ - for (j = 1 ; j < 16 ; j++) { - unsigned char des = dp[j]; - unsigned char k = 0; - - /* look up this descriptor in the table */ - while (cache_table[k].descriptor != 0) { - if (cache_table[k].descriptor == des) { - if (only_trace && cache_table[k].cache_type != LVL_TRACE) - break; - switch (cache_table[k].cache_type) { - case LVL_1_INST: - l1i += cache_table[k].size; - break; - case LVL_1_DATA: - l1d += cache_table[k].size; - break; - case LVL_2: - l2 += cache_table[k].size; - break; - case LVL_3: - l3 += cache_table[k].size; - break; - case LVL_TRACE: - trace += cache_table[k].size; - break; - } - - break; - } - - k++; - } - } - } - } - - if (new_l1d) - l1d = new_l1d; - - if (new_l1i) - l1i = new_l1i; - - if (new_l2) { - l2 = new_l2; -#ifdef CONFIG_SMP - per_cpu(cpu_llc_id, cpu) = l2_id; -#endif - } - - if (new_l3) { - l3 = new_l3; -#ifdef CONFIG_SMP - per_cpu(cpu_llc_id, cpu) = l3_id; -#endif - } - -#ifdef CONFIG_SMP - /* - * If cpu_llc_id is not yet set, this means cpuid_level < 4 which in - * turns means that the only possibility is SMT (as indicated in - * cpuid1). Since cpuid2 doesn't specify shared caches, and we know - * that SMT shares all caches, we can unconditionally set cpu_llc_id to - * c->phys_proc_id. - */ - if (per_cpu(cpu_llc_id, cpu) == BAD_APICID) - per_cpu(cpu_llc_id, cpu) = c->phys_proc_id; -#endif - - c->x86_cache_size = l3 ? l3 : (l2 ? l2 : (l1i+l1d)); - - return l2; -} - -static int __cache_amd_cpumap_setup(unsigned int cpu, int index, - struct _cpuid4_info_regs *base) -{ - struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); - struct cacheinfo *this_leaf; - int i, sibling; - - /* - * For L3, always use the pre-calculated cpu_llc_shared_mask - * to derive shared_cpu_map. - */ - if (index == 3) { - for_each_cpu(i, cpu_llc_shared_mask(cpu)) { - this_cpu_ci = get_cpu_cacheinfo(i); - if (!this_cpu_ci->info_list) - continue; - this_leaf = this_cpu_ci->info_list + index; - for_each_cpu(sibling, cpu_llc_shared_mask(cpu)) { - if (!cpu_online(sibling)) - continue; - cpumask_set_cpu(sibling, - &this_leaf->shared_cpu_map); - } - } - } else if (boot_cpu_has(X86_FEATURE_TOPOEXT)) { - unsigned int apicid, nshared, first, last; - - nshared = base->eax.split.num_threads_sharing + 1; - apicid = cpu_data(cpu).apicid; - first = apicid - (apicid % nshared); - last = first + nshared - 1; - - for_each_online_cpu(i) { - this_cpu_ci = get_cpu_cacheinfo(i); - if (!this_cpu_ci->info_list) - continue; - - apicid = cpu_data(i).apicid; - if ((apicid < first) || (apicid > last)) - continue; - - this_leaf = this_cpu_ci->info_list + index; - - for_each_online_cpu(sibling) { - apicid = cpu_data(sibling).apicid; - if ((apicid < first) || (apicid > last)) - continue; - cpumask_set_cpu(sibling, - &this_leaf->shared_cpu_map); - } - } - } else - return 0; - - return 1; -} - -static void __cache_cpumap_setup(unsigned int cpu, int index, - struct _cpuid4_info_regs *base) -{ - struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); - struct cacheinfo *this_leaf, *sibling_leaf; - unsigned long num_threads_sharing; - int index_msb, i; - struct cpuinfo_x86 *c = &cpu_data(cpu); - - if (c->x86_vendor == X86_VENDOR_AMD) { - if (__cache_amd_cpumap_setup(cpu, index, base)) - return; - } - - this_leaf = this_cpu_ci->info_list + index; - num_threads_sharing = 1 + base->eax.split.num_threads_sharing; - - cpumask_set_cpu(cpu, &this_leaf->shared_cpu_map); - if (num_threads_sharing == 1) - return; - - index_msb = get_count_order(num_threads_sharing); - - for_each_online_cpu(i) - if (cpu_data(i).apicid >> index_msb == c->apicid >> index_msb) { - struct cpu_cacheinfo *sib_cpu_ci = get_cpu_cacheinfo(i); - - if (i == cpu || !sib_cpu_ci->info_list) - continue;/* skip if itself or no cacheinfo */ - sibling_leaf = sib_cpu_ci->info_list + index; - cpumask_set_cpu(i, &this_leaf->shared_cpu_map); - cpumask_set_cpu(cpu, &sibling_leaf->shared_cpu_map); - } -} - -static void ci_leaf_init(struct cacheinfo *this_leaf, - struct _cpuid4_info_regs *base) -{ - this_leaf->id = base->id; - this_leaf->attributes = CACHE_ID; - this_leaf->level = base->eax.split.level; - this_leaf->type = cache_type_map[base->eax.split.type]; - this_leaf->coherency_line_size = - base->ebx.split.coherency_line_size + 1; - this_leaf->ways_of_associativity = - base->ebx.split.ways_of_associativity + 1; - this_leaf->size = base->size; - this_leaf->number_of_sets = base->ecx.split.number_of_sets + 1; - this_leaf->physical_line_partition = - base->ebx.split.physical_line_partition + 1; - this_leaf->priv = base->nb; -} - -static int __init_cache_level(unsigned int cpu) -{ - struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); - - if (!num_cache_leaves) - return -ENOENT; - if (!this_cpu_ci) - return -EINVAL; - this_cpu_ci->num_levels = 3; - this_cpu_ci->num_leaves = num_cache_leaves; - return 0; -} - -/* - * The max shared threads number comes from CPUID.4:EAX[25-14] with input - * ECX as cache index. Then right shift apicid by the number's order to get - * cache id for this cache node. - */ -static void get_cache_id(int cpu, struct _cpuid4_info_regs *id4_regs) -{ - struct cpuinfo_x86 *c = &cpu_data(cpu); - unsigned long num_threads_sharing; - int index_msb; - - num_threads_sharing = 1 + id4_regs->eax.split.num_threads_sharing; - index_msb = get_count_order(num_threads_sharing); - id4_regs->id = c->apicid >> index_msb; -} - -static int __populate_cache_leaves(unsigned int cpu) -{ - unsigned int idx, ret; - struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); - struct cacheinfo *this_leaf = this_cpu_ci->info_list; - struct _cpuid4_info_regs id4_regs = {}; - - for (idx = 0; idx < this_cpu_ci->num_leaves; idx++) { - ret = cpuid4_cache_lookup_regs(idx, &id4_regs); - if (ret) - return ret; - get_cache_id(cpu, &id4_regs); - ci_leaf_init(this_leaf++, &id4_regs); - __cache_cpumap_setup(cpu, idx, &id4_regs); - } - this_cpu_ci->cpu_map_populated = true; - - return 0; -} - -DEFINE_SMP_CALL_CACHE_FUNCTION(init_cache_level) -DEFINE_SMP_CALL_CACHE_FUNCTION(populate_cache_leaves) diff --git a/arch/x86/kernel/cpu/intel_rdt.c b/arch/x86/kernel/cpu/intel_rdt.c index 589b948e6e01..ec4754f81cbd 100644 --- a/arch/x86/kernel/cpu/intel_rdt.c +++ b/arch/x86/kernel/cpu/intel_rdt.c @@ -33,8 +33,8 @@ #include #include "intel_rdt.h" -#define MAX_MBA_BW 100u #define MBA_IS_LINEAR 0x4 +#define MBA_MAX_MBPS U32_MAX /* Mutex to protect rdtgroup access. */ DEFINE_MUTEX(rdtgroup_mutex); @@ -178,7 +178,7 @@ struct rdt_resource rdt_resources_all[] = { .msr_update = mba_wrmsr, .cache_level = 3, .parse_ctrlval = parse_bw, - .format_str = "%d=%*d", + .format_str = "%d=%*u", .fflags = RFTYPE_RES_MB, }, }; @@ -230,6 +230,14 @@ static inline void cache_alloc_hsw_probe(void) rdt_alloc_capable = true; } +bool is_mba_sc(struct rdt_resource *r) +{ + if (!r) + return rdt_resources_all[RDT_RESOURCE_MBA].membw.mba_sc; + + return r->membw.mba_sc; +} + /* * rdt_get_mb_table() - get a mapping of bandwidth(b/w) percentage values * exposed to user interface and the h/w understandable delay values. @@ -341,7 +349,7 @@ static int get_cache_id(int cpu, int level) * that can be written to QOS_MSRs. * There are currently no SKUs which support non linear delay values. */ -static u32 delay_bw_map(unsigned long bw, struct rdt_resource *r) +u32 delay_bw_map(unsigned long bw, struct rdt_resource *r) { if (r->membw.delay_linear) return MAX_MBA_BW - bw; @@ -431,25 +439,40 @@ struct rdt_domain *rdt_find_domain(struct rdt_resource *r, int id, return NULL; } +void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm) +{ + int i; + + /* + * Initialize the Control MSRs to having no control. + * For Cache Allocation: Set all bits in cbm + * For Memory Allocation: Set b/w requested to 100% + * and the bandwidth in MBps to U32_MAX + */ + for (i = 0; i < r->num_closid; i++, dc++, dm++) { + *dc = r->default_ctrl; + *dm = MBA_MAX_MBPS; + } +} + static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d) { struct msr_param m; - u32 *dc; - int i; + u32 *dc, *dm; dc = kmalloc_array(r->num_closid, sizeof(*d->ctrl_val), GFP_KERNEL); if (!dc) return -ENOMEM; - d->ctrl_val = dc; + dm = kmalloc_array(r->num_closid, sizeof(*d->mbps_val), GFP_KERNEL); + if (!dm) { + kfree(dc); + return -ENOMEM; + } - /* - * Initialize the Control MSRs to having no control. - * For Cache Allocation: Set all bits in cbm - * For Memory Allocation: Set b/w requested to 100 - */ - for (i = 0; i < r->num_closid; i++, dc++) - *dc = r->default_ctrl; + d->ctrl_val = dc; + d->mbps_val = dm; + setup_default_ctrlval(r, dc, dm); m.low = 0; m.high = r->num_closid; @@ -588,6 +611,7 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r) } kfree(d->ctrl_val); + kfree(d->mbps_val); kfree(d->rmid_busy_llc); kfree(d->mbm_total); kfree(d->mbm_local); @@ -821,6 +845,8 @@ static __init void rdt_quirks(void) case INTEL_FAM6_SKYLAKE_X: if (boot_cpu_data.x86_stepping <= 4) set_rdt_options("!cmt,!mbmtotal,!mbmlocal,!l3cat"); + else + set_rdt_options("!l3cat"); } } diff --git a/arch/x86/kernel/cpu/intel_rdt.h b/arch/x86/kernel/cpu/intel_rdt.h index 3fd7a70ee04a..39752825e376 100644 --- a/arch/x86/kernel/cpu/intel_rdt.h +++ b/arch/x86/kernel/cpu/intel_rdt.h @@ -28,6 +28,7 @@ #define MBM_CNTR_WIDTH 24 #define MBM_OVERFLOW_INTERVAL 1000 +#define MAX_MBA_BW 100u #define RMID_VAL_ERROR BIT_ULL(63) #define RMID_VAL_UNAVAIL BIT_ULL(62) @@ -180,10 +181,20 @@ struct rftype { * struct mbm_state - status for each MBM counter in each domain * @chunks: Total data moved (multiply by rdt_group.mon_scale to get bytes) * @prev_msr Value of IA32_QM_CTR for this RMID last time we read it + * @chunks_bw Total local data moved. Used for bandwidth calculation + * @prev_bw_msr:Value of previous IA32_QM_CTR for bandwidth counting + * @prev_bw The most recent bandwidth in MBps + * @delta_bw Difference between the current and previous bandwidth + * @delta_comp Indicates whether to compute the delta_bw */ struct mbm_state { u64 chunks; u64 prev_msr; + u64 chunks_bw; + u64 prev_bw_msr; + u32 prev_bw; + u32 delta_bw; + bool delta_comp; }; /** @@ -202,6 +213,7 @@ struct mbm_state { * @cqm_work_cpu: * worker cpu for CQM h/w counters * @ctrl_val: array of cache or mem ctrl values (indexed by CLOSID) + * @mbps_val: When mba_sc is enabled, this holds the bandwidth in MBps * @new_ctrl: new ctrl value to be loaded * @have_new_ctrl: did user provide new_ctrl for this domain */ @@ -217,6 +229,7 @@ struct rdt_domain { int mbm_work_cpu; int cqm_work_cpu; u32 *ctrl_val; + u32 *mbps_val; u32 new_ctrl; bool have_new_ctrl; }; @@ -259,6 +272,7 @@ struct rdt_cache { * @min_bw: Minimum memory bandwidth percentage user can request * @bw_gran: Granularity at which the memory bandwidth is allocated * @delay_linear: True if memory B/W delay is in linear scale + * @mba_sc: True if MBA software controller(mba_sc) is enabled * @mb_map: Mapping of memory B/W percentage to memory B/W delay */ struct rdt_membw { @@ -266,6 +280,7 @@ struct rdt_membw { u32 min_bw; u32 bw_gran; u32 delay_linear; + bool mba_sc; u32 *mb_map; }; @@ -445,6 +460,9 @@ void mon_event_read(struct rmid_read *rr, struct rdt_domain *d, void mbm_setup_overflow_handler(struct rdt_domain *dom, unsigned long delay_ms); void mbm_handle_overflow(struct work_struct *work); +bool is_mba_sc(struct rdt_resource *r); +void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm); +u32 delay_bw_map(unsigned long bw, struct rdt_resource *r); void cqm_setup_limbo_handler(struct rdt_domain *dom, unsigned long delay_ms); void cqm_handle_limbo(struct work_struct *work); bool has_busy_rmid(struct rdt_resource *r, struct rdt_domain *d); diff --git a/arch/x86/kernel/cpu/intel_rdt_ctrlmondata.c b/arch/x86/kernel/cpu/intel_rdt_ctrlmondata.c index 23e1d5c249c6..116d57b248d3 100644 --- a/arch/x86/kernel/cpu/intel_rdt_ctrlmondata.c +++ b/arch/x86/kernel/cpu/intel_rdt_ctrlmondata.c @@ -53,7 +53,8 @@ static bool bw_validate(char *buf, unsigned long *data, struct rdt_resource *r) return false; } - if (bw < r->membw.min_bw || bw > r->default_ctrl) { + if ((bw < r->membw.min_bw || bw > r->default_ctrl) && + !is_mba_sc(r)) { rdt_last_cmd_printf("MB value %ld out of range [%d,%d]\n", bw, r->membw.min_bw, r->default_ctrl); return false; @@ -179,6 +180,8 @@ static int update_domains(struct rdt_resource *r, int closid) struct msr_param msr_param; cpumask_var_t cpu_mask; struct rdt_domain *d; + bool mba_sc; + u32 *dc; int cpu; if (!zalloc_cpumask_var(&cpu_mask, GFP_KERNEL)) @@ -188,13 +191,20 @@ static int update_domains(struct rdt_resource *r, int closid) msr_param.high = msr_param.low + 1; msr_param.res = r; + mba_sc = is_mba_sc(r); list_for_each_entry(d, &r->domains, list) { - if (d->have_new_ctrl && d->new_ctrl != d->ctrl_val[closid]) { + dc = !mba_sc ? d->ctrl_val : d->mbps_val; + if (d->have_new_ctrl && d->new_ctrl != dc[closid]) { cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask); - d->ctrl_val[closid] = d->new_ctrl; + dc[closid] = d->new_ctrl; } } - if (cpumask_empty(cpu_mask)) + + /* + * Avoid writing the control msr with control values when + * MBA software controller is enabled + */ + if (cpumask_empty(cpu_mask) || mba_sc) goto done; cpu = get_cpu(); /* Update CBM on this cpu if it's in cpu_mask. */ @@ -282,13 +292,17 @@ static void show_doms(struct seq_file *s, struct rdt_resource *r, int closid) { struct rdt_domain *dom; bool sep = false; + u32 ctrl_val; seq_printf(s, "%*s:", max_name_width, r->name); list_for_each_entry(dom, &r->domains, list) { if (sep) seq_puts(s, ";"); + + ctrl_val = (!is_mba_sc(r) ? dom->ctrl_val[closid] : + dom->mbps_val[closid]); seq_printf(s, r->format_str, dom->id, max_data_width, - dom->ctrl_val[closid]); + ctrl_val); sep = true; } seq_puts(s, "\n"); diff --git a/arch/x86/kernel/cpu/intel_rdt_monitor.c b/arch/x86/kernel/cpu/intel_rdt_monitor.c index 681450eee428..b0f3aed76b75 100644 --- a/arch/x86/kernel/cpu/intel_rdt_monitor.c +++ b/arch/x86/kernel/cpu/intel_rdt_monitor.c @@ -225,10 +225,18 @@ void free_rmid(u32 rmid) list_add_tail(&entry->list, &rmid_free_lru); } +static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr) +{ + u64 shift = 64 - MBM_CNTR_WIDTH, chunks; + + chunks = (cur_msr << shift) - (prev_msr << shift); + return chunks >>= shift; +} + static int __mon_event_count(u32 rmid, struct rmid_read *rr) { - u64 chunks, shift, tval; struct mbm_state *m; + u64 chunks, tval; tval = __rmid_read(rmid, rr->evtid); if (tval & (RMID_VAL_ERROR | RMID_VAL_UNAVAIL)) { @@ -254,14 +262,12 @@ static int __mon_event_count(u32 rmid, struct rmid_read *rr) } if (rr->first) { - m->prev_msr = tval; - m->chunks = 0; + memset(m, 0, sizeof(struct mbm_state)); + m->prev_bw_msr = m->prev_msr = tval; return 0; } - shift = 64 - MBM_CNTR_WIDTH; - chunks = (tval << shift) - (m->prev_msr << shift); - chunks >>= shift; + chunks = mbm_overflow_count(m->prev_msr, tval); m->chunks += chunks; m->prev_msr = tval; @@ -269,6 +275,32 @@ static int __mon_event_count(u32 rmid, struct rmid_read *rr) return 0; } +/* + * Supporting function to calculate the memory bandwidth + * and delta bandwidth in MBps. + */ +static void mbm_bw_count(u32 rmid, struct rmid_read *rr) +{ + struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3]; + struct mbm_state *m = &rr->d->mbm_local[rmid]; + u64 tval, cur_bw, chunks; + + tval = __rmid_read(rmid, rr->evtid); + if (tval & (RMID_VAL_ERROR | RMID_VAL_UNAVAIL)) + return; + + chunks = mbm_overflow_count(m->prev_bw_msr, tval); + m->chunks_bw += chunks; + m->chunks = m->chunks_bw; + cur_bw = (chunks * r->mon_scale) >> 20; + + if (m->delta_comp) + m->delta_bw = abs(cur_bw - m->prev_bw); + m->delta_comp = false; + m->prev_bw = cur_bw; + m->prev_bw_msr = tval; +} + /* * This is called via IPI to read the CQM/MBM counters * on a domain. @@ -297,6 +329,118 @@ void mon_event_count(void *info) } } +/* + * Feedback loop for MBA software controller (mba_sc) + * + * mba_sc is a feedback loop where we periodically read MBM counters and + * adjust the bandwidth percentage values via the IA32_MBA_THRTL_MSRs so + * that: + * + * current bandwdith(cur_bw) < user specified bandwidth(user_bw) + * + * This uses the MBM counters to measure the bandwidth and MBA throttle + * MSRs to control the bandwidth for a particular rdtgrp. It builds on the + * fact that resctrl rdtgroups have both monitoring and control. + * + * The frequency of the checks is 1s and we just tag along the MBM overflow + * timer. Having 1s interval makes the calculation of bandwidth simpler. + * + * Although MBA's goal is to restrict the bandwidth to a maximum, there may + * be a need to increase the bandwidth to avoid uncecessarily restricting + * the L2 <-> L3 traffic. + * + * Since MBA controls the L2 external bandwidth where as MBM measures the + * L3 external bandwidth the following sequence could lead to such a + * situation. + * + * Consider an rdtgroup which had high L3 <-> memory traffic in initial + * phases -> mba_sc kicks in and reduced bandwidth percentage values -> but + * after some time rdtgroup has mostly L2 <-> L3 traffic. + * + * In this case we may restrict the rdtgroup's L2 <-> L3 traffic as its + * throttle MSRs already have low percentage values. To avoid + * unnecessarily restricting such rdtgroups, we also increase the bandwidth. + */ +static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_domain *dom_mbm) +{ + u32 closid, rmid, cur_msr, cur_msr_val, new_msr_val; + struct mbm_state *pmbm_data, *cmbm_data; + u32 cur_bw, delta_bw, user_bw; + struct rdt_resource *r_mba; + struct rdt_domain *dom_mba; + struct list_head *head; + struct rdtgroup *entry; + + r_mba = &rdt_resources_all[RDT_RESOURCE_MBA]; + closid = rgrp->closid; + rmid = rgrp->mon.rmid; + pmbm_data = &dom_mbm->mbm_local[rmid]; + + dom_mba = get_domain_from_cpu(smp_processor_id(), r_mba); + if (!dom_mba) { + pr_warn_once("Failure to get domain for MBA update\n"); + return; + } + + cur_bw = pmbm_data->prev_bw; + user_bw = dom_mba->mbps_val[closid]; + delta_bw = pmbm_data->delta_bw; + cur_msr_val = dom_mba->ctrl_val[closid]; + + /* + * For Ctrl groups read data from child monitor groups. + */ + head = &rgrp->mon.crdtgrp_list; + list_for_each_entry(entry, head, mon.crdtgrp_list) { + cmbm_data = &dom_mbm->mbm_local[entry->mon.rmid]; + cur_bw += cmbm_data->prev_bw; + delta_bw += cmbm_data->delta_bw; + } + + /* + * Scale up/down the bandwidth linearly for the ctrl group. The + * bandwidth step is the bandwidth granularity specified by the + * hardware. + * + * The delta_bw is used when increasing the bandwidth so that we + * dont alternately increase and decrease the control values + * continuously. + * + * For ex: consider cur_bw = 90MBps, user_bw = 100MBps and if + * bandwidth step is 20MBps(> user_bw - cur_bw), we would keep + * switching between 90 and 110 continuously if we only check + * cur_bw < user_bw. + */ + if (cur_msr_val > r_mba->membw.min_bw && user_bw < cur_bw) { + new_msr_val = cur_msr_val - r_mba->membw.bw_gran; + } else if (cur_msr_val < MAX_MBA_BW && + (user_bw > (cur_bw + delta_bw))) { + new_msr_val = cur_msr_val + r_mba->membw.bw_gran; + } else { + return; + } + + cur_msr = r_mba->msr_base + closid; + wrmsrl(cur_msr, delay_bw_map(new_msr_val, r_mba)); + dom_mba->ctrl_val[closid] = new_msr_val; + + /* + * Delta values are updated dynamically package wise for each + * rdtgrp everytime the throttle MSR changes value. + * + * This is because (1)the increase in bandwidth is not perfectly + * linear and only "approximately" linear even when the hardware + * says it is linear.(2)Also since MBA is a core specific + * mechanism, the delta values vary based on number of cores used + * by the rdtgrp. + */ + pmbm_data->delta_comp = true; + list_for_each_entry(entry, head, mon.crdtgrp_list) { + cmbm_data = &dom_mbm->mbm_local[entry->mon.rmid]; + cmbm_data->delta_comp = true; + } +} + static void mbm_update(struct rdt_domain *d, int rmid) { struct rmid_read rr; @@ -314,7 +458,16 @@ static void mbm_update(struct rdt_domain *d, int rmid) } if (is_mbm_local_enabled()) { rr.evtid = QOS_L3_MBM_LOCAL_EVENT_ID; - __mon_event_count(rmid, &rr); + + /* + * Call the MBA software controller only for the + * control groups and when user has enabled + * the software controller explicitly. + */ + if (!is_mba_sc(NULL)) + __mon_event_count(rmid, &rr); + else + mbm_bw_count(rmid, &rr); } } @@ -385,6 +538,9 @@ void mbm_handle_overflow(struct work_struct *work) head = &prgrp->mon.crdtgrp_list; list_for_each_entry(crgrp, head, mon.crdtgrp_list) mbm_update(d, crgrp->mon.rmid); + + if (is_mba_sc(NULL)) + update_mba_bw(prgrp, d); } schedule_delayed_work_on(cpu, &d->mbm_over, delay); diff --git a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c index fca759d272a1..749856a2e736 100644 --- a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c +++ b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c @@ -1005,6 +1005,11 @@ static void l2_qos_cfg_update(void *arg) wrmsrl(IA32_L2_QOS_CFG, *enable ? L2_QOS_CDP_ENABLE : 0ULL); } +static inline bool is_mba_linear(void) +{ + return rdt_resources_all[RDT_RESOURCE_MBA].membw.delay_linear; +} + static int set_cache_qos_cfg(int level, bool enable) { void (*update)(void *arg); @@ -1041,6 +1046,28 @@ static int set_cache_qos_cfg(int level, bool enable) return 0; } +/* + * Enable or disable the MBA software controller + * which helps user specify bandwidth in MBps. + * MBA software controller is supported only if + * MBM is supported and MBA is in linear scale. + */ +static int set_mba_sc(bool mba_sc) +{ + struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_MBA]; + struct rdt_domain *d; + + if (!is_mbm_enabled() || !is_mba_linear() || + mba_sc == is_mba_sc(r)) + return -EINVAL; + + r->membw.mba_sc = mba_sc; + list_for_each_entry(d, &r->domains, list) + setup_default_ctrlval(r, d->ctrl_val, d->mbps_val); + + return 0; +} + static int cdp_enable(int level, int data_type, int code_type) { struct rdt_resource *r_ldata = &rdt_resources_all[data_type]; @@ -1123,6 +1150,10 @@ static int parse_rdtgroupfs_options(char *data) ret = cdpl2_enable(); if (ret) goto out; + } else if (!strcmp(token, "mba_MBps")) { + ret = set_mba_sc(true); + if (ret) + goto out; } else { ret = -EINVAL; goto out; @@ -1445,6 +1476,8 @@ static void rdt_kill_sb(struct super_block *sb) cpus_read_lock(); mutex_lock(&rdtgroup_mutex); + set_mba_sc(false); + /*Put everything back to default values. */ for_each_alloc_enabled_rdt_resource(r) reset_all_ctrls(r); diff --git a/arch/x86/kernel/cpu/mcheck/mce-inject.c b/arch/x86/kernel/cpu/mcheck/mce-inject.c index 475cb4f5f14f..c805a06e14c3 100644 --- a/arch/x86/kernel/cpu/mcheck/mce-inject.c +++ b/arch/x86/kernel/cpu/mcheck/mce-inject.c @@ -48,7 +48,7 @@ static struct dentry *dfs_inj; static u8 n_banks; -#define MAX_FLAG_OPT_SIZE 3 +#define MAX_FLAG_OPT_SIZE 4 #define NBCFG 0x44 enum injection_type { diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c index 42cf2880d0ed..e4cf6ff1c2e1 100644 --- a/arch/x86/kernel/cpu/mcheck/mce.c +++ b/arch/x86/kernel/cpu/mcheck/mce.c @@ -1457,7 +1457,7 @@ static int __mcheck_cpu_mce_banks_init(void) int i; u8 num_banks = mca_cfg.banks; - mce_banks = kzalloc(num_banks * sizeof(struct mce_bank), GFP_KERNEL); + mce_banks = kcalloc(num_banks, sizeof(struct mce_bank), GFP_KERNEL); if (!mce_banks) return -ENOMEM; @@ -1727,6 +1727,21 @@ static void __mcheck_cpu_init_early(struct cpuinfo_x86 *c) } } +static void mce_centaur_feature_init(struct cpuinfo_x86 *c) +{ + struct mca_config *cfg = &mca_cfg; + + /* + * All newer Centaur CPUs support MCE broadcasting. Enable + * synchronization with a one second timeout. + */ + if ((c->x86 == 6 && c->x86_model == 0xf && c->x86_stepping >= 0xe) || + c->x86 > 6) { + if (cfg->monarch_timeout < 0) + cfg->monarch_timeout = USEC_PER_SEC; + } +} + static void __mcheck_cpu_init_vendor(struct cpuinfo_x86 *c) { switch (c->x86_vendor) { @@ -1739,6 +1754,9 @@ static void __mcheck_cpu_init_vendor(struct cpuinfo_x86 *c) mce_amd_feature_init(c); break; } + case X86_VENDOR_CENTAUR: + mce_centaur_feature_init(c); + break; default: break; diff --git a/arch/x86/kernel/cpu/mcheck/mce_amd.c b/arch/x86/kernel/cpu/mcheck/mce_amd.c index c8e038800591..dd33c357548f 100644 --- a/arch/x86/kernel/cpu/mcheck/mce_amd.c +++ b/arch/x86/kernel/cpu/mcheck/mce_amd.c @@ -436,8 +436,7 @@ static void deferred_error_interrupt_enable(struct cpuinfo_x86 *c) wrmsr(MSR_CU_DEF_ERR, low, high); } -static u32 smca_get_block_address(unsigned int cpu, unsigned int bank, - unsigned int block) +static u32 smca_get_block_address(unsigned int bank, unsigned int block) { u32 low, high; u32 addr = 0; @@ -456,13 +455,13 @@ static u32 smca_get_block_address(unsigned int cpu, unsigned int bank, * For SMCA enabled processors, BLKPTR field of the first MISC register * (MCx_MISC0) indicates presence of additional MISC regs set (MISC1-4). */ - if (rdmsr_safe_on_cpu(cpu, MSR_AMD64_SMCA_MCx_CONFIG(bank), &low, &high)) + if (rdmsr_safe(MSR_AMD64_SMCA_MCx_CONFIG(bank), &low, &high)) goto out; if (!(low & MCI_CONFIG_MCAX)) goto out; - if (!rdmsr_safe_on_cpu(cpu, MSR_AMD64_SMCA_MCx_MISC(bank), &low, &high) && + if (!rdmsr_safe(MSR_AMD64_SMCA_MCx_MISC(bank), &low, &high) && (low & MASK_BLKPTR_LO)) addr = MSR_AMD64_SMCA_MCx_MISCy(bank, block - 1); @@ -471,7 +470,7 @@ out: return addr; } -static u32 get_block_address(unsigned int cpu, u32 current_addr, u32 low, u32 high, +static u32 get_block_address(u32 current_addr, u32 low, u32 high, unsigned int bank, unsigned int block) { u32 addr = 0, offset = 0; @@ -480,7 +479,7 @@ static u32 get_block_address(unsigned int cpu, u32 current_addr, u32 low, u32 hi return addr; if (mce_flags.smca) - return smca_get_block_address(cpu, bank, block); + return smca_get_block_address(bank, block); /* Fall back to method we used for older processors: */ switch (block) { @@ -558,7 +557,7 @@ void mce_amd_feature_init(struct cpuinfo_x86 *c) smca_configure(bank, cpu); for (block = 0; block < NR_BLOCKS; ++block) { - address = get_block_address(cpu, address, low, high, bank, block); + address = get_block_address(address, low, high, bank, block); if (!address) break; @@ -1175,7 +1174,7 @@ static int allocate_threshold_blocks(unsigned int cpu, unsigned int bank, if (err) goto out_free; recurse: - address = get_block_address(cpu, address, low, high, bank, ++block); + address = get_block_address(address, low, high, bank, ++block); if (!address) return 0; @@ -1385,7 +1384,7 @@ int mce_threshold_create_device(unsigned int cpu) if (bp) return 0; - bp = kzalloc(sizeof(struct threshold_bank *) * mca_cfg.banks, + bp = kcalloc(mca_cfg.banks, sizeof(struct threshold_bank *), GFP_KERNEL); if (!bp) return -ENOMEM; diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c index 77e201301528..08286269fd24 100644 --- a/arch/x86/kernel/cpu/microcode/core.c +++ b/arch/x86/kernel/cpu/microcode/core.c @@ -70,7 +70,7 @@ static DEFINE_MUTEX(microcode_mutex); /* * Serialize late loading so that CPUs get updated one-by-one. */ -static DEFINE_SPINLOCK(update_lock); +static DEFINE_RAW_SPINLOCK(update_lock); struct ucode_cpu_info ucode_cpu_info[NR_CPUS]; @@ -560,9 +560,9 @@ static int __reload_late(void *info) if (__wait_for_cpus(&late_cpus_in, NSEC_PER_SEC)) return -1; - spin_lock(&update_lock); + raw_spin_lock(&update_lock); apply_microcode_local(&err); - spin_unlock(&update_lock); + raw_spin_unlock(&update_lock); /* siblings return UCODE_OK because their engine got updated already */ if (err > UCODE_NFOUND) { diff --git a/arch/x86/kernel/cpu/mtrr/Makefile b/arch/x86/kernel/cpu/mtrr/Makefile index ad9e5ed81181..2ad9107ee980 100644 --- a/arch/x86/kernel/cpu/mtrr/Makefile +++ b/arch/x86/kernel/cpu/mtrr/Makefile @@ -1,3 +1,3 @@ -obj-y := main.o if.o generic.o cleanup.o +obj-y := mtrr.o if.o generic.o cleanup.o obj-$(CONFIG_X86_32) += amd.o cyrix.o centaur.o diff --git a/arch/x86/kernel/cpu/mtrr/if.c b/arch/x86/kernel/cpu/mtrr/if.c index 558444b23923..4021d3859499 100644 --- a/arch/x86/kernel/cpu/mtrr/if.c +++ b/arch/x86/kernel/cpu/mtrr/if.c @@ -43,7 +43,7 @@ mtrr_file_add(unsigned long base, unsigned long size, max = num_var_ranges; if (fcount == NULL) { - fcount = kzalloc(max * sizeof *fcount, GFP_KERNEL); + fcount = kcalloc(max, sizeof(*fcount), GFP_KERNEL); if (!fcount) return -ENOMEM; FILE_FCOUNT(file) = fcount; @@ -106,17 +106,9 @@ mtrr_write(struct file *file, const char __user *buf, size_t len, loff_t * ppos) memset(line, 0, LINE_SIZE); - length = len; - length--; - - if (length > LINE_SIZE - 1) - length = LINE_SIZE - 1; - + length = strncpy_from_user(line, buf, LINE_SIZE - 1); if (length < 0) - return -EINVAL; - - if (copy_from_user(line, buf, length)) - return -EFAULT; + return length; linelen = strlen(line); ptr = line + linelen - 1; @@ -149,17 +141,16 @@ mtrr_write(struct file *file, const char __user *buf, size_t len, loff_t * ppos) return -EINVAL; ptr = skip_spaces(ptr + 5); - for (i = 0; i < MTRR_NUM_TYPES; ++i) { - if (strcmp(ptr, mtrr_strings[i])) - continue; - base >>= PAGE_SHIFT; - size >>= PAGE_SHIFT; - err = mtrr_add_page((unsigned long)base, (unsigned long)size, i, true); - if (err < 0) - return err; - return len; - } - return -EINVAL; + i = match_string(mtrr_strings, MTRR_NUM_TYPES, ptr); + if (i < 0) + return i; + + base >>= PAGE_SHIFT; + size >>= PAGE_SHIFT; + err = mtrr_add_page((unsigned long)base, (unsigned long)size, i, true); + if (err < 0) + return err; + return len; } static long diff --git a/arch/x86/kernel/cpu/mtrr/main.c b/arch/x86/kernel/cpu/mtrr/main.c deleted file mode 100644 index 7468de429087..000000000000 --- a/arch/x86/kernel/cpu/mtrr/main.c +++ /dev/null @@ -1,887 +0,0 @@ -/* Generic MTRR (Memory Type Range Register) driver. - - Copyright (C) 1997-2000 Richard Gooch - Copyright (c) 2002 Patrick Mochel - - This library is free software; you can redistribute it and/or - modify it under the terms of the GNU Library General Public - License as published by the Free Software Foundation; either - version 2 of the License, or (at your option) any later version. - - This library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Library General Public License for more details. - - You should have received a copy of the GNU Library General Public - License along with this library; if not, write to the Free - Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. - - Richard Gooch may be reached by email at rgooch@atnf.csiro.au - The postal address is: - Richard Gooch, c/o ATNF, P. O. Box 76, Epping, N.S.W., 2121, Australia. - - Source: "Pentium Pro Family Developer's Manual, Volume 3: - Operating System Writer's Guide" (Intel document number 242692), - section 11.11.7 - - This was cleaned and made readable by Patrick Mochel - on 6-7 March 2002. - Source: Intel Architecture Software Developers Manual, Volume 3: - System Programming Guide; Section 9.11. (1997 edition - PPro). -*/ - -#define DEBUG - -#include /* FIXME: kvm_para.h needs this */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include -#include - -#include "mtrr.h" - -/* arch_phys_wc_add returns an MTRR register index plus this offset. */ -#define MTRR_TO_PHYS_WC_OFFSET 1000 - -u32 num_var_ranges; -static bool __mtrr_enabled; - -static bool mtrr_enabled(void) -{ - return __mtrr_enabled; -} - -unsigned int mtrr_usage_table[MTRR_MAX_VAR_RANGES]; -static DEFINE_MUTEX(mtrr_mutex); - -u64 size_or_mask, size_and_mask; -static bool mtrr_aps_delayed_init; - -static const struct mtrr_ops *mtrr_ops[X86_VENDOR_NUM] __ro_after_init; - -const struct mtrr_ops *mtrr_if; - -static void set_mtrr(unsigned int reg, unsigned long base, - unsigned long size, mtrr_type type); - -void __init set_mtrr_ops(const struct mtrr_ops *ops) -{ - if (ops->vendor && ops->vendor < X86_VENDOR_NUM) - mtrr_ops[ops->vendor] = ops; -} - -/* Returns non-zero if we have the write-combining memory type */ -static int have_wrcomb(void) -{ - struct pci_dev *dev; - - dev = pci_get_class(PCI_CLASS_BRIDGE_HOST << 8, NULL); - if (dev != NULL) { - /* - * ServerWorks LE chipsets < rev 6 have problems with - * write-combining. Don't allow it and leave room for other - * chipsets to be tagged - */ - if (dev->vendor == PCI_VENDOR_ID_SERVERWORKS && - dev->device == PCI_DEVICE_ID_SERVERWORKS_LE && - dev->revision <= 5) { - pr_info("mtrr: Serverworks LE rev < 6 detected. Write-combining disabled.\n"); - pci_dev_put(dev); - return 0; - } - /* - * Intel 450NX errata # 23. Non ascending cacheline evictions to - * write combining memory may resulting in data corruption - */ - if (dev->vendor == PCI_VENDOR_ID_INTEL && - dev->device == PCI_DEVICE_ID_INTEL_82451NX) { - pr_info("mtrr: Intel 450NX MMC detected. Write-combining disabled.\n"); - pci_dev_put(dev); - return 0; - } - pci_dev_put(dev); - } - return mtrr_if->have_wrcomb ? mtrr_if->have_wrcomb() : 0; -} - -/* This function returns the number of variable MTRRs */ -static void __init set_num_var_ranges(void) -{ - unsigned long config = 0, dummy; - - if (use_intel()) - rdmsr(MSR_MTRRcap, config, dummy); - else if (is_cpu(AMD)) - config = 2; - else if (is_cpu(CYRIX) || is_cpu(CENTAUR)) - config = 8; - - num_var_ranges = config & 0xff; -} - -static void __init init_table(void) -{ - int i, max; - - max = num_var_ranges; - for (i = 0; i < max; i++) - mtrr_usage_table[i] = 1; -} - -struct set_mtrr_data { - unsigned long smp_base; - unsigned long smp_size; - unsigned int smp_reg; - mtrr_type smp_type; -}; - -/** - * mtrr_rendezvous_handler - Work done in the synchronization handler. Executed - * by all the CPUs. - * @info: pointer to mtrr configuration data - * - * Returns nothing. - */ -static int mtrr_rendezvous_handler(void *info) -{ - struct set_mtrr_data *data = info; - - /* - * We use this same function to initialize the mtrrs during boot, - * resume, runtime cpu online and on an explicit request to set a - * specific MTRR. - * - * During boot or suspend, the state of the boot cpu's mtrrs has been - * saved, and we want to replicate that across all the cpus that come - * online (either at the end of boot or resume or during a runtime cpu - * online). If we're doing that, @reg is set to something special and on - * all the cpu's we do mtrr_if->set_all() (On the logical cpu that - * started the boot/resume sequence, this might be a duplicate - * set_all()). - */ - if (data->smp_reg != ~0U) { - mtrr_if->set(data->smp_reg, data->smp_base, - data->smp_size, data->smp_type); - } else if (mtrr_aps_delayed_init || !cpu_online(smp_processor_id())) { - mtrr_if->set_all(); - } - return 0; -} - -static inline int types_compatible(mtrr_type type1, mtrr_type type2) -{ - return type1 == MTRR_TYPE_UNCACHABLE || - type2 == MTRR_TYPE_UNCACHABLE || - (type1 == MTRR_TYPE_WRTHROUGH && type2 == MTRR_TYPE_WRBACK) || - (type1 == MTRR_TYPE_WRBACK && type2 == MTRR_TYPE_WRTHROUGH); -} - -/** - * set_mtrr - update mtrrs on all processors - * @reg: mtrr in question - * @base: mtrr base - * @size: mtrr size - * @type: mtrr type - * - * This is kinda tricky, but fortunately, Intel spelled it out for us cleanly: - * - * 1. Queue work to do the following on all processors: - * 2. Disable Interrupts - * 3. Wait for all procs to do so - * 4. Enter no-fill cache mode - * 5. Flush caches - * 6. Clear PGE bit - * 7. Flush all TLBs - * 8. Disable all range registers - * 9. Update the MTRRs - * 10. Enable all range registers - * 11. Flush all TLBs and caches again - * 12. Enter normal cache mode and reenable caching - * 13. Set PGE - * 14. Wait for buddies to catch up - * 15. Enable interrupts. - * - * What does that mean for us? Well, stop_machine() will ensure that - * the rendezvous handler is started on each CPU. And in lockstep they - * do the state transition of disabling interrupts, updating MTRR's - * (the CPU vendors may each do it differently, so we call mtrr_if->set() - * callback and let them take care of it.) and enabling interrupts. - * - * Note that the mechanism is the same for UP systems, too; all the SMP stuff - * becomes nops. - */ -static void -set_mtrr(unsigned int reg, unsigned long base, unsigned long size, mtrr_type type) -{ - struct set_mtrr_data data = { .smp_reg = reg, - .smp_base = base, - .smp_size = size, - .smp_type = type - }; - - stop_machine(mtrr_rendezvous_handler, &data, cpu_online_mask); -} - -static void set_mtrr_cpuslocked(unsigned int reg, unsigned long base, - unsigned long size, mtrr_type type) -{ - struct set_mtrr_data data = { .smp_reg = reg, - .smp_base = base, - .smp_size = size, - .smp_type = type - }; - - stop_machine_cpuslocked(mtrr_rendezvous_handler, &data, cpu_online_mask); -} - -static void set_mtrr_from_inactive_cpu(unsigned int reg, unsigned long base, - unsigned long size, mtrr_type type) -{ - struct set_mtrr_data data = { .smp_reg = reg, - .smp_base = base, - .smp_size = size, - .smp_type = type - }; - - stop_machine_from_inactive_cpu(mtrr_rendezvous_handler, &data, - cpu_callout_mask); -} - -/** - * mtrr_add_page - Add a memory type region - * @base: Physical base address of region in pages (in units of 4 kB!) - * @size: Physical size of region in pages (4 kB) - * @type: Type of MTRR desired - * @increment: If this is true do usage counting on the region - * - * Memory type region registers control the caching on newer Intel and - * non Intel processors. This function allows drivers to request an - * MTRR is added. The details and hardware specifics of each processor's - * implementation are hidden from the caller, but nevertheless the - * caller should expect to need to provide a power of two size on an - * equivalent power of two boundary. - * - * If the region cannot be added either because all regions are in use - * or the CPU cannot support it a negative value is returned. On success - * the register number for this entry is returned, but should be treated - * as a cookie only. - * - * On a multiprocessor machine the changes are made to all processors. - * This is required on x86 by the Intel processors. - * - * The available types are - * - * %MTRR_TYPE_UNCACHABLE - No caching - * - * %MTRR_TYPE_WRBACK - Write data back in bursts whenever - * - * %MTRR_TYPE_WRCOMB - Write data back soon but allow bursts - * - * %MTRR_TYPE_WRTHROUGH - Cache reads but not writes - * - * BUGS: Needs a quiet flag for the cases where drivers do not mind - * failures and do not wish system log messages to be sent. - */ -int mtrr_add_page(unsigned long base, unsigned long size, - unsigned int type, bool increment) -{ - unsigned long lbase, lsize; - int i, replace, error; - mtrr_type ltype; - - if (!mtrr_enabled()) - return -ENXIO; - - error = mtrr_if->validate_add_page(base, size, type); - if (error) - return error; - - if (type >= MTRR_NUM_TYPES) { - pr_warn("mtrr: type: %u invalid\n", type); - return -EINVAL; - } - - /* If the type is WC, check that this processor supports it */ - if ((type == MTRR_TYPE_WRCOMB) && !have_wrcomb()) { - pr_warn("mtrr: your processor doesn't support write-combining\n"); - return -ENOSYS; - } - - if (!size) { - pr_warn("mtrr: zero sized request\n"); - return -EINVAL; - } - - if ((base | (base + size - 1)) >> - (boot_cpu_data.x86_phys_bits - PAGE_SHIFT)) { - pr_warn("mtrr: base or size exceeds the MTRR width\n"); - return -EINVAL; - } - - error = -EINVAL; - replace = -1; - - /* No CPU hotplug when we change MTRR entries */ - get_online_cpus(); - - /* Search for existing MTRR */ - mutex_lock(&mtrr_mutex); - for (i = 0; i < num_var_ranges; ++i) { - mtrr_if->get(i, &lbase, &lsize, <ype); - if (!lsize || base > lbase + lsize - 1 || - base + size - 1 < lbase) - continue; - /* - * At this point we know there is some kind of - * overlap/enclosure - */ - if (base < lbase || base + size - 1 > lbase + lsize - 1) { - if (base <= lbase && - base + size - 1 >= lbase + lsize - 1) { - /* New region encloses an existing region */ - if (type == ltype) { - replace = replace == -1 ? i : -2; - continue; - } else if (types_compatible(type, ltype)) - continue; - } - pr_warn("mtrr: 0x%lx000,0x%lx000 overlaps existing" - " 0x%lx000,0x%lx000\n", base, size, lbase, - lsize); - goto out; - } - /* New region is enclosed by an existing region */ - if (ltype != type) { - if (types_compatible(type, ltype)) - continue; - pr_warn("mtrr: type mismatch for %lx000,%lx000 old: %s new: %s\n", - base, size, mtrr_attrib_to_str(ltype), - mtrr_attrib_to_str(type)); - goto out; - } - if (increment) - ++mtrr_usage_table[i]; - error = i; - goto out; - } - /* Search for an empty MTRR */ - i = mtrr_if->get_free_region(base, size, replace); - if (i >= 0) { - set_mtrr_cpuslocked(i, base, size, type); - if (likely(replace < 0)) { - mtrr_usage_table[i] = 1; - } else { - mtrr_usage_table[i] = mtrr_usage_table[replace]; - if (increment) - mtrr_usage_table[i]++; - if (unlikely(replace != i)) { - set_mtrr_cpuslocked(replace, 0, 0, 0); - mtrr_usage_table[replace] = 0; - } - } - } else { - pr_info("mtrr: no more MTRRs available\n"); - } - error = i; - out: - mutex_unlock(&mtrr_mutex); - put_online_cpus(); - return error; -} - -static int mtrr_check(unsigned long base, unsigned long size) -{ - if ((base & (PAGE_SIZE - 1)) || (size & (PAGE_SIZE - 1))) { - pr_warn("mtrr: size and base must be multiples of 4 kiB\n"); - pr_debug("mtrr: size: 0x%lx base: 0x%lx\n", size, base); - dump_stack(); - return -1; - } - return 0; -} - -/** - * mtrr_add - Add a memory type region - * @base: Physical base address of region - * @size: Physical size of region - * @type: Type of MTRR desired - * @increment: If this is true do usage counting on the region - * - * Memory type region registers control the caching on newer Intel and - * non Intel processors. This function allows drivers to request an - * MTRR is added. The details and hardware specifics of each processor's - * implementation are hidden from the caller, but nevertheless the - * caller should expect to need to provide a power of two size on an - * equivalent power of two boundary. - * - * If the region cannot be added either because all regions are in use - * or the CPU cannot support it a negative value is returned. On success - * the register number for this entry is returned, but should be treated - * as a cookie only. - * - * On a multiprocessor machine the changes are made to all processors. - * This is required on x86 by the Intel processors. - * - * The available types are - * - * %MTRR_TYPE_UNCACHABLE - No caching - * - * %MTRR_TYPE_WRBACK - Write data back in bursts whenever - * - * %MTRR_TYPE_WRCOMB - Write data back soon but allow bursts - * - * %MTRR_TYPE_WRTHROUGH - Cache reads but not writes - * - * BUGS: Needs a quiet flag for the cases where drivers do not mind - * failures and do not wish system log messages to be sent. - */ -int mtrr_add(unsigned long base, unsigned long size, unsigned int type, - bool increment) -{ - if (!mtrr_enabled()) - return -ENODEV; - if (mtrr_check(base, size)) - return -EINVAL; - return mtrr_add_page(base >> PAGE_SHIFT, size >> PAGE_SHIFT, type, - increment); -} - -/** - * mtrr_del_page - delete a memory type region - * @reg: Register returned by mtrr_add - * @base: Physical base address - * @size: Size of region - * - * If register is supplied then base and size are ignored. This is - * how drivers should call it. - * - * Releases an MTRR region. If the usage count drops to zero the - * register is freed and the region returns to default state. - * On success the register is returned, on failure a negative error - * code. - */ -int mtrr_del_page(int reg, unsigned long base, unsigned long size) -{ - int i, max; - mtrr_type ltype; - unsigned long lbase, lsize; - int error = -EINVAL; - - if (!mtrr_enabled()) - return -ENODEV; - - max = num_var_ranges; - /* No CPU hotplug when we change MTRR entries */ - get_online_cpus(); - mutex_lock(&mtrr_mutex); - if (reg < 0) { - /* Search for existing MTRR */ - for (i = 0; i < max; ++i) { - mtrr_if->get(i, &lbase, &lsize, <ype); - if (lbase == base && lsize == size) { - reg = i; - break; - } - } - if (reg < 0) { - pr_debug("mtrr: no MTRR for %lx000,%lx000 found\n", - base, size); - goto out; - } - } - if (reg >= max) { - pr_warn("mtrr: register: %d too big\n", reg); - goto out; - } - mtrr_if->get(reg, &lbase, &lsize, <ype); - if (lsize < 1) { - pr_warn("mtrr: MTRR %d not used\n", reg); - goto out; - } - if (mtrr_usage_table[reg] < 1) { - pr_warn("mtrr: reg: %d has count=0\n", reg); - goto out; - } - if (--mtrr_usage_table[reg] < 1) - set_mtrr_cpuslocked(reg, 0, 0, 0); - error = reg; - out: - mutex_unlock(&mtrr_mutex); - put_online_cpus(); - return error; -} - -/** - * mtrr_del - delete a memory type region - * @reg: Register returned by mtrr_add - * @base: Physical base address - * @size: Size of region - * - * If register is supplied then base and size are ignored. This is - * how drivers should call it. - * - * Releases an MTRR region. If the usage count drops to zero the - * register is freed and the region returns to default state. - * On success the register is returned, on failure a negative error - * code. - */ -int mtrr_del(int reg, unsigned long base, unsigned long size) -{ - if (!mtrr_enabled()) - return -ENODEV; - if (mtrr_check(base, size)) - return -EINVAL; - return mtrr_del_page(reg, base >> PAGE_SHIFT, size >> PAGE_SHIFT); -} - -/** - * arch_phys_wc_add - add a WC MTRR and handle errors if PAT is unavailable - * @base: Physical base address - * @size: Size of region - * - * If PAT is available, this does nothing. If PAT is unavailable, it - * attempts to add a WC MTRR covering size bytes starting at base and - * logs an error if this fails. - * - * The called should provide a power of two size on an equivalent - * power of two boundary. - * - * Drivers must store the return value to pass to mtrr_del_wc_if_needed, - * but drivers should not try to interpret that return value. - */ -int arch_phys_wc_add(unsigned long base, unsigned long size) -{ - int ret; - - if (pat_enabled() || !mtrr_enabled()) - return 0; /* Success! (We don't need to do anything.) */ - - ret = mtrr_add(base, size, MTRR_TYPE_WRCOMB, true); - if (ret < 0) { - pr_warn("Failed to add WC MTRR for [%p-%p]; performance may suffer.", - (void *)base, (void *)(base + size - 1)); - return ret; - } - return ret + MTRR_TO_PHYS_WC_OFFSET; -} -EXPORT_SYMBOL(arch_phys_wc_add); - -/* - * arch_phys_wc_del - undoes arch_phys_wc_add - * @handle: Return value from arch_phys_wc_add - * - * This cleans up after mtrr_add_wc_if_needed. - * - * The API guarantees that mtrr_del_wc_if_needed(error code) and - * mtrr_del_wc_if_needed(0) do nothing. - */ -void arch_phys_wc_del(int handle) -{ - if (handle >= 1) { - WARN_ON(handle < MTRR_TO_PHYS_WC_OFFSET); - mtrr_del(handle - MTRR_TO_PHYS_WC_OFFSET, 0, 0); - } -} -EXPORT_SYMBOL(arch_phys_wc_del); - -/* - * arch_phys_wc_index - translates arch_phys_wc_add's return value - * @handle: Return value from arch_phys_wc_add - * - * This will turn the return value from arch_phys_wc_add into an mtrr - * index suitable for debugging. - * - * Note: There is no legitimate use for this function, except possibly - * in printk line. Alas there is an illegitimate use in some ancient - * drm ioctls. - */ -int arch_phys_wc_index(int handle) -{ - if (handle < MTRR_TO_PHYS_WC_OFFSET) - return -1; - else - return handle - MTRR_TO_PHYS_WC_OFFSET; -} -EXPORT_SYMBOL_GPL(arch_phys_wc_index); - -/* - * HACK ALERT! - * These should be called implicitly, but we can't yet until all the initcall - * stuff is done... - */ -static void __init init_ifs(void) -{ -#ifndef CONFIG_X86_64 - amd_init_mtrr(); - cyrix_init_mtrr(); - centaur_init_mtrr(); -#endif -} - -/* The suspend/resume methods are only for CPU without MTRR. CPU using generic - * MTRR driver doesn't require this - */ -struct mtrr_value { - mtrr_type ltype; - unsigned long lbase; - unsigned long lsize; -}; - -static struct mtrr_value mtrr_value[MTRR_MAX_VAR_RANGES]; - -static int mtrr_save(void) -{ - int i; - - for (i = 0; i < num_var_ranges; i++) { - mtrr_if->get(i, &mtrr_value[i].lbase, - &mtrr_value[i].lsize, - &mtrr_value[i].ltype); - } - return 0; -} - -static void mtrr_restore(void) -{ - int i; - - for (i = 0; i < num_var_ranges; i++) { - if (mtrr_value[i].lsize) { - set_mtrr(i, mtrr_value[i].lbase, - mtrr_value[i].lsize, - mtrr_value[i].ltype); - } - } -} - - - -static struct syscore_ops mtrr_syscore_ops = { - .suspend = mtrr_save, - .resume = mtrr_restore, -}; - -int __initdata changed_by_mtrr_cleanup; - -#define SIZE_OR_MASK_BITS(n) (~((1ULL << ((n) - PAGE_SHIFT)) - 1)) -/** - * mtrr_bp_init - initialize mtrrs on the boot CPU - * - * This needs to be called early; before any of the other CPUs are - * initialized (i.e. before smp_init()). - * - */ -void __init mtrr_bp_init(void) -{ - u32 phys_addr; - - init_ifs(); - - phys_addr = 32; - - if (boot_cpu_has(X86_FEATURE_MTRR)) { - mtrr_if = &generic_mtrr_ops; - size_or_mask = SIZE_OR_MASK_BITS(36); - size_and_mask = 0x00f00000; - phys_addr = 36; - - /* - * This is an AMD specific MSR, but we assume(hope?) that - * Intel will implement it too when they extend the address - * bus of the Xeon. - */ - if (cpuid_eax(0x80000000) >= 0x80000008) { - phys_addr = cpuid_eax(0x80000008) & 0xff; - /* CPUID workaround for Intel 0F33/0F34 CPU */ - if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && - boot_cpu_data.x86 == 0xF && - boot_cpu_data.x86_model == 0x3 && - (boot_cpu_data.x86_stepping == 0x3 || - boot_cpu_data.x86_stepping == 0x4)) - phys_addr = 36; - - size_or_mask = SIZE_OR_MASK_BITS(phys_addr); - size_and_mask = ~size_or_mask & 0xfffff00000ULL; - } else if (boot_cpu_data.x86_vendor == X86_VENDOR_CENTAUR && - boot_cpu_data.x86 == 6) { - /* - * VIA C* family have Intel style MTRRs, - * but don't support PAE - */ - size_or_mask = SIZE_OR_MASK_BITS(32); - size_and_mask = 0; - phys_addr = 32; - } - } else { - switch (boot_cpu_data.x86_vendor) { - case X86_VENDOR_AMD: - if (cpu_feature_enabled(X86_FEATURE_K6_MTRR)) { - /* Pre-Athlon (K6) AMD CPU MTRRs */ - mtrr_if = mtrr_ops[X86_VENDOR_AMD]; - size_or_mask = SIZE_OR_MASK_BITS(32); - size_and_mask = 0; - } - break; - case X86_VENDOR_CENTAUR: - if (cpu_feature_enabled(X86_FEATURE_CENTAUR_MCR)) { - mtrr_if = mtrr_ops[X86_VENDOR_CENTAUR]; - size_or_mask = SIZE_OR_MASK_BITS(32); - size_and_mask = 0; - } - break; - case X86_VENDOR_CYRIX: - if (cpu_feature_enabled(X86_FEATURE_CYRIX_ARR)) { - mtrr_if = mtrr_ops[X86_VENDOR_CYRIX]; - size_or_mask = SIZE_OR_MASK_BITS(32); - size_and_mask = 0; - } - break; - default: - break; - } - } - - if (mtrr_if) { - __mtrr_enabled = true; - set_num_var_ranges(); - init_table(); - if (use_intel()) { - /* BIOS may override */ - __mtrr_enabled = get_mtrr_state(); - - if (mtrr_enabled()) - mtrr_bp_pat_init(); - - if (mtrr_cleanup(phys_addr)) { - changed_by_mtrr_cleanup = 1; - mtrr_if->set_all(); - } - } - } - - if (!mtrr_enabled()) { - pr_info("MTRR: Disabled\n"); - - /* - * PAT initialization relies on MTRR's rendezvous handler. - * Skip PAT init until the handler can initialize both - * features independently. - */ - pat_disable("MTRRs disabled, skipping PAT initialization too."); - } -} - -void mtrr_ap_init(void) -{ - if (!mtrr_enabled()) - return; - - if (!use_intel() || mtrr_aps_delayed_init) - return; - /* - * Ideally we should hold mtrr_mutex here to avoid mtrr entries - * changed, but this routine will be called in cpu boot time, - * holding the lock breaks it. - * - * This routine is called in two cases: - * - * 1. very earily time of software resume, when there absolutely - * isn't mtrr entry changes; - * - * 2. cpu hotadd time. We let mtrr_add/del_page hold cpuhotplug - * lock to prevent mtrr entry changes - */ - set_mtrr_from_inactive_cpu(~0U, 0, 0, 0); -} - -/** - * Save current fixed-range MTRR state of the first cpu in cpu_online_mask. - */ -void mtrr_save_state(void) -{ - int first_cpu; - - if (!mtrr_enabled()) - return; - - first_cpu = cpumask_first(cpu_online_mask); - smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1); -} - -void set_mtrr_aps_delayed_init(void) -{ - if (!mtrr_enabled()) - return; - if (!use_intel()) - return; - - mtrr_aps_delayed_init = true; -} - -/* - * Delayed MTRR initialization for all AP's - */ -void mtrr_aps_init(void) -{ - if (!use_intel() || !mtrr_enabled()) - return; - - /* - * Check if someone has requested the delay of AP MTRR initialization, - * by doing set_mtrr_aps_delayed_init(), prior to this point. If not, - * then we are done. - */ - if (!mtrr_aps_delayed_init) - return; - - set_mtrr(~0U, 0, 0, 0); - mtrr_aps_delayed_init = false; -} - -void mtrr_bp_restore(void) -{ - if (!use_intel() || !mtrr_enabled()) - return; - - mtrr_if->set_all(); -} - -static int __init mtrr_init_finialize(void) -{ - if (!mtrr_enabled()) - return 0; - - if (use_intel()) { - if (!changed_by_mtrr_cleanup) - mtrr_state_warn(); - return 0; - } - - /* - * The CPU has no MTRR and seems to not support SMP. They have - * specific drivers, we use a tricky method to support - * suspend/resume for them. - * - * TBD: is there any system with such CPU which supports - * suspend/resume? If no, we should remove the code. - */ - register_syscore_ops(&mtrr_syscore_ops); - - return 0; -} -subsys_initcall(mtrr_init_finialize); diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtrr.c new file mode 100644 index 000000000000..9a19c800fe40 --- /dev/null +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -0,0 +1,890 @@ +/* Generic MTRR (Memory Type Range Register) driver. + + Copyright (C) 1997-2000 Richard Gooch + Copyright (c) 2002 Patrick Mochel + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Library General Public + License as published by the Free Software Foundation; either + version 2 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Library General Public License for more details. + + You should have received a copy of the GNU Library General Public + License along with this library; if not, write to the Free + Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + + Richard Gooch may be reached by email at rgooch@atnf.csiro.au + The postal address is: + Richard Gooch, c/o ATNF, P. O. Box 76, Epping, N.S.W., 2121, Australia. + + Source: "Pentium Pro Family Developer's Manual, Volume 3: + Operating System Writer's Guide" (Intel document number 242692), + section 11.11.7 + + This was cleaned and made readable by Patrick Mochel + on 6-7 March 2002. + Source: Intel Architecture Software Developers Manual, Volume 3: + System Programming Guide; Section 9.11. (1997 edition - PPro). +*/ + +#define DEBUG + +#include /* FIXME: kvm_para.h needs this */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include "mtrr.h" + +/* arch_phys_wc_add returns an MTRR register index plus this offset. */ +#define MTRR_TO_PHYS_WC_OFFSET 1000 + +u32 num_var_ranges; +static bool __mtrr_enabled; + +static bool mtrr_enabled(void) +{ + return __mtrr_enabled; +} + +unsigned int mtrr_usage_table[MTRR_MAX_VAR_RANGES]; +static DEFINE_MUTEX(mtrr_mutex); + +u64 size_or_mask, size_and_mask; +static bool mtrr_aps_delayed_init; + +static const struct mtrr_ops *mtrr_ops[X86_VENDOR_NUM] __ro_after_init; + +const struct mtrr_ops *mtrr_if; + +static void set_mtrr(unsigned int reg, unsigned long base, + unsigned long size, mtrr_type type); + +void __init set_mtrr_ops(const struct mtrr_ops *ops) +{ + if (ops->vendor && ops->vendor < X86_VENDOR_NUM) + mtrr_ops[ops->vendor] = ops; +} + +/* Returns non-zero if we have the write-combining memory type */ +static int have_wrcomb(void) +{ + struct pci_dev *dev; + + dev = pci_get_class(PCI_CLASS_BRIDGE_HOST << 8, NULL); + if (dev != NULL) { + /* + * ServerWorks LE chipsets < rev 6 have problems with + * write-combining. Don't allow it and leave room for other + * chipsets to be tagged + */ + if (dev->vendor == PCI_VENDOR_ID_SERVERWORKS && + dev->device == PCI_DEVICE_ID_SERVERWORKS_LE && + dev->revision <= 5) { + pr_info("Serverworks LE rev < 6 detected. Write-combining disabled.\n"); + pci_dev_put(dev); + return 0; + } + /* + * Intel 450NX errata # 23. Non ascending cacheline evictions to + * write combining memory may resulting in data corruption + */ + if (dev->vendor == PCI_VENDOR_ID_INTEL && + dev->device == PCI_DEVICE_ID_INTEL_82451NX) { + pr_info("Intel 450NX MMC detected. Write-combining disabled.\n"); + pci_dev_put(dev); + return 0; + } + pci_dev_put(dev); + } + return mtrr_if->have_wrcomb ? mtrr_if->have_wrcomb() : 0; +} + +/* This function returns the number of variable MTRRs */ +static void __init set_num_var_ranges(void) +{ + unsigned long config = 0, dummy; + + if (use_intel()) + rdmsr(MSR_MTRRcap, config, dummy); + else if (is_cpu(AMD)) + config = 2; + else if (is_cpu(CYRIX) || is_cpu(CENTAUR)) + config = 8; + + num_var_ranges = config & 0xff; +} + +static void __init init_table(void) +{ + int i, max; + + max = num_var_ranges; + for (i = 0; i < max; i++) + mtrr_usage_table[i] = 1; +} + +struct set_mtrr_data { + unsigned long smp_base; + unsigned long smp_size; + unsigned int smp_reg; + mtrr_type smp_type; +}; + +/** + * mtrr_rendezvous_handler - Work done in the synchronization handler. Executed + * by all the CPUs. + * @info: pointer to mtrr configuration data + * + * Returns nothing. + */ +static int mtrr_rendezvous_handler(void *info) +{ + struct set_mtrr_data *data = info; + + /* + * We use this same function to initialize the mtrrs during boot, + * resume, runtime cpu online and on an explicit request to set a + * specific MTRR. + * + * During boot or suspend, the state of the boot cpu's mtrrs has been + * saved, and we want to replicate that across all the cpus that come + * online (either at the end of boot or resume or during a runtime cpu + * online). If we're doing that, @reg is set to something special and on + * all the cpu's we do mtrr_if->set_all() (On the logical cpu that + * started the boot/resume sequence, this might be a duplicate + * set_all()). + */ + if (data->smp_reg != ~0U) { + mtrr_if->set(data->smp_reg, data->smp_base, + data->smp_size, data->smp_type); + } else if (mtrr_aps_delayed_init || !cpu_online(smp_processor_id())) { + mtrr_if->set_all(); + } + return 0; +} + +static inline int types_compatible(mtrr_type type1, mtrr_type type2) +{ + return type1 == MTRR_TYPE_UNCACHABLE || + type2 == MTRR_TYPE_UNCACHABLE || + (type1 == MTRR_TYPE_WRTHROUGH && type2 == MTRR_TYPE_WRBACK) || + (type1 == MTRR_TYPE_WRBACK && type2 == MTRR_TYPE_WRTHROUGH); +} + +/** + * set_mtrr - update mtrrs on all processors + * @reg: mtrr in question + * @base: mtrr base + * @size: mtrr size + * @type: mtrr type + * + * This is kinda tricky, but fortunately, Intel spelled it out for us cleanly: + * + * 1. Queue work to do the following on all processors: + * 2. Disable Interrupts + * 3. Wait for all procs to do so + * 4. Enter no-fill cache mode + * 5. Flush caches + * 6. Clear PGE bit + * 7. Flush all TLBs + * 8. Disable all range registers + * 9. Update the MTRRs + * 10. Enable all range registers + * 11. Flush all TLBs and caches again + * 12. Enter normal cache mode and reenable caching + * 13. Set PGE + * 14. Wait for buddies to catch up + * 15. Enable interrupts. + * + * What does that mean for us? Well, stop_machine() will ensure that + * the rendezvous handler is started on each CPU. And in lockstep they + * do the state transition of disabling interrupts, updating MTRR's + * (the CPU vendors may each do it differently, so we call mtrr_if->set() + * callback and let them take care of it.) and enabling interrupts. + * + * Note that the mechanism is the same for UP systems, too; all the SMP stuff + * becomes nops. + */ +static void +set_mtrr(unsigned int reg, unsigned long base, unsigned long size, mtrr_type type) +{ + struct set_mtrr_data data = { .smp_reg = reg, + .smp_base = base, + .smp_size = size, + .smp_type = type + }; + + stop_machine(mtrr_rendezvous_handler, &data, cpu_online_mask); +} + +static void set_mtrr_cpuslocked(unsigned int reg, unsigned long base, + unsigned long size, mtrr_type type) +{ + struct set_mtrr_data data = { .smp_reg = reg, + .smp_base = base, + .smp_size = size, + .smp_type = type + }; + + stop_machine_cpuslocked(mtrr_rendezvous_handler, &data, cpu_online_mask); +} + +static void set_mtrr_from_inactive_cpu(unsigned int reg, unsigned long base, + unsigned long size, mtrr_type type) +{ + struct set_mtrr_data data = { .smp_reg = reg, + .smp_base = base, + .smp_size = size, + .smp_type = type + }; + + stop_machine_from_inactive_cpu(mtrr_rendezvous_handler, &data, + cpu_callout_mask); +} + +/** + * mtrr_add_page - Add a memory type region + * @base: Physical base address of region in pages (in units of 4 kB!) + * @size: Physical size of region in pages (4 kB) + * @type: Type of MTRR desired + * @increment: If this is true do usage counting on the region + * + * Memory type region registers control the caching on newer Intel and + * non Intel processors. This function allows drivers to request an + * MTRR is added. The details and hardware specifics of each processor's + * implementation are hidden from the caller, but nevertheless the + * caller should expect to need to provide a power of two size on an + * equivalent power of two boundary. + * + * If the region cannot be added either because all regions are in use + * or the CPU cannot support it a negative value is returned. On success + * the register number for this entry is returned, but should be treated + * as a cookie only. + * + * On a multiprocessor machine the changes are made to all processors. + * This is required on x86 by the Intel processors. + * + * The available types are + * + * %MTRR_TYPE_UNCACHABLE - No caching + * + * %MTRR_TYPE_WRBACK - Write data back in bursts whenever + * + * %MTRR_TYPE_WRCOMB - Write data back soon but allow bursts + * + * %MTRR_TYPE_WRTHROUGH - Cache reads but not writes + * + * BUGS: Needs a quiet flag for the cases where drivers do not mind + * failures and do not wish system log messages to be sent. + */ +int mtrr_add_page(unsigned long base, unsigned long size, + unsigned int type, bool increment) +{ + unsigned long lbase, lsize; + int i, replace, error; + mtrr_type ltype; + + if (!mtrr_enabled()) + return -ENXIO; + + error = mtrr_if->validate_add_page(base, size, type); + if (error) + return error; + + if (type >= MTRR_NUM_TYPES) { + pr_warn("type: %u invalid\n", type); + return -EINVAL; + } + + /* If the type is WC, check that this processor supports it */ + if ((type == MTRR_TYPE_WRCOMB) && !have_wrcomb()) { + pr_warn("your processor doesn't support write-combining\n"); + return -ENOSYS; + } + + if (!size) { + pr_warn("zero sized request\n"); + return -EINVAL; + } + + if ((base | (base + size - 1)) >> + (boot_cpu_data.x86_phys_bits - PAGE_SHIFT)) { + pr_warn("base or size exceeds the MTRR width\n"); + return -EINVAL; + } + + error = -EINVAL; + replace = -1; + + /* No CPU hotplug when we change MTRR entries */ + get_online_cpus(); + + /* Search for existing MTRR */ + mutex_lock(&mtrr_mutex); + for (i = 0; i < num_var_ranges; ++i) { + mtrr_if->get(i, &lbase, &lsize, <ype); + if (!lsize || base > lbase + lsize - 1 || + base + size - 1 < lbase) + continue; + /* + * At this point we know there is some kind of + * overlap/enclosure + */ + if (base < lbase || base + size - 1 > lbase + lsize - 1) { + if (base <= lbase && + base + size - 1 >= lbase + lsize - 1) { + /* New region encloses an existing region */ + if (type == ltype) { + replace = replace == -1 ? i : -2; + continue; + } else if (types_compatible(type, ltype)) + continue; + } + pr_warn("0x%lx000,0x%lx000 overlaps existing 0x%lx000,0x%lx000\n", base, size, lbase, + lsize); + goto out; + } + /* New region is enclosed by an existing region */ + if (ltype != type) { + if (types_compatible(type, ltype)) + continue; + pr_warn("type mismatch for %lx000,%lx000 old: %s new: %s\n", + base, size, mtrr_attrib_to_str(ltype), + mtrr_attrib_to_str(type)); + goto out; + } + if (increment) + ++mtrr_usage_table[i]; + error = i; + goto out; + } + /* Search for an empty MTRR */ + i = mtrr_if->get_free_region(base, size, replace); + if (i >= 0) { + set_mtrr_cpuslocked(i, base, size, type); + if (likely(replace < 0)) { + mtrr_usage_table[i] = 1; + } else { + mtrr_usage_table[i] = mtrr_usage_table[replace]; + if (increment) + mtrr_usage_table[i]++; + if (unlikely(replace != i)) { + set_mtrr_cpuslocked(replace, 0, 0, 0); + mtrr_usage_table[replace] = 0; + } + } + } else { + pr_info("no more MTRRs available\n"); + } + error = i; + out: + mutex_unlock(&mtrr_mutex); + put_online_cpus(); + return error; +} + +static int mtrr_check(unsigned long base, unsigned long size) +{ + if ((base & (PAGE_SIZE - 1)) || (size & (PAGE_SIZE - 1))) { + pr_warn("size and base must be multiples of 4 kiB\n"); + pr_debug("size: 0x%lx base: 0x%lx\n", size, base); + dump_stack(); + return -1; + } + return 0; +} + +/** + * mtrr_add - Add a memory type region + * @base: Physical base address of region + * @size: Physical size of region + * @type: Type of MTRR desired + * @increment: If this is true do usage counting on the region + * + * Memory type region registers control the caching on newer Intel and + * non Intel processors. This function allows drivers to request an + * MTRR is added. The details and hardware specifics of each processor's + * implementation are hidden from the caller, but nevertheless the + * caller should expect to need to provide a power of two size on an + * equivalent power of two boundary. + * + * If the region cannot be added either because all regions are in use + * or the CPU cannot support it a negative value is returned. On success + * the register number for this entry is returned, but should be treated + * as a cookie only. + * + * On a multiprocessor machine the changes are made to all processors. + * This is required on x86 by the Intel processors. + * + * The available types are + * + * %MTRR_TYPE_UNCACHABLE - No caching + * + * %MTRR_TYPE_WRBACK - Write data back in bursts whenever + * + * %MTRR_TYPE_WRCOMB - Write data back soon but allow bursts + * + * %MTRR_TYPE_WRTHROUGH - Cache reads but not writes + * + * BUGS: Needs a quiet flag for the cases where drivers do not mind + * failures and do not wish system log messages to be sent. + */ +int mtrr_add(unsigned long base, unsigned long size, unsigned int type, + bool increment) +{ + if (!mtrr_enabled()) + return -ENODEV; + if (mtrr_check(base, size)) + return -EINVAL; + return mtrr_add_page(base >> PAGE_SHIFT, size >> PAGE_SHIFT, type, + increment); +} + +/** + * mtrr_del_page - delete a memory type region + * @reg: Register returned by mtrr_add + * @base: Physical base address + * @size: Size of region + * + * If register is supplied then base and size are ignored. This is + * how drivers should call it. + * + * Releases an MTRR region. If the usage count drops to zero the + * register is freed and the region returns to default state. + * On success the register is returned, on failure a negative error + * code. + */ +int mtrr_del_page(int reg, unsigned long base, unsigned long size) +{ + int i, max; + mtrr_type ltype; + unsigned long lbase, lsize; + int error = -EINVAL; + + if (!mtrr_enabled()) + return -ENODEV; + + max = num_var_ranges; + /* No CPU hotplug when we change MTRR entries */ + get_online_cpus(); + mutex_lock(&mtrr_mutex); + if (reg < 0) { + /* Search for existing MTRR */ + for (i = 0; i < max; ++i) { + mtrr_if->get(i, &lbase, &lsize, <ype); + if (lbase == base && lsize == size) { + reg = i; + break; + } + } + if (reg < 0) { + pr_debug("no MTRR for %lx000,%lx000 found\n", + base, size); + goto out; + } + } + if (reg >= max) { + pr_warn("register: %d too big\n", reg); + goto out; + } + mtrr_if->get(reg, &lbase, &lsize, <ype); + if (lsize < 1) { + pr_warn("MTRR %d not used\n", reg); + goto out; + } + if (mtrr_usage_table[reg] < 1) { + pr_warn("reg: %d has count=0\n", reg); + goto out; + } + if (--mtrr_usage_table[reg] < 1) + set_mtrr_cpuslocked(reg, 0, 0, 0); + error = reg; + out: + mutex_unlock(&mtrr_mutex); + put_online_cpus(); + return error; +} + +/** + * mtrr_del - delete a memory type region + * @reg: Register returned by mtrr_add + * @base: Physical base address + * @size: Size of region + * + * If register is supplied then base and size are ignored. This is + * how drivers should call it. + * + * Releases an MTRR region. If the usage count drops to zero the + * register is freed and the region returns to default state. + * On success the register is returned, on failure a negative error + * code. + */ +int mtrr_del(int reg, unsigned long base, unsigned long size) +{ + if (!mtrr_enabled()) + return -ENODEV; + if (mtrr_check(base, size)) + return -EINVAL; + return mtrr_del_page(reg, base >> PAGE_SHIFT, size >> PAGE_SHIFT); +} + +/** + * arch_phys_wc_add - add a WC MTRR and handle errors if PAT is unavailable + * @base: Physical base address + * @size: Size of region + * + * If PAT is available, this does nothing. If PAT is unavailable, it + * attempts to add a WC MTRR covering size bytes starting at base and + * logs an error if this fails. + * + * The called should provide a power of two size on an equivalent + * power of two boundary. + * + * Drivers must store the return value to pass to mtrr_del_wc_if_needed, + * but drivers should not try to interpret that return value. + */ +int arch_phys_wc_add(unsigned long base, unsigned long size) +{ + int ret; + + if (pat_enabled() || !mtrr_enabled()) + return 0; /* Success! (We don't need to do anything.) */ + + ret = mtrr_add(base, size, MTRR_TYPE_WRCOMB, true); + if (ret < 0) { + pr_warn("Failed to add WC MTRR for [%p-%p]; performance may suffer.", + (void *)base, (void *)(base + size - 1)); + return ret; + } + return ret + MTRR_TO_PHYS_WC_OFFSET; +} +EXPORT_SYMBOL(arch_phys_wc_add); + +/* + * arch_phys_wc_del - undoes arch_phys_wc_add + * @handle: Return value from arch_phys_wc_add + * + * This cleans up after mtrr_add_wc_if_needed. + * + * The API guarantees that mtrr_del_wc_if_needed(error code) and + * mtrr_del_wc_if_needed(0) do nothing. + */ +void arch_phys_wc_del(int handle) +{ + if (handle >= 1) { + WARN_ON(handle < MTRR_TO_PHYS_WC_OFFSET); + mtrr_del(handle - MTRR_TO_PHYS_WC_OFFSET, 0, 0); + } +} +EXPORT_SYMBOL(arch_phys_wc_del); + +/* + * arch_phys_wc_index - translates arch_phys_wc_add's return value + * @handle: Return value from arch_phys_wc_add + * + * This will turn the return value from arch_phys_wc_add into an mtrr + * index suitable for debugging. + * + * Note: There is no legitimate use for this function, except possibly + * in printk line. Alas there is an illegitimate use in some ancient + * drm ioctls. + */ +int arch_phys_wc_index(int handle) +{ + if (handle < MTRR_TO_PHYS_WC_OFFSET) + return -1; + else + return handle - MTRR_TO_PHYS_WC_OFFSET; +} +EXPORT_SYMBOL_GPL(arch_phys_wc_index); + +/* + * HACK ALERT! + * These should be called implicitly, but we can't yet until all the initcall + * stuff is done... + */ +static void __init init_ifs(void) +{ +#ifndef CONFIG_X86_64 + amd_init_mtrr(); + cyrix_init_mtrr(); + centaur_init_mtrr(); +#endif +} + +/* The suspend/resume methods are only for CPU without MTRR. CPU using generic + * MTRR driver doesn't require this + */ +struct mtrr_value { + mtrr_type ltype; + unsigned long lbase; + unsigned long lsize; +}; + +static struct mtrr_value mtrr_value[MTRR_MAX_VAR_RANGES]; + +static int mtrr_save(void) +{ + int i; + + for (i = 0; i < num_var_ranges; i++) { + mtrr_if->get(i, &mtrr_value[i].lbase, + &mtrr_value[i].lsize, + &mtrr_value[i].ltype); + } + return 0; +} + +static void mtrr_restore(void) +{ + int i; + + for (i = 0; i < num_var_ranges; i++) { + if (mtrr_value[i].lsize) { + set_mtrr(i, mtrr_value[i].lbase, + mtrr_value[i].lsize, + mtrr_value[i].ltype); + } + } +} + + + +static struct syscore_ops mtrr_syscore_ops = { + .suspend = mtrr_save, + .resume = mtrr_restore, +}; + +int __initdata changed_by_mtrr_cleanup; + +#define SIZE_OR_MASK_BITS(n) (~((1ULL << ((n) - PAGE_SHIFT)) - 1)) +/** + * mtrr_bp_init - initialize mtrrs on the boot CPU + * + * This needs to be called early; before any of the other CPUs are + * initialized (i.e. before smp_init()). + * + */ +void __init mtrr_bp_init(void) +{ + u32 phys_addr; + + init_ifs(); + + phys_addr = 32; + + if (boot_cpu_has(X86_FEATURE_MTRR)) { + mtrr_if = &generic_mtrr_ops; + size_or_mask = SIZE_OR_MASK_BITS(36); + size_and_mask = 0x00f00000; + phys_addr = 36; + + /* + * This is an AMD specific MSR, but we assume(hope?) that + * Intel will implement it too when they extend the address + * bus of the Xeon. + */ + if (cpuid_eax(0x80000000) >= 0x80000008) { + phys_addr = cpuid_eax(0x80000008) & 0xff; + /* CPUID workaround for Intel 0F33/0F34 CPU */ + if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL && + boot_cpu_data.x86 == 0xF && + boot_cpu_data.x86_model == 0x3 && + (boot_cpu_data.x86_stepping == 0x3 || + boot_cpu_data.x86_stepping == 0x4)) + phys_addr = 36; + + size_or_mask = SIZE_OR_MASK_BITS(phys_addr); + size_and_mask = ~size_or_mask & 0xfffff00000ULL; + } else if (boot_cpu_data.x86_vendor == X86_VENDOR_CENTAUR && + boot_cpu_data.x86 == 6) { + /* + * VIA C* family have Intel style MTRRs, + * but don't support PAE + */ + size_or_mask = SIZE_OR_MASK_BITS(32); + size_and_mask = 0; + phys_addr = 32; + } + } else { + switch (boot_cpu_data.x86_vendor) { + case X86_VENDOR_AMD: + if (cpu_feature_enabled(X86_FEATURE_K6_MTRR)) { + /* Pre-Athlon (K6) AMD CPU MTRRs */ + mtrr_if = mtrr_ops[X86_VENDOR_AMD]; + size_or_mask = SIZE_OR_MASK_BITS(32); + size_and_mask = 0; + } + break; + case X86_VENDOR_CENTAUR: + if (cpu_feature_enabled(X86_FEATURE_CENTAUR_MCR)) { + mtrr_if = mtrr_ops[X86_VENDOR_CENTAUR]; + size_or_mask = SIZE_OR_MASK_BITS(32); + size_and_mask = 0; + } + break; + case X86_VENDOR_CYRIX: + if (cpu_feature_enabled(X86_FEATURE_CYRIX_ARR)) { + mtrr_if = mtrr_ops[X86_VENDOR_CYRIX]; + size_or_mask = SIZE_OR_MASK_BITS(32); + size_and_mask = 0; + } + break; + default: + break; + } + } + + if (mtrr_if) { + __mtrr_enabled = true; + set_num_var_ranges(); + init_table(); + if (use_intel()) { + /* BIOS may override */ + __mtrr_enabled = get_mtrr_state(); + + if (mtrr_enabled()) + mtrr_bp_pat_init(); + + if (mtrr_cleanup(phys_addr)) { + changed_by_mtrr_cleanup = 1; + mtrr_if->set_all(); + } + } + } + + if (!mtrr_enabled()) { + pr_info("Disabled\n"); + + /* + * PAT initialization relies on MTRR's rendezvous handler. + * Skip PAT init until the handler can initialize both + * features independently. + */ + pat_disable("MTRRs disabled, skipping PAT initialization too."); + } +} + +void mtrr_ap_init(void) +{ + if (!mtrr_enabled()) + return; + + if (!use_intel() || mtrr_aps_delayed_init) + return; + + rcu_cpu_starting(smp_processor_id()); + + /* + * Ideally we should hold mtrr_mutex here to avoid mtrr entries + * changed, but this routine will be called in cpu boot time, + * holding the lock breaks it. + * + * This routine is called in two cases: + * + * 1. very earily time of software resume, when there absolutely + * isn't mtrr entry changes; + * + * 2. cpu hotadd time. We let mtrr_add/del_page hold cpuhotplug + * lock to prevent mtrr entry changes + */ + set_mtrr_from_inactive_cpu(~0U, 0, 0, 0); +} + +/** + * Save current fixed-range MTRR state of the first cpu in cpu_online_mask. + */ +void mtrr_save_state(void) +{ + int first_cpu; + + if (!mtrr_enabled()) + return; + + first_cpu = cpumask_first(cpu_online_mask); + smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1); +} + +void set_mtrr_aps_delayed_init(void) +{ + if (!mtrr_enabled()) + return; + if (!use_intel()) + return; + + mtrr_aps_delayed_init = true; +} + +/* + * Delayed MTRR initialization for all AP's + */ +void mtrr_aps_init(void) +{ + if (!use_intel() || !mtrr_enabled()) + return; + + /* + * Check if someone has requested the delay of AP MTRR initialization, + * by doing set_mtrr_aps_delayed_init(), prior to this point. If not, + * then we are done. + */ + if (!mtrr_aps_delayed_init) + return; + + set_mtrr(~0U, 0, 0, 0); + mtrr_aps_delayed_init = false; +} + +void mtrr_bp_restore(void) +{ + if (!use_intel() || !mtrr_enabled()) + return; + + mtrr_if->set_all(); +} + +static int __init mtrr_init_finialize(void) +{ + if (!mtrr_enabled()) + return 0; + + if (use_intel()) { + if (!changed_by_mtrr_cleanup) + mtrr_state_warn(); + return 0; + } + + /* + * The CPU has no MTRR and seems to not support SMP. They have + * specific drivers, we use a tricky method to support + * suspend/resume for them. + * + * TBD: is there any system with such CPU which supports + * suspend/resume? If no, we should remove the code. + */ + register_syscore_ops(&mtrr_syscore_ops); + + return 0; +} +subsys_initcall(mtrr_init_finialize); diff --git a/arch/x86/kernel/cpu/topology.c b/arch/x86/kernel/cpu/topology.c index b099024d339c..81c0afb39d0a 100644 --- a/arch/x86/kernel/cpu/topology.c +++ b/arch/x86/kernel/cpu/topology.c @@ -27,7 +27,7 @@ * exists, use it for populating initial_apicid and cpu topology * detection. */ -void detect_extended_topology(struct cpuinfo_x86 *c) +int detect_extended_topology(struct cpuinfo_x86 *c) { #ifdef CONFIG_SMP unsigned int eax, ebx, ecx, edx, sub_index; @@ -36,7 +36,7 @@ void detect_extended_topology(struct cpuinfo_x86 *c) static bool printed; if (c->cpuid_level < 0xb) - return; + return -1; cpuid_count(0xb, SMT_LEVEL, &eax, &ebx, &ecx, &edx); @@ -44,7 +44,7 @@ void detect_extended_topology(struct cpuinfo_x86 *c) * check if the cpuid leaf 0xb is actually implemented. */ if (ebx == 0 || (LEAFB_SUBTYPE(ecx) != SMT_TYPE)) - return; + return -1; set_cpu_cap(c, X86_FEATURE_XTOPOLOGY); @@ -95,6 +95,6 @@ void detect_extended_topology(struct cpuinfo_x86 *c) c->cpu_core_id); printed = 1; } - return; #endif + return 0; } diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c index 18fa9d74c182..666a284116ac 100644 --- a/arch/x86/kernel/dumpstack.c +++ b/arch/x86/kernel/dumpstack.c @@ -22,11 +22,14 @@ #include #include +#define OPCODE_BUFSIZE 64 + int panic_on_unrecovered_nmi; int panic_on_io_nmi; -static unsigned int code_bytes = 64; static int die_counter; +static struct pt_regs exec_summary_regs; + bool in_task_stack(unsigned long *stack, struct task_struct *task, struct stack_info *info) { @@ -69,9 +72,62 @@ static void printk_stack_address(unsigned long address, int reliable, printk("%s %s%pB\n", log_lvl, reliable ? "" : "? ", (void *)address); } +/* + * There are a couple of reasons for the 2/3rd prologue, courtesy of Linus: + * + * In case where we don't have the exact kernel image (which, if we did, we can + * simply disassemble and navigate to the RIP), the purpose of the bigger + * prologue is to have more context and to be able to correlate the code from + * the different toolchains better. + * + * In addition, it helps in recreating the register allocation of the failing + * kernel and thus make sense of the register dump. + * + * What is more, the additional complication of a variable length insn arch like + * x86 warrants having longer byte sequence before rIP so that the disassembler + * can "sync" up properly and find instruction boundaries when decoding the + * opcode bytes. + * + * Thus, the 2/3rds prologue and 64 byte OPCODE_BUFSIZE is just a random + * guesstimate in attempt to achieve all of the above. + */ +void show_opcodes(u8 *rip, const char *loglvl) +{ + unsigned int code_prologue = OPCODE_BUFSIZE * 2 / 3; + u8 opcodes[OPCODE_BUFSIZE]; + u8 *ip; + int i; + + printk("%sCode: ", loglvl); + + ip = (u8 *)rip - code_prologue; + if (probe_kernel_read(opcodes, ip, OPCODE_BUFSIZE)) { + pr_cont("Bad RIP value.\n"); + return; + } + + for (i = 0; i < OPCODE_BUFSIZE; i++, ip++) { + if (ip == rip) + pr_cont("<%02x> ", opcodes[i]); + else + pr_cont("%02x ", opcodes[i]); + } + pr_cont("\n"); +} + +void show_ip(struct pt_regs *regs, const char *loglvl) +{ +#ifdef CONFIG_X86_32 + printk("%sEIP: %pS\n", loglvl, (void *)regs->ip); +#else + printk("%sRIP: %04x:%pS\n", loglvl, (int)regs->cs, (void *)regs->ip); +#endif + show_opcodes((u8 *)regs->ip, loglvl); +} + void show_iret_regs(struct pt_regs *regs) { - printk(KERN_DEFAULT "RIP: %04x:%pS\n", (int)regs->cs, (void *)regs->ip); + show_ip(regs, KERN_DEFAULT); printk(KERN_DEFAULT "RSP: %04x:%016lx EFLAGS: %08lx", (int)regs->ss, regs->sp, regs->flags); } @@ -267,7 +323,6 @@ unsigned long oops_begin(void) bust_spinlocks(1); return flags; } -EXPORT_SYMBOL_GPL(oops_begin); NOKPROBE_SYMBOL(oops_begin); void __noreturn rewind_stack_do_exit(int signr); @@ -287,6 +342,9 @@ void oops_end(unsigned long flags, struct pt_regs *regs, int signr) raw_local_irq_restore(flags); oops_exit(); + /* Executive summary in case the oops scrolled away */ + __show_regs(&exec_summary_regs, true); + if (!signr) return; if (in_interrupt()) @@ -305,10 +363,10 @@ NOKPROBE_SYMBOL(oops_end); int __die(const char *str, struct pt_regs *regs, long err) { -#ifdef CONFIG_X86_32 - unsigned short ss; - unsigned long sp; -#endif + /* Save the regs of the first oops for the executive summary later. */ + if (!die_counter) + exec_summary_regs = *regs; + printk(KERN_DEFAULT "%s: %04lx [#%d]%s%s%s%s%s\n", str, err & 0xffff, ++die_counter, IS_ENABLED(CONFIG_PREEMPT) ? " PREEMPT" : "", @@ -318,26 +376,13 @@ int __die(const char *str, struct pt_regs *regs, long err) IS_ENABLED(CONFIG_PAGE_TABLE_ISOLATION) ? (boot_cpu_has(X86_FEATURE_PTI) ? " PTI" : " NOPTI") : ""); + show_regs(regs); + print_modules(); + if (notify_die(DIE_OOPS, str, regs, err, current->thread.trap_nr, SIGSEGV) == NOTIFY_STOP) return 1; - print_modules(); - show_regs(regs); -#ifdef CONFIG_X86_32 - if (user_mode(regs)) { - sp = regs->sp; - ss = regs->ss; - } else { - sp = kernel_stack_pointer(regs); - savesegment(ss, ss); - } - printk(KERN_EMERG "EIP: %pS SS:ESP: %04x:%08lx\n", - (void *)regs->ip, ss, sp); -#else - /* Executive summary in case the oops scrolled away */ - printk(KERN_ALERT "RIP: %pS RSP: %016lx\n", (void *)regs->ip, regs->sp); -#endif return 0; } NOKPROBE_SYMBOL(__die); @@ -356,30 +401,9 @@ void die(const char *str, struct pt_regs *regs, long err) oops_end(flags, regs, sig); } -static int __init code_bytes_setup(char *s) -{ - ssize_t ret; - unsigned long val; - - if (!s) - return -EINVAL; - - ret = kstrtoul(s, 0, &val); - if (ret) - return ret; - - code_bytes = val; - if (code_bytes > 8192) - code_bytes = 8192; - - return 1; -} -__setup("code_bytes=", code_bytes_setup); - void show_regs(struct pt_regs *regs) { bool all = true; - int i; show_regs_print_info(KERN_DEFAULT); @@ -389,36 +413,8 @@ void show_regs(struct pt_regs *regs) __show_regs(regs, all); /* - * When in-kernel, we also print out the stack and code at the - * time of the fault.. + * When in-kernel, we also print out the stack at the time of the fault.. */ - if (!user_mode(regs)) { - unsigned int code_prologue = code_bytes * 43 / 64; - unsigned int code_len = code_bytes; - unsigned char c; - u8 *ip; - + if (!user_mode(regs)) show_trace_log_lvl(current, regs, NULL, KERN_DEFAULT); - - printk(KERN_DEFAULT "Code: "); - - ip = (u8 *)regs->ip - code_prologue; - if (ip < (u8 *)PAGE_OFFSET || probe_kernel_address(ip, c)) { - /* try starting at IP */ - ip = (u8 *)regs->ip; - code_len = code_len - code_prologue + 1; - } - for (i = 0; i < code_len; i++, ip++) { - if (ip < (u8 *)PAGE_OFFSET || - probe_kernel_address(ip, c)) { - pr_cont(" Bad RIP value."); - break; - } - if (ip == (u8 *)regs->ip) - pr_cont("<%02x> ", c); - else - pr_cont("%02x ", c); - } - } - pr_cont("\n"); } diff --git a/arch/x86/kernel/e820.c b/arch/x86/kernel/e820.c index 6a2cb1442e05..d1f25c831447 100644 --- a/arch/x86/kernel/e820.c +++ b/arch/x86/kernel/e820.c @@ -155,7 +155,8 @@ static void __init __e820__range_add(struct e820_table *table, u64 start, u64 si int x = table->nr_entries; if (x >= ARRAY_SIZE(table->entries)) { - pr_err("e820: too many entries; ignoring [mem %#010llx-%#010llx]\n", start, start + size - 1); + pr_err("too many entries; ignoring [mem %#010llx-%#010llx]\n", + start, start + size - 1); return; } @@ -190,9 +191,10 @@ void __init e820__print_table(char *who) int i; for (i = 0; i < e820_table->nr_entries; i++) { - pr_info("%s: [mem %#018Lx-%#018Lx] ", who, - e820_table->entries[i].addr, - e820_table->entries[i].addr + e820_table->entries[i].size - 1); + pr_info("%s: [mem %#018Lx-%#018Lx] ", + who, + e820_table->entries[i].addr, + e820_table->entries[i].addr + e820_table->entries[i].size - 1); e820_print_type(e820_table->entries[i].type); pr_cont("\n"); @@ -574,7 +576,7 @@ void __init e820__update_table_print(void) if (e820__update_table(e820_table)) return; - pr_info("e820: modified physical RAM map:\n"); + pr_info("modified physical RAM map:\n"); e820__print_table("modified"); } @@ -636,9 +638,8 @@ __init void e820__setup_pci_gap(void) if (!found) { #ifdef CONFIG_X86_64 gapstart = (max_pfn << PAGE_SHIFT) + 1024*1024; - pr_err( - "e820: Cannot find an available gap in the 32-bit address range\n" - "e820: PCI devices with unassigned 32-bit BARs may not work!\n"); + pr_err("Cannot find an available gap in the 32-bit address range\n"); + pr_err("PCI devices with unassigned 32-bit BARs may not work!\n"); #else gapstart = 0x10000000; #endif @@ -649,7 +650,8 @@ __init void e820__setup_pci_gap(void) */ pci_mem_start = gapstart; - pr_info("e820: [mem %#010lx-%#010lx] available for PCI devices\n", gapstart, gapstart + gapsize - 1); + pr_info("[mem %#010lx-%#010lx] available for PCI devices\n", + gapstart, gapstart + gapsize - 1); } /* @@ -711,7 +713,7 @@ void __init e820__memory_setup_extended(u64 phys_addr, u32 data_len) memcpy(e820_table_firmware, e820_table, sizeof(*e820_table_firmware)); early_memunmap(sdata, data_len); - pr_info("e820: extended physical RAM map:\n"); + pr_info("extended physical RAM map:\n"); e820__print_table("extended"); } @@ -780,7 +782,7 @@ u64 __init e820__memblock_alloc_reserved(u64 size, u64 align) addr = __memblock_alloc_base(size, align, MEMBLOCK_ALLOC_ACCESSIBLE); if (addr) { e820__range_update_kexec(addr, size, E820_TYPE_RAM, E820_TYPE_RESERVED); - pr_info("e820: update e820_table_kexec for e820__memblock_alloc_reserved()\n"); + pr_info("update e820_table_kexec for e820__memblock_alloc_reserved()\n"); e820__update_table_kexec(); } @@ -830,8 +832,8 @@ static unsigned long __init e820_end_pfn(unsigned long limit_pfn, enum e820_type if (last_pfn > max_arch_pfn) last_pfn = max_arch_pfn; - pr_info("e820: last_pfn = %#lx max_arch_pfn = %#lx\n", - last_pfn, max_arch_pfn); + pr_info("last_pfn = %#lx max_arch_pfn = %#lx\n", + last_pfn, max_arch_pfn); return last_pfn; } @@ -1005,7 +1007,7 @@ void __init e820__finish_early_params(void) if (e820__update_table(e820_table) < 0) early_panic("Invalid user supplied memory map"); - pr_info("e820: user-defined physical RAM map:\n"); + pr_info("user-defined physical RAM map:\n"); e820__print_table("user"); } } @@ -1238,7 +1240,7 @@ void __init e820__memory_setup(void) memcpy(e820_table_kexec, e820_table, sizeof(*e820_table_kexec)); memcpy(e820_table_firmware, e820_table, sizeof(*e820_table_firmware)); - pr_info("e820: BIOS-provided physical RAM map:\n"); + pr_info("BIOS-provided physical RAM map:\n"); e820__print_table(who); } diff --git a/arch/x86/kernel/early-quirks.c b/arch/x86/kernel/early-quirks.c index bae0d32e327b..da5d8ac60062 100644 --- a/arch/x86/kernel/early-quirks.c +++ b/arch/x86/kernel/early-quirks.c @@ -28,8 +28,6 @@ #include #include -#define dev_err(msg) pr_err("pci 0000:%02x:%02x.%d: %s", bus, slot, func, msg) - static void __init fix_hypertransport_config(int num, int slot, int func) { u32 htcfg; @@ -617,7 +615,8 @@ static void __init apple_airport_reset(int bus, int slot, int func) pmcsr = read_pci_config_16(bus, slot, func, BCM4331_PM_CAP + PCI_PM_CTRL); if ((pmcsr & PCI_PM_CTRL_STATE_MASK) != PCI_D0) { - dev_err("Cannot power up Apple AirPort card\n"); + pr_err("pci 0000:%02x:%02x.%d: Cannot power up Apple AirPort card\n", + bus, slot, func); return; } } @@ -628,7 +627,8 @@ static void __init apple_airport_reset(int bus, int slot, int func) mmio = early_ioremap(addr, BCM4331_MMIO_SIZE); if (!mmio) { - dev_err("Cannot iomap Apple AirPort card\n"); + pr_err("pci 0000:%02x:%02x.%d: Cannot iomap Apple AirPort card\n", + bus, slot, func); return; } diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 2d29e47c056e..a21d6ace648e 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -6,6 +6,10 @@ */ #define DISABLE_BRANCH_PROFILING + +/* cpu_feature_enabled() cannot be used this early */ +#define USE_EARLY_PGTABLE_L5 + #include #include #include @@ -32,11 +36,6 @@ #include #include -#ifdef CONFIG_X86_5LEVEL -#undef pgtable_l5_enabled -#define pgtable_l5_enabled __pgtable_l5_enabled -#endif - /* * Manage page tables very early on. */ @@ -45,8 +44,7 @@ static unsigned int __initdata next_early_pgt; pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX); #ifdef CONFIG_X86_5LEVEL -unsigned int __pgtable_l5_enabled __ro_after_init; -EXPORT_SYMBOL(__pgtable_l5_enabled); +unsigned int __pgtable_l5_enabled __initdata; unsigned int pgdir_shift __ro_after_init = 39; EXPORT_SYMBOL(pgdir_shift); unsigned int ptrs_per_p4d __ro_after_init = 1; @@ -82,13 +80,14 @@ static unsigned int __head *fixup_int(void *ptr, unsigned long physaddr) static bool __head check_la57_support(unsigned long physaddr) { - if (native_cpuid_eax(0) < 7) - return false; - - if (!(native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) + /* + * 5-level paging is detected and enabled at kernel decomression + * stage. Only check if it has been enabled there. + */ + if (!(native_read_cr4() & X86_CR4_LA57)) return false; - *fixup_int(&pgtable_l5_enabled, physaddr) = 1; + *fixup_int(&__pgtable_l5_enabled, physaddr) = 1; *fixup_int(&pgdir_shift, physaddr) = 48; *fixup_int(&ptrs_per_p4d, physaddr) = 512; *fixup_long(&page_offset_base, physaddr) = __PAGE_OFFSET_BASE_L5; @@ -281,7 +280,7 @@ again: * critical -- __PAGE_OFFSET would point us back into the dynamic * range and we might end up looping forever... */ - if (!pgtable_l5_enabled) + if (!pgtable_l5_enabled()) p4d_p = pgd_p; else if (pgd) p4d_p = (p4dval_t *)((pgd & PTE_PFN_MASK) + __START_KERNEL_map - phys_base); diff --git a/arch/x86/kernel/head_32.S b/arch/x86/kernel/head_32.S index b59e4fb40fd9..abe6df15a8fb 100644 --- a/arch/x86/kernel/head_32.S +++ b/arch/x86/kernel/head_32.S @@ -375,7 +375,7 @@ ENDPROC(startup_32_smp) */ __INIT setup_once: -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR /* * Configure the stack canary. The linker can't handle this by * relocation. Manually set base address in stack canary diff --git a/arch/x86/kernel/hpet.c b/arch/x86/kernel/hpet.c index 8ce4212e2b8d..346b24883911 100644 --- a/arch/x86/kernel/hpet.c +++ b/arch/x86/kernel/hpet.c @@ -610,7 +610,7 @@ static void hpet_msi_capability_lookup(unsigned int start_timer) if (!hpet_domain) return; - hpet_devs = kzalloc(sizeof(struct hpet_dev) * num_timers, GFP_KERNEL); + hpet_devs = kcalloc(num_timers, sizeof(struct hpet_dev), GFP_KERNEL); if (!hpet_devs) return; @@ -966,8 +966,8 @@ int __init hpet_enable(void) #endif cfg = hpet_readl(HPET_CFG); - hpet_boot_cfg = kmalloc((last + 2) * sizeof(*hpet_boot_cfg), - GFP_KERNEL); + hpet_boot_cfg = kmalloc_array(last + 2, sizeof(*hpet_boot_cfg), + GFP_KERNEL); if (hpet_boot_cfg) *hpet_boot_cfg = cfg; else @@ -975,8 +975,7 @@ int __init hpet_enable(void) cfg &= ~(HPET_CFG_ENABLE | HPET_CFG_LEGACY); hpet_writel(cfg, HPET_CFG); if (cfg) - pr_warn("HPET: Unrecognized bits %#x set in global cfg\n", - cfg); + pr_warn("Unrecognized bits %#x set in global cfg\n", cfg); for (i = 0; i <= last; ++i) { cfg = hpet_readl(HPET_Tn_CFG(i)); @@ -988,7 +987,7 @@ int __init hpet_enable(void) | HPET_TN_64BIT_CAP | HPET_TN_32BIT | HPET_TN_ROUTE | HPET_TN_FSB | HPET_TN_FSB_CAP); if (cfg) - pr_warn("HPET: Unrecognized bits %#x set in cfg#%u\n", + pr_warn("Unrecognized bits %#x set in cfg#%u\n", cfg, i); } hpet_print_config(); diff --git a/arch/x86/kernel/i8237.c b/arch/x86/kernel/i8237.c index 8eeaa81de066..0a3e70fd00d6 100644 --- a/arch/x86/kernel/i8237.c +++ b/arch/x86/kernel/i8237.c @@ -9,10 +9,12 @@ * your option) any later version. */ +#include #include #include #include +#include /* * This module just handles suspend/resume issues with the @@ -49,6 +51,29 @@ static struct syscore_ops i8237_syscore_ops = { static int __init i8237A_init_ops(void) { + /* + * From SKL PCH onwards, the legacy DMA device is removed in which the + * I/O ports (81h-83h, 87h, 89h-8Bh, 8Fh) related to it are removed + * as well. All removed ports must return 0xff for a inb() request. + * + * Note: DMA_PAGE_2 (port 0x81) should not be checked for detecting + * the presence of DMA device since it may be used by BIOS to decode + * LPC traffic for POST codes. Original LPC only decodes one byte of + * port 0x80 but some BIOS may choose to enhance PCH LPC port 0x8x + * decoding. + */ + if (dma_inb(DMA_PAGE_0) == 0xFF) + return -ENODEV; + + /* + * It is not required to load this driver as newer SoC may not + * support 8237 DMA or bus mastering from LPC. Platform firmware + * must announce the support for such legacy devices via + * ACPI_FADT_LEGACY_DEVICES field in FADT table. + */ + if (x86_pnpbios_disabled() && dmi_get_bios_year() >= 2017) + return -ENODEV; + register_syscore_ops(&i8237_syscore_ops); return 0; } diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c index 2c3a1b4294eb..74383a3780dc 100644 --- a/arch/x86/kernel/idt.c +++ b/arch/x86/kernel/idt.c @@ -317,15 +317,12 @@ void __init idt_setup_apic_and_irq_gates(void) set_intr_gate(i, entry); } - for_each_clear_bit_from(i, system_vectors, NR_VECTORS) { #ifdef CONFIG_X86_LOCAL_APIC + for_each_clear_bit_from(i, system_vectors, NR_VECTORS) { set_bit(i, system_vectors); set_intr_gate(i, spurious_interrupt); -#else - entry = irq_entries_start + 8 * (i - FIRST_EXTERNAL_VECTOR); - set_intr_gate(i, entry); -#endif } +#endif } /** diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c index a15fe0e92cf9..108c48d0d40e 100644 --- a/arch/x86/kernel/jailhouse.c +++ b/arch/x86/kernel/jailhouse.c @@ -37,7 +37,7 @@ static uint32_t __init jailhouse_detect(void) return jailhouse_cpuid_base(); } -static void jailhouse_get_wallclock(struct timespec *now) +static void jailhouse_get_wallclock(struct timespec64 *now) { memset(now, 0, sizeof(*now)); } diff --git a/arch/x86/kernel/ksysfs.c b/arch/x86/kernel/ksysfs.c index 8c1cc08f514f..163ae706a0d4 100644 --- a/arch/x86/kernel/ksysfs.c +++ b/arch/x86/kernel/ksysfs.c @@ -283,7 +283,7 @@ static int __init create_setup_data_nodes(struct kobject *parent) if (ret) goto out_setup_data_kobj; - kobjp = kmalloc(sizeof(*kobjp) * nr, GFP_KERNEL); + kobjp = kmalloc_array(nr, sizeof(*kobjp), GFP_KERNEL); if (!kobjp) { ret = -ENOMEM; goto out_setup_data_kobj; diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c index 8b26c9e01cc4..bf8d1eb7fca3 100644 --- a/arch/x86/kernel/kvmclock.c +++ b/arch/x86/kernel/kvmclock.c @@ -53,7 +53,7 @@ static struct pvclock_wall_clock *wall_clock; * have elapsed since the hypervisor wrote the data. So we try to account for * that with system time */ -static void kvm_get_wallclock(struct timespec *now) +static void kvm_get_wallclock(struct timespec64 *now) { struct pvclock_vcpu_time_info *vcpu_time; int low, high; @@ -72,7 +72,7 @@ static void kvm_get_wallclock(struct timespec *now) put_cpu(); } -static int kvm_set_wallclock(const struct timespec *now) +static int kvm_set_wallclock(const struct timespec64 *now) { return -ENODEV; } diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c index 6010449ca6d2..4c8acdfdc5a7 100644 --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -354,7 +354,8 @@ void arch_crash_save_vmcoreinfo(void) { VMCOREINFO_NUMBER(phys_base); VMCOREINFO_SYMBOL(init_top_pgt); - VMCOREINFO_NUMBER(pgtable_l5_enabled); + vmcoreinfo_append_str("NUMBER(pgtable_l5_enabled)=%d\n", + pgtable_l5_enabled()); #ifdef CONFIG_NUMA VMCOREINFO_SYMBOL(node_data); diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index 77625b60a510..ab5d9dd668d2 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -15,13 +15,11 @@ #include #include -static int forbid_dac __read_mostly; +static bool disable_dac_quirk __read_mostly; const struct dma_map_ops *dma_ops = &dma_direct_ops; EXPORT_SYMBOL(dma_ops); -static int iommu_sac_force __read_mostly; - #ifdef CONFIG_IOMMU_DEBUG int panic_on_overflow __read_mostly = 1; int force_iommu __read_mostly = 1; @@ -55,9 +53,6 @@ struct device x86_dma_fallback_dev = { }; EXPORT_SYMBOL(x86_dma_fallback_dev); -/* Number of entries preallocated for DMA-API debugging */ -#define PREALLOC_DMA_DEBUG_ENTRIES 65536 - void __init pci_iommu_alloc(void) { struct iommu_table_entry *p; @@ -76,7 +71,7 @@ void __init pci_iommu_alloc(void) } } -bool arch_dma_alloc_attrs(struct device **dev, gfp_t *gfp) +bool arch_dma_alloc_attrs(struct device **dev) { if (!*dev) *dev = &x86_dma_fallback_dev; @@ -125,13 +120,13 @@ static __init int iommu_setup(char *p) if (!strncmp(p, "nomerge", 7)) iommu_merge = 0; if (!strncmp(p, "forcesac", 8)) - iommu_sac_force = 1; + pr_warn("forcesac option ignored.\n"); if (!strncmp(p, "allowdac", 8)) - forbid_dac = 0; + pr_warn("allowdac option ignored.\n"); if (!strncmp(p, "nodac", 5)) - forbid_dac = 1; + pr_warn("nodac option ignored.\n"); if (!strncmp(p, "usedac", 6)) { - forbid_dac = -1; + disable_dac_quirk = true; return 1; } #ifdef CONFIG_SWIOTLB @@ -156,40 +151,9 @@ static __init int iommu_setup(char *p) } early_param("iommu", iommu_setup); -int arch_dma_supported(struct device *dev, u64 mask) -{ -#ifdef CONFIG_PCI - if (mask > 0xffffffff && forbid_dac > 0) { - dev_info(dev, "PCI: Disallowing DAC for device\n"); - return 0; - } -#endif - - /* Tell the device to use SAC when IOMMU force is on. This - allows the driver to use cheaper accesses in some cases. - - Problem with this is that if we overflow the IOMMU area and - return DAC as fallback address the device may not handle it - correctly. - - As a special case some controllers have a 39bit address - mode that is as efficient as 32bit (aic79xx). Don't force - SAC for these. Assume all masks <= 40 bits are of this - type. Normally this doesn't make any difference, but gives - more gentle handling of IOMMU overflow. */ - if (iommu_sac_force && (mask >= DMA_BIT_MASK(40))) { - dev_info(dev, "Force SAC with mask %Lx\n", mask); - return 0; - } - - return 1; -} -EXPORT_SYMBOL(arch_dma_supported); - static int __init pci_iommu_init(void) { struct iommu_table_entry *p; - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); #ifdef CONFIG_PCI dma_debug_add_bus(&pci_bus_type); @@ -209,11 +173,17 @@ rootfs_initcall(pci_iommu_init); #ifdef CONFIG_PCI /* Many VIA bridges seem to corrupt data for DAC. Disable it here */ +static int via_no_dac_cb(struct pci_dev *pdev, void *data) +{ + pdev->dev.dma_32bit_limit = true; + return 0; +} + static void via_no_dac(struct pci_dev *dev) { - if (forbid_dac == 0) { + if (!disable_dac_quirk) { dev_info(&dev->dev, "disabling DAC on VIA PCI bridge\n"); - forbid_dac = 1; + pci_walk_bus(dev->subordinate, via_no_dac_cb, NULL); } } DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_VIA, PCI_ANY_ID, diff --git a/arch/x86/kernel/perf_regs.c b/arch/x86/kernel/perf_regs.c index e47b2dbbdef3..c06c4c16c6b6 100644 --- a/arch/x86/kernel/perf_regs.c +++ b/arch/x86/kernel/perf_regs.c @@ -151,17 +151,19 @@ void perf_get_regs_user(struct perf_regs *regs_user, regs_user_copy->sp = user_regs->sp; regs_user_copy->cs = user_regs->cs; regs_user_copy->ss = user_regs->ss; - /* - * Most system calls don't save these registers, don't report them. + * Store user space frame-pointer value on sample + * to facilitate stack unwinding for cases when + * user space executable code has such support + * enabled at compile time: */ + regs_user_copy->bp = user_regs->bp; + regs_user_copy->bx = -1; - regs_user_copy->bp = -1; regs_user_copy->r12 = -1; regs_user_copy->r13 = -1; regs_user_copy->r14 = -1; regs_user_copy->r15 = -1; - /* * For this to be at all useful, we need a reasonable guess for * the ABI. Be careful: we're in NMI context, and we're diff --git a/arch/x86/kernel/platform-quirks.c b/arch/x86/kernel/platform-quirks.c index 235fe6008ac8..b348a672f71d 100644 --- a/arch/x86/kernel/platform-quirks.c +++ b/arch/x86/kernel/platform-quirks.c @@ -33,9 +33,14 @@ void __init x86_early_init_platform_quirks(void) x86_platform.set_legacy_features(); } +bool __init x86_pnpbios_disabled(void) +{ + return x86_platform.legacy.devices.pnpbios == 0; +} + #if defined(CONFIG_PNPBIOS) bool __init arch_pnpbios_disabled(void) { - return x86_platform.legacy.devices.pnpbios == 0; + return x86_pnpbios_disabled(); } #endif diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c index 5224c6099184..0ae659de21eb 100644 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -76,16 +76,14 @@ void __show_regs(struct pt_regs *regs, int all) savesegment(gs, gs); } - printk(KERN_DEFAULT "EIP: %pS\n", (void *)regs->ip); - printk(KERN_DEFAULT "EFLAGS: %08lx CPU: %d\n", regs->flags, - raw_smp_processor_id()); + show_ip(regs, KERN_DEFAULT); printk(KERN_DEFAULT "EAX: %08lx EBX: %08lx ECX: %08lx EDX: %08lx\n", regs->ax, regs->bx, regs->cx, regs->dx); printk(KERN_DEFAULT "ESI: %08lx EDI: %08lx EBP: %08lx ESP: %08lx\n", regs->si, regs->di, regs->bp, sp); - printk(KERN_DEFAULT " DS: %04x ES: %04x FS: %04x GS: %04x SS: %04x\n", - (u16)regs->ds, (u16)regs->es, (u16)regs->fs, gs, ss); + printk(KERN_DEFAULT "DS: %04x ES: %04x FS: %04x GS: %04x SS: %04x EFLAGS: %08lx\n", + (u16)regs->ds, (u16)regs->es, (u16)regs->fs, gs, ss, regs->flags); if (!all) return; diff --git a/arch/x86/kernel/ptrace.c b/arch/x86/kernel/ptrace.c index ed5c4cdf0a34..e2ee403865eb 100644 --- a/arch/x86/kernel/ptrace.c +++ b/arch/x86/kernel/ptrace.c @@ -1377,7 +1377,6 @@ static void fill_sigtrap_info(struct task_struct *tsk, tsk->thread.trap_nr = X86_TRAP_DB; tsk->thread.error_code = error_code; - memset(info, 0, sizeof(*info)); info->si_signo = SIGTRAP; info->si_code = si_code; info->si_addr = user_mode(regs) ? (void __user *)regs->ip : NULL; @@ -1395,6 +1394,7 @@ void send_sigtrap(struct task_struct *tsk, struct pt_regs *regs, { struct siginfo info; + clear_siginfo(&info); fill_sigtrap_info(tsk, regs, error_code, si_code, &info); /* Send us the fake SIGTRAP */ force_sig_info(SIGTRAP, &info, tsk); diff --git a/arch/x86/kernel/pvclock.c b/arch/x86/kernel/pvclock.c index 761f6af6efa5..637982efecd8 100644 --- a/arch/x86/kernel/pvclock.c +++ b/arch/x86/kernel/pvclock.c @@ -123,28 +123,35 @@ u64 pvclock_clocksource_read(struct pvclock_vcpu_time_info *src) void pvclock_read_wallclock(struct pvclock_wall_clock *wall_clock, struct pvclock_vcpu_time_info *vcpu_time, - struct timespec *ts) + struct timespec64 *ts) { u32 version; u64 delta; - struct timespec now; + struct timespec64 now; /* get wallclock at system boot */ do { version = wall_clock->version; rmb(); /* fetch version before time */ + /* + * Note: wall_clock->sec is a u32 value, so it can + * only store dates between 1970 and 2106. To allow + * times beyond that, we need to create a new hypercall + * interface with an extended pvclock_wall_clock structure + * like ARM has. + */ now.tv_sec = wall_clock->sec; now.tv_nsec = wall_clock->nsec; rmb(); /* fetch time before checking version */ } while ((wall_clock->version & 1) || (version != wall_clock->version)); delta = pvclock_clocksource_read(vcpu_time); /* time since system boot */ - delta += now.tv_sec * (u64)NSEC_PER_SEC + now.tv_nsec; + delta += now.tv_sec * NSEC_PER_SEC + now.tv_nsec; now.tv_nsec = do_div(delta, NSEC_PER_SEC); now.tv_sec = delta; - set_normalized_timespec(ts, now.tv_sec, now.tv_nsec); + set_normalized_timespec64(ts, now.tv_sec, now.tv_nsec); } void pvclock_set_pvti_cpu0_va(struct pvclock_vsyscall_time_info *pvti) diff --git a/arch/x86/kernel/rtc.c b/arch/x86/kernel/rtc.c index f7b82ed7b5b5..586f718b8e95 100644 --- a/arch/x86/kernel/rtc.c +++ b/arch/x86/kernel/rtc.c @@ -39,7 +39,7 @@ EXPORT_SYMBOL(rtc_lock); * jump to the next second precisely 500 ms later. Check the Motorola * MC146818A or Dallas DS12887 data sheet for details. */ -int mach_set_rtc_mmss(const struct timespec *now) +int mach_set_rtc_mmss(const struct timespec64 *now) { unsigned long long nowtime = now->tv_sec; struct rtc_time tm; @@ -60,7 +60,7 @@ int mach_set_rtc_mmss(const struct timespec *now) return retval; } -void mach_get_cmos_time(struct timespec *now) +void mach_get_cmos_time(struct timespec64 *now) { unsigned int status, year, mon, day, hour, min, sec, century = 0; unsigned long flags; @@ -118,7 +118,7 @@ void mach_get_cmos_time(struct timespec *now) } else year += CMOS_YEARS_OFFS; - now->tv_sec = mktime(year, mon, day, hour, min, sec); + now->tv_sec = mktime64(year, mon, day, hour, min, sec); now->tv_nsec = 0; } @@ -145,13 +145,13 @@ void rtc_cmos_write(unsigned char val, unsigned char addr) } EXPORT_SYMBOL(rtc_cmos_write); -int update_persistent_clock(struct timespec now) +int update_persistent_clock64(struct timespec64 now) { return x86_platform.set_wallclock(&now); } /* not static: needed by APM */ -void read_persistent_clock(struct timespec *ts) +void read_persistent_clock64(struct timespec64 *ts) { x86_platform.get_wallclock(ts); } diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index 5c623dfe39d1..2f86d883dd95 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -1312,11 +1312,3 @@ static int __init register_kernel_offset_dumper(void) return 0; } __initcall(register_kernel_offset_dumper); - -void arch_show_smap(struct seq_file *m, struct vm_area_struct *vma) -{ - if (!boot_cpu_has(X86_FEATURE_OSPKE)) - return; - - seq_printf(m, "ProtectionKey: %8u\n", vma_pkey(vma)); -} diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index da270b95fe4d..445ca11ff863 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -688,6 +688,12 @@ setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs) sigset_t *set = sigmask_to_save(); compat_sigset_t *cset = (compat_sigset_t *) set; + /* + * Increment event counter and perform fixup for the pre-signal + * frame. + */ + rseq_signal_deliver(regs); + /* Set up the stack frame */ if (is_ia32_frame(ksig)) { if (ksig->ka.sa.sa_flags & SA_SIGINFO) diff --git a/arch/x86/kernel/signal_compat.c b/arch/x86/kernel/signal_compat.c index 14c057f29979..9ccbf0576cd0 100644 --- a/arch/x86/kernel/signal_compat.c +++ b/arch/x86/kernel/signal_compat.c @@ -29,7 +29,7 @@ static inline void signal_compat_build_tests(void) BUILD_BUG_ON(NSIGFPE != 15); BUILD_BUG_ON(NSIGSEGV != 7); BUILD_BUG_ON(NSIGBUS != 5); - BUILD_BUG_ON(NSIGTRAP != 4); + BUILD_BUG_ON(NSIGTRAP != 5); BUILD_BUG_ON(NSIGCHLD != 6); BUILD_BUG_ON(NSIGSYS != 1); diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 9dd324ae4832..c2f7d1d2a5c3 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -81,13 +81,6 @@ #include #include -/* Number of siblings per CPU package */ -int smp_num_siblings = 1; -EXPORT_SYMBOL(smp_num_siblings); - -/* Last level cache ID of each logical CPU */ -DEFINE_PER_CPU_READ_MOSTLY(u16, cpu_llc_id) = BAD_APICID; - /* representing HT siblings of each logical CPU */ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map); EXPORT_PER_CPU_SYMBOL(cpu_sibling_map); diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c index a3f15ed545b5..6a78d4b36a79 100644 --- a/arch/x86/kernel/sys_x86_64.c +++ b/arch/x86/kernel/sys_x86_64.c @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 +#include #include #include #include @@ -19,7 +20,6 @@ #include #include -#include #include #include #include diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 03f3d7695dac..a535dd64de63 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -299,6 +299,7 @@ static void do_error_trap(struct pt_regs *regs, long error_code, char *str, if (notify_die(DIE_TRAP, str, regs, error_code, trapnr, signr) != NOTIFY_STOP) { cond_local_irq_enable(regs); + clear_siginfo(&info); do_trap(trapnr, signr, str, regs, error_code, fill_trap_info(regs, signr, trapnr, &info)); } @@ -854,6 +855,7 @@ static void math_error(struct pt_regs *regs, int error_code, int trapnr) task->thread.trap_nr = trapnr; task->thread.error_code = error_code; + clear_siginfo(&info); info.si_signo = SIGFPE; info.si_errno = 0; info.si_addr = (void __user *)uprobe_get_trap_addr(regs); @@ -929,6 +931,7 @@ dotraplinkage void do_iret_error(struct pt_regs *regs, long error_code) RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU"); local_irq_enable(); + clear_siginfo(&info); info.si_signo = SIGILL; info.si_errno = 0; info.si_code = ILL_BADSTK; diff --git a/arch/x86/kernel/umip.c b/arch/x86/kernel/umip.c index f44ce0fb3583..ff20b35e98dd 100644 --- a/arch/x86/kernel/umip.c +++ b/arch/x86/kernel/umip.c @@ -278,6 +278,7 @@ static void force_sig_info_umip_fault(void __user *addr, struct pt_regs *regs) tsk->thread.error_code = X86_PF_USER | X86_PF_WRITE; tsk->thread.trap_nr = X86_TRAP_PF; + clear_siginfo(&info); info.si_signo = SIGSEGV; info.si_errno = 0; info.si_code = SEGV_MAPERR; diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c index c84bb5396958..58d8d800875d 100644 --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -1083,8 +1083,8 @@ arch_uretprobe_hijack_return_addr(unsigned long trampoline_vaddr, struct pt_regs return orig_ret_vaddr; if (nleft != rasize) { - pr_err("uprobe: return address clobbered: pid=%d, %%sp=%#lx, " - "%%ip=%#lx\n", current->pid, regs->sp, regs->ip); + pr_err("return address clobbered: pid=%d, %%sp=%#lx, %%ip=%#lx\n", + current->pid, regs->sp, regs->ip); force_sig_info(SIGSEGV, SEND_SIG_FORCED, current); } diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 795f3a80e576..5e1458f609a1 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -117,11 +117,11 @@ SECTIONS #ifdef CONFIG_X86_64 . = ALIGN(PAGE_SIZE); - VMLINUX_SYMBOL(__entry_trampoline_start) = .; + __entry_trampoline_start = .; _entry_trampoline = .; *(.entry_trampoline) . = ALIGN(PAGE_SIZE); - VMLINUX_SYMBOL(__entry_trampoline_end) = .; + __entry_trampoline_end = .; ASSERT(. - _entry_trampoline == PAGE_SIZE, "entry trampoline is too big"); #endif diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index 92bf2f2e7cdd..7e042e3d47fd 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -203,8 +203,9 @@ int kvm_vcpu_ioctl_set_cpuid(struct kvm_vcpu *vcpu, goto out; r = -ENOMEM; if (cpuid->nent) { - cpuid_entries = vmalloc(sizeof(struct kvm_cpuid_entry) * - cpuid->nent); + cpuid_entries = + vmalloc(array_size(sizeof(struct kvm_cpuid_entry), + cpuid->nent)); if (!cpuid_entries) goto out; r = -EFAULT; @@ -379,7 +380,8 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function, /* cpuid 0x80000008.ebx */ const u32 kvm_cpuid_8000_0008_ebx_x86_features = - F(AMD_IBPB) | F(AMD_IBRS) | F(VIRT_SSBD); + F(AMD_IBPB) | F(AMD_IBRS) | F(AMD_SSBD) | F(VIRT_SSBD) | + F(AMD_SSB_NO); /* cpuid 0xC0000001.edx */ const u32 kvm_cpuid_C000_0001_edx_x86_features = @@ -403,7 +405,8 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function, const u32 kvm_cpuid_7_0_ecx_x86_features = F(AVX512VBMI) | F(LA57) | F(PKU) | 0 /*OSPKE*/ | F(AVX512_VPOPCNTDQ) | F(UMIP) | F(AVX512_VBMI2) | F(GFNI) | - F(VAES) | F(VPCLMULQDQ) | F(AVX512_VNNI) | F(AVX512_BITALG); + F(VAES) | F(VPCLMULQDQ) | F(AVX512_VNNI) | F(AVX512_BITALG) | + F(CLDEMOTE); /* cpuid 7.0.edx*/ const u32 kvm_cpuid_7_0_edx_x86_features = @@ -664,7 +667,12 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function, entry->ebx |= F(VIRT_SSBD); entry->ebx &= kvm_cpuid_8000_0008_ebx_x86_features; cpuid_mask(&entry->ebx, CPUID_8000_0008_EBX); - if (boot_cpu_has(X86_FEATURE_LS_CFG_SSBD)) + /* + * The preference is to use SPEC CTRL MSR instead of the + * VIRT_SPEC MSR. + */ + if (boot_cpu_has(X86_FEATURE_LS_CFG_SSBD) && + !boot_cpu_has(X86_FEATURE_AMD_SSBD)) entry->ebx |= F(VIRT_SSBD); break; } @@ -778,7 +786,8 @@ int kvm_dev_ioctl_get_cpuid(struct kvm_cpuid2 *cpuid, return -EINVAL; r = -ENOMEM; - cpuid_entries = vzalloc(sizeof(struct kvm_cpuid_entry2) * cpuid->nent); + cpuid_entries = vzalloc(array_size(sizeof(struct kvm_cpuid_entry2), + cpuid->nent)); if (!cpuid_entries) goto out; diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c index b3705ae52824..4c4f4263420c 100644 --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -812,6 +812,19 @@ static inline int jmp_rel(struct x86_emulate_ctxt *ctxt, int rel) return assign_eip_near(ctxt, ctxt->_eip + rel); } +static int linear_read_system(struct x86_emulate_ctxt *ctxt, ulong linear, + void *data, unsigned size) +{ + return ctxt->ops->read_std(ctxt, linear, data, size, &ctxt->exception, true); +} + +static int linear_write_system(struct x86_emulate_ctxt *ctxt, + ulong linear, void *data, + unsigned int size) +{ + return ctxt->ops->write_std(ctxt, linear, data, size, &ctxt->exception, true); +} + static int segmented_read_std(struct x86_emulate_ctxt *ctxt, struct segmented_address addr, void *data, @@ -823,7 +836,7 @@ static int segmented_read_std(struct x86_emulate_ctxt *ctxt, rc = linearize(ctxt, addr, size, false, &linear); if (rc != X86EMUL_CONTINUE) return rc; - return ctxt->ops->read_std(ctxt, linear, data, size, &ctxt->exception); + return ctxt->ops->read_std(ctxt, linear, data, size, &ctxt->exception, false); } static int segmented_write_std(struct x86_emulate_ctxt *ctxt, @@ -837,7 +850,7 @@ static int segmented_write_std(struct x86_emulate_ctxt *ctxt, rc = linearize(ctxt, addr, size, true, &linear); if (rc != X86EMUL_CONTINUE) return rc; - return ctxt->ops->write_std(ctxt, linear, data, size, &ctxt->exception); + return ctxt->ops->write_std(ctxt, linear, data, size, &ctxt->exception, false); } /* @@ -1496,8 +1509,7 @@ static int read_interrupt_descriptor(struct x86_emulate_ctxt *ctxt, return emulate_gp(ctxt, index << 3 | 0x2); addr = dt.address + index * 8; - return ctxt->ops->read_std(ctxt, addr, desc, sizeof *desc, - &ctxt->exception); + return linear_read_system(ctxt, addr, desc, sizeof *desc); } static void get_descriptor_table_ptr(struct x86_emulate_ctxt *ctxt, @@ -1560,8 +1572,7 @@ static int read_segment_descriptor(struct x86_emulate_ctxt *ctxt, if (rc != X86EMUL_CONTINUE) return rc; - return ctxt->ops->read_std(ctxt, *desc_addr_p, desc, sizeof(*desc), - &ctxt->exception); + return linear_read_system(ctxt, *desc_addr_p, desc, sizeof(*desc)); } /* allowed just for 8 bytes segments */ @@ -1575,8 +1586,7 @@ static int write_segment_descriptor(struct x86_emulate_ctxt *ctxt, if (rc != X86EMUL_CONTINUE) return rc; - return ctxt->ops->write_std(ctxt, addr, desc, sizeof *desc, - &ctxt->exception); + return linear_write_system(ctxt, addr, desc, sizeof *desc); } static int __load_segment_descriptor(struct x86_emulate_ctxt *ctxt, @@ -1737,8 +1747,7 @@ static int __load_segment_descriptor(struct x86_emulate_ctxt *ctxt, return ret; } } else if (ctxt->mode == X86EMUL_MODE_PROT64) { - ret = ctxt->ops->read_std(ctxt, desc_addr+8, &base3, - sizeof(base3), &ctxt->exception); + ret = linear_read_system(ctxt, desc_addr+8, &base3, sizeof(base3)); if (ret != X86EMUL_CONTINUE) return ret; if (emul_is_noncanonical_address(get_desc_base(&seg_desc) | @@ -2051,11 +2060,11 @@ static int __emulate_int_real(struct x86_emulate_ctxt *ctxt, int irq) eip_addr = dt.address + (irq << 2); cs_addr = dt.address + (irq << 2) + 2; - rc = ops->read_std(ctxt, cs_addr, &cs, 2, &ctxt->exception); + rc = linear_read_system(ctxt, cs_addr, &cs, 2); if (rc != X86EMUL_CONTINUE) return rc; - rc = ops->read_std(ctxt, eip_addr, &eip, 2, &ctxt->exception); + rc = linear_read_system(ctxt, eip_addr, &eip, 2); if (rc != X86EMUL_CONTINUE) return rc; @@ -2919,12 +2928,12 @@ static bool emulator_io_port_access_allowed(struct x86_emulate_ctxt *ctxt, #ifdef CONFIG_X86_64 base |= ((u64)base3) << 32; #endif - r = ops->read_std(ctxt, base + 102, &io_bitmap_ptr, 2, NULL); + r = ops->read_std(ctxt, base + 102, &io_bitmap_ptr, 2, NULL, true); if (r != X86EMUL_CONTINUE) return false; if (io_bitmap_ptr + port/8 > desc_limit_scaled(&tr_seg)) return false; - r = ops->read_std(ctxt, base + io_bitmap_ptr + port/8, &perm, 2, NULL); + r = ops->read_std(ctxt, base + io_bitmap_ptr + port/8, &perm, 2, NULL, true); if (r != X86EMUL_CONTINUE) return false; if ((perm >> bit_idx) & mask) @@ -3053,35 +3062,30 @@ static int task_switch_16(struct x86_emulate_ctxt *ctxt, u16 tss_selector, u16 old_tss_sel, ulong old_tss_base, struct desc_struct *new_desc) { - const struct x86_emulate_ops *ops = ctxt->ops; struct tss_segment_16 tss_seg; int ret; u32 new_tss_base = get_desc_base(new_desc); - ret = ops->read_std(ctxt, old_tss_base, &tss_seg, sizeof tss_seg, - &ctxt->exception); + ret = linear_read_system(ctxt, old_tss_base, &tss_seg, sizeof tss_seg); if (ret != X86EMUL_CONTINUE) return ret; save_state_to_tss16(ctxt, &tss_seg); - ret = ops->write_std(ctxt, old_tss_base, &tss_seg, sizeof tss_seg, - &ctxt->exception); + ret = linear_write_system(ctxt, old_tss_base, &tss_seg, sizeof tss_seg); if (ret != X86EMUL_CONTINUE) return ret; - ret = ops->read_std(ctxt, new_tss_base, &tss_seg, sizeof tss_seg, - &ctxt->exception); + ret = linear_read_system(ctxt, new_tss_base, &tss_seg, sizeof tss_seg); if (ret != X86EMUL_CONTINUE) return ret; if (old_tss_sel != 0xffff) { tss_seg.prev_task_link = old_tss_sel; - ret = ops->write_std(ctxt, new_tss_base, - &tss_seg.prev_task_link, - sizeof tss_seg.prev_task_link, - &ctxt->exception); + ret = linear_write_system(ctxt, new_tss_base, + &tss_seg.prev_task_link, + sizeof tss_seg.prev_task_link); if (ret != X86EMUL_CONTINUE) return ret; } @@ -3197,38 +3201,34 @@ static int task_switch_32(struct x86_emulate_ctxt *ctxt, u16 tss_selector, u16 old_tss_sel, ulong old_tss_base, struct desc_struct *new_desc) { - const struct x86_emulate_ops *ops = ctxt->ops; struct tss_segment_32 tss_seg; int ret; u32 new_tss_base = get_desc_base(new_desc); u32 eip_offset = offsetof(struct tss_segment_32, eip); u32 ldt_sel_offset = offsetof(struct tss_segment_32, ldt_selector); - ret = ops->read_std(ctxt, old_tss_base, &tss_seg, sizeof tss_seg, - &ctxt->exception); + ret = linear_read_system(ctxt, old_tss_base, &tss_seg, sizeof tss_seg); if (ret != X86EMUL_CONTINUE) return ret; save_state_to_tss32(ctxt, &tss_seg); /* Only GP registers and segment selectors are saved */ - ret = ops->write_std(ctxt, old_tss_base + eip_offset, &tss_seg.eip, - ldt_sel_offset - eip_offset, &ctxt->exception); + ret = linear_write_system(ctxt, old_tss_base + eip_offset, &tss_seg.eip, + ldt_sel_offset - eip_offset); if (ret != X86EMUL_CONTINUE) return ret; - ret = ops->read_std(ctxt, new_tss_base, &tss_seg, sizeof tss_seg, - &ctxt->exception); + ret = linear_read_system(ctxt, new_tss_base, &tss_seg, sizeof tss_seg); if (ret != X86EMUL_CONTINUE) return ret; if (old_tss_sel != 0xffff) { tss_seg.prev_task_link = old_tss_sel; - ret = ops->write_std(ctxt, new_tss_base, - &tss_seg.prev_task_link, - sizeof tss_seg.prev_task_link, - &ctxt->exception); + ret = linear_write_system(ctxt, new_tss_base, + &tss_seg.prev_task_link, + sizeof tss_seg.prev_task_link); if (ret != X86EMUL_CONTINUE) return ret; } @@ -4189,7 +4189,9 @@ static int check_cr_write(struct x86_emulate_ctxt *ctxt) maxphyaddr = eax & 0xff; else maxphyaddr = 36; - rsvd = rsvd_bits(maxphyaddr, 62); + rsvd = rsvd_bits(maxphyaddr, 63); + if (ctxt->ops->get_cr(ctxt, 4) & X86_CR4_PCIDE) + rsvd &= ~CR3_PCID_INVD; } if (new_val & rsvd) diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index 46ff64da44ca..af8caf965baa 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -1242,6 +1242,121 @@ int kvm_hv_get_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata) return kvm_hv_get_msr(vcpu, msr, pdata); } +static __always_inline int get_sparse_bank_no(u64 valid_bank_mask, int bank_no) +{ + int i = 0, j; + + if (!(valid_bank_mask & BIT_ULL(bank_no))) + return -1; + + for (j = 0; j < bank_no; j++) + if (valid_bank_mask & BIT_ULL(j)) + i++; + + return i; +} + +static u64 kvm_hv_flush_tlb(struct kvm_vcpu *current_vcpu, u64 ingpa, + u16 rep_cnt, bool ex) +{ + struct kvm *kvm = current_vcpu->kvm; + struct kvm_vcpu_hv *hv_current = ¤t_vcpu->arch.hyperv; + struct hv_tlb_flush_ex flush_ex; + struct hv_tlb_flush flush; + struct kvm_vcpu *vcpu; + unsigned long vcpu_bitmap[BITS_TO_LONGS(KVM_MAX_VCPUS)] = {0}; + unsigned long valid_bank_mask = 0; + u64 sparse_banks[64]; + int sparse_banks_len, i; + bool all_cpus; + + if (!ex) { + if (unlikely(kvm_read_guest(kvm, ingpa, &flush, sizeof(flush)))) + return HV_STATUS_INVALID_HYPERCALL_INPUT; + + trace_kvm_hv_flush_tlb(flush.processor_mask, + flush.address_space, flush.flags); + + sparse_banks[0] = flush.processor_mask; + all_cpus = flush.flags & HV_FLUSH_ALL_PROCESSORS; + } else { + if (unlikely(kvm_read_guest(kvm, ingpa, &flush_ex, + sizeof(flush_ex)))) + return HV_STATUS_INVALID_HYPERCALL_INPUT; + + trace_kvm_hv_flush_tlb_ex(flush_ex.hv_vp_set.valid_bank_mask, + flush_ex.hv_vp_set.format, + flush_ex.address_space, + flush_ex.flags); + + valid_bank_mask = flush_ex.hv_vp_set.valid_bank_mask; + all_cpus = flush_ex.hv_vp_set.format != + HV_GENERIC_SET_SPARSE_4K; + + sparse_banks_len = bitmap_weight(&valid_bank_mask, 64) * + sizeof(sparse_banks[0]); + + if (!sparse_banks_len && !all_cpus) + goto ret_success; + + if (!all_cpus && + kvm_read_guest(kvm, + ingpa + offsetof(struct hv_tlb_flush_ex, + hv_vp_set.bank_contents), + sparse_banks, + sparse_banks_len)) + return HV_STATUS_INVALID_HYPERCALL_INPUT; + } + + cpumask_clear(&hv_current->tlb_lush); + + kvm_for_each_vcpu(i, vcpu, kvm) { + struct kvm_vcpu_hv *hv = &vcpu->arch.hyperv; + int bank = hv->vp_index / 64, sbank = 0; + + if (!all_cpus) { + /* Banks >64 can't be represented */ + if (bank >= 64) + continue; + + /* Non-ex hypercalls can only address first 64 vCPUs */ + if (!ex && bank) + continue; + + if (ex) { + /* + * Check is the bank of this vCPU is in sparse + * set and get the sparse bank number. + */ + sbank = get_sparse_bank_no(valid_bank_mask, + bank); + + if (sbank < 0) + continue; + } + + if (!(sparse_banks[sbank] & BIT_ULL(hv->vp_index % 64))) + continue; + } + + /* + * vcpu->arch.cr3 may not be up-to-date for running vCPUs so we + * can't analyze it here, flush TLB regardless of the specified + * address space. + */ + __set_bit(i, vcpu_bitmap); + } + + kvm_make_vcpus_request_mask(kvm, + KVM_REQ_TLB_FLUSH | KVM_REQUEST_NO_WAKEUP, + vcpu_bitmap, &hv_current->tlb_lush); + +ret_success: + /* We always do full TLB flush, set rep_done = rep_cnt. */ + return (u64)HV_STATUS_SUCCESS | + ((u64)rep_cnt << HV_HYPERCALL_REP_COMP_OFFSET); +} + bool kvm_hv_hypercall_enabled(struct kvm *kvm) { return READ_ONCE(kvm->arch.hyperv.hv_hypercall) & HV_X64_MSR_HYPERCALL_ENABLE; @@ -1315,7 +1430,7 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) { u64 param, ingpa, outgpa, ret = HV_STATUS_SUCCESS; uint16_t code, rep_idx, rep_cnt; - bool fast, longmode; + bool fast, longmode, rep; /* * hypercall generates UD from non zero cpl and real mode @@ -1345,31 +1460,34 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) #endif code = param & 0xffff; - fast = (param >> 16) & 0x1; - rep_cnt = (param >> 32) & 0xfff; - rep_idx = (param >> 48) & 0xfff; + fast = !!(param & HV_HYPERCALL_FAST_BIT); + rep_cnt = (param >> HV_HYPERCALL_REP_COMP_OFFSET) & 0xfff; + rep_idx = (param >> HV_HYPERCALL_REP_START_OFFSET) & 0xfff; + rep = !!(rep_cnt || rep_idx); trace_kvm_hv_hypercall(code, fast, rep_cnt, rep_idx, ingpa, outgpa); - /* Hypercall continuation is not supported yet */ - if (rep_cnt || rep_idx) { - ret = HV_STATUS_INVALID_HYPERCALL_CODE; - goto out; - } - switch (code) { case HVCALL_NOTIFY_LONG_SPIN_WAIT: + if (unlikely(rep)) { + ret = HV_STATUS_INVALID_HYPERCALL_INPUT; + break; + } kvm_vcpu_on_spin(vcpu, true); break; case HVCALL_SIGNAL_EVENT: + if (unlikely(rep)) { + ret = HV_STATUS_INVALID_HYPERCALL_INPUT; + break; + } ret = kvm_hvcall_signal_event(vcpu, fast, ingpa); if (ret != HV_STATUS_INVALID_PORT_ID) break; /* maybe userspace knows this conn_id: fall through */ case HVCALL_POST_MESSAGE: /* don't bother userspace if it has no way to handle it */ - if (!vcpu_to_synic(vcpu)->active) { - ret = HV_STATUS_INVALID_HYPERCALL_CODE; + if (unlikely(rep || !vcpu_to_synic(vcpu)->active)) { + ret = HV_STATUS_INVALID_HYPERCALL_INPUT; break; } vcpu->run->exit_reason = KVM_EXIT_HYPERV; @@ -1380,12 +1498,39 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace; return 0; + case HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST: + if (unlikely(fast || !rep_cnt || rep_idx)) { + ret = HV_STATUS_INVALID_HYPERCALL_INPUT; + break; + } + ret = kvm_hv_flush_tlb(vcpu, ingpa, rep_cnt, false); + break; + case HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE: + if (unlikely(fast || rep)) { + ret = HV_STATUS_INVALID_HYPERCALL_INPUT; + break; + } + ret = kvm_hv_flush_tlb(vcpu, ingpa, rep_cnt, false); + break; + case HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX: + if (unlikely(fast || !rep_cnt || rep_idx)) { + ret = HV_STATUS_INVALID_HYPERCALL_INPUT; + break; + } + ret = kvm_hv_flush_tlb(vcpu, ingpa, rep_cnt, true); + break; + case HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX: + if (unlikely(fast || rep)) { + ret = HV_STATUS_INVALID_HYPERCALL_INPUT; + break; + } + ret = kvm_hv_flush_tlb(vcpu, ingpa, rep_cnt, true); + break; default: ret = HV_STATUS_INVALID_HYPERCALL_CODE; break; } -out: return kvm_hv_hypercall_complete(vcpu, ret); } diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 3773c4625114..b5cd8465d44f 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -2002,13 +2002,11 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 value) } } - if ((old_value ^ value) & X2APIC_ENABLE) { - if (value & X2APIC_ENABLE) { - kvm_apic_set_x2apic_id(apic, vcpu->vcpu_id); - kvm_x86_ops->set_virtual_x2apic_mode(vcpu, true); - } else - kvm_x86_ops->set_virtual_x2apic_mode(vcpu, false); - } + if (((old_value ^ value) & X2APIC_ENABLE) && (value & X2APIC_ENABLE)) + kvm_apic_set_x2apic_id(apic, vcpu->vcpu_id); + + if ((old_value ^ value) & (MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE)) + kvm_x86_ops->set_virtual_apic_mode(vcpu); apic->base_address = apic->vcpu->arch.apic_base & MSR_IA32_APICBASE_BASE; diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h index edce055e9fd7..ed0ed39abd36 100644 --- a/arch/x86/kvm/lapic.h +++ b/arch/x86/kvm/lapic.h @@ -16,6 +16,13 @@ #define APIC_BUS_CYCLE_NS 1 #define APIC_BUS_FREQUENCY (1000000000ULL / APIC_BUS_CYCLE_NS) +enum lapic_mode { + LAPIC_MODE_DISABLED = 0, + LAPIC_MODE_INVALID = X2APIC_ENABLE, + LAPIC_MODE_XAPIC = MSR_IA32_APICBASE_ENABLE, + LAPIC_MODE_X2APIC = MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE, +}; + struct kvm_timer { struct hrtimer timer; s64 period; /* unit: ns */ @@ -89,6 +96,7 @@ u64 kvm_get_apic_base(struct kvm_vcpu *vcpu); int kvm_set_apic_base(struct kvm_vcpu *vcpu, struct msr_data *msr_info); int kvm_apic_get_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s); int kvm_apic_set_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s); +enum lapic_mode kvm_get_apic_mode(struct kvm_vcpu *vcpu); int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu); u64 kvm_get_lapic_tscdeadline_msr(struct kvm_vcpu *vcpu); @@ -220,4 +228,10 @@ void kvm_lapic_switch_to_hv_timer(struct kvm_vcpu *vcpu); void kvm_lapic_expired_hv_timer(struct kvm_vcpu *vcpu); bool kvm_lapic_hv_timer_in_use(struct kvm_vcpu *vcpu); void kvm_lapic_restart_hv_timer(struct kvm_vcpu *vcpu); + +static inline enum lapic_mode kvm_apic_mode(u64 apic_base) +{ + return apic_base & (MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE); +} + #endif diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c index 8494dbae41b9..d594690d8b95 100644 --- a/arch/x86/kvm/mmu.c +++ b/arch/x86/kvm/mmu.c @@ -222,7 +222,6 @@ static const u64 shadow_acc_track_saved_bits_mask = PT64_EPT_READABLE_MASK | static const u64 shadow_acc_track_saved_bits_shift = PT64_SECOND_AVAIL_BITS_SHIFT; static void mmu_spte_set(u64 *sptep, u64 spte); -static void mmu_free_roots(struct kvm_vcpu *vcpu); void kvm_mmu_set_mmio_spte_mask(u64 mmio_mask, u64 mmio_value) { @@ -3007,6 +3006,7 @@ static void kvm_send_hwpoison_signal(unsigned long address, struct task_struct * { siginfo_t info; + clear_siginfo(&info); info.si_signo = SIGBUS; info.si_errno = 0; info.si_code = BUS_MCEERR_AR; @@ -3342,51 +3342,48 @@ out_unlock: return RET_PF_RETRY; } - -static void mmu_free_roots(struct kvm_vcpu *vcpu) +static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa, + struct list_head *invalid_list) { - int i; struct kvm_mmu_page *sp; - LIST_HEAD(invalid_list); - if (!VALID_PAGE(vcpu->arch.mmu.root_hpa)) + if (!VALID_PAGE(*root_hpa)) return; - if (vcpu->arch.mmu.shadow_root_level >= PT64_ROOT_4LEVEL && - (vcpu->arch.mmu.root_level >= PT64_ROOT_4LEVEL || - vcpu->arch.mmu.direct_map)) { - hpa_t root = vcpu->arch.mmu.root_hpa; + sp = page_header(*root_hpa & PT64_BASE_ADDR_MASK); + --sp->root_count; + if (!sp->root_count && sp->role.invalid) + kvm_mmu_prepare_zap_page(kvm, sp, invalid_list); - spin_lock(&vcpu->kvm->mmu_lock); - sp = page_header(root); - --sp->root_count; - if (!sp->root_count && sp->role.invalid) { - kvm_mmu_prepare_zap_page(vcpu->kvm, sp, &invalid_list); - kvm_mmu_commit_zap_page(vcpu->kvm, &invalid_list); - } - spin_unlock(&vcpu->kvm->mmu_lock); - vcpu->arch.mmu.root_hpa = INVALID_PAGE; + *root_hpa = INVALID_PAGE; +} + +void kvm_mmu_free_roots(struct kvm_vcpu *vcpu) +{ + int i; + LIST_HEAD(invalid_list); + struct kvm_mmu *mmu = &vcpu->arch.mmu; + + if (!VALID_PAGE(mmu->root_hpa)) return; - } spin_lock(&vcpu->kvm->mmu_lock); - for (i = 0; i < 4; ++i) { - hpa_t root = vcpu->arch.mmu.pae_root[i]; - if (root) { - root &= PT64_BASE_ADDR_MASK; - sp = page_header(root); - --sp->root_count; - if (!sp->root_count && sp->role.invalid) - kvm_mmu_prepare_zap_page(vcpu->kvm, sp, - &invalid_list); - } - vcpu->arch.mmu.pae_root[i] = INVALID_PAGE; + if (mmu->shadow_root_level >= PT64_ROOT_4LEVEL && + (mmu->root_level >= PT64_ROOT_4LEVEL || mmu->direct_map)) { + mmu_free_root_page(vcpu->kvm, &mmu->root_hpa, &invalid_list); + } else { + for (i = 0; i < 4; ++i) + if (mmu->pae_root[i] != 0) + mmu_free_root_page(vcpu->kvm, &mmu->pae_root[i], + &invalid_list); + mmu->root_hpa = INVALID_PAGE; } + kvm_mmu_commit_zap_page(vcpu->kvm, &invalid_list); spin_unlock(&vcpu->kvm->mmu_lock); - vcpu->arch.mmu.root_hpa = INVALID_PAGE; } +EXPORT_SYMBOL_GPL(kvm_mmu_free_roots); static int mmu_check_root(struct kvm_vcpu *vcpu, gfn_t root_gfn) { @@ -3719,7 +3716,6 @@ static int handle_mmio_page_fault(struct kvm_vcpu *vcpu, u64 addr, bool direct) */ return RET_PF_RETRY; } -EXPORT_SYMBOL_GPL(handle_mmio_page_fault); static bool page_fault_handle_page_track(struct kvm_vcpu *vcpu, u32 error_code, gfn_t gfn) @@ -3811,6 +3807,14 @@ static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn, struct kvm_memory_slot *slot; bool async; + /* + * Don't expose private memslots to L2. + */ + if (is_guest_mode(vcpu) && !kvm_is_visible_gfn(vcpu->kvm, gfn)) { + *pfn = KVM_PFN_NOSLOT; + return false; + } + slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn); async = false; *pfn = __gfn_to_pfn_memslot(slot, gfn, false, &async, write, writable); @@ -3950,7 +3954,7 @@ static void nonpaging_init_context(struct kvm_vcpu *vcpu, void kvm_mmu_new_cr3(struct kvm_vcpu *vcpu) { - mmu_free_roots(vcpu); + kvm_mmu_free_roots(vcpu); } static unsigned long get_cr3(struct kvm_vcpu *vcpu) @@ -4472,6 +4476,7 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu) struct kvm_mmu *context = &vcpu->arch.mmu; context->base_role.word = 0; + context->base_role.guest_mode = is_guest_mode(vcpu); context->base_role.smm = is_smm(vcpu); context->base_role.ad_disabled = (shadow_accessed_mask == 0); context->page_fault = tdp_page_fault; @@ -4538,6 +4543,7 @@ void kvm_init_shadow_mmu(struct kvm_vcpu *vcpu) = smep && !is_write_protection(vcpu); context->base_role.smap_andnot_wp = smap && !is_write_protection(vcpu); + context->base_role.guest_mode = is_guest_mode(vcpu); context->base_role.smm = is_smm(vcpu); reset_shadow_zero_bits_mask(vcpu, context); } @@ -4563,7 +4569,7 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, context->root_hpa = INVALID_PAGE; context->direct_map = false; context->base_role.ad_disabled = !accessed_dirty; - + context->base_role.guest_mode = 1; update_permission_bitmask(vcpu, context, true); update_pkru_bitmask(vcpu, context, true); update_last_nonleaf_level(vcpu, context); @@ -4663,7 +4669,7 @@ EXPORT_SYMBOL_GPL(kvm_mmu_load); void kvm_mmu_unload(struct kvm_vcpu *vcpu) { - mmu_free_roots(vcpu); + kvm_mmu_free_roots(vcpu); WARN_ON(VALID_PAGE(vcpu->arch.mmu.root_hpa)); } EXPORT_SYMBOL_GPL(kvm_mmu_unload); @@ -4824,6 +4830,7 @@ static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, mask.smep_andnot_wp = 1; mask.smap_andnot_wp = 1; mask.smm = 1; + mask.guest_mode = 1; mask.ad_disabled = 1; /* diff --git a/arch/x86/kvm/page_track.c b/arch/x86/kvm/page_track.c index 01c1371f39f8..3052a59a3065 100644 --- a/arch/x86/kvm/page_track.c +++ b/arch/x86/kvm/page_track.c @@ -40,8 +40,9 @@ int kvm_page_track_create_memslot(struct kvm_memory_slot *slot, int i; for (i = 0; i < KVM_PAGE_TRACK_MAX; i++) { - slot->arch.gfn_track[i] = kvzalloc(npages * - sizeof(*slot->arch.gfn_track[i]), GFP_KERNEL); + slot->arch.gfn_track[i] = + kvcalloc(npages, sizeof(*slot->arch.gfn_track[i]), + GFP_KERNEL); if (!slot->arch.gfn_track[i]) goto track_free; } diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c index 26110c202b19..f059a73f0fd0 100644 --- a/arch/x86/kvm/svm.c +++ b/arch/x86/kvm/svm.c @@ -1001,7 +1001,9 @@ static int svm_cpu_init(int cpu) if (svm_sev_enabled()) { r = -ENOMEM; - sd->sev_vmcbs = kmalloc((max_sev_asid + 1) * sizeof(void *), GFP_KERNEL); + sd->sev_vmcbs = kmalloc_array(max_sev_asid + 1, + sizeof(void *), + GFP_KERNEL); if (!sd->sev_vmcbs) goto err_1; } @@ -1768,7 +1770,10 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, unsigned long npages, npinned, size; unsigned long locked, lock_limit; struct page **pages; - int first, last; + unsigned long first, last; + + if (ulen == 0 || uaddr + ulen < uaddr) + return NULL; /* Calculate number of pages. */ first = (uaddr & PAGE_MASK) >> PAGE_SHIFT; @@ -1855,13 +1860,13 @@ static void __unregister_enc_region_locked(struct kvm *kvm, static struct kvm *svm_vm_alloc(void) { - struct kvm_svm *kvm_svm = kzalloc(sizeof(struct kvm_svm), GFP_KERNEL); + struct kvm_svm *kvm_svm = vzalloc(sizeof(struct kvm_svm)); return &kvm_svm->kvm; } static void svm_vm_free(struct kvm *kvm) { - kfree(to_kvm_svm(kvm)); + vfree(to_kvm_svm(kvm)); } static void sev_vm_destroy(struct kvm *kvm) @@ -4115,7 +4120,8 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) break; case MSR_IA32_SPEC_CTRL: if (!msr_info->host_initiated && - !guest_cpuid_has(vcpu, X86_FEATURE_AMD_IBRS)) + !guest_cpuid_has(vcpu, X86_FEATURE_AMD_IBRS) && + !guest_cpuid_has(vcpu, X86_FEATURE_AMD_SSBD)) return 1; msr_info->data = svm->spec_ctrl; @@ -4217,11 +4223,12 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) break; case MSR_IA32_SPEC_CTRL: if (!msr->host_initiated && - !guest_cpuid_has(vcpu, X86_FEATURE_AMD_IBRS)) + !guest_cpuid_has(vcpu, X86_FEATURE_AMD_IBRS) && + !guest_cpuid_has(vcpu, X86_FEATURE_AMD_SSBD)) return 1; /* The STIBP bit doesn't fault even if it's not advertised */ - if (data & ~(SPEC_CTRL_IBRS | SPEC_CTRL_STIBP)) + if (data & ~(SPEC_CTRL_IBRS | SPEC_CTRL_STIBP | SPEC_CTRL_SSBD)) return 1; svm->spec_ctrl = data; @@ -5060,7 +5067,7 @@ static void update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr) set_cr_intercept(svm, INTERCEPT_CR8_WRITE); } -static void svm_set_virtual_x2apic_mode(struct kvm_vcpu *vcpu, bool set) +static void svm_set_virtual_apic_mode(struct kvm_vcpu *vcpu) { return; } @@ -6947,6 +6954,9 @@ static int svm_register_enc_region(struct kvm *kvm, if (!sev_guest(kvm)) return -ENOTTY; + if (range->addr > ULONG_MAX || range->size > ULONG_MAX) + return -EINVAL; + region = kzalloc(sizeof(*region), GFP_KERNEL); if (!region) return -ENOMEM; @@ -7098,7 +7108,7 @@ static struct kvm_x86_ops svm_x86_ops __ro_after_init = { .enable_nmi_window = enable_nmi_window, .enable_irq_window = enable_irq_window, .update_cr8_intercept = update_cr8_intercept, - .set_virtual_x2apic_mode = svm_set_virtual_x2apic_mode, + .set_virtual_apic_mode = svm_set_virtual_apic_mode, .get_enable_apicv = svm_get_enable_apicv, .refresh_apicv_exec_ctrl = svm_refresh_apicv_exec_ctrl, .load_eoi_exitmap = svm_load_eoi_exitmap, diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h index 9807c314c478..0f997683404f 100644 --- a/arch/x86/kvm/trace.h +++ b/arch/x86/kvm/trace.h @@ -1367,6 +1367,57 @@ TRACE_EVENT(kvm_hv_timer_state, __entry->vcpu_id, __entry->hv_timer_in_use) ); + +/* + * Tracepoint for kvm_hv_flush_tlb. + */ +TRACE_EVENT(kvm_hv_flush_tlb, + TP_PROTO(u64 processor_mask, u64 address_space, u64 flags), + TP_ARGS(processor_mask, address_space, flags), + + TP_STRUCT__entry( + __field(u64, processor_mask) + __field(u64, address_space) + __field(u64, flags) + ), + + TP_fast_assign( + __entry->processor_mask = processor_mask; + __entry->address_space = address_space; + __entry->flags = flags; + ), + + TP_printk("processor_mask 0x%llx address_space 0x%llx flags 0x%llx", + __entry->processor_mask, __entry->address_space, + __entry->flags) +); + +/* + * Tracepoint for kvm_hv_flush_tlb_ex. + */ +TRACE_EVENT(kvm_hv_flush_tlb_ex, + TP_PROTO(u64 valid_bank_mask, u64 format, u64 address_space, u64 flags), + TP_ARGS(valid_bank_mask, format, address_space, flags), + + TP_STRUCT__entry( + __field(u64, valid_bank_mask) + __field(u64, format) + __field(u64, address_space) + __field(u64, flags) + ), + + TP_fast_assign( + __entry->valid_bank_mask = valid_bank_mask; + __entry->format = format; + __entry->address_space = address_space; + __entry->flags = flags; + ), + + TP_printk("valid_bank_mask 0x%llx format 0x%llx " + "address_space 0x%llx flags 0x%llx", + __entry->valid_bank_mask, __entry->format, + __entry->address_space, __entry->flags) +); #endif /* _TRACE_KVM_H */ #undef TRACE_INCLUDE_PATH diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c index 40aa29204baf..559a12b6184d 100644 --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -242,7 +242,11 @@ struct shared_msr_entry { * underlying hardware which will be used to run L2. * This structure is packed to ensure that its layout is identical across * machines (necessary for live migration). - * If there are changes in this struct, VMCS12_REVISION must be changed. + * + * IMPORTANT: Changing the layout of existing fields in this structure + * will break save/restore compatibility with older kvm releases. When + * adding new fields, either use space in the reserved padding* arrays + * or add the new fields to the end of the structure. */ typedef u64 natural_width; struct __packed vmcs12 { @@ -265,17 +269,14 @@ struct __packed vmcs12 { u64 virtual_apic_page_addr; u64 apic_access_addr; u64 posted_intr_desc_addr; - u64 vm_function_control; u64 ept_pointer; u64 eoi_exit_bitmap0; u64 eoi_exit_bitmap1; u64 eoi_exit_bitmap2; u64 eoi_exit_bitmap3; - u64 eptp_list_address; u64 xss_exit_bitmap; u64 guest_physical_address; u64 vmcs_link_pointer; - u64 pml_address; u64 guest_ia32_debugctl; u64 guest_ia32_pat; u64 guest_ia32_efer; @@ -288,7 +289,12 @@ struct __packed vmcs12 { u64 host_ia32_pat; u64 host_ia32_efer; u64 host_ia32_perf_global_ctrl; - u64 padding64[8]; /* room for future expansion */ + u64 vmread_bitmap; + u64 vmwrite_bitmap; + u64 vm_function_control; + u64 eptp_list_address; + u64 pml_address; + u64 padding64[3]; /* room for future expansion */ /* * To allow migration of L1 (complete with its L2 guests) between * machines of different natural widths (32 or 64 bit), we cannot have @@ -397,7 +403,6 @@ struct __packed vmcs12 { u16 guest_ldtr_selector; u16 guest_tr_selector; u16 guest_intr_status; - u16 guest_pml_index; u16 host_es_selector; u16 host_cs_selector; u16 host_ss_selector; @@ -405,12 +410,172 @@ struct __packed vmcs12 { u16 host_fs_selector; u16 host_gs_selector; u16 host_tr_selector; + u16 guest_pml_index; }; +/* + * For save/restore compatibility, the vmcs12 field offsets must not change. + */ +#define CHECK_OFFSET(field, loc) \ + BUILD_BUG_ON_MSG(offsetof(struct vmcs12, field) != (loc), \ + "Offset of " #field " in struct vmcs12 has changed.") + +static inline void vmx_check_vmcs12_offsets(void) { + CHECK_OFFSET(revision_id, 0); + CHECK_OFFSET(abort, 4); + CHECK_OFFSET(launch_state, 8); + CHECK_OFFSET(io_bitmap_a, 40); + CHECK_OFFSET(io_bitmap_b, 48); + CHECK_OFFSET(msr_bitmap, 56); + CHECK_OFFSET(vm_exit_msr_store_addr, 64); + CHECK_OFFSET(vm_exit_msr_load_addr, 72); + CHECK_OFFSET(vm_entry_msr_load_addr, 80); + CHECK_OFFSET(tsc_offset, 88); + CHECK_OFFSET(virtual_apic_page_addr, 96); + CHECK_OFFSET(apic_access_addr, 104); + CHECK_OFFSET(posted_intr_desc_addr, 112); + CHECK_OFFSET(ept_pointer, 120); + CHECK_OFFSET(eoi_exit_bitmap0, 128); + CHECK_OFFSET(eoi_exit_bitmap1, 136); + CHECK_OFFSET(eoi_exit_bitmap2, 144); + CHECK_OFFSET(eoi_exit_bitmap3, 152); + CHECK_OFFSET(xss_exit_bitmap, 160); + CHECK_OFFSET(guest_physical_address, 168); + CHECK_OFFSET(vmcs_link_pointer, 176); + CHECK_OFFSET(guest_ia32_debugctl, 184); + CHECK_OFFSET(guest_ia32_pat, 192); + CHECK_OFFSET(guest_ia32_efer, 200); + CHECK_OFFSET(guest_ia32_perf_global_ctrl, 208); + CHECK_OFFSET(guest_pdptr0, 216); + CHECK_OFFSET(guest_pdptr1, 224); + CHECK_OFFSET(guest_pdptr2, 232); + CHECK_OFFSET(guest_pdptr3, 240); + CHECK_OFFSET(guest_bndcfgs, 248); + CHECK_OFFSET(host_ia32_pat, 256); + CHECK_OFFSET(host_ia32_efer, 264); + CHECK_OFFSET(host_ia32_perf_global_ctrl, 272); + CHECK_OFFSET(vmread_bitmap, 280); + CHECK_OFFSET(vmwrite_bitmap, 288); + CHECK_OFFSET(vm_function_control, 296); + CHECK_OFFSET(eptp_list_address, 304); + CHECK_OFFSET(pml_address, 312); + CHECK_OFFSET(cr0_guest_host_mask, 344); + CHECK_OFFSET(cr4_guest_host_mask, 352); + CHECK_OFFSET(cr0_read_shadow, 360); + CHECK_OFFSET(cr4_read_shadow, 368); + CHECK_OFFSET(cr3_target_value0, 376); + CHECK_OFFSET(cr3_target_value1, 384); + CHECK_OFFSET(cr3_target_value2, 392); + CHECK_OFFSET(cr3_target_value3, 400); + CHECK_OFFSET(exit_qualification, 408); + CHECK_OFFSET(guest_linear_address, 416); + CHECK_OFFSET(guest_cr0, 424); + CHECK_OFFSET(guest_cr3, 432); + CHECK_OFFSET(guest_cr4, 440); + CHECK_OFFSET(guest_es_base, 448); + CHECK_OFFSET(guest_cs_base, 456); + CHECK_OFFSET(guest_ss_base, 464); + CHECK_OFFSET(guest_ds_base, 472); + CHECK_OFFSET(guest_fs_base, 480); + CHECK_OFFSET(guest_gs_base, 488); + CHECK_OFFSET(guest_ldtr_base, 496); + CHECK_OFFSET(guest_tr_base, 504); + CHECK_OFFSET(guest_gdtr_base, 512); + CHECK_OFFSET(guest_idtr_base, 520); + CHECK_OFFSET(guest_dr7, 528); + CHECK_OFFSET(guest_rsp, 536); + CHECK_OFFSET(guest_rip, 544); + CHECK_OFFSET(guest_rflags, 552); + CHECK_OFFSET(guest_pending_dbg_exceptions, 560); + CHECK_OFFSET(guest_sysenter_esp, 568); + CHECK_OFFSET(guest_sysenter_eip, 576); + CHECK_OFFSET(host_cr0, 584); + CHECK_OFFSET(host_cr3, 592); + CHECK_OFFSET(host_cr4, 600); + CHECK_OFFSET(host_fs_base, 608); + CHECK_OFFSET(host_gs_base, 616); + CHECK_OFFSET(host_tr_base, 624); + CHECK_OFFSET(host_gdtr_base, 632); + CHECK_OFFSET(host_idtr_base, 640); + CHECK_OFFSET(host_ia32_sysenter_esp, 648); + CHECK_OFFSET(host_ia32_sysenter_eip, 656); + CHECK_OFFSET(host_rsp, 664); + CHECK_OFFSET(host_rip, 672); + CHECK_OFFSET(pin_based_vm_exec_control, 744); + CHECK_OFFSET(cpu_based_vm_exec_control, 748); + CHECK_OFFSET(exception_bitmap, 752); + CHECK_OFFSET(page_fault_error_code_mask, 756); + CHECK_OFFSET(page_fault_error_code_match, 760); + CHECK_OFFSET(cr3_target_count, 764); + CHECK_OFFSET(vm_exit_controls, 768); + CHECK_OFFSET(vm_exit_msr_store_count, 772); + CHECK_OFFSET(vm_exit_msr_load_count, 776); + CHECK_OFFSET(vm_entry_controls, 780); + CHECK_OFFSET(vm_entry_msr_load_count, 784); + CHECK_OFFSET(vm_entry_intr_info_field, 788); + CHECK_OFFSET(vm_entry_exception_error_code, 792); + CHECK_OFFSET(vm_entry_instruction_len, 796); + CHECK_OFFSET(tpr_threshold, 800); + CHECK_OFFSET(secondary_vm_exec_control, 804); + CHECK_OFFSET(vm_instruction_error, 808); + CHECK_OFFSET(vm_exit_reason, 812); + CHECK_OFFSET(vm_exit_intr_info, 816); + CHECK_OFFSET(vm_exit_intr_error_code, 820); + CHECK_OFFSET(idt_vectoring_info_field, 824); + CHECK_OFFSET(idt_vectoring_error_code, 828); + CHECK_OFFSET(vm_exit_instruction_len, 832); + CHECK_OFFSET(vmx_instruction_info, 836); + CHECK_OFFSET(guest_es_limit, 840); + CHECK_OFFSET(guest_cs_limit, 844); + CHECK_OFFSET(guest_ss_limit, 848); + CHECK_OFFSET(guest_ds_limit, 852); + CHECK_OFFSET(guest_fs_limit, 856); + CHECK_OFFSET(guest_gs_limit, 860); + CHECK_OFFSET(guest_ldtr_limit, 864); + CHECK_OFFSET(guest_tr_limit, 868); + CHECK_OFFSET(guest_gdtr_limit, 872); + CHECK_OFFSET(guest_idtr_limit, 876); + CHECK_OFFSET(guest_es_ar_bytes, 880); + CHECK_OFFSET(guest_cs_ar_bytes, 884); + CHECK_OFFSET(guest_ss_ar_bytes, 888); + CHECK_OFFSET(guest_ds_ar_bytes, 892); + CHECK_OFFSET(guest_fs_ar_bytes, 896); + CHECK_OFFSET(guest_gs_ar_bytes, 900); + CHECK_OFFSET(guest_ldtr_ar_bytes, 904); + CHECK_OFFSET(guest_tr_ar_bytes, 908); + CHECK_OFFSET(guest_interruptibility_info, 912); + CHECK_OFFSET(guest_activity_state, 916); + CHECK_OFFSET(guest_sysenter_cs, 920); + CHECK_OFFSET(host_ia32_sysenter_cs, 924); + CHECK_OFFSET(vmx_preemption_timer_value, 928); + CHECK_OFFSET(virtual_processor_id, 960); + CHECK_OFFSET(posted_intr_nv, 962); + CHECK_OFFSET(guest_es_selector, 964); + CHECK_OFFSET(guest_cs_selector, 966); + CHECK_OFFSET(guest_ss_selector, 968); + CHECK_OFFSET(guest_ds_selector, 970); + CHECK_OFFSET(guest_fs_selector, 972); + CHECK_OFFSET(guest_gs_selector, 974); + CHECK_OFFSET(guest_ldtr_selector, 976); + CHECK_OFFSET(guest_tr_selector, 978); + CHECK_OFFSET(guest_intr_status, 980); + CHECK_OFFSET(host_es_selector, 982); + CHECK_OFFSET(host_cs_selector, 984); + CHECK_OFFSET(host_ss_selector, 986); + CHECK_OFFSET(host_ds_selector, 988); + CHECK_OFFSET(host_fs_selector, 990); + CHECK_OFFSET(host_gs_selector, 992); + CHECK_OFFSET(host_tr_selector, 994); + CHECK_OFFSET(guest_pml_index, 996); +} + /* * VMCS12_REVISION is an arbitrary id that should be changed if the content or * layout of struct vmcs12 is changed. MSR_IA32_VMX_BASIC returns this id, and * VMPTRLD verifies that the VMCS region that L1 is loading contains this id. + * + * IMPORTANT: Changing this value will break save/restore compatibility with + * older kvm releases. */ #define VMCS12_REVISION 0x11e57ed0 @@ -481,7 +646,8 @@ struct nested_vmx { bool sync_shadow_vmcs; bool dirty_vmcs12; - bool change_vmcs01_virtual_x2apic_mode; + bool change_vmcs01_virtual_apic_mode; + /* L2 must run next, and mustn't decide to exit to L1. */ bool nested_run_pending; @@ -761,6 +927,7 @@ static const unsigned short vmcs_field_to_offset_table[] = { FIELD64(VM_EXIT_MSR_STORE_ADDR, vm_exit_msr_store_addr), FIELD64(VM_EXIT_MSR_LOAD_ADDR, vm_exit_msr_load_addr), FIELD64(VM_ENTRY_MSR_LOAD_ADDR, vm_entry_msr_load_addr), + FIELD64(PML_ADDRESS, pml_address), FIELD64(TSC_OFFSET, tsc_offset), FIELD64(VIRTUAL_APIC_PAGE_ADDR, virtual_apic_page_addr), FIELD64(APIC_ACCESS_ADDR, apic_access_addr), @@ -772,10 +939,11 @@ static const unsigned short vmcs_field_to_offset_table[] = { FIELD64(EOI_EXIT_BITMAP2, eoi_exit_bitmap2), FIELD64(EOI_EXIT_BITMAP3, eoi_exit_bitmap3), FIELD64(EPTP_LIST_ADDRESS, eptp_list_address), + FIELD64(VMREAD_BITMAP, vmread_bitmap), + FIELD64(VMWRITE_BITMAP, vmwrite_bitmap), FIELD64(XSS_EXIT_BITMAP, xss_exit_bitmap), FIELD64(GUEST_PHYSICAL_ADDRESS, guest_physical_address), FIELD64(VMCS_LINK_POINTER, vmcs_link_pointer), - FIELD64(PML_ADDRESS, pml_address), FIELD64(GUEST_IA32_DEBUGCTL, guest_ia32_debugctl), FIELD64(GUEST_IA32_PAT, guest_ia32_pat), FIELD64(GUEST_IA32_EFER, guest_ia32_efer), @@ -1089,6 +1257,16 @@ static inline u16 evmcs_read16(unsigned long field) return *(u16 *)((char *)current_evmcs + offset); } +static inline void evmcs_touch_msr_bitmap(void) +{ + if (unlikely(!current_evmcs)) + return; + + if (current_evmcs->hv_enlightenments_control.msr_bitmap) + current_evmcs->hv_clean_fields &= + ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_MSR_BITMAP; +} + static void evmcs_load(u64 phys_addr) { struct hv_vp_assist_page *vp_ap = @@ -1173,6 +1351,7 @@ static inline u32 evmcs_read32(unsigned long field) { return 0; } static inline u16 evmcs_read16(unsigned long field) { return 0; } static inline void evmcs_load(u64 phys_addr) {} static inline void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf) {} +static inline void evmcs_touch_msr_bitmap(void) {} #endif /* IS_ENABLED(CONFIG_HYPERV) */ static inline bool is_exception_n(u32 intr_info, u8 vector) @@ -1393,6 +1572,11 @@ static inline bool cpu_has_vmx_invept_global(void) return vmx_capability.ept & VMX_EPT_EXTENT_GLOBAL_BIT; } +static inline bool cpu_has_vmx_invvpid_individual_addr(void) +{ + return vmx_capability.vpid & VMX_VPID_EXTENT_INDIVIDUAL_ADDR_BIT; +} + static inline bool cpu_has_vmx_invvpid_single(void) { return vmx_capability.vpid & VMX_VPID_EXTENT_SINGLE_CONTEXT_BIT; @@ -1510,6 +1694,17 @@ static inline unsigned nested_cpu_vmx_misc_cr3_count(struct kvm_vcpu *vcpu) return vmx_misc_cr3_count(to_vmx(vcpu)->nested.msrs.misc_low); } +/* + * Do the virtual VMX capability MSRs specify that L1 can use VMWRITE + * to modify any valid field of the VMCS, or are the VM-exit + * information fields read-only? + */ +static inline bool nested_cpu_has_vmwrite_any_field(struct kvm_vcpu *vcpu) +{ + return to_vmx(vcpu)->nested.msrs.misc_low & + MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS; +} + static inline bool nested_cpu_has(struct vmcs12 *vmcs12, u32 bit) { return vmcs12->cpu_based_vm_exec_control & bit; @@ -3127,6 +3322,7 @@ static void nested_vmx_setup_ctls_msrs(struct nested_vmx_msrs *msrs, bool apicv) msrs->misc_high); msrs->misc_low &= VMX_MISC_SAVE_EFER_LMA; msrs->misc_low |= + MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS | VMX_MISC_EMULATED_PREEMPTION_TIMER_RATE | VMX_MISC_ACTIVITY_HLT; msrs->misc_high = 0; @@ -3300,6 +3496,15 @@ static int vmx_restore_vmx_misc(struct vcpu_vmx *vmx, u64 data) vmx->nested.msrs.misc_low = data; vmx->nested.msrs.misc_high = data >> 32; + + /* + * If L1 has read-only VM-exit information fields, use the + * less permissive vmx_vmwrite_bitmap to specify write + * permissions for the shadow VMCS. + */ + if (enable_shadow_vmcs && !nested_cpu_has_vmwrite_any_field(&vmx->vcpu)) + vmcs_write64(VMWRITE_BITMAP, __pa(vmx_vmwrite_bitmap)); + return 0; } @@ -3354,6 +3559,13 @@ static int vmx_set_vmx_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 data) { struct vcpu_vmx *vmx = to_vmx(vcpu); + /* + * Don't allow changes to the VMX capability MSRs while the vCPU + * is in VMX operation. + */ + if (vmx->nested.vmxon) + return -EBUSY; + switch (msr_index) { case MSR_IA32_VMX_BASIC: return vmx_restore_vmx_basic(vmx, data); @@ -4216,6 +4428,15 @@ static int alloc_loaded_vmcs(struct loaded_vmcs *loaded_vmcs) if (!loaded_vmcs->msr_bitmap) goto out_vmcs; memset(loaded_vmcs->msr_bitmap, 0xff, PAGE_SIZE); + + if (IS_ENABLED(CONFIG_HYPERV) && + static_branch_unlikely(&enable_evmcs) && + (ms_hyperv.nested_features & HV_X64_NESTED_MSR_BITMAP)) { + struct hv_enlightened_vmcs *evmcs = + (struct hv_enlightened_vmcs *)loaded_vmcs->vmcs; + + evmcs->hv_enlightenments_control.msr_bitmap = 1; + } } return 0; @@ -5329,6 +5550,9 @@ static void __always_inline vmx_disable_intercept_for_msr(unsigned long *msr_bit if (!cpu_has_vmx_msr_bitmap()) return; + if (static_branch_unlikely(&enable_evmcs)) + evmcs_touch_msr_bitmap(); + /* * See Intel PRM Vol. 3, 20.6.9 (MSR-Bitmap Address). Early manuals * have the write-low and read-high bitmap offsets the wrong way round. @@ -5364,6 +5588,9 @@ static void __always_inline vmx_enable_intercept_for_msr(unsigned long *msr_bitm if (!cpu_has_vmx_msr_bitmap()) return; + if (static_branch_unlikely(&enable_evmcs)) + evmcs_touch_msr_bitmap(); + /* * See Intel PRM Vol. 3, 20.6.9 (MSR-Bitmap Address). Early manuals * have the write-low and read-high bitmap offsets the wrong way round. @@ -5946,8 +6173,14 @@ static void vmx_vcpu_setup(struct vcpu_vmx *vmx) int i; if (enable_shadow_vmcs) { + /* + * At vCPU creation, "VMWRITE to any supported field + * in the VMCS" is supported, so use the more + * permissive vmx_vmread_bitmap to specify both read + * and write permissions for the shadow VMCS. + */ vmcs_write64(VMREAD_BITMAP, __pa(vmx_vmread_bitmap)); - vmcs_write64(VMWRITE_BITMAP, __pa(vmx_vmwrite_bitmap)); + vmcs_write64(VMWRITE_BITMAP, __pa(vmx_vmread_bitmap)); } if (cpu_has_vmx_msr_bitmap()) vmcs_write64(MSR_BITMAP, __pa(vmx->vmcs01.msr_bitmap)); @@ -7588,8 +7821,7 @@ static int nested_vmx_get_vmptr(struct kvm_vcpu *vcpu, gpa_t *vmpointer) vmcs_read32(VMX_INSTRUCTION_INFO), false, &gva)) return 1; - if (kvm_read_guest_virt(&vcpu->arch.emulate_ctxt, gva, vmpointer, - sizeof(*vmpointer), &e)) { + if (kvm_read_guest_virt(vcpu, gva, vmpointer, sizeof(*vmpointer), &e)) { kvm_inject_page_fault(vcpu, &e); return 1; } @@ -7670,6 +7902,12 @@ static int handle_vmon(struct kvm_vcpu *vcpu) return 1; } + /* CPL=0 must be checked manually. */ + if (vmx_get_cpl(vcpu)) { + kvm_queue_exception(vcpu, UD_VECTOR); + return 1; + } + if (vmx->nested.vmxon) { nested_vmx_failValid(vcpu, VMXERR_VMXON_IN_VMX_ROOT_OPERATION); return kvm_skip_emulated_instruction(vcpu); @@ -7729,6 +7967,11 @@ static int handle_vmon(struct kvm_vcpu *vcpu) */ static int nested_vmx_check_permission(struct kvm_vcpu *vcpu) { + if (vmx_get_cpl(vcpu)) { + kvm_queue_exception(vcpu, UD_VECTOR); + return 0; + } + if (!to_vmx(vcpu)->nested.vmxon) { kvm_queue_exception(vcpu, UD_VECTOR); return 0; @@ -7928,23 +8171,42 @@ static inline int vmcs12_write_any(struct kvm_vcpu *vcpu, } +/* + * Copy the writable VMCS shadow fields back to the VMCS12, in case + * they have been modified by the L1 guest. Note that the "read-only" + * VM-exit information fields are actually writable if the vCPU is + * configured to support "VMWRITE to any supported field in the VMCS." + */ static void copy_shadow_to_vmcs12(struct vcpu_vmx *vmx) { - int i; + const u16 *fields[] = { + shadow_read_write_fields, + shadow_read_only_fields + }; + const int max_fields[] = { + max_shadow_read_write_fields, + max_shadow_read_only_fields + }; + int i, q; unsigned long field; u64 field_value; struct vmcs *shadow_vmcs = vmx->vmcs01.shadow_vmcs; - const u16 *fields = shadow_read_write_fields; - const int num_fields = max_shadow_read_write_fields; preempt_disable(); vmcs_load(shadow_vmcs); - for (i = 0; i < num_fields; i++) { - field = fields[i]; - field_value = __vmcs_readl(field); - vmcs12_write_any(&vmx->vcpu, field, field_value); + for (q = 0; q < ARRAY_SIZE(fields); q++) { + for (i = 0; i < max_fields[q]; i++) { + field = fields[q][i]; + field_value = __vmcs_readl(field); + vmcs12_write_any(&vmx->vcpu, field, field_value); + } + /* + * Skip the VM-exit information fields if they are read-only. + */ + if (!nested_cpu_has_vmwrite_any_field(&vmx->vcpu)) + break; } vmcs_clear(shadow_vmcs); @@ -8029,9 +8291,9 @@ static int handle_vmread(struct kvm_vcpu *vcpu) if (get_vmx_mem_address(vcpu, exit_qualification, vmx_instruction_info, true, &gva)) return 1; - /* _system ok, as hardware has verified cpl=0 */ - kvm_write_guest_virt_system(&vcpu->arch.emulate_ctxt, gva, - &field_value, (is_long_mode(vcpu) ? 8 : 4), NULL); + /* _system ok, nested_vmx_check_permission has verified cpl=0 */ + kvm_write_guest_virt_system(vcpu, gva, &field_value, + (is_long_mode(vcpu) ? 8 : 4), NULL); } nested_vmx_succeed(vcpu); @@ -8069,8 +8331,8 @@ static int handle_vmwrite(struct kvm_vcpu *vcpu) if (get_vmx_mem_address(vcpu, exit_qualification, vmx_instruction_info, false, &gva)) return 1; - if (kvm_read_guest_virt(&vcpu->arch.emulate_ctxt, gva, - &field_value, (is_64_bit_mode(vcpu) ? 8 : 4), &e)) { + if (kvm_read_guest_virt(vcpu, gva, &field_value, + (is_64_bit_mode(vcpu) ? 8 : 4), &e)) { kvm_inject_page_fault(vcpu, &e); return 1; } @@ -8078,7 +8340,12 @@ static int handle_vmwrite(struct kvm_vcpu *vcpu) field = kvm_register_readl(vcpu, (((vmx_instruction_info) >> 28) & 0xf)); - if (vmcs_field_readonly(field)) { + /* + * If the vCPU supports "VMWRITE to any supported field in the + * VMCS," then the "read-only" fields are actually read/write. + */ + if (vmcs_field_readonly(field) && + !nested_cpu_has_vmwrite_any_field(vcpu)) { nested_vmx_failValid(vcpu, VMXERR_VMWRITE_READ_ONLY_VMCS_COMPONENT); return kvm_skip_emulated_instruction(vcpu); @@ -8189,10 +8456,10 @@ static int handle_vmptrst(struct kvm_vcpu *vcpu) if (get_vmx_mem_address(vcpu, exit_qualification, vmx_instruction_info, true, &vmcs_gva)) return 1; - /* ok to use *_system, as hardware has verified cpl=0 */ - if (kvm_write_guest_virt_system(&vcpu->arch.emulate_ctxt, vmcs_gva, - (void *)&to_vmx(vcpu)->nested.current_vmptr, - sizeof(u64), &e)) { + /* *_system ok, nested_vmx_check_permission has verified cpl=0 */ + if (kvm_write_guest_virt_system(vcpu, vmcs_gva, + (void *)&to_vmx(vcpu)->nested.current_vmptr, + sizeof(u64), &e)) { kvm_inject_page_fault(vcpu, &e); return 1; } @@ -8239,8 +8506,7 @@ static int handle_invept(struct kvm_vcpu *vcpu) if (get_vmx_mem_address(vcpu, vmcs_readl(EXIT_QUALIFICATION), vmx_instruction_info, false, &gva)) return 1; - if (kvm_read_guest_virt(&vcpu->arch.emulate_ctxt, gva, &operand, - sizeof(operand), &e)) { + if (kvm_read_guest_virt(vcpu, gva, &operand, sizeof(operand), &e)) { kvm_inject_page_fault(vcpu, &e); return 1; } @@ -8304,8 +8570,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) if (get_vmx_mem_address(vcpu, vmcs_readl(EXIT_QUALIFICATION), vmx_instruction_info, false, &gva)) return 1; - if (kvm_read_guest_virt(&vcpu->arch.emulate_ctxt, gva, &operand, - sizeof(operand), &e)) { + if (kvm_read_guest_virt(vcpu, gva, &operand, sizeof(operand), &e)) { kvm_inject_page_fault(vcpu, &e); return 1; } @@ -8317,12 +8582,19 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) switch (type) { case VMX_VPID_EXTENT_INDIVIDUAL_ADDR: - if (is_noncanonical_address(operand.gla, vcpu)) { + if (!operand.vpid || + is_noncanonical_address(operand.gla, vcpu)) { nested_vmx_failValid(vcpu, VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); return kvm_skip_emulated_instruction(vcpu); } - /* fall through */ + if (cpu_has_vmx_invvpid_individual_addr() && + vmx->nested.vpid02) { + __invvpid(VMX_VPID_EXTENT_INDIVIDUAL_ADDR, + vmx->nested.vpid02, operand.gla); + } else + __vmx_flush_tlb(vcpu, vmx->nested.vpid02, true); + break; case VMX_VPID_EXTENT_SINGLE_CONTEXT: case VMX_VPID_EXTENT_SINGLE_NON_GLOBAL: if (!operand.vpid) { @@ -8330,15 +8602,16 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); return kvm_skip_emulated_instruction(vcpu); } + __vmx_flush_tlb(vcpu, vmx->nested.vpid02, true); break; case VMX_VPID_EXTENT_ALL_CONTEXT: + __vmx_flush_tlb(vcpu, vmx->nested.vpid02, true); break; default: WARN_ON_ONCE(1); return kvm_skip_emulated_instruction(vcpu); } - __vmx_flush_tlb(vcpu, vmx->nested.vpid02, true); nested_vmx_succeed(vcpu); return kvm_skip_emulated_instruction(vcpu); @@ -8842,11 +9115,13 @@ static bool nested_vmx_exit_reflected(struct kvm_vcpu *vcpu, u32 exit_reason) case EXIT_REASON_TPR_BELOW_THRESHOLD: return nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW); case EXIT_REASON_APIC_ACCESS: - return nested_cpu_has2(vmcs12, - SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES); case EXIT_REASON_APIC_WRITE: case EXIT_REASON_EOI_INDUCED: - /* apic_write and eoi_induced should exit unconditionally. */ + /* + * The controls for "virtualize APIC accesses," "APIC- + * register virtualization," and "virtual-interrupt + * delivery" only come from vmcs12. + */ return true; case EXIT_REASON_EPT_VIOLATION: /* @@ -9253,31 +9528,43 @@ static void update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr) vmcs_write32(TPR_THRESHOLD, irr); } -static void vmx_set_virtual_x2apic_mode(struct kvm_vcpu *vcpu, bool set) +static void vmx_set_virtual_apic_mode(struct kvm_vcpu *vcpu) { u32 sec_exec_control; + if (!lapic_in_kernel(vcpu)) + return; + /* Postpone execution until vmcs01 is the current VMCS. */ if (is_guest_mode(vcpu)) { - to_vmx(vcpu)->nested.change_vmcs01_virtual_x2apic_mode = true; + to_vmx(vcpu)->nested.change_vmcs01_virtual_apic_mode = true; return; } - if (!cpu_has_vmx_virtualize_x2apic_mode()) - return; - if (!cpu_need_tpr_shadow(vcpu)) return; sec_exec_control = vmcs_read32(SECONDARY_VM_EXEC_CONTROL); + sec_exec_control &= ~(SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES | + SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE); - if (set) { - sec_exec_control &= ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES; - sec_exec_control |= SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE; - } else { - sec_exec_control &= ~SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE; - sec_exec_control |= SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES; - vmx_flush_tlb(vcpu, true); + switch (kvm_get_apic_mode(vcpu)) { + case LAPIC_MODE_INVALID: + WARN_ONCE(true, "Invalid local APIC state"); + case LAPIC_MODE_DISABLED: + break; + case LAPIC_MODE_XAPIC: + if (flexpriority_enabled) { + sec_exec_control |= + SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES; + vmx_flush_tlb(vcpu, true); + } + break; + case LAPIC_MODE_X2APIC: + if (cpu_has_vmx_virtualize_x2apic_mode()) + sec_exec_control |= + SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE; + break; } vmcs_write32(SECONDARY_VM_EXEC_CONTROL, sec_exec_control); @@ -9286,24 +9573,7 @@ static void vmx_set_virtual_x2apic_mode(struct kvm_vcpu *vcpu, bool set) static void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu, hpa_t hpa) { - struct vcpu_vmx *vmx = to_vmx(vcpu); - - /* - * Currently we do not handle the nested case where L2 has an - * APIC access page of its own; that page is still pinned. - * Hence, we skip the case where the VCPU is in guest mode _and_ - * L1 prepared an APIC access page for L2. - * - * For the case where L1 and L2 share the same APIC access page - * (flexpriority=Y but SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES clear - * in the vmcs12), this function will only update either the vmcs01 - * or the vmcs02. If the former, the vmcs02 will be updated by - * prepare_vmcs02. If the latter, the vmcs01 will be updated in - * the next L2->L1 exit. - */ - if (!is_guest_mode(vcpu) || - !nested_cpu_has2(get_vmcs12(&vmx->vcpu), - SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) { + if (!is_guest_mode(vcpu)) { vmcs_write64(APIC_ACCESS_ADDR, hpa); vmx_flush_tlb(vcpu, true); } @@ -9943,13 +10213,13 @@ STACK_FRAME_NON_STANDARD(vmx_vcpu_run); static struct kvm *vmx_vm_alloc(void) { - struct kvm_vmx *kvm_vmx = kzalloc(sizeof(struct kvm_vmx), GFP_KERNEL); + struct kvm_vmx *kvm_vmx = vzalloc(sizeof(struct kvm_vmx)); return &kvm_vmx->kvm; } static void vmx_vm_free(struct kvm *kvm) { - kfree(to_kvm_vmx(kvm)); + vfree(to_kvm_vmx(kvm)); } static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs) @@ -10387,11 +10657,6 @@ static void nested_get_vmcs12_pages(struct kvm_vcpu *vcpu, vmcs_clear_bits(SECONDARY_VM_EXEC_CONTROL, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES); } - } else if (!(nested_cpu_has_virt_x2apic_mode(vmcs12)) && - cpu_need_virtualize_apic_accesses(&vmx->vcpu)) { - vmcs_set_bits(SECONDARY_VM_EXEC_CONTROL, - SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES); - kvm_vcpu_reload_apic_access_page(vcpu); } if (nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW)) { @@ -10871,8 +11136,7 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool ne return 0; } -static void prepare_vmcs02_full(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, - bool from_vmentry) +static void prepare_vmcs02_full(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) { struct vcpu_vmx *vmx = to_vmx(vcpu); @@ -11006,13 +11270,13 @@ static void prepare_vmcs02_full(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, * is assigned to entry_failure_code on failure. */ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, - bool from_vmentry, u32 *entry_failure_code) + u32 *entry_failure_code) { struct vcpu_vmx *vmx = to_vmx(vcpu); u32 exec_control, vmcs12_exec_ctrl; if (vmx->nested.dirty_vmcs12) { - prepare_vmcs02_full(vcpu, vmcs12, from_vmentry); + prepare_vmcs02_full(vcpu, vmcs12); vmx->nested.dirty_vmcs12 = false; } @@ -11032,7 +11296,7 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, * HOST_FS_BASE, HOST_GS_BASE. */ - if (from_vmentry && + if (vmx->nested.nested_run_pending && (vmcs12->vm_entry_controls & VM_ENTRY_LOAD_DEBUG_CONTROLS)) { kvm_set_dr(vcpu, 7, vmcs12->guest_dr7); vmcs_write64(GUEST_IA32_DEBUGCTL, vmcs12->guest_ia32_debugctl); @@ -11040,7 +11304,7 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, kvm_set_dr(vcpu, 7, vcpu->arch.dr7); vmcs_write64(GUEST_IA32_DEBUGCTL, vmx->nested.vmcs01_debugctl); } - if (from_vmentry) { + if (vmx->nested.nested_run_pending) { vmcs_write32(VM_ENTRY_INTR_INFO_FIELD, vmcs12->vm_entry_intr_info_field); vmcs_write32(VM_ENTRY_EXCEPTION_ERROR_CODE, @@ -11172,7 +11436,7 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, ~VM_ENTRY_IA32E_MODE) | (vmcs_config.vmentry_ctrl & ~VM_ENTRY_IA32E_MODE)); - if (from_vmentry && + if (vmx->nested.nested_run_pending && (vmcs12->vm_entry_controls & VM_ENTRY_LOAD_IA32_PAT)) { vmcs_write64(GUEST_IA32_PAT, vmcs12->guest_ia32_pat); vcpu->arch.pat = vmcs12->guest_ia32_pat; @@ -11197,7 +11461,7 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, if (nested_cpu_has_vpid(vmcs12) && vmx->nested.vpid02) { if (vmcs12->virtual_processor_id != vmx->nested.last_vpid) { vmx->nested.last_vpid = vmcs12->virtual_processor_id; - __vmx_flush_tlb(vcpu, to_vmx(vcpu)->nested.vpid02, true); + __vmx_flush_tlb(vcpu, vmx->nested.vpid02, true); } } else { vmx_flush_tlb(vcpu, true); @@ -11240,7 +11504,7 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, vmx_set_cr4(vcpu, vmcs12->guest_cr4); vmcs_writel(CR4_READ_SHADOW, nested_read_cr4(vmcs12)); - if (from_vmentry && + if (vmx->nested.nested_run_pending && (vmcs12->vm_entry_controls & VM_ENTRY_LOAD_IA32_EFER)) vcpu->arch.efer = vmcs12->guest_ia32_efer; else if (vmcs12->vm_entry_controls & VM_ENTRY_IA32E_MODE) @@ -11418,7 +11682,7 @@ static int check_vmentry_postreqs(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, return 0; } -static int enter_vmx_non_root_mode(struct kvm_vcpu *vcpu, bool from_vmentry) +static int enter_vmx_non_root_mode(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx = to_vmx(vcpu); struct vmcs12 *vmcs12 = get_vmcs12(vcpu); @@ -11438,7 +11702,7 @@ static int enter_vmx_non_root_mode(struct kvm_vcpu *vcpu, bool from_vmentry) vcpu->arch.tsc_offset += vmcs12->tsc_offset; r = EXIT_REASON_INVALID_STATE; - if (prepare_vmcs02(vcpu, vmcs12, from_vmentry, &exit_qual)) + if (prepare_vmcs02(vcpu, vmcs12, &exit_qual)) goto fail; nested_get_vmcs12_pages(vcpu, vmcs12); @@ -11540,20 +11804,22 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch) * the nested entry. */ - ret = enter_vmx_non_root_mode(vcpu, true); - if (ret) + vmx->nested.nested_run_pending = 1; + ret = enter_vmx_non_root_mode(vcpu); + if (ret) { + vmx->nested.nested_run_pending = 0; return ret; + } /* * If we're entering a halted L2 vcpu and the L2 vcpu won't be woken * by event injection, halt vcpu. */ if ((vmcs12->guest_activity_state == GUEST_ACTIVITY_HLT) && - !(vmcs12->vm_entry_intr_info_field & INTR_INFO_VALID_MASK)) + !(vmcs12->vm_entry_intr_info_field & INTR_INFO_VALID_MASK)) { + vmx->nested.nested_run_pending = 0; return kvm_vcpu_halt(vcpu); - - vmx->nested.nested_run_pending = 1; - + } return 1; out: @@ -11925,12 +12191,20 @@ static void load_vmcs12_host_state(struct kvm_vcpu *vcpu, load_vmcs12_mmu_host_state(vcpu, vmcs12); - if (enable_vpid) { - /* - * Trivially support vpid by letting L2s share their parent - * L1's vpid. TODO: move to a more elaborate solution, giving - * each L2 its own vpid and exposing the vpid feature to L1. - */ + /* + * If vmcs01 don't use VPID, CPU flushes TLB on every + * VMEntry/VMExit. Thus, no need to flush TLB. + * + * If vmcs12 uses VPID, TLB entries populated by L2 are + * tagged with vmx->nested.vpid02 while L1 entries are tagged + * with vmx->vpid. Thus, no need to flush TLB. + * + * Therefore, flush TLB only in case vmcs01 uses VPID and + * vmcs12 don't use VPID as in this case L1 & L2 TLB entries + * are both tagged with vmx->vpid. + */ + if (enable_vpid && + !(nested_cpu_has_vpid(vmcs12) && to_vmx(vcpu)->nested.vpid02)) { vmx_flush_tlb(vcpu, true); } @@ -12069,10 +12343,9 @@ static void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 exit_reason, if (kvm_has_tsc_control) decache_tsc_multiplier(vmx); - if (vmx->nested.change_vmcs01_virtual_x2apic_mode) { - vmx->nested.change_vmcs01_virtual_x2apic_mode = false; - vmx_set_virtual_x2apic_mode(vcpu, - vcpu->arch.apic_base & X2APIC_ENABLE); + if (vmx->nested.change_vmcs01_virtual_apic_mode) { + vmx->nested.change_vmcs01_virtual_apic_mode = false; + vmx_set_virtual_apic_mode(vcpu); } else if (!nested_cpu_has_ept(vmcs12) && nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) { @@ -12236,7 +12509,7 @@ static inline int u64_shl_div_u64(u64 a, unsigned int shift, static int vmx_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc) { struct vcpu_vmx *vmx; - u64 tscl, guest_tscl, delta_tsc; + u64 tscl, guest_tscl, delta_tsc, lapic_timer_advance_cycles; if (kvm_mwait_in_guest(vcpu->kvm)) return -EOPNOTSUPP; @@ -12245,6 +12518,12 @@ static int vmx_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc) tscl = rdtsc(); guest_tscl = kvm_read_l1_tsc(vcpu, tscl); delta_tsc = max(guest_deadline_tsc, guest_tscl) - guest_tscl; + lapic_timer_advance_cycles = nsec_to_cycles(vcpu, lapic_timer_advance_ns); + + if (delta_tsc > lapic_timer_advance_cycles) + delta_tsc -= lapic_timer_advance_cycles; + else + delta_tsc = 0; /* Convert to host delta tsc if tsc scaling is enabled */ if (vcpu->arch.tsc_scaling_ratio != kvm_default_tsc_scaling_ratio && @@ -12615,7 +12894,7 @@ static int vmx_pre_leave_smm(struct kvm_vcpu *vcpu, u64 smbase) if (vmx->nested.smm.guest_mode) { vcpu->arch.hflags &= ~HF_SMM_MASK; - ret = enter_vmx_non_root_mode(vcpu, false); + ret = enter_vmx_non_root_mode(vcpu); vcpu->arch.hflags |= HF_SMM_MASK; if (ret) return ret; @@ -12700,7 +12979,7 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = { .enable_nmi_window = enable_nmi_window, .enable_irq_window = enable_irq_window, .update_cr8_intercept = update_cr8_intercept, - .set_virtual_x2apic_mode = vmx_set_virtual_x2apic_mode, + .set_virtual_apic_mode = vmx_set_virtual_apic_mode, .set_apic_access_page_addr = vmx_set_apic_access_page_addr, .get_enable_apicv = vmx_get_enable_apicv, .refresh_apicv_exec_ctrl = vmx_refresh_apicv_exec_ctrl, @@ -12812,6 +13091,7 @@ static int __init vmx_init(void) rcu_assign_pointer(crash_vmclear_loaded_vmcss, crash_vmclear_local_loaded_vmcss); #endif + vmx_check_vmcs12_offsets(); return 0; } diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 71e7cda6d014..0046aa70205a 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -138,6 +138,7 @@ module_param(tsc_tolerance_ppm, uint, S_IRUGO | S_IWUSR); /* lapic timer advance (tscdeadline mode only) in nanoseconds */ unsigned int __read_mostly lapic_timer_advance_ns = 0; module_param(lapic_timer_advance_ns, uint, S_IRUGO | S_IWUSR); +EXPORT_SYMBOL_GPL(lapic_timer_advance_ns); static bool __read_mostly vector_hashing = true; module_param(vector_hashing, bool, S_IRUGO); @@ -318,23 +319,27 @@ u64 kvm_get_apic_base(struct kvm_vcpu *vcpu) } EXPORT_SYMBOL_GPL(kvm_get_apic_base); +enum lapic_mode kvm_get_apic_mode(struct kvm_vcpu *vcpu) +{ + return kvm_apic_mode(kvm_get_apic_base(vcpu)); +} +EXPORT_SYMBOL_GPL(kvm_get_apic_mode); + int kvm_set_apic_base(struct kvm_vcpu *vcpu, struct msr_data *msr_info) { - u64 old_state = vcpu->arch.apic_base & - (MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE); - u64 new_state = msr_info->data & - (MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE); + enum lapic_mode old_mode = kvm_get_apic_mode(vcpu); + enum lapic_mode new_mode = kvm_apic_mode(msr_info->data); u64 reserved_bits = ((~0ULL) << cpuid_maxphyaddr(vcpu)) | 0x2ff | (guest_cpuid_has(vcpu, X86_FEATURE_X2APIC) ? 0 : X2APIC_ENABLE); - if ((msr_info->data & reserved_bits) || new_state == X2APIC_ENABLE) - return 1; - if (!msr_info->host_initiated && - ((new_state == MSR_IA32_APICBASE_ENABLE && - old_state == (MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE)) || - (new_state == (MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE) && - old_state == 0))) + if ((msr_info->data & reserved_bits) != 0 || new_mode == LAPIC_MODE_INVALID) return 1; + if (!msr_info->host_initiated) { + if (old_mode == LAPIC_MODE_X2APIC && new_mode == LAPIC_MODE_XAPIC) + return 1; + if (old_mode == LAPIC_MODE_DISABLED && new_mode == LAPIC_MODE_X2APIC) + return 1; + } kvm_lapic_set_base(vcpu, msr_info->data); return 0; @@ -856,7 +861,7 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) } if (is_long_mode(vcpu) && - (cr3 & rsvd_bits(cpuid_maxphyaddr(vcpu), 62))) + (cr3 & rsvd_bits(cpuid_maxphyaddr(vcpu), 63))) return 1; else if (is_pae(vcpu) && is_paging(vcpu) && !load_pdptrs(vcpu, vcpu->arch.walk_mmu, cr3)) @@ -1761,7 +1766,7 @@ static int do_monotonic_boot(s64 *t, u64 *tsc_timestamp) return mode; } -static int do_realtime(struct timespec *ts, u64 *tsc_timestamp) +static int do_realtime(struct timespec64 *ts, u64 *tsc_timestamp) { struct pvclock_gtod_data *gtod = &pvclock_gtod_data; unsigned long seq; @@ -1794,7 +1799,7 @@ static bool kvm_get_time_and_clockread(s64 *kernel_ns, u64 *tsc_timestamp) } /* returns true if host is using TSC based clocksource */ -static bool kvm_get_walltime_and_clockread(struct timespec *ts, +static bool kvm_get_walltime_and_clockread(struct timespec64 *ts, u64 *tsc_timestamp) { /* checked again under seqlock below */ @@ -2868,6 +2873,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_HYPERV_SYNIC2: case KVM_CAP_HYPERV_VP_INDEX: case KVM_CAP_HYPERV_EVENTFD: + case KVM_CAP_HYPERV_TLBFLUSH: case KVM_CAP_PCI_SEGMENT: case KVM_CAP_DEBUGREGS: case KVM_CAP_X86_ROBUST_SINGLESTEP: @@ -2894,7 +2900,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = KVM_CLOCK_TSC_STABLE; break; case KVM_CAP_X86_DISABLE_EXITS: - r |= KVM_X86_DISABLE_EXITS_HTL | KVM_X86_DISABLE_EXITS_PAUSE; + r |= KVM_X86_DISABLE_EXITS_HLT | KVM_X86_DISABLE_EXITS_PAUSE; if(kvm_can_mwait_in_guest()) r |= KVM_X86_DISABLE_EXITS_MWAIT; break; @@ -3962,7 +3968,7 @@ out_nofree: return r; } -int kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf) +vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf) { return VM_FAULT_SIGBUS; } @@ -4248,7 +4254,7 @@ split_irqchip_unlock: if ((cap->args[0] & KVM_X86_DISABLE_EXITS_MWAIT) && kvm_can_mwait_in_guest()) kvm->arch.mwait_in_guest = true; - if (cap->args[0] & KVM_X86_DISABLE_EXITS_HTL) + if (cap->args[0] & KVM_X86_DISABLE_EXITS_HLT) kvm->arch.hlt_in_guest = true; if (cap->args[0] & KVM_X86_DISABLE_EXITS_PAUSE) kvm->arch.pause_in_guest = true; @@ -4787,11 +4793,10 @@ static int kvm_fetch_guest_virt(struct x86_emulate_ctxt *ctxt, return X86EMUL_CONTINUE; } -int kvm_read_guest_virt(struct x86_emulate_ctxt *ctxt, +int kvm_read_guest_virt(struct kvm_vcpu *vcpu, gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception) { - struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); u32 access = (kvm_x86_ops->get_cpl(vcpu) == 3) ? PFERR_USER_MASK : 0; return kvm_read_guest_virt_helper(addr, val, bytes, vcpu, access, @@ -4799,12 +4804,17 @@ int kvm_read_guest_virt(struct x86_emulate_ctxt *ctxt, } EXPORT_SYMBOL_GPL(kvm_read_guest_virt); -static int kvm_read_guest_virt_system(struct x86_emulate_ctxt *ctxt, - gva_t addr, void *val, unsigned int bytes, - struct x86_exception *exception) +static int emulator_read_std(struct x86_emulate_ctxt *ctxt, + gva_t addr, void *val, unsigned int bytes, + struct x86_exception *exception, bool system) { struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); - return kvm_read_guest_virt_helper(addr, val, bytes, vcpu, 0, exception); + u32 access = 0; + + if (!system && kvm_x86_ops->get_cpl(vcpu) == 3) + access |= PFERR_USER_MASK; + + return kvm_read_guest_virt_helper(addr, val, bytes, vcpu, access, exception); } static int kvm_read_guest_phys_system(struct x86_emulate_ctxt *ctxt, @@ -4816,18 +4826,16 @@ static int kvm_read_guest_phys_system(struct x86_emulate_ctxt *ctxt, return r < 0 ? X86EMUL_IO_NEEDED : X86EMUL_CONTINUE; } -int kvm_write_guest_virt_system(struct x86_emulate_ctxt *ctxt, - gva_t addr, void *val, - unsigned int bytes, - struct x86_exception *exception) +static int kvm_write_guest_virt_helper(gva_t addr, void *val, unsigned int bytes, + struct kvm_vcpu *vcpu, u32 access, + struct x86_exception *exception) { - struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); void *data = val; int r = X86EMUL_CONTINUE; while (bytes) { gpa_t gpa = vcpu->arch.walk_mmu->gva_to_gpa(vcpu, addr, - PFERR_WRITE_MASK, + access, exception); unsigned offset = addr & (PAGE_SIZE-1); unsigned towrite = min(bytes, (unsigned)PAGE_SIZE - offset); @@ -4848,6 +4856,27 @@ int kvm_write_guest_virt_system(struct x86_emulate_ctxt *ctxt, out: return r; } + +static int emulator_write_std(struct x86_emulate_ctxt *ctxt, gva_t addr, void *val, + unsigned int bytes, struct x86_exception *exception, + bool system) +{ + struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); + u32 access = PFERR_WRITE_MASK; + + if (!system && kvm_x86_ops->get_cpl(vcpu) == 3) + access |= PFERR_USER_MASK; + + return kvm_write_guest_virt_helper(addr, val, bytes, vcpu, + access, exception); +} + +int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu, gva_t addr, void *val, + unsigned int bytes, struct x86_exception *exception) +{ + return kvm_write_guest_virt_helper(addr, val, bytes, vcpu, + PFERR_WRITE_MASK, exception); +} EXPORT_SYMBOL_GPL(kvm_write_guest_virt_system); int handle_ud(struct kvm_vcpu *vcpu) @@ -4858,8 +4887,8 @@ int handle_ud(struct kvm_vcpu *vcpu) struct x86_exception e; if (force_emulation_prefix && - kvm_read_guest_virt(&vcpu->arch.emulate_ctxt, - kvm_get_linear_rip(vcpu), sig, sizeof(sig), &e) == 0 && + kvm_read_guest_virt(vcpu, kvm_get_linear_rip(vcpu), + sig, sizeof(sig), &e) == 0 && memcmp(sig, "\xf\xbkvm", sizeof(sig)) == 0) { kvm_rip_write(vcpu, kvm_rip_read(vcpu) + sizeof(sig)); emul_type = 0; @@ -5600,8 +5629,8 @@ static int emulator_pre_leave_smm(struct x86_emulate_ctxt *ctxt, u64 smbase) static const struct x86_emulate_ops emulate_ops = { .read_gpr = emulator_read_gpr, .write_gpr = emulator_write_gpr, - .read_std = kvm_read_guest_virt_system, - .write_std = kvm_write_guest_virt_system, + .read_std = emulator_read_std, + .write_std = emulator_write_std, .read_phys = kvm_read_guest_phys_system, .fetch = kvm_fetch_guest_virt, .read_emulated = emulator_read_emulated, @@ -6617,7 +6646,7 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr, unsigned long clock_type) { struct kvm_clock_pairing clock_pairing; - struct timespec ts; + struct timespec64 ts; u64 cycle; int ret; @@ -8538,7 +8567,7 @@ int kvm_arch_hardware_setup(void) /* * Make sure the user can only configure tsc_khz values that * fit into a signed integer. - * A min value is not calculated needed because it will always + * A min value is not calculated because it will always * be 1 on all machines. */ u64 max = min(0x7fffffffULL, @@ -8871,13 +8900,14 @@ int kvm_arch_create_memslot(struct kvm *kvm, struct kvm_memory_slot *slot, slot->base_gfn, level) + 1; slot->arch.rmap[i] = - kvzalloc(lpages * sizeof(*slot->arch.rmap[i]), GFP_KERNEL); + kvcalloc(lpages, sizeof(*slot->arch.rmap[i]), + GFP_KERNEL); if (!slot->arch.rmap[i]) goto out_free; if (i == 0) continue; - linfo = kvzalloc(lpages * sizeof(*linfo), GFP_KERNEL); + linfo = kvcalloc(lpages, sizeof(*linfo), GFP_KERNEL); if (!linfo) goto out_free; diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index c9492f764902..331993c49dae 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -247,11 +247,11 @@ int kvm_inject_realmode_interrupt(struct kvm_vcpu *vcpu, int irq, int inc_eip); void kvm_write_tsc(struct kvm_vcpu *vcpu, struct msr_data *msr); u64 get_kvmclock_ns(struct kvm *kvm); -int kvm_read_guest_virt(struct x86_emulate_ctxt *ctxt, +int kvm_read_guest_virt(struct kvm_vcpu *vcpu, gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception); -int kvm_write_guest_virt_system(struct x86_emulate_ctxt *ctxt, +int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu, gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception); diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S index 9a53a06e5a3e..298ef1479240 100644 --- a/arch/x86/lib/memcpy_64.S +++ b/arch/x86/lib/memcpy_64.S @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -183,12 +184,15 @@ ENTRY(memcpy_orig) ENDPROC(memcpy_orig) #ifndef CONFIG_UML + +MCSAFE_TEST_CTL + /* - * memcpy_mcsafe_unrolled - memory copy with machine check exception handling + * __memcpy_mcsafe - memory copy with machine check exception handling * Note that we only catch machine checks when reading the source addresses. * Writes to target are posted and don't generate machine checks. */ -ENTRY(memcpy_mcsafe_unrolled) +ENTRY(__memcpy_mcsafe) cmpl $8, %edx /* Less than 8 bytes? Go to byte copy loop */ jb .L_no_whole_words @@ -204,58 +208,33 @@ ENTRY(memcpy_mcsafe_unrolled) subl $8, %ecx negl %ecx subl %ecx, %edx -.L_copy_leading_bytes: +.L_read_leading_bytes: movb (%rsi), %al + MCSAFE_TEST_SRC %rsi 1 .E_leading_bytes + MCSAFE_TEST_DST %rdi 1 .E_leading_bytes +.L_write_leading_bytes: movb %al, (%rdi) incq %rsi incq %rdi decl %ecx - jnz .L_copy_leading_bytes + jnz .L_read_leading_bytes .L_8byte_aligned: - /* Figure out how many whole cache lines (64-bytes) to copy */ - movl %edx, %ecx - andl $63, %edx - shrl $6, %ecx - jz .L_no_whole_cache_lines - - /* Loop copying whole cache lines */ -.L_cache_w0: movq (%rsi), %r8 -.L_cache_w1: movq 1*8(%rsi), %r9 -.L_cache_w2: movq 2*8(%rsi), %r10 -.L_cache_w3: movq 3*8(%rsi), %r11 - movq %r8, (%rdi) - movq %r9, 1*8(%rdi) - movq %r10, 2*8(%rdi) - movq %r11, 3*8(%rdi) -.L_cache_w4: movq 4*8(%rsi), %r8 -.L_cache_w5: movq 5*8(%rsi), %r9 -.L_cache_w6: movq 6*8(%rsi), %r10 -.L_cache_w7: movq 7*8(%rsi), %r11 - movq %r8, 4*8(%rdi) - movq %r9, 5*8(%rdi) - movq %r10, 6*8(%rdi) - movq %r11, 7*8(%rdi) - leaq 64(%rsi), %rsi - leaq 64(%rdi), %rdi - decl %ecx - jnz .L_cache_w0 - - /* Are there any trailing 8-byte words? */ -.L_no_whole_cache_lines: movl %edx, %ecx andl $7, %edx shrl $3, %ecx jz .L_no_whole_words - /* Copy trailing words */ -.L_copy_trailing_words: +.L_read_words: movq (%rsi), %r8 - mov %r8, (%rdi) - leaq 8(%rsi), %rsi - leaq 8(%rdi), %rdi + MCSAFE_TEST_SRC %rsi 8 .E_read_words + MCSAFE_TEST_DST %rdi 8 .E_write_words +.L_write_words: + movq %r8, (%rdi) + addq $8, %rsi + addq $8, %rdi decl %ecx - jnz .L_copy_trailing_words + jnz .L_read_words /* Any trailing bytes? */ .L_no_whole_words: @@ -264,38 +243,55 @@ ENTRY(memcpy_mcsafe_unrolled) /* Copy trailing bytes */ movl %edx, %ecx -.L_copy_trailing_bytes: +.L_read_trailing_bytes: movb (%rsi), %al + MCSAFE_TEST_SRC %rsi 1 .E_trailing_bytes + MCSAFE_TEST_DST %rdi 1 .E_trailing_bytes +.L_write_trailing_bytes: movb %al, (%rdi) incq %rsi incq %rdi decl %ecx - jnz .L_copy_trailing_bytes + jnz .L_read_trailing_bytes /* Copy successful. Return zero */ .L_done_memcpy_trap: xorq %rax, %rax ret -ENDPROC(memcpy_mcsafe_unrolled) -EXPORT_SYMBOL_GPL(memcpy_mcsafe_unrolled) +ENDPROC(__memcpy_mcsafe) +EXPORT_SYMBOL_GPL(__memcpy_mcsafe) .section .fixup, "ax" - /* Return -EFAULT for any failure */ -.L_memcpy_mcsafe_fail: - mov $-EFAULT, %rax + /* + * Return number of bytes not copied for any failure. Note that + * there is no "tail" handling since the source buffer is 8-byte + * aligned and poison is cacheline aligned. + */ +.E_read_words: + shll $3, %ecx +.E_leading_bytes: + addl %edx, %ecx +.E_trailing_bytes: + mov %ecx, %eax ret + /* + * For write fault handling, given the destination is unaligned, + * we handle faults on multi-byte writes with a byte-by-byte + * copy up to the write-protected page. + */ +.E_write_words: + shll $3, %ecx + addl %edx, %ecx + movl %ecx, %edx + jmp mcsafe_handle_tail + .previous - _ASM_EXTABLE_FAULT(.L_copy_leading_bytes, .L_memcpy_mcsafe_fail) - _ASM_EXTABLE_FAULT(.L_cache_w0, .L_memcpy_mcsafe_fail) - _ASM_EXTABLE_FAULT(.L_cache_w1, .L_memcpy_mcsafe_fail) - _ASM_EXTABLE_FAULT(.L_cache_w2, .L_memcpy_mcsafe_fail) - _ASM_EXTABLE_FAULT(.L_cache_w3, .L_memcpy_mcsafe_fail) - _ASM_EXTABLE_FAULT(.L_cache_w4, .L_memcpy_mcsafe_fail) - _ASM_EXTABLE_FAULT(.L_cache_w5, .L_memcpy_mcsafe_fail) - _ASM_EXTABLE_FAULT(.L_cache_w6, .L_memcpy_mcsafe_fail) - _ASM_EXTABLE_FAULT(.L_cache_w7, .L_memcpy_mcsafe_fail) - _ASM_EXTABLE_FAULT(.L_copy_trailing_words, .L_memcpy_mcsafe_fail) - _ASM_EXTABLE_FAULT(.L_copy_trailing_bytes, .L_memcpy_mcsafe_fail) + _ASM_EXTABLE_FAULT(.L_read_leading_bytes, .E_leading_bytes) + _ASM_EXTABLE_FAULT(.L_read_words, .E_read_words) + _ASM_EXTABLE_FAULT(.L_read_trailing_bytes, .E_trailing_bytes) + _ASM_EXTABLE(.L_write_leading_bytes, .E_leading_bytes) + _ASM_EXTABLE(.L_write_words, .E_write_words) + _ASM_EXTABLE(.L_write_trailing_bytes, .E_trailing_bytes) #endif diff --git a/arch/x86/lib/usercopy_64.c b/arch/x86/lib/usercopy_64.c index 75d3776123cc..9c5606d88f61 100644 --- a/arch/x86/lib/usercopy_64.c +++ b/arch/x86/lib/usercopy_64.c @@ -23,13 +23,13 @@ unsigned long __clear_user(void __user *addr, unsigned long size) asm volatile( " testq %[size8],%[size8]\n" " jz 4f\n" - "0: movq %[zero],(%[dst])\n" - " addq %[eight],%[dst]\n" + "0: movq $0,(%[dst])\n" + " addq $8,%[dst]\n" " decl %%ecx ; jnz 0b\n" "4: movq %[size1],%%rcx\n" " testl %%ecx,%%ecx\n" " jz 2f\n" - "1: movb %b[zero],(%[dst])\n" + "1: movb $0,(%[dst])\n" " incq %[dst]\n" " decl %%ecx ; jnz 1b\n" "2:\n" @@ -40,8 +40,7 @@ unsigned long __clear_user(void __user *addr, unsigned long size) _ASM_EXTABLE(0b,3b) _ASM_EXTABLE(1b,2b) : [size8] "=&c"(size), [dst] "=&D" (__d0) - : [size1] "r"(size & 7), "[size8]" (size / 8), "[dst]"(addr), - [zero] "r" (0UL), [eight] "r" (8UL)); + : [size1] "r"(size & 7), "[size8]" (size / 8), "[dst]"(addr)); clac(); return size; } @@ -75,6 +74,27 @@ copy_user_handle_tail(char *to, char *from, unsigned len) return len; } +/* + * Similar to copy_user_handle_tail, probe for the write fault point, + * but reuse __memcpy_mcsafe in case a new read error is encountered. + * clac() is handled in _copy_to_iter_mcsafe(). + */ +__visible unsigned long +mcsafe_handle_tail(char *to, char *from, unsigned len) +{ + for (; len; --len, to++, from++) { + /* + * Call the assembly routine back directly since + * memcpy_mcsafe() may silently fallback to memcpy. + */ + unsigned long rem = __memcpy_mcsafe(to, from, 1); + + if (rem) + break; + } + return len; +} + #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE /** * clean_cache_range - write back a cache range with CLWB diff --git a/arch/x86/mm/dump_pagetables.c b/arch/x86/mm/dump_pagetables.c index cc7ff5957194..2f3c9196b834 100644 --- a/arch/x86/mm/dump_pagetables.c +++ b/arch/x86/mm/dump_pagetables.c @@ -360,7 +360,7 @@ static inline bool kasan_page_table(struct seq_file *m, struct pg_state *st, void *pt) { if (__pa(pt) == __pa(kasan_zero_pmd) || - (pgtable_l5_enabled && __pa(pt) == __pa(kasan_zero_p4d)) || + (pgtable_l5_enabled() && __pa(pt) == __pa(kasan_zero_p4d)) || __pa(pt) == __pa(kasan_zero_pud)) { pgprotval_t prot = pte_flags(kasan_zero_pte[0]); note_page(m, st, __pgprot(prot), 0, 5); @@ -476,8 +476,8 @@ static void walk_p4d_level(struct seq_file *m, struct pg_state *st, pgd_t addr, } } -#define pgd_large(a) (pgtable_l5_enabled ? pgd_large(a) : p4d_large(__p4d(pgd_val(a)))) -#define pgd_none(a) (pgtable_l5_enabled ? pgd_none(a) : p4d_none(__p4d(pgd_val(a)))) +#define pgd_large(a) (pgtable_l5_enabled() ? pgd_large(a) : p4d_large(__p4d(pgd_val(a)))) +#define pgd_none(a) (pgtable_l5_enabled() ? pgd_none(a) : p4d_none(__p4d(pgd_val(a)))) static inline bool is_hypervisor_range(int idx) { diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 73bd8c95ac71..9a84a0d08727 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -209,6 +209,7 @@ force_sig_info_fault(int si_signo, int si_code, unsigned long address, unsigned lsb = 0; siginfo_t info; + clear_siginfo(&info); info.si_signo = si_signo; info.si_errno = 0; info.si_code = si_code; @@ -439,7 +440,7 @@ static noinline int vmalloc_fault(unsigned long address) if (pgd_none(*pgd_k)) return -1; - if (pgtable_l5_enabled) { + if (pgtable_l5_enabled()) { if (pgd_none(*pgd)) { set_pgd(pgd, *pgd_k); arch_flush_lazy_mmu_mode(); @@ -454,7 +455,7 @@ static noinline int vmalloc_fault(unsigned long address) if (p4d_none(*p4d_k)) return -1; - if (p4d_none(*p4d) && !pgtable_l5_enabled) { + if (p4d_none(*p4d) && !pgtable_l5_enabled()) { set_p4d(p4d, *p4d_k); arch_flush_lazy_mmu_mode(); } else { @@ -828,6 +829,8 @@ static inline void show_signal_msg(struct pt_regs *regs, unsigned long error_code, unsigned long address, struct task_struct *tsk) { + const char *loglvl = task_pid_nr(tsk) > 1 ? KERN_INFO : KERN_EMERG; + if (!unhandled_signal(tsk, SIGSEGV)) return; @@ -835,13 +838,14 @@ show_signal_msg(struct pt_regs *regs, unsigned long error_code, return; printk("%s%s[%d]: segfault at %lx ip %px sp %px error %lx", - task_pid_nr(tsk) > 1 ? KERN_INFO : KERN_EMERG, - tsk->comm, task_pid_nr(tsk), address, + loglvl, tsk->comm, task_pid_nr(tsk), address, (void *)regs->ip, (void *)regs->sp, error_code); print_vma_addr(KERN_CONT " in ", regs->ip); printk(KERN_CONT "\n"); + + show_opcodes((u8 *)regs->ip, loglvl); } static void diff --git a/arch/x86/mm/ident_map.c b/arch/x86/mm/ident_map.c index a2f0c7e20fb0..fe7a12599d8e 100644 --- a/arch/x86/mm/ident_map.c +++ b/arch/x86/mm/ident_map.c @@ -123,7 +123,7 @@ int kernel_ident_mapping_init(struct x86_mapping_info *info, pgd_t *pgd_page, result = ident_p4d_init(info, p4d, addr, next); if (result) return result; - if (pgtable_l5_enabled) { + if (pgtable_l5_enabled()) { set_pgd(pgd, __pgd(__pa(p4d) | info->kernpg_flag)); } else { /* diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index fec82b577c18..cee58a972cb2 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -706,7 +706,9 @@ void __init init_mem_mapping(void) */ int devmem_is_allowed(unsigned long pagenr) { - if (page_is_ram(pagenr)) { + if (region_intersects(PFN_PHYS(pagenr), PAGE_SIZE, + IORESOURCE_SYSTEM_RAM, IORES_DESC_NONE) + != REGION_DISJOINT) { /* * For disallowed memory regions in the low 1MB range, * request that the page be shown as all zeros. diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c index c893c6a3d707..979e0a02cbe1 100644 --- a/arch/x86/mm/init_32.c +++ b/arch/x86/mm/init_32.c @@ -692,7 +692,7 @@ void __init initmem_init(void) high_memory = (void *) __va(max_low_pfn * PAGE_SIZE - 1) + 1; #endif - memblock_set_node(0, (phys_addr_t)ULLONG_MAX, &memblock.memory, 0); + memblock_set_node(0, PHYS_ADDR_MAX, &memblock.memory, 0); sparse_memory_present_with_active_regions(0); #ifdef CONFIG_FLATMEM diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 0a400606dea0..045f492d5f68 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -180,7 +180,7 @@ static void sync_global_pgds_l4(unsigned long start, unsigned long end) */ void sync_global_pgds(unsigned long start, unsigned long end) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled()) sync_global_pgds_l5(start, end); else sync_global_pgds_l4(start, end); @@ -643,7 +643,7 @@ phys_p4d_init(p4d_t *p4d_page, unsigned long paddr, unsigned long paddr_end, unsigned long vaddr = (unsigned long)__va(paddr); int i = p4d_index(vaddr); - if (!pgtable_l5_enabled) + if (!pgtable_l5_enabled()) return phys_pud_init((pud_t *) p4d_page, paddr, paddr_end, page_size_mask); for (; i < PTRS_PER_P4D; i++, paddr = paddr_next) { @@ -723,7 +723,7 @@ kernel_physical_mapping_init(unsigned long paddr_start, page_size_mask); spin_lock(&init_mm.page_table_lock); - if (pgtable_l5_enabled) + if (pgtable_l5_enabled()) pgd_populate(&init_mm, pgd, p4d); else p4d_populate(&init_mm, p4d_offset(pgd, vaddr), (pud_t *) p4d); @@ -742,7 +742,7 @@ kernel_physical_mapping_init(unsigned long paddr_start, #ifndef CONFIG_NUMA void __init initmem_init(void) { - memblock_set_node(0, (phys_addr_t)ULLONG_MAX, &memblock.memory, 0); + memblock_set_node(0, PHYS_ADDR_MAX, &memblock.memory, 0); } #endif @@ -1100,7 +1100,7 @@ remove_p4d_table(p4d_t *p4d_start, unsigned long addr, unsigned long end, * 5-level case we should free them. This code will have to change * to adapt for boot-time switching between 4 and 5 level page tables. */ - if (pgtable_l5_enabled) + if (pgtable_l5_enabled()) free_pud_table(pud_base, p4d); } diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index 980dbebd0ca7..e3e77527f8df 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -2,10 +2,8 @@ #define DISABLE_BRANCH_PROFILING #define pr_fmt(fmt) "kasan: " fmt -#ifdef CONFIG_X86_5LEVEL -/* Too early to use cpu_feature_enabled() */ -#define pgtable_l5_enabled __pgtable_l5_enabled -#endif +/* cpu_feature_enabled() cannot be used this early */ +#define USE_EARLY_PGTABLE_L5 #include #include @@ -182,7 +180,7 @@ static void __init clear_pgds(unsigned long start, * With folded p4d, pgd_clear() is nop, use p4d_clear() * instead. */ - if (pgtable_l5_enabled) + if (pgtable_l5_enabled()) pgd_clear(pgd); else p4d_clear(p4d_offset(pgd, start)); @@ -197,7 +195,7 @@ static inline p4d_t *early_p4d_offset(pgd_t *pgd, unsigned long addr) { unsigned long p4d; - if (!pgtable_l5_enabled) + if (!pgtable_l5_enabled()) return (p4d_t *)pgd; p4d = __pa_nodebug(pgd_val(*pgd)) & PTE_PFN_MASK; @@ -284,7 +282,7 @@ void __init kasan_early_init(void) for (i = 0; i < PTRS_PER_PUD; i++) kasan_zero_pud[i] = __pud(pud_val); - for (i = 0; pgtable_l5_enabled && i < PTRS_PER_P4D; i++) + for (i = 0; pgtable_l5_enabled() && i < PTRS_PER_P4D; i++) kasan_zero_p4d[i] = __p4d(p4d_val); kasan_map_early_shadow(early_top_pgt); @@ -315,7 +313,7 @@ void __init kasan_init(void) * bunch of things like kernel code, modules, EFI mapping, etc. * We need to take extra steps to not overwrite them. */ - if (pgtable_l5_enabled) { + if (pgtable_l5_enabled()) { void *ptr; ptr = (void *)pgd_page_vaddr(*pgd_offset_k(KASAN_SHADOW_END)); diff --git a/arch/x86/mm/kaslr.c b/arch/x86/mm/kaslr.c index 615cc03ced84..61db77b0eda9 100644 --- a/arch/x86/mm/kaslr.c +++ b/arch/x86/mm/kaslr.c @@ -78,7 +78,7 @@ void __init kernel_randomize_memory(void) struct rnd_state rand_state; unsigned long remain_entropy; - vaddr_start = pgtable_l5_enabled ? __PAGE_OFFSET_BASE_L5 : __PAGE_OFFSET_BASE_L4; + vaddr_start = pgtable_l5_enabled() ? __PAGE_OFFSET_BASE_L5 : __PAGE_OFFSET_BASE_L4; vaddr = vaddr_start; /* @@ -124,7 +124,7 @@ void __init kernel_randomize_memory(void) */ entropy = remain_entropy / (ARRAY_SIZE(kaslr_regions) - i); prandom_bytes_state(&rand_state, &rand, sizeof(rand)); - if (pgtable_l5_enabled) + if (pgtable_l5_enabled()) entropy = (rand % (entropy + 1)) & P4D_MASK; else entropy = (rand % (entropy + 1)) & PUD_MASK; @@ -136,7 +136,7 @@ void __init kernel_randomize_memory(void) * randomization alignment. */ vaddr += get_padding(&kaslr_regions[i]); - if (pgtable_l5_enabled) + if (pgtable_l5_enabled()) vaddr = round_up(vaddr + 1, P4D_SIZE); else vaddr = round_up(vaddr + 1, PUD_SIZE); @@ -212,7 +212,7 @@ void __meminit init_trampoline(void) return; } - if (pgtable_l5_enabled) + if (pgtable_l5_enabled()) init_trampoline_p4d(); else init_trampoline_pud(); diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 1b2197d13832..7ae36868aed2 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -527,6 +527,7 @@ void __init sme_enable(struct boot_params *bp) /* SEV state cannot be controlled by a command line option */ sme_me_mask = me_mask; sev_enabled = true; + physical_mask &= ~sme_me_mask; return; } @@ -561,4 +562,6 @@ void __init sme_enable(struct boot_params *bp) sme_me_mask = 0; else sme_me_mask = active_by_default ? me_mask : 0; + + physical_mask &= ~sme_me_mask; } diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c index 25504d5aa816..fa150855647c 100644 --- a/arch/x86/mm/numa.c +++ b/arch/x86/mm/numa.c @@ -136,13 +136,13 @@ static int __init numa_add_memblk_to(int nid, u64 start, u64 end, /* whine about and ignore invalid blks */ if (start > end || nid < 0 || nid >= MAX_NUMNODES) { - pr_warning("NUMA: Warning: invalid memblk node %d [mem %#010Lx-%#010Lx]\n", - nid, start, end - 1); + pr_warn("Warning: invalid memblk node %d [mem %#010Lx-%#010Lx]\n", + nid, start, end - 1); return 0; } if (mi->nr_blks >= NR_NODE_MEMBLKS) { - pr_err("NUMA: too many memblk ranges\n"); + pr_err("too many memblk ranges\n"); return -EINVAL; } @@ -267,14 +267,14 @@ int __init numa_cleanup_meminfo(struct numa_meminfo *mi) */ if (bi->end > bj->start && bi->start < bj->end) { if (bi->nid != bj->nid) { - pr_err("NUMA: node %d [mem %#010Lx-%#010Lx] overlaps with node %d [mem %#010Lx-%#010Lx]\n", + pr_err("node %d [mem %#010Lx-%#010Lx] overlaps with node %d [mem %#010Lx-%#010Lx]\n", bi->nid, bi->start, bi->end - 1, bj->nid, bj->start, bj->end - 1); return -EINVAL; } - pr_warning("NUMA: Warning: node %d [mem %#010Lx-%#010Lx] overlaps with itself [mem %#010Lx-%#010Lx]\n", - bi->nid, bi->start, bi->end - 1, - bj->start, bj->end - 1); + pr_warn("Warning: node %d [mem %#010Lx-%#010Lx] overlaps with itself [mem %#010Lx-%#010Lx]\n", + bi->nid, bi->start, bi->end - 1, + bj->start, bj->end - 1); } /* @@ -364,7 +364,7 @@ static int __init numa_alloc_distance(void) phys = memblock_find_in_range(0, PFN_PHYS(max_pfn_mapped), size, PAGE_SIZE); if (!phys) { - pr_warning("NUMA: Warning: can't allocate distance table!\n"); + pr_warn("Warning: can't allocate distance table!\n"); /* don't retry until explicitly reset */ numa_distance = (void *)1LU; return -ENOMEM; @@ -410,14 +410,14 @@ void __init numa_set_distance(int from, int to, int distance) if (from >= numa_distance_cnt || to >= numa_distance_cnt || from < 0 || to < 0) { - pr_warn_once("NUMA: Warning: node ids are out of bound, from=%d to=%d distance=%d\n", - from, to, distance); + pr_warn_once("Warning: node ids are out of bound, from=%d to=%d distance=%d\n", + from, to, distance); return; } if ((u8)distance != distance || (from == to && distance != LOCAL_DISTANCE)) { - pr_warn_once("NUMA: Warning: invalid distance parameter, from=%d to=%d distance=%d\n", + pr_warn_once("Warning: invalid distance parameter, from=%d to=%d distance=%d\n", from, to, distance); return; } diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index ffc8c13c50e4..47b5951e592b 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -8,6 +8,11 @@ #include #include +#ifdef CONFIG_DYNAMIC_PHYSICAL_MASK +phys_addr_t physical_mask __ro_after_init = (1ULL << __PHYSICAL_MASK_SHIFT) - 1; +EXPORT_SYMBOL(physical_mask); +#endif + #define PGALLOC_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO) #ifdef CONFIG_HIGHPTE @@ -114,13 +119,12 @@ static inline void pgd_list_del(pgd_t *pgd) static void pgd_set_mm(pgd_t *pgd, struct mm_struct *mm) { - BUILD_BUG_ON(sizeof(virt_to_page(pgd)->index) < sizeof(mm)); - virt_to_page(pgd)->index = (pgoff_t)mm; + virt_to_page(pgd)->pt_mm = mm; } struct mm_struct *pgd_page_get_mm(struct page *page) { - return (struct mm_struct *)page->index; + return page->pt_mm; } static void pgd_ctor(struct mm_struct *mm, pgd_t *pgd) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index e055d1a06699..6eb1f34c3c85 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -157,7 +157,7 @@ static void sync_current_stack_to_mm(struct mm_struct *mm) unsigned long sp = current_stack_pointer; pgd_t *pgd = pgd_offset(mm, sp); - if (pgtable_l5_enabled) { + if (pgtable_l5_enabled()) { if (unlikely(pgd_none(*pgd))) { pgd_t *pgd_ref = pgd_offset_k(sp); diff --git a/arch/x86/net/Makefile b/arch/x86/net/Makefile index fefb4b619598..59e123da580c 100644 --- a/arch/x86/net/Makefile +++ b/arch/x86/net/Makefile @@ -1,6 +1,9 @@ # # Arch-specific network modules # -OBJECT_FILES_NON_STANDARD_bpf_jit.o += y -obj-$(CONFIG_BPF_JIT) += bpf_jit.o bpf_jit_comp.o +ifeq ($(CONFIG_X86_32),y) + obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o +else + obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o +endif diff --git a/arch/x86/net/bpf_jit.S b/arch/x86/net/bpf_jit.S deleted file mode 100644 index b33093f84528..000000000000 --- a/arch/x86/net/bpf_jit.S +++ /dev/null @@ -1,154 +0,0 @@ -/* bpf_jit.S : BPF JIT helper functions - * - * Copyright (C) 2011 Eric Dumazet (eric.dumazet@gmail.com) - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; version 2 - * of the License. - */ -#include -#include - -/* - * Calling convention : - * rbx : skb pointer (callee saved) - * esi : offset of byte(s) to fetch in skb (can be scratched) - * r10 : copy of skb->data - * r9d : hlen = skb->len - skb->data_len - */ -#define SKBDATA %r10 -#define SKF_MAX_NEG_OFF $(-0x200000) /* SKF_LL_OFF from filter.h */ - -#define FUNC(name) \ - .globl name; \ - .type name, @function; \ - name: - -FUNC(sk_load_word) - test %esi,%esi - js bpf_slow_path_word_neg - -FUNC(sk_load_word_positive_offset) - mov %r9d,%eax # hlen - sub %esi,%eax # hlen - offset - cmp $3,%eax - jle bpf_slow_path_word - mov (SKBDATA,%rsi),%eax - bswap %eax /* ntohl() */ - ret - -FUNC(sk_load_half) - test %esi,%esi - js bpf_slow_path_half_neg - -FUNC(sk_load_half_positive_offset) - mov %r9d,%eax - sub %esi,%eax # hlen - offset - cmp $1,%eax - jle bpf_slow_path_half - movzwl (SKBDATA,%rsi),%eax - rol $8,%ax # ntohs() - ret - -FUNC(sk_load_byte) - test %esi,%esi - js bpf_slow_path_byte_neg - -FUNC(sk_load_byte_positive_offset) - cmp %esi,%r9d /* if (offset >= hlen) goto bpf_slow_path_byte */ - jle bpf_slow_path_byte - movzbl (SKBDATA,%rsi),%eax - ret - -/* rsi contains offset and can be scratched */ -#define bpf_slow_path_common(LEN) \ - lea 32(%rbp), %rdx;\ - FRAME_BEGIN; \ - mov %rbx, %rdi; /* arg1 == skb */ \ - push %r9; \ - push SKBDATA; \ -/* rsi already has offset */ \ - mov $LEN,%ecx; /* len */ \ - call skb_copy_bits; \ - test %eax,%eax; \ - pop SKBDATA; \ - pop %r9; \ - FRAME_END - - -bpf_slow_path_word: - bpf_slow_path_common(4) - js bpf_error - mov 32(%rbp),%eax - bswap %eax - ret - -bpf_slow_path_half: - bpf_slow_path_common(2) - js bpf_error - mov 32(%rbp),%ax - rol $8,%ax - movzwl %ax,%eax - ret - -bpf_slow_path_byte: - bpf_slow_path_common(1) - js bpf_error - movzbl 32(%rbp),%eax - ret - -#define sk_negative_common(SIZE) \ - FRAME_BEGIN; \ - mov %rbx, %rdi; /* arg1 == skb */ \ - push %r9; \ - push SKBDATA; \ -/* rsi already has offset */ \ - mov $SIZE,%edx; /* size */ \ - call bpf_internal_load_pointer_neg_helper; \ - test %rax,%rax; \ - pop SKBDATA; \ - pop %r9; \ - FRAME_END; \ - jz bpf_error - -bpf_slow_path_word_neg: - cmp SKF_MAX_NEG_OFF, %esi /* test range */ - jl bpf_error /* offset lower -> error */ - -FUNC(sk_load_word_negative_offset) - sk_negative_common(4) - mov (%rax), %eax - bswap %eax - ret - -bpf_slow_path_half_neg: - cmp SKF_MAX_NEG_OFF, %esi - jl bpf_error - -FUNC(sk_load_half_negative_offset) - sk_negative_common(2) - mov (%rax),%ax - rol $8,%ax - movzwl %ax,%eax - ret - -bpf_slow_path_byte_neg: - cmp SKF_MAX_NEG_OFF, %esi - jl bpf_error - -FUNC(sk_load_byte_negative_offset) - sk_negative_common(1) - movzbl (%rax), %eax - ret - -bpf_error: -# force a return 0 from jit handler - xor %eax,%eax - mov (%rbp),%rbx - mov 8(%rbp),%r13 - mov 16(%rbp),%r14 - mov 24(%rbp),%r15 - add $40, %rbp - leaveq - ret diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 263c8453815e..2580cd2e98b1 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1,4 +1,5 @@ -/* bpf_jit_comp.c : BPF JIT compiler +/* + * bpf_jit_comp.c: BPF JIT compiler * * Copyright (C) 2011-2013 Eric Dumazet (eric.dumazet@gmail.com) * Internal BPF Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com @@ -16,15 +17,6 @@ #include #include -/* - * assembly code in arch/x86/net/bpf_jit.S - */ -extern u8 sk_load_word[], sk_load_half[], sk_load_byte[]; -extern u8 sk_load_word_positive_offset[], sk_load_half_positive_offset[]; -extern u8 sk_load_byte_positive_offset[]; -extern u8 sk_load_word_negative_offset[], sk_load_half_negative_offset[]; -extern u8 sk_load_byte_negative_offset[]; - static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len) { if (len == 1) @@ -45,14 +37,15 @@ static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len) #define EMIT2(b1, b2) EMIT((b1) + ((b2) << 8), 2) #define EMIT3(b1, b2, b3) EMIT((b1) + ((b2) << 8) + ((b3) << 16), 3) #define EMIT4(b1, b2, b3, b4) EMIT((b1) + ((b2) << 8) + ((b3) << 16) + ((b4) << 24), 4) + #define EMIT1_off32(b1, off) \ - do {EMIT1(b1); EMIT(off, 4); } while (0) + do { EMIT1(b1); EMIT(off, 4); } while (0) #define EMIT2_off32(b1, b2, off) \ - do {EMIT2(b1, b2); EMIT(off, 4); } while (0) + do { EMIT2(b1, b2); EMIT(off, 4); } while (0) #define EMIT3_off32(b1, b2, b3, off) \ - do {EMIT3(b1, b2, b3); EMIT(off, 4); } while (0) + do { EMIT3(b1, b2, b3); EMIT(off, 4); } while (0) #define EMIT4_off32(b1, b2, b3, b4, off) \ - do {EMIT4(b1, b2, b3, b4); EMIT(off, 4); } while (0) + do { EMIT4(b1, b2, b3, b4); EMIT(off, 4); } while (0) static bool is_imm8(int value) { @@ -70,9 +63,10 @@ static bool is_uimm32(u64 value) } /* mov dst, src */ -#define EMIT_mov(DST, SRC) \ - do {if (DST != SRC) \ - EMIT3(add_2mod(0x48, DST, SRC), 0x89, add_2reg(0xC0, DST, SRC)); \ +#define EMIT_mov(DST, SRC) \ + do { \ + if (DST != SRC) \ + EMIT3(add_2mod(0x48, DST, SRC), 0x89, add_2reg(0xC0, DST, SRC)); \ } while (0) static int bpf_size_to_x86_bytes(int bpf_size) @@ -89,7 +83,8 @@ static int bpf_size_to_x86_bytes(int bpf_size) return 0; } -/* list of x86 cond jumps opcodes (. + s8) +/* + * List of x86 cond jumps opcodes (. + s8) * Add 0x10 (and an extra 0x0f) to generate far jumps (. + s32) */ #define X86_JB 0x72 @@ -103,38 +98,37 @@ static int bpf_size_to_x86_bytes(int bpf_size) #define X86_JLE 0x7E #define X86_JG 0x7F -#define CHOOSE_LOAD_FUNC(K, func) \ - ((int)K < 0 ? ((int)K >= SKF_LL_OFF ? func##_negative_offset : func) : func##_positive_offset) - -/* pick a register outside of BPF range for JIT internal work */ +/* Pick a register outside of BPF range for JIT internal work */ #define AUX_REG (MAX_BPF_JIT_REG + 1) -/* The following table maps BPF registers to x64 registers. +/* + * The following table maps BPF registers to x86-64 registers. * - * x64 register r12 is unused, since if used as base address + * x86-64 register R12 is unused, since if used as base address * register in load/store instructions, it always needs an * extra byte of encoding and is callee saved. * - * r9 caches skb->len - skb->data_len - * r10 caches skb->data, and used for blinding (if enabled) + * Also x86-64 register R9 is unused. x86-64 register R10 is + * used for blinding (if enabled). */ static const int reg2hex[] = { - [BPF_REG_0] = 0, /* rax */ - [BPF_REG_1] = 7, /* rdi */ - [BPF_REG_2] = 6, /* rsi */ - [BPF_REG_3] = 2, /* rdx */ - [BPF_REG_4] = 1, /* rcx */ - [BPF_REG_5] = 0, /* r8 */ - [BPF_REG_6] = 3, /* rbx callee saved */ - [BPF_REG_7] = 5, /* r13 callee saved */ - [BPF_REG_8] = 6, /* r14 callee saved */ - [BPF_REG_9] = 7, /* r15 callee saved */ - [BPF_REG_FP] = 5, /* rbp readonly */ - [BPF_REG_AX] = 2, /* r10 temp register */ - [AUX_REG] = 3, /* r11 temp register */ + [BPF_REG_0] = 0, /* RAX */ + [BPF_REG_1] = 7, /* RDI */ + [BPF_REG_2] = 6, /* RSI */ + [BPF_REG_3] = 2, /* RDX */ + [BPF_REG_4] = 1, /* RCX */ + [BPF_REG_5] = 0, /* R8 */ + [BPF_REG_6] = 3, /* RBX callee saved */ + [BPF_REG_7] = 5, /* R13 callee saved */ + [BPF_REG_8] = 6, /* R14 callee saved */ + [BPF_REG_9] = 7, /* R15 callee saved */ + [BPF_REG_FP] = 5, /* RBP readonly */ + [BPF_REG_AX] = 2, /* R10 temp register */ + [AUX_REG] = 3, /* R11 temp register */ }; -/* is_ereg() == true if BPF register 'reg' maps to x64 r8..r15 +/* + * is_ereg() == true if BPF register 'reg' maps to x86-64 r8..r15 * which need extra byte of encoding. * rax,rcx,...,rbp have simpler encoding */ @@ -153,7 +147,7 @@ static bool is_axreg(u32 reg) return reg == BPF_REG_0; } -/* add modifiers if 'reg' maps to x64 registers r8..r15 */ +/* Add modifiers if 'reg' maps to x86-64 registers R8..R15 */ static u8 add_1mod(u8 byte, u32 reg) { if (is_ereg(reg)) @@ -170,13 +164,13 @@ static u8 add_2mod(u8 byte, u32 r1, u32 r2) return byte; } -/* encode 'dst_reg' register into x64 opcode 'byte' */ +/* Encode 'dst_reg' register into x86-64 opcode 'byte' */ static u8 add_1reg(u8 byte, u32 dst_reg) { return byte + reg2hex[dst_reg]; } -/* encode 'dst_reg' and 'src_reg' registers into x64 opcode 'byte' */ +/* Encode 'dst_reg' and 'src_reg' registers into x86-64 opcode 'byte' */ static u8 add_2reg(u8 byte, u32 dst_reg, u32 src_reg) { return byte + reg2hex[dst_reg] + (reg2hex[src_reg] << 3); @@ -184,27 +178,24 @@ static u8 add_2reg(u8 byte, u32 dst_reg, u32 src_reg) static void jit_fill_hole(void *area, unsigned int size) { - /* fill whole space with int3 instructions */ + /* Fill whole space with INT3 instructions */ memset(area, 0xcc, size); } struct jit_context { - int cleanup_addr; /* epilogue code offset */ - bool seen_ld_abs; - bool seen_ax_reg; + int cleanup_addr; /* Epilogue code offset */ }; -/* maximum number of bytes emitted while JITing one eBPF insn */ +/* Maximum number of bytes emitted while JITing one eBPF insn */ #define BPF_MAX_INSN_SIZE 128 #define BPF_INSN_SAFETY 64 -#define AUX_STACK_SPACE \ - (32 /* space for rbx, r13, r14, r15 */ + \ - 8 /* space for skb_copy_bits() buffer */) +#define AUX_STACK_SPACE 40 /* Space for RBX, R13, R14, R15, tailcnt */ -#define PROLOGUE_SIZE 37 +#define PROLOGUE_SIZE 37 -/* emit x64 prologue code for BPF program and check it's size. +/* + * Emit x86-64 prologue code for BPF program and check its size. * bpf_tail_call helper will skip it while jumping into another program */ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf) @@ -212,8 +203,11 @@ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf) u8 *prog = *pprog; int cnt = 0; - EMIT1(0x55); /* push rbp */ - EMIT3(0x48, 0x89, 0xE5); /* mov rbp,rsp */ + /* push rbp */ + EMIT1(0x55); + + /* mov rbp,rsp */ + EMIT3(0x48, 0x89, 0xE5); /* sub rsp, rounded_stack_depth + AUX_STACK_SPACE */ EMIT3_off32(0x48, 0x81, 0xEC, @@ -222,19 +216,8 @@ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf) /* sub rbp, AUX_STACK_SPACE */ EMIT4(0x48, 0x83, 0xED, AUX_STACK_SPACE); - /* all classic BPF filters use R6(rbx) save it */ - /* mov qword ptr [rbp+0],rbx */ EMIT4(0x48, 0x89, 0x5D, 0); - - /* bpf_convert_filter() maps classic BPF register X to R7 and uses R8 - * as temporary, so all tcpdump filters need to spill/fill R7(r13) and - * R8(r14). R9(r15) spill could be made conditional, but there is only - * one 'bpf_error' return path out of helper functions inside bpf_jit.S - * The overhead of extra spill is negligible for any filter other - * than synthetic ones. Therefore not worth adding complexity. - */ - /* mov qword ptr [rbp+8],r13 */ EMIT4(0x4C, 0x89, 0x6D, 8); /* mov qword ptr [rbp+16],r14 */ @@ -243,9 +226,10 @@ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf) EMIT4(0x4C, 0x89, 0x7D, 24); if (!ebpf_from_cbpf) { - /* Clear the tail call counter (tail_call_cnt): for eBPF tail + /* + * Clear the tail call counter (tail_call_cnt): for eBPF tail * calls we need to reset the counter to 0. It's done in two - * instructions, resetting rax register to 0, and moving it + * instructions, resetting RAX register to 0, and moving it * to the counter location. */ @@ -260,7 +244,9 @@ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf) *pprog = prog; } -/* generate the following code: +/* + * Generate the following code: + * * ... bpf_tail_call(void *ctx, struct bpf_array *array, u64 index) ... * if (index >= array->map.max_entries) * goto out; @@ -278,23 +264,26 @@ static void emit_bpf_tail_call(u8 **pprog) int label1, label2, label3; int cnt = 0; - /* rdi - pointer to ctx + /* + * rdi - pointer to ctx * rsi - pointer to bpf_array * rdx - index in bpf_array */ - /* if (index >= array->map.max_entries) - * goto out; + /* + * if (index >= array->map.max_entries) + * goto out; */ EMIT2(0x89, 0xD2); /* mov edx, edx */ EMIT3(0x39, 0x56, /* cmp dword ptr [rsi + 16], edx */ offsetof(struct bpf_array, map.max_entries)); -#define OFFSET1 (41 + RETPOLINE_RAX_BPF_JIT_SIZE) /* number of bytes to jump */ +#define OFFSET1 (41 + RETPOLINE_RAX_BPF_JIT_SIZE) /* Number of bytes to jump */ EMIT2(X86_JBE, OFFSET1); /* jbe out */ label1 = cnt; - /* if (tail_call_cnt > MAX_TAIL_CALL_CNT) - * goto out; + /* + * if (tail_call_cnt > MAX_TAIL_CALL_CNT) + * goto out; */ EMIT2_off32(0x8B, 0x85, 36); /* mov eax, dword ptr [rbp + 36] */ EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT); /* cmp eax, MAX_TAIL_CALL_CNT */ @@ -308,8 +297,9 @@ static void emit_bpf_tail_call(u8 **pprog) EMIT4_off32(0x48, 0x8B, 0x84, 0xD6, /* mov rax, [rsi + rdx * 8 + offsetof(...)] */ offsetof(struct bpf_array, ptrs)); - /* if (prog == NULL) - * goto out; + /* + * if (prog == NULL) + * goto out; */ EMIT3(0x48, 0x85, 0xC0); /* test rax,rax */ #define OFFSET3 (8 + RETPOLINE_RAX_BPF_JIT_SIZE) @@ -321,7 +311,8 @@ static void emit_bpf_tail_call(u8 **pprog) offsetof(struct bpf_prog, bpf_func)); EMIT4(0x48, 0x83, 0xC0, PROLOGUE_SIZE); /* add rax, prologue_size */ - /* now we're ready to jump into next BPF program + /* + * Wow we're ready to jump into next BPF program * rdi == ctx (1st arg) * rax == prog->bpf_func + prologue_size */ @@ -334,26 +325,6 @@ static void emit_bpf_tail_call(u8 **pprog) *pprog = prog; } - -static void emit_load_skb_data_hlen(u8 **pprog) -{ - u8 *prog = *pprog; - int cnt = 0; - - /* r9d = skb->len - skb->data_len (headlen) - * r10 = skb->data - */ - /* mov %r9d, off32(%rdi) */ - EMIT3_off32(0x44, 0x8b, 0x8f, offsetof(struct sk_buff, len)); - - /* sub %r9d, off32(%rdi) */ - EMIT3_off32(0x44, 0x2b, 0x8f, offsetof(struct sk_buff, data_len)); - - /* mov %r10, off32(%rdi) */ - EMIT3_off32(0x4c, 0x8b, 0x97, offsetof(struct sk_buff, data)); - *pprog = prog; -} - static void emit_mov_imm32(u8 **pprog, bool sign_propagate, u32 dst_reg, const u32 imm32) { @@ -361,7 +332,8 @@ static void emit_mov_imm32(u8 **pprog, bool sign_propagate, u8 b1, b2, b3; int cnt = 0; - /* optimization: if imm32 is positive, use 'mov %eax, imm32' + /* + * Optimization: if imm32 is positive, use 'mov %eax, imm32' * (which zero-extends imm32) to save 2 bytes. */ if (sign_propagate && (s32)imm32 < 0) { @@ -373,7 +345,8 @@ static void emit_mov_imm32(u8 **pprog, bool sign_propagate, goto done; } - /* optimization: if imm32 is zero, use 'xor %eax, %eax' + /* + * Optimization: if imm32 is zero, use 'xor %eax, %eax' * to save 3 bytes. */ if (imm32 == 0) { @@ -400,7 +373,8 @@ static void emit_mov_imm64(u8 **pprog, u32 dst_reg, int cnt = 0; if (is_uimm32(((u64)imm32_hi << 32) | (u32)imm32_lo)) { - /* For emitting plain u32, where sign bit must not be + /* + * For emitting plain u32, where sign bit must not be * propagated LLVM tends to load imm64 over mov32 * directly, so save couple of bytes by just doing * 'mov %eax, imm32' instead. @@ -439,8 +413,6 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, { struct bpf_insn *insn = bpf_prog->insnsi; int insn_cnt = bpf_prog->len; - bool seen_ld_abs = ctx->seen_ld_abs | (oldproglen == 0); - bool seen_ax_reg = ctx->seen_ax_reg | (oldproglen == 0); bool seen_exit = false; u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY]; int i, cnt = 0; @@ -450,9 +422,6 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, emit_prologue(&prog, bpf_prog->aux->stack_depth, bpf_prog_was_classic(bpf_prog)); - if (seen_ld_abs) - emit_load_skb_data_hlen(&prog); - for (i = 0; i < insn_cnt; i++, insn++) { const s32 imm32 = insn->imm; u32 dst_reg = insn->dst_reg; @@ -460,13 +429,9 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 b2 = 0, b3 = 0; s64 jmp_offset; u8 jmp_cond; - bool reload_skb_data; int ilen; u8 *func; - if (dst_reg == BPF_REG_AX || src_reg == BPF_REG_AX) - ctx->seen_ax_reg = seen_ax_reg = true; - switch (insn->code) { /* ALU */ case BPF_ALU | BPF_ADD | BPF_X: @@ -525,7 +490,8 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, else if (is_ereg(dst_reg)) EMIT1(add_1mod(0x40, dst_reg)); - /* b3 holds 'normal' opcode, b2 short form only valid + /* + * b3 holds 'normal' opcode, b2 short form only valid * in case dst is eax/rax. */ switch (BPF_OP(insn->code)) { @@ -593,7 +559,8 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, /* mov rax, dst_reg */ EMIT_mov(BPF_REG_0, dst_reg); - /* xor edx, edx + /* + * xor edx, edx * equivalent to 'xor rdx, rdx', but one byte less */ EMIT2(0x31, 0xd2); @@ -655,7 +622,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, } break; } - /* shifts */ + /* Shifts */ case BPF_ALU | BPF_LSH | BPF_K: case BPF_ALU | BPF_RSH | BPF_K: case BPF_ALU | BPF_ARSH | BPF_K: @@ -686,7 +653,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, case BPF_ALU64 | BPF_RSH | BPF_X: case BPF_ALU64 | BPF_ARSH | BPF_X: - /* check for bad case when dst_reg == rcx */ + /* Check for bad case when dst_reg == rcx */ if (dst_reg == BPF_REG_4) { /* mov r11, dst_reg */ EMIT_mov(AUX_REG, dst_reg); @@ -724,13 +691,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, case BPF_ALU | BPF_END | BPF_FROM_BE: switch (imm32) { case 16: - /* emit 'ror %ax, 8' to swap lower 2 bytes */ + /* Emit 'ror %ax, 8' to swap lower 2 bytes */ EMIT1(0x66); if (is_ereg(dst_reg)) EMIT1(0x41); EMIT3(0xC1, add_1reg(0xC8, dst_reg), 8); - /* emit 'movzwl eax, ax' */ + /* Emit 'movzwl eax, ax' */ if (is_ereg(dst_reg)) EMIT3(0x45, 0x0F, 0xB7); else @@ -738,7 +705,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, EMIT1(add_2reg(0xC0, dst_reg, dst_reg)); break; case 32: - /* emit 'bswap eax' to swap lower 4 bytes */ + /* Emit 'bswap eax' to swap lower 4 bytes */ if (is_ereg(dst_reg)) EMIT2(0x41, 0x0F); else @@ -746,7 +713,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, EMIT1(add_1reg(0xC8, dst_reg)); break; case 64: - /* emit 'bswap rax' to swap 8 bytes */ + /* Emit 'bswap rax' to swap 8 bytes */ EMIT3(add_1mod(0x48, dst_reg), 0x0F, add_1reg(0xC8, dst_reg)); break; @@ -756,7 +723,8 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, case BPF_ALU | BPF_END | BPF_FROM_LE: switch (imm32) { case 16: - /* emit 'movzwl eax, ax' to zero extend 16-bit + /* + * Emit 'movzwl eax, ax' to zero extend 16-bit * into 64 bit */ if (is_ereg(dst_reg)) @@ -766,7 +734,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, EMIT1(add_2reg(0xC0, dst_reg, dst_reg)); break; case 32: - /* emit 'mov eax, eax' to clear upper 32-bits */ + /* Emit 'mov eax, eax' to clear upper 32-bits */ if (is_ereg(dst_reg)) EMIT1(0x45); EMIT2(0x89, add_2reg(0xC0, dst_reg, dst_reg)); @@ -809,9 +777,9 @@ st: if (is_imm8(insn->off)) /* STX: *(u8*)(dst_reg + off) = src_reg */ case BPF_STX | BPF_MEM | BPF_B: - /* emit 'mov byte ptr [rax + off], al' */ + /* Emit 'mov byte ptr [rax + off], al' */ if (is_ereg(dst_reg) || is_ereg(src_reg) || - /* have to add extra byte for x86 SIL, DIL regs */ + /* We have to add extra byte for x86 SIL, DIL regs */ src_reg == BPF_REG_1 || src_reg == BPF_REG_2) EMIT2(add_2mod(0x40, dst_reg, src_reg), 0x88); else @@ -840,25 +808,26 @@ stx: if (is_imm8(insn->off)) /* LDX: dst_reg = *(u8*)(src_reg + off) */ case BPF_LDX | BPF_MEM | BPF_B: - /* emit 'movzx rax, byte ptr [rax + off]' */ + /* Emit 'movzx rax, byte ptr [rax + off]' */ EMIT3(add_2mod(0x48, src_reg, dst_reg), 0x0F, 0xB6); goto ldx; case BPF_LDX | BPF_MEM | BPF_H: - /* emit 'movzx rax, word ptr [rax + off]' */ + /* Emit 'movzx rax, word ptr [rax + off]' */ EMIT3(add_2mod(0x48, src_reg, dst_reg), 0x0F, 0xB7); goto ldx; case BPF_LDX | BPF_MEM | BPF_W: - /* emit 'mov eax, dword ptr [rax+0x14]' */ + /* Emit 'mov eax, dword ptr [rax+0x14]' */ if (is_ereg(dst_reg) || is_ereg(src_reg)) EMIT2(add_2mod(0x40, src_reg, dst_reg), 0x8B); else EMIT1(0x8B); goto ldx; case BPF_LDX | BPF_MEM | BPF_DW: - /* emit 'mov rax, qword ptr [rax+0x14]' */ + /* Emit 'mov rax, qword ptr [rax+0x14]' */ EMIT2(add_2mod(0x48, src_reg, dst_reg), 0x8B); -ldx: /* if insn->off == 0 we can save one extra byte, but - * special case of x86 r13 which always needs an offset +ldx: /* + * If insn->off == 0 we can save one extra byte, but + * special case of x86 R13 which always needs an offset * is not worth the hassle */ if (is_imm8(insn->off)) @@ -870,7 +839,7 @@ ldx: /* if insn->off == 0 we can save one extra byte, but /* STX XADD: lock *(u32*)(dst_reg + off) += src_reg */ case BPF_STX | BPF_XADD | BPF_W: - /* emit 'lock add dword ptr [rax + off], eax' */ + /* Emit 'lock add dword ptr [rax + off], eax' */ if (is_ereg(dst_reg) || is_ereg(src_reg)) EMIT3(0xF0, add_2mod(0x40, dst_reg, src_reg), 0x01); else @@ -889,35 +858,12 @@ xadd: if (is_imm8(insn->off)) case BPF_JMP | BPF_CALL: func = (u8 *) __bpf_call_base + imm32; jmp_offset = func - (image + addrs[i]); - if (seen_ld_abs) { - reload_skb_data = bpf_helper_changes_pkt_data(func); - if (reload_skb_data) { - EMIT1(0x57); /* push %rdi */ - jmp_offset += 22; /* pop, mov, sub, mov */ - } else { - EMIT2(0x41, 0x52); /* push %r10 */ - EMIT2(0x41, 0x51); /* push %r9 */ - /* need to adjust jmp offset, since - * pop %r9, pop %r10 take 4 bytes after call insn - */ - jmp_offset += 4; - } - } if (!imm32 || !is_simm32(jmp_offset)) { - pr_err("unsupported bpf func %d addr %p image %p\n", + pr_err("unsupported BPF func %d addr %p image %p\n", imm32, func, image); return -EINVAL; } EMIT1_off32(0xE8, jmp_offset); - if (seen_ld_abs) { - if (reload_skb_data) { - EMIT1(0x5F); /* pop %rdi */ - emit_load_skb_data_hlen(&prog); - } else { - EMIT2(0x41, 0x59); /* pop %r9 */ - EMIT2(0x41, 0x5A); /* pop %r10 */ - } - } break; case BPF_JMP | BPF_TAIL_CALL: @@ -970,7 +916,7 @@ xadd: if (is_imm8(insn->off)) else EMIT2_off32(0x81, add_1reg(0xF8, dst_reg), imm32); -emit_cond_jmp: /* convert BPF opcode to x86 */ +emit_cond_jmp: /* Convert BPF opcode to x86 */ switch (BPF_OP(insn->code)) { case BPF_JEQ: jmp_cond = X86_JE; @@ -996,22 +942,22 @@ emit_cond_jmp: /* convert BPF opcode to x86 */ jmp_cond = X86_JBE; break; case BPF_JSGT: - /* signed '>', GT in x86 */ + /* Signed '>', GT in x86 */ jmp_cond = X86_JG; break; case BPF_JSLT: - /* signed '<', LT in x86 */ + /* Signed '<', LT in x86 */ jmp_cond = X86_JL; break; case BPF_JSGE: - /* signed '>=', GE in x86 */ + /* Signed '>=', GE in x86 */ jmp_cond = X86_JGE; break; case BPF_JSLE: - /* signed '<=', LE in x86 */ + /* Signed '<=', LE in x86 */ jmp_cond = X86_JLE; break; - default: /* to silence gcc warning */ + default: /* to silence GCC warning */ return -EFAULT; } jmp_offset = addrs[i + insn->off] - addrs[i]; @@ -1039,7 +985,7 @@ emit_cond_jmp: /* convert BPF opcode to x86 */ jmp_offset = addrs[i + insn->off] - addrs[i]; if (!jmp_offset) - /* optimize out nop jumps */ + /* Optimize out nop jumps */ break; emit_jmp: if (is_imm8(jmp_offset)) { @@ -1052,66 +998,13 @@ emit_jmp: } break; - case BPF_LD | BPF_IND | BPF_W: - func = sk_load_word; - goto common_load; - case BPF_LD | BPF_ABS | BPF_W: - func = CHOOSE_LOAD_FUNC(imm32, sk_load_word); -common_load: - ctx->seen_ld_abs = seen_ld_abs = true; - jmp_offset = func - (image + addrs[i]); - if (!func || !is_simm32(jmp_offset)) { - pr_err("unsupported bpf func %d addr %p image %p\n", - imm32, func, image); - return -EINVAL; - } - if (BPF_MODE(insn->code) == BPF_ABS) { - /* mov %esi, imm32 */ - EMIT1_off32(0xBE, imm32); - } else { - /* mov %rsi, src_reg */ - EMIT_mov(BPF_REG_2, src_reg); - if (imm32) { - if (is_imm8(imm32)) - /* add %esi, imm8 */ - EMIT3(0x83, 0xC6, imm32); - else - /* add %esi, imm32 */ - EMIT2_off32(0x81, 0xC6, imm32); - } - } - /* skb pointer is in R6 (%rbx), it will be copied into - * %rdi if skb_copy_bits() call is necessary. - * sk_load_* helpers also use %r10 and %r9d. - * See bpf_jit.S - */ - if (seen_ax_reg) - /* r10 = skb->data, mov %r10, off32(%rbx) */ - EMIT3_off32(0x4c, 0x8b, 0x93, - offsetof(struct sk_buff, data)); - EMIT1_off32(0xE8, jmp_offset); /* call */ - break; - - case BPF_LD | BPF_IND | BPF_H: - func = sk_load_half; - goto common_load; - case BPF_LD | BPF_ABS | BPF_H: - func = CHOOSE_LOAD_FUNC(imm32, sk_load_half); - goto common_load; - case BPF_LD | BPF_IND | BPF_B: - func = sk_load_byte; - goto common_load; - case BPF_LD | BPF_ABS | BPF_B: - func = CHOOSE_LOAD_FUNC(imm32, sk_load_byte); - goto common_load; - case BPF_JMP | BPF_EXIT: if (seen_exit) { jmp_offset = ctx->cleanup_addr - addrs[i]; goto emit_jmp; } seen_exit = true; - /* update cleanup_addr */ + /* Update cleanup_addr */ ctx->cleanup_addr = proglen; /* mov rbx, qword ptr [rbp+0] */ EMIT4(0x48, 0x8B, 0x5D, 0); @@ -1129,10 +1022,11 @@ common_load: break; default: - /* By design x64 JIT should support all BPF instructions + /* + * By design x86-64 JIT should support all BPF instructions. * This error will be seen if new instruction was added - * to interpreter, but not to JIT - * or if there is junk in bpf_prog + * to the interpreter, but not to the JIT, or if there is + * junk in bpf_prog. */ pr_err("bpf_jit: unknown opcode %02x\n", insn->code); return -EINVAL; @@ -1184,7 +1078,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) return orig_prog; tmp = bpf_jit_blind_constants(prog); - /* If blinding was requested and we failed during blinding, + /* + * If blinding was requested and we failed during blinding, * we must fall back to the interpreter. */ if (IS_ERR(tmp)) @@ -1212,14 +1107,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) extra_pass = true; goto skip_init_addrs; } - addrs = kmalloc(prog->len * sizeof(*addrs), GFP_KERNEL); + addrs = kmalloc_array(prog->len, sizeof(*addrs), GFP_KERNEL); if (!addrs) { prog = orig_prog; goto out_addrs; } - /* Before first pass, make a rough estimation of addrs[] - * each bpf instruction is translated to less than 64 bytes + /* + * Before first pass, make a rough estimation of addrs[] + * each BPF instruction is translated to less than 64 bytes */ for (proglen = 0, i = 0; i < prog->len; i++) { proglen += 64; @@ -1228,10 +1124,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) ctx.cleanup_addr = proglen; skip_init_addrs: - /* JITed image shrinks with every pass and the loop iterates - * until the image stops shrinking. Very large bpf programs + /* + * JITed image shrinks with every pass and the loop iterates + * until the image stops shrinking. Very large BPF programs * may converge on the last pass. In such case do one more - * pass to emit the final image + * pass to emit the final image. */ for (pass = 0; pass < 20 || image; pass++) { proglen = do_jit(prog, addrs, image, oldproglen, &ctx); diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c new file mode 100644 index 000000000000..55799873ebe5 --- /dev/null +++ b/arch/x86/net/bpf_jit_comp32.c @@ -0,0 +1,2419 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Just-In-Time compiler for eBPF filters on IA32 (32bit x86) + * + * Author: Wang YanQing (udknight@gmail.com) + * The code based on code and ideas from: + * Eric Dumazet (eric.dumazet@gmail.com) + * and from: + * Shubham Bansal + */ + +#include +#include +#include +#include +#include +#include +#include + +/* + * eBPF prog stack layout: + * + * high + * original ESP => +-----+ + * | | callee saved registers + * +-----+ + * | ... | eBPF JIT scratch space + * BPF_FP,IA32_EBP => +-----+ + * | ... | eBPF prog stack + * +-----+ + * |RSVD | JIT scratchpad + * current ESP => +-----+ + * | | + * | ... | Function call stack + * | | + * +-----+ + * low + * + * The callee saved registers: + * + * high + * original ESP => +------------------+ \ + * | ebp | | + * current EBP => +------------------+ } callee saved registers + * | ebx,esi,edi | | + * +------------------+ / + * low + */ + +static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len) +{ + if (len == 1) + *ptr = bytes; + else if (len == 2) + *(u16 *)ptr = bytes; + else { + *(u32 *)ptr = bytes; + barrier(); + } + return ptr + len; +} + +#define EMIT(bytes, len) \ + do { prog = emit_code(prog, bytes, len); cnt += len; } while (0) + +#define EMIT1(b1) EMIT(b1, 1) +#define EMIT2(b1, b2) EMIT((b1) + ((b2) << 8), 2) +#define EMIT3(b1, b2, b3) EMIT((b1) + ((b2) << 8) + ((b3) << 16), 3) +#define EMIT4(b1, b2, b3, b4) \ + EMIT((b1) + ((b2) << 8) + ((b3) << 16) + ((b4) << 24), 4) + +#define EMIT1_off32(b1, off) \ + do { EMIT1(b1); EMIT(off, 4); } while (0) +#define EMIT2_off32(b1, b2, off) \ + do { EMIT2(b1, b2); EMIT(off, 4); } while (0) +#define EMIT3_off32(b1, b2, b3, off) \ + do { EMIT3(b1, b2, b3); EMIT(off, 4); } while (0) +#define EMIT4_off32(b1, b2, b3, b4, off) \ + do { EMIT4(b1, b2, b3, b4); EMIT(off, 4); } while (0) + +#define jmp_label(label, jmp_insn_len) (label - cnt - jmp_insn_len) + +static bool is_imm8(int value) +{ + return value <= 127 && value >= -128; +} + +static bool is_simm32(s64 value) +{ + return value == (s64) (s32) value; +} + +#define STACK_OFFSET(k) (k) +#define TCALL_CNT (MAX_BPF_JIT_REG + 0) /* Tail Call Count */ + +#define IA32_EAX (0x0) +#define IA32_EBX (0x3) +#define IA32_ECX (0x1) +#define IA32_EDX (0x2) +#define IA32_ESI (0x6) +#define IA32_EDI (0x7) +#define IA32_EBP (0x5) +#define IA32_ESP (0x4) + +/* + * List of x86 cond jumps opcodes (. + s8) + * Add 0x10 (and an extra 0x0f) to generate far jumps (. + s32) + */ +#define IA32_JB 0x72 +#define IA32_JAE 0x73 +#define IA32_JE 0x74 +#define IA32_JNE 0x75 +#define IA32_JBE 0x76 +#define IA32_JA 0x77 +#define IA32_JL 0x7C +#define IA32_JGE 0x7D +#define IA32_JLE 0x7E +#define IA32_JG 0x7F + +/* + * Map eBPF registers to IA32 32bit registers or stack scratch space. + * + * 1. All the registers, R0-R10, are mapped to scratch space on stack. + * 2. We need two 64 bit temp registers to do complex operations on eBPF + * registers. + * 3. For performance reason, the BPF_REG_AX for blinding constant, is + * mapped to real hardware register pair, IA32_ESI and IA32_EDI. + * + * As the eBPF registers are all 64 bit registers and IA32 has only 32 bit + * registers, we have to map each eBPF registers with two IA32 32 bit regs + * or scratch memory space and we have to build eBPF 64 bit register from those. + * + * We use IA32_EAX, IA32_EDX, IA32_ECX, IA32_EBX as temporary registers. + */ +static const u8 bpf2ia32[][2] = { + /* Return value from in-kernel function, and exit value from eBPF */ + [BPF_REG_0] = {STACK_OFFSET(0), STACK_OFFSET(4)}, + + /* The arguments from eBPF program to in-kernel function */ + /* Stored on stack scratch space */ + [BPF_REG_1] = {STACK_OFFSET(8), STACK_OFFSET(12)}, + [BPF_REG_2] = {STACK_OFFSET(16), STACK_OFFSET(20)}, + [BPF_REG_3] = {STACK_OFFSET(24), STACK_OFFSET(28)}, + [BPF_REG_4] = {STACK_OFFSET(32), STACK_OFFSET(36)}, + [BPF_REG_5] = {STACK_OFFSET(40), STACK_OFFSET(44)}, + + /* Callee saved registers that in-kernel function will preserve */ + /* Stored on stack scratch space */ + [BPF_REG_6] = {STACK_OFFSET(48), STACK_OFFSET(52)}, + [BPF_REG_7] = {STACK_OFFSET(56), STACK_OFFSET(60)}, + [BPF_REG_8] = {STACK_OFFSET(64), STACK_OFFSET(68)}, + [BPF_REG_9] = {STACK_OFFSET(72), STACK_OFFSET(76)}, + + /* Read only Frame Pointer to access Stack */ + [BPF_REG_FP] = {STACK_OFFSET(80), STACK_OFFSET(84)}, + + /* Temporary register for blinding constants. */ + [BPF_REG_AX] = {IA32_ESI, IA32_EDI}, + + /* Tail call count. Stored on stack scratch space. */ + [TCALL_CNT] = {STACK_OFFSET(88), STACK_OFFSET(92)}, +}; + +#define dst_lo dst[0] +#define dst_hi dst[1] +#define src_lo src[0] +#define src_hi src[1] + +#define STACK_ALIGNMENT 8 +/* + * Stack space for BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, + * BPF_REG_5, BPF_REG_6, BPF_REG_7, BPF_REG_8, BPF_REG_9, + * BPF_REG_FP, BPF_REG_AX and Tail call counts. + */ +#define SCRATCH_SIZE 96 + +/* Total stack size used in JITed code */ +#define _STACK_SIZE (stack_depth + SCRATCH_SIZE) + +#define STACK_SIZE ALIGN(_STACK_SIZE, STACK_ALIGNMENT) + +/* Get the offset of eBPF REGISTERs stored on scratch space. */ +#define STACK_VAR(off) (off) + +/* Encode 'dst_reg' register into IA32 opcode 'byte' */ +static u8 add_1reg(u8 byte, u32 dst_reg) +{ + return byte + dst_reg; +} + +/* Encode 'dst_reg' and 'src_reg' registers into IA32 opcode 'byte' */ +static u8 add_2reg(u8 byte, u32 dst_reg, u32 src_reg) +{ + return byte + dst_reg + (src_reg << 3); +} + +static void jit_fill_hole(void *area, unsigned int size) +{ + /* Fill whole space with int3 instructions */ + memset(area, 0xcc, size); +} + +static inline void emit_ia32_mov_i(const u8 dst, const u32 val, bool dstk, + u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + + if (dstk) { + if (val == 0) { + /* xor eax,eax */ + EMIT2(0x33, add_2reg(0xC0, IA32_EAX, IA32_EAX)); + /* mov dword ptr [ebp+off],eax */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst)); + } else { + EMIT3_off32(0xC7, add_1reg(0x40, IA32_EBP), + STACK_VAR(dst), val); + } + } else { + if (val == 0) + EMIT2(0x33, add_2reg(0xC0, dst, dst)); + else + EMIT2_off32(0xC7, add_1reg(0xC0, dst), + val); + } + *pprog = prog; +} + +/* dst = imm (4 bytes)*/ +static inline void emit_ia32_mov_r(const u8 dst, const u8 src, bool dstk, + bool sstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u8 sreg = sstk ? IA32_EAX : src; + + if (sstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), STACK_VAR(src)); + if (dstk) + /* mov dword ptr [ebp+off],eax */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, sreg), STACK_VAR(dst)); + else + /* mov dst,sreg */ + EMIT2(0x89, add_2reg(0xC0, dst, sreg)); + + *pprog = prog; +} + +/* dst = src */ +static inline void emit_ia32_mov_r64(const bool is64, const u8 dst[], + const u8 src[], bool dstk, + bool sstk, u8 **pprog) +{ + emit_ia32_mov_r(dst_lo, src_lo, dstk, sstk, pprog); + if (is64) + /* complete 8 byte move */ + emit_ia32_mov_r(dst_hi, src_hi, dstk, sstk, pprog); + else + /* zero out high 4 bytes */ + emit_ia32_mov_i(dst_hi, 0, dstk, pprog); +} + +/* Sign extended move */ +static inline void emit_ia32_mov_i64(const bool is64, const u8 dst[], + const u32 val, bool dstk, u8 **pprog) +{ + u32 hi = 0; + + if (is64 && (val & (1<<31))) + hi = (u32)~0; + emit_ia32_mov_i(dst_lo, val, dstk, pprog); + emit_ia32_mov_i(dst_hi, hi, dstk, pprog); +} + +/* + * ALU operation (32 bit) + * dst = dst * src + */ +static inline void emit_ia32_mul_r(const u8 dst, const u8 src, bool dstk, + bool sstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u8 sreg = sstk ? IA32_ECX : src; + + if (sstk) + /* mov ecx,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), STACK_VAR(src)); + + if (dstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), STACK_VAR(dst)); + else + /* mov eax,dst */ + EMIT2(0x8B, add_2reg(0xC0, dst, IA32_EAX)); + + + EMIT2(0xF7, add_1reg(0xE0, sreg)); + + if (dstk) + /* mov dword ptr [ebp+off],eax */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst)); + else + /* mov dst,eax */ + EMIT2(0x89, add_2reg(0xC0, dst, IA32_EAX)); + + *pprog = prog; +} + +static inline void emit_ia32_to_le_r64(const u8 dst[], s32 val, + bool dstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + + if (dstk && val != 64) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + switch (val) { + case 16: + /* + * Emit 'movzwl eax,ax' to zero extend 16-bit + * into 64 bit + */ + EMIT2(0x0F, 0xB7); + EMIT1(add_2reg(0xC0, dreg_lo, dreg_lo)); + /* xor dreg_hi,dreg_hi */ + EMIT2(0x33, add_2reg(0xC0, dreg_hi, dreg_hi)); + break; + case 32: + /* xor dreg_hi,dreg_hi */ + EMIT2(0x33, add_2reg(0xC0, dreg_hi, dreg_hi)); + break; + case 64: + /* nop */ + break; + } + + if (dstk && val != 64) { + /* mov dword ptr [ebp+off],dreg_lo */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_lo), + STACK_VAR(dst_lo)); + /* mov dword ptr [ebp+off],dreg_hi */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_hi), + STACK_VAR(dst_hi)); + } + *pprog = prog; +} + +static inline void emit_ia32_to_be_r64(const u8 dst[], s32 val, + bool dstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + switch (val) { + case 16: + /* Emit 'ror %ax, 8' to swap lower 2 bytes */ + EMIT1(0x66); + EMIT3(0xC1, add_1reg(0xC8, dreg_lo), 8); + + EMIT2(0x0F, 0xB7); + EMIT1(add_2reg(0xC0, dreg_lo, dreg_lo)); + + /* xor dreg_hi,dreg_hi */ + EMIT2(0x33, add_2reg(0xC0, dreg_hi, dreg_hi)); + break; + case 32: + /* Emit 'bswap eax' to swap lower 4 bytes */ + EMIT1(0x0F); + EMIT1(add_1reg(0xC8, dreg_lo)); + + /* xor dreg_hi,dreg_hi */ + EMIT2(0x33, add_2reg(0xC0, dreg_hi, dreg_hi)); + break; + case 64: + /* Emit 'bswap eax' to swap lower 4 bytes */ + EMIT1(0x0F); + EMIT1(add_1reg(0xC8, dreg_lo)); + + /* Emit 'bswap edx' to swap lower 4 bytes */ + EMIT1(0x0F); + EMIT1(add_1reg(0xC8, dreg_hi)); + + /* mov ecx,dreg_hi */ + EMIT2(0x89, add_2reg(0xC0, IA32_ECX, dreg_hi)); + /* mov dreg_hi,dreg_lo */ + EMIT2(0x89, add_2reg(0xC0, dreg_hi, dreg_lo)); + /* mov dreg_lo,ecx */ + EMIT2(0x89, add_2reg(0xC0, dreg_lo, IA32_ECX)); + + break; + } + if (dstk) { + /* mov dword ptr [ebp+off],dreg_lo */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_lo), + STACK_VAR(dst_lo)); + /* mov dword ptr [ebp+off],dreg_hi */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_hi), + STACK_VAR(dst_hi)); + } + *pprog = prog; +} + +/* + * ALU operation (32 bit) + * dst = dst (div|mod) src + */ +static inline void emit_ia32_div_mod_r(const u8 op, const u8 dst, const u8 src, + bool dstk, bool sstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + + if (sstk) + /* mov ecx,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), + STACK_VAR(src)); + else if (src != IA32_ECX) + /* mov ecx,src */ + EMIT2(0x8B, add_2reg(0xC0, src, IA32_ECX)); + + if (dstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst)); + else + /* mov eax,dst */ + EMIT2(0x8B, add_2reg(0xC0, dst, IA32_EAX)); + + /* xor edx,edx */ + EMIT2(0x31, add_2reg(0xC0, IA32_EDX, IA32_EDX)); + /* div ecx */ + EMIT2(0xF7, add_1reg(0xF0, IA32_ECX)); + + if (op == BPF_MOD) { + if (dstk) + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst)); + else + EMIT2(0x89, add_2reg(0xC0, dst, IA32_EDX)); + } else { + if (dstk) + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst)); + else + EMIT2(0x89, add_2reg(0xC0, dst, IA32_EAX)); + } + *pprog = prog; +} + +/* + * ALU operation (32 bit) + * dst = dst (shift) src + */ +static inline void emit_ia32_shift_r(const u8 op, const u8 dst, const u8 src, + bool dstk, bool sstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u8 dreg = dstk ? IA32_EAX : dst; + u8 b2; + + if (dstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), STACK_VAR(dst)); + + if (sstk) + /* mov ecx,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), STACK_VAR(src)); + else if (src != IA32_ECX) + /* mov ecx,src */ + EMIT2(0x8B, add_2reg(0xC0, src, IA32_ECX)); + + switch (op) { + case BPF_LSH: + b2 = 0xE0; break; + case BPF_RSH: + b2 = 0xE8; break; + case BPF_ARSH: + b2 = 0xF8; break; + default: + return; + } + EMIT2(0xD3, add_1reg(b2, dreg)); + + if (dstk) + /* mov dword ptr [ebp+off],dreg */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg), STACK_VAR(dst)); + *pprog = prog; +} + +/* + * ALU operation (32 bit) + * dst = dst (op) src + */ +static inline void emit_ia32_alu_r(const bool is64, const bool hi, const u8 op, + const u8 dst, const u8 src, bool dstk, + bool sstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u8 sreg = sstk ? IA32_EAX : src; + u8 dreg = dstk ? IA32_EDX : dst; + + if (sstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), STACK_VAR(src)); + + if (dstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), STACK_VAR(dst)); + + switch (BPF_OP(op)) { + /* dst = dst + src */ + case BPF_ADD: + if (hi && is64) + EMIT2(0x11, add_2reg(0xC0, dreg, sreg)); + else + EMIT2(0x01, add_2reg(0xC0, dreg, sreg)); + break; + /* dst = dst - src */ + case BPF_SUB: + if (hi && is64) + EMIT2(0x19, add_2reg(0xC0, dreg, sreg)); + else + EMIT2(0x29, add_2reg(0xC0, dreg, sreg)); + break; + /* dst = dst | src */ + case BPF_OR: + EMIT2(0x09, add_2reg(0xC0, dreg, sreg)); + break; + /* dst = dst & src */ + case BPF_AND: + EMIT2(0x21, add_2reg(0xC0, dreg, sreg)); + break; + /* dst = dst ^ src */ + case BPF_XOR: + EMIT2(0x31, add_2reg(0xC0, dreg, sreg)); + break; + } + + if (dstk) + /* mov dword ptr [ebp+off],dreg */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg), + STACK_VAR(dst)); + *pprog = prog; +} + +/* ALU operation (64 bit) */ +static inline void emit_ia32_alu_r64(const bool is64, const u8 op, + const u8 dst[], const u8 src[], + bool dstk, bool sstk, + u8 **pprog) +{ + u8 *prog = *pprog; + + emit_ia32_alu_r(is64, false, op, dst_lo, src_lo, dstk, sstk, &prog); + if (is64) + emit_ia32_alu_r(is64, true, op, dst_hi, src_hi, dstk, sstk, + &prog); + else + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + *pprog = prog; +} + +/* + * ALU operation (32 bit) + * dst = dst (op) val + */ +static inline void emit_ia32_alu_i(const bool is64, const bool hi, const u8 op, + const u8 dst, const s32 val, bool dstk, + u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u8 dreg = dstk ? IA32_EAX : dst; + u8 sreg = IA32_EDX; + + if (dstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), STACK_VAR(dst)); + + if (!is_imm8(val)) + /* mov edx,imm32*/ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_EDX), val); + + switch (op) { + /* dst = dst + val */ + case BPF_ADD: + if (hi && is64) { + if (is_imm8(val)) + EMIT3(0x83, add_1reg(0xD0, dreg), val); + else + EMIT2(0x11, add_2reg(0xC0, dreg, sreg)); + } else { + if (is_imm8(val)) + EMIT3(0x83, add_1reg(0xC0, dreg), val); + else + EMIT2(0x01, add_2reg(0xC0, dreg, sreg)); + } + break; + /* dst = dst - val */ + case BPF_SUB: + if (hi && is64) { + if (is_imm8(val)) + EMIT3(0x83, add_1reg(0xD8, dreg), val); + else + EMIT2(0x19, add_2reg(0xC0, dreg, sreg)); + } else { + if (is_imm8(val)) + EMIT3(0x83, add_1reg(0xE8, dreg), val); + else + EMIT2(0x29, add_2reg(0xC0, dreg, sreg)); + } + break; + /* dst = dst | val */ + case BPF_OR: + if (is_imm8(val)) + EMIT3(0x83, add_1reg(0xC8, dreg), val); + else + EMIT2(0x09, add_2reg(0xC0, dreg, sreg)); + break; + /* dst = dst & val */ + case BPF_AND: + if (is_imm8(val)) + EMIT3(0x83, add_1reg(0xE0, dreg), val); + else + EMIT2(0x21, add_2reg(0xC0, dreg, sreg)); + break; + /* dst = dst ^ val */ + case BPF_XOR: + if (is_imm8(val)) + EMIT3(0x83, add_1reg(0xF0, dreg), val); + else + EMIT2(0x31, add_2reg(0xC0, dreg, sreg)); + break; + case BPF_NEG: + EMIT2(0xF7, add_1reg(0xD8, dreg)); + break; + } + + if (dstk) + /* mov dword ptr [ebp+off],dreg */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg), + STACK_VAR(dst)); + *pprog = prog; +} + +/* ALU operation (64 bit) */ +static inline void emit_ia32_alu_i64(const bool is64, const u8 op, + const u8 dst[], const u32 val, + bool dstk, u8 **pprog) +{ + u8 *prog = *pprog; + u32 hi = 0; + + if (is64 && (val & (1<<31))) + hi = (u32)~0; + + emit_ia32_alu_i(is64, false, op, dst_lo, val, dstk, &prog); + if (is64) + emit_ia32_alu_i(is64, true, op, dst_hi, hi, dstk, &prog); + else + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + + *pprog = prog; +} + +/* dst = ~dst (64 bit) */ +static inline void emit_ia32_neg64(const u8 dst[], bool dstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + + /* xor ecx,ecx */ + EMIT2(0x31, add_2reg(0xC0, IA32_ECX, IA32_ECX)); + /* sub dreg_lo,ecx */ + EMIT2(0x2B, add_2reg(0xC0, dreg_lo, IA32_ECX)); + /* mov dreg_lo,ecx */ + EMIT2(0x89, add_2reg(0xC0, dreg_lo, IA32_ECX)); + + /* xor ecx,ecx */ + EMIT2(0x31, add_2reg(0xC0, IA32_ECX, IA32_ECX)); + /* sbb dreg_hi,ecx */ + EMIT2(0x19, add_2reg(0xC0, dreg_hi, IA32_ECX)); + /* mov dreg_hi,ecx */ + EMIT2(0x89, add_2reg(0xC0, dreg_hi, IA32_ECX)); + + if (dstk) { + /* mov dword ptr [ebp+off],dreg_lo */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_lo), + STACK_VAR(dst_lo)); + /* mov dword ptr [ebp+off],dreg_hi */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_hi), + STACK_VAR(dst_hi)); + } + *pprog = prog; +} + +/* dst = dst << src */ +static inline void emit_ia32_lsh_r64(const u8 dst[], const u8 src[], + bool dstk, bool sstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + static int jmp_label1 = -1; + static int jmp_label2 = -1; + static int jmp_label3 = -1; + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + + if (sstk) + /* mov ecx,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), + STACK_VAR(src_lo)); + else + /* mov ecx,src_lo */ + EMIT2(0x8B, add_2reg(0xC0, src_lo, IA32_ECX)); + + /* cmp ecx,32 */ + EMIT3(0x83, add_1reg(0xF8, IA32_ECX), 32); + /* Jumps when >= 32 */ + if (is_imm8(jmp_label(jmp_label1, 2))) + EMIT2(IA32_JAE, jmp_label(jmp_label1, 2)); + else + EMIT2_off32(0x0F, IA32_JAE + 0x10, jmp_label(jmp_label1, 6)); + + /* < 32 */ + /* shl dreg_hi,cl */ + EMIT2(0xD3, add_1reg(0xE0, dreg_hi)); + /* mov ebx,dreg_lo */ + EMIT2(0x8B, add_2reg(0xC0, dreg_lo, IA32_EBX)); + /* shl dreg_lo,cl */ + EMIT2(0xD3, add_1reg(0xE0, dreg_lo)); + + /* IA32_ECX = -IA32_ECX + 32 */ + /* neg ecx */ + EMIT2(0xF7, add_1reg(0xD8, IA32_ECX)); + /* add ecx,32 */ + EMIT3(0x83, add_1reg(0xC0, IA32_ECX), 32); + + /* shr ebx,cl */ + EMIT2(0xD3, add_1reg(0xE8, IA32_EBX)); + /* or dreg_hi,ebx */ + EMIT2(0x09, add_2reg(0xC0, dreg_hi, IA32_EBX)); + + /* goto out; */ + if (is_imm8(jmp_label(jmp_label3, 2))) + EMIT2(0xEB, jmp_label(jmp_label3, 2)); + else + EMIT1_off32(0xE9, jmp_label(jmp_label3, 5)); + + /* >= 32 */ + if (jmp_label1 == -1) + jmp_label1 = cnt; + + /* cmp ecx,64 */ + EMIT3(0x83, add_1reg(0xF8, IA32_ECX), 64); + /* Jumps when >= 64 */ + if (is_imm8(jmp_label(jmp_label2, 2))) + EMIT2(IA32_JAE, jmp_label(jmp_label2, 2)); + else + EMIT2_off32(0x0F, IA32_JAE + 0x10, jmp_label(jmp_label2, 6)); + + /* >= 32 && < 64 */ + /* sub ecx,32 */ + EMIT3(0x83, add_1reg(0xE8, IA32_ECX), 32); + /* shl dreg_lo,cl */ + EMIT2(0xD3, add_1reg(0xE0, dreg_lo)); + /* mov dreg_hi,dreg_lo */ + EMIT2(0x89, add_2reg(0xC0, dreg_hi, dreg_lo)); + + /* xor dreg_lo,dreg_lo */ + EMIT2(0x33, add_2reg(0xC0, dreg_lo, dreg_lo)); + + /* goto out; */ + if (is_imm8(jmp_label(jmp_label3, 2))) + EMIT2(0xEB, jmp_label(jmp_label3, 2)); + else + EMIT1_off32(0xE9, jmp_label(jmp_label3, 5)); + + /* >= 64 */ + if (jmp_label2 == -1) + jmp_label2 = cnt; + /* xor dreg_lo,dreg_lo */ + EMIT2(0x33, add_2reg(0xC0, dreg_lo, dreg_lo)); + /* xor dreg_hi,dreg_hi */ + EMIT2(0x33, add_2reg(0xC0, dreg_hi, dreg_hi)); + + if (jmp_label3 == -1) + jmp_label3 = cnt; + + if (dstk) { + /* mov dword ptr [ebp+off],dreg_lo */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_lo), + STACK_VAR(dst_lo)); + /* mov dword ptr [ebp+off],dreg_hi */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_hi), + STACK_VAR(dst_hi)); + } + /* out: */ + *pprog = prog; +} + +/* dst = dst >> src (signed)*/ +static inline void emit_ia32_arsh_r64(const u8 dst[], const u8 src[], + bool dstk, bool sstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + static int jmp_label1 = -1; + static int jmp_label2 = -1; + static int jmp_label3 = -1; + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + + if (sstk) + /* mov ecx,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), + STACK_VAR(src_lo)); + else + /* mov ecx,src_lo */ + EMIT2(0x8B, add_2reg(0xC0, src_lo, IA32_ECX)); + + /* cmp ecx,32 */ + EMIT3(0x83, add_1reg(0xF8, IA32_ECX), 32); + /* Jumps when >= 32 */ + if (is_imm8(jmp_label(jmp_label1, 2))) + EMIT2(IA32_JAE, jmp_label(jmp_label1, 2)); + else + EMIT2_off32(0x0F, IA32_JAE + 0x10, jmp_label(jmp_label1, 6)); + + /* < 32 */ + /* lshr dreg_lo,cl */ + EMIT2(0xD3, add_1reg(0xE8, dreg_lo)); + /* mov ebx,dreg_hi */ + EMIT2(0x8B, add_2reg(0xC0, dreg_hi, IA32_EBX)); + /* ashr dreg_hi,cl */ + EMIT2(0xD3, add_1reg(0xF8, dreg_hi)); + + /* IA32_ECX = -IA32_ECX + 32 */ + /* neg ecx */ + EMIT2(0xF7, add_1reg(0xD8, IA32_ECX)); + /* add ecx,32 */ + EMIT3(0x83, add_1reg(0xC0, IA32_ECX), 32); + + /* shl ebx,cl */ + EMIT2(0xD3, add_1reg(0xE0, IA32_EBX)); + /* or dreg_lo,ebx */ + EMIT2(0x09, add_2reg(0xC0, dreg_lo, IA32_EBX)); + + /* goto out; */ + if (is_imm8(jmp_label(jmp_label3, 2))) + EMIT2(0xEB, jmp_label(jmp_label3, 2)); + else + EMIT1_off32(0xE9, jmp_label(jmp_label3, 5)); + + /* >= 32 */ + if (jmp_label1 == -1) + jmp_label1 = cnt; + + /* cmp ecx,64 */ + EMIT3(0x83, add_1reg(0xF8, IA32_ECX), 64); + /* Jumps when >= 64 */ + if (is_imm8(jmp_label(jmp_label2, 2))) + EMIT2(IA32_JAE, jmp_label(jmp_label2, 2)); + else + EMIT2_off32(0x0F, IA32_JAE + 0x10, jmp_label(jmp_label2, 6)); + + /* >= 32 && < 64 */ + /* sub ecx,32 */ + EMIT3(0x83, add_1reg(0xE8, IA32_ECX), 32); + /* ashr dreg_hi,cl */ + EMIT2(0xD3, add_1reg(0xF8, dreg_hi)); + /* mov dreg_lo,dreg_hi */ + EMIT2(0x89, add_2reg(0xC0, dreg_lo, dreg_hi)); + + /* ashr dreg_hi,imm8 */ + EMIT3(0xC1, add_1reg(0xF8, dreg_hi), 31); + + /* goto out; */ + if (is_imm8(jmp_label(jmp_label3, 2))) + EMIT2(0xEB, jmp_label(jmp_label3, 2)); + else + EMIT1_off32(0xE9, jmp_label(jmp_label3, 5)); + + /* >= 64 */ + if (jmp_label2 == -1) + jmp_label2 = cnt; + /* ashr dreg_hi,imm8 */ + EMIT3(0xC1, add_1reg(0xF8, dreg_hi), 31); + /* mov dreg_lo,dreg_hi */ + EMIT2(0x89, add_2reg(0xC0, dreg_lo, dreg_hi)); + + if (jmp_label3 == -1) + jmp_label3 = cnt; + + if (dstk) { + /* mov dword ptr [ebp+off],dreg_lo */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_lo), + STACK_VAR(dst_lo)); + /* mov dword ptr [ebp+off],dreg_hi */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_hi), + STACK_VAR(dst_hi)); + } + /* out: */ + *pprog = prog; +} + +/* dst = dst >> src */ +static inline void emit_ia32_rsh_r64(const u8 dst[], const u8 src[], bool dstk, + bool sstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + static int jmp_label1 = -1; + static int jmp_label2 = -1; + static int jmp_label3 = -1; + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + + if (sstk) + /* mov ecx,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), + STACK_VAR(src_lo)); + else + /* mov ecx,src_lo */ + EMIT2(0x8B, add_2reg(0xC0, src_lo, IA32_ECX)); + + /* cmp ecx,32 */ + EMIT3(0x83, add_1reg(0xF8, IA32_ECX), 32); + /* Jumps when >= 32 */ + if (is_imm8(jmp_label(jmp_label1, 2))) + EMIT2(IA32_JAE, jmp_label(jmp_label1, 2)); + else + EMIT2_off32(0x0F, IA32_JAE + 0x10, jmp_label(jmp_label1, 6)); + + /* < 32 */ + /* lshr dreg_lo,cl */ + EMIT2(0xD3, add_1reg(0xE8, dreg_lo)); + /* mov ebx,dreg_hi */ + EMIT2(0x8B, add_2reg(0xC0, dreg_hi, IA32_EBX)); + /* shr dreg_hi,cl */ + EMIT2(0xD3, add_1reg(0xE8, dreg_hi)); + + /* IA32_ECX = -IA32_ECX + 32 */ + /* neg ecx */ + EMIT2(0xF7, add_1reg(0xD8, IA32_ECX)); + /* add ecx,32 */ + EMIT3(0x83, add_1reg(0xC0, IA32_ECX), 32); + + /* shl ebx,cl */ + EMIT2(0xD3, add_1reg(0xE0, IA32_EBX)); + /* or dreg_lo,ebx */ + EMIT2(0x09, add_2reg(0xC0, dreg_lo, IA32_EBX)); + + /* goto out; */ + if (is_imm8(jmp_label(jmp_label3, 2))) + EMIT2(0xEB, jmp_label(jmp_label3, 2)); + else + EMIT1_off32(0xE9, jmp_label(jmp_label3, 5)); + + /* >= 32 */ + if (jmp_label1 == -1) + jmp_label1 = cnt; + /* cmp ecx,64 */ + EMIT3(0x83, add_1reg(0xF8, IA32_ECX), 64); + /* Jumps when >= 64 */ + if (is_imm8(jmp_label(jmp_label2, 2))) + EMIT2(IA32_JAE, jmp_label(jmp_label2, 2)); + else + EMIT2_off32(0x0F, IA32_JAE + 0x10, jmp_label(jmp_label2, 6)); + + /* >= 32 && < 64 */ + /* sub ecx,32 */ + EMIT3(0x83, add_1reg(0xE8, IA32_ECX), 32); + /* shr dreg_hi,cl */ + EMIT2(0xD3, add_1reg(0xE8, dreg_hi)); + /* mov dreg_lo,dreg_hi */ + EMIT2(0x89, add_2reg(0xC0, dreg_lo, dreg_hi)); + /* xor dreg_hi,dreg_hi */ + EMIT2(0x33, add_2reg(0xC0, dreg_hi, dreg_hi)); + + /* goto out; */ + if (is_imm8(jmp_label(jmp_label3, 2))) + EMIT2(0xEB, jmp_label(jmp_label3, 2)); + else + EMIT1_off32(0xE9, jmp_label(jmp_label3, 5)); + + /* >= 64 */ + if (jmp_label2 == -1) + jmp_label2 = cnt; + /* xor dreg_lo,dreg_lo */ + EMIT2(0x33, add_2reg(0xC0, dreg_lo, dreg_lo)); + /* xor dreg_hi,dreg_hi */ + EMIT2(0x33, add_2reg(0xC0, dreg_hi, dreg_hi)); + + if (jmp_label3 == -1) + jmp_label3 = cnt; + + if (dstk) { + /* mov dword ptr [ebp+off],dreg_lo */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_lo), + STACK_VAR(dst_lo)); + /* mov dword ptr [ebp+off],dreg_hi */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_hi), + STACK_VAR(dst_hi)); + } + /* out: */ + *pprog = prog; +} + +/* dst = dst << val */ +static inline void emit_ia32_lsh_i64(const u8 dst[], const u32 val, + bool dstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + /* Do LSH operation */ + if (val < 32) { + /* shl dreg_hi,imm8 */ + EMIT3(0xC1, add_1reg(0xE0, dreg_hi), val); + /* mov ebx,dreg_lo */ + EMIT2(0x8B, add_2reg(0xC0, dreg_lo, IA32_EBX)); + /* shl dreg_lo,imm8 */ + EMIT3(0xC1, add_1reg(0xE0, dreg_lo), val); + + /* IA32_ECX = 32 - val */ + /* mov ecx,val */ + EMIT2(0xB1, val); + /* movzx ecx,ecx */ + EMIT3(0x0F, 0xB6, add_2reg(0xC0, IA32_ECX, IA32_ECX)); + /* neg ecx */ + EMIT2(0xF7, add_1reg(0xD8, IA32_ECX)); + /* add ecx,32 */ + EMIT3(0x83, add_1reg(0xC0, IA32_ECX), 32); + + /* shr ebx,cl */ + EMIT2(0xD3, add_1reg(0xE8, IA32_EBX)); + /* or dreg_hi,ebx */ + EMIT2(0x09, add_2reg(0xC0, dreg_hi, IA32_EBX)); + } else if (val >= 32 && val < 64) { + u32 value = val - 32; + + /* shl dreg_lo,imm8 */ + EMIT3(0xC1, add_1reg(0xE0, dreg_lo), value); + /* mov dreg_hi,dreg_lo */ + EMIT2(0x89, add_2reg(0xC0, dreg_hi, dreg_lo)); + /* xor dreg_lo,dreg_lo */ + EMIT2(0x33, add_2reg(0xC0, dreg_lo, dreg_lo)); + } else { + /* xor dreg_lo,dreg_lo */ + EMIT2(0x33, add_2reg(0xC0, dreg_lo, dreg_lo)); + /* xor dreg_hi,dreg_hi */ + EMIT2(0x33, add_2reg(0xC0, dreg_hi, dreg_hi)); + } + + if (dstk) { + /* mov dword ptr [ebp+off],dreg_lo */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_lo), + STACK_VAR(dst_lo)); + /* mov dword ptr [ebp+off],dreg_hi */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_hi), + STACK_VAR(dst_hi)); + } + *pprog = prog; +} + +/* dst = dst >> val */ +static inline void emit_ia32_rsh_i64(const u8 dst[], const u32 val, + bool dstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + + /* Do RSH operation */ + if (val < 32) { + /* shr dreg_lo,imm8 */ + EMIT3(0xC1, add_1reg(0xE8, dreg_lo), val); + /* mov ebx,dreg_hi */ + EMIT2(0x8B, add_2reg(0xC0, dreg_hi, IA32_EBX)); + /* shr dreg_hi,imm8 */ + EMIT3(0xC1, add_1reg(0xE8, dreg_hi), val); + + /* IA32_ECX = 32 - val */ + /* mov ecx,val */ + EMIT2(0xB1, val); + /* movzx ecx,ecx */ + EMIT3(0x0F, 0xB6, add_2reg(0xC0, IA32_ECX, IA32_ECX)); + /* neg ecx */ + EMIT2(0xF7, add_1reg(0xD8, IA32_ECX)); + /* add ecx,32 */ + EMIT3(0x83, add_1reg(0xC0, IA32_ECX), 32); + + /* shl ebx,cl */ + EMIT2(0xD3, add_1reg(0xE0, IA32_EBX)); + /* or dreg_lo,ebx */ + EMIT2(0x09, add_2reg(0xC0, dreg_lo, IA32_EBX)); + } else if (val >= 32 && val < 64) { + u32 value = val - 32; + + /* shr dreg_hi,imm8 */ + EMIT3(0xC1, add_1reg(0xE8, dreg_hi), value); + /* mov dreg_lo,dreg_hi */ + EMIT2(0x89, add_2reg(0xC0, dreg_lo, dreg_hi)); + /* xor dreg_hi,dreg_hi */ + EMIT2(0x33, add_2reg(0xC0, dreg_hi, dreg_hi)); + } else { + /* xor dreg_lo,dreg_lo */ + EMIT2(0x33, add_2reg(0xC0, dreg_lo, dreg_lo)); + /* xor dreg_hi,dreg_hi */ + EMIT2(0x33, add_2reg(0xC0, dreg_hi, dreg_hi)); + } + + if (dstk) { + /* mov dword ptr [ebp+off],dreg_lo */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_lo), + STACK_VAR(dst_lo)); + /* mov dword ptr [ebp+off],dreg_hi */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_hi), + STACK_VAR(dst_hi)); + } + *pprog = prog; +} + +/* dst = dst >> val (signed) */ +static inline void emit_ia32_arsh_i64(const u8 dst[], const u32 val, + bool dstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + /* Do RSH operation */ + if (val < 32) { + /* shr dreg_lo,imm8 */ + EMIT3(0xC1, add_1reg(0xE8, dreg_lo), val); + /* mov ebx,dreg_hi */ + EMIT2(0x8B, add_2reg(0xC0, dreg_hi, IA32_EBX)); + /* ashr dreg_hi,imm8 */ + EMIT3(0xC1, add_1reg(0xF8, dreg_hi), val); + + /* IA32_ECX = 32 - val */ + /* mov ecx,val */ + EMIT2(0xB1, val); + /* movzx ecx,ecx */ + EMIT3(0x0F, 0xB6, add_2reg(0xC0, IA32_ECX, IA32_ECX)); + /* neg ecx */ + EMIT2(0xF7, add_1reg(0xD8, IA32_ECX)); + /* add ecx,32 */ + EMIT3(0x83, add_1reg(0xC0, IA32_ECX), 32); + + /* shl ebx,cl */ + EMIT2(0xD3, add_1reg(0xE0, IA32_EBX)); + /* or dreg_lo,ebx */ + EMIT2(0x09, add_2reg(0xC0, dreg_lo, IA32_EBX)); + } else if (val >= 32 && val < 64) { + u32 value = val - 32; + + /* ashr dreg_hi,imm8 */ + EMIT3(0xC1, add_1reg(0xF8, dreg_hi), value); + /* mov dreg_lo,dreg_hi */ + EMIT2(0x89, add_2reg(0xC0, dreg_lo, dreg_hi)); + + /* ashr dreg_hi,imm8 */ + EMIT3(0xC1, add_1reg(0xF8, dreg_hi), 31); + } else { + /* ashr dreg_hi,imm8 */ + EMIT3(0xC1, add_1reg(0xF8, dreg_hi), 31); + /* mov dreg_lo,dreg_hi */ + EMIT2(0x89, add_2reg(0xC0, dreg_lo, dreg_hi)); + } + + if (dstk) { + /* mov dword ptr [ebp+off],dreg_lo */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_lo), + STACK_VAR(dst_lo)); + /* mov dword ptr [ebp+off],dreg_hi */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, dreg_hi), + STACK_VAR(dst_hi)); + } + *pprog = prog; +} + +static inline void emit_ia32_mul_r64(const u8 dst[], const u8 src[], bool dstk, + bool sstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + + if (dstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_hi)); + else + /* mov eax,dst_hi */ + EMIT2(0x8B, add_2reg(0xC0, dst_hi, IA32_EAX)); + + if (sstk) + /* mul dword ptr [ebp+off] */ + EMIT3(0xF7, add_1reg(0x60, IA32_EBP), STACK_VAR(src_lo)); + else + /* mul src_lo */ + EMIT2(0xF7, add_1reg(0xE0, src_lo)); + + /* mov ecx,eax */ + EMIT2(0x89, add_2reg(0xC0, IA32_ECX, IA32_EAX)); + + if (dstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + else + /* mov eax,dst_lo */ + EMIT2(0x8B, add_2reg(0xC0, dst_lo, IA32_EAX)); + + if (sstk) + /* mul dword ptr [ebp+off] */ + EMIT3(0xF7, add_1reg(0x60, IA32_EBP), STACK_VAR(src_hi)); + else + /* mul src_hi */ + EMIT2(0xF7, add_1reg(0xE0, src_hi)); + + /* add eax,eax */ + EMIT2(0x01, add_2reg(0xC0, IA32_ECX, IA32_EAX)); + + if (dstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + else + /* mov eax,dst_lo */ + EMIT2(0x8B, add_2reg(0xC0, dst_lo, IA32_EAX)); + + if (sstk) + /* mul dword ptr [ebp+off] */ + EMIT3(0xF7, add_1reg(0x60, IA32_EBP), STACK_VAR(src_lo)); + else + /* mul src_lo */ + EMIT2(0xF7, add_1reg(0xE0, src_lo)); + + /* add ecx,edx */ + EMIT2(0x01, add_2reg(0xC0, IA32_ECX, IA32_EDX)); + + if (dstk) { + /* mov dword ptr [ebp+off],eax */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + /* mov dword ptr [ebp+off],ecx */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_ECX), + STACK_VAR(dst_hi)); + } else { + /* mov dst_lo,eax */ + EMIT2(0x89, add_2reg(0xC0, dst_lo, IA32_EAX)); + /* mov dst_hi,ecx */ + EMIT2(0x89, add_2reg(0xC0, dst_hi, IA32_ECX)); + } + + *pprog = prog; +} + +static inline void emit_ia32_mul_i64(const u8 dst[], const u32 val, + bool dstk, u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + u32 hi; + + hi = val & (1<<31) ? (u32)~0 : 0; + /* movl eax,imm32 */ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_EAX), val); + if (dstk) + /* mul dword ptr [ebp+off] */ + EMIT3(0xF7, add_1reg(0x60, IA32_EBP), STACK_VAR(dst_hi)); + else + /* mul dst_hi */ + EMIT2(0xF7, add_1reg(0xE0, dst_hi)); + + /* mov ecx,eax */ + EMIT2(0x89, add_2reg(0xC0, IA32_ECX, IA32_EAX)); + + /* movl eax,imm32 */ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_EAX), hi); + if (dstk) + /* mul dword ptr [ebp+off] */ + EMIT3(0xF7, add_1reg(0x60, IA32_EBP), STACK_VAR(dst_lo)); + else + /* mul dst_lo */ + EMIT2(0xF7, add_1reg(0xE0, dst_lo)); + /* add ecx,eax */ + EMIT2(0x01, add_2reg(0xC0, IA32_ECX, IA32_EAX)); + + /* movl eax,imm32 */ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_EAX), val); + if (dstk) + /* mul dword ptr [ebp+off] */ + EMIT3(0xF7, add_1reg(0x60, IA32_EBP), STACK_VAR(dst_lo)); + else + /* mul dst_lo */ + EMIT2(0xF7, add_1reg(0xE0, dst_lo)); + + /* add ecx,edx */ + EMIT2(0x01, add_2reg(0xC0, IA32_ECX, IA32_EDX)); + + if (dstk) { + /* mov dword ptr [ebp+off],eax */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + /* mov dword ptr [ebp+off],ecx */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_ECX), + STACK_VAR(dst_hi)); + } else { + /* mov dword ptr [ebp+off],eax */ + EMIT2(0x89, add_2reg(0xC0, dst_lo, IA32_EAX)); + /* mov dword ptr [ebp+off],ecx */ + EMIT2(0x89, add_2reg(0xC0, dst_hi, IA32_ECX)); + } + + *pprog = prog; +} + +static int bpf_size_to_x86_bytes(int bpf_size) +{ + if (bpf_size == BPF_W) + return 4; + else if (bpf_size == BPF_H) + return 2; + else if (bpf_size == BPF_B) + return 1; + else if (bpf_size == BPF_DW) + return 4; /* imm32 */ + else + return 0; +} + +struct jit_context { + int cleanup_addr; /* Epilogue code offset */ +}; + +/* Maximum number of bytes emitted while JITing one eBPF insn */ +#define BPF_MAX_INSN_SIZE 128 +#define BPF_INSN_SAFETY 64 + +#define PROLOGUE_SIZE 35 + +/* + * Emit prologue code for BPF program and check it's size. + * bpf_tail_call helper will skip it while jumping into another program. + */ +static void emit_prologue(u8 **pprog, u32 stack_depth) +{ + u8 *prog = *pprog; + int cnt = 0; + const u8 *r1 = bpf2ia32[BPF_REG_1]; + const u8 fplo = bpf2ia32[BPF_REG_FP][0]; + const u8 fphi = bpf2ia32[BPF_REG_FP][1]; + const u8 *tcc = bpf2ia32[TCALL_CNT]; + + /* push ebp */ + EMIT1(0x55); + /* mov ebp,esp */ + EMIT2(0x89, 0xE5); + /* push edi */ + EMIT1(0x57); + /* push esi */ + EMIT1(0x56); + /* push ebx */ + EMIT1(0x53); + + /* sub esp,STACK_SIZE */ + EMIT2_off32(0x81, 0xEC, STACK_SIZE); + /* sub ebp,SCRATCH_SIZE+4+12*/ + EMIT3(0x83, add_1reg(0xE8, IA32_EBP), SCRATCH_SIZE + 16); + /* xor ebx,ebx */ + EMIT2(0x31, add_2reg(0xC0, IA32_EBX, IA32_EBX)); + + /* Set up BPF prog stack base register */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EBP), STACK_VAR(fplo)); + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EBX), STACK_VAR(fphi)); + + /* Move BPF_CTX (EAX) to BPF_REG_R1 */ + /* mov dword ptr [ebp+off],eax */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EAX), STACK_VAR(r1[0])); + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EBX), STACK_VAR(r1[1])); + + /* Initialize Tail Count */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EBX), STACK_VAR(tcc[0])); + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EBX), STACK_VAR(tcc[1])); + + BUILD_BUG_ON(cnt != PROLOGUE_SIZE); + *pprog = prog; +} + +/* Emit epilogue code for BPF program */ +static void emit_epilogue(u8 **pprog, u32 stack_depth) +{ + u8 *prog = *pprog; + const u8 *r0 = bpf2ia32[BPF_REG_0]; + int cnt = 0; + + /* mov eax,dword ptr [ebp+off]*/ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), STACK_VAR(r0[0])); + /* mov edx,dword ptr [ebp+off]*/ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), STACK_VAR(r0[1])); + + /* add ebp,SCRATCH_SIZE+4+12*/ + EMIT3(0x83, add_1reg(0xC0, IA32_EBP), SCRATCH_SIZE + 16); + + /* mov ebx,dword ptr [ebp-12]*/ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EBX), -12); + /* mov esi,dword ptr [ebp-8]*/ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ESI), -8); + /* mov edi,dword ptr [ebp-4]*/ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDI), -4); + + EMIT1(0xC9); /* leave */ + EMIT1(0xC3); /* ret */ + *pprog = prog; +} + +/* + * Generate the following code: + * ... bpf_tail_call(void *ctx, struct bpf_array *array, u64 index) ... + * if (index >= array->map.max_entries) + * goto out; + * if (++tail_call_cnt > MAX_TAIL_CALL_CNT) + * goto out; + * prog = array->ptrs[index]; + * if (prog == NULL) + * goto out; + * goto *(prog->bpf_func + prologue_size); + * out: + */ +static void emit_bpf_tail_call(u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + const u8 *r1 = bpf2ia32[BPF_REG_1]; + const u8 *r2 = bpf2ia32[BPF_REG_2]; + const u8 *r3 = bpf2ia32[BPF_REG_3]; + const u8 *tcc = bpf2ia32[TCALL_CNT]; + u32 lo, hi; + static int jmp_label1 = -1; + + /* + * if (index >= array->map.max_entries) + * goto out; + */ + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), STACK_VAR(r2[0])); + /* mov edx,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), STACK_VAR(r3[0])); + + /* cmp dword ptr [eax+off],edx */ + EMIT3(0x39, add_2reg(0x40, IA32_EAX, IA32_EDX), + offsetof(struct bpf_array, map.max_entries)); + /* jbe out */ + EMIT2(IA32_JBE, jmp_label(jmp_label1, 2)); + + /* + * if (tail_call_cnt > MAX_TAIL_CALL_CNT) + * goto out; + */ + lo = (u32)MAX_TAIL_CALL_CNT; + hi = (u32)((u64)MAX_TAIL_CALL_CNT >> 32); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), STACK_VAR(tcc[0])); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EBX), STACK_VAR(tcc[1])); + + /* cmp edx,hi */ + EMIT3(0x83, add_1reg(0xF8, IA32_EBX), hi); + EMIT2(IA32_JNE, 3); + /* cmp ecx,lo */ + EMIT3(0x83, add_1reg(0xF8, IA32_ECX), lo); + + /* ja out */ + EMIT2(IA32_JAE, jmp_label(jmp_label1, 2)); + + /* add eax,0x1 */ + EMIT3(0x83, add_1reg(0xC0, IA32_ECX), 0x01); + /* adc ebx,0x0 */ + EMIT3(0x83, add_1reg(0xD0, IA32_EBX), 0x00); + + /* mov dword ptr [ebp+off],eax */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_ECX), STACK_VAR(tcc[0])); + /* mov dword ptr [ebp+off],edx */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EBX), STACK_VAR(tcc[1])); + + /* prog = array->ptrs[index]; */ + /* mov edx, [eax + edx * 4 + offsetof(...)] */ + EMIT3_off32(0x8B, 0x94, 0x90, offsetof(struct bpf_array, ptrs)); + + /* + * if (prog == NULL) + * goto out; + */ + /* test edx,edx */ + EMIT2(0x85, add_2reg(0xC0, IA32_EDX, IA32_EDX)); + /* je out */ + EMIT2(IA32_JE, jmp_label(jmp_label1, 2)); + + /* goto *(prog->bpf_func + prologue_size); */ + /* mov edx, dword ptr [edx + 32] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EDX, IA32_EDX), + offsetof(struct bpf_prog, bpf_func)); + /* add edx,prologue_size */ + EMIT3(0x83, add_1reg(0xC0, IA32_EDX), PROLOGUE_SIZE); + + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), STACK_VAR(r1[0])); + + /* + * Now we're ready to jump into next BPF program: + * eax == ctx (1st arg) + * edx == prog->bpf_func + prologue_size + */ + RETPOLINE_EDX_BPF_JIT(); + + if (jmp_label1 == -1) + jmp_label1 = cnt; + + /* out: */ + *pprog = prog; +} + +/* Push the scratch stack register on top of the stack. */ +static inline void emit_push_r64(const u8 src[], u8 **pprog) +{ + u8 *prog = *pprog; + int cnt = 0; + + /* mov ecx,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), STACK_VAR(src_hi)); + /* push ecx */ + EMIT1(0x51); + + /* mov ecx,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), STACK_VAR(src_lo)); + /* push ecx */ + EMIT1(0x51); + + *pprog = prog; +} + +static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, + int oldproglen, struct jit_context *ctx) +{ + struct bpf_insn *insn = bpf_prog->insnsi; + int insn_cnt = bpf_prog->len; + bool seen_exit = false; + u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY]; + int i, cnt = 0; + int proglen = 0; + u8 *prog = temp; + + emit_prologue(&prog, bpf_prog->aux->stack_depth); + + for (i = 0; i < insn_cnt; i++, insn++) { + const s32 imm32 = insn->imm; + const bool is64 = BPF_CLASS(insn->code) == BPF_ALU64; + const bool dstk = insn->dst_reg == BPF_REG_AX ? false : true; + const bool sstk = insn->src_reg == BPF_REG_AX ? false : true; + const u8 code = insn->code; + const u8 *dst = bpf2ia32[insn->dst_reg]; + const u8 *src = bpf2ia32[insn->src_reg]; + const u8 *r0 = bpf2ia32[BPF_REG_0]; + s64 jmp_offset; + u8 jmp_cond; + int ilen; + u8 *func; + + switch (code) { + /* ALU operations */ + /* dst = src */ + case BPF_ALU | BPF_MOV | BPF_K: + case BPF_ALU | BPF_MOV | BPF_X: + case BPF_ALU64 | BPF_MOV | BPF_K: + case BPF_ALU64 | BPF_MOV | BPF_X: + switch (BPF_SRC(code)) { + case BPF_X: + emit_ia32_mov_r64(is64, dst, src, dstk, + sstk, &prog); + break; + case BPF_K: + /* Sign-extend immediate value to dst reg */ + emit_ia32_mov_i64(is64, dst, imm32, + dstk, &prog); + break; + } + break; + /* dst = dst + src/imm */ + /* dst = dst - src/imm */ + /* dst = dst | src/imm */ + /* dst = dst & src/imm */ + /* dst = dst ^ src/imm */ + /* dst = dst * src/imm */ + /* dst = dst << src */ + /* dst = dst >> src */ + case BPF_ALU | BPF_ADD | BPF_K: + case BPF_ALU | BPF_ADD | BPF_X: + case BPF_ALU | BPF_SUB | BPF_K: + case BPF_ALU | BPF_SUB | BPF_X: + case BPF_ALU | BPF_OR | BPF_K: + case BPF_ALU | BPF_OR | BPF_X: + case BPF_ALU | BPF_AND | BPF_K: + case BPF_ALU | BPF_AND | BPF_X: + case BPF_ALU | BPF_XOR | BPF_K: + case BPF_ALU | BPF_XOR | BPF_X: + case BPF_ALU64 | BPF_ADD | BPF_K: + case BPF_ALU64 | BPF_ADD | BPF_X: + case BPF_ALU64 | BPF_SUB | BPF_K: + case BPF_ALU64 | BPF_SUB | BPF_X: + case BPF_ALU64 | BPF_OR | BPF_K: + case BPF_ALU64 | BPF_OR | BPF_X: + case BPF_ALU64 | BPF_AND | BPF_K: + case BPF_ALU64 | BPF_AND | BPF_X: + case BPF_ALU64 | BPF_XOR | BPF_K: + case BPF_ALU64 | BPF_XOR | BPF_X: + switch (BPF_SRC(code)) { + case BPF_X: + emit_ia32_alu_r64(is64, BPF_OP(code), dst, + src, dstk, sstk, &prog); + break; + case BPF_K: + emit_ia32_alu_i64(is64, BPF_OP(code), dst, + imm32, dstk, &prog); + break; + } + break; + case BPF_ALU | BPF_MUL | BPF_K: + case BPF_ALU | BPF_MUL | BPF_X: + switch (BPF_SRC(code)) { + case BPF_X: + emit_ia32_mul_r(dst_lo, src_lo, dstk, + sstk, &prog); + break; + case BPF_K: + /* mov ecx,imm32*/ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_ECX), + imm32); + emit_ia32_mul_r(dst_lo, IA32_ECX, dstk, + false, &prog); + break; + } + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + break; + case BPF_ALU | BPF_LSH | BPF_X: + case BPF_ALU | BPF_RSH | BPF_X: + case BPF_ALU | BPF_ARSH | BPF_K: + case BPF_ALU | BPF_ARSH | BPF_X: + switch (BPF_SRC(code)) { + case BPF_X: + emit_ia32_shift_r(BPF_OP(code), dst_lo, src_lo, + dstk, sstk, &prog); + break; + case BPF_K: + /* mov ecx,imm32*/ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_ECX), + imm32); + emit_ia32_shift_r(BPF_OP(code), dst_lo, + IA32_ECX, dstk, false, + &prog); + break; + } + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + break; + /* dst = dst / src(imm) */ + /* dst = dst % src(imm) */ + case BPF_ALU | BPF_DIV | BPF_K: + case BPF_ALU | BPF_DIV | BPF_X: + case BPF_ALU | BPF_MOD | BPF_K: + case BPF_ALU | BPF_MOD | BPF_X: + switch (BPF_SRC(code)) { + case BPF_X: + emit_ia32_div_mod_r(BPF_OP(code), dst_lo, + src_lo, dstk, sstk, &prog); + break; + case BPF_K: + /* mov ecx,imm32*/ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_ECX), + imm32); + emit_ia32_div_mod_r(BPF_OP(code), dst_lo, + IA32_ECX, dstk, false, + &prog); + break; + } + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + break; + case BPF_ALU64 | BPF_DIV | BPF_K: + case BPF_ALU64 | BPF_DIV | BPF_X: + case BPF_ALU64 | BPF_MOD | BPF_K: + case BPF_ALU64 | BPF_MOD | BPF_X: + goto notyet; + /* dst = dst >> imm */ + /* dst = dst << imm */ + case BPF_ALU | BPF_RSH | BPF_K: + case BPF_ALU | BPF_LSH | BPF_K: + if (unlikely(imm32 > 31)) + return -EINVAL; + /* mov ecx,imm32*/ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_ECX), imm32); + emit_ia32_shift_r(BPF_OP(code), dst_lo, IA32_ECX, dstk, + false, &prog); + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + break; + /* dst = dst << imm */ + case BPF_ALU64 | BPF_LSH | BPF_K: + if (unlikely(imm32 > 63)) + return -EINVAL; + emit_ia32_lsh_i64(dst, imm32, dstk, &prog); + break; + /* dst = dst >> imm */ + case BPF_ALU64 | BPF_RSH | BPF_K: + if (unlikely(imm32 > 63)) + return -EINVAL; + emit_ia32_rsh_i64(dst, imm32, dstk, &prog); + break; + /* dst = dst << src */ + case BPF_ALU64 | BPF_LSH | BPF_X: + emit_ia32_lsh_r64(dst, src, dstk, sstk, &prog); + break; + /* dst = dst >> src */ + case BPF_ALU64 | BPF_RSH | BPF_X: + emit_ia32_rsh_r64(dst, src, dstk, sstk, &prog); + break; + /* dst = dst >> src (signed) */ + case BPF_ALU64 | BPF_ARSH | BPF_X: + emit_ia32_arsh_r64(dst, src, dstk, sstk, &prog); + break; + /* dst = dst >> imm (signed) */ + case BPF_ALU64 | BPF_ARSH | BPF_K: + if (unlikely(imm32 > 63)) + return -EINVAL; + emit_ia32_arsh_i64(dst, imm32, dstk, &prog); + break; + /* dst = ~dst */ + case BPF_ALU | BPF_NEG: + emit_ia32_alu_i(is64, false, BPF_OP(code), + dst_lo, 0, dstk, &prog); + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + break; + /* dst = ~dst (64 bit) */ + case BPF_ALU64 | BPF_NEG: + emit_ia32_neg64(dst, dstk, &prog); + break; + /* dst = dst * src/imm */ + case BPF_ALU64 | BPF_MUL | BPF_X: + case BPF_ALU64 | BPF_MUL | BPF_K: + switch (BPF_SRC(code)) { + case BPF_X: + emit_ia32_mul_r64(dst, src, dstk, sstk, &prog); + break; + case BPF_K: + emit_ia32_mul_i64(dst, imm32, dstk, &prog); + break; + } + break; + /* dst = htole(dst) */ + case BPF_ALU | BPF_END | BPF_FROM_LE: + emit_ia32_to_le_r64(dst, imm32, dstk, &prog); + break; + /* dst = htobe(dst) */ + case BPF_ALU | BPF_END | BPF_FROM_BE: + emit_ia32_to_be_r64(dst, imm32, dstk, &prog); + break; + /* dst = imm64 */ + case BPF_LD | BPF_IMM | BPF_DW: { + s32 hi, lo = imm32; + + hi = insn[1].imm; + emit_ia32_mov_i(dst_lo, lo, dstk, &prog); + emit_ia32_mov_i(dst_hi, hi, dstk, &prog); + insn++; + i++; + break; + } + /* ST: *(u8*)(dst_reg + off) = imm */ + case BPF_ST | BPF_MEM | BPF_H: + case BPF_ST | BPF_MEM | BPF_B: + case BPF_ST | BPF_MEM | BPF_W: + case BPF_ST | BPF_MEM | BPF_DW: + if (dstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + else + /* mov eax,dst_lo */ + EMIT2(0x8B, add_2reg(0xC0, dst_lo, IA32_EAX)); + + switch (BPF_SIZE(code)) { + case BPF_B: + EMIT(0xC6, 1); break; + case BPF_H: + EMIT2(0x66, 0xC7); break; + case BPF_W: + case BPF_DW: + EMIT(0xC7, 1); break; + } + + if (is_imm8(insn->off)) + EMIT2(add_1reg(0x40, IA32_EAX), insn->off); + else + EMIT1_off32(add_1reg(0x80, IA32_EAX), + insn->off); + EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(code))); + + if (BPF_SIZE(code) == BPF_DW) { + u32 hi; + + hi = imm32 & (1<<31) ? (u32)~0 : 0; + EMIT2_off32(0xC7, add_1reg(0x80, IA32_EAX), + insn->off + 4); + EMIT(hi, 4); + } + break; + + /* STX: *(u8*)(dst_reg + off) = src_reg */ + case BPF_STX | BPF_MEM | BPF_B: + case BPF_STX | BPF_MEM | BPF_H: + case BPF_STX | BPF_MEM | BPF_W: + case BPF_STX | BPF_MEM | BPF_DW: + if (dstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + else + /* mov eax,dst_lo */ + EMIT2(0x8B, add_2reg(0xC0, dst_lo, IA32_EAX)); + + if (sstk) + /* mov edx,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(src_lo)); + else + /* mov edx,src_lo */ + EMIT2(0x8B, add_2reg(0xC0, src_lo, IA32_EDX)); + + switch (BPF_SIZE(code)) { + case BPF_B: + EMIT(0x88, 1); break; + case BPF_H: + EMIT2(0x66, 0x89); break; + case BPF_W: + case BPF_DW: + EMIT(0x89, 1); break; + } + + if (is_imm8(insn->off)) + EMIT2(add_2reg(0x40, IA32_EAX, IA32_EDX), + insn->off); + else + EMIT1_off32(add_2reg(0x80, IA32_EAX, IA32_EDX), + insn->off); + + if (BPF_SIZE(code) == BPF_DW) { + if (sstk) + /* mov edi,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, + IA32_EDX), + STACK_VAR(src_hi)); + else + /* mov edi,src_hi */ + EMIT2(0x8B, add_2reg(0xC0, src_hi, + IA32_EDX)); + EMIT1(0x89); + if (is_imm8(insn->off + 4)) { + EMIT2(add_2reg(0x40, IA32_EAX, + IA32_EDX), + insn->off + 4); + } else { + EMIT1(add_2reg(0x80, IA32_EAX, + IA32_EDX)); + EMIT(insn->off + 4, 4); + } + } + break; + + /* LDX: dst_reg = *(u8*)(src_reg + off) */ + case BPF_LDX | BPF_MEM | BPF_B: + case BPF_LDX | BPF_MEM | BPF_H: + case BPF_LDX | BPF_MEM | BPF_W: + case BPF_LDX | BPF_MEM | BPF_DW: + if (sstk) + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(src_lo)); + else + /* mov eax,dword ptr [ebp+off] */ + EMIT2(0x8B, add_2reg(0xC0, src_lo, IA32_EAX)); + + switch (BPF_SIZE(code)) { + case BPF_B: + EMIT2(0x0F, 0xB6); break; + case BPF_H: + EMIT2(0x0F, 0xB7); break; + case BPF_W: + case BPF_DW: + EMIT(0x8B, 1); break; + } + + if (is_imm8(insn->off)) + EMIT2(add_2reg(0x40, IA32_EAX, IA32_EDX), + insn->off); + else + EMIT1_off32(add_2reg(0x80, IA32_EAX, IA32_EDX), + insn->off); + + if (dstk) + /* mov dword ptr [ebp+off],edx */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_lo)); + else + /* mov dst_lo,edx */ + EMIT2(0x89, add_2reg(0xC0, dst_lo, IA32_EDX)); + switch (BPF_SIZE(code)) { + case BPF_B: + case BPF_H: + case BPF_W: + if (dstk) { + EMIT3(0xC7, add_1reg(0x40, IA32_EBP), + STACK_VAR(dst_hi)); + EMIT(0x0, 4); + } else { + EMIT3(0xC7, add_1reg(0xC0, dst_hi), 0); + } + break; + case BPF_DW: + EMIT2_off32(0x8B, + add_2reg(0x80, IA32_EAX, IA32_EDX), + insn->off + 4); + if (dstk) + EMIT3(0x89, + add_2reg(0x40, IA32_EBP, + IA32_EDX), + STACK_VAR(dst_hi)); + else + EMIT2(0x89, + add_2reg(0xC0, dst_hi, IA32_EDX)); + break; + default: + break; + } + break; + /* call */ + case BPF_JMP | BPF_CALL: + { + const u8 *r1 = bpf2ia32[BPF_REG_1]; + const u8 *r2 = bpf2ia32[BPF_REG_2]; + const u8 *r3 = bpf2ia32[BPF_REG_3]; + const u8 *r4 = bpf2ia32[BPF_REG_4]; + const u8 *r5 = bpf2ia32[BPF_REG_5]; + + if (insn->src_reg == BPF_PSEUDO_CALL) + goto notyet; + + func = (u8 *) __bpf_call_base + imm32; + jmp_offset = func - (image + addrs[i]); + + if (!imm32 || !is_simm32(jmp_offset)) { + pr_err("unsupported BPF func %d addr %p image %p\n", + imm32, func, image); + return -EINVAL; + } + + /* mov eax,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(r1[0])); + /* mov edx,dword ptr [ebp+off] */ + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(r1[1])); + + emit_push_r64(r5, &prog); + emit_push_r64(r4, &prog); + emit_push_r64(r3, &prog); + emit_push_r64(r2, &prog); + + EMIT1_off32(0xE8, jmp_offset + 9); + + /* mov dword ptr [ebp+off],eax */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(r0[0])); + /* mov dword ptr [ebp+off],edx */ + EMIT3(0x89, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(r0[1])); + + /* add esp,32 */ + EMIT3(0x83, add_1reg(0xC0, IA32_ESP), 32); + break; + } + case BPF_JMP | BPF_TAIL_CALL: + emit_bpf_tail_call(&prog); + break; + + /* cond jump */ + case BPF_JMP | BPF_JEQ | BPF_X: + case BPF_JMP | BPF_JNE | BPF_X: + case BPF_JMP | BPF_JGT | BPF_X: + case BPF_JMP | BPF_JLT | BPF_X: + case BPF_JMP | BPF_JGE | BPF_X: + case BPF_JMP | BPF_JLE | BPF_X: + case BPF_JMP | BPF_JSGT | BPF_X: + case BPF_JMP | BPF_JSLE | BPF_X: + case BPF_JMP | BPF_JSLT | BPF_X: + case BPF_JMP | BPF_JSGE | BPF_X: { + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + u8 sreg_lo = sstk ? IA32_ECX : src_lo; + u8 sreg_hi = sstk ? IA32_EBX : src_hi; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + + if (sstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), + STACK_VAR(src_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EBX), + STACK_VAR(src_hi)); + } + + /* cmp dreg_hi,sreg_hi */ + EMIT2(0x39, add_2reg(0xC0, dreg_hi, sreg_hi)); + EMIT2(IA32_JNE, 2); + /* cmp dreg_lo,sreg_lo */ + EMIT2(0x39, add_2reg(0xC0, dreg_lo, sreg_lo)); + goto emit_cond_jmp; + } + case BPF_JMP | BPF_JSET | BPF_X: { + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + u8 sreg_lo = sstk ? IA32_ECX : src_lo; + u8 sreg_hi = sstk ? IA32_EBX : src_hi; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + + if (sstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_ECX), + STACK_VAR(src_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EBX), + STACK_VAR(src_hi)); + } + /* and dreg_lo,sreg_lo */ + EMIT2(0x23, add_2reg(0xC0, sreg_lo, dreg_lo)); + /* and dreg_hi,sreg_hi */ + EMIT2(0x23, add_2reg(0xC0, sreg_hi, dreg_hi)); + /* or dreg_lo,dreg_hi */ + EMIT2(0x09, add_2reg(0xC0, dreg_lo, dreg_hi)); + goto emit_cond_jmp; + } + case BPF_JMP | BPF_JSET | BPF_K: { + u32 hi; + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + u8 sreg_lo = IA32_ECX; + u8 sreg_hi = IA32_EBX; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + hi = imm32 & (1<<31) ? (u32)~0 : 0; + + /* mov ecx,imm32 */ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_ECX), imm32); + /* mov ebx,imm32 */ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_EBX), hi); + + /* and dreg_lo,sreg_lo */ + EMIT2(0x23, add_2reg(0xC0, sreg_lo, dreg_lo)); + /* and dreg_hi,sreg_hi */ + EMIT2(0x23, add_2reg(0xC0, sreg_hi, dreg_hi)); + /* or dreg_lo,dreg_hi */ + EMIT2(0x09, add_2reg(0xC0, dreg_lo, dreg_hi)); + goto emit_cond_jmp; + } + case BPF_JMP | BPF_JEQ | BPF_K: + case BPF_JMP | BPF_JNE | BPF_K: + case BPF_JMP | BPF_JGT | BPF_K: + case BPF_JMP | BPF_JLT | BPF_K: + case BPF_JMP | BPF_JGE | BPF_K: + case BPF_JMP | BPF_JLE | BPF_K: + case BPF_JMP | BPF_JSGT | BPF_K: + case BPF_JMP | BPF_JSLE | BPF_K: + case BPF_JMP | BPF_JSLT | BPF_K: + case BPF_JMP | BPF_JSGE | BPF_K: { + u32 hi; + u8 dreg_lo = dstk ? IA32_EAX : dst_lo; + u8 dreg_hi = dstk ? IA32_EDX : dst_hi; + u8 sreg_lo = IA32_ECX; + u8 sreg_hi = IA32_EBX; + + if (dstk) { + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EAX), + STACK_VAR(dst_lo)); + EMIT3(0x8B, add_2reg(0x40, IA32_EBP, IA32_EDX), + STACK_VAR(dst_hi)); + } + + hi = imm32 & (1<<31) ? (u32)~0 : 0; + /* mov ecx,imm32 */ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_ECX), imm32); + /* mov ebx,imm32 */ + EMIT2_off32(0xC7, add_1reg(0xC0, IA32_EBX), hi); + + /* cmp dreg_hi,sreg_hi */ + EMIT2(0x39, add_2reg(0xC0, dreg_hi, sreg_hi)); + EMIT2(IA32_JNE, 2); + /* cmp dreg_lo,sreg_lo */ + EMIT2(0x39, add_2reg(0xC0, dreg_lo, sreg_lo)); + +emit_cond_jmp: /* Convert BPF opcode to x86 */ + switch (BPF_OP(code)) { + case BPF_JEQ: + jmp_cond = IA32_JE; + break; + case BPF_JSET: + case BPF_JNE: + jmp_cond = IA32_JNE; + break; + case BPF_JGT: + /* GT is unsigned '>', JA in x86 */ + jmp_cond = IA32_JA; + break; + case BPF_JLT: + /* LT is unsigned '<', JB in x86 */ + jmp_cond = IA32_JB; + break; + case BPF_JGE: + /* GE is unsigned '>=', JAE in x86 */ + jmp_cond = IA32_JAE; + break; + case BPF_JLE: + /* LE is unsigned '<=', JBE in x86 */ + jmp_cond = IA32_JBE; + break; + case BPF_JSGT: + /* Signed '>', GT in x86 */ + jmp_cond = IA32_JG; + break; + case BPF_JSLT: + /* Signed '<', LT in x86 */ + jmp_cond = IA32_JL; + break; + case BPF_JSGE: + /* Signed '>=', GE in x86 */ + jmp_cond = IA32_JGE; + break; + case BPF_JSLE: + /* Signed '<=', LE in x86 */ + jmp_cond = IA32_JLE; + break; + default: /* to silence GCC warning */ + return -EFAULT; + } + jmp_offset = addrs[i + insn->off] - addrs[i]; + if (is_imm8(jmp_offset)) { + EMIT2(jmp_cond, jmp_offset); + } else if (is_simm32(jmp_offset)) { + EMIT2_off32(0x0F, jmp_cond + 0x10, jmp_offset); + } else { + pr_err("cond_jmp gen bug %llx\n", jmp_offset); + return -EFAULT; + } + + break; + } + case BPF_JMP | BPF_JA: + if (insn->off == -1) + /* -1 jmp instructions will always jump + * backwards two bytes. Explicitly handling + * this case avoids wasting too many passes + * when there are long sequences of replaced + * dead code. + */ + jmp_offset = -2; + else + jmp_offset = addrs[i + insn->off] - addrs[i]; + + if (!jmp_offset) + /* Optimize out nop jumps */ + break; +emit_jmp: + if (is_imm8(jmp_offset)) { + EMIT2(0xEB, jmp_offset); + } else if (is_simm32(jmp_offset)) { + EMIT1_off32(0xE9, jmp_offset); + } else { + pr_err("jmp gen bug %llx\n", jmp_offset); + return -EFAULT; + } + break; + /* STX XADD: lock *(u32 *)(dst + off) += src */ + case BPF_STX | BPF_XADD | BPF_W: + /* STX XADD: lock *(u64 *)(dst + off) += src */ + case BPF_STX | BPF_XADD | BPF_DW: + goto notyet; + case BPF_JMP | BPF_EXIT: + if (seen_exit) { + jmp_offset = ctx->cleanup_addr - addrs[i]; + goto emit_jmp; + } + seen_exit = true; + /* Update cleanup_addr */ + ctx->cleanup_addr = proglen; + emit_epilogue(&prog, bpf_prog->aux->stack_depth); + break; +notyet: + pr_info_once("*** NOT YET: opcode %02x ***\n", code); + return -EFAULT; + default: + /* + * This error will be seen if new instruction was added + * to interpreter, but not to JIT or if there is junk in + * bpf_prog + */ + pr_err("bpf_jit: unknown opcode %02x\n", code); + return -EINVAL; + } + + ilen = prog - temp; + if (ilen > BPF_MAX_INSN_SIZE) { + pr_err("bpf_jit: fatal insn size error\n"); + return -EFAULT; + } + + if (image) { + if (unlikely(proglen + ilen > oldproglen)) { + pr_err("bpf_jit: fatal error\n"); + return -EFAULT; + } + memcpy(image + proglen, temp, ilen); + } + proglen += ilen; + addrs[i] = proglen; + prog = temp; + } + return proglen; +} + +struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) +{ + struct bpf_binary_header *header = NULL; + struct bpf_prog *tmp, *orig_prog = prog; + int proglen, oldproglen = 0; + struct jit_context ctx = {}; + bool tmp_blinded = false; + u8 *image = NULL; + int *addrs; + int pass; + int i; + + if (!prog->jit_requested) + return orig_prog; + + tmp = bpf_jit_blind_constants(prog); + /* + * If blinding was requested and we failed during blinding, + * we must fall back to the interpreter. + */ + if (IS_ERR(tmp)) + return orig_prog; + if (tmp != prog) { + tmp_blinded = true; + prog = tmp; + } + + addrs = kmalloc_array(prog->len, sizeof(*addrs), GFP_KERNEL); + if (!addrs) { + prog = orig_prog; + goto out; + } + + /* + * Before first pass, make a rough estimation of addrs[] + * each BPF instruction is translated to less than 64 bytes + */ + for (proglen = 0, i = 0; i < prog->len; i++) { + proglen += 64; + addrs[i] = proglen; + } + ctx.cleanup_addr = proglen; + + /* + * JITed image shrinks with every pass and the loop iterates + * until the image stops shrinking. Very large BPF programs + * may converge on the last pass. In such case do one more + * pass to emit the final image. + */ + for (pass = 0; pass < 20 || image; pass++) { + proglen = do_jit(prog, addrs, image, oldproglen, &ctx); + if (proglen <= 0) { +out_image: + image = NULL; + if (header) + bpf_jit_binary_free(header); + prog = orig_prog; + goto out_addrs; + } + if (image) { + if (proglen != oldproglen) { + pr_err("bpf_jit: proglen=%d != oldproglen=%d\n", + proglen, oldproglen); + goto out_image; + } + break; + } + if (proglen == oldproglen) { + header = bpf_jit_binary_alloc(proglen, &image, + 1, jit_fill_hole); + if (!header) { + prog = orig_prog; + goto out_addrs; + } + } + oldproglen = proglen; + cond_resched(); + } + + if (bpf_jit_enable > 1) + bpf_jit_dump(prog->len, proglen, pass + 1, image); + + if (image) { + bpf_jit_binary_lock_ro(header); + prog->bpf_func = (void *)image; + prog->jited = 1; + prog->jited_len = proglen; + } else { + prog = orig_prog; + } + +out_addrs: + kfree(addrs); +out: + if (tmp_blinded) + bpf_jit_prog_release_other(prog, prog == orig_prog ? + tmp : orig_prog); + return prog; +} diff --git a/arch/x86/pci/early.c b/arch/x86/pci/early.c index f0114007e915..e5f753cbb1c3 100644 --- a/arch/x86/pci/early.c +++ b/arch/x86/pci/early.c @@ -59,24 +59,15 @@ int early_pci_allowed(void) void early_dump_pci_device(u8 bus, u8 slot, u8 func) { + u32 value[256 / 4]; int i; - int j; - u32 val; - printk(KERN_INFO "pci 0000:%02x:%02x.%d config space:", - bus, slot, func); + pr_info("pci 0000:%02x:%02x.%d config space:\n", bus, slot, func); - for (i = 0; i < 256; i += 4) { - if (!(i & 0x0f)) - printk("\n %02x:",i); + for (i = 0; i < 256; i += 4) + value[i / 4] = read_pci_config(bus, slot, func, i); - val = read_pci_config(bus, slot, func, i); - for (j = 0; j < 4; j++) { - printk(" %02x", val & 0xff); - val >>= 8; - } - } - printk("\n"); + print_hex_dump(KERN_INFO, "", DUMP_PREFIX_OFFSET, 16, 1, value, 256, false); } void early_dump_pci_devices(void) diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c index 54ef19e90705..13f4485ca388 100644 --- a/arch/x86/pci/fixup.c +++ b/arch/x86/pci/fixup.c @@ -636,6 +636,10 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x2030, quirk_no_aersid); DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x2031, quirk_no_aersid); DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x2032, quirk_no_aersid); DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x2033, quirk_no_aersid); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x334a, quirk_no_aersid); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x334b, quirk_no_aersid); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x334c, quirk_no_aersid); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x334d, quirk_no_aersid); #ifdef CONFIG_PHYS_ADDR_T_64BIT diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c index 9542a746dc50..9112d1cb397b 100644 --- a/arch/x86/pci/xen.c +++ b/arch/x86/pci/xen.c @@ -168,7 +168,7 @@ static int xen_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) if (type == PCI_CAP_ID_MSI && nvec > 1) return 1; - v = kzalloc(sizeof(int) * max(1, nvec), GFP_KERNEL); + v = kcalloc(max(1, nvec), sizeof(int), GFP_KERNEL); if (!v) return -ENOMEM; diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c index bed7e7f4e44c..e01f7ceb9e7a 100644 --- a/arch/x86/platform/efi/efi_64.c +++ b/arch/x86/platform/efi/efi_64.c @@ -225,7 +225,7 @@ int __init efi_alloc_page_tables(void) pud = pud_alloc(&init_mm, p4d, EFI_VA_END); if (!pud) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled()) free_page((unsigned long) pgd_page_vaddr(*pgd)); free_pages((unsigned long)efi_pgd, PGD_ALLOCATION_ORDER); return -ENOMEM; diff --git a/arch/x86/platform/intel-mid/intel_mid_vrtc.c b/arch/x86/platform/intel-mid/intel_mid_vrtc.c index 58024862a7eb..a52914aa3b6c 100644 --- a/arch/x86/platform/intel-mid/intel_mid_vrtc.c +++ b/arch/x86/platform/intel-mid/intel_mid_vrtc.c @@ -57,7 +57,7 @@ void vrtc_cmos_write(unsigned char val, unsigned char reg) } EXPORT_SYMBOL_GPL(vrtc_cmos_write); -void vrtc_get_time(struct timespec *now) +void vrtc_get_time(struct timespec64 *now) { u8 sec, min, hour, mday, mon; unsigned long flags; @@ -83,18 +83,18 @@ void vrtc_get_time(struct timespec *now) pr_info("vRTC: sec: %d min: %d hour: %d day: %d " "mon: %d year: %d\n", sec, min, hour, mday, mon, year); - now->tv_sec = mktime(year, mon, mday, hour, min, sec); + now->tv_sec = mktime64(year, mon, mday, hour, min, sec); now->tv_nsec = 0; } -int vrtc_set_mmss(const struct timespec *now) +int vrtc_set_mmss(const struct timespec64 *now) { unsigned long flags; struct rtc_time tm; int year; int retval = 0; - rtc_time_to_tm(now->tv_sec, &tm); + rtc_time64_to_tm(now->tv_sec, &tm); if (!rtc_valid_tm(&tm) && tm.tm_year >= 72) { /* * tm.year is the number of years since 1900, and the @@ -110,8 +110,8 @@ int vrtc_set_mmss(const struct timespec *now) vrtc_cmos_write(tm.tm_sec, RTC_SECONDS); spin_unlock_irqrestore(&rtc_lock, flags); } else { - pr_err("%s: Invalid vRTC value: write of %lx to vRTC failed\n", - __func__, now->tv_sec); + pr_err("%s: Invalid vRTC value: write of %llx to vRTC failed\n", + __func__, (s64)now->tv_sec); retval = -EINVAL; } return retval; diff --git a/arch/x86/platform/uv/tlb_uv.c b/arch/x86/platform/uv/tlb_uv.c index b36caae0fb2f..ca446da48fd2 100644 --- a/arch/x86/platform/uv/tlb_uv.c +++ b/arch/x86/platform/uv/tlb_uv.c @@ -615,7 +615,7 @@ static int uv2_3_wait_completion(struct bau_desc *bau_desc, /* spin on the status MMR, waiting for it to go idle */ while (descriptor_stat != UV2H_DESC_IDLE) { - if ((descriptor_stat == UV2H_DESC_SOURCE_TIMEOUT)) { + if (descriptor_stat == UV2H_DESC_SOURCE_TIMEOUT) { /* * A h/w bug on the destination side may * have prevented the message being marked @@ -2142,7 +2142,7 @@ static int __init init_per_cpu(int nuvhubs, int base_part_pnode) if (is_uv3_hub() || is_uv2_hub() || is_uv1_hub()) timeout_us = calculate_destination_timeout(); - vp = kmalloc(nuvhubs * sizeof(struct uvhub_desc), GFP_KERNEL); + vp = kmalloc_array(nuvhubs, sizeof(struct uvhub_desc), GFP_KERNEL); uvhub_descs = (struct uvhub_desc *)vp; memset(uvhub_descs, 0, nuvhubs * sizeof(struct uvhub_desc)); uvhub_mask = kzalloc((nuvhubs+7)/8, GFP_KERNEL); diff --git a/arch/x86/platform/uv/uv_irq.c b/arch/x86/platform/uv/uv_irq.c index e4cb9f4cde8a..fc13cbbb2dce 100644 --- a/arch/x86/platform/uv/uv_irq.c +++ b/arch/x86/platform/uv/uv_irq.c @@ -47,11 +47,6 @@ static void uv_program_mmr(struct irq_cfg *cfg, struct uv_irq_2_mmr_pnode *info) static void uv_noop(struct irq_data *data) { } -static void uv_ack_apic(struct irq_data *data) -{ - ack_APIC_irq(); -} - static int uv_set_irq_affinity(struct irq_data *data, const struct cpumask *mask, bool force) @@ -73,7 +68,7 @@ static struct irq_chip uv_irq_chip = { .name = "UV-CORE", .irq_mask = uv_noop, .irq_unmask = uv_noop, - .irq_eoi = uv_ack_apic, + .irq_eoi = apic_ack_irq, .irq_set_affinity = uv_set_irq_affinity, }; diff --git a/arch/x86/platform/uv/uv_time.c b/arch/x86/platform/uv/uv_time.c index b082d71b08ee..a36b368eea08 100644 --- a/arch/x86/platform/uv/uv_time.c +++ b/arch/x86/platform/uv/uv_time.c @@ -158,7 +158,7 @@ static __init int uv_rtc_allocate_timers(void) { int cpu; - blade_info = kzalloc(uv_possible_blades * sizeof(void *), GFP_KERNEL); + blade_info = kcalloc(uv_possible_blades, sizeof(void *), GFP_KERNEL); if (!blade_info) return -ENOMEM; diff --git a/arch/x86/power/hibernate_64.c b/arch/x86/power/hibernate_64.c index ccf4a49bb065..67ccf64c8bd8 100644 --- a/arch/x86/power/hibernate_64.c +++ b/arch/x86/power/hibernate_64.c @@ -72,7 +72,7 @@ static int set_up_temporary_text_mapping(pgd_t *pgd) * tables used by the image kernel. */ - if (pgtable_l5_enabled) { + if (pgtable_l5_enabled()) { p4d = (p4d_t *)get_safe_page(GFP_ATOMIC); if (!p4d) return -ENOMEM; diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig index 13ed827c7c66..9d529f22fd9d 100644 --- a/arch/x86/um/Kconfig +++ b/arch/x86/um/Kconfig @@ -1,5 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 -mainmenu "User Mode Linux/$SUBARCH $KERNELVERSION Kernel Configuration" +mainmenu "User Mode Linux/$(SUBARCH) $(KERNELVERSION) Kernel Configuration" + +comment "Compiler: $(CC_VERSION_TEXT)" + +source "scripts/Kconfig.include" source "arch/um/Kconfig.common" @@ -16,8 +20,8 @@ config UML_X86 select GENERIC_FIND_FIRST_BIT config 64BIT - bool "64-bit kernel" if SUBARCH = "x86" - default SUBARCH != "i386" + bool "64-bit kernel" if "$(SUBARCH)" = "x86" + default "$(SUBARCH)" != "i386" config X86_32 def_bool !64BIT diff --git a/arch/x86/um/vdso/Makefile b/arch/x86/um/vdso/Makefile index 10003359e633..b2d6967262b2 100644 --- a/arch/x86/um/vdso/Makefile +++ b/arch/x86/um/vdso/Makefile @@ -23,14 +23,14 @@ $(obj)/vdso.o: $(obj)/vdso.so targets += vdso.so vdso.so.dbg vdso.lds $(vobjs-y) -export CPPFLAGS_vdso.lds += -P -C +CPPFLAGS_vdso.lds += -P -C VDSO_LDFLAGS_vdso.lds = -m64 -Wl,-soname=linux-vdso.so.1 \ -Wl,-z,max-page-size=4096 -Wl,-z,common-page-size=4096 $(obj)/vdso.o: $(src)/vdso.S $(obj)/vdso.so -$(obj)/vdso.so.dbg: $(src)/vdso.lds $(vobjs) FORCE +$(obj)/vdso.so.dbg: $(obj)/vdso.lds $(vobjs) FORCE $(call if_changed,vdso) $(obj)/%.so: OBJCOPYFLAGS := -S diff --git a/arch/x86/xen/efi.c b/arch/x86/xen/efi.c index a18703be9ead..1804b27f9632 100644 --- a/arch/x86/xen/efi.c +++ b/arch/x86/xen/efi.c @@ -115,6 +115,61 @@ static efi_system_table_t __init *xen_efi_probe(void) return &efi_systab_xen; } +/* + * Determine whether we're in secure boot mode. + * + * Please keep the logic in sync with + * drivers/firmware/efi/libstub/secureboot.c:efi_get_secureboot(). + */ +static enum efi_secureboot_mode xen_efi_get_secureboot(void) +{ + static efi_guid_t efi_variable_guid = EFI_GLOBAL_VARIABLE_GUID; + static efi_guid_t shim_guid = EFI_SHIM_LOCK_GUID; + efi_status_t status; + u8 moksbstate, secboot, setupmode; + unsigned long size; + + size = sizeof(secboot); + status = efi.get_variable(L"SecureBoot", &efi_variable_guid, + NULL, &size, &secboot); + + if (status == EFI_NOT_FOUND) + return efi_secureboot_mode_disabled; + + if (status != EFI_SUCCESS) + goto out_efi_err; + + size = sizeof(setupmode); + status = efi.get_variable(L"SetupMode", &efi_variable_guid, + NULL, &size, &setupmode); + + if (status != EFI_SUCCESS) + goto out_efi_err; + + if (secboot == 0 || setupmode == 1) + return efi_secureboot_mode_disabled; + + /* See if a user has put the shim into insecure mode. */ + size = sizeof(moksbstate); + status = efi.get_variable(L"MokSBStateRT", &shim_guid, + NULL, &size, &moksbstate); + + /* If it fails, we don't care why. Default to secure. */ + if (status != EFI_SUCCESS) + goto secure_boot_enabled; + + if (moksbstate == 1) + return efi_secureboot_mode_disabled; + + secure_boot_enabled: + pr_info("UEFI Secure Boot is enabled.\n"); + return efi_secureboot_mode_enabled; + + out_efi_err: + pr_err("Could not determine UEFI Secure Boot status.\n"); + return efi_secureboot_mode_unknown; +} + void __init xen_efi_init(void) { efi_system_table_t *efi_systab_xen; @@ -129,6 +184,8 @@ void __init xen_efi_init(void) boot_params.efi_info.efi_systab = (__u32)__pa(efi_systab_xen); boot_params.efi_info.efi_systab_hi = (__u32)(__pa(efi_systab_xen) >> 32); + boot_params.secure_boot = xen_efi_get_secureboot(); + set_bit(EFI_BOOT, &efi.flags); set_bit(EFI_PARAVIRT, &efi.flags); set_bit(EFI_64BIT, &efi.flags); diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c index 2d76106788a3..96fc2f0fdbfe 100644 --- a/arch/x86/xen/mmu.c +++ b/arch/x86/xen/mmu.c @@ -63,37 +63,44 @@ static noinline void xen_flush_tlb_all(void) #define REMAP_BATCH_SIZE 16 struct remap_data { - xen_pfn_t *mfn; + xen_pfn_t *pfn; bool contiguous; + bool no_translate; pgprot_t prot; struct mmu_update *mmu_update; }; -static int remap_area_mfn_pte_fn(pte_t *ptep, pgtable_t token, +static int remap_area_pfn_pte_fn(pte_t *ptep, pgtable_t token, unsigned long addr, void *data) { struct remap_data *rmd = data; - pte_t pte = pte_mkspecial(mfn_pte(*rmd->mfn, rmd->prot)); + pte_t pte = pte_mkspecial(mfn_pte(*rmd->pfn, rmd->prot)); - /* If we have a contiguous range, just update the mfn itself, - else update pointer to be "next mfn". */ + /* + * If we have a contiguous range, just update the pfn itself, + * else update pointer to be "next pfn". + */ if (rmd->contiguous) - (*rmd->mfn)++; + (*rmd->pfn)++; else - rmd->mfn++; + rmd->pfn++; - rmd->mmu_update->ptr = virt_to_machine(ptep).maddr | MMU_NORMAL_PT_UPDATE; + rmd->mmu_update->ptr = virt_to_machine(ptep).maddr; + rmd->mmu_update->ptr |= rmd->no_translate ? + MMU_PT_UPDATE_NO_TRANSLATE : + MMU_NORMAL_PT_UPDATE; rmd->mmu_update->val = pte_val_ma(pte); rmd->mmu_update++; return 0; } -static int do_remap_gfn(struct vm_area_struct *vma, +static int do_remap_pfn(struct vm_area_struct *vma, unsigned long addr, - xen_pfn_t *gfn, int nr, + xen_pfn_t *pfn, int nr, int *err_ptr, pgprot_t prot, - unsigned domid, + unsigned int domid, + bool no_translate, struct page **pages) { int err = 0; @@ -104,11 +111,14 @@ static int do_remap_gfn(struct vm_area_struct *vma, BUG_ON(!((vma->vm_flags & (VM_PFNMAP | VM_IO)) == (VM_PFNMAP | VM_IO))); - rmd.mfn = gfn; + rmd.pfn = pfn; rmd.prot = prot; - /* We use the err_ptr to indicate if there we are doing a contiguous - * mapping or a discontigious mapping. */ + /* + * We use the err_ptr to indicate if there we are doing a contiguous + * mapping or a discontigious mapping. + */ rmd.contiguous = !err_ptr; + rmd.no_translate = no_translate; while (nr) { int index = 0; @@ -119,7 +129,7 @@ static int do_remap_gfn(struct vm_area_struct *vma, rmd.mmu_update = mmu_update; err = apply_to_page_range(vma->vm_mm, addr, range, - remap_area_mfn_pte_fn, &rmd); + remap_area_pfn_pte_fn, &rmd); if (err) goto out; @@ -173,7 +183,8 @@ int xen_remap_domain_gfn_range(struct vm_area_struct *vma, if (xen_feature(XENFEAT_auto_translated_physmap)) return -EOPNOTSUPP; - return do_remap_gfn(vma, addr, &gfn, nr, NULL, prot, domid, pages); + return do_remap_pfn(vma, addr, &gfn, nr, NULL, prot, domid, false, + pages); } EXPORT_SYMBOL_GPL(xen_remap_domain_gfn_range); @@ -192,10 +203,25 @@ int xen_remap_domain_gfn_array(struct vm_area_struct *vma, * cause of "wrong memory was mapped in". */ BUG_ON(err_ptr == NULL); - return do_remap_gfn(vma, addr, gfn, nr, err_ptr, prot, domid, pages); + return do_remap_pfn(vma, addr, gfn, nr, err_ptr, prot, domid, + false, pages); } EXPORT_SYMBOL_GPL(xen_remap_domain_gfn_array); +int xen_remap_domain_mfn_array(struct vm_area_struct *vma, + unsigned long addr, + xen_pfn_t *mfn, int nr, + int *err_ptr, pgprot_t prot, + unsigned int domid, struct page **pages) +{ + if (xen_feature(XENFEAT_auto_translated_physmap)) + return -EOPNOTSUPP; + + return do_remap_pfn(vma, addr, mfn, nr, err_ptr, prot, domid, + true, pages); +} +EXPORT_SYMBOL_GPL(xen_remap_domain_mfn_array); + /* Returns: 0 success */ int xen_unmap_domain_gfn_range(struct vm_area_struct *vma, int nr, struct page **pages) diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c index 29163c43ebbd..e0f1bcf01d63 100644 --- a/arch/x86/xen/time.c +++ b/arch/x86/xen/time.c @@ -57,7 +57,7 @@ static u64 xen_clocksource_get_cycles(struct clocksource *cs) return xen_clocksource_read(); } -static void xen_read_wallclock(struct timespec *ts) +static void xen_read_wallclock(struct timespec64 *ts) { struct shared_info *s = HYPERVISOR_shared_info; struct pvclock_wall_clock *wall_clock = &(s->wc); @@ -68,12 +68,12 @@ static void xen_read_wallclock(struct timespec *ts) put_cpu_var(xen_vcpu); } -static void xen_get_wallclock(struct timespec *now) +static void xen_get_wallclock(struct timespec64 *now) { xen_read_wallclock(now); } -static int xen_set_wallclock(const struct timespec *now) +static int xen_set_wallclock(const struct timespec64 *now) { return -ENODEV; } @@ -461,7 +461,7 @@ static void __init xen_time_init(void) { struct pvclock_vcpu_time_info *pvti; int cpu = smp_processor_id(); - struct timespec tp; + struct timespec64 tp; /* As Dom0 is never moved, no penalty on using TSC there */ if (xen_initial_domain()) @@ -479,7 +479,7 @@ static void __init xen_time_init(void) /* Set initial system time with full resolution */ xen_read_wallclock(&tp); - do_settimeofday(&tp); + do_settimeofday64(&tp); setup_force_cpu_cap(X86_FEATURE_TSC); diff --git a/arch/x86/xen/xen-pvh.S b/arch/x86/xen/xen-pvh.S index e1a5fbeae08d..ca2d3b2bf2af 100644 --- a/arch/x86/xen/xen-pvh.S +++ b/arch/x86/xen/xen-pvh.S @@ -54,12 +54,19 @@ * charge of setting up it's own stack, GDT and IDT. */ +#define PVH_GDT_ENTRY_CS 1 +#define PVH_GDT_ENTRY_DS 2 +#define PVH_GDT_ENTRY_CANARY 3 +#define PVH_CS_SEL (PVH_GDT_ENTRY_CS * 8) +#define PVH_DS_SEL (PVH_GDT_ENTRY_DS * 8) +#define PVH_CANARY_SEL (PVH_GDT_ENTRY_CANARY * 8) + ENTRY(pvh_start_xen) cld lgdt (_pa(gdt)) - mov $(__BOOT_DS),%eax + mov $PVH_DS_SEL,%eax mov %eax,%ds mov %eax,%es mov %eax,%ss @@ -93,11 +100,17 @@ ENTRY(pvh_start_xen) mov %eax, %cr0 /* Jump to 64-bit mode. */ - ljmp $__KERNEL_CS, $_pa(1f) + ljmp $PVH_CS_SEL, $_pa(1f) /* 64-bit entry point. */ .code64 1: + /* Set base address in stack canary descriptor. */ + mov $MSR_GS_BASE,%ecx + mov $_pa(canary), %eax + xor %edx, %edx + wrmsr + call xen_prepare_pvh /* startup_64 expects boot_params in %rsi. */ @@ -107,6 +120,17 @@ ENTRY(pvh_start_xen) #else /* CONFIG_X86_64 */ + /* Set base address in stack canary descriptor. */ + movl $_pa(gdt_start),%eax + movl $_pa(canary),%ecx + movw %cx, (PVH_GDT_ENTRY_CANARY * 8) + 2(%eax) + shrl $16, %ecx + movb %cl, (PVH_GDT_ENTRY_CANARY * 8) + 4(%eax) + movb %ch, (PVH_GDT_ENTRY_CANARY * 8) + 7(%eax) + + mov $PVH_CANARY_SEL,%eax + mov %eax,%gs + call mk_early_pgtbl_32 mov $_pa(initial_page_table), %eax @@ -116,13 +140,13 @@ ENTRY(pvh_start_xen) or $(X86_CR0_PG | X86_CR0_PE), %eax mov %eax, %cr0 - ljmp $__BOOT_CS, $1f + ljmp $PVH_CS_SEL, $1f 1: call xen_prepare_pvh mov $_pa(pvh_bootparams), %esi /* startup_32 doesn't expect paging and PAE to be on. */ - ljmp $__BOOT_CS, $_pa(2f) + ljmp $PVH_CS_SEL, $_pa(2f) 2: mov %cr0, %eax and $~X86_CR0_PG, %eax @@ -131,7 +155,7 @@ ENTRY(pvh_start_xen) and $~X86_CR4_PAE, %eax mov %eax, %cr4 - ljmp $__BOOT_CS, $_pa(startup_32) + ljmp $PVH_CS_SEL, $_pa(startup_32) #endif END(pvh_start_xen) @@ -143,16 +167,19 @@ gdt: .word 0 gdt_start: .quad 0x0000000000000000 /* NULL descriptor */ - .quad 0x0000000000000000 /* reserved */ #ifdef CONFIG_X86_64 - .quad GDT_ENTRY(0xa09a, 0, 0xfffff) /* __KERNEL_CS */ + .quad GDT_ENTRY(0xa09a, 0, 0xfffff) /* PVH_CS_SEL */ #else - .quad GDT_ENTRY(0xc09a, 0, 0xfffff) /* __KERNEL_CS */ + .quad GDT_ENTRY(0xc09a, 0, 0xfffff) /* PVH_CS_SEL */ #endif - .quad GDT_ENTRY(0xc092, 0, 0xfffff) /* __KERNEL_DS */ + .quad GDT_ENTRY(0xc092, 0, 0xfffff) /* PVH_DS_SEL */ + .quad GDT_ENTRY(0x4090, 0, 0x18) /* PVH_CANARY_SEL */ gdt_end: - .balign 4 + .balign 16 +canary: + .fill 48, 1, 0 + early_stack: .fill 256, 1, 0 early_stack_end: diff --git a/arch/xtensa/Kconfig b/arch/xtensa/Kconfig index c921e8bccdc8..d575e8701955 100644 --- a/arch/xtensa/Kconfig +++ b/arch/xtensa/Kconfig @@ -17,9 +17,7 @@ config XTENSA select GENERIC_SCHED_CLOCK select GENERIC_STRNCPY_FROM_USER if KASAN select HAVE_ARCH_KASAN if MMU - select HAVE_CC_STACKPROTECTOR select HAVE_DEBUG_KMEMLEAK - select HAVE_DMA_API_DEBUG select HAVE_DMA_CONTIGUOUS select HAVE_EXIT_THREAD select HAVE_FUNCTION_TRACER @@ -29,6 +27,7 @@ config XTENSA select HAVE_MEMBLOCK select HAVE_OPROFILE select HAVE_PERF_EVENTS + select HAVE_STACKPROTECTOR select IRQ_DOMAIN select MODULES_USE_ELF_RELA select NO_BOOTMEM diff --git a/arch/xtensa/include/asm/Kbuild b/arch/xtensa/include/asm/Kbuild index 436b20337168..e5e1e61c538c 100644 --- a/arch/xtensa/include/asm/Kbuild +++ b/arch/xtensa/include/asm/Kbuild @@ -1,4 +1,5 @@ generic-y += bug.h +generic-y += compat.h generic-y += device.h generic-y += div64.h generic-y += dma-contiguous.h diff --git a/arch/xtensa/include/asm/cacheflush.h b/arch/xtensa/include/asm/cacheflush.h index 397d6a1a4224..a0d50be5a8cb 100644 --- a/arch/xtensa/include/asm/cacheflush.h +++ b/arch/xtensa/include/asm/cacheflush.h @@ -88,7 +88,7 @@ static inline void __invalidate_icache_page_alias(unsigned long virt, * * Pages can get remapped. Because this might change the 'color' of that page, * we have to flush the cache before the PTE is changed. - * (see also Documentation/cachetlb.txt) + * (see also Documentation/core-api/cachetlb.rst) */ #if defined(CONFIG_MMU) && \ @@ -152,7 +152,7 @@ void local_flush_cache_page(struct vm_area_struct *vma, __invalidate_icache_range(start,(end) - (start)); \ } while (0) -/* This is not required, see Documentation/cachetlb.txt */ +/* This is not required, see Documentation/core-api/cachetlb.rst */ #define flush_icache_page(vma,page) do { } while (0) #define flush_dcache_mmap_lock(mapping) do { } while (0) diff --git a/arch/xtensa/include/asm/pci.h b/arch/xtensa/include/asm/pci.h index d5a82153a7c5..883024054b05 100644 --- a/arch/xtensa/include/asm/pci.h +++ b/arch/xtensa/include/asm/pci.h @@ -20,8 +20,6 @@ #define pcibios_assign_all_busses() 0 -extern struct pci_controller* pcibios_alloc_controller(void); - /* Assume some values. (We should revise them, if necessary) */ #define PCIBIOS_MIN_IO 0x2000 @@ -42,8 +40,6 @@ extern struct pci_controller* pcibios_alloc_controller(void); * decisions. */ -#define PCI_DMA_BUS_IS_PHYS (1) - /* Tell PCI code what kind of PCI resource mappings we support */ #define HAVE_PCI_MMAP 1 #define ARCH_GENERIC_PCI_MMAP_RESOURCE 1 diff --git a/arch/xtensa/include/uapi/asm/msgbuf.h b/arch/xtensa/include/uapi/asm/msgbuf.h index 36e2e103ca38..d6915e9f071c 100644 --- a/arch/xtensa/include/uapi/asm/msgbuf.h +++ b/arch/xtensa/include/uapi/asm/msgbuf.h @@ -7,7 +7,6 @@ * between kernel and user space. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem * - 2 miscellaneous 32-bit values * * This file is subject to the terms and conditions of the GNU General @@ -21,19 +20,19 @@ struct msqid64_ds { struct ipc64_perm msg_perm; #ifdef __XTENSA_EB__ - unsigned int __unused1; - __kernel_time_t msg_stime; /* last msgsnd time */ - unsigned int __unused2; - __kernel_time_t msg_rtime; /* last msgrcv time */ - unsigned int __unused3; - __kernel_time_t msg_ctime; /* last change time */ + unsigned long msg_stime_high; + unsigned long msg_stime; /* last msgsnd time */ + unsigned long msg_rtime_high; + unsigned long msg_rtime; /* last msgrcv time */ + unsigned long msg_ctime_high; + unsigned long msg_ctime; /* last change time */ #elif defined(__XTENSA_EL__) - __kernel_time_t msg_stime; /* last msgsnd time */ - unsigned int __unused1; - __kernel_time_t msg_rtime; /* last msgrcv time */ - unsigned int __unused2; - __kernel_time_t msg_ctime; /* last change time */ - unsigned int __unused3; + unsigned long msg_stime; /* last msgsnd time */ + unsigned long msg_stime_high; + unsigned long msg_rtime; /* last msgrcv time */ + unsigned long msg_rtime_high; + unsigned long msg_ctime; /* last change time */ + unsigned long msg_ctime_high; #else # error processor byte order undefined! #endif diff --git a/arch/xtensa/include/uapi/asm/sembuf.h b/arch/xtensa/include/uapi/asm/sembuf.h index f61b6331a10c..09f348d643f1 100644 --- a/arch/xtensa/include/uapi/asm/sembuf.h +++ b/arch/xtensa/include/uapi/asm/sembuf.h @@ -14,7 +14,6 @@ * between kernel and user space. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem * - 2 miscellaneous 32-bit values * */ @@ -27,15 +26,15 @@ struct semid64_ds { struct ipc64_perm sem_perm; /* permissions .. see ipc.h */ #ifdef __XTENSA_EL__ - __kernel_time_t sem_otime; /* last semop time */ - unsigned long __unused1; - __kernel_time_t sem_ctime; /* last change time */ - unsigned long __unused2; + unsigned long sem_otime; /* last semop time */ + unsigned long sem_otime_high; + unsigned long sem_ctime; /* last change time */ + unsigned long sem_ctime_high; #else - unsigned long __unused1; - __kernel_time_t sem_otime; /* last semop time */ - unsigned long __unused2; - __kernel_time_t sem_ctime; /* last change time */ + unsigned long sem_otime_high; + unsigned long sem_otime; /* last semop time */ + unsigned long sem_ctime_high; + unsigned long sem_ctime; /* last change time */ #endif unsigned long sem_nsems; /* no. of semaphores in array */ unsigned long __unused3; diff --git a/arch/xtensa/include/uapi/asm/shmbuf.h b/arch/xtensa/include/uapi/asm/shmbuf.h index 26550bdc8430..554a57a6a90f 100644 --- a/arch/xtensa/include/uapi/asm/shmbuf.h +++ b/arch/xtensa/include/uapi/asm/shmbuf.h @@ -4,10 +4,10 @@ * * The shmid64_ds structure for Xtensa architecture. * Note extra padding because this structure is passed back and forth - * between kernel and user space. + * between kernel and user space, but the padding is on the wrong + * side for big-endian xtensa, for historic reasons. * * Pad space is left for: - * - 64-bit time_t to solve y2038 problem * - 2 miscellaneous 32-bit values * * This file is subject to the terms and conditions of the GNU General Public @@ -20,42 +20,21 @@ #ifndef _XTENSA_SHMBUF_H #define _XTENSA_SHMBUF_H -#if defined (__XTENSA_EL__) struct shmid64_ds { struct ipc64_perm shm_perm; /* operation perms */ size_t shm_segsz; /* size of segment (bytes) */ - __kernel_time_t shm_atime; /* last attach time */ - unsigned long __unused1; - __kernel_time_t shm_dtime; /* last detach time */ - unsigned long __unused2; - __kernel_time_t shm_ctime; /* last change time */ - unsigned long __unused3; + unsigned long shm_atime; /* last attach time */ + unsigned long shm_atime_high; + unsigned long shm_dtime; /* last detach time */ + unsigned long shm_dtime_high; + unsigned long shm_ctime; /* last change time */ + unsigned long shm_ctime_high; __kernel_pid_t shm_cpid; /* pid of creator */ __kernel_pid_t shm_lpid; /* pid of last operator */ unsigned long shm_nattch; /* no. of current attaches */ unsigned long __unused4; unsigned long __unused5; }; -#elif defined (__XTENSA_EB__) -struct shmid64_ds { - struct ipc64_perm shm_perm; /* operation perms */ - size_t shm_segsz; /* size of segment (bytes) */ - __kernel_time_t shm_atime; /* last attach time */ - unsigned long __unused1; - __kernel_time_t shm_dtime; /* last detach time */ - unsigned long __unused2; - __kernel_time_t shm_ctime; /* last change time */ - unsigned long __unused3; - __kernel_pid_t shm_cpid; /* pid of creator */ - __kernel_pid_t shm_lpid; /* pid of last operator */ - unsigned long shm_nattch; /* no. of current attaches */ - unsigned long __unused4; - unsigned long __unused5; -}; -#else -# error endian order not defined -#endif - struct shminfo64 { unsigned long shmmax; diff --git a/arch/xtensa/kernel/asm-offsets.c b/arch/xtensa/kernel/asm-offsets.c index 022cf918ec20..67904f55f188 100644 --- a/arch/xtensa/kernel/asm-offsets.c +++ b/arch/xtensa/kernel/asm-offsets.c @@ -76,7 +76,7 @@ int main(void) DEFINE(TASK_PID, offsetof (struct task_struct, pid)); DEFINE(TASK_THREAD, offsetof (struct task_struct, thread)); DEFINE(TASK_THREAD_INFO, offsetof (struct task_struct, stack)); -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR DEFINE(TASK_STACK_CANARY, offsetof(struct task_struct, stack_canary)); #endif DEFINE(TASK_STRUCT_SIZE, sizeof (struct task_struct)); diff --git a/arch/xtensa/kernel/entry.S b/arch/xtensa/kernel/entry.S index 5caff0744f3c..9cbc380e9572 100644 --- a/arch/xtensa/kernel/entry.S +++ b/arch/xtensa/kernel/entry.S @@ -1971,7 +1971,7 @@ ENTRY(_switch_to) s32i a1, a2, THREAD_SP # save stack pointer #endif -#if defined(CONFIG_CC_STACKPROTECTOR) && !defined(CONFIG_SMP) +#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP) movi a6, __stack_chk_guard l32i a8, a3, TASK_STACK_CANARY s32i a8, a6, 0 diff --git a/arch/xtensa/kernel/pci-dma.c b/arch/xtensa/kernel/pci-dma.c index 732631ce250f..392b4a80ebc2 100644 --- a/arch/xtensa/kernel/pci-dma.c +++ b/arch/xtensa/kernel/pci-dma.c @@ -261,12 +261,3 @@ const struct dma_map_ops xtensa_dma_map_ops = { .mapping_error = xtensa_dma_mapping_error, }; EXPORT_SYMBOL(xtensa_dma_map_ops); - -#define PREALLOC_DMA_DEBUG_ENTRIES (1 << 16) - -static int __init xtensa_dma_init(void) -{ - dma_debug_init(PREALLOC_DMA_DEBUG_ENTRIES); - return 0; -} -fs_initcall(xtensa_dma_init); diff --git a/arch/xtensa/kernel/pci.c b/arch/xtensa/kernel/pci.c index b7c7a60c7000..21f13e9aabe1 100644 --- a/arch/xtensa/kernel/pci.c +++ b/arch/xtensa/kernel/pci.c @@ -41,8 +41,8 @@ * pci_bus_add_device */ -struct pci_controller* pci_ctrl_head; -struct pci_controller** pci_ctrl_tail = &pci_ctrl_head; +static struct pci_controller *pci_ctrl_head; +static struct pci_controller **pci_ctrl_tail = &pci_ctrl_head; static int pci_bus_count; @@ -80,50 +80,6 @@ pcibios_align_resource(void *data, const struct resource *res, return start; } -int -pcibios_enable_resources(struct pci_dev *dev, int mask) -{ - u16 cmd, old_cmd; - int idx; - struct resource *r; - - pci_read_config_word(dev, PCI_COMMAND, &cmd); - old_cmd = cmd; - for(idx=0; idx<6; idx++) { - r = &dev->resource[idx]; - if (!r->start && r->end) { - pr_err("PCI: Device %s not available because " - "of resource collisions\n", pci_name(dev)); - return -EINVAL; - } - if (r->flags & IORESOURCE_IO) - cmd |= PCI_COMMAND_IO; - if (r->flags & IORESOURCE_MEM) - cmd |= PCI_COMMAND_MEMORY; - } - if (dev->resource[PCI_ROM_RESOURCE].start) - cmd |= PCI_COMMAND_MEMORY; - if (cmd != old_cmd) { - pr_info("PCI: Enabling device %s (%04x -> %04x)\n", - pci_name(dev), old_cmd, cmd); - pci_write_config_word(dev, PCI_COMMAND, cmd); - } - return 0; -} - -struct pci_controller * __init pcibios_alloc_controller(void) -{ - struct pci_controller *pci_ctrl; - - pci_ctrl = (struct pci_controller *)alloc_bootmem(sizeof(*pci_ctrl)); - memset(pci_ctrl, 0, sizeof(struct pci_controller)); - - *pci_ctrl_tail = pci_ctrl; - pci_ctrl_tail = &pci_ctrl->next; - - return pci_ctrl; -} - static void __init pci_controller_apertures(struct pci_controller *pci_ctrl, struct list_head *resources) { @@ -223,8 +179,7 @@ int pcibios_enable_device(struct pci_dev *dev, int mask) for (idx=0; idx<6; idx++) { r = &dev->resource[idx]; if (!r->start && r->end) { - pr_err("PCI: Device %s not available because " - "of resource collisions\n", pci_name(dev)); + pci_err(dev, "can't enable device: resource collisions\n"); return -EINVAL; } if (r->flags & IORESOURCE_IO) @@ -233,29 +188,13 @@ int pcibios_enable_device(struct pci_dev *dev, int mask) cmd |= PCI_COMMAND_MEMORY; } if (cmd != old_cmd) { - pr_info("PCI: Enabling device %s (%04x -> %04x)\n", - pci_name(dev), old_cmd, cmd); + pci_info(dev, "enabling device (%04x -> %04x)\n", old_cmd, cmd); pci_write_config_word(dev, PCI_COMMAND, cmd); } return 0; } -#ifdef CONFIG_PROC_FS - -/* - * Return the index of the PCI controller for device pdev. - */ - -int -pci_controller_num(struct pci_dev *dev) -{ - struct pci_controller *pci_ctrl = (struct pci_controller*) dev->sysdata; - return pci_ctrl->index; -} - -#endif /* CONFIG_PROC_FS */ - /* * Platform support for /proc/bus/pci/X/Y mmap()s. * -- paulus. diff --git a/arch/xtensa/kernel/process.c b/arch/xtensa/kernel/process.c index 8dd0593fb2c4..483dcfb6e681 100644 --- a/arch/xtensa/kernel/process.c +++ b/arch/xtensa/kernel/process.c @@ -58,7 +58,7 @@ void (*pm_power_off)(void) = NULL; EXPORT_SYMBOL(pm_power_off); -#ifdef CONFIG_CC_STACKPROTECTOR +#ifdef CONFIG_STACKPROTECTOR #include unsigned long __stack_chk_guard __read_mostly; EXPORT_SYMBOL(__stack_chk_guard); diff --git a/arch/xtensa/kernel/traps.c b/arch/xtensa/kernel/traps.c index 32c5207f1226..86507fa7c2d7 100644 --- a/arch/xtensa/kernel/traps.c +++ b/arch/xtensa/kernel/traps.c @@ -323,8 +323,6 @@ do_illegal_instruction(struct pt_regs *regs) void do_unaligned_user (struct pt_regs *regs) { - siginfo_t info; - __die_if_kernel("Unhandled unaligned exception in kernel", regs, SIGKILL); @@ -334,12 +332,7 @@ do_unaligned_user (struct pt_regs *regs) "(pid = %d, pc = %#010lx)\n", regs->excvaddr, current->comm, task_pid_nr(current), regs->pc); - info.si_signo = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRALN; - info.si_addr = (void *) regs->excvaddr; - force_sig_info(SIGSEGV, &info, current); - + force_sig_fault(SIGBUS, BUS_ADRALN, (void *) regs->excvaddr, current); } #endif diff --git a/arch/xtensa/mm/fault.c b/arch/xtensa/mm/fault.c index 8b9b6f44bb06..c111a833205a 100644 --- a/arch/xtensa/mm/fault.c +++ b/arch/xtensa/mm/fault.c @@ -39,13 +39,13 @@ void do_page_fault(struct pt_regs *regs) struct mm_struct *mm = current->mm; unsigned int exccause = regs->exccause; unsigned int address = regs->excvaddr; - siginfo_t info; + int code; int is_write, is_exec; int fault; unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; - info.si_code = SEGV_MAPERR; + code = SEGV_MAPERR; /* We fault-in kernel-space virtual memory on-demand. The * 'reference' page table is init_mm.pgd. @@ -91,7 +91,7 @@ retry: */ good_area: - info.si_code = SEGV_ACCERR; + code = SEGV_ACCERR; if (is_write) { if (!(vma->vm_flags & VM_WRITE)) @@ -157,11 +157,7 @@ bad_area: if (user_mode(regs)) { current->thread.bad_vaddr = address; current->thread.error_code = is_write; - info.si_signo = SIGSEGV; - info.si_errno = 0; - /* info.si_code has been set above */ - info.si_addr = (void *) address; - force_sig_info(SIGSEGV, &info, current); + force_sig_fault(SIGSEGV, code, (void *) address, current); return; } bad_page_fault(regs, address, SIGSEGV); @@ -186,11 +182,7 @@ do_sigbus: * or user mode. */ current->thread.bad_vaddr = address; - info.si_code = SIGBUS; - info.si_errno = 0; - info.si_code = BUS_ADRERR; - info.si_addr = (void *) address; - force_sig_info(SIGBUS, &info, current); + force_sig_fault(SIGBUS, BUS_ADRERR, (void *) address, current); /* Kernel mode? Handle exceptions or die */ if (!user_mode(regs)) diff --git a/arch/xtensa/platforms/iss/console.c b/arch/xtensa/platforms/iss/console.c index 92f567f9a21e..af81a62faba6 100644 --- a/arch/xtensa/platforms/iss/console.c +++ b/arch/xtensa/platforms/iss/console.c @@ -153,19 +153,6 @@ static int rs_proc_show(struct seq_file *m, void *v) return 0; } -static int rs_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, rs_proc_show, NULL); -} - -static const struct file_operations rs_proc_fops = { - .owner = THIS_MODULE, - .open = rs_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static const struct tty_operations serial_ops = { .open = rs_open, .close = rs_close, @@ -176,7 +163,7 @@ static const struct tty_operations serial_ops = { .chars_in_buffer = rs_chars_in_buffer, .hangup = rs_hangup, .wait_until_sent = rs_wait_until_sent, - .proc_fops = &rs_proc_fops, + .proc_show = rs_proc_show, }; int __init rs_init(void) diff --git a/block/Kconfig b/block/Kconfig index 28ec55752b68..eb50fd4977c2 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -114,7 +114,7 @@ config BLK_DEV_THROTTLING one needs to mount and use blkio cgroup controller for creating cgroups and specifying per device IO rate policies. - See Documentation/cgroups/blkio-controller.txt for more information. + See Documentation/cgroup-v1/blkio-controller.txt for more information. config BLK_DEV_THROTTLING_LOW bool "Block throttling .low limit interface support (EXPERIMENTAL)" diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index d819dc77fe65..a9e8633388f4 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -55,13 +55,13 @@ BFQG_FLAG_FNS(empty) /* This should be called with the scheduler lock held. */ static void bfqg_stats_update_group_wait_time(struct bfqg_stats *stats) { - unsigned long long now; + u64 now; if (!bfqg_stats_waiting(stats)) return; - now = sched_clock(); - if (time_after64(now, stats->start_group_wait_time)) + now = ktime_get_ns(); + if (now > stats->start_group_wait_time) blkg_stat_add(&stats->group_wait_time, now - stats->start_group_wait_time); bfqg_stats_clear_waiting(stats); @@ -77,20 +77,20 @@ static void bfqg_stats_set_start_group_wait_time(struct bfq_group *bfqg, return; if (bfqg == curr_bfqg) return; - stats->start_group_wait_time = sched_clock(); + stats->start_group_wait_time = ktime_get_ns(); bfqg_stats_mark_waiting(stats); } /* This should be called with the scheduler lock held. */ static void bfqg_stats_end_empty_time(struct bfqg_stats *stats) { - unsigned long long now; + u64 now; if (!bfqg_stats_empty(stats)) return; - now = sched_clock(); - if (time_after64(now, stats->start_empty_time)) + now = ktime_get_ns(); + if (now > stats->start_empty_time) blkg_stat_add(&stats->empty_time, now - stats->start_empty_time); bfqg_stats_clear_empty(stats); @@ -116,7 +116,7 @@ void bfqg_stats_set_start_empty_time(struct bfq_group *bfqg) if (bfqg_stats_empty(stats)) return; - stats->start_empty_time = sched_clock(); + stats->start_empty_time = ktime_get_ns(); bfqg_stats_mark_empty(stats); } @@ -125,9 +125,9 @@ void bfqg_stats_update_idle_time(struct bfq_group *bfqg) struct bfqg_stats *stats = &bfqg->stats; if (bfqg_stats_idling(stats)) { - unsigned long long now = sched_clock(); + u64 now = ktime_get_ns(); - if (time_after64(now, stats->start_idle_time)) + if (now > stats->start_idle_time) blkg_stat_add(&stats->idle_time, now - stats->start_idle_time); bfqg_stats_clear_idling(stats); @@ -138,7 +138,7 @@ void bfqg_stats_set_start_idle_time(struct bfq_group *bfqg) { struct bfqg_stats *stats = &bfqg->stats; - stats->start_idle_time = sched_clock(); + stats->start_idle_time = ktime_get_ns(); bfqg_stats_mark_idling(stats); } @@ -171,18 +171,18 @@ void bfqg_stats_update_io_merged(struct bfq_group *bfqg, unsigned int op) blkg_rwstat_add(&bfqg->stats.merged, op, 1); } -void bfqg_stats_update_completion(struct bfq_group *bfqg, uint64_t start_time, - uint64_t io_start_time, unsigned int op) +void bfqg_stats_update_completion(struct bfq_group *bfqg, u64 start_time_ns, + u64 io_start_time_ns, unsigned int op) { struct bfqg_stats *stats = &bfqg->stats; - unsigned long long now = sched_clock(); + u64 now = ktime_get_ns(); - if (time_after64(now, io_start_time)) + if (now > io_start_time_ns) blkg_rwstat_add(&stats->service_time, op, - now - io_start_time); - if (time_after64(io_start_time, start_time)) + now - io_start_time_ns); + if (io_start_time_ns > start_time_ns) blkg_rwstat_add(&stats->wait_time, op, - io_start_time - start_time); + io_start_time_ns - start_time_ns); } #else /* CONFIG_BFQ_GROUP_IOSCHED && CONFIG_DEBUG_BLK_CGROUP */ @@ -191,8 +191,8 @@ void bfqg_stats_update_io_add(struct bfq_group *bfqg, struct bfq_queue *bfqq, unsigned int op) { } void bfqg_stats_update_io_remove(struct bfq_group *bfqg, unsigned int op) { } void bfqg_stats_update_io_merged(struct bfq_group *bfqg, unsigned int op) { } -void bfqg_stats_update_completion(struct bfq_group *bfqg, uint64_t start_time, - uint64_t io_start_time, unsigned int op) { } +void bfqg_stats_update_completion(struct bfq_group *bfqg, u64 start_time_ns, + u64 io_start_time_ns, unsigned int op) { } void bfqg_stats_update_dequeue(struct bfq_group *bfqg) { } void bfqg_stats_set_start_empty_time(struct bfq_group *bfqg) { } void bfqg_stats_update_idle_time(struct bfq_group *bfqg) { } diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 771ae9730ac6..495b9ddb3355 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -49,9 +49,39 @@ * * In particular, to provide these low-latency guarantees, BFQ * explicitly privileges the I/O of two classes of time-sensitive - * applications: interactive and soft real-time. This feature enables - * BFQ to provide applications in these classes with a very low - * latency. Finally, BFQ also features additional heuristics for + * applications: interactive and soft real-time. In more detail, BFQ + * behaves this way if the low_latency parameter is set (default + * configuration). This feature enables BFQ to provide applications in + * these classes with a very low latency. + * + * To implement this feature, BFQ constantly tries to detect whether + * the I/O requests in a bfq_queue come from an interactive or a soft + * real-time application. For brevity, in these cases, the queue is + * said to be interactive or soft real-time. In both cases, BFQ + * privileges the service of the queue, over that of non-interactive + * and non-soft-real-time queues. This privileging is performed, + * mainly, by raising the weight of the queue. So, for brevity, we + * call just weight-raising periods the time periods during which a + * queue is privileged, because deemed interactive or soft real-time. + * + * The detection of soft real-time queues/applications is described in + * detail in the comments on the function + * bfq_bfqq_softrt_next_start. On the other hand, the detection of an + * interactive queue works as follows: a queue is deemed interactive + * if it is constantly non empty only for a limited time interval, + * after which it does become empty. The queue may be deemed + * interactive again (for a limited time), if it restarts being + * constantly non empty, provided that this happens only after the + * queue has remained empty for a given minimum idle time. + * + * By default, BFQ computes automatically the above maximum time + * interval, i.e., the time interval after which a constantly + * non-empty queue stops being deemed interactive. Since a queue is + * weight-raised while it is deemed interactive, this maximum time + * interval happens to coincide with the (maximum) duration of the + * weight-raising for interactive queues. + * + * Finally, BFQ also features additional heuristics for * preserving both a low latency and a high throughput on NCQ-capable, * rotational or flash-based devices, and to get the job done quickly * for applications consisting in many I/O-bound processes. @@ -61,14 +91,14 @@ * all low-latency heuristics for that device, by setting low_latency * to 0. * - * BFQ is described in [1], where also a reference to the initial, more - * theoretical paper on BFQ can be found. The interested reader can find - * in the latter paper full details on the main algorithm, as well as - * formulas of the guarantees and formal proofs of all the properties. - * With respect to the version of BFQ presented in these papers, this - * implementation adds a few more heuristics, such as the one that - * guarantees a low latency to soft real-time applications, and a - * hierarchical extension based on H-WF2Q+. + * BFQ is described in [1], where also a reference to the initial, + * more theoretical paper on BFQ can be found. The interested reader + * can find in the latter paper full details on the main algorithm, as + * well as formulas of the guarantees and formal proofs of all the + * properties. With respect to the version of BFQ presented in these + * papers, this implementation adds a few more heuristics, such as the + * ones that guarantee a low latency to interactive and soft real-time + * applications, and a hierarchical extension based on H-WF2Q+. * * B-WF2Q+ is based on WF2Q+, which is described in [2], together with * H-WF2Q+, while the augmented tree used here to implement B-WF2Q+ @@ -218,56 +248,46 @@ static struct kmem_cache *bfq_pool; #define BFQ_RATE_SHIFT 16 /* - * By default, BFQ computes the duration of the weight raising for - * interactive applications automatically, using the following formula: - * duration = (R / r) * T, where r is the peak rate of the device, and - * R and T are two reference parameters. - * In particular, R is the peak rate of the reference device (see - * below), and T is a reference time: given the systems that are - * likely to be installed on the reference device according to its - * speed class, T is about the maximum time needed, under BFQ and - * while reading two files in parallel, to load typical large - * applications on these systems (see the comments on - * max_service_from_wr below, for more details on how T is obtained). - * In practice, the slower/faster the device at hand is, the more/less - * it takes to load applications with respect to the reference device. - * Accordingly, the longer/shorter BFQ grants weight raising to - * interactive applications. - * - * BFQ uses four different reference pairs (R, T), depending on: - * . whether the device is rotational or non-rotational; - * . whether the device is slow, such as old or portable HDDs, as well as - * SD cards, or fast, such as newer HDDs and SSDs. + * When configured for computing the duration of the weight-raising + * for interactive queues automatically (see the comments at the + * beginning of this file), BFQ does it using the following formula: + * duration = (ref_rate / r) * ref_wr_duration, + * where r is the peak rate of the device, and ref_rate and + * ref_wr_duration are two reference parameters. In particular, + * ref_rate is the peak rate of the reference storage device (see + * below), and ref_wr_duration is about the maximum time needed, with + * BFQ and while reading two files in parallel, to load typical large + * applications on the reference device (see the comments on + * max_service_from_wr below, for more details on how ref_wr_duration + * is obtained). In practice, the slower/faster the device at hand + * is, the more/less it takes to load applications with respect to the + * reference device. Accordingly, the longer/shorter BFQ grants + * weight raising to interactive applications. * - * The device's speed class is dynamically (re)detected in - * bfq_update_peak_rate() every time the estimated peak rate is updated. + * BFQ uses two different reference pairs (ref_rate, ref_wr_duration), + * depending on whether the device is rotational or non-rotational. * - * In the following definitions, R_slow[0]/R_fast[0] and - * T_slow[0]/T_fast[0] are the reference values for a slow/fast - * rotational device, whereas R_slow[1]/R_fast[1] and - * T_slow[1]/T_fast[1] are the reference values for a slow/fast - * non-rotational device. Finally, device_speed_thresh are the - * thresholds used to switch between speed classes. The reference - * rates are not the actual peak rates of the devices used as a - * reference, but slightly lower values. The reason for using these - * slightly lower values is that the peak-rate estimator tends to - * yield slightly lower values than the actual peak rate (it can yield - * the actual peak rate only if there is only one process doing I/O, - * and the process does sequential I/O). + * In the following definitions, ref_rate[0] and ref_wr_duration[0] + * are the reference values for a rotational device, whereas + * ref_rate[1] and ref_wr_duration[1] are the reference values for a + * non-rotational device. The reference rates are not the actual peak + * rates of the devices used as a reference, but slightly lower + * values. The reason for using slightly lower values is that the + * peak-rate estimator tends to yield slightly lower values than the + * actual peak rate (it can yield the actual peak rate only if there + * is only one process doing I/O, and the process does sequential + * I/O). * - * Both the reference peak rates and the thresholds are measured in - * sectors/usec, left-shifted by BFQ_RATE_SHIFT. + * The reference peak rates are measured in sectors/usec, left-shifted + * by BFQ_RATE_SHIFT. */ -static int R_slow[2] = {1000, 10700}; -static int R_fast[2] = {14000, 33000}; +static int ref_rate[2] = {14000, 33000}; /* - * To improve readability, a conversion function is used to initialize the - * following arrays, which entails that they can be initialized only in a - * function. + * To improve readability, a conversion function is used to initialize + * the following array, which entails that the array can be + * initialized only in a function. */ -static int T_slow[2]; -static int T_fast[2]; -static int device_speed_thresh[2]; +static int ref_wr_duration[2]; /* * BFQ uses the above-detailed, time-based weight-raising mechanism to @@ -486,46 +506,6 @@ static struct request *bfq_choose_req(struct bfq_data *bfqd, } } -/* - * See the comments on bfq_limit_depth for the purpose of - * the depths set in the function. - */ -static void bfq_update_depths(struct bfq_data *bfqd, struct sbitmap_queue *bt) -{ - bfqd->sb_shift = bt->sb.shift; - - /* - * In-word depths if no bfq_queue is being weight-raised: - * leaving 25% of tags only for sync reads. - * - * In next formulas, right-shift the value - * (1U<sb_shift), instead of computing directly - * (1U<<(bfqd->sb_shift - something)), to be robust against - * any possible value of bfqd->sb_shift, without having to - * limit 'something'. - */ - /* no more than 50% of tags for async I/O */ - bfqd->word_depths[0][0] = max((1U<sb_shift)>>1, 1U); - /* - * no more than 75% of tags for sync writes (25% extra tags - * w.r.t. async I/O, to prevent async I/O from starving sync - * writes) - */ - bfqd->word_depths[0][1] = max(((1U<sb_shift) * 3)>>2, 1U); - - /* - * In-word depths in case some bfq_queue is being weight- - * raised: leaving ~63% of tags for sync reads. This is the - * highest percentage for which, in our tests, application - * start-up times didn't suffer from any regression due to tag - * shortage. - */ - /* no more than ~18% of tags for async I/O */ - bfqd->word_depths[1][0] = max(((1U<sb_shift) * 3)>>4, 1U); - /* no more than ~37% of tags for sync writes (~20% extra tags) */ - bfqd->word_depths[1][1] = max(((1U<sb_shift) * 6)>>4, 1U); -} - /* * Async I/O can easily starve sync I/O (both sync reads and sync * writes), by consuming all tags. Similarly, storms of sync writes, @@ -535,25 +515,11 @@ static void bfq_update_depths(struct bfq_data *bfqd, struct sbitmap_queue *bt) */ static void bfq_limit_depth(unsigned int op, struct blk_mq_alloc_data *data) { - struct blk_mq_tags *tags = blk_mq_tags_from_data(data); struct bfq_data *bfqd = data->q->elevator->elevator_data; - struct sbitmap_queue *bt; if (op_is_sync(op) && !op_is_write(op)) return; - if (data->flags & BLK_MQ_REQ_RESERVED) { - if (unlikely(!tags->nr_reserved_tags)) { - WARN_ON_ONCE(1); - return; - } - bt = &tags->breserved_tags; - } else - bt = &tags->bitmap_tags; - - if (unlikely(bfqd->sb_shift != bt->sb.shift)) - bfq_update_depths(bfqd, bt); - data->shallow_depth = bfqd->word_depths[!!bfqd->wr_busy_queues][op_is_sync(op)]; @@ -906,26 +872,30 @@ static unsigned int bfq_wr_duration(struct bfq_data *bfqd) if (bfqd->bfq_wr_max_time > 0) return bfqd->bfq_wr_max_time; - dur = bfqd->RT_prod; + dur = bfqd->rate_dur_prod; do_div(dur, bfqd->peak_rate); /* - * Limit duration between 3 and 13 seconds. Tests show that - * higher values than 13 seconds often yield the opposite of - * the desired result, i.e., worsen responsiveness by letting - * non-interactive and non-soft-real-time applications - * preserve weight raising for a too long time interval. + * Limit duration between 3 and 25 seconds. The upper limit + * has been conservatively set after the following worst case: + * on a QEMU/KVM virtual machine + * - running in a slow PC + * - with a virtual disk stacked on a slow low-end 5400rpm HDD + * - serving a heavy I/O workload, such as the sequential reading + * of several files + * mplayer took 23 seconds to start, if constantly weight-raised. + * + * As for higher values than that accomodating the above bad + * scenario, tests show that higher values would often yield + * the opposite of the desired result, i.e., would worsen + * responsiveness by allowing non-interactive applications to + * preserve weight raising for too long. * * On the other end, lower values than 3 seconds make it * difficult for most interactive tasks to complete their jobs * before weight-raising finishes. */ - if (dur > msecs_to_jiffies(13000)) - dur = msecs_to_jiffies(13000); - else if (dur < msecs_to_jiffies(3000)) - dur = msecs_to_jiffies(3000); - - return dur; + return clamp_val(dur, msecs_to_jiffies(3000), msecs_to_jiffies(25000)); } /* switch back from soft real-time to interactive weight raising */ @@ -1392,15 +1362,6 @@ static bool bfq_bfqq_update_budg_for_activation(struct bfq_data *bfqd, return wr_or_deserves_wr; } -/* - * Return the farthest future time instant according to jiffies - * macros. - */ -static unsigned long bfq_greatest_from_now(void) -{ - return jiffies + MAX_JIFFY_OFFSET; -} - /* * Return the farthest past time instant according to jiffies * macros. @@ -1545,7 +1506,8 @@ static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, in_burst = bfq_bfqq_in_large_burst(bfqq); soft_rt = bfqd->bfq_wr_max_softrt_rate > 0 && !in_burst && - time_is_before_jiffies(bfqq->soft_rt_next_start); + time_is_before_jiffies(bfqq->soft_rt_next_start) && + bfqq->dispatched == 0; *interactive = !in_burst && idle_for_long_time; wr_or_deserves_wr = bfqd->low_latency && (bfqq->wr_coeff > 1 || @@ -1858,6 +1820,8 @@ static int bfq_request_merge(struct request_queue *q, struct request **req, return ELEVATOR_NO_MERGE; } +static struct bfq_queue *bfq_init_rq(struct request *rq); + static void bfq_request_merged(struct request_queue *q, struct request *req, enum elv_merge type) { @@ -1866,7 +1830,7 @@ static void bfq_request_merged(struct request_queue *q, struct request *req, blk_rq_pos(req) < blk_rq_pos(container_of(rb_prev(&req->rb_node), struct request, rb_node))) { - struct bfq_queue *bfqq = RQ_BFQQ(req); + struct bfq_queue *bfqq = bfq_init_rq(req); struct bfq_data *bfqd = bfqq->bfqd; struct request *prev, *next_rq; @@ -1891,14 +1855,25 @@ static void bfq_request_merged(struct request_queue *q, struct request *req, } } +/* + * This function is called to notify the scheduler that the requests + * rq and 'next' have been merged, with 'next' going away. BFQ + * exploits this hook to address the following issue: if 'next' has a + * fifo_time lower that rq, then the fifo_time of rq must be set to + * the value of 'next', to not forget the greater age of 'next'. + * + * NOTE: in this function we assume that rq is in a bfq_queue, basing + * on that rq is picked from the hash table q->elevator->hash, which, + * in its turn, is filled only with I/O requests present in + * bfq_queues, while BFQ is in use for the request queue q. In fact, + * the function that fills this hash table (elv_rqhash_add) is called + * only by bfq_insert_request. + */ static void bfq_requests_merged(struct request_queue *q, struct request *rq, struct request *next) { - struct bfq_queue *bfqq = RQ_BFQQ(rq), *next_bfqq = RQ_BFQQ(next); - - if (!RB_EMPTY_NODE(&rq->rb_node)) - goto end; - spin_lock_irq(&bfqq->bfqd->lock); + struct bfq_queue *bfqq = bfq_init_rq(rq), + *next_bfqq = bfq_init_rq(next); /* * If next and rq belong to the same bfq_queue and next is older @@ -1920,11 +1895,6 @@ static void bfq_requests_merged(struct request_queue *q, struct request *rq, if (bfqq->next_rq == next) bfqq->next_rq = rq; - bfq_remove_request(q, next); - bfqg_stats_update_io_remove(bfqq_group(bfqq), next->cmd_flags); - - spin_unlock_irq(&bfqq->bfqd->lock); -end: bfqg_stats_update_io_merged(bfqq_group(bfqq), next->cmd_flags); } @@ -2506,37 +2476,15 @@ static unsigned long bfq_calc_max_budget(struct bfq_data *bfqd) /* * Update parameters related to throughput and responsiveness, as a * function of the estimated peak rate. See comments on - * bfq_calc_max_budget(), and on T_slow and T_fast arrays. + * bfq_calc_max_budget(), and on the ref_wr_duration array. */ static void update_thr_responsiveness_params(struct bfq_data *bfqd) { - int dev_type = blk_queue_nonrot(bfqd->queue); - - if (bfqd->bfq_user_max_budget == 0) + if (bfqd->bfq_user_max_budget == 0) { bfqd->bfq_max_budget = bfq_calc_max_budget(bfqd); - - if (bfqd->device_speed == BFQ_BFQD_FAST && - bfqd->peak_rate < device_speed_thresh[dev_type]) { - bfqd->device_speed = BFQ_BFQD_SLOW; - bfqd->RT_prod = R_slow[dev_type] * - T_slow[dev_type]; - } else if (bfqd->device_speed == BFQ_BFQD_SLOW && - bfqd->peak_rate > device_speed_thresh[dev_type]) { - bfqd->device_speed = BFQ_BFQD_FAST; - bfqd->RT_prod = R_fast[dev_type] * - T_fast[dev_type]; + bfq_log(bfqd, "new max_budget = %d", bfqd->bfq_max_budget); } - - bfq_log(bfqd, -"dev_type %s dev_speed_class = %s (%llu sects/sec), thresh %llu setcs/sec", - dev_type == 0 ? "ROT" : "NONROT", - bfqd->device_speed == BFQ_BFQD_FAST ? "FAST" : "SLOW", - bfqd->device_speed == BFQ_BFQD_FAST ? - (USEC_PER_SEC*(u64)R_fast[dev_type])>>BFQ_RATE_SHIFT : - (USEC_PER_SEC*(u64)R_slow[dev_type])>>BFQ_RATE_SHIFT, - (USEC_PER_SEC*(u64)device_speed_thresh[dev_type])>> - BFQ_RATE_SHIFT); } static void bfq_reset_rate_computation(struct bfq_data *bfqd, @@ -3265,23 +3213,6 @@ void bfq_bfqq_expire(struct bfq_data *bfqd, bfqq->soft_rt_next_start = bfq_bfqq_softrt_next_start(bfqd, bfqq); else { - /* - * The application is still waiting for the - * completion of one or more requests: - * prevent it from possibly being incorrectly - * deemed as soft real-time by setting its - * soft_rt_next_start to infinity. In fact, - * without this assignment, the application - * would be incorrectly deemed as soft - * real-time if: - * 1) it issued a new request before the - * completion of all its in-flight - * requests, and - * 2) at that time, its soft_rt_next_start - * happened to be in the past. - */ - bfqq->soft_rt_next_start = - bfq_greatest_from_now(); /* * Schedule an update of soft_rt_next_start to when * the task may be discovered to be isochronous. @@ -4540,14 +4471,12 @@ static inline void bfq_update_insert_stats(struct request_queue *q, unsigned int cmd_flags) {} #endif -static void bfq_prepare_request(struct request *rq, struct bio *bio); - static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, bool at_head) { struct request_queue *q = hctx->queue; struct bfq_data *bfqd = q->elevator->elevator_data; - struct bfq_queue *bfqq = RQ_BFQQ(rq); + struct bfq_queue *bfqq; bool idle_timer_disabled = false; unsigned int cmd_flags; @@ -4562,24 +4491,13 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, blk_mq_sched_request_inserted(rq); spin_lock_irq(&bfqd->lock); + bfqq = bfq_init_rq(rq); if (at_head || blk_rq_is_passthrough(rq)) { if (at_head) list_add(&rq->queuelist, &bfqd->dispatch); else list_add_tail(&rq->queuelist, &bfqd->dispatch); - } else { - if (WARN_ON_ONCE(!bfqq)) { - /* - * This should never happen. Most likely rq is - * a requeued regular request, being - * re-inserted without being first - * re-prepared. Do a prepare, to avoid - * failure. - */ - bfq_prepare_request(rq, rq->bio); - bfqq = RQ_BFQQ(rq); - } - + } else { /* bfqq is assumed to be non null here */ idle_timer_disabled = __bfq_insert_request(bfqd, rq); /* * Update bfqq, because, if a queue merge has occurred @@ -4778,8 +4696,8 @@ static void bfq_finish_requeue_request(struct request *rq) if (rq->rq_flags & RQF_STARTED) bfqg_stats_update_completion(bfqq_group(bfqq), - rq_start_time_ns(rq), - rq_io_start_time_ns(rq), + rq->start_time_ns, + rq->io_start_time_ns, rq->cmd_flags); if (likely(rq->rq_flags & RQF_STARTED)) { @@ -4922,11 +4840,48 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, } /* - * Allocate bfq data structures associated with this request. + * Only reset private fields. The actual request preparation will be + * performed by bfq_init_rq, when rq is either inserted or merged. See + * comments on bfq_init_rq for the reason behind this delayed + * preparation. */ static void bfq_prepare_request(struct request *rq, struct bio *bio) +{ + /* + * Regardless of whether we have an icq attached, we have to + * clear the scheduler pointers, as they might point to + * previously allocated bic/bfqq structs. + */ + rq->elv.priv[0] = rq->elv.priv[1] = NULL; +} + +/* + * If needed, init rq, allocate bfq data structures associated with + * rq, and increment reference counters in the destination bfq_queue + * for rq. Return the destination bfq_queue for rq, or NULL is rq is + * not associated with any bfq_queue. + * + * This function is invoked by the functions that perform rq insertion + * or merging. One may have expected the above preparation operations + * to be performed in bfq_prepare_request, and not delayed to when rq + * is inserted or merged. The rationale behind this delayed + * preparation is that, after the prepare_request hook is invoked for + * rq, rq may still be transformed into a request with no icq, i.e., a + * request not associated with any queue. No bfq hook is invoked to + * signal this tranformation. As a consequence, should these + * preparation operations be performed when the prepare_request hook + * is invoked, and should rq be transformed one moment later, bfq + * would end up in an inconsistent state, because it would have + * incremented some queue counters for an rq destined to + * transformation, without any chance to correctly lower these + * counters back. In contrast, no transformation can still happen for + * rq after rq has been inserted or merged. So, it is safe to execute + * these preparation operations when rq is finally inserted or merged. + */ +static struct bfq_queue *bfq_init_rq(struct request *rq) { struct request_queue *q = rq->q; + struct bio *bio = rq->bio; struct bfq_data *bfqd = q->elevator->elevator_data; struct bfq_io_cq *bic; const int is_sync = rq_is_sync(rq); @@ -4934,20 +4889,21 @@ static void bfq_prepare_request(struct request *rq, struct bio *bio) bool new_queue = false; bool bfqq_already_existing = false, split = false; + if (unlikely(!rq->elv.icq)) + return NULL; + /* - * Even if we don't have an icq attached, we should still clear - * the scheduler pointers, as they might point to previously - * allocated bic/bfqq structs. + * Assuming that elv.priv[1] is set only if everything is set + * for this rq. This holds true, because this function is + * invoked only for insertion or merging, and, after such + * events, a request cannot be manipulated any longer before + * being removed from bfq. */ - if (!rq->elv.icq) { - rq->elv.priv[0] = rq->elv.priv[1] = NULL; - return; - } + if (rq->elv.priv[1]) + return rq->elv.priv[1]; bic = icq_to_bic(rq->elv.icq); - spin_lock_irq(&bfqd->lock); - bfq_check_ioprio_change(bic, bio); bfq_bic_update_cgroup(bic, bio); @@ -5006,7 +4962,7 @@ static void bfq_prepare_request(struct request *rq, struct bio *bio) if (unlikely(bfq_bfqq_just_created(bfqq))) bfq_handle_burst(bfqd, bfqq); - spin_unlock_irq(&bfqd->lock); + return bfqq; } static void bfq_idle_slice_timer_body(struct bfq_queue *bfqq) @@ -5105,6 +5061,64 @@ void bfq_put_async_queues(struct bfq_data *bfqd, struct bfq_group *bfqg) __bfq_put_async_bfqq(bfqd, &bfqg->async_idle_bfqq); } +/* + * See the comments on bfq_limit_depth for the purpose of + * the depths set in the function. Return minimum shallow depth we'll use. + */ +static unsigned int bfq_update_depths(struct bfq_data *bfqd, + struct sbitmap_queue *bt) +{ + unsigned int i, j, min_shallow = UINT_MAX; + + /* + * In-word depths if no bfq_queue is being weight-raised: + * leaving 25% of tags only for sync reads. + * + * In next formulas, right-shift the value + * (1U<sb.shift), instead of computing directly + * (1U<<(bt->sb.shift - something)), to be robust against + * any possible value of bt->sb.shift, without having to + * limit 'something'. + */ + /* no more than 50% of tags for async I/O */ + bfqd->word_depths[0][0] = max((1U << bt->sb.shift) >> 1, 1U); + /* + * no more than 75% of tags for sync writes (25% extra tags + * w.r.t. async I/O, to prevent async I/O from starving sync + * writes) + */ + bfqd->word_depths[0][1] = max(((1U << bt->sb.shift) * 3) >> 2, 1U); + + /* + * In-word depths in case some bfq_queue is being weight- + * raised: leaving ~63% of tags for sync reads. This is the + * highest percentage for which, in our tests, application + * start-up times didn't suffer from any regression due to tag + * shortage. + */ + /* no more than ~18% of tags for async I/O */ + bfqd->word_depths[1][0] = max(((1U << bt->sb.shift) * 3) >> 4, 1U); + /* no more than ~37% of tags for sync writes (~20% extra tags) */ + bfqd->word_depths[1][1] = max(((1U << bt->sb.shift) * 6) >> 4, 1U); + + for (i = 0; i < 2; i++) + for (j = 0; j < 2; j++) + min_shallow = min(min_shallow, bfqd->word_depths[i][j]); + + return min_shallow; +} + +static int bfq_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int index) +{ + struct bfq_data *bfqd = hctx->queue->elevator->elevator_data; + struct blk_mq_tags *tags = hctx->sched_tags; + unsigned int min_shallow; + + min_shallow = bfq_update_depths(bfqd, &tags->bitmap_tags); + sbitmap_queue_min_shallow_depth(&tags->bitmap_tags, min_shallow); + return 0; +} + static void bfq_exit_queue(struct elevator_queue *e) { struct bfq_data *bfqd = e->elevator_data; @@ -5242,14 +5256,12 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) bfqd->wr_busy_queues = 0; /* - * Begin by assuming, optimistically, that the device is a - * high-speed one, and that its peak rate is equal to 2/3 of - * the highest reference rate. + * Begin by assuming, optimistically, that the device peak + * rate is equal to 2/3 of the highest reference rate. */ - bfqd->RT_prod = R_fast[blk_queue_nonrot(bfqd->queue)] * - T_fast[blk_queue_nonrot(bfqd->queue)]; - bfqd->peak_rate = R_fast[blk_queue_nonrot(bfqd->queue)] * 2 / 3; - bfqd->device_speed = BFQ_BFQD_FAST; + bfqd->rate_dur_prod = ref_rate[blk_queue_nonrot(bfqd->queue)] * + ref_wr_duration[blk_queue_nonrot(bfqd->queue)]; + bfqd->peak_rate = ref_rate[blk_queue_nonrot(bfqd->queue)] * 2 / 3; spin_lock_init(&bfqd->lock); @@ -5526,6 +5538,7 @@ static struct elevator_type iosched_bfq_mq = { .requests_merged = bfq_requests_merged, .request_merged = bfq_request_merged, .has_work = bfq_has_work, + .init_hctx = bfq_init_hctx, .init_sched = bfq_init_queue, .exit_sched = bfq_exit_queue, }, @@ -5556,8 +5569,8 @@ static int __init bfq_init(void) /* * Times to load large popular applications for the typical * systems installed on the reference devices (see the - * comments before the definitions of the next two - * arrays). Actually, we use slightly slower values, as the + * comments before the definition of the next + * array). Actually, we use slightly lower values, as the * estimated peak rate tends to be smaller than the actual * peak rate. The reason for this last fact is that estimates * are computed over much shorter time intervals than the long @@ -5566,25 +5579,8 @@ static int __init bfq_init(void) * scheduler cannot rely on a peak-rate-evaluation workload to * be run for a long time. */ - T_slow[0] = msecs_to_jiffies(3500); /* actually 4 sec */ - T_slow[1] = msecs_to_jiffies(6000); /* actually 6.5 sec */ - T_fast[0] = msecs_to_jiffies(7000); /* actually 8 sec */ - T_fast[1] = msecs_to_jiffies(2500); /* actually 3 sec */ - - /* - * Thresholds that determine the switch between speed classes - * (see the comments before the definition of the array - * device_speed_thresh). These thresholds are biased towards - * transitions to the fast class. This is safer than the - * opposite bias. In fact, a wrong transition to the slow - * class results in short weight-raising periods, because the - * speed of the device then tends to be higher that the - * reference peak rate. On the opposite end, a wrong - * transition to the fast class tends to increase - * weight-raising periods, because of the opposite reason. - */ - device_speed_thresh[0] = (4 * R_slow[0]) / 3; - device_speed_thresh[1] = (4 * R_slow[1]) / 3; + ref_wr_duration[0] = msecs_to_jiffies(7000); /* actually 8 sec */ + ref_wr_duration[1] = msecs_to_jiffies(2500); /* actually 3 sec */ ret = elv_register(&iosched_bfq_mq); if (ret) diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index ae2f3dadec44..0f712e03b035 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -399,11 +399,6 @@ struct bfq_io_cq { struct bfq_ttime saved_ttime; }; -enum bfq_device_speed { - BFQ_BFQD_FAST, - BFQ_BFQD_SLOW, -}; - /** * struct bfq_data - per-device data structure. * @@ -611,12 +606,11 @@ struct bfq_data { /* Max service-rate for a soft real-time queue, in sectors/sec */ unsigned int bfq_wr_max_softrt_rate; /* - * Cached value of the product R*T, used for computing the - * maximum duration of weight raising automatically. + * Cached value of the product ref_rate*ref_wr_duration, used + * for computing the maximum duration of weight raising + * automatically. */ - u64 RT_prod; - /* device-speed class for the low-latency heuristic */ - enum bfq_device_speed device_speed; + u64 rate_dur_prod; /* fallback dummy bfqq for extreme OOM conditions */ struct bfq_queue oom_bfqq; @@ -635,12 +629,6 @@ struct bfq_data { /* bfqq associated with the task issuing current bio for merging */ struct bfq_queue *bio_bfqq; - /* - * Cached sbitmap shift, used to compute depth limits in - * bfq_update_depths. - */ - unsigned int sb_shift; - /* * Depth limits used in bfq_limit_depth (see comments on the * function) @@ -732,9 +720,9 @@ struct bfqg_stats { /* total time with empty current active q with other requests queued */ struct blkg_stat empty_time; /* fields after this shouldn't be cleared on stat reset */ - uint64_t start_group_wait_time; - uint64_t start_idle_time; - uint64_t start_empty_time; + u64 start_group_wait_time; + u64 start_idle_time; + u64 start_empty_time; uint16_t flags; #endif /* CONFIG_BFQ_GROUP_IOSCHED && CONFIG_DEBUG_BLK_CGROUP */ }; @@ -856,8 +844,8 @@ void bfqg_stats_update_io_add(struct bfq_group *bfqg, struct bfq_queue *bfqq, unsigned int op); void bfqg_stats_update_io_remove(struct bfq_group *bfqg, unsigned int op); void bfqg_stats_update_io_merged(struct bfq_group *bfqg, unsigned int op); -void bfqg_stats_update_completion(struct bfq_group *bfqg, uint64_t start_time, - uint64_t io_start_time, unsigned int op); +void bfqg_stats_update_completion(struct bfq_group *bfqg, u64 start_time_ns, + u64 io_start_time_ns, unsigned int op); void bfqg_stats_update_dequeue(struct bfq_group *bfqg); void bfqg_stats_set_start_empty_time(struct bfq_group *bfqg); void bfqg_stats_update_idle_time(struct bfq_group *bfqg); diff --git a/block/bio-integrity.c b/block/bio-integrity.c index 9cfdd6c83b5b..add7c7c85335 100644 --- a/block/bio-integrity.c +++ b/block/bio-integrity.c @@ -56,12 +56,12 @@ struct bio_integrity_payload *bio_integrity_alloc(struct bio *bio, struct bio_set *bs = bio->bi_pool; unsigned inline_vecs; - if (!bs || !bs->bio_integrity_pool) { + if (!bs || !mempool_initialized(&bs->bio_integrity_pool)) { bip = kmalloc(sizeof(struct bio_integrity_payload) + sizeof(struct bio_vec) * nr_vecs, gfp_mask); inline_vecs = nr_vecs; } else { - bip = mempool_alloc(bs->bio_integrity_pool, gfp_mask); + bip = mempool_alloc(&bs->bio_integrity_pool, gfp_mask); inline_vecs = BIP_INLINE_VECS; } @@ -74,7 +74,7 @@ struct bio_integrity_payload *bio_integrity_alloc(struct bio *bio, unsigned long idx = 0; bip->bip_vec = bvec_alloc(gfp_mask, nr_vecs, &idx, - bs->bvec_integrity_pool); + &bs->bvec_integrity_pool); if (!bip->bip_vec) goto err; bip->bip_max_vcnt = bvec_nr_vecs(idx); @@ -90,7 +90,7 @@ struct bio_integrity_payload *bio_integrity_alloc(struct bio *bio, return bip; err: - mempool_free(bip, bs->bio_integrity_pool); + mempool_free(bip, &bs->bio_integrity_pool); return ERR_PTR(-ENOMEM); } EXPORT_SYMBOL(bio_integrity_alloc); @@ -111,10 +111,10 @@ static void bio_integrity_free(struct bio *bio) kfree(page_address(bip->bip_vec->bv_page) + bip->bip_vec->bv_offset); - if (bs && bs->bio_integrity_pool) { - bvec_free(bs->bvec_integrity_pool, bip->bip_vec, bip->bip_slab); + if (bs && mempool_initialized(&bs->bio_integrity_pool)) { + bvec_free(&bs->bvec_integrity_pool, bip->bip_vec, bip->bip_slab); - mempool_free(bip, bs->bio_integrity_pool); + mempool_free(bip, &bs->bio_integrity_pool); } else { kfree(bip); } @@ -465,16 +465,15 @@ EXPORT_SYMBOL(bio_integrity_clone); int bioset_integrity_create(struct bio_set *bs, int pool_size) { - if (bs->bio_integrity_pool) + if (mempool_initialized(&bs->bio_integrity_pool)) return 0; - bs->bio_integrity_pool = mempool_create_slab_pool(pool_size, bip_slab); - if (!bs->bio_integrity_pool) + if (mempool_init_slab_pool(&bs->bio_integrity_pool, + pool_size, bip_slab)) return -1; - bs->bvec_integrity_pool = biovec_create_pool(pool_size); - if (!bs->bvec_integrity_pool) { - mempool_destroy(bs->bio_integrity_pool); + if (biovec_init_pool(&bs->bvec_integrity_pool, pool_size)) { + mempool_exit(&bs->bio_integrity_pool); return -1; } @@ -484,8 +483,8 @@ EXPORT_SYMBOL(bioset_integrity_create); void bioset_integrity_free(struct bio_set *bs) { - mempool_destroy(bs->bio_integrity_pool); - mempool_destroy(bs->bvec_integrity_pool); + mempool_exit(&bs->bio_integrity_pool); + mempool_exit(&bs->bvec_integrity_pool); } EXPORT_SYMBOL(bioset_integrity_free); diff --git a/block/bio.c b/block/bio.c index 53e0f0a1ed94..9710e275f230 100644 --- a/block/bio.c +++ b/block/bio.c @@ -53,7 +53,7 @@ static struct biovec_slab bvec_slabs[BVEC_POOL_NR] __read_mostly = { * fs_bio_set is the bio_set containing bio and iovec memory pools used by * IO code that does not need private memory pools. */ -struct bio_set *fs_bio_set; +struct bio_set fs_bio_set; EXPORT_SYMBOL(fs_bio_set); /* @@ -254,7 +254,7 @@ static void bio_free(struct bio *bio) bio_uninit(bio); if (bs) { - bvec_free(bs->bvec_pool, bio->bi_io_vec, BVEC_POOL_IDX(bio)); + bvec_free(&bs->bvec_pool, bio->bi_io_vec, BVEC_POOL_IDX(bio)); /* * If we have front padding, adjust the bio pointer before freeing @@ -262,7 +262,7 @@ static void bio_free(struct bio *bio) p = bio; p -= bs->front_pad; - mempool_free(p, bs->bio_pool); + mempool_free(p, &bs->bio_pool); } else { /* Bio was allocated by bio_kmalloc() */ kfree(bio); @@ -454,7 +454,8 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned int nr_iovecs, inline_vecs = nr_iovecs; } else { /* should not use nobvec bioset for nr_iovecs > 0 */ - if (WARN_ON_ONCE(!bs->bvec_pool && nr_iovecs > 0)) + if (WARN_ON_ONCE(!mempool_initialized(&bs->bvec_pool) && + nr_iovecs > 0)) return NULL; /* * generic_make_request() converts recursion to iteration; this @@ -483,11 +484,11 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned int nr_iovecs, bs->rescue_workqueue) gfp_mask &= ~__GFP_DIRECT_RECLAIM; - p = mempool_alloc(bs->bio_pool, gfp_mask); + p = mempool_alloc(&bs->bio_pool, gfp_mask); if (!p && gfp_mask != saved_gfp) { punt_bios_to_rescuer(bs); gfp_mask = saved_gfp; - p = mempool_alloc(bs->bio_pool, gfp_mask); + p = mempool_alloc(&bs->bio_pool, gfp_mask); } front_pad = bs->front_pad; @@ -503,11 +504,11 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned int nr_iovecs, if (nr_iovecs > inline_vecs) { unsigned long idx = 0; - bvl = bvec_alloc(gfp_mask, nr_iovecs, &idx, bs->bvec_pool); + bvl = bvec_alloc(gfp_mask, nr_iovecs, &idx, &bs->bvec_pool); if (!bvl && gfp_mask != saved_gfp) { punt_bios_to_rescuer(bs); gfp_mask = saved_gfp; - bvl = bvec_alloc(gfp_mask, nr_iovecs, &idx, bs->bvec_pool); + bvl = bvec_alloc(gfp_mask, nr_iovecs, &idx, &bs->bvec_pool); } if (unlikely(!bvl)) @@ -524,25 +525,25 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned int nr_iovecs, return bio; err_free: - mempool_free(p, bs->bio_pool); + mempool_free(p, &bs->bio_pool); return NULL; } EXPORT_SYMBOL(bio_alloc_bioset); -void zero_fill_bio(struct bio *bio) +void zero_fill_bio_iter(struct bio *bio, struct bvec_iter start) { unsigned long flags; struct bio_vec bv; struct bvec_iter iter; - bio_for_each_segment(bv, bio, iter) { + __bio_for_each_segment(bv, bio, iter, start) { char *data = bvec_kmap_irq(&bv, &flags); memset(data, 0, bv.bv_len); flush_dcache_page(bv.bv_page); bvec_kunmap_irq(data, &flags); } } -EXPORT_SYMBOL(zero_fill_bio); +EXPORT_SYMBOL(zero_fill_bio_iter); /** * bio_put - release a reference to a bio @@ -773,7 +774,7 @@ int bio_add_pc_page(struct request_queue *q, struct bio *bio, struct page return 0; } - if (bio->bi_vcnt >= bio->bi_max_vecs) + if (bio_full(bio)) return 0; /* @@ -821,52 +822,82 @@ int bio_add_pc_page(struct request_queue *q, struct bio *bio, struct page EXPORT_SYMBOL(bio_add_pc_page); /** - * bio_add_page - attempt to add page to bio - * @bio: destination bio - * @page: page to add - * @len: vec entry length - * @offset: vec entry offset + * __bio_try_merge_page - try appending data to an existing bvec. + * @bio: destination bio + * @page: page to add + * @len: length of the data to add + * @off: offset of the data in @page * - * Attempt to add a page to the bio_vec maplist. This will only fail - * if either bio->bi_vcnt == bio->bi_max_vecs or it's a cloned bio. + * Try to add the data at @page + @off to the last bvec of @bio. This is a + * a useful optimisation for file systems with a block size smaller than the + * page size. + * + * Return %true on success or %false on failure. */ -int bio_add_page(struct bio *bio, struct page *page, - unsigned int len, unsigned int offset) +bool __bio_try_merge_page(struct bio *bio, struct page *page, + unsigned int len, unsigned int off) { - struct bio_vec *bv; - - /* - * cloned bio must not modify vec list - */ if (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED))) - return 0; + return false; - /* - * For filesystems with a blocksize smaller than the pagesize - * we will often be called with the same page as last time and - * a consecutive offset. Optimize this special case. - */ if (bio->bi_vcnt > 0) { - bv = &bio->bi_io_vec[bio->bi_vcnt - 1]; + struct bio_vec *bv = &bio->bi_io_vec[bio->bi_vcnt - 1]; - if (page == bv->bv_page && - offset == bv->bv_offset + bv->bv_len) { + if (page == bv->bv_page && off == bv->bv_offset + bv->bv_len) { bv->bv_len += len; - goto done; + bio->bi_iter.bi_size += len; + return true; } } + return false; +} +EXPORT_SYMBOL_GPL(__bio_try_merge_page); - if (bio->bi_vcnt >= bio->bi_max_vecs) - return 0; +/** + * __bio_add_page - add page to a bio in a new segment + * @bio: destination bio + * @page: page to add + * @len: length of the data to add + * @off: offset of the data in @page + * + * Add the data at @page + @off to @bio as a new bvec. The caller must ensure + * that @bio has space for another bvec. + */ +void __bio_add_page(struct bio *bio, struct page *page, + unsigned int len, unsigned int off) +{ + struct bio_vec *bv = &bio->bi_io_vec[bio->bi_vcnt]; - bv = &bio->bi_io_vec[bio->bi_vcnt]; - bv->bv_page = page; - bv->bv_len = len; - bv->bv_offset = offset; + WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)); + WARN_ON_ONCE(bio_full(bio)); + + bv->bv_page = page; + bv->bv_offset = off; + bv->bv_len = len; - bio->bi_vcnt++; -done: bio->bi_iter.bi_size += len; + bio->bi_vcnt++; +} +EXPORT_SYMBOL_GPL(__bio_add_page); + +/** + * bio_add_page - attempt to add page to bio + * @bio: destination bio + * @page: page to add + * @len: vec entry length + * @offset: vec entry offset + * + * Attempt to add a page to the bio_vec maplist. This will only fail + * if either bio->bi_vcnt == bio->bi_max_vecs or it's a cloned bio. + */ +int bio_add_page(struct bio *bio, struct page *page, + unsigned int len, unsigned int offset) +{ + if (!__bio_try_merge_page(bio, page, len, offset)) { + if (bio_full(bio)) + return 0; + __bio_add_page(bio, page, len, offset); + } return len; } EXPORT_SYMBOL(bio_add_page); @@ -970,27 +1001,68 @@ void bio_advance(struct bio *bio, unsigned bytes) } EXPORT_SYMBOL(bio_advance); +void bio_copy_data_iter(struct bio *dst, struct bvec_iter *dst_iter, + struct bio *src, struct bvec_iter *src_iter) +{ + struct bio_vec src_bv, dst_bv; + void *src_p, *dst_p; + unsigned bytes; + + while (src_iter->bi_size && dst_iter->bi_size) { + src_bv = bio_iter_iovec(src, *src_iter); + dst_bv = bio_iter_iovec(dst, *dst_iter); + + bytes = min(src_bv.bv_len, dst_bv.bv_len); + + src_p = kmap_atomic(src_bv.bv_page); + dst_p = kmap_atomic(dst_bv.bv_page); + + memcpy(dst_p + dst_bv.bv_offset, + src_p + src_bv.bv_offset, + bytes); + + kunmap_atomic(dst_p); + kunmap_atomic(src_p); + + flush_dcache_page(dst_bv.bv_page); + + bio_advance_iter(src, src_iter, bytes); + bio_advance_iter(dst, dst_iter, bytes); + } +} +EXPORT_SYMBOL(bio_copy_data_iter); + /** - * bio_copy_data - copy contents of data buffers from one chain of bios to - * another - * @src: source bio list - * @dst: destination bio list - * - * If @src and @dst are single bios, bi_next must be NULL - otherwise, treats - * @src and @dst as linked lists of bios. + * bio_copy_data - copy contents of data buffers from one bio to another + * @src: source bio + * @dst: destination bio * * Stops when it reaches the end of either @src or @dst - that is, copies * min(src->bi_size, dst->bi_size) bytes (or the equivalent for lists of bios). */ void bio_copy_data(struct bio *dst, struct bio *src) { - struct bvec_iter src_iter, dst_iter; - struct bio_vec src_bv, dst_bv; - void *src_p, *dst_p; - unsigned bytes; + struct bvec_iter src_iter = src->bi_iter; + struct bvec_iter dst_iter = dst->bi_iter; - src_iter = src->bi_iter; - dst_iter = dst->bi_iter; + bio_copy_data_iter(dst, &dst_iter, src, &src_iter); +} +EXPORT_SYMBOL(bio_copy_data); + +/** + * bio_list_copy_data - copy contents of data buffers from one chain of bios to + * another + * @src: source bio list + * @dst: destination bio list + * + * Stops when it reaches the end of either the @src list or @dst list - that is, + * copies min(src->bi_size, dst->bi_size) bytes (or the equivalent for lists of + * bios). + */ +void bio_list_copy_data(struct bio *dst, struct bio *src) +{ + struct bvec_iter src_iter = src->bi_iter; + struct bvec_iter dst_iter = dst->bi_iter; while (1) { if (!src_iter.bi_size) { @@ -1009,26 +1081,10 @@ void bio_copy_data(struct bio *dst, struct bio *src) dst_iter = dst->bi_iter; } - src_bv = bio_iter_iovec(src, src_iter); - dst_bv = bio_iter_iovec(dst, dst_iter); - - bytes = min(src_bv.bv_len, dst_bv.bv_len); - - src_p = kmap_atomic(src_bv.bv_page); - dst_p = kmap_atomic(dst_bv.bv_page); - - memcpy(dst_p + dst_bv.bv_offset, - src_p + src_bv.bv_offset, - bytes); - - kunmap_atomic(dst_p); - kunmap_atomic(src_p); - - bio_advance_iter(src, &src_iter, bytes); - bio_advance_iter(dst, &dst_iter, bytes); + bio_copy_data_iter(dst, &dst_iter, src, &src_iter); } } -EXPORT_SYMBOL(bio_copy_data); +EXPORT_SYMBOL(bio_list_copy_data); struct bio_map_data { int is_our_pages; @@ -1584,6 +1640,7 @@ void bio_set_pages_dirty(struct bio *bio) set_page_dirty_lock(page); } } +EXPORT_SYMBOL_GPL(bio_set_pages_dirty); static void bio_release_pages(struct bio *bio) { @@ -1667,6 +1724,7 @@ void bio_check_pages_dirty(struct bio *bio) bio_put(bio); } } +EXPORT_SYMBOL_GPL(bio_check_pages_dirty); void generic_start_io_acct(struct request_queue *q, int rw, unsigned long sectors, struct hd_struct *part) @@ -1749,6 +1807,9 @@ again: if (!bio_integrity_endio(bio)) return; + if (WARN_ONCE(bio->bi_next, "driver left bi_next not NULL")) + bio->bi_next = NULL; + /* * Need to have a real endio function for chained bios, otherwise * various corner cases will break (like stacking block devices that @@ -1848,30 +1909,38 @@ EXPORT_SYMBOL_GPL(bio_trim); * create memory pools for biovec's in a bio_set. * use the global biovec slabs created for general use. */ -mempool_t *biovec_create_pool(int pool_entries) +int biovec_init_pool(mempool_t *pool, int pool_entries) { struct biovec_slab *bp = bvec_slabs + BVEC_POOL_MAX; - return mempool_create_slab_pool(pool_entries, bp->slab); + return mempool_init_slab_pool(pool, pool_entries, bp->slab); } -void bioset_free(struct bio_set *bs) +/* + * bioset_exit - exit a bioset initialized with bioset_init() + * + * May be called on a zeroed but uninitialized bioset (i.e. allocated with + * kzalloc()). + */ +void bioset_exit(struct bio_set *bs) { if (bs->rescue_workqueue) destroy_workqueue(bs->rescue_workqueue); + bs->rescue_workqueue = NULL; - mempool_destroy(bs->bio_pool); - mempool_destroy(bs->bvec_pool); + mempool_exit(&bs->bio_pool); + mempool_exit(&bs->bvec_pool); bioset_integrity_free(bs); - bio_put_slab(bs); - - kfree(bs); + if (bs->bio_slab) + bio_put_slab(bs); + bs->bio_slab = NULL; } -EXPORT_SYMBOL(bioset_free); +EXPORT_SYMBOL(bioset_exit); /** - * bioset_create - Create a bio_set + * bioset_init - Initialize a bio_set + * @bs: pool to initialize * @pool_size: Number of bio and bio_vecs to cache in the mempool * @front_pad: Number of bytes to allocate in front of the returned bio * @flags: Flags to modify behavior, currently %BIOSET_NEED_BVECS @@ -1890,16 +1959,12 @@ EXPORT_SYMBOL(bioset_free); * dispatch queued requests when the mempool runs out of space. * */ -struct bio_set *bioset_create(unsigned int pool_size, - unsigned int front_pad, - int flags) +int bioset_init(struct bio_set *bs, + unsigned int pool_size, + unsigned int front_pad, + int flags) { unsigned int back_pad = BIO_INLINE_VECS * sizeof(struct bio_vec); - struct bio_set *bs; - - bs = kzalloc(sizeof(*bs), GFP_KERNEL); - if (!bs) - return NULL; bs->front_pad = front_pad; @@ -1908,34 +1973,47 @@ struct bio_set *bioset_create(unsigned int pool_size, INIT_WORK(&bs->rescue_work, bio_alloc_rescue); bs->bio_slab = bio_find_or_create_slab(front_pad + back_pad); - if (!bs->bio_slab) { - kfree(bs); - return NULL; - } + if (!bs->bio_slab) + return -ENOMEM; - bs->bio_pool = mempool_create_slab_pool(pool_size, bs->bio_slab); - if (!bs->bio_pool) + if (mempool_init_slab_pool(&bs->bio_pool, pool_size, bs->bio_slab)) goto bad; - if (flags & BIOSET_NEED_BVECS) { - bs->bvec_pool = biovec_create_pool(pool_size); - if (!bs->bvec_pool) - goto bad; - } + if ((flags & BIOSET_NEED_BVECS) && + biovec_init_pool(&bs->bvec_pool, pool_size)) + goto bad; if (!(flags & BIOSET_NEED_RESCUER)) - return bs; + return 0; bs->rescue_workqueue = alloc_workqueue("bioset", WQ_MEM_RECLAIM, 0); if (!bs->rescue_workqueue) goto bad; - return bs; + return 0; bad: - bioset_free(bs); - return NULL; + bioset_exit(bs); + return -ENOMEM; +} +EXPORT_SYMBOL(bioset_init); + +/* + * Initialize and setup a new bio_set, based on the settings from + * another bio_set. + */ +int bioset_init_from_src(struct bio_set *bs, struct bio_set *src) +{ + int flags; + + flags = 0; + if (src->bvec_pool.min_nr) + flags |= BIOSET_NEED_BVECS; + if (src->rescue_workqueue) + flags |= BIOSET_NEED_RESCUER; + + return bioset_init(bs, src->bio_pool.min_nr, src->front_pad, flags); } -EXPORT_SYMBOL(bioset_create); +EXPORT_SYMBOL(bioset_init_from_src); #ifdef CONFIG_BLK_CGROUP @@ -2013,18 +2091,18 @@ static int __init init_bio(void) { bio_slab_max = 2; bio_slab_nr = 0; - bio_slabs = kzalloc(bio_slab_max * sizeof(struct bio_slab), GFP_KERNEL); + bio_slabs = kcalloc(bio_slab_max, sizeof(struct bio_slab), + GFP_KERNEL); if (!bio_slabs) panic("bio: can't allocate bios\n"); bio_integrity_init(); biovec_init_slabs(); - fs_bio_set = bioset_create(BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS); - if (!fs_bio_set) + if (bioset_init(&fs_bio_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS)) panic("bio: can't allocate bios\n"); - if (bioset_integrity_create(fs_bio_set, BIO_POOL_SIZE)) + if (bioset_integrity_create(&fs_bio_set, BIO_POOL_SIZE)) panic("bio: can't create integrity pool\n"); return 0; diff --git a/block/blk-core.c b/block/blk-core.c index 85909b431eb0..cf0ee764b908 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -196,15 +196,8 @@ void blk_rq_init(struct request_queue *q, struct request *rq) RB_CLEAR_NODE(&rq->rb_node); rq->tag = -1; rq->internal_tag = -1; - rq->start_time = jiffies; - set_start_time_ns(rq); + rq->start_time_ns = ktime_get_ns(); rq->part = NULL; - seqcount_init(&rq->gstate_seq); - u64_stats_init(&rq->aborted_gstate_sync); - /* - * See comment of blk_mq_init_request - */ - WRITE_ONCE(rq->gstate, MQ_RQ_GEN_INC); } EXPORT_SYMBOL(blk_rq_init); @@ -280,6 +273,10 @@ static void req_bio_endio(struct request *rq, struct bio *bio, bio_advance(bio, nbytes); /* don't actually finish bio if it's part of flush sequence */ + /* + * XXX this code looks suspicious - it's not consistent with advancing + * req->bio in caller + */ if (bio->bi_iter.bi_size == 0 && !(rq->rq_flags & RQF_FLUSH_SEQ)) bio_endio(bio); } @@ -360,7 +357,6 @@ EXPORT_SYMBOL(blk_start_queue_async); void blk_start_queue(struct request_queue *q) { lockdep_assert_held(q->queue_lock); - WARN_ON(!in_interrupt() && !irqs_disabled()); WARN_ON_ONCE(q->mq_ops); queue_flag_clear(QUEUE_FLAG_STOPPED, q); @@ -996,18 +992,24 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id, spinlock_t *lock) { struct request_queue *q; + int ret; q = kmem_cache_alloc_node(blk_requestq_cachep, gfp_mask | __GFP_ZERO, node_id); if (!q) return NULL; + INIT_LIST_HEAD(&q->queue_head); + q->last_merge = NULL; + q->end_sector = 0; + q->boundary_rq = NULL; + q->id = ida_simple_get(&blk_queue_ida, 0, 0, gfp_mask); if (q->id < 0) goto fail_q; - q->bio_split = bioset_create(BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS); - if (!q->bio_split) + ret = bioset_init(&q->bio_split, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS); + if (ret) goto fail_id; q->backing_dev_info = bdi_alloc_node(gfp_mask, node_id); @@ -1079,7 +1081,7 @@ fail_bdi: fail_stats: bdi_put(q->backing_dev_info); fail_split: - bioset_free(q->bio_split); + bioset_exit(&q->bio_split); fail_id: ida_simple_remove(&blk_queue_ida, q->id); fail_q: @@ -1173,16 +1175,8 @@ int blk_init_allocated_queue(struct request_queue *q) q->sg_reserved_size = INT_MAX; - /* Protect q->elevator from elevator_change */ - mutex_lock(&q->sysfs_lock); - - /* init elevator */ - if (elevator_init(q, NULL)) { - mutex_unlock(&q->sysfs_lock); + if (elevator_init(q)) goto out_exit_flush_rq; - } - - mutex_unlock(&q->sysfs_lock); return 0; out_exit_flush_rq: @@ -1334,6 +1328,7 @@ int blk_update_nr_requests(struct request_queue *q, unsigned int nr) * @op: operation and flags * @bio: bio to allocate request for (can be %NULL) * @flags: BLQ_MQ_REQ_* flags + * @gfp_mask: allocator flags * * Get a free request from @q. This function may fail under memory * pressure or if @q is dead. @@ -1343,7 +1338,7 @@ int blk_update_nr_requests(struct request_queue *q, unsigned int nr) * Returns request pointer on success, with @q->queue_lock *not held*. */ static struct request *__get_request(struct request_list *rl, unsigned int op, - struct bio *bio, blk_mq_req_flags_t flags) + struct bio *bio, blk_mq_req_flags_t flags, gfp_t gfp_mask) { struct request_queue *q = rl->q; struct request *rq; @@ -1352,8 +1347,6 @@ static struct request *__get_request(struct request_list *rl, unsigned int op, struct io_cq *icq = NULL; const bool is_sync = op_is_sync(op); int may_queue; - gfp_t gfp_mask = flags & BLK_MQ_REQ_NOWAIT ? GFP_ATOMIC : - __GFP_DIRECT_RECLAIM; req_flags_t rq_flags = RQF_ALLOCED; lockdep_assert_held(q->queue_lock); @@ -1517,8 +1510,9 @@ rq_starved: * @op: operation and flags * @bio: bio to allocate request for (can be %NULL) * @flags: BLK_MQ_REQ_* flags. + * @gfp: allocator flags * - * Get a free request from @q. If %__GFP_DIRECT_RECLAIM is set in @gfp_mask, + * Get a free request from @q. If %BLK_MQ_REQ_NOWAIT is set in @flags, * this function keeps retrying under memory pressure and fails iff @q is dead. * * Must be called with @q->queue_lock held and, @@ -1526,7 +1520,7 @@ rq_starved: * Returns request pointer on success, with @q->queue_lock *not held*. */ static struct request *get_request(struct request_queue *q, unsigned int op, - struct bio *bio, blk_mq_req_flags_t flags) + struct bio *bio, blk_mq_req_flags_t flags, gfp_t gfp) { const bool is_sync = op_is_sync(op); DEFINE_WAIT(wait); @@ -1538,7 +1532,7 @@ static struct request *get_request(struct request_queue *q, unsigned int op, rl = blk_get_rl(q, bio); /* transferred to @rq on success */ retry: - rq = __get_request(rl, op, bio, flags); + rq = __get_request(rl, op, bio, flags, gfp); if (!IS_ERR(rq)) return rq; @@ -1579,8 +1573,7 @@ static struct request *blk_old_get_request(struct request_queue *q, unsigned int op, blk_mq_req_flags_t flags) { struct request *rq; - gfp_t gfp_mask = flags & BLK_MQ_REQ_NOWAIT ? GFP_ATOMIC : - __GFP_DIRECT_RECLAIM; + gfp_t gfp_mask = flags & BLK_MQ_REQ_NOWAIT ? GFP_ATOMIC : GFP_NOIO; int ret = 0; WARN_ON_ONCE(q->mq_ops); @@ -1592,7 +1585,7 @@ static struct request *blk_old_get_request(struct request_queue *q, if (ret) return ERR_PTR(ret); spin_lock_irq(q->queue_lock); - rq = get_request(q, op, NULL, flags); + rq = get_request(q, op, NULL, flags, gfp_mask); if (IS_ERR(rq)) { spin_unlock_irq(q->queue_lock); blk_queue_exit(q); @@ -1607,13 +1600,13 @@ static struct request *blk_old_get_request(struct request_queue *q, } /** - * blk_get_request_flags - allocate a request + * blk_get_request - allocate a request * @q: request queue to allocate a request for * @op: operation (REQ_OP_*) and REQ_* flags, e.g. REQ_SYNC. * @flags: BLK_MQ_REQ_* flags, e.g. BLK_MQ_REQ_NOWAIT. */ -struct request *blk_get_request_flags(struct request_queue *q, unsigned int op, - blk_mq_req_flags_t flags) +struct request *blk_get_request(struct request_queue *q, unsigned int op, + blk_mq_req_flags_t flags) { struct request *req; @@ -1632,14 +1625,6 @@ struct request *blk_get_request_flags(struct request_queue *q, unsigned int op, return req; } -EXPORT_SYMBOL(blk_get_request_flags); - -struct request *blk_get_request(struct request_queue *q, unsigned int op, - gfp_t gfp_mask) -{ - return blk_get_request_flags(q, op, gfp_mask & __GFP_DIRECT_RECLAIM ? - 0 : BLK_MQ_REQ_NOWAIT); -} EXPORT_SYMBOL(blk_get_request); /** @@ -1660,7 +1645,7 @@ void blk_requeue_request(struct request_queue *q, struct request *rq) blk_delete_timer(rq); blk_clear_rq_complete(rq); trace_block_rq_requeue(q, rq); - wbt_requeue(q->rq_wb, &rq->issue_stat); + wbt_requeue(q->rq_wb, rq); if (rq->rq_flags & RQF_QUEUED) blk_queue_end_tag(q, rq); @@ -1767,7 +1752,7 @@ void __blk_put_request(struct request_queue *q, struct request *req) /* this is a bio leak */ WARN_ON(req->bio != NULL); - wbt_done(q->rq_wb, &req->issue_stat); + wbt_done(q->rq_wb, req); /* * Request may not have originated from ll_rw_blk. if not, @@ -2066,7 +2051,7 @@ get_rq: * Returns with the queue unlocked. */ blk_queue_enter_live(q); - req = get_request(q, bio->bi_opf, bio, 0); + req = get_request(q, bio->bi_opf, bio, 0, GFP_NOIO); if (IS_ERR(req)) { blk_queue_exit(q); __wbt_done(q->rq_wb, wb_acct); @@ -2078,7 +2063,7 @@ get_rq: goto out_unlock; } - wbt_track(&req->issue_stat, wb_acct); + wbt_track(req, wb_acct); /* * After dropping the lock and possibly sleeping here, our request @@ -2235,10 +2220,10 @@ static inline int blk_partition_remap(struct bio *bio) if (bio_check_eod(bio, part_nr_sects_read(p))) goto out; bio->bi_iter.bi_sector += p->start_sect; - bio->bi_partno = 0; trace_block_bio_remap(bio->bi_disk->queue, bio, part_devt(p), bio->bi_iter.bi_sector - p->start_sect); } + bio->bi_partno = 0; ret = 0; out: rcu_read_unlock(); @@ -2392,7 +2377,9 @@ blk_qc_t generic_make_request(struct bio *bio) if (bio->bi_opf & REQ_NOWAIT) flags = BLK_MQ_REQ_NOWAIT; - if (blk_queue_enter(q, flags) < 0) { + if (bio_flagged(bio, BIO_QUEUE_ENTERED)) + blk_queue_enter_live(q); + else if (blk_queue_enter(q, flags) < 0) { if (!blk_queue_dying(q) && (bio->bi_opf & REQ_NOWAIT)) bio_wouldblock_error(bio); else @@ -2727,7 +2714,7 @@ void blk_account_io_completion(struct request *req, unsigned int bytes) } } -void blk_account_io_done(struct request *req) +void blk_account_io_done(struct request *req, u64 now) { /* * Account IO completion. flush_rq isn't accounted as a @@ -2735,11 +2722,12 @@ void blk_account_io_done(struct request *req) * containing request is enough. */ if (blk_do_io_stat(req) && !(req->rq_flags & RQF_FLUSH_SEQ)) { - unsigned long duration = jiffies - req->start_time; + unsigned long duration; const int rw = rq_data_dir(req); struct hd_struct *part; int cpu; + duration = nsecs_to_jiffies(now - req->start_time_ns); cpu = part_stat_lock(); part = req->part; @@ -2970,10 +2958,8 @@ static void blk_dequeue_request(struct request *rq) * and to it is freed is accounted as io that is in progress at * the driver side. */ - if (blk_account_rq(rq)) { + if (blk_account_rq(rq)) q->in_flight[rq_is_sync(rq)]++; - set_io_start_time_ns(rq); - } } /** @@ -2992,9 +2978,12 @@ void blk_start_request(struct request *req) blk_dequeue_request(req); if (test_bit(QUEUE_FLAG_STATS, &req->q->queue_flags)) { - blk_stat_set_issue(&req->issue_stat, blk_rq_sectors(req)); + req->io_start_time_ns = ktime_get_ns(); +#ifdef CONFIG_BLK_DEV_THROTTLING_LOW + req->throtl_size = blk_rq_sectors(req); +#endif req->rq_flags |= RQF_STATS; - wbt_issue(req->q->rq_wb, &req->issue_stat); + wbt_issue(req->q->rq_wb, req); } BUG_ON(blk_rq_is_complete(req)); @@ -3092,8 +3081,10 @@ bool blk_update_request(struct request *req, blk_status_t error, struct bio *bio = req->bio; unsigned bio_bytes = min(bio->bi_iter.bi_size, nr_bytes); - if (bio_bytes == bio->bi_iter.bi_size) + if (bio_bytes == bio->bi_iter.bi_size) { req->bio = bio->bi_next; + bio->bi_next = NULL; + } /* Completion has already been traced */ bio_clear_flag(bio, BIO_TRACE_COMPLETION); @@ -3190,12 +3181,13 @@ EXPORT_SYMBOL_GPL(blk_unprep_request); void blk_finish_request(struct request *req, blk_status_t error) { struct request_queue *q = req->q; + u64 now = ktime_get_ns(); lockdep_assert_held(req->q->queue_lock); WARN_ON_ONCE(q->mq_ops); if (req->rq_flags & RQF_STATS) - blk_stat_add(req); + blk_stat_add(req, now); if (req->rq_flags & RQF_QUEUED) blk_queue_end_tag(q, req); @@ -3210,10 +3202,10 @@ void blk_finish_request(struct request *req, blk_status_t error) if (req->rq_flags & RQF_DONTPREP) blk_unprep_request(req); - blk_account_io_done(req); + blk_account_io_done(req, now); if (req->end_io) { - wbt_done(req->q->rq_wb, &req->issue_stat); + wbt_done(req->q->rq_wb, req); req->end_io(req, error); } else { if (blk_bidi_rq(req)) @@ -3519,7 +3511,7 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src, struct bio *bio, *bio_src; if (!bs) - bs = fs_bio_set; + bs = &fs_bio_set; __rq_for_each_bio(bio_src, rq_src) { bio = bio_clone_fast(bio_src, gfp_mask, bs); @@ -3630,7 +3622,7 @@ static void queue_unplugged(struct request_queue *q, unsigned int depth, blk_run_queue_async(q); else __blk_run_queue(q); - spin_unlock(q->queue_lock); + spin_unlock_irq(q->queue_lock); } static void flush_plug_callbacks(struct blk_plug *plug, bool from_schedule) @@ -3678,7 +3670,6 @@ EXPORT_SYMBOL(blk_check_plugged); void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule) { struct request_queue *q; - unsigned long flags; struct request *rq; LIST_HEAD(list); unsigned int depth; @@ -3698,11 +3689,6 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule) q = NULL; depth = 0; - /* - * Save and disable interrupts here, to avoid doing it for every - * queue lock we have to take. - */ - local_irq_save(flags); while (!list_empty(&list)) { rq = list_entry_rq(list.next); list_del_init(&rq->queuelist); @@ -3715,7 +3701,7 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule) queue_unplugged(q, depth, from_schedule); q = rq->q; depth = 0; - spin_lock(q->queue_lock); + spin_lock_irq(q->queue_lock); } /* @@ -3742,8 +3728,6 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule) */ if (q) queue_unplugged(q, depth, from_schedule); - - local_irq_restore(flags); } void blk_finish_plug(struct blk_plug *plug) diff --git a/block/blk-flush.c b/block/blk-flush.c index f17170675917..ce41f666de3e 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -94,7 +94,7 @@ enum { }; static bool blk_kick_flush(struct request_queue *q, - struct blk_flush_queue *fq); + struct blk_flush_queue *fq, unsigned int flags); static unsigned int blk_flush_policy(unsigned long fflags, struct request *rq) { @@ -169,9 +169,11 @@ static bool blk_flush_complete_seq(struct request *rq, struct request_queue *q = rq->q; struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx]; bool queued = false, kicked; + unsigned int cmd_flags; BUG_ON(rq->flush.seq & seq); rq->flush.seq |= seq; + cmd_flags = rq->cmd_flags; if (likely(!error)) seq = blk_flush_cur_seq(rq); @@ -212,7 +214,7 @@ static bool blk_flush_complete_seq(struct request *rq, BUG(); } - kicked = blk_kick_flush(q, fq); + kicked = blk_kick_flush(q, fq, cmd_flags); return kicked | queued; } @@ -281,6 +283,7 @@ static void flush_end_io(struct request *flush_rq, blk_status_t error) * blk_kick_flush - consider issuing flush request * @q: request_queue being kicked * @fq: flush queue + * @flags: cmd_flags of the original request * * Flush related states of @q have changed, consider issuing flush request. * Please read the comment at the top of this file for more info. @@ -291,7 +294,8 @@ static void flush_end_io(struct request *flush_rq, blk_status_t error) * RETURNS: * %true if flush was issued, %false otherwise. */ -static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq) +static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq, + unsigned int flags) { struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx]; struct request *first_rq = @@ -346,6 +350,7 @@ static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq) } flush_rq->cmd_flags = REQ_OP_FLUSH | REQ_PREFLUSH; + flush_rq->cmd_flags |= (flags & REQ_DRV) | (flags & REQ_FAILFAST_MASK); flush_rq->rq_flags |= RQF_FLUSH_SEQ; flush_rq->rq_disk = first_rq->rq_disk; flush_rq->end_io = flush_end_io; diff --git a/block/blk-integrity.c b/block/blk-integrity.c index feb30570eaf5..6121611e1316 100644 --- a/block/blk-integrity.c +++ b/block/blk-integrity.c @@ -333,34 +333,34 @@ static ssize_t integrity_device_show(struct blk_integrity *bi, char *page) } static struct integrity_sysfs_entry integrity_format_entry = { - .attr = { .name = "format", .mode = S_IRUGO }, + .attr = { .name = "format", .mode = 0444 }, .show = integrity_format_show, }; static struct integrity_sysfs_entry integrity_tag_size_entry = { - .attr = { .name = "tag_size", .mode = S_IRUGO }, + .attr = { .name = "tag_size", .mode = 0444 }, .show = integrity_tag_size_show, }; static struct integrity_sysfs_entry integrity_interval_entry = { - .attr = { .name = "protection_interval_bytes", .mode = S_IRUGO }, + .attr = { .name = "protection_interval_bytes", .mode = 0444 }, .show = integrity_interval_show, }; static struct integrity_sysfs_entry integrity_verify_entry = { - .attr = { .name = "read_verify", .mode = S_IRUGO | S_IWUSR }, + .attr = { .name = "read_verify", .mode = 0644 }, .show = integrity_verify_show, .store = integrity_verify_store, }; static struct integrity_sysfs_entry integrity_generate_entry = { - .attr = { .name = "write_generate", .mode = S_IRUGO | S_IWUSR }, + .attr = { .name = "write_generate", .mode = 0644 }, .show = integrity_generate_show, .store = integrity_generate_store, }; static struct integrity_sysfs_entry integrity_device_entry = { - .attr = { .name = "device_is_integrity_capable", .mode = S_IRUGO }, + .attr = { .name = "device_is_integrity_capable", .mode = 0444 }, .show = integrity_device_show, }; diff --git a/block/blk-lib.c b/block/blk-lib.c index a676084d4740..8faa70f26fcd 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -62,10 +62,16 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, unsigned int req_sects; sector_t end_sect, tmp; - /* Make sure bi_size doesn't overflow */ - req_sects = min_t(sector_t, nr_sects, UINT_MAX >> 9); + /* + * Issue in chunks of the user defined max discard setting, + * ensuring that bi_size doesn't overflow + */ + req_sects = min_t(sector_t, nr_sects, + q->limits.max_discard_sectors); + if (req_sects > UINT_MAX >> 9) + req_sects = UINT_MAX >> 9; - /** + /* * If splitting a request, and the next starting sector would be * misaligned, stop the discard at the previous aligned sector. */ diff --git a/block/blk-merge.c b/block/blk-merge.c index 782940c65d8a..aaec38cc37b8 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -188,16 +188,16 @@ void blk_queue_split(struct request_queue *q, struct bio **bio) switch (bio_op(*bio)) { case REQ_OP_DISCARD: case REQ_OP_SECURE_ERASE: - split = blk_bio_discard_split(q, *bio, q->bio_split, &nsegs); + split = blk_bio_discard_split(q, *bio, &q->bio_split, &nsegs); break; case REQ_OP_WRITE_ZEROES: - split = blk_bio_write_zeroes_split(q, *bio, q->bio_split, &nsegs); + split = blk_bio_write_zeroes_split(q, *bio, &q->bio_split, &nsegs); break; case REQ_OP_WRITE_SAME: - split = blk_bio_write_same_split(q, *bio, q->bio_split, &nsegs); + split = blk_bio_write_same_split(q, *bio, &q->bio_split, &nsegs); break; default: - split = blk_bio_segment_split(q, *bio, q->bio_split, &nsegs); + split = blk_bio_segment_split(q, *bio, &q->bio_split, &nsegs); break; } @@ -210,6 +210,16 @@ void blk_queue_split(struct request_queue *q, struct bio **bio) /* there isn't chance to merge the splitted bio */ split->bi_opf |= REQ_NOMERGE; + /* + * Since we're recursing into make_request here, ensure + * that we mark this bio as already having entered the queue. + * If not, and the queue is going away, we can get stuck + * forever on waiting for the queue reference to drop. But + * that will never happen, as we're already holding a + * reference to it. + */ + bio_set_flag(*bio, BIO_QUEUE_ENTERED); + bio_chain(split, *bio); trace_block_split(q, split, (*bio)->bi_iter.bi_sector); generic_make_request(*bio); @@ -724,13 +734,12 @@ static struct request *attempt_merge(struct request_queue *q, } /* - * At this point we have either done a back merge - * or front merge. We need the smaller start_time of - * the merged requests to be the current request - * for accounting purposes. + * At this point we have either done a back merge or front merge. We + * need the smaller start_time_ns of the merged requests to be the + * current request for accounting purposes. */ - if (time_after(req->start_time, next->start_time)) - req->start_time = next->start_time; + if (next->start_time_ns < req->start_time_ns) + req->start_time_ns = next->start_time_ns; req->biotail->bi_next = next->bio; req->biotail = next->biotail; diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 3080e18cb859..ffa622366922 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -344,7 +344,6 @@ static const char *const rqf_name[] = { RQF_NAME(STATS), RQF_NAME(SPECIAL_PAYLOAD), RQF_NAME(ZONE_WRITE_LOCKED), - RQF_NAME(MQ_TIMEOUT_EXPIRED), RQF_NAME(MQ_POLL_SLEPT), }; #undef RQF_NAME diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 25c14c58385c..56c493c6cd90 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -268,19 +268,16 @@ bool blk_mq_sched_try_merge(struct request_queue *q, struct bio *bio, EXPORT_SYMBOL_GPL(blk_mq_sched_try_merge); /* - * Reverse check our software queue for entries that we could potentially - * merge with. Currently includes a hand-wavy stop count of 8, to not spend - * too much time checking for merges. + * Iterate list of requests and see if we can merge this bio with any + * of them. */ -static bool blk_mq_attempt_merge(struct request_queue *q, - struct blk_mq_ctx *ctx, struct bio *bio) +bool blk_mq_bio_list_merge(struct request_queue *q, struct list_head *list, + struct bio *bio) { struct request *rq; int checked = 8; - lockdep_assert_held(&ctx->lock); - - list_for_each_entry_reverse(rq, &ctx->rq_list, queuelist) { + list_for_each_entry_reverse(rq, list, queuelist) { bool merged = false; if (!checked--) @@ -305,13 +302,30 @@ static bool blk_mq_attempt_merge(struct request_queue *q, continue; } - if (merged) - ctx->rq_merged++; return merged; } return false; } +EXPORT_SYMBOL_GPL(blk_mq_bio_list_merge); + +/* + * Reverse check our software queue for entries that we could potentially + * merge with. Currently includes a hand-wavy stop count of 8, to not spend + * too much time checking for merges. + */ +static bool blk_mq_attempt_merge(struct request_queue *q, + struct blk_mq_ctx *ctx, struct bio *bio) +{ + lockdep_assert_held(&ctx->lock); + + if (blk_mq_bio_list_merge(q, &ctx->rq_list, bio)) { + ctx->rq_merged++; + return true; + } + + return false; +} bool __blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio) { @@ -571,6 +585,7 @@ int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) if (!e) { q->elevator = NULL; + q->nr_requests = q->tag_set->queue_depth; return 0; } @@ -633,14 +648,3 @@ void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e) blk_mq_sched_tags_teardown(q); q->elevator = NULL; } - -int blk_mq_sched_init(struct request_queue *q) -{ - int ret; - - mutex_lock(&q->sysfs_lock); - ret = elevator_init(q, NULL); - mutex_unlock(&q->sysfs_lock); - - return ret; -} diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h index 1e9c9018ace1..0cb8f938dff9 100644 --- a/block/blk-mq-sched.h +++ b/block/blk-mq-sched.h @@ -33,8 +33,6 @@ int blk_mq_sched_init_hctx(struct request_queue *q, struct blk_mq_hw_ctx *hctx, void blk_mq_sched_exit_hctx(struct request_queue *q, struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx); -int blk_mq_sched_init(struct request_queue *q); - static inline bool blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio) { diff --git a/block/blk-mq-sysfs.c b/block/blk-mq-sysfs.c index a54b4b070f1c..aafb44224c89 100644 --- a/block/blk-mq-sysfs.c +++ b/block/blk-mq-sysfs.c @@ -166,15 +166,15 @@ static struct attribute *default_ctx_attrs[] = { }; static struct blk_mq_hw_ctx_sysfs_entry blk_mq_hw_sysfs_nr_tags = { - .attr = {.name = "nr_tags", .mode = S_IRUGO }, + .attr = {.name = "nr_tags", .mode = 0444 }, .show = blk_mq_hw_sysfs_nr_tags_show, }; static struct blk_mq_hw_ctx_sysfs_entry blk_mq_hw_sysfs_nr_reserved_tags = { - .attr = {.name = "nr_reserved_tags", .mode = S_IRUGO }, + .attr = {.name = "nr_reserved_tags", .mode = 0444 }, .show = blk_mq_hw_sysfs_nr_reserved_tags_show, }; static struct blk_mq_hw_ctx_sysfs_entry blk_mq_hw_sysfs_cpus = { - .attr = {.name = "cpu_list", .mode = S_IRUGO }, + .attr = {.name = "cpu_list", .mode = 0444 }, .show = blk_mq_hw_sysfs_cpus_show, }; diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index 336dde07b230..09b2ee6694fb 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -134,6 +134,8 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data) ws = bt_wait_ptr(bt, data->hctx); drop_ctx = data->ctx == NULL; do { + struct sbitmap_queue *bt_prev; + /* * We're out of tags on this hardware queue, kick any * pending IO submits before going to sleep waiting for @@ -159,6 +161,7 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data) if (data->ctx) blk_mq_put_ctx(data->ctx); + bt_prev = bt; io_schedule(); data->ctx = blk_mq_get_ctx(data->q); @@ -170,6 +173,15 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data) bt = &tags->bitmap_tags; finish_wait(&ws->wait, &wait); + + /* + * If destination hw queue is changed, fake wake up on + * previous queue for compensating the wake up miss, so + * other allocations on previous queue won't be starved. + */ + if (bt != bt_prev) + sbitmap_queue_wake_up(bt_prev); + ws = bt_wait_ptr(bt, data->hctx); } while (1); @@ -259,7 +271,7 @@ static bool bt_tags_iter(struct sbitmap *bitmap, unsigned int bitnr, void *data) * test and set the bit before assining ->rqs[]. */ rq = tags->rqs[bitnr]; - if (rq) + if (rq && blk_mq_rq_state(rq) == MQ_RQ_IN_FLIGHT) iter_data->fn(rq, iter_data->data, reserved); return true; @@ -299,35 +311,6 @@ void blk_mq_tagset_busy_iter(struct blk_mq_tag_set *tagset, } EXPORT_SYMBOL(blk_mq_tagset_busy_iter); -int blk_mq_tagset_iter(struct blk_mq_tag_set *set, void *data, - int (fn)(void *, struct request *)) -{ - int i, j, ret = 0; - - if (WARN_ON_ONCE(!fn)) - goto out; - - for (i = 0; i < set->nr_hw_queues; i++) { - struct blk_mq_tags *tags = set->tags[i]; - - if (!tags) - continue; - - for (j = 0; j < tags->nr_tags; j++) { - if (!tags->static_rqs[j]) - continue; - - ret = fn(data, tags->static_rqs[j]); - if (ret) - goto out; - } - } - -out: - return ret; -} -EXPORT_SYMBOL_GPL(blk_mq_tagset_iter); - void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_iter_fn *fn, void *priv) { diff --git a/block/blk-mq.c b/block/blk-mq.c index 9ce9cac16c3f..70c65bb6c013 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -309,7 +309,8 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data, RB_CLEAR_NODE(&rq->rb_node); rq->rq_disk = NULL; rq->part = NULL; - rq->start_time = jiffies; + rq->start_time_ns = ktime_get_ns(); + rq->io_start_time_ns = 0; rq->nr_phys_segments = 0; #if defined(CONFIG_BLK_DEV_INTEGRITY) rq->nr_integrity_segments = 0; @@ -328,11 +329,10 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data, #ifdef CONFIG_BLK_CGROUP rq->rl = NULL; - set_start_time_ns(rq); - rq->io_start_time_ns = 0; #endif data->ctx->rq_dispatched[op_is_sync(op)]++; + refcount_set(&rq->ref, 1); return rq; } @@ -361,9 +361,11 @@ static struct request *blk_mq_get_request(struct request_queue *q, /* * Flush requests are special and go directly to the - * dispatch list. + * dispatch list. Don't include reserved tags in the + * limiting, as it isn't useful. */ - if (!op_is_flush(op) && e->type->ops.mq.limit_depth) + if (!op_is_flush(op) && e->type->ops.mq.limit_depth && + !(data->flags & BLK_MQ_REQ_RESERVED)) e->type->ops.mq.limit_depth(op, data); } @@ -464,13 +466,27 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q, } EXPORT_SYMBOL_GPL(blk_mq_alloc_request_hctx); +static void __blk_mq_free_request(struct request *rq) +{ + struct request_queue *q = rq->q; + struct blk_mq_ctx *ctx = rq->mq_ctx; + struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, ctx->cpu); + const int sched_tag = rq->internal_tag; + + if (rq->tag != -1) + blk_mq_put_tag(hctx, hctx->tags, ctx, rq->tag); + if (sched_tag != -1) + blk_mq_put_tag(hctx, hctx->sched_tags, ctx, sched_tag); + blk_mq_sched_restart(hctx); + blk_queue_exit(q); +} + void blk_mq_free_request(struct request *rq) { struct request_queue *q = rq->q; struct elevator_queue *e = q->elevator; struct blk_mq_ctx *ctx = rq->mq_ctx; struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, ctx->cpu); - const int sched_tag = rq->internal_tag; if (rq->rq_flags & RQF_ELVPRIV) { if (e && e->type->ops.mq.finish_request) @@ -488,27 +504,30 @@ void blk_mq_free_request(struct request *rq) if (unlikely(laptop_mode && !blk_rq_is_passthrough(rq))) laptop_io_completion(q->backing_dev_info); - wbt_done(q->rq_wb, &rq->issue_stat); + wbt_done(q->rq_wb, rq); if (blk_rq_rl(rq)) blk_put_rl(blk_rq_rl(rq)); - blk_mq_rq_update_state(rq, MQ_RQ_IDLE); - if (rq->tag != -1) - blk_mq_put_tag(hctx, hctx->tags, ctx, rq->tag); - if (sched_tag != -1) - blk_mq_put_tag(hctx, hctx->sched_tags, ctx, sched_tag); - blk_mq_sched_restart(hctx); - blk_queue_exit(q); + WRITE_ONCE(rq->state, MQ_RQ_IDLE); + if (refcount_dec_and_test(&rq->ref)) + __blk_mq_free_request(rq); } EXPORT_SYMBOL_GPL(blk_mq_free_request); inline void __blk_mq_end_request(struct request *rq, blk_status_t error) { - blk_account_io_done(rq); + u64 now = ktime_get_ns(); + + if (rq->rq_flags & RQF_STATS) { + blk_mq_poll_stats_start(rq->q); + blk_stat_add(rq, now); + } + + blk_account_io_done(rq, now); if (rq->end_io) { - wbt_done(rq->q->rq_wb, &rq->issue_stat); + wbt_done(rq->q->rq_wb, rq); rq->end_io(rq, error); } else { if (unlikely(blk_bidi_rq(rq))) @@ -539,15 +558,12 @@ static void __blk_mq_complete_request(struct request *rq) bool shared = false; int cpu; - WARN_ON_ONCE(blk_mq_rq_state(rq) != MQ_RQ_IN_FLIGHT); - blk_mq_rq_update_state(rq, MQ_RQ_COMPLETE); + if (cmpxchg(&rq->state, MQ_RQ_IN_FLIGHT, MQ_RQ_COMPLETE) != + MQ_RQ_IN_FLIGHT) + return; if (rq->internal_tag != -1) blk_mq_sched_completed_request(rq); - if (rq->rq_flags & RQF_STATS) { - blk_mq_poll_stats_start(rq->q); - blk_stat_add(rq); - } if (!test_bit(QUEUE_FLAG_SAME_COMP, &rq->q->queue_flags)) { rq->q->softirq_done_fn(rq); @@ -589,36 +605,6 @@ static void hctx_lock(struct blk_mq_hw_ctx *hctx, int *srcu_idx) *srcu_idx = srcu_read_lock(hctx->srcu); } -static void blk_mq_rq_update_aborted_gstate(struct request *rq, u64 gstate) -{ - unsigned long flags; - - /* - * blk_mq_rq_aborted_gstate() is used from the completion path and - * can thus be called from irq context. u64_stats_fetch in the - * middle of update on the same CPU leads to lockup. Disable irq - * while updating. - */ - local_irq_save(flags); - u64_stats_update_begin(&rq->aborted_gstate_sync); - rq->aborted_gstate = gstate; - u64_stats_update_end(&rq->aborted_gstate_sync); - local_irq_restore(flags); -} - -static u64 blk_mq_rq_aborted_gstate(struct request *rq) -{ - unsigned int start; - u64 aborted_gstate; - - do { - start = u64_stats_fetch_begin(&rq->aborted_gstate_sync); - aborted_gstate = rq->aborted_gstate; - } while (u64_stats_fetch_retry(&rq->aborted_gstate_sync, start)); - - return aborted_gstate; -} - /** * blk_mq_complete_request - end I/O on a request * @rq: the request being processed @@ -629,28 +615,9 @@ static u64 blk_mq_rq_aborted_gstate(struct request *rq) **/ void blk_mq_complete_request(struct request *rq) { - struct request_queue *q = rq->q; - struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, rq->mq_ctx->cpu); - int srcu_idx; - - if (unlikely(blk_should_fake_timeout(q))) + if (unlikely(blk_should_fake_timeout(rq->q))) return; - - /* - * If @rq->aborted_gstate equals the current instance, timeout is - * claiming @rq and we lost. This is synchronized through - * hctx_lock(). See blk_mq_timeout_work() for details. - * - * Completion path never blocks and we can directly use RCU here - * instead of hctx_lock() which can be either RCU or SRCU. - * However, that would complicate paths which want to synchronize - * against us. Let stay in sync with the issue path so that - * hctx_lock() covers both issue and completion paths. - */ - hctx_lock(hctx, &srcu_idx); - if (blk_mq_rq_aborted_gstate(rq) != rq->gstate) - __blk_mq_complete_request(rq); - hctx_unlock(hctx, srcu_idx); + __blk_mq_complete_request(rq); } EXPORT_SYMBOL(blk_mq_complete_request); @@ -669,32 +636,18 @@ void blk_mq_start_request(struct request *rq) trace_block_rq_issue(q, rq); if (test_bit(QUEUE_FLAG_STATS, &q->queue_flags)) { - blk_stat_set_issue(&rq->issue_stat, blk_rq_sectors(rq)); + rq->io_start_time_ns = ktime_get_ns(); +#ifdef CONFIG_BLK_DEV_THROTTLING_LOW + rq->throtl_size = blk_rq_sectors(rq); +#endif rq->rq_flags |= RQF_STATS; - wbt_issue(q->rq_wb, &rq->issue_stat); + wbt_issue(q->rq_wb, rq); } WARN_ON_ONCE(blk_mq_rq_state(rq) != MQ_RQ_IDLE); - /* - * Mark @rq in-flight which also advances the generation number, - * and register for timeout. Protect with a seqcount to allow the - * timeout path to read both @rq->gstate and @rq->deadline - * coherently. - * - * This is the only place where a request is marked in-flight. If - * the timeout path reads an in-flight @rq->gstate, the - * @rq->deadline it reads together under @rq->gstate_seq is - * guaranteed to be the matching one. - */ - preempt_disable(); - write_seqcount_begin(&rq->gstate_seq); - - blk_mq_rq_update_state(rq, MQ_RQ_IN_FLIGHT); blk_add_timer(rq); - - write_seqcount_end(&rq->gstate_seq); - preempt_enable(); + WRITE_ONCE(rq->state, MQ_RQ_IN_FLIGHT); if (q->dma_drain_size && blk_rq_bytes(rq)) { /* @@ -707,11 +660,6 @@ void blk_mq_start_request(struct request *rq) } EXPORT_SYMBOL(blk_mq_start_request); -/* - * When we reach here because queue is busy, it's safe to change the state - * to IDLE without checking @rq->aborted_gstate because we should still be - * holding the RCU read lock and thus protected against timeout. - */ static void __blk_mq_requeue_request(struct request *rq) { struct request_queue *q = rq->q; @@ -719,10 +667,11 @@ static void __blk_mq_requeue_request(struct request *rq) blk_mq_put_driver_tag(rq); trace_block_rq_requeue(q, rq); - wbt_requeue(q->rq_wb, &rq->issue_stat); + wbt_requeue(q->rq_wb, rq); - if (blk_mq_rq_state(rq) != MQ_RQ_IDLE) { - blk_mq_rq_update_state(rq, MQ_RQ_IDLE); + if (blk_mq_request_started(rq)) { + WRITE_ONCE(rq->state, MQ_RQ_IDLE); + rq->rq_flags &= ~RQF_TIMED_OUT; if (q->dma_drain_size && blk_rq_bytes(rq)) rq->nr_phys_segments--; } @@ -820,101 +769,83 @@ struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, unsigned int tag) } EXPORT_SYMBOL(blk_mq_tag_to_rq); -struct blk_mq_timeout_data { - unsigned long next; - unsigned int next_set; - unsigned int nr_expired; -}; - static void blk_mq_rq_timed_out(struct request *req, bool reserved) { - const struct blk_mq_ops *ops = req->q->mq_ops; - enum blk_eh_timer_return ret = BLK_EH_RESET_TIMER; - - req->rq_flags |= RQF_MQ_TIMEOUT_EXPIRED; + req->rq_flags |= RQF_TIMED_OUT; + if (req->q->mq_ops->timeout) { + enum blk_eh_timer_return ret; - if (ops->timeout) - ret = ops->timeout(req, reserved); - - switch (ret) { - case BLK_EH_HANDLED: - __blk_mq_complete_request(req); - break; - case BLK_EH_RESET_TIMER: - /* - * As nothing prevents from completion happening while - * ->aborted_gstate is set, this may lead to ignored - * completions and further spurious timeouts. - */ - blk_mq_rq_update_aborted_gstate(req, 0); - blk_add_timer(req); - break; - case BLK_EH_NOT_HANDLED: - break; - default: - printk(KERN_ERR "block: bad eh return: %d\n", ret); - break; + ret = req->q->mq_ops->timeout(req, reserved); + if (ret == BLK_EH_DONE) + return; + WARN_ON_ONCE(ret != BLK_EH_RESET_TIMER); } + + req->rq_flags &= ~RQF_TIMED_OUT; + blk_add_timer(req); } -static void blk_mq_check_expired(struct blk_mq_hw_ctx *hctx, - struct request *rq, void *priv, bool reserved) +static bool blk_mq_req_expired(struct request *rq, unsigned long *next) { - struct blk_mq_timeout_data *data = priv; - unsigned long gstate, deadline; - int start; + unsigned long deadline; - might_sleep(); - - if (rq->rq_flags & RQF_MQ_TIMEOUT_EXPIRED) - return; + if (blk_mq_rq_state(rq) != MQ_RQ_IN_FLIGHT) + return false; + if (rq->rq_flags & RQF_TIMED_OUT) + return false; - /* read coherent snapshots of @rq->state_gen and @rq->deadline */ - while (true) { - start = read_seqcount_begin(&rq->gstate_seq); - gstate = READ_ONCE(rq->gstate); - deadline = blk_rq_deadline(rq); - if (!read_seqcount_retry(&rq->gstate_seq, start)) - break; - cond_resched(); - } + deadline = blk_rq_deadline(rq); + if (time_after_eq(jiffies, deadline)) + return true; - /* if in-flight && overdue, mark for abortion */ - if ((gstate & MQ_RQ_STATE_MASK) == MQ_RQ_IN_FLIGHT && - time_after_eq(jiffies, deadline)) { - blk_mq_rq_update_aborted_gstate(rq, gstate); - data->nr_expired++; - hctx->nr_expired++; - } else if (!data->next_set || time_after(data->next, deadline)) { - data->next = deadline; - data->next_set = 1; - } + if (*next == 0) + *next = deadline; + else if (time_after(*next, deadline)) + *next = deadline; + return false; } -static void blk_mq_terminate_expired(struct blk_mq_hw_ctx *hctx, +static void blk_mq_check_expired(struct blk_mq_hw_ctx *hctx, struct request *rq, void *priv, bool reserved) { + unsigned long *next = priv; + + /* + * Just do a quick check if it is expired before locking the request in + * so we're not unnecessarilly synchronizing across CPUs. + */ + if (!blk_mq_req_expired(rq, next)) + return; + + /* + * We have reason to believe the request may be expired. Take a + * reference on the request to lock this request lifetime into its + * currently allocated context to prevent it from being reallocated in + * the event the completion by-passes this timeout handler. + * + * If the reference was already released, then the driver beat the + * timeout handler to posting a natural completion. + */ + if (!refcount_inc_not_zero(&rq->ref)) + return; + /* - * We marked @rq->aborted_gstate and waited for RCU. If there were - * completions that we lost to, they would have finished and - * updated @rq->gstate by now; otherwise, the completion path is - * now guaranteed to see @rq->aborted_gstate and yield. If - * @rq->aborted_gstate still matches @rq->gstate, @rq is ours. + * The request is now locked and cannot be reallocated underneath the + * timeout handler's processing. Re-verify this exact request is truly + * expired; if it is not expired, then the request was completed and + * reallocated as a new request. */ - if (!(rq->rq_flags & RQF_MQ_TIMEOUT_EXPIRED) && - READ_ONCE(rq->gstate) == rq->aborted_gstate) + if (blk_mq_req_expired(rq, next)) blk_mq_rq_timed_out(rq, reserved); + if (refcount_dec_and_test(&rq->ref)) + __blk_mq_free_request(rq); } static void blk_mq_timeout_work(struct work_struct *work) { struct request_queue *q = container_of(work, struct request_queue, timeout_work); - struct blk_mq_timeout_data data = { - .next = 0, - .next_set = 0, - .nr_expired = 0, - }; + unsigned long next = 0; struct blk_mq_hw_ctx *hctx; int i; @@ -934,39 +865,10 @@ static void blk_mq_timeout_work(struct work_struct *work) if (!percpu_ref_tryget(&q->q_usage_counter)) return; - /* scan for the expired ones and set their ->aborted_gstate */ - blk_mq_queue_tag_busy_iter(q, blk_mq_check_expired, &data); - - if (data.nr_expired) { - bool has_rcu = false; - - /* - * Wait till everyone sees ->aborted_gstate. The - * sequential waits for SRCUs aren't ideal. If this ever - * becomes a problem, we can add per-hw_ctx rcu_head and - * wait in parallel. - */ - queue_for_each_hw_ctx(q, hctx, i) { - if (!hctx->nr_expired) - continue; - - if (!(hctx->flags & BLK_MQ_F_BLOCKING)) - has_rcu = true; - else - synchronize_srcu(hctx->srcu); + blk_mq_queue_tag_busy_iter(q, blk_mq_check_expired, &next); - hctx->nr_expired = 0; - } - if (has_rcu) - synchronize_rcu(); - - /* terminate the ones we won */ - blk_mq_queue_tag_busy_iter(q, blk_mq_terminate_expired, NULL); - } - - if (data.next_set) { - data.next = blk_rq_timeout(round_jiffies_up(data.next)); - mod_timer(&q->timeout, data.next); + if (next != 0) { + mod_timer(&q->timeout, next); } else { /* * Request timeouts are handled as a forward rolling timer. If @@ -1029,7 +931,7 @@ static bool dispatch_rq_from_ctx(struct sbitmap *sb, unsigned int bitnr, struct blk_mq_ctx *ctx = hctx->ctxs[bitnr]; spin_lock(&ctx->lock); - if (unlikely(!list_empty(&ctx->rq_list))) { + if (!list_empty(&ctx->rq_list)) { dispatch_data->rq = list_entry_rq(ctx->rq_list.next); list_del_init(&dispatch_data->rq->queuelist); if (list_empty(&ctx->rq_list)) @@ -1578,7 +1480,7 @@ static void blk_mq_run_work_fn(struct work_struct *work) * If we are stopped, don't run the queue. */ if (test_bit(BLK_MQ_S_STOPPED, &hctx->state)) - clear_bit(BLK_MQ_S_STOPPED, &hctx->state); + return; __blk_mq_run_hw_queue(hctx); } @@ -1716,15 +1618,6 @@ static void blk_mq_bio_to_request(struct request *rq, struct bio *bio) blk_account_io_start(rq, true); } -static inline void blk_mq_queue_io(struct blk_mq_hw_ctx *hctx, - struct blk_mq_ctx *ctx, - struct request *rq) -{ - spin_lock(&ctx->lock); - __blk_mq_insert_request(hctx, rq, false); - spin_unlock(&ctx->lock); -} - static blk_qc_t request_to_qc_t(struct blk_mq_hw_ctx *hctx, struct request *rq) { if (rq->tag != -1) @@ -1882,7 +1775,7 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) return BLK_QC_T_NONE; } - wbt_track(&rq->issue_stat, wb_acct); + wbt_track(rq, wb_acct); cookie = request_to_qc_t(data.hctx, rq); @@ -1949,15 +1842,10 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) blk_mq_put_ctx(data.ctx); blk_mq_bio_to_request(rq, bio); blk_mq_try_issue_directly(data.hctx, rq, &cookie); - } else if (q->elevator) { - blk_mq_put_ctx(data.ctx); - blk_mq_bio_to_request(rq, bio); - blk_mq_sched_insert_request(rq, false, true, true); } else { blk_mq_put_ctx(data.ctx); blk_mq_bio_to_request(rq, bio); - blk_mq_queue_io(data.hctx, data.ctx, rq); - blk_mq_run_hw_queue(data.hctx, true); + blk_mq_sched_insert_request(rq, false, true, true); } return cookie; @@ -2020,7 +1908,7 @@ struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, if (!tags) return NULL; - tags->rqs = kzalloc_node(nr_tags * sizeof(struct request *), + tags->rqs = kcalloc_node(nr_tags, sizeof(struct request *), GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, node); if (!tags->rqs) { @@ -2028,9 +1916,9 @@ struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, return NULL; } - tags->static_rqs = kzalloc_node(nr_tags * sizeof(struct request *), - GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, - node); + tags->static_rqs = kcalloc_node(nr_tags, sizeof(struct request *), + GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, + node); if (!tags->static_rqs) { kfree(tags->rqs); blk_mq_free_tags(tags); @@ -2056,15 +1944,7 @@ static int blk_mq_init_request(struct blk_mq_tag_set *set, struct request *rq, return ret; } - seqcount_init(&rq->gstate_seq); - u64_stats_init(&rq->aborted_gstate_sync); - /* - * start gstate with gen 1 instead of 0, otherwise it will be equal - * to aborted_gstate, and be identified timed out by - * blk_mq_terminate_expired. - */ - WRITE_ONCE(rq->gstate, MQ_RQ_GEN_INC); - + WRITE_ONCE(rq->state, MQ_RQ_IDLE); return 0; } @@ -2365,6 +2245,7 @@ static void blk_mq_map_swqueue(struct request_queue *q) queue_for_each_hw_ctx(q, hctx, i) { cpumask_clear(hctx->cpumask); hctx->nr_ctx = 0; + hctx->dispatch_from = NULL; } /* @@ -2473,7 +2354,6 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q) mutex_lock(&set->tag_list_lock); list_del_rcu(&q->tag_set_list); - INIT_LIST_HEAD(&q->tag_set_list); if (list_is_singular(&set->tag_list)) { /* just transitioned to unshared */ set->flags &= ~BLK_MQ_F_TAG_SHARED; @@ -2481,8 +2361,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q) blk_mq_update_tag_set_depth(set, false); } mutex_unlock(&set->tag_list_lock); - synchronize_rcu(); + INIT_LIST_HEAD(&q->tag_set_list); } static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set, @@ -2646,7 +2526,7 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, /* init q->mq_kobj and sw queues' kobjects */ blk_mq_sysfs_init(q); - q->queue_hw_ctx = kzalloc_node(nr_cpu_ids * sizeof(*(q->queue_hw_ctx)), + q->queue_hw_ctx = kcalloc_node(nr_cpu_ids, sizeof(*(q->queue_hw_ctx)), GFP_KERNEL, set->numa_node); if (!q->queue_hw_ctx) goto err_percpu; @@ -2697,7 +2577,7 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, if (!(set->flags & BLK_MQ_F_NO_SCHED)) { int ret; - ret = blk_mq_sched_init(q); + ret = elevator_init_mq(q); if (ret) return ERR_PTR(ret); } @@ -2865,14 +2745,14 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) if (set->nr_hw_queues > nr_cpu_ids) set->nr_hw_queues = nr_cpu_ids; - set->tags = kzalloc_node(nr_cpu_ids * sizeof(struct blk_mq_tags *), + set->tags = kcalloc_node(nr_cpu_ids, sizeof(struct blk_mq_tags *), GFP_KERNEL, set->numa_node); if (!set->tags) return -ENOMEM; ret = -ENOMEM; - set->mq_map = kzalloc_node(sizeof(*set->mq_map) * nr_cpu_ids, - GFP_KERNEL, set->numa_node); + set->mq_map = kcalloc_node(nr_cpu_ids, sizeof(*set->mq_map), + GFP_KERNEL, set->numa_node); if (!set->mq_map) goto out_free_tags; diff --git a/block/blk-mq.h b/block/blk-mq.h index e1bb420dc5d6..89231e439b2f 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -30,20 +30,6 @@ struct blk_mq_ctx { struct kobject kobj; } ____cacheline_aligned_in_smp; -/* - * Bits for request->gstate. The lower two bits carry MQ_RQ_* state value - * and the upper bits the generation number. - */ -enum mq_rq_state { - MQ_RQ_IDLE = 0, - MQ_RQ_IN_FLIGHT = 1, - MQ_RQ_COMPLETE = 2, - - MQ_RQ_STATE_BITS = 2, - MQ_RQ_STATE_MASK = (1 << MQ_RQ_STATE_BITS) - 1, - MQ_RQ_GEN_INC = 1 << MQ_RQ_STATE_BITS, -}; - void blk_mq_freeze_queue(struct request_queue *q); void blk_mq_free_queue(struct request_queue *q); int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr); @@ -107,33 +93,9 @@ void blk_mq_release(struct request_queue *q); * blk_mq_rq_state() - read the current MQ_RQ_* state of a request * @rq: target request. */ -static inline int blk_mq_rq_state(struct request *rq) +static inline enum mq_rq_state blk_mq_rq_state(struct request *rq) { - return READ_ONCE(rq->gstate) & MQ_RQ_STATE_MASK; -} - -/** - * blk_mq_rq_update_state() - set the current MQ_RQ_* state of a request - * @rq: target request. - * @state: new state to set. - * - * Set @rq's state to @state. The caller is responsible for ensuring that - * there are no other updaters. A request can transition into IN_FLIGHT - * only from IDLE and doing so increments the generation number. - */ -static inline void blk_mq_rq_update_state(struct request *rq, - enum mq_rq_state state) -{ - u64 old_val = READ_ONCE(rq->gstate); - u64 new_val = (old_val & ~MQ_RQ_STATE_MASK) | state; - - if (state == MQ_RQ_IN_FLIGHT) { - WARN_ON_ONCE((old_val & MQ_RQ_STATE_MASK) != MQ_RQ_IDLE); - new_val += MQ_RQ_GEN_INC; - } - - /* avoid exposing interim values */ - WRITE_ONCE(rq->gstate, new_val); + return READ_ONCE(rq->state); } static inline struct blk_mq_ctx *__blk_mq_get_ctx(struct request_queue *q, diff --git a/block/blk-stat.c b/block/blk-stat.c index bd365a95fcf8..175c143ac5b9 100644 --- a/block/blk-stat.c +++ b/block/blk-stat.c @@ -47,19 +47,15 @@ static void __blk_stat_add(struct blk_rq_stat *stat, u64 value) stat->nr_samples++; } -void blk_stat_add(struct request *rq) +void blk_stat_add(struct request *rq, u64 now) { struct request_queue *q = rq->q; struct blk_stat_callback *cb; struct blk_rq_stat *stat; int bucket; - u64 now, value; + u64 value; - now = __blk_stat_time(ktime_to_ns(ktime_get())); - if (now < blk_stat_time(&rq->issue_stat)) - return; - - value = now - blk_stat_time(&rq->issue_stat); + value = (now >= rq->io_start_time_ns) ? now - rq->io_start_time_ns : 0; blk_throtl_stat_add(rq, value); diff --git a/block/blk-stat.h b/block/blk-stat.h index 2dd36347252a..78399cdde9c9 100644 --- a/block/blk-stat.h +++ b/block/blk-stat.h @@ -8,21 +8,6 @@ #include #include -/* - * from upper: - * 3 bits: reserved for other usage - * 12 bits: size - * 49 bits: time - */ -#define BLK_STAT_RES_BITS 3 -#define BLK_STAT_SIZE_BITS 12 -#define BLK_STAT_RES_SHIFT (64 - BLK_STAT_RES_BITS) -#define BLK_STAT_SIZE_SHIFT (BLK_STAT_RES_SHIFT - BLK_STAT_SIZE_BITS) -#define BLK_STAT_TIME_MASK ((1ULL << BLK_STAT_SIZE_SHIFT) - 1) -#define BLK_STAT_SIZE_MASK \ - (((1ULL << BLK_STAT_SIZE_BITS) - 1) << BLK_STAT_SIZE_SHIFT) -#define BLK_STAT_RES_MASK (~((1ULL << BLK_STAT_RES_SHIFT) - 1)) - /** * struct blk_stat_callback - Block statistics callback. * @@ -80,35 +65,7 @@ struct blk_stat_callback { struct blk_queue_stats *blk_alloc_queue_stats(void); void blk_free_queue_stats(struct blk_queue_stats *); -void blk_stat_add(struct request *); - -static inline u64 __blk_stat_time(u64 time) -{ - return time & BLK_STAT_TIME_MASK; -} - -static inline u64 blk_stat_time(struct blk_issue_stat *stat) -{ - return __blk_stat_time(stat->stat); -} - -static inline sector_t blk_capped_size(sector_t size) -{ - return size & ((1ULL << BLK_STAT_SIZE_BITS) - 1); -} - -static inline sector_t blk_stat_size(struct blk_issue_stat *stat) -{ - return (stat->stat & BLK_STAT_SIZE_MASK) >> BLK_STAT_SIZE_SHIFT; -} - -static inline void blk_stat_set_issue(struct blk_issue_stat *stat, - sector_t size) -{ - stat->stat = (stat->stat & BLK_STAT_RES_MASK) | - (ktime_to_ns(ktime_get()) & BLK_STAT_TIME_MASK) | - (((u64)blk_capped_size(size)) << BLK_STAT_SIZE_SHIFT); -} +void blk_stat_add(struct request *rq, u64 now); /* record time/size info in request but not add a callback */ void blk_stat_enable_accounting(struct request_queue *q); diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index d00d1b0ec109..94987b1f69e1 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -491,188 +491,198 @@ static ssize_t queue_wc_store(struct request_queue *q, const char *page, return count; } +static ssize_t queue_fua_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%u\n", test_bit(QUEUE_FLAG_FUA, &q->queue_flags)); +} + static ssize_t queue_dax_show(struct request_queue *q, char *page) { return queue_var_show(blk_queue_dax(q), page); } static struct queue_sysfs_entry queue_requests_entry = { - .attr = {.name = "nr_requests", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "nr_requests", .mode = 0644 }, .show = queue_requests_show, .store = queue_requests_store, }; static struct queue_sysfs_entry queue_ra_entry = { - .attr = {.name = "read_ahead_kb", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "read_ahead_kb", .mode = 0644 }, .show = queue_ra_show, .store = queue_ra_store, }; static struct queue_sysfs_entry queue_max_sectors_entry = { - .attr = {.name = "max_sectors_kb", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "max_sectors_kb", .mode = 0644 }, .show = queue_max_sectors_show, .store = queue_max_sectors_store, }; static struct queue_sysfs_entry queue_max_hw_sectors_entry = { - .attr = {.name = "max_hw_sectors_kb", .mode = S_IRUGO }, + .attr = {.name = "max_hw_sectors_kb", .mode = 0444 }, .show = queue_max_hw_sectors_show, }; static struct queue_sysfs_entry queue_max_segments_entry = { - .attr = {.name = "max_segments", .mode = S_IRUGO }, + .attr = {.name = "max_segments", .mode = 0444 }, .show = queue_max_segments_show, }; static struct queue_sysfs_entry queue_max_discard_segments_entry = { - .attr = {.name = "max_discard_segments", .mode = S_IRUGO }, + .attr = {.name = "max_discard_segments", .mode = 0444 }, .show = queue_max_discard_segments_show, }; static struct queue_sysfs_entry queue_max_integrity_segments_entry = { - .attr = {.name = "max_integrity_segments", .mode = S_IRUGO }, + .attr = {.name = "max_integrity_segments", .mode = 0444 }, .show = queue_max_integrity_segments_show, }; static struct queue_sysfs_entry queue_max_segment_size_entry = { - .attr = {.name = "max_segment_size", .mode = S_IRUGO }, + .attr = {.name = "max_segment_size", .mode = 0444 }, .show = queue_max_segment_size_show, }; static struct queue_sysfs_entry queue_iosched_entry = { - .attr = {.name = "scheduler", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "scheduler", .mode = 0644 }, .show = elv_iosched_show, .store = elv_iosched_store, }; static struct queue_sysfs_entry queue_hw_sector_size_entry = { - .attr = {.name = "hw_sector_size", .mode = S_IRUGO }, + .attr = {.name = "hw_sector_size", .mode = 0444 }, .show = queue_logical_block_size_show, }; static struct queue_sysfs_entry queue_logical_block_size_entry = { - .attr = {.name = "logical_block_size", .mode = S_IRUGO }, + .attr = {.name = "logical_block_size", .mode = 0444 }, .show = queue_logical_block_size_show, }; static struct queue_sysfs_entry queue_physical_block_size_entry = { - .attr = {.name = "physical_block_size", .mode = S_IRUGO }, + .attr = {.name = "physical_block_size", .mode = 0444 }, .show = queue_physical_block_size_show, }; static struct queue_sysfs_entry queue_chunk_sectors_entry = { - .attr = {.name = "chunk_sectors", .mode = S_IRUGO }, + .attr = {.name = "chunk_sectors", .mode = 0444 }, .show = queue_chunk_sectors_show, }; static struct queue_sysfs_entry queue_io_min_entry = { - .attr = {.name = "minimum_io_size", .mode = S_IRUGO }, + .attr = {.name = "minimum_io_size", .mode = 0444 }, .show = queue_io_min_show, }; static struct queue_sysfs_entry queue_io_opt_entry = { - .attr = {.name = "optimal_io_size", .mode = S_IRUGO }, + .attr = {.name = "optimal_io_size", .mode = 0444 }, .show = queue_io_opt_show, }; static struct queue_sysfs_entry queue_discard_granularity_entry = { - .attr = {.name = "discard_granularity", .mode = S_IRUGO }, + .attr = {.name = "discard_granularity", .mode = 0444 }, .show = queue_discard_granularity_show, }; static struct queue_sysfs_entry queue_discard_max_hw_entry = { - .attr = {.name = "discard_max_hw_bytes", .mode = S_IRUGO }, + .attr = {.name = "discard_max_hw_bytes", .mode = 0444 }, .show = queue_discard_max_hw_show, }; static struct queue_sysfs_entry queue_discard_max_entry = { - .attr = {.name = "discard_max_bytes", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "discard_max_bytes", .mode = 0644 }, .show = queue_discard_max_show, .store = queue_discard_max_store, }; static struct queue_sysfs_entry queue_discard_zeroes_data_entry = { - .attr = {.name = "discard_zeroes_data", .mode = S_IRUGO }, + .attr = {.name = "discard_zeroes_data", .mode = 0444 }, .show = queue_discard_zeroes_data_show, }; static struct queue_sysfs_entry queue_write_same_max_entry = { - .attr = {.name = "write_same_max_bytes", .mode = S_IRUGO }, + .attr = {.name = "write_same_max_bytes", .mode = 0444 }, .show = queue_write_same_max_show, }; static struct queue_sysfs_entry queue_write_zeroes_max_entry = { - .attr = {.name = "write_zeroes_max_bytes", .mode = S_IRUGO }, + .attr = {.name = "write_zeroes_max_bytes", .mode = 0444 }, .show = queue_write_zeroes_max_show, }; static struct queue_sysfs_entry queue_nonrot_entry = { - .attr = {.name = "rotational", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "rotational", .mode = 0644 }, .show = queue_show_nonrot, .store = queue_store_nonrot, }; static struct queue_sysfs_entry queue_zoned_entry = { - .attr = {.name = "zoned", .mode = S_IRUGO }, + .attr = {.name = "zoned", .mode = 0444 }, .show = queue_zoned_show, }; static struct queue_sysfs_entry queue_nomerges_entry = { - .attr = {.name = "nomerges", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "nomerges", .mode = 0644 }, .show = queue_nomerges_show, .store = queue_nomerges_store, }; static struct queue_sysfs_entry queue_rq_affinity_entry = { - .attr = {.name = "rq_affinity", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "rq_affinity", .mode = 0644 }, .show = queue_rq_affinity_show, .store = queue_rq_affinity_store, }; static struct queue_sysfs_entry queue_iostats_entry = { - .attr = {.name = "iostats", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "iostats", .mode = 0644 }, .show = queue_show_iostats, .store = queue_store_iostats, }; static struct queue_sysfs_entry queue_random_entry = { - .attr = {.name = "add_random", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "add_random", .mode = 0644 }, .show = queue_show_random, .store = queue_store_random, }; static struct queue_sysfs_entry queue_poll_entry = { - .attr = {.name = "io_poll", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "io_poll", .mode = 0644 }, .show = queue_poll_show, .store = queue_poll_store, }; static struct queue_sysfs_entry queue_poll_delay_entry = { - .attr = {.name = "io_poll_delay", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "io_poll_delay", .mode = 0644 }, .show = queue_poll_delay_show, .store = queue_poll_delay_store, }; static struct queue_sysfs_entry queue_wc_entry = { - .attr = {.name = "write_cache", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "write_cache", .mode = 0644 }, .show = queue_wc_show, .store = queue_wc_store, }; +static struct queue_sysfs_entry queue_fua_entry = { + .attr = {.name = "fua", .mode = 0444 }, + .show = queue_fua_show, +}; + static struct queue_sysfs_entry queue_dax_entry = { - .attr = {.name = "dax", .mode = S_IRUGO }, + .attr = {.name = "dax", .mode = 0444 }, .show = queue_dax_show, }; static struct queue_sysfs_entry queue_wb_lat_entry = { - .attr = {.name = "wbt_lat_usec", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "wbt_lat_usec", .mode = 0644 }, .show = queue_wb_lat_show, .store = queue_wb_lat_store, }; #ifdef CONFIG_BLK_DEV_THROTTLING_LOW static struct queue_sysfs_entry throtl_sample_time_entry = { - .attr = {.name = "throttle_sample_time", .mode = S_IRUGO | S_IWUSR }, + .attr = {.name = "throttle_sample_time", .mode = 0644 }, .show = blk_throtl_sample_time_show, .store = blk_throtl_sample_time_store, }; @@ -708,6 +718,7 @@ static struct attribute *default_attrs[] = { &queue_random_entry.attr, &queue_poll_entry.attr, &queue_wc_entry.attr, + &queue_fua_entry.attr, &queue_dax_entry.attr, &queue_wb_lat_entry.attr, &queue_poll_delay_entry.attr, @@ -813,8 +824,7 @@ static void __blk_release_queue(struct work_struct *work) if (q->mq_ops) blk_mq_debugfs_unregister(q); - if (q->bio_split) - bioset_free(q->bio_split); + bioset_exit(&q->bio_split); ida_simple_remove(&blk_queue_ida, q->id); call_rcu(&q->rcu_head, blk_free_queue_rcu); diff --git a/block/blk-tag.c b/block/blk-tag.c index 09f19c6c52ce..fbc153aef166 100644 --- a/block/blk-tag.c +++ b/block/blk-tag.c @@ -99,12 +99,12 @@ init_tag_map(struct request_queue *q, struct blk_queue_tag *tags, int depth) __func__, depth); } - tag_index = kzalloc(depth * sizeof(struct request *), GFP_ATOMIC); + tag_index = kcalloc(depth, sizeof(struct request *), GFP_ATOMIC); if (!tag_index) goto fail; nr_ulongs = ALIGN(depth, BITS_PER_LONG) / BITS_PER_LONG; - tag_map = kzalloc(nr_ulongs * sizeof(unsigned long), GFP_ATOMIC); + tag_map = kcalloc(nr_ulongs, sizeof(unsigned long), GFP_ATOMIC); if (!tag_map) goto fail; @@ -188,7 +188,6 @@ int blk_queue_init_tags(struct request_queue *q, int depth, */ q->queue_tags = tags; queue_flag_set_unlocked(QUEUE_FLAG_QUEUED, q); - INIT_LIST_HEAD(&q->tag_busy_list); return 0; } EXPORT_SYMBOL(blk_queue_init_tags); @@ -374,27 +373,6 @@ int blk_queue_start_tag(struct request_queue *q, struct request *rq) rq->tag = tag; bqt->tag_index[tag] = rq; blk_start_request(rq); - list_add(&rq->queuelist, &q->tag_busy_list); return 0; } EXPORT_SYMBOL(blk_queue_start_tag); - -/** - * blk_queue_invalidate_tags - invalidate all pending tags - * @q: the request queue for the device - * - * Description: - * Hardware conditions may dictate a need to stop all pending requests. - * In this case, we will safely clear the block side of the tag queue and - * readd all requests to the request queue in the right order. - **/ -void blk_queue_invalidate_tags(struct request_queue *q) -{ - struct list_head *tmp, *n; - - lockdep_assert_held(q->queue_lock); - - list_for_each_safe(tmp, n, &q->tag_busy_list) - blk_requeue_request(q, list_entry_rq(tmp)); -} -EXPORT_SYMBOL(blk_queue_invalidate_tags); diff --git a/block/blk-throttle.c b/block/blk-throttle.c index c5a131673733..82282e6fdcf8 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -36,8 +36,6 @@ static int throtl_quantum = 32; */ #define LATENCY_FILTERED_HD (1000L) /* 1ms */ -#define SKIP_LATENCY (((u64)1) << BLK_STAT_RES_SHIFT) - static struct blkcg_policy blkcg_policy_throtl; /* A workqueue to queue throttle related work */ @@ -821,7 +819,7 @@ static bool throtl_slice_used(struct throtl_grp *tg, bool rw) if (time_in_range(jiffies, tg->slice_start[rw], tg->slice_end[rw])) return false; - return 1; + return true; } /* Trim the used slices and adjust slice start accordingly */ @@ -931,7 +929,7 @@ static bool tg_with_in_iops_limit(struct throtl_grp *tg, struct bio *bio, if (wait) *wait = jiffy_wait; - return 0; + return false; } static bool tg_with_in_bps_limit(struct throtl_grp *tg, struct bio *bio, @@ -974,7 +972,7 @@ static bool tg_with_in_bps_limit(struct throtl_grp *tg, struct bio *bio, jiffy_wait = jiffy_wait + (jiffy_elapsed_rnd - jiffy_elapsed); if (wait) *wait = jiffy_wait; - return 0; + return false; } /* @@ -1024,7 +1022,7 @@ static bool tg_may_dispatch(struct throtl_grp *tg, struct bio *bio, tg_with_in_iops_limit(tg, bio, &iops_wait)) { if (wait) *wait = 0; - return 1; + return true; } max_wait = max(bps_wait, iops_wait); @@ -1035,7 +1033,7 @@ static bool tg_may_dispatch(struct throtl_grp *tg, struct bio *bio, if (time_before(tg->slice_end[rw], jiffies + max_wait)) throtl_extend_slice(tg, rw, jiffies + max_wait); - return 0; + return false; } static void throtl_charge_bio(struct throtl_grp *tg, struct bio *bio) @@ -1209,7 +1207,7 @@ static int throtl_select_dispatch(struct throtl_service_queue *parent_sq) while (1) { struct throtl_grp *tg = throtl_rb_first(parent_sq); - struct throtl_service_queue *sq = &tg->service_queue; + struct throtl_service_queue *sq; if (!tg) break; @@ -1221,6 +1219,7 @@ static int throtl_select_dispatch(struct throtl_service_queue *parent_sq) nr_disp += throtl_dispatch_tg(tg); + sq = &tg->service_queue; if (sq->nr_queued[0] || sq->nr_queued[1]) tg_update_disptime(tg); @@ -2139,7 +2138,7 @@ static void blk_throtl_assoc_bio(struct throtl_grp *tg, struct bio *bio) bio->bi_cg_private = tg; blkg_get(tg_to_blkg(tg)); } - blk_stat_set_issue(&bio->bi_issue_stat, bio_sectors(bio)); + bio_issue_init(&bio->bi_issue, bio_sectors(bio)); #endif } @@ -2251,7 +2250,7 @@ out: #ifdef CONFIG_BLK_DEV_THROTTLING_LOW if (throttled || !td->track_bio_latency) - bio->bi_issue_stat.stat |= SKIP_LATENCY; + bio->bi_issue.value |= BIO_ISSUE_THROTL_SKIP_LATENCY; #endif return throttled; } @@ -2281,8 +2280,7 @@ void blk_throtl_stat_add(struct request *rq, u64 time_ns) struct request_queue *q = rq->q; struct throtl_data *td = q->td; - throtl_track_latency(td, blk_stat_size(&rq->issue_stat), - req_op(rq), time_ns >> 10); + throtl_track_latency(td, rq->throtl_size, req_op(rq), time_ns >> 10); } void blk_throtl_bio_endio(struct bio *bio) @@ -2302,8 +2300,8 @@ void blk_throtl_bio_endio(struct bio *bio) finish_time_ns = ktime_get_ns(); tg->last_finish_time = finish_time_ns >> 10; - start_time = blk_stat_time(&bio->bi_issue_stat) >> 10; - finish_time = __blk_stat_time(finish_time_ns) >> 10; + start_time = bio_issue_time(&bio->bi_issue) >> 10; + finish_time = __bio_issue_time(finish_time_ns) >> 10; if (!start_time || finish_time <= start_time) { blkg_put(tg_to_blkg(tg)); return; @@ -2311,16 +2309,15 @@ void blk_throtl_bio_endio(struct bio *bio) lat = finish_time - start_time; /* this is only for bio based driver */ - if (!(bio->bi_issue_stat.stat & SKIP_LATENCY)) - throtl_track_latency(tg->td, blk_stat_size(&bio->bi_issue_stat), - bio_op(bio), lat); + if (!(bio->bi_issue.value & BIO_ISSUE_THROTL_SKIP_LATENCY)) + throtl_track_latency(tg->td, bio_issue_size(&bio->bi_issue), + bio_op(bio), lat); if (tg->latency_target && lat >= tg->td->filtered_latency) { int bucket; unsigned int threshold; - bucket = request_bucket_index( - blk_stat_size(&bio->bi_issue_stat)); + bucket = request_bucket_index(bio_issue_size(&bio->bi_issue)); threshold = tg->td->avg_buckets[rw][bucket].latency + tg->latency_target; if (lat > threshold) diff --git a/block/blk-timeout.c b/block/blk-timeout.c index 652d4d4d3e97..4b8a48d48ba1 100644 --- a/block/blk-timeout.c +++ b/block/blk-timeout.c @@ -86,14 +86,11 @@ static void blk_rq_timed_out(struct request *req) if (q->rq_timed_out_fn) ret = q->rq_timed_out_fn(req); switch (ret) { - case BLK_EH_HANDLED: - __blk_complete_request(req); - break; case BLK_EH_RESET_TIMER: blk_add_timer(req); blk_clear_rq_complete(req); break; - case BLK_EH_NOT_HANDLED: + case BLK_EH_DONE: /* * LLD handles this for now but in the future * we can send a request msg to abort the command @@ -214,7 +211,6 @@ void blk_add_timer(struct request *req) req->timeout = q->rq_timeout; blk_rq_set_deadline(req, jiffies + req->timeout); - req->rq_flags &= ~RQF_MQ_TIMEOUT_EXPIRED; /* * Only the non-mq case needs to add the request to a protected list. diff --git a/block/blk-wbt.c b/block/blk-wbt.c index f92fc84b5e2c..4f89b28fa652 100644 --- a/block/blk-wbt.c +++ b/block/blk-wbt.c @@ -29,6 +29,26 @@ #define CREATE_TRACE_POINTS #include +static inline void wbt_clear_state(struct request *rq) +{ + rq->wbt_flags = 0; +} + +static inline enum wbt_flags wbt_flags(struct request *rq) +{ + return rq->wbt_flags; +} + +static inline bool wbt_is_tracked(struct request *rq) +{ + return rq->wbt_flags & WBT_TRACKED; +} + +static inline bool wbt_is_read(struct request *rq) +{ + return rq->wbt_flags & WBT_READ; +} + enum { /* * Default setting, we'll scale up (to 75% of QD max) or down (min 1) @@ -101,9 +121,15 @@ static bool wb_recent_wait(struct rq_wb *rwb) return time_before(jiffies, wb->dirty_sleep + HZ); } -static inline struct rq_wait *get_rq_wait(struct rq_wb *rwb, bool is_kswapd) +static inline struct rq_wait *get_rq_wait(struct rq_wb *rwb, + enum wbt_flags wb_acct) { - return &rwb->rq_wait[is_kswapd]; + if (wb_acct & WBT_KSWAPD) + return &rwb->rq_wait[WBT_RWQ_KSWAPD]; + else if (wb_acct & WBT_DISCARD) + return &rwb->rq_wait[WBT_RWQ_DISCARD]; + + return &rwb->rq_wait[WBT_RWQ_BG]; } static void rwb_wake_all(struct rq_wb *rwb) @@ -126,7 +152,7 @@ void __wbt_done(struct rq_wb *rwb, enum wbt_flags wb_acct) if (!(wb_acct & WBT_TRACKED)) return; - rqw = get_rq_wait(rwb, wb_acct & WBT_KSWAPD); + rqw = get_rq_wait(rwb, wb_acct); inflight = atomic_dec_return(&rqw->inflight); /* @@ -139,10 +165,13 @@ void __wbt_done(struct rq_wb *rwb, enum wbt_flags wb_acct) } /* - * If the device does write back caching, drop further down - * before we wake people up. + * For discards, our limit is always the background. For writes, if + * the device does write back caching, drop further down before we + * wake people up. */ - if (rwb->wc && !wb_recent_wait(rwb)) + if (wb_acct & WBT_DISCARD) + limit = rwb->wb_background; + else if (rwb->wc && !wb_recent_wait(rwb)) limit = 0; else limit = rwb->wb_normal; @@ -165,24 +194,24 @@ void __wbt_done(struct rq_wb *rwb, enum wbt_flags wb_acct) * Called on completion of a request. Note that it's also called when * a request is merged, when the request gets freed. */ -void wbt_done(struct rq_wb *rwb, struct blk_issue_stat *stat) +void wbt_done(struct rq_wb *rwb, struct request *rq) { if (!rwb) return; - if (!wbt_is_tracked(stat)) { - if (rwb->sync_cookie == stat) { + if (!wbt_is_tracked(rq)) { + if (rwb->sync_cookie == rq) { rwb->sync_issue = 0; rwb->sync_cookie = NULL; } - if (wbt_is_read(stat)) + if (wbt_is_read(rq)) wb_timestamp(rwb, &rwb->last_comp); } else { - WARN_ON_ONCE(stat == rwb->sync_cookie); - __wbt_done(rwb, wbt_stat_to_mask(stat)); + WARN_ON_ONCE(rq == rwb->sync_cookie); + __wbt_done(rwb, wbt_flags(rq)); } - wbt_clear_state(stat); + wbt_clear_state(rq); } /* @@ -479,6 +508,9 @@ static inline unsigned int get_limit(struct rq_wb *rwb, unsigned long rw) { unsigned int limit; + if ((rw & REQ_OP_MASK) == REQ_OP_DISCARD) + return rwb->wb_background; + /* * At this point we know it's a buffered write. If this is * kswapd trying to free memory, or REQ_SYNC is set, then @@ -529,11 +561,12 @@ static inline bool may_queue(struct rq_wb *rwb, struct rq_wait *rqw, * Block if we will exceed our limit, or if we are currently waiting for * the timer to kick off queuing again. */ -static void __wbt_wait(struct rq_wb *rwb, unsigned long rw, spinlock_t *lock) +static void __wbt_wait(struct rq_wb *rwb, enum wbt_flags wb_acct, + unsigned long rw, spinlock_t *lock) __releases(lock) __acquires(lock) { - struct rq_wait *rqw = get_rq_wait(rwb, current_is_kswapd()); + struct rq_wait *rqw = get_rq_wait(rwb, wb_acct); DEFINE_WAIT(wait); if (may_queue(rwb, rqw, &wait, rw)) @@ -559,21 +592,20 @@ static void __wbt_wait(struct rq_wb *rwb, unsigned long rw, spinlock_t *lock) static inline bool wbt_should_throttle(struct rq_wb *rwb, struct bio *bio) { - const int op = bio_op(bio); - - /* - * If not a WRITE, do nothing - */ - if (op != REQ_OP_WRITE) - return false; - - /* - * Don't throttle WRITE_ODIRECT - */ - if ((bio->bi_opf & (REQ_SYNC | REQ_IDLE)) == (REQ_SYNC | REQ_IDLE)) + switch (bio_op(bio)) { + case REQ_OP_WRITE: + /* + * Don't throttle WRITE_ODIRECT + */ + if ((bio->bi_opf & (REQ_SYNC | REQ_IDLE)) == + (REQ_SYNC | REQ_IDLE)) + return false; + /* fallthrough */ + case REQ_OP_DISCARD: + return true; + default: return false; - - return true; + } } /* @@ -584,7 +616,7 @@ static inline bool wbt_should_throttle(struct rq_wb *rwb, struct bio *bio) */ enum wbt_flags wbt_wait(struct rq_wb *rwb, struct bio *bio, spinlock_t *lock) { - unsigned int ret = 0; + enum wbt_flags ret = 0; if (!rwb_enabled(rwb)) return 0; @@ -598,41 +630,42 @@ enum wbt_flags wbt_wait(struct rq_wb *rwb, struct bio *bio, spinlock_t *lock) return ret; } - __wbt_wait(rwb, bio->bi_opf, lock); + if (current_is_kswapd()) + ret |= WBT_KSWAPD; + if (bio_op(bio) == REQ_OP_DISCARD) + ret |= WBT_DISCARD; + + __wbt_wait(rwb, ret, bio->bi_opf, lock); if (!blk_stat_is_active(rwb->cb)) rwb_arm_timer(rwb); - if (current_is_kswapd()) - ret |= WBT_KSWAPD; - return ret | WBT_TRACKED; } -void wbt_issue(struct rq_wb *rwb, struct blk_issue_stat *stat) +void wbt_issue(struct rq_wb *rwb, struct request *rq) { if (!rwb_enabled(rwb)) return; /* - * Track sync issue, in case it takes a long time to complete. Allows - * us to react quicker, if a sync IO takes a long time to complete. - * Note that this is just a hint. 'stat' can go away when the - * request completes, so it's important we never dereference it. We - * only use the address to compare with, which is why we store the - * sync_issue time locally. + * Track sync issue, in case it takes a long time to complete. Allows us + * to react quicker, if a sync IO takes a long time to complete. Note + * that this is just a hint. The request can go away when it completes, + * so it's important we never dereference it. We only use the address to + * compare with, which is why we store the sync_issue time locally. */ - if (wbt_is_read(stat) && !rwb->sync_issue) { - rwb->sync_cookie = stat; - rwb->sync_issue = blk_stat_time(stat); + if (wbt_is_read(rq) && !rwb->sync_issue) { + rwb->sync_cookie = rq; + rwb->sync_issue = rq->io_start_time_ns; } } -void wbt_requeue(struct rq_wb *rwb, struct blk_issue_stat *stat) +void wbt_requeue(struct rq_wb *rwb, struct request *rq) { if (!rwb_enabled(rwb)) return; - if (stat == rwb->sync_cookie) { + if (rq == rwb->sync_cookie) { rwb->sync_issue = 0; rwb->sync_cookie = NULL; } @@ -701,7 +734,7 @@ static int wbt_data_dir(const struct request *rq) if (op == REQ_OP_READ) return READ; - else if (op == REQ_OP_WRITE || op == REQ_OP_FLUSH) + else if (op_is_write(op)) return WRITE; /* don't account */ @@ -713,8 +746,6 @@ int wbt_init(struct request_queue *q) struct rq_wb *rwb; int i; - BUILD_BUG_ON(WBT_NR_BITS > BLK_STAT_RES_BITS); - rwb = kzalloc(sizeof(*rwb), GFP_KERNEL); if (!rwb) return -ENOMEM; diff --git a/block/blk-wbt.h b/block/blk-wbt.h index a232c98fbf4d..300df531d0a6 100644 --- a/block/blk-wbt.h +++ b/block/blk-wbt.h @@ -14,12 +14,16 @@ enum wbt_flags { WBT_TRACKED = 1, /* write, tracked for throttling */ WBT_READ = 2, /* read */ WBT_KSWAPD = 4, /* write, from kswapd */ + WBT_DISCARD = 8, /* discard */ - WBT_NR_BITS = 3, /* number of bits */ + WBT_NR_BITS = 4, /* number of bits */ }; enum { - WBT_NUM_RWQ = 2, + WBT_RWQ_BG = 0, + WBT_RWQ_KSWAPD, + WBT_RWQ_DISCARD, + WBT_NUM_RWQ, }; /* @@ -31,31 +35,6 @@ enum { WBT_STATE_ON_MANUAL = 2, }; -static inline void wbt_clear_state(struct blk_issue_stat *stat) -{ - stat->stat &= ~BLK_STAT_RES_MASK; -} - -static inline enum wbt_flags wbt_stat_to_mask(struct blk_issue_stat *stat) -{ - return (stat->stat & BLK_STAT_RES_MASK) >> BLK_STAT_RES_SHIFT; -} - -static inline void wbt_track(struct blk_issue_stat *stat, enum wbt_flags wb_acct) -{ - stat->stat |= ((u64) wb_acct) << BLK_STAT_RES_SHIFT; -} - -static inline bool wbt_is_tracked(struct blk_issue_stat *stat) -{ - return (stat->stat >> BLK_STAT_RES_SHIFT) & WBT_TRACKED; -} - -static inline bool wbt_is_read(struct blk_issue_stat *stat) -{ - return (stat->stat >> BLK_STAT_RES_SHIFT) & WBT_READ; -} - struct rq_wait { wait_queue_head_t wait; atomic_t inflight; @@ -84,7 +63,7 @@ struct rq_wb { struct blk_stat_callback *cb; - s64 sync_issue; + u64 sync_issue; void *sync_cookie; unsigned int wc; @@ -109,14 +88,19 @@ static inline unsigned int wbt_inflight(struct rq_wb *rwb) #ifdef CONFIG_BLK_WBT +static inline void wbt_track(struct request *rq, enum wbt_flags flags) +{ + rq->wbt_flags |= flags; +} + void __wbt_done(struct rq_wb *, enum wbt_flags); -void wbt_done(struct rq_wb *, struct blk_issue_stat *); +void wbt_done(struct rq_wb *, struct request *); enum wbt_flags wbt_wait(struct rq_wb *, struct bio *, spinlock_t *); int wbt_init(struct request_queue *); void wbt_exit(struct request_queue *); void wbt_update_limits(struct rq_wb *); -void wbt_requeue(struct rq_wb *, struct blk_issue_stat *); -void wbt_issue(struct rq_wb *, struct blk_issue_stat *); +void wbt_requeue(struct rq_wb *, struct request *); +void wbt_issue(struct rq_wb *, struct request *); void wbt_disable_default(struct request_queue *); void wbt_enable_default(struct request_queue *); @@ -127,10 +111,13 @@ u64 wbt_default_latency_nsec(struct request_queue *); #else +static inline void wbt_track(struct request *rq, enum wbt_flags flags) +{ +} static inline void __wbt_done(struct rq_wb *rwb, enum wbt_flags flags) { } -static inline void wbt_done(struct rq_wb *rwb, struct blk_issue_stat *stat) +static inline void wbt_done(struct rq_wb *rwb, struct request *rq) { } static inline enum wbt_flags wbt_wait(struct rq_wb *rwb, struct bio *bio, @@ -148,10 +135,10 @@ static inline void wbt_exit(struct request_queue *q) static inline void wbt_update_limits(struct rq_wb *rwb) { } -static inline void wbt_requeue(struct rq_wb *rwb, struct blk_issue_stat *stat) +static inline void wbt_requeue(struct rq_wb *rwb, struct request *rq) { } -static inline void wbt_issue(struct rq_wb *rwb, struct blk_issue_stat *stat) +static inline void wbt_issue(struct rq_wb *rwb, struct request *rq) { } static inline void wbt_disable_default(struct request_queue *q) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 08e84ef2bc05..51000914e23f 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -328,7 +328,11 @@ int blkdev_report_zones_ioctl(struct block_device *bdev, fmode_t mode, if (!rep.nr_zones) return -EINVAL; - zones = kcalloc(rep.nr_zones, sizeof(struct blk_zone), GFP_KERNEL); + if (rep.nr_zones > INT_MAX / sizeof(struct blk_zone)) + return -ERANGE; + + zones = kvmalloc_array(rep.nr_zones, sizeof(struct blk_zone), + GFP_KERNEL | __GFP_ZERO); if (!zones) return -ENOMEM; @@ -350,7 +354,7 @@ int blkdev_report_zones_ioctl(struct block_device *bdev, fmode_t mode, } out: - kfree(zones); + kvfree(zones); return ret; } diff --git a/block/blk.h b/block/blk.h index b034fd2460c4..8d23aea96ce9 100644 --- a/block/blk.h +++ b/block/blk.h @@ -186,7 +186,7 @@ unsigned int blk_plug_queued_count(struct request_queue *q); void blk_account_io_start(struct request *req, bool new_io); void blk_account_io_completion(struct request *req, unsigned int bytes); -void blk_account_io_done(struct request *req); +void blk_account_io_done(struct request *req, u64 now); /* * EH timer and IO completion will both attempt to 'grab' the request, make @@ -231,6 +231,9 @@ static inline void elv_deactivate_rq(struct request_queue *q, struct request *rq e->type->ops.sq.elevator_deactivate_req_fn(q, rq); } +int elevator_init(struct request_queue *); +int elevator_init_mq(struct request_queue *q); +void elevator_exit(struct request_queue *, struct elevator_queue *); int elv_register_queue(struct request_queue *q); void elv_unregister_queue(struct request_queue *q); diff --git a/block/bounce.c b/block/bounce.c index dd0b93f2a871..fd31347b7836 100644 --- a/block/bounce.c +++ b/block/bounce.c @@ -28,28 +28,29 @@ #define POOL_SIZE 64 #define ISA_POOL_SIZE 16 -static struct bio_set *bounce_bio_set, *bounce_bio_split; -static mempool_t *page_pool, *isa_page_pool; +static struct bio_set bounce_bio_set, bounce_bio_split; +static mempool_t page_pool, isa_page_pool; #if defined(CONFIG_HIGHMEM) static __init int init_emergency_pool(void) { + int ret; #if defined(CONFIG_HIGHMEM) && !defined(CONFIG_MEMORY_HOTPLUG) if (max_pfn <= max_low_pfn) return 0; #endif - page_pool = mempool_create_page_pool(POOL_SIZE, 0); - BUG_ON(!page_pool); + ret = mempool_init_page_pool(&page_pool, POOL_SIZE, 0); + BUG_ON(ret); pr_info("pool size: %d pages\n", POOL_SIZE); - bounce_bio_set = bioset_create(BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS); - BUG_ON(!bounce_bio_set); - if (bioset_integrity_create(bounce_bio_set, BIO_POOL_SIZE)) + ret = bioset_init(&bounce_bio_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS); + BUG_ON(ret); + if (bioset_integrity_create(&bounce_bio_set, BIO_POOL_SIZE)) BUG_ON(1); - bounce_bio_split = bioset_create(BIO_POOL_SIZE, 0, 0); - BUG_ON(!bounce_bio_split); + ret = bioset_init(&bounce_bio_split, BIO_POOL_SIZE, 0, 0); + BUG_ON(ret); return 0; } @@ -63,14 +64,11 @@ __initcall(init_emergency_pool); */ static void bounce_copy_vec(struct bio_vec *to, unsigned char *vfrom) { - unsigned long flags; unsigned char *vto; - local_irq_save(flags); vto = kmap_atomic(to->bv_page); memcpy(vto + to->bv_offset, vfrom, to->bv_len); kunmap_atomic(vto); - local_irq_restore(flags); } #else /* CONFIG_HIGHMEM */ @@ -94,12 +92,14 @@ static void *mempool_alloc_pages_isa(gfp_t gfp_mask, void *data) */ int init_emergency_isa_pool(void) { - if (isa_page_pool) + int ret; + + if (mempool_initialized(&isa_page_pool)) return 0; - isa_page_pool = mempool_create(ISA_POOL_SIZE, mempool_alloc_pages_isa, - mempool_free_pages, (void *) 0); - BUG_ON(!isa_page_pool); + ret = mempool_init(&isa_page_pool, ISA_POOL_SIZE, mempool_alloc_pages_isa, + mempool_free_pages, (void *) 0); + BUG_ON(ret); pr_info("isa pool size: %d pages\n", ISA_POOL_SIZE); return 0; @@ -166,13 +166,13 @@ static void bounce_end_io(struct bio *bio, mempool_t *pool) static void bounce_end_io_write(struct bio *bio) { - bounce_end_io(bio, page_pool); + bounce_end_io(bio, &page_pool); } static void bounce_end_io_write_isa(struct bio *bio) { - bounce_end_io(bio, isa_page_pool); + bounce_end_io(bio, &isa_page_pool); } static void __bounce_end_io_read(struct bio *bio, mempool_t *pool) @@ -187,12 +187,12 @@ static void __bounce_end_io_read(struct bio *bio, mempool_t *pool) static void bounce_end_io_read(struct bio *bio) { - __bounce_end_io_read(bio, page_pool); + __bounce_end_io_read(bio, &page_pool); } static void bounce_end_io_read_isa(struct bio *bio) { - __bounce_end_io_read(bio, isa_page_pool); + __bounce_end_io_read(bio, &isa_page_pool); } static void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig, @@ -217,13 +217,13 @@ static void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig, return; if (!passthrough && sectors < bio_sectors(*bio_orig)) { - bio = bio_split(*bio_orig, sectors, GFP_NOIO, bounce_bio_split); + bio = bio_split(*bio_orig, sectors, GFP_NOIO, &bounce_bio_split); bio_chain(bio, *bio_orig); generic_make_request(*bio_orig); *bio_orig = bio; } bio = bio_clone_bioset(*bio_orig, GFP_NOIO, passthrough ? NULL : - bounce_bio_set); + &bounce_bio_set); bio_for_each_segment_all(to, bio, i) { struct page *page = to->bv_page; @@ -250,7 +250,7 @@ static void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig, bio->bi_flags |= (1 << BIO_BOUNCED); - if (pool == page_pool) { + if (pool == &page_pool) { bio->bi_end_io = bounce_end_io_write; if (rw == READ) bio->bi_end_io = bounce_end_io_read; @@ -282,10 +282,10 @@ void blk_queue_bounce(struct request_queue *q, struct bio **bio_orig) if (!(q->bounce_gfp & GFP_DMA)) { if (q->limits.bounce_pfn >= blk_max_pfn) return; - pool = page_pool; + pool = &page_pool; } else { - BUG_ON(!isa_page_pool); - pool = isa_page_pool; + BUG_ON(!mempool_initialized(&isa_page_pool)); + pool = &isa_page_pool; } /* diff --git a/block/bsg-lib.c b/block/bsg-lib.c index fc2e5ff2c4b9..9419def8c017 100644 --- a/block/bsg-lib.c +++ b/block/bsg-lib.c @@ -303,11 +303,9 @@ static void bsg_exit_rq(struct request_queue *q, struct request *req) * @name: device to give bsg device * @job_fn: bsg job handler * @dd_job_size: size of LLD data needed for each job - * @release: @dev release function */ struct request_queue *bsg_setup_queue(struct device *dev, const char *name, - bsg_job_fn *job_fn, int dd_job_size, - void (*release)(struct device *)) + bsg_job_fn *job_fn, int dd_job_size) { struct request_queue *q; int ret; @@ -331,7 +329,7 @@ struct request_queue *bsg_setup_queue(struct device *dev, const char *name, blk_queue_softirq_done(q, bsg_softirq_done); blk_queue_rq_timeout(q, BLK_DEFAULT_SG_TIMEOUT); - ret = bsg_register_queue(q, dev, name, &bsg_transport_ops, release); + ret = bsg_register_queue(q, dev, name, &bsg_transport_ops); if (ret) { printk(KERN_ERR "%s: bsg interface failed to " "initialize - register queue\n", dev->kobj.name); diff --git a/block/bsg.c b/block/bsg.c index defa06c11858..66602c489956 100644 --- a/block/bsg.c +++ b/block/bsg.c @@ -226,8 +226,7 @@ bsg_map_hdr(struct request_queue *q, struct sg_io_v4 *hdr, fmode_t mode) return ERR_PTR(ret); rq = blk_get_request(q, hdr->dout_xfer_len ? - REQ_OP_SCSI_OUT : REQ_OP_SCSI_IN, - GFP_KERNEL); + REQ_OP_SCSI_OUT : REQ_OP_SCSI_IN, 0); if (IS_ERR(rq)) return rq; @@ -249,7 +248,7 @@ bsg_map_hdr(struct request_queue *q, struct sg_io_v4 *hdr, fmode_t mode) goto out; } - next_rq = blk_get_request(q, REQ_OP_SCSI_IN, GFP_KERNEL); + next_rq = blk_get_request(q, REQ_OP_SCSI_IN, 0); if (IS_ERR(next_rq)) { ret = PTR_ERR(next_rq); goto out; @@ -650,18 +649,6 @@ static struct bsg_device *bsg_alloc_device(void) return bd; } -static void bsg_kref_release_function(struct kref *kref) -{ - struct bsg_class_device *bcd = - container_of(kref, struct bsg_class_device, ref); - struct device *parent = bcd->parent; - - if (bcd->release) - bcd->release(bcd->parent); - - put_device(parent); -} - static int bsg_put_device(struct bsg_device *bd) { int ret = 0, do_free; @@ -694,7 +681,6 @@ static int bsg_put_device(struct bsg_device *bd) kfree(bd); out: - kref_put(&q->bsg_dev.ref, bsg_kref_release_function); if (do_free) blk_put_queue(q); return ret; @@ -707,6 +693,8 @@ static struct bsg_device *bsg_add_device(struct inode *inode, struct bsg_device *bd; unsigned char buf[32]; + lockdep_assert_held(&bsg_mutex); + if (!blk_get_queue(rq)) return ERR_PTR(-ENXIO); @@ -721,14 +709,12 @@ static struct bsg_device *bsg_add_device(struct inode *inode, bsg_set_block(bd, file); atomic_set(&bd->ref_count, 1); - mutex_lock(&bsg_mutex); hlist_add_head(&bd->dev_list, bsg_dev_idx_hash(iminor(inode))); strncpy(bd->name, dev_name(rq->bsg_dev.class_dev), sizeof(bd->name) - 1); bsg_dbg(bd, "bound to <%s>, max queue %d\n", format_dev_t(buf, inode->i_rdev), bd->max_queue); - mutex_unlock(&bsg_mutex); return bd; } @@ -736,7 +722,7 @@ static struct bsg_device *__bsg_get_device(int minor, struct request_queue *q) { struct bsg_device *bd; - mutex_lock(&bsg_mutex); + lockdep_assert_held(&bsg_mutex); hlist_for_each_entry(bd, bsg_dev_idx_hash(minor), dev_list) { if (bd->queue == q) { @@ -746,7 +732,6 @@ static struct bsg_device *__bsg_get_device(int minor, struct request_queue *q) } bd = NULL; found: - mutex_unlock(&bsg_mutex); return bd; } @@ -760,21 +745,18 @@ static struct bsg_device *bsg_get_device(struct inode *inode, struct file *file) */ mutex_lock(&bsg_mutex); bcd = idr_find(&bsg_minor_idr, iminor(inode)); - if (bcd) - kref_get(&bcd->ref); - mutex_unlock(&bsg_mutex); - if (!bcd) - return ERR_PTR(-ENODEV); + if (!bcd) { + bd = ERR_PTR(-ENODEV); + goto out_unlock; + } bd = __bsg_get_device(iminor(inode), bcd->queue); - if (bd) - return bd; - - bd = bsg_add_device(inode, bcd->queue, file); - if (IS_ERR(bd)) - kref_put(&bcd->ref, bsg_kref_release_function); + if (!bd) + bd = bsg_add_device(inode, bcd->queue, file); +out_unlock: + mutex_unlock(&bsg_mutex); return bd; } @@ -913,25 +895,17 @@ void bsg_unregister_queue(struct request_queue *q) sysfs_remove_link(&q->kobj, "bsg"); device_unregister(bcd->class_dev); bcd->class_dev = NULL; - kref_put(&bcd->ref, bsg_kref_release_function); mutex_unlock(&bsg_mutex); } EXPORT_SYMBOL_GPL(bsg_unregister_queue); int bsg_register_queue(struct request_queue *q, struct device *parent, - const char *name, const struct bsg_ops *ops, - void (*release)(struct device *)) + const char *name, const struct bsg_ops *ops) { struct bsg_class_device *bcd; dev_t dev; int ret; struct device *class_dev = NULL; - const char *devname; - - if (name) - devname = name; - else - devname = dev_name(parent); /* * we need a proper transport to send commands, not a stacked device @@ -955,15 +929,12 @@ int bsg_register_queue(struct request_queue *q, struct device *parent, bcd->minor = ret; bcd->queue = q; - bcd->parent = get_device(parent); - bcd->release = release; bcd->ops = ops; - kref_init(&bcd->ref); dev = MKDEV(bsg_major, bcd->minor); - class_dev = device_create(bsg_class, parent, dev, NULL, "%s", devname); + class_dev = device_create(bsg_class, parent, dev, NULL, "%s", name); if (IS_ERR(class_dev)) { ret = PTR_ERR(class_dev); - goto put_dev; + goto idr_remove; } bcd->class_dev = class_dev; @@ -978,8 +949,7 @@ int bsg_register_queue(struct request_queue *q, struct device *parent, unregister_class_dev: device_unregister(class_dev); -put_dev: - put_device(parent); +idr_remove: idr_remove(&bsg_minor_idr, bcd->minor); unlock: mutex_unlock(&bsg_mutex); @@ -993,7 +963,7 @@ int bsg_scsi_register_queue(struct request_queue *q, struct device *parent) return -EINVAL; } - return bsg_register_queue(q, parent, NULL, &bsg_scsi_ops, NULL); + return bsg_register_queue(q, parent, dev_name(parent), &bsg_scsi_ops); } EXPORT_SYMBOL_GPL(bsg_scsi_register_queue); diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c index 9f342ef1ad42..82b6c27b3245 100644 --- a/block/cfq-iosched.c +++ b/block/cfq-iosched.c @@ -210,9 +210,9 @@ struct cfqg_stats { /* total time with empty current active q with other requests queued */ struct blkg_stat empty_time; /* fields after this shouldn't be cleared on stat reset */ - uint64_t start_group_wait_time; - uint64_t start_idle_time; - uint64_t start_empty_time; + u64 start_group_wait_time; + u64 start_idle_time; + u64 start_empty_time; uint16_t flags; #endif /* CONFIG_DEBUG_BLK_CGROUP */ #endif /* CONFIG_CFQ_GROUP_IOSCHED */ @@ -491,13 +491,13 @@ CFQG_FLAG_FNS(empty) /* This should be called with the queue_lock held. */ static void cfqg_stats_update_group_wait_time(struct cfqg_stats *stats) { - unsigned long long now; + u64 now; if (!cfqg_stats_waiting(stats)) return; - now = sched_clock(); - if (time_after64(now, stats->start_group_wait_time)) + now = ktime_get_ns(); + if (now > stats->start_group_wait_time) blkg_stat_add(&stats->group_wait_time, now - stats->start_group_wait_time); cfqg_stats_clear_waiting(stats); @@ -513,20 +513,20 @@ static void cfqg_stats_set_start_group_wait_time(struct cfq_group *cfqg, return; if (cfqg == curr_cfqg) return; - stats->start_group_wait_time = sched_clock(); + stats->start_group_wait_time = ktime_get_ns(); cfqg_stats_mark_waiting(stats); } /* This should be called with the queue_lock held. */ static void cfqg_stats_end_empty_time(struct cfqg_stats *stats) { - unsigned long long now; + u64 now; if (!cfqg_stats_empty(stats)) return; - now = sched_clock(); - if (time_after64(now, stats->start_empty_time)) + now = ktime_get_ns(); + if (now > stats->start_empty_time) blkg_stat_add(&stats->empty_time, now - stats->start_empty_time); cfqg_stats_clear_empty(stats); @@ -552,7 +552,7 @@ static void cfqg_stats_set_start_empty_time(struct cfq_group *cfqg) if (cfqg_stats_empty(stats)) return; - stats->start_empty_time = sched_clock(); + stats->start_empty_time = ktime_get_ns(); cfqg_stats_mark_empty(stats); } @@ -561,9 +561,9 @@ static void cfqg_stats_update_idle_time(struct cfq_group *cfqg) struct cfqg_stats *stats = &cfqg->stats; if (cfqg_stats_idling(stats)) { - unsigned long long now = sched_clock(); + u64 now = ktime_get_ns(); - if (time_after64(now, stats->start_idle_time)) + if (now > stats->start_idle_time) blkg_stat_add(&stats->idle_time, now - stats->start_idle_time); cfqg_stats_clear_idling(stats); @@ -576,7 +576,7 @@ static void cfqg_stats_set_start_idle_time(struct cfq_group *cfqg) BUG_ON(cfqg_stats_idling(stats)); - stats->start_idle_time = sched_clock(); + stats->start_idle_time = ktime_get_ns(); cfqg_stats_mark_idling(stats); } @@ -701,17 +701,19 @@ static inline void cfqg_stats_update_io_merged(struct cfq_group *cfqg, } static inline void cfqg_stats_update_completion(struct cfq_group *cfqg, - uint64_t start_time, uint64_t io_start_time, - unsigned int op) + u64 start_time_ns, + u64 io_start_time_ns, + unsigned int op) { struct cfqg_stats *stats = &cfqg->stats; - unsigned long long now = sched_clock(); + u64 now = ktime_get_ns(); - if (time_after64(now, io_start_time)) - blkg_rwstat_add(&stats->service_time, op, now - io_start_time); - if (time_after64(io_start_time, start_time)) + if (now > io_start_time_ns) + blkg_rwstat_add(&stats->service_time, op, + now - io_start_time_ns); + if (io_start_time_ns > start_time_ns) blkg_rwstat_add(&stats->wait_time, op, - io_start_time - start_time); + io_start_time_ns - start_time_ns); } /* @stats = 0 */ @@ -797,8 +799,9 @@ static inline void cfqg_stats_update_io_remove(struct cfq_group *cfqg, static inline void cfqg_stats_update_io_merged(struct cfq_group *cfqg, unsigned int op) { } static inline void cfqg_stats_update_completion(struct cfq_group *cfqg, - uint64_t start_time, uint64_t io_start_time, - unsigned int op) { } + u64 start_time_ns, + u64 io_start_time_ns, + unsigned int op) { } #endif /* CONFIG_CFQ_GROUP_IOSCHED */ @@ -4225,8 +4228,8 @@ static void cfq_completed_request(struct request_queue *q, struct request *rq) cfqd->rq_in_driver--; cfqq->dispatched--; (RQ_CFQG(rq))->dispatched--; - cfqg_stats_update_completion(cfqq->cfqg, rq_start_time_ns(rq), - rq_io_start_time_ns(rq), rq->cmd_flags); + cfqg_stats_update_completion(cfqq->cfqg, rq->start_time_ns, + rq->io_start_time_ns, rq->cmd_flags); cfqd->rq_in_flight[cfq_cfqq_sync(cfqq)]--; @@ -4242,16 +4245,7 @@ static void cfq_completed_request(struct request_queue *q, struct request *rq) cfqq_type(cfqq)); st->ttime.last_end_request = now; - /* - * We have to do this check in jiffies since start_time is in - * jiffies and it is not trivial to convert to ns. If - * cfq_fifo_expire[1] ever comes close to 1 jiffie, this test - * will become problematic but so far we are fine (the default - * is 128 ms). - */ - if (!time_after(rq->start_time + - nsecs_to_jiffies(cfqd->cfq_fifo_expire[1]), - jiffies)) + if (rq->start_time_ns + cfqd->cfq_fifo_expire[1] <= now) cfqd->last_delayed_sync = now; } @@ -4792,7 +4786,7 @@ USEC_STORE_FUNCTION(cfq_target_latency_us_store, &cfqd->cfq_target_latency, 1, U #undef USEC_STORE_FUNCTION #define CFQ_ATTR(name) \ - __ATTR(name, S_IRUGO|S_IWUSR, cfq_##name##_show, cfq_##name##_store) + __ATTR(name, 0644, cfq_##name##_show, cfq_##name##_store) static struct elv_fs_entry cfq_attrs[] = { CFQ_ATTR(quantum), diff --git a/block/deadline-iosched.c b/block/deadline-iosched.c index 9de9f156e203..ef2f1f09e9b3 100644 --- a/block/deadline-iosched.c +++ b/block/deadline-iosched.c @@ -512,8 +512,7 @@ STORE_FUNCTION(deadline_fifo_batch_store, &dd->fifo_batch, 0, INT_MAX, 0); #undef STORE_FUNCTION #define DD_ATTR(name) \ - __ATTR(name, S_IRUGO|S_IWUSR, deadline_##name##_show, \ - deadline_##name##_store) + __ATTR(name, 0644, deadline_##name##_show, deadline_##name##_store) static struct elv_fs_entry deadline_attrs[] = { DD_ATTR(read_expire), diff --git a/block/elevator.c b/block/elevator.c index e87e9b43aba0..fa828b5bfd4b 100644 --- a/block/elevator.c +++ b/block/elevator.c @@ -199,76 +199,46 @@ static void elevator_release(struct kobject *kobj) kfree(e); } -int elevator_init(struct request_queue *q, char *name) +/* + * Use the default elevator specified by config boot param for non-mq devices, + * or by config option. Don't try to load modules as we could be running off + * async and request_module() isn't allowed from async. + */ +int elevator_init(struct request_queue *q) { struct elevator_type *e = NULL; - int err; + int err = 0; /* * q->sysfs_lock must be held to provide mutual exclusion between * elevator_switch() and here. */ - lockdep_assert_held(&q->sysfs_lock); - + mutex_lock(&q->sysfs_lock); if (unlikely(q->elevator)) - return 0; - - INIT_LIST_HEAD(&q->queue_head); - q->last_merge = NULL; - q->end_sector = 0; - q->boundary_rq = NULL; - - if (name) { - e = elevator_get(q, name, true); - if (!e) - return -EINVAL; - } + goto out_unlock; - /* - * Use the default elevator specified by config boot param for - * non-mq devices, or by config option. Don't try to load modules - * as we could be running off async and request_module() isn't - * allowed from async. - */ - if (!e && !q->mq_ops && *chosen_elevator) { + if (*chosen_elevator) { e = elevator_get(q, chosen_elevator, false); if (!e) printk(KERN_ERR "I/O scheduler %s not found\n", chosen_elevator); } + if (!e) + e = elevator_get(q, CONFIG_DEFAULT_IOSCHED, false); if (!e) { - /* - * For blk-mq devices, we default to using mq-deadline, - * if available, for single queue devices. If deadline - * isn't available OR we have multiple queues, default - * to "none". - */ - if (q->mq_ops) { - if (q->nr_hw_queues == 1) - e = elevator_get(q, "mq-deadline", false); - if (!e) - return 0; - } else - e = elevator_get(q, CONFIG_DEFAULT_IOSCHED, false); - - if (!e) { - printk(KERN_ERR - "Default I/O scheduler not found. " \ - "Using noop.\n"); - e = elevator_get(q, "noop", false); - } + printk(KERN_ERR + "Default I/O scheduler not found. Using noop.\n"); + e = elevator_get(q, "noop", false); } - if (e->uses_mq) - err = blk_mq_init_sched(q, e); - else - err = e->ops.sq.elevator_init_fn(q, e); + err = e->ops.sq.elevator_init_fn(q, e); if (err) elevator_put(e); +out_unlock: + mutex_unlock(&q->sysfs_lock); return err; } -EXPORT_SYMBOL(elevator_init); void elevator_exit(struct request_queue *q, struct elevator_queue *e) { @@ -281,7 +251,6 @@ void elevator_exit(struct request_queue *q, struct elevator_queue *e) kobject_put(&e->kobj); } -EXPORT_SYMBOL(elevator_exit); static inline void __elv_rqhash_del(struct request *rq) { @@ -1004,6 +973,40 @@ out: return ret; } +/* + * For blk-mq devices, we default to using mq-deadline, if available, for single + * queue devices. If deadline isn't available OR we have multiple queues, + * default to "none". + */ +int elevator_init_mq(struct request_queue *q) +{ + struct elevator_type *e; + int err = 0; + + if (q->nr_hw_queues != 1) + return 0; + + /* + * q->sysfs_lock must be held to provide mutual exclusion between + * elevator_switch() and here. + */ + mutex_lock(&q->sysfs_lock); + if (unlikely(q->elevator)) + goto out_unlock; + + e = elevator_get(q, "mq-deadline", false); + if (!e) + goto out_unlock; + + err = blk_mq_init_sched(q, e); + if (err) + elevator_put(e); +out_unlock: + mutex_unlock(&q->sysfs_lock); + return err; +} + + /* * switch to new_e io scheduler. be careful not to introduce deadlocks - * we don't free the old io scheduler, before we have allocated what we diff --git a/block/genhd.c b/block/genhd.c index c4513fe1adda..f1543a45e73b 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -1027,18 +1027,6 @@ static const struct seq_operations partitions_op = { .stop = disk_seqf_stop, .show = show_partition }; - -static int partitions_open(struct inode *inode, struct file *file) -{ - return seq_open(file, &partitions_op); -} - -static const struct file_operations proc_partitions_operations = { - .open = partitions_open, - .read = seq_read, - .llseek = seq_lseek, - .release = seq_release, -}; #endif @@ -1139,28 +1127,25 @@ static ssize_t disk_discard_alignment_show(struct device *dev, return sprintf(buf, "%d\n", queue_discard_alignment(disk->queue)); } -static DEVICE_ATTR(range, S_IRUGO, disk_range_show, NULL); -static DEVICE_ATTR(ext_range, S_IRUGO, disk_ext_range_show, NULL); -static DEVICE_ATTR(removable, S_IRUGO, disk_removable_show, NULL); -static DEVICE_ATTR(hidden, S_IRUGO, disk_hidden_show, NULL); -static DEVICE_ATTR(ro, S_IRUGO, disk_ro_show, NULL); -static DEVICE_ATTR(size, S_IRUGO, part_size_show, NULL); -static DEVICE_ATTR(alignment_offset, S_IRUGO, disk_alignment_offset_show, NULL); -static DEVICE_ATTR(discard_alignment, S_IRUGO, disk_discard_alignment_show, - NULL); -static DEVICE_ATTR(capability, S_IRUGO, disk_capability_show, NULL); -static DEVICE_ATTR(stat, S_IRUGO, part_stat_show, NULL); -static DEVICE_ATTR(inflight, S_IRUGO, part_inflight_show, NULL); -static DEVICE_ATTR(badblocks, S_IRUGO | S_IWUSR, disk_badblocks_show, - disk_badblocks_store); +static DEVICE_ATTR(range, 0444, disk_range_show, NULL); +static DEVICE_ATTR(ext_range, 0444, disk_ext_range_show, NULL); +static DEVICE_ATTR(removable, 0444, disk_removable_show, NULL); +static DEVICE_ATTR(hidden, 0444, disk_hidden_show, NULL); +static DEVICE_ATTR(ro, 0444, disk_ro_show, NULL); +static DEVICE_ATTR(size, 0444, part_size_show, NULL); +static DEVICE_ATTR(alignment_offset, 0444, disk_alignment_offset_show, NULL); +static DEVICE_ATTR(discard_alignment, 0444, disk_discard_alignment_show, NULL); +static DEVICE_ATTR(capability, 0444, disk_capability_show, NULL); +static DEVICE_ATTR(stat, 0444, part_stat_show, NULL); +static DEVICE_ATTR(inflight, 0444, part_inflight_show, NULL); +static DEVICE_ATTR(badblocks, 0644, disk_badblocks_show, disk_badblocks_store); #ifdef CONFIG_FAIL_MAKE_REQUEST static struct device_attribute dev_attr_fail = - __ATTR(make-it-fail, S_IRUGO|S_IWUSR, part_fail_show, part_fail_store); + __ATTR(make-it-fail, 0644, part_fail_show, part_fail_store); #endif #ifdef CONFIG_FAIL_IO_TIMEOUT static struct device_attribute dev_attr_fail_timeout = - __ATTR(io-timeout-fail, S_IRUGO|S_IWUSR, part_timeout_show, - part_timeout_store); + __ATTR(io-timeout-fail, 0644, part_timeout_show, part_timeout_store); #endif static struct attribute *disk_attrs[] = { @@ -1377,22 +1362,10 @@ static const struct seq_operations diskstats_op = { .show = diskstats_show }; -static int diskstats_open(struct inode *inode, struct file *file) -{ - return seq_open(file, &diskstats_op); -} - -static const struct file_operations proc_diskstats_operations = { - .open = diskstats_open, - .read = seq_read, - .llseek = seq_lseek, - .release = seq_release, -}; - static int __init proc_genhd_init(void) { - proc_create("diskstats", 0, NULL, &proc_diskstats_operations); - proc_create("partitions", 0, NULL, &proc_partitions_operations); + proc_create_seq("diskstats", 0, NULL, &diskstats_op); + proc_create_seq("partitions", 0, NULL, &partitions_op); return 0; } module_init(proc_genhd_init); @@ -1924,9 +1897,9 @@ static ssize_t disk_events_poll_msecs_store(struct device *dev, return count; } -static const DEVICE_ATTR(events, S_IRUGO, disk_events_show, NULL); -static const DEVICE_ATTR(events_async, S_IRUGO, disk_events_async_show, NULL); -static const DEVICE_ATTR(events_poll_msecs, S_IRUGO|S_IWUSR, +static const DEVICE_ATTR(events, 0444, disk_events_show, NULL); +static const DEVICE_ATTR(events_async, 0444, disk_events_async_show, NULL); +static const DEVICE_ATTR(events_poll_msecs, 0644, disk_events_poll_msecs_show, disk_events_poll_msecs_store); diff --git a/block/ioprio.c b/block/ioprio.c index 6f5d0b6625e3..f9821080c92c 100644 --- a/block/ioprio.c +++ b/block/ioprio.c @@ -61,15 +61,10 @@ int set_task_ioprio(struct task_struct *task, int ioprio) } EXPORT_SYMBOL_GPL(set_task_ioprio); -SYSCALL_DEFINE3(ioprio_set, int, which, int, who, int, ioprio) +int ioprio_check_cap(int ioprio) { int class = IOPRIO_PRIO_CLASS(ioprio); int data = IOPRIO_PRIO_DATA(ioprio); - struct task_struct *p, *g; - struct user_struct *user; - struct pid *pgrp; - kuid_t uid; - int ret; switch (class) { case IOPRIO_CLASS_RT: @@ -92,6 +87,21 @@ SYSCALL_DEFINE3(ioprio_set, int, which, int, who, int, ioprio) return -EINVAL; } + return 0; +} + +SYSCALL_DEFINE3(ioprio_set, int, which, int, who, int, ioprio) +{ + struct task_struct *p, *g; + struct user_struct *user; + struct pid *pgrp; + kuid_t uid; + int ret; + + ret = ioprio_check_cap(ioprio); + if (ret) + return ret; + ret = -ESRCH; rcu_read_lock(); switch (which) { diff --git a/block/kyber-iosched.c b/block/kyber-iosched.c index 0d6d25e32e1f..a1660bafc912 100644 --- a/block/kyber-iosched.c +++ b/block/kyber-iosched.c @@ -72,6 +72,19 @@ static const unsigned int kyber_batch_size[] = { [KYBER_OTHER] = 8, }; +/* + * There is a same mapping between ctx & hctx and kcq & khd, + * we use request->mq_ctx->index_hw to index the kcq in khd. + */ +struct kyber_ctx_queue { + /* + * Used to ensure operations on rq_list and kcq_map to be an atmoic one. + * Also protect the rqs on rq_list when merge. + */ + spinlock_t lock; + struct list_head rq_list[KYBER_NUM_DOMAINS]; +} ____cacheline_aligned_in_smp; + struct kyber_queue_data { struct request_queue *q; @@ -99,6 +112,8 @@ struct kyber_hctx_data { struct list_head rqs[KYBER_NUM_DOMAINS]; unsigned int cur_domain; unsigned int batching; + struct kyber_ctx_queue *kcqs; + struct sbitmap kcq_map[KYBER_NUM_DOMAINS]; wait_queue_entry_t domain_wait[KYBER_NUM_DOMAINS]; struct sbq_wait_state *domain_ws[KYBER_NUM_DOMAINS]; atomic_t wait_index[KYBER_NUM_DOMAINS]; @@ -107,10 +122,8 @@ struct kyber_hctx_data { static int kyber_domain_wake(wait_queue_entry_t *wait, unsigned mode, int flags, void *key); -static int rq_sched_domain(const struct request *rq) +static unsigned int kyber_sched_domain(unsigned int op) { - unsigned int op = rq->cmd_flags; - if ((op & REQ_OP_MASK) == REQ_OP_READ) return KYBER_READ; else if ((op & REQ_OP_MASK) == REQ_OP_WRITE && op_is_sync(op)) @@ -284,6 +297,11 @@ static unsigned int kyber_sched_tags_shift(struct kyber_queue_data *kqd) return kqd->q->queue_hw_ctx[0]->sched_tags->bitmap_tags.sb.shift; } +static int kyber_bucket_fn(const struct request *rq) +{ + return kyber_sched_domain(rq->cmd_flags); +} + static struct kyber_queue_data *kyber_queue_data_alloc(struct request_queue *q) { struct kyber_queue_data *kqd; @@ -297,7 +315,7 @@ static struct kyber_queue_data *kyber_queue_data_alloc(struct request_queue *q) goto err; kqd->q = q; - kqd->cb = blk_stat_alloc_callback(kyber_stat_timer_fn, rq_sched_domain, + kqd->cb = blk_stat_alloc_callback(kyber_stat_timer_fn, kyber_bucket_fn, KYBER_NUM_DOMAINS, kqd); if (!kqd->cb) goto err_kqd; @@ -376,8 +394,18 @@ static void kyber_exit_sched(struct elevator_queue *e) kfree(kqd); } +static void kyber_ctx_queue_init(struct kyber_ctx_queue *kcq) +{ + unsigned int i; + + spin_lock_init(&kcq->lock); + for (i = 0; i < KYBER_NUM_DOMAINS; i++) + INIT_LIST_HEAD(&kcq->rq_list[i]); +} + static int kyber_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) { + struct kyber_queue_data *kqd = hctx->queue->elevator->elevator_data; struct kyber_hctx_data *khd; int i; @@ -385,6 +413,24 @@ static int kyber_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) if (!khd) return -ENOMEM; + khd->kcqs = kmalloc_array_node(hctx->nr_ctx, + sizeof(struct kyber_ctx_queue), + GFP_KERNEL, hctx->numa_node); + if (!khd->kcqs) + goto err_khd; + + for (i = 0; i < hctx->nr_ctx; i++) + kyber_ctx_queue_init(&khd->kcqs[i]); + + for (i = 0; i < KYBER_NUM_DOMAINS; i++) { + if (sbitmap_init_node(&khd->kcq_map[i], hctx->nr_ctx, + ilog2(8), GFP_KERNEL, hctx->numa_node)) { + while (--i >= 0) + sbitmap_free(&khd->kcq_map[i]); + goto err_kcqs; + } + } + spin_lock_init(&khd->lock); for (i = 0; i < KYBER_NUM_DOMAINS; i++) { @@ -400,12 +446,26 @@ static int kyber_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) khd->batching = 0; hctx->sched_data = khd; + sbitmap_queue_min_shallow_depth(&hctx->sched_tags->bitmap_tags, + kqd->async_depth); return 0; + +err_kcqs: + kfree(khd->kcqs); +err_khd: + kfree(khd); + return -ENOMEM; } static void kyber_exit_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) { + struct kyber_hctx_data *khd = hctx->sched_data; + int i; + + for (i = 0; i < KYBER_NUM_DOMAINS; i++) + sbitmap_free(&khd->kcq_map[i]); + kfree(khd->kcqs); kfree(hctx->sched_data); } @@ -427,7 +487,7 @@ static void rq_clear_domain_token(struct kyber_queue_data *kqd, nr = rq_get_domain_token(rq); if (nr != -1) { - sched_domain = rq_sched_domain(rq); + sched_domain = kyber_sched_domain(rq->cmd_flags); sbitmap_queue_clear(&kqd->domain_tokens[sched_domain], nr, rq->mq_ctx->cpu); } @@ -446,11 +506,51 @@ static void kyber_limit_depth(unsigned int op, struct blk_mq_alloc_data *data) } } +static bool kyber_bio_merge(struct blk_mq_hw_ctx *hctx, struct bio *bio) +{ + struct kyber_hctx_data *khd = hctx->sched_data; + struct blk_mq_ctx *ctx = blk_mq_get_ctx(hctx->queue); + struct kyber_ctx_queue *kcq = &khd->kcqs[ctx->index_hw]; + unsigned int sched_domain = kyber_sched_domain(bio->bi_opf); + struct list_head *rq_list = &kcq->rq_list[sched_domain]; + bool merged; + + spin_lock(&kcq->lock); + merged = blk_mq_bio_list_merge(hctx->queue, rq_list, bio); + spin_unlock(&kcq->lock); + blk_mq_put_ctx(ctx); + + return merged; +} + static void kyber_prepare_request(struct request *rq, struct bio *bio) { rq_set_domain_token(rq, -1); } +static void kyber_insert_requests(struct blk_mq_hw_ctx *hctx, + struct list_head *rq_list, bool at_head) +{ + struct kyber_hctx_data *khd = hctx->sched_data; + struct request *rq, *next; + + list_for_each_entry_safe(rq, next, rq_list, queuelist) { + unsigned int sched_domain = kyber_sched_domain(rq->cmd_flags); + struct kyber_ctx_queue *kcq = &khd->kcqs[rq->mq_ctx->index_hw]; + struct list_head *head = &kcq->rq_list[sched_domain]; + + spin_lock(&kcq->lock); + if (at_head) + list_move(&rq->queuelist, head); + else + list_move_tail(&rq->queuelist, head); + sbitmap_set_bit(&khd->kcq_map[sched_domain], + rq->mq_ctx->index_hw); + blk_mq_sched_request_inserted(rq); + spin_unlock(&kcq->lock); + } +} + static void kyber_finish_request(struct request *rq) { struct kyber_queue_data *kqd = rq->q->elevator->elevator_data; @@ -469,7 +569,7 @@ static void kyber_completed_request(struct request *rq) * Check if this request met our latency goal. If not, quickly gather * some statistics and start throttling. */ - sched_domain = rq_sched_domain(rq); + sched_domain = kyber_sched_domain(rq->cmd_flags); switch (sched_domain) { case KYBER_READ: target = kqd->read_lat_nsec; @@ -485,29 +585,48 @@ static void kyber_completed_request(struct request *rq) if (blk_stat_is_active(kqd->cb)) return; - now = __blk_stat_time(ktime_to_ns(ktime_get())); - if (now < blk_stat_time(&rq->issue_stat)) + now = ktime_get_ns(); + if (now < rq->io_start_time_ns) return; - latency = now - blk_stat_time(&rq->issue_stat); + latency = now - rq->io_start_time_ns; if (latency > target) blk_stat_activate_msecs(kqd->cb, 10); } -static void kyber_flush_busy_ctxs(struct kyber_hctx_data *khd, - struct blk_mq_hw_ctx *hctx) +struct flush_kcq_data { + struct kyber_hctx_data *khd; + unsigned int sched_domain; + struct list_head *list; +}; + +static bool flush_busy_kcq(struct sbitmap *sb, unsigned int bitnr, void *data) { - LIST_HEAD(rq_list); - struct request *rq, *next; + struct flush_kcq_data *flush_data = data; + struct kyber_ctx_queue *kcq = &flush_data->khd->kcqs[bitnr]; - blk_mq_flush_busy_ctxs(hctx, &rq_list); - list_for_each_entry_safe(rq, next, &rq_list, queuelist) { - unsigned int sched_domain; + spin_lock(&kcq->lock); + list_splice_tail_init(&kcq->rq_list[flush_data->sched_domain], + flush_data->list); + sbitmap_clear_bit(sb, bitnr); + spin_unlock(&kcq->lock); - sched_domain = rq_sched_domain(rq); - list_move_tail(&rq->queuelist, &khd->rqs[sched_domain]); - } + return true; +} + +static void kyber_flush_busy_kcqs(struct kyber_hctx_data *khd, + unsigned int sched_domain, + struct list_head *list) +{ + struct flush_kcq_data data = { + .khd = khd, + .sched_domain = sched_domain, + .list = list, + }; + + sbitmap_for_each_set(&khd->kcq_map[sched_domain], + flush_busy_kcq, &data); } static int kyber_domain_wake(wait_queue_entry_t *wait, unsigned mode, int flags, @@ -570,26 +689,23 @@ static int kyber_get_domain_token(struct kyber_queue_data *kqd, static struct request * kyber_dispatch_cur_domain(struct kyber_queue_data *kqd, struct kyber_hctx_data *khd, - struct blk_mq_hw_ctx *hctx, - bool *flushed) + struct blk_mq_hw_ctx *hctx) { struct list_head *rqs; struct request *rq; int nr; rqs = &khd->rqs[khd->cur_domain]; - rq = list_first_entry_or_null(rqs, struct request, queuelist); /* - * If there wasn't already a pending request and we haven't flushed the - * software queues yet, flush the software queues and check again. + * If we already have a flushed request, then we just need to get a + * token for it. Otherwise, if there are pending requests in the kcqs, + * flush the kcqs, but only if we can get a token. If not, we should + * leave the requests in the kcqs so that they can be merged. Note that + * khd->lock serializes the flushes, so if we observed any bit set in + * the kcq_map, we will always get a request. */ - if (!rq && !*flushed) { - kyber_flush_busy_ctxs(khd, hctx); - *flushed = true; - rq = list_first_entry_or_null(rqs, struct request, queuelist); - } - + rq = list_first_entry_or_null(rqs, struct request, queuelist); if (rq) { nr = kyber_get_domain_token(kqd, khd, hctx); if (nr >= 0) { @@ -598,6 +714,16 @@ kyber_dispatch_cur_domain(struct kyber_queue_data *kqd, list_del_init(&rq->queuelist); return rq; } + } else if (sbitmap_any_bit_set(&khd->kcq_map[khd->cur_domain])) { + nr = kyber_get_domain_token(kqd, khd, hctx); + if (nr >= 0) { + kyber_flush_busy_kcqs(khd, khd->cur_domain, rqs); + rq = list_first_entry(rqs, struct request, queuelist); + khd->batching++; + rq_set_domain_token(rq, nr); + list_del_init(&rq->queuelist); + return rq; + } } /* There were either no pending requests or no tokens. */ @@ -608,7 +734,6 @@ static struct request *kyber_dispatch_request(struct blk_mq_hw_ctx *hctx) { struct kyber_queue_data *kqd = hctx->queue->elevator->elevator_data; struct kyber_hctx_data *khd = hctx->sched_data; - bool flushed = false; struct request *rq; int i; @@ -619,7 +744,7 @@ static struct request *kyber_dispatch_request(struct blk_mq_hw_ctx *hctx) * from the batch. */ if (khd->batching < kyber_batch_size[khd->cur_domain]) { - rq = kyber_dispatch_cur_domain(kqd, khd, hctx, &flushed); + rq = kyber_dispatch_cur_domain(kqd, khd, hctx); if (rq) goto out; } @@ -640,7 +765,7 @@ static struct request *kyber_dispatch_request(struct blk_mq_hw_ctx *hctx) else khd->cur_domain++; - rq = kyber_dispatch_cur_domain(kqd, khd, hctx, &flushed); + rq = kyber_dispatch_cur_domain(kqd, khd, hctx); if (rq) goto out; } @@ -657,10 +782,12 @@ static bool kyber_has_work(struct blk_mq_hw_ctx *hctx) int i; for (i = 0; i < KYBER_NUM_DOMAINS; i++) { - if (!list_empty_careful(&khd->rqs[i])) + if (!list_empty_careful(&khd->rqs[i]) || + sbitmap_any_bit_set(&khd->kcq_map[i])) return true; } - return sbitmap_any_bit_set(&hctx->ctx_map); + + return false; } #define KYBER_LAT_SHOW_STORE(op) \ @@ -831,7 +958,9 @@ static struct elevator_type kyber_sched = { .init_hctx = kyber_init_hctx, .exit_hctx = kyber_exit_hctx, .limit_depth = kyber_limit_depth, + .bio_merge = kyber_bio_merge, .prepare_request = kyber_prepare_request, + .insert_requests = kyber_insert_requests, .finish_request = kyber_finish_request, .requeue_request = kyber_finish_request, .completed_request = kyber_completed_request, diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 8ec0ba9f5386..099a9e05854c 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -630,8 +630,7 @@ STORE_FUNCTION(deadline_fifo_batch_store, &dd->fifo_batch, 0, INT_MAX, 0); #undef STORE_FUNCTION #define DD_ATTR(name) \ - __ATTR(name, S_IRUGO|S_IWUSR, deadline_##name##_show, \ - deadline_##name##_store) + __ATTR(name, 0644, deadline_##name##_show, deadline_##name##_store) static struct elv_fs_entry deadline_attrs[] = { DD_ATTR(read_expire), diff --git a/block/partition-generic.c b/block/partition-generic.c index db57cced9b98..3dcfd4ec0e11 100644 --- a/block/partition-generic.c +++ b/block/partition-generic.c @@ -179,18 +179,17 @@ ssize_t part_fail_store(struct device *dev, } #endif -static DEVICE_ATTR(partition, S_IRUGO, part_partition_show, NULL); -static DEVICE_ATTR(start, S_IRUGO, part_start_show, NULL); -static DEVICE_ATTR(size, S_IRUGO, part_size_show, NULL); -static DEVICE_ATTR(ro, S_IRUGO, part_ro_show, NULL); -static DEVICE_ATTR(alignment_offset, S_IRUGO, part_alignment_offset_show, NULL); -static DEVICE_ATTR(discard_alignment, S_IRUGO, part_discard_alignment_show, - NULL); -static DEVICE_ATTR(stat, S_IRUGO, part_stat_show, NULL); -static DEVICE_ATTR(inflight, S_IRUGO, part_inflight_show, NULL); +static DEVICE_ATTR(partition, 0444, part_partition_show, NULL); +static DEVICE_ATTR(start, 0444, part_start_show, NULL); +static DEVICE_ATTR(size, 0444, part_size_show, NULL); +static DEVICE_ATTR(ro, 0444, part_ro_show, NULL); +static DEVICE_ATTR(alignment_offset, 0444, part_alignment_offset_show, NULL); +static DEVICE_ATTR(discard_alignment, 0444, part_discard_alignment_show, NULL); +static DEVICE_ATTR(stat, 0444, part_stat_show, NULL); +static DEVICE_ATTR(inflight, 0444, part_inflight_show, NULL); #ifdef CONFIG_FAIL_MAKE_REQUEST static struct device_attribute dev_attr_fail = - __ATTR(make-it-fail, S_IRUGO|S_IWUSR, part_fail_show, part_fail_store); + __ATTR(make-it-fail, 0644, part_fail_show, part_fail_store); #endif static struct attribute *part_attrs[] = { @@ -291,8 +290,7 @@ static ssize_t whole_disk_show(struct device *dev, { return 0; } -static DEVICE_ATTR(whole_disk, S_IRUSR | S_IRGRP | S_IROTH, - whole_disk_show, NULL); +static DEVICE_ATTR(whole_disk, 0444, whole_disk_show, NULL); /* * Must be called either with bd_mutex held, before a disk can be opened or @@ -518,7 +516,7 @@ rescan: if (disk->fops->revalidate_disk) disk->fops->revalidate_disk(disk); - check_disk_size_change(disk, bdev); + check_disk_size_change(disk, bdev, true); bdev->bd_invalidated = 0; if (!get_capacity(disk) || !(state = check_partition(disk, bdev))) return 0; @@ -643,7 +641,7 @@ int invalidate_partitions(struct gendisk *disk, struct block_device *bdev) return res; set_capacity(disk, 0); - check_disk_size_change(disk, bdev); + check_disk_size_change(disk, bdev, false); bdev->bd_invalidated = 0; /* tell userspace that the media / partition table may have changed */ kobject_uevent(&disk_to_dev(disk)->kobj, KOBJ_CHANGE); diff --git a/block/partitions/check.c b/block/partitions/check.c index 720145c49066..ffe408fead0c 100644 --- a/block/partitions/check.c +++ b/block/partitions/check.c @@ -122,7 +122,7 @@ static struct parsed_partitions *allocate_partitions(struct gendisk *hd) return NULL; nr = disk_max_parts(hd); - state->parts = vzalloc(nr * sizeof(state->parts[0])); + state->parts = vzalloc(array_size(nr, sizeof(state->parts[0]))); if (!state->parts) { kfree(state); return NULL; diff --git a/block/partitions/cmdline.c b/block/partitions/cmdline.c index e333583ab38c..60fb3df9897c 100644 --- a/block/partitions/cmdline.c +++ b/block/partitions/cmdline.c @@ -58,6 +58,62 @@ static int __init cmdline_parts_setup(char *s) } __setup("blkdevparts=", cmdline_parts_setup); +static bool has_overlaps(sector_t from, sector_t size, + sector_t from2, sector_t size2) +{ + sector_t end = from + size; + sector_t end2 = from2 + size2; + + if (from >= from2 && from < end2) + return true; + + if (end > from2 && end <= end2) + return true; + + if (from2 >= from && from2 < end) + return true; + + if (end2 > from && end2 <= end) + return true; + + return false; +} + +static inline void overlaps_warns_header(void) +{ + pr_warn("Overlapping partitions are used in command line partitions."); + pr_warn("Don't use filesystems on overlapping partitions:"); +} + +static void cmdline_parts_verifier(int slot, struct parsed_partitions *state) +{ + int i; + bool header = true; + + for (; slot < state->limit && state->parts[slot].has_info; slot++) { + for (i = slot+1; i < state->limit && state->parts[i].has_info; + i++) { + if (has_overlaps(state->parts[slot].from, + state->parts[slot].size, + state->parts[i].from, + state->parts[i].size)) { + if (header) { + header = false; + overlaps_warns_header(); + } + pr_warn("%s[%llu,%llu] overlaps with " + "%s[%llu,%llu].", + state->parts[slot].info.volname, + (u64)state->parts[slot].from << 9, + (u64)state->parts[slot].size << 9, + state->parts[i].info.volname, + (u64)state->parts[i].from << 9, + (u64)state->parts[i].size << 9); + } + } + } +} + /* * Purpose: allocate cmdline partitions. * Returns: @@ -93,6 +149,7 @@ int cmdline_partition(struct parsed_partitions *state) disk_size = get_capacity(state->bdev->bd_disk) << 9; cmdline_parts_set(parts, disk_size, 1, add_part, (void *)state); + cmdline_parts_verifier(1, state); strlcat(state->pp_buf, "\n", PAGE_SIZE); diff --git a/block/partitions/ldm.c b/block/partitions/ldm.c index 2a365c756648..0417937dfe99 100644 --- a/block/partitions/ldm.c +++ b/block/partitions/ldm.c @@ -378,7 +378,7 @@ static bool ldm_validate_tocblocks(struct parsed_partitions *state, BUG_ON(!state || !ldb); ph = &ldb->ph; tb[0] = &ldb->toc; - tb[1] = kmalloc(sizeof(*tb[1]) * 3, GFP_KERNEL); + tb[1] = kmalloc_array(3, sizeof(*tb[1]), GFP_KERNEL); if (!tb[1]) { ldm_crit("Out of memory."); goto err; diff --git a/block/scsi_ioctl.c b/block/scsi_ioctl.c index 60b471f8621b..533f4aee8567 100644 --- a/block/scsi_ioctl.c +++ b/block/scsi_ioctl.c @@ -321,8 +321,7 @@ static int sg_io(struct request_queue *q, struct gendisk *bd_disk, at_head = 1; ret = -ENOMEM; - rq = blk_get_request(q, writing ? REQ_OP_SCSI_OUT : REQ_OP_SCSI_IN, - GFP_KERNEL); + rq = blk_get_request(q, writing ? REQ_OP_SCSI_OUT : REQ_OP_SCSI_IN, 0); if (IS_ERR(rq)) return PTR_ERR(rq); req = scsi_req(rq); @@ -449,8 +448,7 @@ int sg_scsi_ioctl(struct request_queue *q, struct gendisk *disk, fmode_t mode, } - rq = blk_get_request(q, in_len ? REQ_OP_SCSI_OUT : REQ_OP_SCSI_IN, - __GFP_RECLAIM); + rq = blk_get_request(q, in_len ? REQ_OP_SCSI_OUT : REQ_OP_SCSI_IN, 0); if (IS_ERR(rq)) { err = PTR_ERR(rq); goto error_free_buffer; @@ -501,7 +499,7 @@ int sg_scsi_ioctl(struct request_queue *q, struct gendisk *disk, fmode_t mode, break; } - if (bytes && blk_rq_map_kern(q, rq, buffer, bytes, __GFP_RECLAIM)) { + if (bytes && blk_rq_map_kern(q, rq, buffer, bytes, GFP_NOIO)) { err = DRIVER_ERROR << 24; goto error; } @@ -538,7 +536,7 @@ static int __blk_send_generic(struct request_queue *q, struct gendisk *bd_disk, struct request *rq; int err; - rq = blk_get_request(q, REQ_OP_SCSI_OUT, __GFP_RECLAIM); + rq = blk_get_request(q, REQ_OP_SCSI_OUT, 0); if (IS_ERR(rq)) return PTR_ERR(rq); rq->timeout = BLK_DEFAULT_SG_TIMEOUT; diff --git a/certs/Kconfig b/certs/Kconfig index 5f7663df6e8e..c94e93d8bccf 100644 --- a/certs/Kconfig +++ b/certs/Kconfig @@ -13,7 +13,7 @@ config MODULE_SIG_KEY If this option is unchanged from its default "certs/signing_key.pem", then the kernel will automatically generate the private key and - certificate as described in Documentation/module-signing.txt + certificate as described in Documentation/admin-guide/module-signing.rst config SYSTEM_TRUSTED_KEYRING bool "Provide system-wide ring of trusted keys" diff --git a/crypto/Kconfig b/crypto/Kconfig index 76e8c88c97b4..f3e40ac56d93 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -289,6 +289,107 @@ config CRYPTO_CHACHA20POLY1305 with the Poly1305 authenticator. It is defined in RFC7539 for use in IETF protocols. +config CRYPTO_AEGIS128 + tristate "AEGIS-128 AEAD algorithm" + select CRYPTO_AEAD + select CRYPTO_AES # for AES S-box tables + help + Support for the AEGIS-128 dedicated AEAD algorithm. + +config CRYPTO_AEGIS128L + tristate "AEGIS-128L AEAD algorithm" + select CRYPTO_AEAD + select CRYPTO_AES # for AES S-box tables + help + Support for the AEGIS-128L dedicated AEAD algorithm. + +config CRYPTO_AEGIS256 + tristate "AEGIS-256 AEAD algorithm" + select CRYPTO_AEAD + select CRYPTO_AES # for AES S-box tables + help + Support for the AEGIS-256 dedicated AEAD algorithm. + +config CRYPTO_AEGIS128_AESNI_SSE2 + tristate "AEGIS-128 AEAD algorithm (x86_64 AESNI+SSE2 implementation)" + depends on X86 && 64BIT + select CRYPTO_AEAD + select CRYPTO_CRYPTD + help + AESNI+SSE2 implementation of the AEGSI-128 dedicated AEAD algorithm. + +config CRYPTO_AEGIS128L_AESNI_SSE2 + tristate "AEGIS-128L AEAD algorithm (x86_64 AESNI+SSE2 implementation)" + depends on X86 && 64BIT + select CRYPTO_AEAD + select CRYPTO_CRYPTD + help + AESNI+SSE2 implementation of the AEGSI-128L dedicated AEAD algorithm. + +config CRYPTO_AEGIS256_AESNI_SSE2 + tristate "AEGIS-256 AEAD algorithm (x86_64 AESNI+SSE2 implementation)" + depends on X86 && 64BIT + select CRYPTO_AEAD + select CRYPTO_CRYPTD + help + AESNI+SSE2 implementation of the AEGSI-256 dedicated AEAD algorithm. + +config CRYPTO_MORUS640 + tristate "MORUS-640 AEAD algorithm" + select CRYPTO_AEAD + help + Support for the MORUS-640 dedicated AEAD algorithm. + +config CRYPTO_MORUS640_GLUE + tristate + depends on X86 + select CRYPTO_AEAD + select CRYPTO_CRYPTD + help + Common glue for SIMD optimizations of the MORUS-640 dedicated AEAD + algorithm. + +config CRYPTO_MORUS640_SSE2 + tristate "MORUS-640 AEAD algorithm (x86_64 SSE2 implementation)" + depends on X86 && 64BIT + select CRYPTO_AEAD + select CRYPTO_MORUS640_GLUE + help + SSE2 implementation of the MORUS-640 dedicated AEAD algorithm. + +config CRYPTO_MORUS1280 + tristate "MORUS-1280 AEAD algorithm" + select CRYPTO_AEAD + help + Support for the MORUS-1280 dedicated AEAD algorithm. + +config CRYPTO_MORUS1280_GLUE + tristate + depends on X86 + select CRYPTO_AEAD + select CRYPTO_CRYPTD + help + Common glue for SIMD optimizations of the MORUS-1280 dedicated AEAD + algorithm. + +config CRYPTO_MORUS1280_SSE2 + tristate "MORUS-1280 AEAD algorithm (x86_64 SSE2 implementation)" + depends on X86 && 64BIT + select CRYPTO_AEAD + select CRYPTO_MORUS1280_GLUE + help + SSE2 optimizedimplementation of the MORUS-1280 dedicated AEAD + algorithm. + +config CRYPTO_MORUS1280_AVX2 + tristate "MORUS-1280 AEAD algorithm (x86_64 AVX2 implementation)" + depends on X86 && 64BIT + select CRYPTO_AEAD + select CRYPTO_MORUS1280_GLUE + help + AVX2 optimized implementation of the MORUS-1280 dedicated AEAD + algorithm. + config CRYPTO_SEQIV tristate "Sequence Number IV Generator" select CRYPTO_AEAD @@ -1335,34 +1436,6 @@ config CRYPTO_SALSA20 The Salsa20 stream cipher algorithm is designed by Daniel J. Bernstein . See -config CRYPTO_SALSA20_586 - tristate "Salsa20 stream cipher algorithm (i586)" - depends on (X86 || UML_X86) && !64BIT - select CRYPTO_BLKCIPHER - select CRYPTO_SALSA20 - help - Salsa20 stream cipher algorithm. - - Salsa20 is a stream cipher submitted to eSTREAM, the ECRYPT - Stream Cipher Project. See - - The Salsa20 stream cipher algorithm is designed by Daniel J. - Bernstein . See - -config CRYPTO_SALSA20_X86_64 - tristate "Salsa20 stream cipher algorithm (x86_64)" - depends on (X86 || UML_X86) && 64BIT - select CRYPTO_BLKCIPHER - select CRYPTO_SALSA20 - help - Salsa20 stream cipher algorithm. - - Salsa20 is a stream cipher submitted to eSTREAM, the ECRYPT - Stream Cipher Project. See - - The Salsa20 stream cipher algorithm is designed by Daniel J. - Bernstein . See - config CRYPTO_CHACHA20 tristate "ChaCha20 cipher algorithm" select CRYPTO_BLKCIPHER @@ -1695,6 +1768,15 @@ config CRYPTO_LZ4HC help This is the LZ4 high compression mode algorithm. +config CRYPTO_ZSTD + tristate "Zstd compression algorithm" + select CRYPTO_ALGAPI + select CRYPTO_ACOMP2 + select ZSTD_COMPRESS + select ZSTD_DECOMPRESS + help + This is the zstd algorithm. + comment "Random Number Generation" config CRYPTO_ANSI_CPRNG diff --git a/crypto/Makefile b/crypto/Makefile index 3a5f01616f74..6d1d40eeb964 100644 --- a/crypto/Makefile +++ b/crypto/Makefile @@ -86,6 +86,11 @@ obj-$(CONFIG_CRYPTO_KEYWRAP) += keywrap.o obj-$(CONFIG_CRYPTO_GCM) += gcm.o obj-$(CONFIG_CRYPTO_CCM) += ccm.o obj-$(CONFIG_CRYPTO_CHACHA20POLY1305) += chacha20poly1305.o +obj-$(CONFIG_CRYPTO_AEGIS128) += aegis128.o +obj-$(CONFIG_CRYPTO_AEGIS128L) += aegis128l.o +obj-$(CONFIG_CRYPTO_AEGIS256) += aegis256.o +obj-$(CONFIG_CRYPTO_MORUS640) += morus640.o +obj-$(CONFIG_CRYPTO_MORUS1280) += morus1280.o obj-$(CONFIG_CRYPTO_PCRYPT) += pcrypt.o obj-$(CONFIG_CRYPTO_CRYPTD) += cryptd.o obj-$(CONFIG_CRYPTO_MCRYPTD) += mcryptd.o @@ -137,6 +142,7 @@ obj-$(CONFIG_CRYPTO_USER_API_HASH) += algif_hash.o obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.o obj-$(CONFIG_CRYPTO_USER_API_RNG) += algif_rng.o obj-$(CONFIG_CRYPTO_USER_API_AEAD) += algif_aead.o +obj-$(CONFIG_CRYPTO_ZSTD) += zstd.o ecdh_generic-y := ecc.o ecdh_generic-y += ecdh.o diff --git a/crypto/aegis.h b/crypto/aegis.h new file mode 100644 index 000000000000..f1c6900ddb80 --- /dev/null +++ b/crypto/aegis.h @@ -0,0 +1,80 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * AEGIS common definitions + * + * Copyright (c) 2018 Ondrej Mosnacek + * Copyright (c) 2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#ifndef _CRYPTO_AEGIS_H +#define _CRYPTO_AEGIS_H + +#include +#include + +#define AEGIS_BLOCK_SIZE 16 + +union aegis_block { + __le64 words64[AEGIS_BLOCK_SIZE / sizeof(__le64)]; + u32 words32[AEGIS_BLOCK_SIZE / sizeof(u32)]; + u8 bytes[AEGIS_BLOCK_SIZE]; +}; + +#define AEGIS_BLOCK_ALIGN (__alignof__(union aegis_block)) +#define AEGIS_ALIGNED(p) IS_ALIGNED((uintptr_t)p, AEGIS_BLOCK_ALIGN) + +static const union aegis_block crypto_aegis_const[2] = { + { .words64 = { + cpu_to_le64(U64_C(0x0d08050302010100)), + cpu_to_le64(U64_C(0x6279e99059372215)), + } }, + { .words64 = { + cpu_to_le64(U64_C(0xf12fc26d55183ddb)), + cpu_to_le64(U64_C(0xdd28b57342311120)), + } }, +}; + +static void crypto_aegis_block_xor(union aegis_block *dst, + const union aegis_block *src) +{ + dst->words64[0] ^= src->words64[0]; + dst->words64[1] ^= src->words64[1]; +} + +static void crypto_aegis_block_and(union aegis_block *dst, + const union aegis_block *src) +{ + dst->words64[0] &= src->words64[0]; + dst->words64[1] &= src->words64[1]; +} + +static void crypto_aegis_aesenc(union aegis_block *dst, + const union aegis_block *src, + const union aegis_block *key) +{ + u32 *d = dst->words32; + const u8 *s = src->bytes; + const u32 *k = key->words32; + const u32 *t0 = crypto_ft_tab[0]; + const u32 *t1 = crypto_ft_tab[1]; + const u32 *t2 = crypto_ft_tab[2]; + const u32 *t3 = crypto_ft_tab[3]; + u32 d0, d1, d2, d3; + + d0 = t0[s[ 0]] ^ t1[s[ 5]] ^ t2[s[10]] ^ t3[s[15]] ^ k[0]; + d1 = t0[s[ 4]] ^ t1[s[ 9]] ^ t2[s[14]] ^ t3[s[ 3]] ^ k[1]; + d2 = t0[s[ 8]] ^ t1[s[13]] ^ t2[s[ 2]] ^ t3[s[ 7]] ^ k[2]; + d3 = t0[s[12]] ^ t1[s[ 1]] ^ t2[s[ 6]] ^ t3[s[11]] ^ k[3]; + + d[0] = d0; + d[1] = d1; + d[2] = d2; + d[3] = d3; +} + +#endif /* _CRYPTO_AEGIS_H */ diff --git a/crypto/aegis128.c b/crypto/aegis128.c new file mode 100644 index 000000000000..38271303ce16 --- /dev/null +++ b/crypto/aegis128.c @@ -0,0 +1,463 @@ +/* + * The AEGIS-128 Authenticated-Encryption Algorithm + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "aegis.h" + +#define AEGIS128_NONCE_SIZE 16 +#define AEGIS128_STATE_BLOCKS 5 +#define AEGIS128_KEY_SIZE 16 +#define AEGIS128_MIN_AUTH_SIZE 8 +#define AEGIS128_MAX_AUTH_SIZE 16 + +struct aegis_state { + union aegis_block blocks[AEGIS128_STATE_BLOCKS]; +}; + +struct aegis_ctx { + union aegis_block key; +}; + +struct aegis128_ops { + int (*skcipher_walk_init)(struct skcipher_walk *walk, + struct aead_request *req, bool atomic); + + void (*crypt_chunk)(struct aegis_state *state, u8 *dst, + const u8 *src, unsigned int size); +}; + +static void crypto_aegis128_update(struct aegis_state *state) +{ + union aegis_block tmp; + unsigned int i; + + tmp = state->blocks[AEGIS128_STATE_BLOCKS - 1]; + for (i = AEGIS128_STATE_BLOCKS - 1; i > 0; i--) + crypto_aegis_aesenc(&state->blocks[i], &state->blocks[i - 1], + &state->blocks[i]); + crypto_aegis_aesenc(&state->blocks[0], &tmp, &state->blocks[0]); +} + +static void crypto_aegis128_update_a(struct aegis_state *state, + const union aegis_block *msg) +{ + crypto_aegis128_update(state); + crypto_aegis_block_xor(&state->blocks[0], msg); +} + +static void crypto_aegis128_update_u(struct aegis_state *state, const void *msg) +{ + crypto_aegis128_update(state); + crypto_xor(state->blocks[0].bytes, msg, AEGIS_BLOCK_SIZE); +} + +static void crypto_aegis128_init(struct aegis_state *state, + const union aegis_block *key, + const u8 *iv) +{ + union aegis_block key_iv; + unsigned int i; + + key_iv = *key; + crypto_xor(key_iv.bytes, iv, AEGIS_BLOCK_SIZE); + + state->blocks[0] = key_iv; + state->blocks[1] = crypto_aegis_const[1]; + state->blocks[2] = crypto_aegis_const[0]; + state->blocks[3] = *key; + state->blocks[4] = *key; + + crypto_aegis_block_xor(&state->blocks[3], &crypto_aegis_const[0]); + crypto_aegis_block_xor(&state->blocks[4], &crypto_aegis_const[1]); + + for (i = 0; i < 5; i++) { + crypto_aegis128_update_a(state, key); + crypto_aegis128_update_a(state, &key_iv); + } +} + +static void crypto_aegis128_ad(struct aegis_state *state, + const u8 *src, unsigned int size) +{ + if (AEGIS_ALIGNED(src)) { + const union aegis_block *src_blk = + (const union aegis_block *)src; + + while (size >= AEGIS_BLOCK_SIZE) { + crypto_aegis128_update_a(state, src_blk); + + size -= AEGIS_BLOCK_SIZE; + src_blk++; + } + } else { + while (size >= AEGIS_BLOCK_SIZE) { + crypto_aegis128_update_u(state, src); + + size -= AEGIS_BLOCK_SIZE; + src += AEGIS_BLOCK_SIZE; + } + } +} + +static void crypto_aegis128_encrypt_chunk(struct aegis_state *state, u8 *dst, + const u8 *src, unsigned int size) +{ + union aegis_block tmp; + + if (AEGIS_ALIGNED(src) && AEGIS_ALIGNED(dst)) { + while (size >= AEGIS_BLOCK_SIZE) { + union aegis_block *dst_blk = + (union aegis_block *)dst; + const union aegis_block *src_blk = + (const union aegis_block *)src; + + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + crypto_aegis_block_xor(&tmp, src_blk); + + crypto_aegis128_update_a(state, src_blk); + + *dst_blk = tmp; + + size -= AEGIS_BLOCK_SIZE; + src += AEGIS_BLOCK_SIZE; + dst += AEGIS_BLOCK_SIZE; + } + } else { + while (size >= AEGIS_BLOCK_SIZE) { + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + crypto_xor(tmp.bytes, src, AEGIS_BLOCK_SIZE); + + crypto_aegis128_update_u(state, src); + + memcpy(dst, tmp.bytes, AEGIS_BLOCK_SIZE); + + size -= AEGIS_BLOCK_SIZE; + src += AEGIS_BLOCK_SIZE; + dst += AEGIS_BLOCK_SIZE; + } + } + + if (size > 0) { + union aegis_block msg = {}; + memcpy(msg.bytes, src, size); + + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + + crypto_aegis128_update_a(state, &msg); + + crypto_aegis_block_xor(&msg, &tmp); + + memcpy(dst, msg.bytes, size); + } +} + +static void crypto_aegis128_decrypt_chunk(struct aegis_state *state, u8 *dst, + const u8 *src, unsigned int size) +{ + union aegis_block tmp; + + if (AEGIS_ALIGNED(src) && AEGIS_ALIGNED(dst)) { + while (size >= AEGIS_BLOCK_SIZE) { + union aegis_block *dst_blk = + (union aegis_block *)dst; + const union aegis_block *src_blk = + (const union aegis_block *)src; + + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + crypto_aegis_block_xor(&tmp, src_blk); + + crypto_aegis128_update_a(state, &tmp); + + *dst_blk = tmp; + + size -= AEGIS_BLOCK_SIZE; + src += AEGIS_BLOCK_SIZE; + dst += AEGIS_BLOCK_SIZE; + } + } else { + while (size >= AEGIS_BLOCK_SIZE) { + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + crypto_xor(tmp.bytes, src, AEGIS_BLOCK_SIZE); + + crypto_aegis128_update_a(state, &tmp); + + memcpy(dst, tmp.bytes, AEGIS_BLOCK_SIZE); + + size -= AEGIS_BLOCK_SIZE; + src += AEGIS_BLOCK_SIZE; + dst += AEGIS_BLOCK_SIZE; + } + } + + if (size > 0) { + union aegis_block msg = {}; + memcpy(msg.bytes, src, size); + + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + crypto_aegis_block_xor(&msg, &tmp); + + memset(msg.bytes + size, 0, AEGIS_BLOCK_SIZE - size); + + crypto_aegis128_update_a(state, &msg); + + memcpy(dst, msg.bytes, size); + } +} + +static void crypto_aegis128_process_ad(struct aegis_state *state, + struct scatterlist *sg_src, + unsigned int assoclen) +{ + struct scatter_walk walk; + union aegis_block buf; + unsigned int pos = 0; + + scatterwalk_start(&walk, sg_src); + while (assoclen != 0) { + unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int left = size; + void *mapped = scatterwalk_map(&walk); + const u8 *src = (const u8 *)mapped; + + if (pos + size >= AEGIS_BLOCK_SIZE) { + if (pos > 0) { + unsigned int fill = AEGIS_BLOCK_SIZE - pos; + memcpy(buf.bytes + pos, src, fill); + crypto_aegis128_update_a(state, &buf); + pos = 0; + left -= fill; + src += fill; + } + + crypto_aegis128_ad(state, src, left); + src += left & ~(AEGIS_BLOCK_SIZE - 1); + left &= AEGIS_BLOCK_SIZE - 1; + } + + memcpy(buf.bytes + pos, src, left); + + pos += left; + assoclen -= size; + scatterwalk_unmap(mapped); + scatterwalk_advance(&walk, size); + scatterwalk_done(&walk, 0, assoclen); + } + + if (pos > 0) { + memset(buf.bytes + pos, 0, AEGIS_BLOCK_SIZE - pos); + crypto_aegis128_update_a(state, &buf); + } +} + +static void crypto_aegis128_process_crypt(struct aegis_state *state, + struct aead_request *req, + const struct aegis128_ops *ops) +{ + struct skcipher_walk walk; + u8 *src, *dst; + unsigned int chunksize; + + ops->skcipher_walk_init(&walk, req, false); + + while (walk.nbytes) { + src = walk.src.virt.addr; + dst = walk.dst.virt.addr; + chunksize = walk.nbytes; + + ops->crypt_chunk(state, dst, src, chunksize); + + skcipher_walk_done(&walk, 0); + } +} + +static void crypto_aegis128_final(struct aegis_state *state, + union aegis_block *tag_xor, + u64 assoclen, u64 cryptlen) +{ + u64 assocbits = assoclen * 8; + u64 cryptbits = cryptlen * 8; + + union aegis_block tmp; + unsigned int i; + + tmp.words64[0] = cpu_to_le64(assocbits); + tmp.words64[1] = cpu_to_le64(cryptbits); + + crypto_aegis_block_xor(&tmp, &state->blocks[3]); + + for (i = 0; i < 7; i++) + crypto_aegis128_update_a(state, &tmp); + + for (i = 0; i < AEGIS128_STATE_BLOCKS; i++) + crypto_aegis_block_xor(tag_xor, &state->blocks[i]); +} + +static int crypto_aegis128_setkey(struct crypto_aead *aead, const u8 *key, + unsigned int keylen) +{ + struct aegis_ctx *ctx = crypto_aead_ctx(aead); + + if (keylen != AEGIS128_KEY_SIZE) { + crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; + } + + memcpy(ctx->key.bytes, key, AEGIS128_KEY_SIZE); + return 0; +} + +static int crypto_aegis128_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + if (authsize > AEGIS128_MAX_AUTH_SIZE) + return -EINVAL; + if (authsize < AEGIS128_MIN_AUTH_SIZE) + return -EINVAL; + return 0; +} + +static void crypto_aegis128_crypt(struct aead_request *req, + union aegis_block *tag_xor, + unsigned int cryptlen, + const struct aegis128_ops *ops) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_ctx *ctx = crypto_aead_ctx(tfm); + struct aegis_state state; + + crypto_aegis128_init(&state, &ctx->key, req->iv); + crypto_aegis128_process_ad(&state, req->src, req->assoclen); + crypto_aegis128_process_crypt(&state, req, ops); + crypto_aegis128_final(&state, tag_xor, req->assoclen, cryptlen); +} + +static int crypto_aegis128_encrypt(struct aead_request *req) +{ + static const struct aegis128_ops ops = { + .skcipher_walk_init = skcipher_walk_aead_encrypt, + .crypt_chunk = crypto_aegis128_encrypt_chunk, + }; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + union aegis_block tag = {}; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen; + + crypto_aegis128_crypt(req, &tag, cryptlen, &ops); + + scatterwalk_map_and_copy(tag.bytes, req->dst, req->assoclen + cryptlen, + authsize, 1); + return 0; +} + +static int crypto_aegis128_decrypt(struct aead_request *req) +{ + static const struct aegis128_ops ops = { + .skcipher_walk_init = skcipher_walk_aead_decrypt, + .crypt_chunk = crypto_aegis128_decrypt_chunk, + }; + static const u8 zeros[AEGIS128_MAX_AUTH_SIZE] = {}; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + union aegis_block tag; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen - authsize; + + scatterwalk_map_and_copy(tag.bytes, req->src, req->assoclen + cryptlen, + authsize, 0); + + crypto_aegis128_crypt(req, &tag, cryptlen, &ops); + + return crypto_memneq(tag.bytes, zeros, authsize) ? -EBADMSG : 0; +} + +static int crypto_aegis128_init_tfm(struct crypto_aead *tfm) +{ + return 0; +} + +static void crypto_aegis128_exit_tfm(struct crypto_aead *tfm) +{ +} + +static struct aead_alg crypto_aegis128_alg = { + .setkey = crypto_aegis128_setkey, + .setauthsize = crypto_aegis128_setauthsize, + .encrypt = crypto_aegis128_encrypt, + .decrypt = crypto_aegis128_decrypt, + .init = crypto_aegis128_init_tfm, + .exit = crypto_aegis128_exit_tfm, + + .ivsize = AEGIS128_NONCE_SIZE, + .maxauthsize = AEGIS128_MAX_AUTH_SIZE, + .chunksize = AEGIS_BLOCK_SIZE, + + .base = { + .cra_flags = CRYPTO_ALG_TYPE_AEAD, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct aegis_ctx), + .cra_alignmask = 0, + + .cra_priority = 100, + + .cra_name = "aegis128", + .cra_driver_name = "aegis128-generic", + + .cra_module = THIS_MODULE, + } +}; + +static int __init crypto_aegis128_module_init(void) +{ + return crypto_register_aead(&crypto_aegis128_alg); +} + +static void __exit crypto_aegis128_module_exit(void) +{ + crypto_unregister_aead(&crypto_aegis128_alg); +} + +module_init(crypto_aegis128_module_init); +module_exit(crypto_aegis128_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("AEGIS-128 AEAD algorithm"); +MODULE_ALIAS_CRYPTO("aegis128"); +MODULE_ALIAS_CRYPTO("aegis128-generic"); diff --git a/crypto/aegis128l.c b/crypto/aegis128l.c new file mode 100644 index 000000000000..0cc1a7525c85 --- /dev/null +++ b/crypto/aegis128l.c @@ -0,0 +1,527 @@ +/* + * The AEGIS-128L Authenticated-Encryption Algorithm + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "aegis.h" + +#define AEGIS128L_CHUNK_BLOCKS 2 +#define AEGIS128L_CHUNK_SIZE (AEGIS128L_CHUNK_BLOCKS * AEGIS_BLOCK_SIZE) +#define AEGIS128L_NONCE_SIZE 16 +#define AEGIS128L_STATE_BLOCKS 8 +#define AEGIS128L_KEY_SIZE 16 +#define AEGIS128L_MIN_AUTH_SIZE 8 +#define AEGIS128L_MAX_AUTH_SIZE 16 + +union aegis_chunk { + union aegis_block blocks[AEGIS128L_CHUNK_BLOCKS]; + u8 bytes[AEGIS128L_CHUNK_SIZE]; +}; + +struct aegis_state { + union aegis_block blocks[AEGIS128L_STATE_BLOCKS]; +}; + +struct aegis_ctx { + union aegis_block key; +}; + +struct aegis128l_ops { + int (*skcipher_walk_init)(struct skcipher_walk *walk, + struct aead_request *req, bool atomic); + + void (*crypt_chunk)(struct aegis_state *state, u8 *dst, + const u8 *src, unsigned int size); +}; + +static void crypto_aegis128l_update(struct aegis_state *state) +{ + union aegis_block tmp; + unsigned int i; + + tmp = state->blocks[AEGIS128L_STATE_BLOCKS - 1]; + for (i = AEGIS128L_STATE_BLOCKS - 1; i > 0; i--) + crypto_aegis_aesenc(&state->blocks[i], &state->blocks[i - 1], + &state->blocks[i]); + crypto_aegis_aesenc(&state->blocks[0], &tmp, &state->blocks[0]); +} + +static void crypto_aegis128l_update_a(struct aegis_state *state, + const union aegis_chunk *msg) +{ + crypto_aegis128l_update(state); + crypto_aegis_block_xor(&state->blocks[0], &msg->blocks[0]); + crypto_aegis_block_xor(&state->blocks[4], &msg->blocks[1]); +} + +static void crypto_aegis128l_update_u(struct aegis_state *state, + const void *msg) +{ + crypto_aegis128l_update(state); + crypto_xor(state->blocks[0].bytes, msg + 0 * AEGIS_BLOCK_SIZE, + AEGIS_BLOCK_SIZE); + crypto_xor(state->blocks[4].bytes, msg + 1 * AEGIS_BLOCK_SIZE, + AEGIS_BLOCK_SIZE); +} + +static void crypto_aegis128l_init(struct aegis_state *state, + const union aegis_block *key, + const u8 *iv) +{ + union aegis_block key_iv; + union aegis_chunk chunk; + unsigned int i; + + memcpy(chunk.blocks[0].bytes, iv, AEGIS_BLOCK_SIZE); + chunk.blocks[1] = *key; + + key_iv = *key; + crypto_aegis_block_xor(&key_iv, &chunk.blocks[0]); + + state->blocks[0] = key_iv; + state->blocks[1] = crypto_aegis_const[1]; + state->blocks[2] = crypto_aegis_const[0]; + state->blocks[3] = crypto_aegis_const[1]; + state->blocks[4] = key_iv; + state->blocks[5] = *key; + state->blocks[6] = *key; + state->blocks[7] = *key; + + crypto_aegis_block_xor(&state->blocks[5], &crypto_aegis_const[0]); + crypto_aegis_block_xor(&state->blocks[6], &crypto_aegis_const[1]); + crypto_aegis_block_xor(&state->blocks[7], &crypto_aegis_const[0]); + + for (i = 0; i < 10; i++) { + crypto_aegis128l_update_a(state, &chunk); + } +} + +static void crypto_aegis128l_ad(struct aegis_state *state, + const u8 *src, unsigned int size) +{ + if (AEGIS_ALIGNED(src)) { + const union aegis_chunk *src_chunk = + (const union aegis_chunk *)src; + + while (size >= AEGIS128L_CHUNK_SIZE) { + crypto_aegis128l_update_a(state, src_chunk); + + size -= AEGIS128L_CHUNK_SIZE; + src_chunk += 1; + } + } else { + while (size >= AEGIS128L_CHUNK_SIZE) { + crypto_aegis128l_update_u(state, src); + + size -= AEGIS128L_CHUNK_SIZE; + src += AEGIS128L_CHUNK_SIZE; + } + } +} + +static void crypto_aegis128l_encrypt_chunk(struct aegis_state *state, u8 *dst, + const u8 *src, unsigned int size) +{ + union aegis_chunk tmp; + union aegis_block *tmp0 = &tmp.blocks[0]; + union aegis_block *tmp1 = &tmp.blocks[1]; + + if (AEGIS_ALIGNED(src) && AEGIS_ALIGNED(dst)) { + while (size >= AEGIS128L_CHUNK_SIZE) { + union aegis_chunk *dst_blk = + (union aegis_chunk *)dst; + const union aegis_chunk *src_blk = + (const union aegis_chunk *)src; + + *tmp0 = state->blocks[2]; + crypto_aegis_block_and(tmp0, &state->blocks[3]); + crypto_aegis_block_xor(tmp0, &state->blocks[6]); + crypto_aegis_block_xor(tmp0, &state->blocks[1]); + crypto_aegis_block_xor(tmp0, &src_blk->blocks[0]); + + *tmp1 = state->blocks[6]; + crypto_aegis_block_and(tmp1, &state->blocks[7]); + crypto_aegis_block_xor(tmp1, &state->blocks[5]); + crypto_aegis_block_xor(tmp1, &state->blocks[2]); + crypto_aegis_block_xor(tmp1, &src_blk->blocks[1]); + + crypto_aegis128l_update_a(state, src_blk); + + *dst_blk = tmp; + + size -= AEGIS128L_CHUNK_SIZE; + src += AEGIS128L_CHUNK_SIZE; + dst += AEGIS128L_CHUNK_SIZE; + } + } else { + while (size >= AEGIS128L_CHUNK_SIZE) { + *tmp0 = state->blocks[2]; + crypto_aegis_block_and(tmp0, &state->blocks[3]); + crypto_aegis_block_xor(tmp0, &state->blocks[6]); + crypto_aegis_block_xor(tmp0, &state->blocks[1]); + crypto_xor(tmp0->bytes, src + 0 * AEGIS_BLOCK_SIZE, + AEGIS_BLOCK_SIZE); + + *tmp1 = state->blocks[6]; + crypto_aegis_block_and(tmp1, &state->blocks[7]); + crypto_aegis_block_xor(tmp1, &state->blocks[5]); + crypto_aegis_block_xor(tmp1, &state->blocks[2]); + crypto_xor(tmp1->bytes, src + 1 * AEGIS_BLOCK_SIZE, + AEGIS_BLOCK_SIZE); + + crypto_aegis128l_update_u(state, src); + + memcpy(dst, tmp.bytes, AEGIS128L_CHUNK_SIZE); + + size -= AEGIS128L_CHUNK_SIZE; + src += AEGIS128L_CHUNK_SIZE; + dst += AEGIS128L_CHUNK_SIZE; + } + } + + if (size > 0) { + union aegis_chunk msg = {}; + memcpy(msg.bytes, src, size); + + *tmp0 = state->blocks[2]; + crypto_aegis_block_and(tmp0, &state->blocks[3]); + crypto_aegis_block_xor(tmp0, &state->blocks[6]); + crypto_aegis_block_xor(tmp0, &state->blocks[1]); + + *tmp1 = state->blocks[6]; + crypto_aegis_block_and(tmp1, &state->blocks[7]); + crypto_aegis_block_xor(tmp1, &state->blocks[5]); + crypto_aegis_block_xor(tmp1, &state->blocks[2]); + + crypto_aegis128l_update_a(state, &msg); + + crypto_aegis_block_xor(&msg.blocks[0], tmp0); + crypto_aegis_block_xor(&msg.blocks[1], tmp1); + + memcpy(dst, msg.bytes, size); + } +} + +static void crypto_aegis128l_decrypt_chunk(struct aegis_state *state, u8 *dst, + const u8 *src, unsigned int size) +{ + union aegis_chunk tmp; + union aegis_block *tmp0 = &tmp.blocks[0]; + union aegis_block *tmp1 = &tmp.blocks[1]; + + if (AEGIS_ALIGNED(src) && AEGIS_ALIGNED(dst)) { + while (size >= AEGIS128L_CHUNK_SIZE) { + union aegis_chunk *dst_blk = + (union aegis_chunk *)dst; + const union aegis_chunk *src_blk = + (const union aegis_chunk *)src; + + *tmp0 = state->blocks[2]; + crypto_aegis_block_and(tmp0, &state->blocks[3]); + crypto_aegis_block_xor(tmp0, &state->blocks[6]); + crypto_aegis_block_xor(tmp0, &state->blocks[1]); + crypto_aegis_block_xor(tmp0, &src_blk->blocks[0]); + + *tmp1 = state->blocks[6]; + crypto_aegis_block_and(tmp1, &state->blocks[7]); + crypto_aegis_block_xor(tmp1, &state->blocks[5]); + crypto_aegis_block_xor(tmp1, &state->blocks[2]); + crypto_aegis_block_xor(tmp1, &src_blk->blocks[1]); + + crypto_aegis128l_update_a(state, &tmp); + + *dst_blk = tmp; + + size -= AEGIS128L_CHUNK_SIZE; + src += AEGIS128L_CHUNK_SIZE; + dst += AEGIS128L_CHUNK_SIZE; + } + } else { + while (size >= AEGIS128L_CHUNK_SIZE) { + *tmp0 = state->blocks[2]; + crypto_aegis_block_and(tmp0, &state->blocks[3]); + crypto_aegis_block_xor(tmp0, &state->blocks[6]); + crypto_aegis_block_xor(tmp0, &state->blocks[1]); + crypto_xor(tmp0->bytes, src + 0 * AEGIS_BLOCK_SIZE, + AEGIS_BLOCK_SIZE); + + *tmp1 = state->blocks[6]; + crypto_aegis_block_and(tmp1, &state->blocks[7]); + crypto_aegis_block_xor(tmp1, &state->blocks[5]); + crypto_aegis_block_xor(tmp1, &state->blocks[2]); + crypto_xor(tmp1->bytes, src + 1 * AEGIS_BLOCK_SIZE, + AEGIS_BLOCK_SIZE); + + crypto_aegis128l_update_a(state, &tmp); + + memcpy(dst, tmp.bytes, AEGIS128L_CHUNK_SIZE); + + size -= AEGIS128L_CHUNK_SIZE; + src += AEGIS128L_CHUNK_SIZE; + dst += AEGIS128L_CHUNK_SIZE; + } + } + + if (size > 0) { + union aegis_chunk msg = {}; + memcpy(msg.bytes, src, size); + + *tmp0 = state->blocks[2]; + crypto_aegis_block_and(tmp0, &state->blocks[3]); + crypto_aegis_block_xor(tmp0, &state->blocks[6]); + crypto_aegis_block_xor(tmp0, &state->blocks[1]); + crypto_aegis_block_xor(&msg.blocks[0], tmp0); + + *tmp1 = state->blocks[6]; + crypto_aegis_block_and(tmp1, &state->blocks[7]); + crypto_aegis_block_xor(tmp1, &state->blocks[5]); + crypto_aegis_block_xor(tmp1, &state->blocks[2]); + crypto_aegis_block_xor(&msg.blocks[1], tmp1); + + memset(msg.bytes + size, 0, AEGIS128L_CHUNK_SIZE - size); + + crypto_aegis128l_update_a(state, &msg); + + memcpy(dst, msg.bytes, size); + } +} + +static void crypto_aegis128l_process_ad(struct aegis_state *state, + struct scatterlist *sg_src, + unsigned int assoclen) +{ + struct scatter_walk walk; + union aegis_chunk buf; + unsigned int pos = 0; + + scatterwalk_start(&walk, sg_src); + while (assoclen != 0) { + unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int left = size; + void *mapped = scatterwalk_map(&walk); + const u8 *src = (const u8 *)mapped; + + if (pos + size >= AEGIS128L_CHUNK_SIZE) { + if (pos > 0) { + unsigned int fill = AEGIS128L_CHUNK_SIZE - pos; + memcpy(buf.bytes + pos, src, fill); + crypto_aegis128l_update_a(state, &buf); + pos = 0; + left -= fill; + src += fill; + } + + crypto_aegis128l_ad(state, src, left); + src += left & ~(AEGIS128L_CHUNK_SIZE - 1); + left &= AEGIS128L_CHUNK_SIZE - 1; + } + + memcpy(buf.bytes + pos, src, left); + + pos += left; + assoclen -= size; + scatterwalk_unmap(mapped); + scatterwalk_advance(&walk, size); + scatterwalk_done(&walk, 0, assoclen); + } + + if (pos > 0) { + memset(buf.bytes + pos, 0, AEGIS128L_CHUNK_SIZE - pos); + crypto_aegis128l_update_a(state, &buf); + } +} + +static void crypto_aegis128l_process_crypt(struct aegis_state *state, + struct aead_request *req, + const struct aegis128l_ops *ops) +{ + struct skcipher_walk walk; + u8 *src, *dst; + unsigned int chunksize; + + ops->skcipher_walk_init(&walk, req, false); + + while (walk.nbytes) { + src = walk.src.virt.addr; + dst = walk.dst.virt.addr; + chunksize = walk.nbytes; + + ops->crypt_chunk(state, dst, src, chunksize); + + skcipher_walk_done(&walk, 0); + } +} + +static void crypto_aegis128l_final(struct aegis_state *state, + union aegis_block *tag_xor, + u64 assoclen, u64 cryptlen) +{ + u64 assocbits = assoclen * 8; + u64 cryptbits = cryptlen * 8; + + union aegis_chunk tmp; + unsigned int i; + + tmp.blocks[0].words64[0] = cpu_to_le64(assocbits); + tmp.blocks[0].words64[1] = cpu_to_le64(cryptbits); + + crypto_aegis_block_xor(&tmp.blocks[0], &state->blocks[2]); + + tmp.blocks[1] = tmp.blocks[0]; + for (i = 0; i < 7; i++) + crypto_aegis128l_update_a(state, &tmp); + + for (i = 0; i < 7; i++) + crypto_aegis_block_xor(tag_xor, &state->blocks[i]); +} + +static int crypto_aegis128l_setkey(struct crypto_aead *aead, const u8 *key, + unsigned int keylen) +{ + struct aegis_ctx *ctx = crypto_aead_ctx(aead); + + if (keylen != AEGIS128L_KEY_SIZE) { + crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; + } + + memcpy(ctx->key.bytes, key, AEGIS128L_KEY_SIZE); + return 0; +} + +static int crypto_aegis128l_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + if (authsize > AEGIS128L_MAX_AUTH_SIZE) + return -EINVAL; + if (authsize < AEGIS128L_MIN_AUTH_SIZE) + return -EINVAL; + return 0; +} + +static void crypto_aegis128l_crypt(struct aead_request *req, + union aegis_block *tag_xor, + unsigned int cryptlen, + const struct aegis128l_ops *ops) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_ctx *ctx = crypto_aead_ctx(tfm); + struct aegis_state state; + + crypto_aegis128l_init(&state, &ctx->key, req->iv); + crypto_aegis128l_process_ad(&state, req->src, req->assoclen); + crypto_aegis128l_process_crypt(&state, req, ops); + crypto_aegis128l_final(&state, tag_xor, req->assoclen, cryptlen); +} + +static int crypto_aegis128l_encrypt(struct aead_request *req) +{ + static const struct aegis128l_ops ops = { + .skcipher_walk_init = skcipher_walk_aead_encrypt, + .crypt_chunk = crypto_aegis128l_encrypt_chunk, + }; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + union aegis_block tag = {}; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen; + + crypto_aegis128l_crypt(req, &tag, cryptlen, &ops); + + scatterwalk_map_and_copy(tag.bytes, req->dst, req->assoclen + cryptlen, + authsize, 1); + return 0; +} + +static int crypto_aegis128l_decrypt(struct aead_request *req) +{ + static const struct aegis128l_ops ops = { + .skcipher_walk_init = skcipher_walk_aead_decrypt, + .crypt_chunk = crypto_aegis128l_decrypt_chunk, + }; + static const u8 zeros[AEGIS128L_MAX_AUTH_SIZE] = {}; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + union aegis_block tag; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen - authsize; + + scatterwalk_map_and_copy(tag.bytes, req->src, req->assoclen + cryptlen, + authsize, 0); + + crypto_aegis128l_crypt(req, &tag, cryptlen, &ops); + + return crypto_memneq(tag.bytes, zeros, authsize) ? -EBADMSG : 0; +} + +static int crypto_aegis128l_init_tfm(struct crypto_aead *tfm) +{ + return 0; +} + +static void crypto_aegis128l_exit_tfm(struct crypto_aead *tfm) +{ +} + +static struct aead_alg crypto_aegis128l_alg = { + .setkey = crypto_aegis128l_setkey, + .setauthsize = crypto_aegis128l_setauthsize, + .encrypt = crypto_aegis128l_encrypt, + .decrypt = crypto_aegis128l_decrypt, + .init = crypto_aegis128l_init_tfm, + .exit = crypto_aegis128l_exit_tfm, + + .ivsize = AEGIS128L_NONCE_SIZE, + .maxauthsize = AEGIS128L_MAX_AUTH_SIZE, + .chunksize = AEGIS128L_CHUNK_SIZE, + + .base = { + .cra_flags = CRYPTO_ALG_TYPE_AEAD, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct aegis_ctx), + .cra_alignmask = 0, + + .cra_priority = 100, + + .cra_name = "aegis128l", + .cra_driver_name = "aegis128l-generic", + + .cra_module = THIS_MODULE, + } +}; + +static int __init crypto_aegis128l_module_init(void) +{ + return crypto_register_aead(&crypto_aegis128l_alg); +} + +static void __exit crypto_aegis128l_module_exit(void) +{ + crypto_unregister_aead(&crypto_aegis128l_alg); +} + +module_init(crypto_aegis128l_module_init); +module_exit(crypto_aegis128l_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("AEGIS-128L AEAD algorithm"); +MODULE_ALIAS_CRYPTO("aegis128l"); +MODULE_ALIAS_CRYPTO("aegis128l-generic"); diff --git a/crypto/aegis256.c b/crypto/aegis256.c new file mode 100644 index 000000000000..a489d741d33a --- /dev/null +++ b/crypto/aegis256.c @@ -0,0 +1,478 @@ +/* + * The AEGIS-256 Authenticated-Encryption Algorithm + * + * Copyright (c) 2017-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "aegis.h" + +#define AEGIS256_NONCE_SIZE 32 +#define AEGIS256_STATE_BLOCKS 6 +#define AEGIS256_KEY_SIZE 32 +#define AEGIS256_MIN_AUTH_SIZE 8 +#define AEGIS256_MAX_AUTH_SIZE 16 + +struct aegis_state { + union aegis_block blocks[AEGIS256_STATE_BLOCKS]; +}; + +struct aegis_ctx { + union aegis_block key[AEGIS256_KEY_SIZE / AEGIS_BLOCK_SIZE]; +}; + +struct aegis256_ops { + int (*skcipher_walk_init)(struct skcipher_walk *walk, + struct aead_request *req, bool atomic); + + void (*crypt_chunk)(struct aegis_state *state, u8 *dst, + const u8 *src, unsigned int size); +}; + +static void crypto_aegis256_update(struct aegis_state *state) +{ + union aegis_block tmp; + unsigned int i; + + tmp = state->blocks[AEGIS256_STATE_BLOCKS - 1]; + for (i = AEGIS256_STATE_BLOCKS - 1; i > 0; i--) + crypto_aegis_aesenc(&state->blocks[i], &state->blocks[i - 1], + &state->blocks[i]); + crypto_aegis_aesenc(&state->blocks[0], &tmp, &state->blocks[0]); +} + +static void crypto_aegis256_update_a(struct aegis_state *state, + const union aegis_block *msg) +{ + crypto_aegis256_update(state); + crypto_aegis_block_xor(&state->blocks[0], msg); +} + +static void crypto_aegis256_update_u(struct aegis_state *state, const void *msg) +{ + crypto_aegis256_update(state); + crypto_xor(state->blocks[0].bytes, msg, AEGIS_BLOCK_SIZE); +} + +static void crypto_aegis256_init(struct aegis_state *state, + const union aegis_block *key, + const u8 *iv) +{ + union aegis_block key_iv[2]; + unsigned int i; + + key_iv[0] = key[0]; + key_iv[1] = key[1]; + crypto_xor(key_iv[0].bytes, iv + 0 * AEGIS_BLOCK_SIZE, + AEGIS_BLOCK_SIZE); + crypto_xor(key_iv[1].bytes, iv + 1 * AEGIS_BLOCK_SIZE, + AEGIS_BLOCK_SIZE); + + state->blocks[0] = key_iv[0]; + state->blocks[1] = key_iv[1]; + state->blocks[2] = crypto_aegis_const[1]; + state->blocks[3] = crypto_aegis_const[0]; + state->blocks[4] = key[0]; + state->blocks[5] = key[1]; + + crypto_aegis_block_xor(&state->blocks[4], &crypto_aegis_const[0]); + crypto_aegis_block_xor(&state->blocks[5], &crypto_aegis_const[1]); + + for (i = 0; i < 4; i++) { + crypto_aegis256_update_a(state, &key[0]); + crypto_aegis256_update_a(state, &key[1]); + crypto_aegis256_update_a(state, &key_iv[0]); + crypto_aegis256_update_a(state, &key_iv[1]); + } +} + +static void crypto_aegis256_ad(struct aegis_state *state, + const u8 *src, unsigned int size) +{ + if (AEGIS_ALIGNED(src)) { + const union aegis_block *src_blk = + (const union aegis_block *)src; + + while (size >= AEGIS_BLOCK_SIZE) { + crypto_aegis256_update_a(state, src_blk); + + size -= AEGIS_BLOCK_SIZE; + src_blk++; + } + } else { + while (size >= AEGIS_BLOCK_SIZE) { + crypto_aegis256_update_u(state, src); + + size -= AEGIS_BLOCK_SIZE; + src += AEGIS_BLOCK_SIZE; + } + } +} + +static void crypto_aegis256_encrypt_chunk(struct aegis_state *state, u8 *dst, + const u8 *src, unsigned int size) +{ + union aegis_block tmp; + + if (AEGIS_ALIGNED(src) && AEGIS_ALIGNED(dst)) { + while (size >= AEGIS_BLOCK_SIZE) { + union aegis_block *dst_blk = + (union aegis_block *)dst; + const union aegis_block *src_blk = + (const union aegis_block *)src; + + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[5]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + crypto_aegis_block_xor(&tmp, src_blk); + + crypto_aegis256_update_a(state, src_blk); + + *dst_blk = tmp; + + size -= AEGIS_BLOCK_SIZE; + src += AEGIS_BLOCK_SIZE; + dst += AEGIS_BLOCK_SIZE; + } + } else { + while (size >= AEGIS_BLOCK_SIZE) { + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[5]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + crypto_xor(tmp.bytes, src, AEGIS_BLOCK_SIZE); + + crypto_aegis256_update_u(state, src); + + memcpy(dst, tmp.bytes, AEGIS_BLOCK_SIZE); + + size -= AEGIS_BLOCK_SIZE; + src += AEGIS_BLOCK_SIZE; + dst += AEGIS_BLOCK_SIZE; + } + } + + if (size > 0) { + union aegis_block msg = {}; + memcpy(msg.bytes, src, size); + + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[5]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + + crypto_aegis256_update_a(state, &msg); + + crypto_aegis_block_xor(&msg, &tmp); + + memcpy(dst, msg.bytes, size); + } +} + +static void crypto_aegis256_decrypt_chunk(struct aegis_state *state, u8 *dst, + const u8 *src, unsigned int size) +{ + union aegis_block tmp; + + if (AEGIS_ALIGNED(src) && AEGIS_ALIGNED(dst)) { + while (size >= AEGIS_BLOCK_SIZE) { + union aegis_block *dst_blk = + (union aegis_block *)dst; + const union aegis_block *src_blk = + (const union aegis_block *)src; + + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[5]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + crypto_aegis_block_xor(&tmp, src_blk); + + crypto_aegis256_update_a(state, &tmp); + + *dst_blk = tmp; + + size -= AEGIS_BLOCK_SIZE; + src += AEGIS_BLOCK_SIZE; + dst += AEGIS_BLOCK_SIZE; + } + } else { + while (size >= AEGIS_BLOCK_SIZE) { + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[5]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + crypto_xor(tmp.bytes, src, AEGIS_BLOCK_SIZE); + + crypto_aegis256_update_a(state, &tmp); + + memcpy(dst, tmp.bytes, AEGIS_BLOCK_SIZE); + + size -= AEGIS_BLOCK_SIZE; + src += AEGIS_BLOCK_SIZE; + dst += AEGIS_BLOCK_SIZE; + } + } + + if (size > 0) { + union aegis_block msg = {}; + memcpy(msg.bytes, src, size); + + tmp = state->blocks[2]; + crypto_aegis_block_and(&tmp, &state->blocks[3]); + crypto_aegis_block_xor(&tmp, &state->blocks[5]); + crypto_aegis_block_xor(&tmp, &state->blocks[4]); + crypto_aegis_block_xor(&tmp, &state->blocks[1]); + crypto_aegis_block_xor(&msg, &tmp); + + memset(msg.bytes + size, 0, AEGIS_BLOCK_SIZE - size); + + crypto_aegis256_update_a(state, &msg); + + memcpy(dst, msg.bytes, size); + } +} + +static void crypto_aegis256_process_ad(struct aegis_state *state, + struct scatterlist *sg_src, + unsigned int assoclen) +{ + struct scatter_walk walk; + union aegis_block buf; + unsigned int pos = 0; + + scatterwalk_start(&walk, sg_src); + while (assoclen != 0) { + unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int left = size; + void *mapped = scatterwalk_map(&walk); + const u8 *src = (const u8 *)mapped; + + if (pos + size >= AEGIS_BLOCK_SIZE) { + if (pos > 0) { + unsigned int fill = AEGIS_BLOCK_SIZE - pos; + memcpy(buf.bytes + pos, src, fill); + crypto_aegis256_update_a(state, &buf); + pos = 0; + left -= fill; + src += fill; + } + + crypto_aegis256_ad(state, src, left); + src += left & ~(AEGIS_BLOCK_SIZE - 1); + left &= AEGIS_BLOCK_SIZE - 1; + } + + memcpy(buf.bytes + pos, src, left); + + pos += left; + assoclen -= size; + scatterwalk_unmap(mapped); + scatterwalk_advance(&walk, size); + scatterwalk_done(&walk, 0, assoclen); + } + + if (pos > 0) { + memset(buf.bytes + pos, 0, AEGIS_BLOCK_SIZE - pos); + crypto_aegis256_update_a(state, &buf); + } +} + +static void crypto_aegis256_process_crypt(struct aegis_state *state, + struct aead_request *req, + const struct aegis256_ops *ops) +{ + struct skcipher_walk walk; + u8 *src, *dst; + unsigned int chunksize; + + ops->skcipher_walk_init(&walk, req, false); + + while (walk.nbytes) { + src = walk.src.virt.addr; + dst = walk.dst.virt.addr; + chunksize = walk.nbytes; + + ops->crypt_chunk(state, dst, src, chunksize); + + skcipher_walk_done(&walk, 0); + } +} + +static void crypto_aegis256_final(struct aegis_state *state, + union aegis_block *tag_xor, + u64 assoclen, u64 cryptlen) +{ + u64 assocbits = assoclen * 8; + u64 cryptbits = cryptlen * 8; + + union aegis_block tmp; + unsigned int i; + + tmp.words64[0] = cpu_to_le64(assocbits); + tmp.words64[1] = cpu_to_le64(cryptbits); + + crypto_aegis_block_xor(&tmp, &state->blocks[3]); + + for (i = 0; i < 7; i++) + crypto_aegis256_update_a(state, &tmp); + + for (i = 0; i < AEGIS256_STATE_BLOCKS; i++) + crypto_aegis_block_xor(tag_xor, &state->blocks[i]); +} + +static int crypto_aegis256_setkey(struct crypto_aead *aead, const u8 *key, + unsigned int keylen) +{ + struct aegis_ctx *ctx = crypto_aead_ctx(aead); + + if (keylen != AEGIS256_KEY_SIZE) { + crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; + } + + memcpy(ctx->key[0].bytes, key, AEGIS_BLOCK_SIZE); + memcpy(ctx->key[1].bytes, key + AEGIS_BLOCK_SIZE, + AEGIS_BLOCK_SIZE); + return 0; +} + +static int crypto_aegis256_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + if (authsize > AEGIS256_MAX_AUTH_SIZE) + return -EINVAL; + if (authsize < AEGIS256_MIN_AUTH_SIZE) + return -EINVAL; + return 0; +} + +static void crypto_aegis256_crypt(struct aead_request *req, + union aegis_block *tag_xor, + unsigned int cryptlen, + const struct aegis256_ops *ops) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct aegis_ctx *ctx = crypto_aead_ctx(tfm); + struct aegis_state state; + + crypto_aegis256_init(&state, ctx->key, req->iv); + crypto_aegis256_process_ad(&state, req->src, req->assoclen); + crypto_aegis256_process_crypt(&state, req, ops); + crypto_aegis256_final(&state, tag_xor, req->assoclen, cryptlen); +} + +static int crypto_aegis256_encrypt(struct aead_request *req) +{ + static const struct aegis256_ops ops = { + .skcipher_walk_init = skcipher_walk_aead_encrypt, + .crypt_chunk = crypto_aegis256_encrypt_chunk, + }; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + union aegis_block tag = {}; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen; + + crypto_aegis256_crypt(req, &tag, cryptlen, &ops); + + scatterwalk_map_and_copy(tag.bytes, req->dst, req->assoclen + cryptlen, + authsize, 1); + return 0; +} + +static int crypto_aegis256_decrypt(struct aead_request *req) +{ + static const struct aegis256_ops ops = { + .skcipher_walk_init = skcipher_walk_aead_decrypt, + .crypt_chunk = crypto_aegis256_decrypt_chunk, + }; + static const u8 zeros[AEGIS256_MAX_AUTH_SIZE] = {}; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + union aegis_block tag; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen - authsize; + + scatterwalk_map_and_copy(tag.bytes, req->src, req->assoclen + cryptlen, + authsize, 0); + + crypto_aegis256_crypt(req, &tag, cryptlen, &ops); + + return crypto_memneq(tag.bytes, zeros, authsize) ? -EBADMSG : 0; +} + +static int crypto_aegis256_init_tfm(struct crypto_aead *tfm) +{ + return 0; +} + +static void crypto_aegis256_exit_tfm(struct crypto_aead *tfm) +{ +} + +static struct aead_alg crypto_aegis256_alg = { + .setkey = crypto_aegis256_setkey, + .setauthsize = crypto_aegis256_setauthsize, + .encrypt = crypto_aegis256_encrypt, + .decrypt = crypto_aegis256_decrypt, + .init = crypto_aegis256_init_tfm, + .exit = crypto_aegis256_exit_tfm, + + .ivsize = AEGIS256_NONCE_SIZE, + .maxauthsize = AEGIS256_MAX_AUTH_SIZE, + .chunksize = AEGIS_BLOCK_SIZE, + + .base = { + .cra_flags = CRYPTO_ALG_TYPE_AEAD, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct aegis_ctx), + .cra_alignmask = 0, + + .cra_priority = 100, + + .cra_name = "aegis256", + .cra_driver_name = "aegis256-generic", + + .cra_module = THIS_MODULE, + } +}; + +static int __init crypto_aegis256_module_init(void) +{ + return crypto_register_aead(&crypto_aegis256_alg); +} + +static void __exit crypto_aegis256_module_exit(void) +{ + crypto_unregister_aead(&crypto_aegis256_alg); +} + +module_init(crypto_aegis256_module_init); +module_exit(crypto_aegis256_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("AEGIS-256 AEAD algorithm"); +MODULE_ALIAS_CRYPTO("aegis256"); +MODULE_ALIAS_CRYPTO("aegis256-generic"); diff --git a/crypto/af_alg.c b/crypto/af_alg.c index 7846c0c20cfe..49fa8582138b 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -347,7 +347,6 @@ static const struct proto_ops alg_proto_ops = { .sendpage = sock_no_sendpage, .sendmsg = sock_no_sendmsg, .recvmsg = sock_no_recvmsg, - .poll = sock_no_poll, .bind = alg_bind, .release = af_alg_release, @@ -501,8 +500,8 @@ int af_alg_alloc_tsgl(struct sock *sk) sg = sgl->sg; if (!sg || sgl->cur >= MAX_SGL_ENTS) { - sgl = sock_kmalloc(sk, sizeof(*sgl) + - sizeof(sgl->sg[0]) * (MAX_SGL_ENTS + 1), + sgl = sock_kmalloc(sk, + struct_size(sgl, sg, (MAX_SGL_ENTS + 1)), GFP_KERNEL); if (!sgl) return -ENOMEM; @@ -1061,19 +1060,12 @@ void af_alg_async_cb(struct crypto_async_request *_req, int err) } EXPORT_SYMBOL_GPL(af_alg_async_cb); -/** - * af_alg_poll - poll system call handler - */ -__poll_t af_alg_poll(struct file *file, struct socket *sock, - poll_table *wait) +__poll_t af_alg_poll_mask(struct socket *sock, __poll_t events) { struct sock *sk = sock->sk; struct alg_sock *ask = alg_sk(sk); struct af_alg_ctx *ctx = ask->private; - __poll_t mask; - - sock_poll_wait(file, sk_sleep(sk), wait); - mask = 0; + __poll_t mask = 0; if (!ctx->more || ctx->used) mask |= EPOLLIN | EPOLLRDNORM; @@ -1083,7 +1075,7 @@ __poll_t af_alg_poll(struct file *file, struct socket *sock, return mask; } -EXPORT_SYMBOL_GPL(af_alg_poll); +EXPORT_SYMBOL_GPL(af_alg_poll_mask); /** * af_alg_alloc_areq - allocate struct af_alg_async_req diff --git a/crypto/algapi.c b/crypto/algapi.c index 2a0271b5f62a..c0755cf4f53f 100644 --- a/crypto/algapi.c +++ b/crypto/algapi.c @@ -10,6 +10,7 @@ * */ +#include #include #include #include @@ -59,6 +60,15 @@ static int crypto_check_alg(struct crypto_alg *alg) if (alg->cra_blocksize > PAGE_SIZE / 8) return -EINVAL; + if (!alg->cra_type && (alg->cra_flags & CRYPTO_ALG_TYPE_MASK) == + CRYPTO_ALG_TYPE_CIPHER) { + if (alg->cra_alignmask > MAX_CIPHER_ALIGNMASK) + return -EINVAL; + + if (alg->cra_blocksize > MAX_CIPHER_BLOCKSIZE) + return -EINVAL; + } + if (alg->cra_priority < 0) return -EINVAL; diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c index 4b07edd5a9ff..825524f27438 100644 --- a/crypto/algif_aead.c +++ b/crypto/algif_aead.c @@ -255,8 +255,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg, processed - as); if (!areq->tsgl_entries) areq->tsgl_entries = 1; - areq->tsgl = sock_kmalloc(sk, sizeof(*areq->tsgl) * - areq->tsgl_entries, + areq->tsgl = sock_kmalloc(sk, array_size(sizeof(*areq->tsgl), + areq->tsgl_entries), GFP_KERNEL); if (!areq->tsgl) { err = -ENOMEM; @@ -375,7 +375,7 @@ static struct proto_ops algif_aead_ops = { .sendmsg = aead_sendmsg, .sendpage = af_alg_sendpage, .recvmsg = aead_recvmsg, - .poll = af_alg_poll, + .poll_mask = af_alg_poll_mask, }; static int aead_check_key(struct socket *sock) @@ -471,7 +471,7 @@ static struct proto_ops algif_aead_ops_nokey = { .sendmsg = aead_sendmsg_nokey, .sendpage = aead_sendpage_nokey, .recvmsg = aead_recvmsg_nokey, - .poll = af_alg_poll, + .poll_mask = af_alg_poll_mask, }; static void *aead_bind(const char *name, u32 type, u32 mask) diff --git a/crypto/algif_hash.c b/crypto/algif_hash.c index 6c9b1927a520..bfcf595fd8f9 100644 --- a/crypto/algif_hash.c +++ b/crypto/algif_hash.c @@ -288,7 +288,6 @@ static struct proto_ops algif_hash_ops = { .mmap = sock_no_mmap, .bind = sock_no_bind, .setsockopt = sock_no_setsockopt, - .poll = sock_no_poll, .release = af_alg_release, .sendmsg = hash_sendmsg, @@ -396,7 +395,6 @@ static struct proto_ops algif_hash_ops_nokey = { .mmap = sock_no_mmap, .bind = sock_no_bind, .setsockopt = sock_no_setsockopt, - .poll = sock_no_poll, .release = af_alg_release, .sendmsg = hash_sendmsg_nokey, diff --git a/crypto/algif_rng.c b/crypto/algif_rng.c index 150c2b6480ed..22df3799a17b 100644 --- a/crypto/algif_rng.c +++ b/crypto/algif_rng.c @@ -106,7 +106,6 @@ static struct proto_ops algif_rng_ops = { .bind = sock_no_bind, .accept = sock_no_accept, .setsockopt = sock_no_setsockopt, - .poll = sock_no_poll, .sendmsg = sock_no_sendmsg, .sendpage = sock_no_sendpage, diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c index c4e885df4564..4c04eb9888ad 100644 --- a/crypto/algif_skcipher.c +++ b/crypto/algif_skcipher.c @@ -100,7 +100,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, areq->tsgl_entries = af_alg_count_tsgl(sk, len, 0); if (!areq->tsgl_entries) areq->tsgl_entries = 1; - areq->tsgl = sock_kmalloc(sk, sizeof(*areq->tsgl) * areq->tsgl_entries, + areq->tsgl = sock_kmalloc(sk, array_size(sizeof(*areq->tsgl), + areq->tsgl_entries), GFP_KERNEL); if (!areq->tsgl) { err = -ENOMEM; @@ -205,7 +206,7 @@ static struct proto_ops algif_skcipher_ops = { .sendmsg = skcipher_sendmsg, .sendpage = af_alg_sendpage, .recvmsg = skcipher_recvmsg, - .poll = af_alg_poll, + .poll_mask = af_alg_poll_mask, }; static int skcipher_check_key(struct socket *sock) @@ -301,7 +302,7 @@ static struct proto_ops algif_skcipher_ops_nokey = { .sendmsg = skcipher_sendmsg_nokey, .sendpage = skcipher_sendpage_nokey, .recvmsg = skcipher_recvmsg_nokey, - .poll = af_alg_poll, + .poll_mask = af_alg_poll_mask, }; static void *skcipher_bind(const char *name, u32 type, u32 mask) diff --git a/crypto/asymmetric_keys/asymmetric_type.c b/crypto/asymmetric_keys/asymmetric_type.c index 39aecad286fe..26539e9a8bda 100644 --- a/crypto/asymmetric_keys/asymmetric_type.c +++ b/crypto/asymmetric_keys/asymmetric_type.c @@ -1,6 +1,6 @@ /* Asymmetric public-key cryptography key type * - * See Documentation/security/asymmetric-keys.txt + * See Documentation/crypto/asymmetric-keys.txt * * Copyright (C) 2012 Red Hat, Inc. All Rights Reserved. * Written by David Howells (dhowells@redhat.com) diff --git a/crypto/asymmetric_keys/signature.c b/crypto/asymmetric_keys/signature.c index 11b7ba170904..28198314bc39 100644 --- a/crypto/asymmetric_keys/signature.c +++ b/crypto/asymmetric_keys/signature.c @@ -1,6 +1,6 @@ /* Signature verification with an asymmetric key * - * See Documentation/security/asymmetric-keys.txt + * See Documentation/crypto/asymmetric-keys.txt * * Copyright (C) 2012 Red Hat, Inc. All Rights Reserved. * Written by David Howells (dhowells@redhat.com) diff --git a/crypto/authenc.c b/crypto/authenc.c index d3d6d72fe649..4fa8d40d947b 100644 --- a/crypto/authenc.c +++ b/crypto/authenc.c @@ -108,6 +108,7 @@ static int crypto_authenc_setkey(struct crypto_aead *authenc, const u8 *key, CRYPTO_TFM_RES_MASK); out: + memzero_explicit(&keys, sizeof(keys)); return err; badkey: diff --git a/crypto/authencesn.c b/crypto/authencesn.c index 15f91ddd7f0e..50b804747e20 100644 --- a/crypto/authencesn.c +++ b/crypto/authencesn.c @@ -90,6 +90,7 @@ static int crypto_authenc_esn_setkey(struct crypto_aead *authenc_esn, const u8 * CRYPTO_TFM_RES_MASK); out: + memzero_explicit(&keys, sizeof(keys)); return err; badkey: diff --git a/crypto/cfb.c b/crypto/cfb.c index 94ee39bed758..a0d68c09e1b9 100644 --- a/crypto/cfb.c +++ b/crypto/cfb.c @@ -53,9 +53,8 @@ static void crypto_cfb_encrypt_one(struct crypto_skcipher *tfm, static void crypto_cfb_final(struct skcipher_walk *walk, struct crypto_skcipher *tfm) { - const unsigned int bsize = crypto_cfb_bsize(tfm); const unsigned long alignmask = crypto_skcipher_alignmask(tfm); - u8 tmp[bsize + alignmask]; + u8 tmp[MAX_CIPHER_BLOCKSIZE + MAX_CIPHER_ALIGNMASK]; u8 *stream = PTR_ALIGN(tmp + 0, alignmask + 1); u8 *src = walk->src.virt.addr; u8 *dst = walk->dst.virt.addr; @@ -94,7 +93,7 @@ static int crypto_cfb_encrypt_inplace(struct skcipher_walk *walk, unsigned int nbytes = walk->nbytes; u8 *src = walk->src.virt.addr; u8 *iv = walk->iv; - u8 tmp[bsize]; + u8 tmp[MAX_CIPHER_BLOCKSIZE]; do { crypto_cfb_encrypt_one(tfm, iv, tmp); @@ -164,7 +163,7 @@ static int crypto_cfb_decrypt_inplace(struct skcipher_walk *walk, unsigned int nbytes = walk->nbytes; u8 *src = walk->src.virt.addr; u8 *iv = walk->iv; - u8 tmp[bsize]; + u8 tmp[MAX_CIPHER_BLOCKSIZE]; do { crypto_cfb_encrypt_one(tfm, iv, tmp); diff --git a/crypto/cipher.c b/crypto/cipher.c index 94fa3551476b..57836c30a49a 100644 --- a/crypto/cipher.c +++ b/crypto/cipher.c @@ -13,6 +13,7 @@ * */ +#include #include #include #include @@ -67,7 +68,7 @@ static void cipher_crypt_unaligned(void (*fn)(struct crypto_tfm *, u8 *, { unsigned long alignmask = crypto_tfm_alg_alignmask(tfm); unsigned int size = crypto_tfm_alg_blocksize(tfm); - u8 buffer[size + alignmask]; + u8 buffer[MAX_CIPHER_BLOCKSIZE + MAX_CIPHER_ALIGNMASK]; u8 *tmp = (u8 *)ALIGN((unsigned long)buffer, alignmask + 1); memcpy(tmp, src, size); diff --git a/crypto/crc32_generic.c b/crypto/crc32_generic.c index 718cbce8d169..00facd27bcc2 100644 --- a/crypto/crc32_generic.c +++ b/crypto/crc32_generic.c @@ -29,6 +29,7 @@ * This is crypto api shash wrappers to crc32_le. */ +#include #include #include #include @@ -39,11 +40,6 @@ #define CHKSUM_BLOCK_SIZE 1 #define CHKSUM_DIGEST_SIZE 4 -static u32 __crc32_le(u32 crc, unsigned char const *p, size_t len) -{ - return crc32_le(crc, p, len); -} - /** No default init with ~0 */ static int crc32_cra_init(struct crypto_tfm *tfm) { @@ -54,7 +50,6 @@ static int crc32_cra_init(struct crypto_tfm *tfm) return 0; } - /* * Setting the seed allows arbitrary accumulators and flexible XOR policy * If your algorithm starts with ~0, then XOR with ~0 before you set @@ -69,7 +64,7 @@ static int crc32_setkey(struct crypto_shash *hash, const u8 *key, crypto_shash_set_flags(hash, CRYPTO_TFM_RES_BAD_KEY_LEN); return -EINVAL; } - *mctx = le32_to_cpup((__le32 *)key); + *mctx = get_unaligned_le32(key); return 0; } @@ -88,7 +83,7 @@ static int crc32_update(struct shash_desc *desc, const u8 *data, { u32 *crcp = shash_desc_ctx(desc); - *crcp = __crc32_le(*crcp, data, len); + *crcp = crc32_le(*crcp, data, len); return 0; } @@ -96,7 +91,7 @@ static int crc32_update(struct shash_desc *desc, const u8 *data, static int __crc32_finup(u32 *crcp, const u8 *data, unsigned int len, u8 *out) { - *(__le32 *)out = cpu_to_le32(__crc32_le(*crcp, data, len)); + put_unaligned_le32(crc32_le(*crcp, data, len), out); return 0; } @@ -110,7 +105,7 @@ static int crc32_final(struct shash_desc *desc, u8 *out) { u32 *crcp = shash_desc_ctx(desc); - *(__le32 *)out = cpu_to_le32p(crcp); + put_unaligned_le32(*crcp, out); return 0; } diff --git a/crypto/crc32c_generic.c b/crypto/crc32c_generic.c index 372320399622..7283066ecc98 100644 --- a/crypto/crc32c_generic.c +++ b/crypto/crc32c_generic.c @@ -35,6 +35,7 @@ * */ +#include #include #include #include @@ -82,7 +83,7 @@ static int chksum_setkey(struct crypto_shash *tfm, const u8 *key, crypto_shash_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN); return -EINVAL; } - mctx->key = le32_to_cpu(*(__le32 *)key); + mctx->key = get_unaligned_le32(key); return 0; } @@ -99,13 +100,13 @@ static int chksum_final(struct shash_desc *desc, u8 *out) { struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); - *(__le32 *)out = ~cpu_to_le32p(&ctx->crc); + put_unaligned_le32(~ctx->crc, out); return 0; } static int __chksum_finup(u32 *crcp, const u8 *data, unsigned int len, u8 *out) { - *(__le32 *)out = ~cpu_to_le32(__crc32c_le(*crcp, data, len)); + put_unaligned_le32(~__crc32c_le(*crcp, data, len), out); return 0; } @@ -148,7 +149,6 @@ static struct shash_alg alg = { .cra_priority = 100, .cra_flags = CRYPTO_ALG_OPTIONAL_KEY, .cra_blocksize = CHKSUM_BLOCK_SIZE, - .cra_alignmask = 3, .cra_ctxsize = sizeof(struct chksum_ctx), .cra_module = THIS_MODULE, .cra_init = crc32c_cra_init, diff --git a/crypto/ctr.c b/crypto/ctr.c index 854d924f9d8e..435b75bd619e 100644 --- a/crypto/ctr.c +++ b/crypto/ctr.c @@ -58,7 +58,7 @@ static void crypto_ctr_crypt_final(struct blkcipher_walk *walk, unsigned int bsize = crypto_cipher_blocksize(tfm); unsigned long alignmask = crypto_cipher_alignmask(tfm); u8 *ctrblk = walk->iv; - u8 tmp[bsize + alignmask]; + u8 tmp[MAX_CIPHER_BLOCKSIZE + MAX_CIPHER_ALIGNMASK]; u8 *keystream = PTR_ALIGN(tmp + 0, alignmask + 1); u8 *src = walk->src.virt.addr; u8 *dst = walk->dst.virt.addr; @@ -106,7 +106,7 @@ static int crypto_ctr_crypt_inplace(struct blkcipher_walk *walk, unsigned int nbytes = walk->nbytes; u8 *ctrblk = walk->iv; u8 *src = walk->src.virt.addr; - u8 tmp[bsize + alignmask]; + u8 tmp[MAX_CIPHER_BLOCKSIZE + MAX_CIPHER_ALIGNMASK]; u8 *keystream = PTR_ALIGN(tmp + 0, alignmask + 1); do { diff --git a/crypto/cts.c b/crypto/cts.c index 4773c188e6d9..4e28d83ae37d 100644 --- a/crypto/cts.c +++ b/crypto/cts.c @@ -40,6 +40,7 @@ * rfc3962 includes errata information in its Appendix A. */ +#include #include #include #include @@ -104,7 +105,7 @@ static int cts_cbc_encrypt(struct skcipher_request *req) struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct skcipher_request *subreq = &rctx->subreq; int bsize = crypto_skcipher_blocksize(tfm); - u8 d[bsize * 2] __aligned(__alignof__(u32)); + u8 d[MAX_CIPHER_BLOCKSIZE * 2] __aligned(__alignof__(u32)); struct scatterlist *sg; unsigned int offset; int lastn; @@ -183,7 +184,7 @@ static int cts_cbc_decrypt(struct skcipher_request *req) struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct skcipher_request *subreq = &rctx->subreq; int bsize = crypto_skcipher_blocksize(tfm); - u8 d[bsize * 2] __aligned(__alignof__(u32)); + u8 d[MAX_CIPHER_BLOCKSIZE * 2] __aligned(__alignof__(u32)); struct scatterlist *sg; unsigned int offset; u8 *space; diff --git a/crypto/ecc.c b/crypto/ecc.c index 9c066b5ac12d..815541309a95 100644 --- a/crypto/ecc.c +++ b/crypto/ecc.c @@ -515,7 +515,7 @@ static void vli_mmod_fast_256(u64 *result, const u64 *product, static bool vli_mmod_fast(u64 *result, u64 *product, const u64 *curve_prime, unsigned int ndigits) { - u64 tmp[2 * ndigits]; + u64 tmp[2 * ECC_MAX_DIGITS]; switch (ndigits) { case 3: @@ -536,7 +536,7 @@ static bool vli_mmod_fast(u64 *result, u64 *product, static void vli_mod_mult_fast(u64 *result, const u64 *left, const u64 *right, const u64 *curve_prime, unsigned int ndigits) { - u64 product[2 * ndigits]; + u64 product[2 * ECC_MAX_DIGITS]; vli_mult(product, left, right, ndigits); vli_mmod_fast(result, product, curve_prime, ndigits); @@ -546,7 +546,7 @@ static void vli_mod_mult_fast(u64 *result, const u64 *left, const u64 *right, static void vli_mod_square_fast(u64 *result, const u64 *left, const u64 *curve_prime, unsigned int ndigits) { - u64 product[2 * ndigits]; + u64 product[2 * ECC_MAX_DIGITS]; vli_square(product, left, ndigits); vli_mmod_fast(result, product, curve_prime, ndigits); @@ -560,8 +560,8 @@ static void vli_mod_square_fast(u64 *result, const u64 *left, static void vli_mod_inv(u64 *result, const u64 *input, const u64 *mod, unsigned int ndigits) { - u64 a[ndigits], b[ndigits]; - u64 u[ndigits], v[ndigits]; + u64 a[ECC_MAX_DIGITS], b[ECC_MAX_DIGITS]; + u64 u[ECC_MAX_DIGITS], v[ECC_MAX_DIGITS]; u64 carry; int cmp_result; @@ -649,8 +649,8 @@ static void ecc_point_double_jacobian(u64 *x1, u64 *y1, u64 *z1, u64 *curve_prime, unsigned int ndigits) { /* t1 = x, t2 = y, t3 = z */ - u64 t4[ndigits]; - u64 t5[ndigits]; + u64 t4[ECC_MAX_DIGITS]; + u64 t5[ECC_MAX_DIGITS]; if (vli_is_zero(z1, ndigits)) return; @@ -711,7 +711,7 @@ static void ecc_point_double_jacobian(u64 *x1, u64 *y1, u64 *z1, static void apply_z(u64 *x1, u64 *y1, u64 *z, u64 *curve_prime, unsigned int ndigits) { - u64 t1[ndigits]; + u64 t1[ECC_MAX_DIGITS]; vli_mod_square_fast(t1, z, curve_prime, ndigits); /* z^2 */ vli_mod_mult_fast(x1, x1, t1, curve_prime, ndigits); /* x1 * z^2 */ @@ -724,7 +724,7 @@ static void xycz_initial_double(u64 *x1, u64 *y1, u64 *x2, u64 *y2, u64 *p_initial_z, u64 *curve_prime, unsigned int ndigits) { - u64 z[ndigits]; + u64 z[ECC_MAX_DIGITS]; vli_set(x2, x1, ndigits); vli_set(y2, y1, ndigits); @@ -750,7 +750,7 @@ static void xycz_add(u64 *x1, u64 *y1, u64 *x2, u64 *y2, u64 *curve_prime, unsigned int ndigits) { /* t1 = X1, t2 = Y1, t3 = X2, t4 = Y2 */ - u64 t5[ndigits]; + u64 t5[ECC_MAX_DIGITS]; /* t5 = x2 - x1 */ vli_mod_sub(t5, x2, x1, curve_prime, ndigits); @@ -791,9 +791,9 @@ static void xycz_add_c(u64 *x1, u64 *y1, u64 *x2, u64 *y2, u64 *curve_prime, unsigned int ndigits) { /* t1 = X1, t2 = Y1, t3 = X2, t4 = Y2 */ - u64 t5[ndigits]; - u64 t6[ndigits]; - u64 t7[ndigits]; + u64 t5[ECC_MAX_DIGITS]; + u64 t6[ECC_MAX_DIGITS]; + u64 t7[ECC_MAX_DIGITS]; /* t5 = x2 - x1 */ vli_mod_sub(t5, x2, x1, curve_prime, ndigits); @@ -846,9 +846,9 @@ static void ecc_point_mult(struct ecc_point *result, unsigned int ndigits) { /* R0 and R1 */ - u64 rx[2][ndigits]; - u64 ry[2][ndigits]; - u64 z[ndigits]; + u64 rx[2][ECC_MAX_DIGITS]; + u64 ry[2][ECC_MAX_DIGITS]; + u64 z[ECC_MAX_DIGITS]; int i, nb; int num_bits = vli_num_bits(scalar, ndigits); @@ -943,13 +943,13 @@ int ecc_is_key_valid(unsigned int curve_id, unsigned int ndigits, int ecc_gen_privkey(unsigned int curve_id, unsigned int ndigits, u64 *privkey) { const struct ecc_curve *curve = ecc_get_curve(curve_id); - u64 priv[ndigits]; + u64 priv[ECC_MAX_DIGITS]; unsigned int nbytes = ndigits << ECC_DIGITS_TO_BYTES_SHIFT; unsigned int nbits = vli_num_bits(curve->n, ndigits); int err; /* Check that N is included in Table 1 of FIPS 186-4, section 6.1.1 */ - if (nbits < 160) + if (nbits < 160 || ndigits > ARRAY_SIZE(priv)) return -EINVAL; /* @@ -988,10 +988,10 @@ int ecc_make_pub_key(unsigned int curve_id, unsigned int ndigits, { int ret = 0; struct ecc_point *pk; - u64 priv[ndigits]; + u64 priv[ECC_MAX_DIGITS]; const struct ecc_curve *curve = ecc_get_curve(curve_id); - if (!private_key || !curve) { + if (!private_key || !curve || ndigits > ARRAY_SIZE(priv)) { ret = -EINVAL; goto out; } @@ -1025,30 +1025,25 @@ int crypto_ecdh_shared_secret(unsigned int curve_id, unsigned int ndigits, { int ret = 0; struct ecc_point *product, *pk; - u64 *priv, *rand_z; + u64 priv[ECC_MAX_DIGITS]; + u64 rand_z[ECC_MAX_DIGITS]; + unsigned int nbytes; const struct ecc_curve *curve = ecc_get_curve(curve_id); - if (!private_key || !public_key || !curve) { + if (!private_key || !public_key || !curve || + ndigits > ARRAY_SIZE(priv) || ndigits > ARRAY_SIZE(rand_z)) { ret = -EINVAL; goto out; } - priv = kmalloc_array(ndigits, sizeof(*priv), GFP_KERNEL); - if (!priv) { - ret = -ENOMEM; - goto out; - } + nbytes = ndigits << ECC_DIGITS_TO_BYTES_SHIFT; - rand_z = kmalloc_array(ndigits, sizeof(*rand_z), GFP_KERNEL); - if (!rand_z) { - ret = -ENOMEM; - goto kfree_out; - } + get_random_bytes(rand_z, nbytes); pk = ecc_alloc_point(ndigits); if (!pk) { ret = -ENOMEM; - goto kfree_out; + goto out; } product = ecc_alloc_point(ndigits); @@ -1057,8 +1052,6 @@ int crypto_ecdh_shared_secret(unsigned int curve_id, unsigned int ndigits, goto err_alloc_product; } - get_random_bytes(rand_z, ndigits << ECC_DIGITS_TO_BYTES_SHIFT); - ecc_swap_digits(public_key, pk->x, ndigits); ecc_swap_digits(&public_key[ndigits], pk->y, ndigits); ecc_swap_digits(private_key, priv, ndigits); @@ -1073,9 +1066,6 @@ int crypto_ecdh_shared_secret(unsigned int curve_id, unsigned int ndigits, ecc_free_point(product); err_alloc_product: ecc_free_point(pk); -kfree_out: - kzfree(priv); - kzfree(rand_z); out: return ret; } diff --git a/crypto/ecc.h b/crypto/ecc.h index e4fd4492c765..f75a86baa3bd 100644 --- a/crypto/ecc.h +++ b/crypto/ecc.h @@ -26,7 +26,9 @@ #ifndef _CRYPTO_ECC_H #define _CRYPTO_ECC_H -#define ECC_MAX_DIGITS 4 /* 256 */ +#define ECC_CURVE_NIST_P192_DIGITS 3 +#define ECC_CURVE_NIST_P256_DIGITS 4 +#define ECC_MAX_DIGITS ECC_CURVE_NIST_P256_DIGITS #define ECC_DIGITS_TO_BYTES_SHIFT 3 diff --git a/crypto/ecdh.c b/crypto/ecdh.c index d2ec33f0e098..bf6300175b9c 100644 --- a/crypto/ecdh.c +++ b/crypto/ecdh.c @@ -30,8 +30,8 @@ static inline struct ecdh_ctx *ecdh_get_ctx(struct crypto_kpp *tfm) static unsigned int ecdh_supported_curve(unsigned int curve_id) { switch (curve_id) { - case ECC_CURVE_NIST_P192: return 3; - case ECC_CURVE_NIST_P256: return 4; + case ECC_CURVE_NIST_P192: return ECC_CURVE_NIST_P192_DIGITS; + case ECC_CURVE_NIST_P256: return ECC_CURVE_NIST_P256_DIGITS; default: return 0; } } diff --git a/crypto/morus1280.c b/crypto/morus1280.c new file mode 100644 index 000000000000..6180b2557836 --- /dev/null +++ b/crypto/morus1280.c @@ -0,0 +1,549 @@ +/* + * The MORUS-1280 Authenticated-Encryption Algorithm + * + * Copyright (c) 2016-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MORUS1280_WORD_SIZE 8 +#define MORUS1280_BLOCK_SIZE (MORUS_BLOCK_WORDS * MORUS1280_WORD_SIZE) +#define MORUS1280_BLOCK_ALIGN (__alignof__(__le64)) +#define MORUS1280_ALIGNED(p) IS_ALIGNED((uintptr_t)p, MORUS1280_BLOCK_ALIGN) + +struct morus1280_block { + u64 words[MORUS_BLOCK_WORDS]; +}; + +union morus1280_block_in { + __le64 words[MORUS_BLOCK_WORDS]; + u8 bytes[MORUS1280_BLOCK_SIZE]; +}; + +struct morus1280_state { + struct morus1280_block s[MORUS_STATE_BLOCKS]; +}; + +struct morus1280_ctx { + struct morus1280_block key; +}; + +struct morus1280_ops { + int (*skcipher_walk_init)(struct skcipher_walk *walk, + struct aead_request *req, bool atomic); + + void (*crypt_chunk)(struct morus1280_state *state, + u8 *dst, const u8 *src, unsigned int size); +}; + +static const struct morus1280_block crypto_morus1280_const[1] = { + { .words = { + U64_C(0x0d08050302010100), + U64_C(0x6279e99059372215), + U64_C(0xf12fc26d55183ddb), + U64_C(0xdd28b57342311120), + } }, +}; + +static void crypto_morus1280_round(struct morus1280_block *b0, + struct morus1280_block *b1, + struct morus1280_block *b2, + struct morus1280_block *b3, + struct morus1280_block *b4, + const struct morus1280_block *m, + unsigned int b, unsigned int w) +{ + unsigned int i; + struct morus1280_block tmp; + + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + b0->words[i] ^= b1->words[i] & b2->words[i]; + b0->words[i] ^= b3->words[i]; + b0->words[i] ^= m->words[i]; + b0->words[i] = rol64(b0->words[i], b); + } + + tmp = *b3; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) + b3->words[(i + w) % MORUS_BLOCK_WORDS] = tmp.words[i]; +} + +static void crypto_morus1280_update(struct morus1280_state *state, + const struct morus1280_block *m) +{ + static const struct morus1280_block z = {}; + + struct morus1280_block *s = state->s; + + crypto_morus1280_round(&s[0], &s[1], &s[2], &s[3], &s[4], &z, 13, 1); + crypto_morus1280_round(&s[1], &s[2], &s[3], &s[4], &s[0], m, 46, 2); + crypto_morus1280_round(&s[2], &s[3], &s[4], &s[0], &s[1], m, 38, 3); + crypto_morus1280_round(&s[3], &s[4], &s[0], &s[1], &s[2], m, 7, 2); + crypto_morus1280_round(&s[4], &s[0], &s[1], &s[2], &s[3], m, 4, 1); +} + +static void crypto_morus1280_load_a(struct morus1280_block *dst, const u8 *src) +{ + unsigned int i; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + dst->words[i] = le64_to_cpu(*(const __le64 *)src); + src += MORUS1280_WORD_SIZE; + } +} + +static void crypto_morus1280_load_u(struct morus1280_block *dst, const u8 *src) +{ + unsigned int i; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + dst->words[i] = get_unaligned_le64(src); + src += MORUS1280_WORD_SIZE; + } +} + +static void crypto_morus1280_load(struct morus1280_block *dst, const u8 *src) +{ + if (MORUS1280_ALIGNED(src)) + crypto_morus1280_load_a(dst, src); + else + crypto_morus1280_load_u(dst, src); +} + +static void crypto_morus1280_store_a(u8 *dst, const struct morus1280_block *src) +{ + unsigned int i; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + *(__le64 *)dst = cpu_to_le64(src->words[i]); + dst += MORUS1280_WORD_SIZE; + } +} + +static void crypto_morus1280_store_u(u8 *dst, const struct morus1280_block *src) +{ + unsigned int i; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + put_unaligned_le64(src->words[i], dst); + dst += MORUS1280_WORD_SIZE; + } +} + +static void crypto_morus1280_store(u8 *dst, const struct morus1280_block *src) +{ + if (MORUS1280_ALIGNED(dst)) + crypto_morus1280_store_a(dst, src); + else + crypto_morus1280_store_u(dst, src); +} + +static void crypto_morus1280_ad(struct morus1280_state *state, const u8 *src, + unsigned int size) +{ + struct morus1280_block m; + + if (MORUS1280_ALIGNED(src)) { + while (size >= MORUS1280_BLOCK_SIZE) { + crypto_morus1280_load_a(&m, src); + crypto_morus1280_update(state, &m); + + size -= MORUS1280_BLOCK_SIZE; + src += MORUS1280_BLOCK_SIZE; + } + } else { + while (size >= MORUS1280_BLOCK_SIZE) { + crypto_morus1280_load_u(&m, src); + crypto_morus1280_update(state, &m); + + size -= MORUS1280_BLOCK_SIZE; + src += MORUS1280_BLOCK_SIZE; + } + } +} + +static void crypto_morus1280_core(const struct morus1280_state *state, + struct morus1280_block *blk) +{ + unsigned int i; + + for (i = 0; i < MORUS_BLOCK_WORDS; i++) + blk->words[(i + 3) % MORUS_BLOCK_WORDS] ^= state->s[1].words[i]; + + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + blk->words[i] ^= state->s[0].words[i]; + blk->words[i] ^= state->s[2].words[i] & state->s[3].words[i]; + } +} + +static void crypto_morus1280_encrypt_chunk(struct morus1280_state *state, + u8 *dst, const u8 *src, + unsigned int size) +{ + struct morus1280_block c, m; + + if (MORUS1280_ALIGNED(src) && MORUS1280_ALIGNED(dst)) { + while (size >= MORUS1280_BLOCK_SIZE) { + crypto_morus1280_load_a(&m, src); + c = m; + crypto_morus1280_core(state, &c); + crypto_morus1280_store_a(dst, &c); + crypto_morus1280_update(state, &m); + + src += MORUS1280_BLOCK_SIZE; + dst += MORUS1280_BLOCK_SIZE; + size -= MORUS1280_BLOCK_SIZE; + } + } else { + while (size >= MORUS1280_BLOCK_SIZE) { + crypto_morus1280_load_u(&m, src); + c = m; + crypto_morus1280_core(state, &c); + crypto_morus1280_store_u(dst, &c); + crypto_morus1280_update(state, &m); + + src += MORUS1280_BLOCK_SIZE; + dst += MORUS1280_BLOCK_SIZE; + size -= MORUS1280_BLOCK_SIZE; + } + } + + if (size > 0) { + union morus1280_block_in tail; + + memcpy(tail.bytes, src, size); + memset(tail.bytes + size, 0, MORUS1280_BLOCK_SIZE - size); + + crypto_morus1280_load_a(&m, tail.bytes); + c = m; + crypto_morus1280_core(state, &c); + crypto_morus1280_store_a(tail.bytes, &c); + crypto_morus1280_update(state, &m); + + memcpy(dst, tail.bytes, size); + } +} + +static void crypto_morus1280_decrypt_chunk(struct morus1280_state *state, + u8 *dst, const u8 *src, + unsigned int size) +{ + struct morus1280_block m; + + if (MORUS1280_ALIGNED(src) && MORUS1280_ALIGNED(dst)) { + while (size >= MORUS1280_BLOCK_SIZE) { + crypto_morus1280_load_a(&m, src); + crypto_morus1280_core(state, &m); + crypto_morus1280_store_a(dst, &m); + crypto_morus1280_update(state, &m); + + src += MORUS1280_BLOCK_SIZE; + dst += MORUS1280_BLOCK_SIZE; + size -= MORUS1280_BLOCK_SIZE; + } + } else { + while (size >= MORUS1280_BLOCK_SIZE) { + crypto_morus1280_load_u(&m, src); + crypto_morus1280_core(state, &m); + crypto_morus1280_store_u(dst, &m); + crypto_morus1280_update(state, &m); + + src += MORUS1280_BLOCK_SIZE; + dst += MORUS1280_BLOCK_SIZE; + size -= MORUS1280_BLOCK_SIZE; + } + } + + if (size > 0) { + union morus1280_block_in tail; + + memcpy(tail.bytes, src, size); + memset(tail.bytes + size, 0, MORUS1280_BLOCK_SIZE - size); + + crypto_morus1280_load_a(&m, tail.bytes); + crypto_morus1280_core(state, &m); + crypto_morus1280_store_a(tail.bytes, &m); + memset(tail.bytes + size, 0, MORUS1280_BLOCK_SIZE - size); + crypto_morus1280_load_a(&m, tail.bytes); + crypto_morus1280_update(state, &m); + + memcpy(dst, tail.bytes, size); + } +} + +static void crypto_morus1280_init(struct morus1280_state *state, + const struct morus1280_block *key, + const u8 *iv) +{ + static const struct morus1280_block z = {}; + + union morus1280_block_in tmp; + unsigned int i; + + memcpy(tmp.bytes, iv, MORUS_NONCE_SIZE); + memset(tmp.bytes + MORUS_NONCE_SIZE, 0, + MORUS1280_BLOCK_SIZE - MORUS_NONCE_SIZE); + + crypto_morus1280_load(&state->s[0], tmp.bytes); + state->s[1] = *key; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) + state->s[2].words[i] = U64_C(0xFFFFFFFFFFFFFFFF); + state->s[3] = z; + state->s[4] = crypto_morus1280_const[0]; + + for (i = 0; i < 16; i++) + crypto_morus1280_update(state, &z); + + for (i = 0; i < MORUS_BLOCK_WORDS; i++) + state->s[1].words[i] ^= key->words[i]; +} + +static void crypto_morus1280_process_ad(struct morus1280_state *state, + struct scatterlist *sg_src, + unsigned int assoclen) +{ + struct scatter_walk walk; + struct morus1280_block m; + union morus1280_block_in buf; + unsigned int pos = 0; + + scatterwalk_start(&walk, sg_src); + while (assoclen != 0) { + unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int left = size; + void *mapped = scatterwalk_map(&walk); + const u8 *src = (const u8 *)mapped; + + if (pos + size >= MORUS1280_BLOCK_SIZE) { + if (pos > 0) { + unsigned int fill = MORUS1280_BLOCK_SIZE - pos; + memcpy(buf.bytes + pos, src, fill); + + crypto_morus1280_load_a(&m, buf.bytes); + crypto_morus1280_update(state, &m); + + pos = 0; + left -= fill; + src += fill; + } + + crypto_morus1280_ad(state, src, left); + src += left & ~(MORUS1280_BLOCK_SIZE - 1); + left &= MORUS1280_BLOCK_SIZE - 1; + } + + memcpy(buf.bytes + pos, src, left); + + pos += left; + assoclen -= size; + scatterwalk_unmap(mapped); + scatterwalk_advance(&walk, size); + scatterwalk_done(&walk, 0, assoclen); + } + + if (pos > 0) { + memset(buf.bytes + pos, 0, MORUS1280_BLOCK_SIZE - pos); + + crypto_morus1280_load_a(&m, buf.bytes); + crypto_morus1280_update(state, &m); + } +} + +static void crypto_morus1280_process_crypt(struct morus1280_state *state, + struct aead_request *req, + const struct morus1280_ops *ops) +{ + struct skcipher_walk walk; + u8 *dst; + const u8 *src; + + ops->skcipher_walk_init(&walk, req, false); + + while (walk.nbytes) { + src = walk.src.virt.addr; + dst = walk.dst.virt.addr; + + ops->crypt_chunk(state, dst, src, walk.nbytes); + + skcipher_walk_done(&walk, 0); + } +} + +static void crypto_morus1280_final(struct morus1280_state *state, + struct morus1280_block *tag_xor, + u64 assoclen, u64 cryptlen) +{ + u64 assocbits = assoclen * 8; + u64 cryptbits = cryptlen * 8; + + struct morus1280_block tmp; + unsigned int i; + + tmp.words[0] = cpu_to_le64(assocbits); + tmp.words[1] = cpu_to_le64(cryptbits); + tmp.words[2] = 0; + tmp.words[3] = 0; + + for (i = 0; i < MORUS_BLOCK_WORDS; i++) + state->s[4].words[i] ^= state->s[0].words[i]; + + for (i = 0; i < 10; i++) + crypto_morus1280_update(state, &tmp); + + crypto_morus1280_core(state, tag_xor); +} + +static int crypto_morus1280_setkey(struct crypto_aead *aead, const u8 *key, + unsigned int keylen) +{ + struct morus1280_ctx *ctx = crypto_aead_ctx(aead); + union morus1280_block_in tmp; + + if (keylen == MORUS1280_BLOCK_SIZE) + crypto_morus1280_load(&ctx->key, key); + else if (keylen == MORUS1280_BLOCK_SIZE / 2) { + memcpy(tmp.bytes, key, keylen); + memcpy(tmp.bytes + keylen, key, keylen); + + crypto_morus1280_load(&ctx->key, tmp.bytes); + } else { + crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; + } + + return 0; +} + +static int crypto_morus1280_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + return (authsize <= MORUS_MAX_AUTH_SIZE) ? 0 : -EINVAL; +} + +static void crypto_morus1280_crypt(struct aead_request *req, + struct morus1280_block *tag_xor, + unsigned int cryptlen, + const struct morus1280_ops *ops) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct morus1280_ctx *ctx = crypto_aead_ctx(tfm); + struct morus1280_state state; + + crypto_morus1280_init(&state, &ctx->key, req->iv); + crypto_morus1280_process_ad(&state, req->src, req->assoclen); + crypto_morus1280_process_crypt(&state, req, ops); + crypto_morus1280_final(&state, tag_xor, req->assoclen, cryptlen); +} + +static int crypto_morus1280_encrypt(struct aead_request *req) +{ + static const struct morus1280_ops ops = { + .skcipher_walk_init = skcipher_walk_aead_encrypt, + .crypt_chunk = crypto_morus1280_encrypt_chunk, + }; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct morus1280_block tag = {}; + union morus1280_block_in tag_out; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen; + + crypto_morus1280_crypt(req, &tag, cryptlen, &ops); + crypto_morus1280_store(tag_out.bytes, &tag); + + scatterwalk_map_and_copy(tag_out.bytes, req->dst, + req->assoclen + cryptlen, authsize, 1); + return 0; +} + +static int crypto_morus1280_decrypt(struct aead_request *req) +{ + static const struct morus1280_ops ops = { + .skcipher_walk_init = skcipher_walk_aead_decrypt, + .crypt_chunk = crypto_morus1280_decrypt_chunk, + }; + static const u8 zeros[MORUS1280_BLOCK_SIZE] = {}; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + union morus1280_block_in tag_in; + struct morus1280_block tag; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen - authsize; + + scatterwalk_map_and_copy(tag_in.bytes, req->src, + req->assoclen + cryptlen, authsize, 0); + + crypto_morus1280_load(&tag, tag_in.bytes); + crypto_morus1280_crypt(req, &tag, cryptlen, &ops); + crypto_morus1280_store(tag_in.bytes, &tag); + + return crypto_memneq(tag_in.bytes, zeros, authsize) ? -EBADMSG : 0; +} + +static int crypto_morus1280_init_tfm(struct crypto_aead *tfm) +{ + return 0; +} + +static void crypto_morus1280_exit_tfm(struct crypto_aead *tfm) +{ +} + +static struct aead_alg crypto_morus1280_alg = { + .setkey = crypto_morus1280_setkey, + .setauthsize = crypto_morus1280_setauthsize, + .encrypt = crypto_morus1280_encrypt, + .decrypt = crypto_morus1280_decrypt, + .init = crypto_morus1280_init_tfm, + .exit = crypto_morus1280_exit_tfm, + + .ivsize = MORUS_NONCE_SIZE, + .maxauthsize = MORUS_MAX_AUTH_SIZE, + .chunksize = MORUS1280_BLOCK_SIZE, + + .base = { + .cra_flags = CRYPTO_ALG_TYPE_AEAD, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct morus1280_ctx), + .cra_alignmask = 0, + + .cra_priority = 100, + + .cra_name = "morus1280", + .cra_driver_name = "morus1280-generic", + + .cra_module = THIS_MODULE, + } +}; + + +static int __init crypto_morus1280_module_init(void) +{ + return crypto_register_aead(&crypto_morus1280_alg); +} + +static void __exit crypto_morus1280_module_exit(void) +{ + crypto_unregister_aead(&crypto_morus1280_alg); +} + +module_init(crypto_morus1280_module_init); +module_exit(crypto_morus1280_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("MORUS-1280 AEAD algorithm"); +MODULE_ALIAS_CRYPTO("morus1280"); +MODULE_ALIAS_CRYPTO("morus1280-generic"); diff --git a/crypto/morus640.c b/crypto/morus640.c new file mode 100644 index 000000000000..9fbcde307daf --- /dev/null +++ b/crypto/morus640.c @@ -0,0 +1,544 @@ +/* + * The MORUS-640 Authenticated-Encryption Algorithm + * + * Copyright (c) 2016-2018 Ondrej Mosnacek + * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; either version 2 of the License, or (at your option) + * any later version. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MORUS640_WORD_SIZE 4 +#define MORUS640_BLOCK_SIZE (MORUS_BLOCK_WORDS * MORUS640_WORD_SIZE) +#define MORUS640_BLOCK_ALIGN (__alignof__(__le32)) +#define MORUS640_ALIGNED(p) IS_ALIGNED((uintptr_t)p, MORUS640_BLOCK_ALIGN) + +struct morus640_block { + u32 words[MORUS_BLOCK_WORDS]; +}; + +union morus640_block_in { + __le32 words[MORUS_BLOCK_WORDS]; + u8 bytes[MORUS640_BLOCK_SIZE]; +}; + +struct morus640_state { + struct morus640_block s[MORUS_STATE_BLOCKS]; +}; + +struct morus640_ctx { + struct morus640_block key; +}; + +struct morus640_ops { + int (*skcipher_walk_init)(struct skcipher_walk *walk, + struct aead_request *req, bool atomic); + + void (*crypt_chunk)(struct morus640_state *state, + u8 *dst, const u8 *src, unsigned int size); +}; + +static const struct morus640_block crypto_morus640_const[2] = { + { .words = { + U32_C(0x02010100), + U32_C(0x0d080503), + U32_C(0x59372215), + U32_C(0x6279e990), + } }, + { .words = { + U32_C(0x55183ddb), + U32_C(0xf12fc26d), + U32_C(0x42311120), + U32_C(0xdd28b573), + } }, +}; + +static void crypto_morus640_round(struct morus640_block *b0, + struct morus640_block *b1, + struct morus640_block *b2, + struct morus640_block *b3, + struct morus640_block *b4, + const struct morus640_block *m, + unsigned int b, unsigned int w) +{ + unsigned int i; + struct morus640_block tmp; + + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + b0->words[i] ^= b1->words[i] & b2->words[i]; + b0->words[i] ^= b3->words[i]; + b0->words[i] ^= m->words[i]; + b0->words[i] = rol32(b0->words[i], b); + } + + tmp = *b3; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) + b3->words[(i + w) % MORUS_BLOCK_WORDS] = tmp.words[i]; +} + +static void crypto_morus640_update(struct morus640_state *state, + const struct morus640_block *m) +{ + static const struct morus640_block z = {}; + + struct morus640_block *s = state->s; + + crypto_morus640_round(&s[0], &s[1], &s[2], &s[3], &s[4], &z, 5, 1); + crypto_morus640_round(&s[1], &s[2], &s[3], &s[4], &s[0], m, 31, 2); + crypto_morus640_round(&s[2], &s[3], &s[4], &s[0], &s[1], m, 7, 3); + crypto_morus640_round(&s[3], &s[4], &s[0], &s[1], &s[2], m, 22, 2); + crypto_morus640_round(&s[4], &s[0], &s[1], &s[2], &s[3], m, 13, 1); +} + +static void crypto_morus640_load_a(struct morus640_block *dst, const u8 *src) +{ + unsigned int i; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + dst->words[i] = le32_to_cpu(*(const __le32 *)src); + src += MORUS640_WORD_SIZE; + } +} + +static void crypto_morus640_load_u(struct morus640_block *dst, const u8 *src) +{ + unsigned int i; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + dst->words[i] = get_unaligned_le32(src); + src += MORUS640_WORD_SIZE; + } +} + +static void crypto_morus640_load(struct morus640_block *dst, const u8 *src) +{ + if (MORUS640_ALIGNED(src)) + crypto_morus640_load_a(dst, src); + else + crypto_morus640_load_u(dst, src); +} + +static void crypto_morus640_store_a(u8 *dst, const struct morus640_block *src) +{ + unsigned int i; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + *(__le32 *)dst = cpu_to_le32(src->words[i]); + dst += MORUS640_WORD_SIZE; + } +} + +static void crypto_morus640_store_u(u8 *dst, const struct morus640_block *src) +{ + unsigned int i; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + put_unaligned_le32(src->words[i], dst); + dst += MORUS640_WORD_SIZE; + } +} + +static void crypto_morus640_store(u8 *dst, const struct morus640_block *src) +{ + if (MORUS640_ALIGNED(dst)) + crypto_morus640_store_a(dst, src); + else + crypto_morus640_store_u(dst, src); +} + +static void crypto_morus640_ad(struct morus640_state *state, const u8 *src, + unsigned int size) +{ + struct morus640_block m; + + if (MORUS640_ALIGNED(src)) { + while (size >= MORUS640_BLOCK_SIZE) { + crypto_morus640_load_a(&m, src); + crypto_morus640_update(state, &m); + + size -= MORUS640_BLOCK_SIZE; + src += MORUS640_BLOCK_SIZE; + } + } else { + while (size >= MORUS640_BLOCK_SIZE) { + crypto_morus640_load_u(&m, src); + crypto_morus640_update(state, &m); + + size -= MORUS640_BLOCK_SIZE; + src += MORUS640_BLOCK_SIZE; + } + } +} + +static void crypto_morus640_core(const struct morus640_state *state, + struct morus640_block *blk) +{ + unsigned int i; + + for (i = 0; i < MORUS_BLOCK_WORDS; i++) + blk->words[(i + 3) % MORUS_BLOCK_WORDS] ^= state->s[1].words[i]; + + for (i = 0; i < MORUS_BLOCK_WORDS; i++) { + blk->words[i] ^= state->s[0].words[i]; + blk->words[i] ^= state->s[2].words[i] & state->s[3].words[i]; + } +} + +static void crypto_morus640_encrypt_chunk(struct morus640_state *state, u8 *dst, + const u8 *src, unsigned int size) +{ + struct morus640_block c, m; + + if (MORUS640_ALIGNED(src) && MORUS640_ALIGNED(dst)) { + while (size >= MORUS640_BLOCK_SIZE) { + crypto_morus640_load_a(&m, src); + c = m; + crypto_morus640_core(state, &c); + crypto_morus640_store_a(dst, &c); + crypto_morus640_update(state, &m); + + src += MORUS640_BLOCK_SIZE; + dst += MORUS640_BLOCK_SIZE; + size -= MORUS640_BLOCK_SIZE; + } + } else { + while (size >= MORUS640_BLOCK_SIZE) { + crypto_morus640_load_u(&m, src); + c = m; + crypto_morus640_core(state, &c); + crypto_morus640_store_u(dst, &c); + crypto_morus640_update(state, &m); + + src += MORUS640_BLOCK_SIZE; + dst += MORUS640_BLOCK_SIZE; + size -= MORUS640_BLOCK_SIZE; + } + } + + if (size > 0) { + union morus640_block_in tail; + + memcpy(tail.bytes, src, size); + memset(tail.bytes + size, 0, MORUS640_BLOCK_SIZE - size); + + crypto_morus640_load_a(&m, tail.bytes); + c = m; + crypto_morus640_core(state, &c); + crypto_morus640_store_a(tail.bytes, &c); + crypto_morus640_update(state, &m); + + memcpy(dst, tail.bytes, size); + } +} + +static void crypto_morus640_decrypt_chunk(struct morus640_state *state, u8 *dst, + const u8 *src, unsigned int size) +{ + struct morus640_block m; + + if (MORUS640_ALIGNED(src) && MORUS640_ALIGNED(dst)) { + while (size >= MORUS640_BLOCK_SIZE) { + crypto_morus640_load_a(&m, src); + crypto_morus640_core(state, &m); + crypto_morus640_store_a(dst, &m); + crypto_morus640_update(state, &m); + + src += MORUS640_BLOCK_SIZE; + dst += MORUS640_BLOCK_SIZE; + size -= MORUS640_BLOCK_SIZE; + } + } else { + while (size >= MORUS640_BLOCK_SIZE) { + crypto_morus640_load_u(&m, src); + crypto_morus640_core(state, &m); + crypto_morus640_store_u(dst, &m); + crypto_morus640_update(state, &m); + + src += MORUS640_BLOCK_SIZE; + dst += MORUS640_BLOCK_SIZE; + size -= MORUS640_BLOCK_SIZE; + } + } + + if (size > 0) { + union morus640_block_in tail; + + memcpy(tail.bytes, src, size); + + crypto_morus640_load_a(&m, src); + crypto_morus640_core(state, &m); + crypto_morus640_store_a(tail.bytes, &m); + memset(tail.bytes + size, 0, MORUS640_BLOCK_SIZE - size); + crypto_morus640_load_a(&m, tail.bytes); + crypto_morus640_update(state, &m); + + memcpy(dst, tail.bytes, size); + } +} + +static void crypto_morus640_init(struct morus640_state *state, + const struct morus640_block *key, + const u8 *iv) +{ + static const struct morus640_block z = {}; + + unsigned int i; + + crypto_morus640_load(&state->s[0], iv); + state->s[1] = *key; + for (i = 0; i < MORUS_BLOCK_WORDS; i++) + state->s[2].words[i] = U32_C(0xFFFFFFFF); + state->s[3] = crypto_morus640_const[0]; + state->s[4] = crypto_morus640_const[1]; + + for (i = 0; i < 16; i++) + crypto_morus640_update(state, &z); + + for (i = 0; i < MORUS_BLOCK_WORDS; i++) + state->s[1].words[i] ^= key->words[i]; +} + +static void crypto_morus640_process_ad(struct morus640_state *state, + struct scatterlist *sg_src, + unsigned int assoclen) +{ + struct scatter_walk walk; + struct morus640_block m; + union morus640_block_in buf; + unsigned int pos = 0; + + scatterwalk_start(&walk, sg_src); + while (assoclen != 0) { + unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int left = size; + void *mapped = scatterwalk_map(&walk); + const u8 *src = (const u8 *)mapped; + + if (pos + size >= MORUS640_BLOCK_SIZE) { + if (pos > 0) { + unsigned int fill = MORUS640_BLOCK_SIZE - pos; + memcpy(buf.bytes + pos, src, fill); + + crypto_morus640_load_a(&m, buf.bytes); + crypto_morus640_update(state, &m); + + pos = 0; + left -= fill; + src += fill; + } + + crypto_morus640_ad(state, src, left); + src += left & ~(MORUS640_BLOCK_SIZE - 1); + left &= MORUS640_BLOCK_SIZE - 1; + } + + memcpy(buf.bytes + pos, src, left); + + pos += left; + assoclen -= size; + scatterwalk_unmap(mapped); + scatterwalk_advance(&walk, size); + scatterwalk_done(&walk, 0, assoclen); + } + + if (pos > 0) { + memset(buf.bytes + pos, 0, MORUS640_BLOCK_SIZE - pos); + + crypto_morus640_load_a(&m, buf.bytes); + crypto_morus640_update(state, &m); + } +} + +static void crypto_morus640_process_crypt(struct morus640_state *state, + struct aead_request *req, + const struct morus640_ops *ops) +{ + struct skcipher_walk walk; + u8 *dst; + const u8 *src; + + ops->skcipher_walk_init(&walk, req, false); + + while (walk.nbytes) { + src = walk.src.virt.addr; + dst = walk.dst.virt.addr; + + ops->crypt_chunk(state, dst, src, walk.nbytes); + + skcipher_walk_done(&walk, 0); + } +} + +static void crypto_morus640_final(struct morus640_state *state, + struct morus640_block *tag_xor, + u64 assoclen, u64 cryptlen) +{ + u64 assocbits = assoclen * 8; + u64 cryptbits = cryptlen * 8; + + u32 assocbits_lo = (u32)assocbits; + u32 assocbits_hi = (u32)(assocbits >> 32); + u32 cryptbits_lo = (u32)cryptbits; + u32 cryptbits_hi = (u32)(cryptbits >> 32); + + struct morus640_block tmp; + unsigned int i; + + tmp.words[0] = cpu_to_le32(assocbits_lo); + tmp.words[1] = cpu_to_le32(assocbits_hi); + tmp.words[2] = cpu_to_le32(cryptbits_lo); + tmp.words[3] = cpu_to_le32(cryptbits_hi); + + for (i = 0; i < MORUS_BLOCK_WORDS; i++) + state->s[4].words[i] ^= state->s[0].words[i]; + + for (i = 0; i < 10; i++) + crypto_morus640_update(state, &tmp); + + crypto_morus640_core(state, tag_xor); +} + +static int crypto_morus640_setkey(struct crypto_aead *aead, const u8 *key, + unsigned int keylen) +{ + struct morus640_ctx *ctx = crypto_aead_ctx(aead); + + if (keylen != MORUS640_BLOCK_SIZE) { + crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; + } + + crypto_morus640_load(&ctx->key, key); + return 0; +} + +static int crypto_morus640_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + return (authsize <= MORUS_MAX_AUTH_SIZE) ? 0 : -EINVAL; +} + +static void crypto_morus640_crypt(struct aead_request *req, + struct morus640_block *tag_xor, + unsigned int cryptlen, + const struct morus640_ops *ops) +{ + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct morus640_ctx *ctx = crypto_aead_ctx(tfm); + struct morus640_state state; + + crypto_morus640_init(&state, &ctx->key, req->iv); + crypto_morus640_process_ad(&state, req->src, req->assoclen); + crypto_morus640_process_crypt(&state, req, ops); + crypto_morus640_final(&state, tag_xor, req->assoclen, cryptlen); +} + +static int crypto_morus640_encrypt(struct aead_request *req) +{ + static const struct morus640_ops ops = { + .skcipher_walk_init = skcipher_walk_aead_encrypt, + .crypt_chunk = crypto_morus640_encrypt_chunk, + }; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct morus640_block tag = {}; + union morus640_block_in tag_out; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen; + + crypto_morus640_crypt(req, &tag, cryptlen, &ops); + crypto_morus640_store(tag_out.bytes, &tag); + + scatterwalk_map_and_copy(tag_out.bytes, req->dst, + req->assoclen + cryptlen, authsize, 1); + return 0; +} + +static int crypto_morus640_decrypt(struct aead_request *req) +{ + static const struct morus640_ops ops = { + .skcipher_walk_init = skcipher_walk_aead_decrypt, + .crypt_chunk = crypto_morus640_decrypt_chunk, + }; + static const u8 zeros[MORUS640_BLOCK_SIZE] = {}; + + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + union morus640_block_in tag_in; + struct morus640_block tag; + unsigned int authsize = crypto_aead_authsize(tfm); + unsigned int cryptlen = req->cryptlen - authsize; + + scatterwalk_map_and_copy(tag_in.bytes, req->src, + req->assoclen + cryptlen, authsize, 0); + + crypto_morus640_load(&tag, tag_in.bytes); + crypto_morus640_crypt(req, &tag, cryptlen, &ops); + crypto_morus640_store(tag_in.bytes, &tag); + + return crypto_memneq(tag_in.bytes, zeros, authsize) ? -EBADMSG : 0; +} + +static int crypto_morus640_init_tfm(struct crypto_aead *tfm) +{ + return 0; +} + +static void crypto_morus640_exit_tfm(struct crypto_aead *tfm) +{ +} + +static struct aead_alg crypto_morus640_alg = { + .setkey = crypto_morus640_setkey, + .setauthsize = crypto_morus640_setauthsize, + .encrypt = crypto_morus640_encrypt, + .decrypt = crypto_morus640_decrypt, + .init = crypto_morus640_init_tfm, + .exit = crypto_morus640_exit_tfm, + + .ivsize = MORUS_NONCE_SIZE, + .maxauthsize = MORUS_MAX_AUTH_SIZE, + .chunksize = MORUS640_BLOCK_SIZE, + + .base = { + .cra_flags = CRYPTO_ALG_TYPE_AEAD, + .cra_blocksize = 1, + .cra_ctxsize = sizeof(struct morus640_ctx), + .cra_alignmask = 0, + + .cra_priority = 100, + + .cra_name = "morus640", + .cra_driver_name = "morus640-generic", + + .cra_module = THIS_MODULE, + } +}; + +static int __init crypto_morus640_module_init(void) +{ + return crypto_register_aead(&crypto_morus640_alg); +} + +static void __exit crypto_morus640_module_exit(void) +{ + crypto_unregister_aead(&crypto_morus640_alg); +} + +module_init(crypto_morus640_module_init); +module_exit(crypto_morus640_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ondrej Mosnacek "); +MODULE_DESCRIPTION("MORUS-640 AEAD algorithm"); +MODULE_ALIAS_CRYPTO("morus640"); +MODULE_ALIAS_CRYPTO("morus640-generic"); diff --git a/crypto/pcbc.c b/crypto/pcbc.c index d9e45a958720..ef802f6e9642 100644 --- a/crypto/pcbc.c +++ b/crypto/pcbc.c @@ -14,6 +14,7 @@ * */ +#include #include #include #include @@ -72,7 +73,7 @@ static int crypto_pcbc_encrypt_inplace(struct skcipher_request *req, unsigned int nbytes = walk->nbytes; u8 *src = walk->src.virt.addr; u8 *iv = walk->iv; - u8 tmpbuf[bsize]; + u8 tmpbuf[MAX_CIPHER_BLOCKSIZE]; do { memcpy(tmpbuf, src, bsize); @@ -144,7 +145,7 @@ static int crypto_pcbc_decrypt_inplace(struct skcipher_request *req, unsigned int nbytes = walk->nbytes; u8 *src = walk->src.virt.addr; u8 *iv = walk->iv; - u8 tmpbuf[bsize] __aligned(__alignof__(u32)); + u8 tmpbuf[MAX_CIPHER_BLOCKSIZE] __aligned(__alignof__(u32)); do { memcpy(tmpbuf, src, bsize); diff --git a/crypto/proc.c b/crypto/proc.c index 822fcef6d91c..f4eb6139973e 100644 --- a/crypto/proc.c +++ b/crypto/proc.c @@ -94,21 +94,9 @@ static const struct seq_operations crypto_seq_ops = { .show = c_show }; -static int crypto_info_open(struct inode *inode, struct file *file) -{ - return seq_open(file, &crypto_seq_ops); -} - -static const struct file_operations proc_crypto_ops = { - .open = crypto_info_open, - .read = seq_read, - .llseek = seq_lseek, - .release = seq_release -}; - void __init crypto_init_proc(void) { - proc_create("crypto", 0, NULL, &proc_crypto_ops); + proc_create_seq("crypto", 0, NULL, &crypto_seq_ops); } void __exit crypto_exit_proc(void) diff --git a/crypto/rsa.c b/crypto/rsa.c index b067f3a93880..4167980c243d 100644 --- a/crypto/rsa.c +++ b/crypto/rsa.c @@ -215,7 +215,6 @@ static int rsa_verify(struct akcipher_request *req) goto err_free_m; } - ret = -ENOMEM; s = mpi_read_raw_from_sgl(req->src, req->src_len); if (!s) { ret = -ENOMEM; diff --git a/crypto/salsa20_generic.c b/crypto/salsa20_generic.c index 5074006a56c3..8c77bc78a09f 100644 --- a/crypto/salsa20_generic.c +++ b/crypto/salsa20_generic.c @@ -21,9 +21,17 @@ #include #include -#include #include +#define SALSA20_IV_SIZE 8 +#define SALSA20_MIN_KEY_SIZE 16 +#define SALSA20_MAX_KEY_SIZE 32 +#define SALSA20_BLOCK_SIZE 64 + +struct salsa20_ctx { + u32 initial_state[16]; +}; + static void salsa20_block(u32 *state, __le32 *stream) { u32 x[16]; @@ -93,16 +101,15 @@ static void salsa20_docrypt(u32 *state, u8 *dst, const u8 *src, } } -void crypto_salsa20_init(u32 *state, const struct salsa20_ctx *ctx, +static void salsa20_init(u32 *state, const struct salsa20_ctx *ctx, const u8 *iv) { memcpy(state, ctx->initial_state, sizeof(ctx->initial_state)); state[6] = get_unaligned_le32(iv + 0); state[7] = get_unaligned_le32(iv + 4); } -EXPORT_SYMBOL_GPL(crypto_salsa20_init); -int crypto_salsa20_setkey(struct crypto_skcipher *tfm, const u8 *key, +static int salsa20_setkey(struct crypto_skcipher *tfm, const u8 *key, unsigned int keysize) { static const char sigma[16] = "expand 32-byte k"; @@ -143,7 +150,6 @@ int crypto_salsa20_setkey(struct crypto_skcipher *tfm, const u8 *key, return 0; } -EXPORT_SYMBOL_GPL(crypto_salsa20_setkey); static int salsa20_crypt(struct skcipher_request *req) { @@ -155,7 +161,7 @@ static int salsa20_crypt(struct skcipher_request *req) err = skcipher_walk_virt(&walk, req, true); - crypto_salsa20_init(state, ctx, walk.iv); + salsa20_init(state, ctx, walk.iv); while (walk.nbytes > 0) { unsigned int nbytes = walk.nbytes; @@ -183,7 +189,7 @@ static struct skcipher_alg alg = { .max_keysize = SALSA20_MAX_KEY_SIZE, .ivsize = SALSA20_IV_SIZE, .chunksize = SALSA20_BLOCK_SIZE, - .setkey = crypto_salsa20_setkey, + .setkey = salsa20_setkey, .encrypt = salsa20_crypt, .decrypt = salsa20_crypt, }; diff --git a/crypto/sm4_generic.c b/crypto/sm4_generic.c index f537a2766c55..c18eebfd5edd 100644 --- a/crypto/sm4_generic.c +++ b/crypto/sm4_generic.c @@ -190,21 +190,23 @@ static void sm4_do_crypt(const u32 *rk, u32 *out, const u32 *in) /* encrypt a block of text */ -static void sm4_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +void crypto_sm4_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { const struct crypto_sm4_ctx *ctx = crypto_tfm_ctx(tfm); sm4_do_crypt(ctx->rkey_enc, (u32 *)out, (u32 *)in); } +EXPORT_SYMBOL_GPL(crypto_sm4_encrypt); /* decrypt a block of text */ -static void sm4_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) +void crypto_sm4_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in) { const struct crypto_sm4_ctx *ctx = crypto_tfm_ctx(tfm); sm4_do_crypt(ctx->rkey_dec, (u32 *)out, (u32 *)in); } +EXPORT_SYMBOL_GPL(crypto_sm4_decrypt); static struct crypto_alg sm4_alg = { .cra_name = "sm4", @@ -219,8 +221,8 @@ static struct crypto_alg sm4_alg = { .cia_min_keysize = SM4_KEY_SIZE, .cia_max_keysize = SM4_KEY_SIZE, .cia_setkey = crypto_sm4_set_key, - .cia_encrypt = sm4_encrypt, - .cia_decrypt = sm4_decrypt + .cia_encrypt = crypto_sm4_encrypt, + .cia_decrypt = crypto_sm4_decrypt } } }; diff --git a/crypto/tcrypt.c b/crypto/tcrypt.c index 51fe7c8744ae..d5bcdd905007 100644 --- a/crypto/tcrypt.c +++ b/crypto/tcrypt.c @@ -158,9 +158,9 @@ struct test_mb_aead_data { }; static int do_mult_aead_op(struct test_mb_aead_data *data, int enc, - u32 num_mb) + u32 num_mb, int *rc) { - int i, rc[num_mb], err = 0; + int i, err = 0; /* Fire up a bunch of concurrent requests */ for (i = 0; i < num_mb; i++) { @@ -188,18 +188,26 @@ static int test_mb_aead_jiffies(struct test_mb_aead_data *data, int enc, { unsigned long start, end; int bcount; - int ret; + int ret = 0; + int *rc; + + rc = kcalloc(num_mb, sizeof(*rc), GFP_KERNEL); + if (!rc) + return -ENOMEM; for (start = jiffies, end = start + secs * HZ, bcount = 0; time_before(jiffies, end); bcount++) { - ret = do_mult_aead_op(data, enc, num_mb); + ret = do_mult_aead_op(data, enc, num_mb, rc); if (ret) - return ret; + goto out; } pr_cont("%d operations in %d seconds (%ld bytes)\n", bcount * num_mb, secs, (long)bcount * blen * num_mb); - return 0; + +out: + kfree(rc); + return ret; } static int test_mb_aead_cycles(struct test_mb_aead_data *data, int enc, @@ -208,10 +216,15 @@ static int test_mb_aead_cycles(struct test_mb_aead_data *data, int enc, unsigned long cycles = 0; int ret = 0; int i; + int *rc; + + rc = kcalloc(num_mb, sizeof(*rc), GFP_KERNEL); + if (!rc) + return -ENOMEM; /* Warm-up run. */ for (i = 0; i < 4; i++) { - ret = do_mult_aead_op(data, enc, num_mb); + ret = do_mult_aead_op(data, enc, num_mb, rc); if (ret) goto out; } @@ -221,7 +234,7 @@ static int test_mb_aead_cycles(struct test_mb_aead_data *data, int enc, cycles_t start, end; start = get_cycles(); - ret = do_mult_aead_op(data, enc, num_mb); + ret = do_mult_aead_op(data, enc, num_mb, rc); end = get_cycles(); if (ret) @@ -230,11 +243,11 @@ static int test_mb_aead_cycles(struct test_mb_aead_data *data, int enc, cycles += end - start; } -out: - if (ret == 0) - pr_cont("1 operation in %lu cycles (%d bytes)\n", - (cycles + 4) / (8 * num_mb), blen); + pr_cont("1 operation in %lu cycles (%d bytes)\n", + (cycles + 4) / (8 * num_mb), blen); +out: + kfree(rc); return ret; } @@ -705,9 +718,10 @@ struct test_mb_ahash_data { char *xbuf[XBUFSIZE]; }; -static inline int do_mult_ahash_op(struct test_mb_ahash_data *data, u32 num_mb) +static inline int do_mult_ahash_op(struct test_mb_ahash_data *data, u32 num_mb, + int *rc) { - int i, rc[num_mb], err = 0; + int i, err = 0; /* Fire up a bunch of concurrent requests */ for (i = 0; i < num_mb; i++) @@ -731,18 +745,26 @@ static int test_mb_ahash_jiffies(struct test_mb_ahash_data *data, int blen, { unsigned long start, end; int bcount; - int ret; + int ret = 0; + int *rc; + + rc = kcalloc(num_mb, sizeof(*rc), GFP_KERNEL); + if (!rc) + return -ENOMEM; for (start = jiffies, end = start + secs * HZ, bcount = 0; time_before(jiffies, end); bcount++) { - ret = do_mult_ahash_op(data, num_mb); + ret = do_mult_ahash_op(data, num_mb, rc); if (ret) - return ret; + goto out; } pr_cont("%d operations in %d seconds (%ld bytes)\n", bcount * num_mb, secs, (long)bcount * blen * num_mb); - return 0; + +out: + kfree(rc); + return ret; } static int test_mb_ahash_cycles(struct test_mb_ahash_data *data, int blen, @@ -751,10 +773,15 @@ static int test_mb_ahash_cycles(struct test_mb_ahash_data *data, int blen, unsigned long cycles = 0; int ret = 0; int i; + int *rc; + + rc = kcalloc(num_mb, sizeof(*rc), GFP_KERNEL); + if (!rc) + return -ENOMEM; /* Warm-up run. */ for (i = 0; i < 4; i++) { - ret = do_mult_ahash_op(data, num_mb); + ret = do_mult_ahash_op(data, num_mb, rc); if (ret) goto out; } @@ -764,7 +791,7 @@ static int test_mb_ahash_cycles(struct test_mb_ahash_data *data, int blen, cycles_t start, end; start = get_cycles(); - ret = do_mult_ahash_op(data, num_mb); + ret = do_mult_ahash_op(data, num_mb, rc); end = get_cycles(); if (ret) @@ -773,11 +800,11 @@ static int test_mb_ahash_cycles(struct test_mb_ahash_data *data, int blen, cycles += end - start; } -out: - if (ret == 0) - pr_cont("1 operation in %lu cycles (%d bytes)\n", - (cycles + 4) / (8 * num_mb), blen); + pr_cont("1 operation in %lu cycles (%d bytes)\n", + (cycles + 4) / (8 * num_mb), blen); +out: + kfree(rc); return ret; } @@ -1118,9 +1145,9 @@ struct test_mb_skcipher_data { }; static int do_mult_acipher_op(struct test_mb_skcipher_data *data, int enc, - u32 num_mb) + u32 num_mb, int *rc) { - int i, rc[num_mb], err = 0; + int i, err = 0; /* Fire up a bunch of concurrent requests */ for (i = 0; i < num_mb; i++) { @@ -1148,18 +1175,26 @@ static int test_mb_acipher_jiffies(struct test_mb_skcipher_data *data, int enc, { unsigned long start, end; int bcount; - int ret; + int ret = 0; + int *rc; + + rc = kcalloc(num_mb, sizeof(*rc), GFP_KERNEL); + if (!rc) + return -ENOMEM; for (start = jiffies, end = start + secs * HZ, bcount = 0; time_before(jiffies, end); bcount++) { - ret = do_mult_acipher_op(data, enc, num_mb); + ret = do_mult_acipher_op(data, enc, num_mb, rc); if (ret) - return ret; + goto out; } pr_cont("%d operations in %d seconds (%ld bytes)\n", bcount * num_mb, secs, (long)bcount * blen * num_mb); - return 0; + +out: + kfree(rc); + return ret; } static int test_mb_acipher_cycles(struct test_mb_skcipher_data *data, int enc, @@ -1168,10 +1203,15 @@ static int test_mb_acipher_cycles(struct test_mb_skcipher_data *data, int enc, unsigned long cycles = 0; int ret = 0; int i; + int *rc; + + rc = kcalloc(num_mb, sizeof(*rc), GFP_KERNEL); + if (!rc) + return -ENOMEM; /* Warm-up run. */ for (i = 0; i < 4; i++) { - ret = do_mult_acipher_op(data, enc, num_mb); + ret = do_mult_acipher_op(data, enc, num_mb, rc); if (ret) goto out; } @@ -1181,7 +1221,7 @@ static int test_mb_acipher_cycles(struct test_mb_skcipher_data *data, int enc, cycles_t start, end; start = get_cycles(); - ret = do_mult_acipher_op(data, enc, num_mb); + ret = do_mult_acipher_op(data, enc, num_mb, rc); end = get_cycles(); if (ret) @@ -1190,11 +1230,11 @@ static int test_mb_acipher_cycles(struct test_mb_skcipher_data *data, int enc, cycles += end - start; } -out: - if (ret == 0) - pr_cont("1 operation in %lu cycles (%d bytes)\n", - (cycles + 4) / (8 * num_mb), blen); + pr_cont("1 operation in %lu cycles (%d bytes)\n", + (cycles + 4) / (8 * num_mb), blen); +out: + kfree(rc); return ret; } @@ -1606,7 +1646,7 @@ static inline int tcrypt_test(const char *alg) return ret; } -static int do_test(const char *alg, u32 type, u32 mask, int m) +static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb) { int i; int ret = 0; @@ -1621,7 +1661,7 @@ static int do_test(const char *alg, u32 type, u32 mask, int m) } for (i = 1; i < 200; i++) - ret += do_test(NULL, 0, 0, i); + ret += do_test(NULL, 0, 0, i, num_mb); break; case 1: @@ -1902,10 +1942,6 @@ static int do_test(const char *alg, u32 type, u32 mask, int m) ret += tcrypt_test("vmac(aes)"); break; - case 110: - ret += tcrypt_test("hmac(crc32)"); - break; - case 111: ret += tcrypt_test("hmac(sha3-224)"); break; @@ -2903,7 +2939,7 @@ static int __init tcrypt_mod_init(void) goto err_free_tv; } - err = do_test(alg, type, mask, mode); + err = do_test(alg, type, mask, mode, num_mb); if (err) { printk(KERN_ERR "tcrypt: one or more tests failed!\n"); diff --git a/crypto/testmgr.c b/crypto/testmgr.c index af4a01c5037b..11e45352fd0b 100644 --- a/crypto/testmgr.c +++ b/crypto/testmgr.c @@ -84,10 +84,8 @@ struct aead_test_suite { }; struct cipher_test_suite { - struct { - const struct cipher_testvec *vecs; - unsigned int count; - } enc, dec; + const struct cipher_testvec *vecs; + unsigned int count; }; struct comp_test_suite { @@ -605,7 +603,8 @@ static int __test_aead(struct crypto_aead *tfm, int enc, goto out_nooutbuf; /* avoid "the frame size is larger than 1024 bytes" compiler warning */ - sg = kmalloc(sizeof(*sg) * 8 * (diff_dst ? 4 : 2), GFP_KERNEL); + sg = kmalloc(array3_size(sizeof(*sg), 8, (diff_dst ? 4 : 2)), + GFP_KERNEL); if (!sg) goto out_nosg; sgout = &sg[16]; @@ -988,6 +987,7 @@ static int test_cipher(struct crypto_cipher *tfm, int enc, unsigned int i, j, k; char *q; const char *e; + const char *input, *result; void *data; char *xbuf[XBUFSIZE]; int ret = -ENOMEM; @@ -1008,14 +1008,16 @@ static int test_cipher(struct crypto_cipher *tfm, int enc, if (fips_enabled && template[i].fips_skip) continue; + input = enc ? template[i].ptext : template[i].ctext; + result = enc ? template[i].ctext : template[i].ptext; j++; ret = -EINVAL; - if (WARN_ON(template[i].ilen > PAGE_SIZE)) + if (WARN_ON(template[i].len > PAGE_SIZE)) goto out; data = xbuf[0]; - memcpy(data, template[i].input, template[i].ilen); + memcpy(data, input, template[i].len); crypto_cipher_clear_flags(tfm, ~0); if (template[i].wk) @@ -1031,7 +1033,7 @@ static int test_cipher(struct crypto_cipher *tfm, int enc, } else if (ret) continue; - for (k = 0; k < template[i].ilen; + for (k = 0; k < template[i].len; k += crypto_cipher_blocksize(tfm)) { if (enc) crypto_cipher_encrypt_one(tfm, data + k, @@ -1042,10 +1044,10 @@ static int test_cipher(struct crypto_cipher *tfm, int enc, } q = data; - if (memcmp(q, template[i].result, template[i].rlen)) { + if (memcmp(q, result, template[i].len)) { printk(KERN_ERR "alg: cipher: Test %d failed " "on %s for %s\n", j, e, algo); - hexdump(q, template[i].rlen); + hexdump(q, template[i].len); ret = -EINVAL; goto out; } @@ -1073,6 +1075,7 @@ static int __test_skcipher(struct crypto_skcipher *tfm, int enc, struct scatterlist sgout[8]; const char *e, *d; struct crypto_wait wait; + const char *input, *result; void *data; char iv[MAX_IVLEN]; char *xbuf[XBUFSIZE]; @@ -1116,19 +1119,21 @@ static int __test_skcipher(struct crypto_skcipher *tfm, int enc, if (fips_enabled && template[i].fips_skip) continue; - if (template[i].iv) + if (template[i].iv && !(template[i].generates_iv && enc)) memcpy(iv, template[i].iv, ivsize); else memset(iv, 0, MAX_IVLEN); + input = enc ? template[i].ptext : template[i].ctext; + result = enc ? template[i].ctext : template[i].ptext; j++; ret = -EINVAL; - if (WARN_ON(align_offset + template[i].ilen > PAGE_SIZE)) + if (WARN_ON(align_offset + template[i].len > PAGE_SIZE)) goto out; data = xbuf[0]; data += align_offset; - memcpy(data, template[i].input, template[i].ilen); + memcpy(data, input, template[i].len); crypto_skcipher_clear_flags(tfm, ~0); if (template[i].wk) @@ -1144,15 +1149,15 @@ static int __test_skcipher(struct crypto_skcipher *tfm, int enc, } else if (ret) continue; - sg_init_one(&sg[0], data, template[i].ilen); + sg_init_one(&sg[0], data, template[i].len); if (diff_dst) { data = xoutbuf[0]; data += align_offset; - sg_init_one(&sgout[0], data, template[i].ilen); + sg_init_one(&sgout[0], data, template[i].len); } skcipher_request_set_crypt(req, sg, (diff_dst) ? sgout : sg, - template[i].ilen, iv); + template[i].len, iv); ret = crypto_wait_req(enc ? crypto_skcipher_encrypt(req) : crypto_skcipher_decrypt(req), &wait); @@ -1163,17 +1168,16 @@ static int __test_skcipher(struct crypto_skcipher *tfm, int enc, } q = data; - if (memcmp(q, template[i].result, template[i].rlen)) { + if (memcmp(q, result, template[i].len)) { pr_err("alg: skcipher%s: Test %d failed (invalid result) on %s for %s\n", d, j, e, algo); - hexdump(q, template[i].rlen); + hexdump(q, template[i].len); ret = -EINVAL; goto out; } - if (template[i].iv_out && - memcmp(iv, template[i].iv_out, - crypto_skcipher_ivsize(tfm))) { + if (template[i].generates_iv && enc && + memcmp(iv, template[i].iv, crypto_skcipher_ivsize(tfm))) { pr_err("alg: skcipher%s: Test %d failed (invalid output IV) on %s for %s\n", d, j, e, algo); hexdump(iv, crypto_skcipher_ivsize(tfm)); @@ -1194,11 +1198,13 @@ static int __test_skcipher(struct crypto_skcipher *tfm, int enc, if (fips_enabled && template[i].fips_skip) continue; - if (template[i].iv) + if (template[i].iv && !(template[i].generates_iv && enc)) memcpy(iv, template[i].iv, ivsize); else memset(iv, 0, MAX_IVLEN); + input = enc ? template[i].ptext : template[i].ctext; + result = enc ? template[i].ctext : template[i].ptext; j++; crypto_skcipher_clear_flags(tfm, ~0); if (template[i].wk) @@ -1226,7 +1232,7 @@ static int __test_skcipher(struct crypto_skcipher *tfm, int enc, q = xbuf[IDX[k] >> PAGE_SHIFT] + offset_in_page(IDX[k]); - memcpy(q, template[i].input + temp, template[i].tap[k]); + memcpy(q, input + temp, template[i].tap[k]); if (offset_in_page(q) + template[i].tap[k] < PAGE_SIZE) q[template[i].tap[k]] = 0; @@ -1248,7 +1254,7 @@ static int __test_skcipher(struct crypto_skcipher *tfm, int enc, } skcipher_request_set_crypt(req, sg, (diff_dst) ? sgout : sg, - template[i].ilen, iv); + template[i].len, iv); ret = crypto_wait_req(enc ? crypto_skcipher_encrypt(req) : crypto_skcipher_decrypt(req), &wait); @@ -1269,8 +1275,7 @@ static int __test_skcipher(struct crypto_skcipher *tfm, int enc, q = xbuf[IDX[k] >> PAGE_SHIFT] + offset_in_page(IDX[k]); - if (memcmp(q, template[i].result + temp, - template[i].tap[k])) { + if (memcmp(q, result + temp, template[i].tap[k])) { pr_err("alg: skcipher%s: Chunk test %d failed on %s at page %u for %s\n", d, j, e, k, algo); hexdump(q, template[i].tap[k]); @@ -1342,19 +1347,30 @@ static int test_comp(struct crypto_comp *tfm, int ctcount, int dtcount) { const char *algo = crypto_tfm_alg_driver_name(crypto_comp_tfm(tfm)); + char *output, *decomp_output; unsigned int i; - char result[COMP_BUF_SIZE]; int ret; + output = kmalloc(COMP_BUF_SIZE, GFP_KERNEL); + if (!output) + return -ENOMEM; + + decomp_output = kmalloc(COMP_BUF_SIZE, GFP_KERNEL); + if (!decomp_output) { + kfree(output); + return -ENOMEM; + } + for (i = 0; i < ctcount; i++) { int ilen; unsigned int dlen = COMP_BUF_SIZE; - memset(result, 0, sizeof (result)); + memset(output, 0, sizeof(COMP_BUF_SIZE)); + memset(decomp_output, 0, sizeof(COMP_BUF_SIZE)); ilen = ctemplate[i].inlen; ret = crypto_comp_compress(tfm, ctemplate[i].input, - ilen, result, &dlen); + ilen, output, &dlen); if (ret) { printk(KERN_ERR "alg: comp: compression failed " "on test %d for %s: ret=%d\n", i + 1, algo, @@ -1362,7 +1378,17 @@ static int test_comp(struct crypto_comp *tfm, goto out; } - if (dlen != ctemplate[i].outlen) { + ilen = dlen; + dlen = COMP_BUF_SIZE; + ret = crypto_comp_decompress(tfm, output, + ilen, decomp_output, &dlen); + if (ret) { + pr_err("alg: comp: compression failed: decompress: on test %d for %s failed: ret=%d\n", + i + 1, algo, -ret); + goto out; + } + + if (dlen != ctemplate[i].inlen) { printk(KERN_ERR "alg: comp: Compression test %d " "failed for %s: output len = %d\n", i + 1, algo, dlen); @@ -1370,10 +1396,11 @@ static int test_comp(struct crypto_comp *tfm, goto out; } - if (memcmp(result, ctemplate[i].output, dlen)) { - printk(KERN_ERR "alg: comp: Compression test %d " - "failed for %s\n", i + 1, algo); - hexdump(result, dlen); + if (memcmp(decomp_output, ctemplate[i].input, + ctemplate[i].inlen)) { + pr_err("alg: comp: compression failed: output differs: on test %d for %s\n", + i + 1, algo); + hexdump(decomp_output, dlen); ret = -EINVAL; goto out; } @@ -1383,11 +1410,11 @@ static int test_comp(struct crypto_comp *tfm, int ilen; unsigned int dlen = COMP_BUF_SIZE; - memset(result, 0, sizeof (result)); + memset(decomp_output, 0, sizeof(COMP_BUF_SIZE)); ilen = dtemplate[i].inlen; ret = crypto_comp_decompress(tfm, dtemplate[i].input, - ilen, result, &dlen); + ilen, decomp_output, &dlen); if (ret) { printk(KERN_ERR "alg: comp: decompression failed " "on test %d for %s: ret=%d\n", i + 1, algo, @@ -1403,10 +1430,10 @@ static int test_comp(struct crypto_comp *tfm, goto out; } - if (memcmp(result, dtemplate[i].output, dlen)) { + if (memcmp(decomp_output, dtemplate[i].output, dlen)) { printk(KERN_ERR "alg: comp: Decompression test %d " "failed for %s\n", i + 1, algo); - hexdump(result, dlen); + hexdump(decomp_output, dlen); ret = -EINVAL; goto out; } @@ -1415,11 +1442,13 @@ static int test_comp(struct crypto_comp *tfm, ret = 0; out: + kfree(decomp_output); + kfree(output); return ret; } static int test_acomp(struct crypto_acomp *tfm, - const struct comp_testvec *ctemplate, + const struct comp_testvec *ctemplate, const struct comp_testvec *dtemplate, int ctcount, int dtcount) { @@ -1681,8 +1710,9 @@ out: static int alg_test_cipher(const struct alg_test_desc *desc, const char *driver, u32 type, u32 mask) { + const struct cipher_test_suite *suite = &desc->suite.cipher; struct crypto_cipher *tfm; - int err = 0; + int err; tfm = crypto_alloc_cipher(driver, type, mask); if (IS_ERR(tfm)) { @@ -1691,18 +1721,10 @@ static int alg_test_cipher(const struct alg_test_desc *desc, return PTR_ERR(tfm); } - if (desc->suite.cipher.enc.vecs) { - err = test_cipher(tfm, ENCRYPT, desc->suite.cipher.enc.vecs, - desc->suite.cipher.enc.count); - if (err) - goto out; - } - - if (desc->suite.cipher.dec.vecs) - err = test_cipher(tfm, DECRYPT, desc->suite.cipher.dec.vecs, - desc->suite.cipher.dec.count); + err = test_cipher(tfm, ENCRYPT, suite->vecs, suite->count); + if (!err) + err = test_cipher(tfm, DECRYPT, suite->vecs, suite->count); -out: crypto_free_cipher(tfm); return err; } @@ -1710,8 +1732,9 @@ out: static int alg_test_skcipher(const struct alg_test_desc *desc, const char *driver, u32 type, u32 mask) { + const struct cipher_test_suite *suite = &desc->suite.cipher; struct crypto_skcipher *tfm; - int err = 0; + int err; tfm = crypto_alloc_skcipher(driver, type, mask); if (IS_ERR(tfm)) { @@ -1720,18 +1743,10 @@ static int alg_test_skcipher(const struct alg_test_desc *desc, return PTR_ERR(tfm); } - if (desc->suite.cipher.enc.vecs) { - err = test_skcipher(tfm, ENCRYPT, desc->suite.cipher.enc.vecs, - desc->suite.cipher.enc.count); - if (err) - goto out; - } - - if (desc->suite.cipher.dec.vecs) - err = test_skcipher(tfm, DECRYPT, desc->suite.cipher.dec.vecs, - desc->suite.cipher.dec.count); + err = test_skcipher(tfm, ENCRYPT, suite->vecs, suite->count); + if (!err) + err = test_skcipher(tfm, DECRYPT, suite->vecs, suite->count); -out: crypto_free_skcipher(tfm); return err; } @@ -1774,8 +1789,9 @@ static int alg_test_comp(const struct alg_test_desc *desc, const char *driver, return err; } -static int alg_test_hash(const struct alg_test_desc *desc, const char *driver, - u32 type, u32 mask) +static int __alg_test_hash(const struct hash_testvec *template, + unsigned int tcount, const char *driver, + u32 type, u32 mask) { struct crypto_ahash *tfm; int err; @@ -1787,16 +1803,51 @@ static int alg_test_hash(const struct alg_test_desc *desc, const char *driver, return PTR_ERR(tfm); } - err = test_hash(tfm, desc->suite.hash.vecs, - desc->suite.hash.count, true); + err = test_hash(tfm, template, tcount, true); if (!err) - err = test_hash(tfm, desc->suite.hash.vecs, - desc->suite.hash.count, false); - + err = test_hash(tfm, template, tcount, false); crypto_free_ahash(tfm); return err; } +static int alg_test_hash(const struct alg_test_desc *desc, const char *driver, + u32 type, u32 mask) +{ + const struct hash_testvec *template = desc->suite.hash.vecs; + unsigned int tcount = desc->suite.hash.count; + unsigned int nr_unkeyed, nr_keyed; + int err; + + /* + * For OPTIONAL_KEY algorithms, we have to do all the unkeyed tests + * first, before setting a key on the tfm. To make this easier, we + * require that the unkeyed test vectors (if any) are listed first. + */ + + for (nr_unkeyed = 0; nr_unkeyed < tcount; nr_unkeyed++) { + if (template[nr_unkeyed].ksize) + break; + } + for (nr_keyed = 0; nr_unkeyed + nr_keyed < tcount; nr_keyed++) { + if (!template[nr_unkeyed + nr_keyed].ksize) { + pr_err("alg: hash: test vectors for %s out of order, " + "unkeyed ones must come first\n", desc->alg); + return -EINVAL; + } + } + + err = 0; + if (nr_unkeyed) { + err = __alg_test_hash(template, nr_unkeyed, driver, type, mask); + template += nr_unkeyed; + } + + if (!err && nr_keyed) + err = __alg_test_hash(template, nr_keyed, driver, type, mask); + + return err; +} + static int alg_test_crc32c(const struct alg_test_desc *desc, const char *driver, u32 type, u32 mask) { @@ -2316,6 +2367,33 @@ static int alg_test_null(const struct alg_test_desc *desc, /* Please keep this list sorted by algorithm name. */ static const struct alg_test_desc alg_test_descs[] = { { + .alg = "aegis128", + .test = alg_test_aead, + .suite = { + .aead = { + .enc = __VECS(aegis128_enc_tv_template), + .dec = __VECS(aegis128_dec_tv_template), + } + } + }, { + .alg = "aegis128l", + .test = alg_test_aead, + .suite = { + .aead = { + .enc = __VECS(aegis128l_enc_tv_template), + .dec = __VECS(aegis128l_dec_tv_template), + } + } + }, { + .alg = "aegis256", + .test = alg_test_aead, + .suite = { + .aead = { + .enc = __VECS(aegis256_enc_tv_template), + .dec = __VECS(aegis256_dec_tv_template), + } + } + }, { .alg = "ansi_cprng", .test = alg_test_cprng, .suite = { @@ -2488,93 +2566,70 @@ static const struct alg_test_desc alg_test_descs[] = { .test = alg_test_skcipher, .fips_allowed = 1, .suite = { - .cipher = { - .enc = __VECS(aes_cbc_enc_tv_template), - .dec = __VECS(aes_cbc_dec_tv_template) - } - } + .cipher = __VECS(aes_cbc_tv_template) + }, }, { .alg = "cbc(anubis)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(anubis_cbc_enc_tv_template), - .dec = __VECS(anubis_cbc_dec_tv_template) - } - } + .cipher = __VECS(anubis_cbc_tv_template) + }, }, { .alg = "cbc(blowfish)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(bf_cbc_enc_tv_template), - .dec = __VECS(bf_cbc_dec_tv_template) - } - } + .cipher = __VECS(bf_cbc_tv_template) + }, }, { .alg = "cbc(camellia)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(camellia_cbc_enc_tv_template), - .dec = __VECS(camellia_cbc_dec_tv_template) - } - } + .cipher = __VECS(camellia_cbc_tv_template) + }, }, { .alg = "cbc(cast5)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(cast5_cbc_enc_tv_template), - .dec = __VECS(cast5_cbc_dec_tv_template) - } - } + .cipher = __VECS(cast5_cbc_tv_template) + }, }, { .alg = "cbc(cast6)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(cast6_cbc_enc_tv_template), - .dec = __VECS(cast6_cbc_dec_tv_template) - } - } + .cipher = __VECS(cast6_cbc_tv_template) + }, }, { .alg = "cbc(des)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(des_cbc_enc_tv_template), - .dec = __VECS(des_cbc_dec_tv_template) - } - } + .cipher = __VECS(des_cbc_tv_template) + }, }, { .alg = "cbc(des3_ede)", .test = alg_test_skcipher, .fips_allowed = 1, .suite = { - .cipher = { - .enc = __VECS(des3_ede_cbc_enc_tv_template), - .dec = __VECS(des3_ede_cbc_dec_tv_template) - } - } + .cipher = __VECS(des3_ede_cbc_tv_template) + }, + }, { + /* Same as cbc(aes) except the key is stored in + * hardware secure memory which we reference by index + */ + .alg = "cbc(paes)", + .test = alg_test_null, + .fips_allowed = 1, }, { .alg = "cbc(serpent)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(serpent_cbc_enc_tv_template), - .dec = __VECS(serpent_cbc_dec_tv_template) - } - } + .cipher = __VECS(serpent_cbc_tv_template) + }, }, { .alg = "cbc(twofish)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(tf_cbc_enc_tv_template), - .dec = __VECS(tf_cbc_dec_tv_template) - } - } + .cipher = __VECS(tf_cbc_tv_template) + }, }, { .alg = "cbcmac(aes)", .fips_allowed = 1, @@ -2596,11 +2651,8 @@ static const struct alg_test_desc alg_test_descs[] = { .alg = "chacha20", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(chacha20_enc_tv_template), - .dec = __VECS(chacha20_enc_tv_template), - } - } + .cipher = __VECS(chacha20_tv_template) + }, }, { .alg = "cmac(aes)", .fips_allowed = 1, @@ -2643,92 +2695,69 @@ static const struct alg_test_desc alg_test_descs[] = { .test = alg_test_skcipher, .fips_allowed = 1, .suite = { - .cipher = { - .enc = __VECS(aes_ctr_enc_tv_template), - .dec = __VECS(aes_ctr_dec_tv_template) - } + .cipher = __VECS(aes_ctr_tv_template) } }, { .alg = "ctr(blowfish)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(bf_ctr_enc_tv_template), - .dec = __VECS(bf_ctr_dec_tv_template) - } + .cipher = __VECS(bf_ctr_tv_template) } }, { .alg = "ctr(camellia)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(camellia_ctr_enc_tv_template), - .dec = __VECS(camellia_ctr_dec_tv_template) - } + .cipher = __VECS(camellia_ctr_tv_template) } }, { .alg = "ctr(cast5)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(cast5_ctr_enc_tv_template), - .dec = __VECS(cast5_ctr_dec_tv_template) - } + .cipher = __VECS(cast5_ctr_tv_template) } }, { .alg = "ctr(cast6)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(cast6_ctr_enc_tv_template), - .dec = __VECS(cast6_ctr_dec_tv_template) - } + .cipher = __VECS(cast6_ctr_tv_template) } }, { .alg = "ctr(des)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(des_ctr_enc_tv_template), - .dec = __VECS(des_ctr_dec_tv_template) - } + .cipher = __VECS(des_ctr_tv_template) } }, { .alg = "ctr(des3_ede)", .test = alg_test_skcipher, .fips_allowed = 1, .suite = { - .cipher = { - .enc = __VECS(des3_ede_ctr_enc_tv_template), - .dec = __VECS(des3_ede_ctr_dec_tv_template) - } + .cipher = __VECS(des3_ede_ctr_tv_template) } + }, { + /* Same as ctr(aes) except the key is stored in + * hardware secure memory which we reference by index + */ + .alg = "ctr(paes)", + .test = alg_test_null, + .fips_allowed = 1, }, { .alg = "ctr(serpent)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(serpent_ctr_enc_tv_template), - .dec = __VECS(serpent_ctr_dec_tv_template) - } + .cipher = __VECS(serpent_ctr_tv_template) } }, { .alg = "ctr(twofish)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(tf_ctr_enc_tv_template), - .dec = __VECS(tf_ctr_dec_tv_template) - } + .cipher = __VECS(tf_ctr_tv_template) } }, { .alg = "cts(cbc(aes))", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(cts_mode_enc_tv_template), - .dec = __VECS(cts_mode_dec_tv_template) - } + .cipher = __VECS(cts_mode_tv_template) } }, { .alg = "deflate", @@ -2876,64 +2905,43 @@ static const struct alg_test_desc alg_test_descs[] = { .test = alg_test_skcipher, .fips_allowed = 1, .suite = { - .cipher = { - .enc = __VECS(aes_enc_tv_template), - .dec = __VECS(aes_dec_tv_template) - } + .cipher = __VECS(aes_tv_template) } }, { .alg = "ecb(anubis)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(anubis_enc_tv_template), - .dec = __VECS(anubis_dec_tv_template) - } + .cipher = __VECS(anubis_tv_template) } }, { .alg = "ecb(arc4)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(arc4_enc_tv_template), - .dec = __VECS(arc4_dec_tv_template) - } + .cipher = __VECS(arc4_tv_template) } }, { .alg = "ecb(blowfish)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(bf_enc_tv_template), - .dec = __VECS(bf_dec_tv_template) - } + .cipher = __VECS(bf_tv_template) } }, { .alg = "ecb(camellia)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(camellia_enc_tv_template), - .dec = __VECS(camellia_dec_tv_template) - } + .cipher = __VECS(camellia_tv_template) } }, { .alg = "ecb(cast5)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(cast5_enc_tv_template), - .dec = __VECS(cast5_dec_tv_template) - } + .cipher = __VECS(cast5_tv_template) } }, { .alg = "ecb(cast6)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(cast6_enc_tv_template), - .dec = __VECS(cast6_dec_tv_template) - } + .cipher = __VECS(cast6_tv_template) } }, { .alg = "ecb(cipher_null)", @@ -2943,134 +2951,96 @@ static const struct alg_test_desc alg_test_descs[] = { .alg = "ecb(des)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(des_enc_tv_template), - .dec = __VECS(des_dec_tv_template) - } + .cipher = __VECS(des_tv_template) } }, { .alg = "ecb(des3_ede)", .test = alg_test_skcipher, .fips_allowed = 1, .suite = { - .cipher = { - .enc = __VECS(des3_ede_enc_tv_template), - .dec = __VECS(des3_ede_dec_tv_template) - } + .cipher = __VECS(des3_ede_tv_template) } }, { .alg = "ecb(fcrypt)", .test = alg_test_skcipher, .suite = { .cipher = { - .enc = { - .vecs = fcrypt_pcbc_enc_tv_template, - .count = 1 - }, - .dec = { - .vecs = fcrypt_pcbc_dec_tv_template, - .count = 1 - } + .vecs = fcrypt_pcbc_tv_template, + .count = 1 } } }, { .alg = "ecb(khazad)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(khazad_enc_tv_template), - .dec = __VECS(khazad_dec_tv_template) - } + .cipher = __VECS(khazad_tv_template) } + }, { + /* Same as ecb(aes) except the key is stored in + * hardware secure memory which we reference by index + */ + .alg = "ecb(paes)", + .test = alg_test_null, + .fips_allowed = 1, }, { .alg = "ecb(seed)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(seed_enc_tv_template), - .dec = __VECS(seed_dec_tv_template) - } + .cipher = __VECS(seed_tv_template) } }, { .alg = "ecb(serpent)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(serpent_enc_tv_template), - .dec = __VECS(serpent_dec_tv_template) - } + .cipher = __VECS(serpent_tv_template) } }, { .alg = "ecb(sm4)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(sm4_enc_tv_template), - .dec = __VECS(sm4_dec_tv_template) - } + .cipher = __VECS(sm4_tv_template) } }, { .alg = "ecb(speck128)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(speck128_enc_tv_template), - .dec = __VECS(speck128_dec_tv_template) - } + .cipher = __VECS(speck128_tv_template) } }, { .alg = "ecb(speck64)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(speck64_enc_tv_template), - .dec = __VECS(speck64_dec_tv_template) - } + .cipher = __VECS(speck64_tv_template) } }, { .alg = "ecb(tea)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(tea_enc_tv_template), - .dec = __VECS(tea_dec_tv_template) - } + .cipher = __VECS(tea_tv_template) } }, { .alg = "ecb(tnepres)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(tnepres_enc_tv_template), - .dec = __VECS(tnepres_dec_tv_template) - } + .cipher = __VECS(tnepres_tv_template) } }, { .alg = "ecb(twofish)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(tf_enc_tv_template), - .dec = __VECS(tf_dec_tv_template) - } + .cipher = __VECS(tf_tv_template) } }, { .alg = "ecb(xeta)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(xeta_enc_tv_template), - .dec = __VECS(xeta_dec_tv_template) - } + .cipher = __VECS(xeta_tv_template) } }, { .alg = "ecb(xtea)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(xtea_enc_tv_template), - .dec = __VECS(xtea_dec_tv_template) - } + .cipher = __VECS(xtea_tv_template) } }, { .alg = "ecdh", @@ -3096,12 +3066,6 @@ static const struct alg_test_desc alg_test_descs[] = { .suite = { .hash = __VECS(ghash_tv_template) } - }, { - .alg = "hmac(crc32)", - .test = alg_test_hash, - .suite = { - .hash = __VECS(bfin_crc_tv_template) - } }, { .alg = "hmac(md5)", .test = alg_test_hash, @@ -3192,55 +3156,37 @@ static const struct alg_test_desc alg_test_descs[] = { .test = alg_test_skcipher, .fips_allowed = 1, .suite = { - .cipher = { - .enc = __VECS(aes_kw_enc_tv_template), - .dec = __VECS(aes_kw_dec_tv_template) - } + .cipher = __VECS(aes_kw_tv_template) } }, { .alg = "lrw(aes)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(aes_lrw_enc_tv_template), - .dec = __VECS(aes_lrw_dec_tv_template) - } + .cipher = __VECS(aes_lrw_tv_template) } }, { .alg = "lrw(camellia)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(camellia_lrw_enc_tv_template), - .dec = __VECS(camellia_lrw_dec_tv_template) - } + .cipher = __VECS(camellia_lrw_tv_template) } }, { .alg = "lrw(cast6)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(cast6_lrw_enc_tv_template), - .dec = __VECS(cast6_lrw_dec_tv_template) - } + .cipher = __VECS(cast6_lrw_tv_template) } }, { .alg = "lrw(serpent)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(serpent_lrw_enc_tv_template), - .dec = __VECS(serpent_lrw_dec_tv_template) - } + .cipher = __VECS(serpent_lrw_tv_template) } }, { .alg = "lrw(twofish)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(tf_lrw_enc_tv_template), - .dec = __VECS(tf_lrw_dec_tv_template) - } + .cipher = __VECS(tf_lrw_tv_template) } }, { .alg = "lz4", @@ -3290,24 +3236,43 @@ static const struct alg_test_desc alg_test_descs[] = { .suite = { .hash = __VECS(michael_mic_tv_template) } + }, { + .alg = "morus1280", + .test = alg_test_aead, + .suite = { + .aead = { + .enc = __VECS(morus1280_enc_tv_template), + .dec = __VECS(morus1280_dec_tv_template), + } + } + }, { + .alg = "morus640", + .test = alg_test_aead, + .suite = { + .aead = { + .enc = __VECS(morus640_enc_tv_template), + .dec = __VECS(morus640_dec_tv_template), + } + } }, { .alg = "ofb(aes)", .test = alg_test_skcipher, .fips_allowed = 1, .suite = { - .cipher = { - .enc = __VECS(aes_ofb_enc_tv_template), - .dec = __VECS(aes_ofb_dec_tv_template) - } + .cipher = __VECS(aes_ofb_tv_template) } + }, { + /* Same as ofb(aes) except the key is stored in + * hardware secure memory which we reference by index + */ + .alg = "ofb(paes)", + .test = alg_test_null, + .fips_allowed = 1, }, { .alg = "pcbc(fcrypt)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(fcrypt_pcbc_enc_tv_template), - .dec = __VECS(fcrypt_pcbc_dec_tv_template) - } + .cipher = __VECS(fcrypt_pcbc_tv_template) } }, { .alg = "pkcs1pad(rsa,sha224)", @@ -3339,10 +3304,7 @@ static const struct alg_test_desc alg_test_descs[] = { .test = alg_test_skcipher, .fips_allowed = 1, .suite = { - .cipher = { - .enc = __VECS(aes_ctr_rfc3686_enc_tv_template), - .dec = __VECS(aes_ctr_rfc3686_dec_tv_template) - } + .cipher = __VECS(aes_ctr_rfc3686_tv_template) } }, { .alg = "rfc4106(gcm(aes))", @@ -3426,9 +3388,7 @@ static const struct alg_test_desc alg_test_descs[] = { .alg = "salsa20", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(salsa20_stream_enc_tv_template) - } + .cipher = __VECS(salsa20_stream_tv_template) } }, { .alg = "sha1", @@ -3552,65 +3512,59 @@ static const struct alg_test_desc alg_test_descs[] = { .test = alg_test_skcipher, .fips_allowed = 1, .suite = { - .cipher = { - .enc = __VECS(aes_xts_enc_tv_template), - .dec = __VECS(aes_xts_dec_tv_template) - } + .cipher = __VECS(aes_xts_tv_template) } }, { .alg = "xts(camellia)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(camellia_xts_enc_tv_template), - .dec = __VECS(camellia_xts_dec_tv_template) - } + .cipher = __VECS(camellia_xts_tv_template) } }, { .alg = "xts(cast6)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(cast6_xts_enc_tv_template), - .dec = __VECS(cast6_xts_dec_tv_template) - } + .cipher = __VECS(cast6_xts_tv_template) } + }, { + /* Same as xts(aes) except the key is stored in + * hardware secure memory which we reference by index + */ + .alg = "xts(paes)", + .test = alg_test_null, + .fips_allowed = 1, }, { .alg = "xts(serpent)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(serpent_xts_enc_tv_template), - .dec = __VECS(serpent_xts_dec_tv_template) - } + .cipher = __VECS(serpent_xts_tv_template) } }, { .alg = "xts(speck128)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(speck128_xts_enc_tv_template), - .dec = __VECS(speck128_xts_dec_tv_template) - } + .cipher = __VECS(speck128_xts_tv_template) } }, { .alg = "xts(speck64)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(speck64_xts_enc_tv_template), - .dec = __VECS(speck64_xts_dec_tv_template) - } + .cipher = __VECS(speck64_xts_tv_template) } }, { .alg = "xts(twofish)", .test = alg_test_skcipher, .suite = { - .cipher = { - .enc = __VECS(tf_xts_enc_tv_template), - .dec = __VECS(tf_xts_dec_tv_template) - } + .cipher = __VECS(tf_xts_tv_template) } + }, { + .alg = "xts4096(paes)", + .test = alg_test_null, + .fips_allowed = 1, + }, { + .alg = "xts512(paes)", + .test = alg_test_null, + .fips_allowed = 1, }, { .alg = "zlib-deflate", .test = alg_test_comp, @@ -3621,6 +3575,16 @@ static const struct alg_test_desc alg_test_descs[] = { .decomp = __VECS(zlib_deflate_decomp_tv_template) } } + }, { + .alg = "zstd", + .test = alg_test_comp, + .fips_allowed = 1, + .suite = { + .comp = { + .comp = __VECS(zstd_comp_tv_template), + .decomp = __VECS(zstd_decomp_tv_template) + } + } } }; diff --git a/crypto/testmgr.h b/crypto/testmgr.h index 004c0a0f8004..b950aa234e43 100644 --- a/crypto/testmgr.h +++ b/crypto/testmgr.h @@ -44,14 +44,13 @@ struct hash_testvec { }; /* - * cipher_testvec: structure to describe a cipher test - * @key: A pointer to a key used by the test - * @klen: The length of @key - * @iv: A pointer to the IV used by the test - * @input: A pointer to data used as input - * @ilen The length of data in @input - * @result: A pointer to what the test need to produce - * @rlen: The length of data in @result + * cipher_testvec: structure to describe a symmetric cipher test + * @key: Pointer to key + * @klen: Length of @key in bytes + * @iv: Pointer to IV (optional for some ciphers) + * @ptext: Pointer to plaintext + * @ctext: Pointer to ciphertext + * @len: Length of @ptext and @ctext in bytes * @fail: If set to one, the test need to fail * @wk: Does the test need CRYPTO_TFM_REQ_WEAK_KEY * ( e.g. test needs to fail due to a weak key ) @@ -60,23 +59,23 @@ struct hash_testvec { * @also_non_np: if set to 1, the test will be also done without * splitting data in @np SGs * @fips_skip: Skip the test vector in FIPS mode + * @generates_iv: Encryption should ignore the given IV, and output @iv. + * Decryption takes @iv. Needed for AES Keywrap ("kw(aes)"). */ - struct cipher_testvec { const char *key; const char *iv; - const char *iv_out; - const char *input; - const char *result; + const char *ptext; + const char *ctext; unsigned short tap[MAX_TAP]; int np; unsigned char also_non_np; bool fail; unsigned char wk; /* weak key flag */ unsigned char klen; - unsigned short ilen; - unsigned short rlen; + unsigned short len; bool fips_skip; + bool generates_iv; }; struct aead_testvec { @@ -5542,111 +5541,121 @@ static const struct hash_testvec poly1305_tv_template[] = { /* * DES test vectors. */ -static const struct cipher_testvec des_enc_tv_template[] = { +static const struct cipher_testvec des_tv_template[] = { { /* From Applied Cryptography */ .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xe7", - .ilen = 8, - .result = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d", - .rlen = 8, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xe7", + .ctext = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d", + .len = 8, }, { /* Same key, different plaintext block */ .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, - .input = "\x22\x33\x44\x55\x66\x77\x88\x99", - .ilen = 8, - .result = "\xf7\x9c\x89\x2a\x33\x8f\x4a\x8b", - .rlen = 8, + .ptext = "\x22\x33\x44\x55\x66\x77\x88\x99", + .ctext = "\xf7\x9c\x89\x2a\x33\x8f\x4a\x8b", + .len = 8, }, { /* Sbox test from NBS */ .key = "\x7c\xa1\x10\x45\x4a\x1a\x6e\x57", .klen = 8, - .input = "\x01\xa1\xd6\xd0\x39\x77\x67\x42", - .ilen = 8, - .result = "\x69\x0f\x5b\x0d\x9a\x26\x93\x9b", - .rlen = 8, + .ptext = "\x01\xa1\xd6\xd0\x39\x77\x67\x42", + .ctext = "\x69\x0f\x5b\x0d\x9a\x26\x93\x9b", + .len = 8, }, { /* Three blocks */ .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xe7" + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xe7" "\x22\x33\x44\x55\x66\x77\x88\x99" "\xca\xfe\xba\xbe\xfe\xed\xbe\xef", - .ilen = 24, - .result = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" + .ctext = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" "\xf7\x9c\x89\x2a\x33\x8f\x4a\x8b" "\xb4\x99\x26\xf7\x1f\xe1\xd4\x90", - .rlen = 24, + .len = 24, }, { /* Weak key */ .fail = true, .wk = 1, .key = "\x01\x01\x01\x01\x01\x01\x01\x01", .klen = 8, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xe7", - .ilen = 8, - .result = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d", - .rlen = 8, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xe7", + .ctext = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d", + .len = 8, }, { /* Two blocks -- for testing encryption across pages */ .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xe7" + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xe7" "\x22\x33\x44\x55\x66\x77\x88\x99", - .ilen = 16, - .result = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" + .ctext = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" "\xf7\x9c\x89\x2a\x33\x8f\x4a\x8b", - .rlen = 16, + .len = 16, + .np = 2, + .tap = { 8, 8 } + }, { + .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .klen = 8, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xe7" + "\xa3\x99\x7b\xca\xaf\x69\xa0\xf5", + .ctext = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" + "\x69\x0f\x5b\x0d\x9a\x26\x93\x9b", + .len = 16, .np = 2, .tap = { 8, 8 } + }, { + .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .klen = 8, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xe7" + "\xa3\x99\x7b\xca\xaf\x69\xa0\xf5", + .ctext = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" + "\x69\x0f\x5b\x0d\x9a\x26\x93\x9b", + .len = 16, + .np = 3, + .tap = { 3, 12, 1 } }, { /* Four blocks -- for testing encryption with chunking */ .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xe7" + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xe7" "\x22\x33\x44\x55\x66\x77\x88\x99" "\xca\xfe\xba\xbe\xfe\xed\xbe\xef" "\x22\x33\x44\x55\x66\x77\x88\x99", - .ilen = 32, - .result = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" + .ctext = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" "\xf7\x9c\x89\x2a\x33\x8f\x4a\x8b" "\xb4\x99\x26\xf7\x1f\xe1\xd4\x90" "\xf7\x9c\x89\x2a\x33\x8f\x4a\x8b", - .rlen = 32, + .len = 32, .np = 3, .tap = { 14, 10, 8 } }, { .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xe7" + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xe7" "\x22\x33\x44\x55\x66\x77\x88\x99" "\xca\xfe\xba\xbe\xfe\xed\xbe\xef", - .ilen = 24, - .result = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" + .ctext = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" "\xf7\x9c\x89\x2a\x33\x8f\x4a\x8b" "\xb4\x99\x26\xf7\x1f\xe1\xd4\x90", - .rlen = 24, + .len = 24, .np = 4, .tap = { 2, 1, 3, 18 } }, { .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xe7" + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xe7" "\x22\x33\x44\x55\x66\x77\x88\x99", - .ilen = 16, - .result = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" + .ctext = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" "\xf7\x9c\x89\x2a\x33\x8f\x4a\x8b", - .rlen = 16, + .len = 16, .np = 5, .tap = { 2, 2, 2, 2, 8 } }, { .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xe7", - .ilen = 8, - .result = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d", - .rlen = 8, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xe7", + .ctext = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d", + .len = 8, .np = 8, .tap = { 1, 1, 1, 1, 1, 1, 1, 1 } }, { /* Generated with Crypto++ */ .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55", .klen = 8, - .input = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" + .ptext = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" @@ -5677,86 +5686,7 @@ static const struct cipher_testvec des_enc_tv_template[] = { "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB", - .ilen = 248, - .result = "\x88\xCB\x1F\xAB\x2F\x2A\x49\x57" - "\x92\xB9\x77\xFF\x2F\x47\x58\xDD" - "\xD7\x8A\x91\x95\x26\x33\x78\xB2" - "\x33\xBA\xB2\x3E\x02\xF5\x1F\xEF" - "\x98\xC5\xA6\xD2\x7D\x79\xEC\xB3" - "\x45\xF3\x4C\x61\xAC\x6C\xC2\x55" - "\xE5\xD3\x06\x58\x8A\x42\x3E\xDD" - "\x3D\x20\x45\xE9\x6F\x0D\x25\xA8" - "\xA5\xC7\x69\xCE\xD5\x3B\x7B\xC9" - "\x9E\x65\xE7\xA3\xF2\xE4\x18\x94" - "\xD2\x81\xE9\x33\x2B\x2D\x49\xC4" - "\xFE\xDA\x7F\xE2\xF2\x8C\x9C\xDC" - "\x73\x58\x11\x1F\x81\xD7\x21\x1A" - "\x80\xD0\x0D\xE8\x45\xD6\xD8\xD5" - "\x2E\x51\x16\xCA\x09\x89\x54\x62" - "\xF7\x04\x3D\x75\xB9\xA3\x84\xF4" - "\x62\xF0\x02\x58\x83\xAF\x30\x87" - "\x85\x3F\x01\xCD\x8E\x58\x42\xC4" - "\x41\x73\xE0\x15\x0A\xE6\x2E\x80" - "\x94\xF8\x5B\x3A\x4E\xDF\x51\xB2" - "\x9D\xE4\xC4\x9D\xF7\x3F\xF8\x8E" - "\x37\x22\x4D\x00\x2A\xEF\xC1\x0F" - "\x14\xA0\x66\xAB\x79\x39\xD0\x8E" - "\xE9\x95\x61\x74\x12\xED\x07\xD7" - "\xDD\x95\xDC\x7B\x57\x25\x27\x9C" - "\x51\x96\x16\xF7\x94\x61\xB8\x87" - "\xF0\x21\x1B\x32\xFB\x07\x0F\x29" - "\x56\xBD\x9D\x22\xA2\x9F\xA2\xB9" - "\x46\x31\x4C\x5E\x2E\x95\x61\xEF" - "\xE1\x58\x39\x09\xB4\x8B\x40\xAC" - "\x5F\x62\xC7\x72\xD9\xFC\xCB\x9A", - .rlen = 248, - .also_non_np = 1, - .np = 3, - .tap = { 248 - 10, 2, 8 }, - }, -}; - -static const struct cipher_testvec des_dec_tv_template[] = { - { /* From Applied Cryptography */ - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .input = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d", - .ilen = 8, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xe7", - .rlen = 8, - }, { /* Sbox test from NBS */ - .key = "\x7c\xa1\x10\x45\x4a\x1a\x6e\x57", - .klen = 8, - .input = "\x69\x0f\x5b\x0d\x9a\x26\x93\x9b", - .ilen = 8, - .result = "\x01\xa1\xd6\xd0\x39\x77\x67\x42", - .rlen = 8, - }, { /* Two blocks, for chunking test */ - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .input = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" - "\x69\x0f\x5b\x0d\x9a\x26\x93\x9b", - .ilen = 16, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xe7" - "\xa3\x99\x7b\xca\xaf\x69\xa0\xf5", - .rlen = 16, - .np = 2, - .tap = { 8, 8 } - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .input = "\xc9\x57\x44\x25\x6a\x5e\xd3\x1d" - "\x69\x0f\x5b\x0d\x9a\x26\x93\x9b", - .ilen = 16, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xe7" - "\xa3\x99\x7b\xca\xaf\x69\xa0\xf5", - .rlen = 16, - .np = 3, - .tap = { 3, 12, 1 } - }, { /* Generated with Crypto++ */ - .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55", - .klen = 8, - .input = "\x88\xCB\x1F\xAB\x2F\x2A\x49\x57" + .ctext = "\x88\xCB\x1F\xAB\x2F\x2A\x49\x57" "\x92\xB9\x77\xFF\x2F\x47\x58\xDD" "\xD7\x8A\x91\x95\x26\x33\x78\xB2" "\x33\xBA\xB2\x3E\x02\xF5\x1F\xEF" @@ -5787,102 +5717,68 @@ static const struct cipher_testvec des_dec_tv_template[] = { "\x46\x31\x4C\x5E\x2E\x95\x61\xEF" "\xE1\x58\x39\x09\xB4\x8B\x40\xAC" "\x5F\x62\xC7\x72\xD9\xFC\xCB\x9A", - .ilen = 248, - .result = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB", - .rlen = 248, + .len = 248, .also_non_np = 1, .np = 3, .tap = { 248 - 10, 2, 8 }, }, }; -static const struct cipher_testvec des_cbc_enc_tv_template[] = { +static const struct cipher_testvec des_cbc_tv_template[] = { { /* From OpenSSL */ .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, .iv = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .input = "\x37\x36\x35\x34\x33\x32\x31\x20" + .ptext = "\x37\x36\x35\x34\x33\x32\x31\x20" "\x4e\x6f\x77\x20\x69\x73\x20\x74" "\x68\x65\x20\x74\x69\x6d\x65\x20", - .ilen = 24, - .result = "\xcc\xd1\x73\xff\xab\x20\x39\xf4" + .ctext = "\xcc\xd1\x73\xff\xab\x20\x39\xf4" "\xac\xd8\xae\xfd\xdf\xd8\xa1\xeb" "\x46\x8e\x91\x15\x78\x88\xba\x68", - .rlen = 24, + .len = 24, }, { /* FIPS Pub 81 */ .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, .iv = "\x12\x34\x56\x78\x90\xab\xcd\xef", - .input = "\x4e\x6f\x77\x20\x69\x73\x20\x74", - .ilen = 8, - .result = "\xe5\xc7\xcd\xde\x87\x2b\xf2\x7c", - .rlen = 8, + .ptext = "\x4e\x6f\x77\x20\x69\x73\x20\x74", + .ctext = "\xe5\xc7\xcd\xde\x87\x2b\xf2\x7c", + .len = 8, }, { .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, .iv = "\xe5\xc7\xcd\xde\x87\x2b\xf2\x7c", - .input = "\x68\x65\x20\x74\x69\x6d\x65\x20", - .ilen = 8, - .result = "\x43\xe9\x34\x00\x8c\x38\x9c\x0f", - .rlen = 8, + .ptext = "\x68\x65\x20\x74\x69\x6d\x65\x20", + .ctext = "\x43\xe9\x34\x00\x8c\x38\x9c\x0f", + .len = 8, }, { .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, .iv = "\x43\xe9\x34\x00\x8c\x38\x9c\x0f", - .input = "\x66\x6f\x72\x20\x61\x6c\x6c\x20", - .ilen = 8, - .result = "\x68\x37\x88\x49\x9a\x7c\x05\xf6", - .rlen = 8, + .ptext = "\x66\x6f\x72\x20\x61\x6c\x6c\x20", + .ctext = "\x68\x37\x88\x49\x9a\x7c\x05\xf6", + .len = 8, + .np = 2, + .tap = { 4, 4 }, + .also_non_np = 1, }, { /* Copy of openssl vector for chunk testing */ /* From OpenSSL */ .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", .klen = 8, .iv = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .input = "\x37\x36\x35\x34\x33\x32\x31\x20" + .ptext = "\x37\x36\x35\x34\x33\x32\x31\x20" "\x4e\x6f\x77\x20\x69\x73\x20\x74" "\x68\x65\x20\x74\x69\x6d\x65\x20", - .ilen = 24, - .result = "\xcc\xd1\x73\xff\xab\x20\x39\xf4" + .ctext = "\xcc\xd1\x73\xff\xab\x20\x39\xf4" "\xac\xd8\xae\xfd\xdf\xd8\xa1\xeb" "\x46\x8e\x91\x15\x78\x88\xba\x68", - .rlen = 24, + .len = 24, .np = 2, .tap = { 13, 11 } }, { /* Generated with Crypto++ */ .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55", .klen = 8, .iv = "\xE7\x82\x1D\xB8\x53\x11\xAC\x47", - .input = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" + .ptext = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" @@ -5913,85 +5809,7 @@ static const struct cipher_testvec des_cbc_enc_tv_template[] = { "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB", - .ilen = 248, - .result = "\x71\xCC\x56\x1C\x87\x2C\x43\x20" - "\x1C\x20\x13\x09\xF9\x2B\x40\x47" - "\x99\x10\xD1\x1B\x65\x33\x33\xBA" - "\x88\x0D\xA2\xD1\x86\xFF\x4D\xF4" - "\x5A\x0C\x12\x96\x32\x57\xAA\x26" - "\xA7\xF4\x32\x8D\xBC\x10\x31\x9E" - "\x81\x72\x74\xDE\x30\x19\x69\x49" - "\x54\x9C\xC3\xEB\x0B\x97\xDD\xD1" - "\xE8\x6D\x0D\x05\x83\xA5\x12\x08" - "\x47\xF8\x88\x03\x86\x51\x3C\xEF" - "\xE7\x11\x73\x4D\x44\x2B\xE2\x16" - "\xE8\xA5\x06\x50\x66\x70\x0E\x14" - "\xBA\x21\x3B\xD5\x23\x5B\xA7\x8F" - "\x56\xB6\xA7\x44\xDB\x86\xAB\x69" - "\x33\x3C\xBE\x64\xC4\x22\xD3\xFE" - "\x49\x90\x88\x6A\x09\x8F\x76\x59" - "\xCB\xB7\xA0\x2D\x79\x75\x92\x8A" - "\x82\x1D\xC2\xFE\x09\x1F\x78\x6B" - "\x2F\xD6\xA4\x87\x1E\xC4\x53\x63" - "\x80\x02\x61\x2F\xE3\x46\xB6\xB5" - "\xAA\x95\xF4\xEE\xA7\x64\x2B\x4F" - "\x20\xCF\xD2\x47\x4E\x39\x65\xB3" - "\x11\x87\xA2\x6C\x49\x7E\x36\xC7" - "\x62\x8B\x48\x0D\x6A\x64\x00\xBD" - "\x71\x91\x8C\xE9\x70\x19\x01\x4F" - "\x4E\x68\x23\xBA\xDA\x24\x2E\x45" - "\x02\x14\x33\x21\xAE\x58\x4B\xCF" - "\x3B\x4B\xE8\xF8\xF6\x4F\x34\x93" - "\xD7\x07\x8A\xD7\x18\x92\x36\x8C" - "\x82\xA9\xBD\x6A\x31\x91\x39\x11" - "\xC6\x4A\xF3\x55\xC7\x29\x2E\x63", - .rlen = 248, - .also_non_np = 1, - .np = 3, - .tap = { 248 - 10, 2, 8 }, - }, -}; - -static const struct cipher_testvec des_cbc_dec_tv_template[] = { - { /* FIPS Pub 81 */ - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .iv = "\x12\x34\x56\x78\x90\xab\xcd\xef", - .input = "\xe5\xc7\xcd\xde\x87\x2b\xf2\x7c", - .ilen = 8, - .result = "\x4e\x6f\x77\x20\x69\x73\x20\x74", - .rlen = 8, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .iv = "\xe5\xc7\xcd\xde\x87\x2b\xf2\x7c", - .input = "\x43\xe9\x34\x00\x8c\x38\x9c\x0f", - .ilen = 8, - .result = "\x68\x65\x20\x74\x69\x6d\x65\x20", - .rlen = 8, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .iv = "\x43\xe9\x34\x00\x8c\x38\x9c\x0f", - .input = "\x68\x37\x88\x49\x9a\x7c\x05\xf6", - .ilen = 8, - .result = "\x66\x6f\x72\x20\x61\x6c\x6c\x20", - .rlen = 8, - }, { /* Copy of above, for chunk testing */ - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .iv = "\x43\xe9\x34\x00\x8c\x38\x9c\x0f", - .input = "\x68\x37\x88\x49\x9a\x7c\x05\xf6", - .ilen = 8, - .result = "\x66\x6f\x72\x20\x61\x6c\x6c\x20", - .rlen = 8, - .np = 2, - .tap = { 4, 4 } - }, { /* Generated with Crypto++ */ - .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55", - .klen = 8, - .iv = "\xE7\x82\x1D\xB8\x53\x11\xAC\x47", - .input = "\x71\xCC\x56\x1C\x87\x2C\x43\x20" + .ctext = "\x71\xCC\x56\x1C\x87\x2C\x43\x20" "\x1C\x20\x13\x09\xF9\x2B\x40\x47" "\x99\x10\xD1\x1B\x65\x33\x33\xBA" "\x88\x0D\xA2\xD1\x86\xFF\x4D\xF4" @@ -6022,51 +5840,19 @@ static const struct cipher_testvec des_cbc_dec_tv_template[] = { "\xD7\x07\x8A\xD7\x18\x92\x36\x8C" "\x82\xA9\xBD\x6A\x31\x91\x39\x11" "\xC6\x4A\xF3\x55\xC7\x29\x2E\x63", - .ilen = 248, - .result = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB", - .rlen = 248, + .len = 248, .also_non_np = 1, .np = 3, .tap = { 248 - 10, 2, 8 }, }, }; -static const struct cipher_testvec des_ctr_enc_tv_template[] = { +static const struct cipher_testvec des_ctr_tv_template[] = { { /* Generated with Crypto++ */ .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55", .klen = 8, .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" + .ptext = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" @@ -6097,8 +5883,7 @@ static const struct cipher_testvec des_ctr_enc_tv_template[] = { "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB", - .ilen = 248, - .result = "\x2F\x96\x06\x0F\x50\xC9\x68\x03" + .ctext = "\x2F\x96\x06\x0F\x50\xC9\x68\x03" "\x0F\x31\xD4\x64\xA5\x29\x77\x35" "\xBC\x7A\x9F\x19\xE7\x0D\x33\x3E" "\x12\x0B\x8C\xAE\x48\xAE\xD9\x02" @@ -6129,7 +5914,7 @@ static const struct cipher_testvec des_ctr_enc_tv_template[] = { "\x5C\xC4\x15\xC9\x9A\x21\xC5\xCD" "\x19\x7F\x99\x19\x53\xCE\x1D\x14" "\x69\x74\xA1\x06\x46\x0F\x4E\x75", - .rlen = 248, + .len = 248, .also_non_np = 1, .np = 3, .tap = { 248 - 10, 2, 8 }, @@ -6137,7 +5922,7 @@ static const struct cipher_testvec des_ctr_enc_tv_template[] = { .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55", .klen = 8, .iv = "\xE7\x82\x1D\xB8\x53\x11\xAC\x47", - .input = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" + .ptext = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" @@ -6168,8 +5953,7 @@ static const struct cipher_testvec des_ctr_enc_tv_template[] = { "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" "\xC6\x2F\xBB\x24\x8D\x19\x82", - .ilen = 247, - .result = "\x62\xE5\xF4\xDC\x99\xE7\x89\xE3" + .ctext = "\x62\xE5\xF4\xDC\x99\xE7\x89\xE3" "\xF4\x10\xCC\x21\x99\xEB\xDC\x15" "\x19\x13\x93\x27\x9D\xB6\x6F\x45" "\x17\x55\x61\x72\xC8\xD3\x7F\xA5" @@ -6200,193 +5984,44 @@ static const struct cipher_testvec des_ctr_enc_tv_template[] = { "\xDF\x85\x2D\xE1\xB2\xD6\xAB\x94" "\xA5\xA6\xE7\xB0\x51\x36\x52\x37" "\x91\x45\x05\x3E\x58\xBF\x32", - .rlen = 247, + .len = 247, .also_non_np = 1, .np = 2, .tap = { 247 - 8, 8 }, }, }; -static const struct cipher_testvec des_ctr_dec_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55", - .klen = 8, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x2F\x96\x06\x0F\x50\xC9\x68\x03" - "\x0F\x31\xD4\x64\xA5\x29\x77\x35" - "\xBC\x7A\x9F\x19\xE7\x0D\x33\x3E" - "\x12\x0B\x8C\xAE\x48\xAE\xD9\x02" - "\x0A\xD4\xB0\xD6\x37\xB2\x65\x1C" - "\x4B\x65\xEB\x24\xB5\x8E\xAD\x47" - "\x0D\xDA\x79\x77\xA0\x29\xA0\x2B" - "\xC8\x0F\x85\xDC\x03\x13\xA9\x04" - "\x19\x40\xBE\xBE\x5C\x49\x4A\x69" - "\xED\xE8\xE1\x9E\x14\x43\x74\xDE" - "\xEC\x6E\x11\x3F\x36\xEF\x7B\xFB" - "\xBE\x4C\x91\x43\x22\x65\x72\x48" - "\xE2\x12\xED\x88\xAC\xA7\xC9\x91" - "\x14\xA2\x36\x1C\x29\xFF\xC8\x4F" - "\x72\x5C\x4B\xB0\x1E\x93\xC2\xFA" - "\x9D\x53\x86\xA0\xAE\xC6\xB7\x3C" - "\x59\x0C\xD0\x8F\xA6\xD8\xA4\x31" - "\xB7\x30\x1C\x21\x38\xFB\x68\x8C" - "\x2E\xF5\x6E\x73\xC3\x16\x5F\x12" - "\x0C\x33\xB9\x1E\x7B\x70\xDE\x86" - "\x32\xB3\xC1\x16\xAB\xD9\x49\x0B" - "\x96\x28\x72\x6B\xF3\x30\xA9\xEB" - "\x69\xE2\x1E\x58\x46\xA2\x8E\xC7" - "\xC0\xEF\x07\xB7\x77\x2C\x00\x05" - "\x46\xBD\xFE\x53\x81\x8B\xA4\x03" - "\x20\x0F\xDB\x78\x0B\x1F\x53\x04" - "\x4C\x60\x4C\xC3\x2A\x86\x86\x7E" - "\x13\xD2\x26\xED\x5D\x3E\x9C\xF2" - "\x5C\xC4\x15\xC9\x9A\x21\xC5\xCD" - "\x19\x7F\x99\x19\x53\xCE\x1D\x14" - "\x69\x74\xA1\x06\x46\x0F\x4E\x75", - .ilen = 248, - .result = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB", - .rlen = 248, - .also_non_np = 1, - .np = 3, - .tap = { 248 - 10, 2, 8 }, - }, { /* Generated with Crypto++ */ - .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55", - .klen = 8, - .iv = "\xE7\x82\x1D\xB8\x53\x11\xAC\x47", - .input = "\x62\xE5\xF4\xDC\x99\xE7\x89\xE3" - "\xF4\x10\xCC\x21\x99\xEB\xDC\x15" - "\x19\x13\x93\x27\x9D\xB6\x6F\x45" - "\x17\x55\x61\x72\xC8\xD3\x7F\xA5" - "\x32\xD0\xD3\x02\x15\xA4\x05\x23" - "\x9C\x23\x61\x60\x77\x7B\x6C\x95" - "\x26\x49\x42\x2E\xF3\xC1\x8C\x6D" - "\xC8\x47\xD5\x94\xE7\x53\xC8\x23" - "\x1B\xA5\x0B\xCB\x12\xD3\x7A\x12" - "\xA4\x42\x15\x34\xF7\x5F\xDC\x58" - "\x5B\x58\x4C\xAD\xD1\x33\x8E\xE6" - "\xE5\xA0\xDA\x4D\x94\x3D\x63\xA8" - "\x02\x82\xBB\x16\xB8\xDC\xB5\x58" - "\xC3\x2D\x79\xE4\x25\x79\x43\xF9" - "\x6D\xD3\xCA\xC0\xE8\x12\xD4\x7E" - "\x04\x25\x79\xFD\x27\xFB\xC4\xEA" - "\x32\x94\x48\x92\xF3\x68\x1A\x7F" - "\x36\x33\x43\x79\xF7\xCA\xC2\x38" - "\xC0\x68\xD4\x53\xA9\xCC\x43\x0C" - "\x40\x57\x3E\xED\x00\x9F\x22\x6E" - "\x80\x99\x0B\xCC\x40\x63\x46\x8A" - "\xE8\xC4\x9B\x6D\x7A\x08\x6E\xA9" - "\x6F\x84\xBC\xB3\xF4\x95\x0B\x2D" - "\x6A\xBA\x37\x50\xC3\xCF\x9F\x7C" - "\x59\x5E\xDE\x0B\x30\xFA\x34\x8A" - "\xF8\xD1\xA2\xF8\x4E\xBD\x5D\x5E" - "\x7D\x71\x99\xE0\xF6\xE5\x7C\xE0" - "\x6D\xEE\x82\x89\x92\xD4\xF5\xD7" - "\xDF\x85\x2D\xE1\xB2\xD6\xAB\x94" - "\xA5\xA6\xE7\xB0\x51\x36\x52\x37" - "\x91\x45\x05\x3E\x58\xBF\x32", - .ilen = 247, - .result = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82", - .rlen = 247, - .also_non_np = 1, - .np = 2, - .tap = { 247 - 8, 8 }, - }, -}; - -static const struct cipher_testvec des3_ede_enc_tv_template[] = { +static const struct cipher_testvec des3_ede_tv_template[] = { { /* These are from openssl */ .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" "\x55\x55\x55\x55\x55\x55\x55\x55" "\xfe\xdc\xba\x98\x76\x54\x32\x10", .klen = 24, - .input = "\x73\x6f\x6d\x65\x64\x61\x74\x61", - .ilen = 8, - .result = "\x18\xd7\x48\xe5\x63\x62\x05\x72", - .rlen = 8, + .ptext = "\x73\x6f\x6d\x65\x64\x61\x74\x61", + .ctext = "\x18\xd7\x48\xe5\x63\x62\x05\x72", + .len = 8, }, { .key = "\x03\x52\x02\x07\x67\x20\x82\x17" "\x86\x02\x87\x66\x59\x08\x21\x98" "\x64\x05\x6a\xbd\xfe\xa9\x34\x57", .klen = 24, - .input = "\x73\x71\x75\x69\x67\x67\x6c\x65", - .ilen = 8, - .result = "\xc0\x7d\x2a\x0f\xa5\x66\xfa\x30", - .rlen = 8, + .ptext = "\x73\x71\x75\x69\x67\x67\x6c\x65", + .ctext = "\xc0\x7d\x2a\x0f\xa5\x66\xfa\x30", + .len = 8, }, { .key = "\x10\x46\x10\x34\x89\x98\x80\x20" "\x91\x07\xd0\x15\x89\x19\x01\x01" "\x19\x07\x92\x10\x98\x1a\x01\x01", .klen = 24, - .input = "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 8, - .result = "\xe1\xef\x62\xc3\x32\xfe\x82\x5b", - .rlen = 8, + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\xe1\xef\x62\xc3\x32\xfe\x82\x5b", + .len = 8, }, { /* Generated with Crypto++ */ .key = "\xF3\x9C\xD6\xF3\x9C\xB9\x5A\x67" "\x00\x5A\x67\x00\x2D\xCE\xEB\x2D" "\xCE\xEB\xB4\x51\x72\xB4\x51\x72", .klen = 24, - .input = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" + .ptext = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" "\x8B\x12\x86\x69\xF0\x5B\xCF\x56" "\x39\xAD\x34\x9F\x66\xEA\x7D\xC4" "\x48\xD3\xBA\x0D\xB1\x18\xE3\x4A" @@ -6448,8 +6083,7 @@ static const struct cipher_testvec des3_ede_enc_tv_template[] = { "\xFB\x42\xF6\x59\x20\x54\x3F\x86" "\x69\x9D\x64\xCF\x56\xDA\xAD\x34" "\xB8\x03\xEA\x7D\xE1\x48\xD3\x47", - .ilen = 496, - .result = "\x4E\x9A\x40\x3D\x61\x7D\x17\xFA" + .ctext = "\x4E\x9A\x40\x3D\x61\x7D\x17\xFA" "\x16\x86\x88\x0B\xD8\xAE\xF8\xE4" "\x81\x01\x04\x00\x76\xFA\xED\xD3" "\x44\x7E\x21\x9D\xF0\xFB\x2B\x64" @@ -6511,110 +6145,61 @@ static const struct cipher_testvec des3_ede_enc_tv_template[] = { "\x12\xE1\x71\x4A\xF9\x2A\xF5\xF6" "\x93\x03\xD7\x51\x09\xFA\xBE\x68" "\xD8\x45\xFF\x33\xBA\xBB\x2B\x63", - .rlen = 496, + .len = 496, .also_non_np = 1, .np = 3, .tap = { 496 - 20, 4, 16 }, }, }; -static const struct cipher_testvec des3_ede_dec_tv_template[] = { - { /* These are from openssl */ - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\x55\x55\x55\x55\x55\x55\x55\x55" - "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .klen = 24, - .input = "\x18\xd7\x48\xe5\x63\x62\x05\x72", - .ilen = 8, - .result = "\x73\x6f\x6d\x65\x64\x61\x74\x61", - .rlen = 8, - }, { - .key = "\x03\x52\x02\x07\x67\x20\x82\x17" - "\x86\x02\x87\x66\x59\x08\x21\x98" - "\x64\x05\x6a\xbd\xfe\xa9\x34\x57", - .klen = 24, - .input = "\xc0\x7d\x2a\x0f\xa5\x66\xfa\x30", - .ilen = 8, - .result = "\x73\x71\x75\x69\x67\x67\x6c\x65", - .rlen = 8, - }, { - .key = "\x10\x46\x10\x34\x89\x98\x80\x20" - "\x91\x07\xd0\x15\x89\x19\x01\x01" - "\x19\x07\x92\x10\x98\x1a\x01\x01", +static const struct cipher_testvec des3_ede_cbc_tv_template[] = { + { /* Generated from openssl */ + .key = "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" + "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" + "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", .klen = 24, - .input = "\xe1\xef\x62\xc3\x32\xfe\x82\x5b", - .ilen = 8, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 8, + .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .ptext = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" + "\x53\x20\x63\x65\x65\x72\x73\x74" + "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" + "\x20\x79\x65\x53\x72\x63\x74\x65" + "\x20\x73\x6f\x54\x20\x6f\x61\x4d" + "\x79\x6e\x53\x20\x63\x65\x65\x72" + "\x73\x74\x54\x20\x6f\x6f\x4d\x20" + "\x6e\x61\x20\x79\x65\x53\x72\x63" + "\x74\x65\x20\x73\x6f\x54\x20\x6f" + "\x61\x4d\x79\x6e\x53\x20\x63\x65" + "\x65\x72\x73\x74\x54\x20\x6f\x6f" + "\x4d\x20\x6e\x61\x20\x79\x65\x53" + "\x72\x63\x74\x65\x20\x73\x6f\x54" + "\x20\x6f\x61\x4d\x79\x6e\x53\x20" + "\x63\x65\x65\x72\x73\x74\x54\x20" + "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", + .ctext = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" + "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" + "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" + "\x12\x56\x5c\x53\x96\xb6\x00\x7d" + "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" + "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" + "\x76\xd1\xda\x0c\x94\x67\xbb\x04" + "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" + "\x22\x64\x47\xaa\x8f\x75\x13\xbf" + "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" + "\x71\x63\x2e\x89\x7b\x1e\x12\xca" + "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" + "\xd6\xf9\x21\x31\x62\x44\x45\xa6" + "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" + "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" + "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19", + .len = 128, }, { /* Generated with Crypto++ */ - .key = "\xF3\x9C\xD6\xF3\x9C\xB9\x5A\x67" - "\x00\x5A\x67\x00\x2D\xCE\xEB\x2D" - "\xCE\xEB\xB4\x51\x72\xB4\x51\x72", + .key = "\x9C\xD6\xF3\x9C\xB9\x5A\x67\x00" + "\x5A\x67\x00\x2D\xCE\xEB\x2D\xCE" + "\xEB\xB4\x51\x72\xB4\x51\x72\x1F", .klen = 24, - .input = "\x4E\x9A\x40\x3D\x61\x7D\x17\xFA" - "\x16\x86\x88\x0B\xD8\xAE\xF8\xE4" - "\x81\x01\x04\x00\x76\xFA\xED\xD3" - "\x44\x7E\x21\x9D\xF0\xFB\x2B\x64" - "\xCA\x4E\x90\xE0\xC0\x63\x28\x92" - "\xF3\x1F\xA4\x53\x2C\x77\xCC\x77" - "\x69\x56\xD0\x19\xAD\x00\x2D\x97" - "\xBC\xDE\x49\x6A\x82\xBC\x16\xE2" - "\x2F\x3E\x72\xEE\xD1\xCE\xFC\x1B" - "\xEA\x32\x56\xE4\x0B\xAF\x27\x36" - "\xAF\x08\xB9\x61\xB7\x48\x23\x27" - "\xEE\x4D\xC8\x79\x56\x06\xEB\xC7" - "\x5B\xCA\x0A\xC6\x5E\x5C\xCB\xB6" - "\x9D\xDA\x04\x59\xE2\x09\x48\x7E" - "\x6B\x37\xC6\xFE\x92\xA9\x1E\x6E" - "\x0D\x19\xFA\x33\x0F\xEE\x36\x68" - "\x11\xBB\xF9\x5A\x73\xAB\x3A\xEA" - "\xAC\x28\xD8\xD5\x27\xE8\x6B\x16" - "\x45\x86\x50\x01\x70\x35\x99\x92" - "\xDF\x0C\x07\x88\x8B\x7F\x9E\x4B" - "\xD2\x04\x84\x90\xC4\x27\xDF\x0A" - "\x49\xA8\xA7\x1A\x6D\x78\x16\xCA" - "\xB3\x18\x5C\xC3\x93\x63\x5A\x68" - "\x77\x02\xBA\xED\x62\x71\xB1\xD9" - "\x5E\xE5\x6F\x1A\xCC\x1D\xBE\x2E" - "\x11\xF3\xA6\x97\xCA\x8E\xBF\xB4" - "\x56\xA1\x36\x6B\xB1\x0A\x3E\x70" - "\xEA\xD7\xCD\x72\x7B\x79\xC8\xAD" - "\x6B\xFE\xFB\xBA\x64\xAE\x19\xC1" - "\x82\xCF\x8A\xA1\x50\x17\x7F\xB2" - "\x6F\x7B\x0F\x52\xC5\x3E\x4A\x52" - "\x3F\xD9\x3F\x01\xA6\x41\x1A\xB3" - "\xB3\x7A\x0E\x8E\x75\xB2\xB1\x5F" - "\xDB\xEA\x84\x13\x26\x6C\x85\x4E" - "\xAE\x6B\xDC\xE7\xE7\xAD\xB0\x06" - "\x5C\xBA\x92\xD0\x30\xBB\x8D\xD2" - "\xAE\x4C\x70\x85\xA0\x07\xE3\x2C" - "\xD1\x27\x9C\xCF\xDB\x13\xB7\xE5" - "\xF9\x6A\x02\xD0\x39\x9D\xB6\xE7" - "\xD1\x17\x25\x08\xF9\xA9\xA6\x67" - "\x38\x80\xD1\x22\xAB\x1A\xD7\x26" - "\xAD\xCA\x19\x1B\xFA\x18\xA7\x57" - "\x31\xEC\xC9\xED\xDB\x79\xC0\x48" - "\xAC\x31\x9F\x03\x8B\x62\x5B\x7E" - "\x0E\xA6\xD0\x64\xEE\xEA\x00\xFC" - "\x58\xC8\xDE\x51\x4E\x17\x15\x11" - "\x66\x58\xB6\x90\xDC\xDF\xA1\x49" - "\xCA\x79\xE9\x31\x31\x42\xDC\x56" - "\x0B\xCD\xB6\x0D\xC7\x64\xF7\x19" - "\xD9\x42\x05\x7F\xBC\x2F\xFC\x90" - "\xAE\x29\x86\xAA\x43\x7A\x4F\x6B" - "\xCE\xEA\xBC\x31\x8D\x65\x9D\x46" - "\xEA\x77\xB4\xF9\x58\xEA\x5D\x84" - "\xE4\xDC\x14\xBB\xBD\x15\x0E\xDA" - "\xD8\xE4\xA4\x5D\x61\xF9\x58\x0F" - "\xE4\x82\x77\xCE\x87\xC0\x09\xF0" - "\xD6\x10\x9E\x34\xE1\x0C\x67\x55" - "\x7B\x6D\xD5\x51\x4B\x00\xEE\xBA" - "\xF2\x7B\xBE\x75\x07\x42\x9D\x99" - "\x12\xE1\x71\x4A\xF9\x2A\xF5\xF6" - "\x93\x03\xD7\x51\x09\xFA\xBE\x68" - "\xD8\x45\xFF\x33\xBA\xBB\x2B\x63", - .ilen = 496, - .result = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" + .iv = "\xB2\xD7\x48\xED\x06\x44\xF9\x12" + "\xB7\x28\x4D\x83\x24\x59\xF2\x17", + .ptext = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" "\x8B\x12\x86\x69\xF0\x5B\xCF\x56" "\x39\xAD\x34\x9F\x66\xEA\x7D\xC4" "\x48\xD3\xBA\x0D\xB1\x18\xE3\x4A" @@ -6676,125 +6261,7 @@ static const struct cipher_testvec des3_ede_dec_tv_template[] = { "\xFB\x42\xF6\x59\x20\x54\x3F\x86" "\x69\x9D\x64\xCF\x56\xDA\xAD\x34" "\xB8\x03\xEA\x7D\xE1\x48\xD3\x47", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec des3_ede_cbc_enc_tv_template[] = { - { /* Generated from openssl */ - .key = "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" - "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" - "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", - .klen = 24, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .ilen = 128, - .result = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" - "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" - "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" - "\x12\x56\x5c\x53\x96\xb6\x00\x7d" - "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" - "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" - "\x76\xd1\xda\x0c\x94\x67\xbb\x04" - "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" - "\x22\x64\x47\xaa\x8f\x75\x13\xbf" - "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" - "\x71\x63\x2e\x89\x7b\x1e\x12\xca" - "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" - "\xd6\xf9\x21\x31\x62\x44\x45\xa6" - "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" - "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" - "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19", - .rlen = 128, - }, { /* Generated with Crypto++ */ - .key = "\x9C\xD6\xF3\x9C\xB9\x5A\x67\x00" - "\x5A\x67\x00\x2D\xCE\xEB\x2D\xCE" - "\xEB\xB4\x51\x72\xB4\x51\x72\x1F", - .klen = 24, - .iv = "\xB2\xD7\x48\xED\x06\x44\xF9\x12" - "\xB7\x28\x4D\x83\x24\x59\xF2\x17", - .input = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" - "\x8B\x12\x86\x69\xF0\x5B\xCF\x56" - "\x39\xAD\x34\x9F\x66\xEA\x7D\xC4" - "\x48\xD3\xBA\x0D\xB1\x18\xE3\x4A" - "\xFE\x41\x28\x5C\x27\x8E\x11\x85" - "\x6C\xF7\x5E\xC2\x55\x3C\xA0\x0B" - "\x92\x65\xE9\x70\xDB\x4F\xD6\xB9" - "\x00\xB4\x1F\xE6\x49\xFD\x44\x2F" - "\x53\x3A\x8D\x14\x98\x63\xCA\x5D" - "\xC1\xA8\x33\xA7\x0E\x91\x78\xEC" - "\x77\xDE\x42\xD5\xBC\x07\x8B\x12" - "\xE5\x4C\xF0\x5B\x22\x56\x39\x80" - "\x6B\x9F\x66\xC9\x50\xC4\xAF\x36" - "\xBA\x0D\x94\x7F\xE3\x4A\xDD\x41" - "\x28\xB3\x1A\x8E\x11\xF8\x43\xF7" - "\x5E\x21\x55\x3C\x87\x6E\x92\x65" - "\xCC\x57\xDB\xA2\x35\xB9\x00\xEB" - "\x72\xE6\x49\xD0\x44\x2F\xB6\x19" - "\x8D\x14\xFF\x46\xCA\x5D\x24\xA8" - "\x33\x9A\x6D\x91\x78\xC3\x77\xDE" - "\xA1\x08\xBC\x07\xEE\x71\xE5\x4C" - "\xD7\x5B\x22\xB5\x1C\x80\x6B\xF2" - "\x45\xC9\x50\x3B\xAF\x36\x99\x60" - "\x94\x7F\xC6\x4A\xDD\xA4\x0F\xB3" - "\x1A\xED\x74\xF8\x43\x2A\x5E\x21" - "\x88\x13\x87\x6E\xF1\x58\xCC\x57" - "\x3E\xA2\x35\x9C\x67\xEB\x72\xC5" - "\x49\xD0\xBB\x02\xB6\x19\xE0\x4B" - "\xFF\x46\x29\x5D\x24\x8F\x16\x9A" - "\x6D\xF4\x5F\xC3\xAA\x3D\xA1\x08" - "\x93\x7A\xEE\x71\xD8\x4C\xD7\xBE" - "\x01\xB5\x1C\xE7\x4E\xF2\x45\x2C" - "\x50\x3B\x82\x15\x99\x60\xCB\x52" - "\xC6\xA9\x30\xA4\x0F\x96\x79\xED" - "\x74\xDF\x43\x2A\xBD\x04\x88\x13" - "\xFA\x4D\xF1\x58\x23\x57\x3E\x81" - "\x68\x9C\x67\xCE\x51\xC5\xAC\x37" - "\xBB\x02\x95\x7C\xE0\x4B\xD2\x46" - "\x29\xB0\x1B\x8F\x16\xF9\x40\xF4" - "\x5F\x26\xAA\x3D\x84\x6F\x93\x7A" - "\xCD\x54\xD8\xA3\x0A\xBE\x01\xE8" - "\x73\xE7\x4E\xD1\x45\x2C\xB7\x1E" - "\x82\x15\xFC\x47\xCB\x52\x25\xA9" - "\x30\x9B\x62\x96\x79\xC0\x74\xDF" - "\xA6\x09\xBD\x04\xEF\x76\xFA\x4D" - "\xD4\x58\x23\x8A\x1D\x81\x68\xF3" - "\x5A\xCE\x51\x38\xAC\x37\x9E\x61" - "\x95\x7C\xC7\x4B\xD2\xA5\x0C\xB0" - "\x1B\xE2\x75\xF9\x40\x2B\x5F\x26" - "\x89\x10\x84\x6F\xF6\x59\xCD\x54" - "\x3F\xA3\x0A\x9D\x64\xE8\x73\xDA" - "\x4E\xD1\xB8\x03\xB7\x1E\xE1\x48" - "\xFC\x47\x2E\x52\x25\x8C\x17\x9B" - "\x62\xF5\x5C\xC0\xAB\x32\xA6\x09" - "\x90\x7B\xEF\x76\xD9\x4D\xD4\xBF" - "\x06\x8A\x1D\xE4\x4F\xF3\x5A\x2D" - "\x51\x38\x83\x6A\x9E\x61\xC8\x53" - "\xC7\xAE\x31\xA5\x0C\x97\x7E\xE2" - "\x75\xDC\x40\x2B\xB2\x05\x89\x10" - "\xFB\x42\xF6\x59\x20\x54\x3F\x86" - "\x69\x9D\x64\xCF\x56\xDA\xAD\x34" - "\xB8\x03\xEA\x7D\xE1\x48\xD3\x47", - .ilen = 496, - .result = "\xF8\xF6\xB5\x60\x5C\x5A\x75\x84" + .ctext = "\xF8\xF6\xB5\x60\x5C\x5A\x75\x84" "\x87\x81\x53\xBA\xC9\x6F\xEC\xD5" "\x1E\x68\x8E\x85\x12\x86\x1D\x38" "\x1C\x91\x40\xCC\x69\x6A\xD5\x35" @@ -6856,125 +6323,22 @@ static const struct cipher_testvec des3_ede_cbc_enc_tv_template[] = { "\x90\xE9\xFA\x4B\x00\x10\xAC\x58" "\x83\x70\xFF\x86\xE6\xAA\x0F\x1F" "\x95\x63\x73\xA2\x44\xAC\xF8\xA5", - .rlen = 496, + .len = 496, .also_non_np = 1, .np = 3, .tap = { 496 - 20, 4, 16 }, }, }; -static const struct cipher_testvec des3_ede_cbc_dec_tv_template[] = { - { /* Generated from openssl */ - .key = "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" - "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" - "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", - .klen = 24, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .input = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" - "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" - "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" - "\x12\x56\x5c\x53\x96\xb6\x00\x7d" - "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" - "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" - "\x76\xd1\xda\x0c\x94\x67\xbb\x04" - "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" - "\x22\x64\x47\xaa\x8f\x75\x13\xbf" - "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" - "\x71\x63\x2e\x89\x7b\x1e\x12\xca" - "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" - "\xd6\xf9\x21\x31\x62\x44\x45\xa6" - "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" - "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" - "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19", - .ilen = 128, - .result = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .rlen = 128, - }, { /* Generated with Crypto++ */ +static const struct cipher_testvec des3_ede_ctr_tv_template[] = { + { /* Generated with Crypto++ */ .key = "\x9C\xD6\xF3\x9C\xB9\x5A\x67\x00" "\x5A\x67\x00\x2D\xCE\xEB\x2D\xCE" "\xEB\xB4\x51\x72\xB4\x51\x72\x1F", .klen = 24, - .iv = "\xB2\xD7\x48\xED\x06\x44\xF9\x12" - "\xB7\x28\x4D\x83\x24\x59\xF2\x17", - .input = "\xF8\xF6\xB5\x60\x5C\x5A\x75\x84" - "\x87\x81\x53\xBA\xC9\x6F\xEC\xD5" - "\x1E\x68\x8E\x85\x12\x86\x1D\x38" - "\x1C\x91\x40\xCC\x69\x6A\xD5\x35" - "\x0D\x7C\xB5\x07\x7C\x7B\x2A\xAF" - "\x32\xBC\xA1\xB3\x84\x31\x1B\x3C" - "\x0A\x2B\xFA\xD3\x9F\xB0\x8C\x37" - "\x8F\x9D\xA7\x6D\x6C\xFA\xD7\x90" - "\xE3\x69\x54\xED\x3A\xC4\xF1\x6B" - "\xB1\xCC\xFB\x7D\xD8\x8E\x17\x0B" - "\x9C\xF6\x4C\xD6\xFF\x03\x4E\xD9" - "\xE6\xA5\xAD\x25\xE6\x17\x69\x63" - "\x11\x35\x61\x94\x88\x7B\x1C\x48" - "\xF1\x24\x20\x29\x6B\x93\x1A\x8E" - "\x43\x03\x89\xD8\xB1\xDA\x47\x7B" - "\x79\x3A\x83\x76\xDA\xAE\xC6\xBB" - "\x22\xF8\xE8\x3D\x9A\x65\x54\xD8" - "\x4C\xE9\xE7\xE4\x63\x2F\x5C\x73" - "\x5A\xC3\xAE\x46\xA8\xCD\x57\xE6" - "\x67\x88\xA5\x20\x6F\x5F\x97\xC7" - "\xCC\x15\xA2\x0A\x93\xEA\x33\xE7" - "\x03\x5F\xEC\x64\x30\x6F\xEE\xD7" - "\x7E\xDF\xD6\xE9\x6F\x3F\xD6\x1E" - "\xBE\x67\x6C\x5B\x97\xA0\x09\xE6" - "\xEE\xFE\x55\xA3\x29\x65\xE0\x12" - "\xA1\x6A\x8A\x6F\xF2\xE6\xF1\x96" - "\x87\xFB\x9C\x05\xDD\x80\xEC\xFF" - "\xC5\xED\x50\xFE\xFC\x91\xCD\xCE" - "\x25\x2C\x5F\xD9\xAD\x95\x7D\x99" - "\xF0\x05\xC4\x71\x46\x5F\xF9\x0D" - "\xD2\x63\xDF\x9B\x96\x2E\x2B\xA6" - "\x2B\x1C\xD5\xFB\x96\x24\x60\x60" - "\x54\x40\xB8\x62\xA4\xF8\x46\x95" - "\x73\x28\xA3\xA6\x16\x2B\x17\xE7" - "\x7A\xF8\x62\x54\x3B\x64\x69\xE1" - "\x71\x34\x29\x5B\x4E\x05\x9B\xFA" - "\x5E\xF1\x96\xB7\xCE\x16\x9B\x59" - "\xF1\x1A\x4C\x51\x26\xFD\x79\xE2" - "\x3B\x8E\x71\x69\x6A\x91\xB6\x65" - "\x32\x09\xB8\xE4\x09\x1F\xEA\x39" - "\xCE\x20\x65\x9F\xD6\xD1\xC7\xF0" - "\x73\x50\x08\x56\x20\x9B\x94\x23" - "\x14\x39\xB7\x2B\xB1\x2D\x6D\x6F" - "\x41\x5B\xCC\xE2\x18\xAE\x62\x89" - "\x78\x8E\x67\x23\xD0\xFB\x2B\xE5" - "\x25\xC9\x48\x97\xB5\xD3\x17\xD5" - "\x6A\x9F\xA7\x48\x0C\x2B\x73\x3B" - "\x57\x08\xAE\x91\xF2\xB7\x57\x89" - "\xF4\xD0\xB0\x07\xB0\x42\x6C\xAF" - "\x98\x1A\xE7\xD1\xAC\x1E\xB5\x02" - "\xD4\x56\x42\x79\x79\x7F\x2A\x77" - "\x25\xE9\x7D\xC1\x88\x19\x2B\x49" - "\x6F\x46\x59\xAB\x56\x1F\x61\xE0" - "\x0C\x24\x9C\xC9\x5B\x63\xA9\x12" - "\xCF\x88\x96\xB6\xA8\x24\xC6\xA8" - "\x21\x85\x1A\x62\x7E\x34\xBB\xEB" - "\xBD\x02\x2A\xC7\xD8\x89\x80\xC5" - "\xB1\xBB\x60\xA5\x22\xFC\x6F\x38" - "\x02\x80\xA3\x28\x22\x75\xE1\xE9" - "\x90\xE9\xFA\x4B\x00\x10\xAC\x58" - "\x83\x70\xFF\x86\xE6\xAA\x0F\x1F" - "\x95\x63\x73\xA2\x44\xAC\xF8\xA5", - .ilen = 496, - .result = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" + .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" + "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", + .ptext = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" "\x8B\x12\x86\x69\xF0\x5B\xCF\x56" "\x39\xAD\x34\x9F\x66\xEA\x7D\xC4" "\x48\xD3\xBA\x0D\xB1\x18\xE3\x4A" @@ -7036,147 +6400,69 @@ static const struct cipher_testvec des3_ede_cbc_dec_tv_template[] = { "\xFB\x42\xF6\x59\x20\x54\x3F\x86" "\x69\x9D\x64\xCF\x56\xDA\xAD\x34" "\xB8\x03\xEA\x7D\xE1\x48\xD3\x47", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec des3_ede_ctr_enc_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\x9C\xD6\xF3\x9C\xB9\x5A\x67\x00" - "\x5A\x67\x00\x2D\xCE\xEB\x2D\xCE" - "\xEB\xB4\x51\x72\xB4\x51\x72\x1F", - .klen = 24, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" - "\x8B\x12\x86\x69\xF0\x5B\xCF\x56" - "\x39\xAD\x34\x9F\x66\xEA\x7D\xC4" - "\x48\xD3\xBA\x0D\xB1\x18\xE3\x4A" - "\xFE\x41\x28\x5C\x27\x8E\x11\x85" - "\x6C\xF7\x5E\xC2\x55\x3C\xA0\x0B" - "\x92\x65\xE9\x70\xDB\x4F\xD6\xB9" - "\x00\xB4\x1F\xE6\x49\xFD\x44\x2F" - "\x53\x3A\x8D\x14\x98\x63\xCA\x5D" - "\xC1\xA8\x33\xA7\x0E\x91\x78\xEC" - "\x77\xDE\x42\xD5\xBC\x07\x8B\x12" - "\xE5\x4C\xF0\x5B\x22\x56\x39\x80" - "\x6B\x9F\x66\xC9\x50\xC4\xAF\x36" - "\xBA\x0D\x94\x7F\xE3\x4A\xDD\x41" - "\x28\xB3\x1A\x8E\x11\xF8\x43\xF7" - "\x5E\x21\x55\x3C\x87\x6E\x92\x65" - "\xCC\x57\xDB\xA2\x35\xB9\x00\xEB" - "\x72\xE6\x49\xD0\x44\x2F\xB6\x19" - "\x8D\x14\xFF\x46\xCA\x5D\x24\xA8" - "\x33\x9A\x6D\x91\x78\xC3\x77\xDE" - "\xA1\x08\xBC\x07\xEE\x71\xE5\x4C" - "\xD7\x5B\x22\xB5\x1C\x80\x6B\xF2" - "\x45\xC9\x50\x3B\xAF\x36\x99\x60" - "\x94\x7F\xC6\x4A\xDD\xA4\x0F\xB3" - "\x1A\xED\x74\xF8\x43\x2A\x5E\x21" - "\x88\x13\x87\x6E\xF1\x58\xCC\x57" - "\x3E\xA2\x35\x9C\x67\xEB\x72\xC5" - "\x49\xD0\xBB\x02\xB6\x19\xE0\x4B" - "\xFF\x46\x29\x5D\x24\x8F\x16\x9A" - "\x6D\xF4\x5F\xC3\xAA\x3D\xA1\x08" - "\x93\x7A\xEE\x71\xD8\x4C\xD7\xBE" - "\x01\xB5\x1C\xE7\x4E\xF2\x45\x2C" - "\x50\x3B\x82\x15\x99\x60\xCB\x52" - "\xC6\xA9\x30\xA4\x0F\x96\x79\xED" - "\x74\xDF\x43\x2A\xBD\x04\x88\x13" - "\xFA\x4D\xF1\x58\x23\x57\x3E\x81" - "\x68\x9C\x67\xCE\x51\xC5\xAC\x37" - "\xBB\x02\x95\x7C\xE0\x4B\xD2\x46" - "\x29\xB0\x1B\x8F\x16\xF9\x40\xF4" - "\x5F\x26\xAA\x3D\x84\x6F\x93\x7A" - "\xCD\x54\xD8\xA3\x0A\xBE\x01\xE8" - "\x73\xE7\x4E\xD1\x45\x2C\xB7\x1E" - "\x82\x15\xFC\x47\xCB\x52\x25\xA9" - "\x30\x9B\x62\x96\x79\xC0\x74\xDF" - "\xA6\x09\xBD\x04\xEF\x76\xFA\x4D" - "\xD4\x58\x23\x8A\x1D\x81\x68\xF3" - "\x5A\xCE\x51\x38\xAC\x37\x9E\x61" - "\x95\x7C\xC7\x4B\xD2\xA5\x0C\xB0" - "\x1B\xE2\x75\xF9\x40\x2B\x5F\x26" - "\x89\x10\x84\x6F\xF6\x59\xCD\x54" - "\x3F\xA3\x0A\x9D\x64\xE8\x73\xDA" - "\x4E\xD1\xB8\x03\xB7\x1E\xE1\x48" - "\xFC\x47\x2E\x52\x25\x8C\x17\x9B" - "\x62\xF5\x5C\xC0\xAB\x32\xA6\x09" - "\x90\x7B\xEF\x76\xD9\x4D\xD4\xBF" - "\x06\x8A\x1D\xE4\x4F\xF3\x5A\x2D" - "\x51\x38\x83\x6A\x9E\x61\xC8\x53" - "\xC7\xAE\x31\xA5\x0C\x97\x7E\xE2" - "\x75\xDC\x40\x2B\xB2\x05\x89\x10" - "\xFB\x42\xF6\x59\x20\x54\x3F\x86" - "\x69\x9D\x64\xCF\x56\xDA\xAD\x34" - "\xB8\x03\xEA\x7D\xE1\x48\xD3\x47", - .ilen = 496, - .result = "\x07\xC2\x08\x20\x72\x1F\x49\xEF" - "\x19\xCD\x6F\x32\x53\x05\x22\x15" - "\xA2\x85\x2B\xDB\x85\xD2\xD8\xB9" - "\xDD\x0D\x1B\x45\xCB\x69\x11\xD4" - "\xEA\xBE\xB2\x45\x5D\x0C\xAE\xBE" - "\xA0\xC1\x27\xAC\x65\x9F\x53\x7E" - "\xAF\xC2\x1B\xB5\xB8\x6D\x36\x0C" - "\x25\xC0\xF8\x6D\x0B\x29\x01\xDA" - "\x13\x78\xDC\x89\x12\x12\x43\xFA" - "\xF6\x12\xEF\x8D\x87\x62\x78\x83" - "\xE2\xBE\x41\x20\x4C\x6D\x35\x1B" - "\xD1\x0C\x30\xCF\xE2\xDE\x2B\x03" - "\xBF\x45\x73\xD4\xE5\x59\x95\xD1" - "\xB3\x9B\x27\x62\x97\xBD\xDE\x7F" - "\xA4\xD2\x39\x80\xAA\x50\x23\xF0" - "\x74\x88\x3D\xA8\x6A\x18\x79\x3B" - "\xC4\x96\x6C\x8D\x22\x40\x92\x6E" - "\xD6\xAD\x2A\x1F\xDE\x63\xC0\xE7" - "\x07\xF7\x2D\xF7\xB5\xF3\xF0\xCC" - "\x01\x7C\x2A\x9B\xC2\x10\xCA\xAA" - "\xFD\x2B\x3F\xC5\xF3\xF6\xFC\x9B" - "\x45\xDB\x53\xE4\x5B\xF3\xC9\x7B" - "\x8E\x52\xFF\xC8\x02\xB8\xAC\x9D" - "\xA1\x00\x39\xDA\x3D\x2D\x0E\x01" - "\x09\x7D\x8D\x5E\xBE\x53\xB9\xB0" - "\x8E\xE7\xE2\x96\x6A\xB2\x78\xEA" - "\xDE\x23\x8B\xA5\xFA\x5C\xE3\xDA" - "\xBF\x8E\x31\x6A\x55\xD1\x6A\xB2" - "\xB5\x46\x6F\xA5\xF0\xEE\xBA\x1F" - "\x9F\x98\xB0\x66\x4F\xD0\x3F\xA9" - "\xDF\x5F\x58\xC4\xF4\xFF\x75\x5C" - "\x40\x3A\x09\x7E\x6E\x1C\x97\xD4" - "\xCC\xE7\xE7\x71\xCF\x0B\x15\x08" - "\x71\xFA\x07\x97\xCD\xE6\xCA\x1D" - "\x14\x28\x0C\xCF\x99\x13\x7A\xF1" - "\xEB\xFA\xFA\x92\x07\xDE\x1D\xA1" - "\xD3\x36\x69\xFE\x51\x4D\x9F\x2E" - "\x83\x37\x4F\x1F\x48\x30\xED\x04" - "\x4D\xA4\xEF\x3A\xCA\x76\xF4\x1C" - "\x41\x8F\x63\x37\x78\x2F\x86\xA6" - "\xEF\x41\x7E\xD2\xAF\x88\xAB\x67" - "\x52\x71\xC3\x8E\xF8\x26\x93\x72" - "\xAA\xD6\x0E\xE7\x0B\x46\xB1\x3A" - "\xB4\x08\xA9\xA8\xA0\xCF\x20\x0C" - "\x52\xBC\x8B\x05\x56\xB2\xBC\x31" - "\x9B\x74\xB9\x29\x29\x96\x9A\x50" - "\xDC\x45\xDC\x1A\xEB\x0C\x64\xD4" - "\xD3\x05\x7E\x59\x55\xC3\xF4\x90" - "\xC2\xAB\xF8\x9B\x8A\xDA\xCE\xA1" - "\xC3\xF4\xAD\x77\xDD\x44\xC8\xAC" - "\xA3\xF1\xC9\xD2\x19\x5C\xB0\xCA" - "\xA2\x34\xC1\xF7\x6C\xFD\xAC\x65" - "\x32\xDC\x48\xC4\xF2\x00\x6B\x77" - "\xF1\x7D\x76\xAC\xC0\x31\x63\x2A" - "\xA5\x3A\x62\xC8\x91\xB1\x03\x65" - "\xCB\x43\xD1\x06\xDF\xC3\x67\xBC" - "\xDC\xE0\xCD\x35\xCE\x49\x65\xA0" - "\x52\x7B\xA7\x0D\x07\xA9\x1B\xB0" - "\x40\x77\x72\xC2\xEA\x0E\x3A\x78" - "\x46\xB9\x91\xB6\xE7\x3D\x51\x42" - "\xFD\x51\xB0\xC6\x2C\x63\x13\x78" - "\x5C\xEE\xFC\xCF\xC4\x70\x00\x34", - .rlen = 496, + .ctext = "\x07\xC2\x08\x20\x72\x1F\x49\xEF" + "\x19\xCD\x6F\x32\x53\x05\x22\x15" + "\xA2\x85\x2B\xDB\x85\xD2\xD8\xB9" + "\xDD\x0D\x1B\x45\xCB\x69\x11\xD4" + "\xEA\xBE\xB2\x45\x5D\x0C\xAE\xBE" + "\xA0\xC1\x27\xAC\x65\x9F\x53\x7E" + "\xAF\xC2\x1B\xB5\xB8\x6D\x36\x0C" + "\x25\xC0\xF8\x6D\x0B\x29\x01\xDA" + "\x13\x78\xDC\x89\x12\x12\x43\xFA" + "\xF6\x12\xEF\x8D\x87\x62\x78\x83" + "\xE2\xBE\x41\x20\x4C\x6D\x35\x1B" + "\xD1\x0C\x30\xCF\xE2\xDE\x2B\x03" + "\xBF\x45\x73\xD4\xE5\x59\x95\xD1" + "\xB3\x9B\x27\x62\x97\xBD\xDE\x7F" + "\xA4\xD2\x39\x80\xAA\x50\x23\xF0" + "\x74\x88\x3D\xA8\x6A\x18\x79\x3B" + "\xC4\x96\x6C\x8D\x22\x40\x92\x6E" + "\xD6\xAD\x2A\x1F\xDE\x63\xC0\xE7" + "\x07\xF7\x2D\xF7\xB5\xF3\xF0\xCC" + "\x01\x7C\x2A\x9B\xC2\x10\xCA\xAA" + "\xFD\x2B\x3F\xC5\xF3\xF6\xFC\x9B" + "\x45\xDB\x53\xE4\x5B\xF3\xC9\x7B" + "\x8E\x52\xFF\xC8\x02\xB8\xAC\x9D" + "\xA1\x00\x39\xDA\x3D\x2D\x0E\x01" + "\x09\x7D\x8D\x5E\xBE\x53\xB9\xB0" + "\x8E\xE7\xE2\x96\x6A\xB2\x78\xEA" + "\xDE\x23\x8B\xA5\xFA\x5C\xE3\xDA" + "\xBF\x8E\x31\x6A\x55\xD1\x6A\xB2" + "\xB5\x46\x6F\xA5\xF0\xEE\xBA\x1F" + "\x9F\x98\xB0\x66\x4F\xD0\x3F\xA9" + "\xDF\x5F\x58\xC4\xF4\xFF\x75\x5C" + "\x40\x3A\x09\x7E\x6E\x1C\x97\xD4" + "\xCC\xE7\xE7\x71\xCF\x0B\x15\x08" + "\x71\xFA\x07\x97\xCD\xE6\xCA\x1D" + "\x14\x28\x0C\xCF\x99\x13\x7A\xF1" + "\xEB\xFA\xFA\x92\x07\xDE\x1D\xA1" + "\xD3\x36\x69\xFE\x51\x4D\x9F\x2E" + "\x83\x37\x4F\x1F\x48\x30\xED\x04" + "\x4D\xA4\xEF\x3A\xCA\x76\xF4\x1C" + "\x41\x8F\x63\x37\x78\x2F\x86\xA6" + "\xEF\x41\x7E\xD2\xAF\x88\xAB\x67" + "\x52\x71\xC3\x8E\xF8\x26\x93\x72" + "\xAA\xD6\x0E\xE7\x0B\x46\xB1\x3A" + "\xB4\x08\xA9\xA8\xA0\xCF\x20\x0C" + "\x52\xBC\x8B\x05\x56\xB2\xBC\x31" + "\x9B\x74\xB9\x29\x29\x96\x9A\x50" + "\xDC\x45\xDC\x1A\xEB\x0C\x64\xD4" + "\xD3\x05\x7E\x59\x55\xC3\xF4\x90" + "\xC2\xAB\xF8\x9B\x8A\xDA\xCE\xA1" + "\xC3\xF4\xAD\x77\xDD\x44\xC8\xAC" + "\xA3\xF1\xC9\xD2\x19\x5C\xB0\xCA" + "\xA2\x34\xC1\xF7\x6C\xFD\xAC\x65" + "\x32\xDC\x48\xC4\xF2\x00\x6B\x77" + "\xF1\x7D\x76\xAC\xC0\x31\x63\x2A" + "\xA5\x3A\x62\xC8\x91\xB1\x03\x65" + "\xCB\x43\xD1\x06\xDF\xC3\x67\xBC" + "\xDC\xE0\xCD\x35\xCE\x49\x65\xA0" + "\x52\x7B\xA7\x0D\x07\xA9\x1B\xB0" + "\x40\x77\x72\xC2\xEA\x0E\x3A\x78" + "\x46\xB9\x91\xB6\xE7\x3D\x51\x42" + "\xFD\x51\xB0\xC6\x2C\x63\x13\x78" + "\x5C\xEE\xFC\xCF\xC4\x70\x00\x34", + .len = 496, .also_non_np = 1, .np = 3, .tap = { 496 - 20, 4, 16 }, @@ -7187,7 +6473,7 @@ static const struct cipher_testvec des3_ede_ctr_enc_tv_template[] = { .klen = 24, .iv = "\xB2\xD7\x48\xED\x06\x44\xF9\x12" "\xB7\x28\x4D\x83\x24\x59\xF2\x17", - .input = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" + .ptext = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" "\x8B\x12\x86\x69\xF0\x5B\xCF\x56" "\x39\xAD\x34\x9F\x66\xEA\x7D\xC4" "\x48\xD3\xBA\x0D\xB1\x18\xE3\x4A" @@ -7250,8 +6536,7 @@ static const struct cipher_testvec des3_ede_ctr_enc_tv_template[] = { "\x69\x9D\x64\xCF\x56\xDA\xAD\x34" "\xB8\x03\xEA\x7D\xE1\x48\xD3\x47" "\x2E\xB1\x18", - .ilen = 499, - .result = "\x23\xFF\x5C\x99\x75\xBB\x1F\xD4" + .ctext = "\x23\xFF\x5C\x99\x75\xBB\x1F\xD4" "\xBC\x27\x9D\x36\x60\xA9\xC9\xF7" "\x94\x9D\x1B\xFF\x8E\x95\x57\x89" "\x8C\x2E\x33\x70\x43\x61\xE6\xD2" @@ -7314,285 +6599,7 @@ static const struct cipher_testvec des3_ede_ctr_enc_tv_template[] = { "\x5C\x60\x1F\x6E\xA7\xE2\xDC\xE7" "\x32\x0F\x05\x2F\xF2\x4C\x95\x3B" "\xF2\x79\xD9", - .rlen = 499, - .also_non_np = 1, - .np = 2, - .tap = { 499 - 16, 16 }, - }, -}; - -static const struct cipher_testvec des3_ede_ctr_dec_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\x9C\xD6\xF3\x9C\xB9\x5A\x67\x00" - "\x5A\x67\x00\x2D\xCE\xEB\x2D\xCE" - "\xEB\xB4\x51\x72\xB4\x51\x72\x1F", - .klen = 24, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x07\xC2\x08\x20\x72\x1F\x49\xEF" - "\x19\xCD\x6F\x32\x53\x05\x22\x15" - "\xA2\x85\x2B\xDB\x85\xD2\xD8\xB9" - "\xDD\x0D\x1B\x45\xCB\x69\x11\xD4" - "\xEA\xBE\xB2\x45\x5D\x0C\xAE\xBE" - "\xA0\xC1\x27\xAC\x65\x9F\x53\x7E" - "\xAF\xC2\x1B\xB5\xB8\x6D\x36\x0C" - "\x25\xC0\xF8\x6D\x0B\x29\x01\xDA" - "\x13\x78\xDC\x89\x12\x12\x43\xFA" - "\xF6\x12\xEF\x8D\x87\x62\x78\x83" - "\xE2\xBE\x41\x20\x4C\x6D\x35\x1B" - "\xD1\x0C\x30\xCF\xE2\xDE\x2B\x03" - "\xBF\x45\x73\xD4\xE5\x59\x95\xD1" - "\xB3\x9B\x27\x62\x97\xBD\xDE\x7F" - "\xA4\xD2\x39\x80\xAA\x50\x23\xF0" - "\x74\x88\x3D\xA8\x6A\x18\x79\x3B" - "\xC4\x96\x6C\x8D\x22\x40\x92\x6E" - "\xD6\xAD\x2A\x1F\xDE\x63\xC0\xE7" - "\x07\xF7\x2D\xF7\xB5\xF3\xF0\xCC" - "\x01\x7C\x2A\x9B\xC2\x10\xCA\xAA" - "\xFD\x2B\x3F\xC5\xF3\xF6\xFC\x9B" - "\x45\xDB\x53\xE4\x5B\xF3\xC9\x7B" - "\x8E\x52\xFF\xC8\x02\xB8\xAC\x9D" - "\xA1\x00\x39\xDA\x3D\x2D\x0E\x01" - "\x09\x7D\x8D\x5E\xBE\x53\xB9\xB0" - "\x8E\xE7\xE2\x96\x6A\xB2\x78\xEA" - "\xDE\x23\x8B\xA5\xFA\x5C\xE3\xDA" - "\xBF\x8E\x31\x6A\x55\xD1\x6A\xB2" - "\xB5\x46\x6F\xA5\xF0\xEE\xBA\x1F" - "\x9F\x98\xB0\x66\x4F\xD0\x3F\xA9" - "\xDF\x5F\x58\xC4\xF4\xFF\x75\x5C" - "\x40\x3A\x09\x7E\x6E\x1C\x97\xD4" - "\xCC\xE7\xE7\x71\xCF\x0B\x15\x08" - "\x71\xFA\x07\x97\xCD\xE6\xCA\x1D" - "\x14\x28\x0C\xCF\x99\x13\x7A\xF1" - "\xEB\xFA\xFA\x92\x07\xDE\x1D\xA1" - "\xD3\x36\x69\xFE\x51\x4D\x9F\x2E" - "\x83\x37\x4F\x1F\x48\x30\xED\x04" - "\x4D\xA4\xEF\x3A\xCA\x76\xF4\x1C" - "\x41\x8F\x63\x37\x78\x2F\x86\xA6" - "\xEF\x41\x7E\xD2\xAF\x88\xAB\x67" - "\x52\x71\xC3\x8E\xF8\x26\x93\x72" - "\xAA\xD6\x0E\xE7\x0B\x46\xB1\x3A" - "\xB4\x08\xA9\xA8\xA0\xCF\x20\x0C" - "\x52\xBC\x8B\x05\x56\xB2\xBC\x31" - "\x9B\x74\xB9\x29\x29\x96\x9A\x50" - "\xDC\x45\xDC\x1A\xEB\x0C\x64\xD4" - "\xD3\x05\x7E\x59\x55\xC3\xF4\x90" - "\xC2\xAB\xF8\x9B\x8A\xDA\xCE\xA1" - "\xC3\xF4\xAD\x77\xDD\x44\xC8\xAC" - "\xA3\xF1\xC9\xD2\x19\x5C\xB0\xCA" - "\xA2\x34\xC1\xF7\x6C\xFD\xAC\x65" - "\x32\xDC\x48\xC4\xF2\x00\x6B\x77" - "\xF1\x7D\x76\xAC\xC0\x31\x63\x2A" - "\xA5\x3A\x62\xC8\x91\xB1\x03\x65" - "\xCB\x43\xD1\x06\xDF\xC3\x67\xBC" - "\xDC\xE0\xCD\x35\xCE\x49\x65\xA0" - "\x52\x7B\xA7\x0D\x07\xA9\x1B\xB0" - "\x40\x77\x72\xC2\xEA\x0E\x3A\x78" - "\x46\xB9\x91\xB6\xE7\x3D\x51\x42" - "\xFD\x51\xB0\xC6\x2C\x63\x13\x78" - "\x5C\xEE\xFC\xCF\xC4\x70\x00\x34", - .ilen = 496, - .result = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" - "\x8B\x12\x86\x69\xF0\x5B\xCF\x56" - "\x39\xAD\x34\x9F\x66\xEA\x7D\xC4" - "\x48\xD3\xBA\x0D\xB1\x18\xE3\x4A" - "\xFE\x41\x28\x5C\x27\x8E\x11\x85" - "\x6C\xF7\x5E\xC2\x55\x3C\xA0\x0B" - "\x92\x65\xE9\x70\xDB\x4F\xD6\xB9" - "\x00\xB4\x1F\xE6\x49\xFD\x44\x2F" - "\x53\x3A\x8D\x14\x98\x63\xCA\x5D" - "\xC1\xA8\x33\xA7\x0E\x91\x78\xEC" - "\x77\xDE\x42\xD5\xBC\x07\x8B\x12" - "\xE5\x4C\xF0\x5B\x22\x56\x39\x80" - "\x6B\x9F\x66\xC9\x50\xC4\xAF\x36" - "\xBA\x0D\x94\x7F\xE3\x4A\xDD\x41" - "\x28\xB3\x1A\x8E\x11\xF8\x43\xF7" - "\x5E\x21\x55\x3C\x87\x6E\x92\x65" - "\xCC\x57\xDB\xA2\x35\xB9\x00\xEB" - "\x72\xE6\x49\xD0\x44\x2F\xB6\x19" - "\x8D\x14\xFF\x46\xCA\x5D\x24\xA8" - "\x33\x9A\x6D\x91\x78\xC3\x77\xDE" - "\xA1\x08\xBC\x07\xEE\x71\xE5\x4C" - "\xD7\x5B\x22\xB5\x1C\x80\x6B\xF2" - "\x45\xC9\x50\x3B\xAF\x36\x99\x60" - "\x94\x7F\xC6\x4A\xDD\xA4\x0F\xB3" - "\x1A\xED\x74\xF8\x43\x2A\x5E\x21" - "\x88\x13\x87\x6E\xF1\x58\xCC\x57" - "\x3E\xA2\x35\x9C\x67\xEB\x72\xC5" - "\x49\xD0\xBB\x02\xB6\x19\xE0\x4B" - "\xFF\x46\x29\x5D\x24\x8F\x16\x9A" - "\x6D\xF4\x5F\xC3\xAA\x3D\xA1\x08" - "\x93\x7A\xEE\x71\xD8\x4C\xD7\xBE" - "\x01\xB5\x1C\xE7\x4E\xF2\x45\x2C" - "\x50\x3B\x82\x15\x99\x60\xCB\x52" - "\xC6\xA9\x30\xA4\x0F\x96\x79\xED" - "\x74\xDF\x43\x2A\xBD\x04\x88\x13" - "\xFA\x4D\xF1\x58\x23\x57\x3E\x81" - "\x68\x9C\x67\xCE\x51\xC5\xAC\x37" - "\xBB\x02\x95\x7C\xE0\x4B\xD2\x46" - "\x29\xB0\x1B\x8F\x16\xF9\x40\xF4" - "\x5F\x26\xAA\x3D\x84\x6F\x93\x7A" - "\xCD\x54\xD8\xA3\x0A\xBE\x01\xE8" - "\x73\xE7\x4E\xD1\x45\x2C\xB7\x1E" - "\x82\x15\xFC\x47\xCB\x52\x25\xA9" - "\x30\x9B\x62\x96\x79\xC0\x74\xDF" - "\xA6\x09\xBD\x04\xEF\x76\xFA\x4D" - "\xD4\x58\x23\x8A\x1D\x81\x68\xF3" - "\x5A\xCE\x51\x38\xAC\x37\x9E\x61" - "\x95\x7C\xC7\x4B\xD2\xA5\x0C\xB0" - "\x1B\xE2\x75\xF9\x40\x2B\x5F\x26" - "\x89\x10\x84\x6F\xF6\x59\xCD\x54" - "\x3F\xA3\x0A\x9D\x64\xE8\x73\xDA" - "\x4E\xD1\xB8\x03\xB7\x1E\xE1\x48" - "\xFC\x47\x2E\x52\x25\x8C\x17\x9B" - "\x62\xF5\x5C\xC0\xAB\x32\xA6\x09" - "\x90\x7B\xEF\x76\xD9\x4D\xD4\xBF" - "\x06\x8A\x1D\xE4\x4F\xF3\x5A\x2D" - "\x51\x38\x83\x6A\x9E\x61\xC8\x53" - "\xC7\xAE\x31\xA5\x0C\x97\x7E\xE2" - "\x75\xDC\x40\x2B\xB2\x05\x89\x10" - "\xFB\x42\xF6\x59\x20\x54\x3F\x86" - "\x69\x9D\x64\xCF\x56\xDA\xAD\x34" - "\xB8\x03\xEA\x7D\xE1\x48\xD3\x47", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, { /* Generated with Crypto++ */ - .key = "\x9C\xD6\xF3\x9C\xB9\x5A\x67\x00" - "\x5A\x67\x00\x2D\xCE\xEB\x2D\xCE" - "\xEB\xB4\x51\x72\xB4\x51\x72\x1F", - .klen = 24, - .iv = "\xB2\xD7\x48\xED\x06\x44\xF9\x12" - "\xB7\x28\x4D\x83\x24\x59\xF2\x17", - .input = "\x23\xFF\x5C\x99\x75\xBB\x1F\xD4" - "\xBC\x27\x9D\x36\x60\xA9\xC9\xF7" - "\x94\x9D\x1B\xFF\x8E\x95\x57\x89" - "\x8C\x2E\x33\x70\x43\x61\xE6\xD2" - "\x82\x33\x63\xB6\xC4\x34\x5E\xF8" - "\x96\x07\xA7\xD2\x3B\x8E\xC9\xAA" - "\x7C\xA0\x55\x89\x2E\xE1\x85\x25" - "\x14\x04\xDA\x6B\xE0\xEE\x56\xCF" - "\x08\x2E\x69\xD4\x54\xDE\x22\x84" - "\x69\xA6\xA7\xD3\x3A\x9A\xE8\x05" - "\x63\xDB\xBF\x46\x3A\x26\x2E\x0F" - "\x58\x5C\x46\xEA\x07\x40\xDA\xE1" - "\x14\x1D\xCD\x4F\x06\xC0\xCA\x54" - "\x1E\xC9\x45\x85\x67\x7C\xC2\xB5" - "\x97\x5D\x61\x78\x2E\x46\xEC\x6A" - "\x53\xF4\xD0\xAE\xFA\xB4\x86\x29" - "\x9F\x17\x33\x24\xD8\xB9\xB2\x05" - "\x93\x88\xEA\xF7\xA0\x70\x69\x49" - "\x88\x6B\x73\x40\x41\x8D\xD9\xD9" - "\x7E\x78\xE9\xBE\x6C\x14\x22\x7A" - "\x66\xE1\xDA\xED\x10\xFF\x69\x1D" - "\xB9\xAA\xF2\x56\x72\x1B\x23\xE2" - "\x45\x54\x8B\xA3\x70\x23\xB4\x5E" - "\x8E\x96\xC9\x05\x00\xB3\xB6\xC2" - "\x2A\x02\x43\x7A\x62\xD5\xC8\xD2" - "\xC2\xD0\xE4\x78\xA1\x7B\x3E\xE8" - "\x9F\x7F\x7D\x40\x54\x30\x3B\xC0" - "\xA5\x54\xFD\xCA\x25\xEC\x44\x3E" - "\x1A\x54\x7F\x88\xD0\xE1\xFE\x71" - "\xCE\x05\x49\x89\xBA\xD6\x72\xE7" - "\xD6\x5D\x3F\xA2\xD9\xAB\xC5\x02" - "\xD6\x43\x22\xAF\xA2\xE4\x80\x85" - "\xD7\x87\xB9\xEA\x43\xDB\xC8\xEF" - "\x5C\x82\x2E\x98\x0D\x30\x41\x6B" - "\x08\x48\x8D\xF0\xF8\x60\xD7\x9D" - "\xE9\xDE\x40\xAD\x0D\xAD\x0D\x58" - "\x2A\x98\x35\xFE\xF7\xDD\x4B\x40" - "\xDE\xB0\x05\xD9\x7B\x09\x4D\xBC" - "\x42\xC0\xF1\x15\x0B\xFA\x26\x6B" - "\xC6\x12\x13\x4F\xCB\x35\xBA\x35" - "\xDD\x7A\x36\x9C\x12\x57\x55\x83" - "\x78\x58\x09\xD0\xB0\xCF\x7C\x5C" - "\x38\xCF\xBD\x79\x5B\x13\x4D\x97" - "\xC1\x85\x6F\x97\xC9\xE8\xC2\xA4" - "\x98\xE2\xBD\x77\x6B\x53\x39\x1A" - "\x28\x10\xE7\xE0\xE7\xDE\x9D\x69" - "\x78\x6F\x8E\xD2\xD9\x5D\xD2\x15" - "\x9E\xB5\x4D\x8C\xC0\x78\x22\x2F" - "\x17\x11\x2E\x99\xD7\xE3\xA4\x4F" - "\x65\xA5\x6B\x03\x2C\x35\x6F\xDA" - "\x8A\x19\x08\xE1\x08\x48\x59\x51" - "\x53\x4B\xD1\xDF\xDA\x14\x50\x5F" - "\xDF\xB5\x8C\xDF\xC6\xFD\x85\xFA" - "\xD4\xF9\x64\x45\x65\x0D\x7D\xF4" - "\xC8\xCD\x3F\x32\xAF\xDD\x30\xED" - "\x7B\xAA\xAC\xF0\xDA\x7F\xDF\x75" - "\x1C\xA4\xF1\xCB\x5E\x4F\x0B\xB4" - "\x97\x73\x28\xDE\xCF\xAF\x82\xBD" - "\xC4\xBA\xB4\x9C\x0D\x16\x77\x42" - "\x42\x39\x7C\x53\xA4\xD4\xDD\x40" - "\x5C\x60\x1F\x6E\xA7\xE2\xDC\xE7" - "\x32\x0F\x05\x2F\xF2\x4C\x95\x3B" - "\xF2\x79\xD9", - .ilen = 499, - .result = "\x05\xEC\x77\xFB\x42\xD5\x59\x20" - "\x8B\x12\x86\x69\xF0\x5B\xCF\x56" - "\x39\xAD\x34\x9F\x66\xEA\x7D\xC4" - "\x48\xD3\xBA\x0D\xB1\x18\xE3\x4A" - "\xFE\x41\x28\x5C\x27\x8E\x11\x85" - "\x6C\xF7\x5E\xC2\x55\x3C\xA0\x0B" - "\x92\x65\xE9\x70\xDB\x4F\xD6\xB9" - "\x00\xB4\x1F\xE6\x49\xFD\x44\x2F" - "\x53\x3A\x8D\x14\x98\x63\xCA\x5D" - "\xC1\xA8\x33\xA7\x0E\x91\x78\xEC" - "\x77\xDE\x42\xD5\xBC\x07\x8B\x12" - "\xE5\x4C\xF0\x5B\x22\x56\x39\x80" - "\x6B\x9F\x66\xC9\x50\xC4\xAF\x36" - "\xBA\x0D\x94\x7F\xE3\x4A\xDD\x41" - "\x28\xB3\x1A\x8E\x11\xF8\x43\xF7" - "\x5E\x21\x55\x3C\x87\x6E\x92\x65" - "\xCC\x57\xDB\xA2\x35\xB9\x00\xEB" - "\x72\xE6\x49\xD0\x44\x2F\xB6\x19" - "\x8D\x14\xFF\x46\xCA\x5D\x24\xA8" - "\x33\x9A\x6D\x91\x78\xC3\x77\xDE" - "\xA1\x08\xBC\x07\xEE\x71\xE5\x4C" - "\xD7\x5B\x22\xB5\x1C\x80\x6B\xF2" - "\x45\xC9\x50\x3B\xAF\x36\x99\x60" - "\x94\x7F\xC6\x4A\xDD\xA4\x0F\xB3" - "\x1A\xED\x74\xF8\x43\x2A\x5E\x21" - "\x88\x13\x87\x6E\xF1\x58\xCC\x57" - "\x3E\xA2\x35\x9C\x67\xEB\x72\xC5" - "\x49\xD0\xBB\x02\xB6\x19\xE0\x4B" - "\xFF\x46\x29\x5D\x24\x8F\x16\x9A" - "\x6D\xF4\x5F\xC3\xAA\x3D\xA1\x08" - "\x93\x7A\xEE\x71\xD8\x4C\xD7\xBE" - "\x01\xB5\x1C\xE7\x4E\xF2\x45\x2C" - "\x50\x3B\x82\x15\x99\x60\xCB\x52" - "\xC6\xA9\x30\xA4\x0F\x96\x79\xED" - "\x74\xDF\x43\x2A\xBD\x04\x88\x13" - "\xFA\x4D\xF1\x58\x23\x57\x3E\x81" - "\x68\x9C\x67\xCE\x51\xC5\xAC\x37" - "\xBB\x02\x95\x7C\xE0\x4B\xD2\x46" - "\x29\xB0\x1B\x8F\x16\xF9\x40\xF4" - "\x5F\x26\xAA\x3D\x84\x6F\x93\x7A" - "\xCD\x54\xD8\xA3\x0A\xBE\x01\xE8" - "\x73\xE7\x4E\xD1\x45\x2C\xB7\x1E" - "\x82\x15\xFC\x47\xCB\x52\x25\xA9" - "\x30\x9B\x62\x96\x79\xC0\x74\xDF" - "\xA6\x09\xBD\x04\xEF\x76\xFA\x4D" - "\xD4\x58\x23\x8A\x1D\x81\x68\xF3" - "\x5A\xCE\x51\x38\xAC\x37\x9E\x61" - "\x95\x7C\xC7\x4B\xD2\xA5\x0C\xB0" - "\x1B\xE2\x75\xF9\x40\x2B\x5F\x26" - "\x89\x10\x84\x6F\xF6\x59\xCD\x54" - "\x3F\xA3\x0A\x9D\x64\xE8\x73\xDA" - "\x4E\xD1\xB8\x03\xB7\x1E\xE1\x48" - "\xFC\x47\x2E\x52\x25\x8C\x17\x9B" - "\x62\xF5\x5C\xC0\xAB\x32\xA6\x09" - "\x90\x7B\xEF\x76\xD9\x4D\xD4\xBF" - "\x06\x8A\x1D\xE4\x4F\xF3\x5A\x2D" - "\x51\x38\x83\x6A\x9E\x61\xC8\x53" - "\xC7\xAE\x31\xA5\x0C\x97\x7E\xE2" - "\x75\xDC\x40\x2B\xB2\x05\x89\x10" - "\xFB\x42\xF6\x59\x20\x54\x3F\x86" - "\x69\x9D\x64\xCF\x56\xDA\xAD\x34" - "\xB8\x03\xEA\x7D\xE1\x48\xD3\x47" - "\x2E\xB1\x18", - .rlen = 499, + .len = 499, .also_non_np = 1, .np = 2, .tap = { 499 - 16, 16 }, @@ -7602,45 +6609,40 @@ static const struct cipher_testvec des3_ede_ctr_dec_tv_template[] = { /* * Blowfish test vectors. */ -static const struct cipher_testvec bf_enc_tv_template[] = { +static const struct cipher_testvec bf_tv_template[] = { { /* DES test vectors from OpenSSL */ .key = "\x00\x00\x00\x00\x00\x00\x00\x00", .klen = 8, - .input = "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 8, - .result = "\x4e\xf9\x97\x45\x61\x98\xdd\x78", - .rlen = 8, + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\x4e\xf9\x97\x45\x61\x98\xdd\x78", + .len = 8, }, { .key = "\x1f\x1f\x1f\x1f\x0e\x0e\x0e\x0e", .klen = 8, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .ilen = 8, - .result = "\xa7\x90\x79\x51\x08\xea\x3c\xae", - .rlen = 8, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .ctext = "\xa7\x90\x79\x51\x08\xea\x3c\xae", + .len = 8, }, { .key = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", .klen = 8, - .input = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .ilen = 8, - .result = "\xe8\x7a\x24\x4e\x2c\xc8\x5e\x82", - .rlen = 8, + .ptext = "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .ctext = "\xe8\x7a\x24\x4e\x2c\xc8\x5e\x82", + .len = 8, }, { /* Vary the keylength... */ .key = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87" "\x78\x69\x5a\x4b\x3c\x2d\x1e\x0f", .klen = 16, - .input = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .ilen = 8, - .result = "\x93\x14\x28\x87\xee\x3b\xe1\x5c", - .rlen = 8, + .ptext = "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .ctext = "\x93\x14\x28\x87\xee\x3b\xe1\x5c", + .len = 8, }, { .key = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87" "\x78\x69\x5a\x4b\x3c\x2d\x1e\x0f" "\x00\x11\x22\x33\x44", .klen = 21, - .input = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .ilen = 8, - .result = "\xe6\xf5\x1e\xd7\x9b\x9d\xb2\x1f", - .rlen = 8, + .ptext = "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .ctext = "\xe6\xf5\x1e\xd7\x9b\x9d\xb2\x1f", + .len = 8, }, { /* Generated with bf488 */ .key = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87" "\x78\x69\x5a\x4b\x3c\x2d\x1e\x0f" @@ -7650,17 +6652,16 @@ static const struct cipher_testvec bf_enc_tv_template[] = { "\x1f\x1f\x1f\x1f\x0e\x0e\x0e\x0e" "\xff\xff\xff\xff\xff\xff\xff\xff", .klen = 56, - .input = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .ilen = 8, - .result = "\xc0\x45\x04\x01\x2e\x4e\x1f\x53", - .rlen = 8, + .ptext = "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .ctext = "\xc0\x45\x04\x01\x2e\x4e\x1f\x53", + .len = 8, }, { /* Generated with Crypto++ */ .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -7723,8 +6724,7 @@ static const struct cipher_testvec bf_enc_tv_template[] = { "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" "\x2B\xC2\x59\xF0\x64\xFB\x92\x06", - .ilen = 504, - .result = "\x96\x87\x3D\x0C\x7B\xFB\xBD\x1F" + .ctext = "\x96\x87\x3D\x0C\x7B\xFB\xBD\x1F" "\xE3\xC1\x99\x6D\x39\xD4\xC2\x7D" "\xD7\x87\xA1\xF2\xDF\x51\x71\x26" "\xC2\xF4\x6D\xFF\xF6\xCD\x6B\x40" @@ -7787,136 +6787,36 @@ static const struct cipher_testvec bf_enc_tv_template[] = { "\x54\x59\x09\xA5\x3D\xC5\x84\x68" "\x56\xEB\x36\x77\x3D\xAA\xB8\xF5" "\xC9\x1A\xFB\x5D\xDE\xBB\x43\xF4", - .rlen = 504, + .len = 504, .also_non_np = 1, .np = 3, .tap = { 504 - 10, 2, 8 }, }, }; -static const struct cipher_testvec bf_dec_tv_template[] = { - { /* DES test vectors from OpenSSL */ - .key = "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 8, - .input = "\x4e\xf9\x97\x45\x61\x98\xdd\x78", - .ilen = 8, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 8, - }, { - .key = "\x1f\x1f\x1f\x1f\x0e\x0e\x0e\x0e", - .klen = 8, - .input = "\xa7\x90\x79\x51\x08\xea\x3c\xae", - .ilen = 8, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .rlen = 8, - }, { - .key = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", - .klen = 8, - .input = "\xe8\x7a\x24\x4e\x2c\xc8\x5e\x82", - .ilen = 8, - .result = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .rlen = 8, - }, { /* Vary the keylength... */ - .key = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87" - "\x78\x69\x5a\x4b\x3c\x2d\x1e\x0f", +static const struct cipher_testvec bf_cbc_tv_template[] = { + { /* From OpenSSL */ + .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", .klen = 16, - .input = "\x93\x14\x28\x87\xee\x3b\xe1\x5c", - .ilen = 8, - .result = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .rlen = 8, - }, { - .key = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87" - "\x78\x69\x5a\x4b\x3c\x2d\x1e\x0f" - "\x00\x11\x22\x33\x44", - .klen = 21, - .input = "\xe6\xf5\x1e\xd7\x9b\x9d\xb2\x1f", - .ilen = 8, - .result = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .rlen = 8, - }, { /* Generated with bf488, using OpenSSL, Libgcrypt and Nettle */ - .key = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87" - "\x78\x69\x5a\x4b\x3c\x2d\x1e\x0f" - "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x04\x68\x91\x04\xc2\xfd\x3b\x2f" - "\x58\x40\x23\x64\x1a\xba\x61\x76" - "\x1f\x1f\x1f\x1f\x0e\x0e\x0e\x0e" - "\xff\xff\xff\xff\xff\xff\xff\xff", - .klen = 56, - .input = "\xc0\x45\x04\x01\x2e\x4e\x1f\x53", - .ilen = 8, - .result = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .rlen = 8, + .iv = "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .ptext = "\x37\x36\x35\x34\x33\x32\x31\x20" + "\x4e\x6f\x77\x20\x69\x73\x20\x74" + "\x68\x65\x20\x74\x69\x6d\x65\x20" + "\x66\x6f\x72\x20\x00\x00\x00\x00", + .ctext = "\x6b\x77\xb4\xd6\x30\x06\xde\xe6" + "\x05\xb1\x56\xe2\x74\x03\x97\x93" + "\x58\xde\xb9\xe7\x15\x46\x16\xd9" + "\x59\xf1\x65\x2b\xd5\xff\x92\xcc", + .len = 32, }, { /* Generated with Crypto++ */ .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, - .input = "\x96\x87\x3D\x0C\x7B\xFB\xBD\x1F" - "\xE3\xC1\x99\x6D\x39\xD4\xC2\x7D" - "\xD7\x87\xA1\xF2\xDF\x51\x71\x26" - "\xC2\xF4\x6D\xFF\xF6\xCD\x6B\x40" - "\xE1\xB3\xBF\xD4\x38\x2B\xC8\x3B" - "\xD3\xB2\xD4\x61\xC7\x9F\x06\xE9" - "\xCD\xF3\x88\x39\x39\x7A\xDF\x19" - "\xE8\x03\x2A\x0B\x9E\xA0\x2B\x86" - "\x31\xF8\x9D\xB1\xEE\x78\x9D\xB5" - "\xCD\x8B\x7C\x2E\xF5\xA2\x2D\x5D" - "\x6E\x66\xAF\x38\x6C\xD3\x13\xED" - "\x14\xEA\x5D\xD0\x17\x77\x0F\x4A" - "\x50\xF2\xD0\x0F\xC8\xF7\x1E\x7B" - "\x9D\x5B\x54\x65\x4F\x16\x8A\x97" - "\xF3\xF6\xD4\xAA\x87\x36\x77\x72" - "\x99\x4A\xB5\x5E\x88\xC3\xCD\x7D" - "\x1D\x97\xF9\x11\xBD\xE0\x1F\x1F" - "\x96\x3E\x4B\x22\xF4\xC0\xE6\xB8" - "\x47\x82\x98\x23\x33\x36\xBC\x1B" - "\x36\xE7\xF6\xCF\x97\x37\x16\xC0" - "\x87\x31\x8B\xB0\xDB\x19\x42\xA5" - "\x1F\x90\x7E\x66\x34\xDD\x5E\xE9" - "\x4F\xB2\x2B\x9A\xDE\xB3\x5D\x71" - "\x4D\x68\xF0\xDC\xA6\xEA\xE3\x9B" - "\x60\x00\x55\x57\x06\x8B\xD5\xB3" - "\x86\x30\x78\xDA\x33\x9A\x9D\xCC" - "\xBA\x0B\x81\x06\x77\x43\xC7\xC9" - "\xDB\x37\x60\x11\x45\x59\x6D\x2D" - "\x90\x3D\x65\x3E\xD0\x13\xC6\x3C" - "\x0E\x78\x7D\x9A\x00\xD6\x2F\x0B" - "\x3B\x53\x19\x1E\xA8\x9B\x11\xD9" - "\x98\xE4\x7F\xC3\x6E\x51\x24\x70" - "\x9F\x04\x9C\xC2\x9E\x44\x84\xE3" - "\xE0\x8A\x44\xA2\x5C\x94\x74\x34" - "\x37\x52\x7C\x03\xE8\x8E\x97\xE1" - "\x5B\x5C\x0E\xB0\x70\xFE\x54\x3F" - "\xD8\x65\xA9\xC5\xCD\xEC\xF4\x45" - "\x55\xC5\xA7\xA3\x19\x80\x28\x51" - "\xBE\x64\x4A\xC1\xD4\xE1\xBE\xEB" - "\x73\x4C\xB6\xF9\x5F\x6D\x82\xBC" - "\x3E\x42\x14\x49\x88\x51\xBF\x68" - "\x45\x75\x27\x1B\x0A\x72\xED\xAF" - "\xDA\xC4\x4D\x67\x0D\xEE\x75\xE3" - "\x34\xDD\x91\x19\x42\x3A\xCB\xDA" - "\x38\xFA\x3C\x93\x62\xF2\xE3\x81" - "\xB3\xE4\xBB\xF6\x0D\x0B\x1D\x09" - "\x9C\x52\x0D\x50\x63\xA4\xB2\xD2" - "\x82\xA0\x23\x3F\x1F\xB6\xED\x6E" - "\xC2\x9C\x1C\xD0\x9A\x40\xB6\xFC" - "\x36\x56\x6E\x85\x73\xD7\x52\xBA" - "\x35\x5E\x32\x89\x5D\x42\xF5\x36" - "\x52\x8D\x46\x7D\xC8\x71\xAD\x33" - "\xE1\xAF\x6A\xA8\xEC\xBA\x1C\xDC" - "\xFE\x88\xE6\x16\xE4\xC8\x13\x00" - "\x3C\xDA\x59\x32\x38\x19\xD5\xEB" - "\xB6\x7F\x78\x45\x1B\x8E\x07\x8C" - "\x66\x52\x75\xFF\xAF\xCE\x2D\x2B" - "\x22\x29\xCA\xB3\x5F\x7F\xE3\x29" - "\xB2\xB8\x9D\xEB\x16\xC8\xC5\x1D" - "\xC9\x0D\x59\x82\x27\x57\x9D\x42" - "\x54\x59\x09\xA5\x3D\xC5\x84\x68" - "\x56\xEB\x36\x77\x3D\xAA\xB8\xF5" - "\xC9\x1A\xFB\x5D\xDE\xBB\x43\xF4", - .ilen = 504, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -7979,194 +6879,7 @@ static const struct cipher_testvec bf_dec_tv_template[] = { "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" "\x2B\xC2\x59\xF0\x64\xFB\x92\x06", - .rlen = 504, - .also_non_np = 1, - .np = 3, - .tap = { 504 - 10, 2, 8 }, - }, -}; - -static const struct cipher_testvec bf_cbc_enc_tv_template[] = { - { /* From OpenSSL */ - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", - .klen = 16, - .iv = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .input = "\x37\x36\x35\x34\x33\x32\x31\x20" - "\x4e\x6f\x77\x20\x69\x73\x20\x74" - "\x68\x65\x20\x74\x69\x6d\x65\x20" - "\x66\x6f\x72\x20\x00\x00\x00\x00", - .ilen = 32, - .result = "\x6b\x77\xb4\xd6\x30\x06\xde\xe6" - "\x05\xb1\x56\xe2\x74\x03\x97\x93" - "\x58\xde\xb9\xe7\x15\x46\x16\xd9" - "\x59\xf1\x65\x2b\xd5\xff\x92\xcc", - .rlen = 32, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59\xF0\x64\xFB\x92\x06", - .ilen = 504, - .result = "\xB4\xFE\xA5\xBB\x3D\x2C\x27\x06" - "\x06\x2B\x3A\x92\xB2\xF5\x5E\x62" - "\x84\xCD\xF7\x66\x7E\x41\x6C\x8E" - "\x1B\xD9\x02\xB6\x48\xB0\x87\x25" - "\x01\x9C\x93\x63\x51\x60\x82\xD2" - "\x4D\xE5\xC2\xB7\xAE\x60\xD8\xAD" - "\x9F\xAB\x6C\xFA\x20\x05\xDA\x6F" - "\x1F\xD1\xD8\x36\x0F\xB5\x16\x69" - "\x3C\xAF\xB3\x30\x18\x33\xE6\xB5" - "\x43\x29\x9D\x94\xF4\x2F\x0A\x65" - "\x40\xB2\xB2\xB2\x42\x89\xEE\x8A" - "\x60\xD3\x52\xA8\xED\x91\xDF\xE1" - "\x91\x73\x7C\x28\xA1\x14\xC3\x4C" - "\x82\x72\x4B\x7D\x7D\x32\xD5\x19" - "\xE8\xB8\x6B\x30\x21\x09\x0E\x27" - "\x10\x9D\x2D\x3A\x6A\x4B\x7B\xE6" - "\x8D\x4E\x02\x32\xFF\x7F\x8E\x13" - "\xB0\x96\xF4\xC2\xA1\x60\x8A\x69" - "\xEF\x0F\x86\xD0\x25\x13\x1A\x7C" - "\x6E\xF0\x41\xA3\xFB\xB3\xAB\x40" - "\x7D\x19\xA0\x11\x4F\x3E\x1D\x43" - "\x65\xFE\x15\x40\xD0\x62\x41\x02" - "\xEA\x0C\x7A\xC3\x84\xEE\xB0\xBE" - "\xBE\xC8\x57\x51\xCD\x4F\xAD\x5C" - "\xCC\x79\xBA\x0D\x85\x3A\xED\x6B" - "\xAC\x6B\xA3\x4D\xBC\xE8\x02\x6A" - "\xC2\x6D\xBD\x5E\x89\x95\x86\x43" - "\x2C\x17\x4B\xC6\x40\xA2\xBD\x24" - "\x04\xF0\x86\x08\x78\x18\x42\xE0" - "\x39\x1B\x22\x9E\x89\x4C\x04\x6B" - "\x65\xC5\xB6\x0E\xF6\x63\xFC\xD7" - "\xAE\x9E\x87\x13\xCC\xD3\x1A\xEC" - "\xF0\x51\xCC\x93\x68\xFC\xE9\x19" - "\x7C\x4E\x9B\xCC\x17\xAD\xD2\xFC" - "\x97\x18\x92\xFF\x15\x11\xCE\xED" - "\x04\x41\x05\xA3\x92\xFF\x3B\xE6" - "\xB6\x8C\x90\xC6\xCD\x15\xA0\x04" - "\x25\x8B\x5D\x5B\x5F\xDB\xAE\x68" - "\xEF\xB3\x61\x18\xDB\x83\x9B\x39" - "\xCA\x82\xD1\x88\xF0\xA2\x5C\x02" - "\x87\xBD\x8D\x8F\xBB\x62\xF0\x35" - "\x75\x6F\x06\x81\x0A\x97\x4D\xF0" - "\x43\x12\x73\x77\xDB\x91\x83\x5B" - "\xE7\x3A\xA6\x07\x7B\xBF\x2C\x50" - "\x94\xDE\x7B\x65\xDA\x1C\xF1\x9F" - "\x7E\x12\x40\xB2\x3E\x19\x23\xF1" - "\x7C\x1B\x5F\xA8\xF3\xAC\x63\x87" - "\xEB\x3E\x0C\xBE\xA3\x63\x97\x88" - "\x8D\x27\xC6\x2A\xF8\xF2\x67\x9A" - "\x0D\x14\x16\x2B\x6F\xCB\xD4\x76" - "\x14\x48\x2E\xDE\x2A\x44\x5E\x45" - "\xF1\x97\x82\xEF\xB7\xAE\xED\x3A" - "\xED\x73\xD3\x79\xF7\x38\x1D\xD0" - "\xC5\xF8\x69\x83\x28\x84\x87\x56" - "\x3F\xAE\x81\x04\x79\x1F\xD1\x09" - "\xC5\xE5\x05\x0D\x64\x16\xCE\x42" - "\xC5\xF8\xDB\x57\x89\x33\x22\xFC" - "\xB4\xD7\x94\xB9\xF3\xCC\x02\x90" - "\x02\xBA\x55\x1E\x24\x3E\x02\x1D" - "\xC6\xCD\x8F\xD9\xBD\xED\xB0\x51" - "\xCD\xE9\xD5\x0C\xFE\x12\x39\xA9" - "\x93\x9B\xEE\xB5\x97\x41\xD2\xA0" - "\xB4\x98\xD8\x6B\x74\xE7\x65\xF4", - .rlen = 504, - .also_non_np = 1, - .np = 3, - .tap = { 504 - 10, 2, 8 }, - }, -}; - -static const struct cipher_testvec bf_cbc_dec_tv_template[] = { - { /* From OpenSSL */ - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", - .klen = 16, - .iv = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .input = "\x6b\x77\xb4\xd6\x30\x06\xde\xe6" - "\x05\xb1\x56\xe2\x74\x03\x97\x93" - "\x58\xde\xb9\xe7\x15\x46\x16\xd9" - "\x59\xf1\x65\x2b\xd5\xff\x92\xcc", - .ilen = 32, - .result = "\x37\x36\x35\x34\x33\x32\x31\x20" - "\x4e\x6f\x77\x20\x69\x73\x20\x74" - "\x68\x65\x20\x74\x69\x6d\x65\x20" - "\x66\x6f\x72\x20\x00\x00\x00\x00", - .rlen = 32, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\xB4\xFE\xA5\xBB\x3D\x2C\x27\x06" + .ctext = "\xB4\xFE\xA5\xBB\x3D\x2C\x27\x06" "\x06\x2B\x3A\x92\xB2\xF5\x5E\x62" "\x84\xCD\xF7\x66\x7E\x41\x6C\x8E" "\x1B\xD9\x02\xB6\x48\xB0\x87\x25" @@ -8229,78 +6942,14 @@ static const struct cipher_testvec bf_cbc_dec_tv_template[] = { "\xCD\xE9\xD5\x0C\xFE\x12\x39\xA9" "\x93\x9B\xEE\xB5\x97\x41\xD2\xA0" "\xB4\x98\xD8\x6B\x74\xE7\x65\xF4", - .ilen = 504, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59\xF0\x64\xFB\x92\x06", - .rlen = 504, + .len = 504, .also_non_np = 1, .np = 3, .tap = { 504 - 10, 2, 8 }, }, }; -static const struct cipher_testvec bf_ctr_enc_tv_template[] = { +static const struct cipher_testvec bf_ctr_tv_template[] = { { /* Generated with Crypto++ */ .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" @@ -8308,7 +6957,7 @@ static const struct cipher_testvec bf_ctr_enc_tv_template[] = { "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -8371,8 +7020,7 @@ static const struct cipher_testvec bf_ctr_enc_tv_template[] = { "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" "\x2B\xC2\x59\xF0\x64\xFB\x92\x06", - .ilen = 504, - .result = "\xC7\xA3\xDF\xB9\x05\xF4\x9E\x8D" + .ctext = "\xC7\xA3\xDF\xB9\x05\xF4\x9E\x8D" "\x9E\xDF\x38\x18\x83\x07\xEF\xC1" "\x93\x3C\xAA\xAA\xFE\x06\x42\xCC" "\x0D\x70\x86\x5A\x44\xAD\x85\x17" @@ -8435,7 +7083,7 @@ static const struct cipher_testvec bf_ctr_enc_tv_template[] = { "\x64\xBB\x15\xB8\x56\xCF\xEE\xE5" "\x32\x44\x96\x1C\xD8\xEB\x95\xD2" "\xF3\x71\xEF\xEB\x4E\xBB\x4D\x29", - .rlen = 504, + .len = 504, }, { /* Generated with Crypto++ */ .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" @@ -8443,7 +7091,7 @@ static const struct cipher_testvec bf_ctr_enc_tv_template[] = { "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -8506,8 +7154,7 @@ static const struct cipher_testvec bf_ctr_enc_tv_template[] = { "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" "\x2B\xC2\x59\xF0\x64\xFB\x92", - .ilen = 503, - .result = "\xC7\xA3\xDF\xB9\x05\xF4\x9E\x8D" + .ctext = "\xC7\xA3\xDF\xB9\x05\xF4\x9E\x8D" "\x9E\xDF\x38\x18\x83\x07\xEF\xC1" "\x93\x3C\xAA\xAA\xFE\x06\x42\xCC" "\x0D\x70\x86\x5A\x44\xAD\x85\x17" @@ -8570,7 +7217,7 @@ static const struct cipher_testvec bf_ctr_enc_tv_template[] = { "\x64\xBB\x15\xB8\x56\xCF\xEE\xE5" "\x32\x44\x96\x1C\xD8\xEB\x95\xD2" "\xF3\x71\xEF\xEB\x4E\xBB\x4D", - .rlen = 503, + .len = 503, .also_non_np = 1, .np = 2, .tap = { 503 - 8, 8 }, @@ -8581,7 +7228,7 @@ static const struct cipher_testvec bf_ctr_enc_tv_template[] = { "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -8644,8 +7291,7 @@ static const struct cipher_testvec bf_ctr_enc_tv_template[] = { "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" "\x2B\xC2\x59\xF0\x64\xFB\x92\x06", - .ilen = 504, - .result = "\x5F\x58\x6E\x60\x51\x6E\xDC\x3D" + .ctext = "\x5F\x58\x6E\x60\x51\x6E\xDC\x3D" "\xD1\xBB\xF7\xB7\xFD\x04\x44\x82" "\xDC\x9F\x4B\x02\xF1\xD2\x5A\x6F" "\x25\xF9\x27\x21\xF2\xD2\x9A\x01" @@ -8708,83 +7354,47 @@ static const struct cipher_testvec bf_ctr_enc_tv_template[] = { "\xB5\x77\xBA\x1C\xEE\x71\xFA\xB0" "\x16\x4C\x18\x6B\xF2\x69\xA0\x07" "\xEF\xBE\xEC\x69\xAC\xA8\x63\x9E", - .rlen = 504, + .len = 504, }, }; -static const struct cipher_testvec bf_ctr_dec_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\xC7\xA3\xDF\xB9\x05\xF4\x9E\x8D" - "\x9E\xDF\x38\x18\x83\x07\xEF\xC1" - "\x93\x3C\xAA\xAA\xFE\x06\x42\xCC" - "\x0D\x70\x86\x5A\x44\xAD\x85\x17" - "\xE4\x1F\x5E\xA5\x89\xAC\x32\xBC" - "\x3D\xA7\xE9\x0A\x5C\x70\x4D\xDE" - "\x99\x38\x07\xCA\x1D\x21\xC1\x11" - "\x97\xEB\x98\x75\xC4\x73\x45\x83" - "\x46\x1C\x9C\x91\x87\xC1\xA0\x56" - "\x98\xA1\x8B\xDB\x22\x76\xBD\x62" - "\xA4\xBC\xE8\x86\xDA\xD2\x51\x13" - "\x13\xD2\x96\x68\x69\x10\x67\x0C" - "\xD0\x17\x25\x7C\xB2\xAE\x4F\x93" - "\xA6\x82\x20\xCF\x0F\xA6\x47\x79" - "\x88\x09\x40\x59\xBD\x12\x64\xB5" - "\x19\x38\x0D\xFF\x86\xD9\x42\x20" - "\x81\x0D\x96\x99\xAF\x22\x1F\x94" - "\x5C\x6E\xEC\xEA\xA3\x39\xCB\x09" - "\x43\x19\x7F\xD0\xBB\x10\xC2\x49" - "\xF7\xE9\xF2\xEE\xBF\xF7\xF8\xB3" - "\x0E\x1A\xF1\x8D\x70\x82\x0C\x04" - "\xFD\x29\x1A\xAC\xC0\x92\x48\x34" - "\x6A\xE3\x1D\x4F\xFC\x1C\x72\x6A" - "\x57\xCB\xAD\xD0\x98\xAB\xB1\x01" - "\x03\x6A\x45\xDD\x07\x71\x5F\x5B" - "\xB5\x4A\xE4\xE5\xB9\xB9\xBC\xAC" - "\x44\xF7\x41\xA4\x5F\x2E\xE9\x28" - "\xE3\x05\xD2\x94\x78\x4C\x33\x1B" - "\xBD\xC1\x6E\x51\xD9\xAD\xD9\x86" - "\x15\x4A\x78\xAE\x7B\xAD\x3B\xBC" - "\x2F\xE0\x0E\xC5\x7B\x54\x97\x5F" - "\x60\x51\x14\x65\xF9\x91\xE9\xDA" - "\x9A\xBC\xFC\x19\x29\x67\xAA\x63" - "\x5E\xF2\x48\x88\xEB\x79\xE1\xE4" - "\xF7\xF6\x4C\xA9\xE2\x8C\x3B\xE0" - "\xED\x52\xAE\x90\x8F\x5B\x98\x34" - "\x29\x94\x34\x7F\xF9\x6C\x1E\xB6" - "\xA4\xE7\x2D\x06\x54\x9D\xC3\x02" - "\xC1\x90\xA4\x72\x31\x6B\x24\x51" - "\x0B\xB3\x7C\x63\x15\xBA\xAF\x5D" - "\x41\xE0\x37\x6D\xBE\x41\x58\xDE" - "\xF2\x07\x62\x99\xBE\xC1\x8C\x0F" - "\x0F\x28\xFB\x8F\x0E\x1D\x91\xE2" - "\xDA\x99\x5C\x49\xBA\x9C\xA8\x86" - "\x82\x63\x11\xB3\x54\x49\x00\x08" - "\x07\xF2\xE8\x1F\x34\x49\x61\xF4" - "\x81\xE9\xF6\xA9\x5A\x28\x60\x1F" - "\x66\x99\x08\x06\xF2\xE8\x2D\xD1" - "\xD0\x67\xBA\x32\x1F\x02\x86\x7B" - "\xFB\x79\x3D\xC5\xB1\x7F\x15\xAF" - "\xD7\xBF\x31\x46\x22\x7F\xAE\x5B" - "\x8B\x95\x47\xC2\xB1\x62\xA1\xCE" - "\x52\xAC\x9C\x8B\xC2\x49\x7F\xBC" - "\x9C\x89\xB8\xB6\xCA\xE3\x8F\xEA" - "\xAC\xB4\x5D\xE4\x50\xDC\x3A\xB5" - "\x91\x04\x94\x99\x03\x3B\x42\x6D" - "\x9C\x4A\x02\xF5\xB5\x38\x98\xA8" - "\x5C\x97\x2E\x4D\x79\x67\x71\xAF" - "\xF0\x70\x77\xFF\x2D\xDA\xA0\x9E" - "\x23\x8D\xD6\xA6\x68\x10\x78\x9A" - "\x64\xBB\x15\xB8\x56\xCF\xEE\xE5" - "\x32\x44\x96\x1C\xD8\xEB\x95\xD2" - "\xF3\x71\xEF\xEB\x4E\xBB\x4D\x29", - .ilen = 504, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" +/* + * Twofish test vectors. + */ +static const struct cipher_testvec tf_tv_template[] = { + { + .key = zeroed_string, + .klen = 16, + .ptext = zeroed_string, + .ctext = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" + "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a", + .len = 16, + }, { + .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\xfe\xdc\xba\x98\x76\x54\x32\x10" + "\x00\x11\x22\x33\x44\x55\x66\x77", + .klen = 24, + .ptext = zeroed_string, + .ctext = "\xcf\xd1\xd2\xe5\xa9\xbe\x9c\xdf" + "\x50\x1f\x13\xb8\x92\xbd\x22\x48", + .len = 16, + }, { + .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\xfe\xdc\xba\x98\x76\x54\x32\x10" + "\x00\x11\x22\x33\x44\x55\x66\x77" + "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", + .klen = 32, + .ptext = zeroed_string, + .ctext = "\x37\x52\x7b\xe0\x05\x23\x34\xb8" + "\x9f\x0c\xfc\xca\xe8\x7c\xfa\x20", + .len = 16, + }, { /* Generated with Crypto++ */ + .key = "\x3F\x85\x62\x3F\x1C\xF9\xD6\x1C" + "\xF9\xD6\xB3\x90\x6D\x4A\x90\x6D" + "\x4A\x27\x04\xE1\x27\x04\xE1\xBE" + "\x9B\x78\xBE\x9B\x78\x55\x32\x0F", + .klen = 32, + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -8845,81 +7455,124 @@ static const struct cipher_testvec bf_ctr_dec_tv_template[] = { "\x86\x1D\xB4\x28\xBF\x56\xED\x61" "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59\xF0\x64\xFB\x92\x06", - .rlen = 504, + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\x88\xCB\x1E\xC2\xAF\x8A\x97\xFF" + "\xF6\x90\x46\x9C\x4A\x0F\x08\xDC" + "\xDE\xAB\xAD\xFA\xFC\xA8\xC2\x3D" + "\xE0\xE4\x8B\x3F\xD5\xA3\xF7\x14" + "\x34\x9E\xB6\x08\xB2\xDD\xA8\xF5" + "\xDF\xFA\xC7\xE8\x09\x50\x76\x08" + "\xA2\xB6\x6A\x59\xC0\x2B\x6D\x05" + "\x89\xF6\x82\xF0\xD3\xDB\x06\x02" + "\xB5\x11\x5C\x5E\x79\x1A\xAC\x43" + "\x5C\xC0\x30\x4B\x6B\x16\xA1\x40" + "\x80\x27\x88\xBA\x2C\x74\x42\xE0" + "\x1B\xA5\x85\x08\xB9\xE6\x22\x7A" + "\x36\x3B\x0D\x9F\xA0\x22\x6C\x2A" + "\x91\x75\x47\xBC\x67\x21\x4E\xF9" + "\xEA\xFF\xD9\xD5\xC0\xFC\x9E\x2C" + "\x3E\xAD\xC6\x61\x0E\x93\x7A\x22" + "\x09\xC8\x8D\xC1\x8E\xB4\x8B\x5C" + "\xC6\x24\x42\xB8\x23\x66\x80\xA9" + "\x32\x0B\x7A\x29\xBF\xB3\x0B\x63" + "\x43\x27\x13\xA9\xBE\xEB\xBD\xF3" + "\x33\x62\x70\xE2\x1B\x86\x7A\xA1" + "\x51\x4A\x16\xFE\x29\x63\x7E\xD0" + "\x7A\xA4\x6E\x2C\xF8\xC1\xDB\xE8" + "\xCB\x4D\xD2\x8C\x04\x14\xB4\x66" + "\x41\xB7\x3A\x96\x16\x7C\x1D\x5B" + "\xB6\x41\x42\x64\x43\xEE\x6E\x7C" + "\x8B\xAF\x01\x9C\xA4\x6E\x75\x8F" + "\xDE\x10\x9F\xA6\xE7\xD6\x44\x97" + "\x66\xA3\x96\x0F\x1C\x25\x60\xF5" + "\x3C\x2E\x32\x69\x0E\x82\xFF\x27" + "\x0F\xB5\x06\xDA\xD8\x31\x15\x6C" + "\xDF\x18\x6C\x87\xF5\x3B\x11\x9A" + "\x1B\x42\x1F\x5B\x29\x19\x96\x13" + "\x68\x2E\x5E\x08\x1C\x8F\x32\x4B" + "\x81\x77\x6D\xF4\xA0\x01\x42\xEC" + "\xDD\x5B\xFD\x3A\x8E\x6A\x14\xFB" + "\x83\x54\xDF\x0F\x86\xB7\xEA\x40" + "\x46\x39\xF7\x2A\x89\x8D\x4E\x96" + "\x5F\x5F\x6D\x76\xC6\x13\x9D\x3D" + "\x1D\x5F\x0C\x7D\xE2\xBC\xC2\x16" + "\x16\xBE\x89\x3E\xB0\x61\xA2\x5D" + "\xAF\xD1\x40\x5F\x1A\xB8\x26\x41" + "\xC6\xBD\x36\xEF\xED\x29\x50\x6D" + "\x10\xEF\x26\xE8\xA8\x93\x11\x3F" + "\x2D\x1F\x88\x20\x77\x45\xF5\x66" + "\x08\xB9\xF1\xEF\xB1\x93\xA8\x81" + "\x65\xC5\xCD\x3E\x8C\x06\x60\x2C" + "\xB2\x10\x7A\xCA\x05\x25\x59\xDB" + "\xC7\x28\xF5\x20\x35\x52\x9E\x62" + "\xF8\x88\x24\x1C\x4D\x84\x12\x39" + "\x39\xE4\x2E\xF4\xD4\x9D\x2B\xBC" + "\x87\x66\xE6\xC0\x6B\x31\x9A\x66" + "\x03\xDC\x95\xD8\x6B\xD0\x30\x8F" + "\xDF\x8F\x8D\xFA\xEC\x1F\x08\xBD" + "\xA3\x63\xE2\x71\x4F\x03\x94\x87" + "\x50\xDF\x15\x1F\xED\x3A\xA3\x7F" + "\x1F\x2A\xB5\xA1\x69\xAC\x4B\x0D" + "\x84\x9B\x2A\xE9\x55\xDD\x46\x91" + "\x15\x33\xF3\x2B\x9B\x46\x97\x00" + "\xF0\x29\xD8\x59\x5D\x33\x37\xF9" + "\x58\x33\x9B\x78\xC7\x58\x48\x6B" + "\x2C\x75\x64\xC4\xCA\xC1\x7E\xD5", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec tf_cbc_tv_template[] = { + { /* Generated with Nettle */ + .key = zeroed_string, + .klen = 16, + .iv = zeroed_string, + .ptext = zeroed_string, + .ctext = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" + "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a", + .len = 16, + }, { + .key = zeroed_string, + .klen = 16, + .iv = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" + "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a", + .ptext = zeroed_string, + .ctext = "\xd4\x91\xdb\x16\xe7\xb1\xc3\x9e" + "\x86\xcb\x08\x6b\x78\x9f\x54\x19", + .len = 16, + }, { + .key = zeroed_string, + .klen = 16, + .iv = "\xd4\x91\xdb\x16\xe7\xb1\xc3\x9e" + "\x86\xcb\x08\x6b\x78\x9f\x54\x19", + .ptext = zeroed_string, + .ctext = "\x05\xef\x8c\x61\xa8\x11\x58\x26" + "\x34\xba\x5c\xb7\x10\x6a\xa6\x41", + .len = 16, + }, { + .key = zeroed_string, + .klen = 16, + .iv = zeroed_string, + .ptext = zeroed_string, + .ctext = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" + "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a" + "\xd4\x91\xdb\x16\xe7\xb1\xc3\x9e" + "\x86\xcb\x08\x6b\x78\x9f\x54\x19" + "\x05\xef\x8c\x61\xa8\x11\x58\x26" + "\x34\xba\x5c\xb7\x10\x6a\xa6\x41", + .len = 48, }, { /* Generated with Crypto++ */ .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\xC7\xA3\xDF\xB9\x05\xF4\x9E\x8D" - "\x9E\xDF\x38\x18\x83\x07\xEF\xC1" - "\x93\x3C\xAA\xAA\xFE\x06\x42\xCC" - "\x0D\x70\x86\x5A\x44\xAD\x85\x17" - "\xE4\x1F\x5E\xA5\x89\xAC\x32\xBC" - "\x3D\xA7\xE9\x0A\x5C\x70\x4D\xDE" - "\x99\x38\x07\xCA\x1D\x21\xC1\x11" - "\x97\xEB\x98\x75\xC4\x73\x45\x83" - "\x46\x1C\x9C\x91\x87\xC1\xA0\x56" - "\x98\xA1\x8B\xDB\x22\x76\xBD\x62" - "\xA4\xBC\xE8\x86\xDA\xD2\x51\x13" - "\x13\xD2\x96\x68\x69\x10\x67\x0C" - "\xD0\x17\x25\x7C\xB2\xAE\x4F\x93" - "\xA6\x82\x20\xCF\x0F\xA6\x47\x79" - "\x88\x09\x40\x59\xBD\x12\x64\xB5" - "\x19\x38\x0D\xFF\x86\xD9\x42\x20" - "\x81\x0D\x96\x99\xAF\x22\x1F\x94" - "\x5C\x6E\xEC\xEA\xA3\x39\xCB\x09" - "\x43\x19\x7F\xD0\xBB\x10\xC2\x49" - "\xF7\xE9\xF2\xEE\xBF\xF7\xF8\xB3" - "\x0E\x1A\xF1\x8D\x70\x82\x0C\x04" - "\xFD\x29\x1A\xAC\xC0\x92\x48\x34" - "\x6A\xE3\x1D\x4F\xFC\x1C\x72\x6A" - "\x57\xCB\xAD\xD0\x98\xAB\xB1\x01" - "\x03\x6A\x45\xDD\x07\x71\x5F\x5B" - "\xB5\x4A\xE4\xE5\xB9\xB9\xBC\xAC" - "\x44\xF7\x41\xA4\x5F\x2E\xE9\x28" - "\xE3\x05\xD2\x94\x78\x4C\x33\x1B" - "\xBD\xC1\x6E\x51\xD9\xAD\xD9\x86" - "\x15\x4A\x78\xAE\x7B\xAD\x3B\xBC" - "\x2F\xE0\x0E\xC5\x7B\x54\x97\x5F" - "\x60\x51\x14\x65\xF9\x91\xE9\xDA" - "\x9A\xBC\xFC\x19\x29\x67\xAA\x63" - "\x5E\xF2\x48\x88\xEB\x79\xE1\xE4" - "\xF7\xF6\x4C\xA9\xE2\x8C\x3B\xE0" - "\xED\x52\xAE\x90\x8F\x5B\x98\x34" - "\x29\x94\x34\x7F\xF9\x6C\x1E\xB6" - "\xA4\xE7\x2D\x06\x54\x9D\xC3\x02" - "\xC1\x90\xA4\x72\x31\x6B\x24\x51" - "\x0B\xB3\x7C\x63\x15\xBA\xAF\x5D" - "\x41\xE0\x37\x6D\xBE\x41\x58\xDE" - "\xF2\x07\x62\x99\xBE\xC1\x8C\x0F" - "\x0F\x28\xFB\x8F\x0E\x1D\x91\xE2" - "\xDA\x99\x5C\x49\xBA\x9C\xA8\x86" - "\x82\x63\x11\xB3\x54\x49\x00\x08" - "\x07\xF2\xE8\x1F\x34\x49\x61\xF4" - "\x81\xE9\xF6\xA9\x5A\x28\x60\x1F" - "\x66\x99\x08\x06\xF2\xE8\x2D\xD1" - "\xD0\x67\xBA\x32\x1F\x02\x86\x7B" - "\xFB\x79\x3D\xC5\xB1\x7F\x15\xAF" - "\xD7\xBF\x31\x46\x22\x7F\xAE\x5B" - "\x8B\x95\x47\xC2\xB1\x62\xA1\xCE" - "\x52\xAC\x9C\x8B\xC2\x49\x7F\xBC" - "\x9C\x89\xB8\xB6\xCA\xE3\x8F\xEA" - "\xAC\xB4\x5D\xE4\x50\xDC\x3A\xB5" - "\x91\x04\x94\x99\x03\x3B\x42\x6D" - "\x9C\x4A\x02\xF5\xB5\x38\x98\xA8" - "\x5C\x97\x2E\x4D\x79\x67\x71\xAF" - "\xF0\x70\x77\xFF\x2D\xDA\xA0\x9E" - "\x23\x8D\xD6\xA6\x68\x10\x78\x9A" - "\x64\xBB\x15\xB8\x56\xCF\xEE\xE5" - "\x32\x44\x96\x1C\xD8\xEB\x95\xD2" - "\xF3\x71\xEF\xEB\x4E\xBB\x4D", - .ilen = 503, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -8980,84 +7633,86 @@ static const struct cipher_testvec bf_ctr_dec_tv_template[] = { "\x86\x1D\xB4\x28\xBF\x56\xED\x61" "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59\xF0\x64\xFB\x92", - .rlen = 503, - .also_non_np = 1, - .np = 2, - .tap = { 503 - 8, 8 }, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x5F\x58\x6E\x60\x51\x6E\xDC\x3D" - "\xD1\xBB\xF7\xB7\xFD\x04\x44\x82" - "\xDC\x9F\x4B\x02\xF1\xD2\x5A\x6F" - "\x25\xF9\x27\x21\xF2\xD2\x9A\x01" - "\xBD\xAD\x3D\x93\x87\xCA\x0D\xFE" - "\xB7\x2C\x17\x1F\x42\x8C\x13\xB2" - "\x62\x44\x72\xB9\x5D\xC0\xF8\x37" - "\xDF\xEA\x78\x81\x8F\xA6\x34\xB2" - "\x07\x09\x7C\xB9\x3A\xA0\x2B\x18" - "\x34\x6A\x9D\x3D\xA5\xEB\xF4\x60" - "\xF8\x98\xA2\x39\x81\x23\x6C\xA9" - "\x70\xCA\xCC\x45\xD8\x1F\xDF\x44" - "\x2A\x67\x7A\x88\x28\xDC\x36\x83" - "\x18\xD7\x48\x43\x17\x2B\x1B\xE6" - "\x0B\x82\x59\x14\x26\x67\x08\x09" - "\x5B\x5D\x38\xD0\x81\xCE\x54\x2A" - "\xCD\x22\x94\x42\xF5\xBA\x74\x7E" - "\xD9\x00\x40\xA9\x0D\x0B\xBD\x8E" - "\xC4\x8E\x5E\x17\x8F\x48\xE2\xB8" - "\xF4\xCC\x19\x76\xAB\x48\x29\xAA" - "\x81\xD5\xCE\xD5\x8A\x3B\xC9\x21" - "\xEF\x50\x4F\x04\x02\xBF\xE1\x1F" - "\x59\x28\x1A\xE4\x18\x16\xA0\x29" - "\xBF\x34\xA9\x2D\x28\x83\xC0\x5E" - "\xEA\x44\xC4\x6E\xAB\x24\x79\x9D" - "\x2D\xA1\xE8\x55\xCA\x74\xFC\xBD" - "\xFE\xDD\xDA\xA5\xFB\x34\x90\x31" - "\x0E\x62\x28\x9B\xDC\xD7\xA1\xBB" - "\xF0\x1A\xB3\xE2\xD0\xFA\xBD\xE8" - "\x5C\x5A\x10\x67\xF6\x6A\x17\x3F" - "\xC5\xE9\x09\x08\xDD\x22\x77\x42" - "\x26\x6A\x6A\x7A\x3F\x87\x80\x0C" - "\xF0\xFF\x15\x8E\x84\x86\xC0\x10" - "\x0F\x8D\x33\x06\xB8\x72\xA4\x47" - "\x6B\xED\x2E\x05\x94\x6C\x5C\x5B" - "\x13\xF6\x77\xEE\x3B\x16\xDF\xC2" - "\x63\x66\x07\x6D\x3F\x6C\x51\x7C" - "\x1C\xAC\x80\xB6\x58\x48\xB7\x9D" - "\xB4\x19\xD8\x19\x45\x66\x27\x02" - "\xA1\xA9\x99\xF3\x1F\xE5\xA7\x1D" - "\x31\xE7\x1B\x0D\xFF\xBB\xB5\xA1" - "\xF5\x9C\x45\x1E\x18\x19\xA1\xE7" - "\xC2\xF1\xBF\x68\xC3\xEC\xCF\x53" - "\x67\xA6\x2B\x7D\x3C\x6D\x24\xC3" - "\xE8\xE6\x07\x5A\x09\xE0\x32\xA8" - "\x52\xF6\xE9\xED\x0E\xC6\x0A\x6A" - "\xFC\x60\x2A\xE0\x93\xCE\xB8\x2E" - "\xA2\xA8\x0E\x79\x9E\x34\x5D\x37" - "\x6F\x12\xFE\x48\x7B\xE7\xB9\x22" - "\x29\xE8\xD7\xBE\x5D\xD1\x8B\xD9" - "\x91\x51\x4E\x71\xF2\x98\x85\x16" - "\x25\x7A\x76\x8A\x51\x0E\x65\x14" - "\x81\xB5\x3A\x37\xFD\xEC\xB5\x8A" - "\xE1\xCF\x41\x72\x14\x29\x4C\xF0" - "\x20\xD9\x9A\xC5\x66\xA4\x03\x76" - "\x5B\xA4\x15\x4F\x0E\x64\x39\x40" - "\x25\xF9\x20\x22\xF5\x88\xF5\xBA" - "\xE4\xDF\x45\x61\xBF\x8D\x7A\x24" - "\x4B\x92\x71\xD9\x2F\x77\xA7\x95" - "\xA8\x7F\x61\xD5\xA4\x57\xB0\xFB" - "\xB5\x77\xBA\x1C\xEE\x71\xFA\xB0" - "\x16\x4C\x18\x6B\xF2\x69\xA0\x07" - "\xEF\xBE\xEC\x69\xAC\xA8\x63\x9E", - .ilen = 504, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\xC8\xFF\xF2\x53\xA6\x27\x09\xD1" + "\x33\x38\xC2\xC0\x0C\x14\x7E\xB5" + "\x26\x1B\x05\x0C\x05\x12\x3F\xC0" + "\xF9\x1C\x02\x28\x40\x96\x6F\xD0" + "\x3D\x32\xDF\xDA\x56\x00\x6E\xEE" + "\x5B\x2A\x72\x9D\xC2\x4D\x19\xBC" + "\x8C\x53\xFA\x87\x6F\xDD\x81\xA3" + "\xB1\xD3\x44\x65\xDF\xE7\x63\x38" + "\x4A\xFC\xDC\xEC\x3F\x26\x8E\xB8" + "\x43\xFC\xFE\x18\xB5\x11\x6D\x31" + "\x81\x8B\x0D\x75\xF6\x80\xEC\x84" + "\x04\xB9\xE6\x09\x63\xED\x39\xDB" + "\xC3\xF6\x14\xD6\x6E\x5E\x8B\xBD" + "\x3E\xFA\xD7\x98\x50\x6F\xD9\x63" + "\x02\xCD\x0D\x39\x4B\x0D\xEC\x80" + "\xE3\x6A\x17\xF4\xCC\xAD\xFF\x68" + "\x45\xDD\xC8\x83\x1D\x41\x96\x0D" + "\x91\x2E\x05\xD3\x59\x82\xE0\x43" + "\x90\x4F\xB9\xF7\xAD\x6B\x2E\xAF" + "\xA7\x84\x00\x53\xCD\x6F\xD1\x0C" + "\x4E\xF9\x5A\x23\xFB\xCA\xC7\xD3" + "\xA9\xAA\x9D\xB2\x3F\x66\xF1\xAC" + "\x25\x21\x8F\xF7\xEF\xF2\x6A\xDF" + "\xE8\xDA\x75\x1A\x8A\xF1\xDD\x38" + "\x1F\xF9\x3D\x68\x4A\xBB\x9E\x34" + "\x1F\x66\x1F\x9C\x2B\x54\xFF\x60" + "\x7F\x29\x4B\x55\x80\x8F\x4E\xA7" + "\xA6\x9A\x0A\xD9\x0D\x19\x00\xF8" + "\x1F\xBC\x0C\x40\x6B\xEC\x99\x25" + "\x94\x70\x74\x0E\x1D\xC5\xBC\x12" + "\xF3\x42\xBE\x95\xBF\xFB\x4E\x55" + "\x9A\xB9\xCE\x14\x16\x5B\xDC\xD3" + "\x75\x42\x62\x04\x31\x1F\x95\x7C" + "\x66\x1A\x97\xDC\x2F\x40\x5C\x39" + "\x78\xE6\x02\xDB\x49\xE1\xC6\x47" + "\xC2\x78\x9A\xBB\xF3\xBE\xCB\x93" + "\xD8\xB8\xE8\xBB\x8C\xB3\x9B\xA7" + "\xC2\x89\xF3\x91\x88\x83\x3D\xF0" + "\x29\xA2\xCD\xB5\x79\x16\xC2\x40" + "\x11\x03\x8E\x9C\xFD\xC9\x43\xC4" + "\xC2\x19\xF0\x4A\x32\xEF\x0C\x2B" + "\xD3\x2B\xE9\xD4\x4C\xDE\x95\xCF" + "\x04\x03\xD3\x2C\x7F\x82\xC8\xFA" + "\x0F\xD8\x7A\x39\x7B\x01\x41\x9C" + "\x78\xB6\xC9\xBF\xF9\x78\x57\x88" + "\xB1\xA5\xE1\xE0\xD9\x16\xD4\xC8" + "\xEE\xC4\xBE\x7B\x55\x59\x00\x48" + "\x1B\xBC\x14\xFA\x2A\x9D\xC9\x1C" + "\xFB\x28\x3F\x95\xDD\xB7\xD6\xCE" + "\x3A\x7F\x09\x0C\x0E\x69\x30\x7D" + "\xBC\x68\x9C\x91\x2A\x59\x57\x04" + "\xED\x1A\x1E\x00\xB1\x85\x92\x04" + "\x28\x8C\x0C\x3C\xC1\xD5\x12\xF7" + "\x4C\x3E\xB0\xE7\x86\x62\x68\x91" + "\xFC\xC4\xE2\xCE\xA6\xDC\x5E\x93" + "\x5D\x8D\x8C\x68\xB3\xB2\xB9\x64" + "\x16\xB8\xC8\x6F\xD8\xEE\x21\xBD" + "\xAC\x18\x0C\x7D\x0D\x05\xAB\xF1" + "\xFA\xDD\xE2\x48\xDF\x4C\x02\x39" + "\x69\xA1\x62\xBD\x49\x3A\x9D\x91" + "\x30\x70\x56\xA4\x37\xDD\x7C\xC0" + "\x0A\xA3\x30\x10\x26\x25\x41\x2C", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec tf_ctr_tv_template[] = { + { /* Generated with Crypto++ */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .klen = 32, + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -9118,52 +7773,79 @@ static const struct cipher_testvec bf_ctr_dec_tv_template[] = { "\x86\x1D\xB4\x28\xBF\x56\xED\x61" "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59\xF0\x64\xFB\x92\x06", - .rlen = 504, - }, -}; - -/* - * Twofish test vectors. - */ -static const struct cipher_testvec tf_enc_tv_template[] = { - { - .key = zeroed_string, - .klen = 16, - .input = zeroed_string, - .ilen = 16, - .result = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" - "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a", - .rlen = 16, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10" - "\x00\x11\x22\x33\x44\x55\x66\x77", - .klen = 24, - .input = zeroed_string, - .ilen = 16, - .result = "\xcf\xd1\xd2\xe5\xa9\xbe\x9c\xdf" - "\x50\x1f\x13\xb8\x92\xbd\x22\x48", - .rlen = 16, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10" - "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", - .klen = 32, - .input = zeroed_string, - .ilen = 16, - .result = "\x37\x52\x7b\xe0\x05\x23\x34\xb8" - "\x9f\x0c\xfc\xca\xe8\x7c\xfa\x20", - .rlen = 16, + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\xDF\xDD\x69\xFA\xB0\x2E\xFD\xFE" + "\x70\x9E\xC5\x4B\xC9\xD4\xA1\x30" + "\x26\x9B\x89\xA1\xEE\x43\xE0\x52" + "\x55\x17\x4E\xC7\x0E\x33\x1F\xF1" + "\x9F\x8D\x40\x9F\x24\xFD\x92\xA0" + "\xBC\x8F\x35\xDD\x67\x38\xD8\xAA" + "\xCF\xF8\x48\xCA\xFB\xE4\x5C\x60" + "\x01\x41\x21\x12\x38\xAB\x52\x4F" + "\xA8\x57\x20\xE0\x21\x6A\x17\x0D" + "\x0E\xF9\x8E\x49\x42\x00\x3C\x94" + "\x14\xC0\xD0\x8D\x8A\x98\xEB\x29" + "\xEC\xAE\x96\x44\xC0\x3C\x48\xDC" + "\x29\x35\x25\x2F\xE7\x11\x6C\x68" + "\xC8\x67\x0A\x2F\xF4\x07\xBE\xF9" + "\x2C\x31\x87\x40\xAB\xB2\xB6\xFA" + "\xD2\xC9\x6D\x5C\x50\xE9\xE6\x7E" + "\xE3\x0A\xD2\xD5\x6D\x8D\x64\x9E" + "\x70\xCE\x03\x76\xDD\xE0\xF0\x8C" + "\x84\x86\x8B\x6A\xFE\xC7\xF9\x69" + "\x2E\xFE\xFC\xC2\xC4\x1A\x55\x58" + "\xB3\xBE\xE2\x7E\xED\x39\x42\x6C" + "\xB4\x42\x97\x9A\xEC\xE1\x0A\x06" + "\x02\xC5\x03\x9D\xC4\x48\x15\x66" + "\x35\x6A\xC2\xC9\xA2\x26\x30\xBB" + "\xDB\x2D\xC8\x08\x2B\xA0\x29\x1A" + "\x23\x61\x48\xEA\x80\x04\x27\xAA" + "\x69\x49\xE8\xE8\x4A\x83\x6B\x5A" + "\xCA\x7C\xD3\xB1\xB5\x0B\xCC\x23" + "\x74\x1F\xA9\x87\xCD\xED\xC0\x2D" + "\xBF\xEB\xCF\x16\x2D\x2A\x2E\x1D" + "\x96\xBA\x36\x11\x45\x41\xDA\xCE" + "\xA4\x48\x80\x8B\x06\xF4\x98\x89" + "\x8B\x23\x08\x53\xF4\xD4\x5A\x24" + "\x8B\xF8\x43\x73\xD1\xEE\xC4\xB0" + "\xF8\xFE\x09\x0C\x75\x05\x38\x0B" + "\x7C\x81\xDE\x9D\xE4\x61\x37\x63" + "\x63\xAD\x12\xD2\x04\xB9\xCE\x45" + "\x5A\x1A\x6E\xB3\x78\x2A\xA4\x74" + "\x86\xD0\xE3\xFF\xDA\x38\x9C\xB5" + "\xB8\xB1\xDB\x38\x2F\xC5\x6A\xB4" + "\xEB\x6E\x96\xE8\x43\x80\xB5\x51" + "\x61\x2D\x48\xAA\x07\x65\x11\x8C" + "\x48\xE3\x90\x7E\x78\x3A\xEC\x97" + "\x05\x3D\x84\xE7\x90\x2B\xAA\xBD" + "\x83\x29\x0E\x1A\x81\x73\x7B\xE0" + "\x7A\x01\x4A\x37\x3B\x77\x7F\x8D" + "\x49\xA4\x2F\x6E\xBE\x68\x99\x08" + "\x99\xAA\x4C\x12\x04\xAE\x1F\x77" + "\x35\x88\xF1\x65\x06\x0A\x0B\x4D" + "\x47\xF9\x50\x38\x5D\x71\xF9\x6E" + "\xDE\xEC\x61\x35\x2C\x4C\x96\x50" + "\xE8\x28\x93\x9C\x7E\x01\xC6\x04" + "\xB2\xD6\xBC\x6C\x17\xEB\xC1\x7D" + "\x11\xE9\x43\x83\x76\xAA\x53\x37" + "\x0C\x1D\x39\x89\x53\x72\x09\x7E" + "\xD9\x85\x16\x04\xA5\x2C\x05\x6F" + "\x17\x0C\x6E\x66\xAA\x84\xA7\xD9" + "\xE2\xD9\xC4\xEB\x43\x3E\xB1\x8D" + "\x7C\x36\xC7\x71\x70\x9C\x10\xD8" + "\xE8\x47\x2A\x4D\xFD\xA1\xBC\xE3" + "\xB9\x32\xE2\xC1\x82\xAC\xFE\xCC" + "\xC5\xC9\x7F\x9E\xCF\x33\x7A\xDF", + .len = 496, }, { /* Generated with Crypto++ */ - .key = "\x3F\x85\x62\x3F\x1C\xF9\xD6\x1C" - "\xF9\xD6\xB3\x90\x6D\x4A\x90\x6D" - "\x4A\x27\x04\xE1\x27\x04\xE1\xBE" - "\x9B\x78\xBE\x9B\x78\x55\x32\x0F", + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" + "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -9225,176 +7907,78 @@ static const struct cipher_testvec tf_enc_tv_template[] = { "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\x88\xCB\x1E\xC2\xAF\x8A\x97\xFF" - "\xF6\x90\x46\x9C\x4A\x0F\x08\xDC" - "\xDE\xAB\xAD\xFA\xFC\xA8\xC2\x3D" - "\xE0\xE4\x8B\x3F\xD5\xA3\xF7\x14" - "\x34\x9E\xB6\x08\xB2\xDD\xA8\xF5" - "\xDF\xFA\xC7\xE8\x09\x50\x76\x08" - "\xA2\xB6\x6A\x59\xC0\x2B\x6D\x05" - "\x89\xF6\x82\xF0\xD3\xDB\x06\x02" - "\xB5\x11\x5C\x5E\x79\x1A\xAC\x43" - "\x5C\xC0\x30\x4B\x6B\x16\xA1\x40" - "\x80\x27\x88\xBA\x2C\x74\x42\xE0" - "\x1B\xA5\x85\x08\xB9\xE6\x22\x7A" - "\x36\x3B\x0D\x9F\xA0\x22\x6C\x2A" - "\x91\x75\x47\xBC\x67\x21\x4E\xF9" - "\xEA\xFF\xD9\xD5\xC0\xFC\x9E\x2C" - "\x3E\xAD\xC6\x61\x0E\x93\x7A\x22" - "\x09\xC8\x8D\xC1\x8E\xB4\x8B\x5C" - "\xC6\x24\x42\xB8\x23\x66\x80\xA9" - "\x32\x0B\x7A\x29\xBF\xB3\x0B\x63" - "\x43\x27\x13\xA9\xBE\xEB\xBD\xF3" - "\x33\x62\x70\xE2\x1B\x86\x7A\xA1" - "\x51\x4A\x16\xFE\x29\x63\x7E\xD0" - "\x7A\xA4\x6E\x2C\xF8\xC1\xDB\xE8" - "\xCB\x4D\xD2\x8C\x04\x14\xB4\x66" - "\x41\xB7\x3A\x96\x16\x7C\x1D\x5B" - "\xB6\x41\x42\x64\x43\xEE\x6E\x7C" - "\x8B\xAF\x01\x9C\xA4\x6E\x75\x8F" - "\xDE\x10\x9F\xA6\xE7\xD6\x44\x97" - "\x66\xA3\x96\x0F\x1C\x25\x60\xF5" - "\x3C\x2E\x32\x69\x0E\x82\xFF\x27" - "\x0F\xB5\x06\xDA\xD8\x31\x15\x6C" - "\xDF\x18\x6C\x87\xF5\x3B\x11\x9A" - "\x1B\x42\x1F\x5B\x29\x19\x96\x13" - "\x68\x2E\x5E\x08\x1C\x8F\x32\x4B" - "\x81\x77\x6D\xF4\xA0\x01\x42\xEC" - "\xDD\x5B\xFD\x3A\x8E\x6A\x14\xFB" - "\x83\x54\xDF\x0F\x86\xB7\xEA\x40" - "\x46\x39\xF7\x2A\x89\x8D\x4E\x96" - "\x5F\x5F\x6D\x76\xC6\x13\x9D\x3D" - "\x1D\x5F\x0C\x7D\xE2\xBC\xC2\x16" - "\x16\xBE\x89\x3E\xB0\x61\xA2\x5D" - "\xAF\xD1\x40\x5F\x1A\xB8\x26\x41" - "\xC6\xBD\x36\xEF\xED\x29\x50\x6D" - "\x10\xEF\x26\xE8\xA8\x93\x11\x3F" - "\x2D\x1F\x88\x20\x77\x45\xF5\x66" - "\x08\xB9\xF1\xEF\xB1\x93\xA8\x81" - "\x65\xC5\xCD\x3E\x8C\x06\x60\x2C" - "\xB2\x10\x7A\xCA\x05\x25\x59\xDB" - "\xC7\x28\xF5\x20\x35\x52\x9E\x62" - "\xF8\x88\x24\x1C\x4D\x84\x12\x39" - "\x39\xE4\x2E\xF4\xD4\x9D\x2B\xBC" - "\x87\x66\xE6\xC0\x6B\x31\x9A\x66" - "\x03\xDC\x95\xD8\x6B\xD0\x30\x8F" - "\xDF\x8F\x8D\xFA\xEC\x1F\x08\xBD" - "\xA3\x63\xE2\x71\x4F\x03\x94\x87" - "\x50\xDF\x15\x1F\xED\x3A\xA3\x7F" - "\x1F\x2A\xB5\xA1\x69\xAC\x4B\x0D" - "\x84\x9B\x2A\xE9\x55\xDD\x46\x91" - "\x15\x33\xF3\x2B\x9B\x46\x97\x00" - "\xF0\x29\xD8\x59\x5D\x33\x37\xF9" - "\x58\x33\x9B\x78\xC7\x58\x48\x6B" - "\x2C\x75\x64\xC4\xCA\xC1\x7E\xD5", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec tf_dec_tv_template[] = { - { - .key = zeroed_string, - .klen = 16, - .input = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" - "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10" - "\x00\x11\x22\x33\x44\x55\x66\x77", - .klen = 24, - .input = "\xcf\xd1\xd2\xe5\xa9\xbe\x9c\xdf" - "\x50\x1f\x13\xb8\x92\xbd\x22\x48", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10" - "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", - .klen = 32, - .input = "\x37\x52\x7b\xe0\x05\x23\x34\xb8" - "\x9f\x0c\xfc\xca\xe8\x7c\xfa\x20", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, + .ctext = "\xEB\x44\xAF\x49\x27\xB8\xFB\x44" + "\x4C\xA6\xC3\x0C\x8B\xD0\x01\x0C" + "\x53\xC8\x16\x38\xDE\x40\x4F\x91" + "\x25\x6D\x4C\xA0\x9A\x87\x1E\xDA" + "\x88\x7E\x89\xE9\x67\x2B\x83\xA2" + "\x5F\x2E\x23\x3E\x45\xB9\x77\x7B" + "\xA6\x7E\x47\x36\x81\x9F\x9B\xF3" + "\xE0\xF0\xD7\x47\xA9\xC8\xEF\x33" + "\x0C\x43\xFE\x67\x50\x0A\x2C\x3E" + "\xA0\xE1\x25\x8E\x80\x07\x4A\xC0" + "\x64\x89\x9F\x6A\x27\x96\x07\xA6" + "\x9B\xC8\x1B\x21\x60\xAE\x5D\x01" + "\xE2\xCD\xC8\xAA\x6C\x9D\x1C\x34" + "\x39\x18\x09\xA4\x82\x59\x78\xE7" + "\xFC\x59\x65\xF2\x94\xFF\xFB\xE2" + "\x3C\xDA\xB1\x90\x95\xBF\x91\xE3" + "\xE6\x87\x31\x9E\x16\x85\xAD\xB1" + "\x4C\xAE\x43\x4D\x19\x58\xB5\x5E" + "\x2E\xF5\x09\xAA\x39\xF4\xC0\xB3" + "\xD4\x4D\xDB\x73\x7A\xD4\xF1\xBF" + "\x89\x16\x4D\x2D\xA2\x26\x33\x72" + "\x18\x33\x7E\xD6\xD2\x16\xA4\x54" + "\xF4\x8C\xB3\x52\xDF\x21\x9C\xEB" + "\xBF\x49\xD3\xF9\x05\x06\xCB\xD2" + "\xA9\xD2\x3B\x6E\x19\x8C\xBC\x19" + "\xAB\x89\xD6\xD8\xCD\x56\x89\x5E" + "\xAC\x00\xE3\x50\x63\x4A\x80\x9A" + "\x05\xBC\x50\x39\xD3\x32\xD9\x0D" + "\xE3\x20\x0D\x75\x54\xEC\xE6\x31" + "\x14\xB9\x3A\x59\x00\x43\x37\x8E" + "\x8C\x5A\x79\x62\x14\x76\x8A\xAE" + "\x8F\xCC\xA1\x6C\x38\x78\xDD\x2D" + "\x8B\x6D\xEA\xBD\x7B\x25\xFF\x60" + "\xC9\x87\xB1\x79\x1E\xA5\x86\x68" + "\x81\xB4\xE2\xC1\x05\x7D\x3A\x73" + "\xD0\xDA\x75\x77\x9E\x05\x27\xF1" + "\x08\xA9\x66\x64\x6C\xBC\x82\x17" + "\x2C\x23\x5F\x62\x4D\x02\x1A\x58" + "\xE7\xB7\x23\x6D\xE2\x20\xDA\xEF" + "\xB4\xB3\x3F\xB2\x2B\x69\x98\x83" + "\x95\x87\x13\x57\x60\xD7\xB5\xB1" + "\xEE\x0A\x2F\x95\x36\x4C\x76\x5D" + "\x5F\xD9\x19\xED\xB9\xA5\x48\xBF" + "\xC8\xAB\x0F\x71\xCC\x61\x8E\x0A" + "\xD0\x29\x44\xA8\xB9\xC1\xE8\xC8" + "\xC9\xA8\x28\x81\xFB\x50\xF2\xF0" + "\x26\xAE\x39\xB8\x91\xCD\xA8\xAC" + "\xDE\x55\x1B\x50\x14\x53\x44\x17" + "\x54\x46\xFC\xB1\xE4\x07\x6B\x9A" + "\x01\x14\xF0\x2E\x2E\xDB\x46\x1B" + "\x1A\x09\x97\xA9\xB6\x97\x79\x06" + "\xFB\xCB\x85\xCF\xDD\xA1\x41\xB1" + "\x00\xAA\xF7\xE0\x89\x73\xFB\xE5" + "\xBF\x84\xDB\xC9\xCD\xC4\xA2\x0D" + "\x3B\xAC\xF9\xDF\x96\xBF\x88\x23" + "\x41\x67\xA1\x24\x99\x7E\xCC\x9B" + "\x02\x8F\x6A\x49\xF6\x25\xBA\x7A" + "\xF4\x78\xFD\x79\x62\x63\x4F\x14" + "\xD6\x11\x11\x04\x05\x5F\x7E\xEA" + "\x4C\xB6\xF8\xF4\x5F\x48\x52\x54" + "\x94\x63\xA8\x4E\xCF\xD2\x1B\x1B" + "\x22\x18\x6A\xAF\x6E\x3E\xE1\x0D", + .len = 496, }, { /* Generated with Crypto++ */ - .key = "\x3F\x85\x62\x3F\x1C\xF9\xD6\x1C" - "\xF9\xD6\xB3\x90\x6D\x4A\x90\x6D" - "\x4A\x27\x04\xE1\x27\x04\xE1\xBE" - "\x9B\x78\xBE\x9B\x78\x55\x32\x0F", + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, - .input = "\x88\xCB\x1E\xC2\xAF\x8A\x97\xFF" - "\xF6\x90\x46\x9C\x4A\x0F\x08\xDC" - "\xDE\xAB\xAD\xFA\xFC\xA8\xC2\x3D" - "\xE0\xE4\x8B\x3F\xD5\xA3\xF7\x14" - "\x34\x9E\xB6\x08\xB2\xDD\xA8\xF5" - "\xDF\xFA\xC7\xE8\x09\x50\x76\x08" - "\xA2\xB6\x6A\x59\xC0\x2B\x6D\x05" - "\x89\xF6\x82\xF0\xD3\xDB\x06\x02" - "\xB5\x11\x5C\x5E\x79\x1A\xAC\x43" - "\x5C\xC0\x30\x4B\x6B\x16\xA1\x40" - "\x80\x27\x88\xBA\x2C\x74\x42\xE0" - "\x1B\xA5\x85\x08\xB9\xE6\x22\x7A" - "\x36\x3B\x0D\x9F\xA0\x22\x6C\x2A" - "\x91\x75\x47\xBC\x67\x21\x4E\xF9" - "\xEA\xFF\xD9\xD5\xC0\xFC\x9E\x2C" - "\x3E\xAD\xC6\x61\x0E\x93\x7A\x22" - "\x09\xC8\x8D\xC1\x8E\xB4\x8B\x5C" - "\xC6\x24\x42\xB8\x23\x66\x80\xA9" - "\x32\x0B\x7A\x29\xBF\xB3\x0B\x63" - "\x43\x27\x13\xA9\xBE\xEB\xBD\xF3" - "\x33\x62\x70\xE2\x1B\x86\x7A\xA1" - "\x51\x4A\x16\xFE\x29\x63\x7E\xD0" - "\x7A\xA4\x6E\x2C\xF8\xC1\xDB\xE8" - "\xCB\x4D\xD2\x8C\x04\x14\xB4\x66" - "\x41\xB7\x3A\x96\x16\x7C\x1D\x5B" - "\xB6\x41\x42\x64\x43\xEE\x6E\x7C" - "\x8B\xAF\x01\x9C\xA4\x6E\x75\x8F" - "\xDE\x10\x9F\xA6\xE7\xD6\x44\x97" - "\x66\xA3\x96\x0F\x1C\x25\x60\xF5" - "\x3C\x2E\x32\x69\x0E\x82\xFF\x27" - "\x0F\xB5\x06\xDA\xD8\x31\x15\x6C" - "\xDF\x18\x6C\x87\xF5\x3B\x11\x9A" - "\x1B\x42\x1F\x5B\x29\x19\x96\x13" - "\x68\x2E\x5E\x08\x1C\x8F\x32\x4B" - "\x81\x77\x6D\xF4\xA0\x01\x42\xEC" - "\xDD\x5B\xFD\x3A\x8E\x6A\x14\xFB" - "\x83\x54\xDF\x0F\x86\xB7\xEA\x40" - "\x46\x39\xF7\x2A\x89\x8D\x4E\x96" - "\x5F\x5F\x6D\x76\xC6\x13\x9D\x3D" - "\x1D\x5F\x0C\x7D\xE2\xBC\xC2\x16" - "\x16\xBE\x89\x3E\xB0\x61\xA2\x5D" - "\xAF\xD1\x40\x5F\x1A\xB8\x26\x41" - "\xC6\xBD\x36\xEF\xED\x29\x50\x6D" - "\x10\xEF\x26\xE8\xA8\x93\x11\x3F" - "\x2D\x1F\x88\x20\x77\x45\xF5\x66" - "\x08\xB9\xF1\xEF\xB1\x93\xA8\x81" - "\x65\xC5\xCD\x3E\x8C\x06\x60\x2C" - "\xB2\x10\x7A\xCA\x05\x25\x59\xDB" - "\xC7\x28\xF5\x20\x35\x52\x9E\x62" - "\xF8\x88\x24\x1C\x4D\x84\x12\x39" - "\x39\xE4\x2E\xF4\xD4\x9D\x2B\xBC" - "\x87\x66\xE6\xC0\x6B\x31\x9A\x66" - "\x03\xDC\x95\xD8\x6B\xD0\x30\x8F" - "\xDF\x8F\x8D\xFA\xEC\x1F\x08\xBD" - "\xA3\x63\xE2\x71\x4F\x03\x94\x87" - "\x50\xDF\x15\x1F\xED\x3A\xA3\x7F" - "\x1F\x2A\xB5\xA1\x69\xAC\x4B\x0D" - "\x84\x9B\x2A\xE9\x55\xDD\x46\x91" - "\x15\x33\xF3\x2B\x9B\x46\x97\x00" - "\xF0\x29\xD8\x59\x5D\x33\x37\xF9" - "\x58\x33\x9B\x78\xC7\x58\x48\x6B" - "\x2C\x75\x64\xC4\xCA\xC1\x7E\xD5", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -9455,20657 +8039,1063 @@ static const struct cipher_testvec tf_dec_tv_template[] = { "\x86\x1D\xB4\x28\xBF\x56\xED\x61" "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec tf_cbc_enc_tv_template[] = { - { /* Generated with Nettle */ - .key = zeroed_string, - .klen = 16, - .iv = zeroed_string, - .input = zeroed_string, - .ilen = 16, - .result = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" - "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a", - .rlen = 16, - }, { - .key = zeroed_string, - .klen = 16, - .iv = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" - "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a", - .input = zeroed_string, - .ilen = 16, - .result = "\xd4\x91\xdb\x16\xe7\xb1\xc3\x9e" - "\x86\xcb\x08\x6b\x78\x9f\x54\x19", - .rlen = 16, - }, { - .key = zeroed_string, - .klen = 16, - .iv = "\xd4\x91\xdb\x16\xe7\xb1\xc3\x9e" - "\x86\xcb\x08\x6b\x78\x9f\x54\x19", - .input = zeroed_string, - .ilen = 16, - .result = "\x05\xef\x8c\x61\xa8\x11\x58\x26" - "\x34\xba\x5c\xb7\x10\x6a\xa6\x41", - .rlen = 16, - }, { - .key = zeroed_string, - .klen = 16, - .iv = zeroed_string, - .input = zeroed_string, - .ilen = 48, - .result = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" - "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a" - "\xd4\x91\xdb\x16\xe7\xb1\xc3\x9e" - "\x86\xcb\x08\x6b\x78\x9f\x54\x19" - "\x05\xef\x8c\x61\xa8\x11\x58\x26" - "\x34\xba\x5c\xb7\x10\x6a\xa6\x41", - .rlen = 48, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\xC8\xFF\xF2\x53\xA6\x27\x09\xD1" - "\x33\x38\xC2\xC0\x0C\x14\x7E\xB5" - "\x26\x1B\x05\x0C\x05\x12\x3F\xC0" - "\xF9\x1C\x02\x28\x40\x96\x6F\xD0" - "\x3D\x32\xDF\xDA\x56\x00\x6E\xEE" - "\x5B\x2A\x72\x9D\xC2\x4D\x19\xBC" - "\x8C\x53\xFA\x87\x6F\xDD\x81\xA3" - "\xB1\xD3\x44\x65\xDF\xE7\x63\x38" - "\x4A\xFC\xDC\xEC\x3F\x26\x8E\xB8" - "\x43\xFC\xFE\x18\xB5\x11\x6D\x31" - "\x81\x8B\x0D\x75\xF6\x80\xEC\x84" - "\x04\xB9\xE6\x09\x63\xED\x39\xDB" - "\xC3\xF6\x14\xD6\x6E\x5E\x8B\xBD" - "\x3E\xFA\xD7\x98\x50\x6F\xD9\x63" - "\x02\xCD\x0D\x39\x4B\x0D\xEC\x80" - "\xE3\x6A\x17\xF4\xCC\xAD\xFF\x68" - "\x45\xDD\xC8\x83\x1D\x41\x96\x0D" - "\x91\x2E\x05\xD3\x59\x82\xE0\x43" - "\x90\x4F\xB9\xF7\xAD\x6B\x2E\xAF" - "\xA7\x84\x00\x53\xCD\x6F\xD1\x0C" - "\x4E\xF9\x5A\x23\xFB\xCA\xC7\xD3" - "\xA9\xAA\x9D\xB2\x3F\x66\xF1\xAC" - "\x25\x21\x8F\xF7\xEF\xF2\x6A\xDF" - "\xE8\xDA\x75\x1A\x8A\xF1\xDD\x38" - "\x1F\xF9\x3D\x68\x4A\xBB\x9E\x34" - "\x1F\x66\x1F\x9C\x2B\x54\xFF\x60" - "\x7F\x29\x4B\x55\x80\x8F\x4E\xA7" - "\xA6\x9A\x0A\xD9\x0D\x19\x00\xF8" - "\x1F\xBC\x0C\x40\x6B\xEC\x99\x25" - "\x94\x70\x74\x0E\x1D\xC5\xBC\x12" - "\xF3\x42\xBE\x95\xBF\xFB\x4E\x55" - "\x9A\xB9\xCE\x14\x16\x5B\xDC\xD3" - "\x75\x42\x62\x04\x31\x1F\x95\x7C" - "\x66\x1A\x97\xDC\x2F\x40\x5C\x39" - "\x78\xE6\x02\xDB\x49\xE1\xC6\x47" - "\xC2\x78\x9A\xBB\xF3\xBE\xCB\x93" - "\xD8\xB8\xE8\xBB\x8C\xB3\x9B\xA7" - "\xC2\x89\xF3\x91\x88\x83\x3D\xF0" - "\x29\xA2\xCD\xB5\x79\x16\xC2\x40" - "\x11\x03\x8E\x9C\xFD\xC9\x43\xC4" - "\xC2\x19\xF0\x4A\x32\xEF\x0C\x2B" - "\xD3\x2B\xE9\xD4\x4C\xDE\x95\xCF" - "\x04\x03\xD3\x2C\x7F\x82\xC8\xFA" - "\x0F\xD8\x7A\x39\x7B\x01\x41\x9C" - "\x78\xB6\xC9\xBF\xF9\x78\x57\x88" - "\xB1\xA5\xE1\xE0\xD9\x16\xD4\xC8" - "\xEE\xC4\xBE\x7B\x55\x59\x00\x48" - "\x1B\xBC\x14\xFA\x2A\x9D\xC9\x1C" - "\xFB\x28\x3F\x95\xDD\xB7\xD6\xCE" - "\x3A\x7F\x09\x0C\x0E\x69\x30\x7D" - "\xBC\x68\x9C\x91\x2A\x59\x57\x04" - "\xED\x1A\x1E\x00\xB1\x85\x92\x04" - "\x28\x8C\x0C\x3C\xC1\xD5\x12\xF7" - "\x4C\x3E\xB0\xE7\x86\x62\x68\x91" - "\xFC\xC4\xE2\xCE\xA6\xDC\x5E\x93" - "\x5D\x8D\x8C\x68\xB3\xB2\xB9\x64" - "\x16\xB8\xC8\x6F\xD8\xEE\x21\xBD" - "\xAC\x18\x0C\x7D\x0D\x05\xAB\xF1" - "\xFA\xDD\xE2\x48\xDF\x4C\x02\x39" - "\x69\xA1\x62\xBD\x49\x3A\x9D\x91" - "\x30\x70\x56\xA4\x37\xDD\x7C\xC0" - "\x0A\xA3\x30\x10\x26\x25\x41\x2C", - .rlen = 496, + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" + "\x2B\xC2\x59", + .ctext = "\xDF\xDD\x69\xFA\xB0\x2E\xFD\xFE" + "\x70\x9E\xC5\x4B\xC9\xD4\xA1\x30" + "\x26\x9B\x89\xA1\xEE\x43\xE0\x52" + "\x55\x17\x4E\xC7\x0E\x33\x1F\xF1" + "\x9F\x8D\x40\x9F\x24\xFD\x92\xA0" + "\xBC\x8F\x35\xDD\x67\x38\xD8\xAA" + "\xCF\xF8\x48\xCA\xFB\xE4\x5C\x60" + "\x01\x41\x21\x12\x38\xAB\x52\x4F" + "\xA8\x57\x20\xE0\x21\x6A\x17\x0D" + "\x0E\xF9\x8E\x49\x42\x00\x3C\x94" + "\x14\xC0\xD0\x8D\x8A\x98\xEB\x29" + "\xEC\xAE\x96\x44\xC0\x3C\x48\xDC" + "\x29\x35\x25\x2F\xE7\x11\x6C\x68" + "\xC8\x67\x0A\x2F\xF4\x07\xBE\xF9" + "\x2C\x31\x87\x40\xAB\xB2\xB6\xFA" + "\xD2\xC9\x6D\x5C\x50\xE9\xE6\x7E" + "\xE3\x0A\xD2\xD5\x6D\x8D\x64\x9E" + "\x70\xCE\x03\x76\xDD\xE0\xF0\x8C" + "\x84\x86\x8B\x6A\xFE\xC7\xF9\x69" + "\x2E\xFE\xFC\xC2\xC4\x1A\x55\x58" + "\xB3\xBE\xE2\x7E\xED\x39\x42\x6C" + "\xB4\x42\x97\x9A\xEC\xE1\x0A\x06" + "\x02\xC5\x03\x9D\xC4\x48\x15\x66" + "\x35\x6A\xC2\xC9\xA2\x26\x30\xBB" + "\xDB\x2D\xC8\x08\x2B\xA0\x29\x1A" + "\x23\x61\x48\xEA\x80\x04\x27\xAA" + "\x69\x49\xE8\xE8\x4A\x83\x6B\x5A" + "\xCA\x7C\xD3\xB1\xB5\x0B\xCC\x23" + "\x74\x1F\xA9\x87\xCD\xED\xC0\x2D" + "\xBF\xEB\xCF\x16\x2D\x2A\x2E\x1D" + "\x96\xBA\x36\x11\x45\x41\xDA\xCE" + "\xA4\x48\x80\x8B\x06\xF4\x98\x89" + "\x8B\x23\x08\x53\xF4\xD4\x5A\x24" + "\x8B\xF8\x43\x73\xD1\xEE\xC4\xB0" + "\xF8\xFE\x09\x0C\x75\x05\x38\x0B" + "\x7C\x81\xDE\x9D\xE4\x61\x37\x63" + "\x63\xAD\x12\xD2\x04\xB9\xCE\x45" + "\x5A\x1A\x6E\xB3\x78\x2A\xA4\x74" + "\x86\xD0\xE3\xFF\xDA\x38\x9C\xB5" + "\xB8\xB1\xDB\x38\x2F\xC5\x6A\xB4" + "\xEB\x6E\x96\xE8\x43\x80\xB5\x51" + "\x61\x2D\x48\xAA\x07\x65\x11\x8C" + "\x48\xE3\x90\x7E\x78\x3A\xEC\x97" + "\x05\x3D\x84\xE7\x90\x2B\xAA\xBD" + "\x83\x29\x0E\x1A\x81\x73\x7B\xE0" + "\x7A\x01\x4A\x37\x3B\x77\x7F\x8D" + "\x49\xA4\x2F\x6E\xBE\x68\x99\x08" + "\x99\xAA\x4C\x12\x04\xAE\x1F\x77" + "\x35\x88\xF1\x65\x06\x0A\x0B\x4D" + "\x47\xF9\x50\x38\x5D\x71\xF9\x6E" + "\xDE\xEC\x61\x35\x2C\x4C\x96\x50" + "\xE8\x28\x93\x9C\x7E\x01\xC6\x04" + "\xB2\xD6\xBC\x6C\x17\xEB\xC1\x7D" + "\x11\xE9\x43\x83\x76\xAA\x53\x37" + "\x0C\x1D\x39\x89\x53\x72\x09\x7E" + "\xD9\x85\x16\x04\xA5\x2C\x05\x6F" + "\x17\x0C\x6E\x66\xAA\x84\xA7\xD9" + "\xE2\xD9\xC4\xEB\x43\x3E\xB1\x8D" + "\x7C\x36\xC7\x71\x70\x9C\x10\xD8" + "\xE8\x47\x2A\x4D\xFD\xA1\xBC\xE3" + "\xB9\x32\xE2\xC1\x82\xAC\xFE\xCC" + "\xC5\xC9\x7F\x9E\xCF\x33\x7A\xDF" + "\x6C\x82\x9D", + .len = 499, .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, + .np = 2, + .tap = { 499 - 16, 16 }, }, }; -static const struct cipher_testvec tf_cbc_dec_tv_template[] = { - { /* Reverse of the first four above */ - .key = zeroed_string, - .klen = 16, - .iv = zeroed_string, - .input = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" - "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, +static const struct cipher_testvec tf_lrw_tv_template[] = { + /* Generated from AES-LRW test vectors */ + { + .key = "\x45\x62\xac\x25\xf8\x28\x17\x6d" + "\x4c\x26\x84\x14\xb5\x68\x01\x85" + "\x25\x8e\x2a\x05\xe7\x3e\x9d\x03" + "\xee\x5a\x83\x0c\xcc\x09\x4c\x87", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\xa1\x6c\x50\x69\x26\xa4\xef\x7b" + "\x7c\xc6\x91\xeb\x72\xdd\x9b\xee", + .len = 16, }, { - .key = zeroed_string, - .klen = 16, - .iv = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" - "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a", - .input = "\xd4\x91\xdb\x16\xe7\xb1\xc3\x9e" - "\x86\xcb\x08\x6b\x78\x9f\x54\x19", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, + .key = "\x59\x70\x47\x14\xf5\x57\x47\x8c" + "\xd7\x79\xe8\x0f\x54\x88\x79\x44" + "\x0d\x48\xf0\xb7\xb1\x5a\x53\xea" + "\x1c\xaa\x6b\x29\xc2\xca\xfb\xaf", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x02", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\xab\x72\x0a\xad\x3b\x0c\xf0\xc9" + "\x42\x2f\xf1\xae\xf1\x3c\xb1\xbd", + .len = 16, }, { - .key = zeroed_string, - .klen = 16, - .iv = "\xd4\x91\xdb\x16\xe7\xb1\xc3\x9e" - "\x86\xcb\x08\x6b\x78\x9f\x54\x19", - .input = "\x05\xef\x8c\x61\xa8\x11\x58\x26" - "\x34\xba\x5c\xb7\x10\x6a\xa6\x41", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, + .key = "\xd8\x2a\x91\x34\xb2\x6a\x56\x50" + "\x30\xfe\x69\xe2\x37\x7f\x98\x47" + "\xcd\xf9\x0b\x16\x0c\x64\x8f\xb6" + "\xb0\x0d\x0d\x1b\xae\x85\x87\x1f", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x02\x00\x00\x00\x00", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x85\xa7\x56\x67\x08\xfa\x42\xe1" + "\x22\xe6\x82\xfc\xd9\xb4\xd7\xd4", + .len = 16, }, { - .key = zeroed_string, - .klen = 16, - .iv = zeroed_string, - .input = "\x9f\x58\x9f\x5c\xf6\x12\x2c\x32" - "\xb6\xbf\xec\x2f\x2a\xe8\xc3\x5a" - "\xd4\x91\xdb\x16\xe7\xb1\xc3\x9e" - "\x86\xcb\x08\x6b\x78\x9f\x54\x19" - "\x05\xef\x8c\x61\xa8\x11\x58\x26" - "\x34\xba\x5c\xb7\x10\x6a\xa6\x41", - .ilen = 48, - .result = zeroed_string, - .rlen = 48, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .key = "\x0f\x6a\xef\xf8\xd3\xd2\xbb\x15" + "\x25\x83\xf7\x3c\x1f\x01\x28\x74" + "\xca\xc6\xbc\x35\x4d\x4a\x65\x54" + "\x90\xae\x61\xcf\x7b\xae\xbd\xcc" + "\xad\xe4\x94\xc5\x4a\x29\xae\x70", + .klen = 40, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\xd2\xaf\x69\x35\x24\x1d\x0e\x1c" + "\x84\x8b\x05\xe4\xa2\x2f\x16\xf5", + .len = 16, + }, { + .key = "\x8a\xd4\xee\x10\x2f\xbd\x81\xff" + "\xf8\x86\xce\xac\x93\xc5\xad\xc6" + "\xa0\x19\x07\xc0\x9d\xf7\xbb\xdd" + "\x52\x13\xb2\xb7\xf0\xff\x11\xd8" + "\xd6\x08\xd0\xcd\x2e\xb1\x17\x6f", + .klen = 40, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x02\x00\x00\x00\x00", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x4a\x23\x56\xd7\xff\x90\xd0\x9a" + "\x0d\x7c\x26\xfc\xf0\xf0\xf6\xe4", + .len = 16, + }, { + .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" + "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" + "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" + "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" + "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" + "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x30\xaf\x26\x05\x9d\x5d\x0a\x58" + "\xe2\xe7\xce\x8a\xb2\x56\x6d\x76", + .len = 16, + }, { + .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" + "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" + "\xb2\xfb\x64\xce\x60\x97\x87\x8d" + "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" + "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" + "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x02\x00\x00\x00\x00", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\xdf\xcf\xdc\xd2\xe1\xcf\x86\x75" + "\x17\x66\x5e\x0c\x14\xa1\x3d\x40", + .len = 16, + }, { + .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" + "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" + "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" + "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" + "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" + "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" + "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" + "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" + "\x50\x38\x1f\x71\x49\xb6\x57\xd6" + "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" + "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" + "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" + "\xda\x10\x8e\xed\xa2\xa4\x87\xab" + "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" + "\xc9\xac\x42\x31\x95\x7c\xc9\x04" + "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" + "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" + "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" + "\x4c\x96\x12\xed\x7c\x92\x03\x01" + "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" + "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" + "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" + "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" + "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" + "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" + "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" + "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" + "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" + "\x76\x12\x73\x44\x1a\x56\xd7\x72" + "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" + "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" + "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" + "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" + "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" + "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" + "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" + "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" + "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" + "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" + "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" + "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" + "\x8d\x23\x31\x74\x84\xeb\x88\x6e" + "\xcc\xb9\xbc\x22\x83\x19\x07\x22" + "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" + "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" + "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" + "\x3c\xce\x8f\x42\x60\x71\xa7\x75" + "\x08\x40\x65\x8a\x82\xbf\xf5\x43" + "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" + "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" + "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" + "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" + "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" + "\x62\x73\x65\xfd\x46\x63\x25\x3d" + "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" + "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" + "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" + "\xc5\x68\x77\x84\x32\x2b\xcc\x85" + "\x74\x96\xf0\x12\x77\x61\xb9\xeb" + "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" + "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" + "\xda\x39\x87\x45\xc0\x2b\xbb\x01" + "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" + "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" + "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" + "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" + "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" + "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" + "\x21\xc4\xc2\x75\x67\x89\x37\x0a", + .ctext = "\x30\x38\xeb\xaf\x12\x43\x1a\x89" + "\x62\xa2\x36\xe5\xcf\x77\x1e\xd9" + "\x08\xc3\x0d\xdd\x95\xab\x19\x96" + "\x27\x52\x41\xc3\xca\xfb\xf6\xee" + "\x40\x2d\xdf\xdd\x00\x0c\xb9\x0a" + "\x3a\xf0\xc0\xd1\xda\x63\x9e\x45" + "\x42\xe9\x29\xc0\xb4\x07\xb4\x31" + "\x66\x77\x72\xb5\xb6\xb3\x57\x46" + "\x34\x9a\xfe\x03\xaf\x6b\x36\x07" + "\x63\x8e\xc2\x5d\xa6\x0f\xb6\x7d" + "\xfb\x6d\x82\x51\xb6\x98\xd0\x71" + "\xe7\x10\x7a\xdf\xb2\xbd\xf1\x1d" + "\x72\x2b\x54\x13\xe3\x6d\x79\x37" + "\xa9\x39\x2c\xdf\x21\xab\x87\xd5" + "\xee\xef\x9a\x12\x50\x39\x2e\x1b" + "\x7d\xe6\x6a\x27\x48\xb9\xe7\xac" + "\xaa\xcd\x79\x5f\xf2\xf3\xa0\x08" + "\x6f\x2c\xf4\x0e\xd1\xb8\x89\x25" + "\x31\x9d\xef\xb1\x1d\x27\x55\x04" + "\xc9\x8c\xb7\x68\xdc\xb6\x67\x8a" + "\xdb\xcf\x22\xf2\x3b\x6f\xce\xbb" + "\x26\xbe\x4f\x27\x04\x42\xd1\x44" + "\x4c\x08\xa3\x95\x4c\x7f\x1a\xaf" + "\x1d\x28\x14\xfd\xb1\x1a\x34\x18" + "\xf5\x1e\x28\x69\x95\x6a\x5a\xba" + "\x8e\xb2\x58\x1d\x28\x17\x13\x3d" + "\x38\x7d\x14\x8d\xab\x5d\xf9\xe8" + "\x3c\x0f\x2b\x0d\x2b\x08\xb4\x4b" + "\x6b\x0d\xc8\xa7\x84\xc2\x3a\x1a" + "\xb7\xbd\xda\x92\x29\xb8\x5b\x5a" + "\x63\xa5\x99\x82\x09\x72\x8f\xc6" + "\xa4\x62\x24\x69\x8c\x2d\x26\x00" + "\x99\x83\x91\xd6\xc6\xcf\x57\x67" + "\x38\xea\xf2\xfc\x29\xe0\x73\x39" + "\xf9\x13\x94\x6d\xe2\x58\x28\x75" + "\x3e\xae\x71\x90\x07\x70\x1c\x38" + "\x5b\x4c\x1e\xb5\xa5\x3b\x20\xef" + "\xb1\x4c\x3e\x1a\x72\x62\xbb\x22" + "\x82\x09\xe3\x18\x3f\x4f\x48\xfc" + "\xdd\xac\xfc\xb6\x09\xdb\xd2\x7b" + "\xd6\xb7\x7e\x41\x2f\x14\xf5\x0e" + "\xc3\xac\x4a\xed\xe7\x82\xef\x31" + "\x1f\x1a\x51\x1e\x29\x60\xc8\x98" + "\x93\x51\x1d\x3d\x62\x59\x83\x82" + "\x0c\xf1\xd7\x8d\xac\x33\x44\x81" + "\x3c\x59\xb7\xd4\x5b\x65\x82\xc4" + "\xec\xdc\x24\xfd\x0e\x1a\x79\x94" + "\x34\xb0\x62\xfa\x98\x49\x26\x1f" + "\xf4\x9e\x40\x44\x5b\x1f\xf8\xbe" + "\x36\xff\xc6\xc6\x9d\xf2\xd6\xcc" + "\x63\x93\x29\xb9\x0b\x6d\xd7\x6c" + "\xdb\xf6\x21\x80\xf7\x5a\x37\x15" + "\x0c\xe3\x36\xc8\x74\x75\x20\x91" + "\xdf\x52\x2d\x0c\xe7\x45\xff\x46" + "\xb3\xf4\xec\xc2\xbd\xd3\x37\xb6" + "\x26\xa2\x5d\x7d\x61\xbf\x10\x46" + "\x57\x8d\x05\x96\x70\x0b\xd6\x41" + "\x5c\xe9\xd3\x54\x81\x39\x3a\xdd" + "\x5f\x92\x81\x6e\x35\x03\xd4\x72" + "\x3d\x5a\xe7\xb9\x3b\x0c\x84\x23" + "\x45\x5d\xec\x72\xc1\x52\xef\x2e" + "\x81\x00\xd3\xfe\x4c\x3c\x05\x61" + "\x80\x18\xc4\x6c\x03\xd3\xb7\xba" + "\x11\xd7\xb8\x6e\xea\xe1\x80\x30", + .len = 512, + .also_non_np = 1, + .np = 3, + .tap = { 512 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec tf_xts_tv_template[] = { + /* Generated from AES-XTS test vectors */ +{ + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\xC8\xFF\xF2\x53\xA6\x27\x09\xD1" - "\x33\x38\xC2\xC0\x0C\x14\x7E\xB5" - "\x26\x1B\x05\x0C\x05\x12\x3F\xC0" - "\xF9\x1C\x02\x28\x40\x96\x6F\xD0" - "\x3D\x32\xDF\xDA\x56\x00\x6E\xEE" - "\x5B\x2A\x72\x9D\xC2\x4D\x19\xBC" - "\x8C\x53\xFA\x87\x6F\xDD\x81\xA3" - "\xB1\xD3\x44\x65\xDF\xE7\x63\x38" - "\x4A\xFC\xDC\xEC\x3F\x26\x8E\xB8" - "\x43\xFC\xFE\x18\xB5\x11\x6D\x31" - "\x81\x8B\x0D\x75\xF6\x80\xEC\x84" - "\x04\xB9\xE6\x09\x63\xED\x39\xDB" - "\xC3\xF6\x14\xD6\x6E\x5E\x8B\xBD" - "\x3E\xFA\xD7\x98\x50\x6F\xD9\x63" - "\x02\xCD\x0D\x39\x4B\x0D\xEC\x80" - "\xE3\x6A\x17\xF4\xCC\xAD\xFF\x68" - "\x45\xDD\xC8\x83\x1D\x41\x96\x0D" - "\x91\x2E\x05\xD3\x59\x82\xE0\x43" - "\x90\x4F\xB9\xF7\xAD\x6B\x2E\xAF" - "\xA7\x84\x00\x53\xCD\x6F\xD1\x0C" - "\x4E\xF9\x5A\x23\xFB\xCA\xC7\xD3" - "\xA9\xAA\x9D\xB2\x3F\x66\xF1\xAC" - "\x25\x21\x8F\xF7\xEF\xF2\x6A\xDF" - "\xE8\xDA\x75\x1A\x8A\xF1\xDD\x38" - "\x1F\xF9\x3D\x68\x4A\xBB\x9E\x34" - "\x1F\x66\x1F\x9C\x2B\x54\xFF\x60" - "\x7F\x29\x4B\x55\x80\x8F\x4E\xA7" - "\xA6\x9A\x0A\xD9\x0D\x19\x00\xF8" - "\x1F\xBC\x0C\x40\x6B\xEC\x99\x25" - "\x94\x70\x74\x0E\x1D\xC5\xBC\x12" - "\xF3\x42\xBE\x95\xBF\xFB\x4E\x55" - "\x9A\xB9\xCE\x14\x16\x5B\xDC\xD3" - "\x75\x42\x62\x04\x31\x1F\x95\x7C" - "\x66\x1A\x97\xDC\x2F\x40\x5C\x39" - "\x78\xE6\x02\xDB\x49\xE1\xC6\x47" - "\xC2\x78\x9A\xBB\xF3\xBE\xCB\x93" - "\xD8\xB8\xE8\xBB\x8C\xB3\x9B\xA7" - "\xC2\x89\xF3\x91\x88\x83\x3D\xF0" - "\x29\xA2\xCD\xB5\x79\x16\xC2\x40" - "\x11\x03\x8E\x9C\xFD\xC9\x43\xC4" - "\xC2\x19\xF0\x4A\x32\xEF\x0C\x2B" - "\xD3\x2B\xE9\xD4\x4C\xDE\x95\xCF" - "\x04\x03\xD3\x2C\x7F\x82\xC8\xFA" - "\x0F\xD8\x7A\x39\x7B\x01\x41\x9C" - "\x78\xB6\xC9\xBF\xF9\x78\x57\x88" - "\xB1\xA5\xE1\xE0\xD9\x16\xD4\xC8" - "\xEE\xC4\xBE\x7B\x55\x59\x00\x48" - "\x1B\xBC\x14\xFA\x2A\x9D\xC9\x1C" - "\xFB\x28\x3F\x95\xDD\xB7\xD6\xCE" - "\x3A\x7F\x09\x0C\x0E\x69\x30\x7D" - "\xBC\x68\x9C\x91\x2A\x59\x57\x04" - "\xED\x1A\x1E\x00\xB1\x85\x92\x04" - "\x28\x8C\x0C\x3C\xC1\xD5\x12\xF7" - "\x4C\x3E\xB0\xE7\x86\x62\x68\x91" - "\xFC\xC4\xE2\xCE\xA6\xDC\x5E\x93" - "\x5D\x8D\x8C\x68\xB3\xB2\xB9\x64" - "\x16\xB8\xC8\x6F\xD8\xEE\x21\xBD" - "\xAC\x18\x0C\x7D\x0D\x05\xAB\xF1" - "\xFA\xDD\xE2\x48\xDF\x4C\x02\x39" - "\x69\xA1\x62\xBD\x49\x3A\x9D\x91" - "\x30\x70\x56\xA4\x37\xDD\x7C\xC0" - "\x0A\xA3\x30\x10\x26\x25\x41\x2C", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec tf_ctr_enc_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\x4b\xc9\x44\x4a\x11\xa3\xef\xac" + "\x30\x74\xe4\x44\x52\x77\x97\x43" + "\xa7\x60\xb2\x45\x2e\xf9\x00\x90" + "\x9f\xaa\xfd\x89\x6e\x9d\x4a\xe0", + .len = 32, + }, { + .key = "\x11\x11\x11\x11\x11\x11\x11\x11" + "\x11\x11\x11\x11\x11\x11\x11\x11" + "\x22\x22\x22\x22\x22\x22\x22\x22" + "\x22\x22\x22\x22\x22\x22\x22\x22", .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\xDF\xDD\x69\xFA\xB0\x2E\xFD\xFE" - "\x70\x9E\xC5\x4B\xC9\xD4\xA1\x30" - "\x26\x9B\x89\xA1\xEE\x43\xE0\x52" - "\x55\x17\x4E\xC7\x0E\x33\x1F\xF1" - "\x9F\x8D\x40\x9F\x24\xFD\x92\xA0" - "\xBC\x8F\x35\xDD\x67\x38\xD8\xAA" - "\xCF\xF8\x48\xCA\xFB\xE4\x5C\x60" - "\x01\x41\x21\x12\x38\xAB\x52\x4F" - "\xA8\x57\x20\xE0\x21\x6A\x17\x0D" - "\x0E\xF9\x8E\x49\x42\x00\x3C\x94" - "\x14\xC0\xD0\x8D\x8A\x98\xEB\x29" - "\xEC\xAE\x96\x44\xC0\x3C\x48\xDC" - "\x29\x35\x25\x2F\xE7\x11\x6C\x68" - "\xC8\x67\x0A\x2F\xF4\x07\xBE\xF9" - "\x2C\x31\x87\x40\xAB\xB2\xB6\xFA" - "\xD2\xC9\x6D\x5C\x50\xE9\xE6\x7E" - "\xE3\x0A\xD2\xD5\x6D\x8D\x64\x9E" - "\x70\xCE\x03\x76\xDD\xE0\xF0\x8C" - "\x84\x86\x8B\x6A\xFE\xC7\xF9\x69" - "\x2E\xFE\xFC\xC2\xC4\x1A\x55\x58" - "\xB3\xBE\xE2\x7E\xED\x39\x42\x6C" - "\xB4\x42\x97\x9A\xEC\xE1\x0A\x06" - "\x02\xC5\x03\x9D\xC4\x48\x15\x66" - "\x35\x6A\xC2\xC9\xA2\x26\x30\xBB" - "\xDB\x2D\xC8\x08\x2B\xA0\x29\x1A" - "\x23\x61\x48\xEA\x80\x04\x27\xAA" - "\x69\x49\xE8\xE8\x4A\x83\x6B\x5A" - "\xCA\x7C\xD3\xB1\xB5\x0B\xCC\x23" - "\x74\x1F\xA9\x87\xCD\xED\xC0\x2D" - "\xBF\xEB\xCF\x16\x2D\x2A\x2E\x1D" - "\x96\xBA\x36\x11\x45\x41\xDA\xCE" - "\xA4\x48\x80\x8B\x06\xF4\x98\x89" - "\x8B\x23\x08\x53\xF4\xD4\x5A\x24" - "\x8B\xF8\x43\x73\xD1\xEE\xC4\xB0" - "\xF8\xFE\x09\x0C\x75\x05\x38\x0B" - "\x7C\x81\xDE\x9D\xE4\x61\x37\x63" - "\x63\xAD\x12\xD2\x04\xB9\xCE\x45" - "\x5A\x1A\x6E\xB3\x78\x2A\xA4\x74" - "\x86\xD0\xE3\xFF\xDA\x38\x9C\xB5" - "\xB8\xB1\xDB\x38\x2F\xC5\x6A\xB4" - "\xEB\x6E\x96\xE8\x43\x80\xB5\x51" - "\x61\x2D\x48\xAA\x07\x65\x11\x8C" - "\x48\xE3\x90\x7E\x78\x3A\xEC\x97" - "\x05\x3D\x84\xE7\x90\x2B\xAA\xBD" - "\x83\x29\x0E\x1A\x81\x73\x7B\xE0" - "\x7A\x01\x4A\x37\x3B\x77\x7F\x8D" - "\x49\xA4\x2F\x6E\xBE\x68\x99\x08" - "\x99\xAA\x4C\x12\x04\xAE\x1F\x77" - "\x35\x88\xF1\x65\x06\x0A\x0B\x4D" - "\x47\xF9\x50\x38\x5D\x71\xF9\x6E" - "\xDE\xEC\x61\x35\x2C\x4C\x96\x50" - "\xE8\x28\x93\x9C\x7E\x01\xC6\x04" - "\xB2\xD6\xBC\x6C\x17\xEB\xC1\x7D" - "\x11\xE9\x43\x83\x76\xAA\x53\x37" - "\x0C\x1D\x39\x89\x53\x72\x09\x7E" - "\xD9\x85\x16\x04\xA5\x2C\x05\x6F" - "\x17\x0C\x6E\x66\xAA\x84\xA7\xD9" - "\xE2\xD9\xC4\xEB\x43\x3E\xB1\x8D" - "\x7C\x36\xC7\x71\x70\x9C\x10\xD8" - "\xE8\x47\x2A\x4D\xFD\xA1\xBC\xE3" - "\xB9\x32\xE2\xC1\x82\xAC\xFE\xCC" - "\xC5\xC9\x7F\x9E\xCF\x33\x7A\xDF", - .rlen = 496, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44", + .ctext = "\x57\x0e\x8f\xe5\x2a\x35\x61\x4f" + "\x32\xd3\xbd\x36\x05\x15\x44\x2c" + "\x58\x06\xf7\xf8\x00\xa8\xb6\xd5" + "\xc6\x28\x92\xdb\xd8\x34\xa2\xe9", + .len = 32, + }, { + .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" + "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" + "\x22\x22\x22\x22\x22\x22\x22\x22" + "\x22\x22\x22\x22\x22\x22\x22\x22", .klen = 32, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\xEB\x44\xAF\x49\x27\xB8\xFB\x44" - "\x4C\xA6\xC3\x0C\x8B\xD0\x01\x0C" - "\x53\xC8\x16\x38\xDE\x40\x4F\x91" - "\x25\x6D\x4C\xA0\x9A\x87\x1E\xDA" - "\x88\x7E\x89\xE9\x67\x2B\x83\xA2" - "\x5F\x2E\x23\x3E\x45\xB9\x77\x7B" - "\xA6\x7E\x47\x36\x81\x9F\x9B\xF3" - "\xE0\xF0\xD7\x47\xA9\xC8\xEF\x33" - "\x0C\x43\xFE\x67\x50\x0A\x2C\x3E" - "\xA0\xE1\x25\x8E\x80\x07\x4A\xC0" - "\x64\x89\x9F\x6A\x27\x96\x07\xA6" - "\x9B\xC8\x1B\x21\x60\xAE\x5D\x01" - "\xE2\xCD\xC8\xAA\x6C\x9D\x1C\x34" - "\x39\x18\x09\xA4\x82\x59\x78\xE7" - "\xFC\x59\x65\xF2\x94\xFF\xFB\xE2" - "\x3C\xDA\xB1\x90\x95\xBF\x91\xE3" - "\xE6\x87\x31\x9E\x16\x85\xAD\xB1" - "\x4C\xAE\x43\x4D\x19\x58\xB5\x5E" - "\x2E\xF5\x09\xAA\x39\xF4\xC0\xB3" - "\xD4\x4D\xDB\x73\x7A\xD4\xF1\xBF" - "\x89\x16\x4D\x2D\xA2\x26\x33\x72" - "\x18\x33\x7E\xD6\xD2\x16\xA4\x54" - "\xF4\x8C\xB3\x52\xDF\x21\x9C\xEB" - "\xBF\x49\xD3\xF9\x05\x06\xCB\xD2" - "\xA9\xD2\x3B\x6E\x19\x8C\xBC\x19" - "\xAB\x89\xD6\xD8\xCD\x56\x89\x5E" - "\xAC\x00\xE3\x50\x63\x4A\x80\x9A" - "\x05\xBC\x50\x39\xD3\x32\xD9\x0D" - "\xE3\x20\x0D\x75\x54\xEC\xE6\x31" - "\x14\xB9\x3A\x59\x00\x43\x37\x8E" - "\x8C\x5A\x79\x62\x14\x76\x8A\xAE" - "\x8F\xCC\xA1\x6C\x38\x78\xDD\x2D" - "\x8B\x6D\xEA\xBD\x7B\x25\xFF\x60" - "\xC9\x87\xB1\x79\x1E\xA5\x86\x68" - "\x81\xB4\xE2\xC1\x05\x7D\x3A\x73" - "\xD0\xDA\x75\x77\x9E\x05\x27\xF1" - "\x08\xA9\x66\x64\x6C\xBC\x82\x17" - "\x2C\x23\x5F\x62\x4D\x02\x1A\x58" - "\xE7\xB7\x23\x6D\xE2\x20\xDA\xEF" - "\xB4\xB3\x3F\xB2\x2B\x69\x98\x83" - "\x95\x87\x13\x57\x60\xD7\xB5\xB1" - "\xEE\x0A\x2F\x95\x36\x4C\x76\x5D" - "\x5F\xD9\x19\xED\xB9\xA5\x48\xBF" - "\xC8\xAB\x0F\x71\xCC\x61\x8E\x0A" - "\xD0\x29\x44\xA8\xB9\xC1\xE8\xC8" - "\xC9\xA8\x28\x81\xFB\x50\xF2\xF0" - "\x26\xAE\x39\xB8\x91\xCD\xA8\xAC" - "\xDE\x55\x1B\x50\x14\x53\x44\x17" - "\x54\x46\xFC\xB1\xE4\x07\x6B\x9A" - "\x01\x14\xF0\x2E\x2E\xDB\x46\x1B" - "\x1A\x09\x97\xA9\xB6\x97\x79\x06" - "\xFB\xCB\x85\xCF\xDD\xA1\x41\xB1" - "\x00\xAA\xF7\xE0\x89\x73\xFB\xE5" - "\xBF\x84\xDB\xC9\xCD\xC4\xA2\x0D" - "\x3B\xAC\xF9\xDF\x96\xBF\x88\x23" - "\x41\x67\xA1\x24\x99\x7E\xCC\x9B" - "\x02\x8F\x6A\x49\xF6\x25\xBA\x7A" - "\xF4\x78\xFD\x79\x62\x63\x4F\x14" - "\xD6\x11\x11\x04\x05\x5F\x7E\xEA" - "\x4C\xB6\xF8\xF4\x5F\x48\x52\x54" - "\x94\x63\xA8\x4E\xCF\xD2\x1B\x1B" - "\x22\x18\x6A\xAF\x6E\x3E\xE1\x0D", - .rlen = 496, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59", - .ilen = 499, - .result = "\xDF\xDD\x69\xFA\xB0\x2E\xFD\xFE" - "\x70\x9E\xC5\x4B\xC9\xD4\xA1\x30" - "\x26\x9B\x89\xA1\xEE\x43\xE0\x52" - "\x55\x17\x4E\xC7\x0E\x33\x1F\xF1" - "\x9F\x8D\x40\x9F\x24\xFD\x92\xA0" - "\xBC\x8F\x35\xDD\x67\x38\xD8\xAA" - "\xCF\xF8\x48\xCA\xFB\xE4\x5C\x60" - "\x01\x41\x21\x12\x38\xAB\x52\x4F" - "\xA8\x57\x20\xE0\x21\x6A\x17\x0D" - "\x0E\xF9\x8E\x49\x42\x00\x3C\x94" - "\x14\xC0\xD0\x8D\x8A\x98\xEB\x29" - "\xEC\xAE\x96\x44\xC0\x3C\x48\xDC" - "\x29\x35\x25\x2F\xE7\x11\x6C\x68" - "\xC8\x67\x0A\x2F\xF4\x07\xBE\xF9" - "\x2C\x31\x87\x40\xAB\xB2\xB6\xFA" - "\xD2\xC9\x6D\x5C\x50\xE9\xE6\x7E" - "\xE3\x0A\xD2\xD5\x6D\x8D\x64\x9E" - "\x70\xCE\x03\x76\xDD\xE0\xF0\x8C" - "\x84\x86\x8B\x6A\xFE\xC7\xF9\x69" - "\x2E\xFE\xFC\xC2\xC4\x1A\x55\x58" - "\xB3\xBE\xE2\x7E\xED\x39\x42\x6C" - "\xB4\x42\x97\x9A\xEC\xE1\x0A\x06" - "\x02\xC5\x03\x9D\xC4\x48\x15\x66" - "\x35\x6A\xC2\xC9\xA2\x26\x30\xBB" - "\xDB\x2D\xC8\x08\x2B\xA0\x29\x1A" - "\x23\x61\x48\xEA\x80\x04\x27\xAA" - "\x69\x49\xE8\xE8\x4A\x83\x6B\x5A" - "\xCA\x7C\xD3\xB1\xB5\x0B\xCC\x23" - "\x74\x1F\xA9\x87\xCD\xED\xC0\x2D" - "\xBF\xEB\xCF\x16\x2D\x2A\x2E\x1D" - "\x96\xBA\x36\x11\x45\x41\xDA\xCE" - "\xA4\x48\x80\x8B\x06\xF4\x98\x89" - "\x8B\x23\x08\x53\xF4\xD4\x5A\x24" - "\x8B\xF8\x43\x73\xD1\xEE\xC4\xB0" - "\xF8\xFE\x09\x0C\x75\x05\x38\x0B" - "\x7C\x81\xDE\x9D\xE4\x61\x37\x63" - "\x63\xAD\x12\xD2\x04\xB9\xCE\x45" - "\x5A\x1A\x6E\xB3\x78\x2A\xA4\x74" - "\x86\xD0\xE3\xFF\xDA\x38\x9C\xB5" - "\xB8\xB1\xDB\x38\x2F\xC5\x6A\xB4" - "\xEB\x6E\x96\xE8\x43\x80\xB5\x51" - "\x61\x2D\x48\xAA\x07\x65\x11\x8C" - "\x48\xE3\x90\x7E\x78\x3A\xEC\x97" - "\x05\x3D\x84\xE7\x90\x2B\xAA\xBD" - "\x83\x29\x0E\x1A\x81\x73\x7B\xE0" - "\x7A\x01\x4A\x37\x3B\x77\x7F\x8D" - "\x49\xA4\x2F\x6E\xBE\x68\x99\x08" - "\x99\xAA\x4C\x12\x04\xAE\x1F\x77" - "\x35\x88\xF1\x65\x06\x0A\x0B\x4D" - "\x47\xF9\x50\x38\x5D\x71\xF9\x6E" - "\xDE\xEC\x61\x35\x2C\x4C\x96\x50" - "\xE8\x28\x93\x9C\x7E\x01\xC6\x04" - "\xB2\xD6\xBC\x6C\x17\xEB\xC1\x7D" - "\x11\xE9\x43\x83\x76\xAA\x53\x37" - "\x0C\x1D\x39\x89\x53\x72\x09\x7E" - "\xD9\x85\x16\x04\xA5\x2C\x05\x6F" - "\x17\x0C\x6E\x66\xAA\x84\xA7\xD9" - "\xE2\xD9\xC4\xEB\x43\x3E\xB1\x8D" - "\x7C\x36\xC7\x71\x70\x9C\x10\xD8" - "\xE8\x47\x2A\x4D\xFD\xA1\xBC\xE3" - "\xB9\x32\xE2\xC1\x82\xAC\xFE\xCC" - "\xC5\xC9\x7F\x9E\xCF\x33\x7A\xDF" - "\x6C\x82\x9D", - .rlen = 499, - .also_non_np = 1, - .np = 2, - .tap = { 499 - 16, 16 }, - }, -}; - -static const struct cipher_testvec tf_ctr_dec_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44", + .ctext = "\x96\x45\x8f\x8d\x7a\x75\xb1\xde" + "\x40\x0c\x89\x56\xf6\x4d\xa7\x07" + "\x38\xbb\x5b\xe9\xcd\x84\xae\xb2" + "\x7b\x6a\x62\xf4\x8c\xb5\x37\xea", + .len = 32, + }, { + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x31\x41\x59\x26\x53\x58\x97\x93" + "\x23\x84\x62\x64\x33\x83\x27\x95", .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\xDF\xDD\x69\xFA\xB0\x2E\xFD\xFE" - "\x70\x9E\xC5\x4B\xC9\xD4\xA1\x30" - "\x26\x9B\x89\xA1\xEE\x43\xE0\x52" - "\x55\x17\x4E\xC7\x0E\x33\x1F\xF1" - "\x9F\x8D\x40\x9F\x24\xFD\x92\xA0" - "\xBC\x8F\x35\xDD\x67\x38\xD8\xAA" - "\xCF\xF8\x48\xCA\xFB\xE4\x5C\x60" - "\x01\x41\x21\x12\x38\xAB\x52\x4F" - "\xA8\x57\x20\xE0\x21\x6A\x17\x0D" - "\x0E\xF9\x8E\x49\x42\x00\x3C\x94" - "\x14\xC0\xD0\x8D\x8A\x98\xEB\x29" - "\xEC\xAE\x96\x44\xC0\x3C\x48\xDC" - "\x29\x35\x25\x2F\xE7\x11\x6C\x68" - "\xC8\x67\x0A\x2F\xF4\x07\xBE\xF9" - "\x2C\x31\x87\x40\xAB\xB2\xB6\xFA" - "\xD2\xC9\x6D\x5C\x50\xE9\xE6\x7E" - "\xE3\x0A\xD2\xD5\x6D\x8D\x64\x9E" - "\x70\xCE\x03\x76\xDD\xE0\xF0\x8C" - "\x84\x86\x8B\x6A\xFE\xC7\xF9\x69" - "\x2E\xFE\xFC\xC2\xC4\x1A\x55\x58" - "\xB3\xBE\xE2\x7E\xED\x39\x42\x6C" - "\xB4\x42\x97\x9A\xEC\xE1\x0A\x06" - "\x02\xC5\x03\x9D\xC4\x48\x15\x66" - "\x35\x6A\xC2\xC9\xA2\x26\x30\xBB" - "\xDB\x2D\xC8\x08\x2B\xA0\x29\x1A" - "\x23\x61\x48\xEA\x80\x04\x27\xAA" - "\x69\x49\xE8\xE8\x4A\x83\x6B\x5A" - "\xCA\x7C\xD3\xB1\xB5\x0B\xCC\x23" - "\x74\x1F\xA9\x87\xCD\xED\xC0\x2D" - "\xBF\xEB\xCF\x16\x2D\x2A\x2E\x1D" - "\x96\xBA\x36\x11\x45\x41\xDA\xCE" - "\xA4\x48\x80\x8B\x06\xF4\x98\x89" - "\x8B\x23\x08\x53\xF4\xD4\x5A\x24" - "\x8B\xF8\x43\x73\xD1\xEE\xC4\xB0" - "\xF8\xFE\x09\x0C\x75\x05\x38\x0B" - "\x7C\x81\xDE\x9D\xE4\x61\x37\x63" - "\x63\xAD\x12\xD2\x04\xB9\xCE\x45" - "\x5A\x1A\x6E\xB3\x78\x2A\xA4\x74" - "\x86\xD0\xE3\xFF\xDA\x38\x9C\xB5" - "\xB8\xB1\xDB\x38\x2F\xC5\x6A\xB4" - "\xEB\x6E\x96\xE8\x43\x80\xB5\x51" - "\x61\x2D\x48\xAA\x07\x65\x11\x8C" - "\x48\xE3\x90\x7E\x78\x3A\xEC\x97" - "\x05\x3D\x84\xE7\x90\x2B\xAA\xBD" - "\x83\x29\x0E\x1A\x81\x73\x7B\xE0" - "\x7A\x01\x4A\x37\x3B\x77\x7F\x8D" - "\x49\xA4\x2F\x6E\xBE\x68\x99\x08" - "\x99\xAA\x4C\x12\x04\xAE\x1F\x77" - "\x35\x88\xF1\x65\x06\x0A\x0B\x4D" - "\x47\xF9\x50\x38\x5D\x71\xF9\x6E" - "\xDE\xEC\x61\x35\x2C\x4C\x96\x50" - "\xE8\x28\x93\x9C\x7E\x01\xC6\x04" - "\xB2\xD6\xBC\x6C\x17\xEB\xC1\x7D" - "\x11\xE9\x43\x83\x76\xAA\x53\x37" - "\x0C\x1D\x39\x89\x53\x72\x09\x7E" - "\xD9\x85\x16\x04\xA5\x2C\x05\x6F" - "\x17\x0C\x6E\x66\xAA\x84\xA7\xD9" - "\xE2\xD9\xC4\xEB\x43\x3E\xB1\x8D" - "\x7C\x36\xC7\x71\x70\x9C\x10\xD8" - "\xE8\x47\x2A\x4D\xFD\xA1\xBC\xE3" - "\xB9\x32\xE2\xC1\x82\xAC\xFE\xCC" - "\xC5\xC9\x7F\x9E\xCF\x33\x7A\xDF", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\xEB\x44\xAF\x49\x27\xB8\xFB\x44" - "\x4C\xA6\xC3\x0C\x8B\xD0\x01\x0C" - "\x53\xC8\x16\x38\xDE\x40\x4F\x91" - "\x25\x6D\x4C\xA0\x9A\x87\x1E\xDA" - "\x88\x7E\x89\xE9\x67\x2B\x83\xA2" - "\x5F\x2E\x23\x3E\x45\xB9\x77\x7B" - "\xA6\x7E\x47\x36\x81\x9F\x9B\xF3" - "\xE0\xF0\xD7\x47\xA9\xC8\xEF\x33" - "\x0C\x43\xFE\x67\x50\x0A\x2C\x3E" - "\xA0\xE1\x25\x8E\x80\x07\x4A\xC0" - "\x64\x89\x9F\x6A\x27\x96\x07\xA6" - "\x9B\xC8\x1B\x21\x60\xAE\x5D\x01" - "\xE2\xCD\xC8\xAA\x6C\x9D\x1C\x34" - "\x39\x18\x09\xA4\x82\x59\x78\xE7" - "\xFC\x59\x65\xF2\x94\xFF\xFB\xE2" - "\x3C\xDA\xB1\x90\x95\xBF\x91\xE3" - "\xE6\x87\x31\x9E\x16\x85\xAD\xB1" - "\x4C\xAE\x43\x4D\x19\x58\xB5\x5E" - "\x2E\xF5\x09\xAA\x39\xF4\xC0\xB3" - "\xD4\x4D\xDB\x73\x7A\xD4\xF1\xBF" - "\x89\x16\x4D\x2D\xA2\x26\x33\x72" - "\x18\x33\x7E\xD6\xD2\x16\xA4\x54" - "\xF4\x8C\xB3\x52\xDF\x21\x9C\xEB" - "\xBF\x49\xD3\xF9\x05\x06\xCB\xD2" - "\xA9\xD2\x3B\x6E\x19\x8C\xBC\x19" - "\xAB\x89\xD6\xD8\xCD\x56\x89\x5E" - "\xAC\x00\xE3\x50\x63\x4A\x80\x9A" - "\x05\xBC\x50\x39\xD3\x32\xD9\x0D" - "\xE3\x20\x0D\x75\x54\xEC\xE6\x31" - "\x14\xB9\x3A\x59\x00\x43\x37\x8E" - "\x8C\x5A\x79\x62\x14\x76\x8A\xAE" - "\x8F\xCC\xA1\x6C\x38\x78\xDD\x2D" - "\x8B\x6D\xEA\xBD\x7B\x25\xFF\x60" - "\xC9\x87\xB1\x79\x1E\xA5\x86\x68" - "\x81\xB4\xE2\xC1\x05\x7D\x3A\x73" - "\xD0\xDA\x75\x77\x9E\x05\x27\xF1" - "\x08\xA9\x66\x64\x6C\xBC\x82\x17" - "\x2C\x23\x5F\x62\x4D\x02\x1A\x58" - "\xE7\xB7\x23\x6D\xE2\x20\xDA\xEF" - "\xB4\xB3\x3F\xB2\x2B\x69\x98\x83" - "\x95\x87\x13\x57\x60\xD7\xB5\xB1" - "\xEE\x0A\x2F\x95\x36\x4C\x76\x5D" - "\x5F\xD9\x19\xED\xB9\xA5\x48\xBF" - "\xC8\xAB\x0F\x71\xCC\x61\x8E\x0A" - "\xD0\x29\x44\xA8\xB9\xC1\xE8\xC8" - "\xC9\xA8\x28\x81\xFB\x50\xF2\xF0" - "\x26\xAE\x39\xB8\x91\xCD\xA8\xAC" - "\xDE\x55\x1B\x50\x14\x53\x44\x17" - "\x54\x46\xFC\xB1\xE4\x07\x6B\x9A" - "\x01\x14\xF0\x2E\x2E\xDB\x46\x1B" - "\x1A\x09\x97\xA9\xB6\x97\x79\x06" - "\xFB\xCB\x85\xCF\xDD\xA1\x41\xB1" - "\x00\xAA\xF7\xE0\x89\x73\xFB\xE5" - "\xBF\x84\xDB\xC9\xCD\xC4\xA2\x0D" - "\x3B\xAC\xF9\xDF\x96\xBF\x88\x23" - "\x41\x67\xA1\x24\x99\x7E\xCC\x9B" - "\x02\x8F\x6A\x49\xF6\x25\xBA\x7A" - "\xF4\x78\xFD\x79\x62\x63\x4F\x14" - "\xD6\x11\x11\x04\x05\x5F\x7E\xEA" - "\x4C\xB6\xF8\xF4\x5F\x48\x52\x54" - "\x94\x63\xA8\x4E\xCF\xD2\x1B\x1B" - "\x22\x18\x6A\xAF\x6E\x3E\xE1\x0D", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\xDF\xDD\x69\xFA\xB0\x2E\xFD\xFE" - "\x70\x9E\xC5\x4B\xC9\xD4\xA1\x30" - "\x26\x9B\x89\xA1\xEE\x43\xE0\x52" - "\x55\x17\x4E\xC7\x0E\x33\x1F\xF1" - "\x9F\x8D\x40\x9F\x24\xFD\x92\xA0" - "\xBC\x8F\x35\xDD\x67\x38\xD8\xAA" - "\xCF\xF8\x48\xCA\xFB\xE4\x5C\x60" - "\x01\x41\x21\x12\x38\xAB\x52\x4F" - "\xA8\x57\x20\xE0\x21\x6A\x17\x0D" - "\x0E\xF9\x8E\x49\x42\x00\x3C\x94" - "\x14\xC0\xD0\x8D\x8A\x98\xEB\x29" - "\xEC\xAE\x96\x44\xC0\x3C\x48\xDC" - "\x29\x35\x25\x2F\xE7\x11\x6C\x68" - "\xC8\x67\x0A\x2F\xF4\x07\xBE\xF9" - "\x2C\x31\x87\x40\xAB\xB2\xB6\xFA" - "\xD2\xC9\x6D\x5C\x50\xE9\xE6\x7E" - "\xE3\x0A\xD2\xD5\x6D\x8D\x64\x9E" - "\x70\xCE\x03\x76\xDD\xE0\xF0\x8C" - "\x84\x86\x8B\x6A\xFE\xC7\xF9\x69" - "\x2E\xFE\xFC\xC2\xC4\x1A\x55\x58" - "\xB3\xBE\xE2\x7E\xED\x39\x42\x6C" - "\xB4\x42\x97\x9A\xEC\xE1\x0A\x06" - "\x02\xC5\x03\x9D\xC4\x48\x15\x66" - "\x35\x6A\xC2\xC9\xA2\x26\x30\xBB" - "\xDB\x2D\xC8\x08\x2B\xA0\x29\x1A" - "\x23\x61\x48\xEA\x80\x04\x27\xAA" - "\x69\x49\xE8\xE8\x4A\x83\x6B\x5A" - "\xCA\x7C\xD3\xB1\xB5\x0B\xCC\x23" - "\x74\x1F\xA9\x87\xCD\xED\xC0\x2D" - "\xBF\xEB\xCF\x16\x2D\x2A\x2E\x1D" - "\x96\xBA\x36\x11\x45\x41\xDA\xCE" - "\xA4\x48\x80\x8B\x06\xF4\x98\x89" - "\x8B\x23\x08\x53\xF4\xD4\x5A\x24" - "\x8B\xF8\x43\x73\xD1\xEE\xC4\xB0" - "\xF8\xFE\x09\x0C\x75\x05\x38\x0B" - "\x7C\x81\xDE\x9D\xE4\x61\x37\x63" - "\x63\xAD\x12\xD2\x04\xB9\xCE\x45" - "\x5A\x1A\x6E\xB3\x78\x2A\xA4\x74" - "\x86\xD0\xE3\xFF\xDA\x38\x9C\xB5" - "\xB8\xB1\xDB\x38\x2F\xC5\x6A\xB4" - "\xEB\x6E\x96\xE8\x43\x80\xB5\x51" - "\x61\x2D\x48\xAA\x07\x65\x11\x8C" - "\x48\xE3\x90\x7E\x78\x3A\xEC\x97" - "\x05\x3D\x84\xE7\x90\x2B\xAA\xBD" - "\x83\x29\x0E\x1A\x81\x73\x7B\xE0" - "\x7A\x01\x4A\x37\x3B\x77\x7F\x8D" - "\x49\xA4\x2F\x6E\xBE\x68\x99\x08" - "\x99\xAA\x4C\x12\x04\xAE\x1F\x77" - "\x35\x88\xF1\x65\x06\x0A\x0B\x4D" - "\x47\xF9\x50\x38\x5D\x71\xF9\x6E" - "\xDE\xEC\x61\x35\x2C\x4C\x96\x50" - "\xE8\x28\x93\x9C\x7E\x01\xC6\x04" - "\xB2\xD6\xBC\x6C\x17\xEB\xC1\x7D" - "\x11\xE9\x43\x83\x76\xAA\x53\x37" - "\x0C\x1D\x39\x89\x53\x72\x09\x7E" - "\xD9\x85\x16\x04\xA5\x2C\x05\x6F" - "\x17\x0C\x6E\x66\xAA\x84\xA7\xD9" - "\xE2\xD9\xC4\xEB\x43\x3E\xB1\x8D" - "\x7C\x36\xC7\x71\x70\x9C\x10\xD8" - "\xE8\x47\x2A\x4D\xFD\xA1\xBC\xE3" - "\xB9\x32\xE2\xC1\x82\xAC\xFE\xCC" - "\xC5\xC9\x7F\x9E\xCF\x33\x7A\xDF" - "\x6C\x82\x9D", - .ilen = 499, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59", - .rlen = 499, - .also_non_np = 1, - .np = 2, - .tap = { 499 - 16, 16 }, - }, -}; - -static const struct cipher_testvec tf_lrw_enc_tv_template[] = { - /* Generated from AES-LRW test vectors */ - { - .key = "\x45\x62\xac\x25\xf8\x28\x17\x6d" - "\x4c\x26\x84\x14\xb5\x68\x01\x85" - "\x25\x8e\x2a\x05\xe7\x3e\x9d\x03" - "\xee\x5a\x83\x0c\xcc\x09\x4c\x87", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\xa1\x6c\x50\x69\x26\xa4\xef\x7b" - "\x7c\xc6\x91\xeb\x72\xdd\x9b\xee", - .rlen = 16, - }, { - .key = "\x59\x70\x47\x14\xf5\x57\x47\x8c" - "\xd7\x79\xe8\x0f\x54\x88\x79\x44" - "\x0d\x48\xf0\xb7\xb1\x5a\x53\xea" - "\x1c\xaa\x6b\x29\xc2\xca\xfb\xaf", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x02", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\xab\x72\x0a\xad\x3b\x0c\xf0\xc9" - "\x42\x2f\xf1\xae\xf1\x3c\xb1\xbd", - .rlen = 16, - }, { - .key = "\xd8\x2a\x91\x34\xb2\x6a\x56\x50" - "\x30\xfe\x69\xe2\x37\x7f\x98\x47" - "\xcd\xf9\x0b\x16\x0c\x64\x8f\xb6" - "\xb0\x0d\x0d\x1b\xae\x85\x87\x1f", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x85\xa7\x56\x67\x08\xfa\x42\xe1" - "\x22\xe6\x82\xfc\xd9\xb4\xd7\xd4", - .rlen = 16, - }, { - .key = "\x0f\x6a\xef\xf8\xd3\xd2\xbb\x15" - "\x25\x83\xf7\x3c\x1f\x01\x28\x74" - "\xca\xc6\xbc\x35\x4d\x4a\x65\x54" - "\x90\xae\x61\xcf\x7b\xae\xbd\xcc" - "\xad\xe4\x94\xc5\x4a\x29\xae\x70", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\xd2\xaf\x69\x35\x24\x1d\x0e\x1c" - "\x84\x8b\x05\xe4\xa2\x2f\x16\xf5", - .rlen = 16, - }, { - .key = "\x8a\xd4\xee\x10\x2f\xbd\x81\xff" - "\xf8\x86\xce\xac\x93\xc5\xad\xc6" - "\xa0\x19\x07\xc0\x9d\xf7\xbb\xdd" - "\x52\x13\xb2\xb7\xf0\xff\x11\xd8" - "\xd6\x08\xd0\xcd\x2e\xb1\x17\x6f", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x4a\x23\x56\xd7\xff\x90\xd0\x9a" - "\x0d\x7c\x26\xfc\xf0\xf0\xf6\xe4", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x30\xaf\x26\x05\x9d\x5d\x0a\x58" - "\xe2\xe7\xce\x8a\xb2\x56\x6d\x76", - .rlen = 16, - }, { - .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" - "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" - "\xb2\xfb\x64\xce\x60\x97\x87\x8d" - "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" - "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" - "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\xdf\xcf\xdc\xd2\xe1\xcf\x86\x75" - "\x17\x66\x5e\x0c\x14\xa1\x3d\x40", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" - "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" - "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" - "\x50\x38\x1f\x71\x49\xb6\x57\xd6" - "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" - "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" - "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" - "\xda\x10\x8e\xed\xa2\xa4\x87\xab" - "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" - "\xc9\xac\x42\x31\x95\x7c\xc9\x04" - "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" - "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" - "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" - "\x4c\x96\x12\xed\x7c\x92\x03\x01" - "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" - "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" - "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" - "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" - "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" - "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" - "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" - "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" - "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" - "\x76\x12\x73\x44\x1a\x56\xd7\x72" - "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" - "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" - "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" - "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" - "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" - "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" - "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" - "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" - "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" - "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" - "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" - "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" - "\x8d\x23\x31\x74\x84\xeb\x88\x6e" - "\xcc\xb9\xbc\x22\x83\x19\x07\x22" - "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" - "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" - "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" - "\x3c\xce\x8f\x42\x60\x71\xa7\x75" - "\x08\x40\x65\x8a\x82\xbf\xf5\x43" - "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" - "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" - "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" - "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" - "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" - "\x62\x73\x65\xfd\x46\x63\x25\x3d" - "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" - "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" - "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" - "\xc5\x68\x77\x84\x32\x2b\xcc\x85" - "\x74\x96\xf0\x12\x77\x61\xb9\xeb" - "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" - "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" - "\xda\x39\x87\x45\xc0\x2b\xbb\x01" - "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" - "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" - "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" - "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" - "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" - "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" - "\x21\xc4\xc2\x75\x67\x89\x37\x0a", - .ilen = 512, - .result = "\x30\x38\xeb\xaf\x12\x43\x1a\x89" - "\x62\xa2\x36\xe5\xcf\x77\x1e\xd9" - "\x08\xc3\x0d\xdd\x95\xab\x19\x96" - "\x27\x52\x41\xc3\xca\xfb\xf6\xee" - "\x40\x2d\xdf\xdd\x00\x0c\xb9\x0a" - "\x3a\xf0\xc0\xd1\xda\x63\x9e\x45" - "\x42\xe9\x29\xc0\xb4\x07\xb4\x31" - "\x66\x77\x72\xb5\xb6\xb3\x57\x46" - "\x34\x9a\xfe\x03\xaf\x6b\x36\x07" - "\x63\x8e\xc2\x5d\xa6\x0f\xb6\x7d" - "\xfb\x6d\x82\x51\xb6\x98\xd0\x71" - "\xe7\x10\x7a\xdf\xb2\xbd\xf1\x1d" - "\x72\x2b\x54\x13\xe3\x6d\x79\x37" - "\xa9\x39\x2c\xdf\x21\xab\x87\xd5" - "\xee\xef\x9a\x12\x50\x39\x2e\x1b" - "\x7d\xe6\x6a\x27\x48\xb9\xe7\xac" - "\xaa\xcd\x79\x5f\xf2\xf3\xa0\x08" - "\x6f\x2c\xf4\x0e\xd1\xb8\x89\x25" - "\x31\x9d\xef\xb1\x1d\x27\x55\x04" - "\xc9\x8c\xb7\x68\xdc\xb6\x67\x8a" - "\xdb\xcf\x22\xf2\x3b\x6f\xce\xbb" - "\x26\xbe\x4f\x27\x04\x42\xd1\x44" - "\x4c\x08\xa3\x95\x4c\x7f\x1a\xaf" - "\x1d\x28\x14\xfd\xb1\x1a\x34\x18" - "\xf5\x1e\x28\x69\x95\x6a\x5a\xba" - "\x8e\xb2\x58\x1d\x28\x17\x13\x3d" - "\x38\x7d\x14\x8d\xab\x5d\xf9\xe8" - "\x3c\x0f\x2b\x0d\x2b\x08\xb4\x4b" - "\x6b\x0d\xc8\xa7\x84\xc2\x3a\x1a" - "\xb7\xbd\xda\x92\x29\xb8\x5b\x5a" - "\x63\xa5\x99\x82\x09\x72\x8f\xc6" - "\xa4\x62\x24\x69\x8c\x2d\x26\x00" - "\x99\x83\x91\xd6\xc6\xcf\x57\x67" - "\x38\xea\xf2\xfc\x29\xe0\x73\x39" - "\xf9\x13\x94\x6d\xe2\x58\x28\x75" - "\x3e\xae\x71\x90\x07\x70\x1c\x38" - "\x5b\x4c\x1e\xb5\xa5\x3b\x20\xef" - "\xb1\x4c\x3e\x1a\x72\x62\xbb\x22" - "\x82\x09\xe3\x18\x3f\x4f\x48\xfc" - "\xdd\xac\xfc\xb6\x09\xdb\xd2\x7b" - "\xd6\xb7\x7e\x41\x2f\x14\xf5\x0e" - "\xc3\xac\x4a\xed\xe7\x82\xef\x31" - "\x1f\x1a\x51\x1e\x29\x60\xc8\x98" - "\x93\x51\x1d\x3d\x62\x59\x83\x82" - "\x0c\xf1\xd7\x8d\xac\x33\x44\x81" - "\x3c\x59\xb7\xd4\x5b\x65\x82\xc4" - "\xec\xdc\x24\xfd\x0e\x1a\x79\x94" - "\x34\xb0\x62\xfa\x98\x49\x26\x1f" - "\xf4\x9e\x40\x44\x5b\x1f\xf8\xbe" - "\x36\xff\xc6\xc6\x9d\xf2\xd6\xcc" - "\x63\x93\x29\xb9\x0b\x6d\xd7\x6c" - "\xdb\xf6\x21\x80\xf7\x5a\x37\x15" - "\x0c\xe3\x36\xc8\x74\x75\x20\x91" - "\xdf\x52\x2d\x0c\xe7\x45\xff\x46" - "\xb3\xf4\xec\xc2\xbd\xd3\x37\xb6" - "\x26\xa2\x5d\x7d\x61\xbf\x10\x46" - "\x57\x8d\x05\x96\x70\x0b\xd6\x41" - "\x5c\xe9\xd3\x54\x81\x39\x3a\xdd" - "\x5f\x92\x81\x6e\x35\x03\xd4\x72" - "\x3d\x5a\xe7\xb9\x3b\x0c\x84\x23" - "\x45\x5d\xec\x72\xc1\x52\xef\x2e" - "\x81\x00\xd3\xfe\x4c\x3c\x05\x61" - "\x80\x18\xc4\x6c\x03\xd3\xb7\xba" - "\x11\xd7\xb8\x6e\xea\xe1\x80\x30", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec tf_lrw_dec_tv_template[] = { - /* Generated from AES-LRW test vectors */ - /* same as enc vectors with input and result reversed */ - { - .key = "\x45\x62\xac\x25\xf8\x28\x17\x6d" - "\x4c\x26\x84\x14\xb5\x68\x01\x85" - "\x25\x8e\x2a\x05\xe7\x3e\x9d\x03" - "\xee\x5a\x83\x0c\xcc\x09\x4c\x87", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\xa1\x6c\x50\x69\x26\xa4\xef\x7b" - "\x7c\xc6\x91\xeb\x72\xdd\x9b\xee", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\x59\x70\x47\x14\xf5\x57\x47\x8c" - "\xd7\x79\xe8\x0f\x54\x88\x79\x44" - "\x0d\x48\xf0\xb7\xb1\x5a\x53\xea" - "\x1c\xaa\x6b\x29\xc2\xca\xfb\xaf", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x02", - .input = "\xab\x72\x0a\xad\x3b\x0c\xf0\xc9" - "\x42\x2f\xf1\xae\xf1\x3c\xb1\xbd", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\xd8\x2a\x91\x34\xb2\x6a\x56\x50" - "\x30\xfe\x69\xe2\x37\x7f\x98\x47" - "\xcd\xf9\x0b\x16\x0c\x64\x8f\xb6" - "\xb0\x0d\x0d\x1b\xae\x85\x87\x1f", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x85\xa7\x56\x67\x08\xfa\x42\xe1" - "\x22\xe6\x82\xfc\xd9\xb4\xd7\xd4", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\x0f\x6a\xef\xf8\xd3\xd2\xbb\x15" - "\x25\x83\xf7\x3c\x1f\x01\x28\x74" - "\xca\xc6\xbc\x35\x4d\x4a\x65\x54" - "\x90\xae\x61\xcf\x7b\xae\xbd\xcc" - "\xad\xe4\x94\xc5\x4a\x29\xae\x70", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\xd2\xaf\x69\x35\x24\x1d\x0e\x1c" - "\x84\x8b\x05\xe4\xa2\x2f\x16\xf5", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\x8a\xd4\xee\x10\x2f\xbd\x81\xff" - "\xf8\x86\xce\xac\x93\xc5\xad\xc6" - "\xa0\x19\x07\xc0\x9d\xf7\xbb\xdd" - "\x52\x13\xb2\xb7\xf0\xff\x11\xd8" - "\xd6\x08\xd0\xcd\x2e\xb1\x17\x6f", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x4a\x23\x56\xd7\xff\x90\xd0\x9a" - "\x0d\x7c\x26\xfc\xf0\xf0\xf6\xe4", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\xaf\x26\x05\x9d\x5d\x0a\x58" - "\xe2\xe7\xce\x8a\xb2\x56\x6d\x76", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" - "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" - "\xb2\xfb\x64\xce\x60\x97\x87\x8d" - "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" - "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" - "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\xdf\xcf\xdc\xd2\xe1\xcf\x86\x75" - "\x17\x66\x5e\x0c\x14\xa1\x3d\x40", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x38\xeb\xaf\x12\x43\x1a\x89" - "\x62\xa2\x36\xe5\xcf\x77\x1e\xd9" - "\x08\xc3\x0d\xdd\x95\xab\x19\x96" - "\x27\x52\x41\xc3\xca\xfb\xf6\xee" - "\x40\x2d\xdf\xdd\x00\x0c\xb9\x0a" - "\x3a\xf0\xc0\xd1\xda\x63\x9e\x45" - "\x42\xe9\x29\xc0\xb4\x07\xb4\x31" - "\x66\x77\x72\xb5\xb6\xb3\x57\x46" - "\x34\x9a\xfe\x03\xaf\x6b\x36\x07" - "\x63\x8e\xc2\x5d\xa6\x0f\xb6\x7d" - "\xfb\x6d\x82\x51\xb6\x98\xd0\x71" - "\xe7\x10\x7a\xdf\xb2\xbd\xf1\x1d" - "\x72\x2b\x54\x13\xe3\x6d\x79\x37" - "\xa9\x39\x2c\xdf\x21\xab\x87\xd5" - "\xee\xef\x9a\x12\x50\x39\x2e\x1b" - "\x7d\xe6\x6a\x27\x48\xb9\xe7\xac" - "\xaa\xcd\x79\x5f\xf2\xf3\xa0\x08" - "\x6f\x2c\xf4\x0e\xd1\xb8\x89\x25" - "\x31\x9d\xef\xb1\x1d\x27\x55\x04" - "\xc9\x8c\xb7\x68\xdc\xb6\x67\x8a" - "\xdb\xcf\x22\xf2\x3b\x6f\xce\xbb" - "\x26\xbe\x4f\x27\x04\x42\xd1\x44" - "\x4c\x08\xa3\x95\x4c\x7f\x1a\xaf" - "\x1d\x28\x14\xfd\xb1\x1a\x34\x18" - "\xf5\x1e\x28\x69\x95\x6a\x5a\xba" - "\x8e\xb2\x58\x1d\x28\x17\x13\x3d" - "\x38\x7d\x14\x8d\xab\x5d\xf9\xe8" - "\x3c\x0f\x2b\x0d\x2b\x08\xb4\x4b" - "\x6b\x0d\xc8\xa7\x84\xc2\x3a\x1a" - "\xb7\xbd\xda\x92\x29\xb8\x5b\x5a" - "\x63\xa5\x99\x82\x09\x72\x8f\xc6" - "\xa4\x62\x24\x69\x8c\x2d\x26\x00" - "\x99\x83\x91\xd6\xc6\xcf\x57\x67" - "\x38\xea\xf2\xfc\x29\xe0\x73\x39" - "\xf9\x13\x94\x6d\xe2\x58\x28\x75" - "\x3e\xae\x71\x90\x07\x70\x1c\x38" - "\x5b\x4c\x1e\xb5\xa5\x3b\x20\xef" - "\xb1\x4c\x3e\x1a\x72\x62\xbb\x22" - "\x82\x09\xe3\x18\x3f\x4f\x48\xfc" - "\xdd\xac\xfc\xb6\x09\xdb\xd2\x7b" - "\xd6\xb7\x7e\x41\x2f\x14\xf5\x0e" - "\xc3\xac\x4a\xed\xe7\x82\xef\x31" - "\x1f\x1a\x51\x1e\x29\x60\xc8\x98" - "\x93\x51\x1d\x3d\x62\x59\x83\x82" - "\x0c\xf1\xd7\x8d\xac\x33\x44\x81" - "\x3c\x59\xb7\xd4\x5b\x65\x82\xc4" - "\xec\xdc\x24\xfd\x0e\x1a\x79\x94" - "\x34\xb0\x62\xfa\x98\x49\x26\x1f" - "\xf4\x9e\x40\x44\x5b\x1f\xf8\xbe" - "\x36\xff\xc6\xc6\x9d\xf2\xd6\xcc" - "\x63\x93\x29\xb9\x0b\x6d\xd7\x6c" - "\xdb\xf6\x21\x80\xf7\x5a\x37\x15" - "\x0c\xe3\x36\xc8\x74\x75\x20\x91" - "\xdf\x52\x2d\x0c\xe7\x45\xff\x46" - "\xb3\xf4\xec\xc2\xbd\xd3\x37\xb6" - "\x26\xa2\x5d\x7d\x61\xbf\x10\x46" - "\x57\x8d\x05\x96\x70\x0b\xd6\x41" - "\x5c\xe9\xd3\x54\x81\x39\x3a\xdd" - "\x5f\x92\x81\x6e\x35\x03\xd4\x72" - "\x3d\x5a\xe7\xb9\x3b\x0c\x84\x23" - "\x45\x5d\xec\x72\xc1\x52\xef\x2e" - "\x81\x00\xd3\xfe\x4c\x3c\x05\x61" - "\x80\x18\xc4\x6c\x03\xd3\xb7\xba" - "\x11\xd7\xb8\x6e\xea\xe1\x80\x30", - .ilen = 512, - .result = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" - "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" - "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" - "\x50\x38\x1f\x71\x49\xb6\x57\xd6" - "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" - "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" - "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" - "\xda\x10\x8e\xed\xa2\xa4\x87\xab" - "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" - "\xc9\xac\x42\x31\x95\x7c\xc9\x04" - "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" - "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" - "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" - "\x4c\x96\x12\xed\x7c\x92\x03\x01" - "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" - "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" - "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" - "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" - "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" - "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" - "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" - "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" - "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" - "\x76\x12\x73\x44\x1a\x56\xd7\x72" - "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" - "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" - "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" - "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" - "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" - "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" - "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" - "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" - "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" - "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" - "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" - "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" - "\x8d\x23\x31\x74\x84\xeb\x88\x6e" - "\xcc\xb9\xbc\x22\x83\x19\x07\x22" - "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" - "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" - "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" - "\x3c\xce\x8f\x42\x60\x71\xa7\x75" - "\x08\x40\x65\x8a\x82\xbf\xf5\x43" - "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" - "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" - "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" - "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" - "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" - "\x62\x73\x65\xfd\x46\x63\x25\x3d" - "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" - "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" - "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" - "\xc5\x68\x77\x84\x32\x2b\xcc\x85" - "\x74\x96\xf0\x12\x77\x61\xb9\xeb" - "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" - "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" - "\xda\x39\x87\x45\xc0\x2b\xbb\x01" - "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" - "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" - "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" - "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" - "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" - "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" - "\x21\xc4\xc2\x75\x67\x89\x37\x0a", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec tf_xts_enc_tv_template[] = { - /* Generated from AES-XTS test vectors */ -{ - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 32, - .result = "\x4b\xc9\x44\x4a\x11\xa3\xef\xac" - "\x30\x74\xe4\x44\x52\x77\x97\x43" - "\xa7\x60\xb2\x45\x2e\xf9\x00\x90" - "\x9f\xaa\xfd\x89\x6e\x9d\x4a\xe0", - .rlen = 32, - }, { - .key = "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\x57\x0e\x8f\xe5\x2a\x35\x61\x4f" - "\x32\xd3\xbd\x36\x05\x15\x44\x2c" - "\x58\x06\xf7\xf8\x00\xa8\xb6\xd5" - "\xc6\x28\x92\xdb\xd8\x34\xa2\xe9", - .rlen = 32, - }, { - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\x96\x45\x8f\x8d\x7a\x75\xb1\xde" - "\x40\x0c\x89\x56\xf6\x4d\xa7\x07" - "\x38\xbb\x5b\xe9\xcd\x84\xae\xb2" - "\x7b\x6a\x62\xf4\x8c\xb5\x37\xea", - .rlen = 32, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\xa9\x78\xae\x1e\xea\xa2\x44\x4c" - "\xa2\x7a\x64\x1f\xaf\x46\xc1\xe0" - "\x6c\xb2\xf3\x92\x9a\xd6\x7d\x58" - "\xb8\x2d\xb9\x5d\x58\x07\x66\x50" - "\xea\x35\x35\x8c\xb2\x46\x61\x06" - "\x5d\x65\xfc\x57\x8f\x69\x74\xab" - "\x8a\x06\x69\xb5\x6c\xda\x66\xc7" - "\x52\x90\xbb\x8e\x6d\x8b\xb5\xa2" - "\x78\x1d\xc2\xa9\xc2\x73\x00\xc3" - "\x32\x36\x7c\x97\x6b\x4e\x8a\x50" - "\xe4\x91\x83\x96\x8f\xf4\x94\x1a" - "\xa6\x27\xe1\x33\xcb\x91\xc6\x5f" - "\x94\x75\xbc\xd7\x3e\x3e\x6f\x9e" - "\xa9\x31\x80\x5e\xe5\xdb\xc8\x53" - "\x01\x73\x68\x32\x25\x19\xfa\xfb" - "\xe4\xcf\xb9\x3e\xa2\xa0\x8f\x31" - "\xbf\x54\x06\x93\xa8\xb1\x0f\xb6" - "\x7c\x3c\xde\x6f\x0f\xfb\x0c\x11" - "\x39\x80\x39\x09\x97\x65\xf2\x83" - "\xae\xe6\xa1\x6f\x47\xb8\x49\xde" - "\x99\x36\x20\x7d\x97\x3b\xec\xfa" - "\xb4\x33\x6e\x7a\xc7\x46\x84\x49" - "\x91\xcd\xe1\x57\x0d\xed\x40\x08" - "\x13\xf1\x4e\x3e\xa4\xa4\x5c\xe6" - "\xd2\x0c\x20\x8f\x3e\xdf\x3f\x47" - "\x9a\x2f\xde\x6d\x66\xc9\x99\x4a" - "\x2d\x9e\x9d\x4b\x1a\x27\xa2\x12" - "\x99\xf0\xf8\xb1\xb6\xf6\x57\xc3" - "\xca\x1c\xa3\x8e\xed\x39\x28\xb5" - "\x10\x1b\x4b\x08\x42\x00\x4a\xd3" - "\xad\x5a\xc6\x8e\xc8\xbb\x95\xc4" - "\x4b\xaa\xfe\xd5\x42\xa8\xa3\x6d" - "\x3c\xf3\x34\x91\x2d\xb4\xdd\x20" - "\x0c\x90\x6d\xa3\x9b\x66\x9d\x24" - "\x02\xa6\xa9\x3f\x3f\x58\x5d\x47" - "\x24\x65\x63\x7e\xbd\x8c\xe6\x52" - "\x7d\xef\x33\x53\x63\xec\xaa\x0b" - "\x64\x15\xa9\xa6\x1f\x10\x00\x38" - "\x35\xa8\xe7\xbe\x23\x70\x22\xe0" - "\xd3\xb9\xe6\xfd\xe6\xaa\x03\x50" - "\xf3\x3c\x27\x36\x8b\xcc\xfe\x9c" - "\x9c\xa3\xb3\xe7\x68\x9b\xa2\x71" - "\xe0\x07\xd9\x1f\x68\x1f\xac\x5e" - "\x7a\x74\x85\xa9\x6a\x90\xab\x2c" - "\x38\x51\xbc\x1f\x43\x4a\x56\x1c" - "\xf8\x47\x03\x4e\x67\xa8\x1f\x99" - "\x04\x39\x73\x32\xb2\x86\x79\xe7" - "\x14\x28\x70\xb8\xe2\x7d\x69\x85" - "\xb6\x0f\xc5\xd0\xd0\x01\x5c\xe6" - "\x09\x0f\x75\xf7\xb6\x81\xd2\x11" - "\x20\x9c\xa1\xee\x11\x44\x79\xd0" - "\xb2\x34\x77\xda\x10\x9a\x6f\x6f" - "\xef\x7c\xd9\xdc\x35\xb7\x61\xdd" - "\xf1\xa4\xc6\x1c\xbf\x05\x22\xac" - "\xfe\x2f\x85\x00\x44\xdf\x33\x16" - "\x35\xb6\xa3\xd3\x70\xdf\x69\x35" - "\x6a\xc7\xb4\x99\x45\x27\xc8\x8e" - "\x5a\x14\x30\xd0\x55\x3e\x4f\x64" - "\x0d\x38\xe3\xdf\x8b\xa8\x93\x26" - "\x75\xae\xf6\xb5\x23\x0b\x17\x31" - "\xbf\x27\xb8\xb5\x94\x31\xa7\x8f" - "\x43\xc4\x46\x24\x22\x4f\x8f\x7e" - "\xe5\xf4\x6d\x1e\x0e\x18\x7a\xbb" - "\xa6\x8f\xfb\x49\x49\xd8\x7e\x5a", - .rlen = 512, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\xd7\x4b\x93\x7d\x13\xa2\xa2\xe1" - "\x35\x39\x71\x88\x76\x1e\xc9\xea" - "\x86\xad\xf3\x14\x48\x3d\x5e\xe9" - "\xe9\x2d\xb2\x56\x59\x35\x9d\xec" - "\x84\xfa\x7e\x9d\x6d\x33\x36\x8f" - "\xce\xf4\xa9\x21\x0b\x5f\x96\xec" - "\xcb\xf9\x57\x68\x33\x88\x39\xbf" - "\x2f\xbb\x59\x03\xbd\x66\x8b\x11" - "\x11\x65\x51\x2e\xb8\x67\x05\xd1" - "\x27\x11\x5c\xd4\xcc\x97\xc2\xb3" - "\xa9\x55\xaf\x07\x56\xd1\xdc\xf5" - "\x85\xdc\x46\xe6\xf0\x24\xeb\x93" - "\x4d\xf0\x9b\xf5\x73\x1c\xda\x03" - "\x22\xc8\x3a\x4f\xb4\x19\x91\x09" - "\x54\x0b\xf6\xfe\x17\x3d\x1a\x53" - "\x72\x60\x79\xcb\x0e\x32\x8a\x77" - "\xd5\xed\xdb\x33\xd7\x62\x16\x69" - "\x63\xe0\xab\xb5\xf6\x9c\x5f\x3d" - "\x69\x35\x61\x86\xf8\x86\xb9\x89" - "\x6e\x59\x35\xac\xf6\x6b\x33\xa0" - "\xea\xef\x96\x62\xd8\xa9\xcf\x56" - "\xbf\xdb\x8a\xfd\xa1\x82\x77\x73" - "\x3d\x94\x4a\x49\x42\x6d\x08\x60" - "\xa1\xea\xab\xb6\x88\x13\x94\xb8" - "\x51\x98\xdb\x35\x85\xdf\xf6\xb9" - "\x8f\xcd\xdf\x80\xd3\x40\x2d\x72" - "\xb8\xb2\x6c\x02\x43\x35\x22\x2a" - "\x31\xed\xcd\x16\x19\xdf\x62\x0f" - "\x29\xcf\x87\x04\xec\x02\x4f\xe4" - "\xa2\xed\x73\xc6\x69\xd3\x7e\x89" - "\x0b\x76\x10\x7c\xd6\xf9\x6a\x25" - "\xed\xcc\x60\x5d\x61\x20\xc1\x97" - "\x56\x91\x57\x28\xbe\x71\x0d\xcd" - "\xde\xc4\x9e\x55\x91\xbe\xd1\x28" - "\x9b\x90\xeb\x73\xf3\x68\x51\xc6" - "\xdf\x82\xcc\xd8\x1f\xce\x5b\x27" - "\xc0\x60\x5e\x33\xd6\xa7\x20\xea" - "\xb2\x54\xc7\x5d\x6a\x3b\x67\x47" - "\xcf\xa0\xe3\xab\x86\xaf\xc1\x42" - "\xe6\xb0\x23\x4a\xaf\x53\xdf\xa0" - "\xad\x12\x32\x31\x03\xf7\x21\xbe" - "\x2d\xd5\x82\x42\xb6\x4a\x3d\xcd" - "\xd8\x81\x77\xa9\x49\x98\x6c\x09" - "\xc5\xa3\x61\x12\x62\x85\x6b\xcd" - "\xb3\xf4\x20\x0c\x41\xc4\x05\x37" - "\x46\x5f\xeb\x71\x8b\xf1\xaf\x6e" - "\xba\xf3\x50\x2e\xfe\xa8\x37\xeb" - "\xe8\x8c\x4f\xa4\x0c\xf1\x31\xc8" - "\x6e\x71\x4f\xa5\xd7\x97\x73\xe0" - "\x93\x4a\x2f\xda\x7b\xe0\x20\x54" - "\x1f\x8d\x85\x79\x0b\x7b\x5e\x75" - "\xb9\x07\x67\xcc\xc8\xe7\x21\x15" - "\xa7\xc8\x98\xff\x4b\x80\x1c\x12" - "\xa8\x54\xe1\x38\x52\xe6\x74\x81" - "\x97\x47\xa1\x41\x0e\xc0\x50\xe3" - "\x55\x0e\xc3\xa7\x70\x77\xce\x07" - "\xed\x8c\x88\xe6\xa1\x5b\x14\xec" - "\xe6\xde\x06\x6d\x74\xc5\xd9\xfa" - "\xe5\x2f\x5a\xff\xc8\x05\xee\x27" - "\x35\x61\xbf\x0b\x19\x78\x9b\xd2" - "\x04\xc7\x05\xb1\x79\xb4\xff\x5f" - "\xf3\xea\x67\x52\x78\xc2\xce\x70" - "\xa4\x05\x0b\xb2\xb3\xa8\x30\x97" - "\x37\x30\xe1\x91\x8d\xb3\x2a\xff", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec tf_xts_dec_tv_template[] = { - /* Generated from AES-XTS test vectors */ - /* same as enc vectors with input and result reversed */ - { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x4b\xc9\x44\x4a\x11\xa3\xef\xac" - "\x30\x74\xe4\x44\x52\x77\x97\x43" - "\xa7\x60\xb2\x45\x2e\xf9\x00\x90" - "\x9f\xaa\xfd\x89\x6e\x9d\x4a\xe0", - .ilen = 32, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 32, - }, { - .key = "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x57\x0e\x8f\xe5\x2a\x35\x61\x4f" - "\x32\xd3\xbd\x36\x05\x15\x44\x2c" - "\x58\x06\xf7\xf8\x00\xa8\xb6\xd5" - "\xc6\x28\x92\xdb\xd8\x34\xa2\xe9", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x96\x45\x8f\x8d\x7a\x75\xb1\xde" - "\x40\x0c\x89\x56\xf6\x4d\xa7\x07" - "\x38\xbb\x5b\xe9\xcd\x84\xae\xb2" - "\x7b\x6a\x62\xf4\x8c\xb5\x37\xea", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xa9\x78\xae\x1e\xea\xa2\x44\x4c" - "\xa2\x7a\x64\x1f\xaf\x46\xc1\xe0" - "\x6c\xb2\xf3\x92\x9a\xd6\x7d\x58" - "\xb8\x2d\xb9\x5d\x58\x07\x66\x50" - "\xea\x35\x35\x8c\xb2\x46\x61\x06" - "\x5d\x65\xfc\x57\x8f\x69\x74\xab" - "\x8a\x06\x69\xb5\x6c\xda\x66\xc7" - "\x52\x90\xbb\x8e\x6d\x8b\xb5\xa2" - "\x78\x1d\xc2\xa9\xc2\x73\x00\xc3" - "\x32\x36\x7c\x97\x6b\x4e\x8a\x50" - "\xe4\x91\x83\x96\x8f\xf4\x94\x1a" - "\xa6\x27\xe1\x33\xcb\x91\xc6\x5f" - "\x94\x75\xbc\xd7\x3e\x3e\x6f\x9e" - "\xa9\x31\x80\x5e\xe5\xdb\xc8\x53" - "\x01\x73\x68\x32\x25\x19\xfa\xfb" - "\xe4\xcf\xb9\x3e\xa2\xa0\x8f\x31" - "\xbf\x54\x06\x93\xa8\xb1\x0f\xb6" - "\x7c\x3c\xde\x6f\x0f\xfb\x0c\x11" - "\x39\x80\x39\x09\x97\x65\xf2\x83" - "\xae\xe6\xa1\x6f\x47\xb8\x49\xde" - "\x99\x36\x20\x7d\x97\x3b\xec\xfa" - "\xb4\x33\x6e\x7a\xc7\x46\x84\x49" - "\x91\xcd\xe1\x57\x0d\xed\x40\x08" - "\x13\xf1\x4e\x3e\xa4\xa4\x5c\xe6" - "\xd2\x0c\x20\x8f\x3e\xdf\x3f\x47" - "\x9a\x2f\xde\x6d\x66\xc9\x99\x4a" - "\x2d\x9e\x9d\x4b\x1a\x27\xa2\x12" - "\x99\xf0\xf8\xb1\xb6\xf6\x57\xc3" - "\xca\x1c\xa3\x8e\xed\x39\x28\xb5" - "\x10\x1b\x4b\x08\x42\x00\x4a\xd3" - "\xad\x5a\xc6\x8e\xc8\xbb\x95\xc4" - "\x4b\xaa\xfe\xd5\x42\xa8\xa3\x6d" - "\x3c\xf3\x34\x91\x2d\xb4\xdd\x20" - "\x0c\x90\x6d\xa3\x9b\x66\x9d\x24" - "\x02\xa6\xa9\x3f\x3f\x58\x5d\x47" - "\x24\x65\x63\x7e\xbd\x8c\xe6\x52" - "\x7d\xef\x33\x53\x63\xec\xaa\x0b" - "\x64\x15\xa9\xa6\x1f\x10\x00\x38" - "\x35\xa8\xe7\xbe\x23\x70\x22\xe0" - "\xd3\xb9\xe6\xfd\xe6\xaa\x03\x50" - "\xf3\x3c\x27\x36\x8b\xcc\xfe\x9c" - "\x9c\xa3\xb3\xe7\x68\x9b\xa2\x71" - "\xe0\x07\xd9\x1f\x68\x1f\xac\x5e" - "\x7a\x74\x85\xa9\x6a\x90\xab\x2c" - "\x38\x51\xbc\x1f\x43\x4a\x56\x1c" - "\xf8\x47\x03\x4e\x67\xa8\x1f\x99" - "\x04\x39\x73\x32\xb2\x86\x79\xe7" - "\x14\x28\x70\xb8\xe2\x7d\x69\x85" - "\xb6\x0f\xc5\xd0\xd0\x01\x5c\xe6" - "\x09\x0f\x75\xf7\xb6\x81\xd2\x11" - "\x20\x9c\xa1\xee\x11\x44\x79\xd0" - "\xb2\x34\x77\xda\x10\x9a\x6f\x6f" - "\xef\x7c\xd9\xdc\x35\xb7\x61\xdd" - "\xf1\xa4\xc6\x1c\xbf\x05\x22\xac" - "\xfe\x2f\x85\x00\x44\xdf\x33\x16" - "\x35\xb6\xa3\xd3\x70\xdf\x69\x35" - "\x6a\xc7\xb4\x99\x45\x27\xc8\x8e" - "\x5a\x14\x30\xd0\x55\x3e\x4f\x64" - "\x0d\x38\xe3\xdf\x8b\xa8\x93\x26" - "\x75\xae\xf6\xb5\x23\x0b\x17\x31" - "\xbf\x27\xb8\xb5\x94\x31\xa7\x8f" - "\x43\xc4\x46\x24\x22\x4f\x8f\x7e" - "\xe5\xf4\x6d\x1e\x0e\x18\x7a\xbb" - "\xa6\x8f\xfb\x49\x49\xd8\x7e\x5a", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xd7\x4b\x93\x7d\x13\xa2\xa2\xe1" - "\x35\x39\x71\x88\x76\x1e\xc9\xea" - "\x86\xad\xf3\x14\x48\x3d\x5e\xe9" - "\xe9\x2d\xb2\x56\x59\x35\x9d\xec" - "\x84\xfa\x7e\x9d\x6d\x33\x36\x8f" - "\xce\xf4\xa9\x21\x0b\x5f\x96\xec" - "\xcb\xf9\x57\x68\x33\x88\x39\xbf" - "\x2f\xbb\x59\x03\xbd\x66\x8b\x11" - "\x11\x65\x51\x2e\xb8\x67\x05\xd1" - "\x27\x11\x5c\xd4\xcc\x97\xc2\xb3" - "\xa9\x55\xaf\x07\x56\xd1\xdc\xf5" - "\x85\xdc\x46\xe6\xf0\x24\xeb\x93" - "\x4d\xf0\x9b\xf5\x73\x1c\xda\x03" - "\x22\xc8\x3a\x4f\xb4\x19\x91\x09" - "\x54\x0b\xf6\xfe\x17\x3d\x1a\x53" - "\x72\x60\x79\xcb\x0e\x32\x8a\x77" - "\xd5\xed\xdb\x33\xd7\x62\x16\x69" - "\x63\xe0\xab\xb5\xf6\x9c\x5f\x3d" - "\x69\x35\x61\x86\xf8\x86\xb9\x89" - "\x6e\x59\x35\xac\xf6\x6b\x33\xa0" - "\xea\xef\x96\x62\xd8\xa9\xcf\x56" - "\xbf\xdb\x8a\xfd\xa1\x82\x77\x73" - "\x3d\x94\x4a\x49\x42\x6d\x08\x60" - "\xa1\xea\xab\xb6\x88\x13\x94\xb8" - "\x51\x98\xdb\x35\x85\xdf\xf6\xb9" - "\x8f\xcd\xdf\x80\xd3\x40\x2d\x72" - "\xb8\xb2\x6c\x02\x43\x35\x22\x2a" - "\x31\xed\xcd\x16\x19\xdf\x62\x0f" - "\x29\xcf\x87\x04\xec\x02\x4f\xe4" - "\xa2\xed\x73\xc6\x69\xd3\x7e\x89" - "\x0b\x76\x10\x7c\xd6\xf9\x6a\x25" - "\xed\xcc\x60\x5d\x61\x20\xc1\x97" - "\x56\x91\x57\x28\xbe\x71\x0d\xcd" - "\xde\xc4\x9e\x55\x91\xbe\xd1\x28" - "\x9b\x90\xeb\x73\xf3\x68\x51\xc6" - "\xdf\x82\xcc\xd8\x1f\xce\x5b\x27" - "\xc0\x60\x5e\x33\xd6\xa7\x20\xea" - "\xb2\x54\xc7\x5d\x6a\x3b\x67\x47" - "\xcf\xa0\xe3\xab\x86\xaf\xc1\x42" - "\xe6\xb0\x23\x4a\xaf\x53\xdf\xa0" - "\xad\x12\x32\x31\x03\xf7\x21\xbe" - "\x2d\xd5\x82\x42\xb6\x4a\x3d\xcd" - "\xd8\x81\x77\xa9\x49\x98\x6c\x09" - "\xc5\xa3\x61\x12\x62\x85\x6b\xcd" - "\xb3\xf4\x20\x0c\x41\xc4\x05\x37" - "\x46\x5f\xeb\x71\x8b\xf1\xaf\x6e" - "\xba\xf3\x50\x2e\xfe\xa8\x37\xeb" - "\xe8\x8c\x4f\xa4\x0c\xf1\x31\xc8" - "\x6e\x71\x4f\xa5\xd7\x97\x73\xe0" - "\x93\x4a\x2f\xda\x7b\xe0\x20\x54" - "\x1f\x8d\x85\x79\x0b\x7b\x5e\x75" - "\xb9\x07\x67\xcc\xc8\xe7\x21\x15" - "\xa7\xc8\x98\xff\x4b\x80\x1c\x12" - "\xa8\x54\xe1\x38\x52\xe6\x74\x81" - "\x97\x47\xa1\x41\x0e\xc0\x50\xe3" - "\x55\x0e\xc3\xa7\x70\x77\xce\x07" - "\xed\x8c\x88\xe6\xa1\x5b\x14\xec" - "\xe6\xde\x06\x6d\x74\xc5\xd9\xfa" - "\xe5\x2f\x5a\xff\xc8\x05\xee\x27" - "\x35\x61\xbf\x0b\x19\x78\x9b\xd2" - "\x04\xc7\x05\xb1\x79\xb4\xff\x5f" - "\xf3\xea\x67\x52\x78\xc2\xce\x70" - "\xa4\x05\x0b\xb2\xb3\xa8\x30\x97" - "\x37\x30\xe1\x91\x8d\xb3\x2a\xff", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -/* - * Serpent test vectors. These are backwards because Serpent writes - * octet sequences in right-to-left mode. - */ -static const struct cipher_testvec serpent_enc_tv_template[] = { - { - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .ilen = 16, - .result = "\x12\x07\xfc\xce\x9b\xd0\xd6\x47" - "\x6a\xe9\x8f\xbe\xd1\x43\xa0\xe2", - .rlen = 16, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .klen = 16, - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .ilen = 16, - .result = "\x4c\x7d\x8a\x32\x80\x72\xa2\x2c" - "\x82\x3e\x4a\x1f\x3a\xcd\xa1\x6d", - .rlen = 16, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .klen = 32, - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .ilen = 16, - .result = "\xde\x26\x9f\xf8\x33\xe4\x32\xb8" - "\x5b\x2e\x88\xd2\x70\x1c\xe7\x5c", - .rlen = 16, - }, { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80", - .klen = 16, - .input = zeroed_string, - .ilen = 16, - .result = "\xdd\xd2\x6b\x98\xa5\xff\xd8\x2c" - "\x05\x34\x5a\x9d\xad\xbf\xaf\x49", - .rlen = 16, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\xFB\xB0\x5D\xDE\xC0\xFE\xFC\xEB" - "\xB1\x80\x10\x43\xDE\x62\x70\xBD" - "\xFA\x8A\x93\xEA\x6B\xF7\xC5\xD7" - "\x0C\xD1\xBB\x29\x25\x14\x4C\x22" - "\x77\xA6\x38\x00\xDB\xB9\xE2\x07" - "\xD1\xAC\x82\xBA\xEA\x67\xAA\x39" - "\x99\x34\x89\x5B\x54\xE9\x12\x13" - "\x3B\x04\xE5\x12\x42\xC5\x79\xAB" - "\x0D\xC7\x3C\x58\x2D\xA3\x98\xF6" - "\xE4\x61\x9E\x17\x0B\xCE\xE8\xAA" - "\xB5\x6C\x1A\x3A\x67\x52\x81\x6A" - "\x04\xFF\x8A\x1B\x96\xFE\xE6\x87" - "\x3C\xD4\x39\x7D\x36\x9B\x03\xD5" - "\xB6\xA0\x75\x3C\x83\xE6\x1C\x73" - "\x9D\x74\x2B\x77\x53\x2D\xE5\xBD" - "\x69\xDA\x7A\x01\xF5\x6A\x70\x39" - "\x30\xD4\x2C\xF2\x8E\x06\x4B\x39" - "\xB3\x12\x1D\xB3\x17\x46\xE6\xD6" - "\xB6\x31\x36\x34\x38\x3C\x1D\x69" - "\x9F\x47\x28\x9A\x1D\x96\x70\x54" - "\x8E\x88\xCB\xE0\xF5\x6A\xAE\x0A" - "\x3C\xD5\x93\x1C\x21\xC9\x14\x3A" - "\x23\x9C\x9B\x79\xC7\x75\xC8\x39" - "\xA6\xAC\x65\x9A\x99\x37\xAF\x6D" - "\xBD\xB5\x32\xFD\xD8\x9C\x95\x7B" - "\xC6\x6A\x80\x64\xEA\xEF\x6D\x3F" - "\xA9\xFE\x5B\x16\xA3\xCF\x32\xC8" - "\xEF\x50\x22\x20\x93\x30\xBE\xE2" - "\x38\x05\x65\xAF\xBA\xB6\xE4\x72" - "\xA9\xEE\x05\x42\x88\xBD\x9D\x49" - "\xAD\x93\xCA\x4D\x45\x11\x43\x4D" - "\xB8\xF5\x74\x2B\x48\xE7\x21\xE4" - "\x4E\x3A\x4C\xDE\x65\x7A\x5A\xAD" - "\x86\xE6\x23\xEC\x6B\xA7\x17\xE6" - "\xF6\xA1\xAC\x29\xAE\xF9\x9B\x69" - "\x73\x65\x65\x51\xD6\x0B\x4E\x8C" - "\x17\x15\x9D\xB0\xCF\xB2\x42\x2B" - "\x51\xC3\x03\xE8\xB7\x7D\x2D\x39" - "\xE8\x10\x93\x16\xC8\x68\x4C\x60" - "\x87\x70\x14\xD0\x01\x57\xCB\x42" - "\x13\x59\xB1\x7F\x12\x4F\xBB\xC7" - "\xBD\x2B\xD4\xA9\x12\x26\x4F\xDE" - "\xFD\x72\xEC\xD7\x6F\x97\x14\x90" - "\x0E\x37\x13\xE6\x67\x1D\xE5\xFE" - "\x9E\x18\x3C\x8F\x3A\x3F\x59\x9B" - "\x71\x80\x05\x35\x3F\x40\x0B\x21" - "\x76\xE5\xEF\x42\x6C\xDB\x31\x05" - "\x5F\x05\xCF\x14\xE3\xF0\x61\xA2" - "\x49\x03\x5E\x77\x2E\x20\xBA\xA1" - "\xAF\x46\x51\xC0\x2B\xC4\x64\x1E" - "\x65\xCC\x51\x58\x0A\xDF\xF0\x5F" - "\x75\x9F\x48\xCD\x81\xEC\xC3\xF6" - "\xED\xC9\x4B\x7B\x4E\x26\x23\xE1" - "\xBB\xE9\x83\x0B\xCF\xE4\xDE\x00" - "\x48\xFF\xBF\x6C\xB4\x72\x16\xEF" - "\xC7\x46\xEE\x48\x8C\xB8\xAF\x45" - "\x91\x76\xE7\x6E\x65\x3D\x15\x86" - "\x10\xF8\xDB\x66\x97\x7C\x43\x4D" - "\x79\x12\x4E\xCE\x06\xD1\xD1\x6A" - "\x34\xC1\xC9\xF2\x28\x4A\xCD\x02" - "\x75\x55\x9B\xFF\x36\x73\xAB\x7C" - "\xF4\x46\x2E\xEB\xAC\xF3\xD2\xB7", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec tnepres_enc_tv_template[] = { - { /* KeySize=128, PT=0, I=1 */ - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .key = "\x80\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 16, - .ilen = 16, - .result = "\x49\xaf\xbf\xad\x9d\x5a\x34\x05" - "\x2c\xd8\xff\xa5\x98\x6b\xd2\xdd", - .rlen = 16, - }, { /* KeySize=192, PT=0, I=1 */ - .key = "\x80\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 24, - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 16, - .result = "\xe7\x8e\x54\x02\xc7\x19\x55\x68" - "\xac\x36\x78\xf7\xa3\xf6\x0c\x66", - .rlen = 16, - }, { /* KeySize=256, PT=0, I=1 */ - .key = "\x80\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 32, - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 16, - .result = "\xab\xed\x96\xe7\x66\xbf\x28\xcb" - "\xc0\xeb\xd2\x1a\x82\xef\x08\x19", - .rlen = 16, - }, { /* KeySize=256, I=257 */ - .key = "\x1f\x1e\x1d\x1c\x1b\x1a\x19\x18" - "\x17\x16\x15\x14\x13\x12\x11\x10" - "\x0f\x0e\x0d\x0c\x0b\x0a\x09\x08" - "\x07\x06\x05\x04\x03\x02\x01\x00", - .klen = 32, - .input = "\x0f\x0e\x0d\x0c\x0b\x0a\x09\x08" - "\x07\x06\x05\x04\x03\x02\x01\x00", - .ilen = 16, - .result = "\x5c\xe7\x1c\x70\xd2\x88\x2e\x5b" - "\xb8\x32\xe4\x33\xf8\x9f\x26\xde", - .rlen = 16, - }, -}; - - -static const struct cipher_testvec serpent_dec_tv_template[] = { - { - .input = "\x12\x07\xfc\xce\x9b\xd0\xd6\x47" - "\x6a\xe9\x8f\xbe\xd1\x43\xa0\xe2", - .ilen = 16, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .rlen = 16, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .klen = 16, - .input = "\x4c\x7d\x8a\x32\x80\x72\xa2\x2c" - "\x82\x3e\x4a\x1f\x3a\xcd\xa1\x6d", - .ilen = 16, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .rlen = 16, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .klen = 32, - .input = "\xde\x26\x9f\xf8\x33\xe4\x32\xb8" - "\x5b\x2e\x88\xd2\x70\x1c\xe7\x5c", - .ilen = 16, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .rlen = 16, - }, { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80", - .klen = 16, - .input = "\xdd\xd2\x6b\x98\xa5\xff\xd8\x2c" - "\x05\x34\x5a\x9d\xad\xbf\xaf\x49", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .input = "\xFB\xB0\x5D\xDE\xC0\xFE\xFC\xEB" - "\xB1\x80\x10\x43\xDE\x62\x70\xBD" - "\xFA\x8A\x93\xEA\x6B\xF7\xC5\xD7" - "\x0C\xD1\xBB\x29\x25\x14\x4C\x22" - "\x77\xA6\x38\x00\xDB\xB9\xE2\x07" - "\xD1\xAC\x82\xBA\xEA\x67\xAA\x39" - "\x99\x34\x89\x5B\x54\xE9\x12\x13" - "\x3B\x04\xE5\x12\x42\xC5\x79\xAB" - "\x0D\xC7\x3C\x58\x2D\xA3\x98\xF6" - "\xE4\x61\x9E\x17\x0B\xCE\xE8\xAA" - "\xB5\x6C\x1A\x3A\x67\x52\x81\x6A" - "\x04\xFF\x8A\x1B\x96\xFE\xE6\x87" - "\x3C\xD4\x39\x7D\x36\x9B\x03\xD5" - "\xB6\xA0\x75\x3C\x83\xE6\x1C\x73" - "\x9D\x74\x2B\x77\x53\x2D\xE5\xBD" - "\x69\xDA\x7A\x01\xF5\x6A\x70\x39" - "\x30\xD4\x2C\xF2\x8E\x06\x4B\x39" - "\xB3\x12\x1D\xB3\x17\x46\xE6\xD6" - "\xB6\x31\x36\x34\x38\x3C\x1D\x69" - "\x9F\x47\x28\x9A\x1D\x96\x70\x54" - "\x8E\x88\xCB\xE0\xF5\x6A\xAE\x0A" - "\x3C\xD5\x93\x1C\x21\xC9\x14\x3A" - "\x23\x9C\x9B\x79\xC7\x75\xC8\x39" - "\xA6\xAC\x65\x9A\x99\x37\xAF\x6D" - "\xBD\xB5\x32\xFD\xD8\x9C\x95\x7B" - "\xC6\x6A\x80\x64\xEA\xEF\x6D\x3F" - "\xA9\xFE\x5B\x16\xA3\xCF\x32\xC8" - "\xEF\x50\x22\x20\x93\x30\xBE\xE2" - "\x38\x05\x65\xAF\xBA\xB6\xE4\x72" - "\xA9\xEE\x05\x42\x88\xBD\x9D\x49" - "\xAD\x93\xCA\x4D\x45\x11\x43\x4D" - "\xB8\xF5\x74\x2B\x48\xE7\x21\xE4" - "\x4E\x3A\x4C\xDE\x65\x7A\x5A\xAD" - "\x86\xE6\x23\xEC\x6B\xA7\x17\xE6" - "\xF6\xA1\xAC\x29\xAE\xF9\x9B\x69" - "\x73\x65\x65\x51\xD6\x0B\x4E\x8C" - "\x17\x15\x9D\xB0\xCF\xB2\x42\x2B" - "\x51\xC3\x03\xE8\xB7\x7D\x2D\x39" - "\xE8\x10\x93\x16\xC8\x68\x4C\x60" - "\x87\x70\x14\xD0\x01\x57\xCB\x42" - "\x13\x59\xB1\x7F\x12\x4F\xBB\xC7" - "\xBD\x2B\xD4\xA9\x12\x26\x4F\xDE" - "\xFD\x72\xEC\xD7\x6F\x97\x14\x90" - "\x0E\x37\x13\xE6\x67\x1D\xE5\xFE" - "\x9E\x18\x3C\x8F\x3A\x3F\x59\x9B" - "\x71\x80\x05\x35\x3F\x40\x0B\x21" - "\x76\xE5\xEF\x42\x6C\xDB\x31\x05" - "\x5F\x05\xCF\x14\xE3\xF0\x61\xA2" - "\x49\x03\x5E\x77\x2E\x20\xBA\xA1" - "\xAF\x46\x51\xC0\x2B\xC4\x64\x1E" - "\x65\xCC\x51\x58\x0A\xDF\xF0\x5F" - "\x75\x9F\x48\xCD\x81\xEC\xC3\xF6" - "\xED\xC9\x4B\x7B\x4E\x26\x23\xE1" - "\xBB\xE9\x83\x0B\xCF\xE4\xDE\x00" - "\x48\xFF\xBF\x6C\xB4\x72\x16\xEF" - "\xC7\x46\xEE\x48\x8C\xB8\xAF\x45" - "\x91\x76\xE7\x6E\x65\x3D\x15\x86" - "\x10\xF8\xDB\x66\x97\x7C\x43\x4D" - "\x79\x12\x4E\xCE\x06\xD1\xD1\x6A" - "\x34\xC1\xC9\xF2\x28\x4A\xCD\x02" - "\x75\x55\x9B\xFF\x36\x73\xAB\x7C" - "\xF4\x46\x2E\xEB\xAC\xF3\xD2\xB7", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec tnepres_dec_tv_template[] = { - { - .input = "\x41\xcc\x6b\x31\x59\x31\x45\x97" - "\x6d\x6f\xbb\x38\x4b\x37\x21\x28", - .ilen = 16, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .rlen = 16, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .klen = 16, - .input = "\xea\xf4\xd7\xfc\xd8\x01\x34\x47" - "\x81\x45\x0b\xfa\x0c\xd6\xad\x6e", - .ilen = 16, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .rlen = 16, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .klen = 32, - .input = "\x64\xa9\x1a\x37\xed\x9f\xe7\x49" - "\xa8\x4e\x76\xd6\xf5\x0d\x78\xee", - .ilen = 16, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .rlen = 16, - }, { /* KeySize=128, I=121 */ - .key = "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80", - .klen = 16, - .input = "\x3d\xda\xbf\xc0\x06\xda\xab\x06" - "\x46\x2a\xf4\xef\x81\x54\x4e\x26", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, - }, -}; - -static const struct cipher_testvec serpent_cbc_enc_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\x80\xCF\x11\x41\x1A\xB9\x4B\x9C" - "\xFF\xB7\x6C\xEA\xF0\xAF\x77\x6E" - "\x71\x75\x95\x9D\x4E\x1C\xCF\xAD" - "\x81\x34\xE9\x8F\xAE\x5A\x91\x1C" - "\x38\x63\x35\x7E\x79\x18\x0A\xE8" - "\x67\x06\x76\xD5\xFF\x22\x2F\xDA" - "\xB6\x2D\x57\x13\xB6\x3C\xBC\x97" - "\xFE\x53\x75\x35\x97\x7F\x51\xEA" - "\xDF\x5D\xE8\x9D\xCC\xD9\xAE\xE7" - "\x62\x67\xFF\x04\xC2\x18\x22\x5F" - "\x2E\x06\xC1\xE2\x26\xCD\xC6\x1E" - "\xE5\x2C\x4E\x87\x23\xDD\xF0\x41" - "\x08\xA5\xB4\x3E\x07\x1E\x0B\xBB" - "\x72\x84\xF8\x0A\x3F\x38\x5E\x91" - "\x15\x26\xE1\xDB\xA4\x3D\x74\xD2" - "\x41\x1E\x3F\xA9\xC6\x7D\x2A\xAB" - "\x27\xDF\x89\x1D\x86\x3E\xF7\x5A" - "\xF6\xE3\x0F\xC7\x6B\x4C\x96\x7C" - "\x2D\x12\xA5\x05\x92\xCB\xD7\x4A" - "\x4D\x1E\x88\x21\xE1\x63\xB4\xFC" - "\x4A\xF2\xCD\x35\xB9\xD7\x70\x97" - "\x5A\x5E\x7E\x96\x52\x20\xDC\x25" - "\xE9\x6B\x36\xB4\xE0\x98\x85\x2C" - "\x3C\xD2\xF7\x78\x8A\x73\x26\x9B" - "\xAF\x0B\x11\xE8\x4D\x67\x23\xE9" - "\x77\xDF\x58\xF6\x6F\x9E\xA4\xC5" - "\x10\xA1\x82\x0E\x80\xA0\x8F\x4B" - "\xA1\xC0\x12\x54\x4E\xC9\x20\x92" - "\x11\x00\x10\x4E\xB3\x7C\xCA\x63" - "\xE5\x3F\xD3\x41\x37\xCD\x74\xB7" - "\xA5\x7C\x61\xB8\x0B\x7A\x7F\x4D" - "\xFE\x96\x7D\x1B\xBE\x60\x37\xB7" - "\x81\x92\x66\x67\x15\x1E\x39\x98" - "\x52\xC0\xF4\x69\xC0\x99\x4F\x5A" - "\x2E\x32\xAD\x7C\x8B\xE9\xAD\x05" - "\x55\xF9\x0A\x1F\x97\x5C\xFA\x2B" - "\xF4\x99\x76\x3A\x6E\x4D\xE1\x4C" - "\x14\x4E\x6F\x87\xEE\x1A\x85\xA3" - "\x96\xC6\x66\x49\xDA\x0D\x71\xAC" - "\x04\x05\x46\xD3\x90\x0F\x64\x64" - "\x01\x66\x2C\x62\x5D\x34\xD1\xCB" - "\x3A\x24\xCE\x95\xEF\xAE\x2C\x97" - "\x0E\x0C\x1D\x36\x49\xEB\xE9\x3D" - "\x62\xA6\x19\x28\x9E\x26\xB4\x3F" - "\xD7\x55\x42\x3C\xCD\x72\x0A\xF0" - "\x7D\xE9\x95\x45\x86\xED\xB1\xE0" - "\x8D\xE9\xC5\x86\x13\x24\x28\x7D" - "\x74\xEF\xCA\x50\x12\x7E\x64\x8F" - "\x1B\xF5\x5B\xFE\xE2\xAC\xFA\xE7" - "\xBD\x38\x8C\x11\x20\xEF\xB1\xAA" - "\x7B\xE5\xE5\x78\xAD\x9D\x2D\xA2" - "\x8E\xDD\x48\xB3\xEF\x18\x92\x7E" - "\xE6\x75\x0D\x54\x64\x11\xA3\x3A" - "\xDB\x97\x0F\xD3\xDF\x07\xD3\x7E" - "\x1E\xD1\x87\xE4\x74\xBB\x46\xF4" - "\xBA\x23\x2D\x8D\x29\x07\x12\xCF" - "\x34\xCD\x72\x7F\x01\x30\xE7\xA0" - "\xF8\xDD\xA8\x08\xF0\xBC\xB1\xA2" - "\xCC\xE1\x6B\x5F\xBE\xEA\xF1\xE4" - "\x02\xC4\xAF\xFA\xAD\x31\xF4\xBF" - "\xFC\x66\xAA\x37\xF2\x37\x39\x6B" - "\xBC\x08\x3A\xA2\x29\xB3\xDF\xD1", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec serpent_cbc_dec_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x80\xCF\x11\x41\x1A\xB9\x4B\x9C" - "\xFF\xB7\x6C\xEA\xF0\xAF\x77\x6E" - "\x71\x75\x95\x9D\x4E\x1C\xCF\xAD" - "\x81\x34\xE9\x8F\xAE\x5A\x91\x1C" - "\x38\x63\x35\x7E\x79\x18\x0A\xE8" - "\x67\x06\x76\xD5\xFF\x22\x2F\xDA" - "\xB6\x2D\x57\x13\xB6\x3C\xBC\x97" - "\xFE\x53\x75\x35\x97\x7F\x51\xEA" - "\xDF\x5D\xE8\x9D\xCC\xD9\xAE\xE7" - "\x62\x67\xFF\x04\xC2\x18\x22\x5F" - "\x2E\x06\xC1\xE2\x26\xCD\xC6\x1E" - "\xE5\x2C\x4E\x87\x23\xDD\xF0\x41" - "\x08\xA5\xB4\x3E\x07\x1E\x0B\xBB" - "\x72\x84\xF8\x0A\x3F\x38\x5E\x91" - "\x15\x26\xE1\xDB\xA4\x3D\x74\xD2" - "\x41\x1E\x3F\xA9\xC6\x7D\x2A\xAB" - "\x27\xDF\x89\x1D\x86\x3E\xF7\x5A" - "\xF6\xE3\x0F\xC7\x6B\x4C\x96\x7C" - "\x2D\x12\xA5\x05\x92\xCB\xD7\x4A" - "\x4D\x1E\x88\x21\xE1\x63\xB4\xFC" - "\x4A\xF2\xCD\x35\xB9\xD7\x70\x97" - "\x5A\x5E\x7E\x96\x52\x20\xDC\x25" - "\xE9\x6B\x36\xB4\xE0\x98\x85\x2C" - "\x3C\xD2\xF7\x78\x8A\x73\x26\x9B" - "\xAF\x0B\x11\xE8\x4D\x67\x23\xE9" - "\x77\xDF\x58\xF6\x6F\x9E\xA4\xC5" - "\x10\xA1\x82\x0E\x80\xA0\x8F\x4B" - "\xA1\xC0\x12\x54\x4E\xC9\x20\x92" - "\x11\x00\x10\x4E\xB3\x7C\xCA\x63" - "\xE5\x3F\xD3\x41\x37\xCD\x74\xB7" - "\xA5\x7C\x61\xB8\x0B\x7A\x7F\x4D" - "\xFE\x96\x7D\x1B\xBE\x60\x37\xB7" - "\x81\x92\x66\x67\x15\x1E\x39\x98" - "\x52\xC0\xF4\x69\xC0\x99\x4F\x5A" - "\x2E\x32\xAD\x7C\x8B\xE9\xAD\x05" - "\x55\xF9\x0A\x1F\x97\x5C\xFA\x2B" - "\xF4\x99\x76\x3A\x6E\x4D\xE1\x4C" - "\x14\x4E\x6F\x87\xEE\x1A\x85\xA3" - "\x96\xC6\x66\x49\xDA\x0D\x71\xAC" - "\x04\x05\x46\xD3\x90\x0F\x64\x64" - "\x01\x66\x2C\x62\x5D\x34\xD1\xCB" - "\x3A\x24\xCE\x95\xEF\xAE\x2C\x97" - "\x0E\x0C\x1D\x36\x49\xEB\xE9\x3D" - "\x62\xA6\x19\x28\x9E\x26\xB4\x3F" - "\xD7\x55\x42\x3C\xCD\x72\x0A\xF0" - "\x7D\xE9\x95\x45\x86\xED\xB1\xE0" - "\x8D\xE9\xC5\x86\x13\x24\x28\x7D" - "\x74\xEF\xCA\x50\x12\x7E\x64\x8F" - "\x1B\xF5\x5B\xFE\xE2\xAC\xFA\xE7" - "\xBD\x38\x8C\x11\x20\xEF\xB1\xAA" - "\x7B\xE5\xE5\x78\xAD\x9D\x2D\xA2" - "\x8E\xDD\x48\xB3\xEF\x18\x92\x7E" - "\xE6\x75\x0D\x54\x64\x11\xA3\x3A" - "\xDB\x97\x0F\xD3\xDF\x07\xD3\x7E" - "\x1E\xD1\x87\xE4\x74\xBB\x46\xF4" - "\xBA\x23\x2D\x8D\x29\x07\x12\xCF" - "\x34\xCD\x72\x7F\x01\x30\xE7\xA0" - "\xF8\xDD\xA8\x08\xF0\xBC\xB1\xA2" - "\xCC\xE1\x6B\x5F\xBE\xEA\xF1\xE4" - "\x02\xC4\xAF\xFA\xAD\x31\xF4\xBF" - "\xFC\x66\xAA\x37\xF2\x37\x39\x6B" - "\xBC\x08\x3A\xA2\x29\xB3\xDF\xD1", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec serpent_ctr_enc_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\x84\x68\xEC\xF2\x1C\x88\x20\xCA" - "\x37\x69\xE3\x3A\x22\x85\x48\x46" - "\x70\xAA\x25\xB4\xCD\x8B\x04\x4E" - "\x8D\x15\x2B\x98\xDF\x7B\x6D\xB9" - "\xE0\x4A\x73\x00\x65\xB6\x1A\x0D" - "\x5C\x60\xDF\x34\xDC\x60\x4C\xDF" - "\xB5\x1F\x26\x8C\xDA\xC1\x11\xA8" - "\x80\xFA\x37\x7A\x89\xAA\xAE\x7B" - "\x92\x6E\xB9\xDC\xC9\x62\x4F\x88" - "\x0A\x5D\x97\x2F\x6B\xAC\x03\x7C" - "\x22\xF6\x55\x5A\xFA\x35\xA5\x17" - "\xA1\x5C\x5E\x2B\x63\x2D\xB9\x91" - "\x3E\x83\x26\x00\x4E\xD5\xBE\xCE" - "\x79\xC4\x3D\xFC\x70\xA0\xAD\x96" - "\xBA\x58\x2A\x1C\xDF\xC2\x3A\xA5" - "\x7C\xB5\x12\x89\xED\xBF\xB6\x09" - "\x13\x4F\x7D\x61\x3C\x5C\x27\xFC" - "\x5D\xE1\x4F\xA1\xEA\xB3\xCA\xB9" - "\xE6\xD0\x97\x81\xDE\xD1\xFB\x8A" - "\x30\xDB\xA3\x5D\xEC\x25\x0B\x86" - "\x71\xC8\xA7\x67\xE8\xBC\x7D\x4C" - "\xAE\x82\xD3\x73\x31\x09\xCB\xB3" - "\x4D\xD4\xC0\x8A\x2B\xFA\xA6\x55" - "\x39\x0A\xBC\x6E\x75\xAB\xC2\xE2" - "\x8A\xF2\x26\xCD\x63\x38\x35\xF7" - "\xAE\x12\x83\xCD\x8A\x9E\x7E\x4C" - "\xFE\x4D\xD7\xCE\x5C\x6E\x4C\xAF" - "\xE3\xCD\x76\xA7\x87\xA1\x54\x7C" - "\xEC\x32\xC7\x83\x2A\xFF\xF8\xEA" - "\x87\xB2\x47\xA3\x9D\xC2\x9C\xA2" - "\xB7\x2C\x7C\x1A\x24\xCB\x88\x61" - "\xFF\xA7\x1A\x16\x01\xDD\x4B\xFC" - "\x2E\xE0\x48\x67\x09\x42\xCC\x91" - "\xBE\x20\x38\xC0\x5E\x3B\x95\x00" - "\xA1\x96\x66\x0B\x8A\xE9\x9E\xF7" - "\x6B\x34\x0A\x51\xC0\x3B\xEB\x71" - "\x07\x97\x38\x4B\x5C\x56\x98\x67" - "\x78\x9C\xD0\x0E\x2B\xB5\x67\x90" - "\x75\xF8\xFE\x6D\x4E\x85\xCC\x0D" - "\x18\x06\x15\x9D\x5A\x10\x13\x37" - "\xA3\xD6\x68\xA2\xDF\x7E\xC7\x12" - "\xC9\x0D\x4D\x91\xB0\x2A\x55\xFF" - "\x6F\x73\x13\xDF\x28\xB5\x2A\x2C" - "\xE4\xFC\x20\xD9\xF1\x7A\x82\xB1" - "\xCB\x57\xB6\x3D\x8C\xF4\x8E\x27" - "\x37\xDC\x35\xF3\x79\x01\x53\xA4" - "\x7B\x37\xDE\x7C\x04\xAE\x50\xDB" - "\x9B\x1E\x8C\x07\xA7\x52\x49\x50" - "\x34\x25\x65\xDD\xA9\x8F\x7E\xBD" - "\x7A\xC9\x36\xAE\xDE\x21\x48\x64" - "\xC2\x02\xBA\xBE\x11\x1E\x3D\x9C" - "\x98\x52\xCC\x04\xBD\x5E\x61\x26" - "\x10\xD3\x21\xD9\x6E\x25\x98\x77" - "\x8E\x98\x63\xF6\xF6\x52\xFB\x13" - "\xAA\x30\xF2\xB9\xA4\x43\x53\x39" - "\x1C\x97\x07\x7E\x6B\xFF\x3D\x43" - "\xA6\x71\x6B\x66\x8F\x58\x3F\x71" - "\x90\x47\x40\x92\xE6\x69\xD1\x96" - "\x34\xB3\x3B\xE5\x43\xE4\xD5\x56" - "\xB2\xE6\x7E\x86\x7A\x12\x17\x5B" - "\x30\xF3\x9B\x0D\xFA\x57\xE4\x50" - "\x40\x53\x77\x8C\x15\xF8\x8D\x13", - .rlen = 496, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59", - .ilen = 499, - .result = "\x84\x68\xEC\xF2\x1C\x88\x20\xCA" - "\x37\x69\xE3\x3A\x22\x85\x48\x46" - "\x70\xAA\x25\xB4\xCD\x8B\x04\x4E" - "\x8D\x15\x2B\x98\xDF\x7B\x6D\xB9" - "\xE0\x4A\x73\x00\x65\xB6\x1A\x0D" - "\x5C\x60\xDF\x34\xDC\x60\x4C\xDF" - "\xB5\x1F\x26\x8C\xDA\xC1\x11\xA8" - "\x80\xFA\x37\x7A\x89\xAA\xAE\x7B" - "\x92\x6E\xB9\xDC\xC9\x62\x4F\x88" - "\x0A\x5D\x97\x2F\x6B\xAC\x03\x7C" - "\x22\xF6\x55\x5A\xFA\x35\xA5\x17" - "\xA1\x5C\x5E\x2B\x63\x2D\xB9\x91" - "\x3E\x83\x26\x00\x4E\xD5\xBE\xCE" - "\x79\xC4\x3D\xFC\x70\xA0\xAD\x96" - "\xBA\x58\x2A\x1C\xDF\xC2\x3A\xA5" - "\x7C\xB5\x12\x89\xED\xBF\xB6\x09" - "\x13\x4F\x7D\x61\x3C\x5C\x27\xFC" - "\x5D\xE1\x4F\xA1\xEA\xB3\xCA\xB9" - "\xE6\xD0\x97\x81\xDE\xD1\xFB\x8A" - "\x30\xDB\xA3\x5D\xEC\x25\x0B\x86" - "\x71\xC8\xA7\x67\xE8\xBC\x7D\x4C" - "\xAE\x82\xD3\x73\x31\x09\xCB\xB3" - "\x4D\xD4\xC0\x8A\x2B\xFA\xA6\x55" - "\x39\x0A\xBC\x6E\x75\xAB\xC2\xE2" - "\x8A\xF2\x26\xCD\x63\x38\x35\xF7" - "\xAE\x12\x83\xCD\x8A\x9E\x7E\x4C" - "\xFE\x4D\xD7\xCE\x5C\x6E\x4C\xAF" - "\xE3\xCD\x76\xA7\x87\xA1\x54\x7C" - "\xEC\x32\xC7\x83\x2A\xFF\xF8\xEA" - "\x87\xB2\x47\xA3\x9D\xC2\x9C\xA2" - "\xB7\x2C\x7C\x1A\x24\xCB\x88\x61" - "\xFF\xA7\x1A\x16\x01\xDD\x4B\xFC" - "\x2E\xE0\x48\x67\x09\x42\xCC\x91" - "\xBE\x20\x38\xC0\x5E\x3B\x95\x00" - "\xA1\x96\x66\x0B\x8A\xE9\x9E\xF7" - "\x6B\x34\x0A\x51\xC0\x3B\xEB\x71" - "\x07\x97\x38\x4B\x5C\x56\x98\x67" - "\x78\x9C\xD0\x0E\x2B\xB5\x67\x90" - "\x75\xF8\xFE\x6D\x4E\x85\xCC\x0D" - "\x18\x06\x15\x9D\x5A\x10\x13\x37" - "\xA3\xD6\x68\xA2\xDF\x7E\xC7\x12" - "\xC9\x0D\x4D\x91\xB0\x2A\x55\xFF" - "\x6F\x73\x13\xDF\x28\xB5\x2A\x2C" - "\xE4\xFC\x20\xD9\xF1\x7A\x82\xB1" - "\xCB\x57\xB6\x3D\x8C\xF4\x8E\x27" - "\x37\xDC\x35\xF3\x79\x01\x53\xA4" - "\x7B\x37\xDE\x7C\x04\xAE\x50\xDB" - "\x9B\x1E\x8C\x07\xA7\x52\x49\x50" - "\x34\x25\x65\xDD\xA9\x8F\x7E\xBD" - "\x7A\xC9\x36\xAE\xDE\x21\x48\x64" - "\xC2\x02\xBA\xBE\x11\x1E\x3D\x9C" - "\x98\x52\xCC\x04\xBD\x5E\x61\x26" - "\x10\xD3\x21\xD9\x6E\x25\x98\x77" - "\x8E\x98\x63\xF6\xF6\x52\xFB\x13" - "\xAA\x30\xF2\xB9\xA4\x43\x53\x39" - "\x1C\x97\x07\x7E\x6B\xFF\x3D\x43" - "\xA6\x71\x6B\x66\x8F\x58\x3F\x71" - "\x90\x47\x40\x92\xE6\x69\xD1\x96" - "\x34\xB3\x3B\xE5\x43\xE4\xD5\x56" - "\xB2\xE6\x7E\x86\x7A\x12\x17\x5B" - "\x30\xF3\x9B\x0D\xFA\x57\xE4\x50" - "\x40\x53\x77\x8C\x15\xF8\x8D\x13" - "\x38\xE2\xE5", - .rlen = 499, - .also_non_np = 1, - .np = 2, - .tap = { 499 - 16, 16 }, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\x06\x9A\xF8\xB4\x53\x88\x62\xFC" - "\x68\xB8\x2E\xDF\xC1\x05\x0F\x3D" - "\xAF\x4D\x95\xAE\xC4\xE9\x1C\xDC" - "\xF6\x2B\x8F\x90\x89\xF6\x7E\x1A" - "\xA6\xB9\xE4\xF4\xFA\xCA\xE5\x7E" - "\x71\x28\x06\x4F\xE8\x08\x39\xDA" - "\xA5\x0E\xC8\xC0\xB8\x16\xE5\x69" - "\xE5\xCA\xEC\x4F\x63\x2C\xC0\x9B" - "\x9F\x3E\x39\x79\xF0\xCD\x64\x35" - "\x4A\xD3\xC8\xA9\x31\xCD\x48\x5B" - "\x92\x3D\x8F\x3F\x96\xBD\xB3\x18" - "\x74\x2A\x5D\x29\x3F\x57\x8F\xE2" - "\x67\x9A\xE0\xE5\xD4\x4A\xE2\x47" - "\xBC\xF6\xEB\x14\xF3\x8C\x20\xC2" - "\x7D\xE2\x43\x81\x86\x72\x2E\xB1" - "\x39\xF6\x95\xE1\x1F\xCB\x76\x33" - "\x5B\x7D\x23\x0F\x3A\x67\x2A\x2F" - "\xB9\x37\x9D\xDD\x1F\x16\xA1\x3C" - "\x70\xFE\x52\xAA\x93\x3C\xC4\x46" - "\xB1\xE5\xFF\xDA\xAF\xE2\x84\xFE" - "\x25\x92\xB2\x63\xBD\x49\x77\xB4" - "\x22\xA4\x6A\xD5\x04\xE0\x45\x58" - "\x1C\x34\x96\x7C\x03\x0C\x13\xA2" - "\x05\x22\xE2\xCB\x5A\x35\x03\x09" - "\x40\xD2\x82\x05\xCA\x58\x73\xF2" - "\x29\x5E\x01\x47\x13\x32\x78\xBE" - "\x06\xB0\x51\xDB\x6C\x31\xA0\x1C" - "\x74\xBC\x8D\x25\xDF\xF8\x65\xD1" - "\x38\x35\x11\x26\x4A\xB4\x06\x32" - "\xFA\xD2\x07\x77\xB3\x74\x98\x80" - "\x61\x59\xA8\x9F\xF3\x6F\x2A\xBF" - "\xE6\xA5\x9A\xC4\x6B\xA6\x49\x6F" - "\xBC\x47\xD9\xFB\xC6\xEF\x25\x65" - "\x96\xAC\x9F\xE4\x81\x4B\xD8\xBA" - "\xD6\x9B\xC9\x6D\x58\x40\x81\x02" - "\x73\x44\x4E\x43\x6E\x37\xBB\x11" - "\xE3\xF9\xB8\x2F\xEC\x76\x34\xEA" - "\x90\xCD\xB7\x2E\x0E\x32\x71\xE8" - "\xBB\x4E\x0B\x98\xA4\x17\x17\x5B" - "\x07\xB5\x82\x3A\xC4\xE8\x42\x51" - "\x5A\x4C\x4E\x7D\xBF\xC4\xC0\x4F" - "\x68\xB8\xC6\x4A\x32\x6F\x0B\xD7" - "\x85\xED\x6B\xFB\x72\xD2\xA5\x8F" - "\xBF\xF9\xAC\x59\x50\xA8\x08\x70" - "\xEC\xBD\x0A\xBF\xE5\x87\xA1\xC2" - "\x92\x14\x78\xAF\xE8\xEA\x2E\xDD" - "\xC1\x03\x9A\xAA\x89\x8B\x32\x46" - "\x5B\x18\x27\xBA\x46\xAA\x64\xDE" - "\xE3\xD5\xA3\xFC\x7B\x5B\x61\xDB" - "\x7E\xDA\xEC\x30\x17\x19\xF8\x80" - "\xB5\x5E\x27\xB5\x37\x3A\x1F\x28" - "\x07\x73\xC3\x63\xCE\xFF\x8C\xFE" - "\x81\x4E\xF8\x24\xF3\xB8\xC7\xE8" - "\x16\x9A\xCC\x58\x2F\x88\x1C\x4B" - "\xBB\x33\xA2\x73\xF0\x1C\x89\x0E" - "\xDC\x34\x27\x89\x98\xCE\x1C\xA2" - "\xD8\xB8\x90\xBE\xEC\x72\x28\x13" - "\xAC\x7B\xF1\xD0\x7F\x7A\x28\x50" - "\xB7\x99\x65\x8A\xC9\xC6\x21\x34" - "\x7F\x67\x9D\xB7\x2C\xCC\xF5\x17" - "\x2B\x89\xAC\xB0\xD7\x1E\x47\xB0" - "\x61\xAF\xD4\x63\x6D\xB8\x2D\x20", - .rlen = 496, - }, -}; - -static const struct cipher_testvec serpent_ctr_dec_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x84\x68\xEC\xF2\x1C\x88\x20\xCA" - "\x37\x69\xE3\x3A\x22\x85\x48\x46" - "\x70\xAA\x25\xB4\xCD\x8B\x04\x4E" - "\x8D\x15\x2B\x98\xDF\x7B\x6D\xB9" - "\xE0\x4A\x73\x00\x65\xB6\x1A\x0D" - "\x5C\x60\xDF\x34\xDC\x60\x4C\xDF" - "\xB5\x1F\x26\x8C\xDA\xC1\x11\xA8" - "\x80\xFA\x37\x7A\x89\xAA\xAE\x7B" - "\x92\x6E\xB9\xDC\xC9\x62\x4F\x88" - "\x0A\x5D\x97\x2F\x6B\xAC\x03\x7C" - "\x22\xF6\x55\x5A\xFA\x35\xA5\x17" - "\xA1\x5C\x5E\x2B\x63\x2D\xB9\x91" - "\x3E\x83\x26\x00\x4E\xD5\xBE\xCE" - "\x79\xC4\x3D\xFC\x70\xA0\xAD\x96" - "\xBA\x58\x2A\x1C\xDF\xC2\x3A\xA5" - "\x7C\xB5\x12\x89\xED\xBF\xB6\x09" - "\x13\x4F\x7D\x61\x3C\x5C\x27\xFC" - "\x5D\xE1\x4F\xA1\xEA\xB3\xCA\xB9" - "\xE6\xD0\x97\x81\xDE\xD1\xFB\x8A" - "\x30\xDB\xA3\x5D\xEC\x25\x0B\x86" - "\x71\xC8\xA7\x67\xE8\xBC\x7D\x4C" - "\xAE\x82\xD3\x73\x31\x09\xCB\xB3" - "\x4D\xD4\xC0\x8A\x2B\xFA\xA6\x55" - "\x39\x0A\xBC\x6E\x75\xAB\xC2\xE2" - "\x8A\xF2\x26\xCD\x63\x38\x35\xF7" - "\xAE\x12\x83\xCD\x8A\x9E\x7E\x4C" - "\xFE\x4D\xD7\xCE\x5C\x6E\x4C\xAF" - "\xE3\xCD\x76\xA7\x87\xA1\x54\x7C" - "\xEC\x32\xC7\x83\x2A\xFF\xF8\xEA" - "\x87\xB2\x47\xA3\x9D\xC2\x9C\xA2" - "\xB7\x2C\x7C\x1A\x24\xCB\x88\x61" - "\xFF\xA7\x1A\x16\x01\xDD\x4B\xFC" - "\x2E\xE0\x48\x67\x09\x42\xCC\x91" - "\xBE\x20\x38\xC0\x5E\x3B\x95\x00" - "\xA1\x96\x66\x0B\x8A\xE9\x9E\xF7" - "\x6B\x34\x0A\x51\xC0\x3B\xEB\x71" - "\x07\x97\x38\x4B\x5C\x56\x98\x67" - "\x78\x9C\xD0\x0E\x2B\xB5\x67\x90" - "\x75\xF8\xFE\x6D\x4E\x85\xCC\x0D" - "\x18\x06\x15\x9D\x5A\x10\x13\x37" - "\xA3\xD6\x68\xA2\xDF\x7E\xC7\x12" - "\xC9\x0D\x4D\x91\xB0\x2A\x55\xFF" - "\x6F\x73\x13\xDF\x28\xB5\x2A\x2C" - "\xE4\xFC\x20\xD9\xF1\x7A\x82\xB1" - "\xCB\x57\xB6\x3D\x8C\xF4\x8E\x27" - "\x37\xDC\x35\xF3\x79\x01\x53\xA4" - "\x7B\x37\xDE\x7C\x04\xAE\x50\xDB" - "\x9B\x1E\x8C\x07\xA7\x52\x49\x50" - "\x34\x25\x65\xDD\xA9\x8F\x7E\xBD" - "\x7A\xC9\x36\xAE\xDE\x21\x48\x64" - "\xC2\x02\xBA\xBE\x11\x1E\x3D\x9C" - "\x98\x52\xCC\x04\xBD\x5E\x61\x26" - "\x10\xD3\x21\xD9\x6E\x25\x98\x77" - "\x8E\x98\x63\xF6\xF6\x52\xFB\x13" - "\xAA\x30\xF2\xB9\xA4\x43\x53\x39" - "\x1C\x97\x07\x7E\x6B\xFF\x3D\x43" - "\xA6\x71\x6B\x66\x8F\x58\x3F\x71" - "\x90\x47\x40\x92\xE6\x69\xD1\x96" - "\x34\xB3\x3B\xE5\x43\xE4\xD5\x56" - "\xB2\xE6\x7E\x86\x7A\x12\x17\x5B" - "\x30\xF3\x9B\x0D\xFA\x57\xE4\x50" - "\x40\x53\x77\x8C\x15\xF8\x8D\x13", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x84\x68\xEC\xF2\x1C\x88\x20\xCA" - "\x37\x69\xE3\x3A\x22\x85\x48\x46" - "\x70\xAA\x25\xB4\xCD\x8B\x04\x4E" - "\x8D\x15\x2B\x98\xDF\x7B\x6D\xB9" - "\xE0\x4A\x73\x00\x65\xB6\x1A\x0D" - "\x5C\x60\xDF\x34\xDC\x60\x4C\xDF" - "\xB5\x1F\x26\x8C\xDA\xC1\x11\xA8" - "\x80\xFA\x37\x7A\x89\xAA\xAE\x7B" - "\x92\x6E\xB9\xDC\xC9\x62\x4F\x88" - "\x0A\x5D\x97\x2F\x6B\xAC\x03\x7C" - "\x22\xF6\x55\x5A\xFA\x35\xA5\x17" - "\xA1\x5C\x5E\x2B\x63\x2D\xB9\x91" - "\x3E\x83\x26\x00\x4E\xD5\xBE\xCE" - "\x79\xC4\x3D\xFC\x70\xA0\xAD\x96" - "\xBA\x58\x2A\x1C\xDF\xC2\x3A\xA5" - "\x7C\xB5\x12\x89\xED\xBF\xB6\x09" - "\x13\x4F\x7D\x61\x3C\x5C\x27\xFC" - "\x5D\xE1\x4F\xA1\xEA\xB3\xCA\xB9" - "\xE6\xD0\x97\x81\xDE\xD1\xFB\x8A" - "\x30\xDB\xA3\x5D\xEC\x25\x0B\x86" - "\x71\xC8\xA7\x67\xE8\xBC\x7D\x4C" - "\xAE\x82\xD3\x73\x31\x09\xCB\xB3" - "\x4D\xD4\xC0\x8A\x2B\xFA\xA6\x55" - "\x39\x0A\xBC\x6E\x75\xAB\xC2\xE2" - "\x8A\xF2\x26\xCD\x63\x38\x35\xF7" - "\xAE\x12\x83\xCD\x8A\x9E\x7E\x4C" - "\xFE\x4D\xD7\xCE\x5C\x6E\x4C\xAF" - "\xE3\xCD\x76\xA7\x87\xA1\x54\x7C" - "\xEC\x32\xC7\x83\x2A\xFF\xF8\xEA" - "\x87\xB2\x47\xA3\x9D\xC2\x9C\xA2" - "\xB7\x2C\x7C\x1A\x24\xCB\x88\x61" - "\xFF\xA7\x1A\x16\x01\xDD\x4B\xFC" - "\x2E\xE0\x48\x67\x09\x42\xCC\x91" - "\xBE\x20\x38\xC0\x5E\x3B\x95\x00" - "\xA1\x96\x66\x0B\x8A\xE9\x9E\xF7" - "\x6B\x34\x0A\x51\xC0\x3B\xEB\x71" - "\x07\x97\x38\x4B\x5C\x56\x98\x67" - "\x78\x9C\xD0\x0E\x2B\xB5\x67\x90" - "\x75\xF8\xFE\x6D\x4E\x85\xCC\x0D" - "\x18\x06\x15\x9D\x5A\x10\x13\x37" - "\xA3\xD6\x68\xA2\xDF\x7E\xC7\x12" - "\xC9\x0D\x4D\x91\xB0\x2A\x55\xFF" - "\x6F\x73\x13\xDF\x28\xB5\x2A\x2C" - "\xE4\xFC\x20\xD9\xF1\x7A\x82\xB1" - "\xCB\x57\xB6\x3D\x8C\xF4\x8E\x27" - "\x37\xDC\x35\xF3\x79\x01\x53\xA4" - "\x7B\x37\xDE\x7C\x04\xAE\x50\xDB" - "\x9B\x1E\x8C\x07\xA7\x52\x49\x50" - "\x34\x25\x65\xDD\xA9\x8F\x7E\xBD" - "\x7A\xC9\x36\xAE\xDE\x21\x48\x64" - "\xC2\x02\xBA\xBE\x11\x1E\x3D\x9C" - "\x98\x52\xCC\x04\xBD\x5E\x61\x26" - "\x10\xD3\x21\xD9\x6E\x25\x98\x77" - "\x8E\x98\x63\xF6\xF6\x52\xFB\x13" - "\xAA\x30\xF2\xB9\xA4\x43\x53\x39" - "\x1C\x97\x07\x7E\x6B\xFF\x3D\x43" - "\xA6\x71\x6B\x66\x8F\x58\x3F\x71" - "\x90\x47\x40\x92\xE6\x69\xD1\x96" - "\x34\xB3\x3B\xE5\x43\xE4\xD5\x56" - "\xB2\xE6\x7E\x86\x7A\x12\x17\x5B" - "\x30\xF3\x9B\x0D\xFA\x57\xE4\x50" - "\x40\x53\x77\x8C\x15\xF8\x8D\x13" - "\x38\xE2\xE5", - .ilen = 499, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59", - .rlen = 499, - .also_non_np = 1, - .np = 2, - .tap = { 499 - 16, 16 }, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x06\x9A\xF8\xB4\x53\x88\x62\xFC" - "\x68\xB8\x2E\xDF\xC1\x05\x0F\x3D" - "\xAF\x4D\x95\xAE\xC4\xE9\x1C\xDC" - "\xF6\x2B\x8F\x90\x89\xF6\x7E\x1A" - "\xA6\xB9\xE4\xF4\xFA\xCA\xE5\x7E" - "\x71\x28\x06\x4F\xE8\x08\x39\xDA" - "\xA5\x0E\xC8\xC0\xB8\x16\xE5\x69" - "\xE5\xCA\xEC\x4F\x63\x2C\xC0\x9B" - "\x9F\x3E\x39\x79\xF0\xCD\x64\x35" - "\x4A\xD3\xC8\xA9\x31\xCD\x48\x5B" - "\x92\x3D\x8F\x3F\x96\xBD\xB3\x18" - "\x74\x2A\x5D\x29\x3F\x57\x8F\xE2" - "\x67\x9A\xE0\xE5\xD4\x4A\xE2\x47" - "\xBC\xF6\xEB\x14\xF3\x8C\x20\xC2" - "\x7D\xE2\x43\x81\x86\x72\x2E\xB1" - "\x39\xF6\x95\xE1\x1F\xCB\x76\x33" - "\x5B\x7D\x23\x0F\x3A\x67\x2A\x2F" - "\xB9\x37\x9D\xDD\x1F\x16\xA1\x3C" - "\x70\xFE\x52\xAA\x93\x3C\xC4\x46" - "\xB1\xE5\xFF\xDA\xAF\xE2\x84\xFE" - "\x25\x92\xB2\x63\xBD\x49\x77\xB4" - "\x22\xA4\x6A\xD5\x04\xE0\x45\x58" - "\x1C\x34\x96\x7C\x03\x0C\x13\xA2" - "\x05\x22\xE2\xCB\x5A\x35\x03\x09" - "\x40\xD2\x82\x05\xCA\x58\x73\xF2" - "\x29\x5E\x01\x47\x13\x32\x78\xBE" - "\x06\xB0\x51\xDB\x6C\x31\xA0\x1C" - "\x74\xBC\x8D\x25\xDF\xF8\x65\xD1" - "\x38\x35\x11\x26\x4A\xB4\x06\x32" - "\xFA\xD2\x07\x77\xB3\x74\x98\x80" - "\x61\x59\xA8\x9F\xF3\x6F\x2A\xBF" - "\xE6\xA5\x9A\xC4\x6B\xA6\x49\x6F" - "\xBC\x47\xD9\xFB\xC6\xEF\x25\x65" - "\x96\xAC\x9F\xE4\x81\x4B\xD8\xBA" - "\xD6\x9B\xC9\x6D\x58\x40\x81\x02" - "\x73\x44\x4E\x43\x6E\x37\xBB\x11" - "\xE3\xF9\xB8\x2F\xEC\x76\x34\xEA" - "\x90\xCD\xB7\x2E\x0E\x32\x71\xE8" - "\xBB\x4E\x0B\x98\xA4\x17\x17\x5B" - "\x07\xB5\x82\x3A\xC4\xE8\x42\x51" - "\x5A\x4C\x4E\x7D\xBF\xC4\xC0\x4F" - "\x68\xB8\xC6\x4A\x32\x6F\x0B\xD7" - "\x85\xED\x6B\xFB\x72\xD2\xA5\x8F" - "\xBF\xF9\xAC\x59\x50\xA8\x08\x70" - "\xEC\xBD\x0A\xBF\xE5\x87\xA1\xC2" - "\x92\x14\x78\xAF\xE8\xEA\x2E\xDD" - "\xC1\x03\x9A\xAA\x89\x8B\x32\x46" - "\x5B\x18\x27\xBA\x46\xAA\x64\xDE" - "\xE3\xD5\xA3\xFC\x7B\x5B\x61\xDB" - "\x7E\xDA\xEC\x30\x17\x19\xF8\x80" - "\xB5\x5E\x27\xB5\x37\x3A\x1F\x28" - "\x07\x73\xC3\x63\xCE\xFF\x8C\xFE" - "\x81\x4E\xF8\x24\xF3\xB8\xC7\xE8" - "\x16\x9A\xCC\x58\x2F\x88\x1C\x4B" - "\xBB\x33\xA2\x73\xF0\x1C\x89\x0E" - "\xDC\x34\x27\x89\x98\xCE\x1C\xA2" - "\xD8\xB8\x90\xBE\xEC\x72\x28\x13" - "\xAC\x7B\xF1\xD0\x7F\x7A\x28\x50" - "\xB7\x99\x65\x8A\xC9\xC6\x21\x34" - "\x7F\x67\x9D\xB7\x2C\xCC\xF5\x17" - "\x2B\x89\xAC\xB0\xD7\x1E\x47\xB0" - "\x61\xAF\xD4\x63\x6D\xB8\x2D\x20", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - }, -}; - -static const struct cipher_testvec serpent_lrw_enc_tv_template[] = { - /* Generated from AES-LRW test vectors */ - { - .key = "\x45\x62\xac\x25\xf8\x28\x17\x6d" - "\x4c\x26\x84\x14\xb5\x68\x01\x85" - "\x25\x8e\x2a\x05\xe7\x3e\x9d\x03" - "\xee\x5a\x83\x0c\xcc\x09\x4c\x87", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x6f\xbf\xd4\xa4\x5d\x71\x16\x79" - "\x63\x9c\xa6\x8e\x40\xbe\x0d\x8a", - .rlen = 16, - }, { - .key = "\x59\x70\x47\x14\xf5\x57\x47\x8c" - "\xd7\x79\xe8\x0f\x54\x88\x79\x44" - "\x0d\x48\xf0\xb7\xb1\x5a\x53\xea" - "\x1c\xaa\x6b\x29\xc2\xca\xfb\xaf", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x02", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\xfd\xb2\x66\x98\x80\x96\x55\xad" - "\x08\x94\x54\x9c\x21\x7c\x69\xe3", - .rlen = 16, - }, { - .key = "\xd8\x2a\x91\x34\xb2\x6a\x56\x50" - "\x30\xfe\x69\xe2\x37\x7f\x98\x47" - "\xcd\xf9\x0b\x16\x0c\x64\x8f\xb6" - "\xb0\x0d\x0d\x1b\xae\x85\x87\x1f", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x14\x5e\x3d\x70\xc0\x6e\x9c\x34" - "\x5b\x5e\xcf\x0f\xe4\x8c\x21\x5c", - .rlen = 16, - }, { - .key = "\x0f\x6a\xef\xf8\xd3\xd2\xbb\x15" - "\x25\x83\xf7\x3c\x1f\x01\x28\x74" - "\xca\xc6\xbc\x35\x4d\x4a\x65\x54" - "\x90\xae\x61\xcf\x7b\xae\xbd\xcc" - "\xad\xe4\x94\xc5\x4a\x29\xae\x70", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x25\x39\xaa\xa5\xf0\x65\xc8\xdc" - "\x5d\x45\x95\x30\x8f\xff\x2f\x1b", - .rlen = 16, - }, { - .key = "\x8a\xd4\xee\x10\x2f\xbd\x81\xff" - "\xf8\x86\xce\xac\x93\xc5\xad\xc6" - "\xa0\x19\x07\xc0\x9d\xf7\xbb\xdd" - "\x52\x13\xb2\xb7\xf0\xff\x11\xd8" - "\xd6\x08\xd0\xcd\x2e\xb1\x17\x6f", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x0c\x20\x20\x63\xd6\x8b\xfc\x8f" - "\xc0\xe2\x17\xbb\xd2\x59\x6f\x26", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\xc1\x35\x2e\x53\xf0\x96\x4d\x9c" - "\x2e\x18\xe6\x99\xcd\xd3\x15\x68", - .rlen = 16, - }, { - .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" - "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" - "\xb2\xfb\x64\xce\x60\x97\x87\x8d" - "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" - "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" - "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x86\x0a\xc6\xa9\x1a\x9f\xe7\xe6" - "\x64\x3b\x33\xd6\xd5\x84\xd6\xdf", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" - "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" - "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" - "\x50\x38\x1f\x71\x49\xb6\x57\xd6" - "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" - "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" - "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" - "\xda\x10\x8e\xed\xa2\xa4\x87\xab" - "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" - "\xc9\xac\x42\x31\x95\x7c\xc9\x04" - "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" - "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" - "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" - "\x4c\x96\x12\xed\x7c\x92\x03\x01" - "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" - "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" - "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" - "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" - "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" - "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" - "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" - "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" - "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" - "\x76\x12\x73\x44\x1a\x56\xd7\x72" - "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" - "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" - "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" - "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" - "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" - "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" - "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" - "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" - "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" - "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" - "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" - "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" - "\x8d\x23\x31\x74\x84\xeb\x88\x6e" - "\xcc\xb9\xbc\x22\x83\x19\x07\x22" - "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" - "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" - "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" - "\x3c\xce\x8f\x42\x60\x71\xa7\x75" - "\x08\x40\x65\x8a\x82\xbf\xf5\x43" - "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" - "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" - "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" - "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" - "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" - "\x62\x73\x65\xfd\x46\x63\x25\x3d" - "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" - "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" - "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" - "\xc5\x68\x77\x84\x32\x2b\xcc\x85" - "\x74\x96\xf0\x12\x77\x61\xb9\xeb" - "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" - "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" - "\xda\x39\x87\x45\xc0\x2b\xbb\x01" - "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" - "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" - "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" - "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" - "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" - "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" - "\x21\xc4\xc2\x75\x67\x89\x37\x0a", - .ilen = 512, - .result = "\xe3\x5a\x38\x0f\x4d\x92\x3a\x74" - "\x15\xb1\x50\x8c\x9a\xd8\x99\x1d" - "\x82\xec\xf1\x5f\x03\x6d\x02\x58" - "\x90\x67\xfc\xdd\x8d\xe1\x38\x08" - "\x7b\xc9\x9b\x4b\x04\x09\x50\x15" - "\xce\xab\xda\x33\x30\x20\x12\xfa" - "\x83\xc4\xa6\x9a\x2e\x7d\x90\xd9" - "\xa6\xa6\x67\x43\xb4\xa7\xa8\x5c" - "\xbb\x6a\x49\x2b\x8b\xf8\xd0\x22" - "\xe5\x9e\xba\xe8\x8c\x67\xb8\x5b" - "\x60\xbc\xf5\xa4\x95\x4e\x66\xe5" - "\x6d\x8e\xa9\xf6\x65\x2e\x04\xf5" - "\xba\xb5\xdb\x88\xc2\xf6\x7a\x4b" - "\x89\x58\x7c\x9a\xae\x26\xe8\xb7" - "\xb7\x28\xcc\xd6\xcc\xa5\x98\x4d" - "\xb9\x91\xcb\xb4\xe4\x8b\x96\x47" - "\x5f\x03\x8b\xdd\x94\xd1\xee\x12" - "\xa7\x83\x80\xf2\xc1\x15\x74\x4f" - "\x49\xf9\xb0\x7e\x6f\xdc\x73\x2f" - "\xe2\xcf\xe0\x1b\x34\xa5\xa0\x52" - "\xfb\x3c\x5d\x85\x91\xe6\x6d\x98" - "\x04\xd6\xdd\x4c\x00\x64\xd9\x54" - "\x5c\x3c\x08\x1d\x4c\x06\x9f\xb8" - "\x1c\x4d\x8d\xdc\xa4\x3c\xb9\x3b" - "\x9e\x85\xce\xc3\xa8\x4a\x0c\xd9" - "\x04\xc3\x6f\x17\x66\xa9\x1f\x59" - "\xd9\xe2\x19\x36\xa3\x88\xb8\x0b" - "\x0f\x4a\x4d\xf8\xc8\x6f\xd5\x43" - "\xeb\xa0\xab\x1f\x61\xc0\x06\xeb" - "\x93\xb7\xb8\x6f\x0d\xbd\x07\x49" - "\xb3\xac\x5d\xcf\x31\xa0\x27\x26" - "\x21\xbe\x94\x2e\x19\xea\xf4\xee" - "\xb5\x13\x89\xf7\x94\x0b\xef\x59" - "\x44\xc5\x78\x8b\x3c\x3b\x71\x20" - "\xf9\x35\x0c\x70\x74\xdc\x5b\xc2" - "\xb4\x11\x0e\x2c\x61\xa1\x52\x46" - "\x18\x11\x16\xc6\x86\x44\xa7\xaf" - "\xd5\x0c\x7d\xa6\x9e\x25\x2d\x1b" - "\x9a\x8f\x0f\xf8\x6a\x61\xa0\xea" - "\x3f\x0e\x90\xd6\x8f\x83\x30\x64" - "\xb5\x51\x2d\x08\x3c\xcd\x99\x36" - "\x96\xd4\xb1\xb5\x48\x30\xca\x48" - "\xf7\x11\xa8\xf5\x97\x8a\x6a\x6d" - "\x12\x33\x2f\xc0\xe8\xda\xec\x8a" - "\xe1\x88\x72\x63\xde\x20\xa3\xe1" - "\x8e\xac\x84\x37\x35\xf5\xf7\x3f" - "\x00\x02\x0e\xe4\xc1\x53\x68\x3f" - "\xaa\xd5\xac\x52\x3d\x20\x2f\x4d" - "\x7c\x83\xd0\xbd\xaa\x97\x35\x36" - "\x98\x88\x59\x5d\xe7\x24\xe3\x90" - "\x9d\x30\x47\xa7\xc3\x60\x35\xf4" - "\xd5\xdb\x0e\x4d\x44\xc1\x81\x8b" - "\xfd\xbd\xc3\x2b\xba\x68\xfe\x8d" - "\x49\x5a\x3c\x8a\xa3\x01\xae\x25" - "\x42\xab\xd2\x87\x1b\x35\xd6\xd2" - "\xd7\x70\x1c\x1f\x72\xd1\xe1\x39" - "\x1c\x58\xa2\xb4\xd0\x78\x55\x72" - "\x76\x59\xea\xd9\xd7\x6e\x63\x8b" - "\xcc\x9b\xa7\x74\x89\xfc\xa3\x68" - "\x86\x28\xd1\xbb\x54\x8d\x66\xad" - "\x2a\x92\xf9\x4e\x04\x3d\xae\xfd" - "\x1b\x2b\x7f\xc3\x2f\x1a\x78\x0a" - "\x5c\xc6\x84\xfe\x7c\xcb\x26\xfd" - "\xd9\x51\x0f\xd7\x94\x2f\xc5\xa7", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec serpent_lrw_dec_tv_template[] = { - /* Generated from AES-LRW test vectors */ - /* same as enc vectors with input and result reversed */ - { - .key = "\x45\x62\xac\x25\xf8\x28\x17\x6d" - "\x4c\x26\x84\x14\xb5\x68\x01\x85" - "\x25\x8e\x2a\x05\xe7\x3e\x9d\x03" - "\xee\x5a\x83\x0c\xcc\x09\x4c\x87", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x6f\xbf\xd4\xa4\x5d\x71\x16\x79" - "\x63\x9c\xa6\x8e\x40\xbe\x0d\x8a", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\x59\x70\x47\x14\xf5\x57\x47\x8c" - "\xd7\x79\xe8\x0f\x54\x88\x79\x44" - "\x0d\x48\xf0\xb7\xb1\x5a\x53\xea" - "\x1c\xaa\x6b\x29\xc2\xca\xfb\xaf", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x02", - .input = "\xfd\xb2\x66\x98\x80\x96\x55\xad" - "\x08\x94\x54\x9c\x21\x7c\x69\xe3", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\xd8\x2a\x91\x34\xb2\x6a\x56\x50" - "\x30\xfe\x69\xe2\x37\x7f\x98\x47" - "\xcd\xf9\x0b\x16\x0c\x64\x8f\xb6" - "\xb0\x0d\x0d\x1b\xae\x85\x87\x1f", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x14\x5e\x3d\x70\xc0\x6e\x9c\x34" - "\x5b\x5e\xcf\x0f\xe4\x8c\x21\x5c", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\x0f\x6a\xef\xf8\xd3\xd2\xbb\x15" - "\x25\x83\xf7\x3c\x1f\x01\x28\x74" - "\xca\xc6\xbc\x35\x4d\x4a\x65\x54" - "\x90\xae\x61\xcf\x7b\xae\xbd\xcc" - "\xad\xe4\x94\xc5\x4a\x29\xae\x70", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x25\x39\xaa\xa5\xf0\x65\xc8\xdc" - "\x5d\x45\x95\x30\x8f\xff\x2f\x1b", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\x8a\xd4\xee\x10\x2f\xbd\x81\xff" - "\xf8\x86\xce\xac\x93\xc5\xad\xc6" - "\xa0\x19\x07\xc0\x9d\xf7\xbb\xdd" - "\x52\x13\xb2\xb7\xf0\xff\x11\xd8" - "\xd6\x08\xd0\xcd\x2e\xb1\x17\x6f", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x0c\x20\x20\x63\xd6\x8b\xfc\x8f" - "\xc0\xe2\x17\xbb\xd2\x59\x6f\x26", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\xc1\x35\x2e\x53\xf0\x96\x4d\x9c" - "\x2e\x18\xe6\x99\xcd\xd3\x15\x68", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" - "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" - "\xb2\xfb\x64\xce\x60\x97\x87\x8d" - "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" - "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" - "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x86\x0a\xc6\xa9\x1a\x9f\xe7\xe6" - "\x64\x3b\x33\xd6\xd5\x84\xd6\xdf", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\xe3\x5a\x38\x0f\x4d\x92\x3a\x74" - "\x15\xb1\x50\x8c\x9a\xd8\x99\x1d" - "\x82\xec\xf1\x5f\x03\x6d\x02\x58" - "\x90\x67\xfc\xdd\x8d\xe1\x38\x08" - "\x7b\xc9\x9b\x4b\x04\x09\x50\x15" - "\xce\xab\xda\x33\x30\x20\x12\xfa" - "\x83\xc4\xa6\x9a\x2e\x7d\x90\xd9" - "\xa6\xa6\x67\x43\xb4\xa7\xa8\x5c" - "\xbb\x6a\x49\x2b\x8b\xf8\xd0\x22" - "\xe5\x9e\xba\xe8\x8c\x67\xb8\x5b" - "\x60\xbc\xf5\xa4\x95\x4e\x66\xe5" - "\x6d\x8e\xa9\xf6\x65\x2e\x04\xf5" - "\xba\xb5\xdb\x88\xc2\xf6\x7a\x4b" - "\x89\x58\x7c\x9a\xae\x26\xe8\xb7" - "\xb7\x28\xcc\xd6\xcc\xa5\x98\x4d" - "\xb9\x91\xcb\xb4\xe4\x8b\x96\x47" - "\x5f\x03\x8b\xdd\x94\xd1\xee\x12" - "\xa7\x83\x80\xf2\xc1\x15\x74\x4f" - "\x49\xf9\xb0\x7e\x6f\xdc\x73\x2f" - "\xe2\xcf\xe0\x1b\x34\xa5\xa0\x52" - "\xfb\x3c\x5d\x85\x91\xe6\x6d\x98" - "\x04\xd6\xdd\x4c\x00\x64\xd9\x54" - "\x5c\x3c\x08\x1d\x4c\x06\x9f\xb8" - "\x1c\x4d\x8d\xdc\xa4\x3c\xb9\x3b" - "\x9e\x85\xce\xc3\xa8\x4a\x0c\xd9" - "\x04\xc3\x6f\x17\x66\xa9\x1f\x59" - "\xd9\xe2\x19\x36\xa3\x88\xb8\x0b" - "\x0f\x4a\x4d\xf8\xc8\x6f\xd5\x43" - "\xeb\xa0\xab\x1f\x61\xc0\x06\xeb" - "\x93\xb7\xb8\x6f\x0d\xbd\x07\x49" - "\xb3\xac\x5d\xcf\x31\xa0\x27\x26" - "\x21\xbe\x94\x2e\x19\xea\xf4\xee" - "\xb5\x13\x89\xf7\x94\x0b\xef\x59" - "\x44\xc5\x78\x8b\x3c\x3b\x71\x20" - "\xf9\x35\x0c\x70\x74\xdc\x5b\xc2" - "\xb4\x11\x0e\x2c\x61\xa1\x52\x46" - "\x18\x11\x16\xc6\x86\x44\xa7\xaf" - "\xd5\x0c\x7d\xa6\x9e\x25\x2d\x1b" - "\x9a\x8f\x0f\xf8\x6a\x61\xa0\xea" - "\x3f\x0e\x90\xd6\x8f\x83\x30\x64" - "\xb5\x51\x2d\x08\x3c\xcd\x99\x36" - "\x96\xd4\xb1\xb5\x48\x30\xca\x48" - "\xf7\x11\xa8\xf5\x97\x8a\x6a\x6d" - "\x12\x33\x2f\xc0\xe8\xda\xec\x8a" - "\xe1\x88\x72\x63\xde\x20\xa3\xe1" - "\x8e\xac\x84\x37\x35\xf5\xf7\x3f" - "\x00\x02\x0e\xe4\xc1\x53\x68\x3f" - "\xaa\xd5\xac\x52\x3d\x20\x2f\x4d" - "\x7c\x83\xd0\xbd\xaa\x97\x35\x36" - "\x98\x88\x59\x5d\xe7\x24\xe3\x90" - "\x9d\x30\x47\xa7\xc3\x60\x35\xf4" - "\xd5\xdb\x0e\x4d\x44\xc1\x81\x8b" - "\xfd\xbd\xc3\x2b\xba\x68\xfe\x8d" - "\x49\x5a\x3c\x8a\xa3\x01\xae\x25" - "\x42\xab\xd2\x87\x1b\x35\xd6\xd2" - "\xd7\x70\x1c\x1f\x72\xd1\xe1\x39" - "\x1c\x58\xa2\xb4\xd0\x78\x55\x72" - "\x76\x59\xea\xd9\xd7\x6e\x63\x8b" - "\xcc\x9b\xa7\x74\x89\xfc\xa3\x68" - "\x86\x28\xd1\xbb\x54\x8d\x66\xad" - "\x2a\x92\xf9\x4e\x04\x3d\xae\xfd" - "\x1b\x2b\x7f\xc3\x2f\x1a\x78\x0a" - "\x5c\xc6\x84\xfe\x7c\xcb\x26\xfd" - "\xd9\x51\x0f\xd7\x94\x2f\xc5\xa7", - .ilen = 512, - .result = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" - "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" - "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" - "\x50\x38\x1f\x71\x49\xb6\x57\xd6" - "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" - "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" - "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" - "\xda\x10\x8e\xed\xa2\xa4\x87\xab" - "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" - "\xc9\xac\x42\x31\x95\x7c\xc9\x04" - "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" - "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" - "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" - "\x4c\x96\x12\xed\x7c\x92\x03\x01" - "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" - "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" - "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" - "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" - "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" - "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" - "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" - "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" - "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" - "\x76\x12\x73\x44\x1a\x56\xd7\x72" - "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" - "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" - "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" - "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" - "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" - "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" - "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" - "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" - "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" - "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" - "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" - "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" - "\x8d\x23\x31\x74\x84\xeb\x88\x6e" - "\xcc\xb9\xbc\x22\x83\x19\x07\x22" - "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" - "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" - "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" - "\x3c\xce\x8f\x42\x60\x71\xa7\x75" - "\x08\x40\x65\x8a\x82\xbf\xf5\x43" - "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" - "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" - "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" - "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" - "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" - "\x62\x73\x65\xfd\x46\x63\x25\x3d" - "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" - "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" - "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" - "\xc5\x68\x77\x84\x32\x2b\xcc\x85" - "\x74\x96\xf0\x12\x77\x61\xb9\xeb" - "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" - "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" - "\xda\x39\x87\x45\xc0\x2b\xbb\x01" - "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" - "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" - "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" - "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" - "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" - "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" - "\x21\xc4\xc2\x75\x67\x89\x37\x0a", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec serpent_xts_enc_tv_template[] = { - /* Generated from AES-XTS test vectors */ - { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 32, - .result = "\xe1\x08\xb8\x1d\x2c\xf5\x33\x64" - "\xc8\x12\x04\xc7\xb3\x70\xe8\xc4" - "\x6a\x31\xc5\xf3\x00\xca\xb9\x16" - "\xde\xe2\x77\x66\xf7\xfe\x62\x08", - .rlen = 32, - }, { - .key = "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\x1a\x0a\x09\x5f\xcd\x07\x07\x98" - "\x41\x86\x12\xaf\xb3\xd7\x68\x13" - "\xed\x81\xcd\x06\x87\x43\x1a\xbb" - "\x13\x3d\xd6\x1e\x2b\xe1\x77\xbe", - .rlen = 32, - }, { - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\xf9\x9b\x28\xb8\x5c\xaf\x8c\x61" - "\xb6\x1c\x81\x8f\x2c\x87\x60\x89" - "\x0d\x8d\x7a\xe8\x60\x48\xcc\x86" - "\xc1\x68\x45\xaa\x00\xe9\x24\xc5", - .rlen = 32, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\xfe\x47\x4a\xc8\x60\x7e\xb4\x8b" - "\x0d\x10\xf4\xb0\x0d\xba\xf8\x53" - "\x65\x6e\x38\x4b\xdb\xaa\xb1\x9e" - "\x28\xca\xb0\x22\xb3\x85\x75\xf4" - "\x00\x5c\x75\x14\x06\xd6\x25\x82" - "\xe6\xcb\x08\xf7\x29\x90\x23\x8e" - "\xa4\x68\x57\xe4\xf0\xd8\x32\xf3" - "\x80\x51\x67\xb5\x0b\x85\x69\xe8" - "\x19\xfe\xc4\xc7\x3e\xea\x90\xd3" - "\x8f\xa3\xf2\x0a\xac\x17\x4b\xa0" - "\x63\x5a\x16\x0f\xf0\xce\x66\x1f" - "\x2c\x21\x07\xf1\xa4\x03\xa3\x44" - "\x41\x61\x87\x5d\x6b\xb3\xef\xd4" - "\xfc\xaa\x32\x7e\x55\x58\x04\x41" - "\xc9\x07\x33\xc6\xa2\x68\xd6\x5a" - "\x55\x79\x4b\x6f\xcf\x89\xb9\x19" - "\xe5\x54\x13\x15\xb2\x1a\xfa\x15" - "\xc2\xf0\x06\x59\xfa\xa0\x25\x05" - "\x58\xfa\x43\x91\x16\x85\x40\xbb" - "\x0d\x34\x4d\xc5\x1e\x20\xd5\x08" - "\xcd\x22\x22\x41\x11\x9f\x6c\x7c" - "\x8d\x57\xc9\xba\x57\xe8\x2c\xf7" - "\xa0\x42\xa8\xde\xfc\xa3\xca\x98" - "\x4b\x43\xb1\xce\x4b\xbf\x01\x67" - "\x6e\x29\x60\xbd\x10\x14\x84\x82" - "\x83\x82\x0c\x63\x73\x92\x02\x7c" - "\x55\x37\x20\x80\x17\x51\xc8\xbc" - "\x46\x02\xcb\x38\x07\x6d\xe2\x85" - "\xaa\x29\xaf\x24\x58\x0d\xf0\x75" - "\x08\x0a\xa5\x34\x25\x16\xf3\x74" - "\xa7\x0b\x97\xbe\xc1\xa9\xdc\x29" - "\x1a\x0a\x56\xc1\x1a\x91\x97\x8c" - "\x0b\xc7\x16\xed\x5a\x22\xa6\x2e" - "\x8c\x2b\x4f\x54\x76\x47\x53\x8e" - "\xe8\x00\xec\x92\xb9\x55\xe6\xa2" - "\xf3\xe2\x4f\x6a\x66\x60\xd0\x87" - "\xe6\xd1\xcc\xe3\x6a\xc5\x2d\x21" - "\xcc\x9d\x6a\xb6\x75\xaa\xe2\x19" - "\x21\x9f\xa1\x5e\x4c\xfd\x72\xf9" - "\x94\x4e\x63\xc7\xae\xfc\xed\x47" - "\xe2\xfe\x7a\x63\x77\xfe\x97\x82" - "\xb1\x10\x6e\x36\x1d\xe1\xc4\x80" - "\xec\x69\x41\xec\xa7\x8a\xe0\x2f" - "\xe3\x49\x26\xa2\x41\xb2\x08\x0f" - "\x28\xb4\xa7\x39\xa1\x99\x2d\x1e" - "\x43\x42\x35\xd0\xcf\xec\x77\x67" - "\xb2\x3b\x9e\x1c\x35\xde\x4f\x5e" - "\x73\x3f\x5d\x6f\x07\x4b\x2e\x50" - "\xab\x6c\x6b\xff\xea\x00\x67\xaa" - "\x0e\x82\x32\xdd\x3d\xb5\xe5\x76" - "\x2b\x77\x3f\xbe\x12\x75\xfb\x92" - "\xc6\x89\x67\x4d\xca\xf7\xd4\x50" - "\xc0\x74\x47\xcc\xd9\x0a\xd4\xc6" - "\x3b\x17\x2e\xe3\x35\xbb\x53\xb5" - "\x86\xad\x51\xcc\xd5\x96\xb8\xdc" - "\x03\x57\xe6\x98\x52\x2f\x61\x62" - "\xc4\x5c\x9c\x36\x71\x07\xfb\x94" - "\xe3\x02\xc4\x2b\x08\x75\xc7\x35" - "\xfb\x2e\x88\x7b\xbb\x67\x00\xe1" - "\xc9\xdd\x99\xb2\x13\x53\x1a\x4e" - "\x76\x87\x19\x04\x1a\x2f\x38\x3e" - "\xef\x91\x64\x1d\x18\x07\x4e\x31" - "\x88\x21\x7c\xb0\xa5\x12\x4c\x3c" - "\xb0\x20\xbd\xda\xdf\xf9\x7c\xdd", - .rlen = 512, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\x2b\xc9\xb4\x6b\x10\x94\xa9\x32" - "\xaa\xb0\x20\xc6\x44\x3d\x74\x1f" - "\x75\x01\xa7\xf6\xf5\xf7\x62\x1b" - "\x80\x1b\x82\xcb\x01\x59\x91\x7f" - "\x80\x3a\x98\xf0\xd2\xca\xc4\xc3" - "\x34\xfd\xe6\x11\xf9\x33\x45\x12" - "\x48\xc5\x8c\x25\xf1\xc5\xc5\x23" - "\xd3\x44\xb4\x73\xd5\x04\xc0\xb7" - "\xca\x2f\xf5\xcd\xc5\xb4\xdd\xb0" - "\xf4\x60\xe8\xfb\xc6\x9c\xc5\x78" - "\xcd\xec\x7d\xdc\x19\x9c\x72\x64" - "\x63\x0b\x38\x2e\x76\xdd\x2d\x36" - "\x49\xb0\x1d\xea\x78\x9e\x00\xca" - "\x20\xcc\x1b\x1e\x98\x74\xab\xed" - "\x79\xf7\xd0\x6c\xd8\x93\x80\x29" - "\xac\xa5\x5e\x34\xa9\xab\xa0\x55" - "\x9a\xea\xaa\x95\x4d\x7b\xfe\x46" - "\x26\x8a\xfd\x88\xa2\xa8\xa6\xae" - "\x25\x42\x17\xbf\x76\x8f\x1c\x3d" - "\xec\x9a\xda\x64\x96\xb5\x61\xff" - "\x99\xeb\x12\x96\x85\x82\x9d\xd5" - "\x81\x85\x14\xa8\x59\xac\x8c\x94" - "\xbb\x3b\x85\x2b\xdf\xb3\x0c\xba" - "\x82\xc6\x4d\xca\x86\xea\x53\x28" - "\x4c\xe0\x4e\x31\xe3\x73\x2f\x79" - "\x9d\x42\xe1\x03\xe3\x8b\xc4\xff" - "\x05\xca\x81\x7b\xda\xa2\xde\x63" - "\x3a\x10\xbe\xc2\xac\x32\xc4\x05" - "\x47\x7e\xef\x67\xe2\x5f\x5b\xae" - "\xed\xf1\x70\x34\x16\x9a\x07\x7b" - "\xf2\x25\x2b\xb0\xf8\x3c\x15\x9a" - "\xa6\x59\x55\x5f\xc1\xf4\x1e\xcd" - "\x93\x1f\x06\xba\xd4\x9a\x22\x69" - "\xfa\x8e\x95\x0d\xf3\x23\x59\x2c" - "\xfe\x00\xba\xf0\x0e\xbc\x6d\xd6" - "\x62\xf0\x7a\x0e\x83\x3e\xdb\x32" - "\xfd\x43\x7d\xda\x42\x51\x87\x43" - "\x9d\xf9\xef\xf4\x30\x97\xf8\x09" - "\x88\xfc\x3f\x93\x70\xc1\x4a\xec" - "\x27\x5f\x11\xac\x71\xc7\x48\x46" - "\x2f\xf9\xdf\x8d\x9f\xf7\x2e\x56" - "\x0d\x4e\xb0\x32\x76\xce\x86\x81" - "\xcd\xdf\xe4\x00\xbf\xfd\x5f\x24" - "\xaf\xf7\x9a\xde\xff\x18\xac\x14" - "\x90\xc5\x01\x39\x34\x0f\x24\xf3" - "\x13\x2f\x5e\x4f\x30\x9a\x36\x40" - "\xec\xea\xbc\xcd\x9e\x0e\x5b\x23" - "\x50\x88\x97\x40\x69\xb1\x37\xf5" - "\xc3\x15\xf9\x3f\xb7\x79\x64\xe8" - "\x7b\x10\x20\xb9\x2b\x46\x83\x5b" - "\xd8\x39\xfc\xe4\xfa\x88\x52\xf2" - "\x72\xb0\x97\x4e\x89\xb3\x48\x00" - "\xc1\x16\x73\x50\x77\xba\xa6\x65" - "\x20\x2d\xb0\x02\x27\x89\xda\x99" - "\x45\xfb\xe9\xd3\x1d\x39\x2f\xd6" - "\x2a\xda\x09\x12\x11\xaf\xe6\x57" - "\x01\x04\x8a\xff\x86\x8b\xac\xf8" - "\xee\xe4\x1c\x98\x5b\xcf\x6b\x76" - "\xa3\x0e\x33\x74\x40\x18\x39\x72" - "\x66\x50\x31\xfd\x70\xdf\xe8\x51" - "\x96\x21\x36\xb2\x9b\xfa\x85\xd1" - "\x30\x05\xc8\x92\x98\x80\xff\x7a" - "\xaf\x43\x0b\xc5\x20\x41\x92\x20" - "\xd4\xa0\x91\x98\x11\x5f\x4d\xb1", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec serpent_xts_dec_tv_template[] = { - /* Generated from AES-XTS test vectors */ - /* same as enc vectors with input and result reversed */ - { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xe1\x08\xb8\x1d\x2c\xf5\x33\x64" - "\xc8\x12\x04\xc7\xb3\x70\xe8\xc4" - "\x6a\x31\xc5\xf3\x00\xca\xb9\x16" - "\xde\xe2\x77\x66\xf7\xfe\x62\x08", - .ilen = 32, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 32, - }, { - .key = "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x1a\x0a\x09\x5f\xcd\x07\x07\x98" - "\x41\x86\x12\xaf\xb3\xd7\x68\x13" - "\xed\x81\xcd\x06\x87\x43\x1a\xbb" - "\x13\x3d\xd6\x1e\x2b\xe1\x77\xbe", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xf9\x9b\x28\xb8\x5c\xaf\x8c\x61" - "\xb6\x1c\x81\x8f\x2c\x87\x60\x89" - "\x0d\x8d\x7a\xe8\x60\x48\xcc\x86" - "\xc1\x68\x45\xaa\x00\xe9\x24\xc5", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xfe\x47\x4a\xc8\x60\x7e\xb4\x8b" - "\x0d\x10\xf4\xb0\x0d\xba\xf8\x53" - "\x65\x6e\x38\x4b\xdb\xaa\xb1\x9e" - "\x28\xca\xb0\x22\xb3\x85\x75\xf4" - "\x00\x5c\x75\x14\x06\xd6\x25\x82" - "\xe6\xcb\x08\xf7\x29\x90\x23\x8e" - "\xa4\x68\x57\xe4\xf0\xd8\x32\xf3" - "\x80\x51\x67\xb5\x0b\x85\x69\xe8" - "\x19\xfe\xc4\xc7\x3e\xea\x90\xd3" - "\x8f\xa3\xf2\x0a\xac\x17\x4b\xa0" - "\x63\x5a\x16\x0f\xf0\xce\x66\x1f" - "\x2c\x21\x07\xf1\xa4\x03\xa3\x44" - "\x41\x61\x87\x5d\x6b\xb3\xef\xd4" - "\xfc\xaa\x32\x7e\x55\x58\x04\x41" - "\xc9\x07\x33\xc6\xa2\x68\xd6\x5a" - "\x55\x79\x4b\x6f\xcf\x89\xb9\x19" - "\xe5\x54\x13\x15\xb2\x1a\xfa\x15" - "\xc2\xf0\x06\x59\xfa\xa0\x25\x05" - "\x58\xfa\x43\x91\x16\x85\x40\xbb" - "\x0d\x34\x4d\xc5\x1e\x20\xd5\x08" - "\xcd\x22\x22\x41\x11\x9f\x6c\x7c" - "\x8d\x57\xc9\xba\x57\xe8\x2c\xf7" - "\xa0\x42\xa8\xde\xfc\xa3\xca\x98" - "\x4b\x43\xb1\xce\x4b\xbf\x01\x67" - "\x6e\x29\x60\xbd\x10\x14\x84\x82" - "\x83\x82\x0c\x63\x73\x92\x02\x7c" - "\x55\x37\x20\x80\x17\x51\xc8\xbc" - "\x46\x02\xcb\x38\x07\x6d\xe2\x85" - "\xaa\x29\xaf\x24\x58\x0d\xf0\x75" - "\x08\x0a\xa5\x34\x25\x16\xf3\x74" - "\xa7\x0b\x97\xbe\xc1\xa9\xdc\x29" - "\x1a\x0a\x56\xc1\x1a\x91\x97\x8c" - "\x0b\xc7\x16\xed\x5a\x22\xa6\x2e" - "\x8c\x2b\x4f\x54\x76\x47\x53\x8e" - "\xe8\x00\xec\x92\xb9\x55\xe6\xa2" - "\xf3\xe2\x4f\x6a\x66\x60\xd0\x87" - "\xe6\xd1\xcc\xe3\x6a\xc5\x2d\x21" - "\xcc\x9d\x6a\xb6\x75\xaa\xe2\x19" - "\x21\x9f\xa1\x5e\x4c\xfd\x72\xf9" - "\x94\x4e\x63\xc7\xae\xfc\xed\x47" - "\xe2\xfe\x7a\x63\x77\xfe\x97\x82" - "\xb1\x10\x6e\x36\x1d\xe1\xc4\x80" - "\xec\x69\x41\xec\xa7\x8a\xe0\x2f" - "\xe3\x49\x26\xa2\x41\xb2\x08\x0f" - "\x28\xb4\xa7\x39\xa1\x99\x2d\x1e" - "\x43\x42\x35\xd0\xcf\xec\x77\x67" - "\xb2\x3b\x9e\x1c\x35\xde\x4f\x5e" - "\x73\x3f\x5d\x6f\x07\x4b\x2e\x50" - "\xab\x6c\x6b\xff\xea\x00\x67\xaa" - "\x0e\x82\x32\xdd\x3d\xb5\xe5\x76" - "\x2b\x77\x3f\xbe\x12\x75\xfb\x92" - "\xc6\x89\x67\x4d\xca\xf7\xd4\x50" - "\xc0\x74\x47\xcc\xd9\x0a\xd4\xc6" - "\x3b\x17\x2e\xe3\x35\xbb\x53\xb5" - "\x86\xad\x51\xcc\xd5\x96\xb8\xdc" - "\x03\x57\xe6\x98\x52\x2f\x61\x62" - "\xc4\x5c\x9c\x36\x71\x07\xfb\x94" - "\xe3\x02\xc4\x2b\x08\x75\xc7\x35" - "\xfb\x2e\x88\x7b\xbb\x67\x00\xe1" - "\xc9\xdd\x99\xb2\x13\x53\x1a\x4e" - "\x76\x87\x19\x04\x1a\x2f\x38\x3e" - "\xef\x91\x64\x1d\x18\x07\x4e\x31" - "\x88\x21\x7c\xb0\xa5\x12\x4c\x3c" - "\xb0\x20\xbd\xda\xdf\xf9\x7c\xdd", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x2b\xc9\xb4\x6b\x10\x94\xa9\x32" - "\xaa\xb0\x20\xc6\x44\x3d\x74\x1f" - "\x75\x01\xa7\xf6\xf5\xf7\x62\x1b" - "\x80\x1b\x82\xcb\x01\x59\x91\x7f" - "\x80\x3a\x98\xf0\xd2\xca\xc4\xc3" - "\x34\xfd\xe6\x11\xf9\x33\x45\x12" - "\x48\xc5\x8c\x25\xf1\xc5\xc5\x23" - "\xd3\x44\xb4\x73\xd5\x04\xc0\xb7" - "\xca\x2f\xf5\xcd\xc5\xb4\xdd\xb0" - "\xf4\x60\xe8\xfb\xc6\x9c\xc5\x78" - "\xcd\xec\x7d\xdc\x19\x9c\x72\x64" - "\x63\x0b\x38\x2e\x76\xdd\x2d\x36" - "\x49\xb0\x1d\xea\x78\x9e\x00\xca" - "\x20\xcc\x1b\x1e\x98\x74\xab\xed" - "\x79\xf7\xd0\x6c\xd8\x93\x80\x29" - "\xac\xa5\x5e\x34\xa9\xab\xa0\x55" - "\x9a\xea\xaa\x95\x4d\x7b\xfe\x46" - "\x26\x8a\xfd\x88\xa2\xa8\xa6\xae" - "\x25\x42\x17\xbf\x76\x8f\x1c\x3d" - "\xec\x9a\xda\x64\x96\xb5\x61\xff" - "\x99\xeb\x12\x96\x85\x82\x9d\xd5" - "\x81\x85\x14\xa8\x59\xac\x8c\x94" - "\xbb\x3b\x85\x2b\xdf\xb3\x0c\xba" - "\x82\xc6\x4d\xca\x86\xea\x53\x28" - "\x4c\xe0\x4e\x31\xe3\x73\x2f\x79" - "\x9d\x42\xe1\x03\xe3\x8b\xc4\xff" - "\x05\xca\x81\x7b\xda\xa2\xde\x63" - "\x3a\x10\xbe\xc2\xac\x32\xc4\x05" - "\x47\x7e\xef\x67\xe2\x5f\x5b\xae" - "\xed\xf1\x70\x34\x16\x9a\x07\x7b" - "\xf2\x25\x2b\xb0\xf8\x3c\x15\x9a" - "\xa6\x59\x55\x5f\xc1\xf4\x1e\xcd" - "\x93\x1f\x06\xba\xd4\x9a\x22\x69" - "\xfa\x8e\x95\x0d\xf3\x23\x59\x2c" - "\xfe\x00\xba\xf0\x0e\xbc\x6d\xd6" - "\x62\xf0\x7a\x0e\x83\x3e\xdb\x32" - "\xfd\x43\x7d\xda\x42\x51\x87\x43" - "\x9d\xf9\xef\xf4\x30\x97\xf8\x09" - "\x88\xfc\x3f\x93\x70\xc1\x4a\xec" - "\x27\x5f\x11\xac\x71\xc7\x48\x46" - "\x2f\xf9\xdf\x8d\x9f\xf7\x2e\x56" - "\x0d\x4e\xb0\x32\x76\xce\x86\x81" - "\xcd\xdf\xe4\x00\xbf\xfd\x5f\x24" - "\xaf\xf7\x9a\xde\xff\x18\xac\x14" - "\x90\xc5\x01\x39\x34\x0f\x24\xf3" - "\x13\x2f\x5e\x4f\x30\x9a\x36\x40" - "\xec\xea\xbc\xcd\x9e\x0e\x5b\x23" - "\x50\x88\x97\x40\x69\xb1\x37\xf5" - "\xc3\x15\xf9\x3f\xb7\x79\x64\xe8" - "\x7b\x10\x20\xb9\x2b\x46\x83\x5b" - "\xd8\x39\xfc\xe4\xfa\x88\x52\xf2" - "\x72\xb0\x97\x4e\x89\xb3\x48\x00" - "\xc1\x16\x73\x50\x77\xba\xa6\x65" - "\x20\x2d\xb0\x02\x27\x89\xda\x99" - "\x45\xfb\xe9\xd3\x1d\x39\x2f\xd6" - "\x2a\xda\x09\x12\x11\xaf\xe6\x57" - "\x01\x04\x8a\xff\x86\x8b\xac\xf8" - "\xee\xe4\x1c\x98\x5b\xcf\x6b\x76" - "\xa3\x0e\x33\x74\x40\x18\x39\x72" - "\x66\x50\x31\xfd\x70\xdf\xe8\x51" - "\x96\x21\x36\xb2\x9b\xfa\x85\xd1" - "\x30\x05\xc8\x92\x98\x80\xff\x7a" - "\xaf\x43\x0b\xc5\x20\x41\x92\x20" - "\xd4\xa0\x91\x98\x11\x5f\x4d\xb1", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -/* - * SM4 test vector taken from the draft RFC - * https://tools.ietf.org/html/draft-crypto-sm4-00#ref-GBT.32907-2016 - */ - -static const struct cipher_testvec sm4_enc_tv_template[] = { - { /* SM4 Appendix A: Example Calculations. Example 1. */ - .key = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" - "\xFE\xDC\xBA\x98\x76\x54\x32\x10", - .klen = 16, - .input = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" - "\xFE\xDC\xBA\x98\x76\x54\x32\x10", - .ilen = 16, - .result = "\x68\x1E\xDF\x34\xD2\x06\x96\x5E" - "\x86\xB3\xE9\x4F\x53\x6E\x42\x46", - .rlen = 16, - }, { /* - * SM4 Appendix A: Example Calculations. - * Last 10 iterations of Example 2. - */ - .key = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" - "\xFE\xDC\xBA\x98\x76\x54\x32\x10", - .klen = 16, - .input = "\x99\x4a\xc3\xe7\xc3\x57\x89\x6a" - "\x81\xfc\xa8\xe\x38\x3e\xef\x80" - "\xb1\x98\xf2\xde\x3f\x4b\xae\xd1" - "\xf0\xf1\x30\x4c\x1\x27\x5a\x8f" - "\x45\xe1\x39\xb7\xae\xff\x1f\x27" - "\xad\x57\x15\xab\x31\x5d\xc\xef" - "\x8c\xc8\x80\xbd\x11\x98\xf3\x7b" - "\xa2\xdd\x14\x20\xf9\xe8\xbb\x82" - "\xf7\x32\xca\x4b\xa8\xf7\xb3\x4d" - "\x27\xd1\xcd\xe6\xb6\x65\x5a\x23" - "\xc2\xf3\x54\x84\x53\xe3\xb9\x20" - "\xa5\x37\x0\xbe\xe7\x7b\x48\xfb" - "\x21\x3d\x9e\x48\x1d\x9e\xf5\xbf" - "\x77\xd5\xb4\x4a\x53\x71\x94\x7a" - "\x88\xa6\x6e\x6\x93\xca\x43\xa5" - "\xc4\xf6\xcd\x53\x4b\x7b\x8e\xfe" - "\xb4\x28\x7c\x42\x29\x32\x5d\x88" - "\xed\xce\x0\x19\xe\x16\x2\x6e" - "\x87\xff\x2c\xac\xe8\xe7\xe9\xbf" - "\x31\x51\xec\x47\xc3\x51\x83\xc1", - .ilen = 160, - .result = "\xb1\x98\xf2\xde\x3f\x4b\xae\xd1" - "\xf0\xf1\x30\x4c\x1\x27\x5a\x8f" - "\x45\xe1\x39\xb7\xae\xff\x1f\x27" - "\xad\x57\x15\xab\x31\x5d\xc\xef" - "\x8c\xc8\x80\xbd\x11\x98\xf3\x7b" - "\xa2\xdd\x14\x20\xf9\xe8\xbb\x82" - "\xf7\x32\xca\x4b\xa8\xf7\xb3\x4d" - "\x27\xd1\xcd\xe6\xb6\x65\x5a\x23" - "\xc2\xf3\x54\x84\x53\xe3\xb9\x20" - "\xa5\x37\x0\xbe\xe7\x7b\x48\xfb" - "\x21\x3d\x9e\x48\x1d\x9e\xf5\xbf" - "\x77\xd5\xb4\x4a\x53\x71\x94\x7a" - "\x88\xa6\x6e\x6\x93\xca\x43\xa5" - "\xc4\xf6\xcd\x53\x4b\x7b\x8e\xfe" - "\xb4\x28\x7c\x42\x29\x32\x5d\x88" - "\xed\xce\x0\x19\xe\x16\x2\x6e" - "\x87\xff\x2c\xac\xe8\xe7\xe9\xbf" - "\x31\x51\xec\x47\xc3\x51\x83\xc1" - "\x59\x52\x98\xc7\xc6\xfd\x27\x1f" - "\x4\x2\xf8\x4\xc3\x3d\x3f\x66", - .rlen = 160 - } -}; - -static const struct cipher_testvec sm4_dec_tv_template[] = { - { /* SM4 Appendix A: Example Calculations. Example 1. */ - .key = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" - "\xFE\xDC\xBA\x98\x76\x54\x32\x10", - .klen = 16, - .input = "\x68\x1E\xDF\x34\xD2\x06\x96\x5E" - "\x86\xB3\xE9\x4F\x53\x6E\x42\x46", - .ilen = 16, - .result = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" - "\xFE\xDC\xBA\x98\x76\x54\x32\x10", - .rlen = 16, - }, { /* - * SM4 Appendix A: Example Calculations. - * Last 10 iterations of Example 2. - */ - .key = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" - "\xFE\xDC\xBA\x98\x76\x54\x32\x10", - .klen = 16, - .input = "\xb1\x98\xf2\xde\x3f\x4b\xae\xd1" - "\xf0\xf1\x30\x4c\x1\x27\x5a\x8f" - "\x45\xe1\x39\xb7\xae\xff\x1f\x27" - "\xad\x57\x15\xab\x31\x5d\xc\xef" - "\x8c\xc8\x80\xbd\x11\x98\xf3\x7b" - "\xa2\xdd\x14\x20\xf9\xe8\xbb\x82" - "\xf7\x32\xca\x4b\xa8\xf7\xb3\x4d" - "\x27\xd1\xcd\xe6\xb6\x65\x5a\x23" - "\xc2\xf3\x54\x84\x53\xe3\xb9\x20" - "\xa5\x37\x0\xbe\xe7\x7b\x48\xfb" - "\x21\x3d\x9e\x48\x1d\x9e\xf5\xbf" - "\x77\xd5\xb4\x4a\x53\x71\x94\x7a" - "\x88\xa6\x6e\x6\x93\xca\x43\xa5" - "\xc4\xf6\xcd\x53\x4b\x7b\x8e\xfe" - "\xb4\x28\x7c\x42\x29\x32\x5d\x88" - "\xed\xce\x0\x19\xe\x16\x2\x6e" - "\x87\xff\x2c\xac\xe8\xe7\xe9\xbf" - "\x31\x51\xec\x47\xc3\x51\x83\xc1" - "\x59\x52\x98\xc7\xc6\xfd\x27\x1f" - "\x4\x2\xf8\x4\xc3\x3d\x3f\x66", - .ilen = 160, - .result = "\x99\x4a\xc3\xe7\xc3\x57\x89\x6a" - "\x81\xfc\xa8\xe\x38\x3e\xef\x80" - "\xb1\x98\xf2\xde\x3f\x4b\xae\xd1" - "\xf0\xf1\x30\x4c\x1\x27\x5a\x8f" - "\x45\xe1\x39\xb7\xae\xff\x1f\x27" - "\xad\x57\x15\xab\x31\x5d\xc\xef" - "\x8c\xc8\x80\xbd\x11\x98\xf3\x7b" - "\xa2\xdd\x14\x20\xf9\xe8\xbb\x82" - "\xf7\x32\xca\x4b\xa8\xf7\xb3\x4d" - "\x27\xd1\xcd\xe6\xb6\x65\x5a\x23" - "\xc2\xf3\x54\x84\x53\xe3\xb9\x20" - "\xa5\x37\x0\xbe\xe7\x7b\x48\xfb" - "\x21\x3d\x9e\x48\x1d\x9e\xf5\xbf" - "\x77\xd5\xb4\x4a\x53\x71\x94\x7a" - "\x88\xa6\x6e\x6\x93\xca\x43\xa5" - "\xc4\xf6\xcd\x53\x4b\x7b\x8e\xfe" - "\xb4\x28\x7c\x42\x29\x32\x5d\x88" - "\xed\xce\x0\x19\xe\x16\x2\x6e" - "\x87\xff\x2c\xac\xe8\xe7\xe9\xbf" - "\x31\x51\xec\x47\xc3\x51\x83\xc1", - .rlen = 160 - } -}; - -/* - * Speck test vectors taken from the original paper: - * "The Simon and Speck Families of Lightweight Block Ciphers" - * https://eprint.iacr.org/2013/404.pdf - * - * Note that the paper does not make byte and word order clear. But it was - * confirmed with the authors that the intended orders are little endian byte - * order and (y, x) word order. Equivalently, the printed test vectors, when - * looking at only the bytes (ignoring the whitespace that divides them into - * words), are backwards: the left-most byte is actually the one with the - * highest memory address, while the right-most byte is actually the one with - * the lowest memory address. - */ - -static const struct cipher_testvec speck128_enc_tv_template[] = { - { /* Speck128/128 */ - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .klen = 16, - .input = "\x20\x6d\x61\x64\x65\x20\x69\x74" - "\x20\x65\x71\x75\x69\x76\x61\x6c", - .ilen = 16, - .result = "\x18\x0d\x57\x5c\xdf\xfe\x60\x78" - "\x65\x32\x78\x79\x51\x98\x5d\xa6", - .rlen = 16, - }, { /* Speck128/192 */ - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17", - .klen = 24, - .input = "\x65\x6e\x74\x20\x74\x6f\x20\x43" - "\x68\x69\x65\x66\x20\x48\x61\x72", - .ilen = 16, - .result = "\x86\x18\x3c\xe0\x5d\x18\xbc\xf9" - "\x66\x55\x13\x13\x3a\xcf\xe4\x1b", - .rlen = 16, - }, { /* Speck128/256 */ - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .klen = 32, - .input = "\x70\x6f\x6f\x6e\x65\x72\x2e\x20" - "\x49\x6e\x20\x74\x68\x6f\x73\x65", - .ilen = 16, - .result = "\x43\x8f\x18\x9c\x8d\xb4\xee\x4e" - "\x3e\xf5\xc0\x05\x04\x01\x09\x41", - .rlen = 16, - }, -}; - -static const struct cipher_testvec speck128_dec_tv_template[] = { - { /* Speck128/128 */ - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .klen = 16, - .input = "\x18\x0d\x57\x5c\xdf\xfe\x60\x78" - "\x65\x32\x78\x79\x51\x98\x5d\xa6", - .ilen = 16, - .result = "\x20\x6d\x61\x64\x65\x20\x69\x74" - "\x20\x65\x71\x75\x69\x76\x61\x6c", - .rlen = 16, - }, { /* Speck128/192 */ - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17", - .klen = 24, - .input = "\x86\x18\x3c\xe0\x5d\x18\xbc\xf9" - "\x66\x55\x13\x13\x3a\xcf\xe4\x1b", - .ilen = 16, - .result = "\x65\x6e\x74\x20\x74\x6f\x20\x43" - "\x68\x69\x65\x66\x20\x48\x61\x72", - .rlen = 16, - }, { /* Speck128/256 */ - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .klen = 32, - .input = "\x43\x8f\x18\x9c\x8d\xb4\xee\x4e" - "\x3e\xf5\xc0\x05\x04\x01\x09\x41", - .ilen = 16, - .result = "\x70\x6f\x6f\x6e\x65\x72\x2e\x20" - "\x49\x6e\x20\x74\x68\x6f\x73\x65", - .rlen = 16, - }, -}; - -/* - * Speck128-XTS test vectors, taken from the AES-XTS test vectors with the - * result recomputed with Speck128 as the cipher - */ - -static const struct cipher_testvec speck128_xts_enc_tv_template[] = { - { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 32, - .result = "\xbe\xa0\xe7\x03\xd7\xfe\xab\x62" - "\x3b\x99\x4a\x64\x74\x77\xac\xed" - "\xd8\xf4\xa6\xcf\xae\xb9\x07\x42" - "\x51\xd9\xb6\x1d\xe0\x5e\xbc\x54", - .rlen = 32, - }, { - .key = "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\xfb\x53\x81\x75\x6f\x9f\x34\xad" - "\x7e\x01\xed\x7b\xcc\xda\x4e\x4a" - "\xd4\x84\xa4\x53\xd5\x88\x73\x1b" - "\xfd\xcb\xae\x0d\xf3\x04\xee\xe6", - .rlen = 32, - }, { - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\x21\x52\x84\x15\xd1\xf7\x21\x55" - "\xd9\x75\x4a\xd3\xc5\xdb\x9f\x7d" - "\xda\x63\xb2\xf1\x82\xb0\x89\x59" - "\x86\xd4\xaa\xaa\xdd\xff\x4f\x92", - .rlen = 32, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\x57\xb5\xf8\x71\x6e\x6d\xdd\x82" - "\x53\xd0\xed\x2d\x30\xc1\x20\xef" - "\x70\x67\x5e\xff\x09\x70\xbb\xc1" - "\x3a\x7b\x48\x26\xd9\x0b\xf4\x48" - "\xbe\xce\xb1\xc7\xb2\x67\xc4\xa7" - "\x76\xf8\x36\x30\xb7\xb4\x9a\xd9" - "\xf5\x9d\xd0\x7b\xc1\x06\x96\x44" - "\x19\xc5\x58\x84\x63\xb9\x12\x68" - "\x68\xc7\xaa\x18\x98\xf2\x1f\x5c" - "\x39\xa6\xd8\x32\x2b\xc3\x51\xfd" - "\x74\x79\x2e\xb4\x44\xd7\x69\xc4" - "\xfc\x29\xe6\xed\x26\x1e\xa6\x9d" - "\x1c\xbe\x00\x0e\x7f\x3a\xca\xfb" - "\x6d\x13\x65\xa0\xf9\x31\x12\xe2" - "\x26\xd1\xec\x2b\x0a\x8b\x59\x99" - "\xa7\x49\xa0\x0e\x09\x33\x85\x50" - "\xc3\x23\xca\x7a\xdd\x13\x45\x5f" - "\xde\x4c\xa7\xcb\x00\x8a\x66\x6f" - "\xa2\xb6\xb1\x2e\xe1\xa0\x18\xf6" - "\xad\xf3\xbd\xeb\xc7\xef\x55\x4f" - "\x79\x91\x8d\x36\x13\x7b\xd0\x4a" - "\x6c\x39\xfb\x53\xb8\x6f\x02\x51" - "\xa5\x20\xac\x24\x1c\x73\x59\x73" - "\x58\x61\x3a\x87\x58\xb3\x20\x56" - "\x39\x06\x2b\x4d\xd3\x20\x2b\x89" - "\x3f\xa2\xf0\x96\xeb\x7f\xa4\xcd" - "\x11\xae\xbd\xcb\x3a\xb4\xd9\x91" - "\x09\x35\x71\x50\x65\xac\x92\xe3" - "\x7b\x32\xc0\x7a\xdd\xd4\xc3\x92" - "\x6f\xeb\x79\xde\x6f\xd3\x25\xc9" - "\xcd\x63\xf5\x1e\x7a\x3b\x26\x9d" - "\x77\x04\x80\xa9\xbf\x38\xb5\xbd" - "\xb8\x05\x07\xbd\xfd\xab\x7b\xf8" - "\x2a\x26\xcc\x49\x14\x6d\x55\x01" - "\x06\x94\xd8\xb2\x2d\x53\x83\x1b" - "\x8f\xd4\xdd\x57\x12\x7e\x18\xba" - "\x8e\xe2\x4d\x80\xef\x7e\x6b\x9d" - "\x24\xa9\x60\xa4\x97\x85\x86\x2a" - "\x01\x00\x09\xf1\xcb\x4a\x24\x1c" - "\xd8\xf6\xe6\x5b\xe7\x5d\xf2\xc4" - "\x97\x1c\x10\xc6\x4d\x66\x4f\x98" - "\x87\x30\xac\xd5\xea\x73\x49\x10" - "\x80\xea\xe5\x5f\x4d\x5f\x03\x33" - "\x66\x02\x35\x3d\x60\x06\x36\x4f" - "\x14\x1c\xd8\x07\x1f\x78\xd0\xf8" - "\x4f\x6c\x62\x7c\x15\xa5\x7c\x28" - "\x7c\xcc\xeb\x1f\xd1\x07\x90\x93" - "\x7e\xc2\xa8\x3a\x80\xc0\xf5\x30" - "\xcc\x75\xcf\x16\x26\xa9\x26\x3b" - "\xe7\x68\x2f\x15\x21\x5b\xe4\x00" - "\xbd\x48\x50\xcd\x75\x70\xc4\x62" - "\xbb\x41\xfb\x89\x4a\x88\x3b\x3b" - "\x51\x66\x02\x69\x04\x97\x36\xd4" - "\x75\xae\x0b\xa3\x42\xf8\xca\x79" - "\x8f\x93\xe9\xcc\x38\xbd\xd6\xd2" - "\xf9\x70\x4e\xc3\x6a\x8e\x25\xbd" - "\xea\x15\x5a\xa0\x85\x7e\x81\x0d" - "\x03\xe7\x05\x39\xf5\x05\x26\xee" - "\xec\xaa\x1f\x3d\xc9\x98\x76\x01" - "\x2c\xf4\xfc\xa3\x88\x77\x38\xc4" - "\x50\x65\x50\x6d\x04\x1f\xdf\x5a" - "\xaa\xf2\x01\xa9\xc1\x8d\xee\xca" - "\x47\x26\xef\x39\xb8\xb4\xf2\xd1" - "\xd6\xbb\x1b\x2a\xc1\x34\x14\xcf", - .rlen = 512, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\xc5\x85\x2a\x4b\x73\xe4\xf6\xf1" - "\x7e\xf9\xf6\xe9\xa3\x73\x36\xcb" - "\xaa\xb6\x22\xb0\x24\x6e\x3d\x73" - "\x92\x99\xde\xd3\x76\xed\xcd\x63" - "\x64\x3a\x22\x57\xc1\x43\x49\xd4" - "\x79\x36\x31\x19\x62\xae\x10\x7e" - "\x7d\xcf\x7a\xe2\x6b\xce\x27\xfa" - "\xdc\x3d\xd9\x83\xd3\x42\x4c\xe0" - "\x1b\xd6\x1d\x1a\x6f\xd2\x03\x00" - "\xfc\x81\x99\x8a\x14\x62\xf5\x7e" - "\x0d\xe7\x12\xe8\x17\x9d\x0b\xec" - "\xe2\xf7\xc9\xa7\x63\xd1\x79\xb6" - "\x62\x62\x37\xfe\x0a\x4c\x4a\x37" - "\x70\xc7\x5e\x96\x5f\xbc\x8e\x9e" - "\x85\x3c\x4f\x26\x64\x85\xbc\x68" - "\xb0\xe0\x86\x5e\x26\x41\xce\x11" - "\x50\xda\x97\x14\xe9\x9e\xc7\x6d" - "\x3b\xdc\x43\xde\x2b\x27\x69\x7d" - "\xfc\xb0\x28\xbd\x8f\xb1\xc6\x31" - "\x14\x4d\xf0\x74\x37\xfd\x07\x25" - "\x96\x55\xe5\xfc\x9e\x27\x2a\x74" - "\x1b\x83\x4d\x15\x83\xac\x57\xa0" - "\xac\xa5\xd0\x38\xef\x19\x56\x53" - "\x25\x4b\xfc\xce\x04\x23\xe5\x6b" - "\xf6\xc6\x6c\x32\x0b\xb3\x12\xc5" - "\xed\x22\x34\x1c\x5d\xed\x17\x06" - "\x36\xa3\xe6\x77\xb9\x97\x46\xb8" - "\xe9\x3f\x7e\xc7\xbc\x13\x5c\xdc" - "\x6e\x3f\x04\x5e\xd1\x59\xa5\x82" - "\x35\x91\x3d\x1b\xe4\x97\x9f\x92" - "\x1c\x5e\x5f\x6f\x41\xd4\x62\xa1" - "\x8d\x39\xfc\x42\xfb\x38\x80\xb9" - "\x0a\xe3\xcc\x6a\x93\xd9\x7a\xb1" - "\xe9\x69\xaf\x0a\x6b\x75\x38\xa7" - "\xa1\xbf\xf7\xda\x95\x93\x4b\x78" - "\x19\xf5\x94\xf9\xd2\x00\x33\x37" - "\xcf\xf5\x9e\x9c\xf3\xcc\xa6\xee" - "\x42\xb2\x9e\x2c\x5f\x48\x23\x26" - "\x15\x25\x17\x03\x3d\xfe\x2c\xfc" - "\xeb\xba\xda\xe0\x00\x05\xb6\xa6" - "\x07\xb3\xe8\x36\x5b\xec\x5b\xbf" - "\xd6\x5b\x00\x74\xc6\x97\xf1\x6a" - "\x49\xa1\xc3\xfa\x10\x52\xb9\x14" - "\xad\xb7\x73\xf8\x78\x12\xc8\x59" - "\x17\x80\x4c\x57\x39\xf1\x6d\x80" - "\x25\x77\x0f\x5e\x7d\xf0\xaf\x21" - "\xec\xce\xb7\xc8\x02\x8a\xed\x53" - "\x2c\x25\x68\x2e\x1f\x85\x5e\x67" - "\xd1\x07\x7a\x3a\x89\x08\xe0\x34" - "\xdc\xdb\x26\xb4\x6b\x77\xfc\x40" - "\x31\x15\x72\xa0\xf0\x73\xd9\x3b" - "\xd5\xdb\xfe\xfc\x8f\xa9\x44\xa2" - "\x09\x9f\xc6\x33\xe5\xe2\x88\xe8" - "\xf3\xf0\x1a\xf4\xce\x12\x0f\xd6" - "\xf7\x36\xe6\xa4\xf4\x7a\x10\x58" - "\xcc\x1f\x48\x49\x65\x47\x75\xe9" - "\x28\xe1\x65\x7b\xf2\xc4\xb5\x07" - "\xf2\xec\x76\xd8\x8f\x09\xf3\x16" - "\xa1\x51\x89\x3b\xeb\x96\x42\xac" - "\x65\xe0\x67\x63\x29\xdc\xb4\x7d" - "\xf2\x41\x51\x6a\xcb\xde\x3c\xfb" - "\x66\x8d\x13\xca\xe0\x59\x2a\x00" - "\xc9\x53\x4c\xe6\x9e\xe2\x73\xd5" - "\x67\x19\xb2\xbd\x9a\x63\xd7\x5c", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - } -}; - -static const struct cipher_testvec speck128_xts_dec_tv_template[] = { - { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xbe\xa0\xe7\x03\xd7\xfe\xab\x62" - "\x3b\x99\x4a\x64\x74\x77\xac\xed" - "\xd8\xf4\xa6\xcf\xae\xb9\x07\x42" - "\x51\xd9\xb6\x1d\xe0\x5e\xbc\x54", - .ilen = 32, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 32, - }, { - .key = "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xfb\x53\x81\x75\x6f\x9f\x34\xad" - "\x7e\x01\xed\x7b\xcc\xda\x4e\x4a" - "\xd4\x84\xa4\x53\xd5\x88\x73\x1b" - "\xfd\xcb\xae\x0d\xf3\x04\xee\xe6", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x21\x52\x84\x15\xd1\xf7\x21\x55" - "\xd9\x75\x4a\xd3\xc5\xdb\x9f\x7d" - "\xda\x63\xb2\xf1\x82\xb0\x89\x59" - "\x86\xd4\xaa\xaa\xdd\xff\x4f\x92", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x57\xb5\xf8\x71\x6e\x6d\xdd\x82" - "\x53\xd0\xed\x2d\x30\xc1\x20\xef" - "\x70\x67\x5e\xff\x09\x70\xbb\xc1" - "\x3a\x7b\x48\x26\xd9\x0b\xf4\x48" - "\xbe\xce\xb1\xc7\xb2\x67\xc4\xa7" - "\x76\xf8\x36\x30\xb7\xb4\x9a\xd9" - "\xf5\x9d\xd0\x7b\xc1\x06\x96\x44" - "\x19\xc5\x58\x84\x63\xb9\x12\x68" - "\x68\xc7\xaa\x18\x98\xf2\x1f\x5c" - "\x39\xa6\xd8\x32\x2b\xc3\x51\xfd" - "\x74\x79\x2e\xb4\x44\xd7\x69\xc4" - "\xfc\x29\xe6\xed\x26\x1e\xa6\x9d" - "\x1c\xbe\x00\x0e\x7f\x3a\xca\xfb" - "\x6d\x13\x65\xa0\xf9\x31\x12\xe2" - "\x26\xd1\xec\x2b\x0a\x8b\x59\x99" - "\xa7\x49\xa0\x0e\x09\x33\x85\x50" - "\xc3\x23\xca\x7a\xdd\x13\x45\x5f" - "\xde\x4c\xa7\xcb\x00\x8a\x66\x6f" - "\xa2\xb6\xb1\x2e\xe1\xa0\x18\xf6" - "\xad\xf3\xbd\xeb\xc7\xef\x55\x4f" - "\x79\x91\x8d\x36\x13\x7b\xd0\x4a" - "\x6c\x39\xfb\x53\xb8\x6f\x02\x51" - "\xa5\x20\xac\x24\x1c\x73\x59\x73" - "\x58\x61\x3a\x87\x58\xb3\x20\x56" - "\x39\x06\x2b\x4d\xd3\x20\x2b\x89" - "\x3f\xa2\xf0\x96\xeb\x7f\xa4\xcd" - "\x11\xae\xbd\xcb\x3a\xb4\xd9\x91" - "\x09\x35\x71\x50\x65\xac\x92\xe3" - "\x7b\x32\xc0\x7a\xdd\xd4\xc3\x92" - "\x6f\xeb\x79\xde\x6f\xd3\x25\xc9" - "\xcd\x63\xf5\x1e\x7a\x3b\x26\x9d" - "\x77\x04\x80\xa9\xbf\x38\xb5\xbd" - "\xb8\x05\x07\xbd\xfd\xab\x7b\xf8" - "\x2a\x26\xcc\x49\x14\x6d\x55\x01" - "\x06\x94\xd8\xb2\x2d\x53\x83\x1b" - "\x8f\xd4\xdd\x57\x12\x7e\x18\xba" - "\x8e\xe2\x4d\x80\xef\x7e\x6b\x9d" - "\x24\xa9\x60\xa4\x97\x85\x86\x2a" - "\x01\x00\x09\xf1\xcb\x4a\x24\x1c" - "\xd8\xf6\xe6\x5b\xe7\x5d\xf2\xc4" - "\x97\x1c\x10\xc6\x4d\x66\x4f\x98" - "\x87\x30\xac\xd5\xea\x73\x49\x10" - "\x80\xea\xe5\x5f\x4d\x5f\x03\x33" - "\x66\x02\x35\x3d\x60\x06\x36\x4f" - "\x14\x1c\xd8\x07\x1f\x78\xd0\xf8" - "\x4f\x6c\x62\x7c\x15\xa5\x7c\x28" - "\x7c\xcc\xeb\x1f\xd1\x07\x90\x93" - "\x7e\xc2\xa8\x3a\x80\xc0\xf5\x30" - "\xcc\x75\xcf\x16\x26\xa9\x26\x3b" - "\xe7\x68\x2f\x15\x21\x5b\xe4\x00" - "\xbd\x48\x50\xcd\x75\x70\xc4\x62" - "\xbb\x41\xfb\x89\x4a\x88\x3b\x3b" - "\x51\x66\x02\x69\x04\x97\x36\xd4" - "\x75\xae\x0b\xa3\x42\xf8\xca\x79" - "\x8f\x93\xe9\xcc\x38\xbd\xd6\xd2" - "\xf9\x70\x4e\xc3\x6a\x8e\x25\xbd" - "\xea\x15\x5a\xa0\x85\x7e\x81\x0d" - "\x03\xe7\x05\x39\xf5\x05\x26\xee" - "\xec\xaa\x1f\x3d\xc9\x98\x76\x01" - "\x2c\xf4\xfc\xa3\x88\x77\x38\xc4" - "\x50\x65\x50\x6d\x04\x1f\xdf\x5a" - "\xaa\xf2\x01\xa9\xc1\x8d\xee\xca" - "\x47\x26\xef\x39\xb8\xb4\xf2\xd1" - "\xd6\xbb\x1b\x2a\xc1\x34\x14\xcf", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xc5\x85\x2a\x4b\x73\xe4\xf6\xf1" - "\x7e\xf9\xf6\xe9\xa3\x73\x36\xcb" - "\xaa\xb6\x22\xb0\x24\x6e\x3d\x73" - "\x92\x99\xde\xd3\x76\xed\xcd\x63" - "\x64\x3a\x22\x57\xc1\x43\x49\xd4" - "\x79\x36\x31\x19\x62\xae\x10\x7e" - "\x7d\xcf\x7a\xe2\x6b\xce\x27\xfa" - "\xdc\x3d\xd9\x83\xd3\x42\x4c\xe0" - "\x1b\xd6\x1d\x1a\x6f\xd2\x03\x00" - "\xfc\x81\x99\x8a\x14\x62\xf5\x7e" - "\x0d\xe7\x12\xe8\x17\x9d\x0b\xec" - "\xe2\xf7\xc9\xa7\x63\xd1\x79\xb6" - "\x62\x62\x37\xfe\x0a\x4c\x4a\x37" - "\x70\xc7\x5e\x96\x5f\xbc\x8e\x9e" - "\x85\x3c\x4f\x26\x64\x85\xbc\x68" - "\xb0\xe0\x86\x5e\x26\x41\xce\x11" - "\x50\xda\x97\x14\xe9\x9e\xc7\x6d" - "\x3b\xdc\x43\xde\x2b\x27\x69\x7d" - "\xfc\xb0\x28\xbd\x8f\xb1\xc6\x31" - "\x14\x4d\xf0\x74\x37\xfd\x07\x25" - "\x96\x55\xe5\xfc\x9e\x27\x2a\x74" - "\x1b\x83\x4d\x15\x83\xac\x57\xa0" - "\xac\xa5\xd0\x38\xef\x19\x56\x53" - "\x25\x4b\xfc\xce\x04\x23\xe5\x6b" - "\xf6\xc6\x6c\x32\x0b\xb3\x12\xc5" - "\xed\x22\x34\x1c\x5d\xed\x17\x06" - "\x36\xa3\xe6\x77\xb9\x97\x46\xb8" - "\xe9\x3f\x7e\xc7\xbc\x13\x5c\xdc" - "\x6e\x3f\x04\x5e\xd1\x59\xa5\x82" - "\x35\x91\x3d\x1b\xe4\x97\x9f\x92" - "\x1c\x5e\x5f\x6f\x41\xd4\x62\xa1" - "\x8d\x39\xfc\x42\xfb\x38\x80\xb9" - "\x0a\xe3\xcc\x6a\x93\xd9\x7a\xb1" - "\xe9\x69\xaf\x0a\x6b\x75\x38\xa7" - "\xa1\xbf\xf7\xda\x95\x93\x4b\x78" - "\x19\xf5\x94\xf9\xd2\x00\x33\x37" - "\xcf\xf5\x9e\x9c\xf3\xcc\xa6\xee" - "\x42\xb2\x9e\x2c\x5f\x48\x23\x26" - "\x15\x25\x17\x03\x3d\xfe\x2c\xfc" - "\xeb\xba\xda\xe0\x00\x05\xb6\xa6" - "\x07\xb3\xe8\x36\x5b\xec\x5b\xbf" - "\xd6\x5b\x00\x74\xc6\x97\xf1\x6a" - "\x49\xa1\xc3\xfa\x10\x52\xb9\x14" - "\xad\xb7\x73\xf8\x78\x12\xc8\x59" - "\x17\x80\x4c\x57\x39\xf1\x6d\x80" - "\x25\x77\x0f\x5e\x7d\xf0\xaf\x21" - "\xec\xce\xb7\xc8\x02\x8a\xed\x53" - "\x2c\x25\x68\x2e\x1f\x85\x5e\x67" - "\xd1\x07\x7a\x3a\x89\x08\xe0\x34" - "\xdc\xdb\x26\xb4\x6b\x77\xfc\x40" - "\x31\x15\x72\xa0\xf0\x73\xd9\x3b" - "\xd5\xdb\xfe\xfc\x8f\xa9\x44\xa2" - "\x09\x9f\xc6\x33\xe5\xe2\x88\xe8" - "\xf3\xf0\x1a\xf4\xce\x12\x0f\xd6" - "\xf7\x36\xe6\xa4\xf4\x7a\x10\x58" - "\xcc\x1f\x48\x49\x65\x47\x75\xe9" - "\x28\xe1\x65\x7b\xf2\xc4\xb5\x07" - "\xf2\xec\x76\xd8\x8f\x09\xf3\x16" - "\xa1\x51\x89\x3b\xeb\x96\x42\xac" - "\x65\xe0\x67\x63\x29\xdc\xb4\x7d" - "\xf2\x41\x51\x6a\xcb\xde\x3c\xfb" - "\x66\x8d\x13\xca\xe0\x59\x2a\x00" - "\xc9\x53\x4c\xe6\x9e\xe2\x73\xd5" - "\x67\x19\xb2\xbd\x9a\x63\xd7\x5c", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - } -}; - -static const struct cipher_testvec speck64_enc_tv_template[] = { - { /* Speck64/96 */ - .key = "\x00\x01\x02\x03\x08\x09\x0a\x0b" - "\x10\x11\x12\x13", - .klen = 12, - .input = "\x65\x61\x6e\x73\x20\x46\x61\x74", - .ilen = 8, - .result = "\x6c\x94\x75\x41\xec\x52\x79\x9f", - .rlen = 8, - }, { /* Speck64/128 */ - .key = "\x00\x01\x02\x03\x08\x09\x0a\x0b" - "\x10\x11\x12\x13\x18\x19\x1a\x1b", - .klen = 16, - .input = "\x2d\x43\x75\x74\x74\x65\x72\x3b", - .ilen = 8, - .result = "\x8b\x02\x4e\x45\x48\xa5\x6f\x8c", - .rlen = 8, - }, -}; - -static const struct cipher_testvec speck64_dec_tv_template[] = { - { /* Speck64/96 */ - .key = "\x00\x01\x02\x03\x08\x09\x0a\x0b" - "\x10\x11\x12\x13", - .klen = 12, - .input = "\x6c\x94\x75\x41\xec\x52\x79\x9f", - .ilen = 8, - .result = "\x65\x61\x6e\x73\x20\x46\x61\x74", - .rlen = 8, - }, { /* Speck64/128 */ - .key = "\x00\x01\x02\x03\x08\x09\x0a\x0b" - "\x10\x11\x12\x13\x18\x19\x1a\x1b", - .klen = 16, - .input = "\x8b\x02\x4e\x45\x48\xa5\x6f\x8c", - .ilen = 8, - .result = "\x2d\x43\x75\x74\x74\x65\x72\x3b", - .rlen = 8, - }, -}; - -/* - * Speck64-XTS test vectors, taken from the AES-XTS test vectors with the result - * recomputed with Speck64 as the cipher, and key lengths adjusted - */ - -static const struct cipher_testvec speck64_xts_enc_tv_template[] = { - { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 24, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 32, - .result = "\x84\xaf\x54\x07\x19\xd4\x7c\xa6" - "\xe4\xfe\xdf\xc4\x1f\x34\xc3\xc2" - "\x80\xf5\x72\xe7\xcd\xf0\x99\x22" - "\x35\xa7\x2f\x06\xef\xdc\x51\xaa", - .rlen = 32, - }, { - .key = "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 24, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\x12\x56\x73\xcd\x15\x87\xa8\x59" - "\xcf\x84\xae\xd9\x1c\x66\xd6\x9f" - "\xb3\x12\x69\x7e\x36\xeb\x52\xff" - "\x62\xdd\xba\x90\xb3\xe1\xee\x99", - .rlen = 32, - }, { - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 24, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\x15\x1b\xe4\x2c\xa2\x5a\x2d\x2c" - "\x27\x36\xc0\xbf\x5d\xea\x36\x37" - "\x2d\x1a\x88\xbc\x66\xb5\xd0\x0b" - "\xa1\xbc\x19\xb2\x0f\x3b\x75\x34", - .rlen = 32, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93", - .klen = 24, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\xaf\xa1\x81\xa6\x32\xbb\x15\x8e" - "\xf8\x95\x2e\xd3\xe6\xee\x7e\x09" - "\x0c\x1a\xf5\x02\x97\x8b\xe3\xb3" - "\x11\xc7\x39\x96\xd0\x95\xf4\x56" - "\xf4\xdd\x03\x38\x01\x44\x2c\xcf" - "\x88\xae\x8e\x3c\xcd\xe7\xaa\x66" - "\xfe\x3d\xc6\xfb\x01\x23\x51\x43" - "\xd5\xd2\x13\x86\x94\x34\xe9\x62" - "\xf9\x89\xe3\xd1\x7b\xbe\xf8\xef" - "\x76\x35\x04\x3f\xdb\x23\x9d\x0b" - "\x85\x42\xb9\x02\xd6\xcc\xdb\x96" - "\xa7\x6b\x27\xb6\xd4\x45\x8f\x7d" - "\xae\xd2\x04\xd5\xda\xc1\x7e\x24" - "\x8c\x73\xbe\x48\x7e\xcf\x65\x28" - "\x29\xe5\xbe\x54\x30\xcb\x46\x95" - "\x4f\x2e\x8a\x36\xc8\x27\xc5\xbe" - "\xd0\x1a\xaf\xab\x26\xcd\x9e\x69" - "\xa1\x09\x95\x71\x26\xe9\xc4\xdf" - "\xe6\x31\xc3\x46\xda\xaf\x0b\x41" - "\x1f\xab\xb1\x8e\xd6\xfc\x0b\xb3" - "\x82\xc0\x37\x27\xfc\x91\xa7\x05" - "\xfb\xc5\xdc\x2b\x74\x96\x48\x43" - "\x5d\x9c\x19\x0f\x60\x63\x3a\x1f" - "\x6f\xf0\x03\xbe\x4d\xfd\xc8\x4a" - "\xc6\xa4\x81\x6d\xc3\x12\x2a\x5c" - "\x07\xff\xf3\x72\x74\x48\xb5\x40" - "\x50\xb5\xdd\x90\x43\x31\x18\x15" - "\x7b\xf2\xa6\xdb\x83\xc8\x4b\x4a" - "\x29\x93\x90\x8b\xda\x07\xf0\x35" - "\x6d\x90\x88\x09\x4e\x83\xf5\x5b" - "\x94\x12\xbb\x33\x27\x1d\x3f\x23" - "\x51\xa8\x7c\x07\xa2\xae\x77\xa6" - "\x50\xfd\xcc\xc0\x4f\x80\x7a\x9f" - "\x66\xdd\xcd\x75\x24\x8b\x33\xf7" - "\x20\xdb\x83\x9b\x4f\x11\x63\x6e" - "\xcf\x37\xef\xc9\x11\x01\x5c\x45" - "\x32\x99\x7c\x3c\x9e\x42\x89\xe3" - "\x70\x6d\x15\x9f\xb1\xe6\xb6\x05" - "\xfe\x0c\xb9\x49\x2d\x90\x6d\xcc" - "\x5d\x3f\xc1\xfe\x89\x0a\x2e\x2d" - "\xa0\xa8\x89\x3b\x73\x39\xa5\x94" - "\x4c\xa4\xa6\xbb\xa7\x14\x46\x89" - "\x10\xff\xaf\xef\xca\xdd\x4f\x80" - "\xb3\xdf\x3b\xab\xd4\xe5\x5a\xc7" - "\x33\xca\x00\x8b\x8b\x3f\xea\xec" - "\x68\x8a\xc2\x6d\xfd\xd4\x67\x0f" - "\x22\x31\xe1\x0e\xfe\x5a\x04\xd5" - "\x64\xa3\xf1\x1a\x76\x28\xcc\x35" - "\x36\xa7\x0a\x74\xf7\x1c\x44\x9b" - "\xc7\x1b\x53\x17\x02\xea\xd1\xad" - "\x13\x51\x73\xc0\xa0\xb2\x05\x32" - "\xa8\xa2\x37\x2e\xe1\x7a\x3a\x19" - "\x26\xb4\x6c\x62\x5d\xb3\x1a\x1d" - "\x59\xda\xee\x1a\x22\x18\xda\x0d" - "\x88\x0f\x55\x8b\x72\x62\xfd\xc1" - "\x69\x13\xcd\x0d\x5f\xc1\x09\x52" - "\xee\xd6\xe3\x84\x4d\xee\xf6\x88" - "\xaf\x83\xdc\x76\xf4\xc0\x93\x3f" - "\x4a\x75\x2f\xb0\x0b\x3e\xc4\x54" - "\x7d\x69\x8d\x00\x62\x77\x0d\x14" - "\xbe\x7c\xa6\x7d\xc5\x24\x4f\xf3" - "\x50\xf7\x5f\xf4\xc2\xca\x41\x97" - "\x37\xbe\x75\x74\xcd\xf0\x75\x6e" - "\x25\x23\x94\xbd\xda\x8d\xb0\xd4", - .rlen = 512, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27", - .klen = 32, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\x55\xed\x71\xd3\x02\x8e\x15\x3b" - "\xc6\x71\x29\x2d\x3e\x89\x9f\x59" - "\x68\x6a\xcc\x8a\x56\x97\xf3\x95" - "\x4e\x51\x08\xda\x2a\xf8\x6f\x3c" - "\x78\x16\xea\x80\xdb\x33\x75\x94" - "\xf9\x29\xc4\x2b\x76\x75\x97\xc7" - "\xf2\x98\x2c\xf9\xff\xc8\xd5\x2b" - "\x18\xf1\xaf\xcf\x7c\xc5\x0b\xee" - "\xad\x3c\x76\x7c\xe6\x27\xa2\x2a" - "\xe4\x66\xe1\xab\xa2\x39\xfc\x7c" - "\xf5\xec\x32\x74\xa3\xb8\x03\x88" - "\x52\xfc\x2e\x56\x3f\xa1\xf0\x9f" - "\x84\x5e\x46\xed\x20\x89\xb6\x44" - "\x8d\xd0\xed\x54\x47\x16\xbe\x95" - "\x8a\xb3\x6b\x72\xc4\x32\x52\x13" - "\x1b\xb0\x82\xbe\xac\xf9\x70\xa6" - "\x44\x18\xdd\x8c\x6e\xca\x6e\x45" - "\x8f\x1e\x10\x07\x57\x25\x98\x7b" - "\x17\x8c\x78\xdd\x80\xa7\xd9\xd8" - "\x63\xaf\xb9\x67\x57\xfd\xbc\xdb" - "\x44\xe9\xc5\x65\xd1\xc7\x3b\xff" - "\x20\xa0\x80\x1a\xc3\x9a\xad\x5e" - "\x5d\x3b\xd3\x07\xd9\xf5\xfd\x3d" - "\x4a\x8b\xa8\xd2\x6e\x7a\x51\x65" - "\x6c\x8e\x95\xe0\x45\xc9\x5f\x4a" - "\x09\x3c\x3d\x71\x7f\x0c\x84\x2a" - "\xc8\x48\x52\x1a\xc2\xd5\xd6\x78" - "\x92\x1e\xa0\x90\x2e\xea\xf0\xf3" - "\xdc\x0f\xb1\xaf\x0d\x9b\x06\x2e" - "\x35\x10\x30\x82\x0d\xe7\xc5\x9b" - "\xde\x44\x18\xbd\x9f\xd1\x45\xa9" - "\x7b\x7a\x4a\xad\x35\x65\x27\xca" - "\xb2\xc3\xd4\x9b\x71\x86\x70\xee" - "\xf1\x89\x3b\x85\x4b\x5b\xaa\xaf" - "\xfc\x42\xc8\x31\x59\xbe\x16\x60" - "\x4f\xf9\xfa\x12\xea\xd0\xa7\x14" - "\xf0\x7a\xf3\xd5\x8d\xbd\x81\xef" - "\x52\x7f\x29\x51\x94\x20\x67\x3c" - "\xd1\xaf\x77\x9f\x22\x5a\x4e\x63" - "\xe7\xff\x73\x25\xd1\xdd\x96\x8a" - "\x98\x52\x6d\xf3\xac\x3e\xf2\x18" - "\x6d\xf6\x0a\x29\xa6\x34\x3d\xed" - "\xe3\x27\x0d\x9d\x0a\x02\x44\x7e" - "\x5a\x7e\x67\x0f\x0a\x9e\xd6\xad" - "\x91\xe6\x4d\x81\x8c\x5c\x59\xaa" - "\xfb\xeb\x56\x53\xd2\x7d\x4c\x81" - "\x65\x53\x0f\x41\x11\xbd\x98\x99" - "\xf9\xc6\xfa\x51\x2e\xa3\xdd\x8d" - "\x84\x98\xf9\x34\xed\x33\x2a\x1f" - "\x82\xed\xc1\x73\x98\xd3\x02\xdc" - "\xe6\xc2\x33\x1d\xa2\xb4\xca\x76" - "\x63\x51\x34\x9d\x96\x12\xae\xce" - "\x83\xc9\x76\x5e\xa4\x1b\x53\x37" - "\x17\xd5\xc0\x80\x1d\x62\xf8\x3d" - "\x54\x27\x74\xbb\x10\x86\x57\x46" - "\x68\xe1\xed\x14\xe7\x9d\xfc\x84" - "\x47\xbc\xc2\xf8\x19\x4b\x99\xcf" - "\x7a\xe9\xc4\xb8\x8c\x82\x72\x4d" - "\x7b\x4f\x38\x55\x36\x71\x64\xc1" - "\xfc\x5c\x75\x52\x33\x02\x18\xf8" - "\x17\xe1\x2b\xc2\x43\x39\xbd\x76" - "\x9b\x63\x76\x32\x2f\x19\x72\x10" - "\x9f\x21\x0c\xf1\x66\x50\x7f\xa5" - "\x0d\x1f\x46\xe0\xba\xd3\x2f\x3c", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - } -}; - -static const struct cipher_testvec speck64_xts_dec_tv_template[] = { - { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 24, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x84\xaf\x54\x07\x19\xd4\x7c\xa6" - "\xe4\xfe\xdf\xc4\x1f\x34\xc3\xc2" - "\x80\xf5\x72\xe7\xcd\xf0\x99\x22" - "\x35\xa7\x2f\x06\xef\xdc\x51\xaa", - .ilen = 32, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 32, - }, { - .key = "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 24, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x12\x56\x73\xcd\x15\x87\xa8\x59" - "\xcf\x84\xae\xd9\x1c\x66\xd6\x9f" - "\xb3\x12\x69\x7e\x36\xeb\x52\xff" - "\x62\xdd\xba\x90\xb3\xe1\xee\x99", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 24, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x15\x1b\xe4\x2c\xa2\x5a\x2d\x2c" - "\x27\x36\xc0\xbf\x5d\xea\x36\x37" - "\x2d\x1a\x88\xbc\x66\xb5\xd0\x0b" - "\xa1\xbc\x19\xb2\x0f\x3b\x75\x34", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93", - .klen = 24, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xaf\xa1\x81\xa6\x32\xbb\x15\x8e" - "\xf8\x95\x2e\xd3\xe6\xee\x7e\x09" - "\x0c\x1a\xf5\x02\x97\x8b\xe3\xb3" - "\x11\xc7\x39\x96\xd0\x95\xf4\x56" - "\xf4\xdd\x03\x38\x01\x44\x2c\xcf" - "\x88\xae\x8e\x3c\xcd\xe7\xaa\x66" - "\xfe\x3d\xc6\xfb\x01\x23\x51\x43" - "\xd5\xd2\x13\x86\x94\x34\xe9\x62" - "\xf9\x89\xe3\xd1\x7b\xbe\xf8\xef" - "\x76\x35\x04\x3f\xdb\x23\x9d\x0b" - "\x85\x42\xb9\x02\xd6\xcc\xdb\x96" - "\xa7\x6b\x27\xb6\xd4\x45\x8f\x7d" - "\xae\xd2\x04\xd5\xda\xc1\x7e\x24" - "\x8c\x73\xbe\x48\x7e\xcf\x65\x28" - "\x29\xe5\xbe\x54\x30\xcb\x46\x95" - "\x4f\x2e\x8a\x36\xc8\x27\xc5\xbe" - "\xd0\x1a\xaf\xab\x26\xcd\x9e\x69" - "\xa1\x09\x95\x71\x26\xe9\xc4\xdf" - "\xe6\x31\xc3\x46\xda\xaf\x0b\x41" - "\x1f\xab\xb1\x8e\xd6\xfc\x0b\xb3" - "\x82\xc0\x37\x27\xfc\x91\xa7\x05" - "\xfb\xc5\xdc\x2b\x74\x96\x48\x43" - "\x5d\x9c\x19\x0f\x60\x63\x3a\x1f" - "\x6f\xf0\x03\xbe\x4d\xfd\xc8\x4a" - "\xc6\xa4\x81\x6d\xc3\x12\x2a\x5c" - "\x07\xff\xf3\x72\x74\x48\xb5\x40" - "\x50\xb5\xdd\x90\x43\x31\x18\x15" - "\x7b\xf2\xa6\xdb\x83\xc8\x4b\x4a" - "\x29\x93\x90\x8b\xda\x07\xf0\x35" - "\x6d\x90\x88\x09\x4e\x83\xf5\x5b" - "\x94\x12\xbb\x33\x27\x1d\x3f\x23" - "\x51\xa8\x7c\x07\xa2\xae\x77\xa6" - "\x50\xfd\xcc\xc0\x4f\x80\x7a\x9f" - "\x66\xdd\xcd\x75\x24\x8b\x33\xf7" - "\x20\xdb\x83\x9b\x4f\x11\x63\x6e" - "\xcf\x37\xef\xc9\x11\x01\x5c\x45" - "\x32\x99\x7c\x3c\x9e\x42\x89\xe3" - "\x70\x6d\x15\x9f\xb1\xe6\xb6\x05" - "\xfe\x0c\xb9\x49\x2d\x90\x6d\xcc" - "\x5d\x3f\xc1\xfe\x89\x0a\x2e\x2d" - "\xa0\xa8\x89\x3b\x73\x39\xa5\x94" - "\x4c\xa4\xa6\xbb\xa7\x14\x46\x89" - "\x10\xff\xaf\xef\xca\xdd\x4f\x80" - "\xb3\xdf\x3b\xab\xd4\xe5\x5a\xc7" - "\x33\xca\x00\x8b\x8b\x3f\xea\xec" - "\x68\x8a\xc2\x6d\xfd\xd4\x67\x0f" - "\x22\x31\xe1\x0e\xfe\x5a\x04\xd5" - "\x64\xa3\xf1\x1a\x76\x28\xcc\x35" - "\x36\xa7\x0a\x74\xf7\x1c\x44\x9b" - "\xc7\x1b\x53\x17\x02\xea\xd1\xad" - "\x13\x51\x73\xc0\xa0\xb2\x05\x32" - "\xa8\xa2\x37\x2e\xe1\x7a\x3a\x19" - "\x26\xb4\x6c\x62\x5d\xb3\x1a\x1d" - "\x59\xda\xee\x1a\x22\x18\xda\x0d" - "\x88\x0f\x55\x8b\x72\x62\xfd\xc1" - "\x69\x13\xcd\x0d\x5f\xc1\x09\x52" - "\xee\xd6\xe3\x84\x4d\xee\xf6\x88" - "\xaf\x83\xdc\x76\xf4\xc0\x93\x3f" - "\x4a\x75\x2f\xb0\x0b\x3e\xc4\x54" - "\x7d\x69\x8d\x00\x62\x77\x0d\x14" - "\xbe\x7c\xa6\x7d\xc5\x24\x4f\xf3" - "\x50\xf7\x5f\xf4\xc2\xca\x41\x97" - "\x37\xbe\x75\x74\xcd\xf0\x75\x6e" - "\x25\x23\x94\xbd\xda\x8d\xb0\xd4", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27", - .klen = 32, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x55\xed\x71\xd3\x02\x8e\x15\x3b" - "\xc6\x71\x29\x2d\x3e\x89\x9f\x59" - "\x68\x6a\xcc\x8a\x56\x97\xf3\x95" - "\x4e\x51\x08\xda\x2a\xf8\x6f\x3c" - "\x78\x16\xea\x80\xdb\x33\x75\x94" - "\xf9\x29\xc4\x2b\x76\x75\x97\xc7" - "\xf2\x98\x2c\xf9\xff\xc8\xd5\x2b" - "\x18\xf1\xaf\xcf\x7c\xc5\x0b\xee" - "\xad\x3c\x76\x7c\xe6\x27\xa2\x2a" - "\xe4\x66\xe1\xab\xa2\x39\xfc\x7c" - "\xf5\xec\x32\x74\xa3\xb8\x03\x88" - "\x52\xfc\x2e\x56\x3f\xa1\xf0\x9f" - "\x84\x5e\x46\xed\x20\x89\xb6\x44" - "\x8d\xd0\xed\x54\x47\x16\xbe\x95" - "\x8a\xb3\x6b\x72\xc4\x32\x52\x13" - "\x1b\xb0\x82\xbe\xac\xf9\x70\xa6" - "\x44\x18\xdd\x8c\x6e\xca\x6e\x45" - "\x8f\x1e\x10\x07\x57\x25\x98\x7b" - "\x17\x8c\x78\xdd\x80\xa7\xd9\xd8" - "\x63\xaf\xb9\x67\x57\xfd\xbc\xdb" - "\x44\xe9\xc5\x65\xd1\xc7\x3b\xff" - "\x20\xa0\x80\x1a\xc3\x9a\xad\x5e" - "\x5d\x3b\xd3\x07\xd9\xf5\xfd\x3d" - "\x4a\x8b\xa8\xd2\x6e\x7a\x51\x65" - "\x6c\x8e\x95\xe0\x45\xc9\x5f\x4a" - "\x09\x3c\x3d\x71\x7f\x0c\x84\x2a" - "\xc8\x48\x52\x1a\xc2\xd5\xd6\x78" - "\x92\x1e\xa0\x90\x2e\xea\xf0\xf3" - "\xdc\x0f\xb1\xaf\x0d\x9b\x06\x2e" - "\x35\x10\x30\x82\x0d\xe7\xc5\x9b" - "\xde\x44\x18\xbd\x9f\xd1\x45\xa9" - "\x7b\x7a\x4a\xad\x35\x65\x27\xca" - "\xb2\xc3\xd4\x9b\x71\x86\x70\xee" - "\xf1\x89\x3b\x85\x4b\x5b\xaa\xaf" - "\xfc\x42\xc8\x31\x59\xbe\x16\x60" - "\x4f\xf9\xfa\x12\xea\xd0\xa7\x14" - "\xf0\x7a\xf3\xd5\x8d\xbd\x81\xef" - "\x52\x7f\x29\x51\x94\x20\x67\x3c" - "\xd1\xaf\x77\x9f\x22\x5a\x4e\x63" - "\xe7\xff\x73\x25\xd1\xdd\x96\x8a" - "\x98\x52\x6d\xf3\xac\x3e\xf2\x18" - "\x6d\xf6\x0a\x29\xa6\x34\x3d\xed" - "\xe3\x27\x0d\x9d\x0a\x02\x44\x7e" - "\x5a\x7e\x67\x0f\x0a\x9e\xd6\xad" - "\x91\xe6\x4d\x81\x8c\x5c\x59\xaa" - "\xfb\xeb\x56\x53\xd2\x7d\x4c\x81" - "\x65\x53\x0f\x41\x11\xbd\x98\x99" - "\xf9\xc6\xfa\x51\x2e\xa3\xdd\x8d" - "\x84\x98\xf9\x34\xed\x33\x2a\x1f" - "\x82\xed\xc1\x73\x98\xd3\x02\xdc" - "\xe6\xc2\x33\x1d\xa2\xb4\xca\x76" - "\x63\x51\x34\x9d\x96\x12\xae\xce" - "\x83\xc9\x76\x5e\xa4\x1b\x53\x37" - "\x17\xd5\xc0\x80\x1d\x62\xf8\x3d" - "\x54\x27\x74\xbb\x10\x86\x57\x46" - "\x68\xe1\xed\x14\xe7\x9d\xfc\x84" - "\x47\xbc\xc2\xf8\x19\x4b\x99\xcf" - "\x7a\xe9\xc4\xb8\x8c\x82\x72\x4d" - "\x7b\x4f\x38\x55\x36\x71\x64\xc1" - "\xfc\x5c\x75\x52\x33\x02\x18\xf8" - "\x17\xe1\x2b\xc2\x43\x39\xbd\x76" - "\x9b\x63\x76\x32\x2f\x19\x72\x10" - "\x9f\x21\x0c\xf1\x66\x50\x7f\xa5" - "\x0d\x1f\x46\xe0\xba\xd3\x2f\x3c", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - } -}; - -/* Cast6 test vectors from RFC 2612 */ -static const struct cipher_testvec cast6_enc_tv_template[] = { - { - .key = "\x23\x42\xbb\x9e\xfa\x38\x54\x2c" - "\x0a\xf7\x56\x47\xf2\x9f\x61\x5d", - .klen = 16, - .input = zeroed_string, - .ilen = 16, - .result = "\xc8\x42\xa0\x89\x72\xb4\x3d\x20" - "\x83\x6c\x91\xd1\xb7\x53\x0f\x6b", - .rlen = 16, - }, { - .key = "\x23\x42\xbb\x9e\xfa\x38\x54\x2c" - "\xbe\xd0\xac\x83\x94\x0a\xc2\x98" - "\xba\xc7\x7a\x77\x17\x94\x28\x63", - .klen = 24, - .input = zeroed_string, - .ilen = 16, - .result = "\x1b\x38\x6c\x02\x10\xdc\xad\xcb" - "\xdd\x0e\x41\xaa\x08\xa7\xa7\xe8", - .rlen = 16, - }, { - .key = "\x23\x42\xbb\x9e\xfa\x38\x54\x2c" - "\xbe\xd0\xac\x83\x94\x0a\xc2\x98" - "\x8d\x7c\x47\xce\x26\x49\x08\x46" - "\x1c\xc1\xb5\x13\x7a\xe6\xb6\x04", - .klen = 32, - .input = zeroed_string, - .ilen = 16, - .result = "\x4f\x6a\x20\x38\x28\x68\x97\xb9" - "\xc9\x87\x01\x36\x55\x33\x17\xfa", - .rlen = 16, - }, { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\xC3\x70\x22\x32\xF5\x80\xCB\x54" - "\xFC\x30\xE0\xF6\xEB\x39\x57\xA6" - "\xB6\xB9\xC5\xA4\x91\x55\x14\x97" - "\xC1\x20\xFF\x6C\x5C\xF0\x67\xEA" - "\x2F\xED\xD8\xC9\xFB\x38\x3F\xFE" - "\x93\xBE\xDC\x00\xD3\x7F\xAD\x4C" - "\x5A\x08\x92\xD1\x47\x0C\xFA\x6C" - "\xD0\x6A\x99\x10\x72\xF8\x47\x62" - "\x81\x42\xF8\xD8\xF5\xBB\x94\x08" - "\xAA\x97\xA2\x8B\x69\xB3\xD2\x7E" - "\xBC\xB5\x00\x0C\xE5\x44\x4B\x58" - "\xE8\x63\xDC\xB3\xC4\xE5\x23\x12" - "\x5A\x72\x85\x47\x8B\xEC\x9F\x26" - "\x84\xB6\xED\x10\x33\x63\x9B\x5F" - "\x4D\x53\xEE\x94\x45\x8B\x60\x58" - "\x86\x20\xF9\x1E\x82\x08\x3E\x58" - "\x60\x1B\x34\x19\x02\xBE\x4E\x09" - "\xBB\x7C\x15\xCC\x60\x27\x55\x7A" - "\x12\xB8\xD8\x08\x89\x3C\xA6\xF3" - "\xF1\xDD\xA7\x07\xA3\x12\x85\x28" - "\xE9\x57\xAC\x80\x0C\x5C\x0F\x3A" - "\x5D\xC2\x91\xC7\x90\xE4\x8C\x43" - "\x92\xE4\x7C\x26\x69\x4D\x83\x68" - "\x14\x96\x42\x47\xBD\xA9\xE4\x8A" - "\x33\x19\xEB\x54\x8E\x0D\x4B\x6E" - "\x91\x51\xB5\x36\x08\xDE\x1C\x06" - "\x03\xBD\xDE\x81\x26\xF7\x99\xC2" - "\xBA\xF7\x6D\x87\x0D\xE4\xA6\xCF" - "\xC1\xF5\x27\x05\xB8\x02\x57\x72" - "\xE6\x42\x13\x0B\xC6\x47\x05\x74" - "\x24\x15\xF7\x0D\xC2\x23\x9D\xB9" - "\x3C\x77\x18\x93\xBA\xB4\xFC\x8C" - "\x98\x82\x67\x67\xB4\xD7\xD3\x43" - "\x23\x08\x02\xB7\x9B\x99\x05\xFB" - "\xD3\xB5\x00\x0A\xA9\x9D\x66\xD6" - "\x2E\x49\x58\xD0\xA8\x57\x29\x7F" - "\x0A\x0E\x7D\xFC\x92\x83\xCC\x67" - "\xA2\xB1\x70\x3A\x8F\x87\x4A\x8D" - "\x17\xE2\x58\x2B\x88\x0D\x68\x62" - "\xBF\x35\xD1\x6F\xC0\xF0\x18\x62" - "\xB2\xC7\x2D\x58\xC7\x16\xDE\x08" - "\xEB\x84\x1D\x25\xA7\x38\x94\x06" - "\x93\x9D\xF8\xFE\x88\x71\xE7\x84" - "\x2C\xA0\x38\xA3\x1D\x48\xCF\x29" - "\x0B\xBC\xD8\x50\x99\x1A\x26\xFB" - "\x8E\x75\x3D\x73\xEB\x6A\xED\x29" - "\xE0\x8E\xED\xFC\xFE\x6F\xF6\xBA" - "\x41\xE2\x10\x4C\x01\x8B\x69\x2B" - "\x25\x3F\x4D\x70\x7B\x92\xD6\x3B" - "\xAC\xF9\x77\x18\xD9\x6A\x30\xA6" - "\x2E\xFA\x30\xFF\xC8\xD5\x1D\x06" - "\x59\x28\x1D\x86\x43\x04\x5D\x3B" - "\x99\x4C\x04\x5A\x21\x17\x8B\x76" - "\x8F\x72\xCB\xA1\x9C\x29\x4C\xC3" - "\x65\xA2\x58\x2A\xC5\x66\x24\xBF" - "\xBA\xE6\x0C\xDD\x34\x24\x74\xC8" - "\x84\x0A\x66\x2C\xBE\x8F\x32\xA9" - "\xE7\xE4\xA1\xD7\xDA\xAB\x23\x1E" - "\xEB\xEE\x6C\x94\x6F\x9C\x2E\xD1" - "\x49\x2C\xF3\xD4\x90\xCC\x93\x4C" - "\x84\x52\x6D\x68\xDE\xC6\x64\xB2" - "\x11\x74\x93\x57\xB4\x7E\xC6\x00", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast6_dec_tv_template[] = { - { - .key = "\x23\x42\xbb\x9e\xfa\x38\x54\x2c" - "\x0a\xf7\x56\x47\xf2\x9f\x61\x5d", - .klen = 16, - .input = "\xc8\x42\xa0\x89\x72\xb4\x3d\x20" - "\x83\x6c\x91\xd1\xb7\x53\x0f\x6b", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, - }, { - .key = "\x23\x42\xbb\x9e\xfa\x38\x54\x2c" - "\xbe\xd0\xac\x83\x94\x0a\xc2\x98" - "\xba\xc7\x7a\x77\x17\x94\x28\x63", - .klen = 24, - .input = "\x1b\x38\x6c\x02\x10\xdc\xad\xcb" - "\xdd\x0e\x41\xaa\x08\xa7\xa7\xe8", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, - }, { - .key = "\x23\x42\xbb\x9e\xfa\x38\x54\x2c" - "\xbe\xd0\xac\x83\x94\x0a\xc2\x98" - "\x8d\x7c\x47\xce\x26\x49\x08\x46" - "\x1c\xc1\xb5\x13\x7a\xe6\xb6\x04", - .klen = 32, - .input = "\x4f\x6a\x20\x38\x28\x68\x97\xb9" - "\xc9\x87\x01\x36\x55\x33\x17\xfa", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, - }, { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\xC3\x70\x22\x32\xF5\x80\xCB\x54" - "\xFC\x30\xE0\xF6\xEB\x39\x57\xA6" - "\xB6\xB9\xC5\xA4\x91\x55\x14\x97" - "\xC1\x20\xFF\x6C\x5C\xF0\x67\xEA" - "\x2F\xED\xD8\xC9\xFB\x38\x3F\xFE" - "\x93\xBE\xDC\x00\xD3\x7F\xAD\x4C" - "\x5A\x08\x92\xD1\x47\x0C\xFA\x6C" - "\xD0\x6A\x99\x10\x72\xF8\x47\x62" - "\x81\x42\xF8\xD8\xF5\xBB\x94\x08" - "\xAA\x97\xA2\x8B\x69\xB3\xD2\x7E" - "\xBC\xB5\x00\x0C\xE5\x44\x4B\x58" - "\xE8\x63\xDC\xB3\xC4\xE5\x23\x12" - "\x5A\x72\x85\x47\x8B\xEC\x9F\x26" - "\x84\xB6\xED\x10\x33\x63\x9B\x5F" - "\x4D\x53\xEE\x94\x45\x8B\x60\x58" - "\x86\x20\xF9\x1E\x82\x08\x3E\x58" - "\x60\x1B\x34\x19\x02\xBE\x4E\x09" - "\xBB\x7C\x15\xCC\x60\x27\x55\x7A" - "\x12\xB8\xD8\x08\x89\x3C\xA6\xF3" - "\xF1\xDD\xA7\x07\xA3\x12\x85\x28" - "\xE9\x57\xAC\x80\x0C\x5C\x0F\x3A" - "\x5D\xC2\x91\xC7\x90\xE4\x8C\x43" - "\x92\xE4\x7C\x26\x69\x4D\x83\x68" - "\x14\x96\x42\x47\xBD\xA9\xE4\x8A" - "\x33\x19\xEB\x54\x8E\x0D\x4B\x6E" - "\x91\x51\xB5\x36\x08\xDE\x1C\x06" - "\x03\xBD\xDE\x81\x26\xF7\x99\xC2" - "\xBA\xF7\x6D\x87\x0D\xE4\xA6\xCF" - "\xC1\xF5\x27\x05\xB8\x02\x57\x72" - "\xE6\x42\x13\x0B\xC6\x47\x05\x74" - "\x24\x15\xF7\x0D\xC2\x23\x9D\xB9" - "\x3C\x77\x18\x93\xBA\xB4\xFC\x8C" - "\x98\x82\x67\x67\xB4\xD7\xD3\x43" - "\x23\x08\x02\xB7\x9B\x99\x05\xFB" - "\xD3\xB5\x00\x0A\xA9\x9D\x66\xD6" - "\x2E\x49\x58\xD0\xA8\x57\x29\x7F" - "\x0A\x0E\x7D\xFC\x92\x83\xCC\x67" - "\xA2\xB1\x70\x3A\x8F\x87\x4A\x8D" - "\x17\xE2\x58\x2B\x88\x0D\x68\x62" - "\xBF\x35\xD1\x6F\xC0\xF0\x18\x62" - "\xB2\xC7\x2D\x58\xC7\x16\xDE\x08" - "\xEB\x84\x1D\x25\xA7\x38\x94\x06" - "\x93\x9D\xF8\xFE\x88\x71\xE7\x84" - "\x2C\xA0\x38\xA3\x1D\x48\xCF\x29" - "\x0B\xBC\xD8\x50\x99\x1A\x26\xFB" - "\x8E\x75\x3D\x73\xEB\x6A\xED\x29" - "\xE0\x8E\xED\xFC\xFE\x6F\xF6\xBA" - "\x41\xE2\x10\x4C\x01\x8B\x69\x2B" - "\x25\x3F\x4D\x70\x7B\x92\xD6\x3B" - "\xAC\xF9\x77\x18\xD9\x6A\x30\xA6" - "\x2E\xFA\x30\xFF\xC8\xD5\x1D\x06" - "\x59\x28\x1D\x86\x43\x04\x5D\x3B" - "\x99\x4C\x04\x5A\x21\x17\x8B\x76" - "\x8F\x72\xCB\xA1\x9C\x29\x4C\xC3" - "\x65\xA2\x58\x2A\xC5\x66\x24\xBF" - "\xBA\xE6\x0C\xDD\x34\x24\x74\xC8" - "\x84\x0A\x66\x2C\xBE\x8F\x32\xA9" - "\xE7\xE4\xA1\xD7\xDA\xAB\x23\x1E" - "\xEB\xEE\x6C\x94\x6F\x9C\x2E\xD1" - "\x49\x2C\xF3\xD4\x90\xCC\x93\x4C" - "\x84\x52\x6D\x68\xDE\xC6\x64\xB2" - "\x11\x74\x93\x57\xB4\x7E\xC6\x00", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast6_cbc_enc_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\xDF\x77\x68\x96\xC7\xBA\xF8\xE2" - "\x0E\x24\x99\x1A\xAA\xF3\xC6\x9F" - "\xA0\x73\xB3\x70\xC3\x68\x64\x70" - "\xAD\x33\x02\xFB\x88\x74\xAA\x78" - "\xC7\x47\x1A\x18\x61\x2D\xAC\x9F" - "\x7E\x6F\xDF\x05\x13\x76\xA6\x72" - "\xB7\x13\x09\x0F\x7D\x38\xDF\x25" - "\x4E\xFD\x50\x45\xFA\x35\x6A\xC0" - "\x57\x95\xE1\x21\x26\x10\x9A\x21" - "\xA1\x8A\x51\x05\xD1\xB1\x78\x35" - "\x98\xF5\xAE\xC0\xC1\x8B\x94\xFF" - "\xD0\x69\x3F\x42\xC2\x01\xA7\x9B" - "\x23\x16\x47\x72\x81\x13\x3A\x72" - "\xEC\xD9\x40\x88\x00\x9C\xB0\xA8" - "\x9C\xAC\xCE\x11\x73\x7B\x63\x3E" - "\xA3\x63\x98\x7D\x35\xE4\xD9\x83" - "\xE2\xD0\x52\x87\x0C\x1F\xB0\xB3" - "\x41\x1A\x93\x8D\x76\x31\x9F\xF2" - "\xFE\x09\xA3\x8F\x22\x6A\x3B\xB9" - "\x6C\x9E\xE4\xA1\xA0\xC4\xE7\xA1" - "\x21\x9C\x1A\xCA\x65\xDE\x44\x03" - "\x99\xF2\xD2\x39\xE3\x3F\x0F\x37" - "\x53\x50\x23\xA4\x81\x6E\xDA\xFB" - "\xF8\x7B\x01\xD7\xB2\x32\x9C\xB8" - "\xB1\x0E\x99\x17\xB5\x38\xF9\xD7" - "\x86\x2D\x6E\x94\x5C\x99\x9D\xB3" - "\xD3\x63\x4B\x2A\x7D\x44\x6A\xB2" - "\xC1\x03\xE6\x5A\x37\xD8\x64\x18" - "\xAA\x32\xCE\x29\xED\xC0\xA2\xCB" - "\x8D\xAF\xCD\xBE\x8F\xB6\xEC\xB4" - "\x89\x05\x81\x6E\x71\x4F\xC3\x28" - "\x10\xC1\x62\xC4\x41\xE9\xD2\x39" - "\xF3\x22\x39\x12\x2C\xC2\x95\x2D" - "\xBF\x93\x58\x4B\x04\xD1\x8D\x57" - "\xAE\xEB\x60\x03\x56\x35\xAD\x5A" - "\xE9\xC3\xFF\x4E\x31\xE1\x37\xF8" - "\x7D\xEE\x65\x8A\xB6\x88\x1A\x3E" - "\x07\x09\x82\xBA\xF0\x80\x8A\xD0" - "\xA0\x3F\x6A\xE9\x24\x87\x19\x65" - "\x73\x3F\x12\x91\x47\x54\xBA\x39" - "\x30\x5B\x1E\xE5\xC2\xF9\x3F\xEF" - "\xD6\x75\xF9\xB8\x7C\x8B\x05\x76" - "\xEE\xB7\x08\x25\x4B\xB6\x7B\x47" - "\x72\xC0\x4C\xD4\xDA\xE0\x75\xF1" - "\x7C\xE8\x94\x9E\x16\x6E\xB8\x12" - "\xA1\xC1\x6E\x3B\x1C\x59\x41\x2D" - "\x23\xFA\x7D\x77\xB8\x46\x75\xFE" - "\x4F\x10\xD3\x09\x60\xA1\x36\x96" - "\x5B\xC2\xDC\x6E\x84\x7D\x9B\x14" - "\x80\x21\x83\x58\x3C\x76\xFD\x28" - "\x1D\xF9\x93\x13\xD7\x0E\x62\x14" - "\x5A\xC5\x4E\x08\xA5\x56\xA4\x3C" - "\x68\x93\x44\x70\xDF\xCF\x4A\x51" - "\x0B\x81\x29\x41\xE5\x62\x4D\x36" - "\xB3\xEA\x94\xA6\xB9\xDD\x3F\x09" - "\x62\x34\xA0\x6A\x7E\x7D\xF5\xF6" - "\x01\x91\xB4\x27\xDA\x59\xD6\x17" - "\x56\x4D\x82\x62\x37\xA3\x48\x01" - "\x99\x91\x77\xB2\x08\x6B\x2C\x37" - "\xC5\x5C\xAD\xB6\x07\xB6\x84\xF3" - "\x4D\x59\x7D\xC5\x28\x69\xFA\x92" - "\x22\x46\x89\x2D\x0F\x2B\x08\x24", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast6_cbc_dec_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\xDF\x77\x68\x96\xC7\xBA\xF8\xE2" - "\x0E\x24\x99\x1A\xAA\xF3\xC6\x9F" - "\xA0\x73\xB3\x70\xC3\x68\x64\x70" - "\xAD\x33\x02\xFB\x88\x74\xAA\x78" - "\xC7\x47\x1A\x18\x61\x2D\xAC\x9F" - "\x7E\x6F\xDF\x05\x13\x76\xA6\x72" - "\xB7\x13\x09\x0F\x7D\x38\xDF\x25" - "\x4E\xFD\x50\x45\xFA\x35\x6A\xC0" - "\x57\x95\xE1\x21\x26\x10\x9A\x21" - "\xA1\x8A\x51\x05\xD1\xB1\x78\x35" - "\x98\xF5\xAE\xC0\xC1\x8B\x94\xFF" - "\xD0\x69\x3F\x42\xC2\x01\xA7\x9B" - "\x23\x16\x47\x72\x81\x13\x3A\x72" - "\xEC\xD9\x40\x88\x00\x9C\xB0\xA8" - "\x9C\xAC\xCE\x11\x73\x7B\x63\x3E" - "\xA3\x63\x98\x7D\x35\xE4\xD9\x83" - "\xE2\xD0\x52\x87\x0C\x1F\xB0\xB3" - "\x41\x1A\x93\x8D\x76\x31\x9F\xF2" - "\xFE\x09\xA3\x8F\x22\x6A\x3B\xB9" - "\x6C\x9E\xE4\xA1\xA0\xC4\xE7\xA1" - "\x21\x9C\x1A\xCA\x65\xDE\x44\x03" - "\x99\xF2\xD2\x39\xE3\x3F\x0F\x37" - "\x53\x50\x23\xA4\x81\x6E\xDA\xFB" - "\xF8\x7B\x01\xD7\xB2\x32\x9C\xB8" - "\xB1\x0E\x99\x17\xB5\x38\xF9\xD7" - "\x86\x2D\x6E\x94\x5C\x99\x9D\xB3" - "\xD3\x63\x4B\x2A\x7D\x44\x6A\xB2" - "\xC1\x03\xE6\x5A\x37\xD8\x64\x18" - "\xAA\x32\xCE\x29\xED\xC0\xA2\xCB" - "\x8D\xAF\xCD\xBE\x8F\xB6\xEC\xB4" - "\x89\x05\x81\x6E\x71\x4F\xC3\x28" - "\x10\xC1\x62\xC4\x41\xE9\xD2\x39" - "\xF3\x22\x39\x12\x2C\xC2\x95\x2D" - "\xBF\x93\x58\x4B\x04\xD1\x8D\x57" - "\xAE\xEB\x60\x03\x56\x35\xAD\x5A" - "\xE9\xC3\xFF\x4E\x31\xE1\x37\xF8" - "\x7D\xEE\x65\x8A\xB6\x88\x1A\x3E" - "\x07\x09\x82\xBA\xF0\x80\x8A\xD0" - "\xA0\x3F\x6A\xE9\x24\x87\x19\x65" - "\x73\x3F\x12\x91\x47\x54\xBA\x39" - "\x30\x5B\x1E\xE5\xC2\xF9\x3F\xEF" - "\xD6\x75\xF9\xB8\x7C\x8B\x05\x76" - "\xEE\xB7\x08\x25\x4B\xB6\x7B\x47" - "\x72\xC0\x4C\xD4\xDA\xE0\x75\xF1" - "\x7C\xE8\x94\x9E\x16\x6E\xB8\x12" - "\xA1\xC1\x6E\x3B\x1C\x59\x41\x2D" - "\x23\xFA\x7D\x77\xB8\x46\x75\xFE" - "\x4F\x10\xD3\x09\x60\xA1\x36\x96" - "\x5B\xC2\xDC\x6E\x84\x7D\x9B\x14" - "\x80\x21\x83\x58\x3C\x76\xFD\x28" - "\x1D\xF9\x93\x13\xD7\x0E\x62\x14" - "\x5A\xC5\x4E\x08\xA5\x56\xA4\x3C" - "\x68\x93\x44\x70\xDF\xCF\x4A\x51" - "\x0B\x81\x29\x41\xE5\x62\x4D\x36" - "\xB3\xEA\x94\xA6\xB9\xDD\x3F\x09" - "\x62\x34\xA0\x6A\x7E\x7D\xF5\xF6" - "\x01\x91\xB4\x27\xDA\x59\xD6\x17" - "\x56\x4D\x82\x62\x37\xA3\x48\x01" - "\x99\x91\x77\xB2\x08\x6B\x2C\x37" - "\xC5\x5C\xAD\xB6\x07\xB6\x84\xF3" - "\x4D\x59\x7D\xC5\x28\x69\xFA\x92" - "\x22\x46\x89\x2D\x0F\x2B\x08\x24", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast6_ctr_enc_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A", - .ilen = 17, - .result = "\x26\x0A\xF1\xE2\x3F\x8A\xEF\xA3" - "\x53\x9A\x5E\x1B\x2A\x1A\xC6\x0A" - "\x57", - .rlen = 17, - }, { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\x26\x0A\xF1\xE2\x3F\x8A\xEF\xA3" - "\x53\x9A\x5E\x1B\x2A\x1A\xC6\x0A" - "\x57\xA3\xEF\x47\x2A\xE8\x88\xA7" - "\x3C\xD0\xEC\xB9\x94\x50\x7D\x56" - "\xBC\xE1\xC1\xF5\xE1\xEE\x12\xF8" - "\x4F\x03\x82\x3A\x93\x6B\x4C\xD3" - "\xE3\xF3\xFA\xC2\x23\x55\x98\x20" - "\x49\x76\x9B\x6B\xC1\x23\xBF\xE5" - "\xD4\xC4\x2F\x61\xE1\x67\x2A\x30" - "\x6F\x29\xCA\x54\xF8\x1B\xA6\x7D" - "\x66\x45\xEE\xC8\x19\xBE\x50\xF0" - "\x5F\x65\xF8\x1E\x4D\x07\x87\xD9" - "\xD3\xD9\x1B\x09\x89\xFD\x42\xC5" - "\xDB\xEB\x86\xF1\x67\x04\x0F\x5C" - "\x81\xDF\x82\x12\xC7\x4C\x1B\x07" - "\xDE\xE6\xFA\x29\x86\xD1\xB0\xBA" - "\x3D\x6A\x69\x76\xEC\x0F\xB4\xE6" - "\xCD\xA7\xF8\xA8\xB8\xE0\x33\xF5" - "\x49\x61\x22\x52\x64\x8C\x46\x41" - "\x1F\x48\x5F\x4F\xA2\x89\x36\x17" - "\x20\xF8\x2F\x8F\x4B\xFA\xF2\xC0" - "\x1E\x18\xA2\xF8\xB7\x6D\x98\xE3" - "\x00\x14\x15\x59\xC1\x30\x64\xAF" - "\xA8\x01\x38\xAB\xD4\x8B\xEC\x7C" - "\x44\x9A\xC6\x2C\x2E\x2B\x2B\xF4" - "\x02\x37\xC4\x69\xEF\x36\xC1\xF3" - "\xA0\xFB\xFE\x29\xAD\x39\xCF\xD0" - "\x51\x73\xA3\x22\x42\x41\xAB\xD2" - "\x0F\x50\x14\xB9\x54\xD3\xD4\xFA" - "\xBF\xC9\xBB\xCE\xC4\x1D\x2D\xAF" - "\xC9\x3F\x07\x87\x42\x4B\x3A\x54" - "\x34\x8E\x37\xA3\x03\x6F\x65\x66" - "\xDB\x44\xC3\xE8\xD7\xDD\x7D\xDD" - "\x61\xB4\x2B\x80\xA3\x98\x13\xF5" - "\x5A\xD3\x34\x58\xC3\x6E\xF6\xB8" - "\x0A\xC6\x50\x01\x8E\xD5\x6C\x7D" - "\xFE\x16\xB6\xCF\xFC\x51\x40\xAE" - "\xB3\x15\xAC\x90\x6F\x0B\x28\x3A" - "\x60\x40\x38\x90\x20\x46\xC7\xB3" - "\x0B\x12\x6D\x3B\x15\x14\xF9\xF4" - "\x11\x41\x76\x6B\xB3\x60\x82\x3C" - "\x84\xFB\x08\x2E\x92\x25\xCB\x79" - "\x6F\x58\xC5\x94\x00\x00\x47\xB6" - "\x9E\xDC\x0F\x29\x70\x46\x20\x76" - "\x65\x75\x66\x5C\x00\x96\xB3\xE1" - "\x0B\xA7\x11\x8B\x2E\x61\x4E\x45" - "\x73\xFC\x91\xAB\x79\x41\x23\x14" - "\x13\xB6\x72\x6C\x46\xB3\x03\x11" - "\xE4\xF1\xEE\xC9\x7A\xCF\x96\x32" - "\xB6\xF0\x8B\x97\xB4\xCF\x82\xB7" - "\x15\x48\x44\x99\x09\xF6\xE0\xD7" - "\xBC\xF1\x5B\x91\x4F\x30\x22\xA2" - "\x45\xC4\x68\x55\xC2\xBE\xA7\xD2" - "\x12\x53\x35\x9C\xF9\xE7\x35\x5D" - "\x81\xE4\x86\x42\xC3\x58\xFB\xF0" - "\x38\x9B\x8E\x5A\xEF\x83\x33\x0F" - "\x00\x4E\x3F\x9F\xF5\x84\x62\xC4" - "\x19\x35\x88\x22\x45\x59\x0E\x8F" - "\xEC\x27\xDD\x4A\xA4\x1F\xBC\x41" - "\x9B\x66\x8D\x32\xBA\x81\x34\x87" - "\x0E\x74\x33\x30\x62\xB9\x89\xDF" - "\xF9\xC5\xDD\x27\xB3\x39\xCB\xCB", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast6_ctr_dec_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x26\x0A\xF1\xE2\x3F\x8A\xEF\xA3" - "\x53\x9A\x5E\x1B\x2A\x1A\xC6\x0A" - "\x57", - .ilen = 17, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A", - .rlen = 17, - }, { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x26\x0A\xF1\xE2\x3F\x8A\xEF\xA3" - "\x53\x9A\x5E\x1B\x2A\x1A\xC6\x0A" - "\x57\xA3\xEF\x47\x2A\xE8\x88\xA7" - "\x3C\xD0\xEC\xB9\x94\x50\x7D\x56" - "\xBC\xE1\xC1\xF5\xE1\xEE\x12\xF8" - "\x4F\x03\x82\x3A\x93\x6B\x4C\xD3" - "\xE3\xF3\xFA\xC2\x23\x55\x98\x20" - "\x49\x76\x9B\x6B\xC1\x23\xBF\xE5" - "\xD4\xC4\x2F\x61\xE1\x67\x2A\x30" - "\x6F\x29\xCA\x54\xF8\x1B\xA6\x7D" - "\x66\x45\xEE\xC8\x19\xBE\x50\xF0" - "\x5F\x65\xF8\x1E\x4D\x07\x87\xD9" - "\xD3\xD9\x1B\x09\x89\xFD\x42\xC5" - "\xDB\xEB\x86\xF1\x67\x04\x0F\x5C" - "\x81\xDF\x82\x12\xC7\x4C\x1B\x07" - "\xDE\xE6\xFA\x29\x86\xD1\xB0\xBA" - "\x3D\x6A\x69\x76\xEC\x0F\xB4\xE6" - "\xCD\xA7\xF8\xA8\xB8\xE0\x33\xF5" - "\x49\x61\x22\x52\x64\x8C\x46\x41" - "\x1F\x48\x5F\x4F\xA2\x89\x36\x17" - "\x20\xF8\x2F\x8F\x4B\xFA\xF2\xC0" - "\x1E\x18\xA2\xF8\xB7\x6D\x98\xE3" - "\x00\x14\x15\x59\xC1\x30\x64\xAF" - "\xA8\x01\x38\xAB\xD4\x8B\xEC\x7C" - "\x44\x9A\xC6\x2C\x2E\x2B\x2B\xF4" - "\x02\x37\xC4\x69\xEF\x36\xC1\xF3" - "\xA0\xFB\xFE\x29\xAD\x39\xCF\xD0" - "\x51\x73\xA3\x22\x42\x41\xAB\xD2" - "\x0F\x50\x14\xB9\x54\xD3\xD4\xFA" - "\xBF\xC9\xBB\xCE\xC4\x1D\x2D\xAF" - "\xC9\x3F\x07\x87\x42\x4B\x3A\x54" - "\x34\x8E\x37\xA3\x03\x6F\x65\x66" - "\xDB\x44\xC3\xE8\xD7\xDD\x7D\xDD" - "\x61\xB4\x2B\x80\xA3\x98\x13\xF5" - "\x5A\xD3\x34\x58\xC3\x6E\xF6\xB8" - "\x0A\xC6\x50\x01\x8E\xD5\x6C\x7D" - "\xFE\x16\xB6\xCF\xFC\x51\x40\xAE" - "\xB3\x15\xAC\x90\x6F\x0B\x28\x3A" - "\x60\x40\x38\x90\x20\x46\xC7\xB3" - "\x0B\x12\x6D\x3B\x15\x14\xF9\xF4" - "\x11\x41\x76\x6B\xB3\x60\x82\x3C" - "\x84\xFB\x08\x2E\x92\x25\xCB\x79" - "\x6F\x58\xC5\x94\x00\x00\x47\xB6" - "\x9E\xDC\x0F\x29\x70\x46\x20\x76" - "\x65\x75\x66\x5C\x00\x96\xB3\xE1" - "\x0B\xA7\x11\x8B\x2E\x61\x4E\x45" - "\x73\xFC\x91\xAB\x79\x41\x23\x14" - "\x13\xB6\x72\x6C\x46\xB3\x03\x11" - "\xE4\xF1\xEE\xC9\x7A\xCF\x96\x32" - "\xB6\xF0\x8B\x97\xB4\xCF\x82\xB7" - "\x15\x48\x44\x99\x09\xF6\xE0\xD7" - "\xBC\xF1\x5B\x91\x4F\x30\x22\xA2" - "\x45\xC4\x68\x55\xC2\xBE\xA7\xD2" - "\x12\x53\x35\x9C\xF9\xE7\x35\x5D" - "\x81\xE4\x86\x42\xC3\x58\xFB\xF0" - "\x38\x9B\x8E\x5A\xEF\x83\x33\x0F" - "\x00\x4E\x3F\x9F\xF5\x84\x62\xC4" - "\x19\x35\x88\x22\x45\x59\x0E\x8F" - "\xEC\x27\xDD\x4A\xA4\x1F\xBC\x41" - "\x9B\x66\x8D\x32\xBA\x81\x34\x87" - "\x0E\x74\x33\x30\x62\xB9\x89\xDF" - "\xF9\xC5\xDD\x27\xB3\x39\xCB\xCB", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast6_lrw_enc_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" - "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" - "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" - "\x50\x38\x1f\x71\x49\xb6\x57\xd6" - "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" - "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" - "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" - "\xda\x10\x8e\xed\xa2\xa4\x87\xab" - "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" - "\xc9\xac\x42\x31\x95\x7c\xc9\x04" - "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" - "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" - "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" - "\x4c\x96\x12\xed\x7c\x92\x03\x01" - "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" - "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" - "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" - "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" - "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" - "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" - "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" - "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" - "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" - "\x76\x12\x73\x44\x1a\x56\xd7\x72" - "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" - "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" - "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" - "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" - "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" - "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" - "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" - "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" - "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" - "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" - "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" - "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" - "\x8d\x23\x31\x74\x84\xeb\x88\x6e" - "\xcc\xb9\xbc\x22\x83\x19\x07\x22" - "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" - "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" - "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" - "\x3c\xce\x8f\x42\x60\x71\xa7\x75" - "\x08\x40\x65\x8a\x82\xbf\xf5\x43" - "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" - "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" - "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" - "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" - "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" - "\x62\x73\x65\xfd\x46\x63\x25\x3d" - "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" - "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" - "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" - "\xc5\x68\x77\x84\x32\x2b\xcc\x85" - "\x74\x96\xf0\x12\x77\x61\xb9\xeb" - "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" - "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" - "\xda\x39\x87\x45\xc0\x2b\xbb\x01" - "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" - "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" - "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" - "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" - "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" - "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" - "\x21\xc4\xc2\x75\x67\x89\x37\x0a", - .ilen = 512, - .result = "\x55\x25\x09\x8B\xB5\xD5\xF8\xBF" - "\x37\x4A\xFE\x3C\x47\xD8\xE6\xEB" - "\xCA\xA4\x9B\xB0\xAB\x6D\x64\xCA" - "\x58\xB6\x73\xF0\xD7\x52\x34\xEF" - "\xFB\x3E\x96\x81\xB7\x71\x34\xA4" - "\x55\x20\xBE\x39\x5A\x2B\xF9\xD1" - "\x65\x0B\xDA\xD3\x7E\xB3\xA6\xF7" - "\x2E\x0B\x5A\x52\xDB\x39\x8C\x9B" - "\x61\x17\x5F\xAF\xB6\x5A\xC8\x08" - "\xA7\xB7\x2A\x11\x7C\x97\x38\x9D" - "\x59\x0E\x66\x59\x5E\xD8\x8B\xCE" - "\x70\xE0\xC3\x42\xB0\x8C\x0F\xBA" - "\xB2\x0D\x81\xB6\xBE\x61\x1C\x2D" - "\x7E\xEA\x91\x25\xAC\xEC\xF8\x28" - "\x80\x1D\xF0\x30\xBA\x62\x77\x7D" - "\xDB\x15\x69\xDF\xFA\x2A\x81\x64" - "\x95\x5B\xA4\x7F\x3E\x4F\xE3\x30" - "\xB0\x5C\xC2\x05\xF8\xF0\x29\xE7" - "\x0A\xA0\x66\xB2\x5D\x0F\x39\x2B" - "\xB4\xB3\x00\xA9\xD0\xAB\x63\x61" - "\x5E\xDB\xFC\x11\x74\x25\x96\x65" - "\xE8\xE2\x34\x57\x77\x15\x5E\x70" - "\xFF\x10\x90\xC3\x64\xF0\x11\x0A" - "\x63\x3A\xD3\x55\x92\x15\x4B\x0C" - "\xC7\x08\x89\x17\x3B\x99\xAD\x63" - "\xE7\x06\xDF\x52\xBC\x15\x64\x45" - "\x9D\x7A\xFB\x69\xBC\x2D\x6E\xA9" - "\x35\xD9\xD8\xF5\x0C\xC4\xA2\x23" - "\x9C\x18\x8B\xA8\x8C\xFE\xF8\x0E" - "\xBD\xAB\x60\x1A\x51\x17\x54\x27" - "\xB6\xE8\xBE\x0F\xA9\xA5\x82\x19" - "\x2F\x6F\x20\xA7\x47\xED\x74\x6C" - "\x4E\xC1\xF8\x8C\x14\xF3\xBB\x1F" - "\xED\x4D\x8F\x7C\x37\xEF\x19\xA1" - "\x07\x16\xDE\x76\xCC\x5E\x94\x02" - "\xFB\xBF\xE4\x81\x50\xCE\xFC\x0F" - "\x9E\xCF\x3D\xF6\x67\x00\xBF\xA7" - "\x6E\x21\x58\x36\x06\xDE\xB3\xD4" - "\xA2\xFA\xD8\x4E\xE0\xB9\x7F\x23" - "\x51\x21\x2B\x32\x68\xAA\xF8\xA8" - "\x93\x08\xB5\x6D\xE6\x43\x2C\xB7" - "\x31\xB2\x0F\xD0\xA2\x51\xC0\x25" - "\x30\xC7\x10\x3F\x97\x27\x01\x8E" - "\xFA\xD8\x4F\x78\xD8\x2E\x1D\xEB" - "\xA1\x37\x52\x0F\x7B\x5E\x87\xA8" - "\x22\xE2\xE6\x92\xA7\x5F\x11\x32" - "\xCC\x93\x34\xFC\xD1\x7E\xAE\x54" - "\xBC\x6A\x1B\x91\xD1\x2E\x21\xEC" - "\x5D\xF1\xC4\xF1\x55\x20\xBF\xE5" - "\x96\x3D\x69\x91\x20\x4E\xF2\x61" - "\xDA\x77\xFE\xEE\xC3\x74\x57\x2A" - "\x78\x39\xB0\xE0\xCF\x12\x56\xD6" - "\x05\xDC\xF9\x19\x66\x44\x1D\xF9" - "\x82\x37\xD4\xC2\x60\xB6\x31\xDF" - "\x0C\xAF\xBC\x8B\x55\x9A\xC8\x2D" - "\xAB\xA7\x88\x7B\x41\xE8\x29\xC9" - "\x9B\x8D\xA7\x00\x86\x25\xB6\x14" - "\xF5\x13\x73\xD7\x4B\x6B\x83\xF3" - "\xAF\x96\x00\xE4\xB7\x3C\x65\xA6" - "\x15\xB7\x94\x7D\x4E\x70\x4C\x75" - "\xF3\xB4\x02\xA9\x17\x1C\x7A\x0A" - "\xC0\xD5\x33\x11\x56\xDE\xDC\xF5" - "\x8D\xD9\xCD\x3B\x22\x67\x18\xC7" - "\xC4\xF5\x99\x61\xBC\xBB\x5B\x46", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast6_lrw_dec_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x55\x25\x09\x8B\xB5\xD5\xF8\xBF" - "\x37\x4A\xFE\x3C\x47\xD8\xE6\xEB" - "\xCA\xA4\x9B\xB0\xAB\x6D\x64\xCA" - "\x58\xB6\x73\xF0\xD7\x52\x34\xEF" - "\xFB\x3E\x96\x81\xB7\x71\x34\xA4" - "\x55\x20\xBE\x39\x5A\x2B\xF9\xD1" - "\x65\x0B\xDA\xD3\x7E\xB3\xA6\xF7" - "\x2E\x0B\x5A\x52\xDB\x39\x8C\x9B" - "\x61\x17\x5F\xAF\xB6\x5A\xC8\x08" - "\xA7\xB7\x2A\x11\x7C\x97\x38\x9D" - "\x59\x0E\x66\x59\x5E\xD8\x8B\xCE" - "\x70\xE0\xC3\x42\xB0\x8C\x0F\xBA" - "\xB2\x0D\x81\xB6\xBE\x61\x1C\x2D" - "\x7E\xEA\x91\x25\xAC\xEC\xF8\x28" - "\x80\x1D\xF0\x30\xBA\x62\x77\x7D" - "\xDB\x15\x69\xDF\xFA\x2A\x81\x64" - "\x95\x5B\xA4\x7F\x3E\x4F\xE3\x30" - "\xB0\x5C\xC2\x05\xF8\xF0\x29\xE7" - "\x0A\xA0\x66\xB2\x5D\x0F\x39\x2B" - "\xB4\xB3\x00\xA9\xD0\xAB\x63\x61" - "\x5E\xDB\xFC\x11\x74\x25\x96\x65" - "\xE8\xE2\x34\x57\x77\x15\x5E\x70" - "\xFF\x10\x90\xC3\x64\xF0\x11\x0A" - "\x63\x3A\xD3\x55\x92\x15\x4B\x0C" - "\xC7\x08\x89\x17\x3B\x99\xAD\x63" - "\xE7\x06\xDF\x52\xBC\x15\x64\x45" - "\x9D\x7A\xFB\x69\xBC\x2D\x6E\xA9" - "\x35\xD9\xD8\xF5\x0C\xC4\xA2\x23" - "\x9C\x18\x8B\xA8\x8C\xFE\xF8\x0E" - "\xBD\xAB\x60\x1A\x51\x17\x54\x27" - "\xB6\xE8\xBE\x0F\xA9\xA5\x82\x19" - "\x2F\x6F\x20\xA7\x47\xED\x74\x6C" - "\x4E\xC1\xF8\x8C\x14\xF3\xBB\x1F" - "\xED\x4D\x8F\x7C\x37\xEF\x19\xA1" - "\x07\x16\xDE\x76\xCC\x5E\x94\x02" - "\xFB\xBF\xE4\x81\x50\xCE\xFC\x0F" - "\x9E\xCF\x3D\xF6\x67\x00\xBF\xA7" - "\x6E\x21\x58\x36\x06\xDE\xB3\xD4" - "\xA2\xFA\xD8\x4E\xE0\xB9\x7F\x23" - "\x51\x21\x2B\x32\x68\xAA\xF8\xA8" - "\x93\x08\xB5\x6D\xE6\x43\x2C\xB7" - "\x31\xB2\x0F\xD0\xA2\x51\xC0\x25" - "\x30\xC7\x10\x3F\x97\x27\x01\x8E" - "\xFA\xD8\x4F\x78\xD8\x2E\x1D\xEB" - "\xA1\x37\x52\x0F\x7B\x5E\x87\xA8" - "\x22\xE2\xE6\x92\xA7\x5F\x11\x32" - "\xCC\x93\x34\xFC\xD1\x7E\xAE\x54" - "\xBC\x6A\x1B\x91\xD1\x2E\x21\xEC" - "\x5D\xF1\xC4\xF1\x55\x20\xBF\xE5" - "\x96\x3D\x69\x91\x20\x4E\xF2\x61" - "\xDA\x77\xFE\xEE\xC3\x74\x57\x2A" - "\x78\x39\xB0\xE0\xCF\x12\x56\xD6" - "\x05\xDC\xF9\x19\x66\x44\x1D\xF9" - "\x82\x37\xD4\xC2\x60\xB6\x31\xDF" - "\x0C\xAF\xBC\x8B\x55\x9A\xC8\x2D" - "\xAB\xA7\x88\x7B\x41\xE8\x29\xC9" - "\x9B\x8D\xA7\x00\x86\x25\xB6\x14" - "\xF5\x13\x73\xD7\x4B\x6B\x83\xF3" - "\xAF\x96\x00\xE4\xB7\x3C\x65\xA6" - "\x15\xB7\x94\x7D\x4E\x70\x4C\x75" - "\xF3\xB4\x02\xA9\x17\x1C\x7A\x0A" - "\xC0\xD5\x33\x11\x56\xDE\xDC\xF5" - "\x8D\xD9\xCD\x3B\x22\x67\x18\xC7" - "\xC4\xF5\x99\x61\xBC\xBB\x5B\x46", - .ilen = 512, - .result = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" - "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" - "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" - "\x50\x38\x1f\x71\x49\xb6\x57\xd6" - "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" - "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" - "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" - "\xda\x10\x8e\xed\xa2\xa4\x87\xab" - "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" - "\xc9\xac\x42\x31\x95\x7c\xc9\x04" - "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" - "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" - "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" - "\x4c\x96\x12\xed\x7c\x92\x03\x01" - "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" - "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" - "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" - "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" - "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" - "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" - "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" - "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" - "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" - "\x76\x12\x73\x44\x1a\x56\xd7\x72" - "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" - "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" - "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" - "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" - "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" - "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" - "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" - "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" - "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" - "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" - "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" - "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" - "\x8d\x23\x31\x74\x84\xeb\x88\x6e" - "\xcc\xb9\xbc\x22\x83\x19\x07\x22" - "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" - "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" - "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" - "\x3c\xce\x8f\x42\x60\x71\xa7\x75" - "\x08\x40\x65\x8a\x82\xbf\xf5\x43" - "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" - "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" - "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" - "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" - "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" - "\x62\x73\x65\xfd\x46\x63\x25\x3d" - "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" - "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" - "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" - "\xc5\x68\x77\x84\x32\x2b\xcc\x85" - "\x74\x96\xf0\x12\x77\x61\xb9\xeb" - "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" - "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" - "\xda\x39\x87\x45\xc0\x2b\xbb\x01" - "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" - "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" - "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" - "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" - "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" - "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" - "\x21\xc4\xc2\x75\x67\x89\x37\x0a", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast6_xts_enc_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\xDE\x6F\x22\xA5\xE8\x39\xE8\x78" - "\x88\x5A\x4F\x8D\x82\x76\x52\x6D" - "\xB2\x41\x16\xF4\x2B\xA6\xEB\xF6" - "\xE2\xC5\x62\x8D\x61\xA1\x01\xED" - "\xD9\x38\x01\xC1\x43\x63\x4E\x88" - "\xC9\x4B\x5A\x88\x80\xB7\x5C\x71" - "\x47\xEE\x11\xD8\xB7\x2D\x5D\x13" - "\x1A\xB1\x68\x5B\x61\xA7\xA9\x81" - "\x8B\x83\xA1\x6A\xAA\x36\xD6\xB6" - "\x60\x54\x09\x32\xFE\x6A\x76\x2E" - "\x28\xFF\xD5\xD6\xDD\x1D\x45\x7D" - "\xF0\x8B\xF3\x32\x4E\x6C\x12\xCB" - "\xB8\x25\x70\xF8\x40\xBC\x90\x1B" - "\x11\xC3\x59\xAF\xF0\x2F\x92\xDD" - "\xD3\x3B\xCF\x60\xA1\x78\x94\x57" - "\xAF\x76\xC1\x67\xA6\x3C\xCD\x98" - "\xB1\xF7\x27\xB9\xA3\xBD\x10\xEA" - "\xCD\x8B\xC2\xF2\x14\xF2\xB2\x67" - "\x05\xDD\x1D\x58\x6E\x2F\x95\x08" - "\x3A\xF8\x78\x76\x82\x56\xA7\xEC" - "\x51\x4B\x85\x77\xC2\x4C\x4A\x34" - "\x71\x38\x17\x91\x44\xE8\xFC\x65" - "\x99\x0D\x52\x91\xEE\xF8\xEF\x27" - "\x2A\x9E\x6E\x78\xC4\x26\x87\xF4" - "\x8A\xF0\x2D\x04\xE8\x14\x92\x5D" - "\x59\x22\x9B\x29\x5C\x18\xF0\xC3" - "\x47\xF3\x76\xD8\xE4\xF3\x1B\xD1" - "\x70\xA3\x0D\xB5\x70\x02\x1D\xA3" - "\x91\x3B\x49\x73\x18\xAB\xD4\xC9" - "\xC3\x1E\xEF\x1F\xFE\xD5\x59\x8A" - "\xD7\xF6\xC9\x71\x67\x79\xD7\x0E" - "\xBE\x1F\x8E\xEC\x55\x7E\x4F\x24" - "\xE6\x87\xEA\xFE\x96\x25\x67\x8E" - "\x93\x03\xFA\xFF\xCE\xAF\xB2\x3C" - "\x6F\xEB\x57\xFB\xD3\x28\x87\xA9" - "\xCE\xC2\xF5\x9C\xC6\x67\xB5\x97" - "\x49\xF7\x04\xCB\xEF\x84\x98\x33" - "\xAF\x38\xD3\x04\x1C\x24\x71\x38" - "\xC7\x71\xDD\x43\x0D\x12\x4A\x18" - "\xBA\xC4\xAF\xBA\xB2\x5B\xEB\x95" - "\x02\x43\x5D\xCE\x19\xCC\xCD\x66" - "\x91\x0B\x8C\x7F\x51\xC4\xBF\x3C" - "\x8B\xF1\xCC\xAA\x29\xD7\x87\xCB" - "\x3E\xC5\xF3\xC9\x75\xE8\xA3\x5B" - "\x30\x45\xA9\xB7\xAF\x80\x64\x6F" - "\x75\x4A\xA7\xC0\x6D\x19\x6B\xDE" - "\x17\xDE\x6D\xEA\x87\x9F\x95\xAE" - "\xF5\x3C\xEE\x54\xB8\x27\x84\xF8" - "\x97\xA3\xE1\x6F\x38\x24\x34\x88" - "\xCE\xBD\x32\x52\xE0\x00\x6C\x94" - "\xC9\xD7\x5D\x37\x81\x33\x2E\x7F" - "\x4F\x7E\x2E\x0D\x94\xBD\xEA\x59" - "\x34\x39\xA8\x35\x12\xB7\xBC\xAC" - "\xEA\x52\x9C\x78\x02\x6D\x92\x36" - "\xFB\x59\x2B\xA4\xEA\x7B\x1B\x83" - "\xE1\x4D\x5E\x2A\x7E\x92\xB1\x64" - "\xDE\xE0\x27\x4B\x0A\x6F\x4C\xE3" - "\xB0\xEB\x31\xE4\x69\x95\xAB\x35" - "\x8B\x2C\xF5\x6B\x7F\xF1\xA2\x82" - "\xF8\xD9\x47\x82\xA9\x82\x03\x91" - "\x69\x1F\xBE\x4C\xE7\xC7\x34\x2F" - "\x45\x72\x80\x17\x81\xBD\x9D\x62" - "\xA1\xAC\xE8\xCF\xC6\x74\xCF\xDC" - "\x22\x60\x4E\xE8\xA4\x5D\x85\xB9", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast6_xts_dec_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xDE\x6F\x22\xA5\xE8\x39\xE8\x78" - "\x88\x5A\x4F\x8D\x82\x76\x52\x6D" - "\xB2\x41\x16\xF4\x2B\xA6\xEB\xF6" - "\xE2\xC5\x62\x8D\x61\xA1\x01\xED" - "\xD9\x38\x01\xC1\x43\x63\x4E\x88" - "\xC9\x4B\x5A\x88\x80\xB7\x5C\x71" - "\x47\xEE\x11\xD8\xB7\x2D\x5D\x13" - "\x1A\xB1\x68\x5B\x61\xA7\xA9\x81" - "\x8B\x83\xA1\x6A\xAA\x36\xD6\xB6" - "\x60\x54\x09\x32\xFE\x6A\x76\x2E" - "\x28\xFF\xD5\xD6\xDD\x1D\x45\x7D" - "\xF0\x8B\xF3\x32\x4E\x6C\x12\xCB" - "\xB8\x25\x70\xF8\x40\xBC\x90\x1B" - "\x11\xC3\x59\xAF\xF0\x2F\x92\xDD" - "\xD3\x3B\xCF\x60\xA1\x78\x94\x57" - "\xAF\x76\xC1\x67\xA6\x3C\xCD\x98" - "\xB1\xF7\x27\xB9\xA3\xBD\x10\xEA" - "\xCD\x8B\xC2\xF2\x14\xF2\xB2\x67" - "\x05\xDD\x1D\x58\x6E\x2F\x95\x08" - "\x3A\xF8\x78\x76\x82\x56\xA7\xEC" - "\x51\x4B\x85\x77\xC2\x4C\x4A\x34" - "\x71\x38\x17\x91\x44\xE8\xFC\x65" - "\x99\x0D\x52\x91\xEE\xF8\xEF\x27" - "\x2A\x9E\x6E\x78\xC4\x26\x87\xF4" - "\x8A\xF0\x2D\x04\xE8\x14\x92\x5D" - "\x59\x22\x9B\x29\x5C\x18\xF0\xC3" - "\x47\xF3\x76\xD8\xE4\xF3\x1B\xD1" - "\x70\xA3\x0D\xB5\x70\x02\x1D\xA3" - "\x91\x3B\x49\x73\x18\xAB\xD4\xC9" - "\xC3\x1E\xEF\x1F\xFE\xD5\x59\x8A" - "\xD7\xF6\xC9\x71\x67\x79\xD7\x0E" - "\xBE\x1F\x8E\xEC\x55\x7E\x4F\x24" - "\xE6\x87\xEA\xFE\x96\x25\x67\x8E" - "\x93\x03\xFA\xFF\xCE\xAF\xB2\x3C" - "\x6F\xEB\x57\xFB\xD3\x28\x87\xA9" - "\xCE\xC2\xF5\x9C\xC6\x67\xB5\x97" - "\x49\xF7\x04\xCB\xEF\x84\x98\x33" - "\xAF\x38\xD3\x04\x1C\x24\x71\x38" - "\xC7\x71\xDD\x43\x0D\x12\x4A\x18" - "\xBA\xC4\xAF\xBA\xB2\x5B\xEB\x95" - "\x02\x43\x5D\xCE\x19\xCC\xCD\x66" - "\x91\x0B\x8C\x7F\x51\xC4\xBF\x3C" - "\x8B\xF1\xCC\xAA\x29\xD7\x87\xCB" - "\x3E\xC5\xF3\xC9\x75\xE8\xA3\x5B" - "\x30\x45\xA9\xB7\xAF\x80\x64\x6F" - "\x75\x4A\xA7\xC0\x6D\x19\x6B\xDE" - "\x17\xDE\x6D\xEA\x87\x9F\x95\xAE" - "\xF5\x3C\xEE\x54\xB8\x27\x84\xF8" - "\x97\xA3\xE1\x6F\x38\x24\x34\x88" - "\xCE\xBD\x32\x52\xE0\x00\x6C\x94" - "\xC9\xD7\x5D\x37\x81\x33\x2E\x7F" - "\x4F\x7E\x2E\x0D\x94\xBD\xEA\x59" - "\x34\x39\xA8\x35\x12\xB7\xBC\xAC" - "\xEA\x52\x9C\x78\x02\x6D\x92\x36" - "\xFB\x59\x2B\xA4\xEA\x7B\x1B\x83" - "\xE1\x4D\x5E\x2A\x7E\x92\xB1\x64" - "\xDE\xE0\x27\x4B\x0A\x6F\x4C\xE3" - "\xB0\xEB\x31\xE4\x69\x95\xAB\x35" - "\x8B\x2C\xF5\x6B\x7F\xF1\xA2\x82" - "\xF8\xD9\x47\x82\xA9\x82\x03\x91" - "\x69\x1F\xBE\x4C\xE7\xC7\x34\x2F" - "\x45\x72\x80\x17\x81\xBD\x9D\x62" - "\xA1\xAC\xE8\xCF\xC6\x74\xCF\xDC" - "\x22\x60\x4E\xE8\xA4\x5D\x85\xB9", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - - -/* - * AES test vectors. - */ -static const struct cipher_testvec aes_enc_tv_template[] = { - { /* From FIPS-197 */ - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .klen = 16, - .input = "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", - .ilen = 16, - .result = "\x69\xc4\xe0\xd8\x6a\x7b\x04\x30" - "\xd8\xcd\xb7\x80\x70\xb4\xc5\x5a", - .rlen = 16, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17", - .klen = 24, - .input = "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", - .ilen = 16, - .result = "\xdd\xa9\x7c\xa4\x86\x4c\xdf\xe0" - "\x6e\xaf\x70\xa0\xec\x0d\x71\x91", - .rlen = 16, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .klen = 32, - .input = "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", - .ilen = 16, - .result = "\x8e\xa2\xb7\xca\x51\x67\x45\xbf" - "\xea\xfc\x49\x90\x4b\x49\x60\x89", - .rlen = 16, - }, { /* Generated with Crypto++ */ - .key = "\xA6\xC9\x83\xA6\xC9\xEC\x0F\x32" - "\x55\x0F\x32\x55\x78\x9B\xBE\x78" - "\x9B\xBE\xE1\x04\x27\xE1\x04\x27" - "\x4A\x6D\x90\x4A\x6D\x90\xB3\xD6", - .klen = 32, - .input = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" - "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" - "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" - "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" - "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" - "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" - "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" - "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" - "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" - "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" - "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" - "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" - "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" - "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" - "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" - "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" - "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" - "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" - "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" - "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" - "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" - "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" - "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" - "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" - "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" - "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" - "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" - "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" - "\x20\x89\x15\x7E\xE7\x50\xDC\x45" - "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" - "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" - "\xED\x56\xBF\x28\xB4\x1D\x86\x12", - .ilen = 496, - .result = "\x71\x73\xF7\xDB\x24\x93\x21\x6D" - "\x61\x1E\xBB\x63\x42\x79\xDB\x64" - "\x6F\x82\xC0\xCA\xA3\x9B\xFA\x0B" - "\xD9\x08\xC7\x4A\x90\xAE\x8F\x5F" - "\x5E\x06\xF0\x5F\x31\x51\x18\x37" - "\x45\xD7\xCA\x3A\xFD\x6C\x3F\xE1" - "\xDD\x8D\x22\x65\x2B\x00\x50\xCE" - "\xBA\x28\x67\xD7\xCE\x0E\x0D\xEA" - "\x78\x69\x7F\xAE\x8F\x8B\x69\x37" - "\x75\xE0\xDC\x96\xE0\xB7\xF4\x09" - "\xCB\x6D\xA2\xFB\xDA\xAF\x09\xF8" - "\x81\x82\x27\xFA\x45\x9C\x29\xA4" - "\x22\x8B\x78\x69\x5B\x46\xF9\x39" - "\x1B\xCC\xF9\x1D\x09\xEB\xBC\x5C" - "\x41\x72\x51\x97\x1D\x07\x49\xA0" - "\x1B\x8E\x65\x4B\xB2\x6A\x12\x03" - "\x6A\x60\x95\xAC\xBD\xAC\x1A\x64" - "\xDE\x5A\xA5\xF0\x83\x2F\xCB\xCA" - "\x22\x74\xA6\x6C\x9B\x73\xCE\x3F" - "\xE1\x8B\x22\x17\x59\x0C\x47\x89" - "\x33\xA1\xD6\x47\x03\x19\x4F\xA8" - "\x67\x69\xF0\x5B\xF0\x20\xAD\x06" - "\x27\x81\x92\xD8\xC5\xBA\x98\x12" - "\xBE\x24\xB5\x2F\x75\x02\xC2\xAD" - "\x12\x2F\x07\x32\xEE\x39\xAF\x64" - "\x05\x8F\xB3\xD4\xEB\x1B\x46\x6E" - "\xD9\x21\xF9\xC4\xB7\xC9\x45\x68" - "\xB4\xA1\x74\x9F\x82\x47\xEB\xCC" - "\xBD\x0A\x14\x95\x0F\x8B\xA8\x2F" - "\x4B\x1B\xA7\xBF\x82\xA6\x43\x0C" - "\xB9\x39\x4A\xA8\x10\x6F\x50\x7B" - "\x25\xFB\x26\x81\xE0\x2F\xF0\x96" - "\x8D\x8B\xAC\x92\x0F\xF6\xED\x64" - "\x63\x29\x4C\x8E\x18\x13\xC5\xBF" - "\xFC\xA0\xD9\xBF\x7C\x3A\x0E\x29" - "\x6F\xD1\x6C\x6F\xA5\xDA\xBF\xB1" - "\x30\xEA\x44\x2D\xC3\x8F\x16\xE1" - "\x66\xFA\xA3\x21\x3E\xFC\x13\xCA" - "\xF0\xF6\xF0\x59\xBD\x8F\x38\x50" - "\x31\xCB\x69\x3F\x96\x15\xD6\xF5" - "\xAE\xFF\xF6\xAA\x41\x85\x4C\x10" - "\x58\xE3\xF9\x44\xE6\x28\xDA\x9A" - "\xDC\x6A\x80\x34\x73\x97\x1B\xC5" - "\xCA\x26\x16\x77\x0E\x60\xAB\x89" - "\x0F\x04\x27\xBD\xCE\x3E\x71\xB4" - "\xA0\xD7\x22\x7E\xDB\xEB\x24\x70" - "\x42\x71\x51\x78\x70\xB3\xE0\x3D" - "\x84\x8E\x8D\x7B\xD0\x6D\xEA\x92" - "\x11\x08\x42\x4F\xE5\xAD\x26\x92" - "\xD2\x00\xAE\xA8\xE3\x4B\x37\x47" - "\x22\xC1\x95\xC1\x63\x7F\xCB\x03" - "\xF3\xE3\xD7\x9D\x60\xC7\xBC\xEA" - "\x35\xA2\xFD\x45\x52\x39\x13\x6F" - "\xC1\x53\xF3\x53\xDF\x33\x84\xD7" - "\xD2\xC8\x37\xB0\x75\xE3\x41\x46" - "\xB3\xC7\x83\x2E\x8A\xBB\xA4\xE5" - "\x7F\x3C\xFD\x8B\xEB\xEA\x63\xBD" - "\xB7\x46\xE7\xBF\x09\x9C\x0D\x0F" - "\x40\x86\x7F\x51\xE1\x11\x9C\xCB" - "\x88\xE6\x68\x47\xE3\x2B\xC5\xFF" - "\x09\x79\xA0\x43\x5C\x0D\x08\x58" - "\x17\xBB\xC0\x6B\x62\x3F\x56\xE9", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec aes_dec_tv_template[] = { - { /* From FIPS-197 */ - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .klen = 16, - .input = "\x69\xc4\xe0\xd8\x6a\x7b\x04\x30" - "\xd8\xcd\xb7\x80\x70\xb4\xc5\x5a", - .ilen = 16, - .result = "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", - .rlen = 16, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17", - .klen = 24, - .input = "\xdd\xa9\x7c\xa4\x86\x4c\xdf\xe0" - "\x6e\xaf\x70\xa0\xec\x0d\x71\x91", - .ilen = 16, - .result = "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", - .rlen = 16, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .klen = 32, - .input = "\x8e\xa2\xb7\xca\x51\x67\x45\xbf" - "\xea\xfc\x49\x90\x4b\x49\x60\x89", - .ilen = 16, - .result = "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", - .rlen = 16, - }, { /* Generated with Crypto++ */ - .key = "\xA6\xC9\x83\xA6\xC9\xEC\x0F\x32" - "\x55\x0F\x32\x55\x78\x9B\xBE\x78" - "\x9B\xBE\xE1\x04\x27\xE1\x04\x27" - "\x4A\x6D\x90\x4A\x6D\x90\xB3\xD6", - .klen = 32, - .input = "\x71\x73\xF7\xDB\x24\x93\x21\x6D" - "\x61\x1E\xBB\x63\x42\x79\xDB\x64" - "\x6F\x82\xC0\xCA\xA3\x9B\xFA\x0B" - "\xD9\x08\xC7\x4A\x90\xAE\x8F\x5F" - "\x5E\x06\xF0\x5F\x31\x51\x18\x37" - "\x45\xD7\xCA\x3A\xFD\x6C\x3F\xE1" - "\xDD\x8D\x22\x65\x2B\x00\x50\xCE" - "\xBA\x28\x67\xD7\xCE\x0E\x0D\xEA" - "\x78\x69\x7F\xAE\x8F\x8B\x69\x37" - "\x75\xE0\xDC\x96\xE0\xB7\xF4\x09" - "\xCB\x6D\xA2\xFB\xDA\xAF\x09\xF8" - "\x81\x82\x27\xFA\x45\x9C\x29\xA4" - "\x22\x8B\x78\x69\x5B\x46\xF9\x39" - "\x1B\xCC\xF9\x1D\x09\xEB\xBC\x5C" - "\x41\x72\x51\x97\x1D\x07\x49\xA0" - "\x1B\x8E\x65\x4B\xB2\x6A\x12\x03" - "\x6A\x60\x95\xAC\xBD\xAC\x1A\x64" - "\xDE\x5A\xA5\xF0\x83\x2F\xCB\xCA" - "\x22\x74\xA6\x6C\x9B\x73\xCE\x3F" - "\xE1\x8B\x22\x17\x59\x0C\x47\x89" - "\x33\xA1\xD6\x47\x03\x19\x4F\xA8" - "\x67\x69\xF0\x5B\xF0\x20\xAD\x06" - "\x27\x81\x92\xD8\xC5\xBA\x98\x12" - "\xBE\x24\xB5\x2F\x75\x02\xC2\xAD" - "\x12\x2F\x07\x32\xEE\x39\xAF\x64" - "\x05\x8F\xB3\xD4\xEB\x1B\x46\x6E" - "\xD9\x21\xF9\xC4\xB7\xC9\x45\x68" - "\xB4\xA1\x74\x9F\x82\x47\xEB\xCC" - "\xBD\x0A\x14\x95\x0F\x8B\xA8\x2F" - "\x4B\x1B\xA7\xBF\x82\xA6\x43\x0C" - "\xB9\x39\x4A\xA8\x10\x6F\x50\x7B" - "\x25\xFB\x26\x81\xE0\x2F\xF0\x96" - "\x8D\x8B\xAC\x92\x0F\xF6\xED\x64" - "\x63\x29\x4C\x8E\x18\x13\xC5\xBF" - "\xFC\xA0\xD9\xBF\x7C\x3A\x0E\x29" - "\x6F\xD1\x6C\x6F\xA5\xDA\xBF\xB1" - "\x30\xEA\x44\x2D\xC3\x8F\x16\xE1" - "\x66\xFA\xA3\x21\x3E\xFC\x13\xCA" - "\xF0\xF6\xF0\x59\xBD\x8F\x38\x50" - "\x31\xCB\x69\x3F\x96\x15\xD6\xF5" - "\xAE\xFF\xF6\xAA\x41\x85\x4C\x10" - "\x58\xE3\xF9\x44\xE6\x28\xDA\x9A" - "\xDC\x6A\x80\x34\x73\x97\x1B\xC5" - "\xCA\x26\x16\x77\x0E\x60\xAB\x89" - "\x0F\x04\x27\xBD\xCE\x3E\x71\xB4" - "\xA0\xD7\x22\x7E\xDB\xEB\x24\x70" - "\x42\x71\x51\x78\x70\xB3\xE0\x3D" - "\x84\x8E\x8D\x7B\xD0\x6D\xEA\x92" - "\x11\x08\x42\x4F\xE5\xAD\x26\x92" - "\xD2\x00\xAE\xA8\xE3\x4B\x37\x47" - "\x22\xC1\x95\xC1\x63\x7F\xCB\x03" - "\xF3\xE3\xD7\x9D\x60\xC7\xBC\xEA" - "\x35\xA2\xFD\x45\x52\x39\x13\x6F" - "\xC1\x53\xF3\x53\xDF\x33\x84\xD7" - "\xD2\xC8\x37\xB0\x75\xE3\x41\x46" - "\xB3\xC7\x83\x2E\x8A\xBB\xA4\xE5" - "\x7F\x3C\xFD\x8B\xEB\xEA\x63\xBD" - "\xB7\x46\xE7\xBF\x09\x9C\x0D\x0F" - "\x40\x86\x7F\x51\xE1\x11\x9C\xCB" - "\x88\xE6\x68\x47\xE3\x2B\xC5\xFF" - "\x09\x79\xA0\x43\x5C\x0D\x08\x58" - "\x17\xBB\xC0\x6B\x62\x3F\x56\xE9", - .ilen = 496, - .result = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" - "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" - "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" - "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" - "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" - "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" - "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" - "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" - "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" - "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" - "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" - "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" - "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" - "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" - "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" - "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" - "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" - "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" - "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" - "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" - "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" - "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" - "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" - "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" - "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" - "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" - "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" - "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" - "\x20\x89\x15\x7E\xE7\x50\xDC\x45" - "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" - "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" - "\xED\x56\xBF\x28\xB4\x1D\x86\x12", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec aes_cbc_enc_tv_template[] = { - { /* From RFC 3602 */ - .key = "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" - "\x51\x2e\x03\xd5\x34\x12\x00\x06", - .klen = 16, - .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" - "\xb4\x22\xda\x80\x2c\x9f\xac\x41", - .input = "Single block msg", - .ilen = 16, - .result = "\xe3\x53\x77\x9c\x10\x79\xae\xb8" - "\x27\x08\x94\x2d\xbe\x77\x18\x1a", - .rlen = 16, - .also_non_np = 1, - .np = 8, - .tap = { 3, 2, 3, 2, 3, 1, 1, 1 }, - }, { - .key = "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" - "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", - .klen = 16, - .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" - "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .ilen = 32, - .result = "\xd2\x96\xcd\x94\xc2\xcc\xcf\x8a" - "\x3a\x86\x30\x28\xb5\xe1\xdc\x0a" - "\x75\x86\x60\x2d\x25\x3c\xff\xf9" - "\x1b\x82\x66\xbe\xa6\xd6\x1a\xb1", - .rlen = 32, - }, { /* From NIST SP800-38A */ - .key = "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" - "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" - "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", - .klen = 24, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\x4f\x02\x1d\xb2\x43\xbc\x63\x3d" - "\x71\x78\x18\x3a\x9f\xa0\x71\xe8" - "\xb4\xd9\xad\xa9\xad\x7d\xed\xf4" - "\xe5\xe7\x38\x76\x3f\x69\x14\x5a" - "\x57\x1b\x24\x20\x12\xfb\x7a\xe0" - "\x7f\xa9\xba\xac\x3d\xf1\x02\xe0" - "\x08\xb0\xe2\x79\x88\x59\x88\x81" - "\xd9\x20\xa9\xe6\x4f\x56\x15\xcd", - .rlen = 64, - }, { - .key = "\x60\x3d\xeb\x10\x15\xca\x71\xbe" - "\x2b\x73\xae\xf0\x85\x7d\x77\x81" - "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" - "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", - .klen = 32, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\xf5\x8c\x4c\x04\xd6\xe5\xf1\xba" - "\x77\x9e\xab\xfb\x5f\x7b\xfb\xd6" - "\x9c\xfc\x4e\x96\x7e\xdb\x80\x8d" - "\x67\x9f\x77\x7b\xc6\x70\x2c\x7d" - "\x39\xf2\x33\x69\xa9\xd9\xba\xcf" - "\xa5\x30\xe2\x63\x04\x23\x14\x61" - "\xb2\xeb\x05\xe2\xc3\x9b\xe9\xfc" - "\xda\x6c\x19\x07\x8c\x6a\x9d\x1b", - .rlen = 64, - }, { /* Generated with Crypto++ */ - .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55" - "\x0F\x32\x55\x78\x9B\xBE\x78\x9B" - "\xBE\xE1\x04\x27\xE1\x04\x27\x4A" - "\x6D\x90\x4A\x6D\x90\xB3\xD6\xF9", - .klen = 32, - .iv = "\xE7\x82\x1D\xB8\x53\x11\xAC\x47" - "\xE2\x7D\x18\xD6\x71\x0C\xA7\x42", - .input = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" - "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" - "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" - "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" - "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" - "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" - "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" - "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" - "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" - "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" - "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" - "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" - "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" - "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" - "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" - "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" - "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" - "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" - "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" - "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" - "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" - "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" - "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" - "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" - "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" - "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" - "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" - "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" - "\x20\x89\x15\x7E\xE7\x50\xDC\x45" - "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" - "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" - "\xED\x56\xBF\x28\xB4\x1D\x86\x12", - .ilen = 496, - .result = "\xEA\x65\x8A\x19\xB0\x66\xC1\x3F" - "\xCE\xF1\x97\x75\xC1\xFD\xB5\xAF" - "\x52\x65\xF7\xFF\xBC\xD8\x2D\x9F" - "\x2F\xB9\x26\x9B\x6F\x10\xB7\xB8" - "\x26\xA1\x02\x46\xA2\xAD\xC6\xC0" - "\x11\x15\xFF\x6D\x1E\x82\x04\xA6" - "\xB1\x74\xD1\x08\x13\xFD\x90\x7C" - "\xF5\xED\xD3\xDB\x5A\x0A\x0C\x2F" - "\x0A\x70\xF1\x88\x07\xCF\x21\x26" - "\x40\x40\x8A\xF5\x53\xF7\x24\x4F" - "\x83\x38\x43\x5F\x08\x99\xEB\xE3" - "\xDC\x02\x64\x67\x50\x6E\x15\xC3" - "\x01\x1A\xA0\x81\x13\x65\xA6\x73" - "\x71\xA6\x3B\x91\x83\x77\xBE\xFA" - "\xDB\x71\x73\xA6\xC1\xAE\x43\xC3" - "\x36\xCE\xD6\xEB\xF9\x30\x1C\x4F" - "\x80\x38\x5E\x9C\x6E\xAB\x98\x2F" - "\x53\xAF\xCF\xC8\x9A\xB8\x86\x43" - "\x3E\x86\xE7\xA1\xF4\x2F\x30\x40" - "\x03\xA8\x6C\x50\x42\x9F\x77\x59" - "\x89\xA0\xC5\xEC\x9A\xB8\xDD\x99" - "\x16\x24\x02\x07\x48\xAE\xF2\x31" - "\x34\x0E\xC3\x85\xFE\x1C\x95\x99" - "\x87\x58\x98\x8B\xE7\xC6\xC5\x70" - "\x73\x81\x07\x7C\x56\x2F\xD8\x1B" - "\xB7\xB9\x2B\xAB\xE3\x01\x87\x0F" - "\xD8\xBB\xC0\x0D\xAC\x2C\x2F\x98" - "\x3C\x0B\xA2\x99\x4A\x8C\xF7\x04" - "\xE0\xE0\xCF\xD1\x81\x5B\xFE\xF5" - "\x24\x04\xFD\xB8\xDF\x13\xD8\xCD" - "\xF1\xE3\x3D\x98\x50\x02\x77\x9E" - "\xBC\x22\xAB\xFA\xC2\x43\x1F\x66" - "\x20\x02\x23\xDA\xDF\xA0\x89\xF6" - "\xD8\xF3\x45\x24\x53\x6F\x16\x77" - "\x02\x3E\x7B\x36\x5F\xA0\x3B\x78" - "\x63\xA2\xBD\xB5\xA4\xCA\x1E\xD3" - "\x57\xBC\x0B\x9F\x43\x51\x28\x4F" - "\x07\x50\x6C\x68\x12\x07\xCF\xFA" - "\x6B\x72\x0B\xEB\xF8\x88\x90\x2C" - "\x7E\xF5\x91\xD1\x03\xD8\xD5\xBD" - "\x22\x39\x7B\x16\x03\x01\x69\xAF" - "\x3D\x38\x66\x28\x0C\xBE\x5B\xC5" - "\x03\xB4\x2F\x51\x8A\x56\x17\x2B" - "\x88\x42\x6D\x40\x68\x8F\xD0\x11" - "\x19\xF9\x1F\x43\x79\x95\x31\xFA" - "\x28\x7A\x3D\xF7\x66\xEB\xEF\xAC" - "\x06\xB2\x01\xAD\xDB\x68\xDB\xEC" - "\x8D\x53\x6E\x72\x68\xA3\xC7\x63" - "\x43\x2B\x78\xE0\x04\x29\x8F\x72" - "\xB2\x2C\xE6\x84\x03\x30\x6D\xCD" - "\x26\x92\x37\xE1\x2F\xBB\x8B\x9D" - "\xE4\x4C\xF6\x93\xBC\xD9\xAD\x44" - "\x52\x65\xC7\xB0\x0E\x3F\x0E\x61" - "\x56\x5D\x1C\x6D\xA7\x05\x2E\xBC" - "\x58\x08\x15\xAB\x12\xAB\x17\x4A" - "\x5E\x1C\xF2\xCD\xB8\xA2\xAE\xFB" - "\x9B\x2E\x0E\x85\x34\x80\x0E\x3F" - "\x4C\xB8\xDB\xCE\x1C\x90\xA1\x61" - "\x6C\x69\x09\x35\x9E\xD4\xF4\xAD" - "\xBC\x06\x41\xE3\x01\xB4\x4E\x0A" - "\xE0\x1F\x91\xF8\x82\x96\x2D\x65" - "\xA3\xAA\x13\xCC\x50\xFF\x7B\x02", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec aes_cbc_dec_tv_template[] = { - { /* From RFC 3602 */ - .key = "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" - "\x51\x2e\x03\xd5\x34\x12\x00\x06", - .klen = 16, - .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" - "\xb4\x22\xda\x80\x2c\x9f\xac\x41", - .input = "\xe3\x53\x77\x9c\x10\x79\xae\xb8" - "\x27\x08\x94\x2d\xbe\x77\x18\x1a", - .ilen = 16, - .result = "Single block msg", - .rlen = 16, - .also_non_np = 1, - .np = 8, - .tap = { 3, 2, 3, 2, 3, 1, 1, 1 }, - }, { - .key = "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" - "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", - .klen = 16, - .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" - "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", - .input = "\xd2\x96\xcd\x94\xc2\xcc\xcf\x8a" - "\x3a\x86\x30\x28\xb5\xe1\xdc\x0a" - "\x75\x86\x60\x2d\x25\x3c\xff\xf9" - "\x1b\x82\x66\xbe\xa6\xd6\x1a\xb1", - .ilen = 32, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .rlen = 32, - }, { /* From NIST SP800-38A */ - .key = "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" - "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" - "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", - .klen = 24, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .input = "\x4f\x02\x1d\xb2\x43\xbc\x63\x3d" - "\x71\x78\x18\x3a\x9f\xa0\x71\xe8" - "\xb4\xd9\xad\xa9\xad\x7d\xed\xf4" - "\xe5\xe7\x38\x76\x3f\x69\x14\x5a" - "\x57\x1b\x24\x20\x12\xfb\x7a\xe0" - "\x7f\xa9\xba\xac\x3d\xf1\x02\xe0" - "\x08\xb0\xe2\x79\x88\x59\x88\x81" - "\xd9\x20\xa9\xe6\x4f\x56\x15\xcd", - .ilen = 64, - .result = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .rlen = 64, - }, { - .key = "\x60\x3d\xeb\x10\x15\xca\x71\xbe" - "\x2b\x73\xae\xf0\x85\x7d\x77\x81" - "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" - "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", - .klen = 32, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .input = "\xf5\x8c\x4c\x04\xd6\xe5\xf1\xba" - "\x77\x9e\xab\xfb\x5f\x7b\xfb\xd6" - "\x9c\xfc\x4e\x96\x7e\xdb\x80\x8d" - "\x67\x9f\x77\x7b\xc6\x70\x2c\x7d" - "\x39\xf2\x33\x69\xa9\xd9\xba\xcf" - "\xa5\x30\xe2\x63\x04\x23\x14\x61" - "\xb2\xeb\x05\xe2\xc3\x9b\xe9\xfc" - "\xda\x6c\x19\x07\x8c\x6a\x9d\x1b", - .ilen = 64, - .result = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .rlen = 64, - }, { /* Generated with Crypto++ */ - .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55" - "\x0F\x32\x55\x78\x9B\xBE\x78\x9B" - "\xBE\xE1\x04\x27\xE1\x04\x27\x4A" - "\x6D\x90\x4A\x6D\x90\xB3\xD6\xF9", - .klen = 32, - .iv = "\xE7\x82\x1D\xB8\x53\x11\xAC\x47" - "\xE2\x7D\x18\xD6\x71\x0C\xA7\x42", - .input = "\xEA\x65\x8A\x19\xB0\x66\xC1\x3F" - "\xCE\xF1\x97\x75\xC1\xFD\xB5\xAF" - "\x52\x65\xF7\xFF\xBC\xD8\x2D\x9F" - "\x2F\xB9\x26\x9B\x6F\x10\xB7\xB8" - "\x26\xA1\x02\x46\xA2\xAD\xC6\xC0" - "\x11\x15\xFF\x6D\x1E\x82\x04\xA6" - "\xB1\x74\xD1\x08\x13\xFD\x90\x7C" - "\xF5\xED\xD3\xDB\x5A\x0A\x0C\x2F" - "\x0A\x70\xF1\x88\x07\xCF\x21\x26" - "\x40\x40\x8A\xF5\x53\xF7\x24\x4F" - "\x83\x38\x43\x5F\x08\x99\xEB\xE3" - "\xDC\x02\x64\x67\x50\x6E\x15\xC3" - "\x01\x1A\xA0\x81\x13\x65\xA6\x73" - "\x71\xA6\x3B\x91\x83\x77\xBE\xFA" - "\xDB\x71\x73\xA6\xC1\xAE\x43\xC3" - "\x36\xCE\xD6\xEB\xF9\x30\x1C\x4F" - "\x80\x38\x5E\x9C\x6E\xAB\x98\x2F" - "\x53\xAF\xCF\xC8\x9A\xB8\x86\x43" - "\x3E\x86\xE7\xA1\xF4\x2F\x30\x40" - "\x03\xA8\x6C\x50\x42\x9F\x77\x59" - "\x89\xA0\xC5\xEC\x9A\xB8\xDD\x99" - "\x16\x24\x02\x07\x48\xAE\xF2\x31" - "\x34\x0E\xC3\x85\xFE\x1C\x95\x99" - "\x87\x58\x98\x8B\xE7\xC6\xC5\x70" - "\x73\x81\x07\x7C\x56\x2F\xD8\x1B" - "\xB7\xB9\x2B\xAB\xE3\x01\x87\x0F" - "\xD8\xBB\xC0\x0D\xAC\x2C\x2F\x98" - "\x3C\x0B\xA2\x99\x4A\x8C\xF7\x04" - "\xE0\xE0\xCF\xD1\x81\x5B\xFE\xF5" - "\x24\x04\xFD\xB8\xDF\x13\xD8\xCD" - "\xF1\xE3\x3D\x98\x50\x02\x77\x9E" - "\xBC\x22\xAB\xFA\xC2\x43\x1F\x66" - "\x20\x02\x23\xDA\xDF\xA0\x89\xF6" - "\xD8\xF3\x45\x24\x53\x6F\x16\x77" - "\x02\x3E\x7B\x36\x5F\xA0\x3B\x78" - "\x63\xA2\xBD\xB5\xA4\xCA\x1E\xD3" - "\x57\xBC\x0B\x9F\x43\x51\x28\x4F" - "\x07\x50\x6C\x68\x12\x07\xCF\xFA" - "\x6B\x72\x0B\xEB\xF8\x88\x90\x2C" - "\x7E\xF5\x91\xD1\x03\xD8\xD5\xBD" - "\x22\x39\x7B\x16\x03\x01\x69\xAF" - "\x3D\x38\x66\x28\x0C\xBE\x5B\xC5" - "\x03\xB4\x2F\x51\x8A\x56\x17\x2B" - "\x88\x42\x6D\x40\x68\x8F\xD0\x11" - "\x19\xF9\x1F\x43\x79\x95\x31\xFA" - "\x28\x7A\x3D\xF7\x66\xEB\xEF\xAC" - "\x06\xB2\x01\xAD\xDB\x68\xDB\xEC" - "\x8D\x53\x6E\x72\x68\xA3\xC7\x63" - "\x43\x2B\x78\xE0\x04\x29\x8F\x72" - "\xB2\x2C\xE6\x84\x03\x30\x6D\xCD" - "\x26\x92\x37\xE1\x2F\xBB\x8B\x9D" - "\xE4\x4C\xF6\x93\xBC\xD9\xAD\x44" - "\x52\x65\xC7\xB0\x0E\x3F\x0E\x61" - "\x56\x5D\x1C\x6D\xA7\x05\x2E\xBC" - "\x58\x08\x15\xAB\x12\xAB\x17\x4A" - "\x5E\x1C\xF2\xCD\xB8\xA2\xAE\xFB" - "\x9B\x2E\x0E\x85\x34\x80\x0E\x3F" - "\x4C\xB8\xDB\xCE\x1C\x90\xA1\x61" - "\x6C\x69\x09\x35\x9E\xD4\xF4\xAD" - "\xBC\x06\x41\xE3\x01\xB4\x4E\x0A" - "\xE0\x1F\x91\xF8\x82\x96\x2D\x65" - "\xA3\xAA\x13\xCC\x50\xFF\x7B\x02", - .ilen = 496, - .result = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" - "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" - "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" - "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" - "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" - "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" - "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" - "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" - "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" - "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" - "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" - "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" - "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" - "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" - "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" - "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" - "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" - "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" - "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" - "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" - "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" - "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" - "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" - "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" - "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" - "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" - "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" - "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" - "\x20\x89\x15\x7E\xE7\x50\xDC\x45" - "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" - "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" - "\xED\x56\xBF\x28\xB4\x1D\x86\x12", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct aead_testvec hmac_md5_ecb_cipher_null_enc_tv_template[] = { - { /* Input data from RFC 2410 Case 1 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x00" /* enc key length */ - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 8 + 16 + 0, - .iv = "", - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .ilen = 8, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xaa\x42\xfe\x43\x8d\xea\xa3\x5a" - "\xb9\x3d\x9f\xb1\xa3\x8e\x9b\xae", - .rlen = 8 + 16, - }, { /* Input data from RFC 2410 Case 2 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x00" /* enc key length */ - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 8 + 16 + 0, - .iv = "", - .input = "Network Security People Have A Strange Sense Of Humor", - .ilen = 53, - .result = "Network Security People Have A Strange Sense Of Humor" - "\x73\xa5\x3e\x1c\x08\x0e\x8a\x8a" - "\x8e\xb5\x5f\x90\x8e\xfe\x13\x23", - .rlen = 53 + 16, - }, -}; - -static const struct aead_testvec hmac_md5_ecb_cipher_null_dec_tv_template[] = { - { -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x00" /* enc key length */ - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 8 + 16 + 0, - .iv = "", - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xaa\x42\xfe\x43\x8d\xea\xa3\x5a" - "\xb9\x3d\x9f\xb1\xa3\x8e\x9b\xae", - .ilen = 8 + 16, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .rlen = 8, - }, { -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x00" /* enc key length */ - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 8 + 16 + 0, - .iv = "", - .input = "Network Security People Have A Strange Sense Of Humor" - "\x73\xa5\x3e\x1c\x08\x0e\x8a\x8a" - "\x8e\xb5\x5f\x90\x8e\xfe\x13\x23", - .ilen = 53 + 16, - .result = "Network Security People Have A Strange Sense Of Humor", - .rlen = 53, - }, -}; - -static const struct aead_testvec hmac_sha1_aes_cbc_enc_tv_temp[] = { - { /* RFC 3602 Case 1 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00" - "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" - "\x51\x2e\x03\xd5\x34\x12\x00\x06", - .klen = 8 + 20 + 16, - .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" - "\xb4\x22\xda\x80\x2c\x9f\xac\x41", - .assoc = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" - "\xb4\x22\xda\x80\x2c\x9f\xac\x41", - .alen = 16, - .input = "Single block msg", - .ilen = 16, - .result = "\xe3\x53\x77\x9c\x10\x79\xae\xb8" - "\x27\x08\x94\x2d\xbe\x77\x18\x1a" - "\x1b\x13\xcb\xaf\x89\x5e\xe1\x2c" - "\x13\xc5\x2e\xa3\xcc\xed\xdc\xb5" - "\x03\x71\xa2\x06", - .rlen = 16 + 20, - }, { /* RFC 3602 Case 2 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33" - "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" - "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", - .klen = 8 + 20 + 16, - .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" - "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", - .assoc = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" - "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", - .alen = 16, - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .ilen = 32, - .result = "\xd2\x96\xcd\x94\xc2\xcc\xcf\x8a" - "\x3a\x86\x30\x28\xb5\xe1\xdc\x0a" - "\x75\x86\x60\x2d\x25\x3c\xff\xf9" - "\x1b\x82\x66\xbe\xa6\xd6\x1a\xb1" - "\xad\x9b\x4c\x5c\x85\xe1\xda\xae" - "\xee\x81\x4e\xd7\xdb\x74\xcf\x58" - "\x65\x39\xf8\xde", - .rlen = 32 + 20, - }, { /* RFC 3602 Case 3 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55" - "\x6c\x3e\xa0\x47\x76\x30\xce\x21" - "\xa2\xce\x33\x4a\xa7\x46\xc2\xcd", - .klen = 8 + 20 + 16, - .iv = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" - "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", - .assoc = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" - "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", - .alen = 16, - .input = "This is a 48-byte message (exactly 3 AES blocks)", - .ilen = 48, - .result = "\xd0\xa0\x2b\x38\x36\x45\x17\x53" - "\xd4\x93\x66\x5d\x33\xf0\xe8\x86" - "\x2d\xea\x54\xcd\xb2\x93\xab\xc7" - "\x50\x69\x39\x27\x67\x72\xf8\xd5" - "\x02\x1c\x19\x21\x6b\xad\x52\x5c" - "\x85\x79\x69\x5d\x83\xba\x26\x84" - "\xc2\xec\x0c\xf8\x7f\x05\xba\xca" - "\xff\xee\x4c\xd0\x93\xe6\x36\x7f" - "\x8d\x62\xf2\x1e", - .rlen = 48 + 20, - }, { /* RFC 3602 Case 4 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55" - "\x56\xe4\x7a\x38\xc5\x59\x89\x74" - "\xbc\x46\x90\x3d\xba\x29\x03\x49", - .klen = 8 + 20 + 16, - .iv = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" - "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", - .assoc = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" - "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", - .alen = 16, - .input = "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf", - .ilen = 64, - .result = "\xc3\x0e\x32\xff\xed\xc0\x77\x4e" - "\x6a\xff\x6a\xf0\x86\x9f\x71\xaa" - "\x0f\x3a\xf0\x7a\x9a\x31\xa9\xc6" - "\x84\xdb\x20\x7e\xb0\xef\x8e\x4e" - "\x35\x90\x7a\xa6\x32\xc3\xff\xdf" - "\x86\x8b\xb7\xb2\x9d\x3d\x46\xad" - "\x83\xce\x9f\x9a\x10\x2e\xe9\x9d" - "\x49\xa5\x3e\x87\xf4\xc3\xda\x55" - "\x1c\x45\x57\xa9\x56\xcb\xa9\x2d" - "\x18\xac\xf1\xc7\x5d\xd1\xcd\x0d" - "\x1d\xbe\xc6\xe9", - .rlen = 64 + 20, - }, { /* RFC 3602 Case 5 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55" - "\x90\xd3\x82\xb4\x10\xee\xba\x7a" - "\xd9\x38\xc4\x6c\xec\x1a\x82\xbf", - .klen = 8 + 20 + 16, - .iv = "\xe9\x6e\x8c\x08\xab\x46\x57\x63" - "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\xe9\x6e\x8c\x08\xab\x46\x57\x63" - "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", - .alen = 24, - .input = "\x08\x00\x0e\xbd\xa7\x0a\x00\x00" - "\x8e\x9c\x08\x3d\xb9\x5b\x07\x00" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x01\x02\x03\x04\x05\x06\x07\x08" - "\x09\x0a\x0b\x0c\x0d\x0e\x0e\x01", - .ilen = 80, - .result = "\xf6\x63\xc2\x5d\x32\x5c\x18\xc6" - "\xa9\x45\x3e\x19\x4e\x12\x08\x49" - "\xa4\x87\x0b\x66\xcc\x6b\x99\x65" - "\x33\x00\x13\xb4\x89\x8d\xc8\x56" - "\xa4\x69\x9e\x52\x3a\x55\xdb\x08" - "\x0b\x59\xec\x3a\x8e\x4b\x7e\x52" - "\x77\x5b\x07\xd1\xdb\x34\xed\x9c" - "\x53\x8a\xb5\x0c\x55\x1b\x87\x4a" - "\xa2\x69\xad\xd0\x47\xad\x2d\x59" - "\x13\xac\x19\xb7\xcf\xba\xd4\xa6" - "\x58\xc6\x84\x75\xe4\xe9\x6b\x0c" - "\xe1\xc5\x0b\x73\x4d\x82\x55\xa8" - "\x85\xe1\x59\xf7", - .rlen = 80 + 20, - }, { /* NIST SP800-38A F.2.3 CBC-AES192.Encrypt */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x18" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55" - "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" - "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" - "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", - .klen = 8 + 20 + 24, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .alen = 16, - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\x4f\x02\x1d\xb2\x43\xbc\x63\x3d" - "\x71\x78\x18\x3a\x9f\xa0\x71\xe8" - "\xb4\xd9\xad\xa9\xad\x7d\xed\xf4" - "\xe5\xe7\x38\x76\x3f\x69\x14\x5a" - "\x57\x1b\x24\x20\x12\xfb\x7a\xe0" - "\x7f\xa9\xba\xac\x3d\xf1\x02\xe0" - "\x08\xb0\xe2\x79\x88\x59\x88\x81" - "\xd9\x20\xa9\xe6\x4f\x56\x15\xcd" - "\x73\xe3\x19\x3f\x8b\xc9\xc6\xf4" - "\x5a\xf1\x5b\xa8\x98\x07\xc5\x36" - "\x47\x4c\xfc\x36", - .rlen = 64 + 20, - }, { /* NIST SP800-38A F.2.5 CBC-AES256.Encrypt */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x20" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55" - "\x60\x3d\xeb\x10\x15\xca\x71\xbe" - "\x2b\x73\xae\xf0\x85\x7d\x77\x81" - "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" - "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", - .klen = 8 + 20 + 32, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .alen = 16, - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\xf5\x8c\x4c\x04\xd6\xe5\xf1\xba" - "\x77\x9e\xab\xfb\x5f\x7b\xfb\xd6" - "\x9c\xfc\x4e\x96\x7e\xdb\x80\x8d" - "\x67\x9f\x77\x7b\xc6\x70\x2c\x7d" - "\x39\xf2\x33\x69\xa9\xd9\xba\xcf" - "\xa5\x30\xe2\x63\x04\x23\x14\x61" - "\xb2\xeb\x05\xe2\xc3\x9b\xe9\xfc" - "\xda\x6c\x19\x07\x8c\x6a\x9d\x1b" - "\xa3\xe8\x9b\x17\xe3\xf4\x7f\xde" - "\x1b\x9f\xc6\x81\x26\x43\x4a\x87" - "\x51\xee\xd6\x4e", - .rlen = 64 + 20, - }, -}; - -static const struct aead_testvec hmac_sha1_ecb_cipher_null_enc_tv_temp[] = { - { /* Input data from RFC 2410 Case 1 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x00" /* enc key length */ - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00", - .klen = 8 + 20 + 0, - .iv = "", - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .ilen = 8, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\x40\xc3\x0a\xa1\xc9\xa0\x28\xab" - "\x99\x5e\x19\x04\xd1\x72\xef\xb8" - "\x8c\x5e\xe4\x08", - .rlen = 8 + 20, - }, { /* Input data from RFC 2410 Case 2 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x00" /* enc key length */ - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00", - .klen = 8 + 20 + 0, - .iv = "", - .input = "Network Security People Have A Strange Sense Of Humor", - .ilen = 53, - .result = "Network Security People Have A Strange Sense Of Humor" - "\x75\x6f\x42\x1e\xf8\x50\x21\xd2" - "\x65\x47\xee\x8e\x1a\xef\x16\xf6" - "\x91\x56\xe4\xd6", - .rlen = 53 + 20, - }, -}; - -static const struct aead_testvec hmac_sha1_ecb_cipher_null_dec_tv_temp[] = { - { -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x00" /* enc key length */ - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00", - .klen = 8 + 20 + 0, - .iv = "", - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\x40\xc3\x0a\xa1\xc9\xa0\x28\xab" - "\x99\x5e\x19\x04\xd1\x72\xef\xb8" - "\x8c\x5e\xe4\x08", - .ilen = 8 + 20, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .rlen = 8, - }, { -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x00" /* enc key length */ - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00", - .klen = 8 + 20 + 0, - .iv = "", - .input = "Network Security People Have A Strange Sense Of Humor" - "\x75\x6f\x42\x1e\xf8\x50\x21\xd2" - "\x65\x47\xee\x8e\x1a\xef\x16\xf6" - "\x91\x56\xe4\xd6", - .ilen = 53 + 20, - .result = "Network Security People Have A Strange Sense Of Humor", - .rlen = 53, - }, -}; - -static const struct aead_testvec hmac_sha256_aes_cbc_enc_tv_temp[] = { - { /* RFC 3602 Case 1 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" - "\x51\x2e\x03\xd5\x34\x12\x00\x06", - .klen = 8 + 32 + 16, - .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" - "\xb4\x22\xda\x80\x2c\x9f\xac\x41", - .assoc = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" - "\xb4\x22\xda\x80\x2c\x9f\xac\x41", - .alen = 16, - .input = "Single block msg", - .ilen = 16, - .result = "\xe3\x53\x77\x9c\x10\x79\xae\xb8" - "\x27\x08\x94\x2d\xbe\x77\x18\x1a" - "\xcc\xde\x2d\x6a\xae\xf1\x0b\xcc" - "\x38\x06\x38\x51\xb4\xb8\xf3\x5b" - "\x5c\x34\xa6\xa3\x6e\x0b\x05\xe5" - "\x6a\x6d\x44\xaa\x26\xa8\x44\xa5", - .rlen = 16 + 32, - }, { /* RFC 3602 Case 2 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" - "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", - .klen = 8 + 32 + 16, - .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" - "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", - .assoc = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" - "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", - .alen = 16, - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .ilen = 32, - .result = "\xd2\x96\xcd\x94\xc2\xcc\xcf\x8a" - "\x3a\x86\x30\x28\xb5\xe1\xdc\x0a" - "\x75\x86\x60\x2d\x25\x3c\xff\xf9" - "\x1b\x82\x66\xbe\xa6\xd6\x1a\xb1" - "\xf5\x33\x53\xf3\x68\x85\x2a\x99" - "\x0e\x06\x58\x8f\xba\xf6\x06\xda" - "\x49\x69\x0d\x5b\xd4\x36\x06\x62" - "\x35\x5e\x54\x58\x53\x4d\xdf\xbf", - .rlen = 32 + 32, - }, { /* RFC 3602 Case 3 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x6c\x3e\xa0\x47\x76\x30\xce\x21" - "\xa2\xce\x33\x4a\xa7\x46\xc2\xcd", - .klen = 8 + 32 + 16, - .iv = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" - "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", - .assoc = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" - "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", - .alen = 16, - .input = "This is a 48-byte message (exactly 3 AES blocks)", - .ilen = 48, - .result = "\xd0\xa0\x2b\x38\x36\x45\x17\x53" - "\xd4\x93\x66\x5d\x33\xf0\xe8\x86" - "\x2d\xea\x54\xcd\xb2\x93\xab\xc7" - "\x50\x69\x39\x27\x67\x72\xf8\xd5" - "\x02\x1c\x19\x21\x6b\xad\x52\x5c" - "\x85\x79\x69\x5d\x83\xba\x26\x84" - "\x68\xb9\x3e\x90\x38\xa0\x88\x01" - "\xe7\xc6\xce\x10\x31\x2f\x9b\x1d" - "\x24\x78\xfb\xbe\x02\xe0\x4f\x40" - "\x10\xbd\xaa\xc6\xa7\x79\xe0\x1a", - .rlen = 48 + 32, - }, { /* RFC 3602 Case 4 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x56\xe4\x7a\x38\xc5\x59\x89\x74" - "\xbc\x46\x90\x3d\xba\x29\x03\x49", - .klen = 8 + 32 + 16, - .iv = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" - "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", - .assoc = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" - "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", - .alen = 16, - .input = "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf", - .ilen = 64, - .result = "\xc3\x0e\x32\xff\xed\xc0\x77\x4e" - "\x6a\xff\x6a\xf0\x86\x9f\x71\xaa" - "\x0f\x3a\xf0\x7a\x9a\x31\xa9\xc6" - "\x84\xdb\x20\x7e\xb0\xef\x8e\x4e" - "\x35\x90\x7a\xa6\x32\xc3\xff\xdf" - "\x86\x8b\xb7\xb2\x9d\x3d\x46\xad" - "\x83\xce\x9f\x9a\x10\x2e\xe9\x9d" - "\x49\xa5\x3e\x87\xf4\xc3\xda\x55" - "\x7a\x1b\xd4\x3c\xdb\x17\x95\xe2" - "\xe0\x93\xec\xc9\x9f\xf7\xce\xd8" - "\x3f\x54\xe2\x49\x39\xe3\x71\x25" - "\x2b\x6c\xe9\x5d\xec\xec\x2b\x64", - .rlen = 64 + 32, - }, { /* RFC 3602 Case 5 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x90\xd3\x82\xb4\x10\xee\xba\x7a" - "\xd9\x38\xc4\x6c\xec\x1a\x82\xbf", - .klen = 8 + 32 + 16, - .iv = "\xe9\x6e\x8c\x08\xab\x46\x57\x63" - "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\xe9\x6e\x8c\x08\xab\x46\x57\x63" - "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", - .alen = 24, - .input = "\x08\x00\x0e\xbd\xa7\x0a\x00\x00" - "\x8e\x9c\x08\x3d\xb9\x5b\x07\x00" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x01\x02\x03\x04\x05\x06\x07\x08" - "\x09\x0a\x0b\x0c\x0d\x0e\x0e\x01", - .ilen = 80, - .result = "\xf6\x63\xc2\x5d\x32\x5c\x18\xc6" - "\xa9\x45\x3e\x19\x4e\x12\x08\x49" - "\xa4\x87\x0b\x66\xcc\x6b\x99\x65" - "\x33\x00\x13\xb4\x89\x8d\xc8\x56" - "\xa4\x69\x9e\x52\x3a\x55\xdb\x08" - "\x0b\x59\xec\x3a\x8e\x4b\x7e\x52" - "\x77\x5b\x07\xd1\xdb\x34\xed\x9c" - "\x53\x8a\xb5\x0c\x55\x1b\x87\x4a" - "\xa2\x69\xad\xd0\x47\xad\x2d\x59" - "\x13\xac\x19\xb7\xcf\xba\xd4\xa6" - "\xbb\xd4\x0f\xbe\xa3\x3b\x4c\xb8" - "\x3a\xd2\xe1\x03\x86\xa5\x59\xb7" - "\x73\xc3\x46\x20\x2c\xb1\xef\x68" - "\xbb\x8a\x32\x7e\x12\x8c\x69\xcf", - .rlen = 80 + 32, - }, { /* NIST SP800-38A F.2.3 CBC-AES192.Encrypt */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x18" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" - "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" - "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", - .klen = 8 + 32 + 24, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .alen = 16, - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\x4f\x02\x1d\xb2\x43\xbc\x63\x3d" - "\x71\x78\x18\x3a\x9f\xa0\x71\xe8" - "\xb4\xd9\xad\xa9\xad\x7d\xed\xf4" - "\xe5\xe7\x38\x76\x3f\x69\x14\x5a" - "\x57\x1b\x24\x20\x12\xfb\x7a\xe0" - "\x7f\xa9\xba\xac\x3d\xf1\x02\xe0" - "\x08\xb0\xe2\x79\x88\x59\x88\x81" - "\xd9\x20\xa9\xe6\x4f\x56\x15\xcd" - "\x2f\xee\x5f\xdb\x66\xfe\x79\x09" - "\x61\x81\x31\xea\x5b\x3d\x8e\xfb" - "\xca\x71\x85\x93\xf7\x85\x55\x8b" - "\x7a\xe4\x94\xca\x8b\xba\x19\x33", - .rlen = 64 + 32, - }, { /* NIST SP800-38A F.2.5 CBC-AES256.Encrypt */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x20" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x60\x3d\xeb\x10\x15\xca\x71\xbe" - "\x2b\x73\xae\xf0\x85\x7d\x77\x81" - "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" - "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", - .klen = 8 + 32 + 32, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .alen = 16, - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\xf5\x8c\x4c\x04\xd6\xe5\xf1\xba" - "\x77\x9e\xab\xfb\x5f\x7b\xfb\xd6" - "\x9c\xfc\x4e\x96\x7e\xdb\x80\x8d" - "\x67\x9f\x77\x7b\xc6\x70\x2c\x7d" - "\x39\xf2\x33\x69\xa9\xd9\xba\xcf" - "\xa5\x30\xe2\x63\x04\x23\x14\x61" - "\xb2\xeb\x05\xe2\xc3\x9b\xe9\xfc" - "\xda\x6c\x19\x07\x8c\x6a\x9d\x1b" - "\x24\x29\xed\xc2\x31\x49\xdb\xb1" - "\x8f\x74\xbd\x17\x92\x03\xbe\x8f" - "\xf3\x61\xde\x1c\xe9\xdb\xcd\xd0" - "\xcc\xce\xe9\x85\x57\xcf\x6f\x5f", - .rlen = 64 + 32, - }, -}; - -static const struct aead_testvec hmac_sha512_aes_cbc_enc_tv_temp[] = { - { /* RFC 3602 Case 1 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" - "\x51\x2e\x03\xd5\x34\x12\x00\x06", - .klen = 8 + 64 + 16, - .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" - "\xb4\x22\xda\x80\x2c\x9f\xac\x41", - .assoc = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" - "\xb4\x22\xda\x80\x2c\x9f\xac\x41", - .alen = 16, - .input = "Single block msg", - .ilen = 16, - .result = "\xe3\x53\x77\x9c\x10\x79\xae\xb8" - "\x27\x08\x94\x2d\xbe\x77\x18\x1a" - "\x3f\xdc\xad\x90\x03\x63\x5e\x68" - "\xc3\x13\xdd\xa4\x5c\x4d\x54\xa7" - "\x19\x6e\x03\x75\x2b\xa1\x62\xce" - "\xe0\xc6\x96\x75\xb2\x14\xca\x96" - "\xec\xbd\x50\x08\x07\x64\x1a\x49" - "\xe8\x9a\x7c\x06\x3d\xcb\xff\xb2" - "\xfa\x20\x89\xdd\x9c\xac\x9e\x16" - "\x18\x8a\xa0\x6d\x01\x6c\xa3\x3a", - .rlen = 16 + 64, - }, { /* RFC 3602 Case 2 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" - "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", - .klen = 8 + 64 + 16, - .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" - "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", - .assoc = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" - "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", - .alen = 16, - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .ilen = 32, - .result = "\xd2\x96\xcd\x94\xc2\xcc\xcf\x8a" - "\x3a\x86\x30\x28\xb5\xe1\xdc\x0a" - "\x75\x86\x60\x2d\x25\x3c\xff\xf9" - "\x1b\x82\x66\xbe\xa6\xd6\x1a\xb1" - "\xda\xb2\x0c\xb2\x26\xc4\xd5\xef" - "\x60\x38\xa4\x5e\x9a\x8c\x1b\x41" - "\x03\x9f\xc4\x64\x7f\x01\x42\x9b" - "\x0e\x1b\xea\xef\xbc\x88\x19\x5e" - "\x31\x7e\xc2\x95\xfc\x09\x32\x0a" - "\x46\x32\x7c\x41\x9c\x59\x3e\xe9" - "\x8f\x9f\xd4\x31\xd6\x22\xbd\xf8" - "\xf7\x0a\x94\xe5\xa9\xc3\xf6\x9d", - .rlen = 32 + 64, - }, { /* RFC 3602 Case 3 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x33\x44\x55\x66\x77\x88\x99\xaa" - "\xbb\xcc\xdd\xee\xff\x11\x22\x33" - "\x44\x55\x66\x77\x88\x99\xaa\xbb" - "\xcc\xdd\xee\xff\x11\x22\x33\x44" - "\x6c\x3e\xa0\x47\x76\x30\xce\x21" - "\xa2\xce\x33\x4a\xa7\x46\xc2\xcd", - .klen = 8 + 64 + 16, - .iv = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" - "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", - .assoc = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" - "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", - .alen = 16, - .input = "This is a 48-byte message (exactly 3 AES blocks)", - .ilen = 48, - .result = "\xd0\xa0\x2b\x38\x36\x45\x17\x53" - "\xd4\x93\x66\x5d\x33\xf0\xe8\x86" - "\x2d\xea\x54\xcd\xb2\x93\xab\xc7" - "\x50\x69\x39\x27\x67\x72\xf8\xd5" - "\x02\x1c\x19\x21\x6b\xad\x52\x5c" - "\x85\x79\x69\x5d\x83\xba\x26\x84" - "\x64\x19\x17\x5b\x57\xe0\x21\x0f" - "\xca\xdb\xa1\x26\x38\x14\xa2\x69" - "\xdb\x54\x67\x80\xc0\x54\xe0\xfd" - "\x3e\x91\xe7\x91\x7f\x13\x38\x44" - "\xb7\xb1\xd6\xc8\x7d\x48\x8d\x41" - "\x08\xea\x29\x6c\x74\x67\x3f\xb0" - "\xac\x7f\x5c\x1d\xf5\xee\x22\x66" - "\x27\xa6\xb6\x13\xba\xba\xf0\xc2", - .rlen = 48 + 64, - }, { /* RFC 3602 Case 4 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x33\x44\x55\x66\x77\x88\x99\xaa" - "\xbb\xcc\xdd\xee\xff\x11\x22\x33" - "\x44\x55\x66\x77\x88\x99\xaa\xbb" - "\xcc\xdd\xee\xff\x11\x22\x33\x44" - "\x56\xe4\x7a\x38\xc5\x59\x89\x74" - "\xbc\x46\x90\x3d\xba\x29\x03\x49", - .klen = 8 + 64 + 16, - .iv = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" - "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", - .assoc = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" - "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", - .alen = 16, - .input = "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf", - .ilen = 64, - .result = "\xc3\x0e\x32\xff\xed\xc0\x77\x4e" - "\x6a\xff\x6a\xf0\x86\x9f\x71\xaa" - "\x0f\x3a\xf0\x7a\x9a\x31\xa9\xc6" - "\x84\xdb\x20\x7e\xb0\xef\x8e\x4e" - "\x35\x90\x7a\xa6\x32\xc3\xff\xdf" - "\x86\x8b\xb7\xb2\x9d\x3d\x46\xad" - "\x83\xce\x9f\x9a\x10\x2e\xe9\x9d" - "\x49\xa5\x3e\x87\xf4\xc3\xda\x55" - "\x82\xcd\x42\x28\x21\x20\x15\xcc" - "\xb7\xb2\x48\x40\xc7\x64\x41\x3a" - "\x61\x32\x82\x85\xcf\x27\xed\xb4" - "\xe4\x68\xa2\xf5\x79\x26\x27\xb2" - "\x51\x67\x6a\xc4\xf0\x66\x55\x50" - "\xbc\x6f\xed\xd5\x8d\xde\x23\x7c" - "\x62\x98\x14\xd7\x2f\x37\x8d\xdf" - "\xf4\x33\x80\xeb\x8e\xb4\xa4\xda", - .rlen = 64 + 64, - }, { /* RFC 3602 Case 5 */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x10" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x33\x44\x55\x66\x77\x88\x99\xaa" - "\xbb\xcc\xdd\xee\xff\x11\x22\x33" - "\x44\x55\x66\x77\x88\x99\xaa\xbb" - "\xcc\xdd\xee\xff\x11\x22\x33\x44" - "\x90\xd3\x82\xb4\x10\xee\xba\x7a" - "\xd9\x38\xc4\x6c\xec\x1a\x82\xbf", - .klen = 8 + 64 + 16, - .iv = "\xe9\x6e\x8c\x08\xab\x46\x57\x63" - "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\xe9\x6e\x8c\x08\xab\x46\x57\x63" - "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", - .alen = 24, - .input = "\x08\x00\x0e\xbd\xa7\x0a\x00\x00" - "\x8e\x9c\x08\x3d\xb9\x5b\x07\x00" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x01\x02\x03\x04\x05\x06\x07\x08" - "\x09\x0a\x0b\x0c\x0d\x0e\x0e\x01", - .ilen = 80, - .result = "\xf6\x63\xc2\x5d\x32\x5c\x18\xc6" - "\xa9\x45\x3e\x19\x4e\x12\x08\x49" - "\xa4\x87\x0b\x66\xcc\x6b\x99\x65" - "\x33\x00\x13\xb4\x89\x8d\xc8\x56" - "\xa4\x69\x9e\x52\x3a\x55\xdb\x08" - "\x0b\x59\xec\x3a\x8e\x4b\x7e\x52" - "\x77\x5b\x07\xd1\xdb\x34\xed\x9c" - "\x53\x8a\xb5\x0c\x55\x1b\x87\x4a" - "\xa2\x69\xad\xd0\x47\xad\x2d\x59" - "\x13\xac\x19\xb7\xcf\xba\xd4\xa6" - "\x74\x84\x94\xe2\xd7\x7a\xf9\xbf" - "\x00\x8a\xa2\xd5\xb7\xf3\x60\xcf" - "\xa0\x47\xdf\x4e\x09\xf4\xb1\x7f" - "\x14\xd9\x3d\x53\x8e\x12\xb3\x00" - "\x4c\x0a\x4e\x32\x40\x43\x88\xce" - "\x92\x26\xc1\x76\x20\x11\xeb\xba" - "\x62\x4f\x9a\x62\x25\xc3\x75\x80" - "\xb7\x0a\x17\xf5\xd7\x94\xb4\x14", - .rlen = 80 + 64, - }, { /* NIST SP800-38A F.2.3 CBC-AES192.Encrypt */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x18" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x33\x44\x55\x66\x77\x88\x99\xaa" - "\xbb\xcc\xdd\xee\xff\x11\x22\x33" - "\x44\x55\x66\x77\x88\x99\xaa\xbb" - "\xcc\xdd\xee\xff\x11\x22\x33\x44" - "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" - "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" - "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", - .klen = 8 + 64 + 24, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .alen = 16, - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\x4f\x02\x1d\xb2\x43\xbc\x63\x3d" - "\x71\x78\x18\x3a\x9f\xa0\x71\xe8" - "\xb4\xd9\xad\xa9\xad\x7d\xed\xf4" - "\xe5\xe7\x38\x76\x3f\x69\x14\x5a" - "\x57\x1b\x24\x20\x12\xfb\x7a\xe0" - "\x7f\xa9\xba\xac\x3d\xf1\x02\xe0" - "\x08\xb0\xe2\x79\x88\x59\x88\x81" - "\xd9\x20\xa9\xe6\x4f\x56\x15\xcd" - "\x77\x4b\x69\x9d\x3a\x0d\xb4\x99" - "\x8f\xc6\x8e\x0e\x72\x58\xe3\x56" - "\xbb\x21\xd2\x7d\x93\x11\x17\x91" - "\xc4\x83\xfd\x0a\xea\x71\xfe\x77" - "\xae\x6f\x0a\xa5\xf0\xcf\xe1\x35" - "\xba\x03\xd5\x32\xfa\x5f\x41\x58" - "\x8d\x43\x98\xa7\x94\x16\x07\x02" - "\x0f\xb6\x81\x50\x28\x95\x2e\x75", - .rlen = 64 + 64, - }, { /* NIST SP800-38A F.2.5 CBC-AES256.Encrypt */ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x20" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x33\x44\x55\x66\x77\x88\x99\xaa" - "\xbb\xcc\xdd\xee\xff\x11\x22\x33" - "\x44\x55\x66\x77\x88\x99\xaa\xbb" - "\xcc\xdd\xee\xff\x11\x22\x33\x44" - "\x60\x3d\xeb\x10\x15\xca\x71\xbe" - "\x2b\x73\xae\xf0\x85\x7d\x77\x81" - "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" - "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", - .klen = 8 + 64 + 32, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .alen = 16, - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\xf5\x8c\x4c\x04\xd6\xe5\xf1\xba" - "\x77\x9e\xab\xfb\x5f\x7b\xfb\xd6" - "\x9c\xfc\x4e\x96\x7e\xdb\x80\x8d" - "\x67\x9f\x77\x7b\xc6\x70\x2c\x7d" - "\x39\xf2\x33\x69\xa9\xd9\xba\xcf" - "\xa5\x30\xe2\x63\x04\x23\x14\x61" - "\xb2\xeb\x05\xe2\xc3\x9b\xe9\xfc" - "\xda\x6c\x19\x07\x8c\x6a\x9d\x1b" - "\xb2\x27\x69\x7f\x45\x64\x79\x2b" - "\xb7\xb8\x4c\xd4\x75\x94\x68\x40" - "\x2a\xea\x91\xc7\x3f\x7c\xed\x7b" - "\x95\x2c\x9b\xa8\xf5\xe5\x52\x8d" - "\x6b\xe1\xae\xf1\x74\xfa\x0d\x0c" - "\xe3\x8d\x64\xc3\x8d\xff\x7c\x8c" - "\xdb\xbf\xa0\xb4\x01\xa2\xa8\xa2" - "\x2c\xb1\x62\x2c\x10\xca\xf1\x21", - .rlen = 64 + 64, - }, -}; - -static const struct aead_testvec hmac_sha1_des_cbc_enc_tv_temp[] = { - { /*Generated with cryptopp*/ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x08" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55" - "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24", - .klen = 8 + 20 + 8, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .alen = 16, - .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .ilen = 128, - .result = "\x70\xd6\xde\x64\x87\x17\xf1\xe8" - "\x54\x31\x85\x37\xed\x6b\x01\x8d" - "\xe3\xcc\xe0\x1d\x5e\xf3\xfe\xf1" - "\x41\xaa\x33\x91\xa7\x7d\x99\x88" - "\x4d\x85\x6e\x2f\xa3\x69\xf5\x82" - "\x3a\x6f\x25\xcb\x7d\x58\x1f\x9b" - "\xaa\x9c\x11\xd5\x76\x67\xce\xde" - "\x56\xd7\x5a\x80\x69\xea\x3a\x02" - "\xf0\xc7\x7c\xe3\xcb\x40\xe5\x52" - "\xd1\x10\x92\x78\x0b\x8e\x5b\xf1" - "\xe3\x26\x1f\xe1\x15\x41\xc7\xba" - "\x99\xdb\x08\x51\x1c\xd3\x01\xf4" - "\x87\x47\x39\xb8\xd2\xdd\xbd\xfb" - "\x66\x13\xdf\x1c\x01\x44\xf0\x7a" - "\x1a\x6b\x13\xf5\xd5\x0b\xb8\xba" - "\x53\xba\xe1\x76\xe3\x82\x07\x86" - "\x95\x16\x20\x09\xf5\x95\x19\xfd" - "\x3c\xc7\xe0\x42\xc0\x14\x69\xfa" - "\x5c\x44\xa9\x37", - .rlen = 128 + 20, - }, -}; - -static const struct aead_testvec hmac_sha224_des_cbc_enc_tv_temp[] = { - { /*Generated with cryptopp*/ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x08" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24", - .klen = 8 + 24 + 8, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .alen = 16, - .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .ilen = 128, - .result = "\x70\xd6\xde\x64\x87\x17\xf1\xe8" - "\x54\x31\x85\x37\xed\x6b\x01\x8d" - "\xe3\xcc\xe0\x1d\x5e\xf3\xfe\xf1" - "\x41\xaa\x33\x91\xa7\x7d\x99\x88" - "\x4d\x85\x6e\x2f\xa3\x69\xf5\x82" - "\x3a\x6f\x25\xcb\x7d\x58\x1f\x9b" - "\xaa\x9c\x11\xd5\x76\x67\xce\xde" - "\x56\xd7\x5a\x80\x69\xea\x3a\x02" - "\xf0\xc7\x7c\xe3\xcb\x40\xe5\x52" - "\xd1\x10\x92\x78\x0b\x8e\x5b\xf1" - "\xe3\x26\x1f\xe1\x15\x41\xc7\xba" - "\x99\xdb\x08\x51\x1c\xd3\x01\xf4" - "\x87\x47\x39\xb8\xd2\xdd\xbd\xfb" - "\x66\x13\xdf\x1c\x01\x44\xf0\x7a" - "\x1a\x6b\x13\xf5\xd5\x0b\xb8\xba" - "\x53\xba\xe1\x76\xe3\x82\x07\x86" - "\x9c\x2d\x7e\xee\x20\x34\x55\x0a" - "\xce\xb5\x4e\x64\x53\xe7\xbf\x91" - "\xab\xd4\xd9\xda\xc9\x12\xae\xf7", - .rlen = 128 + 24, - }, -}; - -static const struct aead_testvec hmac_sha256_des_cbc_enc_tv_temp[] = { - { /*Generated with cryptopp*/ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x08" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24", - .klen = 8 + 32 + 8, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .alen = 16, - .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .ilen = 128, - .result = "\x70\xd6\xde\x64\x87\x17\xf1\xe8" - "\x54\x31\x85\x37\xed\x6b\x01\x8d" - "\xe3\xcc\xe0\x1d\x5e\xf3\xfe\xf1" - "\x41\xaa\x33\x91\xa7\x7d\x99\x88" - "\x4d\x85\x6e\x2f\xa3\x69\xf5\x82" - "\x3a\x6f\x25\xcb\x7d\x58\x1f\x9b" - "\xaa\x9c\x11\xd5\x76\x67\xce\xde" - "\x56\xd7\x5a\x80\x69\xea\x3a\x02" - "\xf0\xc7\x7c\xe3\xcb\x40\xe5\x52" - "\xd1\x10\x92\x78\x0b\x8e\x5b\xf1" - "\xe3\x26\x1f\xe1\x15\x41\xc7\xba" - "\x99\xdb\x08\x51\x1c\xd3\x01\xf4" - "\x87\x47\x39\xb8\xd2\xdd\xbd\xfb" - "\x66\x13\xdf\x1c\x01\x44\xf0\x7a" - "\x1a\x6b\x13\xf5\xd5\x0b\xb8\xba" - "\x53\xba\xe1\x76\xe3\x82\x07\x86" - "\xc6\x58\xa1\x60\x70\x91\x39\x36" - "\x50\xf6\x5d\xab\x4b\x51\x4e\x5e" - "\xde\x63\xde\x76\x52\xde\x9f\xba" - "\x90\xcf\x15\xf2\xbb\x6e\x84\x00", - .rlen = 128 + 32, - }, -}; - -static const struct aead_testvec hmac_sha384_des_cbc_enc_tv_temp[] = { - { /*Generated with cryptopp*/ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x08" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x33\x44\x55\x66\x77\x88\x99\xaa" - "\xbb\xcc\xdd\xee\xff\x11\x22\x33" - "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24", - .klen = 8 + 48 + 8, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .alen = 16, - .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .ilen = 128, - .result = "\x70\xd6\xde\x64\x87\x17\xf1\xe8" - "\x54\x31\x85\x37\xed\x6b\x01\x8d" - "\xe3\xcc\xe0\x1d\x5e\xf3\xfe\xf1" - "\x41\xaa\x33\x91\xa7\x7d\x99\x88" - "\x4d\x85\x6e\x2f\xa3\x69\xf5\x82" - "\x3a\x6f\x25\xcb\x7d\x58\x1f\x9b" - "\xaa\x9c\x11\xd5\x76\x67\xce\xde" - "\x56\xd7\x5a\x80\x69\xea\x3a\x02" - "\xf0\xc7\x7c\xe3\xcb\x40\xe5\x52" - "\xd1\x10\x92\x78\x0b\x8e\x5b\xf1" - "\xe3\x26\x1f\xe1\x15\x41\xc7\xba" - "\x99\xdb\x08\x51\x1c\xd3\x01\xf4" - "\x87\x47\x39\xb8\xd2\xdd\xbd\xfb" - "\x66\x13\xdf\x1c\x01\x44\xf0\x7a" - "\x1a\x6b\x13\xf5\xd5\x0b\xb8\xba" - "\x53\xba\xe1\x76\xe3\x82\x07\x86" - "\xa8\x8e\x9c\x74\x8c\x2b\x99\xa0" - "\xc8\x8c\xef\x25\x07\x83\x11\x3a" - "\x31\x8d\xbe\x3b\x6a\xd7\x96\xfe" - "\x5e\x67\xb5\x74\xe7\xe7\x85\x61" - "\x6a\x95\x26\x75\xcc\x53\x89\xf3" - "\x74\xc9\x2a\x76\x20\xa2\x64\x62", - .rlen = 128 + 48, - }, -}; - -static const struct aead_testvec hmac_sha512_des_cbc_enc_tv_temp[] = { - { /*Generated with cryptopp*/ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x08" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x33\x44\x55\x66\x77\x88\x99\xaa" - "\xbb\xcc\xdd\xee\xff\x11\x22\x33" - "\x44\x55\x66\x77\x88\x99\xaa\xbb" - "\xcc\xdd\xee\xff\x11\x22\x33\x44" - "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24", - .klen = 8 + 64 + 8, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .alen = 16, - .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .ilen = 128, - .result = "\x70\xd6\xde\x64\x87\x17\xf1\xe8" - "\x54\x31\x85\x37\xed\x6b\x01\x8d" - "\xe3\xcc\xe0\x1d\x5e\xf3\xfe\xf1" - "\x41\xaa\x33\x91\xa7\x7d\x99\x88" - "\x4d\x85\x6e\x2f\xa3\x69\xf5\x82" - "\x3a\x6f\x25\xcb\x7d\x58\x1f\x9b" - "\xaa\x9c\x11\xd5\x76\x67\xce\xde" - "\x56\xd7\x5a\x80\x69\xea\x3a\x02" - "\xf0\xc7\x7c\xe3\xcb\x40\xe5\x52" - "\xd1\x10\x92\x78\x0b\x8e\x5b\xf1" - "\xe3\x26\x1f\xe1\x15\x41\xc7\xba" - "\x99\xdb\x08\x51\x1c\xd3\x01\xf4" - "\x87\x47\x39\xb8\xd2\xdd\xbd\xfb" - "\x66\x13\xdf\x1c\x01\x44\xf0\x7a" - "\x1a\x6b\x13\xf5\xd5\x0b\xb8\xba" - "\x53\xba\xe1\x76\xe3\x82\x07\x86" - "\xc6\x2c\x73\x88\xb0\x9d\x5f\x3e" - "\x5b\x78\xca\x0e\xab\x8a\xa3\xbb" - "\xd9\x1d\xc3\xe3\x05\xac\x76\xfb" - "\x58\x83\xda\x67\xfb\x21\x24\xa2" - "\xb1\xa7\xd7\x66\xa6\x8d\xa6\x93" - "\x97\xe2\xe3\xb8\xaa\x48\x85\xee" - "\x8c\xf6\x07\x95\x1f\xa6\x6c\x96" - "\x99\xc7\x5c\x8d\xd8\xb5\x68\x7b", - .rlen = 128 + 64, - }, -}; - -static const struct aead_testvec hmac_sha1_des3_ede_cbc_enc_tv_temp[] = { - { /*Generated with cryptopp*/ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x18" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55" - "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" - "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" - "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", - .klen = 8 + 20 + 24, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .alen = 16, - .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .ilen = 128, - .result = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" - "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" - "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" - "\x12\x56\x5c\x53\x96\xb6\x00\x7d" - "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" - "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" - "\x76\xd1\xda\x0c\x94\x67\xbb\x04" - "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" - "\x22\x64\x47\xaa\x8f\x75\x13\xbf" - "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" - "\x71\x63\x2e\x89\x7b\x1e\x12\xca" - "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" - "\xd6\xf9\x21\x31\x62\x44\x45\xa6" - "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" - "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" - "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19" - "\x67\x6d\xb1\xf5\xb8\x10\xdc\xc6" - "\x75\x86\x96\x6b\xb1\xc5\xe4\xcf" - "\xd1\x60\x91\xb3", - .rlen = 128 + 20, - }, -}; - -static const struct aead_testvec hmac_sha224_des3_ede_cbc_enc_tv_temp[] = { - { /*Generated with cryptopp*/ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x18" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" - "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" - "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", - .klen = 8 + 24 + 24, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .alen = 16, - .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .ilen = 128, - .result = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" - "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" - "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" - "\x12\x56\x5c\x53\x96\xb6\x00\x7d" - "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" - "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" - "\x76\xd1\xda\x0c\x94\x67\xbb\x04" - "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" - "\x22\x64\x47\xaa\x8f\x75\x13\xbf" - "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" - "\x71\x63\x2e\x89\x7b\x1e\x12\xca" - "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" - "\xd6\xf9\x21\x31\x62\x44\x45\xa6" - "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" - "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" - "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19" - "\x15\x24\x7f\x5a\x45\x4a\x66\xce" - "\x2b\x0b\x93\x99\x2f\x9d\x0c\x6c" - "\x56\x1f\xe1\xa6\x41\xb2\x4c\xd0", - .rlen = 128 + 24, - }, -}; - -static const struct aead_testvec hmac_sha256_des3_ede_cbc_enc_tv_temp[] = { - { /*Generated with cryptopp*/ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x18" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" - "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" - "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", - .klen = 8 + 32 + 24, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .alen = 16, - .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .ilen = 128, - .result = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" - "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" - "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" - "\x12\x56\x5c\x53\x96\xb6\x00\x7d" - "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" - "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" - "\x76\xd1\xda\x0c\x94\x67\xbb\x04" - "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" - "\x22\x64\x47\xaa\x8f\x75\x13\xbf" - "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" - "\x71\x63\x2e\x89\x7b\x1e\x12\xca" - "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" - "\xd6\xf9\x21\x31\x62\x44\x45\xa6" - "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" - "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" - "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19" - "\x73\xb0\xea\x9f\xe8\x18\x80\xd6" - "\x56\x38\x44\xc0\xdb\xe3\x4f\x71" - "\xf7\xce\xd1\xd3\xf8\xbd\x3e\x4f" - "\xca\x43\x95\xdf\x80\x61\x81\xa9", - .rlen = 128 + 32, - }, -}; - -static const struct aead_testvec hmac_sha384_des3_ede_cbc_enc_tv_temp[] = { - { /*Generated with cryptopp*/ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x18" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x33\x44\x55\x66\x77\x88\x99\xaa" - "\xbb\xcc\xdd\xee\xff\x11\x22\x33" - "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" - "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" - "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", - .klen = 8 + 48 + 24, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .alen = 16, - .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .ilen = 128, - .result = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" - "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" - "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" - "\x12\x56\x5c\x53\x96\xb6\x00\x7d" - "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" - "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" - "\x76\xd1\xda\x0c\x94\x67\xbb\x04" - "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" - "\x22\x64\x47\xaa\x8f\x75\x13\xbf" - "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" - "\x71\x63\x2e\x89\x7b\x1e\x12\xca" - "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" - "\xd6\xf9\x21\x31\x62\x44\x45\xa6" - "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" - "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" - "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19" - "\x6d\x77\xfc\x80\x9d\x8a\x9c\xb7" - "\x70\xe7\x93\xbf\x73\xe6\x9f\x83" - "\x99\x62\x23\xe6\x5b\xd0\xda\x18" - "\xa4\x32\x8a\x0b\x46\xd7\xf0\x39" - "\x36\x5d\x13\x2f\x86\x10\x78\xd6" - "\xd6\xbe\x5c\xb9\x15\x89\xf9\x1b", - .rlen = 128 + 48, - }, -}; - -static const struct aead_testvec hmac_sha512_des3_ede_cbc_enc_tv_temp[] = { - { /*Generated with cryptopp*/ -#ifdef __LITTLE_ENDIAN - .key = "\x08\x00" /* rta length */ - "\x01\x00" /* rta type */ -#else - .key = "\x00\x08" /* rta length */ - "\x00\x01" /* rta type */ -#endif - "\x00\x00\x00\x18" /* enc key length */ - "\x11\x22\x33\x44\x55\x66\x77\x88" - "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" - "\x22\x33\x44\x55\x66\x77\x88\x99" - "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" - "\x33\x44\x55\x66\x77\x88\x99\xaa" - "\xbb\xcc\xdd\xee\xff\x11\x22\x33" - "\x44\x55\x66\x77\x88\x99\xaa\xbb" - "\xcc\xdd\xee\xff\x11\x22\x33\x44" - "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" - "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" - "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", - .klen = 8 + 64 + 24, - .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" - "\x7D\x33\x88\x93\x0F\x93\xB2\x42", - .alen = 16, - .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" - "\x53\x20\x63\x65\x65\x72\x73\x74" - "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" - "\x20\x79\x65\x53\x72\x63\x74\x65" - "\x20\x73\x6f\x54\x20\x6f\x61\x4d" - "\x79\x6e\x53\x20\x63\x65\x65\x72" - "\x73\x74\x54\x20\x6f\x6f\x4d\x20" - "\x6e\x61\x20\x79\x65\x53\x72\x63" - "\x74\x65\x20\x73\x6f\x54\x20\x6f" - "\x61\x4d\x79\x6e\x53\x20\x63\x65" - "\x65\x72\x73\x74\x54\x20\x6f\x6f" - "\x4d\x20\x6e\x61\x20\x79\x65\x53" - "\x72\x63\x74\x65\x20\x73\x6f\x54" - "\x20\x6f\x61\x4d\x79\x6e\x53\x20" - "\x63\x65\x65\x72\x73\x74\x54\x20" - "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", - .ilen = 128, - .result = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" - "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" - "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" - "\x12\x56\x5c\x53\x96\xb6\x00\x7d" - "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" - "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" - "\x76\xd1\xda\x0c\x94\x67\xbb\x04" - "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" - "\x22\x64\x47\xaa\x8f\x75\x13\xbf" - "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" - "\x71\x63\x2e\x89\x7b\x1e\x12\xca" - "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" - "\xd6\xf9\x21\x31\x62\x44\x45\xa6" - "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" - "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" - "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19" - "\x41\xb5\x1f\xbb\xbd\x4e\xb8\x32" - "\x22\x86\x4e\x57\x1b\x2a\xd8\x6e" - "\xa9\xfb\xc8\xf3\xbf\x2d\xae\x2b" - "\x3b\xbc\x41\xe8\x38\xbb\xf1\x60" - "\x4c\x68\xa9\x4e\x8c\x73\xa7\xc0" - "\x2a\x74\xd4\x65\x12\xcb\x55\xf2" - "\xd5\x02\x6d\xe6\xaf\xc9\x2f\xf2" - "\x57\xaa\x85\xf7\xf3\x6a\xcb\xdb", - .rlen = 128 + 64, - }, -}; - -static const struct cipher_testvec aes_lrw_enc_tv_template[] = { - /* from http://grouper.ieee.org/groups/1619/email/pdf00017.pdf */ - { /* LRW-32-AES 1 */ - .key = "\x45\x62\xac\x25\xf8\x28\x17\x6d" - "\x4c\x26\x84\x14\xb5\x68\x01\x85" - "\x25\x8e\x2a\x05\xe7\x3e\x9d\x03" - "\xee\x5a\x83\x0c\xcc\x09\x4c\x87", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\xf1\xb2\x73\xcd\x65\xa3\xdf\x5f" - "\xe9\x5d\x48\x92\x54\x63\x4e\xb8", - .rlen = 16, - }, { /* LRW-32-AES 2 */ - .key = "\x59\x70\x47\x14\xf5\x57\x47\x8c" - "\xd7\x79\xe8\x0f\x54\x88\x79\x44" - "\x0d\x48\xf0\xb7\xb1\x5a\x53\xea" - "\x1c\xaa\x6b\x29\xc2\xca\xfb\xaf", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x02", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x00\xc8\x2b\xae\x95\xbb\xcd\xe5" - "\x27\x4f\x07\x69\xb2\x60\xe1\x36", - .rlen = 16, - }, { /* LRW-32-AES 3 */ - .key = "\xd8\x2a\x91\x34\xb2\x6a\x56\x50" - "\x30\xfe\x69\xe2\x37\x7f\x98\x47" - "\xcd\xf9\x0b\x16\x0c\x64\x8f\xb6" - "\xb0\x0d\x0d\x1b\xae\x85\x87\x1f", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x76\x32\x21\x83\xed\x8f\xf1\x82" - "\xf9\x59\x62\x03\x69\x0e\x5e\x01", - .rlen = 16, - }, { /* LRW-32-AES 4 */ - .key = "\x0f\x6a\xef\xf8\xd3\xd2\xbb\x15" - "\x25\x83\xf7\x3c\x1f\x01\x28\x74" - "\xca\xc6\xbc\x35\x4d\x4a\x65\x54" - "\x90\xae\x61\xcf\x7b\xae\xbd\xcc" - "\xad\xe4\x94\xc5\x4a\x29\xae\x70", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x9c\x0f\x15\x2f\x55\xa2\xd8\xf0" - "\xd6\x7b\x8f\x9e\x28\x22\xbc\x41", - .rlen = 16, - }, { /* LRW-32-AES 5 */ - .key = "\x8a\xd4\xee\x10\x2f\xbd\x81\xff" - "\xf8\x86\xce\xac\x93\xc5\xad\xc6" - "\xa0\x19\x07\xc0\x9d\xf7\xbb\xdd" - "\x52\x13\xb2\xb7\xf0\xff\x11\xd8" - "\xd6\x08\xd0\xcd\x2e\xb1\x17\x6f", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\xd4\x27\x6a\x7f\x14\x91\x3d\x65" - "\xc8\x60\x48\x02\x87\xe3\x34\x06", - .rlen = 16, - }, { /* LRW-32-AES 6 */ - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\xbd\x06\xb8\xe1\xdb\x98\x89\x9e" - "\xc4\x98\xe4\x91\xcf\x1c\x70\x2b", - .rlen = 16, - }, { /* LRW-32-AES 7 */ - .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" - "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" - "\xb2\xfb\x64\xce\x60\x97\x87\x8d" - "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" - "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" - "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x5b\x90\x8e\xc1\xab\xdd\x67\x5f" - "\x3d\x69\x8a\x95\x53\xc8\x9c\xe5", - .rlen = 16, - }, { -/* http://www.mail-archive.com/stds-p1619@listserv.ieee.org/msg00173.html */ - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" - "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" - "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" - "\x50\x38\x1f\x71\x49\xb6\x57\xd6" - "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" - "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" - "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" - "\xda\x10\x8e\xed\xa2\xa4\x87\xab" - "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" - "\xc9\xac\x42\x31\x95\x7c\xc9\x04" - "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" - "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" - "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" - "\x4c\x96\x12\xed\x7c\x92\x03\x01" - "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" - "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" - "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" - "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" - "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" - "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" - "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" - "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" - "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" - "\x76\x12\x73\x44\x1a\x56\xd7\x72" - "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" - "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" - "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" - "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" - "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" - "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" - "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" - "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" - "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" - "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" - "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" - "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" - "\x8d\x23\x31\x74\x84\xeb\x88\x6e" - "\xcc\xb9\xbc\x22\x83\x19\x07\x22" - "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" - "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" - "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" - "\x3c\xce\x8f\x42\x60\x71\xa7\x75" - "\x08\x40\x65\x8a\x82\xbf\xf5\x43" - "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" - "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" - "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" - "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" - "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" - "\x62\x73\x65\xfd\x46\x63\x25\x3d" - "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" - "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" - "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" - "\xc5\x68\x77\x84\x32\x2b\xcc\x85" - "\x74\x96\xf0\x12\x77\x61\xb9\xeb" - "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" - "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" - "\xda\x39\x87\x45\xc0\x2b\xbb\x01" - "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" - "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" - "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" - "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" - "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" - "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" - "\x21\xc4\xc2\x75\x67\x89\x37\x0a", - .ilen = 512, - .result = "\x1a\x1d\xa9\x30\xad\xf9\x2f\x9b" - "\xb6\x1d\xae\xef\xf0\x2f\xf8\x5a" - "\x39\x3c\xbf\x2a\xb2\x45\xb2\x23" - "\x1b\x63\x3c\xcf\xaa\xbe\xcf\x4e" - "\xfa\xe8\x29\xc2\x20\x68\x2b\x3c" - "\x2e\x8b\xf7\x6e\x25\xbd\xe3\x3d" - "\x66\x27\xd6\xaf\xd6\x64\x3e\xe3" - "\xe8\x58\x46\x97\x39\x51\x07\xde" - "\xcb\x37\xbc\xa9\xc0\x5f\x75\xc3" - "\x0e\x84\x23\x1d\x16\xd4\x1c\x59" - "\x9c\x1a\x02\x55\xab\x3a\x97\x1d" - "\xdf\xdd\xc7\x06\x51\xd7\x70\xae" - "\x23\xc6\x8c\xf5\x1e\xa0\xe5\x82" - "\xb8\xb2\xbf\x04\xa0\x32\x8e\x68" - "\xeb\xaf\x6e\x2d\x94\x22\x2f\xce" - "\x4c\xb5\x59\xe2\xa2\x2f\xa0\x98" - "\x1a\x97\xc6\xd4\xb5\x00\x59\xf2" - "\x84\x14\x72\xb1\x9a\x6e\xa3\x7f" - "\xea\x20\xe7\xcb\x65\x77\x3a\xdf" - "\xc8\x97\x67\x15\xc2\x2a\x27\xcc" - "\x18\x55\xa1\x24\x0b\x24\x24\xaf" - "\x5b\xec\x68\xb8\xc8\xf5\xba\x63" - "\xff\xed\x89\xce\xd5\x3d\x88\xf3" - "\x25\xef\x05\x7c\x3a\xef\xeb\xd8" - "\x7a\x32\x0d\xd1\x1e\x58\x59\x99" - "\x90\x25\xb5\x26\xb0\xe3\x2b\x6c" - "\x4c\xa9\x8b\x84\x4f\x5e\x01\x50" - "\x41\x30\x58\xc5\x62\x74\x52\x1d" - "\x45\x24\x6a\x42\x64\x4f\x97\x1c" - "\xa8\x66\xb5\x6d\x79\xd4\x0d\x48" - "\xc5\x5f\xf3\x90\x32\xdd\xdd\xe1" - "\xe4\xa9\x9f\xfc\xc3\x52\x5a\x46" - "\xe4\x81\x84\x95\x36\x59\x7a\x6b" - "\xaa\xb3\x60\xad\xce\x9f\x9f\x28" - "\xe0\x01\x75\x22\xc4\x4e\xa9\x62" - "\x5c\x62\x0d\x00\xcb\x13\xe8\x43" - "\x72\xd4\x2d\x53\x46\xb5\xd1\x16" - "\x22\x18\xdf\x34\x33\xf5\xd6\x1c" - "\xb8\x79\x78\x97\x94\xff\x72\x13" - "\x4c\x27\xfc\xcb\xbf\x01\x53\xa6" - "\xb4\x50\x6e\xde\xdf\xb5\x43\xa4" - "\x59\xdf\x52\xf9\x7c\xe0\x11\x6f" - "\x2d\x14\x8e\x24\x61\x2c\xe1\x17" - "\xcc\xce\x51\x0c\x19\x8a\x82\x30" - "\x94\xd5\x3d\x6a\x53\x06\x5e\xbd" - "\xb7\xeb\xfa\xfd\x27\x51\xde\x85" - "\x1e\x86\x53\x11\x53\x94\x00\xee" - "\x2b\x8c\x08\x2a\xbf\xdd\xae\x11" - "\xcb\x1e\xa2\x07\x9a\x80\xcf\x62" - "\x9b\x09\xdc\x95\x3c\x96\x8e\xb1" - "\x09\xbd\xe4\xeb\xdb\xca\x70\x7a" - "\x9e\xfa\x31\x18\x45\x3c\x21\x33" - "\xb0\xb3\x2b\xea\xf3\x71\x2d\xe1" - "\x03\xad\x1b\x48\xd4\x67\x27\xf0" - "\x62\xe4\x3d\xfb\x9b\x08\x76\xe7" - "\xdd\x2b\x01\x39\x04\x5a\x58\x7a" - "\xf7\x11\x90\xec\xbd\x51\x5c\x32" - "\x6b\xd7\x35\x39\x02\x6b\xf2\xa6" - "\xd0\x0d\x07\xe1\x06\xc4\x5b\x7d" - "\xe4\x6a\xd7\xee\x15\x1f\x83\xb4" - "\xa3\xa7\x5e\xc3\x90\xb7\xef\xd3" - "\xb7\x4f\xf8\x92\x4c\xb7\x3c\x29" - "\xcd\x7e\x2b\x5d\x43\xea\x42\xe7" - "\x74\x3f\x7d\x58\x88\x75\xde\x3e", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - } -}; - -static const struct cipher_testvec aes_lrw_dec_tv_template[] = { - /* from http://grouper.ieee.org/groups/1619/email/pdf00017.pdf */ - /* same as enc vectors with input and result reversed */ - { /* LRW-32-AES 1 */ - .key = "\x45\x62\xac\x25\xf8\x28\x17\x6d" - "\x4c\x26\x84\x14\xb5\x68\x01\x85" - "\x25\x8e\x2a\x05\xe7\x3e\x9d\x03" - "\xee\x5a\x83\x0c\xcc\x09\x4c\x87", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\xf1\xb2\x73\xcd\x65\xa3\xdf\x5f" - "\xe9\x5d\x48\x92\x54\x63\x4e\xb8", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { /* LRW-32-AES 2 */ - .key = "\x59\x70\x47\x14\xf5\x57\x47\x8c" - "\xd7\x79\xe8\x0f\x54\x88\x79\x44" - "\x0d\x48\xf0\xb7\xb1\x5a\x53\xea" - "\x1c\xaa\x6b\x29\xc2\xca\xfb\xaf", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x02", - .input = "\x00\xc8\x2b\xae\x95\xbb\xcd\xe5" - "\x27\x4f\x07\x69\xb2\x60\xe1\x36", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { /* LRW-32-AES 3 */ - .key = "\xd8\x2a\x91\x34\xb2\x6a\x56\x50" - "\x30\xfe\x69\xe2\x37\x7f\x98\x47" - "\xcd\xf9\x0b\x16\x0c\x64\x8f\xb6" - "\xb0\x0d\x0d\x1b\xae\x85\x87\x1f", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x76\x32\x21\x83\xed\x8f\xf1\x82" - "\xf9\x59\x62\x03\x69\x0e\x5e\x01", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { /* LRW-32-AES 4 */ - .key = "\x0f\x6a\xef\xf8\xd3\xd2\xbb\x15" - "\x25\x83\xf7\x3c\x1f\x01\x28\x74" - "\xca\xc6\xbc\x35\x4d\x4a\x65\x54" - "\x90\xae\x61\xcf\x7b\xae\xbd\xcc" - "\xad\xe4\x94\xc5\x4a\x29\xae\x70", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x9c\x0f\x15\x2f\x55\xa2\xd8\xf0" - "\xd6\x7b\x8f\x9e\x28\x22\xbc\x41", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { /* LRW-32-AES 5 */ - .key = "\x8a\xd4\xee\x10\x2f\xbd\x81\xff" - "\xf8\x86\xce\xac\x93\xc5\xad\xc6" - "\xa0\x19\x07\xc0\x9d\xf7\xbb\xdd" - "\x52\x13\xb2\xb7\xf0\xff\x11\xd8" - "\xd6\x08\xd0\xcd\x2e\xb1\x17\x6f", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\xd4\x27\x6a\x7f\x14\x91\x3d\x65" - "\xc8\x60\x48\x02\x87\xe3\x34\x06", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { /* LRW-32-AES 6 */ - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\xbd\x06\xb8\xe1\xdb\x98\x89\x9e" - "\xc4\x98\xe4\x91\xcf\x1c\x70\x2b", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { /* LRW-32-AES 7 */ - .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" - "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" - "\xb2\xfb\x64\xce\x60\x97\x87\x8d" - "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" - "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" - "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x5b\x90\x8e\xc1\xab\xdd\x67\x5f" - "\x3d\x69\x8a\x95\x53\xc8\x9c\xe5", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { -/* http://www.mail-archive.com/stds-p1619@listserv.ieee.org/msg00173.html */ - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x1a\x1d\xa9\x30\xad\xf9\x2f\x9b" - "\xb6\x1d\xae\xef\xf0\x2f\xf8\x5a" - "\x39\x3c\xbf\x2a\xb2\x45\xb2\x23" - "\x1b\x63\x3c\xcf\xaa\xbe\xcf\x4e" - "\xfa\xe8\x29\xc2\x20\x68\x2b\x3c" - "\x2e\x8b\xf7\x6e\x25\xbd\xe3\x3d" - "\x66\x27\xd6\xaf\xd6\x64\x3e\xe3" - "\xe8\x58\x46\x97\x39\x51\x07\xde" - "\xcb\x37\xbc\xa9\xc0\x5f\x75\xc3" - "\x0e\x84\x23\x1d\x16\xd4\x1c\x59" - "\x9c\x1a\x02\x55\xab\x3a\x97\x1d" - "\xdf\xdd\xc7\x06\x51\xd7\x70\xae" - "\x23\xc6\x8c\xf5\x1e\xa0\xe5\x82" - "\xb8\xb2\xbf\x04\xa0\x32\x8e\x68" - "\xeb\xaf\x6e\x2d\x94\x22\x2f\xce" - "\x4c\xb5\x59\xe2\xa2\x2f\xa0\x98" - "\x1a\x97\xc6\xd4\xb5\x00\x59\xf2" - "\x84\x14\x72\xb1\x9a\x6e\xa3\x7f" - "\xea\x20\xe7\xcb\x65\x77\x3a\xdf" - "\xc8\x97\x67\x15\xc2\x2a\x27\xcc" - "\x18\x55\xa1\x24\x0b\x24\x24\xaf" - "\x5b\xec\x68\xb8\xc8\xf5\xba\x63" - "\xff\xed\x89\xce\xd5\x3d\x88\xf3" - "\x25\xef\x05\x7c\x3a\xef\xeb\xd8" - "\x7a\x32\x0d\xd1\x1e\x58\x59\x99" - "\x90\x25\xb5\x26\xb0\xe3\x2b\x6c" - "\x4c\xa9\x8b\x84\x4f\x5e\x01\x50" - "\x41\x30\x58\xc5\x62\x74\x52\x1d" - "\x45\x24\x6a\x42\x64\x4f\x97\x1c" - "\xa8\x66\xb5\x6d\x79\xd4\x0d\x48" - "\xc5\x5f\xf3\x90\x32\xdd\xdd\xe1" - "\xe4\xa9\x9f\xfc\xc3\x52\x5a\x46" - "\xe4\x81\x84\x95\x36\x59\x7a\x6b" - "\xaa\xb3\x60\xad\xce\x9f\x9f\x28" - "\xe0\x01\x75\x22\xc4\x4e\xa9\x62" - "\x5c\x62\x0d\x00\xcb\x13\xe8\x43" - "\x72\xd4\x2d\x53\x46\xb5\xd1\x16" - "\x22\x18\xdf\x34\x33\xf5\xd6\x1c" - "\xb8\x79\x78\x97\x94\xff\x72\x13" - "\x4c\x27\xfc\xcb\xbf\x01\x53\xa6" - "\xb4\x50\x6e\xde\xdf\xb5\x43\xa4" - "\x59\xdf\x52\xf9\x7c\xe0\x11\x6f" - "\x2d\x14\x8e\x24\x61\x2c\xe1\x17" - "\xcc\xce\x51\x0c\x19\x8a\x82\x30" - "\x94\xd5\x3d\x6a\x53\x06\x5e\xbd" - "\xb7\xeb\xfa\xfd\x27\x51\xde\x85" - "\x1e\x86\x53\x11\x53\x94\x00\xee" - "\x2b\x8c\x08\x2a\xbf\xdd\xae\x11" - "\xcb\x1e\xa2\x07\x9a\x80\xcf\x62" - "\x9b\x09\xdc\x95\x3c\x96\x8e\xb1" - "\x09\xbd\xe4\xeb\xdb\xca\x70\x7a" - "\x9e\xfa\x31\x18\x45\x3c\x21\x33" - "\xb0\xb3\x2b\xea\xf3\x71\x2d\xe1" - "\x03\xad\x1b\x48\xd4\x67\x27\xf0" - "\x62\xe4\x3d\xfb\x9b\x08\x76\xe7" - "\xdd\x2b\x01\x39\x04\x5a\x58\x7a" - "\xf7\x11\x90\xec\xbd\x51\x5c\x32" - "\x6b\xd7\x35\x39\x02\x6b\xf2\xa6" - "\xd0\x0d\x07\xe1\x06\xc4\x5b\x7d" - "\xe4\x6a\xd7\xee\x15\x1f\x83\xb4" - "\xa3\xa7\x5e\xc3\x90\xb7\xef\xd3" - "\xb7\x4f\xf8\x92\x4c\xb7\x3c\x29" - "\xcd\x7e\x2b\x5d\x43\xea\x42\xe7" - "\x74\x3f\x7d\x58\x88\x75\xde\x3e", - .ilen = 512, - .result = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" - "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" - "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" - "\x50\x38\x1f\x71\x49\xb6\x57\xd6" - "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" - "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" - "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" - "\xda\x10\x8e\xed\xa2\xa4\x87\xab" - "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" - "\xc9\xac\x42\x31\x95\x7c\xc9\x04" - "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" - "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" - "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" - "\x4c\x96\x12\xed\x7c\x92\x03\x01" - "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" - "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" - "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" - "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" - "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" - "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" - "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" - "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" - "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" - "\x76\x12\x73\x44\x1a\x56\xd7\x72" - "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" - "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" - "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" - "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" - "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" - "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" - "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" - "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" - "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" - "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" - "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" - "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" - "\x8d\x23\x31\x74\x84\xeb\x88\x6e" - "\xcc\xb9\xbc\x22\x83\x19\x07\x22" - "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" - "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" - "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" - "\x3c\xce\x8f\x42\x60\x71\xa7\x75" - "\x08\x40\x65\x8a\x82\xbf\xf5\x43" - "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" - "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" - "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" - "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" - "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" - "\x62\x73\x65\xfd\x46\x63\x25\x3d" - "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" - "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" - "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" - "\xc5\x68\x77\x84\x32\x2b\xcc\x85" - "\x74\x96\xf0\x12\x77\x61\xb9\xeb" - "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" - "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" - "\xda\x39\x87\x45\xc0\x2b\xbb\x01" - "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" - "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" - "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" - "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" - "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" - "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" - "\x21\xc4\xc2\x75\x67\x89\x37\x0a", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - } -}; - -static const struct cipher_testvec aes_xts_enc_tv_template[] = { - /* http://grouper.ieee.org/groups/1619/email/pdf00086.pdf */ - { /* XTS-AES 1 */ - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 32, - .fips_skip = 1, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 32, - .result = "\x91\x7c\xf6\x9e\xbd\x68\xb2\xec" - "\x9b\x9f\xe9\xa3\xea\xdd\xa6\x92" - "\xcd\x43\xd2\xf5\x95\x98\xed\x85" - "\x8c\x02\xc2\x65\x2f\xbf\x92\x2e", - .rlen = 32, - }, { /* XTS-AES 2 */ - .key = "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\xc4\x54\x18\x5e\x6a\x16\x93\x6e" - "\x39\x33\x40\x38\xac\xef\x83\x8b" - "\xfb\x18\x6f\xff\x74\x80\xad\xc4" - "\x28\x93\x82\xec\xd6\xd3\x94\xf0", - .rlen = 32, - }, { /* XTS-AES 3 */ - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\xaf\x85\x33\x6b\x59\x7a\xfc\x1a" - "\x90\x0b\x2e\xb2\x1e\xc9\x49\xd2" - "\x92\xdf\x4c\x04\x7e\x0b\x21\x53" - "\x21\x86\xa5\x97\x1a\x22\x7a\x89", - .rlen = 32, - }, { /* XTS-AES 4 */ - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\x27\xa7\x47\x9b\xef\xa1\xd4\x76" - "\x48\x9f\x30\x8c\xd4\xcf\xa6\xe2" - "\xa9\x6e\x4b\xbe\x32\x08\xff\x25" - "\x28\x7d\xd3\x81\x96\x16\xe8\x9c" - "\xc7\x8c\xf7\xf5\xe5\x43\x44\x5f" - "\x83\x33\xd8\xfa\x7f\x56\x00\x00" - "\x05\x27\x9f\xa5\xd8\xb5\xe4\xad" - "\x40\xe7\x36\xdd\xb4\xd3\x54\x12" - "\x32\x80\x63\xfd\x2a\xab\x53\xe5" - "\xea\x1e\x0a\x9f\x33\x25\x00\xa5" - "\xdf\x94\x87\xd0\x7a\x5c\x92\xcc" - "\x51\x2c\x88\x66\xc7\xe8\x60\xce" - "\x93\xfd\xf1\x66\xa2\x49\x12\xb4" - "\x22\x97\x61\x46\xae\x20\xce\x84" - "\x6b\xb7\xdc\x9b\xa9\x4a\x76\x7a" - "\xae\xf2\x0c\x0d\x61\xad\x02\x65" - "\x5e\xa9\x2d\xc4\xc4\xe4\x1a\x89" - "\x52\xc6\x51\xd3\x31\x74\xbe\x51" - "\xa1\x0c\x42\x11\x10\xe6\xd8\x15" - "\x88\xed\xe8\x21\x03\xa2\x52\xd8" - "\xa7\x50\xe8\x76\x8d\xef\xff\xed" - "\x91\x22\x81\x0a\xae\xb9\x9f\x91" - "\x72\xaf\x82\xb6\x04\xdc\x4b\x8e" - "\x51\xbc\xb0\x82\x35\xa6\xf4\x34" - "\x13\x32\xe4\xca\x60\x48\x2a\x4b" - "\xa1\xa0\x3b\x3e\x65\x00\x8f\xc5" - "\xda\x76\xb7\x0b\xf1\x69\x0d\xb4" - "\xea\xe2\x9c\x5f\x1b\xad\xd0\x3c" - "\x5c\xcf\x2a\x55\xd7\x05\xdd\xcd" - "\x86\xd4\x49\x51\x1c\xeb\x7e\xc3" - "\x0b\xf1\x2b\x1f\xa3\x5b\x91\x3f" - "\x9f\x74\x7a\x8a\xfd\x1b\x13\x0e" - "\x94\xbf\xf9\x4e\xff\xd0\x1a\x91" - "\x73\x5c\xa1\x72\x6a\xcd\x0b\x19" - "\x7c\x4e\x5b\x03\x39\x36\x97\xe1" - "\x26\x82\x6f\xb6\xbb\xde\x8e\xcc" - "\x1e\x08\x29\x85\x16\xe2\xc9\xed" - "\x03\xff\x3c\x1b\x78\x60\xf6\xde" - "\x76\xd4\xce\xcd\x94\xc8\x11\x98" - "\x55\xef\x52\x97\xca\x67\xe9\xf3" - "\xe7\xff\x72\xb1\xe9\x97\x85\xca" - "\x0a\x7e\x77\x20\xc5\xb3\x6d\xc6" - "\xd7\x2c\xac\x95\x74\xc8\xcb\xbc" - "\x2f\x80\x1e\x23\xe5\x6f\xd3\x44" - "\xb0\x7f\x22\x15\x4b\xeb\xa0\xf0" - "\x8c\xe8\x89\x1e\x64\x3e\xd9\x95" - "\xc9\x4d\x9a\x69\xc9\xf1\xb5\xf4" - "\x99\x02\x7a\x78\x57\x2a\xee\xbd" - "\x74\xd2\x0c\xc3\x98\x81\xc2\x13" - "\xee\x77\x0b\x10\x10\xe4\xbe\xa7" - "\x18\x84\x69\x77\xae\x11\x9f\x7a" - "\x02\x3a\xb5\x8c\xca\x0a\xd7\x52" - "\xaf\xe6\x56\xbb\x3c\x17\x25\x6a" - "\x9f\x6e\x9b\xf1\x9f\xdd\x5a\x38" - "\xfc\x82\xbb\xe8\x72\xc5\x53\x9e" - "\xdb\x60\x9e\xf4\xf7\x9c\x20\x3e" - "\xbb\x14\x0f\x2e\x58\x3c\xb2\xad" - "\x15\xb4\xaa\x5b\x65\x50\x16\xa8" - "\x44\x92\x77\xdb\xd4\x77\xef\x2c" - "\x8d\x6c\x01\x7d\xb7\x38\xb1\x8d" - "\xeb\x4a\x42\x7d\x19\x23\xce\x3f" - "\xf2\x62\x73\x57\x79\xa4\x18\xf2" - "\x0a\x28\x2d\xf9\x20\x14\x7b\xea" - "\xbe\x42\x1e\xe5\x31\x9d\x05\x68", - .rlen = 512, - }, { /* XTS-AES 10, XTS-AES-256, data unit 512 bytes */ - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\x1c\x3b\x3a\x10\x2f\x77\x03\x86" - "\xe4\x83\x6c\x99\xe3\x70\xcf\x9b" - "\xea\x00\x80\x3f\x5e\x48\x23\x57" - "\xa4\xae\x12\xd4\x14\xa3\xe6\x3b" - "\x5d\x31\xe2\x76\xf8\xfe\x4a\x8d" - "\x66\xb3\x17\xf9\xac\x68\x3f\x44" - "\x68\x0a\x86\xac\x35\xad\xfc\x33" - "\x45\xbe\xfe\xcb\x4b\xb1\x88\xfd" - "\x57\x76\x92\x6c\x49\xa3\x09\x5e" - "\xb1\x08\xfd\x10\x98\xba\xec\x70" - "\xaa\xa6\x69\x99\xa7\x2a\x82\xf2" - "\x7d\x84\x8b\x21\xd4\xa7\x41\xb0" - "\xc5\xcd\x4d\x5f\xff\x9d\xac\x89" - "\xae\xba\x12\x29\x61\xd0\x3a\x75" - "\x71\x23\xe9\x87\x0f\x8a\xcf\x10" - "\x00\x02\x08\x87\x89\x14\x29\xca" - "\x2a\x3e\x7a\x7d\x7d\xf7\xb1\x03" - "\x55\x16\x5c\x8b\x9a\x6d\x0a\x7d" - "\xe8\xb0\x62\xc4\x50\x0d\xc4\xcd" - "\x12\x0c\x0f\x74\x18\xda\xe3\xd0" - "\xb5\x78\x1c\x34\x80\x3f\xa7\x54" - "\x21\xc7\x90\xdf\xe1\xde\x18\x34" - "\xf2\x80\xd7\x66\x7b\x32\x7f\x6c" - "\x8c\xd7\x55\x7e\x12\xac\x3a\x0f" - "\x93\xec\x05\xc5\x2e\x04\x93\xef" - "\x31\xa1\x2d\x3d\x92\x60\xf7\x9a" - "\x28\x9d\x6a\x37\x9b\xc7\x0c\x50" - "\x84\x14\x73\xd1\xa8\xcc\x81\xec" - "\x58\x3e\x96\x45\xe0\x7b\x8d\x96" - "\x70\x65\x5b\xa5\xbb\xcf\xec\xc6" - "\xdc\x39\x66\x38\x0a\xd8\xfe\xcb" - "\x17\xb6\xba\x02\x46\x9a\x02\x0a" - "\x84\xe1\x8e\x8f\x84\x25\x20\x70" - "\xc1\x3e\x9f\x1f\x28\x9b\xe5\x4f" - "\xbc\x48\x14\x57\x77\x8f\x61\x60" - "\x15\xe1\x32\x7a\x02\xb1\x40\xf1" - "\x50\x5e\xb3\x09\x32\x6d\x68\x37" - "\x8f\x83\x74\x59\x5c\x84\x9d\x84" - "\xf4\xc3\x33\xec\x44\x23\x88\x51" - "\x43\xcb\x47\xbd\x71\xc5\xed\xae" - "\x9b\xe6\x9a\x2f\xfe\xce\xb1\xbe" - "\xc9\xde\x24\x4f\xbe\x15\x99\x2b" - "\x11\xb7\x7c\x04\x0f\x12\xbd\x8f" - "\x6a\x97\x5a\x44\xa0\xf9\x0c\x29" - "\xa9\xab\xc3\xd4\xd8\x93\x92\x72" - "\x84\xc5\x87\x54\xcc\xe2\x94\x52" - "\x9f\x86\x14\xdc\xd2\xab\xa9\x91" - "\x92\x5f\xed\xc4\xae\x74\xff\xac" - "\x6e\x33\x3b\x93\xeb\x4a\xff\x04" - "\x79\xda\x9a\x41\x0e\x44\x50\xe0" - "\xdd\x7a\xe4\xc6\xe2\x91\x09\x00" - "\x57\x5d\xa4\x01\xfc\x07\x05\x9f" - "\x64\x5e\x8b\x7e\x9b\xfd\xef\x33" - "\x94\x30\x54\xff\x84\x01\x14\x93" - "\xc2\x7b\x34\x29\xea\xed\xb4\xed" - "\x53\x76\x44\x1a\x77\xed\x43\x85" - "\x1a\xd7\x7f\x16\xf5\x41\xdf\xd2" - "\x69\xd5\x0d\x6a\x5f\x14\xfb\x0a" - "\xab\x1c\xbb\x4c\x15\x50\xbe\x97" - "\xf7\xab\x40\x66\x19\x3c\x4c\xaa" - "\x77\x3d\xad\x38\x01\x4b\xd2\x09" - "\x2f\xa7\x55\xc8\x24\xbb\x5e\x54" - "\xc4\xf3\x6f\xfd\xa9\xfc\xea\x70" - "\xb9\xc6\xe6\x93\xe1\x48\xc1\x51", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - } -}; - -static const struct cipher_testvec aes_xts_dec_tv_template[] = { - /* http://grouper.ieee.org/groups/1619/email/pdf00086.pdf */ - { /* XTS-AES 1 */ - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 32, - .fips_skip = 1, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x91\x7c\xf6\x9e\xbd\x68\xb2\xec" - "\x9b\x9f\xe9\xa3\xea\xdd\xa6\x92" - "\xcd\x43\xd2\xf5\x95\x98\xed\x85" - "\x8c\x02\xc2\x65\x2f\xbf\x92\x2e", - .ilen = 32, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 32, - }, { /* XTS-AES 2 */ - .key = "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xc4\x54\x18\x5e\x6a\x16\x93\x6e" - "\x39\x33\x40\x38\xac\xef\x83\x8b" - "\xfb\x18\x6f\xff\x74\x80\xad\xc4" - "\x28\x93\x82\xec\xd6\xd3\x94\xf0", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { /* XTS-AES 3 */ - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xaf\x85\x33\x6b\x59\x7a\xfc\x1a" - "\x90\x0b\x2e\xb2\x1e\xc9\x49\xd2" - "\x92\xdf\x4c\x04\x7e\x0b\x21\x53" - "\x21\x86\xa5\x97\x1a\x22\x7a\x89", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { /* XTS-AES 4 */ - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x27\xa7\x47\x9b\xef\xa1\xd4\x76" - "\x48\x9f\x30\x8c\xd4\xcf\xa6\xe2" - "\xa9\x6e\x4b\xbe\x32\x08\xff\x25" - "\x28\x7d\xd3\x81\x96\x16\xe8\x9c" - "\xc7\x8c\xf7\xf5\xe5\x43\x44\x5f" - "\x83\x33\xd8\xfa\x7f\x56\x00\x00" - "\x05\x27\x9f\xa5\xd8\xb5\xe4\xad" - "\x40\xe7\x36\xdd\xb4\xd3\x54\x12" - "\x32\x80\x63\xfd\x2a\xab\x53\xe5" - "\xea\x1e\x0a\x9f\x33\x25\x00\xa5" - "\xdf\x94\x87\xd0\x7a\x5c\x92\xcc" - "\x51\x2c\x88\x66\xc7\xe8\x60\xce" - "\x93\xfd\xf1\x66\xa2\x49\x12\xb4" - "\x22\x97\x61\x46\xae\x20\xce\x84" - "\x6b\xb7\xdc\x9b\xa9\x4a\x76\x7a" - "\xae\xf2\x0c\x0d\x61\xad\x02\x65" - "\x5e\xa9\x2d\xc4\xc4\xe4\x1a\x89" - "\x52\xc6\x51\xd3\x31\x74\xbe\x51" - "\xa1\x0c\x42\x11\x10\xe6\xd8\x15" - "\x88\xed\xe8\x21\x03\xa2\x52\xd8" - "\xa7\x50\xe8\x76\x8d\xef\xff\xed" - "\x91\x22\x81\x0a\xae\xb9\x9f\x91" - "\x72\xaf\x82\xb6\x04\xdc\x4b\x8e" - "\x51\xbc\xb0\x82\x35\xa6\xf4\x34" - "\x13\x32\xe4\xca\x60\x48\x2a\x4b" - "\xa1\xa0\x3b\x3e\x65\x00\x8f\xc5" - "\xda\x76\xb7\x0b\xf1\x69\x0d\xb4" - "\xea\xe2\x9c\x5f\x1b\xad\xd0\x3c" - "\x5c\xcf\x2a\x55\xd7\x05\xdd\xcd" - "\x86\xd4\x49\x51\x1c\xeb\x7e\xc3" - "\x0b\xf1\x2b\x1f\xa3\x5b\x91\x3f" - "\x9f\x74\x7a\x8a\xfd\x1b\x13\x0e" - "\x94\xbf\xf9\x4e\xff\xd0\x1a\x91" - "\x73\x5c\xa1\x72\x6a\xcd\x0b\x19" - "\x7c\x4e\x5b\x03\x39\x36\x97\xe1" - "\x26\x82\x6f\xb6\xbb\xde\x8e\xcc" - "\x1e\x08\x29\x85\x16\xe2\xc9\xed" - "\x03\xff\x3c\x1b\x78\x60\xf6\xde" - "\x76\xd4\xce\xcd\x94\xc8\x11\x98" - "\x55\xef\x52\x97\xca\x67\xe9\xf3" - "\xe7\xff\x72\xb1\xe9\x97\x85\xca" - "\x0a\x7e\x77\x20\xc5\xb3\x6d\xc6" - "\xd7\x2c\xac\x95\x74\xc8\xcb\xbc" - "\x2f\x80\x1e\x23\xe5\x6f\xd3\x44" - "\xb0\x7f\x22\x15\x4b\xeb\xa0\xf0" - "\x8c\xe8\x89\x1e\x64\x3e\xd9\x95" - "\xc9\x4d\x9a\x69\xc9\xf1\xb5\xf4" - "\x99\x02\x7a\x78\x57\x2a\xee\xbd" - "\x74\xd2\x0c\xc3\x98\x81\xc2\x13" - "\xee\x77\x0b\x10\x10\xe4\xbe\xa7" - "\x18\x84\x69\x77\xae\x11\x9f\x7a" - "\x02\x3a\xb5\x8c\xca\x0a\xd7\x52" - "\xaf\xe6\x56\xbb\x3c\x17\x25\x6a" - "\x9f\x6e\x9b\xf1\x9f\xdd\x5a\x38" - "\xfc\x82\xbb\xe8\x72\xc5\x53\x9e" - "\xdb\x60\x9e\xf4\xf7\x9c\x20\x3e" - "\xbb\x14\x0f\x2e\x58\x3c\xb2\xad" - "\x15\xb4\xaa\x5b\x65\x50\x16\xa8" - "\x44\x92\x77\xdb\xd4\x77\xef\x2c" - "\x8d\x6c\x01\x7d\xb7\x38\xb1\x8d" - "\xeb\x4a\x42\x7d\x19\x23\xce\x3f" - "\xf2\x62\x73\x57\x79\xa4\x18\xf2" - "\x0a\x28\x2d\xf9\x20\x14\x7b\xea" - "\xbe\x42\x1e\xe5\x31\x9d\x05\x68", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - }, { /* XTS-AES 10, XTS-AES-256, data unit 512 bytes */ - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x1c\x3b\x3a\x10\x2f\x77\x03\x86" - "\xe4\x83\x6c\x99\xe3\x70\xcf\x9b" - "\xea\x00\x80\x3f\x5e\x48\x23\x57" - "\xa4\xae\x12\xd4\x14\xa3\xe6\x3b" - "\x5d\x31\xe2\x76\xf8\xfe\x4a\x8d" - "\x66\xb3\x17\xf9\xac\x68\x3f\x44" - "\x68\x0a\x86\xac\x35\xad\xfc\x33" - "\x45\xbe\xfe\xcb\x4b\xb1\x88\xfd" - "\x57\x76\x92\x6c\x49\xa3\x09\x5e" - "\xb1\x08\xfd\x10\x98\xba\xec\x70" - "\xaa\xa6\x69\x99\xa7\x2a\x82\xf2" - "\x7d\x84\x8b\x21\xd4\xa7\x41\xb0" - "\xc5\xcd\x4d\x5f\xff\x9d\xac\x89" - "\xae\xba\x12\x29\x61\xd0\x3a\x75" - "\x71\x23\xe9\x87\x0f\x8a\xcf\x10" - "\x00\x02\x08\x87\x89\x14\x29\xca" - "\x2a\x3e\x7a\x7d\x7d\xf7\xb1\x03" - "\x55\x16\x5c\x8b\x9a\x6d\x0a\x7d" - "\xe8\xb0\x62\xc4\x50\x0d\xc4\xcd" - "\x12\x0c\x0f\x74\x18\xda\xe3\xd0" - "\xb5\x78\x1c\x34\x80\x3f\xa7\x54" - "\x21\xc7\x90\xdf\xe1\xde\x18\x34" - "\xf2\x80\xd7\x66\x7b\x32\x7f\x6c" - "\x8c\xd7\x55\x7e\x12\xac\x3a\x0f" - "\x93\xec\x05\xc5\x2e\x04\x93\xef" - "\x31\xa1\x2d\x3d\x92\x60\xf7\x9a" - "\x28\x9d\x6a\x37\x9b\xc7\x0c\x50" - "\x84\x14\x73\xd1\xa8\xcc\x81\xec" - "\x58\x3e\x96\x45\xe0\x7b\x8d\x96" - "\x70\x65\x5b\xa5\xbb\xcf\xec\xc6" - "\xdc\x39\x66\x38\x0a\xd8\xfe\xcb" - "\x17\xb6\xba\x02\x46\x9a\x02\x0a" - "\x84\xe1\x8e\x8f\x84\x25\x20\x70" - "\xc1\x3e\x9f\x1f\x28\x9b\xe5\x4f" - "\xbc\x48\x14\x57\x77\x8f\x61\x60" - "\x15\xe1\x32\x7a\x02\xb1\x40\xf1" - "\x50\x5e\xb3\x09\x32\x6d\x68\x37" - "\x8f\x83\x74\x59\x5c\x84\x9d\x84" - "\xf4\xc3\x33\xec\x44\x23\x88\x51" - "\x43\xcb\x47\xbd\x71\xc5\xed\xae" - "\x9b\xe6\x9a\x2f\xfe\xce\xb1\xbe" - "\xc9\xde\x24\x4f\xbe\x15\x99\x2b" - "\x11\xb7\x7c\x04\x0f\x12\xbd\x8f" - "\x6a\x97\x5a\x44\xa0\xf9\x0c\x29" - "\xa9\xab\xc3\xd4\xd8\x93\x92\x72" - "\x84\xc5\x87\x54\xcc\xe2\x94\x52" - "\x9f\x86\x14\xdc\xd2\xab\xa9\x91" - "\x92\x5f\xed\xc4\xae\x74\xff\xac" - "\x6e\x33\x3b\x93\xeb\x4a\xff\x04" - "\x79\xda\x9a\x41\x0e\x44\x50\xe0" - "\xdd\x7a\xe4\xc6\xe2\x91\x09\x00" - "\x57\x5d\xa4\x01\xfc\x07\x05\x9f" - "\x64\x5e\x8b\x7e\x9b\xfd\xef\x33" - "\x94\x30\x54\xff\x84\x01\x14\x93" - "\xc2\x7b\x34\x29\xea\xed\xb4\xed" - "\x53\x76\x44\x1a\x77\xed\x43\x85" - "\x1a\xd7\x7f\x16\xf5\x41\xdf\xd2" - "\x69\xd5\x0d\x6a\x5f\x14\xfb\x0a" - "\xab\x1c\xbb\x4c\x15\x50\xbe\x97" - "\xf7\xab\x40\x66\x19\x3c\x4c\xaa" - "\x77\x3d\xad\x38\x01\x4b\xd2\x09" - "\x2f\xa7\x55\xc8\x24\xbb\x5e\x54" - "\xc4\xf3\x6f\xfd\xa9\xfc\xea\x70" - "\xb9\xc6\xe6\x93\xe1\x48\xc1\x51", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - } -}; - - -static const struct cipher_testvec aes_ctr_enc_tv_template[] = { - { /* From NIST Special Publication 800-38A, Appendix F.5 */ - .key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6" - "\xab\xf7\x15\x88\x09\xcf\x4f\x3c", - .klen = 16, - .iv = "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\x87\x4d\x61\x91\xb6\x20\xe3\x26" - "\x1b\xef\x68\x64\x99\x0d\xb6\xce" - "\x98\x06\xf6\x6b\x79\x70\xfd\xff" - "\x86\x17\x18\x7b\xb9\xff\xfd\xff" - "\x5a\xe4\xdf\x3e\xdb\xd5\xd3\x5e" - "\x5b\x4f\x09\x02\x0d\xb0\x3e\xab" - "\x1e\x03\x1d\xda\x2f\xbe\x03\xd1" - "\x79\x21\x70\xa0\xf3\x00\x9c\xee", - .rlen = 64, - }, { - .key = "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" - "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" - "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", - .klen = 24, - .iv = "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\x1a\xbc\x93\x24\x17\x52\x1c\xa2" - "\x4f\x2b\x04\x59\xfe\x7e\x6e\x0b" - "\x09\x03\x39\xec\x0a\xa6\xfa\xef" - "\xd5\xcc\xc2\xc6\xf4\xce\x8e\x94" - "\x1e\x36\xb2\x6b\xd1\xeb\xc6\x70" - "\xd1\xbd\x1d\x66\x56\x20\xab\xf7" - "\x4f\x78\xa7\xf6\xd2\x98\x09\x58" - "\x5a\x97\xda\xec\x58\xc6\xb0\x50", - .rlen = 64, - }, { - .key = "\x60\x3d\xeb\x10\x15\xca\x71\xbe" - "\x2b\x73\xae\xf0\x85\x7d\x77\x81" - "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" - "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", - .klen = 32, - .iv = "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\x60\x1e\xc3\x13\x77\x57\x89\xa5" - "\xb7\xa7\xf5\x04\xbb\xf3\xd2\x28" - "\xf4\x43\xe3\xca\x4d\x62\xb5\x9a" - "\xca\x84\xe9\x90\xca\xca\xf5\xc5" - "\x2b\x09\x30\xda\xa2\x3d\xe9\x4c" - "\xe8\x70\x17\xba\x2d\x84\x98\x8d" - "\xdf\xc9\xc5\x8d\xb6\x7a\xad\xa6" - "\x13\xc2\xdd\x08\x45\x79\x41\xa6", - .rlen = 64, - }, { /* Generated with Crypto++ */ - .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55" - "\x0F\x32\x55\x78\x9B\xBE\x78\x9B" - "\xBE\xE1\x04\x27\xE1\x04\x27\x4A" - "\x6D\x90\x4A\x6D\x90\xB3\xD6\xF9", - .klen = 32, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" - "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" - "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" - "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" - "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" - "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" - "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" - "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" - "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" - "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" - "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" - "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" - "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" - "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" - "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" - "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" - "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" - "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" - "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" - "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" - "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" - "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" - "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" - "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" - "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" - "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" - "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" - "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" - "\x20\x89\x15\x7E\xE7\x50\xDC\x45" - "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" - "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" - "\xED\x56\xBF\x28\xB4\x1D\x86\x12", - .ilen = 496, - .result = "\x04\xF3\xD3\x88\x17\xEF\xDC\xEF" - "\x8B\x04\xF8\x3A\x66\x8D\x1A\x53" - "\x57\x1F\x4B\x23\xE4\xA0\xAF\xF9" - "\x69\x95\x35\x98\x8D\x4D\x8C\xC1" - "\xF0\xB2\x7F\x80\xBB\x54\x28\xA2" - "\x7A\x1B\x9F\x77\xEC\x0E\x6E\xDE" - "\xF0\xEC\xB8\xE4\x20\x62\xEE\xDB" - "\x5D\xF5\xDD\xE3\x54\xFC\xDD\xEB" - "\x6A\xEE\x65\xA1\x21\xD6\xD7\x81" - "\x47\x61\x12\x4D\xC2\x8C\xFA\x78" - "\x1F\x28\x02\x01\xC3\xFC\x1F\xEC" - "\x0F\x10\x4F\xB3\x12\x45\xC6\x3B" - "\x7E\x08\xF9\x5A\xD0\x5D\x73\x2D" - "\x58\xA4\xE5\xCB\x1C\xB4\xCE\x74" - "\x32\x41\x1F\x31\x9C\x08\xA2\x5D" - "\x67\xEB\x72\x1D\xF8\xE7\x70\x54" - "\x34\x4B\x31\x69\x84\x66\x96\x44" - "\x56\xCC\x1E\xD9\xE6\x13\x6A\xB9" - "\x2D\x0A\x05\x45\x2D\x90\xCC\xDF" - "\x16\x5C\x5F\x79\x34\x52\x54\xFE" - "\xFE\xCD\xAD\x04\x2E\xAD\x86\x06" - "\x1F\x37\xE8\x28\xBC\xD3\x8F\x5B" - "\x92\x66\x87\x3B\x8A\x0A\x1A\xCC" - "\x6E\xAB\x9F\x0B\xFA\x5C\xE6\xFD" - "\x3C\x98\x08\x12\xEC\xAA\x9E\x11" - "\xCA\xB2\x1F\xCE\x5E\x5B\xB2\x72" - "\x9C\xCC\x5D\xC5\xE0\x32\xC0\x56" - "\xD5\x45\x16\xD2\xAF\x13\x66\xF7" - "\x8C\x67\xAC\x79\xB2\xAF\x56\x27" - "\x3F\xCC\xFE\xCB\x1E\xC0\x75\xF1" - "\xA7\xC9\xC3\x1D\x8E\xDD\xF9\xD4" - "\x42\xC8\x21\x08\x16\xF7\x01\xD7" - "\xAC\x8E\x3F\x1D\x56\xC1\x06\xE4" - "\x9C\x62\xD6\xA5\x6A\x50\x44\xB3" - "\x35\x1C\x82\xB9\x10\xF9\x42\xA1" - "\xFC\x74\x9B\x44\x4F\x25\x02\xE3" - "\x08\xF5\xD4\x32\x39\x08\x11\xE8" - "\xD2\x6B\x50\x53\xD4\x08\xD1\x6B" - "\x3A\x4A\x68\x7B\x7C\xCD\x46\x5E" - "\x0D\x07\x19\xDB\x67\xD7\x98\x91" - "\xD7\x17\x10\x9B\x7B\x8A\x9B\x33" - "\xAE\xF3\x00\xA6\xD4\x15\xD9\xEA" - "\x85\x99\x22\xE8\x91\x38\x70\x83" - "\x93\x01\x24\x6C\xFA\x9A\xB9\x07" - "\xEA\x8D\x3B\xD9\x2A\x43\x59\x16" - "\x2F\x69\xEE\x84\x36\x44\x76\x98" - "\xF3\x04\x2A\x7C\x74\x3D\x29\x2B" - "\x0D\xAD\x8F\x44\x82\x9E\x57\x8D" - "\xAC\xED\x18\x1F\x50\xA4\xF5\x98" - "\x1F\xBD\x92\x91\x1B\x2D\xA6\xD6" - "\xD2\xE3\x02\xAA\x92\x3B\xC6\xB3" - "\x1B\x39\x72\xD5\x26\xCA\x04\xE0" - "\xFC\x58\x78\xBB\xB1\x3F\xA1\x9C" - "\x42\x24\x3E\x2E\x22\xBB\x4B\xBA" - "\xF4\x52\x0A\xE6\xAE\x47\xB4\x7D" - "\x1D\xA8\xBE\x81\x1A\x75\xDA\xAC" - "\xA6\x25\x1E\xEF\x3A\xC0\x6C\x63" - "\xEF\xDC\xC9\x79\x10\x26\xE8\x61" - "\x29\xFC\xA4\x05\xDF\x7D\x5C\x63" - "\x10\x09\x9B\x46\x9B\xF2\x2C\x2B" - "\xFA\x3A\x05\x4C\xFA\xD1\xFF\xFE" - "\xF1\x4C\xE5\xB2\x91\x64\x0C\x51", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, { /* Generated with Crypto++ */ - .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55" - "\x0F\x32\x55\x78\x9B\xBE\x78\x9B" - "\xBE\xE1\x04\x27\xE1\x04\x27\x4A" - "\x6D\x90\x4A\x6D\x90\xB3\xD6\xF9", - .klen = 32, - .iv = "\xE7\x82\x1D\xB8\x53\x11\xAC\x47" - "\xE2\x7D\x18\xD6\x71\x0C\xA7\x42", - .input = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" - "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" - "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" - "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" - "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" - "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" - "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" - "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" - "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" - "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" - "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" - "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" - "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" - "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" - "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" - "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" - "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" - "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" - "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" - "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" - "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" - "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" - "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" - "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" - "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" - "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" - "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" - "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" - "\x20\x89\x15\x7E\xE7\x50\xDC\x45" - "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" - "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" - "\xED\x56\xBF\x28\xB4\x1D\x86\x12" - "\x7B\xE4\x4D", - .ilen = 499, - .result = "\xDA\x4E\x3F\xBC\xE8\xB6\x3A\xA2" - "\xD5\x4D\x84\x4A\xA9\x0C\xE1\xA5" - "\xB8\x73\xBC\xF9\xBB\x59\x2F\x44" - "\x8B\xAB\x82\x6C\xB4\x32\x9A\xDE" - "\x5A\x0B\xDB\x7A\x6B\xF2\x38\x9F" - "\x06\xF7\xF7\xFF\xFF\xC0\x8A\x2E" - "\x76\xEA\x06\x32\x23\xF3\x59\x2E" - "\x75\xDE\x71\x86\x3C\x98\x23\x44" - "\x5B\xF2\xFA\x6A\x00\xBB\xC1\xAD" - "\x58\xBD\x3E\x6F\x2E\xB4\x19\x04" - "\x70\x8B\x92\x55\x23\xE9\x6A\x3A" - "\x78\x7A\x1B\x10\x85\x52\x9C\x12" - "\xE4\x55\x81\x21\xCE\x53\xD0\x3B" - "\x63\x77\x2C\x74\xD1\xF5\x60\xF3" - "\xA1\xDE\x44\x3C\x8F\x4D\x2F\xDD" - "\x8A\xFE\x3C\x42\x8E\xD3\xF2\x8E" - "\xA8\x28\x69\x65\x31\xE1\x45\x83" - "\xE4\x49\xC4\x9C\xA7\x28\xAA\x21" - "\xCD\x5D\x0F\x15\xB7\x93\x07\x26" - "\xB0\x65\x6D\x91\x90\x23\x7A\xC6" - "\xDB\x68\xB0\xA1\x8E\xA4\x76\x4E" - "\xC6\x91\x83\x20\x92\x4D\x63\x7A" - "\x45\x18\x18\x74\x19\xAD\x71\x01" - "\x6B\x23\xAD\x9D\x4E\xE4\x6E\x46" - "\xC9\x73\x7A\xF9\x02\x95\xF4\x07" - "\x0E\x7A\xA6\xC5\xAE\xFA\x15\x2C" - "\x51\x71\xF1\xDC\x22\xB6\xAC\xD8" - "\x19\x24\x44\xBC\x0C\xFB\x3C\x2D" - "\xB1\x50\x47\x15\x0E\xDB\xB6\xD7" - "\xE8\x61\xE5\x95\x52\x1E\x3E\x49" - "\x70\xE9\x66\x04\x4C\xE1\xAF\xBD" - "\xDD\x15\x3B\x20\x59\x24\xFF\xB0" - "\x39\xAA\xE7\xBF\x23\xA3\x6E\xD5" - "\x15\xF0\x61\x4F\xAE\x89\x10\x58" - "\x5A\x33\x95\x52\x2A\xB5\x77\x9C" - "\xA5\x43\x80\x40\x27\x2D\xAE\xD9" - "\x3F\xE0\x80\x94\x78\x79\xCB\x7E" - "\xAD\x12\x44\x4C\xEC\x27\xB0\xEE" - "\x0B\x05\x2A\x82\x99\x58\xBB\x7A" - "\x8D\x6D\x9D\x8E\xE2\x8E\xE7\x93" - "\x2F\xB3\x09\x8D\x06\xD5\xEE\x70" - "\x16\xAE\x35\xC5\x52\x0F\x46\x1F" - "\x71\xF9\x5E\xF2\x67\xDC\x98\x2F" - "\xA3\x23\xAA\xD5\xD0\x49\xF4\xA6" - "\xF6\xB8\x32\xCD\xD6\x85\x73\x60" - "\x59\x20\xE7\x55\x0E\x91\xE2\x0C" - "\x3F\x1C\xEB\x3D\xDF\x52\x64\xF2" - "\x7D\x8B\x5D\x63\x16\xB9\xB2\x5D" - "\x5E\xAB\xB2\x97\xAB\x78\x44\xE7" - "\xC6\x72\x20\xC5\x90\x9B\xDC\x5D" - "\xB0\xEF\x44\xEF\x87\x31\x8D\xF4" - "\xFB\x81\x5D\xF7\x96\x96\xD4\x50" - "\x89\xA7\xF6\xB9\x67\x76\x40\x9E" - "\x9D\x40\xD5\x2C\x30\xB8\x01\x8F" - "\xE4\x7B\x71\x48\xA9\xA0\xA0\x1D" - "\x87\x52\xA4\x91\xA9\xD7\xA9\x51" - "\xD9\x59\xF7\xCC\x63\x22\xC1\x8D" - "\x84\x7B\xD8\x22\x32\x5C\x6F\x1D" - "\x6E\x9F\xFA\xDD\x49\x40\xDC\x37" - "\x14\x8C\xE1\x80\x1B\xDD\x36\x2A" - "\xD0\xE9\x54\x99\x5D\xBA\x3B\x11" - "\xD8\xFE\xC9\x5B\x5C\x25\xE5\x76" - "\xFB\xF2\x3F", - .rlen = 499, - .also_non_np = 1, - .np = 2, - .tap = { 499 - 16, 16 }, - }, -}; - -static const struct cipher_testvec aes_ctr_dec_tv_template[] = { - { /* From NIST Special Publication 800-38A, Appendix F.5 */ - .key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6" - "\xab\xf7\x15\x88\x09\xcf\x4f\x3c", - .klen = 16, - .iv = "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .input = "\x87\x4d\x61\x91\xb6\x20\xe3\x26" - "\x1b\xef\x68\x64\x99\x0d\xb6\xce" - "\x98\x06\xf6\x6b\x79\x70\xfd\xff" - "\x86\x17\x18\x7b\xb9\xff\xfd\xff" - "\x5a\xe4\xdf\x3e\xdb\xd5\xd3\x5e" - "\x5b\x4f\x09\x02\x0d\xb0\x3e\xab" - "\x1e\x03\x1d\xda\x2f\xbe\x03\xd1" - "\x79\x21\x70\xa0\xf3\x00\x9c\xee", - .ilen = 64, - .result = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .rlen = 64, - }, { - .key = "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" - "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" - "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", - .klen = 24, - .iv = "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .input = "\x1a\xbc\x93\x24\x17\x52\x1c\xa2" - "\x4f\x2b\x04\x59\xfe\x7e\x6e\x0b" - "\x09\x03\x39\xec\x0a\xa6\xfa\xef" - "\xd5\xcc\xc2\xc6\xf4\xce\x8e\x94" - "\x1e\x36\xb2\x6b\xd1\xeb\xc6\x70" - "\xd1\xbd\x1d\x66\x56\x20\xab\xf7" - "\x4f\x78\xa7\xf6\xd2\x98\x09\x58" - "\x5a\x97\xda\xec\x58\xc6\xb0\x50", - .ilen = 64, - .result = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .rlen = 64, - }, { - .key = "\x60\x3d\xeb\x10\x15\xca\x71\xbe" - "\x2b\x73\xae\xf0\x85\x7d\x77\x81" - "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" - "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", - .klen = 32, - .iv = "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .input = "\x60\x1e\xc3\x13\x77\x57\x89\xa5" - "\xb7\xa7\xf5\x04\xbb\xf3\xd2\x28" - "\xf4\x43\xe3\xca\x4d\x62\xb5\x9a" - "\xca\x84\xe9\x90\xca\xca\xf5\xc5" - "\x2b\x09\x30\xda\xa2\x3d\xe9\x4c" - "\xe8\x70\x17\xba\x2d\x84\x98\x8d" - "\xdf\xc9\xc5\x8d\xb6\x7a\xad\xa6" - "\x13\xc2\xdd\x08\x45\x79\x41\xa6", - .ilen = 64, - .result = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .rlen = 64, - }, { /* Generated with Crypto++ */ - .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55" - "\x0F\x32\x55\x78\x9B\xBE\x78\x9B" - "\xBE\xE1\x04\x27\xE1\x04\x27\x4A" - "\x6D\x90\x4A\x6D\x90\xB3\xD6\xF9", - .klen = 32, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x04\xF3\xD3\x88\x17\xEF\xDC\xEF" - "\x8B\x04\xF8\x3A\x66\x8D\x1A\x53" - "\x57\x1F\x4B\x23\xE4\xA0\xAF\xF9" - "\x69\x95\x35\x98\x8D\x4D\x8C\xC1" - "\xF0\xB2\x7F\x80\xBB\x54\x28\xA2" - "\x7A\x1B\x9F\x77\xEC\x0E\x6E\xDE" - "\xF0\xEC\xB8\xE4\x20\x62\xEE\xDB" - "\x5D\xF5\xDD\xE3\x54\xFC\xDD\xEB" - "\x6A\xEE\x65\xA1\x21\xD6\xD7\x81" - "\x47\x61\x12\x4D\xC2\x8C\xFA\x78" - "\x1F\x28\x02\x01\xC3\xFC\x1F\xEC" - "\x0F\x10\x4F\xB3\x12\x45\xC6\x3B" - "\x7E\x08\xF9\x5A\xD0\x5D\x73\x2D" - "\x58\xA4\xE5\xCB\x1C\xB4\xCE\x74" - "\x32\x41\x1F\x31\x9C\x08\xA2\x5D" - "\x67\xEB\x72\x1D\xF8\xE7\x70\x54" - "\x34\x4B\x31\x69\x84\x66\x96\x44" - "\x56\xCC\x1E\xD9\xE6\x13\x6A\xB9" - "\x2D\x0A\x05\x45\x2D\x90\xCC\xDF" - "\x16\x5C\x5F\x79\x34\x52\x54\xFE" - "\xFE\xCD\xAD\x04\x2E\xAD\x86\x06" - "\x1F\x37\xE8\x28\xBC\xD3\x8F\x5B" - "\x92\x66\x87\x3B\x8A\x0A\x1A\xCC" - "\x6E\xAB\x9F\x0B\xFA\x5C\xE6\xFD" - "\x3C\x98\x08\x12\xEC\xAA\x9E\x11" - "\xCA\xB2\x1F\xCE\x5E\x5B\xB2\x72" - "\x9C\xCC\x5D\xC5\xE0\x32\xC0\x56" - "\xD5\x45\x16\xD2\xAF\x13\x66\xF7" - "\x8C\x67\xAC\x79\xB2\xAF\x56\x27" - "\x3F\xCC\xFE\xCB\x1E\xC0\x75\xF1" - "\xA7\xC9\xC3\x1D\x8E\xDD\xF9\xD4" - "\x42\xC8\x21\x08\x16\xF7\x01\xD7" - "\xAC\x8E\x3F\x1D\x56\xC1\x06\xE4" - "\x9C\x62\xD6\xA5\x6A\x50\x44\xB3" - "\x35\x1C\x82\xB9\x10\xF9\x42\xA1" - "\xFC\x74\x9B\x44\x4F\x25\x02\xE3" - "\x08\xF5\xD4\x32\x39\x08\x11\xE8" - "\xD2\x6B\x50\x53\xD4\x08\xD1\x6B" - "\x3A\x4A\x68\x7B\x7C\xCD\x46\x5E" - "\x0D\x07\x19\xDB\x67\xD7\x98\x91" - "\xD7\x17\x10\x9B\x7B\x8A\x9B\x33" - "\xAE\xF3\x00\xA6\xD4\x15\xD9\xEA" - "\x85\x99\x22\xE8\x91\x38\x70\x83" - "\x93\x01\x24\x6C\xFA\x9A\xB9\x07" - "\xEA\x8D\x3B\xD9\x2A\x43\x59\x16" - "\x2F\x69\xEE\x84\x36\x44\x76\x98" - "\xF3\x04\x2A\x7C\x74\x3D\x29\x2B" - "\x0D\xAD\x8F\x44\x82\x9E\x57\x8D" - "\xAC\xED\x18\x1F\x50\xA4\xF5\x98" - "\x1F\xBD\x92\x91\x1B\x2D\xA6\xD6" - "\xD2\xE3\x02\xAA\x92\x3B\xC6\xB3" - "\x1B\x39\x72\xD5\x26\xCA\x04\xE0" - "\xFC\x58\x78\xBB\xB1\x3F\xA1\x9C" - "\x42\x24\x3E\x2E\x22\xBB\x4B\xBA" - "\xF4\x52\x0A\xE6\xAE\x47\xB4\x7D" - "\x1D\xA8\xBE\x81\x1A\x75\xDA\xAC" - "\xA6\x25\x1E\xEF\x3A\xC0\x6C\x63" - "\xEF\xDC\xC9\x79\x10\x26\xE8\x61" - "\x29\xFC\xA4\x05\xDF\x7D\x5C\x63" - "\x10\x09\x9B\x46\x9B\xF2\x2C\x2B" - "\xFA\x3A\x05\x4C\xFA\xD1\xFF\xFE" - "\xF1\x4C\xE5\xB2\x91\x64\x0C\x51", - .ilen = 496, - .result = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" - "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" - "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" - "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" - "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" - "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" - "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" - "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" - "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" - "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" - "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" - "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" - "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" - "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" - "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" - "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" - "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" - "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" - "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" - "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" - "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" - "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" - "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" - "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" - "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" - "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" - "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" - "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" - "\x20\x89\x15\x7E\xE7\x50\xDC\x45" - "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" - "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" - "\xED\x56\xBF\x28\xB4\x1D\x86\x12", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, { /* Generated with Crypto++ */ - .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55" - "\x0F\x32\x55\x78\x9B\xBE\x78\x9B" - "\xBE\xE1\x04\x27\xE1\x04\x27\x4A" - "\x6D\x90\x4A\x6D\x90\xB3\xD6\xF9", - .klen = 32, - .iv = "\xE7\x82\x1D\xB8\x53\x11\xAC\x47" - "\xE2\x7D\x18\xD6\x71\x0C\xA7\x42", - .input = "\xDA\x4E\x3F\xBC\xE8\xB6\x3A\xA2" - "\xD5\x4D\x84\x4A\xA9\x0C\xE1\xA5" - "\xB8\x73\xBC\xF9\xBB\x59\x2F\x44" - "\x8B\xAB\x82\x6C\xB4\x32\x9A\xDE" - "\x5A\x0B\xDB\x7A\x6B\xF2\x38\x9F" - "\x06\xF7\xF7\xFF\xFF\xC0\x8A\x2E" - "\x76\xEA\x06\x32\x23\xF3\x59\x2E" - "\x75\xDE\x71\x86\x3C\x98\x23\x44" - "\x5B\xF2\xFA\x6A\x00\xBB\xC1\xAD" - "\x58\xBD\x3E\x6F\x2E\xB4\x19\x04" - "\x70\x8B\x92\x55\x23\xE9\x6A\x3A" - "\x78\x7A\x1B\x10\x85\x52\x9C\x12" - "\xE4\x55\x81\x21\xCE\x53\xD0\x3B" - "\x63\x77\x2C\x74\xD1\xF5\x60\xF3" - "\xA1\xDE\x44\x3C\x8F\x4D\x2F\xDD" - "\x8A\xFE\x3C\x42\x8E\xD3\xF2\x8E" - "\xA8\x28\x69\x65\x31\xE1\x45\x83" - "\xE4\x49\xC4\x9C\xA7\x28\xAA\x21" - "\xCD\x5D\x0F\x15\xB7\x93\x07\x26" - "\xB0\x65\x6D\x91\x90\x23\x7A\xC6" - "\xDB\x68\xB0\xA1\x8E\xA4\x76\x4E" - "\xC6\x91\x83\x20\x92\x4D\x63\x7A" - "\x45\x18\x18\x74\x19\xAD\x71\x01" - "\x6B\x23\xAD\x9D\x4E\xE4\x6E\x46" - "\xC9\x73\x7A\xF9\x02\x95\xF4\x07" - "\x0E\x7A\xA6\xC5\xAE\xFA\x15\x2C" - "\x51\x71\xF1\xDC\x22\xB6\xAC\xD8" - "\x19\x24\x44\xBC\x0C\xFB\x3C\x2D" - "\xB1\x50\x47\x15\x0E\xDB\xB6\xD7" - "\xE8\x61\xE5\x95\x52\x1E\x3E\x49" - "\x70\xE9\x66\x04\x4C\xE1\xAF\xBD" - "\xDD\x15\x3B\x20\x59\x24\xFF\xB0" - "\x39\xAA\xE7\xBF\x23\xA3\x6E\xD5" - "\x15\xF0\x61\x4F\xAE\x89\x10\x58" - "\x5A\x33\x95\x52\x2A\xB5\x77\x9C" - "\xA5\x43\x80\x40\x27\x2D\xAE\xD9" - "\x3F\xE0\x80\x94\x78\x79\xCB\x7E" - "\xAD\x12\x44\x4C\xEC\x27\xB0\xEE" - "\x0B\x05\x2A\x82\x99\x58\xBB\x7A" - "\x8D\x6D\x9D\x8E\xE2\x8E\xE7\x93" - "\x2F\xB3\x09\x8D\x06\xD5\xEE\x70" - "\x16\xAE\x35\xC5\x52\x0F\x46\x1F" - "\x71\xF9\x5E\xF2\x67\xDC\x98\x2F" - "\xA3\x23\xAA\xD5\xD0\x49\xF4\xA6" - "\xF6\xB8\x32\xCD\xD6\x85\x73\x60" - "\x59\x20\xE7\x55\x0E\x91\xE2\x0C" - "\x3F\x1C\xEB\x3D\xDF\x52\x64\xF2" - "\x7D\x8B\x5D\x63\x16\xB9\xB2\x5D" - "\x5E\xAB\xB2\x97\xAB\x78\x44\xE7" - "\xC6\x72\x20\xC5\x90\x9B\xDC\x5D" - "\xB0\xEF\x44\xEF\x87\x31\x8D\xF4" - "\xFB\x81\x5D\xF7\x96\x96\xD4\x50" - "\x89\xA7\xF6\xB9\x67\x76\x40\x9E" - "\x9D\x40\xD5\x2C\x30\xB8\x01\x8F" - "\xE4\x7B\x71\x48\xA9\xA0\xA0\x1D" - "\x87\x52\xA4\x91\xA9\xD7\xA9\x51" - "\xD9\x59\xF7\xCC\x63\x22\xC1\x8D" - "\x84\x7B\xD8\x22\x32\x5C\x6F\x1D" - "\x6E\x9F\xFA\xDD\x49\x40\xDC\x37" - "\x14\x8C\xE1\x80\x1B\xDD\x36\x2A" - "\xD0\xE9\x54\x99\x5D\xBA\x3B\x11" - "\xD8\xFE\xC9\x5B\x5C\x25\xE5\x76" - "\xFB\xF2\x3F", - .ilen = 499, - .result = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" - "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" - "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" - "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" - "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" - "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" - "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" - "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" - "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" - "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" - "\x22\x8B\x17\x80\xE9\x52\xDE\x47" - "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" - "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" - "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" - "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" - "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" - "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" - "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" - "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" - "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" - "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" - "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" - "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" - "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" - "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" - "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" - "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" - "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" - "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" - "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" - "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" - "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" - "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" - "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" - "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" - "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" - "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" - "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" - "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" - "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" - "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" - "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" - "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" - "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" - "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" - "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" - "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" - "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" - "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" - "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" - "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" - "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" - "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" - "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" - "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" - "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" - "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" - "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" - "\x20\x89\x15\x7E\xE7\x50\xDC\x45" - "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" - "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" - "\xED\x56\xBF\x28\xB4\x1D\x86\x12" - "\x7B\xE4\x4D", - .rlen = 499, - .also_non_np = 1, - .np = 2, - .tap = { 499 - 16, 16 }, - }, -}; - -static const struct cipher_testvec aes_ctr_rfc3686_enc_tv_template[] = { - { /* From RFC 3686 */ - .key = "\xae\x68\x52\xf8\x12\x10\x67\xcc" - "\x4b\xf7\xa5\x76\x55\x77\xf3\x9e" - "\x00\x00\x00\x30", - .klen = 20, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "Single block msg", - .ilen = 16, - .result = "\xe4\x09\x5d\x4f\xb7\xa7\xb3\x79" - "\x2d\x61\x75\xa3\x26\x13\x11\xb8", - .rlen = 16, - }, { - .key = "\x7e\x24\x06\x78\x17\xfa\xe0\xd7" - "\x43\xd6\xce\x1f\x32\x53\x91\x63" - "\x00\x6c\xb6\xdb", - .klen = 20, - .iv = "\xc0\x54\x3b\x59\xda\x48\xd9\x0b", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .ilen = 32, - .result = "\x51\x04\xa1\x06\x16\x8a\x72\xd9" - "\x79\x0d\x41\xee\x8e\xda\xd3\x88" - "\xeb\x2e\x1e\xfc\x46\xda\x57\xc8" - "\xfc\xe6\x30\xdf\x91\x41\xbe\x28", - .rlen = 32, - }, { - .key = "\x16\xaf\x5b\x14\x5f\xc9\xf5\x79" - "\xc1\x75\xf9\x3e\x3b\xfb\x0e\xed" - "\x86\x3d\x06\xcc\xfd\xb7\x85\x15" - "\x00\x00\x00\x48", - .klen = 28, - .iv = "\x36\x73\x3c\x14\x7d\x6d\x93\xcb", - .input = "Single block msg", - .ilen = 16, - .result = "\x4b\x55\x38\x4f\xe2\x59\xc9\xc8" - "\x4e\x79\x35\xa0\x03\xcb\xe9\x28", - .rlen = 16, - }, { - .key = "\x7c\x5c\xb2\x40\x1b\x3d\xc3\x3c" - "\x19\xe7\x34\x08\x19\xe0\xf6\x9c" - "\x67\x8c\x3d\xb8\xe6\xf6\xa9\x1a" - "\x00\x96\xb0\x3b", - .klen = 28, - .iv = "\x02\x0c\x6e\xad\xc2\xcb\x50\x0d", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .ilen = 32, - .result = "\x45\x32\x43\xfc\x60\x9b\x23\x32" - "\x7e\xdf\xaa\xfa\x71\x31\xcd\x9f" - "\x84\x90\x70\x1c\x5a\xd4\xa7\x9c" - "\xfc\x1f\xe0\xff\x42\xf4\xfb\x00", - .rlen = 32, - }, { - .key = "\x77\x6b\xef\xf2\x85\x1d\xb0\x6f" - "\x4c\x8a\x05\x42\xc8\x69\x6f\x6c" - "\x6a\x81\xaf\x1e\xec\x96\xb4\xd3" - "\x7f\xc1\xd6\x89\xe6\xc1\xc1\x04" - "\x00\x00\x00\x60", - .klen = 36, - .iv = "\xdb\x56\x72\xc9\x7a\xa8\xf0\xb2", - .input = "Single block msg", - .ilen = 16, - .result = "\x14\x5a\xd0\x1d\xbf\x82\x4e\xc7" - "\x56\x08\x63\xdc\x71\xe3\xe0\xc0", - .rlen = 16, - }, { - .key = "\xf6\xd6\x6d\x6b\xd5\x2d\x59\xbb" - "\x07\x96\x36\x58\x79\xef\xf8\x86" - "\xc6\x6d\xd5\x1a\x5b\x6a\x99\x74" - "\x4b\x50\x59\x0c\x87\xa2\x38\x84" - "\x00\xfa\xac\x24", - .klen = 36, - .iv = "\xc1\x58\x5e\xf1\x5a\x43\xd8\x75", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .ilen = 32, - .result = "\xf0\x5e\x23\x1b\x38\x94\x61\x2c" - "\x49\xee\x00\x0b\x80\x4e\xb2\xa9" - "\xb8\x30\x6b\x50\x8f\x83\x9d\x6a" - "\x55\x30\x83\x1d\x93\x44\xaf\x1c", - .rlen = 32, - }, { - // generated using Crypto++ - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x00\x00\x00\x00", - .klen = 32 + 4, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x03\x06\x09\x0c\x0f\x12\x15" - "\x18\x1b\x1e\x21\x24\x27\x2a\x2d" - "\x30\x33\x36\x39\x3c\x3f\x42\x45" - "\x48\x4b\x4e\x51\x54\x57\x5a\x5d" - "\x60\x63\x66\x69\x6c\x6f\x72\x75" - "\x78\x7b\x7e\x81\x84\x87\x8a\x8d" - "\x90\x93\x96\x99\x9c\x9f\xa2\xa5" - "\xa8\xab\xae\xb1\xb4\xb7\xba\xbd" - "\xc0\xc3\xc6\xc9\xcc\xcf\xd2\xd5" - "\xd8\xdb\xde\xe1\xe4\xe7\xea\xed" - "\xf0\xf3\xf6\xf9\xfc\xff\x02\x05" - "\x08\x0b\x0e\x11\x14\x17\x1a\x1d" - "\x20\x23\x26\x29\x2c\x2f\x32\x35" - "\x38\x3b\x3e\x41\x44\x47\x4a\x4d" - "\x50\x53\x56\x59\x5c\x5f\x62\x65" - "\x68\x6b\x6e\x71\x74\x77\x7a\x7d" - "\x80\x83\x86\x89\x8c\x8f\x92\x95" - "\x98\x9b\x9e\xa1\xa4\xa7\xaa\xad" - "\xb0\xb3\xb6\xb9\xbc\xbf\xc2\xc5" - "\xc8\xcb\xce\xd1\xd4\xd7\xda\xdd" - "\xe0\xe3\xe6\xe9\xec\xef\xf2\xf5" - "\xf8\xfb\xfe\x01\x04\x07\x0a\x0d" - "\x10\x13\x16\x19\x1c\x1f\x22\x25" - "\x28\x2b\x2e\x31\x34\x37\x3a\x3d" - "\x40\x43\x46\x49\x4c\x4f\x52\x55" - "\x58\x5b\x5e\x61\x64\x67\x6a\x6d" - "\x70\x73\x76\x79\x7c\x7f\x82\x85" - "\x88\x8b\x8e\x91\x94\x97\x9a\x9d" - "\xa0\xa3\xa6\xa9\xac\xaf\xb2\xb5" - "\xb8\xbb\xbe\xc1\xc4\xc7\xca\xcd" - "\xd0\xd3\xd6\xd9\xdc\xdf\xe2\xe5" - "\xe8\xeb\xee\xf1\xf4\xf7\xfa\xfd" - "\x00\x05\x0a\x0f\x14\x19\x1e\x23" - "\x28\x2d\x32\x37\x3c\x41\x46\x4b" - "\x50\x55\x5a\x5f\x64\x69\x6e\x73" - "\x78\x7d\x82\x87\x8c\x91\x96\x9b" - "\xa0\xa5\xaa\xaf\xb4\xb9\xbe\xc3" - "\xc8\xcd\xd2\xd7\xdc\xe1\xe6\xeb" - "\xf0\xf5\xfa\xff\x04\x09\x0e\x13" - "\x18\x1d\x22\x27\x2c\x31\x36\x3b" - "\x40\x45\x4a\x4f\x54\x59\x5e\x63" - "\x68\x6d\x72\x77\x7c\x81\x86\x8b" - "\x90\x95\x9a\x9f\xa4\xa9\xae\xb3" - "\xb8\xbd\xc2\xc7\xcc\xd1\xd6\xdb" - "\xe0\xe5\xea\xef\xf4\xf9\xfe\x03" - "\x08\x0d\x12\x17\x1c\x21\x26\x2b" - "\x30\x35\x3a\x3f\x44\x49\x4e\x53" - "\x58\x5d\x62\x67\x6c\x71\x76\x7b" - "\x80\x85\x8a\x8f\x94\x99\x9e\xa3" - "\xa8\xad\xb2\xb7\xbc\xc1\xc6\xcb" - "\xd0\xd5\xda\xdf\xe4\xe9\xee\xf3" - "\xf8\xfd\x02\x07\x0c\x11\x16\x1b" - "\x20\x25\x2a\x2f\x34\x39\x3e\x43" - "\x48\x4d\x52\x57\x5c\x61\x66\x6b" - "\x70\x75\x7a\x7f\x84\x89\x8e\x93" - "\x98\x9d\xa2\xa7\xac\xb1\xb6\xbb" - "\xc0\xc5\xca\xcf\xd4\xd9\xde\xe3" - "\xe8\xed\xf2\xf7\xfc\x01\x06\x0b" - "\x10\x15\x1a\x1f\x24\x29\x2e\x33" - "\x38\x3d\x42\x47\x4c\x51\x56\x5b" - "\x60\x65\x6a\x6f\x74\x79\x7e\x83" - "\x88\x8d\x92\x97\x9c\xa1\xa6\xab" - "\xb0\xb5\xba\xbf\xc4\xc9\xce\xd3" - "\xd8\xdd\xe2\xe7\xec\xf1\xf6\xfb" - "\x00\x07\x0e\x15\x1c\x23\x2a\x31" - "\x38\x3f\x46\x4d\x54\x5b\x62\x69" - "\x70\x77\x7e\x85\x8c\x93\x9a\xa1" - "\xa8\xaf\xb6\xbd\xc4\xcb\xd2\xd9" - "\xe0\xe7\xee\xf5\xfc\x03\x0a\x11" - "\x18\x1f\x26\x2d\x34\x3b\x42\x49" - "\x50\x57\x5e\x65\x6c\x73\x7a\x81" - "\x88\x8f\x96\x9d\xa4\xab\xb2\xb9" - "\xc0\xc7\xce\xd5\xdc\xe3\xea\xf1" - "\xf8\xff\x06\x0d\x14\x1b\x22\x29" - "\x30\x37\x3e\x45\x4c\x53\x5a\x61" - "\x68\x6f\x76\x7d\x84\x8b\x92\x99" - "\xa0\xa7\xae\xb5\xbc\xc3\xca\xd1" - "\xd8\xdf\xe6\xed\xf4\xfb\x02\x09" - "\x10\x17\x1e\x25\x2c\x33\x3a\x41" - "\x48\x4f\x56\x5d\x64\x6b\x72\x79" - "\x80\x87\x8e\x95\x9c\xa3\xaa\xb1" - "\xb8\xbf\xc6\xcd\xd4\xdb\xe2\xe9" - "\xf0\xf7\xfe\x05\x0c\x13\x1a\x21" - "\x28\x2f\x36\x3d\x44\x4b\x52\x59" - "\x60\x67\x6e\x75\x7c\x83\x8a\x91" - "\x98\x9f\xa6\xad\xb4\xbb\xc2\xc9" - "\xd0\xd7\xde\xe5\xec\xf3\xfa\x01" - "\x08\x0f\x16\x1d\x24\x2b\x32\x39" - "\x40\x47\x4e\x55\x5c\x63\x6a\x71" - "\x78\x7f\x86\x8d\x94\x9b\xa2\xa9" - "\xb0\xb7\xbe\xc5\xcc\xd3\xda\xe1" - "\xe8\xef\xf6\xfd\x04\x0b\x12\x19" - "\x20\x27\x2e\x35\x3c\x43\x4a\x51" - "\x58\x5f\x66\x6d\x74\x7b\x82\x89" - "\x90\x97\x9e\xa5\xac\xb3\xba\xc1" - "\xc8\xcf\xd6\xdd\xe4\xeb\xf2\xf9" - "\x00\x09\x12\x1b\x24\x2d\x36\x3f" - "\x48\x51\x5a\x63\x6c\x75\x7e\x87" - "\x90\x99\xa2\xab\xb4\xbd\xc6\xcf" - "\xd8\xe1\xea\xf3\xfc\x05\x0e\x17" - "\x20\x29\x32\x3b\x44\x4d\x56\x5f" - "\x68\x71\x7a\x83\x8c\x95\x9e\xa7" - "\xb0\xb9\xc2\xcb\xd4\xdd\xe6\xef" - "\xf8\x01\x0a\x13\x1c\x25\x2e\x37" - "\x40\x49\x52\x5b\x64\x6d\x76\x7f" - "\x88\x91\x9a\xa3\xac\xb5\xbe\xc7" - "\xd0\xd9\xe2\xeb\xf4\xfd\x06\x0f" - "\x18\x21\x2a\x33\x3c\x45\x4e\x57" - "\x60\x69\x72\x7b\x84\x8d\x96\x9f" - "\xa8\xb1\xba\xc3\xcc\xd5\xde\xe7" - "\xf0\xf9\x02\x0b\x14\x1d\x26\x2f" - "\x38\x41\x4a\x53\x5c\x65\x6e\x77" - "\x80\x89\x92\x9b\xa4\xad\xb6\xbf" - "\xc8\xd1\xda\xe3\xec\xf5\xfe\x07" - "\x10\x19\x22\x2b\x34\x3d\x46\x4f" - "\x58\x61\x6a\x73\x7c\x85\x8e\x97" - "\xa0\xa9\xb2\xbb\xc4\xcd\xd6\xdf" - "\xe8\xf1\xfa\x03\x0c\x15\x1e\x27" - "\x30\x39\x42\x4b\x54\x5d\x66\x6f" - "\x78\x81\x8a\x93\x9c\xa5\xae\xb7" - "\xc0\xc9\xd2\xdb\xe4\xed\xf6\xff" - "\x08\x11\x1a\x23\x2c\x35\x3e\x47" - "\x50\x59\x62\x6b\x74\x7d\x86\x8f" - "\x98\xa1\xaa\xb3\xbc\xc5\xce\xd7" - "\xe0\xe9\xf2\xfb\x04\x0d\x16\x1f" - "\x28\x31\x3a\x43\x4c\x55\x5e\x67" - "\x70\x79\x82\x8b\x94\x9d\xa6\xaf" - "\xb8\xc1\xca\xd3\xdc\xe5\xee\xf7" - "\x00\x0b\x16\x21\x2c\x37\x42\x4d" - "\x58\x63\x6e\x79\x84\x8f\x9a\xa5" - "\xb0\xbb\xc6\xd1\xdc\xe7\xf2\xfd" - "\x08\x13\x1e\x29\x34\x3f\x4a\x55" - "\x60\x6b\x76\x81\x8c\x97\xa2\xad" - "\xb8\xc3\xce\xd9\xe4\xef\xfa\x05" - "\x10\x1b\x26\x31\x3c\x47\x52\x5d" - "\x68\x73\x7e\x89\x94\x9f\xaa\xb5" - "\xc0\xcb\xd6\xe1\xec\xf7\x02\x0d" - "\x18\x23\x2e\x39\x44\x4f\x5a\x65" - "\x70\x7b\x86\x91\x9c\xa7\xb2\xbd" - "\xc8\xd3\xde\xe9\xf4\xff\x0a\x15" - "\x20\x2b\x36\x41\x4c\x57\x62\x6d" - "\x78\x83\x8e\x99\xa4\xaf\xba\xc5" - "\xd0\xdb\xe6\xf1\xfc\x07\x12\x1d" - "\x28\x33\x3e\x49\x54\x5f\x6a\x75" - "\x80\x8b\x96\xa1\xac\xb7\xc2\xcd" - "\xd8\xe3\xee\xf9\x04\x0f\x1a\x25" - "\x30\x3b\x46\x51\x5c\x67\x72\x7d" - "\x88\x93\x9e\xa9\xb4\xbf\xca\xd5" - "\xe0\xeb\xf6\x01\x0c\x17\x22\x2d" - "\x38\x43\x4e\x59\x64\x6f\x7a\x85" - "\x90\x9b\xa6\xb1\xbc\xc7\xd2\xdd" - "\xe8\xf3\xfe\x09\x14\x1f\x2a\x35" - "\x40\x4b\x56\x61\x6c\x77\x82\x8d" - "\x98\xa3\xae\xb9\xc4\xcf\xda\xe5" - "\xf0\xfb\x06\x11\x1c\x27\x32\x3d" - "\x48\x53\x5e\x69\x74\x7f\x8a\x95" - "\xa0\xab\xb6\xc1\xcc\xd7\xe2\xed" - "\xf8\x03\x0e\x19\x24\x2f\x3a\x45" - "\x50\x5b\x66\x71\x7c\x87\x92\x9d" - "\xa8\xb3\xbe\xc9\xd4\xdf\xea\xf5" - "\x00\x0d\x1a\x27\x34\x41\x4e\x5b" - "\x68\x75\x82\x8f\x9c\xa9\xb6\xc3" - "\xd0\xdd\xea\xf7\x04\x11\x1e\x2b" - "\x38\x45\x52\x5f\x6c\x79\x86\x93" - "\xa0\xad\xba\xc7\xd4\xe1\xee\xfb" - "\x08\x15\x22\x2f\x3c\x49\x56\x63" - "\x70\x7d\x8a\x97\xa4\xb1\xbe\xcb" - "\xd8\xe5\xf2\xff\x0c\x19\x26\x33" - "\x40\x4d\x5a\x67\x74\x81\x8e\x9b" - "\xa8\xb5\xc2\xcf\xdc\xe9\xf6\x03" - "\x10\x1d\x2a\x37\x44\x51\x5e\x6b" - "\x78\x85\x92\x9f\xac\xb9\xc6\xd3" - "\xe0\xed\xfa\x07\x14\x21\x2e\x3b" - "\x48\x55\x62\x6f\x7c\x89\x96\xa3" - "\xb0\xbd\xca\xd7\xe4\xf1\xfe\x0b" - "\x18\x25\x32\x3f\x4c\x59\x66\x73" - "\x80\x8d\x9a\xa7\xb4\xc1\xce\xdb" - "\xe8\xf5\x02\x0f\x1c\x29\x36\x43" - "\x50\x5d\x6a\x77\x84\x91\x9e\xab" - "\xb8\xc5\xd2\xdf\xec\xf9\x06\x13" - "\x20\x2d\x3a\x47\x54\x61\x6e\x7b" - "\x88\x95\xa2\xaf\xbc\xc9\xd6\xe3" - "\xf0\xfd\x0a\x17\x24\x31\x3e\x4b" - "\x58\x65\x72\x7f\x8c\x99\xa6\xb3" - "\xc0\xcd\xda\xe7\xf4\x01\x0e\x1b" - "\x28\x35\x42\x4f\x5c\x69\x76\x83" - "\x90\x9d\xaa\xb7\xc4\xd1\xde\xeb" - "\xf8\x05\x12\x1f\x2c\x39\x46\x53" - "\x60\x6d\x7a\x87\x94\xa1\xae\xbb" - "\xc8\xd5\xe2\xef\xfc\x09\x16\x23" - "\x30\x3d\x4a\x57\x64\x71\x7e\x8b" - "\x98\xa5\xb2\xbf\xcc\xd9\xe6\xf3" - "\x00\x0f\x1e\x2d\x3c\x4b\x5a\x69" - "\x78\x87\x96\xa5\xb4\xc3\xd2\xe1" - "\xf0\xff\x0e\x1d\x2c\x3b\x4a\x59" - "\x68\x77\x86\x95\xa4\xb3\xc2\xd1" - "\xe0\xef\xfe\x0d\x1c\x2b\x3a\x49" - "\x58\x67\x76\x85\x94\xa3\xb2\xc1" - "\xd0\xdf\xee\xfd\x0c\x1b\x2a\x39" - "\x48\x57\x66\x75\x84\x93\xa2\xb1" - "\xc0\xcf\xde\xed\xfc\x0b\x1a\x29" - "\x38\x47\x56\x65\x74\x83\x92\xa1" - "\xb0\xbf\xce\xdd\xec\xfb\x0a\x19" - "\x28\x37\x46\x55\x64\x73\x82\x91" - "\xa0\xaf\xbe\xcd\xdc\xeb\xfa\x09" - "\x18\x27\x36\x45\x54\x63\x72\x81" - "\x90\x9f\xae\xbd\xcc\xdb\xea\xf9" - "\x08\x17\x26\x35\x44\x53\x62\x71" - "\x80\x8f\x9e\xad\xbc\xcb\xda\xe9" - "\xf8\x07\x16\x25\x34\x43\x52\x61" - "\x70\x7f\x8e\x9d\xac\xbb\xca\xd9" - "\xe8\xf7\x06\x15\x24\x33\x42\x51" - "\x60\x6f\x7e\x8d\x9c\xab\xba\xc9" - "\xd8\xe7\xf6\x05\x14\x23\x32\x41" - "\x50\x5f\x6e\x7d\x8c\x9b\xaa\xb9" - "\xc8\xd7\xe6\xf5\x04\x13\x22\x31" - "\x40\x4f\x5e\x6d\x7c\x8b\x9a\xa9" - "\xb8\xc7\xd6\xe5\xf4\x03\x12\x21" - "\x30\x3f\x4e\x5d\x6c\x7b\x8a\x99" - "\xa8\xb7\xc6\xd5\xe4\xf3\x02\x11" - "\x20\x2f\x3e\x4d\x5c\x6b\x7a\x89" - "\x98\xa7\xb6\xc5\xd4\xe3\xf2\x01" - "\x10\x1f\x2e\x3d\x4c\x5b\x6a\x79" - "\x88\x97\xa6\xb5\xc4\xd3\xe2\xf1" - "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x88\x99\xaa\xbb\xcc\xdd\xee\xff" - "\x10\x21\x32\x43\x54\x65\x76\x87" - "\x98\xa9\xba\xcb\xdc\xed\xfe\x0f" - "\x20\x31\x42\x53\x64\x75\x86\x97" - "\xa8\xb9\xca\xdb\xec\xfd\x0e\x1f" - "\x30\x41\x52\x63\x74\x85\x96\xa7" - "\xb8\xc9\xda\xeb\xfc\x0d\x1e\x2f" - "\x40\x51\x62\x73\x84\x95\xa6\xb7" - "\xc8\xd9\xea\xfb\x0c\x1d\x2e\x3f" - "\x50\x61\x72\x83\x94\xa5\xb6\xc7" - "\xd8\xe9\xfa\x0b\x1c\x2d\x3e\x4f" - "\x60\x71\x82\x93\xa4\xb5\xc6\xd7" - "\xe8\xf9\x0a\x1b\x2c\x3d\x4e\x5f" - "\x70\x81\x92\xa3\xb4\xc5\xd6\xe7" - "\xf8\x09\x1a\x2b\x3c\x4d\x5e\x6f" - "\x80\x91\xa2\xb3\xc4\xd5\xe6\xf7" - "\x08\x19\x2a\x3b\x4c\x5d\x6e\x7f" - "\x90\xa1\xb2\xc3\xd4\xe5\xf6\x07" - "\x18\x29\x3a\x4b\x5c\x6d\x7e\x8f" - "\xa0\xb1\xc2\xd3\xe4\xf5\x06\x17" - "\x28\x39\x4a\x5b\x6c\x7d\x8e\x9f" - "\xb0\xc1\xd2\xe3\xf4\x05\x16\x27" - "\x38\x49\x5a\x6b\x7c\x8d\x9e\xaf" - "\xc0\xd1\xe2\xf3\x04\x15\x26\x37" - "\x48\x59\x6a\x7b\x8c\x9d\xae\xbf" - "\xd0\xe1\xf2\x03\x14\x25\x36\x47" - "\x58\x69\x7a\x8b\x9c\xad\xbe\xcf" - "\xe0\xf1\x02\x13\x24\x35\x46\x57" - "\x68\x79\x8a\x9b\xac\xbd\xce\xdf" - "\xf0\x01\x12\x23\x34\x45\x56\x67" - "\x78\x89\x9a\xab\xbc\xcd\xde\xef" - "\x00\x13\x26\x39\x4c\x5f\x72\x85" - "\x98\xab\xbe\xd1\xe4\xf7\x0a\x1d" - "\x30\x43\x56\x69\x7c\x8f\xa2\xb5" - "\xc8\xdb\xee\x01\x14\x27\x3a\x4d" - "\x60\x73\x86\x99\xac\xbf\xd2\xe5" - "\xf8\x0b\x1e\x31\x44\x57\x6a\x7d" - "\x90\xa3\xb6\xc9\xdc\xef\x02\x15" - "\x28\x3b\x4e\x61\x74\x87\x9a\xad" - "\xc0\xd3\xe6\xf9\x0c\x1f\x32\x45" - "\x58\x6b\x7e\x91\xa4\xb7\xca\xdd" - "\xf0\x03\x16\x29\x3c\x4f\x62\x75" - "\x88\x9b\xae\xc1\xd4\xe7\xfa\x0d" - "\x20\x33\x46\x59\x6c\x7f\x92\xa5" - "\xb8\xcb\xde\xf1\x04\x17\x2a\x3d" - "\x50\x63\x76\x89\x9c\xaf\xc2\xd5" - "\xe8\xfb\x0e\x21\x34\x47\x5a\x6d" - "\x80\x93\xa6\xb9\xcc\xdf\xf2\x05" - "\x18\x2b\x3e\x51\x64\x77\x8a\x9d" - "\xb0\xc3\xd6\xe9\xfc\x0f\x22\x35" - "\x48\x5b\x6e\x81\x94\xa7\xba\xcd" - "\xe0\xf3\x06\x19\x2c\x3f\x52\x65" - "\x78\x8b\x9e\xb1\xc4\xd7\xea\xfd" - "\x10\x23\x36\x49\x5c\x6f\x82\x95" - "\xa8\xbb\xce\xe1\xf4\x07\x1a\x2d" - "\x40\x53\x66\x79\x8c\x9f\xb2\xc5" - "\xd8\xeb\xfe\x11\x24\x37\x4a\x5d" - "\x70\x83\x96\xa9\xbc\xcf\xe2\xf5" - "\x08\x1b\x2e\x41\x54\x67\x7a\x8d" - "\xa0\xb3\xc6\xd9\xec\xff\x12\x25" - "\x38\x4b\x5e\x71\x84\x97\xaa\xbd" - "\xd0\xe3\xf6\x09\x1c\x2f\x42\x55" - "\x68\x7b\x8e\xa1\xb4\xc7\xda\xed" - "\x00\x15\x2a\x3f\x54\x69\x7e\x93" - "\xa8\xbd\xd2\xe7\xfc\x11\x26\x3b" - "\x50\x65\x7a\x8f\xa4\xb9\xce\xe3" - "\xf8\x0d\x22\x37\x4c\x61\x76\x8b" - "\xa0\xb5\xca\xdf\xf4\x09\x1e\x33" - "\x48\x5d\x72\x87\x9c\xb1\xc6\xdb" - "\xf0\x05\x1a\x2f\x44\x59\x6e\x83" - "\x98\xad\xc2\xd7\xec\x01\x16\x2b" - "\x40\x55\x6a\x7f\x94\xa9\xbe\xd3" - "\xe8\xfd\x12\x27\x3c\x51\x66\x7b" - "\x90\xa5\xba\xcf\xe4\xf9\x0e\x23" - "\x38\x4d\x62\x77\x8c\xa1\xb6\xcb" - "\xe0\xf5\x0a\x1f\x34\x49\x5e\x73" - "\x88\x9d\xb2\xc7\xdc\xf1\x06\x1b" - "\x30\x45\x5a\x6f\x84\x99\xae\xc3" - "\xd8\xed\x02\x17\x2c\x41\x56\x6b" - "\x80\x95\xaa\xbf\xd4\xe9\xfe\x13" - "\x28\x3d\x52\x67\x7c\x91\xa6\xbb" - "\xd0\xe5\xfa\x0f\x24\x39\x4e\x63" - "\x78\x8d\xa2\xb7\xcc\xe1\xf6\x0b" - "\x20\x35\x4a\x5f\x74\x89\x9e\xb3" - "\xc8\xdd\xf2\x07\x1c\x31\x46\x5b" - "\x70\x85\x9a\xaf\xc4\xd9\xee\x03" - "\x18\x2d\x42\x57\x6c\x81\x96\xab" - "\xc0\xd5\xea\xff\x14\x29\x3e\x53" - "\x68\x7d\x92\xa7\xbc\xd1\xe6\xfb" - "\x10\x25\x3a\x4f\x64\x79\x8e\xa3" - "\xb8\xcd\xe2\xf7\x0c\x21\x36\x4b" - "\x60\x75\x8a\x9f\xb4\xc9\xde\xf3" - "\x08\x1d\x32\x47\x5c\x71\x86\x9b" - "\xb0\xc5\xda\xef\x04\x19\x2e\x43" - "\x58\x6d\x82\x97\xac\xc1\xd6\xeb" - "\x00\x17\x2e\x45\x5c\x73\x8a\xa1" - "\xb8\xcf\xe6\xfd\x14\x2b\x42\x59" - "\x70\x87\x9e\xb5\xcc\xe3\xfa\x11" - "\x28\x3f\x56\x6d\x84\x9b\xb2\xc9" - "\xe0\xf7\x0e\x25\x3c\x53\x6a\x81" - "\x98\xaf\xc6\xdd\xf4\x0b\x22\x39" - "\x50\x67\x7e\x95\xac\xc3\xda\xf1" - "\x08\x1f\x36\x4d\x64\x7b\x92\xa9" - "\xc0\xd7\xee\x05\x1c\x33\x4a\x61" - "\x78\x8f\xa6\xbd\xd4\xeb\x02\x19" - "\x30\x47\x5e\x75\x8c\xa3\xba\xd1" - "\xe8\xff\x16\x2d\x44\x5b\x72\x89" - "\xa0\xb7\xce\xe5\xfc\x13\x2a\x41" - "\x58\x6f\x86\x9d\xb4\xcb\xe2\xf9" - "\x10\x27\x3e\x55\x6c\x83\x9a\xb1" - "\xc8\xdf\xf6\x0d\x24\x3b\x52\x69" - "\x80\x97\xae\xc5\xdc\xf3\x0a\x21" - "\x38\x4f\x66\x7d\x94\xab\xc2\xd9" - "\xf0\x07\x1e\x35\x4c\x63\x7a\x91" - "\xa8\xbf\xd6\xed\x04\x1b\x32\x49" - "\x60\x77\x8e\xa5\xbc\xd3\xea\x01" - "\x18\x2f\x46\x5d\x74\x8b\xa2\xb9" - "\xd0\xe7\xfe\x15\x2c\x43\x5a\x71" - "\x88\x9f\xb6\xcd\xe4\xfb\x12\x29" - "\x40\x57\x6e\x85\x9c\xb3\xca\xe1" - "\xf8\x0f\x26\x3d\x54\x6b\x82\x99" - "\xb0\xc7\xde\xf5\x0c\x23\x3a\x51" - "\x68\x7f\x96\xad\xc4\xdb\xf2\x09" - "\x20\x37\x4e\x65\x7c\x93\xaa\xc1" - "\xd8\xef\x06\x1d\x34\x4b\x62\x79" - "\x90\xa7\xbe\xd5\xec\x03\x1a\x31" - "\x48\x5f\x76\x8d\xa4\xbb\xd2\xe9" - "\x00\x19\x32\x4b\x64\x7d\x96\xaf" - "\xc8\xe1\xfa\x13\x2c\x45\x5e\x77" - "\x90\xa9\xc2\xdb\xf4\x0d\x26\x3f" - "\x58\x71\x8a\xa3\xbc\xd5\xee\x07" - "\x20\x39\x52\x6b\x84\x9d\xb6\xcf" - "\xe8\x01\x1a\x33\x4c\x65\x7e\x97" - "\xb0\xc9\xe2\xfb\x14\x2d\x46\x5f" - "\x78\x91\xaa\xc3\xdc\xf5\x0e\x27" - "\x40\x59\x72\x8b\xa4\xbd\xd6\xef" - "\x08\x21\x3a\x53\x6c\x85\x9e\xb7" - "\xd0\xe9\x02\x1b\x34\x4d\x66\x7f" - "\x98\xb1\xca\xe3\xfc\x15\x2e\x47" - "\x60\x79\x92\xab\xc4\xdd\xf6\x0f" - "\x28\x41\x5a\x73\x8c\xa5\xbe\xd7" - "\xf0\x09\x22\x3b\x54\x6d\x86\x9f" - "\xb8\xd1\xea\x03\x1c\x35\x4e\x67" - "\x80\x99\xb2\xcb\xe4\xfd\x16\x2f" - "\x48\x61\x7a\x93\xac\xc5\xde\xf7" - "\x10\x29\x42\x5b\x74\x8d\xa6\xbf" - "\xd8\xf1\x0a\x23\x3c\x55\x6e\x87" - "\xa0\xb9\xd2\xeb\x04\x1d\x36\x4f" - "\x68\x81\x9a\xb3\xcc\xe5\xfe\x17" - "\x30\x49\x62\x7b\x94\xad\xc6\xdf" - "\xf8\x11\x2a\x43\x5c\x75\x8e\xa7" - "\xc0\xd9\xf2\x0b\x24\x3d\x56\x6f" - "\x88\xa1\xba\xd3\xec\x05\x1e\x37" - "\x50\x69\x82\x9b\xb4\xcd\xe6\xff" - "\x18\x31\x4a\x63\x7c\x95\xae\xc7" - "\xe0\xf9\x12\x2b\x44\x5d\x76\x8f" - "\xa8\xc1\xda\xf3\x0c\x25\x3e\x57" - "\x70\x89\xa2\xbb\xd4\xed\x06\x1f" - "\x38\x51\x6a\x83\x9c\xb5\xce\xe7" - "\x00\x1b\x36\x51\x6c\x87\xa2\xbd" - "\xd8\xf3\x0e\x29\x44\x5f\x7a\x95" - "\xb0\xcb\xe6\x01\x1c\x37\x52\x6d" - "\x88\xa3\xbe\xd9\xf4\x0f\x2a\x45" - "\x60\x7b\x96\xb1\xcc\xe7\x02\x1d" - "\x38\x53\x6e\x89\xa4\xbf\xda\xf5" - "\x10\x2b\x46\x61\x7c\x97\xb2\xcd" - "\xe8\x03\x1e\x39\x54\x6f\x8a\xa5" - "\xc0\xdb\xf6\x11\x2c\x47\x62\x7d" - "\x98\xb3\xce\xe9\x04\x1f\x3a\x55" - "\x70\x8b\xa6\xc1\xdc\xf7\x12\x2d" - "\x48\x63\x7e\x99\xb4\xcf\xea\x05" - "\x20\x3b\x56\x71\x8c\xa7\xc2\xdd" - "\xf8\x13\x2e\x49\x64\x7f\x9a\xb5" - "\xd0\xeb\x06\x21\x3c\x57\x72\x8d" - "\xa8\xc3\xde\xf9\x14\x2f\x4a\x65" - "\x80\x9b\xb6\xd1\xec\x07\x22\x3d" - "\x58\x73\x8e\xa9\xc4\xdf\xfa\x15" - "\x30\x4b\x66\x81\x9c\xb7\xd2\xed" - "\x08\x23\x3e\x59\x74\x8f\xaa\xc5" - "\xe0\xfb\x16\x31\x4c\x67\x82\x9d" - "\xb8\xd3\xee\x09\x24\x3f\x5a\x75" - "\x90\xab\xc6\xe1\xfc\x17\x32\x4d" - "\x68\x83\x9e\xb9\xd4\xef\x0a\x25" - "\x40\x5b\x76\x91\xac\xc7\xe2\xfd" - "\x18\x33\x4e\x69\x84\x9f\xba\xd5" - "\xf0\x0b\x26\x41\x5c\x77\x92\xad" - "\xc8\xe3\xfe\x19\x34\x4f\x6a\x85" - "\xa0\xbb\xd6\xf1\x0c\x27\x42\x5d" - "\x78\x93\xae\xc9\xe4\xff\x1a\x35" - "\x50\x6b\x86\xa1\xbc\xd7\xf2\x0d" - "\x28\x43\x5e\x79\x94\xaf\xca\xe5" - "\x00\x1d\x3a\x57\x74\x91\xae\xcb" - "\xe8\x05\x22\x3f\x5c\x79\x96\xb3" - "\xd0\xed\x0a\x27\x44\x61\x7e\x9b" - "\xb8\xd5\xf2\x0f\x2c\x49\x66\x83" - "\xa0\xbd\xda\xf7\x14\x31\x4e\x6b" - "\x88\xa5\xc2\xdf\xfc\x19\x36\x53" - "\x70\x8d\xaa\xc7\xe4\x01\x1e\x3b" - "\x58\x75\x92\xaf\xcc\xe9\x06\x23" - "\x40\x5d\x7a\x97\xb4\xd1\xee\x0b" - "\x28\x45\x62\x7f\x9c\xb9\xd6\xf3" - "\x10\x2d\x4a\x67\x84\xa1\xbe\xdb" - "\xf8\x15\x32\x4f\x6c\x89\xa6\xc3" - "\xe0\xfd\x1a\x37\x54\x71\x8e\xab" - "\xc8\xe5\x02\x1f\x3c\x59\x76\x93" - "\xb0\xcd\xea\x07\x24\x41\x5e\x7b" - "\x98\xb5\xd2\xef\x0c\x29\x46\x63" - "\x80\x9d\xba\xd7\xf4\x11\x2e\x4b" - "\x68\x85\xa2\xbf\xdc\xf9\x16\x33" - "\x50\x6d\x8a\xa7\xc4\xe1\xfe\x1b" - "\x38\x55\x72\x8f\xac\xc9\xe6\x03" - "\x20\x3d\x5a\x77\x94\xb1\xce\xeb" - "\x08\x25\x42\x5f\x7c\x99\xb6\xd3" - "\xf0\x0d\x2a\x47\x64\x81\x9e\xbb" - "\xd8\xf5\x12\x2f\x4c\x69\x86\xa3" - "\xc0\xdd\xfa\x17\x34\x51\x6e\x8b" - "\xa8\xc5\xe2\xff\x1c\x39\x56\x73" - "\x90\xad\xca\xe7\x04\x21\x3e\x5b" - "\x78\x95\xb2\xcf\xec\x09\x26\x43" - "\x60\x7d\x9a\xb7\xd4\xf1\x0e\x2b" - "\x48\x65\x82\x9f\xbc\xd9\xf6\x13" - "\x30\x4d\x6a\x87\xa4\xc1\xde\xfb" - "\x18\x35\x52\x6f\x8c\xa9\xc6\xe3" - "\x00\x1f\x3e\x5d\x7c\x9b\xba\xd9" - "\xf8\x17\x36\x55\x74\x93\xb2\xd1" - "\xf0\x0f\x2e\x4d\x6c\x8b\xaa\xc9" - "\xe8\x07\x26\x45\x64\x83\xa2\xc1" - "\xe0\xff\x1e\x3d\x5c\x7b\x9a\xb9" - "\xd8\xf7\x16\x35\x54\x73\x92\xb1" - "\xd0\xef\x0e\x2d\x4c\x6b\x8a\xa9" - "\xc8\xe7\x06\x25\x44\x63\x82\xa1" - "\xc0\xdf\xfe\x1d\x3c\x5b\x7a\x99" - "\xb8\xd7\xf6\x15\x34\x53\x72\x91" - "\xb0\xcf\xee\x0d\x2c\x4b\x6a\x89" - "\xa8\xc7\xe6\x05\x24\x43\x62\x81" - "\xa0\xbf\xde\xfd\x1c\x3b\x5a\x79" - "\x98\xb7\xd6\xf5\x14\x33\x52\x71" - "\x90\xaf\xce\xed\x0c\x2b\x4a\x69" - "\x88\xa7\xc6\xe5\x04\x23\x42\x61" - "\x80\x9f\xbe\xdd\xfc\x1b\x3a\x59" - "\x78\x97\xb6\xd5\xf4\x13\x32\x51" - "\x70\x8f\xae\xcd\xec\x0b\x2a\x49" - "\x68\x87\xa6\xc5\xe4\x03\x22\x41" - "\x60\x7f\x9e\xbd\xdc\xfb\x1a\x39" - "\x58\x77\x96\xb5\xd4\xf3\x12\x31" - "\x50\x6f\x8e\xad\xcc\xeb\x0a\x29" - "\x48\x67\x86\xa5\xc4\xe3\x02\x21" - "\x40\x5f\x7e\x9d\xbc\xdb\xfa\x19" - "\x38\x57\x76\x95\xb4\xd3\xf2\x11" - "\x30\x4f\x6e\x8d\xac\xcb\xea\x09" - "\x28\x47\x66\x85\xa4\xc3\xe2\x01" - "\x20\x3f\x5e\x7d\x9c\xbb\xda\xf9" - "\x18\x37\x56\x75\x94\xb3\xd2\xf1" - "\x10\x2f\x4e\x6d\x8c\xab\xca\xe9" - "\x08\x27\x46\x65\x84\xa3\xc2\xe1" - "\x00\x21\x42\x63", - .ilen = 4100, - .result = - "\xf0\x5c\x74\xad\x4e\xbc\x99\xe2" - "\xae\xff\x91\x3a\x44\xcf\x38\x32" - "\x1e\xad\xa7\xcd\xa1\x39\x95\xaa" - "\x10\xb1\xb3\x2e\x04\x31\x8f\x86" - "\xf2\x62\x74\x70\x0c\xa4\x46\x08" - "\xa8\xb7\x99\xa8\xe9\xd2\x73\x79" - "\x7e\x6e\xd4\x8f\x1e\xc7\x8e\x31" - "\x0b\xfa\x4b\xce\xfd\xf3\x57\x71" - "\xe9\x46\x03\xa5\x3d\x34\x00\xe2" - "\x18\xff\x75\x6d\x06\x2d\x00\xab" - "\xb9\x3e\x6c\x59\xc5\x84\x06\xb5" - "\x8b\xd0\x89\x9c\x4a\x79\x16\xc6" - "\x3d\x74\x54\xfa\x44\xcd\x23\x26" - "\x5c\xcf\x7e\x28\x92\x32\xbf\xdf" - "\xa7\x20\x3c\x74\x58\x2a\x9a\xde" - "\x61\x00\x1c\x4f\xff\x59\xc4\x22" - "\xac\x3c\xd0\xe8\x6c\xf9\x97\x1b" - "\x58\x9b\xad\x71\xe8\xa9\xb5\x0d" - "\xee\x2f\x04\x1f\x7f\xbc\x99\xee" - "\x84\xff\x42\x60\xdc\x3a\x18\xa5" - "\x81\xf9\xef\xdc\x7a\x0f\x65\x41" - "\x2f\xa3\xd3\xf9\xc2\xcb\xc0\x4d" - "\x8f\xd3\x76\x96\xad\x49\x6d\x38" - "\x3d\x39\x0b\x6c\x80\xb7\x54\x69" - "\xf0\x2c\x90\x02\x29\x0d\x1c\x12" - "\xad\x55\xc3\x8b\x68\xd9\xcc\xb3" - "\xb2\x64\x33\x90\x5e\xca\x4b\xe2" - "\xfb\x75\xdc\x63\xf7\x9f\x82\x74" - "\xf0\xc9\xaa\x7f\xe9\x2a\x9b\x33" - "\xbc\x88\x00\x7f\xca\xb2\x1f\x14" - "\xdb\xc5\x8e\x7b\x11\x3c\x3e\x08" - "\xf3\x83\xe8\xe0\x94\x86\x2e\x92" - "\x78\x6b\x01\xc9\xc7\x83\xba\x21" - "\x6a\x25\x15\x33\x4e\x45\x08\xec" - "\x35\xdb\xe0\x6e\x31\x51\x79\xa9" - "\x42\x44\x65\xc1\xa0\xf1\xf9\x2a" - "\x70\xd5\xb6\xc6\xc1\x8c\x39\xfc" - "\x25\xa6\x55\xd9\xdd\x2d\x4c\xec" - "\x49\xc6\xeb\x0e\xa8\x25\x2a\x16" - "\x1b\x66\x84\xda\xe2\x92\xe5\xc0" - "\xc8\x53\x07\xaf\x80\x84\xec\xfd" - "\xcd\xd1\x6e\xcd\x6f\x6a\xf5\x36" - "\xc5\x15\xe5\x25\x7d\x77\xd1\x1a" - "\x93\x36\xa9\xcf\x7c\xa4\x54\x4a" - "\x06\x51\x48\x4e\xf6\x59\x87\xd2" - "\x04\x02\xef\xd3\x44\xde\x76\x31" - "\xb3\x34\x17\x1b\x9d\x66\x11\x9f" - "\x1e\xcc\x17\xe9\xc7\x3c\x1b\xe7" - "\xcb\x50\x08\xfc\xdc\x2b\x24\xdb" - "\x65\x83\xd0\x3b\xe3\x30\xea\x94" - "\x6c\xe7\xe8\x35\x32\xc7\xdb\x64" - "\xb4\x01\xab\x36\x2c\x77\x13\xaf" - "\xf8\x2b\x88\x3f\x54\x39\xc4\x44" - "\xfe\xef\x6f\x68\x34\xbe\x0f\x05" - "\x16\x6d\xf6\x0a\x30\xe7\xe3\xed" - "\xc4\xde\x3c\x1b\x13\xd8\xdb\xfe" - "\x41\x62\xe5\x28\xd4\x8d\xa3\xc7" - "\x93\x97\xc6\x48\x45\x1d\x9f\x83" - "\xdf\x4b\x40\x3e\x42\x25\x87\x80" - "\x4c\x7d\xa8\xd4\x98\x23\x95\x75" - "\x41\x8c\xda\x41\x9b\xd4\xa7\x06" - "\xb5\xf1\x71\x09\x53\xbe\xca\xbf" - "\x32\x03\xed\xf0\x50\x1c\x56\x39" - "\x5b\xa4\x75\x18\xf7\x9b\x58\xef" - "\x53\xfc\x2a\x38\x23\x15\x75\xcd" - "\x45\xe5\x5a\x82\x55\xba\x21\xfa" - "\xd4\xbd\xc6\x94\x7c\xc5\x80\x12" - "\xf7\x4b\x32\xc4\x9a\x82\xd8\x28" - "\x8f\xd9\xc2\x0f\x60\x03\xbe\x5e" - "\x21\xd6\x5f\x58\xbf\x5c\xb1\x32" - "\x82\x8d\xa9\xe5\xf2\x66\x1a\xc0" - "\xa0\xbc\x58\x2f\x71\xf5\x2f\xed" - "\xd1\x26\xb9\xd8\x49\x5a\x07\x19" - "\x01\x7c\x59\xb0\xf8\xa4\xb7\xd3" - "\x7b\x1a\x8c\x38\xf4\x50\xa4\x59" - "\xb0\xcc\x41\x0b\x88\x7f\xe5\x31" - "\xb3\x42\xba\xa2\x7e\xd4\x32\x71" - "\x45\x87\x48\xa9\xc2\xf2\x89\xb3" - "\xe4\xa7\x7e\x52\x15\x61\xfa\xfe" - "\xc9\xdd\x81\xeb\x13\xab\xab\xc3" - "\x98\x59\xd8\x16\x3d\x14\x7a\x1c" - "\x3c\x41\x9a\x16\x16\x9b\xd2\xd2" - "\x69\x3a\x29\x23\xac\x86\x32\xa5" - "\x48\x9c\x9e\xf3\x47\x77\x81\x70" - "\x24\xe8\x85\xd2\xf5\xb5\xfa\xff" - "\x59\x6a\xd3\x50\x59\x43\x59\xde" - "\xd9\xf1\x55\xa5\x0c\xc3\x1a\x1a" - "\x18\x34\x0d\x1a\x63\x33\xed\x10" - "\xe0\x1d\x2a\x18\xd2\xc0\x54\xa8" - "\xca\xb5\x9a\xd3\xdd\xca\x45\x84" - "\x50\xe7\x0f\xfe\xa4\x99\x5a\xbe" - "\x43\x2d\x9a\xcb\x92\x3f\x5a\x1d" - "\x85\xd8\xc9\xdf\x68\xc9\x12\x80" - "\x56\x0c\xdc\x00\xdc\x3a\x7d\x9d" - "\xa3\xa2\xe8\x4d\xbf\xf9\x70\xa0" - "\xa4\x13\x4f\x6b\xaf\x0a\x89\x7f" - "\xda\xf0\xbf\x9b\xc8\x1d\xe5\xf8" - "\x2e\x8b\x07\xb5\x73\x1b\xcc\xa2" - "\xa6\xad\x30\xbc\x78\x3c\x5b\x10" - "\xfa\x5e\x62\x2d\x9e\x64\xb3\x33" - "\xce\xf9\x1f\x86\xe7\x8b\xa2\xb8" - "\xe8\x99\x57\x8c\x11\xed\x66\xd9" - "\x3c\x72\xb9\xc3\xe6\x4e\x17\x3a" - "\x6a\xcb\x42\x24\x06\xed\x3e\x4e" - "\xa3\xe8\x6a\x94\xda\x0d\x4e\xd5" - "\x14\x19\xcf\xb6\x26\xd8\x2e\xcc" - "\x64\x76\x38\x49\x4d\xfe\x30\x6d" - "\xe4\xc8\x8c\x7b\xc4\xe0\x35\xba" - "\x22\x6e\x76\xe1\x1a\xf2\x53\xc3" - "\x28\xa2\x82\x1f\x61\x69\xad\xc1" - "\x7b\x28\x4b\x1e\x6c\x85\x95\x9b" - "\x51\xb5\x17\x7f\x12\x69\x8c\x24" - "\xd5\xc7\x5a\x5a\x11\x54\xff\x5a" - "\xf7\x16\xc3\x91\xa6\xf0\xdc\x0a" - "\xb6\xa7\x4a\x0d\x7a\x58\xfe\xa5" - "\xf5\xcb\x8f\x7b\x0e\xea\x57\xe7" - "\xbd\x79\xd6\x1c\x88\x23\x6c\xf2" - "\x4d\x29\x77\x53\x35\x6a\x00\x8d" - "\xcd\xa3\x58\xbe\x77\x99\x18\xf8" - "\xe6\xe1\x8f\xe9\x37\x8f\xe3\xe2" - "\x5a\x8a\x93\x25\xaf\xf3\x78\x80" - "\xbe\xa6\x1b\xc6\xac\x8b\x1c\x91" - "\x58\xe1\x9f\x89\x35\x9d\x1d\x21" - "\x29\x9f\xf4\x99\x02\x27\x0f\xa8" - "\x4f\x79\x94\x2b\x33\x2c\xda\xa2" - "\x26\x39\x83\x94\xef\x27\xd8\x53" - "\x8f\x66\x0d\xe4\x41\x7d\x34\xcd" - "\x43\x7c\x95\x0a\x53\xef\x66\xda" - "\x7e\x9b\xf3\x93\xaf\xd0\x73\x71" - "\xba\x40\x9b\x74\xf8\xd7\xd7\x41" - "\x6d\xaf\x72\x9c\x8d\x21\x87\x3c" - "\xfd\x0a\x90\xa9\x47\x96\x9e\xd3" - "\x88\xee\x73\xcf\x66\x2f\x52\x56" - "\x6d\xa9\x80\x4c\xe2\x6f\x62\x88" - "\x3f\x0e\x54\x17\x48\x80\x5d\xd3" - "\xc3\xda\x25\x3d\xa1\xc8\xcb\x9f" - "\x9b\x70\xb3\xa1\xeb\x04\x52\xa1" - "\xf2\x22\x0f\xfc\xc8\x18\xfa\xf9" - "\x85\x9c\xf1\xac\xeb\x0c\x02\x46" - "\x75\xd2\xf5\x2c\xe3\xd2\x59\x94" - "\x12\xf3\x3c\xfc\xd7\x92\xfa\x36" - "\xba\x61\x34\x38\x7c\xda\x48\x3e" - "\x08\xc9\x39\x23\x5e\x02\x2c\x1a" - "\x18\x7e\xb4\xd9\xfd\x9e\x40\x02" - "\xb1\x33\x37\x32\xe7\xde\xd6\xd0" - "\x7c\x58\x65\x4b\xf8\x34\x27\x9c" - "\x44\xb4\xbd\xe9\xe9\x4c\x78\x7d" - "\x4b\x9f\xce\xb1\xcd\x47\xa5\x37" - "\xe5\x6d\xbd\xb9\x43\x94\x0a\xd4" - "\xd6\xf9\x04\x5f\xb5\x66\x6c\x1a" - "\x35\x12\xe3\x36\x28\x27\x36\x58" - "\x01\x2b\x79\xe4\xba\x6d\x10\x7d" - "\x65\xdf\x84\x95\xf4\xd5\xb6\x8f" - "\x2b\x9f\x96\x00\x86\x60\xf0\x21" - "\x76\xa8\x6a\x8c\x28\x1c\xb3\x6b" - "\x97\xd7\xb6\x53\x2a\xcc\xab\x40" - "\x9d\x62\x79\x58\x52\xe6\x65\xb7" - "\xab\x55\x67\x9c\x89\x7c\x03\xb0" - "\x73\x59\xc5\x81\xf5\x18\x17\x5c" - "\x89\xf3\x78\x35\x44\x62\x78\x72" - "\xd0\x96\xeb\x31\xe7\x87\x77\x14" - "\x99\x51\xf2\x59\x26\x9e\xb5\xa6" - "\x45\xfe\x6e\xbd\x07\x4c\x94\x5a" - "\xa5\x7d\xfc\xf1\x2b\x77\xe2\xfe" - "\x17\xd4\x84\xa0\xac\xb5\xc7\xda" - "\xa9\x1a\xb6\xf3\x74\x11\xb4\x9d" - "\xfb\x79\x2e\x04\x2d\x50\x28\x83" - "\xbf\xc6\x52\xd3\x34\xd6\xe8\x7a" - "\xb6\xea\xe7\xa8\x6c\x15\x1e\x2c" - "\x57\xbc\x48\x4e\x5f\x5c\xb6\x92" - "\xd2\x49\x77\x81\x6d\x90\x70\xae" - "\x98\xa1\x03\x0d\x6b\xb9\x77\x14" - "\xf1\x4e\x23\xd3\xf8\x68\xbd\xc2" - "\xfe\x04\xb7\x5c\xc5\x17\x60\x8f" - "\x65\x54\xa4\x7a\x42\xdc\x18\x0d" - "\xb5\xcf\x0f\xd3\xc7\x91\x66\x1b" - "\x45\x42\x27\x75\x50\xe5\xee\xb8" - "\x7f\x33\x2c\xba\x4a\x92\x4d\x2c" - "\x3c\xe3\x0d\x80\x01\xba\x0d\x29" - "\xd8\x3c\xe9\x13\x16\x57\xe6\xea" - "\x94\x52\xe7\x00\x4d\x30\xb0\x0f" - "\x35\xb8\xb8\xa7\xb1\xb5\x3b\x44" - "\xe1\x2f\xfd\x88\xed\x43\xe7\x52" - "\x10\x93\xb3\x8a\x30\x6b\x0a\xf7" - "\x23\xc6\x50\x9d\x4a\xb0\xde\xc3" - "\xdc\x9b\x2f\x01\x56\x36\x09\xc5" - "\x2f\x6b\xfe\xf1\xd8\x27\x45\x03" - "\x30\x5e\x5c\x5b\xb4\x62\x0e\x1a" - "\xa9\x21\x2b\x92\x94\x87\x62\x57" - "\x4c\x10\x74\x1a\xf1\x0a\xc5\x84" - "\x3b\x9e\x72\x02\xd7\xcc\x09\x56" - "\xbd\x54\xc1\xf0\xc3\xe3\xb3\xf8" - "\xd2\x0d\x61\xcb\xef\xce\x0d\x05" - "\xb0\x98\xd9\x8e\x4f\xf9\xbc\x93" - "\xa6\xea\xc8\xcf\x10\x53\x4b\xf1" - "\xec\xfc\x89\xf9\x64\xb0\x22\xbf" - "\x9e\x55\x46\x9f\x7c\x50\x8e\x84" - "\x54\x20\x98\xd7\x6c\x40\x1e\xdb" - "\x69\x34\x78\x61\x24\x21\x9c\x8a" - "\xb3\x62\x31\x8b\x6e\xf5\x2a\x35" - "\x86\x13\xb1\x6c\x64\x2e\x41\xa5" - "\x05\xf2\x42\xba\xd2\x3a\x0d\x8e" - "\x8a\x59\x94\x3c\xcf\x36\x27\x82" - "\xc2\x45\xee\x58\xcd\x88\xb4\xec" - "\xde\xb2\x96\x0a\xaf\x38\x6f\x88" - "\xd7\xd8\xe1\xdf\xb9\x96\xa9\x0a" - "\xb1\x95\x28\x86\x20\xe9\x17\x49" - "\xa2\x29\x38\xaa\xa5\xe9\x6e\xf1" - "\x19\x27\xc0\xd5\x2a\x22\xc3\x0b" - "\xdb\x7c\x73\x10\xb9\xba\x89\x76" - "\x54\xae\x7d\x71\xb3\x93\xf6\x32" - "\xe6\x47\x43\x55\xac\xa0\x0d\xc2" - "\x93\x27\x4a\x8e\x0e\x74\x15\xc7" - "\x0b\x85\xd9\x0c\xa9\x30\x7a\x3e" - "\xea\x8f\x85\x6d\x3a\x12\x4f\x72" - "\x69\x58\x7a\x80\xbb\xb5\x97\xf3" - "\xcf\x70\xd2\x5d\xdd\x4d\x21\x79" - "\x54\x4d\xe4\x05\xe8\xbd\xc2\x62" - "\xb1\x3b\x77\x1c\xd6\x5c\xf3\xa0" - "\x79\x00\xa8\x6c\x29\xd9\x18\x24" - "\x36\xa2\x46\xc0\x96\x65\x7f\xbd" - "\x2a\xed\x36\x16\x0c\xaa\x9f\xf4" - "\xc5\xb4\xe2\x12\xed\x69\xed\x4f" - "\x26\x2c\x39\x52\x89\x98\xe7\x2c" - "\x99\xa4\x9e\xa3\x9b\x99\x46\x7a" - "\x3a\xdc\xa8\x59\xa3\xdb\xc3\x3b" - "\x95\x0d\x3b\x09\x6e\xee\x83\x5d" - "\x32\x4d\xed\xab\xfa\x98\x14\x4e" - "\xc3\x15\x45\x53\x61\xc4\x93\xbd" - "\x90\xf4\x99\x95\x4c\xe6\x76\x92" - "\x29\x90\x46\x30\x92\x69\x7d\x13" - "\xf2\xa5\xcd\x69\x49\x44\xb2\x0f" - "\x63\x40\x36\x5f\x09\xe2\x78\xf8" - "\x91\xe3\xe2\xfa\x10\xf7\xc8\x24" - "\xa8\x89\x32\x5c\x37\x25\x1d\xb2" - "\xea\x17\x8a\x0a\xa9\x64\xc3\x7c" - "\x3c\x7c\xbd\xc6\x79\x34\xe7\xe2" - "\x85\x8e\xbf\xf8\xde\x92\xa0\xae" - "\x20\xc4\xf6\xbb\x1f\x38\x19\x0e" - "\xe8\x79\x9c\xa1\x23\xe9\x54\x7e" - "\x37\x2f\xe2\x94\x32\xaf\xa0\x23" - "\x49\xe4\xc0\xb3\xac\x00\x8f\x36" - "\x05\xc4\xa6\x96\xec\x05\x98\x4f" - "\x96\x67\x57\x1f\x20\x86\x1b\x2d" - "\x69\xe4\x29\x93\x66\x5f\xaf\x6b" - "\x88\x26\x2c\x67\x02\x4b\x52\xd0" - "\x83\x7a\x43\x1f\xc0\x71\x15\x25" - "\x77\x65\x08\x60\x11\x76\x4c\x8d" - "\xed\xa9\x27\xc6\xb1\x2a\x2c\x6a" - "\x4a\x97\xf5\xc6\xb7\x70\x42\xd3" - "\x03\xd1\x24\x95\xec\x6d\xab\x38" - "\x72\xce\xe2\x8b\x33\xd7\x51\x09" - "\xdc\x45\xe0\x09\x96\x32\xf3\xc4" - "\x84\xdc\x73\x73\x2d\x1b\x11\x98" - "\xc5\x0e\x69\x28\x94\xc7\xb5\x4d" - "\xc8\x8a\xd0\xaa\x13\x2e\x18\x74" - "\xdd\xd1\x1e\xf3\x90\xe8\xfc\x9a" - "\x72\x4a\x0e\xd1\xe4\xfb\x0d\x96" - "\xd1\x0c\x79\x85\x1b\x1c\xfe\xe1" - "\x62\x8f\x7a\x73\x32\xab\xc8\x18" - "\x69\xe3\x34\x30\xdf\x13\xa6\xe5" - "\xe8\x0e\x67\x7f\x81\x11\xb4\x60" - "\xc7\xbd\x79\x65\x50\xdc\xc4\x5b" - "\xde\x39\xa4\x01\x72\x63\xf3\xd1" - "\x64\x4e\xdf\xfc\x27\x92\x37\x0d" - "\x57\xcd\x11\x4f\x11\x04\x8e\x1d" - "\x16\xf7\xcd\x92\x9a\x99\x30\x14" - "\xf1\x7c\x67\x1b\x1f\x41\x0b\xe8" - "\x32\xe8\xb8\xc1\x4f\x54\x86\x4f" - "\xe5\x79\x81\x73\xcd\x43\x59\x68" - "\x73\x02\x3b\x78\x21\x72\x43\x00" - "\x49\x17\xf7\x00\xaf\x68\x24\x53" - "\x05\x0a\xc3\x33\xe0\x33\x3f\x69" - "\xd2\x84\x2f\x0b\xed\xde\x04\xf4" - "\x11\x94\x13\x69\x51\x09\x28\xde" - "\x57\x5c\xef\xdc\x9a\x49\x1c\x17" - "\x97\xf3\x96\xc1\x7f\x5d\x2e\x7d" - "\x55\xb8\xb3\x02\x09\xb3\x1f\xe7" - "\xc9\x8d\xa3\x36\x34\x8a\x77\x13" - "\x30\x63\x4c\xa5\xcd\xc3\xe0\x7e" - "\x05\xa1\x7b\x0c\xcb\x74\x47\x31" - "\x62\x03\x43\xf1\x87\xb4\xb0\x85" - "\x87\x8e\x4b\x25\xc7\xcf\xae\x4b" - "\x36\x46\x3e\x62\xbc\x6f\xeb\x5f" - "\x73\xac\xe6\x07\xee\xc1\xa1\xd6" - "\xc4\xab\xc9\xd6\x89\x45\xe1\xf1" - "\x04\x4e\x1a\x6f\xbb\x4f\x3a\xa3" - "\xa0\xcb\xa3\x0a\xd8\x71\x35\x55" - "\xe4\xbc\x2e\x04\x06\xe6\xff\x5b" - "\x1c\xc0\x11\x7c\xc5\x17\xf3\x38" - "\xcf\xe9\xba\x0f\x0e\xef\x02\xc2" - "\x8d\xc6\xbc\x4b\x67\x20\x95\xd7" - "\x2c\x45\x5b\x86\x44\x8c\x6f\x2e" - "\x7e\x9f\x1c\x77\xba\x6b\x0e\xa3" - "\x69\xdc\xab\x24\x57\x60\x47\xc1" - "\xd1\xa5\x9d\x23\xe6\xb1\x37\xfe" - "\x93\xd2\x4c\x46\xf9\x0c\xc6\xfb" - "\xd6\x9d\x99\x69\xab\x7a\x07\x0c" - "\x65\xe7\xc4\x08\x96\xe2\xa5\x01" - "\x3f\x46\x07\x05\x7e\xe8\x9a\x90" - "\x50\xdc\xe9\x7a\xea\xa1\x39\x6e" - "\x66\xe4\x6f\xa5\x5f\xb2\xd9\x5b" - "\xf5\xdb\x2a\x32\xf0\x11\x6f\x7c" - "\x26\x10\x8f\x3d\x80\xe9\x58\xf7" - "\xe0\xa8\x57\xf8\xdb\x0e\xce\x99" - "\x63\x19\x3d\xd5\xec\x1b\x77\x69" - "\x98\xf6\xe4\x5f\x67\x17\x4b\x09" - "\x85\x62\x82\x70\x18\xe2\x9a\x78" - "\xe2\x62\xbd\xb4\xf1\x42\xc6\xfb" - "\x08\xd0\xbd\xeb\x4e\x09\xf2\xc8" - "\x1e\xdc\x3d\x32\x21\x56\x9c\x4f" - "\x35\xf3\x61\x06\x72\x84\xc4\x32" - "\xf2\xf1\xfa\x0b\x2f\xc3\xdb\x02" - "\x04\xc2\xde\x57\x64\x60\x8d\xcf" - "\xcb\x86\x5d\x97\x3e\xb1\x9c\x01" - "\xd6\x28\x8f\x99\xbc\x46\xeb\x05" - "\xaf\x7e\xb8\x21\x2a\x56\x85\x1c" - "\xb3\x71\xa0\xde\xca\x96\xf1\x78" - "\x49\xa2\x99\x81\x80\x5c\x01\xf5" - "\xa0\xa2\x56\x63\xe2\x70\x07\xa5" - "\x95\xd6\x85\xeb\x36\x9e\xa9\x51" - "\x66\x56\x5f\x1d\x02\x19\xe2\xf6" - "\x4f\x73\x38\x09\x75\x64\x48\xe0" - "\xf1\x7e\x0e\xe8\x9d\xf9\xed\x94" - "\xfe\x16\x26\x62\x49\x74\xf4\xb0" - "\xd4\xa9\x6c\xb0\xfd\x53\xe9\x81" - "\xe0\x7a\xbf\xcf\xb5\xc4\x01\x81" - "\x79\x99\x77\x01\x3b\xe9\xa2\xb6" - "\xe6\x6a\x8a\x9e\x56\x1c\x8d\x1e" - "\x8f\x06\x55\x2c\x6c\xdc\x92\x87" - "\x64\x3b\x4b\x19\xa1\x13\x64\x1d" - "\x4a\xe9\xc0\x00\xb8\x95\xef\x6b" - "\x1a\x86\x6d\x37\x52\x02\xc2\xe0" - "\xc8\xbb\x42\x0c\x02\x21\x4a\xc9" - "\xef\xa0\x54\xe4\x5e\x16\x53\x81" - "\x70\x62\x10\xaf\xde\xb8\xb5\xd3" - "\xe8\x5e\x6c\xc3\x8a\x3e\x18\x07" - "\xf2\x2f\x7d\xa7\xe1\x3d\x4e\xb4" - "\x26\xa7\xa3\x93\x86\xb2\x04\x1e" - "\x53\x5d\x86\xd6\xde\x65\xca\xe3" - "\x4e\xc1\xcf\xef\xc8\x70\x1b\x83" - "\x13\xdd\x18\x8b\x0d\x76\xd2\xf6" - "\x37\x7a\x93\x7a\x50\x11\x9f\x96" - "\x86\x25\xfd\xac\xdc\xbe\x18\x93" - "\x19\x6b\xec\x58\x4f\xb9\x75\xa7" - "\xdd\x3f\x2f\xec\xc8\x5a\x84\xab" - "\xd5\xe4\x8a\x07\xf6\x4d\x23\xd6" - "\x03\xfb\x03\x6a\xea\x66\xbf\xd4" - "\xb1\x34\xfb\x78\xe9\x55\xdc\x7c" - "\x3d\x9c\xe5\x9a\xac\xc3\x7a\x80" - "\x24\x6d\xa0\xef\x25\x7c\xb7\xea" - "\xce\x4d\x5f\x18\x60\xce\x87\x22" - "\x66\x2f\xd5\xdd\xdd\x02\x21\x75" - "\x82\xa0\x1f\x58\xc6\xd3\x62\xf7" - "\x32\xd8\xaf\x1e\x07\x77\x51\x96" - "\xd5\x6b\x1e\x7e\x80\x02\xe8\x67" - "\xea\x17\x0b\x10\xd2\x3f\x28\x25" - "\x4f\x05\x77\x02\x14\x69\xf0\x2c" - "\xbe\x0c\xf1\x74\x30\xd1\xb9\x9b" - "\xfc\x8c\xbb\x04\x16\xd9\xba\xc3" - "\xbc\x91\x8a\xc4\x30\xa4\xb0\x12" - "\x4c\x21\x87\xcb\xc9\x1d\x16\x96" - "\x07\x6f\x23\x54\xb9\x6f\x79\xe5" - "\x64\xc0\x64\xda\xb1\xae\xdd\x60" - "\x6c\x1a\x9d\xd3\x04\x8e\x45\xb0" - "\x92\x61\xd0\x48\x81\xed\x5e\x1d" - "\xa0\xc9\xa4\x33\xc7\x13\x51\x5d" - "\x7f\x83\x73\xb6\x70\x18\x65\x3e" - "\x2f\x0e\x7a\x12\x39\x98\xab\xd8" - "\x7e\x6f\xa3\xd1\xba\x56\xad\xbd" - "\xf0\x03\x01\x1c\x85\x35\x9f\xeb" - "\x19\x63\xa1\xaf\xfe\x2d\x35\x50" - "\x39\xa0\x65\x7c\x95\x7e\x6b\xfe" - "\xc1\xac\x07\x7c\x98\x4f\xbe\x57" - "\xa7\x22\xec\xe2\x7e\x29\x09\x53" - "\xe8\xbf\xb4\x7e\x3f\x8f\xfc\x14" - "\xce\x54\xf9\x18\x58\xb5\xff\x44" - "\x05\x9d\xce\x1b\xb6\x82\x23\xc8" - "\x2e\xbc\x69\xbb\x4a\x29\x0f\x65" - "\x94\xf0\x63\x06\x0e\xef\x8c\xbd" - "\xff\xfd\xb0\x21\x6e\x57\x05\x75" - "\xda\xd5\xc4\xeb\x8d\x32\xf7\x50" - "\xd3\x6f\x22\xed\x5f\x8e\xa2\x5b" - "\x80\x8c\xc8\x78\x40\x24\x4b\x89" - "\x30\xce\x7a\x97\x0e\xc4\xaf\xef" - "\x9b\xb4\xcd\x66\x74\x14\x04\x2b" - "\xf7\xce\x0b\x1c\x6e\xc2\x78\x8c" - "\xca\xc5\xd0\x1c\x95\x4a\x91\x2d" - "\xa7\x20\xeb\x86\x52\xb7\x67\xd8" - "\x0c\xd6\x04\x14\xde\x51\x74\x75" - "\xe7\x11\xb4\x87\xa3\x3d\x2d\xad" - "\x4f\xef\xa0\x0f\x70\x00\x6d\x13" - "\x19\x1d\x41\x50\xe9\xd8\xf0\x32" - "\x71\xbc\xd3\x11\xf2\xac\xbe\xaf" - "\x75\x46\x65\x4e\x07\x34\x37\xa3" - "\x89\xfe\x75\xd4\x70\x4c\xc6\x3f" - "\x69\x24\x0e\x38\x67\x43\x8c\xde" - "\x06\xb5\xb8\xe7\xc4\xf0\x41\x8f" - "\xf0\xbd\x2f\x0b\xb9\x18\xf8\xde" - "\x64\xb1\xdb\xee\x00\x50\x77\xe1" - "\xc7\xff\xa6\xfa\xdd\x70\xf4\xe3" - "\x93\xe9\x77\x35\x3d\x4b\x2f\x2b" - "\x6d\x55\xf0\xfc\x88\x54\x4e\x89" - "\xc1\x8a\x23\x31\x2d\x14\x2a\xb8" - "\x1b\x15\xdd\x9e\x6e\x7b\xda\x05" - "\x91\x7d\x62\x64\x96\x72\xde\xfc" - "\xc1\xec\xf0\x23\x51\x6f\xdb\x5b" - "\x1d\x08\x57\xce\x09\xb8\xf6\xcd" - "\x8d\x95\xf2\x20\xbf\x0f\x20\x57" - "\x98\x81\x84\x4f\x15\x5c\x76\xe7" - "\x3e\x0a\x3a\x6c\xc4\x8a\xbe\x78" - "\x74\x77\xc3\x09\x4b\x5d\x48\xe4" - "\xc8\xcb\x0b\xea\x17\x28\xcf\xcf" - "\x31\x32\x44\xa4\xe5\x0e\x1a\x98" - "\x94\xc4\xf0\xff\xae\x3e\x44\xe8" - "\xa5\xb3\xb5\x37\x2f\xe8\xaf\x6f" - "\x28\xc1\x37\x5f\x31\xd2\xb9\x33" - "\xb1\xb2\x52\x94\x75\x2c\x29\x59" - "\x06\xc2\x25\xe8\x71\x65\x4e\xed" - "\xc0\x9c\xb1\xbb\x25\xdc\x6c\xe7" - "\x4b\xa5\x7a\x54\x7a\x60\xff\x7a" - "\xe0\x50\x40\x96\x35\x63\xe4\x0b" - "\x76\xbd\xa4\x65\x00\x1b\x57\x88" - "\xae\xed\x39\x88\x42\x11\x3c\xed" - "\x85\x67\x7d\xb9\x68\x82\xe9\x43" - "\x3c\x47\x53\xfa\xe8\xf8\x9f\x1f" - "\x9f\xef\x0f\xf7\x30\xd9\x30\x0e" - "\xb9\x9f\x69\x18\x2f\x7e\xf8\xf8" - "\xf8\x8c\x0f\xd4\x02\x4d\xea\xcd" - "\x0a\x9c\x6f\x71\x6d\x5a\x4c\x60" - "\xce\x20\x56\x32\xc6\xc5\x99\x1f" - "\x09\xe6\x4e\x18\x1a\x15\x13\xa8" - "\x7d\xb1\x6b\xc0\xb2\x6d\xf8\x26" - "\x66\xf8\x3d\x18\x74\x70\x66\x7a" - "\x34\x17\xde\xba\x47\xf1\x06\x18" - "\xcb\xaf\xeb\x4a\x1e\x8f\xa7\x77" - "\xe0\x3b\x78\x62\x66\xc9\x10\xea" - "\x1f\xb7\x29\x0a\x45\xa1\x1d\x1e" - "\x1d\xe2\x65\x61\x50\x9c\xd7\x05" - "\xf2\x0b\x5b\x12\x61\x02\xc8\xe5" - "\x63\x4f\x20\x0c\x07\x17\x33\x5e" - "\x03\x9a\x53\x0f\x2e\x55\xfe\x50" - "\x43\x7d\xd0\xb6\x7e\x5a\xda\xae" - "\x58\xef\x15\xa9\x83\xd9\x46\xb1" - "\x42\xaa\xf5\x02\x6c\xce\x92\x06" - "\x1b\xdb\x66\x45\x91\x79\xc2\x2d" - "\xe6\x53\xd3\x14\xfd\xbb\x44\x63" - "\xc6\xd7\x3d\x7a\x0c\x75\x78\x9d" - "\x5c\xa6\x39\xb3\xe5\x63\xca\x8b" - "\xfe\xd3\xef\x60\x83\xf6\x8e\x70" - "\xb6\x67\xc7\x77\xed\x23\xef\x4c" - "\xf0\xed\x2d\x07\x59\x6f\xc1\x01" - "\x34\x37\x08\xab\xd9\x1f\x09\xb1" - "\xce\x5b\x17\xff\x74\xf8\x9c\xd5" - "\x2c\x56\x39\x79\x0f\x69\x44\x75" - "\x58\x27\x01\xc4\xbf\xa7\xa1\x1d" - "\x90\x17\x77\x86\x5a\x3f\xd9\xd1" - "\x0e\xa0\x10\xf8\xec\x1e\xa5\x7f" - "\x5e\x36\xd1\xe3\x04\x2c\x70\xf7" - "\x8e\xc0\x98\x2f\x6c\x94\x2b\x41" - "\xb7\x60\x00\xb7\x2e\xb8\x02\x8d" - "\xb8\xb0\xd3\x86\xba\x1d\xd7\x90" - "\xd6\xb6\xe1\xfc\xd7\xd8\x28\x06" - "\x63\x9b\xce\x61\x24\x79\xc0\x70" - "\x52\xd0\xb6\xd4\x28\x95\x24\x87" - "\x03\x1f\xb7\x9a\xda\xa3\xfb\x52" - "\x5b\x68\xe7\x4c\x8c\x24\xe1\x42" - "\xf7\xd5\xfd\xad\x06\x32\x9f\xba" - "\xc1\xfc\xdd\xc6\xfc\xfc\xb3\x38" - "\x74\x56\x58\x40\x02\x37\x52\x2c" - "\x55\xcc\xb3\x9e\x7a\xe9\xd4\x38" - "\x41\x5e\x0c\x35\xe2\x11\xd1\x13" - "\xf8\xb7\x8d\x72\x6b\x22\x2a\xb0" - "\xdb\x08\xba\x35\xb9\x3f\xc8\xd3" - "\x24\x90\xec\x58\xd2\x09\xc7\x2d" - "\xed\x38\x80\x36\x72\x43\x27\x49" - "\x4a\x80\x8a\xa2\xe8\xd3\xda\x30" - "\x7d\xb6\x82\x37\x86\x92\x86\x3e" - "\x08\xb2\x28\x5a\x55\x44\x24\x7d" - "\x40\x48\x8a\xb6\x89\x58\x08\xa0" - "\xd6\x6d\x3a\x17\xbf\xf6\x54\xa2" - "\xf5\xd3\x8c\x0f\x78\x12\x57\x8b" - "\xd5\xc2\xfd\x58\x5b\x7f\x38\xe3" - "\xcc\xb7\x7c\x48\xb3\x20\xe8\x81" - "\x14\x32\x45\x05\xe0\xdb\x9f\x75" - "\x85\xb4\x6a\xfc\x95\xe3\x54\x22" - "\x12\xee\x30\xfe\xd8\x30\xef\x34" - "\x50\xab\x46\x30\x98\x2f\xb7\xc0" - "\x15\xa2\x83\xb6\xf2\x06\x21\xa2" - "\xc3\x26\x37\x14\xd1\x4d\xb5\x10" - "\x52\x76\x4d\x6a\xee\xb5\x2b\x15" - "\xb7\xf9\x51\xe8\x2a\xaf\xc7\xfa" - "\x77\xaf\xb0\x05\x4d\xd1\x68\x8e" - "\x74\x05\x9f\x9d\x93\xa5\x3e\x7f" - "\x4e\x5f\x9d\xcb\x09\xc7\x83\xe3" - "\x02\x9d\x27\x1f\xef\x85\x05\x8d" - "\xec\x55\x88\x0f\x0d\x7c\x4c\xe8" - "\xa1\x75\xa0\xd8\x06\x47\x14\xef" - "\xaa\x61\xcf\x26\x15\xad\xd8\xa3" - "\xaa\x75\xf2\x78\x4a\x5a\x61\xdf" - "\x8b\xc7\x04\xbc\xb2\x32\xd2\x7e" - "\x42\xee\xb4\x2f\x51\xff\x7b\x2e" - "\xd3\x02\xe8\xdc\x5d\x0d\x50\xdc" - "\xae\xb7\x46\xf9\xa8\xe6\xd0\x16" - "\xcc\xe6\x2c\x81\xc7\xad\xe9\xf0" - "\x05\x72\x6d\x3d\x0a\x7a\xa9\x02" - "\xac\x82\x93\x6e\xb6\x1c\x28\xfc" - "\x44\x12\xfb\x73\x77\xd4\x13\x39" - "\x29\x88\x8a\xf3\x5c\xa6\x36\xa0" - "\x2a\xed\x7e\xb1\x1d\xd6\x4c\x6b" - "\x41\x01\x18\x5d\x5d\x07\x97\xa6" - "\x4b\xef\x31\x18\xea\xac\xb1\x84" - "\x21\xed\xda\x86", - .rlen = 4100, - .np = 2, - .tap = { 4064, 36 }, - }, -}; - -static const struct cipher_testvec aes_ctr_rfc3686_dec_tv_template[] = { - { /* From RFC 3686 */ - .key = "\xae\x68\x52\xf8\x12\x10\x67\xcc" - "\x4b\xf7\xa5\x76\x55\x77\xf3\x9e" - "\x00\x00\x00\x30", - .klen = 20, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xe4\x09\x5d\x4f\xb7\xa7\xb3\x79" - "\x2d\x61\x75\xa3\x26\x13\x11\xb8", - .ilen = 16, - .result = "Single block msg", - .rlen = 16, - }, { - .key = "\x7e\x24\x06\x78\x17\xfa\xe0\xd7" - "\x43\xd6\xce\x1f\x32\x53\x91\x63" - "\x00\x6c\xb6\xdb", - .klen = 20, - .iv = "\xc0\x54\x3b\x59\xda\x48\xd9\x0b", - .input = "\x51\x04\xa1\x06\x16\x8a\x72\xd9" - "\x79\x0d\x41\xee\x8e\xda\xd3\x88" - "\xeb\x2e\x1e\xfc\x46\xda\x57\xc8" - "\xfc\xe6\x30\xdf\x91\x41\xbe\x28", - .ilen = 32, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .rlen = 32, - }, { - .key = "\x16\xaf\x5b\x14\x5f\xc9\xf5\x79" - "\xc1\x75\xf9\x3e\x3b\xfb\x0e\xed" - "\x86\x3d\x06\xcc\xfd\xb7\x85\x15" - "\x00\x00\x00\x48", - .klen = 28, - .iv = "\x36\x73\x3c\x14\x7d\x6d\x93\xcb", - .input = "\x4b\x55\x38\x4f\xe2\x59\xc9\xc8" - "\x4e\x79\x35\xa0\x03\xcb\xe9\x28", - .ilen = 16, - .result = "Single block msg", - .rlen = 16, - }, { - .key = "\x7c\x5c\xb2\x40\x1b\x3d\xc3\x3c" - "\x19\xe7\x34\x08\x19\xe0\xf6\x9c" - "\x67\x8c\x3d\xb8\xe6\xf6\xa9\x1a" - "\x00\x96\xb0\x3b", - .klen = 28, - .iv = "\x02\x0c\x6e\xad\xc2\xcb\x50\x0d", - .input = "\x45\x32\x43\xfc\x60\x9b\x23\x32" - "\x7e\xdf\xaa\xfa\x71\x31\xcd\x9f" - "\x84\x90\x70\x1c\x5a\xd4\xa7\x9c" - "\xfc\x1f\xe0\xff\x42\xf4\xfb\x00", - .ilen = 32, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .rlen = 32, - }, { - .key = "\x77\x6b\xef\xf2\x85\x1d\xb0\x6f" - "\x4c\x8a\x05\x42\xc8\x69\x6f\x6c" - "\x6a\x81\xaf\x1e\xec\x96\xb4\xd3" - "\x7f\xc1\xd6\x89\xe6\xc1\xc1\x04" - "\x00\x00\x00\x60", - .klen = 36, - .iv = "\xdb\x56\x72\xc9\x7a\xa8\xf0\xb2", - .input = "\x14\x5a\xd0\x1d\xbf\x82\x4e\xc7" - "\x56\x08\x63\xdc\x71\xe3\xe0\xc0", - .ilen = 16, - .result = "Single block msg", - .rlen = 16, - }, { - .key = "\xf6\xd6\x6d\x6b\xd5\x2d\x59\xbb" - "\x07\x96\x36\x58\x79\xef\xf8\x86" - "\xc6\x6d\xd5\x1a\x5b\x6a\x99\x74" - "\x4b\x50\x59\x0c\x87\xa2\x38\x84" - "\x00\xfa\xac\x24", - .klen = 36, - .iv = "\xc1\x58\x5e\xf1\x5a\x43\xd8\x75", - .input = "\xf0\x5e\x23\x1b\x38\x94\x61\x2c" - "\x49\xee\x00\x0b\x80\x4e\xb2\xa9" - "\xb8\x30\x6b\x50\x8f\x83\x9d\x6a" - "\x55\x30\x83\x1d\x93\x44\xaf\x1c", - .ilen = 32, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .rlen = 32, - }, -}; - -static const struct cipher_testvec aes_ofb_enc_tv_template[] = { - /* From NIST Special Publication 800-38A, Appendix F.5 */ - { - .key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6" - "\xab\xf7\x15\x88\x09\xcf\x4f\x3c", - .klen = 16, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07\x08" - "\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .ilen = 64, - .result = "\x3b\x3f\xd9\x2e\xb7\x2d\xad\x20" - "\x33\x34\x49\xf8\xe8\x3c\xfb\x4a" - "\x77\x89\x50\x8d\x16\x91\x8f\x03\xf5" - "\x3c\x52\xda\xc5\x4e\xd8\x25" - "\x97\x40\x05\x1e\x9c\x5f\xec\xf6\x43" - "\x44\xf7\xa8\x22\x60\xed\xcc" - "\x30\x4c\x65\x28\xf6\x59\xc7\x78" - "\x66\xa5\x10\xd9\xc1\xd6\xae\x5e", - .rlen = 64, - } -}; - -static const struct cipher_testvec aes_ofb_dec_tv_template[] = { - /* From NIST Special Publication 800-38A, Appendix F.5 */ - { - .key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6" - "\xab\xf7\x15\x88\x09\xcf\x4f\x3c", - .klen = 16, - .iv = "\x00\x01\x02\x03\x04\x05\x06\x07\x08" - "\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .input = "\x3b\x3f\xd9\x2e\xb7\x2d\xad\x20" - "\x33\x34\x49\xf8\xe8\x3c\xfb\x4a" - "\x77\x89\x50\x8d\x16\x91\x8f\x03\xf5" - "\x3c\x52\xda\xc5\x4e\xd8\x25" - "\x97\x40\x05\x1e\x9c\x5f\xec\xf6\x43" - "\x44\xf7\xa8\x22\x60\xed\xcc" - "\x30\x4c\x65\x28\xf6\x59\xc7\x78" - "\x66\xa5\x10\xd9\xc1\xd6\xae\x5e", - .ilen = 64, - .result = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" - "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" - "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" - "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" - "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" - "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" - "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" - "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", - .rlen = 64, - } -}; - -static const struct aead_testvec aes_gcm_enc_tv_template[] = { - { /* From McGrew & Viega - http://citeseer.ist.psu.edu/656989.html */ - .key = zeroed_string, - .klen = 16, - .result = "\x58\xe2\xfc\xce\xfa\x7e\x30\x61" - "\x36\x7f\x1d\x57\xa4\xe7\x45\x5a", - .rlen = 16, - }, { - .key = zeroed_string, - .klen = 16, - .input = zeroed_string, - .ilen = 16, - .result = "\x03\x88\xda\xce\x60\xb6\xa3\x92" - "\xf3\x28\xc2\xb9\x71\xb2\xfe\x78" - "\xab\x6e\x47\xd4\x2c\xec\x13\xbd" - "\xf5\x3a\x67\xb2\x12\x57\xbd\xdf", - .rlen = 32, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08", - .klen = 16, - .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" - "\xde\xca\xf8\x88", - .input = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" - "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" - "\x86\xa7\xa9\x53\x15\x34\xf7\xda" - "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" - "\x1c\x3c\x0c\x95\x95\x68\x09\x53" - "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" - "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" - "\xba\x63\x7b\x39\x1a\xaf\xd2\x55", - .ilen = 64, - .result = "\x42\x83\x1e\xc2\x21\x77\x74\x24" - "\x4b\x72\x21\xb7\x84\xd0\xd4\x9c" - "\xe3\xaa\x21\x2f\x2c\x02\xa4\xe0" - "\x35\xc1\x7e\x23\x29\xac\xa1\x2e" - "\x21\xd5\x14\xb2\x54\x66\x93\x1c" - "\x7d\x8f\x6a\x5a\xac\x84\xaa\x05" - "\x1b\xa3\x0b\x39\x6a\x0a\xac\x97" - "\x3d\x58\xe0\x91\x47\x3f\x59\x85" - "\x4d\x5c\x2a\xf3\x27\xcd\x64\xa6" - "\x2c\xf3\x5a\xbd\x2b\xa6\xfa\xb4", - .rlen = 80, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08", - .klen = 16, - .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" - "\xde\xca\xf8\x88", - .input = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" - "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" - "\x86\xa7\xa9\x53\x15\x34\xf7\xda" - "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" - "\x1c\x3c\x0c\x95\x95\x68\x09\x53" - "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" - "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" - "\xba\x63\x7b\x39", - .ilen = 60, - .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef" - "\xfe\xed\xfa\xce\xde\xad\xbe\xef" - "\xab\xad\xda\xd2", - .alen = 20, - .result = "\x42\x83\x1e\xc2\x21\x77\x74\x24" - "\x4b\x72\x21\xb7\x84\xd0\xd4\x9c" - "\xe3\xaa\x21\x2f\x2c\x02\xa4\xe0" - "\x35\xc1\x7e\x23\x29\xac\xa1\x2e" - "\x21\xd5\x14\xb2\x54\x66\x93\x1c" - "\x7d\x8f\x6a\x5a\xac\x84\xaa\x05" - "\x1b\xa3\x0b\x39\x6a\x0a\xac\x97" - "\x3d\x58\xe0\x91" - "\x5b\xc9\x4f\xbc\x32\x21\xa5\xdb" - "\x94\xfa\xe9\x5a\xe7\x12\x1a\x47", - .rlen = 76, - }, { - .key = zeroed_string, - .klen = 24, - .result = "\xcd\x33\xb2\x8a\xc7\x73\xf7\x4b" - "\xa0\x0e\xd1\xf3\x12\x57\x24\x35", - .rlen = 16, - }, { - .key = zeroed_string, - .klen = 24, - .input = zeroed_string, - .ilen = 16, - .result = "\x98\xe7\x24\x7c\x07\xf0\xfe\x41" - "\x1c\x26\x7e\x43\x84\xb0\xf6\x00" - "\x2f\xf5\x8d\x80\x03\x39\x27\xab" - "\x8e\xf4\xd4\x58\x75\x14\xf0\xfb", - .rlen = 32, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\xfe\xff\xe9\x92\x86\x65\x73\x1c", - .klen = 24, - .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" - "\xde\xca\xf8\x88", - .input = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" - "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" - "\x86\xa7\xa9\x53\x15\x34\xf7\xda" - "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" - "\x1c\x3c\x0c\x95\x95\x68\x09\x53" - "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" - "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" - "\xba\x63\x7b\x39\x1a\xaf\xd2\x55", - .ilen = 64, - .result = "\x39\x80\xca\x0b\x3c\x00\xe8\x41" - "\xeb\x06\xfa\xc4\x87\x2a\x27\x57" - "\x85\x9e\x1c\xea\xa6\xef\xd9\x84" - "\x62\x85\x93\xb4\x0c\xa1\xe1\x9c" - "\x7d\x77\x3d\x00\xc1\x44\xc5\x25" - "\xac\x61\x9d\x18\xc8\x4a\x3f\x47" - "\x18\xe2\x44\x8b\x2f\xe3\x24\xd9" - "\xcc\xda\x27\x10\xac\xad\xe2\x56" - "\x99\x24\xa7\xc8\x58\x73\x36\xbf" - "\xb1\x18\x02\x4d\xb8\x67\x4a\x14", - .rlen = 80, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\xfe\xff\xe9\x92\x86\x65\x73\x1c", - .klen = 24, - .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" - "\xde\xca\xf8\x88", - .input = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" - "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" - "\x86\xa7\xa9\x53\x15\x34\xf7\xda" - "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" - "\x1c\x3c\x0c\x95\x95\x68\x09\x53" - "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" - "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" - "\xba\x63\x7b\x39", - .ilen = 60, - .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef" - "\xfe\xed\xfa\xce\xde\xad\xbe\xef" - "\xab\xad\xda\xd2", - .alen = 20, - .result = "\x39\x80\xca\x0b\x3c\x00\xe8\x41" - "\xeb\x06\xfa\xc4\x87\x2a\x27\x57" - "\x85\x9e\x1c\xea\xa6\xef\xd9\x84" - "\x62\x85\x93\xb4\x0c\xa1\xe1\x9c" - "\x7d\x77\x3d\x00\xc1\x44\xc5\x25" - "\xac\x61\x9d\x18\xc8\x4a\x3f\x47" - "\x18\xe2\x44\x8b\x2f\xe3\x24\xd9" - "\xcc\xda\x27\x10" - "\x25\x19\x49\x8e\x80\xf1\x47\x8f" - "\x37\xba\x55\xbd\x6d\x27\x61\x8c", - .rlen = 76, - .np = 2, - .tap = { 32, 28 }, - .anp = 2, - .atap = { 8, 12 } - }, { - .key = zeroed_string, - .klen = 32, - .result = "\x53\x0f\x8a\xfb\xc7\x45\x36\xb9" - "\xa9\x63\xb4\xf1\xc4\xcb\x73\x8b", - .rlen = 16, - } -}; - -static const struct aead_testvec aes_gcm_dec_tv_template[] = { - { /* From McGrew & Viega - http://citeseer.ist.psu.edu/656989.html */ - .key = zeroed_string, - .klen = 32, - .input = "\xce\xa7\x40\x3d\x4d\x60\x6b\x6e" - "\x07\x4e\xc5\xd3\xba\xf3\x9d\x18" - "\xd0\xd1\xc8\xa7\x99\x99\x6b\xf0" - "\x26\x5b\x98\xb5\xd4\x8a\xb9\x19", - .ilen = 32, - .result = zeroed_string, - .rlen = 16, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08", - .klen = 32, - .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" - "\xde\xca\xf8\x88", - .input = "\x52\x2d\xc1\xf0\x99\x56\x7d\x07" - "\xf4\x7f\x37\xa3\x2a\x84\x42\x7d" - "\x64\x3a\x8c\xdc\xbf\xe5\xc0\xc9" - "\x75\x98\xa2\xbd\x25\x55\xd1\xaa" - "\x8c\xb0\x8e\x48\x59\x0d\xbb\x3d" - "\xa7\xb0\x8b\x10\x56\x82\x88\x38" - "\xc5\xf6\x1e\x63\x93\xba\x7a\x0a" - "\xbc\xc9\xf6\x62\x89\x80\x15\xad" - "\xb0\x94\xda\xc5\xd9\x34\x71\xbd" - "\xec\x1a\x50\x22\x70\xe3\xcc\x6c", - .ilen = 80, - .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" - "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" - "\x86\xa7\xa9\x53\x15\x34\xf7\xda" - "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" - "\x1c\x3c\x0c\x95\x95\x68\x09\x53" - "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" - "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" - "\xba\x63\x7b\x39\x1a\xaf\xd2\x55", - .rlen = 64, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08", - .klen = 32, - .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" - "\xde\xca\xf8\x88", - .input = "\x52\x2d\xc1\xf0\x99\x56\x7d\x07" - "\xf4\x7f\x37\xa3\x2a\x84\x42\x7d" - "\x64\x3a\x8c\xdc\xbf\xe5\xc0\xc9" - "\x75\x98\xa2\xbd\x25\x55\xd1\xaa" - "\x8c\xb0\x8e\x48\x59\x0d\xbb\x3d" - "\xa7\xb0\x8b\x10\x56\x82\x88\x38" - "\xc5\xf6\x1e\x63\x93\xba\x7a\x0a" - "\xbc\xc9\xf6\x62" - "\x76\xfc\x6e\xce\x0f\x4e\x17\x68" - "\xcd\xdf\x88\x53\xbb\x2d\x55\x1b", - .ilen = 76, - .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef" - "\xfe\xed\xfa\xce\xde\xad\xbe\xef" - "\xab\xad\xda\xd2", - .alen = 20, - .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" - "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" - "\x86\xa7\xa9\x53\x15\x34\xf7\xda" - "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" - "\x1c\x3c\x0c\x95\x95\x68\x09\x53" - "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" - "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" - "\xba\x63\x7b\x39", - .rlen = 60, - .np = 2, - .tap = { 48, 28 }, - .anp = 3, - .atap = { 8, 8, 4 } - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08", - .klen = 16, - .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" - "\xde\xca\xf8\x88", - .input = "\x42\x83\x1e\xc2\x21\x77\x74\x24" - "\x4b\x72\x21\xb7\x84\xd0\xd4\x9c" - "\xe3\xaa\x21\x2f\x2c\x02\xa4\xe0" - "\x35\xc1\x7e\x23\x29\xac\xa1\x2e" - "\x21\xd5\x14\xb2\x54\x66\x93\x1c" - "\x7d\x8f\x6a\x5a\xac\x84\xaa\x05" - "\x1b\xa3\x0b\x39\x6a\x0a\xac\x97" - "\x3d\x58\xe0\x91\x47\x3f\x59\x85" - "\x4d\x5c\x2a\xf3\x27\xcd\x64\xa6" - "\x2c\xf3\x5a\xbd\x2b\xa6\xfa\xb4", - .ilen = 80, - .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" - "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" - "\x86\xa7\xa9\x53\x15\x34\xf7\xda" - "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" - "\x1c\x3c\x0c\x95\x95\x68\x09\x53" - "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" - "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" - "\xba\x63\x7b\x39\x1a\xaf\xd2\x55", - .rlen = 64, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08", - .klen = 16, - .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" - "\xde\xca\xf8\x88", - .input = "\x42\x83\x1e\xc2\x21\x77\x74\x24" - "\x4b\x72\x21\xb7\x84\xd0\xd4\x9c" - "\xe3\xaa\x21\x2f\x2c\x02\xa4\xe0" - "\x35\xc1\x7e\x23\x29\xac\xa1\x2e" - "\x21\xd5\x14\xb2\x54\x66\x93\x1c" - "\x7d\x8f\x6a\x5a\xac\x84\xaa\x05" - "\x1b\xa3\x0b\x39\x6a\x0a\xac\x97" - "\x3d\x58\xe0\x91" - "\x5b\xc9\x4f\xbc\x32\x21\xa5\xdb" - "\x94\xfa\xe9\x5a\xe7\x12\x1a\x47", - .ilen = 76, - .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef" - "\xfe\xed\xfa\xce\xde\xad\xbe\xef" - "\xab\xad\xda\xd2", - .alen = 20, - .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" - "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" - "\x86\xa7\xa9\x53\x15\x34\xf7\xda" - "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" - "\x1c\x3c\x0c\x95\x95\x68\x09\x53" - "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" - "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" - "\xba\x63\x7b\x39", - .rlen = 60, - }, { - .key = zeroed_string, - .klen = 24, - .input = "\x98\xe7\x24\x7c\x07\xf0\xfe\x41" - "\x1c\x26\x7e\x43\x84\xb0\xf6\x00" - "\x2f\xf5\x8d\x80\x03\x39\x27\xab" - "\x8e\xf4\xd4\x58\x75\x14\xf0\xfb", - .ilen = 32, - .result = zeroed_string, - .rlen = 16, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\xfe\xff\xe9\x92\x86\x65\x73\x1c", - .klen = 24, - .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" - "\xde\xca\xf8\x88", - .input = "\x39\x80\xca\x0b\x3c\x00\xe8\x41" - "\xeb\x06\xfa\xc4\x87\x2a\x27\x57" - "\x85\x9e\x1c\xea\xa6\xef\xd9\x84" - "\x62\x85\x93\xb4\x0c\xa1\xe1\x9c" - "\x7d\x77\x3d\x00\xc1\x44\xc5\x25" - "\xac\x61\x9d\x18\xc8\x4a\x3f\x47" - "\x18\xe2\x44\x8b\x2f\xe3\x24\xd9" - "\xcc\xda\x27\x10\xac\xad\xe2\x56" - "\x99\x24\xa7\xc8\x58\x73\x36\xbf" - "\xb1\x18\x02\x4d\xb8\x67\x4a\x14", - .ilen = 80, - .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" - "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" - "\x86\xa7\xa9\x53\x15\x34\xf7\xda" - "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" - "\x1c\x3c\x0c\x95\x95\x68\x09\x53" - "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" - "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" - "\xba\x63\x7b\x39\x1a\xaf\xd2\x55", - .rlen = 64, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\xfe\xff\xe9\x92\x86\x65\x73\x1c", - .klen = 24, - .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" - "\xde\xca\xf8\x88", - .input = "\x39\x80\xca\x0b\x3c\x00\xe8\x41" - "\xeb\x06\xfa\xc4\x87\x2a\x27\x57" - "\x85\x9e\x1c\xea\xa6\xef\xd9\x84" - "\x62\x85\x93\xb4\x0c\xa1\xe1\x9c" - "\x7d\x77\x3d\x00\xc1\x44\xc5\x25" - "\xac\x61\x9d\x18\xc8\x4a\x3f\x47" - "\x18\xe2\x44\x8b\x2f\xe3\x24\xd9" - "\xcc\xda\x27\x10" - "\x25\x19\x49\x8e\x80\xf1\x47\x8f" - "\x37\xba\x55\xbd\x6d\x27\x61\x8c", - .ilen = 76, - .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef" - "\xfe\xed\xfa\xce\xde\xad\xbe\xef" - "\xab\xad\xda\xd2", - .alen = 20, - .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" - "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" - "\x86\xa7\xa9\x53\x15\x34\xf7\xda" - "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" - "\x1c\x3c\x0c\x95\x95\x68\x09\x53" - "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" - "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" - "\xba\x63\x7b\x39", - .rlen = 60, - } -}; - -static const struct aead_testvec aes_gcm_rfc4106_enc_tv_template[] = { - { /* Generated using Crypto++ */ - .key = zeroed_string, - .klen = 20, - .iv = zeroed_string, - .input = zeroed_string, - .ilen = 16, - .assoc = zeroed_string, - .alen = 16, - .result = "\x03\x88\xDA\xCE\x60\xB6\xA3\x92" - "\xF3\x28\xC2\xB9\x71\xB2\xFE\x78" - "\x97\xFE\x4C\x23\x37\x42\x01\xE0" - "\x81\x9F\x8D\xC5\xD7\x41\xA0\x1B", - .rlen = 32, - },{ - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00\x00", - .klen = 20, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = zeroed_string, - .ilen = 16, - .assoc = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .result = "\xC0\x0D\x8B\x42\x0F\x8F\x34\x18" - "\x88\xB1\xC5\xBC\xC5\xB6\xD6\x28" - "\x6A\x9D\xDF\x11\x5E\xFE\x5E\x9D" - "\x2F\x70\x44\x92\xF7\xF2\xE3\xEF", - .rlen = 32, - - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00\x00", - .klen = 20, - .iv = zeroed_string, - .input = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .ilen = 16, - .assoc = zeroed_string, - .alen = 16, - .result = "\x4B\xB1\xB5\xE3\x25\x71\x70\xDE" - "\x7F\xC9\x9C\xA5\x14\x19\xF2\xAC" - "\x0B\x8F\x88\x69\x17\xE6\xB4\x3C" - "\xB1\x68\xFD\x14\x52\x64\x61\xB2", - .rlen = 32, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00\x00", - .klen = 20, - .iv = zeroed_string, - .input = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .ilen = 16, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .alen = 16, - .result = "\x4B\xB1\xB5\xE3\x25\x71\x70\xDE" - "\x7F\xC9\x9C\xA5\x14\x19\xF2\xAC" - "\x90\x92\xB7\xE3\x5F\xA3\x9A\x63" - "\x7E\xD7\x1F\xD8\xD3\x7C\x4B\xF5", - .rlen = 32, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00\x00", - .klen = 20, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .ilen = 16, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .result = "\xC1\x0C\x8A\x43\x0E\x8E\x35\x19" - "\x89\xB0\xC4\xBD\xC4\xB7\xD7\x29" - "\x64\x50\xF9\x32\x13\xFB\x74\x61" - "\xF4\xED\x52\xD3\xC5\x10\x55\x3C", - .rlen = 32, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00\x00", - .klen = 20, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .ilen = 64, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .result = "\xC1\x0C\x8A\x43\x0E\x8E\x35\x19" - "\x89\xB0\xC4\xBD\xC4\xB7\xD7\x29" - "\x98\x14\xA1\x42\x37\x80\xFD\x90" - "\x68\x12\x01\xA8\x91\x89\xB9\x83" - "\x5B\x11\x77\x12\x9B\xFF\x24\x89" - "\x94\x5F\x18\x12\xBA\x27\x09\x39" - "\x99\x96\x76\x42\x15\x1C\xCD\xCB" - "\xDC\xD3\xDA\x65\x73\xAF\x80\xCD" - "\xD2\xB6\xC2\x4A\x76\xC2\x92\x85" - "\xBD\xCF\x62\x98\x58\x14\xE5\xBD", - .rlen = 80, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x00\x00\x00\x00", - .klen = 20, - .iv = "\x00\x00\x45\x67\x89\xab\xcd\xef", - .input = "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff", - .ilen = 192, - .assoc = "\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa" - "\xaa\xaa\xaa\xaa\x00\x00\x45\x67" - "\x89\xab\xcd\xef", - .alen = 20, - .result = "\xC1\x76\x33\x85\xE2\x9B\x5F\xDE" - "\xDE\x89\x3D\x42\xE7\xC9\x69\x8A" - "\x44\x6D\xC3\x88\x46\x2E\xC2\x01" - "\x5E\xF6\x0C\x39\xF0\xC4\xA5\x82" - "\xCD\xE8\x31\xCC\x0A\x4C\xE4\x44" - "\x41\xA9\x82\x6F\x22\xA1\x23\x1A" - "\xA8\xE3\x16\xFD\x31\x5C\x27\x31" - "\xF1\x7F\x01\x63\xA3\xAF\x70\xA1" - "\xCF\x07\x57\x41\x67\xD0\xC4\x42" - "\xDB\x18\xC6\x4C\x4C\xE0\x3D\x9F" - "\x05\x07\xFB\x13\x7D\x4A\xCA\x5B" - "\xF0\xBF\x64\x7E\x05\xB1\x72\xEE" - "\x7C\x3B\xD4\xCD\x14\x03\xB2\x2C" - "\xD3\xA9\xEE\xFA\x17\xFC\x9C\xDF" - "\xC7\x75\x40\xFF\xAE\xAD\x1E\x59" - "\x2F\x30\x24\xFB\xAD\x6B\x10\xFA" - "\x6C\x9F\x5B\xE7\x25\xD5\xD0\x25" - "\xAC\x4A\x4B\xDA\xFC\x7A\x85\x1B" - "\x7E\x13\x06\x82\x08\x17\xA4\x35" - "\xEC\xC5\x8D\x63\x96\x81\x0A\x8F" - "\xA3\x05\x38\x95\x20\x1A\x47\x04" - "\x6F\x6D\xDA\x8F\xEF\xC1\x76\x35" - "\x6B\xC7\x4D\x0F\x94\x12\xCA\x3E" - "\x2E\xD5\x03\x2E\x86\x7E\xAA\x3B" - "\x37\x08\x1C\xCF\xBA\x5D\x71\x46" - "\x80\x72\xB0\x4C\x82\x0D\x60\x3C", - .rlen = 208, - }, { /* From draft-mcgrew-gcm-test-01 */ - .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" - "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" - "\x2E\x44\x3B\x68", - .klen = 20, - .iv = "\x49\x56\xED\x7E\x3B\x24\x4C\xFE", - .input = "\x45\x00\x00\x48\x69\x9A\x00\x00" - "\x80\x11\x4D\xB7\xC0\xA8\x01\x02" - "\xC0\xA8\x01\x01\x0A\x9B\xF1\x56" - "\x38\xD3\x01\x00\x00\x01\x00\x00" - "\x00\x00\x00\x00\x04\x5F\x73\x69" - "\x70\x04\x5F\x75\x64\x70\x03\x73" - "\x69\x70\x09\x63\x79\x62\x65\x72" - "\x63\x69\x74\x79\x02\x64\x6B\x00" - "\x00\x21\x00\x01\x01\x02\x02\x01", - .ilen = 72, - .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" - "\x00\x00\x00\x00\x49\x56\xED\x7E" - "\x3B\x24\x4C\xFE", - .alen = 20, - .result = "\xFE\xCF\x53\x7E\x72\x9D\x5B\x07" - "\xDC\x30\xDF\x52\x8D\xD2\x2B\x76" - "\x8D\x1B\x98\x73\x66\x96\xA6\xFD" - "\x34\x85\x09\xFA\x13\xCE\xAC\x34" - "\xCF\xA2\x43\x6F\x14\xA3\xF3\xCF" - "\x65\x92\x5B\xF1\xF4\xA1\x3C\x5D" - "\x15\xB2\x1E\x18\x84\xF5\xFF\x62" - "\x47\xAE\xAB\xB7\x86\xB9\x3B\xCE" - "\x61\xBC\x17\xD7\x68\xFD\x97\x32" - "\x45\x90\x18\x14\x8F\x6C\xBE\x72" - "\x2F\xD0\x47\x96\x56\x2D\xFD\xB4", - .rlen = 88, - }, { - .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\x6D\x6A\x8F\x94\x67\x30\x83\x08" - "\xCA\xFE\xBA\xBE", - .klen = 20, - .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .input = "\x45\x00\x00\x3E\x69\x8F\x00\x00" - "\x80\x11\x4D\xCC\xC0\xA8\x01\x02" - "\xC0\xA8\x01\x01\x0A\x98\x00\x35" - "\x00\x2A\x23\x43\xB2\xD0\x01\x00" - "\x00\x01\x00\x00\x00\x00\x00\x00" - "\x03\x73\x69\x70\x09\x63\x79\x62" - "\x65\x72\x63\x69\x74\x79\x02\x64" - "\x6B\x00\x00\x01\x00\x01\x00\x01", - .ilen = 64, - .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" - "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .alen = 16, - .result = "\xDE\xB2\x2C\xD9\xB0\x7C\x72\xC1" - "\x6E\x3A\x65\xBE\xEB\x8D\xF3\x04" - "\xA5\xA5\x89\x7D\x33\xAE\x53\x0F" - "\x1B\xA7\x6D\x5D\x11\x4D\x2A\x5C" - "\x3D\xE8\x18\x27\xC1\x0E\x9A\x4F" - "\x51\x33\x0D\x0E\xEC\x41\x66\x42" - "\xCF\xBB\x85\xA5\xB4\x7E\x48\xA4" - "\xEC\x3B\x9B\xA9\x5D\x91\x8B\xD1" - "\x83\xB7\x0D\x3A\xA8\xBC\x6E\xE4" - "\xC3\x09\xE9\xD8\x5A\x41\xAD\x4A", - .rlen = 80, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x11\x22\x33\x44", - .klen = 36, - .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", - .input = "\x45\x00\x00\x30\x69\xA6\x40\x00" - "\x80\x06\x26\x90\xC0\xA8\x01\x02" - "\x93\x89\x15\x5E\x0A\x9E\x00\x8B" - "\x2D\xC5\x7E\xE0\x00\x00\x00\x00" - "\x70\x02\x40\x00\x20\xBF\x00\x00" - "\x02\x04\x05\xB4\x01\x01\x04\x02" - "\x01\x02\x02\x01", - .ilen = 52, - .assoc = "\x4A\x2C\xBF\xE3\x00\x00\x00\x02" - "\x01\x02\x03\x04\x05\x06\x07\x08", - .alen = 16, - .result = "\xFF\x42\x5C\x9B\x72\x45\x99\xDF" - "\x7A\x3B\xCD\x51\x01\x94\xE0\x0D" - "\x6A\x78\x10\x7F\x1B\x0B\x1C\xBF" - "\x06\xEF\xAE\x9D\x65\xA5\xD7\x63" - "\x74\x8A\x63\x79\x85\x77\x1D\x34" - "\x7F\x05\x45\x65\x9F\x14\xE9\x9D" - "\xEF\x84\x2D\x8E\xB3\x35\xF4\xEE" - "\xCF\xDB\xF8\x31\x82\x4B\x4C\x49" - "\x15\x95\x6C\x96", - .rlen = 68, - }, { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00", - .klen = 20, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x45\x00\x00\x3C\x99\xC5\x00\x00" - "\x80\x01\xCB\x7A\x40\x67\x93\x18" - "\x01\x01\x01\x01\x08\x00\x07\x5C" - "\x02\x00\x44\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x75\x76\x77\x61\x62\x63\x64\x65" - "\x66\x67\x68\x69\x01\x02\x02\x01", - .ilen = 64, - .assoc = "\x00\x00\x00\x00\x00\x00\x00\x01" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .alen = 16, - .result = "\x46\x88\xDA\xF2\xF9\x73\xA3\x92" - "\x73\x29\x09\xC3\x31\xD5\x6D\x60" - "\xF6\x94\xAB\xAA\x41\x4B\x5E\x7F" - "\xF5\xFD\xCD\xFF\xF5\xE9\xA2\x84" - "\x45\x64\x76\x49\x27\x19\xFF\xB6" - "\x4D\xE7\xD9\xDC\xA1\xE1\xD8\x94" - "\xBC\x3B\xD5\x78\x73\xED\x4D\x18" - "\x1D\x19\xD4\xD5\xC8\xC1\x8A\xF3" - "\xF8\x21\xD4\x96\xEE\xB0\x96\xE9" - "\x8A\xD2\xB6\x9E\x47\x99\xC7\x1D", - .rlen = 80, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E\x43", - .klen = 20, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .input = "\x45\x00\x00\x3C\x99\xC3\x00\x00" - "\x80\x01\xCB\x7C\x40\x67\x93\x18" - "\x01\x01\x01\x01\x08\x00\x08\x5C" - "\x02\x00\x43\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x75\x76\x77\x61\x62\x63\x64\x65" - "\x66\x67\x68\x69\x01\x02\x02\x01", - .ilen = 64, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .result = "\xFB\xA2\xCA\xA4\x85\x3C\xF9\xF0" - "\xF2\x2C\xB1\x0D\x86\xDD\x83\xB0" - "\xFE\xC7\x56\x91\xCF\x1A\x04\xB0" - "\x0D\x11\x38\xEC\x9C\x35\x79\x17" - "\x65\xAC\xBD\x87\x01\xAD\x79\x84" - "\x5B\xF9\xFE\x3F\xBA\x48\x7B\xC9" - "\x17\x55\xE6\x66\x2B\x4C\x8D\x0D" - "\x1F\x5E\x22\x73\x95\x30\x32\x0A" - "\xE0\xD7\x31\xCC\x97\x8E\xCA\xFA" - "\xEA\xE8\x8F\x00\xE8\x0D\x6E\x48", - .rlen = 80, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E\x43", - .klen = 20, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .input = "\x45\x00\x00\x1C\x42\xA2\x00\x00" - "\x80\x01\x44\x1F\x40\x67\x93\xB6" - "\xE0\x00\x00\x02\x0A\x00\xF5\xFF" - "\x01\x02\x02\x01", - .ilen = 28, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .result = "\xFB\xA2\xCA\x84\x5E\x5D\xF9\xF0" - "\xF2\x2C\x3E\x6E\x86\xDD\x83\x1E" - "\x1F\xC6\x57\x92\xCD\x1A\xF9\x13" - "\x0E\x13\x79\xED\x36\x9F\x07\x1F" - "\x35\xE0\x34\xBE\x95\xF1\x12\xE4" - "\xE7\xD0\x5D\x35", - .rlen = 44, - }, { - .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\x6D\x6A\x8F\x94\x67\x30\x83\x08" - "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\xCA\xFE\xBA\xBE", - .klen = 28, - .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .input = "\x45\x00\x00\x28\xA4\xAD\x40\x00" - "\x40\x06\x78\x80\x0A\x01\x03\x8F" - "\x0A\x01\x06\x12\x80\x23\x06\xB8" - "\xCB\x71\x26\x02\xDD\x6B\xB0\x3E" - "\x50\x10\x16\xD0\x75\x68\x00\x01", - .ilen = 40, - .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" - "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .alen = 16, - .result = "\xA5\xB1\xF8\x06\x60\x29\xAE\xA4" - "\x0E\x59\x8B\x81\x22\xDE\x02\x42" - "\x09\x38\xB3\xAB\x33\xF8\x28\xE6" - "\x87\xB8\x85\x8B\x5B\xFB\xDB\xD0" - "\x31\x5B\x27\x45\x21\x44\xCC\x77" - "\x95\x45\x7B\x96\x52\x03\x7F\x53" - "\x18\x02\x7B\x5B\x4C\xD7\xA6\x36", - .rlen = 56, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xDE\xCA\xF8\x88", - .klen = 20, - .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", - .input = "\x45\x00\x00\x49\x33\xBA\x00\x00" - "\x7F\x11\x91\x06\xC3\xFB\x1D\x10" - "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" - "\x00\x35\xDD\x7B\x80\x03\x02\xD5" - "\x00\x00\x4E\x20\x00\x1E\x8C\x18" - "\xD7\x5B\x81\xDC\x91\xBA\xA0\x47" - "\x6B\x91\xB9\x24\xB2\x80\x38\x9D" - "\x92\xC9\x63\xBA\xC0\x46\xEC\x95" - "\x9B\x62\x66\xC0\x47\x22\xB1\x49" - "\x23\x01\x01\x01", - .ilen = 76, - .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" - "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" - "\xCE\xFA\xCE\x74", - .alen = 20, - .result = "\x18\xA6\xFD\x42\xF7\x2C\xBF\x4A" - "\xB2\xA2\xEA\x90\x1F\x73\xD8\x14" - "\xE3\xE7\xF2\x43\xD9\x54\x12\xE1" - "\xC3\x49\xC1\xD2\xFB\xEC\x16\x8F" - "\x91\x90\xFE\xEB\xAF\x2C\xB0\x19" - "\x84\xE6\x58\x63\x96\x5D\x74\x72" - "\xB7\x9D\xA3\x45\xE0\xE7\x80\x19" - "\x1F\x0D\x2F\x0E\x0F\x49\x6C\x22" - "\x6F\x21\x27\xB2\x7D\xB3\x57\x24" - "\xE7\x84\x5D\x68\x65\x1F\x57\xE6" - "\x5F\x35\x4F\x75\xFF\x17\x01\x57" - "\x69\x62\x34\x36", - .rlen = 92, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x73\x61\x6C\x74", - .klen = 36, - .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", - .input = "\x45\x08\x00\x28\x73\x2C\x00\x00" - "\x40\x06\xE9\xF9\x0A\x01\x06\x12" - "\x0A\x01\x03\x8F\x06\xB8\x80\x23" - "\xDD\x6B\xAF\xBE\xCB\x71\x26\x02" - "\x50\x10\x1F\x64\x6D\x54\x00\x01", - .ilen = 40, - .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" - "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" - "\x69\x76\x65\x63", - .alen = 20, - .result = "\xF2\xD6\x9E\xCD\xBD\x5A\x0D\x5B" - "\x8D\x5E\xF3\x8B\xAD\x4D\xA5\x8D" - "\x1F\x27\x8F\xDE\x98\xEF\x67\x54" - "\x9D\x52\x4A\x30\x18\xD9\xA5\x7F" - "\xF4\xD3\xA3\x1C\xE6\x73\x11\x9E" - "\x45\x16\x26\xC2\x41\x57\x71\xE3" - "\xB7\xEE\xBC\xA6\x14\xC8\x9B\x35", - .rlen = 56, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E\x43", - .klen = 20, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .input = "\x45\x00\x00\x49\x33\x3E\x00\x00" - "\x7F\x11\x91\x82\xC3\xFB\x1D\x10" - "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" - "\x00\x35\xCB\x45\x80\x03\x02\x5B" - "\x00\x00\x01\xE0\x00\x1E\x8C\x18" - "\xD6\x57\x59\xD5\x22\x84\xA0\x35" - "\x2C\x71\x47\x5C\x88\x80\x39\x1C" - "\x76\x4D\x6E\x5E\xE0\x49\x6B\x32" - "\x5A\xE2\x70\xC0\x38\x99\x49\x39" - "\x15\x01\x01\x01", - .ilen = 76, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .result = "\xFB\xA2\xCA\xD1\x2F\xC1\xF9\xF0" - "\x0D\x3C\xEB\xF3\x05\x41\x0D\xB8" - "\x3D\x77\x84\xB6\x07\x32\x3D\x22" - "\x0F\x24\xB0\xA9\x7D\x54\x18\x28" - "\x00\xCA\xDB\x0F\x68\xD9\x9E\xF0" - "\xE0\xC0\xC8\x9A\xE9\xBE\xA8\x88" - "\x4E\x52\xD6\x5B\xC1\xAF\xD0\x74" - "\x0F\x74\x24\x44\x74\x7B\x5B\x39" - "\xAB\x53\x31\x63\xAA\xD4\x55\x0E" - "\xE5\x16\x09\x75\xCD\xB6\x08\xC5" - "\x76\x91\x89\x60\x97\x63\xB8\xE1" - "\x8C\xAA\x81\xE2", - .rlen = 92, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x73\x61\x6C\x74", - .klen = 36, - .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", - .input = "\x63\x69\x73\x63\x6F\x01\x72\x75" - "\x6C\x65\x73\x01\x74\x68\x65\x01" - "\x6E\x65\x74\x77\x65\x01\x64\x65" - "\x66\x69\x6E\x65\x01\x74\x68\x65" - "\x74\x65\x63\x68\x6E\x6F\x6C\x6F" - "\x67\x69\x65\x73\x01\x74\x68\x61" - "\x74\x77\x69\x6C\x6C\x01\x64\x65" - "\x66\x69\x6E\x65\x74\x6F\x6D\x6F" - "\x72\x72\x6F\x77\x01\x02\x02\x01", - .ilen = 72, - .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" - "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" - "\x69\x76\x65\x63", - .alen = 20, - .result = "\xD4\xB7\xED\x86\xA1\x77\x7F\x2E" - "\xA1\x3D\x69\x73\xD3\x24\xC6\x9E" - "\x7B\x43\xF8\x26\xFB\x56\x83\x12" - "\x26\x50\x8B\xEB\xD2\xDC\xEB\x18" - "\xD0\xA6\xDF\x10\xE5\x48\x7D\xF0" - "\x74\x11\x3E\x14\xC6\x41\x02\x4E" - "\x3E\x67\x73\xD9\x1A\x62\xEE\x42" - "\x9B\x04\x3A\x10\xE3\xEF\xE6\xB0" - "\x12\xA4\x93\x63\x41\x23\x64\xF8" - "\xC0\xCA\xC5\x87\xF2\x49\xE5\x6B" - "\x11\xE2\x4F\x30\xE4\x4C\xCC\x76", - .rlen = 88, - }, { - .key = "\x7D\x77\x3D\x00\xC1\x44\xC5\x25" - "\xAC\x61\x9D\x18\xC8\x4A\x3F\x47" - "\xD9\x66\x42\x67", - .klen = 20, - .iv = "\x43\x45\x7E\x91\x82\x44\x3B\xC6", - .input = "\x01\x02\x02\x01", - .ilen = 4, - .assoc = "\x33\x54\x67\xAE\xFF\xFF\xFF\xFF" - "\x43\x45\x7E\x91\x82\x44\x3B\xC6", - .alen = 16, - .result = "\x43\x7F\x86\x6B\xCB\x3F\x69\x9F" - "\xE9\xB0\x82\x2B\xAC\x96\x1C\x45" - "\x04\xBE\xF2\x70", - .rlen = 20, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xDE\xCA\xF8\x88", - .klen = 20, - .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", - .input = "\x74\x6F\x01\x62\x65\x01\x6F\x72" - "\x01\x6E\x6F\x74\x01\x74\x6F\x01" - "\x62\x65\x00\x01", - .ilen = 20, - .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" - "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" - "\xCE\xFA\xCE\x74", - .alen = 20, - .result = "\x29\xC9\xFC\x69\xA1\x97\xD0\x38" - "\xCC\xDD\x14\xE2\xDD\xFC\xAA\x05" - "\x43\x33\x21\x64\x41\x25\x03\x52" - "\x43\x03\xED\x3C\x6C\x5F\x28\x38" - "\x43\xAF\x8C\x3E", - .rlen = 36, - }, { - .key = "\x6C\x65\x67\x61\x6C\x69\x7A\x65" - "\x6D\x61\x72\x69\x6A\x75\x61\x6E" - "\x61\x61\x6E\x64\x64\x6F\x69\x74" - "\x62\x65\x66\x6F\x72\x65\x69\x61" - "\x74\x75\x72\x6E", - .klen = 36, - .iv = "\x33\x30\x21\x69\x67\x65\x74\x6D", - .input = "\x45\x00\x00\x30\xDA\x3A\x00\x00" - "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" - "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01", - .ilen = 52, - .assoc = "\x79\x6B\x69\x63\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\x33\x30\x21\x69" - "\x67\x65\x74\x6D", - .alen = 20, - .result = "\xF9\x7A\xB2\xAA\x35\x6D\x8E\xDC" - "\xE1\x76\x44\xAC\x8C\x78\xE2\x5D" - "\xD2\x4D\xED\xBB\x29\xEB\xF1\xB6" - "\x4A\x27\x4B\x39\xB4\x9C\x3A\x86" - "\x4C\xD3\xD7\x8C\xA4\xAE\x68\xA3" - "\x2B\x42\x45\x8F\xB5\x7D\xBE\x82" - "\x1D\xCC\x63\xB9\xD0\x93\x7B\xA2" - "\x94\x5F\x66\x93\x68\x66\x1A\x32" - "\x9F\xB4\xC0\x53", - .rlen = 68, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E\x43", - .klen = 20, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .input = "\x45\x00\x00\x30\xDA\x3A\x00\x00" - "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" - "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01", - .ilen = 52, - .assoc = "\x3F\x7E\xF6\x42\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .result = "\xFB\xA2\xCA\xA8\xC6\xC5\xF9\xF0" - "\xF2\x2C\xA5\x4A\x06\x12\x10\xAD" - "\x3F\x6E\x57\x91\xCF\x1A\xCA\x21" - "\x0D\x11\x7C\xEC\x9C\x35\x79\x17" - "\x65\xAC\xBD\x87\x01\xAD\x79\x84" - "\x5B\xF9\xFE\x3F\xBA\x48\x7B\xC9" - "\x63\x21\x93\x06\x84\xEE\xCA\xDB" - "\x56\x91\x25\x46\xE7\xA9\x5C\x97" - "\x40\xD7\xCB\x05", - .rlen = 68, - }, { - .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" - "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" - "\x22\x43\x3C\x64", - .klen = 20, - .iv = "\x48\x55\xEC\x7D\x3A\x23\x4B\xFD", - .input = "\x08\x00\xC6\xCD\x02\x00\x07\x00" - "\x61\x62\x63\x64\x65\x66\x67\x68" - "\x69\x6A\x6B\x6C\x6D\x6E\x6F\x70" - "\x71\x72\x73\x74\x01\x02\x02\x01", - .ilen = 32, - .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" - "\x00\x00\x00\x07\x48\x55\xEC\x7D" - "\x3A\x23\x4B\xFD", - .alen = 20, - .result = "\x74\x75\x2E\x8A\xEB\x5D\x87\x3C" - "\xD7\xC0\xF4\xAC\xC3\x6C\x4B\xFF" - "\x84\xB7\xD7\xB9\x8F\x0C\xA8\xB6" - "\xAC\xDA\x68\x94\xBC\x61\x90\x69" - "\xEF\x9C\xBC\x28\xFE\x1B\x56\xA7" - "\xC4\xE0\xD5\x8C\x86\xCD\x2B\xC0", - .rlen = 48, - } -}; - -static const struct aead_testvec aes_gcm_rfc4106_dec_tv_template[] = { - { /* Generated using Crypto++ */ - .key = zeroed_string, - .klen = 20, - .iv = zeroed_string, - .input = "\x03\x88\xDA\xCE\x60\xB6\xA3\x92" - "\xF3\x28\xC2\xB9\x71\xB2\xFE\x78" - "\x97\xFE\x4C\x23\x37\x42\x01\xE0" - "\x81\x9F\x8D\xC5\xD7\x41\xA0\x1B", - .ilen = 32, - .assoc = zeroed_string, - .alen = 16, - .result = zeroed_string, - .rlen = 16, - - },{ - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00\x00", - .klen = 20, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\xC0\x0D\x8B\x42\x0F\x8F\x34\x18" - "\x88\xB1\xC5\xBC\xC5\xB6\xD6\x28" - "\x6A\x9D\xDF\x11\x5E\xFE\x5E\x9D" - "\x2F\x70\x44\x92\xF7\xF2\xE3\xEF", - .ilen = 32, - .assoc = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .result = zeroed_string, - .rlen = 16, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00\x00", - .klen = 20, - .iv = zeroed_string, - .input = "\x4B\xB1\xB5\xE3\x25\x71\x70\xDE" - "\x7F\xC9\x9C\xA5\x14\x19\xF2\xAC" - "\x0B\x8F\x88\x69\x17\xE6\xB4\x3C" - "\xB1\x68\xFD\x14\x52\x64\x61\xB2", - .ilen = 32, - .assoc = zeroed_string, - .alen = 16, - .result = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .rlen = 16, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00\x00", - .klen = 20, - .iv = zeroed_string, - .input = "\x4B\xB1\xB5\xE3\x25\x71\x70\xDE" - "\x7F\xC9\x9C\xA5\x14\x19\xF2\xAC" - "\x90\x92\xB7\xE3\x5F\xA3\x9A\x63" - "\x7E\xD7\x1F\xD8\xD3\x7C\x4B\xF5", - .ilen = 32, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .alen = 16, - .result = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .rlen = 16, - - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00\x00", - .klen = 20, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\xC1\x0C\x8A\x43\x0E\x8E\x35\x19" - "\x89\xB0\xC4\xBD\xC4\xB7\xD7\x29" - "\x64\x50\xF9\x32\x13\xFB\x74\x61" - "\xF4\xED\x52\xD3\xC5\x10\x55\x3C", - .ilen = 32, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .result = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .rlen = 16, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00\x00", - .klen = 20, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\xC1\x0C\x8A\x43\x0E\x8E\x35\x19" - "\x89\xB0\xC4\xBD\xC4\xB7\xD7\x29" - "\x98\x14\xA1\x42\x37\x80\xFD\x90" - "\x68\x12\x01\xA8\x91\x89\xB9\x83" - "\x5B\x11\x77\x12\x9B\xFF\x24\x89" - "\x94\x5F\x18\x12\xBA\x27\x09\x39" - "\x99\x96\x76\x42\x15\x1C\xCD\xCB" - "\xDC\xD3\xDA\x65\x73\xAF\x80\xCD" - "\xD2\xB6\xC2\x4A\x76\xC2\x92\x85" - "\xBD\xCF\x62\x98\x58\x14\xE5\xBD", - .ilen = 80, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .result = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .rlen = 64, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x00\x00\x00\x00", - .klen = 20, - .iv = "\x00\x00\x45\x67\x89\xab\xcd\xef", - .input = "\xC1\x76\x33\x85\xE2\x9B\x5F\xDE" - "\xDE\x89\x3D\x42\xE7\xC9\x69\x8A" - "\x44\x6D\xC3\x88\x46\x2E\xC2\x01" - "\x5E\xF6\x0C\x39\xF0\xC4\xA5\x82" - "\xCD\xE8\x31\xCC\x0A\x4C\xE4\x44" - "\x41\xA9\x82\x6F\x22\xA1\x23\x1A" - "\xA8\xE3\x16\xFD\x31\x5C\x27\x31" - "\xF1\x7F\x01\x63\xA3\xAF\x70\xA1" - "\xCF\x07\x57\x41\x67\xD0\xC4\x42" - "\xDB\x18\xC6\x4C\x4C\xE0\x3D\x9F" - "\x05\x07\xFB\x13\x7D\x4A\xCA\x5B" - "\xF0\xBF\x64\x7E\x05\xB1\x72\xEE" - "\x7C\x3B\xD4\xCD\x14\x03\xB2\x2C" - "\xD3\xA9\xEE\xFA\x17\xFC\x9C\xDF" - "\xC7\x75\x40\xFF\xAE\xAD\x1E\x59" - "\x2F\x30\x24\xFB\xAD\x6B\x10\xFA" - "\x6C\x9F\x5B\xE7\x25\xD5\xD0\x25" - "\xAC\x4A\x4B\xDA\xFC\x7A\x85\x1B" - "\x7E\x13\x06\x82\x08\x17\xA4\x35" - "\xEC\xC5\x8D\x63\x96\x81\x0A\x8F" - "\xA3\x05\x38\x95\x20\x1A\x47\x04" - "\x6F\x6D\xDA\x8F\xEF\xC1\x76\x35" - "\x6B\xC7\x4D\x0F\x94\x12\xCA\x3E" - "\x2E\xD5\x03\x2E\x86\x7E\xAA\x3B" - "\x37\x08\x1C\xCF\xBA\x5D\x71\x46" - "\x80\x72\xB0\x4C\x82\x0D\x60\x3C", - .ilen = 208, - .assoc = "\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa" - "\xaa\xaa\xaa\xaa\x00\x00\x45\x67" - "\x89\xab\xcd\xef", - .alen = 20, - .result = "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff", - .rlen = 192, - }, { - .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" - "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" - "\x2E\x44\x3B\x68", - .klen = 20, - .iv = "\x49\x56\xED\x7E\x3B\x24\x4C\xFE", - .result = "\x45\x00\x00\x48\x69\x9A\x00\x00" - "\x80\x11\x4D\xB7\xC0\xA8\x01\x02" - "\xC0\xA8\x01\x01\x0A\x9B\xF1\x56" - "\x38\xD3\x01\x00\x00\x01\x00\x00" - "\x00\x00\x00\x00\x04\x5F\x73\x69" - "\x70\x04\x5F\x75\x64\x70\x03\x73" - "\x69\x70\x09\x63\x79\x62\x65\x72" - "\x63\x69\x74\x79\x02\x64\x6B\x00" - "\x00\x21\x00\x01\x01\x02\x02\x01", - .rlen = 72, - .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" - "\x00\x00\x00\x00\x49\x56\xED\x7E" - "\x3B\x24\x4C\xFE", - .alen = 20, - .input = "\xFE\xCF\x53\x7E\x72\x9D\x5B\x07" - "\xDC\x30\xDF\x52\x8D\xD2\x2B\x76" - "\x8D\x1B\x98\x73\x66\x96\xA6\xFD" - "\x34\x85\x09\xFA\x13\xCE\xAC\x34" - "\xCF\xA2\x43\x6F\x14\xA3\xF3\xCF" - "\x65\x92\x5B\xF1\xF4\xA1\x3C\x5D" - "\x15\xB2\x1E\x18\x84\xF5\xFF\x62" - "\x47\xAE\xAB\xB7\x86\xB9\x3B\xCE" - "\x61\xBC\x17\xD7\x68\xFD\x97\x32" - "\x45\x90\x18\x14\x8F\x6C\xBE\x72" - "\x2F\xD0\x47\x96\x56\x2D\xFD\xB4", - .ilen = 88, - }, { - .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\x6D\x6A\x8F\x94\x67\x30\x83\x08" - "\xCA\xFE\xBA\xBE", - .klen = 20, - .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .result = "\x45\x00\x00\x3E\x69\x8F\x00\x00" - "\x80\x11\x4D\xCC\xC0\xA8\x01\x02" - "\xC0\xA8\x01\x01\x0A\x98\x00\x35" - "\x00\x2A\x23\x43\xB2\xD0\x01\x00" - "\x00\x01\x00\x00\x00\x00\x00\x00" - "\x03\x73\x69\x70\x09\x63\x79\x62" - "\x65\x72\x63\x69\x74\x79\x02\x64" - "\x6B\x00\x00\x01\x00\x01\x00\x01", - .rlen = 64, - .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" - "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .alen = 16, - .input = "\xDE\xB2\x2C\xD9\xB0\x7C\x72\xC1" - "\x6E\x3A\x65\xBE\xEB\x8D\xF3\x04" - "\xA5\xA5\x89\x7D\x33\xAE\x53\x0F" - "\x1B\xA7\x6D\x5D\x11\x4D\x2A\x5C" - "\x3D\xE8\x18\x27\xC1\x0E\x9A\x4F" - "\x51\x33\x0D\x0E\xEC\x41\x66\x42" - "\xCF\xBB\x85\xA5\xB4\x7E\x48\xA4" - "\xEC\x3B\x9B\xA9\x5D\x91\x8B\xD1" - "\x83\xB7\x0D\x3A\xA8\xBC\x6E\xE4" - "\xC3\x09\xE9\xD8\x5A\x41\xAD\x4A", - .ilen = 80, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x11\x22\x33\x44", - .klen = 36, - .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", - .result = "\x45\x00\x00\x30\x69\xA6\x40\x00" - "\x80\x06\x26\x90\xC0\xA8\x01\x02" - "\x93\x89\x15\x5E\x0A\x9E\x00\x8B" - "\x2D\xC5\x7E\xE0\x00\x00\x00\x00" - "\x70\x02\x40\x00\x20\xBF\x00\x00" - "\x02\x04\x05\xB4\x01\x01\x04\x02" - "\x01\x02\x02\x01", - .rlen = 52, - .assoc = "\x4A\x2C\xBF\xE3\x00\x00\x00\x02" - "\x01\x02\x03\x04\x05\x06\x07\x08", - .alen = 16, - .input = "\xFF\x42\x5C\x9B\x72\x45\x99\xDF" - "\x7A\x3B\xCD\x51\x01\x94\xE0\x0D" - "\x6A\x78\x10\x7F\x1B\x0B\x1C\xBF" - "\x06\xEF\xAE\x9D\x65\xA5\xD7\x63" - "\x74\x8A\x63\x79\x85\x77\x1D\x34" - "\x7F\x05\x45\x65\x9F\x14\xE9\x9D" - "\xEF\x84\x2D\x8E\xB3\x35\xF4\xEE" - "\xCF\xDB\xF8\x31\x82\x4B\x4C\x49" - "\x15\x95\x6C\x96", - .ilen = 68, - }, { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00", - .klen = 20, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .result = "\x45\x00\x00\x3C\x99\xC5\x00\x00" - "\x80\x01\xCB\x7A\x40\x67\x93\x18" - "\x01\x01\x01\x01\x08\x00\x07\x5C" - "\x02\x00\x44\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x75\x76\x77\x61\x62\x63\x64\x65" - "\x66\x67\x68\x69\x01\x02\x02\x01", - .rlen = 64, - .assoc = "\x00\x00\x00\x00\x00\x00\x00\x01" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .alen = 16, - .input = "\x46\x88\xDA\xF2\xF9\x73\xA3\x92" - "\x73\x29\x09\xC3\x31\xD5\x6D\x60" - "\xF6\x94\xAB\xAA\x41\x4B\x5E\x7F" - "\xF5\xFD\xCD\xFF\xF5\xE9\xA2\x84" - "\x45\x64\x76\x49\x27\x19\xFF\xB6" - "\x4D\xE7\xD9\xDC\xA1\xE1\xD8\x94" - "\xBC\x3B\xD5\x78\x73\xED\x4D\x18" - "\x1D\x19\xD4\xD5\xC8\xC1\x8A\xF3" - "\xF8\x21\xD4\x96\xEE\xB0\x96\xE9" - "\x8A\xD2\xB6\x9E\x47\x99\xC7\x1D", - .ilen = 80, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E\x43", - .klen = 20, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .result = "\x45\x00\x00\x3C\x99\xC3\x00\x00" - "\x80\x01\xCB\x7C\x40\x67\x93\x18" - "\x01\x01\x01\x01\x08\x00\x08\x5C" - "\x02\x00\x43\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x75\x76\x77\x61\x62\x63\x64\x65" - "\x66\x67\x68\x69\x01\x02\x02\x01", - .rlen = 64, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .input = "\xFB\xA2\xCA\xA4\x85\x3C\xF9\xF0" - "\xF2\x2C\xB1\x0D\x86\xDD\x83\xB0" - "\xFE\xC7\x56\x91\xCF\x1A\x04\xB0" - "\x0D\x11\x38\xEC\x9C\x35\x79\x17" - "\x65\xAC\xBD\x87\x01\xAD\x79\x84" - "\x5B\xF9\xFE\x3F\xBA\x48\x7B\xC9" - "\x17\x55\xE6\x66\x2B\x4C\x8D\x0D" - "\x1F\x5E\x22\x73\x95\x30\x32\x0A" - "\xE0\xD7\x31\xCC\x97\x8E\xCA\xFA" - "\xEA\xE8\x8F\x00\xE8\x0D\x6E\x48", - .ilen = 80, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E\x43", - .klen = 20, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .result = "\x45\x00\x00\x1C\x42\xA2\x00\x00" - "\x80\x01\x44\x1F\x40\x67\x93\xB6" - "\xE0\x00\x00\x02\x0A\x00\xF5\xFF" - "\x01\x02\x02\x01", - .rlen = 28, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .input = "\xFB\xA2\xCA\x84\x5E\x5D\xF9\xF0" - "\xF2\x2C\x3E\x6E\x86\xDD\x83\x1E" - "\x1F\xC6\x57\x92\xCD\x1A\xF9\x13" - "\x0E\x13\x79\xED\x36\x9F\x07\x1F" - "\x35\xE0\x34\xBE\x95\xF1\x12\xE4" - "\xE7\xD0\x5D\x35", - .ilen = 44, - }, { - .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\x6D\x6A\x8F\x94\x67\x30\x83\x08" - "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\xCA\xFE\xBA\xBE", - .klen = 28, - .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .result = "\x45\x00\x00\x28\xA4\xAD\x40\x00" - "\x40\x06\x78\x80\x0A\x01\x03\x8F" - "\x0A\x01\x06\x12\x80\x23\x06\xB8" - "\xCB\x71\x26\x02\xDD\x6B\xB0\x3E" - "\x50\x10\x16\xD0\x75\x68\x00\x01", - .rlen = 40, - .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" - "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .alen = 16, - .input = "\xA5\xB1\xF8\x06\x60\x29\xAE\xA4" - "\x0E\x59\x8B\x81\x22\xDE\x02\x42" - "\x09\x38\xB3\xAB\x33\xF8\x28\xE6" - "\x87\xB8\x85\x8B\x5B\xFB\xDB\xD0" - "\x31\x5B\x27\x45\x21\x44\xCC\x77" - "\x95\x45\x7B\x96\x52\x03\x7F\x53" - "\x18\x02\x7B\x5B\x4C\xD7\xA6\x36", - .ilen = 56, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xDE\xCA\xF8\x88", - .klen = 20, - .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", - .result = "\x45\x00\x00\x49\x33\xBA\x00\x00" - "\x7F\x11\x91\x06\xC3\xFB\x1D\x10" - "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" - "\x00\x35\xDD\x7B\x80\x03\x02\xD5" - "\x00\x00\x4E\x20\x00\x1E\x8C\x18" - "\xD7\x5B\x81\xDC\x91\xBA\xA0\x47" - "\x6B\x91\xB9\x24\xB2\x80\x38\x9D" - "\x92\xC9\x63\xBA\xC0\x46\xEC\x95" - "\x9B\x62\x66\xC0\x47\x22\xB1\x49" - "\x23\x01\x01\x01", - .rlen = 76, - .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" - "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" - "\xCE\xFA\xCE\x74", - .alen = 20, - .input = "\x18\xA6\xFD\x42\xF7\x2C\xBF\x4A" - "\xB2\xA2\xEA\x90\x1F\x73\xD8\x14" - "\xE3\xE7\xF2\x43\xD9\x54\x12\xE1" - "\xC3\x49\xC1\xD2\xFB\xEC\x16\x8F" - "\x91\x90\xFE\xEB\xAF\x2C\xB0\x19" - "\x84\xE6\x58\x63\x96\x5D\x74\x72" - "\xB7\x9D\xA3\x45\xE0\xE7\x80\x19" - "\x1F\x0D\x2F\x0E\x0F\x49\x6C\x22" - "\x6F\x21\x27\xB2\x7D\xB3\x57\x24" - "\xE7\x84\x5D\x68\x65\x1F\x57\xE6" - "\x5F\x35\x4F\x75\xFF\x17\x01\x57" - "\x69\x62\x34\x36", - .ilen = 92, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x73\x61\x6C\x74", - .klen = 36, - .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", - .result = "\x45\x08\x00\x28\x73\x2C\x00\x00" - "\x40\x06\xE9\xF9\x0A\x01\x06\x12" - "\x0A\x01\x03\x8F\x06\xB8\x80\x23" - "\xDD\x6B\xAF\xBE\xCB\x71\x26\x02" - "\x50\x10\x1F\x64\x6D\x54\x00\x01", - .rlen = 40, - .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" - "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" - "\x69\x76\x65\x63", - .alen = 20, - .input = "\xF2\xD6\x9E\xCD\xBD\x5A\x0D\x5B" - "\x8D\x5E\xF3\x8B\xAD\x4D\xA5\x8D" - "\x1F\x27\x8F\xDE\x98\xEF\x67\x54" - "\x9D\x52\x4A\x30\x18\xD9\xA5\x7F" - "\xF4\xD3\xA3\x1C\xE6\x73\x11\x9E" - "\x45\x16\x26\xC2\x41\x57\x71\xE3" - "\xB7\xEE\xBC\xA6\x14\xC8\x9B\x35", - .ilen = 56, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E\x43", - .klen = 20, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .result = "\x45\x00\x00\x49\x33\x3E\x00\x00" - "\x7F\x11\x91\x82\xC3\xFB\x1D\x10" - "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" - "\x00\x35\xCB\x45\x80\x03\x02\x5B" - "\x00\x00\x01\xE0\x00\x1E\x8C\x18" - "\xD6\x57\x59\xD5\x22\x84\xA0\x35" - "\x2C\x71\x47\x5C\x88\x80\x39\x1C" - "\x76\x4D\x6E\x5E\xE0\x49\x6B\x32" - "\x5A\xE2\x70\xC0\x38\x99\x49\x39" - "\x15\x01\x01\x01", - .rlen = 76, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .input = "\xFB\xA2\xCA\xD1\x2F\xC1\xF9\xF0" - "\x0D\x3C\xEB\xF3\x05\x41\x0D\xB8" - "\x3D\x77\x84\xB6\x07\x32\x3D\x22" - "\x0F\x24\xB0\xA9\x7D\x54\x18\x28" - "\x00\xCA\xDB\x0F\x68\xD9\x9E\xF0" - "\xE0\xC0\xC8\x9A\xE9\xBE\xA8\x88" - "\x4E\x52\xD6\x5B\xC1\xAF\xD0\x74" - "\x0F\x74\x24\x44\x74\x7B\x5B\x39" - "\xAB\x53\x31\x63\xAA\xD4\x55\x0E" - "\xE5\x16\x09\x75\xCD\xB6\x08\xC5" - "\x76\x91\x89\x60\x97\x63\xB8\xE1" - "\x8C\xAA\x81\xE2", - .ilen = 92, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x73\x61\x6C\x74", - .klen = 36, - .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", - .result = "\x63\x69\x73\x63\x6F\x01\x72\x75" - "\x6C\x65\x73\x01\x74\x68\x65\x01" - "\x6E\x65\x74\x77\x65\x01\x64\x65" - "\x66\x69\x6E\x65\x01\x74\x68\x65" - "\x74\x65\x63\x68\x6E\x6F\x6C\x6F" - "\x67\x69\x65\x73\x01\x74\x68\x61" - "\x74\x77\x69\x6C\x6C\x01\x64\x65" - "\x66\x69\x6E\x65\x74\x6F\x6D\x6F" - "\x72\x72\x6F\x77\x01\x02\x02\x01", - .rlen = 72, - .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" - "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" - "\x69\x76\x65\x63", - .alen = 20, - .input = "\xD4\xB7\xED\x86\xA1\x77\x7F\x2E" - "\xA1\x3D\x69\x73\xD3\x24\xC6\x9E" - "\x7B\x43\xF8\x26\xFB\x56\x83\x12" - "\x26\x50\x8B\xEB\xD2\xDC\xEB\x18" - "\xD0\xA6\xDF\x10\xE5\x48\x7D\xF0" - "\x74\x11\x3E\x14\xC6\x41\x02\x4E" - "\x3E\x67\x73\xD9\x1A\x62\xEE\x42" - "\x9B\x04\x3A\x10\xE3\xEF\xE6\xB0" - "\x12\xA4\x93\x63\x41\x23\x64\xF8" - "\xC0\xCA\xC5\x87\xF2\x49\xE5\x6B" - "\x11\xE2\x4F\x30\xE4\x4C\xCC\x76", - .ilen = 88, - }, { - .key = "\x7D\x77\x3D\x00\xC1\x44\xC5\x25" - "\xAC\x61\x9D\x18\xC8\x4A\x3F\x47" - "\xD9\x66\x42\x67", - .klen = 20, - .iv = "\x43\x45\x7E\x91\x82\x44\x3B\xC6", - .result = "\x01\x02\x02\x01", - .rlen = 4, - .assoc = "\x33\x54\x67\xAE\xFF\xFF\xFF\xFF" - "\x43\x45\x7E\x91\x82\x44\x3B\xC6", - .alen = 16, - .input = "\x43\x7F\x86\x6B\xCB\x3F\x69\x9F" - "\xE9\xB0\x82\x2B\xAC\x96\x1C\x45" - "\x04\xBE\xF2\x70", - .ilen = 20, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xDE\xCA\xF8\x88", - .klen = 20, - .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", - .result = "\x74\x6F\x01\x62\x65\x01\x6F\x72" - "\x01\x6E\x6F\x74\x01\x74\x6F\x01" - "\x62\x65\x00\x01", - .rlen = 20, - .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" - "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" - "\xCE\xFA\xCE\x74", - .alen = 20, - .input = "\x29\xC9\xFC\x69\xA1\x97\xD0\x38" - "\xCC\xDD\x14\xE2\xDD\xFC\xAA\x05" - "\x43\x33\x21\x64\x41\x25\x03\x52" - "\x43\x03\xED\x3C\x6C\x5F\x28\x38" - "\x43\xAF\x8C\x3E", - .ilen = 36, - }, { - .key = "\x6C\x65\x67\x61\x6C\x69\x7A\x65" - "\x6D\x61\x72\x69\x6A\x75\x61\x6E" - "\x61\x61\x6E\x64\x64\x6F\x69\x74" - "\x62\x65\x66\x6F\x72\x65\x69\x61" - "\x74\x75\x72\x6E", - .klen = 36, - .iv = "\x33\x30\x21\x69\x67\x65\x74\x6D", - .result = "\x45\x00\x00\x30\xDA\x3A\x00\x00" - "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" - "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01", - .rlen = 52, - .assoc = "\x79\x6B\x69\x63\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\x33\x30\x21\x69" - "\x67\x65\x74\x6D", - .alen = 20, - .input = "\xF9\x7A\xB2\xAA\x35\x6D\x8E\xDC" - "\xE1\x76\x44\xAC\x8C\x78\xE2\x5D" - "\xD2\x4D\xED\xBB\x29\xEB\xF1\xB6" - "\x4A\x27\x4B\x39\xB4\x9C\x3A\x86" - "\x4C\xD3\xD7\x8C\xA4\xAE\x68\xA3" - "\x2B\x42\x45\x8F\xB5\x7D\xBE\x82" - "\x1D\xCC\x63\xB9\xD0\x93\x7B\xA2" - "\x94\x5F\x66\x93\x68\x66\x1A\x32" - "\x9F\xB4\xC0\x53", - .ilen = 68, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E\x43", - .klen = 20, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .result = "\x45\x00\x00\x30\xDA\x3A\x00\x00" - "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" - "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01", - .rlen = 52, - .assoc = "\x3F\x7E\xF6\x42\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .input = "\xFB\xA2\xCA\xA8\xC6\xC5\xF9\xF0" - "\xF2\x2C\xA5\x4A\x06\x12\x10\xAD" - "\x3F\x6E\x57\x91\xCF\x1A\xCA\x21" - "\x0D\x11\x7C\xEC\x9C\x35\x79\x17" - "\x65\xAC\xBD\x87\x01\xAD\x79\x84" - "\x5B\xF9\xFE\x3F\xBA\x48\x7B\xC9" - "\x63\x21\x93\x06\x84\xEE\xCA\xDB" - "\x56\x91\x25\x46\xE7\xA9\x5C\x97" - "\x40\xD7\xCB\x05", - .ilen = 68, - }, { - .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" - "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" - "\x22\x43\x3C\x64", - .klen = 20, - .iv = "\x48\x55\xEC\x7D\x3A\x23\x4B\xFD", - .result = "\x08\x00\xC6\xCD\x02\x00\x07\x00" - "\x61\x62\x63\x64\x65\x66\x67\x68" - "\x69\x6A\x6B\x6C\x6D\x6E\x6F\x70" - "\x71\x72\x73\x74\x01\x02\x02\x01", - .rlen = 32, - .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" - "\x00\x00\x00\x07\x48\x55\xEC\x7D" - "\x3A\x23\x4B\xFD", - .alen = 20, - .input = "\x74\x75\x2E\x8A\xEB\x5D\x87\x3C" - "\xD7\xC0\xF4\xAC\xC3\x6C\x4B\xFF" - "\x84\xB7\xD7\xB9\x8F\x0C\xA8\xB6" - "\xAC\xDA\x68\x94\xBC\x61\x90\x69" - "\xEF\x9C\xBC\x28\xFE\x1B\x56\xA7" - "\xC4\xE0\xD5\x8C\x86\xCD\x2B\xC0", - .ilen = 48, - } -}; - -static const struct aead_testvec aes_gcm_rfc4543_enc_tv_template[] = { - { /* From draft-mcgrew-gcm-test-01 */ - .key = "\x4c\x80\xcd\xef\xbb\x5d\x10\xda" - "\x90\x6a\xc7\x3c\x36\x13\xa6\x34" - "\x22\x43\x3c\x64", - .klen = 20, - .iv = zeroed_string, - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x07" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .alen = 16, - .input = "\x45\x00\x00\x30\xda\x3a\x00\x00" - "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" - "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6a\x6b\x6c" - "\x6d\x6e\x6f\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01", - .ilen = 52, - .result = "\x45\x00\x00\x30\xda\x3a\x00\x00" - "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" - "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6a\x6b\x6c" - "\x6d\x6e\x6f\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01\xf2\xa9\xa8\x36" - "\xe1\x55\x10\x6a\xa8\xdc\xd6\x18" - "\xe4\x09\x9a\xaa", - .rlen = 68, - } -}; - -static const struct aead_testvec aes_gcm_rfc4543_dec_tv_template[] = { - { /* From draft-mcgrew-gcm-test-01 */ - .key = "\x4c\x80\xcd\xef\xbb\x5d\x10\xda" - "\x90\x6a\xc7\x3c\x36\x13\xa6\x34" - "\x22\x43\x3c\x64", - .klen = 20, - .iv = zeroed_string, - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x07" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .alen = 16, - .input = "\x45\x00\x00\x30\xda\x3a\x00\x00" - "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" - "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6a\x6b\x6c" - "\x6d\x6e\x6f\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01\xf2\xa9\xa8\x36" - "\xe1\x55\x10\x6a\xa8\xdc\xd6\x18" - "\xe4\x09\x9a\xaa", - .ilen = 68, - .result = "\x45\x00\x00\x30\xda\x3a\x00\x00" - "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" - "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6a\x6b\x6c" - "\x6d\x6e\x6f\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01", - .rlen = 52, - }, { /* nearly same as previous, but should fail */ - .key = "\x4c\x80\xcd\xef\xbb\x5d\x10\xda" - "\x90\x6a\xc7\x3c\x36\x13\xa6\x34" - "\x22\x43\x3c\x64", - .klen = 20, - .iv = zeroed_string, - .assoc = "\x00\x00\x43\x21\x00\x00\x00\x07" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .alen = 16, - .input = "\x45\x00\x00\x30\xda\x3a\x00\x00" - "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" - "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6a\x6b\x6c" - "\x6d\x6e\x6f\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01\xf2\xa9\xa8\x36" - "\xe1\x55\x10\x6a\xa8\xdc\xd6\x18" - "\x00\x00\x00\x00", - .ilen = 68, - .novrfy = 1, - .result = "\x45\x00\x00\x30\xda\x3a\x00\x00" - "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" - "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6a\x6b\x6c" - "\x6d\x6e\x6f\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01", - .rlen = 52, - }, -}; - -static const struct aead_testvec aes_ccm_enc_tv_template[] = { - { /* From RFC 3610 */ - .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", - .klen = 16, - .iv = "\x01\x00\x00\x00\x03\x02\x01\x00" - "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07", - .alen = 8, - .input = "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e", - .ilen = 23, - .result = "\x58\x8c\x97\x9a\x61\xc6\x63\xd2" - "\xf0\x66\xd0\xc2\xc0\xf9\x89\x80" - "\x6d\x5f\x6b\x61\xda\xc3\x84\x17" - "\xe8\xd1\x2c\xfd\xf9\x26\xe0", - .rlen = 31, - }, { - .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", - .klen = 16, - .iv = "\x01\x00\x00\x00\x07\x06\x05\x04" - "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b", - .alen = 12, - .input = "\x0c\x0d\x0e\x0f\x10\x11\x12\x13" - "\x14\x15\x16\x17\x18\x19\x1a\x1b" - "\x1c\x1d\x1e\x1f", - .ilen = 20, - .result = "\xdc\xf1\xfb\x7b\x5d\x9e\x23\xfb" - "\x9d\x4e\x13\x12\x53\x65\x8a\xd8" - "\x6e\xbd\xca\x3e\x51\xe8\x3f\x07" - "\x7d\x9c\x2d\x93", - .rlen = 28, - }, { - .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", - .klen = 16, - .iv = "\x01\x00\x00\x00\x0b\x0a\x09\x08" - "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07", - .alen = 8, - .input = "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20", - .ilen = 25, - .result = "\x82\x53\x1a\x60\xcc\x24\x94\x5a" - "\x4b\x82\x79\x18\x1a\xb5\xc8\x4d" - "\xf2\x1c\xe7\xf9\xb7\x3f\x42\xe1" - "\x97\xea\x9c\x07\xe5\x6b\x5e\xb1" - "\x7e\x5f\x4e", - .rlen = 35, - }, { - .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", - .klen = 16, - .iv = "\x01\x00\x00\x00\x0c\x0b\x0a\x09" - "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b", - .alen = 12, - .input = "\x0c\x0d\x0e\x0f\x10\x11\x12\x13" - "\x14\x15\x16\x17\x18\x19\x1a\x1b" - "\x1c\x1d\x1e", - .ilen = 19, - .result = "\x07\x34\x25\x94\x15\x77\x85\x15" - "\x2b\x07\x40\x98\x33\x0a\xbb\x14" - "\x1b\x94\x7b\x56\x6a\xa9\x40\x6b" - "\x4d\x99\x99\x88\xdd", - .rlen = 29, - }, { - .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" - "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", - .klen = 16, - .iv = "\x01\x00\x33\x56\x8e\xf7\xb2\x63" - "\x3c\x96\x96\x76\x6c\xfa\x00\x00", - .assoc = "\x63\x01\x8f\x76\xdc\x8a\x1b\xcb", - .alen = 8, - .input = "\x90\x20\xea\x6f\x91\xbd\xd8\x5a" - "\xfa\x00\x39\xba\x4b\xaf\xf9\xbf" - "\xb7\x9c\x70\x28\x94\x9c\xd0\xec", - .ilen = 24, - .result = "\x4c\xcb\x1e\x7c\xa9\x81\xbe\xfa" - "\xa0\x72\x6c\x55\xd3\x78\x06\x12" - "\x98\xc8\x5c\x92\x81\x4a\xbc\x33" - "\xc5\x2e\xe8\x1d\x7d\x77\xc0\x8a", - .rlen = 32, - }, { - .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" - "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", - .klen = 16, - .iv = "\x01\x00\xd5\x60\x91\x2d\x3f\x70" - "\x3c\x96\x96\x76\x6c\xfa\x00\x00", - .assoc = "\xcd\x90\x44\xd2\xb7\x1f\xdb\x81" - "\x20\xea\x60\xc0", - .alen = 12, - .input = "\x64\x35\xac\xba\xfb\x11\xa8\x2e" - "\x2f\x07\x1d\x7c\xa4\xa5\xeb\xd9" - "\x3a\x80\x3b\xa8\x7f", - .ilen = 21, - .result = "\x00\x97\x69\xec\xab\xdf\x48\x62" - "\x55\x94\xc5\x92\x51\xe6\x03\x57" - "\x22\x67\x5e\x04\xc8\x47\x09\x9e" - "\x5a\xe0\x70\x45\x51", - .rlen = 29, - }, { - .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" - "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", - .klen = 16, - .iv = "\x01\x00\x42\xff\xf8\xf1\x95\x1c" - "\x3c\x96\x96\x76\x6c\xfa\x00\x00", - .assoc = "\xd8\x5b\xc7\xe6\x9f\x94\x4f\xb8", - .alen = 8, - .input = "\x8a\x19\xb9\x50\xbc\xf7\x1a\x01" - "\x8e\x5e\x67\x01\xc9\x17\x87\x65" - "\x98\x09\xd6\x7d\xbe\xdd\x18", - .ilen = 23, - .result = "\xbc\x21\x8d\xaa\x94\x74\x27\xb6" - "\xdb\x38\x6a\x99\xac\x1a\xef\x23" - "\xad\xe0\xb5\x29\x39\xcb\x6a\x63" - "\x7c\xf9\xbe\xc2\x40\x88\x97\xc6" - "\xba", - .rlen = 33, - }, { - /* This is taken from FIPS CAVS. */ - .key = "\x83\xac\x54\x66\xc2\xeb\xe5\x05" - "\x2e\x01\xd1\xfc\x5d\x82\x66\x2e", - .klen = 16, - .iv = "\x03\x96\xac\x59\x30\x07\xa1\xe2\xa2\xc7\x55\x24\0\0\0\0", - .alen = 0, - .input = "\x19\xc8\x81\xf6\xe9\x86\xff\x93" - "\x0b\x78\x67\xe5\xbb\xb7\xfc\x6e" - "\x83\x77\xb3\xa6\x0c\x8c\x9f\x9c" - "\x35\x2e\xad\xe0\x62\xf9\x91\xa1", - .ilen = 32, - .result = "\xab\x6f\xe1\x69\x1d\x19\x99\xa8" - "\x92\xa0\xc4\x6f\x7e\xe2\x8b\xb1" - "\x70\xbb\x8c\xa6\x4c\x6e\x97\x8a" - "\x57\x2b\xbe\x5d\x98\xa6\xb1\x32" - "\xda\x24\xea\xd9\xa1\x39\x98\xfd" - "\xa4\xbe\xd9\xf2\x1a\x6d\x22\xa8", - .rlen = 48, - }, { - .key = "\x1e\x2c\x7e\x01\x41\x9a\xef\xc0" - "\x0d\x58\x96\x6e\x5c\xa2\x4b\xd3", - .klen = 16, - .iv = "\x03\x4f\xa3\x19\xd3\x01\x5a\xd8" - "\x30\x60\x15\x56\x00\x00\x00\x00", - .assoc = "\xda\xe6\x28\x9c\x45\x2d\xfd\x63" - "\x5e\xda\x4c\xb6\xe6\xfc\xf9\xb7" - "\x0c\x56\xcb\xe4\xe0\x05\x7a\xe1" - "\x0a\x63\x09\x78\xbc\x2c\x55\xde", - .alen = 32, - .input = "\x87\xa3\x36\xfd\x96\xb3\x93\x78" - "\xa9\x28\x63\xba\x12\xa3\x14\x85" - "\x57\x1e\x06\xc9\x7b\x21\xef\x76" - "\x7f\x38\x7e\x8e\x29\xa4\x3e\x7e", - .ilen = 32, - .result = "\x8a\x1e\x11\xf0\x02\x6b\xe2\x19" - "\xfc\x70\xc4\x6d\x8e\xb7\x99\xab" - "\xc5\x4b\xa2\xac\xd3\xf3\x48\xff" - "\x3b\xb5\xce\x53\xef\xde\xbb\x02" - "\xa9\x86\x15\x6c\x13\xfe\xda\x0a" - "\x22\xb8\x29\x3d\xd8\x39\x9a\x23", - .rlen = 48, - }, { - .key = "\xf4\x6b\xc2\x75\x62\xfe\xb4\xe1" - "\xa3\xf0\xff\xdd\x4e\x4b\x12\x75" - "\x53\x14\x73\x66\x8d\x88\xf6\x80", - .klen = 24, - .iv = "\x03\xa0\x20\x35\x26\xf2\x21\x8d" - "\x50\x20\xda\xe2\x00\x00\x00\x00", - .assoc = "\x5b\x9e\x13\x67\x02\x5e\xef\xc1" - "\x6c\xf9\xd7\x1e\x52\x8f\x7a\x47" - "\xe9\xd4\xcf\x20\x14\x6e\xf0\x2d" - "\xd8\x9e\x2b\x56\x10\x23\x56\xe7", - .alen = 32, - .result = "\x36\xea\x7a\x70\x08\xdc\x6a\xbc" - "\xad\x0c\x7a\x63\xf6\x61\xfd\x9b", - .rlen = 16, - }, { - .key = "\x56\xdf\x5c\x8f\x26\x3f\x0e\x42" - "\xef\x7a\xd3\xce\xfc\x84\x60\x62" - "\xca\xb4\x40\xaf\x5f\xc9\xc9\x01", - .klen = 24, - .iv = "\x03\xd6\x3c\x8c\x86\x84\xb6\xcd" - "\xef\x09\x2e\x94\x00\x00\x00\x00", - .assoc = "\x02\x65\x78\x3c\xe9\x21\x30\x91" - "\xb1\xb9\xda\x76\x9a\x78\x6d\x95" - "\xf2\x88\x32\xa3\xf2\x50\xcb\x4c" - "\xe3\x00\x73\x69\x84\x69\x87\x79", - .alen = 32, - .input = "\x9f\xd2\x02\x4b\x52\x49\x31\x3c" - "\x43\x69\x3a\x2d\x8e\x70\xad\x7e" - "\xe0\xe5\x46\x09\x80\x89\x13\xb2" - "\x8c\x8b\xd9\x3f\x86\xfb\xb5\x6b", - .ilen = 32, - .result = "\x39\xdf\x7c\x3c\x5a\x29\xb9\x62" - "\x5d\x51\xc2\x16\xd8\xbd\x06\x9f" - "\x9b\x6a\x09\x70\xc1\x51\x83\xc2" - "\x66\x88\x1d\x4f\x9a\xda\xe0\x1e" - "\xc7\x79\x11\x58\xe5\x6b\x20\x40" - "\x7a\xea\x46\x42\x8b\xe4\x6f\xe1", - .rlen = 48, - }, { - .key = "\xe0\x8d\x99\x71\x60\xd7\x97\x1a" - "\xbd\x01\x99\xd5\x8a\xdf\x71\x3a" - "\xd3\xdf\x24\x4b\x5e\x3d\x4b\x4e" - "\x30\x7a\xb9\xd8\x53\x0a\x5e\x2b", - .klen = 32, - .iv = "\x03\x1e\x29\x91\xad\x8e\xc1\x53" - "\x0a\xcf\x2d\xbe\x00\x00\x00\x00", - .assoc = "\x19\xb6\x1f\x57\xc4\xf3\xf0\x8b" - "\x78\x2b\x94\x02\x29\x0f\x42\x27" - "\x6b\x75\xcb\x98\x34\x08\x7e\x79" - "\xe4\x3e\x49\x0d\x84\x8b\x22\x87", - .alen = 32, - .input = "\xe1\xd9\xd8\x13\xeb\x3a\x75\x3f" - "\x9d\xbd\x5f\x66\xbe\xdc\xbb\x66" - "\xbf\x17\x99\x62\x4a\x39\x27\x1f" - "\x1d\xdc\x24\xae\x19\x2f\x98\x4c", - .ilen = 32, - .result = "\x19\xb8\x61\x33\x45\x2b\x43\x96" - "\x6f\x51\xd0\x20\x30\x7d\x9b\xc6" - "\x26\x3d\xf8\xc9\x65\x16\xa8\x9f" - "\xf0\x62\x17\x34\xf2\x1e\x8d\x75" - "\x4e\x13\xcc\xc0\xc3\x2a\x54\x2d", - .rlen = 40, - }, { - .key = "\x7c\xc8\x18\x3b\x8d\x99\xe0\x7c" - "\x45\x41\xb8\xbd\x5c\xa7\xc2\x32" - "\x8a\xb8\x02\x59\xa4\xfe\xa9\x2c" - "\x09\x75\x9a\x9b\x3c\x9b\x27\x39", - .klen = 32, - .iv = "\x03\xf9\xd9\x4e\x63\xb5\x3d\x9d" - "\x43\xf6\x1e\x50\0\0\0\0", - .assoc = "\x57\xf5\x6b\x8b\x57\x5c\x3d\x3b" - "\x13\x02\x01\x0c\x83\x4c\x96\x35" - "\x8e\xd6\x39\xcf\x7d\x14\x9b\x94" - "\xb0\x39\x36\xe6\x8f\x57\xe0\x13", - .alen = 32, - .input = "\x3b\x6c\x29\x36\xb6\xef\x07\xa6" - "\x83\x72\x07\x4f\xcf\xfa\x66\x89" - "\x5f\xca\xb1\xba\xd5\x8f\x2c\x27" - "\x30\xdb\x75\x09\x93\xd4\x65\xe4", - .ilen = 32, - .result = "\xb0\x88\x5a\x33\xaa\xe5\xc7\x1d" - "\x85\x23\xc7\xc6\x2f\xf4\x1e\x3d" - "\xcc\x63\x44\x25\x07\x78\x4f\x9e" - "\x96\xb8\x88\xeb\xbc\x48\x1f\x06" - "\x39\xaf\x39\xac\xd8\x4a\x80\x39" - "\x7b\x72\x8a\xf7", - .rlen = 44, - }, { - .key = "\xab\xd0\xe9\x33\x07\x26\xe5\x83" - "\x8c\x76\x95\xd4\xb6\xdc\xf3\x46" - "\xf9\x8f\xad\xe3\x02\x13\x83\x77" - "\x3f\xb0\xf1\xa1\xa1\x22\x0f\x2b", - .klen = 32, - .iv = "\x03\x24\xa7\x8b\x07\xcb\xcc\x0e" - "\xe6\x33\xbf\xf5\x00\x00\x00\x00", - .assoc = "\xd4\xdb\x30\x1d\x03\xfe\xfd\x5f" - "\x87\xd4\x8c\xb6\xb6\xf1\x7a\x5d" - "\xab\x90\x65\x8d\x8e\xca\x4d\x4f" - "\x16\x0c\x40\x90\x4b\xc7\x36\x73", - .alen = 32, - .input = "\xf5\xc6\x7d\x48\xc1\xb7\xe6\x92" - "\x97\x5a\xca\xc4\xa9\x6d\xf9\x3d" - "\x6c\xde\xbc\xf1\x90\xea\x6a\xb2" - "\x35\x86\x36\xaf\x5c\xfe\x4b\x3a", - .ilen = 32, - .result = "\x83\x6f\x40\x87\x72\xcf\xc1\x13" - "\xef\xbb\x80\x21\x04\x6c\x58\x09" - "\x07\x1b\xfc\xdf\xc0\x3f\x5b\xc7" - "\xe0\x79\xa8\x6e\x71\x7c\x3f\xcf" - "\x5c\xda\xb2\x33\xe5\x13\xe2\x0d" - "\x74\xd1\xef\xb5\x0f\x3a\xb5\xf8", - .rlen = 48, - } -}; - -static const struct aead_testvec aes_ccm_dec_tv_template[] = { - { /* From RFC 3610 */ - .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", - .klen = 16, - .iv = "\x01\x00\x00\x00\x03\x02\x01\x00" - "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07", - .alen = 8, - .input = "\x58\x8c\x97\x9a\x61\xc6\x63\xd2" - "\xf0\x66\xd0\xc2\xc0\xf9\x89\x80" - "\x6d\x5f\x6b\x61\xda\xc3\x84\x17" - "\xe8\xd1\x2c\xfd\xf9\x26\xe0", - .ilen = 31, - .result = "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e", - .rlen = 23, - }, { - .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", - .klen = 16, - .iv = "\x01\x00\x00\x00\x07\x06\x05\x04" - "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b", - .alen = 12, - .input = "\xdc\xf1\xfb\x7b\x5d\x9e\x23\xfb" - "\x9d\x4e\x13\x12\x53\x65\x8a\xd8" - "\x6e\xbd\xca\x3e\x51\xe8\x3f\x07" - "\x7d\x9c\x2d\x93", - .ilen = 28, - .result = "\x0c\x0d\x0e\x0f\x10\x11\x12\x13" - "\x14\x15\x16\x17\x18\x19\x1a\x1b" - "\x1c\x1d\x1e\x1f", - .rlen = 20, - }, { - .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", - .klen = 16, - .iv = "\x01\x00\x00\x00\x0b\x0a\x09\x08" - "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07", - .alen = 8, - .input = "\x82\x53\x1a\x60\xcc\x24\x94\x5a" - "\x4b\x82\x79\x18\x1a\xb5\xc8\x4d" - "\xf2\x1c\xe7\xf9\xb7\x3f\x42\xe1" - "\x97\xea\x9c\x07\xe5\x6b\x5e\xb1" - "\x7e\x5f\x4e", - .ilen = 35, - .result = "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20", - .rlen = 25, - }, { - .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", - .klen = 16, - .iv = "\x01\x00\x00\x00\x0c\x0b\x0a\x09" - "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", - .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b", - .alen = 12, - .input = "\x07\x34\x25\x94\x15\x77\x85\x15" - "\x2b\x07\x40\x98\x33\x0a\xbb\x14" - "\x1b\x94\x7b\x56\x6a\xa9\x40\x6b" - "\x4d\x99\x99\x88\xdd", - .ilen = 29, - .result = "\x0c\x0d\x0e\x0f\x10\x11\x12\x13" - "\x14\x15\x16\x17\x18\x19\x1a\x1b" - "\x1c\x1d\x1e", - .rlen = 19, - }, { - .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" - "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", - .klen = 16, - .iv = "\x01\x00\x33\x56\x8e\xf7\xb2\x63" - "\x3c\x96\x96\x76\x6c\xfa\x00\x00", - .assoc = "\x63\x01\x8f\x76\xdc\x8a\x1b\xcb", - .alen = 8, - .input = "\x4c\xcb\x1e\x7c\xa9\x81\xbe\xfa" - "\xa0\x72\x6c\x55\xd3\x78\x06\x12" - "\x98\xc8\x5c\x92\x81\x4a\xbc\x33" - "\xc5\x2e\xe8\x1d\x7d\x77\xc0\x8a", - .ilen = 32, - .result = "\x90\x20\xea\x6f\x91\xbd\xd8\x5a" - "\xfa\x00\x39\xba\x4b\xaf\xf9\xbf" - "\xb7\x9c\x70\x28\x94\x9c\xd0\xec", - .rlen = 24, - }, { - .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" - "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", - .klen = 16, - .iv = "\x01\x00\xd5\x60\x91\x2d\x3f\x70" - "\x3c\x96\x96\x76\x6c\xfa\x00\x00", - .assoc = "\xcd\x90\x44\xd2\xb7\x1f\xdb\x81" - "\x20\xea\x60\xc0", - .alen = 12, - .input = "\x00\x97\x69\xec\xab\xdf\x48\x62" - "\x55\x94\xc5\x92\x51\xe6\x03\x57" - "\x22\x67\x5e\x04\xc8\x47\x09\x9e" - "\x5a\xe0\x70\x45\x51", - .ilen = 29, - .result = "\x64\x35\xac\xba\xfb\x11\xa8\x2e" - "\x2f\x07\x1d\x7c\xa4\xa5\xeb\xd9" - "\x3a\x80\x3b\xa8\x7f", - .rlen = 21, - }, { - .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" - "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", - .klen = 16, - .iv = "\x01\x00\x42\xff\xf8\xf1\x95\x1c" - "\x3c\x96\x96\x76\x6c\xfa\x00\x00", - .assoc = "\xd8\x5b\xc7\xe6\x9f\x94\x4f\xb8", - .alen = 8, - .input = "\xbc\x21\x8d\xaa\x94\x74\x27\xb6" - "\xdb\x38\x6a\x99\xac\x1a\xef\x23" - "\xad\xe0\xb5\x29\x39\xcb\x6a\x63" - "\x7c\xf9\xbe\xc2\x40\x88\x97\xc6" - "\xba", - .ilen = 33, - .result = "\x8a\x19\xb9\x50\xbc\xf7\x1a\x01" - "\x8e\x5e\x67\x01\xc9\x17\x87\x65" - "\x98\x09\xd6\x7d\xbe\xdd\x18", - .rlen = 23, - }, { - /* This is taken from FIPS CAVS. */ - .key = "\xab\x2f\x8a\x74\xb7\x1c\xd2\xb1" - "\xff\x80\x2e\x48\x7d\x82\xf8\xb9", - .klen = 16, - .iv = "\x03\xc6\xfb\x7d\x80\x0d\x13\xab" - "\xd8\xa6\xb2\xd8\x00\x00\x00\x00", - .alen = 0, - .input = "\xd5\xe8\x93\x9f\xc7\x89\x2e\x2b", - .ilen = 8, - .result = "\x00", - .rlen = 0, - .novrfy = 1, - }, { - .key = "\xab\x2f\x8a\x74\xb7\x1c\xd2\xb1" - "\xff\x80\x2e\x48\x7d\x82\xf8\xb9", - .klen = 16, - .iv = "\x03\xaf\x94\x87\x78\x35\x82\x81" - "\x7f\x88\x94\x68\x00\x00\x00\x00", - .alen = 0, - .input = "\x41\x3c\xb8\x87\x73\xcb\xf3\xf3", - .ilen = 8, - .result = "\x00", - .rlen = 0, - }, { - .key = "\x61\x0e\x8c\xae\xe3\x23\xb6\x38" - "\x76\x1c\xf6\x3a\x67\xa3\x9c\xd8", - .klen = 16, - .iv = "\x03\xc6\xfb\x7d\x80\x0d\x13\xab" - "\xd8\xa6\xb2\xd8\x00\x00\x00\x00", - .assoc = "\xf3\x94\x87\x78\x35\x82\x81\x7f" - "\x88\x94\x68\xb1\x78\x6b\x2b\xd6" - "\x04\x1f\x4e\xed\x78\xd5\x33\x66" - "\xd8\x94\x99\x91\x81\x54\x62\x57", - .alen = 32, - .input = "\xf0\x7c\x29\x02\xae\x1c\x2f\x55" - "\xd0\xd1\x3d\x1a\xa3\x6d\xe4\x0a" - "\x86\xb0\x87\x6b\x62\x33\x8c\x34" - "\xce\xab\x57\xcc\x79\x0b\xe0\x6f" - "\x5c\x3e\x48\x1f\x6c\x46\xf7\x51" - "\x8b\x84\x83\x2a\xc1\x05\xb8\xc5", - .ilen = 48, - .result = "\x50\x82\x3e\x07\xe2\x1e\xb6\xfb" - "\x33\xe4\x73\xce\xd2\xfb\x95\x79" - "\xe8\xb4\xb5\x77\x11\x10\x62\x6f" - "\x6a\x82\xd1\x13\xec\xf5\xd0\x48", - .rlen = 32, - .novrfy = 1, - }, { - .key = "\x61\x0e\x8c\xae\xe3\x23\xb6\x38" - "\x76\x1c\xf6\x3a\x67\xa3\x9c\xd8", - .klen = 16, - .iv = "\x03\x05\xe0\xc9\x0f\xed\x34\xea" - "\x97\xd4\x3b\xdf\x00\x00\x00\x00", - .assoc = "\x49\x5c\x50\x1f\x1d\x94\xcc\x81" - "\xba\xb7\xb6\x03\xaf\xa5\xc1\xa1" - "\xd8\x5c\x42\x68\xe0\x6c\xda\x89" - "\x05\xac\x56\xac\x1b\x2a\xd3\x86", - .alen = 32, - .input = "\x39\xbe\x7d\x15\x62\x77\xf3\x3c" - "\xad\x83\x52\x6d\x71\x03\x25\x1c" - "\xed\x81\x3a\x9a\x16\x7d\x19\x80" - "\x72\x04\x72\xd0\xf6\xff\x05\x0f" - "\xb7\x14\x30\x00\x32\x9e\xa0\xa6" - "\x9e\x5a\x18\xa1\xb8\xfe\xdb\xd3", - .ilen = 48, - .result = "\x75\x05\xbe\xc2\xd9\x1e\xde\x60" - "\x47\x3d\x8c\x7d\xbd\xb5\xd9\xb7" - "\xf2\xae\x61\x05\x8f\x82\x24\x3f" - "\x9c\x67\x91\xe1\x38\x4f\xe4\x0c", - .rlen = 32, - }, { - .key = "\x39\xbb\xa7\xbe\x59\x97\x9e\x73" - "\xa2\xbc\x6b\x98\xd7\x75\x7f\xe3" - "\xa4\x48\x93\x39\x26\x71\x4a\xc6", - .klen = 24, - .iv = "\x03\xee\x49\x83\xe9\xa9\xff\xe9" - "\x57\xba\xfd\x9e\x00\x00\x00\x00", - .assoc = "\x44\xa6\x2c\x05\xe9\xe1\x43\xb1" - "\x58\x7c\xf2\x5c\x6d\x39\x0a\x64" - "\xa4\xf0\x13\x05\xd1\x77\x99\x67" - "\x11\xc4\xc6\xdb\x00\x56\x36\x61", - .alen = 32, - .input = "\x71\x99\xfa\xf4\x44\x12\x68\x9b", - .ilen = 8, - .result = "\x00", - .rlen = 0, - }, { - .key = "\x58\x5d\xa0\x96\x65\x1a\x04\xd7" - "\x96\xe5\xc5\x68\xaa\x95\x35\xe0" - "\x29\xa0\xba\x9e\x48\x78\xd1\xba", - .klen = 24, - .iv = "\x03\xee\x49\x83\xe9\xa9\xff\xe9" - "\x57\xba\xfd\x9e\x00\x00\x00\x00", - .assoc = "\x44\xa6\x2c\x05\xe9\xe1\x43\xb1" - "\x58\x7c\xf2\x5c\x6d\x39\x0a\x64" - "\xa4\xf0\x13\x05\xd1\x77\x99\x67" - "\x11\xc4\xc6\xdb\x00\x56\x36\x61", - .alen = 32, - .input = "\xfb\xe5\x5d\x34\xbe\xe5\xe8\xe7" - "\x5a\xef\x2f\xbf\x1f\x7f\xd4\xb2" - "\x66\xca\x61\x1e\x96\x7a\x61\xb3" - "\x1c\x16\x45\x52\xba\x04\x9c\x9f" - "\xb1\xd2\x40\xbc\x52\x7c\x6f\xb1", - .ilen = 40, - .result = "\x85\x34\x66\x42\xc8\x92\x0f\x36" - "\x58\xe0\x6b\x91\x3c\x98\x5c\xbb" - "\x0a\x85\xcc\x02\xad\x7a\x96\xe9" - "\x65\x43\xa4\xc3\x0f\xdc\x55\x81", - .rlen = 32, - }, { - .key = "\x58\x5d\xa0\x96\x65\x1a\x04\xd7" - "\x96\xe5\xc5\x68\xaa\x95\x35\xe0" - "\x29\xa0\xba\x9e\x48\x78\xd1\xba", - .klen = 24, - .iv = "\x03\xd1\xfc\x57\x9c\xfe\xb8\x9c" - "\xad\x71\xaa\x1f\x00\x00\x00\x00", - .assoc = "\x86\x67\xa5\xa9\x14\x5f\x0d\xc6" - "\xff\x14\xc7\x44\xbf\x6c\x3a\xc3" - "\xff\xb6\x81\xbd\xe2\xd5\x06\xc7" - "\x3c\xa1\x52\x13\x03\x8a\x23\x3a", - .alen = 32, - .input = "\x3f\x66\xb0\x9d\xe5\x4b\x38\x00" - "\xc6\x0e\x6e\xe5\xd6\x98\xa6\x37" - "\x8c\x26\x33\xc6\xb2\xa2\x17\xfa" - "\x64\x19\xc0\x30\xd7\xfc\x14\x6b" - "\xe3\x33\xc2\x04\xb0\x37\xbe\x3f" - "\xa9\xb4\x2d\x68\x03\xa3\x44\xef", - .ilen = 48, - .result = "\x02\x87\x4d\x28\x80\x6e\xb2\xed" - "\x99\x2a\xa8\xca\x04\x25\x45\x90" - "\x1d\xdd\x5a\xd9\xe4\xdb\x9c\x9c" - "\x49\xe9\x01\xfe\xa7\x80\x6d\x6b", - .rlen = 32, - .novrfy = 1, - }, { - .key = "\xa4\x4b\x54\x29\x0a\xb8\x6d\x01" - "\x5b\x80\x2a\xcf\x25\xc4\xb7\x5c" - "\x20\x2c\xad\x30\xc2\x2b\x41\xfb" - "\x0e\x85\xbc\x33\xad\x0f\x2b\xff", - .klen = 32, - .iv = "\x03\xee\x49\x83\xe9\xa9\xff\xe9" - "\x57\xba\xfd\x9e\x00\x00\x00\x00", - .alen = 0, - .input = "\x1f\xb8\x8f\xa3\xdd\x54\x00\xf2", - .ilen = 8, - .result = "\x00", - .rlen = 0, - }, { - .key = "\x39\xbb\xa7\xbe\x59\x97\x9e\x73" - "\xa2\xbc\x6b\x98\xd7\x75\x7f\xe3" - "\xa4\x48\x93\x39\x26\x71\x4a\xc6" - "\xae\x8f\x11\x4c\xc2\x9c\x4a\xbb", - .klen = 32, - .iv = "\x03\x85\x34\x66\x42\xc8\x92\x0f" - "\x36\x58\xe0\x6b\x00\x00\x00\x00", - .alen = 0, - .input = "\x48\x01\x5e\x02\x24\x04\x66\x47" - "\xa1\xea\x6f\xaf\xe8\xfc\xfb\xdd" - "\xa5\xa9\x87\x8d\x84\xee\x2e\x77" - "\xbb\x86\xb9\xf5\x5c\x6c\xff\xf6" - "\x72\xc3\x8e\xf7\x70\xb1\xb2\x07" - "\xbc\xa8\xa3\xbd\x83\x7c\x1d\x2a", - .ilen = 48, - .result = "\xdc\x56\xf2\x71\xb0\xb1\xa0\x6c" - "\xf0\x97\x3a\xfb\x6d\xe7\x32\x99" - "\x3e\xaf\x70\x5e\xb2\x4d\xea\x39" - "\x89\xd4\x75\x7a\x63\xb1\xda\x93", - .rlen = 32, - .novrfy = 1, - }, { - .key = "\x58\x5d\xa0\x96\x65\x1a\x04\xd7" - "\x96\xe5\xc5\x68\xaa\x95\x35\xe0" - "\x29\xa0\xba\x9e\x48\x78\xd1\xba" - "\x0d\x1a\x53\x3b\xb5\xe3\xf8\x8b", - .klen = 32, - .iv = "\x03\xcf\x76\x3f\xd9\x95\x75\x8f" - "\x44\x89\x40\x7b\x00\x00\x00\x00", - .assoc = "\x8f\x86\x6c\x4d\x1d\xc5\x39\x88" - "\xc8\xf3\x5c\x52\x10\x63\x6f\x2b" - "\x8a\x2a\xc5\x6f\x30\x23\x58\x7b" - "\xfb\x36\x03\x11\xb4\xd9\xf2\xfe", - .alen = 32, - .input = "\x48\x58\xd6\xf3\xad\x63\x58\xbf" - "\xae\xc7\x5e\xae\x83\x8f\x7b\xe4" - "\x78\x5c\x4c\x67\x71\x89\x94\xbf" - "\x47\xf1\x63\x7e\x1c\x59\xbd\xc5" - "\x7f\x44\x0a\x0c\x01\x18\x07\x92" - "\xe1\xd3\x51\xce\x32\x6d\x0c\x5b", - .ilen = 48, - .result = "\xc2\x54\xc8\xde\x78\x87\x77\x40" - "\x49\x71\xe4\xb7\xe7\xcb\x76\x61" - "\x0a\x41\xb9\xe9\xc0\x76\x54\xab" - "\x04\x49\x3b\x19\x93\x57\x25\x5d", - .rlen = 32, - }, -}; - -/* - * rfc4309 refers to section 8 of rfc3610 for test vectors, but they all - * use a 13-byte nonce, we only support an 11-byte nonce. Worse, - * they use AD lengths which are not valid ESP header lengths. - * - * These vectors are copied/generated from the ones for rfc4106 with - * the key truncated by one byte.. - */ -static const struct aead_testvec aes_ccm_rfc4309_enc_tv_template[] = { - { /* Generated using Crypto++ */ - .key = zeroed_string, - .klen = 19, - .iv = zeroed_string, - .input = zeroed_string, - .ilen = 16, - .assoc = zeroed_string, - .alen = 16, - .result = "\x2E\x9A\xCA\x6B\xDA\x54\xFC\x6F" - "\x12\x50\xE8\xDE\x81\x3C\x63\x08" - "\x1A\x22\xBA\x75\xEE\xD4\xD5\xB5" - "\x27\x50\x01\xAC\x03\x33\x39\xFB", - .rlen = 32, - },{ - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00", - .klen = 19, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = zeroed_string, - .ilen = 16, - .assoc = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .result = "\xCF\xB9\x99\x17\xC8\x86\x0E\x7F" - "\x7E\x76\xF8\xE6\xF8\xCC\x1F\x17" - "\x6A\xE0\x53\x9F\x4B\x73\x7E\xDA" - "\x08\x09\x4E\xC4\x1E\xAD\xC6\xB0", - .rlen = 32, - - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00", - .klen = 19, - .iv = zeroed_string, - .input = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .ilen = 16, - .assoc = zeroed_string, - .alen = 16, - .result = "\x33\xDE\x73\xBC\xA6\xCE\x4E\xA6" - "\x61\xF4\xF5\x41\x03\x4A\xE3\x86" - "\xA1\xE2\xC2\x42\x2B\x81\x70\x40" - "\xFD\x7F\x76\xD1\x03\x07\xBB\x0C", - .rlen = 32, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00", - .klen = 19, - .iv = zeroed_string, - .input = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .ilen = 16, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .alen = 16, - .result = "\x33\xDE\x73\xBC\xA6\xCE\x4E\xA6" - "\x61\xF4\xF5\x41\x03\x4A\xE3\x86" - "\x5B\xC0\x73\xE0\x2B\x73\x68\xC9" - "\x2D\x8C\x58\xC2\x90\x3D\xB0\x3E", - .rlen = 32, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00", - .klen = 19, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .ilen = 16, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .result = "\xCE\xB8\x98\x16\xC9\x87\x0F\x7E" - "\x7F\x77\xF9\xE7\xF9\xCD\x1E\x16" - "\x43\x8E\x76\x57\x3B\xB4\x05\xE8" - "\xA9\x9B\xBF\x25\xE0\x4F\xC0\xED", - .rlen = 32, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00", - .klen = 19, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .ilen = 64, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .result = "\xCE\xB8\x98\x16\xC9\x87\x0F\x7E" - "\x7F\x77\xF9\xE7\xF9\xCD\x1E\x16" - "\x9C\xA4\x97\x83\x3F\x01\xA5\xF4" - "\x43\x09\xE7\xB8\xE9\xD1\xD7\x02" - "\x9B\xAB\x39\x18\xEB\x94\x34\x36" - "\xE6\xC5\xC8\x9B\x00\x81\x9E\x49" - "\x1D\x78\xE1\x48\xE3\xE9\xEA\x8E" - "\x3A\x2B\x67\x5D\x35\x6A\x0F\xDB" - "\x02\x73\xDD\xE7\x30\x4A\x30\x54" - "\x1A\x9D\x09\xCA\xC8\x1C\x32\x5F", - .rlen = 80, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x00\x00\x00", - .klen = 19, - .iv = "\x00\x00\x45\x67\x89\xab\xcd\xef", - .input = "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff", - .ilen = 192, - .assoc = "\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa" - "\xaa\xaa\xaa\xaa\x00\x00\x45\x67" - "\x89\xab\xcd\xef", - .alen = 20, - .result = "\x64\x17\xDC\x24\x9D\x92\xBA\x5E" - "\x7C\x64\x6D\x33\x46\x77\xAC\xB1" - "\x5C\x9E\xE2\xC7\x27\x11\x3E\x95" - "\x7D\xBE\x28\xC8\xC1\xCA\x5E\x8C" - "\xB4\xE2\xDE\x9F\x53\x59\x26\xDB" - "\x0C\xD4\xE4\x07\x9A\xE6\x3E\x01" - "\x58\x0D\x3E\x3D\xD5\x21\xEB\x04" - "\x06\x9D\x5F\xB9\x02\x49\x1A\x2B" - "\xBA\xF0\x4E\x3B\x85\x50\x5B\x09" - "\xFE\xEC\xFC\x54\xEC\x0C\xE2\x79" - "\x8A\x2F\x5F\xD7\x05\x5D\xF1\x6D" - "\x22\xEB\xD1\x09\x80\x3F\x5A\x70" - "\xB2\xB9\xD3\x63\x99\xC2\x4D\x1B" - "\x36\x12\x00\x89\xAA\x5D\x55\xDA" - "\x1D\x5B\xD8\x3C\x5F\x09\xD2\xE6" - "\x39\x41\x5C\xF0\xBE\x26\x4E\x5F" - "\x2B\x50\x44\x52\xC2\x10\x7D\x38" - "\x82\x64\x83\x0C\xAE\x49\xD0\xE5" - "\x4F\xE5\x66\x4C\x58\x7A\xEE\x43" - "\x3B\x51\xFE\xBA\x24\x8A\xFE\xDC" - "\x19\x6D\x60\x66\x61\xF9\x9A\x3F" - "\x75\xFC\x38\x53\x5B\xB5\xCD\x52" - "\x4F\xE5\xE4\xC9\xFE\x10\xCB\x98" - "\xF0\x06\x5B\x07\xAB\xBB\xF4\x0E" - "\x2D\xC2\xDD\x5D\xDD\x22\x9A\xCC" - "\x39\xAB\x63\xA5\x3D\x9C\x51\x8A", - .rlen = 208, - }, { /* From draft-mcgrew-gcm-test-01 */ - .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" - "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" - "\x2E\x44\x3B", - .klen = 19, - .iv = "\x49\x56\xED\x7E\x3B\x24\x4C\xFE", - .input = "\x45\x00\x00\x48\x69\x9A\x00\x00" - "\x80\x11\x4D\xB7\xC0\xA8\x01\x02" - "\xC0\xA8\x01\x01\x0A\x9B\xF1\x56" - "\x38\xD3\x01\x00\x00\x01\x00\x00" - "\x00\x00\x00\x00\x04\x5F\x73\x69" - "\x70\x04\x5F\x75\x64\x70\x03\x73" - "\x69\x70\x09\x63\x79\x62\x65\x72" - "\x63\x69\x74\x79\x02\x64\x6B\x00" - "\x00\x21\x00\x01\x01\x02\x02\x01", - .ilen = 72, - .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" - "\x00\x00\x00\x00\x49\x56\xED\x7E" - "\x3B\x24\x4C\xFE", - .alen = 20, - .result = "\x89\xBA\x3E\xEF\xE6\xD6\xCF\xDB" - "\x83\x60\xF5\xBA\x3A\x56\x79\xE6" - "\x7E\x0C\x53\xCF\x9E\x87\xE0\x4E" - "\x1A\x26\x01\x24\xC7\x2E\x3D\xBF" - "\x29\x2C\x91\xC1\xB8\xA8\xCF\xE0" - "\x39\xF8\x53\x6D\x31\x22\x2B\xBF" - "\x98\x81\xFC\x34\xEE\x85\x36\xCD" - "\x26\xDB\x6C\x7A\x0C\x77\x8A\x35" - "\x18\x85\x54\xB2\xBC\xDD\x3F\x43" - "\x61\x06\x8A\xDF\x86\x3F\xB4\xAC" - "\x97\xDC\xBD\xFD\x92\x10\xC5\xFF", - .rlen = 88, - }, { - .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\x6D\x6A\x8F\x94\x67\x30\x83\x08" - "\xCA\xFE\xBA", - .klen = 19, - .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .input = "\x45\x00\x00\x3E\x69\x8F\x00\x00" - "\x80\x11\x4D\xCC\xC0\xA8\x01\x02" - "\xC0\xA8\x01\x01\x0A\x98\x00\x35" - "\x00\x2A\x23\x43\xB2\xD0\x01\x00" - "\x00\x01\x00\x00\x00\x00\x00\x00" - "\x03\x73\x69\x70\x09\x63\x79\x62" - "\x65\x72\x63\x69\x74\x79\x02\x64" - "\x6B\x00\x00\x01\x00\x01\x00\x01", - .ilen = 64, - .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" - "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .alen = 16, - .result = "\x4B\xC2\x70\x60\x64\xD2\xF3\xC8" - "\xE5\x26\x8A\xDE\xB8\x7E\x7D\x16" - "\x56\xC7\xD2\x88\xBA\x8D\x58\xAF" - "\xF5\x71\xB6\x37\x84\xA7\xB1\x99" - "\x51\x5C\x0D\xA0\x27\xDE\xE7\x2D" - "\xEF\x25\x88\x1F\x1D\x77\x11\xFF" - "\xDB\xED\xEE\x56\x16\xC5\x5C\x9B" - "\x00\x62\x1F\x68\x4E\x7C\xA0\x97" - "\x10\x72\x7E\x53\x13\x3B\x68\xE4" - "\x30\x99\x91\x79\x09\xEA\xFF\x6A", - .rlen = 80, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x11\x22\x33", - .klen = 35, - .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", - .input = "\x45\x00\x00\x30\x69\xA6\x40\x00" - "\x80\x06\x26\x90\xC0\xA8\x01\x02" - "\x93\x89\x15\x5E\x0A\x9E\x00\x8B" - "\x2D\xC5\x7E\xE0\x00\x00\x00\x00" - "\x70\x02\x40\x00\x20\xBF\x00\x00" - "\x02\x04\x05\xB4\x01\x01\x04\x02" - "\x01\x02\x02\x01", - .ilen = 52, - .assoc = "\x4A\x2C\xBF\xE3\x00\x00\x00\x02" - "\x01\x02\x03\x04\x05\x06\x07\x08", - .alen = 16, - .result = "\xD6\x31\x0D\x2B\x3D\x6F\xBD\x2F" - "\x58\x41\x7E\xFF\x9A\x9E\x09\xB4" - "\x1A\xF7\xF6\x42\x31\xCD\xBF\xAD" - "\x27\x0E\x2C\xF2\xDB\x10\xDF\x55" - "\x8F\x0D\xD7\xAC\x23\xBD\x42\x10" - "\xD0\xB2\xAF\xD8\x37\xAC\x6B\x0B" - "\x11\xD4\x0B\x12\xEC\xB4\xB1\x92" - "\x23\xA6\x10\xB0\x26\xD6\xD9\x26" - "\x5A\x48\x6A\x3E", - .rlen = 68, - }, { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00", - .klen = 19, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x45\x00\x00\x3C\x99\xC5\x00\x00" - "\x80\x01\xCB\x7A\x40\x67\x93\x18" - "\x01\x01\x01\x01\x08\x00\x07\x5C" - "\x02\x00\x44\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x75\x76\x77\x61\x62\x63\x64\x65" - "\x66\x67\x68\x69\x01\x02\x02\x01", - .ilen = 64, - .assoc = "\x00\x00\x00\x00\x00\x00\x00\x01" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .alen = 16, - .result = "\x6B\x9A\xCA\x57\x43\x91\xFC\x6F" - "\x92\x51\x23\xA4\xC1\x5B\xF0\x10" - "\xF3\x13\xF4\xF8\xA1\x9A\xB4\xDC" - "\x89\xC8\xF8\x42\x62\x95\xB7\xCB" - "\xB8\xF5\x0F\x1B\x2E\x94\xA2\xA7" - "\xBF\xFB\x8A\x92\x13\x63\xD1\x3C" - "\x08\xF5\xE8\xA6\xAA\xF6\x34\xF9" - "\x42\x05\xAF\xB3\xE7\x9A\xFC\xEE" - "\x36\x25\xC1\x10\x12\x1C\xCA\x82" - "\xEA\xE6\x63\x5A\x57\x28\xA9\x9A", - .rlen = 80, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E", - .klen = 19, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .input = "\x45\x00\x00\x3C\x99\xC3\x00\x00" - "\x80\x01\xCB\x7C\x40\x67\x93\x18" - "\x01\x01\x01\x01\x08\x00\x08\x5C" - "\x02\x00\x43\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x75\x76\x77\x61\x62\x63\x64\x65" - "\x66\x67\x68\x69\x01\x02\x02\x01", - .ilen = 64, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .result = "\x6A\x6B\x45\x2B\x7C\x67\x52\xF6" - "\x10\x60\x40\x62\x6B\x4F\x97\x8E" - "\x0B\xB2\x22\x97\xCB\x21\xE0\x90" - "\xA2\xE7\xD1\x41\x30\xE4\x4B\x1B" - "\x79\x01\x58\x50\x01\x06\xE1\xE0" - "\x2C\x83\x79\xD3\xDE\x46\x97\x1A" - "\x30\xB8\xE5\xDF\xD7\x12\x56\x75" - "\xD0\x95\xB7\xB8\x91\x42\xF7\xFD" - "\x97\x57\xCA\xC1\x20\xD0\x86\xB9" - "\x66\x9D\xB4\x2B\x96\x22\xAC\x67", - .rlen = 80, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E", - .klen = 19, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .input = "\x45\x00\x00\x1C\x42\xA2\x00\x00" - "\x80\x01\x44\x1F\x40\x67\x93\xB6" - "\xE0\x00\x00\x02\x0A\x00\xF5\xFF" - "\x01\x02\x02\x01", - .ilen = 28, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .result = "\x6A\x6B\x45\x0B\xA7\x06\x52\xF6" - "\x10\x60\xCF\x01\x6B\x4F\x97\x20" - "\xEA\xB3\x23\x94\xC9\x21\x1D\x33" - "\xA1\xE5\x90\x40\x05\x37\x45\x70" - "\xB5\xD6\x09\x0A\x23\x73\x33\xF9" - "\x08\xB4\x22\xE4", - .rlen = 44, - }, { - .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\x6D\x6A\x8F\x94\x67\x30\x83\x08" - "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\xCA\xFE\xBA", - .klen = 27, - .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .input = "\x45\x00\x00\x28\xA4\xAD\x40\x00" - "\x40\x06\x78\x80\x0A\x01\x03\x8F" - "\x0A\x01\x06\x12\x80\x23\x06\xB8" - "\xCB\x71\x26\x02\xDD\x6B\xB0\x3E" - "\x50\x10\x16\xD0\x75\x68\x00\x01", - .ilen = 40, - .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" - "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .alen = 16, - .result = "\x05\x22\x15\xD1\x52\x56\x85\x04" - "\xA8\x5C\x5D\x6D\x7E\x6E\xF5\xFA" - "\xEA\x16\x37\x50\xF3\xDF\x84\x3B" - "\x2F\x32\x18\x57\x34\x2A\x8C\x23" - "\x67\xDF\x6D\x35\x7B\x54\x0D\xFB" - "\x34\xA5\x9F\x6C\x48\x30\x1E\x22" - "\xFE\xB1\x22\x17\x17\x8A\xB9\x5B", - .rlen = 56, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xDE\xCA\xF8", - .klen = 19, - .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", - .input = "\x45\x00\x00\x49\x33\xBA\x00\x00" - "\x7F\x11\x91\x06\xC3\xFB\x1D\x10" - "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" - "\x00\x35\xDD\x7B\x80\x03\x02\xD5" - "\x00\x00\x4E\x20\x00\x1E\x8C\x18" - "\xD7\x5B\x81\xDC\x91\xBA\xA0\x47" - "\x6B\x91\xB9\x24\xB2\x80\x38\x9D" - "\x92\xC9\x63\xBA\xC0\x46\xEC\x95" - "\x9B\x62\x66\xC0\x47\x22\xB1\x49" - "\x23\x01\x01\x01", - .ilen = 76, - .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" - "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" - "\xCE\xFA\xCE\x74", - .alen = 20, - .result = "\x92\xD0\x53\x79\x33\x38\xD5\xF3" - "\x7D\xE4\x7A\x8E\x86\x03\xC9\x90" - "\x96\x35\xAB\x9C\xFB\xE8\xA3\x76" - "\xE9\xE9\xE2\xD1\x2E\x11\x0E\x00" - "\xFA\xCE\xB5\x9E\x02\xA7\x7B\xEA" - "\x71\x9A\x58\xFB\xA5\x8A\xE1\xB7" - "\x9C\x39\x9D\xE3\xB5\x6E\x69\xE6" - "\x63\xC9\xDB\x05\x69\x51\x12\xAD" - "\x3E\x00\x32\x73\x86\xF2\xEE\xF5" - "\x0F\xE8\x81\x7E\x84\xD3\xC0\x0D" - "\x76\xD6\x55\xC6\xB4\xC2\x34\xC7" - "\x12\x25\x0B\xF9", - .rlen = 92, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x73\x61\x6C", - .klen = 35, - .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", - .input = "\x45\x08\x00\x28\x73\x2C\x00\x00" - "\x40\x06\xE9\xF9\x0A\x01\x06\x12" - "\x0A\x01\x03\x8F\x06\xB8\x80\x23" - "\xDD\x6B\xAF\xBE\xCB\x71\x26\x02" - "\x50\x10\x1F\x64\x6D\x54\x00\x01", - .ilen = 40, - .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" - "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" - "\x69\x76\x65\x63", - .alen = 20, - .result = "\xCC\x74\xB7\xD3\xB0\x38\x50\x42" - "\x2C\x64\x87\x46\x1E\x34\x10\x05" - "\x29\x6B\xBB\x36\xE9\x69\xAD\x92" - "\x82\xA1\x10\x6A\xEB\x0F\xDC\x7D" - "\x08\xBA\xF3\x91\xCA\xAA\x61\xDA" - "\x62\xF4\x14\x61\x5C\x9D\xB5\xA7" - "\xEE\xD7\xB9\x7E\x87\x99\x9B\x7D", - .rlen = 56, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E", - .klen = 19, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .input = "\x45\x00\x00\x49\x33\x3E\x00\x00" - "\x7F\x11\x91\x82\xC3\xFB\x1D\x10" - "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" - "\x00\x35\xCB\x45\x80\x03\x02\x5B" - "\x00\x00\x01\xE0\x00\x1E\x8C\x18" - "\xD6\x57\x59\xD5\x22\x84\xA0\x35" - "\x2C\x71\x47\x5C\x88\x80\x39\x1C" - "\x76\x4D\x6E\x5E\xE0\x49\x6B\x32" - "\x5A\xE2\x70\xC0\x38\x99\x49\x39" - "\x15\x01\x01\x01", - .ilen = 76, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .result = "\x6A\x6B\x45\x5E\xD6\x9A\x52\xF6" - "\xEF\x70\x1A\x9C\xE8\xD3\x19\x86" - "\xC8\x02\xF0\xB0\x03\x09\xD9\x02" - "\xA0\xD2\x59\x04\xD1\x85\x2A\x24" - "\x1C\x67\x3E\xD8\x68\x72\x06\x94" - "\x97\xBA\x4F\x76\x8D\xB0\x44\x5B" - "\x69\xBF\xD5\xE2\x3D\xF1\x0B\x0C" - "\xC0\xBF\xB1\x8F\x70\x09\x9E\xCE" - "\xA5\xF2\x55\x58\x84\xFA\xF9\xB5" - "\x23\xF4\x84\x40\x74\x14\x8A\x6B" - "\xDB\xD7\x67\xED\xA4\x93\xF3\x47" - "\xCC\xF7\x46\x6F", - .rlen = 92, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x73\x61\x6C", - .klen = 35, - .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", - .input = "\x63\x69\x73\x63\x6F\x01\x72\x75" - "\x6C\x65\x73\x01\x74\x68\x65\x01" - "\x6E\x65\x74\x77\x65\x01\x64\x65" - "\x66\x69\x6E\x65\x01\x74\x68\x65" - "\x74\x65\x63\x68\x6E\x6F\x6C\x6F" - "\x67\x69\x65\x73\x01\x74\x68\x61" - "\x74\x77\x69\x6C\x6C\x01\x64\x65" - "\x66\x69\x6E\x65\x74\x6F\x6D\x6F" - "\x72\x72\x6F\x77\x01\x02\x02\x01", - .ilen = 72, - .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" - "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" - "\x69\x76\x65\x63", - .alen = 20, - .result = "\xEA\x15\xC4\x98\xAC\x15\x22\x37" - "\x00\x07\x1D\xBE\x60\x5D\x73\x16" - "\x4D\x0F\xCC\xCE\x8A\xD0\x49\xD4" - "\x39\xA3\xD1\xB1\x21\x0A\x92\x1A" - "\x2C\xCF\x8F\x9D\xC9\x91\x0D\xB4" - "\x15\xFC\xBC\xA5\xC5\xBF\x54\xE5" - "\x1C\xC7\x32\x41\x07\x7B\x2C\xB6" - "\x5C\x23\x7C\x93\xEA\xEF\x23\x1C" - "\x73\xF4\xE7\x12\x84\x4C\x37\x0A" - "\x4A\x8F\x06\x37\x48\xF9\xF9\x05" - "\x55\x13\x40\xC3\xD5\x55\x3A\x3D", - .rlen = 88, - }, { - .key = "\x7D\x77\x3D\x00\xC1\x44\xC5\x25" - "\xAC\x61\x9D\x18\xC8\x4A\x3F\x47" - "\xD9\x66\x42", - .klen = 19, - .iv = "\x43\x45\x7E\x91\x82\x44\x3B\xC6", - .input = "\x01\x02\x02\x01", - .ilen = 4, - .assoc = "\x33\x54\x67\xAE\xFF\xFF\xFF\xFF" - "\x43\x45\x7E\x91\x82\x44\x3B\xC6", - .alen = 16, - .result = "\x4C\x72\x63\x30\x2F\xE6\x56\xDD" - "\xD0\xD8\x60\x9D\x8B\xEF\x85\x90" - "\xF7\x61\x24\x62", - .rlen = 20, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xDE\xCA\xF8", - .klen = 19, - .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", - .input = "\x74\x6F\x01\x62\x65\x01\x6F\x72" - "\x01\x6E\x6F\x74\x01\x74\x6F\x01" - "\x62\x65\x00\x01", - .ilen = 20, - .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" - "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" - "\xCE\xFA\xCE\x74", - .alen = 20, - .result = "\xA3\xBF\x52\x52\x65\x83\xBA\x81" - "\x03\x9B\x84\xFC\x44\x8C\xBB\x81" - "\x36\xE1\x78\xBB\xA5\x49\x3A\xD0" - "\xF0\x6B\x21\xAF\x98\xC0\x34\xDC" - "\x17\x17\x65\xAD", - .rlen = 36, - }, { - .key = "\x6C\x65\x67\x61\x6C\x69\x7A\x65" - "\x6D\x61\x72\x69\x6A\x75\x61\x6E" - "\x61\x61\x6E\x64\x64\x6F\x69\x74" - "\x62\x65\x66\x6F\x72\x65\x69\x61" - "\x74\x75\x72", - .klen = 35, - .iv = "\x33\x30\x21\x69\x67\x65\x74\x6D", - .input = "\x45\x00\x00\x30\xDA\x3A\x00\x00" - "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" - "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01", - .ilen = 52, - .assoc = "\x79\x6B\x69\x63\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\x33\x30\x21\x69" - "\x67\x65\x74\x6D", - .alen = 20, - .result = "\x96\xFD\x86\xF8\xD1\x98\xFF\x10" - "\xAB\x8C\xDA\x8A\x5A\x08\x38\x1A" - "\x48\x59\x80\x18\x1A\x18\x1A\x04" - "\xC9\x0D\xE3\xE7\x0E\xA4\x0B\x75" - "\x92\x9C\x52\x5C\x0B\xFB\xF8\xAF" - "\x16\xC3\x35\xA8\xE7\xCE\x84\x04" - "\xEB\x40\x6B\x7A\x8E\x75\xBB\x42" - "\xE0\x63\x4B\x21\x44\xA2\x2B\x2B" - "\x39\xDB\xC8\xDC", - .rlen = 68, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E", - .klen = 19, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .input = "\x45\x00\x00\x30\xDA\x3A\x00\x00" - "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" - "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01", - .ilen = 52, - .assoc = "\x3F\x7E\xF6\x42\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .result = "\x6A\x6B\x45\x27\x3F\x9E\x52\xF6" - "\x10\x60\x54\x25\xEB\x80\x04\x93" - "\xCA\x1B\x23\x97\xCB\x21\x2E\x01" - "\xA2\xE7\x95\x41\x30\xE4\x4B\x1B" - "\x79\x01\x58\x50\x01\x06\xE1\xE0" - "\x2C\x83\x79\xD3\xDE\x46\x97\x1A" - "\x44\xCC\x90\xBF\x00\x94\x94\x92" - "\x20\x17\x0C\x1B\x55\xDE\x7E\x68" - "\xF4\x95\x5D\x4F", - .rlen = 68, - }, { - .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" - "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" - "\x22\x43\x3C", - .klen = 19, - .iv = "\x48\x55\xEC\x7D\x3A\x23\x4B\xFD", - .input = "\x08\x00\xC6\xCD\x02\x00\x07\x00" - "\x61\x62\x63\x64\x65\x66\x67\x68" - "\x69\x6A\x6B\x6C\x6D\x6E\x6F\x70" - "\x71\x72\x73\x74\x01\x02\x02\x01", - .ilen = 32, - .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" - "\x00\x00\x00\x07\x48\x55\xEC\x7D" - "\x3A\x23\x4B\xFD", - .alen = 20, - .result = "\x67\xE9\x28\xB3\x1C\xA4\x6D\x02" - "\xF0\xB5\x37\xB6\x6B\x2F\xF5\x4F" - "\xF8\xA3\x4C\x53\xB8\x12\x09\xBF" - "\x58\x7D\xCF\x29\xA3\x41\x68\x6B" - "\xCE\xE8\x79\x85\x3C\xB0\x3A\x8F" - "\x16\xB0\xA1\x26\xC9\xBC\xBC\xA6", - .rlen = 48, - } -}; - -static const struct aead_testvec aes_ccm_rfc4309_dec_tv_template[] = { - { /* Generated using Crypto++ */ - .key = zeroed_string, - .klen = 19, - .iv = zeroed_string, - .result = zeroed_string, - .rlen = 16, - .assoc = zeroed_string, - .alen = 16, - .input = "\x2E\x9A\xCA\x6B\xDA\x54\xFC\x6F" - "\x12\x50\xE8\xDE\x81\x3C\x63\x08" - "\x1A\x22\xBA\x75\xEE\xD4\xD5\xB5" - "\x27\x50\x01\xAC\x03\x33\x39\xFB", - .ilen = 32, - },{ - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00", - .klen = 19, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .result = zeroed_string, - .rlen = 16, - .assoc = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .input = "\xCF\xB9\x99\x17\xC8\x86\x0E\x7F" - "\x7E\x76\xF8\xE6\xF8\xCC\x1F\x17" - "\x6A\xE0\x53\x9F\x4B\x73\x7E\xDA" - "\x08\x09\x4E\xC4\x1E\xAD\xC6\xB0", - .ilen = 32, - - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00", - .klen = 19, - .iv = zeroed_string, - .result = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .rlen = 16, - .assoc = zeroed_string, - .alen = 16, - .input = "\x33\xDE\x73\xBC\xA6\xCE\x4E\xA6" - "\x61\xF4\xF5\x41\x03\x4A\xE3\x86" - "\xA1\xE2\xC2\x42\x2B\x81\x70\x40" - "\xFD\x7F\x76\xD1\x03\x07\xBB\x0C", - .ilen = 32, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00", - .klen = 19, - .iv = zeroed_string, - .result = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .rlen = 16, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .alen = 16, - .input = "\x33\xDE\x73\xBC\xA6\xCE\x4E\xA6" - "\x61\xF4\xF5\x41\x03\x4A\xE3\x86" - "\x5B\xC0\x73\xE0\x2B\x73\x68\xC9" - "\x2D\x8C\x58\xC2\x90\x3D\xB0\x3E", - .ilen = 32, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00", - .klen = 19, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .result = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .rlen = 16, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .input = "\xCE\xB8\x98\x16\xC9\x87\x0F\x7E" - "\x7F\x77\xF9\xE7\xF9\xCD\x1E\x16" - "\x43\x8E\x76\x57\x3B\xB4\x05\xE8" - "\xA9\x9B\xBF\x25\xE0\x4F\xC0\xED", - .ilen = 32, - }, { - .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" - "\x6d\x6a\x8f\x94\x67\x30\x83\x08" - "\x00\x00\x00", - .klen = 19, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", - .result = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x01\x01\x01\x01\x01\x01\x01\x01", - .rlen = 64, - .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .alen = 16, - .input = "\xCE\xB8\x98\x16\xC9\x87\x0F\x7E" - "\x7F\x77\xF9\xE7\xF9\xCD\x1E\x16" - "\x9C\xA4\x97\x83\x3F\x01\xA5\xF4" - "\x43\x09\xE7\xB8\xE9\xD1\xD7\x02" - "\x9B\xAB\x39\x18\xEB\x94\x34\x36" - "\xE6\xC5\xC8\x9B\x00\x81\x9E\x49" - "\x1D\x78\xE1\x48\xE3\xE9\xEA\x8E" - "\x3A\x2B\x67\x5D\x35\x6A\x0F\xDB" - "\x02\x73\xDD\xE7\x30\x4A\x30\x54" - "\x1A\x9D\x09\xCA\xC8\x1C\x32\x5F", - .ilen = 80, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x00\x00\x00", - .klen = 19, - .iv = "\x00\x00\x45\x67\x89\xab\xcd\xef", - .result = "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff" - "\xff\xff\xff\xff\xff\xff\xff\xff", - .rlen = 192, - .assoc = "\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa" - "\xaa\xaa\xaa\xaa\x00\x00\x45\x67" - "\x89\xab\xcd\xef", - .alen = 20, - .input = "\x64\x17\xDC\x24\x9D\x92\xBA\x5E" - "\x7C\x64\x6D\x33\x46\x77\xAC\xB1" - "\x5C\x9E\xE2\xC7\x27\x11\x3E\x95" - "\x7D\xBE\x28\xC8\xC1\xCA\x5E\x8C" - "\xB4\xE2\xDE\x9F\x53\x59\x26\xDB" - "\x0C\xD4\xE4\x07\x9A\xE6\x3E\x01" - "\x58\x0D\x3E\x3D\xD5\x21\xEB\x04" - "\x06\x9D\x5F\xB9\x02\x49\x1A\x2B" - "\xBA\xF0\x4E\x3B\x85\x50\x5B\x09" - "\xFE\xEC\xFC\x54\xEC\x0C\xE2\x79" - "\x8A\x2F\x5F\xD7\x05\x5D\xF1\x6D" - "\x22\xEB\xD1\x09\x80\x3F\x5A\x70" - "\xB2\xB9\xD3\x63\x99\xC2\x4D\x1B" - "\x36\x12\x00\x89\xAA\x5D\x55\xDA" - "\x1D\x5B\xD8\x3C\x5F\x09\xD2\xE6" - "\x39\x41\x5C\xF0\xBE\x26\x4E\x5F" - "\x2B\x50\x44\x52\xC2\x10\x7D\x38" - "\x82\x64\x83\x0C\xAE\x49\xD0\xE5" - "\x4F\xE5\x66\x4C\x58\x7A\xEE\x43" - "\x3B\x51\xFE\xBA\x24\x8A\xFE\xDC" - "\x19\x6D\x60\x66\x61\xF9\x9A\x3F" - "\x75\xFC\x38\x53\x5B\xB5\xCD\x52" - "\x4F\xE5\xE4\xC9\xFE\x10\xCB\x98" - "\xF0\x06\x5B\x07\xAB\xBB\xF4\x0E" - "\x2D\xC2\xDD\x5D\xDD\x22\x9A\xCC" - "\x39\xAB\x63\xA5\x3D\x9C\x51\x8A", - .ilen = 208, - }, { /* From draft-mcgrew-gcm-test-01 */ - .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" - "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" - "\x2E\x44\x3B", - .klen = 19, - .iv = "\x49\x56\xED\x7E\x3B\x24\x4C\xFE", - .result = "\x45\x00\x00\x48\x69\x9A\x00\x00" - "\x80\x11\x4D\xB7\xC0\xA8\x01\x02" - "\xC0\xA8\x01\x01\x0A\x9B\xF1\x56" - "\x38\xD3\x01\x00\x00\x01\x00\x00" - "\x00\x00\x00\x00\x04\x5F\x73\x69" - "\x70\x04\x5F\x75\x64\x70\x03\x73" - "\x69\x70\x09\x63\x79\x62\x65\x72" - "\x63\x69\x74\x79\x02\x64\x6B\x00" - "\x00\x21\x00\x01\x01\x02\x02\x01", - .rlen = 72, - .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" - "\x00\x00\x00\x00\x49\x56\xED\x7E" - "\x3B\x24\x4C\xFE", - .alen = 20, - .input = "\x89\xBA\x3E\xEF\xE6\xD6\xCF\xDB" - "\x83\x60\xF5\xBA\x3A\x56\x79\xE6" - "\x7E\x0C\x53\xCF\x9E\x87\xE0\x4E" - "\x1A\x26\x01\x24\xC7\x2E\x3D\xBF" - "\x29\x2C\x91\xC1\xB8\xA8\xCF\xE0" - "\x39\xF8\x53\x6D\x31\x22\x2B\xBF" - "\x98\x81\xFC\x34\xEE\x85\x36\xCD" - "\x26\xDB\x6C\x7A\x0C\x77\x8A\x35" - "\x18\x85\x54\xB2\xBC\xDD\x3F\x43" - "\x61\x06\x8A\xDF\x86\x3F\xB4\xAC" - "\x97\xDC\xBD\xFD\x92\x10\xC5\xFF", - .ilen = 88, - }, { - .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\x6D\x6A\x8F\x94\x67\x30\x83\x08" - "\xCA\xFE\xBA", - .klen = 19, - .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .result = "\x45\x00\x00\x3E\x69\x8F\x00\x00" - "\x80\x11\x4D\xCC\xC0\xA8\x01\x02" - "\xC0\xA8\x01\x01\x0A\x98\x00\x35" - "\x00\x2A\x23\x43\xB2\xD0\x01\x00" - "\x00\x01\x00\x00\x00\x00\x00\x00" - "\x03\x73\x69\x70\x09\x63\x79\x62" - "\x65\x72\x63\x69\x74\x79\x02\x64" - "\x6B\x00\x00\x01\x00\x01\x00\x01", - .rlen = 64, - .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" - "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .alen = 16, - .input = "\x4B\xC2\x70\x60\x64\xD2\xF3\xC8" - "\xE5\x26\x8A\xDE\xB8\x7E\x7D\x16" - "\x56\xC7\xD2\x88\xBA\x8D\x58\xAF" - "\xF5\x71\xB6\x37\x84\xA7\xB1\x99" - "\x51\x5C\x0D\xA0\x27\xDE\xE7\x2D" - "\xEF\x25\x88\x1F\x1D\x77\x11\xFF" - "\xDB\xED\xEE\x56\x16\xC5\x5C\x9B" - "\x00\x62\x1F\x68\x4E\x7C\xA0\x97" - "\x10\x72\x7E\x53\x13\x3B\x68\xE4" - "\x30\x99\x91\x79\x09\xEA\xFF\x6A", - .ilen = 80, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x11\x22\x33", - .klen = 35, - .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", - .result = "\x45\x00\x00\x30\x69\xA6\x40\x00" - "\x80\x06\x26\x90\xC0\xA8\x01\x02" - "\x93\x89\x15\x5E\x0A\x9E\x00\x8B" - "\x2D\xC5\x7E\xE0\x00\x00\x00\x00" - "\x70\x02\x40\x00\x20\xBF\x00\x00" - "\x02\x04\x05\xB4\x01\x01\x04\x02" - "\x01\x02\x02\x01", - .rlen = 52, - .assoc = "\x4A\x2C\xBF\xE3\x00\x00\x00\x02" - "\x01\x02\x03\x04\x05\x06\x07\x08", - .alen = 16, - .input = "\xD6\x31\x0D\x2B\x3D\x6F\xBD\x2F" - "\x58\x41\x7E\xFF\x9A\x9E\x09\xB4" - "\x1A\xF7\xF6\x42\x31\xCD\xBF\xAD" - "\x27\x0E\x2C\xF2\xDB\x10\xDF\x55" - "\x8F\x0D\xD7\xAC\x23\xBD\x42\x10" - "\xD0\xB2\xAF\xD8\x37\xAC\x6B\x0B" - "\x11\xD4\x0B\x12\xEC\xB4\xB1\x92" - "\x23\xA6\x10\xB0\x26\xD6\xD9\x26" - "\x5A\x48\x6A\x3E", - .ilen = 68, - }, { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00", - .klen = 19, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .result = "\x45\x00\x00\x3C\x99\xC5\x00\x00" - "\x80\x01\xCB\x7A\x40\x67\x93\x18" - "\x01\x01\x01\x01\x08\x00\x07\x5C" - "\x02\x00\x44\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x75\x76\x77\x61\x62\x63\x64\x65" - "\x66\x67\x68\x69\x01\x02\x02\x01", - .rlen = 64, - .assoc = "\x00\x00\x00\x00\x00\x00\x00\x01" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .alen = 16, - .input = "\x6B\x9A\xCA\x57\x43\x91\xFC\x6F" - "\x92\x51\x23\xA4\xC1\x5B\xF0\x10" - "\xF3\x13\xF4\xF8\xA1\x9A\xB4\xDC" - "\x89\xC8\xF8\x42\x62\x95\xB7\xCB" - "\xB8\xF5\x0F\x1B\x2E\x94\xA2\xA7" - "\xBF\xFB\x8A\x92\x13\x63\xD1\x3C" - "\x08\xF5\xE8\xA6\xAA\xF6\x34\xF9" - "\x42\x05\xAF\xB3\xE7\x9A\xFC\xEE" - "\x36\x25\xC1\x10\x12\x1C\xCA\x82" - "\xEA\xE6\x63\x5A\x57\x28\xA9\x9A", - .ilen = 80, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E", - .klen = 19, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .result = "\x45\x00\x00\x3C\x99\xC3\x00\x00" - "\x80\x01\xCB\x7C\x40\x67\x93\x18" - "\x01\x01\x01\x01\x08\x00\x08\x5C" - "\x02\x00\x43\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x75\x76\x77\x61\x62\x63\x64\x65" - "\x66\x67\x68\x69\x01\x02\x02\x01", - .rlen = 64, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .input = "\x6A\x6B\x45\x2B\x7C\x67\x52\xF6" - "\x10\x60\x40\x62\x6B\x4F\x97\x8E" - "\x0B\xB2\x22\x97\xCB\x21\xE0\x90" - "\xA2\xE7\xD1\x41\x30\xE4\x4B\x1B" - "\x79\x01\x58\x50\x01\x06\xE1\xE0" - "\x2C\x83\x79\xD3\xDE\x46\x97\x1A" - "\x30\xB8\xE5\xDF\xD7\x12\x56\x75" - "\xD0\x95\xB7\xB8\x91\x42\xF7\xFD" - "\x97\x57\xCA\xC1\x20\xD0\x86\xB9" - "\x66\x9D\xB4\x2B\x96\x22\xAC\x67", - .ilen = 80, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E", - .klen = 19, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .result = "\x45\x00\x00\x1C\x42\xA2\x00\x00" - "\x80\x01\x44\x1F\x40\x67\x93\xB6" - "\xE0\x00\x00\x02\x0A\x00\xF5\xFF" - "\x01\x02\x02\x01", - .rlen = 28, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .input = "\x6A\x6B\x45\x0B\xA7\x06\x52\xF6" - "\x10\x60\xCF\x01\x6B\x4F\x97\x20" - "\xEA\xB3\x23\x94\xC9\x21\x1D\x33" - "\xA1\xE5\x90\x40\x05\x37\x45\x70" - "\xB5\xD6\x09\x0A\x23\x73\x33\xF9" - "\x08\xB4\x22\xE4", - .ilen = 44, - }, { - .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\x6D\x6A\x8F\x94\x67\x30\x83\x08" - "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" - "\xCA\xFE\xBA", - .klen = 27, - .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .result = "\x45\x00\x00\x28\xA4\xAD\x40\x00" - "\x40\x06\x78\x80\x0A\x01\x03\x8F" - "\x0A\x01\x06\x12\x80\x23\x06\xB8" - "\xCB\x71\x26\x02\xDD\x6B\xB0\x3E" - "\x50\x10\x16\xD0\x75\x68\x00\x01", - .rlen = 40, - .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" - "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", - .alen = 16, - .input = "\x05\x22\x15\xD1\x52\x56\x85\x04" - "\xA8\x5C\x5D\x6D\x7E\x6E\xF5\xFA" - "\xEA\x16\x37\x50\xF3\xDF\x84\x3B" - "\x2F\x32\x18\x57\x34\x2A\x8C\x23" - "\x67\xDF\x6D\x35\x7B\x54\x0D\xFB" - "\x34\xA5\x9F\x6C\x48\x30\x1E\x22" - "\xFE\xB1\x22\x17\x17\x8A\xB9\x5B", - .ilen = 56, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xDE\xCA\xF8", - .klen = 19, - .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", - .result = "\x45\x00\x00\x49\x33\xBA\x00\x00" - "\x7F\x11\x91\x06\xC3\xFB\x1D\x10" - "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" - "\x00\x35\xDD\x7B\x80\x03\x02\xD5" - "\x00\x00\x4E\x20\x00\x1E\x8C\x18" - "\xD7\x5B\x81\xDC\x91\xBA\xA0\x47" - "\x6B\x91\xB9\x24\xB2\x80\x38\x9D" - "\x92\xC9\x63\xBA\xC0\x46\xEC\x95" - "\x9B\x62\x66\xC0\x47\x22\xB1\x49" - "\x23\x01\x01\x01", - .rlen = 76, - .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" - "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" - "\xCE\xFA\xCE\x74", - .alen = 20, - .input = "\x92\xD0\x53\x79\x33\x38\xD5\xF3" - "\x7D\xE4\x7A\x8E\x86\x03\xC9\x90" - "\x96\x35\xAB\x9C\xFB\xE8\xA3\x76" - "\xE9\xE9\xE2\xD1\x2E\x11\x0E\x00" - "\xFA\xCE\xB5\x9E\x02\xA7\x7B\xEA" - "\x71\x9A\x58\xFB\xA5\x8A\xE1\xB7" - "\x9C\x39\x9D\xE3\xB5\x6E\x69\xE6" - "\x63\xC9\xDB\x05\x69\x51\x12\xAD" - "\x3E\x00\x32\x73\x86\xF2\xEE\xF5" - "\x0F\xE8\x81\x7E\x84\xD3\xC0\x0D" - "\x76\xD6\x55\xC6\xB4\xC2\x34\xC7" - "\x12\x25\x0B\xF9", - .ilen = 92, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x73\x61\x6C", - .klen = 35, - .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", - .result = "\x45\x08\x00\x28\x73\x2C\x00\x00" - "\x40\x06\xE9\xF9\x0A\x01\x06\x12" - "\x0A\x01\x03\x8F\x06\xB8\x80\x23" - "\xDD\x6B\xAF\xBE\xCB\x71\x26\x02" - "\x50\x10\x1F\x64\x6D\x54\x00\x01", - .rlen = 40, - .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" - "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" - "\x69\x76\x65\x63", - .alen = 20, - .input = "\xCC\x74\xB7\xD3\xB0\x38\x50\x42" - "\x2C\x64\x87\x46\x1E\x34\x10\x05" - "\x29\x6B\xBB\x36\xE9\x69\xAD\x92" - "\x82\xA1\x10\x6A\xEB\x0F\xDC\x7D" - "\x08\xBA\xF3\x91\xCA\xAA\x61\xDA" - "\x62\xF4\x14\x61\x5C\x9D\xB5\xA7" - "\xEE\xD7\xB9\x7E\x87\x99\x9B\x7D", - .ilen = 56, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E", - .klen = 19, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .result = "\x45\x00\x00\x49\x33\x3E\x00\x00" - "\x7F\x11\x91\x82\xC3\xFB\x1D\x10" - "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" - "\x00\x35\xCB\x45\x80\x03\x02\x5B" - "\x00\x00\x01\xE0\x00\x1E\x8C\x18" - "\xD6\x57\x59\xD5\x22\x84\xA0\x35" - "\x2C\x71\x47\x5C\x88\x80\x39\x1C" - "\x76\x4D\x6E\x5E\xE0\x49\x6B\x32" - "\x5A\xE2\x70\xC0\x38\x99\x49\x39" - "\x15\x01\x01\x01", - .rlen = 76, - .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .input = "\x6A\x6B\x45\x5E\xD6\x9A\x52\xF6" - "\xEF\x70\x1A\x9C\xE8\xD3\x19\x86" - "\xC8\x02\xF0\xB0\x03\x09\xD9\x02" - "\xA0\xD2\x59\x04\xD1\x85\x2A\x24" - "\x1C\x67\x3E\xD8\x68\x72\x06\x94" - "\x97\xBA\x4F\x76\x8D\xB0\x44\x5B" - "\x69\xBF\xD5\xE2\x3D\xF1\x0B\x0C" - "\xC0\xBF\xB1\x8F\x70\x09\x9E\xCE" - "\xA5\xF2\x55\x58\x84\xFA\xF9\xB5" - "\x23\xF4\x84\x40\x74\x14\x8A\x6B" - "\xDB\xD7\x67\xED\xA4\x93\xF3\x47" - "\xCC\xF7\x46\x6F", - .ilen = 92, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\x73\x61\x6C", - .klen = 35, - .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", - .result = "\x63\x69\x73\x63\x6F\x01\x72\x75" - "\x6C\x65\x73\x01\x74\x68\x65\x01" - "\x6E\x65\x74\x77\x65\x01\x64\x65" - "\x66\x69\x6E\x65\x01\x74\x68\x65" - "\x74\x65\x63\x68\x6E\x6F\x6C\x6F" - "\x67\x69\x65\x73\x01\x74\x68\x61" - "\x74\x77\x69\x6C\x6C\x01\x64\x65" - "\x66\x69\x6E\x65\x74\x6F\x6D\x6F" - "\x72\x72\x6F\x77\x01\x02\x02\x01", - .rlen = 72, - .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" - "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" - "\x69\x76\x65\x63", - .alen = 20, - .input = "\xEA\x15\xC4\x98\xAC\x15\x22\x37" - "\x00\x07\x1D\xBE\x60\x5D\x73\x16" - "\x4D\x0F\xCC\xCE\x8A\xD0\x49\xD4" - "\x39\xA3\xD1\xB1\x21\x0A\x92\x1A" - "\x2C\xCF\x8F\x9D\xC9\x91\x0D\xB4" - "\x15\xFC\xBC\xA5\xC5\xBF\x54\xE5" - "\x1C\xC7\x32\x41\x07\x7B\x2C\xB6" - "\x5C\x23\x7C\x93\xEA\xEF\x23\x1C" - "\x73\xF4\xE7\x12\x84\x4C\x37\x0A" - "\x4A\x8F\x06\x37\x48\xF9\xF9\x05" - "\x55\x13\x40\xC3\xD5\x55\x3A\x3D", - .ilen = 88, - }, { - .key = "\x7D\x77\x3D\x00\xC1\x44\xC5\x25" - "\xAC\x61\x9D\x18\xC8\x4A\x3F\x47" - "\xD9\x66\x42", - .klen = 19, - .iv = "\x43\x45\x7E\x91\x82\x44\x3B\xC6", - .result = "\x01\x02\x02\x01", - .rlen = 4, - .assoc = "\x33\x54\x67\xAE\xFF\xFF\xFF\xFF" - "\x43\x45\x7E\x91\x82\x44\x3B\xC6", - .alen = 16, - .input = "\x4C\x72\x63\x30\x2F\xE6\x56\xDD" - "\xD0\xD8\x60\x9D\x8B\xEF\x85\x90" - "\xF7\x61\x24\x62", - .ilen = 20, - }, { - .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" - "\x34\x45\x56\x67\x78\x89\x9A\xAB" - "\xDE\xCA\xF8", - .klen = 19, - .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", - .result = "\x74\x6F\x01\x62\x65\x01\x6F\x72" - "\x01\x6E\x6F\x74\x01\x74\x6F\x01" - "\x62\x65\x00\x01", - .rlen = 20, - .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" - "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" - "\xCE\xFA\xCE\x74", - .alen = 20, - .input = "\xA3\xBF\x52\x52\x65\x83\xBA\x81" - "\x03\x9B\x84\xFC\x44\x8C\xBB\x81" - "\x36\xE1\x78\xBB\xA5\x49\x3A\xD0" - "\xF0\x6B\x21\xAF\x98\xC0\x34\xDC" - "\x17\x17\x65\xAD", - .ilen = 36, - }, { - .key = "\x6C\x65\x67\x61\x6C\x69\x7A\x65" - "\x6D\x61\x72\x69\x6A\x75\x61\x6E" - "\x61\x61\x6E\x64\x64\x6F\x69\x74" - "\x62\x65\x66\x6F\x72\x65\x69\x61" - "\x74\x75\x72", - .klen = 35, - .iv = "\x33\x30\x21\x69\x67\x65\x74\x6D", - .result = "\x45\x00\x00\x30\xDA\x3A\x00\x00" - "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" - "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01", - .rlen = 52, - .assoc = "\x79\x6B\x69\x63\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\x33\x30\x21\x69" - "\x67\x65\x74\x6D", - .alen = 20, - .input = "\x96\xFD\x86\xF8\xD1\x98\xFF\x10" - "\xAB\x8C\xDA\x8A\x5A\x08\x38\x1A" - "\x48\x59\x80\x18\x1A\x18\x1A\x04" - "\xC9\x0D\xE3\xE7\x0E\xA4\x0B\x75" - "\x92\x9C\x52\x5C\x0B\xFB\xF8\xAF" - "\x16\xC3\x35\xA8\xE7\xCE\x84\x04" - "\xEB\x40\x6B\x7A\x8E\x75\xBB\x42" - "\xE0\x63\x4B\x21\x44\xA2\x2B\x2B" - "\x39\xDB\xC8\xDC", - .ilen = 68, - }, { - .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" - "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" - "\x57\x69\x0E", - .klen = 19, - .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", - .result = "\x45\x00\x00\x30\xDA\x3A\x00\x00" - "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" - "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" - "\x02\x00\x07\x00\x61\x62\x63\x64" - "\x65\x66\x67\x68\x69\x6A\x6B\x6C" - "\x6D\x6E\x6F\x70\x71\x72\x73\x74" - "\x01\x02\x02\x01", - .rlen = 52, - .assoc = "\x3F\x7E\xF6\x42\x10\x10\x10\x10" - "\x10\x10\x10\x10\x4E\x28\x00\x00" - "\xA2\xFC\xA1\xA3", - .alen = 20, - .input = "\x6A\x6B\x45\x27\x3F\x9E\x52\xF6" - "\x10\x60\x54\x25\xEB\x80\x04\x93" - "\xCA\x1B\x23\x97\xCB\x21\x2E\x01" - "\xA2\xE7\x95\x41\x30\xE4\x4B\x1B" - "\x79\x01\x58\x50\x01\x06\xE1\xE0" - "\x2C\x83\x79\xD3\xDE\x46\x97\x1A" - "\x44\xCC\x90\xBF\x00\x94\x94\x92" - "\x20\x17\x0C\x1B\x55\xDE\x7E\x68" - "\xF4\x95\x5D\x4F", - .ilen = 68, - }, { - .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" - "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" - "\x22\x43\x3C", - .klen = 19, - .iv = "\x48\x55\xEC\x7D\x3A\x23\x4B\xFD", - .result = "\x08\x00\xC6\xCD\x02\x00\x07\x00" - "\x61\x62\x63\x64\x65\x66\x67\x68" - "\x69\x6A\x6B\x6C\x6D\x6E\x6F\x70" - "\x71\x72\x73\x74\x01\x02\x02\x01", - .rlen = 32, - .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" - "\x00\x00\x00\x07\x48\x55\xEC\x7D" - "\x3A\x23\x4B\xFD", - .alen = 20, - .input = "\x67\xE9\x28\xB3\x1C\xA4\x6D\x02" - "\xF0\xB5\x37\xB6\x6B\x2F\xF5\x4F" - "\xF8\xA3\x4C\x53\xB8\x12\x09\xBF" - "\x58\x7D\xCF\x29\xA3\x41\x68\x6B" - "\xCE\xE8\x79\x85\x3C\xB0\x3A\x8F" - "\x16\xB0\xA1\x26\xC9\xBC\xBC\xA6", - .ilen = 48, - } -}; - -/* - * ChaCha20-Poly1305 AEAD test vectors from RFC7539 2.8.2./A.5. - */ -static const struct aead_testvec rfc7539_enc_tv_template[] = { - { - .key = "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f", - .klen = 32, - .iv = "\x07\x00\x00\x00\x40\x41\x42\x43" - "\x44\x45\x46\x47", - .assoc = "\x50\x51\x52\x53\xc0\xc1\xc2\xc3" - "\xc4\xc5\xc6\xc7", - .alen = 12, - .input = "\x4c\x61\x64\x69\x65\x73\x20\x61" - "\x6e\x64\x20\x47\x65\x6e\x74\x6c" - "\x65\x6d\x65\x6e\x20\x6f\x66\x20" - "\x74\x68\x65\x20\x63\x6c\x61\x73" - "\x73\x20\x6f\x66\x20\x27\x39\x39" - "\x3a\x20\x49\x66\x20\x49\x20\x63" - "\x6f\x75\x6c\x64\x20\x6f\x66\x66" - "\x65\x72\x20\x79\x6f\x75\x20\x6f" - "\x6e\x6c\x79\x20\x6f\x6e\x65\x20" - "\x74\x69\x70\x20\x66\x6f\x72\x20" - "\x74\x68\x65\x20\x66\x75\x74\x75" - "\x72\x65\x2c\x20\x73\x75\x6e\x73" - "\x63\x72\x65\x65\x6e\x20\x77\x6f" - "\x75\x6c\x64\x20\x62\x65\x20\x69" - "\x74\x2e", - .ilen = 114, - .result = "\xd3\x1a\x8d\x34\x64\x8e\x60\xdb" - "\x7b\x86\xaf\xbc\x53\xef\x7e\xc2" - "\xa4\xad\xed\x51\x29\x6e\x08\xfe" - "\xa9\xe2\xb5\xa7\x36\xee\x62\xd6" - "\x3d\xbe\xa4\x5e\x8c\xa9\x67\x12" - "\x82\xfa\xfb\x69\xda\x92\x72\x8b" - "\x1a\x71\xde\x0a\x9e\x06\x0b\x29" - "\x05\xd6\xa5\xb6\x7e\xcd\x3b\x36" - "\x92\xdd\xbd\x7f\x2d\x77\x8b\x8c" - "\x98\x03\xae\xe3\x28\x09\x1b\x58" - "\xfa\xb3\x24\xe4\xfa\xd6\x75\x94" - "\x55\x85\x80\x8b\x48\x31\xd7\xbc" - "\x3f\xf4\xde\xf0\x8e\x4b\x7a\x9d" - "\xe5\x76\xd2\x65\x86\xce\xc6\x4b" - "\x61\x16\x1a\xe1\x0b\x59\x4f\x09" - "\xe2\x6a\x7e\x90\x2e\xcb\xd0\x60" - "\x06\x91", - .rlen = 130, - }, { - .key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a" - "\xf3\x33\x88\x86\x04\xf6\xb5\xf0" - "\x47\x39\x17\xc1\x40\x2b\x80\x09" - "\x9d\xca\x5c\xbc\x20\x70\x75\xc0", - .klen = 32, - .iv = "\x00\x00\x00\x00\x01\x02\x03\x04" - "\x05\x06\x07\x08", - .assoc = "\xf3\x33\x88\x86\x00\x00\x00\x00" - "\x00\x00\x4e\x91", - .alen = 12, - .input = "\x49\x6e\x74\x65\x72\x6e\x65\x74" - "\x2d\x44\x72\x61\x66\x74\x73\x20" - "\x61\x72\x65\x20\x64\x72\x61\x66" - "\x74\x20\x64\x6f\x63\x75\x6d\x65" - "\x6e\x74\x73\x20\x76\x61\x6c\x69" - "\x64\x20\x66\x6f\x72\x20\x61\x20" - "\x6d\x61\x78\x69\x6d\x75\x6d\x20" - "\x6f\x66\x20\x73\x69\x78\x20\x6d" - "\x6f\x6e\x74\x68\x73\x20\x61\x6e" - "\x64\x20\x6d\x61\x79\x20\x62\x65" - "\x20\x75\x70\x64\x61\x74\x65\x64" - "\x2c\x20\x72\x65\x70\x6c\x61\x63" - "\x65\x64\x2c\x20\x6f\x72\x20\x6f" - "\x62\x73\x6f\x6c\x65\x74\x65\x64" - "\x20\x62\x79\x20\x6f\x74\x68\x65" - "\x72\x20\x64\x6f\x63\x75\x6d\x65" - "\x6e\x74\x73\x20\x61\x74\x20\x61" - "\x6e\x79\x20\x74\x69\x6d\x65\x2e" - "\x20\x49\x74\x20\x69\x73\x20\x69" - "\x6e\x61\x70\x70\x72\x6f\x70\x72" - "\x69\x61\x74\x65\x20\x74\x6f\x20" - "\x75\x73\x65\x20\x49\x6e\x74\x65" - "\x72\x6e\x65\x74\x2d\x44\x72\x61" - "\x66\x74\x73\x20\x61\x73\x20\x72" - "\x65\x66\x65\x72\x65\x6e\x63\x65" - "\x20\x6d\x61\x74\x65\x72\x69\x61" - "\x6c\x20\x6f\x72\x20\x74\x6f\x20" - "\x63\x69\x74\x65\x20\x74\x68\x65" - "\x6d\x20\x6f\x74\x68\x65\x72\x20" - "\x74\x68\x61\x6e\x20\x61\x73\x20" - "\x2f\xe2\x80\x9c\x77\x6f\x72\x6b" - "\x20\x69\x6e\x20\x70\x72\x6f\x67" - "\x72\x65\x73\x73\x2e\x2f\xe2\x80" - "\x9d", - .ilen = 265, - .result = "\x64\xa0\x86\x15\x75\x86\x1a\xf4" - "\x60\xf0\x62\xc7\x9b\xe6\x43\xbd" - "\x5e\x80\x5c\xfd\x34\x5c\xf3\x89" - "\xf1\x08\x67\x0a\xc7\x6c\x8c\xb2" - "\x4c\x6c\xfc\x18\x75\x5d\x43\xee" - "\xa0\x9e\xe9\x4e\x38\x2d\x26\xb0" - "\xbd\xb7\xb7\x3c\x32\x1b\x01\x00" - "\xd4\xf0\x3b\x7f\x35\x58\x94\xcf" - "\x33\x2f\x83\x0e\x71\x0b\x97\xce" - "\x98\xc8\xa8\x4a\xbd\x0b\x94\x81" - "\x14\xad\x17\x6e\x00\x8d\x33\xbd" - "\x60\xf9\x82\xb1\xff\x37\xc8\x55" - "\x97\x97\xa0\x6e\xf4\xf0\xef\x61" - "\xc1\x86\x32\x4e\x2b\x35\x06\x38" - "\x36\x06\x90\x7b\x6a\x7c\x02\xb0" - "\xf9\xf6\x15\x7b\x53\xc8\x67\xe4" - "\xb9\x16\x6c\x76\x7b\x80\x4d\x46" - "\xa5\x9b\x52\x16\xcd\xe7\xa4\xe9" - "\x90\x40\xc5\xa4\x04\x33\x22\x5e" - "\xe2\x82\xa1\xb0\xa0\x6c\x52\x3e" - "\xaf\x45\x34\xd7\xf8\x3f\xa1\x15" - "\x5b\x00\x47\x71\x8c\xbc\x54\x6a" - "\x0d\x07\x2b\x04\xb3\x56\x4e\xea" - "\x1b\x42\x22\x73\xf5\x48\x27\x1a" - "\x0b\xb2\x31\x60\x53\xfa\x76\x99" - "\x19\x55\xeb\xd6\x31\x59\x43\x4e" - "\xce\xbb\x4e\x46\x6d\xae\x5a\x10" - "\x73\xa6\x72\x76\x27\x09\x7a\x10" - "\x49\xe6\x17\xd9\x1d\x36\x10\x94" - "\xfa\x68\xf0\xff\x77\x98\x71\x30" - "\x30\x5b\xea\xba\x2e\xda\x04\xdf" - "\x99\x7b\x71\x4d\x6c\x6f\x2c\x29" - "\xa6\xad\x5c\xb4\x02\x2b\x02\x70" - "\x9b\xee\xad\x9d\x67\x89\x0c\xbb" - "\x22\x39\x23\x36\xfe\xa1\x85\x1f" - "\x38", - .rlen = 281, - }, -}; - -static const struct aead_testvec rfc7539_dec_tv_template[] = { - { - .key = "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f", - .klen = 32, - .iv = "\x07\x00\x00\x00\x40\x41\x42\x43" - "\x44\x45\x46\x47", - .assoc = "\x50\x51\x52\x53\xc0\xc1\xc2\xc3" - "\xc4\xc5\xc6\xc7", - .alen = 12, - .input = "\xd3\x1a\x8d\x34\x64\x8e\x60\xdb" - "\x7b\x86\xaf\xbc\x53\xef\x7e\xc2" - "\xa4\xad\xed\x51\x29\x6e\x08\xfe" - "\xa9\xe2\xb5\xa7\x36\xee\x62\xd6" - "\x3d\xbe\xa4\x5e\x8c\xa9\x67\x12" - "\x82\xfa\xfb\x69\xda\x92\x72\x8b" - "\x1a\x71\xde\x0a\x9e\x06\x0b\x29" - "\x05\xd6\xa5\xb6\x7e\xcd\x3b\x36" - "\x92\xdd\xbd\x7f\x2d\x77\x8b\x8c" - "\x98\x03\xae\xe3\x28\x09\x1b\x58" - "\xfa\xb3\x24\xe4\xfa\xd6\x75\x94" - "\x55\x85\x80\x8b\x48\x31\xd7\xbc" - "\x3f\xf4\xde\xf0\x8e\x4b\x7a\x9d" - "\xe5\x76\xd2\x65\x86\xce\xc6\x4b" - "\x61\x16\x1a\xe1\x0b\x59\x4f\x09" - "\xe2\x6a\x7e\x90\x2e\xcb\xd0\x60" - "\x06\x91", - .ilen = 130, - .result = "\x4c\x61\x64\x69\x65\x73\x20\x61" - "\x6e\x64\x20\x47\x65\x6e\x74\x6c" - "\x65\x6d\x65\x6e\x20\x6f\x66\x20" - "\x74\x68\x65\x20\x63\x6c\x61\x73" - "\x73\x20\x6f\x66\x20\x27\x39\x39" - "\x3a\x20\x49\x66\x20\x49\x20\x63" - "\x6f\x75\x6c\x64\x20\x6f\x66\x66" - "\x65\x72\x20\x79\x6f\x75\x20\x6f" - "\x6e\x6c\x79\x20\x6f\x6e\x65\x20" - "\x74\x69\x70\x20\x66\x6f\x72\x20" - "\x74\x68\x65\x20\x66\x75\x74\x75" - "\x72\x65\x2c\x20\x73\x75\x6e\x73" - "\x63\x72\x65\x65\x6e\x20\x77\x6f" - "\x75\x6c\x64\x20\x62\x65\x20\x69" - "\x74\x2e", - .rlen = 114, - }, { - .key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a" - "\xf3\x33\x88\x86\x04\xf6\xb5\xf0" - "\x47\x39\x17\xc1\x40\x2b\x80\x09" - "\x9d\xca\x5c\xbc\x20\x70\x75\xc0", - .klen = 32, - .iv = "\x00\x00\x00\x00\x01\x02\x03\x04" - "\x05\x06\x07\x08", - .assoc = "\xf3\x33\x88\x86\x00\x00\x00\x00" - "\x00\x00\x4e\x91", - .alen = 12, - .input = "\x64\xa0\x86\x15\x75\x86\x1a\xf4" - "\x60\xf0\x62\xc7\x9b\xe6\x43\xbd" - "\x5e\x80\x5c\xfd\x34\x5c\xf3\x89" - "\xf1\x08\x67\x0a\xc7\x6c\x8c\xb2" - "\x4c\x6c\xfc\x18\x75\x5d\x43\xee" - "\xa0\x9e\xe9\x4e\x38\x2d\x26\xb0" - "\xbd\xb7\xb7\x3c\x32\x1b\x01\x00" - "\xd4\xf0\x3b\x7f\x35\x58\x94\xcf" - "\x33\x2f\x83\x0e\x71\x0b\x97\xce" - "\x98\xc8\xa8\x4a\xbd\x0b\x94\x81" - "\x14\xad\x17\x6e\x00\x8d\x33\xbd" - "\x60\xf9\x82\xb1\xff\x37\xc8\x55" - "\x97\x97\xa0\x6e\xf4\xf0\xef\x61" - "\xc1\x86\x32\x4e\x2b\x35\x06\x38" - "\x36\x06\x90\x7b\x6a\x7c\x02\xb0" - "\xf9\xf6\x15\x7b\x53\xc8\x67\xe4" - "\xb9\x16\x6c\x76\x7b\x80\x4d\x46" - "\xa5\x9b\x52\x16\xcd\xe7\xa4\xe9" - "\x90\x40\xc5\xa4\x04\x33\x22\x5e" - "\xe2\x82\xa1\xb0\xa0\x6c\x52\x3e" - "\xaf\x45\x34\xd7\xf8\x3f\xa1\x15" - "\x5b\x00\x47\x71\x8c\xbc\x54\x6a" - "\x0d\x07\x2b\x04\xb3\x56\x4e\xea" - "\x1b\x42\x22\x73\xf5\x48\x27\x1a" - "\x0b\xb2\x31\x60\x53\xfa\x76\x99" - "\x19\x55\xeb\xd6\x31\x59\x43\x4e" - "\xce\xbb\x4e\x46\x6d\xae\x5a\x10" - "\x73\xa6\x72\x76\x27\x09\x7a\x10" - "\x49\xe6\x17\xd9\x1d\x36\x10\x94" - "\xfa\x68\xf0\xff\x77\x98\x71\x30" - "\x30\x5b\xea\xba\x2e\xda\x04\xdf" - "\x99\x7b\x71\x4d\x6c\x6f\x2c\x29" - "\xa6\xad\x5c\xb4\x02\x2b\x02\x70" - "\x9b\xee\xad\x9d\x67\x89\x0c\xbb" - "\x22\x39\x23\x36\xfe\xa1\x85\x1f" - "\x38", - .ilen = 281, - .result = "\x49\x6e\x74\x65\x72\x6e\x65\x74" - "\x2d\x44\x72\x61\x66\x74\x73\x20" - "\x61\x72\x65\x20\x64\x72\x61\x66" - "\x74\x20\x64\x6f\x63\x75\x6d\x65" - "\x6e\x74\x73\x20\x76\x61\x6c\x69" - "\x64\x20\x66\x6f\x72\x20\x61\x20" - "\x6d\x61\x78\x69\x6d\x75\x6d\x20" - "\x6f\x66\x20\x73\x69\x78\x20\x6d" - "\x6f\x6e\x74\x68\x73\x20\x61\x6e" - "\x64\x20\x6d\x61\x79\x20\x62\x65" - "\x20\x75\x70\x64\x61\x74\x65\x64" - "\x2c\x20\x72\x65\x70\x6c\x61\x63" - "\x65\x64\x2c\x20\x6f\x72\x20\x6f" - "\x62\x73\x6f\x6c\x65\x74\x65\x64" - "\x20\x62\x79\x20\x6f\x74\x68\x65" - "\x72\x20\x64\x6f\x63\x75\x6d\x65" - "\x6e\x74\x73\x20\x61\x74\x20\x61" - "\x6e\x79\x20\x74\x69\x6d\x65\x2e" - "\x20\x49\x74\x20\x69\x73\x20\x69" - "\x6e\x61\x70\x70\x72\x6f\x70\x72" - "\x69\x61\x74\x65\x20\x74\x6f\x20" - "\x75\x73\x65\x20\x49\x6e\x74\x65" - "\x72\x6e\x65\x74\x2d\x44\x72\x61" - "\x66\x74\x73\x20\x61\x73\x20\x72" - "\x65\x66\x65\x72\x65\x6e\x63\x65" - "\x20\x6d\x61\x74\x65\x72\x69\x61" - "\x6c\x20\x6f\x72\x20\x74\x6f\x20" - "\x63\x69\x74\x65\x20\x74\x68\x65" - "\x6d\x20\x6f\x74\x68\x65\x72\x20" - "\x74\x68\x61\x6e\x20\x61\x73\x20" - "\x2f\xe2\x80\x9c\x77\x6f\x72\x6b" - "\x20\x69\x6e\x20\x70\x72\x6f\x67" - "\x72\x65\x73\x73\x2e\x2f\xe2\x80" - "\x9d", - .rlen = 265, - }, -}; - -/* - * draft-irtf-cfrg-chacha20-poly1305 - */ -static const struct aead_testvec rfc7539esp_enc_tv_template[] = { - { - .key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a" - "\xf3\x33\x88\x86\x04\xf6\xb5\xf0" - "\x47\x39\x17\xc1\x40\x2b\x80\x09" - "\x9d\xca\x5c\xbc\x20\x70\x75\xc0" - "\x00\x00\x00\x00", - .klen = 36, - .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", - .assoc = "\xf3\x33\x88\x86\x00\x00\x00\x00" - "\x00\x00\x4e\x91\x01\x02\x03\x04" - "\x05\x06\x07\x08", - .alen = 20, - .input = "\x49\x6e\x74\x65\x72\x6e\x65\x74" - "\x2d\x44\x72\x61\x66\x74\x73\x20" - "\x61\x72\x65\x20\x64\x72\x61\x66" - "\x74\x20\x64\x6f\x63\x75\x6d\x65" - "\x6e\x74\x73\x20\x76\x61\x6c\x69" - "\x64\x20\x66\x6f\x72\x20\x61\x20" - "\x6d\x61\x78\x69\x6d\x75\x6d\x20" - "\x6f\x66\x20\x73\x69\x78\x20\x6d" - "\x6f\x6e\x74\x68\x73\x20\x61\x6e" - "\x64\x20\x6d\x61\x79\x20\x62\x65" - "\x20\x75\x70\x64\x61\x74\x65\x64" - "\x2c\x20\x72\x65\x70\x6c\x61\x63" - "\x65\x64\x2c\x20\x6f\x72\x20\x6f" - "\x62\x73\x6f\x6c\x65\x74\x65\x64" - "\x20\x62\x79\x20\x6f\x74\x68\x65" - "\x72\x20\x64\x6f\x63\x75\x6d\x65" - "\x6e\x74\x73\x20\x61\x74\x20\x61" - "\x6e\x79\x20\x74\x69\x6d\x65\x2e" - "\x20\x49\x74\x20\x69\x73\x20\x69" - "\x6e\x61\x70\x70\x72\x6f\x70\x72" - "\x69\x61\x74\x65\x20\x74\x6f\x20" - "\x75\x73\x65\x20\x49\x6e\x74\x65" - "\x72\x6e\x65\x74\x2d\x44\x72\x61" - "\x66\x74\x73\x20\x61\x73\x20\x72" - "\x65\x66\x65\x72\x65\x6e\x63\x65" - "\x20\x6d\x61\x74\x65\x72\x69\x61" - "\x6c\x20\x6f\x72\x20\x74\x6f\x20" - "\x63\x69\x74\x65\x20\x74\x68\x65" - "\x6d\x20\x6f\x74\x68\x65\x72\x20" - "\x74\x68\x61\x6e\x20\x61\x73\x20" - "\x2f\xe2\x80\x9c\x77\x6f\x72\x6b" - "\x20\x69\x6e\x20\x70\x72\x6f\x67" - "\x72\x65\x73\x73\x2e\x2f\xe2\x80" - "\x9d", - .ilen = 265, - .result = "\x64\xa0\x86\x15\x75\x86\x1a\xf4" - "\x60\xf0\x62\xc7\x9b\xe6\x43\xbd" - "\x5e\x80\x5c\xfd\x34\x5c\xf3\x89" - "\xf1\x08\x67\x0a\xc7\x6c\x8c\xb2" - "\x4c\x6c\xfc\x18\x75\x5d\x43\xee" - "\xa0\x9e\xe9\x4e\x38\x2d\x26\xb0" - "\xbd\xb7\xb7\x3c\x32\x1b\x01\x00" - "\xd4\xf0\x3b\x7f\x35\x58\x94\xcf" - "\x33\x2f\x83\x0e\x71\x0b\x97\xce" - "\x98\xc8\xa8\x4a\xbd\x0b\x94\x81" - "\x14\xad\x17\x6e\x00\x8d\x33\xbd" - "\x60\xf9\x82\xb1\xff\x37\xc8\x55" - "\x97\x97\xa0\x6e\xf4\xf0\xef\x61" - "\xc1\x86\x32\x4e\x2b\x35\x06\x38" - "\x36\x06\x90\x7b\x6a\x7c\x02\xb0" - "\xf9\xf6\x15\x7b\x53\xc8\x67\xe4" - "\xb9\x16\x6c\x76\x7b\x80\x4d\x46" - "\xa5\x9b\x52\x16\xcd\xe7\xa4\xe9" - "\x90\x40\xc5\xa4\x04\x33\x22\x5e" - "\xe2\x82\xa1\xb0\xa0\x6c\x52\x3e" - "\xaf\x45\x34\xd7\xf8\x3f\xa1\x15" - "\x5b\x00\x47\x71\x8c\xbc\x54\x6a" - "\x0d\x07\x2b\x04\xb3\x56\x4e\xea" - "\x1b\x42\x22\x73\xf5\x48\x27\x1a" - "\x0b\xb2\x31\x60\x53\xfa\x76\x99" - "\x19\x55\xeb\xd6\x31\x59\x43\x4e" - "\xce\xbb\x4e\x46\x6d\xae\x5a\x10" - "\x73\xa6\x72\x76\x27\x09\x7a\x10" - "\x49\xe6\x17\xd9\x1d\x36\x10\x94" - "\xfa\x68\xf0\xff\x77\x98\x71\x30" - "\x30\x5b\xea\xba\x2e\xda\x04\xdf" - "\x99\x7b\x71\x4d\x6c\x6f\x2c\x29" - "\xa6\xad\x5c\xb4\x02\x2b\x02\x70" - "\x9b\xee\xad\x9d\x67\x89\x0c\xbb" - "\x22\x39\x23\x36\xfe\xa1\x85\x1f" - "\x38", - .rlen = 281, - }, -}; - -static const struct aead_testvec rfc7539esp_dec_tv_template[] = { - { - .key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a" - "\xf3\x33\x88\x86\x04\xf6\xb5\xf0" - "\x47\x39\x17\xc1\x40\x2b\x80\x09" - "\x9d\xca\x5c\xbc\x20\x70\x75\xc0" - "\x00\x00\x00\x00", - .klen = 36, - .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", - .assoc = "\xf3\x33\x88\x86\x00\x00\x00\x00" - "\x00\x00\x4e\x91\x01\x02\x03\x04" - "\x05\x06\x07\x08", - .alen = 20, - .input = "\x64\xa0\x86\x15\x75\x86\x1a\xf4" - "\x60\xf0\x62\xc7\x9b\xe6\x43\xbd" - "\x5e\x80\x5c\xfd\x34\x5c\xf3\x89" - "\xf1\x08\x67\x0a\xc7\x6c\x8c\xb2" - "\x4c\x6c\xfc\x18\x75\x5d\x43\xee" - "\xa0\x9e\xe9\x4e\x38\x2d\x26\xb0" - "\xbd\xb7\xb7\x3c\x32\x1b\x01\x00" - "\xd4\xf0\x3b\x7f\x35\x58\x94\xcf" - "\x33\x2f\x83\x0e\x71\x0b\x97\xce" - "\x98\xc8\xa8\x4a\xbd\x0b\x94\x81" - "\x14\xad\x17\x6e\x00\x8d\x33\xbd" - "\x60\xf9\x82\xb1\xff\x37\xc8\x55" - "\x97\x97\xa0\x6e\xf4\xf0\xef\x61" - "\xc1\x86\x32\x4e\x2b\x35\x06\x38" - "\x36\x06\x90\x7b\x6a\x7c\x02\xb0" - "\xf9\xf6\x15\x7b\x53\xc8\x67\xe4" - "\xb9\x16\x6c\x76\x7b\x80\x4d\x46" - "\xa5\x9b\x52\x16\xcd\xe7\xa4\xe9" - "\x90\x40\xc5\xa4\x04\x33\x22\x5e" - "\xe2\x82\xa1\xb0\xa0\x6c\x52\x3e" - "\xaf\x45\x34\xd7\xf8\x3f\xa1\x15" - "\x5b\x00\x47\x71\x8c\xbc\x54\x6a" - "\x0d\x07\x2b\x04\xb3\x56\x4e\xea" - "\x1b\x42\x22\x73\xf5\x48\x27\x1a" - "\x0b\xb2\x31\x60\x53\xfa\x76\x99" - "\x19\x55\xeb\xd6\x31\x59\x43\x4e" - "\xce\xbb\x4e\x46\x6d\xae\x5a\x10" - "\x73\xa6\x72\x76\x27\x09\x7a\x10" - "\x49\xe6\x17\xd9\x1d\x36\x10\x94" - "\xfa\x68\xf0\xff\x77\x98\x71\x30" - "\x30\x5b\xea\xba\x2e\xda\x04\xdf" - "\x99\x7b\x71\x4d\x6c\x6f\x2c\x29" - "\xa6\xad\x5c\xb4\x02\x2b\x02\x70" - "\x9b\xee\xad\x9d\x67\x89\x0c\xbb" - "\x22\x39\x23\x36\xfe\xa1\x85\x1f" - "\x38", - .ilen = 281, - .result = "\x49\x6e\x74\x65\x72\x6e\x65\x74" - "\x2d\x44\x72\x61\x66\x74\x73\x20" - "\x61\x72\x65\x20\x64\x72\x61\x66" - "\x74\x20\x64\x6f\x63\x75\x6d\x65" - "\x6e\x74\x73\x20\x76\x61\x6c\x69" - "\x64\x20\x66\x6f\x72\x20\x61\x20" - "\x6d\x61\x78\x69\x6d\x75\x6d\x20" - "\x6f\x66\x20\x73\x69\x78\x20\x6d" - "\x6f\x6e\x74\x68\x73\x20\x61\x6e" - "\x64\x20\x6d\x61\x79\x20\x62\x65" - "\x20\x75\x70\x64\x61\x74\x65\x64" - "\x2c\x20\x72\x65\x70\x6c\x61\x63" - "\x65\x64\x2c\x20\x6f\x72\x20\x6f" - "\x62\x73\x6f\x6c\x65\x74\x65\x64" - "\x20\x62\x79\x20\x6f\x74\x68\x65" - "\x72\x20\x64\x6f\x63\x75\x6d\x65" - "\x6e\x74\x73\x20\x61\x74\x20\x61" - "\x6e\x79\x20\x74\x69\x6d\x65\x2e" - "\x20\x49\x74\x20\x69\x73\x20\x69" - "\x6e\x61\x70\x70\x72\x6f\x70\x72" - "\x69\x61\x74\x65\x20\x74\x6f\x20" - "\x75\x73\x65\x20\x49\x6e\x74\x65" - "\x72\x6e\x65\x74\x2d\x44\x72\x61" - "\x66\x74\x73\x20\x61\x73\x20\x72" - "\x65\x66\x65\x72\x65\x6e\x63\x65" - "\x20\x6d\x61\x74\x65\x72\x69\x61" - "\x6c\x20\x6f\x72\x20\x74\x6f\x20" - "\x63\x69\x74\x65\x20\x74\x68\x65" - "\x6d\x20\x6f\x74\x68\x65\x72\x20" - "\x74\x68\x61\x6e\x20\x61\x73\x20" - "\x2f\xe2\x80\x9c\x77\x6f\x72\x6b" - "\x20\x69\x6e\x20\x70\x72\x6f\x67" - "\x72\x65\x73\x73\x2e\x2f\xe2\x80" - "\x9d", - .rlen = 265, - }, -}; - -/* - * All key wrapping test vectors taken from - * http://csrc.nist.gov/groups/STM/cavp/documents/mac/kwtestvectors.zip - * - * Note: as documented in keywrap.c, the ivout for encryption is the first - * semiblock of the ciphertext from the test vector. For decryption, iv is - * the first semiblock of the ciphertext. - */ -static const struct cipher_testvec aes_kw_enc_tv_template[] = { - { - .key = "\x75\x75\xda\x3a\x93\x60\x7c\xc2" - "\xbf\xd8\xce\xc7\xaa\xdf\xd9\xa6", - .klen = 16, - .input = "\x42\x13\x6d\x3c\x38\x4a\x3e\xea" - "\xc9\x5a\x06\x6f\xd2\x8f\xed\x3f", - .ilen = 16, - .result = "\xf6\x85\x94\x81\x6f\x64\xca\xa3" - "\xf5\x6f\xab\xea\x25\x48\xf5\xfb", - .rlen = 16, - .iv_out = "\x03\x1f\x6b\xd7\xe6\x1e\x64\x3d", - }, -}; - -static const struct cipher_testvec aes_kw_dec_tv_template[] = { - { - .key = "\x80\xaa\x99\x73\x27\xa4\x80\x6b" - "\x6a\x7a\x41\xa5\x2b\x86\xc3\x71" - "\x03\x86\xf9\x32\x78\x6e\xf7\x96" - "\x76\xfa\xfb\x90\xb8\x26\x3c\x5f", - .klen = 32, - .input = "\xd3\x3d\x3d\x97\x7b\xf0\xa9\x15" - "\x59\xf9\x9c\x8a\xcd\x29\x3d\x43", - .ilen = 16, - .result = "\x0a\x25\x6b\xa7\x5c\xfa\x03\xaa" - "\xa0\x2b\xa9\x42\x03\xf1\x5b\xaa", - .rlen = 16, - .iv = "\x42\x3c\x96\x0d\x8a\x2a\xc4\xc1", - }, -}; - -/* - * ANSI X9.31 Continuous Pseudo-Random Number Generator (AES mode) - * test vectors, taken from Appendix B.2.9 and B.2.10: - * http://csrc.nist.gov/groups/STM/cavp/documents/rng/RNGVS.pdf - * Only AES-128 is supported at this time. - */ -static const struct cprng_testvec ansi_cprng_aes_tv_template[] = { - { - .key = "\xf3\xb1\x66\x6d\x13\x60\x72\x42" - "\xed\x06\x1c\xab\xb8\xd4\x62\x02", - .klen = 16, - .dt = "\xe6\xb3\xbe\x78\x2a\x23\xfa\x62" - "\xd7\x1d\x4a\xfb\xb0\xe9\x22\xf9", - .dtlen = 16, - .v = "\x80\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .vlen = 16, - .result = "\x59\x53\x1e\xd1\x3b\xb0\xc0\x55" - "\x84\x79\x66\x85\xc1\x2f\x76\x41", - .rlen = 16, - .loops = 1, - }, { - .key = "\xf3\xb1\x66\x6d\x13\x60\x72\x42" - "\xed\x06\x1c\xab\xb8\xd4\x62\x02", - .klen = 16, - .dt = "\xe6\xb3\xbe\x78\x2a\x23\xfa\x62" - "\xd7\x1d\x4a\xfb\xb0\xe9\x22\xfa", - .dtlen = 16, - .v = "\xc0\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .vlen = 16, - .result = "\x7c\x22\x2c\xf4\xca\x8f\xa2\x4c" - "\x1c\x9c\xb6\x41\xa9\xf3\x22\x0d", - .rlen = 16, - .loops = 1, - }, { - .key = "\xf3\xb1\x66\x6d\x13\x60\x72\x42" - "\xed\x06\x1c\xab\xb8\xd4\x62\x02", - .klen = 16, - .dt = "\xe6\xb3\xbe\x78\x2a\x23\xfa\x62" - "\xd7\x1d\x4a\xfb\xb0\xe9\x22\xfb", - .dtlen = 16, - .v = "\xe0\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .vlen = 16, - .result = "\x8a\xaa\x00\x39\x66\x67\x5b\xe5" - "\x29\x14\x28\x81\xa9\x4d\x4e\xc7", - .rlen = 16, - .loops = 1, - }, { - .key = "\xf3\xb1\x66\x6d\x13\x60\x72\x42" - "\xed\x06\x1c\xab\xb8\xd4\x62\x02", - .klen = 16, - .dt = "\xe6\xb3\xbe\x78\x2a\x23\xfa\x62" - "\xd7\x1d\x4a\xfb\xb0\xe9\x22\xfc", - .dtlen = 16, - .v = "\xf0\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .vlen = 16, - .result = "\x88\xdd\xa4\x56\x30\x24\x23\xe5" - "\xf6\x9d\xa5\x7e\x7b\x95\xc7\x3a", - .rlen = 16, - .loops = 1, - }, { - .key = "\xf3\xb1\x66\x6d\x13\x60\x72\x42" - "\xed\x06\x1c\xab\xb8\xd4\x62\x02", - .klen = 16, - .dt = "\xe6\xb3\xbe\x78\x2a\x23\xfa\x62" - "\xd7\x1d\x4a\xfb\xb0\xe9\x22\xfd", - .dtlen = 16, - .v = "\xf8\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .vlen = 16, - .result = "\x05\x25\x92\x46\x61\x79\xd2\xcb" - "\x78\xc4\x0b\x14\x0a\x5a\x9a\xc8", - .rlen = 16, - .loops = 1, - }, { /* Monte Carlo Test */ - .key = "\x9f\x5b\x51\x20\x0b\xf3\x34\xb5" - "\xd8\x2b\xe8\xc3\x72\x55\xc8\x48", - .klen = 16, - .dt = "\x63\x76\xbb\xe5\x29\x02\xba\x3b" - "\x67\xc9\x25\xfa\x70\x1f\x11\xac", - .dtlen = 16, - .v = "\x57\x2c\x8e\x76\x87\x26\x47\x97" - "\x7e\x74\xfb\xdd\xc4\x95\x01\xd1", - .vlen = 16, - .result = "\x48\xe9\xbd\x0d\x06\xee\x18\xfb" - "\xe4\x57\x90\xd5\xc3\xfc\x9b\x73", - .rlen = 16, - .loops = 10000, - }, -}; - -/* - * SP800-90A DRBG Test vectors from - * http://csrc.nist.gov/groups/STM/cavp/documents/drbg/drbgtestvectors.zip - * - * Test vectors for DRBG with prediction resistance. All types of DRBGs - * (Hash, HMAC, CTR) are tested with all permutations of use cases (w/ and - * w/o personalization string, w/ and w/o additional input string). - */ -static const struct drbg_testvec drbg_pr_sha256_tv_template[] = { - { - .entropy = (unsigned char *) - "\x72\x88\x4c\xcd\x6c\x85\x57\x70\xf7\x0b\x8b\x86" - "\xc1\xeb\xd2\x4e\x36\x14\xab\x18\xc4\x9c\xc9\xcf" - "\x1a\xe8\xf7\x7b\x02\x49\x73\xd7\xf1\x42\x7d\xc6" - "\x3f\x29\x2d\xec\xd3\x66\x51\x3f\x1d\x8d\x5b\x4e", - .entropylen = 48, - .entpra = (unsigned char *) - "\x38\x9c\x91\xfa\xc2\xa3\x46\x89\x56\x08\x3f\x62" - "\x73\xd5\x22\xa9\x29\x63\x3a\x1d\xe5\x5d\x5e\x4f" - "\x67\xb0\x67\x7a\x5e\x9e\x0c\x62", - .entprb = (unsigned char *) - "\xb2\x8f\x36\xb2\xf6\x8d\x39\x13\xfa\x6c\x66\xcf" - "\x62\x8a\x7e\x8c\x12\x33\x71\x9c\x69\xe4\xa5\xf0" - "\x8c\xee\xeb\x9c\xf5\x31\x98\x31", - .entprlen = 32, - .expected = (unsigned char *) - "\x52\x7b\xa3\xad\x71\x77\xa4\x49\x42\x04\x61\xc7" - "\xf0\xaf\xa5\xfd\xd3\xb3\x0d\x6a\x61\xba\x35\x49" - "\xbb\xaa\xaf\xe4\x25\x7d\xb5\x48\xaf\x5c\x18\x3d" - "\x33\x8d\x9d\x45\xdf\x98\xd5\x94\xa8\xda\x92\xfe" - "\xc4\x3c\x94\x2a\xcf\x7f\x7b\xf2\xeb\x28\xa9\xf1" - "\xe0\x86\x30\xa8\xfe\xf2\x48\x90\x91\x0c\x75\xb5" - "\x3c\x00\xf0\x4d\x09\x4f\x40\xa7\xa2\x8c\x52\xdf" - "\x52\xef\x17\xbf\x3d\xd1\xa2\x31\xb4\xb8\xdc\xe6" - "\x5b\x0d\x1f\x78\x36\xb4\xe6\x4b\xa7\x11\x25\xd5" - "\x94\xc6\x97\x36\xab\xf0\xe5\x31\x28\x6a\xbb\xce" - "\x30\x81\xa6\x8f\x27\x14\xf8\x1c", - .expectedlen = 128, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\x5d\xf2\x14\xbc\xf6\xb5\x4e\x0b\xf0\x0d\x6f\x2d" - "\xe2\x01\x66\x7b\xd0\xa4\x73\xa4\x21\xdd\xb0\xc0" - "\x51\x79\x09\xf4\xea\xa9\x08\xfa\xa6\x67\xe0\xe1" - "\xd1\x88\xa8\xad\xee\x69\x74\xb3\x55\x06\x9b\xf6", - .entropylen = 48, - .entpra = (unsigned char *) - "\xef\x48\x06\xa2\xc2\x45\xf1\x44\xfa\x34\x2c\xeb" - "\x8d\x78\x3c\x09\x8f\x34\x72\x20\xf2\xe7\xfd\x13" - "\x76\x0a\xf6\xdc\x3c\xf5\xc0\x15", - .entprb = (unsigned char *) - "\x4b\xbe\xe5\x24\xed\x6a\x2d\x0c\xdb\x73\x5e\x09" - "\xf9\xad\x67\x7c\x51\x47\x8b\x6b\x30\x2a\xc6\xde" - "\x76\xaa\x55\x04\x8b\x0a\x72\x95", - .entprlen = 32, - .expected = (unsigned char *) - "\x3b\x14\x71\x99\xa1\xda\xa0\x42\xe6\xc8\x85\x32" - "\x70\x20\x32\x53\x9a\xbe\xd1\x1e\x15\xef\xfb\x4c" - "\x25\x6e\x19\x3a\xf0\xb9\xcb\xde\xf0\x3b\xc6\x18" - "\x4d\x85\x5a\x9b\xf1\xe3\xc2\x23\x03\x93\x08\xdb" - "\xa7\x07\x4b\x33\x78\x40\x4d\xeb\x24\xf5\x6e\x81" - "\x4a\x1b\x6e\xa3\x94\x52\x43\xb0\xaf\x2e\x21\xf4" - "\x42\x46\x8e\x90\xed\x34\x21\x75\xea\xda\x67\xb6" - "\xe4\xf6\xff\xc6\x31\x6c\x9a\x5a\xdb\xb3\x97\x13" - "\x09\xd3\x20\x98\x33\x2d\x6d\xd7\xb5\x6a\xa8\xa9" - "\x9a\x5b\xd6\x87\x52\xa1\x89\x2b\x4b\x9c\x64\x60" - "\x50\x47\xa3\x63\x81\x16\xaf\x19", - .expectedlen = 128, - .addtla = (unsigned char *) - "\xbe\x13\xdb\x2a\xe9\xa8\xfe\x09\x97\xe1\xce\x5d" - "\xe8\xbb\xc0\x7c\x4f\xcb\x62\x19\x3f\x0f\xd2\xad" - "\xa9\xd0\x1d\x59\x02\xc4\xff\x70", - .addtlb = (unsigned char *) - "\x6f\x96\x13\xe2\xa7\xf5\x6c\xfe\xdf\x66\xe3\x31" - "\x63\x76\xbf\x20\x27\x06\x49\xf1\xf3\x01\x77\x41" - "\x9f\xeb\xe4\x38\xfe\x67\x00\xcd", - .addtllen = 32, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\xc6\x1c\xaf\x83\xa2\x56\x38\xf9\xb0\xbc\xd9\x85" - "\xf5\x2e\xc4\x46\x9c\xe1\xb9\x40\x98\x70\x10\x72" - "\xd7\x7d\x15\x85\xa1\x83\x5a\x97\xdf\xc8\xa8\xe8" - "\x03\x4c\xcb\x70\x35\x8b\x90\x94\x46\x8a\x6e\xa1", - .entropylen = 48, - .entpra = (unsigned char *) - "\xc9\x05\xa4\xcf\x28\x80\x4b\x93\x0f\x8b\xc6\xf9" - "\x09\x41\x58\x74\xe9\xec\x28\xc7\x53\x0a\x73\x60" - "\xba\x0a\xde\x57\x5b\x4b\x9f\x29", - .entprb = (unsigned char *) - "\x4f\x31\xd2\xeb\xac\xfa\xa8\xe2\x01\x7d\xf3\xbd" - "\x42\xbd\x20\xa0\x30\x65\x74\xd5\x5d\xd2\xad\xa4" - "\xa9\xeb\x1f\x4d\xf6\xfd\xb8\x26", - .entprlen = 32, - .expected = (unsigned char *) - "\xf6\x13\x05\xcb\x83\x60\x16\x42\x49\x1d\xc6\x25" - "\x3b\x8c\x31\xa3\xbe\x8b\xbd\x1c\xe2\xec\x1d\xde" - "\xbb\xbf\xa1\xac\xa8\x9f\x50\xce\x69\xce\xef\xd5" - "\xd6\xf2\xef\x6a\xf7\x81\x38\xdf\xbc\xa7\x5a\xb9" - "\xb2\x42\x65\xab\xe4\x86\x8d\x2d\x9d\x59\x99\x2c" - "\x5a\x0d\x71\x55\x98\xa4\x45\xc2\x8d\xdb\x05\x5e" - "\x50\x21\xf7\xcd\xe8\x98\x43\xce\x57\x74\x63\x4c" - "\xf3\xb1\xa5\x14\x1e\x9e\x01\xeb\x54\xd9\x56\xae" - "\xbd\xb6\x6f\x1a\x47\x6b\x3b\x44\xe4\xa2\xe9\x3c" - "\x6c\x83\x12\x30\xb8\x78\x7f\x8e\x54\x82\xd4\xfe" - "\x90\x35\x0d\x4c\x4d\x85\xe7\x13", - .expectedlen = 128, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = (unsigned char *) - "\xa5\xbf\xac\x4f\x71\xa1\xbb\x67\x94\xc6\x50\xc7" - "\x2a\x45\x9e\x10\xa8\xed\xf7\x52\x4f\xfe\x21\x90" - "\xa4\x1b\xe1\xe2\x53\xcc\x61\x47", - .perslen = 32, - }, { - .entropy = (unsigned char *) - "\xb6\xc1\x8d\xdf\x99\x54\xbe\x95\x10\x48\xd9\xf6" - "\xd7\x48\xa8\x73\x2d\x74\xde\x1e\xde\x57\x7e\xf4" - "\x7b\x7b\x64\xef\x88\x7a\xa8\x10\x4b\xe1\xc1\x87" - "\xbb\x0b\xe1\x39\x39\x50\xaf\x68\x9c\xa2\xbf\x5e", - .entropylen = 48, - .entpra = (unsigned char *) - "\xdc\x81\x0a\x01\x58\xa7\x2e\xce\xee\x48\x8c\x7c" - "\x77\x9e\x3c\xf1\x17\x24\x7a\xbb\xab\x9f\xca\x12" - "\x19\xaf\x97\x2d\x5f\xf9\xff\xfc", - .entprb = (unsigned char *) - "\xaf\xfc\x4f\x98\x8b\x93\x95\xc1\xb5\x8b\x7f\x73" - "\x6d\xa6\xbe\x6d\x33\xeb\x2c\x82\xb1\xaf\xc1\xb6" - "\xb6\x05\xe2\x44\xaa\xfd\xe7\xdb", - .entprlen = 32, - .expected = (unsigned char *) - "\x51\x79\xde\x1c\x0f\x58\xf3\xf4\xc9\x57\x2e\x31" - "\xa7\x09\xa1\x53\x64\x63\xa2\xc5\x1d\x84\x88\x65" - "\x01\x1b\xc6\x16\x3c\x49\x5b\x42\x8e\x53\xf5\x18" - "\xad\x94\x12\x0d\x4f\x55\xcc\x45\x5c\x98\x0f\x42" - "\x28\x2f\x47\x11\xf9\xc4\x01\x97\x6b\xa0\x94\x50" - "\xa9\xd1\x5e\x06\x54\x3f\xdf\xbb\xc4\x98\xee\x8b" - "\xba\xa9\xfa\x49\xee\x1d\xdc\xfb\x50\xf6\x51\x9f" - "\x6c\x4a\x9a\x6f\x63\xa2\x7d\xad\xaf\x3a\x24\xa0" - "\xd9\x9f\x07\xeb\x15\xee\x26\xe0\xd5\x63\x39\xda" - "\x3c\x59\xd6\x33\x6c\x02\xe8\x05\x71\x46\x68\x44" - "\x63\x4a\x68\x72\xe9\xf5\x55\xfe", - .expectedlen = 128, - .addtla = (unsigned char *) - "\x15\x20\x2f\xf6\x98\x28\x63\xa2\xc4\x4e\xbb\x6c" - "\xb2\x25\x92\x61\x79\xc9\x22\xc4\x61\x54\x96\xff" - "\x4a\x85\xca\x80\xfe\x0d\x1c\xd0", - .addtlb = (unsigned char *) - "\xde\x29\x8e\x03\x42\x61\xa3\x28\x5e\xc8\x80\xc2" - "\x6d\xbf\xad\x13\xe1\x8d\x2a\xc7\xe8\xc7\x18\x89" - "\x42\x58\x9e\xd6\xcc\xad\x7b\x1e", - .addtllen = 32, - .pers = (unsigned char *) - "\x84\xc3\x73\x9e\xce\xb3\xbc\x89\xf7\x62\xb3\xe1" - "\xd7\x48\x45\x8a\xa9\xcc\xe9\xed\xd5\x81\x84\x52" - "\x82\x4c\xdc\x19\xb8\xf8\x92\x5c", - .perslen = 32, - }, -}; - -static const struct drbg_testvec drbg_pr_hmac_sha256_tv_template[] = { - { - .entropy = (unsigned char *) - "\x99\x69\xe5\x4b\x47\x03\xff\x31\x78\x5b\x87\x9a" - "\x7e\x5c\x0e\xae\x0d\x3e\x30\x95\x59\xe9\xfe\x96" - "\xb0\x67\x6d\x49\xd5\x91\xea\x4d\x07\xd2\x0d\x46" - "\xd0\x64\x75\x7d\x30\x23\xca\xc2\x37\x61\x27\xab", - .entropylen = 48, - .entpra = (unsigned char *) - "\xc6\x0f\x29\x99\x10\x0f\x73\x8c\x10\xf7\x47\x92" - "\x67\x6a\x3f\xc4\xa2\x62\xd1\x37\x21\x79\x80\x46" - "\xe2\x9a\x29\x51\x81\x56\x9f\x54", - .entprb = (unsigned char *) - "\xc1\x1d\x45\x24\xc9\x07\x1b\xd3\x09\x60\x15\xfc" - "\xf7\xbc\x24\xa6\x07\xf2\x2f\xa0\x65\xc9\x37\x65" - "\x8a\x2a\x77\xa8\x69\x90\x89\xf4", - .entprlen = 32, - .expected = (unsigned char *) - "\xab\xc0\x15\x85\x60\x94\x80\x3a\x93\x8d\xff\xd2" - "\x0d\xa9\x48\x43\x87\x0e\xf9\x35\xb8\x2c\xfe\xc1" - "\x77\x06\xb8\xf5\x51\xb8\x38\x50\x44\x23\x5d\xd4" - "\x4b\x59\x9f\x94\xb3\x9b\xe7\x8d\xd4\x76\xe0\xcf" - "\x11\x30\x9c\x99\x5a\x73\x34\xe0\xa7\x8b\x37\xbc" - "\x95\x86\x23\x50\x86\xfa\x3b\x63\x7b\xa9\x1c\xf8" - "\xfb\x65\xef\xa2\x2a\x58\x9c\x13\x75\x31\xaa\x7b" - "\x2d\x4e\x26\x07\xaa\xc2\x72\x92\xb0\x1c\x69\x8e" - "\x6e\x01\xae\x67\x9e\xb8\x7c\x01\xa8\x9c\x74\x22" - "\xd4\x37\x2d\x6d\x75\x4a\xba\xbb\x4b\xf8\x96\xfc" - "\xb1\xcd\x09\xd6\x92\xd0\x28\x3f", - .expectedlen = 128, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\xb9\x1f\xe9\xef\xdd\x9b\x7d\x20\xb6\xec\xe0\x2f" - "\xdb\x76\x24\xce\x41\xc8\x3a\x4a\x12\x7f\x3e\x2f" - "\xae\x05\x99\xea\xb5\x06\x71\x0d\x0c\x4c\xb4\x05" - "\x26\xc6\xbd\xf5\x7f\x2a\x3d\xf2\xb5\x49\x7b\xda", - .entropylen = 48, - .entpra = (unsigned char *) - "\xef\x67\x50\x9c\xa7\x7d\xdf\xb7\x2d\x81\x01\xa4" - "\x62\x81\x6a\x69\x5b\xb3\x37\x45\xa7\x34\x8e\x26" - "\x46\xd9\x26\xa2\x19\xd4\x94\x43", - .entprb = (unsigned char *) - "\x97\x75\x53\x53\xba\xb4\xa6\xb2\x91\x60\x71\x79" - "\xd1\x6b\x4a\x24\x9a\x34\x66\xcc\x33\xab\x07\x98" - "\x51\x78\x72\xb2\x79\xfd\x2c\xff", - .entprlen = 32, - .expected = (unsigned char *) - "\x9c\xdc\x63\x8a\x19\x23\x22\x66\x0c\xc5\xb9\xd7" - "\xfb\x2a\xb0\x31\xe3\x8a\x36\xa8\x5a\xa8\x14\xda" - "\x1e\xa9\xcc\xfe\xb8\x26\x44\x83\x9f\xf6\xff\xaa" - "\xc8\x98\xb8\x30\x35\x3b\x3d\x36\xd2\x49\xd4\x40" - "\x62\x0a\x65\x10\x76\x55\xef\xc0\x95\x9c\xa7\xda" - "\x3f\xcf\xb7\x7b\xc6\xe1\x28\x52\xfc\x0c\xe2\x37" - "\x0d\x83\xa7\x51\x4b\x31\x47\x3c\xe1\x3c\xae\x70" - "\x01\xc8\xa3\xd3\xc2\xac\x77\x9c\xd1\x68\x77\x9b" - "\x58\x27\x3b\xa5\x0f\xc2\x7a\x8b\x04\x65\x62\xd5" - "\xe8\xd6\xfe\x2a\xaf\xd3\xd3\xfe\xbd\x18\xfb\xcd" - "\xcd\x66\xb5\x01\x69\x66\xa0\x3c", - .expectedlen = 128, - .addtla = (unsigned char *) - "\x17\xc1\x56\xcb\xcc\x50\xd6\x03\x7d\x45\x76\xa3" - "\x75\x76\xc1\x4a\x66\x1b\x2e\xdf\xb0\x2e\x7d\x56" - "\x6d\x99\x3b\xc6\x58\xda\x03\xf6", - .addtlb = (unsigned char *) - "\x7c\x7b\x4a\x4b\x32\x5e\x6f\x67\x34\xf5\x21\x4c" - "\xf9\x96\xf9\xbf\x1c\x8c\x81\xd3\x9b\x60\x6a\x44" - "\xc6\x03\xa2\xfb\x13\x20\x19\xb7", - .addtllen = 32, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\x13\x54\x96\xfc\x1b\x7d\x28\xf3\x18\xc9\xa7\x89" - "\xb6\xb3\xc8\x72\xac\x00\xd4\x59\x36\x25\x05\xaf" - "\xa5\xdb\x96\xcb\x3c\x58\x46\x87\xa5\xaa\xbf\x20" - "\x3b\xfe\x23\x0e\xd1\xc7\x41\x0f\x3f\xc9\xb3\x67", - .entropylen = 48, - .entpra = (unsigned char *) - "\xe2\xbd\xb7\x48\x08\x06\xf3\xe1\x93\x3c\xac\x79" - "\xa7\x2b\x11\xda\xe3\x2e\xe1\x91\xa5\x02\x19\x57" - "\x20\x28\xad\xf2\x60\xd7\xcd\x45", - .entprb = (unsigned char *) - "\x8b\xd4\x69\xfc\xff\x59\x95\x95\xc6\x51\xde\x71" - "\x68\x5f\xfc\xf9\x4a\xab\xec\x5a\xcb\xbe\xd3\x66" - "\x1f\xfa\x74\xd3\xac\xa6\x74\x60", - .entprlen = 32, - .expected = (unsigned char *) - "\x1f\x9e\xaf\xe4\xd2\x46\xb7\x47\x41\x4c\x65\x99" - "\x01\xe9\x3b\xbb\x83\x0c\x0a\xb0\xc1\x3a\xe2\xb3" - "\x31\x4e\xeb\x93\x73\xee\x0b\x26\xc2\x63\xa5\x75" - "\x45\x99\xd4\x5c\x9f\xa1\xd4\x45\x87\x6b\x20\x61" - "\x40\xea\x78\xa5\x32\xdf\x9e\x66\x17\xaf\xb1\x88" - "\x9e\x2e\x23\xdd\xc1\xda\x13\x97\x88\xa5\xb6\x5e" - "\x90\x14\x4e\xef\x13\xab\x5c\xd9\x2c\x97\x9e\x7c" - "\xd7\xf8\xce\xea\x81\xf5\xcd\x71\x15\x49\x44\xce" - "\x83\xb6\x05\xfb\x7d\x30\xb5\x57\x2c\x31\x4f\xfc" - "\xfe\x80\xb6\xc0\x13\x0c\x5b\x9b\x2e\x8f\x3d\xfc" - "\xc2\xa3\x0c\x11\x1b\x80\x5f\xf3", - .expectedlen = 128, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = (unsigned char *) - "\x64\xb6\xfc\x60\xbc\x61\x76\x23\x6d\x3f\x4a\x0f" - "\xe1\xb4\xd5\x20\x9e\x70\xdd\x03\x53\x6d\xbf\xce" - "\xcd\x56\x80\xbc\xb8\x15\xc8\xaa", - .perslen = 32, - }, { - .entropy = (unsigned char *) - "\xc7\xcc\xbc\x67\x7e\x21\x66\x1e\x27\x2b\x63\xdd" - "\x3a\x78\xdc\xdf\x66\x6d\x3f\x24\xae\xcf\x37\x01" - "\xa9\x0d\x89\x8a\xa7\xdc\x81\x58\xae\xb2\x10\x15" - "\x7e\x18\x44\x6d\x13\xea\xdf\x37\x85\xfe\x81\xfb", - .entropylen = 48, - .entpra = (unsigned char *) - "\x7b\xa1\x91\x5b\x3c\x04\xc4\x1b\x1d\x19\x2f\x1a" - "\x18\x81\x60\x3c\x6c\x62\x91\xb7\xe9\xf5\xcb\x96" - "\xbb\x81\x6a\xcc\xb5\xae\x55\xb6", - .entprb = (unsigned char *) - "\x99\x2c\xc7\x78\x7e\x3b\x88\x12\xef\xbe\xd3\xd2" - "\x7d\x2a\xa5\x86\xda\x8d\x58\x73\x4a\x0a\xb2\x2e" - "\xbb\x4c\x7e\xe3\x9a\xb6\x81\xc1", - .entprlen = 32, - .expected = (unsigned char *) - "\x95\x6f\x95\xfc\x3b\xb7\xfe\x3e\xd0\x4e\x1a\x14" - "\x6c\x34\x7f\x7b\x1d\x0d\x63\x5e\x48\x9c\x69\xe6" - "\x46\x07\xd2\x87\xf3\x86\x52\x3d\x98\x27\x5e\xd7" - "\x54\xe7\x75\x50\x4f\xfb\x4d\xfd\xac\x2f\x4b\x77" - "\xcf\x9e\x8e\xcc\x16\xa2\x24\xcd\x53\xde\x3e\xc5" - "\x55\x5d\xd5\x26\x3f\x89\xdf\xca\x8b\x4e\x1e\xb6" - "\x88\x78\x63\x5c\xa2\x63\x98\x4e\x6f\x25\x59\xb1" - "\x5f\x2b\x23\xb0\x4b\xa5\x18\x5d\xc2\x15\x74\x40" - "\x59\x4c\xb4\x1e\xcf\x9a\x36\xfd\x43\xe2\x03\xb8" - "\x59\x91\x30\x89\x2a\xc8\x5a\x43\x23\x7c\x73\x72" - "\xda\x3f\xad\x2b\xba\x00\x6b\xd1", - .expectedlen = 128, - .addtla = (unsigned char *) - "\x18\xe8\x17\xff\xef\x39\xc7\x41\x5c\x73\x03\x03" - "\xf6\x3d\xe8\x5f\xc8\xab\xe4\xab\x0f\xad\xe8\xd6" - "\x86\x88\x55\x28\xc1\x69\xdd\x76", - .addtlb = (unsigned char *) - "\xac\x07\xfc\xbe\x87\x0e\xd3\xea\x1f\x7e\xb8\xe7" - "\x9d\xec\xe8\xe7\xbc\xf3\x18\x25\x77\x35\x4a\xaa" - "\x00\x99\x2a\xdd\x0a\x00\x50\x82", - .addtllen = 32, - .pers = (unsigned char *) - "\xbc\x55\xab\x3c\xf6\x52\xb0\x11\x3d\x7b\x90\xb8" - "\x24\xc9\x26\x4e\x5a\x1e\x77\x0d\x3d\x58\x4a\xda" - "\xd1\x81\xe9\xf8\xeb\x30\x8f\x6f", - .perslen = 32, - }, -}; - -static const struct drbg_testvec drbg_pr_ctr_aes128_tv_template[] = { - { - .entropy = (unsigned char *) - "\xd1\x44\xc6\x61\x81\x6d\xca\x9d\x15\x28\x8a\x42" - "\x94\xd7\x28\x9c\x43\x77\x19\x29\x1a\x6d\xc3\xa2", - .entropylen = 24, - .entpra = (unsigned char *) - "\x96\xd8\x9e\x45\x32\xc9\xd2\x08\x7a\x6d\x97\x15" - "\xb4\xec\x80\xb1", - .entprb = (unsigned char *) - "\x8b\xb6\x72\xb5\x24\x0b\x98\x65\x95\x95\xe9\xc9" - "\x28\x07\xeb\xc2", - .entprlen = 16, - .expected = (unsigned char *) - "\x70\x19\xd0\x4c\x45\x78\xd6\x68\xa9\x9a\xaa\xfe" - "\xc1\xdf\x27\x9a\x1c\x0d\x0d\xf7\x24\x75\x46\xcc" - "\x77\x6b\xdf\x89\xc6\x94\xdc\x74\x50\x10\x70\x18" - "\x9b\xdc\x96\xb4\x89\x23\x40\x1a\xce\x09\x87\xce" - "\xd2\xf3\xd5\xe4\x51\x67\x74\x11\x5a\xcc\x8b\x3b" - "\x8a\xf1\x23\xa8", - .expectedlen = 64, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\x8e\x83\xe0\xeb\x37\xea\x3e\x53\x5e\x17\x6e\x77" - "\xbd\xb1\x53\x90\xfc\xdc\xc1\x3c\x9a\x88\x22\x94", - .entropylen = 24, - .entpra = (unsigned char *) - "\x6a\x85\xe7\x37\xc8\xf1\x04\x31\x98\x4f\xc8\x73" - "\x67\xd1\x08\xf8", - .entprb = (unsigned char *) - "\xd7\xa4\x68\xe2\x12\x74\xc3\xd9\xf1\xb7\x05\xbc" - "\xd4\xba\x04\x58", - .entprlen = 16, - .expected = (unsigned char *) - "\x78\xd6\xa6\x70\xff\xd1\x82\xf5\xa2\x88\x7f\x6d" - "\x3d\x8c\x39\xb1\xa8\xcb\x2c\x91\xab\x14\x7e\xbc" - "\x95\x45\x9f\x24\xb8\x20\xac\x21\x23\xdb\x72\xd7" - "\x12\x8d\x48\x95\xf3\x19\x0c\x43\xc6\x19\x45\xfc" - "\x8b\xac\x40\x29\x73\x00\x03\x45\x5e\x12\xff\x0c" - "\xc1\x02\x41\x82", - .expectedlen = 64, - .addtla = (unsigned char *) - "\xa2\xd9\x38\xcf\x8b\x29\x67\x5b\x65\x62\x6f\xe8" - "\xeb\xb3\x01\x76", - .addtlb = (unsigned char *) - "\x59\x63\x1e\x81\x8a\x14\xa8\xbb\xa1\xb8\x41\x25" - "\xd0\x7f\xcc\x43", - .addtllen = 16, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\x04\xd9\x49\xa6\xdc\xe8\x6e\xbb\xf1\x08\x77\x2b" - "\x9e\x08\xca\x92\x65\x16\xda\x99\xa2\x59\xf3\xe8", - .entropylen = 24, - .entpra = (unsigned char *) - "\x38\x7e\x3f\x6b\x51\x70\x7b\x20\xec\x53\xd0\x66" - "\xc3\x0f\xe3\xb0", - .entprb = (unsigned char *) - "\xe0\x86\xa6\xaa\x5f\x72\x2f\xad\xf7\xef\x06\xb8" - "\xd6\x9c\x9d\xe8", - .entprlen = 16, - .expected = (unsigned char *) - "\xc9\x0a\xaf\x85\x89\x71\x44\x66\x4f\x25\x0b\x2b" - "\xde\xd8\xfa\xff\x52\x5a\x1b\x32\x5e\x41\x7a\x10" - "\x1f\xef\x1e\x62\x23\xe9\x20\x30\xc9\x0d\xad\x69" - "\xb4\x9c\x5b\xf4\x87\x42\xd5\xae\x5e\x5e\x43\xcc" - "\xd9\xfd\x0b\x93\x4a\xe3\xd4\x06\x37\x36\x0f\x3f" - "\x72\x82\x0c\xcf", - .expectedlen = 64, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = (unsigned char *) - "\xbf\xa4\x9a\x8f\x7b\xd8\xb1\x7a\x9d\xfa\x45\xed" - "\x21\x52\xb3\xad", - .perslen = 16, - }, { - .entropy = (unsigned char *) - "\x92\x89\x8f\x31\xfa\x1c\xff\x6d\x18\x2f\x26\x06" - "\x43\xdf\xf8\x18\xc2\xa4\xd9\x72\xc3\xb9\xb6\x97", - .entropylen = 24, - .entpra = (unsigned char *) - "\x20\x72\x8a\x06\xf8\x6f\x8d\xd4\x41\xe2\x72\xb7" - "\xc4\x2c\xe8\x10", - .entprb = (unsigned char *) - "\x3d\xb0\xf0\x94\xf3\x05\x50\x33\x17\x86\x3e\x22" - "\x08\xf7\xa5\x01", - .entprlen = 16, - .expected = (unsigned char *) - "\x5a\x35\x39\x87\x0f\x4d\x22\xa4\x09\x24\xee\x71" - "\xc9\x6f\xac\x72\x0a\xd6\xf0\x88\x82\xd0\x83\x28" - "\x73\xec\x3f\x93\xd8\xab\x45\x23\xf0\x7e\xac\x45" - "\x14\x5e\x93\x9f\xb1\xd6\x76\x43\x3d\xb6\xe8\x08" - "\x88\xf6\xda\x89\x08\x77\x42\xfe\x1a\xf4\x3f\xc4" - "\x23\xc5\x1f\x68", - .expectedlen = 64, - .addtla = (unsigned char *) - "\x1a\x40\xfa\xe3\xcc\x6c\x7c\xa0\xf8\xda\xba\x59" - "\x23\x6d\xad\x1d", - .addtlb = (unsigned char *) - "\x9f\x72\x76\x6c\xc7\x46\xe5\xed\x2e\x53\x20\x12" - "\xbc\x59\x31\x8c", - .addtllen = 16, - .pers = (unsigned char *) - "\xea\x65\xee\x60\x26\x4e\x7e\xb6\x0e\x82\x68\xc4" - "\x37\x3c\x5c\x0b", - .perslen = 16, - }, -}; - -/* - * SP800-90A DRBG Test vectors from - * http://csrc.nist.gov/groups/STM/cavp/documents/drbg/drbgtestvectors.zip - * - * Test vectors for DRBG without prediction resistance. All types of DRBGs - * (Hash, HMAC, CTR) are tested with all permutations of use cases (w/ and - * w/o personalization string, w/ and w/o additional input string). - */ -static const struct drbg_testvec drbg_nopr_sha256_tv_template[] = { - { - .entropy = (unsigned char *) - "\xa6\x5a\xd0\xf3\x45\xdb\x4e\x0e\xff\xe8\x75\xc3" - "\xa2\xe7\x1f\x42\xc7\x12\x9d\x62\x0f\xf5\xc1\x19" - "\xa9\xef\x55\xf0\x51\x85\xe0\xfb\x85\x81\xf9\x31" - "\x75\x17\x27\x6e\x06\xe9\x60\x7d\xdb\xcb\xcc\x2e", - .entropylen = 48, - .expected = (unsigned char *) - "\xd3\xe1\x60\xc3\x5b\x99\xf3\x40\xb2\x62\x82\x64" - "\xd1\x75\x10\x60\xe0\x04\x5d\xa3\x83\xff\x57\xa5" - "\x7d\x73\xa6\x73\xd2\xb8\xd8\x0d\xaa\xf6\xa6\xc3" - "\x5a\x91\xbb\x45\x79\xd7\x3f\xd0\xc8\xfe\xd1\x11" - "\xb0\x39\x13\x06\x82\x8a\xdf\xed\x52\x8f\x01\x81" - "\x21\xb3\xfe\xbd\xc3\x43\xe7\x97\xb8\x7d\xbb\x63" - "\xdb\x13\x33\xde\xd9\xd1\xec\xe1\x77\xcf\xa6\xb7" - "\x1f\xe8\xab\x1d\xa4\x66\x24\xed\x64\x15\xe5\x1c" - "\xcd\xe2\xc7\xca\x86\xe2\x83\x99\x0e\xea\xeb\x91" - "\x12\x04\x15\x52\x8b\x22\x95\x91\x02\x81\xb0\x2d" - "\xd4\x31\xf4\xc9\xf7\x04\x27\xdf", - .expectedlen = 128, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\x73\xd3\xfb\xa3\x94\x5f\x2b\x5f\xb9\x8f\xf6\x9c" - "\x8a\x93\x17\xae\x19\xc3\x4c\xc3\xd6\xca\xa3\x2d" - "\x16\xfc\x42\xd2\x2d\xd5\x6f\x56\xcc\x1d\x30\xff" - "\x9e\x06\x3e\x09\xce\x58\xe6\x9a\x35\xb3\xa6\x56", - .entropylen = 48, - .expected = (unsigned char *) - "\x71\x7b\x93\x46\x1a\x40\xaa\x35\xa4\xaa\xc5\xe7" - "\x6d\x5b\x5b\x8a\xa0\xdf\x39\x7d\xae\x71\x58\x5b" - "\x3c\x7c\xb4\xf0\x89\xfa\x4a\x8c\xa9\x5c\x54\xc0" - "\x40\xdf\xbc\xce\x26\x81\x34\xf8\xba\x7d\x1c\xe8" - "\xad\x21\xe0\x74\xcf\x48\x84\x30\x1f\xa1\xd5\x4f" - "\x81\x42\x2f\xf4\xdb\x0b\x23\xf8\x73\x27\xb8\x1d" - "\x42\xf8\x44\x58\xd8\x5b\x29\x27\x0a\xf8\x69\x59" - "\xb5\x78\x44\xeb\x9e\xe0\x68\x6f\x42\x9a\xb0\x5b" - "\xe0\x4e\xcb\x6a\xaa\xe2\xd2\xd5\x33\x25\x3e\xe0" - "\x6c\xc7\x6a\x07\xa5\x03\x83\x9f\xe2\x8b\xd1\x1c" - "\x70\xa8\x07\x59\x97\xeb\xf6\xbe", - .expectedlen = 128, - .addtla = (unsigned char *) - "\xf4\xd5\x98\x3d\xa8\xfc\xfa\x37\xb7\x54\x67\x73" - "\xc7\xc3\xdd\x47\x34\x71\x02\x5d\xc1\xa0\xd3\x10" - "\xc1\x8b\xbd\xf5\x66\x34\x6f\xdd", - .addtlb = (unsigned char *) - "\xf7\x9e\x6a\x56\x0e\x73\xe9\xd9\x7a\xd1\x69\xe0" - "\x6f\x8c\x55\x1c\x44\xd1\xce\x6f\x28\xcc\xa4\x4d" - "\xa8\xc0\x85\xd1\x5a\x0c\x59\x40", - .addtllen = 32, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\x2a\x85\xa9\x8b\xd0\xda\x83\xd6\xad\xab\x9f\xbb" - "\x54\x31\x15\x95\x1c\x4d\x49\x9f\x6a\x15\xf6\xe4" - "\x15\x50\x88\x06\x29\x0d\xed\x8d\xb9\x6f\x96\xe1" - "\x83\x9f\xf7\x88\xda\x84\xbf\x44\x28\xd9\x1d\xaa", - .entropylen = 48, - .expected = (unsigned char *) - "\x2d\x55\xde\xc9\xed\x05\x47\x07\x3d\x04\xfc\x28" - "\x0f\x92\xf0\x4d\xd8\x00\x32\x47\x0a\x1b\x1c\x4b" - "\xef\xd9\x97\xa1\x17\x67\xda\x26\x6c\xfe\x76\x46" - "\x6f\xbc\x6d\x82\x4e\x83\x8a\x98\x66\x6c\x01\xb6" - "\xe6\x64\xe0\x08\x10\x6f\xd3\x5d\x90\xe7\x0d\x72" - "\xa6\xa7\xe3\xbb\x98\x11\x12\x56\x23\xc2\x6d\xd1" - "\xc8\xa8\x7a\x39\xf3\x34\xe3\xb8\xf8\x66\x00\x77" - "\x7d\xcf\x3c\x3e\xfa\xc9\x0f\xaf\xe0\x24\xfa\xe9" - "\x84\xf9\x6a\x01\xf6\x35\xdb\x5c\xab\x2a\xef\x4e" - "\xac\xab\x55\xb8\x9b\xef\x98\x68\xaf\x51\xd8\x16" - "\xa5\x5e\xae\xf9\x1e\xd2\xdb\xe6", - .expectedlen = 128, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = (unsigned char *) - "\xa8\x80\xec\x98\x30\x98\x15\xd2\xc6\xc4\x68\xf1" - "\x3a\x1c\xbf\xce\x6a\x40\x14\xeb\x36\x99\x53\xda" - "\x57\x6b\xce\xa4\x1c\x66\x3d\xbc", - .perslen = 32, - }, { - .entropy = (unsigned char *) - "\x69\xed\x82\xa9\xc5\x7b\xbf\xe5\x1d\x2f\xcb\x7a" - "\xd3\x50\x7d\x96\xb4\xb9\x2b\x50\x77\x51\x27\x74" - "\x33\x74\xba\xf1\x30\xdf\x8e\xdf\x87\x1d\x87\xbc" - "\x96\xb2\xc3\xa7\xed\x60\x5e\x61\x4e\x51\x29\x1a", - .entropylen = 48, - .expected = (unsigned char *) - "\xa5\x71\x24\x31\x11\xfe\x13\xe1\xa8\x24\x12\xfb" - "\x37\xa1\x27\xa5\xab\x77\xa1\x9f\xae\x8f\xaf\x13" - "\x93\xf7\x53\x85\x91\xb6\x1b\xab\xd4\x6b\xea\xb6" - "\xef\xda\x4c\x90\x6e\xef\x5f\xde\xe1\xc7\x10\x36" - "\xd5\x67\xbd\x14\xb6\x89\x21\x0c\xc9\x92\x65\x64" - "\xd0\xf3\x23\xe0\x7f\xd1\xe8\x75\xc2\x85\x06\xea" - "\xca\xc0\xcb\x79\x2d\x29\x82\xfc\xaa\x9a\xc6\x95" - "\x7e\xdc\x88\x65\xba\xec\x0e\x16\x87\xec\xa3\x9e" - "\xd8\x8c\x80\xab\x3a\x64\xe0\xcb\x0e\x45\x98\xdd" - "\x7c\x6c\x6c\x26\x11\x13\xc8\xce\xa9\x47\xa6\x06" - "\x57\xa2\x66\xbb\x2d\x7f\xf3\xc1", - .expectedlen = 128, - .addtla = (unsigned char *) - "\x74\xd3\x6d\xda\xe8\xd6\x86\x5f\x63\x01\xfd\xf2" - "\x7d\x06\x29\x6d\x94\xd1\x66\xf0\xd2\x72\x67\x4e" - "\x77\xc5\x3d\x9e\x03\xe3\xa5\x78", - .addtlb = (unsigned char *) - "\xf6\xb6\x3d\xf0\x7c\x26\x04\xc5\x8b\xcd\x3e\x6a" - "\x9f\x9c\x3a\x2e\xdb\x47\x87\xe5\x8e\x00\x5e\x2b" - "\x74\x7f\xa6\xf6\x80\xcd\x9b\x21", - .addtllen = 32, - .pers = (unsigned char *) - "\x74\xa6\xe0\x08\xf9\x27\xee\x1d\x6e\x3c\x28\x20" - "\x87\xdd\xd7\x54\x31\x47\x78\x4b\xe5\x6d\xa3\x73" - "\xa9\x65\xb1\x10\xc1\xdc\x77\x7c", - .perslen = 32, - }, -}; - -static const struct drbg_testvec drbg_nopr_hmac_sha256_tv_template[] = { - { - .entropy = (unsigned char *) - "\xca\x85\x19\x11\x34\x93\x84\xbf\xfe\x89\xde\x1c" - "\xbd\xc4\x6e\x68\x31\xe4\x4d\x34\xa4\xfb\x93\x5e" - "\xe2\x85\xdd\x14\xb7\x1a\x74\x88\x65\x9b\xa9\x6c" - "\x60\x1d\xc6\x9f\xc9\x02\x94\x08\x05\xec\x0c\xa8", - .entropylen = 48, - .expected = (unsigned char *) - "\xe5\x28\xe9\xab\xf2\xde\xce\x54\xd4\x7c\x7e\x75" - "\xe5\xfe\x30\x21\x49\xf8\x17\xea\x9f\xb4\xbe\xe6" - "\xf4\x19\x96\x97\xd0\x4d\x5b\x89\xd5\x4f\xbb\x97" - "\x8a\x15\xb5\xc4\x43\xc9\xec\x21\x03\x6d\x24\x60" - "\xb6\xf7\x3e\xba\xd0\xdc\x2a\xba\x6e\x62\x4a\xbf" - "\x07\x74\x5b\xc1\x07\x69\x4b\xb7\x54\x7b\xb0\x99" - "\x5f\x70\xde\x25\xd6\xb2\x9e\x2d\x30\x11\xbb\x19" - "\xd2\x76\x76\xc0\x71\x62\xc8\xb5\xcc\xde\x06\x68" - "\x96\x1d\xf8\x68\x03\x48\x2c\xb3\x7e\xd6\xd5\xc0" - "\xbb\x8d\x50\xcf\x1f\x50\xd4\x76\xaa\x04\x58\xbd" - "\xab\xa8\x06\xf4\x8b\xe9\xdc\xb8", - .expectedlen = 128, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\xf9\x7a\x3c\xfd\x91\xfa\xa0\x46\xb9\xe6\x1b\x94" - "\x93\xd4\x36\xc4\x93\x1f\x60\x4b\x22\xf1\x08\x15" - "\x21\xb3\x41\x91\x51\xe8\xff\x06\x11\xf3\xa7\xd4" - "\x35\x95\x35\x7d\x58\x12\x0b\xd1\xe2\xdd\x8a\xed", - .entropylen = 48, - .expected = (unsigned char *) - "\xc6\x87\x1c\xff\x08\x24\xfe\x55\xea\x76\x89\xa5" - "\x22\x29\x88\x67\x30\x45\x0e\x5d\x36\x2d\xa5\xbf" - "\x59\x0d\xcf\x9a\xcd\x67\xfe\xd4\xcb\x32\x10\x7d" - "\xf5\xd0\x39\x69\xa6\x6b\x1f\x64\x94\xfd\xf5\xd6" - "\x3d\x5b\x4d\x0d\x34\xea\x73\x99\xa0\x7d\x01\x16" - "\x12\x6d\x0d\x51\x8c\x7c\x55\xba\x46\xe1\x2f\x62" - "\xef\xc8\xfe\x28\xa5\x1c\x9d\x42\x8e\x6d\x37\x1d" - "\x73\x97\xab\x31\x9f\xc7\x3d\xed\x47\x22\xe5\xb4" - "\xf3\x00\x04\x03\x2a\x61\x28\xdf\x5e\x74\x97\xec" - "\xf8\x2c\xa7\xb0\xa5\x0e\x86\x7e\xf6\x72\x8a\x4f" - "\x50\x9a\x8c\x85\x90\x87\x03\x9c", - .expectedlen = 128, - .addtla = (unsigned char *) - "\x51\x72\x89\xaf\xe4\x44\xa0\xfe\x5e\xd1\xa4\x1d" - "\xbb\xb5\xeb\x17\x15\x00\x79\xbd\xd3\x1e\x29\xcf" - "\x2f\xf3\x00\x34\xd8\x26\x8e\x3b", - .addtlb = (unsigned char *) - "\x88\x02\x8d\x29\xef\x80\xb4\xe6\xf0\xfe\x12\xf9" - "\x1d\x74\x49\xfe\x75\x06\x26\x82\xe8\x9c\x57\x14" - "\x40\xc0\xc9\xb5\x2c\x42\xa6\xe0", - .addtllen = 32, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\x8d\xf0\x13\xb4\xd1\x03\x52\x30\x73\x91\x7d\xdf" - "\x6a\x86\x97\x93\x05\x9e\x99\x43\xfc\x86\x54\x54" - "\x9e\x7a\xb2\x2f\x7c\x29\xf1\x22\xda\x26\x25\xaf" - "\x2d\xdd\x4a\xbc\xce\x3c\xf4\xfa\x46\x59\xd8\x4e", - .entropylen = 48, - .expected = (unsigned char *) - "\xb9\x1c\xba\x4c\xc8\x4f\xa2\x5d\xf8\x61\x0b\x81" - "\xb6\x41\x40\x27\x68\xa2\x09\x72\x34\x93\x2e\x37" - "\xd5\x90\xb1\x15\x4c\xbd\x23\xf9\x74\x52\xe3\x10" - "\xe2\x91\xc4\x51\x46\x14\x7f\x0d\xa2\xd8\x17\x61" - "\xfe\x90\xfb\xa6\x4f\x94\x41\x9c\x0f\x66\x2b\x28" - "\xc1\xed\x94\xda\x48\x7b\xb7\xe7\x3e\xec\x79\x8f" - "\xbc\xf9\x81\xb7\x91\xd1\xbe\x4f\x17\x7a\x89\x07" - "\xaa\x3c\x40\x16\x43\xa5\xb6\x2b\x87\xb8\x9d\x66" - "\xb3\xa6\x0e\x40\xd4\xa8\xe4\xe9\xd8\x2a\xf6\xd2" - "\x70\x0e\x6f\x53\x5c\xdb\x51\xf7\x5c\x32\x17\x29" - "\x10\x37\x41\x03\x0c\xcc\x3a\x56", - .expectedlen = 128, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = (unsigned char *) - "\xb5\x71\xe6\x6d\x7c\x33\x8b\xc0\x7b\x76\xad\x37" - "\x57\xbb\x2f\x94\x52\xbf\x7e\x07\x43\x7a\xe8\x58" - "\x1c\xe7\xbc\x7c\x3a\xc6\x51\xa9", - .perslen = 32, - }, { - .entropy = (unsigned char *) - "\xc2\xa5\x66\xa9\xa1\x81\x7b\x15\xc5\xc3\xb7\x78" - "\x17\x7a\xc8\x7c\x24\xe7\x97\xbe\x0a\x84\x5f\x11" - "\xc2\xfe\x39\x9d\xd3\x77\x32\xf2\xcb\x18\x94\xeb" - "\x2b\x97\xb3\xc5\x6e\x62\x83\x29\x51\x6f\x86\xec", - .entropylen = 48, - .expected = (unsigned char *) - "\xb3\xa3\x69\x8d\x77\x76\x99\xa0\xdd\x9f\xa3\xf0" - "\xa9\xfa\x57\x83\x2d\x3c\xef\xac\x5d\xf2\x44\x37" - "\xc6\xd7\x3a\x0f\xe4\x10\x40\xf1\x72\x90\x38\xae" - "\xf1\xe9\x26\x35\x2e\xa5\x9d\xe1\x20\xbf\xb7\xb0" - "\x73\x18\x3a\x34\x10\x6e\xfe\xd6\x27\x8f\xf8\xad" - "\x84\x4b\xa0\x44\x81\x15\xdf\xdd\xf3\x31\x9a\x82" - "\xde\x6b\xb1\x1d\x80\xbd\x87\x1a\x9a\xcd\x35\xc7" - "\x36\x45\xe1\x27\x0f\xb9\xfe\x4f\xa8\x8e\xc0\xe4" - "\x65\x40\x9e\xa0\xcb\xa8\x09\xfe\x2f\x45\xe0\x49" - "\x43\xa2\xe3\x96\xbb\xb7\xdd\x2f\x4e\x07\x95\x30" - "\x35\x24\xcc\x9c\xc5\xea\x54\xa1", - .expectedlen = 128, - .addtla = (unsigned char *) - "\x41\x3d\xd8\x3f\xe5\x68\x35\xab\xd4\x78\xcb\x96" - "\x93\xd6\x76\x35\x90\x1c\x40\x23\x9a\x26\x64\x62" - "\xd3\x13\x3b\x83\xe4\x9c\x82\x0b", - .addtlb = (unsigned char *) - "\xd5\xc4\xa7\x1f\x9d\x6d\x95\xa1\xbe\xdf\x0b\xd2" - "\x24\x7c\x27\x7d\x1f\x84\xa4\xe5\x7a\x4a\x88\x25" - "\xb8\x2a\x2d\x09\x7d\xe6\x3e\xf1", - .addtllen = 32, - .pers = (unsigned char *) - "\x13\xce\x4d\x8d\xd2\xdb\x97\x96\xf9\x41\x56\xc8" - "\xe8\xf0\x76\x9b\x0a\xa1\xc8\x2c\x13\x23\xb6\x15" - "\x36\x60\x3b\xca\x37\xc9\xee\x29", - .perslen = 32, - }, -}; - -static const struct drbg_testvec drbg_nopr_ctr_aes192_tv_template[] = { - { - .entropy = (unsigned char *) - "\xc3\x5c\x2f\xa2\xa8\x9d\x52\xa1\x1f\xa3\x2a\xa9" - "\x6c\x95\xb8\xf1\xc9\xa8\xf9\xcb\x24\x5a\x8b\x40" - "\xf3\xa6\xe5\xa7\xfb\xd9\xd3\xc6\x8e\x27\x7b\xa9" - "\xac\x9b\xbb\x00", - .entropylen = 40, - .expected = (unsigned char *) - "\x8c\x2e\x72\xab\xfd\x9b\xb8\x28\x4d\xb7\x9e\x17" - "\xa4\x3a\x31\x46\xcd\x76\x94\xe3\x52\x49\xfc\x33" - "\x83\x91\x4a\x71\x17\xf4\x13\x68\xe6\xd4\xf1\x48" - "\xff\x49\xbf\x29\x07\x6b\x50\x15\xc5\x9f\x45\x79" - "\x45\x66\x2e\x3d\x35\x03\x84\x3f\x4a\xa5\xa3\xdf" - "\x9a\x9d\xf1\x0d", - .expectedlen = 64, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = NULL, - .perslen = 0, - }, -}; - -static const struct drbg_testvec drbg_nopr_ctr_aes256_tv_template[] = { - { - .entropy = (unsigned char *) - "\x36\x40\x19\x40\xfa\x8b\x1f\xba\x91\xa1\x66\x1f" - "\x21\x1d\x78\xa0\xb9\x38\x9a\x74\xe5\xbc\xcf\xec" - "\xe8\xd7\x66\xaf\x1a\x6d\x3b\x14\x49\x6f\x25\xb0" - "\xf1\x30\x1b\x4f\x50\x1b\xe3\x03\x80\xa1\x37\xeb", - .entropylen = 48, - .expected = (unsigned char *) - "\x58\x62\xeb\x38\xbd\x55\x8d\xd9\x78\xa6\x96\xe6" - "\xdf\x16\x47\x82\xdd\xd8\x87\xe7\xe9\xa6\xc9\xf3" - "\xf1\xfb\xaf\xb7\x89\x41\xb5\x35\xa6\x49\x12\xdf" - "\xd2\x24\xc6\xdc\x74\x54\xe5\x25\x0b\x3d\x97\x16" - "\x5e\x16\x26\x0c\x2f\xaf\x1c\xc7\x73\x5c\xb7\x5f" - "\xb4\xf0\x7e\x1d", - .expectedlen = 64, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = NULL, - .perslen = 0, - }, -}; - -static const struct drbg_testvec drbg_nopr_ctr_aes128_tv_template[] = { - { - .entropy = (unsigned char *) - "\x87\xe1\xc5\x32\x99\x7f\x57\xa3\x5c\x28\x6d\xe8" - "\x64\xbf\xf2\x64\xa3\x9e\x98\xdb\x6c\x10\x78\x7f", - .entropylen = 24, - .expected = (unsigned char *) - "\x2c\x14\x7e\x24\x11\x9a\xd8\xd4\xb2\xed\x61\xc1" - "\x53\xd0\x50\xc9\x24\xff\x59\x75\x15\xf1\x17\x3a" - "\x3d\xf4\x4b\x2c\x84\x28\xef\x89\x0e\xb9\xde\xf3" - "\xe4\x78\x04\xb2\xfd\x9b\x35\x7f\xe1\x3f\x8a\x3e" - "\x10\xc8\x67\x0a\xf9\xdf\x2d\x6c\x96\xfb\xb2\xb8" - "\xcb\x2d\xd6\xb0", - .expectedlen = 64, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\x71\xbd\xce\x35\x42\x7d\x20\xbf\x58\xcf\x17\x74" - "\xce\x72\xd8\x33\x34\x50\x2d\x8f\x5b\x14\xc4\xdd", - .entropylen = 24, - .expected = (unsigned char *) - "\x97\x33\xe8\x20\x12\xe2\x7b\xa1\x46\x8f\xf2\x34" - "\xb3\xc9\xb6\x6b\x20\xb2\x4f\xee\x27\xd8\x0b\x21" - "\x8c\xff\x63\x73\x69\x29\xfb\xf3\x85\xcd\x88\x8e" - "\x43\x2c\x71\x8b\xa2\x55\xd2\x0f\x1d\x7f\xe3\xe1" - "\x2a\xa3\xe9\x2c\x25\x89\xc7\x14\x52\x99\x56\xcc" - "\xc3\xdf\xb3\x81", - .expectedlen = 64, - .addtla = (unsigned char *) - "\x66\xef\x42\xd6\x9a\x8c\x3d\x6d\x4a\x9e\x95\xa6" - "\x91\x4d\x81\x56", - .addtlb = (unsigned char *) - "\xe3\x18\x83\xd9\x4b\x5e\xc4\xcc\xaa\x61\x2f\xbb" - "\x4a\x55\xd1\xc6", - .addtllen = 16, - .pers = NULL, - .perslen = 0, - }, { - .entropy = (unsigned char *) - "\xca\x4b\x1e\xfa\x75\xbd\x69\x36\x38\x73\xb8\xf9" - "\xdb\x4d\x35\x0e\x47\xbf\x6c\x37\x72\xfd\xf7\xa9", - .entropylen = 24, - .expected = (unsigned char *) - "\x59\xc3\x19\x79\x1b\xb1\xf3\x0e\xe9\x34\xae\x6e" - "\x8b\x1f\xad\x1f\x74\xca\x25\x45\x68\xb8\x7f\x75" - "\x12\xf8\xf2\xab\x4c\x23\x01\x03\x05\xe1\x70\xee" - "\x75\xd8\xcb\xeb\x23\x4c\x7a\x23\x6e\x12\x27\xdb" - "\x6f\x7a\xac\x3c\x44\xb7\x87\x4b\x65\x56\x74\x45" - "\x34\x30\x0c\x3d", - .expectedlen = 64, - .addtla = NULL, - .addtlb = NULL, - .addtllen = 0, - .pers = (unsigned char *) - "\xeb\xaa\x60\x2c\x4d\xbe\x33\xff\x1b\xef\xbf\x0a" - "\x0b\xc6\x97\x54", - .perslen = 16, - }, { - .entropy = (unsigned char *) - "\xc0\x70\x1f\x92\x50\x75\x8f\xcd\xf2\xbe\x73\x98" - "\x80\xdb\x66\xeb\x14\x68\xb4\xa5\x87\x9c\x2d\xa6", - .entropylen = 24, - .expected = (unsigned char *) - "\x97\xc0\xc0\xe5\xa0\xcc\xf2\x4f\x33\x63\x48\x8a" - "\xdb\x13\x0a\x35\x89\xbf\x80\x65\x62\xee\x13\x95" - "\x7c\x33\xd3\x7d\xf4\x07\x77\x7a\x2b\x65\x0b\x5f" - "\x45\x5c\x13\xf1\x90\x77\x7f\xc5\x04\x3f\xcc\x1a" - "\x38\xf8\xcd\x1b\xbb\xd5\x57\xd1\x4a\x4c\x2e\x8a" - "\x2b\x49\x1e\x5c", - .expectedlen = 64, - .addtla = (unsigned char *) - "\xf9\x01\xf8\x16\x7a\x1d\xff\xde\x8e\x3c\x83\xe2" - "\x44\x85\xe7\xfe", - .addtlb = (unsigned char *) - "\x17\x1c\x09\x38\xc2\x38\x9f\x97\x87\x60\x55\xb4" - "\x82\x16\x62\x7f", - .addtllen = 16, - .pers = (unsigned char *) - "\x80\x08\xae\xe8\xe9\x69\x40\xc5\x08\x73\xc7\x9f" - "\x8e\xcf\xe0\x02", - .perslen = 16, - }, -}; - -/* Cast5 test vectors from RFC 2144 */ -static const struct cipher_testvec cast5_enc_tv_template[] = { - { - .key = "\x01\x23\x45\x67\x12\x34\x56\x78" - "\x23\x45\x67\x89\x34\x56\x78\x9a", - .klen = 16, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .ilen = 8, - .result = "\x23\x8b\x4f\xe5\x84\x7e\x44\xb2", - .rlen = 8, - }, { - .key = "\x01\x23\x45\x67\x12\x34\x56\x78" - "\x23\x45", - .klen = 10, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .ilen = 8, - .result = "\xeb\x6a\x71\x1a\x2c\x02\x27\x1b", - .rlen = 8, - }, { - .key = "\x01\x23\x45\x67\x12", - .klen = 5, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .ilen = 8, - .result = "\x7a\xc8\x16\xd1\x6e\x9b\x30\x2e", - .rlen = 8, - }, { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", - .klen = 16, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\x8D\xFC\x81\x9C\xCB\xAA\x5A\x1C" - "\x7E\x95\xCF\x40\xAB\x4D\x6F\xEA" - "\xD3\xD9\xB0\x9A\xB7\xC7\xE0\x2E" - "\xD1\x39\x34\x92\x8F\xFA\x14\xF1" - "\xD5\xD2\x7B\x59\x1F\x35\x28\xC2" - "\x20\xD9\x42\x06\xC9\x0B\x10\x04" - "\xF8\x79\xCD\x32\x86\x75\x4C\xB6" - "\x7B\x1C\x52\xB1\x91\x64\x22\x4B" - "\x13\xC7\xAE\x98\x0E\xB5\xCF\x6F" - "\x3F\xF4\x43\x96\x73\x0D\xA2\x05" - "\xDB\xFD\x28\x90\x2C\x56\xB9\x37" - "\x5B\x69\x0C\xAD\x84\x67\xFF\x15" - "\x4A\xD4\xA7\xD3\xDD\x99\x47\x3A" - "\xED\x34\x35\x78\x6B\x91\xC9\x32" - "\xE1\xBF\xBC\xB4\x04\x85\x6A\x39" - "\xC0\xBA\x51\xD0\x0F\x4E\xD1\xE2" - "\x1C\xFD\x0E\x05\x07\xF4\x10\xED" - "\xA2\x17\xFF\xF5\x64\xC6\x1A\x22" - "\xAD\x78\xE7\xD7\x11\xE9\x99\xB9" - "\xAA\xEC\x6F\xF8\x3B\xBF\xCE\x77" - "\x93\xE8\xAD\x1D\x50\x6C\xAE\xBC" - "\xBA\x5C\x80\xD1\x91\x65\x51\x1B" - "\xE8\x0A\xCD\x99\x96\x71\x3D\xB6" - "\x78\x75\x37\x55\xC1\xF5\x90\x40" - "\x34\xF4\x7E\xC8\xCC\x3A\x5F\x6E" - "\x36\xA1\xA1\xC2\x3A\x72\x42\x8E" - "\x0E\x37\x88\xE8\xCE\x83\xCB\xAD" - "\xE0\x69\x77\x50\xC7\x0C\x99\xCA" - "\x19\x5B\x30\x25\x9A\xEF\x9B\x0C" - "\xEF\x8F\x74\x4C\xCF\x49\x4E\xB9" - "\xC5\xAE\x9E\x2E\x78\x9A\xB9\x48" - "\xD5\x81\xE4\x37\x1D\xBF\x27\xD9" - "\xC5\xD6\x65\x43\x45\x8C\xBB\xB6" - "\x55\xF4\x06\xBB\x49\x53\x8B\x1B" - "\x07\xA9\x96\x69\x5B\xCB\x0F\xBC" - "\x93\x85\x90\x0F\x0A\x68\x40\x2A" - "\x95\xED\x2D\x88\xBF\x71\xD0\xBB" - "\xEC\xB0\x77\x6C\x79\xFC\x3C\x05" - "\x49\x3F\xB8\x24\xEF\x8E\x09\xA2" - "\x1D\xEF\x92\x02\x96\xD4\x7F\xC8" - "\x03\xB2\xCA\xDB\x17\x5C\x52\xCF" - "\xDD\x70\x37\x63\xAA\xA5\x83\x20" - "\x52\x02\xF6\xB9\xE7\x6E\x0A\xB6" - "\x79\x03\xA0\xDA\xA3\x79\x21\xBD" - "\xE3\x37\x3A\xC0\xF7\x2C\x32\xBE" - "\x8B\xE8\xA6\x00\xC7\x32\xD5\x06" - "\xBB\xE3\xAB\x06\x21\x82\xB8\x32" - "\x31\x34\x2A\xA7\x1F\x64\x99\xBF" - "\xFA\xDA\x3D\x75\xF7\x48\xD5\x48" - "\x4B\x52\x7E\xF6\x7C\xAB\x67\x59" - "\xC5\xDC\xA8\xC6\x63\x85\x4A\xDF" - "\xF0\x40\x5F\xCF\xE3\x58\x52\x67" - "\x7A\x24\x32\xC5\xEC\x9E\xA9\x6F" - "\x58\x56\xDD\x94\x1F\x71\x8D\xF4" - "\x6E\xFF\x2C\xA7\xA5\xD8\xBA\xAF" - "\x1D\x8B\xA2\x46\xB5\xC4\x9F\x57" - "\x8D\xD8\xB3\x3C\x02\x0D\xBB\x84" - "\xC7\xBD\xB4\x9A\x6E\xBB\xB1\x37" - "\x95\x79\xC4\xA7\xEA\x1D\xDC\x33" - "\x5D\x0B\x3F\x03\x8F\x30\xF9\xAE" - "\x4F\xFE\x24\x9C\x9A\x02\xE5\x57" - "\xF5\xBC\x25\xD6\x02\x56\x57\x1C", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast5_dec_tv_template[] = { - { - .key = "\x01\x23\x45\x67\x12\x34\x56\x78" - "\x23\x45\x67\x89\x34\x56\x78\x9a", - .klen = 16, - .input = "\x23\x8b\x4f\xe5\x84\x7e\x44\xb2", - .ilen = 8, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .rlen = 8, - }, { - .key = "\x01\x23\x45\x67\x12\x34\x56\x78" - "\x23\x45", - .klen = 10, - .input = "\xeb\x6a\x71\x1a\x2c\x02\x27\x1b", - .ilen = 8, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .rlen = 8, - }, { - .key = "\x01\x23\x45\x67\x12", - .klen = 5, - .input = "\x7a\xc8\x16\xd1\x6e\x9b\x30\x2e", - .ilen = 8, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .rlen = 8, - }, { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", - .klen = 16, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\x8D\xFC\x81\x9C\xCB\xAA\x5A\x1C" - "\x7E\x95\xCF\x40\xAB\x4D\x6F\xEA" - "\xD3\xD9\xB0\x9A\xB7\xC7\xE0\x2E" - "\xD1\x39\x34\x92\x8F\xFA\x14\xF1" - "\xD5\xD2\x7B\x59\x1F\x35\x28\xC2" - "\x20\xD9\x42\x06\xC9\x0B\x10\x04" - "\xF8\x79\xCD\x32\x86\x75\x4C\xB6" - "\x7B\x1C\x52\xB1\x91\x64\x22\x4B" - "\x13\xC7\xAE\x98\x0E\xB5\xCF\x6F" - "\x3F\xF4\x43\x96\x73\x0D\xA2\x05" - "\xDB\xFD\x28\x90\x2C\x56\xB9\x37" - "\x5B\x69\x0C\xAD\x84\x67\xFF\x15" - "\x4A\xD4\xA7\xD3\xDD\x99\x47\x3A" - "\xED\x34\x35\x78\x6B\x91\xC9\x32" - "\xE1\xBF\xBC\xB4\x04\x85\x6A\x39" - "\xC0\xBA\x51\xD0\x0F\x4E\xD1\xE2" - "\x1C\xFD\x0E\x05\x07\xF4\x10\xED" - "\xA2\x17\xFF\xF5\x64\xC6\x1A\x22" - "\xAD\x78\xE7\xD7\x11\xE9\x99\xB9" - "\xAA\xEC\x6F\xF8\x3B\xBF\xCE\x77" - "\x93\xE8\xAD\x1D\x50\x6C\xAE\xBC" - "\xBA\x5C\x80\xD1\x91\x65\x51\x1B" - "\xE8\x0A\xCD\x99\x96\x71\x3D\xB6" - "\x78\x75\x37\x55\xC1\xF5\x90\x40" - "\x34\xF4\x7E\xC8\xCC\x3A\x5F\x6E" - "\x36\xA1\xA1\xC2\x3A\x72\x42\x8E" - "\x0E\x37\x88\xE8\xCE\x83\xCB\xAD" - "\xE0\x69\x77\x50\xC7\x0C\x99\xCA" - "\x19\x5B\x30\x25\x9A\xEF\x9B\x0C" - "\xEF\x8F\x74\x4C\xCF\x49\x4E\xB9" - "\xC5\xAE\x9E\x2E\x78\x9A\xB9\x48" - "\xD5\x81\xE4\x37\x1D\xBF\x27\xD9" - "\xC5\xD6\x65\x43\x45\x8C\xBB\xB6" - "\x55\xF4\x06\xBB\x49\x53\x8B\x1B" - "\x07\xA9\x96\x69\x5B\xCB\x0F\xBC" - "\x93\x85\x90\x0F\x0A\x68\x40\x2A" - "\x95\xED\x2D\x88\xBF\x71\xD0\xBB" - "\xEC\xB0\x77\x6C\x79\xFC\x3C\x05" - "\x49\x3F\xB8\x24\xEF\x8E\x09\xA2" - "\x1D\xEF\x92\x02\x96\xD4\x7F\xC8" - "\x03\xB2\xCA\xDB\x17\x5C\x52\xCF" - "\xDD\x70\x37\x63\xAA\xA5\x83\x20" - "\x52\x02\xF6\xB9\xE7\x6E\x0A\xB6" - "\x79\x03\xA0\xDA\xA3\x79\x21\xBD" - "\xE3\x37\x3A\xC0\xF7\x2C\x32\xBE" - "\x8B\xE8\xA6\x00\xC7\x32\xD5\x06" - "\xBB\xE3\xAB\x06\x21\x82\xB8\x32" - "\x31\x34\x2A\xA7\x1F\x64\x99\xBF" - "\xFA\xDA\x3D\x75\xF7\x48\xD5\x48" - "\x4B\x52\x7E\xF6\x7C\xAB\x67\x59" - "\xC5\xDC\xA8\xC6\x63\x85\x4A\xDF" - "\xF0\x40\x5F\xCF\xE3\x58\x52\x67" - "\x7A\x24\x32\xC5\xEC\x9E\xA9\x6F" - "\x58\x56\xDD\x94\x1F\x71\x8D\xF4" - "\x6E\xFF\x2C\xA7\xA5\xD8\xBA\xAF" - "\x1D\x8B\xA2\x46\xB5\xC4\x9F\x57" - "\x8D\xD8\xB3\x3C\x02\x0D\xBB\x84" - "\xC7\xBD\xB4\x9A\x6E\xBB\xB1\x37" - "\x95\x79\xC4\xA7\xEA\x1D\xDC\x33" - "\x5D\x0B\x3F\x03\x8F\x30\xF9\xAE" - "\x4F\xFE\x24\x9C\x9A\x02\xE5\x57" - "\xF5\xBC\x25\xD6\x02\x56\x57\x1C", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast5_cbc_enc_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", - .klen = 16, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\x05\x28\xCE\x61\x90\x80\xE1\x78" - "\xB9\x2A\x97\x7C\xB0\x83\xD8\x1A" - "\xDE\x58\x7F\xD7\xFD\x72\xB8\xFB" - "\xDA\xF0\x6E\x77\x14\x47\x82\xBA" - "\x29\x0E\x25\x6E\xB4\x39\xD9\x7F" - "\x05\xA7\xA7\x3A\xC1\x5D\x9E\x39" - "\xA7\xFB\x0D\x05\x00\xF3\x58\x67" - "\x60\xEC\x73\x77\x46\x85\x9B\x6A" - "\x08\x3E\xBE\x59\xFB\xE4\x96\x34" - "\xB4\x05\x49\x1A\x97\x43\xAD\xA0" - "\xA9\x1E\x6E\x74\xF1\x94\xEC\xA8" - "\xB5\x8A\x20\xEA\x89\x6B\x19\xAA" - "\xA7\xF1\x33\x67\x90\x23\x0D\xEE" - "\x81\xD5\x78\x4F\xD3\x63\xEA\x46" - "\xB5\xB2\x6E\xBB\xCA\x76\x06\x10" - "\x96\x2A\x0A\xBA\xF9\x41\x5A\x1D" - "\x36\x7C\x56\x14\x54\x83\xFA\xA1" - "\x27\xDD\xBA\x8A\x90\x29\xD6\xA6" - "\xFA\x48\x3E\x1E\x23\x6E\x98\xA8" - "\xA7\xD9\x67\x92\x5C\x13\xB4\x71" - "\xA8\xAA\x89\x4A\xA4\xB3\x49\x7C" - "\x7D\x7F\xCE\x6F\x29\x2E\x7E\x37" - "\xC8\x52\x60\xD9\xE7\xCA\x60\x98" - "\xED\xCD\xE8\x60\x83\xAD\x34\x4D" - "\x96\x4A\x99\x2B\xB7\x14\x75\x66" - "\x6C\x2C\x1A\xBA\x4B\xBB\x49\x56" - "\xE1\x86\xA2\x0E\xD0\xF0\x07\xD3" - "\x18\x38\x09\x9C\x0E\x8B\x86\x07" - "\x90\x12\x37\x49\x27\x98\x69\x18" - "\xB0\xCC\xFB\xD3\xBD\x04\xA0\x85" - "\x4B\x22\x97\x07\xB6\x97\xE9\x95" - "\x0F\x88\x36\xA9\x44\x00\xC6\xE9" - "\x27\x53\x5C\x5B\x1F\xD3\xE2\xEE" - "\xD0\xCD\x63\x30\xA9\xC0\xDD\x49" - "\xFE\x16\xA4\x07\x0D\xE2\x5D\x97" - "\xDE\x89\xBA\x2E\xF3\xA9\x5E\xBE" - "\x03\x55\x0E\x02\x41\x4A\x45\x06" - "\xBE\xEA\x32\xF2\xDC\x91\x5C\x20" - "\x94\x02\x30\xD2\xFC\x29\xFA\x8E" - "\x34\xA0\x31\xB8\x34\xBA\xAE\x54" - "\xB5\x88\x1F\xDC\x43\xDC\x22\x9F" - "\xDC\xCE\xD3\xFA\xA4\xA8\xBC\x8A" - "\xC7\x5A\x43\x21\xA5\xB1\xDB\xC3" - "\x84\x3B\xB4\x9B\xB5\xA7\xF1\x0A" - "\xB6\x37\x21\x19\x55\xC2\xBD\x99" - "\x49\x24\xBB\x7C\xB3\x8E\xEF\xD2" - "\x3A\xCF\xA0\x31\x28\x0E\x25\xA2" - "\x11\xB4\x18\x17\x1A\x65\x92\x56" - "\xE8\xE0\x52\x9C\x61\x18\x2A\xB1" - "\x1A\x01\x22\x45\x17\x62\x52\x6C" - "\x91\x44\xCF\x98\xC7\xC0\x79\x26" - "\x32\x66\x6F\x23\x7F\x94\x36\x88" - "\x3C\xC9\xD0\xB7\x45\x30\x31\x86" - "\x3D\xC6\xA3\x98\x62\x84\x1A\x8B" - "\x16\x88\xC7\xA3\xE9\x4F\xE0\x86" - "\xA4\x93\xA8\x34\x5A\xCA\xDF\xCA" - "\x46\x38\xD2\xF4\xE0\x2D\x1E\xC9" - "\x7C\xEF\x53\xB7\x60\x72\x41\xBF" - "\x29\x00\x87\x02\xAF\x44\x4C\xB7" - "\x8C\xF5\x3F\x19\xF4\x80\x45\xA7" - "\x15\x5F\xDB\xE9\xB1\x83\xD2\xE6" - "\x1D\x18\x66\x44\x5B\x8F\x14\xEB", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast5_cbc_dec_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", - .klen = 16, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\x05\x28\xCE\x61\x90\x80\xE1\x78" - "\xB9\x2A\x97\x7C\xB0\x83\xD8\x1A" - "\xDE\x58\x7F\xD7\xFD\x72\xB8\xFB" - "\xDA\xF0\x6E\x77\x14\x47\x82\xBA" - "\x29\x0E\x25\x6E\xB4\x39\xD9\x7F" - "\x05\xA7\xA7\x3A\xC1\x5D\x9E\x39" - "\xA7\xFB\x0D\x05\x00\xF3\x58\x67" - "\x60\xEC\x73\x77\x46\x85\x9B\x6A" - "\x08\x3E\xBE\x59\xFB\xE4\x96\x34" - "\xB4\x05\x49\x1A\x97\x43\xAD\xA0" - "\xA9\x1E\x6E\x74\xF1\x94\xEC\xA8" - "\xB5\x8A\x20\xEA\x89\x6B\x19\xAA" - "\xA7\xF1\x33\x67\x90\x23\x0D\xEE" - "\x81\xD5\x78\x4F\xD3\x63\xEA\x46" - "\xB5\xB2\x6E\xBB\xCA\x76\x06\x10" - "\x96\x2A\x0A\xBA\xF9\x41\x5A\x1D" - "\x36\x7C\x56\x14\x54\x83\xFA\xA1" - "\x27\xDD\xBA\x8A\x90\x29\xD6\xA6" - "\xFA\x48\x3E\x1E\x23\x6E\x98\xA8" - "\xA7\xD9\x67\x92\x5C\x13\xB4\x71" - "\xA8\xAA\x89\x4A\xA4\xB3\x49\x7C" - "\x7D\x7F\xCE\x6F\x29\x2E\x7E\x37" - "\xC8\x52\x60\xD9\xE7\xCA\x60\x98" - "\xED\xCD\xE8\x60\x83\xAD\x34\x4D" - "\x96\x4A\x99\x2B\xB7\x14\x75\x66" - "\x6C\x2C\x1A\xBA\x4B\xBB\x49\x56" - "\xE1\x86\xA2\x0E\xD0\xF0\x07\xD3" - "\x18\x38\x09\x9C\x0E\x8B\x86\x07" - "\x90\x12\x37\x49\x27\x98\x69\x18" - "\xB0\xCC\xFB\xD3\xBD\x04\xA0\x85" - "\x4B\x22\x97\x07\xB6\x97\xE9\x95" - "\x0F\x88\x36\xA9\x44\x00\xC6\xE9" - "\x27\x53\x5C\x5B\x1F\xD3\xE2\xEE" - "\xD0\xCD\x63\x30\xA9\xC0\xDD\x49" - "\xFE\x16\xA4\x07\x0D\xE2\x5D\x97" - "\xDE\x89\xBA\x2E\xF3\xA9\x5E\xBE" - "\x03\x55\x0E\x02\x41\x4A\x45\x06" - "\xBE\xEA\x32\xF2\xDC\x91\x5C\x20" - "\x94\x02\x30\xD2\xFC\x29\xFA\x8E" - "\x34\xA0\x31\xB8\x34\xBA\xAE\x54" - "\xB5\x88\x1F\xDC\x43\xDC\x22\x9F" - "\xDC\xCE\xD3\xFA\xA4\xA8\xBC\x8A" - "\xC7\x5A\x43\x21\xA5\xB1\xDB\xC3" - "\x84\x3B\xB4\x9B\xB5\xA7\xF1\x0A" - "\xB6\x37\x21\x19\x55\xC2\xBD\x99" - "\x49\x24\xBB\x7C\xB3\x8E\xEF\xD2" - "\x3A\xCF\xA0\x31\x28\x0E\x25\xA2" - "\x11\xB4\x18\x17\x1A\x65\x92\x56" - "\xE8\xE0\x52\x9C\x61\x18\x2A\xB1" - "\x1A\x01\x22\x45\x17\x62\x52\x6C" - "\x91\x44\xCF\x98\xC7\xC0\x79\x26" - "\x32\x66\x6F\x23\x7F\x94\x36\x88" - "\x3C\xC9\xD0\xB7\x45\x30\x31\x86" - "\x3D\xC6\xA3\x98\x62\x84\x1A\x8B" - "\x16\x88\xC7\xA3\xE9\x4F\xE0\x86" - "\xA4\x93\xA8\x34\x5A\xCA\xDF\xCA" - "\x46\x38\xD2\xF4\xE0\x2D\x1E\xC9" - "\x7C\xEF\x53\xB7\x60\x72\x41\xBF" - "\x29\x00\x87\x02\xAF\x44\x4C\xB7" - "\x8C\xF5\x3F\x19\xF4\x80\x45\xA7" - "\x15\x5F\xDB\xE9\xB1\x83\xD2\xE6" - "\x1D\x18\x66\x44\x5B\x8F\x14\xEB", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast5_ctr_enc_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", - .klen = 16, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A", - .ilen = 17, - .result = "\xFF\xC4\x2E\x82\x3D\xF8\xA8\x39" - "\x7C\x52\xC4\xD3\xBB\x62\xC6\xA8" - "\x0C", - .rlen = 17, - }, { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", - .klen = 16, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\xFF\xC4\x2E\x82\x3D\xF8\xA8\x39" - "\x7C\x52\xC4\xD3\xBB\x62\xC6\xA8" - "\x0C\x63\xA5\x55\xE3\xF8\x1C\x7F" - "\xDC\x59\xF9\xA0\x52\xAD\x83\xDF" - "\xD5\x3B\x53\x4A\xAA\x1F\x49\x44" - "\xE8\x20\xCC\xF8\x97\xE6\xE0\x3C" - "\x5A\xD2\x83\xEC\xEE\x25\x3F\xCF" - "\x0D\xC2\x79\x80\x99\x6E\xFF\x7B" - "\x64\xB0\x7B\x86\x29\x1D\x9F\x17" - "\x10\xA5\xA5\xEB\x16\x55\x9E\xE3" - "\x88\x18\x52\x56\x48\x58\xD1\x6B" - "\xE8\x74\x6E\x48\xB0\x2E\x69\x63" - "\x32\xAA\xAC\x26\x55\x45\x94\xDE" - "\x30\x26\x26\xE6\x08\x82\x2F\x5F" - "\xA7\x15\x94\x07\x75\x2D\xC6\x3A" - "\x1B\xA0\x39\xFB\xBA\xB9\x06\x56" - "\xF6\x9F\xF1\x2F\x9B\xF3\x89\x8B" - "\x08\xC8\x9D\x5E\x6B\x95\x09\xC7" - "\x98\xB7\x62\xA4\x1D\x25\xFA\xC5" - "\x62\xC8\x5D\x6B\xB4\x85\x88\x7F" - "\x3B\x29\xF9\xB4\x32\x62\x69\xBF" - "\x32\xB8\xEB\xFD\x0E\x26\xAA\xA3" - "\x44\x67\x90\x20\xAC\x41\xDF\x43" - "\xC6\xC7\x19\x9F\x2C\x28\x74\xEB" - "\x3E\x7F\x7A\x80\x5B\xE4\x08\x60" - "\xC7\xC9\x71\x34\x44\xCE\x05\xFD" - "\xA8\x91\xA8\x44\x5E\xD3\x89\x2C" - "\xAE\x59\x0F\x07\x88\x79\x53\x26" - "\xAF\xAC\xCB\x1D\x6F\x08\x25\x62" - "\xD0\x82\x65\x66\xE4\x2A\x29\x1C" - "\x9C\x64\x5F\x49\x9D\xF8\x62\xF9" - "\xED\xC4\x13\x52\x75\xDC\xE4\xF9" - "\x68\x0F\x8A\xCD\xA6\x8D\x75\xAA" - "\x49\xA1\x86\x86\x37\x5C\x6B\x3D" - "\x56\xE5\x6F\xBE\x27\xC0\x10\xF8" - "\x3C\x4D\x17\x35\x14\xDC\x1C\xA0" - "\x6E\xAE\xD1\x10\xDD\x83\x06\xC2" - "\x23\xD3\xC7\x27\x15\x04\x2C\x27" - "\xDD\x1F\x2E\x97\x09\x9C\x33\x7D" - "\xAC\x50\x1B\x2E\xC9\x52\x0C\x14" - "\x4B\x78\xC4\xDE\x07\x6A\x12\x02" - "\x6E\xD7\x4B\x91\xB9\x88\x4D\x02" - "\xC3\xB5\x04\xBC\xE0\x67\xCA\x18" - "\x22\xA1\xAE\x9A\x21\xEF\xB2\x06" - "\x35\xCD\xEC\x37\x70\x2D\xFC\x1E" - "\xA8\x31\xE7\xFC\xE5\x8E\x88\x66" - "\x16\xB5\xC8\x45\x21\x37\xBD\x24" - "\xA9\xD5\x36\x12\x9F\x6E\x67\x80" - "\x87\x54\xD5\xAF\x97\xE1\x15\xA7" - "\x11\xF0\x63\x7B\xE1\x44\x14\x1C" - "\x06\x32\x05\x8C\x6C\xDB\x9B\x36" - "\x6A\x6B\xAD\x3A\x27\x55\x20\x4C" - "\x76\x36\x43\xE8\x16\x60\xB5\xF3" - "\xDF\x5A\xC6\xA5\x69\x78\x59\x51" - "\x54\x68\x65\x06\x84\xDE\x3D\xAE" - "\x38\x91\xBD\xCC\xA2\x8A\xEC\xE6" - "\x9E\x83\xAE\x1E\x8E\x34\x5D\xDE" - "\x91\xCE\x8F\xED\x40\xF7\xC8\x8B" - "\x9A\x13\x4C\xAD\x89\x97\x9E\xD1" - "\x91\x01\xD7\x21\x23\x28\x1E\xCC" - "\x8C\x98\xDB\xDE\xFC\x72\x94\xAA" - "\xC0\x0D\x96\xAA\x23\xF8\xFE\x13", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec cast5_ctr_dec_tv_template[] = { - { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", - .klen = 16, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\xFF\xC4\x2E\x82\x3D\xF8\xA8\x39" - "\x7C\x52\xC4\xD3\xBB\x62\xC6\xA8" - "\x0C", - .ilen = 17, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A", - .rlen = 17, - }, { /* Generated from TF test vectors */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", - .klen = 16, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", - .input = "\xFF\xC4\x2E\x82\x3D\xF8\xA8\x39" - "\x7C\x52\xC4\xD3\xBB\x62\xC6\xA8" - "\x0C\x63\xA5\x55\xE3\xF8\x1C\x7F" - "\xDC\x59\xF9\xA0\x52\xAD\x83\xDF" - "\xD5\x3B\x53\x4A\xAA\x1F\x49\x44" - "\xE8\x20\xCC\xF8\x97\xE6\xE0\x3C" - "\x5A\xD2\x83\xEC\xEE\x25\x3F\xCF" - "\x0D\xC2\x79\x80\x99\x6E\xFF\x7B" - "\x64\xB0\x7B\x86\x29\x1D\x9F\x17" - "\x10\xA5\xA5\xEB\x16\x55\x9E\xE3" - "\x88\x18\x52\x56\x48\x58\xD1\x6B" - "\xE8\x74\x6E\x48\xB0\x2E\x69\x63" - "\x32\xAA\xAC\x26\x55\x45\x94\xDE" - "\x30\x26\x26\xE6\x08\x82\x2F\x5F" - "\xA7\x15\x94\x07\x75\x2D\xC6\x3A" - "\x1B\xA0\x39\xFB\xBA\xB9\x06\x56" - "\xF6\x9F\xF1\x2F\x9B\xF3\x89\x8B" - "\x08\xC8\x9D\x5E\x6B\x95\x09\xC7" - "\x98\xB7\x62\xA4\x1D\x25\xFA\xC5" - "\x62\xC8\x5D\x6B\xB4\x85\x88\x7F" - "\x3B\x29\xF9\xB4\x32\x62\x69\xBF" - "\x32\xB8\xEB\xFD\x0E\x26\xAA\xA3" - "\x44\x67\x90\x20\xAC\x41\xDF\x43" - "\xC6\xC7\x19\x9F\x2C\x28\x74\xEB" - "\x3E\x7F\x7A\x80\x5B\xE4\x08\x60" - "\xC7\xC9\x71\x34\x44\xCE\x05\xFD" - "\xA8\x91\xA8\x44\x5E\xD3\x89\x2C" - "\xAE\x59\x0F\x07\x88\x79\x53\x26" - "\xAF\xAC\xCB\x1D\x6F\x08\x25\x62" - "\xD0\x82\x65\x66\xE4\x2A\x29\x1C" - "\x9C\x64\x5F\x49\x9D\xF8\x62\xF9" - "\xED\xC4\x13\x52\x75\xDC\xE4\xF9" - "\x68\x0F\x8A\xCD\xA6\x8D\x75\xAA" - "\x49\xA1\x86\x86\x37\x5C\x6B\x3D" - "\x56\xE5\x6F\xBE\x27\xC0\x10\xF8" - "\x3C\x4D\x17\x35\x14\xDC\x1C\xA0" - "\x6E\xAE\xD1\x10\xDD\x83\x06\xC2" - "\x23\xD3\xC7\x27\x15\x04\x2C\x27" - "\xDD\x1F\x2E\x97\x09\x9C\x33\x7D" - "\xAC\x50\x1B\x2E\xC9\x52\x0C\x14" - "\x4B\x78\xC4\xDE\x07\x6A\x12\x02" - "\x6E\xD7\x4B\x91\xB9\x88\x4D\x02" - "\xC3\xB5\x04\xBC\xE0\x67\xCA\x18" - "\x22\xA1\xAE\x9A\x21\xEF\xB2\x06" - "\x35\xCD\xEC\x37\x70\x2D\xFC\x1E" - "\xA8\x31\xE7\xFC\xE5\x8E\x88\x66" - "\x16\xB5\xC8\x45\x21\x37\xBD\x24" - "\xA9\xD5\x36\x12\x9F\x6E\x67\x80" - "\x87\x54\xD5\xAF\x97\xE1\x15\xA7" - "\x11\xF0\x63\x7B\xE1\x44\x14\x1C" - "\x06\x32\x05\x8C\x6C\xDB\x9B\x36" - "\x6A\x6B\xAD\x3A\x27\x55\x20\x4C" - "\x76\x36\x43\xE8\x16\x60\xB5\xF3" - "\xDF\x5A\xC6\xA5\x69\x78\x59\x51" - "\x54\x68\x65\x06\x84\xDE\x3D\xAE" - "\x38\x91\xBD\xCC\xA2\x8A\xEC\xE6" - "\x9E\x83\xAE\x1E\x8E\x34\x5D\xDE" - "\x91\xCE\x8F\xED\x40\xF7\xC8\x8B" - "\x9A\x13\x4C\xAD\x89\x97\x9E\xD1" - "\x91\x01\xD7\x21\x23\x28\x1E\xCC" - "\x8C\x98\xDB\xDE\xFC\x72\x94\xAA" - "\xC0\x0D\x96\xAA\x23\xF8\xFE\x13", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - .also_non_np = 1, - .np = 3, - .tap = { 496 - 20, 4, 16 }, - }, -}; - -/* - * ARC4 test vectors from OpenSSL - */ -static const struct cipher_testvec arc4_enc_tv_template[] = { - { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .ilen = 8, - .result = "\x75\xb7\x87\x80\x99\xe0\xc5\x96", - .rlen = 8, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .input = "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 8, - .result = "\x74\x94\xc2\xe7\x10\x4b\x08\x79", - .rlen = 8, - }, { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 8, - .input = "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 8, - .result = "\xde\x18\x89\x41\xa3\x37\x5d\x3a", - .rlen = 8, - }, { - .key = "\xef\x01\x23\x45", - .klen = 4, - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00", - .ilen = 20, - .result = "\xd6\xa1\x41\xa7\xec\x3c\x38\xdf" - "\xbd\x61\x5a\x11\x62\xe1\xc7\xba" - "\x36\xb6\x78\x58", - .rlen = 20, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .input = "\x12\x34\x56\x78\x9A\xBC\xDE\xF0" - "\x12\x34\x56\x78\x9A\xBC\xDE\xF0" - "\x12\x34\x56\x78\x9A\xBC\xDE\xF0" - "\x12\x34\x56\x78", - .ilen = 28, - .result = "\x66\xa0\x94\x9f\x8a\xf7\xd6\x89" - "\x1f\x7f\x83\x2b\xa8\x33\xc0\x0c" - "\x89\x2e\xbe\x30\x14\x3c\xe2\x87" - "\x40\x01\x1e\xcf", - .rlen = 28, - }, { - .key = "\xef\x01\x23\x45", - .klen = 4, - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00", - .ilen = 10, - .result = "\xd6\xa1\x41\xa7\xec\x3c\x38\xdf" - "\xbd\x61", - .rlen = 10, - }, { - .key = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 16, - .input = "\x01\x23\x45\x67\x89\xAB\xCD\xEF", - .ilen = 8, - .result = "\x69\x72\x36\x59\x1B\x52\x42\xB1", - .rlen = 8, - }, -}; - -static const struct cipher_testvec arc4_dec_tv_template[] = { - { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .input = "\x75\xb7\x87\x80\x99\xe0\xc5\x96", - .ilen = 8, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .rlen = 8, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .input = "\x74\x94\xc2\xe7\x10\x4b\x08\x79", - .ilen = 8, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 8, - }, { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 8, - .input = "\xde\x18\x89\x41\xa3\x37\x5d\x3a", - .ilen = 8, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 8, - }, { - .key = "\xef\x01\x23\x45", - .klen = 4, - .input = "\xd6\xa1\x41\xa7\xec\x3c\x38\xdf" - "\xbd\x61\x5a\x11\x62\xe1\xc7\xba" - "\x36\xb6\x78\x58", - .ilen = 20, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00", - .rlen = 20, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", - .klen = 8, - .input = "\x66\xa0\x94\x9f\x8a\xf7\xd6\x89" - "\x1f\x7f\x83\x2b\xa8\x33\xc0\x0c" - "\x89\x2e\xbe\x30\x14\x3c\xe2\x87" - "\x40\x01\x1e\xcf", - .ilen = 28, - .result = "\x12\x34\x56\x78\x9A\xBC\xDE\xF0" - "\x12\x34\x56\x78\x9A\xBC\xDE\xF0" - "\x12\x34\x56\x78\x9A\xBC\xDE\xF0" - "\x12\x34\x56\x78", - .rlen = 28, - }, { - .key = "\xef\x01\x23\x45", - .klen = 4, - .input = "\xd6\xa1\x41\xa7\xec\x3c\x38\xdf" - "\xbd\x61", - .ilen = 10, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00", - .rlen = 10, - }, { - .key = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 16, - .input = "\x69\x72\x36\x59\x1B\x52\x42\xB1", - .ilen = 8, - .result = "\x01\x23\x45\x67\x89\xAB\xCD\xEF", - .rlen = 8, - }, -}; - -/* - * TEA test vectors - */ -static const struct cipher_testvec tea_enc_tv_template[] = { - { - .key = zeroed_string, - .klen = 16, - .input = zeroed_string, - .ilen = 8, - .result = "\x0a\x3a\xea\x41\x40\xa9\xba\x94", - .rlen = 8, - }, { - .key = "\x2b\x02\x05\x68\x06\x14\x49\x76" - "\x77\x5d\x0e\x26\x6c\x28\x78\x43", - .klen = 16, - .input = "\x74\x65\x73\x74\x20\x6d\x65\x2e", - .ilen = 8, - .result = "\x77\x5d\x2a\x6a\xf6\xce\x92\x09", - .rlen = 8, - }, { - .key = "\x09\x65\x43\x11\x66\x44\x39\x25" - "\x51\x3a\x16\x10\x0a\x08\x12\x6e", - .klen = 16, - .input = "\x6c\x6f\x6e\x67\x65\x72\x5f\x74" - "\x65\x73\x74\x5f\x76\x65\x63\x74", - .ilen = 16, - .result = "\xbe\x7a\xbb\x81\x95\x2d\x1f\x1e" - "\xdd\x89\xa1\x25\x04\x21\xdf\x95", - .rlen = 16, - }, { - .key = "\x4d\x76\x32\x17\x05\x3f\x75\x2c" - "\x5d\x04\x16\x36\x15\x72\x63\x2f", - .klen = 16, - .input = "\x54\x65\x61\x20\x69\x73\x20\x67" - "\x6f\x6f\x64\x20\x66\x6f\x72\x20" - "\x79\x6f\x75\x21\x21\x21\x20\x72" - "\x65\x61\x6c\x6c\x79\x21\x21\x21", - .ilen = 32, - .result = "\xe0\x4d\x5d\x3c\xb7\x8c\x36\x47" - "\x94\x18\x95\x91\xa9\xfc\x49\xf8" - "\x44\xd1\x2d\xc2\x99\xb8\x08\x2a" - "\x07\x89\x73\xc2\x45\x92\xc6\x90", - .rlen = 32, - } -}; - -static const struct cipher_testvec tea_dec_tv_template[] = { - { - .key = zeroed_string, - .klen = 16, - .input = "\x0a\x3a\xea\x41\x40\xa9\xba\x94", - .ilen = 8, - .result = zeroed_string, - .rlen = 8, - }, { - .key = "\x2b\x02\x05\x68\x06\x14\x49\x76" - "\x77\x5d\x0e\x26\x6c\x28\x78\x43", - .klen = 16, - .input = "\x77\x5d\x2a\x6a\xf6\xce\x92\x09", - .ilen = 8, - .result = "\x74\x65\x73\x74\x20\x6d\x65\x2e", - .rlen = 8, - }, { - .key = "\x09\x65\x43\x11\x66\x44\x39\x25" - "\x51\x3a\x16\x10\x0a\x08\x12\x6e", - .klen = 16, - .input = "\xbe\x7a\xbb\x81\x95\x2d\x1f\x1e" - "\xdd\x89\xa1\x25\x04\x21\xdf\x95", - .ilen = 16, - .result = "\x6c\x6f\x6e\x67\x65\x72\x5f\x74" - "\x65\x73\x74\x5f\x76\x65\x63\x74", - .rlen = 16, - }, { - .key = "\x4d\x76\x32\x17\x05\x3f\x75\x2c" - "\x5d\x04\x16\x36\x15\x72\x63\x2f", - .klen = 16, - .input = "\xe0\x4d\x5d\x3c\xb7\x8c\x36\x47" - "\x94\x18\x95\x91\xa9\xfc\x49\xf8" - "\x44\xd1\x2d\xc2\x99\xb8\x08\x2a" - "\x07\x89\x73\xc2\x45\x92\xc6\x90", - .ilen = 32, - .result = "\x54\x65\x61\x20\x69\x73\x20\x67" - "\x6f\x6f\x64\x20\x66\x6f\x72\x20" - "\x79\x6f\x75\x21\x21\x21\x20\x72" - "\x65\x61\x6c\x6c\x79\x21\x21\x21", - .rlen = 32, - } -}; - -/* - * XTEA test vectors - */ -static const struct cipher_testvec xtea_enc_tv_template[] = { - { - .key = zeroed_string, - .klen = 16, - .input = zeroed_string, - .ilen = 8, - .result = "\xd8\xd4\xe9\xde\xd9\x1e\x13\xf7", - .rlen = 8, - }, { - .key = "\x2b\x02\x05\x68\x06\x14\x49\x76" - "\x77\x5d\x0e\x26\x6c\x28\x78\x43", - .klen = 16, - .input = "\x74\x65\x73\x74\x20\x6d\x65\x2e", - .ilen = 8, - .result = "\x94\xeb\xc8\x96\x84\x6a\x49\xa8", - .rlen = 8, - }, { - .key = "\x09\x65\x43\x11\x66\x44\x39\x25" - "\x51\x3a\x16\x10\x0a\x08\x12\x6e", - .klen = 16, - .input = "\x6c\x6f\x6e\x67\x65\x72\x5f\x74" - "\x65\x73\x74\x5f\x76\x65\x63\x74", - .ilen = 16, - .result = "\x3e\xce\xae\x22\x60\x56\xa8\x9d" - "\x77\x4d\xd4\xb4\x87\x24\xe3\x9a", - .rlen = 16, - }, { - .key = "\x4d\x76\x32\x17\x05\x3f\x75\x2c" - "\x5d\x04\x16\x36\x15\x72\x63\x2f", - .klen = 16, - .input = "\x54\x65\x61\x20\x69\x73\x20\x67" - "\x6f\x6f\x64\x20\x66\x6f\x72\x20" - "\x79\x6f\x75\x21\x21\x21\x20\x72" - "\x65\x61\x6c\x6c\x79\x21\x21\x21", - .ilen = 32, - .result = "\x99\x81\x9f\x5d\x6f\x4b\x31\x3a" - "\x86\xff\x6f\xd0\xe3\x87\x70\x07" - "\x4d\xb8\xcf\xf3\x99\x50\xb3\xd4" - "\x73\xa2\xfa\xc9\x16\x59\x5d\x81", - .rlen = 32, - } -}; - -static const struct cipher_testvec xtea_dec_tv_template[] = { - { - .key = zeroed_string, - .klen = 16, - .input = "\xd8\xd4\xe9\xde\xd9\x1e\x13\xf7", - .ilen = 8, - .result = zeroed_string, - .rlen = 8, - }, { - .key = "\x2b\x02\x05\x68\x06\x14\x49\x76" - "\x77\x5d\x0e\x26\x6c\x28\x78\x43", - .klen = 16, - .input = "\x94\xeb\xc8\x96\x84\x6a\x49\xa8", - .ilen = 8, - .result = "\x74\x65\x73\x74\x20\x6d\x65\x2e", - .rlen = 8, - }, { - .key = "\x09\x65\x43\x11\x66\x44\x39\x25" - "\x51\x3a\x16\x10\x0a\x08\x12\x6e", - .klen = 16, - .input = "\x3e\xce\xae\x22\x60\x56\xa8\x9d" - "\x77\x4d\xd4\xb4\x87\x24\xe3\x9a", - .ilen = 16, - .result = "\x6c\x6f\x6e\x67\x65\x72\x5f\x74" - "\x65\x73\x74\x5f\x76\x65\x63\x74", - .rlen = 16, - }, { - .key = "\x4d\x76\x32\x17\x05\x3f\x75\x2c" - "\x5d\x04\x16\x36\x15\x72\x63\x2f", - .klen = 16, - .input = "\x99\x81\x9f\x5d\x6f\x4b\x31\x3a" - "\x86\xff\x6f\xd0\xe3\x87\x70\x07" - "\x4d\xb8\xcf\xf3\x99\x50\xb3\xd4" - "\x73\xa2\xfa\xc9\x16\x59\x5d\x81", - .ilen = 32, - .result = "\x54\x65\x61\x20\x69\x73\x20\x67" - "\x6f\x6f\x64\x20\x66\x6f\x72\x20" - "\x79\x6f\x75\x21\x21\x21\x20\x72" - "\x65\x61\x6c\x6c\x79\x21\x21\x21", - .rlen = 32, - } -}; - -/* - * KHAZAD test vectors. - */ -static const struct cipher_testvec khazad_enc_tv_template[] = { - { - .key = "\x80\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 16, - .input = "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 8, - .result = "\x49\xa4\xce\x32\xac\x19\x0e\x3f", - .rlen = 8, - }, { - .key = "\x38\x38\x38\x38\x38\x38\x38\x38" - "\x38\x38\x38\x38\x38\x38\x38\x38", - .klen = 16, - .input = "\x38\x38\x38\x38\x38\x38\x38\x38", - .ilen = 8, - .result = "\x7e\x82\x12\xa1\xd9\x5b\xe4\xf9", - .rlen = 8, - }, { - .key = "\xa2\xa2\xa2\xa2\xa2\xa2\xa2\xa2" - "\xa2\xa2\xa2\xa2\xa2\xa2\xa2\xa2", - .klen = 16, - .input = "\xa2\xa2\xa2\xa2\xa2\xa2\xa2\xa2", - .ilen = 8, - .result = "\xaa\xbe\xc1\x95\xc5\x94\x1a\x9c", - .rlen = 8, - }, { - .key = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f" - "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", - .klen = 16, - .input = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", - .ilen = 8, - .result = "\x04\x74\xf5\x70\x50\x16\xd3\xb8", - .rlen = 8, - }, { - .key = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f" - "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", - .klen = 16, - .input = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f" - "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", - .ilen = 16, - .result = "\x04\x74\xf5\x70\x50\x16\xd3\xb8" - "\x04\x74\xf5\x70\x50\x16\xd3\xb8", - .rlen = 16, - }, -}; - -static const struct cipher_testvec khazad_dec_tv_template[] = { - { - .key = "\x80\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 16, - .input = "\x49\xa4\xce\x32\xac\x19\x0e\x3f", - .ilen = 8, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 8, - }, { - .key = "\x38\x38\x38\x38\x38\x38\x38\x38" - "\x38\x38\x38\x38\x38\x38\x38\x38", - .klen = 16, - .input = "\x7e\x82\x12\xa1\xd9\x5b\xe4\xf9", - .ilen = 8, - .result = "\x38\x38\x38\x38\x38\x38\x38\x38", - .rlen = 8, - }, { - .key = "\xa2\xa2\xa2\xa2\xa2\xa2\xa2\xa2" - "\xa2\xa2\xa2\xa2\xa2\xa2\xa2\xa2", - .klen = 16, - .input = "\xaa\xbe\xc1\x95\xc5\x94\x1a\x9c", - .ilen = 8, - .result = "\xa2\xa2\xa2\xa2\xa2\xa2\xa2\xa2", - .rlen = 8, - }, { - .key = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f" - "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", - .klen = 16, - .input = "\x04\x74\xf5\x70\x50\x16\xd3\xb8", - .ilen = 8, - .result = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", - .rlen = 8, - }, { - .key = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f" - "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", - .klen = 16, - .input = "\x04\x74\xf5\x70\x50\x16\xd3\xb8" - "\x04\x74\xf5\x70\x50\x16\xd3\xb8", - .ilen = 16, - .result = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f" - "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", - .rlen = 16, - }, -}; - -/* - * Anubis test vectors. - */ - -static const struct cipher_testvec anubis_enc_tv_template[] = { - { - .key = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", - .klen = 16, - .input = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", - .ilen = 16, - .result = "\x6d\xc5\xda\xa2\x26\x7d\x62\x6f" - "\x08\xb7\x52\x8e\x6e\x6e\x86\x90", - .rlen = 16, - }, { - - .key = "\x03\x03\x03\x03\x03\x03\x03\x03" - "\x03\x03\x03\x03\x03\x03\x03\x03" - "\x03\x03\x03\x03", - .klen = 20, - .input = "\x03\x03\x03\x03\x03\x03\x03\x03" - "\x03\x03\x03\x03\x03\x03\x03\x03", - .ilen = 16, - .result = "\xdb\xf1\x42\xf4\xd1\x8a\xc7\x49" - "\x87\x41\x6f\x82\x0a\x98\x64\xae", - .rlen = 16, - }, { - .key = "\x24\x24\x24\x24\x24\x24\x24\x24" - "\x24\x24\x24\x24\x24\x24\x24\x24" - "\x24\x24\x24\x24\x24\x24\x24\x24" - "\x24\x24\x24\x24", - .klen = 28, - .input = "\x24\x24\x24\x24\x24\x24\x24\x24" - "\x24\x24\x24\x24\x24\x24\x24\x24", - .ilen = 16, - .result = "\xfd\x1b\x4a\xe3\xbf\xf0\xad\x3d" - "\x06\xd3\x61\x27\xfd\x13\x9e\xde", - .rlen = 16, - }, { - .key = "\x25\x25\x25\x25\x25\x25\x25\x25" - "\x25\x25\x25\x25\x25\x25\x25\x25" - "\x25\x25\x25\x25\x25\x25\x25\x25" - "\x25\x25\x25\x25\x25\x25\x25\x25", - .klen = 32, - .input = "\x25\x25\x25\x25\x25\x25\x25\x25" - "\x25\x25\x25\x25\x25\x25\x25\x25", - .ilen = 16, - .result = "\x1a\x91\xfb\x2b\xb7\x78\x6b\xc4" - "\x17\xd9\xff\x40\x3b\x0e\xe5\xfe", - .rlen = 16, - }, { - .key = "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35", - .klen = 40, - .input = "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35", - .ilen = 16, - .result = "\xa5\x2c\x85\x6f\x9c\xba\xa0\x97" - "\x9e\xc6\x84\x0f\x17\x21\x07\xee", - .rlen = 16, - }, -}; - -static const struct cipher_testvec anubis_dec_tv_template[] = { - { - .key = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", - .klen = 16, - .input = "\x6d\xc5\xda\xa2\x26\x7d\x62\x6f" - "\x08\xb7\x52\x8e\x6e\x6e\x86\x90", - .ilen = 16, - .result = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", - .rlen = 16, - }, { - - .key = "\x03\x03\x03\x03\x03\x03\x03\x03" - "\x03\x03\x03\x03\x03\x03\x03\x03" - "\x03\x03\x03\x03", - .klen = 20, - .input = "\xdb\xf1\x42\xf4\xd1\x8a\xc7\x49" - "\x87\x41\x6f\x82\x0a\x98\x64\xae", - .ilen = 16, - .result = "\x03\x03\x03\x03\x03\x03\x03\x03" - "\x03\x03\x03\x03\x03\x03\x03\x03", - .rlen = 16, - }, { - .key = "\x24\x24\x24\x24\x24\x24\x24\x24" - "\x24\x24\x24\x24\x24\x24\x24\x24" - "\x24\x24\x24\x24\x24\x24\x24\x24" - "\x24\x24\x24\x24", - .klen = 28, - .input = "\xfd\x1b\x4a\xe3\xbf\xf0\xad\x3d" - "\x06\xd3\x61\x27\xfd\x13\x9e\xde", - .ilen = 16, - .result = "\x24\x24\x24\x24\x24\x24\x24\x24" - "\x24\x24\x24\x24\x24\x24\x24\x24", - .rlen = 16, - }, { - .key = "\x25\x25\x25\x25\x25\x25\x25\x25" - "\x25\x25\x25\x25\x25\x25\x25\x25" - "\x25\x25\x25\x25\x25\x25\x25\x25" - "\x25\x25\x25\x25\x25\x25\x25\x25", - .klen = 32, - .input = "\x1a\x91\xfb\x2b\xb7\x78\x6b\xc4" - "\x17\xd9\xff\x40\x3b\x0e\xe5\xfe", - .ilen = 16, - .result = "\x25\x25\x25\x25\x25\x25\x25\x25" - "\x25\x25\x25\x25\x25\x25\x25\x25", - .rlen = 16, - }, { - .key = "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35", - .input = "\xa5\x2c\x85\x6f\x9c\xba\xa0\x97" - "\x9e\xc6\x84\x0f\x17\x21\x07\xee", - .klen = 40, - .ilen = 16, - .result = "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35", - .rlen = 16, - }, -}; - -static const struct cipher_testvec anubis_cbc_enc_tv_template[] = { - { - .key = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", - .klen = 16, - .input = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", - .ilen = 32, - .result = "\x6d\xc5\xda\xa2\x26\x7d\x62\x6f" - "\x08\xb7\x52\x8e\x6e\x6e\x86\x90" - "\x86\xd8\xb5\x6f\x98\x5e\x8a\x66" - "\x4f\x1f\x78\xa1\xbb\x37\xf1\xbe", - .rlen = 32, - }, { - .key = "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35", - .klen = 40, - .input = "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35", - .ilen = 32, - .result = "\xa5\x2c\x85\x6f\x9c\xba\xa0\x97" - "\x9e\xc6\x84\x0f\x17\x21\x07\xee" - "\xa2\xbc\x06\x98\xc6\x4b\xda\x75" - "\x2e\xaa\xbe\x58\xce\x01\x5b\xc7", - .rlen = 32, - }, -}; - -static const struct cipher_testvec anubis_cbc_dec_tv_template[] = { - { - .key = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", - .klen = 16, - .input = "\x6d\xc5\xda\xa2\x26\x7d\x62\x6f" - "\x08\xb7\x52\x8e\x6e\x6e\x86\x90" - "\x86\xd8\xb5\x6f\x98\x5e\x8a\x66" - "\x4f\x1f\x78\xa1\xbb\x37\xf1\xbe", - .ilen = 32, - .result = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" - "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", - .rlen = 32, - }, { - .key = "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35", - .klen = 40, - .input = "\xa5\x2c\x85\x6f\x9c\xba\xa0\x97" - "\x9e\xc6\x84\x0f\x17\x21\x07\xee" - "\xa2\xbc\x06\x98\xc6\x4b\xda\x75" - "\x2e\xaa\xbe\x58\xce\x01\x5b\xc7", - .ilen = 32, - .result = "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35" - "\x35\x35\x35\x35\x35\x35\x35\x35", - .rlen = 32, - }, -}; - -/* - * XETA test vectors - */ -static const struct cipher_testvec xeta_enc_tv_template[] = { - { - .key = zeroed_string, - .klen = 16, - .input = zeroed_string, - .ilen = 8, - .result = "\xaa\x22\x96\xe5\x6c\x61\xf3\x45", - .rlen = 8, - }, { - .key = "\x2b\x02\x05\x68\x06\x14\x49\x76" - "\x77\x5d\x0e\x26\x6c\x28\x78\x43", - .klen = 16, - .input = "\x74\x65\x73\x74\x20\x6d\x65\x2e", - .ilen = 8, - .result = "\x82\x3e\xeb\x35\xdc\xdd\xd9\xc3", - .rlen = 8, - }, { - .key = "\x09\x65\x43\x11\x66\x44\x39\x25" - "\x51\x3a\x16\x10\x0a\x08\x12\x6e", - .klen = 16, - .input = "\x6c\x6f\x6e\x67\x65\x72\x5f\x74" - "\x65\x73\x74\x5f\x76\x65\x63\x74", - .ilen = 16, - .result = "\xe2\x04\xdb\xf2\x89\x85\x9e\xea" - "\x61\x35\xaa\xed\xb5\xcb\x71\x2c", - .rlen = 16, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ctext = "\xa9\x78\xae\x1e\xea\xa2\x44\x4c" + "\xa2\x7a\x64\x1f\xaf\x46\xc1\xe0" + "\x6c\xb2\xf3\x92\x9a\xd6\x7d\x58" + "\xb8\x2d\xb9\x5d\x58\x07\x66\x50" + "\xea\x35\x35\x8c\xb2\x46\x61\x06" + "\x5d\x65\xfc\x57\x8f\x69\x74\xab" + "\x8a\x06\x69\xb5\x6c\xda\x66\xc7" + "\x52\x90\xbb\x8e\x6d\x8b\xb5\xa2" + "\x78\x1d\xc2\xa9\xc2\x73\x00\xc3" + "\x32\x36\x7c\x97\x6b\x4e\x8a\x50" + "\xe4\x91\x83\x96\x8f\xf4\x94\x1a" + "\xa6\x27\xe1\x33\xcb\x91\xc6\x5f" + "\x94\x75\xbc\xd7\x3e\x3e\x6f\x9e" + "\xa9\x31\x80\x5e\xe5\xdb\xc8\x53" + "\x01\x73\x68\x32\x25\x19\xfa\xfb" + "\xe4\xcf\xb9\x3e\xa2\xa0\x8f\x31" + "\xbf\x54\x06\x93\xa8\xb1\x0f\xb6" + "\x7c\x3c\xde\x6f\x0f\xfb\x0c\x11" + "\x39\x80\x39\x09\x97\x65\xf2\x83" + "\xae\xe6\xa1\x6f\x47\xb8\x49\xde" + "\x99\x36\x20\x7d\x97\x3b\xec\xfa" + "\xb4\x33\x6e\x7a\xc7\x46\x84\x49" + "\x91\xcd\xe1\x57\x0d\xed\x40\x08" + "\x13\xf1\x4e\x3e\xa4\xa4\x5c\xe6" + "\xd2\x0c\x20\x8f\x3e\xdf\x3f\x47" + "\x9a\x2f\xde\x6d\x66\xc9\x99\x4a" + "\x2d\x9e\x9d\x4b\x1a\x27\xa2\x12" + "\x99\xf0\xf8\xb1\xb6\xf6\x57\xc3" + "\xca\x1c\xa3\x8e\xed\x39\x28\xb5" + "\x10\x1b\x4b\x08\x42\x00\x4a\xd3" + "\xad\x5a\xc6\x8e\xc8\xbb\x95\xc4" + "\x4b\xaa\xfe\xd5\x42\xa8\xa3\x6d" + "\x3c\xf3\x34\x91\x2d\xb4\xdd\x20" + "\x0c\x90\x6d\xa3\x9b\x66\x9d\x24" + "\x02\xa6\xa9\x3f\x3f\x58\x5d\x47" + "\x24\x65\x63\x7e\xbd\x8c\xe6\x52" + "\x7d\xef\x33\x53\x63\xec\xaa\x0b" + "\x64\x15\xa9\xa6\x1f\x10\x00\x38" + "\x35\xa8\xe7\xbe\x23\x70\x22\xe0" + "\xd3\xb9\xe6\xfd\xe6\xaa\x03\x50" + "\xf3\x3c\x27\x36\x8b\xcc\xfe\x9c" + "\x9c\xa3\xb3\xe7\x68\x9b\xa2\x71" + "\xe0\x07\xd9\x1f\x68\x1f\xac\x5e" + "\x7a\x74\x85\xa9\x6a\x90\xab\x2c" + "\x38\x51\xbc\x1f\x43\x4a\x56\x1c" + "\xf8\x47\x03\x4e\x67\xa8\x1f\x99" + "\x04\x39\x73\x32\xb2\x86\x79\xe7" + "\x14\x28\x70\xb8\xe2\x7d\x69\x85" + "\xb6\x0f\xc5\xd0\xd0\x01\x5c\xe6" + "\x09\x0f\x75\xf7\xb6\x81\xd2\x11" + "\x20\x9c\xa1\xee\x11\x44\x79\xd0" + "\xb2\x34\x77\xda\x10\x9a\x6f\x6f" + "\xef\x7c\xd9\xdc\x35\xb7\x61\xdd" + "\xf1\xa4\xc6\x1c\xbf\x05\x22\xac" + "\xfe\x2f\x85\x00\x44\xdf\x33\x16" + "\x35\xb6\xa3\xd3\x70\xdf\x69\x35" + "\x6a\xc7\xb4\x99\x45\x27\xc8\x8e" + "\x5a\x14\x30\xd0\x55\x3e\x4f\x64" + "\x0d\x38\xe3\xdf\x8b\xa8\x93\x26" + "\x75\xae\xf6\xb5\x23\x0b\x17\x31" + "\xbf\x27\xb8\xb5\x94\x31\xa7\x8f" + "\x43\xc4\x46\x24\x22\x4f\x8f\x7e" + "\xe5\xf4\x6d\x1e\x0e\x18\x7a\xbb" + "\xa6\x8f\xfb\x49\x49\xd8\x7e\x5a", + .len = 512, }, { - .key = "\x4d\x76\x32\x17\x05\x3f\x75\x2c" - "\x5d\x04\x16\x36\x15\x72\x63\x2f", - .klen = 16, - .input = "\x54\x65\x61\x20\x69\x73\x20\x67" - "\x6f\x6f\x64\x20\x66\x6f\x72\x20" - "\x79\x6f\x75\x21\x21\x21\x20\x72" - "\x65\x61\x6c\x6c\x79\x21\x21\x21", - .ilen = 32, - .result = "\x0b\x03\xcd\x8a\xbe\x95\xfd\xb1" - "\xc1\x44\x91\x0b\xa5\xc9\x1b\xb4" - "\xa9\xda\x1e\x9e\xb1\x3e\x2a\x8f" - "\xea\xa5\x6a\x85\xd1\xf4\xa8\xa5", - .rlen = 32, - } + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x62\x49\x77\x57\x24\x70\x93\x69" + "\x99\x59\x57\x49\x66\x96\x76\x27" + "\x31\x41\x59\x26\x53\x58\x97\x93" + "\x23\x84\x62\x64\x33\x83\x27\x95" + "\x02\x88\x41\x97\x16\x93\x99\x37" + "\x51\x05\x82\x09\x74\x94\x45\x92", + .klen = 64, + .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ctext = "\xd7\x4b\x93\x7d\x13\xa2\xa2\xe1" + "\x35\x39\x71\x88\x76\x1e\xc9\xea" + "\x86\xad\xf3\x14\x48\x3d\x5e\xe9" + "\xe9\x2d\xb2\x56\x59\x35\x9d\xec" + "\x84\xfa\x7e\x9d\x6d\x33\x36\x8f" + "\xce\xf4\xa9\x21\x0b\x5f\x96\xec" + "\xcb\xf9\x57\x68\x33\x88\x39\xbf" + "\x2f\xbb\x59\x03\xbd\x66\x8b\x11" + "\x11\x65\x51\x2e\xb8\x67\x05\xd1" + "\x27\x11\x5c\xd4\xcc\x97\xc2\xb3" + "\xa9\x55\xaf\x07\x56\xd1\xdc\xf5" + "\x85\xdc\x46\xe6\xf0\x24\xeb\x93" + "\x4d\xf0\x9b\xf5\x73\x1c\xda\x03" + "\x22\xc8\x3a\x4f\xb4\x19\x91\x09" + "\x54\x0b\xf6\xfe\x17\x3d\x1a\x53" + "\x72\x60\x79\xcb\x0e\x32\x8a\x77" + "\xd5\xed\xdb\x33\xd7\x62\x16\x69" + "\x63\xe0\xab\xb5\xf6\x9c\x5f\x3d" + "\x69\x35\x61\x86\xf8\x86\xb9\x89" + "\x6e\x59\x35\xac\xf6\x6b\x33\xa0" + "\xea\xef\x96\x62\xd8\xa9\xcf\x56" + "\xbf\xdb\x8a\xfd\xa1\x82\x77\x73" + "\x3d\x94\x4a\x49\x42\x6d\x08\x60" + "\xa1\xea\xab\xb6\x88\x13\x94\xb8" + "\x51\x98\xdb\x35\x85\xdf\xf6\xb9" + "\x8f\xcd\xdf\x80\xd3\x40\x2d\x72" + "\xb8\xb2\x6c\x02\x43\x35\x22\x2a" + "\x31\xed\xcd\x16\x19\xdf\x62\x0f" + "\x29\xcf\x87\x04\xec\x02\x4f\xe4" + "\xa2\xed\x73\xc6\x69\xd3\x7e\x89" + "\x0b\x76\x10\x7c\xd6\xf9\x6a\x25" + "\xed\xcc\x60\x5d\x61\x20\xc1\x97" + "\x56\x91\x57\x28\xbe\x71\x0d\xcd" + "\xde\xc4\x9e\x55\x91\xbe\xd1\x28" + "\x9b\x90\xeb\x73\xf3\x68\x51\xc6" + "\xdf\x82\xcc\xd8\x1f\xce\x5b\x27" + "\xc0\x60\x5e\x33\xd6\xa7\x20\xea" + "\xb2\x54\xc7\x5d\x6a\x3b\x67\x47" + "\xcf\xa0\xe3\xab\x86\xaf\xc1\x42" + "\xe6\xb0\x23\x4a\xaf\x53\xdf\xa0" + "\xad\x12\x32\x31\x03\xf7\x21\xbe" + "\x2d\xd5\x82\x42\xb6\x4a\x3d\xcd" + "\xd8\x81\x77\xa9\x49\x98\x6c\x09" + "\xc5\xa3\x61\x12\x62\x85\x6b\xcd" + "\xb3\xf4\x20\x0c\x41\xc4\x05\x37" + "\x46\x5f\xeb\x71\x8b\xf1\xaf\x6e" + "\xba\xf3\x50\x2e\xfe\xa8\x37\xeb" + "\xe8\x8c\x4f\xa4\x0c\xf1\x31\xc8" + "\x6e\x71\x4f\xa5\xd7\x97\x73\xe0" + "\x93\x4a\x2f\xda\x7b\xe0\x20\x54" + "\x1f\x8d\x85\x79\x0b\x7b\x5e\x75" + "\xb9\x07\x67\xcc\xc8\xe7\x21\x15" + "\xa7\xc8\x98\xff\x4b\x80\x1c\x12" + "\xa8\x54\xe1\x38\x52\xe6\x74\x81" + "\x97\x47\xa1\x41\x0e\xc0\x50\xe3" + "\x55\x0e\xc3\xa7\x70\x77\xce\x07" + "\xed\x8c\x88\xe6\xa1\x5b\x14\xec" + "\xe6\xde\x06\x6d\x74\xc5\xd9\xfa" + "\xe5\x2f\x5a\xff\xc8\x05\xee\x27" + "\x35\x61\xbf\x0b\x19\x78\x9b\xd2" + "\x04\xc7\x05\xb1\x79\xb4\xff\x5f" + "\xf3\xea\x67\x52\x78\xc2\xce\x70" + "\xa4\x05\x0b\xb2\xb3\xa8\x30\x97" + "\x37\x30\xe1\x91\x8d\xb3\x2a\xff", + .len = 512, + .also_non_np = 1, + .np = 3, + .tap = { 512 - 20, 4, 16 }, + }, }; -static const struct cipher_testvec xeta_dec_tv_template[] = { +/* + * Serpent test vectors. These are backwards because Serpent writes + * octet sequences in right-to-left mode. + */ +static const struct cipher_testvec serpent_tv_template[] = { { - .key = zeroed_string, - .klen = 16, - .input = "\xaa\x22\x96\xe5\x6c\x61\xf3\x45", - .ilen = 8, - .result = zeroed_string, - .rlen = 8, + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .ctext = "\x12\x07\xfc\xce\x9b\xd0\xd6\x47" + "\x6a\xe9\x8f\xbe\xd1\x43\xa0\xe2", + .len = 16, }, { - .key = "\x2b\x02\x05\x68\x06\x14\x49\x76" - "\x77\x5d\x0e\x26\x6c\x28\x78\x43", + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", .klen = 16, - .input = "\x82\x3e\xeb\x35\xdc\xdd\xd9\xc3", - .ilen = 8, - .result = "\x74\x65\x73\x74\x20\x6d\x65\x2e", - .rlen = 8, + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .ctext = "\x4c\x7d\x8a\x32\x80\x72\xa2\x2c" + "\x82\x3e\x4a\x1f\x3a\xcd\xa1\x6d", + .len = 16, }, { - .key = "\x09\x65\x43\x11\x66\x44\x39\x25" - "\x51\x3a\x16\x10\x0a\x08\x12\x6e", - .klen = 16, - .input = "\xe2\x04\xdb\xf2\x89\x85\x9e\xea" - "\x61\x35\xaa\xed\xb5\xcb\x71\x2c", - .ilen = 16, - .result = "\x6c\x6f\x6e\x67\x65\x72\x5f\x74" - "\x65\x73\x74\x5f\x76\x65\x63\x74", - .rlen = 16, + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .klen = 32, + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .ctext = "\xde\x26\x9f\xf8\x33\xe4\x32\xb8" + "\x5b\x2e\x88\xd2\x70\x1c\xe7\x5c", + .len = 16, }, { - .key = "\x4d\x76\x32\x17\x05\x3f\x75\x2c" - "\x5d\x04\x16\x36\x15\x72\x63\x2f", + .key = "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80", .klen = 16, - .input = "\x0b\x03\xcd\x8a\xbe\x95\xfd\xb1" - "\xc1\x44\x91\x0b\xa5\xc9\x1b\xb4" - "\xa9\xda\x1e\x9e\xb1\x3e\x2a\x8f" - "\xea\xa5\x6a\x85\xd1\xf4\xa8\xa5", - .ilen = 32, - .result = "\x54\x65\x61\x20\x69\x73\x20\x67" - "\x6f\x6f\x64\x20\x66\x6f\x72\x20" - "\x79\x6f\x75\x21\x21\x21\x20\x72" - "\x65\x61\x6c\x6c\x79\x21\x21\x21", - .rlen = 32, - } -}; - -/* - * FCrypt test vectors - */ -static const struct cipher_testvec fcrypt_pcbc_enc_tv_template[] = { - { /* http://www.openafs.org/pipermail/openafs-devel/2000-December/005320.html */ - .key = "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 8, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 8, - .result = "\x0E\x09\x00\xC7\x3E\xF7\xED\x41", - .rlen = 8, - }, { - .key = "\x11\x44\x77\xAA\xDD\x00\x33\x66", - .klen = 8, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x12\x34\x56\x78\x9A\xBC\xDE\xF0", - .ilen = 8, - .result = "\xD8\xED\x78\x74\x77\xEC\x06\x80", - .rlen = 8, - }, { /* From Arla */ - .key = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", - .klen = 8, - .iv = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .input = "The quick brown fox jumps over the lazy dogs.\0\0", - .ilen = 48, - .result = "\x00\xf0\x0e\x11\x75\xe6\x23\x82" - "\xee\xac\x98\x62\x44\x51\xe4\x84" - "\xc3\x59\xd8\xaa\x64\x60\xae\xf7" - "\xd2\xd9\x13\x79\x72\xa3\x45\x03" - "\x23\xb5\x62\xd7\x0c\xf5\x27\xd1" - "\xf8\x91\x3c\xac\x44\x22\x92\xef", - .rlen = 48, - }, { - .key = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .klen = 8, - .iv = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", - .input = "The quick brown fox jumps over the lazy dogs.\0\0", - .ilen = 48, - .result = "\xca\x90\xf5\x9d\xcb\xd4\xd2\x3c" - "\x01\x88\x7f\x3e\x31\x6e\x62\x9d" - "\xd8\xe0\x57\xa3\x06\x3a\x42\x58" - "\x2a\x28\xfe\x72\x52\x2f\xdd\xe0" - "\x19\x89\x09\x1c\x2a\x8e\x8c\x94" - "\xfc\xc7\x68\xe4\x88\xaa\xde\x0f", - .rlen = 48, - }, { /* split-page version */ - .key = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .klen = 8, - .iv = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", - .input = "The quick brown fox jumps over the lazy dogs.\0\0", - .ilen = 48, - .result = "\xca\x90\xf5\x9d\xcb\xd4\xd2\x3c" - "\x01\x88\x7f\x3e\x31\x6e\x62\x9d" - "\xd8\xe0\x57\xa3\x06\x3a\x42\x58" - "\x2a\x28\xfe\x72\x52\x2f\xdd\xe0" - "\x19\x89\x09\x1c\x2a\x8e\x8c\x94" - "\xfc\xc7\x68\xe4\x88\xaa\xde\x0f", - .rlen = 48, - .np = 2, - .tap = { 20, 28 }, - } + .ptext = zeroed_string, + .ctext = "\xdd\xd2\x6b\x98\xa5\xff\xd8\x2c" + "\x05\x34\x5a\x9d\xad\xbf\xaf\x49", + .len = 16, + }, { /* Generated with Crypto++ */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .klen = 32, + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" + "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" + "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" + "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" + "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" + "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" + "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" + "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" + "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" + "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" + "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" + "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" + "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" + "\x29\xC0\x57\xEE\x62\xF9\x90\x04" + "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" + "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" + "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" + "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" + "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" + "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" + "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" + "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" + "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" + "\x57\xEE\x85\x1C\x90\x27\xBE\x32" + "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" + "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" + "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" + "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" + "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" + "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" + "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" + "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" + "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" + "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" + "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" + "\x69\x00\x74\x0B\xA2\x16\xAD\x44" + "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" + "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" + "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" + "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" + "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" + "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" + "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" + "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" + "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" + "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" + "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" + "\x58\xEF\x86\x1D\x91\x28\xBF\x33" + "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" + "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" + "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" + "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" + "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" + "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" + "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" + "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" + "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" + "\x86\x1D\xB4\x28\xBF\x56\xED\x61" + "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" + "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\xFB\xB0\x5D\xDE\xC0\xFE\xFC\xEB" + "\xB1\x80\x10\x43\xDE\x62\x70\xBD" + "\xFA\x8A\x93\xEA\x6B\xF7\xC5\xD7" + "\x0C\xD1\xBB\x29\x25\x14\x4C\x22" + "\x77\xA6\x38\x00\xDB\xB9\xE2\x07" + "\xD1\xAC\x82\xBA\xEA\x67\xAA\x39" + "\x99\x34\x89\x5B\x54\xE9\x12\x13" + "\x3B\x04\xE5\x12\x42\xC5\x79\xAB" + "\x0D\xC7\x3C\x58\x2D\xA3\x98\xF6" + "\xE4\x61\x9E\x17\x0B\xCE\xE8\xAA" + "\xB5\x6C\x1A\x3A\x67\x52\x81\x6A" + "\x04\xFF\x8A\x1B\x96\xFE\xE6\x87" + "\x3C\xD4\x39\x7D\x36\x9B\x03\xD5" + "\xB6\xA0\x75\x3C\x83\xE6\x1C\x73" + "\x9D\x74\x2B\x77\x53\x2D\xE5\xBD" + "\x69\xDA\x7A\x01\xF5\x6A\x70\x39" + "\x30\xD4\x2C\xF2\x8E\x06\x4B\x39" + "\xB3\x12\x1D\xB3\x17\x46\xE6\xD6" + "\xB6\x31\x36\x34\x38\x3C\x1D\x69" + "\x9F\x47\x28\x9A\x1D\x96\x70\x54" + "\x8E\x88\xCB\xE0\xF5\x6A\xAE\x0A" + "\x3C\xD5\x93\x1C\x21\xC9\x14\x3A" + "\x23\x9C\x9B\x79\xC7\x75\xC8\x39" + "\xA6\xAC\x65\x9A\x99\x37\xAF\x6D" + "\xBD\xB5\x32\xFD\xD8\x9C\x95\x7B" + "\xC6\x6A\x80\x64\xEA\xEF\x6D\x3F" + "\xA9\xFE\x5B\x16\xA3\xCF\x32\xC8" + "\xEF\x50\x22\x20\x93\x30\xBE\xE2" + "\x38\x05\x65\xAF\xBA\xB6\xE4\x72" + "\xA9\xEE\x05\x42\x88\xBD\x9D\x49" + "\xAD\x93\xCA\x4D\x45\x11\x43\x4D" + "\xB8\xF5\x74\x2B\x48\xE7\x21\xE4" + "\x4E\x3A\x4C\xDE\x65\x7A\x5A\xAD" + "\x86\xE6\x23\xEC\x6B\xA7\x17\xE6" + "\xF6\xA1\xAC\x29\xAE\xF9\x9B\x69" + "\x73\x65\x65\x51\xD6\x0B\x4E\x8C" + "\x17\x15\x9D\xB0\xCF\xB2\x42\x2B" + "\x51\xC3\x03\xE8\xB7\x7D\x2D\x39" + "\xE8\x10\x93\x16\xC8\x68\x4C\x60" + "\x87\x70\x14\xD0\x01\x57\xCB\x42" + "\x13\x59\xB1\x7F\x12\x4F\xBB\xC7" + "\xBD\x2B\xD4\xA9\x12\x26\x4F\xDE" + "\xFD\x72\xEC\xD7\x6F\x97\x14\x90" + "\x0E\x37\x13\xE6\x67\x1D\xE5\xFE" + "\x9E\x18\x3C\x8F\x3A\x3F\x59\x9B" + "\x71\x80\x05\x35\x3F\x40\x0B\x21" + "\x76\xE5\xEF\x42\x6C\xDB\x31\x05" + "\x5F\x05\xCF\x14\xE3\xF0\x61\xA2" + "\x49\x03\x5E\x77\x2E\x20\xBA\xA1" + "\xAF\x46\x51\xC0\x2B\xC4\x64\x1E" + "\x65\xCC\x51\x58\x0A\xDF\xF0\x5F" + "\x75\x9F\x48\xCD\x81\xEC\xC3\xF6" + "\xED\xC9\x4B\x7B\x4E\x26\x23\xE1" + "\xBB\xE9\x83\x0B\xCF\xE4\xDE\x00" + "\x48\xFF\xBF\x6C\xB4\x72\x16\xEF" + "\xC7\x46\xEE\x48\x8C\xB8\xAF\x45" + "\x91\x76\xE7\x6E\x65\x3D\x15\x86" + "\x10\xF8\xDB\x66\x97\x7C\x43\x4D" + "\x79\x12\x4E\xCE\x06\xD1\xD1\x6A" + "\x34\xC1\xC9\xF2\x28\x4A\xCD\x02" + "\x75\x55\x9B\xFF\x36\x73\xAB\x7C" + "\xF4\x46\x2E\xEB\xAC\xF3\xD2\xB7", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, }; -static const struct cipher_testvec fcrypt_pcbc_dec_tv_template[] = { - { /* http://www.openafs.org/pipermail/openafs-devel/2000-December/005320.html */ - .key = "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 8, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x0E\x09\x00\xC7\x3E\xF7\xED\x41", - .ilen = 8, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 8, - }, { - .key = "\x11\x44\x77\xAA\xDD\x00\x33\x66", - .klen = 8, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xD8\xED\x78\x74\x77\xEC\x06\x80", - .ilen = 8, - .result = "\x12\x34\x56\x78\x9A\xBC\xDE\xF0", - .rlen = 8, - }, { /* From Arla */ - .key = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", - .klen = 8, - .iv = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .input = "\x00\xf0\x0e\x11\x75\xe6\x23\x82" - "\xee\xac\x98\x62\x44\x51\xe4\x84" - "\xc3\x59\xd8\xaa\x64\x60\xae\xf7" - "\xd2\xd9\x13\x79\x72\xa3\x45\x03" - "\x23\xb5\x62\xd7\x0c\xf5\x27\xd1" - "\xf8\x91\x3c\xac\x44\x22\x92\xef", - .ilen = 48, - .result = "The quick brown fox jumps over the lazy dogs.\0\0", - .rlen = 48, - }, { - .key = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .klen = 8, - .iv = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", - .input = "\xca\x90\xf5\x9d\xcb\xd4\xd2\x3c" - "\x01\x88\x7f\x3e\x31\x6e\x62\x9d" - "\xd8\xe0\x57\xa3\x06\x3a\x42\x58" - "\x2a\x28\xfe\x72\x52\x2f\xdd\xe0" - "\x19\x89\x09\x1c\x2a\x8e\x8c\x94" - "\xfc\xc7\x68\xe4\x88\xaa\xde\x0f", - .ilen = 48, - .result = "The quick brown fox jumps over the lazy dogs.\0\0", - .rlen = 48, - }, { /* split-page version */ - .key = "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .klen = 8, - .iv = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", - .input = "\xca\x90\xf5\x9d\xcb\xd4\xd2\x3c" - "\x01\x88\x7f\x3e\x31\x6e\x62\x9d" - "\xd8\xe0\x57\xa3\x06\x3a\x42\x58" - "\x2a\x28\xfe\x72\x52\x2f\xdd\xe0" - "\x19\x89\x09\x1c\x2a\x8e\x8c\x94" - "\xfc\xc7\x68\xe4\x88\xaa\xde\x0f", - .ilen = 48, - .result = "The quick brown fox jumps over the lazy dogs.\0\0", - .rlen = 48, - .np = 2, - .tap = { 20, 28 }, +static const struct cipher_testvec tnepres_tv_template[] = { + { /* KeySize=0 */ + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .ctext = "\x41\xcc\x6b\x31\x59\x31\x45\x97" + "\x6d\x6f\xbb\x38\x4b\x37\x21\x28", + .len = 16, + }, + { /* KeySize=128, PT=0, I=1 */ + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .key = "\x80\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 16, + .ctext = "\x49\xaf\xbf\xad\x9d\x5a\x34\x05" + "\x2c\xd8\xff\xa5\x98\x6b\xd2\xdd", + .len = 16, + }, { /* KeySize=128 */ + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .klen = 16, + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .ctext = "\xea\xf4\xd7\xfc\xd8\x01\x34\x47" + "\x81\x45\x0b\xfa\x0c\xd6\xad\x6e", + .len = 16, + }, { /* KeySize=128, I=121 */ + .key = "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80", + .klen = 16, + .ptext = zeroed_string, + .ctext = "\x3d\xda\xbf\xc0\x06\xda\xab\x06" + "\x46\x2a\xf4\xef\x81\x54\x4e\x26", + .len = 16, + }, { /* KeySize=192, PT=0, I=1 */ + .key = "\x80\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 24, + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\xe7\x8e\x54\x02\xc7\x19\x55\x68" + "\xac\x36\x78\xf7\xa3\xf6\x0c\x66", + .len = 16, + }, { /* KeySize=256, PT=0, I=1 */ + .key = "\x80\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 32, + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\xab\xed\x96\xe7\x66\xbf\x28\xcb" + "\xc0\xeb\xd2\x1a\x82\xef\x08\x19", + .len = 16, + }, { /* KeySize=256, I=257 */ + .key = "\x1f\x1e\x1d\x1c\x1b\x1a\x19\x18" + "\x17\x16\x15\x14\x13\x12\x11\x10" + "\x0f\x0e\x0d\x0c\x0b\x0a\x09\x08" + "\x07\x06\x05\x04\x03\x02\x01\x00", + .klen = 32, + .ptext = "\x0f\x0e\x0d\x0c\x0b\x0a\x09\x08" + "\x07\x06\x05\x04\x03\x02\x01\x00", + .ctext = "\x5c\xe7\x1c\x70\xd2\x88\x2e\x5b" + "\xb8\x32\xe4\x33\xf8\x9f\x26\xde", + .len = 16, + }, { /* KeySize=256 */ + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .klen = 32, + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .ctext = "\x64\xa9\x1a\x37\xed\x9f\xe7\x49" + "\xa8\x4e\x76\xd6\xf5\x0d\x78\xee", + .len = 16, } }; -/* - * CAMELLIA test vectors. - */ -static const struct cipher_testvec camellia_enc_tv_template[] = { - { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .klen = 16, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .ilen = 16, - .result = "\x67\x67\x31\x38\x54\x96\x69\x73" - "\x08\x57\x06\x56\x48\xea\xbe\x43", - .rlen = 16, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10" - "\x00\x11\x22\x33\x44\x55\x66\x77", - .klen = 24, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .ilen = 16, - .result = "\xb4\x99\x34\x01\xb3\xe9\x96\xf8" - "\x4e\xe5\xce\xe7\xd7\x9b\x09\xb9", - .rlen = 16, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10" - "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", +static const struct cipher_testvec serpent_cbc_tv_template[] = { + { /* Generated with Crypto++ */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, - .input = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .ilen = 16, - .result = "\x9a\xcc\x23\x7d\xff\x16\xd7\x6c" - "\x20\xef\x7c\x91\x9e\x3a\x75\x09", - .rlen = 16, - }, { /* Generated with Crypto++ */ - .key = "\x3F\x85\x62\x3F\x1C\xF9\xD6\x1C" - "\xF9\xD6\xB3\x90\x6D\x4A\x90\x6D" - "\x4A\x27\x04\xE1\x27\x04\xE1\xBE" - "\x9B\x78\xBE\x9B\x78\x55\x32\x0F", + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" + "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" + "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" + "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" + "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" + "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" + "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" + "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" + "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" + "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" + "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" + "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" + "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" + "\x29\xC0\x57\xEE\x62\xF9\x90\x04" + "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" + "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" + "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" + "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" + "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" + "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" + "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" + "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" + "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" + "\x57\xEE\x85\x1C\x90\x27\xBE\x32" + "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" + "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" + "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" + "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" + "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" + "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" + "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" + "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" + "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" + "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" + "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" + "\x69\x00\x74\x0B\xA2\x16\xAD\x44" + "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" + "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" + "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" + "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" + "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" + "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" + "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" + "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" + "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" + "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" + "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" + "\x58\xEF\x86\x1D\x91\x28\xBF\x33" + "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" + "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" + "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" + "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" + "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" + "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" + "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" + "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" + "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" + "\x86\x1D\xB4\x28\xBF\x56\xED\x61" + "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" + "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\x80\xCF\x11\x41\x1A\xB9\x4B\x9C" + "\xFF\xB7\x6C\xEA\xF0\xAF\x77\x6E" + "\x71\x75\x95\x9D\x4E\x1C\xCF\xAD" + "\x81\x34\xE9\x8F\xAE\x5A\x91\x1C" + "\x38\x63\x35\x7E\x79\x18\x0A\xE8" + "\x67\x06\x76\xD5\xFF\x22\x2F\xDA" + "\xB6\x2D\x57\x13\xB6\x3C\xBC\x97" + "\xFE\x53\x75\x35\x97\x7F\x51\xEA" + "\xDF\x5D\xE8\x9D\xCC\xD9\xAE\xE7" + "\x62\x67\xFF\x04\xC2\x18\x22\x5F" + "\x2E\x06\xC1\xE2\x26\xCD\xC6\x1E" + "\xE5\x2C\x4E\x87\x23\xDD\xF0\x41" + "\x08\xA5\xB4\x3E\x07\x1E\x0B\xBB" + "\x72\x84\xF8\x0A\x3F\x38\x5E\x91" + "\x15\x26\xE1\xDB\xA4\x3D\x74\xD2" + "\x41\x1E\x3F\xA9\xC6\x7D\x2A\xAB" + "\x27\xDF\x89\x1D\x86\x3E\xF7\x5A" + "\xF6\xE3\x0F\xC7\x6B\x4C\x96\x7C" + "\x2D\x12\xA5\x05\x92\xCB\xD7\x4A" + "\x4D\x1E\x88\x21\xE1\x63\xB4\xFC" + "\x4A\xF2\xCD\x35\xB9\xD7\x70\x97" + "\x5A\x5E\x7E\x96\x52\x20\xDC\x25" + "\xE9\x6B\x36\xB4\xE0\x98\x85\x2C" + "\x3C\xD2\xF7\x78\x8A\x73\x26\x9B" + "\xAF\x0B\x11\xE8\x4D\x67\x23\xE9" + "\x77\xDF\x58\xF6\x6F\x9E\xA4\xC5" + "\x10\xA1\x82\x0E\x80\xA0\x8F\x4B" + "\xA1\xC0\x12\x54\x4E\xC9\x20\x92" + "\x11\x00\x10\x4E\xB3\x7C\xCA\x63" + "\xE5\x3F\xD3\x41\x37\xCD\x74\xB7" + "\xA5\x7C\x61\xB8\x0B\x7A\x7F\x4D" + "\xFE\x96\x7D\x1B\xBE\x60\x37\xB7" + "\x81\x92\x66\x67\x15\x1E\x39\x98" + "\x52\xC0\xF4\x69\xC0\x99\x4F\x5A" + "\x2E\x32\xAD\x7C\x8B\xE9\xAD\x05" + "\x55\xF9\x0A\x1F\x97\x5C\xFA\x2B" + "\xF4\x99\x76\x3A\x6E\x4D\xE1\x4C" + "\x14\x4E\x6F\x87\xEE\x1A\x85\xA3" + "\x96\xC6\x66\x49\xDA\x0D\x71\xAC" + "\x04\x05\x46\xD3\x90\x0F\x64\x64" + "\x01\x66\x2C\x62\x5D\x34\xD1\xCB" + "\x3A\x24\xCE\x95\xEF\xAE\x2C\x97" + "\x0E\x0C\x1D\x36\x49\xEB\xE9\x3D" + "\x62\xA6\x19\x28\x9E\x26\xB4\x3F" + "\xD7\x55\x42\x3C\xCD\x72\x0A\xF0" + "\x7D\xE9\x95\x45\x86\xED\xB1\xE0" + "\x8D\xE9\xC5\x86\x13\x24\x28\x7D" + "\x74\xEF\xCA\x50\x12\x7E\x64\x8F" + "\x1B\xF5\x5B\xFE\xE2\xAC\xFA\xE7" + "\xBD\x38\x8C\x11\x20\xEF\xB1\xAA" + "\x7B\xE5\xE5\x78\xAD\x9D\x2D\xA2" + "\x8E\xDD\x48\xB3\xEF\x18\x92\x7E" + "\xE6\x75\x0D\x54\x64\x11\xA3\x3A" + "\xDB\x97\x0F\xD3\xDF\x07\xD3\x7E" + "\x1E\xD1\x87\xE4\x74\xBB\x46\xF4" + "\xBA\x23\x2D\x8D\x29\x07\x12\xCF" + "\x34\xCD\x72\x7F\x01\x30\xE7\xA0" + "\xF8\xDD\xA8\x08\xF0\xBC\xB1\xA2" + "\xCC\xE1\x6B\x5F\xBE\xEA\xF1\xE4" + "\x02\xC4\xAF\xFA\xAD\x31\xF4\xBF" + "\xFC\x66\xAA\x37\xF2\x37\x39\x6B" + "\xBC\x08\x3A\xA2\x29\xB3\xDF\xD1", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec serpent_ctr_tv_template[] = { + { /* Generated with Crypto++ */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -30146,393 +9136,237 @@ static const struct cipher_testvec camellia_enc_tv_template[] = { "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59\xF0\x64\xFB\x92\x06" - "\x9D\x34\xCB\x3F\xD6\x6D\x04\x78" - "\x0F\xA6\x1A\xB1\x48\xDF\x53\xEA" - "\x81\x18\x8C\x23\xBA\x2E\xC5\x5C" - "\xF3\x67\xFE\x95\x09\xA0\x37\xCE" - "\x42\xD9\x70\x07\x7B\x12\xA9\x1D" - "\xB4\x4B\xE2\x56\xED\x84\x1B\x8F" - "\x26\xBD\x31\xC8\x5F\xF6\x6A\x01" - "\x98\x0C\xA3\x3A\xD1\x45\xDC\x73" - "\x0A\x7E\x15\xAC\x20\xB7\x4E\xE5" - "\x59\xF0\x87\x1E\x92\x29\xC0\x34" - "\xCB\x62\xF9\x6D\x04\x9B\x0F\xA6" - "\x3D\xD4\x48\xDF\x76\x0D\x81\x18" - "\xAF\x23\xBA\x51\xE8\x5C\xF3\x8A" - "\x21\x95\x2C\xC3\x37\xCE\x65\xFC" - "\x70\x07\x9E\x12\xA9\x40\xD7\x4B" - "\xE2\x79\x10\x84\x1B\xB2\x26\xBD" - "\x54\xEB\x5F\xF6\x8D\x01\x98\x2F" - "\xC6\x3A\xD1\x68\xFF\x73\x0A\xA1" - "\x15\xAC\x43\xDA\x4E\xE5\x7C\x13" - "\x87\x1E\xB5\x29\xC0\x57\xEE\x62" - "\xF9\x90\x04\x9B\x32\xC9\x3D\xD4" - "\x6B\x02\x76\x0D\xA4\x18\xAF\x46" - "\xDD\x51\xE8\x7F\x16\x8A\x21\xB8" - "\x2C\xC3\x5A\xF1\x65\xFC\x93\x07" - "\x9E\x35\xCC\x40\xD7\x6E\x05\x79" - "\x10\xA7\x1B\xB2\x49\xE0\x54\xEB" - "\x82\x19\x8D\x24\xBB\x2F\xC6\x5D" - "\xF4\x68\xFF\x96\x0A\xA1\x38\xCF" - "\x43\xDA\x71\x08\x7C\x13\xAA\x1E" - "\xB5\x4C\xE3\x57\xEE\x85\x1C\x90" - "\x27\xBE\x32\xC9\x60\xF7\x6B\x02" - "\x99\x0D\xA4\x3B\xD2\x46\xDD\x74" - "\x0B\x7F\x16\xAD\x21\xB8\x4F\xE6" - "\x5A\xF1\x88\x1F\x93\x2A\xC1\x35" - "\xCC\x63\xFA\x6E\x05\x9C\x10\xA7" - "\x3E\xD5\x49\xE0\x77\x0E\x82\x19" - "\xB0\x24\xBB\x52\xE9\x5D\xF4\x8B" - "\x22\x96\x2D\xC4\x38\xCF\x66\xFD" - "\x71\x08\x9F\x13\xAA\x41\xD8\x4C" - "\xE3\x7A\x11\x85\x1C\xB3\x27\xBE" - "\x55\xEC\x60\xF7\x8E\x02\x99\x30" - "\xC7\x3B\xD2\x69\x00\x74\x0B\xA2" - "\x16\xAD\x44\xDB\x4F\xE6\x7D\x14" - "\x88\x1F\xB6\x2A\xC1\x58\xEF\x63" - "\xFA\x91\x05\x9C\x33\xCA\x3E\xD5" - "\x6C\x03\x77\x0E\xA5\x19\xB0\x47" - "\xDE\x52\xE9\x80\x17\x8B\x22\xB9" - "\x2D\xC4\x5B\xF2\x66\xFD\x94\x08" - "\x9F\x36\xCD\x41\xD8\x6F\x06\x7A" - "\x11\xA8\x1C\xB3\x4A\xE1\x55\xEC" - "\x83\x1A\x8E\x25\xBC\x30\xC7\x5E" - "\xF5\x69\x00\x97\x0B\xA2\x39\xD0" - "\x44\xDB\x72\x09\x7D\x14\xAB\x1F" - "\xB6\x4D\xE4\x58\xEF\x86\x1D\x91" - "\x28\xBF\x33\xCA\x61\xF8\x6C\x03" - "\x9A\x0E\xA5\x3C\xD3\x47\xDE\x75" - "\x0C\x80\x17\xAE\x22\xB9\x50\xE7" - "\x5B\xF2\x89\x20\x94\x2B\xC2\x36" - "\xCD\x64\xFB\x6F\x06\x9D\x11\xA8" - "\x3F\xD6\x4A\xE1\x78\x0F\x83\x1A" - "\xB1\x25\xBC\x53\xEA\x5E\xF5\x8C" - "\x00\x97\x2E\xC5\x39\xD0\x67\xFE" - "\x72\x09\xA0\x14\xAB\x42\xD9\x4D", - .ilen = 1008, - .result = "\xED\xCD\xDB\xB8\x68\xCE\xBD\xEA" - "\x9D\x9D\xCD\x9F\x4F\xFC\x4D\xB7" - "\xA5\xFF\x6F\x43\x0F\xBA\x32\x04" - "\xB3\xC2\xB9\x03\xAA\x91\x56\x29" - "\x0D\xD0\xFD\xC4\x65\xA5\x69\xB9" - "\xF1\xF6\xB1\xA5\xB2\x75\x4F\x8A" - "\x8D\x7D\x1B\x9B\xC7\x68\x72\xF8" - "\x01\x9B\x17\x0A\x29\xE7\x61\x28" - "\x7F\xA7\x50\xCA\x20\x2C\x96\x3B" - "\x6E\x5C\x5D\x3F\xB5\x7F\xF3\x2B" - "\x04\xEF\x9D\xD4\xCE\x41\x28\x8E" - "\x83\x54\xAE\x7C\x82\x46\x10\xC9" - "\xC4\x8A\x1E\x1F\x4C\xA9\xFC\xEC" - "\x3C\x8C\x30\xFC\x59\xD2\x54\xC4" - "\x6F\x50\xC6\xCA\x8C\x14\x5B\x9C" - "\x18\x56\x5B\xF8\x33\x0E\x4A\xDB" - "\xEC\xB5\x6E\x5B\x31\xC4\x0E\x98" - "\x9F\x32\xBA\xA2\x18\xCF\x55\x43" - "\xFE\x80\x8F\x60\xCF\x05\x30\x9B" - "\x70\x50\x1E\x9C\x08\x87\xE6\x20" - "\xD2\xF3\x27\xF8\x2A\x8D\x12\xB2" - "\xBC\x5F\xFE\x52\x52\xF6\x7F\xB6" - "\xB8\x30\x86\x3B\x0F\x94\x1E\x79" - "\x13\x94\x35\xA2\xB1\x35\x5B\x05" - "\x2A\x98\x6B\x96\x4C\xB1\x20\xBE" - "\xB6\x14\xC2\x06\xBF\xFD\x5F\x2A" - "\xF5\x33\xC8\x19\x45\x14\x44\x5D" - "\xFE\x94\x7B\xBB\x63\x13\x57\xC3" - "\x2A\x8F\x6C\x11\x2A\x07\xA7\x6A" - "\xBF\x20\xD3\x99\xC6\x00\x0B\xBF" - "\x83\x46\x25\x3A\xB0\xF6\xC5\xC8" - "\x00\xCA\xE5\x28\x4A\x7C\x95\x9C" - "\x7B\x43\xAB\xF9\xE4\xF8\x74\xAB" - "\xA7\xB8\x9C\x0F\x53\x7B\xB6\x74" - "\x60\x64\x0D\x1C\x80\xD1\x20\x9E" - "\xDC\x14\x27\x9B\xFC\xBD\x5C\x96" - "\xD2\x51\xDC\x96\xEE\xE5\xEA\x2B" - "\x02\x7C\xAA\x3C\xDC\x9D\x7B\x01" - "\x20\xC3\xE1\x0B\xDD\xAB\xF3\x1E" - "\x19\xA8\x84\x29\x5F\xCC\xC3\x5B" - "\xE4\x33\x59\xDC\x12\xEB\x2B\x4D" - "\x5B\x55\x23\xB7\x40\x31\xDE\xEE" - "\x18\xC9\x3C\x4D\xBC\xED\xE0\x42" - "\xAD\xDE\xA0\xA3\xC3\xFE\x44\xD3" - "\xE1\x9A\xDA\xAB\x32\xFC\x1A\xBF" - "\x63\xA9\xF0\x6A\x08\x46\xBD\x48" - "\x83\x06\xAB\x82\x99\x01\x16\x1A" - "\x03\x36\xC5\x59\x6B\xB8\x8C\x9F" - "\xC6\x51\x3D\xE5\x7F\xBF\xAB\xBC" - "\xC9\xA1\x88\x34\x5F\xA9\x7C\x3B" - "\x9F\x1B\x98\x2B\x4F\xFB\x9B\xF0" - "\xCD\xB6\x45\xB2\x29\x2E\x34\x23" - "\xA9\x97\xC0\x22\x8C\x42\x9B\x5F" - "\x40\xC8\xD7\x3D\x82\x9A\x6F\xAA" - "\x74\x83\x29\x05\xE8\xC4\x4D\x01" - "\xB5\xE5\x84\x3F\x7F\xD3\xE0\x99" - "\xDA\xE7\x6F\x30\xFD\xAA\x92\x30" - "\xA5\x46\x8B\xA2\xE6\x58\x62\x7C" - "\x2C\x35\x1B\x38\x85\x7D\xE8\xF3" - "\x87\x4F\xDA\xD8\x5F\xFC\xB6\x44" - "\xD0\xE3\x9B\x8B\xBF\xD6\xB8\xC4" - "\x73\xAE\x1D\x8B\x5B\x74\x8B\xCB" - "\xA4\xAD\xCF\x5D\xD4\x58\xC9\xCD" - "\xF7\x90\x68\xCF\xC9\x11\x52\x3E" - "\xE8\xA1\xA3\x78\x8B\xD0\xAC\x0A" - "\xD4\xC9\xA3\xA5\x55\x30\xC8\x3E" - "\xED\x28\x39\xE9\x63\xED\x41\x70" - "\x51\xE3\xC4\xA0\xFC\xD5\x43\xCB" - "\x4D\x65\xC8\xFD\x3A\x91\x8F\x60" - "\x8A\xA6\x6D\x9D\x3E\x01\x23\x4B" - "\x50\x47\xC9\xDC\x9B\xDE\x37\xC5" - "\xBF\x67\xB1\x6B\x78\x38\xD5\x7E" - "\xB6\xFF\x67\x83\x3B\x6E\xBE\x23" - "\x45\xFA\x1D\x69\x44\xFD\xC6\xB9" - "\xD0\x4A\x92\xD1\xBE\xF6\x4A\xB7" - "\xCA\xA8\xA2\x9E\x13\x87\x57\x92" - "\x64\x7C\x85\x0B\xB3\x29\x37\xD8" - "\xE6\xAA\xAF\xC4\x03\x67\xA3\xBF" - "\x2E\x45\x83\xB6\xD8\x54\x00\x89" - "\xF6\xBC\x3A\x7A\x88\x58\x51\xED" - "\xF4\x4E\x01\xA5\xC3\x2E\xD9\x42" - "\xBD\x6E\x0D\x0B\x21\xB0\x1A\xCC" - "\xA4\xD3\x3F\xDC\x9B\x81\xD8\xF1" - "\xEA\x7A\x6A\xB7\x07\xC9\x6D\x91" - "\x6D\x3A\xF5\x5F\xA6\xFF\x87\x1E" - "\x3F\xDD\xC0\x72\xEA\xAC\x08\x15" - "\x21\xE6\xC6\xB6\x0D\xD8\x51\x86" - "\x2A\x03\x73\xF7\x29\xD4\xC4\xE4" - "\x7F\x95\x10\xF7\xAB\x3F\x92\x23" - "\xD3\xCE\x9C\x2E\x46\x3B\x63\x43" - "\xBB\xC2\x82\x7A\x83\xD5\x55\xE2" - "\xE7\x9B\x2F\x92\xAF\xFD\x81\x56" - "\x79\xFD\x3E\xF9\x46\xE0\x25\xD4" - "\x38\xDE\xBC\x2C\xC4\x7A\x2A\x8F" - "\x94\x4F\xD0\xAD\x9B\x37\x18\xD4" - "\x0E\x4D\x0F\x02\x3A\xDC\x5A\xA2" - "\x39\x25\x55\x20\x5A\xA6\x02\x9F" - "\xE6\x77\x21\x77\xE5\x4B\x7B\x0B" - "\x30\xF8\x5F\x33\x0F\x49\xCD\xFF" - "\xF2\xE4\x35\xF9\xF0\x63\xC3\x7E" - "\xF1\xA6\x73\xB4\xDF\xE7\xBB\x78" - "\xFF\x21\xA9\xF3\xF3\xCF\x5D\xBA" - "\xED\x87\x98\xAC\xFE\x48\x97\x6D" - "\xA6\x7F\x69\x31\xB1\xC4\xFF\x14" - "\xC6\x76\xD4\x10\xDD\xF6\x49\x2C" - "\x9C\xC8\x6D\x76\xC0\x8F\x5F\x55" - "\x2F\x3C\x8A\x30\xAA\xC3\x16\x55" - "\xC6\xFC\x8D\x8B\xB9\xE5\x80\x6C" - "\xC8\x7E\xBD\x65\x58\x36\xD5\xBC" - "\xF0\x33\x52\x29\x70\xF9\x5C\xE9" - "\xAC\x1F\xB5\x73\x56\x66\x54\xAF" - "\x1B\x8F\x7D\xED\xAB\x03\xCE\xE3" - "\xAE\x47\xB6\x69\x86\xE9\x01\x31" - "\x83\x18\x3D\xF4\x74\x7B\xF9\x42" - "\x4C\xFD\x75\x4A\x6D\xF0\x03\xA6" - "\x2B\x20\x63\xDA\x49\x65\x5E\x8B" - "\xC0\x19\xE3\x8D\xD9\xF3\xB0\x34" - "\xD3\x52\xFC\x68\x00\x43\x1B\x37" - "\x31\x93\x51\x1C\x63\x97\x70\xB0" - "\x99\x78\x83\x13\xFD\xCF\x53\x81" - "\x36\x46\xB5\x42\x52\x2F\x32\xEB" - "\x4A\x3D\xF1\x8F\x1C\x54\x2E\xFC" - "\x41\x75\x5A\x8C\x8E\x6F\xE7\x1A" - "\xAE\xEF\x3E\x82\x12\x0B\x74\x72" - "\xF8\xB2\xAA\x7A\xD6\xFF\xFA\x55" - "\x33\x1A\xBB\xD3\xA2\x7E\x97\x66", - .rlen = 1008, - .also_non_np = 1, - .np = 3, - .tap = { 1008 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec camellia_dec_tv_template[] = { - { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .klen = 16, - .input = "\x67\x67\x31\x38\x54\x96\x69\x73" - "\x08\x57\x06\x56\x48\xea\xbe\x43", - .ilen = 16, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .rlen = 16, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10" - "\x00\x11\x22\x33\x44\x55\x66\x77", - .klen = 24, - .input = "\xb4\x99\x34\x01\xb3\xe9\x96\xf8" - "\x4e\xe5\xce\xe7\xd7\x9b\x09\xb9", - .ilen = 16, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .rlen = 16, - }, { - .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10" - "\x00\x11\x22\x33\x44\x55\x66\x77" - "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", - .klen = 32, - .input = "\x9a\xcc\x23\x7d\xff\x16\xd7\x6c" - "\x20\xef\x7c\x91\x9e\x3a\x75\x09", - .ilen = 16, - .result = "\x01\x23\x45\x67\x89\xab\xcd\xef" - "\xfe\xdc\xba\x98\x76\x54\x32\x10", - .rlen = 16, + "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" + "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" + "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" + "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" + "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" + "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" + "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" + "\x58\xEF\x86\x1D\x91\x28\xBF\x33" + "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" + "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" + "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" + "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" + "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" + "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" + "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" + "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" + "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" + "\x86\x1D\xB4\x28\xBF\x56\xED\x61" + "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" + "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\x84\x68\xEC\xF2\x1C\x88\x20\xCA" + "\x37\x69\xE3\x3A\x22\x85\x48\x46" + "\x70\xAA\x25\xB4\xCD\x8B\x04\x4E" + "\x8D\x15\x2B\x98\xDF\x7B\x6D\xB9" + "\xE0\x4A\x73\x00\x65\xB6\x1A\x0D" + "\x5C\x60\xDF\x34\xDC\x60\x4C\xDF" + "\xB5\x1F\x26\x8C\xDA\xC1\x11\xA8" + "\x80\xFA\x37\x7A\x89\xAA\xAE\x7B" + "\x92\x6E\xB9\xDC\xC9\x62\x4F\x88" + "\x0A\x5D\x97\x2F\x6B\xAC\x03\x7C" + "\x22\xF6\x55\x5A\xFA\x35\xA5\x17" + "\xA1\x5C\x5E\x2B\x63\x2D\xB9\x91" + "\x3E\x83\x26\x00\x4E\xD5\xBE\xCE" + "\x79\xC4\x3D\xFC\x70\xA0\xAD\x96" + "\xBA\x58\x2A\x1C\xDF\xC2\x3A\xA5" + "\x7C\xB5\x12\x89\xED\xBF\xB6\x09" + "\x13\x4F\x7D\x61\x3C\x5C\x27\xFC" + "\x5D\xE1\x4F\xA1\xEA\xB3\xCA\xB9" + "\xE6\xD0\x97\x81\xDE\xD1\xFB\x8A" + "\x30\xDB\xA3\x5D\xEC\x25\x0B\x86" + "\x71\xC8\xA7\x67\xE8\xBC\x7D\x4C" + "\xAE\x82\xD3\x73\x31\x09\xCB\xB3" + "\x4D\xD4\xC0\x8A\x2B\xFA\xA6\x55" + "\x39\x0A\xBC\x6E\x75\xAB\xC2\xE2" + "\x8A\xF2\x26\xCD\x63\x38\x35\xF7" + "\xAE\x12\x83\xCD\x8A\x9E\x7E\x4C" + "\xFE\x4D\xD7\xCE\x5C\x6E\x4C\xAF" + "\xE3\xCD\x76\xA7\x87\xA1\x54\x7C" + "\xEC\x32\xC7\x83\x2A\xFF\xF8\xEA" + "\x87\xB2\x47\xA3\x9D\xC2\x9C\xA2" + "\xB7\x2C\x7C\x1A\x24\xCB\x88\x61" + "\xFF\xA7\x1A\x16\x01\xDD\x4B\xFC" + "\x2E\xE0\x48\x67\x09\x42\xCC\x91" + "\xBE\x20\x38\xC0\x5E\x3B\x95\x00" + "\xA1\x96\x66\x0B\x8A\xE9\x9E\xF7" + "\x6B\x34\x0A\x51\xC0\x3B\xEB\x71" + "\x07\x97\x38\x4B\x5C\x56\x98\x67" + "\x78\x9C\xD0\x0E\x2B\xB5\x67\x90" + "\x75\xF8\xFE\x6D\x4E\x85\xCC\x0D" + "\x18\x06\x15\x9D\x5A\x10\x13\x37" + "\xA3\xD6\x68\xA2\xDF\x7E\xC7\x12" + "\xC9\x0D\x4D\x91\xB0\x2A\x55\xFF" + "\x6F\x73\x13\xDF\x28\xB5\x2A\x2C" + "\xE4\xFC\x20\xD9\xF1\x7A\x82\xB1" + "\xCB\x57\xB6\x3D\x8C\xF4\x8E\x27" + "\x37\xDC\x35\xF3\x79\x01\x53\xA4" + "\x7B\x37\xDE\x7C\x04\xAE\x50\xDB" + "\x9B\x1E\x8C\x07\xA7\x52\x49\x50" + "\x34\x25\x65\xDD\xA9\x8F\x7E\xBD" + "\x7A\xC9\x36\xAE\xDE\x21\x48\x64" + "\xC2\x02\xBA\xBE\x11\x1E\x3D\x9C" + "\x98\x52\xCC\x04\xBD\x5E\x61\x26" + "\x10\xD3\x21\xD9\x6E\x25\x98\x77" + "\x8E\x98\x63\xF6\xF6\x52\xFB\x13" + "\xAA\x30\xF2\xB9\xA4\x43\x53\x39" + "\x1C\x97\x07\x7E\x6B\xFF\x3D\x43" + "\xA6\x71\x6B\x66\x8F\x58\x3F\x71" + "\x90\x47\x40\x92\xE6\x69\xD1\x96" + "\x34\xB3\x3B\xE5\x43\xE4\xD5\x56" + "\xB2\xE6\x7E\x86\x7A\x12\x17\x5B" + "\x30\xF3\x9B\x0D\xFA\x57\xE4\x50" + "\x40\x53\x77\x8C\x15\xF8\x8D\x13", + .len = 496, }, { /* Generated with Crypto++ */ - .key = "\x3F\x85\x62\x3F\x1C\xF9\xD6\x1C" - "\xF9\xD6\xB3\x90\x6D\x4A\x90\x6D" - "\x4A\x27\x04\xE1\x27\x04\xE1\xBE" - "\x9B\x78\xBE\x9B\x78\x55\x32\x0F", + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, - .input = "\xED\xCD\xDB\xB8\x68\xCE\xBD\xEA" - "\x9D\x9D\xCD\x9F\x4F\xFC\x4D\xB7" - "\xA5\xFF\x6F\x43\x0F\xBA\x32\x04" - "\xB3\xC2\xB9\x03\xAA\x91\x56\x29" - "\x0D\xD0\xFD\xC4\x65\xA5\x69\xB9" - "\xF1\xF6\xB1\xA5\xB2\x75\x4F\x8A" - "\x8D\x7D\x1B\x9B\xC7\x68\x72\xF8" - "\x01\x9B\x17\x0A\x29\xE7\x61\x28" - "\x7F\xA7\x50\xCA\x20\x2C\x96\x3B" - "\x6E\x5C\x5D\x3F\xB5\x7F\xF3\x2B" - "\x04\xEF\x9D\xD4\xCE\x41\x28\x8E" - "\x83\x54\xAE\x7C\x82\x46\x10\xC9" - "\xC4\x8A\x1E\x1F\x4C\xA9\xFC\xEC" - "\x3C\x8C\x30\xFC\x59\xD2\x54\xC4" - "\x6F\x50\xC6\xCA\x8C\x14\x5B\x9C" - "\x18\x56\x5B\xF8\x33\x0E\x4A\xDB" - "\xEC\xB5\x6E\x5B\x31\xC4\x0E\x98" - "\x9F\x32\xBA\xA2\x18\xCF\x55\x43" - "\xFE\x80\x8F\x60\xCF\x05\x30\x9B" - "\x70\x50\x1E\x9C\x08\x87\xE6\x20" - "\xD2\xF3\x27\xF8\x2A\x8D\x12\xB2" - "\xBC\x5F\xFE\x52\x52\xF6\x7F\xB6" - "\xB8\x30\x86\x3B\x0F\x94\x1E\x79" - "\x13\x94\x35\xA2\xB1\x35\x5B\x05" - "\x2A\x98\x6B\x96\x4C\xB1\x20\xBE" - "\xB6\x14\xC2\x06\xBF\xFD\x5F\x2A" - "\xF5\x33\xC8\x19\x45\x14\x44\x5D" - "\xFE\x94\x7B\xBB\x63\x13\x57\xC3" - "\x2A\x8F\x6C\x11\x2A\x07\xA7\x6A" - "\xBF\x20\xD3\x99\xC6\x00\x0B\xBF" - "\x83\x46\x25\x3A\xB0\xF6\xC5\xC8" - "\x00\xCA\xE5\x28\x4A\x7C\x95\x9C" - "\x7B\x43\xAB\xF9\xE4\xF8\x74\xAB" - "\xA7\xB8\x9C\x0F\x53\x7B\xB6\x74" - "\x60\x64\x0D\x1C\x80\xD1\x20\x9E" - "\xDC\x14\x27\x9B\xFC\xBD\x5C\x96" - "\xD2\x51\xDC\x96\xEE\xE5\xEA\x2B" - "\x02\x7C\xAA\x3C\xDC\x9D\x7B\x01" - "\x20\xC3\xE1\x0B\xDD\xAB\xF3\x1E" - "\x19\xA8\x84\x29\x5F\xCC\xC3\x5B" - "\xE4\x33\x59\xDC\x12\xEB\x2B\x4D" - "\x5B\x55\x23\xB7\x40\x31\xDE\xEE" - "\x18\xC9\x3C\x4D\xBC\xED\xE0\x42" - "\xAD\xDE\xA0\xA3\xC3\xFE\x44\xD3" - "\xE1\x9A\xDA\xAB\x32\xFC\x1A\xBF" - "\x63\xA9\xF0\x6A\x08\x46\xBD\x48" - "\x83\x06\xAB\x82\x99\x01\x16\x1A" - "\x03\x36\xC5\x59\x6B\xB8\x8C\x9F" - "\xC6\x51\x3D\xE5\x7F\xBF\xAB\xBC" - "\xC9\xA1\x88\x34\x5F\xA9\x7C\x3B" - "\x9F\x1B\x98\x2B\x4F\xFB\x9B\xF0" - "\xCD\xB6\x45\xB2\x29\x2E\x34\x23" - "\xA9\x97\xC0\x22\x8C\x42\x9B\x5F" - "\x40\xC8\xD7\x3D\x82\x9A\x6F\xAA" - "\x74\x83\x29\x05\xE8\xC4\x4D\x01" - "\xB5\xE5\x84\x3F\x7F\xD3\xE0\x99" - "\xDA\xE7\x6F\x30\xFD\xAA\x92\x30" - "\xA5\x46\x8B\xA2\xE6\x58\x62\x7C" - "\x2C\x35\x1B\x38\x85\x7D\xE8\xF3" - "\x87\x4F\xDA\xD8\x5F\xFC\xB6\x44" - "\xD0\xE3\x9B\x8B\xBF\xD6\xB8\xC4" - "\x73\xAE\x1D\x8B\x5B\x74\x8B\xCB" - "\xA4\xAD\xCF\x5D\xD4\x58\xC9\xCD" - "\xF7\x90\x68\xCF\xC9\x11\x52\x3E" - "\xE8\xA1\xA3\x78\x8B\xD0\xAC\x0A" - "\xD4\xC9\xA3\xA5\x55\x30\xC8\x3E" - "\xED\x28\x39\xE9\x63\xED\x41\x70" - "\x51\xE3\xC4\xA0\xFC\xD5\x43\xCB" - "\x4D\x65\xC8\xFD\x3A\x91\x8F\x60" - "\x8A\xA6\x6D\x9D\x3E\x01\x23\x4B" - "\x50\x47\xC9\xDC\x9B\xDE\x37\xC5" - "\xBF\x67\xB1\x6B\x78\x38\xD5\x7E" - "\xB6\xFF\x67\x83\x3B\x6E\xBE\x23" - "\x45\xFA\x1D\x69\x44\xFD\xC6\xB9" - "\xD0\x4A\x92\xD1\xBE\xF6\x4A\xB7" - "\xCA\xA8\xA2\x9E\x13\x87\x57\x92" - "\x64\x7C\x85\x0B\xB3\x29\x37\xD8" - "\xE6\xAA\xAF\xC4\x03\x67\xA3\xBF" - "\x2E\x45\x83\xB6\xD8\x54\x00\x89" - "\xF6\xBC\x3A\x7A\x88\x58\x51\xED" - "\xF4\x4E\x01\xA5\xC3\x2E\xD9\x42" - "\xBD\x6E\x0D\x0B\x21\xB0\x1A\xCC" - "\xA4\xD3\x3F\xDC\x9B\x81\xD8\xF1" - "\xEA\x7A\x6A\xB7\x07\xC9\x6D\x91" - "\x6D\x3A\xF5\x5F\xA6\xFF\x87\x1E" - "\x3F\xDD\xC0\x72\xEA\xAC\x08\x15" - "\x21\xE6\xC6\xB6\x0D\xD8\x51\x86" - "\x2A\x03\x73\xF7\x29\xD4\xC4\xE4" - "\x7F\x95\x10\xF7\xAB\x3F\x92\x23" - "\xD3\xCE\x9C\x2E\x46\x3B\x63\x43" - "\xBB\xC2\x82\x7A\x83\xD5\x55\xE2" - "\xE7\x9B\x2F\x92\xAF\xFD\x81\x56" - "\x79\xFD\x3E\xF9\x46\xE0\x25\xD4" - "\x38\xDE\xBC\x2C\xC4\x7A\x2A\x8F" - "\x94\x4F\xD0\xAD\x9B\x37\x18\xD4" - "\x0E\x4D\x0F\x02\x3A\xDC\x5A\xA2" - "\x39\x25\x55\x20\x5A\xA6\x02\x9F" - "\xE6\x77\x21\x77\xE5\x4B\x7B\x0B" - "\x30\xF8\x5F\x33\x0F\x49\xCD\xFF" - "\xF2\xE4\x35\xF9\xF0\x63\xC3\x7E" - "\xF1\xA6\x73\xB4\xDF\xE7\xBB\x78" - "\xFF\x21\xA9\xF3\xF3\xCF\x5D\xBA" - "\xED\x87\x98\xAC\xFE\x48\x97\x6D" - "\xA6\x7F\x69\x31\xB1\xC4\xFF\x14" - "\xC6\x76\xD4\x10\xDD\xF6\x49\x2C" - "\x9C\xC8\x6D\x76\xC0\x8F\x5F\x55" - "\x2F\x3C\x8A\x30\xAA\xC3\x16\x55" - "\xC6\xFC\x8D\x8B\xB9\xE5\x80\x6C" - "\xC8\x7E\xBD\x65\x58\x36\xD5\xBC" - "\xF0\x33\x52\x29\x70\xF9\x5C\xE9" - "\xAC\x1F\xB5\x73\x56\x66\x54\xAF" - "\x1B\x8F\x7D\xED\xAB\x03\xCE\xE3" - "\xAE\x47\xB6\x69\x86\xE9\x01\x31" - "\x83\x18\x3D\xF4\x74\x7B\xF9\x42" - "\x4C\xFD\x75\x4A\x6D\xF0\x03\xA6" - "\x2B\x20\x63\xDA\x49\x65\x5E\x8B" - "\xC0\x19\xE3\x8D\xD9\xF3\xB0\x34" - "\xD3\x52\xFC\x68\x00\x43\x1B\x37" - "\x31\x93\x51\x1C\x63\x97\x70\xB0" - "\x99\x78\x83\x13\xFD\xCF\x53\x81" - "\x36\x46\xB5\x42\x52\x2F\x32\xEB" - "\x4A\x3D\xF1\x8F\x1C\x54\x2E\xFC" - "\x41\x75\x5A\x8C\x8E\x6F\xE7\x1A" - "\xAE\xEF\x3E\x82\x12\x0B\x74\x72" - "\xF8\xB2\xAA\x7A\xD6\xFF\xFA\x55" - "\x33\x1A\xBB\xD3\xA2\x7E\x97\x66", - .ilen = 1008, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" + "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" + "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" + "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" + "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" + "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" + "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" + "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" + "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" + "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" + "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" + "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" + "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" + "\x29\xC0\x57\xEE\x62\xF9\x90\x04" + "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" + "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" + "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" + "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" + "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" + "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" + "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" + "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" + "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" + "\x57\xEE\x85\x1C\x90\x27\xBE\x32" + "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" + "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" + "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" + "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" + "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" + "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" + "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" + "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" + "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" + "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" + "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" + "\x69\x00\x74\x0B\xA2\x16\xAD\x44" + "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" + "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" + "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" + "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" + "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" + "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" + "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" + "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" + "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" + "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" + "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" + "\x58\xEF\x86\x1D\x91\x28\xBF\x33" + "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" + "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" + "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" + "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" + "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" + "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" + "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" + "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" + "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" + "\x86\x1D\xB4\x28\xBF\x56\xED\x61" + "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" + "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" + "\x2B\xC2\x59", + .ctext = "\x84\x68\xEC\xF2\x1C\x88\x20\xCA" + "\x37\x69\xE3\x3A\x22\x85\x48\x46" + "\x70\xAA\x25\xB4\xCD\x8B\x04\x4E" + "\x8D\x15\x2B\x98\xDF\x7B\x6D\xB9" + "\xE0\x4A\x73\x00\x65\xB6\x1A\x0D" + "\x5C\x60\xDF\x34\xDC\x60\x4C\xDF" + "\xB5\x1F\x26\x8C\xDA\xC1\x11\xA8" + "\x80\xFA\x37\x7A\x89\xAA\xAE\x7B" + "\x92\x6E\xB9\xDC\xC9\x62\x4F\x88" + "\x0A\x5D\x97\x2F\x6B\xAC\x03\x7C" + "\x22\xF6\x55\x5A\xFA\x35\xA5\x17" + "\xA1\x5C\x5E\x2B\x63\x2D\xB9\x91" + "\x3E\x83\x26\x00\x4E\xD5\xBE\xCE" + "\x79\xC4\x3D\xFC\x70\xA0\xAD\x96" + "\xBA\x58\x2A\x1C\xDF\xC2\x3A\xA5" + "\x7C\xB5\x12\x89\xED\xBF\xB6\x09" + "\x13\x4F\x7D\x61\x3C\x5C\x27\xFC" + "\x5D\xE1\x4F\xA1\xEA\xB3\xCA\xB9" + "\xE6\xD0\x97\x81\xDE\xD1\xFB\x8A" + "\x30\xDB\xA3\x5D\xEC\x25\x0B\x86" + "\x71\xC8\xA7\x67\xE8\xBC\x7D\x4C" + "\xAE\x82\xD3\x73\x31\x09\xCB\xB3" + "\x4D\xD4\xC0\x8A\x2B\xFA\xA6\x55" + "\x39\x0A\xBC\x6E\x75\xAB\xC2\xE2" + "\x8A\xF2\x26\xCD\x63\x38\x35\xF7" + "\xAE\x12\x83\xCD\x8A\x9E\x7E\x4C" + "\xFE\x4D\xD7\xCE\x5C\x6E\x4C\xAF" + "\xE3\xCD\x76\xA7\x87\xA1\x54\x7C" + "\xEC\x32\xC7\x83\x2A\xFF\xF8\xEA" + "\x87\xB2\x47\xA3\x9D\xC2\x9C\xA2" + "\xB7\x2C\x7C\x1A\x24\xCB\x88\x61" + "\xFF\xA7\x1A\x16\x01\xDD\x4B\xFC" + "\x2E\xE0\x48\x67\x09\x42\xCC\x91" + "\xBE\x20\x38\xC0\x5E\x3B\x95\x00" + "\xA1\x96\x66\x0B\x8A\xE9\x9E\xF7" + "\x6B\x34\x0A\x51\xC0\x3B\xEB\x71" + "\x07\x97\x38\x4B\x5C\x56\x98\x67" + "\x78\x9C\xD0\x0E\x2B\xB5\x67\x90" + "\x75\xF8\xFE\x6D\x4E\x85\xCC\x0D" + "\x18\x06\x15\x9D\x5A\x10\x13\x37" + "\xA3\xD6\x68\xA2\xDF\x7E\xC7\x12" + "\xC9\x0D\x4D\x91\xB0\x2A\x55\xFF" + "\x6F\x73\x13\xDF\x28\xB5\x2A\x2C" + "\xE4\xFC\x20\xD9\xF1\x7A\x82\xB1" + "\xCB\x57\xB6\x3D\x8C\xF4\x8E\x27" + "\x37\xDC\x35\xF3\x79\x01\x53\xA4" + "\x7B\x37\xDE\x7C\x04\xAE\x50\xDB" + "\x9B\x1E\x8C\x07\xA7\x52\x49\x50" + "\x34\x25\x65\xDD\xA9\x8F\x7E\xBD" + "\x7A\xC9\x36\xAE\xDE\x21\x48\x64" + "\xC2\x02\xBA\xBE\x11\x1E\x3D\x9C" + "\x98\x52\xCC\x04\xBD\x5E\x61\x26" + "\x10\xD3\x21\xD9\x6E\x25\x98\x77" + "\x8E\x98\x63\xF6\xF6\x52\xFB\x13" + "\xAA\x30\xF2\xB9\xA4\x43\x53\x39" + "\x1C\x97\x07\x7E\x6B\xFF\x3D\x43" + "\xA6\x71\x6B\x66\x8F\x58\x3F\x71" + "\x90\x47\x40\x92\xE6\x69\xD1\x96" + "\x34\xB3\x3B\xE5\x43\xE4\xD5\x56" + "\xB2\xE6\x7E\x86\x7A\x12\x17\x5B" + "\x30\xF3\x9B\x0D\xFA\x57\xE4\x50" + "\x40\x53\x77\x8C\x15\xF8\x8D\x13" + "\x38\xE2\xE5", + .len = 499, + .also_non_np = 1, + .np = 2, + .tap = { 499 - 16, 16 }, + }, { /* Generated with Crypto++ */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .klen = 32, + .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" + "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -30593,107 +9427,1488 @@ static const struct cipher_testvec camellia_dec_tv_template[] = { "\x86\x1D\xB4\x28\xBF\x56\xED\x61" "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59\xF0\x64\xFB\x92\x06" - "\x9D\x34\xCB\x3F\xD6\x6D\x04\x78" - "\x0F\xA6\x1A\xB1\x48\xDF\x53\xEA" - "\x81\x18\x8C\x23\xBA\x2E\xC5\x5C" - "\xF3\x67\xFE\x95\x09\xA0\x37\xCE" - "\x42\xD9\x70\x07\x7B\x12\xA9\x1D" - "\xB4\x4B\xE2\x56\xED\x84\x1B\x8F" - "\x26\xBD\x31\xC8\x5F\xF6\x6A\x01" - "\x98\x0C\xA3\x3A\xD1\x45\xDC\x73" - "\x0A\x7E\x15\xAC\x20\xB7\x4E\xE5" - "\x59\xF0\x87\x1E\x92\x29\xC0\x34" - "\xCB\x62\xF9\x6D\x04\x9B\x0F\xA6" - "\x3D\xD4\x48\xDF\x76\x0D\x81\x18" - "\xAF\x23\xBA\x51\xE8\x5C\xF3\x8A" - "\x21\x95\x2C\xC3\x37\xCE\x65\xFC" - "\x70\x07\x9E\x12\xA9\x40\xD7\x4B" - "\xE2\x79\x10\x84\x1B\xB2\x26\xBD" - "\x54\xEB\x5F\xF6\x8D\x01\x98\x2F" - "\xC6\x3A\xD1\x68\xFF\x73\x0A\xA1" - "\x15\xAC\x43\xDA\x4E\xE5\x7C\x13" - "\x87\x1E\xB5\x29\xC0\x57\xEE\x62" - "\xF9\x90\x04\x9B\x32\xC9\x3D\xD4" - "\x6B\x02\x76\x0D\xA4\x18\xAF\x46" - "\xDD\x51\xE8\x7F\x16\x8A\x21\xB8" - "\x2C\xC3\x5A\xF1\x65\xFC\x93\x07" - "\x9E\x35\xCC\x40\xD7\x6E\x05\x79" - "\x10\xA7\x1B\xB2\x49\xE0\x54\xEB" - "\x82\x19\x8D\x24\xBB\x2F\xC6\x5D" - "\xF4\x68\xFF\x96\x0A\xA1\x38\xCF" - "\x43\xDA\x71\x08\x7C\x13\xAA\x1E" - "\xB5\x4C\xE3\x57\xEE\x85\x1C\x90" - "\x27\xBE\x32\xC9\x60\xF7\x6B\x02" - "\x99\x0D\xA4\x3B\xD2\x46\xDD\x74" - "\x0B\x7F\x16\xAD\x21\xB8\x4F\xE6" - "\x5A\xF1\x88\x1F\x93\x2A\xC1\x35" - "\xCC\x63\xFA\x6E\x05\x9C\x10\xA7" - "\x3E\xD5\x49\xE0\x77\x0E\x82\x19" - "\xB0\x24\xBB\x52\xE9\x5D\xF4\x8B" - "\x22\x96\x2D\xC4\x38\xCF\x66\xFD" - "\x71\x08\x9F\x13\xAA\x41\xD8\x4C" - "\xE3\x7A\x11\x85\x1C\xB3\x27\xBE" - "\x55\xEC\x60\xF7\x8E\x02\x99\x30" - "\xC7\x3B\xD2\x69\x00\x74\x0B\xA2" - "\x16\xAD\x44\xDB\x4F\xE6\x7D\x14" - "\x88\x1F\xB6\x2A\xC1\x58\xEF\x63" - "\xFA\x91\x05\x9C\x33\xCA\x3E\xD5" - "\x6C\x03\x77\x0E\xA5\x19\xB0\x47" - "\xDE\x52\xE9\x80\x17\x8B\x22\xB9" - "\x2D\xC4\x5B\xF2\x66\xFD\x94\x08" - "\x9F\x36\xCD\x41\xD8\x6F\x06\x7A" - "\x11\xA8\x1C\xB3\x4A\xE1\x55\xEC" - "\x83\x1A\x8E\x25\xBC\x30\xC7\x5E" - "\xF5\x69\x00\x97\x0B\xA2\x39\xD0" - "\x44\xDB\x72\x09\x7D\x14\xAB\x1F" - "\xB6\x4D\xE4\x58\xEF\x86\x1D\x91" - "\x28\xBF\x33\xCA\x61\xF8\x6C\x03" - "\x9A\x0E\xA5\x3C\xD3\x47\xDE\x75" - "\x0C\x80\x17\xAE\x22\xB9\x50\xE7" - "\x5B\xF2\x89\x20\x94\x2B\xC2\x36" - "\xCD\x64\xFB\x6F\x06\x9D\x11\xA8" - "\x3F\xD6\x4A\xE1\x78\x0F\x83\x1A" - "\xB1\x25\xBC\x53\xEA\x5E\xF5\x8C" - "\x00\x97\x2E\xC5\x39\xD0\x67\xFE" - "\x72\x09\xA0\x14\xAB\x42\xD9\x4D", - .rlen = 1008, + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\x06\x9A\xF8\xB4\x53\x88\x62\xFC" + "\x68\xB8\x2E\xDF\xC1\x05\x0F\x3D" + "\xAF\x4D\x95\xAE\xC4\xE9\x1C\xDC" + "\xF6\x2B\x8F\x90\x89\xF6\x7E\x1A" + "\xA6\xB9\xE4\xF4\xFA\xCA\xE5\x7E" + "\x71\x28\x06\x4F\xE8\x08\x39\xDA" + "\xA5\x0E\xC8\xC0\xB8\x16\xE5\x69" + "\xE5\xCA\xEC\x4F\x63\x2C\xC0\x9B" + "\x9F\x3E\x39\x79\xF0\xCD\x64\x35" + "\x4A\xD3\xC8\xA9\x31\xCD\x48\x5B" + "\x92\x3D\x8F\x3F\x96\xBD\xB3\x18" + "\x74\x2A\x5D\x29\x3F\x57\x8F\xE2" + "\x67\x9A\xE0\xE5\xD4\x4A\xE2\x47" + "\xBC\xF6\xEB\x14\xF3\x8C\x20\xC2" + "\x7D\xE2\x43\x81\x86\x72\x2E\xB1" + "\x39\xF6\x95\xE1\x1F\xCB\x76\x33" + "\x5B\x7D\x23\x0F\x3A\x67\x2A\x2F" + "\xB9\x37\x9D\xDD\x1F\x16\xA1\x3C" + "\x70\xFE\x52\xAA\x93\x3C\xC4\x46" + "\xB1\xE5\xFF\xDA\xAF\xE2\x84\xFE" + "\x25\x92\xB2\x63\xBD\x49\x77\xB4" + "\x22\xA4\x6A\xD5\x04\xE0\x45\x58" + "\x1C\x34\x96\x7C\x03\x0C\x13\xA2" + "\x05\x22\xE2\xCB\x5A\x35\x03\x09" + "\x40\xD2\x82\x05\xCA\x58\x73\xF2" + "\x29\x5E\x01\x47\x13\x32\x78\xBE" + "\x06\xB0\x51\xDB\x6C\x31\xA0\x1C" + "\x74\xBC\x8D\x25\xDF\xF8\x65\xD1" + "\x38\x35\x11\x26\x4A\xB4\x06\x32" + "\xFA\xD2\x07\x77\xB3\x74\x98\x80" + "\x61\x59\xA8\x9F\xF3\x6F\x2A\xBF" + "\xE6\xA5\x9A\xC4\x6B\xA6\x49\x6F" + "\xBC\x47\xD9\xFB\xC6\xEF\x25\x65" + "\x96\xAC\x9F\xE4\x81\x4B\xD8\xBA" + "\xD6\x9B\xC9\x6D\x58\x40\x81\x02" + "\x73\x44\x4E\x43\x6E\x37\xBB\x11" + "\xE3\xF9\xB8\x2F\xEC\x76\x34\xEA" + "\x90\xCD\xB7\x2E\x0E\x32\x71\xE8" + "\xBB\x4E\x0B\x98\xA4\x17\x17\x5B" + "\x07\xB5\x82\x3A\xC4\xE8\x42\x51" + "\x5A\x4C\x4E\x7D\xBF\xC4\xC0\x4F" + "\x68\xB8\xC6\x4A\x32\x6F\x0B\xD7" + "\x85\xED\x6B\xFB\x72\xD2\xA5\x8F" + "\xBF\xF9\xAC\x59\x50\xA8\x08\x70" + "\xEC\xBD\x0A\xBF\xE5\x87\xA1\xC2" + "\x92\x14\x78\xAF\xE8\xEA\x2E\xDD" + "\xC1\x03\x9A\xAA\x89\x8B\x32\x46" + "\x5B\x18\x27\xBA\x46\xAA\x64\xDE" + "\xE3\xD5\xA3\xFC\x7B\x5B\x61\xDB" + "\x7E\xDA\xEC\x30\x17\x19\xF8\x80" + "\xB5\x5E\x27\xB5\x37\x3A\x1F\x28" + "\x07\x73\xC3\x63\xCE\xFF\x8C\xFE" + "\x81\x4E\xF8\x24\xF3\xB8\xC7\xE8" + "\x16\x9A\xCC\x58\x2F\x88\x1C\x4B" + "\xBB\x33\xA2\x73\xF0\x1C\x89\x0E" + "\xDC\x34\x27\x89\x98\xCE\x1C\xA2" + "\xD8\xB8\x90\xBE\xEC\x72\x28\x13" + "\xAC\x7B\xF1\xD0\x7F\x7A\x28\x50" + "\xB7\x99\x65\x8A\xC9\xC6\x21\x34" + "\x7F\x67\x9D\xB7\x2C\xCC\xF5\x17" + "\x2B\x89\xAC\xB0\xD7\x1E\x47\xB0" + "\x61\xAF\xD4\x63\x6D\xB8\x2D\x20", + .len = 496, + }, +}; + +static const struct cipher_testvec serpent_lrw_tv_template[] = { + /* Generated from AES-LRW test vectors */ + { + .key = "\x45\x62\xac\x25\xf8\x28\x17\x6d" + "\x4c\x26\x84\x14\xb5\x68\x01\x85" + "\x25\x8e\x2a\x05\xe7\x3e\x9d\x03" + "\xee\x5a\x83\x0c\xcc\x09\x4c\x87", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x6f\xbf\xd4\xa4\x5d\x71\x16\x79" + "\x63\x9c\xa6\x8e\x40\xbe\x0d\x8a", + .len = 16, + }, { + .key = "\x59\x70\x47\x14\xf5\x57\x47\x8c" + "\xd7\x79\xe8\x0f\x54\x88\x79\x44" + "\x0d\x48\xf0\xb7\xb1\x5a\x53\xea" + "\x1c\xaa\x6b\x29\xc2\xca\xfb\xaf", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x02", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\xfd\xb2\x66\x98\x80\x96\x55\xad" + "\x08\x94\x54\x9c\x21\x7c\x69\xe3", + .len = 16, + }, { + .key = "\xd8\x2a\x91\x34\xb2\x6a\x56\x50" + "\x30\xfe\x69\xe2\x37\x7f\x98\x47" + "\xcd\xf9\x0b\x16\x0c\x64\x8f\xb6" + "\xb0\x0d\x0d\x1b\xae\x85\x87\x1f", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x02\x00\x00\x00\x00", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x14\x5e\x3d\x70\xc0\x6e\x9c\x34" + "\x5b\x5e\xcf\x0f\xe4\x8c\x21\x5c", + .len = 16, + }, { + .key = "\x0f\x6a\xef\xf8\xd3\xd2\xbb\x15" + "\x25\x83\xf7\x3c\x1f\x01\x28\x74" + "\xca\xc6\xbc\x35\x4d\x4a\x65\x54" + "\x90\xae\x61\xcf\x7b\xae\xbd\xcc" + "\xad\xe4\x94\xc5\x4a\x29\xae\x70", + .klen = 40, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x25\x39\xaa\xa5\xf0\x65\xc8\xdc" + "\x5d\x45\x95\x30\x8f\xff\x2f\x1b", + .len = 16, + }, { + .key = "\x8a\xd4\xee\x10\x2f\xbd\x81\xff" + "\xf8\x86\xce\xac\x93\xc5\xad\xc6" + "\xa0\x19\x07\xc0\x9d\xf7\xbb\xdd" + "\x52\x13\xb2\xb7\xf0\xff\x11\xd8" + "\xd6\x08\xd0\xcd\x2e\xb1\x17\x6f", + .klen = 40, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x02\x00\x00\x00\x00", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x0c\x20\x20\x63\xd6\x8b\xfc\x8f" + "\xc0\xe2\x17\xbb\xd2\x59\x6f\x26", + .len = 16, + }, { + .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" + "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" + "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" + "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" + "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" + "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\xc1\x35\x2e\x53\xf0\x96\x4d\x9c" + "\x2e\x18\xe6\x99\xcd\xd3\x15\x68", + .len = 16, + }, { + .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" + "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" + "\xb2\xfb\x64\xce\x60\x97\x87\x8d" + "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" + "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" + "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x02\x00\x00\x00\x00", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x86\x0a\xc6\xa9\x1a\x9f\xe7\xe6" + "\x64\x3b\x33\xd6\xd5\x84\xd6\xdf", + .len = 16, + }, { + .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" + "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" + "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" + "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" + "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" + "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" + "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" + "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" + "\x50\x38\x1f\x71\x49\xb6\x57\xd6" + "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" + "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" + "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" + "\xda\x10\x8e\xed\xa2\xa4\x87\xab" + "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" + "\xc9\xac\x42\x31\x95\x7c\xc9\x04" + "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" + "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" + "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" + "\x4c\x96\x12\xed\x7c\x92\x03\x01" + "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" + "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" + "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" + "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" + "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" + "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" + "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" + "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" + "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" + "\x76\x12\x73\x44\x1a\x56\xd7\x72" + "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" + "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" + "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" + "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" + "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" + "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" + "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" + "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" + "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" + "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" + "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" + "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" + "\x8d\x23\x31\x74\x84\xeb\x88\x6e" + "\xcc\xb9\xbc\x22\x83\x19\x07\x22" + "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" + "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" + "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" + "\x3c\xce\x8f\x42\x60\x71\xa7\x75" + "\x08\x40\x65\x8a\x82\xbf\xf5\x43" + "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" + "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" + "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" + "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" + "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" + "\x62\x73\x65\xfd\x46\x63\x25\x3d" + "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" + "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" + "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" + "\xc5\x68\x77\x84\x32\x2b\xcc\x85" + "\x74\x96\xf0\x12\x77\x61\xb9\xeb" + "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" + "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" + "\xda\x39\x87\x45\xc0\x2b\xbb\x01" + "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" + "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" + "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" + "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" + "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" + "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" + "\x21\xc4\xc2\x75\x67\x89\x37\x0a", + .ctext = "\xe3\x5a\x38\x0f\x4d\x92\x3a\x74" + "\x15\xb1\x50\x8c\x9a\xd8\x99\x1d" + "\x82\xec\xf1\x5f\x03\x6d\x02\x58" + "\x90\x67\xfc\xdd\x8d\xe1\x38\x08" + "\x7b\xc9\x9b\x4b\x04\x09\x50\x15" + "\xce\xab\xda\x33\x30\x20\x12\xfa" + "\x83\xc4\xa6\x9a\x2e\x7d\x90\xd9" + "\xa6\xa6\x67\x43\xb4\xa7\xa8\x5c" + "\xbb\x6a\x49\x2b\x8b\xf8\xd0\x22" + "\xe5\x9e\xba\xe8\x8c\x67\xb8\x5b" + "\x60\xbc\xf5\xa4\x95\x4e\x66\xe5" + "\x6d\x8e\xa9\xf6\x65\x2e\x04\xf5" + "\xba\xb5\xdb\x88\xc2\xf6\x7a\x4b" + "\x89\x58\x7c\x9a\xae\x26\xe8\xb7" + "\xb7\x28\xcc\xd6\xcc\xa5\x98\x4d" + "\xb9\x91\xcb\xb4\xe4\x8b\x96\x47" + "\x5f\x03\x8b\xdd\x94\xd1\xee\x12" + "\xa7\x83\x80\xf2\xc1\x15\x74\x4f" + "\x49\xf9\xb0\x7e\x6f\xdc\x73\x2f" + "\xe2\xcf\xe0\x1b\x34\xa5\xa0\x52" + "\xfb\x3c\x5d\x85\x91\xe6\x6d\x98" + "\x04\xd6\xdd\x4c\x00\x64\xd9\x54" + "\x5c\x3c\x08\x1d\x4c\x06\x9f\xb8" + "\x1c\x4d\x8d\xdc\xa4\x3c\xb9\x3b" + "\x9e\x85\xce\xc3\xa8\x4a\x0c\xd9" + "\x04\xc3\x6f\x17\x66\xa9\x1f\x59" + "\xd9\xe2\x19\x36\xa3\x88\xb8\x0b" + "\x0f\x4a\x4d\xf8\xc8\x6f\xd5\x43" + "\xeb\xa0\xab\x1f\x61\xc0\x06\xeb" + "\x93\xb7\xb8\x6f\x0d\xbd\x07\x49" + "\xb3\xac\x5d\xcf\x31\xa0\x27\x26" + "\x21\xbe\x94\x2e\x19\xea\xf4\xee" + "\xb5\x13\x89\xf7\x94\x0b\xef\x59" + "\x44\xc5\x78\x8b\x3c\x3b\x71\x20" + "\xf9\x35\x0c\x70\x74\xdc\x5b\xc2" + "\xb4\x11\x0e\x2c\x61\xa1\x52\x46" + "\x18\x11\x16\xc6\x86\x44\xa7\xaf" + "\xd5\x0c\x7d\xa6\x9e\x25\x2d\x1b" + "\x9a\x8f\x0f\xf8\x6a\x61\xa0\xea" + "\x3f\x0e\x90\xd6\x8f\x83\x30\x64" + "\xb5\x51\x2d\x08\x3c\xcd\x99\x36" + "\x96\xd4\xb1\xb5\x48\x30\xca\x48" + "\xf7\x11\xa8\xf5\x97\x8a\x6a\x6d" + "\x12\x33\x2f\xc0\xe8\xda\xec\x8a" + "\xe1\x88\x72\x63\xde\x20\xa3\xe1" + "\x8e\xac\x84\x37\x35\xf5\xf7\x3f" + "\x00\x02\x0e\xe4\xc1\x53\x68\x3f" + "\xaa\xd5\xac\x52\x3d\x20\x2f\x4d" + "\x7c\x83\xd0\xbd\xaa\x97\x35\x36" + "\x98\x88\x59\x5d\xe7\x24\xe3\x90" + "\x9d\x30\x47\xa7\xc3\x60\x35\xf4" + "\xd5\xdb\x0e\x4d\x44\xc1\x81\x8b" + "\xfd\xbd\xc3\x2b\xba\x68\xfe\x8d" + "\x49\x5a\x3c\x8a\xa3\x01\xae\x25" + "\x42\xab\xd2\x87\x1b\x35\xd6\xd2" + "\xd7\x70\x1c\x1f\x72\xd1\xe1\x39" + "\x1c\x58\xa2\xb4\xd0\x78\x55\x72" + "\x76\x59\xea\xd9\xd7\x6e\x63\x8b" + "\xcc\x9b\xa7\x74\x89\xfc\xa3\x68" + "\x86\x28\xd1\xbb\x54\x8d\x66\xad" + "\x2a\x92\xf9\x4e\x04\x3d\xae\xfd" + "\x1b\x2b\x7f\xc3\x2f\x1a\x78\x0a" + "\x5c\xc6\x84\xfe\x7c\xcb\x26\xfd" + "\xd9\x51\x0f\xd7\x94\x2f\xc5\xa7", + .len = 512, + .also_non_np = 1, + .np = 3, + .tap = { 512 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec serpent_xts_tv_template[] = { + /* Generated from AES-XTS test vectors */ + { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\xe1\x08\xb8\x1d\x2c\xf5\x33\x64" + "\xc8\x12\x04\xc7\xb3\x70\xe8\xc4" + "\x6a\x31\xc5\xf3\x00\xca\xb9\x16" + "\xde\xe2\x77\x66\xf7\xfe\x62\x08", + .len = 32, + }, { + .key = "\x11\x11\x11\x11\x11\x11\x11\x11" + "\x11\x11\x11\x11\x11\x11\x11\x11" + "\x22\x22\x22\x22\x22\x22\x22\x22" + "\x22\x22\x22\x22\x22\x22\x22\x22", + .klen = 32, + .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44", + .ctext = "\x1a\x0a\x09\x5f\xcd\x07\x07\x98" + "\x41\x86\x12\xaf\xb3\xd7\x68\x13" + "\xed\x81\xcd\x06\x87\x43\x1a\xbb" + "\x13\x3d\xd6\x1e\x2b\xe1\x77\xbe", + .len = 32, + }, { + .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" + "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" + "\x22\x22\x22\x22\x22\x22\x22\x22" + "\x22\x22\x22\x22\x22\x22\x22\x22", + .klen = 32, + .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44", + .ctext = "\xf9\x9b\x28\xb8\x5c\xaf\x8c\x61" + "\xb6\x1c\x81\x8f\x2c\x87\x60\x89" + "\x0d\x8d\x7a\xe8\x60\x48\xcc\x86" + "\xc1\x68\x45\xaa\x00\xe9\x24\xc5", + .len = 32, + }, { + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x31\x41\x59\x26\x53\x58\x97\x93" + "\x23\x84\x62\x64\x33\x83\x27\x95", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ctext = "\xfe\x47\x4a\xc8\x60\x7e\xb4\x8b" + "\x0d\x10\xf4\xb0\x0d\xba\xf8\x53" + "\x65\x6e\x38\x4b\xdb\xaa\xb1\x9e" + "\x28\xca\xb0\x22\xb3\x85\x75\xf4" + "\x00\x5c\x75\x14\x06\xd6\x25\x82" + "\xe6\xcb\x08\xf7\x29\x90\x23\x8e" + "\xa4\x68\x57\xe4\xf0\xd8\x32\xf3" + "\x80\x51\x67\xb5\x0b\x85\x69\xe8" + "\x19\xfe\xc4\xc7\x3e\xea\x90\xd3" + "\x8f\xa3\xf2\x0a\xac\x17\x4b\xa0" + "\x63\x5a\x16\x0f\xf0\xce\x66\x1f" + "\x2c\x21\x07\xf1\xa4\x03\xa3\x44" + "\x41\x61\x87\x5d\x6b\xb3\xef\xd4" + "\xfc\xaa\x32\x7e\x55\x58\x04\x41" + "\xc9\x07\x33\xc6\xa2\x68\xd6\x5a" + "\x55\x79\x4b\x6f\xcf\x89\xb9\x19" + "\xe5\x54\x13\x15\xb2\x1a\xfa\x15" + "\xc2\xf0\x06\x59\xfa\xa0\x25\x05" + "\x58\xfa\x43\x91\x16\x85\x40\xbb" + "\x0d\x34\x4d\xc5\x1e\x20\xd5\x08" + "\xcd\x22\x22\x41\x11\x9f\x6c\x7c" + "\x8d\x57\xc9\xba\x57\xe8\x2c\xf7" + "\xa0\x42\xa8\xde\xfc\xa3\xca\x98" + "\x4b\x43\xb1\xce\x4b\xbf\x01\x67" + "\x6e\x29\x60\xbd\x10\x14\x84\x82" + "\x83\x82\x0c\x63\x73\x92\x02\x7c" + "\x55\x37\x20\x80\x17\x51\xc8\xbc" + "\x46\x02\xcb\x38\x07\x6d\xe2\x85" + "\xaa\x29\xaf\x24\x58\x0d\xf0\x75" + "\x08\x0a\xa5\x34\x25\x16\xf3\x74" + "\xa7\x0b\x97\xbe\xc1\xa9\xdc\x29" + "\x1a\x0a\x56\xc1\x1a\x91\x97\x8c" + "\x0b\xc7\x16\xed\x5a\x22\xa6\x2e" + "\x8c\x2b\x4f\x54\x76\x47\x53\x8e" + "\xe8\x00\xec\x92\xb9\x55\xe6\xa2" + "\xf3\xe2\x4f\x6a\x66\x60\xd0\x87" + "\xe6\xd1\xcc\xe3\x6a\xc5\x2d\x21" + "\xcc\x9d\x6a\xb6\x75\xaa\xe2\x19" + "\x21\x9f\xa1\x5e\x4c\xfd\x72\xf9" + "\x94\x4e\x63\xc7\xae\xfc\xed\x47" + "\xe2\xfe\x7a\x63\x77\xfe\x97\x82" + "\xb1\x10\x6e\x36\x1d\xe1\xc4\x80" + "\xec\x69\x41\xec\xa7\x8a\xe0\x2f" + "\xe3\x49\x26\xa2\x41\xb2\x08\x0f" + "\x28\xb4\xa7\x39\xa1\x99\x2d\x1e" + "\x43\x42\x35\xd0\xcf\xec\x77\x67" + "\xb2\x3b\x9e\x1c\x35\xde\x4f\x5e" + "\x73\x3f\x5d\x6f\x07\x4b\x2e\x50" + "\xab\x6c\x6b\xff\xea\x00\x67\xaa" + "\x0e\x82\x32\xdd\x3d\xb5\xe5\x76" + "\x2b\x77\x3f\xbe\x12\x75\xfb\x92" + "\xc6\x89\x67\x4d\xca\xf7\xd4\x50" + "\xc0\x74\x47\xcc\xd9\x0a\xd4\xc6" + "\x3b\x17\x2e\xe3\x35\xbb\x53\xb5" + "\x86\xad\x51\xcc\xd5\x96\xb8\xdc" + "\x03\x57\xe6\x98\x52\x2f\x61\x62" + "\xc4\x5c\x9c\x36\x71\x07\xfb\x94" + "\xe3\x02\xc4\x2b\x08\x75\xc7\x35" + "\xfb\x2e\x88\x7b\xbb\x67\x00\xe1" + "\xc9\xdd\x99\xb2\x13\x53\x1a\x4e" + "\x76\x87\x19\x04\x1a\x2f\x38\x3e" + "\xef\x91\x64\x1d\x18\x07\x4e\x31" + "\x88\x21\x7c\xb0\xa5\x12\x4c\x3c" + "\xb0\x20\xbd\xda\xdf\xf9\x7c\xdd", + .len = 512, + }, { + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x62\x49\x77\x57\x24\x70\x93\x69" + "\x99\x59\x57\x49\x66\x96\x76\x27" + "\x31\x41\x59\x26\x53\x58\x97\x93" + "\x23\x84\x62\x64\x33\x83\x27\x95" + "\x02\x88\x41\x97\x16\x93\x99\x37" + "\x51\x05\x82\x09\x74\x94\x45\x92", + .klen = 64, + .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ctext = "\x2b\xc9\xb4\x6b\x10\x94\xa9\x32" + "\xaa\xb0\x20\xc6\x44\x3d\x74\x1f" + "\x75\x01\xa7\xf6\xf5\xf7\x62\x1b" + "\x80\x1b\x82\xcb\x01\x59\x91\x7f" + "\x80\x3a\x98\xf0\xd2\xca\xc4\xc3" + "\x34\xfd\xe6\x11\xf9\x33\x45\x12" + "\x48\xc5\x8c\x25\xf1\xc5\xc5\x23" + "\xd3\x44\xb4\x73\xd5\x04\xc0\xb7" + "\xca\x2f\xf5\xcd\xc5\xb4\xdd\xb0" + "\xf4\x60\xe8\xfb\xc6\x9c\xc5\x78" + "\xcd\xec\x7d\xdc\x19\x9c\x72\x64" + "\x63\x0b\x38\x2e\x76\xdd\x2d\x36" + "\x49\xb0\x1d\xea\x78\x9e\x00\xca" + "\x20\xcc\x1b\x1e\x98\x74\xab\xed" + "\x79\xf7\xd0\x6c\xd8\x93\x80\x29" + "\xac\xa5\x5e\x34\xa9\xab\xa0\x55" + "\x9a\xea\xaa\x95\x4d\x7b\xfe\x46" + "\x26\x8a\xfd\x88\xa2\xa8\xa6\xae" + "\x25\x42\x17\xbf\x76\x8f\x1c\x3d" + "\xec\x9a\xda\x64\x96\xb5\x61\xff" + "\x99\xeb\x12\x96\x85\x82\x9d\xd5" + "\x81\x85\x14\xa8\x59\xac\x8c\x94" + "\xbb\x3b\x85\x2b\xdf\xb3\x0c\xba" + "\x82\xc6\x4d\xca\x86\xea\x53\x28" + "\x4c\xe0\x4e\x31\xe3\x73\x2f\x79" + "\x9d\x42\xe1\x03\xe3\x8b\xc4\xff" + "\x05\xca\x81\x7b\xda\xa2\xde\x63" + "\x3a\x10\xbe\xc2\xac\x32\xc4\x05" + "\x47\x7e\xef\x67\xe2\x5f\x5b\xae" + "\xed\xf1\x70\x34\x16\x9a\x07\x7b" + "\xf2\x25\x2b\xb0\xf8\x3c\x15\x9a" + "\xa6\x59\x55\x5f\xc1\xf4\x1e\xcd" + "\x93\x1f\x06\xba\xd4\x9a\x22\x69" + "\xfa\x8e\x95\x0d\xf3\x23\x59\x2c" + "\xfe\x00\xba\xf0\x0e\xbc\x6d\xd6" + "\x62\xf0\x7a\x0e\x83\x3e\xdb\x32" + "\xfd\x43\x7d\xda\x42\x51\x87\x43" + "\x9d\xf9\xef\xf4\x30\x97\xf8\x09" + "\x88\xfc\x3f\x93\x70\xc1\x4a\xec" + "\x27\x5f\x11\xac\x71\xc7\x48\x46" + "\x2f\xf9\xdf\x8d\x9f\xf7\x2e\x56" + "\x0d\x4e\xb0\x32\x76\xce\x86\x81" + "\xcd\xdf\xe4\x00\xbf\xfd\x5f\x24" + "\xaf\xf7\x9a\xde\xff\x18\xac\x14" + "\x90\xc5\x01\x39\x34\x0f\x24\xf3" + "\x13\x2f\x5e\x4f\x30\x9a\x36\x40" + "\xec\xea\xbc\xcd\x9e\x0e\x5b\x23" + "\x50\x88\x97\x40\x69\xb1\x37\xf5" + "\xc3\x15\xf9\x3f\xb7\x79\x64\xe8" + "\x7b\x10\x20\xb9\x2b\x46\x83\x5b" + "\xd8\x39\xfc\xe4\xfa\x88\x52\xf2" + "\x72\xb0\x97\x4e\x89\xb3\x48\x00" + "\xc1\x16\x73\x50\x77\xba\xa6\x65" + "\x20\x2d\xb0\x02\x27\x89\xda\x99" + "\x45\xfb\xe9\xd3\x1d\x39\x2f\xd6" + "\x2a\xda\x09\x12\x11\xaf\xe6\x57" + "\x01\x04\x8a\xff\x86\x8b\xac\xf8" + "\xee\xe4\x1c\x98\x5b\xcf\x6b\x76" + "\xa3\x0e\x33\x74\x40\x18\x39\x72" + "\x66\x50\x31\xfd\x70\xdf\xe8\x51" + "\x96\x21\x36\xb2\x9b\xfa\x85\xd1" + "\x30\x05\xc8\x92\x98\x80\xff\x7a" + "\xaf\x43\x0b\xc5\x20\x41\x92\x20" + "\xd4\xa0\x91\x98\x11\x5f\x4d\xb1", + .len = 512, + .also_non_np = 1, + .np = 3, + .tap = { 512 - 20, 4, 16 }, + }, +}; + +/* + * SM4 test vector taken from the draft RFC + * https://tools.ietf.org/html/draft-crypto-sm4-00#ref-GBT.32907-2016 + */ + +static const struct cipher_testvec sm4_tv_template[] = { + { /* SM4 Appendix A: Example Calculations. Example 1. */ + .key = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" + "\xFE\xDC\xBA\x98\x76\x54\x32\x10", + .klen = 16, + .ptext = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" + "\xFE\xDC\xBA\x98\x76\x54\x32\x10", + .ctext = "\x68\x1E\xDF\x34\xD2\x06\x96\x5E" + "\x86\xB3\xE9\x4F\x53\x6E\x42\x46", + .len = 16, + }, { /* + * SM4 Appendix A: Example Calculations. + * Last 10 iterations of Example 2. + */ + .key = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" + "\xFE\xDC\xBA\x98\x76\x54\x32\x10", + .klen = 16, + .ptext = "\x99\x4a\xc3\xe7\xc3\x57\x89\x6a" + "\x81\xfc\xa8\xe\x38\x3e\xef\x80" + "\xb1\x98\xf2\xde\x3f\x4b\xae\xd1" + "\xf0\xf1\x30\x4c\x1\x27\x5a\x8f" + "\x45\xe1\x39\xb7\xae\xff\x1f\x27" + "\xad\x57\x15\xab\x31\x5d\xc\xef" + "\x8c\xc8\x80\xbd\x11\x98\xf3\x7b" + "\xa2\xdd\x14\x20\xf9\xe8\xbb\x82" + "\xf7\x32\xca\x4b\xa8\xf7\xb3\x4d" + "\x27\xd1\xcd\xe6\xb6\x65\x5a\x23" + "\xc2\xf3\x54\x84\x53\xe3\xb9\x20" + "\xa5\x37\x0\xbe\xe7\x7b\x48\xfb" + "\x21\x3d\x9e\x48\x1d\x9e\xf5\xbf" + "\x77\xd5\xb4\x4a\x53\x71\x94\x7a" + "\x88\xa6\x6e\x6\x93\xca\x43\xa5" + "\xc4\xf6\xcd\x53\x4b\x7b\x8e\xfe" + "\xb4\x28\x7c\x42\x29\x32\x5d\x88" + "\xed\xce\x0\x19\xe\x16\x2\x6e" + "\x87\xff\x2c\xac\xe8\xe7\xe9\xbf" + "\x31\x51\xec\x47\xc3\x51\x83\xc1", + .ctext = "\xb1\x98\xf2\xde\x3f\x4b\xae\xd1" + "\xf0\xf1\x30\x4c\x1\x27\x5a\x8f" + "\x45\xe1\x39\xb7\xae\xff\x1f\x27" + "\xad\x57\x15\xab\x31\x5d\xc\xef" + "\x8c\xc8\x80\xbd\x11\x98\xf3\x7b" + "\xa2\xdd\x14\x20\xf9\xe8\xbb\x82" + "\xf7\x32\xca\x4b\xa8\xf7\xb3\x4d" + "\x27\xd1\xcd\xe6\xb6\x65\x5a\x23" + "\xc2\xf3\x54\x84\x53\xe3\xb9\x20" + "\xa5\x37\x0\xbe\xe7\x7b\x48\xfb" + "\x21\x3d\x9e\x48\x1d\x9e\xf5\xbf" + "\x77\xd5\xb4\x4a\x53\x71\x94\x7a" + "\x88\xa6\x6e\x6\x93\xca\x43\xa5" + "\xc4\xf6\xcd\x53\x4b\x7b\x8e\xfe" + "\xb4\x28\x7c\x42\x29\x32\x5d\x88" + "\xed\xce\x0\x19\xe\x16\x2\x6e" + "\x87\xff\x2c\xac\xe8\xe7\xe9\xbf" + "\x31\x51\xec\x47\xc3\x51\x83\xc1" + "\x59\x52\x98\xc7\xc6\xfd\x27\x1f" + "\x4\x2\xf8\x4\xc3\x3d\x3f\x66", + .len = 160 + } +}; + +/* + * Speck test vectors taken from the original paper: + * "The Simon and Speck Families of Lightweight Block Ciphers" + * https://eprint.iacr.org/2013/404.pdf + * + * Note that the paper does not make byte and word order clear. But it was + * confirmed with the authors that the intended orders are little endian byte + * order and (y, x) word order. Equivalently, the printed test vectors, when + * looking at only the bytes (ignoring the whitespace that divides them into + * words), are backwards: the left-most byte is actually the one with the + * highest memory address, while the right-most byte is actually the one with + * the lowest memory address. + */ + +static const struct cipher_testvec speck128_tv_template[] = { + { /* Speck128/128 */ + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .klen = 16, + .ptext = "\x20\x6d\x61\x64\x65\x20\x69\x74" + "\x20\x65\x71\x75\x69\x76\x61\x6c", + .ctext = "\x18\x0d\x57\x5c\xdf\xfe\x60\x78" + "\x65\x32\x78\x79\x51\x98\x5d\xa6", + .len = 16, + }, { /* Speck128/192 */ + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17", + .klen = 24, + .ptext = "\x65\x6e\x74\x20\x74\x6f\x20\x43" + "\x68\x69\x65\x66\x20\x48\x61\x72", + .ctext = "\x86\x18\x3c\xe0\x5d\x18\xbc\xf9" + "\x66\x55\x13\x13\x3a\xcf\xe4\x1b", + .len = 16, + }, { /* Speck128/256 */ + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .klen = 32, + .ptext = "\x70\x6f\x6f\x6e\x65\x72\x2e\x20" + "\x49\x6e\x20\x74\x68\x6f\x73\x65", + .ctext = "\x43\x8f\x18\x9c\x8d\xb4\xee\x4e" + "\x3e\xf5\xc0\x05\x04\x01\x09\x41", + .len = 16, + }, +}; + +/* + * Speck128-XTS test vectors, taken from the AES-XTS test vectors with the + * ciphertext recomputed with Speck128 as the cipher + */ +static const struct cipher_testvec speck128_xts_tv_template[] = { + { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\xbe\xa0\xe7\x03\xd7\xfe\xab\x62" + "\x3b\x99\x4a\x64\x74\x77\xac\xed" + "\xd8\xf4\xa6\xcf\xae\xb9\x07\x42" + "\x51\xd9\xb6\x1d\xe0\x5e\xbc\x54", + .len = 32, + }, { + .key = "\x11\x11\x11\x11\x11\x11\x11\x11" + "\x11\x11\x11\x11\x11\x11\x11\x11" + "\x22\x22\x22\x22\x22\x22\x22\x22" + "\x22\x22\x22\x22\x22\x22\x22\x22", + .klen = 32, + .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44", + .ctext = "\xfb\x53\x81\x75\x6f\x9f\x34\xad" + "\x7e\x01\xed\x7b\xcc\xda\x4e\x4a" + "\xd4\x84\xa4\x53\xd5\x88\x73\x1b" + "\xfd\xcb\xae\x0d\xf3\x04\xee\xe6", + .len = 32, + }, { + .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" + "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" + "\x22\x22\x22\x22\x22\x22\x22\x22" + "\x22\x22\x22\x22\x22\x22\x22\x22", + .klen = 32, + .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44", + .ctext = "\x21\x52\x84\x15\xd1\xf7\x21\x55" + "\xd9\x75\x4a\xd3\xc5\xdb\x9f\x7d" + "\xda\x63\xb2\xf1\x82\xb0\x89\x59" + "\x86\xd4\xaa\xaa\xdd\xff\x4f\x92", + .len = 32, + }, { + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x31\x41\x59\x26\x53\x58\x97\x93" + "\x23\x84\x62\x64\x33\x83\x27\x95", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ctext = "\x57\xb5\xf8\x71\x6e\x6d\xdd\x82" + "\x53\xd0\xed\x2d\x30\xc1\x20\xef" + "\x70\x67\x5e\xff\x09\x70\xbb\xc1" + "\x3a\x7b\x48\x26\xd9\x0b\xf4\x48" + "\xbe\xce\xb1\xc7\xb2\x67\xc4\xa7" + "\x76\xf8\x36\x30\xb7\xb4\x9a\xd9" + "\xf5\x9d\xd0\x7b\xc1\x06\x96\x44" + "\x19\xc5\x58\x84\x63\xb9\x12\x68" + "\x68\xc7\xaa\x18\x98\xf2\x1f\x5c" + "\x39\xa6\xd8\x32\x2b\xc3\x51\xfd" + "\x74\x79\x2e\xb4\x44\xd7\x69\xc4" + "\xfc\x29\xe6\xed\x26\x1e\xa6\x9d" + "\x1c\xbe\x00\x0e\x7f\x3a\xca\xfb" + "\x6d\x13\x65\xa0\xf9\x31\x12\xe2" + "\x26\xd1\xec\x2b\x0a\x8b\x59\x99" + "\xa7\x49\xa0\x0e\x09\x33\x85\x50" + "\xc3\x23\xca\x7a\xdd\x13\x45\x5f" + "\xde\x4c\xa7\xcb\x00\x8a\x66\x6f" + "\xa2\xb6\xb1\x2e\xe1\xa0\x18\xf6" + "\xad\xf3\xbd\xeb\xc7\xef\x55\x4f" + "\x79\x91\x8d\x36\x13\x7b\xd0\x4a" + "\x6c\x39\xfb\x53\xb8\x6f\x02\x51" + "\xa5\x20\xac\x24\x1c\x73\x59\x73" + "\x58\x61\x3a\x87\x58\xb3\x20\x56" + "\x39\x06\x2b\x4d\xd3\x20\x2b\x89" + "\x3f\xa2\xf0\x96\xeb\x7f\xa4\xcd" + "\x11\xae\xbd\xcb\x3a\xb4\xd9\x91" + "\x09\x35\x71\x50\x65\xac\x92\xe3" + "\x7b\x32\xc0\x7a\xdd\xd4\xc3\x92" + "\x6f\xeb\x79\xde\x6f\xd3\x25\xc9" + "\xcd\x63\xf5\x1e\x7a\x3b\x26\x9d" + "\x77\x04\x80\xa9\xbf\x38\xb5\xbd" + "\xb8\x05\x07\xbd\xfd\xab\x7b\xf8" + "\x2a\x26\xcc\x49\x14\x6d\x55\x01" + "\x06\x94\xd8\xb2\x2d\x53\x83\x1b" + "\x8f\xd4\xdd\x57\x12\x7e\x18\xba" + "\x8e\xe2\x4d\x80\xef\x7e\x6b\x9d" + "\x24\xa9\x60\xa4\x97\x85\x86\x2a" + "\x01\x00\x09\xf1\xcb\x4a\x24\x1c" + "\xd8\xf6\xe6\x5b\xe7\x5d\xf2\xc4" + "\x97\x1c\x10\xc6\x4d\x66\x4f\x98" + "\x87\x30\xac\xd5\xea\x73\x49\x10" + "\x80\xea\xe5\x5f\x4d\x5f\x03\x33" + "\x66\x02\x35\x3d\x60\x06\x36\x4f" + "\x14\x1c\xd8\x07\x1f\x78\xd0\xf8" + "\x4f\x6c\x62\x7c\x15\xa5\x7c\x28" + "\x7c\xcc\xeb\x1f\xd1\x07\x90\x93" + "\x7e\xc2\xa8\x3a\x80\xc0\xf5\x30" + "\xcc\x75\xcf\x16\x26\xa9\x26\x3b" + "\xe7\x68\x2f\x15\x21\x5b\xe4\x00" + "\xbd\x48\x50\xcd\x75\x70\xc4\x62" + "\xbb\x41\xfb\x89\x4a\x88\x3b\x3b" + "\x51\x66\x02\x69\x04\x97\x36\xd4" + "\x75\xae\x0b\xa3\x42\xf8\xca\x79" + "\x8f\x93\xe9\xcc\x38\xbd\xd6\xd2" + "\xf9\x70\x4e\xc3\x6a\x8e\x25\xbd" + "\xea\x15\x5a\xa0\x85\x7e\x81\x0d" + "\x03\xe7\x05\x39\xf5\x05\x26\xee" + "\xec\xaa\x1f\x3d\xc9\x98\x76\x01" + "\x2c\xf4\xfc\xa3\x88\x77\x38\xc4" + "\x50\x65\x50\x6d\x04\x1f\xdf\x5a" + "\xaa\xf2\x01\xa9\xc1\x8d\xee\xca" + "\x47\x26\xef\x39\xb8\xb4\xf2\xd1" + "\xd6\xbb\x1b\x2a\xc1\x34\x14\xcf", + .len = 512, + }, { + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x62\x49\x77\x57\x24\x70\x93\x69" + "\x99\x59\x57\x49\x66\x96\x76\x27" + "\x31\x41\x59\x26\x53\x58\x97\x93" + "\x23\x84\x62\x64\x33\x83\x27\x95" + "\x02\x88\x41\x97\x16\x93\x99\x37" + "\x51\x05\x82\x09\x74\x94\x45\x92", + .klen = 64, + .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ctext = "\xc5\x85\x2a\x4b\x73\xe4\xf6\xf1" + "\x7e\xf9\xf6\xe9\xa3\x73\x36\xcb" + "\xaa\xb6\x22\xb0\x24\x6e\x3d\x73" + "\x92\x99\xde\xd3\x76\xed\xcd\x63" + "\x64\x3a\x22\x57\xc1\x43\x49\xd4" + "\x79\x36\x31\x19\x62\xae\x10\x7e" + "\x7d\xcf\x7a\xe2\x6b\xce\x27\xfa" + "\xdc\x3d\xd9\x83\xd3\x42\x4c\xe0" + "\x1b\xd6\x1d\x1a\x6f\xd2\x03\x00" + "\xfc\x81\x99\x8a\x14\x62\xf5\x7e" + "\x0d\xe7\x12\xe8\x17\x9d\x0b\xec" + "\xe2\xf7\xc9\xa7\x63\xd1\x79\xb6" + "\x62\x62\x37\xfe\x0a\x4c\x4a\x37" + "\x70\xc7\x5e\x96\x5f\xbc\x8e\x9e" + "\x85\x3c\x4f\x26\x64\x85\xbc\x68" + "\xb0\xe0\x86\x5e\x26\x41\xce\x11" + "\x50\xda\x97\x14\xe9\x9e\xc7\x6d" + "\x3b\xdc\x43\xde\x2b\x27\x69\x7d" + "\xfc\xb0\x28\xbd\x8f\xb1\xc6\x31" + "\x14\x4d\xf0\x74\x37\xfd\x07\x25" + "\x96\x55\xe5\xfc\x9e\x27\x2a\x74" + "\x1b\x83\x4d\x15\x83\xac\x57\xa0" + "\xac\xa5\xd0\x38\xef\x19\x56\x53" + "\x25\x4b\xfc\xce\x04\x23\xe5\x6b" + "\xf6\xc6\x6c\x32\x0b\xb3\x12\xc5" + "\xed\x22\x34\x1c\x5d\xed\x17\x06" + "\x36\xa3\xe6\x77\xb9\x97\x46\xb8" + "\xe9\x3f\x7e\xc7\xbc\x13\x5c\xdc" + "\x6e\x3f\x04\x5e\xd1\x59\xa5\x82" + "\x35\x91\x3d\x1b\xe4\x97\x9f\x92" + "\x1c\x5e\x5f\x6f\x41\xd4\x62\xa1" + "\x8d\x39\xfc\x42\xfb\x38\x80\xb9" + "\x0a\xe3\xcc\x6a\x93\xd9\x7a\xb1" + "\xe9\x69\xaf\x0a\x6b\x75\x38\xa7" + "\xa1\xbf\xf7\xda\x95\x93\x4b\x78" + "\x19\xf5\x94\xf9\xd2\x00\x33\x37" + "\xcf\xf5\x9e\x9c\xf3\xcc\xa6\xee" + "\x42\xb2\x9e\x2c\x5f\x48\x23\x26" + "\x15\x25\x17\x03\x3d\xfe\x2c\xfc" + "\xeb\xba\xda\xe0\x00\x05\xb6\xa6" + "\x07\xb3\xe8\x36\x5b\xec\x5b\xbf" + "\xd6\x5b\x00\x74\xc6\x97\xf1\x6a" + "\x49\xa1\xc3\xfa\x10\x52\xb9\x14" + "\xad\xb7\x73\xf8\x78\x12\xc8\x59" + "\x17\x80\x4c\x57\x39\xf1\x6d\x80" + "\x25\x77\x0f\x5e\x7d\xf0\xaf\x21" + "\xec\xce\xb7\xc8\x02\x8a\xed\x53" + "\x2c\x25\x68\x2e\x1f\x85\x5e\x67" + "\xd1\x07\x7a\x3a\x89\x08\xe0\x34" + "\xdc\xdb\x26\xb4\x6b\x77\xfc\x40" + "\x31\x15\x72\xa0\xf0\x73\xd9\x3b" + "\xd5\xdb\xfe\xfc\x8f\xa9\x44\xa2" + "\x09\x9f\xc6\x33\xe5\xe2\x88\xe8" + "\xf3\xf0\x1a\xf4\xce\x12\x0f\xd6" + "\xf7\x36\xe6\xa4\xf4\x7a\x10\x58" + "\xcc\x1f\x48\x49\x65\x47\x75\xe9" + "\x28\xe1\x65\x7b\xf2\xc4\xb5\x07" + "\xf2\xec\x76\xd8\x8f\x09\xf3\x16" + "\xa1\x51\x89\x3b\xeb\x96\x42\xac" + "\x65\xe0\x67\x63\x29\xdc\xb4\x7d" + "\xf2\x41\x51\x6a\xcb\xde\x3c\xfb" + "\x66\x8d\x13\xca\xe0\x59\x2a\x00" + "\xc9\x53\x4c\xe6\x9e\xe2\x73\xd5" + "\x67\x19\xb2\xbd\x9a\x63\xd7\x5c", + .len = 512, + .also_non_np = 1, + .np = 3, + .tap = { 512 - 20, 4, 16 }, + } +}; + +static const struct cipher_testvec speck64_tv_template[] = { + { /* Speck64/96 */ + .key = "\x00\x01\x02\x03\x08\x09\x0a\x0b" + "\x10\x11\x12\x13", + .klen = 12, + .ptext = "\x65\x61\x6e\x73\x20\x46\x61\x74", + .ctext = "\x6c\x94\x75\x41\xec\x52\x79\x9f", + .len = 8, + }, { /* Speck64/128 */ + .key = "\x00\x01\x02\x03\x08\x09\x0a\x0b" + "\x10\x11\x12\x13\x18\x19\x1a\x1b", + .klen = 16, + .ptext = "\x2d\x43\x75\x74\x74\x65\x72\x3b", + .ctext = "\x8b\x02\x4e\x45\x48\xa5\x6f\x8c", + .len = 8, + }, +}; + +/* + * Speck64-XTS test vectors, taken from the AES-XTS test vectors with the + * ciphertext recomputed with Speck64 as the cipher, and key lengths adjusted + */ +static const struct cipher_testvec speck64_xts_tv_template[] = { + { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 24, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\x84\xaf\x54\x07\x19\xd4\x7c\xa6" + "\xe4\xfe\xdf\xc4\x1f\x34\xc3\xc2" + "\x80\xf5\x72\xe7\xcd\xf0\x99\x22" + "\x35\xa7\x2f\x06\xef\xdc\x51\xaa", + .len = 32, + }, { + .key = "\x11\x11\x11\x11\x11\x11\x11\x11" + "\x11\x11\x11\x11\x11\x11\x11\x11" + "\x22\x22\x22\x22\x22\x22\x22\x22", + .klen = 24, + .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44", + .ctext = "\x12\x56\x73\xcd\x15\x87\xa8\x59" + "\xcf\x84\xae\xd9\x1c\x66\xd6\x9f" + "\xb3\x12\x69\x7e\x36\xeb\x52\xff" + "\x62\xdd\xba\x90\xb3\xe1\xee\x99", + .len = 32, + }, { + .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" + "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" + "\x22\x22\x22\x22\x22\x22\x22\x22", + .klen = 24, + .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44", + .ctext = "\x15\x1b\xe4\x2c\xa2\x5a\x2d\x2c" + "\x27\x36\xc0\xbf\x5d\xea\x36\x37" + "\x2d\x1a\x88\xbc\x66\xb5\xd0\x0b" + "\xa1\xbc\x19\xb2\x0f\x3b\x75\x34", + .len = 32, + }, { + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x31\x41\x59\x26\x53\x58\x97\x93", + .klen = 24, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ctext = "\xaf\xa1\x81\xa6\x32\xbb\x15\x8e" + "\xf8\x95\x2e\xd3\xe6\xee\x7e\x09" + "\x0c\x1a\xf5\x02\x97\x8b\xe3\xb3" + "\x11\xc7\x39\x96\xd0\x95\xf4\x56" + "\xf4\xdd\x03\x38\x01\x44\x2c\xcf" + "\x88\xae\x8e\x3c\xcd\xe7\xaa\x66" + "\xfe\x3d\xc6\xfb\x01\x23\x51\x43" + "\xd5\xd2\x13\x86\x94\x34\xe9\x62" + "\xf9\x89\xe3\xd1\x7b\xbe\xf8\xef" + "\x76\x35\x04\x3f\xdb\x23\x9d\x0b" + "\x85\x42\xb9\x02\xd6\xcc\xdb\x96" + "\xa7\x6b\x27\xb6\xd4\x45\x8f\x7d" + "\xae\xd2\x04\xd5\xda\xc1\x7e\x24" + "\x8c\x73\xbe\x48\x7e\xcf\x65\x28" + "\x29\xe5\xbe\x54\x30\xcb\x46\x95" + "\x4f\x2e\x8a\x36\xc8\x27\xc5\xbe" + "\xd0\x1a\xaf\xab\x26\xcd\x9e\x69" + "\xa1\x09\x95\x71\x26\xe9\xc4\xdf" + "\xe6\x31\xc3\x46\xda\xaf\x0b\x41" + "\x1f\xab\xb1\x8e\xd6\xfc\x0b\xb3" + "\x82\xc0\x37\x27\xfc\x91\xa7\x05" + "\xfb\xc5\xdc\x2b\x74\x96\x48\x43" + "\x5d\x9c\x19\x0f\x60\x63\x3a\x1f" + "\x6f\xf0\x03\xbe\x4d\xfd\xc8\x4a" + "\xc6\xa4\x81\x6d\xc3\x12\x2a\x5c" + "\x07\xff\xf3\x72\x74\x48\xb5\x40" + "\x50\xb5\xdd\x90\x43\x31\x18\x15" + "\x7b\xf2\xa6\xdb\x83\xc8\x4b\x4a" + "\x29\x93\x90\x8b\xda\x07\xf0\x35" + "\x6d\x90\x88\x09\x4e\x83\xf5\x5b" + "\x94\x12\xbb\x33\x27\x1d\x3f\x23" + "\x51\xa8\x7c\x07\xa2\xae\x77\xa6" + "\x50\xfd\xcc\xc0\x4f\x80\x7a\x9f" + "\x66\xdd\xcd\x75\x24\x8b\x33\xf7" + "\x20\xdb\x83\x9b\x4f\x11\x63\x6e" + "\xcf\x37\xef\xc9\x11\x01\x5c\x45" + "\x32\x99\x7c\x3c\x9e\x42\x89\xe3" + "\x70\x6d\x15\x9f\xb1\xe6\xb6\x05" + "\xfe\x0c\xb9\x49\x2d\x90\x6d\xcc" + "\x5d\x3f\xc1\xfe\x89\x0a\x2e\x2d" + "\xa0\xa8\x89\x3b\x73\x39\xa5\x94" + "\x4c\xa4\xa6\xbb\xa7\x14\x46\x89" + "\x10\xff\xaf\xef\xca\xdd\x4f\x80" + "\xb3\xdf\x3b\xab\xd4\xe5\x5a\xc7" + "\x33\xca\x00\x8b\x8b\x3f\xea\xec" + "\x68\x8a\xc2\x6d\xfd\xd4\x67\x0f" + "\x22\x31\xe1\x0e\xfe\x5a\x04\xd5" + "\x64\xa3\xf1\x1a\x76\x28\xcc\x35" + "\x36\xa7\x0a\x74\xf7\x1c\x44\x9b" + "\xc7\x1b\x53\x17\x02\xea\xd1\xad" + "\x13\x51\x73\xc0\xa0\xb2\x05\x32" + "\xa8\xa2\x37\x2e\xe1\x7a\x3a\x19" + "\x26\xb4\x6c\x62\x5d\xb3\x1a\x1d" + "\x59\xda\xee\x1a\x22\x18\xda\x0d" + "\x88\x0f\x55\x8b\x72\x62\xfd\xc1" + "\x69\x13\xcd\x0d\x5f\xc1\x09\x52" + "\xee\xd6\xe3\x84\x4d\xee\xf6\x88" + "\xaf\x83\xdc\x76\xf4\xc0\x93\x3f" + "\x4a\x75\x2f\xb0\x0b\x3e\xc4\x54" + "\x7d\x69\x8d\x00\x62\x77\x0d\x14" + "\xbe\x7c\xa6\x7d\xc5\x24\x4f\xf3" + "\x50\xf7\x5f\xf4\xc2\xca\x41\x97" + "\x37\xbe\x75\x74\xcd\xf0\x75\x6e" + "\x25\x23\x94\xbd\xda\x8d\xb0\xd4", + .len = 512, + }, { + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x62\x49\x77\x57\x24\x70\x93\x69" + "\x99\x59\x57\x49\x66\x96\x76\x27", + .klen = 32, + .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ctext = "\x55\xed\x71\xd3\x02\x8e\x15\x3b" + "\xc6\x71\x29\x2d\x3e\x89\x9f\x59" + "\x68\x6a\xcc\x8a\x56\x97\xf3\x95" + "\x4e\x51\x08\xda\x2a\xf8\x6f\x3c" + "\x78\x16\xea\x80\xdb\x33\x75\x94" + "\xf9\x29\xc4\x2b\x76\x75\x97\xc7" + "\xf2\x98\x2c\xf9\xff\xc8\xd5\x2b" + "\x18\xf1\xaf\xcf\x7c\xc5\x0b\xee" + "\xad\x3c\x76\x7c\xe6\x27\xa2\x2a" + "\xe4\x66\xe1\xab\xa2\x39\xfc\x7c" + "\xf5\xec\x32\x74\xa3\xb8\x03\x88" + "\x52\xfc\x2e\x56\x3f\xa1\xf0\x9f" + "\x84\x5e\x46\xed\x20\x89\xb6\x44" + "\x8d\xd0\xed\x54\x47\x16\xbe\x95" + "\x8a\xb3\x6b\x72\xc4\x32\x52\x13" + "\x1b\xb0\x82\xbe\xac\xf9\x70\xa6" + "\x44\x18\xdd\x8c\x6e\xca\x6e\x45" + "\x8f\x1e\x10\x07\x57\x25\x98\x7b" + "\x17\x8c\x78\xdd\x80\xa7\xd9\xd8" + "\x63\xaf\xb9\x67\x57\xfd\xbc\xdb" + "\x44\xe9\xc5\x65\xd1\xc7\x3b\xff" + "\x20\xa0\x80\x1a\xc3\x9a\xad\x5e" + "\x5d\x3b\xd3\x07\xd9\xf5\xfd\x3d" + "\x4a\x8b\xa8\xd2\x6e\x7a\x51\x65" + "\x6c\x8e\x95\xe0\x45\xc9\x5f\x4a" + "\x09\x3c\x3d\x71\x7f\x0c\x84\x2a" + "\xc8\x48\x52\x1a\xc2\xd5\xd6\x78" + "\x92\x1e\xa0\x90\x2e\xea\xf0\xf3" + "\xdc\x0f\xb1\xaf\x0d\x9b\x06\x2e" + "\x35\x10\x30\x82\x0d\xe7\xc5\x9b" + "\xde\x44\x18\xbd\x9f\xd1\x45\xa9" + "\x7b\x7a\x4a\xad\x35\x65\x27\xca" + "\xb2\xc3\xd4\x9b\x71\x86\x70\xee" + "\xf1\x89\x3b\x85\x4b\x5b\xaa\xaf" + "\xfc\x42\xc8\x31\x59\xbe\x16\x60" + "\x4f\xf9\xfa\x12\xea\xd0\xa7\x14" + "\xf0\x7a\xf3\xd5\x8d\xbd\x81\xef" + "\x52\x7f\x29\x51\x94\x20\x67\x3c" + "\xd1\xaf\x77\x9f\x22\x5a\x4e\x63" + "\xe7\xff\x73\x25\xd1\xdd\x96\x8a" + "\x98\x52\x6d\xf3\xac\x3e\xf2\x18" + "\x6d\xf6\x0a\x29\xa6\x34\x3d\xed" + "\xe3\x27\x0d\x9d\x0a\x02\x44\x7e" + "\x5a\x7e\x67\x0f\x0a\x9e\xd6\xad" + "\x91\xe6\x4d\x81\x8c\x5c\x59\xaa" + "\xfb\xeb\x56\x53\xd2\x7d\x4c\x81" + "\x65\x53\x0f\x41\x11\xbd\x98\x99" + "\xf9\xc6\xfa\x51\x2e\xa3\xdd\x8d" + "\x84\x98\xf9\x34\xed\x33\x2a\x1f" + "\x82\xed\xc1\x73\x98\xd3\x02\xdc" + "\xe6\xc2\x33\x1d\xa2\xb4\xca\x76" + "\x63\x51\x34\x9d\x96\x12\xae\xce" + "\x83\xc9\x76\x5e\xa4\x1b\x53\x37" + "\x17\xd5\xc0\x80\x1d\x62\xf8\x3d" + "\x54\x27\x74\xbb\x10\x86\x57\x46" + "\x68\xe1\xed\x14\xe7\x9d\xfc\x84" + "\x47\xbc\xc2\xf8\x19\x4b\x99\xcf" + "\x7a\xe9\xc4\xb8\x8c\x82\x72\x4d" + "\x7b\x4f\x38\x55\x36\x71\x64\xc1" + "\xfc\x5c\x75\x52\x33\x02\x18\xf8" + "\x17\xe1\x2b\xc2\x43\x39\xbd\x76" + "\x9b\x63\x76\x32\x2f\x19\x72\x10" + "\x9f\x21\x0c\xf1\x66\x50\x7f\xa5" + "\x0d\x1f\x46\xe0\xba\xd3\x2f\x3c", + .len = 512, .also_non_np = 1, .np = 3, - .tap = { 1008 - 20, 4, 16 }, - }, + .tap = { 512 - 20, 4, 16 }, + } }; -static const struct cipher_testvec camellia_cbc_enc_tv_template[] = { +/* Cast6 test vectors from RFC 2612 */ +static const struct cipher_testvec cast6_tv_template[] = { { - .key = "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" - "\x51\x2e\x03\xd5\x34\x12\x00\x06", - .klen = 16, - .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" - "\xb4\x22\xda\x80\x2c\x9f\xac\x41", - .input = "Single block msg", - .ilen = 16, - .result = "\xea\x32\x12\x76\x3b\x50\x10\xe7" - "\x18\xf6\xfd\x5d\xf6\x8f\x13\x51", - .rlen = 16, - }, { - .key = "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" - "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", - .klen = 16, - .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" - "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .ilen = 32, - .result = "\xa5\xdf\x6e\x50\xda\x70\x6c\x01" - "\x4a\xab\xf3\xf2\xd6\xfc\x6c\xfd" - "\x19\xb4\x3e\x57\x1c\x02\x5e\xa0" - "\x15\x78\xe0\x5e\xf2\xcb\x87\x16", - .rlen = 32, - }, { /* Generated with Crypto++ */ + .key = "\x23\x42\xbb\x9e\xfa\x38\x54\x2c" + "\x0a\xf7\x56\x47\xf2\x9f\x61\x5d", + .klen = 16, + .ptext = zeroed_string, + .ctext = "\xc8\x42\xa0\x89\x72\xb4\x3d\x20" + "\x83\x6c\x91\xd1\xb7\x53\x0f\x6b", + .len = 16, + }, { + .key = "\x23\x42\xbb\x9e\xfa\x38\x54\x2c" + "\xbe\xd0\xac\x83\x94\x0a\xc2\x98" + "\xba\xc7\x7a\x77\x17\x94\x28\x63", + .klen = 24, + .ptext = zeroed_string, + .ctext = "\x1b\x38\x6c\x02\x10\xdc\xad\xcb" + "\xdd\x0e\x41\xaa\x08\xa7\xa7\xe8", + .len = 16, + }, { + .key = "\x23\x42\xbb\x9e\xfa\x38\x54\x2c" + "\xbe\xd0\xac\x83\x94\x0a\xc2\x98" + "\x8d\x7c\x47\xce\x26\x49\x08\x46" + "\x1c\xc1\xb5\x13\x7a\xe6\xb6\x04", + .klen = 32, + .ptext = zeroed_string, + .ctext = "\x4f\x6a\x20\x38\x28\x68\x97\xb9" + "\xc9\x87\x01\x36\x55\x33\x17\xfa", + .len = 16, + }, { /* Generated from TF test vectors */ .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" @@ -30701,7 +10916,7 @@ static const struct cipher_testvec camellia_cbc_enc_tv_template[] = { .klen = 32, .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -30711,696 +10926,16081 @@ static const struct cipher_testvec camellia_cbc_enc_tv_template[] = { "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59\xF0\x64\xFB\x92\x06" - "\x9D\x34\xCB\x3F\xD6\x6D\x04\x78" - "\x0F\xA6\x1A\xB1\x48\xDF\x53\xEA" - "\x81\x18\x8C\x23\xBA\x2E\xC5\x5C" - "\xF3\x67\xFE\x95\x09\xA0\x37\xCE" - "\x42\xD9\x70\x07\x7B\x12\xA9\x1D" - "\xB4\x4B\xE2\x56\xED\x84\x1B\x8F" - "\x26\xBD\x31\xC8\x5F\xF6\x6A\x01" - "\x98\x0C\xA3\x3A\xD1\x45\xDC\x73" - "\x0A\x7E\x15\xAC\x20\xB7\x4E\xE5" - "\x59\xF0\x87\x1E\x92\x29\xC0\x34" - "\xCB\x62\xF9\x6D\x04\x9B\x0F\xA6" - "\x3D\xD4\x48\xDF\x76\x0D\x81\x18" - "\xAF\x23\xBA\x51\xE8\x5C\xF3\x8A" - "\x21\x95\x2C\xC3\x37\xCE\x65\xFC" - "\x70\x07\x9E\x12\xA9\x40\xD7\x4B" - "\xE2\x79\x10\x84\x1B\xB2\x26\xBD" - "\x54\xEB\x5F\xF6\x8D\x01\x98\x2F" - "\xC6\x3A\xD1\x68\xFF\x73\x0A\xA1" - "\x15\xAC\x43\xDA\x4E\xE5\x7C\x13" - "\x87\x1E\xB5\x29\xC0\x57\xEE\x62" - "\xF9\x90\x04\x9B\x32\xC9\x3D\xD4" - "\x6B\x02\x76\x0D\xA4\x18\xAF\x46" - "\xDD\x51\xE8\x7F\x16\x8A\x21\xB8" - "\x2C\xC3\x5A\xF1\x65\xFC\x93\x07" - "\x9E\x35\xCC\x40\xD7\x6E\x05\x79" - "\x10\xA7\x1B\xB2\x49\xE0\x54\xEB" - "\x82\x19\x8D\x24\xBB\x2F\xC6\x5D" - "\xF4\x68\xFF\x96\x0A\xA1\x38\xCF" - "\x43\xDA\x71\x08\x7C\x13\xAA\x1E" - "\xB5\x4C\xE3\x57\xEE\x85\x1C\x90" - "\x27\xBE\x32\xC9\x60\xF7\x6B\x02" - "\x99\x0D\xA4\x3B\xD2\x46\xDD\x74" - "\x0B\x7F\x16\xAD\x21\xB8\x4F\xE6" - "\x5A\xF1\x88\x1F\x93\x2A\xC1\x35" - "\xCC\x63\xFA\x6E\x05\x9C\x10\xA7" - "\x3E\xD5\x49\xE0\x77\x0E\x82\x19" - "\xB0\x24\xBB\x52\xE9\x5D\xF4\x8B" - "\x22\x96\x2D\xC4\x38\xCF\x66\xFD" - "\x71\x08\x9F\x13\xAA\x41\xD8\x4C" - "\xE3\x7A\x11\x85\x1C\xB3\x27\xBE" - "\x55\xEC\x60\xF7\x8E\x02\x99\x30" - "\xC7\x3B\xD2\x69\x00\x74\x0B\xA2" - "\x16\xAD\x44\xDB\x4F\xE6\x7D\x14" - "\x88\x1F\xB6\x2A\xC1\x58\xEF\x63" - "\xFA\x91\x05\x9C\x33\xCA\x3E\xD5" - "\x6C\x03\x77\x0E\xA5\x19\xB0\x47" - "\xDE\x52\xE9\x80\x17\x8B\x22\xB9" - "\x2D\xC4\x5B\xF2\x66\xFD\x94\x08" - "\x9F\x36\xCD\x41\xD8\x6F\x06\x7A" - "\x11\xA8\x1C\xB3\x4A\xE1\x55\xEC" - "\x83\x1A\x8E\x25\xBC\x30\xC7\x5E" - "\xF5\x69\x00\x97\x0B\xA2\x39\xD0" - "\x44\xDB\x72\x09\x7D\x14\xAB\x1F" - "\xB6\x4D\xE4\x58\xEF\x86\x1D\x91" - "\x28\xBF\x33\xCA\x61\xF8\x6C\x03" - "\x9A\x0E\xA5\x3C\xD3\x47\xDE\x75" - "\x0C\x80\x17\xAE\x22\xB9\x50\xE7" - "\x5B\xF2\x89\x20\x94\x2B\xC2\x36" - "\xCD\x64\xFB\x6F\x06\x9D\x11\xA8" - "\x3F\xD6\x4A\xE1\x78\x0F\x83\x1A" - "\xB1\x25\xBC\x53\xEA\x5E\xF5\x8C" - "\x00\x97\x2E\xC5\x39\xD0\x67\xFE" - "\x72\x09\xA0\x14\xAB\x42\xD9\x4D", - .ilen = 1008, - .result = "\xCD\x3E\x2A\x3B\x3E\x94\xC5\x77" - "\xBA\xBB\x5B\xB1\xDE\x7B\xA4\x40" - "\x88\x39\xE3\xFD\x94\x4B\x25\x58" - "\xE1\x4B\xC4\x18\x7A\xFD\x17\x2B" - "\xB9\xF9\xC2\x27\x6A\xB6\x31\x27" - "\xA6\xAD\xEF\xE5\x5D\xE4\x02\x01" - "\x56\x2E\x10\xC2\x2C\xFF\xC6\x83" - "\xB5\xDC\x4F\x63\xAD\x0E\x63\x5E" - "\x56\xC8\x18\x3D\x79\x86\x97\xEF" - "\x57\x0E\x63\xA1\xC1\x41\x48\xB8" - "\x98\xB7\x51\x6D\x18\xF6\x19\x82" - "\x37\x49\x88\xA4\xEF\x91\x21\x47" - "\x03\x28\xEA\x42\xF4\xFB\x7A\x58" - "\x28\x90\x77\x46\xD8\xD2\x35\x16" - "\x44\xA9\x9E\x49\x52\x2A\xE4\x16" - "\x5D\xF7\x65\xEB\x0F\xC9\x29\xE6" - "\xCF\x76\x91\x89\x8A\x94\x39\xFA" - "\x6B\x5F\x63\x53\x74\x43\x91\xF5" - "\x3F\xBC\x88\x53\xB2\x1A\x02\x3F" - "\x9D\x32\x84\xEB\x56\x28\xD6\x06" - "\xD5\xB2\x20\xA9\xFC\xC3\x76\x62" - "\x32\xCC\x86\xC8\x36\x67\x5E\x7E" - "\xA4\xAA\x15\x63\x6B\xA9\x86\xAF" - "\x1A\x52\x82\x36\x5F\xF4\x3F\x7A" - "\x9B\x78\x62\x3B\x02\x28\x60\xB3" - "\xBA\x82\xB1\xDD\xC9\x60\x8F\x47" - "\xF1\x6B\xFE\xE5\x39\x34\xA0\x28" - "\xA4\xB3\xC9\x7E\xED\x28\x8D\x70" - "\xB2\x1D\xFD\xC6\x00\xCF\x1A\x94" - "\x28\xF8\xC1\x34\xB7\x58\xA5\x6C" - "\x1A\x9D\xE4\xE4\xF6\xB9\xB4\xB0" - "\x5D\x51\x54\x9A\x53\xA0\xF9\x32" - "\xBD\x31\x54\x14\x7B\x33\xEE\x17" - "\xD3\xC7\x1F\x48\xBF\x0B\x22\xA2" - "\x7D\x0C\xDF\xD0\x2E\x98\xFA\xD2" - "\xFA\xCF\x24\x1D\x99\x9B\xD0\x7E" - "\xF4\x4F\x88\xFF\x45\x99\x4A\xF4" - "\xF2\x0A\x5B\x3B\x21\xAB\x92\xAE" - "\x40\x78\x91\x95\xC4\x2F\xA3\xE8" - "\x18\xC7\x07\xA6\xC8\xC0\x66\x33" - "\x35\xC0\xB4\xA0\xF8\xEE\x1E\xF3" - "\x40\xF5\x40\x54\xF1\x84\x8C\xEA" - "\x27\x38\x1F\xF8\x77\xC7\xDF\xD8" - "\x1D\xE2\xD9\x59\x40\x4F\x59\xD4" - "\xF8\x17\x99\x8D\x58\x2D\x72\x44" - "\x9D\x1D\x91\x64\xD6\x3F\x0A\x82" - "\xC7\x57\x3D\xEF\xD3\x41\xFA\xA7" - "\x68\xA3\xB8\xA5\x93\x74\x2E\x85" - "\x4C\x9D\x69\x59\xCE\x15\xAE\xBF" - "\x9C\x8F\x14\x64\x5D\x7F\xCF\x0B" - "\xCE\x43\x5D\x28\xC0\x2F\xFB\x18" - "\x79\x9A\xFC\x43\x16\x7C\x6B\x7B" - "\x38\xB8\x48\x36\x66\x4E\x20\x43" - "\xBA\x76\x13\x9A\xC3\xF2\xEB\x52" - "\xD7\xDC\xB2\x67\x63\x14\x25\xCD" - "\xB1\x13\x4B\xDE\x8C\x59\x21\x84" - "\x81\x8D\x97\x23\x45\x33\x7C\xF3" - "\xC5\xBC\x79\x95\xAA\x84\x68\x31" - "\x2D\x1A\x68\xFE\xEC\x92\x94\xDA" - "\x94\x2A\x6F\xD6\xFE\xE5\x76\x97" - "\xF4\x6E\xEE\xCB\x2B\x95\x4E\x36" - "\x5F\x74\x8C\x86\x5B\x71\xD0\x20" - "\x78\x1A\x7F\x18\x8C\xD9\xCD\xF5" - "\x21\x41\x56\x72\x13\xE1\x86\x07" - "\x07\x26\xF3\x4F\x7B\xEA\xB5\x18" - "\xFE\x94\x2D\x9F\xE0\x72\x18\x65" - "\xB2\xA5\x63\x48\xB4\x13\x22\xF7" - "\x25\xF1\x80\xA8\x7F\x54\x86\x7B" - "\x39\xAE\x95\x0C\x09\x32\x22\x2D" - "\x4D\x73\x39\x0C\x09\x2C\x7C\x10" - "\xD0\x4B\x53\xF6\x90\xC5\x99\x2F" - "\x15\xE1\x7F\xC6\xC5\x7A\x52\x14" - "\x65\xEE\x93\x54\xD0\x66\x15\x3C" - "\x4C\x68\xFD\x64\x0F\xF9\x10\x39" - "\x46\x7A\xDD\x97\x20\xEE\xC7\xD2" - "\x98\x4A\xB6\xE6\xF5\xA8\x1F\x4F" - "\xDB\xAB\x6D\xD5\x9B\x34\x16\x97" - "\x2F\x64\xE5\x37\xEF\x0E\xA1\xE9" - "\xBE\x31\x31\x96\x8B\x40\x18\x75" - "\x11\x75\x14\x32\xA5\x2D\x1B\x6B" - "\xDB\x59\xEB\xFA\x3D\x8E\x7C\xC4" - "\xDE\x68\xC8\x9F\xC9\x99\xE3\xC6" - "\x71\xB0\x12\x57\x89\x0D\xC0\x2B" - "\x9F\x12\x6A\x04\x67\xF1\x95\x31" - "\x59\xFD\x84\x95\x2C\x9C\x5B\xEC" - "\x09\xB0\x43\x96\x4A\x64\x80\x40" - "\xB9\x72\x19\xDD\x70\x42\xFA\xB1" - "\x4A\x2C\x0C\x0A\x60\x6E\xE3\x7C" - "\x37\x5A\xBE\xA4\x62\xCF\x29\xAB" - "\x7F\x4D\xA6\xB3\xE2\xB6\x64\xC6" - "\x33\x0B\xF3\xD5\x01\x38\x74\xA4" - "\x67\x1E\x75\x68\xC3\xAD\x76\xE9" - "\xE9\xBC\xF0\xEB\xD8\xFD\x31\x8A" - "\x5F\xC9\x18\x94\x4B\x86\x66\xFC" - "\xBD\x0B\x3D\xB3\x9F\xFA\x1F\xD9" - "\x78\xC4\xE3\x24\x1C\x67\xA2\xF8" - "\x43\xBC\x76\x75\xBF\x6C\x05\xB3" - "\x32\xE8\x7C\x80\xDB\xC7\xB6\x61" - "\x1A\x3E\x2B\xA7\x25\xED\x8F\xA0" - "\x00\x4B\xF8\x90\xCA\xD8\xFB\x12" - "\xAC\x1F\x18\xE9\xD2\x5E\xA2\x8E" - "\xE4\x84\x6B\x9D\xEB\x1E\x6B\xA3" - "\x7B\xDC\xCE\x15\x97\x27\xB2\x65" - "\xBC\x0E\x47\xAB\x55\x13\x53\xAB" - "\x0E\x34\x55\x02\x5F\x27\xC5\x89" - "\xDF\xC5\x70\xC4\xDD\x76\x82\xEE" - "\x68\xA6\x09\xB0\xE5\x5E\xF1\x0C" - "\xE3\xF3\x09\x9B\xFE\x65\x4B\xB8" - "\x30\xEC\xD5\x7C\x6A\xEC\x1D\xD2" - "\x93\xB7\xA1\x1A\x02\xD4\xC0\xD6" - "\x8D\x4D\x83\x9A\xED\x29\x4E\x14" - "\x86\xD5\x3C\x1A\xD5\xB9\x0A\x6A" - "\x72\x22\xD5\x92\x38\xF1\xA1\x86" - "\xB2\x41\x51\xCA\x4E\xAB\x8F\xD3" - "\x80\x56\xC3\xD7\x65\xE1\xB3\x86" - "\xCB\xCE\x98\xA1\xD4\x59\x1C\x06" - "\x01\xED\xF8\x29\x91\x19\x5C\x9A" - "\xEE\x28\x1B\x48\xD7\x32\xEF\x9F" - "\x6C\x2B\x66\x4E\x78\xD5\x8B\x72" - "\x80\xE7\x29\xDC\x23\x55\x98\x54" - "\xB1\xFF\x3E\x95\x56\xA8\x78\x78" - "\xEF\xC4\xA5\x11\x2D\x2B\xD8\x93" - "\x30\x6E\x7E\x51\xBB\x42\x5F\x03" - "\x43\x94\x23\x7E\xEE\xF0\xA5\x79" - "\x55\x01\xD4\x58\xB2\xF2\x85\x49" - "\x70\xC5\xB9\x0B\x3B\x7A\x6E\x6C", - .rlen = 1008, + "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" + "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" + "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" + "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" + "\x29\xC0\x57\xEE\x62\xF9\x90\x04" + "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" + "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" + "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" + "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" + "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" + "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" + "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" + "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" + "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" + "\x57\xEE\x85\x1C\x90\x27\xBE\x32" + "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" + "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" + "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" + "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" + "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" + "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" + "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" + "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" + "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" + "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" + "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" + "\x69\x00\x74\x0B\xA2\x16\xAD\x44" + "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" + "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" + "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" + "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" + "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" + "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" + "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" + "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" + "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" + "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" + "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" + "\x58\xEF\x86\x1D\x91\x28\xBF\x33" + "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" + "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" + "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" + "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" + "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" + "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" + "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" + "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" + "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" + "\x86\x1D\xB4\x28\xBF\x56\xED\x61" + "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" + "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\xC3\x70\x22\x32\xF5\x80\xCB\x54" + "\xFC\x30\xE0\xF6\xEB\x39\x57\xA6" + "\xB6\xB9\xC5\xA4\x91\x55\x14\x97" + "\xC1\x20\xFF\x6C\x5C\xF0\x67\xEA" + "\x2F\xED\xD8\xC9\xFB\x38\x3F\xFE" + "\x93\xBE\xDC\x00\xD3\x7F\xAD\x4C" + "\x5A\x08\x92\xD1\x47\x0C\xFA\x6C" + "\xD0\x6A\x99\x10\x72\xF8\x47\x62" + "\x81\x42\xF8\xD8\xF5\xBB\x94\x08" + "\xAA\x97\xA2\x8B\x69\xB3\xD2\x7E" + "\xBC\xB5\x00\x0C\xE5\x44\x4B\x58" + "\xE8\x63\xDC\xB3\xC4\xE5\x23\x12" + "\x5A\x72\x85\x47\x8B\xEC\x9F\x26" + "\x84\xB6\xED\x10\x33\x63\x9B\x5F" + "\x4D\x53\xEE\x94\x45\x8B\x60\x58" + "\x86\x20\xF9\x1E\x82\x08\x3E\x58" + "\x60\x1B\x34\x19\x02\xBE\x4E\x09" + "\xBB\x7C\x15\xCC\x60\x27\x55\x7A" + "\x12\xB8\xD8\x08\x89\x3C\xA6\xF3" + "\xF1\xDD\xA7\x07\xA3\x12\x85\x28" + "\xE9\x57\xAC\x80\x0C\x5C\x0F\x3A" + "\x5D\xC2\x91\xC7\x90\xE4\x8C\x43" + "\x92\xE4\x7C\x26\x69\x4D\x83\x68" + "\x14\x96\x42\x47\xBD\xA9\xE4\x8A" + "\x33\x19\xEB\x54\x8E\x0D\x4B\x6E" + "\x91\x51\xB5\x36\x08\xDE\x1C\x06" + "\x03\xBD\xDE\x81\x26\xF7\x99\xC2" + "\xBA\xF7\x6D\x87\x0D\xE4\xA6\xCF" + "\xC1\xF5\x27\x05\xB8\x02\x57\x72" + "\xE6\x42\x13\x0B\xC6\x47\x05\x74" + "\x24\x15\xF7\x0D\xC2\x23\x9D\xB9" + "\x3C\x77\x18\x93\xBA\xB4\xFC\x8C" + "\x98\x82\x67\x67\xB4\xD7\xD3\x43" + "\x23\x08\x02\xB7\x9B\x99\x05\xFB" + "\xD3\xB5\x00\x0A\xA9\x9D\x66\xD6" + "\x2E\x49\x58\xD0\xA8\x57\x29\x7F" + "\x0A\x0E\x7D\xFC\x92\x83\xCC\x67" + "\xA2\xB1\x70\x3A\x8F\x87\x4A\x8D" + "\x17\xE2\x58\x2B\x88\x0D\x68\x62" + "\xBF\x35\xD1\x6F\xC0\xF0\x18\x62" + "\xB2\xC7\x2D\x58\xC7\x16\xDE\x08" + "\xEB\x84\x1D\x25\xA7\x38\x94\x06" + "\x93\x9D\xF8\xFE\x88\x71\xE7\x84" + "\x2C\xA0\x38\xA3\x1D\x48\xCF\x29" + "\x0B\xBC\xD8\x50\x99\x1A\x26\xFB" + "\x8E\x75\x3D\x73\xEB\x6A\xED\x29" + "\xE0\x8E\xED\xFC\xFE\x6F\xF6\xBA" + "\x41\xE2\x10\x4C\x01\x8B\x69\x2B" + "\x25\x3F\x4D\x70\x7B\x92\xD6\x3B" + "\xAC\xF9\x77\x18\xD9\x6A\x30\xA6" + "\x2E\xFA\x30\xFF\xC8\xD5\x1D\x06" + "\x59\x28\x1D\x86\x43\x04\x5D\x3B" + "\x99\x4C\x04\x5A\x21\x17\x8B\x76" + "\x8F\x72\xCB\xA1\x9C\x29\x4C\xC3" + "\x65\xA2\x58\x2A\xC5\x66\x24\xBF" + "\xBA\xE6\x0C\xDD\x34\x24\x74\xC8" + "\x84\x0A\x66\x2C\xBE\x8F\x32\xA9" + "\xE7\xE4\xA1\xD7\xDA\xAB\x23\x1E" + "\xEB\xEE\x6C\x94\x6F\x9C\x2E\xD1" + "\x49\x2C\xF3\xD4\x90\xCC\x93\x4C" + "\x84\x52\x6D\x68\xDE\xC6\x64\xB2" + "\x11\x74\x93\x57\xB4\x7E\xC6\x00", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec cast6_cbc_tv_template[] = { + { /* Generated from TF test vectors */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .klen = 32, + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" + "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" + "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" + "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" + "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" + "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" + "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" + "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" + "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" + "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" + "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" + "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" + "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" + "\x29\xC0\x57\xEE\x62\xF9\x90\x04" + "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" + "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" + "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" + "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" + "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" + "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" + "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" + "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" + "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" + "\x57\xEE\x85\x1C\x90\x27\xBE\x32" + "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" + "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" + "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" + "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" + "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" + "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" + "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" + "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" + "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" + "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" + "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" + "\x69\x00\x74\x0B\xA2\x16\xAD\x44" + "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" + "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" + "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" + "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" + "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" + "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" + "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" + "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" + "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" + "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" + "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" + "\x58\xEF\x86\x1D\x91\x28\xBF\x33" + "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" + "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" + "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" + "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" + "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" + "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" + "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" + "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" + "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" + "\x86\x1D\xB4\x28\xBF\x56\xED\x61" + "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" + "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\xDF\x77\x68\x96\xC7\xBA\xF8\xE2" + "\x0E\x24\x99\x1A\xAA\xF3\xC6\x9F" + "\xA0\x73\xB3\x70\xC3\x68\x64\x70" + "\xAD\x33\x02\xFB\x88\x74\xAA\x78" + "\xC7\x47\x1A\x18\x61\x2D\xAC\x9F" + "\x7E\x6F\xDF\x05\x13\x76\xA6\x72" + "\xB7\x13\x09\x0F\x7D\x38\xDF\x25" + "\x4E\xFD\x50\x45\xFA\x35\x6A\xC0" + "\x57\x95\xE1\x21\x26\x10\x9A\x21" + "\xA1\x8A\x51\x05\xD1\xB1\x78\x35" + "\x98\xF5\xAE\xC0\xC1\x8B\x94\xFF" + "\xD0\x69\x3F\x42\xC2\x01\xA7\x9B" + "\x23\x16\x47\x72\x81\x13\x3A\x72" + "\xEC\xD9\x40\x88\x00\x9C\xB0\xA8" + "\x9C\xAC\xCE\x11\x73\x7B\x63\x3E" + "\xA3\x63\x98\x7D\x35\xE4\xD9\x83" + "\xE2\xD0\x52\x87\x0C\x1F\xB0\xB3" + "\x41\x1A\x93\x8D\x76\x31\x9F\xF2" + "\xFE\x09\xA3\x8F\x22\x6A\x3B\xB9" + "\x6C\x9E\xE4\xA1\xA0\xC4\xE7\xA1" + "\x21\x9C\x1A\xCA\x65\xDE\x44\x03" + "\x99\xF2\xD2\x39\xE3\x3F\x0F\x37" + "\x53\x50\x23\xA4\x81\x6E\xDA\xFB" + "\xF8\x7B\x01\xD7\xB2\x32\x9C\xB8" + "\xB1\x0E\x99\x17\xB5\x38\xF9\xD7" + "\x86\x2D\x6E\x94\x5C\x99\x9D\xB3" + "\xD3\x63\x4B\x2A\x7D\x44\x6A\xB2" + "\xC1\x03\xE6\x5A\x37\xD8\x64\x18" + "\xAA\x32\xCE\x29\xED\xC0\xA2\xCB" + "\x8D\xAF\xCD\xBE\x8F\xB6\xEC\xB4" + "\x89\x05\x81\x6E\x71\x4F\xC3\x28" + "\x10\xC1\x62\xC4\x41\xE9\xD2\x39" + "\xF3\x22\x39\x12\x2C\xC2\x95\x2D" + "\xBF\x93\x58\x4B\x04\xD1\x8D\x57" + "\xAE\xEB\x60\x03\x56\x35\xAD\x5A" + "\xE9\xC3\xFF\x4E\x31\xE1\x37\xF8" + "\x7D\xEE\x65\x8A\xB6\x88\x1A\x3E" + "\x07\x09\x82\xBA\xF0\x80\x8A\xD0" + "\xA0\x3F\x6A\xE9\x24\x87\x19\x65" + "\x73\x3F\x12\x91\x47\x54\xBA\x39" + "\x30\x5B\x1E\xE5\xC2\xF9\x3F\xEF" + "\xD6\x75\xF9\xB8\x7C\x8B\x05\x76" + "\xEE\xB7\x08\x25\x4B\xB6\x7B\x47" + "\x72\xC0\x4C\xD4\xDA\xE0\x75\xF1" + "\x7C\xE8\x94\x9E\x16\x6E\xB8\x12" + "\xA1\xC1\x6E\x3B\x1C\x59\x41\x2D" + "\x23\xFA\x7D\x77\xB8\x46\x75\xFE" + "\x4F\x10\xD3\x09\x60\xA1\x36\x96" + "\x5B\xC2\xDC\x6E\x84\x7D\x9B\x14" + "\x80\x21\x83\x58\x3C\x76\xFD\x28" + "\x1D\xF9\x93\x13\xD7\x0E\x62\x14" + "\x5A\xC5\x4E\x08\xA5\x56\xA4\x3C" + "\x68\x93\x44\x70\xDF\xCF\x4A\x51" + "\x0B\x81\x29\x41\xE5\x62\x4D\x36" + "\xB3\xEA\x94\xA6\xB9\xDD\x3F\x09" + "\x62\x34\xA0\x6A\x7E\x7D\xF5\xF6" + "\x01\x91\xB4\x27\xDA\x59\xD6\x17" + "\x56\x4D\x82\x62\x37\xA3\x48\x01" + "\x99\x91\x77\xB2\x08\x6B\x2C\x37" + "\xC5\x5C\xAD\xB6\x07\xB6\x84\xF3" + "\x4D\x59\x7D\xC5\x28\x69\xFA\x92" + "\x22\x46\x89\x2D\x0F\x2B\x08\x24", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec cast6_ctr_tv_template[] = { + { /* Generated from TF test vectors */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .klen = 32, + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" + "\x3A", + .ctext = "\x26\x0A\xF1\xE2\x3F\x8A\xEF\xA3" + "\x53\x9A\x5E\x1B\x2A\x1A\xC6\x0A" + "\x57", + .len = 17, + }, { /* Generated from TF test vectors */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .klen = 32, + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" + "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" + "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" + "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" + "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" + "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" + "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" + "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" + "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" + "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" + "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" + "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" + "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" + "\x29\xC0\x57\xEE\x62\xF9\x90\x04" + "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" + "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" + "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" + "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" + "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" + "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" + "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" + "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" + "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" + "\x57\xEE\x85\x1C\x90\x27\xBE\x32" + "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" + "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" + "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" + "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" + "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" + "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" + "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" + "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" + "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" + "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" + "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" + "\x69\x00\x74\x0B\xA2\x16\xAD\x44" + "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" + "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" + "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" + "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" + "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" + "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" + "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" + "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" + "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" + "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" + "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" + "\x58\xEF\x86\x1D\x91\x28\xBF\x33" + "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" + "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" + "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" + "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" + "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" + "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" + "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" + "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" + "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" + "\x86\x1D\xB4\x28\xBF\x56\xED\x61" + "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" + "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\x26\x0A\xF1\xE2\x3F\x8A\xEF\xA3" + "\x53\x9A\x5E\x1B\x2A\x1A\xC6\x0A" + "\x57\xA3\xEF\x47\x2A\xE8\x88\xA7" + "\x3C\xD0\xEC\xB9\x94\x50\x7D\x56" + "\xBC\xE1\xC1\xF5\xE1\xEE\x12\xF8" + "\x4F\x03\x82\x3A\x93\x6B\x4C\xD3" + "\xE3\xF3\xFA\xC2\x23\x55\x98\x20" + "\x49\x76\x9B\x6B\xC1\x23\xBF\xE5" + "\xD4\xC4\x2F\x61\xE1\x67\x2A\x30" + "\x6F\x29\xCA\x54\xF8\x1B\xA6\x7D" + "\x66\x45\xEE\xC8\x19\xBE\x50\xF0" + "\x5F\x65\xF8\x1E\x4D\x07\x87\xD9" + "\xD3\xD9\x1B\x09\x89\xFD\x42\xC5" + "\xDB\xEB\x86\xF1\x67\x04\x0F\x5C" + "\x81\xDF\x82\x12\xC7\x4C\x1B\x07" + "\xDE\xE6\xFA\x29\x86\xD1\xB0\xBA" + "\x3D\x6A\x69\x76\xEC\x0F\xB4\xE6" + "\xCD\xA7\xF8\xA8\xB8\xE0\x33\xF5" + "\x49\x61\x22\x52\x64\x8C\x46\x41" + "\x1F\x48\x5F\x4F\xA2\x89\x36\x17" + "\x20\xF8\x2F\x8F\x4B\xFA\xF2\xC0" + "\x1E\x18\xA2\xF8\xB7\x6D\x98\xE3" + "\x00\x14\x15\x59\xC1\x30\x64\xAF" + "\xA8\x01\x38\xAB\xD4\x8B\xEC\x7C" + "\x44\x9A\xC6\x2C\x2E\x2B\x2B\xF4" + "\x02\x37\xC4\x69\xEF\x36\xC1\xF3" + "\xA0\xFB\xFE\x29\xAD\x39\xCF\xD0" + "\x51\x73\xA3\x22\x42\x41\xAB\xD2" + "\x0F\x50\x14\xB9\x54\xD3\xD4\xFA" + "\xBF\xC9\xBB\xCE\xC4\x1D\x2D\xAF" + "\xC9\x3F\x07\x87\x42\x4B\x3A\x54" + "\x34\x8E\x37\xA3\x03\x6F\x65\x66" + "\xDB\x44\xC3\xE8\xD7\xDD\x7D\xDD" + "\x61\xB4\x2B\x80\xA3\x98\x13\xF5" + "\x5A\xD3\x34\x58\xC3\x6E\xF6\xB8" + "\x0A\xC6\x50\x01\x8E\xD5\x6C\x7D" + "\xFE\x16\xB6\xCF\xFC\x51\x40\xAE" + "\xB3\x15\xAC\x90\x6F\x0B\x28\x3A" + "\x60\x40\x38\x90\x20\x46\xC7\xB3" + "\x0B\x12\x6D\x3B\x15\x14\xF9\xF4" + "\x11\x41\x76\x6B\xB3\x60\x82\x3C" + "\x84\xFB\x08\x2E\x92\x25\xCB\x79" + "\x6F\x58\xC5\x94\x00\x00\x47\xB6" + "\x9E\xDC\x0F\x29\x70\x46\x20\x76" + "\x65\x75\x66\x5C\x00\x96\xB3\xE1" + "\x0B\xA7\x11\x8B\x2E\x61\x4E\x45" + "\x73\xFC\x91\xAB\x79\x41\x23\x14" + "\x13\xB6\x72\x6C\x46\xB3\x03\x11" + "\xE4\xF1\xEE\xC9\x7A\xCF\x96\x32" + "\xB6\xF0\x8B\x97\xB4\xCF\x82\xB7" + "\x15\x48\x44\x99\x09\xF6\xE0\xD7" + "\xBC\xF1\x5B\x91\x4F\x30\x22\xA2" + "\x45\xC4\x68\x55\xC2\xBE\xA7\xD2" + "\x12\x53\x35\x9C\xF9\xE7\x35\x5D" + "\x81\xE4\x86\x42\xC3\x58\xFB\xF0" + "\x38\x9B\x8E\x5A\xEF\x83\x33\x0F" + "\x00\x4E\x3F\x9F\xF5\x84\x62\xC4" + "\x19\x35\x88\x22\x45\x59\x0E\x8F" + "\xEC\x27\xDD\x4A\xA4\x1F\xBC\x41" + "\x9B\x66\x8D\x32\xBA\x81\x34\x87" + "\x0E\x74\x33\x30\x62\xB9\x89\xDF" + "\xF9\xC5\xDD\x27\xB3\x39\xCB\xCB", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec cast6_lrw_tv_template[] = { + { /* Generated from TF test vectors */ + .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" + "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" + "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" + "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" + "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" + "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" + "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" + "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" + "\x50\x38\x1f\x71\x49\xb6\x57\xd6" + "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" + "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" + "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" + "\xda\x10\x8e\xed\xa2\xa4\x87\xab" + "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" + "\xc9\xac\x42\x31\x95\x7c\xc9\x04" + "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" + "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" + "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" + "\x4c\x96\x12\xed\x7c\x92\x03\x01" + "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" + "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" + "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" + "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" + "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" + "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" + "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" + "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" + "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" + "\x76\x12\x73\x44\x1a\x56\xd7\x72" + "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" + "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" + "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" + "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" + "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" + "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" + "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" + "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" + "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" + "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" + "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" + "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" + "\x8d\x23\x31\x74\x84\xeb\x88\x6e" + "\xcc\xb9\xbc\x22\x83\x19\x07\x22" + "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" + "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" + "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" + "\x3c\xce\x8f\x42\x60\x71\xa7\x75" + "\x08\x40\x65\x8a\x82\xbf\xf5\x43" + "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" + "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" + "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" + "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" + "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" + "\x62\x73\x65\xfd\x46\x63\x25\x3d" + "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" + "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" + "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" + "\xc5\x68\x77\x84\x32\x2b\xcc\x85" + "\x74\x96\xf0\x12\x77\x61\xb9\xeb" + "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" + "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" + "\xda\x39\x87\x45\xc0\x2b\xbb\x01" + "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" + "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" + "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" + "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" + "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" + "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" + "\x21\xc4\xc2\x75\x67\x89\x37\x0a", + .ctext = "\x55\x25\x09\x8B\xB5\xD5\xF8\xBF" + "\x37\x4A\xFE\x3C\x47\xD8\xE6\xEB" + "\xCA\xA4\x9B\xB0\xAB\x6D\x64\xCA" + "\x58\xB6\x73\xF0\xD7\x52\x34\xEF" + "\xFB\x3E\x96\x81\xB7\x71\x34\xA4" + "\x55\x20\xBE\x39\x5A\x2B\xF9\xD1" + "\x65\x0B\xDA\xD3\x7E\xB3\xA6\xF7" + "\x2E\x0B\x5A\x52\xDB\x39\x8C\x9B" + "\x61\x17\x5F\xAF\xB6\x5A\xC8\x08" + "\xA7\xB7\x2A\x11\x7C\x97\x38\x9D" + "\x59\x0E\x66\x59\x5E\xD8\x8B\xCE" + "\x70\xE0\xC3\x42\xB0\x8C\x0F\xBA" + "\xB2\x0D\x81\xB6\xBE\x61\x1C\x2D" + "\x7E\xEA\x91\x25\xAC\xEC\xF8\x28" + "\x80\x1D\xF0\x30\xBA\x62\x77\x7D" + "\xDB\x15\x69\xDF\xFA\x2A\x81\x64" + "\x95\x5B\xA4\x7F\x3E\x4F\xE3\x30" + "\xB0\x5C\xC2\x05\xF8\xF0\x29\xE7" + "\x0A\xA0\x66\xB2\x5D\x0F\x39\x2B" + "\xB4\xB3\x00\xA9\xD0\xAB\x63\x61" + "\x5E\xDB\xFC\x11\x74\x25\x96\x65" + "\xE8\xE2\x34\x57\x77\x15\x5E\x70" + "\xFF\x10\x90\xC3\x64\xF0\x11\x0A" + "\x63\x3A\xD3\x55\x92\x15\x4B\x0C" + "\xC7\x08\x89\x17\x3B\x99\xAD\x63" + "\xE7\x06\xDF\x52\xBC\x15\x64\x45" + "\x9D\x7A\xFB\x69\xBC\x2D\x6E\xA9" + "\x35\xD9\xD8\xF5\x0C\xC4\xA2\x23" + "\x9C\x18\x8B\xA8\x8C\xFE\xF8\x0E" + "\xBD\xAB\x60\x1A\x51\x17\x54\x27" + "\xB6\xE8\xBE\x0F\xA9\xA5\x82\x19" + "\x2F\x6F\x20\xA7\x47\xED\x74\x6C" + "\x4E\xC1\xF8\x8C\x14\xF3\xBB\x1F" + "\xED\x4D\x8F\x7C\x37\xEF\x19\xA1" + "\x07\x16\xDE\x76\xCC\x5E\x94\x02" + "\xFB\xBF\xE4\x81\x50\xCE\xFC\x0F" + "\x9E\xCF\x3D\xF6\x67\x00\xBF\xA7" + "\x6E\x21\x58\x36\x06\xDE\xB3\xD4" + "\xA2\xFA\xD8\x4E\xE0\xB9\x7F\x23" + "\x51\x21\x2B\x32\x68\xAA\xF8\xA8" + "\x93\x08\xB5\x6D\xE6\x43\x2C\xB7" + "\x31\xB2\x0F\xD0\xA2\x51\xC0\x25" + "\x30\xC7\x10\x3F\x97\x27\x01\x8E" + "\xFA\xD8\x4F\x78\xD8\x2E\x1D\xEB" + "\xA1\x37\x52\x0F\x7B\x5E\x87\xA8" + "\x22\xE2\xE6\x92\xA7\x5F\x11\x32" + "\xCC\x93\x34\xFC\xD1\x7E\xAE\x54" + "\xBC\x6A\x1B\x91\xD1\x2E\x21\xEC" + "\x5D\xF1\xC4\xF1\x55\x20\xBF\xE5" + "\x96\x3D\x69\x91\x20\x4E\xF2\x61" + "\xDA\x77\xFE\xEE\xC3\x74\x57\x2A" + "\x78\x39\xB0\xE0\xCF\x12\x56\xD6" + "\x05\xDC\xF9\x19\x66\x44\x1D\xF9" + "\x82\x37\xD4\xC2\x60\xB6\x31\xDF" + "\x0C\xAF\xBC\x8B\x55\x9A\xC8\x2D" + "\xAB\xA7\x88\x7B\x41\xE8\x29\xC9" + "\x9B\x8D\xA7\x00\x86\x25\xB6\x14" + "\xF5\x13\x73\xD7\x4B\x6B\x83\xF3" + "\xAF\x96\x00\xE4\xB7\x3C\x65\xA6" + "\x15\xB7\x94\x7D\x4E\x70\x4C\x75" + "\xF3\xB4\x02\xA9\x17\x1C\x7A\x0A" + "\xC0\xD5\x33\x11\x56\xDE\xDC\xF5" + "\x8D\xD9\xCD\x3B\x22\x67\x18\xC7" + "\xC4\xF5\x99\x61\xBC\xBB\x5B\x46", + .len = 512, + .also_non_np = 1, + .np = 3, + .tap = { 512 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec cast6_xts_tv_template[] = { + { /* Generated from TF test vectors */ + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x62\x49\x77\x57\x24\x70\x93\x69" + "\x99\x59\x57\x49\x66\x96\x76\x27" + "\x31\x41\x59\x26\x53\x58\x97\x93" + "\x23\x84\x62\x64\x33\x83\x27\x95" + "\x02\x88\x41\x97\x16\x93\x99\x37" + "\x51\x05\x82\x09\x74\x94\x45\x92", + .klen = 64, + .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ctext = "\xDE\x6F\x22\xA5\xE8\x39\xE8\x78" + "\x88\x5A\x4F\x8D\x82\x76\x52\x6D" + "\xB2\x41\x16\xF4\x2B\xA6\xEB\xF6" + "\xE2\xC5\x62\x8D\x61\xA1\x01\xED" + "\xD9\x38\x01\xC1\x43\x63\x4E\x88" + "\xC9\x4B\x5A\x88\x80\xB7\x5C\x71" + "\x47\xEE\x11\xD8\xB7\x2D\x5D\x13" + "\x1A\xB1\x68\x5B\x61\xA7\xA9\x81" + "\x8B\x83\xA1\x6A\xAA\x36\xD6\xB6" + "\x60\x54\x09\x32\xFE\x6A\x76\x2E" + "\x28\xFF\xD5\xD6\xDD\x1D\x45\x7D" + "\xF0\x8B\xF3\x32\x4E\x6C\x12\xCB" + "\xB8\x25\x70\xF8\x40\xBC\x90\x1B" + "\x11\xC3\x59\xAF\xF0\x2F\x92\xDD" + "\xD3\x3B\xCF\x60\xA1\x78\x94\x57" + "\xAF\x76\xC1\x67\xA6\x3C\xCD\x98" + "\xB1\xF7\x27\xB9\xA3\xBD\x10\xEA" + "\xCD\x8B\xC2\xF2\x14\xF2\xB2\x67" + "\x05\xDD\x1D\x58\x6E\x2F\x95\x08" + "\x3A\xF8\x78\x76\x82\x56\xA7\xEC" + "\x51\x4B\x85\x77\xC2\x4C\x4A\x34" + "\x71\x38\x17\x91\x44\xE8\xFC\x65" + "\x99\x0D\x52\x91\xEE\xF8\xEF\x27" + "\x2A\x9E\x6E\x78\xC4\x26\x87\xF4" + "\x8A\xF0\x2D\x04\xE8\x14\x92\x5D" + "\x59\x22\x9B\x29\x5C\x18\xF0\xC3" + "\x47\xF3\x76\xD8\xE4\xF3\x1B\xD1" + "\x70\xA3\x0D\xB5\x70\x02\x1D\xA3" + "\x91\x3B\x49\x73\x18\xAB\xD4\xC9" + "\xC3\x1E\xEF\x1F\xFE\xD5\x59\x8A" + "\xD7\xF6\xC9\x71\x67\x79\xD7\x0E" + "\xBE\x1F\x8E\xEC\x55\x7E\x4F\x24" + "\xE6\x87\xEA\xFE\x96\x25\x67\x8E" + "\x93\x03\xFA\xFF\xCE\xAF\xB2\x3C" + "\x6F\xEB\x57\xFB\xD3\x28\x87\xA9" + "\xCE\xC2\xF5\x9C\xC6\x67\xB5\x97" + "\x49\xF7\x04\xCB\xEF\x84\x98\x33" + "\xAF\x38\xD3\x04\x1C\x24\x71\x38" + "\xC7\x71\xDD\x43\x0D\x12\x4A\x18" + "\xBA\xC4\xAF\xBA\xB2\x5B\xEB\x95" + "\x02\x43\x5D\xCE\x19\xCC\xCD\x66" + "\x91\x0B\x8C\x7F\x51\xC4\xBF\x3C" + "\x8B\xF1\xCC\xAA\x29\xD7\x87\xCB" + "\x3E\xC5\xF3\xC9\x75\xE8\xA3\x5B" + "\x30\x45\xA9\xB7\xAF\x80\x64\x6F" + "\x75\x4A\xA7\xC0\x6D\x19\x6B\xDE" + "\x17\xDE\x6D\xEA\x87\x9F\x95\xAE" + "\xF5\x3C\xEE\x54\xB8\x27\x84\xF8" + "\x97\xA3\xE1\x6F\x38\x24\x34\x88" + "\xCE\xBD\x32\x52\xE0\x00\x6C\x94" + "\xC9\xD7\x5D\x37\x81\x33\x2E\x7F" + "\x4F\x7E\x2E\x0D\x94\xBD\xEA\x59" + "\x34\x39\xA8\x35\x12\xB7\xBC\xAC" + "\xEA\x52\x9C\x78\x02\x6D\x92\x36" + "\xFB\x59\x2B\xA4\xEA\x7B\x1B\x83" + "\xE1\x4D\x5E\x2A\x7E\x92\xB1\x64" + "\xDE\xE0\x27\x4B\x0A\x6F\x4C\xE3" + "\xB0\xEB\x31\xE4\x69\x95\xAB\x35" + "\x8B\x2C\xF5\x6B\x7F\xF1\xA2\x82" + "\xF8\xD9\x47\x82\xA9\x82\x03\x91" + "\x69\x1F\xBE\x4C\xE7\xC7\x34\x2F" + "\x45\x72\x80\x17\x81\xBD\x9D\x62" + "\xA1\xAC\xE8\xCF\xC6\x74\xCF\xDC" + "\x22\x60\x4E\xE8\xA4\x5D\x85\xB9", + .len = 512, + .also_non_np = 1, + .np = 3, + .tap = { 512 - 20, 4, 16 }, + }, +}; + +/* + * AES test vectors. + */ +static const struct cipher_testvec aes_tv_template[] = { + { /* From FIPS-197 */ + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .klen = 16, + .ptext = "\x00\x11\x22\x33\x44\x55\x66\x77" + "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", + .ctext = "\x69\xc4\xe0\xd8\x6a\x7b\x04\x30" + "\xd8\xcd\xb7\x80\x70\xb4\xc5\x5a", + .len = 16, + }, { + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17", + .klen = 24, + .ptext = "\x00\x11\x22\x33\x44\x55\x66\x77" + "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", + .ctext = "\xdd\xa9\x7c\xa4\x86\x4c\xdf\xe0" + "\x6e\xaf\x70\xa0\xec\x0d\x71\x91", + .len = 16, + }, { + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .klen = 32, + .ptext = "\x00\x11\x22\x33\x44\x55\x66\x77" + "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", + .ctext = "\x8e\xa2\xb7\xca\x51\x67\x45\xbf" + "\xea\xfc\x49\x90\x4b\x49\x60\x89", + .len = 16, + }, { /* Generated with Crypto++ */ + .key = "\xA6\xC9\x83\xA6\xC9\xEC\x0F\x32" + "\x55\x0F\x32\x55\x78\x9B\xBE\x78" + "\x9B\xBE\xE1\x04\x27\xE1\x04\x27" + "\x4A\x6D\x90\x4A\x6D\x90\xB3\xD6", + .klen = 32, + .ptext = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" + "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" + "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" + "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" + "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" + "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" + "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" + "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" + "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" + "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" + "\x22\x8B\x17\x80\xE9\x52\xDE\x47" + "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" + "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" + "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" + "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" + "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" + "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" + "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" + "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" + "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" + "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" + "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" + "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" + "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" + "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" + "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" + "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" + "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" + "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" + "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" + "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" + "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" + "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" + "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" + "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" + "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" + "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" + "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" + "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" + "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" + "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" + "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" + "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" + "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" + "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" + "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" + "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" + "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" + "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" + "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" + "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" + "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" + "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" + "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" + "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" + "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" + "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" + "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" + "\x20\x89\x15\x7E\xE7\x50\xDC\x45" + "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" + "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" + "\xED\x56\xBF\x28\xB4\x1D\x86\x12", + .ctext = "\x71\x73\xF7\xDB\x24\x93\x21\x6D" + "\x61\x1E\xBB\x63\x42\x79\xDB\x64" + "\x6F\x82\xC0\xCA\xA3\x9B\xFA\x0B" + "\xD9\x08\xC7\x4A\x90\xAE\x8F\x5F" + "\x5E\x06\xF0\x5F\x31\x51\x18\x37" + "\x45\xD7\xCA\x3A\xFD\x6C\x3F\xE1" + "\xDD\x8D\x22\x65\x2B\x00\x50\xCE" + "\xBA\x28\x67\xD7\xCE\x0E\x0D\xEA" + "\x78\x69\x7F\xAE\x8F\x8B\x69\x37" + "\x75\xE0\xDC\x96\xE0\xB7\xF4\x09" + "\xCB\x6D\xA2\xFB\xDA\xAF\x09\xF8" + "\x81\x82\x27\xFA\x45\x9C\x29\xA4" + "\x22\x8B\x78\x69\x5B\x46\xF9\x39" + "\x1B\xCC\xF9\x1D\x09\xEB\xBC\x5C" + "\x41\x72\x51\x97\x1D\x07\x49\xA0" + "\x1B\x8E\x65\x4B\xB2\x6A\x12\x03" + "\x6A\x60\x95\xAC\xBD\xAC\x1A\x64" + "\xDE\x5A\xA5\xF0\x83\x2F\xCB\xCA" + "\x22\x74\xA6\x6C\x9B\x73\xCE\x3F" + "\xE1\x8B\x22\x17\x59\x0C\x47\x89" + "\x33\xA1\xD6\x47\x03\x19\x4F\xA8" + "\x67\x69\xF0\x5B\xF0\x20\xAD\x06" + "\x27\x81\x92\xD8\xC5\xBA\x98\x12" + "\xBE\x24\xB5\x2F\x75\x02\xC2\xAD" + "\x12\x2F\x07\x32\xEE\x39\xAF\x64" + "\x05\x8F\xB3\xD4\xEB\x1B\x46\x6E" + "\xD9\x21\xF9\xC4\xB7\xC9\x45\x68" + "\xB4\xA1\x74\x9F\x82\x47\xEB\xCC" + "\xBD\x0A\x14\x95\x0F\x8B\xA8\x2F" + "\x4B\x1B\xA7\xBF\x82\xA6\x43\x0C" + "\xB9\x39\x4A\xA8\x10\x6F\x50\x7B" + "\x25\xFB\x26\x81\xE0\x2F\xF0\x96" + "\x8D\x8B\xAC\x92\x0F\xF6\xED\x64" + "\x63\x29\x4C\x8E\x18\x13\xC5\xBF" + "\xFC\xA0\xD9\xBF\x7C\x3A\x0E\x29" + "\x6F\xD1\x6C\x6F\xA5\xDA\xBF\xB1" + "\x30\xEA\x44\x2D\xC3\x8F\x16\xE1" + "\x66\xFA\xA3\x21\x3E\xFC\x13\xCA" + "\xF0\xF6\xF0\x59\xBD\x8F\x38\x50" + "\x31\xCB\x69\x3F\x96\x15\xD6\xF5" + "\xAE\xFF\xF6\xAA\x41\x85\x4C\x10" + "\x58\xE3\xF9\x44\xE6\x28\xDA\x9A" + "\xDC\x6A\x80\x34\x73\x97\x1B\xC5" + "\xCA\x26\x16\x77\x0E\x60\xAB\x89" + "\x0F\x04\x27\xBD\xCE\x3E\x71\xB4" + "\xA0\xD7\x22\x7E\xDB\xEB\x24\x70" + "\x42\x71\x51\x78\x70\xB3\xE0\x3D" + "\x84\x8E\x8D\x7B\xD0\x6D\xEA\x92" + "\x11\x08\x42\x4F\xE5\xAD\x26\x92" + "\xD2\x00\xAE\xA8\xE3\x4B\x37\x47" + "\x22\xC1\x95\xC1\x63\x7F\xCB\x03" + "\xF3\xE3\xD7\x9D\x60\xC7\xBC\xEA" + "\x35\xA2\xFD\x45\x52\x39\x13\x6F" + "\xC1\x53\xF3\x53\xDF\x33\x84\xD7" + "\xD2\xC8\x37\xB0\x75\xE3\x41\x46" + "\xB3\xC7\x83\x2E\x8A\xBB\xA4\xE5" + "\x7F\x3C\xFD\x8B\xEB\xEA\x63\xBD" + "\xB7\x46\xE7\xBF\x09\x9C\x0D\x0F" + "\x40\x86\x7F\x51\xE1\x11\x9C\xCB" + "\x88\xE6\x68\x47\xE3\x2B\xC5\xFF" + "\x09\x79\xA0\x43\x5C\x0D\x08\x58" + "\x17\xBB\xC0\x6B\x62\x3F\x56\xE9", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec aes_cbc_tv_template[] = { + { /* From RFC 3602 */ + .key = "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" + "\x51\x2e\x03\xd5\x34\x12\x00\x06", + .klen = 16, + .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" + "\xb4\x22\xda\x80\x2c\x9f\xac\x41", + .ptext = "Single block msg", + .ctext = "\xe3\x53\x77\x9c\x10\x79\xae\xb8" + "\x27\x08\x94\x2d\xbe\x77\x18\x1a", + .len = 16, + .also_non_np = 1, + .np = 8, + .tap = { 3, 2, 3, 2, 3, 1, 1, 1 }, + }, { + .key = "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" + "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", + .klen = 16, + .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" + "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .ctext = "\xd2\x96\xcd\x94\xc2\xcc\xcf\x8a" + "\x3a\x86\x30\x28\xb5\xe1\xdc\x0a" + "\x75\x86\x60\x2d\x25\x3c\xff\xf9" + "\x1b\x82\x66\xbe\xa6\xd6\x1a\xb1", + .len = 32, + }, { /* From NIST SP800-38A */ + .key = "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" + "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" + "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", + .klen = 24, + .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .ptext = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ctext = "\x4f\x02\x1d\xb2\x43\xbc\x63\x3d" + "\x71\x78\x18\x3a\x9f\xa0\x71\xe8" + "\xb4\xd9\xad\xa9\xad\x7d\xed\xf4" + "\xe5\xe7\x38\x76\x3f\x69\x14\x5a" + "\x57\x1b\x24\x20\x12\xfb\x7a\xe0" + "\x7f\xa9\xba\xac\x3d\xf1\x02\xe0" + "\x08\xb0\xe2\x79\x88\x59\x88\x81" + "\xd9\x20\xa9\xe6\x4f\x56\x15\xcd", + .len = 64, + }, { + .key = "\x60\x3d\xeb\x10\x15\xca\x71\xbe" + "\x2b\x73\xae\xf0\x85\x7d\x77\x81" + "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" + "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", + .klen = 32, + .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .ptext = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ctext = "\xf5\x8c\x4c\x04\xd6\xe5\xf1\xba" + "\x77\x9e\xab\xfb\x5f\x7b\xfb\xd6" + "\x9c\xfc\x4e\x96\x7e\xdb\x80\x8d" + "\x67\x9f\x77\x7b\xc6\x70\x2c\x7d" + "\x39\xf2\x33\x69\xa9\xd9\xba\xcf" + "\xa5\x30\xe2\x63\x04\x23\x14\x61" + "\xb2\xeb\x05\xe2\xc3\x9b\xe9\xfc" + "\xda\x6c\x19\x07\x8c\x6a\x9d\x1b", + .len = 64, + }, { /* Generated with Crypto++ */ + .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55" + "\x0F\x32\x55\x78\x9B\xBE\x78\x9B" + "\xBE\xE1\x04\x27\xE1\x04\x27\x4A" + "\x6D\x90\x4A\x6D\x90\xB3\xD6\xF9", + .klen = 32, + .iv = "\xE7\x82\x1D\xB8\x53\x11\xAC\x47" + "\xE2\x7D\x18\xD6\x71\x0C\xA7\x42", + .ptext = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" + "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" + "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" + "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" + "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" + "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" + "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" + "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" + "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" + "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" + "\x22\x8B\x17\x80\xE9\x52\xDE\x47" + "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" + "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" + "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" + "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" + "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" + "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" + "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" + "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" + "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" + "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" + "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" + "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" + "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" + "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" + "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" + "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" + "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" + "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" + "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" + "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" + "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" + "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" + "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" + "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" + "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" + "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" + "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" + "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" + "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" + "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" + "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" + "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" + "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" + "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" + "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" + "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" + "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" + "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" + "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" + "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" + "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" + "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" + "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" + "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" + "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" + "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" + "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" + "\x20\x89\x15\x7E\xE7\x50\xDC\x45" + "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" + "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" + "\xED\x56\xBF\x28\xB4\x1D\x86\x12", + .ctext = "\xEA\x65\x8A\x19\xB0\x66\xC1\x3F" + "\xCE\xF1\x97\x75\xC1\xFD\xB5\xAF" + "\x52\x65\xF7\xFF\xBC\xD8\x2D\x9F" + "\x2F\xB9\x26\x9B\x6F\x10\xB7\xB8" + "\x26\xA1\x02\x46\xA2\xAD\xC6\xC0" + "\x11\x15\xFF\x6D\x1E\x82\x04\xA6" + "\xB1\x74\xD1\x08\x13\xFD\x90\x7C" + "\xF5\xED\xD3\xDB\x5A\x0A\x0C\x2F" + "\x0A\x70\xF1\x88\x07\xCF\x21\x26" + "\x40\x40\x8A\xF5\x53\xF7\x24\x4F" + "\x83\x38\x43\x5F\x08\x99\xEB\xE3" + "\xDC\x02\x64\x67\x50\x6E\x15\xC3" + "\x01\x1A\xA0\x81\x13\x65\xA6\x73" + "\x71\xA6\x3B\x91\x83\x77\xBE\xFA" + "\xDB\x71\x73\xA6\xC1\xAE\x43\xC3" + "\x36\xCE\xD6\xEB\xF9\x30\x1C\x4F" + "\x80\x38\x5E\x9C\x6E\xAB\x98\x2F" + "\x53\xAF\xCF\xC8\x9A\xB8\x86\x43" + "\x3E\x86\xE7\xA1\xF4\x2F\x30\x40" + "\x03\xA8\x6C\x50\x42\x9F\x77\x59" + "\x89\xA0\xC5\xEC\x9A\xB8\xDD\x99" + "\x16\x24\x02\x07\x48\xAE\xF2\x31" + "\x34\x0E\xC3\x85\xFE\x1C\x95\x99" + "\x87\x58\x98\x8B\xE7\xC6\xC5\x70" + "\x73\x81\x07\x7C\x56\x2F\xD8\x1B" + "\xB7\xB9\x2B\xAB\xE3\x01\x87\x0F" + "\xD8\xBB\xC0\x0D\xAC\x2C\x2F\x98" + "\x3C\x0B\xA2\x99\x4A\x8C\xF7\x04" + "\xE0\xE0\xCF\xD1\x81\x5B\xFE\xF5" + "\x24\x04\xFD\xB8\xDF\x13\xD8\xCD" + "\xF1\xE3\x3D\x98\x50\x02\x77\x9E" + "\xBC\x22\xAB\xFA\xC2\x43\x1F\x66" + "\x20\x02\x23\xDA\xDF\xA0\x89\xF6" + "\xD8\xF3\x45\x24\x53\x6F\x16\x77" + "\x02\x3E\x7B\x36\x5F\xA0\x3B\x78" + "\x63\xA2\xBD\xB5\xA4\xCA\x1E\xD3" + "\x57\xBC\x0B\x9F\x43\x51\x28\x4F" + "\x07\x50\x6C\x68\x12\x07\xCF\xFA" + "\x6B\x72\x0B\xEB\xF8\x88\x90\x2C" + "\x7E\xF5\x91\xD1\x03\xD8\xD5\xBD" + "\x22\x39\x7B\x16\x03\x01\x69\xAF" + "\x3D\x38\x66\x28\x0C\xBE\x5B\xC5" + "\x03\xB4\x2F\x51\x8A\x56\x17\x2B" + "\x88\x42\x6D\x40\x68\x8F\xD0\x11" + "\x19\xF9\x1F\x43\x79\x95\x31\xFA" + "\x28\x7A\x3D\xF7\x66\xEB\xEF\xAC" + "\x06\xB2\x01\xAD\xDB\x68\xDB\xEC" + "\x8D\x53\x6E\x72\x68\xA3\xC7\x63" + "\x43\x2B\x78\xE0\x04\x29\x8F\x72" + "\xB2\x2C\xE6\x84\x03\x30\x6D\xCD" + "\x26\x92\x37\xE1\x2F\xBB\x8B\x9D" + "\xE4\x4C\xF6\x93\xBC\xD9\xAD\x44" + "\x52\x65\xC7\xB0\x0E\x3F\x0E\x61" + "\x56\x5D\x1C\x6D\xA7\x05\x2E\xBC" + "\x58\x08\x15\xAB\x12\xAB\x17\x4A" + "\x5E\x1C\xF2\xCD\xB8\xA2\xAE\xFB" + "\x9B\x2E\x0E\x85\x34\x80\x0E\x3F" + "\x4C\xB8\xDB\xCE\x1C\x90\xA1\x61" + "\x6C\x69\x09\x35\x9E\xD4\xF4\xAD" + "\xBC\x06\x41\xE3\x01\xB4\x4E\x0A" + "\xE0\x1F\x91\xF8\x82\x96\x2D\x65" + "\xA3\xAA\x13\xCC\x50\xFF\x7B\x02", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, +}; + +static const struct aead_testvec hmac_md5_ecb_cipher_null_enc_tv_template[] = { + { /* Input data from RFC 2410 Case 1 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x00" /* enc key length */ + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 8 + 16 + 0, + .iv = "", + .input = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .ilen = 8, + .result = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\xaa\x42\xfe\x43\x8d\xea\xa3\x5a" + "\xb9\x3d\x9f\xb1\xa3\x8e\x9b\xae", + .rlen = 8 + 16, + }, { /* Input data from RFC 2410 Case 2 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x00" /* enc key length */ + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 8 + 16 + 0, + .iv = "", + .input = "Network Security People Have A Strange Sense Of Humor", + .ilen = 53, + .result = "Network Security People Have A Strange Sense Of Humor" + "\x73\xa5\x3e\x1c\x08\x0e\x8a\x8a" + "\x8e\xb5\x5f\x90\x8e\xfe\x13\x23", + .rlen = 53 + 16, + }, +}; + +static const struct aead_testvec hmac_md5_ecb_cipher_null_dec_tv_template[] = { + { +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x00" /* enc key length */ + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 8 + 16 + 0, + .iv = "", + .input = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\xaa\x42\xfe\x43\x8d\xea\xa3\x5a" + "\xb9\x3d\x9f\xb1\xa3\x8e\x9b\xae", + .ilen = 8 + 16, + .result = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .rlen = 8, + }, { +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x00" /* enc key length */ + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 8 + 16 + 0, + .iv = "", + .input = "Network Security People Have A Strange Sense Of Humor" + "\x73\xa5\x3e\x1c\x08\x0e\x8a\x8a" + "\x8e\xb5\x5f\x90\x8e\xfe\x13\x23", + .ilen = 53 + 16, + .result = "Network Security People Have A Strange Sense Of Humor", + .rlen = 53, + }, +}; + +static const struct aead_testvec hmac_sha1_aes_cbc_enc_tv_temp[] = { + { /* RFC 3602 Case 1 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00" + "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" + "\x51\x2e\x03\xd5\x34\x12\x00\x06", + .klen = 8 + 20 + 16, + .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" + "\xb4\x22\xda\x80\x2c\x9f\xac\x41", + .assoc = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" + "\xb4\x22\xda\x80\x2c\x9f\xac\x41", + .alen = 16, + .input = "Single block msg", + .ilen = 16, + .result = "\xe3\x53\x77\x9c\x10\x79\xae\xb8" + "\x27\x08\x94\x2d\xbe\x77\x18\x1a" + "\x1b\x13\xcb\xaf\x89\x5e\xe1\x2c" + "\x13\xc5\x2e\xa3\xcc\xed\xdc\xb5" + "\x03\x71\xa2\x06", + .rlen = 16 + 20, + }, { /* RFC 3602 Case 2 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33" + "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" + "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", + .klen = 8 + 20 + 16, + .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" + "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", + .assoc = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" + "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", + .alen = 16, + .input = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .ilen = 32, + .result = "\xd2\x96\xcd\x94\xc2\xcc\xcf\x8a" + "\x3a\x86\x30\x28\xb5\xe1\xdc\x0a" + "\x75\x86\x60\x2d\x25\x3c\xff\xf9" + "\x1b\x82\x66\xbe\xa6\xd6\x1a\xb1" + "\xad\x9b\x4c\x5c\x85\xe1\xda\xae" + "\xee\x81\x4e\xd7\xdb\x74\xcf\x58" + "\x65\x39\xf8\xde", + .rlen = 32 + 20, + }, { /* RFC 3602 Case 3 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55" + "\x6c\x3e\xa0\x47\x76\x30\xce\x21" + "\xa2\xce\x33\x4a\xa7\x46\xc2\xcd", + .klen = 8 + 20 + 16, + .iv = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" + "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", + .assoc = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" + "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", + .alen = 16, + .input = "This is a 48-byte message (exactly 3 AES blocks)", + .ilen = 48, + .result = "\xd0\xa0\x2b\x38\x36\x45\x17\x53" + "\xd4\x93\x66\x5d\x33\xf0\xe8\x86" + "\x2d\xea\x54\xcd\xb2\x93\xab\xc7" + "\x50\x69\x39\x27\x67\x72\xf8\xd5" + "\x02\x1c\x19\x21\x6b\xad\x52\x5c" + "\x85\x79\x69\x5d\x83\xba\x26\x84" + "\xc2\xec\x0c\xf8\x7f\x05\xba\xca" + "\xff\xee\x4c\xd0\x93\xe6\x36\x7f" + "\x8d\x62\xf2\x1e", + .rlen = 48 + 20, + }, { /* RFC 3602 Case 4 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55" + "\x56\xe4\x7a\x38\xc5\x59\x89\x74" + "\xbc\x46\x90\x3d\xba\x29\x03\x49", + .klen = 8 + 20 + 16, + .iv = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" + "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", + .assoc = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" + "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", + .alen = 16, + .input = "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf", + .ilen = 64, + .result = "\xc3\x0e\x32\xff\xed\xc0\x77\x4e" + "\x6a\xff\x6a\xf0\x86\x9f\x71\xaa" + "\x0f\x3a\xf0\x7a\x9a\x31\xa9\xc6" + "\x84\xdb\x20\x7e\xb0\xef\x8e\x4e" + "\x35\x90\x7a\xa6\x32\xc3\xff\xdf" + "\x86\x8b\xb7\xb2\x9d\x3d\x46\xad" + "\x83\xce\x9f\x9a\x10\x2e\xe9\x9d" + "\x49\xa5\x3e\x87\xf4\xc3\xda\x55" + "\x1c\x45\x57\xa9\x56\xcb\xa9\x2d" + "\x18\xac\xf1\xc7\x5d\xd1\xcd\x0d" + "\x1d\xbe\xc6\xe9", + .rlen = 64 + 20, + }, { /* RFC 3602 Case 5 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55" + "\x90\xd3\x82\xb4\x10\xee\xba\x7a" + "\xd9\x38\xc4\x6c\xec\x1a\x82\xbf", + .klen = 8 + 20 + 16, + .iv = "\xe9\x6e\x8c\x08\xab\x46\x57\x63" + "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\xe9\x6e\x8c\x08\xab\x46\x57\x63" + "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", + .alen = 24, + .input = "\x08\x00\x0e\xbd\xa7\x0a\x00\x00" + "\x8e\x9c\x08\x3d\xb9\x5b\x07\x00" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x01\x02\x03\x04\x05\x06\x07\x08" + "\x09\x0a\x0b\x0c\x0d\x0e\x0e\x01", + .ilen = 80, + .result = "\xf6\x63\xc2\x5d\x32\x5c\x18\xc6" + "\xa9\x45\x3e\x19\x4e\x12\x08\x49" + "\xa4\x87\x0b\x66\xcc\x6b\x99\x65" + "\x33\x00\x13\xb4\x89\x8d\xc8\x56" + "\xa4\x69\x9e\x52\x3a\x55\xdb\x08" + "\x0b\x59\xec\x3a\x8e\x4b\x7e\x52" + "\x77\x5b\x07\xd1\xdb\x34\xed\x9c" + "\x53\x8a\xb5\x0c\x55\x1b\x87\x4a" + "\xa2\x69\xad\xd0\x47\xad\x2d\x59" + "\x13\xac\x19\xb7\xcf\xba\xd4\xa6" + "\x58\xc6\x84\x75\xe4\xe9\x6b\x0c" + "\xe1\xc5\x0b\x73\x4d\x82\x55\xa8" + "\x85\xe1\x59\xf7", + .rlen = 80 + 20, + }, { /* NIST SP800-38A F.2.3 CBC-AES192.Encrypt */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x18" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55" + "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" + "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" + "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", + .klen = 8 + 20 + 24, + .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .alen = 16, + .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ilen = 64, + .result = "\x4f\x02\x1d\xb2\x43\xbc\x63\x3d" + "\x71\x78\x18\x3a\x9f\xa0\x71\xe8" + "\xb4\xd9\xad\xa9\xad\x7d\xed\xf4" + "\xe5\xe7\x38\x76\x3f\x69\x14\x5a" + "\x57\x1b\x24\x20\x12\xfb\x7a\xe0" + "\x7f\xa9\xba\xac\x3d\xf1\x02\xe0" + "\x08\xb0\xe2\x79\x88\x59\x88\x81" + "\xd9\x20\xa9\xe6\x4f\x56\x15\xcd" + "\x73\xe3\x19\x3f\x8b\xc9\xc6\xf4" + "\x5a\xf1\x5b\xa8\x98\x07\xc5\x36" + "\x47\x4c\xfc\x36", + .rlen = 64 + 20, + }, { /* NIST SP800-38A F.2.5 CBC-AES256.Encrypt */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x20" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55" + "\x60\x3d\xeb\x10\x15\xca\x71\xbe" + "\x2b\x73\xae\xf0\x85\x7d\x77\x81" + "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" + "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", + .klen = 8 + 20 + 32, + .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .alen = 16, + .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ilen = 64, + .result = "\xf5\x8c\x4c\x04\xd6\xe5\xf1\xba" + "\x77\x9e\xab\xfb\x5f\x7b\xfb\xd6" + "\x9c\xfc\x4e\x96\x7e\xdb\x80\x8d" + "\x67\x9f\x77\x7b\xc6\x70\x2c\x7d" + "\x39\xf2\x33\x69\xa9\xd9\xba\xcf" + "\xa5\x30\xe2\x63\x04\x23\x14\x61" + "\xb2\xeb\x05\xe2\xc3\x9b\xe9\xfc" + "\xda\x6c\x19\x07\x8c\x6a\x9d\x1b" + "\xa3\xe8\x9b\x17\xe3\xf4\x7f\xde" + "\x1b\x9f\xc6\x81\x26\x43\x4a\x87" + "\x51\xee\xd6\x4e", + .rlen = 64 + 20, + }, +}; + +static const struct aead_testvec hmac_sha1_ecb_cipher_null_enc_tv_temp[] = { + { /* Input data from RFC 2410 Case 1 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x00" /* enc key length */ + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00", + .klen = 8 + 20 + 0, + .iv = "", + .input = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .ilen = 8, + .result = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\x40\xc3\x0a\xa1\xc9\xa0\x28\xab" + "\x99\x5e\x19\x04\xd1\x72\xef\xb8" + "\x8c\x5e\xe4\x08", + .rlen = 8 + 20, + }, { /* Input data from RFC 2410 Case 2 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x00" /* enc key length */ + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00", + .klen = 8 + 20 + 0, + .iv = "", + .input = "Network Security People Have A Strange Sense Of Humor", + .ilen = 53, + .result = "Network Security People Have A Strange Sense Of Humor" + "\x75\x6f\x42\x1e\xf8\x50\x21\xd2" + "\x65\x47\xee\x8e\x1a\xef\x16\xf6" + "\x91\x56\xe4\xd6", + .rlen = 53 + 20, + }, +}; + +static const struct aead_testvec hmac_sha1_ecb_cipher_null_dec_tv_temp[] = { + { +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x00" /* enc key length */ + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00", + .klen = 8 + 20 + 0, + .iv = "", + .input = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\x40\xc3\x0a\xa1\xc9\xa0\x28\xab" + "\x99\x5e\x19\x04\xd1\x72\xef\xb8" + "\x8c\x5e\xe4\x08", + .ilen = 8 + 20, + .result = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .rlen = 8, + }, { +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x00" /* enc key length */ + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00", + .klen = 8 + 20 + 0, + .iv = "", + .input = "Network Security People Have A Strange Sense Of Humor" + "\x75\x6f\x42\x1e\xf8\x50\x21\xd2" + "\x65\x47\xee\x8e\x1a\xef\x16\xf6" + "\x91\x56\xe4\xd6", + .ilen = 53 + 20, + .result = "Network Security People Have A Strange Sense Of Humor", + .rlen = 53, + }, +}; + +static const struct aead_testvec hmac_sha256_aes_cbc_enc_tv_temp[] = { + { /* RFC 3602 Case 1 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" + "\x51\x2e\x03\xd5\x34\x12\x00\x06", + .klen = 8 + 32 + 16, + .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" + "\xb4\x22\xda\x80\x2c\x9f\xac\x41", + .assoc = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" + "\xb4\x22\xda\x80\x2c\x9f\xac\x41", + .alen = 16, + .input = "Single block msg", + .ilen = 16, + .result = "\xe3\x53\x77\x9c\x10\x79\xae\xb8" + "\x27\x08\x94\x2d\xbe\x77\x18\x1a" + "\xcc\xde\x2d\x6a\xae\xf1\x0b\xcc" + "\x38\x06\x38\x51\xb4\xb8\xf3\x5b" + "\x5c\x34\xa6\xa3\x6e\x0b\x05\xe5" + "\x6a\x6d\x44\xaa\x26\xa8\x44\xa5", + .rlen = 16 + 32, + }, { /* RFC 3602 Case 2 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" + "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", + .klen = 8 + 32 + 16, + .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" + "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", + .assoc = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" + "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", + .alen = 16, + .input = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .ilen = 32, + .result = "\xd2\x96\xcd\x94\xc2\xcc\xcf\x8a" + "\x3a\x86\x30\x28\xb5\xe1\xdc\x0a" + "\x75\x86\x60\x2d\x25\x3c\xff\xf9" + "\x1b\x82\x66\xbe\xa6\xd6\x1a\xb1" + "\xf5\x33\x53\xf3\x68\x85\x2a\x99" + "\x0e\x06\x58\x8f\xba\xf6\x06\xda" + "\x49\x69\x0d\x5b\xd4\x36\x06\x62" + "\x35\x5e\x54\x58\x53\x4d\xdf\xbf", + .rlen = 32 + 32, + }, { /* RFC 3602 Case 3 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x6c\x3e\xa0\x47\x76\x30\xce\x21" + "\xa2\xce\x33\x4a\xa7\x46\xc2\xcd", + .klen = 8 + 32 + 16, + .iv = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" + "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", + .assoc = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" + "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", + .alen = 16, + .input = "This is a 48-byte message (exactly 3 AES blocks)", + .ilen = 48, + .result = "\xd0\xa0\x2b\x38\x36\x45\x17\x53" + "\xd4\x93\x66\x5d\x33\xf0\xe8\x86" + "\x2d\xea\x54\xcd\xb2\x93\xab\xc7" + "\x50\x69\x39\x27\x67\x72\xf8\xd5" + "\x02\x1c\x19\x21\x6b\xad\x52\x5c" + "\x85\x79\x69\x5d\x83\xba\x26\x84" + "\x68\xb9\x3e\x90\x38\xa0\x88\x01" + "\xe7\xc6\xce\x10\x31\x2f\x9b\x1d" + "\x24\x78\xfb\xbe\x02\xe0\x4f\x40" + "\x10\xbd\xaa\xc6\xa7\x79\xe0\x1a", + .rlen = 48 + 32, + }, { /* RFC 3602 Case 4 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x56\xe4\x7a\x38\xc5\x59\x89\x74" + "\xbc\x46\x90\x3d\xba\x29\x03\x49", + .klen = 8 + 32 + 16, + .iv = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" + "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", + .assoc = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" + "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", + .alen = 16, + .input = "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf", + .ilen = 64, + .result = "\xc3\x0e\x32\xff\xed\xc0\x77\x4e" + "\x6a\xff\x6a\xf0\x86\x9f\x71\xaa" + "\x0f\x3a\xf0\x7a\x9a\x31\xa9\xc6" + "\x84\xdb\x20\x7e\xb0\xef\x8e\x4e" + "\x35\x90\x7a\xa6\x32\xc3\xff\xdf" + "\x86\x8b\xb7\xb2\x9d\x3d\x46\xad" + "\x83\xce\x9f\x9a\x10\x2e\xe9\x9d" + "\x49\xa5\x3e\x87\xf4\xc3\xda\x55" + "\x7a\x1b\xd4\x3c\xdb\x17\x95\xe2" + "\xe0\x93\xec\xc9\x9f\xf7\xce\xd8" + "\x3f\x54\xe2\x49\x39\xe3\x71\x25" + "\x2b\x6c\xe9\x5d\xec\xec\x2b\x64", + .rlen = 64 + 32, + }, { /* RFC 3602 Case 5 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x90\xd3\x82\xb4\x10\xee\xba\x7a" + "\xd9\x38\xc4\x6c\xec\x1a\x82\xbf", + .klen = 8 + 32 + 16, + .iv = "\xe9\x6e\x8c\x08\xab\x46\x57\x63" + "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\xe9\x6e\x8c\x08\xab\x46\x57\x63" + "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", + .alen = 24, + .input = "\x08\x00\x0e\xbd\xa7\x0a\x00\x00" + "\x8e\x9c\x08\x3d\xb9\x5b\x07\x00" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x01\x02\x03\x04\x05\x06\x07\x08" + "\x09\x0a\x0b\x0c\x0d\x0e\x0e\x01", + .ilen = 80, + .result = "\xf6\x63\xc2\x5d\x32\x5c\x18\xc6" + "\xa9\x45\x3e\x19\x4e\x12\x08\x49" + "\xa4\x87\x0b\x66\xcc\x6b\x99\x65" + "\x33\x00\x13\xb4\x89\x8d\xc8\x56" + "\xa4\x69\x9e\x52\x3a\x55\xdb\x08" + "\x0b\x59\xec\x3a\x8e\x4b\x7e\x52" + "\x77\x5b\x07\xd1\xdb\x34\xed\x9c" + "\x53\x8a\xb5\x0c\x55\x1b\x87\x4a" + "\xa2\x69\xad\xd0\x47\xad\x2d\x59" + "\x13\xac\x19\xb7\xcf\xba\xd4\xa6" + "\xbb\xd4\x0f\xbe\xa3\x3b\x4c\xb8" + "\x3a\xd2\xe1\x03\x86\xa5\x59\xb7" + "\x73\xc3\x46\x20\x2c\xb1\xef\x68" + "\xbb\x8a\x32\x7e\x12\x8c\x69\xcf", + .rlen = 80 + 32, + }, { /* NIST SP800-38A F.2.3 CBC-AES192.Encrypt */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x18" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" + "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" + "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", + .klen = 8 + 32 + 24, + .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .alen = 16, + .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ilen = 64, + .result = "\x4f\x02\x1d\xb2\x43\xbc\x63\x3d" + "\x71\x78\x18\x3a\x9f\xa0\x71\xe8" + "\xb4\xd9\xad\xa9\xad\x7d\xed\xf4" + "\xe5\xe7\x38\x76\x3f\x69\x14\x5a" + "\x57\x1b\x24\x20\x12\xfb\x7a\xe0" + "\x7f\xa9\xba\xac\x3d\xf1\x02\xe0" + "\x08\xb0\xe2\x79\x88\x59\x88\x81" + "\xd9\x20\xa9\xe6\x4f\x56\x15\xcd" + "\x2f\xee\x5f\xdb\x66\xfe\x79\x09" + "\x61\x81\x31\xea\x5b\x3d\x8e\xfb" + "\xca\x71\x85\x93\xf7\x85\x55\x8b" + "\x7a\xe4\x94\xca\x8b\xba\x19\x33", + .rlen = 64 + 32, + }, { /* NIST SP800-38A F.2.5 CBC-AES256.Encrypt */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x20" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x60\x3d\xeb\x10\x15\xca\x71\xbe" + "\x2b\x73\xae\xf0\x85\x7d\x77\x81" + "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" + "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", + .klen = 8 + 32 + 32, + .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .alen = 16, + .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ilen = 64, + .result = "\xf5\x8c\x4c\x04\xd6\xe5\xf1\xba" + "\x77\x9e\xab\xfb\x5f\x7b\xfb\xd6" + "\x9c\xfc\x4e\x96\x7e\xdb\x80\x8d" + "\x67\x9f\x77\x7b\xc6\x70\x2c\x7d" + "\x39\xf2\x33\x69\xa9\xd9\xba\xcf" + "\xa5\x30\xe2\x63\x04\x23\x14\x61" + "\xb2\xeb\x05\xe2\xc3\x9b\xe9\xfc" + "\xda\x6c\x19\x07\x8c\x6a\x9d\x1b" + "\x24\x29\xed\xc2\x31\x49\xdb\xb1" + "\x8f\x74\xbd\x17\x92\x03\xbe\x8f" + "\xf3\x61\xde\x1c\xe9\xdb\xcd\xd0" + "\xcc\xce\xe9\x85\x57\xcf\x6f\x5f", + .rlen = 64 + 32, + }, +}; + +static const struct aead_testvec hmac_sha512_aes_cbc_enc_tv_temp[] = { + { /* RFC 3602 Case 1 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" + "\x51\x2e\x03\xd5\x34\x12\x00\x06", + .klen = 8 + 64 + 16, + .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" + "\xb4\x22\xda\x80\x2c\x9f\xac\x41", + .assoc = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" + "\xb4\x22\xda\x80\x2c\x9f\xac\x41", + .alen = 16, + .input = "Single block msg", + .ilen = 16, + .result = "\xe3\x53\x77\x9c\x10\x79\xae\xb8" + "\x27\x08\x94\x2d\xbe\x77\x18\x1a" + "\x3f\xdc\xad\x90\x03\x63\x5e\x68" + "\xc3\x13\xdd\xa4\x5c\x4d\x54\xa7" + "\x19\x6e\x03\x75\x2b\xa1\x62\xce" + "\xe0\xc6\x96\x75\xb2\x14\xca\x96" + "\xec\xbd\x50\x08\x07\x64\x1a\x49" + "\xe8\x9a\x7c\x06\x3d\xcb\xff\xb2" + "\xfa\x20\x89\xdd\x9c\xac\x9e\x16" + "\x18\x8a\xa0\x6d\x01\x6c\xa3\x3a", + .rlen = 16 + 64, + }, { /* RFC 3602 Case 2 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" + "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", + .klen = 8 + 64 + 16, + .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" + "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", + .assoc = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" + "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", + .alen = 16, + .input = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .ilen = 32, + .result = "\xd2\x96\xcd\x94\xc2\xcc\xcf\x8a" + "\x3a\x86\x30\x28\xb5\xe1\xdc\x0a" + "\x75\x86\x60\x2d\x25\x3c\xff\xf9" + "\x1b\x82\x66\xbe\xa6\xd6\x1a\xb1" + "\xda\xb2\x0c\xb2\x26\xc4\xd5\xef" + "\x60\x38\xa4\x5e\x9a\x8c\x1b\x41" + "\x03\x9f\xc4\x64\x7f\x01\x42\x9b" + "\x0e\x1b\xea\xef\xbc\x88\x19\x5e" + "\x31\x7e\xc2\x95\xfc\x09\x32\x0a" + "\x46\x32\x7c\x41\x9c\x59\x3e\xe9" + "\x8f\x9f\xd4\x31\xd6\x22\xbd\xf8" + "\xf7\x0a\x94\xe5\xa9\xc3\xf6\x9d", + .rlen = 32 + 64, + }, { /* RFC 3602 Case 3 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x33\x44\x55\x66\x77\x88\x99\xaa" + "\xbb\xcc\xdd\xee\xff\x11\x22\x33" + "\x44\x55\x66\x77\x88\x99\xaa\xbb" + "\xcc\xdd\xee\xff\x11\x22\x33\x44" + "\x6c\x3e\xa0\x47\x76\x30\xce\x21" + "\xa2\xce\x33\x4a\xa7\x46\xc2\xcd", + .klen = 8 + 64 + 16, + .iv = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" + "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", + .assoc = "\xc7\x82\xdc\x4c\x09\x8c\x66\xcb" + "\xd9\xcd\x27\xd8\x25\x68\x2c\x81", + .alen = 16, + .input = "This is a 48-byte message (exactly 3 AES blocks)", + .ilen = 48, + .result = "\xd0\xa0\x2b\x38\x36\x45\x17\x53" + "\xd4\x93\x66\x5d\x33\xf0\xe8\x86" + "\x2d\xea\x54\xcd\xb2\x93\xab\xc7" + "\x50\x69\x39\x27\x67\x72\xf8\xd5" + "\x02\x1c\x19\x21\x6b\xad\x52\x5c" + "\x85\x79\x69\x5d\x83\xba\x26\x84" + "\x64\x19\x17\x5b\x57\xe0\x21\x0f" + "\xca\xdb\xa1\x26\x38\x14\xa2\x69" + "\xdb\x54\x67\x80\xc0\x54\xe0\xfd" + "\x3e\x91\xe7\x91\x7f\x13\x38\x44" + "\xb7\xb1\xd6\xc8\x7d\x48\x8d\x41" + "\x08\xea\x29\x6c\x74\x67\x3f\xb0" + "\xac\x7f\x5c\x1d\xf5\xee\x22\x66" + "\x27\xa6\xb6\x13\xba\xba\xf0\xc2", + .rlen = 48 + 64, + }, { /* RFC 3602 Case 4 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x33\x44\x55\x66\x77\x88\x99\xaa" + "\xbb\xcc\xdd\xee\xff\x11\x22\x33" + "\x44\x55\x66\x77\x88\x99\xaa\xbb" + "\xcc\xdd\xee\xff\x11\x22\x33\x44" + "\x56\xe4\x7a\x38\xc5\x59\x89\x74" + "\xbc\x46\x90\x3d\xba\x29\x03\x49", + .klen = 8 + 64 + 16, + .iv = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" + "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", + .assoc = "\x8c\xe8\x2e\xef\xbe\xa0\xda\x3c" + "\x44\x69\x9e\xd7\xdb\x51\xb7\xd9", + .alen = 16, + .input = "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf", + .ilen = 64, + .result = "\xc3\x0e\x32\xff\xed\xc0\x77\x4e" + "\x6a\xff\x6a\xf0\x86\x9f\x71\xaa" + "\x0f\x3a\xf0\x7a\x9a\x31\xa9\xc6" + "\x84\xdb\x20\x7e\xb0\xef\x8e\x4e" + "\x35\x90\x7a\xa6\x32\xc3\xff\xdf" + "\x86\x8b\xb7\xb2\x9d\x3d\x46\xad" + "\x83\xce\x9f\x9a\x10\x2e\xe9\x9d" + "\x49\xa5\x3e\x87\xf4\xc3\xda\x55" + "\x82\xcd\x42\x28\x21\x20\x15\xcc" + "\xb7\xb2\x48\x40\xc7\x64\x41\x3a" + "\x61\x32\x82\x85\xcf\x27\xed\xb4" + "\xe4\x68\xa2\xf5\x79\x26\x27\xb2" + "\x51\x67\x6a\xc4\xf0\x66\x55\x50" + "\xbc\x6f\xed\xd5\x8d\xde\x23\x7c" + "\x62\x98\x14\xd7\x2f\x37\x8d\xdf" + "\xf4\x33\x80\xeb\x8e\xb4\xa4\xda", + .rlen = 64 + 64, + }, { /* RFC 3602 Case 5 */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x10" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x33\x44\x55\x66\x77\x88\x99\xaa" + "\xbb\xcc\xdd\xee\xff\x11\x22\x33" + "\x44\x55\x66\x77\x88\x99\xaa\xbb" + "\xcc\xdd\xee\xff\x11\x22\x33\x44" + "\x90\xd3\x82\xb4\x10\xee\xba\x7a" + "\xd9\x38\xc4\x6c\xec\x1a\x82\xbf", + .klen = 8 + 64 + 16, + .iv = "\xe9\x6e\x8c\x08\xab\x46\x57\x63" + "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\xe9\x6e\x8c\x08\xab\x46\x57\x63" + "\xfd\x09\x8d\x45\xdd\x3f\xf8\x93", + .alen = 24, + .input = "\x08\x00\x0e\xbd\xa7\x0a\x00\x00" + "\x8e\x9c\x08\x3d\xb9\x5b\x07\x00" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x01\x02\x03\x04\x05\x06\x07\x08" + "\x09\x0a\x0b\x0c\x0d\x0e\x0e\x01", + .ilen = 80, + .result = "\xf6\x63\xc2\x5d\x32\x5c\x18\xc6" + "\xa9\x45\x3e\x19\x4e\x12\x08\x49" + "\xa4\x87\x0b\x66\xcc\x6b\x99\x65" + "\x33\x00\x13\xb4\x89\x8d\xc8\x56" + "\xa4\x69\x9e\x52\x3a\x55\xdb\x08" + "\x0b\x59\xec\x3a\x8e\x4b\x7e\x52" + "\x77\x5b\x07\xd1\xdb\x34\xed\x9c" + "\x53\x8a\xb5\x0c\x55\x1b\x87\x4a" + "\xa2\x69\xad\xd0\x47\xad\x2d\x59" + "\x13\xac\x19\xb7\xcf\xba\xd4\xa6" + "\x74\x84\x94\xe2\xd7\x7a\xf9\xbf" + "\x00\x8a\xa2\xd5\xb7\xf3\x60\xcf" + "\xa0\x47\xdf\x4e\x09\xf4\xb1\x7f" + "\x14\xd9\x3d\x53\x8e\x12\xb3\x00" + "\x4c\x0a\x4e\x32\x40\x43\x88\xce" + "\x92\x26\xc1\x76\x20\x11\xeb\xba" + "\x62\x4f\x9a\x62\x25\xc3\x75\x80" + "\xb7\x0a\x17\xf5\xd7\x94\xb4\x14", + .rlen = 80 + 64, + }, { /* NIST SP800-38A F.2.3 CBC-AES192.Encrypt */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x18" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x33\x44\x55\x66\x77\x88\x99\xaa" + "\xbb\xcc\xdd\xee\xff\x11\x22\x33" + "\x44\x55\x66\x77\x88\x99\xaa\xbb" + "\xcc\xdd\xee\xff\x11\x22\x33\x44" + "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" + "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" + "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", + .klen = 8 + 64 + 24, + .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .alen = 16, + .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ilen = 64, + .result = "\x4f\x02\x1d\xb2\x43\xbc\x63\x3d" + "\x71\x78\x18\x3a\x9f\xa0\x71\xe8" + "\xb4\xd9\xad\xa9\xad\x7d\xed\xf4" + "\xe5\xe7\x38\x76\x3f\x69\x14\x5a" + "\x57\x1b\x24\x20\x12\xfb\x7a\xe0" + "\x7f\xa9\xba\xac\x3d\xf1\x02\xe0" + "\x08\xb0\xe2\x79\x88\x59\x88\x81" + "\xd9\x20\xa9\xe6\x4f\x56\x15\xcd" + "\x77\x4b\x69\x9d\x3a\x0d\xb4\x99" + "\x8f\xc6\x8e\x0e\x72\x58\xe3\x56" + "\xbb\x21\xd2\x7d\x93\x11\x17\x91" + "\xc4\x83\xfd\x0a\xea\x71\xfe\x77" + "\xae\x6f\x0a\xa5\xf0\xcf\xe1\x35" + "\xba\x03\xd5\x32\xfa\x5f\x41\x58" + "\x8d\x43\x98\xa7\x94\x16\x07\x02" + "\x0f\xb6\x81\x50\x28\x95\x2e\x75", + .rlen = 64 + 64, + }, { /* NIST SP800-38A F.2.5 CBC-AES256.Encrypt */ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x20" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x33\x44\x55\x66\x77\x88\x99\xaa" + "\xbb\xcc\xdd\xee\xff\x11\x22\x33" + "\x44\x55\x66\x77\x88\x99\xaa\xbb" + "\xcc\xdd\xee\xff\x11\x22\x33\x44" + "\x60\x3d\xeb\x10\x15\xca\x71\xbe" + "\x2b\x73\xae\xf0\x85\x7d\x77\x81" + "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" + "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", + .klen = 8 + 64 + 32, + .iv = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .alen = 16, + .input = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ilen = 64, + .result = "\xf5\x8c\x4c\x04\xd6\xe5\xf1\xba" + "\x77\x9e\xab\xfb\x5f\x7b\xfb\xd6" + "\x9c\xfc\x4e\x96\x7e\xdb\x80\x8d" + "\x67\x9f\x77\x7b\xc6\x70\x2c\x7d" + "\x39\xf2\x33\x69\xa9\xd9\xba\xcf" + "\xa5\x30\xe2\x63\x04\x23\x14\x61" + "\xb2\xeb\x05\xe2\xc3\x9b\xe9\xfc" + "\xda\x6c\x19\x07\x8c\x6a\x9d\x1b" + "\xb2\x27\x69\x7f\x45\x64\x79\x2b" + "\xb7\xb8\x4c\xd4\x75\x94\x68\x40" + "\x2a\xea\x91\xc7\x3f\x7c\xed\x7b" + "\x95\x2c\x9b\xa8\xf5\xe5\x52\x8d" + "\x6b\xe1\xae\xf1\x74\xfa\x0d\x0c" + "\xe3\x8d\x64\xc3\x8d\xff\x7c\x8c" + "\xdb\xbf\xa0\xb4\x01\xa2\xa8\xa2" + "\x2c\xb1\x62\x2c\x10\xca\xf1\x21", + .rlen = 64 + 64, + }, +}; + +static const struct aead_testvec hmac_sha1_des_cbc_enc_tv_temp[] = { + { /*Generated with cryptopp*/ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x08" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55" + "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24", + .klen = 8 + 20 + 8, + .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .alen = 16, + .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" + "\x53\x20\x63\x65\x65\x72\x73\x74" + "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" + "\x20\x79\x65\x53\x72\x63\x74\x65" + "\x20\x73\x6f\x54\x20\x6f\x61\x4d" + "\x79\x6e\x53\x20\x63\x65\x65\x72" + "\x73\x74\x54\x20\x6f\x6f\x4d\x20" + "\x6e\x61\x20\x79\x65\x53\x72\x63" + "\x74\x65\x20\x73\x6f\x54\x20\x6f" + "\x61\x4d\x79\x6e\x53\x20\x63\x65" + "\x65\x72\x73\x74\x54\x20\x6f\x6f" + "\x4d\x20\x6e\x61\x20\x79\x65\x53" + "\x72\x63\x74\x65\x20\x73\x6f\x54" + "\x20\x6f\x61\x4d\x79\x6e\x53\x20" + "\x63\x65\x65\x72\x73\x74\x54\x20" + "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", + .ilen = 128, + .result = "\x70\xd6\xde\x64\x87\x17\xf1\xe8" + "\x54\x31\x85\x37\xed\x6b\x01\x8d" + "\xe3\xcc\xe0\x1d\x5e\xf3\xfe\xf1" + "\x41\xaa\x33\x91\xa7\x7d\x99\x88" + "\x4d\x85\x6e\x2f\xa3\x69\xf5\x82" + "\x3a\x6f\x25\xcb\x7d\x58\x1f\x9b" + "\xaa\x9c\x11\xd5\x76\x67\xce\xde" + "\x56\xd7\x5a\x80\x69\xea\x3a\x02" + "\xf0\xc7\x7c\xe3\xcb\x40\xe5\x52" + "\xd1\x10\x92\x78\x0b\x8e\x5b\xf1" + "\xe3\x26\x1f\xe1\x15\x41\xc7\xba" + "\x99\xdb\x08\x51\x1c\xd3\x01\xf4" + "\x87\x47\x39\xb8\xd2\xdd\xbd\xfb" + "\x66\x13\xdf\x1c\x01\x44\xf0\x7a" + "\x1a\x6b\x13\xf5\xd5\x0b\xb8\xba" + "\x53\xba\xe1\x76\xe3\x82\x07\x86" + "\x95\x16\x20\x09\xf5\x95\x19\xfd" + "\x3c\xc7\xe0\x42\xc0\x14\x69\xfa" + "\x5c\x44\xa9\x37", + .rlen = 128 + 20, + }, +}; + +static const struct aead_testvec hmac_sha224_des_cbc_enc_tv_temp[] = { + { /*Generated with cryptopp*/ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x08" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24", + .klen = 8 + 24 + 8, + .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .alen = 16, + .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" + "\x53\x20\x63\x65\x65\x72\x73\x74" + "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" + "\x20\x79\x65\x53\x72\x63\x74\x65" + "\x20\x73\x6f\x54\x20\x6f\x61\x4d" + "\x79\x6e\x53\x20\x63\x65\x65\x72" + "\x73\x74\x54\x20\x6f\x6f\x4d\x20" + "\x6e\x61\x20\x79\x65\x53\x72\x63" + "\x74\x65\x20\x73\x6f\x54\x20\x6f" + "\x61\x4d\x79\x6e\x53\x20\x63\x65" + "\x65\x72\x73\x74\x54\x20\x6f\x6f" + "\x4d\x20\x6e\x61\x20\x79\x65\x53" + "\x72\x63\x74\x65\x20\x73\x6f\x54" + "\x20\x6f\x61\x4d\x79\x6e\x53\x20" + "\x63\x65\x65\x72\x73\x74\x54\x20" + "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", + .ilen = 128, + .result = "\x70\xd6\xde\x64\x87\x17\xf1\xe8" + "\x54\x31\x85\x37\xed\x6b\x01\x8d" + "\xe3\xcc\xe0\x1d\x5e\xf3\xfe\xf1" + "\x41\xaa\x33\x91\xa7\x7d\x99\x88" + "\x4d\x85\x6e\x2f\xa3\x69\xf5\x82" + "\x3a\x6f\x25\xcb\x7d\x58\x1f\x9b" + "\xaa\x9c\x11\xd5\x76\x67\xce\xde" + "\x56\xd7\x5a\x80\x69\xea\x3a\x02" + "\xf0\xc7\x7c\xe3\xcb\x40\xe5\x52" + "\xd1\x10\x92\x78\x0b\x8e\x5b\xf1" + "\xe3\x26\x1f\xe1\x15\x41\xc7\xba" + "\x99\xdb\x08\x51\x1c\xd3\x01\xf4" + "\x87\x47\x39\xb8\xd2\xdd\xbd\xfb" + "\x66\x13\xdf\x1c\x01\x44\xf0\x7a" + "\x1a\x6b\x13\xf5\xd5\x0b\xb8\xba" + "\x53\xba\xe1\x76\xe3\x82\x07\x86" + "\x9c\x2d\x7e\xee\x20\x34\x55\x0a" + "\xce\xb5\x4e\x64\x53\xe7\xbf\x91" + "\xab\xd4\xd9\xda\xc9\x12\xae\xf7", + .rlen = 128 + 24, + }, +}; + +static const struct aead_testvec hmac_sha256_des_cbc_enc_tv_temp[] = { + { /*Generated with cryptopp*/ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x08" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24", + .klen = 8 + 32 + 8, + .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .alen = 16, + .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" + "\x53\x20\x63\x65\x65\x72\x73\x74" + "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" + "\x20\x79\x65\x53\x72\x63\x74\x65" + "\x20\x73\x6f\x54\x20\x6f\x61\x4d" + "\x79\x6e\x53\x20\x63\x65\x65\x72" + "\x73\x74\x54\x20\x6f\x6f\x4d\x20" + "\x6e\x61\x20\x79\x65\x53\x72\x63" + "\x74\x65\x20\x73\x6f\x54\x20\x6f" + "\x61\x4d\x79\x6e\x53\x20\x63\x65" + "\x65\x72\x73\x74\x54\x20\x6f\x6f" + "\x4d\x20\x6e\x61\x20\x79\x65\x53" + "\x72\x63\x74\x65\x20\x73\x6f\x54" + "\x20\x6f\x61\x4d\x79\x6e\x53\x20" + "\x63\x65\x65\x72\x73\x74\x54\x20" + "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", + .ilen = 128, + .result = "\x70\xd6\xde\x64\x87\x17\xf1\xe8" + "\x54\x31\x85\x37\xed\x6b\x01\x8d" + "\xe3\xcc\xe0\x1d\x5e\xf3\xfe\xf1" + "\x41\xaa\x33\x91\xa7\x7d\x99\x88" + "\x4d\x85\x6e\x2f\xa3\x69\xf5\x82" + "\x3a\x6f\x25\xcb\x7d\x58\x1f\x9b" + "\xaa\x9c\x11\xd5\x76\x67\xce\xde" + "\x56\xd7\x5a\x80\x69\xea\x3a\x02" + "\xf0\xc7\x7c\xe3\xcb\x40\xe5\x52" + "\xd1\x10\x92\x78\x0b\x8e\x5b\xf1" + "\xe3\x26\x1f\xe1\x15\x41\xc7\xba" + "\x99\xdb\x08\x51\x1c\xd3\x01\xf4" + "\x87\x47\x39\xb8\xd2\xdd\xbd\xfb" + "\x66\x13\xdf\x1c\x01\x44\xf0\x7a" + "\x1a\x6b\x13\xf5\xd5\x0b\xb8\xba" + "\x53\xba\xe1\x76\xe3\x82\x07\x86" + "\xc6\x58\xa1\x60\x70\x91\x39\x36" + "\x50\xf6\x5d\xab\x4b\x51\x4e\x5e" + "\xde\x63\xde\x76\x52\xde\x9f\xba" + "\x90\xcf\x15\xf2\xbb\x6e\x84\x00", + .rlen = 128 + 32, + }, +}; + +static const struct aead_testvec hmac_sha384_des_cbc_enc_tv_temp[] = { + { /*Generated with cryptopp*/ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x08" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x33\x44\x55\x66\x77\x88\x99\xaa" + "\xbb\xcc\xdd\xee\xff\x11\x22\x33" + "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24", + .klen = 8 + 48 + 8, + .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .alen = 16, + .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" + "\x53\x20\x63\x65\x65\x72\x73\x74" + "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" + "\x20\x79\x65\x53\x72\x63\x74\x65" + "\x20\x73\x6f\x54\x20\x6f\x61\x4d" + "\x79\x6e\x53\x20\x63\x65\x65\x72" + "\x73\x74\x54\x20\x6f\x6f\x4d\x20" + "\x6e\x61\x20\x79\x65\x53\x72\x63" + "\x74\x65\x20\x73\x6f\x54\x20\x6f" + "\x61\x4d\x79\x6e\x53\x20\x63\x65" + "\x65\x72\x73\x74\x54\x20\x6f\x6f" + "\x4d\x20\x6e\x61\x20\x79\x65\x53" + "\x72\x63\x74\x65\x20\x73\x6f\x54" + "\x20\x6f\x61\x4d\x79\x6e\x53\x20" + "\x63\x65\x65\x72\x73\x74\x54\x20" + "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", + .ilen = 128, + .result = "\x70\xd6\xde\x64\x87\x17\xf1\xe8" + "\x54\x31\x85\x37\xed\x6b\x01\x8d" + "\xe3\xcc\xe0\x1d\x5e\xf3\xfe\xf1" + "\x41\xaa\x33\x91\xa7\x7d\x99\x88" + "\x4d\x85\x6e\x2f\xa3\x69\xf5\x82" + "\x3a\x6f\x25\xcb\x7d\x58\x1f\x9b" + "\xaa\x9c\x11\xd5\x76\x67\xce\xde" + "\x56\xd7\x5a\x80\x69\xea\x3a\x02" + "\xf0\xc7\x7c\xe3\xcb\x40\xe5\x52" + "\xd1\x10\x92\x78\x0b\x8e\x5b\xf1" + "\xe3\x26\x1f\xe1\x15\x41\xc7\xba" + "\x99\xdb\x08\x51\x1c\xd3\x01\xf4" + "\x87\x47\x39\xb8\xd2\xdd\xbd\xfb" + "\x66\x13\xdf\x1c\x01\x44\xf0\x7a" + "\x1a\x6b\x13\xf5\xd5\x0b\xb8\xba" + "\x53\xba\xe1\x76\xe3\x82\x07\x86" + "\xa8\x8e\x9c\x74\x8c\x2b\x99\xa0" + "\xc8\x8c\xef\x25\x07\x83\x11\x3a" + "\x31\x8d\xbe\x3b\x6a\xd7\x96\xfe" + "\x5e\x67\xb5\x74\xe7\xe7\x85\x61" + "\x6a\x95\x26\x75\xcc\x53\x89\xf3" + "\x74\xc9\x2a\x76\x20\xa2\x64\x62", + .rlen = 128 + 48, + }, +}; + +static const struct aead_testvec hmac_sha512_des_cbc_enc_tv_temp[] = { + { /*Generated with cryptopp*/ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x08" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x33\x44\x55\x66\x77\x88\x99\xaa" + "\xbb\xcc\xdd\xee\xff\x11\x22\x33" + "\x44\x55\x66\x77\x88\x99\xaa\xbb" + "\xcc\xdd\xee\xff\x11\x22\x33\x44" + "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24", + .klen = 8 + 64 + 8, + .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .alen = 16, + .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" + "\x53\x20\x63\x65\x65\x72\x73\x74" + "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" + "\x20\x79\x65\x53\x72\x63\x74\x65" + "\x20\x73\x6f\x54\x20\x6f\x61\x4d" + "\x79\x6e\x53\x20\x63\x65\x65\x72" + "\x73\x74\x54\x20\x6f\x6f\x4d\x20" + "\x6e\x61\x20\x79\x65\x53\x72\x63" + "\x74\x65\x20\x73\x6f\x54\x20\x6f" + "\x61\x4d\x79\x6e\x53\x20\x63\x65" + "\x65\x72\x73\x74\x54\x20\x6f\x6f" + "\x4d\x20\x6e\x61\x20\x79\x65\x53" + "\x72\x63\x74\x65\x20\x73\x6f\x54" + "\x20\x6f\x61\x4d\x79\x6e\x53\x20" + "\x63\x65\x65\x72\x73\x74\x54\x20" + "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", + .ilen = 128, + .result = "\x70\xd6\xde\x64\x87\x17\xf1\xe8" + "\x54\x31\x85\x37\xed\x6b\x01\x8d" + "\xe3\xcc\xe0\x1d\x5e\xf3\xfe\xf1" + "\x41\xaa\x33\x91\xa7\x7d\x99\x88" + "\x4d\x85\x6e\x2f\xa3\x69\xf5\x82" + "\x3a\x6f\x25\xcb\x7d\x58\x1f\x9b" + "\xaa\x9c\x11\xd5\x76\x67\xce\xde" + "\x56\xd7\x5a\x80\x69\xea\x3a\x02" + "\xf0\xc7\x7c\xe3\xcb\x40\xe5\x52" + "\xd1\x10\x92\x78\x0b\x8e\x5b\xf1" + "\xe3\x26\x1f\xe1\x15\x41\xc7\xba" + "\x99\xdb\x08\x51\x1c\xd3\x01\xf4" + "\x87\x47\x39\xb8\xd2\xdd\xbd\xfb" + "\x66\x13\xdf\x1c\x01\x44\xf0\x7a" + "\x1a\x6b\x13\xf5\xd5\x0b\xb8\xba" + "\x53\xba\xe1\x76\xe3\x82\x07\x86" + "\xc6\x2c\x73\x88\xb0\x9d\x5f\x3e" + "\x5b\x78\xca\x0e\xab\x8a\xa3\xbb" + "\xd9\x1d\xc3\xe3\x05\xac\x76\xfb" + "\x58\x83\xda\x67\xfb\x21\x24\xa2" + "\xb1\xa7\xd7\x66\xa6\x8d\xa6\x93" + "\x97\xe2\xe3\xb8\xaa\x48\x85\xee" + "\x8c\xf6\x07\x95\x1f\xa6\x6c\x96" + "\x99\xc7\x5c\x8d\xd8\xb5\x68\x7b", + .rlen = 128 + 64, + }, +}; + +static const struct aead_testvec hmac_sha1_des3_ede_cbc_enc_tv_temp[] = { + { /*Generated with cryptopp*/ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x18" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55" + "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" + "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" + "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", + .klen = 8 + 20 + 24, + .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .alen = 16, + .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" + "\x53\x20\x63\x65\x65\x72\x73\x74" + "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" + "\x20\x79\x65\x53\x72\x63\x74\x65" + "\x20\x73\x6f\x54\x20\x6f\x61\x4d" + "\x79\x6e\x53\x20\x63\x65\x65\x72" + "\x73\x74\x54\x20\x6f\x6f\x4d\x20" + "\x6e\x61\x20\x79\x65\x53\x72\x63" + "\x74\x65\x20\x73\x6f\x54\x20\x6f" + "\x61\x4d\x79\x6e\x53\x20\x63\x65" + "\x65\x72\x73\x74\x54\x20\x6f\x6f" + "\x4d\x20\x6e\x61\x20\x79\x65\x53" + "\x72\x63\x74\x65\x20\x73\x6f\x54" + "\x20\x6f\x61\x4d\x79\x6e\x53\x20" + "\x63\x65\x65\x72\x73\x74\x54\x20" + "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", + .ilen = 128, + .result = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" + "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" + "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" + "\x12\x56\x5c\x53\x96\xb6\x00\x7d" + "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" + "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" + "\x76\xd1\xda\x0c\x94\x67\xbb\x04" + "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" + "\x22\x64\x47\xaa\x8f\x75\x13\xbf" + "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" + "\x71\x63\x2e\x89\x7b\x1e\x12\xca" + "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" + "\xd6\xf9\x21\x31\x62\x44\x45\xa6" + "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" + "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" + "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19" + "\x67\x6d\xb1\xf5\xb8\x10\xdc\xc6" + "\x75\x86\x96\x6b\xb1\xc5\xe4\xcf" + "\xd1\x60\x91\xb3", + .rlen = 128 + 20, + }, +}; + +static const struct aead_testvec hmac_sha224_des3_ede_cbc_enc_tv_temp[] = { + { /*Generated with cryptopp*/ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x18" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" + "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" + "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", + .klen = 8 + 24 + 24, + .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .alen = 16, + .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" + "\x53\x20\x63\x65\x65\x72\x73\x74" + "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" + "\x20\x79\x65\x53\x72\x63\x74\x65" + "\x20\x73\x6f\x54\x20\x6f\x61\x4d" + "\x79\x6e\x53\x20\x63\x65\x65\x72" + "\x73\x74\x54\x20\x6f\x6f\x4d\x20" + "\x6e\x61\x20\x79\x65\x53\x72\x63" + "\x74\x65\x20\x73\x6f\x54\x20\x6f" + "\x61\x4d\x79\x6e\x53\x20\x63\x65" + "\x65\x72\x73\x74\x54\x20\x6f\x6f" + "\x4d\x20\x6e\x61\x20\x79\x65\x53" + "\x72\x63\x74\x65\x20\x73\x6f\x54" + "\x20\x6f\x61\x4d\x79\x6e\x53\x20" + "\x63\x65\x65\x72\x73\x74\x54\x20" + "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", + .ilen = 128, + .result = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" + "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" + "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" + "\x12\x56\x5c\x53\x96\xb6\x00\x7d" + "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" + "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" + "\x76\xd1\xda\x0c\x94\x67\xbb\x04" + "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" + "\x22\x64\x47\xaa\x8f\x75\x13\xbf" + "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" + "\x71\x63\x2e\x89\x7b\x1e\x12\xca" + "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" + "\xd6\xf9\x21\x31\x62\x44\x45\xa6" + "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" + "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" + "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19" + "\x15\x24\x7f\x5a\x45\x4a\x66\xce" + "\x2b\x0b\x93\x99\x2f\x9d\x0c\x6c" + "\x56\x1f\xe1\xa6\x41\xb2\x4c\xd0", + .rlen = 128 + 24, + }, +}; + +static const struct aead_testvec hmac_sha256_des3_ede_cbc_enc_tv_temp[] = { + { /*Generated with cryptopp*/ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x18" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" + "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" + "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", + .klen = 8 + 32 + 24, + .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .alen = 16, + .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" + "\x53\x20\x63\x65\x65\x72\x73\x74" + "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" + "\x20\x79\x65\x53\x72\x63\x74\x65" + "\x20\x73\x6f\x54\x20\x6f\x61\x4d" + "\x79\x6e\x53\x20\x63\x65\x65\x72" + "\x73\x74\x54\x20\x6f\x6f\x4d\x20" + "\x6e\x61\x20\x79\x65\x53\x72\x63" + "\x74\x65\x20\x73\x6f\x54\x20\x6f" + "\x61\x4d\x79\x6e\x53\x20\x63\x65" + "\x65\x72\x73\x74\x54\x20\x6f\x6f" + "\x4d\x20\x6e\x61\x20\x79\x65\x53" + "\x72\x63\x74\x65\x20\x73\x6f\x54" + "\x20\x6f\x61\x4d\x79\x6e\x53\x20" + "\x63\x65\x65\x72\x73\x74\x54\x20" + "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", + .ilen = 128, + .result = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" + "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" + "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" + "\x12\x56\x5c\x53\x96\xb6\x00\x7d" + "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" + "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" + "\x76\xd1\xda\x0c\x94\x67\xbb\x04" + "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" + "\x22\x64\x47\xaa\x8f\x75\x13\xbf" + "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" + "\x71\x63\x2e\x89\x7b\x1e\x12\xca" + "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" + "\xd6\xf9\x21\x31\x62\x44\x45\xa6" + "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" + "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" + "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19" + "\x73\xb0\xea\x9f\xe8\x18\x80\xd6" + "\x56\x38\x44\xc0\xdb\xe3\x4f\x71" + "\xf7\xce\xd1\xd3\xf8\xbd\x3e\x4f" + "\xca\x43\x95\xdf\x80\x61\x81\xa9", + .rlen = 128 + 32, + }, +}; + +static const struct aead_testvec hmac_sha384_des3_ede_cbc_enc_tv_temp[] = { + { /*Generated with cryptopp*/ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x18" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x33\x44\x55\x66\x77\x88\x99\xaa" + "\xbb\xcc\xdd\xee\xff\x11\x22\x33" + "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" + "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" + "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", + .klen = 8 + 48 + 24, + .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .alen = 16, + .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" + "\x53\x20\x63\x65\x65\x72\x73\x74" + "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" + "\x20\x79\x65\x53\x72\x63\x74\x65" + "\x20\x73\x6f\x54\x20\x6f\x61\x4d" + "\x79\x6e\x53\x20\x63\x65\x65\x72" + "\x73\x74\x54\x20\x6f\x6f\x4d\x20" + "\x6e\x61\x20\x79\x65\x53\x72\x63" + "\x74\x65\x20\x73\x6f\x54\x20\x6f" + "\x61\x4d\x79\x6e\x53\x20\x63\x65" + "\x65\x72\x73\x74\x54\x20\x6f\x6f" + "\x4d\x20\x6e\x61\x20\x79\x65\x53" + "\x72\x63\x74\x65\x20\x73\x6f\x54" + "\x20\x6f\x61\x4d\x79\x6e\x53\x20" + "\x63\x65\x65\x72\x73\x74\x54\x20" + "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", + .ilen = 128, + .result = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" + "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" + "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" + "\x12\x56\x5c\x53\x96\xb6\x00\x7d" + "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" + "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" + "\x76\xd1\xda\x0c\x94\x67\xbb\x04" + "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" + "\x22\x64\x47\xaa\x8f\x75\x13\xbf" + "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" + "\x71\x63\x2e\x89\x7b\x1e\x12\xca" + "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" + "\xd6\xf9\x21\x31\x62\x44\x45\xa6" + "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" + "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" + "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19" + "\x6d\x77\xfc\x80\x9d\x8a\x9c\xb7" + "\x70\xe7\x93\xbf\x73\xe6\x9f\x83" + "\x99\x62\x23\xe6\x5b\xd0\xda\x18" + "\xa4\x32\x8a\x0b\x46\xd7\xf0\x39" + "\x36\x5d\x13\x2f\x86\x10\x78\xd6" + "\xd6\xbe\x5c\xb9\x15\x89\xf9\x1b", + .rlen = 128 + 48, + }, +}; + +static const struct aead_testvec hmac_sha512_des3_ede_cbc_enc_tv_temp[] = { + { /*Generated with cryptopp*/ +#ifdef __LITTLE_ENDIAN + .key = "\x08\x00" /* rta length */ + "\x01\x00" /* rta type */ +#else + .key = "\x00\x08" /* rta length */ + "\x00\x01" /* rta type */ +#endif + "\x00\x00\x00\x18" /* enc key length */ + "\x11\x22\x33\x44\x55\x66\x77\x88" + "\x99\xaa\xbb\xcc\xdd\xee\xff\x11" + "\x22\x33\x44\x55\x66\x77\x88\x99" + "\xaa\xbb\xcc\xdd\xee\xff\x11\x22" + "\x33\x44\x55\x66\x77\x88\x99\xaa" + "\xbb\xcc\xdd\xee\xff\x11\x22\x33" + "\x44\x55\x66\x77\x88\x99\xaa\xbb" + "\xcc\xdd\xee\xff\x11\x22\x33\x44" + "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24" + "\x44\x4D\x99\x5A\x12\xD6\x40\xC0" + "\xEA\xC2\x84\xE8\x14\x95\xDB\xE8", + .klen = 8 + 64 + 24, + .iv = "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x01" + "\x7D\x33\x88\x93\x0F\x93\xB2\x42", + .alen = 16, + .input = "\x6f\x54\x20\x6f\x61\x4d\x79\x6e" + "\x53\x20\x63\x65\x65\x72\x73\x74" + "\x54\x20\x6f\x6f\x4d\x20\x6e\x61" + "\x20\x79\x65\x53\x72\x63\x74\x65" + "\x20\x73\x6f\x54\x20\x6f\x61\x4d" + "\x79\x6e\x53\x20\x63\x65\x65\x72" + "\x73\x74\x54\x20\x6f\x6f\x4d\x20" + "\x6e\x61\x20\x79\x65\x53\x72\x63" + "\x74\x65\x20\x73\x6f\x54\x20\x6f" + "\x61\x4d\x79\x6e\x53\x20\x63\x65" + "\x65\x72\x73\x74\x54\x20\x6f\x6f" + "\x4d\x20\x6e\x61\x20\x79\x65\x53" + "\x72\x63\x74\x65\x20\x73\x6f\x54" + "\x20\x6f\x61\x4d\x79\x6e\x53\x20" + "\x63\x65\x65\x72\x73\x74\x54\x20" + "\x6f\x6f\x4d\x20\x6e\x61\x0a\x79", + .ilen = 128, + .result = "\x0e\x2d\xb6\x97\x3c\x56\x33\xf4" + "\x67\x17\x21\xc7\x6e\x8a\xd5\x49" + "\x74\xb3\x49\x05\xc5\x1c\xd0\xed" + "\x12\x56\x5c\x53\x96\xb6\x00\x7d" + "\x90\x48\xfc\xf5\x8d\x29\x39\xcc" + "\x8a\xd5\x35\x18\x36\x23\x4e\xd7" + "\x76\xd1\xda\x0c\x94\x67\xbb\x04" + "\x8b\xf2\x03\x6c\xa8\xcf\xb6\xea" + "\x22\x64\x47\xaa\x8f\x75\x13\xbf" + "\x9f\xc2\xc3\xf0\xc9\x56\xc5\x7a" + "\x71\x63\x2e\x89\x7b\x1e\x12\xca" + "\xe2\x5f\xaf\xd8\xa4\xf8\xc9\x7a" + "\xd6\xf9\x21\x31\x62\x44\x45\xa6" + "\xd6\xbc\x5a\xd3\x2d\x54\x43\xcc" + "\x9d\xde\xa5\x70\xe9\x42\x45\x8a" + "\x6b\xfa\xb1\x91\x13\xb0\xd9\x19" + "\x41\xb5\x1f\xbb\xbd\x4e\xb8\x32" + "\x22\x86\x4e\x57\x1b\x2a\xd8\x6e" + "\xa9\xfb\xc8\xf3\xbf\x2d\xae\x2b" + "\x3b\xbc\x41\xe8\x38\xbb\xf1\x60" + "\x4c\x68\xa9\x4e\x8c\x73\xa7\xc0" + "\x2a\x74\xd4\x65\x12\xcb\x55\xf2" + "\xd5\x02\x6d\xe6\xaf\xc9\x2f\xf2" + "\x57\xaa\x85\xf7\xf3\x6a\xcb\xdb", + .rlen = 128 + 64, + }, +}; + +static const struct cipher_testvec aes_lrw_tv_template[] = { + /* from http://grouper.ieee.org/groups/1619/email/pdf00017.pdf */ + { /* LRW-32-AES 1 */ + .key = "\x45\x62\xac\x25\xf8\x28\x17\x6d" + "\x4c\x26\x84\x14\xb5\x68\x01\x85" + "\x25\x8e\x2a\x05\xe7\x3e\x9d\x03" + "\xee\x5a\x83\x0c\xcc\x09\x4c\x87", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\xf1\xb2\x73\xcd\x65\xa3\xdf\x5f" + "\xe9\x5d\x48\x92\x54\x63\x4e\xb8", + .len = 16, + }, { /* LRW-32-AES 2 */ + .key = "\x59\x70\x47\x14\xf5\x57\x47\x8c" + "\xd7\x79\xe8\x0f\x54\x88\x79\x44" + "\x0d\x48\xf0\xb7\xb1\x5a\x53\xea" + "\x1c\xaa\x6b\x29\xc2\xca\xfb\xaf", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x02", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x00\xc8\x2b\xae\x95\xbb\xcd\xe5" + "\x27\x4f\x07\x69\xb2\x60\xe1\x36", + .len = 16, + }, { /* LRW-32-AES 3 */ + .key = "\xd8\x2a\x91\x34\xb2\x6a\x56\x50" + "\x30\xfe\x69\xe2\x37\x7f\x98\x47" + "\xcd\xf9\x0b\x16\x0c\x64\x8f\xb6" + "\xb0\x0d\x0d\x1b\xae\x85\x87\x1f", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x02\x00\x00\x00\x00", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x76\x32\x21\x83\xed\x8f\xf1\x82" + "\xf9\x59\x62\x03\x69\x0e\x5e\x01", + .len = 16, + }, { /* LRW-32-AES 4 */ + .key = "\x0f\x6a\xef\xf8\xd3\xd2\xbb\x15" + "\x25\x83\xf7\x3c\x1f\x01\x28\x74" + "\xca\xc6\xbc\x35\x4d\x4a\x65\x54" + "\x90\xae\x61\xcf\x7b\xae\xbd\xcc" + "\xad\xe4\x94\xc5\x4a\x29\xae\x70", + .klen = 40, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x9c\x0f\x15\x2f\x55\xa2\xd8\xf0" + "\xd6\x7b\x8f\x9e\x28\x22\xbc\x41", + .len = 16, + }, { /* LRW-32-AES 5 */ + .key = "\x8a\xd4\xee\x10\x2f\xbd\x81\xff" + "\xf8\x86\xce\xac\x93\xc5\xad\xc6" + "\xa0\x19\x07\xc0\x9d\xf7\xbb\xdd" + "\x52\x13\xb2\xb7\xf0\xff\x11\xd8" + "\xd6\x08\xd0\xcd\x2e\xb1\x17\x6f", + .klen = 40, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x02\x00\x00\x00\x00", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\xd4\x27\x6a\x7f\x14\x91\x3d\x65" + "\xc8\x60\x48\x02\x87\xe3\x34\x06", + .len = 16, + }, { /* LRW-32-AES 6 */ + .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" + "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" + "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" + "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" + "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" + "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\xbd\x06\xb8\xe1\xdb\x98\x89\x9e" + "\xc4\x98\xe4\x91\xcf\x1c\x70\x2b", + .len = 16, + }, { /* LRW-32-AES 7 */ + .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" + "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" + "\xb2\xfb\x64\xce\x60\x97\x87\x8d" + "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" + "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" + "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x02\x00\x00\x00\x00", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x5b\x90\x8e\xc1\xab\xdd\x67\x5f" + "\x3d\x69\x8a\x95\x53\xc8\x9c\xe5", + .len = 16, + }, { +/* http://www.mail-archive.com/stds-p1619@listserv.ieee.org/msg00173.html */ + .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" + "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" + "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" + "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" + "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" + "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" + "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" + "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" + "\x50\x38\x1f\x71\x49\xb6\x57\xd6" + "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" + "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" + "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" + "\xda\x10\x8e\xed\xa2\xa4\x87\xab" + "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" + "\xc9\xac\x42\x31\x95\x7c\xc9\x04" + "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" + "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" + "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" + "\x4c\x96\x12\xed\x7c\x92\x03\x01" + "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" + "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" + "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" + "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" + "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" + "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" + "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" + "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" + "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" + "\x76\x12\x73\x44\x1a\x56\xd7\x72" + "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" + "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" + "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" + "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" + "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" + "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" + "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" + "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" + "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" + "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" + "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" + "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" + "\x8d\x23\x31\x74\x84\xeb\x88\x6e" + "\xcc\xb9\xbc\x22\x83\x19\x07\x22" + "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" + "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" + "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" + "\x3c\xce\x8f\x42\x60\x71\xa7\x75" + "\x08\x40\x65\x8a\x82\xbf\xf5\x43" + "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" + "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" + "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" + "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" + "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" + "\x62\x73\x65\xfd\x46\x63\x25\x3d" + "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" + "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" + "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" + "\xc5\x68\x77\x84\x32\x2b\xcc\x85" + "\x74\x96\xf0\x12\x77\x61\xb9\xeb" + "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" + "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" + "\xda\x39\x87\x45\xc0\x2b\xbb\x01" + "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" + "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" + "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" + "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" + "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" + "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" + "\x21\xc4\xc2\x75\x67\x89\x37\x0a", + .ctext = "\x1a\x1d\xa9\x30\xad\xf9\x2f\x9b" + "\xb6\x1d\xae\xef\xf0\x2f\xf8\x5a" + "\x39\x3c\xbf\x2a\xb2\x45\xb2\x23" + "\x1b\x63\x3c\xcf\xaa\xbe\xcf\x4e" + "\xfa\xe8\x29\xc2\x20\x68\x2b\x3c" + "\x2e\x8b\xf7\x6e\x25\xbd\xe3\x3d" + "\x66\x27\xd6\xaf\xd6\x64\x3e\xe3" + "\xe8\x58\x46\x97\x39\x51\x07\xde" + "\xcb\x37\xbc\xa9\xc0\x5f\x75\xc3" + "\x0e\x84\x23\x1d\x16\xd4\x1c\x59" + "\x9c\x1a\x02\x55\xab\x3a\x97\x1d" + "\xdf\xdd\xc7\x06\x51\xd7\x70\xae" + "\x23\xc6\x8c\xf5\x1e\xa0\xe5\x82" + "\xb8\xb2\xbf\x04\xa0\x32\x8e\x68" + "\xeb\xaf\x6e\x2d\x94\x22\x2f\xce" + "\x4c\xb5\x59\xe2\xa2\x2f\xa0\x98" + "\x1a\x97\xc6\xd4\xb5\x00\x59\xf2" + "\x84\x14\x72\xb1\x9a\x6e\xa3\x7f" + "\xea\x20\xe7\xcb\x65\x77\x3a\xdf" + "\xc8\x97\x67\x15\xc2\x2a\x27\xcc" + "\x18\x55\xa1\x24\x0b\x24\x24\xaf" + "\x5b\xec\x68\xb8\xc8\xf5\xba\x63" + "\xff\xed\x89\xce\xd5\x3d\x88\xf3" + "\x25\xef\x05\x7c\x3a\xef\xeb\xd8" + "\x7a\x32\x0d\xd1\x1e\x58\x59\x99" + "\x90\x25\xb5\x26\xb0\xe3\x2b\x6c" + "\x4c\xa9\x8b\x84\x4f\x5e\x01\x50" + "\x41\x30\x58\xc5\x62\x74\x52\x1d" + "\x45\x24\x6a\x42\x64\x4f\x97\x1c" + "\xa8\x66\xb5\x6d\x79\xd4\x0d\x48" + "\xc5\x5f\xf3\x90\x32\xdd\xdd\xe1" + "\xe4\xa9\x9f\xfc\xc3\x52\x5a\x46" + "\xe4\x81\x84\x95\x36\x59\x7a\x6b" + "\xaa\xb3\x60\xad\xce\x9f\x9f\x28" + "\xe0\x01\x75\x22\xc4\x4e\xa9\x62" + "\x5c\x62\x0d\x00\xcb\x13\xe8\x43" + "\x72\xd4\x2d\x53\x46\xb5\xd1\x16" + "\x22\x18\xdf\x34\x33\xf5\xd6\x1c" + "\xb8\x79\x78\x97\x94\xff\x72\x13" + "\x4c\x27\xfc\xcb\xbf\x01\x53\xa6" + "\xb4\x50\x6e\xde\xdf\xb5\x43\xa4" + "\x59\xdf\x52\xf9\x7c\xe0\x11\x6f" + "\x2d\x14\x8e\x24\x61\x2c\xe1\x17" + "\xcc\xce\x51\x0c\x19\x8a\x82\x30" + "\x94\xd5\x3d\x6a\x53\x06\x5e\xbd" + "\xb7\xeb\xfa\xfd\x27\x51\xde\x85" + "\x1e\x86\x53\x11\x53\x94\x00\xee" + "\x2b\x8c\x08\x2a\xbf\xdd\xae\x11" + "\xcb\x1e\xa2\x07\x9a\x80\xcf\x62" + "\x9b\x09\xdc\x95\x3c\x96\x8e\xb1" + "\x09\xbd\xe4\xeb\xdb\xca\x70\x7a" + "\x9e\xfa\x31\x18\x45\x3c\x21\x33" + "\xb0\xb3\x2b\xea\xf3\x71\x2d\xe1" + "\x03\xad\x1b\x48\xd4\x67\x27\xf0" + "\x62\xe4\x3d\xfb\x9b\x08\x76\xe7" + "\xdd\x2b\x01\x39\x04\x5a\x58\x7a" + "\xf7\x11\x90\xec\xbd\x51\x5c\x32" + "\x6b\xd7\x35\x39\x02\x6b\xf2\xa6" + "\xd0\x0d\x07\xe1\x06\xc4\x5b\x7d" + "\xe4\x6a\xd7\xee\x15\x1f\x83\xb4" + "\xa3\xa7\x5e\xc3\x90\xb7\xef\xd3" + "\xb7\x4f\xf8\x92\x4c\xb7\x3c\x29" + "\xcd\x7e\x2b\x5d\x43\xea\x42\xe7" + "\x74\x3f\x7d\x58\x88\x75\xde\x3e", + .len = 512, + .also_non_np = 1, + .np = 3, + .tap = { 512 - 20, 4, 16 }, + } +}; + +static const struct cipher_testvec aes_xts_tv_template[] = { + /* http://grouper.ieee.org/groups/1619/email/pdf00086.pdf */ + { /* XTS-AES 1 */ + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 32, + .fips_skip = 1, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\x91\x7c\xf6\x9e\xbd\x68\xb2\xec" + "\x9b\x9f\xe9\xa3\xea\xdd\xa6\x92" + "\xcd\x43\xd2\xf5\x95\x98\xed\x85" + "\x8c\x02\xc2\x65\x2f\xbf\x92\x2e", + .len = 32, + }, { /* XTS-AES 2 */ + .key = "\x11\x11\x11\x11\x11\x11\x11\x11" + "\x11\x11\x11\x11\x11\x11\x11\x11" + "\x22\x22\x22\x22\x22\x22\x22\x22" + "\x22\x22\x22\x22\x22\x22\x22\x22", + .klen = 32, + .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44", + .ctext = "\xc4\x54\x18\x5e\x6a\x16\x93\x6e" + "\x39\x33\x40\x38\xac\xef\x83\x8b" + "\xfb\x18\x6f\xff\x74\x80\xad\xc4" + "\x28\x93\x82\xec\xd6\xd3\x94\xf0", + .len = 32, + }, { /* XTS-AES 3 */ + .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" + "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" + "\x22\x22\x22\x22\x22\x22\x22\x22" + "\x22\x22\x22\x22\x22\x22\x22\x22", + .klen = 32, + .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44", + .ctext = "\xaf\x85\x33\x6b\x59\x7a\xfc\x1a" + "\x90\x0b\x2e\xb2\x1e\xc9\x49\xd2" + "\x92\xdf\x4c\x04\x7e\x0b\x21\x53" + "\x21\x86\xa5\x97\x1a\x22\x7a\x89", + .len = 32, + }, { /* XTS-AES 4 */ + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x31\x41\x59\x26\x53\x58\x97\x93" + "\x23\x84\x62\x64\x33\x83\x27\x95", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ctext = "\x27\xa7\x47\x9b\xef\xa1\xd4\x76" + "\x48\x9f\x30\x8c\xd4\xcf\xa6\xe2" + "\xa9\x6e\x4b\xbe\x32\x08\xff\x25" + "\x28\x7d\xd3\x81\x96\x16\xe8\x9c" + "\xc7\x8c\xf7\xf5\xe5\x43\x44\x5f" + "\x83\x33\xd8\xfa\x7f\x56\x00\x00" + "\x05\x27\x9f\xa5\xd8\xb5\xe4\xad" + "\x40\xe7\x36\xdd\xb4\xd3\x54\x12" + "\x32\x80\x63\xfd\x2a\xab\x53\xe5" + "\xea\x1e\x0a\x9f\x33\x25\x00\xa5" + "\xdf\x94\x87\xd0\x7a\x5c\x92\xcc" + "\x51\x2c\x88\x66\xc7\xe8\x60\xce" + "\x93\xfd\xf1\x66\xa2\x49\x12\xb4" + "\x22\x97\x61\x46\xae\x20\xce\x84" + "\x6b\xb7\xdc\x9b\xa9\x4a\x76\x7a" + "\xae\xf2\x0c\x0d\x61\xad\x02\x65" + "\x5e\xa9\x2d\xc4\xc4\xe4\x1a\x89" + "\x52\xc6\x51\xd3\x31\x74\xbe\x51" + "\xa1\x0c\x42\x11\x10\xe6\xd8\x15" + "\x88\xed\xe8\x21\x03\xa2\x52\xd8" + "\xa7\x50\xe8\x76\x8d\xef\xff\xed" + "\x91\x22\x81\x0a\xae\xb9\x9f\x91" + "\x72\xaf\x82\xb6\x04\xdc\x4b\x8e" + "\x51\xbc\xb0\x82\x35\xa6\xf4\x34" + "\x13\x32\xe4\xca\x60\x48\x2a\x4b" + "\xa1\xa0\x3b\x3e\x65\x00\x8f\xc5" + "\xda\x76\xb7\x0b\xf1\x69\x0d\xb4" + "\xea\xe2\x9c\x5f\x1b\xad\xd0\x3c" + "\x5c\xcf\x2a\x55\xd7\x05\xdd\xcd" + "\x86\xd4\x49\x51\x1c\xeb\x7e\xc3" + "\x0b\xf1\x2b\x1f\xa3\x5b\x91\x3f" + "\x9f\x74\x7a\x8a\xfd\x1b\x13\x0e" + "\x94\xbf\xf9\x4e\xff\xd0\x1a\x91" + "\x73\x5c\xa1\x72\x6a\xcd\x0b\x19" + "\x7c\x4e\x5b\x03\x39\x36\x97\xe1" + "\x26\x82\x6f\xb6\xbb\xde\x8e\xcc" + "\x1e\x08\x29\x85\x16\xe2\xc9\xed" + "\x03\xff\x3c\x1b\x78\x60\xf6\xde" + "\x76\xd4\xce\xcd\x94\xc8\x11\x98" + "\x55\xef\x52\x97\xca\x67\xe9\xf3" + "\xe7\xff\x72\xb1\xe9\x97\x85\xca" + "\x0a\x7e\x77\x20\xc5\xb3\x6d\xc6" + "\xd7\x2c\xac\x95\x74\xc8\xcb\xbc" + "\x2f\x80\x1e\x23\xe5\x6f\xd3\x44" + "\xb0\x7f\x22\x15\x4b\xeb\xa0\xf0" + "\x8c\xe8\x89\x1e\x64\x3e\xd9\x95" + "\xc9\x4d\x9a\x69\xc9\xf1\xb5\xf4" + "\x99\x02\x7a\x78\x57\x2a\xee\xbd" + "\x74\xd2\x0c\xc3\x98\x81\xc2\x13" + "\xee\x77\x0b\x10\x10\xe4\xbe\xa7" + "\x18\x84\x69\x77\xae\x11\x9f\x7a" + "\x02\x3a\xb5\x8c\xca\x0a\xd7\x52" + "\xaf\xe6\x56\xbb\x3c\x17\x25\x6a" + "\x9f\x6e\x9b\xf1\x9f\xdd\x5a\x38" + "\xfc\x82\xbb\xe8\x72\xc5\x53\x9e" + "\xdb\x60\x9e\xf4\xf7\x9c\x20\x3e" + "\xbb\x14\x0f\x2e\x58\x3c\xb2\xad" + "\x15\xb4\xaa\x5b\x65\x50\x16\xa8" + "\x44\x92\x77\xdb\xd4\x77\xef\x2c" + "\x8d\x6c\x01\x7d\xb7\x38\xb1\x8d" + "\xeb\x4a\x42\x7d\x19\x23\xce\x3f" + "\xf2\x62\x73\x57\x79\xa4\x18\xf2" + "\x0a\x28\x2d\xf9\x20\x14\x7b\xea" + "\xbe\x42\x1e\xe5\x31\x9d\x05\x68", + .len = 512, + }, { /* XTS-AES 10, XTS-AES-256, data unit 512 bytes */ + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x62\x49\x77\x57\x24\x70\x93\x69" + "\x99\x59\x57\x49\x66\x96\x76\x27" + "\x31\x41\x59\x26\x53\x58\x97\x93" + "\x23\x84\x62\x64\x33\x83\x27\x95" + "\x02\x88\x41\x97\x16\x93\x99\x37" + "\x51\x05\x82\x09\x74\x94\x45\x92", + .klen = 64, + .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ctext = "\x1c\x3b\x3a\x10\x2f\x77\x03\x86" + "\xe4\x83\x6c\x99\xe3\x70\xcf\x9b" + "\xea\x00\x80\x3f\x5e\x48\x23\x57" + "\xa4\xae\x12\xd4\x14\xa3\xe6\x3b" + "\x5d\x31\xe2\x76\xf8\xfe\x4a\x8d" + "\x66\xb3\x17\xf9\xac\x68\x3f\x44" + "\x68\x0a\x86\xac\x35\xad\xfc\x33" + "\x45\xbe\xfe\xcb\x4b\xb1\x88\xfd" + "\x57\x76\x92\x6c\x49\xa3\x09\x5e" + "\xb1\x08\xfd\x10\x98\xba\xec\x70" + "\xaa\xa6\x69\x99\xa7\x2a\x82\xf2" + "\x7d\x84\x8b\x21\xd4\xa7\x41\xb0" + "\xc5\xcd\x4d\x5f\xff\x9d\xac\x89" + "\xae\xba\x12\x29\x61\xd0\x3a\x75" + "\x71\x23\xe9\x87\x0f\x8a\xcf\x10" + "\x00\x02\x08\x87\x89\x14\x29\xca" + "\x2a\x3e\x7a\x7d\x7d\xf7\xb1\x03" + "\x55\x16\x5c\x8b\x9a\x6d\x0a\x7d" + "\xe8\xb0\x62\xc4\x50\x0d\xc4\xcd" + "\x12\x0c\x0f\x74\x18\xda\xe3\xd0" + "\xb5\x78\x1c\x34\x80\x3f\xa7\x54" + "\x21\xc7\x90\xdf\xe1\xde\x18\x34" + "\xf2\x80\xd7\x66\x7b\x32\x7f\x6c" + "\x8c\xd7\x55\x7e\x12\xac\x3a\x0f" + "\x93\xec\x05\xc5\x2e\x04\x93\xef" + "\x31\xa1\x2d\x3d\x92\x60\xf7\x9a" + "\x28\x9d\x6a\x37\x9b\xc7\x0c\x50" + "\x84\x14\x73\xd1\xa8\xcc\x81\xec" + "\x58\x3e\x96\x45\xe0\x7b\x8d\x96" + "\x70\x65\x5b\xa5\xbb\xcf\xec\xc6" + "\xdc\x39\x66\x38\x0a\xd8\xfe\xcb" + "\x17\xb6\xba\x02\x46\x9a\x02\x0a" + "\x84\xe1\x8e\x8f\x84\x25\x20\x70" + "\xc1\x3e\x9f\x1f\x28\x9b\xe5\x4f" + "\xbc\x48\x14\x57\x77\x8f\x61\x60" + "\x15\xe1\x32\x7a\x02\xb1\x40\xf1" + "\x50\x5e\xb3\x09\x32\x6d\x68\x37" + "\x8f\x83\x74\x59\x5c\x84\x9d\x84" + "\xf4\xc3\x33\xec\x44\x23\x88\x51" + "\x43\xcb\x47\xbd\x71\xc5\xed\xae" + "\x9b\xe6\x9a\x2f\xfe\xce\xb1\xbe" + "\xc9\xde\x24\x4f\xbe\x15\x99\x2b" + "\x11\xb7\x7c\x04\x0f\x12\xbd\x8f" + "\x6a\x97\x5a\x44\xa0\xf9\x0c\x29" + "\xa9\xab\xc3\xd4\xd8\x93\x92\x72" + "\x84\xc5\x87\x54\xcc\xe2\x94\x52" + "\x9f\x86\x14\xdc\xd2\xab\xa9\x91" + "\x92\x5f\xed\xc4\xae\x74\xff\xac" + "\x6e\x33\x3b\x93\xeb\x4a\xff\x04" + "\x79\xda\x9a\x41\x0e\x44\x50\xe0" + "\xdd\x7a\xe4\xc6\xe2\x91\x09\x00" + "\x57\x5d\xa4\x01\xfc\x07\x05\x9f" + "\x64\x5e\x8b\x7e\x9b\xfd\xef\x33" + "\x94\x30\x54\xff\x84\x01\x14\x93" + "\xc2\x7b\x34\x29\xea\xed\xb4\xed" + "\x53\x76\x44\x1a\x77\xed\x43\x85" + "\x1a\xd7\x7f\x16\xf5\x41\xdf\xd2" + "\x69\xd5\x0d\x6a\x5f\x14\xfb\x0a" + "\xab\x1c\xbb\x4c\x15\x50\xbe\x97" + "\xf7\xab\x40\x66\x19\x3c\x4c\xaa" + "\x77\x3d\xad\x38\x01\x4b\xd2\x09" + "\x2f\xa7\x55\xc8\x24\xbb\x5e\x54" + "\xc4\xf3\x6f\xfd\xa9\xfc\xea\x70" + "\xb9\xc6\xe6\x93\xe1\x48\xc1\x51", + .len = 512, + .also_non_np = 1, + .np = 3, + .tap = { 512 - 20, 4, 16 }, + } +}; + +static const struct cipher_testvec aes_ctr_tv_template[] = { + { /* From NIST Special Publication 800-38A, Appendix F.5 */ + .key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6" + "\xab\xf7\x15\x88\x09\xcf\x4f\x3c", + .klen = 16, + .iv = "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ptext = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ctext = "\x87\x4d\x61\x91\xb6\x20\xe3\x26" + "\x1b\xef\x68\x64\x99\x0d\xb6\xce" + "\x98\x06\xf6\x6b\x79\x70\xfd\xff" + "\x86\x17\x18\x7b\xb9\xff\xfd\xff" + "\x5a\xe4\xdf\x3e\xdb\xd5\xd3\x5e" + "\x5b\x4f\x09\x02\x0d\xb0\x3e\xab" + "\x1e\x03\x1d\xda\x2f\xbe\x03\xd1" + "\x79\x21\x70\xa0\xf3\x00\x9c\xee", + .len = 64, + }, { + .key = "\x8e\x73\xb0\xf7\xda\x0e\x64\x52" + "\xc8\x10\xf3\x2b\x80\x90\x79\xe5" + "\x62\xf8\xea\xd2\x52\x2c\x6b\x7b", + .klen = 24, + .iv = "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ptext = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ctext = "\x1a\xbc\x93\x24\x17\x52\x1c\xa2" + "\x4f\x2b\x04\x59\xfe\x7e\x6e\x0b" + "\x09\x03\x39\xec\x0a\xa6\xfa\xef" + "\xd5\xcc\xc2\xc6\xf4\xce\x8e\x94" + "\x1e\x36\xb2\x6b\xd1\xeb\xc6\x70" + "\xd1\xbd\x1d\x66\x56\x20\xab\xf7" + "\x4f\x78\xa7\xf6\xd2\x98\x09\x58" + "\x5a\x97\xda\xec\x58\xc6\xb0\x50", + .len = 64, + }, { + .key = "\x60\x3d\xeb\x10\x15\xca\x71\xbe" + "\x2b\x73\xae\xf0\x85\x7d\x77\x81" + "\x1f\x35\x2c\x07\x3b\x61\x08\xd7" + "\x2d\x98\x10\xa3\x09\x14\xdf\xf4", + .klen = 32, + .iv = "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", + .ptext = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ctext = "\x60\x1e\xc3\x13\x77\x57\x89\xa5" + "\xb7\xa7\xf5\x04\xbb\xf3\xd2\x28" + "\xf4\x43\xe3\xca\x4d\x62\xb5\x9a" + "\xca\x84\xe9\x90\xca\xca\xf5\xc5" + "\x2b\x09\x30\xda\xa2\x3d\xe9\x4c" + "\xe8\x70\x17\xba\x2d\x84\x98\x8d" + "\xdf\xc9\xc5\x8d\xb6\x7a\xad\xa6" + "\x13\xc2\xdd\x08\x45\x79\x41\xa6", + .len = 64, + }, { /* Generated with Crypto++ */ + .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55" + "\x0F\x32\x55\x78\x9B\xBE\x78\x9B" + "\xBE\xE1\x04\x27\xE1\x04\x27\x4A" + "\x6D\x90\x4A\x6D\x90\xB3\xD6\xF9", + .klen = 32, + .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" + "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", + .ptext = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" + "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" + "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" + "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" + "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" + "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" + "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" + "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" + "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" + "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" + "\x22\x8B\x17\x80\xE9\x52\xDE\x47" + "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" + "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" + "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" + "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" + "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" + "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" + "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" + "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" + "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" + "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" + "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" + "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" + "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" + "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" + "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" + "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" + "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" + "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" + "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" + "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" + "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" + "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" + "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" + "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" + "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" + "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" + "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" + "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" + "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" + "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" + "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" + "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" + "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" + "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" + "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" + "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" + "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" + "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" + "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" + "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" + "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" + "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" + "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" + "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" + "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" + "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" + "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" + "\x20\x89\x15\x7E\xE7\x50\xDC\x45" + "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" + "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" + "\xED\x56\xBF\x28\xB4\x1D\x86\x12", + .ctext = "\x04\xF3\xD3\x88\x17\xEF\xDC\xEF" + "\x8B\x04\xF8\x3A\x66\x8D\x1A\x53" + "\x57\x1F\x4B\x23\xE4\xA0\xAF\xF9" + "\x69\x95\x35\x98\x8D\x4D\x8C\xC1" + "\xF0\xB2\x7F\x80\xBB\x54\x28\xA2" + "\x7A\x1B\x9F\x77\xEC\x0E\x6E\xDE" + "\xF0\xEC\xB8\xE4\x20\x62\xEE\xDB" + "\x5D\xF5\xDD\xE3\x54\xFC\xDD\xEB" + "\x6A\xEE\x65\xA1\x21\xD6\xD7\x81" + "\x47\x61\x12\x4D\xC2\x8C\xFA\x78" + "\x1F\x28\x02\x01\xC3\xFC\x1F\xEC" + "\x0F\x10\x4F\xB3\x12\x45\xC6\x3B" + "\x7E\x08\xF9\x5A\xD0\x5D\x73\x2D" + "\x58\xA4\xE5\xCB\x1C\xB4\xCE\x74" + "\x32\x41\x1F\x31\x9C\x08\xA2\x5D" + "\x67\xEB\x72\x1D\xF8\xE7\x70\x54" + "\x34\x4B\x31\x69\x84\x66\x96\x44" + "\x56\xCC\x1E\xD9\xE6\x13\x6A\xB9" + "\x2D\x0A\x05\x45\x2D\x90\xCC\xDF" + "\x16\x5C\x5F\x79\x34\x52\x54\xFE" + "\xFE\xCD\xAD\x04\x2E\xAD\x86\x06" + "\x1F\x37\xE8\x28\xBC\xD3\x8F\x5B" + "\x92\x66\x87\x3B\x8A\x0A\x1A\xCC" + "\x6E\xAB\x9F\x0B\xFA\x5C\xE6\xFD" + "\x3C\x98\x08\x12\xEC\xAA\x9E\x11" + "\xCA\xB2\x1F\xCE\x5E\x5B\xB2\x72" + "\x9C\xCC\x5D\xC5\xE0\x32\xC0\x56" + "\xD5\x45\x16\xD2\xAF\x13\x66\xF7" + "\x8C\x67\xAC\x79\xB2\xAF\x56\x27" + "\x3F\xCC\xFE\xCB\x1E\xC0\x75\xF1" + "\xA7\xC9\xC3\x1D\x8E\xDD\xF9\xD4" + "\x42\xC8\x21\x08\x16\xF7\x01\xD7" + "\xAC\x8E\x3F\x1D\x56\xC1\x06\xE4" + "\x9C\x62\xD6\xA5\x6A\x50\x44\xB3" + "\x35\x1C\x82\xB9\x10\xF9\x42\xA1" + "\xFC\x74\x9B\x44\x4F\x25\x02\xE3" + "\x08\xF5\xD4\x32\x39\x08\x11\xE8" + "\xD2\x6B\x50\x53\xD4\x08\xD1\x6B" + "\x3A\x4A\x68\x7B\x7C\xCD\x46\x5E" + "\x0D\x07\x19\xDB\x67\xD7\x98\x91" + "\xD7\x17\x10\x9B\x7B\x8A\x9B\x33" + "\xAE\xF3\x00\xA6\xD4\x15\xD9\xEA" + "\x85\x99\x22\xE8\x91\x38\x70\x83" + "\x93\x01\x24\x6C\xFA\x9A\xB9\x07" + "\xEA\x8D\x3B\xD9\x2A\x43\x59\x16" + "\x2F\x69\xEE\x84\x36\x44\x76\x98" + "\xF3\x04\x2A\x7C\x74\x3D\x29\x2B" + "\x0D\xAD\x8F\x44\x82\x9E\x57\x8D" + "\xAC\xED\x18\x1F\x50\xA4\xF5\x98" + "\x1F\xBD\x92\x91\x1B\x2D\xA6\xD6" + "\xD2\xE3\x02\xAA\x92\x3B\xC6\xB3" + "\x1B\x39\x72\xD5\x26\xCA\x04\xE0" + "\xFC\x58\x78\xBB\xB1\x3F\xA1\x9C" + "\x42\x24\x3E\x2E\x22\xBB\x4B\xBA" + "\xF4\x52\x0A\xE6\xAE\x47\xB4\x7D" + "\x1D\xA8\xBE\x81\x1A\x75\xDA\xAC" + "\xA6\x25\x1E\xEF\x3A\xC0\x6C\x63" + "\xEF\xDC\xC9\x79\x10\x26\xE8\x61" + "\x29\xFC\xA4\x05\xDF\x7D\x5C\x63" + "\x10\x09\x9B\x46\x9B\xF2\x2C\x2B" + "\xFA\x3A\x05\x4C\xFA\xD1\xFF\xFE" + "\xF1\x4C\xE5\xB2\x91\x64\x0C\x51", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, { /* Generated with Crypto++ */ + .key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55" + "\x0F\x32\x55\x78\x9B\xBE\x78\x9B" + "\xBE\xE1\x04\x27\xE1\x04\x27\x4A" + "\x6D\x90\x4A\x6D\x90\xB3\xD6\xF9", + .klen = 32, + .iv = "\xE7\x82\x1D\xB8\x53\x11\xAC\x47" + "\xE2\x7D\x18\xD6\x71\x0C\xA7\x42", + .ptext = "\x50\xB9\x22\xAE\x17\x80\x0C\x75" + "\xDE\x47\xD3\x3C\xA5\x0E\x9A\x03" + "\x6C\xF8\x61\xCA\x33\xBF\x28\x91" + "\x1D\x86\xEF\x58\xE4\x4D\xB6\x1F" + "\xAB\x14\x7D\x09\x72\xDB\x44\xD0" + "\x39\xA2\x0B\x97\x00\x69\xF5\x5E" + "\xC7\x30\xBC\x25\x8E\x1A\x83\xEC" + "\x55\xE1\x4A\xB3\x1C\xA8\x11\x7A" + "\x06\x6F\xD8\x41\xCD\x36\x9F\x08" + "\x94\xFD\x66\xF2\x5B\xC4\x2D\xB9" + "\x22\x8B\x17\x80\xE9\x52\xDE\x47" + "\xB0\x19\xA5\x0E\x77\x03\x6C\xD5" + "\x3E\xCA\x33\x9C\x05\x91\xFA\x63" + "\xEF\x58\xC1\x2A\xB6\x1F\x88\x14" + "\x7D\xE6\x4F\xDB\x44\xAD\x16\xA2" + "\x0B\x74\x00\x69\xD2\x3B\xC7\x30" + "\x99\x02\x8E\xF7\x60\xEC\x55\xBE" + "\x27\xB3\x1C\x85\x11\x7A\xE3\x4C" + "\xD8\x41\xAA\x13\x9F\x08\x71\xFD" + "\x66\xCF\x38\xC4\x2D\x96\x22\x8B" + "\xF4\x5D\xE9\x52\xBB\x24\xB0\x19" + "\x82\x0E\x77\xE0\x49\xD5\x3E\xA7" + "\x10\x9C\x05\x6E\xFA\x63\xCC\x35" + "\xC1\x2A\x93\x1F\x88\xF1\x5A\xE6" + "\x4F\xB8\x21\xAD\x16\x7F\x0B\x74" + "\xDD\x46\xD2\x3B\xA4\x0D\x99\x02" + "\x6B\xF7\x60\xC9\x32\xBE\x27\x90" + "\x1C\x85\xEE\x57\xE3\x4C\xB5\x1E" + "\xAA\x13\x7C\x08\x71\xDA\x43\xCF" + "\x38\xA1\x0A\x96\xFF\x68\xF4\x5D" + "\xC6\x2F\xBB\x24\x8D\x19\x82\xEB" + "\x54\xE0\x49\xB2\x1B\xA7\x10\x79" + "\x05\x6E\xD7\x40\xCC\x35\x9E\x07" + "\x93\xFC\x65\xF1\x5A\xC3\x2C\xB8" + "\x21\x8A\x16\x7F\xE8\x51\xDD\x46" + "\xAF\x18\xA4\x0D\x76\x02\x6B\xD4" + "\x3D\xC9\x32\x9B\x04\x90\xF9\x62" + "\xEE\x57\xC0\x29\xB5\x1E\x87\x13" + "\x7C\xE5\x4E\xDA\x43\xAC\x15\xA1" + "\x0A\x73\xFF\x68\xD1\x3A\xC6\x2F" + "\x98\x01\x8D\xF6\x5F\xEB\x54\xBD" + "\x26\xB2\x1B\x84\x10\x79\xE2\x4B" + "\xD7\x40\xA9\x12\x9E\x07\x70\xFC" + "\x65\xCE\x37\xC3\x2C\x95\x21\x8A" + "\xF3\x5C\xE8\x51\xBA\x23\xAF\x18" + "\x81\x0D\x76\xDF\x48\xD4\x3D\xA6" + "\x0F\x9B\x04\x6D\xF9\x62\xCB\x34" + "\xC0\x29\x92\x1E\x87\xF0\x59\xE5" + "\x4E\xB7\x20\xAC\x15\x7E\x0A\x73" + "\xDC\x45\xD1\x3A\xA3\x0C\x98\x01" + "\x6A\xF6\x5F\xC8\x31\xBD\x26\x8F" + "\x1B\x84\xED\x56\xE2\x4B\xB4\x1D" + "\xA9\x12\x7B\x07\x70\xD9\x42\xCE" + "\x37\xA0\x09\x95\xFE\x67\xF3\x5C" + "\xC5\x2E\xBA\x23\x8C\x18\x81\xEA" + "\x53\xDF\x48\xB1\x1A\xA6\x0F\x78" + "\x04\x6D\xD6\x3F\xCB\x34\x9D\x06" + "\x92\xFB\x64\xF0\x59\xC2\x2B\xB7" + "\x20\x89\x15\x7E\xE7\x50\xDC\x45" + "\xAE\x17\xA3\x0C\x75\x01\x6A\xD3" + "\x3C\xC8\x31\x9A\x03\x8F\xF8\x61" + "\xED\x56\xBF\x28\xB4\x1D\x86\x12" + "\x7B\xE4\x4D", + .ctext = "\xDA\x4E\x3F\xBC\xE8\xB6\x3A\xA2" + "\xD5\x4D\x84\x4A\xA9\x0C\xE1\xA5" + "\xB8\x73\xBC\xF9\xBB\x59\x2F\x44" + "\x8B\xAB\x82\x6C\xB4\x32\x9A\xDE" + "\x5A\x0B\xDB\x7A\x6B\xF2\x38\x9F" + "\x06\xF7\xF7\xFF\xFF\xC0\x8A\x2E" + "\x76\xEA\x06\x32\x23\xF3\x59\x2E" + "\x75\xDE\x71\x86\x3C\x98\x23\x44" + "\x5B\xF2\xFA\x6A\x00\xBB\xC1\xAD" + "\x58\xBD\x3E\x6F\x2E\xB4\x19\x04" + "\x70\x8B\x92\x55\x23\xE9\x6A\x3A" + "\x78\x7A\x1B\x10\x85\x52\x9C\x12" + "\xE4\x55\x81\x21\xCE\x53\xD0\x3B" + "\x63\x77\x2C\x74\xD1\xF5\x60\xF3" + "\xA1\xDE\x44\x3C\x8F\x4D\x2F\xDD" + "\x8A\xFE\x3C\x42\x8E\xD3\xF2\x8E" + "\xA8\x28\x69\x65\x31\xE1\x45\x83" + "\xE4\x49\xC4\x9C\xA7\x28\xAA\x21" + "\xCD\x5D\x0F\x15\xB7\x93\x07\x26" + "\xB0\x65\x6D\x91\x90\x23\x7A\xC6" + "\xDB\x68\xB0\xA1\x8E\xA4\x76\x4E" + "\xC6\x91\x83\x20\x92\x4D\x63\x7A" + "\x45\x18\x18\x74\x19\xAD\x71\x01" + "\x6B\x23\xAD\x9D\x4E\xE4\x6E\x46" + "\xC9\x73\x7A\xF9\x02\x95\xF4\x07" + "\x0E\x7A\xA6\xC5\xAE\xFA\x15\x2C" + "\x51\x71\xF1\xDC\x22\xB6\xAC\xD8" + "\x19\x24\x44\xBC\x0C\xFB\x3C\x2D" + "\xB1\x50\x47\x15\x0E\xDB\xB6\xD7" + "\xE8\x61\xE5\x95\x52\x1E\x3E\x49" + "\x70\xE9\x66\x04\x4C\xE1\xAF\xBD" + "\xDD\x15\x3B\x20\x59\x24\xFF\xB0" + "\x39\xAA\xE7\xBF\x23\xA3\x6E\xD5" + "\x15\xF0\x61\x4F\xAE\x89\x10\x58" + "\x5A\x33\x95\x52\x2A\xB5\x77\x9C" + "\xA5\x43\x80\x40\x27\x2D\xAE\xD9" + "\x3F\xE0\x80\x94\x78\x79\xCB\x7E" + "\xAD\x12\x44\x4C\xEC\x27\xB0\xEE" + "\x0B\x05\x2A\x82\x99\x58\xBB\x7A" + "\x8D\x6D\x9D\x8E\xE2\x8E\xE7\x93" + "\x2F\xB3\x09\x8D\x06\xD5\xEE\x70" + "\x16\xAE\x35\xC5\x52\x0F\x46\x1F" + "\x71\xF9\x5E\xF2\x67\xDC\x98\x2F" + "\xA3\x23\xAA\xD5\xD0\x49\xF4\xA6" + "\xF6\xB8\x32\xCD\xD6\x85\x73\x60" + "\x59\x20\xE7\x55\x0E\x91\xE2\x0C" + "\x3F\x1C\xEB\x3D\xDF\x52\x64\xF2" + "\x7D\x8B\x5D\x63\x16\xB9\xB2\x5D" + "\x5E\xAB\xB2\x97\xAB\x78\x44\xE7" + "\xC6\x72\x20\xC5\x90\x9B\xDC\x5D" + "\xB0\xEF\x44\xEF\x87\x31\x8D\xF4" + "\xFB\x81\x5D\xF7\x96\x96\xD4\x50" + "\x89\xA7\xF6\xB9\x67\x76\x40\x9E" + "\x9D\x40\xD5\x2C\x30\xB8\x01\x8F" + "\xE4\x7B\x71\x48\xA9\xA0\xA0\x1D" + "\x87\x52\xA4\x91\xA9\xD7\xA9\x51" + "\xD9\x59\xF7\xCC\x63\x22\xC1\x8D" + "\x84\x7B\xD8\x22\x32\x5C\x6F\x1D" + "\x6E\x9F\xFA\xDD\x49\x40\xDC\x37" + "\x14\x8C\xE1\x80\x1B\xDD\x36\x2A" + "\xD0\xE9\x54\x99\x5D\xBA\x3B\x11" + "\xD8\xFE\xC9\x5B\x5C\x25\xE5\x76" + "\xFB\xF2\x3F", + .len = 499, .also_non_np = 1, - .np = 3, - .tap = { 1008 - 20, 4, 16 }, + .np = 2, + .tap = { 499 - 16, 16 }, + }, +}; + +static const struct cipher_testvec aes_ctr_rfc3686_tv_template[] = { + { /* From RFC 3686 */ + .key = "\xae\x68\x52\xf8\x12\x10\x67\xcc" + "\x4b\xf7\xa5\x76\x55\x77\xf3\x9e" + "\x00\x00\x00\x30", + .klen = 20, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "Single block msg", + .ctext = "\xe4\x09\x5d\x4f\xb7\xa7\xb3\x79" + "\x2d\x61\x75\xa3\x26\x13\x11\xb8", + .len = 16, + }, { + .key = "\x7e\x24\x06\x78\x17\xfa\xe0\xd7" + "\x43\xd6\xce\x1f\x32\x53\x91\x63" + "\x00\x6c\xb6\xdb", + .klen = 20, + .iv = "\xc0\x54\x3b\x59\xda\x48\xd9\x0b", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .ctext = "\x51\x04\xa1\x06\x16\x8a\x72\xd9" + "\x79\x0d\x41\xee\x8e\xda\xd3\x88" + "\xeb\x2e\x1e\xfc\x46\xda\x57\xc8" + "\xfc\xe6\x30\xdf\x91\x41\xbe\x28", + .len = 32, + }, { + .key = "\x16\xaf\x5b\x14\x5f\xc9\xf5\x79" + "\xc1\x75\xf9\x3e\x3b\xfb\x0e\xed" + "\x86\x3d\x06\xcc\xfd\xb7\x85\x15" + "\x00\x00\x00\x48", + .klen = 28, + .iv = "\x36\x73\x3c\x14\x7d\x6d\x93\xcb", + .ptext = "Single block msg", + .ctext = "\x4b\x55\x38\x4f\xe2\x59\xc9\xc8" + "\x4e\x79\x35\xa0\x03\xcb\xe9\x28", + .len = 16, + }, { + .key = "\x7c\x5c\xb2\x40\x1b\x3d\xc3\x3c" + "\x19\xe7\x34\x08\x19\xe0\xf6\x9c" + "\x67\x8c\x3d\xb8\xe6\xf6\xa9\x1a" + "\x00\x96\xb0\x3b", + .klen = 28, + .iv = "\x02\x0c\x6e\xad\xc2\xcb\x50\x0d", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .ctext = "\x45\x32\x43\xfc\x60\x9b\x23\x32" + "\x7e\xdf\xaa\xfa\x71\x31\xcd\x9f" + "\x84\x90\x70\x1c\x5a\xd4\xa7\x9c" + "\xfc\x1f\xe0\xff\x42\xf4\xfb\x00", + .len = 32, + }, { + .key = "\x77\x6b\xef\xf2\x85\x1d\xb0\x6f" + "\x4c\x8a\x05\x42\xc8\x69\x6f\x6c" + "\x6a\x81\xaf\x1e\xec\x96\xb4\xd3" + "\x7f\xc1\xd6\x89\xe6\xc1\xc1\x04" + "\x00\x00\x00\x60", + .klen = 36, + .iv = "\xdb\x56\x72\xc9\x7a\xa8\xf0\xb2", + .ptext = "Single block msg", + .ctext = "\x14\x5a\xd0\x1d\xbf\x82\x4e\xc7" + "\x56\x08\x63\xdc\x71\xe3\xe0\xc0", + .len = 16, + }, { + .key = "\xf6\xd6\x6d\x6b\xd5\x2d\x59\xbb" + "\x07\x96\x36\x58\x79\xef\xf8\x86" + "\xc6\x6d\xd5\x1a\x5b\x6a\x99\x74" + "\x4b\x50\x59\x0c\x87\xa2\x38\x84" + "\x00\xfa\xac\x24", + .klen = 36, + .iv = "\xc1\x58\x5e\xf1\x5a\x43\xd8\x75", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .ctext = "\xf0\x5e\x23\x1b\x38\x94\x61\x2c" + "\x49\xee\x00\x0b\x80\x4e\xb2\xa9" + "\xb8\x30\x6b\x50\x8f\x83\x9d\x6a" + "\x55\x30\x83\x1d\x93\x44\xaf\x1c", + .len = 32, + }, { + // generated using Crypto++ + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x00\x00\x00\x00", + .klen = 32 + 4, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = + "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20\x21\x22\x23\x24\x25\x26\x27" + "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" + "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" + "\x40\x41\x42\x43\x44\x45\x46\x47" + "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" + "\x50\x51\x52\x53\x54\x55\x56\x57" + "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" + "\x60\x61\x62\x63\x64\x65\x66\x67" + "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" + "\x70\x71\x72\x73\x74\x75\x76\x77" + "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" + "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" + "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" + "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" + "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" + "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" + "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" + "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" + "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" + "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" + "\xe8\xe9\xea\xeb\xec\xed\xee\xef" + "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" + "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" + "\x00\x03\x06\x09\x0c\x0f\x12\x15" + "\x18\x1b\x1e\x21\x24\x27\x2a\x2d" + "\x30\x33\x36\x39\x3c\x3f\x42\x45" + "\x48\x4b\x4e\x51\x54\x57\x5a\x5d" + "\x60\x63\x66\x69\x6c\x6f\x72\x75" + "\x78\x7b\x7e\x81\x84\x87\x8a\x8d" + "\x90\x93\x96\x99\x9c\x9f\xa2\xa5" + "\xa8\xab\xae\xb1\xb4\xb7\xba\xbd" + "\xc0\xc3\xc6\xc9\xcc\xcf\xd2\xd5" + "\xd8\xdb\xde\xe1\xe4\xe7\xea\xed" + "\xf0\xf3\xf6\xf9\xfc\xff\x02\x05" + "\x08\x0b\x0e\x11\x14\x17\x1a\x1d" + "\x20\x23\x26\x29\x2c\x2f\x32\x35" + "\x38\x3b\x3e\x41\x44\x47\x4a\x4d" + "\x50\x53\x56\x59\x5c\x5f\x62\x65" + "\x68\x6b\x6e\x71\x74\x77\x7a\x7d" + "\x80\x83\x86\x89\x8c\x8f\x92\x95" + "\x98\x9b\x9e\xa1\xa4\xa7\xaa\xad" + "\xb0\xb3\xb6\xb9\xbc\xbf\xc2\xc5" + "\xc8\xcb\xce\xd1\xd4\xd7\xda\xdd" + "\xe0\xe3\xe6\xe9\xec\xef\xf2\xf5" + "\xf8\xfb\xfe\x01\x04\x07\x0a\x0d" + "\x10\x13\x16\x19\x1c\x1f\x22\x25" + "\x28\x2b\x2e\x31\x34\x37\x3a\x3d" + "\x40\x43\x46\x49\x4c\x4f\x52\x55" + "\x58\x5b\x5e\x61\x64\x67\x6a\x6d" + "\x70\x73\x76\x79\x7c\x7f\x82\x85" + "\x88\x8b\x8e\x91\x94\x97\x9a\x9d" + "\xa0\xa3\xa6\xa9\xac\xaf\xb2\xb5" + "\xb8\xbb\xbe\xc1\xc4\xc7\xca\xcd" + "\xd0\xd3\xd6\xd9\xdc\xdf\xe2\xe5" + "\xe8\xeb\xee\xf1\xf4\xf7\xfa\xfd" + "\x00\x05\x0a\x0f\x14\x19\x1e\x23" + "\x28\x2d\x32\x37\x3c\x41\x46\x4b" + "\x50\x55\x5a\x5f\x64\x69\x6e\x73" + "\x78\x7d\x82\x87\x8c\x91\x96\x9b" + "\xa0\xa5\xaa\xaf\xb4\xb9\xbe\xc3" + "\xc8\xcd\xd2\xd7\xdc\xe1\xe6\xeb" + "\xf0\xf5\xfa\xff\x04\x09\x0e\x13" + "\x18\x1d\x22\x27\x2c\x31\x36\x3b" + "\x40\x45\x4a\x4f\x54\x59\x5e\x63" + "\x68\x6d\x72\x77\x7c\x81\x86\x8b" + "\x90\x95\x9a\x9f\xa4\xa9\xae\xb3" + "\xb8\xbd\xc2\xc7\xcc\xd1\xd6\xdb" + "\xe0\xe5\xea\xef\xf4\xf9\xfe\x03" + "\x08\x0d\x12\x17\x1c\x21\x26\x2b" + "\x30\x35\x3a\x3f\x44\x49\x4e\x53" + "\x58\x5d\x62\x67\x6c\x71\x76\x7b" + "\x80\x85\x8a\x8f\x94\x99\x9e\xa3" + "\xa8\xad\xb2\xb7\xbc\xc1\xc6\xcb" + "\xd0\xd5\xda\xdf\xe4\xe9\xee\xf3" + "\xf8\xfd\x02\x07\x0c\x11\x16\x1b" + "\x20\x25\x2a\x2f\x34\x39\x3e\x43" + "\x48\x4d\x52\x57\x5c\x61\x66\x6b" + "\x70\x75\x7a\x7f\x84\x89\x8e\x93" + "\x98\x9d\xa2\xa7\xac\xb1\xb6\xbb" + "\xc0\xc5\xca\xcf\xd4\xd9\xde\xe3" + "\xe8\xed\xf2\xf7\xfc\x01\x06\x0b" + "\x10\x15\x1a\x1f\x24\x29\x2e\x33" + "\x38\x3d\x42\x47\x4c\x51\x56\x5b" + "\x60\x65\x6a\x6f\x74\x79\x7e\x83" + "\x88\x8d\x92\x97\x9c\xa1\xa6\xab" + "\xb0\xb5\xba\xbf\xc4\xc9\xce\xd3" + "\xd8\xdd\xe2\xe7\xec\xf1\xf6\xfb" + "\x00\x07\x0e\x15\x1c\x23\x2a\x31" + "\x38\x3f\x46\x4d\x54\x5b\x62\x69" + "\x70\x77\x7e\x85\x8c\x93\x9a\xa1" + "\xa8\xaf\xb6\xbd\xc4\xcb\xd2\xd9" + "\xe0\xe7\xee\xf5\xfc\x03\x0a\x11" + "\x18\x1f\x26\x2d\x34\x3b\x42\x49" + "\x50\x57\x5e\x65\x6c\x73\x7a\x81" + "\x88\x8f\x96\x9d\xa4\xab\xb2\xb9" + "\xc0\xc7\xce\xd5\xdc\xe3\xea\xf1" + "\xf8\xff\x06\x0d\x14\x1b\x22\x29" + "\x30\x37\x3e\x45\x4c\x53\x5a\x61" + "\x68\x6f\x76\x7d\x84\x8b\x92\x99" + "\xa0\xa7\xae\xb5\xbc\xc3\xca\xd1" + "\xd8\xdf\xe6\xed\xf4\xfb\x02\x09" + "\x10\x17\x1e\x25\x2c\x33\x3a\x41" + "\x48\x4f\x56\x5d\x64\x6b\x72\x79" + "\x80\x87\x8e\x95\x9c\xa3\xaa\xb1" + "\xb8\xbf\xc6\xcd\xd4\xdb\xe2\xe9" + "\xf0\xf7\xfe\x05\x0c\x13\x1a\x21" + "\x28\x2f\x36\x3d\x44\x4b\x52\x59" + "\x60\x67\x6e\x75\x7c\x83\x8a\x91" + "\x98\x9f\xa6\xad\xb4\xbb\xc2\xc9" + "\xd0\xd7\xde\xe5\xec\xf3\xfa\x01" + "\x08\x0f\x16\x1d\x24\x2b\x32\x39" + "\x40\x47\x4e\x55\x5c\x63\x6a\x71" + "\x78\x7f\x86\x8d\x94\x9b\xa2\xa9" + "\xb0\xb7\xbe\xc5\xcc\xd3\xda\xe1" + "\xe8\xef\xf6\xfd\x04\x0b\x12\x19" + "\x20\x27\x2e\x35\x3c\x43\x4a\x51" + "\x58\x5f\x66\x6d\x74\x7b\x82\x89" + "\x90\x97\x9e\xa5\xac\xb3\xba\xc1" + "\xc8\xcf\xd6\xdd\xe4\xeb\xf2\xf9" + "\x00\x09\x12\x1b\x24\x2d\x36\x3f" + "\x48\x51\x5a\x63\x6c\x75\x7e\x87" + "\x90\x99\xa2\xab\xb4\xbd\xc6\xcf" + "\xd8\xe1\xea\xf3\xfc\x05\x0e\x17" + "\x20\x29\x32\x3b\x44\x4d\x56\x5f" + "\x68\x71\x7a\x83\x8c\x95\x9e\xa7" + "\xb0\xb9\xc2\xcb\xd4\xdd\xe6\xef" + "\xf8\x01\x0a\x13\x1c\x25\x2e\x37" + "\x40\x49\x52\x5b\x64\x6d\x76\x7f" + "\x88\x91\x9a\xa3\xac\xb5\xbe\xc7" + "\xd0\xd9\xe2\xeb\xf4\xfd\x06\x0f" + "\x18\x21\x2a\x33\x3c\x45\x4e\x57" + "\x60\x69\x72\x7b\x84\x8d\x96\x9f" + "\xa8\xb1\xba\xc3\xcc\xd5\xde\xe7" + "\xf0\xf9\x02\x0b\x14\x1d\x26\x2f" + "\x38\x41\x4a\x53\x5c\x65\x6e\x77" + "\x80\x89\x92\x9b\xa4\xad\xb6\xbf" + "\xc8\xd1\xda\xe3\xec\xf5\xfe\x07" + "\x10\x19\x22\x2b\x34\x3d\x46\x4f" + "\x58\x61\x6a\x73\x7c\x85\x8e\x97" + "\xa0\xa9\xb2\xbb\xc4\xcd\xd6\xdf" + "\xe8\xf1\xfa\x03\x0c\x15\x1e\x27" + "\x30\x39\x42\x4b\x54\x5d\x66\x6f" + "\x78\x81\x8a\x93\x9c\xa5\xae\xb7" + "\xc0\xc9\xd2\xdb\xe4\xed\xf6\xff" + "\x08\x11\x1a\x23\x2c\x35\x3e\x47" + "\x50\x59\x62\x6b\x74\x7d\x86\x8f" + "\x98\xa1\xaa\xb3\xbc\xc5\xce\xd7" + "\xe0\xe9\xf2\xfb\x04\x0d\x16\x1f" + "\x28\x31\x3a\x43\x4c\x55\x5e\x67" + "\x70\x79\x82\x8b\x94\x9d\xa6\xaf" + "\xb8\xc1\xca\xd3\xdc\xe5\xee\xf7" + "\x00\x0b\x16\x21\x2c\x37\x42\x4d" + "\x58\x63\x6e\x79\x84\x8f\x9a\xa5" + "\xb0\xbb\xc6\xd1\xdc\xe7\xf2\xfd" + "\x08\x13\x1e\x29\x34\x3f\x4a\x55" + "\x60\x6b\x76\x81\x8c\x97\xa2\xad" + "\xb8\xc3\xce\xd9\xe4\xef\xfa\x05" + "\x10\x1b\x26\x31\x3c\x47\x52\x5d" + "\x68\x73\x7e\x89\x94\x9f\xaa\xb5" + "\xc0\xcb\xd6\xe1\xec\xf7\x02\x0d" + "\x18\x23\x2e\x39\x44\x4f\x5a\x65" + "\x70\x7b\x86\x91\x9c\xa7\xb2\xbd" + "\xc8\xd3\xde\xe9\xf4\xff\x0a\x15" + "\x20\x2b\x36\x41\x4c\x57\x62\x6d" + "\x78\x83\x8e\x99\xa4\xaf\xba\xc5" + "\xd0\xdb\xe6\xf1\xfc\x07\x12\x1d" + "\x28\x33\x3e\x49\x54\x5f\x6a\x75" + "\x80\x8b\x96\xa1\xac\xb7\xc2\xcd" + "\xd8\xe3\xee\xf9\x04\x0f\x1a\x25" + "\x30\x3b\x46\x51\x5c\x67\x72\x7d" + "\x88\x93\x9e\xa9\xb4\xbf\xca\xd5" + "\xe0\xeb\xf6\x01\x0c\x17\x22\x2d" + "\x38\x43\x4e\x59\x64\x6f\x7a\x85" + "\x90\x9b\xa6\xb1\xbc\xc7\xd2\xdd" + "\xe8\xf3\xfe\x09\x14\x1f\x2a\x35" + "\x40\x4b\x56\x61\x6c\x77\x82\x8d" + "\x98\xa3\xae\xb9\xc4\xcf\xda\xe5" + "\xf0\xfb\x06\x11\x1c\x27\x32\x3d" + "\x48\x53\x5e\x69\x74\x7f\x8a\x95" + "\xa0\xab\xb6\xc1\xcc\xd7\xe2\xed" + "\xf8\x03\x0e\x19\x24\x2f\x3a\x45" + "\x50\x5b\x66\x71\x7c\x87\x92\x9d" + "\xa8\xb3\xbe\xc9\xd4\xdf\xea\xf5" + "\x00\x0d\x1a\x27\x34\x41\x4e\x5b" + "\x68\x75\x82\x8f\x9c\xa9\xb6\xc3" + "\xd0\xdd\xea\xf7\x04\x11\x1e\x2b" + "\x38\x45\x52\x5f\x6c\x79\x86\x93" + "\xa0\xad\xba\xc7\xd4\xe1\xee\xfb" + "\x08\x15\x22\x2f\x3c\x49\x56\x63" + "\x70\x7d\x8a\x97\xa4\xb1\xbe\xcb" + "\xd8\xe5\xf2\xff\x0c\x19\x26\x33" + "\x40\x4d\x5a\x67\x74\x81\x8e\x9b" + "\xa8\xb5\xc2\xcf\xdc\xe9\xf6\x03" + "\x10\x1d\x2a\x37\x44\x51\x5e\x6b" + "\x78\x85\x92\x9f\xac\xb9\xc6\xd3" + "\xe0\xed\xfa\x07\x14\x21\x2e\x3b" + "\x48\x55\x62\x6f\x7c\x89\x96\xa3" + "\xb0\xbd\xca\xd7\xe4\xf1\xfe\x0b" + "\x18\x25\x32\x3f\x4c\x59\x66\x73" + "\x80\x8d\x9a\xa7\xb4\xc1\xce\xdb" + "\xe8\xf5\x02\x0f\x1c\x29\x36\x43" + "\x50\x5d\x6a\x77\x84\x91\x9e\xab" + "\xb8\xc5\xd2\xdf\xec\xf9\x06\x13" + "\x20\x2d\x3a\x47\x54\x61\x6e\x7b" + "\x88\x95\xa2\xaf\xbc\xc9\xd6\xe3" + "\xf0\xfd\x0a\x17\x24\x31\x3e\x4b" + "\x58\x65\x72\x7f\x8c\x99\xa6\xb3" + "\xc0\xcd\xda\xe7\xf4\x01\x0e\x1b" + "\x28\x35\x42\x4f\x5c\x69\x76\x83" + "\x90\x9d\xaa\xb7\xc4\xd1\xde\xeb" + "\xf8\x05\x12\x1f\x2c\x39\x46\x53" + "\x60\x6d\x7a\x87\x94\xa1\xae\xbb" + "\xc8\xd5\xe2\xef\xfc\x09\x16\x23" + "\x30\x3d\x4a\x57\x64\x71\x7e\x8b" + "\x98\xa5\xb2\xbf\xcc\xd9\xe6\xf3" + "\x00\x0f\x1e\x2d\x3c\x4b\x5a\x69" + "\x78\x87\x96\xa5\xb4\xc3\xd2\xe1" + "\xf0\xff\x0e\x1d\x2c\x3b\x4a\x59" + "\x68\x77\x86\x95\xa4\xb3\xc2\xd1" + "\xe0\xef\xfe\x0d\x1c\x2b\x3a\x49" + "\x58\x67\x76\x85\x94\xa3\xb2\xc1" + "\xd0\xdf\xee\xfd\x0c\x1b\x2a\x39" + "\x48\x57\x66\x75\x84\x93\xa2\xb1" + "\xc0\xcf\xde\xed\xfc\x0b\x1a\x29" + "\x38\x47\x56\x65\x74\x83\x92\xa1" + "\xb0\xbf\xce\xdd\xec\xfb\x0a\x19" + "\x28\x37\x46\x55\x64\x73\x82\x91" + "\xa0\xaf\xbe\xcd\xdc\xeb\xfa\x09" + "\x18\x27\x36\x45\x54\x63\x72\x81" + "\x90\x9f\xae\xbd\xcc\xdb\xea\xf9" + "\x08\x17\x26\x35\x44\x53\x62\x71" + "\x80\x8f\x9e\xad\xbc\xcb\xda\xe9" + "\xf8\x07\x16\x25\x34\x43\x52\x61" + "\x70\x7f\x8e\x9d\xac\xbb\xca\xd9" + "\xe8\xf7\x06\x15\x24\x33\x42\x51" + "\x60\x6f\x7e\x8d\x9c\xab\xba\xc9" + "\xd8\xe7\xf6\x05\x14\x23\x32\x41" + "\x50\x5f\x6e\x7d\x8c\x9b\xaa\xb9" + "\xc8\xd7\xe6\xf5\x04\x13\x22\x31" + "\x40\x4f\x5e\x6d\x7c\x8b\x9a\xa9" + "\xb8\xc7\xd6\xe5\xf4\x03\x12\x21" + "\x30\x3f\x4e\x5d\x6c\x7b\x8a\x99" + "\xa8\xb7\xc6\xd5\xe4\xf3\x02\x11" + "\x20\x2f\x3e\x4d\x5c\x6b\x7a\x89" + "\x98\xa7\xb6\xc5\xd4\xe3\xf2\x01" + "\x10\x1f\x2e\x3d\x4c\x5b\x6a\x79" + "\x88\x97\xa6\xb5\xc4\xd3\xe2\xf1" + "\x00\x11\x22\x33\x44\x55\x66\x77" + "\x88\x99\xaa\xbb\xcc\xdd\xee\xff" + "\x10\x21\x32\x43\x54\x65\x76\x87" + "\x98\xa9\xba\xcb\xdc\xed\xfe\x0f" + "\x20\x31\x42\x53\x64\x75\x86\x97" + "\xa8\xb9\xca\xdb\xec\xfd\x0e\x1f" + "\x30\x41\x52\x63\x74\x85\x96\xa7" + "\xb8\xc9\xda\xeb\xfc\x0d\x1e\x2f" + "\x40\x51\x62\x73\x84\x95\xa6\xb7" + "\xc8\xd9\xea\xfb\x0c\x1d\x2e\x3f" + "\x50\x61\x72\x83\x94\xa5\xb6\xc7" + "\xd8\xe9\xfa\x0b\x1c\x2d\x3e\x4f" + "\x60\x71\x82\x93\xa4\xb5\xc6\xd7" + "\xe8\xf9\x0a\x1b\x2c\x3d\x4e\x5f" + "\x70\x81\x92\xa3\xb4\xc5\xd6\xe7" + "\xf8\x09\x1a\x2b\x3c\x4d\x5e\x6f" + "\x80\x91\xa2\xb3\xc4\xd5\xe6\xf7" + "\x08\x19\x2a\x3b\x4c\x5d\x6e\x7f" + "\x90\xa1\xb2\xc3\xd4\xe5\xf6\x07" + "\x18\x29\x3a\x4b\x5c\x6d\x7e\x8f" + "\xa0\xb1\xc2\xd3\xe4\xf5\x06\x17" + "\x28\x39\x4a\x5b\x6c\x7d\x8e\x9f" + "\xb0\xc1\xd2\xe3\xf4\x05\x16\x27" + "\x38\x49\x5a\x6b\x7c\x8d\x9e\xaf" + "\xc0\xd1\xe2\xf3\x04\x15\x26\x37" + "\x48\x59\x6a\x7b\x8c\x9d\xae\xbf" + "\xd0\xe1\xf2\x03\x14\x25\x36\x47" + "\x58\x69\x7a\x8b\x9c\xad\xbe\xcf" + "\xe0\xf1\x02\x13\x24\x35\x46\x57" + "\x68\x79\x8a\x9b\xac\xbd\xce\xdf" + "\xf0\x01\x12\x23\x34\x45\x56\x67" + "\x78\x89\x9a\xab\xbc\xcd\xde\xef" + "\x00\x13\x26\x39\x4c\x5f\x72\x85" + "\x98\xab\xbe\xd1\xe4\xf7\x0a\x1d" + "\x30\x43\x56\x69\x7c\x8f\xa2\xb5" + "\xc8\xdb\xee\x01\x14\x27\x3a\x4d" + "\x60\x73\x86\x99\xac\xbf\xd2\xe5" + "\xf8\x0b\x1e\x31\x44\x57\x6a\x7d" + "\x90\xa3\xb6\xc9\xdc\xef\x02\x15" + "\x28\x3b\x4e\x61\x74\x87\x9a\xad" + "\xc0\xd3\xe6\xf9\x0c\x1f\x32\x45" + "\x58\x6b\x7e\x91\xa4\xb7\xca\xdd" + "\xf0\x03\x16\x29\x3c\x4f\x62\x75" + "\x88\x9b\xae\xc1\xd4\xe7\xfa\x0d" + "\x20\x33\x46\x59\x6c\x7f\x92\xa5" + "\xb8\xcb\xde\xf1\x04\x17\x2a\x3d" + "\x50\x63\x76\x89\x9c\xaf\xc2\xd5" + "\xe8\xfb\x0e\x21\x34\x47\x5a\x6d" + "\x80\x93\xa6\xb9\xcc\xdf\xf2\x05" + "\x18\x2b\x3e\x51\x64\x77\x8a\x9d" + "\xb0\xc3\xd6\xe9\xfc\x0f\x22\x35" + "\x48\x5b\x6e\x81\x94\xa7\xba\xcd" + "\xe0\xf3\x06\x19\x2c\x3f\x52\x65" + "\x78\x8b\x9e\xb1\xc4\xd7\xea\xfd" + "\x10\x23\x36\x49\x5c\x6f\x82\x95" + "\xa8\xbb\xce\xe1\xf4\x07\x1a\x2d" + "\x40\x53\x66\x79\x8c\x9f\xb2\xc5" + "\xd8\xeb\xfe\x11\x24\x37\x4a\x5d" + "\x70\x83\x96\xa9\xbc\xcf\xe2\xf5" + "\x08\x1b\x2e\x41\x54\x67\x7a\x8d" + "\xa0\xb3\xc6\xd9\xec\xff\x12\x25" + "\x38\x4b\x5e\x71\x84\x97\xaa\xbd" + "\xd0\xe3\xf6\x09\x1c\x2f\x42\x55" + "\x68\x7b\x8e\xa1\xb4\xc7\xda\xed" + "\x00\x15\x2a\x3f\x54\x69\x7e\x93" + "\xa8\xbd\xd2\xe7\xfc\x11\x26\x3b" + "\x50\x65\x7a\x8f\xa4\xb9\xce\xe3" + "\xf8\x0d\x22\x37\x4c\x61\x76\x8b" + "\xa0\xb5\xca\xdf\xf4\x09\x1e\x33" + "\x48\x5d\x72\x87\x9c\xb1\xc6\xdb" + "\xf0\x05\x1a\x2f\x44\x59\x6e\x83" + "\x98\xad\xc2\xd7\xec\x01\x16\x2b" + "\x40\x55\x6a\x7f\x94\xa9\xbe\xd3" + "\xe8\xfd\x12\x27\x3c\x51\x66\x7b" + "\x90\xa5\xba\xcf\xe4\xf9\x0e\x23" + "\x38\x4d\x62\x77\x8c\xa1\xb6\xcb" + "\xe0\xf5\x0a\x1f\x34\x49\x5e\x73" + "\x88\x9d\xb2\xc7\xdc\xf1\x06\x1b" + "\x30\x45\x5a\x6f\x84\x99\xae\xc3" + "\xd8\xed\x02\x17\x2c\x41\x56\x6b" + "\x80\x95\xaa\xbf\xd4\xe9\xfe\x13" + "\x28\x3d\x52\x67\x7c\x91\xa6\xbb" + "\xd0\xe5\xfa\x0f\x24\x39\x4e\x63" + "\x78\x8d\xa2\xb7\xcc\xe1\xf6\x0b" + "\x20\x35\x4a\x5f\x74\x89\x9e\xb3" + "\xc8\xdd\xf2\x07\x1c\x31\x46\x5b" + "\x70\x85\x9a\xaf\xc4\xd9\xee\x03" + "\x18\x2d\x42\x57\x6c\x81\x96\xab" + "\xc0\xd5\xea\xff\x14\x29\x3e\x53" + "\x68\x7d\x92\xa7\xbc\xd1\xe6\xfb" + "\x10\x25\x3a\x4f\x64\x79\x8e\xa3" + "\xb8\xcd\xe2\xf7\x0c\x21\x36\x4b" + "\x60\x75\x8a\x9f\xb4\xc9\xde\xf3" + "\x08\x1d\x32\x47\x5c\x71\x86\x9b" + "\xb0\xc5\xda\xef\x04\x19\x2e\x43" + "\x58\x6d\x82\x97\xac\xc1\xd6\xeb" + "\x00\x17\x2e\x45\x5c\x73\x8a\xa1" + "\xb8\xcf\xe6\xfd\x14\x2b\x42\x59" + "\x70\x87\x9e\xb5\xcc\xe3\xfa\x11" + "\x28\x3f\x56\x6d\x84\x9b\xb2\xc9" + "\xe0\xf7\x0e\x25\x3c\x53\x6a\x81" + "\x98\xaf\xc6\xdd\xf4\x0b\x22\x39" + "\x50\x67\x7e\x95\xac\xc3\xda\xf1" + "\x08\x1f\x36\x4d\x64\x7b\x92\xa9" + "\xc0\xd7\xee\x05\x1c\x33\x4a\x61" + "\x78\x8f\xa6\xbd\xd4\xeb\x02\x19" + "\x30\x47\x5e\x75\x8c\xa3\xba\xd1" + "\xe8\xff\x16\x2d\x44\x5b\x72\x89" + "\xa0\xb7\xce\xe5\xfc\x13\x2a\x41" + "\x58\x6f\x86\x9d\xb4\xcb\xe2\xf9" + "\x10\x27\x3e\x55\x6c\x83\x9a\xb1" + "\xc8\xdf\xf6\x0d\x24\x3b\x52\x69" + "\x80\x97\xae\xc5\xdc\xf3\x0a\x21" + "\x38\x4f\x66\x7d\x94\xab\xc2\xd9" + "\xf0\x07\x1e\x35\x4c\x63\x7a\x91" + "\xa8\xbf\xd6\xed\x04\x1b\x32\x49" + "\x60\x77\x8e\xa5\xbc\xd3\xea\x01" + "\x18\x2f\x46\x5d\x74\x8b\xa2\xb9" + "\xd0\xe7\xfe\x15\x2c\x43\x5a\x71" + "\x88\x9f\xb6\xcd\xe4\xfb\x12\x29" + "\x40\x57\x6e\x85\x9c\xb3\xca\xe1" + "\xf8\x0f\x26\x3d\x54\x6b\x82\x99" + "\xb0\xc7\xde\xf5\x0c\x23\x3a\x51" + "\x68\x7f\x96\xad\xc4\xdb\xf2\x09" + "\x20\x37\x4e\x65\x7c\x93\xaa\xc1" + "\xd8\xef\x06\x1d\x34\x4b\x62\x79" + "\x90\xa7\xbe\xd5\xec\x03\x1a\x31" + "\x48\x5f\x76\x8d\xa4\xbb\xd2\xe9" + "\x00\x19\x32\x4b\x64\x7d\x96\xaf" + "\xc8\xe1\xfa\x13\x2c\x45\x5e\x77" + "\x90\xa9\xc2\xdb\xf4\x0d\x26\x3f" + "\x58\x71\x8a\xa3\xbc\xd5\xee\x07" + "\x20\x39\x52\x6b\x84\x9d\xb6\xcf" + "\xe8\x01\x1a\x33\x4c\x65\x7e\x97" + "\xb0\xc9\xe2\xfb\x14\x2d\x46\x5f" + "\x78\x91\xaa\xc3\xdc\xf5\x0e\x27" + "\x40\x59\x72\x8b\xa4\xbd\xd6\xef" + "\x08\x21\x3a\x53\x6c\x85\x9e\xb7" + "\xd0\xe9\x02\x1b\x34\x4d\x66\x7f" + "\x98\xb1\xca\xe3\xfc\x15\x2e\x47" + "\x60\x79\x92\xab\xc4\xdd\xf6\x0f" + "\x28\x41\x5a\x73\x8c\xa5\xbe\xd7" + "\xf0\x09\x22\x3b\x54\x6d\x86\x9f" + "\xb8\xd1\xea\x03\x1c\x35\x4e\x67" + "\x80\x99\xb2\xcb\xe4\xfd\x16\x2f" + "\x48\x61\x7a\x93\xac\xc5\xde\xf7" + "\x10\x29\x42\x5b\x74\x8d\xa6\xbf" + "\xd8\xf1\x0a\x23\x3c\x55\x6e\x87" + "\xa0\xb9\xd2\xeb\x04\x1d\x36\x4f" + "\x68\x81\x9a\xb3\xcc\xe5\xfe\x17" + "\x30\x49\x62\x7b\x94\xad\xc6\xdf" + "\xf8\x11\x2a\x43\x5c\x75\x8e\xa7" + "\xc0\xd9\xf2\x0b\x24\x3d\x56\x6f" + "\x88\xa1\xba\xd3\xec\x05\x1e\x37" + "\x50\x69\x82\x9b\xb4\xcd\xe6\xff" + "\x18\x31\x4a\x63\x7c\x95\xae\xc7" + "\xe0\xf9\x12\x2b\x44\x5d\x76\x8f" + "\xa8\xc1\xda\xf3\x0c\x25\x3e\x57" + "\x70\x89\xa2\xbb\xd4\xed\x06\x1f" + "\x38\x51\x6a\x83\x9c\xb5\xce\xe7" + "\x00\x1b\x36\x51\x6c\x87\xa2\xbd" + "\xd8\xf3\x0e\x29\x44\x5f\x7a\x95" + "\xb0\xcb\xe6\x01\x1c\x37\x52\x6d" + "\x88\xa3\xbe\xd9\xf4\x0f\x2a\x45" + "\x60\x7b\x96\xb1\xcc\xe7\x02\x1d" + "\x38\x53\x6e\x89\xa4\xbf\xda\xf5" + "\x10\x2b\x46\x61\x7c\x97\xb2\xcd" + "\xe8\x03\x1e\x39\x54\x6f\x8a\xa5" + "\xc0\xdb\xf6\x11\x2c\x47\x62\x7d" + "\x98\xb3\xce\xe9\x04\x1f\x3a\x55" + "\x70\x8b\xa6\xc1\xdc\xf7\x12\x2d" + "\x48\x63\x7e\x99\xb4\xcf\xea\x05" + "\x20\x3b\x56\x71\x8c\xa7\xc2\xdd" + "\xf8\x13\x2e\x49\x64\x7f\x9a\xb5" + "\xd0\xeb\x06\x21\x3c\x57\x72\x8d" + "\xa8\xc3\xde\xf9\x14\x2f\x4a\x65" + "\x80\x9b\xb6\xd1\xec\x07\x22\x3d" + "\x58\x73\x8e\xa9\xc4\xdf\xfa\x15" + "\x30\x4b\x66\x81\x9c\xb7\xd2\xed" + "\x08\x23\x3e\x59\x74\x8f\xaa\xc5" + "\xe0\xfb\x16\x31\x4c\x67\x82\x9d" + "\xb8\xd3\xee\x09\x24\x3f\x5a\x75" + "\x90\xab\xc6\xe1\xfc\x17\x32\x4d" + "\x68\x83\x9e\xb9\xd4\xef\x0a\x25" + "\x40\x5b\x76\x91\xac\xc7\xe2\xfd" + "\x18\x33\x4e\x69\x84\x9f\xba\xd5" + "\xf0\x0b\x26\x41\x5c\x77\x92\xad" + "\xc8\xe3\xfe\x19\x34\x4f\x6a\x85" + "\xa0\xbb\xd6\xf1\x0c\x27\x42\x5d" + "\x78\x93\xae\xc9\xe4\xff\x1a\x35" + "\x50\x6b\x86\xa1\xbc\xd7\xf2\x0d" + "\x28\x43\x5e\x79\x94\xaf\xca\xe5" + "\x00\x1d\x3a\x57\x74\x91\xae\xcb" + "\xe8\x05\x22\x3f\x5c\x79\x96\xb3" + "\xd0\xed\x0a\x27\x44\x61\x7e\x9b" + "\xb8\xd5\xf2\x0f\x2c\x49\x66\x83" + "\xa0\xbd\xda\xf7\x14\x31\x4e\x6b" + "\x88\xa5\xc2\xdf\xfc\x19\x36\x53" + "\x70\x8d\xaa\xc7\xe4\x01\x1e\x3b" + "\x58\x75\x92\xaf\xcc\xe9\x06\x23" + "\x40\x5d\x7a\x97\xb4\xd1\xee\x0b" + "\x28\x45\x62\x7f\x9c\xb9\xd6\xf3" + "\x10\x2d\x4a\x67\x84\xa1\xbe\xdb" + "\xf8\x15\x32\x4f\x6c\x89\xa6\xc3" + "\xe0\xfd\x1a\x37\x54\x71\x8e\xab" + "\xc8\xe5\x02\x1f\x3c\x59\x76\x93" + "\xb0\xcd\xea\x07\x24\x41\x5e\x7b" + "\x98\xb5\xd2\xef\x0c\x29\x46\x63" + "\x80\x9d\xba\xd7\xf4\x11\x2e\x4b" + "\x68\x85\xa2\xbf\xdc\xf9\x16\x33" + "\x50\x6d\x8a\xa7\xc4\xe1\xfe\x1b" + "\x38\x55\x72\x8f\xac\xc9\xe6\x03" + "\x20\x3d\x5a\x77\x94\xb1\xce\xeb" + "\x08\x25\x42\x5f\x7c\x99\xb6\xd3" + "\xf0\x0d\x2a\x47\x64\x81\x9e\xbb" + "\xd8\xf5\x12\x2f\x4c\x69\x86\xa3" + "\xc0\xdd\xfa\x17\x34\x51\x6e\x8b" + "\xa8\xc5\xe2\xff\x1c\x39\x56\x73" + "\x90\xad\xca\xe7\x04\x21\x3e\x5b" + "\x78\x95\xb2\xcf\xec\x09\x26\x43" + "\x60\x7d\x9a\xb7\xd4\xf1\x0e\x2b" + "\x48\x65\x82\x9f\xbc\xd9\xf6\x13" + "\x30\x4d\x6a\x87\xa4\xc1\xde\xfb" + "\x18\x35\x52\x6f\x8c\xa9\xc6\xe3" + "\x00\x1f\x3e\x5d\x7c\x9b\xba\xd9" + "\xf8\x17\x36\x55\x74\x93\xb2\xd1" + "\xf0\x0f\x2e\x4d\x6c\x8b\xaa\xc9" + "\xe8\x07\x26\x45\x64\x83\xa2\xc1" + "\xe0\xff\x1e\x3d\x5c\x7b\x9a\xb9" + "\xd8\xf7\x16\x35\x54\x73\x92\xb1" + "\xd0\xef\x0e\x2d\x4c\x6b\x8a\xa9" + "\xc8\xe7\x06\x25\x44\x63\x82\xa1" + "\xc0\xdf\xfe\x1d\x3c\x5b\x7a\x99" + "\xb8\xd7\xf6\x15\x34\x53\x72\x91" + "\xb0\xcf\xee\x0d\x2c\x4b\x6a\x89" + "\xa8\xc7\xe6\x05\x24\x43\x62\x81" + "\xa0\xbf\xde\xfd\x1c\x3b\x5a\x79" + "\x98\xb7\xd6\xf5\x14\x33\x52\x71" + "\x90\xaf\xce\xed\x0c\x2b\x4a\x69" + "\x88\xa7\xc6\xe5\x04\x23\x42\x61" + "\x80\x9f\xbe\xdd\xfc\x1b\x3a\x59" + "\x78\x97\xb6\xd5\xf4\x13\x32\x51" + "\x70\x8f\xae\xcd\xec\x0b\x2a\x49" + "\x68\x87\xa6\xc5\xe4\x03\x22\x41" + "\x60\x7f\x9e\xbd\xdc\xfb\x1a\x39" + "\x58\x77\x96\xb5\xd4\xf3\x12\x31" + "\x50\x6f\x8e\xad\xcc\xeb\x0a\x29" + "\x48\x67\x86\xa5\xc4\xe3\x02\x21" + "\x40\x5f\x7e\x9d\xbc\xdb\xfa\x19" + "\x38\x57\x76\x95\xb4\xd3\xf2\x11" + "\x30\x4f\x6e\x8d\xac\xcb\xea\x09" + "\x28\x47\x66\x85\xa4\xc3\xe2\x01" + "\x20\x3f\x5e\x7d\x9c\xbb\xda\xf9" + "\x18\x37\x56\x75\x94\xb3\xd2\xf1" + "\x10\x2f\x4e\x6d\x8c\xab\xca\xe9" + "\x08\x27\x46\x65\x84\xa3\xc2\xe1" + "\x00\x21\x42\x63", + .ctext = + "\xf0\x5c\x74\xad\x4e\xbc\x99\xe2" + "\xae\xff\x91\x3a\x44\xcf\x38\x32" + "\x1e\xad\xa7\xcd\xa1\x39\x95\xaa" + "\x10\xb1\xb3\x2e\x04\x31\x8f\x86" + "\xf2\x62\x74\x70\x0c\xa4\x46\x08" + "\xa8\xb7\x99\xa8\xe9\xd2\x73\x79" + "\x7e\x6e\xd4\x8f\x1e\xc7\x8e\x31" + "\x0b\xfa\x4b\xce\xfd\xf3\x57\x71" + "\xe9\x46\x03\xa5\x3d\x34\x00\xe2" + "\x18\xff\x75\x6d\x06\x2d\x00\xab" + "\xb9\x3e\x6c\x59\xc5\x84\x06\xb5" + "\x8b\xd0\x89\x9c\x4a\x79\x16\xc6" + "\x3d\x74\x54\xfa\x44\xcd\x23\x26" + "\x5c\xcf\x7e\x28\x92\x32\xbf\xdf" + "\xa7\x20\x3c\x74\x58\x2a\x9a\xde" + "\x61\x00\x1c\x4f\xff\x59\xc4\x22" + "\xac\x3c\xd0\xe8\x6c\xf9\x97\x1b" + "\x58\x9b\xad\x71\xe8\xa9\xb5\x0d" + "\xee\x2f\x04\x1f\x7f\xbc\x99\xee" + "\x84\xff\x42\x60\xdc\x3a\x18\xa5" + "\x81\xf9\xef\xdc\x7a\x0f\x65\x41" + "\x2f\xa3\xd3\xf9\xc2\xcb\xc0\x4d" + "\x8f\xd3\x76\x96\xad\x49\x6d\x38" + "\x3d\x39\x0b\x6c\x80\xb7\x54\x69" + "\xf0\x2c\x90\x02\x29\x0d\x1c\x12" + "\xad\x55\xc3\x8b\x68\xd9\xcc\xb3" + "\xb2\x64\x33\x90\x5e\xca\x4b\xe2" + "\xfb\x75\xdc\x63\xf7\x9f\x82\x74" + "\xf0\xc9\xaa\x7f\xe9\x2a\x9b\x33" + "\xbc\x88\x00\x7f\xca\xb2\x1f\x14" + "\xdb\xc5\x8e\x7b\x11\x3c\x3e\x08" + "\xf3\x83\xe8\xe0\x94\x86\x2e\x92" + "\x78\x6b\x01\xc9\xc7\x83\xba\x21" + "\x6a\x25\x15\x33\x4e\x45\x08\xec" + "\x35\xdb\xe0\x6e\x31\x51\x79\xa9" + "\x42\x44\x65\xc1\xa0\xf1\xf9\x2a" + "\x70\xd5\xb6\xc6\xc1\x8c\x39\xfc" + "\x25\xa6\x55\xd9\xdd\x2d\x4c\xec" + "\x49\xc6\xeb\x0e\xa8\x25\x2a\x16" + "\x1b\x66\x84\xda\xe2\x92\xe5\xc0" + "\xc8\x53\x07\xaf\x80\x84\xec\xfd" + "\xcd\xd1\x6e\xcd\x6f\x6a\xf5\x36" + "\xc5\x15\xe5\x25\x7d\x77\xd1\x1a" + "\x93\x36\xa9\xcf\x7c\xa4\x54\x4a" + "\x06\x51\x48\x4e\xf6\x59\x87\xd2" + "\x04\x02\xef\xd3\x44\xde\x76\x31" + "\xb3\x34\x17\x1b\x9d\x66\x11\x9f" + "\x1e\xcc\x17\xe9\xc7\x3c\x1b\xe7" + "\xcb\x50\x08\xfc\xdc\x2b\x24\xdb" + "\x65\x83\xd0\x3b\xe3\x30\xea\x94" + "\x6c\xe7\xe8\x35\x32\xc7\xdb\x64" + "\xb4\x01\xab\x36\x2c\x77\x13\xaf" + "\xf8\x2b\x88\x3f\x54\x39\xc4\x44" + "\xfe\xef\x6f\x68\x34\xbe\x0f\x05" + "\x16\x6d\xf6\x0a\x30\xe7\xe3\xed" + "\xc4\xde\x3c\x1b\x13\xd8\xdb\xfe" + "\x41\x62\xe5\x28\xd4\x8d\xa3\xc7" + "\x93\x97\xc6\x48\x45\x1d\x9f\x83" + "\xdf\x4b\x40\x3e\x42\x25\x87\x80" + "\x4c\x7d\xa8\xd4\x98\x23\x95\x75" + "\x41\x8c\xda\x41\x9b\xd4\xa7\x06" + "\xb5\xf1\x71\x09\x53\xbe\xca\xbf" + "\x32\x03\xed\xf0\x50\x1c\x56\x39" + "\x5b\xa4\x75\x18\xf7\x9b\x58\xef" + "\x53\xfc\x2a\x38\x23\x15\x75\xcd" + "\x45\xe5\x5a\x82\x55\xba\x21\xfa" + "\xd4\xbd\xc6\x94\x7c\xc5\x80\x12" + "\xf7\x4b\x32\xc4\x9a\x82\xd8\x28" + "\x8f\xd9\xc2\x0f\x60\x03\xbe\x5e" + "\x21\xd6\x5f\x58\xbf\x5c\xb1\x32" + "\x82\x8d\xa9\xe5\xf2\x66\x1a\xc0" + "\xa0\xbc\x58\x2f\x71\xf5\x2f\xed" + "\xd1\x26\xb9\xd8\x49\x5a\x07\x19" + "\x01\x7c\x59\xb0\xf8\xa4\xb7\xd3" + "\x7b\x1a\x8c\x38\xf4\x50\xa4\x59" + "\xb0\xcc\x41\x0b\x88\x7f\xe5\x31" + "\xb3\x42\xba\xa2\x7e\xd4\x32\x71" + "\x45\x87\x48\xa9\xc2\xf2\x89\xb3" + "\xe4\xa7\x7e\x52\x15\x61\xfa\xfe" + "\xc9\xdd\x81\xeb\x13\xab\xab\xc3" + "\x98\x59\xd8\x16\x3d\x14\x7a\x1c" + "\x3c\x41\x9a\x16\x16\x9b\xd2\xd2" + "\x69\x3a\x29\x23\xac\x86\x32\xa5" + "\x48\x9c\x9e\xf3\x47\x77\x81\x70" + "\x24\xe8\x85\xd2\xf5\xb5\xfa\xff" + "\x59\x6a\xd3\x50\x59\x43\x59\xde" + "\xd9\xf1\x55\xa5\x0c\xc3\x1a\x1a" + "\x18\x34\x0d\x1a\x63\x33\xed\x10" + "\xe0\x1d\x2a\x18\xd2\xc0\x54\xa8" + "\xca\xb5\x9a\xd3\xdd\xca\x45\x84" + "\x50\xe7\x0f\xfe\xa4\x99\x5a\xbe" + "\x43\x2d\x9a\xcb\x92\x3f\x5a\x1d" + "\x85\xd8\xc9\xdf\x68\xc9\x12\x80" + "\x56\x0c\xdc\x00\xdc\x3a\x7d\x9d" + "\xa3\xa2\xe8\x4d\xbf\xf9\x70\xa0" + "\xa4\x13\x4f\x6b\xaf\x0a\x89\x7f" + "\xda\xf0\xbf\x9b\xc8\x1d\xe5\xf8" + "\x2e\x8b\x07\xb5\x73\x1b\xcc\xa2" + "\xa6\xad\x30\xbc\x78\x3c\x5b\x10" + "\xfa\x5e\x62\x2d\x9e\x64\xb3\x33" + "\xce\xf9\x1f\x86\xe7\x8b\xa2\xb8" + "\xe8\x99\x57\x8c\x11\xed\x66\xd9" + "\x3c\x72\xb9\xc3\xe6\x4e\x17\x3a" + "\x6a\xcb\x42\x24\x06\xed\x3e\x4e" + "\xa3\xe8\x6a\x94\xda\x0d\x4e\xd5" + "\x14\x19\xcf\xb6\x26\xd8\x2e\xcc" + "\x64\x76\x38\x49\x4d\xfe\x30\x6d" + "\xe4\xc8\x8c\x7b\xc4\xe0\x35\xba" + "\x22\x6e\x76\xe1\x1a\xf2\x53\xc3" + "\x28\xa2\x82\x1f\x61\x69\xad\xc1" + "\x7b\x28\x4b\x1e\x6c\x85\x95\x9b" + "\x51\xb5\x17\x7f\x12\x69\x8c\x24" + "\xd5\xc7\x5a\x5a\x11\x54\xff\x5a" + "\xf7\x16\xc3\x91\xa6\xf0\xdc\x0a" + "\xb6\xa7\x4a\x0d\x7a\x58\xfe\xa5" + "\xf5\xcb\x8f\x7b\x0e\xea\x57\xe7" + "\xbd\x79\xd6\x1c\x88\x23\x6c\xf2" + "\x4d\x29\x77\x53\x35\x6a\x00\x8d" + "\xcd\xa3\x58\xbe\x77\x99\x18\xf8" + "\xe6\xe1\x8f\xe9\x37\x8f\xe3\xe2" + "\x5a\x8a\x93\x25\xaf\xf3\x78\x80" + "\xbe\xa6\x1b\xc6\xac\x8b\x1c\x91" + "\x58\xe1\x9f\x89\x35\x9d\x1d\x21" + "\x29\x9f\xf4\x99\x02\x27\x0f\xa8" + "\x4f\x79\x94\x2b\x33\x2c\xda\xa2" + "\x26\x39\x83\x94\xef\x27\xd8\x53" + "\x8f\x66\x0d\xe4\x41\x7d\x34\xcd" + "\x43\x7c\x95\x0a\x53\xef\x66\xda" + "\x7e\x9b\xf3\x93\xaf\xd0\x73\x71" + "\xba\x40\x9b\x74\xf8\xd7\xd7\x41" + "\x6d\xaf\x72\x9c\x8d\x21\x87\x3c" + "\xfd\x0a\x90\xa9\x47\x96\x9e\xd3" + "\x88\xee\x73\xcf\x66\x2f\x52\x56" + "\x6d\xa9\x80\x4c\xe2\x6f\x62\x88" + "\x3f\x0e\x54\x17\x48\x80\x5d\xd3" + "\xc3\xda\x25\x3d\xa1\xc8\xcb\x9f" + "\x9b\x70\xb3\xa1\xeb\x04\x52\xa1" + "\xf2\x22\x0f\xfc\xc8\x18\xfa\xf9" + "\x85\x9c\xf1\xac\xeb\x0c\x02\x46" + "\x75\xd2\xf5\x2c\xe3\xd2\x59\x94" + "\x12\xf3\x3c\xfc\xd7\x92\xfa\x36" + "\xba\x61\x34\x38\x7c\xda\x48\x3e" + "\x08\xc9\x39\x23\x5e\x02\x2c\x1a" + "\x18\x7e\xb4\xd9\xfd\x9e\x40\x02" + "\xb1\x33\x37\x32\xe7\xde\xd6\xd0" + "\x7c\x58\x65\x4b\xf8\x34\x27\x9c" + "\x44\xb4\xbd\xe9\xe9\x4c\x78\x7d" + "\x4b\x9f\xce\xb1\xcd\x47\xa5\x37" + "\xe5\x6d\xbd\xb9\x43\x94\x0a\xd4" + "\xd6\xf9\x04\x5f\xb5\x66\x6c\x1a" + "\x35\x12\xe3\x36\x28\x27\x36\x58" + "\x01\x2b\x79\xe4\xba\x6d\x10\x7d" + "\x65\xdf\x84\x95\xf4\xd5\xb6\x8f" + "\x2b\x9f\x96\x00\x86\x60\xf0\x21" + "\x76\xa8\x6a\x8c\x28\x1c\xb3\x6b" + "\x97\xd7\xb6\x53\x2a\xcc\xab\x40" + "\x9d\x62\x79\x58\x52\xe6\x65\xb7" + "\xab\x55\x67\x9c\x89\x7c\x03\xb0" + "\x73\x59\xc5\x81\xf5\x18\x17\x5c" + "\x89\xf3\x78\x35\x44\x62\x78\x72" + "\xd0\x96\xeb\x31\xe7\x87\x77\x14" + "\x99\x51\xf2\x59\x26\x9e\xb5\xa6" + "\x45\xfe\x6e\xbd\x07\x4c\x94\x5a" + "\xa5\x7d\xfc\xf1\x2b\x77\xe2\xfe" + "\x17\xd4\x84\xa0\xac\xb5\xc7\xda" + "\xa9\x1a\xb6\xf3\x74\x11\xb4\x9d" + "\xfb\x79\x2e\x04\x2d\x50\x28\x83" + "\xbf\xc6\x52\xd3\x34\xd6\xe8\x7a" + "\xb6\xea\xe7\xa8\x6c\x15\x1e\x2c" + "\x57\xbc\x48\x4e\x5f\x5c\xb6\x92" + "\xd2\x49\x77\x81\x6d\x90\x70\xae" + "\x98\xa1\x03\x0d\x6b\xb9\x77\x14" + "\xf1\x4e\x23\xd3\xf8\x68\xbd\xc2" + "\xfe\x04\xb7\x5c\xc5\x17\x60\x8f" + "\x65\x54\xa4\x7a\x42\xdc\x18\x0d" + "\xb5\xcf\x0f\xd3\xc7\x91\x66\x1b" + "\x45\x42\x27\x75\x50\xe5\xee\xb8" + "\x7f\x33\x2c\xba\x4a\x92\x4d\x2c" + "\x3c\xe3\x0d\x80\x01\xba\x0d\x29" + "\xd8\x3c\xe9\x13\x16\x57\xe6\xea" + "\x94\x52\xe7\x00\x4d\x30\xb0\x0f" + "\x35\xb8\xb8\xa7\xb1\xb5\x3b\x44" + "\xe1\x2f\xfd\x88\xed\x43\xe7\x52" + "\x10\x93\xb3\x8a\x30\x6b\x0a\xf7" + "\x23\xc6\x50\x9d\x4a\xb0\xde\xc3" + "\xdc\x9b\x2f\x01\x56\x36\x09\xc5" + "\x2f\x6b\xfe\xf1\xd8\x27\x45\x03" + "\x30\x5e\x5c\x5b\xb4\x62\x0e\x1a" + "\xa9\x21\x2b\x92\x94\x87\x62\x57" + "\x4c\x10\x74\x1a\xf1\x0a\xc5\x84" + "\x3b\x9e\x72\x02\xd7\xcc\x09\x56" + "\xbd\x54\xc1\xf0\xc3\xe3\xb3\xf8" + "\xd2\x0d\x61\xcb\xef\xce\x0d\x05" + "\xb0\x98\xd9\x8e\x4f\xf9\xbc\x93" + "\xa6\xea\xc8\xcf\x10\x53\x4b\xf1" + "\xec\xfc\x89\xf9\x64\xb0\x22\xbf" + "\x9e\x55\x46\x9f\x7c\x50\x8e\x84" + "\x54\x20\x98\xd7\x6c\x40\x1e\xdb" + "\x69\x34\x78\x61\x24\x21\x9c\x8a" + "\xb3\x62\x31\x8b\x6e\xf5\x2a\x35" + "\x86\x13\xb1\x6c\x64\x2e\x41\xa5" + "\x05\xf2\x42\xba\xd2\x3a\x0d\x8e" + "\x8a\x59\x94\x3c\xcf\x36\x27\x82" + "\xc2\x45\xee\x58\xcd\x88\xb4\xec" + "\xde\xb2\x96\x0a\xaf\x38\x6f\x88" + "\xd7\xd8\xe1\xdf\xb9\x96\xa9\x0a" + "\xb1\x95\x28\x86\x20\xe9\x17\x49" + "\xa2\x29\x38\xaa\xa5\xe9\x6e\xf1" + "\x19\x27\xc0\xd5\x2a\x22\xc3\x0b" + "\xdb\x7c\x73\x10\xb9\xba\x89\x76" + "\x54\xae\x7d\x71\xb3\x93\xf6\x32" + "\xe6\x47\x43\x55\xac\xa0\x0d\xc2" + "\x93\x27\x4a\x8e\x0e\x74\x15\xc7" + "\x0b\x85\xd9\x0c\xa9\x30\x7a\x3e" + "\xea\x8f\x85\x6d\x3a\x12\x4f\x72" + "\x69\x58\x7a\x80\xbb\xb5\x97\xf3" + "\xcf\x70\xd2\x5d\xdd\x4d\x21\x79" + "\x54\x4d\xe4\x05\xe8\xbd\xc2\x62" + "\xb1\x3b\x77\x1c\xd6\x5c\xf3\xa0" + "\x79\x00\xa8\x6c\x29\xd9\x18\x24" + "\x36\xa2\x46\xc0\x96\x65\x7f\xbd" + "\x2a\xed\x36\x16\x0c\xaa\x9f\xf4" + "\xc5\xb4\xe2\x12\xed\x69\xed\x4f" + "\x26\x2c\x39\x52\x89\x98\xe7\x2c" + "\x99\xa4\x9e\xa3\x9b\x99\x46\x7a" + "\x3a\xdc\xa8\x59\xa3\xdb\xc3\x3b" + "\x95\x0d\x3b\x09\x6e\xee\x83\x5d" + "\x32\x4d\xed\xab\xfa\x98\x14\x4e" + "\xc3\x15\x45\x53\x61\xc4\x93\xbd" + "\x90\xf4\x99\x95\x4c\xe6\x76\x92" + "\x29\x90\x46\x30\x92\x69\x7d\x13" + "\xf2\xa5\xcd\x69\x49\x44\xb2\x0f" + "\x63\x40\x36\x5f\x09\xe2\x78\xf8" + "\x91\xe3\xe2\xfa\x10\xf7\xc8\x24" + "\xa8\x89\x32\x5c\x37\x25\x1d\xb2" + "\xea\x17\x8a\x0a\xa9\x64\xc3\x7c" + "\x3c\x7c\xbd\xc6\x79\x34\xe7\xe2" + "\x85\x8e\xbf\xf8\xde\x92\xa0\xae" + "\x20\xc4\xf6\xbb\x1f\x38\x19\x0e" + "\xe8\x79\x9c\xa1\x23\xe9\x54\x7e" + "\x37\x2f\xe2\x94\x32\xaf\xa0\x23" + "\x49\xe4\xc0\xb3\xac\x00\x8f\x36" + "\x05\xc4\xa6\x96\xec\x05\x98\x4f" + "\x96\x67\x57\x1f\x20\x86\x1b\x2d" + "\x69\xe4\x29\x93\x66\x5f\xaf\x6b" + "\x88\x26\x2c\x67\x02\x4b\x52\xd0" + "\x83\x7a\x43\x1f\xc0\x71\x15\x25" + "\x77\x65\x08\x60\x11\x76\x4c\x8d" + "\xed\xa9\x27\xc6\xb1\x2a\x2c\x6a" + "\x4a\x97\xf5\xc6\xb7\x70\x42\xd3" + "\x03\xd1\x24\x95\xec\x6d\xab\x38" + "\x72\xce\xe2\x8b\x33\xd7\x51\x09" + "\xdc\x45\xe0\x09\x96\x32\xf3\xc4" + "\x84\xdc\x73\x73\x2d\x1b\x11\x98" + "\xc5\x0e\x69\x28\x94\xc7\xb5\x4d" + "\xc8\x8a\xd0\xaa\x13\x2e\x18\x74" + "\xdd\xd1\x1e\xf3\x90\xe8\xfc\x9a" + "\x72\x4a\x0e\xd1\xe4\xfb\x0d\x96" + "\xd1\x0c\x79\x85\x1b\x1c\xfe\xe1" + "\x62\x8f\x7a\x73\x32\xab\xc8\x18" + "\x69\xe3\x34\x30\xdf\x13\xa6\xe5" + "\xe8\x0e\x67\x7f\x81\x11\xb4\x60" + "\xc7\xbd\x79\x65\x50\xdc\xc4\x5b" + "\xde\x39\xa4\x01\x72\x63\xf3\xd1" + "\x64\x4e\xdf\xfc\x27\x92\x37\x0d" + "\x57\xcd\x11\x4f\x11\x04\x8e\x1d" + "\x16\xf7\xcd\x92\x9a\x99\x30\x14" + "\xf1\x7c\x67\x1b\x1f\x41\x0b\xe8" + "\x32\xe8\xb8\xc1\x4f\x54\x86\x4f" + "\xe5\x79\x81\x73\xcd\x43\x59\x68" + "\x73\x02\x3b\x78\x21\x72\x43\x00" + "\x49\x17\xf7\x00\xaf\x68\x24\x53" + "\x05\x0a\xc3\x33\xe0\x33\x3f\x69" + "\xd2\x84\x2f\x0b\xed\xde\x04\xf4" + "\x11\x94\x13\x69\x51\x09\x28\xde" + "\x57\x5c\xef\xdc\x9a\x49\x1c\x17" + "\x97\xf3\x96\xc1\x7f\x5d\x2e\x7d" + "\x55\xb8\xb3\x02\x09\xb3\x1f\xe7" + "\xc9\x8d\xa3\x36\x34\x8a\x77\x13" + "\x30\x63\x4c\xa5\xcd\xc3\xe0\x7e" + "\x05\xa1\x7b\x0c\xcb\x74\x47\x31" + "\x62\x03\x43\xf1\x87\xb4\xb0\x85" + "\x87\x8e\x4b\x25\xc7\xcf\xae\x4b" + "\x36\x46\x3e\x62\xbc\x6f\xeb\x5f" + "\x73\xac\xe6\x07\xee\xc1\xa1\xd6" + "\xc4\xab\xc9\xd6\x89\x45\xe1\xf1" + "\x04\x4e\x1a\x6f\xbb\x4f\x3a\xa3" + "\xa0\xcb\xa3\x0a\xd8\x71\x35\x55" + "\xe4\xbc\x2e\x04\x06\xe6\xff\x5b" + "\x1c\xc0\x11\x7c\xc5\x17\xf3\x38" + "\xcf\xe9\xba\x0f\x0e\xef\x02\xc2" + "\x8d\xc6\xbc\x4b\x67\x20\x95\xd7" + "\x2c\x45\x5b\x86\x44\x8c\x6f\x2e" + "\x7e\x9f\x1c\x77\xba\x6b\x0e\xa3" + "\x69\xdc\xab\x24\x57\x60\x47\xc1" + "\xd1\xa5\x9d\x23\xe6\xb1\x37\xfe" + "\x93\xd2\x4c\x46\xf9\x0c\xc6\xfb" + "\xd6\x9d\x99\x69\xab\x7a\x07\x0c" + "\x65\xe7\xc4\x08\x96\xe2\xa5\x01" + "\x3f\x46\x07\x05\x7e\xe8\x9a\x90" + "\x50\xdc\xe9\x7a\xea\xa1\x39\x6e" + "\x66\xe4\x6f\xa5\x5f\xb2\xd9\x5b" + "\xf5\xdb\x2a\x32\xf0\x11\x6f\x7c" + "\x26\x10\x8f\x3d\x80\xe9\x58\xf7" + "\xe0\xa8\x57\xf8\xdb\x0e\xce\x99" + "\x63\x19\x3d\xd5\xec\x1b\x77\x69" + "\x98\xf6\xe4\x5f\x67\x17\x4b\x09" + "\x85\x62\x82\x70\x18\xe2\x9a\x78" + "\xe2\x62\xbd\xb4\xf1\x42\xc6\xfb" + "\x08\xd0\xbd\xeb\x4e\x09\xf2\xc8" + "\x1e\xdc\x3d\x32\x21\x56\x9c\x4f" + "\x35\xf3\x61\x06\x72\x84\xc4\x32" + "\xf2\xf1\xfa\x0b\x2f\xc3\xdb\x02" + "\x04\xc2\xde\x57\x64\x60\x8d\xcf" + "\xcb\x86\x5d\x97\x3e\xb1\x9c\x01" + "\xd6\x28\x8f\x99\xbc\x46\xeb\x05" + "\xaf\x7e\xb8\x21\x2a\x56\x85\x1c" + "\xb3\x71\xa0\xde\xca\x96\xf1\x78" + "\x49\xa2\x99\x81\x80\x5c\x01\xf5" + "\xa0\xa2\x56\x63\xe2\x70\x07\xa5" + "\x95\xd6\x85\xeb\x36\x9e\xa9\x51" + "\x66\x56\x5f\x1d\x02\x19\xe2\xf6" + "\x4f\x73\x38\x09\x75\x64\x48\xe0" + "\xf1\x7e\x0e\xe8\x9d\xf9\xed\x94" + "\xfe\x16\x26\x62\x49\x74\xf4\xb0" + "\xd4\xa9\x6c\xb0\xfd\x53\xe9\x81" + "\xe0\x7a\xbf\xcf\xb5\xc4\x01\x81" + "\x79\x99\x77\x01\x3b\xe9\xa2\xb6" + "\xe6\x6a\x8a\x9e\x56\x1c\x8d\x1e" + "\x8f\x06\x55\x2c\x6c\xdc\x92\x87" + "\x64\x3b\x4b\x19\xa1\x13\x64\x1d" + "\x4a\xe9\xc0\x00\xb8\x95\xef\x6b" + "\x1a\x86\x6d\x37\x52\x02\xc2\xe0" + "\xc8\xbb\x42\x0c\x02\x21\x4a\xc9" + "\xef\xa0\x54\xe4\x5e\x16\x53\x81" + "\x70\x62\x10\xaf\xde\xb8\xb5\xd3" + "\xe8\x5e\x6c\xc3\x8a\x3e\x18\x07" + "\xf2\x2f\x7d\xa7\xe1\x3d\x4e\xb4" + "\x26\xa7\xa3\x93\x86\xb2\x04\x1e" + "\x53\x5d\x86\xd6\xde\x65\xca\xe3" + "\x4e\xc1\xcf\xef\xc8\x70\x1b\x83" + "\x13\xdd\x18\x8b\x0d\x76\xd2\xf6" + "\x37\x7a\x93\x7a\x50\x11\x9f\x96" + "\x86\x25\xfd\xac\xdc\xbe\x18\x93" + "\x19\x6b\xec\x58\x4f\xb9\x75\xa7" + "\xdd\x3f\x2f\xec\xc8\x5a\x84\xab" + "\xd5\xe4\x8a\x07\xf6\x4d\x23\xd6" + "\x03\xfb\x03\x6a\xea\x66\xbf\xd4" + "\xb1\x34\xfb\x78\xe9\x55\xdc\x7c" + "\x3d\x9c\xe5\x9a\xac\xc3\x7a\x80" + "\x24\x6d\xa0\xef\x25\x7c\xb7\xea" + "\xce\x4d\x5f\x18\x60\xce\x87\x22" + "\x66\x2f\xd5\xdd\xdd\x02\x21\x75" + "\x82\xa0\x1f\x58\xc6\xd3\x62\xf7" + "\x32\xd8\xaf\x1e\x07\x77\x51\x96" + "\xd5\x6b\x1e\x7e\x80\x02\xe8\x67" + "\xea\x17\x0b\x10\xd2\x3f\x28\x25" + "\x4f\x05\x77\x02\x14\x69\xf0\x2c" + "\xbe\x0c\xf1\x74\x30\xd1\xb9\x9b" + "\xfc\x8c\xbb\x04\x16\xd9\xba\xc3" + "\xbc\x91\x8a\xc4\x30\xa4\xb0\x12" + "\x4c\x21\x87\xcb\xc9\x1d\x16\x96" + "\x07\x6f\x23\x54\xb9\x6f\x79\xe5" + "\x64\xc0\x64\xda\xb1\xae\xdd\x60" + "\x6c\x1a\x9d\xd3\x04\x8e\x45\xb0" + "\x92\x61\xd0\x48\x81\xed\x5e\x1d" + "\xa0\xc9\xa4\x33\xc7\x13\x51\x5d" + "\x7f\x83\x73\xb6\x70\x18\x65\x3e" + "\x2f\x0e\x7a\x12\x39\x98\xab\xd8" + "\x7e\x6f\xa3\xd1\xba\x56\xad\xbd" + "\xf0\x03\x01\x1c\x85\x35\x9f\xeb" + "\x19\x63\xa1\xaf\xfe\x2d\x35\x50" + "\x39\xa0\x65\x7c\x95\x7e\x6b\xfe" + "\xc1\xac\x07\x7c\x98\x4f\xbe\x57" + "\xa7\x22\xec\xe2\x7e\x29\x09\x53" + "\xe8\xbf\xb4\x7e\x3f\x8f\xfc\x14" + "\xce\x54\xf9\x18\x58\xb5\xff\x44" + "\x05\x9d\xce\x1b\xb6\x82\x23\xc8" + "\x2e\xbc\x69\xbb\x4a\x29\x0f\x65" + "\x94\xf0\x63\x06\x0e\xef\x8c\xbd" + "\xff\xfd\xb0\x21\x6e\x57\x05\x75" + "\xda\xd5\xc4\xeb\x8d\x32\xf7\x50" + "\xd3\x6f\x22\xed\x5f\x8e\xa2\x5b" + "\x80\x8c\xc8\x78\x40\x24\x4b\x89" + "\x30\xce\x7a\x97\x0e\xc4\xaf\xef" + "\x9b\xb4\xcd\x66\x74\x14\x04\x2b" + "\xf7\xce\x0b\x1c\x6e\xc2\x78\x8c" + "\xca\xc5\xd0\x1c\x95\x4a\x91\x2d" + "\xa7\x20\xeb\x86\x52\xb7\x67\xd8" + "\x0c\xd6\x04\x14\xde\x51\x74\x75" + "\xe7\x11\xb4\x87\xa3\x3d\x2d\xad" + "\x4f\xef\xa0\x0f\x70\x00\x6d\x13" + "\x19\x1d\x41\x50\xe9\xd8\xf0\x32" + "\x71\xbc\xd3\x11\xf2\xac\xbe\xaf" + "\x75\x46\x65\x4e\x07\x34\x37\xa3" + "\x89\xfe\x75\xd4\x70\x4c\xc6\x3f" + "\x69\x24\x0e\x38\x67\x43\x8c\xde" + "\x06\xb5\xb8\xe7\xc4\xf0\x41\x8f" + "\xf0\xbd\x2f\x0b\xb9\x18\xf8\xde" + "\x64\xb1\xdb\xee\x00\x50\x77\xe1" + "\xc7\xff\xa6\xfa\xdd\x70\xf4\xe3" + "\x93\xe9\x77\x35\x3d\x4b\x2f\x2b" + "\x6d\x55\xf0\xfc\x88\x54\x4e\x89" + "\xc1\x8a\x23\x31\x2d\x14\x2a\xb8" + "\x1b\x15\xdd\x9e\x6e\x7b\xda\x05" + "\x91\x7d\x62\x64\x96\x72\xde\xfc" + "\xc1\xec\xf0\x23\x51\x6f\xdb\x5b" + "\x1d\x08\x57\xce\x09\xb8\xf6\xcd" + "\x8d\x95\xf2\x20\xbf\x0f\x20\x57" + "\x98\x81\x84\x4f\x15\x5c\x76\xe7" + "\x3e\x0a\x3a\x6c\xc4\x8a\xbe\x78" + "\x74\x77\xc3\x09\x4b\x5d\x48\xe4" + "\xc8\xcb\x0b\xea\x17\x28\xcf\xcf" + "\x31\x32\x44\xa4\xe5\x0e\x1a\x98" + "\x94\xc4\xf0\xff\xae\x3e\x44\xe8" + "\xa5\xb3\xb5\x37\x2f\xe8\xaf\x6f" + "\x28\xc1\x37\x5f\x31\xd2\xb9\x33" + "\xb1\xb2\x52\x94\x75\x2c\x29\x59" + "\x06\xc2\x25\xe8\x71\x65\x4e\xed" + "\xc0\x9c\xb1\xbb\x25\xdc\x6c\xe7" + "\x4b\xa5\x7a\x54\x7a\x60\xff\x7a" + "\xe0\x50\x40\x96\x35\x63\xe4\x0b" + "\x76\xbd\xa4\x65\x00\x1b\x57\x88" + "\xae\xed\x39\x88\x42\x11\x3c\xed" + "\x85\x67\x7d\xb9\x68\x82\xe9\x43" + "\x3c\x47\x53\xfa\xe8\xf8\x9f\x1f" + "\x9f\xef\x0f\xf7\x30\xd9\x30\x0e" + "\xb9\x9f\x69\x18\x2f\x7e\xf8\xf8" + "\xf8\x8c\x0f\xd4\x02\x4d\xea\xcd" + "\x0a\x9c\x6f\x71\x6d\x5a\x4c\x60" + "\xce\x20\x56\x32\xc6\xc5\x99\x1f" + "\x09\xe6\x4e\x18\x1a\x15\x13\xa8" + "\x7d\xb1\x6b\xc0\xb2\x6d\xf8\x26" + "\x66\xf8\x3d\x18\x74\x70\x66\x7a" + "\x34\x17\xde\xba\x47\xf1\x06\x18" + "\xcb\xaf\xeb\x4a\x1e\x8f\xa7\x77" + "\xe0\x3b\x78\x62\x66\xc9\x10\xea" + "\x1f\xb7\x29\x0a\x45\xa1\x1d\x1e" + "\x1d\xe2\x65\x61\x50\x9c\xd7\x05" + "\xf2\x0b\x5b\x12\x61\x02\xc8\xe5" + "\x63\x4f\x20\x0c\x07\x17\x33\x5e" + "\x03\x9a\x53\x0f\x2e\x55\xfe\x50" + "\x43\x7d\xd0\xb6\x7e\x5a\xda\xae" + "\x58\xef\x15\xa9\x83\xd9\x46\xb1" + "\x42\xaa\xf5\x02\x6c\xce\x92\x06" + "\x1b\xdb\x66\x45\x91\x79\xc2\x2d" + "\xe6\x53\xd3\x14\xfd\xbb\x44\x63" + "\xc6\xd7\x3d\x7a\x0c\x75\x78\x9d" + "\x5c\xa6\x39\xb3\xe5\x63\xca\x8b" + "\xfe\xd3\xef\x60\x83\xf6\x8e\x70" + "\xb6\x67\xc7\x77\xed\x23\xef\x4c" + "\xf0\xed\x2d\x07\x59\x6f\xc1\x01" + "\x34\x37\x08\xab\xd9\x1f\x09\xb1" + "\xce\x5b\x17\xff\x74\xf8\x9c\xd5" + "\x2c\x56\x39\x79\x0f\x69\x44\x75" + "\x58\x27\x01\xc4\xbf\xa7\xa1\x1d" + "\x90\x17\x77\x86\x5a\x3f\xd9\xd1" + "\x0e\xa0\x10\xf8\xec\x1e\xa5\x7f" + "\x5e\x36\xd1\xe3\x04\x2c\x70\xf7" + "\x8e\xc0\x98\x2f\x6c\x94\x2b\x41" + "\xb7\x60\x00\xb7\x2e\xb8\x02\x8d" + "\xb8\xb0\xd3\x86\xba\x1d\xd7\x90" + "\xd6\xb6\xe1\xfc\xd7\xd8\x28\x06" + "\x63\x9b\xce\x61\x24\x79\xc0\x70" + "\x52\xd0\xb6\xd4\x28\x95\x24\x87" + "\x03\x1f\xb7\x9a\xda\xa3\xfb\x52" + "\x5b\x68\xe7\x4c\x8c\x24\xe1\x42" + "\xf7\xd5\xfd\xad\x06\x32\x9f\xba" + "\xc1\xfc\xdd\xc6\xfc\xfc\xb3\x38" + "\x74\x56\x58\x40\x02\x37\x52\x2c" + "\x55\xcc\xb3\x9e\x7a\xe9\xd4\x38" + "\x41\x5e\x0c\x35\xe2\x11\xd1\x13" + "\xf8\xb7\x8d\x72\x6b\x22\x2a\xb0" + "\xdb\x08\xba\x35\xb9\x3f\xc8\xd3" + "\x24\x90\xec\x58\xd2\x09\xc7\x2d" + "\xed\x38\x80\x36\x72\x43\x27\x49" + "\x4a\x80\x8a\xa2\xe8\xd3\xda\x30" + "\x7d\xb6\x82\x37\x86\x92\x86\x3e" + "\x08\xb2\x28\x5a\x55\x44\x24\x7d" + "\x40\x48\x8a\xb6\x89\x58\x08\xa0" + "\xd6\x6d\x3a\x17\xbf\xf6\x54\xa2" + "\xf5\xd3\x8c\x0f\x78\x12\x57\x8b" + "\xd5\xc2\xfd\x58\x5b\x7f\x38\xe3" + "\xcc\xb7\x7c\x48\xb3\x20\xe8\x81" + "\x14\x32\x45\x05\xe0\xdb\x9f\x75" + "\x85\xb4\x6a\xfc\x95\xe3\x54\x22" + "\x12\xee\x30\xfe\xd8\x30\xef\x34" + "\x50\xab\x46\x30\x98\x2f\xb7\xc0" + "\x15\xa2\x83\xb6\xf2\x06\x21\xa2" + "\xc3\x26\x37\x14\xd1\x4d\xb5\x10" + "\x52\x76\x4d\x6a\xee\xb5\x2b\x15" + "\xb7\xf9\x51\xe8\x2a\xaf\xc7\xfa" + "\x77\xaf\xb0\x05\x4d\xd1\x68\x8e" + "\x74\x05\x9f\x9d\x93\xa5\x3e\x7f" + "\x4e\x5f\x9d\xcb\x09\xc7\x83\xe3" + "\x02\x9d\x27\x1f\xef\x85\x05\x8d" + "\xec\x55\x88\x0f\x0d\x7c\x4c\xe8" + "\xa1\x75\xa0\xd8\x06\x47\x14\xef" + "\xaa\x61\xcf\x26\x15\xad\xd8\xa3" + "\xaa\x75\xf2\x78\x4a\x5a\x61\xdf" + "\x8b\xc7\x04\xbc\xb2\x32\xd2\x7e" + "\x42\xee\xb4\x2f\x51\xff\x7b\x2e" + "\xd3\x02\xe8\xdc\x5d\x0d\x50\xdc" + "\xae\xb7\x46\xf9\xa8\xe6\xd0\x16" + "\xcc\xe6\x2c\x81\xc7\xad\xe9\xf0" + "\x05\x72\x6d\x3d\x0a\x7a\xa9\x02" + "\xac\x82\x93\x6e\xb6\x1c\x28\xfc" + "\x44\x12\xfb\x73\x77\xd4\x13\x39" + "\x29\x88\x8a\xf3\x5c\xa6\x36\xa0" + "\x2a\xed\x7e\xb1\x1d\xd6\x4c\x6b" + "\x41\x01\x18\x5d\x5d\x07\x97\xa6" + "\x4b\xef\x31\x18\xea\xac\xb1\x84" + "\x21\xed\xda\x86", + .len = 4100, + .np = 2, + .tap = { 4064, 36 }, + }, +}; + +static const struct cipher_testvec aes_ofb_tv_template[] = { + /* From NIST Special Publication 800-38A, Appendix F.5 */ + { + .key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6" + "\xab\xf7\x15\x88\x09\xcf\x4f\x3c", + .klen = 16, + .iv = "\x00\x01\x02\x03\x04\x05\x06\x07\x08" + "\x09\x0a\x0b\x0c\x0d\x0e\x0f", + .ptext = "\x6b\xc1\xbe\xe2\x2e\x40\x9f\x96" + "\xe9\x3d\x7e\x11\x73\x93\x17\x2a" + "\xae\x2d\x8a\x57\x1e\x03\xac\x9c" + "\x9e\xb7\x6f\xac\x45\xaf\x8e\x51" + "\x30\xc8\x1c\x46\xa3\x5c\xe4\x11" + "\xe5\xfb\xc1\x19\x1a\x0a\x52\xef" + "\xf6\x9f\x24\x45\xdf\x4f\x9b\x17" + "\xad\x2b\x41\x7b\xe6\x6c\x37\x10", + .ctext = "\x3b\x3f\xd9\x2e\xb7\x2d\xad\x20" + "\x33\x34\x49\xf8\xe8\x3c\xfb\x4a" + "\x77\x89\x50\x8d\x16\x91\x8f\x03\xf5" + "\x3c\x52\xda\xc5\x4e\xd8\x25" + "\x97\x40\x05\x1e\x9c\x5f\xec\xf6\x43" + "\x44\xf7\xa8\x22\x60\xed\xcc" + "\x30\x4c\x65\x28\xf6\x59\xc7\x78" + "\x66\xa5\x10\xd9\xc1\xd6\xae\x5e", + .len = 64, + } +}; + +static const struct aead_testvec aes_gcm_enc_tv_template[] = { + { /* From McGrew & Viega - http://citeseer.ist.psu.edu/656989.html */ + .key = zeroed_string, + .klen = 16, + .result = "\x58\xe2\xfc\xce\xfa\x7e\x30\x61" + "\x36\x7f\x1d\x57\xa4\xe7\x45\x5a", + .rlen = 16, + }, { + .key = zeroed_string, + .klen = 16, + .input = zeroed_string, + .ilen = 16, + .result = "\x03\x88\xda\xce\x60\xb6\xa3\x92" + "\xf3\x28\xc2\xb9\x71\xb2\xfe\x78" + "\xab\x6e\x47\xd4\x2c\xec\x13\xbd" + "\xf5\x3a\x67\xb2\x12\x57\xbd\xdf", + .rlen = 32, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08", + .klen = 16, + .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" + "\xde\xca\xf8\x88", + .input = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" + "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" + "\x86\xa7\xa9\x53\x15\x34\xf7\xda" + "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" + "\x1c\x3c\x0c\x95\x95\x68\x09\x53" + "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" + "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" + "\xba\x63\x7b\x39\x1a\xaf\xd2\x55", + .ilen = 64, + .result = "\x42\x83\x1e\xc2\x21\x77\x74\x24" + "\x4b\x72\x21\xb7\x84\xd0\xd4\x9c" + "\xe3\xaa\x21\x2f\x2c\x02\xa4\xe0" + "\x35\xc1\x7e\x23\x29\xac\xa1\x2e" + "\x21\xd5\x14\xb2\x54\x66\x93\x1c" + "\x7d\x8f\x6a\x5a\xac\x84\xaa\x05" + "\x1b\xa3\x0b\x39\x6a\x0a\xac\x97" + "\x3d\x58\xe0\x91\x47\x3f\x59\x85" + "\x4d\x5c\x2a\xf3\x27\xcd\x64\xa6" + "\x2c\xf3\x5a\xbd\x2b\xa6\xfa\xb4", + .rlen = 80, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08", + .klen = 16, + .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" + "\xde\xca\xf8\x88", + .input = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" + "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" + "\x86\xa7\xa9\x53\x15\x34\xf7\xda" + "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" + "\x1c\x3c\x0c\x95\x95\x68\x09\x53" + "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" + "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" + "\xba\x63\x7b\x39", + .ilen = 60, + .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef" + "\xfe\xed\xfa\xce\xde\xad\xbe\xef" + "\xab\xad\xda\xd2", + .alen = 20, + .result = "\x42\x83\x1e\xc2\x21\x77\x74\x24" + "\x4b\x72\x21\xb7\x84\xd0\xd4\x9c" + "\xe3\xaa\x21\x2f\x2c\x02\xa4\xe0" + "\x35\xc1\x7e\x23\x29\xac\xa1\x2e" + "\x21\xd5\x14\xb2\x54\x66\x93\x1c" + "\x7d\x8f\x6a\x5a\xac\x84\xaa\x05" + "\x1b\xa3\x0b\x39\x6a\x0a\xac\x97" + "\x3d\x58\xe0\x91" + "\x5b\xc9\x4f\xbc\x32\x21\xa5\xdb" + "\x94\xfa\xe9\x5a\xe7\x12\x1a\x47", + .rlen = 76, + }, { + .key = zeroed_string, + .klen = 24, + .result = "\xcd\x33\xb2\x8a\xc7\x73\xf7\x4b" + "\xa0\x0e\xd1\xf3\x12\x57\x24\x35", + .rlen = 16, + }, { + .key = zeroed_string, + .klen = 24, + .input = zeroed_string, + .ilen = 16, + .result = "\x98\xe7\x24\x7c\x07\xf0\xfe\x41" + "\x1c\x26\x7e\x43\x84\xb0\xf6\x00" + "\x2f\xf5\x8d\x80\x03\x39\x27\xab" + "\x8e\xf4\xd4\x58\x75\x14\xf0\xfb", + .rlen = 32, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\xfe\xff\xe9\x92\x86\x65\x73\x1c", + .klen = 24, + .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" + "\xde\xca\xf8\x88", + .input = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" + "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" + "\x86\xa7\xa9\x53\x15\x34\xf7\xda" + "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" + "\x1c\x3c\x0c\x95\x95\x68\x09\x53" + "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" + "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" + "\xba\x63\x7b\x39\x1a\xaf\xd2\x55", + .ilen = 64, + .result = "\x39\x80\xca\x0b\x3c\x00\xe8\x41" + "\xeb\x06\xfa\xc4\x87\x2a\x27\x57" + "\x85\x9e\x1c\xea\xa6\xef\xd9\x84" + "\x62\x85\x93\xb4\x0c\xa1\xe1\x9c" + "\x7d\x77\x3d\x00\xc1\x44\xc5\x25" + "\xac\x61\x9d\x18\xc8\x4a\x3f\x47" + "\x18\xe2\x44\x8b\x2f\xe3\x24\xd9" + "\xcc\xda\x27\x10\xac\xad\xe2\x56" + "\x99\x24\xa7\xc8\x58\x73\x36\xbf" + "\xb1\x18\x02\x4d\xb8\x67\x4a\x14", + .rlen = 80, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\xfe\xff\xe9\x92\x86\x65\x73\x1c", + .klen = 24, + .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" + "\xde\xca\xf8\x88", + .input = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" + "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" + "\x86\xa7\xa9\x53\x15\x34\xf7\xda" + "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" + "\x1c\x3c\x0c\x95\x95\x68\x09\x53" + "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" + "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" + "\xba\x63\x7b\x39", + .ilen = 60, + .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef" + "\xfe\xed\xfa\xce\xde\xad\xbe\xef" + "\xab\xad\xda\xd2", + .alen = 20, + .result = "\x39\x80\xca\x0b\x3c\x00\xe8\x41" + "\xeb\x06\xfa\xc4\x87\x2a\x27\x57" + "\x85\x9e\x1c\xea\xa6\xef\xd9\x84" + "\x62\x85\x93\xb4\x0c\xa1\xe1\x9c" + "\x7d\x77\x3d\x00\xc1\x44\xc5\x25" + "\xac\x61\x9d\x18\xc8\x4a\x3f\x47" + "\x18\xe2\x44\x8b\x2f\xe3\x24\xd9" + "\xcc\xda\x27\x10" + "\x25\x19\x49\x8e\x80\xf1\x47\x8f" + "\x37\xba\x55\xbd\x6d\x27\x61\x8c", + .rlen = 76, + .np = 2, + .tap = { 32, 28 }, + .anp = 2, + .atap = { 8, 12 } + }, { + .key = zeroed_string, + .klen = 32, + .result = "\x53\x0f\x8a\xfb\xc7\x45\x36\xb9" + "\xa9\x63\xb4\xf1\xc4\xcb\x73\x8b", + .rlen = 16, + } +}; + +static const struct aead_testvec aes_gcm_dec_tv_template[] = { + { /* From McGrew & Viega - http://citeseer.ist.psu.edu/656989.html */ + .key = zeroed_string, + .klen = 32, + .input = "\xce\xa7\x40\x3d\x4d\x60\x6b\x6e" + "\x07\x4e\xc5\xd3\xba\xf3\x9d\x18" + "\xd0\xd1\xc8\xa7\x99\x99\x6b\xf0" + "\x26\x5b\x98\xb5\xd4\x8a\xb9\x19", + .ilen = 32, + .result = zeroed_string, + .rlen = 16, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08", + .klen = 32, + .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" + "\xde\xca\xf8\x88", + .input = "\x52\x2d\xc1\xf0\x99\x56\x7d\x07" + "\xf4\x7f\x37\xa3\x2a\x84\x42\x7d" + "\x64\x3a\x8c\xdc\xbf\xe5\xc0\xc9" + "\x75\x98\xa2\xbd\x25\x55\xd1\xaa" + "\x8c\xb0\x8e\x48\x59\x0d\xbb\x3d" + "\xa7\xb0\x8b\x10\x56\x82\x88\x38" + "\xc5\xf6\x1e\x63\x93\xba\x7a\x0a" + "\xbc\xc9\xf6\x62\x89\x80\x15\xad" + "\xb0\x94\xda\xc5\xd9\x34\x71\xbd" + "\xec\x1a\x50\x22\x70\xe3\xcc\x6c", + .ilen = 80, + .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" + "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" + "\x86\xa7\xa9\x53\x15\x34\xf7\xda" + "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" + "\x1c\x3c\x0c\x95\x95\x68\x09\x53" + "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" + "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" + "\xba\x63\x7b\x39\x1a\xaf\xd2\x55", + .rlen = 64, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08", + .klen = 32, + .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" + "\xde\xca\xf8\x88", + .input = "\x52\x2d\xc1\xf0\x99\x56\x7d\x07" + "\xf4\x7f\x37\xa3\x2a\x84\x42\x7d" + "\x64\x3a\x8c\xdc\xbf\xe5\xc0\xc9" + "\x75\x98\xa2\xbd\x25\x55\xd1\xaa" + "\x8c\xb0\x8e\x48\x59\x0d\xbb\x3d" + "\xa7\xb0\x8b\x10\x56\x82\x88\x38" + "\xc5\xf6\x1e\x63\x93\xba\x7a\x0a" + "\xbc\xc9\xf6\x62" + "\x76\xfc\x6e\xce\x0f\x4e\x17\x68" + "\xcd\xdf\x88\x53\xbb\x2d\x55\x1b", + .ilen = 76, + .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef" + "\xfe\xed\xfa\xce\xde\xad\xbe\xef" + "\xab\xad\xda\xd2", + .alen = 20, + .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" + "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" + "\x86\xa7\xa9\x53\x15\x34\xf7\xda" + "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" + "\x1c\x3c\x0c\x95\x95\x68\x09\x53" + "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" + "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" + "\xba\x63\x7b\x39", + .rlen = 60, + .np = 2, + .tap = { 48, 28 }, + .anp = 3, + .atap = { 8, 8, 4 } + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08", + .klen = 16, + .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" + "\xde\xca\xf8\x88", + .input = "\x42\x83\x1e\xc2\x21\x77\x74\x24" + "\x4b\x72\x21\xb7\x84\xd0\xd4\x9c" + "\xe3\xaa\x21\x2f\x2c\x02\xa4\xe0" + "\x35\xc1\x7e\x23\x29\xac\xa1\x2e" + "\x21\xd5\x14\xb2\x54\x66\x93\x1c" + "\x7d\x8f\x6a\x5a\xac\x84\xaa\x05" + "\x1b\xa3\x0b\x39\x6a\x0a\xac\x97" + "\x3d\x58\xe0\x91\x47\x3f\x59\x85" + "\x4d\x5c\x2a\xf3\x27\xcd\x64\xa6" + "\x2c\xf3\x5a\xbd\x2b\xa6\xfa\xb4", + .ilen = 80, + .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" + "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" + "\x86\xa7\xa9\x53\x15\x34\xf7\xda" + "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" + "\x1c\x3c\x0c\x95\x95\x68\x09\x53" + "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" + "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" + "\xba\x63\x7b\x39\x1a\xaf\xd2\x55", + .rlen = 64, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08", + .klen = 16, + .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" + "\xde\xca\xf8\x88", + .input = "\x42\x83\x1e\xc2\x21\x77\x74\x24" + "\x4b\x72\x21\xb7\x84\xd0\xd4\x9c" + "\xe3\xaa\x21\x2f\x2c\x02\xa4\xe0" + "\x35\xc1\x7e\x23\x29\xac\xa1\x2e" + "\x21\xd5\x14\xb2\x54\x66\x93\x1c" + "\x7d\x8f\x6a\x5a\xac\x84\xaa\x05" + "\x1b\xa3\x0b\x39\x6a\x0a\xac\x97" + "\x3d\x58\xe0\x91" + "\x5b\xc9\x4f\xbc\x32\x21\xa5\xdb" + "\x94\xfa\xe9\x5a\xe7\x12\x1a\x47", + .ilen = 76, + .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef" + "\xfe\xed\xfa\xce\xde\xad\xbe\xef" + "\xab\xad\xda\xd2", + .alen = 20, + .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" + "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" + "\x86\xa7\xa9\x53\x15\x34\xf7\xda" + "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" + "\x1c\x3c\x0c\x95\x95\x68\x09\x53" + "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" + "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" + "\xba\x63\x7b\x39", + .rlen = 60, + }, { + .key = zeroed_string, + .klen = 24, + .input = "\x98\xe7\x24\x7c\x07\xf0\xfe\x41" + "\x1c\x26\x7e\x43\x84\xb0\xf6\x00" + "\x2f\xf5\x8d\x80\x03\x39\x27\xab" + "\x8e\xf4\xd4\x58\x75\x14\xf0\xfb", + .ilen = 32, + .result = zeroed_string, + .rlen = 16, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\xfe\xff\xe9\x92\x86\x65\x73\x1c", + .klen = 24, + .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" + "\xde\xca\xf8\x88", + .input = "\x39\x80\xca\x0b\x3c\x00\xe8\x41" + "\xeb\x06\xfa\xc4\x87\x2a\x27\x57" + "\x85\x9e\x1c\xea\xa6\xef\xd9\x84" + "\x62\x85\x93\xb4\x0c\xa1\xe1\x9c" + "\x7d\x77\x3d\x00\xc1\x44\xc5\x25" + "\xac\x61\x9d\x18\xc8\x4a\x3f\x47" + "\x18\xe2\x44\x8b\x2f\xe3\x24\xd9" + "\xcc\xda\x27\x10\xac\xad\xe2\x56" + "\x99\x24\xa7\xc8\x58\x73\x36\xbf" + "\xb1\x18\x02\x4d\xb8\x67\x4a\x14", + .ilen = 80, + .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" + "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" + "\x86\xa7\xa9\x53\x15\x34\xf7\xda" + "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" + "\x1c\x3c\x0c\x95\x95\x68\x09\x53" + "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" + "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" + "\xba\x63\x7b\x39\x1a\xaf\xd2\x55", + .rlen = 64, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\xfe\xff\xe9\x92\x86\x65\x73\x1c", + .klen = 24, + .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad" + "\xde\xca\xf8\x88", + .input = "\x39\x80\xca\x0b\x3c\x00\xe8\x41" + "\xeb\x06\xfa\xc4\x87\x2a\x27\x57" + "\x85\x9e\x1c\xea\xa6\xef\xd9\x84" + "\x62\x85\x93\xb4\x0c\xa1\xe1\x9c" + "\x7d\x77\x3d\x00\xc1\x44\xc5\x25" + "\xac\x61\x9d\x18\xc8\x4a\x3f\x47" + "\x18\xe2\x44\x8b\x2f\xe3\x24\xd9" + "\xcc\xda\x27\x10" + "\x25\x19\x49\x8e\x80\xf1\x47\x8f" + "\x37\xba\x55\xbd\x6d\x27\x61\x8c", + .ilen = 76, + .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef" + "\xfe\xed\xfa\xce\xde\xad\xbe\xef" + "\xab\xad\xda\xd2", + .alen = 20, + .result = "\xd9\x31\x32\x25\xf8\x84\x06\xe5" + "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a" + "\x86\xa7\xa9\x53\x15\x34\xf7\xda" + "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72" + "\x1c\x3c\x0c\x95\x95\x68\x09\x53" + "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25" + "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57" + "\xba\x63\x7b\x39", + .rlen = 60, + } +}; + +static const struct aead_testvec aes_gcm_rfc4106_enc_tv_template[] = { + { /* Generated using Crypto++ */ + .key = zeroed_string, + .klen = 20, + .iv = zeroed_string, + .input = zeroed_string, + .ilen = 16, + .assoc = zeroed_string, + .alen = 16, + .result = "\x03\x88\xDA\xCE\x60\xB6\xA3\x92" + "\xF3\x28\xC2\xB9\x71\xB2\xFE\x78" + "\x97\xFE\x4C\x23\x37\x42\x01\xE0" + "\x81\x9F\x8D\xC5\xD7\x41\xA0\x1B", + .rlen = 32, + },{ + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00\x00", + .klen = 20, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .input = zeroed_string, + .ilen = 16, + .assoc = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .result = "\xC0\x0D\x8B\x42\x0F\x8F\x34\x18" + "\x88\xB1\xC5\xBC\xC5\xB6\xD6\x28" + "\x6A\x9D\xDF\x11\x5E\xFE\x5E\x9D" + "\x2F\x70\x44\x92\xF7\xF2\xE3\xEF", + .rlen = 32, + + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00\x00", + .klen = 20, + .iv = zeroed_string, + .input = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .ilen = 16, + .assoc = zeroed_string, + .alen = 16, + .result = "\x4B\xB1\xB5\xE3\x25\x71\x70\xDE" + "\x7F\xC9\x9C\xA5\x14\x19\xF2\xAC" + "\x0B\x8F\x88\x69\x17\xE6\xB4\x3C" + "\xB1\x68\xFD\x14\x52\x64\x61\xB2", + .rlen = 32, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00\x00", + .klen = 20, + .iv = zeroed_string, + .input = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .ilen = 16, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .alen = 16, + .result = "\x4B\xB1\xB5\xE3\x25\x71\x70\xDE" + "\x7F\xC9\x9C\xA5\x14\x19\xF2\xAC" + "\x90\x92\xB7\xE3\x5F\xA3\x9A\x63" + "\x7E\xD7\x1F\xD8\xD3\x7C\x4B\xF5", + .rlen = 32, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00\x00", + .klen = 20, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .input = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .ilen = 16, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .result = "\xC1\x0C\x8A\x43\x0E\x8E\x35\x19" + "\x89\xB0\xC4\xBD\xC4\xB7\xD7\x29" + "\x64\x50\xF9\x32\x13\xFB\x74\x61" + "\xF4\xED\x52\xD3\xC5\x10\x55\x3C", + .rlen = 32, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00\x00", + .klen = 20, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .input = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .ilen = 64, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .result = "\xC1\x0C\x8A\x43\x0E\x8E\x35\x19" + "\x89\xB0\xC4\xBD\xC4\xB7\xD7\x29" + "\x98\x14\xA1\x42\x37\x80\xFD\x90" + "\x68\x12\x01\xA8\x91\x89\xB9\x83" + "\x5B\x11\x77\x12\x9B\xFF\x24\x89" + "\x94\x5F\x18\x12\xBA\x27\x09\x39" + "\x99\x96\x76\x42\x15\x1C\xCD\xCB" + "\xDC\xD3\xDA\x65\x73\xAF\x80\xCD" + "\xD2\xB6\xC2\x4A\x76\xC2\x92\x85" + "\xBD\xCF\x62\x98\x58\x14\xE5\xBD", + .rlen = 80, + }, { + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x00\x00\x00\x00", + .klen = 20, + .iv = "\x00\x00\x45\x67\x89\xab\xcd\xef", + .input = "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff", + .ilen = 192, + .assoc = "\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa" + "\xaa\xaa\xaa\xaa\x00\x00\x45\x67" + "\x89\xab\xcd\xef", + .alen = 20, + .result = "\xC1\x76\x33\x85\xE2\x9B\x5F\xDE" + "\xDE\x89\x3D\x42\xE7\xC9\x69\x8A" + "\x44\x6D\xC3\x88\x46\x2E\xC2\x01" + "\x5E\xF6\x0C\x39\xF0\xC4\xA5\x82" + "\xCD\xE8\x31\xCC\x0A\x4C\xE4\x44" + "\x41\xA9\x82\x6F\x22\xA1\x23\x1A" + "\xA8\xE3\x16\xFD\x31\x5C\x27\x31" + "\xF1\x7F\x01\x63\xA3\xAF\x70\xA1" + "\xCF\x07\x57\x41\x67\xD0\xC4\x42" + "\xDB\x18\xC6\x4C\x4C\xE0\x3D\x9F" + "\x05\x07\xFB\x13\x7D\x4A\xCA\x5B" + "\xF0\xBF\x64\x7E\x05\xB1\x72\xEE" + "\x7C\x3B\xD4\xCD\x14\x03\xB2\x2C" + "\xD3\xA9\xEE\xFA\x17\xFC\x9C\xDF" + "\xC7\x75\x40\xFF\xAE\xAD\x1E\x59" + "\x2F\x30\x24\xFB\xAD\x6B\x10\xFA" + "\x6C\x9F\x5B\xE7\x25\xD5\xD0\x25" + "\xAC\x4A\x4B\xDA\xFC\x7A\x85\x1B" + "\x7E\x13\x06\x82\x08\x17\xA4\x35" + "\xEC\xC5\x8D\x63\x96\x81\x0A\x8F" + "\xA3\x05\x38\x95\x20\x1A\x47\x04" + "\x6F\x6D\xDA\x8F\xEF\xC1\x76\x35" + "\x6B\xC7\x4D\x0F\x94\x12\xCA\x3E" + "\x2E\xD5\x03\x2E\x86\x7E\xAA\x3B" + "\x37\x08\x1C\xCF\xBA\x5D\x71\x46" + "\x80\x72\xB0\x4C\x82\x0D\x60\x3C", + .rlen = 208, + }, { /* From draft-mcgrew-gcm-test-01 */ + .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" + "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" + "\x2E\x44\x3B\x68", + .klen = 20, + .iv = "\x49\x56\xED\x7E\x3B\x24\x4C\xFE", + .input = "\x45\x00\x00\x48\x69\x9A\x00\x00" + "\x80\x11\x4D\xB7\xC0\xA8\x01\x02" + "\xC0\xA8\x01\x01\x0A\x9B\xF1\x56" + "\x38\xD3\x01\x00\x00\x01\x00\x00" + "\x00\x00\x00\x00\x04\x5F\x73\x69" + "\x70\x04\x5F\x75\x64\x70\x03\x73" + "\x69\x70\x09\x63\x79\x62\x65\x72" + "\x63\x69\x74\x79\x02\x64\x6B\x00" + "\x00\x21\x00\x01\x01\x02\x02\x01", + .ilen = 72, + .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" + "\x00\x00\x00\x00\x49\x56\xED\x7E" + "\x3B\x24\x4C\xFE", + .alen = 20, + .result = "\xFE\xCF\x53\x7E\x72\x9D\x5B\x07" + "\xDC\x30\xDF\x52\x8D\xD2\x2B\x76" + "\x8D\x1B\x98\x73\x66\x96\xA6\xFD" + "\x34\x85\x09\xFA\x13\xCE\xAC\x34" + "\xCF\xA2\x43\x6F\x14\xA3\xF3\xCF" + "\x65\x92\x5B\xF1\xF4\xA1\x3C\x5D" + "\x15\xB2\x1E\x18\x84\xF5\xFF\x62" + "\x47\xAE\xAB\xB7\x86\xB9\x3B\xCE" + "\x61\xBC\x17\xD7\x68\xFD\x97\x32" + "\x45\x90\x18\x14\x8F\x6C\xBE\x72" + "\x2F\xD0\x47\x96\x56\x2D\xFD\xB4", + .rlen = 88, + }, { + .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\x6D\x6A\x8F\x94\x67\x30\x83\x08" + "\xCA\xFE\xBA\xBE", + .klen = 20, + .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .input = "\x45\x00\x00\x3E\x69\x8F\x00\x00" + "\x80\x11\x4D\xCC\xC0\xA8\x01\x02" + "\xC0\xA8\x01\x01\x0A\x98\x00\x35" + "\x00\x2A\x23\x43\xB2\xD0\x01\x00" + "\x00\x01\x00\x00\x00\x00\x00\x00" + "\x03\x73\x69\x70\x09\x63\x79\x62" + "\x65\x72\x63\x69\x74\x79\x02\x64" + "\x6B\x00\x00\x01\x00\x01\x00\x01", + .ilen = 64, + .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" + "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .alen = 16, + .result = "\xDE\xB2\x2C\xD9\xB0\x7C\x72\xC1" + "\x6E\x3A\x65\xBE\xEB\x8D\xF3\x04" + "\xA5\xA5\x89\x7D\x33\xAE\x53\x0F" + "\x1B\xA7\x6D\x5D\x11\x4D\x2A\x5C" + "\x3D\xE8\x18\x27\xC1\x0E\x9A\x4F" + "\x51\x33\x0D\x0E\xEC\x41\x66\x42" + "\xCF\xBB\x85\xA5\xB4\x7E\x48\xA4" + "\xEC\x3B\x9B\xA9\x5D\x91\x8B\xD1" + "\x83\xB7\x0D\x3A\xA8\xBC\x6E\xE4" + "\xC3\x09\xE9\xD8\x5A\x41\xAD\x4A", + .rlen = 80, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x11\x22\x33\x44", + .klen = 36, + .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", + .input = "\x45\x00\x00\x30\x69\xA6\x40\x00" + "\x80\x06\x26\x90\xC0\xA8\x01\x02" + "\x93\x89\x15\x5E\x0A\x9E\x00\x8B" + "\x2D\xC5\x7E\xE0\x00\x00\x00\x00" + "\x70\x02\x40\x00\x20\xBF\x00\x00" + "\x02\x04\x05\xB4\x01\x01\x04\x02" + "\x01\x02\x02\x01", + .ilen = 52, + .assoc = "\x4A\x2C\xBF\xE3\x00\x00\x00\x02" + "\x01\x02\x03\x04\x05\x06\x07\x08", + .alen = 16, + .result = "\xFF\x42\x5C\x9B\x72\x45\x99\xDF" + "\x7A\x3B\xCD\x51\x01\x94\xE0\x0D" + "\x6A\x78\x10\x7F\x1B\x0B\x1C\xBF" + "\x06\xEF\xAE\x9D\x65\xA5\xD7\x63" + "\x74\x8A\x63\x79\x85\x77\x1D\x34" + "\x7F\x05\x45\x65\x9F\x14\xE9\x9D" + "\xEF\x84\x2D\x8E\xB3\x35\xF4\xEE" + "\xCF\xDB\xF8\x31\x82\x4B\x4C\x49" + "\x15\x95\x6C\x96", + .rlen = 68, + }, { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00", + .klen = 20, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", + .input = "\x45\x00\x00\x3C\x99\xC5\x00\x00" + "\x80\x01\xCB\x7A\x40\x67\x93\x18" + "\x01\x01\x01\x01\x08\x00\x07\x5C" + "\x02\x00\x44\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x75\x76\x77\x61\x62\x63\x64\x65" + "\x66\x67\x68\x69\x01\x02\x02\x01", + .ilen = 64, + .assoc = "\x00\x00\x00\x00\x00\x00\x00\x01" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .alen = 16, + .result = "\x46\x88\xDA\xF2\xF9\x73\xA3\x92" + "\x73\x29\x09\xC3\x31\xD5\x6D\x60" + "\xF6\x94\xAB\xAA\x41\x4B\x5E\x7F" + "\xF5\xFD\xCD\xFF\xF5\xE9\xA2\x84" + "\x45\x64\x76\x49\x27\x19\xFF\xB6" + "\x4D\xE7\xD9\xDC\xA1\xE1\xD8\x94" + "\xBC\x3B\xD5\x78\x73\xED\x4D\x18" + "\x1D\x19\xD4\xD5\xC8\xC1\x8A\xF3" + "\xF8\x21\xD4\x96\xEE\xB0\x96\xE9" + "\x8A\xD2\xB6\x9E\x47\x99\xC7\x1D", + .rlen = 80, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E\x43", + .klen = 20, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .input = "\x45\x00\x00\x3C\x99\xC3\x00\x00" + "\x80\x01\xCB\x7C\x40\x67\x93\x18" + "\x01\x01\x01\x01\x08\x00\x08\x5C" + "\x02\x00\x43\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x75\x76\x77\x61\x62\x63\x64\x65" + "\x66\x67\x68\x69\x01\x02\x02\x01", + .ilen = 64, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .result = "\xFB\xA2\xCA\xA4\x85\x3C\xF9\xF0" + "\xF2\x2C\xB1\x0D\x86\xDD\x83\xB0" + "\xFE\xC7\x56\x91\xCF\x1A\x04\xB0" + "\x0D\x11\x38\xEC\x9C\x35\x79\x17" + "\x65\xAC\xBD\x87\x01\xAD\x79\x84" + "\x5B\xF9\xFE\x3F\xBA\x48\x7B\xC9" + "\x17\x55\xE6\x66\x2B\x4C\x8D\x0D" + "\x1F\x5E\x22\x73\x95\x30\x32\x0A" + "\xE0\xD7\x31\xCC\x97\x8E\xCA\xFA" + "\xEA\xE8\x8F\x00\xE8\x0D\x6E\x48", + .rlen = 80, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E\x43", + .klen = 20, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .input = "\x45\x00\x00\x1C\x42\xA2\x00\x00" + "\x80\x01\x44\x1F\x40\x67\x93\xB6" + "\xE0\x00\x00\x02\x0A\x00\xF5\xFF" + "\x01\x02\x02\x01", + .ilen = 28, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .result = "\xFB\xA2\xCA\x84\x5E\x5D\xF9\xF0" + "\xF2\x2C\x3E\x6E\x86\xDD\x83\x1E" + "\x1F\xC6\x57\x92\xCD\x1A\xF9\x13" + "\x0E\x13\x79\xED\x36\x9F\x07\x1F" + "\x35\xE0\x34\xBE\x95\xF1\x12\xE4" + "\xE7\xD0\x5D\x35", + .rlen = 44, + }, { + .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\x6D\x6A\x8F\x94\x67\x30\x83\x08" + "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\xCA\xFE\xBA\xBE", + .klen = 28, + .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .input = "\x45\x00\x00\x28\xA4\xAD\x40\x00" + "\x40\x06\x78\x80\x0A\x01\x03\x8F" + "\x0A\x01\x06\x12\x80\x23\x06\xB8" + "\xCB\x71\x26\x02\xDD\x6B\xB0\x3E" + "\x50\x10\x16\xD0\x75\x68\x00\x01", + .ilen = 40, + .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" + "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .alen = 16, + .result = "\xA5\xB1\xF8\x06\x60\x29\xAE\xA4" + "\x0E\x59\x8B\x81\x22\xDE\x02\x42" + "\x09\x38\xB3\xAB\x33\xF8\x28\xE6" + "\x87\xB8\x85\x8B\x5B\xFB\xDB\xD0" + "\x31\x5B\x27\x45\x21\x44\xCC\x77" + "\x95\x45\x7B\x96\x52\x03\x7F\x53" + "\x18\x02\x7B\x5B\x4C\xD7\xA6\x36", + .rlen = 56, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xDE\xCA\xF8\x88", + .klen = 20, + .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", + .input = "\x45\x00\x00\x49\x33\xBA\x00\x00" + "\x7F\x11\x91\x06\xC3\xFB\x1D\x10" + "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" + "\x00\x35\xDD\x7B\x80\x03\x02\xD5" + "\x00\x00\x4E\x20\x00\x1E\x8C\x18" + "\xD7\x5B\x81\xDC\x91\xBA\xA0\x47" + "\x6B\x91\xB9\x24\xB2\x80\x38\x9D" + "\x92\xC9\x63\xBA\xC0\x46\xEC\x95" + "\x9B\x62\x66\xC0\x47\x22\xB1\x49" + "\x23\x01\x01\x01", + .ilen = 76, + .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" + "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" + "\xCE\xFA\xCE\x74", + .alen = 20, + .result = "\x18\xA6\xFD\x42\xF7\x2C\xBF\x4A" + "\xB2\xA2\xEA\x90\x1F\x73\xD8\x14" + "\xE3\xE7\xF2\x43\xD9\x54\x12\xE1" + "\xC3\x49\xC1\xD2\xFB\xEC\x16\x8F" + "\x91\x90\xFE\xEB\xAF\x2C\xB0\x19" + "\x84\xE6\x58\x63\x96\x5D\x74\x72" + "\xB7\x9D\xA3\x45\xE0\xE7\x80\x19" + "\x1F\x0D\x2F\x0E\x0F\x49\x6C\x22" + "\x6F\x21\x27\xB2\x7D\xB3\x57\x24" + "\xE7\x84\x5D\x68\x65\x1F\x57\xE6" + "\x5F\x35\x4F\x75\xFF\x17\x01\x57" + "\x69\x62\x34\x36", + .rlen = 92, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x73\x61\x6C\x74", + .klen = 36, + .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", + .input = "\x45\x08\x00\x28\x73\x2C\x00\x00" + "\x40\x06\xE9\xF9\x0A\x01\x06\x12" + "\x0A\x01\x03\x8F\x06\xB8\x80\x23" + "\xDD\x6B\xAF\xBE\xCB\x71\x26\x02" + "\x50\x10\x1F\x64\x6D\x54\x00\x01", + .ilen = 40, + .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" + "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" + "\x69\x76\x65\x63", + .alen = 20, + .result = "\xF2\xD6\x9E\xCD\xBD\x5A\x0D\x5B" + "\x8D\x5E\xF3\x8B\xAD\x4D\xA5\x8D" + "\x1F\x27\x8F\xDE\x98\xEF\x67\x54" + "\x9D\x52\x4A\x30\x18\xD9\xA5\x7F" + "\xF4\xD3\xA3\x1C\xE6\x73\x11\x9E" + "\x45\x16\x26\xC2\x41\x57\x71\xE3" + "\xB7\xEE\xBC\xA6\x14\xC8\x9B\x35", + .rlen = 56, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E\x43", + .klen = 20, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .input = "\x45\x00\x00\x49\x33\x3E\x00\x00" + "\x7F\x11\x91\x82\xC3\xFB\x1D\x10" + "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" + "\x00\x35\xCB\x45\x80\x03\x02\x5B" + "\x00\x00\x01\xE0\x00\x1E\x8C\x18" + "\xD6\x57\x59\xD5\x22\x84\xA0\x35" + "\x2C\x71\x47\x5C\x88\x80\x39\x1C" + "\x76\x4D\x6E\x5E\xE0\x49\x6B\x32" + "\x5A\xE2\x70\xC0\x38\x99\x49\x39" + "\x15\x01\x01\x01", + .ilen = 76, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .result = "\xFB\xA2\xCA\xD1\x2F\xC1\xF9\xF0" + "\x0D\x3C\xEB\xF3\x05\x41\x0D\xB8" + "\x3D\x77\x84\xB6\x07\x32\x3D\x22" + "\x0F\x24\xB0\xA9\x7D\x54\x18\x28" + "\x00\xCA\xDB\x0F\x68\xD9\x9E\xF0" + "\xE0\xC0\xC8\x9A\xE9\xBE\xA8\x88" + "\x4E\x52\xD6\x5B\xC1\xAF\xD0\x74" + "\x0F\x74\x24\x44\x74\x7B\x5B\x39" + "\xAB\x53\x31\x63\xAA\xD4\x55\x0E" + "\xE5\x16\x09\x75\xCD\xB6\x08\xC5" + "\x76\x91\x89\x60\x97\x63\xB8\xE1" + "\x8C\xAA\x81\xE2", + .rlen = 92, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x73\x61\x6C\x74", + .klen = 36, + .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", + .input = "\x63\x69\x73\x63\x6F\x01\x72\x75" + "\x6C\x65\x73\x01\x74\x68\x65\x01" + "\x6E\x65\x74\x77\x65\x01\x64\x65" + "\x66\x69\x6E\x65\x01\x74\x68\x65" + "\x74\x65\x63\x68\x6E\x6F\x6C\x6F" + "\x67\x69\x65\x73\x01\x74\x68\x61" + "\x74\x77\x69\x6C\x6C\x01\x64\x65" + "\x66\x69\x6E\x65\x74\x6F\x6D\x6F" + "\x72\x72\x6F\x77\x01\x02\x02\x01", + .ilen = 72, + .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" + "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" + "\x69\x76\x65\x63", + .alen = 20, + .result = "\xD4\xB7\xED\x86\xA1\x77\x7F\x2E" + "\xA1\x3D\x69\x73\xD3\x24\xC6\x9E" + "\x7B\x43\xF8\x26\xFB\x56\x83\x12" + "\x26\x50\x8B\xEB\xD2\xDC\xEB\x18" + "\xD0\xA6\xDF\x10\xE5\x48\x7D\xF0" + "\x74\x11\x3E\x14\xC6\x41\x02\x4E" + "\x3E\x67\x73\xD9\x1A\x62\xEE\x42" + "\x9B\x04\x3A\x10\xE3\xEF\xE6\xB0" + "\x12\xA4\x93\x63\x41\x23\x64\xF8" + "\xC0\xCA\xC5\x87\xF2\x49\xE5\x6B" + "\x11\xE2\x4F\x30\xE4\x4C\xCC\x76", + .rlen = 88, + }, { + .key = "\x7D\x77\x3D\x00\xC1\x44\xC5\x25" + "\xAC\x61\x9D\x18\xC8\x4A\x3F\x47" + "\xD9\x66\x42\x67", + .klen = 20, + .iv = "\x43\x45\x7E\x91\x82\x44\x3B\xC6", + .input = "\x01\x02\x02\x01", + .ilen = 4, + .assoc = "\x33\x54\x67\xAE\xFF\xFF\xFF\xFF" + "\x43\x45\x7E\x91\x82\x44\x3B\xC6", + .alen = 16, + .result = "\x43\x7F\x86\x6B\xCB\x3F\x69\x9F" + "\xE9\xB0\x82\x2B\xAC\x96\x1C\x45" + "\x04\xBE\xF2\x70", + .rlen = 20, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xDE\xCA\xF8\x88", + .klen = 20, + .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", + .input = "\x74\x6F\x01\x62\x65\x01\x6F\x72" + "\x01\x6E\x6F\x74\x01\x74\x6F\x01" + "\x62\x65\x00\x01", + .ilen = 20, + .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" + "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" + "\xCE\xFA\xCE\x74", + .alen = 20, + .result = "\x29\xC9\xFC\x69\xA1\x97\xD0\x38" + "\xCC\xDD\x14\xE2\xDD\xFC\xAA\x05" + "\x43\x33\x21\x64\x41\x25\x03\x52" + "\x43\x03\xED\x3C\x6C\x5F\x28\x38" + "\x43\xAF\x8C\x3E", + .rlen = 36, + }, { + .key = "\x6C\x65\x67\x61\x6C\x69\x7A\x65" + "\x6D\x61\x72\x69\x6A\x75\x61\x6E" + "\x61\x61\x6E\x64\x64\x6F\x69\x74" + "\x62\x65\x66\x6F\x72\x65\x69\x61" + "\x74\x75\x72\x6E", + .klen = 36, + .iv = "\x33\x30\x21\x69\x67\x65\x74\x6D", + .input = "\x45\x00\x00\x30\xDA\x3A\x00\x00" + "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" + "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01", + .ilen = 52, + .assoc = "\x79\x6B\x69\x63\xFF\xFF\xFF\xFF" + "\xFF\xFF\xFF\xFF\x33\x30\x21\x69" + "\x67\x65\x74\x6D", + .alen = 20, + .result = "\xF9\x7A\xB2\xAA\x35\x6D\x8E\xDC" + "\xE1\x76\x44\xAC\x8C\x78\xE2\x5D" + "\xD2\x4D\xED\xBB\x29\xEB\xF1\xB6" + "\x4A\x27\x4B\x39\xB4\x9C\x3A\x86" + "\x4C\xD3\xD7\x8C\xA4\xAE\x68\xA3" + "\x2B\x42\x45\x8F\xB5\x7D\xBE\x82" + "\x1D\xCC\x63\xB9\xD0\x93\x7B\xA2" + "\x94\x5F\x66\x93\x68\x66\x1A\x32" + "\x9F\xB4\xC0\x53", + .rlen = 68, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E\x43", + .klen = 20, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .input = "\x45\x00\x00\x30\xDA\x3A\x00\x00" + "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" + "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01", + .ilen = 52, + .assoc = "\x3F\x7E\xF6\x42\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .result = "\xFB\xA2\xCA\xA8\xC6\xC5\xF9\xF0" + "\xF2\x2C\xA5\x4A\x06\x12\x10\xAD" + "\x3F\x6E\x57\x91\xCF\x1A\xCA\x21" + "\x0D\x11\x7C\xEC\x9C\x35\x79\x17" + "\x65\xAC\xBD\x87\x01\xAD\x79\x84" + "\x5B\xF9\xFE\x3F\xBA\x48\x7B\xC9" + "\x63\x21\x93\x06\x84\xEE\xCA\xDB" + "\x56\x91\x25\x46\xE7\xA9\x5C\x97" + "\x40\xD7\xCB\x05", + .rlen = 68, + }, { + .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" + "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" + "\x22\x43\x3C\x64", + .klen = 20, + .iv = "\x48\x55\xEC\x7D\x3A\x23\x4B\xFD", + .input = "\x08\x00\xC6\xCD\x02\x00\x07\x00" + "\x61\x62\x63\x64\x65\x66\x67\x68" + "\x69\x6A\x6B\x6C\x6D\x6E\x6F\x70" + "\x71\x72\x73\x74\x01\x02\x02\x01", + .ilen = 32, + .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" + "\x00\x00\x00\x07\x48\x55\xEC\x7D" + "\x3A\x23\x4B\xFD", + .alen = 20, + .result = "\x74\x75\x2E\x8A\xEB\x5D\x87\x3C" + "\xD7\xC0\xF4\xAC\xC3\x6C\x4B\xFF" + "\x84\xB7\xD7\xB9\x8F\x0C\xA8\xB6" + "\xAC\xDA\x68\x94\xBC\x61\x90\x69" + "\xEF\x9C\xBC\x28\xFE\x1B\x56\xA7" + "\xC4\xE0\xD5\x8C\x86\xCD\x2B\xC0", + .rlen = 48, + } +}; + +static const struct aead_testvec aes_gcm_rfc4106_dec_tv_template[] = { + { /* Generated using Crypto++ */ + .key = zeroed_string, + .klen = 20, + .iv = zeroed_string, + .input = "\x03\x88\xDA\xCE\x60\xB6\xA3\x92" + "\xF3\x28\xC2\xB9\x71\xB2\xFE\x78" + "\x97\xFE\x4C\x23\x37\x42\x01\xE0" + "\x81\x9F\x8D\xC5\xD7\x41\xA0\x1B", + .ilen = 32, + .assoc = zeroed_string, + .alen = 16, + .result = zeroed_string, + .rlen = 16, + + },{ + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00\x00", + .klen = 20, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .input = "\xC0\x0D\x8B\x42\x0F\x8F\x34\x18" + "\x88\xB1\xC5\xBC\xC5\xB6\xD6\x28" + "\x6A\x9D\xDF\x11\x5E\xFE\x5E\x9D" + "\x2F\x70\x44\x92\xF7\xF2\xE3\xEF", + .ilen = 32, + .assoc = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .result = zeroed_string, + .rlen = 16, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00\x00", + .klen = 20, + .iv = zeroed_string, + .input = "\x4B\xB1\xB5\xE3\x25\x71\x70\xDE" + "\x7F\xC9\x9C\xA5\x14\x19\xF2\xAC" + "\x0B\x8F\x88\x69\x17\xE6\xB4\x3C" + "\xB1\x68\xFD\x14\x52\x64\x61\xB2", + .ilen = 32, + .assoc = zeroed_string, + .alen = 16, + .result = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .rlen = 16, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00\x00", + .klen = 20, + .iv = zeroed_string, + .input = "\x4B\xB1\xB5\xE3\x25\x71\x70\xDE" + "\x7F\xC9\x9C\xA5\x14\x19\xF2\xAC" + "\x90\x92\xB7\xE3\x5F\xA3\x9A\x63" + "\x7E\xD7\x1F\xD8\xD3\x7C\x4B\xF5", + .ilen = 32, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .alen = 16, + .result = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .rlen = 16, + + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00\x00", + .klen = 20, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .input = "\xC1\x0C\x8A\x43\x0E\x8E\x35\x19" + "\x89\xB0\xC4\xBD\xC4\xB7\xD7\x29" + "\x64\x50\xF9\x32\x13\xFB\x74\x61" + "\xF4\xED\x52\xD3\xC5\x10\x55\x3C", + .ilen = 32, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .result = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .rlen = 16, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00\x00", + .klen = 20, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .input = "\xC1\x0C\x8A\x43\x0E\x8E\x35\x19" + "\x89\xB0\xC4\xBD\xC4\xB7\xD7\x29" + "\x98\x14\xA1\x42\x37\x80\xFD\x90" + "\x68\x12\x01\xA8\x91\x89\xB9\x83" + "\x5B\x11\x77\x12\x9B\xFF\x24\x89" + "\x94\x5F\x18\x12\xBA\x27\x09\x39" + "\x99\x96\x76\x42\x15\x1C\xCD\xCB" + "\xDC\xD3\xDA\x65\x73\xAF\x80\xCD" + "\xD2\xB6\xC2\x4A\x76\xC2\x92\x85" + "\xBD\xCF\x62\x98\x58\x14\xE5\xBD", + .ilen = 80, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .result = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .rlen = 64, + }, { + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x00\x00\x00\x00", + .klen = 20, + .iv = "\x00\x00\x45\x67\x89\xab\xcd\xef", + .input = "\xC1\x76\x33\x85\xE2\x9B\x5F\xDE" + "\xDE\x89\x3D\x42\xE7\xC9\x69\x8A" + "\x44\x6D\xC3\x88\x46\x2E\xC2\x01" + "\x5E\xF6\x0C\x39\xF0\xC4\xA5\x82" + "\xCD\xE8\x31\xCC\x0A\x4C\xE4\x44" + "\x41\xA9\x82\x6F\x22\xA1\x23\x1A" + "\xA8\xE3\x16\xFD\x31\x5C\x27\x31" + "\xF1\x7F\x01\x63\xA3\xAF\x70\xA1" + "\xCF\x07\x57\x41\x67\xD0\xC4\x42" + "\xDB\x18\xC6\x4C\x4C\xE0\x3D\x9F" + "\x05\x07\xFB\x13\x7D\x4A\xCA\x5B" + "\xF0\xBF\x64\x7E\x05\xB1\x72\xEE" + "\x7C\x3B\xD4\xCD\x14\x03\xB2\x2C" + "\xD3\xA9\xEE\xFA\x17\xFC\x9C\xDF" + "\xC7\x75\x40\xFF\xAE\xAD\x1E\x59" + "\x2F\x30\x24\xFB\xAD\x6B\x10\xFA" + "\x6C\x9F\x5B\xE7\x25\xD5\xD0\x25" + "\xAC\x4A\x4B\xDA\xFC\x7A\x85\x1B" + "\x7E\x13\x06\x82\x08\x17\xA4\x35" + "\xEC\xC5\x8D\x63\x96\x81\x0A\x8F" + "\xA3\x05\x38\x95\x20\x1A\x47\x04" + "\x6F\x6D\xDA\x8F\xEF\xC1\x76\x35" + "\x6B\xC7\x4D\x0F\x94\x12\xCA\x3E" + "\x2E\xD5\x03\x2E\x86\x7E\xAA\x3B" + "\x37\x08\x1C\xCF\xBA\x5D\x71\x46" + "\x80\x72\xB0\x4C\x82\x0D\x60\x3C", + .ilen = 208, + .assoc = "\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa" + "\xaa\xaa\xaa\xaa\x00\x00\x45\x67" + "\x89\xab\xcd\xef", + .alen = 20, + .result = "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff", + .rlen = 192, + }, { + .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" + "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" + "\x2E\x44\x3B\x68", + .klen = 20, + .iv = "\x49\x56\xED\x7E\x3B\x24\x4C\xFE", + .result = "\x45\x00\x00\x48\x69\x9A\x00\x00" + "\x80\x11\x4D\xB7\xC0\xA8\x01\x02" + "\xC0\xA8\x01\x01\x0A\x9B\xF1\x56" + "\x38\xD3\x01\x00\x00\x01\x00\x00" + "\x00\x00\x00\x00\x04\x5F\x73\x69" + "\x70\x04\x5F\x75\x64\x70\x03\x73" + "\x69\x70\x09\x63\x79\x62\x65\x72" + "\x63\x69\x74\x79\x02\x64\x6B\x00" + "\x00\x21\x00\x01\x01\x02\x02\x01", + .rlen = 72, + .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" + "\x00\x00\x00\x00\x49\x56\xED\x7E" + "\x3B\x24\x4C\xFE", + .alen = 20, + .input = "\xFE\xCF\x53\x7E\x72\x9D\x5B\x07" + "\xDC\x30\xDF\x52\x8D\xD2\x2B\x76" + "\x8D\x1B\x98\x73\x66\x96\xA6\xFD" + "\x34\x85\x09\xFA\x13\xCE\xAC\x34" + "\xCF\xA2\x43\x6F\x14\xA3\xF3\xCF" + "\x65\x92\x5B\xF1\xF4\xA1\x3C\x5D" + "\x15\xB2\x1E\x18\x84\xF5\xFF\x62" + "\x47\xAE\xAB\xB7\x86\xB9\x3B\xCE" + "\x61\xBC\x17\xD7\x68\xFD\x97\x32" + "\x45\x90\x18\x14\x8F\x6C\xBE\x72" + "\x2F\xD0\x47\x96\x56\x2D\xFD\xB4", + .ilen = 88, + }, { + .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\x6D\x6A\x8F\x94\x67\x30\x83\x08" + "\xCA\xFE\xBA\xBE", + .klen = 20, + .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .result = "\x45\x00\x00\x3E\x69\x8F\x00\x00" + "\x80\x11\x4D\xCC\xC0\xA8\x01\x02" + "\xC0\xA8\x01\x01\x0A\x98\x00\x35" + "\x00\x2A\x23\x43\xB2\xD0\x01\x00" + "\x00\x01\x00\x00\x00\x00\x00\x00" + "\x03\x73\x69\x70\x09\x63\x79\x62" + "\x65\x72\x63\x69\x74\x79\x02\x64" + "\x6B\x00\x00\x01\x00\x01\x00\x01", + .rlen = 64, + .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" + "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .alen = 16, + .input = "\xDE\xB2\x2C\xD9\xB0\x7C\x72\xC1" + "\x6E\x3A\x65\xBE\xEB\x8D\xF3\x04" + "\xA5\xA5\x89\x7D\x33\xAE\x53\x0F" + "\x1B\xA7\x6D\x5D\x11\x4D\x2A\x5C" + "\x3D\xE8\x18\x27\xC1\x0E\x9A\x4F" + "\x51\x33\x0D\x0E\xEC\x41\x66\x42" + "\xCF\xBB\x85\xA5\xB4\x7E\x48\xA4" + "\xEC\x3B\x9B\xA9\x5D\x91\x8B\xD1" + "\x83\xB7\x0D\x3A\xA8\xBC\x6E\xE4" + "\xC3\x09\xE9\xD8\x5A\x41\xAD\x4A", + .ilen = 80, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x11\x22\x33\x44", + .klen = 36, + .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", + .result = "\x45\x00\x00\x30\x69\xA6\x40\x00" + "\x80\x06\x26\x90\xC0\xA8\x01\x02" + "\x93\x89\x15\x5E\x0A\x9E\x00\x8B" + "\x2D\xC5\x7E\xE0\x00\x00\x00\x00" + "\x70\x02\x40\x00\x20\xBF\x00\x00" + "\x02\x04\x05\xB4\x01\x01\x04\x02" + "\x01\x02\x02\x01", + .rlen = 52, + .assoc = "\x4A\x2C\xBF\xE3\x00\x00\x00\x02" + "\x01\x02\x03\x04\x05\x06\x07\x08", + .alen = 16, + .input = "\xFF\x42\x5C\x9B\x72\x45\x99\xDF" + "\x7A\x3B\xCD\x51\x01\x94\xE0\x0D" + "\x6A\x78\x10\x7F\x1B\x0B\x1C\xBF" + "\x06\xEF\xAE\x9D\x65\xA5\xD7\x63" + "\x74\x8A\x63\x79\x85\x77\x1D\x34" + "\x7F\x05\x45\x65\x9F\x14\xE9\x9D" + "\xEF\x84\x2D\x8E\xB3\x35\xF4\xEE" + "\xCF\xDB\xF8\x31\x82\x4B\x4C\x49" + "\x15\x95\x6C\x96", + .ilen = 68, + }, { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00", + .klen = 20, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", + .result = "\x45\x00\x00\x3C\x99\xC5\x00\x00" + "\x80\x01\xCB\x7A\x40\x67\x93\x18" + "\x01\x01\x01\x01\x08\x00\x07\x5C" + "\x02\x00\x44\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x75\x76\x77\x61\x62\x63\x64\x65" + "\x66\x67\x68\x69\x01\x02\x02\x01", + .rlen = 64, + .assoc = "\x00\x00\x00\x00\x00\x00\x00\x01" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .alen = 16, + .input = "\x46\x88\xDA\xF2\xF9\x73\xA3\x92" + "\x73\x29\x09\xC3\x31\xD5\x6D\x60" + "\xF6\x94\xAB\xAA\x41\x4B\x5E\x7F" + "\xF5\xFD\xCD\xFF\xF5\xE9\xA2\x84" + "\x45\x64\x76\x49\x27\x19\xFF\xB6" + "\x4D\xE7\xD9\xDC\xA1\xE1\xD8\x94" + "\xBC\x3B\xD5\x78\x73\xED\x4D\x18" + "\x1D\x19\xD4\xD5\xC8\xC1\x8A\xF3" + "\xF8\x21\xD4\x96\xEE\xB0\x96\xE9" + "\x8A\xD2\xB6\x9E\x47\x99\xC7\x1D", + .ilen = 80, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E\x43", + .klen = 20, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .result = "\x45\x00\x00\x3C\x99\xC3\x00\x00" + "\x80\x01\xCB\x7C\x40\x67\x93\x18" + "\x01\x01\x01\x01\x08\x00\x08\x5C" + "\x02\x00\x43\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x75\x76\x77\x61\x62\x63\x64\x65" + "\x66\x67\x68\x69\x01\x02\x02\x01", + .rlen = 64, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .input = "\xFB\xA2\xCA\xA4\x85\x3C\xF9\xF0" + "\xF2\x2C\xB1\x0D\x86\xDD\x83\xB0" + "\xFE\xC7\x56\x91\xCF\x1A\x04\xB0" + "\x0D\x11\x38\xEC\x9C\x35\x79\x17" + "\x65\xAC\xBD\x87\x01\xAD\x79\x84" + "\x5B\xF9\xFE\x3F\xBA\x48\x7B\xC9" + "\x17\x55\xE6\x66\x2B\x4C\x8D\x0D" + "\x1F\x5E\x22\x73\x95\x30\x32\x0A" + "\xE0\xD7\x31\xCC\x97\x8E\xCA\xFA" + "\xEA\xE8\x8F\x00\xE8\x0D\x6E\x48", + .ilen = 80, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E\x43", + .klen = 20, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .result = "\x45\x00\x00\x1C\x42\xA2\x00\x00" + "\x80\x01\x44\x1F\x40\x67\x93\xB6" + "\xE0\x00\x00\x02\x0A\x00\xF5\xFF" + "\x01\x02\x02\x01", + .rlen = 28, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .input = "\xFB\xA2\xCA\x84\x5E\x5D\xF9\xF0" + "\xF2\x2C\x3E\x6E\x86\xDD\x83\x1E" + "\x1F\xC6\x57\x92\xCD\x1A\xF9\x13" + "\x0E\x13\x79\xED\x36\x9F\x07\x1F" + "\x35\xE0\x34\xBE\x95\xF1\x12\xE4" + "\xE7\xD0\x5D\x35", + .ilen = 44, + }, { + .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\x6D\x6A\x8F\x94\x67\x30\x83\x08" + "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\xCA\xFE\xBA\xBE", + .klen = 28, + .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .result = "\x45\x00\x00\x28\xA4\xAD\x40\x00" + "\x40\x06\x78\x80\x0A\x01\x03\x8F" + "\x0A\x01\x06\x12\x80\x23\x06\xB8" + "\xCB\x71\x26\x02\xDD\x6B\xB0\x3E" + "\x50\x10\x16\xD0\x75\x68\x00\x01", + .rlen = 40, + .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" + "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .alen = 16, + .input = "\xA5\xB1\xF8\x06\x60\x29\xAE\xA4" + "\x0E\x59\x8B\x81\x22\xDE\x02\x42" + "\x09\x38\xB3\xAB\x33\xF8\x28\xE6" + "\x87\xB8\x85\x8B\x5B\xFB\xDB\xD0" + "\x31\x5B\x27\x45\x21\x44\xCC\x77" + "\x95\x45\x7B\x96\x52\x03\x7F\x53" + "\x18\x02\x7B\x5B\x4C\xD7\xA6\x36", + .ilen = 56, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xDE\xCA\xF8\x88", + .klen = 20, + .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", + .result = "\x45\x00\x00\x49\x33\xBA\x00\x00" + "\x7F\x11\x91\x06\xC3\xFB\x1D\x10" + "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" + "\x00\x35\xDD\x7B\x80\x03\x02\xD5" + "\x00\x00\x4E\x20\x00\x1E\x8C\x18" + "\xD7\x5B\x81\xDC\x91\xBA\xA0\x47" + "\x6B\x91\xB9\x24\xB2\x80\x38\x9D" + "\x92\xC9\x63\xBA\xC0\x46\xEC\x95" + "\x9B\x62\x66\xC0\x47\x22\xB1\x49" + "\x23\x01\x01\x01", + .rlen = 76, + .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" + "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" + "\xCE\xFA\xCE\x74", + .alen = 20, + .input = "\x18\xA6\xFD\x42\xF7\x2C\xBF\x4A" + "\xB2\xA2\xEA\x90\x1F\x73\xD8\x14" + "\xE3\xE7\xF2\x43\xD9\x54\x12\xE1" + "\xC3\x49\xC1\xD2\xFB\xEC\x16\x8F" + "\x91\x90\xFE\xEB\xAF\x2C\xB0\x19" + "\x84\xE6\x58\x63\x96\x5D\x74\x72" + "\xB7\x9D\xA3\x45\xE0\xE7\x80\x19" + "\x1F\x0D\x2F\x0E\x0F\x49\x6C\x22" + "\x6F\x21\x27\xB2\x7D\xB3\x57\x24" + "\xE7\x84\x5D\x68\x65\x1F\x57\xE6" + "\x5F\x35\x4F\x75\xFF\x17\x01\x57" + "\x69\x62\x34\x36", + .ilen = 92, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x73\x61\x6C\x74", + .klen = 36, + .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", + .result = "\x45\x08\x00\x28\x73\x2C\x00\x00" + "\x40\x06\xE9\xF9\x0A\x01\x06\x12" + "\x0A\x01\x03\x8F\x06\xB8\x80\x23" + "\xDD\x6B\xAF\xBE\xCB\x71\x26\x02" + "\x50\x10\x1F\x64\x6D\x54\x00\x01", + .rlen = 40, + .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" + "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" + "\x69\x76\x65\x63", + .alen = 20, + .input = "\xF2\xD6\x9E\xCD\xBD\x5A\x0D\x5B" + "\x8D\x5E\xF3\x8B\xAD\x4D\xA5\x8D" + "\x1F\x27\x8F\xDE\x98\xEF\x67\x54" + "\x9D\x52\x4A\x30\x18\xD9\xA5\x7F" + "\xF4\xD3\xA3\x1C\xE6\x73\x11\x9E" + "\x45\x16\x26\xC2\x41\x57\x71\xE3" + "\xB7\xEE\xBC\xA6\x14\xC8\x9B\x35", + .ilen = 56, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E\x43", + .klen = 20, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .result = "\x45\x00\x00\x49\x33\x3E\x00\x00" + "\x7F\x11\x91\x82\xC3\xFB\x1D\x10" + "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" + "\x00\x35\xCB\x45\x80\x03\x02\x5B" + "\x00\x00\x01\xE0\x00\x1E\x8C\x18" + "\xD6\x57\x59\xD5\x22\x84\xA0\x35" + "\x2C\x71\x47\x5C\x88\x80\x39\x1C" + "\x76\x4D\x6E\x5E\xE0\x49\x6B\x32" + "\x5A\xE2\x70\xC0\x38\x99\x49\x39" + "\x15\x01\x01\x01", + .rlen = 76, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .input = "\xFB\xA2\xCA\xD1\x2F\xC1\xF9\xF0" + "\x0D\x3C\xEB\xF3\x05\x41\x0D\xB8" + "\x3D\x77\x84\xB6\x07\x32\x3D\x22" + "\x0F\x24\xB0\xA9\x7D\x54\x18\x28" + "\x00\xCA\xDB\x0F\x68\xD9\x9E\xF0" + "\xE0\xC0\xC8\x9A\xE9\xBE\xA8\x88" + "\x4E\x52\xD6\x5B\xC1\xAF\xD0\x74" + "\x0F\x74\x24\x44\x74\x7B\x5B\x39" + "\xAB\x53\x31\x63\xAA\xD4\x55\x0E" + "\xE5\x16\x09\x75\xCD\xB6\x08\xC5" + "\x76\x91\x89\x60\x97\x63\xB8\xE1" + "\x8C\xAA\x81\xE2", + .ilen = 92, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x73\x61\x6C\x74", + .klen = 36, + .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", + .result = "\x63\x69\x73\x63\x6F\x01\x72\x75" + "\x6C\x65\x73\x01\x74\x68\x65\x01" + "\x6E\x65\x74\x77\x65\x01\x64\x65" + "\x66\x69\x6E\x65\x01\x74\x68\x65" + "\x74\x65\x63\x68\x6E\x6F\x6C\x6F" + "\x67\x69\x65\x73\x01\x74\x68\x61" + "\x74\x77\x69\x6C\x6C\x01\x64\x65" + "\x66\x69\x6E\x65\x74\x6F\x6D\x6F" + "\x72\x72\x6F\x77\x01\x02\x02\x01", + .rlen = 72, + .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" + "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" + "\x69\x76\x65\x63", + .alen = 20, + .input = "\xD4\xB7\xED\x86\xA1\x77\x7F\x2E" + "\xA1\x3D\x69\x73\xD3\x24\xC6\x9E" + "\x7B\x43\xF8\x26\xFB\x56\x83\x12" + "\x26\x50\x8B\xEB\xD2\xDC\xEB\x18" + "\xD0\xA6\xDF\x10\xE5\x48\x7D\xF0" + "\x74\x11\x3E\x14\xC6\x41\x02\x4E" + "\x3E\x67\x73\xD9\x1A\x62\xEE\x42" + "\x9B\x04\x3A\x10\xE3\xEF\xE6\xB0" + "\x12\xA4\x93\x63\x41\x23\x64\xF8" + "\xC0\xCA\xC5\x87\xF2\x49\xE5\x6B" + "\x11\xE2\x4F\x30\xE4\x4C\xCC\x76", + .ilen = 88, + }, { + .key = "\x7D\x77\x3D\x00\xC1\x44\xC5\x25" + "\xAC\x61\x9D\x18\xC8\x4A\x3F\x47" + "\xD9\x66\x42\x67", + .klen = 20, + .iv = "\x43\x45\x7E\x91\x82\x44\x3B\xC6", + .result = "\x01\x02\x02\x01", + .rlen = 4, + .assoc = "\x33\x54\x67\xAE\xFF\xFF\xFF\xFF" + "\x43\x45\x7E\x91\x82\x44\x3B\xC6", + .alen = 16, + .input = "\x43\x7F\x86\x6B\xCB\x3F\x69\x9F" + "\xE9\xB0\x82\x2B\xAC\x96\x1C\x45" + "\x04\xBE\xF2\x70", + .ilen = 20, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xDE\xCA\xF8\x88", + .klen = 20, + .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", + .result = "\x74\x6F\x01\x62\x65\x01\x6F\x72" + "\x01\x6E\x6F\x74\x01\x74\x6F\x01" + "\x62\x65\x00\x01", + .rlen = 20, + .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" + "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" + "\xCE\xFA\xCE\x74", + .alen = 20, + .input = "\x29\xC9\xFC\x69\xA1\x97\xD0\x38" + "\xCC\xDD\x14\xE2\xDD\xFC\xAA\x05" + "\x43\x33\x21\x64\x41\x25\x03\x52" + "\x43\x03\xED\x3C\x6C\x5F\x28\x38" + "\x43\xAF\x8C\x3E", + .ilen = 36, + }, { + .key = "\x6C\x65\x67\x61\x6C\x69\x7A\x65" + "\x6D\x61\x72\x69\x6A\x75\x61\x6E" + "\x61\x61\x6E\x64\x64\x6F\x69\x74" + "\x62\x65\x66\x6F\x72\x65\x69\x61" + "\x74\x75\x72\x6E", + .klen = 36, + .iv = "\x33\x30\x21\x69\x67\x65\x74\x6D", + .result = "\x45\x00\x00\x30\xDA\x3A\x00\x00" + "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" + "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01", + .rlen = 52, + .assoc = "\x79\x6B\x69\x63\xFF\xFF\xFF\xFF" + "\xFF\xFF\xFF\xFF\x33\x30\x21\x69" + "\x67\x65\x74\x6D", + .alen = 20, + .input = "\xF9\x7A\xB2\xAA\x35\x6D\x8E\xDC" + "\xE1\x76\x44\xAC\x8C\x78\xE2\x5D" + "\xD2\x4D\xED\xBB\x29\xEB\xF1\xB6" + "\x4A\x27\x4B\x39\xB4\x9C\x3A\x86" + "\x4C\xD3\xD7\x8C\xA4\xAE\x68\xA3" + "\x2B\x42\x45\x8F\xB5\x7D\xBE\x82" + "\x1D\xCC\x63\xB9\xD0\x93\x7B\xA2" + "\x94\x5F\x66\x93\x68\x66\x1A\x32" + "\x9F\xB4\xC0\x53", + .ilen = 68, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E\x43", + .klen = 20, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .result = "\x45\x00\x00\x30\xDA\x3A\x00\x00" + "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" + "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01", + .rlen = 52, + .assoc = "\x3F\x7E\xF6\x42\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .input = "\xFB\xA2\xCA\xA8\xC6\xC5\xF9\xF0" + "\xF2\x2C\xA5\x4A\x06\x12\x10\xAD" + "\x3F\x6E\x57\x91\xCF\x1A\xCA\x21" + "\x0D\x11\x7C\xEC\x9C\x35\x79\x17" + "\x65\xAC\xBD\x87\x01\xAD\x79\x84" + "\x5B\xF9\xFE\x3F\xBA\x48\x7B\xC9" + "\x63\x21\x93\x06\x84\xEE\xCA\xDB" + "\x56\x91\x25\x46\xE7\xA9\x5C\x97" + "\x40\xD7\xCB\x05", + .ilen = 68, + }, { + .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" + "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" + "\x22\x43\x3C\x64", + .klen = 20, + .iv = "\x48\x55\xEC\x7D\x3A\x23\x4B\xFD", + .result = "\x08\x00\xC6\xCD\x02\x00\x07\x00" + "\x61\x62\x63\x64\x65\x66\x67\x68" + "\x69\x6A\x6B\x6C\x6D\x6E\x6F\x70" + "\x71\x72\x73\x74\x01\x02\x02\x01", + .rlen = 32, + .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" + "\x00\x00\x00\x07\x48\x55\xEC\x7D" + "\x3A\x23\x4B\xFD", + .alen = 20, + .input = "\x74\x75\x2E\x8A\xEB\x5D\x87\x3C" + "\xD7\xC0\xF4\xAC\xC3\x6C\x4B\xFF" + "\x84\xB7\xD7\xB9\x8F\x0C\xA8\xB6" + "\xAC\xDA\x68\x94\xBC\x61\x90\x69" + "\xEF\x9C\xBC\x28\xFE\x1B\x56\xA7" + "\xC4\xE0\xD5\x8C\x86\xCD\x2B\xC0", + .ilen = 48, + } +}; + +static const struct aead_testvec aes_gcm_rfc4543_enc_tv_template[] = { + { /* From draft-mcgrew-gcm-test-01 */ + .key = "\x4c\x80\xcd\xef\xbb\x5d\x10\xda" + "\x90\x6a\xc7\x3c\x36\x13\xa6\x34" + "\x22\x43\x3c\x64", + .klen = 20, + .iv = zeroed_string, + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x07" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .alen = 16, + .input = "\x45\x00\x00\x30\xda\x3a\x00\x00" + "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" + "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6a\x6b\x6c" + "\x6d\x6e\x6f\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01", + .ilen = 52, + .result = "\x45\x00\x00\x30\xda\x3a\x00\x00" + "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" + "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6a\x6b\x6c" + "\x6d\x6e\x6f\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01\xf2\xa9\xa8\x36" + "\xe1\x55\x10\x6a\xa8\xdc\xd6\x18" + "\xe4\x09\x9a\xaa", + .rlen = 68, + } +}; + +static const struct aead_testvec aes_gcm_rfc4543_dec_tv_template[] = { + { /* From draft-mcgrew-gcm-test-01 */ + .key = "\x4c\x80\xcd\xef\xbb\x5d\x10\xda" + "\x90\x6a\xc7\x3c\x36\x13\xa6\x34" + "\x22\x43\x3c\x64", + .klen = 20, + .iv = zeroed_string, + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x07" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .alen = 16, + .input = "\x45\x00\x00\x30\xda\x3a\x00\x00" + "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" + "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6a\x6b\x6c" + "\x6d\x6e\x6f\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01\xf2\xa9\xa8\x36" + "\xe1\x55\x10\x6a\xa8\xdc\xd6\x18" + "\xe4\x09\x9a\xaa", + .ilen = 68, + .result = "\x45\x00\x00\x30\xda\x3a\x00\x00" + "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" + "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6a\x6b\x6c" + "\x6d\x6e\x6f\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01", + .rlen = 52, + }, { /* nearly same as previous, but should fail */ + .key = "\x4c\x80\xcd\xef\xbb\x5d\x10\xda" + "\x90\x6a\xc7\x3c\x36\x13\xa6\x34" + "\x22\x43\x3c\x64", + .klen = 20, + .iv = zeroed_string, + .assoc = "\x00\x00\x43\x21\x00\x00\x00\x07" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .alen = 16, + .input = "\x45\x00\x00\x30\xda\x3a\x00\x00" + "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" + "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6a\x6b\x6c" + "\x6d\x6e\x6f\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01\xf2\xa9\xa8\x36" + "\xe1\x55\x10\x6a\xa8\xdc\xd6\x18" + "\x00\x00\x00\x00", + .ilen = 68, + .novrfy = 1, + .result = "\x45\x00\x00\x30\xda\x3a\x00\x00" + "\x80\x01\xdf\x3b\xc0\xa8\x00\x05" + "\xc0\xa8\x00\x01\x08\x00\xc6\xcd" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6a\x6b\x6c" + "\x6d\x6e\x6f\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01", + .rlen = 52, + }, +}; + +static const struct aead_testvec aes_ccm_enc_tv_template[] = { + { /* From RFC 3610 */ + .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", + .klen = 16, + .iv = "\x01\x00\x00\x00\x03\x02\x01\x00" + "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07", + .alen = 8, + .input = "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e", + .ilen = 23, + .result = "\x58\x8c\x97\x9a\x61\xc6\x63\xd2" + "\xf0\x66\xd0\xc2\xc0\xf9\x89\x80" + "\x6d\x5f\x6b\x61\xda\xc3\x84\x17" + "\xe8\xd1\x2c\xfd\xf9\x26\xe0", + .rlen = 31, + }, { + .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", + .klen = 16, + .iv = "\x01\x00\x00\x00\x07\x06\x05\x04" + "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b", + .alen = 12, + .input = "\x0c\x0d\x0e\x0f\x10\x11\x12\x13" + "\x14\x15\x16\x17\x18\x19\x1a\x1b" + "\x1c\x1d\x1e\x1f", + .ilen = 20, + .result = "\xdc\xf1\xfb\x7b\x5d\x9e\x23\xfb" + "\x9d\x4e\x13\x12\x53\x65\x8a\xd8" + "\x6e\xbd\xca\x3e\x51\xe8\x3f\x07" + "\x7d\x9c\x2d\x93", + .rlen = 28, + }, { + .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", + .klen = 16, + .iv = "\x01\x00\x00\x00\x0b\x0a\x09\x08" + "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07", + .alen = 8, + .input = "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20", + .ilen = 25, + .result = "\x82\x53\x1a\x60\xcc\x24\x94\x5a" + "\x4b\x82\x79\x18\x1a\xb5\xc8\x4d" + "\xf2\x1c\xe7\xf9\xb7\x3f\x42\xe1" + "\x97\xea\x9c\x07\xe5\x6b\x5e\xb1" + "\x7e\x5f\x4e", + .rlen = 35, + }, { + .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", + .klen = 16, + .iv = "\x01\x00\x00\x00\x0c\x0b\x0a\x09" + "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b", + .alen = 12, + .input = "\x0c\x0d\x0e\x0f\x10\x11\x12\x13" + "\x14\x15\x16\x17\x18\x19\x1a\x1b" + "\x1c\x1d\x1e", + .ilen = 19, + .result = "\x07\x34\x25\x94\x15\x77\x85\x15" + "\x2b\x07\x40\x98\x33\x0a\xbb\x14" + "\x1b\x94\x7b\x56\x6a\xa9\x40\x6b" + "\x4d\x99\x99\x88\xdd", + .rlen = 29, + }, { + .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" + "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", + .klen = 16, + .iv = "\x01\x00\x33\x56\x8e\xf7\xb2\x63" + "\x3c\x96\x96\x76\x6c\xfa\x00\x00", + .assoc = "\x63\x01\x8f\x76\xdc\x8a\x1b\xcb", + .alen = 8, + .input = "\x90\x20\xea\x6f\x91\xbd\xd8\x5a" + "\xfa\x00\x39\xba\x4b\xaf\xf9\xbf" + "\xb7\x9c\x70\x28\x94\x9c\xd0\xec", + .ilen = 24, + .result = "\x4c\xcb\x1e\x7c\xa9\x81\xbe\xfa" + "\xa0\x72\x6c\x55\xd3\x78\x06\x12" + "\x98\xc8\x5c\x92\x81\x4a\xbc\x33" + "\xc5\x2e\xe8\x1d\x7d\x77\xc0\x8a", + .rlen = 32, + }, { + .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" + "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", + .klen = 16, + .iv = "\x01\x00\xd5\x60\x91\x2d\x3f\x70" + "\x3c\x96\x96\x76\x6c\xfa\x00\x00", + .assoc = "\xcd\x90\x44\xd2\xb7\x1f\xdb\x81" + "\x20\xea\x60\xc0", + .alen = 12, + .input = "\x64\x35\xac\xba\xfb\x11\xa8\x2e" + "\x2f\x07\x1d\x7c\xa4\xa5\xeb\xd9" + "\x3a\x80\x3b\xa8\x7f", + .ilen = 21, + .result = "\x00\x97\x69\xec\xab\xdf\x48\x62" + "\x55\x94\xc5\x92\x51\xe6\x03\x57" + "\x22\x67\x5e\x04\xc8\x47\x09\x9e" + "\x5a\xe0\x70\x45\x51", + .rlen = 29, + }, { + .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" + "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", + .klen = 16, + .iv = "\x01\x00\x42\xff\xf8\xf1\x95\x1c" + "\x3c\x96\x96\x76\x6c\xfa\x00\x00", + .assoc = "\xd8\x5b\xc7\xe6\x9f\x94\x4f\xb8", + .alen = 8, + .input = "\x8a\x19\xb9\x50\xbc\xf7\x1a\x01" + "\x8e\x5e\x67\x01\xc9\x17\x87\x65" + "\x98\x09\xd6\x7d\xbe\xdd\x18", + .ilen = 23, + .result = "\xbc\x21\x8d\xaa\x94\x74\x27\xb6" + "\xdb\x38\x6a\x99\xac\x1a\xef\x23" + "\xad\xe0\xb5\x29\x39\xcb\x6a\x63" + "\x7c\xf9\xbe\xc2\x40\x88\x97\xc6" + "\xba", + .rlen = 33, + }, { + /* This is taken from FIPS CAVS. */ + .key = "\x83\xac\x54\x66\xc2\xeb\xe5\x05" + "\x2e\x01\xd1\xfc\x5d\x82\x66\x2e", + .klen = 16, + .iv = "\x03\x96\xac\x59\x30\x07\xa1\xe2\xa2\xc7\x55\x24\0\0\0\0", + .alen = 0, + .input = "\x19\xc8\x81\xf6\xe9\x86\xff\x93" + "\x0b\x78\x67\xe5\xbb\xb7\xfc\x6e" + "\x83\x77\xb3\xa6\x0c\x8c\x9f\x9c" + "\x35\x2e\xad\xe0\x62\xf9\x91\xa1", + .ilen = 32, + .result = "\xab\x6f\xe1\x69\x1d\x19\x99\xa8" + "\x92\xa0\xc4\x6f\x7e\xe2\x8b\xb1" + "\x70\xbb\x8c\xa6\x4c\x6e\x97\x8a" + "\x57\x2b\xbe\x5d\x98\xa6\xb1\x32" + "\xda\x24\xea\xd9\xa1\x39\x98\xfd" + "\xa4\xbe\xd9\xf2\x1a\x6d\x22\xa8", + .rlen = 48, + }, { + .key = "\x1e\x2c\x7e\x01\x41\x9a\xef\xc0" + "\x0d\x58\x96\x6e\x5c\xa2\x4b\xd3", + .klen = 16, + .iv = "\x03\x4f\xa3\x19\xd3\x01\x5a\xd8" + "\x30\x60\x15\x56\x00\x00\x00\x00", + .assoc = "\xda\xe6\x28\x9c\x45\x2d\xfd\x63" + "\x5e\xda\x4c\xb6\xe6\xfc\xf9\xb7" + "\x0c\x56\xcb\xe4\xe0\x05\x7a\xe1" + "\x0a\x63\x09\x78\xbc\x2c\x55\xde", + .alen = 32, + .input = "\x87\xa3\x36\xfd\x96\xb3\x93\x78" + "\xa9\x28\x63\xba\x12\xa3\x14\x85" + "\x57\x1e\x06\xc9\x7b\x21\xef\x76" + "\x7f\x38\x7e\x8e\x29\xa4\x3e\x7e", + .ilen = 32, + .result = "\x8a\x1e\x11\xf0\x02\x6b\xe2\x19" + "\xfc\x70\xc4\x6d\x8e\xb7\x99\xab" + "\xc5\x4b\xa2\xac\xd3\xf3\x48\xff" + "\x3b\xb5\xce\x53\xef\xde\xbb\x02" + "\xa9\x86\x15\x6c\x13\xfe\xda\x0a" + "\x22\xb8\x29\x3d\xd8\x39\x9a\x23", + .rlen = 48, + }, { + .key = "\xf4\x6b\xc2\x75\x62\xfe\xb4\xe1" + "\xa3\xf0\xff\xdd\x4e\x4b\x12\x75" + "\x53\x14\x73\x66\x8d\x88\xf6\x80", + .klen = 24, + .iv = "\x03\xa0\x20\x35\x26\xf2\x21\x8d" + "\x50\x20\xda\xe2\x00\x00\x00\x00", + .assoc = "\x5b\x9e\x13\x67\x02\x5e\xef\xc1" + "\x6c\xf9\xd7\x1e\x52\x8f\x7a\x47" + "\xe9\xd4\xcf\x20\x14\x6e\xf0\x2d" + "\xd8\x9e\x2b\x56\x10\x23\x56\xe7", + .alen = 32, + .result = "\x36\xea\x7a\x70\x08\xdc\x6a\xbc" + "\xad\x0c\x7a\x63\xf6\x61\xfd\x9b", + .rlen = 16, + }, { + .key = "\x56\xdf\x5c\x8f\x26\x3f\x0e\x42" + "\xef\x7a\xd3\xce\xfc\x84\x60\x62" + "\xca\xb4\x40\xaf\x5f\xc9\xc9\x01", + .klen = 24, + .iv = "\x03\xd6\x3c\x8c\x86\x84\xb6\xcd" + "\xef\x09\x2e\x94\x00\x00\x00\x00", + .assoc = "\x02\x65\x78\x3c\xe9\x21\x30\x91" + "\xb1\xb9\xda\x76\x9a\x78\x6d\x95" + "\xf2\x88\x32\xa3\xf2\x50\xcb\x4c" + "\xe3\x00\x73\x69\x84\x69\x87\x79", + .alen = 32, + .input = "\x9f\xd2\x02\x4b\x52\x49\x31\x3c" + "\x43\x69\x3a\x2d\x8e\x70\xad\x7e" + "\xe0\xe5\x46\x09\x80\x89\x13\xb2" + "\x8c\x8b\xd9\x3f\x86\xfb\xb5\x6b", + .ilen = 32, + .result = "\x39\xdf\x7c\x3c\x5a\x29\xb9\x62" + "\x5d\x51\xc2\x16\xd8\xbd\x06\x9f" + "\x9b\x6a\x09\x70\xc1\x51\x83\xc2" + "\x66\x88\x1d\x4f\x9a\xda\xe0\x1e" + "\xc7\x79\x11\x58\xe5\x6b\x20\x40" + "\x7a\xea\x46\x42\x8b\xe4\x6f\xe1", + .rlen = 48, + }, { + .key = "\xe0\x8d\x99\x71\x60\xd7\x97\x1a" + "\xbd\x01\x99\xd5\x8a\xdf\x71\x3a" + "\xd3\xdf\x24\x4b\x5e\x3d\x4b\x4e" + "\x30\x7a\xb9\xd8\x53\x0a\x5e\x2b", + .klen = 32, + .iv = "\x03\x1e\x29\x91\xad\x8e\xc1\x53" + "\x0a\xcf\x2d\xbe\x00\x00\x00\x00", + .assoc = "\x19\xb6\x1f\x57\xc4\xf3\xf0\x8b" + "\x78\x2b\x94\x02\x29\x0f\x42\x27" + "\x6b\x75\xcb\x98\x34\x08\x7e\x79" + "\xe4\x3e\x49\x0d\x84\x8b\x22\x87", + .alen = 32, + .input = "\xe1\xd9\xd8\x13\xeb\x3a\x75\x3f" + "\x9d\xbd\x5f\x66\xbe\xdc\xbb\x66" + "\xbf\x17\x99\x62\x4a\x39\x27\x1f" + "\x1d\xdc\x24\xae\x19\x2f\x98\x4c", + .ilen = 32, + .result = "\x19\xb8\x61\x33\x45\x2b\x43\x96" + "\x6f\x51\xd0\x20\x30\x7d\x9b\xc6" + "\x26\x3d\xf8\xc9\x65\x16\xa8\x9f" + "\xf0\x62\x17\x34\xf2\x1e\x8d\x75" + "\x4e\x13\xcc\xc0\xc3\x2a\x54\x2d", + .rlen = 40, + }, { + .key = "\x7c\xc8\x18\x3b\x8d\x99\xe0\x7c" + "\x45\x41\xb8\xbd\x5c\xa7\xc2\x32" + "\x8a\xb8\x02\x59\xa4\xfe\xa9\x2c" + "\x09\x75\x9a\x9b\x3c\x9b\x27\x39", + .klen = 32, + .iv = "\x03\xf9\xd9\x4e\x63\xb5\x3d\x9d" + "\x43\xf6\x1e\x50\0\0\0\0", + .assoc = "\x57\xf5\x6b\x8b\x57\x5c\x3d\x3b" + "\x13\x02\x01\x0c\x83\x4c\x96\x35" + "\x8e\xd6\x39\xcf\x7d\x14\x9b\x94" + "\xb0\x39\x36\xe6\x8f\x57\xe0\x13", + .alen = 32, + .input = "\x3b\x6c\x29\x36\xb6\xef\x07\xa6" + "\x83\x72\x07\x4f\xcf\xfa\x66\x89" + "\x5f\xca\xb1\xba\xd5\x8f\x2c\x27" + "\x30\xdb\x75\x09\x93\xd4\x65\xe4", + .ilen = 32, + .result = "\xb0\x88\x5a\x33\xaa\xe5\xc7\x1d" + "\x85\x23\xc7\xc6\x2f\xf4\x1e\x3d" + "\xcc\x63\x44\x25\x07\x78\x4f\x9e" + "\x96\xb8\x88\xeb\xbc\x48\x1f\x06" + "\x39\xaf\x39\xac\xd8\x4a\x80\x39" + "\x7b\x72\x8a\xf7", + .rlen = 44, + }, { + .key = "\xab\xd0\xe9\x33\x07\x26\xe5\x83" + "\x8c\x76\x95\xd4\xb6\xdc\xf3\x46" + "\xf9\x8f\xad\xe3\x02\x13\x83\x77" + "\x3f\xb0\xf1\xa1\xa1\x22\x0f\x2b", + .klen = 32, + .iv = "\x03\x24\xa7\x8b\x07\xcb\xcc\x0e" + "\xe6\x33\xbf\xf5\x00\x00\x00\x00", + .assoc = "\xd4\xdb\x30\x1d\x03\xfe\xfd\x5f" + "\x87\xd4\x8c\xb6\xb6\xf1\x7a\x5d" + "\xab\x90\x65\x8d\x8e\xca\x4d\x4f" + "\x16\x0c\x40\x90\x4b\xc7\x36\x73", + .alen = 32, + .input = "\xf5\xc6\x7d\x48\xc1\xb7\xe6\x92" + "\x97\x5a\xca\xc4\xa9\x6d\xf9\x3d" + "\x6c\xde\xbc\xf1\x90\xea\x6a\xb2" + "\x35\x86\x36\xaf\x5c\xfe\x4b\x3a", + .ilen = 32, + .result = "\x83\x6f\x40\x87\x72\xcf\xc1\x13" + "\xef\xbb\x80\x21\x04\x6c\x58\x09" + "\x07\x1b\xfc\xdf\xc0\x3f\x5b\xc7" + "\xe0\x79\xa8\x6e\x71\x7c\x3f\xcf" + "\x5c\xda\xb2\x33\xe5\x13\xe2\x0d" + "\x74\xd1\xef\xb5\x0f\x3a\xb5\xf8", + .rlen = 48, + } +}; + +static const struct aead_testvec aes_ccm_dec_tv_template[] = { + { /* From RFC 3610 */ + .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", + .klen = 16, + .iv = "\x01\x00\x00\x00\x03\x02\x01\x00" + "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07", + .alen = 8, + .input = "\x58\x8c\x97\x9a\x61\xc6\x63\xd2" + "\xf0\x66\xd0\xc2\xc0\xf9\x89\x80" + "\x6d\x5f\x6b\x61\xda\xc3\x84\x17" + "\xe8\xd1\x2c\xfd\xf9\x26\xe0", + .ilen = 31, + .result = "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e", + .rlen = 23, + }, { + .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", + .klen = 16, + .iv = "\x01\x00\x00\x00\x07\x06\x05\x04" + "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b", + .alen = 12, + .input = "\xdc\xf1\xfb\x7b\x5d\x9e\x23\xfb" + "\x9d\x4e\x13\x12\x53\x65\x8a\xd8" + "\x6e\xbd\xca\x3e\x51\xe8\x3f\x07" + "\x7d\x9c\x2d\x93", + .ilen = 28, + .result = "\x0c\x0d\x0e\x0f\x10\x11\x12\x13" + "\x14\x15\x16\x17\x18\x19\x1a\x1b" + "\x1c\x1d\x1e\x1f", + .rlen = 20, + }, { + .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", + .klen = 16, + .iv = "\x01\x00\x00\x00\x0b\x0a\x09\x08" + "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07", + .alen = 8, + .input = "\x82\x53\x1a\x60\xcc\x24\x94\x5a" + "\x4b\x82\x79\x18\x1a\xb5\xc8\x4d" + "\xf2\x1c\xe7\xf9\xb7\x3f\x42\xe1" + "\x97\xea\x9c\x07\xe5\x6b\x5e\xb1" + "\x7e\x5f\x4e", + .ilen = 35, + .result = "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" + "\x20", + .rlen = 25, + }, { + .key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" + "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf", + .klen = 16, + .iv = "\x01\x00\x00\x00\x0c\x0b\x0a\x09" + "\xa0\xa1\xa2\xa3\xa4\xa5\x00\x00", + .assoc = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b", + .alen = 12, + .input = "\x07\x34\x25\x94\x15\x77\x85\x15" + "\x2b\x07\x40\x98\x33\x0a\xbb\x14" + "\x1b\x94\x7b\x56\x6a\xa9\x40\x6b" + "\x4d\x99\x99\x88\xdd", + .ilen = 29, + .result = "\x0c\x0d\x0e\x0f\x10\x11\x12\x13" + "\x14\x15\x16\x17\x18\x19\x1a\x1b" + "\x1c\x1d\x1e", + .rlen = 19, + }, { + .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" + "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", + .klen = 16, + .iv = "\x01\x00\x33\x56\x8e\xf7\xb2\x63" + "\x3c\x96\x96\x76\x6c\xfa\x00\x00", + .assoc = "\x63\x01\x8f\x76\xdc\x8a\x1b\xcb", + .alen = 8, + .input = "\x4c\xcb\x1e\x7c\xa9\x81\xbe\xfa" + "\xa0\x72\x6c\x55\xd3\x78\x06\x12" + "\x98\xc8\x5c\x92\x81\x4a\xbc\x33" + "\xc5\x2e\xe8\x1d\x7d\x77\xc0\x8a", + .ilen = 32, + .result = "\x90\x20\xea\x6f\x91\xbd\xd8\x5a" + "\xfa\x00\x39\xba\x4b\xaf\xf9\xbf" + "\xb7\x9c\x70\x28\x94\x9c\xd0\xec", + .rlen = 24, + }, { + .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" + "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", + .klen = 16, + .iv = "\x01\x00\xd5\x60\x91\x2d\x3f\x70" + "\x3c\x96\x96\x76\x6c\xfa\x00\x00", + .assoc = "\xcd\x90\x44\xd2\xb7\x1f\xdb\x81" + "\x20\xea\x60\xc0", + .alen = 12, + .input = "\x00\x97\x69\xec\xab\xdf\x48\x62" + "\x55\x94\xc5\x92\x51\xe6\x03\x57" + "\x22\x67\x5e\x04\xc8\x47\x09\x9e" + "\x5a\xe0\x70\x45\x51", + .ilen = 29, + .result = "\x64\x35\xac\xba\xfb\x11\xa8\x2e" + "\x2f\x07\x1d\x7c\xa4\xa5\xeb\xd9" + "\x3a\x80\x3b\xa8\x7f", + .rlen = 21, + }, { + .key = "\xd7\x82\x8d\x13\xb2\xb0\xbd\xc3" + "\x25\xa7\x62\x36\xdf\x93\xcc\x6b", + .klen = 16, + .iv = "\x01\x00\x42\xff\xf8\xf1\x95\x1c" + "\x3c\x96\x96\x76\x6c\xfa\x00\x00", + .assoc = "\xd8\x5b\xc7\xe6\x9f\x94\x4f\xb8", + .alen = 8, + .input = "\xbc\x21\x8d\xaa\x94\x74\x27\xb6" + "\xdb\x38\x6a\x99\xac\x1a\xef\x23" + "\xad\xe0\xb5\x29\x39\xcb\x6a\x63" + "\x7c\xf9\xbe\xc2\x40\x88\x97\xc6" + "\xba", + .ilen = 33, + .result = "\x8a\x19\xb9\x50\xbc\xf7\x1a\x01" + "\x8e\x5e\x67\x01\xc9\x17\x87\x65" + "\x98\x09\xd6\x7d\xbe\xdd\x18", + .rlen = 23, + }, { + /* This is taken from FIPS CAVS. */ + .key = "\xab\x2f\x8a\x74\xb7\x1c\xd2\xb1" + "\xff\x80\x2e\x48\x7d\x82\xf8\xb9", + .klen = 16, + .iv = "\x03\xc6\xfb\x7d\x80\x0d\x13\xab" + "\xd8\xa6\xb2\xd8\x00\x00\x00\x00", + .alen = 0, + .input = "\xd5\xe8\x93\x9f\xc7\x89\x2e\x2b", + .ilen = 8, + .result = "\x00", + .rlen = 0, + .novrfy = 1, + }, { + .key = "\xab\x2f\x8a\x74\xb7\x1c\xd2\xb1" + "\xff\x80\x2e\x48\x7d\x82\xf8\xb9", + .klen = 16, + .iv = "\x03\xaf\x94\x87\x78\x35\x82\x81" + "\x7f\x88\x94\x68\x00\x00\x00\x00", + .alen = 0, + .input = "\x41\x3c\xb8\x87\x73\xcb\xf3\xf3", + .ilen = 8, + .result = "\x00", + .rlen = 0, + }, { + .key = "\x61\x0e\x8c\xae\xe3\x23\xb6\x38" + "\x76\x1c\xf6\x3a\x67\xa3\x9c\xd8", + .klen = 16, + .iv = "\x03\xc6\xfb\x7d\x80\x0d\x13\xab" + "\xd8\xa6\xb2\xd8\x00\x00\x00\x00", + .assoc = "\xf3\x94\x87\x78\x35\x82\x81\x7f" + "\x88\x94\x68\xb1\x78\x6b\x2b\xd6" + "\x04\x1f\x4e\xed\x78\xd5\x33\x66" + "\xd8\x94\x99\x91\x81\x54\x62\x57", + .alen = 32, + .input = "\xf0\x7c\x29\x02\xae\x1c\x2f\x55" + "\xd0\xd1\x3d\x1a\xa3\x6d\xe4\x0a" + "\x86\xb0\x87\x6b\x62\x33\x8c\x34" + "\xce\xab\x57\xcc\x79\x0b\xe0\x6f" + "\x5c\x3e\x48\x1f\x6c\x46\xf7\x51" + "\x8b\x84\x83\x2a\xc1\x05\xb8\xc5", + .ilen = 48, + .result = "\x50\x82\x3e\x07\xe2\x1e\xb6\xfb" + "\x33\xe4\x73\xce\xd2\xfb\x95\x79" + "\xe8\xb4\xb5\x77\x11\x10\x62\x6f" + "\x6a\x82\xd1\x13\xec\xf5\xd0\x48", + .rlen = 32, + .novrfy = 1, + }, { + .key = "\x61\x0e\x8c\xae\xe3\x23\xb6\x38" + "\x76\x1c\xf6\x3a\x67\xa3\x9c\xd8", + .klen = 16, + .iv = "\x03\x05\xe0\xc9\x0f\xed\x34\xea" + "\x97\xd4\x3b\xdf\x00\x00\x00\x00", + .assoc = "\x49\x5c\x50\x1f\x1d\x94\xcc\x81" + "\xba\xb7\xb6\x03\xaf\xa5\xc1\xa1" + "\xd8\x5c\x42\x68\xe0\x6c\xda\x89" + "\x05\xac\x56\xac\x1b\x2a\xd3\x86", + .alen = 32, + .input = "\x39\xbe\x7d\x15\x62\x77\xf3\x3c" + "\xad\x83\x52\x6d\x71\x03\x25\x1c" + "\xed\x81\x3a\x9a\x16\x7d\x19\x80" + "\x72\x04\x72\xd0\xf6\xff\x05\x0f" + "\xb7\x14\x30\x00\x32\x9e\xa0\xa6" + "\x9e\x5a\x18\xa1\xb8\xfe\xdb\xd3", + .ilen = 48, + .result = "\x75\x05\xbe\xc2\xd9\x1e\xde\x60" + "\x47\x3d\x8c\x7d\xbd\xb5\xd9\xb7" + "\xf2\xae\x61\x05\x8f\x82\x24\x3f" + "\x9c\x67\x91\xe1\x38\x4f\xe4\x0c", + .rlen = 32, + }, { + .key = "\x39\xbb\xa7\xbe\x59\x97\x9e\x73" + "\xa2\xbc\x6b\x98\xd7\x75\x7f\xe3" + "\xa4\x48\x93\x39\x26\x71\x4a\xc6", + .klen = 24, + .iv = "\x03\xee\x49\x83\xe9\xa9\xff\xe9" + "\x57\xba\xfd\x9e\x00\x00\x00\x00", + .assoc = "\x44\xa6\x2c\x05\xe9\xe1\x43\xb1" + "\x58\x7c\xf2\x5c\x6d\x39\x0a\x64" + "\xa4\xf0\x13\x05\xd1\x77\x99\x67" + "\x11\xc4\xc6\xdb\x00\x56\x36\x61", + .alen = 32, + .input = "\x71\x99\xfa\xf4\x44\x12\x68\x9b", + .ilen = 8, + .result = "\x00", + .rlen = 0, + }, { + .key = "\x58\x5d\xa0\x96\x65\x1a\x04\xd7" + "\x96\xe5\xc5\x68\xaa\x95\x35\xe0" + "\x29\xa0\xba\x9e\x48\x78\xd1\xba", + .klen = 24, + .iv = "\x03\xee\x49\x83\xe9\xa9\xff\xe9" + "\x57\xba\xfd\x9e\x00\x00\x00\x00", + .assoc = "\x44\xa6\x2c\x05\xe9\xe1\x43\xb1" + "\x58\x7c\xf2\x5c\x6d\x39\x0a\x64" + "\xa4\xf0\x13\x05\xd1\x77\x99\x67" + "\x11\xc4\xc6\xdb\x00\x56\x36\x61", + .alen = 32, + .input = "\xfb\xe5\x5d\x34\xbe\xe5\xe8\xe7" + "\x5a\xef\x2f\xbf\x1f\x7f\xd4\xb2" + "\x66\xca\x61\x1e\x96\x7a\x61\xb3" + "\x1c\x16\x45\x52\xba\x04\x9c\x9f" + "\xb1\xd2\x40\xbc\x52\x7c\x6f\xb1", + .ilen = 40, + .result = "\x85\x34\x66\x42\xc8\x92\x0f\x36" + "\x58\xe0\x6b\x91\x3c\x98\x5c\xbb" + "\x0a\x85\xcc\x02\xad\x7a\x96\xe9" + "\x65\x43\xa4\xc3\x0f\xdc\x55\x81", + .rlen = 32, + }, { + .key = "\x58\x5d\xa0\x96\x65\x1a\x04\xd7" + "\x96\xe5\xc5\x68\xaa\x95\x35\xe0" + "\x29\xa0\xba\x9e\x48\x78\xd1\xba", + .klen = 24, + .iv = "\x03\xd1\xfc\x57\x9c\xfe\xb8\x9c" + "\xad\x71\xaa\x1f\x00\x00\x00\x00", + .assoc = "\x86\x67\xa5\xa9\x14\x5f\x0d\xc6" + "\xff\x14\xc7\x44\xbf\x6c\x3a\xc3" + "\xff\xb6\x81\xbd\xe2\xd5\x06\xc7" + "\x3c\xa1\x52\x13\x03\x8a\x23\x3a", + .alen = 32, + .input = "\x3f\x66\xb0\x9d\xe5\x4b\x38\x00" + "\xc6\x0e\x6e\xe5\xd6\x98\xa6\x37" + "\x8c\x26\x33\xc6\xb2\xa2\x17\xfa" + "\x64\x19\xc0\x30\xd7\xfc\x14\x6b" + "\xe3\x33\xc2\x04\xb0\x37\xbe\x3f" + "\xa9\xb4\x2d\x68\x03\xa3\x44\xef", + .ilen = 48, + .result = "\x02\x87\x4d\x28\x80\x6e\xb2\xed" + "\x99\x2a\xa8\xca\x04\x25\x45\x90" + "\x1d\xdd\x5a\xd9\xe4\xdb\x9c\x9c" + "\x49\xe9\x01\xfe\xa7\x80\x6d\x6b", + .rlen = 32, + .novrfy = 1, + }, { + .key = "\xa4\x4b\x54\x29\x0a\xb8\x6d\x01" + "\x5b\x80\x2a\xcf\x25\xc4\xb7\x5c" + "\x20\x2c\xad\x30\xc2\x2b\x41\xfb" + "\x0e\x85\xbc\x33\xad\x0f\x2b\xff", + .klen = 32, + .iv = "\x03\xee\x49\x83\xe9\xa9\xff\xe9" + "\x57\xba\xfd\x9e\x00\x00\x00\x00", + .alen = 0, + .input = "\x1f\xb8\x8f\xa3\xdd\x54\x00\xf2", + .ilen = 8, + .result = "\x00", + .rlen = 0, + }, { + .key = "\x39\xbb\xa7\xbe\x59\x97\x9e\x73" + "\xa2\xbc\x6b\x98\xd7\x75\x7f\xe3" + "\xa4\x48\x93\x39\x26\x71\x4a\xc6" + "\xae\x8f\x11\x4c\xc2\x9c\x4a\xbb", + .klen = 32, + .iv = "\x03\x85\x34\x66\x42\xc8\x92\x0f" + "\x36\x58\xe0\x6b\x00\x00\x00\x00", + .alen = 0, + .input = "\x48\x01\x5e\x02\x24\x04\x66\x47" + "\xa1\xea\x6f\xaf\xe8\xfc\xfb\xdd" + "\xa5\xa9\x87\x8d\x84\xee\x2e\x77" + "\xbb\x86\xb9\xf5\x5c\x6c\xff\xf6" + "\x72\xc3\x8e\xf7\x70\xb1\xb2\x07" + "\xbc\xa8\xa3\xbd\x83\x7c\x1d\x2a", + .ilen = 48, + .result = "\xdc\x56\xf2\x71\xb0\xb1\xa0\x6c" + "\xf0\x97\x3a\xfb\x6d\xe7\x32\x99" + "\x3e\xaf\x70\x5e\xb2\x4d\xea\x39" + "\x89\xd4\x75\x7a\x63\xb1\xda\x93", + .rlen = 32, + .novrfy = 1, + }, { + .key = "\x58\x5d\xa0\x96\x65\x1a\x04\xd7" + "\x96\xe5\xc5\x68\xaa\x95\x35\xe0" + "\x29\xa0\xba\x9e\x48\x78\xd1\xba" + "\x0d\x1a\x53\x3b\xb5\xe3\xf8\x8b", + .klen = 32, + .iv = "\x03\xcf\x76\x3f\xd9\x95\x75\x8f" + "\x44\x89\x40\x7b\x00\x00\x00\x00", + .assoc = "\x8f\x86\x6c\x4d\x1d\xc5\x39\x88" + "\xc8\xf3\x5c\x52\x10\x63\x6f\x2b" + "\x8a\x2a\xc5\x6f\x30\x23\x58\x7b" + "\xfb\x36\x03\x11\xb4\xd9\xf2\xfe", + .alen = 32, + .input = "\x48\x58\xd6\xf3\xad\x63\x58\xbf" + "\xae\xc7\x5e\xae\x83\x8f\x7b\xe4" + "\x78\x5c\x4c\x67\x71\x89\x94\xbf" + "\x47\xf1\x63\x7e\x1c\x59\xbd\xc5" + "\x7f\x44\x0a\x0c\x01\x18\x07\x92" + "\xe1\xd3\x51\xce\x32\x6d\x0c\x5b", + .ilen = 48, + .result = "\xc2\x54\xc8\xde\x78\x87\x77\x40" + "\x49\x71\xe4\xb7\xe7\xcb\x76\x61" + "\x0a\x41\xb9\xe9\xc0\x76\x54\xab" + "\x04\x49\x3b\x19\x93\x57\x25\x5d", + .rlen = 32, + }, +}; + +/* + * rfc4309 refers to section 8 of rfc3610 for test vectors, but they all + * use a 13-byte nonce, we only support an 11-byte nonce. Worse, + * they use AD lengths which are not valid ESP header lengths. + * + * These vectors are copied/generated from the ones for rfc4106 with + * the key truncated by one byte.. + */ +static const struct aead_testvec aes_ccm_rfc4309_enc_tv_template[] = { + { /* Generated using Crypto++ */ + .key = zeroed_string, + .klen = 19, + .iv = zeroed_string, + .input = zeroed_string, + .ilen = 16, + .assoc = zeroed_string, + .alen = 16, + .result = "\x2E\x9A\xCA\x6B\xDA\x54\xFC\x6F" + "\x12\x50\xE8\xDE\x81\x3C\x63\x08" + "\x1A\x22\xBA\x75\xEE\xD4\xD5\xB5" + "\x27\x50\x01\xAC\x03\x33\x39\xFB", + .rlen = 32, + },{ + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00", + .klen = 19, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .input = zeroed_string, + .ilen = 16, + .assoc = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .result = "\xCF\xB9\x99\x17\xC8\x86\x0E\x7F" + "\x7E\x76\xF8\xE6\xF8\xCC\x1F\x17" + "\x6A\xE0\x53\x9F\x4B\x73\x7E\xDA" + "\x08\x09\x4E\xC4\x1E\xAD\xC6\xB0", + .rlen = 32, + + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00", + .klen = 19, + .iv = zeroed_string, + .input = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .ilen = 16, + .assoc = zeroed_string, + .alen = 16, + .result = "\x33\xDE\x73\xBC\xA6\xCE\x4E\xA6" + "\x61\xF4\xF5\x41\x03\x4A\xE3\x86" + "\xA1\xE2\xC2\x42\x2B\x81\x70\x40" + "\xFD\x7F\x76\xD1\x03\x07\xBB\x0C", + .rlen = 32, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00", + .klen = 19, + .iv = zeroed_string, + .input = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .ilen = 16, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .alen = 16, + .result = "\x33\xDE\x73\xBC\xA6\xCE\x4E\xA6" + "\x61\xF4\xF5\x41\x03\x4A\xE3\x86" + "\x5B\xC0\x73\xE0\x2B\x73\x68\xC9" + "\x2D\x8C\x58\xC2\x90\x3D\xB0\x3E", + .rlen = 32, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00", + .klen = 19, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .input = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .ilen = 16, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .result = "\xCE\xB8\x98\x16\xC9\x87\x0F\x7E" + "\x7F\x77\xF9\xE7\xF9\xCD\x1E\x16" + "\x43\x8E\x76\x57\x3B\xB4\x05\xE8" + "\xA9\x9B\xBF\x25\xE0\x4F\xC0\xED", + .rlen = 32, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00", + .klen = 19, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .input = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .ilen = 64, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .result = "\xCE\xB8\x98\x16\xC9\x87\x0F\x7E" + "\x7F\x77\xF9\xE7\xF9\xCD\x1E\x16" + "\x9C\xA4\x97\x83\x3F\x01\xA5\xF4" + "\x43\x09\xE7\xB8\xE9\xD1\xD7\x02" + "\x9B\xAB\x39\x18\xEB\x94\x34\x36" + "\xE6\xC5\xC8\x9B\x00\x81\x9E\x49" + "\x1D\x78\xE1\x48\xE3\xE9\xEA\x8E" + "\x3A\x2B\x67\x5D\x35\x6A\x0F\xDB" + "\x02\x73\xDD\xE7\x30\x4A\x30\x54" + "\x1A\x9D\x09\xCA\xC8\x1C\x32\x5F", + .rlen = 80, + }, { + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x00\x00\x00", + .klen = 19, + .iv = "\x00\x00\x45\x67\x89\xab\xcd\xef", + .input = "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff", + .ilen = 192, + .assoc = "\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa" + "\xaa\xaa\xaa\xaa\x00\x00\x45\x67" + "\x89\xab\xcd\xef", + .alen = 20, + .result = "\x64\x17\xDC\x24\x9D\x92\xBA\x5E" + "\x7C\x64\x6D\x33\x46\x77\xAC\xB1" + "\x5C\x9E\xE2\xC7\x27\x11\x3E\x95" + "\x7D\xBE\x28\xC8\xC1\xCA\x5E\x8C" + "\xB4\xE2\xDE\x9F\x53\x59\x26\xDB" + "\x0C\xD4\xE4\x07\x9A\xE6\x3E\x01" + "\x58\x0D\x3E\x3D\xD5\x21\xEB\x04" + "\x06\x9D\x5F\xB9\x02\x49\x1A\x2B" + "\xBA\xF0\x4E\x3B\x85\x50\x5B\x09" + "\xFE\xEC\xFC\x54\xEC\x0C\xE2\x79" + "\x8A\x2F\x5F\xD7\x05\x5D\xF1\x6D" + "\x22\xEB\xD1\x09\x80\x3F\x5A\x70" + "\xB2\xB9\xD3\x63\x99\xC2\x4D\x1B" + "\x36\x12\x00\x89\xAA\x5D\x55\xDA" + "\x1D\x5B\xD8\x3C\x5F\x09\xD2\xE6" + "\x39\x41\x5C\xF0\xBE\x26\x4E\x5F" + "\x2B\x50\x44\x52\xC2\x10\x7D\x38" + "\x82\x64\x83\x0C\xAE\x49\xD0\xE5" + "\x4F\xE5\x66\x4C\x58\x7A\xEE\x43" + "\x3B\x51\xFE\xBA\x24\x8A\xFE\xDC" + "\x19\x6D\x60\x66\x61\xF9\x9A\x3F" + "\x75\xFC\x38\x53\x5B\xB5\xCD\x52" + "\x4F\xE5\xE4\xC9\xFE\x10\xCB\x98" + "\xF0\x06\x5B\x07\xAB\xBB\xF4\x0E" + "\x2D\xC2\xDD\x5D\xDD\x22\x9A\xCC" + "\x39\xAB\x63\xA5\x3D\x9C\x51\x8A", + .rlen = 208, + }, { /* From draft-mcgrew-gcm-test-01 */ + .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" + "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" + "\x2E\x44\x3B", + .klen = 19, + .iv = "\x49\x56\xED\x7E\x3B\x24\x4C\xFE", + .input = "\x45\x00\x00\x48\x69\x9A\x00\x00" + "\x80\x11\x4D\xB7\xC0\xA8\x01\x02" + "\xC0\xA8\x01\x01\x0A\x9B\xF1\x56" + "\x38\xD3\x01\x00\x00\x01\x00\x00" + "\x00\x00\x00\x00\x04\x5F\x73\x69" + "\x70\x04\x5F\x75\x64\x70\x03\x73" + "\x69\x70\x09\x63\x79\x62\x65\x72" + "\x63\x69\x74\x79\x02\x64\x6B\x00" + "\x00\x21\x00\x01\x01\x02\x02\x01", + .ilen = 72, + .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" + "\x00\x00\x00\x00\x49\x56\xED\x7E" + "\x3B\x24\x4C\xFE", + .alen = 20, + .result = "\x89\xBA\x3E\xEF\xE6\xD6\xCF\xDB" + "\x83\x60\xF5\xBA\x3A\x56\x79\xE6" + "\x7E\x0C\x53\xCF\x9E\x87\xE0\x4E" + "\x1A\x26\x01\x24\xC7\x2E\x3D\xBF" + "\x29\x2C\x91\xC1\xB8\xA8\xCF\xE0" + "\x39\xF8\x53\x6D\x31\x22\x2B\xBF" + "\x98\x81\xFC\x34\xEE\x85\x36\xCD" + "\x26\xDB\x6C\x7A\x0C\x77\x8A\x35" + "\x18\x85\x54\xB2\xBC\xDD\x3F\x43" + "\x61\x06\x8A\xDF\x86\x3F\xB4\xAC" + "\x97\xDC\xBD\xFD\x92\x10\xC5\xFF", + .rlen = 88, + }, { + .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\x6D\x6A\x8F\x94\x67\x30\x83\x08" + "\xCA\xFE\xBA", + .klen = 19, + .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .input = "\x45\x00\x00\x3E\x69\x8F\x00\x00" + "\x80\x11\x4D\xCC\xC0\xA8\x01\x02" + "\xC0\xA8\x01\x01\x0A\x98\x00\x35" + "\x00\x2A\x23\x43\xB2\xD0\x01\x00" + "\x00\x01\x00\x00\x00\x00\x00\x00" + "\x03\x73\x69\x70\x09\x63\x79\x62" + "\x65\x72\x63\x69\x74\x79\x02\x64" + "\x6B\x00\x00\x01\x00\x01\x00\x01", + .ilen = 64, + .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" + "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .alen = 16, + .result = "\x4B\xC2\x70\x60\x64\xD2\xF3\xC8" + "\xE5\x26\x8A\xDE\xB8\x7E\x7D\x16" + "\x56\xC7\xD2\x88\xBA\x8D\x58\xAF" + "\xF5\x71\xB6\x37\x84\xA7\xB1\x99" + "\x51\x5C\x0D\xA0\x27\xDE\xE7\x2D" + "\xEF\x25\x88\x1F\x1D\x77\x11\xFF" + "\xDB\xED\xEE\x56\x16\xC5\x5C\x9B" + "\x00\x62\x1F\x68\x4E\x7C\xA0\x97" + "\x10\x72\x7E\x53\x13\x3B\x68\xE4" + "\x30\x99\x91\x79\x09\xEA\xFF\x6A", + .rlen = 80, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x11\x22\x33", + .klen = 35, + .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", + .input = "\x45\x00\x00\x30\x69\xA6\x40\x00" + "\x80\x06\x26\x90\xC0\xA8\x01\x02" + "\x93\x89\x15\x5E\x0A\x9E\x00\x8B" + "\x2D\xC5\x7E\xE0\x00\x00\x00\x00" + "\x70\x02\x40\x00\x20\xBF\x00\x00" + "\x02\x04\x05\xB4\x01\x01\x04\x02" + "\x01\x02\x02\x01", + .ilen = 52, + .assoc = "\x4A\x2C\xBF\xE3\x00\x00\x00\x02" + "\x01\x02\x03\x04\x05\x06\x07\x08", + .alen = 16, + .result = "\xD6\x31\x0D\x2B\x3D\x6F\xBD\x2F" + "\x58\x41\x7E\xFF\x9A\x9E\x09\xB4" + "\x1A\xF7\xF6\x42\x31\xCD\xBF\xAD" + "\x27\x0E\x2C\xF2\xDB\x10\xDF\x55" + "\x8F\x0D\xD7\xAC\x23\xBD\x42\x10" + "\xD0\xB2\xAF\xD8\x37\xAC\x6B\x0B" + "\x11\xD4\x0B\x12\xEC\xB4\xB1\x92" + "\x23\xA6\x10\xB0\x26\xD6\xD9\x26" + "\x5A\x48\x6A\x3E", + .rlen = 68, + }, { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00", + .klen = 19, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", + .input = "\x45\x00\x00\x3C\x99\xC5\x00\x00" + "\x80\x01\xCB\x7A\x40\x67\x93\x18" + "\x01\x01\x01\x01\x08\x00\x07\x5C" + "\x02\x00\x44\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x75\x76\x77\x61\x62\x63\x64\x65" + "\x66\x67\x68\x69\x01\x02\x02\x01", + .ilen = 64, + .assoc = "\x00\x00\x00\x00\x00\x00\x00\x01" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .alen = 16, + .result = "\x6B\x9A\xCA\x57\x43\x91\xFC\x6F" + "\x92\x51\x23\xA4\xC1\x5B\xF0\x10" + "\xF3\x13\xF4\xF8\xA1\x9A\xB4\xDC" + "\x89\xC8\xF8\x42\x62\x95\xB7\xCB" + "\xB8\xF5\x0F\x1B\x2E\x94\xA2\xA7" + "\xBF\xFB\x8A\x92\x13\x63\xD1\x3C" + "\x08\xF5\xE8\xA6\xAA\xF6\x34\xF9" + "\x42\x05\xAF\xB3\xE7\x9A\xFC\xEE" + "\x36\x25\xC1\x10\x12\x1C\xCA\x82" + "\xEA\xE6\x63\x5A\x57\x28\xA9\x9A", + .rlen = 80, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E", + .klen = 19, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .input = "\x45\x00\x00\x3C\x99\xC3\x00\x00" + "\x80\x01\xCB\x7C\x40\x67\x93\x18" + "\x01\x01\x01\x01\x08\x00\x08\x5C" + "\x02\x00\x43\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x75\x76\x77\x61\x62\x63\x64\x65" + "\x66\x67\x68\x69\x01\x02\x02\x01", + .ilen = 64, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .result = "\x6A\x6B\x45\x2B\x7C\x67\x52\xF6" + "\x10\x60\x40\x62\x6B\x4F\x97\x8E" + "\x0B\xB2\x22\x97\xCB\x21\xE0\x90" + "\xA2\xE7\xD1\x41\x30\xE4\x4B\x1B" + "\x79\x01\x58\x50\x01\x06\xE1\xE0" + "\x2C\x83\x79\xD3\xDE\x46\x97\x1A" + "\x30\xB8\xE5\xDF\xD7\x12\x56\x75" + "\xD0\x95\xB7\xB8\x91\x42\xF7\xFD" + "\x97\x57\xCA\xC1\x20\xD0\x86\xB9" + "\x66\x9D\xB4\x2B\x96\x22\xAC\x67", + .rlen = 80, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E", + .klen = 19, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .input = "\x45\x00\x00\x1C\x42\xA2\x00\x00" + "\x80\x01\x44\x1F\x40\x67\x93\xB6" + "\xE0\x00\x00\x02\x0A\x00\xF5\xFF" + "\x01\x02\x02\x01", + .ilen = 28, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .result = "\x6A\x6B\x45\x0B\xA7\x06\x52\xF6" + "\x10\x60\xCF\x01\x6B\x4F\x97\x20" + "\xEA\xB3\x23\x94\xC9\x21\x1D\x33" + "\xA1\xE5\x90\x40\x05\x37\x45\x70" + "\xB5\xD6\x09\x0A\x23\x73\x33\xF9" + "\x08\xB4\x22\xE4", + .rlen = 44, + }, { + .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\x6D\x6A\x8F\x94\x67\x30\x83\x08" + "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\xCA\xFE\xBA", + .klen = 27, + .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .input = "\x45\x00\x00\x28\xA4\xAD\x40\x00" + "\x40\x06\x78\x80\x0A\x01\x03\x8F" + "\x0A\x01\x06\x12\x80\x23\x06\xB8" + "\xCB\x71\x26\x02\xDD\x6B\xB0\x3E" + "\x50\x10\x16\xD0\x75\x68\x00\x01", + .ilen = 40, + .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" + "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .alen = 16, + .result = "\x05\x22\x15\xD1\x52\x56\x85\x04" + "\xA8\x5C\x5D\x6D\x7E\x6E\xF5\xFA" + "\xEA\x16\x37\x50\xF3\xDF\x84\x3B" + "\x2F\x32\x18\x57\x34\x2A\x8C\x23" + "\x67\xDF\x6D\x35\x7B\x54\x0D\xFB" + "\x34\xA5\x9F\x6C\x48\x30\x1E\x22" + "\xFE\xB1\x22\x17\x17\x8A\xB9\x5B", + .rlen = 56, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xDE\xCA\xF8", + .klen = 19, + .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", + .input = "\x45\x00\x00\x49\x33\xBA\x00\x00" + "\x7F\x11\x91\x06\xC3\xFB\x1D\x10" + "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" + "\x00\x35\xDD\x7B\x80\x03\x02\xD5" + "\x00\x00\x4E\x20\x00\x1E\x8C\x18" + "\xD7\x5B\x81\xDC\x91\xBA\xA0\x47" + "\x6B\x91\xB9\x24\xB2\x80\x38\x9D" + "\x92\xC9\x63\xBA\xC0\x46\xEC\x95" + "\x9B\x62\x66\xC0\x47\x22\xB1\x49" + "\x23\x01\x01\x01", + .ilen = 76, + .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" + "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" + "\xCE\xFA\xCE\x74", + .alen = 20, + .result = "\x92\xD0\x53\x79\x33\x38\xD5\xF3" + "\x7D\xE4\x7A\x8E\x86\x03\xC9\x90" + "\x96\x35\xAB\x9C\xFB\xE8\xA3\x76" + "\xE9\xE9\xE2\xD1\x2E\x11\x0E\x00" + "\xFA\xCE\xB5\x9E\x02\xA7\x7B\xEA" + "\x71\x9A\x58\xFB\xA5\x8A\xE1\xB7" + "\x9C\x39\x9D\xE3\xB5\x6E\x69\xE6" + "\x63\xC9\xDB\x05\x69\x51\x12\xAD" + "\x3E\x00\x32\x73\x86\xF2\xEE\xF5" + "\x0F\xE8\x81\x7E\x84\xD3\xC0\x0D" + "\x76\xD6\x55\xC6\xB4\xC2\x34\xC7" + "\x12\x25\x0B\xF9", + .rlen = 92, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x73\x61\x6C", + .klen = 35, + .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", + .input = "\x45\x08\x00\x28\x73\x2C\x00\x00" + "\x40\x06\xE9\xF9\x0A\x01\x06\x12" + "\x0A\x01\x03\x8F\x06\xB8\x80\x23" + "\xDD\x6B\xAF\xBE\xCB\x71\x26\x02" + "\x50\x10\x1F\x64\x6D\x54\x00\x01", + .ilen = 40, + .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" + "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" + "\x69\x76\x65\x63", + .alen = 20, + .result = "\xCC\x74\xB7\xD3\xB0\x38\x50\x42" + "\x2C\x64\x87\x46\x1E\x34\x10\x05" + "\x29\x6B\xBB\x36\xE9\x69\xAD\x92" + "\x82\xA1\x10\x6A\xEB\x0F\xDC\x7D" + "\x08\xBA\xF3\x91\xCA\xAA\x61\xDA" + "\x62\xF4\x14\x61\x5C\x9D\xB5\xA7" + "\xEE\xD7\xB9\x7E\x87\x99\x9B\x7D", + .rlen = 56, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E", + .klen = 19, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .input = "\x45\x00\x00\x49\x33\x3E\x00\x00" + "\x7F\x11\x91\x82\xC3\xFB\x1D\x10" + "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" + "\x00\x35\xCB\x45\x80\x03\x02\x5B" + "\x00\x00\x01\xE0\x00\x1E\x8C\x18" + "\xD6\x57\x59\xD5\x22\x84\xA0\x35" + "\x2C\x71\x47\x5C\x88\x80\x39\x1C" + "\x76\x4D\x6E\x5E\xE0\x49\x6B\x32" + "\x5A\xE2\x70\xC0\x38\x99\x49\x39" + "\x15\x01\x01\x01", + .ilen = 76, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .result = "\x6A\x6B\x45\x5E\xD6\x9A\x52\xF6" + "\xEF\x70\x1A\x9C\xE8\xD3\x19\x86" + "\xC8\x02\xF0\xB0\x03\x09\xD9\x02" + "\xA0\xD2\x59\x04\xD1\x85\x2A\x24" + "\x1C\x67\x3E\xD8\x68\x72\x06\x94" + "\x97\xBA\x4F\x76\x8D\xB0\x44\x5B" + "\x69\xBF\xD5\xE2\x3D\xF1\x0B\x0C" + "\xC0\xBF\xB1\x8F\x70\x09\x9E\xCE" + "\xA5\xF2\x55\x58\x84\xFA\xF9\xB5" + "\x23\xF4\x84\x40\x74\x14\x8A\x6B" + "\xDB\xD7\x67\xED\xA4\x93\xF3\x47" + "\xCC\xF7\x46\x6F", + .rlen = 92, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x73\x61\x6C", + .klen = 35, + .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", + .input = "\x63\x69\x73\x63\x6F\x01\x72\x75" + "\x6C\x65\x73\x01\x74\x68\x65\x01" + "\x6E\x65\x74\x77\x65\x01\x64\x65" + "\x66\x69\x6E\x65\x01\x74\x68\x65" + "\x74\x65\x63\x68\x6E\x6F\x6C\x6F" + "\x67\x69\x65\x73\x01\x74\x68\x61" + "\x74\x77\x69\x6C\x6C\x01\x64\x65" + "\x66\x69\x6E\x65\x74\x6F\x6D\x6F" + "\x72\x72\x6F\x77\x01\x02\x02\x01", + .ilen = 72, + .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" + "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" + "\x69\x76\x65\x63", + .alen = 20, + .result = "\xEA\x15\xC4\x98\xAC\x15\x22\x37" + "\x00\x07\x1D\xBE\x60\x5D\x73\x16" + "\x4D\x0F\xCC\xCE\x8A\xD0\x49\xD4" + "\x39\xA3\xD1\xB1\x21\x0A\x92\x1A" + "\x2C\xCF\x8F\x9D\xC9\x91\x0D\xB4" + "\x15\xFC\xBC\xA5\xC5\xBF\x54\xE5" + "\x1C\xC7\x32\x41\x07\x7B\x2C\xB6" + "\x5C\x23\x7C\x93\xEA\xEF\x23\x1C" + "\x73\xF4\xE7\x12\x84\x4C\x37\x0A" + "\x4A\x8F\x06\x37\x48\xF9\xF9\x05" + "\x55\x13\x40\xC3\xD5\x55\x3A\x3D", + .rlen = 88, + }, { + .key = "\x7D\x77\x3D\x00\xC1\x44\xC5\x25" + "\xAC\x61\x9D\x18\xC8\x4A\x3F\x47" + "\xD9\x66\x42", + .klen = 19, + .iv = "\x43\x45\x7E\x91\x82\x44\x3B\xC6", + .input = "\x01\x02\x02\x01", + .ilen = 4, + .assoc = "\x33\x54\x67\xAE\xFF\xFF\xFF\xFF" + "\x43\x45\x7E\x91\x82\x44\x3B\xC6", + .alen = 16, + .result = "\x4C\x72\x63\x30\x2F\xE6\x56\xDD" + "\xD0\xD8\x60\x9D\x8B\xEF\x85\x90" + "\xF7\x61\x24\x62", + .rlen = 20, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xDE\xCA\xF8", + .klen = 19, + .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", + .input = "\x74\x6F\x01\x62\x65\x01\x6F\x72" + "\x01\x6E\x6F\x74\x01\x74\x6F\x01" + "\x62\x65\x00\x01", + .ilen = 20, + .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" + "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" + "\xCE\xFA\xCE\x74", + .alen = 20, + .result = "\xA3\xBF\x52\x52\x65\x83\xBA\x81" + "\x03\x9B\x84\xFC\x44\x8C\xBB\x81" + "\x36\xE1\x78\xBB\xA5\x49\x3A\xD0" + "\xF0\x6B\x21\xAF\x98\xC0\x34\xDC" + "\x17\x17\x65\xAD", + .rlen = 36, + }, { + .key = "\x6C\x65\x67\x61\x6C\x69\x7A\x65" + "\x6D\x61\x72\x69\x6A\x75\x61\x6E" + "\x61\x61\x6E\x64\x64\x6F\x69\x74" + "\x62\x65\x66\x6F\x72\x65\x69\x61" + "\x74\x75\x72", + .klen = 35, + .iv = "\x33\x30\x21\x69\x67\x65\x74\x6D", + .input = "\x45\x00\x00\x30\xDA\x3A\x00\x00" + "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" + "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01", + .ilen = 52, + .assoc = "\x79\x6B\x69\x63\xFF\xFF\xFF\xFF" + "\xFF\xFF\xFF\xFF\x33\x30\x21\x69" + "\x67\x65\x74\x6D", + .alen = 20, + .result = "\x96\xFD\x86\xF8\xD1\x98\xFF\x10" + "\xAB\x8C\xDA\x8A\x5A\x08\x38\x1A" + "\x48\x59\x80\x18\x1A\x18\x1A\x04" + "\xC9\x0D\xE3\xE7\x0E\xA4\x0B\x75" + "\x92\x9C\x52\x5C\x0B\xFB\xF8\xAF" + "\x16\xC3\x35\xA8\xE7\xCE\x84\x04" + "\xEB\x40\x6B\x7A\x8E\x75\xBB\x42" + "\xE0\x63\x4B\x21\x44\xA2\x2B\x2B" + "\x39\xDB\xC8\xDC", + .rlen = 68, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E", + .klen = 19, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .input = "\x45\x00\x00\x30\xDA\x3A\x00\x00" + "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" + "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01", + .ilen = 52, + .assoc = "\x3F\x7E\xF6\x42\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .result = "\x6A\x6B\x45\x27\x3F\x9E\x52\xF6" + "\x10\x60\x54\x25\xEB\x80\x04\x93" + "\xCA\x1B\x23\x97\xCB\x21\x2E\x01" + "\xA2\xE7\x95\x41\x30\xE4\x4B\x1B" + "\x79\x01\x58\x50\x01\x06\xE1\xE0" + "\x2C\x83\x79\xD3\xDE\x46\x97\x1A" + "\x44\xCC\x90\xBF\x00\x94\x94\x92" + "\x20\x17\x0C\x1B\x55\xDE\x7E\x68" + "\xF4\x95\x5D\x4F", + .rlen = 68, + }, { + .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" + "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" + "\x22\x43\x3C", + .klen = 19, + .iv = "\x48\x55\xEC\x7D\x3A\x23\x4B\xFD", + .input = "\x08\x00\xC6\xCD\x02\x00\x07\x00" + "\x61\x62\x63\x64\x65\x66\x67\x68" + "\x69\x6A\x6B\x6C\x6D\x6E\x6F\x70" + "\x71\x72\x73\x74\x01\x02\x02\x01", + .ilen = 32, + .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" + "\x00\x00\x00\x07\x48\x55\xEC\x7D" + "\x3A\x23\x4B\xFD", + .alen = 20, + .result = "\x67\xE9\x28\xB3\x1C\xA4\x6D\x02" + "\xF0\xB5\x37\xB6\x6B\x2F\xF5\x4F" + "\xF8\xA3\x4C\x53\xB8\x12\x09\xBF" + "\x58\x7D\xCF\x29\xA3\x41\x68\x6B" + "\xCE\xE8\x79\x85\x3C\xB0\x3A\x8F" + "\x16\xB0\xA1\x26\xC9\xBC\xBC\xA6", + .rlen = 48, + } +}; + +static const struct aead_testvec aes_ccm_rfc4309_dec_tv_template[] = { + { /* Generated using Crypto++ */ + .key = zeroed_string, + .klen = 19, + .iv = zeroed_string, + .result = zeroed_string, + .rlen = 16, + .assoc = zeroed_string, + .alen = 16, + .input = "\x2E\x9A\xCA\x6B\xDA\x54\xFC\x6F" + "\x12\x50\xE8\xDE\x81\x3C\x63\x08" + "\x1A\x22\xBA\x75\xEE\xD4\xD5\xB5" + "\x27\x50\x01\xAC\x03\x33\x39\xFB", + .ilen = 32, + },{ + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00", + .klen = 19, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .result = zeroed_string, + .rlen = 16, + .assoc = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .input = "\xCF\xB9\x99\x17\xC8\x86\x0E\x7F" + "\x7E\x76\xF8\xE6\xF8\xCC\x1F\x17" + "\x6A\xE0\x53\x9F\x4B\x73\x7E\xDA" + "\x08\x09\x4E\xC4\x1E\xAD\xC6\xB0", + .ilen = 32, + + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00", + .klen = 19, + .iv = zeroed_string, + .result = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .rlen = 16, + .assoc = zeroed_string, + .alen = 16, + .input = "\x33\xDE\x73\xBC\xA6\xCE\x4E\xA6" + "\x61\xF4\xF5\x41\x03\x4A\xE3\x86" + "\xA1\xE2\xC2\x42\x2B\x81\x70\x40" + "\xFD\x7F\x76\xD1\x03\x07\xBB\x0C", + .ilen = 32, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00", + .klen = 19, + .iv = zeroed_string, + .result = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .rlen = 16, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .alen = 16, + .input = "\x33\xDE\x73\xBC\xA6\xCE\x4E\xA6" + "\x61\xF4\xF5\x41\x03\x4A\xE3\x86" + "\x5B\xC0\x73\xE0\x2B\x73\x68\xC9" + "\x2D\x8C\x58\xC2\x90\x3D\xB0\x3E", + .ilen = 32, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00", + .klen = 19, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .result = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .rlen = 16, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .input = "\xCE\xB8\x98\x16\xC9\x87\x0F\x7E" + "\x7F\x77\xF9\xE7\xF9\xCD\x1E\x16" + "\x43\x8E\x76\x57\x3B\xB4\x05\xE8" + "\xA9\x9B\xBF\x25\xE0\x4F\xC0\xED", + .ilen = 32, + }, { + .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c" + "\x6d\x6a\x8f\x94\x67\x30\x83\x08" + "\x00\x00\x00", + .klen = 19, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x01", + .result = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x01\x01\x01\x01\x01\x01\x01\x01", + .rlen = 64, + .assoc = "\x01\x01\x01\x01\x01\x01\x01\x01" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .alen = 16, + .input = "\xCE\xB8\x98\x16\xC9\x87\x0F\x7E" + "\x7F\x77\xF9\xE7\xF9\xCD\x1E\x16" + "\x9C\xA4\x97\x83\x3F\x01\xA5\xF4" + "\x43\x09\xE7\xB8\xE9\xD1\xD7\x02" + "\x9B\xAB\x39\x18\xEB\x94\x34\x36" + "\xE6\xC5\xC8\x9B\x00\x81\x9E\x49" + "\x1D\x78\xE1\x48\xE3\xE9\xEA\x8E" + "\x3A\x2B\x67\x5D\x35\x6A\x0F\xDB" + "\x02\x73\xDD\xE7\x30\x4A\x30\x54" + "\x1A\x9D\x09\xCA\xC8\x1C\x32\x5F", + .ilen = 80, + }, { + .key = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x00\x00\x00", + .klen = 19, + .iv = "\x00\x00\x45\x67\x89\xab\xcd\xef", + .result = "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff" + "\xff\xff\xff\xff\xff\xff\xff\xff", + .rlen = 192, + .assoc = "\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa" + "\xaa\xaa\xaa\xaa\x00\x00\x45\x67" + "\x89\xab\xcd\xef", + .alen = 20, + .input = "\x64\x17\xDC\x24\x9D\x92\xBA\x5E" + "\x7C\x64\x6D\x33\x46\x77\xAC\xB1" + "\x5C\x9E\xE2\xC7\x27\x11\x3E\x95" + "\x7D\xBE\x28\xC8\xC1\xCA\x5E\x8C" + "\xB4\xE2\xDE\x9F\x53\x59\x26\xDB" + "\x0C\xD4\xE4\x07\x9A\xE6\x3E\x01" + "\x58\x0D\x3E\x3D\xD5\x21\xEB\x04" + "\x06\x9D\x5F\xB9\x02\x49\x1A\x2B" + "\xBA\xF0\x4E\x3B\x85\x50\x5B\x09" + "\xFE\xEC\xFC\x54\xEC\x0C\xE2\x79" + "\x8A\x2F\x5F\xD7\x05\x5D\xF1\x6D" + "\x22\xEB\xD1\x09\x80\x3F\x5A\x70" + "\xB2\xB9\xD3\x63\x99\xC2\x4D\x1B" + "\x36\x12\x00\x89\xAA\x5D\x55\xDA" + "\x1D\x5B\xD8\x3C\x5F\x09\xD2\xE6" + "\x39\x41\x5C\xF0\xBE\x26\x4E\x5F" + "\x2B\x50\x44\x52\xC2\x10\x7D\x38" + "\x82\x64\x83\x0C\xAE\x49\xD0\xE5" + "\x4F\xE5\x66\x4C\x58\x7A\xEE\x43" + "\x3B\x51\xFE\xBA\x24\x8A\xFE\xDC" + "\x19\x6D\x60\x66\x61\xF9\x9A\x3F" + "\x75\xFC\x38\x53\x5B\xB5\xCD\x52" + "\x4F\xE5\xE4\xC9\xFE\x10\xCB\x98" + "\xF0\x06\x5B\x07\xAB\xBB\xF4\x0E" + "\x2D\xC2\xDD\x5D\xDD\x22\x9A\xCC" + "\x39\xAB\x63\xA5\x3D\x9C\x51\x8A", + .ilen = 208, + }, { /* From draft-mcgrew-gcm-test-01 */ + .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" + "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" + "\x2E\x44\x3B", + .klen = 19, + .iv = "\x49\x56\xED\x7E\x3B\x24\x4C\xFE", + .result = "\x45\x00\x00\x48\x69\x9A\x00\x00" + "\x80\x11\x4D\xB7\xC0\xA8\x01\x02" + "\xC0\xA8\x01\x01\x0A\x9B\xF1\x56" + "\x38\xD3\x01\x00\x00\x01\x00\x00" + "\x00\x00\x00\x00\x04\x5F\x73\x69" + "\x70\x04\x5F\x75\x64\x70\x03\x73" + "\x69\x70\x09\x63\x79\x62\x65\x72" + "\x63\x69\x74\x79\x02\x64\x6B\x00" + "\x00\x21\x00\x01\x01\x02\x02\x01", + .rlen = 72, + .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" + "\x00\x00\x00\x00\x49\x56\xED\x7E" + "\x3B\x24\x4C\xFE", + .alen = 20, + .input = "\x89\xBA\x3E\xEF\xE6\xD6\xCF\xDB" + "\x83\x60\xF5\xBA\x3A\x56\x79\xE6" + "\x7E\x0C\x53\xCF\x9E\x87\xE0\x4E" + "\x1A\x26\x01\x24\xC7\x2E\x3D\xBF" + "\x29\x2C\x91\xC1\xB8\xA8\xCF\xE0" + "\x39\xF8\x53\x6D\x31\x22\x2B\xBF" + "\x98\x81\xFC\x34\xEE\x85\x36\xCD" + "\x26\xDB\x6C\x7A\x0C\x77\x8A\x35" + "\x18\x85\x54\xB2\xBC\xDD\x3F\x43" + "\x61\x06\x8A\xDF\x86\x3F\xB4\xAC" + "\x97\xDC\xBD\xFD\x92\x10\xC5\xFF", + .ilen = 88, + }, { + .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\x6D\x6A\x8F\x94\x67\x30\x83\x08" + "\xCA\xFE\xBA", + .klen = 19, + .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .result = "\x45\x00\x00\x3E\x69\x8F\x00\x00" + "\x80\x11\x4D\xCC\xC0\xA8\x01\x02" + "\xC0\xA8\x01\x01\x0A\x98\x00\x35" + "\x00\x2A\x23\x43\xB2\xD0\x01\x00" + "\x00\x01\x00\x00\x00\x00\x00\x00" + "\x03\x73\x69\x70\x09\x63\x79\x62" + "\x65\x72\x63\x69\x74\x79\x02\x64" + "\x6B\x00\x00\x01\x00\x01\x00\x01", + .rlen = 64, + .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" + "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .alen = 16, + .input = "\x4B\xC2\x70\x60\x64\xD2\xF3\xC8" + "\xE5\x26\x8A\xDE\xB8\x7E\x7D\x16" + "\x56\xC7\xD2\x88\xBA\x8D\x58\xAF" + "\xF5\x71\xB6\x37\x84\xA7\xB1\x99" + "\x51\x5C\x0D\xA0\x27\xDE\xE7\x2D" + "\xEF\x25\x88\x1F\x1D\x77\x11\xFF" + "\xDB\xED\xEE\x56\x16\xC5\x5C\x9B" + "\x00\x62\x1F\x68\x4E\x7C\xA0\x97" + "\x10\x72\x7E\x53\x13\x3B\x68\xE4" + "\x30\x99\x91\x79\x09\xEA\xFF\x6A", + .ilen = 80, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x11\x22\x33", + .klen = 35, + .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", + .result = "\x45\x00\x00\x30\x69\xA6\x40\x00" + "\x80\x06\x26\x90\xC0\xA8\x01\x02" + "\x93\x89\x15\x5E\x0A\x9E\x00\x8B" + "\x2D\xC5\x7E\xE0\x00\x00\x00\x00" + "\x70\x02\x40\x00\x20\xBF\x00\x00" + "\x02\x04\x05\xB4\x01\x01\x04\x02" + "\x01\x02\x02\x01", + .rlen = 52, + .assoc = "\x4A\x2C\xBF\xE3\x00\x00\x00\x02" + "\x01\x02\x03\x04\x05\x06\x07\x08", + .alen = 16, + .input = "\xD6\x31\x0D\x2B\x3D\x6F\xBD\x2F" + "\x58\x41\x7E\xFF\x9A\x9E\x09\xB4" + "\x1A\xF7\xF6\x42\x31\xCD\xBF\xAD" + "\x27\x0E\x2C\xF2\xDB\x10\xDF\x55" + "\x8F\x0D\xD7\xAC\x23\xBD\x42\x10" + "\xD0\xB2\xAF\xD8\x37\xAC\x6B\x0B" + "\x11\xD4\x0B\x12\xEC\xB4\xB1\x92" + "\x23\xA6\x10\xB0\x26\xD6\xD9\x26" + "\x5A\x48\x6A\x3E", + .ilen = 68, + }, { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00", + .klen = 19, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", + .result = "\x45\x00\x00\x3C\x99\xC5\x00\x00" + "\x80\x01\xCB\x7A\x40\x67\x93\x18" + "\x01\x01\x01\x01\x08\x00\x07\x5C" + "\x02\x00\x44\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x75\x76\x77\x61\x62\x63\x64\x65" + "\x66\x67\x68\x69\x01\x02\x02\x01", + .rlen = 64, + .assoc = "\x00\x00\x00\x00\x00\x00\x00\x01" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .alen = 16, + .input = "\x6B\x9A\xCA\x57\x43\x91\xFC\x6F" + "\x92\x51\x23\xA4\xC1\x5B\xF0\x10" + "\xF3\x13\xF4\xF8\xA1\x9A\xB4\xDC" + "\x89\xC8\xF8\x42\x62\x95\xB7\xCB" + "\xB8\xF5\x0F\x1B\x2E\x94\xA2\xA7" + "\xBF\xFB\x8A\x92\x13\x63\xD1\x3C" + "\x08\xF5\xE8\xA6\xAA\xF6\x34\xF9" + "\x42\x05\xAF\xB3\xE7\x9A\xFC\xEE" + "\x36\x25\xC1\x10\x12\x1C\xCA\x82" + "\xEA\xE6\x63\x5A\x57\x28\xA9\x9A", + .ilen = 80, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E", + .klen = 19, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .result = "\x45\x00\x00\x3C\x99\xC3\x00\x00" + "\x80\x01\xCB\x7C\x40\x67\x93\x18" + "\x01\x01\x01\x01\x08\x00\x08\x5C" + "\x02\x00\x43\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x75\x76\x77\x61\x62\x63\x64\x65" + "\x66\x67\x68\x69\x01\x02\x02\x01", + .rlen = 64, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .input = "\x6A\x6B\x45\x2B\x7C\x67\x52\xF6" + "\x10\x60\x40\x62\x6B\x4F\x97\x8E" + "\x0B\xB2\x22\x97\xCB\x21\xE0\x90" + "\xA2\xE7\xD1\x41\x30\xE4\x4B\x1B" + "\x79\x01\x58\x50\x01\x06\xE1\xE0" + "\x2C\x83\x79\xD3\xDE\x46\x97\x1A" + "\x30\xB8\xE5\xDF\xD7\x12\x56\x75" + "\xD0\x95\xB7\xB8\x91\x42\xF7\xFD" + "\x97\x57\xCA\xC1\x20\xD0\x86\xB9" + "\x66\x9D\xB4\x2B\x96\x22\xAC\x67", + .ilen = 80, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E", + .klen = 19, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .result = "\x45\x00\x00\x1C\x42\xA2\x00\x00" + "\x80\x01\x44\x1F\x40\x67\x93\xB6" + "\xE0\x00\x00\x02\x0A\x00\xF5\xFF" + "\x01\x02\x02\x01", + .rlen = 28, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .input = "\x6A\x6B\x45\x0B\xA7\x06\x52\xF6" + "\x10\x60\xCF\x01\x6B\x4F\x97\x20" + "\xEA\xB3\x23\x94\xC9\x21\x1D\x33" + "\xA1\xE5\x90\x40\x05\x37\x45\x70" + "\xB5\xD6\x09\x0A\x23\x73\x33\xF9" + "\x08\xB4\x22\xE4", + .ilen = 44, + }, { + .key = "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\x6D\x6A\x8F\x94\x67\x30\x83\x08" + "\xFE\xFF\xE9\x92\x86\x65\x73\x1C" + "\xCA\xFE\xBA", + .klen = 27, + .iv = "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .result = "\x45\x00\x00\x28\xA4\xAD\x40\x00" + "\x40\x06\x78\x80\x0A\x01\x03\x8F" + "\x0A\x01\x06\x12\x80\x23\x06\xB8" + "\xCB\x71\x26\x02\xDD\x6B\xB0\x3E" + "\x50\x10\x16\xD0\x75\x68\x00\x01", + .rlen = 40, + .assoc = "\x00\x00\xA5\xF8\x00\x00\x00\x0A" + "\xFA\xCE\xDB\xAD\xDE\xCA\xF8\x88", + .alen = 16, + .input = "\x05\x22\x15\xD1\x52\x56\x85\x04" + "\xA8\x5C\x5D\x6D\x7E\x6E\xF5\xFA" + "\xEA\x16\x37\x50\xF3\xDF\x84\x3B" + "\x2F\x32\x18\x57\x34\x2A\x8C\x23" + "\x67\xDF\x6D\x35\x7B\x54\x0D\xFB" + "\x34\xA5\x9F\x6C\x48\x30\x1E\x22" + "\xFE\xB1\x22\x17\x17\x8A\xB9\x5B", + .ilen = 56, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xDE\xCA\xF8", + .klen = 19, + .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", + .result = "\x45\x00\x00\x49\x33\xBA\x00\x00" + "\x7F\x11\x91\x06\xC3\xFB\x1D\x10" + "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" + "\x00\x35\xDD\x7B\x80\x03\x02\xD5" + "\x00\x00\x4E\x20\x00\x1E\x8C\x18" + "\xD7\x5B\x81\xDC\x91\xBA\xA0\x47" + "\x6B\x91\xB9\x24\xB2\x80\x38\x9D" + "\x92\xC9\x63\xBA\xC0\x46\xEC\x95" + "\x9B\x62\x66\xC0\x47\x22\xB1\x49" + "\x23\x01\x01\x01", + .rlen = 76, + .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" + "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" + "\xCE\xFA\xCE\x74", + .alen = 20, + .input = "\x92\xD0\x53\x79\x33\x38\xD5\xF3" + "\x7D\xE4\x7A\x8E\x86\x03\xC9\x90" + "\x96\x35\xAB\x9C\xFB\xE8\xA3\x76" + "\xE9\xE9\xE2\xD1\x2E\x11\x0E\x00" + "\xFA\xCE\xB5\x9E\x02\xA7\x7B\xEA" + "\x71\x9A\x58\xFB\xA5\x8A\xE1\xB7" + "\x9C\x39\x9D\xE3\xB5\x6E\x69\xE6" + "\x63\xC9\xDB\x05\x69\x51\x12\xAD" + "\x3E\x00\x32\x73\x86\xF2\xEE\xF5" + "\x0F\xE8\x81\x7E\x84\xD3\xC0\x0D" + "\x76\xD6\x55\xC6\xB4\xC2\x34\xC7" + "\x12\x25\x0B\xF9", + .ilen = 92, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x73\x61\x6C", + .klen = 35, + .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", + .result = "\x45\x08\x00\x28\x73\x2C\x00\x00" + "\x40\x06\xE9\xF9\x0A\x01\x06\x12" + "\x0A\x01\x03\x8F\x06\xB8\x80\x23" + "\xDD\x6B\xAF\xBE\xCB\x71\x26\x02" + "\x50\x10\x1F\x64\x6D\x54\x00\x01", + .rlen = 40, + .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" + "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" + "\x69\x76\x65\x63", + .alen = 20, + .input = "\xCC\x74\xB7\xD3\xB0\x38\x50\x42" + "\x2C\x64\x87\x46\x1E\x34\x10\x05" + "\x29\x6B\xBB\x36\xE9\x69\xAD\x92" + "\x82\xA1\x10\x6A\xEB\x0F\xDC\x7D" + "\x08\xBA\xF3\x91\xCA\xAA\x61\xDA" + "\x62\xF4\x14\x61\x5C\x9D\xB5\xA7" + "\xEE\xD7\xB9\x7E\x87\x99\x9B\x7D", + .ilen = 56, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E", + .klen = 19, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .result = "\x45\x00\x00\x49\x33\x3E\x00\x00" + "\x7F\x11\x91\x82\xC3\xFB\x1D\x10" + "\xC2\xB1\xD3\x26\xC0\x28\x31\xCE" + "\x00\x35\xCB\x45\x80\x03\x02\x5B" + "\x00\x00\x01\xE0\x00\x1E\x8C\x18" + "\xD6\x57\x59\xD5\x22\x84\xA0\x35" + "\x2C\x71\x47\x5C\x88\x80\x39\x1C" + "\x76\x4D\x6E\x5E\xE0\x49\x6B\x32" + "\x5A\xE2\x70\xC0\x38\x99\x49\x39" + "\x15\x01\x01\x01", + .rlen = 76, + .assoc = "\x42\xF6\x7E\x3F\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .input = "\x6A\x6B\x45\x5E\xD6\x9A\x52\xF6" + "\xEF\x70\x1A\x9C\xE8\xD3\x19\x86" + "\xC8\x02\xF0\xB0\x03\x09\xD9\x02" + "\xA0\xD2\x59\x04\xD1\x85\x2A\x24" + "\x1C\x67\x3E\xD8\x68\x72\x06\x94" + "\x97\xBA\x4F\x76\x8D\xB0\x44\x5B" + "\x69\xBF\xD5\xE2\x3D\xF1\x0B\x0C" + "\xC0\xBF\xB1\x8F\x70\x09\x9E\xCE" + "\xA5\xF2\x55\x58\x84\xFA\xF9\xB5" + "\x23\xF4\x84\x40\x74\x14\x8A\x6B" + "\xDB\xD7\x67\xED\xA4\x93\xF3\x47" + "\xCC\xF7\x46\x6F", + .ilen = 92, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\x73\x61\x6C", + .klen = 35, + .iv = "\x61\x6E\x64\x01\x69\x76\x65\x63", + .result = "\x63\x69\x73\x63\x6F\x01\x72\x75" + "\x6C\x65\x73\x01\x74\x68\x65\x01" + "\x6E\x65\x74\x77\x65\x01\x64\x65" + "\x66\x69\x6E\x65\x01\x74\x68\x65" + "\x74\x65\x63\x68\x6E\x6F\x6C\x6F" + "\x67\x69\x65\x73\x01\x74\x68\x61" + "\x74\x77\x69\x6C\x6C\x01\x64\x65" + "\x66\x69\x6E\x65\x74\x6F\x6D\x6F" + "\x72\x72\x6F\x77\x01\x02\x02\x01", + .rlen = 72, + .assoc = "\x17\x40\x5E\x67\x15\x6F\x31\x26" + "\xDD\x0D\xB9\x9B\x61\x6E\x64\x01" + "\x69\x76\x65\x63", + .alen = 20, + .input = "\xEA\x15\xC4\x98\xAC\x15\x22\x37" + "\x00\x07\x1D\xBE\x60\x5D\x73\x16" + "\x4D\x0F\xCC\xCE\x8A\xD0\x49\xD4" + "\x39\xA3\xD1\xB1\x21\x0A\x92\x1A" + "\x2C\xCF\x8F\x9D\xC9\x91\x0D\xB4" + "\x15\xFC\xBC\xA5\xC5\xBF\x54\xE5" + "\x1C\xC7\x32\x41\x07\x7B\x2C\xB6" + "\x5C\x23\x7C\x93\xEA\xEF\x23\x1C" + "\x73\xF4\xE7\x12\x84\x4C\x37\x0A" + "\x4A\x8F\x06\x37\x48\xF9\xF9\x05" + "\x55\x13\x40\xC3\xD5\x55\x3A\x3D", + .ilen = 88, + }, { + .key = "\x7D\x77\x3D\x00\xC1\x44\xC5\x25" + "\xAC\x61\x9D\x18\xC8\x4A\x3F\x47" + "\xD9\x66\x42", + .klen = 19, + .iv = "\x43\x45\x7E\x91\x82\x44\x3B\xC6", + .result = "\x01\x02\x02\x01", + .rlen = 4, + .assoc = "\x33\x54\x67\xAE\xFF\xFF\xFF\xFF" + "\x43\x45\x7E\x91\x82\x44\x3B\xC6", + .alen = 16, + .input = "\x4C\x72\x63\x30\x2F\xE6\x56\xDD" + "\xD0\xD8\x60\x9D\x8B\xEF\x85\x90" + "\xF7\x61\x24\x62", + .ilen = 20, + }, { + .key = "\xAB\xBC\xCD\xDE\xF0\x01\x12\x23" + "\x34\x45\x56\x67\x78\x89\x9A\xAB" + "\xDE\xCA\xF8", + .klen = 19, + .iv = "\xCA\xFE\xDE\xBA\xCE\xFA\xCE\x74", + .result = "\x74\x6F\x01\x62\x65\x01\x6F\x72" + "\x01\x6E\x6F\x74\x01\x74\x6F\x01" + "\x62\x65\x00\x01", + .rlen = 20, + .assoc = "\x00\x00\x01\x00\x00\x00\x00\x00" + "\x00\x00\x00\x01\xCA\xFE\xDE\xBA" + "\xCE\xFA\xCE\x74", + .alen = 20, + .input = "\xA3\xBF\x52\x52\x65\x83\xBA\x81" + "\x03\x9B\x84\xFC\x44\x8C\xBB\x81" + "\x36\xE1\x78\xBB\xA5\x49\x3A\xD0" + "\xF0\x6B\x21\xAF\x98\xC0\x34\xDC" + "\x17\x17\x65\xAD", + .ilen = 36, + }, { + .key = "\x6C\x65\x67\x61\x6C\x69\x7A\x65" + "\x6D\x61\x72\x69\x6A\x75\x61\x6E" + "\x61\x61\x6E\x64\x64\x6F\x69\x74" + "\x62\x65\x66\x6F\x72\x65\x69\x61" + "\x74\x75\x72", + .klen = 35, + .iv = "\x33\x30\x21\x69\x67\x65\x74\x6D", + .result = "\x45\x00\x00\x30\xDA\x3A\x00\x00" + "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" + "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01", + .rlen = 52, + .assoc = "\x79\x6B\x69\x63\xFF\xFF\xFF\xFF" + "\xFF\xFF\xFF\xFF\x33\x30\x21\x69" + "\x67\x65\x74\x6D", + .alen = 20, + .input = "\x96\xFD\x86\xF8\xD1\x98\xFF\x10" + "\xAB\x8C\xDA\x8A\x5A\x08\x38\x1A" + "\x48\x59\x80\x18\x1A\x18\x1A\x04" + "\xC9\x0D\xE3\xE7\x0E\xA4\x0B\x75" + "\x92\x9C\x52\x5C\x0B\xFB\xF8\xAF" + "\x16\xC3\x35\xA8\xE7\xCE\x84\x04" + "\xEB\x40\x6B\x7A\x8E\x75\xBB\x42" + "\xE0\x63\x4B\x21\x44\xA2\x2B\x2B" + "\x39\xDB\xC8\xDC", + .ilen = 68, + }, { + .key = "\x3D\xE0\x98\x74\xB3\x88\xE6\x49" + "\x19\x88\xD0\xC3\x60\x7E\xAE\x1F" + "\x57\x69\x0E", + .klen = 19, + .iv = "\x4E\x28\x00\x00\xA2\xFC\xA1\xA3", + .result = "\x45\x00\x00\x30\xDA\x3A\x00\x00" + "\x80\x01\xDF\x3B\xC0\xA8\x00\x05" + "\xC0\xA8\x00\x01\x08\x00\xC6\xCD" + "\x02\x00\x07\x00\x61\x62\x63\x64" + "\x65\x66\x67\x68\x69\x6A\x6B\x6C" + "\x6D\x6E\x6F\x70\x71\x72\x73\x74" + "\x01\x02\x02\x01", + .rlen = 52, + .assoc = "\x3F\x7E\xF6\x42\x10\x10\x10\x10" + "\x10\x10\x10\x10\x4E\x28\x00\x00" + "\xA2\xFC\xA1\xA3", + .alen = 20, + .input = "\x6A\x6B\x45\x27\x3F\x9E\x52\xF6" + "\x10\x60\x54\x25\xEB\x80\x04\x93" + "\xCA\x1B\x23\x97\xCB\x21\x2E\x01" + "\xA2\xE7\x95\x41\x30\xE4\x4B\x1B" + "\x79\x01\x58\x50\x01\x06\xE1\xE0" + "\x2C\x83\x79\xD3\xDE\x46\x97\x1A" + "\x44\xCC\x90\xBF\x00\x94\x94\x92" + "\x20\x17\x0C\x1B\x55\xDE\x7E\x68" + "\xF4\x95\x5D\x4F", + .ilen = 68, + }, { + .key = "\x4C\x80\xCD\xEF\xBB\x5D\x10\xDA" + "\x90\x6A\xC7\x3C\x36\x13\xA6\x34" + "\x22\x43\x3C", + .klen = 19, + .iv = "\x48\x55\xEC\x7D\x3A\x23\x4B\xFD", + .result = "\x08\x00\xC6\xCD\x02\x00\x07\x00" + "\x61\x62\x63\x64\x65\x66\x67\x68" + "\x69\x6A\x6B\x6C\x6D\x6E\x6F\x70" + "\x71\x72\x73\x74\x01\x02\x02\x01", + .rlen = 32, + .assoc = "\x00\x00\x43\x21\x87\x65\x43\x21" + "\x00\x00\x00\x07\x48\x55\xEC\x7D" + "\x3A\x23\x4B\xFD", + .alen = 20, + .input = "\x67\xE9\x28\xB3\x1C\xA4\x6D\x02" + "\xF0\xB5\x37\xB6\x6B\x2F\xF5\x4F" + "\xF8\xA3\x4C\x53\xB8\x12\x09\xBF" + "\x58\x7D\xCF\x29\xA3\x41\x68\x6B" + "\xCE\xE8\x79\x85\x3C\xB0\x3A\x8F" + "\x16\xB0\xA1\x26\xC9\xBC\xBC\xA6", + .ilen = 48, + } +}; + +/* + * ChaCha20-Poly1305 AEAD test vectors from RFC7539 2.8.2./A.5. + */ +static const struct aead_testvec rfc7539_enc_tv_template[] = { + { + .key = "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f", + .klen = 32, + .iv = "\x07\x00\x00\x00\x40\x41\x42\x43" + "\x44\x45\x46\x47", + .assoc = "\x50\x51\x52\x53\xc0\xc1\xc2\xc3" + "\xc4\xc5\xc6\xc7", + .alen = 12, + .input = "\x4c\x61\x64\x69\x65\x73\x20\x61" + "\x6e\x64\x20\x47\x65\x6e\x74\x6c" + "\x65\x6d\x65\x6e\x20\x6f\x66\x20" + "\x74\x68\x65\x20\x63\x6c\x61\x73" + "\x73\x20\x6f\x66\x20\x27\x39\x39" + "\x3a\x20\x49\x66\x20\x49\x20\x63" + "\x6f\x75\x6c\x64\x20\x6f\x66\x66" + "\x65\x72\x20\x79\x6f\x75\x20\x6f" + "\x6e\x6c\x79\x20\x6f\x6e\x65\x20" + "\x74\x69\x70\x20\x66\x6f\x72\x20" + "\x74\x68\x65\x20\x66\x75\x74\x75" + "\x72\x65\x2c\x20\x73\x75\x6e\x73" + "\x63\x72\x65\x65\x6e\x20\x77\x6f" + "\x75\x6c\x64\x20\x62\x65\x20\x69" + "\x74\x2e", + .ilen = 114, + .result = "\xd3\x1a\x8d\x34\x64\x8e\x60\xdb" + "\x7b\x86\xaf\xbc\x53\xef\x7e\xc2" + "\xa4\xad\xed\x51\x29\x6e\x08\xfe" + "\xa9\xe2\xb5\xa7\x36\xee\x62\xd6" + "\x3d\xbe\xa4\x5e\x8c\xa9\x67\x12" + "\x82\xfa\xfb\x69\xda\x92\x72\x8b" + "\x1a\x71\xde\x0a\x9e\x06\x0b\x29" + "\x05\xd6\xa5\xb6\x7e\xcd\x3b\x36" + "\x92\xdd\xbd\x7f\x2d\x77\x8b\x8c" + "\x98\x03\xae\xe3\x28\x09\x1b\x58" + "\xfa\xb3\x24\xe4\xfa\xd6\x75\x94" + "\x55\x85\x80\x8b\x48\x31\xd7\xbc" + "\x3f\xf4\xde\xf0\x8e\x4b\x7a\x9d" + "\xe5\x76\xd2\x65\x86\xce\xc6\x4b" + "\x61\x16\x1a\xe1\x0b\x59\x4f\x09" + "\xe2\x6a\x7e\x90\x2e\xcb\xd0\x60" + "\x06\x91", + .rlen = 130, + }, { + .key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a" + "\xf3\x33\x88\x86\x04\xf6\xb5\xf0" + "\x47\x39\x17\xc1\x40\x2b\x80\x09" + "\x9d\xca\x5c\xbc\x20\x70\x75\xc0", + .klen = 32, + .iv = "\x00\x00\x00\x00\x01\x02\x03\x04" + "\x05\x06\x07\x08", + .assoc = "\xf3\x33\x88\x86\x00\x00\x00\x00" + "\x00\x00\x4e\x91", + .alen = 12, + .input = "\x49\x6e\x74\x65\x72\x6e\x65\x74" + "\x2d\x44\x72\x61\x66\x74\x73\x20" + "\x61\x72\x65\x20\x64\x72\x61\x66" + "\x74\x20\x64\x6f\x63\x75\x6d\x65" + "\x6e\x74\x73\x20\x76\x61\x6c\x69" + "\x64\x20\x66\x6f\x72\x20\x61\x20" + "\x6d\x61\x78\x69\x6d\x75\x6d\x20" + "\x6f\x66\x20\x73\x69\x78\x20\x6d" + "\x6f\x6e\x74\x68\x73\x20\x61\x6e" + "\x64\x20\x6d\x61\x79\x20\x62\x65" + "\x20\x75\x70\x64\x61\x74\x65\x64" + "\x2c\x20\x72\x65\x70\x6c\x61\x63" + "\x65\x64\x2c\x20\x6f\x72\x20\x6f" + "\x62\x73\x6f\x6c\x65\x74\x65\x64" + "\x20\x62\x79\x20\x6f\x74\x68\x65" + "\x72\x20\x64\x6f\x63\x75\x6d\x65" + "\x6e\x74\x73\x20\x61\x74\x20\x61" + "\x6e\x79\x20\x74\x69\x6d\x65\x2e" + "\x20\x49\x74\x20\x69\x73\x20\x69" + "\x6e\x61\x70\x70\x72\x6f\x70\x72" + "\x69\x61\x74\x65\x20\x74\x6f\x20" + "\x75\x73\x65\x20\x49\x6e\x74\x65" + "\x72\x6e\x65\x74\x2d\x44\x72\x61" + "\x66\x74\x73\x20\x61\x73\x20\x72" + "\x65\x66\x65\x72\x65\x6e\x63\x65" + "\x20\x6d\x61\x74\x65\x72\x69\x61" + "\x6c\x20\x6f\x72\x20\x74\x6f\x20" + "\x63\x69\x74\x65\x20\x74\x68\x65" + "\x6d\x20\x6f\x74\x68\x65\x72\x20" + "\x74\x68\x61\x6e\x20\x61\x73\x20" + "\x2f\xe2\x80\x9c\x77\x6f\x72\x6b" + "\x20\x69\x6e\x20\x70\x72\x6f\x67" + "\x72\x65\x73\x73\x2e\x2f\xe2\x80" + "\x9d", + .ilen = 265, + .result = "\x64\xa0\x86\x15\x75\x86\x1a\xf4" + "\x60\xf0\x62\xc7\x9b\xe6\x43\xbd" + "\x5e\x80\x5c\xfd\x34\x5c\xf3\x89" + "\xf1\x08\x67\x0a\xc7\x6c\x8c\xb2" + "\x4c\x6c\xfc\x18\x75\x5d\x43\xee" + "\xa0\x9e\xe9\x4e\x38\x2d\x26\xb0" + "\xbd\xb7\xb7\x3c\x32\x1b\x01\x00" + "\xd4\xf0\x3b\x7f\x35\x58\x94\xcf" + "\x33\x2f\x83\x0e\x71\x0b\x97\xce" + "\x98\xc8\xa8\x4a\xbd\x0b\x94\x81" + "\x14\xad\x17\x6e\x00\x8d\x33\xbd" + "\x60\xf9\x82\xb1\xff\x37\xc8\x55" + "\x97\x97\xa0\x6e\xf4\xf0\xef\x61" + "\xc1\x86\x32\x4e\x2b\x35\x06\x38" + "\x36\x06\x90\x7b\x6a\x7c\x02\xb0" + "\xf9\xf6\x15\x7b\x53\xc8\x67\xe4" + "\xb9\x16\x6c\x76\x7b\x80\x4d\x46" + "\xa5\x9b\x52\x16\xcd\xe7\xa4\xe9" + "\x90\x40\xc5\xa4\x04\x33\x22\x5e" + "\xe2\x82\xa1\xb0\xa0\x6c\x52\x3e" + "\xaf\x45\x34\xd7\xf8\x3f\xa1\x15" + "\x5b\x00\x47\x71\x8c\xbc\x54\x6a" + "\x0d\x07\x2b\x04\xb3\x56\x4e\xea" + "\x1b\x42\x22\x73\xf5\x48\x27\x1a" + "\x0b\xb2\x31\x60\x53\xfa\x76\x99" + "\x19\x55\xeb\xd6\x31\x59\x43\x4e" + "\xce\xbb\x4e\x46\x6d\xae\x5a\x10" + "\x73\xa6\x72\x76\x27\x09\x7a\x10" + "\x49\xe6\x17\xd9\x1d\x36\x10\x94" + "\xfa\x68\xf0\xff\x77\x98\x71\x30" + "\x30\x5b\xea\xba\x2e\xda\x04\xdf" + "\x99\x7b\x71\x4d\x6c\x6f\x2c\x29" + "\xa6\xad\x5c\xb4\x02\x2b\x02\x70" + "\x9b\xee\xad\x9d\x67\x89\x0c\xbb" + "\x22\x39\x23\x36\xfe\xa1\x85\x1f" + "\x38", + .rlen = 281, + }, +}; + +static const struct aead_testvec rfc7539_dec_tv_template[] = { + { + .key = "\x80\x81\x82\x83\x84\x85\x86\x87" + "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" + "\x90\x91\x92\x93\x94\x95\x96\x97" + "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f", + .klen = 32, + .iv = "\x07\x00\x00\x00\x40\x41\x42\x43" + "\x44\x45\x46\x47", + .assoc = "\x50\x51\x52\x53\xc0\xc1\xc2\xc3" + "\xc4\xc5\xc6\xc7", + .alen = 12, + .input = "\xd3\x1a\x8d\x34\x64\x8e\x60\xdb" + "\x7b\x86\xaf\xbc\x53\xef\x7e\xc2" + "\xa4\xad\xed\x51\x29\x6e\x08\xfe" + "\xa9\xe2\xb5\xa7\x36\xee\x62\xd6" + "\x3d\xbe\xa4\x5e\x8c\xa9\x67\x12" + "\x82\xfa\xfb\x69\xda\x92\x72\x8b" + "\x1a\x71\xde\x0a\x9e\x06\x0b\x29" + "\x05\xd6\xa5\xb6\x7e\xcd\x3b\x36" + "\x92\xdd\xbd\x7f\x2d\x77\x8b\x8c" + "\x98\x03\xae\xe3\x28\x09\x1b\x58" + "\xfa\xb3\x24\xe4\xfa\xd6\x75\x94" + "\x55\x85\x80\x8b\x48\x31\xd7\xbc" + "\x3f\xf4\xde\xf0\x8e\x4b\x7a\x9d" + "\xe5\x76\xd2\x65\x86\xce\xc6\x4b" + "\x61\x16\x1a\xe1\x0b\x59\x4f\x09" + "\xe2\x6a\x7e\x90\x2e\xcb\xd0\x60" + "\x06\x91", + .ilen = 130, + .result = "\x4c\x61\x64\x69\x65\x73\x20\x61" + "\x6e\x64\x20\x47\x65\x6e\x74\x6c" + "\x65\x6d\x65\x6e\x20\x6f\x66\x20" + "\x74\x68\x65\x20\x63\x6c\x61\x73" + "\x73\x20\x6f\x66\x20\x27\x39\x39" + "\x3a\x20\x49\x66\x20\x49\x20\x63" + "\x6f\x75\x6c\x64\x20\x6f\x66\x66" + "\x65\x72\x20\x79\x6f\x75\x20\x6f" + "\x6e\x6c\x79\x20\x6f\x6e\x65\x20" + "\x74\x69\x70\x20\x66\x6f\x72\x20" + "\x74\x68\x65\x20\x66\x75\x74\x75" + "\x72\x65\x2c\x20\x73\x75\x6e\x73" + "\x63\x72\x65\x65\x6e\x20\x77\x6f" + "\x75\x6c\x64\x20\x62\x65\x20\x69" + "\x74\x2e", + .rlen = 114, + }, { + .key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a" + "\xf3\x33\x88\x86\x04\xf6\xb5\xf0" + "\x47\x39\x17\xc1\x40\x2b\x80\x09" + "\x9d\xca\x5c\xbc\x20\x70\x75\xc0", + .klen = 32, + .iv = "\x00\x00\x00\x00\x01\x02\x03\x04" + "\x05\x06\x07\x08", + .assoc = "\xf3\x33\x88\x86\x00\x00\x00\x00" + "\x00\x00\x4e\x91", + .alen = 12, + .input = "\x64\xa0\x86\x15\x75\x86\x1a\xf4" + "\x60\xf0\x62\xc7\x9b\xe6\x43\xbd" + "\x5e\x80\x5c\xfd\x34\x5c\xf3\x89" + "\xf1\x08\x67\x0a\xc7\x6c\x8c\xb2" + "\x4c\x6c\xfc\x18\x75\x5d\x43\xee" + "\xa0\x9e\xe9\x4e\x38\x2d\x26\xb0" + "\xbd\xb7\xb7\x3c\x32\x1b\x01\x00" + "\xd4\xf0\x3b\x7f\x35\x58\x94\xcf" + "\x33\x2f\x83\x0e\x71\x0b\x97\xce" + "\x98\xc8\xa8\x4a\xbd\x0b\x94\x81" + "\x14\xad\x17\x6e\x00\x8d\x33\xbd" + "\x60\xf9\x82\xb1\xff\x37\xc8\x55" + "\x97\x97\xa0\x6e\xf4\xf0\xef\x61" + "\xc1\x86\x32\x4e\x2b\x35\x06\x38" + "\x36\x06\x90\x7b\x6a\x7c\x02\xb0" + "\xf9\xf6\x15\x7b\x53\xc8\x67\xe4" + "\xb9\x16\x6c\x76\x7b\x80\x4d\x46" + "\xa5\x9b\x52\x16\xcd\xe7\xa4\xe9" + "\x90\x40\xc5\xa4\x04\x33\x22\x5e" + "\xe2\x82\xa1\xb0\xa0\x6c\x52\x3e" + "\xaf\x45\x34\xd7\xf8\x3f\xa1\x15" + "\x5b\x00\x47\x71\x8c\xbc\x54\x6a" + "\x0d\x07\x2b\x04\xb3\x56\x4e\xea" + "\x1b\x42\x22\x73\xf5\x48\x27\x1a" + "\x0b\xb2\x31\x60\x53\xfa\x76\x99" + "\x19\x55\xeb\xd6\x31\x59\x43\x4e" + "\xce\xbb\x4e\x46\x6d\xae\x5a\x10" + "\x73\xa6\x72\x76\x27\x09\x7a\x10" + "\x49\xe6\x17\xd9\x1d\x36\x10\x94" + "\xfa\x68\xf0\xff\x77\x98\x71\x30" + "\x30\x5b\xea\xba\x2e\xda\x04\xdf" + "\x99\x7b\x71\x4d\x6c\x6f\x2c\x29" + "\xa6\xad\x5c\xb4\x02\x2b\x02\x70" + "\x9b\xee\xad\x9d\x67\x89\x0c\xbb" + "\x22\x39\x23\x36\xfe\xa1\x85\x1f" + "\x38", + .ilen = 281, + .result = "\x49\x6e\x74\x65\x72\x6e\x65\x74" + "\x2d\x44\x72\x61\x66\x74\x73\x20" + "\x61\x72\x65\x20\x64\x72\x61\x66" + "\x74\x20\x64\x6f\x63\x75\x6d\x65" + "\x6e\x74\x73\x20\x76\x61\x6c\x69" + "\x64\x20\x66\x6f\x72\x20\x61\x20" + "\x6d\x61\x78\x69\x6d\x75\x6d\x20" + "\x6f\x66\x20\x73\x69\x78\x20\x6d" + "\x6f\x6e\x74\x68\x73\x20\x61\x6e" + "\x64\x20\x6d\x61\x79\x20\x62\x65" + "\x20\x75\x70\x64\x61\x74\x65\x64" + "\x2c\x20\x72\x65\x70\x6c\x61\x63" + "\x65\x64\x2c\x20\x6f\x72\x20\x6f" + "\x62\x73\x6f\x6c\x65\x74\x65\x64" + "\x20\x62\x79\x20\x6f\x74\x68\x65" + "\x72\x20\x64\x6f\x63\x75\x6d\x65" + "\x6e\x74\x73\x20\x61\x74\x20\x61" + "\x6e\x79\x20\x74\x69\x6d\x65\x2e" + "\x20\x49\x74\x20\x69\x73\x20\x69" + "\x6e\x61\x70\x70\x72\x6f\x70\x72" + "\x69\x61\x74\x65\x20\x74\x6f\x20" + "\x75\x73\x65\x20\x49\x6e\x74\x65" + "\x72\x6e\x65\x74\x2d\x44\x72\x61" + "\x66\x74\x73\x20\x61\x73\x20\x72" + "\x65\x66\x65\x72\x65\x6e\x63\x65" + "\x20\x6d\x61\x74\x65\x72\x69\x61" + "\x6c\x20\x6f\x72\x20\x74\x6f\x20" + "\x63\x69\x74\x65\x20\x74\x68\x65" + "\x6d\x20\x6f\x74\x68\x65\x72\x20" + "\x74\x68\x61\x6e\x20\x61\x73\x20" + "\x2f\xe2\x80\x9c\x77\x6f\x72\x6b" + "\x20\x69\x6e\x20\x70\x72\x6f\x67" + "\x72\x65\x73\x73\x2e\x2f\xe2\x80" + "\x9d", + .rlen = 265, + }, +}; + +/* + * draft-irtf-cfrg-chacha20-poly1305 + */ +static const struct aead_testvec rfc7539esp_enc_tv_template[] = { + { + .key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a" + "\xf3\x33\x88\x86\x04\xf6\xb5\xf0" + "\x47\x39\x17\xc1\x40\x2b\x80\x09" + "\x9d\xca\x5c\xbc\x20\x70\x75\xc0" + "\x00\x00\x00\x00", + .klen = 36, + .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", + .assoc = "\xf3\x33\x88\x86\x00\x00\x00\x00" + "\x00\x00\x4e\x91\x01\x02\x03\x04" + "\x05\x06\x07\x08", + .alen = 20, + .input = "\x49\x6e\x74\x65\x72\x6e\x65\x74" + "\x2d\x44\x72\x61\x66\x74\x73\x20" + "\x61\x72\x65\x20\x64\x72\x61\x66" + "\x74\x20\x64\x6f\x63\x75\x6d\x65" + "\x6e\x74\x73\x20\x76\x61\x6c\x69" + "\x64\x20\x66\x6f\x72\x20\x61\x20" + "\x6d\x61\x78\x69\x6d\x75\x6d\x20" + "\x6f\x66\x20\x73\x69\x78\x20\x6d" + "\x6f\x6e\x74\x68\x73\x20\x61\x6e" + "\x64\x20\x6d\x61\x79\x20\x62\x65" + "\x20\x75\x70\x64\x61\x74\x65\x64" + "\x2c\x20\x72\x65\x70\x6c\x61\x63" + "\x65\x64\x2c\x20\x6f\x72\x20\x6f" + "\x62\x73\x6f\x6c\x65\x74\x65\x64" + "\x20\x62\x79\x20\x6f\x74\x68\x65" + "\x72\x20\x64\x6f\x63\x75\x6d\x65" + "\x6e\x74\x73\x20\x61\x74\x20\x61" + "\x6e\x79\x20\x74\x69\x6d\x65\x2e" + "\x20\x49\x74\x20\x69\x73\x20\x69" + "\x6e\x61\x70\x70\x72\x6f\x70\x72" + "\x69\x61\x74\x65\x20\x74\x6f\x20" + "\x75\x73\x65\x20\x49\x6e\x74\x65" + "\x72\x6e\x65\x74\x2d\x44\x72\x61" + "\x66\x74\x73\x20\x61\x73\x20\x72" + "\x65\x66\x65\x72\x65\x6e\x63\x65" + "\x20\x6d\x61\x74\x65\x72\x69\x61" + "\x6c\x20\x6f\x72\x20\x74\x6f\x20" + "\x63\x69\x74\x65\x20\x74\x68\x65" + "\x6d\x20\x6f\x74\x68\x65\x72\x20" + "\x74\x68\x61\x6e\x20\x61\x73\x20" + "\x2f\xe2\x80\x9c\x77\x6f\x72\x6b" + "\x20\x69\x6e\x20\x70\x72\x6f\x67" + "\x72\x65\x73\x73\x2e\x2f\xe2\x80" + "\x9d", + .ilen = 265, + .result = "\x64\xa0\x86\x15\x75\x86\x1a\xf4" + "\x60\xf0\x62\xc7\x9b\xe6\x43\xbd" + "\x5e\x80\x5c\xfd\x34\x5c\xf3\x89" + "\xf1\x08\x67\x0a\xc7\x6c\x8c\xb2" + "\x4c\x6c\xfc\x18\x75\x5d\x43\xee" + "\xa0\x9e\xe9\x4e\x38\x2d\x26\xb0" + "\xbd\xb7\xb7\x3c\x32\x1b\x01\x00" + "\xd4\xf0\x3b\x7f\x35\x58\x94\xcf" + "\x33\x2f\x83\x0e\x71\x0b\x97\xce" + "\x98\xc8\xa8\x4a\xbd\x0b\x94\x81" + "\x14\xad\x17\x6e\x00\x8d\x33\xbd" + "\x60\xf9\x82\xb1\xff\x37\xc8\x55" + "\x97\x97\xa0\x6e\xf4\xf0\xef\x61" + "\xc1\x86\x32\x4e\x2b\x35\x06\x38" + "\x36\x06\x90\x7b\x6a\x7c\x02\xb0" + "\xf9\xf6\x15\x7b\x53\xc8\x67\xe4" + "\xb9\x16\x6c\x76\x7b\x80\x4d\x46" + "\xa5\x9b\x52\x16\xcd\xe7\xa4\xe9" + "\x90\x40\xc5\xa4\x04\x33\x22\x5e" + "\xe2\x82\xa1\xb0\xa0\x6c\x52\x3e" + "\xaf\x45\x34\xd7\xf8\x3f\xa1\x15" + "\x5b\x00\x47\x71\x8c\xbc\x54\x6a" + "\x0d\x07\x2b\x04\xb3\x56\x4e\xea" + "\x1b\x42\x22\x73\xf5\x48\x27\x1a" + "\x0b\xb2\x31\x60\x53\xfa\x76\x99" + "\x19\x55\xeb\xd6\x31\x59\x43\x4e" + "\xce\xbb\x4e\x46\x6d\xae\x5a\x10" + "\x73\xa6\x72\x76\x27\x09\x7a\x10" + "\x49\xe6\x17\xd9\x1d\x36\x10\x94" + "\xfa\x68\xf0\xff\x77\x98\x71\x30" + "\x30\x5b\xea\xba\x2e\xda\x04\xdf" + "\x99\x7b\x71\x4d\x6c\x6f\x2c\x29" + "\xa6\xad\x5c\xb4\x02\x2b\x02\x70" + "\x9b\xee\xad\x9d\x67\x89\x0c\xbb" + "\x22\x39\x23\x36\xfe\xa1\x85\x1f" + "\x38", + .rlen = 281, + }, +}; + +static const struct aead_testvec rfc7539esp_dec_tv_template[] = { + { + .key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a" + "\xf3\x33\x88\x86\x04\xf6\xb5\xf0" + "\x47\x39\x17\xc1\x40\x2b\x80\x09" + "\x9d\xca\x5c\xbc\x20\x70\x75\xc0" + "\x00\x00\x00\x00", + .klen = 36, + .iv = "\x01\x02\x03\x04\x05\x06\x07\x08", + .assoc = "\xf3\x33\x88\x86\x00\x00\x00\x00" + "\x00\x00\x4e\x91\x01\x02\x03\x04" + "\x05\x06\x07\x08", + .alen = 20, + .input = "\x64\xa0\x86\x15\x75\x86\x1a\xf4" + "\x60\xf0\x62\xc7\x9b\xe6\x43\xbd" + "\x5e\x80\x5c\xfd\x34\x5c\xf3\x89" + "\xf1\x08\x67\x0a\xc7\x6c\x8c\xb2" + "\x4c\x6c\xfc\x18\x75\x5d\x43\xee" + "\xa0\x9e\xe9\x4e\x38\x2d\x26\xb0" + "\xbd\xb7\xb7\x3c\x32\x1b\x01\x00" + "\xd4\xf0\x3b\x7f\x35\x58\x94\xcf" + "\x33\x2f\x83\x0e\x71\x0b\x97\xce" + "\x98\xc8\xa8\x4a\xbd\x0b\x94\x81" + "\x14\xad\x17\x6e\x00\x8d\x33\xbd" + "\x60\xf9\x82\xb1\xff\x37\xc8\x55" + "\x97\x97\xa0\x6e\xf4\xf0\xef\x61" + "\xc1\x86\x32\x4e\x2b\x35\x06\x38" + "\x36\x06\x90\x7b\x6a\x7c\x02\xb0" + "\xf9\xf6\x15\x7b\x53\xc8\x67\xe4" + "\xb9\x16\x6c\x76\x7b\x80\x4d\x46" + "\xa5\x9b\x52\x16\xcd\xe7\xa4\xe9" + "\x90\x40\xc5\xa4\x04\x33\x22\x5e" + "\xe2\x82\xa1\xb0\xa0\x6c\x52\x3e" + "\xaf\x45\x34\xd7\xf8\x3f\xa1\x15" + "\x5b\x00\x47\x71\x8c\xbc\x54\x6a" + "\x0d\x07\x2b\x04\xb3\x56\x4e\xea" + "\x1b\x42\x22\x73\xf5\x48\x27\x1a" + "\x0b\xb2\x31\x60\x53\xfa\x76\x99" + "\x19\x55\xeb\xd6\x31\x59\x43\x4e" + "\xce\xbb\x4e\x46\x6d\xae\x5a\x10" + "\x73\xa6\x72\x76\x27\x09\x7a\x10" + "\x49\xe6\x17\xd9\x1d\x36\x10\x94" + "\xfa\x68\xf0\xff\x77\x98\x71\x30" + "\x30\x5b\xea\xba\x2e\xda\x04\xdf" + "\x99\x7b\x71\x4d\x6c\x6f\x2c\x29" + "\xa6\xad\x5c\xb4\x02\x2b\x02\x70" + "\x9b\xee\xad\x9d\x67\x89\x0c\xbb" + "\x22\x39\x23\x36\xfe\xa1\x85\x1f" + "\x38", + .ilen = 281, + .result = "\x49\x6e\x74\x65\x72\x6e\x65\x74" + "\x2d\x44\x72\x61\x66\x74\x73\x20" + "\x61\x72\x65\x20\x64\x72\x61\x66" + "\x74\x20\x64\x6f\x63\x75\x6d\x65" + "\x6e\x74\x73\x20\x76\x61\x6c\x69" + "\x64\x20\x66\x6f\x72\x20\x61\x20" + "\x6d\x61\x78\x69\x6d\x75\x6d\x20" + "\x6f\x66\x20\x73\x69\x78\x20\x6d" + "\x6f\x6e\x74\x68\x73\x20\x61\x6e" + "\x64\x20\x6d\x61\x79\x20\x62\x65" + "\x20\x75\x70\x64\x61\x74\x65\x64" + "\x2c\x20\x72\x65\x70\x6c\x61\x63" + "\x65\x64\x2c\x20\x6f\x72\x20\x6f" + "\x62\x73\x6f\x6c\x65\x74\x65\x64" + "\x20\x62\x79\x20\x6f\x74\x68\x65" + "\x72\x20\x64\x6f\x63\x75\x6d\x65" + "\x6e\x74\x73\x20\x61\x74\x20\x61" + "\x6e\x79\x20\x74\x69\x6d\x65\x2e" + "\x20\x49\x74\x20\x69\x73\x20\x69" + "\x6e\x61\x70\x70\x72\x6f\x70\x72" + "\x69\x61\x74\x65\x20\x74\x6f\x20" + "\x75\x73\x65\x20\x49\x6e\x74\x65" + "\x72\x6e\x65\x74\x2d\x44\x72\x61" + "\x66\x74\x73\x20\x61\x73\x20\x72" + "\x65\x66\x65\x72\x65\x6e\x63\x65" + "\x20\x6d\x61\x74\x65\x72\x69\x61" + "\x6c\x20\x6f\x72\x20\x74\x6f\x20" + "\x63\x69\x74\x65\x20\x74\x68\x65" + "\x6d\x20\x6f\x74\x68\x65\x72\x20" + "\x74\x68\x61\x6e\x20\x61\x73\x20" + "\x2f\xe2\x80\x9c\x77\x6f\x72\x6b" + "\x20\x69\x6e\x20\x70\x72\x6f\x67" + "\x72\x65\x73\x73\x2e\x2f\xe2\x80" + "\x9d", + .rlen = 265, + }, +}; + +static const struct aead_testvec aegis128_enc_tv_template[] = { + { + .key = "\x0f\xc9\x8e\x67\x44\x9e\xaa\x86" + "\x20\x36\x2c\x24\xfe\xc9\x30\x81", + .klen = 16, + .iv = "\x1e\x92\x1c\xcf\x88\x3d\x54\x0d" + "\x40\x6d\x59\x48\xfc\x92\x61\x03", + .assoc = "", + .alen = 0, + .input = "", + .ilen = 0, + .result = "\x07\xa5\x11\xf2\x9d\x40\xb8\x6d" + "\xda\xb8\x12\x34\x4c\x53\xd9\x72", + .rlen = 16, + }, { + .key = "\x4b\xed\xc8\x07\x54\x1a\x52\xa2" + "\xa1\x10\xde\xb5\xf8\xed\xf3\x87", + .klen = 16, + .iv = "\x5a\xb7\x56\x6e\x98\xb9\xfd\x29" + "\xc1\x47\x0b\xda\xf6\xb6\x23\x09", + .assoc = "", + .alen = 0, + .input = "\x79", + .ilen = 1, + .result = "\x9e\x78\x52\xae\xcb\x9e\xe4\xd3" + "\x9a\xd7\x5d\xd7\xaa\x9a\xe9\x5a" + "\xcc", + .rlen = 17, + }, { + .key = "\x88\x12\x01\xa6\x64\x96\xfb\xbe" + "\x22\xea\x90\x47\xf2\x11\xb5\x8e", + .klen = 16, + .iv = "\x97\xdb\x90\x0e\xa8\x35\xa5\x45" + "\x42\x21\xbd\x6b\xf0\xda\xe6\x0f", + .assoc = "", + .alen = 0, + .input = "\xb5\x6e\xad\xdd\x30\x72\xfa\x53" + "\x82\x8e\x16\xb4\xed\x6d\x47", + .ilen = 15, + .result = "\xc3\x80\x83\x04\x5f\xaa\x61\xc7" + "\xca\xdd\x6f\xac\x85\x08\xb5\x35" + "\x2b\xc2\x3e\x0b\x1b\x39\x37\x2b" + "\x7a\x21\x16\xb3\xe6\x67\x66", + .rlen = 31, + }, { + .key = "\xc4\x37\x3b\x45\x74\x11\xa4\xda" + "\xa2\xc5\x42\xd8\xec\x36\x78\x94", + .klen = 16, + .iv = "\xd3\x00\xc9\xad\xb8\xb0\x4e\x61" + "\xc3\xfb\x6f\xfd\xea\xff\xa9\x15", + .assoc = "", + .alen = 0, + .input = "\xf2\x92\xe6\x7d\x40\xee\xa3\x6f" + "\x03\x68\xc8\x45\xe7\x91\x0a\x18", + .ilen = 16, + .result = "\x23\x25\x30\xe5\x6a\xb6\x36\x7d" + "\x38\xfd\x3a\xd2\xc2\x58\xa9\x11" + "\x1e\xa8\x30\x9c\x16\xa4\xdb\x65" + "\x51\x10\x16\x27\x70\x9b\x64\x29", + .rlen = 32, + }, { + .key = "\x01\x5c\x75\xe5\x84\x8d\x4d\xf6" + "\x23\x9f\xf4\x6a\xe6\x5a\x3b\x9a", + .klen = 16, + .iv = "\x10\x25\x03\x4c\xc8\x2c\xf7\x7d" + "\x44\xd5\x21\x8e\xe4\x23\x6b\x1c", + .assoc = "", + .alen = 0, + .input = "\x2e\xb7\x20\x1c\x50\x6a\x4b\x8b" + "\x84\x42\x7a\xd7\xe1\xb5\xcd\x1f" + "\xd3", + .ilen = 17, + .result = "\x2a\x8d\x56\x91\xc6\xf3\x56\xa5" + "\x1f\xf0\x89\x2e\x13\xad\xe6\xf6" + "\x46\x80\xb1\x0e\x18\x30\x40\x97" + "\x03\xdf\x64\x3c\xbe\x93\x9e\xc9" + "\x3b", + .rlen = 33, + }, { + .key = "\x3d\x80\xae\x84\x94\x09\xf6\x12" + "\xa4\x79\xa6\xfb\xe0\x7f\xfd\xa0", + .klen = 16, + .iv = "\x4c\x49\x3d\xec\xd8\xa8\xa0\x98" + "\xc5\xb0\xd3\x1f\xde\x48\x2e\x22", + .assoc = "", + .alen = 0, + .input = "\x6b\xdc\x5a\xbb\x60\xe5\xf4\xa6" + "\x05\x1d\x2c\x68\xdb\xda\x8f\x25" + "\xfe\x8d\x45\x19\x1e\xc0\x0b\x99" + "\x88\x11\x39\x12\x1c\x3a\xbb", + .ilen = 31, + .result = "\x4e\xf6\xfa\x13\xde\x43\x63\x4c" + "\xe2\x04\x3e\xe4\x85\x14\xb6\x3f" + "\xb1\x8f\x4c\xdb\x41\xa2\x14\x99" + "\xf5\x53\x0f\x73\x86\x7e\x97\xa1" + "\x4b\x56\x5b\x94\xce\xcd\x74\xcd" + "\x75\xc4\x53\x01\x89\x45\x59", + .rlen = 47, + }, { + .key = "\x7a\xa5\xe8\x23\xa4\x84\x9e\x2d" + "\x25\x53\x58\x8c\xda\xa3\xc0\xa6", + .klen = 16, + .iv = "\x89\x6e\x77\x8b\xe8\x23\x49\xb4" + "\x45\x8a\x85\xb1\xd8\x6c\xf1\x28", + .assoc = "", + .alen = 0, + .input = "\xa7\x00\x93\x5b\x70\x61\x9d\xc2" + "\x86\xf7\xde\xfa\xd5\xfe\x52\x2b" + "\x28\x50\x51\x9d\x24\x60\x8d\xb3" + "\x49\x3e\x17\xea\xf6\x99\x5a\xdd", + .ilen = 32, + .result = "\xa4\x9a\xb7\xfd\xa0\xd4\xd6\x47" + "\x95\xf4\x58\x38\x14\x83\x27\x01" + "\x4c\xed\x32\x2c\xf7\xd6\x31\xf7" + "\x38\x1b\x2c\xc9\xb6\x31\xce\xaa" + "\xa5\x3c\x1a\x18\x5c\xce\xb9\xdf" + "\x51\x52\x77\xf2\x5e\x85\x80\x41", + .rlen = 48, + }, { + .key = "\xb6\xca\x22\xc3\xb4\x00\x47\x49" + "\xa6\x2d\x0a\x1e\xd4\xc7\x83\xad", + .klen = 16, + .iv = "\xc5\x93\xb0\x2a\xf8\x9f\xf1\xd0" + "\xc6\x64\x37\x42\xd2\x90\xb3\x2e", + .assoc = "\xd5", + .alen = 1, + .input = "", + .ilen = 0, + .result = "\xfb\xd4\x83\x71\x9e\x63\xad\x60" + "\xb9\xf9\xeb\x34\x52\x49\xcf\xb7", + .rlen = 16, + }, { + .key = "\xf3\xee\x5c\x62\xc4\x7c\xf0\x65" + "\x27\x08\xbd\xaf\xce\xec\x45\xb3", + .klen = 16, + .iv = "\x02\xb8\xea\xca\x09\x1b\x9a\xec" + "\x47\x3e\xe9\xd4\xcc\xb5\x76\x34", + .assoc = "\x11\x81\x78\x32\x4d\xb9\x44\x73" + "\x68\x75\x16\xf8\xcb\x7e\xa7", + .alen = 15, + .input = "", + .ilen = 0, + .result = "\x0c\xaf\x2e\x96\xf6\x97\x08\x71" + "\x7d\x3a\x84\xc4\x44\x57\x77\x7e", + .rlen = 16, + }, { + .key = "\x2f\x13\x95\x01\xd5\xf7\x99\x81" + "\xa8\xe2\x6f\x41\xc8\x10\x08\xb9", + .klen = 16, + .iv = "\x3f\xdc\x24\x69\x19\x96\x43\x08" + "\xc8\x18\x9b\x65\xc6\xd9\x39\x3b", + .assoc = "\x4e\xa5\xb2\xd1\x5d\x35\xed\x8f" + "\xe8\x4f\xc8\x89\xc5\xa2\x69\xbc", + .alen = 16, + .input = "", + .ilen = 0, + .result = "\xc7\x87\x09\x3b\xc7\x19\x74\x22" + "\x22\xa5\x67\x10\xb2\x36\xb3\x45", + .rlen = 16, + }, { + .key = "\x6c\x38\xcf\xa1\xe5\x73\x41\x9d" + "\x29\xbc\x21\xd2\xc2\x35\xcb\xbf", + .klen = 16, + .iv = "\x7b\x01\x5d\x08\x29\x12\xec\x24" + "\x49\xf3\x4d\xf7\xc0\xfe\xfb\x41", + .assoc = "\x8a\xca\xec\x70\x6d\xb1\x96\xab" + "\x69\x29\x7a\x1b\xbf\xc7\x2c\xc2" + "\x07", + .alen = 17, + .input = "", + .ilen = 0, + .result = "\x02\xc6\x3b\x46\x65\xb2\xef\x91" + "\x31\xf0\x45\x48\x8a\x2a\xed\xe4", + .rlen = 16, + }, { + .key = "\xa8\x5c\x09\x40\xf5\xef\xea\xb8" + "\xaa\x96\xd3\x64\xbc\x59\x8d\xc6", + .klen = 16, + .iv = "\xb8\x26\x97\xa8\x39\x8e\x94\x3f" + "\xca\xcd\xff\x88\xba\x22\xbe\x47", + .assoc = "\xc7\xef\x26\x10\x7d\x2c\x3f\xc6" + "\xea\x03\x2c\xac\xb9\xeb\xef\xc9" + "\x31\x6b\x08\x12\xfc\xd8\x37\x2d" + "\xe0\x17\x3a\x2e\x83\x5c\x8f", + .alen = 31, + .input = "", + .ilen = 0, + .result = "\x20\x85\xa8\xd0\x91\x48\x85\xf3" + "\x5a\x16\xc0\x57\x68\x47\xdd\xcb", + .rlen = 16, + }, { + .key = "\xe5\x81\x42\xdf\x05\x6a\x93\xd4" + "\x2b\x70\x85\xf5\xb6\x7d\x50\xcc", + .klen = 16, + .iv = "\xf4\x4a\xd1\x47\x49\x09\x3d\x5b" + "\x4b\xa7\xb1\x19\xb4\x46\x81\x4d", + .assoc = "\x03\x14\x5f\xaf\x8d\xa8\xe7\xe2" + "\x6b\xde\xde\x3e\xb3\x10\xb1\xcf" + "\x5c\x2d\x14\x96\x01\x78\xb9\x47" + "\xa1\x44\x19\x06\x5d\xbb\x2e\x2f", + .alen = 32, + .input = "", + .ilen = 0, + .result = "\x6a\xf8\x8d\x9c\x42\x75\x35\x79" + "\xc1\x96\xbd\x31\x6e\x69\x1b\x50", + .rlen = 16, + }, { + .key = "\x22\xa6\x7c\x7f\x15\xe6\x3c\xf0" + "\xac\x4b\x37\x86\xb0\xa2\x13\xd2", + .klen = 16, + .iv = "\x31\x6f\x0b\xe6\x59\x85\xe6\x77" + "\xcc\x81\x63\xab\xae\x6b\x43\x54", + .assoc = "\x40", + .alen = 1, + .input = "\x4f", + .ilen = 1, + .result = "\x01\x24\xb1\xba\xf6\xd3\xdf\x83" + "\x70\x45\xe3\x2a\x9d\x5c\x63\x98" + "\x39", + .rlen = 17, + }, { + .key = "\x5e\xcb\xb6\x1e\x25\x62\xe4\x0c" + "\x2d\x25\xe9\x18\xaa\xc6\xd5\xd8", + .klen = 16, + .iv = "\x6d\x94\x44\x86\x69\x00\x8f\x93" + "\x4d\x5b\x15\x3c\xa8\x8f\x06\x5a", + .assoc = "\x7c\x5d\xd3\xee\xad\x9f\x39\x1a" + "\x6d\x92\x42\x61\xa7\x58\x37", + .alen = 15, + .input = "\x8b\x26\x61\x55\xf1\x3e\xe3\xa1" + "\x8d\xc8\x6e\x85\xa5\x21\x67", + .ilen = 15, + .result = "\x18\x78\xc2\x6e\xe1\xf7\xe6\x8a" + "\xca\x0e\x62\x00\xa8\x21\xb5\x21" + "\x3d\x36\xdb\xf7\xcc\x31\x94\x9c" + "\x98\xbd\x71\x7a\xef\xa4\xfa", + .rlen = 31, + }, { + .key = "\x9b\xef\xf0\xbd\x35\xdd\x8d\x28" + "\xad\xff\x9b\xa9\xa4\xeb\x98\xdf", + .klen = 16, + .iv = "\xaa\xb8\x7e\x25\x79\x7c\x37\xaf" + "\xce\x36\xc7\xce\xa2\xb4\xc9\x60", + .assoc = "\xb9\x82\x0c\x8d\xbd\x1b\xe2\x36" + "\xee\x6c\xf4\xf2\xa1\x7d\xf9\xe2", + .alen = 16, + .input = "\xc8\x4b\x9b\xf5\x01\xba\x8c\xbd" + "\x0e\xa3\x21\x16\x9f\x46\x2a\x63", + .ilen = 16, + .result = "\xea\xd1\x81\x75\xb4\x13\x1d\x86" + "\xd4\x17\x26\xe5\xd6\x89\x39\x04" + "\xa9\x6c\xca\xac\x40\x73\xb2\x4c" + "\x9c\xb9\x0e\x79\x4c\x40\x65\xc6", + .rlen = 32, + }, { + .key = "\xd7\x14\x29\x5d\x45\x59\x36\x44" + "\x2e\xd9\x4d\x3b\x9e\x0f\x5b\xe5", + .klen = 16, + .iv = "\xe6\xdd\xb8\xc4\x89\xf8\xe0\xca" + "\x4f\x10\x7a\x5f\x9c\xd8\x8b\x66", + .assoc = "\xf5\xa6\x46\x2c\xce\x97\x8a\x51" + "\x6f\x46\xa6\x83\x9b\xa1\xbc\xe8" + "\x05", + .alen = 17, + .input = "\x05\x70\xd5\x94\x12\x36\x35\xd8" + "\x8f\x7d\xd3\xa8\x99\x6a\xed\x69" + "\xd0", + .ilen = 17, + .result = "\xf4\xb2\x84\xd1\x81\xfa\x98\x1c" + "\x38\x2d\x69\x90\x1c\x71\x38\x98" + "\x9f\xe1\x19\x3b\x63\x91\xaf\x6e" + "\x4b\x07\x2c\xac\x53\xc5\xd5\xfe" + "\x93", + .rlen = 33, + }, { + .key = "\x14\x39\x63\xfc\x56\xd5\xdf\x5f" + "\xaf\xb3\xff\xcc\x98\x33\x1d\xeb", + .klen = 16, + .iv = "\x23\x02\xf1\x64\x9a\x73\x89\xe6" + "\xd0\xea\x2c\xf1\x96\xfc\x4e\x6d", + .assoc = "\x32\xcb\x80\xcc\xde\x12\x33\x6d" + "\xf0\x20\x58\x15\x95\xc6\x7f\xee" + "\x2f\xf9\x4e\x2c\x1b\x98\x43\xc7" + "\x68\x28\x73\x40\x9f\x96\x4a", + .alen = 31, + .input = "\x41\x94\x0e\x33\x22\xb1\xdd\xf4" + "\x10\x57\x85\x39\x93\x8f\xaf\x70" + "\xfa\xa9\xd0\x4d\x5c\x40\x23\xcd" + "\x98\x34\xab\x37\x56\xae\x32", + .ilen = 31, + .result = "\xa0\xe7\x0a\x60\xe7\xb8\x8a\xdb" + "\x94\xd3\x93\xf2\x41\x86\x16\xdd" + "\x4c\xe8\xe7\xe0\x62\x48\x89\x40" + "\xc0\x49\x9b\x63\x32\xec\x8b\xdb" + "\xdc\xa6\xea\x2c\xc2\x7f\xf5\x04" + "\xcb\xe5\x47\xbb\xa7\xd1\x9d", + .rlen = 47, + }, { + .key = "\x50\x5d\x9d\x9b\x66\x50\x88\x7b" + "\x30\x8e\xb1\x5e\x92\x58\xe0\xf1", + .klen = 16, + .iv = "\x5f\x27\x2b\x03\xaa\xef\x32\x02" + "\x50\xc4\xde\x82\x90\x21\x11\x73", + .assoc = "\x6e\xf0\xba\x6b\xee\x8e\xdc\x89" + "\x71\xfb\x0a\xa6\x8f\xea\x41\xf4" + "\x5a\xbb\x59\xb0\x20\x38\xc5\xe0" + "\x29\x56\x52\x19\x79\xf5\xe9\x37", + .alen = 32, + .input = "\x7e\xb9\x48\xd3\x32\x2d\x86\x10" + "\x91\x31\x37\xcb\x8d\xb3\x72\x76" + "\x24\x6b\xdc\xd1\x61\xe0\xa5\xe7" + "\x5a\x61\x8a\x0f\x30\x0d\xd1\xec", + .ilen = 32, + .result = "\x62\xdc\x2d\x68\x2d\x71\xbb\x33" + "\x13\xdf\xc0\x46\xf6\x61\x94\xa7" + "\x60\xd3\xd4\xca\xd9\xbe\x82\xf3" + "\xf1\x5b\xa0\xfa\x15\xba\xda\xea" + "\x87\x68\x47\x08\x5d\xdd\x83\xb0" + "\x60\xf4\x93\x20\xdf\x34\x8f\xea", + .rlen = 48, + }, { + .key = "\x8d\x82\xd6\x3b\x76\xcc\x30\x97" + "\xb1\x68\x63\xef\x8c\x7c\xa3\xf7", + .klen = 16, + .iv = "\x9c\x4b\x65\xa2\xba\x6b\xdb\x1e" + "\xd1\x9e\x90\x13\x8a\x45\xd3\x79", + .assoc = "\xab\x14\xf3\x0a\xfe\x0a\x85\xa5" + "\xf2\xd5\xbc\x38\x89\x0e\x04\xfb" + "\x84\x7d\x65\x34\x25\xd8\x47\xfa" + "\xeb\x83\x31\xf1\x54\x54\x89\x0d" + "\x9d", + .alen = 33, + .input = "\xba\xde\x82\x72\x42\xa9\x2f\x2c" + "\x12\x0b\xe9\x5c\x87\xd7\x35\x7c" + "\x4f\x2e\xe8\x55\x66\x80\x27\x00" + "\x1b\x8f\x68\xe7\x0a\x6c\x71\xc3" + "\x21\x78\x55\x9d\x9c\x65\x7b\xcd" + "\x0a\x34\x97\xff\x47\x37\xb0\x2a" + "\x80\x0d\x19\x98\x33\xa9\x7a\xe3" + "\x2e\x4c\xc6\xf3\x8c\x88\x42\x01" + "\xbd", + .ilen = 65, + .result = "\x84\xc5\x21\xab\xe1\xeb\xbb\x6d" + "\xaa\x2a\xaf\xeb\x3b\x3b\x69\xe7" + "\x2c\x47\xef\x9d\xb7\x53\x36\xb7" + "\xb6\xf5\xe5\xa8\xc9\x9e\x02\xd7" + "\x83\x88\xc2\xbd\x2f\xf9\x10\xc0" + "\xf5\xa1\x6e\xd3\x97\x64\x82\xa3" + "\xfb\xda\x2c\xb1\x94\xa1\x58\x32" + "\xe8\xd4\x39\xfc\x9e\x26\xf9\xf1" + "\x61\xe6\xae\x07\xf2\xe0\xa7\x44" + "\x96\x28\x3b\xee\x6b\xc6\x16\x31" + "\x3f", + .rlen = 81, + }, { + .key = "\xc9\xa7\x10\xda\x86\x48\xd9\xb3" + "\x32\x42\x15\x80\x85\xa1\x65\xfe", + .klen = 16, + .iv = "\xd8\x70\x9f\x42\xca\xe6\x83\x3a" + "\x52\x79\x42\xa5\x84\x6a\x96\x7f", + .assoc = "\xe8\x39\x2d\xaa\x0e\x85\x2d\xc1" + "\x72\xaf\x6e\xc9\x82\x33\xc7\x01" + "\xaf\x40\x70\xb8\x2a\x78\xc9\x14" + "\xac\xb1\x10\xca\x2e\xb3\x28\xe4" + "\xac\xfa\x58\x7f\xe5\x73\x09\x8c" + "\x1d\x40\x87\x8c\xd9\x75\xc0\x55" + "\xa2\xda\x07\xd1\xc2\xa9\xd1\xbb" + "\x09\x4f\x77\x62\x88\x2d\xf2\x68" + "\x54", + .alen = 65, + .input = "\xf7\x02\xbb\x11\x52\x24\xd8\x48" + "\x93\xe6\x9b\xee\x81\xfc\xf7\x82" + "\x79\xf0\xf3\xd9\x6c\x20\xa9\x1a" + "\xdc\xbc\x47\xc0\xe4\xcb\x10\x99" + "\x2f", + .ilen = 33, + .result = "\x8f\x23\x47\xfb\xf2\xac\x23\x83" + "\x77\x09\xac\x74\xef\xd2\x56\xae" + "\x20\x7b\x7b\xca\x45\x8e\xc8\xc2" + "\x50\xbd\xc7\x44\x1c\x54\x98\xd8" + "\x1f\xd0\x9a\x79\xaa\xf9\xe1\xb3" + "\xb4\x98\x5a\x9b\xe4\x4d\xbf\x4e" + "\x39", + .rlen = 49, + }, { + .key = "\x06\xcc\x4a\x79\x96\xc3\x82\xcf" + "\xb3\x1c\xc7\x12\x7f\xc5\x28\x04", + .klen = 16, + .iv = "\x15\x95\xd8\xe1\xda\x62\x2c\x56" + "\xd3\x53\xf4\x36\x7e\x8e\x59\x85", + .assoc = "\x24\x5e\x67\x49\x1e\x01\xd6\xdd" + "\xf3\x89\x20\x5b\x7c\x57\x89\x07", + .alen = 16, + .input = "\x33\x27\xf5\xb1\x62\xa0\x80\x63" + "\x14\xc0\x4d\x7f\x7b\x20\xba\x89", + .ilen = 16, + .result = "\x42\xc3\x58\xfb\x29\xe2\x4a\x56" + "\xf1\xf5\xe1\x51\x55\x4b\x0a\x45" + "\x46\xb5\x8d\xac\xb6\x34\xd8\x8b" + "\xde\x20\x59\x77\xc1\x74\x90", + .rlen = 31, + }, { + .key = "\x42\xf0\x84\x19\xa6\x3f\x2b\xea" + "\x34\xf6\x79\xa3\x79\xe9\xeb\x0a", + .klen = 16, + .iv = "\x51\xb9\x12\x80\xea\xde\xd5\x71" + "\x54\x2d\xa6\xc8\x78\xb2\x1b\x8c", + .assoc = "\x61\x83\xa0\xe8\x2e\x7d\x7f\xf8" + "\x74\x63\xd2\xec\x76\x7c\x4c\x0d", + .alen = 16, + .input = "\x70\x4c\x2f\x50\x72\x1c\x29\x7f" + "\x95\x9a\xff\x10\x75\x45\x7d\x8f", + .ilen = 16, + .result = "\xb2\xfb\xf6\x97\x69\x7a\xe9\xec" + "\xe2\x94\xa1\x8b\xa0\x2b\x60\x72" + "\x1d\x04\xdd\x6a\xef\x46\x8f\x68" + "\xe9\xe0\x17\x45\x70\x12", + .rlen = 30, + }, { + .key = "\x7f\x15\xbd\xb8\xb6\xba\xd3\x06" + "\xb5\xd1\x2b\x35\x73\x0e\xad\x10", + .klen = 16, + .iv = "\x8e\xde\x4c\x20\xfa\x59\x7e\x8d" + "\xd5\x07\x58\x59\x72\xd7\xde\x92", + .assoc = "\x9d\xa7\xda\x88\x3e\xf8\x28\x14" + "\xf5\x3e\x85\x7d\x70\xa0\x0f\x13", + .alen = 16, + .input = "\xac\x70\x69\xef\x82\x97\xd2\x9b" + "\x15\x74\xb1\xa2\x6f\x69\x3f\x95", + .ilen = 16, + .result = "\x47\xda\x54\x42\x51\x72\xc4\x8b" + "\xf5\x57\x0f\x2f\x49\x0e\x11\x3b" + "\x78\x93\xec\xfc\xf4\xff\xe1\x2d", + .rlen = 24, + }, +}; + +/* + * AEGIS-128 test vectors - generated via reference implementation from + * SUPERCOP (https://bench.cr.yp.to/supercop.html): + * + * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz + * (see crypto_aead/aegis128/) + */ +static const struct aead_testvec aegis128_dec_tv_template[] = { + { + .key = "\x0f\xc9\x8e\x67\x44\x9e\xaa\x86" + "\x20\x36\x2c\x24\xfe\xc9\x30\x81", + .klen = 16, + .iv = "\x1e\x92\x1c\xcf\x88\x3d\x54\x0d" + "\x40\x6d\x59\x48\xfc\x92\x61\x03", + .assoc = "", + .alen = 0, + .input = "\x07\xa5\x11\xf2\x9d\x40\xb8\x6d" + "\xda\xb8\x12\x34\x4c\x53\xd9\x72", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x4b\xed\xc8\x07\x54\x1a\x52\xa2" + "\xa1\x10\xde\xb5\xf8\xed\xf3\x87", + .klen = 16, + .iv = "\x5a\xb7\x56\x6e\x98\xb9\xfd\x29" + "\xc1\x47\x0b\xda\xf6\xb6\x23\x09", + .assoc = "", + .alen = 0, + .input = "\x9e\x78\x52\xae\xcb\x9e\xe4\xd3" + "\x9a\xd7\x5d\xd7\xaa\x9a\xe9\x5a" + "\xcc", + .ilen = 17, + .result = "\x79", + .rlen = 1, + }, { + .key = "\x88\x12\x01\xa6\x64\x96\xfb\xbe" + "\x22\xea\x90\x47\xf2\x11\xb5\x8e", + .klen = 16, + .iv = "\x97\xdb\x90\x0e\xa8\x35\xa5\x45" + "\x42\x21\xbd\x6b\xf0\xda\xe6\x0f", + .assoc = "", + .alen = 0, + .input = "\xc3\x80\x83\x04\x5f\xaa\x61\xc7" + "\xca\xdd\x6f\xac\x85\x08\xb5\x35" + "\x2b\xc2\x3e\x0b\x1b\x39\x37\x2b" + "\x7a\x21\x16\xb3\xe6\x67\x66", + .ilen = 31, + .result = "\xb5\x6e\xad\xdd\x30\x72\xfa\x53" + "\x82\x8e\x16\xb4\xed\x6d\x47", + .rlen = 15, + }, { + .key = "\xc4\x37\x3b\x45\x74\x11\xa4\xda" + "\xa2\xc5\x42\xd8\xec\x36\x78\x94", + .klen = 16, + .iv = "\xd3\x00\xc9\xad\xb8\xb0\x4e\x61" + "\xc3\xfb\x6f\xfd\xea\xff\xa9\x15", + .assoc = "", + .alen = 0, + .input = "\x23\x25\x30\xe5\x6a\xb6\x36\x7d" + "\x38\xfd\x3a\xd2\xc2\x58\xa9\x11" + "\x1e\xa8\x30\x9c\x16\xa4\xdb\x65" + "\x51\x10\x16\x27\x70\x9b\x64\x29", + .ilen = 32, + .result = "\xf2\x92\xe6\x7d\x40\xee\xa3\x6f" + "\x03\x68\xc8\x45\xe7\x91\x0a\x18", + .rlen = 16, + }, { + .key = "\x01\x5c\x75\xe5\x84\x8d\x4d\xf6" + "\x23\x9f\xf4\x6a\xe6\x5a\x3b\x9a", + .klen = 16, + .iv = "\x10\x25\x03\x4c\xc8\x2c\xf7\x7d" + "\x44\xd5\x21\x8e\xe4\x23\x6b\x1c", + .assoc = "", + .alen = 0, + .input = "\x2a\x8d\x56\x91\xc6\xf3\x56\xa5" + "\x1f\xf0\x89\x2e\x13\xad\xe6\xf6" + "\x46\x80\xb1\x0e\x18\x30\x40\x97" + "\x03\xdf\x64\x3c\xbe\x93\x9e\xc9" + "\x3b", + .ilen = 33, + .result = "\x2e\xb7\x20\x1c\x50\x6a\x4b\x8b" + "\x84\x42\x7a\xd7\xe1\xb5\xcd\x1f" + "\xd3", + .rlen = 17, + }, { + .key = "\x3d\x80\xae\x84\x94\x09\xf6\x12" + "\xa4\x79\xa6\xfb\xe0\x7f\xfd\xa0", + .klen = 16, + .iv = "\x4c\x49\x3d\xec\xd8\xa8\xa0\x98" + "\xc5\xb0\xd3\x1f\xde\x48\x2e\x22", + .assoc = "", + .alen = 0, + .input = "\x4e\xf6\xfa\x13\xde\x43\x63\x4c" + "\xe2\x04\x3e\xe4\x85\x14\xb6\x3f" + "\xb1\x8f\x4c\xdb\x41\xa2\x14\x99" + "\xf5\x53\x0f\x73\x86\x7e\x97\xa1" + "\x4b\x56\x5b\x94\xce\xcd\x74\xcd" + "\x75\xc4\x53\x01\x89\x45\x59", + .ilen = 47, + .result = "\x6b\xdc\x5a\xbb\x60\xe5\xf4\xa6" + "\x05\x1d\x2c\x68\xdb\xda\x8f\x25" + "\xfe\x8d\x45\x19\x1e\xc0\x0b\x99" + "\x88\x11\x39\x12\x1c\x3a\xbb", + .rlen = 31, + }, { + .key = "\x7a\xa5\xe8\x23\xa4\x84\x9e\x2d" + "\x25\x53\x58\x8c\xda\xa3\xc0\xa6", + .klen = 16, + .iv = "\x89\x6e\x77\x8b\xe8\x23\x49\xb4" + "\x45\x8a\x85\xb1\xd8\x6c\xf1\x28", + .assoc = "", + .alen = 0, + .input = "\xa4\x9a\xb7\xfd\xa0\xd4\xd6\x47" + "\x95\xf4\x58\x38\x14\x83\x27\x01" + "\x4c\xed\x32\x2c\xf7\xd6\x31\xf7" + "\x38\x1b\x2c\xc9\xb6\x31\xce\xaa" + "\xa5\x3c\x1a\x18\x5c\xce\xb9\xdf" + "\x51\x52\x77\xf2\x5e\x85\x80\x41", + .ilen = 48, + .result = "\xa7\x00\x93\x5b\x70\x61\x9d\xc2" + "\x86\xf7\xde\xfa\xd5\xfe\x52\x2b" + "\x28\x50\x51\x9d\x24\x60\x8d\xb3" + "\x49\x3e\x17\xea\xf6\x99\x5a\xdd", + .rlen = 32, + }, { + .key = "\xb6\xca\x22\xc3\xb4\x00\x47\x49" + "\xa6\x2d\x0a\x1e\xd4\xc7\x83\xad", + .klen = 16, + .iv = "\xc5\x93\xb0\x2a\xf8\x9f\xf1\xd0" + "\xc6\x64\x37\x42\xd2\x90\xb3\x2e", + .assoc = "\xd5", + .alen = 1, + .input = "\xfb\xd4\x83\x71\x9e\x63\xad\x60" + "\xb9\xf9\xeb\x34\x52\x49\xcf\xb7", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xf3\xee\x5c\x62\xc4\x7c\xf0\x65" + "\x27\x08\xbd\xaf\xce\xec\x45\xb3", + .klen = 16, + .iv = "\x02\xb8\xea\xca\x09\x1b\x9a\xec" + "\x47\x3e\xe9\xd4\xcc\xb5\x76\x34", + .assoc = "\x11\x81\x78\x32\x4d\xb9\x44\x73" + "\x68\x75\x16\xf8\xcb\x7e\xa7", + .alen = 15, + .input = "\x0c\xaf\x2e\x96\xf6\x97\x08\x71" + "\x7d\x3a\x84\xc4\x44\x57\x77\x7e", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x2f\x13\x95\x01\xd5\xf7\x99\x81" + "\xa8\xe2\x6f\x41\xc8\x10\x08\xb9", + .klen = 16, + .iv = "\x3f\xdc\x24\x69\x19\x96\x43\x08" + "\xc8\x18\x9b\x65\xc6\xd9\x39\x3b", + .assoc = "\x4e\xa5\xb2\xd1\x5d\x35\xed\x8f" + "\xe8\x4f\xc8\x89\xc5\xa2\x69\xbc", + .alen = 16, + .input = "\xc7\x87\x09\x3b\xc7\x19\x74\x22" + "\x22\xa5\x67\x10\xb2\x36\xb3\x45", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x6c\x38\xcf\xa1\xe5\x73\x41\x9d" + "\x29\xbc\x21\xd2\xc2\x35\xcb\xbf", + .klen = 16, + .iv = "\x7b\x01\x5d\x08\x29\x12\xec\x24" + "\x49\xf3\x4d\xf7\xc0\xfe\xfb\x41", + .assoc = "\x8a\xca\xec\x70\x6d\xb1\x96\xab" + "\x69\x29\x7a\x1b\xbf\xc7\x2c\xc2" + "\x07", + .alen = 17, + .input = "\x02\xc6\x3b\x46\x65\xb2\xef\x91" + "\x31\xf0\x45\x48\x8a\x2a\xed\xe4", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xa8\x5c\x09\x40\xf5\xef\xea\xb8" + "\xaa\x96\xd3\x64\xbc\x59\x8d\xc6", + .klen = 16, + .iv = "\xb8\x26\x97\xa8\x39\x8e\x94\x3f" + "\xca\xcd\xff\x88\xba\x22\xbe\x47", + .assoc = "\xc7\xef\x26\x10\x7d\x2c\x3f\xc6" + "\xea\x03\x2c\xac\xb9\xeb\xef\xc9" + "\x31\x6b\x08\x12\xfc\xd8\x37\x2d" + "\xe0\x17\x3a\x2e\x83\x5c\x8f", + .alen = 31, + .input = "\x20\x85\xa8\xd0\x91\x48\x85\xf3" + "\x5a\x16\xc0\x57\x68\x47\xdd\xcb", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xe5\x81\x42\xdf\x05\x6a\x93\xd4" + "\x2b\x70\x85\xf5\xb6\x7d\x50\xcc", + .klen = 16, + .iv = "\xf4\x4a\xd1\x47\x49\x09\x3d\x5b" + "\x4b\xa7\xb1\x19\xb4\x46\x81\x4d", + .assoc = "\x03\x14\x5f\xaf\x8d\xa8\xe7\xe2" + "\x6b\xde\xde\x3e\xb3\x10\xb1\xcf" + "\x5c\x2d\x14\x96\x01\x78\xb9\x47" + "\xa1\x44\x19\x06\x5d\xbb\x2e\x2f", + .alen = 32, + .input = "\x6a\xf8\x8d\x9c\x42\x75\x35\x79" + "\xc1\x96\xbd\x31\x6e\x69\x1b\x50", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x22\xa6\x7c\x7f\x15\xe6\x3c\xf0" + "\xac\x4b\x37\x86\xb0\xa2\x13\xd2", + .klen = 16, + .iv = "\x31\x6f\x0b\xe6\x59\x85\xe6\x77" + "\xcc\x81\x63\xab\xae\x6b\x43\x54", + .assoc = "\x40", + .alen = 1, + .input = "\x01\x24\xb1\xba\xf6\xd3\xdf\x83" + "\x70\x45\xe3\x2a\x9d\x5c\x63\x98" + "\x39", + .ilen = 17, + .result = "\x4f", + .rlen = 1, + }, { + .key = "\x5e\xcb\xb6\x1e\x25\x62\xe4\x0c" + "\x2d\x25\xe9\x18\xaa\xc6\xd5\xd8", + .klen = 16, + .iv = "\x6d\x94\x44\x86\x69\x00\x8f\x93" + "\x4d\x5b\x15\x3c\xa8\x8f\x06\x5a", + .assoc = "\x7c\x5d\xd3\xee\xad\x9f\x39\x1a" + "\x6d\x92\x42\x61\xa7\x58\x37", + .alen = 15, + .input = "\x18\x78\xc2\x6e\xe1\xf7\xe6\x8a" + "\xca\x0e\x62\x00\xa8\x21\xb5\x21" + "\x3d\x36\xdb\xf7\xcc\x31\x94\x9c" + "\x98\xbd\x71\x7a\xef\xa4\xfa", + .ilen = 31, + .result = "\x8b\x26\x61\x55\xf1\x3e\xe3\xa1" + "\x8d\xc8\x6e\x85\xa5\x21\x67", + .rlen = 15, + }, { + .key = "\x9b\xef\xf0\xbd\x35\xdd\x8d\x28" + "\xad\xff\x9b\xa9\xa4\xeb\x98\xdf", + .klen = 16, + .iv = "\xaa\xb8\x7e\x25\x79\x7c\x37\xaf" + "\xce\x36\xc7\xce\xa2\xb4\xc9\x60", + .assoc = "\xb9\x82\x0c\x8d\xbd\x1b\xe2\x36" + "\xee\x6c\xf4\xf2\xa1\x7d\xf9\xe2", + .alen = 16, + .input = "\xea\xd1\x81\x75\xb4\x13\x1d\x86" + "\xd4\x17\x26\xe5\xd6\x89\x39\x04" + "\xa9\x6c\xca\xac\x40\x73\xb2\x4c" + "\x9c\xb9\x0e\x79\x4c\x40\x65\xc6", + .ilen = 32, + .result = "\xc8\x4b\x9b\xf5\x01\xba\x8c\xbd" + "\x0e\xa3\x21\x16\x9f\x46\x2a\x63", + .rlen = 16, + }, { + .key = "\xd7\x14\x29\x5d\x45\x59\x36\x44" + "\x2e\xd9\x4d\x3b\x9e\x0f\x5b\xe5", + .klen = 16, + .iv = "\xe6\xdd\xb8\xc4\x89\xf8\xe0\xca" + "\x4f\x10\x7a\x5f\x9c\xd8\x8b\x66", + .assoc = "\xf5\xa6\x46\x2c\xce\x97\x8a\x51" + "\x6f\x46\xa6\x83\x9b\xa1\xbc\xe8" + "\x05", + .alen = 17, + .input = "\xf4\xb2\x84\xd1\x81\xfa\x98\x1c" + "\x38\x2d\x69\x90\x1c\x71\x38\x98" + "\x9f\xe1\x19\x3b\x63\x91\xaf\x6e" + "\x4b\x07\x2c\xac\x53\xc5\xd5\xfe" + "\x93", + .ilen = 33, + .result = "\x05\x70\xd5\x94\x12\x36\x35\xd8" + "\x8f\x7d\xd3\xa8\x99\x6a\xed\x69" + "\xd0", + .rlen = 17, + }, { + .key = "\x14\x39\x63\xfc\x56\xd5\xdf\x5f" + "\xaf\xb3\xff\xcc\x98\x33\x1d\xeb", + .klen = 16, + .iv = "\x23\x02\xf1\x64\x9a\x73\x89\xe6" + "\xd0\xea\x2c\xf1\x96\xfc\x4e\x6d", + .assoc = "\x32\xcb\x80\xcc\xde\x12\x33\x6d" + "\xf0\x20\x58\x15\x95\xc6\x7f\xee" + "\x2f\xf9\x4e\x2c\x1b\x98\x43\xc7" + "\x68\x28\x73\x40\x9f\x96\x4a", + .alen = 31, + .input = "\xa0\xe7\x0a\x60\xe7\xb8\x8a\xdb" + "\x94\xd3\x93\xf2\x41\x86\x16\xdd" + "\x4c\xe8\xe7\xe0\x62\x48\x89\x40" + "\xc0\x49\x9b\x63\x32\xec\x8b\xdb" + "\xdc\xa6\xea\x2c\xc2\x7f\xf5\x04" + "\xcb\xe5\x47\xbb\xa7\xd1\x9d", + .ilen = 47, + .result = "\x41\x94\x0e\x33\x22\xb1\xdd\xf4" + "\x10\x57\x85\x39\x93\x8f\xaf\x70" + "\xfa\xa9\xd0\x4d\x5c\x40\x23\xcd" + "\x98\x34\xab\x37\x56\xae\x32", + .rlen = 31, + }, { + .key = "\x50\x5d\x9d\x9b\x66\x50\x88\x7b" + "\x30\x8e\xb1\x5e\x92\x58\xe0\xf1", + .klen = 16, + .iv = "\x5f\x27\x2b\x03\xaa\xef\x32\x02" + "\x50\xc4\xde\x82\x90\x21\x11\x73", + .assoc = "\x6e\xf0\xba\x6b\xee\x8e\xdc\x89" + "\x71\xfb\x0a\xa6\x8f\xea\x41\xf4" + "\x5a\xbb\x59\xb0\x20\x38\xc5\xe0" + "\x29\x56\x52\x19\x79\xf5\xe9\x37", + .alen = 32, + .input = "\x62\xdc\x2d\x68\x2d\x71\xbb\x33" + "\x13\xdf\xc0\x46\xf6\x61\x94\xa7" + "\x60\xd3\xd4\xca\xd9\xbe\x82\xf3" + "\xf1\x5b\xa0\xfa\x15\xba\xda\xea" + "\x87\x68\x47\x08\x5d\xdd\x83\xb0" + "\x60\xf4\x93\x20\xdf\x34\x8f\xea", + .ilen = 48, + .result = "\x7e\xb9\x48\xd3\x32\x2d\x86\x10" + "\x91\x31\x37\xcb\x8d\xb3\x72\x76" + "\x24\x6b\xdc\xd1\x61\xe0\xa5\xe7" + "\x5a\x61\x8a\x0f\x30\x0d\xd1\xec", + .rlen = 32, + }, { + .key = "\x8d\x82\xd6\x3b\x76\xcc\x30\x97" + "\xb1\x68\x63\xef\x8c\x7c\xa3\xf7", + .klen = 16, + .iv = "\x9c\x4b\x65\xa2\xba\x6b\xdb\x1e" + "\xd1\x9e\x90\x13\x8a\x45\xd3\x79", + .assoc = "\xab\x14\xf3\x0a\xfe\x0a\x85\xa5" + "\xf2\xd5\xbc\x38\x89\x0e\x04\xfb" + "\x84\x7d\x65\x34\x25\xd8\x47\xfa" + "\xeb\x83\x31\xf1\x54\x54\x89\x0d" + "\x9d", + .alen = 33, + .input = "\x84\xc5\x21\xab\xe1\xeb\xbb\x6d" + "\xaa\x2a\xaf\xeb\x3b\x3b\x69\xe7" + "\x2c\x47\xef\x9d\xb7\x53\x36\xb7" + "\xb6\xf5\xe5\xa8\xc9\x9e\x02\xd7" + "\x83\x88\xc2\xbd\x2f\xf9\x10\xc0" + "\xf5\xa1\x6e\xd3\x97\x64\x82\xa3" + "\xfb\xda\x2c\xb1\x94\xa1\x58\x32" + "\xe8\xd4\x39\xfc\x9e\x26\xf9\xf1" + "\x61\xe6\xae\x07\xf2\xe0\xa7\x44" + "\x96\x28\x3b\xee\x6b\xc6\x16\x31" + "\x3f", + .ilen = 81, + .result = "\xba\xde\x82\x72\x42\xa9\x2f\x2c" + "\x12\x0b\xe9\x5c\x87\xd7\x35\x7c" + "\x4f\x2e\xe8\x55\x66\x80\x27\x00" + "\x1b\x8f\x68\xe7\x0a\x6c\x71\xc3" + "\x21\x78\x55\x9d\x9c\x65\x7b\xcd" + "\x0a\x34\x97\xff\x47\x37\xb0\x2a" + "\x80\x0d\x19\x98\x33\xa9\x7a\xe3" + "\x2e\x4c\xc6\xf3\x8c\x88\x42\x01" + "\xbd", + .rlen = 65, + }, { + .key = "\xc9\xa7\x10\xda\x86\x48\xd9\xb3" + "\x32\x42\x15\x80\x85\xa1\x65\xfe", + .klen = 16, + .iv = "\xd8\x70\x9f\x42\xca\xe6\x83\x3a" + "\x52\x79\x42\xa5\x84\x6a\x96\x7f", + .assoc = "\xe8\x39\x2d\xaa\x0e\x85\x2d\xc1" + "\x72\xaf\x6e\xc9\x82\x33\xc7\x01" + "\xaf\x40\x70\xb8\x2a\x78\xc9\x14" + "\xac\xb1\x10\xca\x2e\xb3\x28\xe4" + "\xac\xfa\x58\x7f\xe5\x73\x09\x8c" + "\x1d\x40\x87\x8c\xd9\x75\xc0\x55" + "\xa2\xda\x07\xd1\xc2\xa9\xd1\xbb" + "\x09\x4f\x77\x62\x88\x2d\xf2\x68" + "\x54", + .alen = 65, + .input = "\x8f\x23\x47\xfb\xf2\xac\x23\x83" + "\x77\x09\xac\x74\xef\xd2\x56\xae" + "\x20\x7b\x7b\xca\x45\x8e\xc8\xc2" + "\x50\xbd\xc7\x44\x1c\x54\x98\xd8" + "\x1f\xd0\x9a\x79\xaa\xf9\xe1\xb3" + "\xb4\x98\x5a\x9b\xe4\x4d\xbf\x4e" + "\x39", + .ilen = 49, + .result = "\xf7\x02\xbb\x11\x52\x24\xd8\x48" + "\x93\xe6\x9b\xee\x81\xfc\xf7\x82" + "\x79\xf0\xf3\xd9\x6c\x20\xa9\x1a" + "\xdc\xbc\x47\xc0\xe4\xcb\x10\x99" + "\x2f", + .rlen = 33, + }, { + .key = "\x06\xcc\x4a\x79\x96\xc3\x82\xcf" + "\xb3\x1c\xc7\x12\x7f\xc5\x28\x04", + .klen = 16, + .iv = "\x15\x95\xd8\xe1\xda\x62\x2c\x56" + "\xd3\x53\xf4\x36\x7e\x8e\x59\x85", + .assoc = "\x24\x5e\x67\x49\x1e\x01\xd6\xdd" + "\xf3\x89\x20\x5b\x7c\x57\x89\x07", + .alen = 16, + .input = "\x42\xc3\x58\xfb\x29\xe2\x4a\x56" + "\xf1\xf5\xe1\x51\x55\x4b\x0a\x45" + "\x46\xb5\x8d\xac\xb6\x34\xd8\x8b" + "\xde\x20\x59\x77\xc1\x74\x90", + .ilen = 31, + .result = "\x33\x27\xf5\xb1\x62\xa0\x80\x63" + "\x14\xc0\x4d\x7f\x7b\x20\xba\x89", + .rlen = 16, + }, { + .key = "\x42\xf0\x84\x19\xa6\x3f\x2b\xea" + "\x34\xf6\x79\xa3\x79\xe9\xeb\x0a", + .klen = 16, + .iv = "\x51\xb9\x12\x80\xea\xde\xd5\x71" + "\x54\x2d\xa6\xc8\x78\xb2\x1b\x8c", + .assoc = "\x61\x83\xa0\xe8\x2e\x7d\x7f\xf8" + "\x74\x63\xd2\xec\x76\x7c\x4c\x0d", + .alen = 16, + .input = "\xb2\xfb\xf6\x97\x69\x7a\xe9\xec" + "\xe2\x94\xa1\x8b\xa0\x2b\x60\x72" + "\x1d\x04\xdd\x6a\xef\x46\x8f\x68" + "\xe9\xe0\x17\x45\x70\x12", + .ilen = 30, + .result = "\x70\x4c\x2f\x50\x72\x1c\x29\x7f" + "\x95\x9a\xff\x10\x75\x45\x7d\x8f", + .rlen = 16, + }, { + .key = "\x7f\x15\xbd\xb8\xb6\xba\xd3\x06" + "\xb5\xd1\x2b\x35\x73\x0e\xad\x10", + .klen = 16, + .iv = "\x8e\xde\x4c\x20\xfa\x59\x7e\x8d" + "\xd5\x07\x58\x59\x72\xd7\xde\x92", + .assoc = "\x9d\xa7\xda\x88\x3e\xf8\x28\x14" + "\xf5\x3e\x85\x7d\x70\xa0\x0f\x13", + .alen = 16, + .input = "\x47\xda\x54\x42\x51\x72\xc4\x8b" + "\xf5\x57\x0f\x2f\x49\x0e\x11\x3b" + "\x78\x93\xec\xfc\xf4\xff\xe1\x2d", + .ilen = 24, + .result = "\xac\x70\x69\xef\x82\x97\xd2\x9b" + "\x15\x74\xb1\xa2\x6f\x69\x3f\x95", + .rlen = 16, + }, +}; + +/* + * AEGIS-128L test vectors - generated via reference implementation from + * SUPERCOP (https://bench.cr.yp.to/supercop.html): + * + * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz + * (see crypto_aead/aegis128l/) + */ +static const struct aead_testvec aegis128l_enc_tv_template[] = { + { + .key = "\x0f\xc9\x8e\x67\x44\x9e\xaa\x86" + "\x20\x36\x2c\x24\xfe\xc9\x30\x81", + .klen = 16, + .iv = "\x1e\x92\x1c\xcf\x88\x3d\x54\x0d" + "\x40\x6d\x59\x48\xfc\x92\x61\x03", + .assoc = "", + .alen = 0, + .input = "", + .ilen = 0, + .result = "\x30\x4f\xf3\xe9\xb1\xfa\x81\xa6" + "\x20\x72\x78\xdd\x93\xc8\x57\xef", + .rlen = 16, + }, { + .key = "\x4b\xed\xc8\x07\x54\x1a\x52\xa2" + "\xa1\x10\xde\xb5\xf8\xed\xf3\x87", + .klen = 16, + .iv = "\x5a\xb7\x56\x6e\x98\xb9\xfd\x29" + "\xc1\x47\x0b\xda\xf6\xb6\x23\x09", + .assoc = "", + .alen = 0, + .input = "\x79", + .ilen = 1, + .result = "\xa9\x24\xa0\xb6\x2d\xdd\x29\xdb" + "\x40\xb3\x71\xc5\x22\x58\x31\x77" + "\x6d", + .rlen = 17, + }, { + .key = "\x88\x12\x01\xa6\x64\x96\xfb\xbe" + "\x22\xea\x90\x47\xf2\x11\xb5\x8e", + .klen = 16, + .iv = "\x97\xdb\x90\x0e\xa8\x35\xa5\x45" + "\x42\x21\xbd\x6b\xf0\xda\xe6\x0f", + .assoc = "", + .alen = 0, + .input = "\xb5\x6e\xad\xdd\x30\x72\xfa\x53" + "\x82\x8e\x16\xb4\xed\x6d\x47", + .ilen = 15, + .result = "\xbb\x0a\x53\xc4\xaa\x7e\xa4\x03" + "\x2b\xee\x62\x99\x7b\x98\x13\x1f" + "\xe0\x76\x4c\x2e\x53\x99\x4f\xbe" + "\xe1\xa8\x04\x7f\xe1\x71\xbe", + .rlen = 31, + }, { + .key = "\xc4\x37\x3b\x45\x74\x11\xa4\xda" + "\xa2\xc5\x42\xd8\xec\x36\x78\x94", + .klen = 16, + .iv = "\xd3\x00\xc9\xad\xb8\xb0\x4e\x61" + "\xc3\xfb\x6f\xfd\xea\xff\xa9\x15", + .assoc = "", + .alen = 0, + .input = "\xf2\x92\xe6\x7d\x40\xee\xa3\x6f" + "\x03\x68\xc8\x45\xe7\x91\x0a\x18", + .ilen = 16, + .result = "\x66\xdf\x6e\x71\xc0\x6e\xa4\x4c" + "\x9d\xb7\x8c\x9a\xdb\x1f\xd2\x2e" + "\x23\xb6\xa4\xfb\xd3\x86\xdd\xbb" + "\xde\x54\x9b\xf5\x92\x8b\x93\xc5", + .rlen = 32, + }, { + .key = "\x01\x5c\x75\xe5\x84\x8d\x4d\xf6" + "\x23\x9f\xf4\x6a\xe6\x5a\x3b\x9a", + .klen = 16, + .iv = "\x10\x25\x03\x4c\xc8\x2c\xf7\x7d" + "\x44\xd5\x21\x8e\xe4\x23\x6b\x1c", + .assoc = "", + .alen = 0, + .input = "\x2e\xb7\x20\x1c\x50\x6a\x4b\x8b" + "\x84\x42\x7a\xd7\xe1\xb5\xcd\x1f" + "\xd3", + .ilen = 17, + .result = "\x4f\xc3\x69\xb6\xd3\xa4\x64\x8b" + "\x71\xc3\x8a\x91\x22\x4f\x1b\xd2" + "\x33\x6d\x86\xbc\xf8\x2f\x06\xf9" + "\x82\x64\xc7\x72\x00\x30\xfc\xf0" + "\xf8", + .rlen = 33, + }, { + .key = "\x3d\x80\xae\x84\x94\x09\xf6\x12" + "\xa4\x79\xa6\xfb\xe0\x7f\xfd\xa0", + .klen = 16, + .iv = "\x4c\x49\x3d\xec\xd8\xa8\xa0\x98" + "\xc5\xb0\xd3\x1f\xde\x48\x2e\x22", + .assoc = "", + .alen = 0, + .input = "\x6b\xdc\x5a\xbb\x60\xe5\xf4\xa6" + "\x05\x1d\x2c\x68\xdb\xda\x8f\x25" + "\xfe\x8d\x45\x19\x1e\xc0\x0b\x99" + "\x88\x11\x39\x12\x1c\x3a\xbb", + .ilen = 31, + .result = "\xe3\x93\x15\xae\x5f\x9d\x3c\xb5" + "\xd6\x9d\xee\xee\xcf\xaa\xaf\xe1" + "\x45\x10\x96\xe0\xbf\x55\x0f\x4c" + "\x1a\xfd\xf4\xda\x4e\x10\xde\xc9" + "\x0e\x6f\xc7\x3c\x49\x94\x41\xfc" + "\x59\x28\x88\x3c\x79\x10\x6b", + .rlen = 47, + }, { + .key = "\x7a\xa5\xe8\x23\xa4\x84\x9e\x2d" + "\x25\x53\x58\x8c\xda\xa3\xc0\xa6", + .klen = 16, + .iv = "\x89\x6e\x77\x8b\xe8\x23\x49\xb4" + "\x45\x8a\x85\xb1\xd8\x6c\xf1\x28", + .assoc = "", + .alen = 0, + .input = "\xa7\x00\x93\x5b\x70\x61\x9d\xc2" + "\x86\xf7\xde\xfa\xd5\xfe\x52\x2b" + "\x28\x50\x51\x9d\x24\x60\x8d\xb3" + "\x49\x3e\x17\xea\xf6\x99\x5a\xdd", + .ilen = 32, + .result = "\x1c\x8e\x22\x34\xfd\xab\xe6\x0d" + "\x1c\x9f\x06\x54\x8b\x0b\xb4\x40" + "\xde\x11\x59\x3e\xfd\x74\xf6\x42" + "\x97\x17\xf7\x24\xb6\x7e\xc4\xc6" + "\x06\xa3\x94\xda\x3d\x7f\x55\x0a" + "\x92\x07\x2f\xa6\xf3\x6b\x2c\xfc", + .rlen = 48, + }, { + .key = "\xb6\xca\x22\xc3\xb4\x00\x47\x49" + "\xa6\x2d\x0a\x1e\xd4\xc7\x83\xad", + .klen = 16, + .iv = "\xc5\x93\xb0\x2a\xf8\x9f\xf1\xd0" + "\xc6\x64\x37\x42\xd2\x90\xb3\x2e", + .assoc = "\xd5", + .alen = 1, + .input = "", + .ilen = 0, + .result = "\xa0\x2a\xb4\x9a\x91\x00\x15\xb8" + "\x0f\x9a\x15\x60\x0e\x9b\x13\x8f", + .rlen = 16, + }, { + .key = "\xf3\xee\x5c\x62\xc4\x7c\xf0\x65" + "\x27\x08\xbd\xaf\xce\xec\x45\xb3", + .klen = 16, + .iv = "\x02\xb8\xea\xca\x09\x1b\x9a\xec" + "\x47\x3e\xe9\xd4\xcc\xb5\x76\x34", + .assoc = "\x11\x81\x78\x32\x4d\xb9\x44\x73" + "\x68\x75\x16\xf8\xcb\x7e\xa7", + .alen = 15, + .input = "", + .ilen = 0, + .result = "\x4c\x26\xad\x9c\x14\xfd\x9c\x8c" + "\x84\xfb\x26\xfb\xd5\xca\x62\x39", + .rlen = 16, + }, { + .key = "\x2f\x13\x95\x01\xd5\xf7\x99\x81" + "\xa8\xe2\x6f\x41\xc8\x10\x08\xb9", + .klen = 16, + .iv = "\x3f\xdc\x24\x69\x19\x96\x43\x08" + "\xc8\x18\x9b\x65\xc6\xd9\x39\x3b", + .assoc = "\x4e\xa5\xb2\xd1\x5d\x35\xed\x8f" + "\xe8\x4f\xc8\x89\xc5\xa2\x69\xbc", + .alen = 16, + .input = "", + .ilen = 0, + .result = "\x45\x85\x0e\x0f\xf4\xae\x96\xa1" + "\x99\x4d\x6d\xb4\x67\x32\xb0\x3a", + .rlen = 16, + }, { + .key = "\x6c\x38\xcf\xa1\xe5\x73\x41\x9d" + "\x29\xbc\x21\xd2\xc2\x35\xcb\xbf", + .klen = 16, + .iv = "\x7b\x01\x5d\x08\x29\x12\xec\x24" + "\x49\xf3\x4d\xf7\xc0\xfe\xfb\x41", + .assoc = "\x8a\xca\xec\x70\x6d\xb1\x96\xab" + "\x69\x29\x7a\x1b\xbf\xc7\x2c\xc2" + "\x07", + .alen = 17, + .input = "", + .ilen = 0, + .result = "\x33\xb1\x42\x97\x8e\x16\x7b\x63" + "\x06\xba\x5b\xcb\xae\x6d\x8b\x56", + .rlen = 16, + }, { + .key = "\xa8\x5c\x09\x40\xf5\xef\xea\xb8" + "\xaa\x96\xd3\x64\xbc\x59\x8d\xc6", + .klen = 16, + .iv = "\xb8\x26\x97\xa8\x39\x8e\x94\x3f" + "\xca\xcd\xff\x88\xba\x22\xbe\x47", + .assoc = "\xc7\xef\x26\x10\x7d\x2c\x3f\xc6" + "\xea\x03\x2c\xac\xb9\xeb\xef\xc9" + "\x31\x6b\x08\x12\xfc\xd8\x37\x2d" + "\xe0\x17\x3a\x2e\x83\x5c\x8f", + .alen = 31, + .input = "", + .ilen = 0, + .result = "\xda\x44\x08\x8c\x2a\xa5\x07\x35" + "\x0b\x54\x4e\x6d\xe3\xfd\xc4\x5f", + .rlen = 16, + }, { + .key = "\xe5\x81\x42\xdf\x05\x6a\x93\xd4" + "\x2b\x70\x85\xf5\xb6\x7d\x50\xcc", + .klen = 16, + .iv = "\xf4\x4a\xd1\x47\x49\x09\x3d\x5b" + "\x4b\xa7\xb1\x19\xb4\x46\x81\x4d", + .assoc = "\x03\x14\x5f\xaf\x8d\xa8\xe7\xe2" + "\x6b\xde\xde\x3e\xb3\x10\xb1\xcf" + "\x5c\x2d\x14\x96\x01\x78\xb9\x47" + "\xa1\x44\x19\x06\x5d\xbb\x2e\x2f", + .alen = 32, + .input = "", + .ilen = 0, + .result = "\x1b\xb1\xf1\xa8\x9e\xc2\xb2\x88" + "\x40\x7f\x7b\x19\x7a\x52\x8c\xf0", + .rlen = 16, + }, { + .key = "\x22\xa6\x7c\x7f\x15\xe6\x3c\xf0" + "\xac\x4b\x37\x86\xb0\xa2\x13\xd2", + .klen = 16, + .iv = "\x31\x6f\x0b\xe6\x59\x85\xe6\x77" + "\xcc\x81\x63\xab\xae\x6b\x43\x54", + .assoc = "\x40", + .alen = 1, + .input = "\x4f", + .ilen = 1, + .result = "\x6e\xc8\xfb\x15\x9d\x98\x49\xc9" + "\xa0\x98\x09\x85\xbe\x56\x8e\x79" + "\xf4", + .rlen = 17, + }, { + .key = "\x5e\xcb\xb6\x1e\x25\x62\xe4\x0c" + "\x2d\x25\xe9\x18\xaa\xc6\xd5\xd8", + .klen = 16, + .iv = "\x6d\x94\x44\x86\x69\x00\x8f\x93" + "\x4d\x5b\x15\x3c\xa8\x8f\x06\x5a", + .assoc = "\x7c\x5d\xd3\xee\xad\x9f\x39\x1a" + "\x6d\x92\x42\x61\xa7\x58\x37", + .alen = 15, + .input = "\x8b\x26\x61\x55\xf1\x3e\xe3\xa1" + "\x8d\xc8\x6e\x85\xa5\x21\x67", + .ilen = 15, + .result = "\x99\x2e\x84\x50\x64\x5c\xab\x29" + "\x20\xba\xb9\x2f\x62\x3a\xce\x2a" + "\x75\x25\x3b\xe3\x40\xe0\x1d\xfc" + "\x20\x63\x0b\x49\x7e\x97\x08", + .rlen = 31, + }, { + .key = "\x9b\xef\xf0\xbd\x35\xdd\x8d\x28" + "\xad\xff\x9b\xa9\xa4\xeb\x98\xdf", + .klen = 16, + .iv = "\xaa\xb8\x7e\x25\x79\x7c\x37\xaf" + "\xce\x36\xc7\xce\xa2\xb4\xc9\x60", + .assoc = "\xb9\x82\x0c\x8d\xbd\x1b\xe2\x36" + "\xee\x6c\xf4\xf2\xa1\x7d\xf9\xe2", + .alen = 16, + .input = "\xc8\x4b\x9b\xf5\x01\xba\x8c\xbd" + "\x0e\xa3\x21\x16\x9f\x46\x2a\x63", + .ilen = 16, + .result = "\xd9\x8e\xfd\x50\x8f\x02\x9f\xee" + "\x78\x08\x12\xec\x09\xaf\x53\x14" + "\x90\x3e\x3d\x76\xad\x71\x21\x08" + "\x77\xe5\x4b\x15\xc2\xe6\xbc\xdb", + .rlen = 32, + }, { + .key = "\xd7\x14\x29\x5d\x45\x59\x36\x44" + "\x2e\xd9\x4d\x3b\x9e\x0f\x5b\xe5", + .klen = 16, + .iv = "\xe6\xdd\xb8\xc4\x89\xf8\xe0\xca" + "\x4f\x10\x7a\x5f\x9c\xd8\x8b\x66", + .assoc = "\xf5\xa6\x46\x2c\xce\x97\x8a\x51" + "\x6f\x46\xa6\x83\x9b\xa1\xbc\xe8" + "\x05", + .alen = 17, + .input = "\x05\x70\xd5\x94\x12\x36\x35\xd8" + "\x8f\x7d\xd3\xa8\x99\x6a\xed\x69" + "\xd0", + .ilen = 17, + .result = "\xf3\xe7\x95\x86\xcf\x34\x95\x96" + "\x17\xfe\x1b\xae\x1b\x31\xf2\x1a" + "\xbd\xbc\xc9\x4e\x11\x29\x09\x5c" + "\x05\xd3\xb4\x2e\x4a\x74\x59\x49" + "\x7d", + .rlen = 33, + }, { + .key = "\x14\x39\x63\xfc\x56\xd5\xdf\x5f" + "\xaf\xb3\xff\xcc\x98\x33\x1d\xeb", + .klen = 16, + .iv = "\x23\x02\xf1\x64\x9a\x73\x89\xe6" + "\xd0\xea\x2c\xf1\x96\xfc\x4e\x6d", + .assoc = "\x32\xcb\x80\xcc\xde\x12\x33\x6d" + "\xf0\x20\x58\x15\x95\xc6\x7f\xee" + "\x2f\xf9\x4e\x2c\x1b\x98\x43\xc7" + "\x68\x28\x73\x40\x9f\x96\x4a", + .alen = 31, + .input = "\x41\x94\x0e\x33\x22\xb1\xdd\xf4" + "\x10\x57\x85\x39\x93\x8f\xaf\x70" + "\xfa\xa9\xd0\x4d\x5c\x40\x23\xcd" + "\x98\x34\xab\x37\x56\xae\x32", + .ilen = 31, + .result = "\x06\x96\xb2\xbf\x63\xf4\x1e\x24" + "\x0d\x19\x15\x61\x65\x3b\x06\x26" + "\x71\xe8\x7e\x16\xdb\x96\x01\x01" + "\x52\xcd\x49\x5b\x07\x33\x4e\xe7" + "\xaa\x91\xf5\xd5\xc6\xfe\x41\xb5" + "\xed\x90\xce\xb9\xcd\xcc\xa1", + .rlen = 47, + }, { + .key = "\x50\x5d\x9d\x9b\x66\x50\x88\x7b" + "\x30\x8e\xb1\x5e\x92\x58\xe0\xf1", + .klen = 16, + .iv = "\x5f\x27\x2b\x03\xaa\xef\x32\x02" + "\x50\xc4\xde\x82\x90\x21\x11\x73", + .assoc = "\x6e\xf0\xba\x6b\xee\x8e\xdc\x89" + "\x71\xfb\x0a\xa6\x8f\xea\x41\xf4" + "\x5a\xbb\x59\xb0\x20\x38\xc5\xe0" + "\x29\x56\x52\x19\x79\xf5\xe9\x37", + .alen = 32, + .input = "\x7e\xb9\x48\xd3\x32\x2d\x86\x10" + "\x91\x31\x37\xcb\x8d\xb3\x72\x76" + "\x24\x6b\xdc\xd1\x61\xe0\xa5\xe7" + "\x5a\x61\x8a\x0f\x30\x0d\xd1\xec", + .ilen = 32, + .result = "\xf9\xd7\xee\x17\xfd\x24\xcd\xf1" + "\xbc\x0f\x35\x97\x97\x0c\x4b\x18" + "\xce\x58\xc8\x3b\xd4\x85\x93\x79" + "\xcc\x9c\xea\xc1\x73\x13\x0b\x4c" + "\xcc\x6f\x28\xf8\xa4\x4e\xb8\x56" + "\x64\x4e\x47\xce\xb2\xb4\x92\xb4", + .rlen = 48, + }, { + .key = "\x8d\x82\xd6\x3b\x76\xcc\x30\x97" + "\xb1\x68\x63\xef\x8c\x7c\xa3\xf7", + .klen = 16, + .iv = "\x9c\x4b\x65\xa2\xba\x6b\xdb\x1e" + "\xd1\x9e\x90\x13\x8a\x45\xd3\x79", + .assoc = "\xab\x14\xf3\x0a\xfe\x0a\x85\xa5" + "\xf2\xd5\xbc\x38\x89\x0e\x04\xfb" + "\x84\x7d\x65\x34\x25\xd8\x47\xfa" + "\xeb\x83\x31\xf1\x54\x54\x89\x0d" + "\x9d", + .alen = 33, + .input = "\xba\xde\x82\x72\x42\xa9\x2f\x2c" + "\x12\x0b\xe9\x5c\x87\xd7\x35\x7c" + "\x4f\x2e\xe8\x55\x66\x80\x27\x00" + "\x1b\x8f\x68\xe7\x0a\x6c\x71\xc3" + "\x21\x78\x55\x9d\x9c\x65\x7b\xcd" + "\x0a\x34\x97\xff\x47\x37\xb0\x2a" + "\x80\x0d\x19\x98\x33\xa9\x7a\xe3" + "\x2e\x4c\xc6\xf3\x8c\x88\x42\x01" + "\xbd", + .ilen = 65, + .result = "\x58\xfa\x3a\x3d\xd9\x88\x63\xe8" + "\xc5\x78\x50\x8b\x4a\xc9\xdf\x7f" + "\x4b\xfa\xc8\x2e\x67\x43\xf3\x63" + "\x42\x8e\x99\x5a\x9c\x0b\x84\x77" + "\xbc\x46\x76\x48\x82\xc7\x57\x96" + "\xe1\x65\xd1\xed\x1d\xdd\x80\x24" + "\xa6\x4d\xa9\xf1\x53\x8b\x5e\x0e" + "\x26\xb9\xcc\x37\xe5\x43\xe1\x5a" + "\x8a\xd6\x8c\x5a\xe4\x95\xd1\x8d" + "\xf7\x33\x64\xc1\xd3\xf2\xfc\x35" + "\x01", + .rlen = 81, + }, { + .key = "\xc9\xa7\x10\xda\x86\x48\xd9\xb3" + "\x32\x42\x15\x80\x85\xa1\x65\xfe", + .klen = 16, + .iv = "\xd8\x70\x9f\x42\xca\xe6\x83\x3a" + "\x52\x79\x42\xa5\x84\x6a\x96\x7f", + .assoc = "\xe8\x39\x2d\xaa\x0e\x85\x2d\xc1" + "\x72\xaf\x6e\xc9\x82\x33\xc7\x01" + "\xaf\x40\x70\xb8\x2a\x78\xc9\x14" + "\xac\xb1\x10\xca\x2e\xb3\x28\xe4" + "\xac\xfa\x58\x7f\xe5\x73\x09\x8c" + "\x1d\x40\x87\x8c\xd9\x75\xc0\x55" + "\xa2\xda\x07\xd1\xc2\xa9\xd1\xbb" + "\x09\x4f\x77\x62\x88\x2d\xf2\x68" + "\x54", + .alen = 65, + .input = "\xf7\x02\xbb\x11\x52\x24\xd8\x48" + "\x93\xe6\x9b\xee\x81\xfc\xf7\x82" + "\x79\xf0\xf3\xd9\x6c\x20\xa9\x1a" + "\xdc\xbc\x47\xc0\xe4\xcb\x10\x99" + "\x2f", + .ilen = 33, + .result = "\x4c\xa9\xac\x71\xed\x10\xa6\x24" + "\xb7\xa7\xdf\x8b\xf5\xc2\x41\xcb" + "\x05\xc9\xd6\x97\xb6\x10\x7f\x17" + "\xc2\xc0\x93\xcf\xe0\x94\xfd\x99" + "\xf2\x62\x25\x28\x01\x23\x6f\x8b" + "\x04\x52\xbc\xb0\x3e\x66\x52\x90" + "\x9f", + .rlen = 49, + }, { + .key = "\x06\xcc\x4a\x79\x96\xc3\x82\xcf" + "\xb3\x1c\xc7\x12\x7f\xc5\x28\x04", + .klen = 16, + .iv = "\x15\x95\xd8\xe1\xda\x62\x2c\x56" + "\xd3\x53\xf4\x36\x7e\x8e\x59\x85", + .assoc = "\x24\x5e\x67\x49\x1e\x01\xd6\xdd" + "\xf3\x89\x20\x5b\x7c\x57\x89\x07", + .alen = 16, + .input = "\x33\x27\xf5\xb1\x62\xa0\x80\x63" + "\x14\xc0\x4d\x7f\x7b\x20\xba\x89", + .ilen = 16, + .result = "\x6d\xed\x04\x7a\x2f\x0c\x30\xa5" + "\x96\xe6\x97\xe4\x10\xeb\x40\x95" + "\xc5\x9a\xdf\x31\xd5\xa5\xa6\xec" + "\x05\xa8\x31\x50\x11\x19\x44", + .rlen = 31, + }, { + .key = "\x42\xf0\x84\x19\xa6\x3f\x2b\xea" + "\x34\xf6\x79\xa3\x79\xe9\xeb\x0a", + .klen = 16, + .iv = "\x51\xb9\x12\x80\xea\xde\xd5\x71" + "\x54\x2d\xa6\xc8\x78\xb2\x1b\x8c", + .assoc = "\x61\x83\xa0\xe8\x2e\x7d\x7f\xf8" + "\x74\x63\xd2\xec\x76\x7c\x4c\x0d", + .alen = 16, + .input = "\x70\x4c\x2f\x50\x72\x1c\x29\x7f" + "\x95\x9a\xff\x10\x75\x45\x7d\x8f", + .ilen = 16, + .result = "\x30\x95\x7d\xea\xdc\x62\xc0\x88" + "\xa1\xe3\x8d\x8c\xac\x04\x10\xa7" + "\xfa\xfa\x07\xbd\xa0\xf0\x36\xeb" + "\x21\x93\x2e\x31\x84\x83", + .rlen = 30, + }, { + .key = "\x7f\x15\xbd\xb8\xb6\xba\xd3\x06" + "\xb5\xd1\x2b\x35\x73\x0e\xad\x10", + .klen = 16, + .iv = "\x8e\xde\x4c\x20\xfa\x59\x7e\x8d" + "\xd5\x07\x58\x59\x72\xd7\xde\x92", + .assoc = "\x9d\xa7\xda\x88\x3e\xf8\x28\x14" + "\xf5\x3e\x85\x7d\x70\xa0\x0f\x13", + .alen = 16, + .input = "\xac\x70\x69\xef\x82\x97\xd2\x9b" + "\x15\x74\xb1\xa2\x6f\x69\x3f\x95", + .ilen = 16, + .result = "\x93\xcd\xee\xd4\xcb\x9d\x8d\x16" + "\x63\x0d\x43\xd5\x49\xca\xa8\x85" + "\x49\xc0\xae\x13\xbc\x26\x1d\x4b", + .rlen = 24, }, }; -static const struct cipher_testvec camellia_cbc_dec_tv_template[] = { +static const struct aead_testvec aegis128l_dec_tv_template[] = { { - .key = "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" - "\x51\x2e\x03\xd5\x34\x12\x00\x06", - .klen = 16, - .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" - "\xb4\x22\xda\x80\x2c\x9f\xac\x41", - .input = "\xea\x32\x12\x76\x3b\x50\x10\xe7" - "\x18\xf6\xfd\x5d\xf6\x8f\x13\x51", - .ilen = 16, - .result = "Single block msg", - .rlen = 16, + .key = "\x0f\xc9\x8e\x67\x44\x9e\xaa\x86" + "\x20\x36\x2c\x24\xfe\xc9\x30\x81", + .klen = 16, + .iv = "\x1e\x92\x1c\xcf\x88\x3d\x54\x0d" + "\x40\x6d\x59\x48\xfc\x92\x61\x03", + .assoc = "", + .alen = 0, + .input = "\x30\x4f\xf3\xe9\xb1\xfa\x81\xa6" + "\x20\x72\x78\xdd\x93\xc8\x57\xef", + .ilen = 16, + .result = "", + .rlen = 0, }, { - .key = "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" - "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", - .klen = 16, - .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" - "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", - .input = "\xa5\xdf\x6e\x50\xda\x70\x6c\x01" - "\x4a\xab\xf3\xf2\xd6\xfc\x6c\xfd" - "\x19\xb4\x3e\x57\x1c\x02\x5e\xa0" - "\x15\x78\xe0\x5e\xf2\xcb\x87\x16", - .ilen = 32, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", - .rlen = 32, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .key = "\x4b\xed\xc8\x07\x54\x1a\x52\xa2" + "\xa1\x10\xde\xb5\xf8\xed\xf3\x87", + .klen = 16, + .iv = "\x5a\xb7\x56\x6e\x98\xb9\xfd\x29" + "\xc1\x47\x0b\xda\xf6\xb6\x23\x09", + .assoc = "", + .alen = 0, + .input = "\xa9\x24\xa0\xb6\x2d\xdd\x29\xdb" + "\x40\xb3\x71\xc5\x22\x58\x31\x77" + "\x6d", + .ilen = 17, + .result = "\x79", + .rlen = 1, + }, { + .key = "\x88\x12\x01\xa6\x64\x96\xfb\xbe" + "\x22\xea\x90\x47\xf2\x11\xb5\x8e", + .klen = 16, + .iv = "\x97\xdb\x90\x0e\xa8\x35\xa5\x45" + "\x42\x21\xbd\x6b\xf0\xda\xe6\x0f", + .assoc = "", + .alen = 0, + .input = "\xbb\x0a\x53\xc4\xaa\x7e\xa4\x03" + "\x2b\xee\x62\x99\x7b\x98\x13\x1f" + "\xe0\x76\x4c\x2e\x53\x99\x4f\xbe" + "\xe1\xa8\x04\x7f\xe1\x71\xbe", + .ilen = 31, + .result = "\xb5\x6e\xad\xdd\x30\x72\xfa\x53" + "\x82\x8e\x16\xb4\xed\x6d\x47", + .rlen = 15, + }, { + .key = "\xc4\x37\x3b\x45\x74\x11\xa4\xda" + "\xa2\xc5\x42\xd8\xec\x36\x78\x94", + .klen = 16, + .iv = "\xd3\x00\xc9\xad\xb8\xb0\x4e\x61" + "\xc3\xfb\x6f\xfd\xea\xff\xa9\x15", + .assoc = "", + .alen = 0, + .input = "\x66\xdf\x6e\x71\xc0\x6e\xa4\x4c" + "\x9d\xb7\x8c\x9a\xdb\x1f\xd2\x2e" + "\x23\xb6\xa4\xfb\xd3\x86\xdd\xbb" + "\xde\x54\x9b\xf5\x92\x8b\x93\xc5", + .ilen = 32, + .result = "\xf2\x92\xe6\x7d\x40\xee\xa3\x6f" + "\x03\x68\xc8\x45\xe7\x91\x0a\x18", + .rlen = 16, + }, { + .key = "\x01\x5c\x75\xe5\x84\x8d\x4d\xf6" + "\x23\x9f\xf4\x6a\xe6\x5a\x3b\x9a", + .klen = 16, + .iv = "\x10\x25\x03\x4c\xc8\x2c\xf7\x7d" + "\x44\xd5\x21\x8e\xe4\x23\x6b\x1c", + .assoc = "", + .alen = 0, + .input = "\x4f\xc3\x69\xb6\xd3\xa4\x64\x8b" + "\x71\xc3\x8a\x91\x22\x4f\x1b\xd2" + "\x33\x6d\x86\xbc\xf8\x2f\x06\xf9" + "\x82\x64\xc7\x72\x00\x30\xfc\xf0" + "\xf8", + .ilen = 33, + .result = "\x2e\xb7\x20\x1c\x50\x6a\x4b\x8b" + "\x84\x42\x7a\xd7\xe1\xb5\xcd\x1f" + "\xd3", + .rlen = 17, + }, { + .key = "\x3d\x80\xae\x84\x94\x09\xf6\x12" + "\xa4\x79\xa6\xfb\xe0\x7f\xfd\xa0", + .klen = 16, + .iv = "\x4c\x49\x3d\xec\xd8\xa8\xa0\x98" + "\xc5\xb0\xd3\x1f\xde\x48\x2e\x22", + .assoc = "", + .alen = 0, + .input = "\xe3\x93\x15\xae\x5f\x9d\x3c\xb5" + "\xd6\x9d\xee\xee\xcf\xaa\xaf\xe1" + "\x45\x10\x96\xe0\xbf\x55\x0f\x4c" + "\x1a\xfd\xf4\xda\x4e\x10\xde\xc9" + "\x0e\x6f\xc7\x3c\x49\x94\x41\xfc" + "\x59\x28\x88\x3c\x79\x10\x6b", + .ilen = 47, + .result = "\x6b\xdc\x5a\xbb\x60\xe5\xf4\xa6" + "\x05\x1d\x2c\x68\xdb\xda\x8f\x25" + "\xfe\x8d\x45\x19\x1e\xc0\x0b\x99" + "\x88\x11\x39\x12\x1c\x3a\xbb", + .rlen = 31, + }, { + .key = "\x7a\xa5\xe8\x23\xa4\x84\x9e\x2d" + "\x25\x53\x58\x8c\xda\xa3\xc0\xa6", + .klen = 16, + .iv = "\x89\x6e\x77\x8b\xe8\x23\x49\xb4" + "\x45\x8a\x85\xb1\xd8\x6c\xf1\x28", + .assoc = "", + .alen = 0, + .input = "\x1c\x8e\x22\x34\xfd\xab\xe6\x0d" + "\x1c\x9f\x06\x54\x8b\x0b\xb4\x40" + "\xde\x11\x59\x3e\xfd\x74\xf6\x42" + "\x97\x17\xf7\x24\xb6\x7e\xc4\xc6" + "\x06\xa3\x94\xda\x3d\x7f\x55\x0a" + "\x92\x07\x2f\xa6\xf3\x6b\x2c\xfc", + .ilen = 48, + .result = "\xa7\x00\x93\x5b\x70\x61\x9d\xc2" + "\x86\xf7\xde\xfa\xd5\xfe\x52\x2b" + "\x28\x50\x51\x9d\x24\x60\x8d\xb3" + "\x49\x3e\x17\xea\xf6\x99\x5a\xdd", + .rlen = 32, + }, { + .key = "\xb6\xca\x22\xc3\xb4\x00\x47\x49" + "\xa6\x2d\x0a\x1e\xd4\xc7\x83\xad", + .klen = 16, + .iv = "\xc5\x93\xb0\x2a\xf8\x9f\xf1\xd0" + "\xc6\x64\x37\x42\xd2\x90\xb3\x2e", + .assoc = "\xd5", + .alen = 1, + .input = "\xa0\x2a\xb4\x9a\x91\x00\x15\xb8" + "\x0f\x9a\x15\x60\x0e\x9b\x13\x8f", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xf3\xee\x5c\x62\xc4\x7c\xf0\x65" + "\x27\x08\xbd\xaf\xce\xec\x45\xb3", + .klen = 16, + .iv = "\x02\xb8\xea\xca\x09\x1b\x9a\xec" + "\x47\x3e\xe9\xd4\xcc\xb5\x76\x34", + .assoc = "\x11\x81\x78\x32\x4d\xb9\x44\x73" + "\x68\x75\x16\xf8\xcb\x7e\xa7", + .alen = 15, + .input = "\x4c\x26\xad\x9c\x14\xfd\x9c\x8c" + "\x84\xfb\x26\xfb\xd5\xca\x62\x39", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x2f\x13\x95\x01\xd5\xf7\x99\x81" + "\xa8\xe2\x6f\x41\xc8\x10\x08\xb9", + .klen = 16, + .iv = "\x3f\xdc\x24\x69\x19\x96\x43\x08" + "\xc8\x18\x9b\x65\xc6\xd9\x39\x3b", + .assoc = "\x4e\xa5\xb2\xd1\x5d\x35\xed\x8f" + "\xe8\x4f\xc8\x89\xc5\xa2\x69\xbc", + .alen = 16, + .input = "\x45\x85\x0e\x0f\xf4\xae\x96\xa1" + "\x99\x4d\x6d\xb4\x67\x32\xb0\x3a", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x6c\x38\xcf\xa1\xe5\x73\x41\x9d" + "\x29\xbc\x21\xd2\xc2\x35\xcb\xbf", + .klen = 16, + .iv = "\x7b\x01\x5d\x08\x29\x12\xec\x24" + "\x49\xf3\x4d\xf7\xc0\xfe\xfb\x41", + .assoc = "\x8a\xca\xec\x70\x6d\xb1\x96\xab" + "\x69\x29\x7a\x1b\xbf\xc7\x2c\xc2" + "\x07", + .alen = 17, + .input = "\x33\xb1\x42\x97\x8e\x16\x7b\x63" + "\x06\xba\x5b\xcb\xae\x6d\x8b\x56", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xa8\x5c\x09\x40\xf5\xef\xea\xb8" + "\xaa\x96\xd3\x64\xbc\x59\x8d\xc6", + .klen = 16, + .iv = "\xb8\x26\x97\xa8\x39\x8e\x94\x3f" + "\xca\xcd\xff\x88\xba\x22\xbe\x47", + .assoc = "\xc7\xef\x26\x10\x7d\x2c\x3f\xc6" + "\xea\x03\x2c\xac\xb9\xeb\xef\xc9" + "\x31\x6b\x08\x12\xfc\xd8\x37\x2d" + "\xe0\x17\x3a\x2e\x83\x5c\x8f", + .alen = 31, + .input = "\xda\x44\x08\x8c\x2a\xa5\x07\x35" + "\x0b\x54\x4e\x6d\xe3\xfd\xc4\x5f", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xe5\x81\x42\xdf\x05\x6a\x93\xd4" + "\x2b\x70\x85\xf5\xb6\x7d\x50\xcc", + .klen = 16, + .iv = "\xf4\x4a\xd1\x47\x49\x09\x3d\x5b" + "\x4b\xa7\xb1\x19\xb4\x46\x81\x4d", + .assoc = "\x03\x14\x5f\xaf\x8d\xa8\xe7\xe2" + "\x6b\xde\xde\x3e\xb3\x10\xb1\xcf" + "\x5c\x2d\x14\x96\x01\x78\xb9\x47" + "\xa1\x44\x19\x06\x5d\xbb\x2e\x2f", + .alen = 32, + .input = "\x1b\xb1\xf1\xa8\x9e\xc2\xb2\x88" + "\x40\x7f\x7b\x19\x7a\x52\x8c\xf0", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x22\xa6\x7c\x7f\x15\xe6\x3c\xf0" + "\xac\x4b\x37\x86\xb0\xa2\x13\xd2", + .klen = 16, + .iv = "\x31\x6f\x0b\xe6\x59\x85\xe6\x77" + "\xcc\x81\x63\xab\xae\x6b\x43\x54", + .assoc = "\x40", + .alen = 1, + .input = "\x6e\xc8\xfb\x15\x9d\x98\x49\xc9" + "\xa0\x98\x09\x85\xbe\x56\x8e\x79" + "\xf4", + .ilen = 17, + .result = "\x4f", + .rlen = 1, + }, { + .key = "\x5e\xcb\xb6\x1e\x25\x62\xe4\x0c" + "\x2d\x25\xe9\x18\xaa\xc6\xd5\xd8", + .klen = 16, + .iv = "\x6d\x94\x44\x86\x69\x00\x8f\x93" + "\x4d\x5b\x15\x3c\xa8\x8f\x06\x5a", + .assoc = "\x7c\x5d\xd3\xee\xad\x9f\x39\x1a" + "\x6d\x92\x42\x61\xa7\x58\x37", + .alen = 15, + .input = "\x99\x2e\x84\x50\x64\x5c\xab\x29" + "\x20\xba\xb9\x2f\x62\x3a\xce\x2a" + "\x75\x25\x3b\xe3\x40\xe0\x1d\xfc" + "\x20\x63\x0b\x49\x7e\x97\x08", + .ilen = 31, + .result = "\x8b\x26\x61\x55\xf1\x3e\xe3\xa1" + "\x8d\xc8\x6e\x85\xa5\x21\x67", + .rlen = 15, + }, { + .key = "\x9b\xef\xf0\xbd\x35\xdd\x8d\x28" + "\xad\xff\x9b\xa9\xa4\xeb\x98\xdf", + .klen = 16, + .iv = "\xaa\xb8\x7e\x25\x79\x7c\x37\xaf" + "\xce\x36\xc7\xce\xa2\xb4\xc9\x60", + .assoc = "\xb9\x82\x0c\x8d\xbd\x1b\xe2\x36" + "\xee\x6c\xf4\xf2\xa1\x7d\xf9\xe2", + .alen = 16, + .input = "\xd9\x8e\xfd\x50\x8f\x02\x9f\xee" + "\x78\x08\x12\xec\x09\xaf\x53\x14" + "\x90\x3e\x3d\x76\xad\x71\x21\x08" + "\x77\xe5\x4b\x15\xc2\xe6\xbc\xdb", + .ilen = 32, + .result = "\xc8\x4b\x9b\xf5\x01\xba\x8c\xbd" + "\x0e\xa3\x21\x16\x9f\x46\x2a\x63", + .rlen = 16, + }, { + .key = "\xd7\x14\x29\x5d\x45\x59\x36\x44" + "\x2e\xd9\x4d\x3b\x9e\x0f\x5b\xe5", + .klen = 16, + .iv = "\xe6\xdd\xb8\xc4\x89\xf8\xe0\xca" + "\x4f\x10\x7a\x5f\x9c\xd8\x8b\x66", + .assoc = "\xf5\xa6\x46\x2c\xce\x97\x8a\x51" + "\x6f\x46\xa6\x83\x9b\xa1\xbc\xe8" + "\x05", + .alen = 17, + .input = "\xf3\xe7\x95\x86\xcf\x34\x95\x96" + "\x17\xfe\x1b\xae\x1b\x31\xf2\x1a" + "\xbd\xbc\xc9\x4e\x11\x29\x09\x5c" + "\x05\xd3\xb4\x2e\x4a\x74\x59\x49" + "\x7d", + .ilen = 33, + .result = "\x05\x70\xd5\x94\x12\x36\x35\xd8" + "\x8f\x7d\xd3\xa8\x99\x6a\xed\x69" + "\xd0", + .rlen = 17, + }, { + .key = "\x14\x39\x63\xfc\x56\xd5\xdf\x5f" + "\xaf\xb3\xff\xcc\x98\x33\x1d\xeb", + .klen = 16, + .iv = "\x23\x02\xf1\x64\x9a\x73\x89\xe6" + "\xd0\xea\x2c\xf1\x96\xfc\x4e\x6d", + .assoc = "\x32\xcb\x80\xcc\xde\x12\x33\x6d" + "\xf0\x20\x58\x15\x95\xc6\x7f\xee" + "\x2f\xf9\x4e\x2c\x1b\x98\x43\xc7" + "\x68\x28\x73\x40\x9f\x96\x4a", + .alen = 31, + .input = "\x06\x96\xb2\xbf\x63\xf4\x1e\x24" + "\x0d\x19\x15\x61\x65\x3b\x06\x26" + "\x71\xe8\x7e\x16\xdb\x96\x01\x01" + "\x52\xcd\x49\x5b\x07\x33\x4e\xe7" + "\xaa\x91\xf5\xd5\xc6\xfe\x41\xb5" + "\xed\x90\xce\xb9\xcd\xcc\xa1", + .ilen = 47, + .result = "\x41\x94\x0e\x33\x22\xb1\xdd\xf4" + "\x10\x57\x85\x39\x93\x8f\xaf\x70" + "\xfa\xa9\xd0\x4d\x5c\x40\x23\xcd" + "\x98\x34\xab\x37\x56\xae\x32", + .rlen = 31, + }, { + .key = "\x50\x5d\x9d\x9b\x66\x50\x88\x7b" + "\x30\x8e\xb1\x5e\x92\x58\xe0\xf1", + .klen = 16, + .iv = "\x5f\x27\x2b\x03\xaa\xef\x32\x02" + "\x50\xc4\xde\x82\x90\x21\x11\x73", + .assoc = "\x6e\xf0\xba\x6b\xee\x8e\xdc\x89" + "\x71\xfb\x0a\xa6\x8f\xea\x41\xf4" + "\x5a\xbb\x59\xb0\x20\x38\xc5\xe0" + "\x29\x56\x52\x19\x79\xf5\xe9\x37", + .alen = 32, + .input = "\xf9\xd7\xee\x17\xfd\x24\xcd\xf1" + "\xbc\x0f\x35\x97\x97\x0c\x4b\x18" + "\xce\x58\xc8\x3b\xd4\x85\x93\x79" + "\xcc\x9c\xea\xc1\x73\x13\x0b\x4c" + "\xcc\x6f\x28\xf8\xa4\x4e\xb8\x56" + "\x64\x4e\x47\xce\xb2\xb4\x92\xb4", + .ilen = 48, + .result = "\x7e\xb9\x48\xd3\x32\x2d\x86\x10" + "\x91\x31\x37\xcb\x8d\xb3\x72\x76" + "\x24\x6b\xdc\xd1\x61\xe0\xa5\xe7" + "\x5a\x61\x8a\x0f\x30\x0d\xd1\xec", + .rlen = 32, + }, { + .key = "\x8d\x82\xd6\x3b\x76\xcc\x30\x97" + "\xb1\x68\x63\xef\x8c\x7c\xa3\xf7", + .klen = 16, + .iv = "\x9c\x4b\x65\xa2\xba\x6b\xdb\x1e" + "\xd1\x9e\x90\x13\x8a\x45\xd3\x79", + .assoc = "\xab\x14\xf3\x0a\xfe\x0a\x85\xa5" + "\xf2\xd5\xbc\x38\x89\x0e\x04\xfb" + "\x84\x7d\x65\x34\x25\xd8\x47\xfa" + "\xeb\x83\x31\xf1\x54\x54\x89\x0d" + "\x9d", + .alen = 33, + .input = "\x58\xfa\x3a\x3d\xd9\x88\x63\xe8" + "\xc5\x78\x50\x8b\x4a\xc9\xdf\x7f" + "\x4b\xfa\xc8\x2e\x67\x43\xf3\x63" + "\x42\x8e\x99\x5a\x9c\x0b\x84\x77" + "\xbc\x46\x76\x48\x82\xc7\x57\x96" + "\xe1\x65\xd1\xed\x1d\xdd\x80\x24" + "\xa6\x4d\xa9\xf1\x53\x8b\x5e\x0e" + "\x26\xb9\xcc\x37\xe5\x43\xe1\x5a" + "\x8a\xd6\x8c\x5a\xe4\x95\xd1\x8d" + "\xf7\x33\x64\xc1\xd3\xf2\xfc\x35" + "\x01", + .ilen = 81, + .result = "\xba\xde\x82\x72\x42\xa9\x2f\x2c" + "\x12\x0b\xe9\x5c\x87\xd7\x35\x7c" + "\x4f\x2e\xe8\x55\x66\x80\x27\x00" + "\x1b\x8f\x68\xe7\x0a\x6c\x71\xc3" + "\x21\x78\x55\x9d\x9c\x65\x7b\xcd" + "\x0a\x34\x97\xff\x47\x37\xb0\x2a" + "\x80\x0d\x19\x98\x33\xa9\x7a\xe3" + "\x2e\x4c\xc6\xf3\x8c\x88\x42\x01" + "\xbd", + .rlen = 65, + }, { + .key = "\xc9\xa7\x10\xda\x86\x48\xd9\xb3" + "\x32\x42\x15\x80\x85\xa1\x65\xfe", + .klen = 16, + .iv = "\xd8\x70\x9f\x42\xca\xe6\x83\x3a" + "\x52\x79\x42\xa5\x84\x6a\x96\x7f", + .assoc = "\xe8\x39\x2d\xaa\x0e\x85\x2d\xc1" + "\x72\xaf\x6e\xc9\x82\x33\xc7\x01" + "\xaf\x40\x70\xb8\x2a\x78\xc9\x14" + "\xac\xb1\x10\xca\x2e\xb3\x28\xe4" + "\xac\xfa\x58\x7f\xe5\x73\x09\x8c" + "\x1d\x40\x87\x8c\xd9\x75\xc0\x55" + "\xa2\xda\x07\xd1\xc2\xa9\xd1\xbb" + "\x09\x4f\x77\x62\x88\x2d\xf2\x68" + "\x54", + .alen = 65, + .input = "\x4c\xa9\xac\x71\xed\x10\xa6\x24" + "\xb7\xa7\xdf\x8b\xf5\xc2\x41\xcb" + "\x05\xc9\xd6\x97\xb6\x10\x7f\x17" + "\xc2\xc0\x93\xcf\xe0\x94\xfd\x99" + "\xf2\x62\x25\x28\x01\x23\x6f\x8b" + "\x04\x52\xbc\xb0\x3e\x66\x52\x90" + "\x9f", + .ilen = 49, + .result = "\xf7\x02\xbb\x11\x52\x24\xd8\x48" + "\x93\xe6\x9b\xee\x81\xfc\xf7\x82" + "\x79\xf0\xf3\xd9\x6c\x20\xa9\x1a" + "\xdc\xbc\x47\xc0\xe4\xcb\x10\x99" + "\x2f", + .rlen = 33, + }, { + .key = "\x06\xcc\x4a\x79\x96\xc3\x82\xcf" + "\xb3\x1c\xc7\x12\x7f\xc5\x28\x04", + .klen = 16, + .iv = "\x15\x95\xd8\xe1\xda\x62\x2c\x56" + "\xd3\x53\xf4\x36\x7e\x8e\x59\x85", + .assoc = "\x24\x5e\x67\x49\x1e\x01\xd6\xdd" + "\xf3\x89\x20\x5b\x7c\x57\x89\x07", + .alen = 16, + .input = "\x6d\xed\x04\x7a\x2f\x0c\x30\xa5" + "\x96\xe6\x97\xe4\x10\xeb\x40\x95" + "\xc5\x9a\xdf\x31\xd5\xa5\xa6\xec" + "\x05\xa8\x31\x50\x11\x19\x44", + .ilen = 31, + .result = "\x33\x27\xf5\xb1\x62\xa0\x80\x63" + "\x14\xc0\x4d\x7f\x7b\x20\xba\x89", + .rlen = 16, + }, { + .key = "\x42\xf0\x84\x19\xa6\x3f\x2b\xea" + "\x34\xf6\x79\xa3\x79\xe9\xeb\x0a", + .klen = 16, + .iv = "\x51\xb9\x12\x80\xea\xde\xd5\x71" + "\x54\x2d\xa6\xc8\x78\xb2\x1b\x8c", + .assoc = "\x61\x83\xa0\xe8\x2e\x7d\x7f\xf8" + "\x74\x63\xd2\xec\x76\x7c\x4c\x0d", + .alen = 16, + .input = "\x30\x95\x7d\xea\xdc\x62\xc0\x88" + "\xa1\xe3\x8d\x8c\xac\x04\x10\xa7" + "\xfa\xfa\x07\xbd\xa0\xf0\x36\xeb" + "\x21\x93\x2e\x31\x84\x83", + .ilen = 30, + .result = "\x70\x4c\x2f\x50\x72\x1c\x29\x7f" + "\x95\x9a\xff\x10\x75\x45\x7d\x8f", + .rlen = 16, + }, { + .key = "\x7f\x15\xbd\xb8\xb6\xba\xd3\x06" + "\xb5\xd1\x2b\x35\x73\x0e\xad\x10", + .klen = 16, + .iv = "\x8e\xde\x4c\x20\xfa\x59\x7e\x8d" + "\xd5\x07\x58\x59\x72\xd7\xde\x92", + .assoc = "\x9d\xa7\xda\x88\x3e\xf8\x28\x14" + "\xf5\x3e\x85\x7d\x70\xa0\x0f\x13", + .alen = 16, + .input = "\x93\xcd\xee\xd4\xcb\x9d\x8d\x16" + "\x63\x0d\x43\xd5\x49\xca\xa8\x85" + "\x49\xc0\xae\x13\xbc\x26\x1d\x4b", + .ilen = 24, + .result = "\xac\x70\x69\xef\x82\x97\xd2\x9b" + "\x15\x74\xb1\xa2\x6f\x69\x3f\x95", + .rlen = 16, + }, +}; + +/* + * AEGIS-256 test vectors - generated via reference implementation from + * SUPERCOP (https://bench.cr.yp.to/supercop.html): + * + * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz + * (see crypto_aead/aegis256/) + */ +static const struct aead_testvec aegis256_enc_tv_template[] = { + { + .key = "\x0f\xc9\x8e\x67\x44\x9e\xaa\x86" + "\x20\x36\x2c\x24\xfe\xc9\x30\x81" + "\xca\xb0\x82\x21\x41\xa8\xe0\x06" + "\x30\x0b\x37\xf6\xb6\x17\xe7\xb5", + .klen = 32, + .iv = "\x1e\x92\x1c\xcf\x88\x3d\x54\x0d" + "\x40\x6d\x59\x48\xfc\x92\x61\x03" + "\x95\x61\x05\x42\x82\x50\xc0\x0c" + "\x60\x16\x6f\xec\x6d\x2f\xcf\x6b", + .assoc = "", + .alen = 0, + .input = "", + .ilen = 0, + .result = "\xd5\x65\x3a\xa9\x03\x51\xd7\xaa" + "\xfa\x4b\xd8\xa2\x41\x9b\xc1\xb2", + .rlen = 16, + }, { + .key = "\x4b\xed\xc8\x07\x54\x1a\x52\xa2" + "\xa1\x10\xde\xb5\xf8\xed\xf3\x87" + "\xf4\x72\x8e\xa5\x46\x48\x62\x20" + "\xf1\x38\x16\xce\x90\x76\x87\x8c", + .klen = 32, + .iv = "\x5a\xb7\x56\x6e\x98\xb9\xfd\x29" + "\xc1\x47\x0b\xda\xf6\xb6\x23\x09" + "\xbf\x23\x11\xc6\x87\xf0\x42\x26" + "\x22\x44\x4e\xc4\x47\x8e\x6e\x41", + .assoc = "", + .alen = 0, + .input = "\x79", + .ilen = 1, + .result = "\x84\xa2\x8f\xad\xdb\x8d\x2c\x16" + "\x9e\x89\xd9\x06\xa6\xa8\x14\x29" + "\x8b", + .rlen = 17, + }, { + .key = "\x88\x12\x01\xa6\x64\x96\xfb\xbe" + "\x22\xea\x90\x47\xf2\x11\xb5\x8e" + "\x1f\x35\x9a\x29\x4b\xe8\xe4\x39" + "\xb3\x66\xf5\xa6\x6a\xd5\x26\x62", + .klen = 32, + .iv = "\x97\xdb\x90\x0e\xa8\x35\xa5\x45" + "\x42\x21\xbd\x6b\xf0\xda\xe6\x0f" + "\xe9\xe5\x1d\x4a\x8c\x90\xc4\x40" + "\xe3\x71\x2d\x9c\x21\xed\x0e\x18", + .assoc = "", + .alen = 0, + .input = "\xb5\x6e\xad\xdd\x30\x72\xfa\x53" + "\x82\x8e\x16\xb4\xed\x6d\x47", + .ilen = 15, + .result = "\x09\x94\x1f\xa6\x13\xc3\x74\x75" + "\x17\xad\x8a\x0e\xd8\x66\x9a\x28" + "\xd7\x30\x66\x09\x2a\xdc\xfa\x2a" + "\x9f\x3b\xd7\xdd\x66\xd1\x2b", + .rlen = 31, + }, { + .key = "\xc4\x37\x3b\x45\x74\x11\xa4\xda" + "\xa2\xc5\x42\xd8\xec\x36\x78\x94" + "\x49\xf7\xa5\xad\x50\x88\x66\x53" + "\x74\x94\xd4\x7f\x44\x34\xc5\x39", + .klen = 32, + .iv = "\xd3\x00\xc9\xad\xb8\xb0\x4e\x61" + "\xc3\xfb\x6f\xfd\xea\xff\xa9\x15" + "\x14\xa8\x28\xce\x92\x30\x46\x59" + "\xa4\x9f\x0b\x75\xfb\x4c\xad\xee", + .assoc = "", + .alen = 0, + .input = "\xf2\x92\xe6\x7d\x40\xee\xa3\x6f" + "\x03\x68\xc8\x45\xe7\x91\x0a\x18", + .ilen = 16, + .result = "\x8a\x46\xa2\x22\x8c\x03\xab\x6f" + "\x54\x63\x4e\x7f\xc9\x8e\xfa\x70" + "\x7b\xe5\x8d\x78\xbc\xe9\xb6\xa1" + "\x29\x17\xc8\x3b\x52\xa4\x98\x72", + .rlen = 32, + }, { + .key = "\x01\x5c\x75\xe5\x84\x8d\x4d\xf6" + "\x23\x9f\xf4\x6a\xe6\x5a\x3b\x9a" + "\x74\xb9\xb1\x32\x55\x28\xe8\x6d" + "\x35\xc1\xb3\x57\x1f\x93\x64\x0f", + .klen = 32, + .iv = "\x10\x25\x03\x4c\xc8\x2c\xf7\x7d" + "\x44\xd5\x21\x8e\xe4\x23\x6b\x1c" + "\x3e\x6a\x34\x53\x97\xd0\xc8\x73" + "\x66\xcd\xea\x4d\xd5\xab\x4c\xc5", + .assoc = "", + .alen = 0, + .input = "\x2e\xb7\x20\x1c\x50\x6a\x4b\x8b" + "\x84\x42\x7a\xd7\xe1\xb5\xcd\x1f" + "\xd3", + .ilen = 17, + .result = "\x71\x6b\x37\x0b\x02\x61\x28\x12" + "\x83\xab\x66\x90\x84\xc7\xd1\xc5" + "\xb2\x7a\xb4\x7b\xb4\xfe\x02\xb2" + "\xc0\x00\x39\x13\xb5\x51\x68\x44" + "\xad", + .rlen = 33, + }, { + .key = "\x3d\x80\xae\x84\x94\x09\xf6\x12" + "\xa4\x79\xa6\xfb\xe0\x7f\xfd\xa0" + "\x9e\x7c\xbc\xb6\x5b\xc8\x6a\x86" + "\xf7\xef\x91\x30\xf9\xf2\x04\xe6", + .klen = 32, + .iv = "\x4c\x49\x3d\xec\xd8\xa8\xa0\x98" + "\xc5\xb0\xd3\x1f\xde\x48\x2e\x22" + "\x69\x2c\x3f\xd7\x9c\x70\x4a\x8d" + "\x27\xfa\xc9\x26\xaf\x0a\xeb\x9c", + .assoc = "", + .alen = 0, + .input = "\x6b\xdc\x5a\xbb\x60\xe5\xf4\xa6" + "\x05\x1d\x2c\x68\xdb\xda\x8f\x25" + "\xfe\x8d\x45\x19\x1e\xc0\x0b\x99" + "\x88\x11\x39\x12\x1c\x3a\xbb", + .ilen = 31, + .result = "\xaf\xa4\x34\x0d\x59\xe6\x1c\x2f" + "\x06\x3b\x52\x18\x49\x75\x1b\xf0" + "\x53\x09\x72\x7b\x45\x79\xe0\xbe" + "\x89\x85\x23\x15\xb8\x79\x07\x4c" + "\x53\x7a\x15\x37\x0a\xee\xb7\xfb" + "\xc4\x1f\x12\x27\xcf\x77\x90", + .rlen = 47, + }, { + .key = "\x7a\xa5\xe8\x23\xa4\x84\x9e\x2d" + "\x25\x53\x58\x8c\xda\xa3\xc0\xa6" + "\xc8\x3e\xc8\x3a\x60\x68\xec\xa0" + "\xb8\x1c\x70\x08\xd3\x51\xa3\xbd", + .klen = 32, + .iv = "\x89\x6e\x77\x8b\xe8\x23\x49\xb4" + "\x45\x8a\x85\xb1\xd8\x6c\xf1\x28" + "\x93\xef\x4b\x5b\xa1\x10\xcc\xa6" + "\xe8\x28\xa8\xfe\x89\x69\x8b\x72", + .assoc = "", + .alen = 0, + .input = "\xa7\x00\x93\x5b\x70\x61\x9d\xc2" + "\x86\xf7\xde\xfa\xd5\xfe\x52\x2b" + "\x28\x50\x51\x9d\x24\x60\x8d\xb3" + "\x49\x3e\x17\xea\xf6\x99\x5a\xdd", + .ilen = 32, + .result = "\xe2\xc9\x0b\x33\x31\x02\xb3\xb4" + "\x33\xfe\xeb\xa8\xb7\x9b\xb2\xd7" + "\xeb\x0f\x05\x2b\xba\xb3\xca\xef" + "\xf6\xd1\xb6\xc0\xb9\x9b\x85\xc5" + "\xbf\x7a\x3e\xcc\x31\x76\x09\x80" + "\x32\x5d\xbb\xe8\x38\x0e\x77\xd3", + .rlen = 48, + }, { + .key = "\xb6\xca\x22\xc3\xb4\x00\x47\x49" + "\xa6\x2d\x0a\x1e\xd4\xc7\x83\xad" + "\xf3\x00\xd4\xbf\x65\x08\x6e\xb9" + "\x7a\x4a\x4f\xe0\xad\xb0\x42\x93", + .klen = 32, + .iv = "\xc5\x93\xb0\x2a\xf8\x9f\xf1\xd0" + "\xc6\x64\x37\x42\xd2\x90\xb3\x2e" + "\xbd\xb1\x57\xe0\xa6\xb0\x4e\xc0" + "\xaa\x55\x87\xd6\x63\xc8\x2a\x49", + .assoc = "\xd5", + .alen = 1, + .input = "", + .ilen = 0, + .result = "\x96\x43\x30\xca\x6c\x4f\xd7\x12" + "\xba\xd9\xb3\x18\x86\xdf\xc3\x52", + .rlen = 16, + }, { + .key = "\xf3\xee\x5c\x62\xc4\x7c\xf0\x65" + "\x27\x08\xbd\xaf\xce\xec\x45\xb3" + "\x1d\xc3\xdf\x43\x6a\xa8\xf0\xd3" + "\x3b\x77\x2e\xb9\x87\x0f\xe1\x6a", + .klen = 32, + .iv = "\x02\xb8\xea\xca\x09\x1b\x9a\xec" + "\x47\x3e\xe9\xd4\xcc\xb5\x76\x34" + "\xe8\x73\x62\x64\xab\x50\xd0\xda" + "\x6b\x83\x66\xaf\x3e\x27\xc9\x1f", + .assoc = "\x11\x81\x78\x32\x4d\xb9\x44\x73" + "\x68\x75\x16\xf8\xcb\x7e\xa7", + .alen = 15, + .input = "", + .ilen = 0, + .result = "\x2f\xab\x45\xe2\xa7\x46\xc5\x83" + "\x11\x9f\xb0\x74\xee\xc7\x03\xdd", + .rlen = 16, + }, { + .key = "\x2f\x13\x95\x01\xd5\xf7\x99\x81" + "\xa8\xe2\x6f\x41\xc8\x10\x08\xb9" + "\x47\x85\xeb\xc7\x6f\x48\x72\xed" + "\xfc\xa5\x0d\x91\x61\x6e\x81\x40", + .klen = 32, + .iv = "\x3f\xdc\x24\x69\x19\x96\x43\x08" + "\xc8\x18\x9b\x65\xc6\xd9\x39\x3b" + "\x12\x35\x6e\xe8\xb0\xf0\x52\xf3" + "\x2d\xb0\x45\x87\x18\x86\x68\xf6", + .assoc = "\x4e\xa5\xb2\xd1\x5d\x35\xed\x8f" + "\xe8\x4f\xc8\x89\xc5\xa2\x69\xbc", + .alen = 16, + .input = "", + .ilen = 0, + .result = "\x16\x44\x73\x33\x5d\xf2\xb9\x04" + "\x6b\x79\x98\xef\xdb\xd5\xc5\xf1", + .rlen = 16, + }, { + .key = "\x6c\x38\xcf\xa1\xe5\x73\x41\x9d" + "\x29\xbc\x21\xd2\xc2\x35\xcb\xbf" + "\x72\x47\xf6\x4b\x74\xe8\xf4\x06" + "\xbe\xd3\xec\x6a\x3b\xcd\x20\x17", + .klen = 32, + .iv = "\x7b\x01\x5d\x08\x29\x12\xec\x24" + "\x49\xf3\x4d\xf7\xc0\xfe\xfb\x41" + "\x3c\xf8\x79\x6c\xb6\x90\xd4\x0d" + "\xee\xde\x23\x60\xf2\xe5\x08\xcc", + .assoc = "\x8a\xca\xec\x70\x6d\xb1\x96\xab" + "\x69\x29\x7a\x1b\xbf\xc7\x2c\xc2" + "\x07", + .alen = 17, + .input = "", + .ilen = 0, + .result = "\xa4\x9b\xb8\x47\xc0\xed\x7a\x45" + "\x98\x54\x8c\xed\x3d\x17\xf0\xdd", + .rlen = 16, + }, { + .key = "\xa8\x5c\x09\x40\xf5\xef\xea\xb8" + "\xaa\x96\xd3\x64\xbc\x59\x8d\xc6" + "\x9c\x0a\x02\xd0\x79\x88\x76\x20" + "\x7f\x00\xca\x42\x15\x2c\xbf\xed", + .klen = 32, + .iv = "\xb8\x26\x97\xa8\x39\x8e\x94\x3f" + "\xca\xcd\xff\x88\xba\x22\xbe\x47" + "\x67\xba\x85\xf1\xbb\x30\x56\x26" + "\xaf\x0b\x02\x38\xcc\x44\xa7\xa3", + .assoc = "\xc7\xef\x26\x10\x7d\x2c\x3f\xc6" + "\xea\x03\x2c\xac\xb9\xeb\xef\xc9" + "\x31\x6b\x08\x12\xfc\xd8\x37\x2d" + "\xe0\x17\x3a\x2e\x83\x5c\x8f", + .alen = 31, + .input = "", + .ilen = 0, + .result = "\x20\x24\xe2\x33\x5c\x60\xc9\xf0" + "\xa4\x96\x2f\x0d\x53\xc2\xf8\xfc", + .rlen = 16, + }, { + .key = "\xe5\x81\x42\xdf\x05\x6a\x93\xd4" + "\x2b\x70\x85\xf5\xb6\x7d\x50\xcc" + "\xc6\xcc\x0e\x54\x7f\x28\xf8\x3a" + "\x40\x2e\xa9\x1a\xf0\x8b\x5e\xc4", + .klen = 32, + .iv = "\xf4\x4a\xd1\x47\x49\x09\x3d\x5b" + "\x4b\xa7\xb1\x19\xb4\x46\x81\x4d" + "\x91\x7c\x91\x75\xc0\xd0\xd8\x40" + "\x71\x39\xe1\x10\xa6\xa3\x46\x7a", + .assoc = "\x03\x14\x5f\xaf\x8d\xa8\xe7\xe2" + "\x6b\xde\xde\x3e\xb3\x10\xb1\xcf" + "\x5c\x2d\x14\x96\x01\x78\xb9\x47" + "\xa1\x44\x19\x06\x5d\xbb\x2e\x2f", + .alen = 32, + .input = "", + .ilen = 0, + .result = "\x6f\x4a\xb9\xe0\xff\x51\xa3\xf1" + "\xd2\x64\x3e\x66\x6a\xb2\x03\xc0", + .rlen = 16, + }, { + .key = "\x22\xa6\x7c\x7f\x15\xe6\x3c\xf0" + "\xac\x4b\x37\x86\xb0\xa2\x13\xd2" + "\xf1\x8e\x19\xd8\x84\xc8\x7a\x53" + "\x02\x5b\x88\xf3\xca\xea\xfe\x9b", + .klen = 32, + .iv = "\x31\x6f\x0b\xe6\x59\x85\xe6\x77" + "\xcc\x81\x63\xab\xae\x6b\x43\x54" + "\xbb\x3f\x9c\xf9\xc5\x70\x5a\x5a" + "\x32\x67\xc0\xe9\x80\x02\xe5\x50", + .assoc = "\x40", + .alen = 1, + .input = "\x4f", + .ilen = 1, + .result = "\x2c\xfb\xad\x7e\xbe\xa0\x9a\x5b" + "\x7a\x3f\x81\xf7\xfc\x1b\x79\x83" + "\xc7", + .rlen = 17, + }, { + .key = "\x5e\xcb\xb6\x1e\x25\x62\xe4\x0c" + "\x2d\x25\xe9\x18\xaa\xc6\xd5\xd8" + "\x1b\x50\x25\x5d\x89\x68\xfc\x6d" + "\xc3\x89\x67\xcb\xa4\x49\x9d\x71", + .klen = 32, + .iv = "\x6d\x94\x44\x86\x69\x00\x8f\x93" + "\x4d\x5b\x15\x3c\xa8\x8f\x06\x5a" + "\xe6\x01\xa8\x7e\xca\x10\xdc\x73" + "\xf4\x94\x9f\xc1\x5a\x61\x85\x27", + .assoc = "\x7c\x5d\xd3\xee\xad\x9f\x39\x1a" + "\x6d\x92\x42\x61\xa7\x58\x37", + .alen = 15, + .input = "\x8b\x26\x61\x55\xf1\x3e\xe3\xa1" + "\x8d\xc8\x6e\x85\xa5\x21\x67", + .ilen = 15, + .result = "\x1f\x7f\xca\x3c\x2b\xe7\x27\xba" + "\x7e\x98\x83\x02\x34\x23\xf7\x94" + "\xde\x35\xe6\x1d\x14\x18\xe5\x38" + "\x14\x80\x6a\xa7\x1b\xae\x1d", + .rlen = 31, + }, { + .key = "\x9b\xef\xf0\xbd\x35\xdd\x8d\x28" + "\xad\xff\x9b\xa9\xa4\xeb\x98\xdf" + "\x46\x13\x31\xe1\x8e\x08\x7e\x87" + "\x85\xb6\x46\xa3\x7e\xa8\x3c\x48", + .klen = 32, + .iv = "\xaa\xb8\x7e\x25\x79\x7c\x37\xaf" + "\xce\x36\xc7\xce\xa2\xb4\xc9\x60" + "\x10\xc3\xb3\x02\xcf\xb0\x5e\x8d" + "\xb5\xc2\x7e\x9a\x35\xc0\x24\xfd", + .assoc = "\xb9\x82\x0c\x8d\xbd\x1b\xe2\x36" + "\xee\x6c\xf4\xf2\xa1\x7d\xf9\xe2", + .alen = 16, + .input = "\xc8\x4b\x9b\xf5\x01\xba\x8c\xbd" + "\x0e\xa3\x21\x16\x9f\x46\x2a\x63", + .ilen = 16, + .result = "\x05\x86\x9e\xd7\x2b\xa3\x97\x01" + "\xbe\x28\x98\x10\x6f\xe9\x61\x32" + "\x96\xbb\xb1\x2e\x8f\x0c\x44\xb9" + "\x46\x2d\x55\xe3\x42\x67\xf2\xaf", + .rlen = 32, + }, { + .key = "\xd7\x14\x29\x5d\x45\x59\x36\x44" + "\x2e\xd9\x4d\x3b\x9e\x0f\x5b\xe5" + "\x70\xd5\x3c\x65\x93\xa8\x00\xa0" + "\x46\xe4\x25\x7c\x58\x08\xdb\x1e", + .klen = 32, + .iv = "\xe6\xdd\xb8\xc4\x89\xf8\xe0\xca" + "\x4f\x10\x7a\x5f\x9c\xd8\x8b\x66" + "\x3b\x86\xbf\x86\xd4\x50\xe0\xa7" + "\x76\xef\x5c\x72\x0f\x1f\xc3\xd4", + .assoc = "\xf5\xa6\x46\x2c\xce\x97\x8a\x51" + "\x6f\x46\xa6\x83\x9b\xa1\xbc\xe8" + "\x05", + .alen = 17, + .input = "\x05\x70\xd5\x94\x12\x36\x35\xd8" + "\x8f\x7d\xd3\xa8\x99\x6a\xed\x69" + "\xd0", + .ilen = 17, + .result = "\x9c\xe0\x06\x7b\x86\xcf\x2e\xd8" + "\x45\x65\x1b\x72\x9b\xaa\xa3\x1e" + "\x87\x9d\x26\xdf\xff\x81\x11\xd2" + "\x47\x41\xb9\x24\xc1\x8a\xa3\x8b" + "\x55", + .rlen = 33, + }, { + .key = "\x14\x39\x63\xfc\x56\xd5\xdf\x5f" + "\xaf\xb3\xff\xcc\x98\x33\x1d\xeb" + "\x9a\x97\x48\xe9\x98\x48\x82\xba" + "\x07\x11\x04\x54\x32\x67\x7b\xf5", + .klen = 32, + .iv = "\x23\x02\xf1\x64\x9a\x73\x89\xe6" + "\xd0\xea\x2c\xf1\x96\xfc\x4e\x6d" + "\x65\x48\xcb\x0a\xda\xf0\x62\xc0" + "\x38\x1d\x3b\x4a\xe9\x7e\x62\xaa", + .assoc = "\x32\xcb\x80\xcc\xde\x12\x33\x6d" + "\xf0\x20\x58\x15\x95\xc6\x7f\xee" + "\x2f\xf9\x4e\x2c\x1b\x98\x43\xc7" + "\x68\x28\x73\x40\x9f\x96\x4a", + .alen = 31, + .input = "\x41\x94\x0e\x33\x22\xb1\xdd\xf4" + "\x10\x57\x85\x39\x93\x8f\xaf\x70" + "\xfa\xa9\xd0\x4d\x5c\x40\x23\xcd" + "\x98\x34\xab\x37\x56\xae\x32", + .ilen = 31, + .result = "\xa0\xc8\xde\x83\x0d\xc3\x4e\xd5" + "\x69\x7f\x7a\xdd\x8c\x46\xda\xba" + "\x0a\x5c\x0e\x7f\xac\xee\x02\xd2" + "\xe5\x4b\x0a\xba\xb8\xa4\x7b\x66" + "\xde\xae\xdb\xc2\xc0\x0b\xf7\x2b" + "\xdf\xb8\xea\xd8\xa9\x38\xed", + .rlen = 47, + }, { + .key = "\x50\x5d\x9d\x9b\x66\x50\x88\x7b" + "\x30\x8e\xb1\x5e\x92\x58\xe0\xf1" + "\xc5\x5a\x53\x6e\x9d\xe8\x04\xd4" + "\xc9\x3f\xe2\x2d\x0c\xc6\x1a\xcb", + .klen = 32, + .iv = "\x5f\x27\x2b\x03\xaa\xef\x32\x02" + "\x50\xc4\xde\x82\x90\x21\x11\x73" + "\x8f\x0a\xd6\x8f\xdf\x90\xe4\xda" + "\xf9\x4a\x1a\x23\xc3\xdd\x02\x81", + .assoc = "\x6e\xf0\xba\x6b\xee\x8e\xdc\x89" + "\x71\xfb\x0a\xa6\x8f\xea\x41\xf4" + "\x5a\xbb\x59\xb0\x20\x38\xc5\xe0" + "\x29\x56\x52\x19\x79\xf5\xe9\x37", + .alen = 32, + .input = "\x7e\xb9\x48\xd3\x32\x2d\x86\x10" + "\x91\x31\x37\xcb\x8d\xb3\x72\x76" + "\x24\x6b\xdc\xd1\x61\xe0\xa5\xe7" + "\x5a\x61\x8a\x0f\x30\x0d\xd1\xec", + .ilen = 32, + .result = "\xd3\x68\x14\x70\x3c\x01\x43\x86" + "\x02\xab\xbe\x75\xaa\xe7\xf5\x53" + "\x5c\x05\xbd\x9b\x19\xbb\x2a\x61" + "\x8f\x69\x05\x75\x8e\xca\x60\x0c" + "\x5b\xa2\x48\x61\x32\x74\x11\x2b" + "\xf6\xcf\x06\x78\x6f\x78\x1a\x4a", + .rlen = 48, + }, { + .key = "\x8d\x82\xd6\x3b\x76\xcc\x30\x97" + "\xb1\x68\x63\xef\x8c\x7c\xa3\xf7" + "\xef\x1c\x5f\xf2\xa3\x88\x86\xed" + "\x8a\x6d\xc1\x05\xe7\x25\xb9\xa2", + .klen = 32, + .iv = "\x9c\x4b\x65\xa2\xba\x6b\xdb\x1e" + "\xd1\x9e\x90\x13\x8a\x45\xd3\x79" + "\xba\xcd\xe2\x13\xe4\x30\x66\xf4" + "\xba\x78\xf9\xfb\x9d\x3c\xa1\x58", + .assoc = "\xab\x14\xf3\x0a\xfe\x0a\x85\xa5" + "\xf2\xd5\xbc\x38\x89\x0e\x04\xfb" + "\x84\x7d\x65\x34\x25\xd8\x47\xfa" + "\xeb\x83\x31\xf1\x54\x54\x89\x0d" + "\x9d", + .alen = 33, + .input = "\xba\xde\x82\x72\x42\xa9\x2f\x2c" + "\x12\x0b\xe9\x5c\x87\xd7\x35\x7c" + "\x4f\x2e\xe8\x55\x66\x80\x27\x00" + "\x1b\x8f\x68\xe7\x0a\x6c\x71\xc3" + "\x21\x78\x55\x9d\x9c\x65\x7b\xcd" + "\x0a\x34\x97\xff\x47\x37\xb0\x2a" + "\x80\x0d\x19\x98\x33\xa9\x7a\xe3" + "\x2e\x4c\xc6\xf3\x8c\x88\x42\x01" + "\xbd", + .ilen = 65, + .result = "\x07\x0a\x35\xb0\x82\x03\x5a\xd2" + "\x15\x3a\x6c\x72\x83\x9b\xb1\x75" + "\xea\xf2\xfc\xff\xc6\xf1\x13\xa4" + "\x1a\x93\x33\x79\x97\x82\x81\xc0" + "\x96\xc2\x00\xab\x39\xae\xa1\x62" + "\x53\xa3\x86\xc9\x07\x8c\xaf\x22" + "\x47\x31\x29\xca\x4a\x95\xf5\xd5" + "\x20\x63\x5a\x54\x80\x2c\x4a\x63" + "\xfb\x18\x73\x31\x4f\x08\x21\x5d" + "\x20\xe9\xc3\x7e\xea\x25\x77\x3a" + "\x65", + .rlen = 81, + }, { + .key = "\xc9\xa7\x10\xda\x86\x48\xd9\xb3" + "\x32\x42\x15\x80\x85\xa1\x65\xfe" + "\x19\xde\x6b\x76\xa8\x28\x08\x07" + "\x4b\x9a\xa0\xdd\xc1\x84\x58\x79", .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\xCD\x3E\x2A\x3B\x3E\x94\xC5\x77" - "\xBA\xBB\x5B\xB1\xDE\x7B\xA4\x40" - "\x88\x39\xE3\xFD\x94\x4B\x25\x58" - "\xE1\x4B\xC4\x18\x7A\xFD\x17\x2B" - "\xB9\xF9\xC2\x27\x6A\xB6\x31\x27" - "\xA6\xAD\xEF\xE5\x5D\xE4\x02\x01" - "\x56\x2E\x10\xC2\x2C\xFF\xC6\x83" - "\xB5\xDC\x4F\x63\xAD\x0E\x63\x5E" - "\x56\xC8\x18\x3D\x79\x86\x97\xEF" - "\x57\x0E\x63\xA1\xC1\x41\x48\xB8" - "\x98\xB7\x51\x6D\x18\xF6\x19\x82" - "\x37\x49\x88\xA4\xEF\x91\x21\x47" - "\x03\x28\xEA\x42\xF4\xFB\x7A\x58" - "\x28\x90\x77\x46\xD8\xD2\x35\x16" - "\x44\xA9\x9E\x49\x52\x2A\xE4\x16" - "\x5D\xF7\x65\xEB\x0F\xC9\x29\xE6" - "\xCF\x76\x91\x89\x8A\x94\x39\xFA" - "\x6B\x5F\x63\x53\x74\x43\x91\xF5" - "\x3F\xBC\x88\x53\xB2\x1A\x02\x3F" - "\x9D\x32\x84\xEB\x56\x28\xD6\x06" - "\xD5\xB2\x20\xA9\xFC\xC3\x76\x62" - "\x32\xCC\x86\xC8\x36\x67\x5E\x7E" - "\xA4\xAA\x15\x63\x6B\xA9\x86\xAF" - "\x1A\x52\x82\x36\x5F\xF4\x3F\x7A" - "\x9B\x78\x62\x3B\x02\x28\x60\xB3" - "\xBA\x82\xB1\xDD\xC9\x60\x8F\x47" - "\xF1\x6B\xFE\xE5\x39\x34\xA0\x28" - "\xA4\xB3\xC9\x7E\xED\x28\x8D\x70" - "\xB2\x1D\xFD\xC6\x00\xCF\x1A\x94" - "\x28\xF8\xC1\x34\xB7\x58\xA5\x6C" - "\x1A\x9D\xE4\xE4\xF6\xB9\xB4\xB0" - "\x5D\x51\x54\x9A\x53\xA0\xF9\x32" - "\xBD\x31\x54\x14\x7B\x33\xEE\x17" - "\xD3\xC7\x1F\x48\xBF\x0B\x22\xA2" - "\x7D\x0C\xDF\xD0\x2E\x98\xFA\xD2" - "\xFA\xCF\x24\x1D\x99\x9B\xD0\x7E" - "\xF4\x4F\x88\xFF\x45\x99\x4A\xF4" - "\xF2\x0A\x5B\x3B\x21\xAB\x92\xAE" - "\x40\x78\x91\x95\xC4\x2F\xA3\xE8" - "\x18\xC7\x07\xA6\xC8\xC0\x66\x33" - "\x35\xC0\xB4\xA0\xF8\xEE\x1E\xF3" - "\x40\xF5\x40\x54\xF1\x84\x8C\xEA" - "\x27\x38\x1F\xF8\x77\xC7\xDF\xD8" - "\x1D\xE2\xD9\x59\x40\x4F\x59\xD4" - "\xF8\x17\x99\x8D\x58\x2D\x72\x44" - "\x9D\x1D\x91\x64\xD6\x3F\x0A\x82" - "\xC7\x57\x3D\xEF\xD3\x41\xFA\xA7" - "\x68\xA3\xB8\xA5\x93\x74\x2E\x85" - "\x4C\x9D\x69\x59\xCE\x15\xAE\xBF" - "\x9C\x8F\x14\x64\x5D\x7F\xCF\x0B" - "\xCE\x43\x5D\x28\xC0\x2F\xFB\x18" - "\x79\x9A\xFC\x43\x16\x7C\x6B\x7B" - "\x38\xB8\x48\x36\x66\x4E\x20\x43" - "\xBA\x76\x13\x9A\xC3\xF2\xEB\x52" - "\xD7\xDC\xB2\x67\x63\x14\x25\xCD" - "\xB1\x13\x4B\xDE\x8C\x59\x21\x84" - "\x81\x8D\x97\x23\x45\x33\x7C\xF3" - "\xC5\xBC\x79\x95\xAA\x84\x68\x31" - "\x2D\x1A\x68\xFE\xEC\x92\x94\xDA" - "\x94\x2A\x6F\xD6\xFE\xE5\x76\x97" - "\xF4\x6E\xEE\xCB\x2B\x95\x4E\x36" - "\x5F\x74\x8C\x86\x5B\x71\xD0\x20" - "\x78\x1A\x7F\x18\x8C\xD9\xCD\xF5" - "\x21\x41\x56\x72\x13\xE1\x86\x07" - "\x07\x26\xF3\x4F\x7B\xEA\xB5\x18" - "\xFE\x94\x2D\x9F\xE0\x72\x18\x65" - "\xB2\xA5\x63\x48\xB4\x13\x22\xF7" - "\x25\xF1\x80\xA8\x7F\x54\x86\x7B" - "\x39\xAE\x95\x0C\x09\x32\x22\x2D" - "\x4D\x73\x39\x0C\x09\x2C\x7C\x10" - "\xD0\x4B\x53\xF6\x90\xC5\x99\x2F" - "\x15\xE1\x7F\xC6\xC5\x7A\x52\x14" - "\x65\xEE\x93\x54\xD0\x66\x15\x3C" - "\x4C\x68\xFD\x64\x0F\xF9\x10\x39" - "\x46\x7A\xDD\x97\x20\xEE\xC7\xD2" - "\x98\x4A\xB6\xE6\xF5\xA8\x1F\x4F" - "\xDB\xAB\x6D\xD5\x9B\x34\x16\x97" - "\x2F\x64\xE5\x37\xEF\x0E\xA1\xE9" - "\xBE\x31\x31\x96\x8B\x40\x18\x75" - "\x11\x75\x14\x32\xA5\x2D\x1B\x6B" - "\xDB\x59\xEB\xFA\x3D\x8E\x7C\xC4" - "\xDE\x68\xC8\x9F\xC9\x99\xE3\xC6" - "\x71\xB0\x12\x57\x89\x0D\xC0\x2B" - "\x9F\x12\x6A\x04\x67\xF1\x95\x31" - "\x59\xFD\x84\x95\x2C\x9C\x5B\xEC" - "\x09\xB0\x43\x96\x4A\x64\x80\x40" - "\xB9\x72\x19\xDD\x70\x42\xFA\xB1" - "\x4A\x2C\x0C\x0A\x60\x6E\xE3\x7C" - "\x37\x5A\xBE\xA4\x62\xCF\x29\xAB" - "\x7F\x4D\xA6\xB3\xE2\xB6\x64\xC6" - "\x33\x0B\xF3\xD5\x01\x38\x74\xA4" - "\x67\x1E\x75\x68\xC3\xAD\x76\xE9" - "\xE9\xBC\xF0\xEB\xD8\xFD\x31\x8A" - "\x5F\xC9\x18\x94\x4B\x86\x66\xFC" - "\xBD\x0B\x3D\xB3\x9F\xFA\x1F\xD9" - "\x78\xC4\xE3\x24\x1C\x67\xA2\xF8" - "\x43\xBC\x76\x75\xBF\x6C\x05\xB3" - "\x32\xE8\x7C\x80\xDB\xC7\xB6\x61" - "\x1A\x3E\x2B\xA7\x25\xED\x8F\xA0" - "\x00\x4B\xF8\x90\xCA\xD8\xFB\x12" - "\xAC\x1F\x18\xE9\xD2\x5E\xA2\x8E" - "\xE4\x84\x6B\x9D\xEB\x1E\x6B\xA3" - "\x7B\xDC\xCE\x15\x97\x27\xB2\x65" - "\xBC\x0E\x47\xAB\x55\x13\x53\xAB" - "\x0E\x34\x55\x02\x5F\x27\xC5\x89" - "\xDF\xC5\x70\xC4\xDD\x76\x82\xEE" - "\x68\xA6\x09\xB0\xE5\x5E\xF1\x0C" - "\xE3\xF3\x09\x9B\xFE\x65\x4B\xB8" - "\x30\xEC\xD5\x7C\x6A\xEC\x1D\xD2" - "\x93\xB7\xA1\x1A\x02\xD4\xC0\xD6" - "\x8D\x4D\x83\x9A\xED\x29\x4E\x14" - "\x86\xD5\x3C\x1A\xD5\xB9\x0A\x6A" - "\x72\x22\xD5\x92\x38\xF1\xA1\x86" - "\xB2\x41\x51\xCA\x4E\xAB\x8F\xD3" - "\x80\x56\xC3\xD7\x65\xE1\xB3\x86" - "\xCB\xCE\x98\xA1\xD4\x59\x1C\x06" - "\x01\xED\xF8\x29\x91\x19\x5C\x9A" - "\xEE\x28\x1B\x48\xD7\x32\xEF\x9F" - "\x6C\x2B\x66\x4E\x78\xD5\x8B\x72" - "\x80\xE7\x29\xDC\x23\x55\x98\x54" - "\xB1\xFF\x3E\x95\x56\xA8\x78\x78" - "\xEF\xC4\xA5\x11\x2D\x2B\xD8\x93" - "\x30\x6E\x7E\x51\xBB\x42\x5F\x03" - "\x43\x94\x23\x7E\xEE\xF0\xA5\x79" - "\x55\x01\xD4\x58\xB2\xF2\x85\x49" - "\x70\xC5\xB9\x0B\x3B\x7A\x6E\x6C", - .ilen = 1008, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59\xF0\x64\xFB\x92\x06" - "\x9D\x34\xCB\x3F\xD6\x6D\x04\x78" - "\x0F\xA6\x1A\xB1\x48\xDF\x53\xEA" - "\x81\x18\x8C\x23\xBA\x2E\xC5\x5C" - "\xF3\x67\xFE\x95\x09\xA0\x37\xCE" - "\x42\xD9\x70\x07\x7B\x12\xA9\x1D" - "\xB4\x4B\xE2\x56\xED\x84\x1B\x8F" - "\x26\xBD\x31\xC8\x5F\xF6\x6A\x01" - "\x98\x0C\xA3\x3A\xD1\x45\xDC\x73" - "\x0A\x7E\x15\xAC\x20\xB7\x4E\xE5" - "\x59\xF0\x87\x1E\x92\x29\xC0\x34" - "\xCB\x62\xF9\x6D\x04\x9B\x0F\xA6" - "\x3D\xD4\x48\xDF\x76\x0D\x81\x18" - "\xAF\x23\xBA\x51\xE8\x5C\xF3\x8A" - "\x21\x95\x2C\xC3\x37\xCE\x65\xFC" - "\x70\x07\x9E\x12\xA9\x40\xD7\x4B" - "\xE2\x79\x10\x84\x1B\xB2\x26\xBD" - "\x54\xEB\x5F\xF6\x8D\x01\x98\x2F" - "\xC6\x3A\xD1\x68\xFF\x73\x0A\xA1" - "\x15\xAC\x43\xDA\x4E\xE5\x7C\x13" - "\x87\x1E\xB5\x29\xC0\x57\xEE\x62" - "\xF9\x90\x04\x9B\x32\xC9\x3D\xD4" - "\x6B\x02\x76\x0D\xA4\x18\xAF\x46" - "\xDD\x51\xE8\x7F\x16\x8A\x21\xB8" - "\x2C\xC3\x5A\xF1\x65\xFC\x93\x07" - "\x9E\x35\xCC\x40\xD7\x6E\x05\x79" - "\x10\xA7\x1B\xB2\x49\xE0\x54\xEB" - "\x82\x19\x8D\x24\xBB\x2F\xC6\x5D" - "\xF4\x68\xFF\x96\x0A\xA1\x38\xCF" - "\x43\xDA\x71\x08\x7C\x13\xAA\x1E" - "\xB5\x4C\xE3\x57\xEE\x85\x1C\x90" - "\x27\xBE\x32\xC9\x60\xF7\x6B\x02" - "\x99\x0D\xA4\x3B\xD2\x46\xDD\x74" - "\x0B\x7F\x16\xAD\x21\xB8\x4F\xE6" - "\x5A\xF1\x88\x1F\x93\x2A\xC1\x35" - "\xCC\x63\xFA\x6E\x05\x9C\x10\xA7" - "\x3E\xD5\x49\xE0\x77\x0E\x82\x19" - "\xB0\x24\xBB\x52\xE9\x5D\xF4\x8B" - "\x22\x96\x2D\xC4\x38\xCF\x66\xFD" - "\x71\x08\x9F\x13\xAA\x41\xD8\x4C" - "\xE3\x7A\x11\x85\x1C\xB3\x27\xBE" - "\x55\xEC\x60\xF7\x8E\x02\x99\x30" - "\xC7\x3B\xD2\x69\x00\x74\x0B\xA2" - "\x16\xAD\x44\xDB\x4F\xE6\x7D\x14" - "\x88\x1F\xB6\x2A\xC1\x58\xEF\x63" - "\xFA\x91\x05\x9C\x33\xCA\x3E\xD5" - "\x6C\x03\x77\x0E\xA5\x19\xB0\x47" - "\xDE\x52\xE9\x80\x17\x8B\x22\xB9" - "\x2D\xC4\x5B\xF2\x66\xFD\x94\x08" - "\x9F\x36\xCD\x41\xD8\x6F\x06\x7A" - "\x11\xA8\x1C\xB3\x4A\xE1\x55\xEC" - "\x83\x1A\x8E\x25\xBC\x30\xC7\x5E" - "\xF5\x69\x00\x97\x0B\xA2\x39\xD0" - "\x44\xDB\x72\x09\x7D\x14\xAB\x1F" - "\xB6\x4D\xE4\x58\xEF\x86\x1D\x91" - "\x28\xBF\x33\xCA\x61\xF8\x6C\x03" - "\x9A\x0E\xA5\x3C\xD3\x47\xDE\x75" - "\x0C\x80\x17\xAE\x22\xB9\x50\xE7" - "\x5B\xF2\x89\x20\x94\x2B\xC2\x36" - "\xCD\x64\xFB\x6F\x06\x9D\x11\xA8" - "\x3F\xD6\x4A\xE1\x78\x0F\x83\x1A" - "\xB1\x25\xBC\x53\xEA\x5E\xF5\x8C" - "\x00\x97\x2E\xC5\x39\xD0\x67\xFE" - "\x72\x09\xA0\x14\xAB\x42\xD9\x4D", - .rlen = 1008, - .also_non_np = 1, - .np = 3, - .tap = { 1008 - 20, 4, 16 }, + .iv = "\xd8\x70\x9f\x42\xca\xe6\x83\x3a" + "\x52\x79\x42\xa5\x84\x6a\x96\x7f" + "\xe4\x8f\xed\x97\xe9\xd0\xe8\x0d" + "\x7c\xa6\xd8\xd4\x77\x9b\x40\x2e", + .assoc = "\xe8\x39\x2d\xaa\x0e\x85\x2d\xc1" + "\x72\xaf\x6e\xc9\x82\x33\xc7\x01" + "\xaf\x40\x70\xb8\x2a\x78\xc9\x14" + "\xac\xb1\x10\xca\x2e\xb3\x28\xe4" + "\xac\xfa\x58\x7f\xe5\x73\x09\x8c" + "\x1d\x40\x87\x8c\xd9\x75\xc0\x55" + "\xa2\xda\x07\xd1\xc2\xa9\xd1\xbb" + "\x09\x4f\x77\x62\x88\x2d\xf2\x68" + "\x54", + .alen = 65, + .input = "\xf7\x02\xbb\x11\x52\x24\xd8\x48" + "\x93\xe6\x9b\xee\x81\xfc\xf7\x82" + "\x79\xf0\xf3\xd9\x6c\x20\xa9\x1a" + "\xdc\xbc\x47\xc0\xe4\xcb\x10\x99" + "\x2f", + .ilen = 33, + .result = "\x33\xc1\xda\xfa\x15\x21\x07\x8e" + "\x93\x68\xea\x64\x7b\x3d\x4b\x6b" + "\x71\x5e\x5e\x6b\x92\xaa\x65\xc2" + "\x7a\x2a\xc1\xa9\x0a\xa1\x24\x81" + "\x26\x3a\x5a\x09\xe8\xce\x73\x72" + "\xde\x7b\x58\x9e\x85\xb9\xa4\x28" + "\xda", + .rlen = 49, + }, { + .key = "\x06\xcc\x4a\x79\x96\xc3\x82\xcf" + "\xb3\x1c\xc7\x12\x7f\xc5\x28\x04" + "\x44\xa1\x76\xfb\xad\xc8\x8a\x21" + "\x0d\xc8\x7f\xb6\x9b\xe3\xf8\x4f", + .klen = 32, + .iv = "\x15\x95\xd8\xe1\xda\x62\x2c\x56" + "\xd3\x53\xf4\x36\x7e\x8e\x59\x85" + "\x0e\x51\xf9\x1c\xee\x70\x6a\x27" + "\x3d\xd3\xb7\xac\x51\xfa\xdf\x05", + .assoc = "\x24\x5e\x67\x49\x1e\x01\xd6\xdd" + "\xf3\x89\x20\x5b\x7c\x57\x89\x07", + .alen = 16, + .input = "\x33\x27\xf5\xb1\x62\xa0\x80\x63" + "\x14\xc0\x4d\x7f\x7b\x20\xba\x89", + .ilen = 16, + .result = "\x3e\xf8\x86\x3d\x39\xf8\x96\x02" + "\x0f\xdf\xc9\x6e\x37\x1e\x57\x99" + "\x07\x2a\x1a\xac\xd1\xda\xfd\x3b" + "\xc7\xff\xbd\xbc\x85\x09\x0b", + .rlen = 31, + }, { + .key = "\x42\xf0\x84\x19\xa6\x3f\x2b\xea" + "\x34\xf6\x79\xa3\x79\xe9\xeb\x0a" + "\x6e\x63\x82\x7f\xb2\x68\x0c\x3a" + "\xce\xf5\x5e\x8e\x75\x42\x97\x26", + .klen = 32, + .iv = "\x51\xb9\x12\x80\xea\xde\xd5\x71" + "\x54\x2d\xa6\xc8\x78\xb2\x1b\x8c" + "\x39\x14\x05\xa0\xf3\x10\xec\x41" + "\xff\x01\x95\x84\x2b\x59\x7f\xdb", + .assoc = "\x61\x83\xa0\xe8\x2e\x7d\x7f\xf8" + "\x74\x63\xd2\xec\x76\x7c\x4c\x0d", + .alen = 16, + .input = "\x70\x4c\x2f\x50\x72\x1c\x29\x7f" + "\x95\x9a\xff\x10\x75\x45\x7d\x8f", + .ilen = 16, + .result = "\x2f\xc4\xd8\x0d\xa6\x07\xef\x2e" + "\x6c\xd9\x84\x63\x70\x97\x61\x37" + "\x08\x2f\x16\x90\x9e\x62\x30\x0d" + "\x62\xd5\xc8\xf0\x46\x1a", + .rlen = 30, + }, { + .key = "\x7f\x15\xbd\xb8\xb6\xba\xd3\x06" + "\xb5\xd1\x2b\x35\x73\x0e\xad\x10" + "\x98\x25\x8d\x03\xb7\x08\x8e\x54" + "\x90\x23\x3d\x67\x4f\xa1\x36\xfc", + .klen = 32, + .iv = "\x8e\xde\x4c\x20\xfa\x59\x7e\x8d" + "\xd5\x07\x58\x59\x72\xd7\xde\x92" + "\x63\xd6\x10\x24\xf8\xb0\x6e\x5a" + "\xc0\x2e\x74\x5d\x06\xb8\x1e\xb2", + .assoc = "\x9d\xa7\xda\x88\x3e\xf8\x28\x14" + "\xf5\x3e\x85\x7d\x70\xa0\x0f\x13", + .alen = 16, + .input = "\xac\x70\x69\xef\x82\x97\xd2\x9b" + "\x15\x74\xb1\xa2\x6f\x69\x3f\x95", + .ilen = 16, + .result = "\xce\xf3\x17\x87\x49\xc2\x00\x46" + "\xc6\x12\x5c\x8f\x81\x38\xaa\x55" + "\xf8\x67\x75\xf1\x75\xe3\x2a\x24", + .rlen = 24, + }, +}; + +static const struct aead_testvec aegis256_dec_tv_template[] = { + { + .key = "\x0f\xc9\x8e\x67\x44\x9e\xaa\x86" + "\x20\x36\x2c\x24\xfe\xc9\x30\x81" + "\xca\xb0\x82\x21\x41\xa8\xe0\x06" + "\x30\x0b\x37\xf6\xb6\x17\xe7\xb5", + .klen = 32, + .iv = "\x1e\x92\x1c\xcf\x88\x3d\x54\x0d" + "\x40\x6d\x59\x48\xfc\x92\x61\x03" + "\x95\x61\x05\x42\x82\x50\xc0\x0c" + "\x60\x16\x6f\xec\x6d\x2f\xcf\x6b", + .assoc = "", + .alen = 0, + .input = "\xd5\x65\x3a\xa9\x03\x51\xd7\xaa" + "\xfa\x4b\xd8\xa2\x41\x9b\xc1\xb2", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x4b\xed\xc8\x07\x54\x1a\x52\xa2" + "\xa1\x10\xde\xb5\xf8\xed\xf3\x87" + "\xf4\x72\x8e\xa5\x46\x48\x62\x20" + "\xf1\x38\x16\xce\x90\x76\x87\x8c", + .klen = 32, + .iv = "\x5a\xb7\x56\x6e\x98\xb9\xfd\x29" + "\xc1\x47\x0b\xda\xf6\xb6\x23\x09" + "\xbf\x23\x11\xc6\x87\xf0\x42\x26" + "\x22\x44\x4e\xc4\x47\x8e\x6e\x41", + .assoc = "", + .alen = 0, + .input = "\x84\xa2\x8f\xad\xdb\x8d\x2c\x16" + "\x9e\x89\xd9\x06\xa6\xa8\x14\x29" + "\x8b", + .ilen = 17, + .result = "\x79", + .rlen = 1, + }, { + .key = "\x88\x12\x01\xa6\x64\x96\xfb\xbe" + "\x22\xea\x90\x47\xf2\x11\xb5\x8e" + "\x1f\x35\x9a\x29\x4b\xe8\xe4\x39" + "\xb3\x66\xf5\xa6\x6a\xd5\x26\x62", + .klen = 32, + .iv = "\x97\xdb\x90\x0e\xa8\x35\xa5\x45" + "\x42\x21\xbd\x6b\xf0\xda\xe6\x0f" + "\xe9\xe5\x1d\x4a\x8c\x90\xc4\x40" + "\xe3\x71\x2d\x9c\x21\xed\x0e\x18", + .assoc = "", + .alen = 0, + .input = "\x09\x94\x1f\xa6\x13\xc3\x74\x75" + "\x17\xad\x8a\x0e\xd8\x66\x9a\x28" + "\xd7\x30\x66\x09\x2a\xdc\xfa\x2a" + "\x9f\x3b\xd7\xdd\x66\xd1\x2b", + .ilen = 31, + .result = "\xb5\x6e\xad\xdd\x30\x72\xfa\x53" + "\x82\x8e\x16\xb4\xed\x6d\x47", + .rlen = 15, + }, { + .key = "\xc4\x37\x3b\x45\x74\x11\xa4\xda" + "\xa2\xc5\x42\xd8\xec\x36\x78\x94" + "\x49\xf7\xa5\xad\x50\x88\x66\x53" + "\x74\x94\xd4\x7f\x44\x34\xc5\x39", + .klen = 32, + .iv = "\xd3\x00\xc9\xad\xb8\xb0\x4e\x61" + "\xc3\xfb\x6f\xfd\xea\xff\xa9\x15" + "\x14\xa8\x28\xce\x92\x30\x46\x59" + "\xa4\x9f\x0b\x75\xfb\x4c\xad\xee", + .assoc = "", + .alen = 0, + .input = "\x8a\x46\xa2\x22\x8c\x03\xab\x6f" + "\x54\x63\x4e\x7f\xc9\x8e\xfa\x70" + "\x7b\xe5\x8d\x78\xbc\xe9\xb6\xa1" + "\x29\x17\xc8\x3b\x52\xa4\x98\x72", + .ilen = 32, + .result = "\xf2\x92\xe6\x7d\x40\xee\xa3\x6f" + "\x03\x68\xc8\x45\xe7\x91\x0a\x18", + .rlen = 16, + }, { + .key = "\x01\x5c\x75\xe5\x84\x8d\x4d\xf6" + "\x23\x9f\xf4\x6a\xe6\x5a\x3b\x9a" + "\x74\xb9\xb1\x32\x55\x28\xe8\x6d" + "\x35\xc1\xb3\x57\x1f\x93\x64\x0f", + .klen = 32, + .iv = "\x10\x25\x03\x4c\xc8\x2c\xf7\x7d" + "\x44\xd5\x21\x8e\xe4\x23\x6b\x1c" + "\x3e\x6a\x34\x53\x97\xd0\xc8\x73" + "\x66\xcd\xea\x4d\xd5\xab\x4c\xc5", + .assoc = "", + .alen = 0, + .input = "\x71\x6b\x37\x0b\x02\x61\x28\x12" + "\x83\xab\x66\x90\x84\xc7\xd1\xc5" + "\xb2\x7a\xb4\x7b\xb4\xfe\x02\xb2" + "\xc0\x00\x39\x13\xb5\x51\x68\x44" + "\xad", + .ilen = 33, + .result = "\x2e\xb7\x20\x1c\x50\x6a\x4b\x8b" + "\x84\x42\x7a\xd7\xe1\xb5\xcd\x1f" + "\xd3", + .rlen = 17, + }, { + .key = "\x3d\x80\xae\x84\x94\x09\xf6\x12" + "\xa4\x79\xa6\xfb\xe0\x7f\xfd\xa0" + "\x9e\x7c\xbc\xb6\x5b\xc8\x6a\x86" + "\xf7\xef\x91\x30\xf9\xf2\x04\xe6", + .klen = 32, + .iv = "\x4c\x49\x3d\xec\xd8\xa8\xa0\x98" + "\xc5\xb0\xd3\x1f\xde\x48\x2e\x22" + "\x69\x2c\x3f\xd7\x9c\x70\x4a\x8d" + "\x27\xfa\xc9\x26\xaf\x0a\xeb\x9c", + .assoc = "", + .alen = 0, + .input = "\xaf\xa4\x34\x0d\x59\xe6\x1c\x2f" + "\x06\x3b\x52\x18\x49\x75\x1b\xf0" + "\x53\x09\x72\x7b\x45\x79\xe0\xbe" + "\x89\x85\x23\x15\xb8\x79\x07\x4c" + "\x53\x7a\x15\x37\x0a\xee\xb7\xfb" + "\xc4\x1f\x12\x27\xcf\x77\x90", + .ilen = 47, + .result = "\x6b\xdc\x5a\xbb\x60\xe5\xf4\xa6" + "\x05\x1d\x2c\x68\xdb\xda\x8f\x25" + "\xfe\x8d\x45\x19\x1e\xc0\x0b\x99" + "\x88\x11\x39\x12\x1c\x3a\xbb", + .rlen = 31, + }, { + .key = "\x7a\xa5\xe8\x23\xa4\x84\x9e\x2d" + "\x25\x53\x58\x8c\xda\xa3\xc0\xa6" + "\xc8\x3e\xc8\x3a\x60\x68\xec\xa0" + "\xb8\x1c\x70\x08\xd3\x51\xa3\xbd", + .klen = 32, + .iv = "\x89\x6e\x77\x8b\xe8\x23\x49\xb4" + "\x45\x8a\x85\xb1\xd8\x6c\xf1\x28" + "\x93\xef\x4b\x5b\xa1\x10\xcc\xa6" + "\xe8\x28\xa8\xfe\x89\x69\x8b\x72", + .assoc = "", + .alen = 0, + .input = "\xe2\xc9\x0b\x33\x31\x02\xb3\xb4" + "\x33\xfe\xeb\xa8\xb7\x9b\xb2\xd7" + "\xeb\x0f\x05\x2b\xba\xb3\xca\xef" + "\xf6\xd1\xb6\xc0\xb9\x9b\x85\xc5" + "\xbf\x7a\x3e\xcc\x31\x76\x09\x80" + "\x32\x5d\xbb\xe8\x38\x0e\x77\xd3", + .ilen = 48, + .result = "\xa7\x00\x93\x5b\x70\x61\x9d\xc2" + "\x86\xf7\xde\xfa\xd5\xfe\x52\x2b" + "\x28\x50\x51\x9d\x24\x60\x8d\xb3" + "\x49\x3e\x17\xea\xf6\x99\x5a\xdd", + .rlen = 32, + }, { + .key = "\xb6\xca\x22\xc3\xb4\x00\x47\x49" + "\xa6\x2d\x0a\x1e\xd4\xc7\x83\xad" + "\xf3\x00\xd4\xbf\x65\x08\x6e\xb9" + "\x7a\x4a\x4f\xe0\xad\xb0\x42\x93", + .klen = 32, + .iv = "\xc5\x93\xb0\x2a\xf8\x9f\xf1\xd0" + "\xc6\x64\x37\x42\xd2\x90\xb3\x2e" + "\xbd\xb1\x57\xe0\xa6\xb0\x4e\xc0" + "\xaa\x55\x87\xd6\x63\xc8\x2a\x49", + .assoc = "\xd5", + .alen = 1, + .input = "\x96\x43\x30\xca\x6c\x4f\xd7\x12" + "\xba\xd9\xb3\x18\x86\xdf\xc3\x52", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xf3\xee\x5c\x62\xc4\x7c\xf0\x65" + "\x27\x08\xbd\xaf\xce\xec\x45\xb3" + "\x1d\xc3\xdf\x43\x6a\xa8\xf0\xd3" + "\x3b\x77\x2e\xb9\x87\x0f\xe1\x6a", + .klen = 32, + .iv = "\x02\xb8\xea\xca\x09\x1b\x9a\xec" + "\x47\x3e\xe9\xd4\xcc\xb5\x76\x34" + "\xe8\x73\x62\x64\xab\x50\xd0\xda" + "\x6b\x83\x66\xaf\x3e\x27\xc9\x1f", + .assoc = "\x11\x81\x78\x32\x4d\xb9\x44\x73" + "\x68\x75\x16\xf8\xcb\x7e\xa7", + .alen = 15, + .input = "\x2f\xab\x45\xe2\xa7\x46\xc5\x83" + "\x11\x9f\xb0\x74\xee\xc7\x03\xdd", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x2f\x13\x95\x01\xd5\xf7\x99\x81" + "\xa8\xe2\x6f\x41\xc8\x10\x08\xb9" + "\x47\x85\xeb\xc7\x6f\x48\x72\xed" + "\xfc\xa5\x0d\x91\x61\x6e\x81\x40", + .klen = 32, + .iv = "\x3f\xdc\x24\x69\x19\x96\x43\x08" + "\xc8\x18\x9b\x65\xc6\xd9\x39\x3b" + "\x12\x35\x6e\xe8\xb0\xf0\x52\xf3" + "\x2d\xb0\x45\x87\x18\x86\x68\xf6", + .assoc = "\x4e\xa5\xb2\xd1\x5d\x35\xed\x8f" + "\xe8\x4f\xc8\x89\xc5\xa2\x69\xbc", + .alen = 16, + .input = "\x16\x44\x73\x33\x5d\xf2\xb9\x04" + "\x6b\x79\x98\xef\xdb\xd5\xc5\xf1", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x6c\x38\xcf\xa1\xe5\x73\x41\x9d" + "\x29\xbc\x21\xd2\xc2\x35\xcb\xbf" + "\x72\x47\xf6\x4b\x74\xe8\xf4\x06" + "\xbe\xd3\xec\x6a\x3b\xcd\x20\x17", + .klen = 32, + .iv = "\x7b\x01\x5d\x08\x29\x12\xec\x24" + "\x49\xf3\x4d\xf7\xc0\xfe\xfb\x41" + "\x3c\xf8\x79\x6c\xb6\x90\xd4\x0d" + "\xee\xde\x23\x60\xf2\xe5\x08\xcc", + .assoc = "\x8a\xca\xec\x70\x6d\xb1\x96\xab" + "\x69\x29\x7a\x1b\xbf\xc7\x2c\xc2" + "\x07", + .alen = 17, + .input = "\xa4\x9b\xb8\x47\xc0\xed\x7a\x45" + "\x98\x54\x8c\xed\x3d\x17\xf0\xdd", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xa8\x5c\x09\x40\xf5\xef\xea\xb8" + "\xaa\x96\xd3\x64\xbc\x59\x8d\xc6" + "\x9c\x0a\x02\xd0\x79\x88\x76\x20" + "\x7f\x00\xca\x42\x15\x2c\xbf\xed", + .klen = 32, + .iv = "\xb8\x26\x97\xa8\x39\x8e\x94\x3f" + "\xca\xcd\xff\x88\xba\x22\xbe\x47" + "\x67\xba\x85\xf1\xbb\x30\x56\x26" + "\xaf\x0b\x02\x38\xcc\x44\xa7\xa3", + .assoc = "\xc7\xef\x26\x10\x7d\x2c\x3f\xc6" + "\xea\x03\x2c\xac\xb9\xeb\xef\xc9" + "\x31\x6b\x08\x12\xfc\xd8\x37\x2d" + "\xe0\x17\x3a\x2e\x83\x5c\x8f", + .alen = 31, + .input = "\x20\x24\xe2\x33\x5c\x60\xc9\xf0" + "\xa4\x96\x2f\x0d\x53\xc2\xf8\xfc", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xe5\x81\x42\xdf\x05\x6a\x93\xd4" + "\x2b\x70\x85\xf5\xb6\x7d\x50\xcc" + "\xc6\xcc\x0e\x54\x7f\x28\xf8\x3a" + "\x40\x2e\xa9\x1a\xf0\x8b\x5e\xc4", + .klen = 32, + .iv = "\xf4\x4a\xd1\x47\x49\x09\x3d\x5b" + "\x4b\xa7\xb1\x19\xb4\x46\x81\x4d" + "\x91\x7c\x91\x75\xc0\xd0\xd8\x40" + "\x71\x39\xe1\x10\xa6\xa3\x46\x7a", + .assoc = "\x03\x14\x5f\xaf\x8d\xa8\xe7\xe2" + "\x6b\xde\xde\x3e\xb3\x10\xb1\xcf" + "\x5c\x2d\x14\x96\x01\x78\xb9\x47" + "\xa1\x44\x19\x06\x5d\xbb\x2e\x2f", + .alen = 32, + .input = "\x6f\x4a\xb9\xe0\xff\x51\xa3\xf1" + "\xd2\x64\x3e\x66\x6a\xb2\x03\xc0", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x22\xa6\x7c\x7f\x15\xe6\x3c\xf0" + "\xac\x4b\x37\x86\xb0\xa2\x13\xd2" + "\xf1\x8e\x19\xd8\x84\xc8\x7a\x53" + "\x02\x5b\x88\xf3\xca\xea\xfe\x9b", + .klen = 32, + .iv = "\x31\x6f\x0b\xe6\x59\x85\xe6\x77" + "\xcc\x81\x63\xab\xae\x6b\x43\x54" + "\xbb\x3f\x9c\xf9\xc5\x70\x5a\x5a" + "\x32\x67\xc0\xe9\x80\x02\xe5\x50", + .assoc = "\x40", + .alen = 1, + .input = "\x2c\xfb\xad\x7e\xbe\xa0\x9a\x5b" + "\x7a\x3f\x81\xf7\xfc\x1b\x79\x83" + "\xc7", + .ilen = 17, + .result = "\x4f", + .rlen = 1, + }, { + .key = "\x5e\xcb\xb6\x1e\x25\x62\xe4\x0c" + "\x2d\x25\xe9\x18\xaa\xc6\xd5\xd8" + "\x1b\x50\x25\x5d\x89\x68\xfc\x6d" + "\xc3\x89\x67\xcb\xa4\x49\x9d\x71", + .klen = 32, + .iv = "\x6d\x94\x44\x86\x69\x00\x8f\x93" + "\x4d\x5b\x15\x3c\xa8\x8f\x06\x5a" + "\xe6\x01\xa8\x7e\xca\x10\xdc\x73" + "\xf4\x94\x9f\xc1\x5a\x61\x85\x27", + .assoc = "\x7c\x5d\xd3\xee\xad\x9f\x39\x1a" + "\x6d\x92\x42\x61\xa7\x58\x37", + .alen = 15, + .input = "\x1f\x7f\xca\x3c\x2b\xe7\x27\xba" + "\x7e\x98\x83\x02\x34\x23\xf7\x94" + "\xde\x35\xe6\x1d\x14\x18\xe5\x38" + "\x14\x80\x6a\xa7\x1b\xae\x1d", + .ilen = 31, + .result = "\x8b\x26\x61\x55\xf1\x3e\xe3\xa1" + "\x8d\xc8\x6e\x85\xa5\x21\x67", + .rlen = 15, + }, { + .key = "\x9b\xef\xf0\xbd\x35\xdd\x8d\x28" + "\xad\xff\x9b\xa9\xa4\xeb\x98\xdf" + "\x46\x13\x31\xe1\x8e\x08\x7e\x87" + "\x85\xb6\x46\xa3\x7e\xa8\x3c\x48", + .klen = 32, + .iv = "\xaa\xb8\x7e\x25\x79\x7c\x37\xaf" + "\xce\x36\xc7\xce\xa2\xb4\xc9\x60" + "\x10\xc3\xb3\x02\xcf\xb0\x5e\x8d" + "\xb5\xc2\x7e\x9a\x35\xc0\x24\xfd", + .assoc = "\xb9\x82\x0c\x8d\xbd\x1b\xe2\x36" + "\xee\x6c\xf4\xf2\xa1\x7d\xf9\xe2", + .alen = 16, + .input = "\x05\x86\x9e\xd7\x2b\xa3\x97\x01" + "\xbe\x28\x98\x10\x6f\xe9\x61\x32" + "\x96\xbb\xb1\x2e\x8f\x0c\x44\xb9" + "\x46\x2d\x55\xe3\x42\x67\xf2\xaf", + .ilen = 32, + .result = "\xc8\x4b\x9b\xf5\x01\xba\x8c\xbd" + "\x0e\xa3\x21\x16\x9f\x46\x2a\x63", + .rlen = 16, + }, { + .key = "\xd7\x14\x29\x5d\x45\x59\x36\x44" + "\x2e\xd9\x4d\x3b\x9e\x0f\x5b\xe5" + "\x70\xd5\x3c\x65\x93\xa8\x00\xa0" + "\x46\xe4\x25\x7c\x58\x08\xdb\x1e", + .klen = 32, + .iv = "\xe6\xdd\xb8\xc4\x89\xf8\xe0\xca" + "\x4f\x10\x7a\x5f\x9c\xd8\x8b\x66" + "\x3b\x86\xbf\x86\xd4\x50\xe0\xa7" + "\x76\xef\x5c\x72\x0f\x1f\xc3\xd4", + .assoc = "\xf5\xa6\x46\x2c\xce\x97\x8a\x51" + "\x6f\x46\xa6\x83\x9b\xa1\xbc\xe8" + "\x05", + .alen = 17, + .input = "\x9c\xe0\x06\x7b\x86\xcf\x2e\xd8" + "\x45\x65\x1b\x72\x9b\xaa\xa3\x1e" + "\x87\x9d\x26\xdf\xff\x81\x11\xd2" + "\x47\x41\xb9\x24\xc1\x8a\xa3\x8b" + "\x55", + .ilen = 33, + .result = "\x05\x70\xd5\x94\x12\x36\x35\xd8" + "\x8f\x7d\xd3\xa8\x99\x6a\xed\x69" + "\xd0", + .rlen = 17, + }, { + .key = "\x14\x39\x63\xfc\x56\xd5\xdf\x5f" + "\xaf\xb3\xff\xcc\x98\x33\x1d\xeb" + "\x9a\x97\x48\xe9\x98\x48\x82\xba" + "\x07\x11\x04\x54\x32\x67\x7b\xf5", + .klen = 32, + .iv = "\x23\x02\xf1\x64\x9a\x73\x89\xe6" + "\xd0\xea\x2c\xf1\x96\xfc\x4e\x6d" + "\x65\x48\xcb\x0a\xda\xf0\x62\xc0" + "\x38\x1d\x3b\x4a\xe9\x7e\x62\xaa", + .assoc = "\x32\xcb\x80\xcc\xde\x12\x33\x6d" + "\xf0\x20\x58\x15\x95\xc6\x7f\xee" + "\x2f\xf9\x4e\x2c\x1b\x98\x43\xc7" + "\x68\x28\x73\x40\x9f\x96\x4a", + .alen = 31, + .input = "\xa0\xc8\xde\x83\x0d\xc3\x4e\xd5" + "\x69\x7f\x7a\xdd\x8c\x46\xda\xba" + "\x0a\x5c\x0e\x7f\xac\xee\x02\xd2" + "\xe5\x4b\x0a\xba\xb8\xa4\x7b\x66" + "\xde\xae\xdb\xc2\xc0\x0b\xf7\x2b" + "\xdf\xb8\xea\xd8\xa9\x38\xed", + .ilen = 47, + .result = "\x41\x94\x0e\x33\x22\xb1\xdd\xf4" + "\x10\x57\x85\x39\x93\x8f\xaf\x70" + "\xfa\xa9\xd0\x4d\x5c\x40\x23\xcd" + "\x98\x34\xab\x37\x56\xae\x32", + .rlen = 31, + }, { + .key = "\x50\x5d\x9d\x9b\x66\x50\x88\x7b" + "\x30\x8e\xb1\x5e\x92\x58\xe0\xf1" + "\xc5\x5a\x53\x6e\x9d\xe8\x04\xd4" + "\xc9\x3f\xe2\x2d\x0c\xc6\x1a\xcb", + .klen = 32, + .iv = "\x5f\x27\x2b\x03\xaa\xef\x32\x02" + "\x50\xc4\xde\x82\x90\x21\x11\x73" + "\x8f\x0a\xd6\x8f\xdf\x90\xe4\xda" + "\xf9\x4a\x1a\x23\xc3\xdd\x02\x81", + .assoc = "\x6e\xf0\xba\x6b\xee\x8e\xdc\x89" + "\x71\xfb\x0a\xa6\x8f\xea\x41\xf4" + "\x5a\xbb\x59\xb0\x20\x38\xc5\xe0" + "\x29\x56\x52\x19\x79\xf5\xe9\x37", + .alen = 32, + .input = "\xd3\x68\x14\x70\x3c\x01\x43\x86" + "\x02\xab\xbe\x75\xaa\xe7\xf5\x53" + "\x5c\x05\xbd\x9b\x19\xbb\x2a\x61" + "\x8f\x69\x05\x75\x8e\xca\x60\x0c" + "\x5b\xa2\x48\x61\x32\x74\x11\x2b" + "\xf6\xcf\x06\x78\x6f\x78\x1a\x4a", + .ilen = 48, + .result = "\x7e\xb9\x48\xd3\x32\x2d\x86\x10" + "\x91\x31\x37\xcb\x8d\xb3\x72\x76" + "\x24\x6b\xdc\xd1\x61\xe0\xa5\xe7" + "\x5a\x61\x8a\x0f\x30\x0d\xd1\xec", + .rlen = 32, + }, { + .key = "\x8d\x82\xd6\x3b\x76\xcc\x30\x97" + "\xb1\x68\x63\xef\x8c\x7c\xa3\xf7" + "\xef\x1c\x5f\xf2\xa3\x88\x86\xed" + "\x8a\x6d\xc1\x05\xe7\x25\xb9\xa2", + .klen = 32, + .iv = "\x9c\x4b\x65\xa2\xba\x6b\xdb\x1e" + "\xd1\x9e\x90\x13\x8a\x45\xd3\x79" + "\xba\xcd\xe2\x13\xe4\x30\x66\xf4" + "\xba\x78\xf9\xfb\x9d\x3c\xa1\x58", + .assoc = "\xab\x14\xf3\x0a\xfe\x0a\x85\xa5" + "\xf2\xd5\xbc\x38\x89\x0e\x04\xfb" + "\x84\x7d\x65\x34\x25\xd8\x47\xfa" + "\xeb\x83\x31\xf1\x54\x54\x89\x0d" + "\x9d", + .alen = 33, + .input = "\x07\x0a\x35\xb0\x82\x03\x5a\xd2" + "\x15\x3a\x6c\x72\x83\x9b\xb1\x75" + "\xea\xf2\xfc\xff\xc6\xf1\x13\xa4" + "\x1a\x93\x33\x79\x97\x82\x81\xc0" + "\x96\xc2\x00\xab\x39\xae\xa1\x62" + "\x53\xa3\x86\xc9\x07\x8c\xaf\x22" + "\x47\x31\x29\xca\x4a\x95\xf5\xd5" + "\x20\x63\x5a\x54\x80\x2c\x4a\x63" + "\xfb\x18\x73\x31\x4f\x08\x21\x5d" + "\x20\xe9\xc3\x7e\xea\x25\x77\x3a" + "\x65", + .ilen = 81, + .result = "\xba\xde\x82\x72\x42\xa9\x2f\x2c" + "\x12\x0b\xe9\x5c\x87\xd7\x35\x7c" + "\x4f\x2e\xe8\x55\x66\x80\x27\x00" + "\x1b\x8f\x68\xe7\x0a\x6c\x71\xc3" + "\x21\x78\x55\x9d\x9c\x65\x7b\xcd" + "\x0a\x34\x97\xff\x47\x37\xb0\x2a" + "\x80\x0d\x19\x98\x33\xa9\x7a\xe3" + "\x2e\x4c\xc6\xf3\x8c\x88\x42\x01" + "\xbd", + .rlen = 65, + }, { + .key = "\xc9\xa7\x10\xda\x86\x48\xd9\xb3" + "\x32\x42\x15\x80\x85\xa1\x65\xfe" + "\x19\xde\x6b\x76\xa8\x28\x08\x07" + "\x4b\x9a\xa0\xdd\xc1\x84\x58\x79", + .klen = 32, + .iv = "\xd8\x70\x9f\x42\xca\xe6\x83\x3a" + "\x52\x79\x42\xa5\x84\x6a\x96\x7f" + "\xe4\x8f\xed\x97\xe9\xd0\xe8\x0d" + "\x7c\xa6\xd8\xd4\x77\x9b\x40\x2e", + .assoc = "\xe8\x39\x2d\xaa\x0e\x85\x2d\xc1" + "\x72\xaf\x6e\xc9\x82\x33\xc7\x01" + "\xaf\x40\x70\xb8\x2a\x78\xc9\x14" + "\xac\xb1\x10\xca\x2e\xb3\x28\xe4" + "\xac\xfa\x58\x7f\xe5\x73\x09\x8c" + "\x1d\x40\x87\x8c\xd9\x75\xc0\x55" + "\xa2\xda\x07\xd1\xc2\xa9\xd1\xbb" + "\x09\x4f\x77\x62\x88\x2d\xf2\x68" + "\x54", + .alen = 65, + .input = "\x33\xc1\xda\xfa\x15\x21\x07\x8e" + "\x93\x68\xea\x64\x7b\x3d\x4b\x6b" + "\x71\x5e\x5e\x6b\x92\xaa\x65\xc2" + "\x7a\x2a\xc1\xa9\x0a\xa1\x24\x81" + "\x26\x3a\x5a\x09\xe8\xce\x73\x72" + "\xde\x7b\x58\x9e\x85\xb9\xa4\x28" + "\xda", + .ilen = 49, + .result = "\xf7\x02\xbb\x11\x52\x24\xd8\x48" + "\x93\xe6\x9b\xee\x81\xfc\xf7\x82" + "\x79\xf0\xf3\xd9\x6c\x20\xa9\x1a" + "\xdc\xbc\x47\xc0\xe4\xcb\x10\x99" + "\x2f", + .rlen = 33, + }, { + .key = "\x06\xcc\x4a\x79\x96\xc3\x82\xcf" + "\xb3\x1c\xc7\x12\x7f\xc5\x28\x04" + "\x44\xa1\x76\xfb\xad\xc8\x8a\x21" + "\x0d\xc8\x7f\xb6\x9b\xe3\xf8\x4f", + .klen = 32, + .iv = "\x15\x95\xd8\xe1\xda\x62\x2c\x56" + "\xd3\x53\xf4\x36\x7e\x8e\x59\x85" + "\x0e\x51\xf9\x1c\xee\x70\x6a\x27" + "\x3d\xd3\xb7\xac\x51\xfa\xdf\x05", + .assoc = "\x24\x5e\x67\x49\x1e\x01\xd6\xdd" + "\xf3\x89\x20\x5b\x7c\x57\x89\x07", + .alen = 16, + .input = "\x3e\xf8\x86\x3d\x39\xf8\x96\x02" + "\x0f\xdf\xc9\x6e\x37\x1e\x57\x99" + "\x07\x2a\x1a\xac\xd1\xda\xfd\x3b" + "\xc7\xff\xbd\xbc\x85\x09\x0b", + .ilen = 31, + .result = "\x33\x27\xf5\xb1\x62\xa0\x80\x63" + "\x14\xc0\x4d\x7f\x7b\x20\xba\x89", + .rlen = 16, + }, { + .key = "\x42\xf0\x84\x19\xa6\x3f\x2b\xea" + "\x34\xf6\x79\xa3\x79\xe9\xeb\x0a" + "\x6e\x63\x82\x7f\xb2\x68\x0c\x3a" + "\xce\xf5\x5e\x8e\x75\x42\x97\x26", + .klen = 32, + .iv = "\x51\xb9\x12\x80\xea\xde\xd5\x71" + "\x54\x2d\xa6\xc8\x78\xb2\x1b\x8c" + "\x39\x14\x05\xa0\xf3\x10\xec\x41" + "\xff\x01\x95\x84\x2b\x59\x7f\xdb", + .assoc = "\x61\x83\xa0\xe8\x2e\x7d\x7f\xf8" + "\x74\x63\xd2\xec\x76\x7c\x4c\x0d", + .alen = 16, + .input = "\x2f\xc4\xd8\x0d\xa6\x07\xef\x2e" + "\x6c\xd9\x84\x63\x70\x97\x61\x37" + "\x08\x2f\x16\x90\x9e\x62\x30\x0d" + "\x62\xd5\xc8\xf0\x46\x1a", + .ilen = 30, + .result = "\x70\x4c\x2f\x50\x72\x1c\x29\x7f" + "\x95\x9a\xff\x10\x75\x45\x7d\x8f", + .rlen = 16, + }, { + .key = "\x7f\x15\xbd\xb8\xb6\xba\xd3\x06" + "\xb5\xd1\x2b\x35\x73\x0e\xad\x10" + "\x98\x25\x8d\x03\xb7\x08\x8e\x54" + "\x90\x23\x3d\x67\x4f\xa1\x36\xfc", + .klen = 32, + .iv = "\x8e\xde\x4c\x20\xfa\x59\x7e\x8d" + "\xd5\x07\x58\x59\x72\xd7\xde\x92" + "\x63\xd6\x10\x24\xf8\xb0\x6e\x5a" + "\xc0\x2e\x74\x5d\x06\xb8\x1e\xb2", + .assoc = "\x9d\xa7\xda\x88\x3e\xf8\x28\x14" + "\xf5\x3e\x85\x7d\x70\xa0\x0f\x13", + .alen = 16, + .input = "\xce\xf3\x17\x87\x49\xc2\x00\x46" + "\xc6\x12\x5c\x8f\x81\x38\xaa\x55" + "\xf8\x67\x75\xf1\x75\xe3\x2a\x24", + .ilen = 24, + .result = "\xac\x70\x69\xef\x82\x97\xd2\x9b" + "\x15\x74\xb1\xa2\x6f\x69\x3f\x95", + .rlen = 16, + }, +}; + +/* + * MORUS-640 test vectors - generated via reference implementation from + * SUPERCOP (https://bench.cr.yp.to/supercop.html): + * + * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz + * (see crypto_aead/morus640128v2/) + */ +static const struct aead_testvec morus640_enc_tv_template[] = { + { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 16, + .iv = "\x0f\xc9\x8e\x67\x44\x9e\xaa\x86" + "\x20\x36\x2c\x24\xfe\xc9\x30\x81", + .assoc = "", + .alen = 0, + .input = "", + .ilen = 0, + .result = "\x89\x62\x7d\xf3\x07\x9d\x52\x05" + "\x53\xc3\x04\x60\x93\xb4\x37\x9a", + .rlen = 16, + }, { + .key = "\x3c\x24\x39\x9f\x10\x7b\xa8\x1b" + "\x80\xda\xb2\x91\xf9\x24\xc2\x06", + .klen = 16, + .iv = "\x4b\xed\xc8\x07\x54\x1a\x52\xa2" + "\xa1\x10\xde\xb5\xf8\xed\xf3\x87", + .assoc = "", + .alen = 0, + .input = "\x69", + .ilen = 1, + .result = "\xa8\x8d\xe4\x90\xb5\x50\x8f\x78" + "\xb6\x10\x9a\x59\x5f\x61\x37\x70" + "\x09", + .rlen = 17, + }, { + .key = "\x79\x49\x73\x3e\x20\xf7\x51\x37" + "\x01\xb4\x64\x22\xf3\x48\x85\x0c", + .klen = 16, + .iv = "\x88\x12\x01\xa6\x64\x96\xfb\xbe" + "\x22\xea\x90\x47\xf2\x11\xb5\x8e", + .assoc = "", + .alen = 0, + .input = "\xa6\xa4\x1e\x76\xec\xd4\x50\xcc" + "\x62\x58\xe9\x8f\xef\xa4\x17", + .ilen = 15, + .result = "\x76\xdd\xb9\x05\x3d\xce\x61\x38" + "\xf3\xef\xf7\xe5\xd7\xfd\x70\xa5" + "\xcf\x9d\x64\xb8\x0a\x9f\xfd\x8b" + "\xd4\x6e\xfe\xd9\xc8\x63\x4b", + .rlen = 31, + }, { + .key = "\xb5\x6e\xad\xdd\x30\x72\xfa\x53" + "\x82\x8e\x16\xb4\xed\x6d\x47\x12", + .klen = 16, + .iv = "\xc4\x37\x3b\x45\x74\x11\xa4\xda" + "\xa2\xc5\x42\xd8\xec\x36\x78\x94", + .assoc = "", + .alen = 0, + .input = "\xe2\xc9\x58\x15\xfc\x4f\xf8\xe8" + "\xe3\x32\x9b\x21\xe9\xc8\xd9\x97", + .ilen = 16, + .result = "\xdc\x72\xe8\x14\xfb\x63\xad\x72" + "\x1f\x57\x9a\x1f\x88\x81\xdb\xd6" + "\xc1\x91\x9d\xb9\x25\xc4\x99\x4c" + "\x97\xcd\x8a\x0c\x9d\x68\x00\x1c", + .rlen = 32, + }, { + .key = "\xf2\x92\xe6\x7d\x40\xee\xa3\x6f" + "\x03\x68\xc8\x45\xe7\x91\x0a\x18", + .klen = 16, + .iv = "\x01\x5c\x75\xe5\x84\x8d\x4d\xf6" + "\x23\x9f\xf4\x6a\xe6\x5a\x3b\x9a", + .assoc = "", + .alen = 0, + .input = "\x1f\xee\x92\xb4\x0c\xcb\xa1\x04" + "\x64\x0c\x4d\xb2\xe3\xec\x9c\x9d" + "\x09", + .ilen = 17, + .result = "\x6b\x4f\x3b\x90\x9a\xa2\xb3\x82" + "\x0a\xb8\x55\xee\xeb\x73\x4d\x7f" + "\x54\x11\x3a\x8a\x31\xa3\xb5\xf2" + "\xcd\x49\xdb\xf3\xee\x26\xbd\xa2" + "\x0d", + .rlen = 33, + }, { + .key = "\x2e\xb7\x20\x1c\x50\x6a\x4b\x8b" + "\x84\x42\x7a\xd7\xe1\xb5\xcd\x1f", + .klen = 16, + .iv = "\x3d\x80\xae\x84\x94\x09\xf6\x12" + "\xa4\x79\xa6\xfb\xe0\x7f\xfd\xa0", + .assoc = "", + .alen = 0, + .input = "\x5c\x13\xcb\x54\x1c\x47\x4a\x1f" + "\xe5\xe6\xff\x44\xdd\x11\x5f\xa3" + "\x33\xdd\xc2\xf8\xdd\x18\x2b\x93" + "\x57\x05\x01\x1c\x66\x22\xd3", + .ilen = 31, + .result = "\x59\xd1\x0f\x6b\xee\x27\x84\x92" + "\xb7\xa9\xb5\xdd\x02\xa4\x12\xa5" + "\x50\x32\xb4\x9a\x2e\x35\x83\x55" + "\x36\x12\x12\xed\xa3\x31\xc5\x30" + "\xa7\xe2\x4a\x6d\x05\x59\x43\x91" + "\x75\xfa\x6c\x17\xc6\x73\xca", + .rlen = 47, + }, { + .key = "\x6b\xdc\x5a\xbb\x60\xe5\xf4\xa6" + "\x05\x1d\x2c\x68\xdb\xda\x8f\x25", + .klen = 16, + .iv = "\x7a\xa5\xe8\x23\xa4\x84\x9e\x2d" + "\x25\x53\x58\x8c\xda\xa3\xc0\xa6", + .assoc = "", + .alen = 0, + .input = "\x98\x37\x05\xf3\x2c\xc2\xf3\x3b" + "\x66\xc0\xb1\xd5\xd7\x35\x21\xaa" + "\x5d\x9f\xce\x7c\xe2\xb8\xad\xad" + "\x19\x33\xe0\xf4\x40\x81\x72\x28", + .ilen = 32, + .result = "\xdb\x49\x68\x0f\x91\x5b\x21\xb1" + "\xcf\x50\xb2\x4c\x32\xe1\xa6\x69" + "\xc0\xfb\x44\x1f\xa0\x9a\xeb\x39" + "\x1b\xde\x68\x38\xcc\x27\x52\xc5" + "\xf6\x3e\x74\xea\x66\x5b\x5f\x0c" + "\x65\x9e\x58\xe6\x52\xa2\xfe\x59", + .rlen = 48, + }, { + .key = "\xa7\x00\x93\x5b\x70\x61\x9d\xc2" + "\x86\xf7\xde\xfa\xd5\xfe\x52\x2b", + .klen = 16, + .iv = "\xb6\xca\x22\xc3\xb4\x00\x47\x49" + "\xa6\x2d\x0a\x1e\xd4\xc7\x83\xad", + .assoc = "\xc5", + .alen = 1, + .input = "", + .ilen = 0, + .result = "\x56\xe7\x24\x52\xdd\x95\x60\x5b" + "\x09\x48\x39\x69\x9c\xb3\x62\x46", + .rlen = 16, + }, { + .key = "\xe4\x25\xcd\xfa\x80\xdd\x46\xde" + "\x07\xd1\x90\x8b\xcf\x23\x15\x31", + .klen = 16, + .iv = "\xf3\xee\x5c\x62\xc4\x7c\xf0\x65" + "\x27\x08\xbd\xaf\xce\xec\x45\xb3", + .assoc = "\x02\xb8\xea\xca\x09\x1b\x9a\xec" + "\x47\x3e\xe9\xd4\xcc\xb5\x76", + .alen = 15, + .input = "", + .ilen = 0, + .result = "\xdd\xfa\x6c\x1f\x5d\x86\x87\x01" + "\x13\xe5\x73\x46\x46\xf2\x5c\xe1", + .rlen = 16, + }, { + .key = "\x20\x4a\x07\x99\x91\x58\xee\xfa" + "\x88\xab\x42\x1c\xc9\x47\xd7\x38", + .klen = 16, + .iv = "\x2f\x13\x95\x01\xd5\xf7\x99\x81" + "\xa8\xe2\x6f\x41\xc8\x10\x08\xb9", + .assoc = "\x3f\xdc\x24\x69\x19\x96\x43\x08" + "\xc8\x18\x9b\x65\xc6\xd9\x39\x3b", + .alen = 16, + .input = "", + .ilen = 0, + .result = "\xa6\x1b\xb9\xd7\x5e\x3c\xcf\xac" + "\xa9\x21\x45\x0b\x16\x52\xf7\xe1", + .rlen = 16, + }, { + .key = "\x5d\x6f\x41\x39\xa1\xd4\x97\x16" + "\x09\x85\xf4\xae\xc3\x6b\x9a\x3e", + .klen = 16, + .iv = "\x6c\x38\xcf\xa1\xe5\x73\x41\x9d" + "\x29\xbc\x21\xd2\xc2\x35\xcb\xbf", + .assoc = "\x7b\x01\x5d\x08\x29\x12\xec\x24" + "\x49\xf3\x4d\xf7\xc0\xfe\xfb\x41" + "\x3c", + .alen = 17, + .input = "", + .ilen = 0, + .result = "\x15\xff\xde\x3b\x34\xfc\xf6\xf9" + "\xbb\xa8\x62\xad\x0a\xf5\x48\x60", + .rlen = 16, + }, { + .key = "\x99\x93\x7a\xd8\xb1\x50\x40\x31" + "\x8a\x60\xa6\x3f\xbd\x90\x5d\x44", + .klen = 16, + .iv = "\xa8\x5c\x09\x40\xf5\xef\xea\xb8" + "\xaa\x96\xd3\x64\xbc\x59\x8d\xc6", + .assoc = "\xb8\x26\x97\xa8\x39\x8e\x94\x3f" + "\xca\xcd\xff\x88\xba\x22\xbe\x47" + "\x67\xba\x85\xf1\xbb\x30\x56\x26" + "\xaf\x0b\x02\x38\xcc\x44\xa7", + .alen = 31, + .input = "", + .ilen = 0, + .result = "\xd2\x9d\xf8\x3b\xd7\x84\xe9\x2d" + "\x4b\xef\x75\x16\x0a\x99\xae\x6b", + .rlen = 16, + }, { + .key = "\xd6\xb8\xb4\x77\xc1\xcb\xe9\x4d" + "\x0a\x3a\x58\xd1\xb7\xb4\x1f\x4a", + .klen = 16, + .iv = "\xe5\x81\x42\xdf\x05\x6a\x93\xd4" + "\x2b\x70\x85\xf5\xb6\x7d\x50\xcc", + .assoc = "\xf4\x4a\xd1\x47\x49\x09\x3d\x5b" + "\x4b\xa7\xb1\x19\xb4\x46\x81\x4d" + "\x91\x7c\x91\x75\xc0\xd0\xd8\x40" + "\x71\x39\xe1\x10\xa6\xa3\x46\x7a", + .alen = 32, + .input = "", + .ilen = 0, + .result = "\xe4\x8d\xa7\xa7\x45\xc1\x31\x4f" + "\xce\xfb\xaf\xd6\xc2\xe6\xee\xc0", + .rlen = 16, + }, { + .key = "\x12\xdd\xee\x17\xd1\x47\x92\x69" + "\x8b\x14\x0a\x62\xb1\xd9\xe2\x50", + .klen = 16, + .iv = "\x22\xa6\x7c\x7f\x15\xe6\x3c\xf0" + "\xac\x4b\x37\x86\xb0\xa2\x13\xd2", + .assoc = "\x31", + .alen = 1, + .input = "\x40", + .ilen = 1, + .result = "\xe2\x67\x38\x4f\xb9\xad\x7d\x38" + "\x01\xfe\x84\x14\x85\xf8\xd1\xe3" + "\x22", + .rlen = 17, + }, { + .key = "\x4f\x01\x27\xb6\xe1\xc3\x3a\x85" + "\x0c\xee\xbc\xf4\xab\xfd\xa5\x57", + .klen = 16, + .iv = "\x5e\xcb\xb6\x1e\x25\x62\xe4\x0c" + "\x2d\x25\xe9\x18\xaa\xc6\xd5\xd8", + .assoc = "\x6d\x94\x44\x86\x69\x00\x8f\x93" + "\x4d\x5b\x15\x3c\xa8\x8f\x06", + .alen = 15, + .input = "\x7c\x5d\xd3\xee\xad\x9f\x39\x1a" + "\x6d\x92\x42\x61\xa7\x58\x37", + .ilen = 15, + .result = "\x77\x32\x61\xeb\xb4\x33\x29\x92" + "\x29\x95\xc5\x8e\x85\x76\xab\xfc" + "\x07\x95\xa7\x44\x74\xf7\x22\xff" + "\xd8\xd8\x36\x3d\x8a\x7f\x9e", + .rlen = 31, + }, { + .key = "\x8b\x26\x61\x55\xf1\x3e\xe3\xa1" + "\x8d\xc8\x6e\x85\xa5\x21\x67\x5d", + .klen = 16, + .iv = "\x9b\xef\xf0\xbd\x35\xdd\x8d\x28" + "\xad\xff\x9b\xa9\xa4\xeb\x98\xdf", + .assoc = "\xaa\xb8\x7e\x25\x79\x7c\x37\xaf" + "\xce\x36\xc7\xce\xa2\xb4\xc9\x60", + .alen = 16, + .input = "\xb9\x82\x0c\x8d\xbd\x1b\xe2\x36" + "\xee\x6c\xf4\xf2\xa1\x7d\xf9\xe2", + .ilen = 16, + .result = "\xd8\xfd\x44\x45\xf6\x42\x12\x38" + "\xf2\x0b\xea\x4f\x9e\x11\x61\x07" + "\x48\x67\x98\x18\x9b\xd0\x0c\x59" + "\x67\xa4\x11\xb3\x2b\xd6\xc1\x70", + .rlen = 32, + }, { + .key = "\xc8\x4b\x9b\xf5\x01\xba\x8c\xbd" + "\x0e\xa3\x21\x16\x9f\x46\x2a\x63", + .klen = 16, + .iv = "\xd7\x14\x29\x5d\x45\x59\x36\x44" + "\x2e\xd9\x4d\x3b\x9e\x0f\x5b\xe5", + .assoc = "\xe6\xdd\xb8\xc4\x89\xf8\xe0\xca" + "\x4f\x10\x7a\x5f\x9c\xd8\x8b\x66" + "\x3b", + .alen = 17, + .input = "\xf5\xa6\x46\x2c\xce\x97\x8a\x51" + "\x6f\x46\xa6\x83\x9b\xa1\xbc\xe8" + "\x05", + .ilen = 17, + .result = "\xb1\xab\x53\x4e\xc7\x40\x16\xb6" + "\x71\x3a\x00\x9f\x41\x88\xb0\xb2" + "\x71\x83\x85\x5f\xc8\x79\x0a\x99" + "\x99\xdc\x89\x1c\x88\xd2\x3e\xf9" + "\x83", + .rlen = 33, + }, { + .key = "\x05\x70\xd5\x94\x12\x36\x35\xd8" + "\x8f\x7d\xd3\xa8\x99\x6a\xed\x69", + .klen = 16, + .iv = "\x14\x39\x63\xfc\x56\xd5\xdf\x5f" + "\xaf\xb3\xff\xcc\x98\x33\x1d\xeb", + .assoc = "\x23\x02\xf1\x64\x9a\x73\x89\xe6" + "\xd0\xea\x2c\xf1\x96\xfc\x4e\x6d" + "\x65\x48\xcb\x0a\xda\xf0\x62\xc0" + "\x38\x1d\x3b\x4a\xe9\x7e\x62", + .alen = 31, + .input = "\x32\xcb\x80\xcc\xde\x12\x33\x6d" + "\xf0\x20\x58\x15\x95\xc6\x7f\xee" + "\x2f\xf9\x4e\x2c\x1b\x98\x43\xc7" + "\x68\x28\x73\x40\x9f\x96\x4a", + .ilen = 31, + .result = "\x29\xc4\xf0\x03\xc1\x86\xdf\x06" + "\x5c\x7b\xef\x64\x87\x00\xd1\x37" + "\xa7\x08\xbc\x7f\x8f\x41\x54\xd0" + "\x3e\xf1\xc3\xa2\x96\x84\xdd\x2a" + "\x2d\x21\x30\xf9\x02\xdb\x06\x0c" + "\xf1\x5a\x66\x69\xe0\xca\x83", + .rlen = 47, + }, { + .key = "\x41\x94\x0e\x33\x22\xb1\xdd\xf4" + "\x10\x57\x85\x39\x93\x8f\xaf\x70", + .klen = 16, + .iv = "\x50\x5d\x9d\x9b\x66\x50\x88\x7b" + "\x30\x8e\xb1\x5e\x92\x58\xe0\xf1", + .assoc = "\x5f\x27\x2b\x03\xaa\xef\x32\x02" + "\x50\xc4\xde\x82\x90\x21\x11\x73" + "\x8f\x0a\xd6\x8f\xdf\x90\xe4\xda" + "\xf9\x4a\x1a\x23\xc3\xdd\x02\x81", + .alen = 32, + .input = "\x6e\xf0\xba\x6b\xee\x8e\xdc\x89" + "\x71\xfb\x0a\xa6\x8f\xea\x41\xf4" + "\x5a\xbb\x59\xb0\x20\x38\xc5\xe0" + "\x29\x56\x52\x19\x79\xf5\xe9\x37", + .ilen = 32, + .result = "\xe2\x2e\x44\xdf\xd3\x60\x6d\xb2" + "\x70\x57\x37\xc5\xc2\x4f\x8d\x14" + "\xc6\xbf\x8b\xec\xf5\x62\x67\xf2" + "\x2f\xa1\xe6\xd6\xa7\xb1\x8c\x54" + "\xe5\x6b\x49\xf9\x6e\x90\xc3\xaa" + "\x7a\x00\x2e\x4d\x7f\x31\x2e\x81", + .rlen = 48, + }, { + .key = "\x7e\xb9\x48\xd3\x32\x2d\x86\x10" + "\x91\x31\x37\xcb\x8d\xb3\x72\x76", + .klen = 16, + .iv = "\x8d\x82\xd6\x3b\x76\xcc\x30\x97" + "\xb1\x68\x63\xef\x8c\x7c\xa3\xf7", + .assoc = "\x9c\x4b\x65\xa2\xba\x6b\xdb\x1e" + "\xd1\x9e\x90\x13\x8a\x45\xd3\x79" + "\xba\xcd\xe2\x13\xe4\x30\x66\xf4" + "\xba\x78\xf9\xfb\x9d\x3c\xa1\x58" + "\x1a", + .alen = 33, + .input = "\xab\x14\xf3\x0a\xfe\x0a\x85\xa5" + "\xf2\xd5\xbc\x38\x89\x0e\x04\xfb" + "\x84\x7d\x65\x34\x25\xd8\x47\xfa" + "\xeb\x83\x31\xf1\x54\x54\x89\x0d" + "\x9d\x4d\x54\x51\x84\x61\xf6\x8e" + "\x03\x31\xf2\x25\x16\xcc\xaa\xc6" + "\x75\x73\x20\x30\x59\x54\xb2\xf0" + "\x3a\x4b\xe0\x23\x8e\xa6\x08\x35" + "\x8a", + .ilen = 65, + .result = "\xc7\xca\x26\x61\x57\xee\xa2\xb9" + "\xb1\x37\xde\x95\x06\x90\x11\x08" + "\x4d\x30\x9f\x24\xc0\x56\xb7\xe1" + "\x0b\x9f\xd2\x57\xe9\xd2\xb1\x76" + "\x56\x9a\xb4\x58\xc5\x08\xfc\xb5" + "\xf2\x31\x9b\xc9\xcd\xb3\x64\xdb" + "\x6f\x50\xbf\xf4\x73\x9d\xfb\x6b" + "\xef\x35\x25\x48\xed\xcf\x29\xa8" + "\xac\xc3\xb9\xcb\x61\x8f\x73\x92" + "\x2c\x7a\x6f\xda\xf9\x09\x6f\xe1" + "\xc4", + .rlen = 81, + }, { + .key = "\xba\xde\x82\x72\x42\xa9\x2f\x2c" + "\x12\x0b\xe9\x5c\x87\xd7\x35\x7c", + .klen = 16, + .iv = "\xc9\xa7\x10\xda\x86\x48\xd9\xb3" + "\x32\x42\x15\x80\x85\xa1\x65\xfe", + .assoc = "\xd8\x70\x9f\x42\xca\xe6\x83\x3a" + "\x52\x79\x42\xa5\x84\x6a\x96\x7f" + "\xe4\x8f\xed\x97\xe9\xd0\xe8\x0d" + "\x7c\xa6\xd8\xd4\x77\x9b\x40\x2e" + "\x28\xce\x57\x34\xcd\x6e\x84\x4c" + "\x17\x3c\xe1\xb2\xa8\x0b\xbb\xf1" + "\x96\x41\x0d\x69\xe8\x54\x0a\xc8" + "\x15\x4e\x91\x92\x89\x4b\xb7\x9b" + "\x21", + .alen = 65, + .input = "\xe8\x39\x2d\xaa\x0e\x85\x2d\xc1" + "\x72\xaf\x6e\xc9\x82\x33\xc7\x01" + "\xaf\x40\x70\xb8\x2a\x78\xc9\x14" + "\xac\xb1\x10\xca\x2e\xb3\x28\xe4" + "\xac", + .ilen = 33, + .result = "\x57\xcd\x3d\x46\xc5\xf9\x68\x3b" + "\x2c\x0f\xb4\x7e\x7b\x64\x3e\x40" + "\xf3\x78\x63\x34\x89\x79\x39\x6b" + "\x61\x64\x4a\x9a\xfa\x70\xa4\xd3" + "\x54\x0b\xea\x05\xa6\x95\x64\xed" + "\x3d\x69\xa2\x0c\x27\x56\x2f\x34" + "\x66", + .rlen = 49, + }, { + .key = "\xf7\x02\xbb\x11\x52\x24\xd8\x48" + "\x93\xe6\x9b\xee\x81\xfc\xf7\x82", + .klen = 16, + .iv = "\x06\xcc\x4a\x79\x96\xc3\x82\xcf" + "\xb3\x1c\xc7\x12\x7f\xc5\x28\x04", + .assoc = "\x15\x95\xd8\xe1\xda\x62\x2c\x56" + "\xd3\x53\xf4\x36\x7e\x8e\x59\x85", + .alen = 16, + .input = "\x24\x5e\x67\x49\x1e\x01\xd6\xdd" + "\xf3\x89\x20\x5b\x7c\x57\x89\x07", + .ilen = 16, + .result = "\xfc\x85\x06\x28\x8f\xe8\x23\x1f" + "\x33\x98\x87\xde\x08\xb6\xb6\xae" + "\x3e\xa4\xf8\x19\xf1\x92\x60\x39" + "\xb9\x6b\x3f\xdf\xc8\xcb\x30", + .rlen = 31, + }, { + .key = "\x33\x27\xf5\xb1\x62\xa0\x80\x63" + "\x14\xc0\x4d\x7f\x7b\x20\xba\x89", + .klen = 16, + .iv = "\x42\xf0\x84\x19\xa6\x3f\x2b\xea" + "\x34\xf6\x79\xa3\x79\xe9\xeb\x0a", + .assoc = "\x51\xb9\x12\x80\xea\xde\xd5\x71" + "\x54\x2d\xa6\xc8\x78\xb2\x1b\x8c", + .alen = 16, + .input = "\x61\x83\xa0\xe8\x2e\x7d\x7f\xf8" + "\x74\x63\xd2\xec\x76\x7c\x4c\x0d", + .ilen = 16, + .result = "\x74\x7d\x70\x07\xe9\xba\x01\xee" + "\x6c\xc6\x6f\x50\x25\x33\xbe\x50" + "\x17\xb8\x17\x62\xed\x80\xa2\xf5" + "\x03\xde\x85\x71\x5d\x34", + .rlen = 30, + }, { + .key = "\x70\x4c\x2f\x50\x72\x1c\x29\x7f" + "\x95\x9a\xff\x10\x75\x45\x7d\x8f", + .klen = 16, + .iv = "\x7f\x15\xbd\xb8\xb6\xba\xd3\x06" + "\xb5\xd1\x2b\x35\x73\x0e\xad\x10", + .assoc = "\x8e\xde\x4c\x20\xfa\x59\x7e\x8d" + "\xd5\x07\x58\x59\x72\xd7\xde\x92", + .alen = 16, + .input = "\x9d\xa7\xda\x88\x3e\xf8\x28\x14" + "\xf5\x3e\x85\x7d\x70\xa0\x0f\x13", + .ilen = 16, + .result = "\xf4\xb3\x85\xf9\xac\xde\xb1\x38" + "\x29\xfd\x6c\x7c\x49\xe5\x1d\xaf" + "\xba\xea\xd4\xfa\x3f\x11\x33\x98", + .rlen = 24, + }, { + .key = "\xac\x70\x69\xef\x82\x97\xd2\x9b" + "\x15\x74\xb1\xa2\x6f\x69\x3f\x95", + .klen = 16, + .iv = "\xbb\x3a\xf7\x57\xc6\x36\x7c\x22" + "\x36\xab\xde\xc6\x6d\x32\x70\x17", + .assoc = "\xcb\x03\x85\xbf\x0a\xd5\x26\xa9" + "\x56\xe1\x0a\xeb\x6c\xfb\xa1\x98", + .alen = 16, + .input = "\xda\xcc\x14\x27\x4e\x74\xd1\x30" + "\x76\x18\x37\x0f\x6a\xc4\xd1\x1a", + .ilen = 16, + .result = "\xe6\x5c\x49\x4f\x78\xf3\x62\x86" + "\xe1\xb7\xa5\xc3\x32\x88\x3c\x8c" + "\x6e", + .rlen = 17, + }, +}; + +static const struct aead_testvec morus640_dec_tv_template[] = { + { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 16, + .iv = "\x0f\xc9\x8e\x67\x44\x9e\xaa\x86" + "\x20\x36\x2c\x24\xfe\xc9\x30\x81", + .assoc = "", + .alen = 0, + .input = "\x89\x62\x7d\xf3\x07\x9d\x52\x05" + "\x53\xc3\x04\x60\x93\xb4\x37\x9a", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x3c\x24\x39\x9f\x10\x7b\xa8\x1b" + "\x80\xda\xb2\x91\xf9\x24\xc2\x06", + .klen = 16, + .iv = "\x4b\xed\xc8\x07\x54\x1a\x52\xa2" + "\xa1\x10\xde\xb5\xf8\xed\xf3\x87", + .assoc = "", + .alen = 0, + .input = "\xa8\x8d\xe4\x90\xb5\x50\x8f\x78" + "\xb6\x10\x9a\x59\x5f\x61\x37\x70" + "\x09", + .ilen = 17, + .result = "\x69", + .rlen = 1, + }, { + .key = "\x79\x49\x73\x3e\x20\xf7\x51\x37" + "\x01\xb4\x64\x22\xf3\x48\x85\x0c", + .klen = 16, + .iv = "\x88\x12\x01\xa6\x64\x96\xfb\xbe" + "\x22\xea\x90\x47\xf2\x11\xb5\x8e", + .assoc = "", + .alen = 0, + .input = "\x76\xdd\xb9\x05\x3d\xce\x61\x38" + "\xf3\xef\xf7\xe5\xd7\xfd\x70\xa5" + "\xcf\x9d\x64\xb8\x0a\x9f\xfd\x8b" + "\xd4\x6e\xfe\xd9\xc8\x63\x4b", + .ilen = 31, + .result = "\xa6\xa4\x1e\x76\xec\xd4\x50\xcc" + "\x62\x58\xe9\x8f\xef\xa4\x17", + .rlen = 15, + }, { + .key = "\xb5\x6e\xad\xdd\x30\x72\xfa\x53" + "\x82\x8e\x16\xb4\xed\x6d\x47\x12", + .klen = 16, + .iv = "\xc4\x37\x3b\x45\x74\x11\xa4\xda" + "\xa2\xc5\x42\xd8\xec\x36\x78\x94", + .assoc = "", + .alen = 0, + .input = "\xdc\x72\xe8\x14\xfb\x63\xad\x72" + "\x1f\x57\x9a\x1f\x88\x81\xdb\xd6" + "\xc1\x91\x9d\xb9\x25\xc4\x99\x4c" + "\x97\xcd\x8a\x0c\x9d\x68\x00\x1c", + .ilen = 32, + .result = "\xe2\xc9\x58\x15\xfc\x4f\xf8\xe8" + "\xe3\x32\x9b\x21\xe9\xc8\xd9\x97", + .rlen = 16, + }, { + .key = "\xf2\x92\xe6\x7d\x40\xee\xa3\x6f" + "\x03\x68\xc8\x45\xe7\x91\x0a\x18", + .klen = 16, + .iv = "\x01\x5c\x75\xe5\x84\x8d\x4d\xf6" + "\x23\x9f\xf4\x6a\xe6\x5a\x3b\x9a", + .assoc = "", + .alen = 0, + .input = "\x6b\x4f\x3b\x90\x9a\xa2\xb3\x82" + "\x0a\xb8\x55\xee\xeb\x73\x4d\x7f" + "\x54\x11\x3a\x8a\x31\xa3\xb5\xf2" + "\xcd\x49\xdb\xf3\xee\x26\xbd\xa2" + "\x0d", + .ilen = 33, + .result = "\x1f\xee\x92\xb4\x0c\xcb\xa1\x04" + "\x64\x0c\x4d\xb2\xe3\xec\x9c\x9d" + "\x09", + .rlen = 17, + }, { + .key = "\x2e\xb7\x20\x1c\x50\x6a\x4b\x8b" + "\x84\x42\x7a\xd7\xe1\xb5\xcd\x1f", + .klen = 16, + .iv = "\x3d\x80\xae\x84\x94\x09\xf6\x12" + "\xa4\x79\xa6\xfb\xe0\x7f\xfd\xa0", + .assoc = "", + .alen = 0, + .input = "\x59\xd1\x0f\x6b\xee\x27\x84\x92" + "\xb7\xa9\xb5\xdd\x02\xa4\x12\xa5" + "\x50\x32\xb4\x9a\x2e\x35\x83\x55" + "\x36\x12\x12\xed\xa3\x31\xc5\x30" + "\xa7\xe2\x4a\x6d\x05\x59\x43\x91" + "\x75\xfa\x6c\x17\xc6\x73\xca", + .ilen = 47, + .result = "\x5c\x13\xcb\x54\x1c\x47\x4a\x1f" + "\xe5\xe6\xff\x44\xdd\x11\x5f\xa3" + "\x33\xdd\xc2\xf8\xdd\x18\x2b\x93" + "\x57\x05\x01\x1c\x66\x22\xd3", + .rlen = 31, + }, { + .key = "\x6b\xdc\x5a\xbb\x60\xe5\xf4\xa6" + "\x05\x1d\x2c\x68\xdb\xda\x8f\x25", + .klen = 16, + .iv = "\x7a\xa5\xe8\x23\xa4\x84\x9e\x2d" + "\x25\x53\x58\x8c\xda\xa3\xc0\xa6", + .assoc = "", + .alen = 0, + .input = "\xdb\x49\x68\x0f\x91\x5b\x21\xb1" + "\xcf\x50\xb2\x4c\x32\xe1\xa6\x69" + "\xc0\xfb\x44\x1f\xa0\x9a\xeb\x39" + "\x1b\xde\x68\x38\xcc\x27\x52\xc5" + "\xf6\x3e\x74\xea\x66\x5b\x5f\x0c" + "\x65\x9e\x58\xe6\x52\xa2\xfe\x59", + .ilen = 48, + .result = "\x98\x37\x05\xf3\x2c\xc2\xf3\x3b" + "\x66\xc0\xb1\xd5\xd7\x35\x21\xaa" + "\x5d\x9f\xce\x7c\xe2\xb8\xad\xad" + "\x19\x33\xe0\xf4\x40\x81\x72\x28", + .rlen = 32, + }, { + .key = "\xa7\x00\x93\x5b\x70\x61\x9d\xc2" + "\x86\xf7\xde\xfa\xd5\xfe\x52\x2b", + .klen = 16, + .iv = "\xb6\xca\x22\xc3\xb4\x00\x47\x49" + "\xa6\x2d\x0a\x1e\xd4\xc7\x83\xad", + .assoc = "\xc5", + .alen = 1, + .input = "\x56\xe7\x24\x52\xdd\x95\x60\x5b" + "\x09\x48\x39\x69\x9c\xb3\x62\x46", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xe4\x25\xcd\xfa\x80\xdd\x46\xde" + "\x07\xd1\x90\x8b\xcf\x23\x15\x31", + .klen = 16, + .iv = "\xf3\xee\x5c\x62\xc4\x7c\xf0\x65" + "\x27\x08\xbd\xaf\xce\xec\x45\xb3", + .assoc = "\x02\xb8\xea\xca\x09\x1b\x9a\xec" + "\x47\x3e\xe9\xd4\xcc\xb5\x76", + .alen = 15, + .input = "\xdd\xfa\x6c\x1f\x5d\x86\x87\x01" + "\x13\xe5\x73\x46\x46\xf2\x5c\xe1", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x20\x4a\x07\x99\x91\x58\xee\xfa" + "\x88\xab\x42\x1c\xc9\x47\xd7\x38", + .klen = 16, + .iv = "\x2f\x13\x95\x01\xd5\xf7\x99\x81" + "\xa8\xe2\x6f\x41\xc8\x10\x08\xb9", + .assoc = "\x3f\xdc\x24\x69\x19\x96\x43\x08" + "\xc8\x18\x9b\x65\xc6\xd9\x39\x3b", + .alen = 16, + .input = "\xa6\x1b\xb9\xd7\x5e\x3c\xcf\xac" + "\xa9\x21\x45\x0b\x16\x52\xf7\xe1", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x5d\x6f\x41\x39\xa1\xd4\x97\x16" + "\x09\x85\xf4\xae\xc3\x6b\x9a\x3e", + .klen = 16, + .iv = "\x6c\x38\xcf\xa1\xe5\x73\x41\x9d" + "\x29\xbc\x21\xd2\xc2\x35\xcb\xbf", + .assoc = "\x7b\x01\x5d\x08\x29\x12\xec\x24" + "\x49\xf3\x4d\xf7\xc0\xfe\xfb\x41" + "\x3c", + .alen = 17, + .input = "\x15\xff\xde\x3b\x34\xfc\xf6\xf9" + "\xbb\xa8\x62\xad\x0a\xf5\x48\x60", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x99\x93\x7a\xd8\xb1\x50\x40\x31" + "\x8a\x60\xa6\x3f\xbd\x90\x5d\x44", + .klen = 16, + .iv = "\xa8\x5c\x09\x40\xf5\xef\xea\xb8" + "\xaa\x96\xd3\x64\xbc\x59\x8d\xc6", + .assoc = "\xb8\x26\x97\xa8\x39\x8e\x94\x3f" + "\xca\xcd\xff\x88\xba\x22\xbe\x47" + "\x67\xba\x85\xf1\xbb\x30\x56\x26" + "\xaf\x0b\x02\x38\xcc\x44\xa7", + .alen = 31, + .input = "\xd2\x9d\xf8\x3b\xd7\x84\xe9\x2d" + "\x4b\xef\x75\x16\x0a\x99\xae\x6b", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xd6\xb8\xb4\x77\xc1\xcb\xe9\x4d" + "\x0a\x3a\x58\xd1\xb7\xb4\x1f\x4a", + .klen = 16, + .iv = "\xe5\x81\x42\xdf\x05\x6a\x93\xd4" + "\x2b\x70\x85\xf5\xb6\x7d\x50\xcc", + .assoc = "\xf4\x4a\xd1\x47\x49\x09\x3d\x5b" + "\x4b\xa7\xb1\x19\xb4\x46\x81\x4d" + "\x91\x7c\x91\x75\xc0\xd0\xd8\x40" + "\x71\x39\xe1\x10\xa6\xa3\x46\x7a", + .alen = 32, + .input = "\xe4\x8d\xa7\xa7\x45\xc1\x31\x4f" + "\xce\xfb\xaf\xd6\xc2\xe6\xee\xc0", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x12\xdd\xee\x17\xd1\x47\x92\x69" + "\x8b\x14\x0a\x62\xb1\xd9\xe2\x50", + .klen = 16, + .iv = "\x22\xa6\x7c\x7f\x15\xe6\x3c\xf0" + "\xac\x4b\x37\x86\xb0\xa2\x13\xd2", + .assoc = "\x31", + .alen = 1, + .input = "\xe2\x67\x38\x4f\xb9\xad\x7d\x38" + "\x01\xfe\x84\x14\x85\xf8\xd1\xe3" + "\x22", + .ilen = 17, + .result = "\x40", + .rlen = 1, + }, { + .key = "\x4f\x01\x27\xb6\xe1\xc3\x3a\x85" + "\x0c\xee\xbc\xf4\xab\xfd\xa5\x57", + .klen = 16, + .iv = "\x5e\xcb\xb6\x1e\x25\x62\xe4\x0c" + "\x2d\x25\xe9\x18\xaa\xc6\xd5\xd8", + .assoc = "\x6d\x94\x44\x86\x69\x00\x8f\x93" + "\x4d\x5b\x15\x3c\xa8\x8f\x06", + .alen = 15, + .input = "\x77\x32\x61\xeb\xb4\x33\x29\x92" + "\x29\x95\xc5\x8e\x85\x76\xab\xfc" + "\x07\x95\xa7\x44\x74\xf7\x22\xff" + "\xd8\xd8\x36\x3d\x8a\x7f\x9e", + .ilen = 31, + .result = "\x7c\x5d\xd3\xee\xad\x9f\x39\x1a" + "\x6d\x92\x42\x61\xa7\x58\x37", + .rlen = 15, + }, { + .key = "\x8b\x26\x61\x55\xf1\x3e\xe3\xa1" + "\x8d\xc8\x6e\x85\xa5\x21\x67\x5d", + .klen = 16, + .iv = "\x9b\xef\xf0\xbd\x35\xdd\x8d\x28" + "\xad\xff\x9b\xa9\xa4\xeb\x98\xdf", + .assoc = "\xaa\xb8\x7e\x25\x79\x7c\x37\xaf" + "\xce\x36\xc7\xce\xa2\xb4\xc9\x60", + .alen = 16, + .input = "\xd8\xfd\x44\x45\xf6\x42\x12\x38" + "\xf2\x0b\xea\x4f\x9e\x11\x61\x07" + "\x48\x67\x98\x18\x9b\xd0\x0c\x59" + "\x67\xa4\x11\xb3\x2b\xd6\xc1\x70", + .ilen = 32, + .result = "\xb9\x82\x0c\x8d\xbd\x1b\xe2\x36" + "\xee\x6c\xf4\xf2\xa1\x7d\xf9\xe2", + .rlen = 16, + }, { + .key = "\xc8\x4b\x9b\xf5\x01\xba\x8c\xbd" + "\x0e\xa3\x21\x16\x9f\x46\x2a\x63", + .klen = 16, + .iv = "\xd7\x14\x29\x5d\x45\x59\x36\x44" + "\x2e\xd9\x4d\x3b\x9e\x0f\x5b\xe5", + .assoc = "\xe6\xdd\xb8\xc4\x89\xf8\xe0\xca" + "\x4f\x10\x7a\x5f\x9c\xd8\x8b\x66" + "\x3b", + .alen = 17, + .input = "\xb1\xab\x53\x4e\xc7\x40\x16\xb6" + "\x71\x3a\x00\x9f\x41\x88\xb0\xb2" + "\x71\x83\x85\x5f\xc8\x79\x0a\x99" + "\x99\xdc\x89\x1c\x88\xd2\x3e\xf9" + "\x83", + .ilen = 33, + .result = "\xf5\xa6\x46\x2c\xce\x97\x8a\x51" + "\x6f\x46\xa6\x83\x9b\xa1\xbc\xe8" + "\x05", + .rlen = 17, + }, { + .key = "\x05\x70\xd5\x94\x12\x36\x35\xd8" + "\x8f\x7d\xd3\xa8\x99\x6a\xed\x69", + .klen = 16, + .iv = "\x14\x39\x63\xfc\x56\xd5\xdf\x5f" + "\xaf\xb3\xff\xcc\x98\x33\x1d\xeb", + .assoc = "\x23\x02\xf1\x64\x9a\x73\x89\xe6" + "\xd0\xea\x2c\xf1\x96\xfc\x4e\x6d" + "\x65\x48\xcb\x0a\xda\xf0\x62\xc0" + "\x38\x1d\x3b\x4a\xe9\x7e\x62", + .alen = 31, + .input = "\x29\xc4\xf0\x03\xc1\x86\xdf\x06" + "\x5c\x7b\xef\x64\x87\x00\xd1\x37" + "\xa7\x08\xbc\x7f\x8f\x41\x54\xd0" + "\x3e\xf1\xc3\xa2\x96\x84\xdd\x2a" + "\x2d\x21\x30\xf9\x02\xdb\x06\x0c" + "\xf1\x5a\x66\x69\xe0\xca\x83", + .ilen = 47, + .result = "\x32\xcb\x80\xcc\xde\x12\x33\x6d" + "\xf0\x20\x58\x15\x95\xc6\x7f\xee" + "\x2f\xf9\x4e\x2c\x1b\x98\x43\xc7" + "\x68\x28\x73\x40\x9f\x96\x4a", + .rlen = 31, + }, { + .key = "\x41\x94\x0e\x33\x22\xb1\xdd\xf4" + "\x10\x57\x85\x39\x93\x8f\xaf\x70", + .klen = 16, + .iv = "\x50\x5d\x9d\x9b\x66\x50\x88\x7b" + "\x30\x8e\xb1\x5e\x92\x58\xe0\xf1", + .assoc = "\x5f\x27\x2b\x03\xaa\xef\x32\x02" + "\x50\xc4\xde\x82\x90\x21\x11\x73" + "\x8f\x0a\xd6\x8f\xdf\x90\xe4\xda" + "\xf9\x4a\x1a\x23\xc3\xdd\x02\x81", + .alen = 32, + .input = "\xe2\x2e\x44\xdf\xd3\x60\x6d\xb2" + "\x70\x57\x37\xc5\xc2\x4f\x8d\x14" + "\xc6\xbf\x8b\xec\xf5\x62\x67\xf2" + "\x2f\xa1\xe6\xd6\xa7\xb1\x8c\x54" + "\xe5\x6b\x49\xf9\x6e\x90\xc3\xaa" + "\x7a\x00\x2e\x4d\x7f\x31\x2e\x81", + .ilen = 48, + .result = "\x6e\xf0\xba\x6b\xee\x8e\xdc\x89" + "\x71\xfb\x0a\xa6\x8f\xea\x41\xf4" + "\x5a\xbb\x59\xb0\x20\x38\xc5\xe0" + "\x29\x56\x52\x19\x79\xf5\xe9\x37", + .rlen = 32, + }, { + .key = "\x7e\xb9\x48\xd3\x32\x2d\x86\x10" + "\x91\x31\x37\xcb\x8d\xb3\x72\x76", + .klen = 16, + .iv = "\x8d\x82\xd6\x3b\x76\xcc\x30\x97" + "\xb1\x68\x63\xef\x8c\x7c\xa3\xf7", + .assoc = "\x9c\x4b\x65\xa2\xba\x6b\xdb\x1e" + "\xd1\x9e\x90\x13\x8a\x45\xd3\x79" + "\xba\xcd\xe2\x13\xe4\x30\x66\xf4" + "\xba\x78\xf9\xfb\x9d\x3c\xa1\x58" + "\x1a", + .alen = 33, + .input = "\xc7\xca\x26\x61\x57\xee\xa2\xb9" + "\xb1\x37\xde\x95\x06\x90\x11\x08" + "\x4d\x30\x9f\x24\xc0\x56\xb7\xe1" + "\x0b\x9f\xd2\x57\xe9\xd2\xb1\x76" + "\x56\x9a\xb4\x58\xc5\x08\xfc\xb5" + "\xf2\x31\x9b\xc9\xcd\xb3\x64\xdb" + "\x6f\x50\xbf\xf4\x73\x9d\xfb\x6b" + "\xef\x35\x25\x48\xed\xcf\x29\xa8" + "\xac\xc3\xb9\xcb\x61\x8f\x73\x92" + "\x2c\x7a\x6f\xda\xf9\x09\x6f\xe1" + "\xc4", + .ilen = 81, + .result = "\xab\x14\xf3\x0a\xfe\x0a\x85\xa5" + "\xf2\xd5\xbc\x38\x89\x0e\x04\xfb" + "\x84\x7d\x65\x34\x25\xd8\x47\xfa" + "\xeb\x83\x31\xf1\x54\x54\x89\x0d" + "\x9d\x4d\x54\x51\x84\x61\xf6\x8e" + "\x03\x31\xf2\x25\x16\xcc\xaa\xc6" + "\x75\x73\x20\x30\x59\x54\xb2\xf0" + "\x3a\x4b\xe0\x23\x8e\xa6\x08\x35" + "\x8a", + .rlen = 65, + }, { + .key = "\xba\xde\x82\x72\x42\xa9\x2f\x2c" + "\x12\x0b\xe9\x5c\x87\xd7\x35\x7c", + .klen = 16, + .iv = "\xc9\xa7\x10\xda\x86\x48\xd9\xb3" + "\x32\x42\x15\x80\x85\xa1\x65\xfe", + .assoc = "\xd8\x70\x9f\x42\xca\xe6\x83\x3a" + "\x52\x79\x42\xa5\x84\x6a\x96\x7f" + "\xe4\x8f\xed\x97\xe9\xd0\xe8\x0d" + "\x7c\xa6\xd8\xd4\x77\x9b\x40\x2e" + "\x28\xce\x57\x34\xcd\x6e\x84\x4c" + "\x17\x3c\xe1\xb2\xa8\x0b\xbb\xf1" + "\x96\x41\x0d\x69\xe8\x54\x0a\xc8" + "\x15\x4e\x91\x92\x89\x4b\xb7\x9b" + "\x21", + .alen = 65, + .input = "\x57\xcd\x3d\x46\xc5\xf9\x68\x3b" + "\x2c\x0f\xb4\x7e\x7b\x64\x3e\x40" + "\xf3\x78\x63\x34\x89\x79\x39\x6b" + "\x61\x64\x4a\x9a\xfa\x70\xa4\xd3" + "\x54\x0b\xea\x05\xa6\x95\x64\xed" + "\x3d\x69\xa2\x0c\x27\x56\x2f\x34" + "\x66", + .ilen = 49, + .result = "\xe8\x39\x2d\xaa\x0e\x85\x2d\xc1" + "\x72\xaf\x6e\xc9\x82\x33\xc7\x01" + "\xaf\x40\x70\xb8\x2a\x78\xc9\x14" + "\xac\xb1\x10\xca\x2e\xb3\x28\xe4" + "\xac", + .rlen = 33, + }, { + .key = "\xf7\x02\xbb\x11\x52\x24\xd8\x48" + "\x93\xe6\x9b\xee\x81\xfc\xf7\x82", + .klen = 16, + .iv = "\x06\xcc\x4a\x79\x96\xc3\x82\xcf" + "\xb3\x1c\xc7\x12\x7f\xc5\x28\x04", + .assoc = "\x15\x95\xd8\xe1\xda\x62\x2c\x56" + "\xd3\x53\xf4\x36\x7e\x8e\x59\x85", + .alen = 16, + .input = "\xfc\x85\x06\x28\x8f\xe8\x23\x1f" + "\x33\x98\x87\xde\x08\xb6\xb6\xae" + "\x3e\xa4\xf8\x19\xf1\x92\x60\x39" + "\xb9\x6b\x3f\xdf\xc8\xcb\x30", + .ilen = 31, + .result = "\x24\x5e\x67\x49\x1e\x01\xd6\xdd" + "\xf3\x89\x20\x5b\x7c\x57\x89\x07", + .rlen = 16, + }, { + .key = "\x33\x27\xf5\xb1\x62\xa0\x80\x63" + "\x14\xc0\x4d\x7f\x7b\x20\xba\x89", + .klen = 16, + .iv = "\x42\xf0\x84\x19\xa6\x3f\x2b\xea" + "\x34\xf6\x79\xa3\x79\xe9\xeb\x0a", + .assoc = "\x51\xb9\x12\x80\xea\xde\xd5\x71" + "\x54\x2d\xa6\xc8\x78\xb2\x1b\x8c", + .alen = 16, + .input = "\x74\x7d\x70\x07\xe9\xba\x01\xee" + "\x6c\xc6\x6f\x50\x25\x33\xbe\x50" + "\x17\xb8\x17\x62\xed\x80\xa2\xf5" + "\x03\xde\x85\x71\x5d\x34", + .ilen = 30, + .result = "\x61\x83\xa0\xe8\x2e\x7d\x7f\xf8" + "\x74\x63\xd2\xec\x76\x7c\x4c\x0d", + .rlen = 16, + }, { + .key = "\x70\x4c\x2f\x50\x72\x1c\x29\x7f" + "\x95\x9a\xff\x10\x75\x45\x7d\x8f", + .klen = 16, + .iv = "\x7f\x15\xbd\xb8\xb6\xba\xd3\x06" + "\xb5\xd1\x2b\x35\x73\x0e\xad\x10", + .assoc = "\x8e\xde\x4c\x20\xfa\x59\x7e\x8d" + "\xd5\x07\x58\x59\x72\xd7\xde\x92", + .alen = 16, + .input = "\xf4\xb3\x85\xf9\xac\xde\xb1\x38" + "\x29\xfd\x6c\x7c\x49\xe5\x1d\xaf" + "\xba\xea\xd4\xfa\x3f\x11\x33\x98", + .ilen = 24, + .result = "\x9d\xa7\xda\x88\x3e\xf8\x28\x14" + "\xf5\x3e\x85\x7d\x70\xa0\x0f\x13", + .rlen = 16, + }, { + .key = "\xac\x70\x69\xef\x82\x97\xd2\x9b" + "\x15\x74\xb1\xa2\x6f\x69\x3f\x95", + .klen = 16, + .iv = "\xbb\x3a\xf7\x57\xc6\x36\x7c\x22" + "\x36\xab\xde\xc6\x6d\x32\x70\x17", + .assoc = "\xcb\x03\x85\xbf\x0a\xd5\x26\xa9" + "\x56\xe1\x0a\xeb\x6c\xfb\xa1\x98", + .alen = 16, + .input = "\xe6\x5c\x49\x4f\x78\xf3\x62\x86" + "\xe1\xb7\xa5\xc3\x32\x88\x3c\x8c" + "\x6e", + .ilen = 17, + .result = "\xda\xcc\x14\x27\x4e\x74\xd1\x30" + "\x76\x18\x37\x0f\x6a\xc4\xd1\x1a", + .rlen = 16, + }, +}; + +/* + * MORUS-1280 test vectors - generated via reference implementation from + * SUPERCOP (https://bench.cr.yp.to/supercop.html): + * + * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz + * (see crypto_aead/morus1280128v2/ and crypto_aead/morus1280256v2/ ) + */ +static const struct aead_testvec morus1280_enc_tv_template[] = { + { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 16, + .iv = "\x0f\xc9\x8e\x67\x44\x9e\xaa\x86" + "\x20\x36\x2c\x24\xfe\xc9\x30\x81", + .assoc = "", + .alen = 0, + .input = "", + .ilen = 0, + .result = "\x91\x85\x0f\xf5\x52\x9e\xce\xce" + "\x65\x99\xc7\xbf\xd3\x76\xe8\x98", + .rlen = 16, + }, { + .key = "\x3c\x24\x39\x9f\x10\x7b\xa8\x1b" + "\x80\xda\xb2\x91\xf9\x24\xc2\x06", + .klen = 16, + .iv = "\x4b\xed\xc8\x07\x54\x1a\x52\xa2" + "\xa1\x10\xde\xb5\xf8\xed\xf3\x87", + .assoc = "", + .alen = 0, + .input = "\x69", + .ilen = 1, + .result = "\x88\xc3\x4c\xf0\x2f\x43\x76\x13" + "\x96\xda\x76\x34\x33\x4e\xd5\x39" + "\x73", + .rlen = 17, + }, { + .key = "\x79\x49\x73\x3e\x20\xf7\x51\x37" + "\x01\xb4\x64\x22\xf3\x48\x85\x0c", + .klen = 16, + .iv = "\x88\x12\x01\xa6\x64\x96\xfb\xbe" + "\x22\xea\x90\x47\xf2\x11\xb5\x8e", + .assoc = "", + .alen = 0, + .input = "\xa6\xa4\x1e\x76\xec\xd4\x50\xcc" + "\x62\x58\xe9\x8f\xef\xa4\x17\x91" + "\xb4\x96\x9f\x6b\xce\x38\xa5\x46" + "\x13\x7d\x64\x93\xd7\x05\xf5", + .ilen = 31, + .result = "\x3e\x5c\x3b\x58\x3b\x7d\x2a\x22" + "\x75\x0b\x24\xa6\x0e\xc3\xde\x52" + "\x97\x0b\x64\xd4\xce\x90\x52\xf7" + "\xef\xdb\x6a\x38\xd2\xa8\xa1\x0d" + "\xe0\x61\x33\x24\xc6\x4d\x51\xbc" + "\xa4\x21\x74\xcf\x19\x16\x59", + .rlen = 47, + }, { + .key = "\xb5\x6e\xad\xdd\x30\x72\xfa\x53" + "\x82\x8e\x16\xb4\xed\x6d\x47\x12", + .klen = 16, + .iv = "\xc4\x37\x3b\x45\x74\x11\xa4\xda" + "\xa2\xc5\x42\xd8\xec\x36\x78\x94", + .assoc = "", + .alen = 0, + .input = "\xe2\xc9\x58\x15\xfc\x4f\xf8\xe8" + "\xe3\x32\x9b\x21\xe9\xc8\xd9\x97" + "\xde\x58\xab\xf0\xd3\xd8\x27\x60" + "\xd5\xaa\x43\x6b\xb1\x64\x95\xa4", + .ilen = 32, + .result = "\x30\x82\x9c\x2b\x67\xcb\xf9\x1f" + "\xde\x9f\x77\xb2\xda\x92\x61\x5c" + "\x09\x0b\x2d\x9a\x26\xaa\x1c\x06" + "\xab\x74\xb7\x2b\x95\x5f\x9f\xa1" + "\x9a\xff\x50\xa0\xa2\xff\xc5\xad" + "\x21\x8e\x84\x5c\x12\x61\xb2\xae", + .rlen = 48, + }, { + .key = "\xf2\x92\xe6\x7d\x40\xee\xa3\x6f" + "\x03\x68\xc8\x45\xe7\x91\x0a\x18", + .klen = 16, + .iv = "\x01\x5c\x75\xe5\x84\x8d\x4d\xf6" + "\x23\x9f\xf4\x6a\xe6\x5a\x3b\x9a", + .assoc = "", + .alen = 0, + .input = "\x1f\xee\x92\xb4\x0c\xcb\xa1\x04" + "\x64\x0c\x4d\xb2\xe3\xec\x9c\x9d" + "\x09\x1a\xb7\x74\xd8\x78\xa9\x79" + "\x96\xd8\x22\x43\x8c\xc3\x34\x7b" + "\xc4", + .ilen = 33, + .result = "\x67\x5d\x8e\x45\xc8\x39\xf5\x17" + "\xc1\x1d\x2a\xdd\x88\x67\xda\x1f" + "\x6d\xe8\x37\x28\x5a\xc1\x5e\x9f" + "\xa6\xec\xc6\x92\x05\x4b\xc0\xa3" + "\x63\xef\x88\xa4\x9b\x0a\x5c\xed" + "\x2b\x6a\xac\x63\x52\xaa\x10\x94" + "\xd0", + .rlen = 49, + }, { + .key = "\x2e\xb7\x20\x1c\x50\x6a\x4b\x8b" + "\x84\x42\x7a\xd7\xe1\xb5\xcd\x1f", + .klen = 16, + .iv = "\x3d\x80\xae\x84\x94\x09\xf6\x12" + "\xa4\x79\xa6\xfb\xe0\x7f\xfd\xa0", + .assoc = "", + .alen = 0, + .input = "\x5c\x13\xcb\x54\x1c\x47\x4a\x1f" + "\xe5\xe6\xff\x44\xdd\x11\x5f\xa3" + "\x33\xdd\xc2\xf8\xdd\x18\x2b\x93" + "\x57\x05\x01\x1c\x66\x22\xd3\x51" + "\xd3\xdf\x18\xc9\x30\x66\xed\xb1" + "\x96\x58\xd5\x8c\x64\x8c\x7c\xf5" + "\x01\xd0\x74\x5f\x9b\xaa\xf6\xd1" + "\xe6\x16\xa2\xac\xde\x47\x40", + .ilen = 63, + .result = "\x7d\x61\x1a\x35\x20\xcc\x07\x88" + "\x03\x98\x87\xcf\xc0\x6e\x4d\x19" + "\xe3\xd4\x0b\xfb\x29\x8f\x49\x1a" + "\x3a\x06\x77\xce\x71\x2c\xcd\xdd" + "\xed\xf6\xc9\xbe\xa6\x3b\xb8\xfc" + "\x6c\xbe\x77\xed\x74\x0e\x20\x85" + "\xd0\x65\xde\x24\x6f\xe3\x25\xc5" + "\xdf\x5b\x0f\xbd\x8a\x88\x78\xc9" + "\xe5\x81\x37\xde\x84\x7a\xf6\x84" + "\x99\x7a\x72\x9c\x54\x31\xa1", + .rlen = 79, + }, { + .key = "\x6b\xdc\x5a\xbb\x60\xe5\xf4\xa6" + "\x05\x1d\x2c\x68\xdb\xda\x8f\x25", + .klen = 16, + .iv = "\x7a\xa5\xe8\x23\xa4\x84\x9e\x2d" + "\x25\x53\x58\x8c\xda\xa3\xc0\xa6", + .assoc = "", + .alen = 0, + .input = "\x98\x37\x05\xf3\x2c\xc2\xf3\x3b" + "\x66\xc0\xb1\xd5\xd7\x35\x21\xaa" + "\x5d\x9f\xce\x7c\xe2\xb8\xad\xad" + "\x19\x33\xe0\xf4\x40\x81\x72\x28" + "\xe1\x8b\x1c\xf8\x91\x78\xff\xaf" + "\xb0\x68\x69\xf2\x27\x35\x91\x84" + "\x2e\x37\x5b\x00\x04\xff\x16\x9c" + "\xb5\x19\x39\xeb\xd9\xcd\x29\x9a", + .ilen = 64, + .result = "\x05\xc5\xb1\xf9\x1b\xb9\xab\x2c" + "\xa5\x07\x12\xa7\x12\x39\x60\x66" + "\x30\x81\x4a\x03\x78\x28\x45\x52" + "\xd2\x2b\x24\xfd\x8b\xa5\xb7\x66" + "\x6f\x45\xd7\x3b\x67\x6f\x51\xb9" + "\xc0\x3d\x6c\xca\x1e\xae\xff\xb6" + "\x79\xa9\xe4\x82\x5d\x4c\x2d\xdf" + "\xeb\x71\x40\xc9\x2c\x40\x45\x6d" + "\x73\x77\x01\xf3\x4f\xf3\x9d\x2a" + "\x5d\x57\xa8\xa1\x18\xa2\xad\xcb", + .rlen = 80, + }, { + .key = "\xa7\x00\x93\x5b\x70\x61\x9d\xc2" + "\x86\xf7\xde\xfa\xd5\xfe\x52\x2b", + .klen = 16, + .iv = "\xb6\xca\x22\xc3\xb4\x00\x47\x49" + "\xa6\x2d\x0a\x1e\xd4\xc7\x83\xad", + .assoc = "\xc5", + .alen = 1, + .input = "", + .ilen = 0, + .result = "\x4d\xbf\x11\xac\x7f\x97\x0b\x2e" + "\x89\x3b\x9d\x0f\x83\x1c\x08\xc3", + .rlen = 16, + }, { + .key = "\xe4\x25\xcd\xfa\x80\xdd\x46\xde" + "\x07\xd1\x90\x8b\xcf\x23\x15\x31", + .klen = 16, + .iv = "\xf3\xee\x5c\x62\xc4\x7c\xf0\x65" + "\x27\x08\xbd\xaf\xce\xec\x45\xb3", + .assoc = "\x02\xb8\xea\xca\x09\x1b\x9a\xec" + "\x47\x3e\xe9\xd4\xcc\xb5\x76\x34" + "\xe8\x73\x62\x64\xab\x50\xd0\xda" + "\x6b\x83\x66\xaf\x3e\x27\xc9", + .alen = 31, + .input = "", + .ilen = 0, + .result = "\x5b\xc0\x8d\x54\xe4\xec\xbe\x38" + "\x03\x12\xf9\xcc\x9e\x46\x42\x92", + .rlen = 16, + }, { + .key = "\x20\x4a\x07\x99\x91\x58\xee\xfa" + "\x88\xab\x42\x1c\xc9\x47\xd7\x38", + .klen = 16, + .iv = "\x2f\x13\x95\x01\xd5\xf7\x99\x81" + "\xa8\xe2\x6f\x41\xc8\x10\x08\xb9", + .assoc = "\x3f\xdc\x24\x69\x19\x96\x43\x08" + "\xc8\x18\x9b\x65\xc6\xd9\x39\x3b" + "\x12\x35\x6e\xe8\xb0\xf0\x52\xf3" + "\x2d\xb0\x45\x87\x18\x86\x68\xf6", + .alen = 32, + .input = "", + .ilen = 0, + .result = "\x48\xc5\xc3\x4c\x40\x2e\x2f\xc2" + "\x6d\x65\xe0\x67\x9c\x1d\xa0\xf0", + .rlen = 16, + }, { + .key = "\x5d\x6f\x41\x39\xa1\xd4\x97\x16" + "\x09\x85\xf4\xae\xc3\x6b\x9a\x3e", + .klen = 16, + .iv = "\x6c\x38\xcf\xa1\xe5\x73\x41\x9d" + "\x29\xbc\x21\xd2\xc2\x35\xcb\xbf", + .assoc = "\x7b\x01\x5d\x08\x29\x12\xec\x24" + "\x49\xf3\x4d\xf7\xc0\xfe\xfb\x41" + "\x3c\xf8\x79\x6c\xb6\x90\xd4\x0d" + "\xee\xde\x23\x60\xf2\xe5\x08\xcc" + "\x97", + .alen = 33, + .input = "", + .ilen = 0, + .result = "\x28\x64\x78\x51\x55\xd8\x56\x4a" + "\x58\x3e\xf7\xbe\xee\x21\xfe\x94", + .rlen = 16, + }, { + .key = "\x99\x93\x7a\xd8\xb1\x50\x40\x31" + "\x8a\x60\xa6\x3f\xbd\x90\x5d\x44", + .klen = 16, + .iv = "\xa8\x5c\x09\x40\xf5\xef\xea\xb8" + "\xaa\x96\xd3\x64\xbc\x59\x8d\xc6", + .assoc = "\xb8\x26\x97\xa8\x39\x8e\x94\x3f" + "\xca\xcd\xff\x88\xba\x22\xbe\x47" + "\x67\xba\x85\xf1\xbb\x30\x56\x26" + "\xaf\x0b\x02\x38\xcc\x44\xa7\xa3" + "\xa6\xbf\x31\x93\x60\xcd\xda\x63" + "\x2c\xb1\xaa\x19\xc8\x19\xf8\xeb" + "\x03\xa1\xe8\xbe\x37\x54\xec\xa2" + "\xcd\x2c\x45\x58\xbd\x8e\x80", + .alen = 63, + .input = "", + .ilen = 0, + .result = "\xb3\xa6\x00\x4e\x09\x20\xac\x21" + "\x77\x72\x69\x76\x2d\x36\xe5\xc8", + .rlen = 16, + }, { + .key = "\xd6\xb8\xb4\x77\xc1\xcb\xe9\x4d" + "\x0a\x3a\x58\xd1\xb7\xb4\x1f\x4a", + .klen = 16, + .iv = "\xe5\x81\x42\xdf\x05\x6a\x93\xd4" + "\x2b\x70\x85\xf5\xb6\x7d\x50\xcc", + .assoc = "\xf4\x4a\xd1\x47\x49\x09\x3d\x5b" + "\x4b\xa7\xb1\x19\xb4\x46\x81\x4d" + "\x91\x7c\x91\x75\xc0\xd0\xd8\x40" + "\x71\x39\xe1\x10\xa6\xa3\x46\x7a" + "\xb4\x6b\x35\xc2\xc1\xdf\xed\x60" + "\x46\xc1\x3e\x7f\x8c\xc2\x0e\x7a" + "\x30\x08\xd0\x5f\xa0\xaa\x0c\x6d" + "\x9c\x2f\xdb\x97\xb8\x15\x69\x01", + .alen = 64, + .input = "", + .ilen = 0, + .result = "\x65\x33\x7b\xa1\x63\xf4\x20\xdd" + "\xe4\xb9\x4a\xaa\x9a\x21\xaa\x14", + .rlen = 16, + }, { + .key = "\x12\xdd\xee\x17\xd1\x47\x92\x69" + "\x8b\x14\x0a\x62\xb1\xd9\xe2\x50", + .klen = 16, + .iv = "\x22\xa6\x7c\x7f\x15\xe6\x3c\xf0" + "\xac\x4b\x37\x86\xb0\xa2\x13\xd2", + .assoc = "\x31", + .alen = 1, + .input = "\x40", + .ilen = 1, + .result = "\x1d\x47\x17\x34\x86\xf5\x54\x1a" + "\x6d\x28\xb8\x5d\x6c\xcf\xa0\xb9" + "\xbf", + .rlen = 17, + }, { + .key = "\x4f\x01\x27\xb6\xe1\xc3\x3a\x85" + "\x0c\xee\xbc\xf4\xab\xfd\xa5\x57", + .klen = 16, + .iv = "\x5e\xcb\xb6\x1e\x25\x62\xe4\x0c" + "\x2d\x25\xe9\x18\xaa\xc6\xd5\xd8", + .assoc = "\x6d\x94\x44\x86\x69\x00\x8f\x93" + "\x4d\x5b\x15\x3c\xa8\x8f\x06\x5a" + "\xe6\x01\xa8\x7e\xca\x10\xdc\x73" + "\xf4\x94\x9f\xc1\x5a\x61\x85", + .alen = 31, + .input = "\x7c\x5d\xd3\xee\xad\x9f\x39\x1a" + "\x6d\x92\x42\x61\xa7\x58\x37\xdb" + "\xb0\xb2\x2b\x9f\x0b\xb8\xbd\x7a" + "\x24\xa0\xd6\xb7\x11\x79\x6c", + .ilen = 31, + .result = "\x78\x90\x52\xae\x0f\xf7\x2e\xef" + "\x63\x09\x08\x58\xb5\x56\xbd\x72" + "\x6e\x42\xcf\x27\x04\x7c\xdb\x92" + "\x18\xe9\xa4\x33\x90\xba\x62\xb5" + "\x70\xd3\x88\x9b\x4f\x05\xa7\x51" + "\x85\x87\x17\x09\x42\xed\x4e", + .rlen = 47, + }, { + .key = "\x8b\x26\x61\x55\xf1\x3e\xe3\xa1" + "\x8d\xc8\x6e\x85\xa5\x21\x67\x5d", + .klen = 16, + .iv = "\x9b\xef\xf0\xbd\x35\xdd\x8d\x28" + "\xad\xff\x9b\xa9\xa4\xeb\x98\xdf", + .assoc = "\xaa\xb8\x7e\x25\x79\x7c\x37\xaf" + "\xce\x36\xc7\xce\xa2\xb4\xc9\x60" + "\x10\xc3\xb3\x02\xcf\xb0\x5e\x8d" + "\xb5\xc2\x7e\x9a\x35\xc0\x24\xfd", + .alen = 32, + .input = "\xb9\x82\x0c\x8d\xbd\x1b\xe2\x36" + "\xee\x6c\xf4\xf2\xa1\x7d\xf9\xe2" + "\xdb\x74\x36\x23\x11\x58\x3f\x93" + "\xe5\xcd\xb5\x90\xeb\xd8\x0c\xb3", + .ilen = 32, + .result = "\x1d\x2c\x57\xe0\x50\x38\x3d\x41" + "\x2e\x71\xc8\x3b\x92\x43\x58\xaf" + "\x5a\xfb\xad\x8f\xd9\xd5\x8a\x5e" + "\xdb\xf3\xcd\x3a\x2b\xe1\x2c\x1a" + "\xb0\xed\xe3\x0c\x6e\xf9\xf2\xd6" + "\x90\xe6\xb1\x0e\xa5\x8a\xac\xb7", + .rlen = 48, + }, { + .key = "\xc8\x4b\x9b\xf5\x01\xba\x8c\xbd" + "\x0e\xa3\x21\x16\x9f\x46\x2a\x63", + .klen = 16, + .iv = "\xd7\x14\x29\x5d\x45\x59\x36\x44" + "\x2e\xd9\x4d\x3b\x9e\x0f\x5b\xe5", + .assoc = "\xe6\xdd\xb8\xc4\x89\xf8\xe0\xca" + "\x4f\x10\x7a\x5f\x9c\xd8\x8b\x66" + "\x3b\x86\xbf\x86\xd4\x50\xe0\xa7" + "\x76\xef\x5c\x72\x0f\x1f\xc3\xd4" + "\xee", + .alen = 33, + .input = "\xf5\xa6\x46\x2c\xce\x97\x8a\x51" + "\x6f\x46\xa6\x83\x9b\xa1\xbc\xe8" + "\x05\x36\x42\xa7\x16\xf8\xc1\xad" + "\xa7\xfb\x94\x68\xc5\x37\xab\x8a" + "\x72", + .ilen = 33, + .result = "\x59\x10\x84\x1c\x83\x4c\x8b\xfc" + "\xfd\x2e\x4b\x46\x84\xff\x78\x4e" + "\x50\xda\x5c\xb9\x61\x1d\xf5\xb9" + "\xfe\xbb\x7f\xae\x8c\xc1\x24\xbd" + "\x8c\x6f\x1f\x9b\xce\xc6\xc1\x37" + "\x08\x06\x5a\xe5\x96\x10\x95\xc2" + "\x5e", + .rlen = 49, + }, { + .key = "\x05\x70\xd5\x94\x12\x36\x35\xd8" + "\x8f\x7d\xd3\xa8\x99\x6a\xed\x69", + .klen = 16, + .iv = "\x14\x39\x63\xfc\x56\xd5\xdf\x5f" + "\xaf\xb3\xff\xcc\x98\x33\x1d\xeb", + .assoc = "\x23\x02\xf1\x64\x9a\x73\x89\xe6" + "\xd0\xea\x2c\xf1\x96\xfc\x4e\x6d" + "\x65\x48\xcb\x0a\xda\xf0\x62\xc0" + "\x38\x1d\x3b\x4a\xe9\x7e\x62\xaa" + "\xfd\xc9\x4a\xa9\xa9\x39\x4b\x54" + "\xc8\x0e\x24\x7f\x5e\x10\x7a\x45" + "\x10\x0b\x56\x85\xad\x54\xaa\x66" + "\xa8\x43\xcd\xd4\x9b\xb7\xfa", + .alen = 63, + .input = "\x32\xcb\x80\xcc\xde\x12\x33\x6d" + "\xf0\x20\x58\x15\x95\xc6\x7f\xee" + "\x2f\xf9\x4e\x2c\x1b\x98\x43\xc7" + "\x68\x28\x73\x40\x9f\x96\x4a\x60" + "\x80\xf4\x4b\xf4\xc1\x3d\xd0\x93" + "\xcf\x12\xc9\x59\x8f\x7a\x7f\xa8" + "\x1b\xa5\x50\xed\x87\xa9\x72\x59" + "\x9c\x44\xb2\xa4\x99\x98\x34", + .ilen = 63, + .result = "\x9a\x12\xbc\xdf\x72\xa8\x56\x22" + "\x49\x2d\x07\x92\xfc\x3d\x6d\x5f" + "\xef\x36\x19\xae\x91\xfa\xd6\x63" + "\x46\xea\x8a\x39\x14\x21\xa6\x37" + "\x18\xfc\x97\x3e\x16\xa5\x4d\x39" + "\x45\x2e\x69\xcc\x9c\x5f\xdf\x6d" + "\x5e\xa2\xbf\xac\x83\x32\x72\x52" + "\x58\x58\x23\x40\xfd\xa5\xc2\xe6" + "\xe9\x5a\x50\x98\x00\x58\xc9\x86" + "\x4f\x20\x37\xdb\x7b\x22\xa3", + .rlen = 79, + }, { + .key = "\x41\x94\x0e\x33\x22\xb1\xdd\xf4" + "\x10\x57\x85\x39\x93\x8f\xaf\x70", + .klen = 16, + .iv = "\x50\x5d\x9d\x9b\x66\x50\x88\x7b" + "\x30\x8e\xb1\x5e\x92\x58\xe0\xf1", + .assoc = "\x5f\x27\x2b\x03\xaa\xef\x32\x02" + "\x50\xc4\xde\x82\x90\x21\x11\x73" + "\x8f\x0a\xd6\x8f\xdf\x90\xe4\xda" + "\xf9\x4a\x1a\x23\xc3\xdd\x02\x81" + "\x0b\x76\x4f\xd7\x0a\x4b\x5e\x51" + "\xe3\x1d\xb9\xe5\x21\xb9\x8f\xd4" + "\x3d\x72\x3e\x26\x16\xa9\xca\x32" + "\x77\x47\x63\x14\x95\x3d\xe4\x34", + .alen = 64, + .input = "\x6e\xf0\xba\x6b\xee\x8e\xdc\x89" + "\x71\xfb\x0a\xa6\x8f\xea\x41\xf4" + "\x5a\xbb\x59\xb0\x20\x38\xc5\xe0" + "\x29\x56\x52\x19\x79\xf5\xe9\x37" + "\x8f\xa1\x50\x23\x22\x4f\xe3\x91" + "\xe9\x21\x5e\xbf\x52\x23\x95\x37" + "\x48\x0c\x38\x8f\xf0\xff\x92\x24" + "\x6b\x47\x49\xe3\x94\x1f\x1e\x01", + .ilen = 64, + .result = "\xe6\xeb\x92\x5a\x5b\xf0\x2d\xbb" + "\x23\xec\x35\xe3\xae\xc9\xfb\x0b" + "\x90\x14\x46\xeb\xa8\x8d\xb0\x9b" + "\x39\xda\x8b\x48\xec\xb2\x00\x4e" + "\x80\x6f\x46\x4f\x9b\x1e\xbb\x35" + "\xea\x5a\xbc\xa2\x36\xa5\x89\x45" + "\xc2\xd6\xd7\x15\x0b\xf6\x6c\x56" + "\xec\x99\x7d\x61\xb3\x15\x93\xed" + "\x83\x1e\xd9\x48\x84\x0b\x37\xfe" + "\x95\x74\x44\xd5\x54\xa6\x27\x06", + .rlen = 80, + }, { + .key = "\x7e\xb9\x48\xd3\x32\x2d\x86\x10" + "\x91\x31\x37\xcb\x8d\xb3\x72\x76", + .klen = 16, + .iv = "\x8d\x82\xd6\x3b\x76\xcc\x30\x97" + "\xb1\x68\x63\xef\x8c\x7c\xa3\xf7", + .assoc = "\x9c\x4b\x65\xa2\xba\x6b\xdb\x1e" + "\xd1\x9e\x90\x13\x8a\x45\xd3\x79" + "\xba\xcd\xe2\x13\xe4\x30\x66\xf4" + "\xba\x78\xf9\xfb\x9d\x3c\xa1\x58" + "\x1a\x22\x53\x05\x6b\x5c\x71\x4f" + "\xfd\x2d\x4d\x4c\xe5\x62\xa5\x63" + "\x6a\xda\x26\xc8\x7f\xff\xea\xfd" + "\x46\x4a\xfa\x53\x8f\xc4\xcd\x68" + "\x58", + .alen = 65, + .input = "\xab\x14\xf3\x0a\xfe\x0a\x85\xa5" + "\xf2\xd5\xbc\x38\x89\x0e\x04\xfb" + "\x84\x7d\x65\x34\x25\xd8\x47\xfa" + "\xeb\x83\x31\xf1\x54\x54\x89\x0d" + "\x9d\x4d\x54\x51\x84\x61\xf6\x8e" + "\x03\x31\xf2\x25\x16\xcc\xaa\xc6" + "\x75\x73\x20\x30\x59\x54\xb2\xf0" + "\x3a\x4b\xe0\x23\x8e\xa6\x08\x35" + "\x8a\xdf\x27\xa0\xe4\x60\x99\xae" + "\x8e\x43\xd9\x39\x7b\x10\x40\x67" + "\x5c\x7e\xc9\x70\x63\x34\xca\x59" + "\xfe\x86\xbc\xb7\x9c\x39\xf3\x6d" + "\x6a\x41\x64\x6f\x16\x7f\x65\x7e" + "\x89\x84\x68\xeb\xb0\x51\xbe\x55" + "\x33\x16\x59\x6c\x3b\xef\x88\xad" + "\x2f\xab\xbc\x25\x76\x87\x41\x2f" + "\x36", + .ilen = 129, + .result = "\x89\x24\x27\x86\xdc\xd7\x6b\xd9" + "\xd1\xcd\xdc\x16\xdd\x2c\xc1\xfb" + "\x52\xb5\xb3\xab\x50\x99\x3f\xa0" + "\x38\xa4\x74\xa5\x04\x15\x63\x05" + "\x8f\x54\x81\x06\x5a\x6b\xa4\x63" + "\x6d\xa7\x21\xcb\xff\x42\x30\x8e" + "\x3b\xd1\xca\x3f\x4b\x1a\xb8\xc3" + "\x42\x01\xe6\xbc\x75\x15\x87\xee" + "\xc9\x8e\x65\x01\xd9\xd8\xb5\x9f" + "\x48\x86\xa6\x5f\x2c\xc7\xb5\xb0" + "\xed\x5d\x14\x7c\x3f\x40\xb1\x0b" + "\x72\xef\x94\x8d\x7a\x85\x56\xe5" + "\x56\x08\x15\x56\xba\xaf\xbd\xf0" + "\x20\xef\xa0\xf6\xa9\xad\xa2\xc9" + "\x1c\x3b\x28\x51\x7e\x77\xb2\x18" + "\x4f\x61\x64\x37\x22\x36\x6d\x78" + "\xed\xed\x35\xe8\x83\xa5\xec\x25" + "\x6b\xff\x5f\x1a\x09\x96\x3d\xdc" + "\x20", + .rlen = 145, + }, { + .key = "\xba\xde\x82\x72\x42\xa9\x2f\x2c" + "\x12\x0b\xe9\x5c\x87\xd7\x35\x7c", + .klen = 16, + .iv = "\xc9\xa7\x10\xda\x86\x48\xd9\xb3" + "\x32\x42\x15\x80\x85\xa1\x65\xfe", + .assoc = "\xd8\x70\x9f\x42\xca\xe6\x83\x3a" + "\x52\x79\x42\xa5\x84\x6a\x96\x7f" + "\xe4\x8f\xed\x97\xe9\xd0\xe8\x0d" + "\x7c\xa6\xd8\xd4\x77\x9b\x40\x2e" + "\x28\xce\x57\x34\xcd\x6e\x84\x4c" + "\x17\x3c\xe1\xb2\xa8\x0b\xbb\xf1" + "\x96\x41\x0d\x69\xe8\x54\x0a\xc8" + "\x15\x4e\x91\x92\x89\x4b\xb7\x9b" + "\x21\xf7\x42\x89\xac\x12\x2a\x54" + "\x69\xee\x18\xc7\x8d\xed\xe8\xfd" + "\xbb\x04\x28\xe6\x8a\x3c\x98\xc1" + "\x04\x2d\xa9\xa1\x24\x83\xff\xe9" + "\x55\x7a\xf0\xd1\xf6\x63\x05\xe1" + "\xd9\x1e\x75\x72\xc1\x9f\xae\x32" + "\xe1\x6b\xcd\x9e\x61\x19\x23\x86" + "\xd9\xd2\xaf\x8e\xd5\xd3\xa8\xa9" + "\x51", + .alen = 129, + .input = "\xe8\x39\x2d\xaa\x0e\x85\x2d\xc1" + "\x72\xaf\x6e\xc9\x82\x33\xc7\x01" + "\xaf\x40\x70\xb8\x2a\x78\xc9\x14" + "\xac\xb1\x10\xca\x2e\xb3\x28\xe4" + "\xac\xfa\x58\x7f\xe5\x73\x09\x8c" + "\x1d\x40\x87\x8c\xd9\x75\xc0\x55" + "\xa2\xda\x07\xd1\xc2\xa9\xd1\xbb" + "\x09\x4f\x77\x62\x88\x2d\xf2\x68" + "\x54", + .ilen = 65, + .result = "\x36\x78\xb9\x22\xde\x62\x35\x55" + "\x1a\x7a\xf5\x45\xbc\xd7\x15\x82" + "\x01\xe9\x5a\x07\xea\x46\xaf\x91" + "\xcb\x73\xa5\xee\xe1\xb4\xbf\xc2" + "\xdb\xd2\x9d\x59\xde\xfc\x83\x00" + "\xf5\x46\xac\x97\xd5\x57\xa9\xb9" + "\x1f\x8c\xe8\xca\x68\x8b\x91\x0c" + "\x01\xbe\x0a\xaf\x7c\xf6\x67\xa4" + "\xbf\xbc\x88\x3f\x5d\xd1\xf9\x19" + "\x0f\x9d\xb2\xaf\xb9\x6e\x17\xdf" + "\xa2", + .rlen = 81, + }, { + .key = "\xf7\x02\xbb\x11\x52\x24\xd8\x48" + "\x93\xe6\x9b\xee\x81\xfc\xf7\x82", + .klen = 16, + .iv = "\x06\xcc\x4a\x79\x96\xc3\x82\xcf" + "\xb3\x1c\xc7\x12\x7f\xc5\x28\x04", + .assoc = "\x15\x95\xd8\xe1\xda\x62\x2c\x56" + "\xd3\x53\xf4\x36\x7e\x8e\x59\x85" + "\x0e\x51\xf9\x1c\xee\x70\x6a\x27" + "\x3d\xd3\xb7\xac\x51\xfa\xdf\x05", + .alen = 32, + .input = "\x24\x5e\x67\x49\x1e\x01\xd6\xdd" + "\xf3\x89\x20\x5b\x7c\x57\x89\x07" + "\xd9\x02\x7c\x3d\x2f\x18\x4b\x2d" + "\x6e\xde\xee\xa2\x08\x12\xc7\xba", + .ilen = 32, + .result = "\x08\x1b\x95\x0e\x41\x95\x02\x4b" + "\x9c\xbb\xa8\xd0\x7c\xd3\x44\x6e" + "\x89\x14\x33\x70\x0a\xbc\xea\x39" + "\x88\xaa\x2b\xd5\x73\x11\x55\xf5" + "\x33\x33\x9c\xd7\x42\x34\x49\x8e" + "\x2f\x03\x30\x05\x47\xaf\x34", + .rlen = 47, + }, { + .key = "\x33\x27\xf5\xb1\x62\xa0\x80\x63" + "\x14\xc0\x4d\x7f\x7b\x20\xba\x89", + .klen = 16, + .iv = "\x42\xf0\x84\x19\xa6\x3f\x2b\xea" + "\x34\xf6\x79\xa3\x79\xe9\xeb\x0a", + .assoc = "\x51\xb9\x12\x80\xea\xde\xd5\x71" + "\x54\x2d\xa6\xc8\x78\xb2\x1b\x8c" + "\x39\x14\x05\xa0\xf3\x10\xec\x41" + "\xff\x01\x95\x84\x2b\x59\x7f\xdb", + .alen = 32, + .input = "\x61\x83\xa0\xe8\x2e\x7d\x7f\xf8" + "\x74\x63\xd2\xec\x76\x7c\x4c\x0d" + "\x03\xc4\x88\xc1\x35\xb8\xcd\x47" + "\x2f\x0c\xcd\x7a\xe2\x71\x66\x91", + .ilen = 32, + .result = "\x97\xca\xf4\xe0\x8d\x89\xbf\x68" + "\x0c\x60\xb9\x27\xdf\xaa\x41\xc6" + "\x25\xd8\xf7\x1f\x10\x15\x48\x61" + "\x4c\x95\x00\xdf\x51\x9b\x7f\xe6" + "\x24\x40\x9e\xbe\x3b\xeb\x1b\x98" + "\xb9\x9c\xe5\xef\xf2\x05", + .rlen = 46, + }, { + .key = "\x70\x4c\x2f\x50\x72\x1c\x29\x7f" + "\x95\x9a\xff\x10\x75\x45\x7d\x8f", + .klen = 16, + .iv = "\x7f\x15\xbd\xb8\xb6\xba\xd3\x06" + "\xb5\xd1\x2b\x35\x73\x0e\xad\x10", + .assoc = "\x8e\xde\x4c\x20\xfa\x59\x7e\x8d" + "\xd5\x07\x58\x59\x72\xd7\xde\x92" + "\x63\xd6\x10\x24\xf8\xb0\x6e\x5a" + "\xc0\x2e\x74\x5d\x06\xb8\x1e\xb2", + .alen = 32, + .input = "\x9d\xa7\xda\x88\x3e\xf8\x28\x14" + "\xf5\x3e\x85\x7d\x70\xa0\x0f\x13" + "\x2e\x86\x93\x45\x3a\x58\x4f\x61" + "\xf0\x3a\xac\x53\xbc\xd0\x06\x68", + .ilen = 32, + .result = "\x63\x4c\x2a\x8e\xb4\x6b\x63\x0d" + "\xb5\xec\x9b\x4e\x12\x23\xa3\xcf" + "\x1a\x5a\x70\x15\x5a\x10\x40\x51" + "\xca\x47\x4c\x9d\xc9\x97\xf4\x77" + "\xdb\xc8\x10\x2d\xdc\x65\x20\x3f", + .rlen = 40, + }, { + .key = "\xac\x70\x69\xef\x82\x97\xd2\x9b" + "\x15\x74\xb1\xa2\x6f\x69\x3f\x95", + .klen = 16, + .iv = "\xbb\x3a\xf7\x57\xc6\x36\x7c\x22" + "\x36\xab\xde\xc6\x6d\x32\x70\x17", + .assoc = "\xcb\x03\x85\xbf\x0a\xd5\x26\xa9" + "\x56\xe1\x0a\xeb\x6c\xfb\xa1\x98" + "\x8d\x98\x1c\xa8\xfe\x50\xf0\x74" + "\x81\x5c\x53\x35\xe0\x17\xbd\x88", + .alen = 32, + .input = "\xda\xcc\x14\x27\x4e\x74\xd1\x30" + "\x76\x18\x37\x0f\x6a\xc4\xd1\x1a" + "\x58\x49\x9f\xc9\x3f\xf8\xd1\x7a" + "\xb2\x67\x8b\x2b\x96\x2f\xa5\x3e", + .ilen = 32, + .result = "\xf1\x62\x44\xc7\x5f\x19\xca\x43" + "\x47\x2c\xaf\x68\x82\xbd\x51\xef" + "\x3d\x65\xd8\x45\x2d\x06\x07\x78" + "\x08\x2e\xb3\x23\xcd\x81\x12\x55" + "\x1a", + .rlen = 33, + }, { + .key = "\xe9\x95\xa2\x8f\x93\x13\x7b\xb7" + "\x96\x4e\x63\x33\x69\x8d\x02\x9b" + "\x23\xf9\x22\xeb\x80\xa0\xb1\x81" + "\xe2\x73\xc3\x21\x4d\x47\x8d\xf4", + .klen = 32, + .iv = "\xf8\x5e\x31\xf7\xd7\xb2\x25\x3e" + "\xb7\x85\x90\x58\x67\x57\x33\x1d", + .assoc = "", + .alen = 0, + .input = "", + .ilen = 0, + .result = "\xdf\x2f\x83\xc0\x45\x4a\x2c\xcf" + "\xb9\xd2\x41\xf6\x80\xa1\x52\x70", + .rlen = 16, + }, { + .key = "\x25\xba\xdc\x2e\xa3\x8f\x24\xd3" + "\x17\x29\x15\xc5\x63\xb2\xc5\xa1" + "\x4d\xbc\x2d\x6f\x85\x40\x33\x9a" + "\xa3\xa0\xa1\xfa\x27\xa6\x2c\xca", + .klen = 32, + .iv = "\x34\x83\x6a\x96\xe7\x2d\xce\x5a" + "\x38\x5f\x42\xe9\x61\x7b\xf5\x23", + .assoc = "", + .alen = 0, + .input = "\x53", + .ilen = 1, + .result = "\x01\xd8\x55\x3c\xc0\x5a\x4b\xc7" + "\x01\xf4\x08\xe3\x0d\xf7\xf0\x78" + "\x53", + .rlen = 17, + }, { + .key = "\x62\xdf\x16\xcd\xb3\x0a\xcc\xef" + "\x98\x03\xc7\x56\x5d\xd6\x87\xa8" + "\x77\x7e\x39\xf3\x8a\xe0\xb5\xb4" + "\x65\xce\x80\xd2\x01\x05\xcb\xa1", + .klen = 32, + .iv = "\x71\xa8\xa4\x35\xf7\xa9\x76\x75" + "\xb8\x39\xf4\x7a\x5b\x9f\xb8\x29", + .assoc = "", + .alen = 0, + .input = "\x8f\x3a\xc1\x05\x7f\xe7\xcb\x83" + "\xf9\xa6\x4d\xc3\x58\x31\x19\x2c" + "\xd7\x90\xc2\x56\x4e\xd8\x57\xc7" + "\xf6\xf0\x27\xb4\x25\x4c\x83", + .ilen = 31, + .result = "\xc2\x4b\x41\x0f\x2d\xb9\x62\x07" + "\xff\x8e\x74\xf8\xa1\xa6\xd5\x37" + "\xa5\x64\x31\x5c\xca\x73\x9b\x43" + "\xe6\x70\x63\x46\x95\xcb\xf7\xb5" + "\x20\x8c\x75\x7a\x2a\x17\x2f\xa9" + "\xb8\x4d\x11\x42\xd1\xf8\xf1", + .rlen = 47, + }, { + .key = "\x9e\x03\x4f\x6d\xc3\x86\x75\x0a" + "\x19\xdd\x79\xe8\x57\xfb\x4a\xae" + "\xa2\x40\x45\x77\x90\x80\x37\xce" + "\x26\xfb\x5f\xaa\xdb\x64\x6b\x77", + .klen = 32, + .iv = "\xae\xcc\xde\xd5\x07\x25\x1f\x91" + "\x39\x14\xa6\x0c\x55\xc4\x7b\x30", + .assoc = "", + .alen = 0, + .input = "\xcc\x5f\xfb\xa4\x8f\x63\x74\x9f" + "\x7a\x81\xff\x55\x52\x56\xdc\x33" + "\x01\x52\xcd\xdb\x53\x78\xd9\xe1" + "\xb7\x1d\x06\x8d\xff\xab\x22\x98", + .ilen = 32, + .result = "\xbb\x01\x7c\xd1\x2c\x33\x7b\x37" + "\x0a\xee\xc4\x30\x19\xd7\x3a\x6f" + "\xf8\x2b\x67\xf5\x3b\x84\x87\x2a" + "\xfb\x07\x7a\x82\xb5\xe4\x85\x26" + "\x1e\xa8\xe5\x04\x54\xce\xe5\x5f" + "\xb5\x3f\xc1\xd5\x7f\xbd\xd2\xa6", + .rlen = 48, + }, { + .key = "\xdb\x28\x89\x0c\xd3\x01\x1e\x26" + "\x9a\xb7\x2b\x79\x51\x1f\x0d\xb4" + "\xcc\x03\x50\xfc\x95\x20\xb9\xe7" + "\xe8\x29\x3e\x83\xb5\xc3\x0a\x4e", + .klen = 32, + .iv = "\xea\xf1\x18\x74\x17\xa0\xc8\xad" + "\xba\xee\x58\x9d\x4f\xe8\x3d\x36", + .assoc = "", + .alen = 0, + .input = "\x08\x84\x34\x44\x9f\xde\x1c\xbb" + "\xfb\x5b\xb1\xe6\x4c\x7a\x9f\x39" + "\x2c\x14\xd9\x5f\x59\x18\x5b\xfb" + "\x79\x4b\xe5\x65\xd9\x0a\xc1\x6f" + "\x2e", + .ilen = 33, + .result = "\xc2\xf4\x40\x55\xf9\x59\xff\x73" + "\x08\xf5\x98\x92\x0c\x7b\x35\x9a" + "\xa8\xf4\x42\x7e\x6f\x93\xca\x22" + "\x23\x06\x1e\xf8\x89\x22\xf4\x46" + "\x7c\x7c\x67\x75\xab\xe5\x75\xaa" + "\x15\xd7\x83\x19\xfd\x31\x59\x5b" + "\x32", + .rlen = 49, + }, { + .key = "\x17\x4d\xc3\xab\xe3\x7d\xc7\x42" + "\x1b\x91\xdd\x0a\x4b\x43\xcf\xba" + "\xf6\xc5\x5c\x80\x9a\xc0\x3b\x01" + "\xa9\x56\x1d\x5b\x8f\x22\xa9\x25", + .klen = 32, + .iv = "\x27\x16\x51\x13\x27\x1c\x71\xc9" + "\x3b\xc8\x0a\x2f\x49\x0c\x00\x3c", + .assoc = "", + .alen = 0, + .input = "\x45\xa8\x6e\xe3\xaf\x5a\xc5\xd7" + "\x7c\x35\x63\x77\x46\x9f\x61\x3f" + "\x56\xd7\xe4\xe3\x5e\xb8\xdc\x14" + "\x3a\x79\xc4\x3e\xb3\x69\x61\x46" + "\x3c\xb6\x83\x4e\xb4\x26\xc7\x73" + "\x22\xda\x52\x8b\x7d\x11\x98\xea" + "\x62\xe1\x14\x1e\xdc\xfe\x0f\xad" + "\x20\x76\x5a\xdc\x4e\x71\x13", + .ilen = 63, + .result = "\xc9\x82\x3b\x4b\x87\x84\xa5\xdb" + "\xa0\x8c\xd3\x3e\x7f\x8d\xe8\x28" + "\x2a\xdc\xfa\x01\x84\x87\x9a\x70" + "\x81\x75\x37\x0a\xd2\x75\xa9\xb6" + "\x21\x72\xee\x7e\x65\x95\xe5\xcc" + "\x01\xb7\x39\xa6\x51\x15\xca\xff" + "\x61\xdc\x97\x38\xcc\xf4\xca\xc7" + "\x83\x9b\x05\x11\x72\x60\xf0\xb4" + "\x7e\x06\xab\x0a\xc0\xbb\x59\x23" + "\xaa\x2d\xfc\x4e\x35\x05\x59", + .rlen = 79, + }, { + .key = "\x54\x71\xfd\x4b\xf3\xf9\x6f\x5e" + "\x9c\x6c\x8f\x9c\x45\x68\x92\xc1" + "\x21\x87\x67\x04\x9f\x60\xbd\x1b" + "\x6a\x84\xfc\x34\x6a\x81\x48\xfb", + .klen = 32, + .iv = "\x63\x3b\x8b\xb3\x37\x98\x1a\xe5" + "\xbc\xa2\xbc\xc0\x43\x31\xc2\x42", + .assoc = "", + .alen = 0, + .input = "\x81\xcd\xa8\x82\xbf\xd6\x6e\xf3" + "\xfd\x0f\x15\x09\x40\xc3\x24\x45" + "\x81\x99\xf0\x67\x63\x58\x5e\x2e" + "\xfb\xa6\xa3\x16\x8d\xc8\x00\x1c" + "\x4b\x62\x87\x7c\x15\x38\xda\x70" + "\x3d\xea\xe7\xf2\x40\xba\xae\x79" + "\x8f\x48\xfc\xbf\x45\x53\x2e\x78" + "\xef\x79\xf0\x1b\x49\xf7\xfd\x9c", + .ilen = 64, + .result = "\x11\x7c\x7d\xef\xce\x29\x95\xec" + "\x7e\x9f\x42\xa6\x26\x07\xa1\x75" + "\x2f\x4e\x09\x9a\xf6\x6b\xc2\xfa" + "\x0d\xd0\x17\xdc\x25\x1e\x9b\xdc" + "\x5f\x8c\x1c\x60\x15\x4f\x9b\x20" + "\x7b\xff\xcd\x82\x60\x84\xf4\xa5" + "\x20\x9a\x05\x19\x5b\x02\x0a\x72" + "\x43\x11\x26\x58\xcf\xc5\x41\xcf" + "\x13\xcc\xde\x32\x92\xfa\x86\xf2" + "\xaf\x16\xe8\x8f\xca\xb6\xfd\x54", + .rlen = 80, + }, { + .key = "\x90\x96\x36\xea\x03\x74\x18\x7a" + "\x1d\x46\x42\x2d\x3f\x8c\x54\xc7" + "\x4b\x4a\x73\x89\xa4\x00\x3f\x34" + "\x2c\xb1\xdb\x0c\x44\xe0\xe8\xd2", + .klen = 32, + .iv = "\xa0\x5f\xc5\x52\x47\x13\xc2\x01" + "\x3d\x7c\x6e\x52\x3d\x55\x85\x48", + .assoc = "\xaf", + .alen = 1, + .input = "", + .ilen = 0, + .result = "\x9b\xc5\x3b\x20\x0a\x88\x56\xbe" + "\x69\xdf\xc4\xc4\x02\x46\x3a\xf0", + .rlen = 16, + }, { + .key = "\xcd\xbb\x70\x89\x13\xf0\xc1\x95" + "\x9e\x20\xf4\xbf\x39\xb1\x17\xcd" + "\x76\x0c\x7f\x0d\xa9\xa0\xc1\x4e" + "\xed\xdf\xb9\xe4\x1e\x3f\x87\xa8", + .klen = 32, + .iv = "\xdc\x84\xfe\xf1\x58\x8f\x6b\x1c" + "\xbe\x57\x20\xe3\x37\x7a\x48\x4f", + .assoc = "\xeb\x4d\x8d\x59\x9c\x2e\x15\xa3" + "\xde\x8d\x4d\x07\x36\x43\x78\xd0" + "\x0b\x6d\x84\x4f\x2c\xf0\x82\x5b" + "\x4e\xf6\x29\xd1\x8b\x6f\x56", + .alen = 31, + .input = "", + .ilen = 0, + .result = "\xe0\x6d\xa1\x07\x98\x2f\x40\x2d" + "\x2e\x9a\xd6\x61\x43\xc0\x74\x69", + .rlen = 16, + }, { + .key = "\x0a\xe0\xaa\x29\x24\x6c\x6a\xb1" + "\x1f\xfa\xa6\x50\x33\xd5\xda\xd3" + "\xa0\xce\x8a\x91\xae\x40\x43\x68" + "\xae\x0d\x98\xbd\xf8\x9e\x26\x7f", + .klen = 32, + .iv = "\x19\xa9\x38\x91\x68\x0b\x14\x38" + "\x3f\x31\xd2\x74\x31\x9e\x0a\x55", + .assoc = "\x28\x72\xc7\xf8\xac\xaa\xbe\xbf" + "\x5f\x67\xff\x99\x30\x67\x3b\xd6" + "\x35\x2f\x90\xd3\x31\x90\x04\x74" + "\x0f\x23\x08\xa9\x65\xce\xf6\xea", + .alen = 32, + .input = "", + .ilen = 0, + .result = "\xb9\x57\x13\x3e\x82\x31\x61\x65" + "\x0d\x7f\x6c\x96\x93\x5c\x50\xe2", + .rlen = 16, + }, { + .key = "\x46\x04\xe3\xc8\x34\xe7\x12\xcd" + "\xa0\xd4\x58\xe2\x2d\xf9\x9c\xda" + "\xca\x91\x96\x15\xb4\xe0\xc5\x81" + "\x70\x3a\x77\x95\xd2\xfd\xc5\x55", + .klen = 32, + .iv = "\x55\xcd\x72\x30\x78\x86\xbd\x54" + "\xc0\x0b\x84\x06\x2b\xc2\xcd\x5b", + .assoc = "\x64\x97\x00\x98\xbc\x25\x67\xdb" + "\xe0\x41\xb1\x2a\x2a\x8c\xfe\xdd" + "\x5f\xf2\x9c\x58\x36\x30\x86\x8e" + "\xd1\x51\xe6\x81\x3f\x2d\x95\xc1" + "\x01", + .alen = 33, + .input = "", + .ilen = 0, + .result = "\x81\x96\x34\xde\xbb\x36\xdd\x3e" + "\x4e\x5e\xcb\x44\x21\xb8\x3f\xf1", + .rlen = 16, + }, { + .key = "\x83\x29\x1d\x67\x44\x63\xbb\xe9" + "\x20\xaf\x0a\x73\x27\x1e\x5f\xe0" + "\xf5\x53\xa1\x9a\xb9\x80\x47\x9b" + "\x31\x68\x56\x6e\xac\x5c\x65\x2c", + .klen = 32, + .iv = "\x92\xf2\xac\xcf\x88\x02\x65\x70" + "\x41\xe5\x36\x97\x25\xe7\x90\x61", + .assoc = "\xa1\xbb\x3a\x37\xcc\xa1\x10\xf7" + "\x61\x1c\x63\xbc\x24\xb0\xc0\xe3" + "\x8a\xb4\xa7\xdc\x3b\xd0\x08\xa8" + "\x92\x7f\xc5\x5a\x19\x8c\x34\x97" + "\x0f\x95\x9b\x18\xe4\x8d\xb4\x24" + "\xb9\x33\x28\x18\xe1\x9d\x14\xe0" + "\x64\xb2\x89\x7d\x78\xa8\x05\x7e" + "\x07\x8c\xfc\x88\x2d\xb8\x53", + .alen = 63, + .input = "", + .ilen = 0, + .result = "\x2e\x99\xb6\x79\x57\x56\x80\x36" + "\x8e\xc4\x1c\x12\x7d\x71\x36\x0c", + .rlen = 16, + }, { + .key = "\xbf\x4e\x57\x07\x54\xdf\x64\x05" + "\xa1\x89\xbc\x04\x21\x42\x22\xe6" + "\x1f\x15\xad\x1e\xbe\x20\xc9\xb4" + "\xf3\x95\x35\x46\x86\xbb\x04\x03", + .klen = 32, + .iv = "\xce\x17\xe5\x6f\x98\x7e\x0e\x8c" + "\xc2\xbf\xe8\x29\x1f\x0b\x52\x68", + .assoc = "\xdd\xe0\x74\xd6\xdc\x1d\xb8\x13" + "\xe2\xf6\x15\x4d\x1e\xd4\x83\xe9" + "\xb4\x76\xb3\x60\x40\x70\x8a\xc1" + "\x53\xac\xa4\x32\xf3\xeb\xd3\x6e" + "\x1e\x42\xa0\x46\x45\x9f\xc7\x22" + "\xd3\x43\xbc\x7e\xa5\x47\x2a\x6f" + "\x91\x19\x70\x1e\xe1\xfe\x25\x49" + "\xd6\x8f\x93\xc7\x28\x3f\x3d\x03", + .alen = 64, + .input = "", + .ilen = 0, + .result = "\x7b\x25\x3d\x47\xd4\xa7\x08\xce" + "\x3b\x89\x40\x36\xba\x6d\x0e\xa2", + .rlen = 16, + }, { + .key = "\xfc\x72\x90\xa6\x64\x5a\x0d\x21" + "\x22\x63\x6e\x96\x1b\x67\xe4\xec" + "\x49\xd7\xb9\xa2\xc3\xc0\x4b\xce" + "\xb4\xc3\x14\x1e\x61\x1a\xa3\xd9", + .klen = 32, + .iv = "\x0b\x3c\x1f\x0e\xa8\xf9\xb7\xa7" + "\x42\x9a\x9a\xba\x19\x30\x15\x6e", + .assoc = "\x1a", + .alen = 1, + .input = "\x29", + .ilen = 1, + .result = "\xe6\x09\x6f\x95\x9a\x18\xc8\xf6" + "\x17\x75\x81\x16\xdf\x26\xff\x67" + "\x92", + .rlen = 17, + }, { + .key = "\x38\x97\xca\x45\x74\xd6\xb6\x3c" + "\xa3\x3d\x20\x27\x15\x8b\xa7\xf2" + "\x74\x9a\xc4\x27\xc8\x60\xcd\xe8" + "\x75\xf0\xf2\xf7\x3b\x79\x42\xb0", + .klen = 32, + .iv = "\x47\x60\x59\xad\xb8\x75\x60\xc3" + "\xc3\x74\x4c\x4c\x13\x54\xd8\x74", + .assoc = "\x56\x29\xe7\x15\xfc\x14\x0a\x4a" + "\xe4\xaa\x79\x70\x12\x1d\x08\xf6" + "\x09\xfb\xca\x69\x4b\xb0\x8e\xf5" + "\xd6\x07\x62\xe3\xa8\xa9\x12", + .alen = 31, + .input = "\x66\xf3\x75\x7d\x40\xb3\xb4\xd1" + "\x04\xe1\xa6\x94\x10\xe6\x39\x77" + "\xd3\xac\x4d\x8a\x8c\x58\x6e\xfb" + "\x06\x13\x9a\xd9\x5e\xc0\xfa", + .ilen = 31, + .result = "\x82\xc0\x56\xf0\xd7\xc4\xc9\xfd" + "\x3c\xd1\x2a\xd4\x15\x86\x9d\xda" + "\xea\x6c\x6f\xa1\x33\xb0\x7a\x01" + "\x57\xe7\xf3\x7b\x73\xe7\x54\x10" + "\xc6\x91\xe2\xc6\xa0\x69\xe7\xe6" + "\x76\xc3\xf5\x3a\x76\xfd\x4a", + .rlen = 47, + }, { + .key = "\x75\xbc\x04\xe5\x84\x52\x5e\x58" + "\x24\x17\xd2\xb9\x0e\xaf\x6a\xf9" + "\x9e\x5c\xd0\xab\xcd\x00\x4f\x01" + "\x37\x1e\xd1\xcf\x15\xd8\xe2\x86", + .klen = 32, + .iv = "\x84\x85\x92\x4d\xc8\xf1\x08\xdf" + "\x44\x4e\xff\xdd\x0d\x78\x9a\x7a", + .assoc = "\x93\x4e\x21\xb4\x0c\x90\xb3\x66" + "\x65\x84\x2b\x01\x0b\x42\xcb\xfc" + "\x33\xbd\xd6\xed\x50\x50\x10\x0e" + "\x97\x35\x41\xbb\x82\x08\xb1\xf2", + .alen = 32, + .input = "\xa2\x17\xaf\x1c\x50\x2e\x5d\xed" + "\x85\xbb\x58\x26\x0a\x0b\xfc\x7d" + "\xfe\x6e\x59\x0e\x91\xf8\xf0\x15" + "\xc8\x40\x78\xb1\x38\x1f\x99\xa7", + .ilen = 32, + .result = "\x01\x47\x8e\x6c\xf6\x64\x89\x3a" + "\x71\xce\xe4\xaa\x45\x70\xe6\x84" + "\x62\x48\x08\x64\x86\x6a\xdf\xec" + "\xb4\xa0\xfb\x34\x03\x0c\x19\xf4" + "\x2b\x7b\x36\x73\xec\x54\xa9\x1e" + "\x30\x85\xdb\xe4\xac\xe9\x2c\xca", + .rlen = 48, + }, { + .key = "\xb1\xe1\x3e\x84\x94\xcd\x07\x74" + "\xa5\xf2\x84\x4a\x08\xd4\x2c\xff" + "\xc8\x1e\xdb\x2f\xd2\xa0\xd1\x1b" + "\xf8\x4c\xb0\xa8\xef\x37\x81\x5d", + .klen = 32, + .iv = "\xc0\xaa\xcc\xec\xd8\x6c\xb1\xfb" + "\xc5\x28\xb1\x6e\x07\x9d\x5d\x81", + .assoc = "\xd0\x73\x5a\x54\x1d\x0b\x5b\x82" + "\xe5\x5f\xdd\x93\x05\x66\x8e\x02" + "\x5e\x80\xe1\x71\x55\xf0\x92\x28" + "\x59\x62\x20\x94\x5c\x67\x50\xc8" + "\x58", + .alen = 33, + .input = "\xdf\x3c\xe9\xbc\x61\xaa\x06\x09" + "\x06\x95\x0a\xb7\x04\x2f\xbe\x84" + "\x28\x30\x64\x92\x96\x98\x72\x2e" + "\x89\x6e\x57\x8a\x13\x7e\x38\x7e" + "\xdb", + .ilen = 33, + .result = "\x85\xe0\xf8\x0f\x8e\x49\xe3\x60" + "\xcb\x4a\x54\x94\xcf\xf5\x7e\x34" + "\xe9\xf8\x80\x65\x53\xd0\x72\x70" + "\x4f\x7d\x9d\xd1\x15\x6f\xb9\x2c" + "\xfa\xe8\xdd\xac\x2e\xe1\x3f\x67" + "\x63\x0f\x1a\x59\xb7\x89\xdb\xf4" + "\xc3", + .rlen = 49, + }, { + .key = "\xee\x05\x77\x23\xa5\x49\xb0\x90" + "\x26\xcc\x36\xdc\x02\xf8\xef\x05" + "\xf3\xe1\xe7\xb3\xd8\x40\x53\x35" + "\xb9\x79\x8f\x80\xc9\x96\x20\x33", + .klen = 32, + .iv = "\xfd\xce\x06\x8b\xe9\xe8\x5a\x17" + "\x46\x02\x63\x00\x01\xc1\x20\x87", + .assoc = "\x0c\x98\x94\xf3\x2d\x87\x04\x9e" + "\x66\x39\x8f\x24\xff\x8a\x50\x08" + "\x88\x42\xed\xf6\x5a\x90\x14\x42" + "\x1a\x90\xfe\x6c\x36\xc6\xf0\x9f" + "\x66\xa0\xb5\x2d\x2c\xf8\x25\x15" + "\x55\x90\xa2\x7e\x77\x94\x96\x3a" + "\x71\x1c\xf7\x44\xee\xa8\xc3\x42" + "\xe2\xa3\x84\x04\x0b\xe1\xce", + .alen = 63, + .input = "\x1b\x61\x23\x5b\x71\x26\xae\x25" + "\x87\x6f\xbc\x49\xfe\x53\x81\x8a" + "\x53\xf2\x70\x17\x9b\x38\xf4\x48" + "\x4b\x9b\x36\x62\xed\xdd\xd8\x54" + "\xea\xcb\xb6\x79\x45\xfc\xaa\x54" + "\x5c\x94\x47\x58\xa7\xff\x9c\x9e" + "\x7c\xb6\xf1\xac\xc8\xfd\x8b\x35" + "\xd5\xa4\x6a\xd4\x09\xc2\x08", + .ilen = 63, + .result = "\x00\xe5\x5b\x87\x5c\x20\x22\x8a" + "\xda\x1f\xd3\xff\xbb\xb2\xb0\xf8" + "\xef\xe9\xeb\x9e\x7c\x80\xf4\x2b" + "\x59\xc0\x79\xbc\x17\xa0\x15\x01" + "\xf5\x72\xfb\x5a\xe7\xaf\x07\xe3" + "\x1b\x49\x21\x34\x23\x63\x55\x5e" + "\xee\x4f\x34\x17\xfa\xfe\xa5\x0c" + "\xed\x0b\x23\xea\x9b\xda\x57\x2f" + "\xf6\xa9\xae\x0d\x4e\x40\x96\x45" + "\x7f\xfa\xf0\xbf\xc4\x98\x78", + .rlen = 79, + }, { + .key = "\x2a\x2a\xb1\xc3\xb5\xc5\x59\xac" + "\xa7\xa6\xe8\x6d\xfc\x1d\xb2\x0b" + "\x1d\xa3\xf3\x38\xdd\xe0\xd5\x4e" + "\x7b\xa7\x6e\x58\xa3\xf5\xbf\x0a", + .klen = 32, + .iv = "\x39\xf3\x3f\x2b\xf9\x64\x03\x33" + "\xc7\xdd\x15\x91\xfb\xe6\xe2\x8d", + .assoc = "\x49\xbc\xce\x92\x3d\x02\xad\xba" + "\xe7\x13\x41\xb6\xf9\xaf\x13\x0f" + "\xb2\x04\xf8\x7a\x5f\x30\x96\x5b" + "\xdc\xbd\xdd\x44\x10\x25\x8f\x75" + "\x75\x4d\xb9\x5b\x8e\x0a\x38\x13" + "\x6f\x9f\x36\xe4\x3a\x3e\xac\xc9" + "\x9d\x83\xde\xe5\x57\xfd\xe3\x0e" + "\xb1\xa7\x1b\x44\x05\x67\xb7\x37", + .alen = 64, + .input = "\x58\x85\x5c\xfa\x81\xa1\x57\x40" + "\x08\x4a\x6e\xda\xf8\x78\x44\x90" + "\x7d\xb5\x7b\x9b\xa1\xd8\x76\x62" + "\x0c\xc9\x15\x3b\xc7\x3c\x77\x2b" + "\xf8\x78\xba\xa7\xa6\x0e\xbd\x52" + "\x76\xa3\xdc\xbe\x6b\xa8\xb1\x2d" + "\xa9\x1d\xd8\x4e\x31\x53\xab\x00" + "\xa5\xa7\x01\x13\x04\x49\xf2\x04", + .ilen = 64, + .result = "\x28\xdd\xb9\x4a\x12\xc7\x0a\xe1" + "\x58\x06\x1a\x9b\x8c\x67\xdf\xeb" + "\x35\x35\x60\x9d\x06\x40\x65\xc1" + "\x93\xe8\xb3\x82\x50\x29\xdd\xb5" + "\x2b\xcb\xde\x18\x78\x6b\x42\xbe" + "\x6d\x24\xd0\xb2\x7d\xd7\x08\x8f" + "\x4a\x18\x98\xad\x8c\xf2\x97\xb4" + "\xf4\x77\xe4\xbf\x41\x3b\xc4\x06" + "\xce\x9e\x34\x81\xf0\x89\x11\x13" + "\x02\x65\xa1\x7c\xdf\x07\x33\x06", + .rlen = 80, + }, { + .key = "\x67\x4f\xeb\x62\xc5\x40\x01\xc7" + "\x28\x80\x9a\xfe\xf6\x41\x74\x12" + "\x48\x65\xfe\xbc\xe2\x80\x57\x68" + "\x3c\xd4\x4d\x31\x7d\x54\x5f\xe1", + .klen = 32, + .iv = "\x76\x18\x79\xca\x09\xdf\xac\x4e" + "\x48\xb7\xc7\x23\xf5\x0a\xa5\x93", + .assoc = "\x85\xe1\x08\x32\x4d\x7e\x56\xd5" + "\x68\xed\xf3\x47\xf3\xd3\xd6\x15" + "\xdd\xc7\x04\xfe\x64\xd0\x18\x75" + "\x9d\xeb\xbc\x1d\xea\x84\x2e\x4c" + "\x83\xf9\xbe\x8a\xef\x1c\x4b\x10" + "\x89\xaf\xcb\x4b\xfe\xe7\xc1\x58" + "\xca\xea\xc6\x87\xc0\x53\x03\xd9" + "\x80\xaa\xb2\x83\xff\xee\xa1\x6a" + "\x04", + .alen = 65, + .input = "\x94\xaa\x96\x9a\x91\x1d\x00\x5c" + "\x88\x24\x20\x6b\xf2\x9c\x06\x96" + "\xa7\x77\x87\x1f\xa6\x78\xf8\x7b" + "\xcd\xf6\xf4\x13\xa1\x9b\x16\x02" + "\x07\x24\xbf\xd5\x08\x20\xd0\x4f" + "\x90\xb3\x70\x24\x2f\x51\xc7\xbb" + "\xd6\x84\xc0\xef\x9a\xa8\xca\xcc" + "\x74\xab\x97\x53\xfe\xd0\xdb\x37" + "\x37\x6a\x0e\x9f\x3f\xa3\x2a\xe3" + "\x1b\x34\x6d\x51\x72\x2b\x17\xe7" + "\x4d\xaa\x2c\x18\xda\xa3\x33\x89" + "\x2a\x9f\xf4\xd2\xed\x76\x3d\x3f" + "\x3c\x15\x9d\x8e\x4f\x3c\x27\xb0" + "\x42\x3f\x2f\x8a\xd4\xc2\x10\xb2" + "\x27\x7f\xe3\x34\x80\x02\x49\x4b" + "\x07\x68\x22\x2a\x88\x25\x53\xb2" + "\x2f", + .ilen = 129, + .result = "\x85\x39\x69\x35\xfb\xf9\xb0\xa6" + "\x85\x43\x88\xd0\xd7\x78\x60\x19" + "\x3e\x1f\xb1\xa4\xd6\xc5\x96\xec" + "\xf7\x84\x85\xc7\x27\x0f\x74\x57" + "\x28\x9e\xdd\x90\x3c\x43\x12\xc5" + "\x51\x3d\x39\x8f\xa5\xf4\xe0\x0b" + "\x57\x04\xf1\x6d\xfe\x9b\x84\x27" + "\xe8\xeb\x4d\xda\x02\x0a\xc5\x49" + "\x1a\x55\x5e\x50\x56\x4d\x94\xda" + "\x20\xf8\x12\x54\x50\xb3\x11\xda" + "\xed\x44\x27\x67\xd5\xd1\x8b\x4b" + "\x38\x67\x56\x65\x59\xda\xe6\x97" + "\x81\xae\x2f\x92\x3b\xae\x22\x1c" + "\x91\x59\x38\x18\x00\xe8\xba\x92" + "\x04\x19\x56\xdf\xb0\x82\xeb\x6f" + "\x2e\xdb\x54\x3c\x4b\xbb\x60\x90" + "\x4c\x50\x10\x62\xba\x7a\xb1\x68" + "\x37\xd7\x87\x4e\xe4\x66\x09\x1f" + "\xa5", + .rlen = 145, + }, { + .key = "\xa3\x73\x24\x01\xd5\xbc\xaa\xe3" + "\xa9\x5a\x4c\x90\xf0\x65\x37\x18" + "\x72\x28\x0a\x40\xe7\x20\xd9\x82" + "\xfe\x02\x2b\x09\x57\xb3\xfe\xb7", + .klen = 32, + .iv = "\xb3\x3d\xb3\x69\x19\x5b\x54\x6a" + "\xc9\x91\x79\xb4\xef\x2e\x68\x99", + .assoc = "\xc2\x06\x41\xd1\x5d\xfa\xff\xf1" + "\xe9\xc7\xa5\xd9\xed\xf8\x98\x1b" + "\x07\x89\x10\x82\x6a\x70\x9a\x8f" + "\x5e\x19\x9b\xf5\xc5\xe3\xcd\x22" + "\x92\xa5\xc2\xb8\x51\x2e\x5e\x0e" + "\xa4\xbe\x5f\xb1\xc1\x90\xd7\xe7" + "\xf7\x52\xae\x28\x29\xa8\x22\xa4" + "\x4f\xae\x48\xc2\xfa\x75\x8b\x9e" + "\xce\x83\x2a\x88\x07\x55\xbb\x89" + "\xf6\xdf\xac\xdf\x83\x08\xbf\x7d" + "\xac\x30\x8b\x8e\x02\xac\x00\xf1" + "\x30\x46\xe1\xbc\x75\xbf\x49\xbb" + "\x26\x4e\x29\xf0\x2f\x21\xc6\x13" + "\x92\xd9\x3d\x11\xe4\x10\x00\x8e" + "\xd4\xd4\x58\x65\xa6\x2b\xe3\x25" + "\xb1\x8f\x15\x93\xe7\x71\xb9\x2c" + "\x4b", + .alen = 129, + .input = "\xd1\xcf\xd0\x39\xa1\x99\xa9\x78" + "\x09\xfe\xd2\xfd\xec\xc1\xc9\x9d" + "\xd2\x39\x93\xa3\xab\x18\x7a\x95" + "\x8f\x24\xd3\xeb\x7b\xfa\xb5\xd8" + "\x15\xd1\xc3\x04\x69\x32\xe3\x4d" + "\xaa\xc2\x04\x8b\xf2\xfa\xdc\x4a" + "\x02\xeb\xa8\x90\x03\xfd\xea\x97" + "\x43\xaf\x2e\x92\xf8\x57\xc5\x6a" + "\x00", + .ilen = 65, + .result = "\x7d\xde\x53\x22\xe4\x23\x3b\x30" + "\x78\xde\x35\x90\x7a\xd9\x0b\x93" + "\xf6\x0e\x0b\xed\x40\xee\x10\x9c" + "\x96\x3a\xd3\x34\xb2\xd0\x67\xcf" + "\x63\x7f\x2d\x0c\xcf\x96\xec\x64" + "\x1a\x87\xcc\x7d\x2c\x5e\x81\x4b" + "\xd2\x8f\x4c\x7c\x00\xb1\xb4\xe0" + "\x87\x4d\xb1\xbc\xd8\x78\x2c\x17" + "\xf2\x3b\xd8\x28\x40\xe2\x76\xf6" + "\x20\x13\x83\x46\xaf\xff\xe3\x0f" + "\x72", + .rlen = 81, + }, { + .key = "\xe0\x98\x5e\xa1\xe5\x38\x53\xff" + "\x2a\x35\xfe\x21\xea\x8a\xfa\x1e" + "\x9c\xea\x15\xc5\xec\xc0\x5b\x9b" + "\xbf\x2f\x0a\xe1\x32\x12\x9d\x8e", + .klen = 32, + .iv = "\xef\x61\xed\x08\x29\xd7\xfd\x86" + "\x4a\x6b\x2b\x46\xe9\x53\x2a\xa0", + .assoc = "\xfe\x2a\x7b\x70\x6d\x75\xa7\x0d" + "\x6a\xa2\x57\x6a\xe7\x1c\x5b\x21" + "\x31\x4b\x1b\x07\x6f\x10\x1c\xa8" + "\x20\x46\x7a\xce\x9f\x42\x6d\xf9", + .alen = 32, + .input = "\x0d\xf4\x09\xd8\xb1\x14\x51\x94" + "\x8a\xd8\x84\x8e\xe6\xe5\x8c\xa3" + "\xfc\xfc\x9e\x28\xb0\xb8\xfc\xaf" + "\x50\x52\xb1\xc4\x55\x59\x55\xaf", + .ilen = 32, + .result = "\x5a\xcd\x8c\x57\xf2\x6a\xb6\xbe" + "\x53\xc7\xaa\x9a\x60\x74\x9c\xc4" + "\xa2\xc2\xd0\x6d\xe1\x03\x63\xdc" + "\xbb\x51\x7e\x9c\x89\x73\xde\x4e" + "\x24\xf8\x52\x7c\x15\x41\x0e\xba" + "\x69\x0e\x36\x5f\x2f\x22\x8c", + .rlen = 47, + }, { + .key = "\x1c\xbd\x98\x40\xf5\xb3\xfc\x1b" + "\xaa\x0f\xb0\xb3\xe4\xae\xbc\x24" + "\xc7\xac\x21\x49\xf1\x60\xdd\xb5" + "\x80\x5d\xe9\xba\x0c\x71\x3c\x64", + .klen = 32, + .iv = "\x2c\x86\x26\xa8\x39\x52\xa6\xa2" + "\xcb\x45\xdd\xd7\xe3\x77\xed\xa6", + .assoc = "\x3b\x4f\xb5\x10\x7d\xf1\x50\x29" + "\xeb\x7c\x0a\xfb\xe1\x40\x1e\x27" + "\x5c\x0d\x27\x8b\x74\xb0\x9e\xc2" + "\xe1\x74\x59\xa6\x79\xa1\x0c\xd0", + .alen = 32, + .input = "\x4a\x18\x43\x77\xc1\x90\xfa\xb0" + "\x0b\xb2\x36\x20\xe0\x09\x4e\xa9" + "\x26\xbe\xaa\xac\xb5\x58\x7e\xc8" + "\x11\x7f\x90\x9c\x2f\xb8\xf4\x85", + .ilen = 32, + .result = "\x47\xd6\xce\x78\xd6\xbf\x4a\x51" + "\xb8\xda\x92\x3c\xfd\xda\xac\x8e" + "\x8d\x88\xd7\x4d\x90\xe5\xeb\xa1" + "\xab\xd6\x7c\x76\xad\xea\x7d\x76" + "\x53\xee\xb0\xcd\xd0\x02\xbb\x70" + "\x5b\x6f\x7b\xe2\x8c\xe8", + .rlen = 46, + }, { + .key = "\x59\xe1\xd2\xdf\x05\x2f\xa4\x37" + "\x2b\xe9\x63\x44\xde\xd3\x7f\x2b" + "\xf1\x6f\x2d\xcd\xf6\x00\x5f\xcf" + "\x42\x8a\xc8\x92\xe6\xd0\xdc\x3b", + .klen = 32, + .iv = "\x68\xab\x60\x47\x49\xce\x4f\xbe" + "\x4c\x20\x8f\x68\xdd\x9c\xb0\xac", + .assoc = "\x77\x74\xee\xaf\x8d\x6d\xf9\x45" + "\x6c\x56\xbc\x8d\xdb\x65\xe0\x2e" + "\x86\xd0\x32\x0f\x79\x50\x20\xdb" + "\xa2\xa1\x37\x7e\x53\x00\xab\xa6", + .alen = 32, + .input = "\x86\x3d\x7d\x17\xd1\x0c\xa3\xcc" + "\x8c\x8d\xe8\xb1\xda\x2e\x11\xaf" + "\x51\x80\xb5\x30\xba\xf8\x00\xe2" + "\xd3\xad\x6f\x75\x09\x18\x93\x5c", + .ilen = 32, + .result = "\x9f\xa9\x2b\xa4\x8f\x00\x05\x2b" + "\xe7\x68\x81\x51\xbb\xfb\xdf\x60" + "\xbb\xac\xe8\xc1\xdc\x68\xae\x68" + "\x3a\xcd\x7a\x06\x49\xfe\x80\x11" + "\xe6\x61\x99\xe2\xdd\xbe\x2c\xbf", + .rlen = 40, + }, { + .key = "\x96\x06\x0b\x7f\x15\xab\x4d\x53" + "\xac\xc3\x15\xd6\xd8\xf7\x42\x31" + "\x1b\x31\x38\x51\xfc\xa0\xe1\xe8" + "\x03\xb8\xa7\x6b\xc0\x2f\x7b\x11", + .klen = 32, + .iv = "\xa5\xcf\x9a\xe6\x59\x4a\xf7\xd9" + "\xcd\xfa\x41\xfa\xd7\xc0\x72\xb2", + .assoc = "\xb4\x99\x28\x4e\x9d\xe8\xa2\x60" + "\xed\x30\x6e\x1e\xd5\x89\xa3\x34" + "\xb1\x92\x3e\x93\x7e\xf0\xa2\xf5" + "\x64\xcf\x16\x57\x2d\x5f\x4a\x7d", + .alen = 32, + .input = "\xc3\x62\xb7\xb6\xe2\x87\x4c\xe7" + "\x0d\x67\x9a\x43\xd4\x52\xd4\xb5" + "\x7b\x43\xc1\xb5\xbf\x98\x82\xfc" + "\x94\xda\x4e\x4d\xe4\x77\x32\x32", + .ilen = 32, + .result = "\xe2\x34\xfa\x25\xfd\xfb\x89\x5e" + "\x5b\x4e\x0b\x15\x6e\x39\xfb\x0c" + "\x73\xc7\xd9\x6b\xbe\xce\x9b\x70" + "\xc7\x4f\x96\x16\x03\xfc\xea\xfb" + "\x56", + .rlen = 33, + }, +}; + +static const struct aead_testvec morus1280_dec_tv_template[] = { + { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 16, + .iv = "\x0f\xc9\x8e\x67\x44\x9e\xaa\x86" + "\x20\x36\x2c\x24\xfe\xc9\x30\x81", + .assoc = "", + .alen = 0, + .input = "\x91\x85\x0f\xf5\x52\x9e\xce\xce" + "\x65\x99\xc7\xbf\xd3\x76\xe8\x98", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x3c\x24\x39\x9f\x10\x7b\xa8\x1b" + "\x80\xda\xb2\x91\xf9\x24\xc2\x06", + .klen = 16, + .iv = "\x4b\xed\xc8\x07\x54\x1a\x52\xa2" + "\xa1\x10\xde\xb5\xf8\xed\xf3\x87", + .assoc = "", + .alen = 0, + .input = "\x88\xc3\x4c\xf0\x2f\x43\x76\x13" + "\x96\xda\x76\x34\x33\x4e\xd5\x39" + "\x73", + .ilen = 17, + .result = "\x69", + .rlen = 1, + }, { + .key = "\x79\x49\x73\x3e\x20\xf7\x51\x37" + "\x01\xb4\x64\x22\xf3\x48\x85\x0c", + .klen = 16, + .iv = "\x88\x12\x01\xa6\x64\x96\xfb\xbe" + "\x22\xea\x90\x47\xf2\x11\xb5\x8e", + .assoc = "", + .alen = 0, + .input = "\x3e\x5c\x3b\x58\x3b\x7d\x2a\x22" + "\x75\x0b\x24\xa6\x0e\xc3\xde\x52" + "\x97\x0b\x64\xd4\xce\x90\x52\xf7" + "\xef\xdb\x6a\x38\xd2\xa8\xa1\x0d" + "\xe0\x61\x33\x24\xc6\x4d\x51\xbc" + "\xa4\x21\x74\xcf\x19\x16\x59", + .ilen = 47, + .result = "\xa6\xa4\x1e\x76\xec\xd4\x50\xcc" + "\x62\x58\xe9\x8f\xef\xa4\x17\x91" + "\xb4\x96\x9f\x6b\xce\x38\xa5\x46" + "\x13\x7d\x64\x93\xd7\x05\xf5", + .rlen = 31, + }, { + .key = "\xb5\x6e\xad\xdd\x30\x72\xfa\x53" + "\x82\x8e\x16\xb4\xed\x6d\x47\x12", + .klen = 16, + .iv = "\xc4\x37\x3b\x45\x74\x11\xa4\xda" + "\xa2\xc5\x42\xd8\xec\x36\x78\x94", + .assoc = "", + .alen = 0, + .input = "\x30\x82\x9c\x2b\x67\xcb\xf9\x1f" + "\xde\x9f\x77\xb2\xda\x92\x61\x5c" + "\x09\x0b\x2d\x9a\x26\xaa\x1c\x06" + "\xab\x74\xb7\x2b\x95\x5f\x9f\xa1" + "\x9a\xff\x50\xa0\xa2\xff\xc5\xad" + "\x21\x8e\x84\x5c\x12\x61\xb2\xae", + .ilen = 48, + .result = "\xe2\xc9\x58\x15\xfc\x4f\xf8\xe8" + "\xe3\x32\x9b\x21\xe9\xc8\xd9\x97" + "\xde\x58\xab\xf0\xd3\xd8\x27\x60" + "\xd5\xaa\x43\x6b\xb1\x64\x95\xa4", + .rlen = 32, + }, { + .key = "\xf2\x92\xe6\x7d\x40\xee\xa3\x6f" + "\x03\x68\xc8\x45\xe7\x91\x0a\x18", + .klen = 16, + .iv = "\x01\x5c\x75\xe5\x84\x8d\x4d\xf6" + "\x23\x9f\xf4\x6a\xe6\x5a\x3b\x9a", + .assoc = "", + .alen = 0, + .input = "\x67\x5d\x8e\x45\xc8\x39\xf5\x17" + "\xc1\x1d\x2a\xdd\x88\x67\xda\x1f" + "\x6d\xe8\x37\x28\x5a\xc1\x5e\x9f" + "\xa6\xec\xc6\x92\x05\x4b\xc0\xa3" + "\x63\xef\x88\xa4\x9b\x0a\x5c\xed" + "\x2b\x6a\xac\x63\x52\xaa\x10\x94" + "\xd0", + .ilen = 49, + .result = "\x1f\xee\x92\xb4\x0c\xcb\xa1\x04" + "\x64\x0c\x4d\xb2\xe3\xec\x9c\x9d" + "\x09\x1a\xb7\x74\xd8\x78\xa9\x79" + "\x96\xd8\x22\x43\x8c\xc3\x34\x7b" + "\xc4", + .rlen = 33, + }, { + .key = "\x2e\xb7\x20\x1c\x50\x6a\x4b\x8b" + "\x84\x42\x7a\xd7\xe1\xb5\xcd\x1f", + .klen = 16, + .iv = "\x3d\x80\xae\x84\x94\x09\xf6\x12" + "\xa4\x79\xa6\xfb\xe0\x7f\xfd\xa0", + .assoc = "", + .alen = 0, + .input = "\x7d\x61\x1a\x35\x20\xcc\x07\x88" + "\x03\x98\x87\xcf\xc0\x6e\x4d\x19" + "\xe3\xd4\x0b\xfb\x29\x8f\x49\x1a" + "\x3a\x06\x77\xce\x71\x2c\xcd\xdd" + "\xed\xf6\xc9\xbe\xa6\x3b\xb8\xfc" + "\x6c\xbe\x77\xed\x74\x0e\x20\x85" + "\xd0\x65\xde\x24\x6f\xe3\x25\xc5" + "\xdf\x5b\x0f\xbd\x8a\x88\x78\xc9" + "\xe5\x81\x37\xde\x84\x7a\xf6\x84" + "\x99\x7a\x72\x9c\x54\x31\xa1", + .ilen = 79, + .result = "\x5c\x13\xcb\x54\x1c\x47\x4a\x1f" + "\xe5\xe6\xff\x44\xdd\x11\x5f\xa3" + "\x33\xdd\xc2\xf8\xdd\x18\x2b\x93" + "\x57\x05\x01\x1c\x66\x22\xd3\x51" + "\xd3\xdf\x18\xc9\x30\x66\xed\xb1" + "\x96\x58\xd5\x8c\x64\x8c\x7c\xf5" + "\x01\xd0\x74\x5f\x9b\xaa\xf6\xd1" + "\xe6\x16\xa2\xac\xde\x47\x40", + .rlen = 63, + }, { + .key = "\x6b\xdc\x5a\xbb\x60\xe5\xf4\xa6" + "\x05\x1d\x2c\x68\xdb\xda\x8f\x25", + .klen = 16, + .iv = "\x7a\xa5\xe8\x23\xa4\x84\x9e\x2d" + "\x25\x53\x58\x8c\xda\xa3\xc0\xa6", + .assoc = "", + .alen = 0, + .input = "\x05\xc5\xb1\xf9\x1b\xb9\xab\x2c" + "\xa5\x07\x12\xa7\x12\x39\x60\x66" + "\x30\x81\x4a\x03\x78\x28\x45\x52" + "\xd2\x2b\x24\xfd\x8b\xa5\xb7\x66" + "\x6f\x45\xd7\x3b\x67\x6f\x51\xb9" + "\xc0\x3d\x6c\xca\x1e\xae\xff\xb6" + "\x79\xa9\xe4\x82\x5d\x4c\x2d\xdf" + "\xeb\x71\x40\xc9\x2c\x40\x45\x6d" + "\x73\x77\x01\xf3\x4f\xf3\x9d\x2a" + "\x5d\x57\xa8\xa1\x18\xa2\xad\xcb", + .ilen = 80, + .result = "\x98\x37\x05\xf3\x2c\xc2\xf3\x3b" + "\x66\xc0\xb1\xd5\xd7\x35\x21\xaa" + "\x5d\x9f\xce\x7c\xe2\xb8\xad\xad" + "\x19\x33\xe0\xf4\x40\x81\x72\x28" + "\xe1\x8b\x1c\xf8\x91\x78\xff\xaf" + "\xb0\x68\x69\xf2\x27\x35\x91\x84" + "\x2e\x37\x5b\x00\x04\xff\x16\x9c" + "\xb5\x19\x39\xeb\xd9\xcd\x29\x9a", + .rlen = 64, + }, { + .key = "\xa7\x00\x93\x5b\x70\x61\x9d\xc2" + "\x86\xf7\xde\xfa\xd5\xfe\x52\x2b", + .klen = 16, + .iv = "\xb6\xca\x22\xc3\xb4\x00\x47\x49" + "\xa6\x2d\x0a\x1e\xd4\xc7\x83\xad", + .assoc = "\xc5", + .alen = 1, + .input = "\x4d\xbf\x11\xac\x7f\x97\x0b\x2e" + "\x89\x3b\x9d\x0f\x83\x1c\x08\xc3", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xe4\x25\xcd\xfa\x80\xdd\x46\xde" + "\x07\xd1\x90\x8b\xcf\x23\x15\x31", + .klen = 16, + .iv = "\xf3\xee\x5c\x62\xc4\x7c\xf0\x65" + "\x27\x08\xbd\xaf\xce\xec\x45\xb3", + .assoc = "\x02\xb8\xea\xca\x09\x1b\x9a\xec" + "\x47\x3e\xe9\xd4\xcc\xb5\x76\x34" + "\xe8\x73\x62\x64\xab\x50\xd0\xda" + "\x6b\x83\x66\xaf\x3e\x27\xc9", + .alen = 31, + .input = "\x5b\xc0\x8d\x54\xe4\xec\xbe\x38" + "\x03\x12\xf9\xcc\x9e\x46\x42\x92", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x20\x4a\x07\x99\x91\x58\xee\xfa" + "\x88\xab\x42\x1c\xc9\x47\xd7\x38", + .klen = 16, + .iv = "\x2f\x13\x95\x01\xd5\xf7\x99\x81" + "\xa8\xe2\x6f\x41\xc8\x10\x08\xb9", + .assoc = "\x3f\xdc\x24\x69\x19\x96\x43\x08" + "\xc8\x18\x9b\x65\xc6\xd9\x39\x3b" + "\x12\x35\x6e\xe8\xb0\xf0\x52\xf3" + "\x2d\xb0\x45\x87\x18\x86\x68\xf6", + .alen = 32, + .input = "\x48\xc5\xc3\x4c\x40\x2e\x2f\xc2" + "\x6d\x65\xe0\x67\x9c\x1d\xa0\xf0", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x5d\x6f\x41\x39\xa1\xd4\x97\x16" + "\x09\x85\xf4\xae\xc3\x6b\x9a\x3e", + .klen = 16, + .iv = "\x6c\x38\xcf\xa1\xe5\x73\x41\x9d" + "\x29\xbc\x21\xd2\xc2\x35\xcb\xbf", + .assoc = "\x7b\x01\x5d\x08\x29\x12\xec\x24" + "\x49\xf3\x4d\xf7\xc0\xfe\xfb\x41" + "\x3c\xf8\x79\x6c\xb6\x90\xd4\x0d" + "\xee\xde\x23\x60\xf2\xe5\x08\xcc" + "\x97", + .alen = 33, + .input = "\x28\x64\x78\x51\x55\xd8\x56\x4a" + "\x58\x3e\xf7\xbe\xee\x21\xfe\x94", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x99\x93\x7a\xd8\xb1\x50\x40\x31" + "\x8a\x60\xa6\x3f\xbd\x90\x5d\x44", + .klen = 16, + .iv = "\xa8\x5c\x09\x40\xf5\xef\xea\xb8" + "\xaa\x96\xd3\x64\xbc\x59\x8d\xc6", + .assoc = "\xb8\x26\x97\xa8\x39\x8e\x94\x3f" + "\xca\xcd\xff\x88\xba\x22\xbe\x47" + "\x67\xba\x85\xf1\xbb\x30\x56\x26" + "\xaf\x0b\x02\x38\xcc\x44\xa7\xa3" + "\xa6\xbf\x31\x93\x60\xcd\xda\x63" + "\x2c\xb1\xaa\x19\xc8\x19\xf8\xeb" + "\x03\xa1\xe8\xbe\x37\x54\xec\xa2" + "\xcd\x2c\x45\x58\xbd\x8e\x80", + .alen = 63, + .input = "\xb3\xa6\x00\x4e\x09\x20\xac\x21" + "\x77\x72\x69\x76\x2d\x36\xe5\xc8", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xd6\xb8\xb4\x77\xc1\xcb\xe9\x4d" + "\x0a\x3a\x58\xd1\xb7\xb4\x1f\x4a", + .klen = 16, + .iv = "\xe5\x81\x42\xdf\x05\x6a\x93\xd4" + "\x2b\x70\x85\xf5\xb6\x7d\x50\xcc", + .assoc = "\xf4\x4a\xd1\x47\x49\x09\x3d\x5b" + "\x4b\xa7\xb1\x19\xb4\x46\x81\x4d" + "\x91\x7c\x91\x75\xc0\xd0\xd8\x40" + "\x71\x39\xe1\x10\xa6\xa3\x46\x7a" + "\xb4\x6b\x35\xc2\xc1\xdf\xed\x60" + "\x46\xc1\x3e\x7f\x8c\xc2\x0e\x7a" + "\x30\x08\xd0\x5f\xa0\xaa\x0c\x6d" + "\x9c\x2f\xdb\x97\xb8\x15\x69\x01", + .alen = 64, + .input = "\x65\x33\x7b\xa1\x63\xf4\x20\xdd" + "\xe4\xb9\x4a\xaa\x9a\x21\xaa\x14", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x12\xdd\xee\x17\xd1\x47\x92\x69" + "\x8b\x14\x0a\x62\xb1\xd9\xe2\x50", + .klen = 16, + .iv = "\x22\xa6\x7c\x7f\x15\xe6\x3c\xf0" + "\xac\x4b\x37\x86\xb0\xa2\x13\xd2", + .assoc = "\x31", + .alen = 1, + .input = "\x1d\x47\x17\x34\x86\xf5\x54\x1a" + "\x6d\x28\xb8\x5d\x6c\xcf\xa0\xb9" + "\xbf", + .ilen = 17, + .result = "\x40", + .rlen = 1, + }, { + .key = "\x4f\x01\x27\xb6\xe1\xc3\x3a\x85" + "\x0c\xee\xbc\xf4\xab\xfd\xa5\x57", + .klen = 16, + .iv = "\x5e\xcb\xb6\x1e\x25\x62\xe4\x0c" + "\x2d\x25\xe9\x18\xaa\xc6\xd5\xd8", + .assoc = "\x6d\x94\x44\x86\x69\x00\x8f\x93" + "\x4d\x5b\x15\x3c\xa8\x8f\x06\x5a" + "\xe6\x01\xa8\x7e\xca\x10\xdc\x73" + "\xf4\x94\x9f\xc1\x5a\x61\x85", + .alen = 31, + .input = "\x78\x90\x52\xae\x0f\xf7\x2e\xef" + "\x63\x09\x08\x58\xb5\x56\xbd\x72" + "\x6e\x42\xcf\x27\x04\x7c\xdb\x92" + "\x18\xe9\xa4\x33\x90\xba\x62\xb5" + "\x70\xd3\x88\x9b\x4f\x05\xa7\x51" + "\x85\x87\x17\x09\x42\xed\x4e", + .ilen = 47, + .result = "\x7c\x5d\xd3\xee\xad\x9f\x39\x1a" + "\x6d\x92\x42\x61\xa7\x58\x37\xdb" + "\xb0\xb2\x2b\x9f\x0b\xb8\xbd\x7a" + "\x24\xa0\xd6\xb7\x11\x79\x6c", + .rlen = 31, + }, { + .key = "\x8b\x26\x61\x55\xf1\x3e\xe3\xa1" + "\x8d\xc8\x6e\x85\xa5\x21\x67\x5d", + .klen = 16, + .iv = "\x9b\xef\xf0\xbd\x35\xdd\x8d\x28" + "\xad\xff\x9b\xa9\xa4\xeb\x98\xdf", + .assoc = "\xaa\xb8\x7e\x25\x79\x7c\x37\xaf" + "\xce\x36\xc7\xce\xa2\xb4\xc9\x60" + "\x10\xc3\xb3\x02\xcf\xb0\x5e\x8d" + "\xb5\xc2\x7e\x9a\x35\xc0\x24\xfd", + .alen = 32, + .input = "\x1d\x2c\x57\xe0\x50\x38\x3d\x41" + "\x2e\x71\xc8\x3b\x92\x43\x58\xaf" + "\x5a\xfb\xad\x8f\xd9\xd5\x8a\x5e" + "\xdb\xf3\xcd\x3a\x2b\xe1\x2c\x1a" + "\xb0\xed\xe3\x0c\x6e\xf9\xf2\xd6" + "\x90\xe6\xb1\x0e\xa5\x8a\xac\xb7", + .ilen = 48, + .result = "\xb9\x82\x0c\x8d\xbd\x1b\xe2\x36" + "\xee\x6c\xf4\xf2\xa1\x7d\xf9\xe2" + "\xdb\x74\x36\x23\x11\x58\x3f\x93" + "\xe5\xcd\xb5\x90\xeb\xd8\x0c\xb3", + .rlen = 32, + }, { + .key = "\xc8\x4b\x9b\xf5\x01\xba\x8c\xbd" + "\x0e\xa3\x21\x16\x9f\x46\x2a\x63", + .klen = 16, + .iv = "\xd7\x14\x29\x5d\x45\x59\x36\x44" + "\x2e\xd9\x4d\x3b\x9e\x0f\x5b\xe5", + .assoc = "\xe6\xdd\xb8\xc4\x89\xf8\xe0\xca" + "\x4f\x10\x7a\x5f\x9c\xd8\x8b\x66" + "\x3b\x86\xbf\x86\xd4\x50\xe0\xa7" + "\x76\xef\x5c\x72\x0f\x1f\xc3\xd4" + "\xee", + .alen = 33, + .input = "\x59\x10\x84\x1c\x83\x4c\x8b\xfc" + "\xfd\x2e\x4b\x46\x84\xff\x78\x4e" + "\x50\xda\x5c\xb9\x61\x1d\xf5\xb9" + "\xfe\xbb\x7f\xae\x8c\xc1\x24\xbd" + "\x8c\x6f\x1f\x9b\xce\xc6\xc1\x37" + "\x08\x06\x5a\xe5\x96\x10\x95\xc2" + "\x5e", + .ilen = 49, + .result = "\xf5\xa6\x46\x2c\xce\x97\x8a\x51" + "\x6f\x46\xa6\x83\x9b\xa1\xbc\xe8" + "\x05\x36\x42\xa7\x16\xf8\xc1\xad" + "\xa7\xfb\x94\x68\xc5\x37\xab\x8a" + "\x72", + .rlen = 33, + }, { + .key = "\x05\x70\xd5\x94\x12\x36\x35\xd8" + "\x8f\x7d\xd3\xa8\x99\x6a\xed\x69", + .klen = 16, + .iv = "\x14\x39\x63\xfc\x56\xd5\xdf\x5f" + "\xaf\xb3\xff\xcc\x98\x33\x1d\xeb", + .assoc = "\x23\x02\xf1\x64\x9a\x73\x89\xe6" + "\xd0\xea\x2c\xf1\x96\xfc\x4e\x6d" + "\x65\x48\xcb\x0a\xda\xf0\x62\xc0" + "\x38\x1d\x3b\x4a\xe9\x7e\x62\xaa" + "\xfd\xc9\x4a\xa9\xa9\x39\x4b\x54" + "\xc8\x0e\x24\x7f\x5e\x10\x7a\x45" + "\x10\x0b\x56\x85\xad\x54\xaa\x66" + "\xa8\x43\xcd\xd4\x9b\xb7\xfa", + .alen = 63, + .input = "\x9a\x12\xbc\xdf\x72\xa8\x56\x22" + "\x49\x2d\x07\x92\xfc\x3d\x6d\x5f" + "\xef\x36\x19\xae\x91\xfa\xd6\x63" + "\x46\xea\x8a\x39\x14\x21\xa6\x37" + "\x18\xfc\x97\x3e\x16\xa5\x4d\x39" + "\x45\x2e\x69\xcc\x9c\x5f\xdf\x6d" + "\x5e\xa2\xbf\xac\x83\x32\x72\x52" + "\x58\x58\x23\x40\xfd\xa5\xc2\xe6" + "\xe9\x5a\x50\x98\x00\x58\xc9\x86" + "\x4f\x20\x37\xdb\x7b\x22\xa3", + .ilen = 79, + .result = "\x32\xcb\x80\xcc\xde\x12\x33\x6d" + "\xf0\x20\x58\x15\x95\xc6\x7f\xee" + "\x2f\xf9\x4e\x2c\x1b\x98\x43\xc7" + "\x68\x28\x73\x40\x9f\x96\x4a\x60" + "\x80\xf4\x4b\xf4\xc1\x3d\xd0\x93" + "\xcf\x12\xc9\x59\x8f\x7a\x7f\xa8" + "\x1b\xa5\x50\xed\x87\xa9\x72\x59" + "\x9c\x44\xb2\xa4\x99\x98\x34", + .rlen = 63, + }, { + .key = "\x41\x94\x0e\x33\x22\xb1\xdd\xf4" + "\x10\x57\x85\x39\x93\x8f\xaf\x70", + .klen = 16, + .iv = "\x50\x5d\x9d\x9b\x66\x50\x88\x7b" + "\x30\x8e\xb1\x5e\x92\x58\xe0\xf1", + .assoc = "\x5f\x27\x2b\x03\xaa\xef\x32\x02" + "\x50\xc4\xde\x82\x90\x21\x11\x73" + "\x8f\x0a\xd6\x8f\xdf\x90\xe4\xda" + "\xf9\x4a\x1a\x23\xc3\xdd\x02\x81" + "\x0b\x76\x4f\xd7\x0a\x4b\x5e\x51" + "\xe3\x1d\xb9\xe5\x21\xb9\x8f\xd4" + "\x3d\x72\x3e\x26\x16\xa9\xca\x32" + "\x77\x47\x63\x14\x95\x3d\xe4\x34", + .alen = 64, + .input = "\xe6\xeb\x92\x5a\x5b\xf0\x2d\xbb" + "\x23\xec\x35\xe3\xae\xc9\xfb\x0b" + "\x90\x14\x46\xeb\xa8\x8d\xb0\x9b" + "\x39\xda\x8b\x48\xec\xb2\x00\x4e" + "\x80\x6f\x46\x4f\x9b\x1e\xbb\x35" + "\xea\x5a\xbc\xa2\x36\xa5\x89\x45" + "\xc2\xd6\xd7\x15\x0b\xf6\x6c\x56" + "\xec\x99\x7d\x61\xb3\x15\x93\xed" + "\x83\x1e\xd9\x48\x84\x0b\x37\xfe" + "\x95\x74\x44\xd5\x54\xa6\x27\x06", + .ilen = 80, + .result = "\x6e\xf0\xba\x6b\xee\x8e\xdc\x89" + "\x71\xfb\x0a\xa6\x8f\xea\x41\xf4" + "\x5a\xbb\x59\xb0\x20\x38\xc5\xe0" + "\x29\x56\x52\x19\x79\xf5\xe9\x37" + "\x8f\xa1\x50\x23\x22\x4f\xe3\x91" + "\xe9\x21\x5e\xbf\x52\x23\x95\x37" + "\x48\x0c\x38\x8f\xf0\xff\x92\x24" + "\x6b\x47\x49\xe3\x94\x1f\x1e\x01", + .rlen = 64, + }, { + .key = "\x7e\xb9\x48\xd3\x32\x2d\x86\x10" + "\x91\x31\x37\xcb\x8d\xb3\x72\x76", + .klen = 16, + .iv = "\x8d\x82\xd6\x3b\x76\xcc\x30\x97" + "\xb1\x68\x63\xef\x8c\x7c\xa3\xf7", + .assoc = "\x9c\x4b\x65\xa2\xba\x6b\xdb\x1e" + "\xd1\x9e\x90\x13\x8a\x45\xd3\x79" + "\xba\xcd\xe2\x13\xe4\x30\x66\xf4" + "\xba\x78\xf9\xfb\x9d\x3c\xa1\x58" + "\x1a\x22\x53\x05\x6b\x5c\x71\x4f" + "\xfd\x2d\x4d\x4c\xe5\x62\xa5\x63" + "\x6a\xda\x26\xc8\x7f\xff\xea\xfd" + "\x46\x4a\xfa\x53\x8f\xc4\xcd\x68" + "\x58", + .alen = 65, + .input = "\x89\x24\x27\x86\xdc\xd7\x6b\xd9" + "\xd1\xcd\xdc\x16\xdd\x2c\xc1\xfb" + "\x52\xb5\xb3\xab\x50\x99\x3f\xa0" + "\x38\xa4\x74\xa5\x04\x15\x63\x05" + "\x8f\x54\x81\x06\x5a\x6b\xa4\x63" + "\x6d\xa7\x21\xcb\xff\x42\x30\x8e" + "\x3b\xd1\xca\x3f\x4b\x1a\xb8\xc3" + "\x42\x01\xe6\xbc\x75\x15\x87\xee" + "\xc9\x8e\x65\x01\xd9\xd8\xb5\x9f" + "\x48\x86\xa6\x5f\x2c\xc7\xb5\xb0" + "\xed\x5d\x14\x7c\x3f\x40\xb1\x0b" + "\x72\xef\x94\x8d\x7a\x85\x56\xe5" + "\x56\x08\x15\x56\xba\xaf\xbd\xf0" + "\x20\xef\xa0\xf6\xa9\xad\xa2\xc9" + "\x1c\x3b\x28\x51\x7e\x77\xb2\x18" + "\x4f\x61\x64\x37\x22\x36\x6d\x78" + "\xed\xed\x35\xe8\x83\xa5\xec\x25" + "\x6b\xff\x5f\x1a\x09\x96\x3d\xdc" + "\x20", + .ilen = 145, + .result = "\xab\x14\xf3\x0a\xfe\x0a\x85\xa5" + "\xf2\xd5\xbc\x38\x89\x0e\x04\xfb" + "\x84\x7d\x65\x34\x25\xd8\x47\xfa" + "\xeb\x83\x31\xf1\x54\x54\x89\x0d" + "\x9d\x4d\x54\x51\x84\x61\xf6\x8e" + "\x03\x31\xf2\x25\x16\xcc\xaa\xc6" + "\x75\x73\x20\x30\x59\x54\xb2\xf0" + "\x3a\x4b\xe0\x23\x8e\xa6\x08\x35" + "\x8a\xdf\x27\xa0\xe4\x60\x99\xae" + "\x8e\x43\xd9\x39\x7b\x10\x40\x67" + "\x5c\x7e\xc9\x70\x63\x34\xca\x59" + "\xfe\x86\xbc\xb7\x9c\x39\xf3\x6d" + "\x6a\x41\x64\x6f\x16\x7f\x65\x7e" + "\x89\x84\x68\xeb\xb0\x51\xbe\x55" + "\x33\x16\x59\x6c\x3b\xef\x88\xad" + "\x2f\xab\xbc\x25\x76\x87\x41\x2f" + "\x36", + .rlen = 129, + }, { + .key = "\xba\xde\x82\x72\x42\xa9\x2f\x2c" + "\x12\x0b\xe9\x5c\x87\xd7\x35\x7c", + .klen = 16, + .iv = "\xc9\xa7\x10\xda\x86\x48\xd9\xb3" + "\x32\x42\x15\x80\x85\xa1\x65\xfe", + .assoc = "\xd8\x70\x9f\x42\xca\xe6\x83\x3a" + "\x52\x79\x42\xa5\x84\x6a\x96\x7f" + "\xe4\x8f\xed\x97\xe9\xd0\xe8\x0d" + "\x7c\xa6\xd8\xd4\x77\x9b\x40\x2e" + "\x28\xce\x57\x34\xcd\x6e\x84\x4c" + "\x17\x3c\xe1\xb2\xa8\x0b\xbb\xf1" + "\x96\x41\x0d\x69\xe8\x54\x0a\xc8" + "\x15\x4e\x91\x92\x89\x4b\xb7\x9b" + "\x21\xf7\x42\x89\xac\x12\x2a\x54" + "\x69\xee\x18\xc7\x8d\xed\xe8\xfd" + "\xbb\x04\x28\xe6\x8a\x3c\x98\xc1" + "\x04\x2d\xa9\xa1\x24\x83\xff\xe9" + "\x55\x7a\xf0\xd1\xf6\x63\x05\xe1" + "\xd9\x1e\x75\x72\xc1\x9f\xae\x32" + "\xe1\x6b\xcd\x9e\x61\x19\x23\x86" + "\xd9\xd2\xaf\x8e\xd5\xd3\xa8\xa9" + "\x51", + .alen = 129, + .input = "\x36\x78\xb9\x22\xde\x62\x35\x55" + "\x1a\x7a\xf5\x45\xbc\xd7\x15\x82" + "\x01\xe9\x5a\x07\xea\x46\xaf\x91" + "\xcb\x73\xa5\xee\xe1\xb4\xbf\xc2" + "\xdb\xd2\x9d\x59\xde\xfc\x83\x00" + "\xf5\x46\xac\x97\xd5\x57\xa9\xb9" + "\x1f\x8c\xe8\xca\x68\x8b\x91\x0c" + "\x01\xbe\x0a\xaf\x7c\xf6\x67\xa4" + "\xbf\xbc\x88\x3f\x5d\xd1\xf9\x19" + "\x0f\x9d\xb2\xaf\xb9\x6e\x17\xdf" + "\xa2", + .ilen = 81, + .result = "\xe8\x39\x2d\xaa\x0e\x85\x2d\xc1" + "\x72\xaf\x6e\xc9\x82\x33\xc7\x01" + "\xaf\x40\x70\xb8\x2a\x78\xc9\x14" + "\xac\xb1\x10\xca\x2e\xb3\x28\xe4" + "\xac\xfa\x58\x7f\xe5\x73\x09\x8c" + "\x1d\x40\x87\x8c\xd9\x75\xc0\x55" + "\xa2\xda\x07\xd1\xc2\xa9\xd1\xbb" + "\x09\x4f\x77\x62\x88\x2d\xf2\x68" + "\x54", + .rlen = 65, + }, { + .key = "\xf7\x02\xbb\x11\x52\x24\xd8\x48" + "\x93\xe6\x9b\xee\x81\xfc\xf7\x82", + .klen = 16, + .iv = "\x06\xcc\x4a\x79\x96\xc3\x82\xcf" + "\xb3\x1c\xc7\x12\x7f\xc5\x28\x04", + .assoc = "\x15\x95\xd8\xe1\xda\x62\x2c\x56" + "\xd3\x53\xf4\x36\x7e\x8e\x59\x85" + "\x0e\x51\xf9\x1c\xee\x70\x6a\x27" + "\x3d\xd3\xb7\xac\x51\xfa\xdf\x05", + .alen = 32, + .input = "\x08\x1b\x95\x0e\x41\x95\x02\x4b" + "\x9c\xbb\xa8\xd0\x7c\xd3\x44\x6e" + "\x89\x14\x33\x70\x0a\xbc\xea\x39" + "\x88\xaa\x2b\xd5\x73\x11\x55\xf5" + "\x33\x33\x9c\xd7\x42\x34\x49\x8e" + "\x2f\x03\x30\x05\x47\xaf\x34", + .ilen = 47, + .result = "\x24\x5e\x67\x49\x1e\x01\xd6\xdd" + "\xf3\x89\x20\x5b\x7c\x57\x89\x07" + "\xd9\x02\x7c\x3d\x2f\x18\x4b\x2d" + "\x6e\xde\xee\xa2\x08\x12\xc7\xba", + .rlen = 32, + }, { + .key = "\x33\x27\xf5\xb1\x62\xa0\x80\x63" + "\x14\xc0\x4d\x7f\x7b\x20\xba\x89", + .klen = 16, + .iv = "\x42\xf0\x84\x19\xa6\x3f\x2b\xea" + "\x34\xf6\x79\xa3\x79\xe9\xeb\x0a", + .assoc = "\x51\xb9\x12\x80\xea\xde\xd5\x71" + "\x54\x2d\xa6\xc8\x78\xb2\x1b\x8c" + "\x39\x14\x05\xa0\xf3\x10\xec\x41" + "\xff\x01\x95\x84\x2b\x59\x7f\xdb", + .alen = 32, + .input = "\x97\xca\xf4\xe0\x8d\x89\xbf\x68" + "\x0c\x60\xb9\x27\xdf\xaa\x41\xc6" + "\x25\xd8\xf7\x1f\x10\x15\x48\x61" + "\x4c\x95\x00\xdf\x51\x9b\x7f\xe6" + "\x24\x40\x9e\xbe\x3b\xeb\x1b\x98" + "\xb9\x9c\xe5\xef\xf2\x05", + .ilen = 46, + .result = "\x61\x83\xa0\xe8\x2e\x7d\x7f\xf8" + "\x74\x63\xd2\xec\x76\x7c\x4c\x0d" + "\x03\xc4\x88\xc1\x35\xb8\xcd\x47" + "\x2f\x0c\xcd\x7a\xe2\x71\x66\x91", + .rlen = 32, + }, { + .key = "\x70\x4c\x2f\x50\x72\x1c\x29\x7f" + "\x95\x9a\xff\x10\x75\x45\x7d\x8f", + .klen = 16, + .iv = "\x7f\x15\xbd\xb8\xb6\xba\xd3\x06" + "\xb5\xd1\x2b\x35\x73\x0e\xad\x10", + .assoc = "\x8e\xde\x4c\x20\xfa\x59\x7e\x8d" + "\xd5\x07\x58\x59\x72\xd7\xde\x92" + "\x63\xd6\x10\x24\xf8\xb0\x6e\x5a" + "\xc0\x2e\x74\x5d\x06\xb8\x1e\xb2", + .alen = 32, + .input = "\x63\x4c\x2a\x8e\xb4\x6b\x63\x0d" + "\xb5\xec\x9b\x4e\x12\x23\xa3\xcf" + "\x1a\x5a\x70\x15\x5a\x10\x40\x51" + "\xca\x47\x4c\x9d\xc9\x97\xf4\x77" + "\xdb\xc8\x10\x2d\xdc\x65\x20\x3f", + .ilen = 40, + .result = "\x9d\xa7\xda\x88\x3e\xf8\x28\x14" + "\xf5\x3e\x85\x7d\x70\xa0\x0f\x13" + "\x2e\x86\x93\x45\x3a\x58\x4f\x61" + "\xf0\x3a\xac\x53\xbc\xd0\x06\x68", + .rlen = 32, + }, { + .key = "\xac\x70\x69\xef\x82\x97\xd2\x9b" + "\x15\x74\xb1\xa2\x6f\x69\x3f\x95", + .klen = 16, + .iv = "\xbb\x3a\xf7\x57\xc6\x36\x7c\x22" + "\x36\xab\xde\xc6\x6d\x32\x70\x17", + .assoc = "\xcb\x03\x85\xbf\x0a\xd5\x26\xa9" + "\x56\xe1\x0a\xeb\x6c\xfb\xa1\x98" + "\x8d\x98\x1c\xa8\xfe\x50\xf0\x74" + "\x81\x5c\x53\x35\xe0\x17\xbd\x88", + .alen = 32, + .input = "\xf1\x62\x44\xc7\x5f\x19\xca\x43" + "\x47\x2c\xaf\x68\x82\xbd\x51\xef" + "\x3d\x65\xd8\x45\x2d\x06\x07\x78" + "\x08\x2e\xb3\x23\xcd\x81\x12\x55" + "\x1a", + .ilen = 33, + .result = "\xda\xcc\x14\x27\x4e\x74\xd1\x30" + "\x76\x18\x37\x0f\x6a\xc4\xd1\x1a" + "\x58\x49\x9f\xc9\x3f\xf8\xd1\x7a" + "\xb2\x67\x8b\x2b\x96\x2f\xa5\x3e", + .rlen = 32, + }, { + .key = "\xe9\x95\xa2\x8f\x93\x13\x7b\xb7" + "\x96\x4e\x63\x33\x69\x8d\x02\x9b" + "\x23\xf9\x22\xeb\x80\xa0\xb1\x81" + "\xe2\x73\xc3\x21\x4d\x47\x8d\xf4", + .klen = 32, + .iv = "\xf8\x5e\x31\xf7\xd7\xb2\x25\x3e" + "\xb7\x85\x90\x58\x67\x57\x33\x1d", + .assoc = "", + .alen = 0, + .input = "\xdf\x2f\x83\xc0\x45\x4a\x2c\xcf" + "\xb9\xd2\x41\xf6\x80\xa1\x52\x70", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x25\xba\xdc\x2e\xa3\x8f\x24\xd3" + "\x17\x29\x15\xc5\x63\xb2\xc5\xa1" + "\x4d\xbc\x2d\x6f\x85\x40\x33\x9a" + "\xa3\xa0\xa1\xfa\x27\xa6\x2c\xca", + .klen = 32, + .iv = "\x34\x83\x6a\x96\xe7\x2d\xce\x5a" + "\x38\x5f\x42\xe9\x61\x7b\xf5\x23", + .assoc = "", + .alen = 0, + .input = "\x01\xd8\x55\x3c\xc0\x5a\x4b\xc7" + "\x01\xf4\x08\xe3\x0d\xf7\xf0\x78" + "\x53", + .ilen = 17, + .result = "\x53", + .rlen = 1, + }, { + .key = "\x62\xdf\x16\xcd\xb3\x0a\xcc\xef" + "\x98\x03\xc7\x56\x5d\xd6\x87\xa8" + "\x77\x7e\x39\xf3\x8a\xe0\xb5\xb4" + "\x65\xce\x80\xd2\x01\x05\xcb\xa1", + .klen = 32, + .iv = "\x71\xa8\xa4\x35\xf7\xa9\x76\x75" + "\xb8\x39\xf4\x7a\x5b\x9f\xb8\x29", + .assoc = "", + .alen = 0, + .input = "\xc2\x4b\x41\x0f\x2d\xb9\x62\x07" + "\xff\x8e\x74\xf8\xa1\xa6\xd5\x37" + "\xa5\x64\x31\x5c\xca\x73\x9b\x43" + "\xe6\x70\x63\x46\x95\xcb\xf7\xb5" + "\x20\x8c\x75\x7a\x2a\x17\x2f\xa9" + "\xb8\x4d\x11\x42\xd1\xf8\xf1", + .ilen = 47, + .result = "\x8f\x3a\xc1\x05\x7f\xe7\xcb\x83" + "\xf9\xa6\x4d\xc3\x58\x31\x19\x2c" + "\xd7\x90\xc2\x56\x4e\xd8\x57\xc7" + "\xf6\xf0\x27\xb4\x25\x4c\x83", + .rlen = 31, + }, { + .key = "\x9e\x03\x4f\x6d\xc3\x86\x75\x0a" + "\x19\xdd\x79\xe8\x57\xfb\x4a\xae" + "\xa2\x40\x45\x77\x90\x80\x37\xce" + "\x26\xfb\x5f\xaa\xdb\x64\x6b\x77", + .klen = 32, + .iv = "\xae\xcc\xde\xd5\x07\x25\x1f\x91" + "\x39\x14\xa6\x0c\x55\xc4\x7b\x30", + .assoc = "", + .alen = 0, + .input = "\xbb\x01\x7c\xd1\x2c\x33\x7b\x37" + "\x0a\xee\xc4\x30\x19\xd7\x3a\x6f" + "\xf8\x2b\x67\xf5\x3b\x84\x87\x2a" + "\xfb\x07\x7a\x82\xb5\xe4\x85\x26" + "\x1e\xa8\xe5\x04\x54\xce\xe5\x5f" + "\xb5\x3f\xc1\xd5\x7f\xbd\xd2\xa6", + .ilen = 48, + .result = "\xcc\x5f\xfb\xa4\x8f\x63\x74\x9f" + "\x7a\x81\xff\x55\x52\x56\xdc\x33" + "\x01\x52\xcd\xdb\x53\x78\xd9\xe1" + "\xb7\x1d\x06\x8d\xff\xab\x22\x98", + .rlen = 32, + }, { + .key = "\xdb\x28\x89\x0c\xd3\x01\x1e\x26" + "\x9a\xb7\x2b\x79\x51\x1f\x0d\xb4" + "\xcc\x03\x50\xfc\x95\x20\xb9\xe7" + "\xe8\x29\x3e\x83\xb5\xc3\x0a\x4e", + .klen = 32, + .iv = "\xea\xf1\x18\x74\x17\xa0\xc8\xad" + "\xba\xee\x58\x9d\x4f\xe8\x3d\x36", + .assoc = "", + .alen = 0, + .input = "\xc2\xf4\x40\x55\xf9\x59\xff\x73" + "\x08\xf5\x98\x92\x0c\x7b\x35\x9a" + "\xa8\xf4\x42\x7e\x6f\x93\xca\x22" + "\x23\x06\x1e\xf8\x89\x22\xf4\x46" + "\x7c\x7c\x67\x75\xab\xe5\x75\xaa" + "\x15\xd7\x83\x19\xfd\x31\x59\x5b" + "\x32", + .ilen = 49, + .result = "\x08\x84\x34\x44\x9f\xde\x1c\xbb" + "\xfb\x5b\xb1\xe6\x4c\x7a\x9f\x39" + "\x2c\x14\xd9\x5f\x59\x18\x5b\xfb" + "\x79\x4b\xe5\x65\xd9\x0a\xc1\x6f" + "\x2e", + .rlen = 33, + }, { + .key = "\x17\x4d\xc3\xab\xe3\x7d\xc7\x42" + "\x1b\x91\xdd\x0a\x4b\x43\xcf\xba" + "\xf6\xc5\x5c\x80\x9a\xc0\x3b\x01" + "\xa9\x56\x1d\x5b\x8f\x22\xa9\x25", + .klen = 32, + .iv = "\x27\x16\x51\x13\x27\x1c\x71\xc9" + "\x3b\xc8\x0a\x2f\x49\x0c\x00\x3c", + .assoc = "", + .alen = 0, + .input = "\xc9\x82\x3b\x4b\x87\x84\xa5\xdb" + "\xa0\x8c\xd3\x3e\x7f\x8d\xe8\x28" + "\x2a\xdc\xfa\x01\x84\x87\x9a\x70" + "\x81\x75\x37\x0a\xd2\x75\xa9\xb6" + "\x21\x72\xee\x7e\x65\x95\xe5\xcc" + "\x01\xb7\x39\xa6\x51\x15\xca\xff" + "\x61\xdc\x97\x38\xcc\xf4\xca\xc7" + "\x83\x9b\x05\x11\x72\x60\xf0\xb4" + "\x7e\x06\xab\x0a\xc0\xbb\x59\x23" + "\xaa\x2d\xfc\x4e\x35\x05\x59", + .ilen = 79, + .result = "\x45\xa8\x6e\xe3\xaf\x5a\xc5\xd7" + "\x7c\x35\x63\x77\x46\x9f\x61\x3f" + "\x56\xd7\xe4\xe3\x5e\xb8\xdc\x14" + "\x3a\x79\xc4\x3e\xb3\x69\x61\x46" + "\x3c\xb6\x83\x4e\xb4\x26\xc7\x73" + "\x22\xda\x52\x8b\x7d\x11\x98\xea" + "\x62\xe1\x14\x1e\xdc\xfe\x0f\xad" + "\x20\x76\x5a\xdc\x4e\x71\x13", + .rlen = 63, + }, { + .key = "\x54\x71\xfd\x4b\xf3\xf9\x6f\x5e" + "\x9c\x6c\x8f\x9c\x45\x68\x92\xc1" + "\x21\x87\x67\x04\x9f\x60\xbd\x1b" + "\x6a\x84\xfc\x34\x6a\x81\x48\xfb", + .klen = 32, + .iv = "\x63\x3b\x8b\xb3\x37\x98\x1a\xe5" + "\xbc\xa2\xbc\xc0\x43\x31\xc2\x42", + .assoc = "", + .alen = 0, + .input = "\x11\x7c\x7d\xef\xce\x29\x95\xec" + "\x7e\x9f\x42\xa6\x26\x07\xa1\x75" + "\x2f\x4e\x09\x9a\xf6\x6b\xc2\xfa" + "\x0d\xd0\x17\xdc\x25\x1e\x9b\xdc" + "\x5f\x8c\x1c\x60\x15\x4f\x9b\x20" + "\x7b\xff\xcd\x82\x60\x84\xf4\xa5" + "\x20\x9a\x05\x19\x5b\x02\x0a\x72" + "\x43\x11\x26\x58\xcf\xc5\x41\xcf" + "\x13\xcc\xde\x32\x92\xfa\x86\xf2" + "\xaf\x16\xe8\x8f\xca\xb6\xfd\x54", + .ilen = 80, + .result = "\x81\xcd\xa8\x82\xbf\xd6\x6e\xf3" + "\xfd\x0f\x15\x09\x40\xc3\x24\x45" + "\x81\x99\xf0\x67\x63\x58\x5e\x2e" + "\xfb\xa6\xa3\x16\x8d\xc8\x00\x1c" + "\x4b\x62\x87\x7c\x15\x38\xda\x70" + "\x3d\xea\xe7\xf2\x40\xba\xae\x79" + "\x8f\x48\xfc\xbf\x45\x53\x2e\x78" + "\xef\x79\xf0\x1b\x49\xf7\xfd\x9c", + .rlen = 64, + }, { + .key = "\x90\x96\x36\xea\x03\x74\x18\x7a" + "\x1d\x46\x42\x2d\x3f\x8c\x54\xc7" + "\x4b\x4a\x73\x89\xa4\x00\x3f\x34" + "\x2c\xb1\xdb\x0c\x44\xe0\xe8\xd2", + .klen = 32, + .iv = "\xa0\x5f\xc5\x52\x47\x13\xc2\x01" + "\x3d\x7c\x6e\x52\x3d\x55\x85\x48", + .assoc = "\xaf", + .alen = 1, + .input = "\x9b\xc5\x3b\x20\x0a\x88\x56\xbe" + "\x69\xdf\xc4\xc4\x02\x46\x3a\xf0", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xcd\xbb\x70\x89\x13\xf0\xc1\x95" + "\x9e\x20\xf4\xbf\x39\xb1\x17\xcd" + "\x76\x0c\x7f\x0d\xa9\xa0\xc1\x4e" + "\xed\xdf\xb9\xe4\x1e\x3f\x87\xa8", + .klen = 32, + .iv = "\xdc\x84\xfe\xf1\x58\x8f\x6b\x1c" + "\xbe\x57\x20\xe3\x37\x7a\x48\x4f", + .assoc = "\xeb\x4d\x8d\x59\x9c\x2e\x15\xa3" + "\xde\x8d\x4d\x07\x36\x43\x78\xd0" + "\x0b\x6d\x84\x4f\x2c\xf0\x82\x5b" + "\x4e\xf6\x29\xd1\x8b\x6f\x56", + .alen = 31, + .input = "\xe0\x6d\xa1\x07\x98\x2f\x40\x2d" + "\x2e\x9a\xd6\x61\x43\xc0\x74\x69", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x0a\xe0\xaa\x29\x24\x6c\x6a\xb1" + "\x1f\xfa\xa6\x50\x33\xd5\xda\xd3" + "\xa0\xce\x8a\x91\xae\x40\x43\x68" + "\xae\x0d\x98\xbd\xf8\x9e\x26\x7f", + .klen = 32, + .iv = "\x19\xa9\x38\x91\x68\x0b\x14\x38" + "\x3f\x31\xd2\x74\x31\x9e\x0a\x55", + .assoc = "\x28\x72\xc7\xf8\xac\xaa\xbe\xbf" + "\x5f\x67\xff\x99\x30\x67\x3b\xd6" + "\x35\x2f\x90\xd3\x31\x90\x04\x74" + "\x0f\x23\x08\xa9\x65\xce\xf6\xea", + .alen = 32, + .input = "\xb9\x57\x13\x3e\x82\x31\x61\x65" + "\x0d\x7f\x6c\x96\x93\x5c\x50\xe2", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x46\x04\xe3\xc8\x34\xe7\x12\xcd" + "\xa0\xd4\x58\xe2\x2d\xf9\x9c\xda" + "\xca\x91\x96\x15\xb4\xe0\xc5\x81" + "\x70\x3a\x77\x95\xd2\xfd\xc5\x55", + .klen = 32, + .iv = "\x55\xcd\x72\x30\x78\x86\xbd\x54" + "\xc0\x0b\x84\x06\x2b\xc2\xcd\x5b", + .assoc = "\x64\x97\x00\x98\xbc\x25\x67\xdb" + "\xe0\x41\xb1\x2a\x2a\x8c\xfe\xdd" + "\x5f\xf2\x9c\x58\x36\x30\x86\x8e" + "\xd1\x51\xe6\x81\x3f\x2d\x95\xc1" + "\x01", + .alen = 33, + .input = "\x81\x96\x34\xde\xbb\x36\xdd\x3e" + "\x4e\x5e\xcb\x44\x21\xb8\x3f\xf1", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\x83\x29\x1d\x67\x44\x63\xbb\xe9" + "\x20\xaf\x0a\x73\x27\x1e\x5f\xe0" + "\xf5\x53\xa1\x9a\xb9\x80\x47\x9b" + "\x31\x68\x56\x6e\xac\x5c\x65\x2c", + .klen = 32, + .iv = "\x92\xf2\xac\xcf\x88\x02\x65\x70" + "\x41\xe5\x36\x97\x25\xe7\x90\x61", + .assoc = "\xa1\xbb\x3a\x37\xcc\xa1\x10\xf7" + "\x61\x1c\x63\xbc\x24\xb0\xc0\xe3" + "\x8a\xb4\xa7\xdc\x3b\xd0\x08\xa8" + "\x92\x7f\xc5\x5a\x19\x8c\x34\x97" + "\x0f\x95\x9b\x18\xe4\x8d\xb4\x24" + "\xb9\x33\x28\x18\xe1\x9d\x14\xe0" + "\x64\xb2\x89\x7d\x78\xa8\x05\x7e" + "\x07\x8c\xfc\x88\x2d\xb8\x53", + .alen = 63, + .input = "\x2e\x99\xb6\x79\x57\x56\x80\x36" + "\x8e\xc4\x1c\x12\x7d\x71\x36\x0c", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xbf\x4e\x57\x07\x54\xdf\x64\x05" + "\xa1\x89\xbc\x04\x21\x42\x22\xe6" + "\x1f\x15\xad\x1e\xbe\x20\xc9\xb4" + "\xf3\x95\x35\x46\x86\xbb\x04\x03", + .klen = 32, + .iv = "\xce\x17\xe5\x6f\x98\x7e\x0e\x8c" + "\xc2\xbf\xe8\x29\x1f\x0b\x52\x68", + .assoc = "\xdd\xe0\x74\xd6\xdc\x1d\xb8\x13" + "\xe2\xf6\x15\x4d\x1e\xd4\x83\xe9" + "\xb4\x76\xb3\x60\x40\x70\x8a\xc1" + "\x53\xac\xa4\x32\xf3\xeb\xd3\x6e" + "\x1e\x42\xa0\x46\x45\x9f\xc7\x22" + "\xd3\x43\xbc\x7e\xa5\x47\x2a\x6f" + "\x91\x19\x70\x1e\xe1\xfe\x25\x49" + "\xd6\x8f\x93\xc7\x28\x3f\x3d\x03", + .alen = 64, + .input = "\x7b\x25\x3d\x47\xd4\xa7\x08\xce" + "\x3b\x89\x40\x36\xba\x6d\x0e\xa2", + .ilen = 16, + .result = "", + .rlen = 0, + }, { + .key = "\xfc\x72\x90\xa6\x64\x5a\x0d\x21" + "\x22\x63\x6e\x96\x1b\x67\xe4\xec" + "\x49\xd7\xb9\xa2\xc3\xc0\x4b\xce" + "\xb4\xc3\x14\x1e\x61\x1a\xa3\xd9", + .klen = 32, + .iv = "\x0b\x3c\x1f\x0e\xa8\xf9\xb7\xa7" + "\x42\x9a\x9a\xba\x19\x30\x15\x6e", + .assoc = "\x1a", + .alen = 1, + .input = "\xe6\x09\x6f\x95\x9a\x18\xc8\xf6" + "\x17\x75\x81\x16\xdf\x26\xff\x67" + "\x92", + .ilen = 17, + .result = "\x29", + .rlen = 1, + }, { + .key = "\x38\x97\xca\x45\x74\xd6\xb6\x3c" + "\xa3\x3d\x20\x27\x15\x8b\xa7\xf2" + "\x74\x9a\xc4\x27\xc8\x60\xcd\xe8" + "\x75\xf0\xf2\xf7\x3b\x79\x42\xb0", + .klen = 32, + .iv = "\x47\x60\x59\xad\xb8\x75\x60\xc3" + "\xc3\x74\x4c\x4c\x13\x54\xd8\x74", + .assoc = "\x56\x29\xe7\x15\xfc\x14\x0a\x4a" + "\xe4\xaa\x79\x70\x12\x1d\x08\xf6" + "\x09\xfb\xca\x69\x4b\xb0\x8e\xf5" + "\xd6\x07\x62\xe3\xa8\xa9\x12", + .alen = 31, + .input = "\x82\xc0\x56\xf0\xd7\xc4\xc9\xfd" + "\x3c\xd1\x2a\xd4\x15\x86\x9d\xda" + "\xea\x6c\x6f\xa1\x33\xb0\x7a\x01" + "\x57\xe7\xf3\x7b\x73\xe7\x54\x10" + "\xc6\x91\xe2\xc6\xa0\x69\xe7\xe6" + "\x76\xc3\xf5\x3a\x76\xfd\x4a", + .ilen = 47, + .result = "\x66\xf3\x75\x7d\x40\xb3\xb4\xd1" + "\x04\xe1\xa6\x94\x10\xe6\x39\x77" + "\xd3\xac\x4d\x8a\x8c\x58\x6e\xfb" + "\x06\x13\x9a\xd9\x5e\xc0\xfa", + .rlen = 31, + }, { + .key = "\x75\xbc\x04\xe5\x84\x52\x5e\x58" + "\x24\x17\xd2\xb9\x0e\xaf\x6a\xf9" + "\x9e\x5c\xd0\xab\xcd\x00\x4f\x01" + "\x37\x1e\xd1\xcf\x15\xd8\xe2\x86", + .klen = 32, + .iv = "\x84\x85\x92\x4d\xc8\xf1\x08\xdf" + "\x44\x4e\xff\xdd\x0d\x78\x9a\x7a", + .assoc = "\x93\x4e\x21\xb4\x0c\x90\xb3\x66" + "\x65\x84\x2b\x01\x0b\x42\xcb\xfc" + "\x33\xbd\xd6\xed\x50\x50\x10\x0e" + "\x97\x35\x41\xbb\x82\x08\xb1\xf2", + .alen = 32, + .input = "\x01\x47\x8e\x6c\xf6\x64\x89\x3a" + "\x71\xce\xe4\xaa\x45\x70\xe6\x84" + "\x62\x48\x08\x64\x86\x6a\xdf\xec" + "\xb4\xa0\xfb\x34\x03\x0c\x19\xf4" + "\x2b\x7b\x36\x73\xec\x54\xa9\x1e" + "\x30\x85\xdb\xe4\xac\xe9\x2c\xca", + .ilen = 48, + .result = "\xa2\x17\xaf\x1c\x50\x2e\x5d\xed" + "\x85\xbb\x58\x26\x0a\x0b\xfc\x7d" + "\xfe\x6e\x59\x0e\x91\xf8\xf0\x15" + "\xc8\x40\x78\xb1\x38\x1f\x99\xa7", + .rlen = 32, + }, { + .key = "\xb1\xe1\x3e\x84\x94\xcd\x07\x74" + "\xa5\xf2\x84\x4a\x08\xd4\x2c\xff" + "\xc8\x1e\xdb\x2f\xd2\xa0\xd1\x1b" + "\xf8\x4c\xb0\xa8\xef\x37\x81\x5d", + .klen = 32, + .iv = "\xc0\xaa\xcc\xec\xd8\x6c\xb1\xfb" + "\xc5\x28\xb1\x6e\x07\x9d\x5d\x81", + .assoc = "\xd0\x73\x5a\x54\x1d\x0b\x5b\x82" + "\xe5\x5f\xdd\x93\x05\x66\x8e\x02" + "\x5e\x80\xe1\x71\x55\xf0\x92\x28" + "\x59\x62\x20\x94\x5c\x67\x50\xc8" + "\x58", + .alen = 33, + .input = "\x85\xe0\xf8\x0f\x8e\x49\xe3\x60" + "\xcb\x4a\x54\x94\xcf\xf5\x7e\x34" + "\xe9\xf8\x80\x65\x53\xd0\x72\x70" + "\x4f\x7d\x9d\xd1\x15\x6f\xb9\x2c" + "\xfa\xe8\xdd\xac\x2e\xe1\x3f\x67" + "\x63\x0f\x1a\x59\xb7\x89\xdb\xf4" + "\xc3", + .ilen = 49, + .result = "\xdf\x3c\xe9\xbc\x61\xaa\x06\x09" + "\x06\x95\x0a\xb7\x04\x2f\xbe\x84" + "\x28\x30\x64\x92\x96\x98\x72\x2e" + "\x89\x6e\x57\x8a\x13\x7e\x38\x7e" + "\xdb", + .rlen = 33, + }, { + .key = "\xee\x05\x77\x23\xa5\x49\xb0\x90" + "\x26\xcc\x36\xdc\x02\xf8\xef\x05" + "\xf3\xe1\xe7\xb3\xd8\x40\x53\x35" + "\xb9\x79\x8f\x80\xc9\x96\x20\x33", + .klen = 32, + .iv = "\xfd\xce\x06\x8b\xe9\xe8\x5a\x17" + "\x46\x02\x63\x00\x01\xc1\x20\x87", + .assoc = "\x0c\x98\x94\xf3\x2d\x87\x04\x9e" + "\x66\x39\x8f\x24\xff\x8a\x50\x08" + "\x88\x42\xed\xf6\x5a\x90\x14\x42" + "\x1a\x90\xfe\x6c\x36\xc6\xf0\x9f" + "\x66\xa0\xb5\x2d\x2c\xf8\x25\x15" + "\x55\x90\xa2\x7e\x77\x94\x96\x3a" + "\x71\x1c\xf7\x44\xee\xa8\xc3\x42" + "\xe2\xa3\x84\x04\x0b\xe1\xce", + .alen = 63, + .input = "\x00\xe5\x5b\x87\x5c\x20\x22\x8a" + "\xda\x1f\xd3\xff\xbb\xb2\xb0\xf8" + "\xef\xe9\xeb\x9e\x7c\x80\xf4\x2b" + "\x59\xc0\x79\xbc\x17\xa0\x15\x01" + "\xf5\x72\xfb\x5a\xe7\xaf\x07\xe3" + "\x1b\x49\x21\x34\x23\x63\x55\x5e" + "\xee\x4f\x34\x17\xfa\xfe\xa5\x0c" + "\xed\x0b\x23\xea\x9b\xda\x57\x2f" + "\xf6\xa9\xae\x0d\x4e\x40\x96\x45" + "\x7f\xfa\xf0\xbf\xc4\x98\x78", + .ilen = 79, + .result = "\x1b\x61\x23\x5b\x71\x26\xae\x25" + "\x87\x6f\xbc\x49\xfe\x53\x81\x8a" + "\x53\xf2\x70\x17\x9b\x38\xf4\x48" + "\x4b\x9b\x36\x62\xed\xdd\xd8\x54" + "\xea\xcb\xb6\x79\x45\xfc\xaa\x54" + "\x5c\x94\x47\x58\xa7\xff\x9c\x9e" + "\x7c\xb6\xf1\xac\xc8\xfd\x8b\x35" + "\xd5\xa4\x6a\xd4\x09\xc2\x08", + .rlen = 63, + }, { + .key = "\x2a\x2a\xb1\xc3\xb5\xc5\x59\xac" + "\xa7\xa6\xe8\x6d\xfc\x1d\xb2\x0b" + "\x1d\xa3\xf3\x38\xdd\xe0\xd5\x4e" + "\x7b\xa7\x6e\x58\xa3\xf5\xbf\x0a", + .klen = 32, + .iv = "\x39\xf3\x3f\x2b\xf9\x64\x03\x33" + "\xc7\xdd\x15\x91\xfb\xe6\xe2\x8d", + .assoc = "\x49\xbc\xce\x92\x3d\x02\xad\xba" + "\xe7\x13\x41\xb6\xf9\xaf\x13\x0f" + "\xb2\x04\xf8\x7a\x5f\x30\x96\x5b" + "\xdc\xbd\xdd\x44\x10\x25\x8f\x75" + "\x75\x4d\xb9\x5b\x8e\x0a\x38\x13" + "\x6f\x9f\x36\xe4\x3a\x3e\xac\xc9" + "\x9d\x83\xde\xe5\x57\xfd\xe3\x0e" + "\xb1\xa7\x1b\x44\x05\x67\xb7\x37", + .alen = 64, + .input = "\x28\xdd\xb9\x4a\x12\xc7\x0a\xe1" + "\x58\x06\x1a\x9b\x8c\x67\xdf\xeb" + "\x35\x35\x60\x9d\x06\x40\x65\xc1" + "\x93\xe8\xb3\x82\x50\x29\xdd\xb5" + "\x2b\xcb\xde\x18\x78\x6b\x42\xbe" + "\x6d\x24\xd0\xb2\x7d\xd7\x08\x8f" + "\x4a\x18\x98\xad\x8c\xf2\x97\xb4" + "\xf4\x77\xe4\xbf\x41\x3b\xc4\x06" + "\xce\x9e\x34\x81\xf0\x89\x11\x13" + "\x02\x65\xa1\x7c\xdf\x07\x33\x06", + .ilen = 80, + .result = "\x58\x85\x5c\xfa\x81\xa1\x57\x40" + "\x08\x4a\x6e\xda\xf8\x78\x44\x90" + "\x7d\xb5\x7b\x9b\xa1\xd8\x76\x62" + "\x0c\xc9\x15\x3b\xc7\x3c\x77\x2b" + "\xf8\x78\xba\xa7\xa6\x0e\xbd\x52" + "\x76\xa3\xdc\xbe\x6b\xa8\xb1\x2d" + "\xa9\x1d\xd8\x4e\x31\x53\xab\x00" + "\xa5\xa7\x01\x13\x04\x49\xf2\x04", + .rlen = 64, + }, { + .key = "\x67\x4f\xeb\x62\xc5\x40\x01\xc7" + "\x28\x80\x9a\xfe\xf6\x41\x74\x12" + "\x48\x65\xfe\xbc\xe2\x80\x57\x68" + "\x3c\xd4\x4d\x31\x7d\x54\x5f\xe1", + .klen = 32, + .iv = "\x76\x18\x79\xca\x09\xdf\xac\x4e" + "\x48\xb7\xc7\x23\xf5\x0a\xa5\x93", + .assoc = "\x85\xe1\x08\x32\x4d\x7e\x56\xd5" + "\x68\xed\xf3\x47\xf3\xd3\xd6\x15" + "\xdd\xc7\x04\xfe\x64\xd0\x18\x75" + "\x9d\xeb\xbc\x1d\xea\x84\x2e\x4c" + "\x83\xf9\xbe\x8a\xef\x1c\x4b\x10" + "\x89\xaf\xcb\x4b\xfe\xe7\xc1\x58" + "\xca\xea\xc6\x87\xc0\x53\x03\xd9" + "\x80\xaa\xb2\x83\xff\xee\xa1\x6a" + "\x04", + .alen = 65, + .input = "\x85\x39\x69\x35\xfb\xf9\xb0\xa6" + "\x85\x43\x88\xd0\xd7\x78\x60\x19" + "\x3e\x1f\xb1\xa4\xd6\xc5\x96\xec" + "\xf7\x84\x85\xc7\x27\x0f\x74\x57" + "\x28\x9e\xdd\x90\x3c\x43\x12\xc5" + "\x51\x3d\x39\x8f\xa5\xf4\xe0\x0b" + "\x57\x04\xf1\x6d\xfe\x9b\x84\x27" + "\xe8\xeb\x4d\xda\x02\x0a\xc5\x49" + "\x1a\x55\x5e\x50\x56\x4d\x94\xda" + "\x20\xf8\x12\x54\x50\xb3\x11\xda" + "\xed\x44\x27\x67\xd5\xd1\x8b\x4b" + "\x38\x67\x56\x65\x59\xda\xe6\x97" + "\x81\xae\x2f\x92\x3b\xae\x22\x1c" + "\x91\x59\x38\x18\x00\xe8\xba\x92" + "\x04\x19\x56\xdf\xb0\x82\xeb\x6f" + "\x2e\xdb\x54\x3c\x4b\xbb\x60\x90" + "\x4c\x50\x10\x62\xba\x7a\xb1\x68" + "\x37\xd7\x87\x4e\xe4\x66\x09\x1f" + "\xa5", + .ilen = 145, + .result = "\x94\xaa\x96\x9a\x91\x1d\x00\x5c" + "\x88\x24\x20\x6b\xf2\x9c\x06\x96" + "\xa7\x77\x87\x1f\xa6\x78\xf8\x7b" + "\xcd\xf6\xf4\x13\xa1\x9b\x16\x02" + "\x07\x24\xbf\xd5\x08\x20\xd0\x4f" + "\x90\xb3\x70\x24\x2f\x51\xc7\xbb" + "\xd6\x84\xc0\xef\x9a\xa8\xca\xcc" + "\x74\xab\x97\x53\xfe\xd0\xdb\x37" + "\x37\x6a\x0e\x9f\x3f\xa3\x2a\xe3" + "\x1b\x34\x6d\x51\x72\x2b\x17\xe7" + "\x4d\xaa\x2c\x18\xda\xa3\x33\x89" + "\x2a\x9f\xf4\xd2\xed\x76\x3d\x3f" + "\x3c\x15\x9d\x8e\x4f\x3c\x27\xb0" + "\x42\x3f\x2f\x8a\xd4\xc2\x10\xb2" + "\x27\x7f\xe3\x34\x80\x02\x49\x4b" + "\x07\x68\x22\x2a\x88\x25\x53\xb2" + "\x2f", + .rlen = 129, + }, { + .key = "\xa3\x73\x24\x01\xd5\xbc\xaa\xe3" + "\xa9\x5a\x4c\x90\xf0\x65\x37\x18" + "\x72\x28\x0a\x40\xe7\x20\xd9\x82" + "\xfe\x02\x2b\x09\x57\xb3\xfe\xb7", + .klen = 32, + .iv = "\xb3\x3d\xb3\x69\x19\x5b\x54\x6a" + "\xc9\x91\x79\xb4\xef\x2e\x68\x99", + .assoc = "\xc2\x06\x41\xd1\x5d\xfa\xff\xf1" + "\xe9\xc7\xa5\xd9\xed\xf8\x98\x1b" + "\x07\x89\x10\x82\x6a\x70\x9a\x8f" + "\x5e\x19\x9b\xf5\xc5\xe3\xcd\x22" + "\x92\xa5\xc2\xb8\x51\x2e\x5e\x0e" + "\xa4\xbe\x5f\xb1\xc1\x90\xd7\xe7" + "\xf7\x52\xae\x28\x29\xa8\x22\xa4" + "\x4f\xae\x48\xc2\xfa\x75\x8b\x9e" + "\xce\x83\x2a\x88\x07\x55\xbb\x89" + "\xf6\xdf\xac\xdf\x83\x08\xbf\x7d" + "\xac\x30\x8b\x8e\x02\xac\x00\xf1" + "\x30\x46\xe1\xbc\x75\xbf\x49\xbb" + "\x26\x4e\x29\xf0\x2f\x21\xc6\x13" + "\x92\xd9\x3d\x11\xe4\x10\x00\x8e" + "\xd4\xd4\x58\x65\xa6\x2b\xe3\x25" + "\xb1\x8f\x15\x93\xe7\x71\xb9\x2c" + "\x4b", + .alen = 129, + .input = "\x7d\xde\x53\x22\xe4\x23\x3b\x30" + "\x78\xde\x35\x90\x7a\xd9\x0b\x93" + "\xf6\x0e\x0b\xed\x40\xee\x10\x9c" + "\x96\x3a\xd3\x34\xb2\xd0\x67\xcf" + "\x63\x7f\x2d\x0c\xcf\x96\xec\x64" + "\x1a\x87\xcc\x7d\x2c\x5e\x81\x4b" + "\xd2\x8f\x4c\x7c\x00\xb1\xb4\xe0" + "\x87\x4d\xb1\xbc\xd8\x78\x2c\x17" + "\xf2\x3b\xd8\x28\x40\xe2\x76\xf6" + "\x20\x13\x83\x46\xaf\xff\xe3\x0f" + "\x72", + .ilen = 81, + .result = "\xd1\xcf\xd0\x39\xa1\x99\xa9\x78" + "\x09\xfe\xd2\xfd\xec\xc1\xc9\x9d" + "\xd2\x39\x93\xa3\xab\x18\x7a\x95" + "\x8f\x24\xd3\xeb\x7b\xfa\xb5\xd8" + "\x15\xd1\xc3\x04\x69\x32\xe3\x4d" + "\xaa\xc2\x04\x8b\xf2\xfa\xdc\x4a" + "\x02\xeb\xa8\x90\x03\xfd\xea\x97" + "\x43\xaf\x2e\x92\xf8\x57\xc5\x6a" + "\x00", + .rlen = 65, + }, { + .key = "\xe0\x98\x5e\xa1\xe5\x38\x53\xff" + "\x2a\x35\xfe\x21\xea\x8a\xfa\x1e" + "\x9c\xea\x15\xc5\xec\xc0\x5b\x9b" + "\xbf\x2f\x0a\xe1\x32\x12\x9d\x8e", + .klen = 32, + .iv = "\xef\x61\xed\x08\x29\xd7\xfd\x86" + "\x4a\x6b\x2b\x46\xe9\x53\x2a\xa0", + .assoc = "\xfe\x2a\x7b\x70\x6d\x75\xa7\x0d" + "\x6a\xa2\x57\x6a\xe7\x1c\x5b\x21" + "\x31\x4b\x1b\x07\x6f\x10\x1c\xa8" + "\x20\x46\x7a\xce\x9f\x42\x6d\xf9", + .alen = 32, + .input = "\x5a\xcd\x8c\x57\xf2\x6a\xb6\xbe" + "\x53\xc7\xaa\x9a\x60\x74\x9c\xc4" + "\xa2\xc2\xd0\x6d\xe1\x03\x63\xdc" + "\xbb\x51\x7e\x9c\x89\x73\xde\x4e" + "\x24\xf8\x52\x7c\x15\x41\x0e\xba" + "\x69\x0e\x36\x5f\x2f\x22\x8c", + .ilen = 47, + .result = "\x0d\xf4\x09\xd8\xb1\x14\x51\x94" + "\x8a\xd8\x84\x8e\xe6\xe5\x8c\xa3" + "\xfc\xfc\x9e\x28\xb0\xb8\xfc\xaf" + "\x50\x52\xb1\xc4\x55\x59\x55\xaf", + .rlen = 32, + }, { + .key = "\x1c\xbd\x98\x40\xf5\xb3\xfc\x1b" + "\xaa\x0f\xb0\xb3\xe4\xae\xbc\x24" + "\xc7\xac\x21\x49\xf1\x60\xdd\xb5" + "\x80\x5d\xe9\xba\x0c\x71\x3c\x64", + .klen = 32, + .iv = "\x2c\x86\x26\xa8\x39\x52\xa6\xa2" + "\xcb\x45\xdd\xd7\xe3\x77\xed\xa6", + .assoc = "\x3b\x4f\xb5\x10\x7d\xf1\x50\x29" + "\xeb\x7c\x0a\xfb\xe1\x40\x1e\x27" + "\x5c\x0d\x27\x8b\x74\xb0\x9e\xc2" + "\xe1\x74\x59\xa6\x79\xa1\x0c\xd0", + .alen = 32, + .input = "\x47\xd6\xce\x78\xd6\xbf\x4a\x51" + "\xb8\xda\x92\x3c\xfd\xda\xac\x8e" + "\x8d\x88\xd7\x4d\x90\xe5\xeb\xa1" + "\xab\xd6\x7c\x76\xad\xea\x7d\x76" + "\x53\xee\xb0\xcd\xd0\x02\xbb\x70" + "\x5b\x6f\x7b\xe2\x8c\xe8", + .ilen = 46, + .result = "\x4a\x18\x43\x77\xc1\x90\xfa\xb0" + "\x0b\xb2\x36\x20\xe0\x09\x4e\xa9" + "\x26\xbe\xaa\xac\xb5\x58\x7e\xc8" + "\x11\x7f\x90\x9c\x2f\xb8\xf4\x85", + .rlen = 32, + }, { + .key = "\x59\xe1\xd2\xdf\x05\x2f\xa4\x37" + "\x2b\xe9\x63\x44\xde\xd3\x7f\x2b" + "\xf1\x6f\x2d\xcd\xf6\x00\x5f\xcf" + "\x42\x8a\xc8\x92\xe6\xd0\xdc\x3b", + .klen = 32, + .iv = "\x68\xab\x60\x47\x49\xce\x4f\xbe" + "\x4c\x20\x8f\x68\xdd\x9c\xb0\xac", + .assoc = "\x77\x74\xee\xaf\x8d\x6d\xf9\x45" + "\x6c\x56\xbc\x8d\xdb\x65\xe0\x2e" + "\x86\xd0\x32\x0f\x79\x50\x20\xdb" + "\xa2\xa1\x37\x7e\x53\x00\xab\xa6", + .alen = 32, + .input = "\x9f\xa9\x2b\xa4\x8f\x00\x05\x2b" + "\xe7\x68\x81\x51\xbb\xfb\xdf\x60" + "\xbb\xac\xe8\xc1\xdc\x68\xae\x68" + "\x3a\xcd\x7a\x06\x49\xfe\x80\x11" + "\xe6\x61\x99\xe2\xdd\xbe\x2c\xbf", + .ilen = 40, + .result = "\x86\x3d\x7d\x17\xd1\x0c\xa3\xcc" + "\x8c\x8d\xe8\xb1\xda\x2e\x11\xaf" + "\x51\x80\xb5\x30\xba\xf8\x00\xe2" + "\xd3\xad\x6f\x75\x09\x18\x93\x5c", + .rlen = 32, + }, { + .key = "\x96\x06\x0b\x7f\x15\xab\x4d\x53" + "\xac\xc3\x15\xd6\xd8\xf7\x42\x31" + "\x1b\x31\x38\x51\xfc\xa0\xe1\xe8" + "\x03\xb8\xa7\x6b\xc0\x2f\x7b\x11", + .klen = 32, + .iv = "\xa5\xcf\x9a\xe6\x59\x4a\xf7\xd9" + "\xcd\xfa\x41\xfa\xd7\xc0\x72\xb2", + .assoc = "\xb4\x99\x28\x4e\x9d\xe8\xa2\x60" + "\xed\x30\x6e\x1e\xd5\x89\xa3\x34" + "\xb1\x92\x3e\x93\x7e\xf0\xa2\xf5" + "\x64\xcf\x16\x57\x2d\x5f\x4a\x7d", + .alen = 32, + .input = "\xe2\x34\xfa\x25\xfd\xfb\x89\x5e" + "\x5b\x4e\x0b\x15\x6e\x39\xfb\x0c" + "\x73\xc7\xd9\x6b\xbe\xce\x9b\x70" + "\xc7\x4f\x96\x16\x03\xfc\xea\xfb" + "\x56", + .ilen = 33, + .result = "\xc3\x62\xb7\xb6\xe2\x87\x4c\xe7" + "\x0d\x67\x9a\x43\xd4\x52\xd4\xb5" + "\x7b\x43\xc1\xb5\xbf\x98\x82\xfc" + "\x94\xda\x4e\x4d\xe4\x77\x32\x32", + .rlen = 32, + }, +}; + +/* + * All key wrapping test vectors taken from + * http://csrc.nist.gov/groups/STM/cavp/documents/mac/kwtestvectors.zip + * + * Note: as documented in keywrap.c, the ivout for encryption is the first + * semiblock of the ciphertext from the test vector. For decryption, iv is + * the first semiblock of the ciphertext. + */ +static const struct cipher_testvec aes_kw_tv_template[] = { + { + .key = "\x75\x75\xda\x3a\x93\x60\x7c\xc2" + "\xbf\xd8\xce\xc7\xaa\xdf\xd9\xa6", + .klen = 16, + .ptext = "\x42\x13\x6d\x3c\x38\x4a\x3e\xea" + "\xc9\x5a\x06\x6f\xd2\x8f\xed\x3f", + .ctext = "\xf6\x85\x94\x81\x6f\x64\xca\xa3" + "\xf5\x6f\xab\xea\x25\x48\xf5\xfb", + .len = 16, + .iv = "\x03\x1f\x6b\xd7\xe6\x1e\x64\x3d", + .generates_iv = true, + }, { + .key = "\x80\xaa\x99\x73\x27\xa4\x80\x6b" + "\x6a\x7a\x41\xa5\x2b\x86\xc3\x71" + "\x03\x86\xf9\x32\x78\x6e\xf7\x96" + "\x76\xfa\xfb\x90\xb8\x26\x3c\x5f", + .klen = 32, + .ptext = "\x0a\x25\x6b\xa7\x5c\xfa\x03\xaa" + "\xa0\x2b\xa9\x42\x03\xf1\x5b\xaa", + .ctext = "\xd3\x3d\x3d\x97\x7b\xf0\xa9\x15" + "\x59\xf9\x9c\x8a\xcd\x29\x3d\x43", + .len = 16, + .iv = "\x42\x3c\x96\x0d\x8a\x2a\xc4\xc1", + .generates_iv = true, + }, +}; + +/* + * ANSI X9.31 Continuous Pseudo-Random Number Generator (AES mode) + * test vectors, taken from Appendix B.2.9 and B.2.10: + * http://csrc.nist.gov/groups/STM/cavp/documents/rng/RNGVS.pdf + * Only AES-128 is supported at this time. + */ +static const struct cprng_testvec ansi_cprng_aes_tv_template[] = { + { + .key = "\xf3\xb1\x66\x6d\x13\x60\x72\x42" + "\xed\x06\x1c\xab\xb8\xd4\x62\x02", + .klen = 16, + .dt = "\xe6\xb3\xbe\x78\x2a\x23\xfa\x62" + "\xd7\x1d\x4a\xfb\xb0\xe9\x22\xf9", + .dtlen = 16, + .v = "\x80\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .vlen = 16, + .result = "\x59\x53\x1e\xd1\x3b\xb0\xc0\x55" + "\x84\x79\x66\x85\xc1\x2f\x76\x41", + .rlen = 16, + .loops = 1, + }, { + .key = "\xf3\xb1\x66\x6d\x13\x60\x72\x42" + "\xed\x06\x1c\xab\xb8\xd4\x62\x02", + .klen = 16, + .dt = "\xe6\xb3\xbe\x78\x2a\x23\xfa\x62" + "\xd7\x1d\x4a\xfb\xb0\xe9\x22\xfa", + .dtlen = 16, + .v = "\xc0\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .vlen = 16, + .result = "\x7c\x22\x2c\xf4\xca\x8f\xa2\x4c" + "\x1c\x9c\xb6\x41\xa9\xf3\x22\x0d", + .rlen = 16, + .loops = 1, + }, { + .key = "\xf3\xb1\x66\x6d\x13\x60\x72\x42" + "\xed\x06\x1c\xab\xb8\xd4\x62\x02", + .klen = 16, + .dt = "\xe6\xb3\xbe\x78\x2a\x23\xfa\x62" + "\xd7\x1d\x4a\xfb\xb0\xe9\x22\xfb", + .dtlen = 16, + .v = "\xe0\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .vlen = 16, + .result = "\x8a\xaa\x00\x39\x66\x67\x5b\xe5" + "\x29\x14\x28\x81\xa9\x4d\x4e\xc7", + .rlen = 16, + .loops = 1, + }, { + .key = "\xf3\xb1\x66\x6d\x13\x60\x72\x42" + "\xed\x06\x1c\xab\xb8\xd4\x62\x02", + .klen = 16, + .dt = "\xe6\xb3\xbe\x78\x2a\x23\xfa\x62" + "\xd7\x1d\x4a\xfb\xb0\xe9\x22\xfc", + .dtlen = 16, + .v = "\xf0\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .vlen = 16, + .result = "\x88\xdd\xa4\x56\x30\x24\x23\xe5" + "\xf6\x9d\xa5\x7e\x7b\x95\xc7\x3a", + .rlen = 16, + .loops = 1, + }, { + .key = "\xf3\xb1\x66\x6d\x13\x60\x72\x42" + "\xed\x06\x1c\xab\xb8\xd4\x62\x02", + .klen = 16, + .dt = "\xe6\xb3\xbe\x78\x2a\x23\xfa\x62" + "\xd7\x1d\x4a\xfb\xb0\xe9\x22\xfd", + .dtlen = 16, + .v = "\xf8\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .vlen = 16, + .result = "\x05\x25\x92\x46\x61\x79\xd2\xcb" + "\x78\xc4\x0b\x14\x0a\x5a\x9a\xc8", + .rlen = 16, + .loops = 1, + }, { /* Monte Carlo Test */ + .key = "\x9f\x5b\x51\x20\x0b\xf3\x34\xb5" + "\xd8\x2b\xe8\xc3\x72\x55\xc8\x48", + .klen = 16, + .dt = "\x63\x76\xbb\xe5\x29\x02\xba\x3b" + "\x67\xc9\x25\xfa\x70\x1f\x11\xac", + .dtlen = 16, + .v = "\x57\x2c\x8e\x76\x87\x26\x47\x97" + "\x7e\x74\xfb\xdd\xc4\x95\x01\xd1", + .vlen = 16, + .result = "\x48\xe9\xbd\x0d\x06\xee\x18\xfb" + "\xe4\x57\x90\xd5\xc3\xfc\x9b\x73", + .rlen = 16, + .loops = 10000, + }, +}; + +/* + * SP800-90A DRBG Test vectors from + * http://csrc.nist.gov/groups/STM/cavp/documents/drbg/drbgtestvectors.zip + * + * Test vectors for DRBG with prediction resistance. All types of DRBGs + * (Hash, HMAC, CTR) are tested with all permutations of use cases (w/ and + * w/o personalization string, w/ and w/o additional input string). + */ +static const struct drbg_testvec drbg_pr_sha256_tv_template[] = { + { + .entropy = (unsigned char *) + "\x72\x88\x4c\xcd\x6c\x85\x57\x70\xf7\x0b\x8b\x86" + "\xc1\xeb\xd2\x4e\x36\x14\xab\x18\xc4\x9c\xc9\xcf" + "\x1a\xe8\xf7\x7b\x02\x49\x73\xd7\xf1\x42\x7d\xc6" + "\x3f\x29\x2d\xec\xd3\x66\x51\x3f\x1d\x8d\x5b\x4e", + .entropylen = 48, + .entpra = (unsigned char *) + "\x38\x9c\x91\xfa\xc2\xa3\x46\x89\x56\x08\x3f\x62" + "\x73\xd5\x22\xa9\x29\x63\x3a\x1d\xe5\x5d\x5e\x4f" + "\x67\xb0\x67\x7a\x5e\x9e\x0c\x62", + .entprb = (unsigned char *) + "\xb2\x8f\x36\xb2\xf6\x8d\x39\x13\xfa\x6c\x66\xcf" + "\x62\x8a\x7e\x8c\x12\x33\x71\x9c\x69\xe4\xa5\xf0" + "\x8c\xee\xeb\x9c\xf5\x31\x98\x31", + .entprlen = 32, + .expected = (unsigned char *) + "\x52\x7b\xa3\xad\x71\x77\xa4\x49\x42\x04\x61\xc7" + "\xf0\xaf\xa5\xfd\xd3\xb3\x0d\x6a\x61\xba\x35\x49" + "\xbb\xaa\xaf\xe4\x25\x7d\xb5\x48\xaf\x5c\x18\x3d" + "\x33\x8d\x9d\x45\xdf\x98\xd5\x94\xa8\xda\x92\xfe" + "\xc4\x3c\x94\x2a\xcf\x7f\x7b\xf2\xeb\x28\xa9\xf1" + "\xe0\x86\x30\xa8\xfe\xf2\x48\x90\x91\x0c\x75\xb5" + "\x3c\x00\xf0\x4d\x09\x4f\x40\xa7\xa2\x8c\x52\xdf" + "\x52\xef\x17\xbf\x3d\xd1\xa2\x31\xb4\xb8\xdc\xe6" + "\x5b\x0d\x1f\x78\x36\xb4\xe6\x4b\xa7\x11\x25\xd5" + "\x94\xc6\x97\x36\xab\xf0\xe5\x31\x28\x6a\xbb\xce" + "\x30\x81\xa6\x8f\x27\x14\xf8\x1c", + .expectedlen = 128, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\x5d\xf2\x14\xbc\xf6\xb5\x4e\x0b\xf0\x0d\x6f\x2d" + "\xe2\x01\x66\x7b\xd0\xa4\x73\xa4\x21\xdd\xb0\xc0" + "\x51\x79\x09\xf4\xea\xa9\x08\xfa\xa6\x67\xe0\xe1" + "\xd1\x88\xa8\xad\xee\x69\x74\xb3\x55\x06\x9b\xf6", + .entropylen = 48, + .entpra = (unsigned char *) + "\xef\x48\x06\xa2\xc2\x45\xf1\x44\xfa\x34\x2c\xeb" + "\x8d\x78\x3c\x09\x8f\x34\x72\x20\xf2\xe7\xfd\x13" + "\x76\x0a\xf6\xdc\x3c\xf5\xc0\x15", + .entprb = (unsigned char *) + "\x4b\xbe\xe5\x24\xed\x6a\x2d\x0c\xdb\x73\x5e\x09" + "\xf9\xad\x67\x7c\x51\x47\x8b\x6b\x30\x2a\xc6\xde" + "\x76\xaa\x55\x04\x8b\x0a\x72\x95", + .entprlen = 32, + .expected = (unsigned char *) + "\x3b\x14\x71\x99\xa1\xda\xa0\x42\xe6\xc8\x85\x32" + "\x70\x20\x32\x53\x9a\xbe\xd1\x1e\x15\xef\xfb\x4c" + "\x25\x6e\x19\x3a\xf0\xb9\xcb\xde\xf0\x3b\xc6\x18" + "\x4d\x85\x5a\x9b\xf1\xe3\xc2\x23\x03\x93\x08\xdb" + "\xa7\x07\x4b\x33\x78\x40\x4d\xeb\x24\xf5\x6e\x81" + "\x4a\x1b\x6e\xa3\x94\x52\x43\xb0\xaf\x2e\x21\xf4" + "\x42\x46\x8e\x90\xed\x34\x21\x75\xea\xda\x67\xb6" + "\xe4\xf6\xff\xc6\x31\x6c\x9a\x5a\xdb\xb3\x97\x13" + "\x09\xd3\x20\x98\x33\x2d\x6d\xd7\xb5\x6a\xa8\xa9" + "\x9a\x5b\xd6\x87\x52\xa1\x89\x2b\x4b\x9c\x64\x60" + "\x50\x47\xa3\x63\x81\x16\xaf\x19", + .expectedlen = 128, + .addtla = (unsigned char *) + "\xbe\x13\xdb\x2a\xe9\xa8\xfe\x09\x97\xe1\xce\x5d" + "\xe8\xbb\xc0\x7c\x4f\xcb\x62\x19\x3f\x0f\xd2\xad" + "\xa9\xd0\x1d\x59\x02\xc4\xff\x70", + .addtlb = (unsigned char *) + "\x6f\x96\x13\xe2\xa7\xf5\x6c\xfe\xdf\x66\xe3\x31" + "\x63\x76\xbf\x20\x27\x06\x49\xf1\xf3\x01\x77\x41" + "\x9f\xeb\xe4\x38\xfe\x67\x00\xcd", + .addtllen = 32, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\xc6\x1c\xaf\x83\xa2\x56\x38\xf9\xb0\xbc\xd9\x85" + "\xf5\x2e\xc4\x46\x9c\xe1\xb9\x40\x98\x70\x10\x72" + "\xd7\x7d\x15\x85\xa1\x83\x5a\x97\xdf\xc8\xa8\xe8" + "\x03\x4c\xcb\x70\x35\x8b\x90\x94\x46\x8a\x6e\xa1", + .entropylen = 48, + .entpra = (unsigned char *) + "\xc9\x05\xa4\xcf\x28\x80\x4b\x93\x0f\x8b\xc6\xf9" + "\x09\x41\x58\x74\xe9\xec\x28\xc7\x53\x0a\x73\x60" + "\xba\x0a\xde\x57\x5b\x4b\x9f\x29", + .entprb = (unsigned char *) + "\x4f\x31\xd2\xeb\xac\xfa\xa8\xe2\x01\x7d\xf3\xbd" + "\x42\xbd\x20\xa0\x30\x65\x74\xd5\x5d\xd2\xad\xa4" + "\xa9\xeb\x1f\x4d\xf6\xfd\xb8\x26", + .entprlen = 32, + .expected = (unsigned char *) + "\xf6\x13\x05\xcb\x83\x60\x16\x42\x49\x1d\xc6\x25" + "\x3b\x8c\x31\xa3\xbe\x8b\xbd\x1c\xe2\xec\x1d\xde" + "\xbb\xbf\xa1\xac\xa8\x9f\x50\xce\x69\xce\xef\xd5" + "\xd6\xf2\xef\x6a\xf7\x81\x38\xdf\xbc\xa7\x5a\xb9" + "\xb2\x42\x65\xab\xe4\x86\x8d\x2d\x9d\x59\x99\x2c" + "\x5a\x0d\x71\x55\x98\xa4\x45\xc2\x8d\xdb\x05\x5e" + "\x50\x21\xf7\xcd\xe8\x98\x43\xce\x57\x74\x63\x4c" + "\xf3\xb1\xa5\x14\x1e\x9e\x01\xeb\x54\xd9\x56\xae" + "\xbd\xb6\x6f\x1a\x47\x6b\x3b\x44\xe4\xa2\xe9\x3c" + "\x6c\x83\x12\x30\xb8\x78\x7f\x8e\x54\x82\xd4\xfe" + "\x90\x35\x0d\x4c\x4d\x85\xe7\x13", + .expectedlen = 128, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = (unsigned char *) + "\xa5\xbf\xac\x4f\x71\xa1\xbb\x67\x94\xc6\x50\xc7" + "\x2a\x45\x9e\x10\xa8\xed\xf7\x52\x4f\xfe\x21\x90" + "\xa4\x1b\xe1\xe2\x53\xcc\x61\x47", + .perslen = 32, + }, { + .entropy = (unsigned char *) + "\xb6\xc1\x8d\xdf\x99\x54\xbe\x95\x10\x48\xd9\xf6" + "\xd7\x48\xa8\x73\x2d\x74\xde\x1e\xde\x57\x7e\xf4" + "\x7b\x7b\x64\xef\x88\x7a\xa8\x10\x4b\xe1\xc1\x87" + "\xbb\x0b\xe1\x39\x39\x50\xaf\x68\x9c\xa2\xbf\x5e", + .entropylen = 48, + .entpra = (unsigned char *) + "\xdc\x81\x0a\x01\x58\xa7\x2e\xce\xee\x48\x8c\x7c" + "\x77\x9e\x3c\xf1\x17\x24\x7a\xbb\xab\x9f\xca\x12" + "\x19\xaf\x97\x2d\x5f\xf9\xff\xfc", + .entprb = (unsigned char *) + "\xaf\xfc\x4f\x98\x8b\x93\x95\xc1\xb5\x8b\x7f\x73" + "\x6d\xa6\xbe\x6d\x33\xeb\x2c\x82\xb1\xaf\xc1\xb6" + "\xb6\x05\xe2\x44\xaa\xfd\xe7\xdb", + .entprlen = 32, + .expected = (unsigned char *) + "\x51\x79\xde\x1c\x0f\x58\xf3\xf4\xc9\x57\x2e\x31" + "\xa7\x09\xa1\x53\x64\x63\xa2\xc5\x1d\x84\x88\x65" + "\x01\x1b\xc6\x16\x3c\x49\x5b\x42\x8e\x53\xf5\x18" + "\xad\x94\x12\x0d\x4f\x55\xcc\x45\x5c\x98\x0f\x42" + "\x28\x2f\x47\x11\xf9\xc4\x01\x97\x6b\xa0\x94\x50" + "\xa9\xd1\x5e\x06\x54\x3f\xdf\xbb\xc4\x98\xee\x8b" + "\xba\xa9\xfa\x49\xee\x1d\xdc\xfb\x50\xf6\x51\x9f" + "\x6c\x4a\x9a\x6f\x63\xa2\x7d\xad\xaf\x3a\x24\xa0" + "\xd9\x9f\x07\xeb\x15\xee\x26\xe0\xd5\x63\x39\xda" + "\x3c\x59\xd6\x33\x6c\x02\xe8\x05\x71\x46\x68\x44" + "\x63\x4a\x68\x72\xe9\xf5\x55\xfe", + .expectedlen = 128, + .addtla = (unsigned char *) + "\x15\x20\x2f\xf6\x98\x28\x63\xa2\xc4\x4e\xbb\x6c" + "\xb2\x25\x92\x61\x79\xc9\x22\xc4\x61\x54\x96\xff" + "\x4a\x85\xca\x80\xfe\x0d\x1c\xd0", + .addtlb = (unsigned char *) + "\xde\x29\x8e\x03\x42\x61\xa3\x28\x5e\xc8\x80\xc2" + "\x6d\xbf\xad\x13\xe1\x8d\x2a\xc7\xe8\xc7\x18\x89" + "\x42\x58\x9e\xd6\xcc\xad\x7b\x1e", + .addtllen = 32, + .pers = (unsigned char *) + "\x84\xc3\x73\x9e\xce\xb3\xbc\x89\xf7\x62\xb3\xe1" + "\xd7\x48\x45\x8a\xa9\xcc\xe9\xed\xd5\x81\x84\x52" + "\x82\x4c\xdc\x19\xb8\xf8\x92\x5c", + .perslen = 32, + }, +}; + +static const struct drbg_testvec drbg_pr_hmac_sha256_tv_template[] = { + { + .entropy = (unsigned char *) + "\x99\x69\xe5\x4b\x47\x03\xff\x31\x78\x5b\x87\x9a" + "\x7e\x5c\x0e\xae\x0d\x3e\x30\x95\x59\xe9\xfe\x96" + "\xb0\x67\x6d\x49\xd5\x91\xea\x4d\x07\xd2\x0d\x46" + "\xd0\x64\x75\x7d\x30\x23\xca\xc2\x37\x61\x27\xab", + .entropylen = 48, + .entpra = (unsigned char *) + "\xc6\x0f\x29\x99\x10\x0f\x73\x8c\x10\xf7\x47\x92" + "\x67\x6a\x3f\xc4\xa2\x62\xd1\x37\x21\x79\x80\x46" + "\xe2\x9a\x29\x51\x81\x56\x9f\x54", + .entprb = (unsigned char *) + "\xc1\x1d\x45\x24\xc9\x07\x1b\xd3\x09\x60\x15\xfc" + "\xf7\xbc\x24\xa6\x07\xf2\x2f\xa0\x65\xc9\x37\x65" + "\x8a\x2a\x77\xa8\x69\x90\x89\xf4", + .entprlen = 32, + .expected = (unsigned char *) + "\xab\xc0\x15\x85\x60\x94\x80\x3a\x93\x8d\xff\xd2" + "\x0d\xa9\x48\x43\x87\x0e\xf9\x35\xb8\x2c\xfe\xc1" + "\x77\x06\xb8\xf5\x51\xb8\x38\x50\x44\x23\x5d\xd4" + "\x4b\x59\x9f\x94\xb3\x9b\xe7\x8d\xd4\x76\xe0\xcf" + "\x11\x30\x9c\x99\x5a\x73\x34\xe0\xa7\x8b\x37\xbc" + "\x95\x86\x23\x50\x86\xfa\x3b\x63\x7b\xa9\x1c\xf8" + "\xfb\x65\xef\xa2\x2a\x58\x9c\x13\x75\x31\xaa\x7b" + "\x2d\x4e\x26\x07\xaa\xc2\x72\x92\xb0\x1c\x69\x8e" + "\x6e\x01\xae\x67\x9e\xb8\x7c\x01\xa8\x9c\x74\x22" + "\xd4\x37\x2d\x6d\x75\x4a\xba\xbb\x4b\xf8\x96\xfc" + "\xb1\xcd\x09\xd6\x92\xd0\x28\x3f", + .expectedlen = 128, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\xb9\x1f\xe9\xef\xdd\x9b\x7d\x20\xb6\xec\xe0\x2f" + "\xdb\x76\x24\xce\x41\xc8\x3a\x4a\x12\x7f\x3e\x2f" + "\xae\x05\x99\xea\xb5\x06\x71\x0d\x0c\x4c\xb4\x05" + "\x26\xc6\xbd\xf5\x7f\x2a\x3d\xf2\xb5\x49\x7b\xda", + .entropylen = 48, + .entpra = (unsigned char *) + "\xef\x67\x50\x9c\xa7\x7d\xdf\xb7\x2d\x81\x01\xa4" + "\x62\x81\x6a\x69\x5b\xb3\x37\x45\xa7\x34\x8e\x26" + "\x46\xd9\x26\xa2\x19\xd4\x94\x43", + .entprb = (unsigned char *) + "\x97\x75\x53\x53\xba\xb4\xa6\xb2\x91\x60\x71\x79" + "\xd1\x6b\x4a\x24\x9a\x34\x66\xcc\x33\xab\x07\x98" + "\x51\x78\x72\xb2\x79\xfd\x2c\xff", + .entprlen = 32, + .expected = (unsigned char *) + "\x9c\xdc\x63\x8a\x19\x23\x22\x66\x0c\xc5\xb9\xd7" + "\xfb\x2a\xb0\x31\xe3\x8a\x36\xa8\x5a\xa8\x14\xda" + "\x1e\xa9\xcc\xfe\xb8\x26\x44\x83\x9f\xf6\xff\xaa" + "\xc8\x98\xb8\x30\x35\x3b\x3d\x36\xd2\x49\xd4\x40" + "\x62\x0a\x65\x10\x76\x55\xef\xc0\x95\x9c\xa7\xda" + "\x3f\xcf\xb7\x7b\xc6\xe1\x28\x52\xfc\x0c\xe2\x37" + "\x0d\x83\xa7\x51\x4b\x31\x47\x3c\xe1\x3c\xae\x70" + "\x01\xc8\xa3\xd3\xc2\xac\x77\x9c\xd1\x68\x77\x9b" + "\x58\x27\x3b\xa5\x0f\xc2\x7a\x8b\x04\x65\x62\xd5" + "\xe8\xd6\xfe\x2a\xaf\xd3\xd3\xfe\xbd\x18\xfb\xcd" + "\xcd\x66\xb5\x01\x69\x66\xa0\x3c", + .expectedlen = 128, + .addtla = (unsigned char *) + "\x17\xc1\x56\xcb\xcc\x50\xd6\x03\x7d\x45\x76\xa3" + "\x75\x76\xc1\x4a\x66\x1b\x2e\xdf\xb0\x2e\x7d\x56" + "\x6d\x99\x3b\xc6\x58\xda\x03\xf6", + .addtlb = (unsigned char *) + "\x7c\x7b\x4a\x4b\x32\x5e\x6f\x67\x34\xf5\x21\x4c" + "\xf9\x96\xf9\xbf\x1c\x8c\x81\xd3\x9b\x60\x6a\x44" + "\xc6\x03\xa2\xfb\x13\x20\x19\xb7", + .addtllen = 32, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\x13\x54\x96\xfc\x1b\x7d\x28\xf3\x18\xc9\xa7\x89" + "\xb6\xb3\xc8\x72\xac\x00\xd4\x59\x36\x25\x05\xaf" + "\xa5\xdb\x96\xcb\x3c\x58\x46\x87\xa5\xaa\xbf\x20" + "\x3b\xfe\x23\x0e\xd1\xc7\x41\x0f\x3f\xc9\xb3\x67", + .entropylen = 48, + .entpra = (unsigned char *) + "\xe2\xbd\xb7\x48\x08\x06\xf3\xe1\x93\x3c\xac\x79" + "\xa7\x2b\x11\xda\xe3\x2e\xe1\x91\xa5\x02\x19\x57" + "\x20\x28\xad\xf2\x60\xd7\xcd\x45", + .entprb = (unsigned char *) + "\x8b\xd4\x69\xfc\xff\x59\x95\x95\xc6\x51\xde\x71" + "\x68\x5f\xfc\xf9\x4a\xab\xec\x5a\xcb\xbe\xd3\x66" + "\x1f\xfa\x74\xd3\xac\xa6\x74\x60", + .entprlen = 32, + .expected = (unsigned char *) + "\x1f\x9e\xaf\xe4\xd2\x46\xb7\x47\x41\x4c\x65\x99" + "\x01\xe9\x3b\xbb\x83\x0c\x0a\xb0\xc1\x3a\xe2\xb3" + "\x31\x4e\xeb\x93\x73\xee\x0b\x26\xc2\x63\xa5\x75" + "\x45\x99\xd4\x5c\x9f\xa1\xd4\x45\x87\x6b\x20\x61" + "\x40\xea\x78\xa5\x32\xdf\x9e\x66\x17\xaf\xb1\x88" + "\x9e\x2e\x23\xdd\xc1\xda\x13\x97\x88\xa5\xb6\x5e" + "\x90\x14\x4e\xef\x13\xab\x5c\xd9\x2c\x97\x9e\x7c" + "\xd7\xf8\xce\xea\x81\xf5\xcd\x71\x15\x49\x44\xce" + "\x83\xb6\x05\xfb\x7d\x30\xb5\x57\x2c\x31\x4f\xfc" + "\xfe\x80\xb6\xc0\x13\x0c\x5b\x9b\x2e\x8f\x3d\xfc" + "\xc2\xa3\x0c\x11\x1b\x80\x5f\xf3", + .expectedlen = 128, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = (unsigned char *) + "\x64\xb6\xfc\x60\xbc\x61\x76\x23\x6d\x3f\x4a\x0f" + "\xe1\xb4\xd5\x20\x9e\x70\xdd\x03\x53\x6d\xbf\xce" + "\xcd\x56\x80\xbc\xb8\x15\xc8\xaa", + .perslen = 32, + }, { + .entropy = (unsigned char *) + "\xc7\xcc\xbc\x67\x7e\x21\x66\x1e\x27\x2b\x63\xdd" + "\x3a\x78\xdc\xdf\x66\x6d\x3f\x24\xae\xcf\x37\x01" + "\xa9\x0d\x89\x8a\xa7\xdc\x81\x58\xae\xb2\x10\x15" + "\x7e\x18\x44\x6d\x13\xea\xdf\x37\x85\xfe\x81\xfb", + .entropylen = 48, + .entpra = (unsigned char *) + "\x7b\xa1\x91\x5b\x3c\x04\xc4\x1b\x1d\x19\x2f\x1a" + "\x18\x81\x60\x3c\x6c\x62\x91\xb7\xe9\xf5\xcb\x96" + "\xbb\x81\x6a\xcc\xb5\xae\x55\xb6", + .entprb = (unsigned char *) + "\x99\x2c\xc7\x78\x7e\x3b\x88\x12\xef\xbe\xd3\xd2" + "\x7d\x2a\xa5\x86\xda\x8d\x58\x73\x4a\x0a\xb2\x2e" + "\xbb\x4c\x7e\xe3\x9a\xb6\x81\xc1", + .entprlen = 32, + .expected = (unsigned char *) + "\x95\x6f\x95\xfc\x3b\xb7\xfe\x3e\xd0\x4e\x1a\x14" + "\x6c\x34\x7f\x7b\x1d\x0d\x63\x5e\x48\x9c\x69\xe6" + "\x46\x07\xd2\x87\xf3\x86\x52\x3d\x98\x27\x5e\xd7" + "\x54\xe7\x75\x50\x4f\xfb\x4d\xfd\xac\x2f\x4b\x77" + "\xcf\x9e\x8e\xcc\x16\xa2\x24\xcd\x53\xde\x3e\xc5" + "\x55\x5d\xd5\x26\x3f\x89\xdf\xca\x8b\x4e\x1e\xb6" + "\x88\x78\x63\x5c\xa2\x63\x98\x4e\x6f\x25\x59\xb1" + "\x5f\x2b\x23\xb0\x4b\xa5\x18\x5d\xc2\x15\x74\x40" + "\x59\x4c\xb4\x1e\xcf\x9a\x36\xfd\x43\xe2\x03\xb8" + "\x59\x91\x30\x89\x2a\xc8\x5a\x43\x23\x7c\x73\x72" + "\xda\x3f\xad\x2b\xba\x00\x6b\xd1", + .expectedlen = 128, + .addtla = (unsigned char *) + "\x18\xe8\x17\xff\xef\x39\xc7\x41\x5c\x73\x03\x03" + "\xf6\x3d\xe8\x5f\xc8\xab\xe4\xab\x0f\xad\xe8\xd6" + "\x86\x88\x55\x28\xc1\x69\xdd\x76", + .addtlb = (unsigned char *) + "\xac\x07\xfc\xbe\x87\x0e\xd3\xea\x1f\x7e\xb8\xe7" + "\x9d\xec\xe8\xe7\xbc\xf3\x18\x25\x77\x35\x4a\xaa" + "\x00\x99\x2a\xdd\x0a\x00\x50\x82", + .addtllen = 32, + .pers = (unsigned char *) + "\xbc\x55\xab\x3c\xf6\x52\xb0\x11\x3d\x7b\x90\xb8" + "\x24\xc9\x26\x4e\x5a\x1e\x77\x0d\x3d\x58\x4a\xda" + "\xd1\x81\xe9\xf8\xeb\x30\x8f\x6f", + .perslen = 32, + }, +}; + +static const struct drbg_testvec drbg_pr_ctr_aes128_tv_template[] = { + { + .entropy = (unsigned char *) + "\xd1\x44\xc6\x61\x81\x6d\xca\x9d\x15\x28\x8a\x42" + "\x94\xd7\x28\x9c\x43\x77\x19\x29\x1a\x6d\xc3\xa2", + .entropylen = 24, + .entpra = (unsigned char *) + "\x96\xd8\x9e\x45\x32\xc9\xd2\x08\x7a\x6d\x97\x15" + "\xb4\xec\x80\xb1", + .entprb = (unsigned char *) + "\x8b\xb6\x72\xb5\x24\x0b\x98\x65\x95\x95\xe9\xc9" + "\x28\x07\xeb\xc2", + .entprlen = 16, + .expected = (unsigned char *) + "\x70\x19\xd0\x4c\x45\x78\xd6\x68\xa9\x9a\xaa\xfe" + "\xc1\xdf\x27\x9a\x1c\x0d\x0d\xf7\x24\x75\x46\xcc" + "\x77\x6b\xdf\x89\xc6\x94\xdc\x74\x50\x10\x70\x18" + "\x9b\xdc\x96\xb4\x89\x23\x40\x1a\xce\x09\x87\xce" + "\xd2\xf3\xd5\xe4\x51\x67\x74\x11\x5a\xcc\x8b\x3b" + "\x8a\xf1\x23\xa8", + .expectedlen = 64, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\x8e\x83\xe0\xeb\x37\xea\x3e\x53\x5e\x17\x6e\x77" + "\xbd\xb1\x53\x90\xfc\xdc\xc1\x3c\x9a\x88\x22\x94", + .entropylen = 24, + .entpra = (unsigned char *) + "\x6a\x85\xe7\x37\xc8\xf1\x04\x31\x98\x4f\xc8\x73" + "\x67\xd1\x08\xf8", + .entprb = (unsigned char *) + "\xd7\xa4\x68\xe2\x12\x74\xc3\xd9\xf1\xb7\x05\xbc" + "\xd4\xba\x04\x58", + .entprlen = 16, + .expected = (unsigned char *) + "\x78\xd6\xa6\x70\xff\xd1\x82\xf5\xa2\x88\x7f\x6d" + "\x3d\x8c\x39\xb1\xa8\xcb\x2c\x91\xab\x14\x7e\xbc" + "\x95\x45\x9f\x24\xb8\x20\xac\x21\x23\xdb\x72\xd7" + "\x12\x8d\x48\x95\xf3\x19\x0c\x43\xc6\x19\x45\xfc" + "\x8b\xac\x40\x29\x73\x00\x03\x45\x5e\x12\xff\x0c" + "\xc1\x02\x41\x82", + .expectedlen = 64, + .addtla = (unsigned char *) + "\xa2\xd9\x38\xcf\x8b\x29\x67\x5b\x65\x62\x6f\xe8" + "\xeb\xb3\x01\x76", + .addtlb = (unsigned char *) + "\x59\x63\x1e\x81\x8a\x14\xa8\xbb\xa1\xb8\x41\x25" + "\xd0\x7f\xcc\x43", + .addtllen = 16, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\x04\xd9\x49\xa6\xdc\xe8\x6e\xbb\xf1\x08\x77\x2b" + "\x9e\x08\xca\x92\x65\x16\xda\x99\xa2\x59\xf3\xe8", + .entropylen = 24, + .entpra = (unsigned char *) + "\x38\x7e\x3f\x6b\x51\x70\x7b\x20\xec\x53\xd0\x66" + "\xc3\x0f\xe3\xb0", + .entprb = (unsigned char *) + "\xe0\x86\xa6\xaa\x5f\x72\x2f\xad\xf7\xef\x06\xb8" + "\xd6\x9c\x9d\xe8", + .entprlen = 16, + .expected = (unsigned char *) + "\xc9\x0a\xaf\x85\x89\x71\x44\x66\x4f\x25\x0b\x2b" + "\xde\xd8\xfa\xff\x52\x5a\x1b\x32\x5e\x41\x7a\x10" + "\x1f\xef\x1e\x62\x23\xe9\x20\x30\xc9\x0d\xad\x69" + "\xb4\x9c\x5b\xf4\x87\x42\xd5\xae\x5e\x5e\x43\xcc" + "\xd9\xfd\x0b\x93\x4a\xe3\xd4\x06\x37\x36\x0f\x3f" + "\x72\x82\x0c\xcf", + .expectedlen = 64, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = (unsigned char *) + "\xbf\xa4\x9a\x8f\x7b\xd8\xb1\x7a\x9d\xfa\x45\xed" + "\x21\x52\xb3\xad", + .perslen = 16, + }, { + .entropy = (unsigned char *) + "\x92\x89\x8f\x31\xfa\x1c\xff\x6d\x18\x2f\x26\x06" + "\x43\xdf\xf8\x18\xc2\xa4\xd9\x72\xc3\xb9\xb6\x97", + .entropylen = 24, + .entpra = (unsigned char *) + "\x20\x72\x8a\x06\xf8\x6f\x8d\xd4\x41\xe2\x72\xb7" + "\xc4\x2c\xe8\x10", + .entprb = (unsigned char *) + "\x3d\xb0\xf0\x94\xf3\x05\x50\x33\x17\x86\x3e\x22" + "\x08\xf7\xa5\x01", + .entprlen = 16, + .expected = (unsigned char *) + "\x5a\x35\x39\x87\x0f\x4d\x22\xa4\x09\x24\xee\x71" + "\xc9\x6f\xac\x72\x0a\xd6\xf0\x88\x82\xd0\x83\x28" + "\x73\xec\x3f\x93\xd8\xab\x45\x23\xf0\x7e\xac\x45" + "\x14\x5e\x93\x9f\xb1\xd6\x76\x43\x3d\xb6\xe8\x08" + "\x88\xf6\xda\x89\x08\x77\x42\xfe\x1a\xf4\x3f\xc4" + "\x23\xc5\x1f\x68", + .expectedlen = 64, + .addtla = (unsigned char *) + "\x1a\x40\xfa\xe3\xcc\x6c\x7c\xa0\xf8\xda\xba\x59" + "\x23\x6d\xad\x1d", + .addtlb = (unsigned char *) + "\x9f\x72\x76\x6c\xc7\x46\xe5\xed\x2e\x53\x20\x12" + "\xbc\x59\x31\x8c", + .addtllen = 16, + .pers = (unsigned char *) + "\xea\x65\xee\x60\x26\x4e\x7e\xb6\x0e\x82\x68\xc4" + "\x37\x3c\x5c\x0b", + .perslen = 16, + }, +}; + +/* + * SP800-90A DRBG Test vectors from + * http://csrc.nist.gov/groups/STM/cavp/documents/drbg/drbgtestvectors.zip + * + * Test vectors for DRBG without prediction resistance. All types of DRBGs + * (Hash, HMAC, CTR) are tested with all permutations of use cases (w/ and + * w/o personalization string, w/ and w/o additional input string). + */ +static const struct drbg_testvec drbg_nopr_sha256_tv_template[] = { + { + .entropy = (unsigned char *) + "\xa6\x5a\xd0\xf3\x45\xdb\x4e\x0e\xff\xe8\x75\xc3" + "\xa2\xe7\x1f\x42\xc7\x12\x9d\x62\x0f\xf5\xc1\x19" + "\xa9\xef\x55\xf0\x51\x85\xe0\xfb\x85\x81\xf9\x31" + "\x75\x17\x27\x6e\x06\xe9\x60\x7d\xdb\xcb\xcc\x2e", + .entropylen = 48, + .expected = (unsigned char *) + "\xd3\xe1\x60\xc3\x5b\x99\xf3\x40\xb2\x62\x82\x64" + "\xd1\x75\x10\x60\xe0\x04\x5d\xa3\x83\xff\x57\xa5" + "\x7d\x73\xa6\x73\xd2\xb8\xd8\x0d\xaa\xf6\xa6\xc3" + "\x5a\x91\xbb\x45\x79\xd7\x3f\xd0\xc8\xfe\xd1\x11" + "\xb0\x39\x13\x06\x82\x8a\xdf\xed\x52\x8f\x01\x81" + "\x21\xb3\xfe\xbd\xc3\x43\xe7\x97\xb8\x7d\xbb\x63" + "\xdb\x13\x33\xde\xd9\xd1\xec\xe1\x77\xcf\xa6\xb7" + "\x1f\xe8\xab\x1d\xa4\x66\x24\xed\x64\x15\xe5\x1c" + "\xcd\xe2\xc7\xca\x86\xe2\x83\x99\x0e\xea\xeb\x91" + "\x12\x04\x15\x52\x8b\x22\x95\x91\x02\x81\xb0\x2d" + "\xd4\x31\xf4\xc9\xf7\x04\x27\xdf", + .expectedlen = 128, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\x73\xd3\xfb\xa3\x94\x5f\x2b\x5f\xb9\x8f\xf6\x9c" + "\x8a\x93\x17\xae\x19\xc3\x4c\xc3\xd6\xca\xa3\x2d" + "\x16\xfc\x42\xd2\x2d\xd5\x6f\x56\xcc\x1d\x30\xff" + "\x9e\x06\x3e\x09\xce\x58\xe6\x9a\x35\xb3\xa6\x56", + .entropylen = 48, + .expected = (unsigned char *) + "\x71\x7b\x93\x46\x1a\x40\xaa\x35\xa4\xaa\xc5\xe7" + "\x6d\x5b\x5b\x8a\xa0\xdf\x39\x7d\xae\x71\x58\x5b" + "\x3c\x7c\xb4\xf0\x89\xfa\x4a\x8c\xa9\x5c\x54\xc0" + "\x40\xdf\xbc\xce\x26\x81\x34\xf8\xba\x7d\x1c\xe8" + "\xad\x21\xe0\x74\xcf\x48\x84\x30\x1f\xa1\xd5\x4f" + "\x81\x42\x2f\xf4\xdb\x0b\x23\xf8\x73\x27\xb8\x1d" + "\x42\xf8\x44\x58\xd8\x5b\x29\x27\x0a\xf8\x69\x59" + "\xb5\x78\x44\xeb\x9e\xe0\x68\x6f\x42\x9a\xb0\x5b" + "\xe0\x4e\xcb\x6a\xaa\xe2\xd2\xd5\x33\x25\x3e\xe0" + "\x6c\xc7\x6a\x07\xa5\x03\x83\x9f\xe2\x8b\xd1\x1c" + "\x70\xa8\x07\x59\x97\xeb\xf6\xbe", + .expectedlen = 128, + .addtla = (unsigned char *) + "\xf4\xd5\x98\x3d\xa8\xfc\xfa\x37\xb7\x54\x67\x73" + "\xc7\xc3\xdd\x47\x34\x71\x02\x5d\xc1\xa0\xd3\x10" + "\xc1\x8b\xbd\xf5\x66\x34\x6f\xdd", + .addtlb = (unsigned char *) + "\xf7\x9e\x6a\x56\x0e\x73\xe9\xd9\x7a\xd1\x69\xe0" + "\x6f\x8c\x55\x1c\x44\xd1\xce\x6f\x28\xcc\xa4\x4d" + "\xa8\xc0\x85\xd1\x5a\x0c\x59\x40", + .addtllen = 32, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\x2a\x85\xa9\x8b\xd0\xda\x83\xd6\xad\xab\x9f\xbb" + "\x54\x31\x15\x95\x1c\x4d\x49\x9f\x6a\x15\xf6\xe4" + "\x15\x50\x88\x06\x29\x0d\xed\x8d\xb9\x6f\x96\xe1" + "\x83\x9f\xf7\x88\xda\x84\xbf\x44\x28\xd9\x1d\xaa", + .entropylen = 48, + .expected = (unsigned char *) + "\x2d\x55\xde\xc9\xed\x05\x47\x07\x3d\x04\xfc\x28" + "\x0f\x92\xf0\x4d\xd8\x00\x32\x47\x0a\x1b\x1c\x4b" + "\xef\xd9\x97\xa1\x17\x67\xda\x26\x6c\xfe\x76\x46" + "\x6f\xbc\x6d\x82\x4e\x83\x8a\x98\x66\x6c\x01\xb6" + "\xe6\x64\xe0\x08\x10\x6f\xd3\x5d\x90\xe7\x0d\x72" + "\xa6\xa7\xe3\xbb\x98\x11\x12\x56\x23\xc2\x6d\xd1" + "\xc8\xa8\x7a\x39\xf3\x34\xe3\xb8\xf8\x66\x00\x77" + "\x7d\xcf\x3c\x3e\xfa\xc9\x0f\xaf\xe0\x24\xfa\xe9" + "\x84\xf9\x6a\x01\xf6\x35\xdb\x5c\xab\x2a\xef\x4e" + "\xac\xab\x55\xb8\x9b\xef\x98\x68\xaf\x51\xd8\x16" + "\xa5\x5e\xae\xf9\x1e\xd2\xdb\xe6", + .expectedlen = 128, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = (unsigned char *) + "\xa8\x80\xec\x98\x30\x98\x15\xd2\xc6\xc4\x68\xf1" + "\x3a\x1c\xbf\xce\x6a\x40\x14\xeb\x36\x99\x53\xda" + "\x57\x6b\xce\xa4\x1c\x66\x3d\xbc", + .perslen = 32, + }, { + .entropy = (unsigned char *) + "\x69\xed\x82\xa9\xc5\x7b\xbf\xe5\x1d\x2f\xcb\x7a" + "\xd3\x50\x7d\x96\xb4\xb9\x2b\x50\x77\x51\x27\x74" + "\x33\x74\xba\xf1\x30\xdf\x8e\xdf\x87\x1d\x87\xbc" + "\x96\xb2\xc3\xa7\xed\x60\x5e\x61\x4e\x51\x29\x1a", + .entropylen = 48, + .expected = (unsigned char *) + "\xa5\x71\x24\x31\x11\xfe\x13\xe1\xa8\x24\x12\xfb" + "\x37\xa1\x27\xa5\xab\x77\xa1\x9f\xae\x8f\xaf\x13" + "\x93\xf7\x53\x85\x91\xb6\x1b\xab\xd4\x6b\xea\xb6" + "\xef\xda\x4c\x90\x6e\xef\x5f\xde\xe1\xc7\x10\x36" + "\xd5\x67\xbd\x14\xb6\x89\x21\x0c\xc9\x92\x65\x64" + "\xd0\xf3\x23\xe0\x7f\xd1\xe8\x75\xc2\x85\x06\xea" + "\xca\xc0\xcb\x79\x2d\x29\x82\xfc\xaa\x9a\xc6\x95" + "\x7e\xdc\x88\x65\xba\xec\x0e\x16\x87\xec\xa3\x9e" + "\xd8\x8c\x80\xab\x3a\x64\xe0\xcb\x0e\x45\x98\xdd" + "\x7c\x6c\x6c\x26\x11\x13\xc8\xce\xa9\x47\xa6\x06" + "\x57\xa2\x66\xbb\x2d\x7f\xf3\xc1", + .expectedlen = 128, + .addtla = (unsigned char *) + "\x74\xd3\x6d\xda\xe8\xd6\x86\x5f\x63\x01\xfd\xf2" + "\x7d\x06\x29\x6d\x94\xd1\x66\xf0\xd2\x72\x67\x4e" + "\x77\xc5\x3d\x9e\x03\xe3\xa5\x78", + .addtlb = (unsigned char *) + "\xf6\xb6\x3d\xf0\x7c\x26\x04\xc5\x8b\xcd\x3e\x6a" + "\x9f\x9c\x3a\x2e\xdb\x47\x87\xe5\x8e\x00\x5e\x2b" + "\x74\x7f\xa6\xf6\x80\xcd\x9b\x21", + .addtllen = 32, + .pers = (unsigned char *) + "\x74\xa6\xe0\x08\xf9\x27\xee\x1d\x6e\x3c\x28\x20" + "\x87\xdd\xd7\x54\x31\x47\x78\x4b\xe5\x6d\xa3\x73" + "\xa9\x65\xb1\x10\xc1\xdc\x77\x7c", + .perslen = 32, + }, +}; + +static const struct drbg_testvec drbg_nopr_hmac_sha256_tv_template[] = { + { + .entropy = (unsigned char *) + "\xca\x85\x19\x11\x34\x93\x84\xbf\xfe\x89\xde\x1c" + "\xbd\xc4\x6e\x68\x31\xe4\x4d\x34\xa4\xfb\x93\x5e" + "\xe2\x85\xdd\x14\xb7\x1a\x74\x88\x65\x9b\xa9\x6c" + "\x60\x1d\xc6\x9f\xc9\x02\x94\x08\x05\xec\x0c\xa8", + .entropylen = 48, + .expected = (unsigned char *) + "\xe5\x28\xe9\xab\xf2\xde\xce\x54\xd4\x7c\x7e\x75" + "\xe5\xfe\x30\x21\x49\xf8\x17\xea\x9f\xb4\xbe\xe6" + "\xf4\x19\x96\x97\xd0\x4d\x5b\x89\xd5\x4f\xbb\x97" + "\x8a\x15\xb5\xc4\x43\xc9\xec\x21\x03\x6d\x24\x60" + "\xb6\xf7\x3e\xba\xd0\xdc\x2a\xba\x6e\x62\x4a\xbf" + "\x07\x74\x5b\xc1\x07\x69\x4b\xb7\x54\x7b\xb0\x99" + "\x5f\x70\xde\x25\xd6\xb2\x9e\x2d\x30\x11\xbb\x19" + "\xd2\x76\x76\xc0\x71\x62\xc8\xb5\xcc\xde\x06\x68" + "\x96\x1d\xf8\x68\x03\x48\x2c\xb3\x7e\xd6\xd5\xc0" + "\xbb\x8d\x50\xcf\x1f\x50\xd4\x76\xaa\x04\x58\xbd" + "\xab\xa8\x06\xf4\x8b\xe9\xdc\xb8", + .expectedlen = 128, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\xf9\x7a\x3c\xfd\x91\xfa\xa0\x46\xb9\xe6\x1b\x94" + "\x93\xd4\x36\xc4\x93\x1f\x60\x4b\x22\xf1\x08\x15" + "\x21\xb3\x41\x91\x51\xe8\xff\x06\x11\xf3\xa7\xd4" + "\x35\x95\x35\x7d\x58\x12\x0b\xd1\xe2\xdd\x8a\xed", + .entropylen = 48, + .expected = (unsigned char *) + "\xc6\x87\x1c\xff\x08\x24\xfe\x55\xea\x76\x89\xa5" + "\x22\x29\x88\x67\x30\x45\x0e\x5d\x36\x2d\xa5\xbf" + "\x59\x0d\xcf\x9a\xcd\x67\xfe\xd4\xcb\x32\x10\x7d" + "\xf5\xd0\x39\x69\xa6\x6b\x1f\x64\x94\xfd\xf5\xd6" + "\x3d\x5b\x4d\x0d\x34\xea\x73\x99\xa0\x7d\x01\x16" + "\x12\x6d\x0d\x51\x8c\x7c\x55\xba\x46\xe1\x2f\x62" + "\xef\xc8\xfe\x28\xa5\x1c\x9d\x42\x8e\x6d\x37\x1d" + "\x73\x97\xab\x31\x9f\xc7\x3d\xed\x47\x22\xe5\xb4" + "\xf3\x00\x04\x03\x2a\x61\x28\xdf\x5e\x74\x97\xec" + "\xf8\x2c\xa7\xb0\xa5\x0e\x86\x7e\xf6\x72\x8a\x4f" + "\x50\x9a\x8c\x85\x90\x87\x03\x9c", + .expectedlen = 128, + .addtla = (unsigned char *) + "\x51\x72\x89\xaf\xe4\x44\xa0\xfe\x5e\xd1\xa4\x1d" + "\xbb\xb5\xeb\x17\x15\x00\x79\xbd\xd3\x1e\x29\xcf" + "\x2f\xf3\x00\x34\xd8\x26\x8e\x3b", + .addtlb = (unsigned char *) + "\x88\x02\x8d\x29\xef\x80\xb4\xe6\xf0\xfe\x12\xf9" + "\x1d\x74\x49\xfe\x75\x06\x26\x82\xe8\x9c\x57\x14" + "\x40\xc0\xc9\xb5\x2c\x42\xa6\xe0", + .addtllen = 32, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\x8d\xf0\x13\xb4\xd1\x03\x52\x30\x73\x91\x7d\xdf" + "\x6a\x86\x97\x93\x05\x9e\x99\x43\xfc\x86\x54\x54" + "\x9e\x7a\xb2\x2f\x7c\x29\xf1\x22\xda\x26\x25\xaf" + "\x2d\xdd\x4a\xbc\xce\x3c\xf4\xfa\x46\x59\xd8\x4e", + .entropylen = 48, + .expected = (unsigned char *) + "\xb9\x1c\xba\x4c\xc8\x4f\xa2\x5d\xf8\x61\x0b\x81" + "\xb6\x41\x40\x27\x68\xa2\x09\x72\x34\x93\x2e\x37" + "\xd5\x90\xb1\x15\x4c\xbd\x23\xf9\x74\x52\xe3\x10" + "\xe2\x91\xc4\x51\x46\x14\x7f\x0d\xa2\xd8\x17\x61" + "\xfe\x90\xfb\xa6\x4f\x94\x41\x9c\x0f\x66\x2b\x28" + "\xc1\xed\x94\xda\x48\x7b\xb7\xe7\x3e\xec\x79\x8f" + "\xbc\xf9\x81\xb7\x91\xd1\xbe\x4f\x17\x7a\x89\x07" + "\xaa\x3c\x40\x16\x43\xa5\xb6\x2b\x87\xb8\x9d\x66" + "\xb3\xa6\x0e\x40\xd4\xa8\xe4\xe9\xd8\x2a\xf6\xd2" + "\x70\x0e\x6f\x53\x5c\xdb\x51\xf7\x5c\x32\x17\x29" + "\x10\x37\x41\x03\x0c\xcc\x3a\x56", + .expectedlen = 128, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = (unsigned char *) + "\xb5\x71\xe6\x6d\x7c\x33\x8b\xc0\x7b\x76\xad\x37" + "\x57\xbb\x2f\x94\x52\xbf\x7e\x07\x43\x7a\xe8\x58" + "\x1c\xe7\xbc\x7c\x3a\xc6\x51\xa9", + .perslen = 32, + }, { + .entropy = (unsigned char *) + "\xc2\xa5\x66\xa9\xa1\x81\x7b\x15\xc5\xc3\xb7\x78" + "\x17\x7a\xc8\x7c\x24\xe7\x97\xbe\x0a\x84\x5f\x11" + "\xc2\xfe\x39\x9d\xd3\x77\x32\xf2\xcb\x18\x94\xeb" + "\x2b\x97\xb3\xc5\x6e\x62\x83\x29\x51\x6f\x86\xec", + .entropylen = 48, + .expected = (unsigned char *) + "\xb3\xa3\x69\x8d\x77\x76\x99\xa0\xdd\x9f\xa3\xf0" + "\xa9\xfa\x57\x83\x2d\x3c\xef\xac\x5d\xf2\x44\x37" + "\xc6\xd7\x3a\x0f\xe4\x10\x40\xf1\x72\x90\x38\xae" + "\xf1\xe9\x26\x35\x2e\xa5\x9d\xe1\x20\xbf\xb7\xb0" + "\x73\x18\x3a\x34\x10\x6e\xfe\xd6\x27\x8f\xf8\xad" + "\x84\x4b\xa0\x44\x81\x15\xdf\xdd\xf3\x31\x9a\x82" + "\xde\x6b\xb1\x1d\x80\xbd\x87\x1a\x9a\xcd\x35\xc7" + "\x36\x45\xe1\x27\x0f\xb9\xfe\x4f\xa8\x8e\xc0\xe4" + "\x65\x40\x9e\xa0\xcb\xa8\x09\xfe\x2f\x45\xe0\x49" + "\x43\xa2\xe3\x96\xbb\xb7\xdd\x2f\x4e\x07\x95\x30" + "\x35\x24\xcc\x9c\xc5\xea\x54\xa1", + .expectedlen = 128, + .addtla = (unsigned char *) + "\x41\x3d\xd8\x3f\xe5\x68\x35\xab\xd4\x78\xcb\x96" + "\x93\xd6\x76\x35\x90\x1c\x40\x23\x9a\x26\x64\x62" + "\xd3\x13\x3b\x83\xe4\x9c\x82\x0b", + .addtlb = (unsigned char *) + "\xd5\xc4\xa7\x1f\x9d\x6d\x95\xa1\xbe\xdf\x0b\xd2" + "\x24\x7c\x27\x7d\x1f\x84\xa4\xe5\x7a\x4a\x88\x25" + "\xb8\x2a\x2d\x09\x7d\xe6\x3e\xf1", + .addtllen = 32, + .pers = (unsigned char *) + "\x13\xce\x4d\x8d\xd2\xdb\x97\x96\xf9\x41\x56\xc8" + "\xe8\xf0\x76\x9b\x0a\xa1\xc8\x2c\x13\x23\xb6\x15" + "\x36\x60\x3b\xca\x37\xc9\xee\x29", + .perslen = 32, + }, +}; + +static const struct drbg_testvec drbg_nopr_ctr_aes192_tv_template[] = { + { + .entropy = (unsigned char *) + "\xc3\x5c\x2f\xa2\xa8\x9d\x52\xa1\x1f\xa3\x2a\xa9" + "\x6c\x95\xb8\xf1\xc9\xa8\xf9\xcb\x24\x5a\x8b\x40" + "\xf3\xa6\xe5\xa7\xfb\xd9\xd3\xc6\x8e\x27\x7b\xa9" + "\xac\x9b\xbb\x00", + .entropylen = 40, + .expected = (unsigned char *) + "\x8c\x2e\x72\xab\xfd\x9b\xb8\x28\x4d\xb7\x9e\x17" + "\xa4\x3a\x31\x46\xcd\x76\x94\xe3\x52\x49\xfc\x33" + "\x83\x91\x4a\x71\x17\xf4\x13\x68\xe6\xd4\xf1\x48" + "\xff\x49\xbf\x29\x07\x6b\x50\x15\xc5\x9f\x45\x79" + "\x45\x66\x2e\x3d\x35\x03\x84\x3f\x4a\xa5\xa3\xdf" + "\x9a\x9d\xf1\x0d", + .expectedlen = 64, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = NULL, + .perslen = 0, }, }; -static const struct cipher_testvec camellia_ctr_enc_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" - "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" - "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" - "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" - "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" - "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" - "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" - "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" - "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" - "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" - "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" - "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" - "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" - "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" - "\x29\xC0\x57\xEE\x62\xF9\x90\x04" - "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" - "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" - "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" - "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" - "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" - "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" - "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" - "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" - "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" - "\x57\xEE\x85\x1C\x90\x27\xBE\x32" - "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" - "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" - "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" - "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" - "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" - "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" - "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" - "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" - "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" - "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" - "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" - "\x69\x00\x74\x0B\xA2\x16\xAD\x44" - "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" - "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" - "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" - "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" - "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" - "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" - "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" - "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" - "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" - "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" - "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" - "\x58\xEF\x86\x1D\x91\x28\xBF\x33" - "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" - "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" - "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" - "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" - "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" - "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" - "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" - "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" - "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" - "\x86\x1D\xB4\x28\xBF\x56\xED\x61" - "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" - "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .ilen = 496, - .result = "\xF3\x06\x3A\x84\xCD\xBA\x8E\x11" - "\xB7\x74\x6F\x5C\x97\xFB\x36\xFE" - "\xDE\x71\x58\xD4\x15\xD1\xC1\xA4" - "\xC9\x28\x74\xA6\x6B\xC7\x95\xA6" - "\x6C\x77\xF7\x2F\xDF\xC7\xBB\x85" - "\x60\xFC\xE8\x94\xE8\xB5\x09\x2C" - "\x1E\x43\xEF\x6C\xE9\x98\xC5\xA0" - "\x7B\x13\xE5\x7F\xF8\x49\x9A\x8C" - "\xE6\x7B\x08\xC3\x32\x66\x55\x4E" - "\xA5\x44\x1D\x2C\x18\xC7\x29\x1F" - "\x61\x28\x4A\xE3\xCD\xE5\x47\xB2" - "\x82\x2F\x66\x83\x91\x51\xAE\xD7" - "\x1C\x91\x3C\x57\xE3\x1D\x5A\xC9" - "\xFD\xC5\x58\x58\xEF\xCC\x33\xC9" - "\x0F\xEA\x26\x32\xD1\x15\x19\x2D" - "\x25\xB4\x7F\xB0\xDF\xFB\x88\x60" - "\x4E\x4D\x06\x7D\xCC\x1F\xED\x3B" - "\x68\x84\xD5\xB3\x1B\xE7\xB9\xA1" - "\x68\x8B\x2C\x1A\x44\xDA\x63\xD3" - "\x29\xE9\x59\x32\x1F\x30\x1C\x43" - "\xEA\x3A\xA3\x6B\x54\x3C\xAA\x11" - "\xAD\x38\x20\xC9\xB9\x8A\x64\x66" - "\x5A\x07\x49\xDF\xA1\x9C\xF9\x76" - "\x36\x65\xB6\x81\x8F\x76\x09\xE5" - "\xEB\xD1\x29\xA4\xE4\xF4\x4C\xCD" - "\xAF\xFC\xB9\x16\xD9\xC3\x73\x6A" - "\x33\x12\xF8\x7E\xBC\xCC\x7D\x80" - "\xBF\x3C\x25\x06\x13\x84\xFA\x35" - "\xF7\x40\xFA\xA1\x44\x13\x70\xD8" - "\x01\xF9\x85\x15\x63\xEC\x7D\xB9" - "\x02\xD8\xBA\x41\x6C\x92\x68\x66" - "\x95\xDD\xD6\x42\xE7\xBB\xE1\xFD" - "\x28\x3E\x94\xB6\xBD\xA7\xBF\x47" - "\x58\x8D\xFF\x19\x30\x75\x0D\x48" - "\x94\xE9\xA6\xCD\xB3\x8E\x1E\xCD" - "\x59\xBC\x1A\xAC\x3C\x4F\xA9\xEB" - "\xF4\xA7\xE4\x75\x4A\x18\x40\xC9" - "\x1E\xEC\x06\x9C\x28\x4B\xF7\x2B" - "\xE2\xEF\xD6\x42\x2E\xBB\xFC\x0A" - "\x79\xA2\x99\x28\x93\x1B\x00\x57" - "\x35\x1E\x1A\x93\x90\xA4\x68\x95" - "\x5E\x57\x40\xD5\xA9\xAA\x19\x48" - "\xEC\xFF\x76\x77\xDC\x78\x89\x76" - "\xE5\x3B\x00\xEC\x58\x4D\xD1\xE3" - "\xC8\x6C\x2C\x45\x5E\x5F\xD9\x4E" - "\x71\xA5\x36\x6D\x03\xF1\xC7\xD5" - "\xF3\x63\xC0\xD8\xCB\x2B\xF1\xA8" - "\xB9\x2B\xE6\x0B\xB9\x65\x78\xA0" - "\xC4\x46\xE6\x9B\x8B\x43\x2D\xAB" - "\x70\xA6\xE0\x59\x1E\xAC\x9D\xE0" - "\x76\x44\x45\xF3\x24\x11\x57\x98" - "\x9A\x86\xB4\x12\x80\x28\x86\x20" - "\x23\x9D\x2D\xE9\x38\x32\xB1\xE1" - "\xCF\x0A\x23\x73\x7D\xC5\x80\x3D" - "\x9F\x6D\xA0\xD0\xEE\x93\x8A\x79" - "\x3A\xDD\x1D\xBB\x9E\x26\x5D\x01" - "\x44\xD0\xD4\x4E\xC3\xF1\xE4\x38" - "\x09\x62\x0A\x1A\x4E\xD2\x63\x0F" - "\x6E\x3E\xD2\xA4\x3A\xF4\xF3\xFF" - "\x7E\x42\xEC\xB6\x6F\x4D\x6B\x48" - "\xE6\xA6\x50\x80\x78\x9E\xF1\xB0" - "\x4D\xB2\x0D\x3D\xFC\x40\x25\x4D", - .rlen = 496, - }, { /* Generated with Crypto++ */ +static const struct drbg_testvec drbg_nopr_ctr_aes256_tv_template[] = { + { + .entropy = (unsigned char *) + "\x36\x40\x19\x40\xfa\x8b\x1f\xba\x91\xa1\x66\x1f" + "\x21\x1d\x78\xa0\xb9\x38\x9a\x74\xe5\xbc\xcf\xec" + "\xe8\xd7\x66\xaf\x1a\x6d\x3b\x14\x49\x6f\x25\xb0" + "\xf1\x30\x1b\x4f\x50\x1b\xe3\x03\x80\xa1\x37\xeb", + .entropylen = 48, + .expected = (unsigned char *) + "\x58\x62\xeb\x38\xbd\x55\x8d\xd9\x78\xa6\x96\xe6" + "\xdf\x16\x47\x82\xdd\xd8\x87\xe7\xe9\xa6\xc9\xf3" + "\xf1\xfb\xaf\xb7\x89\x41\xb5\x35\xa6\x49\x12\xdf" + "\xd2\x24\xc6\xdc\x74\x54\xe5\x25\x0b\x3d\x97\x16" + "\x5e\x16\x26\x0c\x2f\xaf\x1c\xc7\x73\x5c\xb7\x5f" + "\xb4\xf0\x7e\x1d", + .expectedlen = 64, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = NULL, + .perslen = 0, + }, +}; + +static const struct drbg_testvec drbg_nopr_ctr_aes128_tv_template[] = { + { + .entropy = (unsigned char *) + "\x87\xe1\xc5\x32\x99\x7f\x57\xa3\x5c\x28\x6d\xe8" + "\x64\xbf\xf2\x64\xa3\x9e\x98\xdb\x6c\x10\x78\x7f", + .entropylen = 24, + .expected = (unsigned char *) + "\x2c\x14\x7e\x24\x11\x9a\xd8\xd4\xb2\xed\x61\xc1" + "\x53\xd0\x50\xc9\x24\xff\x59\x75\x15\xf1\x17\x3a" + "\x3d\xf4\x4b\x2c\x84\x28\xef\x89\x0e\xb9\xde\xf3" + "\xe4\x78\x04\xb2\xfd\x9b\x35\x7f\xe1\x3f\x8a\x3e" + "\x10\xc8\x67\x0a\xf9\xdf\x2d\x6c\x96\xfb\xb2\xb8" + "\xcb\x2d\xd6\xb0", + .expectedlen = 64, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\x71\xbd\xce\x35\x42\x7d\x20\xbf\x58\xcf\x17\x74" + "\xce\x72\xd8\x33\x34\x50\x2d\x8f\x5b\x14\xc4\xdd", + .entropylen = 24, + .expected = (unsigned char *) + "\x97\x33\xe8\x20\x12\xe2\x7b\xa1\x46\x8f\xf2\x34" + "\xb3\xc9\xb6\x6b\x20\xb2\x4f\xee\x27\xd8\x0b\x21" + "\x8c\xff\x63\x73\x69\x29\xfb\xf3\x85\xcd\x88\x8e" + "\x43\x2c\x71\x8b\xa2\x55\xd2\x0f\x1d\x7f\xe3\xe1" + "\x2a\xa3\xe9\x2c\x25\x89\xc7\x14\x52\x99\x56\xcc" + "\xc3\xdf\xb3\x81", + .expectedlen = 64, + .addtla = (unsigned char *) + "\x66\xef\x42\xd6\x9a\x8c\x3d\x6d\x4a\x9e\x95\xa6" + "\x91\x4d\x81\x56", + .addtlb = (unsigned char *) + "\xe3\x18\x83\xd9\x4b\x5e\xc4\xcc\xaa\x61\x2f\xbb" + "\x4a\x55\xd1\xc6", + .addtllen = 16, + .pers = NULL, + .perslen = 0, + }, { + .entropy = (unsigned char *) + "\xca\x4b\x1e\xfa\x75\xbd\x69\x36\x38\x73\xb8\xf9" + "\xdb\x4d\x35\x0e\x47\xbf\x6c\x37\x72\xfd\xf7\xa9", + .entropylen = 24, + .expected = (unsigned char *) + "\x59\xc3\x19\x79\x1b\xb1\xf3\x0e\xe9\x34\xae\x6e" + "\x8b\x1f\xad\x1f\x74\xca\x25\x45\x68\xb8\x7f\x75" + "\x12\xf8\xf2\xab\x4c\x23\x01\x03\x05\xe1\x70\xee" + "\x75\xd8\xcb\xeb\x23\x4c\x7a\x23\x6e\x12\x27\xdb" + "\x6f\x7a\xac\x3c\x44\xb7\x87\x4b\x65\x56\x74\x45" + "\x34\x30\x0c\x3d", + .expectedlen = 64, + .addtla = NULL, + .addtlb = NULL, + .addtllen = 0, + .pers = (unsigned char *) + "\xeb\xaa\x60\x2c\x4d\xbe\x33\xff\x1b\xef\xbf\x0a" + "\x0b\xc6\x97\x54", + .perslen = 16, + }, { + .entropy = (unsigned char *) + "\xc0\x70\x1f\x92\x50\x75\x8f\xcd\xf2\xbe\x73\x98" + "\x80\xdb\x66\xeb\x14\x68\xb4\xa5\x87\x9c\x2d\xa6", + .entropylen = 24, + .expected = (unsigned char *) + "\x97\xc0\xc0\xe5\xa0\xcc\xf2\x4f\x33\x63\x48\x8a" + "\xdb\x13\x0a\x35\x89\xbf\x80\x65\x62\xee\x13\x95" + "\x7c\x33\xd3\x7d\xf4\x07\x77\x7a\x2b\x65\x0b\x5f" + "\x45\x5c\x13\xf1\x90\x77\x7f\xc5\x04\x3f\xcc\x1a" + "\x38\xf8\xcd\x1b\xbb\xd5\x57\xd1\x4a\x4c\x2e\x8a" + "\x2b\x49\x1e\x5c", + .expectedlen = 64, + .addtla = (unsigned char *) + "\xf9\x01\xf8\x16\x7a\x1d\xff\xde\x8e\x3c\x83\xe2" + "\x44\x85\xe7\xfe", + .addtlb = (unsigned char *) + "\x17\x1c\x09\x38\xc2\x38\x9f\x97\x87\x60\x55\xb4" + "\x82\x16\x62\x7f", + .addtllen = 16, + .pers = (unsigned char *) + "\x80\x08\xae\xe8\xe9\x69\x40\xc5\x08\x73\xc7\x9f" + "\x8e\xcf\xe0\x02", + .perslen = 16, + }, +}; + +/* Cast5 test vectors from RFC 2144 */ +static const struct cipher_testvec cast5_tv_template[] = { + { + .key = "\x01\x23\x45\x67\x12\x34\x56\x78" + "\x23\x45\x67\x89\x34\x56\x78\x9a", + .klen = 16, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .ctext = "\x23\x8b\x4f\xe5\x84\x7e\x44\xb2", + .len = 8, + }, { + .key = "\x01\x23\x45\x67\x12\x34\x56\x78" + "\x23\x45", + .klen = 10, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .ctext = "\xeb\x6a\x71\x1a\x2c\x02\x27\x1b", + .len = 8, + }, { + .key = "\x01\x23\x45\x67\x12", + .klen = 5, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .ctext = "\x7a\xc8\x16\xd1\x6e\x9b\x30\x2e", + .len = 8, + }, { /* Generated from TF test vectors */ .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", + .klen = 16, + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -31461,213 +27061,83 @@ static const struct cipher_testvec camellia_ctr_enc_tv_template[] = { "\x86\x1D\xB4\x28\xBF\x56\xED\x61" "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59\xF0\x64\xFB\x92\x06" - "\x9D\x34\xCB\x3F\xD6\x6D\x04\x78" - "\x0F\xA6\x1A\xB1\x48\xDF\x53\xEA" - "\x81\x18\x8C\x23\xBA\x2E\xC5\x5C" - "\xF3\x67\xFE\x95\x09\xA0\x37\xCE" - "\x42\xD9\x70\x07\x7B\x12\xA9\x1D" - "\xB4\x4B\xE2\x56\xED\x84\x1B\x8F" - "\x26\xBD\x31\xC8\x5F\xF6\x6A\x01" - "\x98\x0C\xA3\x3A\xD1\x45\xDC\x73" - "\x0A\x7E\x15\xAC\x20\xB7\x4E\xE5" - "\x59\xF0\x87\x1E\x92\x29\xC0\x34" - "\xCB\x62\xF9\x6D\x04\x9B\x0F\xA6" - "\x3D\xD4\x48\xDF\x76\x0D\x81\x18" - "\xAF\x23\xBA\x51\xE8\x5C\xF3\x8A" - "\x21\x95\x2C\xC3\x37\xCE\x65\xFC" - "\x70\x07\x9E\x12\xA9\x40\xD7\x4B" - "\xE2\x79\x10\x84\x1B\xB2\x26\xBD" - "\x54\xEB\x5F\xF6\x8D\x01\x98\x2F" - "\xC6\x3A\xD1\x68\xFF\x73\x0A\xA1" - "\x15\xAC\x43\xDA\x4E\xE5\x7C\x13" - "\x87\x1E\xB5\x29\xC0\x57\xEE\x62" - "\xF9\x90\x04\x9B\x32\xC9\x3D\xD4" - "\x6B\x02\x76\x0D\xA4\x18\xAF\x46" - "\xDD\x51\xE8\x7F\x16\x8A\x21\xB8" - "\x2C\xC3\x5A\xF1\x65\xFC\x93\x07" - "\x9E\x35\xCC\x40\xD7\x6E\x05\x79" - "\x10\xA7\x1B\xB2\x49\xE0\x54\xEB" - "\x82\x19\x8D\x24\xBB\x2F\xC6\x5D" - "\xF4\x68\xFF\x96\x0A\xA1\x38\xCF" - "\x43\xDA\x71\x08\x7C\x13\xAA\x1E" - "\xB5\x4C\xE3\x57\xEE\x85\x1C\x90" - "\x27\xBE\x32\xC9\x60\xF7\x6B\x02" - "\x99\x0D\xA4\x3B\xD2\x46\xDD\x74" - "\x0B\x7F\x16\xAD\x21\xB8\x4F\xE6" - "\x5A\xF1\x88\x1F\x93\x2A\xC1\x35" - "\xCC\x63\xFA\x6E\x05\x9C\x10\xA7" - "\x3E\xD5\x49\xE0\x77\x0E\x82\x19" - "\xB0\x24\xBB\x52\xE9\x5D\xF4\x8B" - "\x22\x96\x2D\xC4\x38\xCF\x66\xFD" - "\x71\x08\x9F\x13\xAA\x41\xD8\x4C" - "\xE3\x7A\x11\x85\x1C\xB3\x27\xBE" - "\x55\xEC\x60\xF7\x8E\x02\x99\x30" - "\xC7\x3B\xD2\x69\x00\x74\x0B\xA2" - "\x16\xAD\x44\xDB\x4F\xE6\x7D\x14" - "\x88\x1F\xB6\x2A\xC1\x58\xEF\x63" - "\xFA\x91\x05\x9C\x33\xCA\x3E\xD5" - "\x6C\x03\x77\x0E\xA5\x19\xB0\x47" - "\xDE\x52\xE9\x80\x17\x8B\x22\xB9" - "\x2D\xC4\x5B\xF2\x66\xFD\x94\x08" - "\x9F\x36\xCD\x41\xD8\x6F\x06\x7A" - "\x11\xA8\x1C\xB3\x4A\xE1\x55\xEC" - "\x83\x1A\x8E\x25\xBC\x30\xC7\x5E" - "\xF5\x69\x00\x97\x0B\xA2\x39\xD0" - "\x44\xDB\x72\x09\x7D\x14\xAB\x1F" - "\xB6\x4D\xE4\x58\xEF\x86\x1D\x91" - "\x28\xBF\x33\xCA\x61\xF8\x6C\x03" - "\x9A\x0E\xA5\x3C\xD3\x47\xDE\x75" - "\x0C\x80\x17\xAE\x22\xB9\x50\xE7" - "\x5B\xF2\x89\x20\x94\x2B\xC2\x36" - "\xCD\x64\xFB\x6F\x06\x9D\x11\xA8" - "\x3F\xD6\x4A\xE1\x78\x0F\x83\x1A" - "\xB1\x25\xBC\x53\xEA\x5E\xF5\x8C" - "\x00\x97\x2E\xC5\x39\xD0\x67\xFE" - "\x72\x09\xA0\x14\xAB\x42\xD9\x4D" - "\xE4\x7B\x12", - .ilen = 1011, - .result = "\xF3\x06\x3A\x84\xCD\xBA\x8E\x11" - "\xB7\x74\x6F\x5C\x97\xFB\x36\xFE" - "\xDE\x71\x58\xD4\x15\xD1\xC1\xA4" - "\xC9\x28\x74\xA6\x6B\xC7\x95\xA6" - "\x6C\x77\xF7\x2F\xDF\xC7\xBB\x85" - "\x60\xFC\xE8\x94\xE8\xB5\x09\x2C" - "\x1E\x43\xEF\x6C\xE9\x98\xC5\xA0" - "\x7B\x13\xE5\x7F\xF8\x49\x9A\x8C" - "\xE6\x7B\x08\xC3\x32\x66\x55\x4E" - "\xA5\x44\x1D\x2C\x18\xC7\x29\x1F" - "\x61\x28\x4A\xE3\xCD\xE5\x47\xB2" - "\x82\x2F\x66\x83\x91\x51\xAE\xD7" - "\x1C\x91\x3C\x57\xE3\x1D\x5A\xC9" - "\xFD\xC5\x58\x58\xEF\xCC\x33\xC9" - "\x0F\xEA\x26\x32\xD1\x15\x19\x2D" - "\x25\xB4\x7F\xB0\xDF\xFB\x88\x60" - "\x4E\x4D\x06\x7D\xCC\x1F\xED\x3B" - "\x68\x84\xD5\xB3\x1B\xE7\xB9\xA1" - "\x68\x8B\x2C\x1A\x44\xDA\x63\xD3" - "\x29\xE9\x59\x32\x1F\x30\x1C\x43" - "\xEA\x3A\xA3\x6B\x54\x3C\xAA\x11" - "\xAD\x38\x20\xC9\xB9\x8A\x64\x66" - "\x5A\x07\x49\xDF\xA1\x9C\xF9\x76" - "\x36\x65\xB6\x81\x8F\x76\x09\xE5" - "\xEB\xD1\x29\xA4\xE4\xF4\x4C\xCD" - "\xAF\xFC\xB9\x16\xD9\xC3\x73\x6A" - "\x33\x12\xF8\x7E\xBC\xCC\x7D\x80" - "\xBF\x3C\x25\x06\x13\x84\xFA\x35" - "\xF7\x40\xFA\xA1\x44\x13\x70\xD8" - "\x01\xF9\x85\x15\x63\xEC\x7D\xB9" - "\x02\xD8\xBA\x41\x6C\x92\x68\x66" - "\x95\xDD\xD6\x42\xE7\xBB\xE1\xFD" - "\x28\x3E\x94\xB6\xBD\xA7\xBF\x47" - "\x58\x8D\xFF\x19\x30\x75\x0D\x48" - "\x94\xE9\xA6\xCD\xB3\x8E\x1E\xCD" - "\x59\xBC\x1A\xAC\x3C\x4F\xA9\xEB" - "\xF4\xA7\xE4\x75\x4A\x18\x40\xC9" - "\x1E\xEC\x06\x9C\x28\x4B\xF7\x2B" - "\xE2\xEF\xD6\x42\x2E\xBB\xFC\x0A" - "\x79\xA2\x99\x28\x93\x1B\x00\x57" - "\x35\x1E\x1A\x93\x90\xA4\x68\x95" - "\x5E\x57\x40\xD5\xA9\xAA\x19\x48" - "\xEC\xFF\x76\x77\xDC\x78\x89\x76" - "\xE5\x3B\x00\xEC\x58\x4D\xD1\xE3" - "\xC8\x6C\x2C\x45\x5E\x5F\xD9\x4E" - "\x71\xA5\x36\x6D\x03\xF1\xC7\xD5" - "\xF3\x63\xC0\xD8\xCB\x2B\xF1\xA8" - "\xB9\x2B\xE6\x0B\xB9\x65\x78\xA0" - "\xC4\x46\xE6\x9B\x8B\x43\x2D\xAB" - "\x70\xA6\xE0\x59\x1E\xAC\x9D\xE0" - "\x76\x44\x45\xF3\x24\x11\x57\x98" - "\x9A\x86\xB4\x12\x80\x28\x86\x20" - "\x23\x9D\x2D\xE9\x38\x32\xB1\xE1" - "\xCF\x0A\x23\x73\x7D\xC5\x80\x3D" - "\x9F\x6D\xA0\xD0\xEE\x93\x8A\x79" - "\x3A\xDD\x1D\xBB\x9E\x26\x5D\x01" - "\x44\xD0\xD4\x4E\xC3\xF1\xE4\x38" - "\x09\x62\x0A\x1A\x4E\xD2\x63\x0F" - "\x6E\x3E\xD2\xA4\x3A\xF4\xF3\xFF" - "\x7E\x42\xEC\xB6\x6F\x4D\x6B\x48" - "\xE6\xA6\x50\x80\x78\x9E\xF1\xB0" - "\x4D\xB2\x0D\x3D\xFC\x40\x25\x4D" - "\x93\x11\x1C\xE9\xD2\x9F\x6E\x90" - "\xE5\x41\x4A\xE2\x3C\x45\x29\x35" - "\xEC\xD6\x47\x50\xCB\x7B\xA2\x32" - "\xF7\x8B\x62\xF1\xE3\x9A\xFE\xC7" - "\x1D\x8C\x02\x72\x68\x09\xE9\xB6" - "\x4A\x80\xE6\xB1\x56\xDF\x90\xD4" - "\x93\x74\xA4\xCE\x20\x23\xBF\x48" - "\xA5\xDE\x1B\xFA\x40\x69\x31\x98" - "\x62\x6E\xA5\xC7\xBF\x0C\x62\xE5" - "\x6D\xE1\x93\xF1\x83\x10\x1C\xCA" - "\xF6\x5C\x19\xF8\x90\x78\xCB\xE4" - "\x0B\x3A\xB5\xF8\x43\x86\xD3\x3F" - "\xBA\x83\x34\x3C\x42\xCC\x7D\x28" - "\x29\x63\x4F\xD8\x02\x17\xC5\x07" - "\x2C\xA4\xAC\x79\xCB\xC3\xA9\x09" - "\x81\x45\x18\xED\xE4\xCB\x42\x3B" - "\x87\x2D\x23\xDC\xC5\xBA\x45\xBD" - "\x92\xE5\x02\x97\x96\xCE\xAD\xEC" - "\xBA\xD8\x76\xF8\xCA\xC1\x31\xEC" - "\x1E\x4F\x3F\x83\xF8\x33\xE8\x6E" - "\xCC\xF8\x5F\xDD\x65\x50\x99\x69" - "\xAF\x48\xCE\xA5\xBA\xB6\x14\x9F" - "\x05\x93\xB2\xE6\x59\xC8\x28\xFE" - "\x8F\x37\xF9\x64\xB9\xA5\x56\x8F" - "\xF1\x1B\x90\xEF\xAE\xEB\xFC\x09" - "\x11\x7A\xF2\x19\x0A\x0A\x9A\x3C" - "\xE2\x5E\x29\xFA\x31\x9B\xC1\x74" - "\x1E\x10\x3E\x07\xA9\x31\x6D\xF8" - "\x81\xF5\xD5\x8A\x04\x23\x51\xAC" - "\xA2\xE2\x63\xFD\x27\x1F\x79\x5B" - "\x1F\xE8\xDA\x11\x49\x4D\x1C\xBA" - "\x54\xCC\x0F\xBA\x92\x69\xE5\xCB" - "\x41\x1A\x67\xA6\x40\x82\x70\x8C" - "\x19\x79\x08\xA4\x51\x20\x7D\xC9" - "\x12\x27\xAE\x20\x0D\x2C\xA1\x6D" - "\xF4\x55\xD4\xE7\xE6\xD4\x28\x08" - "\x00\x70\x12\x56\x56\x50\xAD\x14" - "\x5C\x3E\xA2\xD1\x36\x3F\x36\x48" - "\xED\xB1\x57\x3E\x5D\x15\xF6\x1E" - "\x53\xE9\xA4\x3E\xED\x7D\xCF\x7D" - "\x29\xAF\xF3\x1E\x51\xA8\x9F\x85" - "\x8B\xF0\xBB\xCE\xCC\x39\xC3\x64" - "\x4B\xF2\xAD\x70\x19\xD4\x44\x8F" - "\x91\x76\xE8\x15\x66\x34\x9F\xF6" - "\x0F\x15\xA4\xA8\x24\xF8\x58\xB1" - "\x38\x46\x47\xC7\x9B\xCA\xE9\x42" - "\x44\xAA\xE6\xB5\x9C\x91\xA4\xD3" - "\x16\xA0\xED\x42\xBE\xB5\x06\x19" - "\xBE\x67\xE8\xBC\x22\x32\xA4\x1E" - "\x93\xEB\xBE\xE9\xE1\x93\xE5\x31" - "\x3A\xA2\x75\xDF\xE3\x6B\xE7\xCC" - "\xB4\x70\x20\xE0\x6D\x82\x7C\xC8" - "\x94\x5C\x5E\x37\x18\xAD\xED\x8B" - "\x44\x86\xCA\x5E\x07\xB7\x70\x8D" - "\x40\x48\x19\x73\x7C\x78\x64\x0B" - "\xDB\x01\xCA\xAE\x63\x19\xE9\xD1" - "\x6B\x2C\x84\x10\x45\x42\x2E\xC3" - "\xDF\x7F\xAA\xE8\x87\x1B\x63\x46" - "\x74\x28\x9D\x05\x30\x20\x62\x41" - "\xC0\x9F\x2C\x36\x2B\x78\xD7\x26" - "\xDF\x58\x51\xED\xFA\xDC\x87\x79" - "\xBF\x8C\xBF\xC4\x0F\xE5\x05\xDA" - "\x45\xE3\x35\x0D\x69\x91\x54\x1C" - "\xE7\x2C\x49\x08\x8B\x72\xFA\x5C" - "\xF1\x6B\xD9", - .rlen = 1011, + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\x8D\xFC\x81\x9C\xCB\xAA\x5A\x1C" + "\x7E\x95\xCF\x40\xAB\x4D\x6F\xEA" + "\xD3\xD9\xB0\x9A\xB7\xC7\xE0\x2E" + "\xD1\x39\x34\x92\x8F\xFA\x14\xF1" + "\xD5\xD2\x7B\x59\x1F\x35\x28\xC2" + "\x20\xD9\x42\x06\xC9\x0B\x10\x04" + "\xF8\x79\xCD\x32\x86\x75\x4C\xB6" + "\x7B\x1C\x52\xB1\x91\x64\x22\x4B" + "\x13\xC7\xAE\x98\x0E\xB5\xCF\x6F" + "\x3F\xF4\x43\x96\x73\x0D\xA2\x05" + "\xDB\xFD\x28\x90\x2C\x56\xB9\x37" + "\x5B\x69\x0C\xAD\x84\x67\xFF\x15" + "\x4A\xD4\xA7\xD3\xDD\x99\x47\x3A" + "\xED\x34\x35\x78\x6B\x91\xC9\x32" + "\xE1\xBF\xBC\xB4\x04\x85\x6A\x39" + "\xC0\xBA\x51\xD0\x0F\x4E\xD1\xE2" + "\x1C\xFD\x0E\x05\x07\xF4\x10\xED" + "\xA2\x17\xFF\xF5\x64\xC6\x1A\x22" + "\xAD\x78\xE7\xD7\x11\xE9\x99\xB9" + "\xAA\xEC\x6F\xF8\x3B\xBF\xCE\x77" + "\x93\xE8\xAD\x1D\x50\x6C\xAE\xBC" + "\xBA\x5C\x80\xD1\x91\x65\x51\x1B" + "\xE8\x0A\xCD\x99\x96\x71\x3D\xB6" + "\x78\x75\x37\x55\xC1\xF5\x90\x40" + "\x34\xF4\x7E\xC8\xCC\x3A\x5F\x6E" + "\x36\xA1\xA1\xC2\x3A\x72\x42\x8E" + "\x0E\x37\x88\xE8\xCE\x83\xCB\xAD" + "\xE0\x69\x77\x50\xC7\x0C\x99\xCA" + "\x19\x5B\x30\x25\x9A\xEF\x9B\x0C" + "\xEF\x8F\x74\x4C\xCF\x49\x4E\xB9" + "\xC5\xAE\x9E\x2E\x78\x9A\xB9\x48" + "\xD5\x81\xE4\x37\x1D\xBF\x27\xD9" + "\xC5\xD6\x65\x43\x45\x8C\xBB\xB6" + "\x55\xF4\x06\xBB\x49\x53\x8B\x1B" + "\x07\xA9\x96\x69\x5B\xCB\x0F\xBC" + "\x93\x85\x90\x0F\x0A\x68\x40\x2A" + "\x95\xED\x2D\x88\xBF\x71\xD0\xBB" + "\xEC\xB0\x77\x6C\x79\xFC\x3C\x05" + "\x49\x3F\xB8\x24\xEF\x8E\x09\xA2" + "\x1D\xEF\x92\x02\x96\xD4\x7F\xC8" + "\x03\xB2\xCA\xDB\x17\x5C\x52\xCF" + "\xDD\x70\x37\x63\xAA\xA5\x83\x20" + "\x52\x02\xF6\xB9\xE7\x6E\x0A\xB6" + "\x79\x03\xA0\xDA\xA3\x79\x21\xBD" + "\xE3\x37\x3A\xC0\xF7\x2C\x32\xBE" + "\x8B\xE8\xA6\x00\xC7\x32\xD5\x06" + "\xBB\xE3\xAB\x06\x21\x82\xB8\x32" + "\x31\x34\x2A\xA7\x1F\x64\x99\xBF" + "\xFA\xDA\x3D\x75\xF7\x48\xD5\x48" + "\x4B\x52\x7E\xF6\x7C\xAB\x67\x59" + "\xC5\xDC\xA8\xC6\x63\x85\x4A\xDF" + "\xF0\x40\x5F\xCF\xE3\x58\x52\x67" + "\x7A\x24\x32\xC5\xEC\x9E\xA9\x6F" + "\x58\x56\xDD\x94\x1F\x71\x8D\xF4" + "\x6E\xFF\x2C\xA7\xA5\xD8\xBA\xAF" + "\x1D\x8B\xA2\x46\xB5\xC4\x9F\x57" + "\x8D\xD8\xB3\x3C\x02\x0D\xBB\x84" + "\xC7\xBD\xB4\x9A\x6E\xBB\xB1\x37" + "\x95\x79\xC4\xA7\xEA\x1D\xDC\x33" + "\x5D\x0B\x3F\x03\x8F\x30\xF9\xAE" + "\x4F\xFE\x24\x9C\x9A\x02\xE5\x57" + "\xF5\xBC\x25\xD6\x02\x56\x57\x1C", + .len = 496, .also_non_np = 1, - .np = 2, - .tap = { 1011 - 16, 16 }, - }, { /* Generated with Crypto++ */ + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec cast5_cbc_tv_template[] = { + { /* Generated from TF test vectors */ .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", + .klen = 16, + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -31728,275 +27198,652 @@ static const struct cipher_testvec camellia_ctr_enc_tv_template[] = { "\x86\x1D\xB4\x28\xBF\x56\xED\x61" "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" - "\x2B\xC2\x59\xF0\x64\xFB\x92\x06" - "\x9D\x34\xCB\x3F\xD6\x6D\x04\x78" - "\x0F\xA6\x1A\xB1\x48\xDF\x53\xEA" - "\x81\x18\x8C\x23\xBA\x2E\xC5\x5C" - "\xF3\x67\xFE\x95\x09\xA0\x37\xCE" - "\x42\xD9\x70\x07\x7B\x12\xA9\x1D" - "\xB4\x4B\xE2\x56\xED\x84\x1B\x8F" - "\x26\xBD\x31\xC8\x5F\xF6\x6A\x01" - "\x98\x0C\xA3\x3A\xD1\x45\xDC\x73" - "\x0A\x7E\x15\xAC\x20\xB7\x4E\xE5" - "\x59\xF0\x87\x1E\x92\x29\xC0\x34" - "\xCB\x62\xF9\x6D\x04\x9B\x0F\xA6" - "\x3D\xD4\x48\xDF\x76\x0D\x81\x18" - "\xAF\x23\xBA\x51\xE8\x5C\xF3\x8A" - "\x21\x95\x2C\xC3\x37\xCE\x65\xFC" - "\x70\x07\x9E\x12\xA9\x40\xD7\x4B" - "\xE2\x79\x10\x84\x1B\xB2\x26\xBD" - "\x54\xEB\x5F\xF6\x8D\x01\x98\x2F" - "\xC6\x3A\xD1\x68\xFF\x73\x0A\xA1" - "\x15\xAC\x43\xDA\x4E\xE5\x7C\x13" - "\x87\x1E\xB5\x29\xC0\x57\xEE\x62" - "\xF9\x90\x04\x9B\x32\xC9\x3D\xD4" - "\x6B\x02\x76\x0D\xA4\x18\xAF\x46" - "\xDD\x51\xE8\x7F\x16\x8A\x21\xB8" - "\x2C\xC3\x5A\xF1\x65\xFC\x93\x07" - "\x9E\x35\xCC\x40\xD7\x6E\x05\x79" - "\x10\xA7\x1B\xB2\x49\xE0\x54\xEB" - "\x82\x19\x8D\x24\xBB\x2F\xC6\x5D" - "\xF4\x68\xFF\x96\x0A\xA1\x38\xCF" - "\x43\xDA\x71\x08\x7C\x13\xAA\x1E" - "\xB5\x4C\xE3\x57\xEE\x85\x1C\x90" - "\x27\xBE\x32\xC9\x60\xF7\x6B\x02" - "\x99\x0D\xA4\x3B\xD2\x46\xDD\x74" - "\x0B\x7F\x16\xAD\x21\xB8\x4F\xE6" - "\x5A\xF1\x88\x1F\x93\x2A\xC1\x35" - "\xCC\x63\xFA\x6E\x05\x9C\x10\xA7" - "\x3E\xD5\x49\xE0\x77\x0E\x82\x19" - "\xB0\x24\xBB\x52\xE9\x5D\xF4\x8B" - "\x22\x96\x2D\xC4\x38\xCF\x66\xFD" - "\x71\x08\x9F\x13\xAA\x41\xD8\x4C" - "\xE3\x7A\x11\x85\x1C\xB3\x27\xBE" - "\x55\xEC\x60\xF7\x8E\x02\x99\x30" - "\xC7\x3B\xD2\x69\x00\x74\x0B\xA2" - "\x16\xAD\x44\xDB\x4F\xE6\x7D\x14" - "\x88\x1F\xB6\x2A\xC1\x58\xEF\x63" - "\xFA\x91\x05\x9C\x33\xCA\x3E\xD5" - "\x6C\x03\x77\x0E\xA5\x19\xB0\x47" - "\xDE\x52\xE9\x80\x17\x8B\x22\xB9" - "\x2D\xC4\x5B\xF2\x66\xFD\x94\x08" - "\x9F\x36\xCD\x41\xD8\x6F\x06\x7A" - "\x11\xA8\x1C\xB3\x4A\xE1\x55\xEC" - "\x83\x1A\x8E\x25\xBC\x30\xC7\x5E" - "\xF5\x69\x00\x97\x0B\xA2\x39\xD0" - "\x44\xDB\x72\x09\x7D\x14\xAB\x1F" - "\xB6\x4D\xE4\x58\xEF\x86\x1D\x91" - "\x28\xBF\x33\xCA\x61\xF8\x6C\x03" - "\x9A\x0E\xA5\x3C\xD3\x47\xDE\x75" - "\x0C\x80\x17\xAE\x22\xB9\x50\xE7" - "\x5B\xF2\x89\x20\x94\x2B\xC2\x36" - "\xCD\x64\xFB\x6F\x06\x9D\x11\xA8" - "\x3F\xD6\x4A\xE1\x78\x0F\x83\x1A" - "\xB1\x25\xBC\x53\xEA\x5E\xF5\x8C" - "\x00\x97\x2E\xC5\x39\xD0\x67\xFE" - "\x72\x09\xA0\x14\xAB\x42\xD9\x4D", - .ilen = 1008, - .result = "\x85\x79\x6C\x8B\x2B\x6D\x14\xF9" - "\xA6\x83\xB6\x80\x5B\x3A\xF3\x7E" - "\x30\x29\xEB\x1F\xDC\x19\x5F\xEB" - "\xF7\xC4\x27\x04\x51\x87\xD7\x6F" - "\xB8\x4E\x07\xFB\xAC\x3B\x08\xB4" - "\x4D\xCB\xE8\xE1\x71\x7D\x4F\x48" - "\xCD\x81\x64\xA5\xC4\x07\x1A\x9A" - "\x4B\x62\x90\x0E\xC8\xB3\x2B\x6B" - "\x8F\x9C\x6E\x72\x4B\xBA\xEF\x07" - "\x2C\x56\x07\x5E\x37\x30\x60\xA9" - "\xE3\xEF\xD6\x69\xE1\xA1\x77\x64" - "\x93\x75\x7A\xB7\x7A\x3B\xE9\x43" - "\x23\x35\x95\x91\x80\x8A\xC7\xCF" - "\xC3\xD5\xBF\xE7\xFE\x4C\x06\x6B" - "\x05\x19\x48\xE2\x62\xBA\x4F\xF2" - "\xFB\xEE\xE4\xCB\x79\x9D\xA3\x10" - "\x1D\x29\x8C\x1D\x7A\x88\x5A\xDD" - "\x4E\xB6\x18\xAA\xCD\xE6\x33\x96" - "\xD9\x0F\x90\x5A\x78\x76\x4D\x77" - "\x3C\x20\x89\x3B\xA3\xF9\x07\xFD" - "\xE4\xE8\x20\x2D\x15\x0A\x63\x49" - "\xF5\x4F\x89\xD8\xDE\xA1\x28\x78" - "\x28\x07\x09\x1B\x03\x94\x1D\x4B" - "\x82\x28\x1E\x1D\x95\xBA\xAC\x85" - "\x71\x6E\x3C\x18\x4B\x77\x74\x79" - "\xBF\x67\x0A\x53\x3C\x94\xD9\x60" - "\xE9\x6D\x40\x34\xA0\x2A\x53\x5D" - "\x27\xD5\x47\xF9\xC3\x4B\x27\x29" - "\xE4\x76\x9C\x3F\xA7\x1C\x87\xFC" - "\x6E\x0F\xCF\x9B\x60\xF0\xF0\x8B" - "\x70\x1C\x84\x81\x72\x4D\xB4\x98" - "\x23\x62\xE7\x6A\x2B\xFC\xA5\xB2" - "\xFF\xF5\x71\x07\xCD\x90\x23\x13" - "\x19\xD7\x79\x36\x6C\x9D\x55\x8B" - "\x93\x78\x86\x05\x69\x46\xD0\xC5" - "\x39\x09\xEB\x79\xEF\xFA\x9F\xAE" - "\xF3\xD5\x44\xC3\xFD\x86\xD2\x7C" - "\x83\x4B\xD8\x75\x9C\x18\x04\x7B" - "\x73\xAD\x72\xA4\xF6\xAB\xCF\x4B" - "\xCC\x01\x45\x90\xA6\x43\x05\x0C" - "\x6C\x4F\x62\x77\x57\x97\x9F\xEE" - "\x75\xA7\x3C\x38\xD1\x0F\x3D\x0E" - "\x2C\x43\x98\xFB\x13\x65\x73\xE4" - "\x3C\x1E\xD6\x90\x08\xF7\xE0\x99" - "\x3B\xF1\x9D\x6C\x48\xA9\x0E\x32" - "\x17\xC2\xCC\x20\xA1\x19\x26\xAA" - "\xE0\x75\x2F\xFB\x54\x66\x0A\xDF" - "\xB5\xF2\x1F\xC1\x34\x3C\x30\x56" - "\xE8\xDC\xF7\x92\x6B\xBF\x17\x24" - "\xEC\x94\xB5\x3B\xD6\xCE\xA2\x54" - "\x10\x7F\x50\xDE\x69\x77\xD5\x37" - "\xFE\x9C\x10\x83\xC5\xEB\xC9\x53" - "\xB7\xF3\xC4\x20\xAF\x0A\x7E\x57" - "\x3A\xE6\x75\xFE\x89\x00\x6E\x48" - "\xFB\x99\x17\x2C\xF6\x64\x40\x95" - "\x5E\xDC\x7A\xA6\x70\xC7\xF4\xDD" - "\x52\x05\x24\x34\xF9\x0E\xC8\x64" - "\x6D\xE2\xD8\x80\x53\x31\x4C\xFE" - "\xB4\x3A\x5F\x19\xCF\x42\x1B\x22" - "\x0B\x2D\x7B\xF1\xC5\x43\xF7\x5E" - "\x12\xA8\x01\x64\x16\x0B\x26\x5A" - "\x0C\x95\x0F\x40\xC5\x5A\x06\x7C" - "\xCF\xF5\xD5\xB7\x7A\x34\x23\xB6" - "\xAA\x9E\xA8\x98\xA2\xF8\x3D\xD3" - "\x3F\x23\x69\x63\x56\x96\x45\xD6" - "\x74\x23\x1D\x5C\x63\xCC\xD8\x78" - "\x16\xE2\x9C\xD2\x80\x02\xF2\x28" - "\x69\x2F\xC4\xA8\x15\x15\x24\x3B" - "\xCB\xF0\x14\xE4\x62\xC8\xF3\xD1" - "\x03\x58\x1B\x33\x77\x74\x1F\xB4" - "\x07\x86\xF2\x21\xB7\x41\xAE\xBF" - "\x25\xC2\xFF\x51\xEF\xEA\xCE\xC4" - "\x5F\xD9\xB8\x18\x6A\xF0\x0F\x0D" - "\xF8\x04\xBB\x6D\x62\x33\x87\x26" - "\x4F\x2F\x14\x6E\xDC\xDB\x66\x09" - "\x2A\xEF\x7D\x84\x10\xAC\x82\x5E" - "\xD2\xE4\xAD\x74\x7A\x6D\xCC\x3A" - "\x7B\x62\xD8\xD6\x07\x2D\xF7\xDF" - "\x9B\xB3\x82\xCF\x9C\x1D\x76\x5C" - "\xAC\x7B\xD4\x9B\x45\xA1\x64\x11" - "\x66\xF1\xA7\x0B\xF9\xDD\x00\xDD" - "\xA4\x45\x3D\x3E\x03\xC9\x2E\xCB" - "\xC3\x14\x84\x72\xFD\x41\xDC\xBD" - "\x75\xBE\xA8\xE5\x16\x48\x64\x39" - "\xCA\xF3\xE6\xDC\x25\x24\xF1\x6D" - "\xB2\x8D\xC5\x38\x54\xD3\x5D\x6D" - "\x0B\x29\x10\x15\x0E\x13\x3B\xAC" - "\x7E\xCC\x9E\x3E\x18\x48\xA6\x02" - "\xEF\x03\xB2\x2E\xE3\xD2\x70\x21" - "\xB4\x19\x26\xBE\x3A\x3D\x05\xE0" - "\xF8\x09\xAF\xE4\x31\x26\x92\x2F" - "\x8F\x55\xAC\xED\x0B\xB2\xA5\x34" - "\xBE\x50\xB1\x02\x22\x96\xE3\x40" - "\x7B\x70\x50\x6E\x3B\xD5\xE5\xA0" - "\x8E\xA2\xAD\x14\x60\x5C\x7A\x2B" - "\x3D\x1B\x7F\xC1\xC0\x2C\x56\x36" - "\xD2\x0A\x32\x06\x97\x34\xB9\xF4" - "\x6F\x9F\x7E\x80\xD0\x9D\xF7\x6A" - "\x21\xC1\xA2\x6A\xB1\x96\x5B\x4D" - "\x7A\x15\x6C\xC4\x4E\xB8\xE0\x9E" - "\x6C\x50\xF3\x9C\xC9\xB5\x23\xB7" - "\xF1\xD4\x29\x4A\x23\xC4\xAD\x1E" - "\x2C\x07\xD2\x43\x5F\x57\x93\xCA" - "\x85\xF9\x9F\xAD\x4C\xF1\xE4\xB1" - "\x1A\x8E\x28\xA4\xB6\x52\x77\x7E" - "\x68\xC6\x47\xB9\x76\xCC\x65\x5F" - "\x0B\xF9\x67\x93\xD8\x0E\x9A\x37" - "\x5F\x41\xED\x64\x6C\xAD\x5F\xED" - "\x3F\x8D\xFB\x8E\x1E\xA0\xE4\x1F" - "\xC2\xC7\xED\x18\x43\xE1\x20\x86" - "\x5D\xBC\x30\x70\x22\xA1\xDC\x53" - "\x10\x3A\x8D\x47\x82\xCD\x7F\x59" - "\x03\x2D\x6D\xF5\xE7\x79\xD4\x07" - "\x68\x2A\xA5\x42\x19\x4D\xAF\xF5" - "\xED\x47\x83\xBC\x5F\x62\x84\xDA" - "\xDA\x41\xFF\xB0\x1D\x64\xA3\xC8" - "\xBD\x4E\xE0\xB8\x7F\xEE\x55\x0A" - "\x4E\x61\xB2\x51\xF6\x9C\x95\xF6" - "\x92\xBB\xF6\xC5\xF0\x09\x86\xDE" - "\x37\x9E\x29\xF9\x2A\x18\x73\x0D" - "\xDC\x7E\x6B\x7B\x1B\x43\x8C\xEA" - "\x13\xC8\x1A\x47\x0A\x2D\x6D\x56" - "\xCD\xD2\xE7\x53\x1A\xAB\x1C\x3C" - "\xC5\x9B\x03\x70\x29\x2A\x49\x09" - "\x67\xA1\xEA\xD6\x3A\x5B\xBF\x71" - "\x1D\x48\x64\x6C\xFB\xC0\x9E\x36", - .rlen = 1008, + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\x05\x28\xCE\x61\x90\x80\xE1\x78" + "\xB9\x2A\x97\x7C\xB0\x83\xD8\x1A" + "\xDE\x58\x7F\xD7\xFD\x72\xB8\xFB" + "\xDA\xF0\x6E\x77\x14\x47\x82\xBA" + "\x29\x0E\x25\x6E\xB4\x39\xD9\x7F" + "\x05\xA7\xA7\x3A\xC1\x5D\x9E\x39" + "\xA7\xFB\x0D\x05\x00\xF3\x58\x67" + "\x60\xEC\x73\x77\x46\x85\x9B\x6A" + "\x08\x3E\xBE\x59\xFB\xE4\x96\x34" + "\xB4\x05\x49\x1A\x97\x43\xAD\xA0" + "\xA9\x1E\x6E\x74\xF1\x94\xEC\xA8" + "\xB5\x8A\x20\xEA\x89\x6B\x19\xAA" + "\xA7\xF1\x33\x67\x90\x23\x0D\xEE" + "\x81\xD5\x78\x4F\xD3\x63\xEA\x46" + "\xB5\xB2\x6E\xBB\xCA\x76\x06\x10" + "\x96\x2A\x0A\xBA\xF9\x41\x5A\x1D" + "\x36\x7C\x56\x14\x54\x83\xFA\xA1" + "\x27\xDD\xBA\x8A\x90\x29\xD6\xA6" + "\xFA\x48\x3E\x1E\x23\x6E\x98\xA8" + "\xA7\xD9\x67\x92\x5C\x13\xB4\x71" + "\xA8\xAA\x89\x4A\xA4\xB3\x49\x7C" + "\x7D\x7F\xCE\x6F\x29\x2E\x7E\x37" + "\xC8\x52\x60\xD9\xE7\xCA\x60\x98" + "\xED\xCD\xE8\x60\x83\xAD\x34\x4D" + "\x96\x4A\x99\x2B\xB7\x14\x75\x66" + "\x6C\x2C\x1A\xBA\x4B\xBB\x49\x56" + "\xE1\x86\xA2\x0E\xD0\xF0\x07\xD3" + "\x18\x38\x09\x9C\x0E\x8B\x86\x07" + "\x90\x12\x37\x49\x27\x98\x69\x18" + "\xB0\xCC\xFB\xD3\xBD\x04\xA0\x85" + "\x4B\x22\x97\x07\xB6\x97\xE9\x95" + "\x0F\x88\x36\xA9\x44\x00\xC6\xE9" + "\x27\x53\x5C\x5B\x1F\xD3\xE2\xEE" + "\xD0\xCD\x63\x30\xA9\xC0\xDD\x49" + "\xFE\x16\xA4\x07\x0D\xE2\x5D\x97" + "\xDE\x89\xBA\x2E\xF3\xA9\x5E\xBE" + "\x03\x55\x0E\x02\x41\x4A\x45\x06" + "\xBE\xEA\x32\xF2\xDC\x91\x5C\x20" + "\x94\x02\x30\xD2\xFC\x29\xFA\x8E" + "\x34\xA0\x31\xB8\x34\xBA\xAE\x54" + "\xB5\x88\x1F\xDC\x43\xDC\x22\x9F" + "\xDC\xCE\xD3\xFA\xA4\xA8\xBC\x8A" + "\xC7\x5A\x43\x21\xA5\xB1\xDB\xC3" + "\x84\x3B\xB4\x9B\xB5\xA7\xF1\x0A" + "\xB6\x37\x21\x19\x55\xC2\xBD\x99" + "\x49\x24\xBB\x7C\xB3\x8E\xEF\xD2" + "\x3A\xCF\xA0\x31\x28\x0E\x25\xA2" + "\x11\xB4\x18\x17\x1A\x65\x92\x56" + "\xE8\xE0\x52\x9C\x61\x18\x2A\xB1" + "\x1A\x01\x22\x45\x17\x62\x52\x6C" + "\x91\x44\xCF\x98\xC7\xC0\x79\x26" + "\x32\x66\x6F\x23\x7F\x94\x36\x88" + "\x3C\xC9\xD0\xB7\x45\x30\x31\x86" + "\x3D\xC6\xA3\x98\x62\x84\x1A\x8B" + "\x16\x88\xC7\xA3\xE9\x4F\xE0\x86" + "\xA4\x93\xA8\x34\x5A\xCA\xDF\xCA" + "\x46\x38\xD2\xF4\xE0\x2D\x1E\xC9" + "\x7C\xEF\x53\xB7\x60\x72\x41\xBF" + "\x29\x00\x87\x02\xAF\x44\x4C\xB7" + "\x8C\xF5\x3F\x19\xF4\x80\x45\xA7" + "\x15\x5F\xDB\xE9\xB1\x83\xD2\xE6" + "\x1D\x18\x66\x44\x5B\x8F\x14\xEB", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec cast5_ctr_tv_template[] = { + { /* Generated from TF test vectors */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", + .klen = 16, + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" + "\x3A", + .ctext = "\xFF\xC4\x2E\x82\x3D\xF8\xA8\x39" + "\x7C\x52\xC4\xD3\xBB\x62\xC6\xA8" + "\x0C", + .len = 17, + }, { /* Generated from TF test vectors */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A", + .klen = 16, + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" + "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" + "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" + "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" + "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" + "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" + "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" + "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" + "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" + "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" + "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" + "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" + "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" + "\x29\xC0\x57\xEE\x62\xF9\x90\x04" + "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" + "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" + "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" + "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" + "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" + "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" + "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" + "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" + "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" + "\x57\xEE\x85\x1C\x90\x27\xBE\x32" + "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" + "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" + "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" + "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" + "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" + "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" + "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" + "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" + "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" + "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" + "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" + "\x69\x00\x74\x0B\xA2\x16\xAD\x44" + "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" + "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" + "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" + "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" + "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" + "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" + "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" + "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" + "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" + "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" + "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" + "\x58\xEF\x86\x1D\x91\x28\xBF\x33" + "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" + "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" + "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" + "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" + "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" + "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" + "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" + "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" + "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" + "\x86\x1D\xB4\x28\xBF\x56\xED\x61" + "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" + "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\xFF\xC4\x2E\x82\x3D\xF8\xA8\x39" + "\x7C\x52\xC4\xD3\xBB\x62\xC6\xA8" + "\x0C\x63\xA5\x55\xE3\xF8\x1C\x7F" + "\xDC\x59\xF9\xA0\x52\xAD\x83\xDF" + "\xD5\x3B\x53\x4A\xAA\x1F\x49\x44" + "\xE8\x20\xCC\xF8\x97\xE6\xE0\x3C" + "\x5A\xD2\x83\xEC\xEE\x25\x3F\xCF" + "\x0D\xC2\x79\x80\x99\x6E\xFF\x7B" + "\x64\xB0\x7B\x86\x29\x1D\x9F\x17" + "\x10\xA5\xA5\xEB\x16\x55\x9E\xE3" + "\x88\x18\x52\x56\x48\x58\xD1\x6B" + "\xE8\x74\x6E\x48\xB0\x2E\x69\x63" + "\x32\xAA\xAC\x26\x55\x45\x94\xDE" + "\x30\x26\x26\xE6\x08\x82\x2F\x5F" + "\xA7\x15\x94\x07\x75\x2D\xC6\x3A" + "\x1B\xA0\x39\xFB\xBA\xB9\x06\x56" + "\xF6\x9F\xF1\x2F\x9B\xF3\x89\x8B" + "\x08\xC8\x9D\x5E\x6B\x95\x09\xC7" + "\x98\xB7\x62\xA4\x1D\x25\xFA\xC5" + "\x62\xC8\x5D\x6B\xB4\x85\x88\x7F" + "\x3B\x29\xF9\xB4\x32\x62\x69\xBF" + "\x32\xB8\xEB\xFD\x0E\x26\xAA\xA3" + "\x44\x67\x90\x20\xAC\x41\xDF\x43" + "\xC6\xC7\x19\x9F\x2C\x28\x74\xEB" + "\x3E\x7F\x7A\x80\x5B\xE4\x08\x60" + "\xC7\xC9\x71\x34\x44\xCE\x05\xFD" + "\xA8\x91\xA8\x44\x5E\xD3\x89\x2C" + "\xAE\x59\x0F\x07\x88\x79\x53\x26" + "\xAF\xAC\xCB\x1D\x6F\x08\x25\x62" + "\xD0\x82\x65\x66\xE4\x2A\x29\x1C" + "\x9C\x64\x5F\x49\x9D\xF8\x62\xF9" + "\xED\xC4\x13\x52\x75\xDC\xE4\xF9" + "\x68\x0F\x8A\xCD\xA6\x8D\x75\xAA" + "\x49\xA1\x86\x86\x37\x5C\x6B\x3D" + "\x56\xE5\x6F\xBE\x27\xC0\x10\xF8" + "\x3C\x4D\x17\x35\x14\xDC\x1C\xA0" + "\x6E\xAE\xD1\x10\xDD\x83\x06\xC2" + "\x23\xD3\xC7\x27\x15\x04\x2C\x27" + "\xDD\x1F\x2E\x97\x09\x9C\x33\x7D" + "\xAC\x50\x1B\x2E\xC9\x52\x0C\x14" + "\x4B\x78\xC4\xDE\x07\x6A\x12\x02" + "\x6E\xD7\x4B\x91\xB9\x88\x4D\x02" + "\xC3\xB5\x04\xBC\xE0\x67\xCA\x18" + "\x22\xA1\xAE\x9A\x21\xEF\xB2\x06" + "\x35\xCD\xEC\x37\x70\x2D\xFC\x1E" + "\xA8\x31\xE7\xFC\xE5\x8E\x88\x66" + "\x16\xB5\xC8\x45\x21\x37\xBD\x24" + "\xA9\xD5\x36\x12\x9F\x6E\x67\x80" + "\x87\x54\xD5\xAF\x97\xE1\x15\xA7" + "\x11\xF0\x63\x7B\xE1\x44\x14\x1C" + "\x06\x32\x05\x8C\x6C\xDB\x9B\x36" + "\x6A\x6B\xAD\x3A\x27\x55\x20\x4C" + "\x76\x36\x43\xE8\x16\x60\xB5\xF3" + "\xDF\x5A\xC6\xA5\x69\x78\x59\x51" + "\x54\x68\x65\x06\x84\xDE\x3D\xAE" + "\x38\x91\xBD\xCC\xA2\x8A\xEC\xE6" + "\x9E\x83\xAE\x1E\x8E\x34\x5D\xDE" + "\x91\xCE\x8F\xED\x40\xF7\xC8\x8B" + "\x9A\x13\x4C\xAD\x89\x97\x9E\xD1" + "\x91\x01\xD7\x21\x23\x28\x1E\xCC" + "\x8C\x98\xDB\xDE\xFC\x72\x94\xAA" + "\xC0\x0D\x96\xAA\x23\xF8\xFE\x13", + .len = 496, + .also_non_np = 1, + .np = 3, + .tap = { 496 - 20, 4, 16 }, + }, +}; + +/* + * ARC4 test vectors from OpenSSL + */ +static const struct cipher_testvec arc4_tv_template[] = { + { + .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .klen = 8, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .ctext = "\x75\xb7\x87\x80\x99\xe0\xc5\x96", + .len = 8, + }, { + .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .klen = 8, + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\x74\x94\xc2\xe7\x10\x4b\x08\x79", + .len = 8, + }, { + .key = "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 8, + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\xde\x18\x89\x41\xa3\x37\x5d\x3a", + .len = 8, + }, { + .key = "\xef\x01\x23\x45", + .klen = 4, + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00", + .ctext = "\xd6\xa1\x41\xa7\xec\x3c\x38\xdf" + "\xbd\x61\x5a\x11\x62\xe1\xc7\xba" + "\x36\xb6\x78\x58", + .len = 20, + }, { + .key = "\x01\x23\x45\x67\x89\xab\xcd\xef", + .klen = 8, + .ptext = "\x12\x34\x56\x78\x9A\xBC\xDE\xF0" + "\x12\x34\x56\x78\x9A\xBC\xDE\xF0" + "\x12\x34\x56\x78\x9A\xBC\xDE\xF0" + "\x12\x34\x56\x78", + .ctext = "\x66\xa0\x94\x9f\x8a\xf7\xd6\x89" + "\x1f\x7f\x83\x2b\xa8\x33\xc0\x0c" + "\x89\x2e\xbe\x30\x14\x3c\xe2\x87" + "\x40\x01\x1e\xcf", + .len = 28, + }, { + .key = "\xef\x01\x23\x45", + .klen = 4, + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00", + .ctext = "\xd6\xa1\x41\xa7\xec\x3c\x38\xdf" + "\xbd\x61", + .len = 10, + }, { + .key = "\x01\x23\x45\x67\x89\xAB\xCD\xEF" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 16, + .ptext = "\x01\x23\x45\x67\x89\xAB\xCD\xEF", + .ctext = "\x69\x72\x36\x59\x1B\x52\x42\xB1", + .len = 8, + }, +}; + +/* + * TEA test vectors + */ +static const struct cipher_testvec tea_tv_template[] = { + { + .key = zeroed_string, + .klen = 16, + .ptext = zeroed_string, + .ctext = "\x0a\x3a\xea\x41\x40\xa9\xba\x94", + .len = 8, + }, { + .key = "\x2b\x02\x05\x68\x06\x14\x49\x76" + "\x77\x5d\x0e\x26\x6c\x28\x78\x43", + .klen = 16, + .ptext = "\x74\x65\x73\x74\x20\x6d\x65\x2e", + .ctext = "\x77\x5d\x2a\x6a\xf6\xce\x92\x09", + .len = 8, + }, { + .key = "\x09\x65\x43\x11\x66\x44\x39\x25" + "\x51\x3a\x16\x10\x0a\x08\x12\x6e", + .klen = 16, + .ptext = "\x6c\x6f\x6e\x67\x65\x72\x5f\x74" + "\x65\x73\x74\x5f\x76\x65\x63\x74", + .ctext = "\xbe\x7a\xbb\x81\x95\x2d\x1f\x1e" + "\xdd\x89\xa1\x25\x04\x21\xdf\x95", + .len = 16, + }, { + .key = "\x4d\x76\x32\x17\x05\x3f\x75\x2c" + "\x5d\x04\x16\x36\x15\x72\x63\x2f", + .klen = 16, + .ptext = "\x54\x65\x61\x20\x69\x73\x20\x67" + "\x6f\x6f\x64\x20\x66\x6f\x72\x20" + "\x79\x6f\x75\x21\x21\x21\x20\x72" + "\x65\x61\x6c\x6c\x79\x21\x21\x21", + .ctext = "\xe0\x4d\x5d\x3c\xb7\x8c\x36\x47" + "\x94\x18\x95\x91\xa9\xfc\x49\xf8" + "\x44\xd1\x2d\xc2\x99\xb8\x08\x2a" + "\x07\x89\x73\xc2\x45\x92\xc6\x90", + .len = 32, + } +}; + +/* + * XTEA test vectors + */ +static const struct cipher_testvec xtea_tv_template[] = { + { + .key = zeroed_string, + .klen = 16, + .ptext = zeroed_string, + .ctext = "\xd8\xd4\xe9\xde\xd9\x1e\x13\xf7", + .len = 8, + }, { + .key = "\x2b\x02\x05\x68\x06\x14\x49\x76" + "\x77\x5d\x0e\x26\x6c\x28\x78\x43", + .klen = 16, + .ptext = "\x74\x65\x73\x74\x20\x6d\x65\x2e", + .ctext = "\x94\xeb\xc8\x96\x84\x6a\x49\xa8", + .len = 8, + }, { + .key = "\x09\x65\x43\x11\x66\x44\x39\x25" + "\x51\x3a\x16\x10\x0a\x08\x12\x6e", + .klen = 16, + .ptext = "\x6c\x6f\x6e\x67\x65\x72\x5f\x74" + "\x65\x73\x74\x5f\x76\x65\x63\x74", + .ctext = "\x3e\xce\xae\x22\x60\x56\xa8\x9d" + "\x77\x4d\xd4\xb4\x87\x24\xe3\x9a", + .len = 16, + }, { + .key = "\x4d\x76\x32\x17\x05\x3f\x75\x2c" + "\x5d\x04\x16\x36\x15\x72\x63\x2f", + .klen = 16, + .ptext = "\x54\x65\x61\x20\x69\x73\x20\x67" + "\x6f\x6f\x64\x20\x66\x6f\x72\x20" + "\x79\x6f\x75\x21\x21\x21\x20\x72" + "\x65\x61\x6c\x6c\x79\x21\x21\x21", + .ctext = "\x99\x81\x9f\x5d\x6f\x4b\x31\x3a" + "\x86\xff\x6f\xd0\xe3\x87\x70\x07" + "\x4d\xb8\xcf\xf3\x99\x50\xb3\xd4" + "\x73\xa2\xfa\xc9\x16\x59\x5d\x81", + .len = 32, + } +}; + +/* + * KHAZAD test vectors. + */ +static const struct cipher_testvec khazad_tv_template[] = { + { + .key = "\x80\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 16, + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\x49\xa4\xce\x32\xac\x19\x0e\x3f", + .len = 8, + }, { + .key = "\x38\x38\x38\x38\x38\x38\x38\x38" + "\x38\x38\x38\x38\x38\x38\x38\x38", + .klen = 16, + .ptext = "\x38\x38\x38\x38\x38\x38\x38\x38", + .ctext = "\x7e\x82\x12\xa1\xd9\x5b\xe4\xf9", + .len = 8, + }, { + .key = "\xa2\xa2\xa2\xa2\xa2\xa2\xa2\xa2" + "\xa2\xa2\xa2\xa2\xa2\xa2\xa2\xa2", + .klen = 16, + .ptext = "\xa2\xa2\xa2\xa2\xa2\xa2\xa2\xa2", + .ctext = "\xaa\xbe\xc1\x95\xc5\x94\x1a\x9c", + .len = 8, + }, { + .key = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f" + "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", + .klen = 16, + .ptext = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", + .ctext = "\x04\x74\xf5\x70\x50\x16\xd3\xb8", + .len = 8, + }, { + .key = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f" + "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", + .klen = 16, + .ptext = "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f" + "\x2f\x2f\x2f\x2f\x2f\x2f\x2f\x2f", + .ctext = "\x04\x74\xf5\x70\x50\x16\xd3\xb8" + "\x04\x74\xf5\x70\x50\x16\xd3\xb8", + .len = 16, }, }; -static const struct cipher_testvec camellia_ctr_dec_tv_template[] = { - { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", +/* + * Anubis test vectors. + */ + +static const struct cipher_testvec anubis_tv_template[] = { + { + .key = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" + "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", + .klen = 16, + .ptext = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" + "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", + .ctext = "\x6d\xc5\xda\xa2\x26\x7d\x62\x6f" + "\x08\xb7\x52\x8e\x6e\x6e\x86\x90", + .len = 16, + }, { + + .key = "\x03\x03\x03\x03\x03\x03\x03\x03" + "\x03\x03\x03\x03\x03\x03\x03\x03" + "\x03\x03\x03\x03", + .klen = 20, + .ptext = "\x03\x03\x03\x03\x03\x03\x03\x03" + "\x03\x03\x03\x03\x03\x03\x03\x03", + .ctext = "\xdb\xf1\x42\xf4\xd1\x8a\xc7\x49" + "\x87\x41\x6f\x82\x0a\x98\x64\xae", + .len = 16, + }, { + .key = "\x24\x24\x24\x24\x24\x24\x24\x24" + "\x24\x24\x24\x24\x24\x24\x24\x24" + "\x24\x24\x24\x24\x24\x24\x24\x24" + "\x24\x24\x24\x24", + .klen = 28, + .ptext = "\x24\x24\x24\x24\x24\x24\x24\x24" + "\x24\x24\x24\x24\x24\x24\x24\x24", + .ctext = "\xfd\x1b\x4a\xe3\xbf\xf0\xad\x3d" + "\x06\xd3\x61\x27\xfd\x13\x9e\xde", + .len = 16, + }, { + .key = "\x25\x25\x25\x25\x25\x25\x25\x25" + "\x25\x25\x25\x25\x25\x25\x25\x25" + "\x25\x25\x25\x25\x25\x25\x25\x25" + "\x25\x25\x25\x25\x25\x25\x25\x25", .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\xF3\x06\x3A\x84\xCD\xBA\x8E\x11" - "\xB7\x74\x6F\x5C\x97\xFB\x36\xFE" - "\xDE\x71\x58\xD4\x15\xD1\xC1\xA4" - "\xC9\x28\x74\xA6\x6B\xC7\x95\xA6" - "\x6C\x77\xF7\x2F\xDF\xC7\xBB\x85" - "\x60\xFC\xE8\x94\xE8\xB5\x09\x2C" - "\x1E\x43\xEF\x6C\xE9\x98\xC5\xA0" - "\x7B\x13\xE5\x7F\xF8\x49\x9A\x8C" - "\xE6\x7B\x08\xC3\x32\x66\x55\x4E" - "\xA5\x44\x1D\x2C\x18\xC7\x29\x1F" - "\x61\x28\x4A\xE3\xCD\xE5\x47\xB2" - "\x82\x2F\x66\x83\x91\x51\xAE\xD7" - "\x1C\x91\x3C\x57\xE3\x1D\x5A\xC9" - "\xFD\xC5\x58\x58\xEF\xCC\x33\xC9" - "\x0F\xEA\x26\x32\xD1\x15\x19\x2D" - "\x25\xB4\x7F\xB0\xDF\xFB\x88\x60" - "\x4E\x4D\x06\x7D\xCC\x1F\xED\x3B" - "\x68\x84\xD5\xB3\x1B\xE7\xB9\xA1" - "\x68\x8B\x2C\x1A\x44\xDA\x63\xD3" - "\x29\xE9\x59\x32\x1F\x30\x1C\x43" - "\xEA\x3A\xA3\x6B\x54\x3C\xAA\x11" - "\xAD\x38\x20\xC9\xB9\x8A\x64\x66" - "\x5A\x07\x49\xDF\xA1\x9C\xF9\x76" - "\x36\x65\xB6\x81\x8F\x76\x09\xE5" - "\xEB\xD1\x29\xA4\xE4\xF4\x4C\xCD" - "\xAF\xFC\xB9\x16\xD9\xC3\x73\x6A" - "\x33\x12\xF8\x7E\xBC\xCC\x7D\x80" - "\xBF\x3C\x25\x06\x13\x84\xFA\x35" - "\xF7\x40\xFA\xA1\x44\x13\x70\xD8" - "\x01\xF9\x85\x15\x63\xEC\x7D\xB9" - "\x02\xD8\xBA\x41\x6C\x92\x68\x66" - "\x95\xDD\xD6\x42\xE7\xBB\xE1\xFD" - "\x28\x3E\x94\xB6\xBD\xA7\xBF\x47" - "\x58\x8D\xFF\x19\x30\x75\x0D\x48" - "\x94\xE9\xA6\xCD\xB3\x8E\x1E\xCD" - "\x59\xBC\x1A\xAC\x3C\x4F\xA9\xEB" - "\xF4\xA7\xE4\x75\x4A\x18\x40\xC9" - "\x1E\xEC\x06\x9C\x28\x4B\xF7\x2B" - "\xE2\xEF\xD6\x42\x2E\xBB\xFC\x0A" - "\x79\xA2\x99\x28\x93\x1B\x00\x57" - "\x35\x1E\x1A\x93\x90\xA4\x68\x95" - "\x5E\x57\x40\xD5\xA9\xAA\x19\x48" - "\xEC\xFF\x76\x77\xDC\x78\x89\x76" - "\xE5\x3B\x00\xEC\x58\x4D\xD1\xE3" - "\xC8\x6C\x2C\x45\x5E\x5F\xD9\x4E" - "\x71\xA5\x36\x6D\x03\xF1\xC7\xD5" - "\xF3\x63\xC0\xD8\xCB\x2B\xF1\xA8" - "\xB9\x2B\xE6\x0B\xB9\x65\x78\xA0" - "\xC4\x46\xE6\x9B\x8B\x43\x2D\xAB" - "\x70\xA6\xE0\x59\x1E\xAC\x9D\xE0" - "\x76\x44\x45\xF3\x24\x11\x57\x98" - "\x9A\x86\xB4\x12\x80\x28\x86\x20" - "\x23\x9D\x2D\xE9\x38\x32\xB1\xE1" - "\xCF\x0A\x23\x73\x7D\xC5\x80\x3D" - "\x9F\x6D\xA0\xD0\xEE\x93\x8A\x79" - "\x3A\xDD\x1D\xBB\x9E\x26\x5D\x01" - "\x44\xD0\xD4\x4E\xC3\xF1\xE4\x38" - "\x09\x62\x0A\x1A\x4E\xD2\x63\x0F" - "\x6E\x3E\xD2\xA4\x3A\xF4\xF3\xFF" - "\x7E\x42\xEC\xB6\x6F\x4D\x6B\x48" - "\xE6\xA6\x50\x80\x78\x9E\xF1\xB0" - "\x4D\xB2\x0D\x3D\xFC\x40\x25\x4D", - .ilen = 496, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .ptext = "\x25\x25\x25\x25\x25\x25\x25\x25" + "\x25\x25\x25\x25\x25\x25\x25\x25", + .ctext = "\x1a\x91\xfb\x2b\xb7\x78\x6b\xc4" + "\x17\xd9\xff\x40\x3b\x0e\xe5\xfe", + .len = 16, + }, { + .key = "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35", + .klen = 40, + .ptext = "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35", + .ctext = "\xa5\x2c\x85\x6f\x9c\xba\xa0\x97" + "\x9e\xc6\x84\x0f\x17\x21\x07\xee", + .len = 16, + }, +}; + +static const struct cipher_testvec anubis_cbc_tv_template[] = { + { + .key = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" + "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", + .klen = 16, + .ptext = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" + "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" + "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe" + "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe", + .ctext = "\x6d\xc5\xda\xa2\x26\x7d\x62\x6f" + "\x08\xb7\x52\x8e\x6e\x6e\x86\x90" + "\x86\xd8\xb5\x6f\x98\x5e\x8a\x66" + "\x4f\x1f\x78\xa1\xbb\x37\xf1\xbe", + .len = 32, + }, { + .key = "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35", + .klen = 40, + .ptext = "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35" + "\x35\x35\x35\x35\x35\x35\x35\x35", + .ctext = "\xa5\x2c\x85\x6f\x9c\xba\xa0\x97" + "\x9e\xc6\x84\x0f\x17\x21\x07\xee" + "\xa2\xbc\x06\x98\xc6\x4b\xda\x75" + "\x2e\xaa\xbe\x58\xce\x01\x5b\xc7", + .len = 32, + }, +}; + +/* + * XETA test vectors + */ +static const struct cipher_testvec xeta_tv_template[] = { + { + .key = zeroed_string, + .klen = 16, + .ptext = zeroed_string, + .ctext = "\xaa\x22\x96\xe5\x6c\x61\xf3\x45", + .len = 8, + }, { + .key = "\x2b\x02\x05\x68\x06\x14\x49\x76" + "\x77\x5d\x0e\x26\x6c\x28\x78\x43", + .klen = 16, + .ptext = "\x74\x65\x73\x74\x20\x6d\x65\x2e", + .ctext = "\x82\x3e\xeb\x35\xdc\xdd\xd9\xc3", + .len = 8, + }, { + .key = "\x09\x65\x43\x11\x66\x44\x39\x25" + "\x51\x3a\x16\x10\x0a\x08\x12\x6e", + .klen = 16, + .ptext = "\x6c\x6f\x6e\x67\x65\x72\x5f\x74" + "\x65\x73\x74\x5f\x76\x65\x63\x74", + .ctext = "\xe2\x04\xdb\xf2\x89\x85\x9e\xea" + "\x61\x35\xaa\xed\xb5\xcb\x71\x2c", + .len = 16, + }, { + .key = "\x4d\x76\x32\x17\x05\x3f\x75\x2c" + "\x5d\x04\x16\x36\x15\x72\x63\x2f", + .klen = 16, + .ptext = "\x54\x65\x61\x20\x69\x73\x20\x67" + "\x6f\x6f\x64\x20\x66\x6f\x72\x20" + "\x79\x6f\x75\x21\x21\x21\x20\x72" + "\x65\x61\x6c\x6c\x79\x21\x21\x21", + .ctext = "\x0b\x03\xcd\x8a\xbe\x95\xfd\xb1" + "\xc1\x44\x91\x0b\xa5\xc9\x1b\xb4" + "\xa9\xda\x1e\x9e\xb1\x3e\x2a\x8f" + "\xea\xa5\x6a\x85\xd1\xf4\xa8\xa5", + .len = 32, + } +}; + +/* + * FCrypt test vectors + */ +static const struct cipher_testvec fcrypt_pcbc_tv_template[] = { + { /* http://www.openafs.org/pipermail/openafs-devel/2000-December/005320.html */ + .key = "\x00\x00\x00\x00\x00\x00\x00\x00", + .klen = 8, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00", + .ctext = "\x0E\x09\x00\xC7\x3E\xF7\xED\x41", + .len = 8, + }, { + .key = "\x11\x44\x77\xAA\xDD\x00\x33\x66", + .klen = 8, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x12\x34\x56\x78\x9A\xBC\xDE\xF0", + .ctext = "\xD8\xED\x78\x74\x77\xEC\x06\x80", + .len = 8, + }, { /* From Arla */ + .key = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", + .klen = 8, + .iv = "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .ptext = "The quick brown fox jumps over the lazy dogs.\0\0", + .ctext = "\x00\xf0\x0e\x11\x75\xe6\x23\x82" + "\xee\xac\x98\x62\x44\x51\xe4\x84" + "\xc3\x59\xd8\xaa\x64\x60\xae\xf7" + "\xd2\xd9\x13\x79\x72\xa3\x45\x03" + "\x23\xb5\x62\xd7\x0c\xf5\x27\xd1" + "\xf8\x91\x3c\xac\x44\x22\x92\xef", + .len = 48, + }, { + .key = "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .klen = 8, + .iv = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", + .ptext = "The quick brown fox jumps over the lazy dogs.\0\0", + .ctext = "\xca\x90\xf5\x9d\xcb\xd4\xd2\x3c" + "\x01\x88\x7f\x3e\x31\x6e\x62\x9d" + "\xd8\xe0\x57\xa3\x06\x3a\x42\x58" + "\x2a\x28\xfe\x72\x52\x2f\xdd\xe0" + "\x19\x89\x09\x1c\x2a\x8e\x8c\x94" + "\xfc\xc7\x68\xe4\x88\xaa\xde\x0f", + .len = 48, + }, { /* split-page version */ + .key = "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .klen = 8, + .iv = "\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87", + .ptext = "The quick brown fox jumps over the lazy dogs.\0\0", + .ctext = "\xca\x90\xf5\x9d\xcb\xd4\xd2\x3c" + "\x01\x88\x7f\x3e\x31\x6e\x62\x9d" + "\xd8\xe0\x57\xa3\x06\x3a\x42\x58" + "\x2a\x28\xfe\x72\x52\x2f\xdd\xe0" + "\x19\x89\x09\x1c\x2a\x8e\x8c\x94" + "\xfc\xc7\x68\xe4\x88\xaa\xde\x0f", + .len = 48, + .np = 2, + .tap = { 20, 28 }, + } +}; + +/* + * CAMELLIA test vectors. + */ +static const struct cipher_testvec camellia_tv_template[] = { + { + .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .klen = 16, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .ctext = "\x67\x67\x31\x38\x54\x96\x69\x73" + "\x08\x57\x06\x56\x48\xea\xbe\x43", + .len = 16, + }, { + .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\xfe\xdc\xba\x98\x76\x54\x32\x10" + "\x00\x11\x22\x33\x44\x55\x66\x77", + .klen = 24, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .ctext = "\xb4\x99\x34\x01\xb3\xe9\x96\xf8" + "\x4e\xe5\xce\xe7\xd7\x9b\x09\xb9", + .len = 16, + }, { + .key = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\xfe\xdc\xba\x98\x76\x54\x32\x10" + "\x00\x11\x22\x33\x44\x55\x66\x77" + "\x88\x99\xaa\xbb\xcc\xdd\xee\xff", + .klen = 32, + .ptext = "\x01\x23\x45\x67\x89\xab\xcd\xef" + "\xfe\xdc\xba\x98\x76\x54\x32\x10", + .ctext = "\x9a\xcc\x23\x7d\xff\x16\xd7\x6c" + "\x20\xef\x7c\x91\x9e\x3a\x75\x09", + .len = 16, + }, { /* Generated with Crypto++ */ + .key = "\x3F\x85\x62\x3F\x1C\xF9\xD6\x1C" + "\xF9\xD6\xB3\x90\x6D\x4A\x90\x6D" + "\x4A\x27\x04\xE1\x27\x04\xE1\xBE" + "\x9B\x78\xBE\x9B\x78\x55\x32\x0F", + .klen = 32, + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -32057,145 +27904,239 @@ static const struct cipher_testvec camellia_ctr_dec_tv_template[] = { "\x86\x1D\xB4\x28\xBF\x56\xED\x61" "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" - "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", - .rlen = 496, - }, { /* Generated with Crypto++ */ - .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" - "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" - "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" - "\x78\xBE\x9B\x78\x55\x32\x0F\x55", - .klen = 32, - .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" - "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", - .input = "\xF3\x06\x3A\x84\xCD\xBA\x8E\x11" - "\xB7\x74\x6F\x5C\x97\xFB\x36\xFE" - "\xDE\x71\x58\xD4\x15\xD1\xC1\xA4" - "\xC9\x28\x74\xA6\x6B\xC7\x95\xA6" - "\x6C\x77\xF7\x2F\xDF\xC7\xBB\x85" - "\x60\xFC\xE8\x94\xE8\xB5\x09\x2C" - "\x1E\x43\xEF\x6C\xE9\x98\xC5\xA0" - "\x7B\x13\xE5\x7F\xF8\x49\x9A\x8C" - "\xE6\x7B\x08\xC3\x32\x66\x55\x4E" - "\xA5\x44\x1D\x2C\x18\xC7\x29\x1F" - "\x61\x28\x4A\xE3\xCD\xE5\x47\xB2" - "\x82\x2F\x66\x83\x91\x51\xAE\xD7" - "\x1C\x91\x3C\x57\xE3\x1D\x5A\xC9" - "\xFD\xC5\x58\x58\xEF\xCC\x33\xC9" - "\x0F\xEA\x26\x32\xD1\x15\x19\x2D" - "\x25\xB4\x7F\xB0\xDF\xFB\x88\x60" - "\x4E\x4D\x06\x7D\xCC\x1F\xED\x3B" - "\x68\x84\xD5\xB3\x1B\xE7\xB9\xA1" - "\x68\x8B\x2C\x1A\x44\xDA\x63\xD3" - "\x29\xE9\x59\x32\x1F\x30\x1C\x43" - "\xEA\x3A\xA3\x6B\x54\x3C\xAA\x11" - "\xAD\x38\x20\xC9\xB9\x8A\x64\x66" - "\x5A\x07\x49\xDF\xA1\x9C\xF9\x76" - "\x36\x65\xB6\x81\x8F\x76\x09\xE5" - "\xEB\xD1\x29\xA4\xE4\xF4\x4C\xCD" - "\xAF\xFC\xB9\x16\xD9\xC3\x73\x6A" - "\x33\x12\xF8\x7E\xBC\xCC\x7D\x80" - "\xBF\x3C\x25\x06\x13\x84\xFA\x35" - "\xF7\x40\xFA\xA1\x44\x13\x70\xD8" - "\x01\xF9\x85\x15\x63\xEC\x7D\xB9" - "\x02\xD8\xBA\x41\x6C\x92\x68\x66" - "\x95\xDD\xD6\x42\xE7\xBB\xE1\xFD" - "\x28\x3E\x94\xB6\xBD\xA7\xBF\x47" - "\x58\x8D\xFF\x19\x30\x75\x0D\x48" - "\x94\xE9\xA6\xCD\xB3\x8E\x1E\xCD" - "\x59\xBC\x1A\xAC\x3C\x4F\xA9\xEB" - "\xF4\xA7\xE4\x75\x4A\x18\x40\xC9" - "\x1E\xEC\x06\x9C\x28\x4B\xF7\x2B" - "\xE2\xEF\xD6\x42\x2E\xBB\xFC\x0A" - "\x79\xA2\x99\x28\x93\x1B\x00\x57" - "\x35\x1E\x1A\x93\x90\xA4\x68\x95" - "\x5E\x57\x40\xD5\xA9\xAA\x19\x48" - "\xEC\xFF\x76\x77\xDC\x78\x89\x76" - "\xE5\x3B\x00\xEC\x58\x4D\xD1\xE3" - "\xC8\x6C\x2C\x45\x5E\x5F\xD9\x4E" - "\x71\xA5\x36\x6D\x03\xF1\xC7\xD5" - "\xF3\x63\xC0\xD8\xCB\x2B\xF1\xA8" - "\xB9\x2B\xE6\x0B\xB9\x65\x78\xA0" - "\xC4\x46\xE6\x9B\x8B\x43\x2D\xAB" - "\x70\xA6\xE0\x59\x1E\xAC\x9D\xE0" - "\x76\x44\x45\xF3\x24\x11\x57\x98" - "\x9A\x86\xB4\x12\x80\x28\x86\x20" - "\x23\x9D\x2D\xE9\x38\x32\xB1\xE1" - "\xCF\x0A\x23\x73\x7D\xC5\x80\x3D" - "\x9F\x6D\xA0\xD0\xEE\x93\x8A\x79" - "\x3A\xDD\x1D\xBB\x9E\x26\x5D\x01" - "\x44\xD0\xD4\x4E\xC3\xF1\xE4\x38" - "\x09\x62\x0A\x1A\x4E\xD2\x63\x0F" - "\x6E\x3E\xD2\xA4\x3A\xF4\xF3\xFF" - "\x7E\x42\xEC\xB6\x6F\x4D\x6B\x48" - "\xE6\xA6\x50\x80\x78\x9E\xF1\xB0" - "\x4D\xB2\x0D\x3D\xFC\x40\x25\x4D" - "\x93\x11\x1C\xE9\xD2\x9F\x6E\x90" - "\xE5\x41\x4A\xE2\x3C\x45\x29\x35" - "\xEC\xD6\x47\x50\xCB\x7B\xA2\x32" - "\xF7\x8B\x62\xF1\xE3\x9A\xFE\xC7" - "\x1D\x8C\x02\x72\x68\x09\xE9\xB6" - "\x4A\x80\xE6\xB1\x56\xDF\x90\xD4" - "\x93\x74\xA4\xCE\x20\x23\xBF\x48" - "\xA5\xDE\x1B\xFA\x40\x69\x31\x98" - "\x62\x6E\xA5\xC7\xBF\x0C\x62\xE5" - "\x6D\xE1\x93\xF1\x83\x10\x1C\xCA" - "\xF6\x5C\x19\xF8\x90\x78\xCB\xE4" - "\x0B\x3A\xB5\xF8\x43\x86\xD3\x3F" - "\xBA\x83\x34\x3C\x42\xCC\x7D\x28" - "\x29\x63\x4F\xD8\x02\x17\xC5\x07" - "\x2C\xA4\xAC\x79\xCB\xC3\xA9\x09" - "\x81\x45\x18\xED\xE4\xCB\x42\x3B" - "\x87\x2D\x23\xDC\xC5\xBA\x45\xBD" - "\x92\xE5\x02\x97\x96\xCE\xAD\xEC" - "\xBA\xD8\x76\xF8\xCA\xC1\x31\xEC" - "\x1E\x4F\x3F\x83\xF8\x33\xE8\x6E" - "\xCC\xF8\x5F\xDD\x65\x50\x99\x69" - "\xAF\x48\xCE\xA5\xBA\xB6\x14\x9F" - "\x05\x93\xB2\xE6\x59\xC8\x28\xFE" - "\x8F\x37\xF9\x64\xB9\xA5\x56\x8F" - "\xF1\x1B\x90\xEF\xAE\xEB\xFC\x09" - "\x11\x7A\xF2\x19\x0A\x0A\x9A\x3C" - "\xE2\x5E\x29\xFA\x31\x9B\xC1\x74" - "\x1E\x10\x3E\x07\xA9\x31\x6D\xF8" - "\x81\xF5\xD5\x8A\x04\x23\x51\xAC" - "\xA2\xE2\x63\xFD\x27\x1F\x79\x5B" - "\x1F\xE8\xDA\x11\x49\x4D\x1C\xBA" - "\x54\xCC\x0F\xBA\x92\x69\xE5\xCB" - "\x41\x1A\x67\xA6\x40\x82\x70\x8C" - "\x19\x79\x08\xA4\x51\x20\x7D\xC9" - "\x12\x27\xAE\x20\x0D\x2C\xA1\x6D" - "\xF4\x55\xD4\xE7\xE6\xD4\x28\x08" - "\x00\x70\x12\x56\x56\x50\xAD\x14" - "\x5C\x3E\xA2\xD1\x36\x3F\x36\x48" - "\xED\xB1\x57\x3E\x5D\x15\xF6\x1E" - "\x53\xE9\xA4\x3E\xED\x7D\xCF\x7D" - "\x29\xAF\xF3\x1E\x51\xA8\x9F\x85" - "\x8B\xF0\xBB\xCE\xCC\x39\xC3\x64" - "\x4B\xF2\xAD\x70\x19\xD4\x44\x8F" - "\x91\x76\xE8\x15\x66\x34\x9F\xF6" - "\x0F\x15\xA4\xA8\x24\xF8\x58\xB1" - "\x38\x46\x47\xC7\x9B\xCA\xE9\x42" - "\x44\xAA\xE6\xB5\x9C\x91\xA4\xD3" - "\x16\xA0\xED\x42\xBE\xB5\x06\x19" - "\xBE\x67\xE8\xBC\x22\x32\xA4\x1E" - "\x93\xEB\xBE\xE9\xE1\x93\xE5\x31" - "\x3A\xA2\x75\xDF\xE3\x6B\xE7\xCC" - "\xB4\x70\x20\xE0\x6D\x82\x7C\xC8" - "\x94\x5C\x5E\x37\x18\xAD\xED\x8B" - "\x44\x86\xCA\x5E\x07\xB7\x70\x8D" - "\x40\x48\x19\x73\x7C\x78\x64\x0B" - "\xDB\x01\xCA\xAE\x63\x19\xE9\xD1" - "\x6B\x2C\x84\x10\x45\x42\x2E\xC3" - "\xDF\x7F\xAA\xE8\x87\x1B\x63\x46" - "\x74\x28\x9D\x05\x30\x20\x62\x41" - "\xC0\x9F\x2C\x36\x2B\x78\xD7\x26" - "\xDF\x58\x51\xED\xFA\xDC\x87\x79" - "\xBF\x8C\xBF\xC4\x0F\xE5\x05\xDA" - "\x45\xE3\x35\x0D\x69\x91\x54\x1C" - "\xE7\x2C\x49\x08\x8B\x72\xFA\x5C" - "\xF1\x6B\xD9", - .ilen = 1011, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" + "\x2B\xC2\x59\xF0\x64\xFB\x92\x06" + "\x9D\x34\xCB\x3F\xD6\x6D\x04\x78" + "\x0F\xA6\x1A\xB1\x48\xDF\x53\xEA" + "\x81\x18\x8C\x23\xBA\x2E\xC5\x5C" + "\xF3\x67\xFE\x95\x09\xA0\x37\xCE" + "\x42\xD9\x70\x07\x7B\x12\xA9\x1D" + "\xB4\x4B\xE2\x56\xED\x84\x1B\x8F" + "\x26\xBD\x31\xC8\x5F\xF6\x6A\x01" + "\x98\x0C\xA3\x3A\xD1\x45\xDC\x73" + "\x0A\x7E\x15\xAC\x20\xB7\x4E\xE5" + "\x59\xF0\x87\x1E\x92\x29\xC0\x34" + "\xCB\x62\xF9\x6D\x04\x9B\x0F\xA6" + "\x3D\xD4\x48\xDF\x76\x0D\x81\x18" + "\xAF\x23\xBA\x51\xE8\x5C\xF3\x8A" + "\x21\x95\x2C\xC3\x37\xCE\x65\xFC" + "\x70\x07\x9E\x12\xA9\x40\xD7\x4B" + "\xE2\x79\x10\x84\x1B\xB2\x26\xBD" + "\x54\xEB\x5F\xF6\x8D\x01\x98\x2F" + "\xC6\x3A\xD1\x68\xFF\x73\x0A\xA1" + "\x15\xAC\x43\xDA\x4E\xE5\x7C\x13" + "\x87\x1E\xB5\x29\xC0\x57\xEE\x62" + "\xF9\x90\x04\x9B\x32\xC9\x3D\xD4" + "\x6B\x02\x76\x0D\xA4\x18\xAF\x46" + "\xDD\x51\xE8\x7F\x16\x8A\x21\xB8" + "\x2C\xC3\x5A\xF1\x65\xFC\x93\x07" + "\x9E\x35\xCC\x40\xD7\x6E\x05\x79" + "\x10\xA7\x1B\xB2\x49\xE0\x54\xEB" + "\x82\x19\x8D\x24\xBB\x2F\xC6\x5D" + "\xF4\x68\xFF\x96\x0A\xA1\x38\xCF" + "\x43\xDA\x71\x08\x7C\x13\xAA\x1E" + "\xB5\x4C\xE3\x57\xEE\x85\x1C\x90" + "\x27\xBE\x32\xC9\x60\xF7\x6B\x02" + "\x99\x0D\xA4\x3B\xD2\x46\xDD\x74" + "\x0B\x7F\x16\xAD\x21\xB8\x4F\xE6" + "\x5A\xF1\x88\x1F\x93\x2A\xC1\x35" + "\xCC\x63\xFA\x6E\x05\x9C\x10\xA7" + "\x3E\xD5\x49\xE0\x77\x0E\x82\x19" + "\xB0\x24\xBB\x52\xE9\x5D\xF4\x8B" + "\x22\x96\x2D\xC4\x38\xCF\x66\xFD" + "\x71\x08\x9F\x13\xAA\x41\xD8\x4C" + "\xE3\x7A\x11\x85\x1C\xB3\x27\xBE" + "\x55\xEC\x60\xF7\x8E\x02\x99\x30" + "\xC7\x3B\xD2\x69\x00\x74\x0B\xA2" + "\x16\xAD\x44\xDB\x4F\xE6\x7D\x14" + "\x88\x1F\xB6\x2A\xC1\x58\xEF\x63" + "\xFA\x91\x05\x9C\x33\xCA\x3E\xD5" + "\x6C\x03\x77\x0E\xA5\x19\xB0\x47" + "\xDE\x52\xE9\x80\x17\x8B\x22\xB9" + "\x2D\xC4\x5B\xF2\x66\xFD\x94\x08" + "\x9F\x36\xCD\x41\xD8\x6F\x06\x7A" + "\x11\xA8\x1C\xB3\x4A\xE1\x55\xEC" + "\x83\x1A\x8E\x25\xBC\x30\xC7\x5E" + "\xF5\x69\x00\x97\x0B\xA2\x39\xD0" + "\x44\xDB\x72\x09\x7D\x14\xAB\x1F" + "\xB6\x4D\xE4\x58\xEF\x86\x1D\x91" + "\x28\xBF\x33\xCA\x61\xF8\x6C\x03" + "\x9A\x0E\xA5\x3C\xD3\x47\xDE\x75" + "\x0C\x80\x17\xAE\x22\xB9\x50\xE7" + "\x5B\xF2\x89\x20\x94\x2B\xC2\x36" + "\xCD\x64\xFB\x6F\x06\x9D\x11\xA8" + "\x3F\xD6\x4A\xE1\x78\x0F\x83\x1A" + "\xB1\x25\xBC\x53\xEA\x5E\xF5\x8C" + "\x00\x97\x2E\xC5\x39\xD0\x67\xFE" + "\x72\x09\xA0\x14\xAB\x42\xD9\x4D", + .ctext = "\xED\xCD\xDB\xB8\x68\xCE\xBD\xEA" + "\x9D\x9D\xCD\x9F\x4F\xFC\x4D\xB7" + "\xA5\xFF\x6F\x43\x0F\xBA\x32\x04" + "\xB3\xC2\xB9\x03\xAA\x91\x56\x29" + "\x0D\xD0\xFD\xC4\x65\xA5\x69\xB9" + "\xF1\xF6\xB1\xA5\xB2\x75\x4F\x8A" + "\x8D\x7D\x1B\x9B\xC7\x68\x72\xF8" + "\x01\x9B\x17\x0A\x29\xE7\x61\x28" + "\x7F\xA7\x50\xCA\x20\x2C\x96\x3B" + "\x6E\x5C\x5D\x3F\xB5\x7F\xF3\x2B" + "\x04\xEF\x9D\xD4\xCE\x41\x28\x8E" + "\x83\x54\xAE\x7C\x82\x46\x10\xC9" + "\xC4\x8A\x1E\x1F\x4C\xA9\xFC\xEC" + "\x3C\x8C\x30\xFC\x59\xD2\x54\xC4" + "\x6F\x50\xC6\xCA\x8C\x14\x5B\x9C" + "\x18\x56\x5B\xF8\x33\x0E\x4A\xDB" + "\xEC\xB5\x6E\x5B\x31\xC4\x0E\x98" + "\x9F\x32\xBA\xA2\x18\xCF\x55\x43" + "\xFE\x80\x8F\x60\xCF\x05\x30\x9B" + "\x70\x50\x1E\x9C\x08\x87\xE6\x20" + "\xD2\xF3\x27\xF8\x2A\x8D\x12\xB2" + "\xBC\x5F\xFE\x52\x52\xF6\x7F\xB6" + "\xB8\x30\x86\x3B\x0F\x94\x1E\x79" + "\x13\x94\x35\xA2\xB1\x35\x5B\x05" + "\x2A\x98\x6B\x96\x4C\xB1\x20\xBE" + "\xB6\x14\xC2\x06\xBF\xFD\x5F\x2A" + "\xF5\x33\xC8\x19\x45\x14\x44\x5D" + "\xFE\x94\x7B\xBB\x63\x13\x57\xC3" + "\x2A\x8F\x6C\x11\x2A\x07\xA7\x6A" + "\xBF\x20\xD3\x99\xC6\x00\x0B\xBF" + "\x83\x46\x25\x3A\xB0\xF6\xC5\xC8" + "\x00\xCA\xE5\x28\x4A\x7C\x95\x9C" + "\x7B\x43\xAB\xF9\xE4\xF8\x74\xAB" + "\xA7\xB8\x9C\x0F\x53\x7B\xB6\x74" + "\x60\x64\x0D\x1C\x80\xD1\x20\x9E" + "\xDC\x14\x27\x9B\xFC\xBD\x5C\x96" + "\xD2\x51\xDC\x96\xEE\xE5\xEA\x2B" + "\x02\x7C\xAA\x3C\xDC\x9D\x7B\x01" + "\x20\xC3\xE1\x0B\xDD\xAB\xF3\x1E" + "\x19\xA8\x84\x29\x5F\xCC\xC3\x5B" + "\xE4\x33\x59\xDC\x12\xEB\x2B\x4D" + "\x5B\x55\x23\xB7\x40\x31\xDE\xEE" + "\x18\xC9\x3C\x4D\xBC\xED\xE0\x42" + "\xAD\xDE\xA0\xA3\xC3\xFE\x44\xD3" + "\xE1\x9A\xDA\xAB\x32\xFC\x1A\xBF" + "\x63\xA9\xF0\x6A\x08\x46\xBD\x48" + "\x83\x06\xAB\x82\x99\x01\x16\x1A" + "\x03\x36\xC5\x59\x6B\xB8\x8C\x9F" + "\xC6\x51\x3D\xE5\x7F\xBF\xAB\xBC" + "\xC9\xA1\x88\x34\x5F\xA9\x7C\x3B" + "\x9F\x1B\x98\x2B\x4F\xFB\x9B\xF0" + "\xCD\xB6\x45\xB2\x29\x2E\x34\x23" + "\xA9\x97\xC0\x22\x8C\x42\x9B\x5F" + "\x40\xC8\xD7\x3D\x82\x9A\x6F\xAA" + "\x74\x83\x29\x05\xE8\xC4\x4D\x01" + "\xB5\xE5\x84\x3F\x7F\xD3\xE0\x99" + "\xDA\xE7\x6F\x30\xFD\xAA\x92\x30" + "\xA5\x46\x8B\xA2\xE6\x58\x62\x7C" + "\x2C\x35\x1B\x38\x85\x7D\xE8\xF3" + "\x87\x4F\xDA\xD8\x5F\xFC\xB6\x44" + "\xD0\xE3\x9B\x8B\xBF\xD6\xB8\xC4" + "\x73\xAE\x1D\x8B\x5B\x74\x8B\xCB" + "\xA4\xAD\xCF\x5D\xD4\x58\xC9\xCD" + "\xF7\x90\x68\xCF\xC9\x11\x52\x3E" + "\xE8\xA1\xA3\x78\x8B\xD0\xAC\x0A" + "\xD4\xC9\xA3\xA5\x55\x30\xC8\x3E" + "\xED\x28\x39\xE9\x63\xED\x41\x70" + "\x51\xE3\xC4\xA0\xFC\xD5\x43\xCB" + "\x4D\x65\xC8\xFD\x3A\x91\x8F\x60" + "\x8A\xA6\x6D\x9D\x3E\x01\x23\x4B" + "\x50\x47\xC9\xDC\x9B\xDE\x37\xC5" + "\xBF\x67\xB1\x6B\x78\x38\xD5\x7E" + "\xB6\xFF\x67\x83\x3B\x6E\xBE\x23" + "\x45\xFA\x1D\x69\x44\xFD\xC6\xB9" + "\xD0\x4A\x92\xD1\xBE\xF6\x4A\xB7" + "\xCA\xA8\xA2\x9E\x13\x87\x57\x92" + "\x64\x7C\x85\x0B\xB3\x29\x37\xD8" + "\xE6\xAA\xAF\xC4\x03\x67\xA3\xBF" + "\x2E\x45\x83\xB6\xD8\x54\x00\x89" + "\xF6\xBC\x3A\x7A\x88\x58\x51\xED" + "\xF4\x4E\x01\xA5\xC3\x2E\xD9\x42" + "\xBD\x6E\x0D\x0B\x21\xB0\x1A\xCC" + "\xA4\xD3\x3F\xDC\x9B\x81\xD8\xF1" + "\xEA\x7A\x6A\xB7\x07\xC9\x6D\x91" + "\x6D\x3A\xF5\x5F\xA6\xFF\x87\x1E" + "\x3F\xDD\xC0\x72\xEA\xAC\x08\x15" + "\x21\xE6\xC6\xB6\x0D\xD8\x51\x86" + "\x2A\x03\x73\xF7\x29\xD4\xC4\xE4" + "\x7F\x95\x10\xF7\xAB\x3F\x92\x23" + "\xD3\xCE\x9C\x2E\x46\x3B\x63\x43" + "\xBB\xC2\x82\x7A\x83\xD5\x55\xE2" + "\xE7\x9B\x2F\x92\xAF\xFD\x81\x56" + "\x79\xFD\x3E\xF9\x46\xE0\x25\xD4" + "\x38\xDE\xBC\x2C\xC4\x7A\x2A\x8F" + "\x94\x4F\xD0\xAD\x9B\x37\x18\xD4" + "\x0E\x4D\x0F\x02\x3A\xDC\x5A\xA2" + "\x39\x25\x55\x20\x5A\xA6\x02\x9F" + "\xE6\x77\x21\x77\xE5\x4B\x7B\x0B" + "\x30\xF8\x5F\x33\x0F\x49\xCD\xFF" + "\xF2\xE4\x35\xF9\xF0\x63\xC3\x7E" + "\xF1\xA6\x73\xB4\xDF\xE7\xBB\x78" + "\xFF\x21\xA9\xF3\xF3\xCF\x5D\xBA" + "\xED\x87\x98\xAC\xFE\x48\x97\x6D" + "\xA6\x7F\x69\x31\xB1\xC4\xFF\x14" + "\xC6\x76\xD4\x10\xDD\xF6\x49\x2C" + "\x9C\xC8\x6D\x76\xC0\x8F\x5F\x55" + "\x2F\x3C\x8A\x30\xAA\xC3\x16\x55" + "\xC6\xFC\x8D\x8B\xB9\xE5\x80\x6C" + "\xC8\x7E\xBD\x65\x58\x36\xD5\xBC" + "\xF0\x33\x52\x29\x70\xF9\x5C\xE9" + "\xAC\x1F\xB5\x73\x56\x66\x54\xAF" + "\x1B\x8F\x7D\xED\xAB\x03\xCE\xE3" + "\xAE\x47\xB6\x69\x86\xE9\x01\x31" + "\x83\x18\x3D\xF4\x74\x7B\xF9\x42" + "\x4C\xFD\x75\x4A\x6D\xF0\x03\xA6" + "\x2B\x20\x63\xDA\x49\x65\x5E\x8B" + "\xC0\x19\xE3\x8D\xD9\xF3\xB0\x34" + "\xD3\x52\xFC\x68\x00\x43\x1B\x37" + "\x31\x93\x51\x1C\x63\x97\x70\xB0" + "\x99\x78\x83\x13\xFD\xCF\x53\x81" + "\x36\x46\xB5\x42\x52\x2F\x32\xEB" + "\x4A\x3D\xF1\x8F\x1C\x54\x2E\xFC" + "\x41\x75\x5A\x8C\x8E\x6F\xE7\x1A" + "\xAE\xEF\x3E\x82\x12\x0B\x74\x72" + "\xF8\xB2\xAA\x7A\xD6\xFF\xFA\x55" + "\x33\x1A\xBB\xD3\xA2\x7E\x97\x66", + .len = 1008, + .also_non_np = 1, + .np = 3, + .tap = { 1008 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec camellia_cbc_tv_template[] = { + { + .key = "\x06\xa9\x21\x40\x36\xb8\xa1\x5b" + "\x51\x2e\x03\xd5\x34\x12\x00\x06", + .klen = 16, + .iv = "\x3d\xaf\xba\x42\x9d\x9e\xb4\x30" + "\xb4\x22\xda\x80\x2c\x9f\xac\x41", + .ptext = "Single block msg", + .ctext = "\xea\x32\x12\x76\x3b\x50\x10\xe7" + "\x18\xf6\xfd\x5d\xf6\x8f\x13\x51", + .len = 16, + }, { + .key = "\xc2\x86\x69\x6d\x88\x7c\x9a\xa0" + "\x61\x1b\xbb\x3e\x20\x25\xa4\x5a", + .klen = 16, + .iv = "\x56\x2e\x17\x99\x6d\x09\x3d\x28" + "\xdd\xb3\xba\x69\x5a\x2e\x6f\x58", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" + "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" + "\x10\x11\x12\x13\x14\x15\x16\x17" + "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f", + .ctext = "\xa5\xdf\x6e\x50\xda\x70\x6c\x01" + "\x4a\xab\xf3\xf2\xd6\xfc\x6c\xfd" + "\x19\xb4\x3e\x57\x1c\x02\x5e\xa0" + "\x15\x78\xe0\x5e\xf2\xcb\x87\x16", + .len = 32, + }, { /* Generated with Crypto++ */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .klen = 32, + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -32320,148 +28261,549 @@ static const struct cipher_testvec camellia_ctr_dec_tv_template[] = { "\x3F\xD6\x4A\xE1\x78\x0F\x83\x1A" "\xB1\x25\xBC\x53\xEA\x5E\xF5\x8C" "\x00\x97\x2E\xC5\x39\xD0\x67\xFE" - "\x72\x09\xA0\x14\xAB\x42\xD9\x4D" - "\xE4\x7B\x12", - .rlen = 1011, + "\x72\x09\xA0\x14\xAB\x42\xD9\x4D", + .ctext = "\xCD\x3E\x2A\x3B\x3E\x94\xC5\x77" + "\xBA\xBB\x5B\xB1\xDE\x7B\xA4\x40" + "\x88\x39\xE3\xFD\x94\x4B\x25\x58" + "\xE1\x4B\xC4\x18\x7A\xFD\x17\x2B" + "\xB9\xF9\xC2\x27\x6A\xB6\x31\x27" + "\xA6\xAD\xEF\xE5\x5D\xE4\x02\x01" + "\x56\x2E\x10\xC2\x2C\xFF\xC6\x83" + "\xB5\xDC\x4F\x63\xAD\x0E\x63\x5E" + "\x56\xC8\x18\x3D\x79\x86\x97\xEF" + "\x57\x0E\x63\xA1\xC1\x41\x48\xB8" + "\x98\xB7\x51\x6D\x18\xF6\x19\x82" + "\x37\x49\x88\xA4\xEF\x91\x21\x47" + "\x03\x28\xEA\x42\xF4\xFB\x7A\x58" + "\x28\x90\x77\x46\xD8\xD2\x35\x16" + "\x44\xA9\x9E\x49\x52\x2A\xE4\x16" + "\x5D\xF7\x65\xEB\x0F\xC9\x29\xE6" + "\xCF\x76\x91\x89\x8A\x94\x39\xFA" + "\x6B\x5F\x63\x53\x74\x43\x91\xF5" + "\x3F\xBC\x88\x53\xB2\x1A\x02\x3F" + "\x9D\x32\x84\xEB\x56\x28\xD6\x06" + "\xD5\xB2\x20\xA9\xFC\xC3\x76\x62" + "\x32\xCC\x86\xC8\x36\x67\x5E\x7E" + "\xA4\xAA\x15\x63\x6B\xA9\x86\xAF" + "\x1A\x52\x82\x36\x5F\xF4\x3F\x7A" + "\x9B\x78\x62\x3B\x02\x28\x60\xB3" + "\xBA\x82\xB1\xDD\xC9\x60\x8F\x47" + "\xF1\x6B\xFE\xE5\x39\x34\xA0\x28" + "\xA4\xB3\xC9\x7E\xED\x28\x8D\x70" + "\xB2\x1D\xFD\xC6\x00\xCF\x1A\x94" + "\x28\xF8\xC1\x34\xB7\x58\xA5\x6C" + "\x1A\x9D\xE4\xE4\xF6\xB9\xB4\xB0" + "\x5D\x51\x54\x9A\x53\xA0\xF9\x32" + "\xBD\x31\x54\x14\x7B\x33\xEE\x17" + "\xD3\xC7\x1F\x48\xBF\x0B\x22\xA2" + "\x7D\x0C\xDF\xD0\x2E\x98\xFA\xD2" + "\xFA\xCF\x24\x1D\x99\x9B\xD0\x7E" + "\xF4\x4F\x88\xFF\x45\x99\x4A\xF4" + "\xF2\x0A\x5B\x3B\x21\xAB\x92\xAE" + "\x40\x78\x91\x95\xC4\x2F\xA3\xE8" + "\x18\xC7\x07\xA6\xC8\xC0\x66\x33" + "\x35\xC0\xB4\xA0\xF8\xEE\x1E\xF3" + "\x40\xF5\x40\x54\xF1\x84\x8C\xEA" + "\x27\x38\x1F\xF8\x77\xC7\xDF\xD8" + "\x1D\xE2\xD9\x59\x40\x4F\x59\xD4" + "\xF8\x17\x99\x8D\x58\x2D\x72\x44" + "\x9D\x1D\x91\x64\xD6\x3F\x0A\x82" + "\xC7\x57\x3D\xEF\xD3\x41\xFA\xA7" + "\x68\xA3\xB8\xA5\x93\x74\x2E\x85" + "\x4C\x9D\x69\x59\xCE\x15\xAE\xBF" + "\x9C\x8F\x14\x64\x5D\x7F\xCF\x0B" + "\xCE\x43\x5D\x28\xC0\x2F\xFB\x18" + "\x79\x9A\xFC\x43\x16\x7C\x6B\x7B" + "\x38\xB8\x48\x36\x66\x4E\x20\x43" + "\xBA\x76\x13\x9A\xC3\xF2\xEB\x52" + "\xD7\xDC\xB2\x67\x63\x14\x25\xCD" + "\xB1\x13\x4B\xDE\x8C\x59\x21\x84" + "\x81\x8D\x97\x23\x45\x33\x7C\xF3" + "\xC5\xBC\x79\x95\xAA\x84\x68\x31" + "\x2D\x1A\x68\xFE\xEC\x92\x94\xDA" + "\x94\x2A\x6F\xD6\xFE\xE5\x76\x97" + "\xF4\x6E\xEE\xCB\x2B\x95\x4E\x36" + "\x5F\x74\x8C\x86\x5B\x71\xD0\x20" + "\x78\x1A\x7F\x18\x8C\xD9\xCD\xF5" + "\x21\x41\x56\x72\x13\xE1\x86\x07" + "\x07\x26\xF3\x4F\x7B\xEA\xB5\x18" + "\xFE\x94\x2D\x9F\xE0\x72\x18\x65" + "\xB2\xA5\x63\x48\xB4\x13\x22\xF7" + "\x25\xF1\x80\xA8\x7F\x54\x86\x7B" + "\x39\xAE\x95\x0C\x09\x32\x22\x2D" + "\x4D\x73\x39\x0C\x09\x2C\x7C\x10" + "\xD0\x4B\x53\xF6\x90\xC5\x99\x2F" + "\x15\xE1\x7F\xC6\xC5\x7A\x52\x14" + "\x65\xEE\x93\x54\xD0\x66\x15\x3C" + "\x4C\x68\xFD\x64\x0F\xF9\x10\x39" + "\x46\x7A\xDD\x97\x20\xEE\xC7\xD2" + "\x98\x4A\xB6\xE6\xF5\xA8\x1F\x4F" + "\xDB\xAB\x6D\xD5\x9B\x34\x16\x97" + "\x2F\x64\xE5\x37\xEF\x0E\xA1\xE9" + "\xBE\x31\x31\x96\x8B\x40\x18\x75" + "\x11\x75\x14\x32\xA5\x2D\x1B\x6B" + "\xDB\x59\xEB\xFA\x3D\x8E\x7C\xC4" + "\xDE\x68\xC8\x9F\xC9\x99\xE3\xC6" + "\x71\xB0\x12\x57\x89\x0D\xC0\x2B" + "\x9F\x12\x6A\x04\x67\xF1\x95\x31" + "\x59\xFD\x84\x95\x2C\x9C\x5B\xEC" + "\x09\xB0\x43\x96\x4A\x64\x80\x40" + "\xB9\x72\x19\xDD\x70\x42\xFA\xB1" + "\x4A\x2C\x0C\x0A\x60\x6E\xE3\x7C" + "\x37\x5A\xBE\xA4\x62\xCF\x29\xAB" + "\x7F\x4D\xA6\xB3\xE2\xB6\x64\xC6" + "\x33\x0B\xF3\xD5\x01\x38\x74\xA4" + "\x67\x1E\x75\x68\xC3\xAD\x76\xE9" + "\xE9\xBC\xF0\xEB\xD8\xFD\x31\x8A" + "\x5F\xC9\x18\x94\x4B\x86\x66\xFC" + "\xBD\x0B\x3D\xB3\x9F\xFA\x1F\xD9" + "\x78\xC4\xE3\x24\x1C\x67\xA2\xF8" + "\x43\xBC\x76\x75\xBF\x6C\x05\xB3" + "\x32\xE8\x7C\x80\xDB\xC7\xB6\x61" + "\x1A\x3E\x2B\xA7\x25\xED\x8F\xA0" + "\x00\x4B\xF8\x90\xCA\xD8\xFB\x12" + "\xAC\x1F\x18\xE9\xD2\x5E\xA2\x8E" + "\xE4\x84\x6B\x9D\xEB\x1E\x6B\xA3" + "\x7B\xDC\xCE\x15\x97\x27\xB2\x65" + "\xBC\x0E\x47\xAB\x55\x13\x53\xAB" + "\x0E\x34\x55\x02\x5F\x27\xC5\x89" + "\xDF\xC5\x70\xC4\xDD\x76\x82\xEE" + "\x68\xA6\x09\xB0\xE5\x5E\xF1\x0C" + "\xE3\xF3\x09\x9B\xFE\x65\x4B\xB8" + "\x30\xEC\xD5\x7C\x6A\xEC\x1D\xD2" + "\x93\xB7\xA1\x1A\x02\xD4\xC0\xD6" + "\x8D\x4D\x83\x9A\xED\x29\x4E\x14" + "\x86\xD5\x3C\x1A\xD5\xB9\x0A\x6A" + "\x72\x22\xD5\x92\x38\xF1\xA1\x86" + "\xB2\x41\x51\xCA\x4E\xAB\x8F\xD3" + "\x80\x56\xC3\xD7\x65\xE1\xB3\x86" + "\xCB\xCE\x98\xA1\xD4\x59\x1C\x06" + "\x01\xED\xF8\x29\x91\x19\x5C\x9A" + "\xEE\x28\x1B\x48\xD7\x32\xEF\x9F" + "\x6C\x2B\x66\x4E\x78\xD5\x8B\x72" + "\x80\xE7\x29\xDC\x23\x55\x98\x54" + "\xB1\xFF\x3E\x95\x56\xA8\x78\x78" + "\xEF\xC4\xA5\x11\x2D\x2B\xD8\x93" + "\x30\x6E\x7E\x51\xBB\x42\x5F\x03" + "\x43\x94\x23\x7E\xEE\xF0\xA5\x79" + "\x55\x01\xD4\x58\xB2\xF2\x85\x49" + "\x70\xC5\xB9\x0B\x3B\x7A\x6E\x6C", + .len = 1008, .also_non_np = 1, - .np = 2, - .tap = { 1011 - 16, 16 }, - }, { /* Generated with Crypto++ */ + .np = 3, + .tap = { 1008 - 20, 4, 16 }, + }, +}; + +static const struct cipher_testvec camellia_ctr_tv_template[] = { + { /* Generated with Crypto++ */ .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" "\x78\xBE\x9B\x78\x55\x32\x0F\x55", .klen = 32, - .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" - "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", - .input = "\x85\x79\x6C\x8B\x2B\x6D\x14\xF9" - "\xA6\x83\xB6\x80\x5B\x3A\xF3\x7E" - "\x30\x29\xEB\x1F\xDC\x19\x5F\xEB" - "\xF7\xC4\x27\x04\x51\x87\xD7\x6F" - "\xB8\x4E\x07\xFB\xAC\x3B\x08\xB4" - "\x4D\xCB\xE8\xE1\x71\x7D\x4F\x48" - "\xCD\x81\x64\xA5\xC4\x07\x1A\x9A" - "\x4B\x62\x90\x0E\xC8\xB3\x2B\x6B" - "\x8F\x9C\x6E\x72\x4B\xBA\xEF\x07" - "\x2C\x56\x07\x5E\x37\x30\x60\xA9" - "\xE3\xEF\xD6\x69\xE1\xA1\x77\x64" - "\x93\x75\x7A\xB7\x7A\x3B\xE9\x43" - "\x23\x35\x95\x91\x80\x8A\xC7\xCF" - "\xC3\xD5\xBF\xE7\xFE\x4C\x06\x6B" - "\x05\x19\x48\xE2\x62\xBA\x4F\xF2" - "\xFB\xEE\xE4\xCB\x79\x9D\xA3\x10" - "\x1D\x29\x8C\x1D\x7A\x88\x5A\xDD" - "\x4E\xB6\x18\xAA\xCD\xE6\x33\x96" - "\xD9\x0F\x90\x5A\x78\x76\x4D\x77" - "\x3C\x20\x89\x3B\xA3\xF9\x07\xFD" - "\xE4\xE8\x20\x2D\x15\x0A\x63\x49" - "\xF5\x4F\x89\xD8\xDE\xA1\x28\x78" - "\x28\x07\x09\x1B\x03\x94\x1D\x4B" - "\x82\x28\x1E\x1D\x95\xBA\xAC\x85" - "\x71\x6E\x3C\x18\x4B\x77\x74\x79" - "\xBF\x67\x0A\x53\x3C\x94\xD9\x60" - "\xE9\x6D\x40\x34\xA0\x2A\x53\x5D" - "\x27\xD5\x47\xF9\xC3\x4B\x27\x29" - "\xE4\x76\x9C\x3F\xA7\x1C\x87\xFC" - "\x6E\x0F\xCF\x9B\x60\xF0\xF0\x8B" - "\x70\x1C\x84\x81\x72\x4D\xB4\x98" - "\x23\x62\xE7\x6A\x2B\xFC\xA5\xB2" - "\xFF\xF5\x71\x07\xCD\x90\x23\x13" - "\x19\xD7\x79\x36\x6C\x9D\x55\x8B" - "\x93\x78\x86\x05\x69\x46\xD0\xC5" - "\x39\x09\xEB\x79\xEF\xFA\x9F\xAE" - "\xF3\xD5\x44\xC3\xFD\x86\xD2\x7C" - "\x83\x4B\xD8\x75\x9C\x18\x04\x7B" - "\x73\xAD\x72\xA4\xF6\xAB\xCF\x4B" - "\xCC\x01\x45\x90\xA6\x43\x05\x0C" - "\x6C\x4F\x62\x77\x57\x97\x9F\xEE" - "\x75\xA7\x3C\x38\xD1\x0F\x3D\x0E" - "\x2C\x43\x98\xFB\x13\x65\x73\xE4" - "\x3C\x1E\xD6\x90\x08\xF7\xE0\x99" - "\x3B\xF1\x9D\x6C\x48\xA9\x0E\x32" - "\x17\xC2\xCC\x20\xA1\x19\x26\xAA" - "\xE0\x75\x2F\xFB\x54\x66\x0A\xDF" - "\xB5\xF2\x1F\xC1\x34\x3C\x30\x56" - "\xE8\xDC\xF7\x92\x6B\xBF\x17\x24" - "\xEC\x94\xB5\x3B\xD6\xCE\xA2\x54" - "\x10\x7F\x50\xDE\x69\x77\xD5\x37" - "\xFE\x9C\x10\x83\xC5\xEB\xC9\x53" - "\xB7\xF3\xC4\x20\xAF\x0A\x7E\x57" - "\x3A\xE6\x75\xFE\x89\x00\x6E\x48" - "\xFB\x99\x17\x2C\xF6\x64\x40\x95" - "\x5E\xDC\x7A\xA6\x70\xC7\xF4\xDD" - "\x52\x05\x24\x34\xF9\x0E\xC8\x64" - "\x6D\xE2\xD8\x80\x53\x31\x4C\xFE" - "\xB4\x3A\x5F\x19\xCF\x42\x1B\x22" - "\x0B\x2D\x7B\xF1\xC5\x43\xF7\x5E" - "\x12\xA8\x01\x64\x16\x0B\x26\x5A" - "\x0C\x95\x0F\x40\xC5\x5A\x06\x7C" - "\xCF\xF5\xD5\xB7\x7A\x34\x23\xB6" - "\xAA\x9E\xA8\x98\xA2\xF8\x3D\xD3" - "\x3F\x23\x69\x63\x56\x96\x45\xD6" - "\x74\x23\x1D\x5C\x63\xCC\xD8\x78" - "\x16\xE2\x9C\xD2\x80\x02\xF2\x28" - "\x69\x2F\xC4\xA8\x15\x15\x24\x3B" - "\xCB\xF0\x14\xE4\x62\xC8\xF3\xD1" - "\x03\x58\x1B\x33\x77\x74\x1F\xB4" - "\x07\x86\xF2\x21\xB7\x41\xAE\xBF" - "\x25\xC2\xFF\x51\xEF\xEA\xCE\xC4" - "\x5F\xD9\xB8\x18\x6A\xF0\x0F\x0D" - "\xF8\x04\xBB\x6D\x62\x33\x87\x26" - "\x4F\x2F\x14\x6E\xDC\xDB\x66\x09" - "\x2A\xEF\x7D\x84\x10\xAC\x82\x5E" - "\xD2\xE4\xAD\x74\x7A\x6D\xCC\x3A" - "\x7B\x62\xD8\xD6\x07\x2D\xF7\xDF" - "\x9B\xB3\x82\xCF\x9C\x1D\x76\x5C" - "\xAC\x7B\xD4\x9B\x45\xA1\x64\x11" - "\x66\xF1\xA7\x0B\xF9\xDD\x00\xDD" - "\xA4\x45\x3D\x3E\x03\xC9\x2E\xCB" - "\xC3\x14\x84\x72\xFD\x41\xDC\xBD" - "\x75\xBE\xA8\xE5\x16\x48\x64\x39" - "\xCA\xF3\xE6\xDC\x25\x24\xF1\x6D" - "\xB2\x8D\xC5\x38\x54\xD3\x5D\x6D" - "\x0B\x29\x10\x15\x0E\x13\x3B\xAC" - "\x7E\xCC\x9E\x3E\x18\x48\xA6\x02" - "\xEF\x03\xB2\x2E\xE3\xD2\x70\x21" - "\xB4\x19\x26\xBE\x3A\x3D\x05\xE0" - "\xF8\x09\xAF\xE4\x31\x26\x92\x2F" - "\x8F\x55\xAC\xED\x0B\xB2\xA5\x34" - "\xBE\x50\xB1\x02\x22\x96\xE3\x40" - "\x7B\x70\x50\x6E\x3B\xD5\xE5\xA0" - "\x8E\xA2\xAD\x14\x60\x5C\x7A\x2B" - "\x3D\x1B\x7F\xC1\xC0\x2C\x56\x36" - "\xD2\x0A\x32\x06\x97\x34\xB9\xF4" - "\x6F\x9F\x7E\x80\xD0\x9D\xF7\x6A" - "\x21\xC1\xA2\x6A\xB1\x96\x5B\x4D" - "\x7A\x15\x6C\xC4\x4E\xB8\xE0\x9E" - "\x6C\x50\xF3\x9C\xC9\xB5\x23\xB7" - "\xF1\xD4\x29\x4A\x23\xC4\xAD\x1E" - "\x2C\x07\xD2\x43\x5F\x57\x93\xCA" - "\x85\xF9\x9F\xAD\x4C\xF1\xE4\xB1" - "\x1A\x8E\x28\xA4\xB6\x52\x77\x7E" - "\x68\xC6\x47\xB9\x76\xCC\x65\x5F" - "\x0B\xF9\x67\x93\xD8\x0E\x9A\x37" - "\x5F\x41\xED\x64\x6C\xAD\x5F\xED" - "\x3F\x8D\xFB\x8E\x1E\xA0\xE4\x1F" - "\xC2\xC7\xED\x18\x43\xE1\x20\x86" - "\x5D\xBC\x30\x70\x22\xA1\xDC\x53" - "\x10\x3A\x8D\x47\x82\xCD\x7F\x59" - "\x03\x2D\x6D\xF5\xE7\x79\xD4\x07" - "\x68\x2A\xA5\x42\x19\x4D\xAF\xF5" - "\xED\x47\x83\xBC\x5F\x62\x84\xDA" - "\xDA\x41\xFF\xB0\x1D\x64\xA3\xC8" - "\xBD\x4E\xE0\xB8\x7F\xEE\x55\x0A" - "\x4E\x61\xB2\x51\xF6\x9C\x95\xF6" - "\x92\xBB\xF6\xC5\xF0\x09\x86\xDE" - "\x37\x9E\x29\xF9\x2A\x18\x73\x0D" - "\xDC\x7E\x6B\x7B\x1B\x43\x8C\xEA" - "\x13\xC8\x1A\x47\x0A\x2D\x6D\x56" - "\xCD\xD2\xE7\x53\x1A\xAB\x1C\x3C" - "\xC5\x9B\x03\x70\x29\x2A\x49\x09" - "\x67\xA1\xEA\xD6\x3A\x5B\xBF\x71" - "\x1D\x48\x64\x6C\xFB\xC0\x9E\x36", - .ilen = 1008, - .result = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" + "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" + "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" + "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" + "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" + "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" + "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" + "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" + "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" + "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" + "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" + "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" + "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" + "\x29\xC0\x57\xEE\x62\xF9\x90\x04" + "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" + "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" + "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" + "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" + "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" + "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" + "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" + "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" + "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" + "\x57\xEE\x85\x1C\x90\x27\xBE\x32" + "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" + "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" + "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" + "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" + "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" + "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" + "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" + "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" + "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" + "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" + "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" + "\x69\x00\x74\x0B\xA2\x16\xAD\x44" + "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" + "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" + "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" + "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" + "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" + "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" + "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" + "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" + "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" + "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" + "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" + "\x58\xEF\x86\x1D\x91\x28\xBF\x33" + "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" + "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" + "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" + "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" + "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" + "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" + "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" + "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" + "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" + "\x86\x1D\xB4\x28\xBF\x56\xED\x61" + "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" + "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7", + .ctext = "\xF3\x06\x3A\x84\xCD\xBA\x8E\x11" + "\xB7\x74\x6F\x5C\x97\xFB\x36\xFE" + "\xDE\x71\x58\xD4\x15\xD1\xC1\xA4" + "\xC9\x28\x74\xA6\x6B\xC7\x95\xA6" + "\x6C\x77\xF7\x2F\xDF\xC7\xBB\x85" + "\x60\xFC\xE8\x94\xE8\xB5\x09\x2C" + "\x1E\x43\xEF\x6C\xE9\x98\xC5\xA0" + "\x7B\x13\xE5\x7F\xF8\x49\x9A\x8C" + "\xE6\x7B\x08\xC3\x32\x66\x55\x4E" + "\xA5\x44\x1D\x2C\x18\xC7\x29\x1F" + "\x61\x28\x4A\xE3\xCD\xE5\x47\xB2" + "\x82\x2F\x66\x83\x91\x51\xAE\xD7" + "\x1C\x91\x3C\x57\xE3\x1D\x5A\xC9" + "\xFD\xC5\x58\x58\xEF\xCC\x33\xC9" + "\x0F\xEA\x26\x32\xD1\x15\x19\x2D" + "\x25\xB4\x7F\xB0\xDF\xFB\x88\x60" + "\x4E\x4D\x06\x7D\xCC\x1F\xED\x3B" + "\x68\x84\xD5\xB3\x1B\xE7\xB9\xA1" + "\x68\x8B\x2C\x1A\x44\xDA\x63\xD3" + "\x29\xE9\x59\x32\x1F\x30\x1C\x43" + "\xEA\x3A\xA3\x6B\x54\x3C\xAA\x11" + "\xAD\x38\x20\xC9\xB9\x8A\x64\x66" + "\x5A\x07\x49\xDF\xA1\x9C\xF9\x76" + "\x36\x65\xB6\x81\x8F\x76\x09\xE5" + "\xEB\xD1\x29\xA4\xE4\xF4\x4C\xCD" + "\xAF\xFC\xB9\x16\xD9\xC3\x73\x6A" + "\x33\x12\xF8\x7E\xBC\xCC\x7D\x80" + "\xBF\x3C\x25\x06\x13\x84\xFA\x35" + "\xF7\x40\xFA\xA1\x44\x13\x70\xD8" + "\x01\xF9\x85\x15\x63\xEC\x7D\xB9" + "\x02\xD8\xBA\x41\x6C\x92\x68\x66" + "\x95\xDD\xD6\x42\xE7\xBB\xE1\xFD" + "\x28\x3E\x94\xB6\xBD\xA7\xBF\x47" + "\x58\x8D\xFF\x19\x30\x75\x0D\x48" + "\x94\xE9\xA6\xCD\xB3\x8E\x1E\xCD" + "\x59\xBC\x1A\xAC\x3C\x4F\xA9\xEB" + "\xF4\xA7\xE4\x75\x4A\x18\x40\xC9" + "\x1E\xEC\x06\x9C\x28\x4B\xF7\x2B" + "\xE2\xEF\xD6\x42\x2E\xBB\xFC\x0A" + "\x79\xA2\x99\x28\x93\x1B\x00\x57" + "\x35\x1E\x1A\x93\x90\xA4\x68\x95" + "\x5E\x57\x40\xD5\xA9\xAA\x19\x48" + "\xEC\xFF\x76\x77\xDC\x78\x89\x76" + "\xE5\x3B\x00\xEC\x58\x4D\xD1\xE3" + "\xC8\x6C\x2C\x45\x5E\x5F\xD9\x4E" + "\x71\xA5\x36\x6D\x03\xF1\xC7\xD5" + "\xF3\x63\xC0\xD8\xCB\x2B\xF1\xA8" + "\xB9\x2B\xE6\x0B\xB9\x65\x78\xA0" + "\xC4\x46\xE6\x9B\x8B\x43\x2D\xAB" + "\x70\xA6\xE0\x59\x1E\xAC\x9D\xE0" + "\x76\x44\x45\xF3\x24\x11\x57\x98" + "\x9A\x86\xB4\x12\x80\x28\x86\x20" + "\x23\x9D\x2D\xE9\x38\x32\xB1\xE1" + "\xCF\x0A\x23\x73\x7D\xC5\x80\x3D" + "\x9F\x6D\xA0\xD0\xEE\x93\x8A\x79" + "\x3A\xDD\x1D\xBB\x9E\x26\x5D\x01" + "\x44\xD0\xD4\x4E\xC3\xF1\xE4\x38" + "\x09\x62\x0A\x1A\x4E\xD2\x63\x0F" + "\x6E\x3E\xD2\xA4\x3A\xF4\xF3\xFF" + "\x7E\x42\xEC\xB6\x6F\x4D\x6B\x48" + "\xE6\xA6\x50\x80\x78\x9E\xF1\xB0" + "\x4D\xB2\x0D\x3D\xFC\x40\x25\x4D", + .len = 496, + }, { /* Generated with Crypto++ */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .klen = 32, + .iv = "\xE2\x24\x89\xEE\x53\xB8\x1D\x5F" + "\xC4\x29\x8E\xF3\x35\x9A\xFF\x64", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" + "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" + "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" + "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" + "\x1E\x92\x29\xC0\x34\xCB\x62\xF9" + "\x6D\x04\x9B\x0F\xA6\x3D\xD4\x48" + "\xDF\x76\x0D\x81\x18\xAF\x23\xBA" + "\x51\xE8\x5C\xF3\x8A\x21\x95\x2C" + "\xC3\x37\xCE\x65\xFC\x70\x07\x9E" + "\x12\xA9\x40\xD7\x4B\xE2\x79\x10" + "\x84\x1B\xB2\x26\xBD\x54\xEB\x5F" + "\xF6\x8D\x01\x98\x2F\xC6\x3A\xD1" + "\x68\xFF\x73\x0A\xA1\x15\xAC\x43" + "\xDA\x4E\xE5\x7C\x13\x87\x1E\xB5" + "\x29\xC0\x57\xEE\x62\xF9\x90\x04" + "\x9B\x32\xC9\x3D\xD4\x6B\x02\x76" + "\x0D\xA4\x18\xAF\x46\xDD\x51\xE8" + "\x7F\x16\x8A\x21\xB8\x2C\xC3\x5A" + "\xF1\x65\xFC\x93\x07\x9E\x35\xCC" + "\x40\xD7\x6E\x05\x79\x10\xA7\x1B" + "\xB2\x49\xE0\x54\xEB\x82\x19\x8D" + "\x24\xBB\x2F\xC6\x5D\xF4\x68\xFF" + "\x96\x0A\xA1\x38\xCF\x43\xDA\x71" + "\x08\x7C\x13\xAA\x1E\xB5\x4C\xE3" + "\x57\xEE\x85\x1C\x90\x27\xBE\x32" + "\xC9\x60\xF7\x6B\x02\x99\x0D\xA4" + "\x3B\xD2\x46\xDD\x74\x0B\x7F\x16" + "\xAD\x21\xB8\x4F\xE6\x5A\xF1\x88" + "\x1F\x93\x2A\xC1\x35\xCC\x63\xFA" + "\x6E\x05\x9C\x10\xA7\x3E\xD5\x49" + "\xE0\x77\x0E\x82\x19\xB0\x24\xBB" + "\x52\xE9\x5D\xF4\x8B\x22\x96\x2D" + "\xC4\x38\xCF\x66\xFD\x71\x08\x9F" + "\x13\xAA\x41\xD8\x4C\xE3\x7A\x11" + "\x85\x1C\xB3\x27\xBE\x55\xEC\x60" + "\xF7\x8E\x02\x99\x30\xC7\x3B\xD2" + "\x69\x00\x74\x0B\xA2\x16\xAD\x44" + "\xDB\x4F\xE6\x7D\x14\x88\x1F\xB6" + "\x2A\xC1\x58\xEF\x63\xFA\x91\x05" + "\x9C\x33\xCA\x3E\xD5\x6C\x03\x77" + "\x0E\xA5\x19\xB0\x47\xDE\x52\xE9" + "\x80\x17\x8B\x22\xB9\x2D\xC4\x5B" + "\xF2\x66\xFD\x94\x08\x9F\x36\xCD" + "\x41\xD8\x6F\x06\x7A\x11\xA8\x1C" + "\xB3\x4A\xE1\x55\xEC\x83\x1A\x8E" + "\x25\xBC\x30\xC7\x5E\xF5\x69\x00" + "\x97\x0B\xA2\x39\xD0\x44\xDB\x72" + "\x09\x7D\x14\xAB\x1F\xB6\x4D\xE4" + "\x58\xEF\x86\x1D\x91\x28\xBF\x33" + "\xCA\x61\xF8\x6C\x03\x9A\x0E\xA5" + "\x3C\xD3\x47\xDE\x75\x0C\x80\x17" + "\xAE\x22\xB9\x50\xE7\x5B\xF2\x89" + "\x20\x94\x2B\xC2\x36\xCD\x64\xFB" + "\x6F\x06\x9D\x11\xA8\x3F\xD6\x4A" + "\xE1\x78\x0F\x83\x1A\xB1\x25\xBC" + "\x53\xEA\x5E\xF5\x8C\x00\x97\x2E" + "\xC5\x39\xD0\x67\xFE\x72\x09\xA0" + "\x14\xAB\x42\xD9\x4D\xE4\x7B\x12" + "\x86\x1D\xB4\x28\xBF\x56\xED\x61" + "\xF8\x8F\x03\x9A\x31\xC8\x3C\xD3" + "\x6A\x01\x75\x0C\xA3\x17\xAE\x45" + "\xDC\x50\xE7\x7E\x15\x89\x20\xB7" + "\x2B\xC2\x59\xF0\x64\xFB\x92\x06" + "\x9D\x34\xCB\x3F\xD6\x6D\x04\x78" + "\x0F\xA6\x1A\xB1\x48\xDF\x53\xEA" + "\x81\x18\x8C\x23\xBA\x2E\xC5\x5C" + "\xF3\x67\xFE\x95\x09\xA0\x37\xCE" + "\x42\xD9\x70\x07\x7B\x12\xA9\x1D" + "\xB4\x4B\xE2\x56\xED\x84\x1B\x8F" + "\x26\xBD\x31\xC8\x5F\xF6\x6A\x01" + "\x98\x0C\xA3\x3A\xD1\x45\xDC\x73" + "\x0A\x7E\x15\xAC\x20\xB7\x4E\xE5" + "\x59\xF0\x87\x1E\x92\x29\xC0\x34" + "\xCB\x62\xF9\x6D\x04\x9B\x0F\xA6" + "\x3D\xD4\x48\xDF\x76\x0D\x81\x18" + "\xAF\x23\xBA\x51\xE8\x5C\xF3\x8A" + "\x21\x95\x2C\xC3\x37\xCE\x65\xFC" + "\x70\x07\x9E\x12\xA9\x40\xD7\x4B" + "\xE2\x79\x10\x84\x1B\xB2\x26\xBD" + "\x54\xEB\x5F\xF6\x8D\x01\x98\x2F" + "\xC6\x3A\xD1\x68\xFF\x73\x0A\xA1" + "\x15\xAC\x43\xDA\x4E\xE5\x7C\x13" + "\x87\x1E\xB5\x29\xC0\x57\xEE\x62" + "\xF9\x90\x04\x9B\x32\xC9\x3D\xD4" + "\x6B\x02\x76\x0D\xA4\x18\xAF\x46" + "\xDD\x51\xE8\x7F\x16\x8A\x21\xB8" + "\x2C\xC3\x5A\xF1\x65\xFC\x93\x07" + "\x9E\x35\xCC\x40\xD7\x6E\x05\x79" + "\x10\xA7\x1B\xB2\x49\xE0\x54\xEB" + "\x82\x19\x8D\x24\xBB\x2F\xC6\x5D" + "\xF4\x68\xFF\x96\x0A\xA1\x38\xCF" + "\x43\xDA\x71\x08\x7C\x13\xAA\x1E" + "\xB5\x4C\xE3\x57\xEE\x85\x1C\x90" + "\x27\xBE\x32\xC9\x60\xF7\x6B\x02" + "\x99\x0D\xA4\x3B\xD2\x46\xDD\x74" + "\x0B\x7F\x16\xAD\x21\xB8\x4F\xE6" + "\x5A\xF1\x88\x1F\x93\x2A\xC1\x35" + "\xCC\x63\xFA\x6E\x05\x9C\x10\xA7" + "\x3E\xD5\x49\xE0\x77\x0E\x82\x19" + "\xB0\x24\xBB\x52\xE9\x5D\xF4\x8B" + "\x22\x96\x2D\xC4\x38\xCF\x66\xFD" + "\x71\x08\x9F\x13\xAA\x41\xD8\x4C" + "\xE3\x7A\x11\x85\x1C\xB3\x27\xBE" + "\x55\xEC\x60\xF7\x8E\x02\x99\x30" + "\xC7\x3B\xD2\x69\x00\x74\x0B\xA2" + "\x16\xAD\x44\xDB\x4F\xE6\x7D\x14" + "\x88\x1F\xB6\x2A\xC1\x58\xEF\x63" + "\xFA\x91\x05\x9C\x33\xCA\x3E\xD5" + "\x6C\x03\x77\x0E\xA5\x19\xB0\x47" + "\xDE\x52\xE9\x80\x17\x8B\x22\xB9" + "\x2D\xC4\x5B\xF2\x66\xFD\x94\x08" + "\x9F\x36\xCD\x41\xD8\x6F\x06\x7A" + "\x11\xA8\x1C\xB3\x4A\xE1\x55\xEC" + "\x83\x1A\x8E\x25\xBC\x30\xC7\x5E" + "\xF5\x69\x00\x97\x0B\xA2\x39\xD0" + "\x44\xDB\x72\x09\x7D\x14\xAB\x1F" + "\xB6\x4D\xE4\x58\xEF\x86\x1D\x91" + "\x28\xBF\x33\xCA\x61\xF8\x6C\x03" + "\x9A\x0E\xA5\x3C\xD3\x47\xDE\x75" + "\x0C\x80\x17\xAE\x22\xB9\x50\xE7" + "\x5B\xF2\x89\x20\x94\x2B\xC2\x36" + "\xCD\x64\xFB\x6F\x06\x9D\x11\xA8" + "\x3F\xD6\x4A\xE1\x78\x0F\x83\x1A" + "\xB1\x25\xBC\x53\xEA\x5E\xF5\x8C" + "\x00\x97\x2E\xC5\x39\xD0\x67\xFE" + "\x72\x09\xA0\x14\xAB\x42\xD9\x4D" + "\xE4\x7B\x12", + .ctext = "\xF3\x06\x3A\x84\xCD\xBA\x8E\x11" + "\xB7\x74\x6F\x5C\x97\xFB\x36\xFE" + "\xDE\x71\x58\xD4\x15\xD1\xC1\xA4" + "\xC9\x28\x74\xA6\x6B\xC7\x95\xA6" + "\x6C\x77\xF7\x2F\xDF\xC7\xBB\x85" + "\x60\xFC\xE8\x94\xE8\xB5\x09\x2C" + "\x1E\x43\xEF\x6C\xE9\x98\xC5\xA0" + "\x7B\x13\xE5\x7F\xF8\x49\x9A\x8C" + "\xE6\x7B\x08\xC3\x32\x66\x55\x4E" + "\xA5\x44\x1D\x2C\x18\xC7\x29\x1F" + "\x61\x28\x4A\xE3\xCD\xE5\x47\xB2" + "\x82\x2F\x66\x83\x91\x51\xAE\xD7" + "\x1C\x91\x3C\x57\xE3\x1D\x5A\xC9" + "\xFD\xC5\x58\x58\xEF\xCC\x33\xC9" + "\x0F\xEA\x26\x32\xD1\x15\x19\x2D" + "\x25\xB4\x7F\xB0\xDF\xFB\x88\x60" + "\x4E\x4D\x06\x7D\xCC\x1F\xED\x3B" + "\x68\x84\xD5\xB3\x1B\xE7\xB9\xA1" + "\x68\x8B\x2C\x1A\x44\xDA\x63\xD3" + "\x29\xE9\x59\x32\x1F\x30\x1C\x43" + "\xEA\x3A\xA3\x6B\x54\x3C\xAA\x11" + "\xAD\x38\x20\xC9\xB9\x8A\x64\x66" + "\x5A\x07\x49\xDF\xA1\x9C\xF9\x76" + "\x36\x65\xB6\x81\x8F\x76\x09\xE5" + "\xEB\xD1\x29\xA4\xE4\xF4\x4C\xCD" + "\xAF\xFC\xB9\x16\xD9\xC3\x73\x6A" + "\x33\x12\xF8\x7E\xBC\xCC\x7D\x80" + "\xBF\x3C\x25\x06\x13\x84\xFA\x35" + "\xF7\x40\xFA\xA1\x44\x13\x70\xD8" + "\x01\xF9\x85\x15\x63\xEC\x7D\xB9" + "\x02\xD8\xBA\x41\x6C\x92\x68\x66" + "\x95\xDD\xD6\x42\xE7\xBB\xE1\xFD" + "\x28\x3E\x94\xB6\xBD\xA7\xBF\x47" + "\x58\x8D\xFF\x19\x30\x75\x0D\x48" + "\x94\xE9\xA6\xCD\xB3\x8E\x1E\xCD" + "\x59\xBC\x1A\xAC\x3C\x4F\xA9\xEB" + "\xF4\xA7\xE4\x75\x4A\x18\x40\xC9" + "\x1E\xEC\x06\x9C\x28\x4B\xF7\x2B" + "\xE2\xEF\xD6\x42\x2E\xBB\xFC\x0A" + "\x79\xA2\x99\x28\x93\x1B\x00\x57" + "\x35\x1E\x1A\x93\x90\xA4\x68\x95" + "\x5E\x57\x40\xD5\xA9\xAA\x19\x48" + "\xEC\xFF\x76\x77\xDC\x78\x89\x76" + "\xE5\x3B\x00\xEC\x58\x4D\xD1\xE3" + "\xC8\x6C\x2C\x45\x5E\x5F\xD9\x4E" + "\x71\xA5\x36\x6D\x03\xF1\xC7\xD5" + "\xF3\x63\xC0\xD8\xCB\x2B\xF1\xA8" + "\xB9\x2B\xE6\x0B\xB9\x65\x78\xA0" + "\xC4\x46\xE6\x9B\x8B\x43\x2D\xAB" + "\x70\xA6\xE0\x59\x1E\xAC\x9D\xE0" + "\x76\x44\x45\xF3\x24\x11\x57\x98" + "\x9A\x86\xB4\x12\x80\x28\x86\x20" + "\x23\x9D\x2D\xE9\x38\x32\xB1\xE1" + "\xCF\x0A\x23\x73\x7D\xC5\x80\x3D" + "\x9F\x6D\xA0\xD0\xEE\x93\x8A\x79" + "\x3A\xDD\x1D\xBB\x9E\x26\x5D\x01" + "\x44\xD0\xD4\x4E\xC3\xF1\xE4\x38" + "\x09\x62\x0A\x1A\x4E\xD2\x63\x0F" + "\x6E\x3E\xD2\xA4\x3A\xF4\xF3\xFF" + "\x7E\x42\xEC\xB6\x6F\x4D\x6B\x48" + "\xE6\xA6\x50\x80\x78\x9E\xF1\xB0" + "\x4D\xB2\x0D\x3D\xFC\x40\x25\x4D" + "\x93\x11\x1C\xE9\xD2\x9F\x6E\x90" + "\xE5\x41\x4A\xE2\x3C\x45\x29\x35" + "\xEC\xD6\x47\x50\xCB\x7B\xA2\x32" + "\xF7\x8B\x62\xF1\xE3\x9A\xFE\xC7" + "\x1D\x8C\x02\x72\x68\x09\xE9\xB6" + "\x4A\x80\xE6\xB1\x56\xDF\x90\xD4" + "\x93\x74\xA4\xCE\x20\x23\xBF\x48" + "\xA5\xDE\x1B\xFA\x40\x69\x31\x98" + "\x62\x6E\xA5\xC7\xBF\x0C\x62\xE5" + "\x6D\xE1\x93\xF1\x83\x10\x1C\xCA" + "\xF6\x5C\x19\xF8\x90\x78\xCB\xE4" + "\x0B\x3A\xB5\xF8\x43\x86\xD3\x3F" + "\xBA\x83\x34\x3C\x42\xCC\x7D\x28" + "\x29\x63\x4F\xD8\x02\x17\xC5\x07" + "\x2C\xA4\xAC\x79\xCB\xC3\xA9\x09" + "\x81\x45\x18\xED\xE4\xCB\x42\x3B" + "\x87\x2D\x23\xDC\xC5\xBA\x45\xBD" + "\x92\xE5\x02\x97\x96\xCE\xAD\xEC" + "\xBA\xD8\x76\xF8\xCA\xC1\x31\xEC" + "\x1E\x4F\x3F\x83\xF8\x33\xE8\x6E" + "\xCC\xF8\x5F\xDD\x65\x50\x99\x69" + "\xAF\x48\xCE\xA5\xBA\xB6\x14\x9F" + "\x05\x93\xB2\xE6\x59\xC8\x28\xFE" + "\x8F\x37\xF9\x64\xB9\xA5\x56\x8F" + "\xF1\x1B\x90\xEF\xAE\xEB\xFC\x09" + "\x11\x7A\xF2\x19\x0A\x0A\x9A\x3C" + "\xE2\x5E\x29\xFA\x31\x9B\xC1\x74" + "\x1E\x10\x3E\x07\xA9\x31\x6D\xF8" + "\x81\xF5\xD5\x8A\x04\x23\x51\xAC" + "\xA2\xE2\x63\xFD\x27\x1F\x79\x5B" + "\x1F\xE8\xDA\x11\x49\x4D\x1C\xBA" + "\x54\xCC\x0F\xBA\x92\x69\xE5\xCB" + "\x41\x1A\x67\xA6\x40\x82\x70\x8C" + "\x19\x79\x08\xA4\x51\x20\x7D\xC9" + "\x12\x27\xAE\x20\x0D\x2C\xA1\x6D" + "\xF4\x55\xD4\xE7\xE6\xD4\x28\x08" + "\x00\x70\x12\x56\x56\x50\xAD\x14" + "\x5C\x3E\xA2\xD1\x36\x3F\x36\x48" + "\xED\xB1\x57\x3E\x5D\x15\xF6\x1E" + "\x53\xE9\xA4\x3E\xED\x7D\xCF\x7D" + "\x29\xAF\xF3\x1E\x51\xA8\x9F\x85" + "\x8B\xF0\xBB\xCE\xCC\x39\xC3\x64" + "\x4B\xF2\xAD\x70\x19\xD4\x44\x8F" + "\x91\x76\xE8\x15\x66\x34\x9F\xF6" + "\x0F\x15\xA4\xA8\x24\xF8\x58\xB1" + "\x38\x46\x47\xC7\x9B\xCA\xE9\x42" + "\x44\xAA\xE6\xB5\x9C\x91\xA4\xD3" + "\x16\xA0\xED\x42\xBE\xB5\x06\x19" + "\xBE\x67\xE8\xBC\x22\x32\xA4\x1E" + "\x93\xEB\xBE\xE9\xE1\x93\xE5\x31" + "\x3A\xA2\x75\xDF\xE3\x6B\xE7\xCC" + "\xB4\x70\x20\xE0\x6D\x82\x7C\xC8" + "\x94\x5C\x5E\x37\x18\xAD\xED\x8B" + "\x44\x86\xCA\x5E\x07\xB7\x70\x8D" + "\x40\x48\x19\x73\x7C\x78\x64\x0B" + "\xDB\x01\xCA\xAE\x63\x19\xE9\xD1" + "\x6B\x2C\x84\x10\x45\x42\x2E\xC3" + "\xDF\x7F\xAA\xE8\x87\x1B\x63\x46" + "\x74\x28\x9D\x05\x30\x20\x62\x41" + "\xC0\x9F\x2C\x36\x2B\x78\xD7\x26" + "\xDF\x58\x51\xED\xFA\xDC\x87\x79" + "\xBF\x8C\xBF\xC4\x0F\xE5\x05\xDA" + "\x45\xE3\x35\x0D\x69\x91\x54\x1C" + "\xE7\x2C\x49\x08\x8B\x72\xFA\x5C" + "\xF1\x6B\xD9", + .len = 1011, + .also_non_np = 1, + .np = 2, + .tap = { 1011 - 16, 16 }, + }, { /* Generated with Crypto++ */ + .key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9" + "\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A" + "\x27\x04\xE1\x27\x04\xE1\xBE\x9B" + "\x78\xBE\x9B\x78\x55\x32\x0F\x55", + .klen = 32, + .iv = "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF" + "\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFD", + .ptext = "\x56\xED\x84\x1B\x8F\x26\xBD\x31" "\xC8\x5F\xF6\x6A\x01\x98\x0C\xA3" "\x3A\xD1\x45\xDC\x73\x0A\x7E\x15" "\xAC\x20\xB7\x4E\xE5\x59\xF0\x87" @@ -32579,273 +28921,146 @@ static const struct cipher_testvec camellia_ctr_dec_tv_template[] = { "\x44\xDB\x72\x09\x7D\x14\xAB\x1F" "\xB6\x4D\xE4\x58\xEF\x86\x1D\x91" "\x28\xBF\x33\xCA\x61\xF8\x6C\x03" - "\x9A\x0E\xA5\x3C\xD3\x47\xDE\x75" - "\x0C\x80\x17\xAE\x22\xB9\x50\xE7" - "\x5B\xF2\x89\x20\x94\x2B\xC2\x36" - "\xCD\x64\xFB\x6F\x06\x9D\x11\xA8" - "\x3F\xD6\x4A\xE1\x78\x0F\x83\x1A" - "\xB1\x25\xBC\x53\xEA\x5E\xF5\x8C" - "\x00\x97\x2E\xC5\x39\xD0\x67\xFE" - "\x72\x09\xA0\x14\xAB\x42\xD9\x4D", - .rlen = 1008, - }, -}; - -static const struct cipher_testvec camellia_lrw_enc_tv_template[] = { - /* Generated from AES-LRW test vectors */ - { - .key = "\x45\x62\xac\x25\xf8\x28\x17\x6d" - "\x4c\x26\x84\x14\xb5\x68\x01\x85" - "\x25\x8e\x2a\x05\xe7\x3e\x9d\x03" - "\xee\x5a\x83\x0c\xcc\x09\x4c\x87", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x92\x68\x19\xd7\xb7\x5b\x0a\x31" - "\x97\xcc\x72\xbe\x99\x17\xeb\x3e", - .rlen = 16, - }, { - .key = "\x59\x70\x47\x14\xf5\x57\x47\x8c" - "\xd7\x79\xe8\x0f\x54\x88\x79\x44" - "\x0d\x48\xf0\xb7\xb1\x5a\x53\xea" - "\x1c\xaa\x6b\x29\xc2\xca\xfb\xaf", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x02", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x73\x09\xb7\x50\xb6\x77\x30\x50" - "\x5c\x8a\x9c\x26\x77\x9d\xfc\x4a", - .rlen = 16, - }, { - .key = "\xd8\x2a\x91\x34\xb2\x6a\x56\x50" - "\x30\xfe\x69\xe2\x37\x7f\x98\x47" - "\xcd\xf9\x0b\x16\x0c\x64\x8f\xb6" - "\xb0\x0d\x0d\x1b\xae\x85\x87\x1f", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x90\xae\x83\xe0\x22\xb9\x60\x91" - "\xfa\xa9\xb7\x98\xe3\xed\x87\x01", - .rlen = 16, - }, { - .key = "\x0f\x6a\xef\xf8\xd3\xd2\xbb\x15" - "\x25\x83\xf7\x3c\x1f\x01\x28\x74" - "\xca\xc6\xbc\x35\x4d\x4a\x65\x54" - "\x90\xae\x61\xcf\x7b\xae\xbd\xcc" - "\xad\xe4\x94\xc5\x4a\x29\xae\x70", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x99\xe9\x6e\xd4\xc9\x21\xa5\xf0" - "\xd8\x83\xef\xd9\x07\x16\x5f\x35", - .rlen = 16, - }, { - .key = "\x8a\xd4\xee\x10\x2f\xbd\x81\xff" - "\xf8\x86\xce\xac\x93\xc5\xad\xc6" - "\xa0\x19\x07\xc0\x9d\xf7\xbb\xdd" - "\x52\x13\xb2\xb7\xf0\xff\x11\xd8" - "\xd6\x08\xd0\xcd\x2e\xb1\x17\x6f", - .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x42\x88\xf4\xcb\x21\x11\x6d\x8e" - "\xde\x1a\xf2\x29\xf1\x4a\xe0\x15", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x40\xaa\x34\x86\x4a\x8f\x78\xb9" - "\xdb\xdb\x0f\x3d\x48\x70\xbe\x8d", - .rlen = 16, - }, { - .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" - "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" - "\xb2\xfb\x64\xce\x60\x97\x87\x8d" - "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" - "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" - "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .ilen = 16, - .result = "\x04\xab\x28\x37\x31\x7a\x26\xab" - "\xa1\x70\x1b\x9c\xe7\xdd\x83\xff", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" - "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" - "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" - "\x50\x38\x1f\x71\x49\xb6\x57\xd6" - "\x8f\xcb\x8d\x6b\xe3\xa6\x29\x90" - "\xfe\x2a\x62\x82\xae\x6d\x8b\xf6" - "\xad\x1e\x9e\x20\x5f\x38\xbe\x04" - "\xda\x10\x8e\xed\xa2\xa4\x87\xab" - "\xda\x6b\xb4\x0c\x75\xba\xd3\x7c" - "\xc9\xac\x42\x31\x95\x7c\xc9\x04" - "\xeb\xd5\x6e\x32\x69\x8a\xdb\xa6" - "\x15\xd7\x3f\x4f\x2f\x66\x69\x03" - "\x9c\x1f\x54\x0f\xde\x1f\xf3\x65" - "\x4c\x96\x12\xed\x7c\x92\x03\x01" - "\x6f\xbc\x35\x93\xac\xf1\x27\xf1" - "\xb4\x96\x82\x5a\x5f\xb0\xa0\x50" - "\x89\xa4\x8e\x66\x44\x85\xcc\xfd" - "\x33\x14\x70\xe3\x96\xb2\xc3\xd3" - "\xbb\x54\x5a\x1a\xf9\x74\xa2\xc5" - "\x2d\x64\x75\xdd\xb4\x54\xe6\x74" - "\x8c\xd3\x9d\x9e\x86\xab\x51\x53" - "\xb7\x93\x3e\x6f\xd0\x4e\x2c\x40" - "\xf6\xa8\x2e\x3e\x9d\xf4\x66\xa5" - "\x76\x12\x73\x44\x1a\x56\xd7\x72" - "\x88\xcd\x21\x8c\x4c\x0f\xfe\xda" - "\x95\xe0\x3a\xa6\xa5\x84\x46\xcd" - "\xd5\x3e\x9d\x3a\xe2\x67\xe6\x60" - "\x1a\xe2\x70\x85\x58\xc2\x1b\x09" - "\xe1\xd7\x2c\xca\xad\xa8\x8f\xf9" - "\xac\xb3\x0e\xdb\xca\x2e\xe2\xb8" - "\x51\x71\xd9\x3c\x6c\xf1\x56\xf8" - "\xea\x9c\xf1\xfb\x0c\xe6\xb7\x10" - "\x1c\xf8\xa9\x7c\xe8\x53\x35\xc1" - "\x90\x3e\x76\x4a\x74\xa4\x21\x2c" - "\xf6\x2c\x4e\x0f\x94\x3a\x88\x2e" - "\x41\x09\x6a\x33\x7d\xf6\xdd\x3f" - "\x8d\x23\x31\x74\x84\xeb\x88\x6e" - "\xcc\xb9\xbc\x22\x83\x19\x07\x22" - "\xa5\x2d\xdf\xa5\xf3\x80\x85\x78" - "\x84\x39\x6a\x6d\x6a\x99\x4f\xa5" - "\x15\xfe\x46\xb0\xe4\x6c\xa5\x41" - "\x3c\xce\x8f\x42\x60\x71\xa7\x75" - "\x08\x40\x65\x8a\x82\xbf\xf5\x43" - "\x71\x96\xa9\x4d\x44\x8a\x20\xbe" - "\xfa\x4d\xbb\xc0\x7d\x31\x96\x65" - "\xe7\x75\xe5\x3e\xfd\x92\x3b\xc9" - "\x55\xbb\x16\x7e\xf7\xc2\x8c\xa4" - "\x40\x1d\xe5\xef\x0e\xdf\xe4\x9a" - "\x62\x73\x65\xfd\x46\x63\x25\x3d" - "\x2b\xaf\xe5\x64\xfe\xa5\x5c\xcf" - "\x24\xf3\xb4\xac\x64\xba\xdf\x4b" - "\xc6\x96\x7d\x81\x2d\x8d\x97\xf7" - "\xc5\x68\x77\x84\x32\x2b\xcc\x85" - "\x74\x96\xf0\x12\x77\x61\xb9\xeb" - "\x71\xaa\x82\xcb\x1c\xdb\x89\xc8" - "\xc6\xb5\xe3\x5c\x7d\x39\x07\x24" - "\xda\x39\x87\x45\xc0\x2b\xbb\x01" - "\xac\xbc\x2a\x5c\x7f\xfc\xe8\xce" - "\x6d\x9c\x6f\xed\xd3\xc1\xa1\xd6" - "\xc5\x55\xa9\x66\x2f\xe1\xc8\x32" - "\xa6\x5d\xa4\x3a\x98\x73\xe8\x45" - "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" - "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" - "\x21\xc4\xc2\x75\x67\x89\x37\x0a", - .ilen = 512, - .result = "\x90\x69\x8e\xf2\x14\x86\x59\xf9" - "\xec\xe7\xfa\x3f\x48\x9d\x7f\x96" - "\x67\x76\xac\x2c\xd2\x63\x18\x93" - "\x13\xf8\xf1\xf6\x71\x77\xb3\xee" - "\x93\xb2\xcc\xf3\x26\xc1\x16\x4f" - "\xd4\xe8\x43\xc1\x68\xa3\x3e\x06" - "\x38\x51\xff\xa8\xb9\xa4\xeb\xb1" - "\x62\xdd\x78\x81\xea\x1d\xef\x04" - "\x1d\x07\xc1\x67\xc8\xd6\x77\xa1" - "\x84\x95\xf4\x9a\xd9\xbc\x2d\xe2" - "\xf6\x80\xfc\x91\x2a\xbc\x42\xa0" - "\x40\x41\x69\xaa\x71\xc0\x37\xec" - "\x39\xf3\xf2\xec\x82\xc3\x88\x79" - "\xbc\xc3\xaa\xb7\xcf\x6a\x72\x80" - "\x4c\xf4\x84\x8f\x13\x9e\x94\x5c" - "\xe5\xb2\x91\xbb\x92\x51\x4d\xf1" - "\xd6\x0d\x71\x6b\x7a\xc2\x2f\x12" - "\x6f\x75\xc7\x80\x99\x50\x84\xcf" - "\xa8\xeb\xd6\xe1\x1c\x59\x81\x7e" - "\xb9\xb3\xde\x7a\x93\x14\x12\xa2" - "\xf7\x43\xb3\x9d\x1a\x87\x65\x91" - "\x42\x08\x40\x82\x06\x1c\x2d\x55" - "\x6e\x48\xd5\x74\x07\x6e\x9d\x80" - "\xeb\xb4\x97\xa1\x36\xdf\xfa\x74" - "\x79\x7f\x5a\x75\xe7\x71\xc8\x8c" - "\x7e\xf8\x3a\x77\xcd\x32\x05\xf9" - "\x3d\xd4\xe9\xa2\xbb\xc4\x8b\x83" - "\x42\x5c\x82\xfa\xe9\x4b\x96\x3b" - "\x7f\x89\x8b\xf9\xf1\x87\xda\xf0" - "\x87\xef\x13\x5d\xf0\xe2\xc5\xc1" - "\xed\x14\xa9\x57\x19\x63\x40\x04" - "\x24\xeb\x6e\x19\xd1\x3d\x70\x78" - "\xeb\xda\x55\x70\x2c\x4f\x41\x5b" - "\x56\x9f\x1a\xd3\xac\xf1\xc0\xc3" - "\x21\xec\xd7\xd2\x55\x32\x7c\x2e" - "\x3c\x48\x8e\xb4\x85\x35\x47\xfe" - "\xe2\x88\x79\x98\x6a\xc9\x8d\xff" - "\xe9\x89\x6e\xb8\xe2\x97\x00\xbd" - "\xa4\x8f\xba\xd0\x8c\xcb\x79\x99" - "\xb3\xb2\xb2\x7a\xc3\xb7\xef\x75" - "\x23\x52\x76\xc3\x50\x6e\x66\xf8" - "\xa2\xe2\xce\xba\x40\x21\x3f\xc9" - "\x0a\x32\x7f\xf7\x08\x8c\x66\xcf" - "\xd3\xdf\x57\x59\x83\xb8\xe1\x85" - "\xd6\x8f\xfb\x48\x1f\x3a\xc4\x2f" - "\xb4\x2d\x58\xab\xd8\x7f\x5e\x3a" - "\xbc\x62\x3e\xe2\x6a\x52\x0d\x76" - "\x2f\x1c\x1a\x30\xed\x95\x2a\x44" - "\x35\xa5\x83\x04\x84\x01\x99\x56" - "\xb7\xe3\x10\x96\xfa\xdc\x19\xdd" - "\xe2\x7f\xcb\xa0\x49\x1b\xff\x4c" - "\x73\xf6\xbb\x94\x00\xe8\xa9\x3d" - "\xe2\x20\xe9\x3f\xfa\x07\x5d\x77" - "\x06\xd5\x4f\x4d\x02\xb8\x40\x1b" - "\x30\xed\x1a\x50\x19\xef\xc4\x2c" - "\x02\xd9\xc5\xd3\x11\x33\x37\xe5" - "\x2b\xa3\x95\xa6\xee\xd8\x74\x1d" - "\x68\xa0\xeb\xbf\xdd\x5e\x99\x96" - "\x91\xc3\x94\x24\xa5\x12\xa2\x37" - "\xb3\xac\xcf\x2a\xfd\x55\x34\xfe" - "\x79\x92\x3e\xe6\x1b\x49\x57\x5d" - "\x93\x6c\x01\xf7\xcc\x4e\x20\xd1" - "\xb2\x1a\xd8\x4c\xbd\x1d\x10\xe9" - "\x5a\xa8\x92\x7f\xba\xe6\x0c\x95", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, + "\x9A\x0E\xA5\x3C\xD3\x47\xDE\x75" + "\x0C\x80\x17\xAE\x22\xB9\x50\xE7" + "\x5B\xF2\x89\x20\x94\x2B\xC2\x36" + "\xCD\x64\xFB\x6F\x06\x9D\x11\xA8" + "\x3F\xD6\x4A\xE1\x78\x0F\x83\x1A" + "\xB1\x25\xBC\x53\xEA\x5E\xF5\x8C" + "\x00\x97\x2E\xC5\x39\xD0\x67\xFE" + "\x72\x09\xA0\x14\xAB\x42\xD9\x4D", + .ctext = "\x85\x79\x6C\x8B\x2B\x6D\x14\xF9" + "\xA6\x83\xB6\x80\x5B\x3A\xF3\x7E" + "\x30\x29\xEB\x1F\xDC\x19\x5F\xEB" + "\xF7\xC4\x27\x04\x51\x87\xD7\x6F" + "\xB8\x4E\x07\xFB\xAC\x3B\x08\xB4" + "\x4D\xCB\xE8\xE1\x71\x7D\x4F\x48" + "\xCD\x81\x64\xA5\xC4\x07\x1A\x9A" + "\x4B\x62\x90\x0E\xC8\xB3\x2B\x6B" + "\x8F\x9C\x6E\x72\x4B\xBA\xEF\x07" + "\x2C\x56\x07\x5E\x37\x30\x60\xA9" + "\xE3\xEF\xD6\x69\xE1\xA1\x77\x64" + "\x93\x75\x7A\xB7\x7A\x3B\xE9\x43" + "\x23\x35\x95\x91\x80\x8A\xC7\xCF" + "\xC3\xD5\xBF\xE7\xFE\x4C\x06\x6B" + "\x05\x19\x48\xE2\x62\xBA\x4F\xF2" + "\xFB\xEE\xE4\xCB\x79\x9D\xA3\x10" + "\x1D\x29\x8C\x1D\x7A\x88\x5A\xDD" + "\x4E\xB6\x18\xAA\xCD\xE6\x33\x96" + "\xD9\x0F\x90\x5A\x78\x76\x4D\x77" + "\x3C\x20\x89\x3B\xA3\xF9\x07\xFD" + "\xE4\xE8\x20\x2D\x15\x0A\x63\x49" + "\xF5\x4F\x89\xD8\xDE\xA1\x28\x78" + "\x28\x07\x09\x1B\x03\x94\x1D\x4B" + "\x82\x28\x1E\x1D\x95\xBA\xAC\x85" + "\x71\x6E\x3C\x18\x4B\x77\x74\x79" + "\xBF\x67\x0A\x53\x3C\x94\xD9\x60" + "\xE9\x6D\x40\x34\xA0\x2A\x53\x5D" + "\x27\xD5\x47\xF9\xC3\x4B\x27\x29" + "\xE4\x76\x9C\x3F\xA7\x1C\x87\xFC" + "\x6E\x0F\xCF\x9B\x60\xF0\xF0\x8B" + "\x70\x1C\x84\x81\x72\x4D\xB4\x98" + "\x23\x62\xE7\x6A\x2B\xFC\xA5\xB2" + "\xFF\xF5\x71\x07\xCD\x90\x23\x13" + "\x19\xD7\x79\x36\x6C\x9D\x55\x8B" + "\x93\x78\x86\x05\x69\x46\xD0\xC5" + "\x39\x09\xEB\x79\xEF\xFA\x9F\xAE" + "\xF3\xD5\x44\xC3\xFD\x86\xD2\x7C" + "\x83\x4B\xD8\x75\x9C\x18\x04\x7B" + "\x73\xAD\x72\xA4\xF6\xAB\xCF\x4B" + "\xCC\x01\x45\x90\xA6\x43\x05\x0C" + "\x6C\x4F\x62\x77\x57\x97\x9F\xEE" + "\x75\xA7\x3C\x38\xD1\x0F\x3D\x0E" + "\x2C\x43\x98\xFB\x13\x65\x73\xE4" + "\x3C\x1E\xD6\x90\x08\xF7\xE0\x99" + "\x3B\xF1\x9D\x6C\x48\xA9\x0E\x32" + "\x17\xC2\xCC\x20\xA1\x19\x26\xAA" + "\xE0\x75\x2F\xFB\x54\x66\x0A\xDF" + "\xB5\xF2\x1F\xC1\x34\x3C\x30\x56" + "\xE8\xDC\xF7\x92\x6B\xBF\x17\x24" + "\xEC\x94\xB5\x3B\xD6\xCE\xA2\x54" + "\x10\x7F\x50\xDE\x69\x77\xD5\x37" + "\xFE\x9C\x10\x83\xC5\xEB\xC9\x53" + "\xB7\xF3\xC4\x20\xAF\x0A\x7E\x57" + "\x3A\xE6\x75\xFE\x89\x00\x6E\x48" + "\xFB\x99\x17\x2C\xF6\x64\x40\x95" + "\x5E\xDC\x7A\xA6\x70\xC7\xF4\xDD" + "\x52\x05\x24\x34\xF9\x0E\xC8\x64" + "\x6D\xE2\xD8\x80\x53\x31\x4C\xFE" + "\xB4\x3A\x5F\x19\xCF\x42\x1B\x22" + "\x0B\x2D\x7B\xF1\xC5\x43\xF7\x5E" + "\x12\xA8\x01\x64\x16\x0B\x26\x5A" + "\x0C\x95\x0F\x40\xC5\x5A\x06\x7C" + "\xCF\xF5\xD5\xB7\x7A\x34\x23\xB6" + "\xAA\x9E\xA8\x98\xA2\xF8\x3D\xD3" + "\x3F\x23\x69\x63\x56\x96\x45\xD6" + "\x74\x23\x1D\x5C\x63\xCC\xD8\x78" + "\x16\xE2\x9C\xD2\x80\x02\xF2\x28" + "\x69\x2F\xC4\xA8\x15\x15\x24\x3B" + "\xCB\xF0\x14\xE4\x62\xC8\xF3\xD1" + "\x03\x58\x1B\x33\x77\x74\x1F\xB4" + "\x07\x86\xF2\x21\xB7\x41\xAE\xBF" + "\x25\xC2\xFF\x51\xEF\xEA\xCE\xC4" + "\x5F\xD9\xB8\x18\x6A\xF0\x0F\x0D" + "\xF8\x04\xBB\x6D\x62\x33\x87\x26" + "\x4F\x2F\x14\x6E\xDC\xDB\x66\x09" + "\x2A\xEF\x7D\x84\x10\xAC\x82\x5E" + "\xD2\xE4\xAD\x74\x7A\x6D\xCC\x3A" + "\x7B\x62\xD8\xD6\x07\x2D\xF7\xDF" + "\x9B\xB3\x82\xCF\x9C\x1D\x76\x5C" + "\xAC\x7B\xD4\x9B\x45\xA1\x64\x11" + "\x66\xF1\xA7\x0B\xF9\xDD\x00\xDD" + "\xA4\x45\x3D\x3E\x03\xC9\x2E\xCB" + "\xC3\x14\x84\x72\xFD\x41\xDC\xBD" + "\x75\xBE\xA8\xE5\x16\x48\x64\x39" + "\xCA\xF3\xE6\xDC\x25\x24\xF1\x6D" + "\xB2\x8D\xC5\x38\x54\xD3\x5D\x6D" + "\x0B\x29\x10\x15\x0E\x13\x3B\xAC" + "\x7E\xCC\x9E\x3E\x18\x48\xA6\x02" + "\xEF\x03\xB2\x2E\xE3\xD2\x70\x21" + "\xB4\x19\x26\xBE\x3A\x3D\x05\xE0" + "\xF8\x09\xAF\xE4\x31\x26\x92\x2F" + "\x8F\x55\xAC\xED\x0B\xB2\xA5\x34" + "\xBE\x50\xB1\x02\x22\x96\xE3\x40" + "\x7B\x70\x50\x6E\x3B\xD5\xE5\xA0" + "\x8E\xA2\xAD\x14\x60\x5C\x7A\x2B" + "\x3D\x1B\x7F\xC1\xC0\x2C\x56\x36" + "\xD2\x0A\x32\x06\x97\x34\xB9\xF4" + "\x6F\x9F\x7E\x80\xD0\x9D\xF7\x6A" + "\x21\xC1\xA2\x6A\xB1\x96\x5B\x4D" + "\x7A\x15\x6C\xC4\x4E\xB8\xE0\x9E" + "\x6C\x50\xF3\x9C\xC9\xB5\x23\xB7" + "\xF1\xD4\x29\x4A\x23\xC4\xAD\x1E" + "\x2C\x07\xD2\x43\x5F\x57\x93\xCA" + "\x85\xF9\x9F\xAD\x4C\xF1\xE4\xB1" + "\x1A\x8E\x28\xA4\xB6\x52\x77\x7E" + "\x68\xC6\x47\xB9\x76\xCC\x65\x5F" + "\x0B\xF9\x67\x93\xD8\x0E\x9A\x37" + "\x5F\x41\xED\x64\x6C\xAD\x5F\xED" + "\x3F\x8D\xFB\x8E\x1E\xA0\xE4\x1F" + "\xC2\xC7\xED\x18\x43\xE1\x20\x86" + "\x5D\xBC\x30\x70\x22\xA1\xDC\x53" + "\x10\x3A\x8D\x47\x82\xCD\x7F\x59" + "\x03\x2D\x6D\xF5\xE7\x79\xD4\x07" + "\x68\x2A\xA5\x42\x19\x4D\xAF\xF5" + "\xED\x47\x83\xBC\x5F\x62\x84\xDA" + "\xDA\x41\xFF\xB0\x1D\x64\xA3\xC8" + "\xBD\x4E\xE0\xB8\x7F\xEE\x55\x0A" + "\x4E\x61\xB2\x51\xF6\x9C\x95\xF6" + "\x92\xBB\xF6\xC5\xF0\x09\x86\xDE" + "\x37\x9E\x29\xF9\x2A\x18\x73\x0D" + "\xDC\x7E\x6B\x7B\x1B\x43\x8C\xEA" + "\x13\xC8\x1A\x47\x0A\x2D\x6D\x56" + "\xCD\xD2\xE7\x53\x1A\xAB\x1C\x3C" + "\xC5\x9B\x03\x70\x29\x2A\x49\x09" + "\x67\xA1\xEA\xD6\x3A\x5B\xBF\x71" + "\x1D\x48\x64\x6C\xFB\xC0\x9E\x36", + .len = 1008, }, }; -static const struct cipher_testvec camellia_lrw_dec_tv_template[] = { +static const struct cipher_testvec camellia_lrw_tv_template[] = { /* Generated from AES-LRW test vectors */ - /* same as enc vectors with input and result reversed */ { .key = "\x45\x62\xac\x25\xf8\x28\x17\x6d" "\x4c\x26\x84\x14\xb5\x68\x01\x85" @@ -32854,12 +29069,11 @@ static const struct cipher_testvec camellia_lrw_dec_tv_template[] = { .klen = 32, .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x92\x68\x19\xd7\xb7\x5b\x0a\x31" - "\x97\xcc\x72\xbe\x99\x17\xeb\x3e", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, + .ctext = "\x92\x68\x19\xd7\xb7\x5b\x0a\x31" + "\x97\xcc\x72\xbe\x99\x17\xeb\x3e", + .len = 16, }, { .key = "\x59\x70\x47\x14\xf5\x57\x47\x8c" "\xd7\x79\xe8\x0f\x54\x88\x79\x44" @@ -32868,12 +29082,11 @@ static const struct cipher_testvec camellia_lrw_dec_tv_template[] = { .klen = 32, .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x02", - .input = "\x73\x09\xb7\x50\xb6\x77\x30\x50" - "\x5c\x8a\x9c\x26\x77\x9d\xfc\x4a", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, + .ctext = "\x73\x09\xb7\x50\xb6\x77\x30\x50" + "\x5c\x8a\x9c\x26\x77\x9d\xfc\x4a", + .len = 16, }, { .key = "\xd8\x2a\x91\x34\xb2\x6a\x56\x50" "\x30\xfe\x69\xe2\x37\x7f\x98\x47" @@ -32882,12 +29095,11 @@ static const struct cipher_testvec camellia_lrw_dec_tv_template[] = { .klen = 32, .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x90\xae\x83\xe0\x22\xb9\x60\x91" - "\xfa\xa9\xb7\x98\xe3\xed\x87\x01", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, + .ctext = "\x90\xae\x83\xe0\x22\xb9\x60\x91" + "\xfa\xa9\xb7\x98\xe3\xed\x87\x01", + .len = 16, }, { .key = "\x0f\x6a\xef\xf8\xd3\xd2\xbb\x15" "\x25\x83\xf7\x3c\x1f\x01\x28\x74" @@ -32897,12 +29109,11 @@ static const struct cipher_testvec camellia_lrw_dec_tv_template[] = { .klen = 40, .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x99\xe9\x6e\xd4\xc9\x21\xa5\xf0" - "\xd8\x83\xef\xd9\x07\x16\x5f\x35", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, + .ctext = "\x99\xe9\x6e\xd4\xc9\x21\xa5\xf0" + "\xd8\x83\xef\xd9\x07\x16\x5f\x35", + .len = 16, }, { .key = "\x8a\xd4\xee\x10\x2f\xbd\x81\xff" "\xf8\x86\xce\xac\x93\xc5\xad\xc6" @@ -32910,122 +29121,54 @@ static const struct cipher_testvec camellia_lrw_dec_tv_template[] = { "\x52\x13\xb2\xb7\xf0\xff\x11\xd8" "\xd6\x08\xd0\xcd\x2e\xb1\x17\x6f", .klen = 40, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x42\x88\xf4\xcb\x21\x11\x6d\x8e" - "\xde\x1a\xf2\x29\xf1\x4a\xe0\x15", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x40\xaa\x34\x86\x4a\x8f\x78\xb9" - "\xdb\xdb\x0f\x3d\x48\x70\xbe\x8d", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" - "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" - "\xb2\xfb\x64\xce\x60\x97\x87\x8d" - "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" - "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" - "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x02\x00\x00\x00\x00", - .input = "\x04\xab\x28\x37\x31\x7a\x26\xab" - "\xa1\x70\x1b\x9c\xe7\xdd\x83\xff", - .ilen = 16, - .result = "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x41\x42\x43\x44\x45\x46", - .rlen = 16, - }, { - .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" - "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" - "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" - "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" - "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" - "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", - .klen = 48, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x90\x69\x8e\xf2\x14\x86\x59\xf9" - "\xec\xe7\xfa\x3f\x48\x9d\x7f\x96" - "\x67\x76\xac\x2c\xd2\x63\x18\x93" - "\x13\xf8\xf1\xf6\x71\x77\xb3\xee" - "\x93\xb2\xcc\xf3\x26\xc1\x16\x4f" - "\xd4\xe8\x43\xc1\x68\xa3\x3e\x06" - "\x38\x51\xff\xa8\xb9\xa4\xeb\xb1" - "\x62\xdd\x78\x81\xea\x1d\xef\x04" - "\x1d\x07\xc1\x67\xc8\xd6\x77\xa1" - "\x84\x95\xf4\x9a\xd9\xbc\x2d\xe2" - "\xf6\x80\xfc\x91\x2a\xbc\x42\xa0" - "\x40\x41\x69\xaa\x71\xc0\x37\xec" - "\x39\xf3\xf2\xec\x82\xc3\x88\x79" - "\xbc\xc3\xaa\xb7\xcf\x6a\x72\x80" - "\x4c\xf4\x84\x8f\x13\x9e\x94\x5c" - "\xe5\xb2\x91\xbb\x92\x51\x4d\xf1" - "\xd6\x0d\x71\x6b\x7a\xc2\x2f\x12" - "\x6f\x75\xc7\x80\x99\x50\x84\xcf" - "\xa8\xeb\xd6\xe1\x1c\x59\x81\x7e" - "\xb9\xb3\xde\x7a\x93\x14\x12\xa2" - "\xf7\x43\xb3\x9d\x1a\x87\x65\x91" - "\x42\x08\x40\x82\x06\x1c\x2d\x55" - "\x6e\x48\xd5\x74\x07\x6e\x9d\x80" - "\xeb\xb4\x97\xa1\x36\xdf\xfa\x74" - "\x79\x7f\x5a\x75\xe7\x71\xc8\x8c" - "\x7e\xf8\x3a\x77\xcd\x32\x05\xf9" - "\x3d\xd4\xe9\xa2\xbb\xc4\x8b\x83" - "\x42\x5c\x82\xfa\xe9\x4b\x96\x3b" - "\x7f\x89\x8b\xf9\xf1\x87\xda\xf0" - "\x87\xef\x13\x5d\xf0\xe2\xc5\xc1" - "\xed\x14\xa9\x57\x19\x63\x40\x04" - "\x24\xeb\x6e\x19\xd1\x3d\x70\x78" - "\xeb\xda\x55\x70\x2c\x4f\x41\x5b" - "\x56\x9f\x1a\xd3\xac\xf1\xc0\xc3" - "\x21\xec\xd7\xd2\x55\x32\x7c\x2e" - "\x3c\x48\x8e\xb4\x85\x35\x47\xfe" - "\xe2\x88\x79\x98\x6a\xc9\x8d\xff" - "\xe9\x89\x6e\xb8\xe2\x97\x00\xbd" - "\xa4\x8f\xba\xd0\x8c\xcb\x79\x99" - "\xb3\xb2\xb2\x7a\xc3\xb7\xef\x75" - "\x23\x52\x76\xc3\x50\x6e\x66\xf8" - "\xa2\xe2\xce\xba\x40\x21\x3f\xc9" - "\x0a\x32\x7f\xf7\x08\x8c\x66\xcf" - "\xd3\xdf\x57\x59\x83\xb8\xe1\x85" - "\xd6\x8f\xfb\x48\x1f\x3a\xc4\x2f" - "\xb4\x2d\x58\xab\xd8\x7f\x5e\x3a" - "\xbc\x62\x3e\xe2\x6a\x52\x0d\x76" - "\x2f\x1c\x1a\x30\xed\x95\x2a\x44" - "\x35\xa5\x83\x04\x84\x01\x99\x56" - "\xb7\xe3\x10\x96\xfa\xdc\x19\xdd" - "\xe2\x7f\xcb\xa0\x49\x1b\xff\x4c" - "\x73\xf6\xbb\x94\x00\xe8\xa9\x3d" - "\xe2\x20\xe9\x3f\xfa\x07\x5d\x77" - "\x06\xd5\x4f\x4d\x02\xb8\x40\x1b" - "\x30\xed\x1a\x50\x19\xef\xc4\x2c" - "\x02\xd9\xc5\xd3\x11\x33\x37\xe5" - "\x2b\xa3\x95\xa6\xee\xd8\x74\x1d" - "\x68\xa0\xeb\xbf\xdd\x5e\x99\x96" - "\x91\xc3\x94\x24\xa5\x12\xa2\x37" - "\xb3\xac\xcf\x2a\xfd\x55\x34\xfe" - "\x79\x92\x3e\xe6\x1b\x49\x57\x5d" - "\x93\x6c\x01\xf7\xcc\x4e\x20\xd1" - "\xb2\x1a\xd8\x4c\xbd\x1d\x10\xe9" - "\x5a\xa8\x92\x7f\xba\xe6\x0c\x95", - .ilen = 512, - .result = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x02\x00\x00\x00\x00", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x42\x88\xf4\xcb\x21\x11\x6d\x8e" + "\xde\x1a\xf2\x29\xf1\x4a\xe0\x15", + .len = 16, + }, { + .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" + "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" + "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" + "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" + "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" + "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x40\xaa\x34\x86\x4a\x8f\x78\xb9" + "\xdb\xdb\x0f\x3d\x48\x70\xbe\x8d", + .len = 16, + }, { + .key = "\xfb\x76\x15\xb2\x3d\x80\x89\x1d" + "\xd4\x70\x98\x0b\xc7\x95\x84\xc8" + "\xb2\xfb\x64\xce\x60\x97\x87\x8d" + "\x17\xfc\xe4\x5a\x49\xe8\x30\xb7" + "\x6e\x78\x17\xe7\x2d\x5e\x12\xd4" + "\x60\x64\x04\x7a\xf1\x2f\x9e\x0c", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x02\x00\x00\x00\x00", + .ptext = "\x30\x31\x32\x33\x34\x35\x36\x37" + "\x38\x39\x41\x42\x43\x44\x45\x46", + .ctext = "\x04\xab\x28\x37\x31\x7a\x26\xab" + "\xa1\x70\x1b\x9c\xe7\xdd\x83\xff", + .len = 16, + }, { + .key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c" + "\x23\x84\xcb\x1c\x77\xd6\x19\x5d" + "\xfe\xf1\xa9\xf3\x7b\xbc\x8d\x21" + "\xa7\x9c\x21\xf8\xcb\x90\x02\x89" + "\xa8\x45\x34\x8e\xc8\xc5\xb5\xf1" + "\x26\xf5\x0e\x76\xfe\xfd\x1b\x1e", + .klen = 48, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x01", + .ptext = "\x05\x11\xb7\x18\xab\xc6\x2d\xac" "\x70\x5d\xf6\x22\x94\xcd\xe5\x6c" "\x17\x6b\xf6\x1c\xf0\xf3\x6e\xf8" "\x50\x38\x1f\x71\x49\xb6\x57\xd6" @@ -33089,14 +29232,78 @@ static const struct cipher_testvec camellia_lrw_dec_tv_template[] = { "\xa4\xc7\xa8\xb4\xf6\x13\x03\xf6" "\xe9\x2e\xc4\x29\x0f\x84\xdb\xc4" "\x21\xc4\xc2\x75\x67\x89\x37\x0a", - .rlen = 512, + .ctext = "\x90\x69\x8e\xf2\x14\x86\x59\xf9" + "\xec\xe7\xfa\x3f\x48\x9d\x7f\x96" + "\x67\x76\xac\x2c\xd2\x63\x18\x93" + "\x13\xf8\xf1\xf6\x71\x77\xb3\xee" + "\x93\xb2\xcc\xf3\x26\xc1\x16\x4f" + "\xd4\xe8\x43\xc1\x68\xa3\x3e\x06" + "\x38\x51\xff\xa8\xb9\xa4\xeb\xb1" + "\x62\xdd\x78\x81\xea\x1d\xef\x04" + "\x1d\x07\xc1\x67\xc8\xd6\x77\xa1" + "\x84\x95\xf4\x9a\xd9\xbc\x2d\xe2" + "\xf6\x80\xfc\x91\x2a\xbc\x42\xa0" + "\x40\x41\x69\xaa\x71\xc0\x37\xec" + "\x39\xf3\xf2\xec\x82\xc3\x88\x79" + "\xbc\xc3\xaa\xb7\xcf\x6a\x72\x80" + "\x4c\xf4\x84\x8f\x13\x9e\x94\x5c" + "\xe5\xb2\x91\xbb\x92\x51\x4d\xf1" + "\xd6\x0d\x71\x6b\x7a\xc2\x2f\x12" + "\x6f\x75\xc7\x80\x99\x50\x84\xcf" + "\xa8\xeb\xd6\xe1\x1c\x59\x81\x7e" + "\xb9\xb3\xde\x7a\x93\x14\x12\xa2" + "\xf7\x43\xb3\x9d\x1a\x87\x65\x91" + "\x42\x08\x40\x82\x06\x1c\x2d\x55" + "\x6e\x48\xd5\x74\x07\x6e\x9d\x80" + "\xeb\xb4\x97\xa1\x36\xdf\xfa\x74" + "\x79\x7f\x5a\x75\xe7\x71\xc8\x8c" + "\x7e\xf8\x3a\x77\xcd\x32\x05\xf9" + "\x3d\xd4\xe9\xa2\xbb\xc4\x8b\x83" + "\x42\x5c\x82\xfa\xe9\x4b\x96\x3b" + "\x7f\x89\x8b\xf9\xf1\x87\xda\xf0" + "\x87\xef\x13\x5d\xf0\xe2\xc5\xc1" + "\xed\x14\xa9\x57\x19\x63\x40\x04" + "\x24\xeb\x6e\x19\xd1\x3d\x70\x78" + "\xeb\xda\x55\x70\x2c\x4f\x41\x5b" + "\x56\x9f\x1a\xd3\xac\xf1\xc0\xc3" + "\x21\xec\xd7\xd2\x55\x32\x7c\x2e" + "\x3c\x48\x8e\xb4\x85\x35\x47\xfe" + "\xe2\x88\x79\x98\x6a\xc9\x8d\xff" + "\xe9\x89\x6e\xb8\xe2\x97\x00\xbd" + "\xa4\x8f\xba\xd0\x8c\xcb\x79\x99" + "\xb3\xb2\xb2\x7a\xc3\xb7\xef\x75" + "\x23\x52\x76\xc3\x50\x6e\x66\xf8" + "\xa2\xe2\xce\xba\x40\x21\x3f\xc9" + "\x0a\x32\x7f\xf7\x08\x8c\x66\xcf" + "\xd3\xdf\x57\x59\x83\xb8\xe1\x85" + "\xd6\x8f\xfb\x48\x1f\x3a\xc4\x2f" + "\xb4\x2d\x58\xab\xd8\x7f\x5e\x3a" + "\xbc\x62\x3e\xe2\x6a\x52\x0d\x76" + "\x2f\x1c\x1a\x30\xed\x95\x2a\x44" + "\x35\xa5\x83\x04\x84\x01\x99\x56" + "\xb7\xe3\x10\x96\xfa\xdc\x19\xdd" + "\xe2\x7f\xcb\xa0\x49\x1b\xff\x4c" + "\x73\xf6\xbb\x94\x00\xe8\xa9\x3d" + "\xe2\x20\xe9\x3f\xfa\x07\x5d\x77" + "\x06\xd5\x4f\x4d\x02\xb8\x40\x1b" + "\x30\xed\x1a\x50\x19\xef\xc4\x2c" + "\x02\xd9\xc5\xd3\x11\x33\x37\xe5" + "\x2b\xa3\x95\xa6\xee\xd8\x74\x1d" + "\x68\xa0\xeb\xbf\xdd\x5e\x99\x96" + "\x91\xc3\x94\x24\xa5\x12\xa2\x37" + "\xb3\xac\xcf\x2a\xfd\x55\x34\xfe" + "\x79\x92\x3e\xe6\x1b\x49\x57\x5d" + "\x93\x6c\x01\xf7\xcc\x4e\x20\xd1" + "\xb2\x1a\xd8\x4c\xbd\x1d\x10\xe9" + "\x5a\xa8\x92\x7f\xba\xe6\x0c\x95", + .len = 512, .also_non_np = 1, .np = 3, .tap = { 512 - 20, 4, 16 }, }, }; -static const struct cipher_testvec camellia_xts_enc_tv_template[] = { +static const struct cipher_testvec camellia_xts_tv_template[] = { /* Generated from AES-XTS test vectors */ { .key = "\x00\x00\x00\x00\x00\x00\x00\x00" @@ -33106,16 +29313,15 @@ static const struct cipher_testvec camellia_xts_enc_tv_template[] = { .klen = 32, .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 32, - .result = "\x06\xcb\xa5\xf1\x04\x63\xb2\x41" + .ctext = "\x06\xcb\xa5\xf1\x04\x63\xb2\x41" "\xdc\xca\xfa\x09\xba\x74\xb9\x05" "\x78\xba\xa4\xf8\x67\x4d\x7e\xad" "\x20\x18\xf5\x0c\x41\x16\x2a\x61", - .rlen = 32, + .len = 32, }, { .key = "\x11\x11\x11\x11\x11\x11\x11\x11" "\x11\x11\x11\x11\x11\x11\x11\x11" @@ -33124,185 +29330,41 @@ static const struct cipher_testvec camellia_xts_enc_tv_template[] = { .klen = 32, .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" "\x44\x44\x44\x44\x44\x44\x44\x44" "\x44\x44\x44\x44\x44\x44\x44\x44" "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\xc2\xb9\xdc\x44\x1d\xdf\xf2\x86" + .ctext = "\xc2\xb9\xdc\x44\x1d\xdf\xf2\x86" "\x8d\x35\x42\x0a\xa5\x5e\x3d\x4f" "\xb5\x37\x06\xff\xbd\xd4\x91\x70" - "\x80\x1f\xb2\x39\x10\x89\x44\xf5", - .rlen = 32, - }, { - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .ilen = 32, - .result = "\x52\x1f\x9d\xf5\x5a\x58\x5a\x7e" - "\x9f\xd0\x8e\x02\x9c\x9a\x6a\xa7" - "\xb4\x3b\xce\xe7\x17\xaa\x89\x6a" - "\x35\x3c\x6b\xb5\x61\x1c\x79\x38", - .rlen = 32, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\xc7\xf9\x0a\xaa\xcb\xb5\x8f\x33" - "\x60\xc3\xe9\x47\x90\xb7\x50\x57" - "\xa3\xad\x81\x2f\xf5\x22\x96\x02" - "\xaa\x7f\xea\xac\x29\x78\xca\x2a" - "\x7c\xcd\x31\x1a\x3c\x40\x0a\x73" - "\x09\x66\xad\x72\x0e\x4d\x5d\x77" - "\xbc\xb8\x76\x80\x37\x59\xa9\x01" - "\x9e\xfb\xdb\x6c\x93\xef\xb6\x8d" - "\x1e\xc1\x94\xa8\xd4\xb5\xb0\x01" - "\xd5\x01\x97\x28\xcd\x7a\x1f\xe8" - "\x08\xda\x76\x00\x65\xcf\x7b\x31" - "\xc6\xfa\xf2\x3b\x00\xa7\x6a\x9e" - "\x6c\x43\x80\x87\xe0\xbb\x4e\xe5" - "\xdc\x8a\xdf\xc3\x1d\x1b\x41\x04" - "\xfb\x54\xdd\x29\x27\xc2\x65\x17" - "\x36\x88\xb0\x85\x8d\x73\x7e\x4b" - "\x1d\x16\x8a\x52\xbc\xa6\xbc\xa4" - "\x8c\xd1\x04\x16\xbf\x8c\x01\x0f" - "\x7e\x6b\x59\x15\x29\xd1\x9b\xd3" - "\x6c\xee\xac\xdc\x45\x58\xca\x5b" - "\x70\x0e\x6a\x12\x86\x82\x79\x9f" - "\x16\xd4\x9d\x67\xcd\x70\x65\x26" - "\x21\x72\x1e\xa1\x94\x8a\x83\x0c" - "\x92\x42\x58\x5e\xa2\xc5\x31\xf3" - "\x7b\xd1\x31\xd4\x15\x80\x31\x61" - "\x5c\x53\x10\xdd\xea\xc8\x83\x5c" - "\x7d\xa7\x05\x66\xcc\x1e\xbb\x05" - "\x47\xae\xb4\x0f\x84\xd8\xf6\xb5" - "\xa1\xc6\x52\x00\x52\xe8\xdc\xd9" - "\x16\x31\xb2\x47\x91\x67\xaa\x28" - "\x2c\x29\x85\xa3\xf7\xf2\x24\x93" - "\x23\x80\x1f\xa8\x1b\x82\x8d\xdc" - "\x9f\x0b\xcd\xb4\x3c\x20\xbc\xec" - "\x4f\xc7\xee\xf8\xfd\xd9\xfb\x7e" - "\x3f\x0d\x23\xfa\x3f\xa7\xcc\x66" - "\x1c\xfe\xa6\x86\xf6\xf7\x85\xc7" - "\x43\xc1\xd4\xfc\xe4\x79\xc9\x1d" - "\xf8\x89\xcd\x20\x27\x84\x5d\x5c" - "\x8e\x4f\x1f\xeb\x08\x21\x4f\xa3" - "\xe0\x7e\x0b\x9c\xe7\x42\xcf\xb7" - "\x3f\x43\xcc\x86\x71\x34\x6a\xd9" - "\x5e\xec\x8f\x36\xc9\x0a\x03\xfe" - "\x18\x41\xdc\x9e\x2e\x75\x20\x3e" - "\xcc\x77\xe0\x8f\xe8\x43\x37\x4c" - "\xed\x1a\x5a\xb3\xfa\x43\xc9\x71" - "\x9f\xc5\xce\xcf\xff\xe7\x77\x1e" - "\x35\x93\xde\x6b\xc0\x6a\x7e\xa9" - "\x34\xb8\x27\x74\x08\xda\xf2\x4a" - "\x23\x5b\x9f\x55\x3a\x57\x82\x52" - "\xea\x6d\xc3\xc7\xf2\xc8\xb5\xdc" - "\xc5\xb9\xbb\xaa\xf2\x29\x9f\x49" - "\x7a\xef\xfe\xdc\x9f\xc9\x28\xe2" - "\x96\x0b\x35\x84\x05\x0d\xd6\x2a" - "\xea\x5a\xbf\x69\xde\xee\x4f\x8f" - "\x84\xb9\xcf\xa7\x57\xea\xe0\xe8" - "\x96\xef\x0f\x0e\xec\xc7\xa6\x74" - "\xb1\xfe\x7a\x6d\x11\xdd\x0e\x15" - "\x4a\x1e\x73\x7f\x55\xea\xf6\xe1" - "\x5b\xb6\x71\xda\xb0\x0c\xba\x26" - "\x5c\x48\x38\x6d\x1c\x32\xb2\x7d" - "\x05\x87\xc2\x1e\x7e\x2d\xd4\x33" - "\xcc\x06\xdb\xe7\x82\x29\x63\xd1" - "\x52\x84\x4f\xee\x27\xe8\x02\xd4" - "\x34\x3c\x69\xc2\xbd\x20\xe6\x7a", - .rlen = 512, + "\x80\x1f\xb2\x39\x10\x89\x44\xf5", + .len = 32, + }, { + .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" + "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" + "\x22\x22\x22\x22\x22\x22\x22\x22" + "\x22\x22\x22\x22\x22\x22\x22\x22", + .klen = 32, + .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44" + "\x44\x44\x44\x44\x44\x44\x44\x44", + .ctext = "\x52\x1f\x9d\xf5\x5a\x58\x5a\x7e" + "\x9f\xd0\x8e\x02\x9c\x9a\x6a\xa7" + "\xb4\x3b\xce\xe7\x17\xaa\x89\x6a" + "\x35\x3c\x6b\xb5\x61\x1c\x79\x38", + .len = 32, }, { .key = "\x27\x18\x28\x18\x28\x45\x90\x45" "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" + "\x23\x84\x62\x64\x33\x83\x27\x95", + .klen = 32, + .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" "\x10\x11\x12\x13\x14\x15\x16\x17" "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" @@ -33366,144 +29428,7 @@ static const struct cipher_testvec camellia_xts_enc_tv_template[] = { "\xe8\xe9\xea\xeb\xec\xed\xee\xef" "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .ilen = 512, - .result = "\x49\xcd\xb8\xbf\x2f\x73\x37\x28" - "\x9a\x7f\x6e\x57\x55\xb8\x07\x88" - "\x4a\x0d\x8b\x55\x60\xed\xb6\x7b" - "\xf1\x74\xac\x96\x05\x7b\x32\xca" - "\xd1\x4e\xf1\x58\x29\x16\x24\x6c" - "\xf2\xb3\xe4\x88\x84\xac\x4d\xee" - "\x97\x07\x82\xf0\x07\x12\x38\x0a" - "\x67\x62\xaf\xfd\x85\x9f\x0a\x55" - "\xa5\x20\xc5\x60\xe4\x68\x53\xa4" - "\x0e\x2e\x65\xe3\xe4\x0c\x30\x7c" - "\x1c\x01\x4f\x55\xa9\x13\xeb\x25" - "\x21\x87\xbc\xd3\xe7\x67\x4f\x38" - "\xa8\x14\x25\x71\xe9\x2e\x4c\x21" - "\x41\x82\x0c\x45\x39\x35\xa8\x75" - "\x03\x29\x01\x84\x8c\xab\x48\xbe" - "\x11\x56\x22\x67\xb7\x67\x1a\x09" - "\xa1\x72\x25\x41\x3c\x39\x65\x80" - "\x7d\x2f\xf8\x2c\x73\x04\x58\x9d" - "\xdd\x16\x8b\x63\x70\x4e\xc5\x17" - "\x21\xe0\x84\x51\x4b\x6f\x05\x52" - "\xe3\x63\x34\xfa\xa4\xaf\x33\x20" - "\xc1\xae\x32\xc4\xb8\x2b\xdb\x76" - "\xd9\x02\x31\x2f\xa3\xc6\xd0\x7b" - "\xaf\x1b\x84\xe3\x9b\xbf\xa6\xe0" - "\xb8\x8a\x13\x88\x71\xf4\x11\xa5" - "\xe9\xa9\x10\x33\xe0\xbe\x49\x89" - "\x41\x22\xf5\x9d\x80\x3e\x3b\x76" - "\x01\x16\x50\x6e\x7c\x6a\x81\xe9" - "\x13\x2c\xde\xb2\x5f\x79\xba\xb2" - "\xb1\x75\xae\xd2\x07\x98\x4b\x69" - "\xae\x7d\x5b\x90\xc2\x6c\xe6\x98" - "\xd3\x4c\xa1\xa3\x9c\xc9\x33\x6a" - "\x0d\x23\xb1\x79\x25\x13\x4b\xe5" - "\xaf\x93\x20\x5c\x7f\x06\x7a\x34" - "\x0b\x78\xe3\x67\x26\xe0\xad\x95" - "\xc5\x4e\x26\x22\xcf\x73\x77\x62" - "\x3e\x10\xd7\x90\x4b\x52\x1c\xc9" - "\xef\x38\x52\x18\x0e\x29\x7e\xef" - "\x34\xfe\x31\x95\xc5\xbc\xa8\xe2" - "\xa8\x4e\x9f\xea\xa6\xf0\xfe\x5d" - "\xc5\x39\x86\xed\x2f\x6d\xa0\xfe" - "\x96\xcd\x41\x10\x78\x4e\x0c\xc9" - "\xc3\x6d\x0f\xb7\xe8\xe0\x62\xab" - "\x8b\xf1\x21\x89\xa1\x12\xaa\xfa" - "\x9d\x70\xbe\x4c\xa8\x98\x89\x01" - "\xb9\xe2\x61\xde\x0c\x4a\x0b\xaa" - "\x89\xf5\x14\x79\x18\x8f\x3b\x0d" - "\x21\x17\xf8\x59\x15\x24\x64\x22" - "\x57\x48\x80\xd5\x3d\x92\x30\x07" - "\xd9\xa1\x4a\x23\x16\x43\x48\x0e" - "\x2b\x2d\x1b\x87\xef\x7e\xbd\xfa" - "\x49\xbc\x7e\x68\x6e\xa8\x46\x95" - "\xad\x5e\xfe\x0a\xa8\xd3\x1a\x5d" - "\x6b\x84\xf3\x00\xba\x52\x05\x02" - "\xe3\x96\x4e\xb6\x79\x3f\x43\xd3" - "\x4d\x3f\xd6\xab\x0a\xc4\x75\x2d" - "\xd1\x08\xc3\x6a\xc8\x37\x29\xa0" - "\xcc\x9a\x05\xdd\x5c\xe1\xff\x66" - "\xf2\x7a\x1d\xf2\xaf\xa9\x48\x89" - "\xf5\x21\x0f\x02\x48\x83\x74\xbf" - "\x2e\xe6\x93\x7b\xa0\xf4\xb1\x2b" - "\xb1\x02\x0a\x5c\x79\x19\x3b\x75" - "\xb7\x16\xd8\x12\x5c\xcd\x7d\x4e" - "\xd5\xc6\x99\xcc\x4e\x6c\x94\x95", - .rlen = 512, - .also_non_np = 1, - .np = 3, - .tap = { 512 - 20, 4, 16 }, - }, -}; - -static const struct cipher_testvec camellia_xts_dec_tv_template[] = { - /* Generated from AES-XTS test vectors */ - /* same as enc vectors with input and result reversed */ - { - .key = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x06\xcb\xa5\xf1\x04\x63\xb2\x41" - "\xdc\xca\xfa\x09\xba\x74\xb9\x05" - "\x78\xba\xa4\xf8\x67\x4d\x7e\xad" - "\x20\x18\xf5\x0c\x41\x16\x2a\x61", - .ilen = 32, - .result = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .rlen = 32, - }, { - .key = "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x11\x11\x11\x11\x11\x11\x11\x11" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xc2\xb9\xdc\x44\x1d\xdf\xf2\x86" - "\x8d\x35\x42\x0a\xa5\x5e\x3d\x4f" - "\xb5\x37\x06\xff\xbd\xd4\x91\x70" - "\x80\x1f\xb2\x39\x10\x89\x44\xf5", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { - .key = "\xff\xfe\xfd\xfc\xfb\xfa\xf9\xf8" - "\xf7\xf6\xf5\xf4\xf3\xf2\xf1\xf0" - "\x22\x22\x22\x22\x22\x22\x22\x22" - "\x22\x22\x22\x22\x22\x22\x22\x22", - .klen = 32, - .iv = "\x33\x33\x33\x33\x33\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x52\x1f\x9d\xf5\x5a\x58\x5a\x7e" - "\x9f\xd0\x8e\x02\x9c\x9a\x6a\xa7" - "\xb4\x3b\xce\xe7\x17\xaa\x89\x6a" - "\x35\x3c\x6b\xb5\x61\x1c\x79\x38", - .ilen = 32, - .result = "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44" - "\x44\x44\x44\x44\x44\x44\x44\x44", - .rlen = 32, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95", - .klen = 32, - .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\xc7\xf9\x0a\xaa\xcb\xb5\x8f\x33" + .ctext = "\xc7\xf9\x0a\xaa\xcb\xb5\x8f\x33" "\x60\xc3\xe9\x47\x90\xb7\x50\x57" "\xa3\xad\x81\x2f\xf5\x22\x96\x02" "\xaa\x7f\xea\xac\x29\x78\xca\x2a" @@ -33567,8 +29492,20 @@ static const struct cipher_testvec camellia_xts_dec_tv_template[] = { "\xcc\x06\xdb\xe7\x82\x29\x63\xd1" "\x52\x84\x4f\xee\x27\xe8\x02\xd4" "\x34\x3c\x69\xc2\xbd\x20\xe6\x7a", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" + .len = 512, + }, { + .key = "\x27\x18\x28\x18\x28\x45\x90\x45" + "\x23\x53\x60\x28\x74\x71\x35\x26" + "\x62\x49\x77\x57\x24\x70\x93\x69" + "\x99\x59\x57\x49\x66\x96\x76\x27" + "\x31\x41\x59\x26\x53\x58\x97\x93" + "\x23\x84\x62\x64\x33\x83\x27\x95" + "\x02\x88\x41\x97\x16\x93\x99\x37" + "\x51\x05\x82\x09\x74\x94\x45\x92", + .klen = 64, + .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" + "\x00\x00\x00\x00\x00\x00\x00\x00", + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" "\x10\x11\x12\x13\x14\x15\x16\x17" "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" @@ -33632,20 +29569,7 @@ static const struct cipher_testvec camellia_xts_dec_tv_template[] = { "\xe8\xe9\xea\xeb\xec\xed\xee\xef" "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, - }, { - .key = "\x27\x18\x28\x18\x28\x45\x90\x45" - "\x23\x53\x60\x28\x74\x71\x35\x26" - "\x62\x49\x77\x57\x24\x70\x93\x69" - "\x99\x59\x57\x49\x66\x96\x76\x27" - "\x31\x41\x59\x26\x53\x58\x97\x93" - "\x23\x84\x62\x64\x33\x83\x27\x95" - "\x02\x88\x41\x97\x16\x93\x99\x37" - "\x51\x05\x82\x09\x74\x94\x45\x92", - .klen = 64, - .iv = "\xff\x00\x00\x00\x00\x00\x00\x00" - "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x49\xcd\xb8\xbf\x2f\x73\x37\x28" + .ctext = "\x49\xcd\xb8\xbf\x2f\x73\x37\x28" "\x9a\x7f\x6e\x57\x55\xb8\x07\x88" "\x4a\x0d\x8b\x55\x60\xed\xb6\x7b" "\xf1\x74\xac\x96\x05\x7b\x32\xca" @@ -33691,90 +29615,25 @@ static const struct cipher_testvec camellia_xts_dec_tv_template[] = { "\x8b\xf1\x21\x89\xa1\x12\xaa\xfa" "\x9d\x70\xbe\x4c\xa8\x98\x89\x01" "\xb9\xe2\x61\xde\x0c\x4a\x0b\xaa" - "\x89\xf5\x14\x79\x18\x8f\x3b\x0d" - "\x21\x17\xf8\x59\x15\x24\x64\x22" - "\x57\x48\x80\xd5\x3d\x92\x30\x07" - "\xd9\xa1\x4a\x23\x16\x43\x48\x0e" - "\x2b\x2d\x1b\x87\xef\x7e\xbd\xfa" - "\x49\xbc\x7e\x68\x6e\xa8\x46\x95" - "\xad\x5e\xfe\x0a\xa8\xd3\x1a\x5d" - "\x6b\x84\xf3\x00\xba\x52\x05\x02" - "\xe3\x96\x4e\xb6\x79\x3f\x43\xd3" - "\x4d\x3f\xd6\xab\x0a\xc4\x75\x2d" - "\xd1\x08\xc3\x6a\xc8\x37\x29\xa0" - "\xcc\x9a\x05\xdd\x5c\xe1\xff\x66" - "\xf2\x7a\x1d\xf2\xaf\xa9\x48\x89" - "\xf5\x21\x0f\x02\x48\x83\x74\xbf" - "\x2e\xe6\x93\x7b\xa0\xf4\xb1\x2b" - "\xb1\x02\x0a\x5c\x79\x19\x3b\x75" - "\xb7\x16\xd8\x12\x5c\xcd\x7d\x4e" - "\xd5\xc6\x99\xcc\x4e\x6c\x94\x95", - .ilen = 512, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" - "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" - "\x10\x11\x12\x13\x14\x15\x16\x17" - "\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" - "\x20\x21\x22\x23\x24\x25\x26\x27" - "\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f" - "\x30\x31\x32\x33\x34\x35\x36\x37" - "\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f" - "\x40\x41\x42\x43\x44\x45\x46\x47" - "\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f" - "\x50\x51\x52\x53\x54\x55\x56\x57" - "\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f" - "\x60\x61\x62\x63\x64\x65\x66\x67" - "\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f" - "\x70\x71\x72\x73\x74\x75\x76\x77" - "\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f" - "\x80\x81\x82\x83\x84\x85\x86\x87" - "\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f" - "\x90\x91\x92\x93\x94\x95\x96\x97" - "\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f" - "\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7" - "\xa8\xa9\xaa\xab\xac\xad\xae\xaf" - "\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7" - "\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf" - "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7" - "\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf" - "\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7" - "\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf" - "\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7" - "\xe8\xe9\xea\xeb\xec\xed\xee\xef" - "\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7" - "\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff", - .rlen = 512, + "\x89\xf5\x14\x79\x18\x8f\x3b\x0d" + "\x21\x17\xf8\x59\x15\x24\x64\x22" + "\x57\x48\x80\xd5\x3d\x92\x30\x07" + "\xd9\xa1\x4a\x23\x16\x43\x48\x0e" + "\x2b\x2d\x1b\x87\xef\x7e\xbd\xfa" + "\x49\xbc\x7e\x68\x6e\xa8\x46\x95" + "\xad\x5e\xfe\x0a\xa8\xd3\x1a\x5d" + "\x6b\x84\xf3\x00\xba\x52\x05\x02" + "\xe3\x96\x4e\xb6\x79\x3f\x43\xd3" + "\x4d\x3f\xd6\xab\x0a\xc4\x75\x2d" + "\xd1\x08\xc3\x6a\xc8\x37\x29\xa0" + "\xcc\x9a\x05\xdd\x5c\xe1\xff\x66" + "\xf2\x7a\x1d\xf2\xaf\xa9\x48\x89" + "\xf5\x21\x0f\x02\x48\x83\x74\xbf" + "\x2e\xe6\x93\x7b\xa0\xf4\xb1\x2b" + "\xb1\x02\x0a\x5c\x79\x19\x3b\x75" + "\xb7\x16\xd8\x12\x5c\xcd\x7d\x4e" + "\xd5\xc6\x99\xcc\x4e\x6c\x94\x95", + .len = 512, .also_non_np = 1, .np = 3, .tap = { 512 - 20, 4, 16 }, @@ -33784,91 +29643,45 @@ static const struct cipher_testvec camellia_xts_dec_tv_template[] = { /* * SEED test vectors */ -static const struct cipher_testvec seed_enc_tv_template[] = { +static const struct cipher_testvec seed_tv_template[] = { { .key = zeroed_string, .klen = 16, - .input = "\x00\x01\x02\x03\x04\x05\x06\x07" + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .ilen = 16, - .result = "\x5e\xba\xc6\xe0\x05\x4e\x16\x68" + .ctext = "\x5e\xba\xc6\xe0\x05\x4e\x16\x68" "\x19\xaf\xf1\xcc\x6d\x34\x6c\xdb", - .rlen = 16, + .len = 16, }, { .key = "\x00\x01\x02\x03\x04\x05\x06\x07" "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", .klen = 16, - .input = zeroed_string, - .ilen = 16, - .result = "\xc1\x1f\x22\xf2\x01\x40\x50\x50" + .ptext = zeroed_string, + .ctext = "\xc1\x1f\x22\xf2\x01\x40\x50\x50" "\x84\x48\x35\x97\xe4\x37\x0f\x43", - .rlen = 16, + .len = 16, }, { .key = "\x47\x06\x48\x08\x51\xe6\x1b\xe8" "\x5d\x74\xbf\xb3\xfd\x95\x61\x85", .klen = 16, - .input = "\x83\xa2\xf8\xa2\x88\x64\x1f\xb9" + .ptext = "\x83\xa2\xf8\xa2\x88\x64\x1f\xb9" "\xa4\xe9\xa5\xcc\x2f\x13\x1c\x7d", - .ilen = 16, - .result = "\xee\x54\xd1\x3e\xbc\xae\x70\x6d" + .ctext = "\xee\x54\xd1\x3e\xbc\xae\x70\x6d" "\x22\x6b\xc3\x14\x2c\xd4\x0d\x4a", - .rlen = 16, + .len = 16, }, { .key = "\x28\xdb\xc3\xbc\x49\xff\xd8\x7d" "\xcf\xa5\x09\xb1\x1d\x42\x2b\xe7", .klen = 16, - .input = "\xb4\x1e\x6b\xe2\xeb\xa8\x4a\x14" + .ptext = "\xb4\x1e\x6b\xe2\xeb\xa8\x4a\x14" "\x8e\x2e\xed\x84\x59\x3c\x5e\xc7", - .ilen = 16, - .result = "\x9b\x9b\x7b\xfc\xd1\x81\x3c\xb9" - "\x5d\x0b\x36\x18\xf4\x0f\x51\x22", - .rlen = 16, - } -}; - -static const struct cipher_testvec seed_dec_tv_template[] = { - { - .key = zeroed_string, - .klen = 16, - .input = "\x5e\xba\xc6\xe0\x05\x4e\x16\x68" - "\x19\xaf\xf1\xcc\x6d\x34\x6c\xdb", - .ilen = 16, - .result = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .rlen = 16, - }, { - .key = "\x00\x01\x02\x03\x04\x05\x06\x07" - "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", - .klen = 16, - .input = "\xc1\x1f\x22\xf2\x01\x40\x50\x50" - "\x84\x48\x35\x97\xe4\x37\x0f\x43", - .ilen = 16, - .result = zeroed_string, - .rlen = 16, - }, { - .key = "\x47\x06\x48\x08\x51\xe6\x1b\xe8" - "\x5d\x74\xbf\xb3\xfd\x95\x61\x85", - .klen = 16, - .input = "\xee\x54\xd1\x3e\xbc\xae\x70\x6d" - "\x22\x6b\xc3\x14\x2c\xd4\x0d\x4a", - .ilen = 16, - .result = "\x83\xa2\xf8\xa2\x88\x64\x1f\xb9" - "\xa4\xe9\xa5\xcc\x2f\x13\x1c\x7d", - .rlen = 16, - }, { - .key = "\x28\xdb\xc3\xbc\x49\xff\xd8\x7d" - "\xcf\xa5\x09\xb1\x1d\x42\x2b\xe7", - .klen = 16, - .input = "\x9b\x9b\x7b\xfc\xd1\x81\x3c\xb9" + .ctext = "\x9b\x9b\x7b\xfc\xd1\x81\x3c\xb9" "\x5d\x0b\x36\x18\xf4\x0f\x51\x22", - .ilen = 16, - .result = "\xb4\x1e\x6b\xe2\xeb\xa8\x4a\x14" - "\x8e\x2e\xed\x84\x59\x3c\x5e\xc7", - .rlen = 16, + .len = 16, } }; -static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { +static const struct cipher_testvec salsa20_stream_tv_template[] = { /* * Testvectors from verified.test-vectors submitted to ECRYPT. * They are truncated to size 39, 64, 111, 129 to test a variety @@ -33879,24 +29692,23 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { "\x08\x09\x0A\x0B\x0C\x0D\x0E\x0F", .klen = 16, .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00", - .ilen = 39, - .result = "\x2D\xD5\xC3\xF7\xBA\x2B\x20\xF7" + .ctext = "\x2D\xD5\xC3\xF7\xBA\x2B\x20\xF7" "\x68\x02\x41\x0C\x68\x86\x88\x89" "\x5A\xD8\xC1\xBD\x4E\xA6\xC9\xB1" "\x40\xFB\x9B\x90\xE2\x10\x49\xBF" "\x58\x3F\x52\x79\x70\xEB\xC1", - .rlen = 39, + .len = 39, }, { /* Set 5, vector 0 */ .key = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", .klen = 16, .iv = "\x80\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" @@ -33904,8 +29716,7 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 64, - .result = "\xB6\x6C\x1E\x44\x46\xDD\x95\x57" + .ctext = "\xB6\x6C\x1E\x44\x46\xDD\x95\x57" "\xE5\x78\xE2\x23\xB0\xB7\x68\x01" "\x7B\x23\xB2\x67\xBB\x02\x34\xAE" "\x46\x26\xBF\x44\x3F\x21\x97\x76" @@ -33913,7 +29724,7 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { "\xCD\x0D\xE9\xA9\x53\x8F\x4A\x09" "\xCA\x9A\xC0\x73\x2E\x30\xBC\xF9" "\x8E\x4F\x13\xE4\xB9\xE2\x01\xD9", - .rlen = 64, + .len = 64, }, { /* Set 3, vector 27 */ .key = "\x1B\x1C\x1D\x1E\x1F\x20\x21\x22" "\x23\x24\x25\x26\x27\x28\x29\x2A" @@ -33921,7 +29732,7 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { "\x33\x34\x35\x36\x37\x38\x39\x3A", .klen = 32, .iv = "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" @@ -33935,8 +29746,7 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00", - .ilen = 111, - .result = "\xAE\x39\x50\x8E\xAC\x9A\xEC\xE7" + .ctext = "\xAE\x39\x50\x8E\xAC\x9A\xEC\xE7" "\xBF\x97\xBB\x20\xB9\xDE\xE4\x1F" "\x87\xD9\x47\xF8\x28\x91\x35\x98" "\xDB\x72\xCC\x23\x29\x48\x56\x5E" @@ -33950,7 +29760,7 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { "\xB1\x41\x3F\x19\x2F\xC4\x3B\xC6" "\x95\x46\x45\x54\xE9\x75\x03\x08" "\x44\xAF\xE5\x8A\x81\x12\x09", - .rlen = 111, + .len = 111, }, { /* Set 5, vector 27 */ .key = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" @@ -33958,7 +29768,7 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { "\x00\x00\x00\x00\x00\x00\x00\x00", .klen = 32, .iv = "\x00\x00\x00\x10\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" @@ -33975,8 +29785,7 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00", - .ilen = 129, - .result = "\xD2\xDB\x1A\x5C\xF1\xC1\xAC\xDB" + .ctext = "\xD2\xDB\x1A\x5C\xF1\xC1\xAC\xDB" "\xE8\x1A\x7A\x43\x40\xEF\x53\x43" "\x5E\x7F\x4B\x1A\x50\x52\x3F\x8D" "\x28\x3D\xCF\x85\x1D\x69\x6E\x60" @@ -33993,7 +29802,7 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { "\x2E\x40\x48\x75\xE9\xE2\x21\x45" "\x0B\xC9\xB6\xB5\x66\xBC\x9A\x59" "\x5A", - .rlen = 129, + .len = 129, }, { /* large test vector generated using Crypto++ */ .key = "\x00\x01\x02\x03\x04\x05\x06\x07" "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" @@ -34002,7 +29811,7 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { .klen = 32, .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = + .ptext = "\x00\x01\x02\x03\x04\x05\x06\x07" "\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f" "\x10\x11\x12\x13\x14\x15\x16\x17" @@ -34516,8 +30325,7 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { "\x10\x2f\x4e\x6d\x8c\xab\xca\xe9" "\x08\x27\x46\x65\x84\xa3\xc2\xe1" "\x00\x21\x42\x63", - .ilen = 4100, - .result = + .ctext = "\xb5\x81\xf5\x64\x18\x73\xe3\xf0" "\x4c\x13\xf2\x77\x18\x60\x65\x5e" "\x29\x01\xce\x98\x55\x53\xf9\x0c" @@ -35031,13 +30839,13 @@ static const struct cipher_testvec salsa20_stream_enc_tv_template[] = { "\xfc\x3f\x09\x7a\x0b\xdc\xc5\x1b" "\x87\x13\xc6\x5b\x59\x8d\xf2\xc8" "\xaf\xdf\x11\x95", - .rlen = 4100, + .len = 4100, .np = 2, .tap = { 4064, 36 }, }, }; -static const struct cipher_testvec chacha20_enc_tv_template[] = { +static const struct cipher_testvec chacha20_tv_template[] = { { /* RFC7539 A.2. Test Vector #1 */ .key = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" @@ -35046,7 +30854,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { .klen = 32, .iv = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .input = "\x00\x00\x00\x00\x00\x00\x00\x00" + .ptext = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" @@ -35054,8 +30862,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00", - .ilen = 64, - .result = "\x76\xb8\xe0\xad\xa0\xf1\x3d\x90" + .ctext = "\x76\xb8\xe0\xad\xa0\xf1\x3d\x90" "\x40\x5d\x6a\xe5\x53\x86\xbd\x28" "\xbd\xd2\x19\xb8\xa0\x8d\xed\x1a" "\xa8\x36\xef\xcc\x8b\x77\x0d\xc7" @@ -35063,7 +30870,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { "\x77\x24\xe0\x3f\xb8\xd8\x4a\x37" "\x6a\x43\xb8\xf4\x15\x18\xa1\x1c" "\xc3\x87\xb6\x69\xb2\xee\x65\x86", - .rlen = 64, + .len = 64, }, { /* RFC7539 A.2. Test Vector #2 */ .key = "\x00\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x00" @@ -35072,7 +30879,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { .klen = 32, .iv = "\x01\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x02", - .input = "\x41\x6e\x79\x20\x73\x75\x62\x6d" + .ptext = "\x41\x6e\x79\x20\x73\x75\x62\x6d" "\x69\x73\x73\x69\x6f\x6e\x20\x74" "\x6f\x20\x74\x68\x65\x20\x49\x45" "\x54\x46\x20\x69\x6e\x74\x65\x6e" @@ -35119,8 +30926,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { "\x20\x77\x68\x69\x63\x68\x20\x61" "\x72\x65\x20\x61\x64\x64\x72\x65" "\x73\x73\x65\x64\x20\x74\x6f", - .ilen = 375, - .result = "\xa3\xfb\xf0\x7d\xf3\xfa\x2f\xde" + .ctext = "\xa3\xfb\xf0\x7d\xf3\xfa\x2f\xde" "\x4f\x37\x6c\xa2\x3e\x82\x73\x70" "\x41\x60\x5d\x9f\x4f\x4f\x57\xbd" "\x8c\xff\x2c\x1d\x4b\x79\x55\xec" @@ -35167,7 +30973,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { "\x7a\xc6\x1d\xd2\x9c\x6f\x21\xba" "\x5b\x86\x2f\x37\x30\xe3\x7c\xfd" "\xc4\xfd\x80\x6c\x22\xf2\x21", - .rlen = 375, + .len = 375, .also_non_np = 1, .np = 3, .tap = { 375 - 20, 4, 16 }, @@ -35180,7 +30986,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { .klen = 32, .iv = "\x2a\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x02", - .input = "\x27\x54\x77\x61\x73\x20\x62\x72" + .ptext = "\x27\x54\x77\x61\x73\x20\x62\x72" "\x69\x6c\x6c\x69\x67\x2c\x20\x61" "\x6e\x64\x20\x74\x68\x65\x20\x73" "\x6c\x69\x74\x68\x79\x20\x74\x6f" @@ -35196,8 +31002,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { "\x68\x65\x20\x6d\x6f\x6d\x65\x20" "\x72\x61\x74\x68\x73\x20\x6f\x75" "\x74\x67\x72\x61\x62\x65\x2e", - .ilen = 127, - .result = "\x62\xe6\x34\x7f\x95\xed\x87\xa4" + .ctext = "\x62\xe6\x34\x7f\x95\xed\x87\xa4" "\x5f\xfa\xe7\x42\x6f\x27\xa1\xdf" "\x5f\xb6\x91\x10\x04\x4c\x0d\x73" "\x11\x8e\xff\xa9\x5b\x01\xe5\xcf" @@ -35213,7 +31018,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { "\x50\xd6\x15\x4b\x6d\xa7\x31\xb1" "\x87\xb5\x8d\xfd\x72\x8a\xfa\x36" "\x75\x7a\x79\x7a\xc1\x88\xd1", - .rlen = 127, + .len = 127, }, { /* Self-made test vector for long data */ .key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a" "\xf3\x33\x88\x86\x04\xf6\xb5\xf0" @@ -35222,7 +31027,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { .klen = 32, .iv = "\x1c\x00\x00\x00\x00\x00\x00\x00" "\x00\x00\x00\x00\x00\x00\x00\x01", - .input = "\x49\xee\xe0\xdc\x24\x90\x40\xcd" + .ptext = "\x49\xee\xe0\xdc\x24\x90\x40\xcd" "\xc5\x40\x8f\x47\x05\xbc\xdd\x81" "\x47\xc6\x8d\xe6\xb1\x8f\xd7\xcb" "\x09\x0e\x6e\x22\x48\x1f\xbf\xb8" @@ -35383,8 +31188,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { "\x08\x7c\xbc\x66\x8a\xb0\xb6\x9f" "\x92\xd6\x41\x7c\x5b\x2a\x00\x79" "\x72", - .ilen = 1281, - .result = "\x45\xe8\xe0\xb6\x9c\xca\xfd\x87" + .ctext = "\x45\xe8\xe0\xb6\x9c\xca\xfd\x87" "\xe8\x1d\x37\x96\x8a\xe3\x40\x35" "\xcf\x5e\x3a\x46\x3d\xfb\xd0\x69" "\xde\xaf\x7a\xd5\x0d\xe9\x52\xec" @@ -35545,7 +31349,7 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { "\x23\x45\x89\x42\xa0\x30\xeb\xbf" "\xa1\xed\xad\xd5\x76\xfa\x24\x8f" "\x98", - .rlen = 1281, + .len = 1281, .also_non_np = 1, .np = 3, .tap = { 1200, 1, 80 }, @@ -35555,132 +31359,28 @@ static const struct cipher_testvec chacha20_enc_tv_template[] = { /* * CTS (Cipher Text Stealing) mode tests */ -static const struct cipher_testvec cts_mode_enc_tv_template[] = { - { /* from rfc3962 */ - .klen = 16, - .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" - "\x74\x65\x72\x69\x79\x61\x6b\x69", - .ilen = 17, - .input = "\x49\x20\x77\x6f\x75\x6c\x64\x20" - "\x6c\x69\x6b\x65\x20\x74\x68\x65" - "\x20", - .rlen = 17, - .result = "\xc6\x35\x35\x68\xf2\xbf\x8c\xb4" - "\xd8\xa5\x80\x36\x2d\xa7\xff\x7f" - "\x97", - }, { - .klen = 16, - .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" - "\x74\x65\x72\x69\x79\x61\x6b\x69", - .ilen = 31, - .input = "\x49\x20\x77\x6f\x75\x6c\x64\x20" - "\x6c\x69\x6b\x65\x20\x74\x68\x65" - "\x20\x47\x65\x6e\x65\x72\x61\x6c" - "\x20\x47\x61\x75\x27\x73\x20", - .rlen = 31, - .result = "\xfc\x00\x78\x3e\x0e\xfd\xb2\xc1" - "\xd4\x45\xd4\xc8\xef\xf7\xed\x22" - "\x97\x68\x72\x68\xd6\xec\xcc\xc0" - "\xc0\x7b\x25\xe2\x5e\xcf\xe5", - }, { - .klen = 16, - .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" - "\x74\x65\x72\x69\x79\x61\x6b\x69", - .ilen = 32, - .input = "\x49\x20\x77\x6f\x75\x6c\x64\x20" - "\x6c\x69\x6b\x65\x20\x74\x68\x65" - "\x20\x47\x65\x6e\x65\x72\x61\x6c" - "\x20\x47\x61\x75\x27\x73\x20\x43", - .rlen = 32, - .result = "\x39\x31\x25\x23\xa7\x86\x62\xd5" - "\xbe\x7f\xcb\xcc\x98\xeb\xf5\xa8" - "\x97\x68\x72\x68\xd6\xec\xcc\xc0" - "\xc0\x7b\x25\xe2\x5e\xcf\xe5\x84", - }, { - .klen = 16, - .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" - "\x74\x65\x72\x69\x79\x61\x6b\x69", - .ilen = 47, - .input = "\x49\x20\x77\x6f\x75\x6c\x64\x20" - "\x6c\x69\x6b\x65\x20\x74\x68\x65" - "\x20\x47\x65\x6e\x65\x72\x61\x6c" - "\x20\x47\x61\x75\x27\x73\x20\x43" - "\x68\x69\x63\x6b\x65\x6e\x2c\x20" - "\x70\x6c\x65\x61\x73\x65\x2c", - .rlen = 47, - .result = "\x97\x68\x72\x68\xd6\xec\xcc\xc0" - "\xc0\x7b\x25\xe2\x5e\xcf\xe5\x84" - "\xb3\xff\xfd\x94\x0c\x16\xa1\x8c" - "\x1b\x55\x49\xd2\xf8\x38\x02\x9e" - "\x39\x31\x25\x23\xa7\x86\x62\xd5" - "\xbe\x7f\xcb\xcc\x98\xeb\xf5", - }, { - .klen = 16, - .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" - "\x74\x65\x72\x69\x79\x61\x6b\x69", - .ilen = 48, - .input = "\x49\x20\x77\x6f\x75\x6c\x64\x20" - "\x6c\x69\x6b\x65\x20\x74\x68\x65" - "\x20\x47\x65\x6e\x65\x72\x61\x6c" - "\x20\x47\x61\x75\x27\x73\x20\x43" - "\x68\x69\x63\x6b\x65\x6e\x2c\x20" - "\x70\x6c\x65\x61\x73\x65\x2c\x20", - .rlen = 48, - .result = "\x97\x68\x72\x68\xd6\xec\xcc\xc0" - "\xc0\x7b\x25\xe2\x5e\xcf\xe5\x84" - "\x9d\xad\x8b\xbb\x96\xc4\xcd\xc0" - "\x3b\xc1\x03\xe1\xa1\x94\xbb\xd8" - "\x39\x31\x25\x23\xa7\x86\x62\xd5" - "\xbe\x7f\xcb\xcc\x98\xeb\xf5\xa8", - }, { - .klen = 16, - .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" - "\x74\x65\x72\x69\x79\x61\x6b\x69", - .ilen = 64, - .input = "\x49\x20\x77\x6f\x75\x6c\x64\x20" - "\x6c\x69\x6b\x65\x20\x74\x68\x65" - "\x20\x47\x65\x6e\x65\x72\x61\x6c" - "\x20\x47\x61\x75\x27\x73\x20\x43" - "\x68\x69\x63\x6b\x65\x6e\x2c\x20" - "\x70\x6c\x65\x61\x73\x65\x2c\x20" - "\x61\x6e\x64\x20\x77\x6f\x6e\x74" - "\x6f\x6e\x20\x73\x6f\x75\x70\x2e", - .rlen = 64, - .result = "\x97\x68\x72\x68\xd6\xec\xcc\xc0" - "\xc0\x7b\x25\xe2\x5e\xcf\xe5\x84" - "\x39\x31\x25\x23\xa7\x86\x62\xd5" - "\xbe\x7f\xcb\xcc\x98\xeb\xf5\xa8" - "\x48\x07\xef\xe8\x36\xee\x89\xa5" - "\x26\x73\x0d\xbc\x2f\x7b\xc8\x40" - "\x9d\xad\x8b\xbb\x96\xc4\xcd\xc0" - "\x3b\xc1\x03\xe1\xa1\x94\xbb\xd8", - } -}; - -static const struct cipher_testvec cts_mode_dec_tv_template[] = { +static const struct cipher_testvec cts_mode_tv_template[] = { { /* from rfc3962 */ .klen = 16, .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" "\x74\x65\x72\x69\x79\x61\x6b\x69", - .rlen = 17, - .result = "\x49\x20\x77\x6f\x75\x6c\x64\x20" + .ptext = "\x49\x20\x77\x6f\x75\x6c\x64\x20" "\x6c\x69\x6b\x65\x20\x74\x68\x65" "\x20", - .ilen = 17, - .input = "\xc6\x35\x35\x68\xf2\xbf\x8c\xb4" + .len = 17, + .ctext = "\xc6\x35\x35\x68\xf2\xbf\x8c\xb4" "\xd8\xa5\x80\x36\x2d\xa7\xff\x7f" "\x97", }, { .klen = 16, .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" "\x74\x65\x72\x69\x79\x61\x6b\x69", - .rlen = 31, - .result = "\x49\x20\x77\x6f\x75\x6c\x64\x20" + .ptext = "\x49\x20\x77\x6f\x75\x6c\x64\x20" "\x6c\x69\x6b\x65\x20\x74\x68\x65" "\x20\x47\x65\x6e\x65\x72\x61\x6c" "\x20\x47\x61\x75\x27\x73\x20", - .ilen = 31, - .input = "\xfc\x00\x78\x3e\x0e\xfd\xb2\xc1" + .len = 31, + .ctext = "\xfc\x00\x78\x3e\x0e\xfd\xb2\xc1" "\xd4\x45\xd4\xc8\xef\xf7\xed\x22" "\x97\x68\x72\x68\xd6\xec\xcc\xc0" "\xc0\x7b\x25\xe2\x5e\xcf\xe5", @@ -35688,13 +31388,12 @@ static const struct cipher_testvec cts_mode_dec_tv_template[] = { .klen = 16, .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" "\x74\x65\x72\x69\x79\x61\x6b\x69", - .rlen = 32, - .result = "\x49\x20\x77\x6f\x75\x6c\x64\x20" + .ptext = "\x49\x20\x77\x6f\x75\x6c\x64\x20" "\x6c\x69\x6b\x65\x20\x74\x68\x65" "\x20\x47\x65\x6e\x65\x72\x61\x6c" "\x20\x47\x61\x75\x27\x73\x20\x43", - .ilen = 32, - .input = "\x39\x31\x25\x23\xa7\x86\x62\xd5" + .len = 32, + .ctext = "\x39\x31\x25\x23\xa7\x86\x62\xd5" "\xbe\x7f\xcb\xcc\x98\xeb\xf5\xa8" "\x97\x68\x72\x68\xd6\xec\xcc\xc0" "\xc0\x7b\x25\xe2\x5e\xcf\xe5\x84", @@ -35702,15 +31401,14 @@ static const struct cipher_testvec cts_mode_dec_tv_template[] = { .klen = 16, .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" "\x74\x65\x72\x69\x79\x61\x6b\x69", - .rlen = 47, - .result = "\x49\x20\x77\x6f\x75\x6c\x64\x20" + .ptext = "\x49\x20\x77\x6f\x75\x6c\x64\x20" "\x6c\x69\x6b\x65\x20\x74\x68\x65" "\x20\x47\x65\x6e\x65\x72\x61\x6c" "\x20\x47\x61\x75\x27\x73\x20\x43" "\x68\x69\x63\x6b\x65\x6e\x2c\x20" "\x70\x6c\x65\x61\x73\x65\x2c", - .ilen = 47, - .input = "\x97\x68\x72\x68\xd6\xec\xcc\xc0" + .len = 47, + .ctext = "\x97\x68\x72\x68\xd6\xec\xcc\xc0" "\xc0\x7b\x25\xe2\x5e\xcf\xe5\x84" "\xb3\xff\xfd\x94\x0c\x16\xa1\x8c" "\x1b\x55\x49\xd2\xf8\x38\x02\x9e" @@ -35720,15 +31418,14 @@ static const struct cipher_testvec cts_mode_dec_tv_template[] = { .klen = 16, .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" "\x74\x65\x72\x69\x79\x61\x6b\x69", - .rlen = 48, - .result = "\x49\x20\x77\x6f\x75\x6c\x64\x20" + .ptext = "\x49\x20\x77\x6f\x75\x6c\x64\x20" "\x6c\x69\x6b\x65\x20\x74\x68\x65" "\x20\x47\x65\x6e\x65\x72\x61\x6c" "\x20\x47\x61\x75\x27\x73\x20\x43" "\x68\x69\x63\x6b\x65\x6e\x2c\x20" "\x70\x6c\x65\x61\x73\x65\x2c\x20", - .ilen = 48, - .input = "\x97\x68\x72\x68\xd6\xec\xcc\xc0" + .len = 48, + .ctext = "\x97\x68\x72\x68\xd6\xec\xcc\xc0" "\xc0\x7b\x25\xe2\x5e\xcf\xe5\x84" "\x9d\xad\x8b\xbb\x96\xc4\xcd\xc0" "\x3b\xc1\x03\xe1\xa1\x94\xbb\xd8" @@ -35738,8 +31435,7 @@ static const struct cipher_testvec cts_mode_dec_tv_template[] = { .klen = 16, .key = "\x63\x68\x69\x63\x6b\x65\x6e\x20" "\x74\x65\x72\x69\x79\x61\x6b\x69", - .rlen = 64, - .result = "\x49\x20\x77\x6f\x75\x6c\x64\x20" + .ptext = "\x49\x20\x77\x6f\x75\x6c\x64\x20" "\x6c\x69\x6b\x65\x20\x74\x68\x65" "\x20\x47\x65\x6e\x65\x72\x61\x6c" "\x20\x47\x61\x75\x27\x73\x20\x43" @@ -35747,8 +31443,8 @@ static const struct cipher_testvec cts_mode_dec_tv_template[] = { "\x70\x6c\x65\x61\x73\x65\x2c\x20" "\x61\x6e\x64\x20\x77\x6f\x6e\x74" "\x6f\x6e\x20\x73\x6f\x75\x70\x2e", - .ilen = 64, - .input = "\x97\x68\x72\x68\xd6\xec\xcc\xc0" + .len = 64, + .ctext = "\x97\x68\x72\x68\xd6\xec\xcc\xc0" "\xc0\x7b\x25\xe2\x5e\xcf\xe5\x84" "\x39\x31\x25\x23\xa7\x86\x62\xd5" "\xbe\x7f\xcb\xcc\x98\xeb\xf5\xa8" @@ -36057,6 +31753,15 @@ static const struct hash_testvec michael_mic_tv_template[] = { * CRC32 test vectors */ static const struct hash_testvec crc32_tv_template[] = { + { + .psize = 0, + .digest = "\x00\x00\x00\x00", + }, + { + .plaintext = "abcdefg", + .psize = 7, + .digest = "\xd8\xb5\x46\xac", + }, { .key = "\x87\xa9\xcb\xed", .ksize = 4, @@ -36493,6 +32198,11 @@ static const struct hash_testvec crc32c_tv_template[] = { .psize = 0, .digest = "\x00\x00\x00\x00", }, + { + .plaintext = "abcdefg", + .psize = 7, + .digest = "\x41\xf4\x27\xe6", + }, { .key = "\x87\xa9\xcb\xed", .ksize = 4, @@ -36921,94 +32631,6 @@ static const struct hash_testvec crc32c_tv_template[] = { } }; -/* - * Blakcifn CRC test vectors - */ -static const struct hash_testvec bfin_crc_tv_template[] = { - { - .psize = 0, - .digest = "\x00\x00\x00\x00", - }, - { - .key = "\x87\xa9\xcb\xed", - .ksize = 4, - .psize = 0, - .digest = "\x87\xa9\xcb\xed", - }, - { - .key = "\xff\xff\xff\xff", - .ksize = 4, - .plaintext = "\x01\x02\x03\x04\x05\x06\x07\x08" - "\x09\x0a\x0b\x0c\x0d\x0e\x0f\x10" - "\x11\x12\x13\x14\x15\x16\x17\x18" - "\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20" - "\x21\x22\x23\x24\x25\x26\x27\x28", - .psize = 40, - .digest = "\x84\x0c\x8d\xa2", - }, - { - .key = "\xff\xff\xff\xff", - .ksize = 4, - .plaintext = "\x01\x02\x03\x04\x05\x06\x07\x08" - "\x09\x0a\x0b\x0c\x0d\x0e\x0f\x10" - "\x11\x12\x13\x14\x15\x16\x17\x18" - "\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20" - "\x21\x22\x23\x24\x25\x26", - .psize = 38, - .digest = "\x8c\x58\xec\xb7", - }, - { - .key = "\xff\xff\xff\xff", - .ksize = 4, - .plaintext = "\x01\x02\x03\x04\x05\x06\x07\x08" - "\x09\x0a\x0b\x0c\x0d\x0e\x0f\x10" - "\x11\x12\x13\x14\x15\x16\x17\x18" - "\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20" - "\x21\x22\x23\x24\x25\x26\x27", - .psize = 39, - .digest = "\xdc\x50\x28\x7b", - }, - { - .key = "\xff\xff\xff\xff", - .ksize = 4, - .plaintext = "\x01\x02\x03\x04\x05\x06\x07\x08" - "\x09\x0a\x0b\x0c\x0d\x0e\x0f\x10" - "\x11\x12\x13\x14\x15\x16\x17\x18" - "\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20" - "\x21\x22\x23\x24\x25\x26\x27\x28" - "\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30" - "\x31\x32\x33\x34\x35\x36\x37\x38" - "\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40" - "\x41\x42\x43\x44\x45\x46\x47\x48" - "\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50" - "\x51\x52\x53\x54\x55\x56\x57\x58" - "\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60" - "\x61\x62\x63\x64\x65\x66\x67\x68" - "\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70" - "\x71\x72\x73\x74\x75\x76\x77\x78" - "\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80" - "\x81\x82\x83\x84\x85\x86\x87\x88" - "\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90" - "\x91\x92\x93\x94\x95\x96\x97\x98" - "\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0" - "\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8" - "\xa9\xaa\xab\xac\xad\xae\xaf\xb0" - "\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8" - "\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0" - "\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8" - "\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0" - "\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8" - "\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0" - "\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8" - "\xe9\xea\xeb\xec\xed\xee\xef\xf0", - .psize = 240, - .digest = "\x10\x19\x4a\x5c", - .np = 2, - .tap = { 31, 209 } - }, - -}; - static const struct comp_testvec lz4_comp_tv_template[] = { { .inlen = 255, @@ -37131,4 +32753,75 @@ static const struct comp_testvec lz4hc_decomp_tv_template[] = { }, }; +static const struct comp_testvec zstd_comp_tv_template[] = { + { + .inlen = 68, + .outlen = 39, + .input = "The algorithm is zstd. " + "The algorithm is zstd. " + "The algorithm is zstd.", + .output = "\x28\xb5\x2f\xfd\x00\x50\xf5\x00\x00\xb8\x54\x68\x65" + "\x20\x61\x6c\x67\x6f\x72\x69\x74\x68\x6d\x20\x69\x73" + "\x20\x7a\x73\x74\x64\x2e\x20\x01\x00\x55\x73\x36\x01" + , + }, + { + .inlen = 244, + .outlen = 151, + .input = "zstd, short for Zstandard, is a fast lossless " + "compression algorithm, targeting real-time " + "compression scenarios at zlib-level and better " + "compression ratios. The zstd compression library " + "provides in-memory compression and decompression " + "functions.", + .output = "\x28\xb5\x2f\xfd\x00\x50\x75\x04\x00\x42\x4b\x1e\x17" + "\x90\x81\x31\x00\xf2\x2f\xe4\x36\xc9\xef\x92\x88\x32" + "\xc9\xf2\x24\x94\xd8\x68\x9a\x0f\x00\x0c\xc4\x31\x6f" + "\x0d\x0c\x38\xac\x5c\x48\x03\xcd\x63\x67\xc0\xf3\xad" + "\x4e\x90\xaa\x78\xa0\xa4\xc5\x99\xda\x2f\xb6\x24\x60" + "\xe2\x79\x4b\xaa\xb6\x6b\x85\x0b\xc9\xc6\x04\x66\x86" + "\xe2\xcc\xe2\x25\x3f\x4f\x09\xcd\xb8\x9d\xdb\xc1\x90" + "\xa9\x11\xbc\x35\x44\x69\x2d\x9c\x64\x4f\x13\x31\x64" + "\xcc\xfb\x4d\x95\x93\x86\x7f\x33\x7f\x1a\xef\xe9\x30" + "\xf9\x67\xa1\x94\x0a\x69\x0f\x60\xcd\xc3\xab\x99\xdc" + "\x42\xed\x97\x05\x00\x33\xc3\x15\x95\x3a\x06\xa0\x0e" + "\x20\xa9\x0e\x82\xb9\x43\x45\x01", + }, +}; + +static const struct comp_testvec zstd_decomp_tv_template[] = { + { + .inlen = 43, + .outlen = 68, + .input = "\x28\xb5\x2f\xfd\x04\x50\xf5\x00\x00\xb8\x54\x68\x65" + "\x20\x61\x6c\x67\x6f\x72\x69\x74\x68\x6d\x20\x69\x73" + "\x20\x7a\x73\x74\x64\x2e\x20\x01\x00\x55\x73\x36\x01" + "\x6b\xf4\x13\x35", + .output = "The algorithm is zstd. " + "The algorithm is zstd. " + "The algorithm is zstd.", + }, + { + .inlen = 155, + .outlen = 244, + .input = "\x28\xb5\x2f\xfd\x04\x50\x75\x04\x00\x42\x4b\x1e\x17" + "\x90\x81\x31\x00\xf2\x2f\xe4\x36\xc9\xef\x92\x88\x32" + "\xc9\xf2\x24\x94\xd8\x68\x9a\x0f\x00\x0c\xc4\x31\x6f" + "\x0d\x0c\x38\xac\x5c\x48\x03\xcd\x63\x67\xc0\xf3\xad" + "\x4e\x90\xaa\x78\xa0\xa4\xc5\x99\xda\x2f\xb6\x24\x60" + "\xe2\x79\x4b\xaa\xb6\x6b\x85\x0b\xc9\xc6\x04\x66\x86" + "\xe2\xcc\xe2\x25\x3f\x4f\x09\xcd\xb8\x9d\xdb\xc1\x90" + "\xa9\x11\xbc\x35\x44\x69\x2d\x9c\x64\x4f\x13\x31\x64" + "\xcc\xfb\x4d\x95\x93\x86\x7f\x33\x7f\x1a\xef\xe9\x30" + "\xf9\x67\xa1\x94\x0a\x69\x0f\x60\xcd\xc3\xab\x99\xdc" + "\x42\xed\x97\x05\x00\x33\xc3\x15\x95\x3a\x06\xa0\x0e" + "\x20\xa9\x0e\x82\xb9\x43\x45\x01\xaa\x6d\xda\x0d", + .output = "zstd, short for Zstandard, is a fast lossless " + "compression algorithm, targeting real-time " + "compression scenarios at zlib-level and better " + "compression ratios. The zstd compression library " + "provides in-memory compression and decompression " + "functions.", + }, +}; #endif /* _CRYPTO_TESTMGR_H */ diff --git a/crypto/zstd.c b/crypto/zstd.c new file mode 100644 index 000000000000..9a76b3ed8b8b --- /dev/null +++ b/crypto/zstd.c @@ -0,0 +1,265 @@ +/* + * Cryptographic API. + * + * Copyright (c) 2017-present, Facebook, Inc. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published by + * the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include + + +#define ZSTD_DEF_LEVEL 3 + +struct zstd_ctx { + ZSTD_CCtx *cctx; + ZSTD_DCtx *dctx; + void *cwksp; + void *dwksp; +}; + +static ZSTD_parameters zstd_params(void) +{ + return ZSTD_getParams(ZSTD_DEF_LEVEL, 0, 0); +} + +static int zstd_comp_init(struct zstd_ctx *ctx) +{ + int ret = 0; + const ZSTD_parameters params = zstd_params(); + const size_t wksp_size = ZSTD_CCtxWorkspaceBound(params.cParams); + + ctx->cwksp = vzalloc(wksp_size); + if (!ctx->cwksp) { + ret = -ENOMEM; + goto out; + } + + ctx->cctx = ZSTD_initCCtx(ctx->cwksp, wksp_size); + if (!ctx->cctx) { + ret = -EINVAL; + goto out_free; + } +out: + return ret; +out_free: + vfree(ctx->cwksp); + goto out; +} + +static int zstd_decomp_init(struct zstd_ctx *ctx) +{ + int ret = 0; + const size_t wksp_size = ZSTD_DCtxWorkspaceBound(); + + ctx->dwksp = vzalloc(wksp_size); + if (!ctx->dwksp) { + ret = -ENOMEM; + goto out; + } + + ctx->dctx = ZSTD_initDCtx(ctx->dwksp, wksp_size); + if (!ctx->dctx) { + ret = -EINVAL; + goto out_free; + } +out: + return ret; +out_free: + vfree(ctx->dwksp); + goto out; +} + +static void zstd_comp_exit(struct zstd_ctx *ctx) +{ + vfree(ctx->cwksp); + ctx->cwksp = NULL; + ctx->cctx = NULL; +} + +static void zstd_decomp_exit(struct zstd_ctx *ctx) +{ + vfree(ctx->dwksp); + ctx->dwksp = NULL; + ctx->dctx = NULL; +} + +static int __zstd_init(void *ctx) +{ + int ret; + + ret = zstd_comp_init(ctx); + if (ret) + return ret; + ret = zstd_decomp_init(ctx); + if (ret) + zstd_comp_exit(ctx); + return ret; +} + +static void *zstd_alloc_ctx(struct crypto_scomp *tfm) +{ + int ret; + struct zstd_ctx *ctx; + + ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); + if (!ctx) + return ERR_PTR(-ENOMEM); + + ret = __zstd_init(ctx); + if (ret) { + kfree(ctx); + return ERR_PTR(ret); + } + + return ctx; +} + +static int zstd_init(struct crypto_tfm *tfm) +{ + struct zstd_ctx *ctx = crypto_tfm_ctx(tfm); + + return __zstd_init(ctx); +} + +static void __zstd_exit(void *ctx) +{ + zstd_comp_exit(ctx); + zstd_decomp_exit(ctx); +} + +static void zstd_free_ctx(struct crypto_scomp *tfm, void *ctx) +{ + __zstd_exit(ctx); + kzfree(ctx); +} + +static void zstd_exit(struct crypto_tfm *tfm) +{ + struct zstd_ctx *ctx = crypto_tfm_ctx(tfm); + + __zstd_exit(ctx); +} + +static int __zstd_compress(const u8 *src, unsigned int slen, + u8 *dst, unsigned int *dlen, void *ctx) +{ + size_t out_len; + struct zstd_ctx *zctx = ctx; + const ZSTD_parameters params = zstd_params(); + + out_len = ZSTD_compressCCtx(zctx->cctx, dst, *dlen, src, slen, params); + if (ZSTD_isError(out_len)) + return -EINVAL; + *dlen = out_len; + return 0; +} + +static int zstd_compress(struct crypto_tfm *tfm, const u8 *src, + unsigned int slen, u8 *dst, unsigned int *dlen) +{ + struct zstd_ctx *ctx = crypto_tfm_ctx(tfm); + + return __zstd_compress(src, slen, dst, dlen, ctx); +} + +static int zstd_scompress(struct crypto_scomp *tfm, const u8 *src, + unsigned int slen, u8 *dst, unsigned int *dlen, + void *ctx) +{ + return __zstd_compress(src, slen, dst, dlen, ctx); +} + +static int __zstd_decompress(const u8 *src, unsigned int slen, + u8 *dst, unsigned int *dlen, void *ctx) +{ + size_t out_len; + struct zstd_ctx *zctx = ctx; + + out_len = ZSTD_decompressDCtx(zctx->dctx, dst, *dlen, src, slen); + if (ZSTD_isError(out_len)) + return -EINVAL; + *dlen = out_len; + return 0; +} + +static int zstd_decompress(struct crypto_tfm *tfm, const u8 *src, + unsigned int slen, u8 *dst, unsigned int *dlen) +{ + struct zstd_ctx *ctx = crypto_tfm_ctx(tfm); + + return __zstd_decompress(src, slen, dst, dlen, ctx); +} + +static int zstd_sdecompress(struct crypto_scomp *tfm, const u8 *src, + unsigned int slen, u8 *dst, unsigned int *dlen, + void *ctx) +{ + return __zstd_decompress(src, slen, dst, dlen, ctx); +} + +static struct crypto_alg alg = { + .cra_name = "zstd", + .cra_flags = CRYPTO_ALG_TYPE_COMPRESS, + .cra_ctxsize = sizeof(struct zstd_ctx), + .cra_module = THIS_MODULE, + .cra_init = zstd_init, + .cra_exit = zstd_exit, + .cra_u = { .compress = { + .coa_compress = zstd_compress, + .coa_decompress = zstd_decompress } } +}; + +static struct scomp_alg scomp = { + .alloc_ctx = zstd_alloc_ctx, + .free_ctx = zstd_free_ctx, + .compress = zstd_scompress, + .decompress = zstd_sdecompress, + .base = { + .cra_name = "zstd", + .cra_driver_name = "zstd-scomp", + .cra_module = THIS_MODULE, + } +}; + +static int __init zstd_mod_init(void) +{ + int ret; + + ret = crypto_register_alg(&alg); + if (ret) + return ret; + + ret = crypto_register_scomp(&scomp); + if (ret) + crypto_unregister_alg(&alg); + + return ret; +} + +static void __exit zstd_mod_fini(void) +{ + crypto_unregister_alg(&alg); + crypto_unregister_scomp(&scomp); +} + +module_init(zstd_mod_init); +module_exit(zstd_mod_fini); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Zstd Compression Algorithm"); +MODULE_ALIAS_CRYPTO("zstd"); diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig index 516d7b36d6fb..b533eeb6139d 100644 --- a/drivers/acpi/Kconfig +++ b/drivers/acpi/Kconfig @@ -547,6 +547,9 @@ config ACPI_CONFIGFS if ARM64 source "drivers/acpi/arm64/Kconfig" + +config ACPI_PPTT + bool endif config TPS68470_PMIC_OPREGION diff --git a/drivers/acpi/Makefile b/drivers/acpi/Makefile index 48e202752754..6d59aa109a91 100644 --- a/drivers/acpi/Makefile +++ b/drivers/acpi/Makefile @@ -88,6 +88,7 @@ obj-$(CONFIG_ACPI_BGRT) += bgrt.o obj-$(CONFIG_ACPI_CPPC_LIB) += cppc_acpi.o obj-$(CONFIG_ACPI_SPCR_TABLE) += spcr.o obj-$(CONFIG_ACPI_DEBUGGER_USER) += acpi_dbg.o +obj-$(CONFIG_ACPI_PPTT) += pptt.o # processor has its own "processor." module_param namespace processor-y := processor_driver.o diff --git a/drivers/acpi/ac.c b/drivers/acpi/ac.c index 2d8de2f8c1ed..cdd3136829f1 100644 --- a/drivers/acpi/ac.c +++ b/drivers/acpi/ac.c @@ -82,11 +82,11 @@ static SIMPLE_DEV_PM_OPS(acpi_ac_pm, NULL, acpi_ac_resume); #ifdef CONFIG_ACPI_PROCFS_POWER extern struct proc_dir_entry *acpi_lock_ac_dir(void); extern void *acpi_unlock_ac_dir(struct proc_dir_entry *acpi_ac_dir); -static int acpi_ac_open_fs(struct inode *inode, struct file *file); #endif static int ac_sleep_before_get_state_ms; +static int ac_check_pmic = 1; static struct acpi_driver acpi_ac_driver = { .name = "ac", @@ -111,16 +111,6 @@ struct acpi_ac { #define to_acpi_ac(x) power_supply_get_drvdata(x) -#ifdef CONFIG_ACPI_PROCFS_POWER -static const struct file_operations acpi_ac_fops = { - .owner = THIS_MODULE, - .open = acpi_ac_open_fs, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; -#endif - /* -------------------------------------------------------------------------- AC Adapter Management -------------------------------------------------------------------------- */ @@ -209,11 +199,6 @@ static int acpi_ac_seq_show(struct seq_file *seq, void *offset) return 0; } -static int acpi_ac_open_fs(struct inode *inode, struct file *file) -{ - return single_open(file, acpi_ac_seq_show, PDE_DATA(inode)); -} - static int acpi_ac_add_fs(struct acpi_ac *ac) { struct proc_dir_entry *entry = NULL; @@ -228,9 +213,8 @@ static int acpi_ac_add_fs(struct acpi_ac *ac) } /* 'state' [R] */ - entry = proc_create_data(ACPI_AC_FILE_STATE, - S_IRUGO, acpi_device_dir(ac->device), - &acpi_ac_fops, ac); + entry = proc_create_single_data(ACPI_AC_FILE_STATE, S_IRUGO, + acpi_device_dir(ac->device), acpi_ac_seq_show, ac); if (!entry) return -ENODEV; return 0; @@ -310,21 +294,43 @@ static int acpi_ac_battery_notify(struct notifier_block *nb, return NOTIFY_OK; } -static int thinkpad_e530_quirk(const struct dmi_system_id *d) +static int __init thinkpad_e530_quirk(const struct dmi_system_id *d) { ac_sleep_before_get_state_ms = 1000; return 0; } -static const struct dmi_system_id ac_dmi_table[] = { +static int __init ac_do_not_check_pmic_quirk(const struct dmi_system_id *d) +{ + ac_check_pmic = 0; + return 0; +} + +static const struct dmi_system_id ac_dmi_table[] __initconst = { { + /* Thinkpad e530 */ .callback = thinkpad_e530_quirk, - .ident = "thinkpad e530", .matches = { DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"), DMI_MATCH(DMI_PRODUCT_NAME, "32597CG"), }, }, + { + /* ECS EF20EA */ + .callback = ac_do_not_check_pmic_quirk, + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "EF20EA"), + }, + }, + { + /* Lenovo Ideapad Miix 320 */ + .callback = ac_do_not_check_pmic_quirk, + .matches = { + DMI_EXACT_MATCH(DMI_SYS_VENDOR, "LENOVO"), + DMI_EXACT_MATCH(DMI_PRODUCT_NAME, "80XF"), + DMI_EXACT_MATCH(DMI_PRODUCT_VERSION, "Lenovo MIIX 320-10ICR"), + }, + }, {}, }; @@ -384,7 +390,6 @@ end: kfree(ac); } - dmi_check_system(ac_dmi_table); return result; } @@ -442,13 +447,17 @@ static int __init acpi_ac_init(void) if (acpi_disabled) return -ENODEV; - for (i = 0; i < ARRAY_SIZE(acpi_ac_blacklist); i++) - if (acpi_dev_present(acpi_ac_blacklist[i].hid, "1", - acpi_ac_blacklist[i].hrv)) { - pr_info(PREFIX "AC: found native %s PMIC, not loading\n", - acpi_ac_blacklist[i].hid); - return -ENODEV; - } + dmi_check_system(ac_dmi_table); + + if (ac_check_pmic) { + for (i = 0; i < ARRAY_SIZE(acpi_ac_blacklist); i++) + if (acpi_dev_present(acpi_ac_blacklist[i].hid, "1", + acpi_ac_blacklist[i].hrv)) { + pr_info(PREFIX "AC: found native %s PMIC, not loading\n", + acpi_ac_blacklist[i].hid); + return -ENODEV; + } + } #ifdef CONFIG_ACPI_PROCFS_POWER acpi_ac_dir = acpi_lock_ac_dir(); diff --git a/drivers/acpi/acpi_apd.c b/drivers/acpi/acpi_apd.c index d553b0087947..2664452fa112 100644 --- a/drivers/acpi/acpi_apd.c +++ b/drivers/acpi/acpi_apd.c @@ -11,6 +11,7 @@ */ #include +#include #include #include #include @@ -72,6 +73,47 @@ static int acpi_apd_setup(struct apd_private_data *pdata) } #ifdef CONFIG_X86_AMD_PLATFORM_DEVICE + +static int misc_check_res(struct acpi_resource *ares, void *data) +{ + struct resource res; + + return !acpi_dev_resource_memory(ares, &res); +} + +static int st_misc_setup(struct apd_private_data *pdata) +{ + struct acpi_device *adev = pdata->adev; + struct platform_device *clkdev; + struct st_clk_data *clk_data; + struct resource_entry *rentry; + struct list_head resource_list; + int ret; + + clk_data = devm_kzalloc(&adev->dev, sizeof(*clk_data), GFP_KERNEL); + if (!clk_data) + return -ENOMEM; + + INIT_LIST_HEAD(&resource_list); + ret = acpi_dev_get_resources(adev, &resource_list, misc_check_res, + NULL); + if (ret < 0) + return -ENOENT; + + list_for_each_entry(rentry, &resource_list, node) { + clk_data->base = devm_ioremap(&adev->dev, rentry->res->start, + resource_size(rentry->res)); + break; + } + + acpi_dev_free_resource_list(&resource_list); + + clkdev = platform_device_register_data(&adev->dev, "clk-st", + PLATFORM_DEVID_NONE, clk_data, + sizeof(*clk_data)); + return PTR_ERR_OR_ZERO(clkdev); +} + static const struct apd_device_desc cz_i2c_desc = { .setup = acpi_apd_setup, .fixed_clk_rate = 133000000, @@ -94,6 +136,10 @@ static const struct apd_device_desc cz_uart_desc = { .fixed_clk_rate = 48000000, .properties = uart_properties, }; + +static const struct apd_device_desc st_misc_desc = { + .setup = st_misc_setup, +}; #endif #ifdef CONFIG_ARM64 @@ -179,6 +225,7 @@ static const struct acpi_device_id acpi_apd_device_ids[] = { { "AMD0020", APD_ADDR(cz_uart_desc) }, { "AMDI0020", APD_ADDR(cz_uart_desc) }, { "AMD0030", }, + { "AMD0040", APD_ADDR(st_misc_desc)}, #endif #ifdef CONFIG_ARM64 { "APMC0D0F", APD_ADDR(xgene_i2c_desc) }, diff --git a/drivers/acpi/acpi_lpss.c b/drivers/acpi/acpi_lpss.c index 2bcffec8dbf0..f8fecfec5df9 100644 --- a/drivers/acpi/acpi_lpss.c +++ b/drivers/acpi/acpi_lpss.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include "internal.h" @@ -69,6 +70,10 @@ ACPI_MODULE_NAME("acpi_lpss"); #define LPSS_SAVE_CTX BIT(4) #define LPSS_NO_D3_DELAY BIT(5) +/* Crystal Cove PMIC shares same ACPI ID between different platforms */ +#define BYT_CRC_HRV 2 +#define CHT_CRC_HRV 3 + struct lpss_private_data; struct lpss_device_desc { @@ -162,7 +167,7 @@ static void byt_pwm_setup(struct lpss_private_data *pdata) if (!adev->pnp.unique_id || strcmp(adev->pnp.unique_id, "1")) return; - if (!acpi_dev_present("INT33FD", NULL, -1)) + if (!acpi_dev_present("INT33FD", NULL, BYT_CRC_HRV)) pwm_add_table(byt_pwm_lookup, ARRAY_SIZE(byt_pwm_lookup)); } @@ -229,11 +234,13 @@ static const struct lpss_device_desc lpt_sdio_dev_desc = { static const struct lpss_device_desc byt_pwm_dev_desc = { .flags = LPSS_SAVE_CTX, + .prv_offset = 0x800, .setup = byt_pwm_setup, }; static const struct lpss_device_desc bsw_pwm_dev_desc = { .flags = LPSS_SAVE_CTX | LPSS_NO_D3_DELAY, + .prv_offset = 0x800, .setup = bsw_pwm_setup, }; @@ -940,9 +947,10 @@ static void lpss_iosf_exit_d3_state(void) mutex_unlock(&lpss_iosf_mutex); } -static int acpi_lpss_suspend(struct device *dev, bool wakeup) +static int acpi_lpss_suspend(struct device *dev, bool runtime) { struct lpss_private_data *pdata = acpi_driver_data(ACPI_COMPANION(dev)); + bool wakeup = runtime || device_may_wakeup(dev); int ret; if (pdata->dev_desc->flags & LPSS_SAVE_CTX) @@ -955,13 +963,14 @@ static int acpi_lpss_suspend(struct device *dev, bool wakeup) * wrong status for devices being about to be powered off. See * lpss_iosf_enter_d3_state() for further information. */ - if (lpss_quirks & LPSS_QUIRK_ALWAYS_POWER_ON && iosf_mbi_available()) + if ((runtime || !pm_suspend_via_firmware()) && + lpss_quirks & LPSS_QUIRK_ALWAYS_POWER_ON && iosf_mbi_available()) lpss_iosf_enter_d3_state(); return ret; } -static int acpi_lpss_resume(struct device *dev) +static int acpi_lpss_resume(struct device *dev, bool runtime) { struct lpss_private_data *pdata = acpi_driver_data(ACPI_COMPANION(dev)); int ret; @@ -970,7 +979,8 @@ static int acpi_lpss_resume(struct device *dev) * This call is kept first to be in symmetry with * acpi_lpss_runtime_suspend() one. */ - if (lpss_quirks & LPSS_QUIRK_ALWAYS_POWER_ON && iosf_mbi_available()) + if ((runtime || !pm_resume_via_firmware()) && + lpss_quirks & LPSS_QUIRK_ALWAYS_POWER_ON && iosf_mbi_available()) lpss_iosf_exit_d3_state(); ret = acpi_dev_resume(dev); @@ -994,12 +1004,12 @@ static int acpi_lpss_suspend_late(struct device *dev) return 0; ret = pm_generic_suspend_late(dev); - return ret ? ret : acpi_lpss_suspend(dev, device_may_wakeup(dev)); + return ret ? ret : acpi_lpss_suspend(dev, false); } static int acpi_lpss_resume_early(struct device *dev) { - int ret = acpi_lpss_resume(dev); + int ret = acpi_lpss_resume(dev, false); return ret ? ret : pm_generic_resume_early(dev); } @@ -1014,7 +1024,7 @@ static int acpi_lpss_runtime_suspend(struct device *dev) static int acpi_lpss_runtime_resume(struct device *dev) { - int ret = acpi_lpss_resume(dev); + int ret = acpi_lpss_resume(dev, true); return ret ? ret : pm_generic_runtime_resume(dev); } diff --git a/drivers/acpi/acpi_platform.c b/drivers/acpi/acpi_platform.c index 88cd949003f3..eaa60c94205a 100644 --- a/drivers/acpi/acpi_platform.c +++ b/drivers/acpi/acpi_platform.c @@ -82,7 +82,7 @@ struct platform_device *acpi_create_platform_device(struct acpi_device *adev, if (count < 0) { return NULL; } else if (count > 0) { - resources = kzalloc(count * sizeof(struct resource), + resources = kcalloc(count, sizeof(struct resource), GFP_KERNEL); if (!resources) { dev_err(&adev->dev, "No memory for resources\n"); diff --git a/drivers/acpi/acpi_video.c b/drivers/acpi/acpi_video.c index 2f2e737be0f8..f0b52266b3ac 100644 --- a/drivers/acpi/acpi_video.c +++ b/drivers/acpi/acpi_video.c @@ -832,8 +832,9 @@ int acpi_video_get_levels(struct acpi_device *device, * in order to account for buggy BIOS which don't export the first two * special levels (see below) */ - br->levels = kmalloc((obj->package.count + ACPI_VIDEO_FIRST_LEVEL) * - sizeof(*br->levels), GFP_KERNEL); + br->levels = kmalloc_array(obj->package.count + ACPI_VIDEO_FIRST_LEVEL, + sizeof(*br->levels), + GFP_KERNEL); if (!br->levels) { result = -ENOMEM; goto out_free; diff --git a/drivers/acpi/acpi_watchdog.c b/drivers/acpi/acpi_watchdog.c index 4bde16fb97d8..95600309ce42 100644 --- a/drivers/acpi/acpi_watchdog.c +++ b/drivers/acpi/acpi_watchdog.c @@ -12,35 +12,51 @@ #define pr_fmt(fmt) "ACPI: watchdog: " fmt #include -#include #include #include #include "internal.h" -static const struct dmi_system_id acpi_watchdog_skip[] = { - { - /* - * On Lenovo Z50-70 there are two issues with the WDAT - * table. First some of the instructions use RTC SRAM - * to store persistent information. This does not work well - * with Linux RTC driver. Second, more important thing is - * that the instructions do not actually reset the system. - * - * On this particular system iTCO_wdt seems to work just - * fine so we prefer that over WDAT for now. - * - * See also https://bugzilla.kernel.org/show_bug.cgi?id=199033. - */ - .ident = "Lenovo Z50-70", - .matches = { - DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"), - DMI_MATCH(DMI_PRODUCT_NAME, "20354"), - DMI_MATCH(DMI_PRODUCT_VERSION, "Lenovo Z50-70"), - }, - }, - {} -}; +#ifdef CONFIG_RTC_MC146818_LIB +#include + +/* + * There are several systems where the WDAT table is accessing RTC SRAM to + * store persistent information. This does not work well with the Linux RTC + * driver so on those systems we skip WDAT driver and prefer iTCO_wdt + * instead. + * + * See also https://bugzilla.kernel.org/show_bug.cgi?id=199033. + */ +static bool acpi_watchdog_uses_rtc(const struct acpi_table_wdat *wdat) +{ + const struct acpi_wdat_entry *entries; + int i; + + entries = (struct acpi_wdat_entry *)(wdat + 1); + for (i = 0; i < wdat->entries; i++) { + const struct acpi_generic_address *gas; + + gas = &entries[i].register_region; + if (gas->space_id == ACPI_ADR_SPACE_SYSTEM_IO) { + switch (gas->address) { + case RTC_PORT(0): + case RTC_PORT(1): + case RTC_PORT(2): + case RTC_PORT(3): + return true; + } + } + } + + return false; +} +#else +static bool acpi_watchdog_uses_rtc(const struct acpi_table_wdat *wdat) +{ + return false; +} +#endif static const struct acpi_table_wdat *acpi_watchdog_get_wdat(void) { @@ -50,9 +66,6 @@ static const struct acpi_table_wdat *acpi_watchdog_get_wdat(void) if (acpi_disabled) return NULL; - if (dmi_check_system(acpi_watchdog_skip)) - return NULL; - status = acpi_get_table(ACPI_SIG_WDAT, 0, (struct acpi_table_header **)&wdat); if (ACPI_FAILURE(status)) { @@ -60,6 +73,11 @@ static const struct acpi_table_wdat *acpi_watchdog_get_wdat(void) return NULL; } + if (acpi_watchdog_uses_rtc(wdat)) { + pr_info("Skipping WDAT on this system because it uses RTC SRAM\n"); + return NULL; + } + return wdat; } diff --git a/drivers/acpi/acpica/acapps.h b/drivers/acpi/acpica/acapps.h index a2a85122fafe..5a9c2febc0fb 100644 --- a/drivers/acpi/acpica/acapps.h +++ b/drivers/acpi/acpica/acapps.h @@ -143,6 +143,8 @@ acpi_status fl_split_input_pathname(char *input_path, char **out_directory_path, char **out_filename); +char *fl_get_file_basename(char *file_pathname); + char *ad_generate_filename(char *prefix, char *table_id); void diff --git a/drivers/acpi/acpica/acglobal.h b/drivers/acpi/acpica/acglobal.h index 0bc550072a21..1e6204518496 100644 --- a/drivers/acpi/acpica/acglobal.h +++ b/drivers/acpi/acpica/acglobal.h @@ -82,7 +82,7 @@ ACPI_GLOBAL(u8, acpi_gbl_global_lock_pending); * interrupt level */ ACPI_GLOBAL(acpi_spinlock, acpi_gbl_gpe_lock); /* For GPE data structs and registers */ -ACPI_GLOBAL(acpi_spinlock, acpi_gbl_hardware_lock); /* For ACPI H/W except GPE registers */ +ACPI_GLOBAL(acpi_raw_spinlock, acpi_gbl_hardware_lock); /* For ACPI H/W except GPE registers */ ACPI_GLOBAL(acpi_spinlock, acpi_gbl_reference_count_lock); /* Mutex for _OSI support */ diff --git a/drivers/acpi/acpica/dbnames.c b/drivers/acpi/acpica/dbnames.c index 170802c62179..992bd7b92540 100644 --- a/drivers/acpi/acpica/dbnames.c +++ b/drivers/acpi/acpica/dbnames.c @@ -189,9 +189,15 @@ void acpi_db_dump_namespace(char *start_arg, char *depth_arg) } acpi_db_set_output_destination(ACPI_DB_DUPLICATE_OUTPUT); - acpi_os_printf("ACPI Namespace (from %4.4s (%p) subtree):\n", - ((struct acpi_namespace_node *)subtree_entry)->name. - ascii, subtree_entry); + + if (((struct acpi_namespace_node *)subtree_entry)->parent) { + acpi_os_printf("ACPI Namespace (from %4.4s (%p) subtree):\n", + ((struct acpi_namespace_node *)subtree_entry)-> + name.ascii, subtree_entry); + } else { + acpi_os_printf("ACPI Namespace (from %s):\n", + ACPI_NAMESPACE_ROOT); + } /* Display the subtree */ @@ -316,6 +322,7 @@ acpi_db_walk_and_match_name(acpi_handle obj_handle, acpi_os_printf("Could Not get pathname for object %p\n", obj_handle); } else { + info.count = 0; info.owner_id = ACPI_OWNER_ID_MAX; info.debug_level = ACPI_UINT32_MAX; info.display_type = ACPI_DISPLAY_SUMMARY | ACPI_DISPLAY_SHORT; diff --git a/drivers/acpi/acpica/dbobject.c b/drivers/acpi/acpica/dbobject.c index 58c3253b533a..a1c76bf21122 100644 --- a/drivers/acpi/acpica/dbobject.c +++ b/drivers/acpi/acpica/dbobject.c @@ -35,6 +35,15 @@ void acpi_db_dump_method_info(acpi_status status, struct acpi_walk_state *walk_state) { struct acpi_thread_state *thread; + struct acpi_namespace_node *node; + + node = walk_state->method_node; + + /* There are no locals or arguments for the module-level code case */ + + if (node == acpi_gbl_root_node) { + return; + } /* Ignore control codes, they are not errors */ @@ -384,8 +393,14 @@ void acpi_db_decode_locals(struct acpi_walk_state *walk_state) struct acpi_namespace_node *node; u8 display_locals = FALSE; - obj_desc = walk_state->method_desc; node = walk_state->method_node; + obj_desc = walk_state->method_desc; + + /* There are no locals for the module-level code case */ + + if (node == acpi_gbl_root_node) { + return; + } if (!node) { acpi_os_printf @@ -452,6 +467,12 @@ void acpi_db_decode_arguments(struct acpi_walk_state *walk_state) node = walk_state->method_node; obj_desc = walk_state->method_desc; + /* There are no arguments for the module-level code case */ + + if (node == acpi_gbl_root_node) { + return; + } + if (!node) { acpi_os_printf ("No method node (Executing subtree for buffer or opregion)\n"); diff --git a/drivers/acpi/acpica/dbtest.c b/drivers/acpi/acpica/dbtest.c index 3892680a5258..8a5462439a97 100644 --- a/drivers/acpi/acpica/dbtest.c +++ b/drivers/acpi/acpica/dbtest.c @@ -30,6 +30,8 @@ acpi_db_test_buffer_type(struct acpi_namespace_node *node, u32 bit_length); static acpi_status acpi_db_test_string_type(struct acpi_namespace_node *node, u32 byte_length); +static acpi_status acpi_db_test_package_type(struct acpi_namespace_node *node); + static acpi_status acpi_db_read_from_object(struct acpi_namespace_node *node, acpi_object_type expected_type, @@ -273,6 +275,11 @@ acpi_db_test_one_object(acpi_handle obj_handle, bit_length = byte_length * 8; break; + case ACPI_TYPE_PACKAGE: + + local_type = ACPI_TYPE_PACKAGE; + break; + case ACPI_TYPE_FIELD_UNIT: case ACPI_TYPE_BUFFER_FIELD: case ACPI_TYPE_LOCAL_REGION_FIELD: @@ -305,6 +312,7 @@ acpi_db_test_one_object(acpi_handle obj_handle, acpi_os_printf("%14s: %4.4s", acpi_ut_get_type_name(node->type), node->name.ascii); + if (!obj_desc) { acpi_os_printf(" Ignoring, no attached object\n"); return (AE_OK); @@ -322,14 +330,13 @@ acpi_db_test_one_object(acpi_handle obj_handle, case ACPI_ADR_SPACE_SYSTEM_MEMORY: case ACPI_ADR_SPACE_SYSTEM_IO: case ACPI_ADR_SPACE_PCI_CONFIG: - case ACPI_ADR_SPACE_EC: break; default: acpi_os_printf - (" %s space is not supported [%4.4s]\n", + (" %s space is not supported in this command [%4.4s]\n", acpi_ut_get_region_name(region_obj->region. space_id), region_obj->region.node->name.ascii); @@ -359,6 +366,11 @@ acpi_db_test_one_object(acpi_handle obj_handle, status = acpi_db_test_buffer_type(node, bit_length); break; + case ACPI_TYPE_PACKAGE: + + status = acpi_db_test_package_type(node); + break; + default: acpi_os_printf(" Ignoring, type not implemented (%2.2X)", @@ -366,6 +378,13 @@ acpi_db_test_one_object(acpi_handle obj_handle, break; } + /* Exit on error, but don't abort the namespace walk */ + + if (ACPI_FAILURE(status)) { + status = AE_OK; + goto exit; + } + switch (node->type) { case ACPI_TYPE_LOCAL_REGION_FIELD: @@ -373,12 +392,14 @@ acpi_db_test_one_object(acpi_handle obj_handle, acpi_os_printf(" (%s)", acpi_ut_get_region_name(region_obj->region. space_id)); + break; default: break; } +exit: acpi_os_printf("\n"); return (status); } @@ -431,7 +452,6 @@ acpi_db_test_integer_type(struct acpi_namespace_node *node, u32 bit_length) if (temp1->integer.value == value_to_write) { value_to_write = 0; } - /* Write a new value */ write_value.type = ACPI_TYPE_INTEGER; @@ -706,6 +726,35 @@ exit: return (status); } +/******************************************************************************* + * + * FUNCTION: acpi_db_test_package_type + * + * PARAMETERS: node - Parent NS node for the object + * + * RETURN: Status + * + * DESCRIPTION: Test read for a Package object. + * + ******************************************************************************/ + +static acpi_status acpi_db_test_package_type(struct acpi_namespace_node *node) +{ + union acpi_object *temp1 = NULL; + acpi_status status; + + /* Read the original value */ + + status = acpi_db_read_from_object(node, ACPI_TYPE_PACKAGE, &temp1); + if (ACPI_FAILURE(status)) { + return (status); + } + + acpi_os_printf(" %8.8X Elements", temp1->package.count); + acpi_os_free(temp1); + return (status); +} + /******************************************************************************* * * FUNCTION: acpi_db_read_from_object @@ -746,8 +795,8 @@ acpi_db_read_from_object(struct acpi_namespace_node *node, acpi_gbl_method_executing = TRUE; status = acpi_evaluate_object(read_handle, NULL, ¶m_objects, &return_obj); - acpi_gbl_method_executing = FALSE; + acpi_gbl_method_executing = FALSE; if (ACPI_FAILURE(status)) { acpi_os_printf("Could not read from object, %s", acpi_format_exception(status)); @@ -760,6 +809,7 @@ acpi_db_read_from_object(struct acpi_namespace_node *node, case ACPI_TYPE_INTEGER: case ACPI_TYPE_BUFFER: case ACPI_TYPE_STRING: + case ACPI_TYPE_PACKAGE: /* * Did we receive the type we wanted? Most important for the * Integer/Buffer case (when a field is larger than an Integer, @@ -771,6 +821,7 @@ acpi_db_read_from_object(struct acpi_namespace_node *node, acpi_ut_get_type_name(expected_type), acpi_ut_get_type_name(ret_value->type)); + acpi_os_free(return_obj.pointer); return (AE_TYPE); } diff --git a/drivers/acpi/acpica/dsdebug.c b/drivers/acpi/acpica/dsdebug.c index 70a2fca60306..9d33f0bb2885 100644 --- a/drivers/acpi/acpica/dsdebug.c +++ b/drivers/acpi/acpica/dsdebug.c @@ -162,9 +162,15 @@ acpi_ds_dump_method_stack(acpi_status status, op->common.next = NULL; #ifdef ACPI_DISASSEMBLER - acpi_os_printf("Failed at "); - acpi_dm_disassemble(next_walk_state, op, - ACPI_UINT32_MAX); + if (walk_state->method_node != + acpi_gbl_root_node) { + + /* More verbose if not module-level code */ + + acpi_os_printf("Failed at "); + acpi_dm_disassemble(next_walk_state, op, + ACPI_UINT32_MAX); + } #endif op->common.next = next; } diff --git a/drivers/acpi/acpica/dswscope.c b/drivers/acpi/acpica/dswscope.c index d1422f984f6e..7592176a8fa2 100644 --- a/drivers/acpi/acpica/dswscope.c +++ b/drivers/acpi/acpica/dswscope.c @@ -115,7 +115,7 @@ acpi_ds_scope_stack_push(struct acpi_namespace_node *node, acpi_ut_get_type_name(old_scope_info-> common.value))); } else { - ACPI_DEBUG_PRINT_RAW((ACPI_DB_EXEC, "[\\___] (%s)", "ROOT")); + ACPI_DEBUG_PRINT_RAW((ACPI_DB_EXEC, ACPI_NAMESPACE_ROOT)); } ACPI_DEBUG_PRINT_RAW((ACPI_DB_EXEC, @@ -166,14 +166,14 @@ acpi_status acpi_ds_scope_stack_pop(struct acpi_walk_state *walk_state) new_scope_info = walk_state->scope_info; if (new_scope_info) { - ACPI_DEBUG_PRINT_RAW((ACPI_DB_EXEC, - "[%4.4s] (%s)\n", + ACPI_DEBUG_PRINT_RAW((ACPI_DB_EXEC, "[%4.4s] (%s)\n", acpi_ut_get_node_name(new_scope_info-> scope.node), acpi_ut_get_type_name(new_scope_info-> common.value))); } else { - ACPI_DEBUG_PRINT_RAW((ACPI_DB_EXEC, "[\\___] (ROOT)\n")); + ACPI_DEBUG_PRINT_RAW((ACPI_DB_EXEC, "%s\n", + ACPI_NAMESPACE_ROOT)); } acpi_ut_delete_generic_state(scope_info); diff --git a/drivers/acpi/acpica/evgpe.c b/drivers/acpi/acpica/evgpe.c index abbd59063906..e10fec99a182 100644 --- a/drivers/acpi/acpica/evgpe.c +++ b/drivers/acpi/acpica/evgpe.c @@ -634,6 +634,12 @@ acpi_ev_detect_gpe(struct acpi_namespace_node *gpe_device, flags = acpi_os_acquire_lock(acpi_gbl_gpe_lock); + if (!gpe_event_info) { + gpe_event_info = acpi_ev_get_gpe_event_info(gpe_device, gpe_number); + if (!gpe_event_info) + goto error_exit; + } + /* Get the info block for the entire GPE register */ gpe_register_info = gpe_event_info->register_info; diff --git a/drivers/acpi/acpica/evxfgpe.c b/drivers/acpi/acpica/evxfgpe.c index c80e3bdf4805..b2d5f66cc1b0 100644 --- a/drivers/acpi/acpica/evxfgpe.c +++ b/drivers/acpi/acpica/evxfgpe.c @@ -637,6 +637,28 @@ unlock_and_exit: ACPI_EXPORT_SYMBOL(acpi_get_gpe_status) +/******************************************************************************* + * + * FUNCTION: acpi_gispatch_gpe + * + * PARAMETERS: gpe_device - Parent GPE Device. NULL for GPE0/GPE1 + * gpe_number - GPE level within the GPE block + * + * RETURN: None + * + * DESCRIPTION: Detect and dispatch a General Purpose Event to either a function + * (e.g. EC) or method (e.g. _Lxx/_Exx) handler. + * + ******************************************************************************/ +void acpi_dispatch_gpe(acpi_handle gpe_device, u32 gpe_number) +{ + ACPI_FUNCTION_TRACE(acpi_dispatch_gpe); + + acpi_ev_detect_gpe(gpe_device, NULL, gpe_number); +} + +ACPI_EXPORT_SYMBOL(acpi_dispatch_gpe) + /******************************************************************************* * * FUNCTION: acpi_finish_gpe diff --git a/drivers/acpi/acpica/exconfig.c b/drivers/acpi/acpica/exconfig.c index f85c6f3271f6..2373a7492151 100644 --- a/drivers/acpi/acpica/exconfig.c +++ b/drivers/acpi/acpica/exconfig.c @@ -489,6 +489,17 @@ acpi_status acpi_ex_unload_table(union acpi_operand_object *ddb_handle) */ ACPI_WARNING((AE_INFO, "Received request to unload an ACPI table")); + /* + * May 2018: Unload is no longer supported for the following reasons: + * 1) A correct implementation on some hosts may not be possible. + * 2) Other ACPI implementations do not correctly/fully support it. + * 3) It requires host device driver support which does not exist. + * (To properly support namespace unload out from underneath.) + * 4) This AML operator has never been seen in the field. + */ + ACPI_EXCEPTION((AE_INFO, AE_NOT_IMPLEMENTED, + "AML Unload operator is not supported")); + /* * Validate the handle * Although the handle is partially validated in acpi_ex_reconfiguration() diff --git a/drivers/acpi/acpica/hwregs.c b/drivers/acpi/acpica/hwregs.c index 27a86ad55b58..3de794bcf8fa 100644 --- a/drivers/acpi/acpica/hwregs.c +++ b/drivers/acpi/acpica/hwregs.c @@ -390,14 +390,14 @@ acpi_status acpi_hw_clear_acpi_status(void) ACPI_BITMASK_ALL_FIXED_STATUS, ACPI_FORMAT_UINT64(acpi_gbl_xpm1a_status.address))); - lock_flags = acpi_os_acquire_lock(acpi_gbl_hardware_lock); + lock_flags = acpi_os_acquire_raw_lock(acpi_gbl_hardware_lock); /* Clear the fixed events in PM1 A/B */ status = acpi_hw_register_write(ACPI_REGISTER_PM1_STATUS, ACPI_BITMASK_ALL_FIXED_STATUS); - acpi_os_release_lock(acpi_gbl_hardware_lock, lock_flags); + acpi_os_release_raw_lock(acpi_gbl_hardware_lock, lock_flags); if (ACPI_FAILURE(status)) { goto exit; diff --git a/drivers/acpi/acpica/hwxface.c b/drivers/acpi/acpica/hwxface.c index 5d1396870bd0..6e39a771a56e 100644 --- a/drivers/acpi/acpica/hwxface.c +++ b/drivers/acpi/acpica/hwxface.c @@ -227,7 +227,7 @@ acpi_status acpi_write_bit_register(u32 register_id, u32 value) return_ACPI_STATUS(AE_BAD_PARAMETER); } - lock_flags = acpi_os_acquire_lock(acpi_gbl_hardware_lock); + lock_flags = acpi_os_acquire_raw_lock(acpi_gbl_hardware_lock); /* * At this point, we know that the parent register is one of the @@ -288,7 +288,7 @@ acpi_status acpi_write_bit_register(u32 register_id, u32 value) unlock_and_exit: - acpi_os_release_lock(acpi_gbl_hardware_lock, lock_flags); + acpi_os_release_raw_lock(acpi_gbl_hardware_lock, lock_flags); return_ACPI_STATUS(status); } diff --git a/drivers/acpi/acpica/nsdump.c b/drivers/acpi/acpica/nsdump.c index 4bdbd1d8431b..90ccffcd770b 100644 --- a/drivers/acpi/acpica/nsdump.c +++ b/drivers/acpi/acpica/nsdump.c @@ -170,6 +170,7 @@ acpi_ns_dump_one_object(acpi_handle obj_handle, } type = this_node->type; + info->count++; /* Check if the owner matches */ @@ -639,6 +640,7 @@ acpi_ns_dump_objects(acpi_object_type type, return; } + info.count = 0; info.debug_level = ACPI_LV_TABLES; info.owner_id = owner_id; info.display_type = display_type; @@ -649,6 +651,7 @@ acpi_ns_dump_objects(acpi_object_type type, acpi_ns_dump_one_object, NULL, (void *)&info, NULL); + acpi_os_printf("\nNamespace node count: %u\n\n", info.count); (void)acpi_ut_release_mutex(ACPI_MTX_NAMESPACE); } diff --git a/drivers/acpi/acpica/psloop.c b/drivers/acpi/acpica/psloop.c index 68422afc365f..bc5f05906bd1 100644 --- a/drivers/acpi/acpica/psloop.c +++ b/drivers/acpi/acpica/psloop.c @@ -515,6 +515,22 @@ acpi_status acpi_ps_parse_loop(struct acpi_walk_state *walk_state) if (ACPI_FAILURE(status)) { return_ACPI_STATUS(status); } + if (walk_state->opcode == AML_SCOPE_OP) { + /* + * If the scope op fails to parse, skip the body of the + * scope op because the parse failure indicates that the + * device may not exist. + */ + walk_state->parser_state.aml = + walk_state->aml + 1; + walk_state->parser_state.aml = + acpi_ps_get_next_package_end + (&walk_state->parser_state); + walk_state->aml = + walk_state->parser_state.aml; + ACPI_ERROR((AE_INFO, + "Skipping Scope block")); + } continue; } @@ -557,7 +573,40 @@ acpi_status acpi_ps_parse_loop(struct acpi_walk_state *walk_state) if (ACPI_FAILURE(status)) { return_ACPI_STATUS(status); } - + if ((walk_state->control_state) && + ((walk_state->control_state->control. + opcode == AML_IF_OP) + || (walk_state->control_state->control. + opcode == AML_WHILE_OP))) { + /* + * If the if/while op fails to parse, we will skip parsing + * the body of the op. + */ + parser_state->aml = + walk_state->control_state->control. + aml_predicate_start + 1; + parser_state->aml = + acpi_ps_get_next_package_end + (parser_state); + walk_state->aml = parser_state->aml; + + ACPI_ERROR((AE_INFO, + "Skipping While/If block")); + if (*walk_state->aml == AML_ELSE_OP) { + ACPI_ERROR((AE_INFO, + "Skipping Else block")); + walk_state->parser_state.aml = + walk_state->aml + 1; + walk_state->parser_state.aml = + acpi_ps_get_next_package_end + (parser_state); + walk_state->aml = + parser_state->aml; + } + ACPI_FREE(acpi_ut_pop_generic_state + (&walk_state->control_state)); + } + op = NULL; continue; } } diff --git a/drivers/acpi/acpica/psobject.c b/drivers/acpi/acpica/psobject.c index 7d9d0151ee54..3138e7a00da8 100644 --- a/drivers/acpi/acpica/psobject.c +++ b/drivers/acpi/acpica/psobject.c @@ -12,6 +12,7 @@ #include "acparser.h" #include "amlcode.h" #include "acconvert.h" +#include "acnamesp.h" #define _COMPONENT ACPI_PARSER ACPI_MODULE_NAME("psobject") @@ -549,6 +550,21 @@ acpi_ps_complete_op(struct acpi_walk_state *walk_state, do { if (*op) { + /* + * These Opcodes need to be removed from the namespace because they + * get created even if these opcodes cannot be created due to + * errors. + */ + if (((*op)->common.aml_opcode == AML_REGION_OP) + || ((*op)->common.aml_opcode == + AML_DATA_REGION_OP)) { + acpi_ns_delete_children((*op)->common. + node); + acpi_ns_remove_node((*op)->common.node); + (*op)->common.node = NULL; + acpi_ps_delete_parse_tree(*op); + } + status2 = acpi_ps_complete_this_op(walk_state, *op); if (ACPI_FAILURE(status2)) { @@ -574,6 +590,20 @@ acpi_ps_complete_op(struct acpi_walk_state *walk_state, #endif walk_state->prev_op = NULL; walk_state->prev_arg_types = walk_state->arg_types; + + if (walk_state->parse_flags & ACPI_PARSE_MODULE_LEVEL) { + /* + * There was something that went wrong while executing code at the + * module-level. We need to skip parsing whatever caused the + * error and keep going. One runtime error during the table load + * should not cause the entire table to not be loaded. This is + * because there could be correct AML beyond the parts that caused + * the runtime error. + */ + ACPI_ERROR((AE_INFO, + "Ignore error and continue table load")); + return_ACPI_STATUS(AE_OK); + } return_ACPI_STATUS(status); } diff --git a/drivers/acpi/acpica/pswalk.c b/drivers/acpi/acpica/pswalk.c index e0a442b8648b..bd6af8c87d48 100644 --- a/drivers/acpi/acpica/pswalk.c +++ b/drivers/acpi/acpica/pswalk.c @@ -25,22 +25,48 @@ ACPI_MODULE_NAME("pswalk") * DESCRIPTION: Delete a portion of or an entire parse tree. * ******************************************************************************/ +#include "amlcode.h" void acpi_ps_delete_parse_tree(union acpi_parse_object *subtree_root) { union acpi_parse_object *op = subtree_root; union acpi_parse_object *next = NULL; union acpi_parse_object *parent = NULL; + u32 level = 0; ACPI_FUNCTION_TRACE_PTR(ps_delete_parse_tree, subtree_root); + ACPI_DEBUG_PRINT((ACPI_DB_PARSE_TREES, " root %p\n", subtree_root)); + /* Visit all nodes in the subtree */ while (op) { - - /* Check if we are not ascending */ - if (op != parent) { + /* This is the descending case */ + + if (ACPI_IS_DEBUG_ENABLED + (ACPI_LV_PARSE_TREES, _COMPONENT)) { + + /* This debug option will print the entire parse tree */ + + acpi_os_printf(" %*.s%s %p", (level * 4), + " ", + acpi_ps_get_opcode_name(op-> + common. + aml_opcode), + op); + + if (op->named.aml_opcode == AML_INT_NAMEPATH_OP) { + acpi_os_printf(" %4.4s", + op->common.value.string); + } + if (op->named.aml_opcode == AML_STRING_OP) { + acpi_os_printf(" %s", + op->common.value.string); + } + acpi_os_printf("\n"); + } + /* Look for an argument or child of the current op */ next = acpi_ps_get_arg(op, 0); @@ -49,6 +75,7 @@ void acpi_ps_delete_parse_tree(union acpi_parse_object *subtree_root) /* Still going downward in tree (Op is not completed yet) */ op = next; + level++; continue; } } @@ -69,6 +96,7 @@ void acpi_ps_delete_parse_tree(union acpi_parse_object *subtree_root) if (next) { op = next; } else { + level--; op = parent; } } diff --git a/drivers/acpi/acpica/rsdump.c b/drivers/acpi/acpica/rsdump.c index b12a0b1cd9ce..6601e71b45e3 100644 --- a/drivers/acpi/acpica/rsdump.c +++ b/drivers/acpi/acpica/rsdump.c @@ -539,7 +539,7 @@ static void acpi_rs_out_title(const char *title) static void acpi_rs_dump_byte_list(u16 length, u8 * data) { - u8 i; + u16 i; for (i = 0; i < length; i++) { acpi_os_printf("%25s%2.2X : %2.2X\n", "Byte", i, data[i]); diff --git a/drivers/acpi/acpica/tbinstal.c b/drivers/acpi/acpica/tbinstal.c index c5085b7ae8c9..5f8e7b561c90 100644 --- a/drivers/acpi/acpica/tbinstal.c +++ b/drivers/acpi/acpica/tbinstal.c @@ -88,7 +88,7 @@ acpi_tb_install_table_with_override(struct acpi_table_desc *new_table_desc, * DESCRIPTION: This function is called to verify and install an ACPI table. * When this function is called by "Load" or "LoadTable" opcodes, * or by acpi_load_table() API, the "Reload" parameter is set. - * After sucessfully returning from this function, table is + * After successfully returning from this function, table is * "INSTALLED" but not "VALIDATED". * ******************************************************************************/ diff --git a/drivers/acpi/acpica/utbuffer.c b/drivers/acpi/acpica/utbuffer.c index 148aeb84e561..fffa6f5ae59e 100644 --- a/drivers/acpi/acpica/utbuffer.c +++ b/drivers/acpi/acpica/utbuffer.c @@ -53,7 +53,7 @@ void acpi_ut_dump_buffer(u8 *buffer, u32 count, u32 display, u32 base_offset) /* Print current offset */ - acpi_os_printf("%6.4X: ", (base_offset + i)); + acpi_os_printf("%8.4X: ", (base_offset + i)); /* Print 16 hex chars */ @@ -219,7 +219,7 @@ acpi_ut_dump_buffer_to_file(ACPI_FILE file, /* Print current offset */ - fprintf(file, "%6.4X: ", (base_offset + i)); + fprintf(file, "%8.4X: ", (base_offset + i)); /* Print 16 hex chars */ diff --git a/drivers/acpi/acpica/uterror.c b/drivers/acpi/acpica/uterror.c index 12d4a0f6b8d2..5a64ddaed8a3 100644 --- a/drivers/acpi/acpica/uterror.c +++ b/drivers/acpi/acpica/uterror.c @@ -182,20 +182,20 @@ acpi_ut_prefixed_namespace_error(const char *module_name, switch (lookup_status) { case AE_ALREADY_EXISTS: - acpi_os_printf(ACPI_MSG_BIOS_ERROR); + acpi_os_printf("\n" ACPI_MSG_BIOS_ERROR); message = "Failure creating"; break; case AE_NOT_FOUND: - acpi_os_printf(ACPI_MSG_BIOS_ERROR); - message = "Failure looking up"; + acpi_os_printf("\n" ACPI_MSG_BIOS_ERROR); + message = "Could not resolve"; break; default: - acpi_os_printf(ACPI_MSG_ERROR); - message = "Failure looking up"; + acpi_os_printf("\n" ACPI_MSG_ERROR); + message = "Failure resolving"; break; } diff --git a/drivers/acpi/acpica/utmutex.c b/drivers/acpi/acpica/utmutex.c index d2d93e388f40..2e465e6a0ab6 100644 --- a/drivers/acpi/acpica/utmutex.c +++ b/drivers/acpi/acpica/utmutex.c @@ -52,7 +52,7 @@ acpi_status acpi_ut_mutex_initialize(void) return_ACPI_STATUS (status); } - status = acpi_os_create_lock (&acpi_gbl_hardware_lock); + status = acpi_os_create_raw_lock(&acpi_gbl_hardware_lock); if (ACPI_FAILURE (status)) { return_ACPI_STATUS (status); } @@ -109,7 +109,7 @@ void acpi_ut_mutex_terminate(void) /* Delete the spinlocks */ acpi_os_delete_lock(acpi_gbl_gpe_lock); - acpi_os_delete_lock(acpi_gbl_hardware_lock); + acpi_os_delete_raw_lock(acpi_gbl_hardware_lock); acpi_os_delete_lock(acpi_gbl_reference_count_lock); /* Delete the reader/writer lock */ diff --git a/drivers/acpi/acpica/utobject.c b/drivers/acpi/acpica/utobject.c index 5b78fe08d7d7..ae6d8cc18cec 100644 --- a/drivers/acpi/acpica/utobject.c +++ b/drivers/acpi/acpica/utobject.c @@ -8,6 +8,7 @@ *****************************************************************************/ #include +#include #include "accommon.h" #include "acnamesp.h" @@ -70,6 +71,7 @@ union acpi_operand_object *acpi_ut_create_internal_object_dbg(const char if (!object) { return_PTR(NULL); } + kmemleak_not_leak(object); switch (type) { case ACPI_TYPE_REGION: diff --git a/drivers/acpi/acpica/utosi.c b/drivers/acpi/acpica/utosi.c index 1b415fa90cf8..64b63c81994b 100644 --- a/drivers/acpi/acpica/utosi.c +++ b/drivers/acpi/acpica/utosi.c @@ -69,6 +69,7 @@ static struct acpi_interface_info acpi_default_supported_interfaces[] = { {"Windows 2015", NULL, 0, ACPI_OSI_WIN_10}, /* Windows 10 - Added 03/2015 */ {"Windows 2016", NULL, 0, ACPI_OSI_WIN_10_RS1}, /* Windows 10 version 1607 - Added 12/2017 */ {"Windows 2017", NULL, 0, ACPI_OSI_WIN_10_RS2}, /* Windows 10 version 1703 - Added 12/2017 */ + {"Windows 2017.2", NULL, 0, ACPI_OSI_WIN_10_RS3}, /* Windows 10 version 1709 - Added 02/2018 */ /* Feature Group Strings */ diff --git a/drivers/acpi/acpica/utprint.c b/drivers/acpi/acpica/utprint.c index 35ffd8d51c65..a98c334c3bb7 100644 --- a/drivers/acpi/acpica/utprint.c +++ b/drivers/acpi/acpica/utprint.c @@ -470,6 +470,7 @@ int vsnprintf(char *string, acpi_size size, const char *format, va_list args) case 'X': type |= ACPI_FORMAT_UPPER; + /* FALLTHROUGH */ case 'x': diff --git a/drivers/acpi/acpica/utstring.c b/drivers/acpi/acpica/utstring.c index bd57a77bbcb2..5bef0b059406 100644 --- a/drivers/acpi/acpica/utstring.c +++ b/drivers/acpi/acpica/utstring.c @@ -141,7 +141,7 @@ void acpi_ut_repair_name(char *name) * Special case for the root node. This can happen if we get an * error during the execution of module-level code. */ - if (ACPI_COMPARE_NAME(name, "\\___")) { + if (ACPI_COMPARE_NAME(name, ACPI_ROOT_PATHNAME)) { return; } diff --git a/drivers/acpi/apei/erst.c b/drivers/acpi/apei/erst.c index 9bff853e85f3..3c5ea7cb693e 100644 --- a/drivers/acpi/apei/erst.c +++ b/drivers/acpi/apei/erst.c @@ -524,7 +524,8 @@ retry: pr_warn(FW_WARN "too many record IDs!\n"); return 0; } - new_entries = kvmalloc(new_size * sizeof(entries[0]), GFP_KERNEL); + new_entries = kvmalloc_array(new_size, sizeof(entries[0]), + GFP_KERNEL); if (!new_entries) return -ENOMEM; memcpy(new_entries, entries, diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c index 1efefe919555..02c6fd9caff7 100644 --- a/drivers/acpi/apei/ghes.c +++ b/drivers/acpi/apei/ghes.c @@ -481,7 +481,7 @@ static void ghes_do_proc(struct ghes *ghes, if (guid_equal(sec_type, &CPER_SEC_PLATFORM_MEM)) { struct cper_sec_mem_err *mem_err = acpi_hest_get_payload(gdata); - ghes_edac_report_mem_error(ghes, sev, mem_err); + ghes_edac_report_mem_error(sev, mem_err); arch_apei_report_mem_error(sev, mem_err); ghes_handle_memory_failure(gdata, sev); @@ -1087,10 +1087,6 @@ static int ghes_probe(struct platform_device *ghes_dev) goto err; } - rc = ghes_edac_register(ghes, &ghes_dev->dev); - if (rc < 0) - goto err; - switch (generic->notify.type) { case ACPI_HEST_NOTIFY_POLLED: timer_setup(&ghes->timer, ghes_poll_func, TIMER_DEFERRABLE); @@ -1102,14 +1098,14 @@ static int ghes_probe(struct platform_device *ghes_dev) if (rc) { pr_err(GHES_PFX "Failed to map GSI to IRQ for generic hardware error source: %d\n", generic->header.source_id); - goto err_edac_unreg; + goto err; } rc = request_irq(ghes->irq, ghes_irq_func, IRQF_SHARED, "GHES IRQ", ghes); if (rc) { pr_err(GHES_PFX "Failed to register IRQ for generic hardware error source: %d\n", generic->header.source_id); - goto err_edac_unreg; + goto err; } break; @@ -1132,14 +1128,16 @@ static int ghes_probe(struct platform_device *ghes_dev) default: BUG(); } + platform_set_drvdata(ghes_dev, ghes); + ghes_edac_register(ghes, &ghes_dev->dev); + /* Handle any pending errors right away */ ghes_proc(ghes); return 0; -err_edac_unreg: - ghes_edac_unregister(ghes); + err: if (ghes) { ghes_fini(ghes); diff --git a/drivers/acpi/apei/hest.c b/drivers/acpi/apei/hest.c index 9cb74115a43d..b1e9f81ebeea 100644 --- a/drivers/acpi/apei/hest.c +++ b/drivers/acpi/apei/hest.c @@ -195,7 +195,8 @@ static int __init hest_ghes_dev_register(unsigned int ghes_count) struct ghes_arr ghes_arr; ghes_arr.count = 0; - ghes_arr.ghes_devs = kmalloc(sizeof(void *) * ghes_count, GFP_KERNEL); + ghes_arr.ghes_devs = kmalloc_array(ghes_count, sizeof(void *), + GFP_KERNEL); if (!ghes_arr.ghes_devs) return -ENOMEM; diff --git a/drivers/acpi/battery.c b/drivers/acpi/battery.c index bdb24d636d9a..b0113a5802a3 100644 --- a/drivers/acpi/battery.c +++ b/drivers/acpi/battery.c @@ -74,6 +74,8 @@ static async_cookie_t async_cookie; static bool battery_driver_registered; static int battery_bix_broken_package; static int battery_notification_delay_ms; +static int battery_ac_is_broken; +static int battery_check_pmic = 1; static unsigned int cache_time = 1000; module_param(cache_time, uint, 0644); MODULE_PARM_DESC(cache_time, "cache time in milliseconds"); @@ -81,14 +83,6 @@ MODULE_PARM_DESC(cache_time, "cache time in milliseconds"); #ifdef CONFIG_ACPI_PROCFS_POWER extern struct proc_dir_entry *acpi_lock_battery_dir(void); extern void *acpi_unlock_battery_dir(struct proc_dir_entry *acpi_battery_dir); - -enum acpi_battery_files { - info_tag = 0, - state_tag, - alarm_tag, - ACPI_BATTERY_NUMFILES, -}; - #endif static const struct acpi_device_id battery_device_ids[] = { @@ -215,6 +209,20 @@ static bool acpi_battery_is_degraded(struct acpi_battery *battery) battery->full_charge_capacity < battery->design_capacity; } +static int acpi_battery_handle_discharging(struct acpi_battery *battery) +{ + /* + * Some devices wrongly report discharging if the battery's charge level + * was above the device's start charging threshold atm the AC adapter + * was plugged in and the device thus did not start a new charge cycle. + */ + if ((battery_ac_is_broken || power_supply_is_system_supplied()) && + battery->rate_now == 0) + return POWER_SUPPLY_STATUS_NOT_CHARGING; + + return POWER_SUPPLY_STATUS_DISCHARGING; +} + static int acpi_battery_get_property(struct power_supply *psy, enum power_supply_property psp, union power_supply_propval *val) @@ -230,7 +238,7 @@ static int acpi_battery_get_property(struct power_supply *psy, switch (psp) { case POWER_SUPPLY_PROP_STATUS: if (battery->state & ACPI_BATTERY_STATE_DISCHARGING) - val->intval = POWER_SUPPLY_STATUS_DISCHARGING; + val->intval = acpi_battery_handle_discharging(battery); else if (battery->state & ACPI_BATTERY_STATE_CHARGING) val->intval = POWER_SUPPLY_STATUS_CHARGING; else if (acpi_battery_is_charged(battery)) @@ -985,9 +993,10 @@ static const char *acpi_battery_units(const struct acpi_battery *battery) "mA" : "mW"; } -static int acpi_battery_print_info(struct seq_file *seq, int result) +static int acpi_battery_info_proc_show(struct seq_file *seq, void *offset) { struct acpi_battery *battery = seq->private; + int result = acpi_battery_update(battery, false); if (result) goto end; @@ -1041,9 +1050,10 @@ static int acpi_battery_print_info(struct seq_file *seq, int result) return result; } -static int acpi_battery_print_state(struct seq_file *seq, int result) +static int acpi_battery_state_proc_show(struct seq_file *seq, void *offset) { struct acpi_battery *battery = seq->private; + int result = acpi_battery_update(battery, false); if (result) goto end; @@ -1088,9 +1098,10 @@ static int acpi_battery_print_state(struct seq_file *seq, int result) return result; } -static int acpi_battery_print_alarm(struct seq_file *seq, int result) +static int acpi_battery_alarm_proc_show(struct seq_file *seq, void *offset) { struct acpi_battery *battery = seq->private; + int result = acpi_battery_update(battery, false); if (result) goto end; @@ -1142,82 +1153,22 @@ static ssize_t acpi_battery_write_alarm(struct file *file, return result; } -typedef int(*print_func)(struct seq_file *seq, int result); - -static print_func acpi_print_funcs[ACPI_BATTERY_NUMFILES] = { - acpi_battery_print_info, - acpi_battery_print_state, - acpi_battery_print_alarm, -}; - -static int acpi_battery_read(int fid, struct seq_file *seq) +static int acpi_battery_alarm_proc_open(struct inode *inode, struct file *file) { - struct acpi_battery *battery = seq->private; - int result = acpi_battery_update(battery, false); - return acpi_print_funcs[fid](seq, result); -} - -#define DECLARE_FILE_FUNCTIONS(_name) \ -static int acpi_battery_read_##_name(struct seq_file *seq, void *offset) \ -{ \ - return acpi_battery_read(_name##_tag, seq); \ -} \ -static int acpi_battery_##_name##_open_fs(struct inode *inode, struct file *file) \ -{ \ - return single_open(file, acpi_battery_read_##_name, PDE_DATA(inode)); \ + return single_open(file, acpi_battery_alarm_proc_show, PDE_DATA(inode)); } -DECLARE_FILE_FUNCTIONS(info); -DECLARE_FILE_FUNCTIONS(state); -DECLARE_FILE_FUNCTIONS(alarm); - -#undef DECLARE_FILE_FUNCTIONS - -#define FILE_DESCRIPTION_RO(_name) \ - { \ - .name = __stringify(_name), \ - .mode = S_IRUGO, \ - .ops = { \ - .open = acpi_battery_##_name##_open_fs, \ - .read = seq_read, \ - .llseek = seq_lseek, \ - .release = single_release, \ - .owner = THIS_MODULE, \ - }, \ - } - -#define FILE_DESCRIPTION_RW(_name) \ - { \ - .name = __stringify(_name), \ - .mode = S_IFREG | S_IRUGO | S_IWUSR, \ - .ops = { \ - .open = acpi_battery_##_name##_open_fs, \ - .read = seq_read, \ - .llseek = seq_lseek, \ - .write = acpi_battery_write_##_name, \ - .release = single_release, \ - .owner = THIS_MODULE, \ - }, \ - } - -static const struct battery_file { - struct file_operations ops; - umode_t mode; - const char *name; -} acpi_battery_file[] = { - FILE_DESCRIPTION_RO(info), - FILE_DESCRIPTION_RO(state), - FILE_DESCRIPTION_RW(alarm), +static const struct file_operations acpi_battery_alarm_fops = { + .owner = THIS_MODULE, + .open = acpi_battery_alarm_proc_open, + .read = seq_read, + .write = acpi_battery_write_alarm, + .llseek = seq_lseek, + .release = single_release, }; -#undef FILE_DESCRIPTION_RO -#undef FILE_DESCRIPTION_RW - static int acpi_battery_add_fs(struct acpi_device *device) { - struct proc_dir_entry *entry = NULL; - int i; - printk(KERN_WARNING PREFIX "Deprecated procfs I/F for battery is loaded," " please retry with CONFIG_ACPI_PROCFS_POWER cleared\n"); if (!acpi_device_dir(device)) { @@ -1227,28 +1178,24 @@ static int acpi_battery_add_fs(struct acpi_device *device) return -ENODEV; } - for (i = 0; i < ACPI_BATTERY_NUMFILES; ++i) { - entry = proc_create_data(acpi_battery_file[i].name, - acpi_battery_file[i].mode, - acpi_device_dir(device), - &acpi_battery_file[i].ops, - acpi_driver_data(device)); - if (!entry) - return -ENODEV; - } + if (!proc_create_single_data("info", S_IRUGO, acpi_device_dir(device), + acpi_battery_info_proc_show, acpi_driver_data(device))) + return -ENODEV; + if (!proc_create_single_data("state", S_IRUGO, acpi_device_dir(device), + acpi_battery_state_proc_show, acpi_driver_data(device))) + return -ENODEV; + if (!proc_create_data("alarm", S_IFREG | S_IRUGO | S_IWUSR, + acpi_device_dir(device), &acpi_battery_alarm_fops, + acpi_driver_data(device))) + return -ENODEV; return 0; } static void acpi_battery_remove_fs(struct acpi_device *device) { - int i; if (!acpi_device_dir(device)) return; - for (i = 0; i < ACPI_BATTERY_NUMFILES; ++i) - remove_proc_entry(acpi_battery_file[i].name, - acpi_device_dir(device)); - - remove_proc_entry(acpi_device_bid(device), acpi_battery_dir); + remove_proc_subtree(acpi_device_bid(device), acpi_battery_dir); acpi_device_dir(device) = NULL; } @@ -1332,23 +1279,64 @@ battery_notification_delay_quirk(const struct dmi_system_id *d) return 0; } +static int __init +battery_ac_is_broken_quirk(const struct dmi_system_id *d) +{ + battery_ac_is_broken = 1; + return 0; +} + +static int __init +battery_do_not_check_pmic_quirk(const struct dmi_system_id *d) +{ + battery_check_pmic = 0; + return 0; +} + static const struct dmi_system_id bat_dmi_table[] __initconst = { { + /* NEC LZ750/LS */ .callback = battery_bix_broken_package_quirk, - .ident = "NEC LZ750/LS", .matches = { DMI_MATCH(DMI_SYS_VENDOR, "NEC"), DMI_MATCH(DMI_PRODUCT_NAME, "PC-LZ750LS"), }, }, { + /* Acer Aspire V5-573G */ .callback = battery_notification_delay_quirk, - .ident = "Acer Aspire V5-573G", .matches = { DMI_MATCH(DMI_SYS_VENDOR, "Acer"), DMI_MATCH(DMI_PRODUCT_NAME, "Aspire V5-573G"), }, }, + { + /* Point of View mobii wintab p800w */ + .callback = battery_ac_is_broken_quirk, + .matches = { + DMI_MATCH(DMI_BOARD_VENDOR, "AMI Corporation"), + DMI_MATCH(DMI_BOARD_NAME, "Aptio CRB"), + DMI_MATCH(DMI_BIOS_VERSION, "3BAIR1013"), + /* Above matches are too generic, add bios-date match */ + DMI_MATCH(DMI_BIOS_DATE, "08/22/2014"), + }, + }, + { + /* ECS EF20EA */ + .callback = battery_do_not_check_pmic_quirk, + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "EF20EA"), + }, + }, + { + /* Lenovo Ideapad Miix 320 */ + .callback = battery_do_not_check_pmic_quirk, + .matches = { + DMI_EXACT_MATCH(DMI_SYS_VENDOR, "LENOVO"), + DMI_EXACT_MATCH(DMI_PRODUCT_NAME, "80XF"), + DMI_EXACT_MATCH(DMI_PRODUCT_VERSION, "Lenovo MIIX 320-10ICR"), + }, + }, {}, }; @@ -1488,16 +1476,18 @@ static void __init acpi_battery_init_async(void *unused, async_cookie_t cookie) unsigned int i; int result; - for (i = 0; i < ARRAY_SIZE(acpi_battery_blacklist); i++) - if (acpi_dev_present(acpi_battery_blacklist[i], "1", -1)) { - pr_info(PREFIX ACPI_BATTERY_DEVICE_NAME - ": found native %s PMIC, not loading\n", - acpi_battery_blacklist[i]); - return; - } - dmi_check_system(bat_dmi_table); + if (battery_check_pmic) { + for (i = 0; i < ARRAY_SIZE(acpi_battery_blacklist); i++) + if (acpi_dev_present(acpi_battery_blacklist[i], "1", -1)) { + pr_info(PREFIX ACPI_BATTERY_DEVICE_NAME + ": found native %s PMIC, not loading\n", + acpi_battery_blacklist[i]); + return; + } + } + #ifdef CONFIG_ACPI_PROCFS_POWER acpi_battery_dir = acpi_lock_battery_dir(); if (!acpi_battery_dir) diff --git a/drivers/acpi/button.c b/drivers/acpi/button.c index f1cc4f9d31cd..2345a5ee2dbb 100644 --- a/drivers/acpi/button.c +++ b/drivers/acpi/button.c @@ -263,19 +263,6 @@ static int acpi_button_state_seq_show(struct seq_file *seq, void *offset) return 0; } -static int acpi_button_state_open_fs(struct inode *inode, struct file *file) -{ - return single_open(file, acpi_button_state_seq_show, PDE_DATA(inode)); -} - -static const struct file_operations acpi_button_state_fops = { - .owner = THIS_MODULE, - .open = acpi_button_state_open_fs, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int acpi_button_add_fs(struct acpi_device *device) { struct acpi_button *button = acpi_driver_data(device); @@ -311,9 +298,9 @@ static int acpi_button_add_fs(struct acpi_device *device) } /* create /proc/acpi/button/lid/LID/state */ - entry = proc_create_data(ACPI_BUTTON_FILE_STATE, - S_IRUGO, acpi_device_dir(device), - &acpi_button_state_fops, device); + entry = proc_create_single_data(ACPI_BUTTON_FILE_STATE, S_IRUGO, + acpi_device_dir(device), acpi_button_state_seq_show, + device); if (!entry) { ret = -ENODEV; goto remove_dev_dir; diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c index 735c74a4cbdb..d9ce4b162e2c 100644 --- a/drivers/acpi/cppc_acpi.c +++ b/drivers/acpi/cppc_acpi.c @@ -39,6 +39,7 @@ #include #include +#include #include #include #include @@ -49,7 +50,7 @@ struct cppc_pcc_data { struct mbox_chan *pcc_channel; void __iomem *pcc_comm_addr; bool pcc_channel_acquired; - ktime_t deadline; + unsigned int deadline_us; unsigned int pcc_mpar, pcc_mrtt, pcc_nominal; bool pending_pcc_write_cmd; /* Any pending/batched PCC write cmds? */ @@ -156,6 +157,9 @@ show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, highest_perf); show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, lowest_perf); show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, nominal_perf); show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, lowest_nonlinear_perf); +show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, lowest_freq); +show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, nominal_freq); + show_cppc_data(cppc_get_perf_ctrs, cppc_perf_fb_ctrs, reference_perf); show_cppc_data(cppc_get_perf_ctrs, cppc_perf_fb_ctrs, wraparound_time); @@ -183,6 +187,8 @@ static struct attribute *cppc_attrs[] = { &lowest_perf.attr, &lowest_nonlinear_perf.attr, &nominal_perf.attr, + &nominal_freq.attr, + &lowest_freq.attr, NULL }; @@ -193,42 +199,31 @@ static struct kobj_type cppc_ktype = { static int check_pcc_chan(int pcc_ss_id, bool chk_err_bit) { - int ret = -EIO, status = 0; + int ret, status; struct cppc_pcc_data *pcc_ss_data = pcc_data[pcc_ss_id]; struct acpi_pcct_shared_memory __iomem *generic_comm_base = pcc_ss_data->pcc_comm_addr; - ktime_t next_deadline = ktime_add(ktime_get(), - pcc_ss_data->deadline); if (!pcc_ss_data->platform_owns_pcc) return 0; - /* Retry in case the remote processor was too slow to catch up. */ - while (!ktime_after(ktime_get(), next_deadline)) { - /* - * Per spec, prior to boot the PCC space wil be initialized by - * platform and should have set the command completion bit when - * PCC can be used by OSPM - */ - status = readw_relaxed(&generic_comm_base->status); - if (status & PCC_CMD_COMPLETE_MASK) { - ret = 0; - if (chk_err_bit && (status & PCC_ERROR_MASK)) - ret = -EIO; - break; - } - /* - * Reducing the bus traffic in case this loop takes longer than - * a few retries. - */ - udelay(3); - } + /* + * Poll PCC status register every 3us(delay_us) for maximum of + * deadline_us(timeout_us) until PCC command complete bit is set(cond) + */ + ret = readw_relaxed_poll_timeout(&generic_comm_base->status, status, + status & PCC_CMD_COMPLETE_MASK, 3, + pcc_ss_data->deadline_us); - if (likely(!ret)) + if (likely(!ret)) { pcc_ss_data->platform_owns_pcc = false; - else - pr_err("PCC check channel failed for ss: %d. Status=%x\n", - pcc_ss_id, status); + if (chk_err_bit && (status & PCC_ERROR_MASK)) + ret = -EIO; + } + + if (unlikely(ret)) + pr_err("PCC check channel failed for ss: %d. ret=%d\n", + pcc_ss_id, ret); return ret; } @@ -580,7 +575,7 @@ static int register_pcc_channel(int pcc_ss_idx) * So add an arbitrary amount of wait on top of Nominal. */ usecs_lat = NUM_RETRIES * cppc_ss->latency; - pcc_data[pcc_ss_idx]->deadline = ns_to_ktime(usecs_lat * NSEC_PER_USEC); + pcc_data[pcc_ss_idx]->deadline_us = usecs_lat; pcc_data[pcc_ss_idx]->pcc_mrtt = cppc_ss->min_turnaround_time; pcc_data[pcc_ss_idx]->pcc_mpar = cppc_ss->max_access_rate; pcc_data[pcc_ss_idx]->pcc_nominal = cppc_ss->latency; @@ -613,7 +608,6 @@ bool __weak cpc_ffh_supported(void) return false; } - /** * pcc_data_alloc() - Allocate the pcc_data memory for pcc subspace * @@ -641,6 +635,34 @@ int pcc_data_alloc(int pcc_ss_id) return 0; } + +/* Check if CPPC revision + num_ent combination is supported */ +static bool is_cppc_supported(int revision, int num_ent) +{ + int expected_num_ent; + + switch (revision) { + case CPPC_V2_REV: + expected_num_ent = CPPC_V2_NUM_ENT; + break; + case CPPC_V3_REV: + expected_num_ent = CPPC_V3_NUM_ENT; + break; + default: + pr_debug("Firmware exports unsupported CPPC revision: %d\n", + revision); + return false; + } + + if (expected_num_ent != num_ent) { + pr_debug("Firmware exports %d entries. Expected: %d for CPPC rev:%d\n", + num_ent, expected_num_ent, revision); + return false; + } + + return true; +} + /* * An example CPC table looks like the following. * @@ -731,14 +753,6 @@ int acpi_cppc_processor_probe(struct acpi_processor *pr) cpc_obj->type); goto out_free; } - - /* Only support CPPCv2. Bail otherwise. */ - if (num_ent != CPPC_NUM_ENT) { - pr_debug("Firmware exports %d entries. Expected: %d\n", - num_ent, CPPC_NUM_ENT); - goto out_free; - } - cpc_ptr->num_entries = num_ent; /* Second entry should be revision. */ @@ -750,12 +764,10 @@ int acpi_cppc_processor_probe(struct acpi_processor *pr) cpc_obj->type); goto out_free; } + cpc_ptr->version = cpc_rev; - if (cpc_rev != CPPC_REV) { - pr_debug("Firmware exports revision:%d. Expected:%d\n", - cpc_rev, CPPC_REV); + if (!is_cppc_supported(cpc_rev, num_ent)) goto out_free; - } /* Iterate through remaining entries in _CPC */ for (i = 2; i < num_ent; i++) { @@ -808,6 +820,18 @@ int acpi_cppc_processor_probe(struct acpi_processor *pr) } } per_cpu(cpu_pcc_subspace_idx, pr->id) = pcc_subspace_id; + + /* + * Initialize the remaining cpc_regs as unsupported. + * Example: In case FW exposes CPPC v2, the below loop will initialize + * LOWEST_FREQ and NOMINAL_FREQ regs as unsupported + */ + for (i = num_ent - 2; i < MAX_CPC_REG_ENT; i++) { + cpc_ptr->cpc_regs[i].type = ACPI_TYPE_INTEGER; + cpc_ptr->cpc_regs[i].cpc_entry.int_value = 0; + } + + /* Store CPU Logical ID */ cpc_ptr->cpu_id = pr->id; @@ -1037,26 +1061,34 @@ int cppc_get_perf_caps(int cpunum, struct cppc_perf_caps *perf_caps) { struct cpc_desc *cpc_desc = per_cpu(cpc_desc_ptr, cpunum); struct cpc_register_resource *highest_reg, *lowest_reg, - *lowest_non_linear_reg, *nominal_reg; - u64 high, low, nom, min_nonlinear; + *lowest_non_linear_reg, *nominal_reg, + *low_freq_reg = NULL, *nom_freq_reg = NULL; + u64 high, low, nom, min_nonlinear, low_f = 0, nom_f = 0; int pcc_ss_id = per_cpu(cpu_pcc_subspace_idx, cpunum); - struct cppc_pcc_data *pcc_ss_data; + struct cppc_pcc_data *pcc_ss_data = NULL; int ret = 0, regs_in_pcc = 0; - if (!cpc_desc || pcc_ss_id < 0) { + if (!cpc_desc) { pr_debug("No CPC descriptor for CPU:%d\n", cpunum); return -ENODEV; } - pcc_ss_data = pcc_data[pcc_ss_id]; highest_reg = &cpc_desc->cpc_regs[HIGHEST_PERF]; lowest_reg = &cpc_desc->cpc_regs[LOWEST_PERF]; lowest_non_linear_reg = &cpc_desc->cpc_regs[LOW_NON_LINEAR_PERF]; nominal_reg = &cpc_desc->cpc_regs[NOMINAL_PERF]; + low_freq_reg = &cpc_desc->cpc_regs[LOWEST_FREQ]; + nom_freq_reg = &cpc_desc->cpc_regs[NOMINAL_FREQ]; /* Are any of the regs PCC ?*/ if (CPC_IN_PCC(highest_reg) || CPC_IN_PCC(lowest_reg) || - CPC_IN_PCC(lowest_non_linear_reg) || CPC_IN_PCC(nominal_reg)) { + CPC_IN_PCC(lowest_non_linear_reg) || CPC_IN_PCC(nominal_reg) || + CPC_IN_PCC(low_freq_reg) || CPC_IN_PCC(nom_freq_reg)) { + if (pcc_ss_id < 0) { + pr_debug("Invalid pcc_ss_id\n"); + return -ENODEV; + } + pcc_ss_data = pcc_data[pcc_ss_id]; regs_in_pcc = 1; down_write(&pcc_ss_data->pcc_lock); /* Ring doorbell once to update PCC subspace */ @@ -1081,6 +1113,17 @@ int cppc_get_perf_caps(int cpunum, struct cppc_perf_caps *perf_caps) if (!high || !low || !nom || !min_nonlinear) ret = -EFAULT; + /* Read optional lowest and nominal frequencies if present */ + if (CPC_SUPPORTED(low_freq_reg)) + cpc_read(cpunum, low_freq_reg, &low_f); + + if (CPC_SUPPORTED(nom_freq_reg)) + cpc_read(cpunum, nom_freq_reg, &nom_f); + + perf_caps->lowest_freq = low_f; + perf_caps->nominal_freq = nom_f; + + out_err: if (regs_in_pcc) up_write(&pcc_ss_data->pcc_lock); @@ -1101,16 +1144,15 @@ int cppc_get_perf_ctrs(int cpunum, struct cppc_perf_fb_ctrs *perf_fb_ctrs) struct cpc_register_resource *delivered_reg, *reference_reg, *ref_perf_reg, *ctr_wrap_reg; int pcc_ss_id = per_cpu(cpu_pcc_subspace_idx, cpunum); - struct cppc_pcc_data *pcc_ss_data; + struct cppc_pcc_data *pcc_ss_data = NULL; u64 delivered, reference, ref_perf, ctr_wrap_time; int ret = 0, regs_in_pcc = 0; - if (!cpc_desc || pcc_ss_id < 0) { + if (!cpc_desc) { pr_debug("No CPC descriptor for CPU:%d\n", cpunum); return -ENODEV; } - pcc_ss_data = pcc_data[pcc_ss_id]; delivered_reg = &cpc_desc->cpc_regs[DELIVERED_CTR]; reference_reg = &cpc_desc->cpc_regs[REFERENCE_CTR]; ref_perf_reg = &cpc_desc->cpc_regs[REFERENCE_PERF]; @@ -1126,6 +1168,11 @@ int cppc_get_perf_ctrs(int cpunum, struct cppc_perf_fb_ctrs *perf_fb_ctrs) /* Are any of the regs PCC ?*/ if (CPC_IN_PCC(delivered_reg) || CPC_IN_PCC(reference_reg) || CPC_IN_PCC(ctr_wrap_reg) || CPC_IN_PCC(ref_perf_reg)) { + if (pcc_ss_id < 0) { + pr_debug("Invalid pcc_ss_id\n"); + return -ENODEV; + } + pcc_ss_data = pcc_data[pcc_ss_id]; down_write(&pcc_ss_data->pcc_lock); regs_in_pcc = 1; /* Ring doorbell once to update PCC subspace */ @@ -1176,15 +1223,14 @@ int cppc_set_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls) struct cpc_desc *cpc_desc = per_cpu(cpc_desc_ptr, cpu); struct cpc_register_resource *desired_reg; int pcc_ss_id = per_cpu(cpu_pcc_subspace_idx, cpu); - struct cppc_pcc_data *pcc_ss_data; + struct cppc_pcc_data *pcc_ss_data = NULL; int ret = 0; - if (!cpc_desc || pcc_ss_id < 0) { + if (!cpc_desc) { pr_debug("No CPC descriptor for CPU:%d\n", cpu); return -ENODEV; } - pcc_ss_data = pcc_data[pcc_ss_id]; desired_reg = &cpc_desc->cpc_regs[DESIRED_PERF]; /* @@ -1195,6 +1241,11 @@ int cppc_set_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls) * achieve that goal here */ if (CPC_IN_PCC(desired_reg)) { + if (pcc_ss_id < 0) { + pr_debug("Invalid pcc_ss_id\n"); + return -ENODEV; + } + pcc_ss_data = pcc_data[pcc_ss_id]; down_read(&pcc_ss_data->pcc_lock); /* BEGIN Phase-I */ if (pcc_ss_data->platform_owns_pcc) { ret = check_pcc_chan(pcc_ss_id, false); diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c index 3d96e4da2d98..a7c2673ffd36 100644 --- a/drivers/acpi/device_pm.c +++ b/drivers/acpi/device_pm.c @@ -1257,10 +1257,7 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on) struct acpi_device *adev = ACPI_COMPANION(dev); if (!adev) - return -ENODEV; - - if (dev->pm_domain) - return -EEXIST; + return 0; /* * Only attach the power domain to the first device if the @@ -1268,7 +1265,7 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on) * management twice. */ if (!acpi_device_is_first_physical_node(adev, dev)) - return -EBUSY; + return 0; acpi_add_pm_notifier(adev, dev, acpi_pm_notify_work_func); dev_pm_domain_set(dev, &acpi_general_pm_domain); @@ -1278,7 +1275,7 @@ int acpi_dev_pm_attach(struct device *dev, bool power_on) } dev->pm_domain->detach = acpi_dev_pm_detach; - return 0; + return 1; } EXPORT_SYMBOL_GPL(acpi_dev_pm_attach); #endif /* CONFIG_PM */ diff --git a/drivers/acpi/ec.c b/drivers/acpi/ec.c index 30a572956557..442a9e24f439 100644 --- a/drivers/acpi/ec.c +++ b/drivers/acpi/ec.c @@ -1034,6 +1034,12 @@ void acpi_ec_unblock_transactions(void) acpi_ec_start(first_ec, true); } +void acpi_ec_dispatch_gpe(void) +{ + if (first_ec) + acpi_dispatch_gpe(NULL, first_ec->gpe); +} + /* -------------------------------------------------------------------------- Event Management -------------------------------------------------------------------------- */ @@ -2031,6 +2037,17 @@ static inline void acpi_ec_query_exit(void) } } +static const struct dmi_system_id acpi_ec_no_wakeup[] = { + { + .ident = "Thinkpad X1 Carbon 6th", + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"), + DMI_MATCH(DMI_PRODUCT_NAME, "20KGS3JF01"), + }, + }, + { }, +}; + int __init acpi_ec_init(void) { int result; @@ -2041,6 +2058,15 @@ int __init acpi_ec_init(void) if (result) return result; + /* + * Disable EC wakeup on following systems to prevent periodic + * wakeup from EC GPE. + */ + if (dmi_check_system(acpi_ec_no_wakeup)) { + ec_no_wakeup = true; + pr_debug("Disabling EC wakeup on suspend-to-idle\n"); + } + /* Drivers must be started after acpi_ec_query_init() */ dsdt_fail = acpi_bus_register_driver(&acpi_ec_driver); /* diff --git a/drivers/acpi/fan.c b/drivers/acpi/fan.c index 3563103590c6..fe0183d48dcd 100644 --- a/drivers/acpi/fan.c +++ b/drivers/acpi/fan.c @@ -298,8 +298,8 @@ static int acpi_fan_get_fps(struct acpi_device *device) } fan->fps_count = obj->package.count - 1; /* minus revision field */ - fan->fps = devm_kzalloc(&device->dev, - fan->fps_count * sizeof(struct acpi_fan_fps), + fan->fps = devm_kcalloc(&device->dev, + fan->fps_count, sizeof(struct acpi_fan_fps), GFP_KERNEL); if (!fan->fps) { dev_err(&device->dev, "Not enough memory\n"); diff --git a/drivers/acpi/internal.h b/drivers/acpi/internal.h index 1d0a501bc7f0..530a3f675490 100644 --- a/drivers/acpi/internal.h +++ b/drivers/acpi/internal.h @@ -188,6 +188,7 @@ int acpi_ec_ecdt_probe(void); int acpi_ec_dsdt_probe(void); void acpi_ec_block_transactions(void); void acpi_ec_unblock_transactions(void); +void acpi_ec_dispatch_gpe(void); int acpi_ec_add_query_handler(struct acpi_ec *ec, u8 query_bit, acpi_handle handle, acpi_ec_query_func func, void *data); diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c index e2235ed3e4be..d15814e1727f 100644 --- a/drivers/acpi/nfit/core.c +++ b/drivers/acpi/nfit/core.c @@ -1082,9 +1082,10 @@ static int __nfit_mem_init(struct acpi_nfit_desc *acpi_desc, continue; nfit_mem->nfit_flush = nfit_flush; flush = nfit_flush->flush; - nfit_mem->flush_wpq = devm_kzalloc(acpi_desc->dev, - flush->hint_count - * sizeof(struct resource), GFP_KERNEL); + nfit_mem->flush_wpq = devm_kcalloc(acpi_desc->dev, + flush->hint_count, + sizeof(struct resource), + GFP_KERNEL); if (!nfit_mem->flush_wpq) return -ENOMEM; for (i = 0; i < flush->hint_count; i++) { @@ -1978,19 +1979,8 @@ static ssize_t range_index_show(struct device *dev, } static DEVICE_ATTR_RO(range_index); -static ssize_t ecc_unit_size_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - struct nd_region *nd_region = to_nd_region(dev); - struct nfit_spa *nfit_spa = nd_region_provider_data(nd_region); - - return sprintf(buf, "%d\n", nfit_spa->clear_err_unit); -} -static DEVICE_ATTR_RO(ecc_unit_size); - static struct attribute *acpi_nfit_region_attributes[] = { &dev_attr_range_index.attr, - &dev_attr_ecc_unit_size.attr, NULL, }; diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c index 0da18bde6a16..7433035ded95 100644 --- a/drivers/acpi/pci_root.c +++ b/drivers/acpi/pci_root.c @@ -153,6 +153,7 @@ static struct pci_osc_bit_struct pci_osc_control_bit[] = { { OSC_PCI_EXPRESS_PME_CONTROL, "PME" }, { OSC_PCI_EXPRESS_AER_CONTROL, "AER" }, { OSC_PCI_EXPRESS_CAPABILITY_CONTROL, "PCIeCapability" }, + { OSC_PCI_EXPRESS_LTR_CONTROL, "LTR" }, }; static void decode_osc_bits(struct acpi_pci_root *root, char *msg, u32 word, @@ -472,9 +473,17 @@ static void negotiate_os_control(struct acpi_pci_root *root, int *no_aspm) } control = OSC_PCI_EXPRESS_CAPABILITY_CONTROL - | OSC_PCI_EXPRESS_NATIVE_HP_CONTROL | OSC_PCI_EXPRESS_PME_CONTROL; + if (IS_ENABLED(CONFIG_PCIEASPM)) + control |= OSC_PCI_EXPRESS_LTR_CONTROL; + + if (IS_ENABLED(CONFIG_HOTPLUG_PCI_PCIE)) + control |= OSC_PCI_EXPRESS_NATIVE_HP_CONTROL; + + if (IS_ENABLED(CONFIG_HOTPLUG_PCI_SHPC)) + control |= OSC_PCI_SHPC_NATIVE_HP_CONTROL; + if (pci_aer_available()) { if (aer_acpi_firmware_first()) dev_info(&device->dev, @@ -900,11 +909,15 @@ struct pci_bus *acpi_pci_root_create(struct acpi_pci_root *root, host_bridge = to_pci_host_bridge(bus->bridge); if (!(root->osc_control_set & OSC_PCI_EXPRESS_NATIVE_HP_CONTROL)) - host_bridge->native_hotplug = 0; + host_bridge->native_pcie_hotplug = 0; + if (!(root->osc_control_set & OSC_PCI_SHPC_NATIVE_HP_CONTROL)) + host_bridge->native_shpc_hotplug = 0; if (!(root->osc_control_set & OSC_PCI_EXPRESS_AER_CONTROL)) host_bridge->native_aer = 0; if (!(root->osc_control_set & OSC_PCI_EXPRESS_PME_CONTROL)) host_bridge->native_pme = 0; + if (!(root->osc_control_set & OSC_PCI_EXPRESS_LTR_CONTROL)) + host_bridge->native_ltr = 0; pci_scan_child_bus(bus); pci_set_host_bridge_release(host_bridge, acpi_pci_root_release_info, diff --git a/drivers/acpi/pptt.c b/drivers/acpi/pptt.c new file mode 100644 index 000000000000..e5ea1974d1e3 --- /dev/null +++ b/drivers/acpi/pptt.c @@ -0,0 +1,655 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * pptt.c - parsing of Processor Properties Topology Table (PPTT) + * + * Copyright (C) 2018, ARM + * + * This file implements parsing of the Processor Properties Topology Table + * which is optionally used to describe the processor and cache topology. + * Due to the relative pointers used throughout the table, this doesn't + * leverage the existing subtable parsing in the kernel. + * + * The PPTT structure is an inverted tree, with each node potentially + * holding one or two inverted tree data structures describing + * the caches available at that level. Each cache structure optionally + * contains properties describing the cache at a given level which can be + * used to override hardware probed values. + */ +#define pr_fmt(fmt) "ACPI PPTT: " fmt + +#include +#include +#include + +static struct acpi_subtable_header *fetch_pptt_subtable(struct acpi_table_header *table_hdr, + u32 pptt_ref) +{ + struct acpi_subtable_header *entry; + + /* there isn't a subtable at reference 0 */ + if (pptt_ref < sizeof(struct acpi_subtable_header)) + return NULL; + + if (pptt_ref + sizeof(struct acpi_subtable_header) > table_hdr->length) + return NULL; + + entry = ACPI_ADD_PTR(struct acpi_subtable_header, table_hdr, pptt_ref); + + if (entry->length == 0) + return NULL; + + if (pptt_ref + entry->length > table_hdr->length) + return NULL; + + return entry; +} + +static struct acpi_pptt_processor *fetch_pptt_node(struct acpi_table_header *table_hdr, + u32 pptt_ref) +{ + return (struct acpi_pptt_processor *)fetch_pptt_subtable(table_hdr, pptt_ref); +} + +static struct acpi_pptt_cache *fetch_pptt_cache(struct acpi_table_header *table_hdr, + u32 pptt_ref) +{ + return (struct acpi_pptt_cache *)fetch_pptt_subtable(table_hdr, pptt_ref); +} + +static struct acpi_subtable_header *acpi_get_pptt_resource(struct acpi_table_header *table_hdr, + struct acpi_pptt_processor *node, + int resource) +{ + u32 *ref; + + if (resource >= node->number_of_priv_resources) + return NULL; + + ref = ACPI_ADD_PTR(u32, node, sizeof(struct acpi_pptt_processor)); + ref += resource; + + return fetch_pptt_subtable(table_hdr, *ref); +} + +static inline bool acpi_pptt_match_type(int table_type, int type) +{ + return ((table_type & ACPI_PPTT_MASK_CACHE_TYPE) == type || + table_type & ACPI_PPTT_CACHE_TYPE_UNIFIED & type); +} + +/** + * acpi_pptt_walk_cache() - Attempt to find the requested acpi_pptt_cache + * @table_hdr: Pointer to the head of the PPTT table + * @local_level: passed res reflects this cache level + * @res: cache resource in the PPTT we want to walk + * @found: returns a pointer to the requested level if found + * @level: the requested cache level + * @type: the requested cache type + * + * Attempt to find a given cache level, while counting the max number + * of cache levels for the cache node. + * + * Given a pptt resource, verify that it is a cache node, then walk + * down each level of caches, counting how many levels are found + * as well as checking the cache type (icache, dcache, unified). If a + * level & type match, then we set found, and continue the search. + * Once the entire cache branch has been walked return its max + * depth. + * + * Return: The cache structure and the level we terminated with. + */ +static int acpi_pptt_walk_cache(struct acpi_table_header *table_hdr, + int local_level, + struct acpi_subtable_header *res, + struct acpi_pptt_cache **found, + int level, int type) +{ + struct acpi_pptt_cache *cache; + + if (res->type != ACPI_PPTT_TYPE_CACHE) + return 0; + + cache = (struct acpi_pptt_cache *) res; + while (cache) { + local_level++; + + if (local_level == level && + cache->flags & ACPI_PPTT_CACHE_TYPE_VALID && + acpi_pptt_match_type(cache->attributes, type)) { + if (*found != NULL && cache != *found) + pr_warn("Found duplicate cache level/type unable to determine uniqueness\n"); + + pr_debug("Found cache @ level %d\n", level); + *found = cache; + /* + * continue looking at this node's resource list + * to verify that we don't find a duplicate + * cache node. + */ + } + cache = fetch_pptt_cache(table_hdr, cache->next_level_of_cache); + } + return local_level; +} + +static struct acpi_pptt_cache *acpi_find_cache_level(struct acpi_table_header *table_hdr, + struct acpi_pptt_processor *cpu_node, + int *starting_level, int level, + int type) +{ + struct acpi_subtable_header *res; + int number_of_levels = *starting_level; + int resource = 0; + struct acpi_pptt_cache *ret = NULL; + int local_level; + + /* walk down from processor node */ + while ((res = acpi_get_pptt_resource(table_hdr, cpu_node, resource))) { + resource++; + + local_level = acpi_pptt_walk_cache(table_hdr, *starting_level, + res, &ret, level, type); + /* + * we are looking for the max depth. Since its potentially + * possible for a given node to have resources with differing + * depths verify that the depth we have found is the largest. + */ + if (number_of_levels < local_level) + number_of_levels = local_level; + } + if (number_of_levels > *starting_level) + *starting_level = number_of_levels; + + return ret; +} + +/** + * acpi_count_levels() - Given a PPTT table, and a cpu node, count the caches + * @table_hdr: Pointer to the head of the PPTT table + * @cpu_node: processor node we wish to count caches for + * + * Given a processor node containing a processing unit, walk into it and count + * how many levels exist solely for it, and then walk up each level until we hit + * the root node (ignore the package level because it may be possible to have + * caches that exist across packages). Count the number of cache levels that + * exist at each level on the way up. + * + * Return: Total number of levels found. + */ +static int acpi_count_levels(struct acpi_table_header *table_hdr, + struct acpi_pptt_processor *cpu_node) +{ + int total_levels = 0; + + do { + acpi_find_cache_level(table_hdr, cpu_node, &total_levels, 0, 0); + cpu_node = fetch_pptt_node(table_hdr, cpu_node->parent); + } while (cpu_node); + + return total_levels; +} + +/** + * acpi_pptt_leaf_node() - Given a processor node, determine if its a leaf + * @table_hdr: Pointer to the head of the PPTT table + * @node: passed node is checked to see if its a leaf + * + * Determine if the *node parameter is a leaf node by iterating the + * PPTT table, looking for nodes which reference it. + * + * Return: 0 if we find a node referencing the passed node (or table error), + * or 1 if we don't. + */ +static int acpi_pptt_leaf_node(struct acpi_table_header *table_hdr, + struct acpi_pptt_processor *node) +{ + struct acpi_subtable_header *entry; + unsigned long table_end; + u32 node_entry; + struct acpi_pptt_processor *cpu_node; + u32 proc_sz; + + table_end = (unsigned long)table_hdr + table_hdr->length; + node_entry = ACPI_PTR_DIFF(node, table_hdr); + entry = ACPI_ADD_PTR(struct acpi_subtable_header, table_hdr, + sizeof(struct acpi_table_pptt)); + proc_sz = sizeof(struct acpi_pptt_processor *); + + while ((unsigned long)entry + proc_sz < table_end) { + cpu_node = (struct acpi_pptt_processor *)entry; + if (entry->type == ACPI_PPTT_TYPE_PROCESSOR && + cpu_node->parent == node_entry) + return 0; + if (entry->length == 0) + return 0; + entry = ACPI_ADD_PTR(struct acpi_subtable_header, entry, + entry->length); + + } + return 1; +} + +/** + * acpi_find_processor_node() - Given a PPTT table find the requested processor + * @table_hdr: Pointer to the head of the PPTT table + * @acpi_cpu_id: cpu we are searching for + * + * Find the subtable entry describing the provided processor. + * This is done by iterating the PPTT table looking for processor nodes + * which have an acpi_processor_id that matches the acpi_cpu_id parameter + * passed into the function. If we find a node that matches this criteria + * we verify that its a leaf node in the topology rather than depending + * on the valid flag, which doesn't need to be set for leaf nodes. + * + * Return: NULL, or the processors acpi_pptt_processor* + */ +static struct acpi_pptt_processor *acpi_find_processor_node(struct acpi_table_header *table_hdr, + u32 acpi_cpu_id) +{ + struct acpi_subtable_header *entry; + unsigned long table_end; + struct acpi_pptt_processor *cpu_node; + u32 proc_sz; + + table_end = (unsigned long)table_hdr + table_hdr->length; + entry = ACPI_ADD_PTR(struct acpi_subtable_header, table_hdr, + sizeof(struct acpi_table_pptt)); + proc_sz = sizeof(struct acpi_pptt_processor *); + + /* find the processor structure associated with this cpuid */ + while ((unsigned long)entry + proc_sz < table_end) { + cpu_node = (struct acpi_pptt_processor *)entry; + + if (entry->length == 0) { + pr_warn("Invalid zero length subtable\n"); + break; + } + if (entry->type == ACPI_PPTT_TYPE_PROCESSOR && + acpi_cpu_id == cpu_node->acpi_processor_id && + acpi_pptt_leaf_node(table_hdr, cpu_node)) { + return (struct acpi_pptt_processor *)entry; + } + + entry = ACPI_ADD_PTR(struct acpi_subtable_header, entry, + entry->length); + } + + return NULL; +} + +static int acpi_find_cache_levels(struct acpi_table_header *table_hdr, + u32 acpi_cpu_id) +{ + int number_of_levels = 0; + struct acpi_pptt_processor *cpu; + + cpu = acpi_find_processor_node(table_hdr, acpi_cpu_id); + if (cpu) + number_of_levels = acpi_count_levels(table_hdr, cpu); + + return number_of_levels; +} + +static u8 acpi_cache_type(enum cache_type type) +{ + switch (type) { + case CACHE_TYPE_DATA: + pr_debug("Looking for data cache\n"); + return ACPI_PPTT_CACHE_TYPE_DATA; + case CACHE_TYPE_INST: + pr_debug("Looking for instruction cache\n"); + return ACPI_PPTT_CACHE_TYPE_INSTR; + default: + case CACHE_TYPE_UNIFIED: + pr_debug("Looking for unified cache\n"); + /* + * It is important that ACPI_PPTT_CACHE_TYPE_UNIFIED + * contains the bit pattern that will match both + * ACPI unified bit patterns because we use it later + * to match both cases. + */ + return ACPI_PPTT_CACHE_TYPE_UNIFIED; + } +} + +static struct acpi_pptt_cache *acpi_find_cache_node(struct acpi_table_header *table_hdr, + u32 acpi_cpu_id, + enum cache_type type, + unsigned int level, + struct acpi_pptt_processor **node) +{ + int total_levels = 0; + struct acpi_pptt_cache *found = NULL; + struct acpi_pptt_processor *cpu_node; + u8 acpi_type = acpi_cache_type(type); + + pr_debug("Looking for CPU %d's level %d cache type %d\n", + acpi_cpu_id, level, acpi_type); + + cpu_node = acpi_find_processor_node(table_hdr, acpi_cpu_id); + + while (cpu_node && !found) { + found = acpi_find_cache_level(table_hdr, cpu_node, + &total_levels, level, acpi_type); + *node = cpu_node; + cpu_node = fetch_pptt_node(table_hdr, cpu_node->parent); + } + + return found; +} + +/* total number of attributes checked by the properties code */ +#define PPTT_CHECKED_ATTRIBUTES 4 + +/** + * update_cache_properties() - Update cacheinfo for the given processor + * @this_leaf: Kernel cache info structure being updated + * @found_cache: The PPTT node describing this cache instance + * @cpu_node: A unique reference to describe this cache instance + * + * The ACPI spec implies that the fields in the cache structures are used to + * extend and correct the information probed from the hardware. Lets only + * set fields that we determine are VALID. + * + * Return: nothing. Side effect of updating the global cacheinfo + */ +static void update_cache_properties(struct cacheinfo *this_leaf, + struct acpi_pptt_cache *found_cache, + struct acpi_pptt_processor *cpu_node) +{ + int valid_flags = 0; + + this_leaf->fw_token = cpu_node; + if (found_cache->flags & ACPI_PPTT_SIZE_PROPERTY_VALID) { + this_leaf->size = found_cache->size; + valid_flags++; + } + if (found_cache->flags & ACPI_PPTT_LINE_SIZE_VALID) { + this_leaf->coherency_line_size = found_cache->line_size; + valid_flags++; + } + if (found_cache->flags & ACPI_PPTT_NUMBER_OF_SETS_VALID) { + this_leaf->number_of_sets = found_cache->number_of_sets; + valid_flags++; + } + if (found_cache->flags & ACPI_PPTT_ASSOCIATIVITY_VALID) { + this_leaf->ways_of_associativity = found_cache->associativity; + valid_flags++; + } + if (found_cache->flags & ACPI_PPTT_WRITE_POLICY_VALID) { + switch (found_cache->attributes & ACPI_PPTT_MASK_WRITE_POLICY) { + case ACPI_PPTT_CACHE_POLICY_WT: + this_leaf->attributes = CACHE_WRITE_THROUGH; + break; + case ACPI_PPTT_CACHE_POLICY_WB: + this_leaf->attributes = CACHE_WRITE_BACK; + break; + } + } + if (found_cache->flags & ACPI_PPTT_ALLOCATION_TYPE_VALID) { + switch (found_cache->attributes & ACPI_PPTT_MASK_ALLOCATION_TYPE) { + case ACPI_PPTT_CACHE_READ_ALLOCATE: + this_leaf->attributes |= CACHE_READ_ALLOCATE; + break; + case ACPI_PPTT_CACHE_WRITE_ALLOCATE: + this_leaf->attributes |= CACHE_WRITE_ALLOCATE; + break; + case ACPI_PPTT_CACHE_RW_ALLOCATE: + case ACPI_PPTT_CACHE_RW_ALLOCATE_ALT: + this_leaf->attributes |= + CACHE_READ_ALLOCATE | CACHE_WRITE_ALLOCATE; + break; + } + } + /* + * If the above flags are valid, and the cache type is NOCACHE + * update the cache type as well. + */ + if (this_leaf->type == CACHE_TYPE_NOCACHE && + valid_flags == PPTT_CHECKED_ATTRIBUTES) + this_leaf->type = CACHE_TYPE_UNIFIED; +} + +static void cache_setup_acpi_cpu(struct acpi_table_header *table, + unsigned int cpu) +{ + struct acpi_pptt_cache *found_cache; + struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); + u32 acpi_cpu_id = get_acpi_id_for_cpu(cpu); + struct cacheinfo *this_leaf; + unsigned int index = 0; + struct acpi_pptt_processor *cpu_node = NULL; + + while (index < get_cpu_cacheinfo(cpu)->num_leaves) { + this_leaf = this_cpu_ci->info_list + index; + found_cache = acpi_find_cache_node(table, acpi_cpu_id, + this_leaf->type, + this_leaf->level, + &cpu_node); + pr_debug("found = %p %p\n", found_cache, cpu_node); + if (found_cache) + update_cache_properties(this_leaf, + found_cache, + cpu_node); + + index++; + } +} + +/* Passing level values greater than this will result in search termination */ +#define PPTT_ABORT_PACKAGE 0xFF + +static struct acpi_pptt_processor *acpi_find_processor_package_id(struct acpi_table_header *table_hdr, + struct acpi_pptt_processor *cpu, + int level, int flag) +{ + struct acpi_pptt_processor *prev_node; + + while (cpu && level) { + if (cpu->flags & flag) + break; + pr_debug("level %d\n", level); + prev_node = fetch_pptt_node(table_hdr, cpu->parent); + if (prev_node == NULL) + break; + cpu = prev_node; + level--; + } + return cpu; +} + +/** + * topology_get_acpi_cpu_tag() - Find a unique topology value for a feature + * @table: Pointer to the head of the PPTT table + * @cpu: Kernel logical cpu number + * @level: A level that terminates the search + * @flag: A flag which terminates the search + * + * Get a unique value given a cpu, and a topology level, that can be + * matched to determine which cpus share common topological features + * at that level. + * + * Return: Unique value, or -ENOENT if unable to locate cpu + */ +static int topology_get_acpi_cpu_tag(struct acpi_table_header *table, + unsigned int cpu, int level, int flag) +{ + struct acpi_pptt_processor *cpu_node; + u32 acpi_cpu_id = get_acpi_id_for_cpu(cpu); + + cpu_node = acpi_find_processor_node(table, acpi_cpu_id); + if (cpu_node) { + cpu_node = acpi_find_processor_package_id(table, cpu_node, + level, flag); + /* Only the first level has a guaranteed id */ + if (level == 0) + return cpu_node->acpi_processor_id; + return ACPI_PTR_DIFF(cpu_node, table); + } + pr_warn_once("PPTT table found, but unable to locate core %d (%d)\n", + cpu, acpi_cpu_id); + return -ENOENT; +} + +static int find_acpi_cpu_topology_tag(unsigned int cpu, int level, int flag) +{ + struct acpi_table_header *table; + acpi_status status; + int retval; + + status = acpi_get_table(ACPI_SIG_PPTT, 0, &table); + if (ACPI_FAILURE(status)) { + pr_warn_once("No PPTT table found, cpu topology may be inaccurate\n"); + return -ENOENT; + } + retval = topology_get_acpi_cpu_tag(table, cpu, level, flag); + pr_debug("Topology Setup ACPI cpu %d, level %d ret = %d\n", + cpu, level, retval); + acpi_put_table(table); + + return retval; +} + +/** + * acpi_find_last_cache_level() - Determines the number of cache levels for a PE + * @cpu: Kernel logical cpu number + * + * Given a logical cpu number, returns the number of levels of cache represented + * in the PPTT. Errors caused by lack of a PPTT table, or otherwise, return 0 + * indicating we didn't find any cache levels. + * + * Return: Cache levels visible to this core. + */ +int acpi_find_last_cache_level(unsigned int cpu) +{ + u32 acpi_cpu_id; + struct acpi_table_header *table; + int number_of_levels = 0; + acpi_status status; + + pr_debug("Cache Setup find last level cpu=%d\n", cpu); + + acpi_cpu_id = get_acpi_id_for_cpu(cpu); + status = acpi_get_table(ACPI_SIG_PPTT, 0, &table); + if (ACPI_FAILURE(status)) { + pr_warn_once("No PPTT table found, cache topology may be inaccurate\n"); + } else { + number_of_levels = acpi_find_cache_levels(table, acpi_cpu_id); + acpi_put_table(table); + } + pr_debug("Cache Setup find last level level=%d\n", number_of_levels); + + return number_of_levels; +} + +/** + * cache_setup_acpi() - Override CPU cache topology with data from the PPTT + * @cpu: Kernel logical cpu number + * + * Updates the global cache info provided by cpu_get_cacheinfo() + * when there are valid properties in the acpi_pptt_cache nodes. A + * successful parse may not result in any updates if none of the + * cache levels have any valid flags set. Futher, a unique value is + * associated with each known CPU cache entry. This unique value + * can be used to determine whether caches are shared between cpus. + * + * Return: -ENOENT on failure to find table, or 0 on success + */ +int cache_setup_acpi(unsigned int cpu) +{ + struct acpi_table_header *table; + acpi_status status; + + pr_debug("Cache Setup ACPI cpu %d\n", cpu); + + status = acpi_get_table(ACPI_SIG_PPTT, 0, &table); + if (ACPI_FAILURE(status)) { + pr_warn_once("No PPTT table found, cache topology may be inaccurate\n"); + return -ENOENT; + } + + cache_setup_acpi_cpu(table, cpu); + acpi_put_table(table); + + return status; +} + +/** + * find_acpi_cpu_topology() - Determine a unique topology value for a given cpu + * @cpu: Kernel logical cpu number + * @level: The topological level for which we would like a unique ID + * + * Determine a topology unique ID for each thread/core/cluster/mc_grouping + * /socket/etc. This ID can then be used to group peers, which will have + * matching ids. + * + * The search terminates when either the requested level is found or + * we reach a root node. Levels beyond the termination point will return the + * same unique ID. The unique id for level 0 is the acpi processor id. All + * other levels beyond this use a generated value to uniquely identify + * a topological feature. + * + * Return: -ENOENT if the PPTT doesn't exist, or the cpu cannot be found. + * Otherwise returns a value which represents a unique topological feature. + */ +int find_acpi_cpu_topology(unsigned int cpu, int level) +{ + return find_acpi_cpu_topology_tag(cpu, level, 0); +} + +/** + * find_acpi_cpu_cache_topology() - Determine a unique cache topology value + * @cpu: Kernel logical cpu number + * @level: The cache level for which we would like a unique ID + * + * Determine a unique ID for each unified cache in the system + * + * Return: -ENOENT if the PPTT doesn't exist, or the cpu cannot be found. + * Otherwise returns a value which represents a unique topological feature. + */ +int find_acpi_cpu_cache_topology(unsigned int cpu, int level) +{ + struct acpi_table_header *table; + struct acpi_pptt_cache *found_cache; + acpi_status status; + u32 acpi_cpu_id = get_acpi_id_for_cpu(cpu); + struct acpi_pptt_processor *cpu_node = NULL; + int ret = -1; + + status = acpi_get_table(ACPI_SIG_PPTT, 0, &table); + if (ACPI_FAILURE(status)) { + pr_warn_once("No PPTT table found, topology may be inaccurate\n"); + return -ENOENT; + } + + found_cache = acpi_find_cache_node(table, acpi_cpu_id, + CACHE_TYPE_UNIFIED, + level, + &cpu_node); + if (found_cache) + ret = ACPI_PTR_DIFF(cpu_node, table); + + acpi_put_table(table); + + return ret; +} + + +/** + * find_acpi_cpu_topology_package() - Determine a unique cpu package value + * @cpu: Kernel logical cpu number + * + * Determine a topology unique package ID for the given cpu. + * This ID can then be used to group peers, which will have matching ids. + * + * The search terminates when either a level is found with the PHYSICAL_PACKAGE + * flag set or we reach a root node. + * + * Return: -ENOENT if the PPTT doesn't exist, or the cpu cannot be found. + * Otherwise returns a value which represents the package for this cpu. + */ +int find_acpi_cpu_topology_package(unsigned int cpu) +{ + return find_acpi_cpu_topology_tag(cpu, PPTT_ABORT_PACKAGE, + ACPI_PPTT_PHYSICAL_PACKAGE); +} diff --git a/drivers/acpi/processor_perflib.c b/drivers/acpi/processor_perflib.c index a651ab3490d8..a303fd0e108c 100644 --- a/drivers/acpi/processor_perflib.c +++ b/drivers/acpi/processor_perflib.c @@ -343,8 +343,9 @@ static int acpi_processor_get_performance_states(struct acpi_processor *pr) pr->performance->state_count = pss->package.count; pr->performance->states = - kmalloc(sizeof(struct acpi_processor_px) * pss->package.count, - GFP_KERNEL); + kmalloc_array(pss->package.count, + sizeof(struct acpi_processor_px), + GFP_KERNEL); if (!pr->performance->states) { result = -ENOMEM; goto end; diff --git a/drivers/acpi/processor_throttling.c b/drivers/acpi/processor_throttling.c index 7f9aff4b8d62..fbc936cf2025 100644 --- a/drivers/acpi/processor_throttling.c +++ b/drivers/acpi/processor_throttling.c @@ -534,8 +534,9 @@ static int acpi_processor_get_throttling_states(struct acpi_processor *pr) pr->throttling.state_count = tss->package.count; pr->throttling.states_tss = - kmalloc(sizeof(struct acpi_processor_tx_tss) * tss->package.count, - GFP_KERNEL); + kmalloc_array(tss->package.count, + sizeof(struct acpi_processor_tx_tss), + GFP_KERNEL); if (!pr->throttling.states_tss) { result = -ENOMEM; goto end; diff --git a/drivers/acpi/reboot.c b/drivers/acpi/reboot.c index 71769fd687b2..6fa9c2a4cfe9 100644 --- a/drivers/acpi/reboot.c +++ b/drivers/acpi/reboot.c @@ -8,8 +8,8 @@ void acpi_reboot(void) { struct acpi_generic_address *rr; struct pci_bus *bus0; - u8 reset_value; unsigned int devfn; + u8 reset_value; if (acpi_disabled) return; @@ -40,7 +40,7 @@ void acpi_reboot(void) /* Form PCI device/function pair. */ devfn = PCI_DEVFN((rr->address >> 32) & 0xffff, (rr->address >> 16) & 0xffff); - printk(KERN_DEBUG "Resetting with ACPI PCI RESET_REG."); + printk(KERN_DEBUG "Resetting with ACPI PCI RESET_REG.\n"); /* Write the value that resets us. */ pci_bus_write_config_byte(bus0, devfn, (rr->address & 0xffff), reset_value); diff --git a/drivers/acpi/sleep.c b/drivers/acpi/sleep.c index 974e58457697..5d0486f1cfcd 100644 --- a/drivers/acpi/sleep.c +++ b/drivers/acpi/sleep.c @@ -989,6 +989,13 @@ static void acpi_s2idle_wake(void) !irqd_is_wakeup_armed(irq_get_irq_data(acpi_sci_irq))) { pm_system_cancel_wakeup(); s2idle_wakeup = true; + /* + * On some platforms with the LPS0 _DSM device noirq resume + * takes too much time for EC wakeup events to survive, so look + * for them now. + */ + if (lps0_device_handle) + acpi_ec_dispatch_gpe(); } } diff --git a/drivers/acpi/sysfs.c b/drivers/acpi/sysfs.c index 4fc59c3bc673..41324f0b1bee 100644 --- a/drivers/acpi/sysfs.c +++ b/drivers/acpi/sysfs.c @@ -857,12 +857,12 @@ void acpi_irq_stats_init(void) num_gpes = acpi_current_gpe_count; num_counters = num_gpes + ACPI_NUM_FIXED_EVENTS + NUM_COUNTERS_EXTRA; - all_attrs = kzalloc(sizeof(struct attribute *) * (num_counters + 1), + all_attrs = kcalloc(num_counters + 1, sizeof(struct attribute *), GFP_KERNEL); if (all_attrs == NULL) return; - all_counters = kzalloc(sizeof(struct event_counter) * (num_counters), + all_counters = kcalloc(num_counters, sizeof(struct event_counter), GFP_KERNEL); if (all_counters == NULL) goto fail; @@ -871,7 +871,7 @@ void acpi_irq_stats_init(void) if (ACPI_FAILURE(status)) goto fail; - counter_attrs = kzalloc(sizeof(struct kobj_attribute) * (num_counters), + counter_attrs = kcalloc(num_counters, sizeof(struct kobj_attribute), GFP_KERNEL); if (counter_attrs == NULL) goto fail; diff --git a/drivers/acpi/tables.c b/drivers/acpi/tables.c index 849c4fb19b03..a3d012b08fc5 100644 --- a/drivers/acpi/tables.c +++ b/drivers/acpi/tables.c @@ -222,7 +222,7 @@ void acpi_table_print_madt_entry(struct acpi_subtable_header *header) * acpi_parse_entries_array - for each proc_num find a suitable subtable * * @id: table id (for debugging purposes) - * @table_size: single entry size + * @table_size: size of the root table * @table_header: where does the table start? * @proc: array of acpi_subtable_proc struct containing entry id * and associated handler with it @@ -233,6 +233,11 @@ void acpi_table_print_madt_entry(struct acpi_subtable_header *header) * on it. Assumption is that there's only single handler for particular * entry id. * + * The table_size is not the size of the complete ACPI table (the length + * field in the header struct), but only the size of the root table; i.e., + * the offset from the very first byte of the complete ACPI table, to the + * first byte of the very first subtable. + * * On success returns sum of all matching entries for all proc handlers. * Otherwise, -ENODEV or -EINVAL is returned. */ @@ -400,7 +405,7 @@ int __init acpi_table_parse(char *id, acpi_tbl_table_handler handler) return -ENODEV; } -/* +/* * The BIOS is supposed to supply a single APIC/MADT, * but some report two. Provide a knob to use either. * (don't you wish instance 0 and 1 were not the same?) @@ -457,7 +462,7 @@ static const char * const table_sigs[] = { ACPI_SIG_UEFI, ACPI_SIG_WAET, ACPI_SIG_WDAT, ACPI_SIG_WDDT, ACPI_SIG_WDRT, ACPI_SIG_DSDT, ACPI_SIG_FADT, ACPI_SIG_PSDT, ACPI_SIG_RSDT, ACPI_SIG_XSDT, ACPI_SIG_SSDT, ACPI_SIG_IORT, - ACPI_SIG_NFIT, ACPI_SIG_HMAT, NULL }; + ACPI_SIG_NFIT, ACPI_SIG_HMAT, ACPI_SIG_PPTT, NULL }; #define ACPI_HEADER_SIZE sizeof(struct acpi_table_header) diff --git a/drivers/amba/bus.c b/drivers/amba/bus.c index 4a3ac31c07d0..41b706403ef7 100644 --- a/drivers/amba/bus.c +++ b/drivers/amba/bus.c @@ -20,6 +20,7 @@ #include #include #include +#include #include @@ -101,8 +102,8 @@ static ssize_t driver_override_store(struct device *_dev, if (strlen(driver_override)) { dev->driver_override = driver_override; } else { - kfree(driver_override); - dev->driver_override = NULL; + kfree(driver_override); + dev->driver_override = NULL; } device_unlock(_dev); @@ -193,15 +194,18 @@ static const struct dev_pm_ops amba_pm = { /* * Primecells are part of the Advanced Microcontroller Bus Architecture, * so we call the bus "amba". + * DMA configuration for platform and AMBA bus is same. So here we reuse + * platform's DMA config routine. */ struct bus_type amba_bustype = { .name = "amba", .dev_groups = amba_dev_groups, .match = amba_match, .uevent = amba_uevent, + .dma_configure = platform_dma_configure, .pm = &amba_pm, - .force_dma = true, }; +EXPORT_SYMBOL_GPL(amba_bustype); static int __init amba_init(void) { @@ -248,7 +252,7 @@ static int amba_probe(struct device *dev) break; ret = dev_pm_domain_attach(dev, true); - if (ret == -EPROBE_DEFER) + if (ret) break; ret = amba_get_enable_pclk(pcdev); @@ -375,7 +379,7 @@ static int amba_device_try_add(struct amba_device *dev, struct resource *parent) } ret = dev_pm_domain_attach(&dev->dev, true); - if (ret == -EPROBE_DEFER) { + if (ret) { iounmap(tmp); goto err_release; } diff --git a/drivers/android/Kconfig b/drivers/android/Kconfig index 7dce3795b887..ee4880bfdcdc 100644 --- a/drivers/android/Kconfig +++ b/drivers/android/Kconfig @@ -10,7 +10,7 @@ if ANDROID config ANDROID_BINDER_IPC bool "Android Binder IPC Driver" - depends on MMU + depends on MMU && !M68K default n ---help--- Binder is used in Android for both communication between processes, @@ -32,19 +32,6 @@ config ANDROID_BINDER_DEVICES created. Each binder device has its own context manager, and is therefore logically separated from the other devices. -config ANDROID_BINDER_IPC_32BIT - bool "Use old (Android 4.4 and earlier) 32-bit binder API" - depends on !64BIT && ANDROID_BINDER_IPC - default y - ---help--- - The Binder API has been changed to support both 32 and 64bit - applications in a mixed environment. - - Enable this to support an old 32-bit Android user-space (v4.4 and - earlier). - - Note that enabling this will break newer Android user-space. - config ANDROID_BINDER_IPC_SELFTEST bool "Android Binder IPC Driver Selftest" depends on ANDROID_BINDER_IPC diff --git a/drivers/android/binder.c b/drivers/android/binder.c index e578eee31589..95283f3bb51c 100644 --- a/drivers/android/binder.c +++ b/drivers/android/binder.c @@ -72,10 +72,6 @@ #include #include -#ifdef CONFIG_ANDROID_BINDER_IPC_32BIT -#define BINDER_IPC_32BIT 1 -#endif - #include #include "binder_alloc.h" #include "binder_trace.h" @@ -2058,8 +2054,8 @@ static size_t binder_validate_object(struct binder_buffer *buffer, u64 offset) struct binder_object_header *hdr; size_t object_size = 0; - if (offset > buffer->data_size - sizeof(*hdr) || - buffer->data_size < sizeof(*hdr) || + if (buffer->data_size < sizeof(*hdr) || + offset > buffer->data_size - sizeof(*hdr) || !IS_ALIGNED(offset, sizeof(u32))) return 0; @@ -3925,10 +3921,11 @@ retry: binder_inner_proc_unlock(proc); if (put_user(e->cmd, (uint32_t __user *)ptr)) return -EFAULT; + cmd = e->cmd; e->cmd = BR_OK; ptr += sizeof(uint32_t); - binder_stat_br(proc, thread, e->cmd); + binder_stat_br(proc, thread, cmd); } break; case BINDER_WORK_TRANSACTION_COMPLETE: { binder_inner_proc_unlock(proc); @@ -4696,7 +4693,7 @@ static void binder_vma_close(struct vm_area_struct *vma) binder_defer_work(proc, BINDER_DEFERRED_PUT_FILES); } -static int binder_vm_fault(struct vm_fault *vmf) +static vm_fault_t binder_vm_fault(struct vm_fault *vmf) { return VM_FAULT_SIGBUS; } @@ -4730,7 +4727,9 @@ static int binder_mmap(struct file *filp, struct vm_area_struct *vma) failure_string = "bad vm_flags"; goto err_bad_arg; } - vma->vm_flags = (vma->vm_flags | VM_DONTCOPY) & ~VM_MAYWRITE; + vma->vm_flags |= VM_DONTCOPY | VM_MIXEDMAP; + vma->vm_flags &= ~VM_MAYWRITE; + vma->vm_ops = &binder_vm_ops; vma->vm_private_data = proc; diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 5a426c877dfb..2628806c64a2 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -219,7 +219,7 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate, mm = alloc->vma_vm_mm; if (mm) { - down_write(&mm->mmap_sem); + down_read(&mm->mmap_sem); vma = alloc->vma; } @@ -288,7 +288,7 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate, /* vm_insert_page does not seem to increment the refcount */ } if (mm) { - up_write(&mm->mmap_sem); + up_read(&mm->mmap_sem); mmput(mm); } return 0; @@ -321,7 +321,7 @@ err_page_ptr_cleared: } err_no_vma: if (mm) { - up_write(&mm->mmap_sem); + up_read(&mm->mmap_sem); mmput(mm); } return vma ? -ENOMEM : -ESRCH; @@ -692,8 +692,8 @@ int binder_alloc_mmap_handler(struct binder_alloc *alloc, } } #endif - alloc->pages = kzalloc(sizeof(alloc->pages[0]) * - ((vma->vm_end - vma->vm_start) / PAGE_SIZE), + alloc->pages = kcalloc((vma->vm_end - vma->vm_start) / PAGE_SIZE, + sizeof(alloc->pages[0]), GFP_KERNEL); if (alloc->pages == NULL) { ret = -ENOMEM; diff --git a/drivers/ata/acard-ahci.c b/drivers/ata/acard-ahci.c index 940ddbc59aa7..583e366be7e2 100644 --- a/drivers/ata/acard-ahci.c +++ b/drivers/ata/acard-ahci.c @@ -271,7 +271,7 @@ static void acard_ahci_qc_prep(struct ata_queued_cmd *qc) * Fill in command table information. First, the header, * a SATA Register - Host to Device command FIS. */ - cmd_tbl = pp->cmd_tbl + qc->tag * AHCI_CMD_TBL_SZ; + cmd_tbl = pp->cmd_tbl + qc->hw_tag * AHCI_CMD_TBL_SZ; ata_tf_to_fis(&qc->tf, qc->dev->link->pmp, 1, cmd_tbl); if (is_atapi) { @@ -294,7 +294,7 @@ static void acard_ahci_qc_prep(struct ata_queued_cmd *qc) if (is_atapi) opts |= AHCI_CMD_ATAPI | AHCI_CMD_PREFETCH; - ahci_fill_cmd_slot(pp, qc->tag, opts); + ahci_fill_cmd_slot(pp, qc->hw_tag, opts); } static bool acard_ahci_qc_fill_rtf(struct ata_queued_cmd *qc) diff --git a/drivers/ata/ahci.h b/drivers/ata/ahci.h index 824bd399f02e..1609ebab4e23 100644 --- a/drivers/ata/ahci.h +++ b/drivers/ata/ahci.h @@ -390,7 +390,7 @@ extern struct device_attribute *ahci_sdev_attrs[]; */ #define AHCI_SHT(drv_name) \ ATA_NCQ_SHT(drv_name), \ - .can_queue = AHCI_MAX_CMDS - 1, \ + .can_queue = AHCI_MAX_CMDS, \ .sg_tablesize = AHCI_MAX_SG, \ .dma_boundary = AHCI_DMA_BOUNDARY, \ .shost_attrs = ahci_shost_attrs, \ diff --git a/drivers/ata/libahci.c b/drivers/ata/libahci.c index e5d90977caec..965842a08743 100644 --- a/drivers/ata/libahci.c +++ b/drivers/ata/libahci.c @@ -1649,7 +1649,7 @@ static void ahci_qc_prep(struct ata_queued_cmd *qc) * Fill in command table information. First, the header, * a SATA Register - Host to Device command FIS. */ - cmd_tbl = pp->cmd_tbl + qc->tag * AHCI_CMD_TBL_SZ; + cmd_tbl = pp->cmd_tbl + qc->hw_tag * AHCI_CMD_TBL_SZ; ata_tf_to_fis(&qc->tf, qc->dev->link->pmp, 1, cmd_tbl); if (is_atapi) { @@ -1670,7 +1670,7 @@ static void ahci_qc_prep(struct ata_queued_cmd *qc) if (is_atapi) opts |= AHCI_CMD_ATAPI | AHCI_CMD_PREFETCH; - ahci_fill_cmd_slot(pp, qc->tag, opts); + ahci_fill_cmd_slot(pp, qc->hw_tag, opts); } static void ahci_fbs_dec_intr(struct ata_port *ap) @@ -2006,7 +2006,7 @@ unsigned int ahci_qc_issue(struct ata_queued_cmd *qc) pp->active_link = qc->dev->link; if (ata_is_ncq(qc->tf.protocol)) - writel(1 << qc->tag, port_mmio + PORT_SCR_ACT); + writel(1 << qc->hw_tag, port_mmio + PORT_SCR_ACT); if (pp->fbs_enabled && pp->fbs_last_dev != qc->dev->link->pmp) { u32 fbs = readl(port_mmio + PORT_FBS); @@ -2016,7 +2016,7 @@ unsigned int ahci_qc_issue(struct ata_queued_cmd *qc) pp->fbs_last_dev = qc->dev->link->pmp; } - writel(1 << qc->tag, port_mmio + PORT_CMD_ISSUE); + writel(1 << qc->hw_tag, port_mmio + PORT_CMD_ISSUE); ahci_sw_activity(qc->dev->link); diff --git a/drivers/ata/libata-core.c b/drivers/ata/libata-core.c index 346b163f6e89..27d15ed7fa3d 100644 --- a/drivers/ata/libata-core.c +++ b/drivers/ata/libata-core.c @@ -759,7 +759,7 @@ int ata_build_rw_tf(struct ata_taskfile *tf, struct ata_device *dev, tf->flags |= ATA_TFLAG_ISADDR | ATA_TFLAG_DEVICE; tf->flags |= tf_flags; - if (ata_ncq_enabled(dev) && likely(tag != ATA_TAG_INTERNAL)) { + if (ata_ncq_enabled(dev) && !ata_tag_internal(tag)) { /* yay, NCQ */ if (!lba_48_ok(block, n_block)) return -ERANGE; @@ -1570,8 +1570,9 @@ unsigned ata_exec_internal_sg(struct ata_device *dev, u8 command = tf->command; int auto_timeout = 0; struct ata_queued_cmd *qc; - unsigned int tag, preempted_tag; - u32 preempted_sactive, preempted_qc_active; + unsigned int preempted_tag; + u32 preempted_sactive; + u64 preempted_qc_active; int preempted_nr_active_links; DECLARE_COMPLETION_ONSTACK(wait); unsigned long flags; @@ -1587,20 +1588,10 @@ unsigned ata_exec_internal_sg(struct ata_device *dev, } /* initialize internal qc */ + qc = __ata_qc_from_tag(ap, ATA_TAG_INTERNAL); - /* XXX: Tag 0 is used for drivers with legacy EH as some - * drivers choke if any other tag is given. This breaks - * ata_tag_internal() test for those drivers. Don't use new - * EH stuff without converting to it. - */ - if (ap->ops->error_handler) - tag = ATA_TAG_INTERNAL; - else - tag = 0; - - qc = __ata_qc_from_tag(ap, tag); - - qc->tag = tag; + qc->tag = ATA_TAG_INTERNAL; + qc->hw_tag = 0; qc->scsicmd = NULL; qc->ap = ap; qc->dev = dev; @@ -2295,7 +2286,7 @@ static int ata_dev_config_ncq(struct ata_device *dev, return 0; } if (ap->flags & ATA_FLAG_NCQ) { - hdepth = min(ap->scsi_host->can_queue, ATA_MAX_QUEUE - 1); + hdepth = min(ap->scsi_host->can_queue, ATA_MAX_QUEUE); dev->flags |= ATA_DFLAG_NCQ; } @@ -3573,9 +3564,11 @@ static int ata_dev_set_mode(struct ata_device *dev) DPRINTK("xfer_shift=%u, xfer_mode=0x%x\n", dev->xfer_shift, (int)dev->xfer_mode); - ata_dev_info(dev, "configured for %s%s\n", - ata_mode_string(ata_xfer_mode2mask(dev->xfer_mode)), - dev_err_whine); + if (!(ehc->i.flags & ATA_EHI_QUIET) || + ehc->i.flags & ATA_EHI_DID_HARDRESET) + ata_dev_info(dev, "configured for %s%s\n", + ata_mode_string(ata_xfer_mode2mask(dev->xfer_mode)), + dev_err_whine); return 0; @@ -4557,9 +4550,6 @@ static const struct ata_blacklist_entry ata_device_blacklist [] = { { "SAMSUNG MZMPC128HBFU-000MV", "CXM14M1Q", ATA_HORKAGE_NOLPM, }, { "SAMSUNG SSD PM830 mSATA *", "CXM13D1Q", ATA_HORKAGE_NOLPM, }, - /* Sandisk devices which are known to not handle LPM well */ - { "SanDisk SD7UB3Q*G1001", NULL, ATA_HORKAGE_NOLPM, }, - /* devices that don't properly handle queued TRIM commands */ { "Micron_M500IT_*", "MU01", ATA_HORKAGE_NO_NCQ_TRIM | ATA_HORKAGE_ZERO_AFTER_TRIM, }, @@ -5136,7 +5126,7 @@ struct ata_queued_cmd *ata_qc_new_init(struct ata_device *dev, int tag) } qc = __ata_qc_from_tag(ap, tag); - qc->tag = tag; + qc->tag = qc->hw_tag = tag; qc->scsicmd = NULL; qc->ap = ap; qc->dev = dev; @@ -5166,7 +5156,7 @@ void ata_qc_free(struct ata_queued_cmd *qc) qc->flags = 0; tag = qc->tag; - if (likely(ata_tag_valid(tag))) { + if (ata_tag_valid(tag)) { qc->tag = ATA_TAG_POISON; if (ap->flags & ATA_FLAG_SAS_HOST) ata_sas_free_tag(tag, ap); @@ -5188,7 +5178,7 @@ void __ata_qc_complete(struct ata_queued_cmd *qc) /* command should be marked inactive atomically with qc completion */ if (ata_is_ncq(qc->tf.protocol)) { - link->sactive &= ~(1 << qc->tag); + link->sactive &= ~(1 << qc->hw_tag); if (!link->sactive) ap->nr_active_links--; } else { @@ -5206,7 +5196,7 @@ void __ata_qc_complete(struct ata_queued_cmd *qc) * is called. (when rc != 0 and atapi request sense is needed) */ qc->flags &= ~ATA_QCFLAG_ACTIVE; - ap->qc_active &= ~(1 << qc->tag); + ap->qc_active &= ~(1ULL << qc->tag); /* call completion callback */ qc->complete_fn(qc); @@ -5363,29 +5353,29 @@ void ata_qc_complete(struct ata_queued_cmd *qc) * RETURNS: * Number of completed commands on success, -errno otherwise. */ -int ata_qc_complete_multiple(struct ata_port *ap, u32 qc_active) +int ata_qc_complete_multiple(struct ata_port *ap, u64 qc_active) { int nr_done = 0; - u32 done_mask; + u64 done_mask; done_mask = ap->qc_active ^ qc_active; if (unlikely(done_mask & qc_active)) { - ata_port_err(ap, "illegal qc_active transition (%08x->%08x)\n", + ata_port_err(ap, "illegal qc_active transition (%08llx->%08llx)\n", ap->qc_active, qc_active); return -EINVAL; } while (done_mask) { struct ata_queued_cmd *qc; - unsigned int tag = __ffs(done_mask); + unsigned int tag = __ffs64(done_mask); qc = ata_qc_from_tag(ap, tag); if (qc) { ata_qc_complete(qc); nr_done++; } - done_mask &= ~(1 << tag); + done_mask &= ~(1ULL << tag); } return nr_done; @@ -5416,11 +5406,11 @@ void ata_qc_issue(struct ata_queued_cmd *qc) WARN_ON_ONCE(ap->ops->error_handler && ata_tag_valid(link->active_tag)); if (ata_is_ncq(prot)) { - WARN_ON_ONCE(link->sactive & (1 << qc->tag)); + WARN_ON_ONCE(link->sactive & (1 << qc->hw_tag)); if (!link->sactive) ap->nr_active_links++; - link->sactive |= 1 << qc->tag; + link->sactive |= 1 << qc->hw_tag; } else { WARN_ON_ONCE(link->sactive); @@ -5429,7 +5419,7 @@ void ata_qc_issue(struct ata_queued_cmd *qc) } qc->flags |= ATA_QCFLAG_ACTIVE; - ap->qc_active |= 1 << qc->tag; + ap->qc_active |= 1ULL << qc->tag; /* * We guarantee to LLDs that they will have at least one @@ -6428,7 +6418,7 @@ void ata_host_init(struct ata_host *host, struct device *dev, { spin_lock_init(&host->lock); mutex_init(&host->eh_mutex); - host->n_tags = ATA_MAX_QUEUE - 1; + host->n_tags = ATA_MAX_QUEUE; host->dev = dev; host->ops = ops; } @@ -6510,7 +6500,7 @@ int ata_host_register(struct ata_host *host, struct scsi_host_template *sht) { int i, rc; - host->n_tags = clamp(sht->can_queue, 1, ATA_MAX_QUEUE - 1); + host->n_tags = clamp(sht->can_queue, 1, ATA_MAX_QUEUE); /* host must have been started */ if (!(host->flags & ATA_HOST_STARTED)) { @@ -6997,7 +6987,7 @@ static void __init ata_parse_force_param(void) if (*p == ',') size++; - ata_force_tbl = kzalloc(sizeof(ata_force_tbl[0]) * size, GFP_KERNEL); + ata_force_tbl = kcalloc(size, sizeof(ata_force_tbl[0]), GFP_KERNEL); if (!ata_force_tbl) { printk(KERN_WARNING "ata: failed to extend force table, " "libata.force ignored\n"); diff --git a/drivers/ata/libata-eh.c b/drivers/ata/libata-eh.c index 513b260bcff1..d5412145d76d 100644 --- a/drivers/ata/libata-eh.c +++ b/drivers/ata/libata-eh.c @@ -500,57 +500,6 @@ void ata_eh_release(struct ata_port *ap) mutex_unlock(&ap->host->eh_mutex); } -/** - * ata_scsi_timed_out - SCSI layer time out callback - * @cmd: timed out SCSI command - * - * Handles SCSI layer timeout. We race with normal completion of - * the qc for @cmd. If the qc is already gone, we lose and let - * the scsi command finish (EH_HANDLED). Otherwise, the qc has - * timed out and EH should be invoked. Prevent ata_qc_complete() - * from finishing it by setting EH_SCHEDULED and return - * EH_NOT_HANDLED. - * - * TODO: kill this function once old EH is gone. - * - * LOCKING: - * Called from timer context - * - * RETURNS: - * EH_HANDLED or EH_NOT_HANDLED - */ -enum blk_eh_timer_return ata_scsi_timed_out(struct scsi_cmnd *cmd) -{ - struct Scsi_Host *host = cmd->device->host; - struct ata_port *ap = ata_shost_to_port(host); - unsigned long flags; - struct ata_queued_cmd *qc; - enum blk_eh_timer_return ret; - - DPRINTK("ENTER\n"); - - if (ap->ops->error_handler) { - ret = BLK_EH_NOT_HANDLED; - goto out; - } - - ret = BLK_EH_HANDLED; - spin_lock_irqsave(ap->lock, flags); - qc = ata_qc_from_tag(ap, ap->link.active_tag); - if (qc) { - WARN_ON(qc->scsicmd != cmd); - qc->flags |= ATA_QCFLAG_EH_SCHEDULED; - qc->err_mask |= AC_ERR_TIMEOUT; - ret = BLK_EH_NOT_HANDLED; - } - spin_unlock_irqrestore(ap->lock, flags); - - out: - DPRINTK("EXIT, ret=%d\n", ret); - return ret; -} -EXPORT_SYMBOL(ata_scsi_timed_out); - static void ata_eh_unload(struct ata_port *ap) { struct ata_link *link; @@ -873,9 +822,12 @@ static int ata_eh_nr_in_flight(struct ata_port *ap) int nr = 0; /* count only non-internal commands */ - for (tag = 0; tag < ATA_MAX_QUEUE - 1; tag++) + for (tag = 0; tag < ATA_MAX_QUEUE; tag++) { + if (ata_tag_internal(tag)) + continue; if (ata_qc_from_tag(ap, tag)) nr++; + } return nr; } @@ -900,7 +852,7 @@ void ata_eh_fastdrain_timerfn(struct timer_list *t) /* No progress during the last interval, tag all * in-flight qcs as timed out and freeze the port. */ - for (tag = 0; tag < ATA_MAX_QUEUE - 1; tag++) { + for (tag = 0; tag < ATA_MAX_QUEUE; tag++) { struct ata_queued_cmd *qc = ata_qc_from_tag(ap, tag); if (qc) qc->err_mask |= AC_ERR_TIMEOUT; @@ -1054,7 +1006,8 @@ static int ata_do_link_abort(struct ata_port *ap, struct ata_link *link) /* we're gonna abort all commands, no need for fast drain */ ata_eh_set_pending(ap, 0); - for (tag = 0; tag < ATA_MAX_QUEUE; tag++) { + /* include internal tag in iteration */ + for (tag = 0; tag <= ATA_MAX_QUEUE; tag++) { struct ata_queued_cmd *qc = ata_qc_from_tag(ap, tag); if (qc && (!link || qc->dev->link == link)) { @@ -1483,6 +1436,10 @@ static const char *ata_err_string(unsigned int err_mask) return "invalid argument"; if (err_mask & AC_ERR_DEV) return "device error"; + if (err_mask & AC_ERR_NCQ) + return "NCQ error"; + if (err_mask & AC_ERR_NODEV_HINT) + return "Polling detection error"; return "unknown error"; } @@ -1866,10 +1823,10 @@ static unsigned int ata_eh_analyze_tf(struct ata_queued_cmd *qc, if (qc->flags & ATA_QCFLAG_SENSE_VALID) { int ret = scsi_check_sense(qc->scsicmd); /* - * SUCCESS here means that the sense code could + * SUCCESS here means that the sense code could be * evaluated and should be passed to the upper layers * for correct evaluation. - * FAILED means the sense code could not interpreted + * FAILED means the sense code could not be interpreted * and the device would need to be reset. * NEEDS_RETRY and ADD_TO_MLQUEUE means that the * command would need to be retried. @@ -2149,6 +2106,21 @@ static inline int ata_eh_worth_retry(struct ata_queued_cmd *qc) return qc->err_mask != AC_ERR_DEV; /* retry if not dev error */ } +/** + * ata_eh_quiet - check if we need to be quiet about a command error + * @qc: qc to check + * + * Look at the qc flags anbd its scsi command request flags to determine + * if we need to be quiet about the command failure. + */ +static inline bool ata_eh_quiet(struct ata_queued_cmd *qc) +{ + if (qc->scsicmd && + qc->scsicmd->request->rq_flags & RQF_QUIET) + qc->flags |= ATA_QCFLAG_QUIET; + return qc->flags & ATA_QCFLAG_QUIET; +} + /** * ata_eh_link_autopsy - analyze error and determine recovery action * @link: host link to perform autopsy on @@ -2166,7 +2138,7 @@ static void ata_eh_link_autopsy(struct ata_link *link) struct ata_eh_context *ehc = &link->eh_context; struct ata_device *dev; unsigned int all_err_mask = 0, eflags = 0; - int tag; + int tag, nr_failed = 0, nr_quiet = 0; u32 serror; int rc; @@ -2218,12 +2190,16 @@ static void ata_eh_link_autopsy(struct ata_link *link) if (qc->err_mask & ~AC_ERR_OTHER) qc->err_mask &= ~AC_ERR_OTHER; - /* SENSE_VALID trumps dev/unknown error and revalidation */ + /* + * SENSE_VALID trumps dev/unknown error and revalidation. Upper + * layers will determine whether the command is worth retrying + * based on the sense data and device class/type. Otherwise, + * determine directly if the command is worth retrying using its + * error mask and flags. + */ if (qc->flags & ATA_QCFLAG_SENSE_VALID) qc->err_mask &= ~(AC_ERR_DEV | AC_ERR_OTHER); - - /* determine whether the command is worth retrying */ - if (ata_eh_worth_retry(qc)) + else if (ata_eh_worth_retry(qc)) qc->flags |= ATA_QCFLAG_RETRY; /* accumulate error info */ @@ -2232,8 +2208,17 @@ static void ata_eh_link_autopsy(struct ata_link *link) if (qc->flags & ATA_QCFLAG_IO) eflags |= ATA_EFLAG_IS_IO; trace_ata_eh_link_autopsy_qc(qc); + + /* Count quiet errors */ + if (ata_eh_quiet(qc)) + nr_quiet++; + nr_failed++; } + /* If all failed commands requested silence, then be quiet */ + if (nr_quiet == nr_failed) + ehc->i.flags |= ATA_EHI_QUIET; + /* enforce default EH actions */ if (ap->pflags & ATA_PFLAG_FROZEN || all_err_mask & (AC_ERR_HSM | AC_ERR_TIMEOUT)) diff --git a/drivers/ata/libata-pmp.c b/drivers/ata/libata-pmp.c index 85aa76116a30..2ae1799f4992 100644 --- a/drivers/ata/libata-pmp.c +++ b/drivers/ata/libata-pmp.c @@ -340,7 +340,7 @@ static int sata_pmp_init_links (struct ata_port *ap, int nr_ports) int i, err; if (!pmp_link) { - pmp_link = kzalloc(sizeof(pmp_link[0]) * SATA_PMP_MAX_PORTS, + pmp_link = kcalloc(SATA_PMP_MAX_PORTS, sizeof(pmp_link[0]), GFP_NOIO); if (!pmp_link) return -ENOMEM; diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c index 89a9d4a2efc8..6a91d04351d9 100644 --- a/drivers/ata/libata-scsi.c +++ b/drivers/ata/libata-scsi.c @@ -872,6 +872,9 @@ static struct ata_queued_cmd *ata_scsi_qc_new(struct ata_device *dev, qc->sg = scsi_sglist(cmd); qc->n_elem = scsi_sg_count(cmd); + + if (cmd->request->rq_flags & RQF_QUIET) + qc->flags |= ATA_QCFLAG_QUIET; } else { cmd->result = (DID_OK << 16) | (QUEUE_FULL << 1); cmd->scsi_done(cmd); @@ -1316,7 +1319,7 @@ static int ata_scsi_dev_config(struct scsi_device *sdev, int depth; depth = min(sdev->host->can_queue, ata_id_queue_depth(dev->id)); - depth = min(ATA_MAX_QUEUE - 1, depth); + depth = min(ATA_MAX_QUEUE, depth); scsi_change_queue_depth(sdev, depth); } @@ -1429,7 +1432,7 @@ int __ata_change_queue_depth(struct ata_port *ap, struct scsi_device *sdev, /* limit and apply queue depth */ queue_depth = min(queue_depth, sdev->host->can_queue); queue_depth = min(queue_depth, ata_id_queue_depth(dev->id)); - queue_depth = min(queue_depth, ATA_MAX_QUEUE - 1); + queue_depth = min(queue_depth, ATA_MAX_QUEUE); if (sdev->queue_depth == queue_depth) return -EINVAL; @@ -1895,7 +1898,7 @@ static unsigned int ata_scsi_rw_xlat(struct ata_queued_cmd *qc) qc->nbytes = n_block * scmd->device->sector_size; rc = ata_build_rw_tf(&qc->tf, qc->dev, block, n_block, tf_flags, - qc->tag, class); + qc->hw_tag, class); if (likely(rc == 0)) return 0; @@ -3233,7 +3236,7 @@ static unsigned int ata_scsi_pass_thru(struct ata_queued_cmd *qc) /* For NCQ commands copy the tag value */ if (ata_is_ncq(tf->protocol)) - tf->nsect = qc->tag << 3; + tf->nsect = qc->hw_tag << 3; /* enforce correct master/slave bit */ tf->device = dev->devno ? @@ -3513,7 +3516,7 @@ static unsigned int ata_scsi_write_same_xlat(struct ata_queued_cmd *qc) tf->protocol = ATA_PROT_NCQ; tf->command = ATA_CMD_FPDMA_SEND; tf->hob_nsect = ATA_SUBCMD_FPDMA_SEND_DSM & 0x1f; - tf->nsect = qc->tag << 3; + tf->nsect = qc->hw_tag << 3; tf->hob_feature = (size / 512) >> 8; tf->feature = size / 512; @@ -3733,7 +3736,7 @@ static unsigned int ata_scsi_zbc_in_xlat(struct ata_queued_cmd *qc) tf->protocol = ATA_PROT_NCQ; tf->command = ATA_CMD_FPDMA_RECV; tf->hob_nsect = ATA_SUBCMD_FPDMA_RECV_ZAC_MGMT_IN & 0x1f; - tf->nsect = qc->tag << 3; + tf->nsect = qc->hw_tag << 3; tf->feature = sect & 0xff; tf->hob_feature = (sect >> 8) & 0xff; tf->auxiliary = ATA_SUBCMD_ZAC_MGMT_IN_REPORT_ZONES | (options << 8); @@ -3812,7 +3815,7 @@ static unsigned int ata_scsi_zbc_out_xlat(struct ata_queued_cmd *qc) tf->protocol = ATA_PROT_NCQ_NODATA; tf->command = ATA_CMD_NCQ_NON_DATA; tf->feature = ATA_SUBCMD_NCQ_NON_DATA_ZAC_MGMT_OUT; - tf->nsect = qc->tag << 3; + tf->nsect = qc->hw_tag << 3; tf->auxiliary = sa | ((u16)all << 8); } else { tf->protocol = ATA_PROT_NODATA; @@ -5051,6 +5054,18 @@ int ata_sas_port_init(struct ata_port *ap) } EXPORT_SYMBOL_GPL(ata_sas_port_init); +int ata_sas_tport_add(struct device *parent, struct ata_port *ap) +{ + return ata_tport_add(parent, ap); +} +EXPORT_SYMBOL_GPL(ata_sas_tport_add); + +void ata_sas_tport_delete(struct ata_port *ap) +{ + ata_tport_delete(ap); +} +EXPORT_SYMBOL_GPL(ata_sas_tport_delete); + /** * ata_sas_port_destroy - Destroy a SATA port allocated by ata_sas_port_alloc * @ap: SATA port to destroy @@ -5117,7 +5132,7 @@ int ata_sas_allocate_tag(struct ata_port *ap) tag = tag < max_queue ? tag : 0; /* the last tag is reserved for internal command. */ - if (tag == ATA_TAG_INTERNAL) + if (ata_tag_internal(tag)) continue; if (!test_and_set_bit(tag, &ap->sas_tag_allocated)) { diff --git a/drivers/ata/libata-zpodd.c b/drivers/ata/libata-zpodd.c index de4ddd0e8550..b3ed8f9953a8 100644 --- a/drivers/ata/libata-zpodd.c +++ b/drivers/ata/libata-zpodd.c @@ -35,7 +35,7 @@ struct zpodd { static int eject_tray(struct ata_device *dev) { struct ata_taskfile tf; - static const char cdb[] = { GPCMD_START_STOP_UNIT, + static const char cdb[ATAPI_CDB_LEN] = { GPCMD_START_STOP_UNIT, 0, 0, 0, 0x02, /* LoEj */ 0, 0, 0, 0, 0, 0, 0, diff --git a/drivers/ata/pata_hpt37x.c b/drivers/ata/pata_hpt37x.c index 3ba843f5cdc0..ef8aaeb0c575 100644 --- a/drivers/ata/pata_hpt37x.c +++ b/drivers/ata/pata_hpt37x.c @@ -224,17 +224,14 @@ static int hpt_dma_blacklisted(const struct ata_device *dev, char *modestr, const char * const list[]) { unsigned char model_num[ATA_ID_PROD_LEN + 1]; - int i = 0; + int i; ata_id_c_string(dev->id, model_num, ATA_ID_PROD, sizeof(model_num)); - while (list[i] != NULL) { - if (!strcmp(list[i], model_num)) { - pr_warn("%s is not supported for %s\n", - modestr, list[i]); - return 1; - } - i++; + i = match_string(list, -1, model_num); + if (i >= 0) { + pr_warn("%s is not supported for %s\n", modestr, list[i]); + return 1; } return 0; } diff --git a/drivers/ata/sata_dwc_460ex.c b/drivers/ata/sata_dwc_460ex.c index ce3d6674ef80..6f142aa54f5f 100644 --- a/drivers/ata/sata_dwc_460ex.c +++ b/drivers/ata/sata_dwc_460ex.c @@ -761,7 +761,7 @@ static void sata_dwc_dma_xfer_complete(struct ata_port *ap, u32 check_status) if (tag > 0) { dev_info(ap->dev, "%s tag=%u cmd=0x%02x dma dir=%s proto=%s dmacr=0x%08x\n", - __func__, qc->tag, qc->tf.command, + __func__, qc->hw_tag, qc->tf.command, get_dma_dir_descript(qc->dma_dir), get_prot_descript(qc->tf.protocol), sata_dwc_readl(&hsdev->sata_dwc_regs->dmacr)); @@ -789,7 +789,7 @@ static int sata_dwc_qc_complete(struct ata_port *ap, struct ata_queued_cmd *qc, { u8 status = 0; u32 mask = 0x0; - u8 tag = qc->tag; + u8 tag = qc->hw_tag; struct sata_dwc_device *hsdev = HSDEV_FROM_AP(ap); struct sata_dwc_device_port *hsdevp = HSDEVP_FROM_AP(ap); hsdev->sactive_queued = 0; @@ -997,7 +997,7 @@ static void sata_dwc_bmdma_setup_by_tag(struct ata_queued_cmd *qc, u8 tag) static void sata_dwc_bmdma_setup(struct ata_queued_cmd *qc) { - u8 tag = qc->tag; + u8 tag = qc->hw_tag; if (ata_is_ncq(qc->tf.protocol)) { dev_dbg(qc->ap->dev, "%s: ap->link.sactive=0x%08x tag=%d\n", @@ -1059,7 +1059,7 @@ static void sata_dwc_bmdma_start_by_tag(struct ata_queued_cmd *qc, u8 tag) static void sata_dwc_bmdma_start(struct ata_queued_cmd *qc) { - u8 tag = qc->tag; + u8 tag = qc->hw_tag; if (ata_is_ncq(qc->tf.protocol)) { dev_dbg(qc->ap->dev, "%s: ap->link.sactive=0x%08x tag=%d\n", @@ -1074,17 +1074,17 @@ static void sata_dwc_bmdma_start(struct ata_queued_cmd *qc) static unsigned int sata_dwc_qc_issue(struct ata_queued_cmd *qc) { u32 sactive; - u8 tag = qc->tag; + u8 tag = qc->hw_tag; struct ata_port *ap = qc->ap; struct sata_dwc_device_port *hsdevp = HSDEVP_FROM_AP(ap); #ifdef DEBUG_NCQ - if (qc->tag > 0 || ap->link.sactive > 1) + if (qc->hw_tag > 0 || ap->link.sactive > 1) dev_info(ap->dev, "%s ap id=%d cmd(0x%02x)=%s qc tag=%d prot=%s ap active_tag=0x%08x ap sactive=0x%08x\n", __func__, ap->print_id, qc->tf.command, ata_get_cmd_descript(qc->tf.command), - qc->tag, get_prot_descript(qc->tf.protocol), + qc->hw_tag, get_prot_descript(qc->tf.protocol), ap->link.active_tag, ap->link.sactive); #endif diff --git a/drivers/ata/sata_fsl.c b/drivers/ata/sata_fsl.c index 95bf3abda6f6..b8d9cfc60374 100644 --- a/drivers/ata/sata_fsl.c +++ b/drivers/ata/sata_fsl.c @@ -519,7 +519,7 @@ static void sata_fsl_qc_prep(struct ata_queued_cmd *qc) struct sata_fsl_port_priv *pp = ap->private_data; struct sata_fsl_host_priv *host_priv = ap->host->private_data; void __iomem *hcr_base = host_priv->hcr_base; - unsigned int tag = sata_fsl_tag(qc->tag, hcr_base); + unsigned int tag = sata_fsl_tag(qc->hw_tag, hcr_base); struct command_desc *cd; u32 desc_info = CMD_DESC_RES | CMD_DESC_SNOOP_ENABLE; u32 num_prde = 0; @@ -566,7 +566,7 @@ static unsigned int sata_fsl_qc_issue(struct ata_queued_cmd *qc) struct ata_port *ap = qc->ap; struct sata_fsl_host_priv *host_priv = ap->host->private_data; void __iomem *hcr_base = host_priv->hcr_base; - unsigned int tag = sata_fsl_tag(qc->tag, hcr_base); + unsigned int tag = sata_fsl_tag(qc->hw_tag, hcr_base); VPRINTK("xx_qc_issue called,CQ=0x%x,CA=0x%x,CE=0x%x,CC=0x%x\n", ioread32(CQ + hcr_base), @@ -595,7 +595,7 @@ static bool sata_fsl_qc_fill_rtf(struct ata_queued_cmd *qc) struct sata_fsl_port_priv *pp = qc->ap->private_data; struct sata_fsl_host_priv *host_priv = qc->ap->host->private_data; void __iomem *hcr_base = host_priv->hcr_base; - unsigned int tag = sata_fsl_tag(qc->tag, hcr_base); + unsigned int tag = sata_fsl_tag(qc->hw_tag, hcr_base); struct command_desc *cd; cd = pp->cmdentry + tag; @@ -1266,7 +1266,7 @@ static void sata_fsl_host_intr(struct ata_port *ap) } VPRINTK("Status of all queues :\n"); - VPRINTK("done_mask/CC = 0x%x, CA = 0x%x, CE=0x%x,CQ=0x%x,apqa=0x%x\n", + VPRINTK("done_mask/CC = 0x%x, CA = 0x%x, CE=0x%x,CQ=0x%x,apqa=0x%llx\n", done_mask, ioread32(hcr_base + CA), ioread32(hcr_base + CE), @@ -1293,7 +1293,7 @@ static void sata_fsl_host_intr(struct ata_port *ap) ata_qc_complete_multiple(ap, ap->qc_active ^ done_mask); return; - } else if ((ap->qc_active & (1 << ATA_TAG_INTERNAL))) { + } else if ((ap->qc_active & (1ULL << ATA_TAG_INTERNAL))) { iowrite32(1, hcr_base + CC); qc = ata_qc_from_tag(ap, ATA_TAG_INTERNAL); diff --git a/drivers/ata/sata_mv.c b/drivers/ata/sata_mv.c index 42d4589b43d4..73ba8e134ca9 100644 --- a/drivers/ata/sata_mv.c +++ b/drivers/ata/sata_mv.c @@ -1802,7 +1802,7 @@ static void mv_fill_sg(struct ata_queued_cmd *qc) struct mv_sg *mv_sg, *last_sg = NULL; unsigned int si; - mv_sg = pp->sg_tbl[qc->tag]; + mv_sg = pp->sg_tbl[qc->hw_tag]; for_each_sg(qc->sg, sg, qc->n_elem, si) { dma_addr_t addr = sg_dma_address(sg); u32 sg_len = sg_dma_len(sg); @@ -1903,9 +1903,9 @@ static void mv_bmdma_setup(struct ata_queued_cmd *qc) writel(0, port_mmio + BMDMA_CMD); /* load PRD table addr. */ - writel((pp->sg_tbl_dma[qc->tag] >> 16) >> 16, + writel((pp->sg_tbl_dma[qc->hw_tag] >> 16) >> 16, port_mmio + BMDMA_PRD_HIGH); - writelfl(pp->sg_tbl_dma[qc->tag], + writelfl(pp->sg_tbl_dma[qc->hw_tag], port_mmio + BMDMA_PRD_LOW); /* issue r/w command */ @@ -2071,17 +2071,17 @@ static void mv_qc_prep(struct ata_queued_cmd *qc) */ if (!(tf->flags & ATA_TFLAG_WRITE)) flags |= CRQB_FLAG_READ; - WARN_ON(MV_MAX_Q_DEPTH <= qc->tag); - flags |= qc->tag << CRQB_TAG_SHIFT; + WARN_ON(MV_MAX_Q_DEPTH <= qc->hw_tag); + flags |= qc->hw_tag << CRQB_TAG_SHIFT; flags |= (qc->dev->link->pmp & 0xf) << CRQB_PMP_SHIFT; /* get current queue index from software */ in_index = pp->req_idx; pp->crqb[in_index].sg_addr = - cpu_to_le32(pp->sg_tbl_dma[qc->tag] & 0xffffffff); + cpu_to_le32(pp->sg_tbl_dma[qc->hw_tag] & 0xffffffff); pp->crqb[in_index].sg_addr_hi = - cpu_to_le32((pp->sg_tbl_dma[qc->tag] >> 16) >> 16); + cpu_to_le32((pp->sg_tbl_dma[qc->hw_tag] >> 16) >> 16); pp->crqb[in_index].ctrl_flags = cpu_to_le16(flags); cw = &pp->crqb[in_index].ata_cmd[0]; @@ -2164,17 +2164,17 @@ static void mv_qc_prep_iie(struct ata_queued_cmd *qc) if (!(tf->flags & ATA_TFLAG_WRITE)) flags |= CRQB_FLAG_READ; - WARN_ON(MV_MAX_Q_DEPTH <= qc->tag); - flags |= qc->tag << CRQB_TAG_SHIFT; - flags |= qc->tag << CRQB_HOSTQ_SHIFT; + WARN_ON(MV_MAX_Q_DEPTH <= qc->hw_tag); + flags |= qc->hw_tag << CRQB_TAG_SHIFT; + flags |= qc->hw_tag << CRQB_HOSTQ_SHIFT; flags |= (qc->dev->link->pmp & 0xf) << CRQB_PMP_SHIFT; /* get current queue index from software */ in_index = pp->req_idx; crqb = (struct mv_crqb_iie *) &pp->crqb[in_index]; - crqb->addr = cpu_to_le32(pp->sg_tbl_dma[qc->tag] & 0xffffffff); - crqb->addr_hi = cpu_to_le32((pp->sg_tbl_dma[qc->tag] >> 16) >> 16); + crqb->addr = cpu_to_le32(pp->sg_tbl_dma[qc->hw_tag] & 0xffffffff); + crqb->addr_hi = cpu_to_le32((pp->sg_tbl_dma[qc->hw_tag] >> 16) >> 16); crqb->flags = cpu_to_le32(flags); crqb->ata_cmd[0] = cpu_to_le32( @@ -2539,7 +2539,7 @@ static int mv_handle_fbs_ncq_dev_err(struct ata_port *ap) failed_links = hweight16(new_map); ata_port_info(ap, - "%s: pmp_map=%04x qc_map=%04x failed_links=%d nr_active_links=%d\n", + "%s: pmp_map=%04x qc_map=%04llx failed_links=%d nr_active_links=%d\n", __func__, pp->delayed_eh_pmp_map, ap->qc_active, failed_links, ap->nr_active_links); @@ -4114,13 +4114,13 @@ static int mv_platform_probe(struct platform_device *pdev) if (!host || !hpriv) return -ENOMEM; - hpriv->port_clks = devm_kzalloc(&pdev->dev, - sizeof(struct clk *) * n_ports, + hpriv->port_clks = devm_kcalloc(&pdev->dev, + n_ports, sizeof(struct clk *), GFP_KERNEL); if (!hpriv->port_clks) return -ENOMEM; - hpriv->port_phys = devm_kzalloc(&pdev->dev, - sizeof(struct phy *) * n_ports, + hpriv->port_phys = devm_kcalloc(&pdev->dev, + n_ports, sizeof(struct phy *), GFP_KERNEL); if (!hpriv->port_phys) return -ENOMEM; diff --git a/drivers/ata/sata_nv.c b/drivers/ata/sata_nv.c index 8c683ddd0f58..10ae11aa1926 100644 --- a/drivers/ata/sata_nv.c +++ b/drivers/ata/sata_nv.c @@ -400,7 +400,7 @@ static struct scsi_host_template nv_adma_sht = { static struct scsi_host_template nv_swncq_sht = { ATA_NCQ_SHT(DRV_NAME), - .can_queue = ATA_MAX_QUEUE, + .can_queue = ATA_MAX_QUEUE - 1, .sg_tablesize = LIBATA_MAX_PRD, .dma_boundary = ATA_DMA_BOUNDARY, .slave_configure = nv_swncq_slave_config, @@ -740,32 +740,16 @@ static int nv_adma_slave_config(struct scsi_device *sdev) sdev1 = ap->host->ports[1]->link.device[0].sdev; if ((port0->flags & NV_ADMA_ATAPI_SETUP_COMPLETE) || (port1->flags & NV_ADMA_ATAPI_SETUP_COMPLETE)) { - /** We have to set the DMA mask to 32-bit if either port is in - ATAPI mode, since they are on the same PCI device which is - used for DMA mapping. If we set the mask we also need to set - the bounce limit on both ports to ensure that the block - layer doesn't feed addresses that cause DMA mapping to - choke. If either SCSI device is not allocated yet, it's OK - since that port will discover its correct setting when it - does get allocated. - Note: Setting 32-bit mask should not fail. */ - if (sdev0) - blk_queue_bounce_limit(sdev0->request_queue, - ATA_DMA_MASK); - if (sdev1) - blk_queue_bounce_limit(sdev1->request_queue, - ATA_DMA_MASK); - - dma_set_mask(&pdev->dev, ATA_DMA_MASK); + /* + * We have to set the DMA mask to 32-bit if either port is in + * ATAPI mode, since they are on the same PCI device which is + * used for DMA mapping. If either SCSI device is not allocated + * yet, it's OK since that port will discover its correct + * setting when it does get allocated. + */ + rc = dma_set_mask(&pdev->dev, ATA_DMA_MASK); } else { - /** This shouldn't fail as it was set to this value before */ - dma_set_mask(&pdev->dev, pp->adma_dma_mask); - if (sdev0) - blk_queue_bounce_limit(sdev0->request_queue, - pp->adma_dma_mask); - if (sdev1) - blk_queue_bounce_limit(sdev1->request_queue, - pp->adma_dma_mask); + rc = dma_set_mask(&pdev->dev, pp->adma_dma_mask); } blk_queue_segment_boundary(sdev->request_queue, segment_boundary); @@ -1131,12 +1115,11 @@ static int nv_adma_port_start(struct ata_port *ap) VPRINTK("ENTER\n"); - /* Ensure DMA mask is set to 32-bit before allocating legacy PRD and - pad buffers */ - rc = dma_set_mask(&pdev->dev, DMA_BIT_MASK(32)); - if (rc) - return rc; - rc = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(32)); + /* + * Ensure DMA mask is set to 32-bit before allocating legacy PRD and + * pad buffers. + */ + rc = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32)); if (rc) return rc; @@ -1156,13 +1139,16 @@ static int nv_adma_port_start(struct ata_port *ap) pp->notifier_clear_block = pp->gen_block + NV_ADMA_NOTIFIER_CLEAR + (4 * ap->port_no); - /* Now that the legacy PRD and padding buffer are allocated we can - safely raise the DMA mask to allocate the CPB/APRD table. - These are allowed to fail since we store the value that ends up - being used to set as the bounce limit in slave_config later if - needed. */ - dma_set_mask(&pdev->dev, DMA_BIT_MASK(64)); - dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64)); + /* + * Now that the legacy PRD and padding buffer are allocated we can + * try to raise the DMA mask to allocate the CPB/APRD table. + */ + rc = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)); + if (rc) { + rc = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32)); + if (rc) + return rc; + } pp->adma_dma_mask = *dev->dma_mask; mem = dmam_alloc_coherent(dev, NV_ADMA_PORT_PRIV_DMA_SZ, @@ -1356,11 +1342,11 @@ static void nv_adma_fill_sg(struct ata_queued_cmd *qc, struct nv_adma_cpb *cpb) for_each_sg(qc->sg, sg, qc->n_elem, si) { aprd = (si < 5) ? &cpb->aprd[si] : - &pp->aprd[NV_ADMA_SGTBL_LEN * qc->tag + (si-5)]; + &pp->aprd[NV_ADMA_SGTBL_LEN * qc->hw_tag + (si-5)]; nv_adma_fill_aprd(qc, sg, si, aprd); } if (si > 5) - cpb->next_aprd = cpu_to_le64(((u64)(pp->aprd_dma + NV_ADMA_SGTBL_SZ * qc->tag))); + cpb->next_aprd = cpu_to_le64(((u64)(pp->aprd_dma + NV_ADMA_SGTBL_SZ * qc->hw_tag))); else cpb->next_aprd = cpu_to_le64(0); } @@ -1385,7 +1371,7 @@ static int nv_adma_use_reg_mode(struct ata_queued_cmd *qc) static void nv_adma_qc_prep(struct ata_queued_cmd *qc) { struct nv_adma_port_priv *pp = qc->ap->private_data; - struct nv_adma_cpb *cpb = &pp->cpb[qc->tag]; + struct nv_adma_cpb *cpb = &pp->cpb[qc->hw_tag]; u8 ctl_flags = NV_CPB_CTL_CPB_VALID | NV_CPB_CTL_IEN; @@ -1403,7 +1389,7 @@ static void nv_adma_qc_prep(struct ata_queued_cmd *qc) wmb(); cpb->len = 3; - cpb->tag = qc->tag; + cpb->tag = qc->hw_tag; cpb->next_cpb_idx = 0; /* turn on NCQ flags for NCQ commands */ @@ -1466,9 +1452,9 @@ static unsigned int nv_adma_qc_issue(struct ata_queued_cmd *qc) pp->last_issue_ncq = curr_ncq; } - writew(qc->tag, mmio + NV_ADMA_APPEND); + writew(qc->hw_tag, mmio + NV_ADMA_APPEND); - DPRINTK("Issued tag %u\n", qc->tag); + DPRINTK("Issued tag %u\n", qc->hw_tag); return 0; } @@ -1730,8 +1716,8 @@ static void nv_swncq_qc_to_dq(struct ata_port *ap, struct ata_queued_cmd *qc) /* queue is full */ WARN_ON(dq->tail - dq->head == ATA_MAX_QUEUE); - dq->defer_bits |= (1 << qc->tag); - dq->tag[dq->tail++ & (ATA_MAX_QUEUE - 1)] = qc->tag; + dq->defer_bits |= (1 << qc->hw_tag); + dq->tag[dq->tail++ & (ATA_MAX_QUEUE - 1)] = qc->hw_tag; } static struct ata_queued_cmd *nv_swncq_qc_from_dq(struct ata_port *ap) @@ -1796,7 +1782,7 @@ static void nv_swncq_ncq_stop(struct ata_port *ap) u32 sactive; u32 done_mask; - ata_port_err(ap, "EH in SWNCQ mode,QC:qc_active 0x%X sactive 0x%X\n", + ata_port_err(ap, "EH in SWNCQ mode,QC:qc_active 0x%llX sactive 0x%X\n", ap->qc_active, ap->link.sactive); ata_port_err(ap, "SWNCQ:qc_active 0x%X defer_bits 0x%X last_issue_tag 0x%x\n " @@ -2010,7 +1996,7 @@ static void nv_swncq_fill_sg(struct ata_queued_cmd *qc) struct ata_bmdma_prd *prd; unsigned int si, idx; - prd = pp->prd + ATA_MAX_PRD * qc->tag; + prd = pp->prd + ATA_MAX_PRD * qc->hw_tag; idx = 0; for_each_sg(qc->sg, sg, qc->n_elem, si) { @@ -2048,16 +2034,16 @@ static unsigned int nv_swncq_issue_atacmd(struct ata_port *ap, DPRINTK("Enter\n"); - writel((1 << qc->tag), pp->sactive_block); - pp->last_issue_tag = qc->tag; - pp->dhfis_bits &= ~(1 << qc->tag); - pp->dmafis_bits &= ~(1 << qc->tag); - pp->qc_active |= (0x1 << qc->tag); + writel((1 << qc->hw_tag), pp->sactive_block); + pp->last_issue_tag = qc->hw_tag; + pp->dhfis_bits &= ~(1 << qc->hw_tag); + pp->dmafis_bits &= ~(1 << qc->hw_tag); + pp->qc_active |= (0x1 << qc->hw_tag); ap->ops->sff_tf_load(ap, &qc->tf); /* load tf registers */ ap->ops->sff_exec_command(ap, &qc->tf); - DPRINTK("Issued tag %u\n", qc->tag); + DPRINTK("Issued tag %u\n", qc->hw_tag); return 0; } @@ -2207,7 +2193,7 @@ static void nv_swncq_dmafis(struct ata_port *ap) rw = qc->tf.flags & ATA_TFLAG_WRITE; /* load PRD table addr. */ - iowrite32(pp->prd_dma + ATA_PRD_TBL_SZ * qc->tag, + iowrite32(pp->prd_dma + ATA_PRD_TBL_SZ * qc->hw_tag, ap->ioaddr.bmdma_addr + ATA_DMA_TABLE_OFS); /* specify data direction, triple-check start bit is clear */ diff --git a/drivers/ata/sata_sil24.c b/drivers/ata/sata_sil24.c index 010ca101d412..319f517137cd 100644 --- a/drivers/ata/sata_sil24.c +++ b/drivers/ata/sata_sil24.c @@ -849,7 +849,7 @@ static void sil24_qc_prep(struct ata_queued_cmd *qc) struct sil24_sge *sge; u16 ctrl = 0; - cb = &pp->cmd_block[sil24_tag(qc->tag)]; + cb = &pp->cmd_block[sil24_tag(qc->hw_tag)]; if (!ata_is_atapi(qc->tf.protocol)) { prb = &cb->ata.prb; @@ -891,7 +891,7 @@ static unsigned int sil24_qc_issue(struct ata_queued_cmd *qc) struct ata_port *ap = qc->ap; struct sil24_port_priv *pp = ap->private_data; void __iomem *port = sil24_port_base(ap); - unsigned int tag = sil24_tag(qc->tag); + unsigned int tag = sil24_tag(qc->hw_tag); dma_addr_t paddr; void __iomem *activate; @@ -911,7 +911,7 @@ static unsigned int sil24_qc_issue(struct ata_queued_cmd *qc) static bool sil24_qc_fill_rtf(struct ata_queued_cmd *qc) { - sil24_read_tf(qc->ap, qc->tag, &qc->result_tf); + sil24_read_tf(qc->ap, qc->hw_tag, &qc->result_tf); return true; } diff --git a/drivers/atm/fore200e.c b/drivers/atm/fore200e.c index 6ebc4e4820fc..99a38115b0a8 100644 --- a/drivers/atm/fore200e.c +++ b/drivers/atm/fore200e.c @@ -2094,7 +2094,8 @@ static int fore200e_alloc_rx_buf(struct fore200e *fore200e) DPRINTK(2, "rx buffers %d / %d are being allocated\n", scheme, magn); /* allocate the array of receive buffers */ - buffer = bsq->buffer = kzalloc(nbr * sizeof(struct buffer), GFP_KERNEL); + buffer = bsq->buffer = kcalloc(nbr, sizeof(struct buffer), + GFP_KERNEL); if (buffer == NULL) return -ENOMEM; diff --git a/drivers/atm/iphase.c b/drivers/atm/iphase.c index be076606d30e..ff81a576347e 100644 --- a/drivers/atm/iphase.c +++ b/drivers/atm/iphase.c @@ -1618,7 +1618,7 @@ static int rx_init(struct atm_dev *dev) skb_queue_head_init(&iadev->rx_dma_q); iadev->rx_free_desc_qhead = NULL; - iadev->rx_open = kzalloc(4 * iadev->num_vc, GFP_KERNEL); + iadev->rx_open = kcalloc(4, iadev->num_vc, GFP_KERNEL); if (!iadev->rx_open) { printk(KERN_ERR DEV_LABEL "itf %d couldn't get free page\n", dev->number); diff --git a/drivers/atm/solos-pci.c b/drivers/atm/solos-pci.c index 0df1a1c80b00..17283018269f 100644 --- a/drivers/atm/solos-pci.c +++ b/drivers/atm/solos-pci.c @@ -1291,7 +1291,8 @@ static int fpga_probe(struct pci_dev *dev, const struct pci_device_id *id) card->using_dma = 1; if (1) { /* All known FPGA versions so far */ card->dma_alignment = 3; - card->dma_bounce = kmalloc(card->nr_ports * BUF_SIZE, GFP_KERNEL); + card->dma_bounce = kmalloc_array(card->nr_ports, + BUF_SIZE, GFP_KERNEL); if (!card->dma_bounce) { dev_warn(&card->dev->dev, "Failed to allocate DMA bounce buffers\n"); err = -ENOMEM; diff --git a/drivers/atm/zatm.c b/drivers/atm/zatm.c index 9c9a22958717..a8d2eb0ceb8d 100644 --- a/drivers/atm/zatm.c +++ b/drivers/atm/zatm.c @@ -1151,8 +1151,8 @@ static void eprom_get_byte(struct zatm_dev *zatm_dev, unsigned char *byte, } -static unsigned char eprom_try_esi(struct atm_dev *dev, unsigned short cmd, - int offset, int swap) +static int eprom_try_esi(struct atm_dev *dev, unsigned short cmd, int offset, + int swap) { unsigned char buf[ZEPROM_SIZE]; struct zatm_dev *zatm_dev; diff --git a/drivers/auxdisplay/Kconfig b/drivers/auxdisplay/Kconfig index 2c2ed9cf8796..57410f9c5d44 100644 --- a/drivers/auxdisplay/Kconfig +++ b/drivers/auxdisplay/Kconfig @@ -14,9 +14,6 @@ menuconfig AUXDISPLAY If you say N, all options in this submenu will be skipped and disabled. -config CHARLCD - tristate "Character LCD core support" if COMPILE_TEST - if AUXDISPLAY config HD44780 @@ -137,8 +134,8 @@ config CFAG12864B_RATE config IMG_ASCII_LCD tristate "Imagination Technologies ASCII LCD Display" depends on HAS_IOMEM - default y if MIPS_MALTA || MIPS_SEAD3 - select SYSCON + default y if MIPS_MALTA + select MFD_SYSCON help Enable this to support the simple ASCII LCD displays found on development boards such as the MIPS Boston, MIPS Malta & MIPS SEAD3 @@ -157,8 +154,6 @@ config HT16K33 Say yes here to add support for Holtek HT16K33, RAM mapping 16*8 LED controller driver with keyscan. -endif # AUXDISPLAY - config ARM_CHARLCD bool "ARM Ltd. Character LCD Driver" depends on PLAT_VERSATILE @@ -169,7 +164,9 @@ config ARM_CHARLCD line and the Linux version on the second line, but that's still useful. -config PANEL +endif # AUXDISPLAY + +menuconfig PANEL tristate "Parallel port LCD/Keypad Panel support" depends on PARPORT select CHARLCD @@ -448,3 +445,6 @@ config PANEL_BOOT_MESSAGE printf()-formatted message is valid with newline and escape codes. endif # PANEL + +config CHARLCD + tristate "Character LCD core support" if COMPILE_TEST diff --git a/drivers/auxdisplay/arm-charlcd.c b/drivers/auxdisplay/arm-charlcd.c index b3176ee92b90..296fb30dfa00 100644 --- a/drivers/auxdisplay/arm-charlcd.c +++ b/drivers/auxdisplay/arm-charlcd.c @@ -1,10 +1,10 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Driver for the on-board character LCD found on some ARM reference boards * This is basically an Hitachi HD44780 LCD with a custom IP block to drive it * http://en.wikipedia.org/wiki/HD44780_Character_LCD * Currently it will just display the text "ARM Linux" and the linux version * - * License terms: GNU General Public License (GPL) version 2 * Author: Linus Walleij */ #include @@ -54,12 +54,14 @@ #define HD_BUSY_FLAG 0x80U /** + * struct charlcd - Private data structure * @dev: a pointer back to containing device * @phybase: the offset to the controller in physical memory * @physize: the size of the physical page * @virtbase: the offset to the controller in virtual memory * @irq: reserved interrupt number * @complete: completion structure for the last LCD command + * @init_work: delayed work structure to initialize the display on boot */ struct charlcd { struct device *dev; diff --git a/drivers/auxdisplay/cfag12864b.c b/drivers/auxdisplay/cfag12864b.c index 41ce4bd96813..7eebae7e322c 100644 --- a/drivers/auxdisplay/cfag12864b.c +++ b/drivers/auxdisplay/cfag12864b.c @@ -1,26 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Filename: cfag12864b.c * Version: 0.1.0 * Description: cfag12864b LCD driver - * License: GPLv2 * Depends: ks0108 * * Author: Copyright (C) Miguel Ojeda Sandonis * Date: 2006-10-31 - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA - * */ #include @@ -347,8 +333,8 @@ static int __init cfag12864b_init(void) goto none; } - cfag12864b_cache = kmalloc(sizeof(unsigned char) * - CFAG12864B_SIZE, GFP_KERNEL); + cfag12864b_cache = kmalloc(CFAG12864B_SIZE, + GFP_KERNEL); if (cfag12864b_cache == NULL) { printk(KERN_ERR CFAG12864B_NAME ": ERROR: " "can't alloc cache buffer (%i bytes)\n", diff --git a/drivers/auxdisplay/cfag12864bfb.c b/drivers/auxdisplay/cfag12864bfb.c index a3874034e2ce..40c8a552a478 100644 --- a/drivers/auxdisplay/cfag12864bfb.c +++ b/drivers/auxdisplay/cfag12864bfb.c @@ -1,26 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Filename: cfag12864bfb.c * Version: 0.1.0 * Description: cfag12864b LCD framebuffer driver - * License: GPLv2 * Depends: cfag12864b * * Author: Copyright (C) Miguel Ojeda Sandonis * Date: 2006-10-31 - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA - * */ #include @@ -37,7 +23,7 @@ #define CFAG12864BFB_NAME "cfag12864bfb" -static struct fb_fix_screeninfo cfag12864bfb_fix = { +static const struct fb_fix_screeninfo cfag12864bfb_fix = { .id = "cfag12864b", .type = FB_TYPE_PACKED_PIXELS, .visual = FB_VISUAL_MONO10, @@ -48,7 +34,7 @@ static struct fb_fix_screeninfo cfag12864bfb_fix = { .accel = FB_ACCEL_NONE, }; -static struct fb_var_screeninfo cfag12864bfb_var = { +static const struct fb_var_screeninfo cfag12864bfb_var = { .xres = CFAG12864B_WIDTH, .yres = CFAG12864B_HEIGHT, .xres_virtual = CFAG12864B_WIDTH, diff --git a/drivers/auxdisplay/charlcd.c b/drivers/auxdisplay/charlcd.c index 642afd88870b..8673fc2b9eb8 100644 --- a/drivers/auxdisplay/charlcd.c +++ b/drivers/auxdisplay/charlcd.c @@ -1,16 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * Character LCD driver for Linux * * Copyright (C) 2000-2008, Willy Tarreau * Copyright (C) 2016-2017 Glider bvba - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. */ #include +#include #include #include #include @@ -192,10 +189,11 @@ static void charlcd_print(struct charlcd *lcd, char c) c = lcd->char_conv[(unsigned char)c]; lcd->ops->write_data(lcd, c); priv->addr.x++; + + /* prevents the cursor from wrapping onto the next line */ + if (priv->addr.x == lcd->bwidth) + charlcd_gotoxy(lcd); } - /* prevents the cursor from wrapping onto the next line */ - if (priv->addr.x == lcd->bwidth) - charlcd_gotoxy(lcd); } static void charlcd_clear_fast(struct charlcd *lcd) @@ -292,6 +290,79 @@ static int charlcd_init_display(struct charlcd *lcd) return 0; } +/* + * Parses an unsigned integer from a string, until a non-digit character + * is found. The empty string is not accepted. No overflow checks are done. + * + * Returns whether the parsing was successful. Only in that case + * the output parameters are written to. + * + * TODO: If the kernel adds an inplace version of kstrtoul(), this function + * could be easily replaced by that. + */ +static bool parse_n(const char *s, unsigned long *res, const char **next_s) +{ + if (!isdigit(*s)) + return false; + + *res = 0; + while (isdigit(*s)) { + *res = *res * 10 + (*s - '0'); + ++s; + } + + *next_s = s; + return true; +} + +/* + * Parses a movement command of the form "(.*);", where the group can be + * any number of subcommands of the form "(x|y)[0-9]+". + * + * Returns whether the command is valid. The position arguments are + * only written if the parsing was successful. + * + * For instance: + * - ";" returns (, ). + * - "x1;" returns (1, ). + * - "y2x1;" returns (1, 2). + * - "x12y34x56;" returns (56, 34). + * - "" fails. + * - "x" fails. + * - "x;" fails. + * - "x1" fails. + * - "xy12;" fails. + * - "x12yy12;" fails. + * - "xx" fails. + */ +static bool parse_xy(const char *s, unsigned long *x, unsigned long *y) +{ + unsigned long new_x = *x; + unsigned long new_y = *y; + + for (;;) { + if (!*s) + return false; + + if (*s == ';') + break; + + if (*s == 'x') { + if (!parse_n(s + 1, &new_x, &s)) + return false; + } else if (*s == 'y') { + if (!parse_n(s + 1, &new_y, &s)) + return false; + } else { + return false; + } + } + + *x = new_x; + *y = new_y; + return true; +} + /* * These are the file operation function for user access to /dev/lcd * This function can also be called from inside the kernel, by @@ -362,6 +433,7 @@ static inline int handle_lcd_special_code(struct charlcd *lcd) break; case 'N': /* Two Lines */ priv->flags |= LCD_FLAG_N; + processed = 1; break; case 'l': /* Shift Cursor Left */ if (priv->addr.x > 0) { @@ -441,9 +513,9 @@ static inline int handle_lcd_special_code(struct charlcd *lcd) shift ^= 4; if (*esc >= '0' && *esc <= '9') { value |= (*esc - '0') << shift; - } else if (*esc >= 'A' && *esc <= 'Z') { + } else if (*esc >= 'A' && *esc <= 'F') { value |= (*esc - 'A' + 10) << shift; - } else if (*esc >= 'a' && *esc <= 'z') { + } else if (*esc >= 'a' && *esc <= 'f') { value |= (*esc - 'a' + 10) << shift; } else { esc++; @@ -469,24 +541,11 @@ static inline int handle_lcd_special_code(struct charlcd *lcd) } case 'x': /* gotoxy : LxXXX[yYYY]; */ case 'y': /* gotoxy : LyYYY[xXXX]; */ - if (!strchr(esc, ';')) - break; - - while (*esc) { - if (*esc == 'x') { - esc++; - if (kstrtoul(esc, 10, &priv->addr.x) < 0) - break; - } else if (*esc == 'y') { - esc++; - if (kstrtoul(esc, 10, &priv->addr.y) < 0) - break; - } else { - break; - } - } + /* If the command is valid, move to the new address */ + if (parse_xy(esc, &priv->addr.x, &priv->addr.y)) + charlcd_gotoxy(lcd); - charlcd_gotoxy(lcd); + /* Regardless of its validity, mark as processed */ processed = 1; break; } @@ -527,7 +586,7 @@ static void charlcd_write_char(struct charlcd *lcd, char c) if ((c != '\n') && priv->esc_seq.len >= 0) { /* yes, let's add this char to the buffer */ priv->esc_seq.buf[priv->esc_seq.len++] = c; - priv->esc_seq.buf[priv->esc_seq.len] = 0; + priv->esc_seq.buf[priv->esc_seq.len] = '\0'; } else { /* aborts any previous escape sequence */ priv->esc_seq.len = -1; @@ -536,7 +595,7 @@ static void charlcd_write_char(struct charlcd *lcd, char c) case LCD_ESCAPE_CHAR: /* start of an escape sequence */ priv->esc_seq.len = 0; - priv->esc_seq.buf[priv->esc_seq.len] = 0; + priv->esc_seq.buf[priv->esc_seq.len] = '\0'; break; case '\b': /* go back one char and clear it */ @@ -555,7 +614,7 @@ static void charlcd_write_char(struct charlcd *lcd, char c) /* back one char again */ lcd->ops->write_cmd(lcd, LCD_CMD_SHIFT); break; - case '\014': + case '\f': /* quickly clear the display */ charlcd_clear_fast(lcd); break; diff --git a/drivers/auxdisplay/hd44780.c b/drivers/auxdisplay/hd44780.c index 036eec404289..78d8f1986fec 100644 --- a/drivers/auxdisplay/hd44780.c +++ b/drivers/auxdisplay/hd44780.c @@ -1,13 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * HD44780 Character LCD driver for Linux * * Copyright (C) 2000-2008, Willy Tarreau * Copyright (C) 2016-2017 Glider bvba - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. */ #include diff --git a/drivers/auxdisplay/ht16k33.c b/drivers/auxdisplay/ht16k33.c index fbfa5b4cc567..a43276c76fc6 100644 --- a/drivers/auxdisplay/ht16k33.c +++ b/drivers/auxdisplay/ht16k33.c @@ -1,18 +1,10 @@ +// SPDX-License-Identifier: GPL-2.0 /* * HT16K33 driver * * Author: Robin van der Gracht * * Copyright: (C) 2016 Protonic Holland. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. */ #include diff --git a/drivers/auxdisplay/ks0108.c b/drivers/auxdisplay/ks0108.c index 816de9eaac26..abfe3fa9e6f4 100644 --- a/drivers/auxdisplay/ks0108.c +++ b/drivers/auxdisplay/ks0108.c @@ -1,26 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Filename: ks0108.c * Version: 0.1.0 * Description: ks0108 LCD Controller driver - * License: GPLv2 * Depends: parport * * Author: Copyright (C) Miguel Ojeda Sandonis * Date: 2006-10-31 - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA - * */ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt diff --git a/drivers/auxdisplay/panel.c b/drivers/auxdisplay/panel.c index ec5e8800f8ad..3b25a643058c 100644 --- a/drivers/auxdisplay/panel.c +++ b/drivers/auxdisplay/panel.c @@ -1,13 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * Front panel driver for Linux * Copyright (C) 2000-2008, Willy Tarreau * Copyright (C) 2016-2017 Glider bvba * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. - * * This code drives an LCD module (/dev/lcd), and a keypad (/dev/keypad) * connected to a parallel printer port. * diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig index 29b0eb452b3a..3e63a900b330 100644 --- a/drivers/base/Kconfig +++ b/drivers/base/Kconfig @@ -70,81 +70,25 @@ config STANDALONE If unsure, say Y. config PREVENT_FIRMWARE_BUILD - bool "Prevent firmware from being built" + bool "Disable drivers features which enable custom firmware building" default y help - Say yes to avoid building firmware. Firmware is usually shipped - with the driver and only when updating the firmware should a - rebuild be made. - If unsure, say Y here. - -config FW_LOADER - tristate "Userspace firmware loading support" if EXPERT - default y - ---help--- - This option is provided for the case where none of the in-tree modules - require userspace firmware loading support, but a module built - out-of-tree does. - -config EXTRA_FIRMWARE - string "External firmware blobs to build into the kernel binary" - depends on FW_LOADER - help - Various drivers in the kernel source tree may require firmware, - which is generally available in your distribution's linux-firmware - package. - - The linux-firmware package should install firmware into - /lib/firmware/ on your system, so they can be loaded by userspace - helpers on request. - - This option allows firmware to be built into the kernel for the case - where the user either cannot or doesn't want to provide it from - userspace at runtime (for example, when the firmware in question is - required for accessing the boot device, and the user doesn't want to - use an initrd). - - This option is a string and takes the (space-separated) names of the - firmware files -- the same names that appear in MODULE_FIRMWARE() - and request_firmware() in the source. These files should exist under - the directory specified by the EXTRA_FIRMWARE_DIR option, which is - /lib/firmware by default. - - For example, you might set CONFIG_EXTRA_FIRMWARE="usb8388.bin", copy - the usb8388.bin file into /lib/firmware, and build the kernel. Then - any request_firmware("usb8388.bin") will be satisfied internally - without needing to call out to userspace. - - WARNING: If you include additional firmware files into your binary - kernel image that are not available under the terms of the GPL, - then it may be a violation of the GPL to distribute the resulting - image since it combines both GPL and non-GPL work. You should - consult a lawyer of your own before distributing such an image. - -config EXTRA_FIRMWARE_DIR - string "Firmware blobs root directory" - depends on EXTRA_FIRMWARE != "" - default "/lib/firmware" - help - This option controls the directory in which the kernel build system - looks for the firmware files listed in the EXTRA_FIRMWARE option. - -config FW_LOADER_USER_HELPER - bool - -config FW_LOADER_USER_HELPER_FALLBACK - bool "Fallback user-helper invocation for firmware loading" - depends on FW_LOADER - select FW_LOADER_USER_HELPER - help - This option enables / disables the invocation of user-helper - (e.g. udev) for loading firmware files as a fallback after the - direct file loading in kernel fails. The user-mode helper is - no longer required unless you have a special firmware file that - resides in a non-standard path. Moreover, the udev support has - been deprecated upstream. - - If you are unsure about this, say N here. + Say yes to disable driver features which enable building a custom + driver firmware at kernel build time. These drivers do not use the + kernel firmware API to load firmware (CONFIG_FW_LOADER), instead they + use their own custom loading mechanism. The required firmware is + usually shipped with the driver, building the driver firmware + should only be needed if you have an updated firmware source. + + Firmware should not be being built as part of kernel, these days + you should always prevent this and say Y here. There are only two + old drivers which enable building of its firmware at kernel build + time: + + o CONFIG_WANXL through CONFIG_WANXL_BUILD_FIRMWARE + o CONFIG_SCSI_AIC79XX through CONFIG_AIC79XX_BUILD_FIRMWARE + +source "drivers/base/firmware_loader/Kconfig" config WANT_DEV_COREDUMP bool diff --git a/drivers/base/Makefile b/drivers/base/Makefile index b074f242a435..704f44295810 100644 --- a/drivers/base/Makefile +++ b/drivers/base/Makefile @@ -8,10 +8,7 @@ obj-y := component.o core.o bus.o dd.o syscore.o \ topology.o container.o property.o cacheinfo.o \ devcon.o obj-$(CONFIG_DEVTMPFS) += devtmpfs.o -obj-$(CONFIG_DMA_CMA) += dma-contiguous.o obj-y += power/ -obj-$(CONFIG_HAS_DMA) += dma-mapping.o -obj-$(CONFIG_HAVE_GENERIC_DMA_COHERENT) += dma-coherent.o obj-$(CONFIG_ISA_BUS_API) += isa.o obj-y += firmware_loader/ obj-$(CONFIG_NUMA) += node.o diff --git a/drivers/base/base.h b/drivers/base/base.h index d800de650fa5..a75c3025fb78 100644 --- a/drivers/base/base.h +++ b/drivers/base/base.h @@ -161,3 +161,6 @@ extern void device_links_driver_cleanup(struct device *dev); extern void device_links_no_driver(struct device *dev); extern bool device_links_busy(struct device *dev); extern void device_links_unbind_consumers(struct device *dev); + +/* device pm support */ +void device_pm_move_to_tail(struct device *dev); diff --git a/drivers/base/bus.c b/drivers/base/bus.c index ef6183306b40..8bfd27ec73d6 100644 --- a/drivers/base/bus.c +++ b/drivers/base/bus.c @@ -184,10 +184,10 @@ static ssize_t unbind_store(struct device_driver *drv, const char *buf, dev = bus_find_device_by_name(bus, NULL, buf); if (dev && dev->driver == drv) { - if (dev->parent) /* Needed for USB */ + if (dev->parent && dev->bus->need_parent_lock) device_lock(dev->parent); device_release_driver(dev); - if (dev->parent) + if (dev->parent && dev->bus->need_parent_lock) device_unlock(dev->parent); err = count; } @@ -211,12 +211,12 @@ static ssize_t bind_store(struct device_driver *drv, const char *buf, dev = bus_find_device_by_name(bus, NULL, buf); if (dev && dev->driver == NULL && driver_match_device(drv, dev)) { - if (dev->parent) /* Needed for USB */ + if (dev->parent && bus->need_parent_lock) device_lock(dev->parent); device_lock(dev); err = driver_probe_device(drv, dev); device_unlock(dev); - if (dev->parent) + if (dev->parent && bus->need_parent_lock) device_unlock(dev->parent); if (err > 0) { @@ -735,10 +735,10 @@ static int __must_check bus_rescan_devices_helper(struct device *dev, int ret = 0; if (!dev->driver) { - if (dev->parent) /* Needed for USB */ + if (dev->parent && dev->bus->need_parent_lock) device_lock(dev->parent); ret = device_attach(dev); - if (dev->parent) + if (dev->parent && dev->bus->need_parent_lock) device_unlock(dev->parent); } return ret < 0 ? ret : 0; @@ -770,10 +770,10 @@ EXPORT_SYMBOL_GPL(bus_rescan_devices); int device_reprobe(struct device *dev) { if (dev->driver) { - if (dev->parent) /* Needed for USB */ + if (dev->parent && dev->bus->need_parent_lock) device_lock(dev->parent); device_release_driver(dev); - if (dev->parent) + if (dev->parent && dev->bus->need_parent_lock) device_unlock(dev->parent); } return bus_rescan_devices_helper(dev, NULL); diff --git a/drivers/base/cacheinfo.c b/drivers/base/cacheinfo.c index edf726267282..2880e2ab01f5 100644 --- a/drivers/base/cacheinfo.c +++ b/drivers/base/cacheinfo.c @@ -32,50 +32,10 @@ struct cpu_cacheinfo *get_cpu_cacheinfo(unsigned int cpu) } #ifdef CONFIG_OF -static int cache_setup_of_node(unsigned int cpu) -{ - struct device_node *np; - struct cacheinfo *this_leaf; - struct device *cpu_dev = get_cpu_device(cpu); - struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); - unsigned int index = 0; - - /* skip if of_node is already populated */ - if (this_cpu_ci->info_list->of_node) - return 0; - - if (!cpu_dev) { - pr_err("No cpu device for CPU %d\n", cpu); - return -ENODEV; - } - np = cpu_dev->of_node; - if (!np) { - pr_err("Failed to find cpu%d device node\n", cpu); - return -ENOENT; - } - - while (index < cache_leaves(cpu)) { - this_leaf = this_cpu_ci->info_list + index; - if (this_leaf->level != 1) - np = of_find_next_cache_node(np); - else - np = of_node_get(np);/* cpu node itself */ - if (!np) - break; - this_leaf->of_node = np; - index++; - } - - if (index != cache_leaves(cpu)) /* not all OF nodes populated */ - return -ENOENT; - - return 0; -} - static inline bool cache_leaves_are_shared(struct cacheinfo *this_leaf, struct cacheinfo *sib_leaf) { - return sib_leaf->of_node == this_leaf->of_node; + return sib_leaf->fw_token == this_leaf->fw_token; } /* OF properties to query for a given cache type */ @@ -111,7 +71,7 @@ static inline int get_cacheinfo_idx(enum cache_type type) return type; } -static void cache_size(struct cacheinfo *this_leaf) +static void cache_size(struct cacheinfo *this_leaf, struct device_node *np) { const char *propname; const __be32 *cache_size; @@ -120,13 +80,14 @@ static void cache_size(struct cacheinfo *this_leaf) ct_idx = get_cacheinfo_idx(this_leaf->type); propname = cache_type_info[ct_idx].size_prop; - cache_size = of_get_property(this_leaf->of_node, propname, NULL); + cache_size = of_get_property(np, propname, NULL); if (cache_size) this_leaf->size = of_read_number(cache_size, 1); } /* not cache_line_size() because that's a macro in include/linux/cache.h */ -static void cache_get_line_size(struct cacheinfo *this_leaf) +static void cache_get_line_size(struct cacheinfo *this_leaf, + struct device_node *np) { const __be32 *line_size; int i, lim, ct_idx; @@ -138,7 +99,7 @@ static void cache_get_line_size(struct cacheinfo *this_leaf) const char *propname; propname = cache_type_info[ct_idx].line_size_props[i]; - line_size = of_get_property(this_leaf->of_node, propname, NULL); + line_size = of_get_property(np, propname, NULL); if (line_size) break; } @@ -147,7 +108,7 @@ static void cache_get_line_size(struct cacheinfo *this_leaf) this_leaf->coherency_line_size = of_read_number(line_size, 1); } -static void cache_nr_sets(struct cacheinfo *this_leaf) +static void cache_nr_sets(struct cacheinfo *this_leaf, struct device_node *np) { const char *propname; const __be32 *nr_sets; @@ -156,7 +117,7 @@ static void cache_nr_sets(struct cacheinfo *this_leaf) ct_idx = get_cacheinfo_idx(this_leaf->type); propname = cache_type_info[ct_idx].nr_sets_prop; - nr_sets = of_get_property(this_leaf->of_node, propname, NULL); + nr_sets = of_get_property(np, propname, NULL); if (nr_sets) this_leaf->number_of_sets = of_read_number(nr_sets, 1); } @@ -175,41 +136,77 @@ static void cache_associativity(struct cacheinfo *this_leaf) this_leaf->ways_of_associativity = (size / nr_sets) / line_size; } -static bool cache_node_is_unified(struct cacheinfo *this_leaf) +static bool cache_node_is_unified(struct cacheinfo *this_leaf, + struct device_node *np) { - return of_property_read_bool(this_leaf->of_node, "cache-unified"); + return of_property_read_bool(np, "cache-unified"); } -static void cache_of_override_properties(unsigned int cpu) +static void cache_of_set_props(struct cacheinfo *this_leaf, + struct device_node *np) { - int index; + /* + * init_cache_level must setup the cache level correctly + * overriding the architecturally specified levels, so + * if type is NONE at this stage, it should be unified + */ + if (this_leaf->type == CACHE_TYPE_NOCACHE && + cache_node_is_unified(this_leaf, np)) + this_leaf->type = CACHE_TYPE_UNIFIED; + cache_size(this_leaf, np); + cache_get_line_size(this_leaf, np); + cache_nr_sets(this_leaf, np); + cache_associativity(this_leaf); +} + +static int cache_setup_of_node(unsigned int cpu) +{ + struct device_node *np; struct cacheinfo *this_leaf; + struct device *cpu_dev = get_cpu_device(cpu); struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); + unsigned int index = 0; - for (index = 0; index < cache_leaves(cpu); index++) { + /* skip if fw_token is already populated */ + if (this_cpu_ci->info_list->fw_token) { + return 0; + } + + if (!cpu_dev) { + pr_err("No cpu device for CPU %d\n", cpu); + return -ENODEV; + } + np = cpu_dev->of_node; + if (!np) { + pr_err("Failed to find cpu%d device node\n", cpu); + return -ENOENT; + } + + while (index < cache_leaves(cpu)) { this_leaf = this_cpu_ci->info_list + index; - /* - * init_cache_level must setup the cache level correctly - * overriding the architecturally specified levels, so - * if type is NONE at this stage, it should be unified - */ - if (this_leaf->type == CACHE_TYPE_NOCACHE && - cache_node_is_unified(this_leaf)) - this_leaf->type = CACHE_TYPE_UNIFIED; - cache_size(this_leaf); - cache_get_line_size(this_leaf); - cache_nr_sets(this_leaf); - cache_associativity(this_leaf); + if (this_leaf->level != 1) + np = of_find_next_cache_node(np); + else + np = of_node_get(np);/* cpu node itself */ + if (!np) + break; + cache_of_set_props(this_leaf, np); + this_leaf->fw_token = np; + index++; } + + if (index != cache_leaves(cpu)) /* not all OF nodes populated */ + return -ENOENT; + + return 0; } #else -static void cache_of_override_properties(unsigned int cpu) { } static inline int cache_setup_of_node(unsigned int cpu) { return 0; } static inline bool cache_leaves_are_shared(struct cacheinfo *this_leaf, struct cacheinfo *sib_leaf) { /* - * For non-DT systems, assume unique level 1 cache, system-wide + * For non-DT/ACPI systems, assume unique level 1 caches, system-wide * shared caches for all other levels. This will be used only if * arch specific code has not populated shared_cpu_map */ @@ -217,6 +214,11 @@ static inline bool cache_leaves_are_shared(struct cacheinfo *this_leaf, } #endif +int __weak cache_setup_acpi(unsigned int cpu) +{ + return -ENOTSUPP; +} + static int cache_shared_cpu_map_setup(unsigned int cpu) { struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); @@ -230,8 +232,8 @@ static int cache_shared_cpu_map_setup(unsigned int cpu) if (of_have_populated_dt()) ret = cache_setup_of_node(cpu); else if (!acpi_disabled) - /* No cache property/hierarchy support yet in ACPI */ - ret = -ENOTSUPP; + ret = cache_setup_acpi(cpu); + if (ret) return ret; @@ -282,16 +284,11 @@ static void cache_shared_cpu_map_remove(unsigned int cpu) cpumask_clear_cpu(cpu, &sib_leaf->shared_cpu_map); cpumask_clear_cpu(sibling, &this_leaf->shared_cpu_map); } - of_node_put(this_leaf->of_node); + if (of_have_populated_dt()) + of_node_put(this_leaf->fw_token); } } -static void cache_override_properties(unsigned int cpu) -{ - if (of_have_populated_dt()) - return cache_of_override_properties(cpu); -} - static void free_cache_attributes(unsigned int cpu) { if (!per_cpu_cacheinfo(cpu)) @@ -325,12 +322,17 @@ static int detect_cache_attributes(unsigned int cpu) if (per_cpu_cacheinfo(cpu) == NULL) return -ENOMEM; + /* + * populate_cache_leaves() may completely setup the cache leaves and + * shared_cpu_map or it may leave it partially setup. + */ ret = populate_cache_leaves(cpu); if (ret) goto free_ci; /* - * For systems using DT for cache hierarchy, of_node and shared_cpu_map - * will be set up here only if they are not populated already + * For systems using DT for cache hierarchy, fw_token + * and shared_cpu_map will be set up here only if they are + * not populated already */ ret = cache_shared_cpu_map_setup(cpu); if (ret) { @@ -338,7 +340,6 @@ static int detect_cache_attributes(unsigned int cpu) goto free_ci; } - cache_override_properties(cpu); return 0; free_ci: diff --git a/drivers/base/core.c b/drivers/base/core.c index b610816eb887..df3e1a44707a 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -144,6 +144,26 @@ static int device_reorder_to_tail(struct device *dev, void *not_used) return 0; } +/** + * device_pm_move_to_tail - Move set of devices to the end of device lists + * @dev: Device to move + * + * This is a device_reorder_to_tail() wrapper taking the requisite locks. + * + * It moves the @dev along with all of its children and all of its consumers + * to the ends of the device_kset and dpm_list, recursively. + */ +void device_pm_move_to_tail(struct device *dev) +{ + int idx; + + idx = device_links_read_lock(); + device_pm_lock(); + device_reorder_to_tail(dev, NULL); + device_pm_unlock(); + device_links_read_unlock(idx); +} + /** * device_link_add - Create a link between two devices. * @consumer: Consumer end of the link. @@ -216,6 +236,13 @@ struct device_link *device_link_add(struct device *consumer, link->rpm_active = true; } pm_runtime_new_link(consumer); + /* + * If the link is being added by the consumer driver at probe + * time, balance the decrementation of the supplier's runtime PM + * usage counter after consumer probe in driver_probe_device(). + */ + if (consumer->links.status == DL_DEV_PROBING) + pm_runtime_get_noresume(supplier); } get_device(supplier); link->supplier = supplier; @@ -235,12 +262,12 @@ struct device_link *device_link_add(struct device *consumer, switch (consumer->links.status) { case DL_DEV_PROBING: /* - * Balance the decrementation of the supplier's - * runtime PM usage counter after consumer probe - * in driver_probe_device(). + * Some callers expect the link creation during + * consumer driver probe to resume the supplier + * even without DL_FLAG_RPM_ACTIVE. */ if (flags & DL_FLAG_PM_RUNTIME) - pm_runtime_get_sync(supplier); + pm_runtime_resume(supplier); link->status = DL_STATE_CONSUMER_PROBE; break; @@ -1467,7 +1494,7 @@ class_dir_create_and_add(struct class *class, struct kobject *parent_kobj) dir = kzalloc(sizeof(*dir), GFP_KERNEL); if (!dir) - return NULL; + return ERR_PTR(-ENOMEM); dir->class = class; kobject_init(&dir->kobj, &class_dir_ktype); @@ -1477,7 +1504,7 @@ class_dir_create_and_add(struct class *class, struct kobject *parent_kobj) retval = kobject_add(&dir->kobj, parent_kobj, "%s", class->name); if (retval < 0) { kobject_put(&dir->kobj); - return NULL; + return ERR_PTR(retval); } return &dir->kobj; } @@ -1784,6 +1811,10 @@ int device_add(struct device *dev) parent = get_device(dev->parent); kobj = get_device_parent(dev, parent); + if (IS_ERR(kobj)) { + error = PTR_ERR(kobj); + goto parent_error; + } if (kobj) dev->kobj.parent = kobj; @@ -1882,6 +1913,7 @@ done: kobject_del(&dev->kobj); Error: cleanup_glue_dir(dev, glue_dir); +parent_error: put_device(parent); name_error: kfree(dev->p); @@ -2406,7 +2438,7 @@ static void device_create_release(struct device *dev) kfree(dev); } -static struct device * +static __printf(6, 0) struct device * device_create_groups_vargs(struct class *class, struct device *parent, dev_t devt, void *drvdata, const struct attribute_group **groups, @@ -2684,7 +2716,7 @@ static int device_move_class_links(struct device *dev, /** * device_move - moves a device to a new parent * @dev: the pointer to the struct device to be moved - * @new_parent: the new parent of the device (can by NULL) + * @new_parent: the new parent of the device (can be NULL) * @dpm_order: how to reorder the dpm_list */ int device_move(struct device *dev, struct device *new_parent, @@ -2701,6 +2733,11 @@ int device_move(struct device *dev, struct device *new_parent, device_pm_lock(); new_parent = get_device(new_parent); new_parent_kobj = get_device_parent(dev, new_parent); + if (IS_ERR(new_parent_kobj)) { + error = PTR_ERR(new_parent_kobj); + put_device(new_parent); + goto out; + } pr_debug("device: '%s': %s: moving to '%s'\n", dev_name(dev), __func__, new_parent ? dev_name(new_parent) : ""); diff --git a/drivers/base/dd.c b/drivers/base/dd.c index c9f54089429b..1435d7281c66 100644 --- a/drivers/base/dd.c +++ b/drivers/base/dd.c @@ -122,9 +122,7 @@ static void deferred_probe_work_func(struct work_struct *work) * the list is a good order for suspend but deferred * probe makes that very unsafe. */ - device_pm_lock(); - device_pm_move_last(dev); - device_pm_unlock(); + device_pm_move_to_tail(dev); dev_dbg(dev, "Retrying from deferred list\n"); if (initcall_debug && !initcalls_done) @@ -817,13 +815,13 @@ static int __driver_attach(struct device *dev, void *data) return ret; } /* ret > 0 means positive match */ - if (dev->parent) /* Needed for USB */ + if (dev->parent && dev->bus->need_parent_lock) device_lock(dev->parent); device_lock(dev); if (!dev->driver) driver_probe_device(drv, dev); device_unlock(dev); - if (dev->parent) + if (dev->parent && dev->bus->need_parent_lock) device_unlock(dev->parent); return 0; @@ -919,7 +917,7 @@ void device_release_driver_internal(struct device *dev, struct device_driver *drv, struct device *parent) { - if (parent) + if (parent && dev->bus->need_parent_lock) device_lock(parent); device_lock(dev); @@ -927,7 +925,7 @@ void device_release_driver_internal(struct device *dev, __device_release_driver(dev, parent); device_unlock(dev); - if (parent) + if (parent && dev->bus->need_parent_lock) device_unlock(parent); } diff --git a/drivers/base/devres.c b/drivers/base/devres.c index 95b67281cd2a..f98a097e73f2 100644 --- a/drivers/base/devres.c +++ b/drivers/base/devres.c @@ -84,9 +84,14 @@ static struct devres_group * node_to_group(struct devres_node *node) static __always_inline struct devres * alloc_dr(dr_release_t release, size_t size, gfp_t gfp, int nid) { - size_t tot_size = sizeof(struct devres) + size; + size_t tot_size; struct devres *dr; + /* We must catch any near-SIZE_MAX cases that could overflow. */ + if (unlikely(check_add_overflow(sizeof(struct devres), size, + &tot_size))) + return NULL; + dr = kmalloc_node_track_caller(tot_size, gfp, nid); if (unlikely(!dr)) return NULL; diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c deleted file mode 100644 index d82566d6e237..000000000000 --- a/drivers/base/dma-mapping.c +++ /dev/null @@ -1,368 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * drivers/base/dma-mapping.c - arch-independent dma-mapping routines - * - * Copyright (c) 2006 SUSE Linux Products GmbH - * Copyright (c) 2006 Tejun Heo - */ - -#include -#include -#include -#include -#include -#include -#include - -/* - * Managed DMA API - */ -struct dma_devres { - size_t size; - void *vaddr; - dma_addr_t dma_handle; - unsigned long attrs; -}; - -static void dmam_release(struct device *dev, void *res) -{ - struct dma_devres *this = res; - - dma_free_attrs(dev, this->size, this->vaddr, this->dma_handle, - this->attrs); -} - -static int dmam_match(struct device *dev, void *res, void *match_data) -{ - struct dma_devres *this = res, *match = match_data; - - if (this->vaddr == match->vaddr) { - WARN_ON(this->size != match->size || - this->dma_handle != match->dma_handle); - return 1; - } - return 0; -} - -/** - * dmam_alloc_coherent - Managed dma_alloc_coherent() - * @dev: Device to allocate coherent memory for - * @size: Size of allocation - * @dma_handle: Out argument for allocated DMA handle - * @gfp: Allocation flags - * - * Managed dma_alloc_coherent(). Memory allocated using this function - * will be automatically released on driver detach. - * - * RETURNS: - * Pointer to allocated memory on success, NULL on failure. - */ -void *dmam_alloc_coherent(struct device *dev, size_t size, - dma_addr_t *dma_handle, gfp_t gfp) -{ - struct dma_devres *dr; - void *vaddr; - - dr = devres_alloc(dmam_release, sizeof(*dr), gfp); - if (!dr) - return NULL; - - vaddr = dma_alloc_coherent(dev, size, dma_handle, gfp); - if (!vaddr) { - devres_free(dr); - return NULL; - } - - dr->vaddr = vaddr; - dr->dma_handle = *dma_handle; - dr->size = size; - - devres_add(dev, dr); - - return vaddr; -} -EXPORT_SYMBOL(dmam_alloc_coherent); - -/** - * dmam_free_coherent - Managed dma_free_coherent() - * @dev: Device to free coherent memory for - * @size: Size of allocation - * @vaddr: Virtual address of the memory to free - * @dma_handle: DMA handle of the memory to free - * - * Managed dma_free_coherent(). - */ -void dmam_free_coherent(struct device *dev, size_t size, void *vaddr, - dma_addr_t dma_handle) -{ - struct dma_devres match_data = { size, vaddr, dma_handle }; - - dma_free_coherent(dev, size, vaddr, dma_handle); - WARN_ON(devres_destroy(dev, dmam_release, dmam_match, &match_data)); -} -EXPORT_SYMBOL(dmam_free_coherent); - -/** - * dmam_alloc_attrs - Managed dma_alloc_attrs() - * @dev: Device to allocate non_coherent memory for - * @size: Size of allocation - * @dma_handle: Out argument for allocated DMA handle - * @gfp: Allocation flags - * @attrs: Flags in the DMA_ATTR_* namespace. - * - * Managed dma_alloc_attrs(). Memory allocated using this function will be - * automatically released on driver detach. - * - * RETURNS: - * Pointer to allocated memory on success, NULL on failure. - */ -void *dmam_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, - gfp_t gfp, unsigned long attrs) -{ - struct dma_devres *dr; - void *vaddr; - - dr = devres_alloc(dmam_release, sizeof(*dr), gfp); - if (!dr) - return NULL; - - vaddr = dma_alloc_attrs(dev, size, dma_handle, gfp, attrs); - if (!vaddr) { - devres_free(dr); - return NULL; - } - - dr->vaddr = vaddr; - dr->dma_handle = *dma_handle; - dr->size = size; - dr->attrs = attrs; - - devres_add(dev, dr); - - return vaddr; -} -EXPORT_SYMBOL(dmam_alloc_attrs); - -#ifdef CONFIG_HAVE_GENERIC_DMA_COHERENT - -static void dmam_coherent_decl_release(struct device *dev, void *res) -{ - dma_release_declared_memory(dev); -} - -/** - * dmam_declare_coherent_memory - Managed dma_declare_coherent_memory() - * @dev: Device to declare coherent memory for - * @phys_addr: Physical address of coherent memory to be declared - * @device_addr: Device address of coherent memory to be declared - * @size: Size of coherent memory to be declared - * @flags: Flags - * - * Managed dma_declare_coherent_memory(). - * - * RETURNS: - * 0 on success, -errno on failure. - */ -int dmam_declare_coherent_memory(struct device *dev, phys_addr_t phys_addr, - dma_addr_t device_addr, size_t size, int flags) -{ - void *res; - int rc; - - res = devres_alloc(dmam_coherent_decl_release, 0, GFP_KERNEL); - if (!res) - return -ENOMEM; - - rc = dma_declare_coherent_memory(dev, phys_addr, device_addr, size, - flags); - if (!rc) - devres_add(dev, res); - else - devres_free(res); - - return rc; -} -EXPORT_SYMBOL(dmam_declare_coherent_memory); - -/** - * dmam_release_declared_memory - Managed dma_release_declared_memory(). - * @dev: Device to release declared coherent memory for - * - * Managed dmam_release_declared_memory(). - */ -void dmam_release_declared_memory(struct device *dev) -{ - WARN_ON(devres_destroy(dev, dmam_coherent_decl_release, NULL, NULL)); -} -EXPORT_SYMBOL(dmam_release_declared_memory); - -#endif - -/* - * Create scatter-list for the already allocated DMA buffer. - */ -int dma_common_get_sgtable(struct device *dev, struct sg_table *sgt, - void *cpu_addr, dma_addr_t handle, size_t size) -{ - struct page *page = virt_to_page(cpu_addr); - int ret; - - ret = sg_alloc_table(sgt, 1, GFP_KERNEL); - if (unlikely(ret)) - return ret; - - sg_set_page(sgt->sgl, page, PAGE_ALIGN(size), 0); - return 0; -} -EXPORT_SYMBOL(dma_common_get_sgtable); - -/* - * Create userspace mapping for the DMA-coherent memory. - */ -int dma_common_mmap(struct device *dev, struct vm_area_struct *vma, - void *cpu_addr, dma_addr_t dma_addr, size_t size) -{ - int ret = -ENXIO; -#ifndef CONFIG_ARCH_NO_COHERENT_DMA_MMAP - unsigned long user_count = vma_pages(vma); - unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT; - unsigned long off = vma->vm_pgoff; - - vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); - - if (dma_mmap_from_dev_coherent(dev, vma, cpu_addr, size, &ret)) - return ret; - - if (off < count && user_count <= (count - off)) - ret = remap_pfn_range(vma, vma->vm_start, - page_to_pfn(virt_to_page(cpu_addr)) + off, - user_count << PAGE_SHIFT, - vma->vm_page_prot); -#endif /* !CONFIG_ARCH_NO_COHERENT_DMA_MMAP */ - - return ret; -} -EXPORT_SYMBOL(dma_common_mmap); - -#ifdef CONFIG_MMU -static struct vm_struct *__dma_common_pages_remap(struct page **pages, - size_t size, unsigned long vm_flags, pgprot_t prot, - const void *caller) -{ - struct vm_struct *area; - - area = get_vm_area_caller(size, vm_flags, caller); - if (!area) - return NULL; - - if (map_vm_area(area, prot, pages)) { - vunmap(area->addr); - return NULL; - } - - return area; -} - -/* - * remaps an array of PAGE_SIZE pages into another vm_area - * Cannot be used in non-sleeping contexts - */ -void *dma_common_pages_remap(struct page **pages, size_t size, - unsigned long vm_flags, pgprot_t prot, - const void *caller) -{ - struct vm_struct *area; - - area = __dma_common_pages_remap(pages, size, vm_flags, prot, caller); - if (!area) - return NULL; - - area->pages = pages; - - return area->addr; -} - -/* - * remaps an allocated contiguous region into another vm_area. - * Cannot be used in non-sleeping contexts - */ - -void *dma_common_contiguous_remap(struct page *page, size_t size, - unsigned long vm_flags, - pgprot_t prot, const void *caller) -{ - int i; - struct page **pages; - struct vm_struct *area; - - pages = kmalloc(sizeof(struct page *) << get_order(size), GFP_KERNEL); - if (!pages) - return NULL; - - for (i = 0; i < (size >> PAGE_SHIFT); i++) - pages[i] = nth_page(page, i); - - area = __dma_common_pages_remap(pages, size, vm_flags, prot, caller); - - kfree(pages); - - if (!area) - return NULL; - return area->addr; -} - -/* - * unmaps a range previously mapped by dma_common_*_remap - */ -void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags) -{ - struct vm_struct *area = find_vm_area(cpu_addr); - - if (!area || (area->flags & vm_flags) != vm_flags) { - WARN(1, "trying to free invalid coherent area: %p\n", cpu_addr); - return; - } - - unmap_kernel_range((unsigned long)cpu_addr, PAGE_ALIGN(size)); - vunmap(cpu_addr); -} -#endif - -/* - * Common configuration to enable DMA API use for a device - */ -#include - -int dma_configure(struct device *dev) -{ - struct device *bridge = NULL, *dma_dev = dev; - enum dev_dma_attr attr; - int ret = 0; - - if (dev_is_pci(dev)) { - bridge = pci_get_host_bridge_device(to_pci_dev(dev)); - dma_dev = bridge; - if (IS_ENABLED(CONFIG_OF) && dma_dev->parent && - dma_dev->parent->of_node) - dma_dev = dma_dev->parent; - } - - if (dma_dev->of_node) { - ret = of_dma_configure(dev, dma_dev->of_node); - } else if (has_acpi_companion(dma_dev)) { - attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode)); - if (attr != DEV_DMA_NOT_SUPPORTED) - ret = acpi_dma_configure(dev, attr); - } - - if (bridge) - pci_put_host_bridge_device(bridge); - - return ret; -} - -void dma_deconfigure(struct device *dev) -{ - of_dma_deconfigure(dev); - acpi_dma_deconfigure(dev); -} diff --git a/drivers/base/driver.c b/drivers/base/driver.c index ba912558a510..857c8f1b876e 100644 --- a/drivers/base/driver.c +++ b/drivers/base/driver.c @@ -148,7 +148,11 @@ int driver_register(struct device_driver *drv) int ret; struct device_driver *other; - BUG_ON(!drv->bus->p); + if (!drv->bus->p) { + pr_err("Driver '%s' was unable to register with bus_type '%s' because the bus was not initialized.\n", + drv->name, drv->bus->name); + return -EINVAL; + } if ((drv->bus->probe && drv->probe) || (drv->bus->remove && drv->remove) || diff --git a/drivers/base/firmware_loader/Kconfig b/drivers/base/firmware_loader/Kconfig new file mode 100644 index 000000000000..eb15d976a9ea --- /dev/null +++ b/drivers/base/firmware_loader/Kconfig @@ -0,0 +1,154 @@ +menu "Firmware loader" + +config FW_LOADER + tristate "Firmware loading facility" if EXPERT + default y + help + This enables the firmware loading facility in the kernel. The kernel + will first look for built-in firmware, if it has any. Next, it will + look for the requested firmware in a series of filesystem paths: + + o firmware_class path module parameter or kernel boot param + o /lib/firmware/updates/UTS_RELEASE + o /lib/firmware/updates + o /lib/firmware/UTS_RELEASE + o /lib/firmware + + Enabling this feature only increases your kernel image by about + 828 bytes, enable this option unless you are certain you don't + need firmware. + + You typically want this built-in (=y) but you can also enable this + as a module, in which case the firmware_class module will be built. + You also want to be sure to enable this built-in if you are going to + enable built-in firmware (CONFIG_EXTRA_FIRMWARE). + +if FW_LOADER + +config EXTRA_FIRMWARE + string "Build named firmware blobs into the kernel binary" + help + Device drivers which require firmware can typically deal with + having the kernel load firmware from the various supported + /lib/firmware/ paths. This option enables you to build into the + kernel firmware files. Built-in firmware searches are preceded + over firmware lookups using your filesystem over the supported + /lib/firmware paths documented on CONFIG_FW_LOADER. + + This may be useful for testing or if the firmware is required early on + in boot and cannot rely on the firmware being placed in an initrd or + initramfs. + + This option is a string and takes the (space-separated) names of the + firmware files -- the same names that appear in MODULE_FIRMWARE() + and request_firmware() in the source. These files should exist under + the directory specified by the EXTRA_FIRMWARE_DIR option, which is + /lib/firmware by default. + + For example, you might set CONFIG_EXTRA_FIRMWARE="usb8388.bin", copy + the usb8388.bin file into /lib/firmware, and build the kernel. Then + any request_firmware("usb8388.bin") will be satisfied internally + inside the kernel without ever looking at your filesystem at runtime. + + WARNING: If you include additional firmware files into your binary + kernel image that are not available under the terms of the GPL, + then it may be a violation of the GPL to distribute the resulting + image since it combines both GPL and non-GPL work. You should + consult a lawyer of your own before distributing such an image. + +config EXTRA_FIRMWARE_DIR + string "Firmware blobs root directory" + depends on EXTRA_FIRMWARE != "" + default "/lib/firmware" + help + This option controls the directory in which the kernel build system + looks for the firmware files listed in the EXTRA_FIRMWARE option. + +config FW_LOADER_USER_HELPER + bool "Enable the firmware sysfs fallback mechanism" + help + This option enables a sysfs loading facility to enable firmware + loading to the kernel through userspace as a fallback mechanism + if and only if the kernel's direct filesystem lookup for the + firmware failed using the different /lib/firmware/ paths, or the + path specified in the firmware_class path module parameter, or the + firmware_class path kernel boot parameter if the firmware_class is + built-in. For details on how to work with the sysfs fallback mechanism + refer to Documentation/driver-api/firmware/fallback-mechanisms.rst. + + The direct filesystem lookup for firmware is always used first now. + + If the kernel's direct filesystem lookup for firmware fails to find + the requested firmware a sysfs fallback loading facility is made + available and userspace is informed about this through uevents. + The uevent can be suppressed if the driver explicitly requested it, + this is known as the driver using the custom fallback mechanism. + If the custom fallback mechanism is used userspace must always + acknowledge failure to find firmware as the timeout for the fallback + mechanism is disabled, and failed requests will linger forever. + + This used to be the default firmware loading facility, and udev used + to listen for uvents to load firmware for the kernel. The firmware + loading facility functionality in udev has been removed, as such it + can no longer be relied upon as a fallback mechanism. Linux no longer + relies on or uses a fallback mechanism in userspace. If you need to + rely on one refer to the permissively licensed firmwared: + + https://github.com/teg/firmwared + + Since this was the default firmware loading facility at one point, + old userspace may exist which relies upon it, and as such this + mechanism can never be removed from the kernel. + + You should only enable this functionality if you are certain you + require a fallback mechanism and have a userspace mechanism ready to + load firmware in case it is not found. One main reason for this may + be if you have drivers which require firmware built-in and for + whatever reason cannot place the required firmware in initramfs. + Another reason kernels may have this feature enabled is to support a + driver which explicitly relies on this fallback mechanism. Only two + drivers need this today: + + o CONFIG_LEDS_LP55XX_COMMON + o CONFIG_DELL_RBU + + Outside of supporting the above drivers, another reason for needing + this may be that your firmware resides outside of the paths the kernel + looks for and cannot possibly be specified using the firmware_class + path module parameter or kernel firmware_class path boot parameter + if firmware_class is built-in. + + A modern use case may be to temporarily mount a custom partition + during provisioning which is only accessible to userspace, and then + to use it to look for and fetch the required firmware. Such type of + driver functionality may not even ever be desirable upstream by + vendors, and as such is only required to be supported as an interface + for provisioning. Since udev's firmware loading facility has been + removed you can use firmwared or a fork of it to customize how you + want to load firmware based on uevents issued. + + Enabling this option will increase your kernel image size by about + 13436 bytes. + + If you are unsure about this, say N here, unless you are Linux + distribution and need to support the above two drivers, or you are + certain you need to support some really custom firmware loading + facility in userspace. + +config FW_LOADER_USER_HELPER_FALLBACK + bool "Force the firmware sysfs fallback mechanism when possible" + depends on FW_LOADER_USER_HELPER + help + Enabling this option forces a sysfs userspace fallback mechanism + to be used for all firmware requests which explicitly do not disable a + a fallback mechanism. Firmware calls which do prohibit a fallback + mechanism is request_firmware_direct(). This option is kept for + backward compatibility purposes given this precise mechanism can also + be enabled by setting the proc sysctl value to true: + + /proc/sys/kernel/firmware_config/force_sysfs_fallback + + If you are unsure about this, say N here. + +endif # FW_LOADER +endmenu diff --git a/drivers/base/firmware_loader/fallback.c b/drivers/base/firmware_loader/fallback.c index 358354148dec..7f732744f0d3 100644 --- a/drivers/base/firmware_loader/fallback.c +++ b/drivers/base/firmware_loader/fallback.c @@ -125,7 +125,7 @@ static ssize_t timeout_show(struct class *class, struct class_attribute *attr, } /** - * firmware_timeout_store - set number of seconds to wait for firmware + * firmware_timeout_store() - set number of seconds to wait for firmware * @class: device class pointer * @attr: device attribute pointer * @buf: buffer to scan for timeout value @@ -239,7 +239,7 @@ static int map_fw_priv_pages(struct fw_priv *fw_priv) } /** - * firmware_loading_store - set value in the 'loading' control file + * firmware_loading_store() - set value in the 'loading' control file * @dev: device pointer * @attr: device attribute pointer * @buf: buffer to scan for loading control value @@ -403,7 +403,7 @@ static int fw_realloc_pages(struct fw_sysfs *fw_sysfs, int min_size) fw_priv->page_array_size * 2); struct page **new_pages; - new_pages = vmalloc(new_array_size * sizeof(void *)); + new_pages = vmalloc(array_size(new_array_size, sizeof(void *))); if (!new_pages) { fw_load_abort(fw_sysfs); return -ENOMEM; @@ -431,7 +431,7 @@ static int fw_realloc_pages(struct fw_sysfs *fw_sysfs, int min_size) } /** - * firmware_data_write - write method for firmware + * firmware_data_write() - write method for firmware * @filp: open sysfs file * @kobj: kobject for the device * @bin_attr: bin_attr structure @@ -512,7 +512,7 @@ static const struct attribute_group *fw_dev_attr_groups[] = { static struct fw_sysfs * fw_create_instance(struct firmware *firmware, const char *fw_name, - struct device *device, unsigned int opt_flags) + struct device *device, enum fw_opt opt_flags) { struct fw_sysfs *fw_sysfs; struct device *f_dev; @@ -537,7 +537,7 @@ exit: } /** - * fw_load_sysfs_fallback - load a firmware via the sysfs fallback mechanism + * fw_load_sysfs_fallback() - load a firmware via the sysfs fallback mechanism * @fw_sysfs: firmware sysfs information for the firmware to load * @opt_flags: flags of options, FW_OPT_* * @timeout: timeout to wait for the load @@ -545,7 +545,7 @@ exit: * In charge of constructing a sysfs fallback interface for firmware loading. **/ static int fw_load_sysfs_fallback(struct fw_sysfs *fw_sysfs, - unsigned int opt_flags, long timeout) + enum fw_opt opt_flags, long timeout) { int retval = 0; struct device *f_dev = &fw_sysfs->dev; @@ -599,7 +599,7 @@ err_put_dev: static int fw_load_from_user_helper(struct firmware *firmware, const char *name, struct device *device, - unsigned int opt_flags) + enum fw_opt opt_flags) { struct fw_sysfs *fw_sysfs; long timeout; @@ -640,7 +640,7 @@ out_unlock: return ret; } -static bool fw_force_sysfs_fallback(unsigned int opt_flags) +static bool fw_force_sysfs_fallback(enum fw_opt opt_flags) { if (fw_fallback_config.force_sysfs_fallback) return true; @@ -649,7 +649,7 @@ static bool fw_force_sysfs_fallback(unsigned int opt_flags) return true; } -static bool fw_run_sysfs_fallback(unsigned int opt_flags) +static bool fw_run_sysfs_fallback(enum fw_opt opt_flags) { if (fw_fallback_config.ignore_sysfs_fallback) { pr_info_once("Ignoring firmware sysfs fallback due to sysctl knob\n"); @@ -662,14 +662,39 @@ static bool fw_run_sysfs_fallback(unsigned int opt_flags) return fw_force_sysfs_fallback(opt_flags); } -int fw_sysfs_fallback(struct firmware *fw, const char *name, - struct device *device, - unsigned int opt_flags, - int ret) +/** + * firmware_fallback_sysfs() - use the fallback mechanism to find firmware + * @fw: pointer to firmware image + * @name: name of firmware file to look for + * @device: device for which firmware is being loaded + * @opt_flags: options to control firmware loading behaviour + * @ret: return value from direct lookup which triggered the fallback mechanism + * + * This function is called if direct lookup for the firmware failed, it enables + * a fallback mechanism through userspace by exposing a sysfs loading + * interface. Userspace is in charge of loading the firmware through the syfs + * loading interface. This syfs fallback mechanism may be disabled completely + * on a system by setting the proc sysctl value ignore_sysfs_fallback to true. + * If this false we check if the internal API caller set the @FW_OPT_NOFALLBACK + * flag, if so it would also disable the fallback mechanism. A system may want + * to enfoce the sysfs fallback mechanism at all times, it can do this by + * setting ignore_sysfs_fallback to false and force_sysfs_fallback to true. + * Enabling force_sysfs_fallback is functionally equivalent to build a kernel + * with CONFIG_FW_LOADER_USER_HELPER_FALLBACK. + **/ +int firmware_fallback_sysfs(struct firmware *fw, const char *name, + struct device *device, + enum fw_opt opt_flags, + int ret) { if (!fw_run_sysfs_fallback(opt_flags)) return ret; - dev_warn(device, "Falling back to user helper\n"); + if (!(opt_flags & FW_OPT_NO_WARN)) + dev_warn(device, "Falling back to syfs fallback for: %s\n", + name); + else + dev_dbg(device, "Falling back to sysfs fallback for: %s\n", + name); return fw_load_from_user_helper(fw, name, device, opt_flags); } diff --git a/drivers/base/firmware_loader/fallback.h b/drivers/base/firmware_loader/fallback.h index f8255670a663..21063503e4ea 100644 --- a/drivers/base/firmware_loader/fallback.h +++ b/drivers/base/firmware_loader/fallback.h @@ -5,6 +5,8 @@ #include #include +#include "firmware.h" + /** * struct firmware_fallback_config - firmware fallback configuration settings * @@ -29,10 +31,10 @@ struct firmware_fallback_config { }; #ifdef CONFIG_FW_LOADER_USER_HELPER -int fw_sysfs_fallback(struct firmware *fw, const char *name, - struct device *device, - unsigned int opt_flags, - int ret); +int firmware_fallback_sysfs(struct firmware *fw, const char *name, + struct device *device, + enum fw_opt opt_flags, + int ret); void kill_pending_fw_fallback_reqs(bool only_kill_custom); void fw_fallback_set_cache_timeout(void); @@ -41,10 +43,10 @@ void fw_fallback_set_default_timeout(void); int register_sysfs_loader(void); void unregister_sysfs_loader(void); #else /* CONFIG_FW_LOADER_USER_HELPER */ -static inline int fw_sysfs_fallback(struct firmware *fw, const char *name, - struct device *device, - unsigned int opt_flags, - int ret) +static inline int firmware_fallback_sysfs(struct firmware *fw, const char *name, + struct device *device, + enum fw_opt opt_flags, + int ret) { /* Keep carrying over the same error */ return ret; diff --git a/drivers/base/firmware_loader/firmware.h b/drivers/base/firmware_loader/firmware.h index 64acbb1a392c..4c1395f8e7ed 100644 --- a/drivers/base/firmware_loader/firmware.h +++ b/drivers/base/firmware_loader/firmware.h @@ -2,6 +2,7 @@ #ifndef __FIRMWARE_LOADER_H #define __FIRMWARE_LOADER_H +#include #include #include #include @@ -10,13 +11,33 @@ #include -/* firmware behavior options */ -#define FW_OPT_UEVENT (1U << 0) -#define FW_OPT_NOWAIT (1U << 1) -#define FW_OPT_USERHELPER (1U << 2) -#define FW_OPT_NO_WARN (1U << 3) -#define FW_OPT_NOCACHE (1U << 4) -#define FW_OPT_NOFALLBACK (1U << 5) +/** + * enum fw_opt - options to control firmware loading behaviour + * + * @FW_OPT_UEVENT: Enables the fallback mechanism to send a kobject uevent + * when the firmware is not found. Userspace is in charge to load the + * firmware using the sysfs loading facility. + * @FW_OPT_NOWAIT: Used to describe the firmware request is asynchronous. + * @FW_OPT_USERHELPER: Enable the fallback mechanism, in case the direct + * filesystem lookup fails at finding the firmware. For details refer to + * firmware_fallback_sysfs(). + * @FW_OPT_NO_WARN: Quiet, avoid printing warning messages. + * @FW_OPT_NOCACHE: Disables firmware caching. Firmware caching is used to + * cache the firmware upon suspend, so that upon resume races against the + * firmware file lookup on storage is avoided. Used for calls where the + * file may be too big, or where the driver takes charge of its own + * firmware caching mechanism. + * @FW_OPT_NOFALLBACK: Disable the fallback mechanism. Takes precedence over + * &FW_OPT_UEVENT and &FW_OPT_USERHELPER. + */ +enum fw_opt { + FW_OPT_UEVENT = BIT(0), + FW_OPT_NOWAIT = BIT(1), + FW_OPT_USERHELPER = BIT(2), + FW_OPT_NO_WARN = BIT(3), + FW_OPT_NOCACHE = BIT(4), + FW_OPT_NOFALLBACK = BIT(5), +}; enum fw_status { FW_STATUS_UNKNOWN, @@ -110,6 +131,6 @@ static inline void fw_state_done(struct fw_priv *fw_priv) } int assign_fw(struct firmware *fw, struct device *device, - unsigned int opt_flags); + enum fw_opt opt_flags); #endif /* __FIRMWARE_LOADER_H */ diff --git a/drivers/base/firmware_loader/main.c b/drivers/base/firmware_loader/main.c index eb34089e4299..0943e7065e0e 100644 --- a/drivers/base/firmware_loader/main.c +++ b/drivers/base/firmware_loader/main.c @@ -443,7 +443,7 @@ static int fw_add_devm_name(struct device *dev, const char *name) #endif int assign_fw(struct firmware *fw, struct device *device, - unsigned int opt_flags) + enum fw_opt opt_flags) { struct fw_priv *fw_priv = fw->priv; int ret; @@ -558,7 +558,7 @@ static void fw_abort_batch_reqs(struct firmware *fw) static int _request_firmware(const struct firmware **firmware_p, const char *name, struct device *device, void *buf, size_t size, - unsigned int opt_flags) + enum fw_opt opt_flags) { struct firmware *fw = NULL; int ret; @@ -581,7 +581,7 @@ _request_firmware(const struct firmware **firmware_p, const char *name, dev_warn(device, "Direct firmware load for %s failed with error %d\n", name, ret); - ret = fw_sysfs_fallback(fw, name, device, opt_flags, ret); + ret = firmware_fallback_sysfs(fw, name, device, opt_flags, ret); } else ret = assign_fw(fw, device, opt_flags); @@ -597,7 +597,7 @@ _request_firmware(const struct firmware **firmware_p, const char *name, } /** - * request_firmware: - send firmware request and wait for it + * request_firmware() - send firmware request and wait for it * @firmware_p: pointer to firmware image * @name: name of firmware file * @device: device for which firmware is being loaded @@ -632,7 +632,34 @@ request_firmware(const struct firmware **firmware_p, const char *name, EXPORT_SYMBOL(request_firmware); /** - * request_firmware_direct: - load firmware directly without usermode helper + * firmware_request_nowarn() - request for an optional fw module + * @firmware: pointer to firmware image + * @name: name of firmware file + * @device: device for which firmware is being loaded + * + * This function is similar in behaviour to request_firmware(), except + * it doesn't produce warning messages when the file is not found. + * The sysfs fallback mechanism is enabled if direct filesystem lookup fails, + * however, however failures to find the firmware file with it are still + * suppressed. It is therefore up to the driver to check for the return value + * of this call and to decide when to inform the users of errors. + **/ +int firmware_request_nowarn(const struct firmware **firmware, const char *name, + struct device *device) +{ + int ret; + + /* Need to pin this module until return */ + __module_get(THIS_MODULE); + ret = _request_firmware(firmware, name, device, NULL, 0, + FW_OPT_UEVENT | FW_OPT_NO_WARN); + module_put(THIS_MODULE); + return ret; +} +EXPORT_SYMBOL_GPL(firmware_request_nowarn); + +/** + * request_firmware_direct() - load firmware directly without usermode helper * @firmware_p: pointer to firmware image * @name: name of firmware file * @device: device for which firmware is being loaded @@ -657,7 +684,7 @@ int request_firmware_direct(const struct firmware **firmware_p, EXPORT_SYMBOL_GPL(request_firmware_direct); /** - * firmware_request_cache: - cache firmware for suspend so resume can use it + * firmware_request_cache() - cache firmware for suspend so resume can use it * @name: name of firmware file * @device: device for which firmware should be cached for * @@ -681,7 +708,7 @@ int firmware_request_cache(struct device *device, const char *name) EXPORT_SYMBOL_GPL(firmware_request_cache); /** - * request_firmware_into_buf - load firmware into a previously allocated buffer + * request_firmware_into_buf() - load firmware into a previously allocated buffer * @firmware_p: pointer to firmware image * @name: name of firmware file * @device: device for which firmware is being loaded and DMA region allocated @@ -713,7 +740,7 @@ request_firmware_into_buf(const struct firmware **firmware_p, const char *name, EXPORT_SYMBOL(request_firmware_into_buf); /** - * release_firmware: - release the resource associated with a firmware image + * release_firmware() - release the resource associated with a firmware image * @fw: firmware resource to release **/ void release_firmware(const struct firmware *fw) @@ -734,7 +761,7 @@ struct firmware_work { struct device *device; void *context; void (*cont)(const struct firmware *fw, void *context); - unsigned int opt_flags; + enum fw_opt opt_flags; }; static void request_firmware_work_func(struct work_struct *work) @@ -755,7 +782,7 @@ static void request_firmware_work_func(struct work_struct *work) } /** - * request_firmware_nowait - asynchronous version of request_firmware + * request_firmware_nowait() - asynchronous version of request_firmware * @module: module requesting the firmware * @uevent: sends uevent to copy the firmware image if this flag * is non-zero else the firmware copy must be done manually. @@ -824,7 +851,7 @@ EXPORT_SYMBOL(request_firmware_nowait); static ASYNC_DOMAIN_EXCLUSIVE(fw_cache_domain); /** - * cache_firmware - cache one firmware image in kernel memory space + * cache_firmware() - cache one firmware image in kernel memory space * @fw_name: the firmware image name * * Cache firmware in kernel memory so that drivers can use it when @@ -866,7 +893,7 @@ static struct fw_priv *lookup_fw_priv(const char *fw_name) } /** - * uncache_firmware - remove one cached firmware image + * uncache_firmware() - remove one cached firmware image * @fw_name: the firmware image name * * Uncache one firmware image which has been cached successfully @@ -1042,7 +1069,7 @@ static void __device_uncache_fw_images(void) } /** - * device_cache_fw_images - cache devices' firmware + * device_cache_fw_images() - cache devices' firmware * * If one device called request_firmware or its nowait version * successfully before, the firmware names are recored into the @@ -1075,7 +1102,7 @@ static void device_cache_fw_images(void) } /** - * device_uncache_fw_images - uncache devices' firmware + * device_uncache_fw_images() - uncache devices' firmware * * uncache all firmwares which have been cached successfully * by device_uncache_fw_images earlier @@ -1092,7 +1119,7 @@ static void device_uncache_fw_images_work(struct work_struct *work) } /** - * device_uncache_fw_images_delay - uncache devices firmwares + * device_uncache_fw_images_delay() - uncache devices firmwares * @delay: number of milliseconds to delay uncache device firmwares * * uncache all devices's firmwares which has been cached successfully diff --git a/drivers/base/memory.c b/drivers/base/memory.c index bffe8616bd55..f5e560188a18 100644 --- a/drivers/base/memory.c +++ b/drivers/base/memory.c @@ -649,13 +649,19 @@ static const struct attribute_group *memory_memblk_attr_groups[] = { static int register_memory(struct memory_block *memory) { + int ret; + memory->dev.bus = &memory_subsys; memory->dev.id = memory->start_section_nr / sections_per_block; memory->dev.release = memory_block_release; memory->dev.groups = memory_memblk_attr_groups; memory->dev.offline = memory->state == MEM_OFFLINE; - return device_register(&memory->dev); + ret = device_register(&memory->dev); + if (ret) + put_device(&memory->dev); + + return ret; } static int init_memory_block(struct memory_block **memory, diff --git a/drivers/base/platform-msi.c b/drivers/base/platform-msi.c index 8e22073aeeed..60d6cc618f1c 100644 --- a/drivers/base/platform-msi.c +++ b/drivers/base/platform-msi.c @@ -101,6 +101,9 @@ static void platform_msi_update_chip_ops(struct msi_domain_info *info) chip->irq_set_affinity = msi_domain_set_affinity; if (!chip->irq_write_msi_msg) chip->irq_write_msi_msg = platform_msi_write_msg; + if (WARN_ON((info->flags & MSI_FLAG_LEVEL_CAPABLE) && + !(chip->flags & IRQCHIP_SUPPORTS_LEVEL_MSI))) + info->flags &= ~MSI_FLAG_LEVEL_CAPABLE; } static void platform_msi_free_descs(struct device *dev, int base, int nvec) diff --git a/drivers/base/platform.c b/drivers/base/platform.c index 8075ddc70a17..dff82a3c2caa 100644 --- a/drivers/base/platform.c +++ b/drivers/base/platform.c @@ -572,17 +572,16 @@ static int platform_drv_probe(struct device *_dev) return ret; ret = dev_pm_domain_attach(_dev, true); - if (ret != -EPROBE_DEFER) { - if (drv->probe) { - ret = drv->probe(dev); - if (ret) - dev_pm_domain_detach(_dev, true); - } else { - /* don't fail if just dev_pm_domain_attach failed */ - ret = 0; - } + if (ret) + goto out; + + if (drv->probe) { + ret = drv->probe(dev); + if (ret) + dev_pm_domain_detach(_dev, true); } +out: if (drv->prevent_deferred_probe && ret == -EPROBE_DEFER) { dev_warn(_dev, "probe deferral not supported\n"); ret = -ENXIO; @@ -1130,6 +1129,22 @@ int platform_pm_restore(struct device *dev) #endif /* CONFIG_HIBERNATE_CALLBACKS */ +int platform_dma_configure(struct device *dev) +{ + enum dev_dma_attr attr; + int ret = 0; + + if (dev->of_node) { + ret = of_dma_configure(dev, dev->of_node, true); + } else if (has_acpi_companion(dev)) { + attr = acpi_get_dma_attr(to_acpi_device_node(dev->fwnode)); + if (attr != DEV_DMA_NOT_SUPPORTED) + ret = acpi_dma_configure(dev, attr); + } + + return ret; +} + static const struct dev_pm_ops platform_dev_pm_ops = { .runtime_suspend = pm_generic_runtime_suspend, .runtime_resume = pm_generic_runtime_resume, @@ -1141,8 +1156,8 @@ struct bus_type platform_bus_type = { .dev_groups = platform_dev_groups, .match = platform_match, .uevent = platform_uevent, + .dma_configure = platform_dma_configure, .pm = &platform_dev_pm_ops, - .force_dma = true, }; EXPORT_SYMBOL_GPL(platform_bus_type); diff --git a/drivers/base/power/common.c b/drivers/base/power/common.c index f6a9ad52cbbf..df41b4780b3b 100644 --- a/drivers/base/power/common.c +++ b/drivers/base/power/common.c @@ -98,28 +98,69 @@ EXPORT_SYMBOL_GPL(dev_pm_put_subsys_data); * Callers must ensure proper synchronization of this function with power * management callbacks. * - * Returns 0 on successfully attached PM domain or negative error code. + * Returns 0 on successfully attached PM domain, or when it is found that the + * device doesn't need a PM domain, else a negative error code. */ int dev_pm_domain_attach(struct device *dev, bool power_on) { int ret; + if (dev->pm_domain) + return 0; + ret = acpi_dev_pm_attach(dev, power_on); - if (ret) + if (!ret) ret = genpd_dev_pm_attach(dev); - return ret; + return ret < 0 ? ret : 0; } EXPORT_SYMBOL_GPL(dev_pm_domain_attach); +/** + * dev_pm_domain_attach_by_id - Associate a device with one of its PM domains. + * @dev: The device used to lookup the PM domain. + * @index: The index of the PM domain. + * + * As @dev may only be attached to a single PM domain, the backend PM domain + * provider creates a virtual device to attach instead. If attachment succeeds, + * the ->detach() callback in the struct dev_pm_domain are assigned by the + * corresponding backend attach function, as to deal with detaching of the + * created virtual device. + * + * This function should typically be invoked by a driver during the probe phase, + * in case its device requires power management through multiple PM domains. The + * driver may benefit from using the received device, to configure device-links + * towards its original device. Depending on the use-case and if needed, the + * links may be dynamically changed by the driver, which allows it to control + * the power to the PM domains independently from each other. + * + * Callers must ensure proper synchronization of this function with power + * management callbacks. + * + * Returns the virtual created device when successfully attached to its PM + * domain, NULL in case @dev don't need a PM domain, else an ERR_PTR(). + * Note that, to detach the returned virtual device, the driver shall call + * dev_pm_domain_detach() on it, typically during the remove phase. + */ +struct device *dev_pm_domain_attach_by_id(struct device *dev, + unsigned int index) +{ + if (dev->pm_domain) + return ERR_PTR(-EEXIST); + + return genpd_dev_pm_attach_by_id(dev, index); +} +EXPORT_SYMBOL_GPL(dev_pm_domain_attach_by_id); + /** * dev_pm_domain_detach - Detach a device from its PM domain. * @dev: Device to detach. * @power_off: Used to indicate whether we should power off the device. * - * This functions will reverse the actions from dev_pm_domain_attach() and thus - * try to detach the @dev from its PM domain. Typically it should be invoked - * from subsystem level code during the remove phase. + * This functions will reverse the actions from dev_pm_domain_attach() and + * dev_pm_domain_attach_by_id(), thus it detaches @dev from its PM domain. + * Typically it should be invoked during the remove phase, either from + * subsystem level code or from drivers. * * Callers must ensure proper synchronization of this function with power * management callbacks. diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c index 1ea0e2502e8e..4925af5c4cf0 100644 --- a/drivers/base/power/domain.c +++ b/drivers/base/power/domain.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -1315,7 +1316,6 @@ EXPORT_SYMBOL_GPL(pm_genpd_syscore_poweron); #endif /* CONFIG_PM_SLEEP */ static struct generic_pm_domain_data *genpd_alloc_dev_data(struct device *dev, - struct generic_pm_domain *genpd, struct gpd_timing_data *td) { struct generic_pm_domain_data *gpd_data; @@ -1377,24 +1377,19 @@ static int genpd_add_device(struct generic_pm_domain *genpd, struct device *dev, struct gpd_timing_data *td) { struct generic_pm_domain_data *gpd_data; - int ret = 0; + int ret; dev_dbg(dev, "%s()\n", __func__); if (IS_ERR_OR_NULL(genpd) || IS_ERR_OR_NULL(dev)) return -EINVAL; - gpd_data = genpd_alloc_dev_data(dev, genpd, td); + gpd_data = genpd_alloc_dev_data(dev, td); if (IS_ERR(gpd_data)) return PTR_ERR(gpd_data); genpd_lock(genpd); - if (genpd->prepared_count > 0) { - ret = -EAGAIN; - goto out; - } - ret = genpd->attach_dev ? genpd->attach_dev(genpd, dev) : 0; if (ret) goto out; @@ -1418,23 +1413,21 @@ static int genpd_add_device(struct generic_pm_domain *genpd, struct device *dev, } /** - * __pm_genpd_add_device - Add a device to an I/O PM domain. + * pm_genpd_add_device - Add a device to an I/O PM domain. * @genpd: PM domain to add the device to. * @dev: Device to be added. - * @td: Set of PM QoS timing parameters to attach to the device. */ -int __pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev, - struct gpd_timing_data *td) +int pm_genpd_add_device(struct generic_pm_domain *genpd, struct device *dev) { int ret; mutex_lock(&gpd_list_lock); - ret = genpd_add_device(genpd, dev, td); + ret = genpd_add_device(genpd, dev, NULL); mutex_unlock(&gpd_list_lock); return ret; } -EXPORT_SYMBOL_GPL(__pm_genpd_add_device); +EXPORT_SYMBOL_GPL(pm_genpd_add_device); static int genpd_remove_device(struct generic_pm_domain *genpd, struct device *dev) @@ -1481,13 +1474,13 @@ static int genpd_remove_device(struct generic_pm_domain *genpd, /** * pm_genpd_remove_device - Remove a device from an I/O PM domain. - * @genpd: PM domain to remove the device from. * @dev: Device to be removed. */ -int pm_genpd_remove_device(struct generic_pm_domain *genpd, - struct device *dev) +int pm_genpd_remove_device(struct device *dev) { - if (!genpd || genpd != genpd_lookup_dev(dev)) + struct generic_pm_domain *genpd = genpd_lookup_dev(dev); + + if (!genpd) return -EINVAL; return genpd_remove_device(genpd, dev); @@ -1696,6 +1689,9 @@ int pm_genpd_init(struct generic_pm_domain *genpd, return ret; } + device_initialize(&genpd->dev); + dev_set_name(&genpd->dev, "%s", genpd->name); + mutex_lock(&gpd_list_lock); list_add(&genpd->gpd_list_node, &gpd_list); mutex_unlock(&gpd_list_lock); @@ -1892,14 +1888,33 @@ int of_genpd_add_provider_simple(struct device_node *np, mutex_lock(&gpd_list_lock); - if (genpd_present(genpd)) { - ret = genpd_add_provider(np, genpd_xlate_simple, genpd); - if (!ret) { - genpd->provider = &np->fwnode; - genpd->has_provider = true; + if (!genpd_present(genpd)) + goto unlock; + + genpd->dev.of_node = np; + + /* Parse genpd OPP table */ + if (genpd->set_performance_state) { + ret = dev_pm_opp_of_add_table(&genpd->dev); + if (ret) { + dev_err(&genpd->dev, "Failed to add OPP table: %d\n", + ret); + goto unlock; } } + ret = genpd_add_provider(np, genpd_xlate_simple, genpd); + if (ret) { + if (genpd->set_performance_state) + dev_pm_opp_of_remove_table(&genpd->dev); + + goto unlock; + } + + genpd->provider = &np->fwnode; + genpd->has_provider = true; + +unlock: mutex_unlock(&gpd_list_lock); return ret; @@ -1914,6 +1929,7 @@ EXPORT_SYMBOL_GPL(of_genpd_add_provider_simple); int of_genpd_add_provider_onecell(struct device_node *np, struct genpd_onecell_data *data) { + struct generic_pm_domain *genpd; unsigned int i; int ret = -EINVAL; @@ -1926,13 +1942,27 @@ int of_genpd_add_provider_onecell(struct device_node *np, data->xlate = genpd_xlate_onecell; for (i = 0; i < data->num_domains; i++) { - if (!data->domains[i]) + genpd = data->domains[i]; + + if (!genpd) continue; - if (!genpd_present(data->domains[i])) + if (!genpd_present(genpd)) goto error; - data->domains[i]->provider = &np->fwnode; - data->domains[i]->has_provider = true; + genpd->dev.of_node = np; + + /* Parse genpd OPP table */ + if (genpd->set_performance_state) { + ret = dev_pm_opp_of_add_table_indexed(&genpd->dev, i); + if (ret) { + dev_err(&genpd->dev, "Failed to add OPP table for index %d: %d\n", + i, ret); + goto error; + } + } + + genpd->provider = &np->fwnode; + genpd->has_provider = true; } ret = genpd_add_provider(np, data->xlate, data); @@ -1945,10 +1975,16 @@ int of_genpd_add_provider_onecell(struct device_node *np, error: while (i--) { - if (!data->domains[i]) + genpd = data->domains[i]; + + if (!genpd) continue; - data->domains[i]->provider = NULL; - data->domains[i]->has_provider = false; + + genpd->provider = NULL; + genpd->has_provider = false; + + if (genpd->set_performance_state) + dev_pm_opp_of_remove_table(&genpd->dev); } mutex_unlock(&gpd_list_lock); @@ -1975,10 +2011,17 @@ void of_genpd_del_provider(struct device_node *np) * provider, set the 'has_provider' to false * so that the PM domain can be safely removed. */ - list_for_each_entry(gpd, &gpd_list, gpd_list_node) - if (gpd->provider == &np->fwnode) + list_for_each_entry(gpd, &gpd_list, gpd_list_node) { + if (gpd->provider == &np->fwnode) { gpd->has_provider = false; + if (!gpd->set_performance_state) + continue; + + dev_pm_opp_of_remove_table(&gpd->dev); + } + } + list_del(&cp->link); of_node_put(cp->node); kfree(cp); @@ -2128,6 +2171,15 @@ struct generic_pm_domain *of_genpd_remove_last(struct device_node *np) } EXPORT_SYMBOL_GPL(of_genpd_remove_last); +static void genpd_release_dev(struct device *dev) +{ + kfree(dev); +} + +static struct bus_type genpd_bus_type = { + .name = "genpd", +}; + /** * genpd_dev_pm_detach - Detach a device from its PM domain. * @dev: Device to detach. @@ -2165,6 +2217,10 @@ static void genpd_dev_pm_detach(struct device *dev, bool power_off) /* Check if PM domain can be powered off after removing this device. */ genpd_queue_power_off_work(pd); + + /* Unregister the device if it was created by genpd. */ + if (dev->bus == &genpd_bus_type) + device_unregister(dev); } static void genpd_dev_pm_sync(struct device *dev) @@ -2178,36 +2234,15 @@ static void genpd_dev_pm_sync(struct device *dev) genpd_queue_power_off_work(pd); } -/** - * genpd_dev_pm_attach - Attach a device to its PM domain using DT. - * @dev: Device to attach. - * - * Parse device's OF node to find a PM domain specifier. If such is found, - * attaches the device to retrieved pm_domain ops. - * - * Both generic and legacy Samsung-specific DT bindings are supported to keep - * backwards compatibility with existing DTBs. - * - * Returns 0 on successfully attached PM domain or negative error code. Note - * that if a power-domain exists for the device, but it cannot be found or - * turned on, then return -EPROBE_DEFER to ensure that the device is not - * probed and to re-try again later. - */ -int genpd_dev_pm_attach(struct device *dev) +static int __genpd_dev_pm_attach(struct device *dev, struct device_node *np, + unsigned int index) { struct of_phandle_args pd_args; struct generic_pm_domain *pd; - unsigned int i; int ret; - if (!dev->of_node) - return -ENODEV; - - if (dev->pm_domain) - return -EEXIST; - - ret = of_parse_phandle_with_args(dev->of_node, "power-domains", - "#power-domain-cells", 0, &pd_args); + ret = of_parse_phandle_with_args(np, "power-domains", + "#power-domain-cells", index, &pd_args); if (ret < 0) return ret; @@ -2223,21 +2258,14 @@ int genpd_dev_pm_attach(struct device *dev) dev_dbg(dev, "adding to PM domain %s\n", pd->name); - for (i = 1; i < GENPD_RETRY_MAX_MS; i <<= 1) { - ret = genpd_add_device(pd, dev, NULL); - if (ret != -EAGAIN) - break; - - mdelay(i); - cond_resched(); - } + ret = genpd_add_device(pd, dev, NULL); mutex_unlock(&gpd_list_lock); if (ret < 0) { if (ret != -EPROBE_DEFER) dev_err(dev, "failed to add to PM domain %s: %d", pd->name, ret); - goto out; + return ret; } dev->pm_domain->detach = genpd_dev_pm_detach; @@ -2246,11 +2274,104 @@ int genpd_dev_pm_attach(struct device *dev) genpd_lock(pd); ret = genpd_power_on(pd, 0); genpd_unlock(pd); -out: - return ret ? -EPROBE_DEFER : 0; + + if (ret) + genpd_remove_device(pd, dev); + + return ret ? -EPROBE_DEFER : 1; +} + +/** + * genpd_dev_pm_attach - Attach a device to its PM domain using DT. + * @dev: Device to attach. + * + * Parse device's OF node to find a PM domain specifier. If such is found, + * attaches the device to retrieved pm_domain ops. + * + * Returns 1 on successfully attached PM domain, 0 when the device don't need a + * PM domain or when multiple power-domains exists for it, else a negative error + * code. Note that if a power-domain exists for the device, but it cannot be + * found or turned on, then return -EPROBE_DEFER to ensure that the device is + * not probed and to re-try again later. + */ +int genpd_dev_pm_attach(struct device *dev) +{ + if (!dev->of_node) + return 0; + + /* + * Devices with multiple PM domains must be attached separately, as we + * can only attach one PM domain per device. + */ + if (of_count_phandle_with_args(dev->of_node, "power-domains", + "#power-domain-cells") != 1) + return 0; + + return __genpd_dev_pm_attach(dev, dev->of_node, 0); } EXPORT_SYMBOL_GPL(genpd_dev_pm_attach); +/** + * genpd_dev_pm_attach_by_id - Associate a device with one of its PM domains. + * @dev: The device used to lookup the PM domain. + * @index: The index of the PM domain. + * + * Parse device's OF node to find a PM domain specifier at the provided @index. + * If such is found, creates a virtual device and attaches it to the retrieved + * pm_domain ops. To deal with detaching of the virtual device, the ->detach() + * callback in the struct dev_pm_domain are assigned to genpd_dev_pm_detach(). + * + * Returns the created virtual device if successfully attached PM domain, NULL + * when the device don't need a PM domain, else an ERR_PTR() in case of + * failures. If a power-domain exists for the device, but cannot be found or + * turned on, then ERR_PTR(-EPROBE_DEFER) is returned to ensure that the device + * is not probed and to re-try again later. + */ +struct device *genpd_dev_pm_attach_by_id(struct device *dev, + unsigned int index) +{ + struct device *genpd_dev; + int num_domains; + int ret; + + if (!dev->of_node) + return NULL; + + /* Deal only with devices using multiple PM domains. */ + num_domains = of_count_phandle_with_args(dev->of_node, "power-domains", + "#power-domain-cells"); + if (num_domains < 2 || index >= num_domains) + return NULL; + + /* Allocate and register device on the genpd bus. */ + genpd_dev = kzalloc(sizeof(*genpd_dev), GFP_KERNEL); + if (!genpd_dev) + return ERR_PTR(-ENOMEM); + + dev_set_name(genpd_dev, "genpd:%u:%s", index, dev_name(dev)); + genpd_dev->bus = &genpd_bus_type; + genpd_dev->release = genpd_release_dev; + + ret = device_register(genpd_dev); + if (ret) { + kfree(genpd_dev); + return ERR_PTR(ret); + } + + /* Try to attach the device to the PM domain at the specified index. */ + ret = __genpd_dev_pm_attach(genpd_dev, dev->of_node, index); + if (ret < 1) { + device_unregister(genpd_dev); + return ret ? ERR_PTR(ret) : NULL; + } + + pm_runtime_set_active(genpd_dev); + pm_runtime_enable(genpd_dev); + + return genpd_dev; +} +EXPORT_SYMBOL_GPL(genpd_dev_pm_attach_by_id); + static const struct of_device_id idle_state_match[] = { { .compatible = "domain-idle-state", }, { } @@ -2361,6 +2482,61 @@ int of_genpd_parse_idle_states(struct device_node *dn, } EXPORT_SYMBOL_GPL(of_genpd_parse_idle_states); +/** + * of_genpd_opp_to_performance_state- Gets performance state of device's + * power domain corresponding to a DT node's "required-opps" property. + * + * @dev: Device for which the performance-state needs to be found. + * @opp_node: DT node where the "required-opps" property is present. This can be + * the device node itself (if it doesn't have an OPP table) or a node + * within the OPP table of a device (if device has an OPP table). + * @state: Pointer to return performance state. + * + * Returns performance state corresponding to the "required-opps" property of + * a DT node. This calls platform specific genpd->opp_to_performance_state() + * callback to translate power domain OPP to performance state. + * + * Returns performance state on success and 0 on failure. + */ +unsigned int of_genpd_opp_to_performance_state(struct device *dev, + struct device_node *opp_node) +{ + struct generic_pm_domain *genpd; + struct dev_pm_opp *opp; + int state = 0; + + genpd = dev_to_genpd(dev); + if (IS_ERR(genpd)) + return 0; + + if (unlikely(!genpd->set_performance_state)) + return 0; + + genpd_lock(genpd); + + opp = of_dev_pm_opp_find_required_opp(&genpd->dev, opp_node); + if (IS_ERR(opp)) { + dev_err(dev, "Failed to find required OPP: %ld\n", + PTR_ERR(opp)); + goto unlock; + } + + state = genpd->opp_to_performance_state(genpd, opp); + dev_pm_opp_put(opp); + +unlock: + genpd_unlock(genpd); + + return state; +} +EXPORT_SYMBOL_GPL(of_genpd_opp_to_performance_state); + +static int __init genpd_bus_init(void) +{ + return bus_register(&genpd_bus_type); +} +core_initcall(genpd_bus_init); + #endif /* CONFIG_PM_GENERIC_DOMAINS_OF */ @@ -2628,6 +2804,19 @@ static int genpd_devices_show(struct seq_file *s, void *data) return ret; } +static int genpd_perf_state_show(struct seq_file *s, void *data) +{ + struct generic_pm_domain *genpd = s->private; + + if (genpd_lock_interruptible(genpd)) + return -ERESTARTSYS; + + seq_printf(s, "%u\n", genpd->performance_state); + + genpd_unlock(genpd); + return 0; +} + #define define_genpd_open_function(name) \ static int genpd_##name##_open(struct inode *inode, struct file *file) \ { \ @@ -2641,6 +2830,7 @@ define_genpd_open_function(idle_states); define_genpd_open_function(active_time); define_genpd_open_function(total_idle_time); define_genpd_open_function(devices); +define_genpd_open_function(perf_state); #define define_genpd_debugfs_fops(name) \ static const struct file_operations genpd_##name##_fops = { \ @@ -2657,6 +2847,7 @@ define_genpd_debugfs_fops(idle_states); define_genpd_debugfs_fops(active_time); define_genpd_debugfs_fops(total_idle_time); define_genpd_debugfs_fops(devices); +define_genpd_debugfs_fops(perf_state); static int __init genpd_debug_init(void) { @@ -2690,6 +2881,9 @@ static int __init genpd_debug_init(void) d, genpd, &genpd_total_idle_time_fops); debugfs_create_file("devices", 0444, d, genpd, &genpd_devices_fops); + if (genpd->set_performance_state) + debugfs_create_file("perf_state", 0444, + d, genpd, &genpd_perf_state_fops); } return 0; diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c index e5e067091572..3f68e2919dc5 100644 --- a/drivers/base/power/main.c +++ b/drivers/base/power/main.c @@ -192,34 +192,31 @@ void device_pm_move_last(struct device *dev) list_move_tail(&dev->power.entry, &dpm_list); } -static ktime_t initcall_debug_start(struct device *dev) +static ktime_t initcall_debug_start(struct device *dev, void *cb) { - ktime_t calltime = 0; - - if (pm_print_times_enabled) { - pr_info("calling %s+ @ %i, parent: %s\n", - dev_name(dev), task_pid_nr(current), - dev->parent ? dev_name(dev->parent) : "none"); - calltime = ktime_get(); - } + if (!pm_print_times_enabled) + return 0; - return calltime; + dev_info(dev, "calling %pF @ %i, parent: %s\n", cb, + task_pid_nr(current), + dev->parent ? dev_name(dev->parent) : "none"); + return ktime_get(); } static void initcall_debug_report(struct device *dev, ktime_t calltime, - int error, pm_message_t state, - const char *info) + void *cb, int error) { ktime_t rettime; s64 nsecs; + if (!pm_print_times_enabled) + return; + rettime = ktime_get(); nsecs = (s64) ktime_to_ns(ktime_sub(rettime, calltime)); - if (pm_print_times_enabled) { - pr_info("call %s+ returned %d after %Ld usecs\n", dev_name(dev), - error, (unsigned long long)nsecs >> 10); - } + dev_info(dev, "%pF returned %d after %Ld usecs\n", cb, error, + (unsigned long long)nsecs >> 10); } /** @@ -446,7 +443,7 @@ static int dpm_run_callback(pm_callback_t cb, struct device *dev, if (!cb) return 0; - calltime = initcall_debug_start(dev); + calltime = initcall_debug_start(dev, cb); pm_dev_dbg(dev, state, info); trace_device_pm_callback_start(dev, info, state.event); @@ -454,7 +451,7 @@ static int dpm_run_callback(pm_callback_t cb, struct device *dev, trace_device_pm_callback_end(dev, error); suspend_report_result(cb, error); - initcall_debug_report(dev, calltime, error, state, info); + initcall_debug_report(dev, calltime, cb, error); return error; } @@ -1664,14 +1661,14 @@ static int legacy_suspend(struct device *dev, pm_message_t state, int error; ktime_t calltime; - calltime = initcall_debug_start(dev); + calltime = initcall_debug_start(dev, cb); trace_device_pm_callback_start(dev, info, state.event); error = cb(dev, state); trace_device_pm_callback_end(dev, error); suspend_report_result(cb, error); - initcall_debug_report(dev, calltime, error, state, info); + initcall_debug_report(dev, calltime, cb, error); return error; } diff --git a/drivers/base/power/power.h b/drivers/base/power/power.h index 86e67e70b509..c511def48b48 100644 --- a/drivers/base/power/power.h +++ b/drivers/base/power/power.h @@ -56,14 +56,6 @@ static inline void device_wakeup_detach_irq(struct device *dev) { } -static inline void device_wakeup_arm_wake_irqs(void) -{ -} - -static inline void device_wakeup_disarm_wake_irqs(void) -{ -} - #endif /* CONFIG_PM_SLEEP */ /* @@ -95,28 +87,6 @@ static inline void pm_runtime_remove(struct device *dev) {} static inline int dpm_sysfs_add(struct device *dev) { return 0; } static inline void dpm_sysfs_remove(struct device *dev) {} -static inline void rpm_sysfs_remove(struct device *dev) {} -static inline int wakeup_sysfs_add(struct device *dev) { return 0; } -static inline void wakeup_sysfs_remove(struct device *dev) {} -static inline int pm_qos_sysfs_add(struct device *dev) { return 0; } -static inline void pm_qos_sysfs_remove(struct device *dev) {} - -static inline void dev_pm_arm_wake_irq(struct wake_irq *wirq) -{ -} - -static inline void dev_pm_disarm_wake_irq(struct wake_irq *wirq) -{ -} - -static inline void dev_pm_enable_wake_irq_check(struct device *dev, - bool can_change_status) -{ -} - -static inline void dev_pm_disable_wake_irq_check(struct device *dev) -{ -} #endif diff --git a/drivers/base/power/runtime.c b/drivers/base/power/runtime.c index 8bef3cb2424d..beb85c31f3fa 100644 --- a/drivers/base/power/runtime.c +++ b/drivers/base/power/runtime.c @@ -1607,6 +1607,8 @@ void pm_runtime_new_link(struct device *dev) void pm_runtime_drop_link(struct device *dev) { + rpm_put_suppliers(dev); + spin_lock_irq(&dev->power.lock); WARN_ON(dev->power.links_count == 0); dev->power.links_count--; diff --git a/drivers/base/power/sysfs.c b/drivers/base/power/sysfs.c index 0f651efc58a1..d713738ce796 100644 --- a/drivers/base/power/sysfs.c +++ b/drivers/base/power/sysfs.c @@ -353,7 +353,7 @@ static ssize_t wakeup_count_show(struct device *dev, spin_lock_irq(&dev->power.lock); if (dev->power.wakeup) { - count = dev->power.wakeup->event_count; + count = dev->power.wakeup->wakeup_count; enabled = true; } spin_unlock_irq(&dev->power.lock); diff --git a/drivers/base/power/wakeup.c b/drivers/base/power/wakeup.c index ea01621ed769..5fa1898755a3 100644 --- a/drivers/base/power/wakeup.c +++ b/drivers/base/power/wakeup.c @@ -57,7 +57,7 @@ static void split_counters(unsigned int *cnt, unsigned int *inpr) /* A preserved old value of the events counter. */ static unsigned int saved_count; -static DEFINE_SPINLOCK(events_lock); +static DEFINE_RAW_SPINLOCK(events_lock); static void pm_wakeup_timer_fn(struct timer_list *t); @@ -183,11 +183,10 @@ void wakeup_source_add(struct wakeup_source *ws) spin_lock_init(&ws->lock); timer_setup(&ws->timer, pm_wakeup_timer_fn, 0); ws->active = false; - ws->last_time = ktime_get(); - spin_lock_irqsave(&events_lock, flags); + raw_spin_lock_irqsave(&events_lock, flags); list_add_rcu(&ws->entry, &wakeup_sources); - spin_unlock_irqrestore(&events_lock, flags); + raw_spin_unlock_irqrestore(&events_lock, flags); } EXPORT_SYMBOL_GPL(wakeup_source_add); @@ -202,9 +201,9 @@ void wakeup_source_remove(struct wakeup_source *ws) if (WARN_ON(!ws)) return; - spin_lock_irqsave(&events_lock, flags); + raw_spin_lock_irqsave(&events_lock, flags); list_del_rcu(&ws->entry); - spin_unlock_irqrestore(&events_lock, flags); + raw_spin_unlock_irqrestore(&events_lock, flags); synchronize_srcu(&wakeup_srcu); } EXPORT_SYMBOL_GPL(wakeup_source_remove); @@ -843,7 +842,7 @@ bool pm_wakeup_pending(void) unsigned long flags; bool ret = false; - spin_lock_irqsave(&events_lock, flags); + raw_spin_lock_irqsave(&events_lock, flags); if (events_check_enabled) { unsigned int cnt, inpr; @@ -851,10 +850,10 @@ bool pm_wakeup_pending(void) ret = (cnt != saved_count || inpr > 0); events_check_enabled = !ret; } - spin_unlock_irqrestore(&events_lock, flags); + raw_spin_unlock_irqrestore(&events_lock, flags); if (ret) { - pr_info("PM: Wakeup pending, aborting suspend\n"); + pr_debug("PM: Wakeup pending, aborting suspend\n"); pm_print_active_wakeup_sources(); } @@ -940,13 +939,13 @@ bool pm_save_wakeup_count(unsigned int count) unsigned long flags; events_check_enabled = false; - spin_lock_irqsave(&events_lock, flags); + raw_spin_lock_irqsave(&events_lock, flags); split_counters(&cnt, &inpr); if (cnt == count && inpr == 0) { saved_count = count; events_check_enabled = true; } - spin_unlock_irqrestore(&events_lock, flags); + raw_spin_unlock_irqrestore(&events_lock, flags); return events_check_enabled; } @@ -1029,32 +1028,75 @@ static int print_wakeup_source_stats(struct seq_file *m, return 0; } -/** - * wakeup_sources_stats_show - Print wakeup sources statistics information. - * @m: seq_file to print the statistics into. - */ -static int wakeup_sources_stats_show(struct seq_file *m, void *unused) +static void *wakeup_sources_stats_seq_start(struct seq_file *m, + loff_t *pos) { struct wakeup_source *ws; - int srcuidx; + loff_t n = *pos; + int *srcuidx = m->private; - seq_puts(m, "name\t\tactive_count\tevent_count\twakeup_count\t" - "expire_count\tactive_since\ttotal_time\tmax_time\t" - "last_change\tprevent_suspend_time\n"); + if (n == 0) { + seq_puts(m, "name\t\tactive_count\tevent_count\twakeup_count\t" + "expire_count\tactive_since\ttotal_time\tmax_time\t" + "last_change\tprevent_suspend_time\n"); + } - srcuidx = srcu_read_lock(&wakeup_srcu); - list_for_each_entry_rcu(ws, &wakeup_sources, entry) - print_wakeup_source_stats(m, ws); - srcu_read_unlock(&wakeup_srcu, srcuidx); + *srcuidx = srcu_read_lock(&wakeup_srcu); + list_for_each_entry_rcu(ws, &wakeup_sources, entry) { + if (n-- <= 0) + return ws; + } + + return NULL; +} + +static void *wakeup_sources_stats_seq_next(struct seq_file *m, + void *v, loff_t *pos) +{ + struct wakeup_source *ws = v; + struct wakeup_source *next_ws = NULL; - print_wakeup_source_stats(m, &deleted_ws); + ++(*pos); + + list_for_each_entry_continue_rcu(ws, &wakeup_sources, entry) { + next_ws = ws; + break; + } + + return next_ws; +} + +static void wakeup_sources_stats_seq_stop(struct seq_file *m, void *v) +{ + int *srcuidx = m->private; + + srcu_read_unlock(&wakeup_srcu, *srcuidx); +} + +/** + * wakeup_sources_stats_seq_show - Print wakeup sources statistics information. + * @m: seq_file to print the statistics into. + * @v: wakeup_source of each iteration + */ +static int wakeup_sources_stats_seq_show(struct seq_file *m, void *v) +{ + struct wakeup_source *ws = v; + + print_wakeup_source_stats(m, ws); return 0; } +static const struct seq_operations wakeup_sources_stats_seq_ops = { + .start = wakeup_sources_stats_seq_start, + .next = wakeup_sources_stats_seq_next, + .stop = wakeup_sources_stats_seq_stop, + .show = wakeup_sources_stats_seq_show, +}; + static int wakeup_sources_stats_open(struct inode *inode, struct file *file) { - return single_open(file, wakeup_sources_stats_show, NULL); + return seq_open_private(file, &wakeup_sources_stats_seq_ops, sizeof(int)); } static const struct file_operations wakeup_sources_stats_fops = { @@ -1062,7 +1104,7 @@ static const struct file_operations wakeup_sources_stats_fops = { .open = wakeup_sources_stats_open, .read = seq_read, .llseek = seq_lseek, - .release = single_release, + .release = seq_release_private, }; static int __init wakeup_sources_debugfs_init(void) diff --git a/drivers/base/property.c b/drivers/base/property.c index 8f205f6461ed..240ab5230ff6 100644 --- a/drivers/base/property.c +++ b/drivers/base/property.c @@ -56,6 +56,72 @@ pset_prop_get(const struct property_set *pset, const char *name) return NULL; } +static const void *property_get_pointer(const struct property_entry *prop) +{ + switch (prop->type) { + case DEV_PROP_U8: + if (prop->is_array) + return prop->pointer.u8_data; + return &prop->value.u8_data; + case DEV_PROP_U16: + if (prop->is_array) + return prop->pointer.u16_data; + return &prop->value.u16_data; + case DEV_PROP_U32: + if (prop->is_array) + return prop->pointer.u32_data; + return &prop->value.u32_data; + case DEV_PROP_U64: + if (prop->is_array) + return prop->pointer.u64_data; + return &prop->value.u64_data; + case DEV_PROP_STRING: + if (prop->is_array) + return prop->pointer.str; + return &prop->value.str; + default: + return NULL; + } +} + +static void property_set_pointer(struct property_entry *prop, const void *pointer) +{ + switch (prop->type) { + case DEV_PROP_U8: + if (prop->is_array) + prop->pointer.u8_data = pointer; + else + prop->value.u8_data = *((u8 *)pointer); + break; + case DEV_PROP_U16: + if (prop->is_array) + prop->pointer.u16_data = pointer; + else + prop->value.u16_data = *((u16 *)pointer); + break; + case DEV_PROP_U32: + if (prop->is_array) + prop->pointer.u32_data = pointer; + else + prop->value.u32_data = *((u32 *)pointer); + break; + case DEV_PROP_U64: + if (prop->is_array) + prop->pointer.u64_data = pointer; + else + prop->value.u64_data = *((u64 *)pointer); + break; + case DEV_PROP_STRING: + if (prop->is_array) + prop->pointer.str = pointer; + else + prop->value.str = pointer; + break; + default: + break; + } +} + static const void *pset_prop_find(const struct property_set *pset, const char *propname, size_t length) { @@ -65,10 +131,7 @@ static const void *pset_prop_find(const struct property_set *pset, prop = pset_prop_get(pset, propname); if (!prop) return ERR_PTR(-EINVAL); - if (prop->is_array) - pointer = prop->pointer.raw_data; - else - pointer = &prop->value.raw_data; + pointer = property_get_pointer(prop); if (!pointer) return ERR_PTR(-ENODATA); if (length > prop->length) @@ -698,16 +761,17 @@ EXPORT_SYMBOL_GPL(fwnode_property_get_reference_args); static void property_entry_free_data(const struct property_entry *p) { + const void *pointer = property_get_pointer(p); size_t i, nval; if (p->is_array) { - if (p->is_string && p->pointer.str) { + if (p->type == DEV_PROP_STRING && p->pointer.str) { nval = p->length / sizeof(const char *); for (i = 0; i < nval; i++) kfree(p->pointer.str[i]); } - kfree(p->pointer.raw_data); - } else if (p->is_string) { + kfree(pointer); + } else if (p->type == DEV_PROP_STRING) { kfree(p->value.str); } kfree(p->name); @@ -716,7 +780,7 @@ static void property_entry_free_data(const struct property_entry *p) static int property_copy_string_array(struct property_entry *dst, const struct property_entry *src) { - char **d; + const char **d; size_t nval = src->length / sizeof(*d); int i; @@ -734,40 +798,44 @@ static int property_copy_string_array(struct property_entry *dst, } } - dst->pointer.raw_data = d; + dst->pointer.str = d; return 0; } static int property_entry_copy_data(struct property_entry *dst, const struct property_entry *src) { + const void *pointer = property_get_pointer(src); + const void *new; int error; if (src->is_array) { if (!src->length) return -ENODATA; - if (src->is_string) { + if (src->type == DEV_PROP_STRING) { error = property_copy_string_array(dst, src); if (error) return error; + new = dst->pointer.str; } else { - dst->pointer.raw_data = kmemdup(src->pointer.raw_data, - src->length, GFP_KERNEL); - if (!dst->pointer.raw_data) + new = kmemdup(pointer, src->length, GFP_KERNEL); + if (!new) return -ENOMEM; } - } else if (src->is_string) { - dst->value.str = kstrdup(src->value.str, GFP_KERNEL); - if (!dst->value.str && src->value.str) + } else if (src->type == DEV_PROP_STRING) { + new = kstrdup(src->value.str, GFP_KERNEL); + if (!new && src->value.str) return -ENOMEM; } else { - dst->value.raw_data = src->value.raw_data; + new = pointer; } dst->length = src->length; dst->is_array = src->is_array; - dst->is_string = src->is_string; + dst->type = src->type; + + property_set_pointer(dst, new); dst->name = kstrdup(src->name, GFP_KERNEL); if (!dst->name) diff --git a/drivers/base/regmap/regmap-mmio.c b/drivers/base/regmap/regmap-mmio.c index 5cadfd3394d8..8741fb5f8f54 100644 --- a/drivers/base/regmap/regmap-mmio.c +++ b/drivers/base/regmap/regmap-mmio.c @@ -206,7 +206,8 @@ static void regmap_mmio_free_context(void *context) if (!IS_ERR(ctx->clk)) { clk_unprepare(ctx->clk); - clk_put(ctx->clk); + if (!ctx->attached_clk) + clk_put(ctx->clk); } kfree(context); } diff --git a/drivers/base/regmap/regmap-slimbus.c b/drivers/base/regmap/regmap-slimbus.c index c90bee81d954..91d501eda8a9 100644 --- a/drivers/base/regmap/regmap-slimbus.c +++ b/drivers/base/regmap/regmap-slimbus.c @@ -41,7 +41,7 @@ static struct regmap_bus regmap_slimbus_bus = { static const struct regmap_bus *regmap_get_slimbus(struct slim_device *slim, const struct regmap_config *config) { - if (config->val_bits == 8 && config->reg_bits == 8) + if (config->val_bits == 8 && config->reg_bits == 16) return ®map_slimbus_bus; return ERR_PTR(-ENOTSUPP); diff --git a/drivers/bcma/main.c b/drivers/bcma/main.c index e6986c7608f1..fc1f4acdd189 100644 --- a/drivers/bcma/main.c +++ b/drivers/bcma/main.c @@ -207,7 +207,7 @@ static void bcma_of_fill_device(struct device *parent, core->irq = bcma_of_get_irq(parent, core, 0); - of_dma_configure(&core->dev, node); + of_dma_configure(&core->dev, node, false); } unsigned int bcma_core_irq(struct bcma_device *core, int num) diff --git a/drivers/block/DAC960.c b/drivers/block/DAC960.c index f781eff7d23e..f6518067aa7d 100644 --- a/drivers/block/DAC960.c +++ b/drivers/block/DAC960.c @@ -1179,7 +1179,6 @@ static bool DAC960_V1_EnableMemoryMailboxInterface(DAC960_Controller_T if (pci_set_dma_mask(Controller->PCIDevice, DMA_BIT_MASK(32))) return DAC960_Failure(Controller, "DMA mask out of range"); - Controller->BounceBufferLimit = DMA_BIT_MASK(32); if ((hw_type == DAC960_PD_Controller) || (hw_type == DAC960_P_Controller)) { CommandMailboxesSize = 0; @@ -1380,11 +1379,8 @@ static bool DAC960_V2_EnableMemoryMailboxInterface(DAC960_Controller_T dma_addr_t CommandMailboxDMA; DAC960_V2_CommandStatus_T CommandStatus; - if (!pci_set_dma_mask(Controller->PCIDevice, DMA_BIT_MASK(64))) - Controller->BounceBufferLimit = DMA_BIT_MASK(64); - else if (!pci_set_dma_mask(Controller->PCIDevice, DMA_BIT_MASK(32))) - Controller->BounceBufferLimit = DMA_BIT_MASK(32); - else + if (pci_set_dma_mask(Controller->PCIDevice, DMA_BIT_MASK(64)) && + pci_set_dma_mask(Controller->PCIDevice, DMA_BIT_MASK(32))) return DAC960_Failure(Controller, "DMA mask out of range"); /* This is a temporary dma mapping, used only in the scope of this function */ @@ -2540,7 +2536,6 @@ static bool DAC960_RegisterBlockDevice(DAC960_Controller_T *Controller) continue; } Controller->RequestQueue[n] = RequestQueue; - blk_queue_bounce_limit(RequestQueue, Controller->BounceBufferLimit); RequestQueue->queuedata = Controller; blk_queue_max_segments(RequestQueue, Controller->DriverScatterGatherLimit); blk_queue_max_hw_sectors(RequestQueue, Controller->MaxBlocksPerCommand); @@ -5724,8 +5719,8 @@ static bool DAC960_CheckStatusBuffer(DAC960_Controller_T *Controller, Controller->CombinedStatusBufferLength = NewStatusBufferLength; return true; } - NewStatusBuffer = kmalloc(2 * Controller->CombinedStatusBufferLength, - GFP_ATOMIC); + NewStatusBuffer = kmalloc_array(2, Controller->CombinedStatusBufferLength, + GFP_ATOMIC); if (NewStatusBuffer == NULL) { DAC960_Warning("Unable to expand Combined Status Buffer - Truncating\n", @@ -6451,19 +6446,6 @@ static int dac960_proc_show(struct seq_file *m, void *v) return 0; } -static int dac960_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, dac960_proc_show, NULL); -} - -static const struct file_operations dac960_proc_fops = { - .owner = THIS_MODULE, - .open = dac960_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int dac960_initial_status_proc_show(struct seq_file *m, void *v) { DAC960_Controller_T *Controller = (DAC960_Controller_T *)m->private; @@ -6471,19 +6453,6 @@ static int dac960_initial_status_proc_show(struct seq_file *m, void *v) return 0; } -static int dac960_initial_status_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, dac960_initial_status_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations dac960_initial_status_proc_fops = { - .owner = THIS_MODULE, - .open = dac960_initial_status_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int dac960_current_status_proc_show(struct seq_file *m, void *v) { DAC960_Controller_T *Controller = (DAC960_Controller_T *) m->private; @@ -6517,19 +6486,6 @@ static int dac960_current_status_proc_show(struct seq_file *m, void *v) return 0; } -static int dac960_current_status_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, dac960_current_status_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations dac960_current_status_proc_fops = { - .owner = THIS_MODULE, - .open = dac960_current_status_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int dac960_user_command_proc_show(struct seq_file *m, void *v) { DAC960_Controller_T *Controller = (DAC960_Controller_T *)m->private; @@ -6584,17 +6540,19 @@ static void DAC960_CreateProcEntries(DAC960_Controller_T *Controller) if (DAC960_ProcDirectoryEntry == NULL) { DAC960_ProcDirectoryEntry = proc_mkdir("rd", NULL); - proc_create("status", 0, DAC960_ProcDirectoryEntry, - &dac960_proc_fops); + proc_create_single("status", 0, DAC960_ProcDirectoryEntry, + dac960_proc_show); } snprintf(Controller->ControllerName, sizeof(Controller->ControllerName), "c%d", Controller->ControllerNumber); ControllerProcEntry = proc_mkdir(Controller->ControllerName, DAC960_ProcDirectoryEntry); - proc_create_data("initial_status", 0, ControllerProcEntry, &dac960_initial_status_proc_fops, Controller); - proc_create_data("current_status", 0, ControllerProcEntry, &dac960_current_status_proc_fops, Controller); - proc_create_data("user_command", S_IWUSR | S_IRUSR, ControllerProcEntry, &dac960_user_command_proc_fops, Controller); + proc_create_single_data("initial_status", 0, ControllerProcEntry, + dac960_initial_status_proc_show, Controller); + proc_create_single_data("current_status", 0, ControllerProcEntry, + dac960_current_status_proc_show, Controller); + proc_create_data("user_command", 0600, ControllerProcEntry, &dac960_user_command_proc_fops, Controller); Controller->ControllerProcEntry = ControllerProcEntry; } diff --git a/drivers/block/DAC960.h b/drivers/block/DAC960.h index 21aff470d268..1439e651928b 100644 --- a/drivers/block/DAC960.h +++ b/drivers/block/DAC960.h @@ -2295,7 +2295,6 @@ typedef struct DAC960_Controller unsigned short MaxBlocksPerCommand; unsigned short ControllerScatterGatherLimit; unsigned short DriverScatterGatherLimit; - u64 BounceBufferLimit; unsigned int CombinedStatusBufferLength; unsigned int InitialStatusLength; unsigned int CurrentStatusLength; diff --git a/drivers/block/aoe/aoeblk.c b/drivers/block/aoe/aoeblk.c index 6797e6c23c8a..429ebb84b592 100644 --- a/drivers/block/aoe/aoeblk.c +++ b/drivers/block/aoe/aoeblk.c @@ -159,14 +159,14 @@ static int aoe_debugfs_open(struct inode *inode, struct file *file) return single_open(file, aoedisk_debugfs_show, inode->i_private); } -static DEVICE_ATTR(state, S_IRUGO, aoedisk_show_state, NULL); -static DEVICE_ATTR(mac, S_IRUGO, aoedisk_show_mac, NULL); -static DEVICE_ATTR(netif, S_IRUGO, aoedisk_show_netif, NULL); +static DEVICE_ATTR(state, 0444, aoedisk_show_state, NULL); +static DEVICE_ATTR(mac, 0444, aoedisk_show_mac, NULL); +static DEVICE_ATTR(netif, 0444, aoedisk_show_netif, NULL); static struct device_attribute dev_attr_firmware_version = { - .attr = { .name = "firmware-version", .mode = S_IRUGO }, + .attr = { .name = "firmware-version", .mode = 0444 }, .show = aoedisk_show_fwver, }; -static DEVICE_ATTR(payload, S_IRUGO, aoedisk_show_payload, NULL); +static DEVICE_ATTR(payload, 0444, aoedisk_show_payload, NULL); static struct attribute *aoe_attrs[] = { &dev_attr_state.attr, @@ -388,7 +388,6 @@ aoeblk_gdalloc(void *vp) d->aoemajor, d->aoeminor); goto err_mempool; } - blk_queue_bounce_limit(q, BLK_BOUNCE_HIGH); spin_lock_irqsave(&d->lock, flags); WARN_ON(!(d->flags & DEVFL_GD_NOW)); diff --git a/drivers/block/aoe/aoecmd.c b/drivers/block/aoe/aoecmd.c index 540bb60cd071..096882e54095 100644 --- a/drivers/block/aoe/aoecmd.c +++ b/drivers/block/aoe/aoecmd.c @@ -1032,8 +1032,9 @@ bvcpy(struct sk_buff *skb, struct bio *bio, struct bvec_iter iter, long cnt) iter.bi_size = cnt; __bio_for_each_segment(bv, bio, iter, iter) { - char *p = page_address(bv.bv_page) + bv.bv_offset; + char *p = kmap_atomic(bv.bv_page) + bv.bv_offset; skb_copy_bits(skb, soff, p, bv.bv_len); + kunmap_atomic(p); soff += bv.bv_len; } } diff --git a/drivers/block/brd.c b/drivers/block/brd.c index 66cb0f857f64..bb976598ee43 100644 --- a/drivers/block/brd.c +++ b/drivers/block/brd.c @@ -331,15 +331,15 @@ static const struct block_device_operations brd_fops = { * And now the modules code and kernel interface. */ static int rd_nr = CONFIG_BLK_DEV_RAM_COUNT; -module_param(rd_nr, int, S_IRUGO); +module_param(rd_nr, int, 0444); MODULE_PARM_DESC(rd_nr, "Maximum number of brd devices"); unsigned long rd_size = CONFIG_BLK_DEV_RAM_SIZE; -module_param(rd_size, ulong, S_IRUGO); +module_param(rd_size, ulong, 0444); MODULE_PARM_DESC(rd_size, "Size of each RAM disk in kbytes."); static int max_part = 1; -module_param(max_part, int, S_IRUGO); +module_param(max_part, int, 0444); MODULE_PARM_DESC(max_part, "Num Minors to reserve between devices"); MODULE_LICENSE("GPL"); @@ -402,6 +402,10 @@ static struct brd_device *brd_alloc(int i) set_capacity(disk, rd_size * 2); disk->queue->backing_dev_info->capabilities |= BDI_CAP_SYNCHRONOUS_IO; + /* Tell the block layer that this is not a rotational device */ + blk_queue_flag_set(QUEUE_FLAG_NONROT, disk->queue); + blk_queue_flag_clear(QUEUE_FLAG_ADD_RANDOM, disk->queue); + return brd; out_free_queue: diff --git a/drivers/block/drbd/drbd_bitmap.c b/drivers/block/drbd/drbd_bitmap.c index 9f4e6f502b84..11a85b740327 100644 --- a/drivers/block/drbd/drbd_bitmap.c +++ b/drivers/block/drbd/drbd_bitmap.c @@ -977,7 +977,7 @@ static void drbd_bm_endio(struct bio *bio) bm_page_unlock_io(device, idx); if (ctx->flags & BM_AIO_COPY_PAGES) - mempool_free(bio->bi_io_vec[0].bv_page, drbd_md_io_page_pool); + mempool_free(bio->bi_io_vec[0].bv_page, &drbd_md_io_page_pool); bio_put(bio); @@ -1014,7 +1014,8 @@ static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_ho bm_set_page_unchanged(b->bm_pages[page_nr]); if (ctx->flags & BM_AIO_COPY_PAGES) { - page = mempool_alloc(drbd_md_io_page_pool, __GFP_HIGHMEM|__GFP_RECLAIM); + page = mempool_alloc(&drbd_md_io_page_pool, + GFP_NOIO | __GFP_HIGHMEM); copy_highpage(page, b->bm_pages[page_nr]); bm_store_page_idx(page, page_nr); } else diff --git a/drivers/block/drbd/drbd_debugfs.c b/drivers/block/drbd/drbd_debugfs.c index ab21976a87b2..5d5e8d6a8a56 100644 --- a/drivers/block/drbd/drbd_debugfs.c +++ b/drivers/block/drbd/drbd_debugfs.c @@ -481,9 +481,9 @@ void drbd_debugfs_resource_add(struct drbd_resource *resource) goto fail; resource->debugfs_res_connections = dentry; - dentry = debugfs_create_file("in_flight_summary", S_IRUSR|S_IRGRP, - resource->debugfs_res, resource, - &in_flight_summary_fops); + dentry = debugfs_create_file("in_flight_summary", 0440, + resource->debugfs_res, resource, + &in_flight_summary_fops); if (IS_ERR_OR_NULL(dentry)) goto fail; resource->debugfs_res_in_flight_summary = dentry; @@ -645,16 +645,16 @@ void drbd_debugfs_connection_add(struct drbd_connection *connection) goto fail; connection->debugfs_conn = dentry; - dentry = debugfs_create_file("callback_history", S_IRUSR|S_IRGRP, - connection->debugfs_conn, connection, - &connection_callback_history_fops); + dentry = debugfs_create_file("callback_history", 0440, + connection->debugfs_conn, connection, + &connection_callback_history_fops); if (IS_ERR_OR_NULL(dentry)) goto fail; connection->debugfs_conn_callback_history = dentry; - dentry = debugfs_create_file("oldest_requests", S_IRUSR|S_IRGRP, - connection->debugfs_conn, connection, - &connection_oldest_requests_fops); + dentry = debugfs_create_file("oldest_requests", 0440, + connection->debugfs_conn, connection, + &connection_oldest_requests_fops); if (IS_ERR_OR_NULL(dentry)) goto fail; connection->debugfs_conn_oldest_requests = dentry; @@ -824,7 +824,7 @@ void drbd_debugfs_device_add(struct drbd_device *device) device->debugfs_minor = dentry; #define DCF(name) do { \ - dentry = debugfs_create_file(#name, S_IRUSR|S_IRGRP, \ + dentry = debugfs_create_file(#name, 0440, \ device->debugfs_vol, device, \ &device_ ## name ## _fops); \ if (IS_ERR_OR_NULL(dentry)) \ diff --git a/drivers/block/drbd/drbd_int.h b/drivers/block/drbd/drbd_int.h index 06ecee1b528e..bc4ed2ed40a2 100644 --- a/drivers/block/drbd/drbd_int.h +++ b/drivers/block/drbd/drbd_int.h @@ -1405,8 +1405,8 @@ extern struct kmem_cache *drbd_request_cache; extern struct kmem_cache *drbd_ee_cache; /* peer requests */ extern struct kmem_cache *drbd_bm_ext_cache; /* bitmap extents */ extern struct kmem_cache *drbd_al_ext_cache; /* activity log extents */ -extern mempool_t *drbd_request_mempool; -extern mempool_t *drbd_ee_mempool; +extern mempool_t drbd_request_mempool; +extern mempool_t drbd_ee_mempool; /* drbd's page pool, used to buffer data received from the peer, * or data requested by the peer. @@ -1432,16 +1432,16 @@ extern wait_queue_head_t drbd_pp_wait; * 128 should be plenty, currently we probably can get away with as few as 1. */ #define DRBD_MIN_POOL_PAGES 128 -extern mempool_t *drbd_md_io_page_pool; +extern mempool_t drbd_md_io_page_pool; /* We also need to make sure we get a bio * when we need it for housekeeping purposes */ -extern struct bio_set *drbd_md_io_bio_set; +extern struct bio_set drbd_md_io_bio_set; /* to allocate from that set */ extern struct bio *bio_alloc_drbd(gfp_t gfp_mask); /* And a bio_set for cloning */ -extern struct bio_set *drbd_io_bio_set; +extern struct bio_set drbd_io_bio_set; extern struct mutex resources_mutex; @@ -1643,7 +1643,7 @@ void drbd_bump_write_ordering(struct drbd_resource *resource, struct drbd_backin /* drbd_proc.c */ extern struct proc_dir_entry *drbd_proc; -extern const struct file_operations drbd_proc_fops; +int drbd_seq_show(struct seq_file *seq, void *v); /* drbd_actlog.c */ extern bool drbd_al_begin_io_prepare(struct drbd_device *device, struct drbd_interval *i); diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index 185f1ef00a7c..a80809bd3057 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -124,11 +124,11 @@ struct kmem_cache *drbd_request_cache; struct kmem_cache *drbd_ee_cache; /* peer requests */ struct kmem_cache *drbd_bm_ext_cache; /* bitmap extents */ struct kmem_cache *drbd_al_ext_cache; /* activity log extents */ -mempool_t *drbd_request_mempool; -mempool_t *drbd_ee_mempool; -mempool_t *drbd_md_io_page_pool; -struct bio_set *drbd_md_io_bio_set; -struct bio_set *drbd_io_bio_set; +mempool_t drbd_request_mempool; +mempool_t drbd_ee_mempool; +mempool_t drbd_md_io_page_pool; +struct bio_set drbd_md_io_bio_set; +struct bio_set drbd_io_bio_set; /* I do not use a standard mempool, because: 1) I want to hand out the pre-allocated objects first. @@ -153,10 +153,10 @@ struct bio *bio_alloc_drbd(gfp_t gfp_mask) { struct bio *bio; - if (!drbd_md_io_bio_set) + if (!bioset_initialized(&drbd_md_io_bio_set)) return bio_alloc(gfp_mask, 1); - bio = bio_alloc_bioset(gfp_mask, 1, drbd_md_io_bio_set); + bio = bio_alloc_bioset(gfp_mask, 1, &drbd_md_io_bio_set); if (!bio) return NULL; return bio; @@ -511,7 +511,8 @@ static void drbd_calc_cpu_mask(cpumask_var_t *cpu_mask) { unsigned int *resources_per_cpu, min_index = ~0; - resources_per_cpu = kzalloc(nr_cpu_ids * sizeof(*resources_per_cpu), GFP_KERNEL); + resources_per_cpu = kcalloc(nr_cpu_ids, sizeof(*resources_per_cpu), + GFP_KERNEL); if (resources_per_cpu) { struct drbd_resource *resource; unsigned int cpu, min = ~0; @@ -2097,16 +2098,11 @@ static void drbd_destroy_mempools(void) /* D_ASSERT(device, atomic_read(&drbd_pp_vacant)==0); */ - if (drbd_io_bio_set) - bioset_free(drbd_io_bio_set); - if (drbd_md_io_bio_set) - bioset_free(drbd_md_io_bio_set); - if (drbd_md_io_page_pool) - mempool_destroy(drbd_md_io_page_pool); - if (drbd_ee_mempool) - mempool_destroy(drbd_ee_mempool); - if (drbd_request_mempool) - mempool_destroy(drbd_request_mempool); + bioset_exit(&drbd_io_bio_set); + bioset_exit(&drbd_md_io_bio_set); + mempool_exit(&drbd_md_io_page_pool); + mempool_exit(&drbd_ee_mempool); + mempool_exit(&drbd_request_mempool); if (drbd_ee_cache) kmem_cache_destroy(drbd_ee_cache); if (drbd_request_cache) @@ -2116,11 +2112,6 @@ static void drbd_destroy_mempools(void) if (drbd_al_ext_cache) kmem_cache_destroy(drbd_al_ext_cache); - drbd_io_bio_set = NULL; - drbd_md_io_bio_set = NULL; - drbd_md_io_page_pool = NULL; - drbd_ee_mempool = NULL; - drbd_request_mempool = NULL; drbd_ee_cache = NULL; drbd_request_cache = NULL; drbd_bm_ext_cache = NULL; @@ -2133,18 +2124,7 @@ static int drbd_create_mempools(void) { struct page *page; const int number = (DRBD_MAX_BIO_SIZE/PAGE_SIZE) * drbd_minor_count; - int i; - - /* prepare our caches and mempools */ - drbd_request_mempool = NULL; - drbd_ee_cache = NULL; - drbd_request_cache = NULL; - drbd_bm_ext_cache = NULL; - drbd_al_ext_cache = NULL; - drbd_pp_pool = NULL; - drbd_md_io_page_pool = NULL; - drbd_md_io_bio_set = NULL; - drbd_io_bio_set = NULL; + int i, ret; /* caches */ drbd_request_cache = kmem_cache_create( @@ -2168,26 +2148,26 @@ static int drbd_create_mempools(void) goto Enomem; /* mempools */ - drbd_io_bio_set = bioset_create(BIO_POOL_SIZE, 0, 0); - if (drbd_io_bio_set == NULL) + ret = bioset_init(&drbd_io_bio_set, BIO_POOL_SIZE, 0, 0); + if (ret) goto Enomem; - drbd_md_io_bio_set = bioset_create(DRBD_MIN_POOL_PAGES, 0, - BIOSET_NEED_BVECS); - if (drbd_md_io_bio_set == NULL) + ret = bioset_init(&drbd_md_io_bio_set, DRBD_MIN_POOL_PAGES, 0, + BIOSET_NEED_BVECS); + if (ret) goto Enomem; - drbd_md_io_page_pool = mempool_create_page_pool(DRBD_MIN_POOL_PAGES, 0); - if (drbd_md_io_page_pool == NULL) + ret = mempool_init_page_pool(&drbd_md_io_page_pool, DRBD_MIN_POOL_PAGES, 0); + if (ret) goto Enomem; - drbd_request_mempool = mempool_create_slab_pool(number, - drbd_request_cache); - if (drbd_request_mempool == NULL) + ret = mempool_init_slab_pool(&drbd_request_mempool, number, + drbd_request_cache); + if (ret) goto Enomem; - drbd_ee_mempool = mempool_create_slab_pool(number, drbd_ee_cache); - if (drbd_ee_mempool == NULL) + ret = mempool_init_slab_pool(&drbd_ee_mempool, number, drbd_ee_cache); + if (ret) goto Enomem; /* drbd's page pool */ @@ -3010,7 +2990,7 @@ static int __init drbd_init(void) goto fail; err = -ENOMEM; - drbd_proc = proc_create_data("drbd", S_IFREG | S_IRUGO , NULL, &drbd_proc_fops, NULL); + drbd_proc = proc_create_single("drbd", S_IFREG | 0444 , NULL, drbd_seq_show); if (!drbd_proc) { pr_err("unable to register proc file\n"); goto fail; diff --git a/drivers/block/drbd/drbd_proc.c b/drivers/block/drbd/drbd_proc.c index 582caeb0de86..74ef29247bb5 100644 --- a/drivers/block/drbd/drbd_proc.c +++ b/drivers/block/drbd/drbd_proc.c @@ -33,18 +33,7 @@ #include #include "drbd_int.h" -static int drbd_proc_open(struct inode *inode, struct file *file); -static int drbd_proc_release(struct inode *inode, struct file *file); - - struct proc_dir_entry *drbd_proc; -const struct file_operations drbd_proc_fops = { - .owner = THIS_MODULE, - .open = drbd_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = drbd_proc_release, -}; static void seq_printf_with_thousands_grouping(struct seq_file *seq, long v) { @@ -235,7 +224,7 @@ static void drbd_syncer_progress(struct drbd_device *device, struct seq_file *se } } -static int drbd_seq_show(struct seq_file *seq, void *v) +int drbd_seq_show(struct seq_file *seq, void *v) { int i, prev_i = -1; const char *sn; @@ -345,24 +334,3 @@ static int drbd_seq_show(struct seq_file *seq, void *v) return 0; } - -static int drbd_proc_open(struct inode *inode, struct file *file) -{ - int err; - - if (try_module_get(THIS_MODULE)) { - err = single_open(file, drbd_seq_show, NULL); - if (err) - module_put(THIS_MODULE); - return err; - } - return -ENODEV; -} - -static int drbd_proc_release(struct inode *inode, struct file *file) -{ - module_put(THIS_MODULE); - return single_release(inode, file); -} - -/* PROC FS stuff end */ diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c index c72dee0ef083..be9450f5ad1c 100644 --- a/drivers/block/drbd/drbd_receiver.c +++ b/drivers/block/drbd/drbd_receiver.c @@ -378,7 +378,7 @@ drbd_alloc_peer_req(struct drbd_peer_device *peer_device, u64 id, sector_t secto if (drbd_insert_fault(device, DRBD_FAULT_AL_EE)) return NULL; - peer_req = mempool_alloc(drbd_ee_mempool, gfp_mask & ~__GFP_HIGHMEM); + peer_req = mempool_alloc(&drbd_ee_mempool, gfp_mask & ~__GFP_HIGHMEM); if (!peer_req) { if (!(gfp_mask & __GFP_NOWARN)) drbd_err(device, "%s: allocation failed\n", __func__); @@ -409,7 +409,7 @@ drbd_alloc_peer_req(struct drbd_peer_device *peer_device, u64 id, sector_t secto return peer_req; fail: - mempool_free(peer_req, drbd_ee_mempool); + mempool_free(peer_req, &drbd_ee_mempool); return NULL; } @@ -426,7 +426,7 @@ void __drbd_free_peer_req(struct drbd_device *device, struct drbd_peer_request * peer_req->flags &= ~EE_CALL_AL_COMPLETE_IO; drbd_al_complete_io(device, &peer_req->i); } - mempool_free(peer_req, drbd_ee_mempool); + mempool_free(peer_req, &drbd_ee_mempool); } int drbd_free_peer_reqs(struct drbd_device *device, struct list_head *list) diff --git a/drivers/block/drbd/drbd_req.c b/drivers/block/drbd/drbd_req.c index a500e738d929..a47e4987ee46 100644 --- a/drivers/block/drbd/drbd_req.c +++ b/drivers/block/drbd/drbd_req.c @@ -55,7 +55,7 @@ static struct drbd_request *drbd_req_new(struct drbd_device *device, struct bio { struct drbd_request *req; - req = mempool_alloc(drbd_request_mempool, GFP_NOIO); + req = mempool_alloc(&drbd_request_mempool, GFP_NOIO); if (!req) return NULL; memset(req, 0, sizeof(*req)); @@ -184,7 +184,7 @@ void drbd_req_destroy(struct kref *kref) } } - mempool_free(req, drbd_request_mempool); + mempool_free(req, &drbd_request_mempool); } static void wake_all_senders(struct drbd_connection *connection) diff --git a/drivers/block/drbd/drbd_req.h b/drivers/block/drbd/drbd_req.h index cb97b3b30962..94c654020f0f 100644 --- a/drivers/block/drbd/drbd_req.h +++ b/drivers/block/drbd/drbd_req.h @@ -269,7 +269,7 @@ enum drbd_req_state_bits { static inline void drbd_req_make_private_bio(struct drbd_request *req, struct bio *bio_src) { struct bio *bio; - bio = bio_clone_fast(bio_src, GFP_NOIO, drbd_io_bio_set); + bio = bio_clone_fast(bio_src, GFP_NOIO, &drbd_io_bio_set); req->private_bio = bio; diff --git a/drivers/block/floppy.c b/drivers/block/floppy.c index 8ec7235fc93b..8871b5044d9e 100644 --- a/drivers/block/floppy.c +++ b/drivers/block/floppy.c @@ -4450,7 +4450,7 @@ static ssize_t floppy_cmos_show(struct device *dev, return sprintf(buf, "%X\n", UDP->cmos); } -static DEVICE_ATTR(cmos, S_IRUGO, floppy_cmos_show, NULL); +static DEVICE_ATTR(cmos, 0444, floppy_cmos_show, NULL); static struct attribute *floppy_dev_attrs[] = { &dev_attr_cmos.attr, diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 55cf554bc914..d6b6f434fd4b 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -76,6 +76,8 @@ #include #include #include +#include + #include "loop.h" #include @@ -522,7 +524,8 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, __rq_for_each_bio(bio, rq) segments += bio_segments(bio); - bvec = kmalloc(sizeof(struct bio_vec) * segments, GFP_NOIO); + bvec = kmalloc_array(segments, sizeof(struct bio_vec), + GFP_NOIO); if (!bvec) return -EIO; cmd->bvec = bvec; @@ -559,6 +562,7 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, cmd->iocb.ki_filp = file; cmd->iocb.ki_complete = lo_rw_aio_complete; cmd->iocb.ki_flags = IOCB_DIRECT; + cmd->iocb.ki_ioprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0); if (cmd->css) kthread_associate_blkcg(cmd->css); @@ -644,6 +648,36 @@ static void loop_reread_partitions(struct loop_device *lo, __func__, lo->lo_number, lo->lo_file_name, rc); } +static inline int is_loop_device(struct file *file) +{ + struct inode *i = file->f_mapping->host; + + return i && S_ISBLK(i->i_mode) && MAJOR(i->i_rdev) == LOOP_MAJOR; +} + +static int loop_validate_file(struct file *file, struct block_device *bdev) +{ + struct inode *inode = file->f_mapping->host; + struct file *f = file; + + /* Avoid recursion */ + while (is_loop_device(f)) { + struct loop_device *l; + + if (f->f_mapping->host->i_bdev == bdev) + return -EBADF; + + l = f->f_mapping->host->i_bdev->bd_disk->private_data; + if (l->lo_state == Lo_unbound) { + return -EINVAL; + } + f = l->lo_backing_file; + } + if (!S_ISREG(inode->i_mode) && !S_ISBLK(inode->i_mode)) + return -EINVAL; + return 0; +} + /* * loop_change_fd switched the backing store of a loopback device to * a new file. This is useful for operating system installers to free up @@ -673,14 +707,15 @@ static int loop_change_fd(struct loop_device *lo, struct block_device *bdev, if (!file) goto out; + error = loop_validate_file(file, bdev); + if (error) + goto out_putf; + inode = file->f_mapping->host; old_file = lo->lo_backing_file; error = -EINVAL; - if (!S_ISREG(inode->i_mode) && !S_ISBLK(inode->i_mode)) - goto out_putf; - /* size of the new backing store needs to be the same */ if (get_loop_size(lo, file) != get_loop_size(lo, old_file)) goto out_putf; @@ -706,13 +741,6 @@ static int loop_change_fd(struct loop_device *lo, struct block_device *bdev, return error; } -static inline int is_loop_device(struct file *file) -{ - struct inode *i = file->f_mapping->host; - - return i && S_ISBLK(i->i_mode) && MAJOR(i->i_rdev) == LOOP_MAJOR; -} - /* loop sysfs attributes */ static ssize_t loop_attr_show(struct device *dev, char *page, @@ -732,7 +760,7 @@ static ssize_t loop_attr_do_show_##_name(struct device *d, \ return loop_attr_show(d, b, loop_attr_##_name##_show); \ } \ static struct device_attribute loop_attr_##_name = \ - __ATTR(_name, S_IRUGO, loop_attr_do_show_##_name, NULL); + __ATTR(_name, 0444, loop_attr_do_show_##_name, NULL); static ssize_t loop_attr_backing_file_show(struct loop_device *lo, char *buf) { @@ -809,16 +837,17 @@ static struct attribute_group loop_attribute_group = { .attrs= loop_attrs, }; -static int loop_sysfs_init(struct loop_device *lo) +static void loop_sysfs_init(struct loop_device *lo) { - return sysfs_create_group(&disk_to_dev(lo->lo_disk)->kobj, - &loop_attribute_group); + lo->sysfs_inited = !sysfs_create_group(&disk_to_dev(lo->lo_disk)->kobj, + &loop_attribute_group); } static void loop_sysfs_exit(struct loop_device *lo) { - sysfs_remove_group(&disk_to_dev(lo->lo_disk)->kobj, - &loop_attribute_group); + if (lo->sysfs_inited) + sysfs_remove_group(&disk_to_dev(lo->lo_disk)->kobj, + &loop_attribute_group); } static void loop_config_discard(struct loop_device *lo) @@ -877,7 +906,7 @@ static int loop_prepare_queue(struct loop_device *lo) static int loop_set_fd(struct loop_device *lo, fmode_t mode, struct block_device *bdev, unsigned int arg) { - struct file *file, *f; + struct file *file; struct inode *inode; struct address_space *mapping; int lo_flags = 0; @@ -896,29 +925,13 @@ static int loop_set_fd(struct loop_device *lo, fmode_t mode, if (lo->lo_state != Lo_unbound) goto out_putf; - /* Avoid recursion */ - f = file; - while (is_loop_device(f)) { - struct loop_device *l; - - if (f->f_mapping->host->i_bdev == bdev) - goto out_putf; - - l = f->f_mapping->host->i_bdev->bd_disk->private_data; - if (l->lo_state == Lo_unbound) { - error = -EINVAL; - goto out_putf; - } - f = l->lo_backing_file; - } + error = loop_validate_file(file, bdev); + if (error) + goto out_putf; mapping = file->f_mapping; inode = mapping->host; - error = -EINVAL; - if (!S_ISREG(inode->i_mode) && !S_ISBLK(inode->i_mode)) - goto out_putf; - if (!(file->f_mode & FMODE_WRITE) || !(mode & FMODE_WRITE) || !file->f_op->write_iter) lo_flags |= LO_FLAGS_READ_ONLY; @@ -1677,9 +1690,9 @@ static const struct block_device_operations lo_fops = { * And now the modules code and kernel interface. */ static int max_loop; -module_param(max_loop, int, S_IRUGO); +module_param(max_loop, int, 0444); MODULE_PARM_DESC(max_loop, "Maximum number of loop devices"); -module_param(max_part, int, S_IRUGO); +module_param(max_part, int, 0444); MODULE_PARM_DESC(max_part, "Maximum number of partitions per loop device"); MODULE_LICENSE("GPL"); MODULE_ALIAS_BLOCKDEV_MAJOR(LOOP_MAJOR); diff --git a/drivers/block/loop.h b/drivers/block/loop.h index b78de9879f4f..4d42c7af7de7 100644 --- a/drivers/block/loop.h +++ b/drivers/block/loop.h @@ -58,6 +58,7 @@ struct loop_device { struct kthread_worker worker; struct task_struct *worker_task; bool use_dio; + bool sysfs_inited; struct request_queue *lo_queue; struct blk_mq_tag_set tag_set; diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c index 769c551e3d71..c73626decb46 100644 --- a/drivers/block/mtip32xx/mtip32xx.c +++ b/drivers/block/mtip32xx/mtip32xx.c @@ -2285,7 +2285,7 @@ static ssize_t mtip_hw_show_status(struct device *dev, return size; } -static DEVICE_ATTR(status, S_IRUGO, mtip_hw_show_status, NULL); +static DEVICE_ATTR(status, 0444, mtip_hw_show_status, NULL); /* debugsfs entries */ @@ -2566,10 +2566,9 @@ static int mtip_hw_debugfs_init(struct driver_data *dd) return -1; } - debugfs_create_file("flags", S_IRUGO, dd->dfs_node, dd, - &mtip_flags_fops); - debugfs_create_file("registers", S_IRUGO, dd->dfs_node, dd, - &mtip_regs_fops); + debugfs_create_file("flags", 0444, dd->dfs_node, dd, &mtip_flags_fops); + debugfs_create_file("registers", 0444, dd->dfs_node, dd, + &mtip_regs_fops); return 0; } @@ -2726,15 +2725,11 @@ static void mtip_softirq_done_fn(struct request *rq) blk_mq_end_request(rq, cmd->status); } -static void mtip_abort_cmd(struct request *req, void *data, - bool reserved) +static void mtip_abort_cmd(struct request *req, void *data, bool reserved) { struct mtip_cmd *cmd = blk_mq_rq_to_pdu(req); struct driver_data *dd = data; - if (!blk_mq_request_started(req)) - return; - dbg_printk(MTIP_DRV_NAME " Aborting request, tag = %d\n", req->tag); clear_bit(req->tag, dd->port->cmds_to_issue); @@ -2742,14 +2737,10 @@ static void mtip_abort_cmd(struct request *req, void *data, mtip_softirq_done_fn(req); } -static void mtip_queue_cmd(struct request *req, void *data, - bool reserved) +static void mtip_queue_cmd(struct request *req, void *data, bool reserved) { struct driver_data *dd = data; - if (!blk_mq_request_started(req)) - return; - set_bit(req->tag, dd->port->cmds_to_issue); blk_abort_request(req); } @@ -3720,7 +3711,8 @@ static enum blk_eh_timer_return mtip_cmd_timeout(struct request *req, struct mtip_cmd *cmd = blk_mq_rq_to_pdu(req); cmd->status = BLK_STS_TIMEOUT; - return BLK_EH_HANDLED; + blk_mq_complete_request(req); + return BLK_EH_DONE; } if (test_bit(req->tag, dd->port->cmds_to_issue)) @@ -3862,7 +3854,6 @@ skip_create_disk: blk_queue_max_hw_sectors(dd->queue, 0xffff); blk_queue_max_segment_size(dd->queue, 0x400000); blk_queue_io_min(dd->queue, 4096); - blk_queue_bounce_limit(dd->queue, dd->pdev->dma_mask); /* Signal trim support */ if (dd->trim_supp == true) { @@ -4273,7 +4264,7 @@ static int mtip_pci_probe(struct pci_dev *pdev, if (!dd->isr_workq) { dev_warn(&pdev->dev, "Can't create wq %d\n", dd->instance); rv = -ENOMEM; - goto block_initialize_err; + goto setmask_err; } memset(cpu_list, 0, sizeof(cpu_list)); @@ -4614,7 +4605,7 @@ static int __init mtip_init(void) } if (dfs_parent) { dfs_device_status = debugfs_create_file("device_status", - S_IRUGO, dfs_parent, NULL, + 0444, dfs_parent, NULL, &mtip_device_status_fops); if (IS_ERR_OR_NULL(dfs_device_status)) { pr_err("Error creating device_status node\n"); diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index afbc202ca6fd..3b7083b8ecbb 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -166,16 +166,19 @@ static ssize_t pid_show(struct device *dev, } static const struct device_attribute pid_attr = { - .attr = { .name = "pid", .mode = S_IRUGO}, + .attr = { .name = "pid", .mode = 0444}, .show = pid_show, }; static void nbd_dev_remove(struct nbd_device *nbd) { struct gendisk *disk = nbd->disk; + struct request_queue *q; + if (disk) { + q = disk->queue; del_gendisk(disk); - blk_cleanup_queue(disk->queue); + blk_cleanup_queue(q); blk_mq_free_tag_set(&nbd->tag_set); disk->private_data = NULL; put_disk(disk); @@ -213,7 +216,15 @@ static void nbd_mark_nsock_dead(struct nbd_device *nbd, struct nbd_sock *nsock, } if (!nsock->dead) { kernel_sock_shutdown(nsock->sock, SHUT_RDWR); - atomic_dec(&nbd->config->live_connections); + if (atomic_dec_return(&nbd->config->live_connections) == 0) { + if (test_and_clear_bit(NBD_DISCONNECT_REQUESTED, + &nbd->config->runtime_flags)) { + set_bit(NBD_DISCONNECTED, + &nbd->config->runtime_flags); + dev_info(nbd_to_dev(nbd), + "Disconnected due to user request.\n"); + } + } } nsock->dead = true; nsock->pending = NULL; @@ -231,9 +242,23 @@ static void nbd_size_clear(struct nbd_device *nbd) static void nbd_size_update(struct nbd_device *nbd) { struct nbd_config *config = nbd->config; + struct block_device *bdev = bdget_disk(nbd->disk, 0); + + if (config->flags & NBD_FLAG_SEND_TRIM) { + nbd->disk->queue->limits.discard_granularity = config->blksize; + nbd->disk->queue->limits.discard_alignment = config->blksize; + blk_queue_max_discard_sectors(nbd->disk->queue, UINT_MAX); + } blk_queue_logical_block_size(nbd->disk->queue, config->blksize); blk_queue_physical_block_size(nbd->disk->queue, config->blksize); set_capacity(nbd->disk, config->bytesize >> 9); + if (bdev) { + if (bdev->bd_disk) + bd_set_size(bdev, config->bytesize); + else + bdev->bd_invalidated = 1; + bdput(bdev); + } kobject_uevent(&nbd_to_dev(nbd)->kobj, KOBJ_CHANGE); } @@ -243,13 +268,15 @@ static void nbd_size_set(struct nbd_device *nbd, loff_t blocksize, struct nbd_config *config = nbd->config; config->blksize = blocksize; config->bytesize = blocksize * nr_blocks; + if (nbd->task_recv != NULL) + nbd_size_update(nbd); } static void nbd_complete_rq(struct request *req) { struct nbd_cmd *cmd = blk_mq_rq_to_pdu(req); - dev_dbg(nbd_to_dev(cmd->nbd), "request %p: %s\n", cmd, + dev_dbg(nbd_to_dev(cmd->nbd), "request %p: %s\n", req, cmd->status ? "failed" : "done"); blk_mq_end_request(req, cmd->status); @@ -286,13 +313,15 @@ static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req, if (!refcount_inc_not_zero(&nbd->config_refs)) { cmd->status = BLK_STS_TIMEOUT; - return BLK_EH_HANDLED; + goto done; } config = nbd->config; if (config->num_connections > 1) { dev_err_ratelimited(nbd_to_dev(nbd), - "Connection timed out, retrying\n"); + "Connection timed out, retrying (%d/%d alive)\n", + atomic_read(&config->live_connections), + config->num_connections); /* * Hooray we have more connections, requeue this IO, the submit * path will put it on a real connection. @@ -314,7 +343,7 @@ static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req, } blk_mq_requeue_request(req, true); nbd_config_put(nbd); - return BLK_EH_NOT_HANDLED; + return BLK_EH_DONE; } } else { dev_err_ratelimited(nbd_to_dev(nbd), @@ -324,8 +353,9 @@ static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req, cmd->status = BLK_STS_IOERR; sock_shutdown(nbd); nbd_config_put(nbd); - - return BLK_EH_HANDLED; +done: + blk_mq_complete_request(req); + return BLK_EH_DONE; } /* @@ -453,7 +483,7 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd, int index) memcpy(request.handle, &tag, sizeof(tag)); dev_dbg(nbd_to_dev(nbd), "request %p: sending control (%s@%llu,%uB)\n", - cmd, nbdcmd_to_ascii(type), + req, nbdcmd_to_ascii(type), (unsigned long long)blk_rq_pos(req) << 9, blk_rq_bytes(req)); result = sock_xmit(nbd, index, 1, &from, (type == NBD_CMD_WRITE) ? MSG_MORE : 0, &sent); @@ -489,7 +519,7 @@ send_pages: int flags = is_last ? 0 : MSG_MORE; dev_dbg(nbd_to_dev(nbd), "request %p: sending %d bytes data\n", - cmd, bvec.bv_len); + req, bvec.bv_len); iov_iter_bvec(&from, ITER_BVEC | WRITE, &bvec, 1, bvec.bv_len); if (skip) { @@ -581,7 +611,7 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index) return cmd; } - dev_dbg(nbd_to_dev(nbd), "request %p: got reply\n", cmd); + dev_dbg(nbd_to_dev(nbd), "request %p: got reply\n", req); if (rq_data_dir(req) != WRITE) { struct req_iterator iter; struct bio_vec bvec; @@ -608,7 +638,7 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index) return ERR_PTR(-EIO); } dev_dbg(nbd_to_dev(nbd), "request %p: got %d bytes data\n", - cmd, bvec.bv_len); + req, bvec.bv_len); } } else { /* See the comment in nbd_queue_rq. */ @@ -647,11 +677,8 @@ static void recv_work(struct work_struct *work) static void nbd_clear_req(struct request *req, void *data, bool reserved) { - struct nbd_cmd *cmd; + struct nbd_cmd *cmd = blk_mq_rq_to_pdu(req); - if (!blk_mq_request_started(req)) - return; - cmd = blk_mq_rq_to_pdu(req); cmd->status = BLK_STS_IOERR; blk_mq_complete_request(req); } @@ -714,10 +741,9 @@ static int wait_for_reconnect(struct nbd_device *nbd) return 0; if (test_bit(NBD_DISCONNECTED, &config->runtime_flags)) return 0; - wait_event_timeout(config->conn_wait, - atomic_read(&config->live_connections), - config->dead_conn_timeout); - return atomic_read(&config->live_connections); + return wait_event_timeout(config->conn_wait, + atomic_read(&config->live_connections) > 0, + config->dead_conn_timeout) > 0; } static int nbd_handle_cmd(struct nbd_cmd *cmd, int index) @@ -950,10 +976,6 @@ static void nbd_bdev_reset(struct block_device *bdev) if (bdev->bd_openers > 1) return; bd_set_size(bdev, 0); - if (max_part > 0) { - blkdev_reread_part(bdev); - bdev->bd_invalidated = 1; - } } static void nbd_parse_flags(struct nbd_device *nbd) @@ -1040,6 +1062,9 @@ static void nbd_config_put(struct nbd_device *nbd) nbd->config = NULL; nbd->tag_set.timeout = 0; + nbd->disk->queue->limits.discard_granularity = 0; + nbd->disk->queue->limits.discard_alignment = 0; + blk_queue_max_discard_sectors(nbd->disk->queue, UINT_MAX); blk_queue_flag_clear(QUEUE_FLAG_DISCARD, nbd->disk->queue); mutex_unlock(&nbd->config_lock); @@ -1109,7 +1134,6 @@ static int nbd_start_device_ioctl(struct nbd_device *nbd, struct block_device *b if (ret) return ret; - bd_set_size(bdev, config->bytesize); if (max_part) bdev->bd_invalidated = 1; mutex_unlock(&nbd->config_lock); @@ -1118,7 +1142,7 @@ static int nbd_start_device_ioctl(struct nbd_device *nbd, struct block_device *b if (ret) sock_shutdown(nbd); mutex_lock(&nbd->config_lock); - bd_set_size(bdev, 0); + nbd_bdev_reset(bdev); /* user requested, ignore socket errors */ if (test_bit(NBD_DISCONNECT_REQUESTED, &config->runtime_flags)) ret = 0; @@ -1269,6 +1293,9 @@ static int nbd_open(struct block_device *bdev, fmode_t mode) refcount_set(&nbd->config_refs, 1); refcount_inc(&nbd->refs); mutex_unlock(&nbd->config_lock); + bdev->bd_invalidated = 1; + } else if (nbd_disconnected(nbd->config)) { + bdev->bd_invalidated = 1; } out: mutex_unlock(&nbd_index_mutex); @@ -1490,8 +1517,9 @@ static int nbd_dev_add(int index) */ blk_queue_flag_set(QUEUE_FLAG_NONROT, disk->queue); blk_queue_flag_clear(QUEUE_FLAG_ADD_RANDOM, disk->queue); - disk->queue->limits.discard_granularity = 512; - blk_queue_max_discard_sectors(disk->queue, UINT_MAX); + disk->queue->limits.discard_granularity = 0; + disk->queue->limits.discard_alignment = 0; + blk_queue_max_discard_sectors(disk->queue, 0); blk_queue_max_segment_size(disk->queue, UINT_MAX); blk_queue_max_segments(disk->queue, USHRT_MAX); blk_queue_max_hw_sectors(disk->queue, 65536); @@ -1755,6 +1783,7 @@ static int nbd_genl_disconnect(struct sk_buff *skb, struct genl_info *info) } mutex_lock(&nbd->config_lock); nbd_disconnect(nbd); + nbd_clear_sock(nbd); mutex_unlock(&nbd->config_lock); if (test_and_clear_bit(NBD_HAS_CONFIG_REF, &nbd->config->runtime_flags)) @@ -2093,7 +2122,8 @@ static int __init nbd_init(void) if (nbds_max > 1UL << (MINORBITS - part_shift)) return -EINVAL; recv_workqueue = alloc_workqueue("knbd-recv", - WQ_MEM_RECLAIM | WQ_HIGHPRI, 0); + WQ_MEM_RECLAIM | WQ_HIGHPRI | + WQ_UNBOUND, 0); if (!recv_workqueue) return -ENOMEM; diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c index a76553293a31..7948049f6c43 100644 --- a/drivers/block/null_blk.c +++ b/drivers/block/null_blk.c @@ -157,23 +157,23 @@ enum { }; static int g_no_sched; -module_param_named(no_sched, g_no_sched, int, S_IRUGO); +module_param_named(no_sched, g_no_sched, int, 0444); MODULE_PARM_DESC(no_sched, "No io scheduler"); static int g_submit_queues = 1; -module_param_named(submit_queues, g_submit_queues, int, S_IRUGO); +module_param_named(submit_queues, g_submit_queues, int, 0444); MODULE_PARM_DESC(submit_queues, "Number of submission queues"); static int g_home_node = NUMA_NO_NODE; -module_param_named(home_node, g_home_node, int, S_IRUGO); +module_param_named(home_node, g_home_node, int, 0444); MODULE_PARM_DESC(home_node, "Home node for the device"); #ifdef CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION static char g_timeout_str[80]; -module_param_string(timeout, g_timeout_str, sizeof(g_timeout_str), S_IRUGO); +module_param_string(timeout, g_timeout_str, sizeof(g_timeout_str), 0444); static char g_requeue_str[80]; -module_param_string(requeue, g_requeue_str, sizeof(g_requeue_str), S_IRUGO); +module_param_string(requeue, g_requeue_str, sizeof(g_requeue_str), 0444); #endif static int g_queue_mode = NULL_Q_MQ; @@ -203,27 +203,27 @@ static const struct kernel_param_ops null_queue_mode_param_ops = { .get = param_get_int, }; -device_param_cb(queue_mode, &null_queue_mode_param_ops, &g_queue_mode, S_IRUGO); +device_param_cb(queue_mode, &null_queue_mode_param_ops, &g_queue_mode, 0444); MODULE_PARM_DESC(queue_mode, "Block interface to use (0=bio,1=rq,2=multiqueue)"); static int g_gb = 250; -module_param_named(gb, g_gb, int, S_IRUGO); +module_param_named(gb, g_gb, int, 0444); MODULE_PARM_DESC(gb, "Size in GB"); static int g_bs = 512; -module_param_named(bs, g_bs, int, S_IRUGO); +module_param_named(bs, g_bs, int, 0444); MODULE_PARM_DESC(bs, "Block size (in bytes)"); static int nr_devices = 1; -module_param(nr_devices, int, S_IRUGO); +module_param(nr_devices, int, 0444); MODULE_PARM_DESC(nr_devices, "Number of devices to register"); static bool g_blocking; -module_param_named(blocking, g_blocking, bool, S_IRUGO); +module_param_named(blocking, g_blocking, bool, 0444); MODULE_PARM_DESC(blocking, "Register as a blocking blk-mq driver device"); static bool shared_tags; -module_param(shared_tags, bool, S_IRUGO); +module_param(shared_tags, bool, 0444); MODULE_PARM_DESC(shared_tags, "Share tag set between devices for blk-mq"); static int g_irqmode = NULL_IRQ_SOFTIRQ; @@ -239,19 +239,19 @@ static const struct kernel_param_ops null_irqmode_param_ops = { .get = param_get_int, }; -device_param_cb(irqmode, &null_irqmode_param_ops, &g_irqmode, S_IRUGO); +device_param_cb(irqmode, &null_irqmode_param_ops, &g_irqmode, 0444); MODULE_PARM_DESC(irqmode, "IRQ completion handler. 0-none, 1-softirq, 2-timer"); static unsigned long g_completion_nsec = 10000; -module_param_named(completion_nsec, g_completion_nsec, ulong, S_IRUGO); +module_param_named(completion_nsec, g_completion_nsec, ulong, 0444); MODULE_PARM_DESC(completion_nsec, "Time in ns to complete a request in hardware. Default: 10,000ns"); static int g_hw_queue_depth = 64; -module_param_named(hw_queue_depth, g_hw_queue_depth, int, S_IRUGO); +module_param_named(hw_queue_depth, g_hw_queue_depth, int, 0444); MODULE_PARM_DESC(hw_queue_depth, "Queue depth for each hardware queue. Default: 64"); static bool g_use_per_node_hctx; -module_param_named(use_per_node_hctx, g_use_per_node_hctx, bool, S_IRUGO); +module_param_named(use_per_node_hctx, g_use_per_node_hctx, bool, 0444); MODULE_PARM_DESC(use_per_node_hctx, "Use per-node allocation for hardware context queues. Default: false"); static struct nullb_device *null_alloc_dev(void); @@ -1365,7 +1365,8 @@ static blk_qc_t null_queue_bio(struct request_queue *q, struct bio *bio) static enum blk_eh_timer_return null_rq_timed_out_fn(struct request *rq) { pr_info("null: rq %p timed out\n", rq); - return BLK_EH_HANDLED; + blk_mq_complete_request(rq); + return BLK_EH_DONE; } static int null_rq_prep_fn(struct request_queue *q, struct request *req) @@ -1427,7 +1428,8 @@ static void null_request_fn(struct request_queue *q) static enum blk_eh_timer_return null_timeout_rq(struct request *rq, bool res) { pr_info("null: rq %p timed out\n", rq); - return BLK_EH_HANDLED; + blk_mq_complete_request(rq); + return BLK_EH_DONE; } static blk_status_t null_queue_rq(struct blk_mq_hw_ctx *hctx, @@ -1573,12 +1575,12 @@ static int setup_commands(struct nullb_queue *nq) struct nullb_cmd *cmd; int i, tag_size; - nq->cmds = kzalloc(nq->queue_depth * sizeof(*cmd), GFP_KERNEL); + nq->cmds = kcalloc(nq->queue_depth, sizeof(*cmd), GFP_KERNEL); if (!nq->cmds) return -ENOMEM; tag_size = ALIGN(nq->queue_depth, BITS_PER_LONG) / BITS_PER_LONG; - nq->tag_map = kzalloc(tag_size * sizeof(unsigned long), GFP_KERNEL); + nq->tag_map = kcalloc(tag_size, sizeof(unsigned long), GFP_KERNEL); if (!nq->tag_map) { kfree(nq->cmds); return -ENOMEM; @@ -1596,8 +1598,9 @@ static int setup_commands(struct nullb_queue *nq) static int setup_queues(struct nullb *nullb) { - nullb->queues = kzalloc(nullb->dev->submit_queues * - sizeof(struct nullb_queue), GFP_KERNEL); + nullb->queues = kcalloc(nullb->dev->submit_queues, + sizeof(struct nullb_queue), + GFP_KERNEL); if (!nullb->queues) return -ENOMEM; diff --git a/drivers/block/paride/pd.c b/drivers/block/paride/pd.c index 27a44b97393a..8961b190e256 100644 --- a/drivers/block/paride/pd.c +++ b/drivers/block/paride/pd.c @@ -740,7 +740,7 @@ static int pd_special_command(struct pd_unit *disk, { struct request *rq; - rq = blk_get_request(disk->gd->queue, REQ_OP_DRV_IN, __GFP_RECLAIM); + rq = blk_get_request(disk->gd->queue, REQ_OP_DRV_IN, 0); if (IS_ERR(rq)) return PTR_ERR(rq); diff --git a/drivers/block/pktcdvd.c b/drivers/block/pktcdvd.c index c61d20c9f3f8..b3f83cd96f33 100644 --- a/drivers/block/pktcdvd.c +++ b/drivers/block/pktcdvd.c @@ -97,8 +97,8 @@ static int pktdev_major; static int write_congestion_on = PKT_WRITE_CONGESTION_ON; static int write_congestion_off = PKT_WRITE_CONGESTION_OFF; static struct mutex ctl_mutex; /* Serialize open/close/setup/teardown */ -static mempool_t *psd_pool; -static struct bio_set *pkt_bio_set; +static mempool_t psd_pool; +static struct bio_set pkt_bio_set; static struct class *class_pktcdvd = NULL; /* /sys/class/pktcdvd */ static struct dentry *pkt_debugfs_root = NULL; /* /sys/kernel/debug/pktcdvd */ @@ -478,8 +478,8 @@ static void pkt_debugfs_dev_new(struct pktcdvd_device *pd) if (!pd->dfs_d_root) return; - pd->dfs_f_info = debugfs_create_file("info", S_IRUGO, - pd->dfs_d_root, pd, &debug_fops); + pd->dfs_f_info = debugfs_create_file("info", 0444, + pd->dfs_d_root, pd, &debug_fops); } static void pkt_debugfs_dev_remove(struct pktcdvd_device *pd) @@ -631,7 +631,7 @@ static inline struct pkt_rb_node *pkt_rbtree_next(struct pkt_rb_node *node) static void pkt_rbtree_erase(struct pktcdvd_device *pd, struct pkt_rb_node *node) { rb_erase(&node->rb_node, &pd->bio_queue); - mempool_free(node, pd->rb_pool); + mempool_free(node, &pd->rb_pool); pd->bio_queue_size--; BUG_ON(pd->bio_queue_size < 0); } @@ -704,13 +704,13 @@ static int pkt_generic_packet(struct pktcdvd_device *pd, struct packet_command * int ret = 0; rq = blk_get_request(q, (cgc->data_direction == CGC_DATA_WRITE) ? - REQ_OP_SCSI_OUT : REQ_OP_SCSI_IN, __GFP_RECLAIM); + REQ_OP_SCSI_OUT : REQ_OP_SCSI_IN, 0); if (IS_ERR(rq)) return PTR_ERR(rq); if (cgc->buflen) { ret = blk_rq_map_kern(q, rq, cgc->buffer, cgc->buflen, - __GFP_RECLAIM); + GFP_NOIO); if (ret) goto out; } @@ -1285,7 +1285,7 @@ static void pkt_start_write(struct pktcdvd_device *pd, struct packet_data *pkt) * Fill-in bvec with data from orig_bios. */ spin_lock(&pkt->lock); - bio_copy_data(pkt->w_bio, pkt->orig_bios.head); + bio_list_copy_data(pkt->w_bio, pkt->orig_bios.head); pkt_set_state(pkt, PACKET_WRITE_WAIT_STATE); spin_unlock(&pkt->lock); @@ -2303,14 +2303,14 @@ static void pkt_end_io_read_cloned(struct bio *bio) psd->bio->bi_status = bio->bi_status; bio_put(bio); bio_endio(psd->bio); - mempool_free(psd, psd_pool); + mempool_free(psd, &psd_pool); pkt_bio_finished(pd); } static void pkt_make_request_read(struct pktcdvd_device *pd, struct bio *bio) { - struct bio *cloned_bio = bio_clone_fast(bio, GFP_NOIO, pkt_bio_set); - struct packet_stacked_data *psd = mempool_alloc(psd_pool, GFP_NOIO); + struct bio *cloned_bio = bio_clone_fast(bio, GFP_NOIO, &pkt_bio_set); + struct packet_stacked_data *psd = mempool_alloc(&psd_pool, GFP_NOIO); psd->pd = pd; psd->bio = bio; @@ -2381,7 +2381,7 @@ static void pkt_make_request_write(struct request_queue *q, struct bio *bio) /* * No matching packet found. Store the bio in the work queue. */ - node = mempool_alloc(pd->rb_pool, GFP_NOIO); + node = mempool_alloc(&pd->rb_pool, GFP_NOIO); node->bio = bio; spin_lock(&pd->lock); BUG_ON(pd->bio_queue_size < 0); @@ -2451,7 +2451,7 @@ static blk_qc_t pkt_make_request(struct request_queue *q, struct bio *bio) split = bio_split(bio, last_zone - bio->bi_iter.bi_sector, - GFP_NOIO, pkt_bio_set); + GFP_NOIO, &pkt_bio_set); bio_chain(split, bio); } else { split = bio; @@ -2538,18 +2538,6 @@ static int pkt_seq_show(struct seq_file *m, void *p) return 0; } -static int pkt_seq_open(struct inode *inode, struct file *file) -{ - return single_open(file, pkt_seq_show, PDE_DATA(inode)); -} - -static const struct file_operations pkt_proc_fops = { - .open = pkt_seq_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release -}; - static int pkt_new_dev(struct pktcdvd_device *pd, dev_t dev) { int i; @@ -2604,7 +2592,7 @@ static int pkt_new_dev(struct pktcdvd_device *pd, dev_t dev) goto out_mem; } - proc_create_data(pd->name, 0, pkt_proc, &pkt_proc_fops, pd); + proc_create_single_data(pd->name, 0, pkt_proc, pkt_seq_show, pd); pkt_dbg(1, pd, "writer mapped to %s\n", bdevname(bdev, b)); return 0; @@ -2707,9 +2695,9 @@ static int pkt_setup_dev(dev_t dev, dev_t* pkt_dev) if (!pd) goto out_mutex; - pd->rb_pool = mempool_create_kmalloc_pool(PKT_RB_POOL_SIZE, - sizeof(struct pkt_rb_node)); - if (!pd->rb_pool) + ret = mempool_init_kmalloc_pool(&pd->rb_pool, PKT_RB_POOL_SIZE, + sizeof(struct pkt_rb_node)); + if (ret) goto out_mem; INIT_LIST_HEAD(&pd->cdrw.pkt_free_list); @@ -2766,7 +2754,7 @@ static int pkt_setup_dev(dev_t dev, dev_t* pkt_dev) out_mem2: put_disk(disk); out_mem: - mempool_destroy(pd->rb_pool); + mempool_exit(&pd->rb_pool); kfree(pd); out_mutex: mutex_unlock(&ctl_mutex); @@ -2817,7 +2805,7 @@ static int pkt_remove_dev(dev_t pkt_dev) blk_cleanup_queue(pd->disk->queue); put_disk(pd->disk); - mempool_destroy(pd->rb_pool); + mempool_exit(&pd->rb_pool); kfree(pd); /* This is safe: open() is still holding a reference. */ @@ -2914,14 +2902,14 @@ static int __init pkt_init(void) mutex_init(&ctl_mutex); - psd_pool = mempool_create_kmalloc_pool(PSD_POOL_SIZE, - sizeof(struct packet_stacked_data)); - if (!psd_pool) - return -ENOMEM; - pkt_bio_set = bioset_create(BIO_POOL_SIZE, 0, 0); - if (!pkt_bio_set) { - mempool_destroy(psd_pool); - return -ENOMEM; + ret = mempool_init_kmalloc_pool(&psd_pool, PSD_POOL_SIZE, + sizeof(struct packet_stacked_data)); + if (ret) + return ret; + ret = bioset_init(&pkt_bio_set, BIO_POOL_SIZE, 0, 0); + if (ret) { + mempool_exit(&psd_pool); + return ret; } ret = register_blkdev(pktdev_major, DRIVER_NAME); @@ -2954,8 +2942,8 @@ out_misc: out: unregister_blkdev(pktdev_major, DRIVER_NAME); out2: - mempool_destroy(psd_pool); - bioset_free(pkt_bio_set); + mempool_exit(&psd_pool); + bioset_exit(&pkt_bio_set); return ret; } @@ -2968,8 +2956,8 @@ static void __exit pkt_exit(void) pkt_sysfs_cleanup(); unregister_blkdev(pktdev_major, DRIVER_NAME); - mempool_destroy(psd_pool); - bioset_free(pkt_bio_set); + mempool_exit(&psd_pool); + bioset_exit(&pkt_bio_set); } MODULE_DESCRIPTION("Packet writing layer for CD/DVD drives"); diff --git a/drivers/block/ps3disk.c b/drivers/block/ps3disk.c index 075662f2cf46..afe1508d82c6 100644 --- a/drivers/block/ps3disk.c +++ b/drivers/block/ps3disk.c @@ -465,8 +465,6 @@ static int ps3disk_probe(struct ps3_system_bus_device *_dev) priv->queue = queue; queue->queuedata = dev; - blk_queue_bounce_limit(queue, BLK_BOUNCE_HIGH); - blk_queue_max_hw_sectors(queue, dev->bounce_size >> 9); blk_queue_segment_boundary(queue, -1UL); blk_queue_dma_alignment(queue, dev->blk_size-1); diff --git a/drivers/block/ps3vram.c b/drivers/block/ps3vram.c index 6a55959cbf78..1e3d5de9d838 100644 --- a/drivers/block/ps3vram.c +++ b/drivers/block/ps3vram.c @@ -407,8 +407,9 @@ static int ps3vram_cache_init(struct ps3_system_bus_device *dev) priv->cache.page_count = CACHE_PAGE_COUNT; priv->cache.page_size = CACHE_PAGE_SIZE; - priv->cache.tags = kzalloc(sizeof(struct ps3vram_tag) * - CACHE_PAGE_COUNT, GFP_KERNEL); + priv->cache.tags = kcalloc(CACHE_PAGE_COUNT, + sizeof(struct ps3vram_tag), + GFP_KERNEL); if (!priv->cache.tags) return -ENOMEM; @@ -521,26 +522,13 @@ static int ps3vram_proc_show(struct seq_file *m, void *v) return 0; } -static int ps3vram_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, ps3vram_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations ps3vram_proc_fops = { - .owner = THIS_MODULE, - .open = ps3vram_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static void ps3vram_proc_init(struct ps3_system_bus_device *dev) { struct ps3vram_priv *priv = ps3_system_bus_get_drvdata(dev); struct proc_dir_entry *pde; - pde = proc_create_data(DEVICE_NAME, 0444, NULL, &ps3vram_proc_fops, - priv); + pde = proc_create_single_data(DEVICE_NAME, 0444, NULL, + ps3vram_proc_show, priv); if (!pde) dev_warn(&dev->core, "failed to create /proc entry\n"); } diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 33b36fea1d73..fa0729c1e776 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -424,7 +424,7 @@ static struct workqueue_struct *rbd_wq; * single-major requires >= 0.75 version of userspace rbd utility. */ static bool single_major = true; -module_param(single_major, bool, S_IRUGO); +module_param(single_major, bool, 0444); MODULE_PARM_DESC(single_major, "Use a single major number for all rbd devices (default: true)"); static ssize_t rbd_add(struct bus_type *bus, const char *buf, @@ -468,11 +468,11 @@ static ssize_t rbd_supported_features_show(struct bus_type *bus, char *buf) return sprintf(buf, "0x%llx\n", RBD_FEATURES_SUPPORTED); } -static BUS_ATTR(add, S_IWUSR, NULL, rbd_add); -static BUS_ATTR(remove, S_IWUSR, NULL, rbd_remove); -static BUS_ATTR(add_single_major, S_IWUSR, NULL, rbd_add_single_major); -static BUS_ATTR(remove_single_major, S_IWUSR, NULL, rbd_remove_single_major); -static BUS_ATTR(supported_features, S_IRUGO, rbd_supported_features_show, NULL); +static BUS_ATTR(add, 0200, NULL, rbd_add); +static BUS_ATTR(remove, 0200, NULL, rbd_remove); +static BUS_ATTR(add_single_major, 0200, NULL, rbd_add_single_major); +static BUS_ATTR(remove_single_major, 0200, NULL, rbd_remove_single_major); +static BUS_ATTR(supported_features, 0444, rbd_supported_features_show, NULL); static struct attribute *rbd_bus_attrs[] = { &bus_attr_add.attr, @@ -2339,6 +2339,7 @@ static bool is_zero_bvecs(struct bio_vec *bvecs, u32 bytes) static int rbd_obj_issue_copyup(struct rbd_obj_request *obj_req, u32 bytes) { unsigned int num_osd_ops = obj_req->osd_req->r_num_ops; + int ret; dout("%s obj_req %p bytes %u\n", __func__, obj_req, bytes); rbd_assert(obj_req->osd_req->r_ops[0].op == CEPH_OSD_OP_STAT); @@ -2353,6 +2354,11 @@ static int rbd_obj_issue_copyup(struct rbd_obj_request *obj_req, u32 bytes) if (!obj_req->osd_req) return -ENOMEM; + ret = osd_req_op_cls_init(obj_req->osd_req, 0, CEPH_OSD_OP_CALL, "rbd", + "copyup"); + if (ret) + return ret; + /* * Only send non-zero copyup data to save some I/O and network * bandwidth -- zero copyup data is equivalent to the object not @@ -2362,9 +2368,6 @@ static int rbd_obj_issue_copyup(struct rbd_obj_request *obj_req, u32 bytes) dout("%s obj_req %p detected zeroes\n", __func__, obj_req); bytes = 0; } - - osd_req_op_cls_init(obj_req->osd_req, 0, CEPH_OSD_OP_CALL, "rbd", - "copyup"); osd_req_op_cls_request_data_bvecs(obj_req->osd_req, 0, obj_req->copyup_bvecs, obj_req->copyup_bvec_count, @@ -3397,7 +3400,6 @@ static void cancel_tasks_sync(struct rbd_device *rbd_dev) { dout("%s rbd_dev %p\n", __func__, rbd_dev); - cancel_delayed_work_sync(&rbd_dev->watch_dwork); cancel_work_sync(&rbd_dev->acquired_lock_work); cancel_work_sync(&rbd_dev->released_lock_work); cancel_delayed_work_sync(&rbd_dev->lock_dwork); @@ -3415,6 +3417,7 @@ static void rbd_unregister_watch(struct rbd_device *rbd_dev) rbd_dev->watch_state = RBD_WATCH_STATE_UNREGISTERED; mutex_unlock(&rbd_dev->watch_mutex); + cancel_delayed_work_sync(&rbd_dev->watch_dwork); ceph_osdc_flush_notifies(&rbd_dev->rbd_client->client->osdc); } @@ -4204,22 +4207,22 @@ static ssize_t rbd_image_refresh(struct device *dev, return size; } -static DEVICE_ATTR(size, S_IRUGO, rbd_size_show, NULL); -static DEVICE_ATTR(features, S_IRUGO, rbd_features_show, NULL); -static DEVICE_ATTR(major, S_IRUGO, rbd_major_show, NULL); -static DEVICE_ATTR(minor, S_IRUGO, rbd_minor_show, NULL); -static DEVICE_ATTR(client_addr, S_IRUGO, rbd_client_addr_show, NULL); -static DEVICE_ATTR(client_id, S_IRUGO, rbd_client_id_show, NULL); -static DEVICE_ATTR(cluster_fsid, S_IRUGO, rbd_cluster_fsid_show, NULL); -static DEVICE_ATTR(config_info, S_IRUSR, rbd_config_info_show, NULL); -static DEVICE_ATTR(pool, S_IRUGO, rbd_pool_show, NULL); -static DEVICE_ATTR(pool_id, S_IRUGO, rbd_pool_id_show, NULL); -static DEVICE_ATTR(name, S_IRUGO, rbd_name_show, NULL); -static DEVICE_ATTR(image_id, S_IRUGO, rbd_image_id_show, NULL); -static DEVICE_ATTR(refresh, S_IWUSR, NULL, rbd_image_refresh); -static DEVICE_ATTR(current_snap, S_IRUGO, rbd_snap_show, NULL); -static DEVICE_ATTR(snap_id, S_IRUGO, rbd_snap_id_show, NULL); -static DEVICE_ATTR(parent, S_IRUGO, rbd_parent_show, NULL); +static DEVICE_ATTR(size, 0444, rbd_size_show, NULL); +static DEVICE_ATTR(features, 0444, rbd_features_show, NULL); +static DEVICE_ATTR(major, 0444, rbd_major_show, NULL); +static DEVICE_ATTR(minor, 0444, rbd_minor_show, NULL); +static DEVICE_ATTR(client_addr, 0444, rbd_client_addr_show, NULL); +static DEVICE_ATTR(client_id, 0444, rbd_client_id_show, NULL); +static DEVICE_ATTR(cluster_fsid, 0444, rbd_cluster_fsid_show, NULL); +static DEVICE_ATTR(config_info, 0400, rbd_config_info_show, NULL); +static DEVICE_ATTR(pool, 0444, rbd_pool_show, NULL); +static DEVICE_ATTR(pool_id, 0444, rbd_pool_id_show, NULL); +static DEVICE_ATTR(name, 0444, rbd_name_show, NULL); +static DEVICE_ATTR(image_id, 0444, rbd_image_id_show, NULL); +static DEVICE_ATTR(refresh, 0200, NULL, rbd_image_refresh); +static DEVICE_ATTR(current_snap, 0444, rbd_snap_show, NULL); +static DEVICE_ATTR(snap_id, 0444, rbd_snap_id_show, NULL); +static DEVICE_ATTR(parent, 0444, rbd_parent_show, NULL); static struct attribute *rbd_attrs[] = { &dev_attr_size.attr, diff --git a/drivers/block/rsxx/core.c b/drivers/block/rsxx/core.c index 34997df132e2..b7d71914a32a 100644 --- a/drivers/block/rsxx/core.c +++ b/drivers/block/rsxx/core.c @@ -247,19 +247,19 @@ static void rsxx_debugfs_dev_new(struct rsxx_cardinfo *card) if (IS_ERR_OR_NULL(card->debugfs_dir)) goto failed_debugfs_dir; - debugfs_stats = debugfs_create_file("stats", S_IRUGO, + debugfs_stats = debugfs_create_file("stats", 0444, card->debugfs_dir, card, &debugfs_stats_fops); if (IS_ERR_OR_NULL(debugfs_stats)) goto failed_debugfs_stats; - debugfs_pci_regs = debugfs_create_file("pci_regs", S_IRUGO, + debugfs_pci_regs = debugfs_create_file("pci_regs", 0444, card->debugfs_dir, card, &debugfs_pci_regs_fops); if (IS_ERR_OR_NULL(debugfs_pci_regs)) goto failed_debugfs_pci_regs; - debugfs_cram = debugfs_create_file("cram", S_IRUGO | S_IWUSR, + debugfs_cram = debugfs_create_file("cram", 0644, card->debugfs_dir, card, &debugfs_cram_fops); if (IS_ERR_OR_NULL(debugfs_cram)) @@ -873,7 +873,8 @@ static int rsxx_pci_probe(struct pci_dev *dev, dev_info(CARD_TO_DEV(card), "Failed reading the number of DMA targets\n"); - card->ctrl = kzalloc(card->n_targets * sizeof(*card->ctrl), GFP_KERNEL); + card->ctrl = kcalloc(card->n_targets, sizeof(*card->ctrl), + GFP_KERNEL); if (!card->ctrl) { st = -ENOMEM; goto failed_dma_setup; diff --git a/drivers/block/rsxx/dma.c b/drivers/block/rsxx/dma.c index beaccf197a5a..8fbc1bf6db3d 100644 --- a/drivers/block/rsxx/dma.c +++ b/drivers/block/rsxx/dma.c @@ -1038,7 +1038,7 @@ int rsxx_eeh_save_issued_dmas(struct rsxx_cardinfo *card) struct rsxx_dma *dma; struct list_head *issued_dmas; - issued_dmas = kzalloc(sizeof(*issued_dmas) * card->n_targets, + issued_dmas = kcalloc(card->n_targets, sizeof(*issued_dmas), GFP_KERNEL); if (!issued_dmas) return -ENOMEM; diff --git a/drivers/block/sx8.c b/drivers/block/sx8.c index 08586dc14e85..4d90e5eba2f5 100644 --- a/drivers/block/sx8.c +++ b/drivers/block/sx8.c @@ -567,7 +567,7 @@ static struct carm_request *carm_get_special(struct carm_host *host) if (!crq) return NULL; - rq = blk_get_request(host->oob_q, REQ_OP_DRV_OUT, GFP_KERNEL); + rq = blk_get_request(host->oob_q, REQ_OP_DRV_OUT, 0); if (IS_ERR(rq)) { spin_lock_irqsave(&host->lock, flags); carm_put_request(host, crq); diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 4a07593c2efd..23752dc99b00 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -298,7 +298,7 @@ static int virtblk_get_id(struct gendisk *disk, char *id_str) struct request *req; int err; - req = blk_get_request(q, REQ_OP_DRV_IN, GFP_KERNEL); + req = blk_get_request(q, REQ_OP_DRV_IN, 0); if (IS_ERR(req)) return PTR_ERR(req); @@ -371,7 +371,7 @@ static ssize_t virtblk_serial_show(struct device *dev, return err; } -static DEVICE_ATTR(serial, S_IRUGO, virtblk_serial_show, NULL); +static DEVICE_ATTR(serial, 0444, virtblk_serial_show, NULL); /* The queue's logical block size must be set before calling this */ static void virtblk_update_capacity(struct virtio_blk *vblk, bool resize) @@ -576,10 +576,10 @@ virtblk_cache_type_show(struct device *dev, struct device_attribute *attr, } static const struct device_attribute dev_attr_cache_type_ro = - __ATTR(cache_type, S_IRUGO, + __ATTR(cache_type, 0444, virtblk_cache_type_show, NULL); static const struct device_attribute dev_attr_cache_type_rw = - __ATTR(cache_type, S_IRUGO|S_IWUSR, + __ATTR(cache_type, 0644, virtblk_cache_type_show, virtblk_cache_type_store); static int virtblk_init_request(struct blk_mq_tag_set *set, struct request *rq, diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c index 987d665e82de..b55b245e8052 100644 --- a/drivers/block/xen-blkback/blkback.c +++ b/drivers/block/xen-blkback/blkback.c @@ -98,7 +98,7 @@ MODULE_PARM_DESC(max_queues, * backend, 4KB page granularity is used. */ unsigned int xen_blkif_max_ring_order = XENBUS_MAX_RING_GRANT_ORDER; -module_param_named(max_ring_page_order, xen_blkif_max_ring_order, int, S_IRUGO); +module_param_named(max_ring_page_order, xen_blkif_max_ring_order, int, 0444); MODULE_PARM_DESC(max_ring_page_order, "Maximum order of pages to be used for the shared ring"); /* * The LRU mechanism to clean the lists of persistent grants needs to diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c index 21c1be1eb226..a4bc74e72c39 100644 --- a/drivers/block/xen-blkback/xenbus.c +++ b/drivers/block/xen-blkback/xenbus.c @@ -139,7 +139,8 @@ static int xen_blkif_alloc_rings(struct xen_blkif *blkif) { unsigned int r; - blkif->rings = kzalloc(blkif->nr_rings * sizeof(struct xen_blkif_ring), GFP_KERNEL); + blkif->rings = kcalloc(blkif->nr_rings, sizeof(struct xen_blkif_ring), + GFP_KERNEL); if (!blkif->rings) return -ENOMEM; @@ -367,7 +368,7 @@ int __init xen_blkif_interface_init(void) out: \ return sprintf(buf, format, result); \ } \ - static DEVICE_ATTR(name, S_IRUGO, show_##name, NULL) + static DEVICE_ATTR(name, 0444, show_##name, NULL) VBD_SHOW_ALLRING(oo_req, "%llu\n"); VBD_SHOW_ALLRING(rd_req, "%llu\n"); @@ -403,7 +404,7 @@ static const struct attribute_group xen_vbdstat_group = { \ return sprintf(buf, format, ##args); \ } \ - static DEVICE_ATTR(name, S_IRUGO, show_##name, NULL) + static DEVICE_ATTR(name, 0444, show_##name, NULL) VBD_SHOW(physical_device, "%x:%x\n", be->major, be->minor); VBD_SHOW(mode, "%s\n", be->mode); diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index 2a8e7813bd1a..b5cedccb5d7d 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -129,13 +129,12 @@ static const struct block_device_operations xlvbd_block_fops; */ static unsigned int xen_blkif_max_segments = 32; -module_param_named(max_indirect_segments, xen_blkif_max_segments, uint, - S_IRUGO); +module_param_named(max_indirect_segments, xen_blkif_max_segments, uint, 0444); MODULE_PARM_DESC(max_indirect_segments, "Maximum amount of segments in indirect requests (default is 32)"); static unsigned int xen_blkif_max_queues = 4; -module_param_named(max_queues, xen_blkif_max_queues, uint, S_IRUGO); +module_param_named(max_queues, xen_blkif_max_queues, uint, 0444); MODULE_PARM_DESC(max_queues, "Maximum number of hardware queues/rings used per virtual disk"); /* @@ -143,7 +142,7 @@ MODULE_PARM_DESC(max_queues, "Maximum number of hardware queues/rings used per v * backend, 4KB page granularity is used. */ static unsigned int xen_blkif_max_ring_order; -module_param_named(max_ring_page_order, xen_blkif_max_ring_order, int, S_IRUGO); +module_param_named(max_ring_page_order, xen_blkif_max_ring_order, int, 0444); MODULE_PARM_DESC(max_ring_page_order, "Maximum order of pages to be used for the shared ring"); #define BLK_RING_SIZE(info) \ @@ -1907,7 +1906,9 @@ static int negotiate_mq(struct blkfront_info *info) if (!info->nr_rings) info->nr_rings = 1; - info->rinfo = kzalloc(sizeof(struct blkfront_ring_info) * info->nr_rings, GFP_KERNEL); + info->rinfo = kcalloc(info->nr_rings, + sizeof(struct blkfront_ring_info), + GFP_KERNEL); if (!info->rinfo) { xenbus_dev_fatal(info->xbdev, -ENOMEM, "allocating ring_info structure"); return -ENOMEM; @@ -2217,15 +2218,18 @@ static int blkfront_setup_indirect(struct blkfront_ring_info *rinfo) } for (i = 0; i < BLK_RING_SIZE(info); i++) { - rinfo->shadow[i].grants_used = kzalloc( - sizeof(rinfo->shadow[i].grants_used[0]) * grants, - GFP_NOIO); - rinfo->shadow[i].sg = kzalloc(sizeof(rinfo->shadow[i].sg[0]) * psegs, GFP_NOIO); - if (info->max_indirect_segments) - rinfo->shadow[i].indirect_grants = kzalloc( - sizeof(rinfo->shadow[i].indirect_grants[0]) * - INDIRECT_GREFS(grants), + rinfo->shadow[i].grants_used = + kcalloc(grants, + sizeof(rinfo->shadow[i].grants_used[0]), GFP_NOIO); + rinfo->shadow[i].sg = kcalloc(psegs, + sizeof(rinfo->shadow[i].sg[0]), + GFP_NOIO); + if (info->max_indirect_segments) + rinfo->shadow[i].indirect_grants = + kcalloc(INDIRECT_GREFS(grants), + sizeof(rinfo->shadow[i].indirect_grants[0]), + GFP_NOIO); if ((rinfo->shadow[i].grants_used == NULL) || (rinfo->shadow[i].sg == NULL) || (info->max_indirect_segments && diff --git a/drivers/block/z2ram.c b/drivers/block/z2ram.c index 8f9130ab5887..d0c5bc4e0703 100644 --- a/drivers/block/z2ram.c +++ b/drivers/block/z2ram.c @@ -197,8 +197,9 @@ static int z2_open(struct block_device *bdev, fmode_t mode) vaddr = (unsigned long)z_remap_nocache_nonser(paddr, size); #endif z2ram_map = - kmalloc((size/Z2RAM_CHUNKSIZE)*sizeof(z2ram_map[0]), - GFP_KERNEL); + kmalloc_array(size / Z2RAM_CHUNKSIZE, + sizeof(z2ram_map[0]), + GFP_KERNEL); if ( z2ram_map == NULL ) { printk( KERN_ERR DEVICE_NAME diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig index ac3a31d433b2..635235759a0a 100644 --- a/drivers/block/zram/Kconfig +++ b/drivers/block/zram/Kconfig @@ -13,7 +13,7 @@ config ZRAM It has several use cases, for example: /tmp storage, use as swap disks and maybe many more. - See zram.txt for more information. + See Documentation/blockdev/zram.txt for more information. config ZRAM_WRITEBACK bool "Write back incompressible page to backing device" @@ -25,4 +25,14 @@ config ZRAM_WRITEBACK For this feature, admin should set up backing device via /sys/block/zramX/backing_dev. - See zram.txt for more infomration. + See Documentation/blockdev/zram.txt for more information. + +config ZRAM_MEMORY_TRACKING + bool "Track zRam block status" + depends on ZRAM && DEBUG_FS + help + With this feature, admin can track the state of allocated blocks + of zRAM. Admin could see the information via + /sys/kernel/debug/zram/zramX/block_state. + + See Documentation/blockdev/zram.txt for more information. diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 0f3fadd71230..7436b2d27fa3 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -31,6 +31,7 @@ #include #include #include +#include #include #include "zram_drv.h" @@ -52,11 +53,28 @@ static size_t huge_class_size; static void zram_free_page(struct zram *zram, size_t index); +static void zram_slot_lock(struct zram *zram, u32 index) +{ + bit_spin_lock(ZRAM_LOCK, &zram->table[index].value); +} + +static void zram_slot_unlock(struct zram *zram, u32 index) +{ + bit_spin_unlock(ZRAM_LOCK, &zram->table[index].value); +} + static inline bool init_done(struct zram *zram) { return zram->disksize; } +static inline bool zram_allocated(struct zram *zram, u32 index) +{ + + return (zram->table[index].value >> (ZRAM_FLAG_SHIFT + 1)) || + zram->table[index].handle; +} + static inline struct zram *dev_to_zram(struct device *dev) { return (struct zram *)dev_to_disk(dev)->private_data; @@ -73,7 +91,7 @@ static void zram_set_handle(struct zram *zram, u32 index, unsigned long handle) } /* flag operations require table entry bit_spin_lock() being held */ -static int zram_test_flag(struct zram *zram, u32 index, +static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) { return zram->table[index].value & BIT(flag); @@ -600,6 +618,114 @@ static int read_from_bdev(struct zram *zram, struct bio_vec *bvec, static void zram_wb_clear(struct zram *zram, u32 index) {} #endif +#ifdef CONFIG_ZRAM_MEMORY_TRACKING + +static struct dentry *zram_debugfs_root; + +static void zram_debugfs_create(void) +{ + zram_debugfs_root = debugfs_create_dir("zram", NULL); +} + +static void zram_debugfs_destroy(void) +{ + debugfs_remove_recursive(zram_debugfs_root); +} + +static void zram_accessed(struct zram *zram, u32 index) +{ + zram->table[index].ac_time = ktime_get_boottime(); +} + +static void zram_reset_access(struct zram *zram, u32 index) +{ + zram->table[index].ac_time = 0; +} + +static ssize_t read_block_state(struct file *file, char __user *buf, + size_t count, loff_t *ppos) +{ + char *kbuf; + ssize_t index, written = 0; + struct zram *zram = file->private_data; + unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; + struct timespec64 ts; + + kbuf = kvmalloc(count, GFP_KERNEL); + if (!kbuf) + return -ENOMEM; + + down_read(&zram->init_lock); + if (!init_done(zram)) { + up_read(&zram->init_lock); + kvfree(kbuf); + return -EINVAL; + } + + for (index = *ppos; index < nr_pages; index++) { + int copied; + + zram_slot_lock(zram, index); + if (!zram_allocated(zram, index)) + goto next; + + ts = ktime_to_timespec64(zram->table[index].ac_time); + copied = snprintf(kbuf + written, count, + "%12zd %12lld.%06lu %c%c%c\n", + index, (s64)ts.tv_sec, + ts.tv_nsec / NSEC_PER_USEC, + zram_test_flag(zram, index, ZRAM_SAME) ? 's' : '.', + zram_test_flag(zram, index, ZRAM_WB) ? 'w' : '.', + zram_test_flag(zram, index, ZRAM_HUGE) ? 'h' : '.'); + + if (count < copied) { + zram_slot_unlock(zram, index); + break; + } + written += copied; + count -= copied; +next: + zram_slot_unlock(zram, index); + *ppos += 1; + } + + up_read(&zram->init_lock); + if (copy_to_user(buf, kbuf, written)) + written = -EFAULT; + kvfree(kbuf); + + return written; +} + +static const struct file_operations proc_zram_block_state_op = { + .open = simple_open, + .read = read_block_state, + .llseek = default_llseek, +}; + +static void zram_debugfs_register(struct zram *zram) +{ + if (!zram_debugfs_root) + return; + + zram->debugfs_dir = debugfs_create_dir(zram->disk->disk_name, + zram_debugfs_root); + debugfs_create_file("block_state", 0400, zram->debugfs_dir, + zram, &proc_zram_block_state_op); +} + +static void zram_debugfs_unregister(struct zram *zram) +{ + debugfs_remove_recursive(zram->debugfs_dir); +} +#else +static void zram_debugfs_create(void) {}; +static void zram_debugfs_destroy(void) {}; +static void zram_accessed(struct zram *zram, u32 index) {}; +static void zram_reset_access(struct zram *zram, u32 index) {}; +static void zram_debugfs_register(struct zram *zram) {}; +static void zram_debugfs_unregister(struct zram *zram) {}; +#endif /* * We switched to per-cpu streams and this attr is not needed anymore. @@ -719,14 +845,15 @@ static ssize_t mm_stat_show(struct device *dev, max_used = atomic_long_read(&zram->stats.max_used_pages); ret = scnprintf(buf, PAGE_SIZE, - "%8llu %8llu %8llu %8lu %8ld %8llu %8lu\n", + "%8llu %8llu %8llu %8lu %8ld %8llu %8lu %8llu\n", orig_size << PAGE_SHIFT, (u64)atomic64_read(&zram->stats.compr_data_size), mem_used << PAGE_SHIFT, zram->limit_pages << PAGE_SHIFT, max_used << PAGE_SHIFT, (u64)atomic64_read(&zram->stats.same_pages), - pool_stats.pages_compacted); + pool_stats.pages_compacted, + (u64)atomic64_read(&zram->stats.huge_pages)); up_read(&zram->init_lock); return ret; @@ -753,16 +880,6 @@ static DEVICE_ATTR_RO(io_stat); static DEVICE_ATTR_RO(mm_stat); static DEVICE_ATTR_RO(debug_stat); -static void zram_slot_lock(struct zram *zram, u32 index) -{ - bit_spin_lock(ZRAM_ACCESS, &zram->table[index].value); -} - -static void zram_slot_unlock(struct zram *zram, u32 index) -{ - bit_spin_unlock(ZRAM_ACCESS, &zram->table[index].value); -} - static void zram_meta_free(struct zram *zram, u64 disksize) { size_t num_pages = disksize >> PAGE_SHIFT; @@ -781,7 +898,7 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) size_t num_pages; num_pages = disksize >> PAGE_SHIFT; - zram->table = vzalloc(num_pages * sizeof(*zram->table)); + zram->table = vzalloc(array_size(num_pages, sizeof(*zram->table))); if (!zram->table) return false; @@ -805,6 +922,13 @@ static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; + zram_reset_access(zram, index); + + if (zram_test_flag(zram, index, ZRAM_HUGE)) { + zram_clear_flag(zram, index, ZRAM_HUGE); + atomic64_dec(&zram->stats.huge_pages); + } + if (zram_wb_enabled(zram) && zram_test_flag(zram, index, ZRAM_WB)) { zram_wb_clear(zram, index); atomic64_dec(&zram->stats.pages_stored); @@ -973,6 +1097,7 @@ compress_again: } if (unlikely(comp_len >= huge_class_size)) { + comp_len = PAGE_SIZE; if (zram_wb_enabled(zram) && allow_wb) { zcomp_stream_put(zram->comp); ret = write_to_bdev(zram, bvec, index, bio, &element); @@ -984,7 +1109,6 @@ compress_again: allow_wb = false; goto compress_again; } - comp_len = PAGE_SIZE; } /* @@ -1046,6 +1170,11 @@ out: zram_slot_lock(zram, index); zram_free_page(zram, index); + if (comp_len == PAGE_SIZE) { + zram_set_flag(zram, index, ZRAM_HUGE); + atomic64_inc(&zram->stats.huge_pages); + } + if (flags) { zram_set_flag(zram, index, flags); zram_set_element(zram, index, element); @@ -1166,6 +1295,10 @@ static int zram_bvec_rw(struct zram *zram, struct bio_vec *bvec, u32 index, generic_end_io_acct(q, rw_acct, &zram->disk->part0, start_time); + zram_slot_lock(zram, index); + zram_accessed(zram, index); + zram_slot_unlock(zram, index); + if (unlikely(ret < 0)) { if (!is_write) atomic64_inc(&zram->stats.failed_reads); @@ -1577,6 +1710,7 @@ static int zram_add(void) } strlcpy(zram->compressor, default_compressor, sizeof(zram->compressor)); + zram_debugfs_register(zram); pr_info("Added device: %s\n", zram->disk->disk_name); return device_id; @@ -1610,6 +1744,7 @@ static int zram_remove(struct zram *zram) zram->claim = true; mutex_unlock(&bdev->bd_mutex); + zram_debugfs_unregister(zram); /* * Remove sysfs first, so no one will perform a disksize * store while we destroy the devices. This also helps during @@ -1712,6 +1847,7 @@ static void destroy_devices(void) { class_unregister(&zram_control_class); idr_for_each(&zram_index_idr, &zram_remove_cb, NULL); + zram_debugfs_destroy(); idr_destroy(&zram_index_idr); unregister_blkdev(zram_major, "zram"); cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); @@ -1733,6 +1869,7 @@ static int __init zram_init(void) return ret; } + zram_debugfs_create(); zram_major = register_blkdev(0, "zram"); if (zram_major <= 0) { pr_err("Unable to get major number\n"); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 008861220723..72c8584b6dff 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -43,10 +43,11 @@ /* Flags for zram pages (table[page_no].value) */ enum zram_pageflags { - /* Page consists the same element */ - ZRAM_SAME = ZRAM_FLAG_SHIFT, - ZRAM_ACCESS, /* page is now accessed */ + /* zram slot is locked */ + ZRAM_LOCK = ZRAM_FLAG_SHIFT, + ZRAM_SAME, /* Page consists the same element */ ZRAM_WB, /* page is stored on backing_device */ + ZRAM_HUGE, /* Incompressible page */ __NR_ZRAM_PAGEFLAGS, }; @@ -60,6 +61,9 @@ struct zram_table_entry { unsigned long element; }; unsigned long value; +#ifdef CONFIG_ZRAM_MEMORY_TRACKING + ktime_t ac_time; +#endif }; struct zram_stats { @@ -71,6 +75,7 @@ struct zram_stats { atomic64_t invalid_io; /* non-page-aligned I/O requests */ atomic64_t notify_free; /* no. of swap slot free notifications */ atomic64_t same_pages; /* no. of same element filled pages */ + atomic64_t huge_pages; /* no. of huge pages */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ atomic64_t writestall; /* no. of write slow paths */ @@ -107,5 +112,8 @@ struct zram { unsigned long nr_pages; spinlock_t bitmap_lock; #endif +#ifdef CONFIG_ZRAM_MEMORY_TRACKING + struct dentry *debugfs_dir; +#endif }; #endif diff --git a/drivers/bluetooth/Kconfig b/drivers/bluetooth/Kconfig index 010f5f579e68..f3c643a0473c 100644 --- a/drivers/bluetooth/Kconfig +++ b/drivers/bluetooth/Kconfig @@ -197,6 +197,7 @@ config BT_HCIUART_BCM config BT_HCIUART_QCA bool "Qualcomm Atheros protocol support" depends on BT_HCIUART + depends on BT_HCIUART_SERDEV select BT_HCIUART_H4 select BT_QCA help diff --git a/drivers/bluetooth/btbcm.c b/drivers/bluetooth/btbcm.c index 6659f113042c..99cde1f9467d 100644 --- a/drivers/bluetooth/btbcm.c +++ b/drivers/bluetooth/btbcm.c @@ -315,10 +315,12 @@ static int btbcm_read_info(struct hci_dev *hdev) return 0; } -static const struct { +struct bcm_subver_table { u16 subver; const char *name; -} bcm_uart_subver_table[] = { +}; + +static const struct bcm_subver_table bcm_uart_subver_table[] = { { 0x4103, "BCM4330B1" }, /* 002.001.003 */ { 0x410e, "BCM43341B0" }, /* 002.001.014 */ { 0x4406, "BCM4324B3" }, /* 002.004.006 */ @@ -330,12 +332,28 @@ static const struct { { } }; -int btbcm_initialize(struct hci_dev *hdev, char *fw_name, size_t len) +static const struct bcm_subver_table bcm_usb_subver_table[] = { + { 0x210b, "BCM43142A0" }, /* 001.001.011 */ + { 0x2112, "BCM4314A0" }, /* 001.001.018 */ + { 0x2118, "BCM20702A0" }, /* 001.001.024 */ + { 0x2126, "BCM4335A0" }, /* 001.001.038 */ + { 0x220e, "BCM20702A1" }, /* 001.002.014 */ + { 0x230f, "BCM4354A2" }, /* 001.003.015 */ + { 0x4106, "BCM4335B0" }, /* 002.001.006 */ + { 0x410e, "BCM20702B0" }, /* 002.001.014 */ + { 0x6109, "BCM4335C0" }, /* 003.001.009 */ + { 0x610c, "BCM4354" }, /* 003.001.012 */ + { } +}; + +int btbcm_initialize(struct hci_dev *hdev, char *fw_name, size_t len, + bool reinit) { - u16 subver, rev; - const char *hw_name = NULL; + u16 subver, rev, pid, vid; + const char *hw_name = "BCM"; struct sk_buff *skb; struct hci_rp_read_local_version *ver; + const struct bcm_subver_table *bcm_subver_table; int i, err; /* Reset */ @@ -354,30 +372,44 @@ int btbcm_initialize(struct hci_dev *hdev, char *fw_name, size_t len) kfree_skb(skb); /* Read controller information */ - err = btbcm_read_info(hdev); - if (err) - return err; + if (!reinit) { + err = btbcm_read_info(hdev); + if (err) + return err; + } - switch ((rev & 0xf000) >> 12) { - case 0: - case 1: - case 2: - case 3: - for (i = 0; bcm_uart_subver_table[i].name; i++) { - if (subver == bcm_uart_subver_table[i].subver) { - hw_name = bcm_uart_subver_table[i].name; - break; - } + /* Upper nibble of rev should be between 0 and 3? */ + if (((rev & 0xf000) >> 12) > 3) + return 0; + + bcm_subver_table = (hdev->bus == HCI_USB) ? bcm_usb_subver_table : + bcm_uart_subver_table; + + for (i = 0; bcm_subver_table[i].name; i++) { + if (subver == bcm_subver_table[i].subver) { + hw_name = bcm_subver_table[i].name; + break; } + } - snprintf(fw_name, len, "brcm/%s.hcd", hw_name ? : "BCM"); - break; - default: - return 0; + if (hdev->bus == HCI_USB) { + /* Read USB Product Info */ + skb = btbcm_read_usb_product(hdev); + if (IS_ERR(skb)) + return PTR_ERR(skb); + + vid = get_unaligned_le16(skb->data + 1); + pid = get_unaligned_le16(skb->data + 3); + kfree_skb(skb); + + snprintf(fw_name, len, "brcm/%s-%4.4x-%4.4x.hcd", + hw_name, vid, pid); + } else { + snprintf(fw_name, len, "brcm/%s.hcd", hw_name); } bt_dev_info(hdev, "%s (%3.3u.%3.3u.%3.3u) build %4.4u", - hw_name ? : "BCM", (subver & 0xe000) >> 13, + hw_name, (subver & 0xe000) >> 13, (subver & 0x1f00) >> 8, (subver & 0x00ff), rev & 0x0fff); return 0; @@ -386,30 +418,14 @@ EXPORT_SYMBOL_GPL(btbcm_initialize); int btbcm_finalize(struct hci_dev *hdev) { - struct sk_buff *skb; - struct hci_rp_read_local_version *ver; - u16 subver, rev; + char fw_name[64]; int err; - /* Reset */ - err = btbcm_reset(hdev); + /* Re-initialize */ + err = btbcm_initialize(hdev, fw_name, sizeof(fw_name), true); if (err) return err; - /* Read Local Version Info */ - skb = btbcm_read_local_version(hdev); - if (IS_ERR(skb)) - return PTR_ERR(skb); - - ver = (struct hci_rp_read_local_version *)skb->data; - rev = le16_to_cpu(ver->hci_rev); - subver = le16_to_cpu(ver->lmp_subver); - kfree_skb(skb); - - bt_dev_info(hdev, "BCM (%3.3u.%3.3u.%3.3u) build %4.4u", - (subver & 0xe000) >> 13, (subver & 0x1f00) >> 8, - (subver & 0x00ff), rev & 0x0fff); - btbcm_check_bdaddr(hdev); set_bit(HCI_QUIRK_STRICT_DUPLICATE_FILTER, &hdev->quirks); @@ -418,95 +434,18 @@ int btbcm_finalize(struct hci_dev *hdev) } EXPORT_SYMBOL_GPL(btbcm_finalize); -static const struct { - u16 subver; - const char *name; -} bcm_usb_subver_table[] = { - { 0x210b, "BCM43142A0" }, /* 001.001.011 */ - { 0x2112, "BCM4314A0" }, /* 001.001.018 */ - { 0x2118, "BCM20702A0" }, /* 001.001.024 */ - { 0x2126, "BCM4335A0" }, /* 001.001.038 */ - { 0x220e, "BCM20702A1" }, /* 001.002.014 */ - { 0x230f, "BCM4354A2" }, /* 001.003.015 */ - { 0x4106, "BCM4335B0" }, /* 002.001.006 */ - { 0x410e, "BCM20702B0" }, /* 002.001.014 */ - { 0x6109, "BCM4335C0" }, /* 003.001.009 */ - { 0x610c, "BCM4354" }, /* 003.001.012 */ - { } -}; - int btbcm_setup_patchram(struct hci_dev *hdev) { char fw_name[64]; const struct firmware *fw; - u16 subver, rev, pid, vid; - const char *hw_name = NULL; struct sk_buff *skb; - struct hci_rp_read_local_version *ver; - int i, err; - - /* Reset */ - err = btbcm_reset(hdev); - if (err) - return err; - - /* Read Local Version Info */ - skb = btbcm_read_local_version(hdev); - if (IS_ERR(skb)) - return PTR_ERR(skb); - - ver = (struct hci_rp_read_local_version *)skb->data; - rev = le16_to_cpu(ver->hci_rev); - subver = le16_to_cpu(ver->lmp_subver); - kfree_skb(skb); + int err; - /* Read controller information */ - err = btbcm_read_info(hdev); + /* Initialize */ + err = btbcm_initialize(hdev, fw_name, sizeof(fw_name), false); if (err) return err; - switch ((rev & 0xf000) >> 12) { - case 0: - case 3: - for (i = 0; bcm_uart_subver_table[i].name; i++) { - if (subver == bcm_uart_subver_table[i].subver) { - hw_name = bcm_uart_subver_table[i].name; - break; - } - } - - snprintf(fw_name, sizeof(fw_name), "brcm/%s.hcd", - hw_name ? : "BCM"); - break; - case 1: - case 2: - /* Read USB Product Info */ - skb = btbcm_read_usb_product(hdev); - if (IS_ERR(skb)) - return PTR_ERR(skb); - - vid = get_unaligned_le16(skb->data + 1); - pid = get_unaligned_le16(skb->data + 3); - kfree_skb(skb); - - for (i = 0; bcm_usb_subver_table[i].name; i++) { - if (subver == bcm_usb_subver_table[i].subver) { - hw_name = bcm_usb_subver_table[i].name; - break; - } - } - - snprintf(fw_name, sizeof(fw_name), "brcm/%s-%4.4x-%4.4x.hcd", - hw_name ? : "BCM", vid, pid); - break; - default: - return 0; - } - - bt_dev_info(hdev, "%s (%3.3u.%3.3u.%3.3u) build %4.4u", - hw_name ? : "BCM", (subver & 0xe000) >> 13, - (subver & 0x1f00) >> 8, (subver & 0x00ff), rev & 0x0fff); - err = request_firmware(&fw, fw_name, &hdev->dev); if (err < 0) { bt_dev_info(hdev, "BCM: Patch %s not found", fw_name); @@ -517,25 +456,11 @@ int btbcm_setup_patchram(struct hci_dev *hdev) release_firmware(fw); - /* Reset */ - err = btbcm_reset(hdev); + /* Re-initialize */ + err = btbcm_initialize(hdev, fw_name, sizeof(fw_name), true); if (err) return err; - /* Read Local Version Info */ - skb = btbcm_read_local_version(hdev); - if (IS_ERR(skb)) - return PTR_ERR(skb); - - ver = (struct hci_rp_read_local_version *)skb->data; - rev = le16_to_cpu(ver->hci_rev); - subver = le16_to_cpu(ver->lmp_subver); - kfree_skb(skb); - - bt_dev_info(hdev, "%s (%3.3u.%3.3u.%3.3u) build %4.4u", - hw_name ? : "BCM", (subver & 0xe000) >> 13, - (subver & 0x1f00) >> 8, (subver & 0x00ff), rev & 0x0fff); - /* Read Local Name */ skb = btbcm_read_local_name(hdev); if (IS_ERR(skb)) diff --git a/drivers/bluetooth/btbcm.h b/drivers/bluetooth/btbcm.h index cfe6ad4cc621..5346515c880c 100644 --- a/drivers/bluetooth/btbcm.h +++ b/drivers/bluetooth/btbcm.h @@ -73,7 +73,8 @@ int btbcm_patchram(struct hci_dev *hdev, const struct firmware *fw); int btbcm_setup_patchram(struct hci_dev *hdev); int btbcm_setup_apple(struct hci_dev *hdev); -int btbcm_initialize(struct hci_dev *hdev, char *fw_name, size_t len); +int btbcm_initialize(struct hci_dev *hdev, char *fw_name, size_t len, + bool reinit); int btbcm_finalize(struct hci_dev *hdev); #else @@ -104,7 +105,7 @@ static inline int btbcm_setup_apple(struct hci_dev *hdev) } static inline int btbcm_initialize(struct hci_dev *hdev, char *fw_name, - size_t len) + size_t len, bool reinit) { return 0; } diff --git a/drivers/bluetooth/btmrvl_debugfs.c b/drivers/bluetooth/btmrvl_debugfs.c index 1828ed8cae7a..c4867576be00 100644 --- a/drivers/bluetooth/btmrvl_debugfs.c +++ b/drivers/bluetooth/btmrvl_debugfs.c @@ -35,15 +35,9 @@ static ssize_t btmrvl_hscfgcmd_write(struct file *file, const char __user *ubuf, size_t count, loff_t *ppos) { struct btmrvl_private *priv = file->private_data; - char buf[16]; long result, ret; - memset(buf, 0, sizeof(buf)); - - if (copy_from_user(&buf, ubuf, min_t(size_t, sizeof(buf) - 1, count))) - return -EFAULT; - - ret = kstrtol(buf, 10, &result); + ret = kstrtol_from_user(ubuf, count, 10, &result); if (ret) return ret; @@ -81,15 +75,9 @@ static ssize_t btmrvl_pscmd_write(struct file *file, const char __user *ubuf, size_t count, loff_t *ppos) { struct btmrvl_private *priv = file->private_data; - char buf[16]; long result, ret; - memset(buf, 0, sizeof(buf)); - - if (copy_from_user(&buf, ubuf, min_t(size_t, sizeof(buf) - 1, count))) - return -EFAULT; - - ret = kstrtol(buf, 10, &result); + ret = kstrtol_from_user(ubuf, count, 10, &result); if (ret) return ret; @@ -127,15 +115,9 @@ static ssize_t btmrvl_hscmd_write(struct file *file, const char __user *ubuf, size_t count, loff_t *ppos) { struct btmrvl_private *priv = file->private_data; - char buf[16]; long result, ret; - memset(buf, 0, sizeof(buf)); - - if (copy_from_user(&buf, ubuf, min_t(size_t, sizeof(buf) - 1, count))) - return -EFAULT; - - ret = kstrtol(buf, 10, &result); + ret = kstrtol_from_user(ubuf, count, 10, &result); if (ret) return ret; @@ -167,35 +149,6 @@ static const struct file_operations btmrvl_hscmd_fops = { .llseek = default_llseek, }; -static ssize_t btmrvl_fwdump_write(struct file *file, const char __user *ubuf, - size_t count, loff_t *ppos) -{ - struct btmrvl_private *priv = file->private_data; - char buf[16]; - bool result; - - memset(buf, 0, sizeof(buf)); - - if (copy_from_user(&buf, ubuf, min_t(size_t, sizeof(buf) - 1, count))) - return -EFAULT; - - if (strtobool(buf, &result)) - return -EINVAL; - - if (!result) - return -EINVAL; - - btmrvl_firmware_dump(priv); - - return count; -} - -static const struct file_operations btmrvl_fwdump_fops = { - .write = btmrvl_fwdump_write, - .open = simple_open, - .llseek = default_llseek, -}; - void btmrvl_debugfs_init(struct hci_dev *hdev) { struct btmrvl_private *priv = hci_get_drvdata(hdev); @@ -226,8 +179,6 @@ void btmrvl_debugfs_init(struct hci_dev *hdev) priv, &btmrvl_hscmd_fops); debugfs_create_file("hscfgcmd", 0644, dbg->config_dir, priv, &btmrvl_hscfgcmd_fops); - debugfs_create_file("fw_dump", 0200, dbg->config_dir, - priv, &btmrvl_fwdump_fops); dbg->status_dir = debugfs_create_dir("status", hdev->debugfs); debugfs_create_u8("curpsmode", 0444, dbg->status_dir, diff --git a/drivers/bluetooth/btmrvl_drv.h b/drivers/bluetooth/btmrvl_drv.h index fc3caf4541ba..f0454541e5fd 100644 --- a/drivers/bluetooth/btmrvl_drv.h +++ b/drivers/bluetooth/btmrvl_drv.h @@ -110,7 +110,6 @@ struct btmrvl_private { u8 *payload, u16 nb); int (*hw_wakeup_firmware)(struct btmrvl_private *priv); int (*hw_process_int_status)(struct btmrvl_private *priv); - void (*firmware_dump)(struct btmrvl_private *priv); spinlock_t driver_lock; /* spinlock used by driver */ #ifdef CONFIG_DEBUG_FS void *debugfs_data; @@ -183,7 +182,6 @@ int btmrvl_send_hscfg_cmd(struct btmrvl_private *priv); int btmrvl_enable_ps(struct btmrvl_private *priv); int btmrvl_prepare_command(struct btmrvl_private *priv); int btmrvl_enable_hs(struct btmrvl_private *priv); -void btmrvl_firmware_dump(struct btmrvl_private *priv); #ifdef CONFIG_DEBUG_FS void btmrvl_debugfs_init(struct hci_dev *hdev); diff --git a/drivers/bluetooth/btmrvl_main.c b/drivers/bluetooth/btmrvl_main.c index f6c694a1b9b0..708ad21683eb 100644 --- a/drivers/bluetooth/btmrvl_main.c +++ b/drivers/bluetooth/btmrvl_main.c @@ -358,12 +358,6 @@ int btmrvl_prepare_command(struct btmrvl_private *priv) return ret; } -void btmrvl_firmware_dump(struct btmrvl_private *priv) -{ - if (priv->firmware_dump) - priv->firmware_dump(priv); -} - static int btmrvl_tx_pkt(struct btmrvl_private *priv, struct sk_buff *skb) { int ret = 0; diff --git a/drivers/bluetooth/btmrvl_sdio.c b/drivers/bluetooth/btmrvl_sdio.c index 6f99b9f3d57f..888bac49a87b 100644 --- a/drivers/bluetooth/btmrvl_sdio.c +++ b/drivers/bluetooth/btmrvl_sdio.c @@ -1311,9 +1311,11 @@ rdwr_status btmrvl_sdio_rdwr_firmware(struct btmrvl_private *priv, } /* This function dump sdio register and memory data */ -static void btmrvl_sdio_dump_firmware(struct btmrvl_private *priv) +static void btmrvl_sdio_coredump(struct device *dev) { - struct btmrvl_sdio_card *card = priv->btmrvl_dev.card; + struct sdio_func *func = dev_to_sdio_func(dev); + struct btmrvl_sdio_card *card; + struct btmrvl_private *priv; int ret = 0; unsigned int reg, reg_start, reg_end; enum rdwr_status stat; @@ -1321,6 +1323,9 @@ static void btmrvl_sdio_dump_firmware(struct btmrvl_private *priv) u8 dump_num = 0, idx, i, read_reg, doneflag = 0; u32 memory_size, fw_dump_len = 0; + card = sdio_get_drvdata(func); + priv = card->priv; + /* dump sdio register first */ btmrvl_sdio_dump_regs(priv); @@ -1547,7 +1552,6 @@ static int btmrvl_sdio_probe(struct sdio_func *func, priv->hw_host_to_card = btmrvl_sdio_host_to_card; priv->hw_wakeup_firmware = btmrvl_sdio_wakeup_fw; priv->hw_process_int_status = btmrvl_sdio_process_int_status; - priv->firmware_dump = btmrvl_sdio_dump_firmware; if (btmrvl_register_hdev(priv)) { BT_ERR("Register hdev failed!"); @@ -1717,6 +1721,7 @@ static struct sdio_driver bt_mrvl_sdio = { .remove = btmrvl_sdio_remove, .drv = { .owner = THIS_MODULE, + .coredump = btmrvl_sdio_coredump, .pm = &btmrvl_sdio_pm_ops, } }; diff --git a/drivers/bluetooth/btqca.c b/drivers/bluetooth/btqca.c index 2793d4180d2f..8219816c54a0 100644 --- a/drivers/bluetooth/btqca.c +++ b/drivers/bluetooth/btqca.c @@ -127,28 +127,41 @@ static void rome_tlv_check_data(struct rome_config *config, BT_DBG("TLV Type\t\t : 0x%x", type_len & 0x000000ff); BT_DBG("Length\t\t : %d bytes", length); + config->dnld_mode = ROME_SKIP_EVT_NONE; + switch (config->type) { case TLV_TYPE_PATCH: tlv_patch = (struct tlv_type_patch *)tlv->data; - BT_DBG("Total Length\t\t : %d bytes", + + /* For Rome version 1.1 to 3.1, all segment commands + * are acked by a vendor specific event (VSE). + * For Rome >= 3.2, the download mode field indicates + * if VSE is skipped by the controller. + * In case VSE is skipped, only the last segment is acked. + */ + config->dnld_mode = tlv_patch->download_mode; + + BT_DBG("Total Length : %d bytes", le32_to_cpu(tlv_patch->total_size)); - BT_DBG("Patch Data Length\t : %d bytes", + BT_DBG("Patch Data Length : %d bytes", le32_to_cpu(tlv_patch->data_length)); BT_DBG("Signing Format Version : 0x%x", tlv_patch->format_version); - BT_DBG("Signature Algorithm\t : 0x%x", + BT_DBG("Signature Algorithm : 0x%x", tlv_patch->signature); - BT_DBG("Reserved\t\t : 0x%x", - le16_to_cpu(tlv_patch->reserved1)); - BT_DBG("Product ID\t\t : 0x%04x", + BT_DBG("Download mode : 0x%x", + tlv_patch->download_mode); + BT_DBG("Reserved : 0x%x", + tlv_patch->reserved1); + BT_DBG("Product ID : 0x%04x", le16_to_cpu(tlv_patch->product_id)); - BT_DBG("Rom Build Version\t : 0x%04x", + BT_DBG("Rom Build Version : 0x%04x", le16_to_cpu(tlv_patch->rom_build)); - BT_DBG("Patch Version\t\t : 0x%04x", + BT_DBG("Patch Version : 0x%04x", le16_to_cpu(tlv_patch->patch_version)); - BT_DBG("Reserved\t\t : 0x%x", + BT_DBG("Reserved : 0x%x", le16_to_cpu(tlv_patch->reserved2)); - BT_DBG("Patch Entry Address\t : 0x%x", + BT_DBG("Patch Entry Address : 0x%x", le32_to_cpu(tlv_patch->entry)); break; @@ -194,8 +207,8 @@ static void rome_tlv_check_data(struct rome_config *config, } } -static int rome_tlv_send_segment(struct hci_dev *hdev, int idx, int seg_size, - const u8 *data) +static int rome_tlv_send_segment(struct hci_dev *hdev, int seg_size, + const u8 *data, enum rome_tlv_dnld_mode mode) { struct sk_buff *skb; struct edl_event_hdr *edl; @@ -203,12 +216,14 @@ static int rome_tlv_send_segment(struct hci_dev *hdev, int idx, int seg_size, u8 cmd[MAX_SIZE_PER_TLV_SEGMENT + 2]; int err = 0; - BT_DBG("%s: Download segment #%d size %d", hdev->name, idx, seg_size); - cmd[0] = EDL_PATCH_TLV_REQ_CMD; cmd[1] = seg_size; memcpy(cmd + 2, data, seg_size); + if (mode == ROME_SKIP_EVT_VSE_CC || mode == ROME_SKIP_EVT_VSE) + return __hci_cmd_send(hdev, EDL_PATCH_CMD_OPCODE, seg_size + 2, + cmd); + skb = __hci_cmd_sync_ev(hdev, EDL_PATCH_CMD_OPCODE, seg_size + 2, cmd, HCI_VENDOR_PKT, HCI_INIT_TIMEOUT); if (IS_ERR(skb)) { @@ -245,47 +260,12 @@ out: return err; } -static int rome_tlv_download_request(struct hci_dev *hdev, - const struct firmware *fw) -{ - const u8 *buffer, *data; - int total_segment, remain_size; - int ret, i; - - if (!fw || !fw->data) - return -EINVAL; - - total_segment = fw->size / MAX_SIZE_PER_TLV_SEGMENT; - remain_size = fw->size % MAX_SIZE_PER_TLV_SEGMENT; - - BT_DBG("%s: Total segment num %d remain size %d total size %zu", - hdev->name, total_segment, remain_size, fw->size); - - data = fw->data; - for (i = 0; i < total_segment; i++) { - buffer = data + i * MAX_SIZE_PER_TLV_SEGMENT; - ret = rome_tlv_send_segment(hdev, i, MAX_SIZE_PER_TLV_SEGMENT, - buffer); - if (ret < 0) - return -EIO; - } - - if (remain_size) { - buffer = data + total_segment * MAX_SIZE_PER_TLV_SEGMENT; - ret = rome_tlv_send_segment(hdev, total_segment, remain_size, - buffer); - if (ret < 0) - return -EIO; - } - - return 0; -} - static int rome_download_firmware(struct hci_dev *hdev, struct rome_config *config) { const struct firmware *fw; - int ret; + const u8 *segment; + int ret, remain, i = 0; bt_dev_info(hdev, "ROME Downloading %s", config->fwname); @@ -298,10 +278,24 @@ static int rome_download_firmware(struct hci_dev *hdev, rome_tlv_check_data(config, fw); - ret = rome_tlv_download_request(hdev, fw); - if (ret) { - BT_ERR("%s: Failed to download file: %s (%d)", hdev->name, - config->fwname, ret); + segment = fw->data; + remain = fw->size; + while (remain > 0) { + int segsize = min(MAX_SIZE_PER_TLV_SEGMENT, remain); + + bt_dev_dbg(hdev, "Send segment %d, size %d", i++, segsize); + + remain -= segsize; + /* The last segment is always acked regardless download mode */ + if (!remain || segsize < MAX_SIZE_PER_TLV_SEGMENT) + config->dnld_mode = ROME_SKIP_EVT_NONE; + + ret = rome_tlv_send_segment(hdev, segsize, segment, + config->dnld_mode); + if (ret) + break; + + segment += segsize; } release_firmware(fw); diff --git a/drivers/bluetooth/btqca.h b/drivers/bluetooth/btqca.h index 65e994b96c47..13d77fd873b6 100644 --- a/drivers/bluetooth/btqca.h +++ b/drivers/bluetooth/btqca.h @@ -61,6 +61,13 @@ enum qca_bardrate { QCA_BAUDRATE_RESERVED }; +enum rome_tlv_dnld_mode { + ROME_SKIP_EVT_NONE, + ROME_SKIP_EVT_VSE, + ROME_SKIP_EVT_CC, + ROME_SKIP_EVT_VSE_CC +}; + enum rome_tlv_type { TLV_TYPE_PATCH = 1, TLV_TYPE_NVM @@ -70,6 +77,7 @@ struct rome_config { u8 type; char fwname[64]; uint8_t user_baud_rate; + enum rome_tlv_dnld_mode dnld_mode; }; struct edl_event_hdr { @@ -94,7 +102,8 @@ struct tlv_type_patch { __le32 data_length; __u8 format_version; __u8 signature; - __le16 reserved1; + __u8 download_mode; + __u8 reserved1; __le16 product_id; __le16 rom_build; __le16 patch_version; diff --git a/drivers/bluetooth/btqcomsmd.c b/drivers/bluetooth/btqcomsmd.c index 2c9a5fc9137d..7df3eed1ef5e 100644 --- a/drivers/bluetooth/btqcomsmd.c +++ b/drivers/bluetooth/btqcomsmd.c @@ -65,6 +65,7 @@ static int btqcomsmd_cmd_callback(struct rpmsg_device *rpdev, void *data, { struct btqcomsmd *btq = priv; + btq->hdev->stat.byte_rx += count; return btqcomsmd_recv(btq->hdev, HCI_EVENT_PKT, data, count); } @@ -76,12 +77,21 @@ static int btqcomsmd_send(struct hci_dev *hdev, struct sk_buff *skb) switch (hci_skb_pkt_type(skb)) { case HCI_ACLDATA_PKT: ret = rpmsg_send(btq->acl_channel, skb->data, skb->len); + if (ret) { + hdev->stat.err_tx++; + break; + } hdev->stat.acl_tx++; hdev->stat.byte_tx += skb->len; break; case HCI_COMMAND_PKT: ret = rpmsg_send(btq->cmd_channel, skb->data, skb->len); + if (ret) { + hdev->stat.err_tx++; + break; + } hdev->stat.cmd_tx++; + hdev->stat.byte_tx += skb->len; break; default: ret = -EILSEQ; diff --git a/drivers/bluetooth/btusb.c b/drivers/bluetooth/btusb.c index b937cc1e2c07..f73a27ea28cc 100644 --- a/drivers/bluetooth/btusb.c +++ b/drivers/bluetooth/btusb.c @@ -276,6 +276,8 @@ static const struct usb_device_id blacklist_table[] = { { USB_DEVICE(0x04ca, 0x3011), .driver_info = BTUSB_QCA_ROME }, { USB_DEVICE(0x04ca, 0x3015), .driver_info = BTUSB_QCA_ROME }, { USB_DEVICE(0x04ca, 0x3016), .driver_info = BTUSB_QCA_ROME }, + { USB_DEVICE(0x04ca, 0x301a), .driver_info = BTUSB_QCA_ROME }, + { USB_DEVICE(0x13d3, 0x3496), .driver_info = BTUSB_QCA_ROME }, /* Broadcom BCM2035 */ { USB_DEVICE(0x0a5c, 0x2009), .driver_info = BTUSB_BCM92035 }, @@ -371,6 +373,9 @@ static const struct usb_device_id blacklist_table[] = { /* Additional Realtek 8723BU Bluetooth devices */ { USB_DEVICE(0x7392, 0xa611), .driver_info = BTUSB_REALTEK }, + /* Additional Realtek 8723DE Bluetooth devices */ + { USB_DEVICE(0x2ff8, 0xb011), .driver_info = BTUSB_REALTEK }, + /* Additional Realtek 8821AE Bluetooth devices */ { USB_DEVICE(0x0b05, 0x17dc), .driver_info = BTUSB_REALTEK }, { USB_DEVICE(0x13d3, 0x3414), .driver_info = BTUSB_REALTEK }, @@ -379,6 +384,7 @@ static const struct usb_device_id blacklist_table[] = { { USB_DEVICE(0x13d3, 0x3462), .driver_info = BTUSB_REALTEK }, /* Additional Realtek 8822BE Bluetooth devices */ + { USB_DEVICE(0x13d3, 0x3526), .driver_info = BTUSB_REALTEK }, { USB_DEVICE(0x0b05, 0x185c), .driver_info = BTUSB_REALTEK }, /* Silicon Wave based devices */ @@ -406,6 +412,13 @@ static const struct dmi_system_id btusb_needs_reset_resume_table[] = { DMI_MATCH(DMI_PRODUCT_NAME, "XPS 13 9360"), }, }, + { + /* Dell Inspiron 5565 (QCA ROME device 0cf3:e009) */ + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."), + DMI_MATCH(DMI_PRODUCT_NAME, "Inspiron 5565"), + }, + }, {} }; @@ -2497,11 +2510,9 @@ static const struct qca_device_info qca_devices_table[] = { { 0x00000302, 28, 4, 18 }, /* Rome 3.2 */ }; -static int btusb_qca_send_vendor_req(struct hci_dev *hdev, u8 request, +static int btusb_qca_send_vendor_req(struct usb_device *udev, u8 request, void *data, u16 size) { - struct btusb_data *btdata = hci_get_drvdata(hdev); - struct usb_device *udev = btdata->udev; int pipe, err; u8 *buf; @@ -2516,7 +2527,7 @@ static int btusb_qca_send_vendor_req(struct hci_dev *hdev, u8 request, err = usb_control_msg(udev, pipe, request, USB_TYPE_VENDOR | USB_DIR_IN, 0, 0, buf, size, USB_CTRL_SET_TIMEOUT); if (err < 0) { - bt_dev_err(hdev, "Failed to access otp area (%d)", err); + dev_err(&udev->dev, "Failed to access otp area (%d)", err); goto done; } @@ -2666,20 +2677,38 @@ static int btusb_setup_qca_load_nvm(struct hci_dev *hdev, return err; } +/* identify the ROM version and check whether patches are needed */ +static bool btusb_qca_need_patch(struct usb_device *udev) +{ + struct qca_version ver; + + if (btusb_qca_send_vendor_req(udev, QCA_GET_TARGET_VERSION, &ver, + sizeof(ver)) < 0) + return false; + /* only low ROM versions need patches */ + return !(le32_to_cpu(ver.rom_version) & ~0xffffU); +} + static int btusb_setup_qca(struct hci_dev *hdev) { + struct btusb_data *btdata = hci_get_drvdata(hdev); + struct usb_device *udev = btdata->udev; const struct qca_device_info *info = NULL; struct qca_version ver; u32 ver_rom; u8 status; int i, err; - err = btusb_qca_send_vendor_req(hdev, QCA_GET_TARGET_VERSION, &ver, + err = btusb_qca_send_vendor_req(udev, QCA_GET_TARGET_VERSION, &ver, sizeof(ver)); if (err < 0) return err; ver_rom = le32_to_cpu(ver.rom_version); + /* Don't care about high ROM versions */ + if (ver_rom & ~0xffffU) + return 0; + for (i = 0; i < ARRAY_SIZE(qca_devices_table); i++) { if (ver_rom == qca_devices_table[i].rom_version) info = &qca_devices_table[i]; @@ -2689,7 +2718,7 @@ static int btusb_setup_qca(struct hci_dev *hdev) return -ENODEV; } - err = btusb_qca_send_vendor_req(hdev, QCA_CHECK_STATUS, &status, + err = btusb_qca_send_vendor_req(udev, QCA_CHECK_STATUS, &status, sizeof(status)); if (err < 0) return err; @@ -2903,7 +2932,8 @@ static int btusb_probe(struct usb_interface *intf, /* Old firmware would otherwise let ath3k driver load * patch and sysconfig files */ - if (le16_to_cpu(udev->descriptor.bcdDevice) <= 0x0001) + if (le16_to_cpu(udev->descriptor.bcdDevice) <= 0x0001 && + !btusb_qca_need_patch(udev)) return -ENODEV; } @@ -3065,6 +3095,7 @@ static int btusb_probe(struct usb_interface *intf, } if (id->driver_info & BTUSB_ATH3012) { + data->setup_on_usb = btusb_setup_qca; hdev->set_bdaddr = btusb_set_bdaddr_ath3012; set_bit(HCI_QUIRK_SIMULTANEOUS_DISCOVERY, &hdev->quirks); set_bit(HCI_QUIRK_STRICT_DUPLICATE_FILTER, &hdev->quirks); diff --git a/drivers/bluetooth/hci_bcm.c b/drivers/bluetooth/hci_bcm.c index 441f5e1deb11..ddbd8c6a0ceb 100644 --- a/drivers/bluetooth/hci_bcm.c +++ b/drivers/bluetooth/hci_bcm.c @@ -380,10 +380,6 @@ static int bcm_open(struct hci_uart *hu) mutex_lock(&bcm_device_lock); if (hu->serdev) { - err = serdev_device_open(hu->serdev); - if (err) - goto err_free; - bcm->dev = serdev_device_get_drvdata(hu->serdev); goto out; } @@ -420,13 +416,10 @@ out: return 0; err_unset_hu: - if (hu->serdev) - serdev_device_close(hu->serdev); #ifdef CONFIG_PM - else + if (!hu->serdev) bcm->dev->hu = NULL; #endif -err_free: mutex_unlock(&bcm_device_lock); hu->priv = NULL; kfree(bcm); @@ -445,7 +438,6 @@ static int bcm_close(struct hci_uart *hu) mutex_lock(&bcm_device_lock); if (hu->serdev) { - serdev_device_close(hu->serdev); bdev = serdev_device_get_drvdata(hu->serdev); } else if (bcm_device_exists(bcm->dev)) { bdev = bcm->dev; @@ -501,7 +493,7 @@ static int bcm_setup(struct hci_uart *hu) hu->hdev->set_diag = bcm_set_diag; hu->hdev->set_bdaddr = btbcm_set_bdaddr; - err = btbcm_initialize(hu->hdev, fw_name, sizeof(fw_name)); + err = btbcm_initialize(hu->hdev, fw_name, sizeof(fw_name), false); if (err) return err; @@ -794,19 +786,21 @@ static const struct acpi_gpio_mapping acpi_bcm_int_first_gpios[] = { { }, }; -#ifdef CONFIG_ACPI -/* IRQ polarity of some chipsets are not defined correctly in ACPI table. */ -static const struct dmi_system_id bcm_active_low_irq_dmi_table[] = { - { /* Handle ThinkPad 8 tablets with BCM2E55 chipset ACPI ID */ - .ident = "Lenovo ThinkPad 8", +/* Some firmware reports an IRQ which does not work (wrong pin in fw table?) */ +static const struct dmi_system_id bcm_broken_irq_dmi_table[] = { + { + .ident = "Meegopad T08", .matches = { - DMI_EXACT_MATCH(DMI_SYS_VENDOR, "LENOVO"), - DMI_EXACT_MATCH(DMI_PRODUCT_VERSION, "ThinkPad 8"), + DMI_EXACT_MATCH(DMI_BOARD_VENDOR, + "To be filled by OEM."), + DMI_EXACT_MATCH(DMI_BOARD_NAME, "T3 MRD"), + DMI_EXACT_MATCH(DMI_BOARD_VERSION, "V1.1"), }, }, { } }; +#ifdef CONFIG_ACPI static int bcm_resource(struct acpi_resource *ares, void *data) { struct bcm_device *dev = data; @@ -904,6 +898,8 @@ static int bcm_gpio_set_shutdown(struct bcm_device *dev, bool powered) static int bcm_get_resources(struct bcm_device *dev) { + const struct dmi_system_id *dmi_id; + dev->name = dev_name(dev->dev); if (x86_apple_machine && !bcm_apple_get_resources(dev)) @@ -936,6 +932,13 @@ static int bcm_get_resources(struct bcm_device *dev) dev->irq = gpiod_to_irq(gpio); } + dmi_id = dmi_first_match(bcm_broken_irq_dmi_table); + if (dmi_id) { + dev_info(dev->dev, "%s: Has a broken IRQ config, disabling IRQ support / runtime-pm\n", + dmi_id->ident); + dev->irq = 0; + } + dev_dbg(dev->dev, "BCM irq: %d\n", dev->irq); return 0; } @@ -944,7 +947,6 @@ static int bcm_get_resources(struct bcm_device *dev) static int bcm_acpi_probe(struct bcm_device *dev) { LIST_HEAD(resources); - const struct dmi_system_id *dmi_id; const struct acpi_gpio_mapping *gpio_mapping = acpi_bcm_int_last_gpios; struct resource_entry *entry; int ret; @@ -991,13 +993,6 @@ static int bcm_acpi_probe(struct bcm_device *dev) dev->irq_active_low = irq_polarity; dev_warn(dev->dev, "Overwriting IRQ polarity to active %s by module-param\n", dev->irq_active_low ? "low" : "high"); - } else { - dmi_id = dmi_first_match(bcm_active_low_irq_dmi_table); - if (dmi_id) { - dev_warn(dev->dev, "%s: Overwriting IRQ polarity to active low", - dmi_id->ident); - dev->irq_active_low = true; - } } return 0; diff --git a/drivers/bluetooth/hci_ldisc.c b/drivers/bluetooth/hci_ldisc.c index b6a71705b7d6..963bb0309e25 100644 --- a/drivers/bluetooth/hci_ldisc.c +++ b/drivers/bluetooth/hci_ldisc.c @@ -195,7 +195,7 @@ restart: clear_bit(HCI_UART_SENDING, &hu->tx_state); } -static void hci_uart_init_work(struct work_struct *work) +void hci_uart_init_work(struct work_struct *work) { struct hci_uart *hu = container_of(work, struct hci_uart, init_ready); int err; @@ -229,15 +229,6 @@ int hci_uart_init_ready(struct hci_uart *hu) } /* ------- Interface to HCI layer ------ */ -/* Initialize device */ -static int hci_uart_open(struct hci_dev *hdev) -{ - BT_DBG("%s %p", hdev->name, hdev); - - /* Nothing to do for UART driver */ - return 0; -} - /* Reset device */ static int hci_uart_flush(struct hci_dev *hdev) { @@ -264,6 +255,17 @@ static int hci_uart_flush(struct hci_dev *hdev) return 0; } +/* Initialize device */ +static int hci_uart_open(struct hci_dev *hdev) +{ + BT_DBG("%s %p", hdev->name, hdev); + + /* Undo clearing this from hci_uart_close() */ + hdev->flush = hci_uart_flush; + + return 0; +} + /* Close device */ static int hci_uart_close(struct hci_dev *hdev) { @@ -447,6 +449,8 @@ static int hci_uart_setup(struct hci_dev *hdev) btbcm_check_bdaddr(hdev); break; #endif + default: + break; } done: diff --git a/drivers/bluetooth/hci_ll.c b/drivers/bluetooth/hci_ll.c index 27e414b4e3a2..3e767f245ed5 100644 --- a/drivers/bluetooth/hci_ll.c +++ b/drivers/bluetooth/hci_ll.c @@ -141,7 +141,6 @@ static int ll_open(struct hci_uart *hu) if (hu->serdev) { struct ll_device *lldev = serdev_device_get_drvdata(hu->serdev); - serdev_device_open(hu->serdev); if (!IS_ERR(lldev->ext_clk)) clk_prepare_enable(lldev->ext_clk); } @@ -179,8 +178,6 @@ static int ll_close(struct hci_uart *hu) gpiod_set_value_cansleep(lldev->enable_gpio, 0); clk_disable_unprepare(lldev->ext_clk); - - serdev_device_close(hu->serdev); } hu->priv = NULL; diff --git a/drivers/bluetooth/hci_nokia.c b/drivers/bluetooth/hci_nokia.c index 3539fd03f47e..2dc33e65d2d0 100644 --- a/drivers/bluetooth/hci_nokia.c +++ b/drivers/bluetooth/hci_nokia.c @@ -29,7 +29,7 @@ #include #include #include -#include +#include #include #include @@ -477,8 +477,6 @@ static int nokia_open(struct hci_uart *hu) dev_dbg(dev, "protocol open"); - serdev_device_open(hu->serdev); - pm_runtime_enable(dev); return 0; @@ -513,7 +511,6 @@ static int nokia_close(struct hci_uart *hu) gpiod_set_value(btdev->wakeup_bt, 0); pm_runtime_disable(&btdev->serdev->dev); - serdev_device_close(btdev->serdev); return 0; } diff --git a/drivers/bluetooth/hci_qca.c b/drivers/bluetooth/hci_qca.c index 05ec530b8a3a..51790dd02afb 100644 --- a/drivers/bluetooth/hci_qca.c +++ b/drivers/bluetooth/hci_qca.c @@ -29,7 +29,12 @@ */ #include +#include #include +#include +#include +#include +#include #include #include @@ -50,6 +55,9 @@ #define IBS_TX_IDLE_TIMEOUT_MS 2000 #define BAUDRATE_SETTLE_TIMEOUT_MS 300 +/* susclk rate */ +#define SUSCLK_RATE_32KHZ 32768 + /* HCI_IBS transmit side sleep protocol states */ enum tx_ibs_states { HCI_IBS_TX_ASLEEP, @@ -111,6 +119,12 @@ struct qca_data { u64 votes_off; }; +struct qca_serdev { + struct hci_uart serdev_hu; + struct gpio_desc *bt_en; + struct clk *susclk; +}; + static void __serial_clock_on(struct tty_struct *tty) { /* TODO: Some chipset requires to enable UART clock on client @@ -386,6 +400,7 @@ static void hci_ibs_wake_retrans_timeout(struct timer_list *t) /* Initialize protocol */ static int qca_open(struct hci_uart *hu) { + struct qca_serdev *qcadev; struct qca_data *qca; BT_DBG("hu %p qca_open", hu); @@ -444,6 +459,13 @@ static int qca_open(struct hci_uart *hu) timer_setup(&qca->tx_idle_timer, hci_ibs_tx_idle_timeout, 0); qca->tx_idle_delay = IBS_TX_IDLE_TIMEOUT_MS; + if (hu->serdev) { + serdev_device_open(hu->serdev); + + qcadev = serdev_device_get_drvdata(hu->serdev); + gpiod_set_value_cansleep(qcadev->bt_en, 1); + } + BT_DBG("HCI_UART_QCA open, tx_idle_delay=%u, wake_retrans=%u", qca->tx_idle_delay, qca->wake_retrans); @@ -512,6 +534,7 @@ static int qca_flush(struct hci_uart *hu) /* Close protocol */ static int qca_close(struct hci_uart *hu) { + struct qca_serdev *qcadev; struct qca_data *qca = hu->priv; BT_DBG("hu %p qca close", hu); @@ -525,6 +548,13 @@ static int qca_close(struct hci_uart *hu) destroy_workqueue(qca->workqueue); qca->hu = NULL; + if (hu->serdev) { + serdev_device_close(hu->serdev); + + qcadev = serdev_device_get_drvdata(hu->serdev); + gpiod_set_value_cansleep(qcadev->bt_en, 0); + } + kfree_skb(qca->rx_skb); hu->priv = NULL; @@ -880,11 +910,19 @@ static int qca_set_baudrate(struct hci_dev *hdev, uint8_t baudrate) */ set_current_state(TASK_UNINTERRUPTIBLE); schedule_timeout(msecs_to_jiffies(BAUDRATE_SETTLE_TIMEOUT_MS)); - set_current_state(TASK_INTERRUPTIBLE); + set_current_state(TASK_RUNNING); return 0; } +static inline void host_set_baudrate(struct hci_uart *hu, unsigned int speed) +{ + if (hu->serdev) + serdev_device_set_baudrate(hu->serdev, speed); + else + hci_uart_set_baudrate(hu, speed); +} + static int qca_setup(struct hci_uart *hu) { struct hci_dev *hdev = hu->hdev; @@ -905,7 +943,7 @@ static int qca_setup(struct hci_uart *hu) speed = hu->proto->init_speed; if (speed) - hci_uart_set_baudrate(hu, speed); + host_set_baudrate(hu, speed); /* Setup user speed if needed */ speed = 0; @@ -924,7 +962,7 @@ static int qca_setup(struct hci_uart *hu) ret); return ret; } - hci_uart_set_baudrate(hu, speed); + host_set_baudrate(hu, speed); } /* Setup patch / NVM configurations */ @@ -935,6 +973,12 @@ static int qca_setup(struct hci_uart *hu) } else if (ret == -ENOENT) { /* No patch/nvm-config found, run with original fw/config */ ret = 0; + } else if (ret == -EAGAIN) { + /* + * Userspace firmware loader will return -EAGAIN in case no + * patch/nvm-config is found, so run with original fw/config. + */ + ret = 0; } /* Setup bdaddr */ @@ -958,12 +1002,80 @@ static struct hci_uart_proto qca_proto = { .dequeue = qca_dequeue, }; +static int qca_serdev_probe(struct serdev_device *serdev) +{ + struct qca_serdev *qcadev; + int err; + + qcadev = devm_kzalloc(&serdev->dev, sizeof(*qcadev), GFP_KERNEL); + if (!qcadev) + return -ENOMEM; + + qcadev->serdev_hu.serdev = serdev; + serdev_device_set_drvdata(serdev, qcadev); + + qcadev->bt_en = devm_gpiod_get(&serdev->dev, "enable", + GPIOD_OUT_LOW); + if (IS_ERR(qcadev->bt_en)) { + dev_err(&serdev->dev, "failed to acquire enable gpio\n"); + return PTR_ERR(qcadev->bt_en); + } + + qcadev->susclk = devm_clk_get(&serdev->dev, NULL); + if (IS_ERR(qcadev->susclk)) { + dev_err(&serdev->dev, "failed to acquire clk\n"); + return PTR_ERR(qcadev->susclk); + } + + err = clk_set_rate(qcadev->susclk, SUSCLK_RATE_32KHZ); + if (err) + return err; + + err = clk_prepare_enable(qcadev->susclk); + if (err) + return err; + + err = hci_uart_register_device(&qcadev->serdev_hu, &qca_proto); + if (err) + clk_disable_unprepare(qcadev->susclk); + + return err; +} + +static void qca_serdev_remove(struct serdev_device *serdev) +{ + struct qca_serdev *qcadev = serdev_device_get_drvdata(serdev); + + hci_uart_unregister_device(&qcadev->serdev_hu); + + clk_disable_unprepare(qcadev->susclk); +} + +static const struct of_device_id qca_bluetooth_of_match[] = { + { .compatible = "qcom,qca6174-bt" }, + { /* sentinel */ } +}; +MODULE_DEVICE_TABLE(of, qca_bluetooth_of_match); + +static struct serdev_device_driver qca_serdev_driver = { + .probe = qca_serdev_probe, + .remove = qca_serdev_remove, + .driver = { + .name = "hci_uart_qca", + .of_match_table = qca_bluetooth_of_match, + }, +}; + int __init qca_init(void) { + serdev_device_driver_register(&qca_serdev_driver); + return hci_uart_register_proto(&qca_proto); } int __exit qca_deinit(void) { + serdev_device_driver_unregister(&qca_serdev_driver); + return hci_uart_unregister_proto(&qca_proto); } diff --git a/drivers/bluetooth/hci_serdev.c b/drivers/bluetooth/hci_serdev.c index e0e6461b9200..aa2543b3c286 100644 --- a/drivers/bluetooth/hci_serdev.c +++ b/drivers/bluetooth/hci_serdev.c @@ -101,14 +101,6 @@ static void hci_uart_write_work(struct work_struct *work) /* ------- Interface to HCI layer ------ */ -/* Initialize device */ -static int hci_uart_open(struct hci_dev *hdev) -{ - BT_DBG("%s %p", hdev->name, hdev); - - return 0; -} - /* Reset device */ static int hci_uart_flush(struct hci_dev *hdev) { @@ -129,6 +121,17 @@ static int hci_uart_flush(struct hci_dev *hdev) return 0; } +/* Initialize device */ +static int hci_uart_open(struct hci_dev *hdev) +{ + BT_DBG("%s %p", hdev->name, hdev); + + /* Undo clearing this from hci_uart_close() */ + hdev->flush = hci_uart_flush; + + return 0; +} + /* Close device */ static int hci_uart_close(struct hci_dev *hdev) { @@ -204,9 +207,8 @@ static int hci_uart_setup(struct hci_dev *hdev) return 0; } - if (skb->len != sizeof(*ver)) { + if (skb->len != sizeof(*ver)) bt_dev_err(hdev, "Event length mismatch for version info"); - } kfree_skb(skb); return 0; @@ -282,10 +284,14 @@ int hci_uart_register_device(struct hci_uart *hu, serdev_device_set_client_ops(hu->serdev, &hci_serdev_client_ops); - err = p->open(hu); + err = serdev_device_open(hu->serdev); if (err) return err; + err = p->open(hu); + if (err) + goto err_open; + hu->proto = p; set_bit(HCI_UART_PROTO_READY, &hu->flags); @@ -302,6 +308,7 @@ int hci_uart_register_device(struct hci_uart *hu, hdev->bus = HCI_UART; hci_set_drvdata(hdev, hu); + INIT_WORK(&hu->init_ready, hci_uart_init_work); INIT_WORK(&hu->write_work, hci_uart_write_work); percpu_init_rwsem(&hu->proto_lock); @@ -351,6 +358,8 @@ err_register: err_alloc: clear_bit(HCI_UART_PROTO_READY, &hu->flags); p->close(hu); +err_open: + serdev_device_close(hu->serdev); return err; } EXPORT_SYMBOL_GPL(hci_uart_register_device); @@ -365,5 +374,6 @@ void hci_uart_unregister_device(struct hci_uart *hu) cancel_work_sync(&hu->write_work); hu->proto->close(hu); + serdev_device_close(hu->serdev); } EXPORT_SYMBOL_GPL(hci_uart_unregister_device); diff --git a/drivers/bluetooth/hci_uart.h b/drivers/bluetooth/hci_uart.h index 66e8c68e4607..00cab2fd7a1b 100644 --- a/drivers/bluetooth/hci_uart.h +++ b/drivers/bluetooth/hci_uart.h @@ -116,6 +116,7 @@ void hci_uart_unregister_device(struct hci_uart *hu); int hci_uart_tx_wakeup(struct hci_uart *hu); int hci_uart_init_ready(struct hci_uart *hu); +void hci_uart_init_work(struct work_struct *work); void hci_uart_set_baudrate(struct hci_uart *hu, unsigned int speed); void hci_uart_set_flow_control(struct hci_uart *hu, bool enable); void hci_uart_set_speeds(struct hci_uart *hu, unsigned int init_speed, diff --git a/drivers/bus/Kconfig b/drivers/bus/Kconfig index 6dc177bf4c42..d1c0b60e9326 100644 --- a/drivers/bus/Kconfig +++ b/drivers/bus/Kconfig @@ -33,7 +33,6 @@ config HISILICON_LPC bool "Support for ISA I/O space on HiSilicon Hip06/7" depends on ARM64 && (ARCH_HISI || COMPILE_TEST) select INDIRECT_PIO - select MFD_CORE if ACPI help Driver to enable I/O access to devices attached to the Low Pin Count bus on the HiSilicon Hip06/7 SoC. diff --git a/drivers/bus/arm-cci.c b/drivers/bus/arm-cci.c index 443e4c3fd357..b8184a903583 100644 --- a/drivers/bus/arm-cci.c +++ b/drivers/bus/arm-cci.c @@ -371,8 +371,6 @@ asmlinkage void __naked cci_enable_port_for_self(void) [sizeof_struct_cpu_port] "i" (sizeof(struct cpu_port)), [sizeof_struct_ace_port] "i" (sizeof(struct cci_ace_port)), [offsetof_port_phys] "i" (offsetof(struct cci_ace_port, phys)) ); - - unreachable(); } /** diff --git a/drivers/bus/fsl-mc/fsl-mc-allocator.c b/drivers/bus/fsl-mc/fsl-mc-allocator.c index fb1442b08962..e906ecfe23dd 100644 --- a/drivers/bus/fsl-mc/fsl-mc-allocator.c +++ b/drivers/bus/fsl-mc/fsl-mc-allocator.c @@ -354,8 +354,8 @@ int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus, if (error < 0) return error; - irq_resources = devm_kzalloc(&mc_bus_dev->dev, - sizeof(*irq_resources) * irq_count, + irq_resources = devm_kcalloc(&mc_bus_dev->dev, + irq_count, sizeof(*irq_resources), GFP_KERNEL); if (!irq_resources) { error = -ENOMEM; @@ -455,7 +455,7 @@ int __must_check fsl_mc_allocate_irqs(struct fsl_mc_device *mc_dev) return -ENOSPC; } - irqs = devm_kzalloc(&mc_dev->dev, irq_count * sizeof(irqs[0]), + irqs = devm_kcalloc(&mc_dev->dev, irq_count, sizeof(irqs[0]), GFP_KERNEL); if (!irqs) return -ENOMEM; diff --git a/drivers/bus/fsl-mc/fsl-mc-msi.c b/drivers/bus/fsl-mc/fsl-mc-msi.c index ec35e255b496..8b9c66d7c4ff 100644 --- a/drivers/bus/fsl-mc/fsl-mc-msi.c +++ b/drivers/bus/fsl-mc/fsl-mc-msi.c @@ -163,6 +163,8 @@ struct irq_domain *fsl_mc_msi_create_irq_domain(struct fwnode_handle *fwnode, { struct irq_domain *domain; + if (WARN_ON((info->flags & MSI_FLAG_LEVEL_CAPABLE))) + info->flags &= ~MSI_FLAG_LEVEL_CAPABLE; if (info->flags & MSI_FLAG_USE_DEF_DOM_OPS) fsl_mc_msi_update_dom_ops(info); if (info->flags & MSI_FLAG_USE_DEF_CHIP_OPS) diff --git a/drivers/bus/hisi_lpc.c b/drivers/bus/hisi_lpc.c index 2d4611e4c339..d5f85455fa62 100644 --- a/drivers/bus/hisi_lpc.c +++ b/drivers/bus/hisi_lpc.c @@ -11,12 +11,12 @@ #include #include #include -#include #include #include #include #include #include +#include #include #define DRV_NAME "hisi-lpc" @@ -341,15 +341,6 @@ static const struct logic_pio_host_ops hisi_lpc_ops = { }; #ifdef CONFIG_ACPI -#define MFD_CHILD_NAME_PREFIX DRV_NAME"-" -#define MFD_CHILD_NAME_LEN (ACPI_ID_LEN + sizeof(MFD_CHILD_NAME_PREFIX) - 1) - -struct hisi_lpc_mfd_cell { - struct mfd_cell_acpi_match acpi_match; - char name[MFD_CHILD_NAME_LEN]; - char pnpid[ACPI_ID_LEN]; -}; - static int hisi_lpc_acpi_xlat_io_res(struct acpi_device *adev, struct acpi_device *host, struct resource *res) @@ -368,7 +359,7 @@ static int hisi_lpc_acpi_xlat_io_res(struct acpi_device *adev, } /* - * hisi_lpc_acpi_set_io_res - set the resources for a child's MFD + * hisi_lpc_acpi_set_io_res - set the resources for a child * @child: the device node to be updated the I/O resource * @hostdev: the device node associated with host controller * @res: double pointer to be set to the address of translated resources @@ -452,78 +443,122 @@ static int hisi_lpc_acpi_set_io_res(struct device *child, return 0; } +static int hisi_lpc_acpi_remove_subdev(struct device *dev, void *unused) +{ + platform_device_unregister(to_platform_device(dev)); + return 0; +} + +struct hisi_lpc_acpi_cell { + const char *hid; + const char *name; + void *pdata; + size_t pdata_size; +}; + /* * hisi_lpc_acpi_probe - probe children for ACPI FW * @hostdev: LPC host device pointer * * Returns 0 when successful, and a negative value for failure. * - * Scan all child devices and create a per-device MFD with - * logical PIO translated IO resources. + * Create a platform device per child, fixing up the resources + * from bus addresses to Logical PIO addresses. + * */ static int hisi_lpc_acpi_probe(struct device *hostdev) { struct acpi_device *adev = ACPI_COMPANION(hostdev); - struct hisi_lpc_mfd_cell *hisi_lpc_mfd_cells; - struct mfd_cell *mfd_cells; struct acpi_device *child; - int size, ret, count = 0, cell_num = 0; - - list_for_each_entry(child, &adev->children, node) - cell_num++; - - /* allocate the mfd cell and companion ACPI info, one per child */ - size = sizeof(*mfd_cells) + sizeof(*hisi_lpc_mfd_cells); - mfd_cells = devm_kcalloc(hostdev, cell_num, size, GFP_KERNEL); - if (!mfd_cells) - return -ENOMEM; + int ret; - hisi_lpc_mfd_cells = (struct hisi_lpc_mfd_cell *)&mfd_cells[cell_num]; /* Only consider the children of the host */ list_for_each_entry(child, &adev->children, node) { - struct mfd_cell *mfd_cell = &mfd_cells[count]; - struct hisi_lpc_mfd_cell *hisi_lpc_mfd_cell = - &hisi_lpc_mfd_cells[count]; - struct mfd_cell_acpi_match *acpi_match = - &hisi_lpc_mfd_cell->acpi_match; - char *name = hisi_lpc_mfd_cell[count].name; - char *pnpid = hisi_lpc_mfd_cell[count].pnpid; - struct mfd_cell_acpi_match match = { - .pnpid = pnpid, + const char *hid = acpi_device_hid(child); + const struct hisi_lpc_acpi_cell *cell; + struct platform_device *pdev; + const struct resource *res; + bool found = false; + int num_res; + + ret = hisi_lpc_acpi_set_io_res(&child->dev, &adev->dev, &res, + &num_res); + if (ret) { + dev_warn(hostdev, "set resource fail (%d)\n", ret); + goto fail; + } + + cell = (struct hisi_lpc_acpi_cell []){ + /* ipmi */ + { + .hid = "IPI0001", + .name = "hisi-lpc-ipmi", + }, + /* 8250-compatible uart */ + { + .hid = "HISI1031", + .name = "serial8250", + .pdata = (struct plat_serial8250_port []) { + { + .iobase = res->start, + .uartclk = 1843200, + .iotype = UPIO_PORT, + .flags = UPF_BOOT_AUTOCONF, + }, + {} + }, + .pdata_size = 2 * + sizeof(struct plat_serial8250_port), + }, + {} }; - /* - * For any instances of this host controller (Hip06 and Hip07 - * are the only chipsets), we would not have multiple slaves - * with the same HID. And in any system we would have just one - * controller active. So don't worrry about MFD name clashes. - */ - snprintf(name, MFD_CHILD_NAME_LEN, MFD_CHILD_NAME_PREFIX"%s", - acpi_device_hid(child)); - snprintf(pnpid, ACPI_ID_LEN, "%s", acpi_device_hid(child)); - - memcpy(acpi_match, &match, sizeof(*acpi_match)); - mfd_cell->name = name; - mfd_cell->acpi_match = acpi_match; - - ret = hisi_lpc_acpi_set_io_res(&child->dev, &adev->dev, - &mfd_cell->resources, - &mfd_cell->num_resources); - if (ret) { - dev_warn(&child->dev, "set resource fail (%d)\n", ret); - return ret; + for (; cell && cell->name; cell++) { + if (!strcmp(cell->hid, hid)) { + found = true; + break; + } } - count++; - } - ret = mfd_add_devices(hostdev, PLATFORM_DEVID_NONE, - mfd_cells, cell_num, NULL, 0, NULL); - if (ret) { - dev_err(hostdev, "failed to add mfd cells (%d)\n", ret); - return ret; + if (!found) { + dev_warn(hostdev, + "could not find cell for child device (%s)\n", + hid); + ret = -ENODEV; + goto fail; + } + + pdev = platform_device_alloc(cell->name, PLATFORM_DEVID_AUTO); + if (!pdev) { + ret = -ENOMEM; + goto fail; + } + + pdev->dev.parent = hostdev; + ACPI_COMPANION_SET(&pdev->dev, child); + + ret = platform_device_add_resources(pdev, res, num_res); + if (ret) + goto fail; + + ret = platform_device_add_data(pdev, cell->pdata, + cell->pdata_size); + if (ret) + goto fail; + + ret = platform_device_add(pdev); + if (ret) + goto fail; + + acpi_device_set_enumerated(child); } return 0; + +fail: + device_for_each_child(hostdev, NULL, + hisi_lpc_acpi_remove_subdev); + return ret; } static const struct acpi_device_id hisi_lpc_acpi_match[] = { diff --git a/drivers/bus/ti-sysc.c b/drivers/bus/ti-sysc.c index 7cd2fd04b212..1cc29629d238 100644 --- a/drivers/bus/ti-sysc.c +++ b/drivers/bus/ti-sysc.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -32,10 +33,18 @@ static const char * const reg_names[] = { "rev", "sysc", "syss", }; enum sysc_clocks { SYSC_FCK, SYSC_ICK, + SYSC_OPTFCK0, + SYSC_OPTFCK1, + SYSC_OPTFCK2, + SYSC_OPTFCK3, + SYSC_OPTFCK4, + SYSC_OPTFCK5, + SYSC_OPTFCK6, + SYSC_OPTFCK7, SYSC_MAX_CLOCKS, }; -static const char * const clock_names[] = { "fck", "ick", }; +static const char * const clock_names[SYSC_ICK + 1] = { "fck", "ick", }; #define SYSC_IDLEMODE_MASK 3 #define SYSC_CLOCKACTIVITY_MASK 3 @@ -48,6 +57,8 @@ static const char * const clock_names[] = { "fck", "ick", }; * @module_va: virtual address of the interconnect target module * @offsets: register offsets from module base * @clocks: clocks used by the interconnect target module + * @clock_roles: clock role names for the found clocks + * @nr_clocks: number of clocks used by the interconnect target module * @legacy_mode: configured for legacy mode if set * @cap: interconnect target module capabilities * @cfg: interconnect target module configuration @@ -61,7 +72,10 @@ struct sysc { u32 module_size; void __iomem *module_va; int offsets[SYSC_MAX_REGS]; - struct clk *clocks[SYSC_MAX_CLOCKS]; + struct clk **clocks; + const char **clock_roles; + int nr_clocks; + struct reset_control *rsts; const char *legacy_mode; const struct sysc_capabilities *cap; struct sysc_config cfg; @@ -88,6 +102,11 @@ static u32 sysc_read(struct sysc *ddata, int offset) return readl_relaxed(ddata->module_va + offset); } +static bool sysc_opt_clks_needed(struct sysc *ddata) +{ + return !!(ddata->cfg.quirks & SYSC_QUIRK_OPT_CLKS_NEEDED); +} + static u32 sysc_read_revision(struct sysc *ddata) { int offset = ddata->offsets[SYSC_REVISION]; @@ -98,21 +117,28 @@ static u32 sysc_read_revision(struct sysc *ddata) return sysc_read(ddata, offset); } -static int sysc_get_one_clock(struct sysc *ddata, - enum sysc_clocks index) +static int sysc_get_one_clock(struct sysc *ddata, const char *name) { - const char *name; - int error; + int error, i, index = -ENODEV; + + if (!strncmp(clock_names[SYSC_FCK], name, 3)) + index = SYSC_FCK; + else if (!strncmp(clock_names[SYSC_ICK], name, 3)) + index = SYSC_ICK; + + if (index < 0) { + for (i = SYSC_OPTFCK0; i < SYSC_MAX_CLOCKS; i++) { + if (!ddata->clocks[i]) { + index = i; + break; + } + } + } - switch (index) { - case SYSC_FCK: - break; - case SYSC_ICK: - break; - default: - return -EINVAL; + if (index < 0) { + dev_err(ddata->dev, "clock %s not added\n", name); + return index; } - name = clock_names[index]; ddata->clocks[index] = devm_clk_get(ddata->dev, name); if (IS_ERR(ddata->clocks[index])) { @@ -138,10 +164,50 @@ static int sysc_get_one_clock(struct sysc *ddata, static int sysc_get_clocks(struct sysc *ddata) { - int i, error; + struct device_node *np = ddata->dev->of_node; + struct property *prop; + const char *name; + int nr_fck = 0, nr_ick = 0, i, error = 0; - for (i = 0; i < SYSC_MAX_CLOCKS; i++) { - error = sysc_get_one_clock(ddata, i); + ddata->clock_roles = devm_kzalloc(ddata->dev, + sizeof(*ddata->clock_roles) * + SYSC_MAX_CLOCKS, + GFP_KERNEL); + if (!ddata->clock_roles) + return -ENOMEM; + + of_property_for_each_string(np, "clock-names", prop, name) { + if (!strncmp(clock_names[SYSC_FCK], name, 3)) + nr_fck++; + if (!strncmp(clock_names[SYSC_ICK], name, 3)) + nr_ick++; + ddata->clock_roles[ddata->nr_clocks] = name; + ddata->nr_clocks++; + } + + if (ddata->nr_clocks < 1) + return 0; + + if (ddata->nr_clocks > SYSC_MAX_CLOCKS) { + dev_err(ddata->dev, "too many clocks for %pOF\n", np); + + return -EINVAL; + } + + if (nr_fck > 1 || nr_ick > 1) { + dev_err(ddata->dev, "max one fck and ick for %pOF\n", np); + + return -EINVAL; + } + + ddata->clocks = devm_kzalloc(ddata->dev, + sizeof(*ddata->clocks) * ddata->nr_clocks, + GFP_KERNEL); + if (!ddata->clocks) + return -ENOMEM; + + for (i = 0; i < ddata->nr_clocks; i++) { + error = sysc_get_one_clock(ddata, ddata->clock_roles[i]); if (error && error != -ENOENT) return error; } @@ -149,6 +215,42 @@ static int sysc_get_clocks(struct sysc *ddata) return 0; } +/** + * sysc_init_resets - reset module on init + * @ddata: device driver data + * + * A module can have both OCP softreset control and external rstctrl. + * If more complicated rstctrl resets are needed, please handle these + * directly from the child device driver and map only the module reset + * for the parent interconnect target module device. + * + * Automatic reset of the module on init can be skipped with the + * "ti,no-reset-on-init" device tree property. + */ +static int sysc_init_resets(struct sysc *ddata) +{ + int error; + + ddata->rsts = + devm_reset_control_array_get_optional_exclusive(ddata->dev); + if (IS_ERR(ddata->rsts)) + return PTR_ERR(ddata->rsts); + + if (ddata->cfg.quirks & SYSC_QUIRK_NO_RESET_ON_INIT) + goto deassert; + + error = reset_control_assert(ddata->rsts); + if (error) + return error; + +deassert: + error = reset_control_deassert(ddata->rsts); + if (error) + return error; + + return 0; +} + /** * sysc_parse_and_check_child_range - parses module IO region from ranges * @ddata: device driver data @@ -533,9 +635,13 @@ static int __maybe_unused sysc_runtime_suspend(struct device *dev) goto idled; } - for (i = 0; i < SYSC_MAX_CLOCKS; i++) { + for (i = 0; i < ddata->nr_clocks; i++) { if (IS_ERR_OR_NULL(ddata->clocks[i])) continue; + + if (i >= SYSC_OPTFCK0 && !sysc_opt_clks_needed(ddata)) + break; + clk_disable(ddata->clocks[i]); } @@ -572,9 +678,13 @@ static int __maybe_unused sysc_runtime_resume(struct device *dev) goto awake; } - for (i = 0; i < SYSC_MAX_CLOCKS; i++) { + for (i = 0; i < ddata->nr_clocks; i++) { if (IS_ERR_OR_NULL(ddata->clocks[i])) continue; + + if (i >= SYSC_OPTFCK0 && !sysc_opt_clks_needed(ddata)) + break; + error = clk_enable(ddata->clocks[i]); if (error) return error; @@ -590,23 +700,103 @@ awake: static int sysc_suspend(struct device *dev) { struct sysc *ddata; + int error; ddata = dev_get_drvdata(dev); + if (ddata->cfg.quirks & (SYSC_QUIRK_RESOURCE_PROVIDER | + SYSC_QUIRK_LEGACY_IDLE)) + return 0; + if (!ddata->enabled) return 0; + dev_dbg(ddata->dev, "%s %s\n", __func__, + ddata->name ? ddata->name : ""); + + error = pm_runtime_put_sync_suspend(dev); + if (error < 0) { + dev_warn(ddata->dev, "%s not idle %i %s\n", + __func__, error, + ddata->name ? ddata->name : ""); + + return 0; + } + ddata->needs_resume = true; - return sysc_runtime_suspend(dev); + return 0; } static int sysc_resume(struct device *dev) +{ + struct sysc *ddata; + int error; + + ddata = dev_get_drvdata(dev); + + if (ddata->cfg.quirks & (SYSC_QUIRK_RESOURCE_PROVIDER | + SYSC_QUIRK_LEGACY_IDLE)) + return 0; + + if (ddata->needs_resume) { + dev_dbg(ddata->dev, "%s %s\n", __func__, + ddata->name ? ddata->name : ""); + + error = pm_runtime_get_sync(dev); + if (error < 0) { + dev_err(ddata->dev, "%s error %i %s\n", + __func__, error, + ddata->name ? ddata->name : ""); + + return error; + } + + ddata->needs_resume = false; + } + + return 0; +} + +static int sysc_noirq_suspend(struct device *dev) { struct sysc *ddata; ddata = dev_get_drvdata(dev); + + if (ddata->cfg.quirks & SYSC_QUIRK_LEGACY_IDLE) + return 0; + + if (!(ddata->cfg.quirks & SYSC_QUIRK_RESOURCE_PROVIDER)) + return 0; + + if (!ddata->enabled) + return 0; + + dev_dbg(ddata->dev, "%s %s\n", __func__, + ddata->name ? ddata->name : ""); + + ddata->needs_resume = true; + + return sysc_runtime_suspend(dev); +} + +static int sysc_noirq_resume(struct device *dev) +{ + struct sysc *ddata; + + ddata = dev_get_drvdata(dev); + + if (ddata->cfg.quirks & SYSC_QUIRK_LEGACY_IDLE) + return 0; + + if (!(ddata->cfg.quirks & SYSC_QUIRK_RESOURCE_PROVIDER)) + return 0; + if (ddata->needs_resume) { + dev_dbg(ddata->dev, "%s %s\n", __func__, + ddata->name ? ddata->name : ""); + ddata->needs_resume = false; return sysc_runtime_resume(dev); @@ -618,6 +808,7 @@ static int sysc_resume(struct device *dev) static const struct dev_pm_ops sysc_pm_ops = { SET_SYSTEM_SLEEP_PM_OPS(sysc_suspend, sysc_resume) + SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(sysc_noirq_suspend, sysc_noirq_resume) SET_RUNTIME_PM_OPS(sysc_runtime_suspend, sysc_runtime_resume, NULL) @@ -649,9 +840,29 @@ struct sysc_revision_quirk { } static const struct sysc_revision_quirk sysc_revision_quirks[] = { + /* These need to use noirq_suspend */ + SYSC_QUIRK("control", 0, 0, 0x10, -1, 0x40000900, 0xffffffff, + SYSC_QUIRK_RESOURCE_PROVIDER), + SYSC_QUIRK("i2c", 0, 0, 0x10, 0x90, 0x5040000a, 0xffffffff, + SYSC_QUIRK_RESOURCE_PROVIDER), + SYSC_QUIRK("mcspi", 0, 0, 0x10, -1, 0x40300a0b, 0xffffffff, + SYSC_QUIRK_RESOURCE_PROVIDER), + SYSC_QUIRK("prcm", 0, 0, -1, -1, 0x40000100, 0xffffffff, + SYSC_QUIRK_RESOURCE_PROVIDER), + SYSC_QUIRK("ocp2scp", 0, 0, 0x10, 0x14, 0x50060005, 0xffffffff, + SYSC_QUIRK_RESOURCE_PROVIDER), + SYSC_QUIRK("padconf", 0, 0, 0x10, -1, 0x4fff0800, 0xffffffff, + SYSC_QUIRK_RESOURCE_PROVIDER), + SYSC_QUIRK("scm", 0, 0, 0x10, -1, 0x40000900, 0xffffffff, + SYSC_QUIRK_RESOURCE_PROVIDER), + SYSC_QUIRK("scrm", 0, 0, -1, -1, 0x00000010, 0xffffffff, + SYSC_QUIRK_RESOURCE_PROVIDER), + SYSC_QUIRK("sdma", 0, 0, 0x2c, 0x28, 0x00010900, 0xffffffff, + SYSC_QUIRK_RESOURCE_PROVIDER), + /* These drivers need to be fixed to not use pm_runtime_irq_safe() */ SYSC_QUIRK("gpio", 0, 0, 0x10, 0x114, 0x50600801, 0xffffffff, - SYSC_QUIRK_LEGACY_IDLE), + SYSC_QUIRK_LEGACY_IDLE | SYSC_QUIRK_OPT_CLKS_IN_RESET), SYSC_QUIRK("mmu", 0, 0, 0x10, 0x14, 0x00000020, 0xffffffff, SYSC_QUIRK_LEGACY_IDLE), SYSC_QUIRK("mmu", 0, 0, 0x10, 0x14, 0x00000030, 0xffffffff, @@ -664,8 +875,40 @@ static const struct sysc_revision_quirk sysc_revision_quirks[] = { SYSC_QUIRK_LEGACY_IDLE), SYSC_QUIRK("timer", 0, 0, 0x10, 0x14, 0x00000015, 0xffffffff, SYSC_QUIRK_LEGACY_IDLE), + /* Some timers on omap4 and later */ + SYSC_QUIRK("timer", 0, 0, 0x10, -1, 0x4fff1301, 0xffffffff, + SYSC_QUIRK_LEGACY_IDLE), SYSC_QUIRK("uart", 0, 0x50, 0x54, 0x58, 0x00000052, 0xffffffff, SYSC_QUIRK_LEGACY_IDLE), + /* Uarts on omap4 and later */ + SYSC_QUIRK("uart", 0, 0x50, 0x54, 0x58, 0x50411e03, 0xffffffff, + SYSC_QUIRK_LEGACY_IDLE), + + /* These devices don't yet suspend properly without legacy setting */ + SYSC_QUIRK("sdio", 0, 0, 0x10, -1, 0x40202301, 0xffffffff, + SYSC_QUIRK_LEGACY_IDLE), + SYSC_QUIRK("wdt", 0, 0, 0x10, 0x14, 0x502a0500, 0xffffffff, + SYSC_QUIRK_LEGACY_IDLE), + SYSC_QUIRK("wdt", 0, 0, 0x10, 0x14, 0x502a0d00, 0xffffffff, + SYSC_QUIRK_LEGACY_IDLE), + +#ifdef DEBUG + SYSC_QUIRK("aess", 0, 0, 0x10, -1, 0x40000000, 0xffffffff, 0), + SYSC_QUIRK("gpu", 0, 0x1fc00, 0x1fc10, -1, 0, 0, 0), + SYSC_QUIRK("hdq1w", 0, 0, 0x14, 0x18, 0x00000006, 0xffffffff, 0), + SYSC_QUIRK("hsi", 0, 0, 0x10, 0x14, 0x50043101, 0xffffffff, 0), + SYSC_QUIRK("iss", 0, 0, 0x10, -1, 0x40000101, 0xffffffff, 0), + SYSC_QUIRK("mcasp", 0, 0, 0x4, -1, 0x44306302, 0xffffffff, 0), + SYSC_QUIRK("mcbsp", 0, -1, 0x8c, -1, 0, 0, 0), + SYSC_QUIRK("mailbox", 0, 0, 0x10, -1, 0x00000400, 0xffffffff, 0), + SYSC_QUIRK("slimbus", 0, 0, 0x10, -1, 0x40000902, 0xffffffff, 0), + SYSC_QUIRK("slimbus", 0, 0, 0x10, -1, 0x40002903, 0xffffffff, 0), + SYSC_QUIRK("spinlock", 0, 0, 0x10, -1, 0x50020000, 0xffffffff, 0), + SYSC_QUIRK("usbhstll", 0, 0, 0x10, 0x14, 0x00000004, 0xffffffff, 0), + SYSC_QUIRK("usb_host_hs", 0, 0, 0x10, 0x14, 0x50700100, 0xffffffff, 0), + SYSC_QUIRK("usb_otg_hs", 0, 0x400, 0x404, 0x408, 0x00000050, + 0xffffffff, 0), +#endif }; static void sysc_init_revision_quirks(struct sysc *ddata) @@ -716,6 +959,7 @@ static int sysc_init_module(struct sysc *ddata) return 0; } + ddata->revision = sysc_read_revision(ddata); pm_runtime_put_sync(ddata->dev); @@ -811,29 +1055,58 @@ static int sysc_init_syss_mask(struct sysc *ddata) } /* - * Many child device drivers need to have fck available to get the clock - * rate for device internal configuration. + * Many child device drivers need to have fck and opt clocks available + * to get the clock rate for device internal configuration etc. */ -static int sysc_child_add_fck(struct sysc *ddata, - struct device *child) +static int sysc_child_add_named_clock(struct sysc *ddata, + struct device *child, + const char *name) { - struct clk *fck; + struct clk *clk; struct clk_lookup *l; - const char *name = clock_names[SYSC_FCK]; + int error = 0; - if (IS_ERR_OR_NULL(ddata->clocks[SYSC_FCK])) + if (!name) return 0; - fck = clk_get(child, name); - if (!IS_ERR(fck)) { - clk_put(fck); + clk = clk_get(child, name); + if (!IS_ERR(clk)) { + clk_put(clk); return -EEXIST; } - l = clkdev_create(ddata->clocks[SYSC_FCK], name, dev_name(child)); + clk = clk_get(ddata->dev, name); + if (IS_ERR(clk)) + return -ENODEV; + + l = clkdev_create(clk, name, dev_name(child)); + if (!l) + error = -ENOMEM; + + clk_put(clk); + + return error; +} + +static int sysc_child_add_clocks(struct sysc *ddata, + struct device *child) +{ + int i, error; + + for (i = 0; i < ddata->nr_clocks; i++) { + error = sysc_child_add_named_clock(ddata, + child, + ddata->clock_roles[i]); + if (error && error != -EEXIST) { + dev_err(ddata->dev, "could not add child clock %s: %i\n", + ddata->clock_roles[i], error); + + return error; + } + } - return l ? 0 : -ENODEV; + return 0; } static struct device_type sysc_device_type = { @@ -891,18 +1164,33 @@ static int sysc_child_suspend_noirq(struct device *dev) ddata = sysc_child_to_parent(dev); + dev_dbg(ddata->dev, "%s %s\n", __func__, + ddata->name ? ddata->name : ""); + error = pm_generic_suspend_noirq(dev); - if (error) + if (error) { + dev_err(dev, "%s error at %i: %i\n", + __func__, __LINE__, error); + return error; + } if (!pm_runtime_status_suspended(dev)) { error = pm_generic_runtime_suspend(dev); - if (error) + if (error) { + dev_err(dev, "%s error at %i: %i\n", + __func__, __LINE__, error); + return error; + } error = sysc_runtime_suspend(ddata->dev); - if (error) + if (error) { + dev_err(dev, "%s error at %i: %i\n", + __func__, __LINE__, error); + return error; + } ddata->child_needs_resume = true; } @@ -917,6 +1205,9 @@ static int sysc_child_resume_noirq(struct device *dev) ddata = sysc_child_to_parent(dev); + dev_dbg(ddata->dev, "%s %s\n", __func__, + ddata->name ? ddata->name : ""); + if (ddata->child_needs_resume) { ddata->child_needs_resume = false; @@ -983,10 +1274,9 @@ static int sysc_notifier_call(struct notifier_block *nb, switch (event) { case BUS_NOTIFY_ADD_DEVICE: - error = sysc_child_add_fck(ddata, dev); - if (error && error != -EEXIST) - dev_warn(ddata->dev, "could not add %s fck: %i\n", - dev_name(dev), error); + error = sysc_child_add_clocks(ddata, dev); + if (error) + return error; sysc_legacy_idle_quirk(ddata, dev); break; default: @@ -1314,6 +1604,11 @@ static void ti_sysc_idle(struct work_struct *work) pm_runtime_put_sync(ddata->dev); } +static const struct of_device_id sysc_match_table[] = { + { .compatible = "simple-bus", }, + { /* sentinel */ }, +}; + static int sysc_probe(struct platform_device *pdev) { struct ti_sysc_platform_data *pdata = dev_get_platdata(&pdev->dev); @@ -1359,8 +1654,11 @@ static int sysc_probe(struct platform_device *pdev) if (error) goto unprepare; - pm_runtime_enable(ddata->dev); + error = sysc_init_resets(ddata); + if (error) + return error; + pm_runtime_enable(ddata->dev); error = sysc_init_module(ddata); if (error) goto unprepare; @@ -1375,8 +1673,8 @@ static int sysc_probe(struct platform_device *pdev) sysc_show_registers(ddata); ddata->dev->type = &sysc_device_type; - error = of_platform_populate(ddata->dev->of_node, - NULL, pdata ? pdata->auxdata : NULL, + error = of_platform_populate(ddata->dev->of_node, sysc_match_table, + pdata ? pdata->auxdata : NULL, ddata->dev); if (error) goto err; @@ -1391,6 +1689,9 @@ static int sysc_probe(struct platform_device *pdev) pm_runtime_put(&pdev->dev); } + if (!of_get_available_child_count(ddata->dev->of_node)) + reset_control_assert(ddata->rsts); + return 0; err: @@ -1420,6 +1721,7 @@ static int sysc_remove(struct platform_device *pdev) pm_runtime_put_sync(&pdev->dev); pm_runtime_disable(&pdev->dev); + reset_control_assert(ddata->rsts); unprepare: sysc_unprepare(ddata); diff --git a/drivers/cdrom/cdrom.c b/drivers/cdrom/cdrom.c index bfc566d3f31a..a78b8e7085e9 100644 --- a/drivers/cdrom/cdrom.c +++ b/drivers/cdrom/cdrom.c @@ -2132,7 +2132,7 @@ static int cdrom_read_cdda_old(struct cdrom_device_info *cdi, __u8 __user *ubuf, */ nr = nframes; do { - cgc.buffer = kmalloc(CD_FRAMESIZE_RAW * nr, GFP_KERNEL); + cgc.buffer = kmalloc_array(nr, CD_FRAMESIZE_RAW, GFP_KERNEL); if (cgc.buffer) break; @@ -2192,7 +2192,7 @@ static int cdrom_read_cdda_bpc(struct cdrom_device_info *cdi, __u8 __user *ubuf, len = nr * CD_FRAMESIZE_RAW; - rq = blk_get_request(q, REQ_OP_SCSI_IN, GFP_KERNEL); + rq = blk_get_request(q, REQ_OP_SCSI_IN, 0); if (IS_ERR(rq)) { ret = PTR_ERR(rq); break; diff --git a/drivers/char/Kconfig b/drivers/char/Kconfig index e538061eadcb..212f447938ae 100644 --- a/drivers/char/Kconfig +++ b/drivers/char/Kconfig @@ -81,7 +81,7 @@ config PRINTER corresponding drivers into the kernel. To compile this driver as a module, choose M here and read - . The module will be called lp. + . The module will be called lp. If you have several parallel ports, you can specify which ports to use with the "lp" kernel command line option. (Try "man bootparam" @@ -540,5 +540,17 @@ source "drivers/s390/char/Kconfig" source "drivers/char/xillybus/Kconfig" +config ADI + tristate "SPARC Privileged ADI driver" + depends on SPARC64 + default m + help + SPARC M7 and newer processors utilize ADI (Application Data + Integrity) to version and protect memory. This driver provides + read/write access to the ADI versions for privileged processes. + This feature is also known as MCD (Memory Corruption Detection) + and SSM (Silicon Secured Memory). Intended consumers of this + driver include crash and makedumpfile. + endmenu diff --git a/drivers/char/Makefile b/drivers/char/Makefile index c97c768cd1dd..b8d42b4e979b 100644 --- a/drivers/char/Makefile +++ b/drivers/char/Makefile @@ -57,3 +57,4 @@ js-rtc-y = rtc.o obj-$(CONFIG_XILLYBUS) += xillybus/ obj-$(CONFIG_POWERNV_OP_PANEL) += powernv-op-panel.o +obj-$(CONFIG_ADI) += adi.o diff --git a/drivers/char/adi.c b/drivers/char/adi.c new file mode 100644 index 000000000000..751d7cc0da1b --- /dev/null +++ b/drivers/char/adi.c @@ -0,0 +1,239 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Privileged ADI driver for sparc64 + * + * Author: Tom Hromatka + */ +#include +#include +#include +#include +#include +#include +#include + +#define MAX_BUF_SZ PAGE_SIZE + +static int adi_open(struct inode *inode, struct file *file) +{ + file->f_mode |= FMODE_UNSIGNED_OFFSET; + return 0; +} + +static int read_mcd_tag(unsigned long addr) +{ + long err; + int ver; + + __asm__ __volatile__( + "1: ldxa [%[addr]] %[asi], %[ver]\n" + " mov 0, %[err]\n" + "2:\n" + " .section .fixup,#alloc,#execinstr\n" + " .align 4\n" + "3: sethi %%hi(2b), %%g1\n" + " jmpl %%g1 + %%lo(2b), %%g0\n" + " mov %[invalid], %[err]\n" + " .previous\n" + " .section __ex_table, \"a\"\n" + " .align 4\n" + " .word 1b, 3b\n" + " .previous\n" + : [ver] "=r" (ver), [err] "=r" (err) + : [addr] "r" (addr), [invalid] "i" (EFAULT), + [asi] "i" (ASI_MCD_REAL) + : "memory", "g1" + ); + + if (err) + return -EFAULT; + else + return ver; +} + +static ssize_t adi_read(struct file *file, char __user *buf, + size_t count, loff_t *offp) +{ + size_t ver_buf_sz, bytes_read = 0; + int ver_buf_idx = 0; + loff_t offset; + u8 *ver_buf; + ssize_t ret; + + ver_buf_sz = min_t(size_t, count, MAX_BUF_SZ); + ver_buf = kmalloc(ver_buf_sz, GFP_KERNEL); + if (!ver_buf) + return -ENOMEM; + + offset = (*offp) * adi_blksize(); + + while (bytes_read < count) { + ret = read_mcd_tag(offset); + if (ret < 0) + goto out; + + ver_buf[ver_buf_idx] = (u8)ret; + ver_buf_idx++; + offset += adi_blksize(); + + if (ver_buf_idx >= ver_buf_sz) { + if (copy_to_user(buf + bytes_read, ver_buf, + ver_buf_sz)) { + ret = -EFAULT; + goto out; + } + + bytes_read += ver_buf_sz; + ver_buf_idx = 0; + + ver_buf_sz = min(count - bytes_read, + (size_t)MAX_BUF_SZ); + } + } + + (*offp) += bytes_read; + ret = bytes_read; +out: + kfree(ver_buf); + return ret; +} + +static int set_mcd_tag(unsigned long addr, u8 ver) +{ + long err; + + __asm__ __volatile__( + "1: stxa %[ver], [%[addr]] %[asi]\n" + " mov 0, %[err]\n" + "2:\n" + " .section .fixup,#alloc,#execinstr\n" + " .align 4\n" + "3: sethi %%hi(2b), %%g1\n" + " jmpl %%g1 + %%lo(2b), %%g0\n" + " mov %[invalid], %[err]\n" + " .previous\n" + " .section __ex_table, \"a\"\n" + " .align 4\n" + " .word 1b, 3b\n" + " .previous\n" + : [err] "=r" (err) + : [ver] "r" (ver), [addr] "r" (addr), + [invalid] "i" (EFAULT), [asi] "i" (ASI_MCD_REAL) + : "memory", "g1" + ); + + if (err) + return -EFAULT; + else + return ver; +} + +static ssize_t adi_write(struct file *file, const char __user *buf, + size_t count, loff_t *offp) +{ + size_t ver_buf_sz, bytes_written = 0; + loff_t offset; + u8 *ver_buf; + ssize_t ret; + int i; + + if (count <= 0) + return -EINVAL; + + ver_buf_sz = min_t(size_t, count, MAX_BUF_SZ); + ver_buf = kmalloc(ver_buf_sz, GFP_KERNEL); + if (!ver_buf) + return -ENOMEM; + + offset = (*offp) * adi_blksize(); + + do { + if (copy_from_user(ver_buf, &buf[bytes_written], + ver_buf_sz)) { + ret = -EFAULT; + goto out; + } + + for (i = 0; i < ver_buf_sz; i++) { + ret = set_mcd_tag(offset, ver_buf[i]); + if (ret < 0) + goto out; + + offset += adi_blksize(); + } + + bytes_written += ver_buf_sz; + ver_buf_sz = min(count - bytes_written, (size_t)MAX_BUF_SZ); + } while (bytes_written < count); + + (*offp) += bytes_written; + ret = bytes_written; +out: + __asm__ __volatile__("membar #Sync"); + kfree(ver_buf); + return ret; +} + +static loff_t adi_llseek(struct file *file, loff_t offset, int whence) +{ + loff_t ret = -EINVAL; + + switch (whence) { + case SEEK_END: + case SEEK_DATA: + case SEEK_HOLE: + /* unsupported */ + return -EINVAL; + case SEEK_CUR: + if (offset == 0) + return file->f_pos; + + offset += file->f_pos; + break; + case SEEK_SET: + break; + } + + if (offset != file->f_pos) { + file->f_pos = offset; + file->f_version = 0; + ret = offset; + } + + return ret; +} + +static const struct file_operations adi_fops = { + .owner = THIS_MODULE, + .llseek = adi_llseek, + .open = adi_open, + .read = adi_read, + .write = adi_write, +}; + +static struct miscdevice adi_miscdev = { + .minor = MISC_DYNAMIC_MINOR, + .name = KBUILD_MODNAME, + .fops = &adi_fops, +}; + +static int __init adi_init(void) +{ + if (!adi_capable()) + return -EPERM; + + return misc_register(&adi_miscdev); +} + +static void __exit adi_exit(void) +{ + misc_deregister(&adi_miscdev); +} + +module_init(adi_init); +module_exit(adi_exit); + +MODULE_AUTHOR("Tom Hromatka "); +MODULE_DESCRIPTION("Privileged interface to ADI"); +MODULE_VERSION("1.0"); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/char/agp/amd-k7-agp.c b/drivers/char/agp/amd-k7-agp.c index b450544dcaf0..6914e4f0ce98 100644 --- a/drivers/char/agp/amd-k7-agp.c +++ b/drivers/char/agp/amd-k7-agp.c @@ -85,7 +85,8 @@ static int amd_create_gatt_pages(int nr_tables) int retval = 0; int i; - tables = kzalloc((nr_tables + 1) * sizeof(struct amd_page_map *),GFP_KERNEL); + tables = kcalloc(nr_tables + 1, sizeof(struct amd_page_map *), + GFP_KERNEL); if (tables == NULL) return -ENOMEM; diff --git a/drivers/char/agp/ati-agp.c b/drivers/char/agp/ati-agp.c index 88b4cbee4dac..20bf5f78a362 100644 --- a/drivers/char/agp/ati-agp.c +++ b/drivers/char/agp/ati-agp.c @@ -108,7 +108,8 @@ static int ati_create_gatt_pages(int nr_tables) int retval = 0; int i; - tables = kzalloc((nr_tables + 1) * sizeof(struct ati_page_map *),GFP_KERNEL); + tables = kcalloc(nr_tables + 1, sizeof(struct ati_page_map *), + GFP_KERNEL); if (tables == NULL) return -ENOMEM; diff --git a/drivers/char/agp/compat_ioctl.c b/drivers/char/agp/compat_ioctl.c index 2053f70ef66b..52ffe1706ce0 100644 --- a/drivers/char/agp/compat_ioctl.c +++ b/drivers/char/agp/compat_ioctl.c @@ -98,11 +98,15 @@ static int compat_agpioc_reserve_wrap(struct agp_file_private *priv, void __user if (ureserve.seg_count >= 16384) return -EINVAL; - usegment = kmalloc(sizeof(*usegment) * ureserve.seg_count, GFP_KERNEL); + usegment = kmalloc_array(ureserve.seg_count, + sizeof(*usegment), + GFP_KERNEL); if (!usegment) return -ENOMEM; - ksegment = kmalloc(sizeof(*ksegment) * kreserve.seg_count, GFP_KERNEL); + ksegment = kmalloc_array(kreserve.seg_count, + sizeof(*ksegment), + GFP_KERNEL); if (!ksegment) { kfree(usegment); return -ENOMEM; diff --git a/drivers/char/agp/isoch.c b/drivers/char/agp/isoch.c index fc8e1bc3347d..31c374b1b91b 100644 --- a/drivers/char/agp/isoch.c +++ b/drivers/char/agp/isoch.c @@ -93,7 +93,8 @@ static int agp_3_5_isochronous_node_enable(struct agp_bridge_data *bridge, * We'll work with an array of isoch_data's (one for each * device in dev_list) throughout this function. */ - if ((master = kmalloc(ndevs * sizeof(*master), GFP_KERNEL)) == NULL) { + master = kmalloc_array(ndevs, sizeof(*master), GFP_KERNEL); + if (master == NULL) { ret = -ENOMEM; goto get_out; } diff --git a/drivers/char/agp/sgi-agp.c b/drivers/char/agp/sgi-agp.c index 3051c73bc383..e7d5bdc02d93 100644 --- a/drivers/char/agp/sgi-agp.c +++ b/drivers/char/agp/sgi-agp.c @@ -280,9 +280,9 @@ static int agp_sgi_init(void) else return 0; - sgi_tioca_agp_bridges = kmalloc(tioca_gart_found * - sizeof(struct agp_bridge_data *), - GFP_KERNEL); + sgi_tioca_agp_bridges = kmalloc_array(tioca_gart_found, + sizeof(struct agp_bridge_data *), + GFP_KERNEL); if (!sgi_tioca_agp_bridges) return -ENOMEM; diff --git a/drivers/char/agp/sworks-agp.c b/drivers/char/agp/sworks-agp.c index 4dbdd3bc9bb8..7729414100ff 100644 --- a/drivers/char/agp/sworks-agp.c +++ b/drivers/char/agp/sworks-agp.c @@ -96,7 +96,7 @@ static int serverworks_create_gatt_pages(int nr_tables) int retval = 0; int i; - tables = kzalloc((nr_tables + 1) * sizeof(struct serverworks_page_map *), + tables = kcalloc(nr_tables + 1, sizeof(struct serverworks_page_map *), GFP_KERNEL); if (tables == NULL) return -ENOMEM; diff --git a/drivers/char/agp/uninorth-agp.c b/drivers/char/agp/uninorth-agp.c index 79d8c84693a1..31fcd0430426 100644 --- a/drivers/char/agp/uninorth-agp.c +++ b/drivers/char/agp/uninorth-agp.c @@ -402,7 +402,9 @@ static int uninorth_create_gatt_table(struct agp_bridge_data *bridge) if (table == NULL) return -ENOMEM; - uninorth_priv.pages_arr = kmalloc((1 << page_order) * sizeof(struct page*), GFP_KERNEL); + uninorth_priv.pages_arr = kmalloc_array(1 << page_order, + sizeof(struct page *), + GFP_KERNEL); if (uninorth_priv.pages_arr == NULL) goto enomem; diff --git a/drivers/char/apm-emulation.c b/drivers/char/apm-emulation.c index a5e2f9e557ea..53436c03dbce 100644 --- a/drivers/char/apm-emulation.c +++ b/drivers/char/apm-emulation.c @@ -461,19 +461,6 @@ static int proc_apm_show(struct seq_file *m, void *v) return 0; } - -static int proc_apm_open(struct inode *inode, struct file *file) -{ - return single_open(file, proc_apm_show, NULL); -} - -static const struct file_operations apm_proc_fops = { - .owner = THIS_MODULE, - .open = proc_apm_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; #endif static int kapmd(void *arg) @@ -657,7 +644,7 @@ static int __init apm_init(void) wake_up_process(kapmd_tsk); #ifdef CONFIG_PROC_FS - proc_create("apm", 0, NULL, &apm_proc_fops); + proc_create_single("apm", 0, NULL, proc_apm_show); #endif ret = misc_register(&apm_device); diff --git a/drivers/char/ds1620.c b/drivers/char/ds1620.c index eb53cbadb68f..a5ecf6dae02e 100644 --- a/drivers/char/ds1620.c +++ b/drivers/char/ds1620.c @@ -345,18 +345,6 @@ static int ds1620_proc_therm_show(struct seq_file *m, void *v) fan_state[netwinder_get_fan()]); return 0; } - -static int ds1620_proc_therm_open(struct inode *inode, struct file *file) -{ - return single_open(file, ds1620_proc_therm_show, NULL); -} - -static const struct file_operations ds1620_proc_therm_fops = { - .open = ds1620_proc_therm_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; #endif static const struct file_operations ds1620_fops = { @@ -404,7 +392,7 @@ static int __init ds1620_init(void) return ret; #ifdef THERM_USE_PROC - if (!proc_create("therm", 0, NULL, &ds1620_proc_therm_fops)) + if (!proc_create_single("therm", 0, NULL, ds1620_proc_therm_show)) printk(KERN_ERR "therm: unable to register /proc/therm\n"); #endif diff --git a/drivers/char/efirtc.c b/drivers/char/efirtc.c index dc62568b7dde..d9aab643997e 100644 --- a/drivers/char/efirtc.c +++ b/drivers/char/efirtc.c @@ -358,19 +358,6 @@ static int efi_rtc_proc_show(struct seq_file *m, void *v) return 0; } - -static int efi_rtc_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, efi_rtc_proc_show, NULL); -} - -static const struct file_operations efi_rtc_proc_fops = { - .open = efi_rtc_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int __init efi_rtc_init(void) { @@ -386,7 +373,7 @@ efi_rtc_init(void) return ret; } - dir = proc_create("driver/efirtc", 0, NULL, &efi_rtc_proc_fops); + dir = proc_create_single("driver/efirtc", 0, NULL, efi_rtc_proc_show); if (dir == NULL) { printk(KERN_ERR "efirtc: can't create /proc/driver/efirtc.\n"); misc_deregister(&efi_rtc_dev); diff --git a/drivers/char/hw_random/Kconfig b/drivers/char/hw_random/Kconfig index d53541e96bee..c34b257d852d 100644 --- a/drivers/char/hw_random/Kconfig +++ b/drivers/char/hw_random/Kconfig @@ -347,6 +347,7 @@ config HW_RANDOM_STM32 tristate "STMicroelectronics STM32 random number generator" depends on HW_RANDOM && (ARCH_STM32 || COMPILE_TEST) depends on HAS_IOMEM + default HW_RANDOM help This driver provides kernel-side support for the Random Number Generator hardware found on STM32 microcontrollers. diff --git a/drivers/char/hw_random/n2-drv.c b/drivers/char/hw_random/n2-drv.c index 92dd4e925315..f8411515fe1c 100644 --- a/drivers/char/hw_random/n2-drv.c +++ b/drivers/char/hw_random/n2-drv.c @@ -435,7 +435,7 @@ static int n2rng_data_read(struct hwrng *rng, u32 *data) *data = np->test_data & 0xffffffff; len = 4; } else { - dev_err(&np->op->dev, "RNG error, restesting\n"); + dev_err(&np->op->dev, "RNG error, retesting\n"); np->flags &= ~N2RNG_FLAG_READY; if (!(np->flags & N2RNG_FLAG_SHUTDOWN)) schedule_delayed_work(&np->work, 0); diff --git a/drivers/char/hw_random/stm32-rng.c b/drivers/char/hw_random/stm32-rng.c index 0d2328da3b76..042860d97b15 100644 --- a/drivers/char/hw_random/stm32-rng.c +++ b/drivers/char/hw_random/stm32-rng.c @@ -187,8 +187,13 @@ static int stm32_rng_runtime_resume(struct device *dev) } #endif -static UNIVERSAL_DEV_PM_OPS(stm32_rng_pm_ops, stm32_rng_runtime_suspend, - stm32_rng_runtime_resume, NULL); +static const struct dev_pm_ops stm32_rng_pm_ops = { + SET_RUNTIME_PM_OPS(stm32_rng_runtime_suspend, + stm32_rng_runtime_resume, NULL) + SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend, + pm_runtime_force_resume) +}; + static const struct of_device_id stm32_rng_match[] = { { diff --git a/drivers/char/hw_random/via-rng.c b/drivers/char/hw_random/via-rng.c index 6e9df558325b..ffe9b0c6c647 100644 --- a/drivers/char/hw_random/via-rng.c +++ b/drivers/char/hw_random/via-rng.c @@ -135,7 +135,7 @@ static int via_rng_init(struct hwrng *rng) * is always enabled if CPUID rng_en is set. There is no * RNG configuration like it used to be the case in this * register */ - if ((c->x86 == 6) && (c->x86_model >= 0x0f)) { + if (((c->x86 == 6) && (c->x86_model >= 0x0f)) || (c->x86 > 6)){ if (!boot_cpu_has(X86_FEATURE_XSTORE_EN)) { pr_err(PFX "can't enable hardware RNG " "if XSTORE is not enabled\n"); diff --git a/drivers/char/ipmi/Kconfig b/drivers/char/ipmi/Kconfig index 3bda116c8aa0..c108441882cc 100644 --- a/drivers/char/ipmi/Kconfig +++ b/drivers/char/ipmi/Kconfig @@ -22,14 +22,6 @@ config IPMI_DMI_DECODE if IPMI_HANDLER -config IPMI_PROC_INTERFACE - bool 'Provide an interface for IPMI stats in /proc (deprecated)' - depends on PROC_FS - default y - help - Do not use this any more, use sysfs for this info. It will be - removed in future kernel versions. - config IPMI_PANIC_EVENT bool 'Generate a panic event to all BMCs on a panic' help @@ -111,6 +103,21 @@ config ASPEED_KCS_IPMI_BMC The driver implements the BMC side of the KCS contorller, it provides the access of KCS IO space for BMC side. +config NPCM7XX_KCS_IPMI_BMC + depends on ARCH_NPCM7XX || COMPILE_TEST + select IPMI_KCS_BMC + select REGMAP_MMIO + tristate "NPCM7xx KCS IPMI BMC driver" + help + Provides a driver for the KCS (Keyboard Controller Style) IPMI + interface found on Nuvoton NPCM7xx SOCs. + + The driver implements the BMC side of the KCS contorller, it + provides the access of KCS IO space for BMC side. + + This support is also available as a module. If so, the module + will be called kcs_bmc_npcm7xx. + config ASPEED_BT_IPMI_BMC depends on ARCH_ASPEED || COMPILE_TEST depends on REGMAP && REGMAP_MMIO && MFD_SYSCON diff --git a/drivers/char/ipmi/Makefile b/drivers/char/ipmi/Makefile index 21e9e872d973..7a3baf301a8f 100644 --- a/drivers/char/ipmi/Makefile +++ b/drivers/char/ipmi/Makefile @@ -24,3 +24,4 @@ obj-$(CONFIG_IPMI_POWEROFF) += ipmi_poweroff.o obj-$(CONFIG_IPMI_KCS_BMC) += kcs_bmc.o obj-$(CONFIG_ASPEED_BT_IPMI_BMC) += bt-bmc.o obj-$(CONFIG_ASPEED_KCS_IPMI_BMC) += kcs_bmc_aspeed.o +obj-$(CONFIG_NPCM7XX_KCS_IPMI_BMC) += kcs_bmc_npcm7xx.o diff --git a/drivers/char/ipmi/ipmi_bt_sm.c b/drivers/char/ipmi/ipmi_bt_sm.c index fd4ea8d87d4b..a3397664f800 100644 --- a/drivers/char/ipmi/ipmi_bt_sm.c +++ b/drivers/char/ipmi/ipmi_bt_sm.c @@ -504,11 +504,12 @@ static enum si_sm_result bt_event(struct si_sm_data *bt, long time) if (status & BT_H_BUSY) /* clear a leftover H_BUSY */ BT_CONTROL(BT_H_BUSY); + bt->timeout = bt->BT_CAP_req2rsp; + /* Read BT capabilities if it hasn't been done yet */ if (!bt->BT_CAP_outreqs) BT_STATE_CHANGE(BT_STATE_CAPABILITIES_BEGIN, SI_SM_CALL_WITHOUT_DELAY); - bt->timeout = bt->BT_CAP_req2rsp; BT_SI_SM_RETURN(SI_SM_IDLE); case BT_STATE_XACTION_START: diff --git a/drivers/char/ipmi/ipmi_devintf.c b/drivers/char/ipmi/ipmi_devintf.c index 8ecfd47806fa..1a486aec99b6 100644 --- a/drivers/char/ipmi/ipmi_devintf.c +++ b/drivers/char/ipmi/ipmi_devintf.c @@ -26,7 +26,7 @@ struct ipmi_file_private { - ipmi_user_t user; + struct ipmi_user *user; spinlock_t recv_msg_lock; struct list_head recv_msgs; struct file *file; @@ -37,7 +37,6 @@ struct ipmi_file_private unsigned int default_retry_time_ms; }; -static DEFINE_MUTEX(ipmi_mutex); static void file_receive_handler(struct ipmi_recv_msg *msg, void *handler_data) { @@ -45,17 +44,15 @@ static void file_receive_handler(struct ipmi_recv_msg *msg, int was_empty; unsigned long flags; - spin_lock_irqsave(&(priv->recv_msg_lock), flags); - - was_empty = list_empty(&(priv->recv_msgs)); - list_add_tail(&(msg->link), &(priv->recv_msgs)); + spin_lock_irqsave(&priv->recv_msg_lock, flags); + was_empty = list_empty(&priv->recv_msgs); + list_add_tail(&msg->link, &priv->recv_msgs); + spin_unlock_irqrestore(&priv->recv_msg_lock, flags); if (was_empty) { wake_up_interruptible(&priv->wait); kill_fasync(&priv->fasync_queue, SIGIO, POLL_IN); } - - spin_unlock_irqrestore(&(priv->recv_msg_lock), flags); } static __poll_t ipmi_poll(struct file *file, poll_table *wait) @@ -68,7 +65,7 @@ static __poll_t ipmi_poll(struct file *file, poll_table *wait) spin_lock_irqsave(&priv->recv_msg_lock, flags); - if (!list_empty(&(priv->recv_msgs))) + if (!list_empty(&priv->recv_msgs)) mask |= (EPOLLIN | EPOLLRDNORM); spin_unlock_irqrestore(&priv->recv_msg_lock, flags); @@ -79,13 +76,8 @@ static __poll_t ipmi_poll(struct file *file, poll_table *wait) static int ipmi_fasync(int fd, struct file *file, int on) { struct ipmi_file_private *priv = file->private_data; - int result; - mutex_lock(&ipmi_mutex); /* could race against open() otherwise */ - result = fasync_helper(fd, file, on, &priv->fasync_queue); - mutex_unlock(&ipmi_mutex); - - return (result); + return fasync_helper(fd, file, on, &priv->fasync_queue); } static const struct ipmi_user_hndl ipmi_hndlrs = @@ -99,18 +91,16 @@ static int ipmi_open(struct inode *inode, struct file *file) int rv; struct ipmi_file_private *priv; - priv = kmalloc(sizeof(*priv), GFP_KERNEL); if (!priv) return -ENOMEM; - mutex_lock(&ipmi_mutex); priv->file = file; rv = ipmi_create_user(if_num, &ipmi_hndlrs, priv, - &(priv->user)); + &priv->user); if (rv) { kfree(priv); goto out; @@ -118,8 +108,8 @@ static int ipmi_open(struct inode *inode, struct file *file) file->private_data = priv; - spin_lock_init(&(priv->recv_msg_lock)); - INIT_LIST_HEAD(&(priv->recv_msgs)); + spin_lock_init(&priv->recv_msg_lock); + INIT_LIST_HEAD(&priv->recv_msgs); init_waitqueue_head(&priv->wait); priv->fasync_queue = NULL; mutex_init(&priv->recv_mutex); @@ -129,7 +119,6 @@ static int ipmi_open(struct inode *inode, struct file *file) priv->default_retry_time_ms = 0; out: - mutex_unlock(&ipmi_mutex); return rv; } @@ -137,7 +126,7 @@ static int ipmi_release(struct inode *inode, struct file *file) { struct ipmi_file_private *priv = file->private_data; int rv; - struct ipmi_recv_msg *msg, *next; + struct ipmi_recv_msg *msg, *next; rv = ipmi_destroy_user(priv->user); if (rv) @@ -146,13 +135,12 @@ static int ipmi_release(struct inode *inode, struct file *file) list_for_each_entry_safe(msg, next, &priv->recv_msgs, link) ipmi_free_recv_msg(msg); - kfree(priv); return 0; } -static int handle_send_req(ipmi_user_t user, +static int handle_send_req(struct ipmi_user *user, struct ipmi_req *req, int retries, unsigned int retry_time_ms) @@ -189,8 +177,7 @@ static int handle_send_req(ipmi_user_t user, if (copy_from_user(msg.data, req->msg.data, - req->msg.data_len)) - { + req->msg.data_len)) { rv = -EFAULT; goto out; } @@ -233,25 +220,24 @@ static int handle_recv(struct ipmi_file_private *priv, mutex_lock(&priv->recv_mutex); /* Grab the message off the list. */ - spin_lock_irqsave(&(priv->recv_msg_lock), flags); + spin_lock_irqsave(&priv->recv_msg_lock, flags); if (list_empty(&(priv->recv_msgs))) { - spin_unlock_irqrestore(&(priv->recv_msg_lock), flags); + spin_unlock_irqrestore(&priv->recv_msg_lock, flags); rv = -EAGAIN; goto recv_err; } entry = priv->recv_msgs.next; msg = list_entry(entry, struct ipmi_recv_msg, link); list_del(entry); - spin_unlock_irqrestore(&(priv->recv_msg_lock), flags); + spin_unlock_irqrestore(&priv->recv_msg_lock, flags); addr_len = ipmi_addr_length(msg->addr.addr_type); - if (rsp->addr_len < addr_len) - { + if (rsp->addr_len < addr_len) { rv = -EINVAL; goto recv_putback_on_err; } - if (copy_to_user(rsp->addr, &(msg->addr), addr_len)) { + if (copy_to_user(rsp->addr, &msg->addr, addr_len)) { rv = -EFAULT; goto recv_putback_on_err; } @@ -273,8 +259,7 @@ static int handle_recv(struct ipmi_file_private *priv, if (copy_to_user(rsp->msg.data, msg->msg.data, - msg->msg.data_len)) - { + msg->msg.data_len)) { rv = -EFAULT; goto recv_putback_on_err; } @@ -294,9 +279,9 @@ static int handle_recv(struct ipmi_file_private *priv, recv_putback_on_err: /* If we got an error, put the message back onto the head of the queue. */ - spin_lock_irqsave(&(priv->recv_msg_lock), flags); - list_add(entry, &(priv->recv_msgs)); - spin_unlock_irqrestore(&(priv->recv_msg_lock), flags); + spin_lock_irqsave(&priv->recv_msg_lock, flags); + list_add(entry, &priv->recv_msgs); + spin_unlock_irqrestore(&priv->recv_msg_lock, flags); recv_err: mutex_unlock(&priv->recv_mutex); return rv; @@ -307,9 +292,9 @@ static int copyout_recv(struct ipmi_recv *rsp, void __user *to) return copy_to_user(to, rsp, sizeof(struct ipmi_recv)) ? -EFAULT : 0; } -static int ipmi_ioctl(struct file *file, - unsigned int cmd, - unsigned long data) +static long ipmi_ioctl(struct file *file, + unsigned int cmd, + unsigned long data) { int rv = -EINVAL; struct ipmi_file_private *priv = file->private_data; @@ -320,16 +305,20 @@ static int ipmi_ioctl(struct file *file, case IPMICTL_SEND_COMMAND: { struct ipmi_req req; + int retries; + unsigned int retry_time_ms; if (copy_from_user(&req, arg, sizeof(req))) { rv = -EFAULT; break; } - rv = handle_send_req(priv->user, - &req, - priv->default_retries, - priv->default_retry_time_ms); + mutex_lock(&priv->recv_mutex); + retries = priv->default_retries; + retry_time_ms = priv->default_retry_time_ms; + mutex_unlock(&priv->recv_mutex); + + rv = handle_send_req(priv->user, &req, retries, retry_time_ms); break; } @@ -569,8 +558,10 @@ static int ipmi_ioctl(struct file *file, break; } + mutex_lock(&priv->recv_mutex); priv->default_retries = parms.retries; priv->default_retry_time_ms = parms.retry_time_ms; + mutex_unlock(&priv->recv_mutex); rv = 0; break; } @@ -579,8 +570,10 @@ static int ipmi_ioctl(struct file *file, { struct ipmi_timing_parms parms; + mutex_lock(&priv->recv_mutex); parms.retries = priv->default_retries; parms.retry_time_ms = priv->default_retry_time_ms; + mutex_unlock(&priv->recv_mutex); if (copy_to_user(arg, &parms, sizeof(parms))) { rv = -EFAULT; @@ -615,30 +608,16 @@ static int ipmi_ioctl(struct file *file, rv = ipmi_set_maintenance_mode(priv->user, mode); break; } + + default: + rv = -ENOTTY; + break; } return rv; } -/* - * Note: it doesn't make sense to take the BKL here but - * not in compat_ipmi_ioctl. -arnd - */ -static long ipmi_unlocked_ioctl(struct file *file, - unsigned int cmd, - unsigned long data) -{ - int ret; - - mutex_lock(&ipmi_mutex); - ret = ipmi_ioctl(file, cmd, data); - mutex_unlock(&ipmi_mutex); - - return ret; -} - #ifdef CONFIG_COMPAT - /* * The following code contains code for supporting 32-bit compatible * ioctls on 64-bit kernels. This allows running 32-bit apps on the @@ -749,15 +728,21 @@ static long compat_ipmi_ioctl(struct file *filep, unsigned int cmd, { struct ipmi_req rp; struct compat_ipmi_req r32; + int retries; + unsigned int retry_time_ms; if (copy_from_user(&r32, compat_ptr(arg), sizeof(r32))) return -EFAULT; get_compat_ipmi_req(&rp, &r32); + mutex_lock(&priv->recv_mutex); + retries = priv->default_retries; + retry_time_ms = priv->default_retry_time_ms; + mutex_unlock(&priv->recv_mutex); + return handle_send_req(priv->user, &rp, - priv->default_retries, - priv->default_retry_time_ms); + retries, retry_time_ms); } case COMPAT_IPMICTL_SEND_COMMAND_SETTIME: { @@ -791,25 +776,13 @@ static long compat_ipmi_ioctl(struct file *filep, unsigned int cmd, return ipmi_ioctl(filep, cmd, arg); } } - -static long unlocked_compat_ipmi_ioctl(struct file *filep, unsigned int cmd, - unsigned long arg) -{ - int ret; - - mutex_lock(&ipmi_mutex); - ret = compat_ipmi_ioctl(filep, cmd, arg); - mutex_unlock(&ipmi_mutex); - - return ret; -} #endif static const struct file_operations ipmi_fops = { .owner = THIS_MODULE, - .unlocked_ioctl = ipmi_unlocked_ioctl, + .unlocked_ioctl = ipmi_ioctl, #ifdef CONFIG_COMPAT - .compat_ioctl = unlocked_compat_ipmi_ioctl, + .compat_ioctl = compat_ipmi_ioctl, #endif .open = ipmi_open, .release = ipmi_release, diff --git a/drivers/char/ipmi/ipmi_msghandler.c b/drivers/char/ipmi/ipmi_msghandler.c index 361148938801..51832b8a2c62 100644 --- a/drivers/char/ipmi/ipmi_msghandler.c +++ b/drivers/char/ipmi/ipmi_msghandler.c @@ -37,11 +37,30 @@ static struct ipmi_recv_msg *ipmi_alloc_recv_msg(void); static int ipmi_init_msghandler(void); static void smi_recv_tasklet(unsigned long); -static void handle_new_recv_msgs(ipmi_smi_t intf); -static void need_waiter(ipmi_smi_t intf); -static int handle_one_recv_msg(ipmi_smi_t intf, +static void handle_new_recv_msgs(struct ipmi_smi *intf); +static void need_waiter(struct ipmi_smi *intf); +static int handle_one_recv_msg(struct ipmi_smi *intf, struct ipmi_smi_msg *msg); +#ifdef DEBUG +static void ipmi_debug_msg(const char *title, unsigned char *data, + unsigned int len) +{ + int i, pos; + char buf[100]; + + pos = snprintf(buf, sizeof(buf), "%s: ", title); + for (i = 0; i < len; i++) + pos += snprintf(buf + pos, sizeof(buf) - pos, + " %2.2x", data[i]); + pr_debug("%s\n", buf); +} +#else +static void ipmi_debug_msg(const char *title, unsigned char *data, + unsigned int len) +{ } +#endif + static int initialized; enum ipmi_panic_event_op { @@ -112,14 +131,13 @@ module_param_cb(panic_op, &panic_op_ops, NULL, 0600); MODULE_PARM_DESC(panic_op, "Sets if the IPMI driver will attempt to store panic information in the event log in the event of a panic. Set to 'none' for no, 'event' for a single event, or 'string' for a generic event and the panic string in IPMI OEM events."); -#ifdef CONFIG_IPMI_PROC_INTERFACE -static struct proc_dir_entry *proc_ipmi_root; -#endif /* CONFIG_IPMI_PROC_INTERFACE */ +#define MAX_EVENTS_IN_QUEUE 25 /* Remain in auto-maintenance mode for this amount of time (in ms). */ -#define IPMI_MAINTENANCE_MODE_TIMEOUT 30000 - -#define MAX_EVENTS_IN_QUEUE 25 +static unsigned long maintenance_mode_timeout_ms = 30000; +module_param(maintenance_mode_timeout_ms, ulong, 0644); +MODULE_PARM_DESC(maintenance_mode_timeout_ms, + "The time (milliseconds) after the last maintenance message that the connection stays in maintenance mode."); /* * Don't let a message sit in a queue forever, always time it with at lest @@ -127,6 +145,31 @@ static struct proc_dir_entry *proc_ipmi_root; */ #define MAX_MSG_TIMEOUT 60000 +/* + * Timeout times below are in milliseconds, and are done off a 1 + * second timer. So setting the value to 1000 would mean anything + * between 0 and 1000ms. So really the only reasonable minimum + * setting it 2000ms, which is between 1 and 2 seconds. + */ + +/* The default timeout for message retries. */ +static unsigned long default_retry_ms = 2000; +module_param(default_retry_ms, ulong, 0644); +MODULE_PARM_DESC(default_retry_ms, + "The time (milliseconds) between retry sends"); + +/* The default timeout for maintenance mode message retries. */ +static unsigned long default_maintenance_retry_ms = 3000; +module_param(default_maintenance_retry_ms, ulong, 0644); +MODULE_PARM_DESC(default_maintenance_retry_ms, + "The time (milliseconds) between retry sends in maintenance mode"); + +/* The default maximum number of retries */ +static unsigned int default_max_retries = 4; +module_param(default_max_retries, uint, 0644); +MODULE_PARM_DESC(default_max_retries, + "The time (milliseconds) between retry sends in maintenance mode"); + /* Call every ~1000 ms. */ #define IPMI_TIMEOUT_TIME 1000 @@ -150,8 +193,12 @@ static struct proc_dir_entry *proc_ipmi_root; struct ipmi_user { struct list_head link; - /* Set to false when the user is destroyed. */ - bool valid; + /* + * Set to NULL when the user is destroyed, a pointer to myself + * so srcu_dereference can be used on it. + */ + struct ipmi_user *self; + struct srcu_struct release_barrier; struct kref refcount; @@ -160,16 +207,33 @@ struct ipmi_user { void *handler_data; /* The interface this user is bound to. */ - ipmi_smi_t intf; + struct ipmi_smi *intf; /* Does this interface receive IPMI events? */ bool gets_events; }; +static struct ipmi_user *acquire_ipmi_user(struct ipmi_user *user, int *index) + __acquires(user->release_barrier) +{ + struct ipmi_user *ruser; + + *index = srcu_read_lock(&user->release_barrier); + ruser = srcu_dereference(user->self, &user->release_barrier); + if (!ruser) + srcu_read_unlock(&user->release_barrier, *index); + return ruser; +} + +static void release_ipmi_user(struct ipmi_user *user, int index) +{ + srcu_read_unlock(&user->release_barrier, index); +} + struct cmd_rcvr { struct list_head link; - ipmi_user_t user; + struct ipmi_user *user; unsigned char netfn; unsigned char cmd; unsigned int chans; @@ -247,13 +311,6 @@ struct ipmi_my_addrinfo { unsigned char lun; }; -#ifdef CONFIG_IPMI_PROC_INTERFACE -struct ipmi_proc_entry { - char *name; - struct ipmi_proc_entry *next; -}; -#endif - /* * Note that the product id, manufacturer id, guid, and device id are * immutable in this structure, so dyn_mutex is not required for @@ -275,7 +332,7 @@ struct bmc_device { }; #define to_bmc_device(x) container_of((x), struct bmc_device, pdev.dev) -static int bmc_get_device_id(ipmi_smi_t intf, struct bmc_device *bmc, +static int bmc_get_device_id(struct ipmi_smi *intf, struct bmc_device *bmc, struct ipmi_device_id *id, bool *guid_set, guid_t *guid); @@ -397,10 +454,11 @@ struct ipmi_smi { struct list_head link; /* - * The list of upper layers that are using me. seq_lock - * protects this. + * The list of upper layers that are using me. seq_lock write + * protects this. Read protection is with srcu. */ struct list_head users; + struct srcu_struct users_srcu; /* Used for wake ups at startup. */ wait_queue_head_t waitq; @@ -420,24 +478,9 @@ struct ipmi_smi { bool in_bmc_register; /* Handle recursive situations. Yuck. */ struct work_struct bmc_reg_work; - /* - * This is the lower-layer's sender routine. Note that you - * must either be holding the ipmi_interfaces_mutex or be in - * an umpreemptible region to use this. You must fetch the - * value into a local variable and make sure it is not NULL. - */ const struct ipmi_smi_handlers *handlers; void *send_info; -#ifdef CONFIG_IPMI_PROC_INTERFACE - /* A list of proc entries for this interface. */ - struct mutex proc_entry_lock; - struct ipmi_proc_entry *proc_entries; - - struct proc_dir_entry *proc_dir; - char proc_dir_name[10]; -#endif - /* Driver-model device for the system interface. */ struct device *si_dev; @@ -502,6 +545,13 @@ struct ipmi_smi { int auto_maintenance_timeout; spinlock_t maintenance_mode_lock; /* Used in a timer... */ + /* + * If we are doing maintenance on something on IPMB, extend + * the timeout time to avoid timeouts writing firmware and + * such. + */ + int ipmb_maintenance_mode_timeout; + /* * A cheap hack, if this is non-null and a message to an * interface comes in with a NULL user, call this routine with @@ -510,7 +560,8 @@ struct ipmi_smi { * * Protected by bmc_reg_mutex. */ - void (*null_user_handler)(ipmi_smi_t intf, struct ipmi_recv_msg *msg); + void (*null_user_handler)(struct ipmi_smi *intf, + struct ipmi_recv_msg *msg); /* * When we are scanning the channels for an SMI, this will @@ -536,12 +587,12 @@ struct ipmi_smi { }; #define to_si_intf_from_dev(device) container_of(device, struct ipmi_smi, dev) -static void __get_guid(ipmi_smi_t intf); -static void __ipmi_bmc_unregister(ipmi_smi_t intf); -static int __ipmi_bmc_register(ipmi_smi_t intf, +static void __get_guid(struct ipmi_smi *intf); +static void __ipmi_bmc_unregister(struct ipmi_smi *intf); +static int __ipmi_bmc_register(struct ipmi_smi *intf, struct ipmi_device_id *id, bool guid_set, guid_t *guid, int intf_num); -static int __scan_channels(ipmi_smi_t intf, struct ipmi_device_id *id); +static int __scan_channels(struct ipmi_smi *intf, struct ipmi_device_id *id); /** @@ -560,6 +611,7 @@ static DEFINE_MUTEX(ipmidriver_mutex); static LIST_HEAD(ipmi_interfaces); static DEFINE_MUTEX(ipmi_interfaces_mutex); +DEFINE_STATIC_SRCU(ipmi_interfaces_srcu); /* * List of watchers that want to know when smi's are added and deleted. @@ -620,7 +672,7 @@ static void free_smi_msg_list(struct list_head *q) } } -static void clean_up_interface_data(ipmi_smi_t intf) +static void clean_up_interface_data(struct ipmi_smi *intf) { int i; struct cmd_rcvr *rcvr, *rcvr2; @@ -652,7 +704,7 @@ static void clean_up_interface_data(ipmi_smi_t intf) static void intf_free(struct kref *ref) { - ipmi_smi_t intf = container_of(ref, struct ipmi_smi, refcount); + struct ipmi_smi *intf = container_of(ref, struct ipmi_smi, refcount); clean_up_interface_data(intf); kfree(intf); @@ -660,65 +712,39 @@ static void intf_free(struct kref *ref) struct watcher_entry { int intf_num; - ipmi_smi_t intf; + struct ipmi_smi *intf; struct list_head link; }; int ipmi_smi_watcher_register(struct ipmi_smi_watcher *watcher) { - ipmi_smi_t intf; - LIST_HEAD(to_deliver); - struct watcher_entry *e, *e2; + struct ipmi_smi *intf; + int index; mutex_lock(&smi_watchers_mutex); - mutex_lock(&ipmi_interfaces_mutex); - - /* Build a list of things to deliver. */ - list_for_each_entry(intf, &ipmi_interfaces, link) { - if (intf->intf_num == -1) - continue; - e = kmalloc(sizeof(*e), GFP_KERNEL); - if (!e) - goto out_err; - kref_get(&intf->refcount); - e->intf = intf; - e->intf_num = intf->intf_num; - list_add_tail(&e->link, &to_deliver); - } - - /* We will succeed, so add it to the list. */ list_add(&watcher->link, &smi_watchers); - mutex_unlock(&ipmi_interfaces_mutex); + index = srcu_read_lock(&ipmi_interfaces_srcu); + list_for_each_entry_rcu(intf, &ipmi_interfaces, link) { + int intf_num = READ_ONCE(intf->intf_num); - list_for_each_entry_safe(e, e2, &to_deliver, link) { - list_del(&e->link); - watcher->new_smi(e->intf_num, e->intf->si_dev); - kref_put(&e->intf->refcount, intf_free); - kfree(e); + if (intf_num == -1) + continue; + watcher->new_smi(intf_num, intf->si_dev); } + srcu_read_unlock(&ipmi_interfaces_srcu, index); mutex_unlock(&smi_watchers_mutex); return 0; - - out_err: - mutex_unlock(&ipmi_interfaces_mutex); - mutex_unlock(&smi_watchers_mutex); - list_for_each_entry_safe(e, e2, &to_deliver, link) { - list_del(&e->link); - kref_put(&e->intf->refcount, intf_free); - kfree(e); - } - return -ENOMEM; } EXPORT_SYMBOL(ipmi_smi_watcher_register); int ipmi_smi_watcher_unregister(struct ipmi_smi_watcher *watcher) { mutex_lock(&smi_watchers_mutex); - list_del(&(watcher->link)); + list_del(&watcher->link); mutex_unlock(&smi_watchers_mutex); return 0; } @@ -732,12 +758,14 @@ call_smi_watchers(int i, struct device *dev) { struct ipmi_smi_watcher *w; + mutex_lock(&smi_watchers_mutex); list_for_each_entry(w, &smi_watchers, link) { if (try_module_get(w->owner)) { w->new_smi(i, dev); module_put(w->owner); } } + mutex_unlock(&smi_watchers_mutex); } static int @@ -831,18 +859,17 @@ unsigned int ipmi_addr_length(int addr_type) } EXPORT_SYMBOL(ipmi_addr_length); -static void deliver_response(struct ipmi_recv_msg *msg) +static int deliver_response(struct ipmi_smi *intf, struct ipmi_recv_msg *msg) { - if (!msg->user) { - ipmi_smi_t intf = msg->user_msg_data; + int rv = 0; + if (!msg->user) { /* Special handling for NULL users. */ if (intf->null_user_handler) { intf->null_user_handler(intf, msg); - ipmi_inc_stat(intf, handled_local_responses); } else { /* No handler, so give up. */ - ipmi_inc_stat(intf, unhandled_local_responses); + rv = -EINVAL; } ipmi_free_recv_msg(msg); } else if (!oops_in_progress) { @@ -851,21 +878,40 @@ static void deliver_response(struct ipmi_recv_msg *msg) * receive handler doesn't much meaning and has a deadlock * risk. At this moment, simply skip it in that case. */ + int index; + struct ipmi_user *user = acquire_ipmi_user(msg->user, &index); - ipmi_user_t user = msg->user; - user->handler->ipmi_recv_hndl(msg, user->handler_data); + if (user) { + user->handler->ipmi_recv_hndl(msg, user->handler_data); + release_ipmi_user(msg->user, index); + } else { + /* User went away, give up. */ + ipmi_free_recv_msg(msg); + rv = -EINVAL; + } } + + return rv; } -static void -deliver_err_response(struct ipmi_recv_msg *msg, int err) +static void deliver_local_response(struct ipmi_smi *intf, + struct ipmi_recv_msg *msg) +{ + if (deliver_response(intf, msg)) + ipmi_inc_stat(intf, unhandled_local_responses); + else + ipmi_inc_stat(intf, handled_local_responses); +} + +static void deliver_err_response(struct ipmi_smi *intf, + struct ipmi_recv_msg *msg, int err) { msg->recv_type = IPMI_RESPONSE_RECV_TYPE; msg->msg_data[0] = err; msg->msg.netfn |= 1; /* Convert to a response. */ msg->msg.data_len = 1; msg->msg.data = msg->msg_data; - deliver_response(msg); + deliver_local_response(intf, msg); } /* @@ -873,7 +919,7 @@ deliver_err_response(struct ipmi_recv_msg *msg, int err) * message with the given timeout to the sequence table. This must be * called with the interface's seq_lock held. */ -static int intf_next_seq(ipmi_smi_t intf, +static int intf_next_seq(struct ipmi_smi *intf, struct ipmi_recv_msg *recv_msg, unsigned long timeout, int retries, @@ -884,6 +930,11 @@ static int intf_next_seq(ipmi_smi_t intf, int rv = 0; unsigned int i; + if (timeout == 0) + timeout = default_retry_ms; + if (retries < 0) + retries = default_max_retries; + for (i = intf->curr_seq; (i+1)%IPMI_IPMB_NUM_SEQ != intf->curr_seq; i = (i+1)%IPMI_IPMB_NUM_SEQ) { if (!intf->seq_table[i].inuse) @@ -921,7 +972,7 @@ static int intf_next_seq(ipmi_smi_t intf, * guard against message coming in after their timeout and the * sequence number being reused). */ -static int intf_find_seq(ipmi_smi_t intf, +static int intf_find_seq(struct ipmi_smi *intf, unsigned char seq, short channel, unsigned char cmd, @@ -935,26 +986,26 @@ static int intf_find_seq(ipmi_smi_t intf, if (seq >= IPMI_IPMB_NUM_SEQ) return -EINVAL; - spin_lock_irqsave(&(intf->seq_lock), flags); + spin_lock_irqsave(&intf->seq_lock, flags); if (intf->seq_table[seq].inuse) { struct ipmi_recv_msg *msg = intf->seq_table[seq].recv_msg; if ((msg->addr.channel == channel) && (msg->msg.cmd == cmd) && (msg->msg.netfn == netfn) - && (ipmi_addr_equal(addr, &(msg->addr)))) { + && (ipmi_addr_equal(addr, &msg->addr))) { *recv_msg = msg; intf->seq_table[seq].inuse = 0; rv = 0; } } - spin_unlock_irqrestore(&(intf->seq_lock), flags); + spin_unlock_irqrestore(&intf->seq_lock, flags); return rv; } /* Start the timer for a specific sequence table entry. */ -static int intf_start_seq_timer(ipmi_smi_t intf, +static int intf_start_seq_timer(struct ipmi_smi *intf, long msgid) { int rv = -ENODEV; @@ -965,24 +1016,24 @@ static int intf_start_seq_timer(ipmi_smi_t intf, GET_SEQ_FROM_MSGID(msgid, seq, seqid); - spin_lock_irqsave(&(intf->seq_lock), flags); + spin_lock_irqsave(&intf->seq_lock, flags); /* * We do this verification because the user can be deleted * while a message is outstanding. */ if ((intf->seq_table[seq].inuse) && (intf->seq_table[seq].seqid == seqid)) { - struct seq_table *ent = &(intf->seq_table[seq]); + struct seq_table *ent = &intf->seq_table[seq]; ent->timeout = ent->orig_timeout; rv = 0; } - spin_unlock_irqrestore(&(intf->seq_lock), flags); + spin_unlock_irqrestore(&intf->seq_lock, flags); return rv; } /* Got an error for the send message for a specific sequence number. */ -static int intf_err_seq(ipmi_smi_t intf, +static int intf_err_seq(struct ipmi_smi *intf, long msgid, unsigned int err) { @@ -995,23 +1046,23 @@ static int intf_err_seq(ipmi_smi_t intf, GET_SEQ_FROM_MSGID(msgid, seq, seqid); - spin_lock_irqsave(&(intf->seq_lock), flags); + spin_lock_irqsave(&intf->seq_lock, flags); /* * We do this verification because the user can be deleted * while a message is outstanding. */ if ((intf->seq_table[seq].inuse) && (intf->seq_table[seq].seqid == seqid)) { - struct seq_table *ent = &(intf->seq_table[seq]); + struct seq_table *ent = &intf->seq_table[seq]; ent->inuse = 0; msg = ent->recv_msg; rv = 0; } - spin_unlock_irqrestore(&(intf->seq_lock), flags); + spin_unlock_irqrestore(&intf->seq_lock, flags); if (msg) - deliver_err_response(msg, err); + deliver_err_response(intf, msg, err); return rv; } @@ -1020,12 +1071,12 @@ static int intf_err_seq(ipmi_smi_t intf, int ipmi_create_user(unsigned int if_num, const struct ipmi_user_hndl *handler, void *handler_data, - ipmi_user_t *user) + struct ipmi_user **user) { unsigned long flags; - ipmi_user_t new_user; - int rv = 0; - ipmi_smi_t intf; + struct ipmi_user *new_user; + int rv = 0, index; + struct ipmi_smi *intf; /* * There is no module usecount here, because it's not @@ -1059,7 +1110,7 @@ int ipmi_create_user(unsigned int if_num, if (!new_user) return -ENOMEM; - mutex_lock(&ipmi_interfaces_mutex); + index = srcu_read_lock(&ipmi_interfaces_srcu); list_for_each_entry_rcu(intf, &ipmi_interfaces, link) { if (intf->intf_num == if_num) goto found; @@ -1069,6 +1120,10 @@ int ipmi_create_user(unsigned int if_num, goto out_kfree; found: + rv = init_srcu_struct(&new_user->release_barrier); + if (rv) + goto out_kfree; + /* Note that each existing user holds a refcount to the interface. */ kref_get(&intf->refcount); @@ -1078,26 +1133,7 @@ int ipmi_create_user(unsigned int if_num, new_user->intf = intf; new_user->gets_events = false; - if (!try_module_get(intf->handlers->owner)) { - rv = -ENODEV; - goto out_kref; - } - - if (intf->handlers->inc_usecount) { - rv = intf->handlers->inc_usecount(intf->send_info); - if (rv) { - module_put(intf->handlers->owner); - goto out_kref; - } - } - - /* - * Hold the lock so intf->handlers is guaranteed to be good - * until now - */ - mutex_unlock(&ipmi_interfaces_mutex); - - new_user->valid = true; + rcu_assign_pointer(new_user->self, new_user); spin_lock_irqsave(&intf->seq_lock, flags); list_add_rcu(&new_user->link, &intf->users); spin_unlock_irqrestore(&intf->seq_lock, flags); @@ -1106,13 +1142,12 @@ int ipmi_create_user(unsigned int if_num, if (atomic_inc_return(&intf->event_waiters) == 1) need_waiter(intf); } + srcu_read_unlock(&ipmi_interfaces_srcu, index); *user = new_user; return 0; -out_kref: - kref_put(&intf->refcount, intf_free); out_kfree: - mutex_unlock(&ipmi_interfaces_mutex); + srcu_read_unlock(&ipmi_interfaces_srcu, index); kfree(new_user); return rv; } @@ -1120,26 +1155,25 @@ EXPORT_SYMBOL(ipmi_create_user); int ipmi_get_smi_info(int if_num, struct ipmi_smi_info *data) { - int rv = 0; - ipmi_smi_t intf; - const struct ipmi_smi_handlers *handlers; + int rv, index; + struct ipmi_smi *intf; - mutex_lock(&ipmi_interfaces_mutex); + index = srcu_read_lock(&ipmi_interfaces_srcu); list_for_each_entry_rcu(intf, &ipmi_interfaces, link) { if (intf->intf_num == if_num) goto found; } + srcu_read_unlock(&ipmi_interfaces_srcu, index); + /* Not found, return an error */ - rv = -EINVAL; - mutex_unlock(&ipmi_interfaces_mutex); - return rv; + return -EINVAL; found: - handlers = intf->handlers; - rv = -ENOSYS; - if (handlers->get_smi_info) - rv = handlers->get_smi_info(intf->send_info, data); - mutex_unlock(&ipmi_interfaces_mutex); + if (!intf->handlers->get_smi_info) + rv = -ENOTTY; + else + rv = intf->handlers->get_smi_info(intf->send_info, data); + srcu_read_unlock(&ipmi_interfaces_srcu, index); return rv; } @@ -1147,19 +1181,34 @@ EXPORT_SYMBOL(ipmi_get_smi_info); static void free_user(struct kref *ref) { - ipmi_user_t user = container_of(ref, struct ipmi_user, refcount); + struct ipmi_user *user = container_of(ref, struct ipmi_user, refcount); kfree(user); } -int ipmi_destroy_user(ipmi_user_t user) +static void _ipmi_destroy_user(struct ipmi_user *user) { - ipmi_smi_t intf = user->intf; + struct ipmi_smi *intf = user->intf; int i; unsigned long flags; struct cmd_rcvr *rcvr; struct cmd_rcvr *rcvrs = NULL; - user->valid = false; + if (!acquire_ipmi_user(user, &i)) { + /* + * The user has already been cleaned up, just make sure + * nothing is using it and return. + */ + synchronize_srcu(&user->release_barrier); + return; + } + + rcu_assign_pointer(user->self, NULL); + release_ipmi_user(user, i); + + synchronize_srcu(&user->release_barrier); + + if (user->handler->shutdown) + user->handler->shutdown(user->handler_data); if (user->handler->ipmi_watchdog_pretimeout) atomic_dec(&intf->event_waiters); @@ -1184,7 +1233,7 @@ int ipmi_destroy_user(ipmi_user_t user) * Remove the user from the command receiver's table. First * we build a list of everything (not using the standard link, * since other things may be using it till we do - * synchronize_rcu()) then free everything in that list. + * synchronize_srcu()) then free everything in that list. */ mutex_lock(&intf->cmd_rcvrs_mutex); list_for_each_entry_rcu(rcvr, &intf->cmd_rcvrs, link) { @@ -1202,109 +1251,156 @@ int ipmi_destroy_user(ipmi_user_t user) kfree(rcvr); } - mutex_lock(&ipmi_interfaces_mutex); - if (intf->handlers) { - module_put(intf->handlers->owner); - if (intf->handlers->dec_usecount) - intf->handlers->dec_usecount(intf->send_info); - } - mutex_unlock(&ipmi_interfaces_mutex); - kref_put(&intf->refcount, intf_free); +} +int ipmi_destroy_user(struct ipmi_user *user) +{ + _ipmi_destroy_user(user); + + cleanup_srcu_struct(&user->release_barrier); kref_put(&user->refcount, free_user); return 0; } EXPORT_SYMBOL(ipmi_destroy_user); -int ipmi_get_version(ipmi_user_t user, +int ipmi_get_version(struct ipmi_user *user, unsigned char *major, unsigned char *minor) { struct ipmi_device_id id; - int rv; + int rv, index; - rv = bmc_get_device_id(user->intf, NULL, &id, NULL, NULL); - if (rv) - return rv; + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; - *major = ipmi_version_major(&id); - *minor = ipmi_version_minor(&id); + rv = bmc_get_device_id(user->intf, NULL, &id, NULL, NULL); + if (!rv) { + *major = ipmi_version_major(&id); + *minor = ipmi_version_minor(&id); + } + release_ipmi_user(user, index); - return 0; + return rv; } EXPORT_SYMBOL(ipmi_get_version); -int ipmi_set_my_address(ipmi_user_t user, +int ipmi_set_my_address(struct ipmi_user *user, unsigned int channel, unsigned char address) { + int index, rv = 0; + + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; + if (channel >= IPMI_MAX_CHANNELS) - return -EINVAL; - user->intf->addrinfo[channel].address = address; - return 0; + rv = -EINVAL; + else + user->intf->addrinfo[channel].address = address; + release_ipmi_user(user, index); + + return rv; } EXPORT_SYMBOL(ipmi_set_my_address); -int ipmi_get_my_address(ipmi_user_t user, +int ipmi_get_my_address(struct ipmi_user *user, unsigned int channel, unsigned char *address) { + int index, rv = 0; + + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; + if (channel >= IPMI_MAX_CHANNELS) - return -EINVAL; - *address = user->intf->addrinfo[channel].address; - return 0; + rv = -EINVAL; + else + *address = user->intf->addrinfo[channel].address; + release_ipmi_user(user, index); + + return rv; } EXPORT_SYMBOL(ipmi_get_my_address); -int ipmi_set_my_LUN(ipmi_user_t user, +int ipmi_set_my_LUN(struct ipmi_user *user, unsigned int channel, unsigned char LUN) { + int index, rv = 0; + + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; + if (channel >= IPMI_MAX_CHANNELS) - return -EINVAL; - user->intf->addrinfo[channel].lun = LUN & 0x3; + rv = -EINVAL; + else + user->intf->addrinfo[channel].lun = LUN & 0x3; + release_ipmi_user(user, index); + return 0; } EXPORT_SYMBOL(ipmi_set_my_LUN); -int ipmi_get_my_LUN(ipmi_user_t user, +int ipmi_get_my_LUN(struct ipmi_user *user, unsigned int channel, unsigned char *address) { + int index, rv = 0; + + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; + if (channel >= IPMI_MAX_CHANNELS) - return -EINVAL; - *address = user->intf->addrinfo[channel].lun; - return 0; + rv = -EINVAL; + else + *address = user->intf->addrinfo[channel].lun; + release_ipmi_user(user, index); + + return rv; } EXPORT_SYMBOL(ipmi_get_my_LUN); -int ipmi_get_maintenance_mode(ipmi_user_t user) +int ipmi_get_maintenance_mode(struct ipmi_user *user) { - int mode; + int mode, index; unsigned long flags; + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; + spin_lock_irqsave(&user->intf->maintenance_mode_lock, flags); mode = user->intf->maintenance_mode; spin_unlock_irqrestore(&user->intf->maintenance_mode_lock, flags); + release_ipmi_user(user, index); return mode; } EXPORT_SYMBOL(ipmi_get_maintenance_mode); -static void maintenance_mode_update(ipmi_smi_t intf) +static void maintenance_mode_update(struct ipmi_smi *intf) { if (intf->handlers->set_maintenance_mode) intf->handlers->set_maintenance_mode( intf->send_info, intf->maintenance_mode_enable); } -int ipmi_set_maintenance_mode(ipmi_user_t user, int mode) +int ipmi_set_maintenance_mode(struct ipmi_user *user, int mode) { - int rv = 0; + int rv = 0, index; unsigned long flags; - ipmi_smi_t intf = user->intf; + struct ipmi_smi *intf = user->intf; + + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; spin_lock_irqsave(&intf->maintenance_mode_lock, flags); if (intf->maintenance_mode != mode) { @@ -1332,17 +1428,23 @@ int ipmi_set_maintenance_mode(ipmi_user_t user, int mode) } out_unlock: spin_unlock_irqrestore(&intf->maintenance_mode_lock, flags); + release_ipmi_user(user, index); return rv; } EXPORT_SYMBOL(ipmi_set_maintenance_mode); -int ipmi_set_gets_events(ipmi_user_t user, bool val) +int ipmi_set_gets_events(struct ipmi_user *user, bool val) { unsigned long flags; - ipmi_smi_t intf = user->intf; + struct ipmi_smi *intf = user->intf; struct ipmi_recv_msg *msg, *msg2; struct list_head msgs; + int index; + + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; INIT_LIST_HEAD(&msgs); @@ -1383,7 +1485,7 @@ int ipmi_set_gets_events(ipmi_user_t user, bool val) list_for_each_entry_safe(msg, msg2, &msgs, link) { msg->user = user; kref_get(&user->refcount); - deliver_response(msg); + deliver_local_response(intf, msg); } spin_lock_irqsave(&intf->events_lock, flags); @@ -1392,12 +1494,13 @@ int ipmi_set_gets_events(ipmi_user_t user, bool val) out: spin_unlock_irqrestore(&intf->events_lock, flags); + release_ipmi_user(user, index); return 0; } EXPORT_SYMBOL(ipmi_set_gets_events); -static struct cmd_rcvr *find_cmd_rcvr(ipmi_smi_t intf, +static struct cmd_rcvr *find_cmd_rcvr(struct ipmi_smi *intf, unsigned char netfn, unsigned char cmd, unsigned char chan) @@ -1412,7 +1515,7 @@ static struct cmd_rcvr *find_cmd_rcvr(ipmi_smi_t intf, return NULL; } -static int is_cmd_rcvr_exclusive(ipmi_smi_t intf, +static int is_cmd_rcvr_exclusive(struct ipmi_smi *intf, unsigned char netfn, unsigned char cmd, unsigned int chans) @@ -1427,19 +1530,24 @@ static int is_cmd_rcvr_exclusive(ipmi_smi_t intf, return 1; } -int ipmi_register_for_cmd(ipmi_user_t user, +int ipmi_register_for_cmd(struct ipmi_user *user, unsigned char netfn, unsigned char cmd, unsigned int chans) { - ipmi_smi_t intf = user->intf; + struct ipmi_smi *intf = user->intf; struct cmd_rcvr *rcvr; - int rv = 0; + int rv = 0, index; + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; rcvr = kmalloc(sizeof(*rcvr), GFP_KERNEL); - if (!rcvr) - return -ENOMEM; + if (!rcvr) { + rv = -ENOMEM; + goto out_release; + } rcvr->cmd = cmd; rcvr->netfn = netfn; rcvr->chans = chans; @@ -1457,24 +1565,30 @@ int ipmi_register_for_cmd(ipmi_user_t user, list_add_rcu(&rcvr->link, &intf->cmd_rcvrs); - out_unlock: +out_unlock: mutex_unlock(&intf->cmd_rcvrs_mutex); if (rv) kfree(rcvr); +out_release: + release_ipmi_user(user, index); return rv; } EXPORT_SYMBOL(ipmi_register_for_cmd); -int ipmi_unregister_for_cmd(ipmi_user_t user, +int ipmi_unregister_for_cmd(struct ipmi_user *user, unsigned char netfn, unsigned char cmd, unsigned int chans) { - ipmi_smi_t intf = user->intf; + struct ipmi_smi *intf = user->intf; struct cmd_rcvr *rcvr; struct cmd_rcvr *rcvrs = NULL; - int i, rv = -ENOENT; + int i, rv = -ENOENT, index; + + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; mutex_lock(&intf->cmd_rcvrs_mutex); for (i = 0; i < IPMI_NUM_CHANNELS; i++) { @@ -1495,12 +1609,14 @@ int ipmi_unregister_for_cmd(ipmi_user_t user, } mutex_unlock(&intf->cmd_rcvrs_mutex); synchronize_rcu(); + release_ipmi_user(user, index); while (rcvrs) { atomic_dec(&intf->event_waiters); rcvr = rcvrs; rcvrs = rcvr->next; kfree(rcvr); } + return rv; } EXPORT_SYMBOL(ipmi_unregister_for_cmd); @@ -1535,21 +1651,19 @@ static inline void format_ipmb_msg(struct ipmi_smi_msg *smi_msg, smi_msg->data[3] = 0; smi_msg->data[i+3] = ipmb_addr->slave_addr; smi_msg->data[i+4] = (msg->netfn << 2) | (ipmb_addr->lun & 0x3); - smi_msg->data[i+5] = ipmb_checksum(&(smi_msg->data[i+3]), 2); + smi_msg->data[i+5] = ipmb_checksum(&smi_msg->data[i + 3], 2); smi_msg->data[i+6] = source_address; smi_msg->data[i+7] = (ipmb_seq << 2) | source_lun; smi_msg->data[i+8] = msg->cmd; /* Now tack on the data to the message. */ if (msg->data_len > 0) - memcpy(&(smi_msg->data[i+9]), msg->data, - msg->data_len); + memcpy(&smi_msg->data[i + 9], msg->data, msg->data_len); smi_msg->data_size = msg->data_len + 9; /* Now calculate the checksum and tack it on. */ smi_msg->data[i+smi_msg->data_size] - = ipmb_checksum(&(smi_msg->data[i+6]), - smi_msg->data_size-6); + = ipmb_checksum(&smi_msg->data[i + 6], smi_msg->data_size - 6); /* * Add on the checksum size and the offset from the @@ -1574,21 +1688,19 @@ static inline void format_lan_msg(struct ipmi_smi_msg *smi_msg, smi_msg->data[3] = lan_addr->session_handle; smi_msg->data[4] = lan_addr->remote_SWID; smi_msg->data[5] = (msg->netfn << 2) | (lan_addr->lun & 0x3); - smi_msg->data[6] = ipmb_checksum(&(smi_msg->data[4]), 2); + smi_msg->data[6] = ipmb_checksum(&smi_msg->data[4], 2); smi_msg->data[7] = lan_addr->local_SWID; smi_msg->data[8] = (ipmb_seq << 2) | source_lun; smi_msg->data[9] = msg->cmd; /* Now tack on the data to the message. */ if (msg->data_len > 0) - memcpy(&(smi_msg->data[10]), msg->data, - msg->data_len); + memcpy(&smi_msg->data[10], msg->data, msg->data_len); smi_msg->data_size = msg->data_len + 10; /* Now calculate the checksum and tack it on. */ smi_msg->data[smi_msg->data_size] - = ipmb_checksum(&(smi_msg->data[7]), - smi_msg->data_size-7); + = ipmb_checksum(&smi_msg->data[7], smi_msg->data_size - 7); /* * Add on the checksum size and the offset from the @@ -1599,7 +1711,7 @@ static inline void format_lan_msg(struct ipmi_smi_msg *smi_msg, smi_msg->msgid = msgid; } -static struct ipmi_smi_msg *smi_add_send_msg(ipmi_smi_t intf, +static struct ipmi_smi_msg *smi_add_send_msg(struct ipmi_smi *intf, struct ipmi_smi_msg *smi_msg, int priority) { @@ -1617,7 +1729,8 @@ static struct ipmi_smi_msg *smi_add_send_msg(ipmi_smi_t intf, } -static void smi_send(ipmi_smi_t intf, const struct ipmi_smi_handlers *handlers, +static void smi_send(struct ipmi_smi *intf, + const struct ipmi_smi_handlers *handlers, struct ipmi_smi_msg *smi_msg, int priority) { int run_to_completion = intf->run_to_completion; @@ -1636,405 +1749,435 @@ static void smi_send(ipmi_smi_t intf, const struct ipmi_smi_handlers *handlers, handlers->sender(intf->send_info, smi_msg); } -/* - * Separate from ipmi_request so that the user does not have to be - * supplied in certain circumstances (mainly at panic time). If - * messages are supplied, they will be freed, even if an error - * occurs. - */ -static int i_ipmi_request(ipmi_user_t user, - ipmi_smi_t intf, - struct ipmi_addr *addr, - long msgid, - struct kernel_ipmi_msg *msg, - void *user_msg_data, - void *supplied_smi, - struct ipmi_recv_msg *supplied_recv, - int priority, - unsigned char source_address, - unsigned char source_lun, - int retries, - unsigned int retry_time_ms) +static bool is_maintenance_mode_cmd(struct kernel_ipmi_msg *msg) { - int rv = 0; - struct ipmi_smi_msg *smi_msg; - struct ipmi_recv_msg *recv_msg; - unsigned long flags; + return (((msg->netfn == IPMI_NETFN_APP_REQUEST) + && ((msg->cmd == IPMI_COLD_RESET_CMD) + || (msg->cmd == IPMI_WARM_RESET_CMD))) + || (msg->netfn == IPMI_NETFN_FIRMWARE_REQUEST)); +} +static int i_ipmi_req_sysintf(struct ipmi_smi *intf, + struct ipmi_addr *addr, + long msgid, + struct kernel_ipmi_msg *msg, + struct ipmi_smi_msg *smi_msg, + struct ipmi_recv_msg *recv_msg, + int retries, + unsigned int retry_time_ms) +{ + struct ipmi_system_interface_addr *smi_addr; - if (supplied_recv) - recv_msg = supplied_recv; - else { - recv_msg = ipmi_alloc_recv_msg(); - if (recv_msg == NULL) - return -ENOMEM; - } - recv_msg->user_msg_data = user_msg_data; + if (msg->netfn & 1) + /* Responses are not allowed to the SMI. */ + return -EINVAL; - if (supplied_smi) - smi_msg = (struct ipmi_smi_msg *) supplied_smi; - else { - smi_msg = ipmi_alloc_smi_msg(); - if (smi_msg == NULL) { - ipmi_free_recv_msg(recv_msg); - return -ENOMEM; - } + smi_addr = (struct ipmi_system_interface_addr *) addr; + if (smi_addr->lun > 3) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EINVAL; } - rcu_read_lock(); - if (intf->in_shutdown) { - rv = -ENODEV; - goto out_err; - } + memcpy(&recv_msg->addr, smi_addr, sizeof(*smi_addr)); - recv_msg->user = user; - if (user) - kref_get(&user->refcount); - recv_msg->msgid = msgid; - /* - * Store the message to send in the receive message so timeout - * responses can get the proper response data. - */ - recv_msg->msg = *msg; + if ((msg->netfn == IPMI_NETFN_APP_REQUEST) + && ((msg->cmd == IPMI_SEND_MSG_CMD) + || (msg->cmd == IPMI_GET_MSG_CMD) + || (msg->cmd == IPMI_READ_EVENT_MSG_BUFFER_CMD))) { + /* + * We don't let the user do these, since we manage + * the sequence numbers. + */ + ipmi_inc_stat(intf, sent_invalid_commands); + return -EINVAL; + } - if (addr->addr_type == IPMI_SYSTEM_INTERFACE_ADDR_TYPE) { - struct ipmi_system_interface_addr *smi_addr; + if (is_maintenance_mode_cmd(msg)) { + unsigned long flags; - if (msg->netfn & 1) { - /* Responses are not allowed to the SMI. */ - rv = -EINVAL; - goto out_err; + spin_lock_irqsave(&intf->maintenance_mode_lock, flags); + intf->auto_maintenance_timeout + = maintenance_mode_timeout_ms; + if (!intf->maintenance_mode + && !intf->maintenance_mode_enable) { + intf->maintenance_mode_enable = true; + maintenance_mode_update(intf); } + spin_unlock_irqrestore(&intf->maintenance_mode_lock, + flags); + } - smi_addr = (struct ipmi_system_interface_addr *) addr; - if (smi_addr->lun > 3) { - ipmi_inc_stat(intf, sent_invalid_commands); - rv = -EINVAL; - goto out_err; - } + if (msg->data_len + 2 > IPMI_MAX_MSG_LENGTH) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EMSGSIZE; + } - memcpy(&recv_msg->addr, smi_addr, sizeof(*smi_addr)); + smi_msg->data[0] = (msg->netfn << 2) | (smi_addr->lun & 0x3); + smi_msg->data[1] = msg->cmd; + smi_msg->msgid = msgid; + smi_msg->user_data = recv_msg; + if (msg->data_len > 0) + memcpy(&smi_msg->data[2], msg->data, msg->data_len); + smi_msg->data_size = msg->data_len + 2; + ipmi_inc_stat(intf, sent_local_commands); - if ((msg->netfn == IPMI_NETFN_APP_REQUEST) - && ((msg->cmd == IPMI_SEND_MSG_CMD) - || (msg->cmd == IPMI_GET_MSG_CMD) - || (msg->cmd == IPMI_READ_EVENT_MSG_BUFFER_CMD))) { - /* - * We don't let the user do these, since we manage - * the sequence numbers. - */ - ipmi_inc_stat(intf, sent_invalid_commands); - rv = -EINVAL; - goto out_err; - } + return 0; +} - if (((msg->netfn == IPMI_NETFN_APP_REQUEST) - && ((msg->cmd == IPMI_COLD_RESET_CMD) - || (msg->cmd == IPMI_WARM_RESET_CMD))) - || (msg->netfn == IPMI_NETFN_FIRMWARE_REQUEST)) { - spin_lock_irqsave(&intf->maintenance_mode_lock, flags); - intf->auto_maintenance_timeout - = IPMI_MAINTENANCE_MODE_TIMEOUT; - if (!intf->maintenance_mode - && !intf->maintenance_mode_enable) { - intf->maintenance_mode_enable = true; - maintenance_mode_update(intf); - } - spin_unlock_irqrestore(&intf->maintenance_mode_lock, - flags); - } +static int i_ipmi_req_ipmb(struct ipmi_smi *intf, + struct ipmi_addr *addr, + long msgid, + struct kernel_ipmi_msg *msg, + struct ipmi_smi_msg *smi_msg, + struct ipmi_recv_msg *recv_msg, + unsigned char source_address, + unsigned char source_lun, + int retries, + unsigned int retry_time_ms) +{ + struct ipmi_ipmb_addr *ipmb_addr; + unsigned char ipmb_seq; + long seqid; + int broadcast = 0; + struct ipmi_channel *chans; + int rv = 0; - if ((msg->data_len + 2) > IPMI_MAX_MSG_LENGTH) { - ipmi_inc_stat(intf, sent_invalid_commands); - rv = -EMSGSIZE; - goto out_err; - } + if (addr->channel >= IPMI_MAX_CHANNELS) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EINVAL; + } - smi_msg->data[0] = (msg->netfn << 2) | (smi_addr->lun & 0x3); - smi_msg->data[1] = msg->cmd; - smi_msg->msgid = msgid; - smi_msg->user_data = recv_msg; - if (msg->data_len > 0) - memcpy(&(smi_msg->data[2]), msg->data, msg->data_len); - smi_msg->data_size = msg->data_len + 2; - ipmi_inc_stat(intf, sent_local_commands); - } else if (is_ipmb_addr(addr) || is_ipmb_bcast_addr(addr)) { - struct ipmi_ipmb_addr *ipmb_addr; - unsigned char ipmb_seq; - long seqid; - int broadcast = 0; - struct ipmi_channel *chans; + chans = READ_ONCE(intf->channel_list)->c; - if (addr->channel >= IPMI_MAX_CHANNELS) { - ipmi_inc_stat(intf, sent_invalid_commands); - rv = -EINVAL; - goto out_err; - } + if (chans[addr->channel].medium != IPMI_CHANNEL_MEDIUM_IPMB) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EINVAL; + } - chans = READ_ONCE(intf->channel_list)->c; + if (addr->addr_type == IPMI_IPMB_BROADCAST_ADDR_TYPE) { + /* + * Broadcasts add a zero at the beginning of the + * message, but otherwise is the same as an IPMB + * address. + */ + addr->addr_type = IPMI_IPMB_ADDR_TYPE; + broadcast = 1; + retries = 0; /* Don't retry broadcasts. */ + } - if (chans[addr->channel].medium != IPMI_CHANNEL_MEDIUM_IPMB) { - ipmi_inc_stat(intf, sent_invalid_commands); - rv = -EINVAL; - goto out_err; - } + /* + * 9 for the header and 1 for the checksum, plus + * possibly one for the broadcast. + */ + if ((msg->data_len + 10 + broadcast) > IPMI_MAX_MSG_LENGTH) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EMSGSIZE; + } - if (retries < 0) { - if (addr->addr_type == IPMI_IPMB_BROADCAST_ADDR_TYPE) - retries = 0; /* Don't retry broadcasts. */ - else - retries = 4; - } - if (addr->addr_type == IPMI_IPMB_BROADCAST_ADDR_TYPE) { - /* - * Broadcasts add a zero at the beginning of the - * message, but otherwise is the same as an IPMB - * address. - */ - addr->addr_type = IPMI_IPMB_ADDR_TYPE; - broadcast = 1; - } + ipmb_addr = (struct ipmi_ipmb_addr *) addr; + if (ipmb_addr->lun > 3) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EINVAL; + } + memcpy(&recv_msg->addr, ipmb_addr, sizeof(*ipmb_addr)); - /* Default to 1 second retries. */ - if (retry_time_ms == 0) - retry_time_ms = 1000; + if (recv_msg->msg.netfn & 0x1) { + /* + * It's a response, so use the user's sequence + * from msgid. + */ + ipmi_inc_stat(intf, sent_ipmb_responses); + format_ipmb_msg(smi_msg, msg, ipmb_addr, msgid, + msgid, broadcast, + source_address, source_lun); /* - * 9 for the header and 1 for the checksum, plus - * possibly one for the broadcast. + * Save the receive message so we can use it + * to deliver the response. */ - if ((msg->data_len + 10 + broadcast) > IPMI_MAX_MSG_LENGTH) { - ipmi_inc_stat(intf, sent_invalid_commands); - rv = -EMSGSIZE; - goto out_err; - } + smi_msg->user_data = recv_msg; + } else { + /* It's a command, so get a sequence for it. */ + unsigned long flags; - ipmb_addr = (struct ipmi_ipmb_addr *) addr; - if (ipmb_addr->lun > 3) { - ipmi_inc_stat(intf, sent_invalid_commands); - rv = -EINVAL; - goto out_err; - } + spin_lock_irqsave(&intf->seq_lock, flags); - memcpy(&recv_msg->addr, ipmb_addr, sizeof(*ipmb_addr)); + if (is_maintenance_mode_cmd(msg)) + intf->ipmb_maintenance_mode_timeout = + maintenance_mode_timeout_ms; - if (recv_msg->msg.netfn & 0x1) { - /* - * It's a response, so use the user's sequence - * from msgid. - */ - ipmi_inc_stat(intf, sent_ipmb_responses); - format_ipmb_msg(smi_msg, msg, ipmb_addr, msgid, - msgid, broadcast, - source_address, source_lun); + if (intf->ipmb_maintenance_mode_timeout && retry_time_ms == 0) + /* Different default in maintenance mode */ + retry_time_ms = default_maintenance_retry_ms; + /* + * Create a sequence number with a 1 second + * timeout and 4 retries. + */ + rv = intf_next_seq(intf, + recv_msg, + retry_time_ms, + retries, + broadcast, + &ipmb_seq, + &seqid); + if (rv) /* - * Save the receive message so we can use it - * to deliver the response. + * We have used up all the sequence numbers, + * probably, so abort. */ - smi_msg->user_data = recv_msg; - } else { - /* It's a command, so get a sequence for it. */ + goto out_err; - spin_lock_irqsave(&(intf->seq_lock), flags); + ipmi_inc_stat(intf, sent_ipmb_commands); - /* - * Create a sequence number with a 1 second - * timeout and 4 retries. - */ - rv = intf_next_seq(intf, - recv_msg, - retry_time_ms, - retries, - broadcast, - &ipmb_seq, - &seqid); - if (rv) { - /* - * We have used up all the sequence numbers, - * probably, so abort. - */ - spin_unlock_irqrestore(&(intf->seq_lock), - flags); - goto out_err; - } + /* + * Store the sequence number in the message, + * so that when the send message response + * comes back we can start the timer. + */ + format_ipmb_msg(smi_msg, msg, ipmb_addr, + STORE_SEQ_IN_MSGID(ipmb_seq, seqid), + ipmb_seq, broadcast, + source_address, source_lun); - ipmi_inc_stat(intf, sent_ipmb_commands); + /* + * Copy the message into the recv message data, so we + * can retransmit it later if necessary. + */ + memcpy(recv_msg->msg_data, smi_msg->data, + smi_msg->data_size); + recv_msg->msg.data = recv_msg->msg_data; + recv_msg->msg.data_len = smi_msg->data_size; - /* - * Store the sequence number in the message, - * so that when the send message response - * comes back we can start the timer. - */ - format_ipmb_msg(smi_msg, msg, ipmb_addr, - STORE_SEQ_IN_MSGID(ipmb_seq, seqid), - ipmb_seq, broadcast, - source_address, source_lun); + /* + * We don't unlock until here, because we need + * to copy the completed message into the + * recv_msg before we release the lock. + * Otherwise, race conditions may bite us. I + * know that's pretty paranoid, but I prefer + * to be correct. + */ +out_err: + spin_unlock_irqrestore(&intf->seq_lock, flags); + } - /* - * Copy the message into the recv message data, so we - * can retransmit it later if necessary. - */ - memcpy(recv_msg->msg_data, smi_msg->data, - smi_msg->data_size); - recv_msg->msg.data = recv_msg->msg_data; - recv_msg->msg.data_len = smi_msg->data_size; + return rv; +} - /* - * We don't unlock until here, because we need - * to copy the completed message into the - * recv_msg before we release the lock. - * Otherwise, race conditions may bite us. I - * know that's pretty paranoid, but I prefer - * to be correct. - */ - spin_unlock_irqrestore(&(intf->seq_lock), flags); - } - } else if (is_lan_addr(addr)) { - struct ipmi_lan_addr *lan_addr; - unsigned char ipmb_seq; - long seqid; - struct ipmi_channel *chans; +static int i_ipmi_req_lan(struct ipmi_smi *intf, + struct ipmi_addr *addr, + long msgid, + struct kernel_ipmi_msg *msg, + struct ipmi_smi_msg *smi_msg, + struct ipmi_recv_msg *recv_msg, + unsigned char source_lun, + int retries, + unsigned int retry_time_ms) +{ + struct ipmi_lan_addr *lan_addr; + unsigned char ipmb_seq; + long seqid; + struct ipmi_channel *chans; + int rv = 0; - if (addr->channel >= IPMI_MAX_CHANNELS) { - ipmi_inc_stat(intf, sent_invalid_commands); - rv = -EINVAL; - goto out_err; - } + if (addr->channel >= IPMI_MAX_CHANNELS) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EINVAL; + } - chans = READ_ONCE(intf->channel_list)->c; + chans = READ_ONCE(intf->channel_list)->c; - if ((chans[addr->channel].medium + if ((chans[addr->channel].medium != IPMI_CHANNEL_MEDIUM_8023LAN) - && (chans[addr->channel].medium - != IPMI_CHANNEL_MEDIUM_ASYNC)) { - ipmi_inc_stat(intf, sent_invalid_commands); - rv = -EINVAL; - goto out_err; - } + && (chans[addr->channel].medium + != IPMI_CHANNEL_MEDIUM_ASYNC)) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EINVAL; + } - retries = 4; + /* 11 for the header and 1 for the checksum. */ + if ((msg->data_len + 12) > IPMI_MAX_MSG_LENGTH) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EMSGSIZE; + } - /* Default to 1 second retries. */ - if (retry_time_ms == 0) - retry_time_ms = 1000; + lan_addr = (struct ipmi_lan_addr *) addr; + if (lan_addr->lun > 3) { + ipmi_inc_stat(intf, sent_invalid_commands); + return -EINVAL; + } - /* 11 for the header and 1 for the checksum. */ - if ((msg->data_len + 12) > IPMI_MAX_MSG_LENGTH) { - ipmi_inc_stat(intf, sent_invalid_commands); - rv = -EMSGSIZE; - goto out_err; - } + memcpy(&recv_msg->addr, lan_addr, sizeof(*lan_addr)); - lan_addr = (struct ipmi_lan_addr *) addr; - if (lan_addr->lun > 3) { - ipmi_inc_stat(intf, sent_invalid_commands); - rv = -EINVAL; - goto out_err; - } + if (recv_msg->msg.netfn & 0x1) { + /* + * It's a response, so use the user's sequence + * from msgid. + */ + ipmi_inc_stat(intf, sent_lan_responses); + format_lan_msg(smi_msg, msg, lan_addr, msgid, + msgid, source_lun); - memcpy(&recv_msg->addr, lan_addr, sizeof(*lan_addr)); + /* + * Save the receive message so we can use it + * to deliver the response. + */ + smi_msg->user_data = recv_msg; + } else { + /* It's a command, so get a sequence for it. */ + unsigned long flags; - if (recv_msg->msg.netfn & 0x1) { - /* - * It's a response, so use the user's sequence - * from msgid. - */ - ipmi_inc_stat(intf, sent_lan_responses); - format_lan_msg(smi_msg, msg, lan_addr, msgid, - msgid, source_lun); + spin_lock_irqsave(&intf->seq_lock, flags); + /* + * Create a sequence number with a 1 second + * timeout and 4 retries. + */ + rv = intf_next_seq(intf, + recv_msg, + retry_time_ms, + retries, + 0, + &ipmb_seq, + &seqid); + if (rv) /* - * Save the receive message so we can use it - * to deliver the response. + * We have used up all the sequence numbers, + * probably, so abort. */ - smi_msg->user_data = recv_msg; - } else { - /* It's a command, so get a sequence for it. */ + goto out_err; - spin_lock_irqsave(&(intf->seq_lock), flags); + ipmi_inc_stat(intf, sent_lan_commands); - /* - * Create a sequence number with a 1 second - * timeout and 4 retries. - */ - rv = intf_next_seq(intf, - recv_msg, - retry_time_ms, - retries, - 0, - &ipmb_seq, - &seqid); - if (rv) { - /* - * We have used up all the sequence numbers, - * probably, so abort. - */ - spin_unlock_irqrestore(&(intf->seq_lock), - flags); - goto out_err; - } + /* + * Store the sequence number in the message, + * so that when the send message response + * comes back we can start the timer. + */ + format_lan_msg(smi_msg, msg, lan_addr, + STORE_SEQ_IN_MSGID(ipmb_seq, seqid), + ipmb_seq, source_lun); + + /* + * Copy the message into the recv message data, so we + * can retransmit it later if necessary. + */ + memcpy(recv_msg->msg_data, smi_msg->data, + smi_msg->data_size); + recv_msg->msg.data = recv_msg->msg_data; + recv_msg->msg.data_len = smi_msg->data_size; + + /* + * We don't unlock until here, because we need + * to copy the completed message into the + * recv_msg before we release the lock. + * Otherwise, race conditions may bite us. I + * know that's pretty paranoid, but I prefer + * to be correct. + */ +out_err: + spin_unlock_irqrestore(&intf->seq_lock, flags); + } - ipmi_inc_stat(intf, sent_lan_commands); + return rv; +} + +/* + * Separate from ipmi_request so that the user does not have to be + * supplied in certain circumstances (mainly at panic time). If + * messages are supplied, they will be freed, even if an error + * occurs. + */ +static int i_ipmi_request(struct ipmi_user *user, + struct ipmi_smi *intf, + struct ipmi_addr *addr, + long msgid, + struct kernel_ipmi_msg *msg, + void *user_msg_data, + void *supplied_smi, + struct ipmi_recv_msg *supplied_recv, + int priority, + unsigned char source_address, + unsigned char source_lun, + int retries, + unsigned int retry_time_ms) +{ + struct ipmi_smi_msg *smi_msg; + struct ipmi_recv_msg *recv_msg; + int rv = 0; + + if (supplied_recv) + recv_msg = supplied_recv; + else { + recv_msg = ipmi_alloc_recv_msg(); + if (recv_msg == NULL) { + rv = -ENOMEM; + goto out; + } + } + recv_msg->user_msg_data = user_msg_data; + + if (supplied_smi) + smi_msg = (struct ipmi_smi_msg *) supplied_smi; + else { + smi_msg = ipmi_alloc_smi_msg(); + if (smi_msg == NULL) { + ipmi_free_recv_msg(recv_msg); + rv = -ENOMEM; + goto out; + } + } - /* - * Store the sequence number in the message, - * so that when the send message response - * comes back we can start the timer. - */ - format_lan_msg(smi_msg, msg, lan_addr, - STORE_SEQ_IN_MSGID(ipmb_seq, seqid), - ipmb_seq, source_lun); + rcu_read_lock(); + if (intf->in_shutdown) { + rv = -ENODEV; + goto out_err; + } - /* - * Copy the message into the recv message data, so we - * can retransmit it later if necessary. - */ - memcpy(recv_msg->msg_data, smi_msg->data, - smi_msg->data_size); - recv_msg->msg.data = recv_msg->msg_data; - recv_msg->msg.data_len = smi_msg->data_size; + recv_msg->user = user; + if (user) + /* The put happens when the message is freed. */ + kref_get(&user->refcount); + recv_msg->msgid = msgid; + /* + * Store the message to send in the receive message so timeout + * responses can get the proper response data. + */ + recv_msg->msg = *msg; - /* - * We don't unlock until here, because we need - * to copy the completed message into the - * recv_msg before we release the lock. - * Otherwise, race conditions may bite us. I - * know that's pretty paranoid, but I prefer - * to be correct. - */ - spin_unlock_irqrestore(&(intf->seq_lock), flags); - } + if (addr->addr_type == IPMI_SYSTEM_INTERFACE_ADDR_TYPE) { + rv = i_ipmi_req_sysintf(intf, addr, msgid, msg, smi_msg, + recv_msg, retries, retry_time_ms); + } else if (is_ipmb_addr(addr) || is_ipmb_bcast_addr(addr)) { + rv = i_ipmi_req_ipmb(intf, addr, msgid, msg, smi_msg, recv_msg, + source_address, source_lun, + retries, retry_time_ms); + } else if (is_lan_addr(addr)) { + rv = i_ipmi_req_lan(intf, addr, msgid, msg, smi_msg, recv_msg, + source_lun, retries, retry_time_ms); } else { /* Unknown address type. */ ipmi_inc_stat(intf, sent_invalid_commands); rv = -EINVAL; - goto out_err; } -#ifdef DEBUG_MSGING - { - int m; - for (m = 0; m < smi_msg->data_size; m++) - printk(" %2.2x", smi_msg->data[m]); - printk("\n"); - } -#endif + if (rv) { +out_err: + ipmi_free_smi_msg(smi_msg); + ipmi_free_recv_msg(recv_msg); + } else { + ipmi_debug_msg("Send", smi_msg->data, smi_msg->data_size); - smi_send(intf, intf->handlers, smi_msg, priority); + smi_send(intf, intf->handlers, smi_msg, priority); + } rcu_read_unlock(); - return 0; - - out_err: - rcu_read_unlock(); - ipmi_free_smi_msg(smi_msg); - ipmi_free_recv_msg(recv_msg); +out: return rv; } -static int check_addr(ipmi_smi_t intf, +static int check_addr(struct ipmi_smi *intf, struct ipmi_addr *addr, unsigned char *saddr, unsigned char *lun) @@ -2046,7 +2189,7 @@ static int check_addr(ipmi_smi_t intf, return 0; } -int ipmi_request_settime(ipmi_user_t user, +int ipmi_request_settime(struct ipmi_user *user, struct ipmi_addr *addr, long msgid, struct kernel_ipmi_msg *msg, @@ -2056,29 +2199,36 @@ int ipmi_request_settime(ipmi_user_t user, unsigned int retry_time_ms) { unsigned char saddr = 0, lun = 0; - int rv; + int rv, index; if (!user) return -EINVAL; + + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; + rv = check_addr(user->intf, addr, &saddr, &lun); - if (rv) - return rv; - return i_ipmi_request(user, - user->intf, - addr, - msgid, - msg, - user_msg_data, - NULL, NULL, - priority, - saddr, - lun, - retries, - retry_time_ms); + if (!rv) + rv = i_ipmi_request(user, + user->intf, + addr, + msgid, + msg, + user_msg_data, + NULL, NULL, + priority, + saddr, + lun, + retries, + retry_time_ms); + + release_ipmi_user(user, index); + return rv; } EXPORT_SYMBOL(ipmi_request_settime); -int ipmi_request_supply_msgs(ipmi_user_t user, +int ipmi_request_supply_msgs(struct ipmi_user *user, struct ipmi_addr *addr, long msgid, struct kernel_ipmi_msg *msg, @@ -2088,29 +2238,37 @@ int ipmi_request_supply_msgs(ipmi_user_t user, int priority) { unsigned char saddr = 0, lun = 0; - int rv; + int rv, index; if (!user) return -EINVAL; + + user = acquire_ipmi_user(user, &index); + if (!user) + return -ENODEV; + rv = check_addr(user->intf, addr, &saddr, &lun); - if (rv) - return rv; - return i_ipmi_request(user, - user->intf, - addr, - msgid, - msg, - user_msg_data, - supplied_smi, - supplied_recv, - priority, - saddr, - lun, - -1, 0); + if (!rv) + rv = i_ipmi_request(user, + user->intf, + addr, + msgid, + msg, + user_msg_data, + supplied_smi, + supplied_recv, + priority, + saddr, + lun, + -1, 0); + + release_ipmi_user(user, index); + return rv; } EXPORT_SYMBOL(ipmi_request_supply_msgs); -static void bmc_device_id_handler(ipmi_smi_t intf, struct ipmi_recv_msg *msg) +static void bmc_device_id_handler(struct ipmi_smi *intf, + struct ipmi_recv_msg *msg) { int rv; @@ -2142,7 +2300,7 @@ static void bmc_device_id_handler(ipmi_smi_t intf, struct ipmi_recv_msg *msg) } static int -send_get_device_id_cmd(ipmi_smi_t intf) +send_get_device_id_cmd(struct ipmi_smi *intf) { struct ipmi_system_interface_addr si; struct kernel_ipmi_msg msg; @@ -2170,7 +2328,7 @@ send_get_device_id_cmd(ipmi_smi_t intf) -1, 0); } -static int __get_device_id(ipmi_smi_t intf, struct bmc_device *bmc) +static int __get_device_id(struct ipmi_smi *intf, struct bmc_device *bmc) { int rv; @@ -2204,7 +2362,7 @@ static int __get_device_id(ipmi_smi_t intf, struct bmc_device *bmc) * Except for the first time this is called (in ipmi_register_smi()), * this will always return good data; */ -static int __bmc_get_device_id(ipmi_smi_t intf, struct bmc_device *bmc, +static int __bmc_get_device_id(struct ipmi_smi *intf, struct bmc_device *bmc, struct ipmi_device_id *id, bool *guid_set, guid_t *guid, int intf_num) { @@ -2337,223 +2495,13 @@ out_noprocessing: return rv; } -static int bmc_get_device_id(ipmi_smi_t intf, struct bmc_device *bmc, +static int bmc_get_device_id(struct ipmi_smi *intf, struct bmc_device *bmc, struct ipmi_device_id *id, bool *guid_set, guid_t *guid) { return __bmc_get_device_id(intf, bmc, id, guid_set, guid, -1); } -#ifdef CONFIG_IPMI_PROC_INTERFACE -static int smi_ipmb_proc_show(struct seq_file *m, void *v) -{ - ipmi_smi_t intf = m->private; - int i; - - seq_printf(m, "%x", intf->addrinfo[0].address); - for (i = 1; i < IPMI_MAX_CHANNELS; i++) - seq_printf(m, " %x", intf->addrinfo[i].address); - seq_putc(m, '\n'); - - return 0; -} - -static int smi_ipmb_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, smi_ipmb_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations smi_ipmb_proc_ops = { - .open = smi_ipmb_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - -static int smi_version_proc_show(struct seq_file *m, void *v) -{ - ipmi_smi_t intf = m->private; - struct ipmi_device_id id; - int rv; - - rv = bmc_get_device_id(intf, NULL, &id, NULL, NULL); - if (rv) - return rv; - - seq_printf(m, "%u.%u\n", - ipmi_version_major(&id), - ipmi_version_minor(&id)); - - return 0; -} - -static int smi_version_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, smi_version_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations smi_version_proc_ops = { - .open = smi_version_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - -static int smi_stats_proc_show(struct seq_file *m, void *v) -{ - ipmi_smi_t intf = m->private; - - seq_printf(m, "sent_invalid_commands: %u\n", - ipmi_get_stat(intf, sent_invalid_commands)); - seq_printf(m, "sent_local_commands: %u\n", - ipmi_get_stat(intf, sent_local_commands)); - seq_printf(m, "handled_local_responses: %u\n", - ipmi_get_stat(intf, handled_local_responses)); - seq_printf(m, "unhandled_local_responses: %u\n", - ipmi_get_stat(intf, unhandled_local_responses)); - seq_printf(m, "sent_ipmb_commands: %u\n", - ipmi_get_stat(intf, sent_ipmb_commands)); - seq_printf(m, "sent_ipmb_command_errs: %u\n", - ipmi_get_stat(intf, sent_ipmb_command_errs)); - seq_printf(m, "retransmitted_ipmb_commands: %u\n", - ipmi_get_stat(intf, retransmitted_ipmb_commands)); - seq_printf(m, "timed_out_ipmb_commands: %u\n", - ipmi_get_stat(intf, timed_out_ipmb_commands)); - seq_printf(m, "timed_out_ipmb_broadcasts: %u\n", - ipmi_get_stat(intf, timed_out_ipmb_broadcasts)); - seq_printf(m, "sent_ipmb_responses: %u\n", - ipmi_get_stat(intf, sent_ipmb_responses)); - seq_printf(m, "handled_ipmb_responses: %u\n", - ipmi_get_stat(intf, handled_ipmb_responses)); - seq_printf(m, "invalid_ipmb_responses: %u\n", - ipmi_get_stat(intf, invalid_ipmb_responses)); - seq_printf(m, "unhandled_ipmb_responses: %u\n", - ipmi_get_stat(intf, unhandled_ipmb_responses)); - seq_printf(m, "sent_lan_commands: %u\n", - ipmi_get_stat(intf, sent_lan_commands)); - seq_printf(m, "sent_lan_command_errs: %u\n", - ipmi_get_stat(intf, sent_lan_command_errs)); - seq_printf(m, "retransmitted_lan_commands: %u\n", - ipmi_get_stat(intf, retransmitted_lan_commands)); - seq_printf(m, "timed_out_lan_commands: %u\n", - ipmi_get_stat(intf, timed_out_lan_commands)); - seq_printf(m, "sent_lan_responses: %u\n", - ipmi_get_stat(intf, sent_lan_responses)); - seq_printf(m, "handled_lan_responses: %u\n", - ipmi_get_stat(intf, handled_lan_responses)); - seq_printf(m, "invalid_lan_responses: %u\n", - ipmi_get_stat(intf, invalid_lan_responses)); - seq_printf(m, "unhandled_lan_responses: %u\n", - ipmi_get_stat(intf, unhandled_lan_responses)); - seq_printf(m, "handled_commands: %u\n", - ipmi_get_stat(intf, handled_commands)); - seq_printf(m, "invalid_commands: %u\n", - ipmi_get_stat(intf, invalid_commands)); - seq_printf(m, "unhandled_commands: %u\n", - ipmi_get_stat(intf, unhandled_commands)); - seq_printf(m, "invalid_events: %u\n", - ipmi_get_stat(intf, invalid_events)); - seq_printf(m, "events: %u\n", - ipmi_get_stat(intf, events)); - seq_printf(m, "failed rexmit LAN msgs: %u\n", - ipmi_get_stat(intf, dropped_rexmit_lan_commands)); - seq_printf(m, "failed rexmit IPMB msgs: %u\n", - ipmi_get_stat(intf, dropped_rexmit_ipmb_commands)); - return 0; -} - -static int smi_stats_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, smi_stats_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations smi_stats_proc_ops = { - .open = smi_stats_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - -int ipmi_smi_add_proc_entry(ipmi_smi_t smi, char *name, - const struct file_operations *proc_ops, - void *data) -{ - int rv = 0; - struct proc_dir_entry *file; - struct ipmi_proc_entry *entry; - - /* Create a list element. */ - entry = kmalloc(sizeof(*entry), GFP_KERNEL); - if (!entry) - return -ENOMEM; - entry->name = kstrdup(name, GFP_KERNEL); - if (!entry->name) { - kfree(entry); - return -ENOMEM; - } - - file = proc_create_data(name, 0, smi->proc_dir, proc_ops, data); - if (!file) { - kfree(entry->name); - kfree(entry); - rv = -ENOMEM; - } else { - mutex_lock(&smi->proc_entry_lock); - /* Stick it on the list. */ - entry->next = smi->proc_entries; - smi->proc_entries = entry; - mutex_unlock(&smi->proc_entry_lock); - } - - return rv; -} -EXPORT_SYMBOL(ipmi_smi_add_proc_entry); - -static int add_proc_entries(ipmi_smi_t smi, int num) -{ - int rv = 0; - - sprintf(smi->proc_dir_name, "%d", num); - smi->proc_dir = proc_mkdir(smi->proc_dir_name, proc_ipmi_root); - if (!smi->proc_dir) - rv = -ENOMEM; - - if (rv == 0) - rv = ipmi_smi_add_proc_entry(smi, "stats", - &smi_stats_proc_ops, - smi); - - if (rv == 0) - rv = ipmi_smi_add_proc_entry(smi, "ipmb", - &smi_ipmb_proc_ops, - smi); - - if (rv == 0) - rv = ipmi_smi_add_proc_entry(smi, "version", - &smi_version_proc_ops, - smi); - - return rv; -} - -static void remove_proc_entries(ipmi_smi_t smi) -{ - struct ipmi_proc_entry *entry; - - mutex_lock(&smi->proc_entry_lock); - while (smi->proc_entries) { - entry = smi->proc_entries; - smi->proc_entries = entry->next; - - remove_proc_entry(entry->name, smi->proc_dir); - kfree(entry->name); - kfree(entry); - } - mutex_unlock(&smi->proc_entry_lock); - remove_proc_entry(smi->proc_dir_name, proc_ipmi_root); -} -#endif /* CONFIG_IPMI_PROC_INTERFACE */ - static ssize_t device_id_show(struct device *dev, struct device_attribute *attr, char *buf) @@ -2885,7 +2833,7 @@ cleanup_bmc_device(struct kref *ref) /* * Must be called with intf->bmc_reg_mutex held. */ -static void __ipmi_bmc_unregister(ipmi_smi_t intf) +static void __ipmi_bmc_unregister(struct ipmi_smi *intf) { struct bmc_device *bmc = intf->bmc; @@ -2905,7 +2853,7 @@ static void __ipmi_bmc_unregister(ipmi_smi_t intf) intf->bmc_registered = false; } -static void ipmi_bmc_unregister(ipmi_smi_t intf) +static void ipmi_bmc_unregister(struct ipmi_smi *intf) { mutex_lock(&intf->bmc_reg_mutex); __ipmi_bmc_unregister(intf); @@ -2915,7 +2863,7 @@ static void ipmi_bmc_unregister(ipmi_smi_t intf) /* * Must be called with intf->bmc_reg_mutex held. */ -static int __ipmi_bmc_register(ipmi_smi_t intf, +static int __ipmi_bmc_register(struct ipmi_smi *intf, struct ipmi_device_id *id, bool guid_set, guid_t *guid, int intf_num) { @@ -3077,7 +3025,7 @@ out_list_del: } static int -send_guid_cmd(ipmi_smi_t intf, int chan) +send_guid_cmd(struct ipmi_smi *intf, int chan) { struct kernel_ipmi_msg msg; struct ipmi_system_interface_addr si; @@ -3104,7 +3052,7 @@ send_guid_cmd(ipmi_smi_t intf, int chan) -1, 0); } -static void guid_handler(ipmi_smi_t intf, struct ipmi_recv_msg *msg) +static void guid_handler(struct ipmi_smi *intf, struct ipmi_recv_msg *msg) { struct bmc_device *bmc = intf->bmc; @@ -3139,7 +3087,7 @@ static void guid_handler(ipmi_smi_t intf, struct ipmi_recv_msg *msg) wake_up(&intf->waitq); } -static void __get_guid(ipmi_smi_t intf) +static void __get_guid(struct ipmi_smi *intf) { int rv; struct bmc_device *bmc = intf->bmc; @@ -3160,7 +3108,7 @@ static void __get_guid(ipmi_smi_t intf) } static int -send_channel_info_cmd(ipmi_smi_t intf, int chan) +send_channel_info_cmd(struct ipmi_smi *intf, int chan) { struct kernel_ipmi_msg msg; unsigned char data[1]; @@ -3190,7 +3138,7 @@ send_channel_info_cmd(ipmi_smi_t intf, int chan) } static void -channel_handler(ipmi_smi_t intf, struct ipmi_recv_msg *msg) +channel_handler(struct ipmi_smi *intf, struct ipmi_recv_msg *msg) { int rv = 0; int ch; @@ -3262,7 +3210,7 @@ channel_handler(ipmi_smi_t intf, struct ipmi_recv_msg *msg) /* * Must be holding intf->bmc_reg_mutex to call this. */ -static int __scan_channels(ipmi_smi_t intf, struct ipmi_device_id *id) +static int __scan_channels(struct ipmi_smi *intf, struct ipmi_device_id *id) { int rv; @@ -3306,7 +3254,7 @@ static int __scan_channels(ipmi_smi_t intf, struct ipmi_device_id *id) return 0; } -static void ipmi_poll(ipmi_smi_t intf) +static void ipmi_poll(struct ipmi_smi *intf) { if (intf->handlers->poll) intf->handlers->poll(intf->send_info); @@ -3314,7 +3262,7 @@ static void ipmi_poll(ipmi_smi_t intf) handle_new_recv_msgs(intf); } -void ipmi_poll_interface(ipmi_user_t user) +void ipmi_poll_interface(struct ipmi_user *user) { ipmi_poll(user->intf); } @@ -3322,7 +3270,8 @@ EXPORT_SYMBOL(ipmi_poll_interface); static void redo_bmc_reg(struct work_struct *work) { - ipmi_smi_t intf = container_of(work, struct ipmi_smi, bmc_reg_work); + struct ipmi_smi *intf = container_of(work, struct ipmi_smi, + bmc_reg_work); if (!intf->in_shutdown) bmc_get_device_id(intf, NULL, NULL, NULL, NULL); @@ -3337,8 +3286,7 @@ int ipmi_register_smi(const struct ipmi_smi_handlers *handlers, { int i, j; int rv; - ipmi_smi_t intf; - ipmi_smi_t tintf; + struct ipmi_smi *intf, *tintf; struct list_head *link; struct ipmi_device_id id; @@ -3362,6 +3310,13 @@ int ipmi_register_smi(const struct ipmi_smi_handlers *handlers, if (!intf) return -ENOMEM; + rv = init_srcu_struct(&intf->users_srcu); + if (rv) { + kfree(intf); + return rv; + } + + intf->bmc = &intf->tmp_bmc; INIT_LIST_HEAD(&intf->bmc->intfs); mutex_init(&intf->bmc->dyn_mutex); @@ -3386,9 +3341,6 @@ int ipmi_register_smi(const struct ipmi_smi_handlers *handlers, intf->seq_table[j].seqid = 0; } intf->curr_seq = 0; -#ifdef CONFIG_IPMI_PROC_INTERFACE - mutex_init(&intf->proc_entry_lock); -#endif spin_lock_init(&intf->waiting_rcv_msgs_lock); INIT_LIST_HEAD(&intf->waiting_rcv_msgs); tasklet_init(&intf->recv_tasklet, @@ -3410,11 +3362,6 @@ int ipmi_register_smi(const struct ipmi_smi_handlers *handlers, for (i = 0; i < IPMI_NUM_STATS; i++) atomic_set(&intf->stats[i], 0); -#ifdef CONFIG_IPMI_PROC_INTERFACE - intf->proc_dir = NULL; -#endif - - mutex_lock(&smi_watchers_mutex); mutex_lock(&ipmi_interfaces_mutex); /* Look for a hole in the numbers. */ i = 0; @@ -3445,25 +3392,14 @@ int ipmi_register_smi(const struct ipmi_smi_handlers *handlers, mutex_lock(&intf->bmc_reg_mutex); rv = __scan_channels(intf, &id); mutex_unlock(&intf->bmc_reg_mutex); - if (rv) - goto out; - -#ifdef CONFIG_IPMI_PROC_INTERFACE - rv = add_proc_entries(intf, i); -#endif out: if (rv) { ipmi_bmc_unregister(intf); -#ifdef CONFIG_IPMI_PROC_INTERFACE - if (intf->proc_dir) - remove_proc_entries(intf); -#endif - intf->handlers = NULL; list_del_rcu(&intf->link); mutex_unlock(&ipmi_interfaces_mutex); - mutex_unlock(&smi_watchers_mutex); - synchronize_rcu(); + synchronize_srcu(&ipmi_interfaces_srcu); + cleanup_srcu_struct(&intf->users_srcu); kref_put(&intf->refcount, intf_free); } else { /* @@ -3474,16 +3410,16 @@ int ipmi_register_smi(const struct ipmi_smi_handlers *handlers, smp_wmb(); intf->intf_num = i; mutex_unlock(&ipmi_interfaces_mutex); + /* After this point the interface is legal to use. */ call_smi_watchers(i, intf->si_dev); - mutex_unlock(&smi_watchers_mutex); } return rv; } EXPORT_SYMBOL(ipmi_register_smi); -static void deliver_smi_err_response(ipmi_smi_t intf, +static void deliver_smi_err_response(struct ipmi_smi *intf, struct ipmi_smi_msg *msg, unsigned char err) { @@ -3495,7 +3431,7 @@ static void deliver_smi_err_response(ipmi_smi_t intf, handle_one_recv_msg(intf, msg); } -static void cleanup_smi_msgs(ipmi_smi_t intf) +static void cleanup_smi_msgs(struct ipmi_smi *intf) { int i; struct seq_table *ent; @@ -3528,60 +3464,58 @@ static void cleanup_smi_msgs(ipmi_smi_t intf) } for (i = 0; i < IPMI_IPMB_NUM_SEQ; i++) { - ent = &(intf->seq_table[i]); + ent = &intf->seq_table[i]; if (!ent->inuse) continue; - deliver_err_response(ent->recv_msg, IPMI_ERR_UNSPECIFIED); + deliver_err_response(intf, ent->recv_msg, IPMI_ERR_UNSPECIFIED); } } -int ipmi_unregister_smi(ipmi_smi_t intf) +void ipmi_unregister_smi(struct ipmi_smi *intf) { struct ipmi_smi_watcher *w; - int intf_num = intf->intf_num; - ipmi_user_t user; + int intf_num = intf->intf_num, index; - mutex_lock(&smi_watchers_mutex); mutex_lock(&ipmi_interfaces_mutex); intf->intf_num = -1; intf->in_shutdown = true; list_del_rcu(&intf->link); mutex_unlock(&ipmi_interfaces_mutex); - synchronize_rcu(); - - cleanup_smi_msgs(intf); - - /* Clean up the effects of users on the lower-level software. */ - mutex_lock(&ipmi_interfaces_mutex); - rcu_read_lock(); - list_for_each_entry_rcu(user, &intf->users, link) { - module_put(intf->handlers->owner); - if (intf->handlers->dec_usecount) - intf->handlers->dec_usecount(intf->send_info); - } - rcu_read_unlock(); - intf->handlers = NULL; - mutex_unlock(&ipmi_interfaces_mutex); + synchronize_srcu(&ipmi_interfaces_srcu); -#ifdef CONFIG_IPMI_PROC_INTERFACE - remove_proc_entries(intf); -#endif - ipmi_bmc_unregister(intf); + /* At this point no users can be added to the interface. */ /* * Call all the watcher interfaces to tell them that - * an interface is gone. + * an interface is going away. */ + mutex_lock(&smi_watchers_mutex); list_for_each_entry(w, &smi_watchers, link) w->smi_gone(intf_num); mutex_unlock(&smi_watchers_mutex); + index = srcu_read_lock(&intf->users_srcu); + while (!list_empty(&intf->users)) { + struct ipmi_user *user = + container_of(list_next_rcu(&intf->users), + struct ipmi_user, link); + + _ipmi_destroy_user(user); + } + srcu_read_unlock(&intf->users_srcu, index); + + intf->handlers->shutdown(intf->send_info); + + cleanup_smi_msgs(intf); + + ipmi_bmc_unregister(intf); + + cleanup_srcu_struct(&intf->users_srcu); kref_put(&intf->refcount, intf_free); - return 0; } EXPORT_SYMBOL(ipmi_unregister_smi); -static int handle_ipmb_get_msg_rsp(ipmi_smi_t intf, +static int handle_ipmb_get_msg_rsp(struct ipmi_smi *intf, struct ipmi_smi_msg *msg) { struct ipmi_ipmb_addr ipmb_addr; @@ -3616,7 +3550,7 @@ static int handle_ipmb_get_msg_rsp(ipmi_smi_t intf, msg->rsp[3] & 0x0f, msg->rsp[8], (msg->rsp[4] >> 2) & (~1), - (struct ipmi_addr *) &(ipmb_addr), + (struct ipmi_addr *) &ipmb_addr, &recv_msg)) { /* * We were unable to find the sequence number, @@ -3626,9 +3560,7 @@ static int handle_ipmb_get_msg_rsp(ipmi_smi_t intf, return 0; } - memcpy(recv_msg->msg_data, - &(msg->rsp[9]), - msg->rsp_size - 9); + memcpy(recv_msg->msg_data, &msg->rsp[9], msg->rsp_size - 9); /* * The other fields matched, so no need to set them, except * for netfn, which needs to be the response that was @@ -3638,13 +3570,15 @@ static int handle_ipmb_get_msg_rsp(ipmi_smi_t intf, recv_msg->msg.data = recv_msg->msg_data; recv_msg->msg.data_len = msg->rsp_size - 10; recv_msg->recv_type = IPMI_RESPONSE_RECV_TYPE; - ipmi_inc_stat(intf, handled_ipmb_responses); - deliver_response(recv_msg); + if (deliver_response(intf, recv_msg)) + ipmi_inc_stat(intf, unhandled_ipmb_responses); + else + ipmi_inc_stat(intf, handled_ipmb_responses); return 0; } -static int handle_ipmb_get_msg_cmd(ipmi_smi_t intf, +static int handle_ipmb_get_msg_cmd(struct ipmi_smi *intf, struct ipmi_smi_msg *msg) { struct cmd_rcvr *rcvr; @@ -3652,7 +3586,7 @@ static int handle_ipmb_get_msg_cmd(ipmi_smi_t intf, unsigned char netfn; unsigned char cmd; unsigned char chan; - ipmi_user_t user = NULL; + struct ipmi_user *user = NULL; struct ipmi_ipmb_addr *ipmb_addr; struct ipmi_recv_msg *recv_msg; @@ -3689,24 +3623,17 @@ static int handle_ipmb_get_msg_cmd(ipmi_smi_t intf, msg->data[2] = msg->rsp[3]; msg->data[3] = msg->rsp[6]; msg->data[4] = ((netfn + 1) << 2) | (msg->rsp[7] & 0x3); - msg->data[5] = ipmb_checksum(&(msg->data[3]), 2); + msg->data[5] = ipmb_checksum(&msg->data[3], 2); msg->data[6] = intf->addrinfo[msg->rsp[3] & 0xf].address; /* rqseq/lun */ msg->data[7] = (msg->rsp[7] & 0xfc) | (msg->rsp[4] & 0x3); msg->data[8] = msg->rsp[8]; /* cmd */ msg->data[9] = IPMI_INVALID_CMD_COMPLETION_CODE; - msg->data[10] = ipmb_checksum(&(msg->data[6]), 4); + msg->data[10] = ipmb_checksum(&msg->data[6], 4); msg->data_size = 11; -#ifdef DEBUG_MSGING - { - int m; - printk("Invalid command:"); - for (m = 0; m < msg->data_size; m++) - printk(" %2.2x", msg->data[m]); - printk("\n"); - } -#endif + ipmi_debug_msg("Invalid command:", msg->data, msg->data_size); + rcu_read_lock(); if (!intf->in_shutdown) { smi_send(intf, intf->handlers, msg, 0); @@ -3719,9 +3646,6 @@ static int handle_ipmb_get_msg_cmd(ipmi_smi_t intf, } rcu_read_unlock(); } else { - /* Deliver the message to the user. */ - ipmi_inc_stat(intf, handled_commands); - recv_msg = ipmi_alloc_recv_msg(); if (!recv_msg) { /* @@ -3755,17 +3679,19 @@ static int handle_ipmb_get_msg_cmd(ipmi_smi_t intf, * at the end also needs to be removed. */ recv_msg->msg.data_len = msg->rsp_size - 10; - memcpy(recv_msg->msg_data, - &(msg->rsp[9]), + memcpy(recv_msg->msg_data, &msg->rsp[9], msg->rsp_size - 10); - deliver_response(recv_msg); + if (deliver_response(intf, recv_msg)) + ipmi_inc_stat(intf, unhandled_commands); + else + ipmi_inc_stat(intf, handled_commands); } } return rv; } -static int handle_lan_get_msg_rsp(ipmi_smi_t intf, +static int handle_lan_get_msg_rsp(struct ipmi_smi *intf, struct ipmi_smi_msg *msg) { struct ipmi_lan_addr lan_addr; @@ -3804,7 +3730,7 @@ static int handle_lan_get_msg_rsp(ipmi_smi_t intf, msg->rsp[3] & 0x0f, msg->rsp[10], (msg->rsp[6] >> 2) & (~1), - (struct ipmi_addr *) &(lan_addr), + (struct ipmi_addr *) &lan_addr, &recv_msg)) { /* * We were unable to find the sequence number, @@ -3814,9 +3740,7 @@ static int handle_lan_get_msg_rsp(ipmi_smi_t intf, return 0; } - memcpy(recv_msg->msg_data, - &(msg->rsp[11]), - msg->rsp_size - 11); + memcpy(recv_msg->msg_data, &msg->rsp[11], msg->rsp_size - 11); /* * The other fields matched, so no need to set them, except * for netfn, which needs to be the response that was @@ -3826,13 +3750,15 @@ static int handle_lan_get_msg_rsp(ipmi_smi_t intf, recv_msg->msg.data = recv_msg->msg_data; recv_msg->msg.data_len = msg->rsp_size - 12; recv_msg->recv_type = IPMI_RESPONSE_RECV_TYPE; - ipmi_inc_stat(intf, handled_lan_responses); - deliver_response(recv_msg); + if (deliver_response(intf, recv_msg)) + ipmi_inc_stat(intf, unhandled_lan_responses); + else + ipmi_inc_stat(intf, handled_lan_responses); return 0; } -static int handle_lan_get_msg_cmd(ipmi_smi_t intf, +static int handle_lan_get_msg_cmd(struct ipmi_smi *intf, struct ipmi_smi_msg *msg) { struct cmd_rcvr *rcvr; @@ -3840,7 +3766,7 @@ static int handle_lan_get_msg_cmd(ipmi_smi_t intf, unsigned char netfn; unsigned char cmd; unsigned char chan; - ipmi_user_t user = NULL; + struct ipmi_user *user = NULL; struct ipmi_lan_addr *lan_addr; struct ipmi_recv_msg *recv_msg; @@ -3878,9 +3804,6 @@ static int handle_lan_get_msg_cmd(ipmi_smi_t intf, */ rv = 0; } else { - /* Deliver the message to the user. */ - ipmi_inc_stat(intf, handled_commands); - recv_msg = ipmi_alloc_recv_msg(); if (!recv_msg) { /* @@ -3916,10 +3839,12 @@ static int handle_lan_get_msg_cmd(ipmi_smi_t intf, * at the end also needs to be removed. */ recv_msg->msg.data_len = msg->rsp_size - 12; - memcpy(recv_msg->msg_data, - &(msg->rsp[11]), + memcpy(recv_msg->msg_data, &msg->rsp[11], msg->rsp_size - 12); - deliver_response(recv_msg); + if (deliver_response(intf, recv_msg)) + ipmi_inc_stat(intf, unhandled_commands); + else + ipmi_inc_stat(intf, handled_commands); } } @@ -3932,7 +3857,7 @@ static int handle_lan_get_msg_cmd(ipmi_smi_t intf, * the OEM. See IPMI 2.0 specification, Chapter 6 and * Chapter 22, sections 22.6 and 22.24 for more details. */ -static int handle_oem_get_msg_cmd(ipmi_smi_t intf, +static int handle_oem_get_msg_cmd(struct ipmi_smi *intf, struct ipmi_smi_msg *msg) { struct cmd_rcvr *rcvr; @@ -3940,7 +3865,7 @@ static int handle_oem_get_msg_cmd(ipmi_smi_t intf, unsigned char netfn; unsigned char cmd; unsigned char chan; - ipmi_user_t user = NULL; + struct ipmi_user *user = NULL; struct ipmi_system_interface_addr *smi_addr; struct ipmi_recv_msg *recv_msg; @@ -3987,9 +3912,6 @@ static int handle_oem_get_msg_cmd(ipmi_smi_t intf, rv = 0; } else { - /* Deliver the message to the user. */ - ipmi_inc_stat(intf, handled_commands); - recv_msg = ipmi_alloc_recv_msg(); if (!recv_msg) { /* @@ -4007,7 +3929,7 @@ static int handle_oem_get_msg_cmd(ipmi_smi_t intf, * requirements */ smi_addr = ((struct ipmi_system_interface_addr *) - &(recv_msg->addr)); + &recv_msg->addr); smi_addr->addr_type = IPMI_SYSTEM_INTERFACE_ADDR_TYPE; smi_addr->channel = IPMI_BMC_CHANNEL; smi_addr->lun = msg->rsp[0] & 3; @@ -4024,10 +3946,12 @@ static int handle_oem_get_msg_cmd(ipmi_smi_t intf, * the Channel Byte in the "GET MESSAGE" command */ recv_msg->msg.data_len = msg->rsp_size - 4; - memcpy(recv_msg->msg_data, - &(msg->rsp[4]), + memcpy(recv_msg->msg_data, &msg->rsp[4], msg->rsp_size - 4); - deliver_response(recv_msg); + if (deliver_response(intf, recv_msg)) + ipmi_inc_stat(intf, unhandled_commands); + else + ipmi_inc_stat(intf, handled_commands); } } @@ -4040,26 +3964,25 @@ static void copy_event_into_recv_msg(struct ipmi_recv_msg *recv_msg, struct ipmi_system_interface_addr *smi_addr; recv_msg->msgid = 0; - smi_addr = (struct ipmi_system_interface_addr *) &(recv_msg->addr); + smi_addr = (struct ipmi_system_interface_addr *) &recv_msg->addr; smi_addr->addr_type = IPMI_SYSTEM_INTERFACE_ADDR_TYPE; smi_addr->channel = IPMI_BMC_CHANNEL; smi_addr->lun = msg->rsp[0] & 3; recv_msg->recv_type = IPMI_ASYNC_EVENT_RECV_TYPE; recv_msg->msg.netfn = msg->rsp[0] >> 2; recv_msg->msg.cmd = msg->rsp[1]; - memcpy(recv_msg->msg_data, &(msg->rsp[3]), msg->rsp_size - 3); + memcpy(recv_msg->msg_data, &msg->rsp[3], msg->rsp_size - 3); recv_msg->msg.data = recv_msg->msg_data; recv_msg->msg.data_len = msg->rsp_size - 3; } -static int handle_read_event_rsp(ipmi_smi_t intf, +static int handle_read_event_rsp(struct ipmi_smi *intf, struct ipmi_smi_msg *msg) { struct ipmi_recv_msg *recv_msg, *recv_msg2; struct list_head msgs; - ipmi_user_t user; - int rv = 0; - int deliver_count = 0; + struct ipmi_user *user; + int rv = 0, deliver_count = 0, index; unsigned long flags; if (msg->rsp_size < 19) { @@ -4083,7 +4006,7 @@ static int handle_read_event_rsp(ipmi_smi_t intf, * Allocate and fill in one message for every user that is * getting events. */ - rcu_read_lock(); + index = srcu_read_lock(&intf->users_srcu); list_for_each_entry_rcu(user, &intf->users, link) { if (!user->gets_events) continue; @@ -4110,15 +4033,15 @@ static int handle_read_event_rsp(ipmi_smi_t intf, copy_event_into_recv_msg(recv_msg, msg); recv_msg->user = user; kref_get(&user->refcount); - list_add_tail(&(recv_msg->link), &msgs); + list_add_tail(&recv_msg->link, &msgs); } - rcu_read_unlock(); + srcu_read_unlock(&intf->users_srcu, index); if (deliver_count) { /* Now deliver all the messages. */ list_for_each_entry_safe(recv_msg, recv_msg2, &msgs, link) { list_del(&recv_msg->link); - deliver_response(recv_msg); + deliver_local_response(intf, recv_msg); } } else if (intf->waiting_events_count < MAX_EVENTS_IN_QUEUE) { /* @@ -4137,7 +4060,7 @@ static int handle_read_event_rsp(ipmi_smi_t intf, } copy_event_into_recv_msg(recv_msg, msg); - list_add_tail(&(recv_msg->link), &(intf->waiting_events)); + list_add_tail(&recv_msg->link, &intf->waiting_events); intf->waiting_events_count++; } else if (!intf->event_msg_printed) { /* @@ -4150,16 +4073,16 @@ static int handle_read_event_rsp(ipmi_smi_t intf, } out: - spin_unlock_irqrestore(&(intf->events_lock), flags); + spin_unlock_irqrestore(&intf->events_lock, flags); return rv; } -static int handle_bmc_rsp(ipmi_smi_t intf, +static int handle_bmc_rsp(struct ipmi_smi *intf, struct ipmi_smi_msg *msg) { struct ipmi_recv_msg *recv_msg; - struct ipmi_user *user; + struct ipmi_system_interface_addr *smi_addr; recv_msg = (struct ipmi_recv_msg *) msg->user_data; if (recv_msg == NULL) { @@ -4168,32 +4091,19 @@ static int handle_bmc_rsp(ipmi_smi_t intf, return 0; } - user = recv_msg->user; - /* Make sure the user still exists. */ - if (user && !user->valid) { - /* The user for the message went away, so give up. */ - ipmi_inc_stat(intf, unhandled_local_responses); - ipmi_free_recv_msg(recv_msg); - } else { - struct ipmi_system_interface_addr *smi_addr; - - ipmi_inc_stat(intf, handled_local_responses); - recv_msg->recv_type = IPMI_RESPONSE_RECV_TYPE; - recv_msg->msgid = msg->msgid; - smi_addr = ((struct ipmi_system_interface_addr *) - &(recv_msg->addr)); - smi_addr->addr_type = IPMI_SYSTEM_INTERFACE_ADDR_TYPE; - smi_addr->channel = IPMI_BMC_CHANNEL; - smi_addr->lun = msg->rsp[0] & 3; - recv_msg->msg.netfn = msg->rsp[0] >> 2; - recv_msg->msg.cmd = msg->rsp[1]; - memcpy(recv_msg->msg_data, - &(msg->rsp[2]), - msg->rsp_size - 2); - recv_msg->msg.data = recv_msg->msg_data; - recv_msg->msg.data_len = msg->rsp_size - 2; - deliver_response(recv_msg); - } + recv_msg->recv_type = IPMI_RESPONSE_RECV_TYPE; + recv_msg->msgid = msg->msgid; + smi_addr = ((struct ipmi_system_interface_addr *) + &recv_msg->addr); + smi_addr->addr_type = IPMI_SYSTEM_INTERFACE_ADDR_TYPE; + smi_addr->channel = IPMI_BMC_CHANNEL; + smi_addr->lun = msg->rsp[0] & 3; + recv_msg->msg.netfn = msg->rsp[0] >> 2; + recv_msg->msg.cmd = msg->rsp[1]; + memcpy(recv_msg->msg_data, &msg->rsp[2], msg->rsp_size - 2); + recv_msg->msg.data = recv_msg->msg_data; + recv_msg->msg.data_len = msg->rsp_size - 2; + deliver_local_response(intf, recv_msg); return 0; } @@ -4203,19 +4113,13 @@ static int handle_bmc_rsp(ipmi_smi_t intf, * 0 if the message should be freed, or -1 if the message should not * be freed or requeued. */ -static int handle_one_recv_msg(ipmi_smi_t intf, +static int handle_one_recv_msg(struct ipmi_smi *intf, struct ipmi_smi_msg *msg) { int requeue; int chan; -#ifdef DEBUG_MSGING - int m; - printk("Recv:"); - for (m = 0; m < msg->rsp_size; m++) - printk(" %2.2x", msg->rsp[m]); - printk("\n"); -#endif + ipmi_debug_msg("Recv:", msg->rsp, msg->rsp_size); if (msg->rsp_size < 2) { /* Message is too small to be correct. */ dev_warn(intf->si_dev, @@ -4252,7 +4156,7 @@ static int handle_one_recv_msg(ipmi_smi_t intf, * It's a response to a response we sent. For this we * deliver a send message response to the user. */ - struct ipmi_recv_msg *recv_msg = msg->user_data; + struct ipmi_recv_msg *recv_msg = msg->user_data; requeue = 0; if (msg->rsp_size < 2) @@ -4267,15 +4171,11 @@ static int handle_one_recv_msg(ipmi_smi_t intf, if (!recv_msg) goto out; - /* Make sure the user still exists. */ - if (!recv_msg->user || !recv_msg->user->valid) - goto out; - recv_msg->recv_type = IPMI_RESPONSE_RESPONSE_TYPE; recv_msg->msg.data = recv_msg->msg_data; recv_msg->msg.data_len = 1; recv_msg->msg_data[0] = msg->rsp[2]; - deliver_response(recv_msg); + deliver_local_response(intf, recv_msg); } else if ((msg->rsp[0] == ((IPMI_NETFN_APP_REQUEST|1) << 2)) && (msg->rsp[1] == IPMI_GET_MSG_CMD)) { struct ipmi_channel *chans; @@ -4367,7 +4267,7 @@ static int handle_one_recv_msg(ipmi_smi_t intf, /* * If there are messages in the queue or pretimeouts, handle them. */ -static void handle_new_recv_msgs(ipmi_smi_t intf) +static void handle_new_recv_msgs(struct ipmi_smi *intf) { struct ipmi_smi_msg *smi_msg; unsigned long flags = 0; @@ -4412,22 +4312,23 @@ static void handle_new_recv_msgs(ipmi_smi_t intf) * deliver pretimeouts to all the users. */ if (atomic_add_unless(&intf->watchdog_pretimeouts_to_deliver, -1, 0)) { - ipmi_user_t user; + struct ipmi_user *user; + int index; - rcu_read_lock(); + index = srcu_read_lock(&intf->users_srcu); list_for_each_entry_rcu(user, &intf->users, link) { if (user->handler->ipmi_watchdog_pretimeout) user->handler->ipmi_watchdog_pretimeout( user->handler_data); } - rcu_read_unlock(); + srcu_read_unlock(&intf->users_srcu, index); } } static void smi_recv_tasklet(unsigned long val) { unsigned long flags = 0; /* keep us warning-free. */ - ipmi_smi_t intf = (ipmi_smi_t) val; + struct ipmi_smi *intf = (struct ipmi_smi *) val; int run_to_completion = intf->run_to_completion; struct ipmi_smi_msg *newmsg = NULL; @@ -4469,7 +4370,7 @@ static void smi_recv_tasklet(unsigned long val) } /* Handle a new message from the lower layer. */ -void ipmi_smi_msg_received(ipmi_smi_t intf, +void ipmi_smi_msg_received(struct ipmi_smi *intf, struct ipmi_smi_msg *msg) { unsigned long flags = 0; /* keep us warning-free. */ @@ -4550,7 +4451,7 @@ free_msg: } EXPORT_SYMBOL(ipmi_smi_msg_received); -void ipmi_smi_watchdog_pretimeout(ipmi_smi_t intf) +void ipmi_smi_watchdog_pretimeout(struct ipmi_smi *intf) { if (intf->in_shutdown) return; @@ -4561,7 +4462,7 @@ void ipmi_smi_watchdog_pretimeout(ipmi_smi_t intf) EXPORT_SYMBOL(ipmi_smi_watchdog_pretimeout); static struct ipmi_smi_msg * -smi_from_recv_msg(ipmi_smi_t intf, struct ipmi_recv_msg *recv_msg, +smi_from_recv_msg(struct ipmi_smi *intf, struct ipmi_recv_msg *recv_msg, unsigned char seq, long seqid) { struct ipmi_smi_msg *smi_msg = ipmi_alloc_smi_msg(); @@ -4576,26 +4477,18 @@ smi_from_recv_msg(ipmi_smi_t intf, struct ipmi_recv_msg *recv_msg, smi_msg->data_size = recv_msg->msg.data_len; smi_msg->msgid = STORE_SEQ_IN_MSGID(seq, seqid); -#ifdef DEBUG_MSGING - { - int m; - printk("Resend: "); - for (m = 0; m < smi_msg->data_size; m++) - printk(" %2.2x", smi_msg->data[m]); - printk("\n"); - } -#endif + ipmi_debug_msg("Resend: ", smi_msg->data, smi_msg->data_size); + return smi_msg; } -static void check_msg_timeout(ipmi_smi_t intf, struct seq_table *ent, +static void check_msg_timeout(struct ipmi_smi *intf, struct seq_table *ent, struct list_head *timeouts, unsigned long timeout_period, int slot, unsigned long *flags, unsigned int *waiting_msgs) { - struct ipmi_recv_msg *msg; - const struct ipmi_smi_handlers *handlers; + struct ipmi_recv_msg *msg; if (intf->in_shutdown) return; @@ -4653,8 +4546,7 @@ static void check_msg_timeout(ipmi_smi_t intf, struct seq_table *ent, * only for messages to the local MC, which don't get * resent. */ - handlers = intf->handlers; - if (handlers) { + if (intf->handlers) { if (is_lan_addr(&ent->recv_msg->addr)) ipmi_inc_stat(intf, retransmitted_lan_commands); @@ -4662,7 +4554,7 @@ static void check_msg_timeout(ipmi_smi_t intf, struct seq_table *ent, ipmi_inc_stat(intf, retransmitted_ipmb_commands); - smi_send(intf, handlers, smi_msg, 0); + smi_send(intf, intf->handlers, smi_msg, 0); } else ipmi_free_smi_msg(smi_msg); @@ -4670,7 +4562,7 @@ static void check_msg_timeout(ipmi_smi_t intf, struct seq_table *ent, } } -static unsigned int ipmi_timeout_handler(ipmi_smi_t intf, +static unsigned int ipmi_timeout_handler(struct ipmi_smi *intf, unsigned long timeout_period) { struct list_head timeouts; @@ -4694,14 +4586,20 @@ static unsigned int ipmi_timeout_handler(ipmi_smi_t intf, */ INIT_LIST_HEAD(&timeouts); spin_lock_irqsave(&intf->seq_lock, flags); + if (intf->ipmb_maintenance_mode_timeout) { + if (intf->ipmb_maintenance_mode_timeout <= timeout_period) + intf->ipmb_maintenance_mode_timeout = 0; + else + intf->ipmb_maintenance_mode_timeout -= timeout_period; + } for (i = 0; i < IPMI_IPMB_NUM_SEQ; i++) - check_msg_timeout(intf, &(intf->seq_table[i]), + check_msg_timeout(intf, &intf->seq_table[i], &timeouts, timeout_period, i, &flags, &waiting_msgs); spin_unlock_irqrestore(&intf->seq_lock, flags); list_for_each_entry_safe(msg, msg2, &timeouts, link) - deliver_err_response(msg, IPMI_TIMEOUT_COMPLETION_CODE); + deliver_err_response(intf, msg, IPMI_TIMEOUT_COMPLETION_CODE); /* * Maintenance mode handling. Check the timeout @@ -4731,7 +4629,7 @@ static unsigned int ipmi_timeout_handler(ipmi_smi_t intf, return waiting_msgs; } -static void ipmi_request_event(ipmi_smi_t intf) +static void ipmi_request_event(struct ipmi_smi *intf) { /* No event requests when in maintenance mode. */ if (intf->maintenance_mode_enable) @@ -4747,13 +4645,13 @@ static atomic_t stop_operation; static void ipmi_timeout(struct timer_list *unused) { - ipmi_smi_t intf; - int nt = 0; + struct ipmi_smi *intf; + int nt = 0, index; if (atomic_read(&stop_operation)) return; - rcu_read_lock(); + index = srcu_read_lock(&ipmi_interfaces_srcu); list_for_each_entry_rcu(intf, &ipmi_interfaces, link) { int lnt = 0; @@ -4776,13 +4674,13 @@ static void ipmi_timeout(struct timer_list *unused) nt += lnt; } - rcu_read_unlock(); + srcu_read_unlock(&ipmi_interfaces_srcu, index); if (nt) mod_timer(&ipmi_timer, jiffies + IPMI_TIMEOUT_JIFFIES); } -static void need_waiter(ipmi_smi_t intf) +static void need_waiter(struct ipmi_smi *intf) { /* Racy, but worst case we start the timer twice. */ if (!timer_pending(&ipmi_timer)) @@ -4853,8 +4751,8 @@ static void dummy_recv_done_handler(struct ipmi_recv_msg *msg) /* * Inside a panic, send a message and wait for a response. */ -static void ipmi_panic_request_and_wait(ipmi_smi_t intf, - struct ipmi_addr *addr, +static void ipmi_panic_request_and_wait(struct ipmi_smi *intf, + struct ipmi_addr *addr, struct kernel_ipmi_msg *msg) { struct ipmi_smi_msg smi_msg; @@ -4885,7 +4783,8 @@ static void ipmi_panic_request_and_wait(ipmi_smi_t intf, ipmi_poll(intf); } -static void event_receiver_fetcher(ipmi_smi_t intf, struct ipmi_recv_msg *msg) +static void event_receiver_fetcher(struct ipmi_smi *intf, + struct ipmi_recv_msg *msg) { if ((msg->addr.addr_type == IPMI_SYSTEM_INTERFACE_ADDR_TYPE) && (msg->msg.netfn == IPMI_NETFN_SENSOR_EVENT_RESPONSE) @@ -4897,7 +4796,7 @@ static void event_receiver_fetcher(ipmi_smi_t intf, struct ipmi_recv_msg *msg) } } -static void device_id_fetcher(ipmi_smi_t intf, struct ipmi_recv_msg *msg) +static void device_id_fetcher(struct ipmi_smi *intf, struct ipmi_recv_msg *msg) { if ((msg->addr.addr_type == IPMI_SYSTEM_INTERFACE_ADDR_TYPE) && (msg->msg.netfn == IPMI_NETFN_APP_RESPONSE) @@ -4912,13 +4811,15 @@ static void device_id_fetcher(ipmi_smi_t intf, struct ipmi_recv_msg *msg) } } -static void send_panic_events(char *str) +static void send_panic_events(struct ipmi_smi *intf, char *str) { - struct kernel_ipmi_msg msg; - ipmi_smi_t intf; - unsigned char data[16]; + struct kernel_ipmi_msg msg; + unsigned char data[16]; struct ipmi_system_interface_addr *si; - struct ipmi_addr addr; + struct ipmi_addr addr; + char *p = str; + struct ipmi_ipmb_addr *ipmb; + int j; if (ipmi_send_panic_event == IPMI_SEND_PANIC_EVENT_NONE) return; @@ -4949,15 +4850,8 @@ static void send_panic_events(char *str) data[7] = str[2]; } - /* For every registered interface, send the event. */ - list_for_each_entry_rcu(intf, &ipmi_interfaces, link) { - if (!intf->handlers || !intf->handlers->poll) - /* Interface is not ready or can't run at panic time. */ - continue; - - /* Send the event announcing the panic. */ - ipmi_panic_request_and_wait(intf, &addr, &msg); - } + /* Send the event announcing the panic. */ + ipmi_panic_request_and_wait(intf, &addr, &msg); /* * On every interface, dump a bunch of OEM event holding the @@ -4966,111 +4860,100 @@ static void send_panic_events(char *str) if (ipmi_send_panic_event != IPMI_SEND_PANIC_EVENT_STRING || !str) return; - /* For every registered interface, send the event. */ - list_for_each_entry_rcu(intf, &ipmi_interfaces, link) { - char *p = str; - struct ipmi_ipmb_addr *ipmb; - int j; - - if (intf->intf_num == -1) - /* Interface was not ready yet. */ - continue; + /* + * intf_num is used as an marker to tell if the + * interface is valid. Thus we need a read barrier to + * make sure data fetched before checking intf_num + * won't be used. + */ + smp_rmb(); - /* - * intf_num is used as an marker to tell if the - * interface is valid. Thus we need a read barrier to - * make sure data fetched before checking intf_num - * won't be used. - */ - smp_rmb(); + /* + * First job here is to figure out where to send the + * OEM events. There's no way in IPMI to send OEM + * events using an event send command, so we have to + * find the SEL to put them in and stick them in + * there. + */ - /* - * First job here is to figure out where to send the - * OEM events. There's no way in IPMI to send OEM - * events using an event send command, so we have to - * find the SEL to put them in and stick them in - * there. - */ + /* Get capabilities from the get device id. */ + intf->local_sel_device = 0; + intf->local_event_generator = 0; + intf->event_receiver = 0; - /* Get capabilities from the get device id. */ - intf->local_sel_device = 0; - intf->local_event_generator = 0; - intf->event_receiver = 0; + /* Request the device info from the local MC. */ + msg.netfn = IPMI_NETFN_APP_REQUEST; + msg.cmd = IPMI_GET_DEVICE_ID_CMD; + msg.data = NULL; + msg.data_len = 0; + intf->null_user_handler = device_id_fetcher; + ipmi_panic_request_and_wait(intf, &addr, &msg); - /* Request the device info from the local MC. */ - msg.netfn = IPMI_NETFN_APP_REQUEST; - msg.cmd = IPMI_GET_DEVICE_ID_CMD; + if (intf->local_event_generator) { + /* Request the event receiver from the local MC. */ + msg.netfn = IPMI_NETFN_SENSOR_EVENT_REQUEST; + msg.cmd = IPMI_GET_EVENT_RECEIVER_CMD; msg.data = NULL; msg.data_len = 0; - intf->null_user_handler = device_id_fetcher; + intf->null_user_handler = event_receiver_fetcher; ipmi_panic_request_and_wait(intf, &addr, &msg); + } + intf->null_user_handler = NULL; - if (intf->local_event_generator) { - /* Request the event receiver from the local MC. */ - msg.netfn = IPMI_NETFN_SENSOR_EVENT_REQUEST; - msg.cmd = IPMI_GET_EVENT_RECEIVER_CMD; - msg.data = NULL; - msg.data_len = 0; - intf->null_user_handler = event_receiver_fetcher; - ipmi_panic_request_and_wait(intf, &addr, &msg); - } - intf->null_user_handler = NULL; + /* + * Validate the event receiver. The low bit must not + * be 1 (it must be a valid IPMB address), it cannot + * be zero, and it must not be my address. + */ + if (((intf->event_receiver & 1) == 0) + && (intf->event_receiver != 0) + && (intf->event_receiver != intf->addrinfo[0].address)) { + /* + * The event receiver is valid, send an IPMB + * message. + */ + ipmb = (struct ipmi_ipmb_addr *) &addr; + ipmb->addr_type = IPMI_IPMB_ADDR_TYPE; + ipmb->channel = 0; /* FIXME - is this right? */ + ipmb->lun = intf->event_receiver_lun; + ipmb->slave_addr = intf->event_receiver; + } else if (intf->local_sel_device) { + /* + * The event receiver was not valid (or was + * me), but I am an SEL device, just dump it + * in my SEL. + */ + si = (struct ipmi_system_interface_addr *) &addr; + si->addr_type = IPMI_SYSTEM_INTERFACE_ADDR_TYPE; + si->channel = IPMI_BMC_CHANNEL; + si->lun = 0; + } else + return; /* No where to send the event. */ + msg.netfn = IPMI_NETFN_STORAGE_REQUEST; /* Storage. */ + msg.cmd = IPMI_ADD_SEL_ENTRY_CMD; + msg.data = data; + msg.data_len = 16; + + j = 0; + while (*p) { + int size = strlen(p); + + if (size > 11) + size = 11; + data[0] = 0; + data[1] = 0; + data[2] = 0xf0; /* OEM event without timestamp. */ + data[3] = intf->addrinfo[0].address; + data[4] = j++; /* sequence # */ /* - * Validate the event receiver. The low bit must not - * be 1 (it must be a valid IPMB address), it cannot - * be zero, and it must not be my address. + * Always give 11 bytes, so strncpy will fill + * it with zeroes for me. */ - if (((intf->event_receiver & 1) == 0) - && (intf->event_receiver != 0) - && (intf->event_receiver != intf->addrinfo[0].address)) { - /* - * The event receiver is valid, send an IPMB - * message. - */ - ipmb = (struct ipmi_ipmb_addr *) &addr; - ipmb->addr_type = IPMI_IPMB_ADDR_TYPE; - ipmb->channel = 0; /* FIXME - is this right? */ - ipmb->lun = intf->event_receiver_lun; - ipmb->slave_addr = intf->event_receiver; - } else if (intf->local_sel_device) { - /* - * The event receiver was not valid (or was - * me), but I am an SEL device, just dump it - * in my SEL. - */ - si = (struct ipmi_system_interface_addr *) &addr; - si->addr_type = IPMI_SYSTEM_INTERFACE_ADDR_TYPE; - si->channel = IPMI_BMC_CHANNEL; - si->lun = 0; - } else - continue; /* No where to send the event. */ - - msg.netfn = IPMI_NETFN_STORAGE_REQUEST; /* Storage. */ - msg.cmd = IPMI_ADD_SEL_ENTRY_CMD; - msg.data = data; - msg.data_len = 16; - - j = 0; - while (*p) { - int size = strlen(p); - - if (size > 11) - size = 11; - data[0] = 0; - data[1] = 0; - data[2] = 0xf0; /* OEM event without timestamp. */ - data[3] = intf->addrinfo[0].address; - data[4] = j++; /* sequence # */ - /* - * Always give 11 bytes, so strncpy will fill - * it with zeroes for me. - */ - strncpy(data+5, p, 11); - p += size; + strncpy(data+5, p, 11); + p += size; - ipmi_panic_request_and_wait(intf, &addr, &msg); - } + ipmi_panic_request_and_wait(intf, &addr, &msg); } } @@ -5080,7 +4963,8 @@ static int panic_event(struct notifier_block *this, unsigned long event, void *ptr) { - ipmi_smi_t intf; + struct ipmi_smi *intf; + struct ipmi_user *user; if (has_panicked) return NOTIFY_DONE; @@ -5088,10 +4972,13 @@ static int panic_event(struct notifier_block *this, /* For every registered interface, set it to run to completion. */ list_for_each_entry_rcu(intf, &ipmi_interfaces, link) { - if (!intf->handlers) + if (!intf->handlers || intf->intf_num == -1) /* Interface is not ready. */ continue; + if (!intf->handlers->poll) + continue; + /* * If we were interrupted while locking xmit_msgs_lock or * waiting_rcv_msgs_lock, the corresponding list may be @@ -5113,9 +5000,15 @@ static int panic_event(struct notifier_block *this, if (intf->handlers->set_run_to_completion) intf->handlers->set_run_to_completion(intf->send_info, 1); - } - send_panic_events(ptr); + list_for_each_entry_rcu(user, &intf->users, link) { + if (user->handler->ipmi_panic_handler) + user->handler->ipmi_panic_handler( + user->handler_data); + } + + send_panic_events(intf, ptr); + } return NOTIFY_DONE; } @@ -5141,16 +5034,6 @@ static int ipmi_init_msghandler(void) pr_info("ipmi message handler version " IPMI_DRIVER_VERSION "\n"); -#ifdef CONFIG_IPMI_PROC_INTERFACE - proc_ipmi_root = proc_mkdir("ipmi", NULL); - if (!proc_ipmi_root) { - pr_err(PFX "Unable to create IPMI proc dir"); - driver_unregister(&ipmidriver.driver); - return -ENOMEM; - } - -#endif /* CONFIG_IPMI_PROC_INTERFACE */ - timer_setup(&ipmi_timer, ipmi_timeout, 0); mod_timer(&ipmi_timer, jiffies + IPMI_TIMEOUT_JIFFIES); @@ -5189,10 +5072,6 @@ static void __exit cleanup_ipmi(void) atomic_inc(&stop_operation); del_timer_sync(&ipmi_timer); -#ifdef CONFIG_IPMI_PROC_INTERFACE - proc_remove(proc_ipmi_root); -#endif /* CONFIG_IPMI_PROC_INTERFACE */ - driver_unregister(&ipmidriver.driver); initialized = 0; diff --git a/drivers/char/ipmi/ipmi_poweroff.c b/drivers/char/ipmi/ipmi_poweroff.c index 7996337852f2..f6e19410dc57 100644 --- a/drivers/char/ipmi/ipmi_poweroff.c +++ b/drivers/char/ipmi/ipmi_poweroff.c @@ -39,9 +39,9 @@ static int ifnum_to_use = -1; /* Our local state. */ static int ready; -static ipmi_user_t ipmi_user; +static struct ipmi_user *ipmi_user; static int ipmi_ifnum; -static void (*specific_poweroff_func)(ipmi_user_t user); +static void (*specific_poweroff_func)(struct ipmi_user *user); /* Holds the old poweroff function so we can restore it on removal. */ static void (*old_poweroff_func)(void); @@ -118,7 +118,7 @@ static const struct ipmi_user_hndl ipmi_poweroff_handler = { }; -static int ipmi_request_wait_for_response(ipmi_user_t user, +static int ipmi_request_wait_for_response(struct ipmi_user *user, struct ipmi_addr *addr, struct kernel_ipmi_msg *send_msg) { @@ -138,7 +138,7 @@ static int ipmi_request_wait_for_response(ipmi_user_t user, } /* Wait for message to complete, spinning. */ -static int ipmi_request_in_rc_mode(ipmi_user_t user, +static int ipmi_request_in_rc_mode(struct ipmi_user *user, struct ipmi_addr *addr, struct kernel_ipmi_msg *send_msg) { @@ -178,9 +178,9 @@ static int ipmi_request_in_rc_mode(ipmi_user_t user, #define IPMI_MOTOROLA_MANUFACTURER_ID 0x0000A1 #define IPMI_MOTOROLA_PPS_IPMC_PRODUCT_ID 0x0051 -static void (*atca_oem_poweroff_hook)(ipmi_user_t user); +static void (*atca_oem_poweroff_hook)(struct ipmi_user *user); -static void pps_poweroff_atca(ipmi_user_t user) +static void pps_poweroff_atca(struct ipmi_user *user) { struct ipmi_system_interface_addr smi_addr; struct kernel_ipmi_msg send_msg; @@ -208,7 +208,7 @@ static void pps_poweroff_atca(ipmi_user_t user) return; } -static int ipmi_atca_detect(ipmi_user_t user) +static int ipmi_atca_detect(struct ipmi_user *user) { struct ipmi_system_interface_addr smi_addr; struct kernel_ipmi_msg send_msg; @@ -245,7 +245,7 @@ static int ipmi_atca_detect(ipmi_user_t user) return !rv; } -static void ipmi_poweroff_atca(ipmi_user_t user) +static void ipmi_poweroff_atca(struct ipmi_user *user) { struct ipmi_system_interface_addr smi_addr; struct kernel_ipmi_msg send_msg; @@ -309,13 +309,13 @@ static void ipmi_poweroff_atca(ipmi_user_t user) #define IPMI_CPI1_PRODUCT_ID 0x000157 #define IPMI_CPI1_MANUFACTURER_ID 0x0108 -static int ipmi_cpi1_detect(ipmi_user_t user) +static int ipmi_cpi1_detect(struct ipmi_user *user) { return ((mfg_id == IPMI_CPI1_MANUFACTURER_ID) && (prod_id == IPMI_CPI1_PRODUCT_ID)); } -static void ipmi_poweroff_cpi1(ipmi_user_t user) +static void ipmi_poweroff_cpi1(struct ipmi_user *user) { struct ipmi_system_interface_addr smi_addr; struct ipmi_ipmb_addr ipmb_addr; @@ -424,7 +424,7 @@ static void ipmi_poweroff_cpi1(ipmi_user_t user) */ #define DELL_IANA_MFR_ID {0xA2, 0x02, 0x00} -static int ipmi_dell_chassis_detect(ipmi_user_t user) +static int ipmi_dell_chassis_detect(struct ipmi_user *user) { const char ipmi_version_major = ipmi_version & 0xF; const char ipmi_version_minor = (ipmi_version >> 4) & 0xF; @@ -445,7 +445,7 @@ static int ipmi_dell_chassis_detect(ipmi_user_t user) #define HP_IANA_MFR_ID 0x0b #define HP_BMC_PROD_ID 0x8201 -static int ipmi_hp_chassis_detect(ipmi_user_t user) +static int ipmi_hp_chassis_detect(struct ipmi_user *user) { if (mfg_id == HP_IANA_MFR_ID && prod_id == HP_BMC_PROD_ID @@ -461,13 +461,13 @@ static int ipmi_hp_chassis_detect(ipmi_user_t user) #define IPMI_NETFN_CHASSIS_REQUEST 0 #define IPMI_CHASSIS_CONTROL_CMD 0x02 -static int ipmi_chassis_detect(ipmi_user_t user) +static int ipmi_chassis_detect(struct ipmi_user *user) { /* Chassis support, use it. */ return (capabilities & 0x80); } -static void ipmi_poweroff_chassis(ipmi_user_t user) +static void ipmi_poweroff_chassis(struct ipmi_user *user) { struct ipmi_system_interface_addr smi_addr; struct kernel_ipmi_msg send_msg; @@ -517,8 +517,8 @@ static void ipmi_poweroff_chassis(ipmi_user_t user) /* Table of possible power off functions. */ struct poweroff_function { char *platform_type; - int (*detect)(ipmi_user_t user); - void (*poweroff_func)(ipmi_user_t user); + int (*detect)(struct ipmi_user *user); + void (*poweroff_func)(struct ipmi_user *user); }; static struct poweroff_function poweroff_functions[] = { diff --git a/drivers/char/ipmi/ipmi_si_intf.c b/drivers/char/ipmi/ipmi_si_intf.c index ff870aa91cfe..ad353be871bf 100644 --- a/drivers/char/ipmi/ipmi_si_intf.c +++ b/drivers/char/ipmi/ipmi_si_intf.c @@ -122,8 +122,8 @@ enum si_stat_indexes { }; struct smi_info { - int intf_num; - ipmi_smi_t intf; + int si_num; + struct ipmi_smi *intf; struct si_sm_data *si_sm; const struct si_sm_handlers *handlers; spinlock_t si_lock; @@ -261,7 +261,6 @@ static int num_max_busy_us; static bool unload_when_empty = true; static int try_smi_init(struct smi_info *smi); -static void shutdown_one_si(struct smi_info *smi_info); static void cleanup_one_si(struct smi_info *smi_info); static void cleanup_ipmi_si(void); @@ -287,10 +286,7 @@ static void deliver_recv_msg(struct smi_info *smi_info, struct ipmi_smi_msg *msg) { /* Deliver the message to the upper layer. */ - if (smi_info->intf) - ipmi_smi_msg_received(smi_info->intf, msg); - else - ipmi_free_smi_msg(msg); + ipmi_smi_msg_received(smi_info->intf, msg); } static void return_hosed_msg(struct smi_info *smi_info, int cCode) @@ -471,8 +467,7 @@ retry: start_clear_flags(smi_info); smi_info->msg_flags &= ~WDT_PRE_TIMEOUT_INT; - if (smi_info->intf) - ipmi_smi_watchdog_pretimeout(smi_info->intf); + ipmi_smi_watchdog_pretimeout(smi_info->intf); } else if (smi_info->msg_flags & RECEIVE_MSG_AVAIL) { /* Messages available. */ smi_info->curr_msg = alloc_msg_handle_irq(smi_info); @@ -798,8 +793,7 @@ restart: * We prefer handling attn over new messages. But don't do * this if there is not yet an upper layer to handle anything. */ - if (likely(smi_info->intf) && - (si_sm_result == SI_SM_ATTN || smi_info->got_attn)) { + if (si_sm_result == SI_SM_ATTN || smi_info->got_attn) { unsigned char msg[2]; if (smi_info->si_state != SI_NORMAL) { @@ -962,8 +956,8 @@ static inline int ipmi_thread_busy_wait(enum si_sm_result smi_result, { unsigned int max_busy_us = 0; - if (smi_info->intf_num < num_max_busy_us) - max_busy_us = kipmid_max_busy_us[smi_info->intf_num]; + if (smi_info->si_num < num_max_busy_us) + max_busy_us = kipmid_max_busy_us[smi_info->si_num]; if (max_busy_us == 0 || smi_result != SI_SM_CALL_WITH_DELAY) ipmi_si_set_not_busy(busy_until); else if (!ipmi_si_is_busy(busy_until)) { @@ -1143,8 +1137,8 @@ irqreturn_t ipmi_si_irq_handler(int irq, void *data) return IRQ_HANDLED; } -static int smi_start_processing(void *send_info, - ipmi_smi_t intf) +static int smi_start_processing(void *send_info, + struct ipmi_smi *intf) { struct smi_info *new_smi = send_info; int enable = 0; @@ -1165,8 +1159,8 @@ static int smi_start_processing(void *send_info, /* * Check if the user forcefully enabled the daemon. */ - if (new_smi->intf_num < num_force_kipmid) - enable = force_kipmid[new_smi->intf_num]; + if (new_smi->si_num < num_force_kipmid) + enable = force_kipmid[new_smi->si_num]; /* * The BT interface is efficient enough to not need a thread, * and there is no need for a thread if we have interrupts. @@ -1176,7 +1170,7 @@ static int smi_start_processing(void *send_info, if (enable) { new_smi->thread = kthread_run(ipmi_thread, new_smi, - "kipmi%d", new_smi->intf_num); + "kipmi%d", new_smi->si_num); if (IS_ERR(new_smi->thread)) { dev_notice(new_smi->io.dev, "Could not start" " kernel thread due to error %ld, only using" @@ -1209,9 +1203,11 @@ static void set_maintenance_mode(void *send_info, bool enable) atomic_set(&smi_info->req_events, 0); } +static void shutdown_smi(void *send_info); static const struct ipmi_smi_handlers handlers = { .owner = THIS_MODULE, .start_processing = smi_start_processing, + .shutdown = shutdown_smi, .get_smi_info = get_smi_info, .sender = sender, .request_events = request_events, @@ -1592,102 +1588,6 @@ out: return rv; } -#ifdef CONFIG_IPMI_PROC_INTERFACE -static int smi_type_proc_show(struct seq_file *m, void *v) -{ - struct smi_info *smi = m->private; - - seq_printf(m, "%s\n", si_to_str[smi->io.si_type]); - - return 0; -} - -static int smi_type_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, smi_type_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations smi_type_proc_ops = { - .open = smi_type_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - -static int smi_si_stats_proc_show(struct seq_file *m, void *v) -{ - struct smi_info *smi = m->private; - - seq_printf(m, "interrupts_enabled: %d\n", - smi->io.irq && !smi->interrupt_disabled); - seq_printf(m, "short_timeouts: %u\n", - smi_get_stat(smi, short_timeouts)); - seq_printf(m, "long_timeouts: %u\n", - smi_get_stat(smi, long_timeouts)); - seq_printf(m, "idles: %u\n", - smi_get_stat(smi, idles)); - seq_printf(m, "interrupts: %u\n", - smi_get_stat(smi, interrupts)); - seq_printf(m, "attentions: %u\n", - smi_get_stat(smi, attentions)); - seq_printf(m, "flag_fetches: %u\n", - smi_get_stat(smi, flag_fetches)); - seq_printf(m, "hosed_count: %u\n", - smi_get_stat(smi, hosed_count)); - seq_printf(m, "complete_transactions: %u\n", - smi_get_stat(smi, complete_transactions)); - seq_printf(m, "events: %u\n", - smi_get_stat(smi, events)); - seq_printf(m, "watchdog_pretimeouts: %u\n", - smi_get_stat(smi, watchdog_pretimeouts)); - seq_printf(m, "incoming_messages: %u\n", - smi_get_stat(smi, incoming_messages)); - return 0; -} - -static int smi_si_stats_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, smi_si_stats_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations smi_si_stats_proc_ops = { - .open = smi_si_stats_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - -static int smi_params_proc_show(struct seq_file *m, void *v) -{ - struct smi_info *smi = m->private; - - seq_printf(m, - "%s,%s,0x%lx,rsp=%d,rsi=%d,rsh=%d,irq=%d,ipmb=%d\n", - si_to_str[smi->io.si_type], - addr_space_to_str[smi->io.addr_type], - smi->io.addr_data, - smi->io.regspacing, - smi->io.regsize, - smi->io.regshift, - smi->io.irq, - smi->io.slave_addr); - - return 0; -} - -static int smi_params_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, smi_params_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations smi_params_proc_ops = { - .open = smi_params_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; -#endif - #define IPMI_SI_ATTR(name) \ static ssize_t ipmi_##name##_show(struct device *dev, \ struct device_attribute *attr, \ @@ -2006,14 +1906,8 @@ int ipmi_si_add_smi(struct si_sm_io *io) list_add_tail(&new_smi->link, &smi_infos); - if (initialized) { + if (initialized) rv = try_smi_init(new_smi); - if (rv) { - cleanup_one_si(new_smi); - mutex_unlock(&smi_infos_lock); - return rv; - } - } out_err: mutex_unlock(&smi_infos_lock); return rv; @@ -2056,19 +1950,19 @@ static int try_smi_init(struct smi_info *new_smi) goto out_err; } - new_smi->intf_num = smi_num; + new_smi->si_num = smi_num; /* Do this early so it's available for logs. */ if (!new_smi->io.dev) { init_name = kasprintf(GFP_KERNEL, "ipmi_si.%d", - new_smi->intf_num); + new_smi->si_num); /* * If we don't already have a device from something * else (like PCI), then register a new one. */ new_smi->pdev = platform_device_alloc("ipmi_si", - new_smi->intf_num); + new_smi->si_num); if (!new_smi->pdev) { pr_err(PFX "Unable to allocate platform device\n"); rv = -ENOMEM; @@ -2182,35 +2076,6 @@ static int try_smi_init(struct smi_info *new_smi) goto out_err; } -#ifdef CONFIG_IPMI_PROC_INTERFACE - rv = ipmi_smi_add_proc_entry(new_smi->intf, "type", - &smi_type_proc_ops, - new_smi); - if (rv) { - dev_err(new_smi->io.dev, - "Unable to create proc entry: %d\n", rv); - goto out_err; - } - - rv = ipmi_smi_add_proc_entry(new_smi->intf, "si_stats", - &smi_si_stats_proc_ops, - new_smi); - if (rv) { - dev_err(new_smi->io.dev, - "Unable to create proc entry: %d\n", rv); - goto out_err; - } - - rv = ipmi_smi_add_proc_entry(new_smi->intf, "params", - &smi_params_proc_ops, - new_smi); - if (rv) { - dev_err(new_smi->io.dev, - "Unable to create proc entry: %d\n", rv); - goto out_err; - } -#endif - /* Don't increment till we know we have succeeded. */ smi_num++; @@ -2223,7 +2088,8 @@ static int try_smi_init(struct smi_info *new_smi) return 0; out_err: - shutdown_one_si(new_smi); + ipmi_unregister_smi(new_smi->intf); + new_smi->intf = NULL; kfree(init_name); @@ -2301,20 +2167,9 @@ skip_fallback_noirq: } module_init(init_ipmi_si); -static void shutdown_one_si(struct smi_info *smi_info) +static void shutdown_smi(void *send_info) { - int rv = 0; - - if (smi_info->intf) { - ipmi_smi_t intf = smi_info->intf; - - smi_info->intf = NULL; - rv = ipmi_unregister_smi(intf); - if (rv) { - pr_err(PFX "Unable to unregister device: errno=%d\n", - rv); - } - } + struct smi_info *smi_info = send_info; if (smi_info->dev_group_added) { device_remove_group(smi_info->io.dev, &ipmi_si_dev_attr_group); @@ -2372,6 +2227,10 @@ static void shutdown_one_si(struct smi_info *smi_info) smi_info->si_sm = NULL; } +/* + * Must be called with smi_infos_lock held, to serialize the + * smi_info->intf check. + */ static void cleanup_one_si(struct smi_info *smi_info) { if (!smi_info) @@ -2379,7 +2238,10 @@ static void cleanup_one_si(struct smi_info *smi_info) list_del(&smi_info->link); - shutdown_one_si(smi_info); + if (smi_info->intf) { + ipmi_unregister_smi(smi_info->intf); + smi_info->intf = NULL; + } if (smi_info->pdev) { if (smi_info->pdev_registered) diff --git a/drivers/char/ipmi/ipmi_ssif.c b/drivers/char/ipmi/ipmi_ssif.c index 35a82f4bfd78..18e4650c233b 100644 --- a/drivers/char/ipmi/ipmi_ssif.c +++ b/drivers/char/ipmi/ipmi_ssif.c @@ -193,8 +193,7 @@ typedef void (*ssif_i2c_done)(struct ssif_info *ssif_info, int result, unsigned char *data, unsigned int len); struct ssif_info { - ipmi_smi_t intf; - int intf_num; + struct ipmi_smi *intf; spinlock_t lock; struct ipmi_smi_msg *waiting_msg; struct ipmi_smi_msg *curr_msg; @@ -290,8 +289,6 @@ struct ssif_info { static bool initialized; -static atomic_t next_intf = ATOMIC_INIT(0); - static void return_hosed_msg(struct ssif_info *ssif_info, struct ipmi_smi_msg *msg); static void start_next_msg(struct ssif_info *ssif_info, unsigned long *flags); @@ -315,17 +312,13 @@ static void ipmi_ssif_unlock_cond(struct ssif_info *ssif_info, static void deliver_recv_msg(struct ssif_info *ssif_info, struct ipmi_smi_msg *msg) { - ipmi_smi_t intf = ssif_info->intf; - - if (!intf) { - ipmi_free_smi_msg(msg); - } else if (msg->rsp_size < 0) { + if (msg->rsp_size < 0) { return_hosed_msg(ssif_info, msg); pr_err(PFX "Malformed message in deliver_recv_msg: rsp_size = %d\n", msg->rsp_size); } else { - ipmi_smi_msg_received(intf, msg); + ipmi_smi_msg_received(ssif_info->intf, msg); } } @@ -452,12 +445,10 @@ static void start_recv_msg_fetch(struct ssif_info *ssif_info, static void handle_flags(struct ssif_info *ssif_info, unsigned long *flags) { if (ssif_info->msg_flags & WDT_PRE_TIMEOUT_INT) { - ipmi_smi_t intf = ssif_info->intf; /* Watchdog pre-timeout */ ssif_inc_stat(ssif_info, watchdog_pretimeouts); start_clear_flags(ssif_info, flags); - if (intf) - ipmi_smi_watchdog_pretimeout(intf); + ipmi_smi_watchdog_pretimeout(ssif_info->intf); } else if (ssif_info->msg_flags & RECEIVE_MSG_AVAIL) /* Messages available. */ start_recv_msg_fetch(ssif_info, flags); @@ -1094,27 +1085,8 @@ static void request_events(void *send_info) } } -static int inc_usecount(void *send_info) -{ - struct ssif_info *ssif_info = send_info; - - if (!i2c_get_adapter(i2c_adapter_id(ssif_info->client->adapter))) - return -ENODEV; - - i2c_use_client(ssif_info->client); - return 0; -} - -static void dec_usecount(void *send_info) -{ - struct ssif_info *ssif_info = send_info; - - i2c_release_client(ssif_info->client); - i2c_put_adapter(ssif_info->client->adapter); -} - -static int ssif_start_processing(void *send_info, - ipmi_smi_t intf) +static int ssif_start_processing(void *send_info, + struct ipmi_smi *intf) { struct ssif_info *ssif_info = send_info; @@ -1225,25 +1197,9 @@ static const struct attribute_group ipmi_ssif_dev_attr_group = { .attrs = ipmi_ssif_dev_attrs, }; -static int ssif_remove(struct i2c_client *client) +static void shutdown_ssif(void *send_info) { - struct ssif_info *ssif_info = i2c_get_clientdata(client); - struct ssif_addr_info *addr_info; - int rv; - - if (!ssif_info) - return 0; - - /* - * After this point, we won't deliver anything asychronously - * to the message handler. We can unregister ourself. - */ - rv = ipmi_unregister_smi(ssif_info->intf); - if (rv) { - pr_err(PFX "Unable to unregister device: errno=%d\n", rv); - return rv; - } - ssif_info->intf = NULL; + struct ssif_info *ssif_info = send_info; device_remove_group(&ssif_info->client->dev, &ipmi_ssif_dev_attr_group); dev_set_drvdata(&ssif_info->client->dev, NULL); @@ -1259,6 +1215,30 @@ static int ssif_remove(struct i2c_client *client) kthread_stop(ssif_info->thread); } + /* + * No message can be outstanding now, we have removed the + * upper layer and it permitted us to do so. + */ + kfree(ssif_info); +} + +static int ssif_remove(struct i2c_client *client) +{ + struct ssif_info *ssif_info = i2c_get_clientdata(client); + struct ipmi_smi *intf; + struct ssif_addr_info *addr_info; + + if (!ssif_info) + return 0; + + /* + * After this point, we won't deliver anything asychronously + * to the message handler. We can unregister ourself. + */ + intf = ssif_info->intf; + ssif_info->intf = NULL; + ipmi_unregister_smi(intf); + list_for_each_entry(addr_info, &ssif_infos, link) { if (addr_info->client == client) { addr_info->client = NULL; @@ -1266,11 +1246,6 @@ static int ssif_remove(struct i2c_client *client) } } - /* - * No message can be outstanding now, we have removed the - * upper layer and it permitted us to do so. - */ - kfree(ssif_info); return 0; } @@ -1341,72 +1316,6 @@ static int ssif_detect(struct i2c_client *client, struct i2c_board_info *info) return rv; } -#ifdef CONFIG_IPMI_PROC_INTERFACE -static int smi_type_proc_show(struct seq_file *m, void *v) -{ - seq_puts(m, "ssif\n"); - - return 0; -} - -static int smi_type_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, smi_type_proc_show, inode->i_private); -} - -static const struct file_operations smi_type_proc_ops = { - .open = smi_type_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - -static int smi_stats_proc_show(struct seq_file *m, void *v) -{ - struct ssif_info *ssif_info = m->private; - - seq_printf(m, "sent_messages: %u\n", - ssif_get_stat(ssif_info, sent_messages)); - seq_printf(m, "sent_messages_parts: %u\n", - ssif_get_stat(ssif_info, sent_messages_parts)); - seq_printf(m, "send_retries: %u\n", - ssif_get_stat(ssif_info, send_retries)); - seq_printf(m, "send_errors: %u\n", - ssif_get_stat(ssif_info, send_errors)); - seq_printf(m, "received_messages: %u\n", - ssif_get_stat(ssif_info, received_messages)); - seq_printf(m, "received_message_parts: %u\n", - ssif_get_stat(ssif_info, received_message_parts)); - seq_printf(m, "receive_retries: %u\n", - ssif_get_stat(ssif_info, receive_retries)); - seq_printf(m, "receive_errors: %u\n", - ssif_get_stat(ssif_info, receive_errors)); - seq_printf(m, "flag_fetches: %u\n", - ssif_get_stat(ssif_info, flag_fetches)); - seq_printf(m, "hosed: %u\n", - ssif_get_stat(ssif_info, hosed)); - seq_printf(m, "events: %u\n", - ssif_get_stat(ssif_info, events)); - seq_printf(m, "watchdog_pretimeouts: %u\n", - ssif_get_stat(ssif_info, watchdog_pretimeouts)); - seq_printf(m, "alerts: %u\n", - ssif_get_stat(ssif_info, alerts)); - return 0; -} - -static int smi_stats_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, smi_stats_proc_show, PDE_DATA(inode)); -} - -static const struct file_operations smi_stats_proc_ops = { - .open = smi_stats_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; -#endif - static int strcmp_nospace(char *s1, char *s2) { while (*s1 && *s2) { @@ -1678,8 +1587,6 @@ static int ssif_probe(struct i2c_client *client, const struct i2c_device_id *id) } found: - ssif_info->intf_num = atomic_inc_return(&next_intf); - if (ssif_dbg_probe) { pr_info("ssif_probe: i2c_probe found device at i2c address %x\n", client->addr); @@ -1697,11 +1604,10 @@ static int ssif_probe(struct i2c_client *client, const struct i2c_device_id *id) ssif_info->handlers.owner = THIS_MODULE; ssif_info->handlers.start_processing = ssif_start_processing; + ssif_info->handlers.shutdown = shutdown_ssif; ssif_info->handlers.get_smi_info = get_smi_info; ssif_info->handlers.sender = sender; ssif_info->handlers.request_events = request_events; - ssif_info->handlers.inc_usecount = inc_usecount; - ssif_info->handlers.dec_usecount = dec_usecount; { unsigned int thread_num; @@ -1740,24 +1646,6 @@ static int ssif_probe(struct i2c_client *client, const struct i2c_device_id *id) goto out_remove_attr; } -#ifdef CONFIG_IPMI_PROC_INTERFACE - rv = ipmi_smi_add_proc_entry(ssif_info->intf, "type", - &smi_type_proc_ops, - ssif_info); - if (rv) { - pr_err(PFX "Unable to create proc entry: %d\n", rv); - goto out_err_unreg; - } - - rv = ipmi_smi_add_proc_entry(ssif_info->intf, "ssif_stats", - &smi_stats_proc_ops, - ssif_info); - if (rv) { - pr_err(PFX "Unable to create proc entry: %d\n", rv); - goto out_err_unreg; - } -#endif - out: if (rv) { /* @@ -1775,11 +1663,6 @@ static int ssif_probe(struct i2c_client *client, const struct i2c_device_id *id) kfree(resp); return rv; -#ifdef CONFIG_IPMI_PROC_INTERFACE -out_err_unreg: - ipmi_unregister_smi(ssif_info->intf); -#endif - out_remove_attr: device_remove_group(&ssif_info->client->dev, &ipmi_ssif_dev_attr_group); dev_set_drvdata(&ssif_info->client->dev, NULL); @@ -1874,7 +1757,8 @@ static unsigned short *ssif_address_list(void) list_for_each_entry(info, &ssif_infos, link) count++; - address_list = kzalloc(sizeof(*address_list) * (count + 1), GFP_KERNEL); + address_list = kcalloc(count + 1, sizeof(*address_list), + GFP_KERNEL); if (!address_list) return NULL; diff --git a/drivers/char/ipmi/ipmi_watchdog.c b/drivers/char/ipmi/ipmi_watchdog.c index 22bc287eac2d..ca1c5c5109f0 100644 --- a/drivers/char/ipmi/ipmi_watchdog.c +++ b/drivers/char/ipmi/ipmi_watchdog.c @@ -125,7 +125,7 @@ static DEFINE_MUTEX(ipmi_watchdog_mutex); static bool nowayout = WATCHDOG_NOWAYOUT; -static ipmi_user_t watchdog_user; +static struct ipmi_user *watchdog_user; static int watchdog_ifnum; /* Default the timeout to 10 seconds. */ @@ -153,7 +153,7 @@ static DEFINE_SPINLOCK(ipmi_read_lock); static char data_to_read; static DECLARE_WAIT_QUEUE_HEAD(read_q); static struct fasync_struct *fasync_q; -static char pretimeout_since_last_heartbeat; +static atomic_t pretimeout_since_last_heartbeat; static char expect_close; static int ifnum_to_use = -1; @@ -303,9 +303,6 @@ MODULE_PARM_DESC(nowayout, "Watchdog cannot be stopped once started " /* Default state of the timer. */ static unsigned char ipmi_watchdog_state = WDOG_TIMEOUT_NONE; -/* If shutting down via IPMI, we ignore the heartbeat. */ -static int ipmi_ignore_heartbeat; - /* Is someone using the watchdog? Only one user is allowed. */ static unsigned long ipmi_wdog_open; @@ -329,35 +326,33 @@ static int testing_nmi; static int nmi_handler_registered; #endif -static int ipmi_heartbeat(void); +static int __ipmi_heartbeat(void); /* - * We use a mutex to make sure that only one thing can send a set - * timeout at one time, because we only have one copy of the data. - * The mutex is claimed when the set_timeout is sent and freed - * when both messages are free. + * We use a mutex to make sure that only one thing can send a set a + * message at one time. The mutex is claimed when a message is sent + * and freed when both the send and receive messages are free. */ -static atomic_t set_timeout_tofree = ATOMIC_INIT(0); -static DEFINE_MUTEX(set_timeout_lock); -static DECLARE_COMPLETION(set_timeout_wait); -static void set_timeout_free_smi(struct ipmi_smi_msg *msg) +static atomic_t msg_tofree = ATOMIC_INIT(0); +static DECLARE_COMPLETION(msg_wait); +static void msg_free_smi(struct ipmi_smi_msg *msg) { - if (atomic_dec_and_test(&set_timeout_tofree)) - complete(&set_timeout_wait); + if (atomic_dec_and_test(&msg_tofree)) + complete(&msg_wait); } -static void set_timeout_free_recv(struct ipmi_recv_msg *msg) +static void msg_free_recv(struct ipmi_recv_msg *msg) { - if (atomic_dec_and_test(&set_timeout_tofree)) - complete(&set_timeout_wait); + if (atomic_dec_and_test(&msg_tofree)) + complete(&msg_wait); } -static struct ipmi_smi_msg set_timeout_smi_msg = { - .done = set_timeout_free_smi +static struct ipmi_smi_msg smi_msg = { + .done = msg_free_smi }; -static struct ipmi_recv_msg set_timeout_recv_msg = { - .done = set_timeout_free_recv +static struct ipmi_recv_msg recv_msg = { + .done = msg_free_recv }; -static int i_ipmi_set_timeout(struct ipmi_smi_msg *smi_msg, +static int __ipmi_set_timeout(struct ipmi_smi_msg *smi_msg, struct ipmi_recv_msg *recv_msg, int *send_heartbeat_now) { @@ -368,9 +363,6 @@ static int i_ipmi_set_timeout(struct ipmi_smi_msg *smi_msg, int hbnow = 0; - /* These can be cleared as we are setting the timeout. */ - pretimeout_since_last_heartbeat = 0; - data[0] = 0; WDOG_SET_TIMER_USE(data[0], WDOG_TIMER_USE_SMS_OS); @@ -414,46 +406,48 @@ static int i_ipmi_set_timeout(struct ipmi_smi_msg *smi_msg, smi_msg, recv_msg, 1); - if (rv) { - printk(KERN_WARNING PFX "set timeout error: %d\n", - rv); - } - - if (send_heartbeat_now) - *send_heartbeat_now = hbnow; + if (rv) + pr_warn(PFX "set timeout error: %d\n", rv); + else if (send_heartbeat_now) + *send_heartbeat_now = hbnow; return rv; } -static int ipmi_set_timeout(int do_heartbeat) +static int _ipmi_set_timeout(int do_heartbeat) { int send_heartbeat_now; int rv; + if (!watchdog_user) + return -ENODEV; - /* We can only send one of these at a time. */ - mutex_lock(&set_timeout_lock); - - atomic_set(&set_timeout_tofree, 2); + atomic_set(&msg_tofree, 2); - rv = i_ipmi_set_timeout(&set_timeout_smi_msg, - &set_timeout_recv_msg, + rv = __ipmi_set_timeout(&smi_msg, + &recv_msg, &send_heartbeat_now); - if (rv) { - mutex_unlock(&set_timeout_lock); - goto out; - } - - wait_for_completion(&set_timeout_wait); + if (rv) + return rv; - mutex_unlock(&set_timeout_lock); + wait_for_completion(&msg_wait); if ((do_heartbeat == IPMI_SET_TIMEOUT_FORCE_HB) - || ((send_heartbeat_now) - && (do_heartbeat == IPMI_SET_TIMEOUT_HB_IF_NECESSARY))) - rv = ipmi_heartbeat(); + || ((send_heartbeat_now) + && (do_heartbeat == IPMI_SET_TIMEOUT_HB_IF_NECESSARY))) + rv = __ipmi_heartbeat(); + + return rv; +} + +static int ipmi_set_timeout(int do_heartbeat) +{ + int rv; + + mutex_lock(&ipmi_watchdog_mutex); + rv = _ipmi_set_timeout(do_heartbeat); + mutex_unlock(&ipmi_watchdog_mutex); -out: return rv; } @@ -531,13 +525,12 @@ static void panic_halt_ipmi_set_timeout(void) while (atomic_read(&panic_done_count) != 0) ipmi_poll_interface(watchdog_user); atomic_add(1, &panic_done_count); - rv = i_ipmi_set_timeout(&panic_halt_smi_msg, + rv = __ipmi_set_timeout(&panic_halt_smi_msg, &panic_halt_recv_msg, &send_heartbeat_now); if (rv) { atomic_sub(1, &panic_done_count); - printk(KERN_WARNING PFX - "Unable to extend the watchdog timeout."); + pr_warn(PFX "Unable to extend the watchdog timeout."); } else { if (send_heartbeat_now) panic_halt_ipmi_heartbeat(); @@ -546,69 +539,22 @@ static void panic_halt_ipmi_set_timeout(void) ipmi_poll_interface(watchdog_user); } -/* - * We use a mutex to make sure that only one thing can send a - * heartbeat at one time, because we only have one copy of the data. - * The semaphore is claimed when the set_timeout is sent and freed - * when both messages are free. - */ -static atomic_t heartbeat_tofree = ATOMIC_INIT(0); -static DEFINE_MUTEX(heartbeat_lock); -static DECLARE_COMPLETION(heartbeat_wait); -static void heartbeat_free_smi(struct ipmi_smi_msg *msg) -{ - if (atomic_dec_and_test(&heartbeat_tofree)) - complete(&heartbeat_wait); -} -static void heartbeat_free_recv(struct ipmi_recv_msg *msg) -{ - if (atomic_dec_and_test(&heartbeat_tofree)) - complete(&heartbeat_wait); -} -static struct ipmi_smi_msg heartbeat_smi_msg = { - .done = heartbeat_free_smi -}; -static struct ipmi_recv_msg heartbeat_recv_msg = { - .done = heartbeat_free_recv -}; - -static int ipmi_heartbeat(void) +static int __ipmi_heartbeat(void) { - struct kernel_ipmi_msg msg; - int rv; + struct kernel_ipmi_msg msg; + int rv; struct ipmi_system_interface_addr addr; - int timeout_retries = 0; - - if (ipmi_ignore_heartbeat) - return 0; - - if (ipmi_start_timer_on_heartbeat) { - ipmi_start_timer_on_heartbeat = 0; - ipmi_watchdog_state = action_val; - return ipmi_set_timeout(IPMI_SET_TIMEOUT_FORCE_HB); - } else if (pretimeout_since_last_heartbeat) { - /* - * A pretimeout occurred, make sure we set the timeout. - * We don't want to set the action, though, we want to - * leave that alone (thus it can't be combined with the - * above operation. - */ - return ipmi_set_timeout(IPMI_SET_TIMEOUT_HB_IF_NECESSARY); - } - - mutex_lock(&heartbeat_lock); + int timeout_retries = 0; restart: - atomic_set(&heartbeat_tofree, 2); - /* * Don't reset the timer if we have the timer turned off, that * re-enables the watchdog. */ - if (ipmi_watchdog_state == WDOG_TIMEOUT_NONE) { - mutex_unlock(&heartbeat_lock); + if (ipmi_watchdog_state == WDOG_TIMEOUT_NONE) return 0; - } + + atomic_set(&msg_tofree, 2); addr.addr_type = IPMI_SYSTEM_INTERFACE_ADDR_TYPE; addr.channel = IPMI_BMC_CHANNEL; @@ -623,26 +569,23 @@ restart: 0, &msg, NULL, - &heartbeat_smi_msg, - &heartbeat_recv_msg, + &smi_msg, + &recv_msg, 1); if (rv) { - mutex_unlock(&heartbeat_lock); - printk(KERN_WARNING PFX "heartbeat failure: %d\n", - rv); + pr_warn(PFX "heartbeat send failure: %d\n", rv); return rv; } /* Wait for the heartbeat to be sent. */ - wait_for_completion(&heartbeat_wait); + wait_for_completion(&msg_wait); - if (heartbeat_recv_msg.msg.data[0] == IPMI_WDOG_TIMER_NOT_INIT_RESP) { + if (recv_msg.msg.data[0] == IPMI_WDOG_TIMER_NOT_INIT_RESP) { timeout_retries++; if (timeout_retries > 3) { - printk(KERN_ERR PFX ": Unable to restore the IPMI" - " watchdog's settings, giving up.\n"); + pr_err(PFX ": Unable to restore the IPMI watchdog's settings, giving up.\n"); rv = -EIO; - goto out_unlock; + goto out; } /* @@ -651,18 +594,17 @@ restart: * to restore the timer's info. Note that we still hold * the heartbeat lock, to keep a heartbeat from happening * in this process, so must say no heartbeat to avoid a - * deadlock on this mutex. + * deadlock on this mutex */ - rv = ipmi_set_timeout(IPMI_SET_TIMEOUT_NO_HB); + rv = _ipmi_set_timeout(IPMI_SET_TIMEOUT_NO_HB); if (rv) { - printk(KERN_ERR PFX ": Unable to send the command to" - " set the watchdog's settings, giving up.\n"); - goto out_unlock; + pr_err(PFX ": Unable to send the command to set the watchdog's settings, giving up.\n"); + goto out; } - /* We might need a new heartbeat, so do it now */ + /* Might need a heartbeat send, go ahead and do it. */ goto restart; - } else if (heartbeat_recv_msg.msg.data[0] != 0) { + } else if (recv_msg.msg.data[0] != 0) { /* * Got an error in the heartbeat response. It was already * reported in ipmi_wdog_msg_handler, but we should return @@ -671,8 +613,43 @@ restart: rv = -EINVAL; } -out_unlock: - mutex_unlock(&heartbeat_lock); +out: + return rv; +} + +static int _ipmi_heartbeat(void) +{ + int rv; + + if (!watchdog_user) + return -ENODEV; + + if (ipmi_start_timer_on_heartbeat) { + ipmi_start_timer_on_heartbeat = 0; + ipmi_watchdog_state = action_val; + rv = _ipmi_set_timeout(IPMI_SET_TIMEOUT_FORCE_HB); + } else if (atomic_cmpxchg(&pretimeout_since_last_heartbeat, 1, 0)) { + /* + * A pretimeout occurred, make sure we set the timeout. + * We don't want to set the action, though, we want to + * leave that alone (thus it can't be combined with the + * above operation. + */ + rv = _ipmi_set_timeout(IPMI_SET_TIMEOUT_HB_IF_NECESSARY); + } else { + rv = __ipmi_heartbeat(); + } + + return rv; +} + +static int ipmi_heartbeat(void) +{ + int rv; + + mutex_lock(&ipmi_watchdog_mutex); + rv = _ipmi_heartbeat(); + mutex_unlock(&ipmi_watchdog_mutex); return rv; } @@ -700,7 +677,7 @@ static int ipmi_ioctl(struct file *file, if (i) return -EFAULT; timeout = val; - return ipmi_set_timeout(IPMI_SET_TIMEOUT_HB_IF_NECESSARY); + return _ipmi_set_timeout(IPMI_SET_TIMEOUT_HB_IF_NECESSARY); case WDIOC_GETTIMEOUT: i = copy_to_user(argp, &timeout, sizeof(timeout)); @@ -713,7 +690,7 @@ static int ipmi_ioctl(struct file *file, if (i) return -EFAULT; pretimeout = val; - return ipmi_set_timeout(IPMI_SET_TIMEOUT_HB_IF_NECESSARY); + return _ipmi_set_timeout(IPMI_SET_TIMEOUT_HB_IF_NECESSARY); case WDIOC_GETPRETIMEOUT: i = copy_to_user(argp, &pretimeout, sizeof(pretimeout)); @@ -722,7 +699,7 @@ static int ipmi_ioctl(struct file *file, return 0; case WDIOC_KEEPALIVE: - return ipmi_heartbeat(); + return _ipmi_heartbeat(); case WDIOC_SETOPTIONS: i = copy_from_user(&val, argp, sizeof(int)); @@ -730,13 +707,13 @@ static int ipmi_ioctl(struct file *file, return -EFAULT; if (val & WDIOS_DISABLECARD) { ipmi_watchdog_state = WDOG_TIMEOUT_NONE; - ipmi_set_timeout(IPMI_SET_TIMEOUT_NO_HB); + _ipmi_set_timeout(IPMI_SET_TIMEOUT_NO_HB); ipmi_start_timer_on_heartbeat = 0; } if (val & WDIOS_ENABLECARD) { ipmi_watchdog_state = action_val; - ipmi_set_timeout(IPMI_SET_TIMEOUT_FORCE_HB); + _ipmi_set_timeout(IPMI_SET_TIMEOUT_FORCE_HB); } return 0; @@ -810,7 +787,7 @@ static ssize_t ipmi_read(struct file *file, * Reading returns if the pretimeout has gone off, and it only does * it once per pretimeout. */ - spin_lock(&ipmi_read_lock); + spin_lock_irq(&ipmi_read_lock); if (!data_to_read) { if (file->f_flags & O_NONBLOCK) { rv = -EAGAIN; @@ -821,9 +798,9 @@ static ssize_t ipmi_read(struct file *file, add_wait_queue(&read_q, &wait); while (!data_to_read) { set_current_state(TASK_INTERRUPTIBLE); - spin_unlock(&ipmi_read_lock); + spin_unlock_irq(&ipmi_read_lock); schedule(); - spin_lock(&ipmi_read_lock); + spin_lock_irq(&ipmi_read_lock); } remove_wait_queue(&read_q, &wait); @@ -835,7 +812,7 @@ static ssize_t ipmi_read(struct file *file, data_to_read = 0; out: - spin_unlock(&ipmi_read_lock); + spin_unlock_irq(&ipmi_read_lock); if (rv == 0) { if (copy_to_user(buf, &data_to_read, 1)) @@ -873,10 +850,10 @@ static __poll_t ipmi_poll(struct file *file, poll_table *wait) poll_wait(file, &read_q, wait); - spin_lock(&ipmi_read_lock); + spin_lock_irq(&ipmi_read_lock); if (data_to_read) mask |= (EPOLLIN | EPOLLRDNORM); - spin_unlock(&ipmi_read_lock); + spin_unlock_irq(&ipmi_read_lock); return mask; } @@ -894,11 +871,13 @@ static int ipmi_close(struct inode *ino, struct file *filep) { if (iminor(ino) == WATCHDOG_MINOR) { if (expect_close == 42) { + mutex_lock(&ipmi_watchdog_mutex); ipmi_watchdog_state = WDOG_TIMEOUT_NONE; - ipmi_set_timeout(IPMI_SET_TIMEOUT_NO_HB); + _ipmi_set_timeout(IPMI_SET_TIMEOUT_NO_HB); + mutex_unlock(&ipmi_watchdog_mutex); } else { - printk(KERN_CRIT PFX - "Unexpected close, not stopping watchdog!\n"); + pr_crit(PFX + "Unexpected close, not stopping watchdog!\n"); ipmi_heartbeat(); } clear_bit(0, &ipmi_wdog_open); @@ -932,11 +911,9 @@ static void ipmi_wdog_msg_handler(struct ipmi_recv_msg *msg, { if (msg->msg.cmd == IPMI_WDOG_RESET_TIMER && msg->msg.data[0] == IPMI_WDOG_TIMER_NOT_INIT_RESP) - printk(KERN_INFO PFX "response: The IPMI controller appears" - " to have been reset, will attempt to reinitialize" - " the watchdog timer\n"); + pr_info(PFX "response: The IPMI controller appears to have been reset, will attempt to reinitialize the watchdog timer\n"); else if (msg->msg.data[0] != 0) - printk(KERN_ERR PFX "response: Error %x on cmd %x\n", + pr_err(PFX "response: Error %x on cmd %x\n", msg->msg.data[0], msg->msg.cmd); @@ -950,12 +927,13 @@ static void ipmi_wdog_pretimeout_handler(void *handler_data) if (atomic_inc_and_test(&preop_panic_excl)) panic("Watchdog pre-timeout"); } else if (preop_val == WDOG_PREOP_GIVE_DATA) { - spin_lock(&ipmi_read_lock); + unsigned long flags; + + spin_lock_irqsave(&ipmi_read_lock, flags); data_to_read = 1; wake_up_interruptible(&read_q); kill_fasync(&fasync_q, SIGIO, POLL_IN); - - spin_unlock(&ipmi_read_lock); + spin_unlock_irqrestore(&ipmi_read_lock, flags); } } @@ -963,12 +941,34 @@ static void ipmi_wdog_pretimeout_handler(void *handler_data) * On some machines, the heartbeat will give an error and not * work unless we re-enable the timer. So do so. */ - pretimeout_since_last_heartbeat = 1; + atomic_set(&pretimeout_since_last_heartbeat, 1); +} + +static void ipmi_wdog_panic_handler(void *user_data) +{ + static int panic_event_handled; + + /* + * On a panic, if we have a panic timeout, make sure to extend + * the watchdog timer to a reasonable value to complete the + * panic, if the watchdog timer is running. Plus the + * pretimeout is meaningless at panic time. + */ + if (watchdog_user && !panic_event_handled && + ipmi_watchdog_state != WDOG_TIMEOUT_NONE) { + /* Make sure we do this only once. */ + panic_event_handled = 1; + + timeout = panic_wdt_timeout; + pretimeout = 0; + panic_halt_ipmi_set_timeout(); + } } static const struct ipmi_user_hndl ipmi_hndlrs = { .ipmi_recv_hndl = ipmi_wdog_msg_handler, - .ipmi_watchdog_pretimeout = ipmi_wdog_pretimeout_handler + .ipmi_watchdog_pretimeout = ipmi_wdog_pretimeout_handler, + .ipmi_panic_handler = ipmi_wdog_panic_handler }; static void ipmi_register_watchdog(int ipmi_intf) @@ -985,7 +985,7 @@ static void ipmi_register_watchdog(int ipmi_intf) rv = ipmi_create_user(ipmi_intf, &ipmi_hndlrs, NULL, &watchdog_user); if (rv < 0) { - printk(KERN_CRIT PFX "Unable to register with ipmi\n"); + pr_crit(PFX "Unable to register with ipmi\n"); goto out; } @@ -1002,7 +1002,7 @@ static void ipmi_register_watchdog(int ipmi_intf) if (rv < 0) { ipmi_destroy_user(watchdog_user); watchdog_user = NULL; - printk(KERN_CRIT PFX "Unable to register misc device\n"); + pr_crit(PFX "Unable to register misc device\n"); } #ifdef HAVE_DIE_NMI @@ -1024,9 +1024,8 @@ static void ipmi_register_watchdog(int ipmi_intf) rv = ipmi_set_timeout(IPMI_SET_TIMEOUT_FORCE_HB); if (rv) { - printk(KERN_WARNING PFX "Error starting timer to" - " test NMI: 0x%x. The NMI pretimeout will" - " likely not work\n", rv); + pr_warn(PFX "Error starting timer to test NMI: 0x%x. The NMI pretimeout will likely not work\n", + rv); rv = 0; goto out_restore; } @@ -1034,9 +1033,7 @@ static void ipmi_register_watchdog(int ipmi_intf) msleep(1500); if (testing_nmi != 2) { - printk(KERN_WARNING PFX "IPMI NMI didn't seem to" - " occur. The NMI pretimeout will" - " likely not work\n"); + pr_warn(PFX "IPMI NMI didn't seem to occur. The NMI pretimeout will likely not work\n"); } out_restore: testing_nmi = 0; @@ -1052,7 +1049,7 @@ static void ipmi_register_watchdog(int ipmi_intf) start_now = 0; /* Disable this function after first startup. */ ipmi_watchdog_state = action_val; ipmi_set_timeout(IPMI_SET_TIMEOUT_FORCE_HB); - printk(KERN_INFO PFX "Starting now!\n"); + pr_info(PFX "Starting now!\n"); } else { /* Stop the timer now. */ ipmi_watchdog_state = WDOG_TIMEOUT_NONE; @@ -1063,34 +1060,38 @@ static void ipmi_register_watchdog(int ipmi_intf) static void ipmi_unregister_watchdog(int ipmi_intf) { int rv; + struct ipmi_user *loc_user = watchdog_user; - if (!watchdog_user) - goto out; + if (!loc_user) + return; if (watchdog_ifnum != ipmi_intf) - goto out; + return; /* Make sure no one can call us any more. */ misc_deregister(&ipmi_wdog_miscdev); + watchdog_user = NULL; + /* * Wait to make sure the message makes it out. The lower layer has * pointers to our buffers, we want to make sure they are done before * we release our memory. */ - while (atomic_read(&set_timeout_tofree)) - schedule_timeout_uninterruptible(1); + while (atomic_read(&msg_tofree)) + msg_free_smi(NULL); + + mutex_lock(&ipmi_watchdog_mutex); /* Disconnect from IPMI. */ - rv = ipmi_destroy_user(watchdog_user); - if (rv) { - printk(KERN_WARNING PFX "error unlinking from IPMI: %d\n", - rv); - } - watchdog_user = NULL; + rv = ipmi_destroy_user(loc_user); + if (rv) + pr_warn(PFX "error unlinking from IPMI: %d\n", rv); - out: - return; + /* If it comes back, restart it properly. */ + ipmi_start_timer_on_heartbeat = 1; + + mutex_unlock(&ipmi_watchdog_mutex); } #ifdef HAVE_DIE_NMI @@ -1124,7 +1125,7 @@ ipmi_nmi(unsigned int val, struct pt_regs *regs) /* On some machines, the heartbeat will give an error and not work unless we re-enable the timer. So do so. */ - pretimeout_since_last_heartbeat = 1; + atomic_set(&pretimeout_since_last_heartbeat, 1); if (atomic_inc_and_test(&preop_panic_excl)) nmi_panic(regs, PFX "pre-timeout"); } @@ -1167,36 +1168,6 @@ static struct notifier_block wdog_reboot_notifier = { .priority = 0 }; -static int wdog_panic_handler(struct notifier_block *this, - unsigned long event, - void *unused) -{ - static int panic_event_handled; - - /* On a panic, if we have a panic timeout, make sure to extend - the watchdog timer to a reasonable value to complete the - panic, if the watchdog timer is running. Plus the - pretimeout is meaningless at panic time. */ - if (watchdog_user && !panic_event_handled && - ipmi_watchdog_state != WDOG_TIMEOUT_NONE) { - /* Make sure we do this only once. */ - panic_event_handled = 1; - - timeout = panic_wdt_timeout; - pretimeout = 0; - panic_halt_ipmi_set_timeout(); - } - - return NOTIFY_OK; -} - -static struct notifier_block wdog_panic_notifier = { - .notifier_call = wdog_panic_handler, - .next = NULL, - .priority = 150 /* priority: INT_MAX >= x >= 0 */ -}; - - static void ipmi_new_smi(int if_num, struct device *device) { ipmi_register_watchdog(if_num); @@ -1288,9 +1259,7 @@ static void check_parms(void) if (preaction_val == WDOG_PRETIMEOUT_NMI) { do_nmi = 1; if (preop_val == WDOG_PREOP_GIVE_DATA) { - printk(KERN_WARNING PFX "Pretimeout op is to give data" - " but NMI pretimeout is enabled, setting" - " pretimeout op to none\n"); + pr_warn(PFX "Pretimeout op is to give data but NMI pretimeout is enabled, setting pretimeout op to none\n"); preop_op("preop_none", NULL); do_nmi = 0; } @@ -1299,8 +1268,7 @@ static void check_parms(void) rv = register_nmi_handler(NMI_UNKNOWN, ipmi_nmi, 0, "ipmi"); if (rv) { - printk(KERN_WARNING PFX - "Can't register nmi handler\n"); + pr_warn(PFX "Can't register nmi handler\n"); return; } else nmi_handler_registered = 1; @@ -1317,27 +1285,24 @@ static int __init ipmi_wdog_init(void) if (action_op(action, NULL)) { action_op("reset", NULL); - printk(KERN_INFO PFX "Unknown action '%s', defaulting to" - " reset\n", action); + pr_info(PFX "Unknown action '%s', defaulting to reset\n", + action); } if (preaction_op(preaction, NULL)) { preaction_op("pre_none", NULL); - printk(KERN_INFO PFX "Unknown preaction '%s', defaulting to" - " none\n", preaction); + pr_info(PFX "Unknown preaction '%s', defaulting to none\n", + preaction); } if (preop_op(preop, NULL)) { preop_op("preop_none", NULL); - printk(KERN_INFO PFX "Unknown preop '%s', defaulting to" - " none\n", preop); + pr_info(PFX "Unknown preop '%s', defaulting to none\n", preop); } check_parms(); register_reboot_notifier(&wdog_reboot_notifier); - atomic_notifier_chain_register(&panic_notifier_list, - &wdog_panic_notifier); rv = ipmi_smi_watcher_register(&smi_watcher); if (rv) { @@ -1345,14 +1310,12 @@ static int __init ipmi_wdog_init(void) if (nmi_handler_registered) unregister_nmi_handler(NMI_UNKNOWN, "ipmi"); #endif - atomic_notifier_chain_unregister(&panic_notifier_list, - &wdog_panic_notifier); unregister_reboot_notifier(&wdog_reboot_notifier); - printk(KERN_WARNING PFX "can't register smi watcher\n"); + pr_warn(PFX "can't register smi watcher\n"); return rv; } - printk(KERN_INFO PFX "driver initialized\n"); + pr_info(PFX "driver initialized\n"); return 0; } @@ -1367,8 +1330,6 @@ static void __exit ipmi_wdog_exit(void) unregister_nmi_handler(NMI_UNKNOWN, "ipmi"); #endif - atomic_notifier_chain_unregister(&panic_notifier_list, - &wdog_panic_notifier); unregister_reboot_notifier(&wdog_reboot_notifier); } module_exit(ipmi_wdog_exit); diff --git a/drivers/char/ipmi/kcs_bmc_npcm7xx.c b/drivers/char/ipmi/kcs_bmc_npcm7xx.c new file mode 100644 index 000000000000..722f7391fe1f --- /dev/null +++ b/drivers/char/ipmi/kcs_bmc_npcm7xx.c @@ -0,0 +1,215 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2018, Nuvoton Corporation. + * Copyright (c) 2018, Intel Corporation. + */ + +#define pr_fmt(fmt) "nuvoton-kcs-bmc: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "kcs_bmc.h" + +#define DEVICE_NAME "npcm-kcs-bmc" +#define KCS_CHANNEL_MAX 3 + +#define KCS1ST 0x0C +#define KCS2ST 0x1E +#define KCS3ST 0x30 + +#define KCS1DO 0x0E +#define KCS2DO 0x20 +#define KCS3DO 0x32 + +#define KCS1DI 0x10 +#define KCS2DI 0x22 +#define KCS3DI 0x34 + +#define KCS1CTL 0x18 +#define KCS2CTL 0x2A +#define KCS3CTL 0x3C +#define KCS_CTL_IBFIE BIT(0) + +#define KCS1IE 0x1C +#define KCS2IE 0x2E +#define KCS3IE 0x40 +#define KCS_IE_IRQE BIT(0) +#define KCS_IE_HIRQE BIT(3) + +/* + * 7.2.4 Core KCS Registers + * Registers in this module are 8 bits. An 8-bit register must be accessed + * by an 8-bit read or write. + * + * sts: KCS Channel n Status Register (KCSnST). + * dob: KCS Channel n Data Out Buffer Register (KCSnDO). + * dib: KCS Channel n Data In Buffer Register (KCSnDI). + * ctl: KCS Channel n Control Register (KCSnCTL). + * ie : KCS Channel n Interrupt Enable Register (KCSnIE). + */ +struct npcm7xx_kcs_reg { + u32 sts; + u32 dob; + u32 dib; + u32 ctl; + u32 ie; +}; + +struct npcm7xx_kcs_bmc { + struct regmap *map; + + const struct npcm7xx_kcs_reg *reg; +}; + +static const struct npcm7xx_kcs_reg npcm7xx_kcs_reg_tbl[KCS_CHANNEL_MAX] = { + { .sts = KCS1ST, .dob = KCS1DO, .dib = KCS1DI, .ctl = KCS1CTL, .ie = KCS1IE }, + { .sts = KCS2ST, .dob = KCS2DO, .dib = KCS2DI, .ctl = KCS2CTL, .ie = KCS2IE }, + { .sts = KCS3ST, .dob = KCS3DO, .dib = KCS3DI, .ctl = KCS3CTL, .ie = KCS3IE }, +}; + +static u8 npcm7xx_kcs_inb(struct kcs_bmc *kcs_bmc, u32 reg) +{ + struct npcm7xx_kcs_bmc *priv = kcs_bmc_priv(kcs_bmc); + u32 val = 0; + int rc; + + rc = regmap_read(priv->map, reg, &val); + WARN(rc != 0, "regmap_read() failed: %d\n", rc); + + return rc == 0 ? (u8)val : 0; +} + +static void npcm7xx_kcs_outb(struct kcs_bmc *kcs_bmc, u32 reg, u8 data) +{ + struct npcm7xx_kcs_bmc *priv = kcs_bmc_priv(kcs_bmc); + int rc; + + rc = regmap_write(priv->map, reg, data); + WARN(rc != 0, "regmap_write() failed: %d\n", rc); +} + +static void npcm7xx_kcs_enable_channel(struct kcs_bmc *kcs_bmc, bool enable) +{ + struct npcm7xx_kcs_bmc *priv = kcs_bmc_priv(kcs_bmc); + + regmap_update_bits(priv->map, priv->reg->ctl, KCS_CTL_IBFIE, + enable ? KCS_CTL_IBFIE : 0); + + regmap_update_bits(priv->map, priv->reg->ie, KCS_IE_IRQE | KCS_IE_HIRQE, + enable ? KCS_IE_IRQE | KCS_IE_HIRQE : 0); +} + +static irqreturn_t npcm7xx_kcs_irq(int irq, void *arg) +{ + struct kcs_bmc *kcs_bmc = arg; + + if (!kcs_bmc_handle_event(kcs_bmc)) + return IRQ_HANDLED; + + return IRQ_NONE; +} + +static int npcm7xx_kcs_config_irq(struct kcs_bmc *kcs_bmc, + struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + int irq; + + irq = platform_get_irq(pdev, 0); + if (irq < 0) + return irq; + + return devm_request_irq(dev, irq, npcm7xx_kcs_irq, IRQF_SHARED, + dev_name(dev), kcs_bmc); +} + +static int npcm7xx_kcs_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct npcm7xx_kcs_bmc *priv; + struct kcs_bmc *kcs_bmc; + u32 chan; + int rc; + + rc = of_property_read_u32(dev->of_node, "kcs_chan", &chan); + if (rc != 0 || chan == 0 || chan > KCS_CHANNEL_MAX) { + dev_err(dev, "no valid 'kcs_chan' configured\n"); + return -ENODEV; + } + + kcs_bmc = kcs_bmc_alloc(dev, sizeof(*priv), chan); + if (!kcs_bmc) + return -ENOMEM; + + priv = kcs_bmc_priv(kcs_bmc); + priv->map = syscon_node_to_regmap(dev->parent->of_node); + if (IS_ERR(priv->map)) { + dev_err(dev, "Couldn't get regmap\n"); + return -ENODEV; + } + priv->reg = &npcm7xx_kcs_reg_tbl[chan - 1]; + + kcs_bmc->ioreg.idr = priv->reg->dib; + kcs_bmc->ioreg.odr = priv->reg->dob; + kcs_bmc->ioreg.str = priv->reg->sts; + kcs_bmc->io_inputb = npcm7xx_kcs_inb; + kcs_bmc->io_outputb = npcm7xx_kcs_outb; + + dev_set_drvdata(dev, kcs_bmc); + + npcm7xx_kcs_enable_channel(kcs_bmc, true); + rc = npcm7xx_kcs_config_irq(kcs_bmc, pdev); + if (rc) + return rc; + + rc = misc_register(&kcs_bmc->miscdev); + if (rc) { + dev_err(dev, "Unable to register device\n"); + return rc; + } + + pr_info("channel=%u idr=0x%x odr=0x%x str=0x%x\n", + chan, + kcs_bmc->ioreg.idr, kcs_bmc->ioreg.odr, kcs_bmc->ioreg.str); + + return 0; +} + +static int npcm7xx_kcs_remove(struct platform_device *pdev) +{ + struct kcs_bmc *kcs_bmc = dev_get_drvdata(&pdev->dev); + + misc_deregister(&kcs_bmc->miscdev); + + return 0; +} + +static const struct of_device_id npcm_kcs_bmc_match[] = { + { .compatible = "nuvoton,npcm750-kcs-bmc" }, + { } +}; +MODULE_DEVICE_TABLE(of, npcm_kcs_bmc_match); + +static struct platform_driver npcm_kcs_bmc_driver = { + .driver = { + .name = DEVICE_NAME, + .of_match_table = npcm_kcs_bmc_match, + }, + .probe = npcm7xx_kcs_probe, + .remove = npcm7xx_kcs_remove, +}; +module_platform_driver(npcm_kcs_bmc_driver); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Avi Fishman "); +MODULE_AUTHOR("Haiyue Wang "); +MODULE_DESCRIPTION("NPCM7xx device interface to the KCS BMC device"); diff --git a/drivers/char/misc.c b/drivers/char/misc.c index 1bb9e7cc82e3..53cfe574d8d4 100644 --- a/drivers/char/misc.c +++ b/drivers/char/misc.c @@ -95,19 +95,6 @@ static const struct seq_operations misc_seq_ops = { .stop = misc_seq_stop, .show = misc_seq_show, }; - -static int misc_seq_open(struct inode *inode, struct file *file) -{ - return seq_open(file, &misc_seq_ops); -} - -static const struct file_operations misc_proc_fops = { - .owner = THIS_MODULE, - .open = misc_seq_open, - .read = seq_read, - .llseek = seq_lseek, - .release = seq_release, -}; #endif static int misc_open(struct inode *inode, struct file *file) @@ -282,7 +269,7 @@ static int __init misc_init(void) int err; struct proc_dir_entry *ret; - ret = proc_create("misc", 0, NULL, &misc_proc_fops); + ret = proc_create_seq("misc", 0, NULL, &misc_seq_ops); misc_class = class_create(THIS_MODULE, "misc"); err = PTR_ERR(misc_class); if (IS_ERR(misc_class)) diff --git a/drivers/char/mspec.c b/drivers/char/mspec.c index 7b75669d3670..058876b55b09 100644 --- a/drivers/char/mspec.c +++ b/drivers/char/mspec.c @@ -191,7 +191,7 @@ mspec_close(struct vm_area_struct *vma) * * Creates a mspec page and maps it to user space. */ -static int +static vm_fault_t mspec_fault(struct vm_fault *vmf) { unsigned long paddr, maddr; @@ -223,14 +223,7 @@ mspec_fault(struct vm_fault *vmf) pfn = paddr >> PAGE_SHIFT; - /* - * vm_insert_pfn can fail with -EBUSY, but in that case it will - * be because another thread has installed the pte first, so it - * is no problem. - */ - vm_insert_pfn(vmf->vma, vmf->address, pfn); - - return VM_FAULT_NOPAGE; + return vmf_insert_pfn(vmf->vma, vmf->address, pfn); } static const struct vm_operations_struct mspec_vm_ops = { diff --git a/drivers/char/nvram.c b/drivers/char/nvram.c index 678fa97e41fb..25264d65e716 100644 --- a/drivers/char/nvram.c +++ b/drivers/char/nvram.c @@ -389,22 +389,9 @@ static int nvram_proc_read(struct seq_file *seq, void *offset) return 0; } -static int nvram_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, nvram_proc_read, NULL); -} - -static const struct file_operations nvram_proc_fops = { - .owner = THIS_MODULE, - .open = nvram_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int nvram_add_proc_fs(void) { - if (!proc_create("driver/nvram", 0, NULL, &nvram_proc_fops)) + if (!proc_create_single("driver/nvram", 0, NULL, nvram_proc_read)) return -ENOMEM; return 0; } diff --git a/drivers/char/pcmcia/synclink_cs.c b/drivers/char/pcmcia/synclink_cs.c index aa502e9fb7fa..66b04194aa9f 100644 --- a/drivers/char/pcmcia/synclink_cs.c +++ b/drivers/char/pcmcia/synclink_cs.c @@ -2616,19 +2616,6 @@ static int mgslpc_proc_show(struct seq_file *m, void *v) return 0; } -static int mgslpc_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, mgslpc_proc_show, NULL); -} - -static const struct file_operations mgslpc_proc_fops = { - .owner = THIS_MODULE, - .open = mgslpc_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - static int rx_alloc_buffers(MGSLPC_INFO *info) { /* each buffer has header and data */ @@ -2815,7 +2802,7 @@ static const struct tty_operations mgslpc_ops = { .tiocmget = tiocmget, .tiocmset = tiocmset, .get_icount = mgslpc_get_icount, - .proc_fops = &mgslpc_proc_fops, + .proc_show = mgslpc_proc_show, }; static int __init synclink_cs_init(void) diff --git a/drivers/char/random.c b/drivers/char/random.c index cd888d4ee605..a8fb0020ba5c 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -402,8 +402,7 @@ static struct poolinfo { /* * Static global variables */ -static DECLARE_WAIT_QUEUE_HEAD(random_read_wait); -static DECLARE_WAIT_QUEUE_HEAD(random_write_wait); +static DECLARE_WAIT_QUEUE_HEAD(random_wait); static struct fasync_struct *fasync; static DEFINE_SPINLOCK(random_ready_list_lock); @@ -722,8 +721,8 @@ retry: /* should we wake readers? */ if (entropy_bits >= random_read_wakeup_bits && - wq_has_sleeper(&random_read_wait)) { - wake_up_interruptible(&random_read_wait); + wq_has_sleeper(&random_wait)) { + wake_up_interruptible_poll(&random_wait, POLLIN); kill_fasync(&fasync, SIGIO, POLL_IN); } /* If the input pool is getting full, send some @@ -1397,7 +1396,7 @@ retry: trace_debit_entropy(r->name, 8 * ibytes); if (ibytes && (r->entropy_count >> ENTROPY_SHIFT) < random_write_wakeup_bits) { - wake_up_interruptible(&random_write_wait); + wake_up_interruptible_poll(&random_wait, POLLOUT); kill_fasync(&fasync, SIGIO, POLL_OUT); } @@ -1839,7 +1838,7 @@ _random_read(int nonblock, char __user *buf, size_t nbytes) if (nonblock) return -EAGAIN; - wait_event_interruptible(random_read_wait, + wait_event_interruptible(random_wait, ENTROPY_BITS(&input_pool) >= random_read_wakeup_bits); if (signal_pending(current)) @@ -1876,14 +1875,17 @@ urandom_read(struct file *file, char __user *buf, size_t nbytes, loff_t *ppos) return ret; } +static struct wait_queue_head * +random_get_poll_head(struct file *file, __poll_t events) +{ + return &random_wait; +} + static __poll_t -random_poll(struct file *file, poll_table * wait) +random_poll_mask(struct file *file, __poll_t events) { - __poll_t mask; + __poll_t mask = 0; - poll_wait(file, &random_read_wait, wait); - poll_wait(file, &random_write_wait, wait); - mask = 0; if (ENTROPY_BITS(&input_pool) >= random_read_wakeup_bits) mask |= EPOLLIN | EPOLLRDNORM; if (ENTROPY_BITS(&input_pool) < random_write_wakeup_bits) @@ -1990,7 +1992,8 @@ static int random_fasync(int fd, struct file *filp, int on) const struct file_operations random_fops = { .read = random_read, .write = random_write, - .poll = random_poll, + .get_poll_head = random_get_poll_head, + .poll_mask = random_poll_mask, .unlocked_ioctl = random_ioctl, .fasync = random_fasync, .llseek = noop_llseek, @@ -2323,7 +2326,7 @@ void add_hwgenerator_randomness(const char *buffer, size_t count, * We'll be woken up again once below random_write_wakeup_thresh, * or when the calling thread is about to terminate. */ - wait_event_interruptible(random_write_wait, kthread_should_stop() || + wait_event_interruptible(random_wait, kthread_should_stop() || ENTROPY_BITS(&input_pool) <= random_write_wakeup_bits); mix_pool_bytes(poolp, buffer, count); credit_entropy_bits(poolp, entropy); diff --git a/drivers/char/raw.c b/drivers/char/raw.c index 293167c6e254..fd6eec8085b4 100644 --- a/drivers/char/raw.c +++ b/drivers/char/raw.c @@ -321,7 +321,8 @@ static int __init raw_init(void) max_raw_minors = MAX_RAW_MINORS; } - raw_devices = vzalloc(sizeof(struct raw_device_data) * max_raw_minors); + raw_devices = vzalloc(array_size(max_raw_minors, + sizeof(struct raw_device_data))); if (!raw_devices) { printk(KERN_ERR "Not enough memory for raw device structures\n"); ret = -ENOMEM; diff --git a/drivers/char/rtc.c b/drivers/char/rtc.c index 57dc546628b5..94fedeeec035 100644 --- a/drivers/char/rtc.c +++ b/drivers/char/rtc.c @@ -171,7 +171,7 @@ static void mask_rtc_irq_bit(unsigned char bit) #endif #ifdef CONFIG_PROC_FS -static int rtc_proc_open(struct inode *inode, struct file *file); +static int rtc_proc_show(struct seq_file *seq, void *v); #endif /* @@ -832,16 +832,6 @@ static struct miscdevice rtc_dev = { .fops = &rtc_fops, }; -#ifdef CONFIG_PROC_FS -static const struct file_operations rtc_proc_fops = { - .owner = THIS_MODULE, - .open = rtc_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; -#endif - static resource_size_t rtc_size; static struct resource * __init rtc_request_region(resource_size_t size) @@ -982,7 +972,7 @@ no_irq: } #ifdef CONFIG_PROC_FS - ent = proc_create("driver/rtc", 0, NULL, &rtc_proc_fops); + ent = proc_create_single("driver/rtc", 0, NULL, rtc_proc_show); if (!ent) printk(KERN_WARNING "rtc: Failed to register with procfs.\n"); #endif @@ -1201,11 +1191,6 @@ static int rtc_proc_show(struct seq_file *seq, void *v) #undef YN #undef NY } - -static int rtc_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, rtc_proc_show, NULL); -} #endif static void rtc_get_rtc_time(struct rtc_time *rtc_tm) diff --git a/drivers/char/toshiba.c b/drivers/char/toshiba.c index 5488516da8ea..802376fe851a 100644 --- a/drivers/char/toshiba.c +++ b/drivers/char/toshiba.c @@ -326,19 +326,6 @@ static int proc_toshiba_show(struct seq_file *m, void *v) key); return 0; } - -static int proc_toshiba_open(struct inode *inode, struct file *file) -{ - return single_open(file, proc_toshiba_show, NULL); -} - -static const struct file_operations proc_toshiba_fops = { - .owner = THIS_MODULE, - .open = proc_toshiba_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; #endif @@ -524,7 +511,7 @@ static int __init toshiba_init(void) { struct proc_dir_entry *pde; - pde = proc_create("toshiba", 0, NULL, &proc_toshiba_fops); + pde = proc_create_single("toshiba", 0, NULL, proc_toshiba_show); if (!pde) { misc_deregister(&tosh_device); return -ENOMEM; diff --git a/drivers/char/tpm/Makefile b/drivers/char/tpm/Makefile index acd758381c58..4e9c33ca1f8f 100644 --- a/drivers/char/tpm/Makefile +++ b/drivers/char/tpm/Makefile @@ -4,11 +4,11 @@ # obj-$(CONFIG_TCG_TPM) += tpm.o tpm-y := tpm-interface.o tpm-dev.o tpm-sysfs.o tpm-chip.o tpm2-cmd.o \ - tpm-dev-common.o tpmrm-dev.o tpm1_eventlog.o tpm2_eventlog.o \ - tpm2-space.o -tpm-$(CONFIG_ACPI) += tpm_ppi.o tpm_eventlog_acpi.o -tpm-$(CONFIG_EFI) += tpm_eventlog_efi.o -tpm-$(CONFIG_OF) += tpm_eventlog_of.o + tpm-dev-common.o tpmrm-dev.o eventlog/common.o eventlog/tpm1.o \ + eventlog/tpm2.o tpm2-space.o +tpm-$(CONFIG_ACPI) += tpm_ppi.o eventlog/acpi.o +tpm-$(CONFIG_EFI) += eventlog/efi.o +tpm-$(CONFIG_OF) += eventlog/of.o obj-$(CONFIG_TCG_TIS_CORE) += tpm_tis_core.o obj-$(CONFIG_TCG_TIS) += tpm_tis.o obj-$(CONFIG_TCG_TIS_SPI) += tpm_tis_spi.o diff --git a/drivers/char/tpm/eventlog/acpi.c b/drivers/char/tpm/eventlog/acpi.c new file mode 100644 index 000000000000..7c53b1973b62 --- /dev/null +++ b/drivers/char/tpm/eventlog/acpi.c @@ -0,0 +1,113 @@ +/* + * Copyright (C) 2005 IBM Corporation + * + * Authors: + * Seiji Munetoh + * Stefan Berger + * Reiner Sailer + * Kylene Hall + * Nayna Jain + * + * Maintained by: + * + * Access to the event log extended by the TCG BIOS of PC platform + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "../tpm.h" +#include "common.h" + +struct acpi_tcpa { + struct acpi_table_header hdr; + u16 platform_class; + union { + struct client_hdr { + u32 log_max_len __packed; + u64 log_start_addr __packed; + } client; + struct server_hdr { + u16 reserved; + u64 log_max_len __packed; + u64 log_start_addr __packed; + } server; + }; +}; + +/* read binary bios log */ +int tpm_read_log_acpi(struct tpm_chip *chip) +{ + struct acpi_tcpa *buff; + acpi_status status; + void __iomem *virt; + u64 len, start; + struct tpm_bios_log *log; + + if (chip->flags & TPM_CHIP_FLAG_TPM2) + return -ENODEV; + + log = &chip->log; + + /* Unfortuntely ACPI does not associate the event log with a specific + * TPM, like PPI. Thus all ACPI TPMs will read the same log. + */ + if (!chip->acpi_dev_handle) + return -ENODEV; + + /* Find TCPA entry in RSDT (ACPI_LOGICAL_ADDRESSING) */ + status = acpi_get_table(ACPI_SIG_TCPA, 1, + (struct acpi_table_header **)&buff); + + if (ACPI_FAILURE(status)) + return -ENODEV; + + switch(buff->platform_class) { + case BIOS_SERVER: + len = buff->server.log_max_len; + start = buff->server.log_start_addr; + break; + case BIOS_CLIENT: + default: + len = buff->client.log_max_len; + start = buff->client.log_start_addr; + break; + } + if (!len) { + dev_warn(&chip->dev, "%s: TCPA log area empty\n", __func__); + return -EIO; + } + + /* malloc EventLog space */ + log->bios_event_log = kmalloc(len, GFP_KERNEL); + if (!log->bios_event_log) + return -ENOMEM; + + log->bios_event_log_end = log->bios_event_log + len; + + virt = acpi_os_map_iomem(start, len); + if (!virt) + goto err; + + memcpy_fromio(log->bios_event_log, virt, len); + + acpi_os_unmap_iomem(virt, len); + return EFI_TCG2_EVENT_LOG_FORMAT_TCG_1_2; + +err: + kfree(log->bios_event_log); + log->bios_event_log = NULL; + return -EIO; + +} diff --git a/drivers/char/tpm/eventlog/common.c b/drivers/char/tpm/eventlog/common.c new file mode 100644 index 000000000000..5a8720df2b51 --- /dev/null +++ b/drivers/char/tpm/eventlog/common.c @@ -0,0 +1,195 @@ +/* + * Copyright (C) 2005, 2012 IBM Corporation + * + * Authors: + * Kent Yoder + * Seiji Munetoh + * Stefan Berger + * Reiner Sailer + * Kylene Hall + * Nayna Jain + * + * Access to the event log created by a system's firmware / BIOS + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + */ + +#include +#include +#include +#include +#include + +#include "../tpm.h" +#include "common.h" + +static int tpm_bios_measurements_open(struct inode *inode, + struct file *file) +{ + int err; + struct seq_file *seq; + struct tpm_chip_seqops *chip_seqops; + const struct seq_operations *seqops; + struct tpm_chip *chip; + + inode_lock(inode); + if (!inode->i_private) { + inode_unlock(inode); + return -ENODEV; + } + chip_seqops = (struct tpm_chip_seqops *)inode->i_private; + seqops = chip_seqops->seqops; + chip = chip_seqops->chip; + get_device(&chip->dev); + inode_unlock(inode); + + /* now register seq file */ + err = seq_open(file, seqops); + if (!err) { + seq = file->private_data; + seq->private = chip; + } + + return err; +} + +static int tpm_bios_measurements_release(struct inode *inode, + struct file *file) +{ + struct seq_file *seq = (struct seq_file *)file->private_data; + struct tpm_chip *chip = (struct tpm_chip *)seq->private; + + put_device(&chip->dev); + + return seq_release(inode, file); +} + +static const struct file_operations tpm_bios_measurements_ops = { + .owner = THIS_MODULE, + .open = tpm_bios_measurements_open, + .read = seq_read, + .llseek = seq_lseek, + .release = tpm_bios_measurements_release, +}; + +static int tpm_read_log(struct tpm_chip *chip) +{ + int rc; + + if (chip->log.bios_event_log != NULL) { + dev_dbg(&chip->dev, + "%s: ERROR - event log already initialized\n", + __func__); + return -EFAULT; + } + + rc = tpm_read_log_acpi(chip); + if (rc != -ENODEV) + return rc; + + rc = tpm_read_log_efi(chip); + if (rc != -ENODEV) + return rc; + + return tpm_read_log_of(chip); +} + +/* + * tpm_bios_log_setup() - Read the event log from the firmware + * @chip: TPM chip to use. + * + * If an event log is found then the securityfs files are setup to + * export it to userspace, otherwise nothing is done. + * + * Returns -ENODEV if the firmware has no event log or securityfs is not + * supported. + */ +int tpm_bios_log_setup(struct tpm_chip *chip) +{ + const char *name = dev_name(&chip->dev); + unsigned int cnt; + int log_version; + int rc = 0; + + rc = tpm_read_log(chip); + if (rc < 0) + return rc; + log_version = rc; + + cnt = 0; + chip->bios_dir[cnt] = securityfs_create_dir(name, NULL); + /* NOTE: securityfs_create_dir can return ENODEV if securityfs is + * compiled out. The caller should ignore the ENODEV return code. + */ + if (IS_ERR(chip->bios_dir[cnt])) + goto err; + cnt++; + + chip->bin_log_seqops.chip = chip; + if (log_version == EFI_TCG2_EVENT_LOG_FORMAT_TCG_2) + chip->bin_log_seqops.seqops = + &tpm2_binary_b_measurements_seqops; + else + chip->bin_log_seqops.seqops = + &tpm1_binary_b_measurements_seqops; + + + chip->bios_dir[cnt] = + securityfs_create_file("binary_bios_measurements", + 0440, chip->bios_dir[0], + (void *)&chip->bin_log_seqops, + &tpm_bios_measurements_ops); + if (IS_ERR(chip->bios_dir[cnt])) + goto err; + cnt++; + + if (!(chip->flags & TPM_CHIP_FLAG_TPM2)) { + + chip->ascii_log_seqops.chip = chip; + chip->ascii_log_seqops.seqops = + &tpm1_ascii_b_measurements_seqops; + + chip->bios_dir[cnt] = + securityfs_create_file("ascii_bios_measurements", + 0440, chip->bios_dir[0], + (void *)&chip->ascii_log_seqops, + &tpm_bios_measurements_ops); + if (IS_ERR(chip->bios_dir[cnt])) + goto err; + cnt++; + } + + return 0; + +err: + rc = PTR_ERR(chip->bios_dir[cnt]); + chip->bios_dir[cnt] = NULL; + tpm_bios_log_teardown(chip); + return rc; +} + +void tpm_bios_log_teardown(struct tpm_chip *chip) +{ + int i; + struct inode *inode; + + /* securityfs_remove currently doesn't take care of handling sync + * between removal and opening of pseudo files. To handle this, a + * workaround is added by making i_private = NULL here during removal + * and to check it during open(), both within inode_lock()/unlock(). + * This design ensures that open() either safely gets kref or fails. + */ + for (i = (TPM_NUM_EVENT_LOG_FILES - 1); i >= 0; i--) { + if (chip->bios_dir[i]) { + inode = d_inode(chip->bios_dir[i]); + inode_lock(inode); + inode->i_private = NULL; + inode_unlock(inode); + securityfs_remove(chip->bios_dir[i]); + } + } +} diff --git a/drivers/char/tpm/eventlog/common.h b/drivers/char/tpm/eventlog/common.h new file mode 100644 index 000000000000..47ff8136ceb5 --- /dev/null +++ b/drivers/char/tpm/eventlog/common.h @@ -0,0 +1,35 @@ +#ifndef __TPM_EVENTLOG_COMMON_H__ +#define __TPM_EVENTLOG_COMMON_H__ + +#include "../tpm.h" + +extern const struct seq_operations tpm1_ascii_b_measurements_seqops; +extern const struct seq_operations tpm1_binary_b_measurements_seqops; +extern const struct seq_operations tpm2_binary_b_measurements_seqops; + +#if defined(CONFIG_ACPI) +int tpm_read_log_acpi(struct tpm_chip *chip); +#else +static inline int tpm_read_log_acpi(struct tpm_chip *chip) +{ + return -ENODEV; +} +#endif +#if defined(CONFIG_OF) +int tpm_read_log_of(struct tpm_chip *chip); +#else +static inline int tpm_read_log_of(struct tpm_chip *chip) +{ + return -ENODEV; +} +#endif +#if defined(CONFIG_EFI) +int tpm_read_log_efi(struct tpm_chip *chip); +#else +static inline int tpm_read_log_efi(struct tpm_chip *chip) +{ + return -ENODEV; +} +#endif + +#endif diff --git a/drivers/char/tpm/eventlog/efi.c b/drivers/char/tpm/eventlog/efi.c new file mode 100644 index 000000000000..3e673ab22cb4 --- /dev/null +++ b/drivers/char/tpm/eventlog/efi.c @@ -0,0 +1,66 @@ +/* + * Copyright (C) 2017 Google + * + * Authors: + * Thiebaud Weksteen + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + */ + +#include +#include + +#include "../tpm.h" +#include "common.h" + +/* read binary bios log from EFI configuration table */ +int tpm_read_log_efi(struct tpm_chip *chip) +{ + + struct linux_efi_tpm_eventlog *log_tbl; + struct tpm_bios_log *log; + u32 log_size; + u8 tpm_log_version; + + if (!(chip->flags & TPM_CHIP_FLAG_TPM2)) + return -ENODEV; + + if (efi.tpm_log == EFI_INVALID_TABLE_ADDR) + return -ENODEV; + + log = &chip->log; + + log_tbl = memremap(efi.tpm_log, sizeof(*log_tbl), MEMREMAP_WB); + if (!log_tbl) { + pr_err("Could not map UEFI TPM log table !\n"); + return -ENOMEM; + } + + log_size = log_tbl->size; + memunmap(log_tbl); + + log_tbl = memremap(efi.tpm_log, sizeof(*log_tbl) + log_size, + MEMREMAP_WB); + if (!log_tbl) { + pr_err("Could not map UEFI TPM log table payload!\n"); + return -ENOMEM; + } + + /* malloc EventLog space */ + log->bios_event_log = kmemdup(log_tbl->log, log_size, GFP_KERNEL); + if (!log->bios_event_log) + goto err_memunmap; + log->bios_event_log_end = log->bios_event_log + log_size; + + tpm_log_version = log_tbl->version; + memunmap(log_tbl); + return tpm_log_version; + +err_memunmap: + memunmap(log_tbl); + return -ENOMEM; +} diff --git a/drivers/char/tpm/eventlog/of.c b/drivers/char/tpm/eventlog/of.c new file mode 100644 index 000000000000..bba5fba6cb3b --- /dev/null +++ b/drivers/char/tpm/eventlog/of.c @@ -0,0 +1,81 @@ +/* + * Copyright 2012 IBM Corporation + * + * Author: Ashley Lai + * Nayna Jain + * + * Maintained by: + * + * Read the event log created by the firmware on PPC64 + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + */ + +#include +#include +#include + +#include "../tpm.h" +#include "common.h" + +int tpm_read_log_of(struct tpm_chip *chip) +{ + struct device_node *np; + const u32 *sizep; + const u64 *basep; + struct tpm_bios_log *log; + u32 size; + u64 base; + + log = &chip->log; + if (chip->dev.parent && chip->dev.parent->of_node) + np = chip->dev.parent->of_node; + else + return -ENODEV; + + if (of_property_read_bool(np, "powered-while-suspended")) + chip->flags |= TPM_CHIP_FLAG_ALWAYS_POWERED; + + sizep = of_get_property(np, "linux,sml-size", NULL); + basep = of_get_property(np, "linux,sml-base", NULL); + if (sizep == NULL && basep == NULL) + return -ENODEV; + if (sizep == NULL || basep == NULL) + return -EIO; + + /* + * For both vtpm/tpm, firmware has log addr and log size in big + * endian format. But in case of vtpm, there is a method called + * sml-handover which is run during kernel init even before + * device tree is setup. This sml-handover function takes care + * of endianness and writes to sml-base and sml-size in little + * endian format. For this reason, vtpm doesn't need conversion + * but physical tpm needs the conversion. + */ + if (of_property_match_string(np, "compatible", "IBM,vtpm") < 0) { + size = be32_to_cpup((__force __be32 *)sizep); + base = be64_to_cpup((__force __be64 *)basep); + } else { + size = *sizep; + base = *basep; + } + + if (size == 0) { + dev_warn(&chip->dev, "%s: Event log area empty\n", __func__); + return -EIO; + } + + log->bios_event_log = kmemdup(__va(base), size, GFP_KERNEL); + if (!log->bios_event_log) + return -ENOMEM; + + log->bios_event_log_end = log->bios_event_log + size; + + if (chip->flags & TPM_CHIP_FLAG_TPM2) + return EFI_TCG2_EVENT_LOG_FORMAT_TCG_2; + return EFI_TCG2_EVENT_LOG_FORMAT_TCG_1_2; +} diff --git a/drivers/char/tpm/eventlog/tpm1.c b/drivers/char/tpm/eventlog/tpm1.c new file mode 100644 index 000000000000..58c84784ba25 --- /dev/null +++ b/drivers/char/tpm/eventlog/tpm1.c @@ -0,0 +1,310 @@ +/* + * Copyright (C) 2005, 2012 IBM Corporation + * + * Authors: + * Kent Yoder + * Seiji Munetoh + * Stefan Berger + * Reiner Sailer + * Kylene Hall + * Nayna Jain + * + * Maintained by: + * + * Access to the event log created by a system's firmware / BIOS + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "../tpm.h" +#include "common.h" + + +static const char* tcpa_event_type_strings[] = { + "PREBOOT", + "POST CODE", + "", + "NO ACTION", + "SEPARATOR", + "ACTION", + "EVENT TAG", + "S-CRTM Contents", + "S-CRTM Version", + "CPU Microcode", + "Platform Config Flags", + "Table of Devices", + "Compact Hash", + "IPL", + "IPL Partition Data", + "Non-Host Code", + "Non-Host Config", + "Non-Host Info" +}; + +static const char* tcpa_pc_event_id_strings[] = { + "", + "SMBIOS", + "BIS Certificate", + "POST BIOS ", + "ESCD ", + "CMOS", + "NVRAM", + "Option ROM", + "Option ROM config", + "", + "Option ROM microcode ", + "S-CRTM Version", + "S-CRTM Contents ", + "POST Contents ", + "Table of Devices", +}; + +/* returns pointer to start of pos. entry of tcg log */ +static void *tpm1_bios_measurements_start(struct seq_file *m, loff_t *pos) +{ + loff_t i; + struct tpm_chip *chip = m->private; + struct tpm_bios_log *log = &chip->log; + void *addr = log->bios_event_log; + void *limit = log->bios_event_log_end; + struct tcpa_event *event; + u32 converted_event_size; + u32 converted_event_type; + + + /* read over *pos measurements */ + for (i = 0; i < *pos; i++) { + event = addr; + + converted_event_size = + do_endian_conversion(event->event_size); + converted_event_type = + do_endian_conversion(event->event_type); + + if ((addr + sizeof(struct tcpa_event)) < limit) { + if ((converted_event_type == 0) && + (converted_event_size == 0)) + return NULL; + addr += (sizeof(struct tcpa_event) + + converted_event_size); + } + } + + /* now check if current entry is valid */ + if ((addr + sizeof(struct tcpa_event)) >= limit) + return NULL; + + event = addr; + + converted_event_size = do_endian_conversion(event->event_size); + converted_event_type = do_endian_conversion(event->event_type); + + if (((converted_event_type == 0) && (converted_event_size == 0)) + || ((addr + sizeof(struct tcpa_event) + converted_event_size) + >= limit)) + return NULL; + + return addr; +} + +static void *tpm1_bios_measurements_next(struct seq_file *m, void *v, + loff_t *pos) +{ + struct tcpa_event *event = v; + struct tpm_chip *chip = m->private; + struct tpm_bios_log *log = &chip->log; + void *limit = log->bios_event_log_end; + u32 converted_event_size; + u32 converted_event_type; + + converted_event_size = do_endian_conversion(event->event_size); + + v += sizeof(struct tcpa_event) + converted_event_size; + + /* now check if current entry is valid */ + if ((v + sizeof(struct tcpa_event)) >= limit) + return NULL; + + event = v; + + converted_event_size = do_endian_conversion(event->event_size); + converted_event_type = do_endian_conversion(event->event_type); + + if (((converted_event_type == 0) && (converted_event_size == 0)) || + ((v + sizeof(struct tcpa_event) + converted_event_size) >= limit)) + return NULL; + + (*pos)++; + return v; +} + +static void tpm1_bios_measurements_stop(struct seq_file *m, void *v) +{ +} + +static int get_event_name(char *dest, struct tcpa_event *event, + unsigned char * event_entry) +{ + const char *name = ""; + /* 41 so there is room for 40 data and 1 nul */ + char data[41] = ""; + int i, n_len = 0, d_len = 0; + struct tcpa_pc_event *pc_event; + + switch (do_endian_conversion(event->event_type)) { + case PREBOOT: + case POST_CODE: + case UNUSED: + case NO_ACTION: + case SCRTM_CONTENTS: + case SCRTM_VERSION: + case CPU_MICROCODE: + case PLATFORM_CONFIG_FLAGS: + case TABLE_OF_DEVICES: + case COMPACT_HASH: + case IPL: + case IPL_PARTITION_DATA: + case NONHOST_CODE: + case NONHOST_CONFIG: + case NONHOST_INFO: + name = tcpa_event_type_strings[do_endian_conversion + (event->event_type)]; + n_len = strlen(name); + break; + case SEPARATOR: + case ACTION: + if (MAX_TEXT_EVENT > + do_endian_conversion(event->event_size)) { + name = event_entry; + n_len = do_endian_conversion(event->event_size); + } + break; + case EVENT_TAG: + pc_event = (struct tcpa_pc_event *)event_entry; + + /* ToDo Row data -> Base64 */ + + switch (do_endian_conversion(pc_event->event_id)) { + case SMBIOS: + case BIS_CERT: + case CMOS: + case NVRAM: + case OPTION_ROM_EXEC: + case OPTION_ROM_CONFIG: + case S_CRTM_VERSION: + name = tcpa_pc_event_id_strings[do_endian_conversion + (pc_event->event_id)]; + n_len = strlen(name); + break; + /* hash data */ + case POST_BIOS_ROM: + case ESCD: + case OPTION_ROM_MICROCODE: + case S_CRTM_CONTENTS: + case POST_CONTENTS: + name = tcpa_pc_event_id_strings[do_endian_conversion + (pc_event->event_id)]; + n_len = strlen(name); + for (i = 0; i < 20; i++) + d_len += sprintf(&data[2*i], "%02x", + pc_event->event_data[i]); + break; + default: + break; + } + default: + break; + } + + return snprintf(dest, MAX_TEXT_EVENT, "[%.*s%.*s]", + n_len, name, d_len, data); + +} + +static int tpm1_binary_bios_measurements_show(struct seq_file *m, void *v) +{ + struct tcpa_event *event = v; + struct tcpa_event temp_event; + char *temp_ptr; + int i; + + memcpy(&temp_event, event, sizeof(struct tcpa_event)); + + /* convert raw integers for endianness */ + temp_event.pcr_index = do_endian_conversion(event->pcr_index); + temp_event.event_type = do_endian_conversion(event->event_type); + temp_event.event_size = do_endian_conversion(event->event_size); + + temp_ptr = (char *) &temp_event; + + for (i = 0; i < (sizeof(struct tcpa_event) - 1) ; i++) + seq_putc(m, temp_ptr[i]); + + temp_ptr = (char *) v; + + for (i = (sizeof(struct tcpa_event) - 1); + i < (sizeof(struct tcpa_event) + temp_event.event_size); i++) + seq_putc(m, temp_ptr[i]); + + return 0; + +} + +static int tpm1_ascii_bios_measurements_show(struct seq_file *m, void *v) +{ + int len = 0; + char *eventname; + struct tcpa_event *event = v; + unsigned char *event_entry = + (unsigned char *)(v + sizeof(struct tcpa_event)); + + eventname = kmalloc(MAX_TEXT_EVENT, GFP_KERNEL); + if (!eventname) { + printk(KERN_ERR "%s: ERROR - No Memory for event name\n ", + __func__); + return -EFAULT; + } + + /* 1st: PCR */ + seq_printf(m, "%2d ", do_endian_conversion(event->pcr_index)); + + /* 2nd: SHA1 */ + seq_printf(m, "%20phN", event->pcr_value); + + /* 3rd: event type identifier */ + seq_printf(m, " %02x", do_endian_conversion(event->event_type)); + + len += get_event_name(eventname, event, event_entry); + + /* 4th: eventname <= max + \'0' delimiter */ + seq_printf(m, " %s\n", eventname); + + kfree(eventname); + return 0; +} + +const struct seq_operations tpm1_ascii_b_measurements_seqops = { + .start = tpm1_bios_measurements_start, + .next = tpm1_bios_measurements_next, + .stop = tpm1_bios_measurements_stop, + .show = tpm1_ascii_bios_measurements_show, +}; + +const struct seq_operations tpm1_binary_b_measurements_seqops = { + .start = tpm1_bios_measurements_start, + .next = tpm1_bios_measurements_next, + .stop = tpm1_bios_measurements_stop, + .show = tpm1_binary_bios_measurements_show, +}; diff --git a/drivers/char/tpm/eventlog/tpm2.c b/drivers/char/tpm/eventlog/tpm2.c new file mode 100644 index 000000000000..1b8fa9de2cac --- /dev/null +++ b/drivers/char/tpm/eventlog/tpm2.c @@ -0,0 +1,210 @@ +/* + * Copyright (C) 2016 IBM Corporation + * + * Authors: + * Nayna Jain + * + * Access to TPM 2.0 event log as written by Firmware. + * It assumes that writer of event log has followed TCG Specification + * for Family "2.0" and written the event data in little endian. + * With that, it doesn't need any endian conversion for structure + * content. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +#include +#include +#include +#include +#include +#include + +#include "../tpm.h" +#include "common.h" + +/* + * calc_tpm2_event_size() - calculate the event size, where event + * is an entry in the TPM 2.0 event log. The event is of type Crypto + * Agile Log Entry Format as defined in TCG EFI Protocol Specification + * Family "2.0". + + * @event: event whose size is to be calculated. + * @event_header: the first event in the event log. + * + * Returns size of the event. If it is an invalid event, returns 0. + */ +static int calc_tpm2_event_size(struct tcg_pcr_event2 *event, + struct tcg_pcr_event *event_header) +{ + struct tcg_efi_specid_event *efispecid; + struct tcg_event_field *event_field; + void *marker; + void *marker_start; + u32 halg_size; + size_t size; + u16 halg; + int i; + int j; + + marker = event; + marker_start = marker; + marker = marker + sizeof(event->pcr_idx) + sizeof(event->event_type) + + sizeof(event->count); + + efispecid = (struct tcg_efi_specid_event *)event_header->event; + + /* Check if event is malformed. */ + if (event->count > efispecid->num_algs) + return 0; + + for (i = 0; i < event->count; i++) { + halg_size = sizeof(event->digests[i].alg_id); + memcpy(&halg, marker, halg_size); + marker = marker + halg_size; + for (j = 0; j < efispecid->num_algs; j++) { + if (halg == efispecid->digest_sizes[j].alg_id) { + marker += + efispecid->digest_sizes[j].digest_size; + break; + } + } + /* Algorithm without known length. Such event is unparseable. */ + if (j == efispecid->num_algs) + return 0; + } + + event_field = (struct tcg_event_field *)marker; + marker = marker + sizeof(event_field->event_size) + + event_field->event_size; + size = marker - marker_start; + + if ((event->event_type == 0) && (event_field->event_size == 0)) + return 0; + + return size; +} + +static void *tpm2_bios_measurements_start(struct seq_file *m, loff_t *pos) +{ + struct tpm_chip *chip = m->private; + struct tpm_bios_log *log = &chip->log; + void *addr = log->bios_event_log; + void *limit = log->bios_event_log_end; + struct tcg_pcr_event *event_header; + struct tcg_pcr_event2 *event; + size_t size; + int i; + + event_header = addr; + size = sizeof(struct tcg_pcr_event) - sizeof(event_header->event) + + event_header->event_size; + + if (*pos == 0) { + if (addr + size < limit) { + if ((event_header->event_type == 0) && + (event_header->event_size == 0)) + return NULL; + return SEQ_START_TOKEN; + } + } + + if (*pos > 0) { + addr += size; + event = addr; + size = calc_tpm2_event_size(event, event_header); + if ((addr + size >= limit) || (size == 0)) + return NULL; + } + + for (i = 0; i < (*pos - 1); i++) { + event = addr; + size = calc_tpm2_event_size(event, event_header); + + if ((addr + size >= limit) || (size == 0)) + return NULL; + addr += size; + } + + return addr; +} + +static void *tpm2_bios_measurements_next(struct seq_file *m, void *v, + loff_t *pos) +{ + struct tcg_pcr_event *event_header; + struct tcg_pcr_event2 *event; + struct tpm_chip *chip = m->private; + struct tpm_bios_log *log = &chip->log; + void *limit = log->bios_event_log_end; + size_t event_size; + void *marker; + + event_header = log->bios_event_log; + + if (v == SEQ_START_TOKEN) { + event_size = sizeof(struct tcg_pcr_event) - + sizeof(event_header->event) + event_header->event_size; + marker = event_header; + } else { + event = v; + event_size = calc_tpm2_event_size(event, event_header); + if (event_size == 0) + return NULL; + marker = event; + } + + marker = marker + event_size; + if (marker >= limit) + return NULL; + v = marker; + event = v; + + event_size = calc_tpm2_event_size(event, event_header); + if (((v + event_size) >= limit) || (event_size == 0)) + return NULL; + + (*pos)++; + return v; +} + +static void tpm2_bios_measurements_stop(struct seq_file *m, void *v) +{ +} + +static int tpm2_binary_bios_measurements_show(struct seq_file *m, void *v) +{ + struct tpm_chip *chip = m->private; + struct tpm_bios_log *log = &chip->log; + struct tcg_pcr_event *event_header = log->bios_event_log; + struct tcg_pcr_event2 *event = v; + void *temp_ptr; + size_t size; + + if (v == SEQ_START_TOKEN) { + size = sizeof(struct tcg_pcr_event) - + sizeof(event_header->event) + event_header->event_size; + + temp_ptr = event_header; + + if (size > 0) + seq_write(m, temp_ptr, size); + } else { + size = calc_tpm2_event_size(event, event_header); + temp_ptr = event; + if (size > 0) + seq_write(m, temp_ptr, size); + } + + return 0; +} + +const struct seq_operations tpm2_binary_b_measurements_seqops = { + .start = tpm2_bios_measurements_start, + .next = tpm2_bios_measurements_next, + .stop = tpm2_bios_measurements_stop, + .show = tpm2_binary_bios_measurements_show, +}; diff --git a/drivers/char/tpm/st33zp24/spi.c b/drivers/char/tpm/st33zp24/spi.c index 0fc4f20b5f83..d7909ab287a8 100644 --- a/drivers/char/tpm/st33zp24/spi.c +++ b/drivers/char/tpm/st33zp24/spi.c @@ -40,7 +40,7 @@ #define ST33ZP24_OK 0x5A #define ST33ZP24_UNDEFINED_ERR 0x80 #define ST33ZP24_BADLOCALITY 0x81 -#define ST33ZP24_TISREGISTER_UKNOWN 0x82 +#define ST33ZP24_TISREGISTER_UNKNOWN 0x82 #define ST33ZP24_LOCALITY_NOT_ACTIVATED 0x83 #define ST33ZP24_HASH_END_BEFORE_HASH_START 0x84 #define ST33ZP24_BAD_COMMAND_ORDER 0x85 @@ -84,7 +84,7 @@ static int st33zp24_status_to_errno(u8 code) return 0; case ST33ZP24_UNDEFINED_ERR: case ST33ZP24_BADLOCALITY: - case ST33ZP24_TISREGISTER_UKNOWN: + case ST33ZP24_TISREGISTER_UNKNOWN: case ST33ZP24_LOCALITY_NOT_ACTIVATED: case ST33ZP24_HASH_END_BEFORE_HASH_START: case ST33ZP24_BAD_COMMAND_ORDER: diff --git a/drivers/char/tpm/st33zp24/st33zp24.c b/drivers/char/tpm/st33zp24/st33zp24.c index f95b9c75175b..abd675bec88c 100644 --- a/drivers/char/tpm/st33zp24/st33zp24.c +++ b/drivers/char/tpm/st33zp24/st33zp24.c @@ -373,8 +373,6 @@ static int st33zp24_send(struct tpm_chip *chip, unsigned char *buf, int ret; u8 data; - if (!chip) - return -EBUSY; if (len < TPM_HEADER_SIZE) return -EBUSY; diff --git a/drivers/char/tpm/tpm-dev-common.c b/drivers/char/tpm/tpm-dev-common.c index 230b99288024..e4a04b2d3c32 100644 --- a/drivers/char/tpm/tpm-dev-common.c +++ b/drivers/char/tpm/tpm-dev-common.c @@ -37,7 +37,7 @@ static void timeout_work(struct work_struct *work) struct file_priv *priv = container_of(work, struct file_priv, work); mutex_lock(&priv->buffer_mutex); - atomic_set(&priv->data_pending, 0); + priv->data_pending = 0; memset(priv->data_buffer, 0, sizeof(priv->data_buffer)); mutex_unlock(&priv->buffer_mutex); } @@ -46,7 +46,6 @@ void tpm_common_open(struct file *file, struct tpm_chip *chip, struct file_priv *priv) { priv->chip = chip; - atomic_set(&priv->data_pending, 0); mutex_init(&priv->buffer_mutex); timer_setup(&priv->user_read_timer, user_reader_timeout, 0); INIT_WORK(&priv->work, timeout_work); @@ -58,29 +57,24 @@ ssize_t tpm_common_read(struct file *file, char __user *buf, size_t size, loff_t *off) { struct file_priv *priv = file->private_data; - ssize_t ret_size; - ssize_t orig_ret_size; + ssize_t ret_size = 0; int rc; del_singleshot_timer_sync(&priv->user_read_timer); flush_work(&priv->work); - ret_size = atomic_read(&priv->data_pending); - if (ret_size > 0) { /* relay data */ - orig_ret_size = ret_size; - if (size < ret_size) - ret_size = size; + mutex_lock(&priv->buffer_mutex); - mutex_lock(&priv->buffer_mutex); + if (priv->data_pending) { + ret_size = min_t(ssize_t, size, priv->data_pending); rc = copy_to_user(buf, priv->data_buffer, ret_size); - memset(priv->data_buffer, 0, orig_ret_size); + memset(priv->data_buffer, 0, priv->data_pending); if (rc) ret_size = -EFAULT; - mutex_unlock(&priv->buffer_mutex); + priv->data_pending = 0; } - atomic_set(&priv->data_pending, 0); - + mutex_unlock(&priv->buffer_mutex); return ret_size; } @@ -91,17 +85,19 @@ ssize_t tpm_common_write(struct file *file, const char __user *buf, size_t in_size = size; ssize_t out_size; + if (in_size > TPM_BUFSIZE) + return -E2BIG; + + mutex_lock(&priv->buffer_mutex); + /* Cannot perform a write until the read has cleared either via * tpm_read or a user_read_timer timeout. This also prevents split * buffered writes from blocking here. */ - if (atomic_read(&priv->data_pending) != 0) + if (priv->data_pending != 0) { + mutex_unlock(&priv->buffer_mutex); return -EBUSY; - - if (in_size > TPM_BUFSIZE) - return -E2BIG; - - mutex_lock(&priv->buffer_mutex); + } if (copy_from_user (priv->data_buffer, (void __user *) buf, in_size)) { @@ -132,7 +128,7 @@ ssize_t tpm_common_write(struct file *file, const char __user *buf, return out_size; } - atomic_set(&priv->data_pending, out_size); + priv->data_pending = out_size; mutex_unlock(&priv->buffer_mutex); /* Set a timeout by which the reader must come claim the result */ @@ -149,5 +145,5 @@ void tpm_common_release(struct file *file, struct file_priv *priv) del_singleshot_timer_sync(&priv->user_read_timer); flush_work(&priv->work); file->private_data = NULL; - atomic_set(&priv->data_pending, 0); + priv->data_pending = 0; } diff --git a/drivers/char/tpm/tpm-dev.h b/drivers/char/tpm/tpm-dev.h index ba3b6f9dacf7..b24cfb4d3ee1 100644 --- a/drivers/char/tpm/tpm-dev.h +++ b/drivers/char/tpm/tpm-dev.h @@ -8,7 +8,7 @@ struct file_priv { struct tpm_chip *chip; /* Data passed to and from the tpm via the read/write calls */ - atomic_t data_pending; + size_t data_pending; struct mutex buffer_mutex; struct timer_list user_read_timer; /* user needs to claim result */ diff --git a/drivers/char/tpm/tpm-interface.c b/drivers/char/tpm/tpm-interface.c index c43a9e28995e..e32f6e85dc6d 100644 --- a/drivers/char/tpm/tpm-interface.c +++ b/drivers/char/tpm/tpm-interface.c @@ -489,7 +489,7 @@ static ssize_t tpm_try_transmit(struct tpm_chip *chip, goto out; } - tpm_msleep(TPM_TIMEOUT); + tpm_msleep(TPM_TIMEOUT_POLL); rmb(); } while (time_before(jiffies, stop)); @@ -587,7 +587,7 @@ ssize_t tpm_transmit(struct tpm_chip *chip, struct tpm_space *space, */ if (rc == TPM2_RC_TESTING && cc == TPM2_CC_SELF_TEST) break; - delay_msec *= 2; + if (delay_msec > TPM2_DURATION_LONG) { if (rc == TPM2_RC_RETRY) dev_err(&chip->dev, "in retry loop\n"); @@ -597,6 +597,7 @@ ssize_t tpm_transmit(struct tpm_chip *chip, struct tpm_space *space, break; } tpm_msleep(delay_msec); + delay_msec *= 2; memcpy(buf, save, save_size); } return ret; diff --git a/drivers/char/tpm/tpm.h b/drivers/char/tpm/tpm.h index 7f2d0f489e9c..4426649e431c 100644 --- a/drivers/char/tpm/tpm.h +++ b/drivers/char/tpm/tpm.h @@ -53,7 +53,10 @@ enum tpm_const { enum tpm_timeout { TPM_TIMEOUT = 5, /* msecs */ TPM_TIMEOUT_RETRY = 100, /* msecs */ - TPM_TIMEOUT_RANGE_US = 300 /* usecs */ + TPM_TIMEOUT_RANGE_US = 300, /* usecs */ + TPM_TIMEOUT_POLL = 1, /* msecs */ + TPM_TIMEOUT_USECS_MIN = 100, /* usecs */ + TPM_TIMEOUT_USECS_MAX = 500 /* usecs */ }; /* TPM addresses */ @@ -590,33 +593,6 @@ int tpm2_prepare_space(struct tpm_chip *chip, struct tpm_space *space, u32 cc, int tpm2_commit_space(struct tpm_chip *chip, struct tpm_space *space, u32 cc, u8 *buf, size_t *bufsiz); -extern const struct seq_operations tpm2_binary_b_measurements_seqops; - -#if defined(CONFIG_ACPI) -int tpm_read_log_acpi(struct tpm_chip *chip); -#else -static inline int tpm_read_log_acpi(struct tpm_chip *chip) -{ - return -ENODEV; -} -#endif -#if defined(CONFIG_OF) -int tpm_read_log_of(struct tpm_chip *chip); -#else -static inline int tpm_read_log_of(struct tpm_chip *chip) -{ - return -ENODEV; -} -#endif -#if defined(CONFIG_EFI) -int tpm_read_log_efi(struct tpm_chip *chip); -#else -static inline int tpm_read_log_efi(struct tpm_chip *chip) -{ - return -ENODEV; -} -#endif - int tpm_bios_log_setup(struct tpm_chip *chip); void tpm_bios_log_teardown(struct tpm_chip *chip); #endif diff --git a/drivers/char/tpm/tpm1_eventlog.c b/drivers/char/tpm/tpm1_eventlog.c deleted file mode 100644 index add798bd69d0..000000000000 --- a/drivers/char/tpm/tpm1_eventlog.c +++ /dev/null @@ -1,476 +0,0 @@ -/* - * Copyright (C) 2005, 2012 IBM Corporation - * - * Authors: - * Kent Yoder - * Seiji Munetoh - * Stefan Berger - * Reiner Sailer - * Kylene Hall - * Nayna Jain - * - * Maintained by: - * - * Access to the event log created by a system's firmware / BIOS - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. - * - */ - -#include -#include -#include -#include -#include -#include -#include - -#include "tpm.h" - - -static const char* tcpa_event_type_strings[] = { - "PREBOOT", - "POST CODE", - "", - "NO ACTION", - "SEPARATOR", - "ACTION", - "EVENT TAG", - "S-CRTM Contents", - "S-CRTM Version", - "CPU Microcode", - "Platform Config Flags", - "Table of Devices", - "Compact Hash", - "IPL", - "IPL Partition Data", - "Non-Host Code", - "Non-Host Config", - "Non-Host Info" -}; - -static const char* tcpa_pc_event_id_strings[] = { - "", - "SMBIOS", - "BIS Certificate", - "POST BIOS ", - "ESCD ", - "CMOS", - "NVRAM", - "Option ROM", - "Option ROM config", - "", - "Option ROM microcode ", - "S-CRTM Version", - "S-CRTM Contents ", - "POST Contents ", - "Table of Devices", -}; - -/* returns pointer to start of pos. entry of tcg log */ -static void *tpm_bios_measurements_start(struct seq_file *m, loff_t *pos) -{ - loff_t i; - struct tpm_chip *chip = m->private; - struct tpm_bios_log *log = &chip->log; - void *addr = log->bios_event_log; - void *limit = log->bios_event_log_end; - struct tcpa_event *event; - u32 converted_event_size; - u32 converted_event_type; - - - /* read over *pos measurements */ - for (i = 0; i < *pos; i++) { - event = addr; - - converted_event_size = - do_endian_conversion(event->event_size); - converted_event_type = - do_endian_conversion(event->event_type); - - if ((addr + sizeof(struct tcpa_event)) < limit) { - if ((converted_event_type == 0) && - (converted_event_size == 0)) - return NULL; - addr += (sizeof(struct tcpa_event) + - converted_event_size); - } - } - - /* now check if current entry is valid */ - if ((addr + sizeof(struct tcpa_event)) >= limit) - return NULL; - - event = addr; - - converted_event_size = do_endian_conversion(event->event_size); - converted_event_type = do_endian_conversion(event->event_type); - - if (((converted_event_type == 0) && (converted_event_size == 0)) - || ((addr + sizeof(struct tcpa_event) + converted_event_size) - >= limit)) - return NULL; - - return addr; -} - -static void *tpm_bios_measurements_next(struct seq_file *m, void *v, - loff_t *pos) -{ - struct tcpa_event *event = v; - struct tpm_chip *chip = m->private; - struct tpm_bios_log *log = &chip->log; - void *limit = log->bios_event_log_end; - u32 converted_event_size; - u32 converted_event_type; - - converted_event_size = do_endian_conversion(event->event_size); - - v += sizeof(struct tcpa_event) + converted_event_size; - - /* now check if current entry is valid */ - if ((v + sizeof(struct tcpa_event)) >= limit) - return NULL; - - event = v; - - converted_event_size = do_endian_conversion(event->event_size); - converted_event_type = do_endian_conversion(event->event_type); - - if (((converted_event_type == 0) && (converted_event_size == 0)) || - ((v + sizeof(struct tcpa_event) + converted_event_size) >= limit)) - return NULL; - - (*pos)++; - return v; -} - -static void tpm_bios_measurements_stop(struct seq_file *m, void *v) -{ -} - -static int get_event_name(char *dest, struct tcpa_event *event, - unsigned char * event_entry) -{ - const char *name = ""; - /* 41 so there is room for 40 data and 1 nul */ - char data[41] = ""; - int i, n_len = 0, d_len = 0; - struct tcpa_pc_event *pc_event; - - switch (do_endian_conversion(event->event_type)) { - case PREBOOT: - case POST_CODE: - case UNUSED: - case NO_ACTION: - case SCRTM_CONTENTS: - case SCRTM_VERSION: - case CPU_MICROCODE: - case PLATFORM_CONFIG_FLAGS: - case TABLE_OF_DEVICES: - case COMPACT_HASH: - case IPL: - case IPL_PARTITION_DATA: - case NONHOST_CODE: - case NONHOST_CONFIG: - case NONHOST_INFO: - name = tcpa_event_type_strings[do_endian_conversion - (event->event_type)]; - n_len = strlen(name); - break; - case SEPARATOR: - case ACTION: - if (MAX_TEXT_EVENT > - do_endian_conversion(event->event_size)) { - name = event_entry; - n_len = do_endian_conversion(event->event_size); - } - break; - case EVENT_TAG: - pc_event = (struct tcpa_pc_event *)event_entry; - - /* ToDo Row data -> Base64 */ - - switch (do_endian_conversion(pc_event->event_id)) { - case SMBIOS: - case BIS_CERT: - case CMOS: - case NVRAM: - case OPTION_ROM_EXEC: - case OPTION_ROM_CONFIG: - case S_CRTM_VERSION: - name = tcpa_pc_event_id_strings[do_endian_conversion - (pc_event->event_id)]; - n_len = strlen(name); - break; - /* hash data */ - case POST_BIOS_ROM: - case ESCD: - case OPTION_ROM_MICROCODE: - case S_CRTM_CONTENTS: - case POST_CONTENTS: - name = tcpa_pc_event_id_strings[do_endian_conversion - (pc_event->event_id)]; - n_len = strlen(name); - for (i = 0; i < 20; i++) - d_len += sprintf(&data[2*i], "%02x", - pc_event->event_data[i]); - break; - default: - break; - } - default: - break; - } - - return snprintf(dest, MAX_TEXT_EVENT, "[%.*s%.*s]", - n_len, name, d_len, data); - -} - -static int tpm_binary_bios_measurements_show(struct seq_file *m, void *v) -{ - struct tcpa_event *event = v; - struct tcpa_event temp_event; - char *temp_ptr; - int i; - - memcpy(&temp_event, event, sizeof(struct tcpa_event)); - - /* convert raw integers for endianness */ - temp_event.pcr_index = do_endian_conversion(event->pcr_index); - temp_event.event_type = do_endian_conversion(event->event_type); - temp_event.event_size = do_endian_conversion(event->event_size); - - temp_ptr = (char *) &temp_event; - - for (i = 0; i < (sizeof(struct tcpa_event) - 1) ; i++) - seq_putc(m, temp_ptr[i]); - - temp_ptr = (char *) v; - - for (i = (sizeof(struct tcpa_event) - 1); - i < (sizeof(struct tcpa_event) + temp_event.event_size); i++) - seq_putc(m, temp_ptr[i]); - - return 0; - -} - -static int tpm_bios_measurements_release(struct inode *inode, - struct file *file) -{ - struct seq_file *seq = (struct seq_file *)file->private_data; - struct tpm_chip *chip = (struct tpm_chip *)seq->private; - - put_device(&chip->dev); - - return seq_release(inode, file); -} - -static int tpm_ascii_bios_measurements_show(struct seq_file *m, void *v) -{ - int len = 0; - char *eventname; - struct tcpa_event *event = v; - unsigned char *event_entry = - (unsigned char *)(v + sizeof(struct tcpa_event)); - - eventname = kmalloc(MAX_TEXT_EVENT, GFP_KERNEL); - if (!eventname) { - printk(KERN_ERR "%s: ERROR - No Memory for event name\n ", - __func__); - return -EFAULT; - } - - /* 1st: PCR */ - seq_printf(m, "%2d ", do_endian_conversion(event->pcr_index)); - - /* 2nd: SHA1 */ - seq_printf(m, "%20phN", event->pcr_value); - - /* 3rd: event type identifier */ - seq_printf(m, " %02x", do_endian_conversion(event->event_type)); - - len += get_event_name(eventname, event, event_entry); - - /* 4th: eventname <= max + \'0' delimiter */ - seq_printf(m, " %s\n", eventname); - - kfree(eventname); - return 0; -} - -static const struct seq_operations tpm_ascii_b_measurements_seqops = { - .start = tpm_bios_measurements_start, - .next = tpm_bios_measurements_next, - .stop = tpm_bios_measurements_stop, - .show = tpm_ascii_bios_measurements_show, -}; - -static const struct seq_operations tpm_binary_b_measurements_seqops = { - .start = tpm_bios_measurements_start, - .next = tpm_bios_measurements_next, - .stop = tpm_bios_measurements_stop, - .show = tpm_binary_bios_measurements_show, -}; - -static int tpm_bios_measurements_open(struct inode *inode, - struct file *file) -{ - int err; - struct seq_file *seq; - struct tpm_chip_seqops *chip_seqops; - const struct seq_operations *seqops; - struct tpm_chip *chip; - - inode_lock(inode); - if (!inode->i_private) { - inode_unlock(inode); - return -ENODEV; - } - chip_seqops = (struct tpm_chip_seqops *)inode->i_private; - seqops = chip_seqops->seqops; - chip = chip_seqops->chip; - get_device(&chip->dev); - inode_unlock(inode); - - /* now register seq file */ - err = seq_open(file, seqops); - if (!err) { - seq = file->private_data; - seq->private = chip; - } - - return err; -} - -static const struct file_operations tpm_bios_measurements_ops = { - .owner = THIS_MODULE, - .open = tpm_bios_measurements_open, - .read = seq_read, - .llseek = seq_lseek, - .release = tpm_bios_measurements_release, -}; - -static int tpm_read_log(struct tpm_chip *chip) -{ - int rc; - - if (chip->log.bios_event_log != NULL) { - dev_dbg(&chip->dev, - "%s: ERROR - event log already initialized\n", - __func__); - return -EFAULT; - } - - rc = tpm_read_log_acpi(chip); - if (rc != -ENODEV) - return rc; - - rc = tpm_read_log_efi(chip); - if (rc != -ENODEV) - return rc; - - return tpm_read_log_of(chip); -} - -/* - * tpm_bios_log_setup() - Read the event log from the firmware - * @chip: TPM chip to use. - * - * If an event log is found then the securityfs files are setup to - * export it to userspace, otherwise nothing is done. - * - * Returns -ENODEV if the firmware has no event log or securityfs is not - * supported. - */ -int tpm_bios_log_setup(struct tpm_chip *chip) -{ - const char *name = dev_name(&chip->dev); - unsigned int cnt; - int log_version; - int rc = 0; - - rc = tpm_read_log(chip); - if (rc < 0) - return rc; - log_version = rc; - - cnt = 0; - chip->bios_dir[cnt] = securityfs_create_dir(name, NULL); - /* NOTE: securityfs_create_dir can return ENODEV if securityfs is - * compiled out. The caller should ignore the ENODEV return code. - */ - if (IS_ERR(chip->bios_dir[cnt])) - goto err; - cnt++; - - chip->bin_log_seqops.chip = chip; - if (log_version == EFI_TCG2_EVENT_LOG_FORMAT_TCG_2) - chip->bin_log_seqops.seqops = - &tpm2_binary_b_measurements_seqops; - else - chip->bin_log_seqops.seqops = - &tpm_binary_b_measurements_seqops; - - - chip->bios_dir[cnt] = - securityfs_create_file("binary_bios_measurements", - 0440, chip->bios_dir[0], - (void *)&chip->bin_log_seqops, - &tpm_bios_measurements_ops); - if (IS_ERR(chip->bios_dir[cnt])) - goto err; - cnt++; - - if (!(chip->flags & TPM_CHIP_FLAG_TPM2)) { - - chip->ascii_log_seqops.chip = chip; - chip->ascii_log_seqops.seqops = - &tpm_ascii_b_measurements_seqops; - - chip->bios_dir[cnt] = - securityfs_create_file("ascii_bios_measurements", - 0440, chip->bios_dir[0], - (void *)&chip->ascii_log_seqops, - &tpm_bios_measurements_ops); - if (IS_ERR(chip->bios_dir[cnt])) - goto err; - cnt++; - } - - return 0; - -err: - rc = PTR_ERR(chip->bios_dir[cnt]); - chip->bios_dir[cnt] = NULL; - tpm_bios_log_teardown(chip); - return rc; -} - -void tpm_bios_log_teardown(struct tpm_chip *chip) -{ - int i; - struct inode *inode; - - /* securityfs_remove currently doesn't take care of handling sync - * between removal and opening of pseudo files. To handle this, a - * workaround is added by making i_private = NULL here during removal - * and to check it during open(), both within inode_lock()/unlock(). - * This design ensures that open() either safely gets kref or fails. - */ - for (i = (TPM_NUM_EVENT_LOG_FILES - 1); i >= 0; i--) { - if (chip->bios_dir[i]) { - inode = d_inode(chip->bios_dir[i]); - inode_lock(inode); - inode->i_private = NULL; - inode_unlock(inode); - securityfs_remove(chip->bios_dir[i]); - } - } -} diff --git a/drivers/char/tpm/tpm2-cmd.c b/drivers/char/tpm/tpm2-cmd.c index 96c77c8e7f40..d31b09099216 100644 --- a/drivers/char/tpm/tpm2-cmd.c +++ b/drivers/char/tpm/tpm2-cmd.c @@ -980,7 +980,7 @@ static int tpm2_get_cc_attrs_tbl(struct tpm_chip *chip) goto out; } - chip->cc_attrs_tbl = devm_kzalloc(&chip->dev, 4 * nr_commands, + chip->cc_attrs_tbl = devm_kcalloc(&chip->dev, 4, nr_commands, GFP_KERNEL); rc = tpm_buf_init(&buf, TPM2_ST_NO_SESSIONS, TPM2_CC_GET_CAPABILITY); diff --git a/drivers/char/tpm/tpm2-space.c b/drivers/char/tpm/tpm2-space.c index 4e4014eabdb9..6122d3276f72 100644 --- a/drivers/char/tpm/tpm2-space.c +++ b/drivers/char/tpm/tpm2-space.c @@ -102,8 +102,9 @@ static int tpm2_load_context(struct tpm_chip *chip, u8 *buf, * TPM_RC_REFERENCE_H0 means the session has been * flushed outside the space */ - rc = -ENOENT; + *handle = 0; tpm_buf_destroy(&tbuf); + return -ENOENT; } else if (rc > 0) { dev_warn(&chip->dev, "%s: failed with a TPM error 0x%04X\n", __func__, rc); diff --git a/drivers/char/tpm/tpm2_eventlog.c b/drivers/char/tpm/tpm2_eventlog.c deleted file mode 100644 index 1ce4411292ba..000000000000 --- a/drivers/char/tpm/tpm2_eventlog.c +++ /dev/null @@ -1,209 +0,0 @@ -/* - * Copyright (C) 2016 IBM Corporation - * - * Authors: - * Nayna Jain - * - * Access to TPM 2.0 event log as written by Firmware. - * It assumes that writer of event log has followed TCG Specification - * for Family "2.0" and written the event data in little endian. - * With that, it doesn't need any endian conversion for structure - * content. - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. - */ - -#include -#include -#include -#include -#include -#include - -#include "tpm.h" - -/* - * calc_tpm2_event_size() - calculate the event size, where event - * is an entry in the TPM 2.0 event log. The event is of type Crypto - * Agile Log Entry Format as defined in TCG EFI Protocol Specification - * Family "2.0". - - * @event: event whose size is to be calculated. - * @event_header: the first event in the event log. - * - * Returns size of the event. If it is an invalid event, returns 0. - */ -static int calc_tpm2_event_size(struct tcg_pcr_event2 *event, - struct tcg_pcr_event *event_header) -{ - struct tcg_efi_specid_event *efispecid; - struct tcg_event_field *event_field; - void *marker; - void *marker_start; - u32 halg_size; - size_t size; - u16 halg; - int i; - int j; - - marker = event; - marker_start = marker; - marker = marker + sizeof(event->pcr_idx) + sizeof(event->event_type) - + sizeof(event->count); - - efispecid = (struct tcg_efi_specid_event *)event_header->event; - - /* Check if event is malformed. */ - if (event->count > efispecid->num_algs) - return 0; - - for (i = 0; i < event->count; i++) { - halg_size = sizeof(event->digests[i].alg_id); - memcpy(&halg, marker, halg_size); - marker = marker + halg_size; - for (j = 0; j < efispecid->num_algs; j++) { - if (halg == efispecid->digest_sizes[j].alg_id) { - marker += - efispecid->digest_sizes[j].digest_size; - break; - } - } - /* Algorithm without known length. Such event is unparseable. */ - if (j == efispecid->num_algs) - return 0; - } - - event_field = (struct tcg_event_field *)marker; - marker = marker + sizeof(event_field->event_size) - + event_field->event_size; - size = marker - marker_start; - - if ((event->event_type == 0) && (event_field->event_size == 0)) - return 0; - - return size; -} - -static void *tpm2_bios_measurements_start(struct seq_file *m, loff_t *pos) -{ - struct tpm_chip *chip = m->private; - struct tpm_bios_log *log = &chip->log; - void *addr = log->bios_event_log; - void *limit = log->bios_event_log_end; - struct tcg_pcr_event *event_header; - struct tcg_pcr_event2 *event; - size_t size; - int i; - - event_header = addr; - size = sizeof(struct tcg_pcr_event) - sizeof(event_header->event) - + event_header->event_size; - - if (*pos == 0) { - if (addr + size < limit) { - if ((event_header->event_type == 0) && - (event_header->event_size == 0)) - return NULL; - return SEQ_START_TOKEN; - } - } - - if (*pos > 0) { - addr += size; - event = addr; - size = calc_tpm2_event_size(event, event_header); - if ((addr + size >= limit) || (size == 0)) - return NULL; - } - - for (i = 0; i < (*pos - 1); i++) { - event = addr; - size = calc_tpm2_event_size(event, event_header); - - if ((addr + size >= limit) || (size == 0)) - return NULL; - addr += size; - } - - return addr; -} - -static void *tpm2_bios_measurements_next(struct seq_file *m, void *v, - loff_t *pos) -{ - struct tcg_pcr_event *event_header; - struct tcg_pcr_event2 *event; - struct tpm_chip *chip = m->private; - struct tpm_bios_log *log = &chip->log; - void *limit = log->bios_event_log_end; - size_t event_size; - void *marker; - - event_header = log->bios_event_log; - - if (v == SEQ_START_TOKEN) { - event_size = sizeof(struct tcg_pcr_event) - - sizeof(event_header->event) + event_header->event_size; - marker = event_header; - } else { - event = v; - event_size = calc_tpm2_event_size(event, event_header); - if (event_size == 0) - return NULL; - marker = event; - } - - marker = marker + event_size; - if (marker >= limit) - return NULL; - v = marker; - event = v; - - event_size = calc_tpm2_event_size(event, event_header); - if (((v + event_size) >= limit) || (event_size == 0)) - return NULL; - - (*pos)++; - return v; -} - -static void tpm2_bios_measurements_stop(struct seq_file *m, void *v) -{ -} - -static int tpm2_binary_bios_measurements_show(struct seq_file *m, void *v) -{ - struct tpm_chip *chip = m->private; - struct tpm_bios_log *log = &chip->log; - struct tcg_pcr_event *event_header = log->bios_event_log; - struct tcg_pcr_event2 *event = v; - void *temp_ptr; - size_t size; - - if (v == SEQ_START_TOKEN) { - size = sizeof(struct tcg_pcr_event) - - sizeof(event_header->event) + event_header->event_size; - - temp_ptr = event_header; - - if (size > 0) - seq_write(m, temp_ptr, size); - } else { - size = calc_tpm2_event_size(event, event_header); - temp_ptr = event; - if (size > 0) - seq_write(m, temp_ptr, size); - } - - return 0; -} - -const struct seq_operations tpm2_binary_b_measurements_seqops = { - .start = tpm2_bios_measurements_start, - .next = tpm2_bios_measurements_next, - .stop = tpm2_bios_measurements_stop, - .show = tpm2_binary_bios_measurements_show, -}; diff --git a/drivers/char/tpm/tpm_crb.c b/drivers/char/tpm/tpm_crb.c index 7f78482cd157..34fbc6cb097b 100644 --- a/drivers/char/tpm/tpm_crb.c +++ b/drivers/char/tpm/tpm_crb.c @@ -511,8 +511,10 @@ static int crb_map_io(struct acpi_device *device, struct crb_priv *priv, priv->regs_t = crb_map_res(dev, priv, &io_res, buf->control_address, sizeof(struct crb_regs_tail)); - if (IS_ERR(priv->regs_t)) - return PTR_ERR(priv->regs_t); + if (IS_ERR(priv->regs_t)) { + ret = PTR_ERR(priv->regs_t); + goto out_relinquish_locality; + } /* * PTT HW bug w/a: wake up the device to access @@ -520,7 +522,7 @@ static int crb_map_io(struct acpi_device *device, struct crb_priv *priv, */ ret = crb_cmd_ready(dev, priv); if (ret) - return ret; + goto out_relinquish_locality; pa_high = ioread32(&priv->regs_t->ctrl_cmd_pa_high); pa_low = ioread32(&priv->regs_t->ctrl_cmd_pa_low); @@ -565,6 +567,8 @@ out: crb_go_idle(dev, priv); +out_relinquish_locality: + __crb_relinquish_locality(dev, priv, 0); return ret; diff --git a/drivers/char/tpm/tpm_eventlog_acpi.c b/drivers/char/tpm/tpm_eventlog_acpi.c deleted file mode 100644 index 66f19e93c216..000000000000 --- a/drivers/char/tpm/tpm_eventlog_acpi.c +++ /dev/null @@ -1,112 +0,0 @@ -/* - * Copyright (C) 2005 IBM Corporation - * - * Authors: - * Seiji Munetoh - * Stefan Berger - * Reiner Sailer - * Kylene Hall - * Nayna Jain - * - * Maintained by: - * - * Access to the event log extended by the TCG BIOS of PC platform - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. - * - */ - -#include -#include -#include -#include -#include -#include -#include - -#include "tpm.h" - -struct acpi_tcpa { - struct acpi_table_header hdr; - u16 platform_class; - union { - struct client_hdr { - u32 log_max_len __packed; - u64 log_start_addr __packed; - } client; - struct server_hdr { - u16 reserved; - u64 log_max_len __packed; - u64 log_start_addr __packed; - } server; - }; -}; - -/* read binary bios log */ -int tpm_read_log_acpi(struct tpm_chip *chip) -{ - struct acpi_tcpa *buff; - acpi_status status; - void __iomem *virt; - u64 len, start; - struct tpm_bios_log *log; - - if (chip->flags & TPM_CHIP_FLAG_TPM2) - return -ENODEV; - - log = &chip->log; - - /* Unfortuntely ACPI does not associate the event log with a specific - * TPM, like PPI. Thus all ACPI TPMs will read the same log. - */ - if (!chip->acpi_dev_handle) - return -ENODEV; - - /* Find TCPA entry in RSDT (ACPI_LOGICAL_ADDRESSING) */ - status = acpi_get_table(ACPI_SIG_TCPA, 1, - (struct acpi_table_header **)&buff); - - if (ACPI_FAILURE(status)) - return -ENODEV; - - switch(buff->platform_class) { - case BIOS_SERVER: - len = buff->server.log_max_len; - start = buff->server.log_start_addr; - break; - case BIOS_CLIENT: - default: - len = buff->client.log_max_len; - start = buff->client.log_start_addr; - break; - } - if (!len) { - dev_warn(&chip->dev, "%s: TCPA log area empty\n", __func__); - return -EIO; - } - - /* malloc EventLog space */ - log->bios_event_log = kmalloc(len, GFP_KERNEL); - if (!log->bios_event_log) - return -ENOMEM; - - log->bios_event_log_end = log->bios_event_log + len; - - virt = acpi_os_map_iomem(start, len); - if (!virt) - goto err; - - memcpy_fromio(log->bios_event_log, virt, len); - - acpi_os_unmap_iomem(virt, len); - return EFI_TCG2_EVENT_LOG_FORMAT_TCG_1_2; - -err: - kfree(log->bios_event_log); - log->bios_event_log = NULL; - return -EIO; - -} diff --git a/drivers/char/tpm/tpm_eventlog_efi.c b/drivers/char/tpm/tpm_eventlog_efi.c deleted file mode 100644 index e3f9ffd341d2..000000000000 --- a/drivers/char/tpm/tpm_eventlog_efi.c +++ /dev/null @@ -1,66 +0,0 @@ -/* - * Copyright (C) 2017 Google - * - * Authors: - * Thiebaud Weksteen - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. - * - */ - -#include -#include - -#include "tpm.h" - -/* read binary bios log from EFI configuration table */ -int tpm_read_log_efi(struct tpm_chip *chip) -{ - - struct linux_efi_tpm_eventlog *log_tbl; - struct tpm_bios_log *log; - u32 log_size; - u8 tpm_log_version; - - if (!(chip->flags & TPM_CHIP_FLAG_TPM2)) - return -ENODEV; - - if (efi.tpm_log == EFI_INVALID_TABLE_ADDR) - return -ENODEV; - - log = &chip->log; - - log_tbl = memremap(efi.tpm_log, sizeof(*log_tbl), MEMREMAP_WB); - if (!log_tbl) { - pr_err("Could not map UEFI TPM log table !\n"); - return -ENOMEM; - } - - log_size = log_tbl->size; - memunmap(log_tbl); - - log_tbl = memremap(efi.tpm_log, sizeof(*log_tbl) + log_size, - MEMREMAP_WB); - if (!log_tbl) { - pr_err("Could not map UEFI TPM log table payload!\n"); - return -ENOMEM; - } - - /* malloc EventLog space */ - log->bios_event_log = kmalloc(log_size, GFP_KERNEL); - if (!log->bios_event_log) - goto err_memunmap; - memcpy(log->bios_event_log, log_tbl->log, log_size); - log->bios_event_log_end = log->bios_event_log + log_size; - - tpm_log_version = log_tbl->version; - memunmap(log_tbl); - return tpm_log_version; - -err_memunmap: - memunmap(log_tbl); - return -ENOMEM; -} diff --git a/drivers/char/tpm/tpm_eventlog_of.c b/drivers/char/tpm/tpm_eventlog_of.c deleted file mode 100644 index 96fd5646f866..000000000000 --- a/drivers/char/tpm/tpm_eventlog_of.c +++ /dev/null @@ -1,82 +0,0 @@ -/* - * Copyright 2012 IBM Corporation - * - * Author: Ashley Lai - * Nayna Jain - * - * Maintained by: - * - * Read the event log created by the firmware on PPC64 - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. - * - */ - -#include -#include -#include - -#include "tpm.h" - -int tpm_read_log_of(struct tpm_chip *chip) -{ - struct device_node *np; - const u32 *sizep; - const u64 *basep; - struct tpm_bios_log *log; - u32 size; - u64 base; - - log = &chip->log; - if (chip->dev.parent && chip->dev.parent->of_node) - np = chip->dev.parent->of_node; - else - return -ENODEV; - - if (of_property_read_bool(np, "powered-while-suspended")) - chip->flags |= TPM_CHIP_FLAG_ALWAYS_POWERED; - - sizep = of_get_property(np, "linux,sml-size", NULL); - basep = of_get_property(np, "linux,sml-base", NULL); - if (sizep == NULL && basep == NULL) - return -ENODEV; - if (sizep == NULL || basep == NULL) - return -EIO; - - /* - * For both vtpm/tpm, firmware has log addr and log size in big - * endian format. But in case of vtpm, there is a method called - * sml-handover which is run during kernel init even before - * device tree is setup. This sml-handover function takes care - * of endianness and writes to sml-base and sml-size in little - * endian format. For this reason, vtpm doesn't need conversion - * but physical tpm needs the conversion. - */ - if (of_property_match_string(np, "compatible", "IBM,vtpm") < 0) { - size = be32_to_cpup(sizep); - base = be64_to_cpup(basep); - } else { - size = *sizep; - base = *basep; - } - - if (size == 0) { - dev_warn(&chip->dev, "%s: Event log area empty\n", __func__); - return -EIO; - } - - log->bios_event_log = kmalloc(size, GFP_KERNEL); - if (!log->bios_event_log) - return -ENOMEM; - - log->bios_event_log_end = log->bios_event_log + size; - - memcpy(log->bios_event_log, __va(base), size); - - if (chip->flags & TPM_CHIP_FLAG_TPM2) - return EFI_TCG2_EVENT_LOG_FORMAT_TCG_2; - return EFI_TCG2_EVENT_LOG_FORMAT_TCG_1_2; -} diff --git a/drivers/char/tpm/tpm_tis_core.c b/drivers/char/tpm/tpm_tis_core.c index 5a1f47b43947..8b46aaa9e049 100644 --- a/drivers/char/tpm/tpm_tis_core.c +++ b/drivers/char/tpm/tpm_tis_core.c @@ -31,12 +31,6 @@ #include "tpm.h" #include "tpm_tis_core.h" -/* This is a polling delay to check for status and burstcount. - * As per ddwg input, expectation is that status check and burstcount - * check should return within few usecs. - */ -#define TPM_POLL_SLEEP 1 /* msec */ - static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value); static bool wait_for_tpm_stat_cond(struct tpm_chip *chip, u8 mask, @@ -90,7 +84,8 @@ again: } } else { do { - tpm_msleep(TPM_POLL_SLEEP); + usleep_range(TPM_TIMEOUT_USECS_MIN, + TPM_TIMEOUT_USECS_MAX); status = chip->ops->status(chip); if ((status & mask) == mask) return 0; @@ -143,13 +138,58 @@ static bool check_locality(struct tpm_chip *chip, int l) return false; } +static bool locality_inactive(struct tpm_chip *chip, int l) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + int rc; + u8 access; + + rc = tpm_tis_read8(priv, TPM_ACCESS(l), &access); + if (rc < 0) + return false; + + if ((access & (TPM_ACCESS_VALID | TPM_ACCESS_ACTIVE_LOCALITY)) + == TPM_ACCESS_VALID) + return true; + + return false; +} + static int release_locality(struct tpm_chip *chip, int l) { struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + unsigned long stop, timeout; + long rc; tpm_tis_write8(priv, TPM_ACCESS(l), TPM_ACCESS_ACTIVE_LOCALITY); - return 0; + stop = jiffies + chip->timeout_a; + + if (chip->flags & TPM_CHIP_FLAG_IRQ) { +again: + timeout = stop - jiffies; + if ((long)timeout <= 0) + return -1; + + rc = wait_event_interruptible_timeout(priv->int_queue, + (locality_inactive(chip, l)), + timeout); + + if (rc > 0) + return 0; + + if (rc == -ERESTARTSYS && freezing(current)) { + clear_thread_flag(TIF_SIGPENDING); + goto again; + } + } else { + do { + if (locality_inactive(chip, l)) + return 0; + tpm_msleep(TPM_TIMEOUT); + } while (time_before(jiffies, stop)); + } + return -1; } static int request_locality(struct tpm_chip *chip, int l) @@ -234,7 +274,7 @@ static int get_burstcount(struct tpm_chip *chip) burstcnt = (value >> 8) & 0xFFFF; if (burstcnt) return burstcnt; - tpm_msleep(TPM_POLL_SLEEP); + usleep_range(TPM_TIMEOUT_USECS_MIN, TPM_TIMEOUT_USECS_MAX); } while (time_before(jiffies, stop)); return -EBUSY; } diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c index 21085515814f..17084cfcf53e 100644 --- a/drivers/char/virtio_console.c +++ b/drivers/char/virtio_console.c @@ -433,8 +433,7 @@ static struct port_buffer *alloc_buf(struct virtio_device *vdev, size_t buf_size * Allocate buffer and the sg list. The sg list array is allocated * directly after the port_buffer struct. */ - buf = kmalloc(sizeof(*buf) + sizeof(struct scatterlist) * pages, - GFP_KERNEL); + buf = kmalloc(struct_size(buf, sg, pages), GFP_KERNEL); if (!buf) goto fail; @@ -1892,13 +1891,14 @@ static int init_vqs(struct ports_device *portdev) nr_ports = portdev->max_nr_ports; nr_queues = use_multiport(portdev) ? (nr_ports + 1) * 2 : 2; - vqs = kmalloc(nr_queues * sizeof(struct virtqueue *), GFP_KERNEL); - io_callbacks = kmalloc(nr_queues * sizeof(vq_callback_t *), GFP_KERNEL); - io_names = kmalloc(nr_queues * sizeof(char *), GFP_KERNEL); - portdev->in_vqs = kmalloc(nr_ports * sizeof(struct virtqueue *), - GFP_KERNEL); - portdev->out_vqs = kmalloc(nr_ports * sizeof(struct virtqueue *), - GFP_KERNEL); + vqs = kmalloc_array(nr_queues, sizeof(struct virtqueue *), GFP_KERNEL); + io_callbacks = kmalloc_array(nr_queues, sizeof(vq_callback_t *), + GFP_KERNEL); + io_names = kmalloc_array(nr_queues, sizeof(char *), GFP_KERNEL); + portdev->in_vqs = kmalloc_array(nr_ports, sizeof(struct virtqueue *), + GFP_KERNEL); + portdev->out_vqs = kmalloc_array(nr_ports, sizeof(struct virtqueue *), + GFP_KERNEL); if (!vqs || !io_callbacks || !io_names || !portdev->in_vqs || !portdev->out_vqs) { err = -ENOMEM; diff --git a/drivers/clk/Kconfig b/drivers/clk/Kconfig index 34968a381d0f..721572a8c429 100644 --- a/drivers/clk/Kconfig +++ b/drivers/clk/Kconfig @@ -277,6 +277,7 @@ config COMMON_CLK_STM32H7 ---help--- Support for stm32h7 SoC family clocks +source "drivers/clk/actions/Kconfig" source "drivers/clk/bcm/Kconfig" source "drivers/clk/hisilicon/Kconfig" source "drivers/clk/imgtec/Kconfig" diff --git a/drivers/clk/Makefile b/drivers/clk/Makefile index de6d06ac790b..ae40cbe770f0 100644 --- a/drivers/clk/Makefile +++ b/drivers/clk/Makefile @@ -33,6 +33,7 @@ obj-$(CONFIG_CLK_HSDK) += clk-hsdk-pll.o obj-$(CONFIG_COMMON_CLK_MAX77686) += clk-max77686.o obj-$(CONFIG_ARCH_MOXART) += clk-moxart.o obj-$(CONFIG_ARCH_NOMADIK) += clk-nomadik.o +obj-$(CONFIG_ARCH_NPCM7XX) += clk-npcm7xx.o obj-$(CONFIG_ARCH_NSPIRE) += clk-nspire.o obj-$(CONFIG_COMMON_CLK_OXNAS) += clk-oxnas.o obj-$(CONFIG_COMMON_CLK_PALMAS) += clk-palmas.o @@ -59,6 +60,7 @@ obj-$(CONFIG_COMMON_CLK_WM831X) += clk-wm831x.o obj-$(CONFIG_COMMON_CLK_XGENE) += clk-xgene.o # please keep this section sorted lexicographically by directory path name +obj-y += actions/ obj-$(CONFIG_COMMON_CLK_AT91) += at91/ obj-$(CONFIG_ARCH_ARTPEC) += axis/ obj-$(CONFIG_ARC_PLAT_AXS10X) += axs10x/ diff --git a/drivers/clk/actions/Kconfig b/drivers/clk/actions/Kconfig new file mode 100644 index 000000000000..8854adb37847 --- /dev/null +++ b/drivers/clk/actions/Kconfig @@ -0,0 +1,14 @@ +config CLK_ACTIONS + bool "Clock driver for Actions Semi SoCs" + depends on ARCH_ACTIONS || COMPILE_TEST + default ARCH_ACTIONS + +if CLK_ACTIONS + +# SoC Drivers + +config CLK_OWL_S900 + bool "Support for the Actions Semi OWL S900 clocks" + depends on (ARM64 && ARCH_ACTIONS) || COMPILE_TEST + default ARM64 && ARCH_ACTIONS +endif diff --git a/drivers/clk/actions/Makefile b/drivers/clk/actions/Makefile new file mode 100644 index 000000000000..76e431434d10 --- /dev/null +++ b/drivers/clk/actions/Makefile @@ -0,0 +1,12 @@ +obj-$(CONFIG_CLK_ACTIONS) += clk-owl.o + +clk-owl-y += owl-common.o +clk-owl-y += owl-gate.o +clk-owl-y += owl-mux.o +clk-owl-y += owl-divider.o +clk-owl-y += owl-factor.o +clk-owl-y += owl-composite.o +clk-owl-y += owl-pll.o + +# SoC support +obj-$(CONFIG_CLK_OWL_S900) += owl-s900.o diff --git a/drivers/clk/actions/owl-common.c b/drivers/clk/actions/owl-common.c new file mode 100644 index 000000000000..61c1071b5180 --- /dev/null +++ b/drivers/clk/actions/owl-common.c @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL common clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#include +#include +#include +#include + +#include "owl-common.h" + +static const struct regmap_config owl_regmap_config = { + .reg_bits = 32, + .reg_stride = 4, + .val_bits = 32, + .max_register = 0x00cc, + .fast_io = true, +}; + +static void owl_clk_set_regmap(const struct owl_clk_desc *desc, + struct regmap *regmap) +{ + int i; + struct owl_clk_common *clks; + + for (i = 0; i < desc->num_clks; i++) { + clks = desc->clks[i]; + if (!clks) + continue; + + clks->regmap = regmap; + } +} + +int owl_clk_regmap_init(struct platform_device *pdev, + const struct owl_clk_desc *desc) +{ + void __iomem *base; + struct regmap *regmap; + struct resource *res; + + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + base = devm_ioremap_resource(&pdev->dev, res); + if (IS_ERR(base)) + return PTR_ERR(base); + + regmap = devm_regmap_init_mmio(&pdev->dev, base, &owl_regmap_config); + if (IS_ERR(regmap)) { + pr_err("failed to init regmap\n"); + return PTR_ERR(regmap); + } + + owl_clk_set_regmap(desc, regmap); + + return 0; +} + +int owl_clk_probe(struct device *dev, struct clk_hw_onecell_data *hw_clks) +{ + int i, ret; + struct clk_hw *hw; + + for (i = 0; i < hw_clks->num; i++) { + + hw = hw_clks->hws[i]; + + if (IS_ERR_OR_NULL(hw)) + continue; + + ret = devm_clk_hw_register(dev, hw); + if (ret) { + dev_err(dev, "Couldn't register clock %d - %s\n", + i, hw->init->name); + return ret; + } + } + + ret = devm_of_clk_add_hw_provider(dev, of_clk_hw_onecell_get, hw_clks); + if (ret) + dev_err(dev, "Failed to add clock provider\n"); + + return ret; +} diff --git a/drivers/clk/actions/owl-common.h b/drivers/clk/actions/owl-common.h new file mode 100644 index 000000000000..4fd726ec54a6 --- /dev/null +++ b/drivers/clk/actions/owl-common.h @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL common clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#ifndef _OWL_COMMON_H_ +#define _OWL_COMMON_H_ + +#include +#include +#include + +struct device_node; + +struct owl_clk_common { + struct regmap *regmap; + struct clk_hw hw; +}; + +struct owl_clk_desc { + struct owl_clk_common **clks; + unsigned long num_clks; + struct clk_hw_onecell_data *hw_clks; +}; + +static inline struct owl_clk_common * + hw_to_owl_clk_common(const struct clk_hw *hw) +{ + return container_of(hw, struct owl_clk_common, hw); +} + +int owl_clk_regmap_init(struct platform_device *pdev, + const struct owl_clk_desc *desc); +int owl_clk_probe(struct device *dev, struct clk_hw_onecell_data *hw_clks); + +#endif /* _OWL_COMMON_H_ */ diff --git a/drivers/clk/actions/owl-composite.c b/drivers/clk/actions/owl-composite.c new file mode 100644 index 000000000000..101706e0c66f --- /dev/null +++ b/drivers/clk/actions/owl-composite.c @@ -0,0 +1,199 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL composite clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#include +#include + +#include "owl-composite.h" + +static u8 owl_comp_get_parent(struct clk_hw *hw) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + + return owl_mux_helper_get_parent(&comp->common, &comp->mux_hw); +} + +static int owl_comp_set_parent(struct clk_hw *hw, u8 index) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + + return owl_mux_helper_set_parent(&comp->common, &comp->mux_hw, index); +} + +static void owl_comp_disable(struct clk_hw *hw) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + struct owl_clk_common *common = &comp->common; + + owl_gate_set(common, &comp->gate_hw, false); +} + +static int owl_comp_enable(struct clk_hw *hw) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + struct owl_clk_common *common = &comp->common; + + owl_gate_set(common, &comp->gate_hw, true); + + return 0; +} + +static int owl_comp_is_enabled(struct clk_hw *hw) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + struct owl_clk_common *common = &comp->common; + + return owl_gate_clk_is_enabled(common, &comp->gate_hw); +} + +static long owl_comp_div_round_rate(struct clk_hw *hw, unsigned long rate, + unsigned long *parent_rate) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + + return owl_divider_helper_round_rate(&comp->common, &comp->rate.div_hw, + rate, parent_rate); +} + +static unsigned long owl_comp_div_recalc_rate(struct clk_hw *hw, + unsigned long parent_rate) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + + return owl_divider_helper_recalc_rate(&comp->common, &comp->rate.div_hw, + parent_rate); +} + +static int owl_comp_div_set_rate(struct clk_hw *hw, unsigned long rate, + unsigned long parent_rate) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + + return owl_divider_helper_set_rate(&comp->common, &comp->rate.div_hw, + rate, parent_rate); +} + +static long owl_comp_fact_round_rate(struct clk_hw *hw, unsigned long rate, + unsigned long *parent_rate) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + + return owl_factor_helper_round_rate(&comp->common, + &comp->rate.factor_hw, + rate, parent_rate); +} + +static unsigned long owl_comp_fact_recalc_rate(struct clk_hw *hw, + unsigned long parent_rate) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + + return owl_factor_helper_recalc_rate(&comp->common, + &comp->rate.factor_hw, + parent_rate); +} + +static int owl_comp_fact_set_rate(struct clk_hw *hw, unsigned long rate, + unsigned long parent_rate) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + + return owl_factor_helper_set_rate(&comp->common, + &comp->rate.factor_hw, + rate, parent_rate); +} + +static long owl_comp_fix_fact_round_rate(struct clk_hw *hw, unsigned long rate, + unsigned long *parent_rate) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + struct clk_fixed_factor *fix_fact_hw = &comp->rate.fix_fact_hw; + + return comp->fix_fact_ops->round_rate(&fix_fact_hw->hw, rate, parent_rate); +} + +static unsigned long owl_comp_fix_fact_recalc_rate(struct clk_hw *hw, + unsigned long parent_rate) +{ + struct owl_composite *comp = hw_to_owl_comp(hw); + struct clk_fixed_factor *fix_fact_hw = &comp->rate.fix_fact_hw; + + return comp->fix_fact_ops->recalc_rate(&fix_fact_hw->hw, parent_rate); + +} + +static int owl_comp_fix_fact_set_rate(struct clk_hw *hw, unsigned long rate, + unsigned long parent_rate) +{ + /* + * We must report success but we can do so unconditionally because + * owl_comp_fix_fact_round_rate returns values that ensure this call is + * a nop. + */ + + return 0; +} + +const struct clk_ops owl_comp_div_ops = { + /* mux_ops */ + .get_parent = owl_comp_get_parent, + .set_parent = owl_comp_set_parent, + + /* gate_ops */ + .disable = owl_comp_disable, + .enable = owl_comp_enable, + .is_enabled = owl_comp_is_enabled, + + /* div_ops */ + .round_rate = owl_comp_div_round_rate, + .recalc_rate = owl_comp_div_recalc_rate, + .set_rate = owl_comp_div_set_rate, +}; + + +const struct clk_ops owl_comp_fact_ops = { + /* mux_ops */ + .get_parent = owl_comp_get_parent, + .set_parent = owl_comp_set_parent, + + /* gate_ops */ + .disable = owl_comp_disable, + .enable = owl_comp_enable, + .is_enabled = owl_comp_is_enabled, + + /* fact_ops */ + .round_rate = owl_comp_fact_round_rate, + .recalc_rate = owl_comp_fact_recalc_rate, + .set_rate = owl_comp_fact_set_rate, +}; + +const struct clk_ops owl_comp_fix_fact_ops = { + /* gate_ops */ + .disable = owl_comp_disable, + .enable = owl_comp_enable, + .is_enabled = owl_comp_is_enabled, + + /* fix_fact_ops */ + .round_rate = owl_comp_fix_fact_round_rate, + .recalc_rate = owl_comp_fix_fact_recalc_rate, + .set_rate = owl_comp_fix_fact_set_rate, +}; + + +const struct clk_ops owl_comp_pass_ops = { + /* mux_ops */ + .get_parent = owl_comp_get_parent, + .set_parent = owl_comp_set_parent, + + /* gate_ops */ + .disable = owl_comp_disable, + .enable = owl_comp_enable, + .is_enabled = owl_comp_is_enabled, +}; diff --git a/drivers/clk/actions/owl-composite.h b/drivers/clk/actions/owl-composite.h new file mode 100644 index 000000000000..b410ed5bf308 --- /dev/null +++ b/drivers/clk/actions/owl-composite.h @@ -0,0 +1,124 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL composite clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#ifndef _OWL_COMPOSITE_H_ +#define _OWL_COMPOSITE_H_ + +#include "owl-common.h" +#include "owl-mux.h" +#include "owl-gate.h" +#include "owl-factor.h" +#include "owl-fixed-factor.h" +#include "owl-divider.h" + +union owl_rate { + struct owl_divider_hw div_hw; + struct owl_factor_hw factor_hw; + struct clk_fixed_factor fix_fact_hw; +}; + +struct owl_composite { + struct owl_mux_hw mux_hw; + struct owl_gate_hw gate_hw; + union owl_rate rate; + + const struct clk_ops *fix_fact_ops; + + struct owl_clk_common common; +}; + +#define OWL_COMP_DIV(_struct, _name, _parent, \ + _mux, _gate, _div, _flags) \ + struct owl_composite _struct = { \ + .mux_hw = _mux, \ + .gate_hw = _gate, \ + .rate.div_hw = _div, \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT_PARENTS(_name, \ + _parent, \ + &owl_comp_div_ops,\ + _flags), \ + }, \ + } + +#define OWL_COMP_DIV_FIXED(_struct, _name, _parent, \ + _gate, _div, _flags) \ + struct owl_composite _struct = { \ + .gate_hw = _gate, \ + .rate.div_hw = _div, \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT(_name, \ + _parent, \ + &owl_comp_div_ops,\ + _flags), \ + }, \ + } + +#define OWL_COMP_FACTOR(_struct, _name, _parent, \ + _mux, _gate, _factor, _flags) \ + struct owl_composite _struct = { \ + .mux_hw = _mux, \ + .gate_hw = _gate, \ + .rate.factor_hw = _factor, \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT_PARENTS(_name, \ + _parent, \ + &owl_comp_fact_ops,\ + _flags), \ + }, \ + } + +#define OWL_COMP_FIXED_FACTOR(_struct, _name, _parent, \ + _gate, _mul, _div, _flags) \ + struct owl_composite _struct = { \ + .gate_hw = _gate, \ + .rate.fix_fact_hw.mult = _mul, \ + .rate.fix_fact_hw.div = _div, \ + .fix_fact_ops = &clk_fixed_factor_ops, \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT(_name, \ + _parent, \ + &owl_comp_fix_fact_ops,\ + _flags), \ + }, \ + } + +#define OWL_COMP_PASS(_struct, _name, _parent, \ + _mux, _gate, _flags) \ + struct owl_composite _struct = { \ + .mux_hw = _mux, \ + .gate_hw = _gate, \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT_PARENTS(_name, \ + _parent, \ + &owl_comp_pass_ops,\ + _flags), \ + }, \ + } + +static inline struct owl_composite *hw_to_owl_comp(const struct clk_hw *hw) +{ + struct owl_clk_common *common = hw_to_owl_clk_common(hw); + + return container_of(common, struct owl_composite, common); +} + +extern const struct clk_ops owl_comp_div_ops; +extern const struct clk_ops owl_comp_fact_ops; +extern const struct clk_ops owl_comp_fix_fact_ops; +extern const struct clk_ops owl_comp_pass_ops; +extern const struct clk_ops clk_fixed_factor_ops; + +#endif /* _OWL_COMPOSITE_H_ */ diff --git a/drivers/clk/actions/owl-divider.c b/drivers/clk/actions/owl-divider.c new file mode 100644 index 000000000000..cddac00fe324 --- /dev/null +++ b/drivers/clk/actions/owl-divider.c @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL divider clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#include +#include + +#include "owl-divider.h" + +long owl_divider_helper_round_rate(struct owl_clk_common *common, + const struct owl_divider_hw *div_hw, + unsigned long rate, + unsigned long *parent_rate) +{ + return divider_round_rate(&common->hw, rate, parent_rate, + div_hw->table, div_hw->width, + div_hw->div_flags); +} + +static long owl_divider_round_rate(struct clk_hw *hw, unsigned long rate, + unsigned long *parent_rate) +{ + struct owl_divider *div = hw_to_owl_divider(hw); + + return owl_divider_helper_round_rate(&div->common, &div->div_hw, + rate, parent_rate); +} + +unsigned long owl_divider_helper_recalc_rate(struct owl_clk_common *common, + const struct owl_divider_hw *div_hw, + unsigned long parent_rate) +{ + unsigned long val; + unsigned int reg; + + regmap_read(common->regmap, div_hw->reg, ®); + val = reg >> div_hw->shift; + val &= (1 << div_hw->width) - 1; + + return divider_recalc_rate(&common->hw, parent_rate, + val, div_hw->table, + div_hw->div_flags, + div_hw->width); +} + +static unsigned long owl_divider_recalc_rate(struct clk_hw *hw, + unsigned long parent_rate) +{ + struct owl_divider *div = hw_to_owl_divider(hw); + + return owl_divider_helper_recalc_rate(&div->common, + &div->div_hw, parent_rate); +} + +int owl_divider_helper_set_rate(const struct owl_clk_common *common, + const struct owl_divider_hw *div_hw, + unsigned long rate, + unsigned long parent_rate) +{ + unsigned long val; + unsigned int reg; + + val = divider_get_val(rate, parent_rate, div_hw->table, + div_hw->width, 0); + + regmap_read(common->regmap, div_hw->reg, ®); + reg &= ~GENMASK(div_hw->width + div_hw->shift - 1, div_hw->shift); + + regmap_write(common->regmap, div_hw->reg, + reg | (val << div_hw->shift)); + + return 0; +} + +static int owl_divider_set_rate(struct clk_hw *hw, unsigned long rate, + unsigned long parent_rate) +{ + struct owl_divider *div = hw_to_owl_divider(hw); + + return owl_divider_helper_set_rate(&div->common, &div->div_hw, + rate, parent_rate); +} + +const struct clk_ops owl_divider_ops = { + .recalc_rate = owl_divider_recalc_rate, + .round_rate = owl_divider_round_rate, + .set_rate = owl_divider_set_rate, +}; diff --git a/drivers/clk/actions/owl-divider.h b/drivers/clk/actions/owl-divider.h new file mode 100644 index 000000000000..92d3e3d23967 --- /dev/null +++ b/drivers/clk/actions/owl-divider.h @@ -0,0 +1,75 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL divider clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#ifndef _OWL_DIVIDER_H_ +#define _OWL_DIVIDER_H_ + +#include "owl-common.h" + +struct owl_divider_hw { + u32 reg; + u8 shift; + u8 width; + u8 div_flags; + struct clk_div_table *table; +}; + +struct owl_divider { + struct owl_divider_hw div_hw; + struct owl_clk_common common; +}; + +#define OWL_DIVIDER_HW(_reg, _shift, _width, _div_flags, _table) \ + { \ + .reg = _reg, \ + .shift = _shift, \ + .width = _width, \ + .div_flags = _div_flags, \ + .table = _table, \ + } + +#define OWL_DIVIDER(_struct, _name, _parent, _reg, \ + _shift, _width, _table, _div_flags, _flags) \ + struct owl_divider _struct = { \ + .div_hw = OWL_DIVIDER_HW(_reg, _shift, _width, \ + _div_flags, _table), \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT(_name, \ + _parent, \ + &owl_divider_ops, \ + _flags), \ + }, \ + } + +static inline struct owl_divider *hw_to_owl_divider(const struct clk_hw *hw) +{ + struct owl_clk_common *common = hw_to_owl_clk_common(hw); + + return container_of(common, struct owl_divider, common); +} + +long owl_divider_helper_round_rate(struct owl_clk_common *common, + const struct owl_divider_hw *div_hw, + unsigned long rate, + unsigned long *parent_rate); + +unsigned long owl_divider_helper_recalc_rate(struct owl_clk_common *common, + const struct owl_divider_hw *div_hw, + unsigned long parent_rate); + +int owl_divider_helper_set_rate(const struct owl_clk_common *common, + const struct owl_divider_hw *div_hw, + unsigned long rate, + unsigned long parent_rate); + +extern const struct clk_ops owl_divider_ops; + +#endif /* _OWL_DIVIDER_H_ */ diff --git a/drivers/clk/actions/owl-factor.c b/drivers/clk/actions/owl-factor.c new file mode 100644 index 000000000000..317d4a9e112e --- /dev/null +++ b/drivers/clk/actions/owl-factor.c @@ -0,0 +1,222 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL factor clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#include +#include +#include + +#include "owl-factor.h" + +static unsigned int _get_table_maxval(const struct clk_factor_table *table) +{ + unsigned int maxval = 0; + const struct clk_factor_table *clkt; + + for (clkt = table; clkt->div; clkt++) + if (clkt->val > maxval) + maxval = clkt->val; + return maxval; +} + +static int _get_table_div_mul(const struct clk_factor_table *table, + unsigned int val, unsigned int *mul, unsigned int *div) +{ + const struct clk_factor_table *clkt; + + for (clkt = table; clkt->div; clkt++) { + if (clkt->val == val) { + *mul = clkt->mul; + *div = clkt->div; + return 1; + } + } + + return 0; +} + +static unsigned int _get_table_val(const struct clk_factor_table *table, + unsigned long rate, unsigned long parent_rate) +{ + const struct clk_factor_table *clkt; + int val = -1; + u64 calc_rate; + + for (clkt = table; clkt->div; clkt++) { + calc_rate = parent_rate * clkt->mul; + do_div(calc_rate, clkt->div); + + if ((unsigned long)calc_rate <= rate) { + val = clkt->val; + break; + } + } + + if (val == -1) + val = _get_table_maxval(table); + + return val; +} + +static int clk_val_best(struct clk_hw *hw, unsigned long rate, + unsigned long *best_parent_rate) +{ + struct owl_factor *factor = hw_to_owl_factor(hw); + struct owl_factor_hw *factor_hw = &factor->factor_hw; + const struct clk_factor_table *clkt = factor_hw->table; + unsigned long parent_rate, try_parent_rate, best = 0, cur_rate; + unsigned long parent_rate_saved = *best_parent_rate; + int bestval = 0; + + if (!rate) + rate = 1; + + if (!(clk_hw_get_flags(hw) & CLK_SET_RATE_PARENT)) { + parent_rate = *best_parent_rate; + bestval = _get_table_val(clkt, rate, parent_rate); + return bestval; + } + + for (clkt = factor_hw->table; clkt->div; clkt++) { + try_parent_rate = rate * clkt->div / clkt->mul; + + if (try_parent_rate == parent_rate_saved) { + pr_debug("%s: [%d %d %d] found try_parent_rate %ld\n", + __func__, clkt->val, clkt->mul, clkt->div, + try_parent_rate); + /* + * It's the most ideal case if the requested rate can be + * divided from parent clock without any need to change + * parent rate, so return the divider immediately. + */ + *best_parent_rate = parent_rate_saved; + return clkt->val; + } + + parent_rate = clk_hw_round_rate(clk_hw_get_parent(hw), + try_parent_rate); + cur_rate = DIV_ROUND_UP(parent_rate, clkt->div) * clkt->mul; + if (cur_rate <= rate && cur_rate > best) { + bestval = clkt->val; + best = cur_rate; + *best_parent_rate = parent_rate; + } + } + + if (!bestval) { + bestval = _get_table_maxval(clkt); + *best_parent_rate = clk_hw_round_rate( + clk_hw_get_parent(hw), 1); + } + + return bestval; +} + +long owl_factor_helper_round_rate(struct owl_clk_common *common, + const struct owl_factor_hw *factor_hw, + unsigned long rate, + unsigned long *parent_rate) +{ + const struct clk_factor_table *clkt = factor_hw->table; + unsigned int val, mul = 0, div = 1; + + val = clk_val_best(&common->hw, rate, parent_rate); + _get_table_div_mul(clkt, val, &mul, &div); + + return *parent_rate * mul / div; +} + +static long owl_factor_round_rate(struct clk_hw *hw, unsigned long rate, + unsigned long *parent_rate) +{ + struct owl_factor *factor = hw_to_owl_factor(hw); + struct owl_factor_hw *factor_hw = &factor->factor_hw; + + return owl_factor_helper_round_rate(&factor->common, factor_hw, + rate, parent_rate); +} + +unsigned long owl_factor_helper_recalc_rate(struct owl_clk_common *common, + const struct owl_factor_hw *factor_hw, + unsigned long parent_rate) +{ + const struct clk_factor_table *clkt = factor_hw->table; + unsigned long long int rate; + u32 reg, val, mul, div; + + div = 0; + mul = 0; + + regmap_read(common->regmap, factor_hw->reg, ®); + + val = reg >> factor_hw->shift; + val &= div_mask(factor_hw); + + _get_table_div_mul(clkt, val, &mul, &div); + if (!div) { + WARN(!(factor_hw->fct_flags & CLK_DIVIDER_ALLOW_ZERO), + "%s: Zero divisor and CLK_DIVIDER_ALLOW_ZERO not set\n", + __clk_get_name(common->hw.clk)); + return parent_rate; + } + + rate = (unsigned long long int)parent_rate * mul; + do_div(rate, div); + + return rate; +} + +static unsigned long owl_factor_recalc_rate(struct clk_hw *hw, + unsigned long parent_rate) +{ + struct owl_factor *factor = hw_to_owl_factor(hw); + struct owl_factor_hw *factor_hw = &factor->factor_hw; + struct owl_clk_common *common = &factor->common; + + return owl_factor_helper_recalc_rate(common, factor_hw, parent_rate); +} + +int owl_factor_helper_set_rate(const struct owl_clk_common *common, + const struct owl_factor_hw *factor_hw, + unsigned long rate, + unsigned long parent_rate) +{ + u32 val, reg; + + val = _get_table_val(factor_hw->table, rate, parent_rate); + + if (val > div_mask(factor_hw)) + val = div_mask(factor_hw); + + regmap_read(common->regmap, factor_hw->reg, ®); + + reg &= ~(div_mask(factor_hw) << factor_hw->shift); + reg |= val << factor_hw->shift; + + regmap_write(common->regmap, factor_hw->reg, reg); + + return 0; +} + +static int owl_factor_set_rate(struct clk_hw *hw, unsigned long rate, + unsigned long parent_rate) +{ + struct owl_factor *factor = hw_to_owl_factor(hw); + struct owl_factor_hw *factor_hw = &factor->factor_hw; + struct owl_clk_common *common = &factor->common; + + return owl_factor_helper_set_rate(common, factor_hw, + rate, parent_rate); +} + +const struct clk_ops owl_factor_ops = { + .round_rate = owl_factor_round_rate, + .recalc_rate = owl_factor_recalc_rate, + .set_rate = owl_factor_set_rate, +}; diff --git a/drivers/clk/actions/owl-factor.h b/drivers/clk/actions/owl-factor.h new file mode 100644 index 000000000000..f1a7ffe896e1 --- /dev/null +++ b/drivers/clk/actions/owl-factor.h @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL factor clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#ifndef _OWL_FACTOR_H_ +#define _OWL_FACTOR_H_ + +#include "owl-common.h" + +struct clk_factor_table { + unsigned int val; + unsigned int mul; + unsigned int div; +}; + +struct owl_factor_hw { + u32 reg; + u8 shift; + u8 width; + u8 fct_flags; + struct clk_factor_table *table; +}; + +struct owl_factor { + struct owl_factor_hw factor_hw; + struct owl_clk_common common; +}; + +#define OWL_FACTOR_HW(_reg, _shift, _width, _fct_flags, _table) \ + { \ + .reg = _reg, \ + .shift = _shift, \ + .width = _width, \ + .fct_flags = _fct_flags, \ + .table = _table, \ + } + +#define OWL_FACTOR(_struct, _name, _parent, _reg, \ + _shift, _width, _table, _fct_flags, _flags) \ + struct owl_factor _struct = { \ + .factor_hw = OWL_FACTOR_HW(_reg, _shift, \ + _width, _fct_flags, _table), \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT(_name, \ + _parent, \ + &owl_factor_ops, \ + _flags), \ + }, \ + } + +#define div_mask(d) ((1 << ((d)->width)) - 1) + +static inline struct owl_factor *hw_to_owl_factor(const struct clk_hw *hw) +{ + struct owl_clk_common *common = hw_to_owl_clk_common(hw); + + return container_of(common, struct owl_factor, common); +} + +long owl_factor_helper_round_rate(struct owl_clk_common *common, + const struct owl_factor_hw *factor_hw, + unsigned long rate, + unsigned long *parent_rate); + +unsigned long owl_factor_helper_recalc_rate(struct owl_clk_common *common, + const struct owl_factor_hw *factor_hw, + unsigned long parent_rate); + +int owl_factor_helper_set_rate(const struct owl_clk_common *common, + const struct owl_factor_hw *factor_hw, + unsigned long rate, + unsigned long parent_rate); + +extern const struct clk_ops owl_factor_ops; + +#endif /* _OWL_FACTOR_H_ */ diff --git a/drivers/clk/actions/owl-fixed-factor.h b/drivers/clk/actions/owl-fixed-factor.h new file mode 100644 index 000000000000..cc9fe36c0964 --- /dev/null +++ b/drivers/clk/actions/owl-fixed-factor.h @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL fixed factor clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#ifndef _OWL_FIXED_FACTOR_H_ +#define _OWL_FIXED_FACTOR_H_ + +#include "owl-common.h" + +#define OWL_FIX_FACT(_struct, _name, _parent, _mul, _div, _flags) \ + struct clk_fixed_factor _struct = { \ + .mult = _mul, \ + .div = _div, \ + .hw.init = CLK_HW_INIT(_name, \ + _parent, \ + &clk_fixed_factor_ops, \ + _flags), \ + } + +extern const struct clk_ops clk_fixed_factor_ops; + +#endif /* _OWL_FIXED_FACTOR_H_ */ diff --git a/drivers/clk/actions/owl-gate.c b/drivers/clk/actions/owl-gate.c new file mode 100644 index 000000000000..f11500ba46a7 --- /dev/null +++ b/drivers/clk/actions/owl-gate.c @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL gate clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#include +#include + +#include "owl-gate.h" + +void owl_gate_set(const struct owl_clk_common *common, + const struct owl_gate_hw *gate_hw, bool enable) +{ + int set = gate_hw->gate_flags & CLK_GATE_SET_TO_DISABLE ? 1 : 0; + u32 reg; + + set ^= enable; + + regmap_read(common->regmap, gate_hw->reg, ®); + + if (set) + reg |= BIT(gate_hw->bit_idx); + else + reg &= ~BIT(gate_hw->bit_idx); + + regmap_write(common->regmap, gate_hw->reg, reg); +} + +static void owl_gate_disable(struct clk_hw *hw) +{ + struct owl_gate *gate = hw_to_owl_gate(hw); + struct owl_clk_common *common = &gate->common; + + owl_gate_set(common, &gate->gate_hw, false); +} + +static int owl_gate_enable(struct clk_hw *hw) +{ + struct owl_gate *gate = hw_to_owl_gate(hw); + struct owl_clk_common *common = &gate->common; + + owl_gate_set(common, &gate->gate_hw, true); + + return 0; +} + +int owl_gate_clk_is_enabled(const struct owl_clk_common *common, + const struct owl_gate_hw *gate_hw) +{ + u32 reg; + + regmap_read(common->regmap, gate_hw->reg, ®); + + if (gate_hw->gate_flags & CLK_GATE_SET_TO_DISABLE) + reg ^= BIT(gate_hw->bit_idx); + + return !!(reg & BIT(gate_hw->bit_idx)); +} + +static int owl_gate_is_enabled(struct clk_hw *hw) +{ + struct owl_gate *gate = hw_to_owl_gate(hw); + struct owl_clk_common *common = &gate->common; + + return owl_gate_clk_is_enabled(common, &gate->gate_hw); +} + +const struct clk_ops owl_gate_ops = { + .disable = owl_gate_disable, + .enable = owl_gate_enable, + .is_enabled = owl_gate_is_enabled, +}; diff --git a/drivers/clk/actions/owl-gate.h b/drivers/clk/actions/owl-gate.h new file mode 100644 index 000000000000..c2d61ceebce2 --- /dev/null +++ b/drivers/clk/actions/owl-gate.h @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL gate clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#ifndef _OWL_GATE_H_ +#define _OWL_GATE_H_ + +#include "owl-common.h" + +struct owl_gate_hw { + u32 reg; + u8 bit_idx; + u8 gate_flags; +}; + +struct owl_gate { + struct owl_gate_hw gate_hw; + struct owl_clk_common common; +}; + +#define OWL_GATE_HW(_reg, _bit_idx, _gate_flags) \ + { \ + .reg = _reg, \ + .bit_idx = _bit_idx, \ + .gate_flags = _gate_flags, \ + } + +#define OWL_GATE(_struct, _name, _parent, _reg, \ + _bit_idx, _gate_flags, _flags) \ + struct owl_gate _struct = { \ + .gate_hw = OWL_GATE_HW(_reg, _bit_idx, _gate_flags), \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT(_name, \ + _parent, \ + &owl_gate_ops, \ + _flags), \ + } \ + } \ + +#define OWL_GATE_NO_PARENT(_struct, _name, _reg, \ + _bit_idx, _gate_flags, _flags) \ + struct owl_gate _struct = { \ + .gate_hw = OWL_GATE_HW(_reg, _bit_idx, _gate_flags), \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT_NO_PARENT(_name, \ + &owl_gate_ops, \ + _flags), \ + }, \ + } \ + +static inline struct owl_gate *hw_to_owl_gate(const struct clk_hw *hw) +{ + struct owl_clk_common *common = hw_to_owl_clk_common(hw); + + return container_of(common, struct owl_gate, common); +} + +void owl_gate_set(const struct owl_clk_common *common, + const struct owl_gate_hw *gate_hw, bool enable); +int owl_gate_clk_is_enabled(const struct owl_clk_common *common, + const struct owl_gate_hw *gate_hw); + +extern const struct clk_ops owl_gate_ops; + +#endif /* _OWL_GATE_H_ */ diff --git a/drivers/clk/actions/owl-mux.c b/drivers/clk/actions/owl-mux.c new file mode 100644 index 000000000000..f9c6cf2540e4 --- /dev/null +++ b/drivers/clk/actions/owl-mux.c @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL mux clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#include +#include + +#include "owl-mux.h" + +u8 owl_mux_helper_get_parent(const struct owl_clk_common *common, + const struct owl_mux_hw *mux_hw) +{ + u32 reg; + u8 parent; + + regmap_read(common->regmap, mux_hw->reg, ®); + parent = reg >> mux_hw->shift; + parent &= BIT(mux_hw->width) - 1; + + return parent; +} + +static u8 owl_mux_get_parent(struct clk_hw *hw) +{ + struct owl_mux *mux = hw_to_owl_mux(hw); + + return owl_mux_helper_get_parent(&mux->common, &mux->mux_hw); +} + +int owl_mux_helper_set_parent(const struct owl_clk_common *common, + struct owl_mux_hw *mux_hw, u8 index) +{ + u32 reg; + + regmap_read(common->regmap, mux_hw->reg, ®); + reg &= ~GENMASK(mux_hw->width + mux_hw->shift - 1, mux_hw->shift); + regmap_write(common->regmap, mux_hw->reg, + reg | (index << mux_hw->shift)); + + return 0; +} + +static int owl_mux_set_parent(struct clk_hw *hw, u8 index) +{ + struct owl_mux *mux = hw_to_owl_mux(hw); + + return owl_mux_helper_set_parent(&mux->common, &mux->mux_hw, index); +} + +const struct clk_ops owl_mux_ops = { + .get_parent = owl_mux_get_parent, + .set_parent = owl_mux_set_parent, + .determine_rate = __clk_mux_determine_rate, +}; diff --git a/drivers/clk/actions/owl-mux.h b/drivers/clk/actions/owl-mux.h new file mode 100644 index 000000000000..834284c8c3ae --- /dev/null +++ b/drivers/clk/actions/owl-mux.h @@ -0,0 +1,61 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL mux clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#ifndef _OWL_MUX_H_ +#define _OWL_MUX_H_ + +#include "owl-common.h" + +struct owl_mux_hw { + u32 reg; + u8 shift; + u8 width; +}; + +struct owl_mux { + struct owl_mux_hw mux_hw; + struct owl_clk_common common; +}; + +#define OWL_MUX_HW(_reg, _shift, _width) \ + { \ + .reg = _reg, \ + .shift = _shift, \ + .width = _width, \ + } + +#define OWL_MUX(_struct, _name, _parents, _reg, \ + _shift, _width, _flags) \ + struct owl_mux _struct = { \ + .mux_hw = OWL_MUX_HW(_reg, _shift, _width), \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT_PARENTS(_name, \ + _parents, \ + &owl_mux_ops, \ + _flags), \ + }, \ + } + +static inline struct owl_mux *hw_to_owl_mux(const struct clk_hw *hw) +{ + struct owl_clk_common *common = hw_to_owl_clk_common(hw); + + return container_of(common, struct owl_mux, common); +} + +u8 owl_mux_helper_get_parent(const struct owl_clk_common *common, + const struct owl_mux_hw *mux_hw); +int owl_mux_helper_set_parent(const struct owl_clk_common *common, + struct owl_mux_hw *mux_hw, u8 index); + +extern const struct clk_ops owl_mux_ops; + +#endif /* _OWL_MUX_H_ */ diff --git a/drivers/clk/actions/owl-pll.c b/drivers/clk/actions/owl-pll.c new file mode 100644 index 000000000000..058e06d7099f --- /dev/null +++ b/drivers/clk/actions/owl-pll.c @@ -0,0 +1,194 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL pll clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#include +#include +#include +#include + +#include "owl-pll.h" + +static u32 owl_pll_calculate_mul(struct owl_pll_hw *pll_hw, unsigned long rate) +{ + u32 mul; + + mul = DIV_ROUND_CLOSEST(rate, pll_hw->bfreq); + if (mul < pll_hw->min_mul) + mul = pll_hw->min_mul; + else if (mul > pll_hw->max_mul) + mul = pll_hw->max_mul; + + return mul &= mul_mask(pll_hw); +} + +static unsigned long _get_table_rate(const struct clk_pll_table *table, + unsigned int val) +{ + const struct clk_pll_table *clkt; + + for (clkt = table; clkt->rate; clkt++) + if (clkt->val == val) + return clkt->rate; + + return 0; +} + +static const struct clk_pll_table *_get_pll_table( + const struct clk_pll_table *table, unsigned long rate) +{ + const struct clk_pll_table *clkt; + + for (clkt = table; clkt->rate; clkt++) { + if (clkt->rate == rate) { + table = clkt; + break; + } else if (clkt->rate < rate) + table = clkt; + } + + return table; +} + +static long owl_pll_round_rate(struct clk_hw *hw, unsigned long rate, + unsigned long *parent_rate) +{ + struct owl_pll *pll = hw_to_owl_pll(hw); + struct owl_pll_hw *pll_hw = &pll->pll_hw; + const struct clk_pll_table *clkt; + u32 mul; + + if (pll_hw->table) { + clkt = _get_pll_table(pll_hw->table, rate); + return clkt->rate; + } + + /* fixed frequency */ + if (pll_hw->width == 0) + return pll_hw->bfreq; + + mul = owl_pll_calculate_mul(pll_hw, rate); + + return pll_hw->bfreq * mul; +} + +static unsigned long owl_pll_recalc_rate(struct clk_hw *hw, + unsigned long parent_rate) +{ + struct owl_pll *pll = hw_to_owl_pll(hw); + struct owl_pll_hw *pll_hw = &pll->pll_hw; + const struct owl_clk_common *common = &pll->common; + u32 val; + + if (pll_hw->table) { + regmap_read(common->regmap, pll_hw->reg, &val); + + val = val >> pll_hw->shift; + val &= mul_mask(pll_hw); + + return _get_table_rate(pll_hw->table, val); + } + + /* fixed frequency */ + if (pll_hw->width == 0) + return pll_hw->bfreq; + + regmap_read(common->regmap, pll_hw->reg, &val); + + val = val >> pll_hw->shift; + val &= mul_mask(pll_hw); + + return pll_hw->bfreq * val; +} + +static int owl_pll_is_enabled(struct clk_hw *hw) +{ + struct owl_pll *pll = hw_to_owl_pll(hw); + struct owl_pll_hw *pll_hw = &pll->pll_hw; + const struct owl_clk_common *common = &pll->common; + u32 reg; + + regmap_read(common->regmap, pll_hw->reg, ®); + + return !!(reg & BIT(pll_hw->bit_idx)); +} + +static void owl_pll_set(const struct owl_clk_common *common, + const struct owl_pll_hw *pll_hw, bool enable) +{ + u32 reg; + + regmap_read(common->regmap, pll_hw->reg, ®); + + if (enable) + reg |= BIT(pll_hw->bit_idx); + else + reg &= ~BIT(pll_hw->bit_idx); + + regmap_write(common->regmap, pll_hw->reg, reg); +} + +static int owl_pll_enable(struct clk_hw *hw) +{ + struct owl_pll *pll = hw_to_owl_pll(hw); + const struct owl_clk_common *common = &pll->common; + + owl_pll_set(common, &pll->pll_hw, true); + + return 0; +} + +static void owl_pll_disable(struct clk_hw *hw) +{ + struct owl_pll *pll = hw_to_owl_pll(hw); + const struct owl_clk_common *common = &pll->common; + + owl_pll_set(common, &pll->pll_hw, false); +} + +static int owl_pll_set_rate(struct clk_hw *hw, unsigned long rate, + unsigned long parent_rate) +{ + struct owl_pll *pll = hw_to_owl_pll(hw); + struct owl_pll_hw *pll_hw = &pll->pll_hw; + const struct owl_clk_common *common = &pll->common; + const struct clk_pll_table *clkt; + u32 val, reg; + + /* fixed frequency */ + if (pll_hw->width == 0) + return 0; + + if (pll_hw->table) { + clkt = _get_pll_table(pll_hw->table, rate); + val = clkt->val; + } else { + val = owl_pll_calculate_mul(pll_hw, rate); + } + + regmap_read(common->regmap, pll_hw->reg, ®); + + reg &= ~mul_mask(pll_hw); + reg |= val << pll_hw->shift; + + regmap_write(common->regmap, pll_hw->reg, reg); + + udelay(PLL_STABILITY_WAIT_US); + + return 0; +} + +const struct clk_ops owl_pll_ops = { + .enable = owl_pll_enable, + .disable = owl_pll_disable, + .is_enabled = owl_pll_is_enabled, + .round_rate = owl_pll_round_rate, + .recalc_rate = owl_pll_recalc_rate, + .set_rate = owl_pll_set_rate, +}; diff --git a/drivers/clk/actions/owl-pll.h b/drivers/clk/actions/owl-pll.h new file mode 100644 index 000000000000..0aae30abd5dc --- /dev/null +++ b/drivers/clk/actions/owl-pll.h @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL pll clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#ifndef _OWL_PLL_H_ +#define _OWL_PLL_H_ + +#include "owl-common.h" + +/* last entry should have rate = 0 */ +struct clk_pll_table { + unsigned int val; + unsigned long rate; +}; + +struct owl_pll_hw { + u32 reg; + u32 bfreq; + u8 bit_idx; + u8 shift; + u8 width; + u8 min_mul; + u8 max_mul; + const struct clk_pll_table *table; +}; + +struct owl_pll { + struct owl_pll_hw pll_hw; + struct owl_clk_common common; +}; + +#define OWL_PLL_HW(_reg, _bfreq, _bit_idx, _shift, \ + _width, _min_mul, _max_mul, _table) \ + { \ + .reg = _reg, \ + .bfreq = _bfreq, \ + .bit_idx = _bit_idx, \ + .shift = _shift, \ + .width = _width, \ + .min_mul = _min_mul, \ + .max_mul = _max_mul, \ + .table = _table, \ + } + +#define OWL_PLL(_struct, _name, _parent, _reg, _bfreq, _bit_idx, \ + _shift, _width, _min_mul, _max_mul, _table, _flags) \ + struct owl_pll _struct = { \ + .pll_hw = OWL_PLL_HW(_reg, _bfreq, _bit_idx, _shift, \ + _width, _min_mul, \ + _max_mul, _table), \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT(_name, \ + _parent, \ + &owl_pll_ops, \ + _flags), \ + }, \ + } + +#define OWL_PLL_NO_PARENT(_struct, _name, _reg, _bfreq, _bit_idx, \ + _shift, _width, _min_mul, _max_mul, _table, _flags) \ + struct owl_pll _struct = { \ + .pll_hw = OWL_PLL_HW(_reg, _bfreq, _bit_idx, _shift, \ + _width, _min_mul, \ + _max_mul, _table), \ + .common = { \ + .regmap = NULL, \ + .hw.init = CLK_HW_INIT_NO_PARENT(_name, \ + &owl_pll_ops, \ + _flags), \ + }, \ + } + +#define mul_mask(m) ((1 << ((m)->width)) - 1) +#define PLL_STABILITY_WAIT_US (50) + +static inline struct owl_pll *hw_to_owl_pll(const struct clk_hw *hw) +{ + struct owl_clk_common *common = hw_to_owl_clk_common(hw); + + return container_of(common, struct owl_pll, common); +} + +extern const struct clk_ops owl_pll_ops; + +#endif /* _OWL_PLL_H_ */ diff --git a/drivers/clk/actions/owl-s900.c b/drivers/clk/actions/owl-s900.c new file mode 100644 index 000000000000..7f60ed6afe63 --- /dev/null +++ b/drivers/clk/actions/owl-s900.c @@ -0,0 +1,721 @@ +// SPDX-License-Identifier: GPL-2.0+ +// +// OWL S900 SoC clock driver +// +// Copyright (c) 2014 Actions Semi Inc. +// Author: David Liu +// +// Copyright (c) 2018 Linaro Ltd. +// Author: Manivannan Sadhasivam + +#include +#include + +#include "owl-common.h" +#include "owl-composite.h" +#include "owl-divider.h" +#include "owl-factor.h" +#include "owl-fixed-factor.h" +#include "owl-gate.h" +#include "owl-mux.h" +#include "owl-pll.h" + +#include + +#define CMU_COREPLL (0x0000) +#define CMU_DEVPLL (0x0004) +#define CMU_DDRPLL (0x0008) +#define CMU_NANDPLL (0x000C) +#define CMU_DISPLAYPLL (0x0010) +#define CMU_AUDIOPLL (0x0014) +#define CMU_TVOUTPLL (0x0018) +#define CMU_BUSCLK (0x001C) +#define CMU_SENSORCLK (0x0020) +#define CMU_LCDCLK (0x0024) +#define CMU_DSICLK (0x0028) +#define CMU_CSICLK (0x002C) +#define CMU_DECLK (0x0030) +#define CMU_BISPCLK (0x0034) +#define CMU_IMXCLK (0x0038) +#define CMU_HDECLK (0x003C) +#define CMU_VDECLK (0x0040) +#define CMU_VCECLK (0x0044) +#define CMU_NANDCCLK (0x004C) +#define CMU_SD0CLK (0x0050) +#define CMU_SD1CLK (0x0054) +#define CMU_SD2CLK (0x0058) +#define CMU_UART0CLK (0x005C) +#define CMU_UART1CLK (0x0060) +#define CMU_UART2CLK (0x0064) +#define CMU_PWM0CLK (0x0070) +#define CMU_PWM1CLK (0x0074) +#define CMU_PWM2CLK (0x0078) +#define CMU_PWM3CLK (0x007C) +#define CMU_USBPLL (0x0080) +#define CMU_ASSISTPLL (0x0084) +#define CMU_EDPCLK (0x0088) +#define CMU_GPU3DCLK (0x0090) +#define CMU_CORECTL (0x009C) +#define CMU_DEVCLKEN0 (0x00A0) +#define CMU_DEVCLKEN1 (0x00A4) +#define CMU_DEVRST0 (0x00A8) +#define CMU_DEVRST1 (0x00AC) +#define CMU_UART3CLK (0x00B0) +#define CMU_UART4CLK (0x00B4) +#define CMU_UART5CLK (0x00B8) +#define CMU_UART6CLK (0x00BC) +#define CMU_TLSCLK (0x00C0) +#define CMU_SD3CLK (0x00C4) +#define CMU_PWM4CLK (0x00C8) +#define CMU_PWM5CLK (0x00CC) + +static struct clk_pll_table clk_audio_pll_table[] = { + { 0, 45158400 }, { 1, 49152000 }, + { 0, 0 }, +}; + +static struct clk_pll_table clk_edp_pll_table[] = { + { 0, 810000000 }, { 1, 135000000 }, { 2, 270000000 }, + { 0, 0 }, +}; + +/* pll clocks */ +static OWL_PLL_NO_PARENT(core_pll_clk, "core_pll_clk", CMU_COREPLL, 24000000, 9, 0, 8, 5, 107, NULL, CLK_IGNORE_UNUSED); +static OWL_PLL_NO_PARENT(dev_pll_clk, "dev_pll_clk", CMU_DEVPLL, 6000000, 8, 0, 8, 20, 180, NULL, CLK_IGNORE_UNUSED); +static OWL_PLL_NO_PARENT(ddr_pll_clk, "ddr_pll_clk", CMU_DDRPLL, 24000000, 8, 0, 8, 5, 45, NULL, CLK_IGNORE_UNUSED); +static OWL_PLL_NO_PARENT(nand_pll_clk, "nand_pll_clk", CMU_NANDPLL, 6000000, 8, 0, 8, 4, 100, NULL, CLK_IGNORE_UNUSED); +static OWL_PLL_NO_PARENT(display_pll_clk, "display_pll_clk", CMU_DISPLAYPLL, 6000000, 8, 0, 8, 20, 180, NULL, CLK_IGNORE_UNUSED); +static OWL_PLL_NO_PARENT(assist_pll_clk, "assist_pll_clk", CMU_ASSISTPLL, 500000000, 0, 0, 0, 0, 0, NULL, CLK_IGNORE_UNUSED); +static OWL_PLL_NO_PARENT(audio_pll_clk, "audio_pll_clk", CMU_AUDIOPLL, 0, 4, 0, 1, 0, 0, clk_audio_pll_table, CLK_IGNORE_UNUSED); +static OWL_PLL(edp_pll_clk, "edp_pll_clk", "edp24M_clk", CMU_EDPCLK, 0, 9, 0, 2, 0, 0, clk_edp_pll_table, CLK_IGNORE_UNUSED); + +static const char *cpu_clk_mux_p[] = { "losc", "hosc", "core_pll_clk", }; +static const char *dev_clk_p[] = { "hosc", "dev_pll_clk", }; +static const char *noc_clk_mux_p[] = { "dev_clk", "assist_pll_clk", }; +static const char *dmm_clk_mux_p[] = { "dev_clk", "nand_pll_clk", "assist_pll_clk", "ddr_clk_src", }; +static const char *bisp_clk_mux_p[] = { "assist_pll_clk", "dev_clk", }; +static const char *csi_clk_mux_p[] = { "display_pll_clk", "dev_clk", }; +static const char *de_clk_mux_p[] = { "assist_pll_clk", "dev_clk", }; +static const char *gpu_clk_mux_p[] = { "dev_clk", "display_pll_clk", "ddr_clk_src", }; +static const char *hde_clk_mux_p[] = { "dev_clk", "display_pll_clk", "ddr_clk_src", }; +static const char *imx_clk_mux_p[] = { "assist_pll_clk", "dev_clk", }; +static const char *lcd_clk_mux_p[] = { "display_pll_clk", "nand_pll_clk", }; +static const char *nand_clk_mux_p[] = { "dev_clk", "nand_pll_clk", }; +static const char *sd_clk_mux_p[] = { "dev_clk", "nand_pll_clk", }; +static const char *sensor_clk_mux_p[] = { "hosc", "bisp_clk", }; +static const char *uart_clk_mux_p[] = { "hosc", "dev_pll_clk", }; +static const char *vce_clk_mux_p[] = { "dev_clk", "display_pll_clk", "assist_pll_clk", "ddr_clk_src", }; +static const char *i2s_clk_mux_p[] = { "audio_pll_clk", }; +static const char *edp_clk_mux_p[] = { "assist_pll_clk", "display_pll_clk", }; + +/* mux clocks */ +static OWL_MUX(cpu_clk, "cpu_clk", cpu_clk_mux_p, CMU_BUSCLK, 0, 2, CLK_SET_RATE_PARENT); +static OWL_MUX(dev_clk, "dev_clk", dev_clk_p, CMU_DEVPLL, 12, 1, CLK_SET_RATE_PARENT); +static OWL_MUX(noc_clk_mux, "noc_clk_mux", noc_clk_mux_p, CMU_BUSCLK, 7, 1, CLK_SET_RATE_PARENT); + +static struct clk_div_table nand_div_table[] = { + { 0, 1 }, { 1, 2 }, { 2, 4 }, { 3, 6 }, + { 4, 8 }, { 5, 10 }, { 6, 12 }, { 7, 14 }, + { 8, 16 }, { 9, 18 }, { 10, 20 }, { 11, 22 }, + { 12, 24 }, { 13, 26 }, { 14, 28 }, { 15, 30 }, + { 0, 0 }, +}; + +static struct clk_div_table apb_div_table[] = { + { 1, 2 }, { 2, 3 }, { 3, 4 }, + { 0, 0 }, +}; + +static struct clk_div_table eth_mac_div_table[] = { + { 0, 2 }, { 1, 4 }, + { 0, 0 }, +}; + +static struct clk_div_table rmii_ref_div_table[] = { + { 0, 4 }, { 1, 10 }, + { 0, 0 }, +}; + +static struct clk_div_table usb3_mac_div_table[] = { + { 1, 2 }, { 2, 3 }, { 3, 4 }, + { 0, 8 }, +}; + +static struct clk_div_table i2s_div_table[] = { + { 0, 1 }, { 1, 2 }, { 2, 3 }, { 3, 4 }, + { 4, 6 }, { 5, 8 }, { 6, 12 }, { 7, 16 }, + { 8, 24 }, + { 0, 0 }, +}; + +static struct clk_div_table hdmia_div_table[] = { + { 0, 1 }, { 1, 2 }, { 2, 3 }, { 3, 4 }, + { 4, 6 }, { 5, 8 }, { 6, 12 }, { 7, 16 }, + { 8, 24 }, + { 0, 0 }, +}; + +/* divider clocks */ +static OWL_DIVIDER(noc_clk_div, "noc_clk_div", "noc_clk", CMU_BUSCLK, 19, 1, NULL, 0, 0); +static OWL_DIVIDER(ahb_clk, "ahb_clk", "noc_clk_div", CMU_BUSCLK, 4, 1, NULL, 0, 0); +static OWL_DIVIDER(apb_clk, "apb_clk", "ahb_clk", CMU_BUSCLK, 8, 2, apb_div_table, 0, 0); +static OWL_DIVIDER(usb3_mac_clk, "usb3_mac_clk", "assist_pll_clk", CMU_ASSISTPLL, 12, 2, usb3_mac_div_table, 0, 0); +static OWL_DIVIDER(rmii_ref_clk, "rmii_ref_clk", "assist_pll_clk", CMU_ASSISTPLL, 8, 1, rmii_ref_div_table, 0, 0); + +static struct clk_factor_table sd_factor_table[] = { + /* bit0 ~ 4 */ + { 0, 1, 1 }, { 1, 1, 2 }, { 2, 1, 3 }, { 3, 1, 4 }, + { 4, 1, 5 }, { 5, 1, 6 }, { 6, 1, 7 }, { 7, 1, 8 }, + { 8, 1, 9 }, { 9, 1, 10 }, { 10, 1, 11 }, { 11, 1, 12 }, + { 12, 1, 13 }, { 13, 1, 14 }, { 14, 1, 15 }, { 15, 1, 16 }, + { 16, 1, 17 }, { 17, 1, 18 }, { 18, 1, 19 }, { 19, 1, 20 }, + { 20, 1, 21 }, { 21, 1, 22 }, { 22, 1, 23 }, { 23, 1, 24 }, + { 24, 1, 25 }, { 25, 1, 26 }, { 26, 1, 27 }, { 27, 1, 28 }, + { 28, 1, 29 }, { 29, 1, 30 }, { 30, 1, 31 }, { 31, 1, 32 }, + + /* bit8: /128 */ + { 256, 1, 1 * 128 }, { 257, 1, 2 * 128 }, { 258, 1, 3 * 128 }, { 259, 1, 4 * 128 }, + { 260, 1, 5 * 128 }, { 261, 1, 6 * 128 }, { 262, 1, 7 * 128 }, { 263, 1, 8 * 128 }, + { 264, 1, 9 * 128 }, { 265, 1, 10 * 128 }, { 266, 1, 11 * 128 }, { 267, 1, 12 * 128 }, + { 268, 1, 13 * 128 }, { 269, 1, 14 * 128 }, { 270, 1, 15 * 128 }, { 271, 1, 16 * 128 }, + { 272, 1, 17 * 128 }, { 273, 1, 18 * 128 }, { 274, 1, 19 * 128 }, { 275, 1, 20 * 128 }, + { 276, 1, 21 * 128 }, { 277, 1, 22 * 128 }, { 278, 1, 23 * 128 }, { 279, 1, 24 * 128 }, + { 280, 1, 25 * 128 }, { 281, 1, 26 * 128 }, { 282, 1, 27 * 128 }, { 283, 1, 28 * 128 }, + { 284, 1, 29 * 128 }, { 285, 1, 30 * 128 }, { 286, 1, 31 * 128 }, { 287, 1, 32 * 128 }, + + { 0, 0 }, +}; + +static struct clk_factor_table dmm_factor_table[] = { + { 0, 1, 1 }, { 1, 2, 3 }, { 2, 1, 2 }, { 3, 1, 3 }, + { 4, 1, 4 }, + { 0, 0, 0 }, +}; + +static struct clk_factor_table noc_factor_table[] = { + { 0, 1, 1 }, { 1, 2, 3 }, { 2, 1, 2 }, { 3, 1, 3 }, { 4, 1, 4 }, + { 0, 0, 0 }, +}; + +static struct clk_factor_table bisp_factor_table[] = { + { 0, 1, 1 }, { 1, 2, 3 }, { 2, 1, 2 }, { 3, 2, 5 }, + { 4, 1, 3 }, { 5, 1, 4 }, { 6, 1, 6 }, { 7, 1, 8 }, + { 0, 0, 0 }, +}; + +/* factor clocks */ +static OWL_FACTOR(noc_clk, "noc_clk", "noc_clk_mux", CMU_BUSCLK, 16, 3, noc_factor_table, 0, 0); +static OWL_FACTOR(de_clk1, "de_clk1", "de_clk", CMU_DECLK, 0, 3, bisp_factor_table, 0, 0); +static OWL_FACTOR(de_clk2, "de_clk2", "de_clk", CMU_DECLK, 4, 3, bisp_factor_table, 0, 0); +static OWL_FACTOR(de_clk3, "de_clk3", "de_clk", CMU_DECLK, 8, 3, bisp_factor_table, 0, 0); + +/* gate clocks */ +static OWL_GATE(gpio_clk, "gpio_clk", "apb_clk", CMU_DEVCLKEN0, 18, 0, 0); +static OWL_GATE_NO_PARENT(gpu_clk, "gpu_clk", CMU_DEVCLKEN0, 30, 0, 0); +static OWL_GATE(dmac_clk, "dmac_clk", "noc_clk_div", CMU_DEVCLKEN0, 1, 0, 0); +static OWL_GATE(timer_clk, "timer_clk", "hosc", CMU_DEVCLKEN1, 27, 0, 0); +static OWL_GATE_NO_PARENT(dsi_clk, "dsi_clk", CMU_DEVCLKEN0, 12, 0, 0); +static OWL_GATE(ddr0_clk, "ddr0_clk", "ddr_pll_clk", CMU_DEVCLKEN0, 31, 0, CLK_IGNORE_UNUSED); +static OWL_GATE(ddr1_clk, "ddr1_clk", "ddr_pll_clk", CMU_DEVCLKEN0, 29, 0, CLK_IGNORE_UNUSED); +static OWL_GATE_NO_PARENT(usb3_480mpll0_clk, "usb3_480mpll0_clk", CMU_USBPLL, 3, 0, 0); +static OWL_GATE_NO_PARENT(usb3_480mphy0_clk, "usb3_480mphy0_clk", CMU_USBPLL, 2, 0, 0); +static OWL_GATE_NO_PARENT(usb3_5gphy_clk, "usb3_5gphy_clk", CMU_USBPLL, 1, 0, 0); +static OWL_GATE_NO_PARENT(usb3_cce_clk, "usb3_cce_clk", CMU_USBPLL, 0, 0, 0); +static OWL_GATE(edp24M_clk, "edp24M_clk", "diff24M", CMU_EDPCLK, 8, 0, 0); +static OWL_GATE(edp_link_clk, "edp_link_clk", "edp_pll_clk", CMU_DEVCLKEN0, 10, 0, 0); +static OWL_GATE_NO_PARENT(usbh0_pllen_clk, "usbh0_pllen_clk", CMU_USBPLL, 12, 0, 0); +static OWL_GATE_NO_PARENT(usbh0_phy_clk, "usbh0_phy_clk", CMU_USBPLL, 10, 0, 0); +static OWL_GATE_NO_PARENT(usbh0_cce_clk, "usbh0_cce_clk", CMU_USBPLL, 8, 0, 0); +static OWL_GATE_NO_PARENT(usbh1_pllen_clk, "usbh1_pllen_clk", CMU_USBPLL, 13, 0, 0); +static OWL_GATE_NO_PARENT(usbh1_phy_clk, "usbh1_phy_clk", CMU_USBPLL, 11, 0, 0); +static OWL_GATE_NO_PARENT(usbh1_cce_clk, "usbh1_cce_clk", CMU_USBPLL, 9, 0, 0); +static OWL_GATE(spi0_clk, "spi0_clk", "ahb_clk", CMU_DEVCLKEN1, 10, 0, CLK_IGNORE_UNUSED); +static OWL_GATE(spi1_clk, "spi1_clk", "ahb_clk", CMU_DEVCLKEN1, 11, 0, CLK_IGNORE_UNUSED); +static OWL_GATE(spi2_clk, "spi2_clk", "ahb_clk", CMU_DEVCLKEN1, 12, 0, CLK_IGNORE_UNUSED); +static OWL_GATE(spi3_clk, "spi3_clk", "ahb_clk", CMU_DEVCLKEN1, 13, 0, CLK_IGNORE_UNUSED); + +/* composite clocks */ +static OWL_COMP_FACTOR(bisp_clk, "bisp_clk", bisp_clk_mux_p, + OWL_MUX_HW(CMU_BISPCLK, 4, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 14, 0), + OWL_FACTOR_HW(CMU_BISPCLK, 0, 3, 0, bisp_factor_table), + 0); + +static OWL_COMP_DIV(csi0_clk, "csi0_clk", csi_clk_mux_p, + OWL_MUX_HW(CMU_CSICLK, 4, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 13, 0), + OWL_DIVIDER_HW(CMU_CSICLK, 0, 4, 0, NULL), + 0); + +static OWL_COMP_DIV(csi1_clk, "csi1_clk", csi_clk_mux_p, + OWL_MUX_HW(CMU_CSICLK, 20, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 15, 0), + OWL_DIVIDER_HW(CMU_CSICLK, 16, 4, 0, NULL), + 0); + +static OWL_COMP_PASS(de_clk, "de_clk", de_clk_mux_p, + OWL_MUX_HW(CMU_DECLK, 12, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 8, 0), + 0); + +static OWL_COMP_FACTOR(dmm_clk, "dmm_clk", dmm_clk_mux_p, + OWL_MUX_HW(CMU_BUSCLK, 10, 2), + OWL_GATE_HW(CMU_DEVCLKEN0, 19, 0), + OWL_FACTOR_HW(CMU_BUSCLK, 12, 3, 0, dmm_factor_table), + CLK_IGNORE_UNUSED); + +static OWL_COMP_FACTOR(edp_clk, "edp_clk", edp_clk_mux_p, + OWL_MUX_HW(CMU_EDPCLK, 19, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 10, 0), + OWL_FACTOR_HW(CMU_EDPCLK, 16, 3, 0, bisp_factor_table), + 0); + +static OWL_COMP_DIV_FIXED(eth_mac_clk, "eth_mac_clk", "assist_pll_clk", + OWL_GATE_HW(CMU_DEVCLKEN1, 22, 0), + OWL_DIVIDER_HW(CMU_ASSISTPLL, 10, 1, 0, eth_mac_div_table), + 0); + +static OWL_COMP_FACTOR(gpu_core_clk, "gpu_core_clk", gpu_clk_mux_p, + OWL_MUX_HW(CMU_GPU3DCLK, 4, 2), + OWL_GATE_HW(CMU_GPU3DCLK, 15, 0), + OWL_FACTOR_HW(CMU_GPU3DCLK, 0, 3, 0, bisp_factor_table), + 0); + +static OWL_COMP_FACTOR(gpu_mem_clk, "gpu_mem_clk", gpu_clk_mux_p, + OWL_MUX_HW(CMU_GPU3DCLK, 20, 2), + OWL_GATE_HW(CMU_GPU3DCLK, 14, 0), + OWL_FACTOR_HW(CMU_GPU3DCLK, 16, 3, 0, bisp_factor_table), + 0); + +static OWL_COMP_FACTOR(gpu_sys_clk, "gpu_sys_clk", gpu_clk_mux_p, + OWL_MUX_HW(CMU_GPU3DCLK, 28, 2), + OWL_GATE_HW(CMU_GPU3DCLK, 13, 0), + OWL_FACTOR_HW(CMU_GPU3DCLK, 24, 3, 0, bisp_factor_table), + 0); + +static OWL_COMP_FACTOR(hde_clk, "hde_clk", hde_clk_mux_p, + OWL_MUX_HW(CMU_HDECLK, 4, 2), + OWL_GATE_HW(CMU_DEVCLKEN0, 27, 0), + OWL_FACTOR_HW(CMU_HDECLK, 0, 3, 0, bisp_factor_table), + 0); + +static OWL_COMP_DIV(hdmia_clk, "hdmia_clk", i2s_clk_mux_p, + OWL_MUX_HW(CMU_AUDIOPLL, 24, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 22, 0), + OWL_DIVIDER_HW(CMU_AUDIOPLL, 24, 4, 0, hdmia_div_table), + 0); + +static OWL_COMP_FIXED_FACTOR(i2c0_clk, "i2c0_clk", "assist_pll_clk", + OWL_GATE_HW(CMU_DEVCLKEN1, 14, 0), + 1, 5, 0); + +static OWL_COMP_FIXED_FACTOR(i2c1_clk, "i2c1_clk", "assist_pll_clk", + OWL_GATE_HW(CMU_DEVCLKEN1, 15, 0), + 1, 5, 0); + +static OWL_COMP_FIXED_FACTOR(i2c2_clk, "i2c2_clk", "assist_pll_clk", + OWL_GATE_HW(CMU_DEVCLKEN1, 30, 0), + 1, 5, 0); + +static OWL_COMP_FIXED_FACTOR(i2c3_clk, "i2c3_clk", "assist_pll_clk", + OWL_GATE_HW(CMU_DEVCLKEN1, 31, 0), + 1, 5, 0); + +static OWL_COMP_FIXED_FACTOR(i2c4_clk, "i2c4_clk", "assist_pll_clk", + OWL_GATE_HW(CMU_DEVCLKEN0, 17, 0), + 1, 5, 0); + +static OWL_COMP_FIXED_FACTOR(i2c5_clk, "i2c5_clk", "assist_pll_clk", + OWL_GATE_HW(CMU_DEVCLKEN1, 1, 0), + 1, 5, 0); + +static OWL_COMP_DIV(i2srx_clk, "i2srx_clk", i2s_clk_mux_p, + OWL_MUX_HW(CMU_AUDIOPLL, 24, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 21, 0), + OWL_DIVIDER_HW(CMU_AUDIOPLL, 20, 4, 0, i2s_div_table), + 0); + +static OWL_COMP_DIV(i2stx_clk, "i2stx_clk", i2s_clk_mux_p, + OWL_MUX_HW(CMU_AUDIOPLL, 24, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 20, 0), + OWL_DIVIDER_HW(CMU_AUDIOPLL, 16, 4, 0, i2s_div_table), + 0); + +static OWL_COMP_FACTOR(imx_clk, "imx_clk", imx_clk_mux_p, + OWL_MUX_HW(CMU_IMXCLK, 4, 1), + OWL_GATE_HW(CMU_DEVCLKEN1, 17, 0), + OWL_FACTOR_HW(CMU_IMXCLK, 0, 3, 0, bisp_factor_table), + 0); + +static OWL_COMP_DIV(lcd_clk, "lcd_clk", lcd_clk_mux_p, + OWL_MUX_HW(CMU_LCDCLK, 12, 2), + OWL_GATE_HW(CMU_DEVCLKEN0, 9, 0), + OWL_DIVIDER_HW(CMU_LCDCLK, 0, 5, 0, NULL), + 0); + +static OWL_COMP_DIV(nand0_clk, "nand0_clk", nand_clk_mux_p, + OWL_MUX_HW(CMU_NANDCCLK, 8, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 4, 0), + OWL_DIVIDER_HW(CMU_NANDCCLK, 0, 4, 0, nand_div_table), + CLK_SET_RATE_PARENT); + +static OWL_COMP_DIV(nand1_clk, "nand1_clk", nand_clk_mux_p, + OWL_MUX_HW(CMU_NANDCCLK, 24, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 11, 0), + OWL_DIVIDER_HW(CMU_NANDCCLK, 16, 4, 0, nand_div_table), + CLK_SET_RATE_PARENT); + +static OWL_COMP_DIV_FIXED(pwm0_clk, "pwm0_clk", "hosc", + OWL_GATE_HW(CMU_DEVCLKEN1, 23, 0), + OWL_DIVIDER_HW(CMU_PWM0CLK, 0, 6, 0, NULL), + 0); + +static OWL_COMP_DIV_FIXED(pwm1_clk, "pwm1_clk", "hosc", + OWL_GATE_HW(CMU_DEVCLKEN1, 24, 0), + OWL_DIVIDER_HW(CMU_PWM1CLK, 0, 6, 0, NULL), + 0); +/* + * pwm2 may be for backlight, do not gate it + * even it is "unused", because it may be + * enabled at boot stage, and in kernel, driver + * has no effective method to know the real status, + * so, the best way is keeping it as what it was. + */ +static OWL_COMP_DIV_FIXED(pwm2_clk, "pwm2_clk", "hosc", + OWL_GATE_HW(CMU_DEVCLKEN1, 25, 0), + OWL_DIVIDER_HW(CMU_PWM2CLK, 0, 6, 0, NULL), + CLK_IGNORE_UNUSED); + +static OWL_COMP_DIV_FIXED(pwm3_clk, "pwm3_clk", "hosc", + OWL_GATE_HW(CMU_DEVCLKEN1, 26, 0), + OWL_DIVIDER_HW(CMU_PWM3CLK, 0, 6, 0, NULL), + 0); + +static OWL_COMP_DIV_FIXED(pwm4_clk, "pwm4_clk", "hosc", + OWL_GATE_HW(CMU_DEVCLKEN1, 4, 0), + OWL_DIVIDER_HW(CMU_PWM4CLK, 0, 6, 0, NULL), + 0); + +static OWL_COMP_DIV_FIXED(pwm5_clk, "pwm5_clk", "hosc", + OWL_GATE_HW(CMU_DEVCLKEN1, 5, 0), + OWL_DIVIDER_HW(CMU_PWM5CLK, 0, 6, 0, NULL), + 0); + +static OWL_COMP_FACTOR(sd0_clk, "sd0_clk", sd_clk_mux_p, + OWL_MUX_HW(CMU_SD0CLK, 9, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 5, 0), + OWL_FACTOR_HW(CMU_SD0CLK, 0, 9, 0, sd_factor_table), + 0); + +static OWL_COMP_FACTOR(sd1_clk, "sd1_clk", sd_clk_mux_p, + OWL_MUX_HW(CMU_SD1CLK, 9, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 6, 0), + OWL_FACTOR_HW(CMU_SD1CLK, 0, 9, 0, sd_factor_table), + 0); + +static OWL_COMP_FACTOR(sd2_clk, "sd2_clk", sd_clk_mux_p, + OWL_MUX_HW(CMU_SD2CLK, 9, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 7, 0), + OWL_FACTOR_HW(CMU_SD2CLK, 0, 9, 0, sd_factor_table), + 0); + +static OWL_COMP_FACTOR(sd3_clk, "sd3_clk", sd_clk_mux_p, + OWL_MUX_HW(CMU_SD3CLK, 9, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 16, 0), + OWL_FACTOR_HW(CMU_SD3CLK, 0, 9, 0, sd_factor_table), + 0); + +static OWL_COMP_DIV(sensor_clk, "sensor_clk", sensor_clk_mux_p, + OWL_MUX_HW(CMU_SENSORCLK, 4, 1), + OWL_GATE_HW(CMU_DEVCLKEN0, 14, 0), + OWL_DIVIDER_HW(CMU_SENSORCLK, 0, 4, 0, NULL), + 0); + +static OWL_COMP_DIV_FIXED(speed_sensor_clk, "speed_sensor_clk", + "hosc", + OWL_GATE_HW(CMU_DEVCLKEN1, 0, 0), + OWL_DIVIDER_HW(CMU_TLSCLK, 0, 4, CLK_DIVIDER_POWER_OF_TWO, NULL), + 0); + +static OWL_COMP_DIV_FIXED(thermal_sensor_clk, "thermal_sensor_clk", + "hosc", + OWL_GATE_HW(CMU_DEVCLKEN1, 2, 0), + OWL_DIVIDER_HW(CMU_TLSCLK, 8, 4, CLK_DIVIDER_POWER_OF_TWO, NULL), + 0); + +static OWL_COMP_DIV(uart0_clk, "uart0_clk", uart_clk_mux_p, + OWL_MUX_HW(CMU_UART0CLK, 16, 1), + OWL_GATE_HW(CMU_DEVCLKEN1, 6, 0), + OWL_DIVIDER_HW(CMU_UART0CLK, 0, 8, CLK_DIVIDER_ROUND_CLOSEST, NULL), + CLK_IGNORE_UNUSED); + +static OWL_COMP_DIV(uart1_clk, "uart1_clk", uart_clk_mux_p, + OWL_MUX_HW(CMU_UART1CLK, 16, 1), + OWL_GATE_HW(CMU_DEVCLKEN1, 7, 0), + OWL_DIVIDER_HW(CMU_UART1CLK, 1, 8, CLK_DIVIDER_ROUND_CLOSEST, NULL), + CLK_IGNORE_UNUSED); + +static OWL_COMP_DIV(uart2_clk, "uart2_clk", uart_clk_mux_p, + OWL_MUX_HW(CMU_UART2CLK, 16, 1), + OWL_GATE_HW(CMU_DEVCLKEN1, 8, 0), + OWL_DIVIDER_HW(CMU_UART2CLK, 0, 8, CLK_DIVIDER_ROUND_CLOSEST, NULL), + CLK_IGNORE_UNUSED); + +static OWL_COMP_DIV(uart3_clk, "uart3_clk", uart_clk_mux_p, + OWL_MUX_HW(CMU_UART3CLK, 16, 1), + OWL_GATE_HW(CMU_DEVCLKEN1, 19, 0), + OWL_DIVIDER_HW(CMU_UART3CLK, 0, 8, CLK_DIVIDER_ROUND_CLOSEST, NULL), + CLK_IGNORE_UNUSED); + +static OWL_COMP_DIV(uart4_clk, "uart4_clk", uart_clk_mux_p, + OWL_MUX_HW(CMU_UART4CLK, 16, 1), + OWL_GATE_HW(CMU_DEVCLKEN1, 20, 0), + OWL_DIVIDER_HW(CMU_UART4CLK, 0, 8, CLK_DIVIDER_ROUND_CLOSEST, NULL), + CLK_IGNORE_UNUSED); + +static OWL_COMP_DIV(uart5_clk, "uart5_clk", uart_clk_mux_p, + OWL_MUX_HW(CMU_UART5CLK, 16, 1), + OWL_GATE_HW(CMU_DEVCLKEN1, 21, 0), + OWL_DIVIDER_HW(CMU_UART5CLK, 0, 8, CLK_DIVIDER_ROUND_CLOSEST, NULL), + CLK_IGNORE_UNUSED); + +static OWL_COMP_DIV(uart6_clk, "uart6_clk", uart_clk_mux_p, + OWL_MUX_HW(CMU_UART6CLK, 16, 1), + OWL_GATE_HW(CMU_DEVCLKEN1, 18, 0), + OWL_DIVIDER_HW(CMU_UART6CLK, 0, 8, CLK_DIVIDER_ROUND_CLOSEST, NULL), + CLK_IGNORE_UNUSED); + +static OWL_COMP_FACTOR(vce_clk, "vce_clk", vce_clk_mux_p, + OWL_MUX_HW(CMU_VCECLK, 4, 2), + OWL_GATE_HW(CMU_DEVCLKEN0, 26, 0), + OWL_FACTOR_HW(CMU_VCECLK, 0, 3, 0, bisp_factor_table), + 0); + +static OWL_COMP_FACTOR(vde_clk, "vde_clk", hde_clk_mux_p, + OWL_MUX_HW(CMU_VDECLK, 4, 2), + OWL_GATE_HW(CMU_DEVCLKEN0, 25, 0), + OWL_FACTOR_HW(CMU_VDECLK, 0, 3, 0, bisp_factor_table), + 0); + +static struct owl_clk_common *s900_clks[] = { + &core_pll_clk.common, + &dev_pll_clk.common, + &ddr_pll_clk.common, + &nand_pll_clk.common, + &display_pll_clk.common, + &assist_pll_clk.common, + &audio_pll_clk.common, + &edp_pll_clk.common, + &cpu_clk.common, + &dev_clk.common, + &noc_clk_mux.common, + &noc_clk_div.common, + &ahb_clk.common, + &apb_clk.common, + &usb3_mac_clk.common, + &rmii_ref_clk.common, + &noc_clk.common, + &de_clk1.common, + &de_clk2.common, + &de_clk3.common, + &gpio_clk.common, + &gpu_clk.common, + &dmac_clk.common, + &timer_clk.common, + &dsi_clk.common, + &ddr0_clk.common, + &ddr1_clk.common, + &usb3_480mpll0_clk.common, + &usb3_480mphy0_clk.common, + &usb3_5gphy_clk.common, + &usb3_cce_clk.common, + &edp24M_clk.common, + &edp_link_clk.common, + &usbh0_pllen_clk.common, + &usbh0_phy_clk.common, + &usbh0_cce_clk.common, + &usbh1_pllen_clk.common, + &usbh1_phy_clk.common, + &usbh1_cce_clk.common, + &i2c0_clk.common, + &i2c1_clk.common, + &i2c2_clk.common, + &i2c3_clk.common, + &i2c4_clk.common, + &i2c5_clk.common, + &spi0_clk.common, + &spi1_clk.common, + &spi2_clk.common, + &spi3_clk.common, + &bisp_clk.common, + &csi0_clk.common, + &csi1_clk.common, + &de_clk.common, + &dmm_clk.common, + &edp_clk.common, + ð_mac_clk.common, + &gpu_core_clk.common, + &gpu_mem_clk.common, + &gpu_sys_clk.common, + &hde_clk.common, + &hdmia_clk.common, + &i2srx_clk.common, + &i2stx_clk.common, + &imx_clk.common, + &lcd_clk.common, + &nand0_clk.common, + &nand1_clk.common, + &pwm0_clk.common, + &pwm1_clk.common, + &pwm2_clk.common, + &pwm3_clk.common, + &pwm4_clk.common, + &pwm5_clk.common, + &sd0_clk.common, + &sd1_clk.common, + &sd2_clk.common, + &sd3_clk.common, + &sensor_clk.common, + &speed_sensor_clk.common, + &thermal_sensor_clk.common, + &uart0_clk.common, + &uart1_clk.common, + &uart2_clk.common, + &uart3_clk.common, + &uart4_clk.common, + &uart5_clk.common, + &uart6_clk.common, + &vce_clk.common, + &vde_clk.common, +}; + +static struct clk_hw_onecell_data s900_hw_clks = { + .hws = { + [CLK_CORE_PLL] = &core_pll_clk.common.hw, + [CLK_DEV_PLL] = &dev_pll_clk.common.hw, + [CLK_DDR_PLL] = &ddr_pll_clk.common.hw, + [CLK_NAND_PLL] = &nand_pll_clk.common.hw, + [CLK_DISPLAY_PLL] = &display_pll_clk.common.hw, + [CLK_ASSIST_PLL] = &assist_pll_clk.common.hw, + [CLK_AUDIO_PLL] = &audio_pll_clk.common.hw, + [CLK_EDP_PLL] = &edp_pll_clk.common.hw, + [CLK_CPU] = &cpu_clk.common.hw, + [CLK_DEV] = &dev_clk.common.hw, + [CLK_NOC_MUX] = &noc_clk_mux.common.hw, + [CLK_NOC_DIV] = &noc_clk_div.common.hw, + [CLK_AHB] = &ahb_clk.common.hw, + [CLK_APB] = &apb_clk.common.hw, + [CLK_USB3_MAC] = &usb3_mac_clk.common.hw, + [CLK_RMII_REF] = &rmii_ref_clk.common.hw, + [CLK_NOC] = &noc_clk.common.hw, + [CLK_DE1] = &de_clk1.common.hw, + [CLK_DE2] = &de_clk2.common.hw, + [CLK_DE3] = &de_clk3.common.hw, + [CLK_GPIO] = &gpio_clk.common.hw, + [CLK_GPU] = &gpu_clk.common.hw, + [CLK_DMAC] = &dmac_clk.common.hw, + [CLK_TIMER] = &timer_clk.common.hw, + [CLK_DSI] = &dsi_clk.common.hw, + [CLK_DDR0] = &ddr0_clk.common.hw, + [CLK_DDR1] = &ddr1_clk.common.hw, + [CLK_USB3_480MPLL0] = &usb3_480mpll0_clk.common.hw, + [CLK_USB3_480MPHY0] = &usb3_480mphy0_clk.common.hw, + [CLK_USB3_5GPHY] = &usb3_5gphy_clk.common.hw, + [CLK_USB3_CCE] = &usb3_cce_clk.common.hw, + [CLK_24M_EDP] = &edp24M_clk.common.hw, + [CLK_EDP_LINK] = &edp_link_clk.common.hw, + [CLK_USB2H0_PLLEN] = &usbh0_pllen_clk.common.hw, + [CLK_USB2H0_PHY] = &usbh0_phy_clk.common.hw, + [CLK_USB2H0_CCE] = &usbh0_cce_clk.common.hw, + [CLK_USB2H1_PLLEN] = &usbh1_pllen_clk.common.hw, + [CLK_USB2H1_PHY] = &usbh1_phy_clk.common.hw, + [CLK_USB2H1_CCE] = &usbh1_cce_clk.common.hw, + [CLK_I2C0] = &i2c0_clk.common.hw, + [CLK_I2C1] = &i2c1_clk.common.hw, + [CLK_I2C2] = &i2c2_clk.common.hw, + [CLK_I2C3] = &i2c3_clk.common.hw, + [CLK_I2C4] = &i2c4_clk.common.hw, + [CLK_I2C5] = &i2c5_clk.common.hw, + [CLK_SPI0] = &spi0_clk.common.hw, + [CLK_SPI1] = &spi1_clk.common.hw, + [CLK_SPI2] = &spi2_clk.common.hw, + [CLK_SPI3] = &spi3_clk.common.hw, + [CLK_BISP] = &bisp_clk.common.hw, + [CLK_CSI0] = &csi0_clk.common.hw, + [CLK_CSI1] = &csi1_clk.common.hw, + [CLK_DE0] = &de_clk.common.hw, + [CLK_DMM] = &dmm_clk.common.hw, + [CLK_EDP] = &edp_clk.common.hw, + [CLK_ETH_MAC] = ð_mac_clk.common.hw, + [CLK_GPU_CORE] = &gpu_core_clk.common.hw, + [CLK_GPU_MEM] = &gpu_mem_clk.common.hw, + [CLK_GPU_SYS] = &gpu_sys_clk.common.hw, + [CLK_HDE] = &hde_clk.common.hw, + [CLK_HDMI_AUDIO] = &hdmia_clk.common.hw, + [CLK_I2SRX] = &i2srx_clk.common.hw, + [CLK_I2STX] = &i2stx_clk.common.hw, + [CLK_IMX] = &imx_clk.common.hw, + [CLK_LCD] = &lcd_clk.common.hw, + [CLK_NAND0] = &nand0_clk.common.hw, + [CLK_NAND1] = &nand1_clk.common.hw, + [CLK_PWM0] = &pwm0_clk.common.hw, + [CLK_PWM1] = &pwm1_clk.common.hw, + [CLK_PWM2] = &pwm2_clk.common.hw, + [CLK_PWM3] = &pwm3_clk.common.hw, + [CLK_PWM4] = &pwm4_clk.common.hw, + [CLK_PWM5] = &pwm5_clk.common.hw, + [CLK_SD0] = &sd0_clk.common.hw, + [CLK_SD1] = &sd1_clk.common.hw, + [CLK_SD2] = &sd2_clk.common.hw, + [CLK_SD3] = &sd3_clk.common.hw, + [CLK_SENSOR] = &sensor_clk.common.hw, + [CLK_SPEED_SENSOR] = &speed_sensor_clk.common.hw, + [CLK_THERMAL_SENSOR] = &thermal_sensor_clk.common.hw, + [CLK_UART0] = &uart0_clk.common.hw, + [CLK_UART1] = &uart1_clk.common.hw, + [CLK_UART2] = &uart2_clk.common.hw, + [CLK_UART3] = &uart3_clk.common.hw, + [CLK_UART4] = &uart4_clk.common.hw, + [CLK_UART5] = &uart5_clk.common.hw, + [CLK_UART6] = &uart6_clk.common.hw, + [CLK_VCE] = &vce_clk.common.hw, + [CLK_VDE] = &vde_clk.common.hw, + }, + .num = CLK_NR_CLKS, +}; + +static const struct owl_clk_desc s900_clk_desc = { + .clks = s900_clks, + .num_clks = ARRAY_SIZE(s900_clks), + + .hw_clks = &s900_hw_clks, +}; + +static int s900_clk_probe(struct platform_device *pdev) +{ + const struct owl_clk_desc *desc; + + desc = &s900_clk_desc; + owl_clk_regmap_init(pdev, desc); + + return owl_clk_probe(&pdev->dev, desc->hw_clks); +} + +static const struct of_device_id s900_clk_of_match[] = { + { .compatible = "actions,s900-cmu", }, + { /* sentinel */ } +}; + +static struct platform_driver s900_clk_driver = { + .probe = s900_clk_probe, + .driver = { + .name = "s900-cmu", + .of_match_table = s900_clk_of_match, + }, +}; + +static int __init s900_clk_init(void) +{ + return platform_driver_register(&s900_clk_driver); +} +core_initcall(s900_clk_init); diff --git a/drivers/clk/at91/clk-pll.c b/drivers/clk/at91/clk-pll.c index 7d3223fc7161..72b6091eb7b9 100644 --- a/drivers/clk/at91/clk-pll.c +++ b/drivers/clk/at91/clk-pll.c @@ -132,19 +132,8 @@ static unsigned long clk_pll_recalc_rate(struct clk_hw *hw, unsigned long parent_rate) { struct clk_pll *pll = to_clk_pll(hw); - unsigned int pllr; - u16 mul; - u8 div; - - regmap_read(pll->regmap, PLL_REG(pll->id), &pllr); - - div = PLL_DIV(pllr); - mul = PLL_MUL(pllr, pll->layout); - - if (!div || !mul) - return 0; - return (parent_rate / div) * (mul + 1); + return (parent_rate / pll->div) * (pll->mul + 1); } static long clk_pll_get_best_div_mul(struct clk_pll *pll, unsigned long rate, diff --git a/drivers/clk/bcm/clk-bcm2835-aux.c b/drivers/clk/bcm/clk-bcm2835-aux.c index 77e276d61702..f225ad29b110 100644 --- a/drivers/clk/bcm/clk-bcm2835-aux.c +++ b/drivers/clk/bcm/clk-bcm2835-aux.c @@ -40,8 +40,10 @@ static int bcm2835_aux_clk_probe(struct platform_device *pdev) if (IS_ERR(reg)) return PTR_ERR(reg); - onecell = devm_kmalloc(dev, sizeof(*onecell) + sizeof(*onecell->hws) * - BCM2835_AUX_CLOCK_COUNT, GFP_KERNEL); + onecell = devm_kmalloc(dev, + struct_size(onecell, hws, + BCM2835_AUX_CLOCK_COUNT), + GFP_KERNEL); if (!onecell) return -ENOMEM; onecell->num = BCM2835_AUX_CLOCK_COUNT; diff --git a/drivers/clk/bcm/clk-bcm2835.c b/drivers/clk/bcm/clk-bcm2835.c index fa0d5c8611a0..7bef0666ae7e 100644 --- a/drivers/clk/bcm/clk-bcm2835.c +++ b/drivers/clk/bcm/clk-bcm2835.c @@ -394,25 +394,21 @@ out: return count * 1000; } -static int bcm2835_debugfs_regset(struct bcm2835_cprman *cprman, u32 base, +static void bcm2835_debugfs_regset(struct bcm2835_cprman *cprman, u32 base, struct debugfs_reg32 *regs, size_t nregs, struct dentry *dentry) { - struct dentry *regdump; struct debugfs_regset32 *regset; regset = devm_kzalloc(cprman->dev, sizeof(*regset), GFP_KERNEL); if (!regset) - return -ENOMEM; + return; regset->regs = regs; regset->nregs = nregs; regset->base = cprman->regs + base; - regdump = debugfs_create_regset32("regdump", S_IRUGO, dentry, - regset); - - return regdump ? 0 : -ENOMEM; + debugfs_create_regset32("regdump", S_IRUGO, dentry, regset); } struct bcm2835_pll_data { @@ -730,7 +726,7 @@ static int bcm2835_pll_set_rate(struct clk_hw *hw, return 0; } -static int bcm2835_pll_debug_init(struct clk_hw *hw, +static void bcm2835_pll_debug_init(struct clk_hw *hw, struct dentry *dentry) { struct bcm2835_pll *pll = container_of(hw, struct bcm2835_pll, hw); @@ -738,9 +734,9 @@ static int bcm2835_pll_debug_init(struct clk_hw *hw, const struct bcm2835_pll_data *data = pll->data; struct debugfs_reg32 *regs; - regs = devm_kzalloc(cprman->dev, 7 * sizeof(*regs), GFP_KERNEL); + regs = devm_kcalloc(cprman->dev, 7, sizeof(*regs), GFP_KERNEL); if (!regs) - return -ENOMEM; + return; regs[0].name = "cm_ctrl"; regs[0].offset = data->cm_ctrl_reg; @@ -757,7 +753,7 @@ static int bcm2835_pll_debug_init(struct clk_hw *hw, regs[6].name = "ana3"; regs[6].offset = data->ana_reg_base + 3 * 4; - return bcm2835_debugfs_regset(cprman, 0, regs, 7, dentry); + bcm2835_debugfs_regset(cprman, 0, regs, 7, dentry); } static const struct clk_ops bcm2835_pll_clk_ops = { @@ -861,24 +857,24 @@ static int bcm2835_pll_divider_set_rate(struct clk_hw *hw, return 0; } -static int bcm2835_pll_divider_debug_init(struct clk_hw *hw, - struct dentry *dentry) +static void bcm2835_pll_divider_debug_init(struct clk_hw *hw, + struct dentry *dentry) { struct bcm2835_pll_divider *divider = bcm2835_pll_divider_from_hw(hw); struct bcm2835_cprman *cprman = divider->cprman; const struct bcm2835_pll_divider_data *data = divider->data; struct debugfs_reg32 *regs; - regs = devm_kzalloc(cprman->dev, 7 * sizeof(*regs), GFP_KERNEL); + regs = devm_kcalloc(cprman->dev, 7, sizeof(*regs), GFP_KERNEL); if (!regs) - return -ENOMEM; + return; regs[0].name = "cm"; regs[0].offset = data->cm_reg; regs[1].name = "a2w"; regs[1].offset = data->a2w_reg; - return bcm2835_debugfs_regset(cprman, 0, regs, 2, dentry); + bcm2835_debugfs_regset(cprman, 0, regs, 2, dentry); } static const struct clk_ops bcm2835_pll_divider_clk_ops = { @@ -1254,15 +1250,14 @@ static struct debugfs_reg32 bcm2835_debugfs_clock_reg32[] = { }, }; -static int bcm2835_clock_debug_init(struct clk_hw *hw, +static void bcm2835_clock_debug_init(struct clk_hw *hw, struct dentry *dentry) { struct bcm2835_clock *clock = bcm2835_clock_from_hw(hw); struct bcm2835_cprman *cprman = clock->cprman; const struct bcm2835_clock_data *data = clock->data; - return bcm2835_debugfs_regset( - cprman, data->ctl_reg, + bcm2835_debugfs_regset(cprman, data->ctl_reg, bcm2835_debugfs_clock_reg32, ARRAY_SIZE(bcm2835_debugfs_clock_reg32), dentry); @@ -1395,7 +1390,7 @@ static struct clk_hw *bcm2835_register_clock(struct bcm2835_cprman *cprman, struct bcm2835_clock *clock; struct clk_init_data init; const char *parents[1 << CM_SRC_BITS]; - size_t i, j; + size_t i; int ret; /* @@ -1405,12 +1400,11 @@ static struct clk_hw *bcm2835_register_clock(struct bcm2835_cprman *cprman, for (i = 0; i < data->num_mux_parents; i++) { parents[i] = data->parents[i]; - for (j = 0; j < ARRAY_SIZE(cprman_parent_names); j++) { - if (strcmp(parents[i], cprman_parent_names[j]) == 0) { - parents[i] = cprman->real_parent_names[j]; - break; - } - } + ret = match_string(cprman_parent_names, + ARRAY_SIZE(cprman_parent_names), + parents[i]); + if (ret >= 0) + parents[i] = cprman->real_parent_names[ret]; } memset(&init, 0, sizeof(init)); @@ -2147,8 +2141,8 @@ static int bcm2835_clk_probe(struct platform_device *pdev) size_t i; int ret; - cprman = devm_kzalloc(dev, sizeof(*cprman) + - sizeof(*cprman->onecell.hws) * asize, + cprman = devm_kzalloc(dev, + struct_size(cprman, onecell.hws, asize), GFP_KERNEL); if (!cprman) return -ENOMEM; diff --git a/drivers/clk/bcm/clk-iproc-asiu.c b/drivers/clk/bcm/clk-iproc-asiu.c index 4360e481368b..6fb8af506777 100644 --- a/drivers/clk/bcm/clk-iproc-asiu.c +++ b/drivers/clk/bcm/clk-iproc-asiu.c @@ -197,8 +197,8 @@ void __init iproc_asiu_setup(struct device_node *node, if (WARN_ON(!asiu)) return; - asiu->clk_data = kzalloc(sizeof(*asiu->clk_data->hws) * num_clks + - sizeof(*asiu->clk_data), GFP_KERNEL); + asiu->clk_data = kzalloc(struct_size(asiu->clk_data, hws, num_clks), + GFP_KERNEL); if (WARN_ON(!asiu->clk_data)) goto err_clks; asiu->clk_data->num = num_clks; diff --git a/drivers/clk/bcm/clk-iproc-pll.c b/drivers/clk/bcm/clk-iproc-pll.c index 43a58ae5a89d..274441e2ddb2 100644 --- a/drivers/clk/bcm/clk-iproc-pll.c +++ b/drivers/clk/bcm/clk-iproc-pll.c @@ -744,8 +744,7 @@ void iproc_pll_clk_setup(struct device_node *node, if (WARN_ON(!pll)) return; - clk_data = kzalloc(sizeof(*clk_data->hws) * num_clks + - sizeof(*clk_data), GFP_KERNEL); + clk_data = kzalloc(struct_size(clk_data, hws, num_clks), GFP_KERNEL); if (WARN_ON(!clk_data)) goto err_clk_data; clk_data->num = num_clks; diff --git a/drivers/clk/bcm/clk-sr.c b/drivers/clk/bcm/clk-sr.c index adc74f4584cf..7b9efc0212a8 100644 --- a/drivers/clk/bcm/clk-sr.c +++ b/drivers/clk/bcm/clk-sr.c @@ -56,8 +56,8 @@ static const struct iproc_pll_ctrl sr_genpll0 = { }; static const struct iproc_clk_ctrl sr_genpll0_clk[] = { - [BCM_SR_GENPLL0_SATA_CLK] = { - .channel = BCM_SR_GENPLL0_SATA_CLK, + [BCM_SR_GENPLL0_125M_CLK] = { + .channel = BCM_SR_GENPLL0_125M_CLK, .flags = IPROC_CLK_AON, .enable = ENABLE_VAL(0x4, 6, 0, 12), .mdiv = REG_VAL(0x18, 0, 9), @@ -102,6 +102,65 @@ static int sr_genpll0_clk_init(struct platform_device *pdev) return 0; } +static const struct iproc_pll_ctrl sr_genpll2 = { + .flags = IPROC_CLK_AON | IPROC_CLK_PLL_HAS_NDIV_FRAC | + IPROC_CLK_PLL_NEEDS_SW_CFG, + .aon = AON_VAL(0x0, 1, 13, 12), + .reset = RESET_VAL(0x0, 12, 11), + .dig_filter = DF_VAL(0x0, 4, 3, 0, 4, 7, 3), + .sw_ctrl = SW_CTRL_VAL(0x10, 31), + .ndiv_int = REG_VAL(0x10, 20, 10), + .ndiv_frac = REG_VAL(0x10, 0, 20), + .pdiv = REG_VAL(0x14, 0, 4), + .status = REG_VAL(0x30, 12, 1), +}; + +static const struct iproc_clk_ctrl sr_genpll2_clk[] = { + [BCM_SR_GENPLL2_NIC_CLK] = { + .channel = BCM_SR_GENPLL2_NIC_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x4, 6, 0, 12), + .mdiv = REG_VAL(0x18, 0, 9), + }, + [BCM_SR_GENPLL2_TS_500_CLK] = { + .channel = BCM_SR_GENPLL2_TS_500_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x4, 7, 1, 13), + .mdiv = REG_VAL(0x18, 10, 9), + }, + [BCM_SR_GENPLL2_125_NITRO_CLK] = { + .channel = BCM_SR_GENPLL2_125_NITRO_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x4, 8, 2, 14), + .mdiv = REG_VAL(0x18, 20, 9), + }, + [BCM_SR_GENPLL2_CHIMP_CLK] = { + .channel = BCM_SR_GENPLL2_CHIMP_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x4, 9, 3, 15), + .mdiv = REG_VAL(0x1c, 0, 9), + }, + [BCM_SR_GENPLL2_NIC_FLASH_CLK] = { + .channel = BCM_SR_GENPLL2_NIC_FLASH_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x4, 10, 4, 16), + .mdiv = REG_VAL(0x1c, 10, 9), + }, + [BCM_SR_GENPLL2_FS4_CLK] = { + .channel = BCM_SR_GENPLL2_FS4_CLK, + .enable = ENABLE_VAL(0x4, 11, 5, 17), + .mdiv = REG_VAL(0x1c, 20, 9), + }, +}; + +static int sr_genpll2_clk_init(struct platform_device *pdev) +{ + iproc_pll_clk_setup(pdev->dev.of_node, + &sr_genpll2, NULL, 0, sr_genpll2_clk, + ARRAY_SIZE(sr_genpll2_clk)); + return 0; +} + static const struct iproc_pll_ctrl sr_genpll3 = { .flags = IPROC_CLK_AON | IPROC_CLK_PLL_HAS_NDIV_FRAC | IPROC_CLK_PLL_NEEDS_SW_CFG, @@ -157,6 +216,30 @@ static const struct iproc_clk_ctrl sr_genpll4_clk[] = { .enable = ENABLE_VAL(0x4, 6, 0, 12), .mdiv = REG_VAL(0x18, 0, 9), }, + [BCM_SR_GENPLL4_TPIU_PLL_CLK] = { + .channel = BCM_SR_GENPLL4_TPIU_PLL_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x4, 7, 1, 13), + .mdiv = REG_VAL(0x18, 10, 9), + }, + [BCM_SR_GENPLL4_NOC_CLK] = { + .channel = BCM_SR_GENPLL4_NOC_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x4, 8, 2, 14), + .mdiv = REG_VAL(0x18, 20, 9), + }, + [BCM_SR_GENPLL4_CHCLK_FS4_CLK] = { + .channel = BCM_SR_GENPLL4_CHCLK_FS4_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x4, 9, 3, 15), + .mdiv = REG_VAL(0x1c, 0, 9), + }, + [BCM_SR_GENPLL4_BRIDGE_FSCPU_CLK] = { + .channel = BCM_SR_GENPLL4_BRIDGE_FSCPU_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x4, 10, 4, 16), + .mdiv = REG_VAL(0x1c, 10, 9), + }, }; static int sr_genpll4_clk_init(struct platform_device *pdev) @@ -181,18 +264,21 @@ static const struct iproc_pll_ctrl sr_genpll5 = { }; static const struct iproc_clk_ctrl sr_genpll5_clk[] = { - [BCM_SR_GENPLL5_FS_CLK] = { - .channel = BCM_SR_GENPLL5_FS_CLK, - .flags = IPROC_CLK_AON, + [BCM_SR_GENPLL5_FS4_HF_CLK] = { + .channel = BCM_SR_GENPLL5_FS4_HF_CLK, .enable = ENABLE_VAL(0x4, 6, 0, 12), .mdiv = REG_VAL(0x18, 0, 9), }, - [BCM_SR_GENPLL5_SPU_CLK] = { - .channel = BCM_SR_GENPLL5_SPU_CLK, - .flags = IPROC_CLK_AON, - .enable = ENABLE_VAL(0x4, 6, 0, 12), + [BCM_SR_GENPLL5_CRYPTO_AE_CLK] = { + .channel = BCM_SR_GENPLL5_CRYPTO_AE_CLK, + .enable = ENABLE_VAL(0x4, 7, 1, 12), .mdiv = REG_VAL(0x18, 10, 9), }, + [BCM_SR_GENPLL5_RAID_AE_CLK] = { + .channel = BCM_SR_GENPLL5_RAID_AE_CLK, + .enable = ENABLE_VAL(0x4, 8, 2, 14), + .mdiv = REG_VAL(0x18, 20, 9), + }, }; static int sr_genpll5_clk_init(struct platform_device *pdev) @@ -214,24 +300,30 @@ static const struct iproc_pll_ctrl sr_lcpll0 = { }; static const struct iproc_clk_ctrl sr_lcpll0_clk[] = { - [BCM_SR_LCPLL0_SATA_REF_CLK] = { - .channel = BCM_SR_LCPLL0_SATA_REF_CLK, + [BCM_SR_LCPLL0_SATA_REFP_CLK] = { + .channel = BCM_SR_LCPLL0_SATA_REFP_CLK, .flags = IPROC_CLK_AON, .enable = ENABLE_VAL(0x0, 7, 1, 13), .mdiv = REG_VAL(0x14, 0, 9), }, - [BCM_SR_LCPLL0_USB_REF_CLK] = { - .channel = BCM_SR_LCPLL0_USB_REF_CLK, + [BCM_SR_LCPLL0_SATA_REFN_CLK] = { + .channel = BCM_SR_LCPLL0_SATA_REFN_CLK, .flags = IPROC_CLK_AON, .enable = ENABLE_VAL(0x0, 8, 2, 14), .mdiv = REG_VAL(0x14, 10, 9), }, - [BCM_SR_LCPLL0_SATA_REFPN_CLK] = { - .channel = BCM_SR_LCPLL0_SATA_REFPN_CLK, + [BCM_SR_LCPLL0_SATA_350_CLK] = { + .channel = BCM_SR_LCPLL0_SATA_350_CLK, .flags = IPROC_CLK_AON, .enable = ENABLE_VAL(0x0, 9, 3, 15), .mdiv = REG_VAL(0x14, 20, 9), }, + [BCM_SR_LCPLL0_SATA_500_CLK] = { + .channel = BCM_SR_LCPLL0_SATA_500_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x0, 10, 4, 16), + .mdiv = REG_VAL(0x18, 0, 9), + }, }; static int sr_lcpll0_clk_init(struct platform_device *pdev) @@ -259,6 +351,18 @@ static const struct iproc_clk_ctrl sr_lcpll1_clk[] = { .enable = ENABLE_VAL(0x0, 7, 1, 13), .mdiv = REG_VAL(0x14, 0, 9), }, + [BCM_SR_LCPLL1_USB_REF_CLK] = { + .channel = BCM_SR_LCPLL1_USB_REF_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x0, 8, 2, 14), + .mdiv = REG_VAL(0x14, 10, 9), + }, + [BCM_SR_LCPLL1_CRMU_TS_CLK] = { + .channel = BCM_SR_LCPLL1_CRMU_TS_CLK, + .flags = IPROC_CLK_AON, + .enable = ENABLE_VAL(0x0, 9, 3, 15), + .mdiv = REG_VAL(0x14, 20, 9), + }, }; static int sr_lcpll1_clk_init(struct platform_device *pdev) @@ -298,6 +402,7 @@ static int sr_lcpll_pcie_clk_init(struct platform_device *pdev) static const struct of_device_id sr_clk_dt_ids[] = { { .compatible = "brcm,sr-genpll0", .data = sr_genpll0_clk_init }, + { .compatible = "brcm,sr-genpll2", .data = sr_genpll2_clk_init }, { .compatible = "brcm,sr-genpll4", .data = sr_genpll4_clk_init }, { .compatible = "brcm,sr-genpll5", .data = sr_genpll5_clk_init }, { .compatible = "brcm,sr-lcpll0", .data = sr_lcpll0_clk_init }, diff --git a/drivers/clk/berlin/berlin2-avpll.c b/drivers/clk/berlin/berlin2-avpll.c index cfcae468e989..aa89b4c9464e 100644 --- a/drivers/clk/berlin/berlin2-avpll.c +++ b/drivers/clk/berlin/berlin2-avpll.c @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2014 Marvell Technology Group Ltd. * * Sebastian Hesselbarth * Alexandre Belloni - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include #include diff --git a/drivers/clk/berlin/berlin2-avpll.h b/drivers/clk/berlin/berlin2-avpll.h index 17e311153b42..f3af34dc2bee 100644 --- a/drivers/clk/berlin/berlin2-avpll.h +++ b/drivers/clk/berlin/berlin2-avpll.h @@ -1,20 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (c) 2014 Marvell Technology Group Ltd. * * Sebastian Hesselbarth * Alexandre Belloni - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #ifndef __BERLIN2_AVPLL_H #define __BERLIN2_AVPLL_H diff --git a/drivers/clk/berlin/berlin2-div.c b/drivers/clk/berlin/berlin2-div.c index 41ab2d392c57..4d0be66aa6a8 100644 --- a/drivers/clk/berlin/berlin2-div.c +++ b/drivers/clk/berlin/berlin2-div.c @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2014 Marvell Technology Group Ltd. * * Alexandre Belloni * Sebastian Hesselbarth - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include #include diff --git a/drivers/clk/berlin/berlin2-div.h b/drivers/clk/berlin/berlin2-div.h index e835ddf8374a..d4da64325190 100644 --- a/drivers/clk/berlin/berlin2-div.h +++ b/drivers/clk/berlin/berlin2-div.h @@ -1,20 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (c) 2014 Marvell Technology Group Ltd. * * Alexandre Belloni * Sebastian Hesselbarth - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #ifndef __BERLIN2_DIV_H #define __BERLIN2_DIV_H diff --git a/drivers/clk/berlin/berlin2-pll.c b/drivers/clk/berlin/berlin2-pll.c index 4ffbe80f6323..9661820717a5 100644 --- a/drivers/clk/berlin/berlin2-pll.c +++ b/drivers/clk/berlin/berlin2-pll.c @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2014 Marvell Technology Group Ltd. * * Alexandre Belloni * Sebastian Hesselbarth - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include #include diff --git a/drivers/clk/berlin/berlin2-pll.h b/drivers/clk/berlin/berlin2-pll.h index 583e024b9bed..3757fb25c4e8 100644 --- a/drivers/clk/berlin/berlin2-pll.h +++ b/drivers/clk/berlin/berlin2-pll.h @@ -1,20 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (c) 2014 Marvell Technology Group Ltd. * * Alexandre Belloni * Sebastian Hesselbarth - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #ifndef __BERLIN2_PLL_H #define __BERLIN2_PLL_H diff --git a/drivers/clk/berlin/bg2.c b/drivers/clk/berlin/bg2.c index e7331ace0337..0b4b44a2579e 100644 --- a/drivers/clk/berlin/bg2.c +++ b/drivers/clk/berlin/bg2.c @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2014 Marvell Technology Group Ltd. * * Sebastian Hesselbarth * Alexandre Belloni - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include @@ -509,8 +498,7 @@ static void __init berlin2_clock_setup(struct device_node *np) u8 avpll_flags = 0; int n, ret; - clk_data = kzalloc(sizeof(*clk_data) + - sizeof(*clk_data->hws) * MAX_CLKS, GFP_KERNEL); + clk_data = kzalloc(struct_size(clk_data, hws, MAX_CLKS), GFP_KERNEL); if (!clk_data) return; clk_data->num = MAX_CLKS; diff --git a/drivers/clk/berlin/bg2q.c b/drivers/clk/berlin/bg2q.c index 67c270b143f7..9b9db743df25 100644 --- a/drivers/clk/berlin/bg2q.c +++ b/drivers/clk/berlin/bg2q.c @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2014 Marvell Technology Group Ltd. * * Alexandre Belloni * Sebastian Hesselbarth - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include @@ -295,8 +284,7 @@ static void __init berlin2q_clock_setup(struct device_node *np) struct clk_hw **hws; int n, ret; - clk_data = kzalloc(sizeof(*clk_data) + - sizeof(*clk_data->hws) * MAX_CLKS, GFP_KERNEL); + clk_data = kzalloc(struct_size(clk_data, hws, MAX_CLKS), GFP_KERNEL); if (!clk_data) return; clk_data->num = MAX_CLKS; diff --git a/drivers/clk/berlin/common.h b/drivers/clk/berlin/common.h index bc68a14c4550..1afb3c29b796 100644 --- a/drivers/clk/berlin/common.h +++ b/drivers/clk/berlin/common.h @@ -1,20 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (c) 2014 Marvell Technology Group Ltd. * * Sebastian Hesselbarth * Alexandre Belloni - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #ifndef __BERLIN2_COMMON_H #define __BERLIN2_COMMON_H diff --git a/drivers/clk/clk-asm9260.c b/drivers/clk/clk-asm9260.c index bf0582cbbf38..44b544157121 100644 --- a/drivers/clk/clk-asm9260.c +++ b/drivers/clk/clk-asm9260.c @@ -273,8 +273,7 @@ static void __init asm9260_acc_init(struct device_node *np) int n; u32 accuracy = 0; - clk_data = kzalloc(sizeof(*clk_data) + - sizeof(*clk_data->hws) * MAX_CLKS, GFP_KERNEL); + clk_data = kzalloc(struct_size(clk_data, hws, MAX_CLKS), GFP_KERNEL); if (!clk_data) return; clk_data->num = MAX_CLKS; diff --git a/drivers/clk/clk-aspeed.c b/drivers/clk/clk-aspeed.c index 5eb50c31e455..38b366b00c57 100644 --- a/drivers/clk/clk-aspeed.c +++ b/drivers/clk/clk-aspeed.c @@ -14,7 +14,9 @@ #include -#define ASPEED_NUM_CLKS 35 +#define ASPEED_NUM_CLKS 36 + +#define ASPEED_RESET2_OFFSET 32 #define ASPEED_RESET_CTRL 0x04 #define ASPEED_CLK_SELECTION 0x08 @@ -30,6 +32,7 @@ #define CLKIN_25MHZ_EN BIT(23) #define AST2400_CLK_SOURCE_SEL BIT(18) #define ASPEED_CLK_SELECTION_2 0xd8 +#define ASPEED_RESET_CTRL2 0xd4 /* Globally visible clocks */ static DEFINE_SPINLOCK(aspeed_clk_lock); @@ -88,7 +91,7 @@ static const struct aspeed_gate_data aspeed_gates[] = { [ASPEED_CLK_GATE_GCLK] = { 1, 7, "gclk-gate", NULL, 0 }, /* 2D engine */ [ASPEED_CLK_GATE_MCLK] = { 2, -1, "mclk-gate", "mpll", CLK_IS_CRITICAL }, /* SDRAM */ [ASPEED_CLK_GATE_VCLK] = { 3, 6, "vclk-gate", NULL, 0 }, /* Video Capture */ - [ASPEED_CLK_GATE_BCLK] = { 4, 10, "bclk-gate", "bclk", 0 }, /* PCIe/PCI */ + [ASPEED_CLK_GATE_BCLK] = { 4, 8, "bclk-gate", "bclk", 0 }, /* PCIe/PCI */ [ASPEED_CLK_GATE_DCLK] = { 5, -1, "dclk-gate", NULL, 0 }, /* DAC */ [ASPEED_CLK_GATE_REFCLK] = { 6, -1, "refclk-gate", "clkin", CLK_IS_CRITICAL }, [ASPEED_CLK_GATE_USBPORT2CLK] = { 7, 3, "usb-port2-gate", NULL, 0 }, /* USB2.0 Host port 2 */ @@ -291,47 +294,72 @@ struct aspeed_reset { #define to_aspeed_reset(p) container_of((p), struct aspeed_reset, rcdev) static const u8 aspeed_resets[] = { + /* SCU04 resets */ [ASPEED_RESET_XDMA] = 25, [ASPEED_RESET_MCTP] = 24, [ASPEED_RESET_ADC] = 23, [ASPEED_RESET_JTAG_MASTER] = 22, [ASPEED_RESET_MIC] = 18, [ASPEED_RESET_PWM] = 9, - [ASPEED_RESET_PCIVGA] = 8, + [ASPEED_RESET_PECI] = 10, [ASPEED_RESET_I2C] = 2, [ASPEED_RESET_AHB] = 1, + + /* + * SCUD4 resets start at an offset to separate them from + * the SCU04 resets. + */ + [ASPEED_RESET_CRT1] = ASPEED_RESET2_OFFSET + 5, }; static int aspeed_reset_deassert(struct reset_controller_dev *rcdev, unsigned long id) { struct aspeed_reset *ar = to_aspeed_reset(rcdev); - u32 rst = BIT(aspeed_resets[id]); + u32 reg = ASPEED_RESET_CTRL; + u32 bit = aspeed_resets[id]; - return regmap_update_bits(ar->map, ASPEED_RESET_CTRL, rst, 0); + if (bit >= ASPEED_RESET2_OFFSET) { + bit -= ASPEED_RESET2_OFFSET; + reg = ASPEED_RESET_CTRL2; + } + + return regmap_update_bits(ar->map, reg, BIT(bit), 0); } static int aspeed_reset_assert(struct reset_controller_dev *rcdev, unsigned long id) { struct aspeed_reset *ar = to_aspeed_reset(rcdev); - u32 rst = BIT(aspeed_resets[id]); + u32 reg = ASPEED_RESET_CTRL; + u32 bit = aspeed_resets[id]; + + if (bit >= ASPEED_RESET2_OFFSET) { + bit -= ASPEED_RESET2_OFFSET; + reg = ASPEED_RESET_CTRL2; + } - return regmap_update_bits(ar->map, ASPEED_RESET_CTRL, rst, rst); + return regmap_update_bits(ar->map, reg, BIT(bit), BIT(bit)); } static int aspeed_reset_status(struct reset_controller_dev *rcdev, unsigned long id) { struct aspeed_reset *ar = to_aspeed_reset(rcdev); - u32 val, rst = BIT(aspeed_resets[id]); - int ret; + u32 reg = ASPEED_RESET_CTRL; + u32 bit = aspeed_resets[id]; + int ret, val; + + if (bit >= ASPEED_RESET2_OFFSET) { + bit -= ASPEED_RESET2_OFFSET; + reg = ASPEED_RESET_CTRL2; + } - ret = regmap_read(ar->map, ASPEED_RESET_CTRL, &val); + ret = regmap_read(ar->map, reg, &val); if (ret) return ret; - return !!(val & rst); + return !!(val & BIT(bit)); } static const struct reset_control_ops aspeed_reset_ops = { @@ -474,6 +502,13 @@ static int aspeed_clk_probe(struct platform_device *pdev) return PTR_ERR(hw); aspeed_clk_data->hws[ASPEED_CLK_BCLK] = hw; + /* Fixed 24MHz clock */ + hw = clk_hw_register_fixed_rate(NULL, "fixed-24m", "clkin", + 0, 24000000); + if (IS_ERR(hw)) + return PTR_ERR(hw); + aspeed_clk_data->hws[ASPEED_CLK_24M] = hw; + /* * TODO: There are a number of clocks that not included in this driver * as more information is required: @@ -627,9 +662,9 @@ static void __init aspeed_cc_init(struct device_node *np) if (!scu_base) return; - aspeed_clk_data = kzalloc(sizeof(*aspeed_clk_data) + - sizeof(*aspeed_clk_data->hws) * ASPEED_NUM_CLKS, - GFP_KERNEL); + aspeed_clk_data = kzalloc(struct_size(aspeed_clk_data, hws, + ASPEED_NUM_CLKS), + GFP_KERNEL); if (!aspeed_clk_data) return; diff --git a/drivers/clk/clk-bulk.c b/drivers/clk/clk-bulk.c index 4c10456f8a32..6904ed6da504 100644 --- a/drivers/clk/clk-bulk.c +++ b/drivers/clk/clk-bulk.c @@ -42,8 +42,9 @@ int __must_check clk_bulk_get(struct device *dev, int num_clks, clks[i].clk = clk_get(dev, clks[i].id); if (IS_ERR(clks[i].clk)) { ret = PTR_ERR(clks[i].clk); - dev_err(dev, "Failed to get clk '%s': %d\n", - clks[i].id, ret); + if (ret != -EPROBE_DEFER) + dev_err(dev, "Failed to get clk '%s': %d\n", + clks[i].id, ret); clks[i].clk = NULL; goto err; } diff --git a/drivers/clk/clk-clps711x.c b/drivers/clk/clk-clps711x.c index 9193f64561f6..2c04396402ab 100644 --- a/drivers/clk/clk-clps711x.c +++ b/drivers/clk/clk-clps711x.c @@ -54,9 +54,9 @@ static struct clps711x_clk * __init _clps711x_clk_init(void __iomem *base, if (!base) return ERR_PTR(-ENOMEM); - clps711x_clk = kzalloc(sizeof(*clps711x_clk) + - sizeof(*clps711x_clk->clk_data.hws) * CLPS711X_CLK_MAX, - GFP_KERNEL); + clps711x_clk = kzalloc(struct_size(clps711x_clk, clk_data.hws, + CLPS711X_CLK_MAX), + GFP_KERNEL); if (!clps711x_clk) return ERR_PTR(-ENOMEM); diff --git a/drivers/clk/clk-efm32gg.c b/drivers/clk/clk-efm32gg.c index f674778fb3ac..f37cf08ff7aa 100644 --- a/drivers/clk/clk-efm32gg.c +++ b/drivers/clk/clk-efm32gg.c @@ -25,8 +25,8 @@ static void __init efm32gg_cmu_init(struct device_node *np) void __iomem *base; struct clk_hw **hws; - clk_data = kzalloc(sizeof(*clk_data) + - sizeof(*clk_data->hws) * CMU_MAX_CLKS, GFP_KERNEL); + clk_data = kzalloc(struct_size(clk_data, hws, CMU_MAX_CLKS), + GFP_KERNEL); if (!clk_data) return; diff --git a/drivers/clk/clk-gemini.c b/drivers/clk/clk-gemini.c index 5e66e6c0205e..b51069e794ff 100644 --- a/drivers/clk/clk-gemini.c +++ b/drivers/clk/clk-gemini.c @@ -399,9 +399,9 @@ static void __init gemini_cc_init(struct device_node *np) int ret; int i; - gemini_clk_data = kzalloc(sizeof(*gemini_clk_data) + - sizeof(*gemini_clk_data->hws) * GEMINI_NUM_CLKS, - GFP_KERNEL); + gemini_clk_data = kzalloc(struct_size(gemini_clk_data, hws, + GEMINI_NUM_CLKS), + GFP_KERNEL); if (!gemini_clk_data) return; diff --git a/drivers/clk/clk-npcm7xx.c b/drivers/clk/clk-npcm7xx.c new file mode 100644 index 000000000000..740af90a9508 --- /dev/null +++ b/drivers/clk/clk-npcm7xx.c @@ -0,0 +1,656 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Nuvoton NPCM7xx Clock Generator + * All the clocks are initialized by the bootloader, so this driver allow only + * reading of current settings directly from the hardware. + * + * Copyright (C) 2018 Nuvoton Technologies tali.perry@nuvoton.com + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +struct npcm7xx_clk_pll { + struct clk_hw hw; + void __iomem *pllcon; + u8 flags; +}; + +#define to_npcm7xx_clk_pll(_hw) container_of(_hw, struct npcm7xx_clk_pll, hw) + +#define PLLCON_LOKI BIT(31) +#define PLLCON_LOKS BIT(30) +#define PLLCON_FBDV GENMASK(27, 16) +#define PLLCON_OTDV2 GENMASK(15, 13) +#define PLLCON_PWDEN BIT(12) +#define PLLCON_OTDV1 GENMASK(10, 8) +#define PLLCON_INDV GENMASK(5, 0) + +static unsigned long npcm7xx_clk_pll_recalc_rate(struct clk_hw *hw, + unsigned long parent_rate) +{ + struct npcm7xx_clk_pll *pll = to_npcm7xx_clk_pll(hw); + unsigned long fbdv, indv, otdv1, otdv2; + unsigned int val; + u64 ret; + + if (parent_rate == 0) { + pr_err("%s: parent rate is zero", __func__); + return 0; + } + + val = readl_relaxed(pll->pllcon); + + indv = FIELD_GET(PLLCON_INDV, val); + fbdv = FIELD_GET(PLLCON_FBDV, val); + otdv1 = FIELD_GET(PLLCON_OTDV1, val); + otdv2 = FIELD_GET(PLLCON_OTDV2, val); + + ret = (u64)parent_rate * fbdv; + do_div(ret, indv * otdv1 * otdv2); + + return ret; +} + +static const struct clk_ops npcm7xx_clk_pll_ops = { + .recalc_rate = npcm7xx_clk_pll_recalc_rate, +}; + +static struct clk_hw * +npcm7xx_clk_register_pll(void __iomem *pllcon, const char *name, + const char *parent_name, unsigned long flags) +{ + struct npcm7xx_clk_pll *pll; + struct clk_init_data init; + struct clk_hw *hw; + int ret; + + pll = kzalloc(sizeof(*pll), GFP_KERNEL); + if (!pll) + return ERR_PTR(-ENOMEM); + + pr_debug("%s reg, name=%s, p=%s\n", __func__, name, parent_name); + + init.name = name; + init.ops = &npcm7xx_clk_pll_ops; + init.parent_names = &parent_name; + init.num_parents = 1; + init.flags = flags; + + pll->pllcon = pllcon; + pll->hw.init = &init; + + hw = &pll->hw; + + ret = clk_hw_register(NULL, hw); + if (ret) { + kfree(pll); + hw = ERR_PTR(ret); + } + + return hw; +} + +#define NPCM7XX_CLKEN1 (0x00) +#define NPCM7XX_CLKEN2 (0x28) +#define NPCM7XX_CLKEN3 (0x30) +#define NPCM7XX_CLKSEL (0x04) +#define NPCM7XX_CLKDIV1 (0x08) +#define NPCM7XX_CLKDIV2 (0x2C) +#define NPCM7XX_CLKDIV3 (0x58) +#define NPCM7XX_PLLCON0 (0x0C) +#define NPCM7XX_PLLCON1 (0x10) +#define NPCM7XX_PLLCON2 (0x54) +#define NPCM7XX_SWRSTR (0x14) +#define NPCM7XX_IRQWAKECON (0x18) +#define NPCM7XX_IRQWAKEFLAG (0x1C) +#define NPCM7XX_IPSRST1 (0x20) +#define NPCM7XX_IPSRST2 (0x24) +#define NPCM7XX_IPSRST3 (0x34) +#define NPCM7XX_WD0RCR (0x38) +#define NPCM7XX_WD1RCR (0x3C) +#define NPCM7XX_WD2RCR (0x40) +#define NPCM7XX_SWRSTC1 (0x44) +#define NPCM7XX_SWRSTC2 (0x48) +#define NPCM7XX_SWRSTC3 (0x4C) +#define NPCM7XX_SWRSTC4 (0x50) +#define NPCM7XX_CORSTC (0x5C) +#define NPCM7XX_PLLCONG (0x60) +#define NPCM7XX_AHBCKFI (0x64) +#define NPCM7XX_SECCNT (0x68) +#define NPCM7XX_CNTR25M (0x6C) + +struct npcm7xx_clk_gate_data { + u32 reg; + u8 bit_idx; + const char *name; + const char *parent_name; + unsigned long flags; + /* + * If this clock is exported via DT, set onecell_idx to constant + * defined in include/dt-bindings/clock/nuvoton, NPCM7XX-clock.h for + * this specific clock. Otherwise, set to -1. + */ + int onecell_idx; +}; + +struct npcm7xx_clk_mux_data { + u8 shift; + u8 mask; + u32 *table; + const char *name; + const char * const *parent_names; + u8 num_parents; + unsigned long flags; + /* + * If this clock is exported via DT, set onecell_idx to constant + * defined in include/dt-bindings/clock/nuvoton, NPCM7XX-clock.h for + * this specific clock. Otherwise, set to -1. + */ + int onecell_idx; + +}; + +struct npcm7xx_clk_div_fixed_data { + u8 mult; + u8 div; + const char *name; + const char *parent_name; + u8 clk_divider_flags; + /* + * If this clock is exported via DT, set onecell_idx to constant + * defined in include/dt-bindings/clock/nuvoton, NPCM7XX-clock.h for + * this specific clock. Otherwise, set to -1. + */ + int onecell_idx; +}; + + +struct npcm7xx_clk_div_data { + u32 reg; + u8 shift; + u8 width; + const char *name; + const char *parent_name; + u8 clk_divider_flags; + unsigned long flags; + /* + * If this clock is exported via DT, set onecell_idx to constant + * defined in include/dt-bindings/clock/nuvoton, NPCM7XX-clock.h for + * this specific clock. Otherwise, set to -1. + */ + int onecell_idx; +}; + +struct npcm7xx_clk_pll_data { + u32 reg; + const char *name; + const char *parent_name; + unsigned long flags; + /* + * If this clock is exported via DT, set onecell_idx to constant + * defined in include/dt-bindings/clock/nuvoton, NPCM7XX-clock.h for + * this specific clock. Otherwise, set to -1. + */ + int onecell_idx; +}; + +/* + * Single copy of strings used to refer to clocks within this driver indexed by + * above enum. + */ +#define NPCM7XX_CLK_S_REFCLK "refclk" +#define NPCM7XX_CLK_S_SYSBYPCK "sysbypck" +#define NPCM7XX_CLK_S_MCBYPCK "mcbypck" +#define NPCM7XX_CLK_S_GFXBYPCK "gfxbypck" +#define NPCM7XX_CLK_S_PLL0 "pll0" +#define NPCM7XX_CLK_S_PLL1 "pll1" +#define NPCM7XX_CLK_S_PLL1_DIV2 "pll1_div2" +#define NPCM7XX_CLK_S_PLL2 "pll2" +#define NPCM7XX_CLK_S_PLL_GFX "pll_gfx" +#define NPCM7XX_CLK_S_PLL2_DIV2 "pll2_div2" +#define NPCM7XX_CLK_S_PIX_MUX "gfx_pixel" +#define NPCM7XX_CLK_S_GPRFSEL_MUX "gprfsel_mux" +#define NPCM7XX_CLK_S_MC_MUX "mc_phy" +#define NPCM7XX_CLK_S_CPU_MUX "cpu" /*AKA system clock.*/ +#define NPCM7XX_CLK_S_MC "mc" +#define NPCM7XX_CLK_S_AXI "axi" /*AKA CLK2*/ +#define NPCM7XX_CLK_S_AHB "ahb" /*AKA CLK4*/ +#define NPCM7XX_CLK_S_CLKOUT_MUX "clkout_mux" +#define NPCM7XX_CLK_S_UART_MUX "uart_mux" +#define NPCM7XX_CLK_S_TIM_MUX "timer_mux" +#define NPCM7XX_CLK_S_SD_MUX "sd_mux" +#define NPCM7XX_CLK_S_GFXM_MUX "gfxm_mux" +#define NPCM7XX_CLK_S_SU_MUX "serial_usb_mux" +#define NPCM7XX_CLK_S_DVC_MUX "dvc_mux" +#define NPCM7XX_CLK_S_GFX_MUX "gfx_mux" +#define NPCM7XX_CLK_S_GFX_PIXEL "gfx_pixel" +#define NPCM7XX_CLK_S_SPI0 "spi0" +#define NPCM7XX_CLK_S_SPI3 "spi3" +#define NPCM7XX_CLK_S_SPIX "spix" +#define NPCM7XX_CLK_S_APB1 "apb1" +#define NPCM7XX_CLK_S_APB2 "apb2" +#define NPCM7XX_CLK_S_APB3 "apb3" +#define NPCM7XX_CLK_S_APB4 "apb4" +#define NPCM7XX_CLK_S_APB5 "apb5" +#define NPCM7XX_CLK_S_TOCK "tock" +#define NPCM7XX_CLK_S_CLKOUT "clkout" +#define NPCM7XX_CLK_S_UART "uart" +#define NPCM7XX_CLK_S_TIMER "timer" +#define NPCM7XX_CLK_S_MMC "mmc" +#define NPCM7XX_CLK_S_SDHC "sdhc" +#define NPCM7XX_CLK_S_ADC "adc" +#define NPCM7XX_CLK_S_GFX "gfx0_gfx1_mem" +#define NPCM7XX_CLK_S_USBIF "serial_usbif" +#define NPCM7XX_CLK_S_USB_HOST "usb_host" +#define NPCM7XX_CLK_S_USB_BRIDGE "usb_bridge" +#define NPCM7XX_CLK_S_PCI "pci" + +static u32 pll_mux_table[] = {0, 1, 2, 3}; +static const char * const pll_mux_parents[] __initconst = { + NPCM7XX_CLK_S_PLL0, + NPCM7XX_CLK_S_PLL1_DIV2, + NPCM7XX_CLK_S_REFCLK, + NPCM7XX_CLK_S_PLL2_DIV2, +}; + +static u32 cpuck_mux_table[] = {0, 1, 2, 3}; +static const char * const cpuck_mux_parents[] __initconst = { + NPCM7XX_CLK_S_PLL0, + NPCM7XX_CLK_S_PLL1_DIV2, + NPCM7XX_CLK_S_REFCLK, + NPCM7XX_CLK_S_SYSBYPCK, +}; + +static u32 pixcksel_mux_table[] = {0, 2}; +static const char * const pixcksel_mux_parents[] __initconst = { + NPCM7XX_CLK_S_PLL_GFX, + NPCM7XX_CLK_S_REFCLK, +}; + +static u32 sucksel_mux_table[] = {2, 3}; +static const char * const sucksel_mux_parents[] __initconst = { + NPCM7XX_CLK_S_REFCLK, + NPCM7XX_CLK_S_PLL2_DIV2, +}; + +static u32 mccksel_mux_table[] = {0, 2, 3}; +static const char * const mccksel_mux_parents[] __initconst = { + NPCM7XX_CLK_S_PLL1_DIV2, + NPCM7XX_CLK_S_REFCLK, + NPCM7XX_CLK_S_MCBYPCK, +}; + +static u32 clkoutsel_mux_table[] = {0, 1, 2, 3, 4}; +static const char * const clkoutsel_mux_parents[] __initconst = { + NPCM7XX_CLK_S_PLL0, + NPCM7XX_CLK_S_PLL1_DIV2, + NPCM7XX_CLK_S_REFCLK, + NPCM7XX_CLK_S_PLL_GFX, // divided by 2 + NPCM7XX_CLK_S_PLL2_DIV2, +}; + +static u32 gfxmsel_mux_table[] = {2, 3}; +static const char * const gfxmsel_mux_parents[] __initconst = { + NPCM7XX_CLK_S_REFCLK, + NPCM7XX_CLK_S_PLL2_DIV2, +}; + +static u32 dvcssel_mux_table[] = {2, 3}; +static const char * const dvcssel_mux_parents[] __initconst = { + NPCM7XX_CLK_S_REFCLK, + NPCM7XX_CLK_S_PLL2, +}; + +static const struct npcm7xx_clk_pll_data npcm7xx_plls[] __initconst = { + {NPCM7XX_PLLCON0, NPCM7XX_CLK_S_PLL0, NPCM7XX_CLK_S_REFCLK, 0, -1}, + + {NPCM7XX_PLLCON1, NPCM7XX_CLK_S_PLL1, + NPCM7XX_CLK_S_REFCLK, 0, -1}, + + {NPCM7XX_PLLCON2, NPCM7XX_CLK_S_PLL2, + NPCM7XX_CLK_S_REFCLK, 0, -1}, + + {NPCM7XX_PLLCONG, NPCM7XX_CLK_S_PLL_GFX, + NPCM7XX_CLK_S_REFCLK, 0, -1}, +}; + +static const struct npcm7xx_clk_mux_data npcm7xx_muxes[] __initconst = { + {0, GENMASK(1, 0), cpuck_mux_table, NPCM7XX_CLK_S_CPU_MUX, + cpuck_mux_parents, ARRAY_SIZE(cpuck_mux_parents), CLK_IS_CRITICAL, + NPCM7XX_CLK_CPU}, + + {4, GENMASK(1, 0), pixcksel_mux_table, NPCM7XX_CLK_S_PIX_MUX, + pixcksel_mux_parents, ARRAY_SIZE(pixcksel_mux_parents), 0, + NPCM7XX_CLK_GFX_PIXEL}, + + {6, GENMASK(1, 0), pll_mux_table, NPCM7XX_CLK_S_SD_MUX, + pll_mux_parents, ARRAY_SIZE(pll_mux_parents), 0, -1}, + + {8, GENMASK(1, 0), pll_mux_table, NPCM7XX_CLK_S_UART_MUX, + pll_mux_parents, ARRAY_SIZE(pll_mux_parents), 0, -1}, + + {10, GENMASK(1, 0), sucksel_mux_table, NPCM7XX_CLK_S_SU_MUX, + sucksel_mux_parents, ARRAY_SIZE(sucksel_mux_parents), 0, -1}, + + {12, GENMASK(1, 0), mccksel_mux_table, NPCM7XX_CLK_S_MC_MUX, + mccksel_mux_parents, ARRAY_SIZE(mccksel_mux_parents), 0, -1}, + + {14, GENMASK(1, 0), pll_mux_table, NPCM7XX_CLK_S_TIM_MUX, + pll_mux_parents, ARRAY_SIZE(pll_mux_parents), 0, -1}, + + {16, GENMASK(1, 0), pll_mux_table, NPCM7XX_CLK_S_GFX_MUX, + pll_mux_parents, ARRAY_SIZE(pll_mux_parents), 0, -1}, + + {18, GENMASK(2, 0), clkoutsel_mux_table, NPCM7XX_CLK_S_CLKOUT_MUX, + clkoutsel_mux_parents, ARRAY_SIZE(clkoutsel_mux_parents), 0, -1}, + + {21, GENMASK(1, 0), gfxmsel_mux_table, NPCM7XX_CLK_S_GFXM_MUX, + gfxmsel_mux_parents, ARRAY_SIZE(gfxmsel_mux_parents), 0, -1}, + + {23, GENMASK(1, 0), dvcssel_mux_table, NPCM7XX_CLK_S_DVC_MUX, + dvcssel_mux_parents, ARRAY_SIZE(dvcssel_mux_parents), 0, -1}, +}; + +/* fixed ratio dividers (no register): */ +static const struct npcm7xx_clk_div_fixed_data npcm7xx_divs_fx[] __initconst = { + { 1, 2, NPCM7XX_CLK_S_MC, NPCM7XX_CLK_S_MC_MUX, 0, NPCM7XX_CLK_MC}, + { 1, 2, NPCM7XX_CLK_S_PLL1_DIV2, NPCM7XX_CLK_S_PLL1, 0, -1}, + { 1, 2, NPCM7XX_CLK_S_PLL2_DIV2, NPCM7XX_CLK_S_PLL2, 0, -1}, +}; + +/* configurable dividers: */ +static const struct npcm7xx_clk_div_data npcm7xx_divs[] __initconst = { + {NPCM7XX_CLKDIV1, 28, 3, NPCM7XX_CLK_S_ADC, + NPCM7XX_CLK_S_TIMER, CLK_DIVIDER_POWER_OF_TWO, 0, NPCM7XX_CLK_ADC}, + /*30-28 ADCCKDIV*/ + {NPCM7XX_CLKDIV1, 26, 2, NPCM7XX_CLK_S_AHB, + NPCM7XX_CLK_S_AXI, 0, CLK_IS_CRITICAL, NPCM7XX_CLK_AHB}, + /*27-26 CLK4DIV*/ + {NPCM7XX_CLKDIV1, 21, 5, NPCM7XX_CLK_S_TIMER, + NPCM7XX_CLK_S_TIM_MUX, 0, 0, NPCM7XX_CLK_TIMER}, + /*25-21 TIMCKDIV*/ + {NPCM7XX_CLKDIV1, 16, 5, NPCM7XX_CLK_S_UART, + NPCM7XX_CLK_S_UART_MUX, 0, 0, NPCM7XX_CLK_UART}, + /*20-16 UARTDIV*/ + {NPCM7XX_CLKDIV1, 11, 5, NPCM7XX_CLK_S_MMC, + NPCM7XX_CLK_S_SD_MUX, 0, 0, NPCM7XX_CLK_MMC}, + /*15-11 MMCCKDIV*/ + {NPCM7XX_CLKDIV1, 6, 5, NPCM7XX_CLK_S_SPI3, + NPCM7XX_CLK_S_AHB, 0, 0, NPCM7XX_CLK_SPI3}, + /*10-6 AHB3CKDIV*/ + {NPCM7XX_CLKDIV1, 2, 4, NPCM7XX_CLK_S_PCI, + NPCM7XX_CLK_S_GFX_MUX, 0, 0, NPCM7XX_CLK_PCI}, + /*5-2 PCICKDIV*/ + {NPCM7XX_CLKDIV1, 0, 1, NPCM7XX_CLK_S_AXI, + NPCM7XX_CLK_S_CPU_MUX, CLK_DIVIDER_POWER_OF_TWO, CLK_IS_CRITICAL, + NPCM7XX_CLK_AXI},/*0 CLK2DIV*/ + + {NPCM7XX_CLKDIV2, 30, 2, NPCM7XX_CLK_S_APB4, + NPCM7XX_CLK_S_AHB, CLK_DIVIDER_POWER_OF_TWO, 0, NPCM7XX_CLK_APB4}, + /*31-30 APB4CKDIV*/ + {NPCM7XX_CLKDIV2, 28, 2, NPCM7XX_CLK_S_APB3, + NPCM7XX_CLK_S_AHB, CLK_DIVIDER_POWER_OF_TWO, 0, NPCM7XX_CLK_APB3}, + /*29-28 APB3CKDIV*/ + {NPCM7XX_CLKDIV2, 26, 2, NPCM7XX_CLK_S_APB2, + NPCM7XX_CLK_S_AHB, CLK_DIVIDER_POWER_OF_TWO, 0, NPCM7XX_CLK_APB2}, + /*27-26 APB2CKDIV*/ + {NPCM7XX_CLKDIV2, 24, 2, NPCM7XX_CLK_S_APB1, + NPCM7XX_CLK_S_AHB, CLK_DIVIDER_POWER_OF_TWO, 0, NPCM7XX_CLK_APB1}, + /*25-24 APB1CKDIV*/ + {NPCM7XX_CLKDIV2, 22, 2, NPCM7XX_CLK_S_APB5, + NPCM7XX_CLK_S_AHB, CLK_DIVIDER_POWER_OF_TWO, 0, NPCM7XX_CLK_APB5}, + /*23-22 APB5CKDIV*/ + {NPCM7XX_CLKDIV2, 16, 5, NPCM7XX_CLK_S_CLKOUT, + NPCM7XX_CLK_S_CLKOUT_MUX, 0, 0, NPCM7XX_CLK_CLKOUT}, + /*20-16 CLKOUTDIV*/ + {NPCM7XX_CLKDIV2, 13, 3, NPCM7XX_CLK_S_GFX, + NPCM7XX_CLK_S_GFX_MUX, 0, 0, NPCM7XX_CLK_GFX}, + /*15-13 GFXCKDIV*/ + {NPCM7XX_CLKDIV2, 8, 5, NPCM7XX_CLK_S_USB_BRIDGE, + NPCM7XX_CLK_S_SU_MUX, 0, 0, NPCM7XX_CLK_SU}, + /*12-8 SUCKDIV*/ + {NPCM7XX_CLKDIV2, 4, 4, NPCM7XX_CLK_S_USB_HOST, + NPCM7XX_CLK_S_SU_MUX, 0, 0, NPCM7XX_CLK_SU48}, + /*7-4 SU48CKDIV*/ + {NPCM7XX_CLKDIV2, 0, 4, NPCM7XX_CLK_S_SDHC, + NPCM7XX_CLK_S_SD_MUX, 0, 0, NPCM7XX_CLK_SDHC} + ,/*3-0 SD1CKDIV*/ + + {NPCM7XX_CLKDIV3, 6, 5, NPCM7XX_CLK_S_SPI0, + NPCM7XX_CLK_S_AHB, 0, 0, NPCM7XX_CLK_SPI0}, + /*10-6 SPI0CKDV*/ + {NPCM7XX_CLKDIV3, 1, 5, NPCM7XX_CLK_S_SPIX, + NPCM7XX_CLK_S_AHB, 0, 0, NPCM7XX_CLK_SPIX}, + /*5-1 SPIXCKDV*/ + +}; + +static const struct npcm7xx_clk_gate_data npcm7xx_gates[] __initconst = { + {NPCM7XX_CLKEN1, 31, "smb1-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN1, 30, "smb0-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN1, 29, "smb7-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN1, 28, "smb6-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN1, 27, "adc-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN1, 26, "wdt-gate", NPCM7XX_CLK_S_TIMER, 0}, + {NPCM7XX_CLKEN1, 25, "usbdev3-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN1, 24, "usbdev6-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN1, 23, "usbdev5-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN1, 22, "usbdev4-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN1, 21, "emc2-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN1, 20, "timer5_9-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN1, 19, "timer0_4-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN1, 18, "pwmm0-gate", NPCM7XX_CLK_S_APB3, 0}, + {NPCM7XX_CLKEN1, 17, "huart-gate", NPCM7XX_CLK_S_UART, 0}, + {NPCM7XX_CLKEN1, 16, "smb5-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN1, 15, "smb4-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN1, 14, "smb3-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN1, 13, "smb2-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN1, 12, "mc-gate", NPCM7XX_CLK_S_MC, 0}, + {NPCM7XX_CLKEN1, 11, "uart01-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN1, 10, "aes-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN1, 9, "peci-gate", NPCM7XX_CLK_S_APB3, 0}, + {NPCM7XX_CLKEN1, 8, "usbdev2-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN1, 7, "uart23-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN1, 6, "emc1-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN1, 5, "usbdev1-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN1, 4, "shm-gate", NPCM7XX_CLK_S_AHB, 0}, + /* bit 3 is reserved */ + {NPCM7XX_CLKEN1, 2, "kcs-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN1, 1, "spi3-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN1, 0, "spi0-gate", NPCM7XX_CLK_S_AHB, 0}, + + {NPCM7XX_CLKEN2, 31, "cp-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN2, 30, "tock-gate", NPCM7XX_CLK_S_TOCK, 0}, + /* bit 29 is reserved */ + {NPCM7XX_CLKEN2, 28, "gmac1-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN2, 27, "usbif-gate", NPCM7XX_CLK_S_USBIF, 0}, + {NPCM7XX_CLKEN2, 26, "usbhost-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN2, 25, "gmac2-gate", NPCM7XX_CLK_S_AHB, 0}, + /* bit 24 is reserved */ + {NPCM7XX_CLKEN2, 23, "pspi2-gate", NPCM7XX_CLK_S_APB5, 0}, + {NPCM7XX_CLKEN2, 22, "pspi1-gate", NPCM7XX_CLK_S_APB5, 0}, + {NPCM7XX_CLKEN2, 21, "3des-gate", NPCM7XX_CLK_S_AHB, 0}, + /* bit 20 is reserved */ + {NPCM7XX_CLKEN2, 19, "siox2-gate", NPCM7XX_CLK_S_APB3, 0}, + {NPCM7XX_CLKEN2, 18, "siox1-gate", NPCM7XX_CLK_S_APB3, 0}, + /* bit 17 is reserved */ + {NPCM7XX_CLKEN2, 16, "fuse-gate", NPCM7XX_CLK_S_APB4, 0}, + /* bit 15 is reserved */ + {NPCM7XX_CLKEN2, 14, "vcd-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN2, 13, "ece-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN2, 12, "vdma-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN2, 11, "ahbpcibrg-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN2, 10, "gfxsys-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN2, 9, "sdhc-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN2, 8, "mmc-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN2, 7, "mft7-gate", NPCM7XX_CLK_S_APB4, 0}, + {NPCM7XX_CLKEN2, 6, "mft6-gate", NPCM7XX_CLK_S_APB4, 0}, + {NPCM7XX_CLKEN2, 5, "mft5-gate", NPCM7XX_CLK_S_APB4, 0}, + {NPCM7XX_CLKEN2, 4, "mft4-gate", NPCM7XX_CLK_S_APB4, 0}, + {NPCM7XX_CLKEN2, 3, "mft3-gate", NPCM7XX_CLK_S_APB4, 0}, + {NPCM7XX_CLKEN2, 2, "mft2-gate", NPCM7XX_CLK_S_APB4, 0}, + {NPCM7XX_CLKEN2, 1, "mft1-gate", NPCM7XX_CLK_S_APB4, 0}, + {NPCM7XX_CLKEN2, 0, "mft0-gate", NPCM7XX_CLK_S_APB4, 0}, + + {NPCM7XX_CLKEN3, 31, "gpiom7-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN3, 30, "gpiom6-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN3, 29, "gpiom5-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN3, 28, "gpiom4-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN3, 27, "gpiom3-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN3, 26, "gpiom2-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN3, 25, "gpiom1-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN3, 24, "gpiom0-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN3, 23, "espi-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN3, 22, "smb11-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN3, 21, "smb10-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN3, 20, "smb9-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN3, 19, "smb8-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN3, 18, "smb15-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN3, 17, "rng-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN3, 16, "timer10_14-gate", NPCM7XX_CLK_S_APB1, 0}, + {NPCM7XX_CLKEN3, 15, "pcirc-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN3, 14, "sececc-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN3, 13, "sha-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN3, 12, "smb14-gate", NPCM7XX_CLK_S_APB2, 0}, + /* bit 11 is reserved */ + /* bit 10 is reserved */ + {NPCM7XX_CLKEN3, 9, "pcimbx-gate", NPCM7XX_CLK_S_AHB, 0}, + /* bit 8 is reserved */ + {NPCM7XX_CLKEN3, 7, "usbdev9-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN3, 6, "usbdev8-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN3, 5, "usbdev7-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN3, 4, "usbdev0-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN3, 3, "smb13-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN3, 2, "spix-gate", NPCM7XX_CLK_S_AHB, 0}, + {NPCM7XX_CLKEN3, 1, "smb12-gate", NPCM7XX_CLK_S_APB2, 0}, + {NPCM7XX_CLKEN3, 0, "pwmm1-gate", NPCM7XX_CLK_S_APB3, 0}, +}; + +static DEFINE_SPINLOCK(npcm7xx_clk_lock); + +static void __init npcm7xx_clk_init(struct device_node *clk_np) +{ + struct clk_hw_onecell_data *npcm7xx_clk_data; + void __iomem *clk_base; + struct resource res; + struct clk_hw *hw; + int ret; + int i; + + ret = of_address_to_resource(clk_np, 0, &res); + if (ret) { + pr_err("%s: failed to get resource, ret %d\n", clk_np->name, + ret); + return; + } + + clk_base = ioremap(res.start, resource_size(&res)); + if (!clk_base) + goto npcm7xx_init_error; + + npcm7xx_clk_data = kzalloc(sizeof(*npcm7xx_clk_data->hws) * + NPCM7XX_NUM_CLOCKS + sizeof(npcm7xx_clk_data), GFP_KERNEL); + if (!npcm7xx_clk_data) + goto npcm7xx_init_np_err; + + npcm7xx_clk_data->num = NPCM7XX_NUM_CLOCKS; + + for (i = 0; i < NPCM7XX_NUM_CLOCKS; i++) + npcm7xx_clk_data->hws[i] = ERR_PTR(-EPROBE_DEFER); + + /* Register plls */ + for (i = 0; i < ARRAY_SIZE(npcm7xx_plls); i++) { + const struct npcm7xx_clk_pll_data *pll_data = &npcm7xx_plls[i]; + + hw = npcm7xx_clk_register_pll(clk_base + pll_data->reg, + pll_data->name, pll_data->parent_name, pll_data->flags); + if (IS_ERR(hw)) { + pr_err("npcm7xx_clk: Can't register pll\n"); + goto npcm7xx_init_fail; + } + + if (pll_data->onecell_idx >= 0) + npcm7xx_clk_data->hws[pll_data->onecell_idx] = hw; + } + + /* Register fixed dividers */ + hw = clk_hw_register_fixed_factor(NULL, NPCM7XX_CLK_S_PLL1_DIV2, + NPCM7XX_CLK_S_PLL1, 0, 1, 2); + if (IS_ERR(hw)) { + pr_err("npcm7xx_clk: Can't register fixed div\n"); + goto npcm7xx_init_fail; + } + + hw = clk_hw_register_fixed_factor(NULL, NPCM7XX_CLK_S_PLL2_DIV2, + NPCM7XX_CLK_S_PLL2, 0, 1, 2); + if (IS_ERR(hw)) { + pr_err("npcm7xx_clk: Can't register div2\n"); + goto npcm7xx_init_fail; + } + + /* Register muxes */ + for (i = 0; i < ARRAY_SIZE(npcm7xx_muxes); i++) { + const struct npcm7xx_clk_mux_data *mux_data = &npcm7xx_muxes[i]; + + hw = clk_hw_register_mux_table(NULL, + mux_data->name, + mux_data->parent_names, mux_data->num_parents, + mux_data->flags, clk_base + NPCM7XX_CLKSEL, + mux_data->shift, mux_data->mask, 0, + mux_data->table, &npcm7xx_clk_lock); + + if (IS_ERR(hw)) { + pr_err("npcm7xx_clk: Can't register mux\n"); + goto npcm7xx_init_fail; + } + + if (mux_data->onecell_idx >= 0) + npcm7xx_clk_data->hws[mux_data->onecell_idx] = hw; + } + + /* Register clock dividers specified in npcm7xx_divs */ + for (i = 0; i < ARRAY_SIZE(npcm7xx_divs); i++) { + const struct npcm7xx_clk_div_data *div_data = &npcm7xx_divs[i]; + + hw = clk_hw_register_divider(NULL, div_data->name, + div_data->parent_name, + div_data->flags, + clk_base + div_data->reg, + div_data->shift, div_data->width, + div_data->clk_divider_flags, &npcm7xx_clk_lock); + if (IS_ERR(hw)) { + pr_err("npcm7xx_clk: Can't register div table\n"); + goto npcm7xx_init_fail; + } + + if (div_data->onecell_idx >= 0) + npcm7xx_clk_data->hws[div_data->onecell_idx] = hw; + } + + ret = of_clk_add_hw_provider(clk_np, of_clk_hw_onecell_get, + npcm7xx_clk_data); + if (ret) + pr_err("failed to add DT provider: %d\n", ret); + + of_node_put(clk_np); + + return; + +npcm7xx_init_fail: + kfree(npcm7xx_clk_data->hws); +npcm7xx_init_np_err: + iounmap(clk_base); +npcm7xx_init_error: + of_node_put(clk_np); +} +CLK_OF_DECLARE(npcm7xx_clk_init, "nuvoton,npcm750-clk", npcm7xx_clk_init); diff --git a/drivers/clk/clk-s2mps11.c b/drivers/clk/clk-s2mps11.c index fbaa84a33c46..d44e0eea31ec 100644 --- a/drivers/clk/clk-s2mps11.c +++ b/drivers/clk/clk-s2mps11.c @@ -147,8 +147,8 @@ static int s2mps11_clk_probe(struct platform_device *pdev) if (!s2mps11_clks) return -ENOMEM; - clk_data = devm_kzalloc(&pdev->dev, sizeof(*clk_data) + - sizeof(*clk_data->hws) * S2MPS11_CLKS_NUM, + clk_data = devm_kzalloc(&pdev->dev, + struct_size(clk_data, hws, S2MPS11_CLKS_NUM), GFP_KERNEL); if (!clk_data) return -ENOMEM; diff --git a/drivers/clk/clk-scmi.c b/drivers/clk/clk-scmi.c index 488c21376b55..bb2a6f2f5516 100644 --- a/drivers/clk/clk-scmi.c +++ b/drivers/clk/clk-scmi.c @@ -137,8 +137,8 @@ static int scmi_clocks_probe(struct scmi_device *sdev) return -EINVAL; } - clk_data = devm_kzalloc(dev, sizeof(*clk_data) + - sizeof(*clk_data->hws) * count, GFP_KERNEL); + clk_data = devm_kzalloc(dev, struct_size(clk_data, hws, count), + GFP_KERNEL); if (!clk_data) return -ENOMEM; diff --git a/drivers/clk/clk-si544.c b/drivers/clk/clk-si544.c index 1c96a9f6c022..1e2a3b8f9454 100644 --- a/drivers/clk/clk-si544.c +++ b/drivers/clk/clk-si544.c @@ -207,6 +207,7 @@ static int si544_calc_muldiv(struct clk_si544_muldiv *settings, /* And the fractional bits using the remainder */ vco = (u64)tmp << 32; + vco += FXO / 2; /* Round to nearest multiple */ do_div(vco, FXO); settings->fb_div_frac = vco; diff --git a/drivers/clk/clk-stm32h7.c b/drivers/clk/clk-stm32h7.c index db2b162c0d4c..d3271eca3779 100644 --- a/drivers/clk/clk-stm32h7.c +++ b/drivers/clk/clk-stm32h7.c @@ -1201,9 +1201,8 @@ static void __init stm32h7_rcc_init(struct device_node *np) const char *hse_clk, *lse_clk, *i2s_clk; struct regmap *pdrm; - clk_data = kzalloc(sizeof(*clk_data) + - sizeof(*clk_data->hws) * STM32H7_MAX_CLKS, - GFP_KERNEL); + clk_data = kzalloc(struct_size(clk_data, hws, STM32H7_MAX_CLKS), + GFP_KERNEL); if (!clk_data) return; diff --git a/drivers/clk/clk-stm32mp1.c b/drivers/clk/clk-stm32mp1.c index edd3cf451401..a907555b2a3d 100644 --- a/drivers/clk/clk-stm32mp1.c +++ b/drivers/clk/clk-stm32mp1.c @@ -579,14 +579,9 @@ clk_stm32_register_gate_ops(struct device *dev, spinlock_t *lock) { struct clk_init_data init = { NULL }; - struct clk_gate *gate; struct clk_hw *hw; int ret; - gate = kzalloc(sizeof(*gate), GFP_KERNEL); - if (!gate) - return ERR_PTR(-ENOMEM); - init.name = name; init.parent_names = &parent_name; init.num_parents = 1; @@ -604,10 +599,8 @@ clk_stm32_register_gate_ops(struct device *dev, hw->init = &init; ret = clk_hw_register(dev, hw); - if (ret) { - kfree(gate); + if (ret) hw = ERR_PTR(ret); - } return hw; } @@ -1988,7 +1981,8 @@ static const struct clock_config stm32mp1_clock_cfg[] = { _DIV(RCC_MCO2CFGR, 4, 4, 0, NULL)), /* Debug clocks */ - GATE(CK_DBG, "ck_sys_dbg", "ck_axi", 0, RCC_DBGCFGR, 8, 0), + GATE(CK_DBG, "ck_sys_dbg", "ck_axi", CLK_IGNORE_UNUSED, + RCC_DBGCFGR, 8, 0), COMPOSITE(CK_TRACE, "ck_trace", ck_trace_src, CLK_OPS_PARENT_ENABLE, _GATE(RCC_DBGCFGR, 9, 0), @@ -2060,9 +2054,8 @@ static int stm32_rcc_init(struct device_node *np, max_binding = data->maxbinding; - clk_data = kzalloc(sizeof(*clk_data) + - sizeof(*clk_data->hws) * max_binding, - GFP_KERNEL); + clk_data = kzalloc(struct_size(clk_data, hws, max_binding), + GFP_KERNEL); if (!clk_data) return -ENOMEM; diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c index 7af555f0e60c..9760b526ca31 100644 --- a/drivers/clk/clk.c +++ b/drivers/clk/clk.c @@ -6,7 +6,7 @@ * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. * - * Standard functionality for the common clock API. See Documentation/clk.txt + * Standard functionality for the common clock API. See Documentation/driver-api/clk.rst */ #include @@ -549,7 +549,8 @@ static void clk_core_rate_unprotect(struct clk_core *core) if (!core) return; - if (WARN_ON(core->protect_count == 0)) + if (WARN(core->protect_count == 0, + "%s already unprotected\n", core->name)) return; if (--core->protect_count > 0) @@ -682,16 +683,18 @@ static void clk_core_unprepare(struct clk_core *core) if (!core) return; - if (WARN_ON(core->prepare_count == 0)) + if (WARN(core->prepare_count == 0, + "%s already unprepared\n", core->name)) return; - if (WARN_ON(core->prepare_count == 1 && core->flags & CLK_IS_CRITICAL)) + if (WARN(core->prepare_count == 1 && core->flags & CLK_IS_CRITICAL, + "Unpreparing critical %s\n", core->name)) return; if (--core->prepare_count > 0) return; - WARN_ON(core->enable_count > 0); + WARN(core->enable_count > 0, "Unpreparing enabled %s\n", core->name); trace_clk_unprepare(core); @@ -809,10 +812,11 @@ static void clk_core_disable(struct clk_core *core) if (!core) return; - if (WARN_ON(core->enable_count == 0)) + if (WARN(core->enable_count == 0, "%s already disabled\n", core->name)) return; - if (WARN_ON(core->enable_count == 1 && core->flags & CLK_IS_CRITICAL)) + if (WARN(core->enable_count == 1 && core->flags & CLK_IS_CRITICAL, + "Disabling critical %s\n", core->name)) return; if (--core->enable_count > 0) @@ -867,7 +871,8 @@ static int clk_core_enable(struct clk_core *core) if (!core) return 0; - if (WARN_ON(core->prepare_count == 0)) + if (WARN(core->prepare_count == 0, + "Enabling unprepared %s\n", core->name)) return -ESHUTDOWN; if (core->enable_count == 0) { @@ -2171,7 +2176,6 @@ void clk_hw_reparent(struct clk_hw *hw, struct clk_hw *new_parent) bool clk_has_parent(struct clk *clk, struct clk *parent) { struct clk_core *core, *parent_core; - unsigned int i; /* NULL clocks should be nops, so return success if either is NULL. */ if (!clk || !parent) @@ -2184,11 +2188,8 @@ bool clk_has_parent(struct clk *clk, struct clk *parent) if (core->parent == parent_core) return true; - for (i = 0; i < core->num_parents; i++) - if (strcmp(core->parent_names[i], parent_core->name) == 0) - return true; - - return false; + return match_string(core->parent_names, core->num_parents, + parent_core->name) >= 0; } EXPORT_SYMBOL_GPL(clk_has_parent); @@ -2609,81 +2610,31 @@ static int possible_parents_show(struct seq_file *s, void *data) } DEFINE_SHOW_ATTRIBUTE(possible_parents); -static int clk_debug_create_one(struct clk_core *core, struct dentry *pdentry) +static void clk_debug_create_one(struct clk_core *core, struct dentry *pdentry) { - struct dentry *d; - int ret = -ENOMEM; - - if (!core || !pdentry) { - ret = -EINVAL; - goto out; - } - - d = debugfs_create_dir(core->name, pdentry); - if (!d) - goto out; + struct dentry *root; - core->dentry = d; - - d = debugfs_create_ulong("clk_rate", 0444, core->dentry, &core->rate); - if (!d) - goto err_out; - - d = debugfs_create_ulong("clk_accuracy", 0444, core->dentry, - &core->accuracy); - if (!d) - goto err_out; - - d = debugfs_create_u32("clk_phase", 0444, core->dentry, &core->phase); - if (!d) - goto err_out; - - d = debugfs_create_file("clk_flags", 0444, core->dentry, core, - &clk_flags_fops); - if (!d) - goto err_out; - - d = debugfs_create_u32("clk_prepare_count", 0444, core->dentry, - &core->prepare_count); - if (!d) - goto err_out; - - d = debugfs_create_u32("clk_enable_count", 0444, core->dentry, - &core->enable_count); - if (!d) - goto err_out; - - d = debugfs_create_u32("clk_protect_count", 0444, core->dentry, - &core->protect_count); - if (!d) - goto err_out; - - d = debugfs_create_u32("clk_notifier_count", 0444, core->dentry, - &core->notifier_count); - if (!d) - goto err_out; + if (!core || !pdentry) + return; - if (core->num_parents > 1) { - d = debugfs_create_file("clk_possible_parents", 0444, - core->dentry, core, &possible_parents_fops); - if (!d) - goto err_out; - } + root = debugfs_create_dir(core->name, pdentry); + core->dentry = root; - if (core->ops->debug_init) { - ret = core->ops->debug_init(core->hw, core->dentry); - if (ret) - goto err_out; - } + debugfs_create_ulong("clk_rate", 0444, root, &core->rate); + debugfs_create_ulong("clk_accuracy", 0444, root, &core->accuracy); + debugfs_create_u32("clk_phase", 0444, root, &core->phase); + debugfs_create_file("clk_flags", 0444, root, core, &clk_flags_fops); + debugfs_create_u32("clk_prepare_count", 0444, root, &core->prepare_count); + debugfs_create_u32("clk_enable_count", 0444, root, &core->enable_count); + debugfs_create_u32("clk_protect_count", 0444, root, &core->protect_count); + debugfs_create_u32("clk_notifier_count", 0444, root, &core->notifier_count); - ret = 0; - goto out; + if (core->num_parents > 1) + debugfs_create_file("clk_possible_parents", 0444, root, core, + &possible_parents_fops); -err_out: - debugfs_remove_recursive(core->dentry); - core->dentry = NULL; -out: - return ret; + if (core->ops->debug_init) + core->ops->debug_init(core->hw, core->dentry); } /** @@ -2694,17 +2645,13 @@ out: * initialized. Otherwise it bails out early since the debugfs clk directory * will be created lazily by clk_debug_init as part of a late_initcall. */ -static int clk_debug_register(struct clk_core *core) +static void clk_debug_register(struct clk_core *core) { - int ret = 0; - mutex_lock(&clk_debug_lock); hlist_add_head(&core->debug_node, &clk_debug_list); if (inited) - ret = clk_debug_create_one(core, rootdir); + clk_debug_create_one(core, rootdir); mutex_unlock(&clk_debug_lock); - - return ret; } /** @@ -2724,19 +2671,6 @@ static void clk_debug_unregister(struct clk_core *core) mutex_unlock(&clk_debug_lock); } -struct dentry *clk_debugfs_add_file(struct clk_hw *hw, char *name, umode_t mode, - void *data, const struct file_operations *fops) -{ - struct dentry *d = NULL; - - if (hw->core->dentry) - d = debugfs_create_file(name, mode, hw->core->dentry, data, - fops); - - return d; -} -EXPORT_SYMBOL_GPL(clk_debugfs_add_file); - /** * clk_debug_init - lazily populate the debugfs clk directory * @@ -2749,32 +2683,17 @@ EXPORT_SYMBOL_GPL(clk_debugfs_add_file); static int __init clk_debug_init(void) { struct clk_core *core; - struct dentry *d; rootdir = debugfs_create_dir("clk", NULL); - if (!rootdir) - return -ENOMEM; - - d = debugfs_create_file("clk_summary", 0444, rootdir, &all_lists, - &clk_summary_fops); - if (!d) - return -ENOMEM; - - d = debugfs_create_file("clk_dump", 0444, rootdir, &all_lists, - &clk_dump_fops); - if (!d) - return -ENOMEM; - - d = debugfs_create_file("clk_orphan_summary", 0444, rootdir, - &orphan_list, &clk_summary_fops); - if (!d) - return -ENOMEM; - - d = debugfs_create_file("clk_orphan_dump", 0444, rootdir, - &orphan_list, &clk_dump_fops); - if (!d) - return -ENOMEM; + debugfs_create_file("clk_summary", 0444, rootdir, &all_lists, + &clk_summary_fops); + debugfs_create_file("clk_dump", 0444, rootdir, &all_lists, + &clk_dump_fops); + debugfs_create_file("clk_orphan_summary", 0444, rootdir, &orphan_list, + &clk_summary_fops); + debugfs_create_file("clk_orphan_dump", 0444, rootdir, &orphan_list, + &clk_dump_fops); mutex_lock(&clk_debug_lock); hlist_for_each_entry(core, &clk_debug_list, debug_node) @@ -2787,7 +2706,7 @@ static int __init clk_debug_init(void) } late_initcall(clk_debug_init); #else -static inline int clk_debug_register(struct clk_core *core) { return 0; } +static inline void clk_debug_register(struct clk_core *core) { } static inline void clk_debug_reparent(struct clk_core *core, struct clk_core *new_parent) { @@ -2828,7 +2747,7 @@ static int __clk_core_init(struct clk_core *core) goto out; } - /* check that clk_ops are sane. See Documentation/clk.txt */ + /* check that clk_ops are sane. See Documentation/driver-api/clk.rst */ if (core->ops->set_rate && !((core->ops->round_rate || core->ops->determine_rate) && core->ops->recalc_rate)) { @@ -3907,7 +3826,7 @@ int of_clk_parent_fill(struct device_node *np, const char **parents, EXPORT_SYMBOL_GPL(of_clk_parent_fill); struct clock_provider { - of_clk_init_cb_t clk_init_cb; + void (*clk_init_cb)(struct device_node *); struct device_node *np; struct list_head node; }; diff --git a/drivers/clk/davinci/da8xx-cfgchip.c b/drivers/clk/davinci/da8xx-cfgchip.c index c971111d2601..aae62a5b8734 100644 --- a/drivers/clk/davinci/da8xx-cfgchip.c +++ b/drivers/clk/davinci/da8xx-cfgchip.c @@ -650,8 +650,8 @@ static int of_da8xx_usb_phy_clk_init(struct device *dev, struct regmap *regmap) struct da8xx_usb0_clk48 *usb0; struct da8xx_usb1_clk48 *usb1; - clk_data = devm_kzalloc(dev, sizeof(*clk_data) + 2 * - sizeof(*clk_data->hws), GFP_KERNEL); + clk_data = devm_kzalloc(dev, struct_size(clk_data, hws, 2), + GFP_KERNEL); if (!clk_data) return -ENOMEM; diff --git a/drivers/clk/davinci/pll-da830.c b/drivers/clk/davinci/pll-da830.c index 929a3d3a9adb..0a0d06fb25fd 100644 --- a/drivers/clk/davinci/pll-da830.c +++ b/drivers/clk/davinci/pll-da830.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -36,11 +37,11 @@ SYSCLK(5, pll0_sysclk5, pll0_pllen, 5, 0); SYSCLK(6, pll0_sysclk6, pll0_pllen, 5, SYSCLK_FIXED_DIV); SYSCLK(7, pll0_sysclk7, pll0_pllen, 5, 0); -int da830_pll_init(struct device *dev, void __iomem *base) +int da830_pll_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { struct clk *clk; - davinci_pll_clk_register(dev, &da830_pll_info, "ref_clk", base); + davinci_pll_clk_register(dev, &da830_pll_info, "ref_clk", base, cfgchip); clk = davinci_pll_sysclk_register(dev, &pll0_sysclk2, base); clk_register_clkdev(clk, "pll0_sysclk2", "da830-psc0"); diff --git a/drivers/clk/davinci/pll-da850.c b/drivers/clk/davinci/pll-da850.c index 2a038b7908cc..0f7198191ea2 100644 --- a/drivers/clk/davinci/pll-da850.c +++ b/drivers/clk/davinci/pll-da850.c @@ -7,10 +7,14 @@ #include #include +#include #include +#include #include #include #include +#include +#include #include #include @@ -81,11 +85,11 @@ static const struct davinci_pll_obsclk_info da850_pll0_obsclk_info = { .ocsrc_mask = GENMASK(4, 0), }; -int da850_pll0_init(struct device *dev, void __iomem *base) +int da850_pll0_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { struct clk *clk; - davinci_pll_clk_register(dev, &da850_pll0_info, "ref_clk", base); + davinci_pll_clk_register(dev, &da850_pll0_info, "ref_clk", base, cfgchip); clk = davinci_pll_sysclk_register(dev, &pll0_sysclk1, base); clk_register_clkdev(clk, "pll0_sysclk1", "da850-psc0"); @@ -134,11 +138,22 @@ static const struct davinci_pll_sysclk_info *da850_pll0_sysclk_info[] = { NULL }; -int of_da850_pll0_init(struct device *dev, void __iomem *base) +void of_da850_pll0_init(struct device_node *node) { - return of_davinci_pll_init(dev, &da850_pll0_info, - &da850_pll0_obsclk_info, - da850_pll0_sysclk_info, 7, base); + void __iomem *base; + struct regmap *cfgchip; + + base = of_iomap(node, 0); + if (!base) { + pr_err("%s: ioremap failed\n", __func__); + return; + } + + cfgchip = syscon_regmap_lookup_by_compatible("ti,da830-cfgchip"); + + of_davinci_pll_init(NULL, node, &da850_pll0_info, + &da850_pll0_obsclk_info, + da850_pll0_sysclk_info, 7, base, cfgchip); } static const struct davinci_pll_clk_info da850_pll1_info = { @@ -179,11 +194,11 @@ static const struct davinci_pll_obsclk_info da850_pll1_obsclk_info = { .ocsrc_mask = GENMASK(4, 0), }; -int da850_pll1_init(struct device *dev, void __iomem *base) +int da850_pll1_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { struct clk *clk; - davinci_pll_clk_register(dev, &da850_pll1_info, "oscin", base); + davinci_pll_clk_register(dev, &da850_pll1_info, "oscin", base, cfgchip); davinci_pll_sysclk_register(dev, &pll1_sysclk1, base); @@ -204,9 +219,9 @@ static const struct davinci_pll_sysclk_info *da850_pll1_sysclk_info[] = { NULL }; -int of_da850_pll1_init(struct device *dev, void __iomem *base) +int of_da850_pll1_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { - return of_davinci_pll_init(dev, &da850_pll1_info, + return of_davinci_pll_init(dev, dev->of_node, &da850_pll1_info, &da850_pll1_obsclk_info, - da850_pll1_sysclk_info, 3, base); + da850_pll1_sysclk_info, 3, base, cfgchip); } diff --git a/drivers/clk/davinci/pll-dm355.c b/drivers/clk/davinci/pll-dm355.c index 5345f8286c50..505aed80be9a 100644 --- a/drivers/clk/davinci/pll-dm355.c +++ b/drivers/clk/davinci/pll-dm355.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -22,16 +23,16 @@ static const struct davinci_pll_clk_info dm355_pll1_info = { PLL_POSTDIV_ALWAYS_ENABLED | PLL_POSTDIV_FIXED_DIV, }; -SYSCLK(1, pll1_sysclk1, pll1, 5, SYSCLK_FIXED_DIV | SYSCLK_ALWAYS_ENABLED); -SYSCLK(2, pll1_sysclk2, pll1, 5, SYSCLK_FIXED_DIV | SYSCLK_ALWAYS_ENABLED); -SYSCLK(3, pll1_sysclk3, pll1, 5, SYSCLK_ALWAYS_ENABLED); -SYSCLK(4, pll1_sysclk4, pll1, 5, SYSCLK_ALWAYS_ENABLED); +SYSCLK(1, pll1_sysclk1, pll1_pllen, 5, SYSCLK_FIXED_DIV | SYSCLK_ALWAYS_ENABLED); +SYSCLK(2, pll1_sysclk2, pll1_pllen, 5, SYSCLK_FIXED_DIV | SYSCLK_ALWAYS_ENABLED); +SYSCLK(3, pll1_sysclk3, pll1_pllen, 5, SYSCLK_ALWAYS_ENABLED); +SYSCLK(4, pll1_sysclk4, pll1_pllen, 5, SYSCLK_ALWAYS_ENABLED); -int dm355_pll1_init(struct device *dev, void __iomem *base) +int dm355_pll1_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { struct clk *clk; - davinci_pll_clk_register(dev, &dm355_pll1_info, "ref_clk", base); + davinci_pll_clk_register(dev, &dm355_pll1_info, "ref_clk", base, cfgchip); clk = davinci_pll_sysclk_register(dev, &pll1_sysclk1, base); clk_register_clkdev(clk, "pll1_sysclk1", "dm355-psc"); @@ -62,17 +63,14 @@ static const struct davinci_pll_clk_info dm355_pll2_info = { PLL_POSTDIV_ALWAYS_ENABLED | PLL_POSTDIV_FIXED_DIV, }; -SYSCLK(1, pll2_sysclk1, pll2, 5, SYSCLK_FIXED_DIV); -SYSCLK(2, pll2_sysclk2, pll2, 5, SYSCLK_FIXED_DIV | SYSCLK_ALWAYS_ENABLED); +SYSCLK(1, pll2_sysclk1, pll2_pllen, 5, SYSCLK_FIXED_DIV | SYSCLK_ALWAYS_ENABLED); -int dm355_pll2_init(struct device *dev, void __iomem *base) +int dm355_pll2_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { - davinci_pll_clk_register(dev, &dm355_pll2_info, "oscin", base); + davinci_pll_clk_register(dev, &dm355_pll2_info, "oscin", base, cfgchip); davinci_pll_sysclk_register(dev, &pll2_sysclk1, base); - davinci_pll_sysclk_register(dev, &pll2_sysclk2, base); - davinci_pll_sysclkbp_clk_register(dev, "pll2_sysclkbp", base); return 0; diff --git a/drivers/clk/davinci/pll-dm365.c b/drivers/clk/davinci/pll-dm365.c index 5f8d9f42d0f3..2d29712753a3 100644 --- a/drivers/clk/davinci/pll-dm365.c +++ b/drivers/clk/davinci/pll-dm365.c @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -56,11 +57,11 @@ static const struct davinci_pll_obsclk_info dm365_pll1_obsclk_info = { .ocsrc_mask = BIT(4), }; -int dm365_pll1_init(struct device *dev, void __iomem *base) +int dm365_pll1_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { struct clk *clk; - davinci_pll_clk_register(dev, &dm365_pll1_info, "ref_clk", base); + davinci_pll_clk_register(dev, &dm365_pll1_info, "ref_clk", base, cfgchip); clk = davinci_pll_sysclk_register(dev, &pll1_sysclk1, base); clk_register_clkdev(clk, "pll1_sysclk1", "dm365-psc"); @@ -119,11 +120,11 @@ static const struct davinci_pll_obsclk_info dm365_pll2_obsclk_info = { .ocsrc_mask = BIT(4), }; -int dm365_pll2_init(struct device *dev, void __iomem *base) +int dm365_pll2_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { struct clk *clk; - davinci_pll_clk_register(dev, &dm365_pll2_info, "oscin", base); + davinci_pll_clk_register(dev, &dm365_pll2_info, "oscin", base, cfgchip); davinci_pll_sysclk_register(dev, &pll2_sysclk1, base); diff --git a/drivers/clk/davinci/pll-dm644x.c b/drivers/clk/davinci/pll-dm644x.c index 69bf785377cf..7650fadfaac8 100644 --- a/drivers/clk/davinci/pll-dm644x.c +++ b/drivers/clk/davinci/pll-dm644x.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -27,11 +28,11 @@ SYSCLK(2, pll1_sysclk2, pll1_pllen, 4, SYSCLK_FIXED_DIV); SYSCLK(3, pll1_sysclk3, pll1_pllen, 4, SYSCLK_FIXED_DIV); SYSCLK(5, pll1_sysclk5, pll1_pllen, 4, SYSCLK_FIXED_DIV); -int dm644x_pll1_init(struct device *dev, void __iomem *base) +int dm644x_pll1_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { struct clk *clk; - davinci_pll_clk_register(dev, &dm644x_pll1_info, "ref_clk", base); + davinci_pll_clk_register(dev, &dm644x_pll1_info, "ref_clk", base, cfgchip); clk = davinci_pll_sysclk_register(dev, &pll1_sysclk1, base); clk_register_clkdev(clk, "pll1_sysclk1", "dm644x-psc"); @@ -66,9 +67,9 @@ static const struct davinci_pll_clk_info dm644x_pll2_info = { SYSCLK(1, pll2_sysclk1, pll2_pllen, 4, 0); SYSCLK(2, pll2_sysclk2, pll2_pllen, 4, 0); -int dm644x_pll2_init(struct device *dev, void __iomem *base) +int dm644x_pll2_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { - davinci_pll_clk_register(dev, &dm644x_pll2_info, "oscin", base); + davinci_pll_clk_register(dev, &dm644x_pll2_info, "oscin", base, cfgchip); davinci_pll_sysclk_register(dev, &pll2_sysclk1, base); diff --git a/drivers/clk/davinci/pll-dm646x.c b/drivers/clk/davinci/pll-dm646x.c index a61cc3256418..26982970df0e 100644 --- a/drivers/clk/davinci/pll-dm646x.c +++ b/drivers/clk/davinci/pll-dm646x.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -29,11 +30,11 @@ SYSCLK(6, pll1_sysclk6, pll1_pllen, 4, 0); SYSCLK(8, pll1_sysclk8, pll1_pllen, 4, 0); SYSCLK(9, pll1_sysclk9, pll1_pllen, 4, 0); -int dm646x_pll1_init(struct device *dev, void __iomem *base) +int dm646x_pll1_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { struct clk *clk; - davinci_pll_clk_register(dev, &dm646x_pll1_info, "ref_clk", base); + davinci_pll_clk_register(dev, &dm646x_pll1_info, "ref_clk", base, cfgchip); clk = davinci_pll_sysclk_register(dev, &pll1_sysclk1, base); clk_register_clkdev(clk, "pll1_sysclk1", "dm646x-psc"); @@ -72,11 +73,11 @@ static const struct davinci_pll_clk_info dm646x_pll2_info = { .flags = 0, }; -SYSCLK(1, pll2_sysclk1, pll2_pllen, 4, 0); +SYSCLK(1, pll2_sysclk1, pll2_pllen, 4, SYSCLK_ALWAYS_ENABLED); -int dm646x_pll2_init(struct device *dev, void __iomem *base) +int dm646x_pll2_init(struct device *dev, void __iomem *base, struct regmap *cfgchip) { - davinci_pll_clk_register(dev, &dm646x_pll2_info, "oscin", base); + davinci_pll_clk_register(dev, &dm646x_pll2_info, "oscin", base, cfgchip); davinci_pll_sysclk_register(dev, &pll2_sysclk1, base); diff --git a/drivers/clk/davinci/pll.c b/drivers/clk/davinci/pll.c index 23a24c944f1d..1c99e992d638 100644 --- a/drivers/clk/davinci/pll.c +++ b/drivers/clk/davinci/pll.c @@ -11,6 +11,7 @@ #include #include +#include #include #include #include @@ -190,7 +191,7 @@ static int davinci_pll_set_rate(struct clk_hw *hw, unsigned long rate, } #ifdef CONFIG_DEBUG_FS -static int davinci_pll_debug_init(struct clk_hw *hw, struct dentry *dentry); +static void davinci_pll_debug_init(struct clk_hw *hw, struct dentry *dentry); #else #define davinci_pll_debug_init NULL #endif @@ -223,6 +224,7 @@ static const struct clk_ops dm365_pll_ops = { /** * davinci_pll_div_register - common *DIV clock implementation + * @dev: The PLL platform device or NULL * @name: the clock name * @parent_name: the parent clock name * @reg: the *DIV register @@ -240,17 +242,21 @@ static struct clk *davinci_pll_div_register(struct device *dev, const struct clk_ops *divider_ops = &clk_divider_ops; struct clk_gate *gate; struct clk_divider *divider; + struct clk *clk; + int ret; - gate = devm_kzalloc(dev, sizeof(*gate), GFP_KERNEL); + gate = kzalloc(sizeof(*gate), GFP_KERNEL); if (!gate) return ERR_PTR(-ENOMEM); gate->reg = reg; gate->bit_idx = DIV_ENABLE_SHIFT; - divider = devm_kzalloc(dev, sizeof(*divider), GFP_KERNEL); - if (!divider) - return ERR_PTR(-ENOMEM); + divider = kzalloc(sizeof(*divider), GFP_KERNEL); + if (!divider) { + ret = -ENOMEM; + goto err_free_gate; + } divider->reg = reg; divider->shift = DIV_RATIO_SHIFT; @@ -261,9 +267,22 @@ static struct clk *davinci_pll_div_register(struct device *dev, divider_ops = &clk_divider_ro_ops; } - return clk_register_composite(dev, name, parent_names, num_parents, - NULL, NULL, ÷r->hw, divider_ops, - &gate->hw, &clk_gate_ops, flags); + clk = clk_register_composite(dev, name, parent_names, num_parents, + NULL, NULL, ÷r->hw, divider_ops, + &gate->hw, &clk_gate_ops, flags); + if (IS_ERR(clk)) { + ret = PTR_ERR(clk); + goto err_free_divider; + } + + return clk; + +err_free_divider: + kfree(divider); +err_free_gate: + kfree(gate); + + return ERR_PTR(ret); } struct davinci_pllen_clk { @@ -321,36 +340,17 @@ static int davinci_pllen_rate_change(struct notifier_block *nb, return NOTIFY_OK; } -static struct davinci_pll_platform_data *davinci_pll_get_pdata(struct device *dev) -{ - struct davinci_pll_platform_data *pdata = dev_get_platdata(dev); - - /* - * Platform data is optional, so allocate a new struct if one was not - * provided. For device tree, this will always be the case. - */ - if (!pdata) - pdata = devm_kzalloc(dev, sizeof(*pdata), GFP_KERNEL); - if (!pdata) - return NULL; - - /* for device tree, we need to fill in the struct */ - if (dev->of_node) - pdata->cfgchip = - syscon_regmap_lookup_by_compatible("ti,da830-cfgchip"); - - return pdata; -} - static struct notifier_block davinci_pllen_notifier = { .notifier_call = davinci_pllen_rate_change, }; /** * davinci_pll_clk_register - Register a PLL clock + * @dev: The PLL platform device or NULL * @info: The device-specific clock info * @parent_name: The parent clock name * @base: The PLL's memory region + * @cfgchip: CFGCHIP syscon regmap for info->unlock_reg or NULL * * This creates a series of clocks that represent the PLL. * @@ -366,9 +366,9 @@ static struct notifier_block davinci_pllen_notifier = { struct clk *davinci_pll_clk_register(struct device *dev, const struct davinci_pll_clk_info *info, const char *parent_name, - void __iomem *base) + void __iomem *base, + struct regmap *cfgchip) { - struct davinci_pll_platform_data *pdata; char prediv_name[MAX_NAME_SIZE]; char pllout_name[MAX_NAME_SIZE]; char postdiv_name[MAX_NAME_SIZE]; @@ -376,11 +376,12 @@ struct clk *davinci_pll_clk_register(struct device *dev, struct clk_init_data init; struct davinci_pll_clk *pllout; struct davinci_pllen_clk *pllen; - struct clk *pllout_clk, *clk; - - pdata = davinci_pll_get_pdata(dev); - if (!pdata) - return ERR_PTR(-ENOMEM); + struct clk *oscin_clk = NULL; + struct clk *prediv_clk = NULL; + struct clk *pllout_clk; + struct clk *postdiv_clk = NULL; + struct clk *pllen_clk; + int ret; if (info->flags & PLL_HAS_CLKMODE) { /* @@ -392,10 +393,10 @@ struct clk *davinci_pll_clk_register(struct device *dev, * a number of different things. In this driver we use it to * mean the signal after the PLLCTL[CLKMODE] switch. */ - clk = clk_register_fixed_factor(dev, OSCIN_CLK_NAME, - parent_name, 0, 1, 1); - if (IS_ERR(clk)) - return clk; + oscin_clk = clk_register_fixed_factor(dev, OSCIN_CLK_NAME, + parent_name, 0, 1, 1); + if (IS_ERR(oscin_clk)) + return oscin_clk; parent_name = OSCIN_CLK_NAME; } @@ -411,30 +412,34 @@ struct clk *davinci_pll_clk_register(struct device *dev, /* Some? DM355 chips don't correctly report the PREDIV value */ if (info->flags & PLL_PREDIV_FIXED8) - clk = clk_register_fixed_factor(dev, prediv_name, - parent_name, flags, 1, 8); + prediv_clk = clk_register_fixed_factor(dev, prediv_name, + parent_name, flags, 1, 8); else - clk = davinci_pll_div_register(dev, prediv_name, + prediv_clk = davinci_pll_div_register(dev, prediv_name, parent_name, base + PREDIV, fixed, flags); - if (IS_ERR(clk)) - return clk; + if (IS_ERR(prediv_clk)) { + ret = PTR_ERR(prediv_clk); + goto err_unregister_oscin; + } parent_name = prediv_name; } /* Unlock writing to PLL registers */ if (info->unlock_reg) { - if (IS_ERR_OR_NULL(pdata->cfgchip)) + if (IS_ERR_OR_NULL(cfgchip)) dev_warn(dev, "Failed to get CFGCHIP (%ld)\n", - PTR_ERR(pdata->cfgchip)); + PTR_ERR(cfgchip)); else - regmap_write_bits(pdata->cfgchip, info->unlock_reg, + regmap_write_bits(cfgchip, info->unlock_reg, info->unlock_mask, 0); } - pllout = devm_kzalloc(dev, sizeof(*pllout), GFP_KERNEL); - if (!pllout) - return ERR_PTR(-ENOMEM); + pllout = kzalloc(sizeof(*pllout), GFP_KERNEL); + if (!pllout) { + ret = -ENOMEM; + goto err_unregister_prediv; + } snprintf(pllout_name, MAX_NAME_SIZE, "%s_pllout", info->name); @@ -456,9 +461,11 @@ struct clk *davinci_pll_clk_register(struct device *dev, pllout->pllm_min = info->pllm_min; pllout->pllm_max = info->pllm_max; - pllout_clk = devm_clk_register(dev, &pllout->hw); - if (IS_ERR(pllout_clk)) - return pllout_clk; + pllout_clk = clk_register(dev, &pllout->hw); + if (IS_ERR(pllout_clk)) { + ret = PTR_ERR(pllout_clk); + goto err_free_pllout; + } clk_hw_set_rate_range(&pllout->hw, info->pllout_min_rate, info->pllout_max_rate); @@ -474,17 +481,21 @@ struct clk *davinci_pll_clk_register(struct device *dev, if (info->flags & PLL_POSTDIV_ALWAYS_ENABLED) flags |= CLK_IS_CRITICAL; - clk = davinci_pll_div_register(dev, postdiv_name, parent_name, - base + POSTDIV, fixed, flags); - if (IS_ERR(clk)) - return clk; + postdiv_clk = davinci_pll_div_register(dev, postdiv_name, + parent_name, base + POSTDIV, fixed, flags); + if (IS_ERR(postdiv_clk)) { + ret = PTR_ERR(postdiv_clk); + goto err_unregister_pllout; + } parent_name = postdiv_name; } - pllen = devm_kzalloc(dev, sizeof(*pllout), GFP_KERNEL); - if (!pllen) - return ERR_PTR(-ENOMEM); + pllen = kzalloc(sizeof(*pllout), GFP_KERNEL); + if (!pllen) { + ret = -ENOMEM; + goto err_unregister_postdiv; + } snprintf(pllen_name, MAX_NAME_SIZE, "%s_pllen", info->name); @@ -497,17 +508,35 @@ struct clk *davinci_pll_clk_register(struct device *dev, pllen->hw.init = &init; pllen->base = base; - clk = devm_clk_register(dev, &pllen->hw); - if (IS_ERR(clk)) - return clk; + pllen_clk = clk_register(dev, &pllen->hw); + if (IS_ERR(pllen_clk)) { + ret = PTR_ERR(pllen_clk); + goto err_free_pllen; + } - clk_notifier_register(clk, &davinci_pllen_notifier); + clk_notifier_register(pllen_clk, &davinci_pllen_notifier); return pllout_clk; + +err_free_pllen: + kfree(pllen); +err_unregister_postdiv: + clk_unregister(postdiv_clk); +err_unregister_pllout: + clk_unregister(pllout_clk); +err_free_pllout: + kfree(pllout); +err_unregister_prediv: + clk_unregister(prediv_clk); +err_unregister_oscin: + clk_unregister(oscin_clk); + + return ERR_PTR(ret); } /** * davinci_pll_auxclk_register - Register bypass clock (AUXCLK) + * @dev: The PLL platform device or NULL * @name: The clock name * @base: The PLL memory region */ @@ -521,6 +550,7 @@ struct clk *davinci_pll_auxclk_register(struct device *dev, /** * davinci_pll_sysclkbp_clk_register - Register bypass divider clock (SYSCLKBP) + * @dev: The PLL platform device or NULL * @name: The clock name * @base: The PLL memory region */ @@ -535,6 +565,7 @@ struct clk *davinci_pll_sysclkbp_clk_register(struct device *dev, /** * davinci_pll_obsclk_register - Register oscillator divider clock (OBSCLK) + * @dev: The PLL platform device or NULL * @info: The clock info * @base: The PLL memory region */ @@ -546,9 +577,11 @@ davinci_pll_obsclk_register(struct device *dev, struct clk_mux *mux; struct clk_gate *gate; struct clk_divider *divider; + struct clk *clk; u32 oscdiv; + int ret; - mux = devm_kzalloc(dev, sizeof(*mux), GFP_KERNEL); + mux = kzalloc(sizeof(*mux), GFP_KERNEL); if (!mux) return ERR_PTR(-ENOMEM); @@ -556,16 +589,20 @@ davinci_pll_obsclk_register(struct device *dev, mux->table = info->table; mux->mask = info->ocsrc_mask; - gate = devm_kzalloc(dev, sizeof(*gate), GFP_KERNEL); - if (!gate) - return ERR_PTR(-ENOMEM); + gate = kzalloc(sizeof(*gate), GFP_KERNEL); + if (!gate) { + ret = -ENOMEM; + goto err_free_mux; + } gate->reg = base + CKEN; gate->bit_idx = CKEN_OBSCLK_SHIFT; - divider = devm_kzalloc(dev, sizeof(*divider), GFP_KERNEL); - if (!divider) - return ERR_PTR(-ENOMEM); + divider = kzalloc(sizeof(*divider), GFP_KERNEL); + if (!divider) { + ret = -ENOMEM; + goto err_free_gate; + } divider->reg = base + OSCDIV; divider->shift = DIV_RATIO_SHIFT; @@ -576,11 +613,27 @@ davinci_pll_obsclk_register(struct device *dev, oscdiv |= BIT(DIV_ENABLE_SHIFT); writel(oscdiv, base + OSCDIV); - return clk_register_composite(dev, info->name, info->parent_names, - info->num_parents, - &mux->hw, &clk_mux_ops, - ÷r->hw, &clk_divider_ops, - &gate->hw, &clk_gate_ops, 0); + clk = clk_register_composite(dev, info->name, info->parent_names, + info->num_parents, + &mux->hw, &clk_mux_ops, + ÷r->hw, &clk_divider_ops, + &gate->hw, &clk_gate_ops, 0); + + if (IS_ERR(clk)) { + ret = PTR_ERR(clk); + goto err_free_divider; + } + + return clk; + +err_free_divider: + kfree(divider); +err_free_gate: + kfree(gate); +err_free_mux: + kfree(mux); + + return ERR_PTR(ret); } /* The PLL SYSCLKn clocks have a mechanism for synchronizing rate changes. */ @@ -616,6 +669,7 @@ static struct notifier_block davinci_pll_sysclk_notifier = { /** * davinci_pll_sysclk_register - Register divider clocks (SYSCLKn) + * @dev: The PLL platform device or NULL * @info: The clock info * @base: The PLL memory region */ @@ -630,6 +684,7 @@ davinci_pll_sysclk_register(struct device *dev, struct clk *clk; u32 reg; u32 flags = 0; + int ret; /* PLLDIVn registers are not entirely consecutive */ if (info->id < 4) @@ -637,16 +692,18 @@ davinci_pll_sysclk_register(struct device *dev, else reg = PLLDIV4 + 4 * (info->id - 4); - gate = devm_kzalloc(dev, sizeof(*gate), GFP_KERNEL); + gate = kzalloc(sizeof(*gate), GFP_KERNEL); if (!gate) return ERR_PTR(-ENOMEM); gate->reg = base + reg; gate->bit_idx = DIV_ENABLE_SHIFT; - divider = devm_kzalloc(dev, sizeof(*divider), GFP_KERNEL); - if (!divider) - return ERR_PTR(-ENOMEM); + divider = kzalloc(sizeof(*divider), GFP_KERNEL); + if (!divider) { + ret = -ENOMEM; + goto err_free_gate; + } divider->reg = base + reg; divider->shift = DIV_RATIO_SHIFT; @@ -668,22 +725,31 @@ davinci_pll_sysclk_register(struct device *dev, clk = clk_register_composite(dev, info->name, &info->parent_name, 1, NULL, NULL, ÷r->hw, divider_ops, &gate->hw, &clk_gate_ops, flags); - if (IS_ERR(clk)) - return clk; + if (IS_ERR(clk)) { + ret = PTR_ERR(clk); + goto err_free_divider; + } clk_notifier_register(clk, &davinci_pll_sysclk_notifier); return clk; + +err_free_divider: + kfree(divider); +err_free_gate: + kfree(gate); + + return ERR_PTR(ret); } -int of_davinci_pll_init(struct device *dev, +int of_davinci_pll_init(struct device *dev, struct device_node *node, const struct davinci_pll_clk_info *info, const struct davinci_pll_obsclk_info *obsclk_info, const struct davinci_pll_sysclk_info **div_info, u8 max_sysclk_id, - void __iomem *base) + void __iomem *base, + struct regmap *cfgchip) { - struct device_node *node = dev->of_node; struct device_node *child; const char *parent_name; struct clk *clk; @@ -693,7 +759,7 @@ int of_davinci_pll_init(struct device *dev, else parent_name = OSCIN_CLK_NAME; - clk = davinci_pll_clk_register(dev, info, parent_name, base); + clk = davinci_pll_clk_register(dev, info, parent_name, base, cfgchip); if (IS_ERR(clk)) { dev_err(dev, "failed to register %s\n", info->name); return PTR_ERR(clk); @@ -711,13 +777,15 @@ int of_davinci_pll_init(struct device *dev, int n_clks = max_sysclk_id + 1; int i; - clk_data = devm_kzalloc(dev, sizeof(*clk_data), GFP_KERNEL); + clk_data = kzalloc(sizeof(*clk_data), GFP_KERNEL); if (!clk_data) return -ENOMEM; - clks = devm_kmalloc_array(dev, n_clks, sizeof(*clks), GFP_KERNEL); - if (!clks) + clks = kmalloc_array(n_clks, sizeof(*clks), GFP_KERNEL); + if (!clks) { + kfree(clk_data); return -ENOMEM; + } clk_data->clks = clks; clk_data->clk_num = n_clks; @@ -770,32 +838,73 @@ int of_davinci_pll_init(struct device *dev, return 0; } +static struct davinci_pll_platform_data *davinci_pll_get_pdata(struct device *dev) +{ + struct davinci_pll_platform_data *pdata = dev_get_platdata(dev); + + /* + * Platform data is optional, so allocate a new struct if one was not + * provided. For device tree, this will always be the case. + */ + if (!pdata) + pdata = devm_kzalloc(dev, sizeof(*pdata), GFP_KERNEL); + if (!pdata) + return NULL; + + /* for device tree, we need to fill in the struct */ + if (dev->of_node) + pdata->cfgchip = + syscon_regmap_lookup_by_compatible("ti,da830-cfgchip"); + + return pdata; +} + +/* needed in early boot for clocksource/clockevent */ +#ifdef CONFIG_ARCH_DAVINCI_DA850 +CLK_OF_DECLARE(da850_pll0, "ti,da850-pll0", of_da850_pll0_init); +#endif + static const struct of_device_id davinci_pll_of_match[] = { - { .compatible = "ti,da850-pll0", .data = of_da850_pll0_init }, +#ifdef CONFIG_ARCH_DAVINCI_DA850 { .compatible = "ti,da850-pll1", .data = of_da850_pll1_init }, +#endif { } }; static const struct platform_device_id davinci_pll_id_table[] = { +#ifdef CONFIG_ARCH_DAVINCI_DA830 { .name = "da830-pll", .driver_data = (kernel_ulong_t)da830_pll_init }, +#endif +#ifdef CONFIG_ARCH_DAVINCI_DA850 { .name = "da850-pll0", .driver_data = (kernel_ulong_t)da850_pll0_init }, { .name = "da850-pll1", .driver_data = (kernel_ulong_t)da850_pll1_init }, +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM355 { .name = "dm355-pll1", .driver_data = (kernel_ulong_t)dm355_pll1_init }, { .name = "dm355-pll2", .driver_data = (kernel_ulong_t)dm355_pll2_init }, +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM365 { .name = "dm365-pll1", .driver_data = (kernel_ulong_t)dm365_pll1_init }, { .name = "dm365-pll2", .driver_data = (kernel_ulong_t)dm365_pll2_init }, +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM644x { .name = "dm644x-pll1", .driver_data = (kernel_ulong_t)dm644x_pll1_init }, { .name = "dm644x-pll2", .driver_data = (kernel_ulong_t)dm644x_pll2_init }, +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM646x { .name = "dm646x-pll1", .driver_data = (kernel_ulong_t)dm646x_pll1_init }, { .name = "dm646x-pll2", .driver_data = (kernel_ulong_t)dm646x_pll2_init }, +#endif { } }; -typedef int (*davinci_pll_init)(struct device *dev, void __iomem *base); +typedef int (*davinci_pll_init)(struct device *dev, void __iomem *base, + struct regmap *cfgchip); static int davinci_pll_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; + struct davinci_pll_platform_data *pdata; const struct of_device_id *of_id; davinci_pll_init pll_init = NULL; struct resource *res; @@ -812,12 +921,18 @@ static int davinci_pll_probe(struct platform_device *pdev) return -EINVAL; } + pdata = davinci_pll_get_pdata(dev); + if (!pdata) { + dev_err(dev, "missing platform data\n"); + return -EINVAL; + } + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); base = devm_ioremap_resource(dev, res); if (IS_ERR(base)) return PTR_ERR(base); - return pll_init(dev, base); + return pll_init(dev, base, pdata->cfgchip); } static struct platform_driver davinci_pll_driver = { @@ -874,26 +989,19 @@ static const struct debugfs_reg32 davinci_pll_regs[] = { DEBUG_REG(PLLDIV9), }; -static int davinci_pll_debug_init(struct clk_hw *hw, struct dentry *dentry) +static void davinci_pll_debug_init(struct clk_hw *hw, struct dentry *dentry) { struct davinci_pll_clk *pll = to_davinci_pll_clk(hw); struct debugfs_regset32 *regset; - struct dentry *d; regset = kzalloc(sizeof(*regset), GFP_KERNEL); if (!regset) - return -ENOMEM; + return; regset->regs = davinci_pll_regs; regset->nregs = ARRAY_SIZE(davinci_pll_regs); regset->base = pll->base; - d = debugfs_create_regset32("registers", 0400, dentry, regset); - if (IS_ERR(d)) { - kfree(regset); - return PTR_ERR(d); - } - - return 0; + debugfs_create_regset32("registers", 0400, dentry, regset); } #endif diff --git a/drivers/clk/davinci/pll.h b/drivers/clk/davinci/pll.h index b1b6fb23f972..7cc354dd29e2 100644 --- a/drivers/clk/davinci/pll.h +++ b/drivers/clk/davinci/pll.h @@ -11,6 +11,7 @@ #include #include #include +#include #include #define PLL_HAS_CLKMODE BIT(0) /* PLL has PLLCTL[CLKMODE] */ @@ -94,7 +95,8 @@ struct davinci_pll_obsclk_info { struct clk *davinci_pll_clk_register(struct device *dev, const struct davinci_pll_clk_info *info, const char *parent_name, - void __iomem *base); + void __iomem *base, + struct regmap *cfgchip); struct clk *davinci_pll_auxclk_register(struct device *dev, const char *name, void __iomem *base); @@ -110,32 +112,29 @@ davinci_pll_sysclk_register(struct device *dev, const struct davinci_pll_sysclk_info *info, void __iomem *base); -int of_davinci_pll_init(struct device *dev, +int of_davinci_pll_init(struct device *dev, struct device_node *node, const struct davinci_pll_clk_info *info, const struct davinci_pll_obsclk_info *obsclk_info, const struct davinci_pll_sysclk_info **div_info, u8 max_sysclk_id, - void __iomem *base); + void __iomem *base, + struct regmap *cfgchip); /* Platform-specific callbacks */ -int da830_pll_init(struct device *dev, void __iomem *base); - -int da850_pll0_init(struct device *dev, void __iomem *base); -int da850_pll1_init(struct device *dev, void __iomem *base); -int of_da850_pll0_init(struct device *dev, void __iomem *base); -int of_da850_pll1_init(struct device *dev, void __iomem *base); - -int dm355_pll1_init(struct device *dev, void __iomem *base); -int dm355_pll2_init(struct device *dev, void __iomem *base); - -int dm365_pll1_init(struct device *dev, void __iomem *base); -int dm365_pll2_init(struct device *dev, void __iomem *base); - -int dm644x_pll1_init(struct device *dev, void __iomem *base); -int dm644x_pll2_init(struct device *dev, void __iomem *base); - -int dm646x_pll1_init(struct device *dev, void __iomem *base); -int dm646x_pll2_init(struct device *dev, void __iomem *base); +#ifdef CONFIG_ARCH_DAVINCI_DA850 +int da850_pll1_init(struct device *dev, void __iomem *base, struct regmap *cfgchip); +void of_da850_pll0_init(struct device_node *node); +int of_da850_pll1_init(struct device *dev, void __iomem *base, struct regmap *cfgchip); +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM355 +int dm355_pll2_init(struct device *dev, void __iomem *base, struct regmap *cfgchip); +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM644x +int dm644x_pll2_init(struct device *dev, void __iomem *base, struct regmap *cfgchip); +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM646x +int dm646x_pll2_init(struct device *dev, void __iomem *base, struct regmap *cfgchip); +#endif #endif /* __CLK_DAVINCI_PLL_H___ */ diff --git a/drivers/clk/davinci/psc-da830.c b/drivers/clk/davinci/psc-da830.c index f61abf5632ff..081b039fcb02 100644 --- a/drivers/clk/davinci/psc-da830.c +++ b/drivers/clk/davinci/psc-da830.c @@ -55,7 +55,8 @@ const struct davinci_psc_init_data da830_psc0_init_data = { .psc_init = &da830_psc0_init, }; -LPSC_CLKDEV2(usb0_clkdev, NULL, "musb-da8xx", +LPSC_CLKDEV3(usb0_clkdev, "fck", "da830-usb-phy-clks", + NULL, "musb-da8xx", NULL, "cppi41-dmaengine"); LPSC_CLKDEV1(usb1_clkdev, NULL, "ohci-da8xx"); /* REVISIT: gpio-davinci.c should be modified to drop con_id */ diff --git a/drivers/clk/davinci/psc-dm355.c b/drivers/clk/davinci/psc-dm355.c index 6995ecea2677..ddd250107c4e 100644 --- a/drivers/clk/davinci/psc-dm355.c +++ b/drivers/clk/davinci/psc-dm355.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -41,14 +42,14 @@ static const struct davinci_lpsc_clk_info dm355_psc_info[] = { LPSC(5, 0, timer3, pll1_auxclk, NULL, 0), LPSC(6, 0, spi1, pll1_sysclk2, spi1_clkdev, 0), LPSC(7, 0, mmcsd1, pll1_sysclk2, mmcsd1_clkdev, 0), - LPSC(8, 0, asp1, pll1_sysclk2, NULL, 0), + LPSC(8, 0, asp1, pll1_sysclk2, mcbsp1_clkdev, 0), LPSC(9, 0, usb, pll1_sysclk2, usb_clkdev, 0), LPSC(10, 0, pwm3, pll1_auxclk, NULL, 0), LPSC(11, 0, spi2, pll1_sysclk2, spi2_clkdev, 0), LPSC(12, 0, rto, pll1_auxclk, NULL, 0), LPSC(14, 0, aemif, pll1_sysclk2, aemif_clkdev, 0), LPSC(15, 0, mmcsd0, pll1_sysclk2, mmcsd0_clkdev, 0), - LPSC(17, 0, asp0, pll1_sysclk2, NULL, 0), + LPSC(17, 0, asp0, pll1_sysclk2, mcbsp0_clkdev, 0), LPSC(18, 0, i2c, pll1_auxclk, i2c_clkdev, 0), LPSC(19, 0, uart0, pll1_auxclk, uart0_clkdev, 0), LPSC(20, 0, uart1, pll1_auxclk, uart1_clkdev, 0), @@ -68,7 +69,7 @@ static const struct davinci_lpsc_clk_info dm355_psc_info[] = { { } }; -static int dm355_psc_init(struct device *dev, void __iomem *base) +int dm355_psc_init(struct device *dev, void __iomem *base) { return davinci_psc_register_clocks(dev, dm355_psc_info, 42, base); } diff --git a/drivers/clk/davinci/psc-dm365.c b/drivers/clk/davinci/psc-dm365.c index 3ad915f37376..8c73086cc676 100644 --- a/drivers/clk/davinci/psc-dm365.c +++ b/drivers/clk/davinci/psc-dm365.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -65,15 +66,28 @@ static const struct davinci_lpsc_clk_info dm365_psc_info[] = { LPSC(31, 0, arm, pll2_sysclk2, NULL, LPSC_ALWAYS_ENABLED), LPSC(38, 0, spi3, pll1_sysclk4, spi3_clkdev, 0), LPSC(39, 0, spi4, pll1_auxclk, spi4_clkdev, 0), - LPSC(40, 0, emac, pll2_sysclk4, emac_clkdev, 0), - LPSC(44, 1, voice_codec, pll1_sysclk3, voice_codec_clkdev, 0), - LPSC(46, 1, vpss_dac, pll1_sysclk3, vpss_dac_clkdev, 0), + LPSC(40, 0, emac, pll1_sysclk4, emac_clkdev, 0), + /* + * The TRM (ARM Subsystem User's Guide) shows two clocks input into + * voice codec module (PLL2 SYSCLK4 with a DIV2 and PLL1 SYSCLK4). Its + * not fully clear from documentation which clock should be considered + * as parent for PSC. The clock chosen here is to maintain + * compatibility with existing code in arch/arm/mach-davinci/dm365.c + */ + LPSC(44, 0, voice_codec, pll2_sysclk4, voice_codec_clkdev, 0), + /* + * Its not fully clear from TRM (ARM Subsystem User's Guide) as to what + * the parent of VPSS DAC LPSC should actually be. PLL1 SYSCLK3 feeds + * into HDVICP and MJCP. The clock chosen here is to remain compatible + * with code existing in arch/arm/mach-davinci/dm365.c + */ + LPSC(46, 0, vpss_dac, pll1_sysclk3, vpss_dac_clkdev, 0), LPSC(47, 0, vpss_master, pll1_sysclk5, vpss_master_clkdev, 0), LPSC(50, 0, mjcp, pll1_sysclk3, NULL, 0), { } }; -static int dm365_psc_init(struct device *dev, void __iomem *base) +int dm365_psc_init(struct device *dev, void __iomem *base) { return davinci_psc_register_clocks(dev, dm365_psc_info, 52, base); } diff --git a/drivers/clk/davinci/psc-dm644x.c b/drivers/clk/davinci/psc-dm644x.c index c22367baa46f..fc0230e3a3d6 100644 --- a/drivers/clk/davinci/psc-dm644x.c +++ b/drivers/clk/davinci/psc-dm644x.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -63,7 +64,7 @@ static const struct davinci_lpsc_clk_info dm644x_psc_info[] = { { } }; -static int dm644x_psc_init(struct device *dev, void __iomem *base) +int dm644x_psc_init(struct device *dev, void __iomem *base) { return davinci_psc_register_clocks(dev, dm644x_psc_info, 41, base); } diff --git a/drivers/clk/davinci/psc-dm646x.c b/drivers/clk/davinci/psc-dm646x.c index 468ef86ea40b..c3f82ed70a80 100644 --- a/drivers/clk/davinci/psc-dm646x.c +++ b/drivers/clk/davinci/psc-dm646x.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -58,7 +59,7 @@ static const struct davinci_lpsc_clk_info dm646x_psc_info[] = { { } }; -static int dm646x_psc_init(struct device *dev, void __iomem *base) +int dm646x_psc_init(struct device *dev, void __iomem *base) { return davinci_psc_register_clocks(dev, dm646x_psc_info, 46, base); } diff --git a/drivers/clk/davinci/psc.c b/drivers/clk/davinci/psc.c index ce170e600f09..fffbed5e263b 100644 --- a/drivers/clk/davinci/psc.c +++ b/drivers/clk/davinci/psc.c @@ -15,6 +15,7 @@ #include #include +#include #include #include #include @@ -63,7 +64,7 @@ struct davinci_psc_data { /** * struct davinci_lpsc_clk - LPSC clock structure - * @dev: the device that provides this LPSC + * @dev: the device that provides this LPSC or NULL * @hw: clk_hw for the LPSC * @pm_domain: power domain for the LPSC * @genpd_clk: clock reference owned by @pm_domain @@ -221,6 +222,7 @@ static void davinci_psc_genpd_detach_dev(struct generic_pm_domain *pm_domain, /** * davinci_lpsc_clk_register - register LPSC clock + * @dev: the clocks's device or NULL * @name: name of this clock * @parent_name: name of clock's parent * @regmap: PSC MMIO region @@ -238,7 +240,7 @@ davinci_lpsc_clk_register(struct device *dev, const char *name, int ret; bool is_on; - lpsc = devm_kzalloc(dev, sizeof(*lpsc), GFP_KERNEL); + lpsc = kzalloc(sizeof(*lpsc), GFP_KERNEL); if (!lpsc) return ERR_PTR(-ENOMEM); @@ -261,9 +263,15 @@ davinci_lpsc_clk_register(struct device *dev, const char *name, lpsc->pd = pd; lpsc->flags = flags; - ret = devm_clk_hw_register(dev, &lpsc->hw); - if (ret < 0) + ret = clk_hw_register(dev, &lpsc->hw); + if (ret < 0) { + kfree(lpsc); return ERR_PTR(ret); + } + + /* for now, genpd is only registered when using device-tree */ + if (!dev || !dev->of_node) + return lpsc; /* genpd attach needs a way to look up this clock */ ret = clk_hw_register_clkdev(&lpsc->hw, name, best_dev_name(dev)); @@ -378,13 +386,15 @@ __davinci_psc_register_clocks(struct device *dev, struct regmap *regmap; int i, ret; - psc = devm_kzalloc(dev, sizeof(*psc), GFP_KERNEL); + psc = kzalloc(sizeof(*psc), GFP_KERNEL); if (!psc) return ERR_PTR(-ENOMEM); - clks = devm_kmalloc_array(dev, num_clks, sizeof(*clks), GFP_KERNEL); - if (!clks) - return ERR_PTR(-ENOMEM); + clks = kmalloc_array(num_clks, sizeof(*clks), GFP_KERNEL); + if (!clks) { + ret = -ENOMEM; + goto err_free_psc; + } psc->clk_data.clks = clks; psc->clk_data.clk_num = num_clks; @@ -396,16 +406,20 @@ __davinci_psc_register_clocks(struct device *dev, for (i = 0; i < num_clks; i++) clks[i] = ERR_PTR(-ENOENT); - pm_domains = devm_kcalloc(dev, num_clks, sizeof(*pm_domains), GFP_KERNEL); - if (!pm_domains) - return ERR_PTR(-ENOMEM); + pm_domains = kcalloc(num_clks, sizeof(*pm_domains), GFP_KERNEL); + if (!pm_domains) { + ret = -ENOMEM; + goto err_free_clks; + } psc->pm_data.domains = pm_domains; psc->pm_data.num_domains = num_clks; - regmap = devm_regmap_init_mmio(dev, base, &davinci_psc_regmap_config); - if (IS_ERR(regmap)) - return ERR_CAST(regmap); + regmap = regmap_init_mmio(dev, base, &davinci_psc_regmap_config); + if (IS_ERR(regmap)) { + ret = PTR_ERR(regmap); + goto err_free_pm_domains; + } for (; info->name; info++) { struct davinci_lpsc_clk *lpsc; @@ -423,6 +437,13 @@ __davinci_psc_register_clocks(struct device *dev, pm_domains[info->md] = &lpsc->pm_domain; } + /* + * for now, a reset controller is only registered when there is a device + * to associate it with. + */ + if (!dev) + return psc; + psc->rcdev.ops = &davinci_psc_reset_ops; psc->rcdev.owner = THIS_MODULE; psc->rcdev.dev = dev; @@ -436,6 +457,15 @@ __davinci_psc_register_clocks(struct device *dev, dev_warn(dev, "Failed to register reset controller (%d)\n", ret); return psc; + +err_free_pm_domains: + kfree(pm_domains); +err_free_clks: + kfree(clks); +err_free_psc: + kfree(psc); + + return ERR_PTR(ret); } int davinci_psc_register_clocks(struct device *dev, @@ -483,20 +513,34 @@ int of_davinci_psc_clk_init(struct device *dev, } static const struct of_device_id davinci_psc_of_match[] = { +#ifdef CONFIG_ARCH_DAVINCI_DA850 { .compatible = "ti,da850-psc0", .data = &of_da850_psc0_init_data }, { .compatible = "ti,da850-psc1", .data = &of_da850_psc1_init_data }, +#endif { } }; static const struct platform_device_id davinci_psc_id_table[] = { +#ifdef CONFIG_ARCH_DAVINCI_DA830 { .name = "da830-psc0", .driver_data = (kernel_ulong_t)&da830_psc0_init_data }, { .name = "da830-psc1", .driver_data = (kernel_ulong_t)&da830_psc1_init_data }, +#endif +#ifdef CONFIG_ARCH_DAVINCI_DA850 { .name = "da850-psc0", .driver_data = (kernel_ulong_t)&da850_psc0_init_data }, { .name = "da850-psc1", .driver_data = (kernel_ulong_t)&da850_psc1_init_data }, +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM355 { .name = "dm355-psc", .driver_data = (kernel_ulong_t)&dm355_psc_init_data }, +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM365 { .name = "dm365-psc", .driver_data = (kernel_ulong_t)&dm365_psc_init_data }, +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM644x { .name = "dm644x-psc", .driver_data = (kernel_ulong_t)&dm644x_psc_init_data }, +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM646x { .name = "dm646x-psc", .driver_data = (kernel_ulong_t)&dm646x_psc_init_data }, +#endif { } }; diff --git a/drivers/clk/davinci/psc.h b/drivers/clk/davinci/psc.h index c2a7df6413fe..6a42529d31a9 100644 --- a/drivers/clk/davinci/psc.h +++ b/drivers/clk/davinci/psc.h @@ -94,15 +94,27 @@ struct davinci_psc_init_data { int (*psc_init)(struct device *dev, void __iomem *base); }; +#ifdef CONFIG_ARCH_DAVINCI_DA830 extern const struct davinci_psc_init_data da830_psc0_init_data; extern const struct davinci_psc_init_data da830_psc1_init_data; +#endif +#ifdef CONFIG_ARCH_DAVINCI_DA850 extern const struct davinci_psc_init_data da850_psc0_init_data; extern const struct davinci_psc_init_data da850_psc1_init_data; extern const struct davinci_psc_init_data of_da850_psc0_init_data; extern const struct davinci_psc_init_data of_da850_psc1_init_data; +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM355 extern const struct davinci_psc_init_data dm355_psc_init_data; +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM356 extern const struct davinci_psc_init_data dm365_psc_init_data; +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM644x extern const struct davinci_psc_init_data dm644x_psc_init_data; +#endif +#ifdef CONFIG_ARCH_DAVINCI_DM646x extern const struct davinci_psc_init_data dm646x_psc_init_data; +#endif #endif /* __CLK_DAVINCI_PSC_H__ */ diff --git a/drivers/clk/hisilicon/Kconfig b/drivers/clk/hisilicon/Kconfig index 1bd43550e4c8..becdb1dd21b5 100644 --- a/drivers/clk/hisilicon/Kconfig +++ b/drivers/clk/hisilicon/Kconfig @@ -44,14 +44,17 @@ config RESET_HISI Build reset controller driver for HiSilicon device chipsets. config STUB_CLK_HI6220 - bool "Hi6220 Stub Clock Driver" - depends on COMMON_CLK_HI6220 && MAILBOX - default ARCH_HISI + bool "Hi6220 Stub Clock Driver" if EXPERT + depends on (COMMON_CLK_HI6220 || COMPILE_TEST) + depends on MAILBOX + default COMMON_CLK_HI6220 help Build the Hisilicon Hi6220 stub clock driver. config STUB_CLK_HI3660 - bool "Hi3660 Stub Clock Driver" - depends on COMMON_CLK_HI3660 && MAILBOX + bool "Hi3660 Stub Clock Driver" if EXPERT + depends on (COMMON_CLK_HI3660 || COMPILE_TEST) + depends on MAILBOX + default COMMON_CLK_HI3660 help Build the Hisilicon Hi3660 stub clock driver. diff --git a/drivers/clk/hisilicon/crg-hi3798cv200.c b/drivers/clk/hisilicon/crg-hi3798cv200.c index 743eec131528..4fe0b2a9baf1 100644 --- a/drivers/clk/hisilicon/crg-hi3798cv200.c +++ b/drivers/clk/hisilicon/crg-hi3798cv200.c @@ -186,6 +186,23 @@ static const struct hisi_gate_clock hi3798cv200_gate_clks[] = { CLK_SET_RATE_PARENT, 0xbc, 0, 0 }, { HISTB_USB2_PHY2_REF_CLK, "clk_u2_phy2_ref", "24m", CLK_SET_RATE_PARENT, 0xbc, 2, 0 }, + /* USB3 */ + { HISTB_USB3_BUS_CLK, "clk_u3_bus", NULL, + CLK_SET_RATE_PARENT, 0xb0, 0, 0 }, + { HISTB_USB3_UTMI_CLK, "clk_u3_utmi", NULL, + CLK_SET_RATE_PARENT, 0xb0, 4, 0 }, + { HISTB_USB3_PIPE_CLK, "clk_u3_pipe", NULL, + CLK_SET_RATE_PARENT, 0xb0, 3, 0 }, + { HISTB_USB3_SUSPEND_CLK, "clk_u3_suspend", NULL, + CLK_SET_RATE_PARENT, 0xb0, 2, 0 }, + { HISTB_USB3_BUS_CLK1, "clk_u3_bus1", NULL, + CLK_SET_RATE_PARENT, 0xb0, 16, 0 }, + { HISTB_USB3_UTMI_CLK1, "clk_u3_utmi1", NULL, + CLK_SET_RATE_PARENT, 0xb0, 20, 0 }, + { HISTB_USB3_PIPE_CLK1, "clk_u3_pipe1", NULL, + CLK_SET_RATE_PARENT, 0xb0, 19, 0 }, + { HISTB_USB3_SUSPEND_CLK1, "clk_u3_suspend1", NULL, + CLK_SET_RATE_PARENT, 0xb0, 18, 0 }, }; static struct hisi_clock_data *hi3798cv200_clk_register( diff --git a/drivers/clk/imx/clk-imx6q.c b/drivers/clk/imx/clk-imx6q.c index 8d518ad5dc13..b9ea7037e193 100644 --- a/drivers/clk/imx/clk-imx6q.c +++ b/drivers/clk/imx/clk-imx6q.c @@ -753,6 +753,8 @@ static void __init imx6q_clocks_init(struct device_node *ccm_node) else clk[IMX6Q_CLK_ECSPI5] = imx_clk_gate2("ecspi5", "ecspi_root", base + 0x6c, 8); clk[IMX6QDL_CLK_ENET] = imx_clk_gate2("enet", "ipg", base + 0x6c, 10); + clk[IMX6QDL_CLK_EPIT1] = imx_clk_gate2("epit1", "ipg", base + 0x6c, 12); + clk[IMX6QDL_CLK_EPIT2] = imx_clk_gate2("epit2", "ipg", base + 0x6c, 14); clk[IMX6QDL_CLK_ESAI_EXTAL] = imx_clk_gate2_shared("esai_extal", "esai_podf", base + 0x6c, 16, &share_count_esai); clk[IMX6QDL_CLK_ESAI_IPG] = imx_clk_gate2_shared("esai_ipg", "ahb", base + 0x6c, 16, &share_count_esai); clk[IMX6QDL_CLK_ESAI_MEM] = imx_clk_gate2_shared("esai_mem", "ahb", base + 0x6c, 16, &share_count_esai); diff --git a/drivers/clk/imx/clk-imx6sl.c b/drivers/clk/imx/clk-imx6sl.c index 9642cdf0fb88..66b1dd1cfad0 100644 --- a/drivers/clk/imx/clk-imx6sl.c +++ b/drivers/clk/imx/clk-imx6sl.c @@ -330,7 +330,7 @@ static void __init imx6sl_clocks_init(struct device_node *ccm_node) clks[IMX6SL_CLK_PERIPH2] = imx_clk_busy_mux("periph2", base + 0x14, 26, 1, base + 0x48, 3, periph2_sels, ARRAY_SIZE(periph2_sels)); /* name parent_name reg shift width */ - clks[IMX6SL_CLK_OCRAM_PODF] = imx_clk_divider("ocram_podf", "ocram_sel", base + 0x14, 16, 3); + clks[IMX6SL_CLK_OCRAM_PODF] = imx_clk_busy_divider("ocram_podf", "ocram_sel", base + 0x14, 16, 3, base + 0x48, 0); clks[IMX6SL_CLK_PERIPH_CLK2_PODF] = imx_clk_divider("periph_clk2_podf", "periph_clk2_sel", base + 0x14, 27, 3); clks[IMX6SL_CLK_PERIPH2_CLK2_PODF] = imx_clk_divider("periph2_clk2_podf", "periph2_clk2_sel", base + 0x14, 0, 3); clks[IMX6SL_CLK_IPG] = imx_clk_divider("ipg", "ahb", base + 0x14, 8, 2); diff --git a/drivers/clk/imx/clk-imx6sx.c b/drivers/clk/imx/clk-imx6sx.c index bc3f9ebf2d9e..10c771b91ef6 100644 --- a/drivers/clk/imx/clk-imx6sx.c +++ b/drivers/clk/imx/clk-imx6sx.c @@ -80,7 +80,7 @@ static const char *lvds_sels[] = { "arm", "pll1_sys", "dummy", "dummy", "dummy", "dummy", "dummy", "pll5_video_div", "dummy", "dummy", "pcie_ref_125m", "dummy", "usbphy1", "usbphy2", }; -static const char *pll_bypass_src_sels[] = { "osc", "lvds1_in", }; +static const char *pll_bypass_src_sels[] = { "osc", "lvds1_in", "lvds2_in", "dummy", }; static const char *pll1_bypass_sels[] = { "pll1", "pll1_bypass_src", }; static const char *pll2_bypass_sels[] = { "pll2", "pll2_bypass_src", }; static const char *pll3_bypass_sels[] = { "pll3", "pll3_bypass_src", }; @@ -97,12 +97,7 @@ static int const clks_init_on[] __initconst = { IMX6SX_CLK_IPMUX1, IMX6SX_CLK_IPMUX2, IMX6SX_CLK_IPMUX3, IMX6SX_CLK_WAKEUP, IMX6SX_CLK_MMDC_P0_FAST, IMX6SX_CLK_MMDC_P0_IPG, IMX6SX_CLK_ROM, IMX6SX_CLK_ARM, IMX6SX_CLK_IPG, IMX6SX_CLK_OCRAM, - IMX6SX_CLK_PER2_MAIN, IMX6SX_CLK_PERCLK, IMX6SX_CLK_M4, - IMX6SX_CLK_QSPI1, IMX6SX_CLK_QSPI2, IMX6SX_CLK_UART_IPG, - IMX6SX_CLK_UART_SERIAL, IMX6SX_CLK_I2C3, IMX6SX_CLK_ECSPI5, - IMX6SX_CLK_CAN1_IPG, IMX6SX_CLK_CAN1_SERIAL, IMX6SX_CLK_CAN2_IPG, - IMX6SX_CLK_CAN2_SERIAL, IMX6SX_CLK_CANFD, IMX6SX_CLK_EPIT1, - IMX6SX_CLK_EPIT2, + IMX6SX_CLK_PER2_MAIN, IMX6SX_CLK_PERCLK, IMX6SX_CLK_TZASC1, }; static const struct clk_div_table clk_enet_ref_table[] = { @@ -158,8 +153,9 @@ static void __init imx6sx_clocks_init(struct device_node *ccm_node) clks[IMX6SX_CLK_IPP_DI0] = of_clk_get_by_name(ccm_node, "ipp_di0"); clks[IMX6SX_CLK_IPP_DI1] = of_clk_get_by_name(ccm_node, "ipp_di1"); - /* Clock source from external clock via CLK1 PAD */ - clks[IMX6SX_CLK_ANACLK1] = imx_obtain_fixed_clock("anaclk1", 0); + /* Clock source from external clock via CLK1/2 PAD */ + clks[IMX6SX_CLK_ANACLK1] = of_clk_get_by_name(ccm_node, "anaclk1"); + clks[IMX6SX_CLK_ANACLK2] = of_clk_get_by_name(ccm_node, "anaclk2"); np = of_find_compatible_node(NULL, NULL, "fsl,imx6sx-anatop"); base = of_iomap(np, 0); @@ -228,7 +224,9 @@ static void __init imx6sx_clocks_init(struct device_node *ccm_node) clks[IMX6SX_CLK_PCIE_REF_125M] = imx_clk_gate("pcie_ref_125m", "pcie_ref", base + 0xe0, 19); clks[IMX6SX_CLK_LVDS1_OUT] = imx_clk_gate_exclusive("lvds1_out", "lvds1_sel", base + 0x160, 10, BIT(12)); + clks[IMX6SX_CLK_LVDS2_OUT] = imx_clk_gate_exclusive("lvds2_out", "lvds2_sel", base + 0x160, 11, BIT(13)); clks[IMX6SX_CLK_LVDS1_IN] = imx_clk_gate_exclusive("lvds1_in", "anaclk1", base + 0x160, 12, BIT(10)); + clks[IMX6SX_CLK_LVDS2_IN] = imx_clk_gate_exclusive("lvds2_in", "anaclk2", base + 0x160, 13, BIT(11)); clks[IMX6SX_CLK_ENET_REF] = clk_register_divider_table(NULL, "enet_ref", "pll6_enet", 0, base + 0xe0, 0, 2, 0, clk_enet_ref_table, @@ -270,6 +268,7 @@ static void __init imx6sx_clocks_init(struct device_node *ccm_node) /* name reg shift width parent_names num_parents */ clks[IMX6SX_CLK_LVDS1_SEL] = imx_clk_mux("lvds1_sel", base + 0x160, 0, 5, lvds_sels, ARRAY_SIZE(lvds_sels)); + clks[IMX6SX_CLK_LVDS2_SEL] = imx_clk_mux("lvds2_sel", base + 0x160, 5, 5, lvds_sels, ARRAY_SIZE(lvds_sels)); np = ccm_node; base = of_iomap(np, 0); diff --git a/drivers/clk/imx/clk-imx6ul.c b/drivers/clk/imx/clk-imx6ul.c index 12320118f8de..ba563ba50b40 100644 --- a/drivers/clk/imx/clk-imx6ul.c +++ b/drivers/clk/imx/clk-imx6ul.c @@ -40,7 +40,7 @@ static const char *axi_alt_sels[] = { "pll2_pfd2_396m", "pll3_pfd1_540m", }; static const char *axi_sels[] = {"periph", "axi_alt_sel", }; static const char *periph_pre_sels[] = { "pll2_bus", "pll2_pfd2_396m", "pll2_pfd0_352m", "pll2_198m", }; static const char *periph2_pre_sels[] = { "pll2_bus", "pll2_pfd2_396m", "pll2_pfd0_352m", "pll4_audio_div", }; -static const char *periph_clk2_sels[] = { "pll3_usb_otg", "osc", "osc", }; +static const char *periph_clk2_sels[] = { "pll3_usb_otg", "osc", "pll2_bypass_src", }; static const char *periph2_clk2_sels[] = { "pll3_usb_otg", "osc", }; static const char *periph_sels[] = { "periph_pre", "periph_clk2", }; static const char *periph2_sels[] = { "periph2_pre", "periph2_clk2", }; @@ -68,6 +68,13 @@ static const char *sim_sels[] = { "sim_podf", "ipp_di0", "ipp_di1", "ldb_di0", " static const char *epdc_pre_sels[] = { "pll2_bus", "pll3_usb_otg", "pll5_video_div", "pll2_pfd0_352m", "pll2_pfd2_396m", "pll3_pfd2_508m", }; static const char *esai_sels[] = { "pll4_audio_div", "pll3_pfd2_508m", "pll5_video_div", "pll3_usb_otg", }; static const char *epdc_sels[] = { "epdc_podf", "ipp_di0", "ipp_di1", "ldb_di0", "ldb_di1", }; +static const char *cko1_sels[] = { "dummy", "dummy", "dummy", "dummy", "dummy", "axi", "enfc", "dummy", "dummy", + "dummy", "lcdif_pix", "ahb", "ipg", "ipg_per", "ckil", "pll4_audio_div", }; +static const char *cko2_sels[] = { "dummy", "dummy", "dummy", "usdhc1", "dummy", "dummy", "ecspi_root", "dummy", + "dummy", "dummy", "dummy", "dummy", "dummy", "dummy", "osc", "dummy", + "dummy", "usdhc2", "sai1", "sai2", "sai3", "dummy", "dummy", "can_root", + "dummy", "dummy", "dummy", "dummy", "uart_serial", "spdif", "dummy", "dummy", }; +static const char *cko_sels[] = { "cko1", "cko2", }; static struct clk *clks[IMX6UL_CLK_END]; static struct clk_onecell_data clk_data; @@ -273,6 +280,10 @@ static void __init imx6ul_clocks_init(struct device_node *ccm_node) clks[IMX6UL_CLK_LDB_DI0_DIV_SEL] = imx_clk_mux("ldb_di0", base + 0x20, 10, 1, ldb_di0_div_sels, ARRAY_SIZE(ldb_di0_div_sels)); clks[IMX6UL_CLK_LDB_DI1_DIV_SEL] = imx_clk_mux("ldb_di1", base + 0x20, 11, 1, ldb_di1_div_sels, ARRAY_SIZE(ldb_di1_div_sels)); + clks[IMX6UL_CLK_CKO1_SEL] = imx_clk_mux("cko1_sel", base + 0x60, 0, 4, cko1_sels, ARRAY_SIZE(cko1_sels)); + clks[IMX6UL_CLK_CKO2_SEL] = imx_clk_mux("cko2_sel", base + 0x60, 16, 5, cko2_sels, ARRAY_SIZE(cko2_sels)); + clks[IMX6UL_CLK_CKO] = imx_clk_mux("cko", base + 0x60, 8, 1, cko_sels, ARRAY_SIZE(cko_sels)); + clks[IMX6UL_CLK_LDB_DI0_DIV_3_5] = imx_clk_fixed_factor("ldb_di0_div_3_5", "ldb_di0_sel", 2, 7); clks[IMX6UL_CLK_LDB_DI0_DIV_7] = imx_clk_fixed_factor("ldb_di0_div_7", "ldb_di0_sel", 1, 7); clks[IMX6UL_CLK_LDB_DI1_DIV_3_5] = imx_clk_fixed_factor("ldb_di1_div_3_5", "qspi1_sel", 2, 7); @@ -316,6 +327,9 @@ static void __init imx6ul_clocks_init(struct device_node *ccm_node) clks[IMX6UL_CLK_LCDIF_PRED] = imx_clk_divider("lcdif_pred", "lcdif_pre_sel", base + 0x38, 12, 3); clks[IMX6UL_CLK_CSI_PODF] = imx_clk_divider("csi_podf", "csi_sel", base + 0x3c, 11, 3); + clks[IMX6UL_CLK_CKO1_PODF] = imx_clk_divider("cko1_podf", "cko1_sel", base + 0x60, 4, 3); + clks[IMX6UL_CLK_CKO2_PODF] = imx_clk_divider("cko2_podf", "cko2_sel", base + 0x60, 21, 3); + clks[IMX6UL_CLK_ARM] = imx_clk_busy_divider("arm", "pll1_sw", base + 0x10, 0, 3, base + 0x48, 16); clks[IMX6UL_CLK_MMDC_PODF] = imx_clk_busy_divider("mmdc_podf", "periph2", base + 0x14, 3, 3, base + 0x48, 2); clks[IMX6UL_CLK_AXI_PODF] = imx_clk_busy_divider("axi_podf", "axi_sel", base + 0x14, 16, 3, base + 0x48, 0); @@ -445,6 +459,10 @@ static void __init imx6ul_clocks_init(struct device_node *ccm_node) clks[IMX6UL_CLK_PWM6] = imx_clk_gate2("pwm6", "perclk", base + 0x80, 28); clks[IMX6UL_CLK_PWM7] = imx_clk_gate2("pwm7", "perclk", base + 0x80, 30); + /* CCOSR */ + clks[IMX6UL_CLK_CKO1] = imx_clk_gate("cko1", "cko1_podf", base + 0x60, 7); + clks[IMX6UL_CLK_CKO2] = imx_clk_gate("cko2", "cko2_podf", base + 0x60, 24); + /* mask handshake of mmdc */ writel_relaxed(BM_CCM_CCDR_MMDC_CH0_MASK, base + CCDR); diff --git a/drivers/clk/imx/clk-imx7d.c b/drivers/clk/imx/clk-imx7d.c index 975a20d3cc94..27217a7ea17e 100644 --- a/drivers/clk/imx/clk-imx7d.c +++ b/drivers/clk/imx/clk-imx7d.c @@ -26,6 +26,8 @@ static u32 share_count_sai1; static u32 share_count_sai2; static u32 share_count_sai3; static u32 share_count_nand; +static u32 share_count_enet1; +static u32 share_count_enet2; static const struct clk_div_table test_div_table[] = { { .val = 3, .div = 1, }, @@ -729,7 +731,7 @@ static void __init imx7d_clocks_init(struct device_node *ccm_node) clks[IMX7D_LCDIF_PIXEL_ROOT_DIV] = imx_clk_divider2("lcdif_pixel_post_div", "lcdif_pixel_pre_div", base + 0xa300, 0, 6); clks[IMX7D_MIPI_DSI_ROOT_DIV] = imx_clk_divider2("mipi_dsi_post_div", "mipi_dsi_pre_div", base + 0xa380, 0, 6); clks[IMX7D_MIPI_CSI_ROOT_DIV] = imx_clk_divider2("mipi_csi_post_div", "mipi_csi_pre_div", base + 0xa400, 0, 6); - clks[IMX7D_MIPI_DPHY_ROOT_DIV] = imx_clk_divider2("mipi_dphy_post_div", "mipi_csi_dphy_div", base + 0xa480, 0, 6); + clks[IMX7D_MIPI_DPHY_ROOT_DIV] = imx_clk_divider2("mipi_dphy_post_div", "mipi_dphy_pre_div", base + 0xa480, 0, 6); clks[IMX7D_SAI1_ROOT_DIV] = imx_clk_divider2("sai1_post_div", "sai1_pre_div", base + 0xa500, 0, 6); clks[IMX7D_SAI2_ROOT_DIV] = imx_clk_divider2("sai2_post_div", "sai2_pre_div", base + 0xa580, 0, 6); clks[IMX7D_SAI3_ROOT_DIV] = imx_clk_divider2("sai3_post_div", "sai3_pre_div", base + 0xa600, 0, 6); @@ -738,7 +740,7 @@ static void __init imx7d_clocks_init(struct device_node *ccm_node) clks[IMX7D_ENET1_TIME_ROOT_DIV] = imx_clk_divider2("enet1_time_post_div", "enet1_time_pre_div", base + 0xa780, 0, 6); clks[IMX7D_ENET2_REF_ROOT_DIV] = imx_clk_divider2("enet2_ref_post_div", "enet2_ref_pre_div", base + 0xa800, 0, 6); clks[IMX7D_ENET2_TIME_ROOT_DIV] = imx_clk_divider2("enet2_time_post_div", "enet2_time_pre_div", base + 0xa880, 0, 6); - clks[IMX7D_ENET_PHY_REF_ROOT_DIV] = imx_clk_divider2("enet_phy_ref_post_div", "enet_phy_ref_pre_div", base + 0xa900, 0, 6); + clks[IMX7D_ENET_PHY_REF_ROOT_CLK] = imx_clk_divider2("enet_phy_ref_root_clk", "enet_phy_ref_pre_div", base + 0xa900, 0, 6); clks[IMX7D_EIM_ROOT_DIV] = imx_clk_divider2("eim_post_div", "eim_pre_div", base + 0xa980, 0, 6); clks[IMX7D_NAND_ROOT_CLK] = imx_clk_divider2("nand_root_clk", "nand_pre_div", base + 0xaa00, 0, 6); clks[IMX7D_QSPI_ROOT_DIV] = imx_clk_divider2("qspi_post_div", "qspi_pre_div", base + 0xaa80, 0, 6); @@ -805,6 +807,10 @@ static void __init imx7d_clocks_init(struct device_node *ccm_node) clks[IMX7D_MIPI_DSI_ROOT_CLK] = imx_clk_gate4("mipi_dsi_root_clk", "mipi_dsi_post_div", base + 0x4650, 0); clks[IMX7D_MIPI_CSI_ROOT_CLK] = imx_clk_gate4("mipi_csi_root_clk", "mipi_csi_post_div", base + 0x4640, 0); clks[IMX7D_MIPI_DPHY_ROOT_CLK] = imx_clk_gate4("mipi_dphy_root_clk", "mipi_dphy_post_div", base + 0x4660, 0); + clks[IMX7D_ENET1_IPG_ROOT_CLK] = imx_clk_gate2_shared2("enet1_ipg_root_clk", "enet_axi_post_div", base + 0x4700, 0, &share_count_enet1); + clks[IMX7D_ENET1_TIME_ROOT_CLK] = imx_clk_gate2_shared2("enet1_time_root_clk", "enet1_time_post_div", base + 0x4700, 0, &share_count_enet1); + clks[IMX7D_ENET2_IPG_ROOT_CLK] = imx_clk_gate2_shared2("enet2_ipg_root_clk", "enet_axi_post_div", base + 0x4710, 0, &share_count_enet2); + clks[IMX7D_ENET2_TIME_ROOT_CLK] = imx_clk_gate2_shared2("enet2_time_root_clk", "enet2_time_post_div", base + 0x4710, 0, &share_count_enet2); clks[IMX7D_SAI1_ROOT_CLK] = imx_clk_gate2_shared2("sai1_root_clk", "sai1_post_div", base + 0x48c0, 0, &share_count_sai1); clks[IMX7D_SAI1_IPG_CLK] = imx_clk_gate2_shared2("sai1_ipg_clk", "ipg_root_clk", base + 0x48c0, 0, &share_count_sai1); clks[IMX7D_SAI2_ROOT_CLK] = imx_clk_gate2_shared2("sai2_root_clk", "sai2_post_div", base + 0x48d0, 0, &share_count_sai2); @@ -812,11 +818,6 @@ static void __init imx7d_clocks_init(struct device_node *ccm_node) clks[IMX7D_SAI3_ROOT_CLK] = imx_clk_gate2_shared2("sai3_root_clk", "sai3_post_div", base + 0x48e0, 0, &share_count_sai3); clks[IMX7D_SAI3_IPG_CLK] = imx_clk_gate2_shared2("sai3_ipg_clk", "ipg_root_clk", base + 0x48e0, 0, &share_count_sai3); clks[IMX7D_SPDIF_ROOT_CLK] = imx_clk_gate4("spdif_root_clk", "spdif_post_div", base + 0x44d0, 0); - clks[IMX7D_ENET1_REF_ROOT_CLK] = imx_clk_gate4("enet1_ref_root_clk", "enet1_ref_post_div", base + 0x44e0, 0); - clks[IMX7D_ENET1_TIME_ROOT_CLK] = imx_clk_gate4("enet1_time_root_clk", "enet1_time_post_div", base + 0x44f0, 0); - clks[IMX7D_ENET2_REF_ROOT_CLK] = imx_clk_gate4("enet2_ref_root_clk", "enet2_ref_post_div", base + 0x4500, 0); - clks[IMX7D_ENET2_TIME_ROOT_CLK] = imx_clk_gate4("enet2_time_root_clk", "enet2_time_post_div", base + 0x4510, 0); - clks[IMX7D_ENET_PHY_REF_ROOT_CLK] = imx_clk_gate4("enet_phy_ref_root_clk", "enet_phy_ref_post_div", base + 0x4520, 0); clks[IMX7D_EIM_ROOT_CLK] = imx_clk_gate4("eim_root_clk", "eim_post_div", base + 0x4160, 0); clks[IMX7D_NAND_RAWNAND_CLK] = imx_clk_gate2_shared2("nand_rawnand_clk", "nand_root_clk", base + 0x4140, 0, &share_count_nand); clks[IMX7D_NAND_USDHC_BUS_RAWNAND_CLK] = imx_clk_gate2_shared2("nand_usdhc_rawnand_clk", "nand_usdhc_root_clk", base + 0x4140, 0, &share_count_nand); @@ -891,6 +892,8 @@ static void __init imx7d_clocks_init(struct device_node *ccm_node) clk_set_parent(clks[IMX7D_PLL_AUDIO_MAIN_BYPASS], clks[IMX7D_PLL_AUDIO_MAIN]); clk_set_parent(clks[IMX7D_PLL_VIDEO_MAIN_BYPASS], clks[IMX7D_PLL_VIDEO_MAIN]); + clk_set_parent(clks[IMX7D_MIPI_CSI_ROOT_SRC], clks[IMX7D_PLL_SYS_PFD3_CLK]); + /* use old gpt clk setting, gpt1 root clk must be twice as gpt counter freq */ clk_set_parent(clks[IMX7D_GPT1_ROOT_SRC], clks[IMX7D_OSC_24M_CLK]); diff --git a/drivers/clk/ingenic/cgu.c b/drivers/clk/ingenic/cgu.c index 56a712c9075f..5ef7d9ba2195 100644 --- a/drivers/clk/ingenic/cgu.c +++ b/drivers/clk/ingenic/cgu.c @@ -43,7 +43,8 @@ static inline bool ingenic_cgu_gate_get(struct ingenic_cgu *cgu, const struct ingenic_cgu_gate_info *info) { - return readl(cgu->base + info->reg) & BIT(info->bit); + return !!(readl(cgu->base + info->reg) & BIT(info->bit)) + ^ info->clear_to_gate; } /** @@ -62,7 +63,7 @@ ingenic_cgu_gate_set(struct ingenic_cgu *cgu, { u32 clkgr = readl(cgu->base + info->reg); - if (val) + if (val ^ info->clear_to_gate) clkgr |= BIT(info->bit); else clkgr &= ~BIT(info->bit); @@ -511,6 +512,9 @@ static int ingenic_clk_enable(struct clk_hw *hw) spin_lock_irqsave(&cgu->lock, flags); ingenic_cgu_gate_set(cgu, &clk_info->gate, false); spin_unlock_irqrestore(&cgu->lock, flags); + + if (clk_info->gate.delay_us) + udelay(clk_info->gate.delay_us); } return 0; diff --git a/drivers/clk/ingenic/cgu.h b/drivers/clk/ingenic/cgu.h index 9da34910bd80..502bcbb61b04 100644 --- a/drivers/clk/ingenic/cgu.h +++ b/drivers/clk/ingenic/cgu.h @@ -111,10 +111,14 @@ struct ingenic_cgu_fixdiv_info { * struct ingenic_cgu_gate_info - information about a clock gate * @reg: offset of the gate control register within the CGU * @bit: offset of the bit in the register that controls the gate + * @clear_to_gate: if set, the clock is gated when the bit is cleared + * @delay_us: delay in microseconds after which the clock is considered stable */ struct ingenic_cgu_gate_info { unsigned reg; u8 bit; + bool clear_to_gate; + u16 delay_us; }; /** @@ -190,7 +194,7 @@ struct ingenic_cgu { /** * struct ingenic_clk - private data for a clock - * @hw: see Documentation/clk.txt + * @hw: see Documentation/driver-api/clk.rst * @cgu: a pointer to the CGU data * @idx: the index of this clock in cgu->clock_info */ diff --git a/drivers/clk/ingenic/jz4770-cgu.c b/drivers/clk/ingenic/jz4770-cgu.c index c78d369b9403..bf46a0df2004 100644 --- a/drivers/clk/ingenic/jz4770-cgu.c +++ b/drivers/clk/ingenic/jz4770-cgu.c @@ -42,7 +42,6 @@ /* bits within the OPCR register */ #define OPCR_SPENDH BIT(5) /* UHC PHY suspend */ -#define OPCR_SPENDN BIT(7) /* OTG PHY suspend */ /* bits within the USBPCR1 register */ #define USBPCR1_UHC_POWER BIT(5) /* UHC PHY power down */ @@ -83,37 +82,6 @@ static const struct clk_ops jz4770_uhc_phy_ops = { .is_enabled = jz4770_uhc_phy_is_enabled, }; -static int jz4770_otg_phy_enable(struct clk_hw *hw) -{ - void __iomem *reg_opcr = cgu->base + CGU_REG_OPCR; - - writel(readl(reg_opcr) | OPCR_SPENDN, reg_opcr); - - /* Wait for the clock to be stable */ - udelay(50); - return 0; -} - -static void jz4770_otg_phy_disable(struct clk_hw *hw) -{ - void __iomem *reg_opcr = cgu->base + CGU_REG_OPCR; - - writel(readl(reg_opcr) & ~OPCR_SPENDN, reg_opcr); -} - -static int jz4770_otg_phy_is_enabled(struct clk_hw *hw) -{ - void __iomem *reg_opcr = cgu->base + CGU_REG_OPCR; - - return !!(readl(reg_opcr) & OPCR_SPENDN); -} - -static const struct clk_ops jz4770_otg_phy_ops = { - .enable = jz4770_otg_phy_enable, - .disable = jz4770_otg_phy_disable, - .is_enabled = jz4770_otg_phy_is_enabled, -}; - static const s8 pll_od_encoding[8] = { 0x0, 0x1, -1, 0x2, -1, -1, -1, 0x3, }; @@ -186,7 +154,7 @@ static const struct ingenic_cgu_clk_info jz4770_cgu_clocks[] = { "h1clk", CGU_CLK_DIV | CGU_CLK_GATE, .parents = { JZ4770_CLK_PLL0, }, .div = { CGU_REG_CPCCR, 24, 1, 4, 22, -1, -1 }, - .gate = { CGU_REG_LCR, 30 }, + .gate = { CGU_REG_CLKGR1, 7 }, }, [JZ4770_CLK_H2CLK] = { "h2clk", CGU_CLK_DIV, @@ -194,9 +162,10 @@ static const struct ingenic_cgu_clk_info jz4770_cgu_clocks[] = { .div = { CGU_REG_CPCCR, 16, 1, 4, 22, -1, -1 }, }, [JZ4770_CLK_C1CLK] = { - "c1clk", CGU_CLK_DIV, + "c1clk", CGU_CLK_DIV | CGU_CLK_GATE, .parents = { JZ4770_CLK_PLL0, }, .div = { CGU_REG_CPCCR, 12, 1, 4, 22, -1, -1 }, + .gate = { CGU_REG_OPCR, 31, true }, // disable CCLK stop on idle }, [JZ4770_CLK_PCLK] = { "pclk", CGU_CLK_DIV, @@ -393,7 +362,7 @@ static const struct ingenic_cgu_clk_info jz4770_cgu_clocks[] = { [JZ4770_CLK_VPU] = { "vpu", CGU_CLK_GATE, .parents = { JZ4770_CLK_H1CLK, }, - .gate = { CGU_REG_CLKGR1, 7 }, + .gate = { CGU_REG_LCR, 30, false, 150 }, }, [JZ4770_CLK_MMC0] = { "mmc0", CGU_CLK_GATE, @@ -410,6 +379,11 @@ static const struct ingenic_cgu_clk_info jz4770_cgu_clocks[] = { .parents = { JZ4770_CLK_MMC2_MUX, }, .gate = { CGU_REG_CLKGR0, 12 }, }, + [JZ4770_CLK_OTG_PHY] = { + "usb_phy", CGU_CLK_GATE, + .parents = { JZ4770_CLK_OTG }, + .gate = { CGU_REG_OPCR, 7, true, 50 }, + }, /* Custom clocks */ @@ -418,11 +392,6 @@ static const struct ingenic_cgu_clk_info jz4770_cgu_clocks[] = { .parents = { JZ4770_CLK_UHC, -1, -1, -1 }, .custom = { &jz4770_uhc_phy_ops }, }, - [JZ4770_CLK_OTG_PHY] = { - "usb_phy", CGU_CLK_CUSTOM, - .parents = { JZ4770_CLK_OTG, -1, -1, -1 }, - .custom = { &jz4770_otg_phy_ops }, - }, [JZ4770_CLK_EXT512] = { "ext/512", CGU_CLK_FIXDIV, diff --git a/drivers/clk/mediatek/Kconfig b/drivers/clk/mediatek/Kconfig index 92afe5989e97..3dd1dab92223 100644 --- a/drivers/clk/mediatek/Kconfig +++ b/drivers/clk/mediatek/Kconfig @@ -60,6 +60,12 @@ config COMMON_CLK_MT2701_AUDSYS ---help--- This driver supports Mediatek MT2701 audsys clocks. +config COMMON_CLK_MT2701_G3DSYS + bool "Clock driver for MediaTek MT2701 g3dsys" + depends on COMMON_CLK_MT2701 + ---help--- + This driver supports MediaTek MT2701 g3dsys clocks. + config COMMON_CLK_MT2712 bool "Clock driver for MediaTek MT2712" depends on (ARCH_MEDIATEK && ARM64) || COMPILE_TEST diff --git a/drivers/clk/mediatek/Makefile b/drivers/clk/mediatek/Makefile index b80eff2abb31..844b55d2770d 100644 --- a/drivers/clk/mediatek/Makefile +++ b/drivers/clk/mediatek/Makefile @@ -9,6 +9,7 @@ obj-$(CONFIG_COMMON_CLK_MT2701) += clk-mt2701.o obj-$(CONFIG_COMMON_CLK_MT2701_AUDSYS) += clk-mt2701-aud.o obj-$(CONFIG_COMMON_CLK_MT2701_BDPSYS) += clk-mt2701-bdp.o obj-$(CONFIG_COMMON_CLK_MT2701_ETHSYS) += clk-mt2701-eth.o +obj-$(CONFIG_COMMON_CLK_MT2701_G3DSYS) += clk-mt2701-g3d.o obj-$(CONFIG_COMMON_CLK_MT2701_HIFSYS) += clk-mt2701-hif.o obj-$(CONFIG_COMMON_CLK_MT2701_IMGSYS) += clk-mt2701-img.o obj-$(CONFIG_COMMON_CLK_MT2701_MMSYS) += clk-mt2701-mm.o diff --git a/drivers/clk/mediatek/clk-mt2701-g3d.c b/drivers/clk/mediatek/clk-mt2701-g3d.c new file mode 100644 index 000000000000..1328c112a38f --- /dev/null +++ b/drivers/clk/mediatek/clk-mt2701-g3d.c @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2018 MediaTek Inc. + * Author: Sean Wang + * + */ + +#include +#include +#include +#include +#include + +#include "clk-mtk.h" +#include "clk-gate.h" + +#include + +#define GATE_G3D(_id, _name, _parent, _shift) { \ + .id = _id, \ + .name = _name, \ + .parent_name = _parent, \ + .regs = &g3d_cg_regs, \ + .shift = _shift, \ + .ops = &mtk_clk_gate_ops_setclr, \ + } + +static const struct mtk_gate_regs g3d_cg_regs = { + .sta_ofs = 0x0, + .set_ofs = 0x4, + .clr_ofs = 0x8, +}; + +static const struct mtk_gate g3d_clks[] = { + GATE_G3D(CLK_G3DSYS_CORE, "g3d_core", "mfg_sel", 0), +}; + +static int clk_mt2701_g3dsys_init(struct platform_device *pdev) +{ + struct clk_onecell_data *clk_data; + struct device_node *node = pdev->dev.of_node; + int r; + + clk_data = mtk_alloc_clk_data(CLK_G3DSYS_NR); + + mtk_clk_register_gates(node, g3d_clks, ARRAY_SIZE(g3d_clks), + clk_data); + + r = of_clk_add_provider(node, of_clk_src_onecell_get, clk_data); + if (r) + dev_err(&pdev->dev, + "could not register clock provider: %s: %d\n", + pdev->name, r); + + mtk_register_reset_controller(node, 1, 0xc); + + return r; +} + +static const struct of_device_id of_match_clk_mt2701_g3d[] = { + { + .compatible = "mediatek,mt2701-g3dsys", + .data = clk_mt2701_g3dsys_init, + }, { + /* sentinel */ + } +}; + +static int clk_mt2701_g3d_probe(struct platform_device *pdev) +{ + int (*clk_init)(struct platform_device *); + int r; + + clk_init = of_device_get_match_data(&pdev->dev); + if (!clk_init) + return -EINVAL; + + r = clk_init(pdev); + if (r) + dev_err(&pdev->dev, + "could not register clock provider: %s: %d\n", + pdev->name, r); + + return r; +} + +static struct platform_driver clk_mt2701_g3d_drv = { + .probe = clk_mt2701_g3d_probe, + .driver = { + .name = "clk-mt2701-g3d", + .of_match_table = of_match_clk_mt2701_g3d, + }, +}; + +builtin_platform_driver(clk_mt2701_g3d_drv); diff --git a/drivers/clk/mediatek/clk-mt2701.c b/drivers/clk/mediatek/clk-mt2701.c index deca7527f92f..4dda8988b2f0 100644 --- a/drivers/clk/mediatek/clk-mt2701.c +++ b/drivers/clk/mediatek/clk-mt2701.c @@ -46,8 +46,6 @@ static const struct mtk_fixed_clk top_fixed_clks[] = { 340 * MHZ), FIXED_CLK(CLK_TOP_HDMI_0_PLL340M, "hdmi_0_pll340m", "clk26m", 340 * MHZ), - FIXED_CLK(CLK_TOP_HDMITX_CLKDIG_CTS, "hdmitx_dig_cts", "clk26m", - 300 * MHZ), FIXED_CLK(CLK_TOP_HADDS2_FB, "hadds2_fbclk", "clk26m", 27 * MHZ), FIXED_CLK(CLK_TOP_WBG_DIG_416M, "wbg_dig_ck_416m", "clk26m", @@ -977,6 +975,10 @@ static const struct mtk_pll_data apmixed_plls[] = { 21, 0x2d0, 4, 0x0, 0x2d4, 0), }; +static const struct mtk_fixed_factor apmixed_fixed_divs[] = { + FACTOR(CLK_APMIXED_HDMI_REF, "hdmi_ref", "tvdpll", 1, 1), +}; + static int mtk_apmixedsys_init(struct platform_device *pdev) { struct clk_onecell_data *clk_data; @@ -988,6 +990,8 @@ static int mtk_apmixedsys_init(struct platform_device *pdev) mtk_clk_register_plls(node, apmixed_plls, ARRAY_SIZE(apmixed_plls), clk_data); + mtk_clk_register_factors(apmixed_fixed_divs, ARRAY_SIZE(apmixed_fixed_divs), + clk_data); return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data); } diff --git a/drivers/clk/meson/Kconfig b/drivers/clk/meson/Kconfig index d5cbec522aec..815659eebea3 100644 --- a/drivers/clk/meson/Kconfig +++ b/drivers/clk/meson/Kconfig @@ -3,6 +3,12 @@ config COMMON_CLK_AMLOGIC depends on OF depends on ARCH_MESON || COMPILE_TEST +config COMMON_CLK_MESON_AO + bool + depends on OF + depends on ARCH_MESON || COMPILE_TEST + select COMMON_CLK_REGMAP_MESON + config COMMON_CLK_REGMAP_MESON bool select REGMAP @@ -21,6 +27,7 @@ config COMMON_CLK_GXBB bool depends on COMMON_CLK_AMLOGIC select RESET_CONTROLLER + select COMMON_CLK_MESON_AO select COMMON_CLK_REGMAP_MESON select MFD_SYSCON help @@ -31,6 +38,7 @@ config COMMON_CLK_AXG bool depends on COMMON_CLK_AMLOGIC select RESET_CONTROLLER + select COMMON_CLK_MESON_AO select COMMON_CLK_REGMAP_MESON select MFD_SYSCON help diff --git a/drivers/clk/meson/Makefile b/drivers/clk/meson/Makefile index ffee82e60b7a..d0d13aeb369a 100644 --- a/drivers/clk/meson/Makefile +++ b/drivers/clk/meson/Makefile @@ -3,7 +3,8 @@ # obj-$(CONFIG_COMMON_CLK_AMLOGIC) += clk-pll.o clk-mpll.o clk-audio-divider.o +obj-$(CONFIG_COMMON_CLK_MESON_AO) += meson-aoclk.o obj-$(CONFIG_COMMON_CLK_MESON8B) += meson8b.o obj-$(CONFIG_COMMON_CLK_GXBB) += gxbb.o gxbb-aoclk.o gxbb-aoclk-32k.o -obj-$(CONFIG_COMMON_CLK_AXG) += axg.o +obj-$(CONFIG_COMMON_CLK_AXG) += axg.o axg-aoclk.o obj-$(CONFIG_COMMON_CLK_REGMAP_MESON) += clk-regmap.o diff --git a/drivers/clk/meson/axg-aoclk.c b/drivers/clk/meson/axg-aoclk.c new file mode 100644 index 000000000000..29e088542387 --- /dev/null +++ b/drivers/clk/meson/axg-aoclk.c @@ -0,0 +1,164 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * Amlogic Meson-AXG Clock Controller Driver + * + * Copyright (c) 2016 Baylibre SAS. + * Author: Michael Turquette + * + * Copyright (c) 2018 Amlogic, inc. + * Author: Qiufang Dai + */ +#include +#include +#include +#include +#include "clk-regmap.h" +#include "meson-aoclk.h" +#include "axg-aoclk.h" + +#define AXG_AO_GATE(_name, _bit) \ +static struct clk_regmap axg_aoclk_##_name = { \ + .data = &(struct clk_regmap_gate_data) { \ + .offset = (AO_RTI_GEN_CNTL_REG0), \ + .bit_idx = (_bit), \ + }, \ + .hw.init = &(struct clk_init_data) { \ + .name = "axg_ao_" #_name, \ + .ops = &clk_regmap_gate_ops, \ + .parent_names = (const char *[]){ "clk81" }, \ + .num_parents = 1, \ + .flags = CLK_IGNORE_UNUSED, \ + }, \ +} + +AXG_AO_GATE(remote, 0); +AXG_AO_GATE(i2c_master, 1); +AXG_AO_GATE(i2c_slave, 2); +AXG_AO_GATE(uart1, 3); +AXG_AO_GATE(uart2, 5); +AXG_AO_GATE(ir_blaster, 6); +AXG_AO_GATE(saradc, 7); + +static struct clk_regmap axg_aoclk_clk81 = { + .data = &(struct clk_regmap_mux_data) { + .offset = AO_RTI_PWR_CNTL_REG0, + .mask = 0x1, + .shift = 8, + }, + .hw.init = &(struct clk_init_data){ + .name = "axg_ao_clk81", + .ops = &clk_regmap_mux_ro_ops, + .parent_names = (const char *[]){ "clk81", "ao_alt_xtal"}, + .num_parents = 2, + }, +}; + +static struct clk_regmap axg_aoclk_saradc_mux = { + .data = &(struct clk_regmap_mux_data) { + .offset = AO_SAR_CLK, + .mask = 0x3, + .shift = 9, + }, + .hw.init = &(struct clk_init_data){ + .name = "axg_ao_saradc_mux", + .ops = &clk_regmap_mux_ops, + .parent_names = (const char *[]){ "xtal", "axg_ao_clk81" }, + .num_parents = 2, + }, +}; + +static struct clk_regmap axg_aoclk_saradc_div = { + .data = &(struct clk_regmap_div_data) { + .offset = AO_SAR_CLK, + .shift = 0, + .width = 8, + }, + .hw.init = &(struct clk_init_data){ + .name = "axg_ao_saradc_div", + .ops = &clk_regmap_divider_ops, + .parent_names = (const char *[]){ "axg_ao_saradc_mux" }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + }, +}; + +static struct clk_regmap axg_aoclk_saradc_gate = { + .data = &(struct clk_regmap_gate_data) { + .offset = AO_SAR_CLK, + .bit_idx = 8, + }, + .hw.init = &(struct clk_init_data){ + .name = "axg_ao_saradc_gate", + .ops = &clk_regmap_gate_ops, + .parent_names = (const char *[]){ "axg_ao_saradc_div" }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + }, +}; + +static const unsigned int axg_aoclk_reset[] = { + [RESET_AO_REMOTE] = 16, + [RESET_AO_I2C_MASTER] = 18, + [RESET_AO_I2C_SLAVE] = 19, + [RESET_AO_UART1] = 17, + [RESET_AO_UART2] = 22, + [RESET_AO_IR_BLASTER] = 23, +}; + +static struct clk_regmap *axg_aoclk_regmap[] = { + [CLKID_AO_REMOTE] = &axg_aoclk_remote, + [CLKID_AO_I2C_MASTER] = &axg_aoclk_i2c_master, + [CLKID_AO_I2C_SLAVE] = &axg_aoclk_i2c_slave, + [CLKID_AO_UART1] = &axg_aoclk_uart1, + [CLKID_AO_UART2] = &axg_aoclk_uart2, + [CLKID_AO_IR_BLASTER] = &axg_aoclk_ir_blaster, + [CLKID_AO_SAR_ADC] = &axg_aoclk_saradc, + [CLKID_AO_CLK81] = &axg_aoclk_clk81, + [CLKID_AO_SAR_ADC_SEL] = &axg_aoclk_saradc_mux, + [CLKID_AO_SAR_ADC_DIV] = &axg_aoclk_saradc_div, + [CLKID_AO_SAR_ADC_CLK] = &axg_aoclk_saradc_gate, +}; + +static const struct clk_hw_onecell_data axg_aoclk_onecell_data = { + .hws = { + [CLKID_AO_REMOTE] = &axg_aoclk_remote.hw, + [CLKID_AO_I2C_MASTER] = &axg_aoclk_i2c_master.hw, + [CLKID_AO_I2C_SLAVE] = &axg_aoclk_i2c_slave.hw, + [CLKID_AO_UART1] = &axg_aoclk_uart1.hw, + [CLKID_AO_UART2] = &axg_aoclk_uart2.hw, + [CLKID_AO_IR_BLASTER] = &axg_aoclk_ir_blaster.hw, + [CLKID_AO_SAR_ADC] = &axg_aoclk_saradc.hw, + [CLKID_AO_CLK81] = &axg_aoclk_clk81.hw, + [CLKID_AO_SAR_ADC_SEL] = &axg_aoclk_saradc_mux.hw, + [CLKID_AO_SAR_ADC_DIV] = &axg_aoclk_saradc_div.hw, + [CLKID_AO_SAR_ADC_CLK] = &axg_aoclk_saradc_gate.hw, + }, + .num = NR_CLKS, +}; + +static const struct meson_aoclk_data axg_aoclkc_data = { + .reset_reg = AO_RTI_GEN_CNTL_REG0, + .num_reset = ARRAY_SIZE(axg_aoclk_reset), + .reset = axg_aoclk_reset, + .num_clks = ARRAY_SIZE(axg_aoclk_regmap), + .clks = axg_aoclk_regmap, + .hw_data = &axg_aoclk_onecell_data, +}; + +static const struct of_device_id axg_aoclkc_match_table[] = { + { + .compatible = "amlogic,meson-axg-aoclkc", + .data = &axg_aoclkc_data, + }, + { } +}; + +static struct platform_driver axg_aoclkc_driver = { + .probe = meson_aoclkc_probe, + .driver = { + .name = "axg-aoclkc", + .of_match_table = axg_aoclkc_match_table, + }, +}; + +builtin_platform_driver(axg_aoclkc_driver); diff --git a/drivers/clk/meson/axg-aoclk.h b/drivers/clk/meson/axg-aoclk.h new file mode 100644 index 000000000000..91384d8dd844 --- /dev/null +++ b/drivers/clk/meson/axg-aoclk.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */ +/* + * Copyright (c) 2017 BayLibre, SAS + * Author: Neil Armstrong + * + * Copyright (c) 2018 Amlogic, inc. + * Author: Qiufang Dai + */ + +#ifndef __AXG_AOCLKC_H +#define __AXG_AOCLKC_H + +#define NR_CLKS 11 +/* AO Configuration Clock registers offsets + * Register offsets from the data sheet must be multiplied by 4. + */ +#define AO_RTI_PWR_CNTL_REG1 0x0C +#define AO_RTI_PWR_CNTL_REG0 0x10 +#define AO_RTI_GEN_CNTL_REG0 0x40 +#define AO_OSCIN_CNTL 0x58 +#define AO_CRT_CLK_CNTL1 0x68 +#define AO_SAR_CLK 0x90 +#define AO_RTC_ALT_CLK_CNTL0 0x94 +#define AO_RTC_ALT_CLK_CNTL1 0x98 + +#include +#include + +#endif /* __AXG_AOCLKC_H */ diff --git a/drivers/clk/meson/axg.c b/drivers/clk/meson/axg.c index 5f5d468c1efe..bd4dbc696b88 100644 --- a/drivers/clk/meson/axg.c +++ b/drivers/clk/meson/axg.c @@ -461,6 +461,7 @@ static struct clk_regmap axg_mpll0_div = { .width = 1, }, .lock = &meson_clk_lock, + .flags = CLK_MESON_MPLL_ROUND_CLOSEST, }, .hw.init = &(struct clk_init_data){ .name = "mpll0_div", @@ -507,6 +508,7 @@ static struct clk_regmap axg_mpll1_div = { .width = 1, }, .lock = &meson_clk_lock, + .flags = CLK_MESON_MPLL_ROUND_CLOSEST, }, .hw.init = &(struct clk_init_data){ .name = "mpll1_div", @@ -553,6 +555,7 @@ static struct clk_regmap axg_mpll2_div = { .width = 1, }, .lock = &meson_clk_lock, + .flags = CLK_MESON_MPLL_ROUND_CLOSEST, }, .hw.init = &(struct clk_init_data){ .name = "mpll2_div", @@ -599,6 +602,7 @@ static struct clk_regmap axg_mpll3_div = { .width = 1, }, .lock = &meson_clk_lock, + .flags = CLK_MESON_MPLL_ROUND_CLOSEST, }, .hw.init = &(struct clk_init_data){ .name = "mpll3_div", diff --git a/drivers/clk/meson/clk-audio-divider.c b/drivers/clk/meson/clk-audio-divider.c index f7ab5b1db342..58f546e04807 100644 --- a/drivers/clk/meson/clk-audio-divider.c +++ b/drivers/clk/meson/clk-audio-divider.c @@ -1,18 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2017 AmLogic, Inc. * Author: Jerome Brunet - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ /* diff --git a/drivers/clk/meson/clk-mpll.c b/drivers/clk/meson/clk-mpll.c index 0df1227b65b3..650f75cc15a9 100644 --- a/drivers/clk/meson/clk-mpll.c +++ b/drivers/clk/meson/clk-mpll.c @@ -1,57 +1,7 @@ +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) /* - * This file is provided under a dual BSD/GPLv2 license. When using or - * redistributing this file, you may do so under either license. - * - * GPL LICENSE SUMMARY - * - * Copyright (c) 2016 AmLogic, Inc. - * Author: Michael Turquette - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of version 2 of the GNU General Public License as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA. - * The full GNU General Public License is included in this distribution - * in the file called COPYING - * - * BSD LICENSE - * * Copyright (c) 2016 AmLogic, Inc. * Author: Michael Turquette - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Intel Corporation nor the names of its - * contributors may be used to endorse or promote products derived - * from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ /* @@ -89,10 +39,23 @@ static long rate_from_params(unsigned long parent_rate, static void params_from_rate(unsigned long requested_rate, unsigned long parent_rate, unsigned int *sdm, - unsigned int *n2) + unsigned int *n2, + u8 flags) { uint64_t div = parent_rate; - unsigned long rem = do_div(div, requested_rate); + uint64_t frac = do_div(div, requested_rate); + + frac *= SDM_DEN; + + if (flags & CLK_MESON_MPLL_ROUND_CLOSEST) + *sdm = DIV_ROUND_CLOSEST_ULL(frac, requested_rate); + else + *sdm = DIV_ROUND_UP_ULL(frac, requested_rate); + + if (*sdm == SDM_DEN) { + *sdm = 0; + div += 1; + } if (div < N2_MIN) { *n2 = N2_MIN; @@ -102,7 +65,6 @@ static void params_from_rate(unsigned long requested_rate, *sdm = SDM_DEN - 1; } else { *n2 = div; - *sdm = DIV_ROUND_UP_ULL((u64)rem * SDM_DEN, requested_rate); } } @@ -125,9 +87,11 @@ static long mpll_round_rate(struct clk_hw *hw, unsigned long rate, unsigned long *parent_rate) { + struct clk_regmap *clk = to_clk_regmap(hw); + struct meson_clk_mpll_data *mpll = meson_clk_mpll_data(clk); unsigned int sdm, n2; - params_from_rate(rate, *parent_rate, &sdm, &n2); + params_from_rate(rate, *parent_rate, &sdm, &n2, mpll->flags); return rate_from_params(*parent_rate, sdm, n2); } @@ -140,7 +104,7 @@ static int mpll_set_rate(struct clk_hw *hw, unsigned int sdm, n2; unsigned long flags = 0; - params_from_rate(rate, parent_rate, &sdm, &n2); + params_from_rate(rate, parent_rate, &sdm, &n2, mpll->flags); if (mpll->lock) spin_lock_irqsave(mpll->lock, flags); diff --git a/drivers/clk/meson/clk-pll.c b/drivers/clk/meson/clk-pll.c index 65a7bd903551..3e04617ac47f 100644 --- a/drivers/clk/meson/clk-pll.c +++ b/drivers/clk/meson/clk-pll.c @@ -1,21 +1,10 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2015 Endless Mobile, Inc. * Author: Carlo Caione * * Copyright (c) 2018 Baylibre, SAS. * Author: Jerome Brunet - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ /* diff --git a/drivers/clk/meson/clk-regmap.c b/drivers/clk/meson/clk-regmap.c index ab7a3556f5b2..305ee307c003 100644 --- a/drivers/clk/meson/clk-regmap.c +++ b/drivers/clk/meson/clk-regmap.c @@ -1,6 +1,8 @@ // SPDX-License-Identifier: GPL-2.0 -// Copyright (c) 2018 BayLibre, SAS. -// Author: Jerome Brunet +/* + * Copyright (c) 2018 BayLibre, SAS. + * Author: Jerome Brunet + */ #include "clk-regmap.h" diff --git a/drivers/clk/meson/clk-regmap.h b/drivers/clk/meson/clk-regmap.h index 627c888026d7..ed2d4348dbe2 100644 --- a/drivers/clk/meson/clk-regmap.h +++ b/drivers/clk/meson/clk-regmap.h @@ -1,6 +1,8 @@ -// SPDX-License-Identifier: GPL-2.0 -// Copyright (c) 2018 BayLibre, SAS. -// Author: Jerome Brunet +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2018 BayLibre, SAS. + * Author: Jerome Brunet + */ #ifndef __CLK_REGMAP_H #define __CLK_REGMAP_H diff --git a/drivers/clk/meson/clkc.h b/drivers/clk/meson/clkc.h index 8fe73c4edca8..2fb084330ee9 100644 --- a/drivers/clk/meson/clkc.h +++ b/drivers/clk/meson/clkc.h @@ -1,18 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (c) 2015 Endless Mobile, Inc. * Author: Carlo Caione - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #ifndef __CLKC_H @@ -97,8 +86,11 @@ struct meson_clk_mpll_data { struct parm ssen; struct parm misc; spinlock_t *lock; + u8 flags; }; +#define CLK_MESON_MPLL_ROUND_CLOSEST BIT(0) + struct meson_clk_audio_div_data { struct parm div; u8 flags; diff --git a/drivers/clk/meson/gxbb-aoclk-32k.c b/drivers/clk/meson/gxbb-aoclk-32k.c index 491634dbc985..680467141a1d 100644 --- a/drivers/clk/meson/gxbb-aoclk-32k.c +++ b/drivers/clk/meson/gxbb-aoclk-32k.c @@ -1,8 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * Copyright (c) 2017 BayLibre, SAS. * Author: Neil Armstrong - * - * SPDX-License-Identifier: GPL-2.0+ */ #include diff --git a/drivers/clk/meson/gxbb-aoclk.c b/drivers/clk/meson/gxbb-aoclk.c index 9ec23ae9a219..42ed61d3c3fb 100644 --- a/drivers/clk/meson/gxbb-aoclk.c +++ b/drivers/clk/meson/gxbb-aoclk.c @@ -1,90 +1,14 @@ +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) /* - * This file is provided under a dual BSD/GPLv2 license. When using or - * redistributing this file, you may do so under either license. - * - * GPL LICENSE SUMMARY - * * Copyright (c) 2016 BayLibre, SAS. * Author: Neil Armstrong - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of version 2 of the GNU General Public License as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, see . - * The full GNU General Public License is included in this distribution - * in the file called COPYING. - * - * BSD LICENSE - * - * Copyright (c) 2016 BayLibre, SAS. - * Author: Neil Armstrong - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Intel Corporation nor the names of its - * contributors may be used to endorse or promote products derived - * from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ -#include -#include #include -#include #include -#include -#include -#include -#include -#include #include "clk-regmap.h" +#include "meson-aoclk.h" #include "gxbb-aoclk.h" -struct gxbb_aoclk_reset_controller { - struct reset_controller_dev reset; - unsigned int *data; - struct regmap *regmap; -}; - -static int gxbb_aoclk_do_reset(struct reset_controller_dev *rcdev, - unsigned long id) -{ - struct gxbb_aoclk_reset_controller *reset = - container_of(rcdev, struct gxbb_aoclk_reset_controller, reset); - - return regmap_write(reset->regmap, AO_RTI_GEN_CNTL_REG0, - BIT(reset->data[id])); -} - -static const struct reset_control_ops gxbb_aoclk_reset_ops = { - .reset = gxbb_aoclk_do_reset, -}; - #define GXBB_AO_GATE(_name, _bit) \ static struct clk_regmap _name##_ao = { \ .data = &(struct clk_regmap_gate_data) { \ @@ -96,7 +20,7 @@ static struct clk_regmap _name##_ao = { \ .ops = &clk_regmap_gate_ops, \ .parent_names = (const char *[]){ "clk81" }, \ .num_parents = 1, \ - .flags = (CLK_SET_RATE_PARENT | CLK_IGNORE_UNUSED), \ + .flags = CLK_IGNORE_UNUSED, \ }, \ } @@ -117,7 +41,7 @@ static struct aoclk_cec_32k cec_32k_ao = { }, }; -static unsigned int gxbb_aoclk_reset[] = { +static const unsigned int gxbb_aoclk_reset[] = { [RESET_AO_REMOTE] = 16, [RESET_AO_I2C_MASTER] = 18, [RESET_AO_I2C_SLAVE] = 19, @@ -135,7 +59,7 @@ static struct clk_regmap *gxbb_aoclk_gate[] = { [CLKID_AO_IR_BLASTER] = &ir_blaster_ao, }; -static struct clk_hw_onecell_data gxbb_aoclk_onecell_data = { +static const struct clk_hw_onecell_data gxbb_aoclk_onecell_data = { .hws = { [CLKID_AO_REMOTE] = &remote_ao.hw, [CLKID_AO_I2C_MASTER] = &i2c_master_ao.hw, @@ -145,58 +69,55 @@ static struct clk_hw_onecell_data gxbb_aoclk_onecell_data = { [CLKID_AO_IR_BLASTER] = &ir_blaster_ao.hw, [CLKID_AO_CEC_32K] = &cec_32k_ao.hw, }, - .num = 7, + .num = NR_CLKS, }; -static int gxbb_aoclkc_probe(struct platform_device *pdev) +static int gxbb_register_cec_ao_32k(struct platform_device *pdev) { - struct gxbb_aoclk_reset_controller *rstc; struct device *dev = &pdev->dev; struct regmap *regmap; - int ret, clkid; - - rstc = devm_kzalloc(dev, sizeof(*rstc), GFP_KERNEL); - if (!rstc) - return -ENOMEM; + int ret; regmap = syscon_node_to_regmap(of_get_parent(dev->of_node)); if (IS_ERR(regmap)) { dev_err(dev, "failed to get regmap\n"); - return -ENODEV; - } - - /* Reset Controller */ - rstc->regmap = regmap; - rstc->data = gxbb_aoclk_reset; - rstc->reset.ops = &gxbb_aoclk_reset_ops; - rstc->reset.nr_resets = ARRAY_SIZE(gxbb_aoclk_reset); - rstc->reset.of_node = dev->of_node; - ret = devm_reset_controller_register(dev, &rstc->reset); - - /* - * Populate regmap and register all clks - */ - for (clkid = 0; clkid < ARRAY_SIZE(gxbb_aoclk_gate); clkid++) { - gxbb_aoclk_gate[clkid]->map = regmap; - - ret = devm_clk_hw_register(dev, - gxbb_aoclk_onecell_data.hws[clkid]); - if (ret) - return ret; + return PTR_ERR(regmap); } /* Specific clocks */ cec_32k_ao.regmap = regmap; ret = devm_clk_hw_register(dev, &cec_32k_ao.hw); + if (ret) { + dev_err(&pdev->dev, "clk cec_32k_ao register failed.\n"); + return ret; + } + + return 0; +} + +static const struct meson_aoclk_data gxbb_aoclkc_data = { + .reset_reg = AO_RTI_GEN_CNTL_REG0, + .num_reset = ARRAY_SIZE(gxbb_aoclk_reset), + .reset = gxbb_aoclk_reset, + .num_clks = ARRAY_SIZE(gxbb_aoclk_gate), + .clks = gxbb_aoclk_gate, + .hw_data = &gxbb_aoclk_onecell_data, +}; + +static int gxbb_aoclkc_probe(struct platform_device *pdev) +{ + int ret = gxbb_register_cec_ao_32k(pdev); if (ret) return ret; - return of_clk_add_hw_provider(dev->of_node, of_clk_hw_onecell_get, - &gxbb_aoclk_onecell_data); + return meson_aoclkc_probe(pdev); } static const struct of_device_id gxbb_aoclkc_match_table[] = { - { .compatible = "amlogic,meson-gx-aoclkc" }, + { + .compatible = "amlogic,meson-gx-aoclkc", + .data = &gxbb_aoclkc_data, + }, { } }; diff --git a/drivers/clk/meson/gxbb-aoclk.h b/drivers/clk/meson/gxbb-aoclk.h index badc4c22b4ee..c514493d989a 100644 --- a/drivers/clk/meson/gxbb-aoclk.h +++ b/drivers/clk/meson/gxbb-aoclk.h @@ -1,13 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0+ */ /* * Copyright (c) 2017 BayLibre, SAS * Author: Neil Armstrong - * - * SPDX-License-Identifier: GPL-2.0+ */ #ifndef __GXBB_AOCLKC_H #define __GXBB_AOCLKC_H +#define NR_CLKS 7 + /* AO Configuration Clock registers offsets */ #define AO_RTI_PWR_CNTL_REG1 0x0c #define AO_RTI_PWR_CNTL_REG0 0x10 @@ -26,4 +27,7 @@ struct aoclk_cec_32k { extern const struct clk_ops meson_aoclk_cec_32k_ops; +#include +#include + #endif /* __GXBB_AOCLKC_H */ diff --git a/drivers/clk/meson/gxbb.c b/drivers/clk/meson/gxbb.c index b1e4d9557610..240658404367 100644 --- a/drivers/clk/meson/gxbb.c +++ b/drivers/clk/meson/gxbb.c @@ -1,20 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0 /* - * AmLogic S905 / GXBB Clock Controller Driver - * * Copyright (c) 2016 AmLogic, Inc. * Michael Turquette - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include @@ -1543,6 +1530,102 @@ static struct clk_regmap gxbb_vapb = { }, }; +/* VDEC clocks */ + +static const char * const gxbb_vdec_parent_names[] = { + "fclk_div4", "fclk_div3", "fclk_div5", "fclk_div7" +}; + +static struct clk_regmap gxbb_vdec_1_sel = { + .data = &(struct clk_regmap_mux_data){ + .offset = HHI_VDEC_CLK_CNTL, + .mask = 0x3, + .shift = 9, + .flags = CLK_MUX_ROUND_CLOSEST, + }, + .hw.init = &(struct clk_init_data){ + .name = "vdec_1_sel", + .ops = &clk_regmap_mux_ops, + .parent_names = gxbb_vdec_parent_names, + .num_parents = ARRAY_SIZE(gxbb_vdec_parent_names), + .flags = CLK_SET_RATE_PARENT, + }, +}; + +static struct clk_regmap gxbb_vdec_1_div = { + .data = &(struct clk_regmap_div_data){ + .offset = HHI_VDEC_CLK_CNTL, + .shift = 0, + .width = 7, + }, + .hw.init = &(struct clk_init_data){ + .name = "vdec_1_div", + .ops = &clk_regmap_divider_ops, + .parent_names = (const char *[]){ "vdec_1_sel" }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + }, +}; + +static struct clk_regmap gxbb_vdec_1 = { + .data = &(struct clk_regmap_gate_data){ + .offset = HHI_VDEC_CLK_CNTL, + .bit_idx = 8, + }, + .hw.init = &(struct clk_init_data) { + .name = "vdec_1", + .ops = &clk_regmap_gate_ops, + .parent_names = (const char *[]){ "vdec_1_div" }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + }, +}; + +static struct clk_regmap gxbb_vdec_hevc_sel = { + .data = &(struct clk_regmap_mux_data){ + .offset = HHI_VDEC2_CLK_CNTL, + .mask = 0x3, + .shift = 25, + .flags = CLK_MUX_ROUND_CLOSEST, + }, + .hw.init = &(struct clk_init_data){ + .name = "vdec_hevc_sel", + .ops = &clk_regmap_mux_ops, + .parent_names = gxbb_vdec_parent_names, + .num_parents = ARRAY_SIZE(gxbb_vdec_parent_names), + .flags = CLK_SET_RATE_PARENT, + }, +}; + +static struct clk_regmap gxbb_vdec_hevc_div = { + .data = &(struct clk_regmap_div_data){ + .offset = HHI_VDEC2_CLK_CNTL, + .shift = 16, + .width = 7, + }, + .hw.init = &(struct clk_init_data){ + .name = "vdec_hevc_div", + .ops = &clk_regmap_divider_ops, + .parent_names = (const char *[]){ "vdec_hevc_sel" }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + }, +}; + +static struct clk_regmap gxbb_vdec_hevc = { + .data = &(struct clk_regmap_gate_data){ + .offset = HHI_VDEC2_CLK_CNTL, + .bit_idx = 24, + }, + .hw.init = &(struct clk_init_data) { + .name = "vdec_hevc", + .ops = &clk_regmap_gate_ops, + .parent_names = (const char *[]){ "vdec_hevc_div" }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + }, +}; + /* Everything Else (EE) domain gates */ static MESON_GATE(gxbb_ddr, HHI_GCLK_MPEG0, 0); static MESON_GATE(gxbb_dos, HHI_GCLK_MPEG0, 1); @@ -1786,6 +1869,12 @@ static struct clk_hw_onecell_data gxbb_hw_onecell_data = { [CLKID_FCLK_DIV4_DIV] = &gxbb_fclk_div4_div.hw, [CLKID_FCLK_DIV5_DIV] = &gxbb_fclk_div5_div.hw, [CLKID_FCLK_DIV7_DIV] = &gxbb_fclk_div7_div.hw, + [CLKID_VDEC_1_SEL] = &gxbb_vdec_1_sel.hw, + [CLKID_VDEC_1_DIV] = &gxbb_vdec_1_div.hw, + [CLKID_VDEC_1] = &gxbb_vdec_1.hw, + [CLKID_VDEC_HEVC_SEL] = &gxbb_vdec_hevc_sel.hw, + [CLKID_VDEC_HEVC_DIV] = &gxbb_vdec_hevc_div.hw, + [CLKID_VDEC_HEVC] = &gxbb_vdec_hevc.hw, [NR_CLKS] = NULL, }, .num = NR_CLKS, @@ -1942,6 +2031,12 @@ static struct clk_hw_onecell_data gxl_hw_onecell_data = { [CLKID_FCLK_DIV4_DIV] = &gxbb_fclk_div4_div.hw, [CLKID_FCLK_DIV5_DIV] = &gxbb_fclk_div5_div.hw, [CLKID_FCLK_DIV7_DIV] = &gxbb_fclk_div7_div.hw, + [CLKID_VDEC_1_SEL] = &gxbb_vdec_1_sel.hw, + [CLKID_VDEC_1_DIV] = &gxbb_vdec_1_div.hw, + [CLKID_VDEC_1] = &gxbb_vdec_1.hw, + [CLKID_VDEC_HEVC_SEL] = &gxbb_vdec_hevc_sel.hw, + [CLKID_VDEC_HEVC_DIV] = &gxbb_vdec_hevc_div.hw, + [CLKID_VDEC_HEVC] = &gxbb_vdec_hevc.hw, [NR_CLKS] = NULL, }, .num = NR_CLKS, @@ -2100,6 +2195,12 @@ static struct clk_regmap *const gx_clk_regmaps[] = { &gxbb_fclk_div4, &gxbb_fclk_div5, &gxbb_fclk_div7, + &gxbb_vdec_1_sel, + &gxbb_vdec_1_div, + &gxbb_vdec_1, + &gxbb_vdec_hevc_sel, + &gxbb_vdec_hevc_div, + &gxbb_vdec_hevc, }; struct clkc_data { diff --git a/drivers/clk/meson/gxbb.h b/drivers/clk/meson/gxbb.h index 9febf3f03739..ec1a812bf1fd 100644 --- a/drivers/clk/meson/gxbb.h +++ b/drivers/clk/meson/gxbb.h @@ -1,57 +1,7 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */ /* - * This file is provided under a dual BSD/GPLv2 license. When using or - * redistributing this file, you may do so under either license. - * - * GPL LICENSE SUMMARY - * * Copyright (c) 2016 AmLogic, Inc. * Author: Michael Turquette - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of version 2 of the GNU General Public License as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA. - * The full GNU General Public License is included in this distribution - * in the file called COPYING - * - * BSD LICENSE - * - * Copyright (c) 2016 BayLibre, Inc. - * Author: Michael Turquette - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * * Neither the name of Intel Corporation nor the names of its - * contributors may be used to endorse or promote products derived - * from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #ifndef __GXBB_H @@ -204,8 +154,12 @@ #define CLKID_FCLK_DIV4_DIV 148 #define CLKID_FCLK_DIV5_DIV 149 #define CLKID_FCLK_DIV7_DIV 150 +#define CLKID_VDEC_1_SEL 151 +#define CLKID_VDEC_1_DIV 152 +#define CLKID_VDEC_HEVC_SEL 154 +#define CLKID_VDEC_HEVC_DIV 155 -#define NR_CLKS 151 +#define NR_CLKS 157 /* include the CLKIDs that have been made part of the DT binding */ #include diff --git a/drivers/clk/meson/meson-aoclk.c b/drivers/clk/meson/meson-aoclk.c new file mode 100644 index 000000000000..f965845917e3 --- /dev/null +++ b/drivers/clk/meson/meson-aoclk.c @@ -0,0 +1,81 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * Amlogic Meson-AXG Clock Controller Driver + * + * Copyright (c) 2016 BayLibre, SAS. + * Author: Neil Armstrong + * + * Copyright (c) 2018 Amlogic, inc. + * Author: Qiufang Dai + * Author: Yixun Lan + */ + +#include +#include +#include +#include +#include "clk-regmap.h" +#include "meson-aoclk.h" + +static int meson_aoclk_do_reset(struct reset_controller_dev *rcdev, + unsigned long id) +{ + struct meson_aoclk_reset_controller *rstc = + container_of(rcdev, struct meson_aoclk_reset_controller, reset); + + return regmap_write(rstc->regmap, rstc->data->reset_reg, + BIT(rstc->data->reset[id])); +} + +static const struct reset_control_ops meson_aoclk_reset_ops = { + .reset = meson_aoclk_do_reset, +}; + +int meson_aoclkc_probe(struct platform_device *pdev) +{ + struct meson_aoclk_reset_controller *rstc; + struct meson_aoclk_data *data; + struct device *dev = &pdev->dev; + struct regmap *regmap; + int ret, clkid; + + data = (struct meson_aoclk_data *) of_device_get_match_data(dev); + if (!data) + return -ENODEV; + + rstc = devm_kzalloc(dev, sizeof(*rstc), GFP_KERNEL); + if (!rstc) + return -ENOMEM; + + regmap = syscon_node_to_regmap(of_get_parent(dev->of_node)); + if (IS_ERR(regmap)) { + dev_err(dev, "failed to get regmap\n"); + return PTR_ERR(regmap); + } + + /* Reset Controller */ + rstc->data = data; + rstc->regmap = regmap; + rstc->reset.ops = &meson_aoclk_reset_ops; + rstc->reset.nr_resets = data->num_reset, + rstc->reset.of_node = dev->of_node; + ret = devm_reset_controller_register(dev, &rstc->reset); + if (ret) { + dev_err(dev, "failed to register reset controller\n"); + return ret; + } + + /* + * Populate regmap and register all clks + */ + for (clkid = 0; clkid < data->num_clks; clkid++) { + data->clks[clkid]->map = regmap; + + ret = devm_clk_hw_register(dev, data->hw_data->hws[clkid]); + if (ret) + return ret; + } + + return devm_of_clk_add_hw_provider(dev, of_clk_hw_onecell_get, + (void *) data->hw_data); +} diff --git a/drivers/clk/meson/meson-aoclk.h b/drivers/clk/meson/meson-aoclk.h new file mode 100644 index 000000000000..ab2819e88922 --- /dev/null +++ b/drivers/clk/meson/meson-aoclk.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: (GPL-2.0+ OR MIT) */ +/* + * Copyright (c) 2017 BayLibre, SAS + * Author: Neil Armstrong + * + * Copyright (c) 2018 Amlogic, inc. + * Author: Qiufang Dai + * Author: Yixun Lan + */ + +#ifndef __MESON_AOCLK_H__ +#define __MESON_AOCLK_H__ + +#include +#include +#include "clk-regmap.h" + +struct meson_aoclk_data { + const unsigned int reset_reg; + const int num_reset; + const unsigned int *reset; + int num_clks; + struct clk_regmap **clks; + const struct clk_hw_onecell_data *hw_data; +}; + +struct meson_aoclk_reset_controller { + struct reset_controller_dev reset; + const struct meson_aoclk_data *data; + struct regmap *regmap; +}; + +int meson_aoclkc_probe(struct platform_device *pdev); +#endif diff --git a/drivers/clk/meson/meson8b.c b/drivers/clk/meson/meson8b.c index d0524ec71aad..7447d96a265f 100644 --- a/drivers/clk/meson/meson8b.c +++ b/drivers/clk/meson/meson8b.c @@ -1,24 +1,10 @@ +// SPDX-License-Identifier: GPL-2.0 /* - * AmLogic S802 (Meson8) / S805 (Meson8b) / S812 (Meson8m2) Clock Controller - * Driver - * * Copyright (c) 2015 Endless Mobile, Inc. * Author: Carlo Caione * * Copyright (c) 2016 BayLibre, Inc. * Michael Turquette - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include @@ -246,6 +232,13 @@ static struct clk_regmap meson8b_fclk_div2 = { .ops = &clk_regmap_gate_ops, .parent_names = (const char *[]){ "fclk_div2_div" }, .num_parents = 1, + /* + * FIXME: Ethernet with a RGMII PHYs is not working if + * fclk_div2 is disabled. it is currently unclear why this + * is. keep it enabled until the Ethernet driver knows how + * to manage this clock. + */ + .flags = CLK_IS_CRITICAL, }, }; @@ -640,6 +633,54 @@ static struct clk_regmap meson8b_cpu_clk = { }, }; +static struct clk_regmap meson8b_nand_clk_sel = { + .data = &(struct clk_regmap_mux_data){ + .offset = HHI_NAND_CLK_CNTL, + .mask = 0x7, + .shift = 9, + .flags = CLK_MUX_ROUND_CLOSEST, + }, + .hw.init = &(struct clk_init_data){ + .name = "nand_clk_sel", + .ops = &clk_regmap_mux_ops, + /* FIXME all other parents are unknown: */ + .parent_names = (const char *[]){ "fclk_div4", "fclk_div3", + "fclk_div5", "fclk_div7", "xtal" }, + .num_parents = 5, + .flags = CLK_SET_RATE_PARENT, + }, +}; + +static struct clk_regmap meson8b_nand_clk_div = { + .data = &(struct clk_regmap_div_data){ + .offset = HHI_NAND_CLK_CNTL, + .shift = 0, + .width = 7, + .flags = CLK_DIVIDER_ROUND_CLOSEST, + }, + .hw.init = &(struct clk_init_data){ + .name = "nand_clk_div", + .ops = &clk_regmap_divider_ops, + .parent_names = (const char *[]){ "nand_clk_sel" }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + }, +}; + +static struct clk_regmap meson8b_nand_clk_gate = { + .data = &(struct clk_regmap_gate_data){ + .offset = HHI_NAND_CLK_CNTL, + .bit_idx = 8, + }, + .hw.init = &(struct clk_init_data){ + .name = "nand_clk_gate", + .ops = &clk_regmap_gate_ops, + .parent_names = (const char *[]){ "nand_clk_div" }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + }, +}; + /* Everything Else (EE) domain gates */ static MESON_GATE(meson8b_ddr, HHI_GCLK_MPEG0, 0); @@ -835,6 +876,9 @@ static struct clk_hw_onecell_data meson8b_hw_onecell_data = { [CLKID_FCLK_DIV4_DIV] = &meson8b_fclk_div4_div.hw, [CLKID_FCLK_DIV5_DIV] = &meson8b_fclk_div5_div.hw, [CLKID_FCLK_DIV7_DIV] = &meson8b_fclk_div7_div.hw, + [CLKID_NAND_SEL] = &meson8b_nand_clk_sel.hw, + [CLKID_NAND_DIV] = &meson8b_nand_clk_div.hw, + [CLKID_NAND_CLK] = &meson8b_nand_clk_gate.hw, [CLK_NR_CLKS] = NULL, }, .num = CLK_NR_CLKS, @@ -940,6 +984,9 @@ static struct clk_regmap *const meson8b_clk_regmaps[] = { &meson8b_fclk_div4, &meson8b_fclk_div5, &meson8b_fclk_div7, + &meson8b_nand_clk_sel, + &meson8b_nand_clk_div, + &meson8b_nand_clk_gate, }; static const struct meson8b_clk_reset_line { diff --git a/drivers/clk/meson/meson8b.h b/drivers/clk/meson/meson8b.h index 6e414bd36981..5d09412b5084 100644 --- a/drivers/clk/meson/meson8b.h +++ b/drivers/clk/meson/meson8b.h @@ -1,21 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (c) 2015 Endless Mobile, Inc. * Author: Carlo Caione * * Copyright (c) 2016 BayLibre, Inc. * Michael Turquette - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #ifndef __MESON8B_H @@ -40,6 +29,7 @@ #define HHI_VID_CLK_CNTL 0x17c /* 0x5f offset in data sheet */ #define HHI_VID_DIVIDER_CNTL 0x198 /* 0x66 offset in data sheet */ #define HHI_SYS_CPU_CLK_CNTL0 0x19c /* 0x67 offset in data sheet */ +#define HHI_NAND_CLK_CNTL 0x25c /* 0x97 offset in data sheet */ #define HHI_MPLL_CNTL 0x280 /* 0xa0 offset in data sheet */ #define HHI_SYS_PLL_CNTL 0x300 /* 0xc0 offset in data sheet */ #define HHI_VID_PLL_CNTL 0x320 /* 0xc8 offset in data sheet */ @@ -83,8 +73,10 @@ #define CLKID_FCLK_DIV4_DIV 107 #define CLKID_FCLK_DIV5_DIV 108 #define CLKID_FCLK_DIV7_DIV 109 +#define CLKID_NAND_SEL 110 +#define CLKID_NAND_DIV 111 -#define CLK_NR_CLKS 110 +#define CLK_NR_CLKS 113 /* * include the CLKID and RESETID that have diff --git a/drivers/clk/mvebu/armada-37xx-periph.c b/drivers/clk/mvebu/armada-37xx-periph.c index 87213ea7fc84..6860bd5a37c5 100644 --- a/drivers/clk/mvebu/armada-37xx-periph.c +++ b/drivers/clk/mvebu/armada-37xx-periph.c @@ -667,9 +667,10 @@ static int armada_3700_periph_clock_probe(struct platform_device *pdev) if (!driver_data) return -ENOMEM; - driver_data->hw_data = devm_kzalloc(dev, sizeof(*driver_data->hw_data) + - sizeof(*driver_data->hw_data->hws) * num_periph, - GFP_KERNEL); + driver_data->hw_data = devm_kzalloc(dev, + struct_size(driver_data->hw_data, + hws, num_periph), + GFP_KERNEL); if (!driver_data->hw_data) return -ENOMEM; driver_data->hw_data->num = num_periph; diff --git a/drivers/clk/mvebu/armada-37xx-tbg.c b/drivers/clk/mvebu/armada-37xx-tbg.c index aa80db11f543..7ff041f73b55 100644 --- a/drivers/clk/mvebu/armada-37xx-tbg.c +++ b/drivers/clk/mvebu/armada-37xx-tbg.c @@ -91,8 +91,8 @@ static int armada_3700_tbg_clock_probe(struct platform_device *pdev) void __iomem *reg; int i, ret; - hw_tbg_data = devm_kzalloc(&pdev->dev, sizeof(*hw_tbg_data) - + sizeof(*hw_tbg_data->hws) * NUM_TBG, + hw_tbg_data = devm_kzalloc(&pdev->dev, + struct_size(hw_tbg_data, hws, NUM_TBG), GFP_KERNEL); if (!hw_tbg_data) return -ENOMEM; diff --git a/drivers/clk/mvebu/clk-corediv.c b/drivers/clk/mvebu/clk-corediv.c index 8491979f4096..68f05c53d40e 100644 --- a/drivers/clk/mvebu/clk-corediv.c +++ b/drivers/clk/mvebu/clk-corediv.c @@ -72,7 +72,7 @@ static const struct clk_corediv_desc mvebu_corediv_desc[] = { }; static const struct clk_corediv_desc mv98dx3236_corediv_desc[] = { - { .mask = 0x0f, .offset = 6, .fieldbit = 26 }, /* NAND clock */ + { .mask = 0x0f, .offset = 6, .fieldbit = 27 }, /* NAND clock */ }; #define to_corediv_clk(p) container_of(p, struct clk_corediv, hw) diff --git a/drivers/clk/qcom/Kconfig b/drivers/clk/qcom/Kconfig index fbf4532f94b8..9c3480dcc38a 100644 --- a/drivers/clk/qcom/Kconfig +++ b/drivers/clk/qcom/Kconfig @@ -218,6 +218,33 @@ config MSM_MMCC_8996 Say Y if you want to support multimedia devices such as display, graphics, video encode/decode, camera, etc. +config MSM_GCC_8998 + tristate "MSM8998 Global Clock Controller" + depends on COMMON_CLK_QCOM + help + Support for the global clock controller on msm8998 devices. + Say Y if you want to use peripheral devices such as UART, SPI, + i2c, USB, UFS, SD/eMMC, PCIe, etc. + +config SDM_GCC_845 + tristate "SDM845 Global Clock Controller" + select QCOM_GDSC + depends on COMMON_CLK_QCOM + help + Support for the global clock controller on SDM845 devices. + Say Y if you want to use peripheral devices such as UART, SPI, + i2C, USB, UFS, SDDC, PCIe, etc. + +config SDM_VIDEOCC_845 + tristate "SDM845 Video Clock Controller" + depends on COMMON_CLK_QCOM + select SDM_GCC_845 + select QCOM_GDSC + help + Support for the video clock controller on SDM845 devices. + Say Y if you want to support video devices and functionality such as + video encode and decode. + config SPMI_PMIC_CLKDIV tristate "SPMI PMIC clkdiv Support" depends on (COMMON_CLK_QCOM && SPMI) || COMPILE_TEST diff --git a/drivers/clk/qcom/Makefile b/drivers/clk/qcom/Makefile index 230332cf317e..762c01137c2f 100644 --- a/drivers/clk/qcom/Makefile +++ b/drivers/clk/qcom/Makefile @@ -30,6 +30,7 @@ obj-$(CONFIG_MSM_GCC_8974) += gcc-msm8974.o obj-$(CONFIG_MSM_GCC_8994) += gcc-msm8994.o obj-$(CONFIG_MSM_GCC_8996) += gcc-msm8996.o obj-$(CONFIG_MSM_LCC_8960) += lcc-msm8960.o +obj-$(CONFIG_MSM_GCC_8998) += gcc-msm8998.o obj-$(CONFIG_MSM_MMCC_8960) += mmcc-msm8960.o obj-$(CONFIG_MSM_MMCC_8974) += mmcc-msm8974.o obj-$(CONFIG_MSM_MMCC_8996) += mmcc-msm8996.o @@ -37,4 +38,6 @@ obj-$(CONFIG_QCOM_A53PLL) += a53-pll.o obj-$(CONFIG_QCOM_CLK_APCS_MSM8916) += apcs-msm8916.o obj-$(CONFIG_QCOM_CLK_RPM) += clk-rpm.o obj-$(CONFIG_QCOM_CLK_SMD_RPM) += clk-smd-rpm.o +obj-$(CONFIG_SDM_GCC_845) += gcc-sdm845.o +obj-$(CONFIG_SDM_VIDEOCC_845) += videocc-sdm845.o obj-$(CONFIG_SPMI_PMIC_CLKDIV) += clk-spmi-pmic-div.o diff --git a/drivers/clk/qcom/clk-alpha-pll.c b/drivers/clk/qcom/clk-alpha-pll.c index 6d04cd96482a..3c49a60072f1 100644 --- a/drivers/clk/qcom/clk-alpha-pll.c +++ b/drivers/clk/qcom/clk-alpha-pll.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2015, The Linux Foundation. All rights reserved. + * Copyright (c) 2015, 2018, The Linux Foundation. All rights reserved. * * This software is licensed under the terms of the GNU General Public * License version 2, as published by the Free Software Foundation, and @@ -58,6 +58,8 @@ #define PLL_TEST_CTL(p) ((p)->offset + (p)->regs[PLL_OFF_TEST_CTL]) #define PLL_TEST_CTL_U(p) ((p)->offset + (p)->regs[PLL_OFF_TEST_CTL_U]) #define PLL_STATUS(p) ((p)->offset + (p)->regs[PLL_OFF_STATUS]) +#define PLL_OPMODE(p) ((p)->offset + (p)->regs[PLL_OFF_OPMODE]) +#define PLL_FRAC(p) ((p)->offset + (p)->regs[PLL_OFF_FRAC]) const u8 clk_alpha_pll_regs[][PLL_OFF_MAX_REGS] = { [CLK_ALPHA_PLL_TYPE_DEFAULT] = { @@ -90,6 +92,18 @@ const u8 clk_alpha_pll_regs[][PLL_OFF_MAX_REGS] = { [PLL_OFF_TEST_CTL] = 0x1c, [PLL_OFF_STATUS] = 0x24, }, + [CLK_ALPHA_PLL_TYPE_FABIA] = { + [PLL_OFF_L_VAL] = 0x04, + [PLL_OFF_USER_CTL] = 0x0c, + [PLL_OFF_USER_CTL_U] = 0x10, + [PLL_OFF_CONFIG_CTL] = 0x14, + [PLL_OFF_CONFIG_CTL_U] = 0x18, + [PLL_OFF_TEST_CTL] = 0x1c, + [PLL_OFF_TEST_CTL_U] = 0x20, + [PLL_OFF_STATUS] = 0x24, + [PLL_OFF_OPMODE] = 0x2c, + [PLL_OFF_FRAC] = 0x38, + }, }; EXPORT_SYMBOL_GPL(clk_alpha_pll_regs); @@ -108,6 +122,12 @@ EXPORT_SYMBOL_GPL(clk_alpha_pll_regs); #define PLL_HUAYRA_N_MASK 0xff #define PLL_HUAYRA_ALPHA_WIDTH 16 +#define FABIA_OPMODE_STANDBY 0x0 +#define FABIA_OPMODE_RUN 0x1 + +#define FABIA_PLL_OUT_MASK 0x7 +#define FABIA_PLL_RATE_MARGIN 500 + #define pll_alpha_width(p) \ ((PLL_ALPHA_VAL_U(p) - PLL_ALPHA_VAL(p) == 4) ? \ ALPHA_REG_BITWIDTH : ALPHA_REG_16BIT_WIDTH) @@ -441,16 +461,12 @@ clk_alpha_pll_recalc_rate(struct clk_hw *hw, unsigned long parent_rate) return alpha_pll_calc_rate(prate, l, a, alpha_width); } -static int clk_alpha_pll_update_latch(struct clk_alpha_pll *pll, - int (*is_enabled)(struct clk_hw *)) + +static int __clk_alpha_pll_update_latch(struct clk_alpha_pll *pll) { int ret; u32 mode; - if (!is_enabled(&pll->clkr.hw) || - !(pll->flags & SUPPORTS_DYNAMIC_UPDATE)) - return 0; - regmap_read(pll->clkr.regmap, PLL_MODE(pll), &mode); /* Latch the input to the PLL */ @@ -489,6 +505,16 @@ static int clk_alpha_pll_update_latch(struct clk_alpha_pll *pll, return 0; } +static int clk_alpha_pll_update_latch(struct clk_alpha_pll *pll, + int (*is_enabled)(struct clk_hw *)) +{ + if (!is_enabled(&pll->clkr.hw) || + !(pll->flags & SUPPORTS_DYNAMIC_UPDATE)) + return 0; + + return __clk_alpha_pll_update_latch(pll); +} + static int __clk_alpha_pll_set_rate(struct clk_hw *hw, unsigned long rate, unsigned long prate, int (*is_enabled)(struct clk_hw *)) @@ -832,3 +858,265 @@ const struct clk_ops clk_alpha_pll_postdiv_ro_ops = { .recalc_rate = clk_alpha_pll_postdiv_recalc_rate, }; EXPORT_SYMBOL_GPL(clk_alpha_pll_postdiv_ro_ops); + +void clk_fabia_pll_configure(struct clk_alpha_pll *pll, struct regmap *regmap, + const struct alpha_pll_config *config) +{ + u32 val, mask; + + if (config->l) + regmap_write(regmap, PLL_L_VAL(pll), config->l); + + if (config->alpha) + regmap_write(regmap, PLL_FRAC(pll), config->alpha); + + if (config->config_ctl_val) + regmap_write(regmap, PLL_CONFIG_CTL(pll), + config->config_ctl_val); + + if (config->post_div_mask) { + mask = config->post_div_mask; + val = config->post_div_val; + regmap_update_bits(regmap, PLL_USER_CTL(pll), mask, val); + } + + regmap_update_bits(regmap, PLL_MODE(pll), PLL_UPDATE_BYPASS, + PLL_UPDATE_BYPASS); + + regmap_update_bits(regmap, PLL_MODE(pll), PLL_RESET_N, PLL_RESET_N); +} +EXPORT_SYMBOL_GPL(clk_fabia_pll_configure); + +static int alpha_pll_fabia_enable(struct clk_hw *hw) +{ + int ret; + struct clk_alpha_pll *pll = to_clk_alpha_pll(hw); + u32 val, opmode_val; + struct regmap *regmap = pll->clkr.regmap; + + ret = regmap_read(regmap, PLL_MODE(pll), &val); + if (ret) + return ret; + + /* If in FSM mode, just vote for it */ + if (val & PLL_VOTE_FSM_ENA) { + ret = clk_enable_regmap(hw); + if (ret) + return ret; + return wait_for_pll_enable_active(pll); + } + + ret = regmap_read(regmap, PLL_OPMODE(pll), &opmode_val); + if (ret) + return ret; + + /* Skip If PLL is already running */ + if ((opmode_val & FABIA_OPMODE_RUN) && (val & PLL_OUTCTRL)) + return 0; + + ret = regmap_update_bits(regmap, PLL_MODE(pll), PLL_OUTCTRL, 0); + if (ret) + return ret; + + ret = regmap_write(regmap, PLL_OPMODE(pll), FABIA_OPMODE_STANDBY); + if (ret) + return ret; + + ret = regmap_update_bits(regmap, PLL_MODE(pll), PLL_RESET_N, + PLL_RESET_N); + if (ret) + return ret; + + ret = regmap_write(regmap, PLL_OPMODE(pll), FABIA_OPMODE_RUN); + if (ret) + return ret; + + ret = wait_for_pll_enable_lock(pll); + if (ret) + return ret; + + ret = regmap_update_bits(regmap, PLL_USER_CTL(pll), + FABIA_PLL_OUT_MASK, FABIA_PLL_OUT_MASK); + if (ret) + return ret; + + return regmap_update_bits(regmap, PLL_MODE(pll), PLL_OUTCTRL, + PLL_OUTCTRL); +} + +static void alpha_pll_fabia_disable(struct clk_hw *hw) +{ + int ret; + struct clk_alpha_pll *pll = to_clk_alpha_pll(hw); + u32 val; + struct regmap *regmap = pll->clkr.regmap; + + ret = regmap_read(regmap, PLL_MODE(pll), &val); + if (ret) + return; + + /* If in FSM mode, just unvote it */ + if (val & PLL_FSM_ENA) { + clk_disable_regmap(hw); + return; + } + + ret = regmap_update_bits(regmap, PLL_MODE(pll), PLL_OUTCTRL, 0); + if (ret) + return; + + /* Disable main outputs */ + ret = regmap_update_bits(regmap, PLL_USER_CTL(pll), FABIA_PLL_OUT_MASK, + 0); + if (ret) + return; + + /* Place the PLL in STANDBY */ + regmap_write(regmap, PLL_OPMODE(pll), FABIA_OPMODE_STANDBY); +} + +static unsigned long alpha_pll_fabia_recalc_rate(struct clk_hw *hw, + unsigned long parent_rate) +{ + struct clk_alpha_pll *pll = to_clk_alpha_pll(hw); + u32 l, frac, alpha_width = pll_alpha_width(pll); + + regmap_read(pll->clkr.regmap, PLL_L_VAL(pll), &l); + regmap_read(pll->clkr.regmap, PLL_FRAC(pll), &frac); + + return alpha_pll_calc_rate(parent_rate, l, frac, alpha_width); +} + +static int alpha_pll_fabia_set_rate(struct clk_hw *hw, unsigned long rate, + unsigned long prate) +{ + struct clk_alpha_pll *pll = to_clk_alpha_pll(hw); + u32 val, l, alpha_width = pll_alpha_width(pll); + u64 a; + unsigned long rrate; + int ret = 0; + + ret = regmap_read(pll->clkr.regmap, PLL_MODE(pll), &val); + if (ret) + return ret; + + rrate = alpha_pll_round_rate(rate, prate, &l, &a, alpha_width); + + /* + * Due to limited number of bits for fractional rate programming, the + * rounded up rate could be marginally higher than the requested rate. + */ + if (rrate > (rate + FABIA_PLL_RATE_MARGIN) || rrate < rate) { + pr_err("Call set rate on the PLL with rounded rates!\n"); + return -EINVAL; + } + + regmap_write(pll->clkr.regmap, PLL_L_VAL(pll), l); + regmap_write(pll->clkr.regmap, PLL_FRAC(pll), a); + + return __clk_alpha_pll_update_latch(pll); +} + +const struct clk_ops clk_alpha_pll_fabia_ops = { + .enable = alpha_pll_fabia_enable, + .disable = alpha_pll_fabia_disable, + .is_enabled = clk_alpha_pll_is_enabled, + .set_rate = alpha_pll_fabia_set_rate, + .recalc_rate = alpha_pll_fabia_recalc_rate, + .round_rate = clk_alpha_pll_round_rate, +}; +EXPORT_SYMBOL_GPL(clk_alpha_pll_fabia_ops); + +const struct clk_ops clk_alpha_pll_fixed_fabia_ops = { + .enable = alpha_pll_fabia_enable, + .disable = alpha_pll_fabia_disable, + .is_enabled = clk_alpha_pll_is_enabled, + .recalc_rate = alpha_pll_fabia_recalc_rate, + .round_rate = clk_alpha_pll_round_rate, +}; +EXPORT_SYMBOL_GPL(clk_alpha_pll_fixed_fabia_ops); + +static unsigned long clk_alpha_pll_postdiv_fabia_recalc_rate(struct clk_hw *hw, + unsigned long parent_rate) +{ + struct clk_alpha_pll_postdiv *pll = to_clk_alpha_pll_postdiv(hw); + u32 i, div = 1, val; + int ret; + + if (!pll->post_div_table) { + pr_err("Missing the post_div_table for the PLL\n"); + return -EINVAL; + } + + ret = regmap_read(pll->clkr.regmap, PLL_USER_CTL(pll), &val); + if (ret) + return ret; + + val >>= pll->post_div_shift; + val &= BIT(pll->width) - 1; + + for (i = 0; i < pll->num_post_div; i++) { + if (pll->post_div_table[i].val == val) { + div = pll->post_div_table[i].div; + break; + } + } + + return (parent_rate / div); +} + +static long clk_alpha_pll_postdiv_fabia_round_rate(struct clk_hw *hw, + unsigned long rate, unsigned long *prate) +{ + struct clk_alpha_pll_postdiv *pll = to_clk_alpha_pll_postdiv(hw); + + if (!pll->post_div_table) { + pr_err("Missing the post_div_table for the PLL\n"); + return -EINVAL; + } + + return divider_round_rate(hw, rate, prate, pll->post_div_table, + pll->width, CLK_DIVIDER_ROUND_CLOSEST); +} + +static int clk_alpha_pll_postdiv_fabia_set_rate(struct clk_hw *hw, + unsigned long rate, unsigned long parent_rate) +{ + struct clk_alpha_pll_postdiv *pll = to_clk_alpha_pll_postdiv(hw); + int i, val = 0, div, ret; + + /* + * If the PLL is in FSM mode, then treat set_rate callback as a + * no-operation. + */ + ret = regmap_read(pll->clkr.regmap, PLL_MODE(pll), &val); + if (ret) + return ret; + + if (val & PLL_VOTE_FSM_ENA) + return 0; + + if (!pll->post_div_table) { + pr_err("Missing the post_div_table for the PLL\n"); + return -EINVAL; + } + + div = DIV_ROUND_UP_ULL((u64)parent_rate, rate); + for (i = 0; i < pll->num_post_div; i++) { + if (pll->post_div_table[i].div == div) { + val = pll->post_div_table[i].val; + break; + } + } + + return regmap_update_bits(pll->clkr.regmap, PLL_USER_CTL(pll), + (BIT(pll->width) - 1) << pll->post_div_shift, + val << pll->post_div_shift); +} + +const struct clk_ops clk_alpha_pll_postdiv_fabia_ops = { + .recalc_rate = clk_alpha_pll_postdiv_fabia_recalc_rate, + .round_rate = clk_alpha_pll_postdiv_fabia_round_rate, + .set_rate = clk_alpha_pll_postdiv_fabia_set_rate, +}; +EXPORT_SYMBOL_GPL(clk_alpha_pll_postdiv_fabia_ops); diff --git a/drivers/clk/qcom/clk-alpha-pll.h b/drivers/clk/qcom/clk-alpha-pll.h index 7593e8a56cf2..f981b486c468 100644 --- a/drivers/clk/qcom/clk-alpha-pll.h +++ b/drivers/clk/qcom/clk-alpha-pll.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2015, The Linux Foundation. All rights reserved. + * Copyright (c) 2015, 2018, The Linux Foundation. All rights reserved. * * This software is licensed under the terms of the GNU General Public * License version 2, as published by the Free Software Foundation, and @@ -22,6 +22,7 @@ enum { CLK_ALPHA_PLL_TYPE_DEFAULT, CLK_ALPHA_PLL_TYPE_HUAYRA, CLK_ALPHA_PLL_TYPE_BRAMMO, + CLK_ALPHA_PLL_TYPE_FABIA, CLK_ALPHA_PLL_TYPE_MAX, }; @@ -36,6 +37,8 @@ enum { PLL_OFF_TEST_CTL, PLL_OFF_TEST_CTL_U, PLL_OFF_STATUS, + PLL_OFF_OPMODE, + PLL_OFF_FRAC, PLL_OFF_MAX_REGS }; @@ -73,6 +76,10 @@ struct clk_alpha_pll { * @offset: base address of registers * @regs: alpha pll register map (see @clk_alpha_pll_regs) * @width: width of post-divider + * @post_div_shift: shift to differentiate between odd & even post-divider + * @post_div_table: table with PLL odd and even post-divider settings + * @num_post_div: Number of PLL post-divider settings + * * @clkr: regmap clock handle */ struct clk_alpha_pll_postdiv { @@ -81,6 +88,9 @@ struct clk_alpha_pll_postdiv { const u8 *regs; struct clk_regmap clkr; + int post_div_shift; + const struct clk_div_table *post_div_table; + size_t num_post_div; }; struct alpha_pll_config { @@ -109,7 +119,13 @@ extern const struct clk_ops clk_alpha_pll_postdiv_ops; extern const struct clk_ops clk_alpha_pll_huayra_ops; extern const struct clk_ops clk_alpha_pll_postdiv_ro_ops; +extern const struct clk_ops clk_alpha_pll_fabia_ops; +extern const struct clk_ops clk_alpha_pll_fixed_fabia_ops; +extern const struct clk_ops clk_alpha_pll_postdiv_fabia_ops; + void clk_alpha_pll_configure(struct clk_alpha_pll *pll, struct regmap *regmap, const struct alpha_pll_config *config); +void clk_fabia_pll_configure(struct clk_alpha_pll *pll, struct regmap *regmap, + const struct alpha_pll_config *config); #endif diff --git a/drivers/clk/qcom/clk-branch.c b/drivers/clk/qcom/clk-branch.c index 26f7af315066..c58c5538b1b6 100644 --- a/drivers/clk/qcom/clk-branch.c +++ b/drivers/clk/qcom/clk-branch.c @@ -77,8 +77,11 @@ static int clk_branch_wait(const struct clk_branch *br, bool enabling, bool voted = br->halt_check & BRANCH_VOTED; const char *name = clk_hw_get_name(&br->clkr.hw); - /* Skip checking halt bit if the clock is in hardware gated mode */ - if (clk_branch_in_hwcg_mode(br)) + /* + * Skip checking halt bit if we're explicitly ignoring the bit or the + * clock is in hardware gated mode + */ + if (br->halt_check == BRANCH_HALT_SKIP || clk_branch_in_hwcg_mode(br)) return 0; if (br->halt_check == BRANCH_HALT_DELAY || (!enabling && voted)) { diff --git a/drivers/clk/qcom/clk-branch.h b/drivers/clk/qcom/clk-branch.h index 284df3f3c55f..1702efb1c511 100644 --- a/drivers/clk/qcom/clk-branch.h +++ b/drivers/clk/qcom/clk-branch.h @@ -42,6 +42,7 @@ struct clk_branch { #define BRANCH_HALT_ENABLE 1 /* pol: 0 = halt */ #define BRANCH_HALT_ENABLE_VOTED (BRANCH_HALT_ENABLE | BRANCH_VOTED) #define BRANCH_HALT_DELAY 2 /* No bit to check; just delay */ +#define BRANCH_HALT_SKIP 3 /* Don't check halt bit */ struct clk_regmap clkr; }; diff --git a/drivers/clk/qcom/clk-rcg.h b/drivers/clk/qcom/clk-rcg.h index 2a7489a84e69..b209a2fe86b9 100644 --- a/drivers/clk/qcom/clk-rcg.h +++ b/drivers/clk/qcom/clk-rcg.h @@ -1,15 +1,5 @@ -/* - * Copyright (c) 2013, The Linux Foundation. All rights reserved. - * - * This software is licensed under the terms of the GNU General Public - * License version 2, as published by the Free Software Foundation, and - * may be copied, distributed, and modified under those terms. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - */ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2013, 2018, The Linux Foundation. All rights reserved. */ #ifndef __QCOM_CLK_RCG_H__ #define __QCOM_CLK_RCG_H__ @@ -144,6 +134,7 @@ extern const struct clk_ops clk_dyn_rcg_ops; * @cmd_rcgr: corresponds to *_CMD_RCGR * @mnd_width: number of bits in m/n/d values * @hid_width: number of bits in half integer divider + * @safe_src_index: safe src index value * @parent_map: map from software's parent index to hardware's src_sel field * @freq_tbl: frequency table * @clkr: regmap clock handle @@ -153,6 +144,7 @@ struct clk_rcg2 { u32 cmd_rcgr; u8 mnd_width; u8 hid_width; + u8 safe_src_index; const struct parent_map *parent_map; const struct freq_tbl *freq_tbl; struct clk_regmap clkr; @@ -167,5 +159,6 @@ extern const struct clk_ops clk_byte_ops; extern const struct clk_ops clk_byte2_ops; extern const struct clk_ops clk_pixel_ops; extern const struct clk_ops clk_gfx3d_ops; +extern const struct clk_ops clk_rcg2_shared_ops; #endif diff --git a/drivers/clk/qcom/clk-rcg2.c b/drivers/clk/qcom/clk-rcg2.c index bbeaf9c09dbb..52208d4165f4 100644 --- a/drivers/clk/qcom/clk-rcg2.c +++ b/drivers/clk/qcom/clk-rcg2.c @@ -1,14 +1,6 @@ +// SPDX-License-Identifier: GPL-2.0 /* - * Copyright (c) 2013, The Linux Foundation. All rights reserved. - * - * This software is licensed under the terms of the GNU General Public - * License version 2, as published by the Free Software Foundation, and - * may be copied, distributed, and modified under those terms. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. + * Copyright (c) 2013, 2018, The Linux Foundation. All rights reserved. */ #include @@ -42,6 +34,7 @@ #define CFG_MODE_SHIFT 12 #define CFG_MODE_MASK (0x3 << CFG_MODE_SHIFT) #define CFG_MODE_DUAL_EDGE (0x2 << CFG_MODE_SHIFT) +#define CFG_HW_CLK_CTRL_MASK BIT(20) #define M_REG 0x8 #define N_REG 0xc @@ -211,6 +204,7 @@ static int _freq_tbl_determine_rate(struct clk_hw *hw, const struct freq_tbl *f, clk_flags = clk_hw_get_flags(hw); p = clk_hw_get_parent_by_index(hw, index); if (clk_flags & CLK_SET_RATE_PARENT) { + rate = f->freq; if (f->pre_div) { rate /= 2; rate *= f->pre_div + 1; @@ -248,7 +242,7 @@ static int clk_rcg2_determine_floor_rate(struct clk_hw *hw, return _freq_tbl_determine_rate(hw, rcg->freq_tbl, req, FLOOR); } -static int clk_rcg2_configure(struct clk_rcg2 *rcg, const struct freq_tbl *f) +static int __clk_rcg2_configure(struct clk_rcg2 *rcg, const struct freq_tbl *f) { u32 cfg, mask; struct clk_hw *hw = &rcg->clkr.hw; @@ -276,13 +270,21 @@ static int clk_rcg2_configure(struct clk_rcg2 *rcg, const struct freq_tbl *f) } mask = BIT(rcg->hid_width) - 1; - mask |= CFG_SRC_SEL_MASK | CFG_MODE_MASK; + mask |= CFG_SRC_SEL_MASK | CFG_MODE_MASK | CFG_HW_CLK_CTRL_MASK; cfg = f->pre_div << CFG_SRC_DIV_SHIFT; cfg |= rcg->parent_map[index].cfg << CFG_SRC_SEL_SHIFT; if (rcg->mnd_width && f->n && (f->m != f->n)) cfg |= CFG_MODE_DUAL_EDGE; - ret = regmap_update_bits(rcg->clkr.regmap, - rcg->cmd_rcgr + CFG_REG, mask, cfg); + + return regmap_update_bits(rcg->clkr.regmap, rcg->cmd_rcgr + CFG_REG, + mask, cfg); +} + +static int clk_rcg2_configure(struct clk_rcg2 *rcg, const struct freq_tbl *f) +{ + int ret; + + ret = __clk_rcg2_configure(rcg, f); if (ret) return ret; @@ -789,3 +791,141 @@ const struct clk_ops clk_gfx3d_ops = { .determine_rate = clk_gfx3d_determine_rate, }; EXPORT_SYMBOL_GPL(clk_gfx3d_ops); + +static int clk_rcg2_set_force_enable(struct clk_hw *hw) +{ + struct clk_rcg2 *rcg = to_clk_rcg2(hw); + const char *name = clk_hw_get_name(hw); + int ret, count; + + ret = regmap_update_bits(rcg->clkr.regmap, rcg->cmd_rcgr + CMD_REG, + CMD_ROOT_EN, CMD_ROOT_EN); + if (ret) + return ret; + + /* wait for RCG to turn ON */ + for (count = 500; count > 0; count--) { + if (clk_rcg2_is_enabled(hw)) + return 0; + + udelay(1); + } + + pr_err("%s: RCG did not turn on\n", name); + return -ETIMEDOUT; +} + +static int clk_rcg2_clear_force_enable(struct clk_hw *hw) +{ + struct clk_rcg2 *rcg = to_clk_rcg2(hw); + + return regmap_update_bits(rcg->clkr.regmap, rcg->cmd_rcgr + CMD_REG, + CMD_ROOT_EN, 0); +} + +static int +clk_rcg2_shared_force_enable_clear(struct clk_hw *hw, const struct freq_tbl *f) +{ + struct clk_rcg2 *rcg = to_clk_rcg2(hw); + int ret; + + ret = clk_rcg2_set_force_enable(hw); + if (ret) + return ret; + + ret = clk_rcg2_configure(rcg, f); + if (ret) + return ret; + + return clk_rcg2_clear_force_enable(hw); +} + +static int clk_rcg2_shared_set_rate(struct clk_hw *hw, unsigned long rate, + unsigned long parent_rate) +{ + struct clk_rcg2 *rcg = to_clk_rcg2(hw); + const struct freq_tbl *f; + + f = qcom_find_freq(rcg->freq_tbl, rate); + if (!f) + return -EINVAL; + + /* + * In case clock is disabled, update the CFG, M, N and D registers + * and don't hit the update bit of CMD register. + */ + if (!__clk_is_enabled(hw->clk)) + return __clk_rcg2_configure(rcg, f); + + return clk_rcg2_shared_force_enable_clear(hw, f); +} + +static int clk_rcg2_shared_set_rate_and_parent(struct clk_hw *hw, + unsigned long rate, unsigned long parent_rate, u8 index) +{ + return clk_rcg2_shared_set_rate(hw, rate, parent_rate); +} + +static int clk_rcg2_shared_enable(struct clk_hw *hw) +{ + struct clk_rcg2 *rcg = to_clk_rcg2(hw); + int ret; + + /* + * Set the update bit because required configuration has already + * been written in clk_rcg2_shared_set_rate() + */ + ret = clk_rcg2_set_force_enable(hw); + if (ret) + return ret; + + ret = update_config(rcg); + if (ret) + return ret; + + return clk_rcg2_clear_force_enable(hw); +} + +static void clk_rcg2_shared_disable(struct clk_hw *hw) +{ + struct clk_rcg2 *rcg = to_clk_rcg2(hw); + u32 cfg; + + /* + * Store current configuration as switching to safe source would clear + * the SRC and DIV of CFG register + */ + regmap_read(rcg->clkr.regmap, rcg->cmd_rcgr + CFG_REG, &cfg); + + /* + * Park the RCG at a safe configuration - sourced off of safe source. + * Force enable and disable the RCG while configuring it to safeguard + * against any update signal coming from the downstream clock. + * The current parent is still prepared and enabled at this point, and + * the safe source is always on while application processor subsystem + * is online. Therefore, the RCG can safely switch its parent. + */ + clk_rcg2_set_force_enable(hw); + + regmap_write(rcg->clkr.regmap, rcg->cmd_rcgr + CFG_REG, + rcg->safe_src_index << CFG_SRC_SEL_SHIFT); + + update_config(rcg); + + clk_rcg2_clear_force_enable(hw); + + /* Write back the stored configuration corresponding to current rate */ + regmap_write(rcg->clkr.regmap, rcg->cmd_rcgr + CFG_REG, cfg); +} + +const struct clk_ops clk_rcg2_shared_ops = { + .enable = clk_rcg2_shared_enable, + .disable = clk_rcg2_shared_disable, + .get_parent = clk_rcg2_get_parent, + .set_parent = clk_rcg2_set_parent, + .recalc_rate = clk_rcg2_recalc_rate, + .determine_rate = clk_rcg2_determine_rate, + .set_rate = clk_rcg2_shared_set_rate, + .set_rate_and_parent = clk_rcg2_shared_set_rate_and_parent, +}; +EXPORT_SYMBOL_GPL(clk_rcg2_shared_ops); diff --git a/drivers/clk/qcom/clk-spmi-pmic-div.c b/drivers/clk/qcom/clk-spmi-pmic-div.c index 8672ab84746f..c90dfdd6c147 100644 --- a/drivers/clk/qcom/clk-spmi-pmic-div.c +++ b/drivers/clk/qcom/clk-spmi-pmic-div.c @@ -239,8 +239,7 @@ static int spmi_pmic_clkdiv_probe(struct platform_device *pdev) if (!nclks) return -EINVAL; - cc = devm_kzalloc(dev, sizeof(*cc) + sizeof(*cc->clks) * nclks, - GFP_KERNEL); + cc = devm_kzalloc(dev, struct_size(cc, clks, nclks), GFP_KERNEL); if (!cc) return -ENOMEM; cc->nclks = nclks; diff --git a/drivers/clk/qcom/common.c b/drivers/clk/qcom/common.c index b8064a336d46..39ce64c2783b 100644 --- a/drivers/clk/qcom/common.c +++ b/drivers/clk/qcom/common.c @@ -228,22 +228,6 @@ int qcom_cc_really_probe(struct platform_device *pdev, if (!cc) return -ENOMEM; - cc->rclks = rclks; - cc->num_rclks = num_clks; - - for (i = 0; i < num_clks; i++) { - if (!rclks[i]) - continue; - - ret = devm_clk_register_regmap(dev, rclks[i]); - if (ret) - return ret; - } - - ret = devm_of_clk_add_hw_provider(dev, qcom_cc_clk_hw_get, cc); - if (ret) - return ret; - reset = &cc->reset; reset->rcdev.of_node = dev->of_node; reset->rcdev.ops = &qcom_reset_ops; @@ -272,6 +256,22 @@ int qcom_cc_really_probe(struct platform_device *pdev, return ret; } + cc->rclks = rclks; + cc->num_rclks = num_clks; + + for (i = 0; i < num_clks; i++) { + if (!rclks[i]) + continue; + + ret = devm_clk_register_regmap(dev, rclks[i]); + if (ret) + return ret; + } + + ret = devm_of_clk_add_hw_provider(dev, qcom_cc_clk_hw_get, cc); + if (ret) + return ret; + return 0; } EXPORT_SYMBOL_GPL(qcom_cc_really_probe); diff --git a/drivers/clk/qcom/gcc-msm8996.c b/drivers/clk/qcom/gcc-msm8996.c index 3d6452932797..9f35b3fe1d97 100644 --- a/drivers/clk/qcom/gcc-msm8996.c +++ b/drivers/clk/qcom/gcc-msm8996.c @@ -1418,6 +1418,7 @@ static struct clk_branch gcc_usb3_phy_aux_clk = { static struct clk_branch gcc_usb3_phy_pipe_clk = { .halt_reg = 0x50004, + .halt_check = BRANCH_HALT_SKIP, .clkr = { .enable_reg = 0x50004, .enable_mask = BIT(0), @@ -2472,6 +2473,7 @@ static struct clk_branch gcc_pcie_0_aux_clk = { static struct clk_branch gcc_pcie_0_pipe_clk = { .halt_reg = 0x6b018, + .halt_check = BRANCH_HALT_SKIP, .clkr = { .enable_reg = 0x6b018, .enable_mask = BIT(0), @@ -2547,6 +2549,7 @@ static struct clk_branch gcc_pcie_1_aux_clk = { static struct clk_branch gcc_pcie_1_pipe_clk = { .halt_reg = 0x6d018, + .halt_check = BRANCH_HALT_SKIP, .clkr = { .enable_reg = 0x6d018, .enable_mask = BIT(0), @@ -2622,6 +2625,7 @@ static struct clk_branch gcc_pcie_2_aux_clk = { static struct clk_branch gcc_pcie_2_pipe_clk = { .halt_reg = 0x6e018, + .halt_check = BRANCH_HALT_SKIP, .clkr = { .enable_reg = 0x6e018, .enable_mask = BIT(0), @@ -2792,6 +2796,7 @@ static struct clk_branch gcc_ufs_tx_symbol_0_clk = { static struct clk_branch gcc_ufs_rx_symbol_0_clk = { .halt_reg = 0x7501c, + .halt_check = BRANCH_HALT_SKIP, .clkr = { .enable_reg = 0x7501c, .enable_mask = BIT(0), @@ -2807,6 +2812,7 @@ static struct clk_branch gcc_ufs_rx_symbol_0_clk = { static struct clk_branch gcc_ufs_rx_symbol_1_clk = { .halt_reg = 0x75020, + .halt_check = BRANCH_HALT_SKIP, .clkr = { .enable_reg = 0x75020, .enable_mask = BIT(0), @@ -3105,7 +3111,7 @@ static struct gdsc aggre0_noc_gdsc = { .name = "aggre0_noc", }, .pwrsts = PWRSTS_OFF_ON, - .flags = VOTABLE, + .flags = VOTABLE | ALWAYS_ON, }; static struct gdsc hlos1_vote_aggre0_noc_gdsc = { diff --git a/drivers/clk/qcom/gcc-msm8998.c b/drivers/clk/qcom/gcc-msm8998.c new file mode 100644 index 000000000000..78d87f5c7098 --- /dev/null +++ b/drivers/clk/qcom/gcc-msm8998.c @@ -0,0 +1,2834 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2016, The Linux Foundation. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "common.h" +#include "clk-regmap.h" +#include "clk-alpha-pll.h" +#include "clk-pll.h" +#include "clk-rcg.h" +#include "clk-branch.h" +#include "reset.h" +#include "gdsc.h" + +#define F(f, s, h, m, n) { (f), (s), (2 * (h) - 1), (m), (n) } + +enum { + P_AUD_REF_CLK, + P_CORE_BI_PLL_TEST_SE, + P_GPLL0_OUT_MAIN, + P_GPLL4_OUT_MAIN, + P_PLL0_EARLY_DIV_CLK_SRC, + P_SLEEP_CLK, + P_XO, +}; + +static const struct parent_map gcc_parent_map_0[] = { + { P_XO, 0 }, + { P_GPLL0_OUT_MAIN, 1 }, + { P_PLL0_EARLY_DIV_CLK_SRC, 6 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_0[] = { + "xo", + "gpll0_out_main", + "gpll0_out_main", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_1[] = { + { P_XO, 0 }, + { P_GPLL0_OUT_MAIN, 1 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_1[] = { + "xo", + "gpll0_out_main", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_2[] = { + { P_XO, 0 }, + { P_GPLL0_OUT_MAIN, 1 }, + { P_SLEEP_CLK, 5 }, + { P_PLL0_EARLY_DIV_CLK_SRC, 6 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_2[] = { + "xo", + "gpll0_out_main", + "core_pi_sleep_clk", + "gpll0_out_main", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_3[] = { + { P_XO, 0 }, + { P_SLEEP_CLK, 5 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_3[] = { + "xo", + "core_pi_sleep_clk", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_4[] = { + { P_XO, 0 }, + { P_GPLL0_OUT_MAIN, 1 }, + { P_GPLL4_OUT_MAIN, 5 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_4[] = { + "xo", + "gpll0_out_main", + "gpll4_out_main", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_5[] = { + { P_XO, 0 }, + { P_GPLL0_OUT_MAIN, 1 }, + { P_AUD_REF_CLK, 2 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_5[] = { + "xo", + "gpll0_out_main", + "aud_ref_clk", + "core_bi_pll_test_se", +}; + +static struct pll_vco fabia_vco[] = { + { 250000000, 2000000000, 0 }, + { 125000000, 1000000000, 1 }, +}; + +static struct clk_alpha_pll gpll0 = { + .offset = 0x0, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .vco_table = fabia_vco, + .num_vco = ARRAY_SIZE(fabia_vco), + .clkr = { + .enable_reg = 0x52000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gpll0", + .parent_names = (const char *[]){ "xo" }, + .num_parents = 1, + .ops = &clk_alpha_pll_ops, + } + }, +}; + +static struct clk_alpha_pll_postdiv gpll0_out_even = { + .offset = 0x0, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll0_out_even", + .parent_names = (const char *[]){ "gpll0" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll0_out_main = { + .offset = 0x0, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll0_out_main", + .parent_names = (const char *[]){ "gpll0" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll0_out_odd = { + .offset = 0x0, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll0_out_odd", + .parent_names = (const char *[]){ "gpll0" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll0_out_test = { + .offset = 0x0, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll0_out_test", + .parent_names = (const char *[]){ "gpll0" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll gpll1 = { + .offset = 0x1000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .vco_table = fabia_vco, + .num_vco = ARRAY_SIZE(fabia_vco), + .clkr = { + .enable_reg = 0x52000, + .enable_mask = BIT(1), + .hw.init = &(struct clk_init_data){ + .name = "gpll1", + .parent_names = (const char *[]){ "xo" }, + .num_parents = 1, + .ops = &clk_alpha_pll_ops, + } + }, +}; + +static struct clk_alpha_pll_postdiv gpll1_out_even = { + .offset = 0x1000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll1_out_even", + .parent_names = (const char *[]){ "gpll1" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll1_out_main = { + .offset = 0x1000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll1_out_main", + .parent_names = (const char *[]){ "gpll1" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll1_out_odd = { + .offset = 0x1000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll1_out_odd", + .parent_names = (const char *[]){ "gpll1" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll1_out_test = { + .offset = 0x1000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll1_out_test", + .parent_names = (const char *[]){ "gpll1" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll gpll2 = { + .offset = 0x2000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .vco_table = fabia_vco, + .num_vco = ARRAY_SIZE(fabia_vco), + .clkr = { + .enable_reg = 0x52000, + .enable_mask = BIT(2), + .hw.init = &(struct clk_init_data){ + .name = "gpll2", + .parent_names = (const char *[]){ "xo" }, + .num_parents = 1, + .ops = &clk_alpha_pll_ops, + } + }, +}; + +static struct clk_alpha_pll_postdiv gpll2_out_even = { + .offset = 0x2000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll2_out_even", + .parent_names = (const char *[]){ "gpll2" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll2_out_main = { + .offset = 0x2000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll2_out_main", + .parent_names = (const char *[]){ "gpll2" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll2_out_odd = { + .offset = 0x2000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll2_out_odd", + .parent_names = (const char *[]){ "gpll2" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll2_out_test = { + .offset = 0x2000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll2_out_test", + .parent_names = (const char *[]){ "gpll2" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll gpll3 = { + .offset = 0x3000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .vco_table = fabia_vco, + .num_vco = ARRAY_SIZE(fabia_vco), + .clkr = { + .enable_reg = 0x52000, + .enable_mask = BIT(3), + .hw.init = &(struct clk_init_data){ + .name = "gpll3", + .parent_names = (const char *[]){ "xo" }, + .num_parents = 1, + .ops = &clk_alpha_pll_ops, + } + }, +}; + +static struct clk_alpha_pll_postdiv gpll3_out_even = { + .offset = 0x3000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll3_out_even", + .parent_names = (const char *[]){ "gpll3" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll3_out_main = { + .offset = 0x3000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll3_out_main", + .parent_names = (const char *[]){ "gpll3" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll3_out_odd = { + .offset = 0x3000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll3_out_odd", + .parent_names = (const char *[]){ "gpll3" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll3_out_test = { + .offset = 0x3000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll3_out_test", + .parent_names = (const char *[]){ "gpll3" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll gpll4 = { + .offset = 0x77000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .vco_table = fabia_vco, + .num_vco = ARRAY_SIZE(fabia_vco), + .clkr = { + .enable_reg = 0x52000, + .enable_mask = BIT(4), + .hw.init = &(struct clk_init_data){ + .name = "gpll4", + .parent_names = (const char *[]){ "xo" }, + .num_parents = 1, + .ops = &clk_alpha_pll_ops, + } + }, +}; + +static struct clk_alpha_pll_postdiv gpll4_out_even = { + .offset = 0x77000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll4_out_even", + .parent_names = (const char *[]){ "gpll4" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll4_out_main = { + .offset = 0x77000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll4_out_main", + .parent_names = (const char *[]){ "gpll4" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll4_out_odd = { + .offset = 0x77000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll4_out_odd", + .parent_names = (const char *[]){ "gpll4" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static struct clk_alpha_pll_postdiv gpll4_out_test = { + .offset = 0x77000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_DEFAULT], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll4_out_test", + .parent_names = (const char *[]){ "gpll4" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_ops, + }, +}; + +static const struct freq_tbl ftbl_blsp1_qup1_i2c_apps_clk_src[] = { + F(19200000, P_XO, 1, 0, 0), + F(50000000, P_GPLL0_OUT_MAIN, 12, 0, 0), + { } +}; + +static struct clk_rcg2 blsp1_qup1_i2c_apps_clk_src = { + .cmd_rcgr = 0x19020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup1_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_blsp1_qup1_spi_apps_clk_src[] = { + F(960000, P_XO, 10, 1, 2), + F(4800000, P_XO, 4, 0, 0), + F(9600000, P_XO, 2, 0, 0), + F(15000000, P_GPLL0_OUT_MAIN, 10, 1, 4), + F(19200000, P_XO, 1, 0, 0), + F(25000000, P_GPLL0_OUT_MAIN, 12, 1, 2), + F(50000000, P_GPLL0_OUT_MAIN, 12, 0, 0), + { } +}; + +static struct clk_rcg2 blsp1_qup1_spi_apps_clk_src = { + .cmd_rcgr = 0x1900c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup1_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_qup2_i2c_apps_clk_src = { + .cmd_rcgr = 0x1b020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup2_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_qup2_spi_apps_clk_src = { + .cmd_rcgr = 0x1b00c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup2_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_qup3_i2c_apps_clk_src = { + .cmd_rcgr = 0x1d020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup3_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_qup3_spi_apps_clk_src = { + .cmd_rcgr = 0x1d00c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup3_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_qup4_i2c_apps_clk_src = { + .cmd_rcgr = 0x1f020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup4_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_qup4_spi_apps_clk_src = { + .cmd_rcgr = 0x1f00c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup4_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_qup5_i2c_apps_clk_src = { + .cmd_rcgr = 0x21020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup5_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_qup5_spi_apps_clk_src = { + .cmd_rcgr = 0x2100c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup5_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_qup6_i2c_apps_clk_src = { + .cmd_rcgr = 0x23020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup6_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_qup6_spi_apps_clk_src = { + .cmd_rcgr = 0x2300c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_qup6_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_blsp1_uart1_apps_clk_src[] = { + F(3686400, P_GPLL0_OUT_MAIN, 1, 96, 15625), + F(7372800, P_GPLL0_OUT_MAIN, 1, 192, 15625), + F(14745600, P_GPLL0_OUT_MAIN, 1, 384, 15625), + F(16000000, P_GPLL0_OUT_MAIN, 5, 2, 15), + F(19200000, P_XO, 1, 0, 0), + F(24000000, P_GPLL0_OUT_MAIN, 5, 1, 5), + F(32000000, P_GPLL0_OUT_MAIN, 1, 4, 75), + F(40000000, P_GPLL0_OUT_MAIN, 15, 0, 0), + F(46400000, P_GPLL0_OUT_MAIN, 1, 29, 375), + F(48000000, P_GPLL0_OUT_MAIN, 12.5, 0, 0), + F(51200000, P_GPLL0_OUT_MAIN, 1, 32, 375), + F(56000000, P_GPLL0_OUT_MAIN, 1, 7, 75), + F(58982400, P_GPLL0_OUT_MAIN, 1, 1536, 15625), + F(60000000, P_GPLL0_OUT_MAIN, 10, 0, 0), + F(63157895, P_GPLL0_OUT_MAIN, 9.5, 0, 0), + { } +}; + +static struct clk_rcg2 blsp1_uart1_apps_clk_src = { + .cmd_rcgr = 0x1a00c, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_uart1_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_uart1_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_uart2_apps_clk_src = { + .cmd_rcgr = 0x1c00c, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_uart1_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_uart2_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp1_uart3_apps_clk_src = { + .cmd_rcgr = 0x1e00c, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_uart1_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp1_uart3_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup1_i2c_apps_clk_src = { + .cmd_rcgr = 0x26020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup1_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup1_spi_apps_clk_src = { + .cmd_rcgr = 0x2600c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup1_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup2_i2c_apps_clk_src = { + .cmd_rcgr = 0x28020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup2_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup2_spi_apps_clk_src = { + .cmd_rcgr = 0x2800c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup2_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup3_i2c_apps_clk_src = { + .cmd_rcgr = 0x2a020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup3_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup3_spi_apps_clk_src = { + .cmd_rcgr = 0x2a00c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup3_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup4_i2c_apps_clk_src = { + .cmd_rcgr = 0x2c020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup4_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup4_spi_apps_clk_src = { + .cmd_rcgr = 0x2c00c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup4_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup5_i2c_apps_clk_src = { + .cmd_rcgr = 0x2e020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup5_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup5_spi_apps_clk_src = { + .cmd_rcgr = 0x2e00c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup5_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup6_i2c_apps_clk_src = { + .cmd_rcgr = 0x30020, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_blsp1_qup1_i2c_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup6_i2c_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_qup6_spi_apps_clk_src = { + .cmd_rcgr = 0x3000c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_qup1_spi_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_qup6_spi_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_uart1_apps_clk_src = { + .cmd_rcgr = 0x2700c, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_uart1_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_uart1_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_uart2_apps_clk_src = { + .cmd_rcgr = 0x2900c, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_uart1_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_uart2_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 blsp2_uart3_apps_clk_src = { + .cmd_rcgr = 0x2b00c, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_blsp1_uart1_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "blsp2_uart3_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gp1_clk_src[] = { + F(19200000, P_XO, 1, 0, 0), + F(100000000, P_GPLL0_OUT_MAIN, 6, 0, 0), + F(200000000, P_GPLL0_OUT_MAIN, 3, 0, 0), + { } +}; + +static struct clk_rcg2 gp1_clk_src = { + .cmd_rcgr = 0x64004, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_2, + .freq_tbl = ftbl_gp1_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gp1_clk_src", + .parent_names = gcc_parent_names_2, + .num_parents = 5, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 gp2_clk_src = { + .cmd_rcgr = 0x65004, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_2, + .freq_tbl = ftbl_gp1_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gp2_clk_src", + .parent_names = gcc_parent_names_2, + .num_parents = 5, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 gp3_clk_src = { + .cmd_rcgr = 0x66004, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_2, + .freq_tbl = ftbl_gp1_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gp3_clk_src", + .parent_names = gcc_parent_names_2, + .num_parents = 5, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_hmss_ahb_clk_src[] = { + F(19200000, P_XO, 1, 0, 0), + F(37500000, P_GPLL0_OUT_MAIN, 16, 0, 0), + F(75000000, P_GPLL0_OUT_MAIN, 8, 0, 0), + { } +}; + +static struct clk_rcg2 hmss_ahb_clk_src = { + .cmd_rcgr = 0x48014, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_hmss_ahb_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "hmss_ahb_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_hmss_rbcpr_clk_src[] = { + F(19200000, P_XO, 1, 0, 0), + { } +}; + +static struct clk_rcg2 hmss_rbcpr_clk_src = { + .cmd_rcgr = 0x48044, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_hmss_rbcpr_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "hmss_rbcpr_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_pcie_aux_clk_src[] = { + F(1010526, P_XO, 1, 1, 19), + { } +}; + +static struct clk_rcg2 pcie_aux_clk_src = { + .cmd_rcgr = 0x6c000, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_3, + .freq_tbl = ftbl_pcie_aux_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "pcie_aux_clk_src", + .parent_names = gcc_parent_names_3, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_pdm2_clk_src[] = { + F(60000000, P_GPLL0_OUT_MAIN, 10, 0, 0), + { } +}; + +static struct clk_rcg2 pdm2_clk_src = { + .cmd_rcgr = 0x33010, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_pdm2_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "pdm2_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_sdcc2_apps_clk_src[] = { + F(144000, P_XO, 16, 3, 25), + F(400000, P_XO, 12, 1, 4), + F(20000000, P_GPLL0_OUT_MAIN, 15, 1, 2), + F(25000000, P_GPLL0_OUT_MAIN, 12, 1, 2), + F(50000000, P_GPLL0_OUT_MAIN, 12, 0, 0), + F(100000000, P_GPLL0_OUT_MAIN, 6, 0, 0), + F(200000000, P_GPLL0_OUT_MAIN, 3, 0, 0), + { } +}; + +static struct clk_rcg2 sdcc2_apps_clk_src = { + .cmd_rcgr = 0x14010, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_4, + .freq_tbl = ftbl_sdcc2_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "sdcc2_apps_clk_src", + .parent_names = gcc_parent_names_4, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_sdcc4_apps_clk_src[] = { + F(144000, P_XO, 16, 3, 25), + F(400000, P_XO, 12, 1, 4), + F(20000000, P_GPLL0_OUT_MAIN, 15, 1, 2), + F(25000000, P_GPLL0_OUT_MAIN, 12, 1, 2), + F(50000000, P_GPLL0_OUT_MAIN, 12, 0, 0), + F(100000000, P_GPLL0_OUT_MAIN, 6, 0, 0), + { } +}; + +static struct clk_rcg2 sdcc4_apps_clk_src = { + .cmd_rcgr = 0x16010, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_sdcc4_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "sdcc4_apps_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_tsif_ref_clk_src[] = { + F(105495, P_XO, 1, 1, 182), + { } +}; + +static struct clk_rcg2 tsif_ref_clk_src = { + .cmd_rcgr = 0x36010, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_5, + .freq_tbl = ftbl_tsif_ref_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "tsif_ref_clk_src", + .parent_names = gcc_parent_names_5, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_ufs_axi_clk_src[] = { + F(100000000, P_GPLL0_OUT_MAIN, 6, 0, 0), + F(200000000, P_GPLL0_OUT_MAIN, 3, 0, 0), + F(240000000, P_GPLL0_OUT_MAIN, 2.5, 0, 0), + { } +}; + +static struct clk_rcg2 ufs_axi_clk_src = { + .cmd_rcgr = 0x75018, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_ufs_axi_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "ufs_axi_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_usb30_master_clk_src[] = { + F(19200000, P_XO, 1, 0, 0), + F(120000000, P_GPLL0_OUT_MAIN, 5, 0, 0), + F(150000000, P_GPLL0_OUT_MAIN, 4, 0, 0), + { } +}; + +static struct clk_rcg2 usb30_master_clk_src = { + .cmd_rcgr = 0xf014, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_usb30_master_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "usb30_master_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 usb30_mock_utmi_clk_src = { + .cmd_rcgr = 0xf028, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_hmss_rbcpr_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "usb30_mock_utmi_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_usb3_phy_aux_clk_src[] = { + F(1200000, P_XO, 16, 0, 0), + { } +}; + +static struct clk_rcg2 usb3_phy_aux_clk_src = { + .cmd_rcgr = 0x5000c, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_3, + .freq_tbl = ftbl_usb3_phy_aux_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "usb3_phy_aux_clk_src", + .parent_names = gcc_parent_names_3, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_branch gcc_aggre1_noc_xo_clk = { + .halt_reg = 0x8202c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8202c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_aggre1_noc_xo_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_aggre1_ufs_axi_clk = { + .halt_reg = 0x82028, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x82028, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_aggre1_ufs_axi_clk", + .parent_names = (const char *[]){ + "ufs_axi_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_aggre1_usb3_axi_clk = { + .halt_reg = 0x82024, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x82024, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_aggre1_usb3_axi_clk", + .parent_names = (const char *[]){ + "usb30_master_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_apss_qdss_tsctr_div2_clk = { + .halt_reg = 0x48090, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x48090, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_apss_qdss_tsctr_div2_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_apss_qdss_tsctr_div8_clk = { + .halt_reg = 0x48094, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x48094, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_apss_qdss_tsctr_div8_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_bimc_hmss_axi_clk = { + .halt_reg = 0x48004, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(22), + .hw.init = &(struct clk_init_data){ + .name = "gcc_bimc_hmss_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_bimc_mss_q6_axi_clk = { + .halt_reg = 0x4401c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x4401c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_bimc_mss_q6_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_ahb_clk = { + .halt_reg = 0x17004, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(17), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup1_i2c_apps_clk = { + .halt_reg = 0x19008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x19008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup1_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup1_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup1_spi_apps_clk = { + .halt_reg = 0x19004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x19004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup1_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup1_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup2_i2c_apps_clk = { + .halt_reg = 0x1b008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x1b008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup2_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup2_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup2_spi_apps_clk = { + .halt_reg = 0x1b004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x1b004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup2_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup2_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup3_i2c_apps_clk = { + .halt_reg = 0x1d008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x1d008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup3_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup3_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup3_spi_apps_clk = { + .halt_reg = 0x1d004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x1d004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup3_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup3_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup4_i2c_apps_clk = { + .halt_reg = 0x1f008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x1f008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup4_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup4_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup4_spi_apps_clk = { + .halt_reg = 0x1f004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x1f004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup4_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup4_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup5_i2c_apps_clk = { + .halt_reg = 0x21008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x21008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup5_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup5_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup5_spi_apps_clk = { + .halt_reg = 0x21004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x21004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup5_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup5_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup6_i2c_apps_clk = { + .halt_reg = 0x23008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x23008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup6_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup6_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_qup6_spi_apps_clk = { + .halt_reg = 0x23004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x23004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_qup6_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp1_qup6_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_sleep_clk = { + .halt_reg = 0x17008, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(16), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_sleep_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_uart1_apps_clk = { + .halt_reg = 0x1a004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x1a004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_uart1_apps_clk", + .parent_names = (const char *[]){ + "blsp1_uart1_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_uart2_apps_clk = { + .halt_reg = 0x1c004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x1c004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_uart2_apps_clk", + .parent_names = (const char *[]){ + "blsp1_uart2_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp1_uart3_apps_clk = { + .halt_reg = 0x1e004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x1e004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp1_uart3_apps_clk", + .parent_names = (const char *[]){ + "blsp1_uart3_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_ahb_clk = { + .halt_reg = 0x25004, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(15), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup1_i2c_apps_clk = { + .halt_reg = 0x26008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x26008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup1_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup1_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup1_spi_apps_clk = { + .halt_reg = 0x26004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x26004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup1_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup1_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup2_i2c_apps_clk = { + .halt_reg = 0x28008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x28008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup2_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup2_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup2_spi_apps_clk = { + .halt_reg = 0x28004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x28004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup2_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup2_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup3_i2c_apps_clk = { + .halt_reg = 0x2a008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x2a008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup3_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup3_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup3_spi_apps_clk = { + .halt_reg = 0x2a004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x2a004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup3_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup3_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup4_i2c_apps_clk = { + .halt_reg = 0x2c008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x2c008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup4_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup4_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup4_spi_apps_clk = { + .halt_reg = 0x2c004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x2c004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup4_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup4_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup5_i2c_apps_clk = { + .halt_reg = 0x2e008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x2e008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup5_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup5_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup5_spi_apps_clk = { + .halt_reg = 0x2e004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x2e004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup5_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup5_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup6_i2c_apps_clk = { + .halt_reg = 0x30008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x30008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup6_i2c_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup6_i2c_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_qup6_spi_apps_clk = { + .halt_reg = 0x30004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x30004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_qup6_spi_apps_clk", + .parent_names = (const char *[]){ + "blsp2_qup6_spi_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_sleep_clk = { + .halt_reg = 0x25008, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(14), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_sleep_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_uart1_apps_clk = { + .halt_reg = 0x27004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x27004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_uart1_apps_clk", + .parent_names = (const char *[]){ + "blsp2_uart1_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_uart2_apps_clk = { + .halt_reg = 0x29004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x29004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_uart2_apps_clk", + .parent_names = (const char *[]){ + "blsp2_uart2_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_blsp2_uart3_apps_clk = { + .halt_reg = 0x2b004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x2b004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_blsp2_uart3_apps_clk", + .parent_names = (const char *[]){ + "blsp2_uart3_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_cfg_noc_usb3_axi_clk = { + .halt_reg = 0x5018, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x5018, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_cfg_noc_usb3_axi_clk", + .parent_names = (const char *[]){ + "usb30_master_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gp1_clk = { + .halt_reg = 0x64000, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x64000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gp1_clk", + .parent_names = (const char *[]){ + "gp1_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gp2_clk = { + .halt_reg = 0x65000, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x65000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gp2_clk", + .parent_names = (const char *[]){ + "gp2_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gp3_clk = { + .halt_reg = 0x66000, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x66000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gp3_clk", + .parent_names = (const char *[]){ + "gp3_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gpu_bimc_gfx_clk = { + .halt_reg = 0x71010, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x71010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gpu_bimc_gfx_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gpu_bimc_gfx_src_clk = { + .halt_reg = 0x7100c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x7100c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gpu_bimc_gfx_src_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gpu_cfg_ahb_clk = { + .halt_reg = 0x71004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x71004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gpu_cfg_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gpu_snoc_dvm_gfx_clk = { + .halt_reg = 0x71018, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x71018, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gpu_snoc_dvm_gfx_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_hmss_ahb_clk = { + .halt_reg = 0x48000, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(21), + .hw.init = &(struct clk_init_data){ + .name = "gcc_hmss_ahb_clk", + .parent_names = (const char *[]){ + "hmss_ahb_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_hmss_at_clk = { + .halt_reg = 0x48010, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x48010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_hmss_at_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_hmss_dvm_bus_clk = { + .halt_reg = 0x4808c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x4808c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_hmss_dvm_bus_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_hmss_rbcpr_clk = { + .halt_reg = 0x48008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x48008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_hmss_rbcpr_clk", + .parent_names = (const char *[]){ + "hmss_rbcpr_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_hmss_trig_clk = { + .halt_reg = 0x4800c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x4800c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_hmss_trig_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_lpass_at_clk = { + .halt_reg = 0x47020, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x47020, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_lpass_at_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_lpass_trig_clk = { + .halt_reg = 0x4701c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x4701c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_lpass_trig_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mmss_noc_cfg_ahb_clk = { + .halt_reg = 0x9004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x9004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mmss_noc_cfg_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mmss_qm_ahb_clk = { + .halt_reg = 0x9030, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x9030, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mmss_qm_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mmss_qm_core_clk = { + .halt_reg = 0x900c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x900c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mmss_qm_core_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mmss_sys_noc_axi_clk = { + .halt_reg = 0x9000, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x9000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mmss_sys_noc_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mss_at_clk = { + .halt_reg = 0x8a00c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8a00c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mss_at_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_aux_clk = { + .halt_reg = 0x6b014, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x6b014, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_aux_clk", + .parent_names = (const char *[]){ + "pcie_aux_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_cfg_ahb_clk = { + .halt_reg = 0x6b010, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x6b010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_cfg_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_mstr_axi_clk = { + .halt_reg = 0x6b00c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x6b00c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_mstr_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_pipe_clk = { + .halt_reg = 0x6b018, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x6b018, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_pipe_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_slv_axi_clk = { + .halt_reg = 0x6b008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x6b008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_slv_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_phy_aux_clk = { + .halt_reg = 0x6f004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x6f004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_phy_aux_clk", + .parent_names = (const char *[]){ + "pcie_aux_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pdm2_clk = { + .halt_reg = 0x3300c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x3300c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pdm2_clk", + .parent_names = (const char *[]){ + "pdm2_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pdm_ahb_clk = { + .halt_reg = 0x33004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x33004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pdm_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pdm_xo4_clk = { + .halt_reg = 0x33008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x33008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pdm_xo4_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_prng_ahb_clk = { + .halt_reg = 0x34004, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(13), + .hw.init = &(struct clk_init_data){ + .name = "gcc_prng_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_sdcc2_ahb_clk = { + .halt_reg = 0x14008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x14008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_sdcc2_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_sdcc2_apps_clk = { + .halt_reg = 0x14004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x14004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_sdcc2_apps_clk", + .parent_names = (const char *[]){ + "sdcc2_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_sdcc4_ahb_clk = { + .halt_reg = 0x16008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x16008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_sdcc4_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_sdcc4_apps_clk = { + .halt_reg = 0x16004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x16004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_sdcc4_apps_clk", + .parent_names = (const char *[]){ + "sdcc4_apps_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_tsif_ahb_clk = { + .halt_reg = 0x36004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x36004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_tsif_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_tsif_inactivity_timers_clk = { + .halt_reg = 0x3600c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x3600c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_tsif_inactivity_timers_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_tsif_ref_clk = { + .halt_reg = 0x36008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x36008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_tsif_ref_clk", + .parent_names = (const char *[]){ + "tsif_ref_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_ahb_clk = { + .halt_reg = 0x7500c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x7500c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_axi_clk = { + .halt_reg = 0x75008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x75008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_axi_clk", + .parent_names = (const char *[]){ + "ufs_axi_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_ice_core_clk = { + .halt_reg = 0x7600c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x7600c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_ice_core_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_phy_aux_clk = { + .halt_reg = 0x76040, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x76040, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_aux_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_rx_symbol_0_clk = { + .halt_reg = 0x75014, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x75014, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_rx_symbol_0_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_rx_symbol_1_clk = { + .halt_reg = 0x7605c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x7605c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_rx_symbol_1_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_tx_symbol_0_clk = { + .halt_reg = 0x75010, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x75010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_tx_symbol_0_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_unipro_core_clk = { + .halt_reg = 0x76008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x76008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_unipro_core_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb30_master_clk = { + .halt_reg = 0xf008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0xf008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_master_clk", + .parent_names = (const char *[]){ + "usb30_master_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb30_mock_utmi_clk = { + .halt_reg = 0xf010, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0xf010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_mock_utmi_clk", + .parent_names = (const char *[]){ + "usb30_mock_utmi_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb30_sleep_clk = { + .halt_reg = 0xf00c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0xf00c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_sleep_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb3_phy_aux_clk = { + .halt_reg = 0x50000, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x50000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_phy_aux_clk", + .parent_names = (const char *[]){ + "usb3_phy_aux_clk_src", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb3_phy_pipe_clk = { + .halt_reg = 0x50004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x50004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_phy_pipe_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb_phy_cfg_ahb2phy_clk = { + .halt_reg = 0x6a004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x6a004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb_phy_cfg_ahb2phy_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct gdsc pcie_0_gdsc = { + .gdscr = 0x6b004, + .gds_hw_ctrl = 0x0, + .pd = { + .name = "pcie_0_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, + .flags = VOTABLE, +}; + +static struct gdsc ufs_gdsc = { + .gdscr = 0x75004, + .gds_hw_ctrl = 0x0, + .pd = { + .name = "ufs_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, + .flags = VOTABLE, +}; + +static struct gdsc usb_30_gdsc = { + .gdscr = 0xf004, + .gds_hw_ctrl = 0x0, + .pd = { + .name = "usb_30_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, + .flags = VOTABLE, +}; + +static struct clk_regmap *gcc_msm8998_clocks[] = { + [BLSP1_QUP1_I2C_APPS_CLK_SRC] = &blsp1_qup1_i2c_apps_clk_src.clkr, + [BLSP1_QUP1_SPI_APPS_CLK_SRC] = &blsp1_qup1_spi_apps_clk_src.clkr, + [BLSP1_QUP2_I2C_APPS_CLK_SRC] = &blsp1_qup2_i2c_apps_clk_src.clkr, + [BLSP1_QUP2_SPI_APPS_CLK_SRC] = &blsp1_qup2_spi_apps_clk_src.clkr, + [BLSP1_QUP3_I2C_APPS_CLK_SRC] = &blsp1_qup3_i2c_apps_clk_src.clkr, + [BLSP1_QUP3_SPI_APPS_CLK_SRC] = &blsp1_qup3_spi_apps_clk_src.clkr, + [BLSP1_QUP4_I2C_APPS_CLK_SRC] = &blsp1_qup4_i2c_apps_clk_src.clkr, + [BLSP1_QUP4_SPI_APPS_CLK_SRC] = &blsp1_qup4_spi_apps_clk_src.clkr, + [BLSP1_QUP5_I2C_APPS_CLK_SRC] = &blsp1_qup5_i2c_apps_clk_src.clkr, + [BLSP1_QUP5_SPI_APPS_CLK_SRC] = &blsp1_qup5_spi_apps_clk_src.clkr, + [BLSP1_QUP6_I2C_APPS_CLK_SRC] = &blsp1_qup6_i2c_apps_clk_src.clkr, + [BLSP1_QUP6_SPI_APPS_CLK_SRC] = &blsp1_qup6_spi_apps_clk_src.clkr, + [BLSP1_UART1_APPS_CLK_SRC] = &blsp1_uart1_apps_clk_src.clkr, + [BLSP1_UART2_APPS_CLK_SRC] = &blsp1_uart2_apps_clk_src.clkr, + [BLSP1_UART3_APPS_CLK_SRC] = &blsp1_uart3_apps_clk_src.clkr, + [BLSP2_QUP1_I2C_APPS_CLK_SRC] = &blsp2_qup1_i2c_apps_clk_src.clkr, + [BLSP2_QUP1_SPI_APPS_CLK_SRC] = &blsp2_qup1_spi_apps_clk_src.clkr, + [BLSP2_QUP2_I2C_APPS_CLK_SRC] = &blsp2_qup2_i2c_apps_clk_src.clkr, + [BLSP2_QUP2_SPI_APPS_CLK_SRC] = &blsp2_qup2_spi_apps_clk_src.clkr, + [BLSP2_QUP3_I2C_APPS_CLK_SRC] = &blsp2_qup3_i2c_apps_clk_src.clkr, + [BLSP2_QUP3_SPI_APPS_CLK_SRC] = &blsp2_qup3_spi_apps_clk_src.clkr, + [BLSP2_QUP4_I2C_APPS_CLK_SRC] = &blsp2_qup4_i2c_apps_clk_src.clkr, + [BLSP2_QUP4_SPI_APPS_CLK_SRC] = &blsp2_qup4_spi_apps_clk_src.clkr, + [BLSP2_QUP5_I2C_APPS_CLK_SRC] = &blsp2_qup5_i2c_apps_clk_src.clkr, + [BLSP2_QUP5_SPI_APPS_CLK_SRC] = &blsp2_qup5_spi_apps_clk_src.clkr, + [BLSP2_QUP6_I2C_APPS_CLK_SRC] = &blsp2_qup6_i2c_apps_clk_src.clkr, + [BLSP2_QUP6_SPI_APPS_CLK_SRC] = &blsp2_qup6_spi_apps_clk_src.clkr, + [BLSP2_UART1_APPS_CLK_SRC] = &blsp2_uart1_apps_clk_src.clkr, + [BLSP2_UART2_APPS_CLK_SRC] = &blsp2_uart2_apps_clk_src.clkr, + [BLSP2_UART3_APPS_CLK_SRC] = &blsp2_uart3_apps_clk_src.clkr, + [GCC_AGGRE1_NOC_XO_CLK] = &gcc_aggre1_noc_xo_clk.clkr, + [GCC_AGGRE1_UFS_AXI_CLK] = &gcc_aggre1_ufs_axi_clk.clkr, + [GCC_AGGRE1_USB3_AXI_CLK] = &gcc_aggre1_usb3_axi_clk.clkr, + [GCC_APSS_QDSS_TSCTR_DIV2_CLK] = &gcc_apss_qdss_tsctr_div2_clk.clkr, + [GCC_APSS_QDSS_TSCTR_DIV8_CLK] = &gcc_apss_qdss_tsctr_div8_clk.clkr, + [GCC_BIMC_HMSS_AXI_CLK] = &gcc_bimc_hmss_axi_clk.clkr, + [GCC_BIMC_MSS_Q6_AXI_CLK] = &gcc_bimc_mss_q6_axi_clk.clkr, + [GCC_BLSP1_AHB_CLK] = &gcc_blsp1_ahb_clk.clkr, + [GCC_BLSP1_QUP1_I2C_APPS_CLK] = &gcc_blsp1_qup1_i2c_apps_clk.clkr, + [GCC_BLSP1_QUP1_SPI_APPS_CLK] = &gcc_blsp1_qup1_spi_apps_clk.clkr, + [GCC_BLSP1_QUP2_I2C_APPS_CLK] = &gcc_blsp1_qup2_i2c_apps_clk.clkr, + [GCC_BLSP1_QUP2_SPI_APPS_CLK] = &gcc_blsp1_qup2_spi_apps_clk.clkr, + [GCC_BLSP1_QUP3_I2C_APPS_CLK] = &gcc_blsp1_qup3_i2c_apps_clk.clkr, + [GCC_BLSP1_QUP3_SPI_APPS_CLK] = &gcc_blsp1_qup3_spi_apps_clk.clkr, + [GCC_BLSP1_QUP4_I2C_APPS_CLK] = &gcc_blsp1_qup4_i2c_apps_clk.clkr, + [GCC_BLSP1_QUP4_SPI_APPS_CLK] = &gcc_blsp1_qup4_spi_apps_clk.clkr, + [GCC_BLSP1_QUP5_I2C_APPS_CLK] = &gcc_blsp1_qup5_i2c_apps_clk.clkr, + [GCC_BLSP1_QUP5_SPI_APPS_CLK] = &gcc_blsp1_qup5_spi_apps_clk.clkr, + [GCC_BLSP1_QUP6_I2C_APPS_CLK] = &gcc_blsp1_qup6_i2c_apps_clk.clkr, + [GCC_BLSP1_QUP6_SPI_APPS_CLK] = &gcc_blsp1_qup6_spi_apps_clk.clkr, + [GCC_BLSP1_SLEEP_CLK] = &gcc_blsp1_sleep_clk.clkr, + [GCC_BLSP1_UART1_APPS_CLK] = &gcc_blsp1_uart1_apps_clk.clkr, + [GCC_BLSP1_UART2_APPS_CLK] = &gcc_blsp1_uart2_apps_clk.clkr, + [GCC_BLSP1_UART3_APPS_CLK] = &gcc_blsp1_uart3_apps_clk.clkr, + [GCC_BLSP2_AHB_CLK] = &gcc_blsp2_ahb_clk.clkr, + [GCC_BLSP2_QUP1_I2C_APPS_CLK] = &gcc_blsp2_qup1_i2c_apps_clk.clkr, + [GCC_BLSP2_QUP1_SPI_APPS_CLK] = &gcc_blsp2_qup1_spi_apps_clk.clkr, + [GCC_BLSP2_QUP2_I2C_APPS_CLK] = &gcc_blsp2_qup2_i2c_apps_clk.clkr, + [GCC_BLSP2_QUP2_SPI_APPS_CLK] = &gcc_blsp2_qup2_spi_apps_clk.clkr, + [GCC_BLSP2_QUP3_I2C_APPS_CLK] = &gcc_blsp2_qup3_i2c_apps_clk.clkr, + [GCC_BLSP2_QUP3_SPI_APPS_CLK] = &gcc_blsp2_qup3_spi_apps_clk.clkr, + [GCC_BLSP2_QUP4_I2C_APPS_CLK] = &gcc_blsp2_qup4_i2c_apps_clk.clkr, + [GCC_BLSP2_QUP4_SPI_APPS_CLK] = &gcc_blsp2_qup4_spi_apps_clk.clkr, + [GCC_BLSP2_QUP5_I2C_APPS_CLK] = &gcc_blsp2_qup5_i2c_apps_clk.clkr, + [GCC_BLSP2_QUP5_SPI_APPS_CLK] = &gcc_blsp2_qup5_spi_apps_clk.clkr, + [GCC_BLSP2_QUP6_I2C_APPS_CLK] = &gcc_blsp2_qup6_i2c_apps_clk.clkr, + [GCC_BLSP2_QUP6_SPI_APPS_CLK] = &gcc_blsp2_qup6_spi_apps_clk.clkr, + [GCC_BLSP2_SLEEP_CLK] = &gcc_blsp2_sleep_clk.clkr, + [GCC_BLSP2_UART1_APPS_CLK] = &gcc_blsp2_uart1_apps_clk.clkr, + [GCC_BLSP2_UART2_APPS_CLK] = &gcc_blsp2_uart2_apps_clk.clkr, + [GCC_BLSP2_UART3_APPS_CLK] = &gcc_blsp2_uart3_apps_clk.clkr, + [GCC_CFG_NOC_USB3_AXI_CLK] = &gcc_cfg_noc_usb3_axi_clk.clkr, + [GCC_GP1_CLK] = &gcc_gp1_clk.clkr, + [GCC_GP2_CLK] = &gcc_gp2_clk.clkr, + [GCC_GP3_CLK] = &gcc_gp3_clk.clkr, + [GCC_GPU_BIMC_GFX_CLK] = &gcc_gpu_bimc_gfx_clk.clkr, + [GCC_GPU_BIMC_GFX_SRC_CLK] = &gcc_gpu_bimc_gfx_src_clk.clkr, + [GCC_GPU_CFG_AHB_CLK] = &gcc_gpu_cfg_ahb_clk.clkr, + [GCC_GPU_SNOC_DVM_GFX_CLK] = &gcc_gpu_snoc_dvm_gfx_clk.clkr, + [GCC_HMSS_AHB_CLK] = &gcc_hmss_ahb_clk.clkr, + [GCC_HMSS_AT_CLK] = &gcc_hmss_at_clk.clkr, + [GCC_HMSS_DVM_BUS_CLK] = &gcc_hmss_dvm_bus_clk.clkr, + [GCC_HMSS_RBCPR_CLK] = &gcc_hmss_rbcpr_clk.clkr, + [GCC_HMSS_TRIG_CLK] = &gcc_hmss_trig_clk.clkr, + [GCC_LPASS_AT_CLK] = &gcc_lpass_at_clk.clkr, + [GCC_LPASS_TRIG_CLK] = &gcc_lpass_trig_clk.clkr, + [GCC_MMSS_NOC_CFG_AHB_CLK] = &gcc_mmss_noc_cfg_ahb_clk.clkr, + [GCC_MMSS_QM_AHB_CLK] = &gcc_mmss_qm_ahb_clk.clkr, + [GCC_MMSS_QM_CORE_CLK] = &gcc_mmss_qm_core_clk.clkr, + [GCC_MMSS_SYS_NOC_AXI_CLK] = &gcc_mmss_sys_noc_axi_clk.clkr, + [GCC_MSS_AT_CLK] = &gcc_mss_at_clk.clkr, + [GCC_PCIE_0_AUX_CLK] = &gcc_pcie_0_aux_clk.clkr, + [GCC_PCIE_0_CFG_AHB_CLK] = &gcc_pcie_0_cfg_ahb_clk.clkr, + [GCC_PCIE_0_MSTR_AXI_CLK] = &gcc_pcie_0_mstr_axi_clk.clkr, + [GCC_PCIE_0_PIPE_CLK] = &gcc_pcie_0_pipe_clk.clkr, + [GCC_PCIE_0_SLV_AXI_CLK] = &gcc_pcie_0_slv_axi_clk.clkr, + [GCC_PCIE_PHY_AUX_CLK] = &gcc_pcie_phy_aux_clk.clkr, + [GCC_PDM2_CLK] = &gcc_pdm2_clk.clkr, + [GCC_PDM_AHB_CLK] = &gcc_pdm_ahb_clk.clkr, + [GCC_PDM_XO4_CLK] = &gcc_pdm_xo4_clk.clkr, + [GCC_PRNG_AHB_CLK] = &gcc_prng_ahb_clk.clkr, + [GCC_SDCC2_AHB_CLK] = &gcc_sdcc2_ahb_clk.clkr, + [GCC_SDCC2_APPS_CLK] = &gcc_sdcc2_apps_clk.clkr, + [GCC_SDCC4_AHB_CLK] = &gcc_sdcc4_ahb_clk.clkr, + [GCC_SDCC4_APPS_CLK] = &gcc_sdcc4_apps_clk.clkr, + [GCC_TSIF_AHB_CLK] = &gcc_tsif_ahb_clk.clkr, + [GCC_TSIF_INACTIVITY_TIMERS_CLK] = &gcc_tsif_inactivity_timers_clk.clkr, + [GCC_TSIF_REF_CLK] = &gcc_tsif_ref_clk.clkr, + [GCC_UFS_AHB_CLK] = &gcc_ufs_ahb_clk.clkr, + [GCC_UFS_AXI_CLK] = &gcc_ufs_axi_clk.clkr, + [GCC_UFS_ICE_CORE_CLK] = &gcc_ufs_ice_core_clk.clkr, + [GCC_UFS_PHY_AUX_CLK] = &gcc_ufs_phy_aux_clk.clkr, + [GCC_UFS_RX_SYMBOL_0_CLK] = &gcc_ufs_rx_symbol_0_clk.clkr, + [GCC_UFS_RX_SYMBOL_1_CLK] = &gcc_ufs_rx_symbol_1_clk.clkr, + [GCC_UFS_TX_SYMBOL_0_CLK] = &gcc_ufs_tx_symbol_0_clk.clkr, + [GCC_UFS_UNIPRO_CORE_CLK] = &gcc_ufs_unipro_core_clk.clkr, + [GCC_USB30_MASTER_CLK] = &gcc_usb30_master_clk.clkr, + [GCC_USB30_MOCK_UTMI_CLK] = &gcc_usb30_mock_utmi_clk.clkr, + [GCC_USB30_SLEEP_CLK] = &gcc_usb30_sleep_clk.clkr, + [GCC_USB3_PHY_AUX_CLK] = &gcc_usb3_phy_aux_clk.clkr, + [GCC_USB3_PHY_PIPE_CLK] = &gcc_usb3_phy_pipe_clk.clkr, + [GCC_USB_PHY_CFG_AHB2PHY_CLK] = &gcc_usb_phy_cfg_ahb2phy_clk.clkr, + [GP1_CLK_SRC] = &gp1_clk_src.clkr, + [GP2_CLK_SRC] = &gp2_clk_src.clkr, + [GP3_CLK_SRC] = &gp3_clk_src.clkr, + [GPLL0] = &gpll0.clkr, + [GPLL0_OUT_EVEN] = &gpll0_out_even.clkr, + [GPLL0_OUT_MAIN] = &gpll0_out_main.clkr, + [GPLL0_OUT_ODD] = &gpll0_out_odd.clkr, + [GPLL0_OUT_TEST] = &gpll0_out_test.clkr, + [GPLL1] = &gpll1.clkr, + [GPLL1_OUT_EVEN] = &gpll1_out_even.clkr, + [GPLL1_OUT_MAIN] = &gpll1_out_main.clkr, + [GPLL1_OUT_ODD] = &gpll1_out_odd.clkr, + [GPLL1_OUT_TEST] = &gpll1_out_test.clkr, + [GPLL2] = &gpll2.clkr, + [GPLL2_OUT_EVEN] = &gpll2_out_even.clkr, + [GPLL2_OUT_MAIN] = &gpll2_out_main.clkr, + [GPLL2_OUT_ODD] = &gpll2_out_odd.clkr, + [GPLL2_OUT_TEST] = &gpll2_out_test.clkr, + [GPLL3] = &gpll3.clkr, + [GPLL3_OUT_EVEN] = &gpll3_out_even.clkr, + [GPLL3_OUT_MAIN] = &gpll3_out_main.clkr, + [GPLL3_OUT_ODD] = &gpll3_out_odd.clkr, + [GPLL3_OUT_TEST] = &gpll3_out_test.clkr, + [GPLL4] = &gpll4.clkr, + [GPLL4_OUT_EVEN] = &gpll4_out_even.clkr, + [GPLL4_OUT_MAIN] = &gpll4_out_main.clkr, + [GPLL4_OUT_ODD] = &gpll4_out_odd.clkr, + [GPLL4_OUT_TEST] = &gpll4_out_test.clkr, + [HMSS_AHB_CLK_SRC] = &hmss_ahb_clk_src.clkr, + [HMSS_RBCPR_CLK_SRC] = &hmss_rbcpr_clk_src.clkr, + [PCIE_AUX_CLK_SRC] = &pcie_aux_clk_src.clkr, + [PDM2_CLK_SRC] = &pdm2_clk_src.clkr, + [SDCC2_APPS_CLK_SRC] = &sdcc2_apps_clk_src.clkr, + [SDCC4_APPS_CLK_SRC] = &sdcc4_apps_clk_src.clkr, + [TSIF_REF_CLK_SRC] = &tsif_ref_clk_src.clkr, + [UFS_AXI_CLK_SRC] = &ufs_axi_clk_src.clkr, + [USB30_MASTER_CLK_SRC] = &usb30_master_clk_src.clkr, + [USB30_MOCK_UTMI_CLK_SRC] = &usb30_mock_utmi_clk_src.clkr, + [USB3_PHY_AUX_CLK_SRC] = &usb3_phy_aux_clk_src.clkr, +}; + +static struct gdsc *gcc_msm8998_gdscs[] = { + [PCIE_0_GDSC] = &pcie_0_gdsc, + [UFS_GDSC] = &ufs_gdsc, + [USB_30_GDSC] = &usb_30_gdsc, +}; + +static const struct qcom_reset_map gcc_msm8998_resets[] = { + [GCC_BLSP1_QUP1_BCR] = { 0x102400 }, + [GCC_BLSP1_QUP2_BCR] = { 0x110592 }, + [GCC_BLSP1_QUP3_BCR] = { 0x118784 }, + [GCC_BLSP1_QUP4_BCR] = { 0x126976 }, + [GCC_BLSP1_QUP5_BCR] = { 0x135168 }, + [GCC_BLSP1_QUP6_BCR] = { 0x143360 }, + [GCC_BLSP2_QUP1_BCR] = { 0x155648 }, + [GCC_BLSP2_QUP2_BCR] = { 0x163840 }, + [GCC_BLSP2_QUP3_BCR] = { 0x172032 }, + [GCC_BLSP2_QUP4_BCR] = { 0x180224 }, + [GCC_BLSP2_QUP5_BCR] = { 0x188416 }, + [GCC_BLSP2_QUP6_BCR] = { 0x196608 }, + [GCC_PCIE_0_BCR] = { 0x438272 }, + [GCC_PDM_BCR] = { 0x208896 }, + [GCC_SDCC2_BCR] = { 0x81920 }, + [GCC_SDCC4_BCR] = { 0x90112 }, + [GCC_TSIF_BCR] = { 0x221184 }, + [GCC_UFS_BCR] = { 0x479232 }, + [GCC_USB_30_BCR] = { 0x61440 }, +}; + +static const struct regmap_config gcc_msm8998_regmap_config = { + .reg_bits = 32, + .reg_stride = 4, + .val_bits = 32, + .max_register = 0x8f000, + .fast_io = true, +}; + +static const struct qcom_cc_desc gcc_msm8998_desc = { + .config = &gcc_msm8998_regmap_config, + .clks = gcc_msm8998_clocks, + .num_clks = ARRAY_SIZE(gcc_msm8998_clocks), + .resets = gcc_msm8998_resets, + .num_resets = ARRAY_SIZE(gcc_msm8998_resets), + .gdscs = gcc_msm8998_gdscs, + .num_gdscs = ARRAY_SIZE(gcc_msm8998_gdscs), +}; + +static int gcc_msm8998_probe(struct platform_device *pdev) +{ + struct regmap *regmap; + int ret; + + regmap = qcom_cc_map(pdev, &gcc_msm8998_desc); + if (IS_ERR(regmap)) + return PTR_ERR(regmap); + + /* + * Set the HMSS_AHB_CLK_SLEEP_ENA bit to allow the hmss_ahb_clk to be + * turned off by hardware during certain apps low power modes. + */ + ret = regmap_update_bits(regmap, 0x52008, BIT(21), BIT(21)); + if (ret) + return ret; + + return qcom_cc_really_probe(pdev, &gcc_msm8998_desc, regmap); +} + +static const struct of_device_id gcc_msm8998_match_table[] = { + { .compatible = "qcom,gcc-msm8998" }, + { } +}; +MODULE_DEVICE_TABLE(of, gcc_msm8998_match_table); + +static struct platform_driver gcc_msm8998_driver = { + .probe = gcc_msm8998_probe, + .driver = { + .name = "gcc-msm8998", + .of_match_table = gcc_msm8998_match_table, + }, +}; + +static int __init gcc_msm8998_init(void) +{ + return platform_driver_register(&gcc_msm8998_driver); +} +core_initcall(gcc_msm8998_init); + +static void __exit gcc_msm8998_exit(void) +{ + platform_driver_unregister(&gcc_msm8998_driver); +} +module_exit(gcc_msm8998_exit); + +MODULE_DESCRIPTION("QCOM GCC msm8998 Driver"); +MODULE_LICENSE("GPL v2"); +MODULE_ALIAS("platform:gcc-msm8998"); diff --git a/drivers/clk/qcom/gcc-sdm845.c b/drivers/clk/qcom/gcc-sdm845.c new file mode 100644 index 000000000000..e78e6f5b99fc --- /dev/null +++ b/drivers/clk/qcom/gcc-sdm845.c @@ -0,0 +1,3465 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2018, The Linux Foundation. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "common.h" +#include "clk-regmap.h" +#include "clk-pll.h" +#include "clk-rcg.h" +#include "clk-branch.h" +#include "clk-alpha-pll.h" +#include "gdsc.h" +#include "reset.h" + +#define F(f, s, h, m, n) { (f), (s), (2 * (h) - 1), (m), (n) } + +enum { + P_BI_TCXO, + P_AUD_REF_CLK, + P_CORE_BI_PLL_TEST_SE, + P_GPLL0_OUT_EVEN, + P_GPLL0_OUT_MAIN, + P_GPLL4_OUT_MAIN, + P_SLEEP_CLK, +}; + +static const struct parent_map gcc_parent_map_0[] = { + { P_BI_TCXO, 0 }, + { P_GPLL0_OUT_MAIN, 1 }, + { P_GPLL0_OUT_EVEN, 6 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_0[] = { + "bi_tcxo", + "gpll0", + "gpll0_out_even", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_1[] = { + { P_BI_TCXO, 0 }, + { P_GPLL0_OUT_MAIN, 1 }, + { P_SLEEP_CLK, 5 }, + { P_GPLL0_OUT_EVEN, 6 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_1[] = { + "bi_tcxo", + "gpll0", + "core_pi_sleep_clk", + "gpll0_out_even", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_2[] = { + { P_BI_TCXO, 0 }, + { P_SLEEP_CLK, 5 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_2[] = { + "bi_tcxo", + "core_pi_sleep_clk", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_3[] = { + { P_BI_TCXO, 0 }, + { P_GPLL0_OUT_MAIN, 1 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_3[] = { + "bi_tcxo", + "gpll0", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_4[] = { + { P_BI_TCXO, 0 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_4[] = { + "bi_tcxo", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_5[] = { + { P_BI_TCXO, 0 }, + { P_GPLL0_OUT_MAIN, 1 }, + { P_GPLL4_OUT_MAIN, 5 }, + { P_GPLL0_OUT_EVEN, 6 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_5[] = { + "bi_tcxo", + "gpll0", + "gpll4", + "gpll0_out_even", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_6[] = { + { P_BI_TCXO, 0 }, + { P_GPLL0_OUT_MAIN, 1 }, + { P_AUD_REF_CLK, 2 }, + { P_GPLL0_OUT_EVEN, 6 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_6[] = { + "bi_tcxo", + "gpll0", + "aud_ref_clk", + "gpll0_out_even", + "core_bi_pll_test_se", +}; + +static const char * const gcc_parent_names_7[] = { + "bi_tcxo", + "gpll0", + "gpll0_out_even", + "core_bi_pll_test_se", +}; + +static const char * const gcc_parent_names_8[] = { + "bi_tcxo", + "gpll0", + "core_bi_pll_test_se", +}; + +static const struct parent_map gcc_parent_map_10[] = { + { P_BI_TCXO, 0 }, + { P_GPLL0_OUT_MAIN, 1 }, + { P_GPLL4_OUT_MAIN, 5 }, + { P_GPLL0_OUT_EVEN, 6 }, + { P_CORE_BI_PLL_TEST_SE, 7 }, +}; + +static const char * const gcc_parent_names_10[] = { + "bi_tcxo", + "gpll0", + "gpll4", + "gpll0_out_even", + "core_bi_pll_test_se", +}; + +static struct clk_alpha_pll gpll0 = { + .offset = 0x0, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_FABIA], + .clkr = { + .enable_reg = 0x52000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gpll0", + .parent_names = (const char *[]){ "bi_tcxo" }, + .num_parents = 1, + .ops = &clk_alpha_pll_fixed_fabia_ops, + }, + }, +}; + +static struct clk_alpha_pll gpll4 = { + .offset = 0x76000, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_FABIA], + .clkr = { + .enable_reg = 0x52000, + .enable_mask = BIT(4), + .hw.init = &(struct clk_init_data){ + .name = "gpll4", + .parent_names = (const char *[]){ "bi_tcxo" }, + .num_parents = 1, + .ops = &clk_alpha_pll_fixed_fabia_ops, + }, + }, +}; + +static const struct clk_div_table post_div_table_fabia_even[] = { + { 0x0, 1 }, + { 0x1, 2 }, + { 0x3, 4 }, + { 0x7, 8 }, + { } +}; + +static struct clk_alpha_pll_postdiv gpll0_out_even = { + .offset = 0x0, + .post_div_shift = 8, + .post_div_table = post_div_table_fabia_even, + .num_post_div = ARRAY_SIZE(post_div_table_fabia_even), + .width = 4, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_FABIA], + .clkr.hw.init = &(struct clk_init_data){ + .name = "gpll0_out_even", + .parent_names = (const char *[]){ "gpll0" }, + .num_parents = 1, + .ops = &clk_alpha_pll_postdiv_fabia_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_cpuss_ahb_clk_src[] = { + F(19200000, P_BI_TCXO, 1, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_cpuss_ahb_clk_src = { + .cmd_rcgr = 0x48014, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_cpuss_ahb_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_cpuss_ahb_clk_src", + .parent_names = gcc_parent_names_7, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_cpuss_rbcpr_clk_src[] = { + F(19200000, P_BI_TCXO, 1, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_cpuss_rbcpr_clk_src = { + .cmd_rcgr = 0x4815c, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_3, + .freq_tbl = ftbl_gcc_cpuss_rbcpr_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_cpuss_rbcpr_clk_src", + .parent_names = gcc_parent_names_8, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_gp1_clk_src[] = { + F(19200000, P_BI_TCXO, 1, 0, 0), + F(25000000, P_GPLL0_OUT_EVEN, 12, 0, 0), + F(50000000, P_GPLL0_OUT_EVEN, 6, 0, 0), + F(100000000, P_GPLL0_OUT_MAIN, 6, 0, 0), + F(200000000, P_GPLL0_OUT_MAIN, 3, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_gp1_clk_src = { + .cmd_rcgr = 0x64004, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_gcc_gp1_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_gp1_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 5, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 gcc_gp2_clk_src = { + .cmd_rcgr = 0x65004, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_gcc_gp1_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_gp2_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 5, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 gcc_gp3_clk_src = { + .cmd_rcgr = 0x66004, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_1, + .freq_tbl = ftbl_gcc_gp1_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_gp3_clk_src", + .parent_names = gcc_parent_names_1, + .num_parents = 5, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_pcie_0_aux_clk_src[] = { + F(9600000, P_BI_TCXO, 2, 0, 0), + F(19200000, P_BI_TCXO, 1, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_pcie_0_aux_clk_src = { + .cmd_rcgr = 0x6b028, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_2, + .freq_tbl = ftbl_gcc_pcie_0_aux_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_aux_clk_src", + .parent_names = gcc_parent_names_2, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 gcc_pcie_1_aux_clk_src = { + .cmd_rcgr = 0x8d028, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_2, + .freq_tbl = ftbl_gcc_pcie_0_aux_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_1_aux_clk_src", + .parent_names = gcc_parent_names_2, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_pcie_phy_refgen_clk_src[] = { + F(19200000, P_BI_TCXO, 1, 0, 0), + F(100000000, P_GPLL0_OUT_MAIN, 6, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_pcie_phy_refgen_clk_src = { + .cmd_rcgr = 0x6f014, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_pcie_phy_refgen_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_phy_refgen_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_pdm2_clk_src[] = { + F(9600000, P_BI_TCXO, 2, 0, 0), + F(19200000, P_BI_TCXO, 1, 0, 0), + F(60000000, P_GPLL0_OUT_MAIN, 10, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_pdm2_clk_src = { + .cmd_rcgr = 0x33010, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_pdm2_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_pdm2_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_qupv3_wrap0_s0_clk_src[] = { + F(7372800, P_GPLL0_OUT_EVEN, 1, 384, 15625), + F(14745600, P_GPLL0_OUT_EVEN, 1, 768, 15625), + F(19200000, P_BI_TCXO, 1, 0, 0), + F(29491200, P_GPLL0_OUT_EVEN, 1, 1536, 15625), + F(32000000, P_GPLL0_OUT_EVEN, 1, 8, 75), + F(48000000, P_GPLL0_OUT_EVEN, 1, 4, 25), + F(64000000, P_GPLL0_OUT_EVEN, 1, 16, 75), + F(80000000, P_GPLL0_OUT_EVEN, 1, 4, 15), + F(96000000, P_GPLL0_OUT_EVEN, 1, 8, 25), + F(100000000, P_GPLL0_OUT_EVEN, 3, 0, 0), + F(102400000, P_GPLL0_OUT_EVEN, 1, 128, 375), + F(112000000, P_GPLL0_OUT_EVEN, 1, 28, 75), + F(117964800, P_GPLL0_OUT_EVEN, 1, 6144, 15625), + F(120000000, P_GPLL0_OUT_EVEN, 2.5, 0, 0), + F(128000000, P_GPLL0_OUT_MAIN, 1, 16, 75), + { } +}; + +static struct clk_rcg2 gcc_qupv3_wrap0_s0_clk_src = { + .cmd_rcgr = 0x17034, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s0_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap0_s1_clk_src = { + .cmd_rcgr = 0x17164, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s1_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap0_s2_clk_src = { + .cmd_rcgr = 0x17294, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s2_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap0_s3_clk_src = { + .cmd_rcgr = 0x173c4, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s3_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap0_s4_clk_src = { + .cmd_rcgr = 0x174f4, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s4_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap0_s5_clk_src = { + .cmd_rcgr = 0x17624, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s5_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap0_s6_clk_src = { + .cmd_rcgr = 0x17754, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s6_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap0_s7_clk_src = { + .cmd_rcgr = 0x17884, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s7_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap1_s0_clk_src = { + .cmd_rcgr = 0x18018, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s0_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap1_s1_clk_src = { + .cmd_rcgr = 0x18148, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s1_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap1_s2_clk_src = { + .cmd_rcgr = 0x18278, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s2_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap1_s3_clk_src = { + .cmd_rcgr = 0x183a8, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s3_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap1_s4_clk_src = { + .cmd_rcgr = 0x184d8, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s4_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap1_s5_clk_src = { + .cmd_rcgr = 0x18608, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s5_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap1_s6_clk_src = { + .cmd_rcgr = 0x18738, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s6_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_qupv3_wrap1_s7_clk_src = { + .cmd_rcgr = 0x18868, + .mnd_width = 16, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_qupv3_wrap0_s0_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s7_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_sdcc2_apps_clk_src[] = { + F(400000, P_BI_TCXO, 12, 1, 4), + F(9600000, P_BI_TCXO, 2, 0, 0), + F(19200000, P_BI_TCXO, 1, 0, 0), + F(25000000, P_GPLL0_OUT_EVEN, 12, 0, 0), + F(50000000, P_GPLL0_OUT_EVEN, 6, 0, 0), + F(100000000, P_GPLL0_OUT_MAIN, 6, 0, 0), + F(201500000, P_GPLL4_OUT_MAIN, 4, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_sdcc2_apps_clk_src = { + .cmd_rcgr = 0x1400c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_10, + .freq_tbl = ftbl_gcc_sdcc2_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_sdcc2_apps_clk_src", + .parent_names = gcc_parent_names_10, + .num_parents = 5, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_sdcc4_apps_clk_src[] = { + F(400000, P_BI_TCXO, 12, 1, 4), + F(9600000, P_BI_TCXO, 2, 0, 0), + F(19200000, P_BI_TCXO, 1, 0, 0), + F(25000000, P_GPLL0_OUT_MAIN, 12, 1, 2), + F(50000000, P_GPLL0_OUT_MAIN, 12, 0, 0), + F(100000000, P_GPLL0_OUT_MAIN, 6, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_sdcc4_apps_clk_src = { + .cmd_rcgr = 0x1600c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_sdcc4_apps_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_sdcc4_apps_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_tsif_ref_clk_src[] = { + F(105495, P_BI_TCXO, 2, 1, 91), + { } +}; + +static struct clk_rcg2 gcc_tsif_ref_clk_src = { + .cmd_rcgr = 0x36010, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_6, + .freq_tbl = ftbl_gcc_tsif_ref_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_tsif_ref_clk_src", + .parent_names = gcc_parent_names_6, + .num_parents = 5, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_ufs_card_axi_clk_src[] = { + F(25000000, P_GPLL0_OUT_EVEN, 12, 0, 0), + F(50000000, P_GPLL0_OUT_EVEN, 6, 0, 0), + F(100000000, P_GPLL0_OUT_MAIN, 6, 0, 0), + F(200000000, P_GPLL0_OUT_MAIN, 3, 0, 0), + F(240000000, P_GPLL0_OUT_MAIN, 2.5, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_ufs_card_axi_clk_src = { + .cmd_rcgr = 0x7501c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_ufs_card_axi_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_axi_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_ufs_card_ice_core_clk_src[] = { + F(37500000, P_GPLL0_OUT_EVEN, 8, 0, 0), + F(75000000, P_GPLL0_OUT_EVEN, 4, 0, 0), + F(150000000, P_GPLL0_OUT_MAIN, 4, 0, 0), + F(300000000, P_GPLL0_OUT_MAIN, 2, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_ufs_card_ice_core_clk_src = { + .cmd_rcgr = 0x7505c, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_ufs_card_ice_core_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_ice_core_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_ufs_card_phy_aux_clk_src = { + .cmd_rcgr = 0x75090, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_4, + .freq_tbl = ftbl_gcc_cpuss_rbcpr_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_phy_aux_clk_src", + .parent_names = gcc_parent_names_4, + .num_parents = 2, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_ufs_card_unipro_core_clk_src[] = { + F(37500000, P_GPLL0_OUT_EVEN, 8, 0, 0), + F(75000000, P_GPLL0_OUT_MAIN, 8, 0, 0), + F(150000000, P_GPLL0_OUT_MAIN, 4, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_ufs_card_unipro_core_clk_src = { + .cmd_rcgr = 0x75074, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_ufs_card_unipro_core_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_unipro_core_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_ufs_phy_axi_clk_src[] = { + F(25000000, P_GPLL0_OUT_EVEN, 12, 0, 0), + F(50000000, P_GPLL0_OUT_EVEN, 6, 0, 0), + F(100000000, P_GPLL0_OUT_MAIN, 6, 0, 0), + F(200000000, P_GPLL0_OUT_MAIN, 3, 0, 0), + F(240000000, P_GPLL0_OUT_MAIN, 2.5, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_ufs_phy_axi_clk_src = { + .cmd_rcgr = 0x7701c, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_ufs_phy_axi_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_axi_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_ufs_phy_ice_core_clk_src = { + .cmd_rcgr = 0x7705c, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_ufs_card_ice_core_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_ice_core_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_ufs_phy_phy_aux_clk_src = { + .cmd_rcgr = 0x77090, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_4, + .freq_tbl = ftbl_gcc_pcie_0_aux_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_phy_aux_clk_src", + .parent_names = gcc_parent_names_4, + .num_parents = 2, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_ufs_phy_unipro_core_clk_src = { + .cmd_rcgr = 0x77074, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_ufs_card_unipro_core_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_unipro_core_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_usb30_prim_master_clk_src[] = { + F(33333333, P_GPLL0_OUT_EVEN, 9, 0, 0), + F(66666667, P_GPLL0_OUT_EVEN, 4.5, 0, 0), + F(133333333, P_GPLL0_OUT_MAIN, 4.5, 0, 0), + F(200000000, P_GPLL0_OUT_MAIN, 3, 0, 0), + F(240000000, P_GPLL0_OUT_MAIN, 2.5, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_usb30_prim_master_clk_src = { + .cmd_rcgr = 0xf018, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_usb30_prim_master_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_prim_master_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_usb30_prim_mock_utmi_clk_src[] = { + F(19200000, P_BI_TCXO, 1, 0, 0), + F(20000000, P_GPLL0_OUT_EVEN, 15, 0, 0), + F(40000000, P_GPLL0_OUT_EVEN, 7.5, 0, 0), + F(60000000, P_GPLL0_OUT_MAIN, 10, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_usb30_prim_mock_utmi_clk_src = { + .cmd_rcgr = 0xf030, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_usb30_prim_mock_utmi_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_prim_mock_utmi_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_usb30_sec_master_clk_src = { + .cmd_rcgr = 0x10018, + .mnd_width = 8, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_usb30_prim_master_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_sec_master_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 gcc_usb30_sec_mock_utmi_clk_src = { + .cmd_rcgr = 0x10030, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_0, + .freq_tbl = ftbl_gcc_usb30_prim_mock_utmi_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_sec_mock_utmi_clk_src", + .parent_names = gcc_parent_names_0, + .num_parents = 4, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 gcc_usb3_prim_phy_aux_clk_src = { + .cmd_rcgr = 0xf05c, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_2, + .freq_tbl = ftbl_gcc_cpuss_rbcpr_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_prim_phy_aux_clk_src", + .parent_names = gcc_parent_names_2, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_rcg2 gcc_usb3_sec_phy_aux_clk_src = { + .cmd_rcgr = 0x1005c, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_2, + .freq_tbl = ftbl_gcc_cpuss_rbcpr_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_sec_phy_aux_clk_src", + .parent_names = gcc_parent_names_2, + .num_parents = 3, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_rcg2 gcc_vs_ctrl_clk_src = { + .cmd_rcgr = 0x7a030, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_3, + .freq_tbl = ftbl_gcc_cpuss_rbcpr_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_vs_ctrl_clk_src", + .parent_names = gcc_parent_names_3, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static const struct freq_tbl ftbl_gcc_vsensor_clk_src[] = { + F(19200000, P_BI_TCXO, 1, 0, 0), + F(300000000, P_GPLL0_OUT_MAIN, 2, 0, 0), + F(600000000, P_GPLL0_OUT_MAIN, 1, 0, 0), + { } +}; + +static struct clk_rcg2 gcc_vsensor_clk_src = { + .cmd_rcgr = 0x7a018, + .mnd_width = 0, + .hid_width = 5, + .parent_map = gcc_parent_map_3, + .freq_tbl = ftbl_gcc_vsensor_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "gcc_vsensor_clk_src", + .parent_names = gcc_parent_names_8, + .num_parents = 3, + .ops = &clk_rcg2_ops, + }, +}; + +static struct clk_branch gcc_aggre_noc_pcie_tbu_clk = { + .halt_reg = 0x90014, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x90014, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_aggre_noc_pcie_tbu_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_aggre_ufs_card_axi_clk = { + .halt_reg = 0x82028, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x82028, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x82028, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_aggre_ufs_card_axi_clk", + .parent_names = (const char *[]){ + "gcc_ufs_card_axi_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_aggre_ufs_phy_axi_clk = { + .halt_reg = 0x82024, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x82024, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x82024, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_aggre_ufs_phy_axi_clk", + .parent_names = (const char *[]){ + "gcc_ufs_phy_axi_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_aggre_usb3_prim_axi_clk = { + .halt_reg = 0x8201c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8201c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_aggre_usb3_prim_axi_clk", + .parent_names = (const char *[]){ + "gcc_usb30_prim_master_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_aggre_usb3_sec_axi_clk = { + .halt_reg = 0x82020, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x82020, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_aggre_usb3_sec_axi_clk", + .parent_names = (const char *[]){ + "gcc_usb30_sec_master_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_apc_vs_clk = { + .halt_reg = 0x7a050, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x7a050, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_apc_vs_clk", + .parent_names = (const char *[]){ + "gcc_vsensor_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_boot_rom_ahb_clk = { + .halt_reg = 0x38004, + .halt_check = BRANCH_HALT_VOTED, + .hwcg_reg = 0x38004, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(10), + .hw.init = &(struct clk_init_data){ + .name = "gcc_boot_rom_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_camera_ahb_clk = { + .halt_reg = 0xb008, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0xb008, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0xb008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_camera_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_camera_axi_clk = { + .halt_reg = 0xb020, + .halt_check = BRANCH_VOTED, + .clkr = { + .enable_reg = 0xb020, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_camera_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_camera_xo_clk = { + .halt_reg = 0xb02c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0xb02c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_camera_xo_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ce1_ahb_clk = { + .halt_reg = 0x4100c, + .halt_check = BRANCH_HALT_VOTED, + .hwcg_reg = 0x4100c, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(3), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ce1_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ce1_axi_clk = { + .halt_reg = 0x41008, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(4), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ce1_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ce1_clk = { + .halt_reg = 0x41004, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(5), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ce1_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_cfg_noc_usb3_prim_axi_clk = { + .halt_reg = 0x502c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x502c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_cfg_noc_usb3_prim_axi_clk", + .parent_names = (const char *[]){ + "gcc_usb30_prim_master_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_cfg_noc_usb3_sec_axi_clk = { + .halt_reg = 0x5030, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x5030, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_cfg_noc_usb3_sec_axi_clk", + .parent_names = (const char *[]){ + "gcc_usb30_sec_master_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_cpuss_ahb_clk = { + .halt_reg = 0x48000, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(21), + .hw.init = &(struct clk_init_data){ + .name = "gcc_cpuss_ahb_clk", + .parent_names = (const char *[]){ + "gcc_cpuss_ahb_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT | CLK_IS_CRITICAL, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_cpuss_rbcpr_clk = { + .halt_reg = 0x48008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x48008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_cpuss_rbcpr_clk", + .parent_names = (const char *[]){ + "gcc_cpuss_rbcpr_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ddrss_gpu_axi_clk = { + .halt_reg = 0x44038, + .halt_check = BRANCH_VOTED, + .clkr = { + .enable_reg = 0x44038, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ddrss_gpu_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_disp_ahb_clk = { + .halt_reg = 0xb00c, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0xb00c, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0xb00c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_disp_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_disp_axi_clk = { + .halt_reg = 0xb024, + .halt_check = BRANCH_VOTED, + .clkr = { + .enable_reg = 0xb024, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_disp_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_disp_gpll0_clk_src = { + .halt_check = BRANCH_HALT_DELAY, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(18), + .hw.init = &(struct clk_init_data){ + .name = "gcc_disp_gpll0_clk_src", + .parent_names = (const char *[]){ + "gpll0", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_disp_gpll0_div_clk_src = { + .halt_check = BRANCH_HALT_DELAY, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(19), + .hw.init = &(struct clk_init_data){ + .name = "gcc_disp_gpll0_div_clk_src", + .parent_names = (const char *[]){ + "gpll0_out_even", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_disp_xo_clk = { + .halt_reg = 0xb030, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0xb030, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_disp_xo_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gp1_clk = { + .halt_reg = 0x64000, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x64000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gp1_clk", + .parent_names = (const char *[]){ + "gcc_gp1_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gp2_clk = { + .halt_reg = 0x65000, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x65000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gp2_clk", + .parent_names = (const char *[]){ + "gcc_gp2_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gp3_clk = { + .halt_reg = 0x66000, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x66000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gp3_clk", + .parent_names = (const char *[]){ + "gcc_gp3_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gpu_cfg_ahb_clk = { + .halt_reg = 0x71004, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x71004, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x71004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gpu_cfg_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gpu_gpll0_clk_src = { + .halt_check = BRANCH_HALT_DELAY, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(15), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gpu_gpll0_clk_src", + .parent_names = (const char *[]){ + "gpll0", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gpu_gpll0_div_clk_src = { + .halt_check = BRANCH_HALT_DELAY, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(16), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gpu_gpll0_div_clk_src", + .parent_names = (const char *[]){ + "gpll0_out_even", + }, + .num_parents = 1, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gpu_iref_clk = { + .halt_reg = 0x8c010, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8c010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gpu_iref_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gpu_memnoc_gfx_clk = { + .halt_reg = 0x7100c, + .halt_check = BRANCH_VOTED, + .clkr = { + .enable_reg = 0x7100c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gpu_memnoc_gfx_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gpu_snoc_dvm_gfx_clk = { + .halt_reg = 0x71018, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x71018, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gpu_snoc_dvm_gfx_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_gpu_vs_clk = { + .halt_reg = 0x7a04c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x7a04c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_gpu_vs_clk", + .parent_names = (const char *[]){ + "gcc_vsensor_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mss_axis2_clk = { + .halt_reg = 0x8a008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8a008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mss_axis2_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mss_cfg_ahb_clk = { + .halt_reg = 0x8a000, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x8a000, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x8a000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mss_cfg_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mss_gpll0_div_clk_src = { + .halt_check = BRANCH_HALT_DELAY, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(17), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mss_gpll0_div_clk_src", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mss_mfab_axis_clk = { + .halt_reg = 0x8a004, + .halt_check = BRANCH_VOTED, + .hwcg_reg = 0x8a004, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x8a004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mss_mfab_axis_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mss_q6_memnoc_axi_clk = { + .halt_reg = 0x8a154, + .halt_check = BRANCH_VOTED, + .clkr = { + .enable_reg = 0x8a154, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mss_q6_memnoc_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mss_snoc_axi_clk = { + .halt_reg = 0x8a150, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8a150, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mss_snoc_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_mss_vs_clk = { + .halt_reg = 0x7a048, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x7a048, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_mss_vs_clk", + .parent_names = (const char *[]){ + "gcc_vsensor_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_aux_clk = { + .halt_reg = 0x6b01c, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(3), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_aux_clk", + .parent_names = (const char *[]){ + "gcc_pcie_0_aux_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_cfg_ahb_clk = { + .halt_reg = 0x6b018, + .halt_check = BRANCH_HALT_VOTED, + .hwcg_reg = 0x6b018, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(2), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_cfg_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_clkref_clk = { + .halt_reg = 0x8c00c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8c00c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_clkref_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_mstr_axi_clk = { + .halt_reg = 0x6b014, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(1), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_mstr_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_pipe_clk = { + .halt_check = BRANCH_HALT_SKIP, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(4), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_pipe_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_slv_axi_clk = { + .halt_reg = 0x6b010, + .halt_check = BRANCH_HALT_VOTED, + .hwcg_reg = 0x6b010, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_slv_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_0_slv_q2a_axi_clk = { + .halt_reg = 0x6b00c, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(5), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_0_slv_q2a_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_1_aux_clk = { + .halt_reg = 0x8d01c, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(29), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_1_aux_clk", + .parent_names = (const char *[]){ + "gcc_pcie_1_aux_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_1_cfg_ahb_clk = { + .halt_reg = 0x8d018, + .halt_check = BRANCH_HALT_VOTED, + .hwcg_reg = 0x8d018, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(28), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_1_cfg_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_1_clkref_clk = { + .halt_reg = 0x8c02c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8c02c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_1_clkref_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_1_mstr_axi_clk = { + .halt_reg = 0x8d014, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(27), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_1_mstr_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_1_pipe_clk = { + .halt_check = BRANCH_HALT_SKIP, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(30), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_1_pipe_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_1_slv_axi_clk = { + .halt_reg = 0x8d010, + .halt_check = BRANCH_HALT_VOTED, + .hwcg_reg = 0x8d010, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(26), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_1_slv_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_1_slv_q2a_axi_clk = { + .halt_reg = 0x8d00c, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(25), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_1_slv_q2a_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_phy_aux_clk = { + .halt_reg = 0x6f004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x6f004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_phy_aux_clk", + .parent_names = (const char *[]){ + "gcc_pcie_0_aux_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pcie_phy_refgen_clk = { + .halt_reg = 0x6f02c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x6f02c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pcie_phy_refgen_clk", + .parent_names = (const char *[]){ + "gcc_pcie_phy_refgen_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pdm2_clk = { + .halt_reg = 0x3300c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x3300c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pdm2_clk", + .parent_names = (const char *[]){ + "gcc_pdm2_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pdm_ahb_clk = { + .halt_reg = 0x33004, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x33004, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x33004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pdm_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_pdm_xo4_clk = { + .halt_reg = 0x33008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x33008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_pdm_xo4_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_prng_ahb_clk = { + .halt_reg = 0x34004, + .halt_check = BRANCH_HALT_VOTED, + .hwcg_reg = 0x34004, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(13), + .hw.init = &(struct clk_init_data){ + .name = "gcc_prng_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qmip_camera_ahb_clk = { + .halt_reg = 0xb014, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0xb014, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0xb014, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qmip_camera_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qmip_disp_ahb_clk = { + .halt_reg = 0xb018, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0xb018, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0xb018, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qmip_disp_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qmip_video_ahb_clk = { + .halt_reg = 0xb010, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0xb010, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0xb010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qmip_video_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap0_s0_clk = { + .halt_reg = 0x17030, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(10), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s0_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap0_s0_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap0_s1_clk = { + .halt_reg = 0x17160, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(11), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s1_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap0_s1_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap0_s2_clk = { + .halt_reg = 0x17290, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(12), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s2_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap0_s2_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap0_s3_clk = { + .halt_reg = 0x173c0, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(13), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s3_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap0_s3_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap0_s4_clk = { + .halt_reg = 0x174f0, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(14), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s4_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap0_s4_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap0_s5_clk = { + .halt_reg = 0x17620, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(15), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s5_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap0_s5_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap0_s6_clk = { + .halt_reg = 0x17750, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(16), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s6_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap0_s6_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap0_s7_clk = { + .halt_reg = 0x17880, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(17), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap0_s7_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap0_s7_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap1_s0_clk = { + .halt_reg = 0x18014, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(22), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s0_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap1_s0_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap1_s1_clk = { + .halt_reg = 0x18144, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(23), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s1_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap1_s1_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap1_s2_clk = { + .halt_reg = 0x18274, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(24), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s2_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap1_s2_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap1_s3_clk = { + .halt_reg = 0x183a4, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(25), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s3_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap1_s3_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap1_s4_clk = { + .halt_reg = 0x184d4, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(26), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s4_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap1_s4_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap1_s5_clk = { + .halt_reg = 0x18604, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(27), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s5_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap1_s5_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap1_s6_clk = { + .halt_reg = 0x18734, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(28), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s6_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap1_s6_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap1_s7_clk = { + .halt_reg = 0x18864, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(29), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap1_s7_clk", + .parent_names = (const char *[]){ + "gcc_qupv3_wrap1_s7_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap_0_m_ahb_clk = { + .halt_reg = 0x17004, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(6), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap_0_m_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap_0_s_ahb_clk = { + .halt_reg = 0x17008, + .halt_check = BRANCH_HALT_VOTED, + .hwcg_reg = 0x17008, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(7), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap_0_s_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap_1_m_ahb_clk = { + .halt_reg = 0x1800c, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(20), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap_1_m_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_qupv3_wrap_1_s_ahb_clk = { + .halt_reg = 0x18010, + .halt_check = BRANCH_HALT_VOTED, + .hwcg_reg = 0x18010, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x5200c, + .enable_mask = BIT(21), + .hw.init = &(struct clk_init_data){ + .name = "gcc_qupv3_wrap_1_s_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_sdcc2_ahb_clk = { + .halt_reg = 0x14008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x14008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_sdcc2_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_sdcc2_apps_clk = { + .halt_reg = 0x14004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x14004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_sdcc2_apps_clk", + .parent_names = (const char *[]){ + "gcc_sdcc2_apps_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_sdcc4_ahb_clk = { + .halt_reg = 0x16008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x16008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_sdcc4_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_sdcc4_apps_clk = { + .halt_reg = 0x16004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x16004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_sdcc4_apps_clk", + .parent_names = (const char *[]){ + "gcc_sdcc4_apps_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_sys_noc_cpuss_ahb_clk = { + .halt_reg = 0x414c, + .halt_check = BRANCH_HALT_VOTED, + .clkr = { + .enable_reg = 0x52004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_sys_noc_cpuss_ahb_clk", + .parent_names = (const char *[]){ + "gcc_cpuss_ahb_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT | CLK_IS_CRITICAL, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_tsif_ahb_clk = { + .halt_reg = 0x36004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x36004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_tsif_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_tsif_inactivity_timers_clk = { + .halt_reg = 0x3600c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x3600c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_tsif_inactivity_timers_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_tsif_ref_clk = { + .halt_reg = 0x36008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x36008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_tsif_ref_clk", + .parent_names = (const char *[]){ + "gcc_tsif_ref_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_card_ahb_clk = { + .halt_reg = 0x75010, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x75010, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x75010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_card_axi_clk = { + .halt_reg = 0x7500c, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x7500c, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x7500c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_axi_clk", + .parent_names = (const char *[]){ + "gcc_ufs_card_axi_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_card_clkref_clk = { + .halt_reg = 0x8c004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8c004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_clkref_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_card_ice_core_clk = { + .halt_reg = 0x75058, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x75058, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x75058, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_ice_core_clk", + .parent_names = (const char *[]){ + "gcc_ufs_card_ice_core_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_card_phy_aux_clk = { + .halt_reg = 0x7508c, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x7508c, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x7508c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_phy_aux_clk", + .parent_names = (const char *[]){ + "gcc_ufs_card_phy_aux_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_card_rx_symbol_0_clk = { + .halt_check = BRANCH_HALT_SKIP, + .clkr = { + .enable_reg = 0x75018, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_rx_symbol_0_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_card_rx_symbol_1_clk = { + .halt_check = BRANCH_HALT_SKIP, + .clkr = { + .enable_reg = 0x750a8, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_rx_symbol_1_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_card_tx_symbol_0_clk = { + .halt_check = BRANCH_HALT_SKIP, + .clkr = { + .enable_reg = 0x75014, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_tx_symbol_0_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_card_unipro_core_clk = { + .halt_reg = 0x75054, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x75054, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x75054, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_card_unipro_core_clk", + .parent_names = (const char *[]){ + "gcc_ufs_card_unipro_core_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_mem_clkref_clk = { + .halt_reg = 0x8c000, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8c000, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_mem_clkref_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_phy_ahb_clk = { + .halt_reg = 0x77010, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x77010, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x77010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_phy_axi_clk = { + .halt_reg = 0x7700c, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x7700c, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x7700c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_axi_clk", + .parent_names = (const char *[]){ + "gcc_ufs_phy_axi_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_phy_ice_core_clk = { + .halt_reg = 0x77058, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x77058, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x77058, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_ice_core_clk", + .parent_names = (const char *[]){ + "gcc_ufs_phy_ice_core_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_phy_phy_aux_clk = { + .halt_reg = 0x7708c, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x7708c, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x7708c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_phy_aux_clk", + .parent_names = (const char *[]){ + "gcc_ufs_phy_phy_aux_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_phy_rx_symbol_0_clk = { + .halt_check = BRANCH_HALT_SKIP, + .clkr = { + .enable_reg = 0x77018, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_rx_symbol_0_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_phy_rx_symbol_1_clk = { + .halt_check = BRANCH_HALT_SKIP, + .clkr = { + .enable_reg = 0x770a8, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_rx_symbol_1_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_phy_tx_symbol_0_clk = { + .halt_check = BRANCH_HALT_SKIP, + .clkr = { + .enable_reg = 0x77014, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_tx_symbol_0_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_ufs_phy_unipro_core_clk = { + .halt_reg = 0x77054, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x77054, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x77054, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_ufs_phy_unipro_core_clk", + .parent_names = (const char *[]){ + "gcc_ufs_phy_unipro_core_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb30_prim_master_clk = { + .halt_reg = 0xf00c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0xf00c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_prim_master_clk", + .parent_names = (const char *[]){ + "gcc_usb30_prim_master_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb30_prim_mock_utmi_clk = { + .halt_reg = 0xf014, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0xf014, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_prim_mock_utmi_clk", + .parent_names = (const char *[]){ + "gcc_usb30_prim_mock_utmi_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb30_prim_sleep_clk = { + .halt_reg = 0xf010, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0xf010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_prim_sleep_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb30_sec_master_clk = { + .halt_reg = 0x1000c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x1000c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_sec_master_clk", + .parent_names = (const char *[]){ + "gcc_usb30_sec_master_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb30_sec_mock_utmi_clk = { + .halt_reg = 0x10014, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x10014, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_sec_mock_utmi_clk", + .parent_names = (const char *[]){ + "gcc_usb30_sec_mock_utmi_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb30_sec_sleep_clk = { + .halt_reg = 0x10010, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x10010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb30_sec_sleep_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb3_prim_clkref_clk = { + .halt_reg = 0x8c008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8c008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_prim_clkref_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb3_prim_phy_aux_clk = { + .halt_reg = 0xf04c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0xf04c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_prim_phy_aux_clk", + .parent_names = (const char *[]){ + "gcc_usb3_prim_phy_aux_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb3_prim_phy_com_aux_clk = { + .halt_reg = 0xf050, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0xf050, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_prim_phy_com_aux_clk", + .parent_names = (const char *[]){ + "gcc_usb3_prim_phy_aux_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb3_prim_phy_pipe_clk = { + .halt_check = BRANCH_HALT_SKIP, + .clkr = { + .enable_reg = 0xf054, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_prim_phy_pipe_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb3_sec_clkref_clk = { + .halt_reg = 0x8c028, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x8c028, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_sec_clkref_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb3_sec_phy_aux_clk = { + .halt_reg = 0x1004c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x1004c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_sec_phy_aux_clk", + .parent_names = (const char *[]){ + "gcc_usb3_sec_phy_aux_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb3_sec_phy_com_aux_clk = { + .halt_reg = 0x10050, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x10050, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_sec_phy_com_aux_clk", + .parent_names = (const char *[]){ + "gcc_usb3_sec_phy_aux_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb3_sec_phy_pipe_clk = { + .halt_check = BRANCH_HALT_SKIP, + .clkr = { + .enable_reg = 0x10054, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb3_sec_phy_pipe_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_usb_phy_cfg_ahb2phy_clk = { + .halt_reg = 0x6a004, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x6a004, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x6a004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_usb_phy_cfg_ahb2phy_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_vdda_vs_clk = { + .halt_reg = 0x7a00c, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x7a00c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_vdda_vs_clk", + .parent_names = (const char *[]){ + "gcc_vsensor_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_vddcx_vs_clk = { + .halt_reg = 0x7a004, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x7a004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_vddcx_vs_clk", + .parent_names = (const char *[]){ + "gcc_vsensor_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_vddmx_vs_clk = { + .halt_reg = 0x7a008, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x7a008, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_vddmx_vs_clk", + .parent_names = (const char *[]){ + "gcc_vsensor_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_video_ahb_clk = { + .halt_reg = 0xb004, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0xb004, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0xb004, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_video_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_video_axi_clk = { + .halt_reg = 0xb01c, + .halt_check = BRANCH_VOTED, + .clkr = { + .enable_reg = 0xb01c, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_video_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_video_xo_clk = { + .halt_reg = 0xb028, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0xb028, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_video_xo_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_vs_ctrl_ahb_clk = { + .halt_reg = 0x7a014, + .halt_check = BRANCH_HALT, + .hwcg_reg = 0x7a014, + .hwcg_bit = 1, + .clkr = { + .enable_reg = 0x7a014, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_vs_ctrl_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch gcc_vs_ctrl_clk = { + .halt_reg = 0x7a010, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x7a010, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "gcc_vs_ctrl_clk", + .parent_names = (const char *[]){ + "gcc_vs_ctrl_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct gdsc pcie_0_gdsc = { + .gdscr = 0x6b004, + .pd = { + .name = "pcie_0_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, + .flags = POLL_CFG_GDSCR, +}; + +static struct gdsc pcie_1_gdsc = { + .gdscr = 0x8d004, + .pd = { + .name = "pcie_1_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, + .flags = POLL_CFG_GDSCR, +}; + +static struct gdsc ufs_card_gdsc = { + .gdscr = 0x75004, + .pd = { + .name = "ufs_card_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, + .flags = POLL_CFG_GDSCR, +}; + +static struct gdsc ufs_phy_gdsc = { + .gdscr = 0x77004, + .pd = { + .name = "ufs_phy_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, + .flags = POLL_CFG_GDSCR, +}; + +static struct gdsc usb30_prim_gdsc = { + .gdscr = 0xf004, + .pd = { + .name = "usb30_prim_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, + .flags = POLL_CFG_GDSCR, +}; + +static struct gdsc usb30_sec_gdsc = { + .gdscr = 0x10004, + .pd = { + .name = "usb30_sec_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, + .flags = POLL_CFG_GDSCR, +}; + +static struct gdsc hlos1_vote_aggre_noc_mmu_audio_tbu_gdsc = { + .gdscr = 0x7d030, + .pd = { + .name = "hlos1_vote_aggre_noc_mmu_audio_tbu_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, +}; + +static struct gdsc hlos1_vote_aggre_noc_mmu_pcie_tbu_gdsc = { + .gdscr = 0x7d03c, + .pd = { + .name = "hlos1_vote_aggre_noc_mmu_pcie_tbu_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, +}; + +static struct gdsc hlos1_vote_aggre_noc_mmu_tbu1_gdsc = { + .gdscr = 0x7d034, + .pd = { + .name = "hlos1_vote_aggre_noc_mmu_tbu1_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, +}; + +static struct gdsc hlos1_vote_aggre_noc_mmu_tbu2_gdsc = { + .gdscr = 0x7d038, + .pd = { + .name = "hlos1_vote_aggre_noc_mmu_tbu2_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, +}; + +static struct gdsc hlos1_vote_mmnoc_mmu_tbu_hf0_gdsc = { + .gdscr = 0x7d040, + .pd = { + .name = "hlos1_vote_mmnoc_mmu_tbu_hf0_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, +}; + +static struct gdsc hlos1_vote_mmnoc_mmu_tbu_hf1_gdsc = { + .gdscr = 0x7d048, + .pd = { + .name = "hlos1_vote_mmnoc_mmu_tbu_hf1_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, +}; + +static struct gdsc hlos1_vote_mmnoc_mmu_tbu_sf_gdsc = { + .gdscr = 0x7d044, + .pd = { + .name = "hlos1_vote_mmnoc_mmu_tbu_sf_gdsc", + }, + .pwrsts = PWRSTS_OFF_ON, +}; + +static struct clk_regmap *gcc_sdm845_clocks[] = { + [GCC_AGGRE_NOC_PCIE_TBU_CLK] = &gcc_aggre_noc_pcie_tbu_clk.clkr, + [GCC_AGGRE_UFS_CARD_AXI_CLK] = &gcc_aggre_ufs_card_axi_clk.clkr, + [GCC_AGGRE_UFS_PHY_AXI_CLK] = &gcc_aggre_ufs_phy_axi_clk.clkr, + [GCC_AGGRE_USB3_PRIM_AXI_CLK] = &gcc_aggre_usb3_prim_axi_clk.clkr, + [GCC_AGGRE_USB3_SEC_AXI_CLK] = &gcc_aggre_usb3_sec_axi_clk.clkr, + [GCC_APC_VS_CLK] = &gcc_apc_vs_clk.clkr, + [GCC_BOOT_ROM_AHB_CLK] = &gcc_boot_rom_ahb_clk.clkr, + [GCC_CAMERA_AHB_CLK] = &gcc_camera_ahb_clk.clkr, + [GCC_CAMERA_AXI_CLK] = &gcc_camera_axi_clk.clkr, + [GCC_CAMERA_XO_CLK] = &gcc_camera_xo_clk.clkr, + [GCC_CE1_AHB_CLK] = &gcc_ce1_ahb_clk.clkr, + [GCC_CE1_AXI_CLK] = &gcc_ce1_axi_clk.clkr, + [GCC_CE1_CLK] = &gcc_ce1_clk.clkr, + [GCC_CFG_NOC_USB3_PRIM_AXI_CLK] = &gcc_cfg_noc_usb3_prim_axi_clk.clkr, + [GCC_CFG_NOC_USB3_SEC_AXI_CLK] = &gcc_cfg_noc_usb3_sec_axi_clk.clkr, + [GCC_CPUSS_AHB_CLK] = &gcc_cpuss_ahb_clk.clkr, + [GCC_CPUSS_AHB_CLK_SRC] = &gcc_cpuss_ahb_clk_src.clkr, + [GCC_CPUSS_RBCPR_CLK] = &gcc_cpuss_rbcpr_clk.clkr, + [GCC_CPUSS_RBCPR_CLK_SRC] = &gcc_cpuss_rbcpr_clk_src.clkr, + [GCC_DDRSS_GPU_AXI_CLK] = &gcc_ddrss_gpu_axi_clk.clkr, + [GCC_DISP_AHB_CLK] = &gcc_disp_ahb_clk.clkr, + [GCC_DISP_AXI_CLK] = &gcc_disp_axi_clk.clkr, + [GCC_DISP_GPLL0_CLK_SRC] = &gcc_disp_gpll0_clk_src.clkr, + [GCC_DISP_GPLL0_DIV_CLK_SRC] = &gcc_disp_gpll0_div_clk_src.clkr, + [GCC_DISP_XO_CLK] = &gcc_disp_xo_clk.clkr, + [GCC_GP1_CLK] = &gcc_gp1_clk.clkr, + [GCC_GP1_CLK_SRC] = &gcc_gp1_clk_src.clkr, + [GCC_GP2_CLK] = &gcc_gp2_clk.clkr, + [GCC_GP2_CLK_SRC] = &gcc_gp2_clk_src.clkr, + [GCC_GP3_CLK] = &gcc_gp3_clk.clkr, + [GCC_GP3_CLK_SRC] = &gcc_gp3_clk_src.clkr, + [GCC_GPU_CFG_AHB_CLK] = &gcc_gpu_cfg_ahb_clk.clkr, + [GCC_GPU_GPLL0_CLK_SRC] = &gcc_gpu_gpll0_clk_src.clkr, + [GCC_GPU_GPLL0_DIV_CLK_SRC] = &gcc_gpu_gpll0_div_clk_src.clkr, + [GCC_GPU_IREF_CLK] = &gcc_gpu_iref_clk.clkr, + [GCC_GPU_MEMNOC_GFX_CLK] = &gcc_gpu_memnoc_gfx_clk.clkr, + [GCC_GPU_SNOC_DVM_GFX_CLK] = &gcc_gpu_snoc_dvm_gfx_clk.clkr, + [GCC_GPU_VS_CLK] = &gcc_gpu_vs_clk.clkr, + [GCC_MSS_AXIS2_CLK] = &gcc_mss_axis2_clk.clkr, + [GCC_MSS_CFG_AHB_CLK] = &gcc_mss_cfg_ahb_clk.clkr, + [GCC_MSS_GPLL0_DIV_CLK_SRC] = &gcc_mss_gpll0_div_clk_src.clkr, + [GCC_MSS_MFAB_AXIS_CLK] = &gcc_mss_mfab_axis_clk.clkr, + [GCC_MSS_Q6_MEMNOC_AXI_CLK] = &gcc_mss_q6_memnoc_axi_clk.clkr, + [GCC_MSS_SNOC_AXI_CLK] = &gcc_mss_snoc_axi_clk.clkr, + [GCC_MSS_VS_CLK] = &gcc_mss_vs_clk.clkr, + [GCC_PCIE_0_AUX_CLK] = &gcc_pcie_0_aux_clk.clkr, + [GCC_PCIE_0_AUX_CLK_SRC] = &gcc_pcie_0_aux_clk_src.clkr, + [GCC_PCIE_0_CFG_AHB_CLK] = &gcc_pcie_0_cfg_ahb_clk.clkr, + [GCC_PCIE_0_CLKREF_CLK] = &gcc_pcie_0_clkref_clk.clkr, + [GCC_PCIE_0_MSTR_AXI_CLK] = &gcc_pcie_0_mstr_axi_clk.clkr, + [GCC_PCIE_0_PIPE_CLK] = &gcc_pcie_0_pipe_clk.clkr, + [GCC_PCIE_0_SLV_AXI_CLK] = &gcc_pcie_0_slv_axi_clk.clkr, + [GCC_PCIE_0_SLV_Q2A_AXI_CLK] = &gcc_pcie_0_slv_q2a_axi_clk.clkr, + [GCC_PCIE_1_AUX_CLK] = &gcc_pcie_1_aux_clk.clkr, + [GCC_PCIE_1_AUX_CLK_SRC] = &gcc_pcie_1_aux_clk_src.clkr, + [GCC_PCIE_1_CFG_AHB_CLK] = &gcc_pcie_1_cfg_ahb_clk.clkr, + [GCC_PCIE_1_CLKREF_CLK] = &gcc_pcie_1_clkref_clk.clkr, + [GCC_PCIE_1_MSTR_AXI_CLK] = &gcc_pcie_1_mstr_axi_clk.clkr, + [GCC_PCIE_1_PIPE_CLK] = &gcc_pcie_1_pipe_clk.clkr, + [GCC_PCIE_1_SLV_AXI_CLK] = &gcc_pcie_1_slv_axi_clk.clkr, + [GCC_PCIE_1_SLV_Q2A_AXI_CLK] = &gcc_pcie_1_slv_q2a_axi_clk.clkr, + [GCC_PCIE_PHY_AUX_CLK] = &gcc_pcie_phy_aux_clk.clkr, + [GCC_PCIE_PHY_REFGEN_CLK] = &gcc_pcie_phy_refgen_clk.clkr, + [GCC_PCIE_PHY_REFGEN_CLK_SRC] = &gcc_pcie_phy_refgen_clk_src.clkr, + [GCC_PDM2_CLK] = &gcc_pdm2_clk.clkr, + [GCC_PDM2_CLK_SRC] = &gcc_pdm2_clk_src.clkr, + [GCC_PDM_AHB_CLK] = &gcc_pdm_ahb_clk.clkr, + [GCC_PDM_XO4_CLK] = &gcc_pdm_xo4_clk.clkr, + [GCC_PRNG_AHB_CLK] = &gcc_prng_ahb_clk.clkr, + [GCC_QMIP_CAMERA_AHB_CLK] = &gcc_qmip_camera_ahb_clk.clkr, + [GCC_QMIP_DISP_AHB_CLK] = &gcc_qmip_disp_ahb_clk.clkr, + [GCC_QMIP_VIDEO_AHB_CLK] = &gcc_qmip_video_ahb_clk.clkr, + [GCC_QUPV3_WRAP0_S0_CLK] = &gcc_qupv3_wrap0_s0_clk.clkr, + [GCC_QUPV3_WRAP0_S0_CLK_SRC] = &gcc_qupv3_wrap0_s0_clk_src.clkr, + [GCC_QUPV3_WRAP0_S1_CLK] = &gcc_qupv3_wrap0_s1_clk.clkr, + [GCC_QUPV3_WRAP0_S1_CLK_SRC] = &gcc_qupv3_wrap0_s1_clk_src.clkr, + [GCC_QUPV3_WRAP0_S2_CLK] = &gcc_qupv3_wrap0_s2_clk.clkr, + [GCC_QUPV3_WRAP0_S2_CLK_SRC] = &gcc_qupv3_wrap0_s2_clk_src.clkr, + [GCC_QUPV3_WRAP0_S3_CLK] = &gcc_qupv3_wrap0_s3_clk.clkr, + [GCC_QUPV3_WRAP0_S3_CLK_SRC] = &gcc_qupv3_wrap0_s3_clk_src.clkr, + [GCC_QUPV3_WRAP0_S4_CLK] = &gcc_qupv3_wrap0_s4_clk.clkr, + [GCC_QUPV3_WRAP0_S4_CLK_SRC] = &gcc_qupv3_wrap0_s4_clk_src.clkr, + [GCC_QUPV3_WRAP0_S5_CLK] = &gcc_qupv3_wrap0_s5_clk.clkr, + [GCC_QUPV3_WRAP0_S5_CLK_SRC] = &gcc_qupv3_wrap0_s5_clk_src.clkr, + [GCC_QUPV3_WRAP0_S6_CLK] = &gcc_qupv3_wrap0_s6_clk.clkr, + [GCC_QUPV3_WRAP0_S6_CLK_SRC] = &gcc_qupv3_wrap0_s6_clk_src.clkr, + [GCC_QUPV3_WRAP0_S7_CLK] = &gcc_qupv3_wrap0_s7_clk.clkr, + [GCC_QUPV3_WRAP0_S7_CLK_SRC] = &gcc_qupv3_wrap0_s7_clk_src.clkr, + [GCC_QUPV3_WRAP1_S0_CLK] = &gcc_qupv3_wrap1_s0_clk.clkr, + [GCC_QUPV3_WRAP1_S0_CLK_SRC] = &gcc_qupv3_wrap1_s0_clk_src.clkr, + [GCC_QUPV3_WRAP1_S1_CLK] = &gcc_qupv3_wrap1_s1_clk.clkr, + [GCC_QUPV3_WRAP1_S1_CLK_SRC] = &gcc_qupv3_wrap1_s1_clk_src.clkr, + [GCC_QUPV3_WRAP1_S2_CLK] = &gcc_qupv3_wrap1_s2_clk.clkr, + [GCC_QUPV3_WRAP1_S2_CLK_SRC] = &gcc_qupv3_wrap1_s2_clk_src.clkr, + [GCC_QUPV3_WRAP1_S3_CLK] = &gcc_qupv3_wrap1_s3_clk.clkr, + [GCC_QUPV3_WRAP1_S3_CLK_SRC] = &gcc_qupv3_wrap1_s3_clk_src.clkr, + [GCC_QUPV3_WRAP1_S4_CLK] = &gcc_qupv3_wrap1_s4_clk.clkr, + [GCC_QUPV3_WRAP1_S4_CLK_SRC] = &gcc_qupv3_wrap1_s4_clk_src.clkr, + [GCC_QUPV3_WRAP1_S5_CLK] = &gcc_qupv3_wrap1_s5_clk.clkr, + [GCC_QUPV3_WRAP1_S5_CLK_SRC] = &gcc_qupv3_wrap1_s5_clk_src.clkr, + [GCC_QUPV3_WRAP1_S6_CLK] = &gcc_qupv3_wrap1_s6_clk.clkr, + [GCC_QUPV3_WRAP1_S6_CLK_SRC] = &gcc_qupv3_wrap1_s6_clk_src.clkr, + [GCC_QUPV3_WRAP1_S7_CLK] = &gcc_qupv3_wrap1_s7_clk.clkr, + [GCC_QUPV3_WRAP1_S7_CLK_SRC] = &gcc_qupv3_wrap1_s7_clk_src.clkr, + [GCC_QUPV3_WRAP_0_M_AHB_CLK] = &gcc_qupv3_wrap_0_m_ahb_clk.clkr, + [GCC_QUPV3_WRAP_0_S_AHB_CLK] = &gcc_qupv3_wrap_0_s_ahb_clk.clkr, + [GCC_QUPV3_WRAP_1_M_AHB_CLK] = &gcc_qupv3_wrap_1_m_ahb_clk.clkr, + [GCC_QUPV3_WRAP_1_S_AHB_CLK] = &gcc_qupv3_wrap_1_s_ahb_clk.clkr, + [GCC_SDCC2_AHB_CLK] = &gcc_sdcc2_ahb_clk.clkr, + [GCC_SDCC2_APPS_CLK] = &gcc_sdcc2_apps_clk.clkr, + [GCC_SDCC2_APPS_CLK_SRC] = &gcc_sdcc2_apps_clk_src.clkr, + [GCC_SDCC4_AHB_CLK] = &gcc_sdcc4_ahb_clk.clkr, + [GCC_SDCC4_APPS_CLK] = &gcc_sdcc4_apps_clk.clkr, + [GCC_SDCC4_APPS_CLK_SRC] = &gcc_sdcc4_apps_clk_src.clkr, + [GCC_SYS_NOC_CPUSS_AHB_CLK] = &gcc_sys_noc_cpuss_ahb_clk.clkr, + [GCC_TSIF_AHB_CLK] = &gcc_tsif_ahb_clk.clkr, + [GCC_TSIF_INACTIVITY_TIMERS_CLK] = + &gcc_tsif_inactivity_timers_clk.clkr, + [GCC_TSIF_REF_CLK] = &gcc_tsif_ref_clk.clkr, + [GCC_TSIF_REF_CLK_SRC] = &gcc_tsif_ref_clk_src.clkr, + [GCC_UFS_CARD_AHB_CLK] = &gcc_ufs_card_ahb_clk.clkr, + [GCC_UFS_CARD_AXI_CLK] = &gcc_ufs_card_axi_clk.clkr, + [GCC_UFS_CARD_AXI_CLK_SRC] = &gcc_ufs_card_axi_clk_src.clkr, + [GCC_UFS_CARD_CLKREF_CLK] = &gcc_ufs_card_clkref_clk.clkr, + [GCC_UFS_CARD_ICE_CORE_CLK] = &gcc_ufs_card_ice_core_clk.clkr, + [GCC_UFS_CARD_ICE_CORE_CLK_SRC] = &gcc_ufs_card_ice_core_clk_src.clkr, + [GCC_UFS_CARD_PHY_AUX_CLK] = &gcc_ufs_card_phy_aux_clk.clkr, + [GCC_UFS_CARD_PHY_AUX_CLK_SRC] = &gcc_ufs_card_phy_aux_clk_src.clkr, + [GCC_UFS_CARD_RX_SYMBOL_0_CLK] = &gcc_ufs_card_rx_symbol_0_clk.clkr, + [GCC_UFS_CARD_RX_SYMBOL_1_CLK] = &gcc_ufs_card_rx_symbol_1_clk.clkr, + [GCC_UFS_CARD_TX_SYMBOL_0_CLK] = &gcc_ufs_card_tx_symbol_0_clk.clkr, + [GCC_UFS_CARD_UNIPRO_CORE_CLK] = &gcc_ufs_card_unipro_core_clk.clkr, + [GCC_UFS_CARD_UNIPRO_CORE_CLK_SRC] = + &gcc_ufs_card_unipro_core_clk_src.clkr, + [GCC_UFS_MEM_CLKREF_CLK] = &gcc_ufs_mem_clkref_clk.clkr, + [GCC_UFS_PHY_AHB_CLK] = &gcc_ufs_phy_ahb_clk.clkr, + [GCC_UFS_PHY_AXI_CLK] = &gcc_ufs_phy_axi_clk.clkr, + [GCC_UFS_PHY_AXI_CLK_SRC] = &gcc_ufs_phy_axi_clk_src.clkr, + [GCC_UFS_PHY_ICE_CORE_CLK] = &gcc_ufs_phy_ice_core_clk.clkr, + [GCC_UFS_PHY_ICE_CORE_CLK_SRC] = &gcc_ufs_phy_ice_core_clk_src.clkr, + [GCC_UFS_PHY_PHY_AUX_CLK] = &gcc_ufs_phy_phy_aux_clk.clkr, + [GCC_UFS_PHY_PHY_AUX_CLK_SRC] = &gcc_ufs_phy_phy_aux_clk_src.clkr, + [GCC_UFS_PHY_RX_SYMBOL_0_CLK] = &gcc_ufs_phy_rx_symbol_0_clk.clkr, + [GCC_UFS_PHY_RX_SYMBOL_1_CLK] = &gcc_ufs_phy_rx_symbol_1_clk.clkr, + [GCC_UFS_PHY_TX_SYMBOL_0_CLK] = &gcc_ufs_phy_tx_symbol_0_clk.clkr, + [GCC_UFS_PHY_UNIPRO_CORE_CLK] = &gcc_ufs_phy_unipro_core_clk.clkr, + [GCC_UFS_PHY_UNIPRO_CORE_CLK_SRC] = + &gcc_ufs_phy_unipro_core_clk_src.clkr, + [GCC_USB30_PRIM_MASTER_CLK] = &gcc_usb30_prim_master_clk.clkr, + [GCC_USB30_PRIM_MASTER_CLK_SRC] = &gcc_usb30_prim_master_clk_src.clkr, + [GCC_USB30_PRIM_MOCK_UTMI_CLK] = &gcc_usb30_prim_mock_utmi_clk.clkr, + [GCC_USB30_PRIM_MOCK_UTMI_CLK_SRC] = + &gcc_usb30_prim_mock_utmi_clk_src.clkr, + [GCC_USB30_PRIM_SLEEP_CLK] = &gcc_usb30_prim_sleep_clk.clkr, + [GCC_USB30_SEC_MASTER_CLK] = &gcc_usb30_sec_master_clk.clkr, + [GCC_USB30_SEC_MASTER_CLK_SRC] = &gcc_usb30_sec_master_clk_src.clkr, + [GCC_USB30_SEC_MOCK_UTMI_CLK] = &gcc_usb30_sec_mock_utmi_clk.clkr, + [GCC_USB30_SEC_MOCK_UTMI_CLK_SRC] = + &gcc_usb30_sec_mock_utmi_clk_src.clkr, + [GCC_USB30_SEC_SLEEP_CLK] = &gcc_usb30_sec_sleep_clk.clkr, + [GCC_USB3_PRIM_CLKREF_CLK] = &gcc_usb3_prim_clkref_clk.clkr, + [GCC_USB3_PRIM_PHY_AUX_CLK] = &gcc_usb3_prim_phy_aux_clk.clkr, + [GCC_USB3_PRIM_PHY_AUX_CLK_SRC] = &gcc_usb3_prim_phy_aux_clk_src.clkr, + [GCC_USB3_PRIM_PHY_COM_AUX_CLK] = &gcc_usb3_prim_phy_com_aux_clk.clkr, + [GCC_USB3_PRIM_PHY_PIPE_CLK] = &gcc_usb3_prim_phy_pipe_clk.clkr, + [GCC_USB3_SEC_CLKREF_CLK] = &gcc_usb3_sec_clkref_clk.clkr, + [GCC_USB3_SEC_PHY_AUX_CLK] = &gcc_usb3_sec_phy_aux_clk.clkr, + [GCC_USB3_SEC_PHY_AUX_CLK_SRC] = &gcc_usb3_sec_phy_aux_clk_src.clkr, + [GCC_USB3_SEC_PHY_COM_AUX_CLK] = &gcc_usb3_sec_phy_com_aux_clk.clkr, + [GCC_USB3_SEC_PHY_PIPE_CLK] = &gcc_usb3_sec_phy_pipe_clk.clkr, + [GCC_USB_PHY_CFG_AHB2PHY_CLK] = &gcc_usb_phy_cfg_ahb2phy_clk.clkr, + [GCC_VDDA_VS_CLK] = &gcc_vdda_vs_clk.clkr, + [GCC_VDDCX_VS_CLK] = &gcc_vddcx_vs_clk.clkr, + [GCC_VDDMX_VS_CLK] = &gcc_vddmx_vs_clk.clkr, + [GCC_VIDEO_AHB_CLK] = &gcc_video_ahb_clk.clkr, + [GCC_VIDEO_AXI_CLK] = &gcc_video_axi_clk.clkr, + [GCC_VIDEO_XO_CLK] = &gcc_video_xo_clk.clkr, + [GCC_VS_CTRL_AHB_CLK] = &gcc_vs_ctrl_ahb_clk.clkr, + [GCC_VS_CTRL_CLK] = &gcc_vs_ctrl_clk.clkr, + [GCC_VS_CTRL_CLK_SRC] = &gcc_vs_ctrl_clk_src.clkr, + [GCC_VSENSOR_CLK_SRC] = &gcc_vsensor_clk_src.clkr, + [GPLL0] = &gpll0.clkr, + [GPLL0_OUT_EVEN] = &gpll0_out_even.clkr, + [GPLL4] = &gpll4.clkr, +}; + +static const struct qcom_reset_map gcc_sdm845_resets[] = { + [GCC_MMSS_BCR] = { 0xb000 }, + [GCC_PCIE_0_BCR] = { 0x6b000 }, + [GCC_PCIE_1_BCR] = { 0x8d000 }, + [GCC_PCIE_PHY_BCR] = { 0x6f000 }, + [GCC_PDM_BCR] = { 0x33000 }, + [GCC_PRNG_BCR] = { 0x34000 }, + [GCC_QUPV3_WRAPPER_0_BCR] = { 0x17000 }, + [GCC_QUPV3_WRAPPER_1_BCR] = { 0x18000 }, + [GCC_QUSB2PHY_PRIM_BCR] = { 0x12000 }, + [GCC_QUSB2PHY_SEC_BCR] = { 0x12004 }, + [GCC_SDCC2_BCR] = { 0x14000 }, + [GCC_SDCC4_BCR] = { 0x16000 }, + [GCC_TSIF_BCR] = { 0x36000 }, + [GCC_UFS_CARD_BCR] = { 0x75000 }, + [GCC_UFS_PHY_BCR] = { 0x77000 }, + [GCC_USB30_PRIM_BCR] = { 0xf000 }, + [GCC_USB30_SEC_BCR] = { 0x10000 }, + [GCC_USB3_PHY_PRIM_BCR] = { 0x50000 }, + [GCC_USB3PHY_PHY_PRIM_BCR] = { 0x50004 }, + [GCC_USB3_DP_PHY_PRIM_BCR] = { 0x50008 }, + [GCC_USB3_PHY_SEC_BCR] = { 0x5000c }, + [GCC_USB3PHY_PHY_SEC_BCR] = { 0x50010 }, + [GCC_USB3_DP_PHY_SEC_BCR] = { 0x50014 }, + [GCC_USB_PHY_CFG_AHB2PHY_BCR] = { 0x6a000 }, + [GCC_PCIE_0_PHY_BCR] = { 0x6c01c }, + [GCC_PCIE_1_PHY_BCR] = { 0x8e01c }, +}; + +static struct gdsc *gcc_sdm845_gdscs[] = { + [PCIE_0_GDSC] = &pcie_0_gdsc, + [PCIE_1_GDSC] = &pcie_1_gdsc, + [UFS_CARD_GDSC] = &ufs_card_gdsc, + [UFS_PHY_GDSC] = &ufs_phy_gdsc, + [USB30_PRIM_GDSC] = &usb30_prim_gdsc, + [USB30_SEC_GDSC] = &usb30_sec_gdsc, + [HLOS1_VOTE_AGGRE_NOC_MMU_AUDIO_TBU_GDSC] = + &hlos1_vote_aggre_noc_mmu_audio_tbu_gdsc, + [HLOS1_VOTE_AGGRE_NOC_MMU_PCIE_TBU_GDSC] = + &hlos1_vote_aggre_noc_mmu_pcie_tbu_gdsc, + [HLOS1_VOTE_AGGRE_NOC_MMU_TBU1_GDSC] = + &hlos1_vote_aggre_noc_mmu_tbu1_gdsc, + [HLOS1_VOTE_AGGRE_NOC_MMU_TBU2_GDSC] = + &hlos1_vote_aggre_noc_mmu_tbu2_gdsc, + [HLOS1_VOTE_MMNOC_MMU_TBU_HF0_GDSC] = + &hlos1_vote_mmnoc_mmu_tbu_hf0_gdsc, + [HLOS1_VOTE_MMNOC_MMU_TBU_HF1_GDSC] = + &hlos1_vote_mmnoc_mmu_tbu_hf1_gdsc, + [HLOS1_VOTE_MMNOC_MMU_TBU_SF_GDSC] = &hlos1_vote_mmnoc_mmu_tbu_sf_gdsc, +}; + +static const struct regmap_config gcc_sdm845_regmap_config = { + .reg_bits = 32, + .reg_stride = 4, + .val_bits = 32, + .max_register = 0x182090, + .fast_io = true, +}; + +static const struct qcom_cc_desc gcc_sdm845_desc = { + .config = &gcc_sdm845_regmap_config, + .clks = gcc_sdm845_clocks, + .num_clks = ARRAY_SIZE(gcc_sdm845_clocks), + .resets = gcc_sdm845_resets, + .num_resets = ARRAY_SIZE(gcc_sdm845_resets), + .gdscs = gcc_sdm845_gdscs, + .num_gdscs = ARRAY_SIZE(gcc_sdm845_gdscs), +}; + +static const struct of_device_id gcc_sdm845_match_table[] = { + { .compatible = "qcom,gcc-sdm845" }, + { } +}; +MODULE_DEVICE_TABLE(of, gcc_sdm845_match_table); + +static int gcc_sdm845_probe(struct platform_device *pdev) +{ + struct regmap *regmap; + + regmap = qcom_cc_map(pdev, &gcc_sdm845_desc); + if (IS_ERR(regmap)) + return PTR_ERR(regmap); + + /* Disable the GPLL0 active input to MMSS and GPU via MISC registers */ + regmap_update_bits(regmap, 0x09ffc, 0x3, 0x3); + regmap_update_bits(regmap, 0x71028, 0x3, 0x3); + + /* Enable CPUSS clocks */ + regmap_update_bits(regmap, 0x48190, BIT(0), 0x1); + regmap_update_bits(regmap, 0x52004, BIT(22), 0x1); + + return qcom_cc_really_probe(pdev, &gcc_sdm845_desc, regmap); +} + +static struct platform_driver gcc_sdm845_driver = { + .probe = gcc_sdm845_probe, + .driver = { + .name = "gcc-sdm845", + .of_match_table = gcc_sdm845_match_table, + }, +}; + +static int __init gcc_sdm845_init(void) +{ + return platform_driver_register(&gcc_sdm845_driver); +} +subsys_initcall(gcc_sdm845_init); + +static void __exit gcc_sdm845_exit(void) +{ + platform_driver_unregister(&gcc_sdm845_driver); +} +module_exit(gcc_sdm845_exit); + +MODULE_DESCRIPTION("QTI GCC SDM845 Driver"); +MODULE_LICENSE("GPL v2"); +MODULE_ALIAS("platform:gcc-sdm845"); diff --git a/drivers/clk/qcom/gdsc.c b/drivers/clk/qcom/gdsc.c index a4f3580587b7..a077133c7ce3 100644 --- a/drivers/clk/qcom/gdsc.c +++ b/drivers/clk/qcom/gdsc.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2015, The Linux Foundation. All rights reserved. + * Copyright (c) 2015, 2017-2018, The Linux Foundation. All rights reserved. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 and @@ -31,6 +31,12 @@ #define HW_CONTROL_MASK BIT(1) #define SW_COLLAPSE_MASK BIT(0) #define GMEM_CLAMP_IO_MASK BIT(0) +#define GMEM_RESET_MASK BIT(4) + +/* CFG_GDSCR */ +#define GDSC_POWER_UP_COMPLETE BIT(16) +#define GDSC_POWER_DOWN_COMPLETE BIT(15) +#define CFG_GDSCR_OFFSET 0x4 /* Wait 2^n CXO cycles between all states. Here, n=2 (4 cycles). */ #define EN_REST_WAIT_VAL (0x2 << 20) @@ -40,20 +46,50 @@ #define RETAIN_MEM BIT(14) #define RETAIN_PERIPH BIT(13) -#define TIMEOUT_US 100 +#define TIMEOUT_US 500 #define domain_to_gdsc(domain) container_of(domain, struct gdsc, pd) -static int gdsc_is_enabled(struct gdsc *sc, unsigned int reg) +enum gdsc_status { + GDSC_OFF, + GDSC_ON +}; + +/* Returns 1 if GDSC status is status, 0 if not, and < 0 on error */ +static int gdsc_check_status(struct gdsc *sc, enum gdsc_status status) { + unsigned int reg; u32 val; int ret; + if (sc->flags & POLL_CFG_GDSCR) + reg = sc->gdscr + CFG_GDSCR_OFFSET; + else if (sc->gds_hw_ctrl) + reg = sc->gds_hw_ctrl; + else + reg = sc->gdscr; + ret = regmap_read(sc->regmap, reg, &val); if (ret) return ret; - return !!(val & PWR_ON_MASK); + if (sc->flags & POLL_CFG_GDSCR) { + switch (status) { + case GDSC_ON: + return !!(val & GDSC_POWER_UP_COMPLETE); + case GDSC_OFF: + return !!(val & GDSC_POWER_DOWN_COMPLETE); + } + } + + switch (status) { + case GDSC_ON: + return !!(val & PWR_ON_MASK); + case GDSC_OFF: + return !(val & PWR_ON_MASK); + } + + return -EINVAL; } static int gdsc_hwctrl(struct gdsc *sc, bool en) @@ -63,34 +99,33 @@ static int gdsc_hwctrl(struct gdsc *sc, bool en) return regmap_update_bits(sc->regmap, sc->gdscr, HW_CONTROL_MASK, val); } -static int gdsc_poll_status(struct gdsc *sc, unsigned int reg, bool en) +static int gdsc_poll_status(struct gdsc *sc, enum gdsc_status status) { ktime_t start; start = ktime_get(); do { - if (gdsc_is_enabled(sc, reg) == en) + if (gdsc_check_status(sc, status)) return 0; } while (ktime_us_delta(ktime_get(), start) < TIMEOUT_US); - if (gdsc_is_enabled(sc, reg) == en) + if (gdsc_check_status(sc, status)) return 0; return -ETIMEDOUT; } -static int gdsc_toggle_logic(struct gdsc *sc, bool en) +static int gdsc_toggle_logic(struct gdsc *sc, enum gdsc_status status) { int ret; - u32 val = en ? 0 : SW_COLLAPSE_MASK; - unsigned int status_reg = sc->gdscr; + u32 val = (status == GDSC_ON) ? 0 : SW_COLLAPSE_MASK; ret = regmap_update_bits(sc->regmap, sc->gdscr, SW_COLLAPSE_MASK, val); if (ret) return ret; /* If disabling votable gdscs, don't poll on status */ - if ((sc->flags & VOTABLE) && !en) { + if ((sc->flags & VOTABLE) && status == GDSC_OFF) { /* * Add a short delay here to ensure that an enable * right after it was disabled does not put it in an @@ -101,7 +136,6 @@ static int gdsc_toggle_logic(struct gdsc *sc, bool en) } if (sc->gds_hw_ctrl) { - status_reg = sc->gds_hw_ctrl; /* * The gds hw controller asserts/de-asserts the status bit soon * after it receives a power on/off request from a master. @@ -115,7 +149,7 @@ static int gdsc_toggle_logic(struct gdsc *sc, bool en) udelay(1); } - return gdsc_poll_status(sc, status_reg, en); + return gdsc_poll_status(sc, status); } static inline int gdsc_deassert_reset(struct gdsc *sc) @@ -166,6 +200,14 @@ static inline void gdsc_assert_clamp_io(struct gdsc *sc) GMEM_CLAMP_IO_MASK, 1); } +static inline void gdsc_assert_reset_aon(struct gdsc *sc) +{ + regmap_update_bits(sc->regmap, sc->clamp_io_ctrl, + GMEM_RESET_MASK, 1); + udelay(1); + regmap_update_bits(sc->regmap, sc->clamp_io_ctrl, + GMEM_RESET_MASK, 0); +} static int gdsc_enable(struct generic_pm_domain *domain) { struct gdsc *sc = domain_to_gdsc(domain); @@ -174,10 +216,19 @@ static int gdsc_enable(struct generic_pm_domain *domain) if (sc->pwrsts == PWRSTS_ON) return gdsc_deassert_reset(sc); - if (sc->flags & CLAMP_IO) + if (sc->flags & SW_RESET) { + gdsc_assert_reset(sc); + udelay(1); + gdsc_deassert_reset(sc); + } + + if (sc->flags & CLAMP_IO) { + if (sc->flags & AON_RESET) + gdsc_assert_reset_aon(sc); gdsc_deassert_clamp_io(sc); + } - ret = gdsc_toggle_logic(sc, true); + ret = gdsc_toggle_logic(sc, GDSC_ON); if (ret) return ret; @@ -222,8 +273,6 @@ static int gdsc_disable(struct generic_pm_domain *domain) /* Turn off HW trigger mode if supported */ if (sc->flags & HW_CTRL) { - unsigned int reg; - ret = gdsc_hwctrl(sc, false); if (ret < 0) return ret; @@ -235,8 +284,7 @@ static int gdsc_disable(struct generic_pm_domain *domain) */ udelay(1); - reg = sc->gds_hw_ctrl ? sc->gds_hw_ctrl : sc->gdscr; - ret = gdsc_poll_status(sc, reg, true); + ret = gdsc_poll_status(sc, GDSC_ON); if (ret) return ret; } @@ -244,7 +292,7 @@ static int gdsc_disable(struct generic_pm_domain *domain) if (sc->pwrsts & PWRSTS_OFF) gdsc_clear_mem_on(sc); - ret = gdsc_toggle_logic(sc, false); + ret = gdsc_toggle_logic(sc, GDSC_OFF); if (ret) return ret; @@ -258,7 +306,6 @@ static int gdsc_init(struct gdsc *sc) { u32 mask, val; int on, ret; - unsigned int reg; /* * Disable HW trigger: collapse/restore occur based on registers writes. @@ -274,13 +321,12 @@ static int gdsc_init(struct gdsc *sc) /* Force gdsc ON if only ON state is supported */ if (sc->pwrsts == PWRSTS_ON) { - ret = gdsc_toggle_logic(sc, true); + ret = gdsc_toggle_logic(sc, GDSC_ON); if (ret) return ret; } - reg = sc->gds_hw_ctrl ? sc->gds_hw_ctrl : sc->gdscr; - on = gdsc_is_enabled(sc, reg); + on = gdsc_check_status(sc, GDSC_ON); if (on < 0) return on; @@ -291,6 +337,14 @@ static int gdsc_init(struct gdsc *sc) if ((sc->flags & VOTABLE) && on) gdsc_enable(&sc->pd); + /* If ALWAYS_ON GDSCs are not ON, turn them ON */ + if (sc->flags & ALWAYS_ON) { + if (!on) + gdsc_enable(&sc->pd); + on = true; + sc->pd.flags |= GENPD_FLAG_ALWAYS_ON; + } + if (on || (sc->pwrsts & PWRSTS_RET)) gdsc_force_mem_on(sc); else diff --git a/drivers/clk/qcom/gdsc.h b/drivers/clk/qcom/gdsc.h index 39648348e5ec..bd1f2c780d0a 100644 --- a/drivers/clk/qcom/gdsc.h +++ b/drivers/clk/qcom/gdsc.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2015, The Linux Foundation. All rights reserved. + * Copyright (c) 2015, 2017-2018, The Linux Foundation. All rights reserved. * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 and @@ -53,6 +53,10 @@ struct gdsc { #define VOTABLE BIT(0) #define CLAMP_IO BIT(1) #define HW_CTRL BIT(2) +#define SW_RESET BIT(3) +#define AON_RESET BIT(4) +#define POLL_CFG_GDSCR BIT(5) +#define ALWAYS_ON BIT(6) struct reset_controller_dev *rcdev; unsigned int *resets; unsigned int reset_count; diff --git a/drivers/clk/qcom/mmcc-msm8996.c b/drivers/clk/qcom/mmcc-msm8996.c index 66a2fa4ec93c..1a25ee4f3658 100644 --- a/drivers/clk/qcom/mmcc-msm8996.c +++ b/drivers/clk/qcom/mmcc-msm8996.c @@ -1245,7 +1245,7 @@ static struct clk_branch mmss_mmagic_ahb_clk = { .name = "mmss_mmagic_ahb_clk", .parent_names = (const char *[]){ "ahb_clk_src" }, .num_parents = 1, - .flags = CLK_SET_RATE_PARENT | CLK_IGNORE_UNUSED, + .flags = CLK_SET_RATE_PARENT | CLK_IS_CRITICAL, .ops = &clk_branch2_ops, }, }, @@ -1260,7 +1260,7 @@ static struct clk_branch mmss_mmagic_cfg_ahb_clk = { .name = "mmss_mmagic_cfg_ahb_clk", .parent_names = (const char *[]){ "ahb_clk_src" }, .num_parents = 1, - .flags = CLK_SET_RATE_PARENT, + .flags = CLK_SET_RATE_PARENT | CLK_IS_CRITICAL, .ops = &clk_branch2_ops, }, }, @@ -1319,7 +1319,7 @@ static struct clk_branch mmagic_camss_axi_clk = { .name = "mmagic_camss_axi_clk", .parent_names = (const char *[]){ "axi_clk_src" }, .num_parents = 1, - .flags = CLK_SET_RATE_PARENT, + .flags = CLK_SET_RATE_PARENT | CLK_IS_CRITICAL, .ops = &clk_branch2_ops, }, }, @@ -1334,7 +1334,7 @@ static struct clk_branch mmagic_camss_noc_cfg_ahb_clk = { .name = "mmagic_camss_noc_cfg_ahb_clk", .parent_names = (const char *[]){ "gcc_mmss_noc_cfg_ahb_clk" }, .num_parents = 1, - .flags = CLK_SET_RATE_PARENT, + .flags = CLK_SET_RATE_PARENT | CLK_IS_CRITICAL, .ops = &clk_branch2_ops, }, }, @@ -1439,7 +1439,7 @@ static struct clk_branch mmagic_mdss_axi_clk = { .name = "mmagic_mdss_axi_clk", .parent_names = (const char *[]){ "axi_clk_src" }, .num_parents = 1, - .flags = CLK_SET_RATE_PARENT, + .flags = CLK_SET_RATE_PARENT | CLK_IS_CRITICAL, .ops = &clk_branch2_ops, }, }, @@ -1454,7 +1454,7 @@ static struct clk_branch mmagic_mdss_noc_cfg_ahb_clk = { .name = "mmagic_mdss_noc_cfg_ahb_clk", .parent_names = (const char *[]){ "gcc_mmss_noc_cfg_ahb_clk" }, .num_parents = 1, - .flags = CLK_SET_RATE_PARENT, + .flags = CLK_SET_RATE_PARENT | CLK_IS_CRITICAL, .ops = &clk_branch2_ops, }, }, @@ -1529,7 +1529,7 @@ static struct clk_branch mmagic_video_axi_clk = { .name = "mmagic_video_axi_clk", .parent_names = (const char *[]){ "axi_clk_src" }, .num_parents = 1, - .flags = CLK_SET_RATE_PARENT, + .flags = CLK_SET_RATE_PARENT | CLK_IS_CRITICAL, .ops = &clk_branch2_ops, }, }, @@ -1544,7 +1544,7 @@ static struct clk_branch mmagic_video_noc_cfg_ahb_clk = { .name = "mmagic_video_noc_cfg_ahb_clk", .parent_names = (const char *[]){ "gcc_mmss_noc_cfg_ahb_clk" }, .num_parents = 1, - .flags = CLK_SET_RATE_PARENT, + .flags = CLK_SET_RATE_PARENT | CLK_IS_CRITICAL, .ops = &clk_branch2_ops, }, }, @@ -2919,7 +2919,7 @@ static struct gdsc mmagic_video_gdsc = { .name = "mmagic_video", }, .pwrsts = PWRSTS_OFF_ON, - .flags = VOTABLE, + .flags = VOTABLE | ALWAYS_ON, }; static struct gdsc mmagic_mdss_gdsc = { @@ -2929,7 +2929,7 @@ static struct gdsc mmagic_mdss_gdsc = { .name = "mmagic_mdss", }, .pwrsts = PWRSTS_OFF_ON, - .flags = VOTABLE, + .flags = VOTABLE | ALWAYS_ON, }; static struct gdsc mmagic_camss_gdsc = { @@ -2939,7 +2939,7 @@ static struct gdsc mmagic_camss_gdsc = { .name = "mmagic_camss", }, .pwrsts = PWRSTS_OFF_ON, - .flags = VOTABLE, + .flags = VOTABLE | ALWAYS_ON, }; static struct gdsc venus_gdsc = { diff --git a/drivers/clk/qcom/videocc-sdm845.c b/drivers/clk/qcom/videocc-sdm845.c new file mode 100644 index 000000000000..9073b7a710ac --- /dev/null +++ b/drivers/clk/qcom/videocc-sdm845.c @@ -0,0 +1,358 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2018, The Linux Foundation. All rights reserved. + */ + +#include +#include +#include +#include + +#include + +#include "common.h" +#include "clk-alpha-pll.h" +#include "clk-branch.h" +#include "clk-rcg.h" +#include "clk-regmap.h" +#include "clk-pll.h" +#include "gdsc.h" + +#define F(f, s, h, m, n) { (f), (s), (2 * (h) - 1), (m), (n) } + +enum { + P_BI_TCXO, + P_CORE_BI_PLL_TEST_SE, + P_VIDEO_PLL0_OUT_EVEN, + P_VIDEO_PLL0_OUT_MAIN, + P_VIDEO_PLL0_OUT_ODD, +}; + +static const struct parent_map video_cc_parent_map_0[] = { + { P_BI_TCXO, 0 }, + { P_VIDEO_PLL0_OUT_MAIN, 1 }, + { P_VIDEO_PLL0_OUT_EVEN, 2 }, + { P_VIDEO_PLL0_OUT_ODD, 3 }, + { P_CORE_BI_PLL_TEST_SE, 4 }, +}; + +static const char * const video_cc_parent_names_0[] = { + "bi_tcxo", + "video_pll0", + "video_pll0_out_even", + "video_pll0_out_odd", + "core_bi_pll_test_se", +}; + +static const struct alpha_pll_config video_pll0_config = { + .l = 0x10, + .alpha = 0xaaab, +}; + +static struct clk_alpha_pll video_pll0 = { + .offset = 0x42c, + .regs = clk_alpha_pll_regs[CLK_ALPHA_PLL_TYPE_FABIA], + .clkr = { + .hw.init = &(struct clk_init_data){ + .name = "video_pll0", + .parent_names = (const char *[]){ "bi_tcxo" }, + .num_parents = 1, + .ops = &clk_alpha_pll_fabia_ops, + }, + }, +}; + +static const struct freq_tbl ftbl_video_cc_venus_clk_src[] = { + F(100000000, P_VIDEO_PLL0_OUT_MAIN, 4, 0, 0), + F(200000000, P_VIDEO_PLL0_OUT_MAIN, 2, 0, 0), + F(330000000, P_VIDEO_PLL0_OUT_MAIN, 1, 0, 0), + F(404000000, P_VIDEO_PLL0_OUT_MAIN, 1, 0, 0), + F(444000000, P_VIDEO_PLL0_OUT_MAIN, 1, 0, 0), + F(533000000, P_VIDEO_PLL0_OUT_MAIN, 1, 0, 0), + { } +}; + +static struct clk_rcg2 video_cc_venus_clk_src = { + .cmd_rcgr = 0x7f0, + .mnd_width = 0, + .hid_width = 5, + .parent_map = video_cc_parent_map_0, + .freq_tbl = ftbl_video_cc_venus_clk_src, + .clkr.hw.init = &(struct clk_init_data){ + .name = "video_cc_venus_clk_src", + .parent_names = video_cc_parent_names_0, + .num_parents = 5, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_rcg2_shared_ops, + }, +}; + +static struct clk_branch video_cc_apb_clk = { + .halt_reg = 0x990, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x990, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "video_cc_apb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch video_cc_at_clk = { + .halt_reg = 0x9f0, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x9f0, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "video_cc_at_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch video_cc_qdss_trig_clk = { + .halt_reg = 0x970, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x970, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "video_cc_qdss_trig_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch video_cc_qdss_tsctr_div8_clk = { + .halt_reg = 0x9d0, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x9d0, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "video_cc_qdss_tsctr_div8_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch video_cc_vcodec0_axi_clk = { + .halt_reg = 0x930, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x930, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "video_cc_vcodec0_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch video_cc_vcodec0_core_clk = { + .halt_reg = 0x890, + .halt_check = BRANCH_VOTED, + .clkr = { + .enable_reg = 0x890, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "video_cc_vcodec0_core_clk", + .parent_names = (const char *[]){ + "video_cc_venus_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch video_cc_vcodec1_axi_clk = { + .halt_reg = 0x950, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x950, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "video_cc_vcodec1_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch video_cc_vcodec1_core_clk = { + .halt_reg = 0x8d0, + .halt_check = BRANCH_VOTED, + .clkr = { + .enable_reg = 0x8d0, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "video_cc_vcodec1_core_clk", + .parent_names = (const char *[]){ + "video_cc_venus_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch video_cc_venus_ahb_clk = { + .halt_reg = 0x9b0, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x9b0, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "video_cc_venus_ahb_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch video_cc_venus_ctl_axi_clk = { + .halt_reg = 0x910, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x910, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "video_cc_venus_ctl_axi_clk", + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct clk_branch video_cc_venus_ctl_core_clk = { + .halt_reg = 0x850, + .halt_check = BRANCH_HALT, + .clkr = { + .enable_reg = 0x850, + .enable_mask = BIT(0), + .hw.init = &(struct clk_init_data){ + .name = "video_cc_venus_ctl_core_clk", + .parent_names = (const char *[]){ + "video_cc_venus_clk_src", + }, + .num_parents = 1, + .flags = CLK_SET_RATE_PARENT, + .ops = &clk_branch2_ops, + }, + }, +}; + +static struct gdsc venus_gdsc = { + .gdscr = 0x814, + .pd = { + .name = "venus_gdsc", + }, + .cxcs = (unsigned int []){ 0x850, 0x910 }, + .cxc_count = 2, + .pwrsts = PWRSTS_OFF_ON, + .flags = POLL_CFG_GDSCR, +}; + +static struct gdsc vcodec0_gdsc = { + .gdscr = 0x874, + .pd = { + .name = "vcodec0_gdsc", + }, + .cxcs = (unsigned int []){ 0x890, 0x930 }, + .cxc_count = 2, + .flags = HW_CTRL | POLL_CFG_GDSCR, + .pwrsts = PWRSTS_OFF_ON, +}; + +static struct gdsc vcodec1_gdsc = { + .gdscr = 0x8b4, + .pd = { + .name = "vcodec1_gdsc", + }, + .cxcs = (unsigned int []){ 0x8d0, 0x950 }, + .cxc_count = 2, + .flags = HW_CTRL | POLL_CFG_GDSCR, + .pwrsts = PWRSTS_OFF_ON, +}; + +static struct clk_regmap *video_cc_sdm845_clocks[] = { + [VIDEO_CC_APB_CLK] = &video_cc_apb_clk.clkr, + [VIDEO_CC_AT_CLK] = &video_cc_at_clk.clkr, + [VIDEO_CC_QDSS_TRIG_CLK] = &video_cc_qdss_trig_clk.clkr, + [VIDEO_CC_QDSS_TSCTR_DIV8_CLK] = &video_cc_qdss_tsctr_div8_clk.clkr, + [VIDEO_CC_VCODEC0_AXI_CLK] = &video_cc_vcodec0_axi_clk.clkr, + [VIDEO_CC_VCODEC0_CORE_CLK] = &video_cc_vcodec0_core_clk.clkr, + [VIDEO_CC_VCODEC1_AXI_CLK] = &video_cc_vcodec1_axi_clk.clkr, + [VIDEO_CC_VCODEC1_CORE_CLK] = &video_cc_vcodec1_core_clk.clkr, + [VIDEO_CC_VENUS_AHB_CLK] = &video_cc_venus_ahb_clk.clkr, + [VIDEO_CC_VENUS_CLK_SRC] = &video_cc_venus_clk_src.clkr, + [VIDEO_CC_VENUS_CTL_AXI_CLK] = &video_cc_venus_ctl_axi_clk.clkr, + [VIDEO_CC_VENUS_CTL_CORE_CLK] = &video_cc_venus_ctl_core_clk.clkr, + [VIDEO_PLL0] = &video_pll0.clkr, +}; + +static struct gdsc *video_cc_sdm845_gdscs[] = { + [VENUS_GDSC] = &venus_gdsc, + [VCODEC0_GDSC] = &vcodec0_gdsc, + [VCODEC1_GDSC] = &vcodec1_gdsc, +}; + +static const struct regmap_config video_cc_sdm845_regmap_config = { + .reg_bits = 32, + .reg_stride = 4, + .val_bits = 32, + .max_register = 0xb90, + .fast_io = true, +}; + +static const struct qcom_cc_desc video_cc_sdm845_desc = { + .config = &video_cc_sdm845_regmap_config, + .clks = video_cc_sdm845_clocks, + .num_clks = ARRAY_SIZE(video_cc_sdm845_clocks), + .gdscs = video_cc_sdm845_gdscs, + .num_gdscs = ARRAY_SIZE(video_cc_sdm845_gdscs), +}; + +static const struct of_device_id video_cc_sdm845_match_table[] = { + { .compatible = "qcom,sdm845-videocc" }, + { } +}; +MODULE_DEVICE_TABLE(of, video_cc_sdm845_match_table); + +static int video_cc_sdm845_probe(struct platform_device *pdev) +{ + struct regmap *regmap; + + regmap = qcom_cc_map(pdev, &video_cc_sdm845_desc); + if (IS_ERR(regmap)) + return PTR_ERR(regmap); + + clk_fabia_pll_configure(&video_pll0, regmap, &video_pll0_config); + + return qcom_cc_really_probe(pdev, &video_cc_sdm845_desc, regmap); +} + +static struct platform_driver video_cc_sdm845_driver = { + .probe = video_cc_sdm845_probe, + .driver = { + .name = "sdm845-videocc", + .of_match_table = video_cc_sdm845_match_table, + }, +}; + +static int __init video_cc_sdm845_init(void) +{ + return platform_driver_register(&video_cc_sdm845_driver); +} +subsys_initcall(video_cc_sdm845_init); + +static void __exit video_cc_sdm845_exit(void) +{ + platform_driver_unregister(&video_cc_sdm845_driver); +} +module_exit(video_cc_sdm845_exit); + +MODULE_LICENSE("GPL v2"); diff --git a/drivers/clk/renesas/Kconfig b/drivers/clk/renesas/Kconfig index ef76c861ec84..f9ba71311727 100644 --- a/drivers/clk/renesas/Kconfig +++ b/drivers/clk/renesas/Kconfig @@ -7,6 +7,7 @@ config CLK_RENESAS select CLK_R8A7740 if ARCH_R8A7740 select CLK_R8A7743 if ARCH_R8A7743 select CLK_R8A7745 if ARCH_R8A7745 + select CLK_R8A77470 if ARCH_R8A77470 select CLK_R8A7778 if ARCH_R8A7778 select CLK_R8A7779 if ARCH_R8A7779 select CLK_R8A7790 if ARCH_R8A7790 @@ -18,6 +19,7 @@ config CLK_RENESAS select CLK_R8A77965 if ARCH_R8A77965 select CLK_R8A77970 if ARCH_R8A77970 select CLK_R8A77980 if ARCH_R8A77980 + select CLK_R8A77990 if ARCH_R8A77990 select CLK_R8A77995 if ARCH_R8A77995 select CLK_SH73A0 if ARCH_SH73A0 @@ -60,6 +62,10 @@ config CLK_R8A7745 bool "RZ/G1E clock support" if COMPILE_TEST select CLK_RCAR_GEN2_CPG +config CLK_R8A77470 + bool "RZ/G1C clock support" if COMPILE_TEST + select CLK_RCAR_GEN2_CPG + config CLK_R8A7778 bool "R-Car M1A clock support" if COMPILE_TEST select CLK_RENESAS_CPG_MSTP @@ -111,6 +117,10 @@ config CLK_R8A77980 bool "R-Car V3H clock support" if COMPILE_TEST select CLK_RCAR_GEN3_CPG +config CLK_R8A77990 + bool "R-Car E3 clock support" if COMPILE_TEST + select CLK_RCAR_GEN3_CPG + config CLK_R8A77995 bool "R-Car D3 clock support" if COMPILE_TEST select CLK_RCAR_GEN3_CPG diff --git a/drivers/clk/renesas/Makefile b/drivers/clk/renesas/Makefile index 6c0f19636e3e..fe5bac9215e5 100644 --- a/drivers/clk/renesas/Makefile +++ b/drivers/clk/renesas/Makefile @@ -6,6 +6,7 @@ obj-$(CONFIG_CLK_R8A73A4) += clk-r8a73a4.o obj-$(CONFIG_CLK_R8A7740) += clk-r8a7740.o obj-$(CONFIG_CLK_R8A7743) += r8a7743-cpg-mssr.o obj-$(CONFIG_CLK_R8A7745) += r8a7745-cpg-mssr.o +obj-$(CONFIG_CLK_R8A77470) += r8a77470-cpg-mssr.o obj-$(CONFIG_CLK_R8A7778) += clk-r8a7778.o obj-$(CONFIG_CLK_R8A7779) += clk-r8a7779.o obj-$(CONFIG_CLK_R8A7790) += r8a7790-cpg-mssr.o @@ -17,6 +18,7 @@ obj-$(CONFIG_CLK_R8A7796) += r8a7796-cpg-mssr.o obj-$(CONFIG_CLK_R8A77965) += r8a77965-cpg-mssr.o obj-$(CONFIG_CLK_R8A77970) += r8a77970-cpg-mssr.o obj-$(CONFIG_CLK_R8A77980) += r8a77980-cpg-mssr.o +obj-$(CONFIG_CLK_R8A77990) += r8a77990-cpg-mssr.o obj-$(CONFIG_CLK_R8A77995) += r8a77995-cpg-mssr.o obj-$(CONFIG_CLK_SH73A0) += clk-sh73a0.o diff --git a/drivers/clk/renesas/clk-r8a7740.c b/drivers/clk/renesas/clk-r8a7740.c index d074f8e982d0..a7a30d2eca41 100644 --- a/drivers/clk/renesas/clk-r8a7740.c +++ b/drivers/clk/renesas/clk-r8a7740.c @@ -161,7 +161,7 @@ static void __init r8a7740_cpg_clocks_init(struct device_node *np) } cpg = kzalloc(sizeof(*cpg), GFP_KERNEL); - clks = kzalloc(num_clks * sizeof(*clks), GFP_KERNEL); + clks = kcalloc(num_clks, sizeof(*clks), GFP_KERNEL); if (cpg == NULL || clks == NULL) { /* We're leaking memory on purpose, there's no point in cleaning * up as the system won't boot anyway. diff --git a/drivers/clk/renesas/clk-r8a7779.c b/drivers/clk/renesas/clk-r8a7779.c index 27fbfafaf2cd..5adcca4656c3 100644 --- a/drivers/clk/renesas/clk-r8a7779.c +++ b/drivers/clk/renesas/clk-r8a7779.c @@ -138,7 +138,7 @@ static void __init r8a7779_cpg_clocks_init(struct device_node *np) } cpg = kzalloc(sizeof(*cpg), GFP_KERNEL); - clks = kzalloc(CPG_NUM_CLOCKS * sizeof(*clks), GFP_KERNEL); + clks = kcalloc(CPG_NUM_CLOCKS, sizeof(*clks), GFP_KERNEL); if (cpg == NULL || clks == NULL) { /* We're leaking memory on purpose, there's no point in cleaning * up as the system won't boot anyway. diff --git a/drivers/clk/renesas/clk-rcar-gen2.c b/drivers/clk/renesas/clk-rcar-gen2.c index ee32a022e6da..bccd62f2cb09 100644 --- a/drivers/clk/renesas/clk-rcar-gen2.c +++ b/drivers/clk/renesas/clk-rcar-gen2.c @@ -417,7 +417,7 @@ static void __init rcar_gen2_cpg_clocks_init(struct device_node *np) } cpg = kzalloc(sizeof(*cpg), GFP_KERNEL); - clks = kzalloc(num_clks * sizeof(*clks), GFP_KERNEL); + clks = kcalloc(num_clks, sizeof(*clks), GFP_KERNEL); if (cpg == NULL || clks == NULL) { /* We're leaking memory on purpose, there's no point in cleaning * up as the system won't boot anyway. diff --git a/drivers/clk/renesas/clk-rz.c b/drivers/clk/renesas/clk-rz.c index 67dd712aa723..ac2f86d626b6 100644 --- a/drivers/clk/renesas/clk-rz.c +++ b/drivers/clk/renesas/clk-rz.c @@ -97,7 +97,7 @@ static void __init rz_cpg_clocks_init(struct device_node *np) return; cpg = kzalloc(sizeof(*cpg), GFP_KERNEL); - clks = kzalloc(num_clks * sizeof(*clks), GFP_KERNEL); + clks = kcalloc(num_clks, sizeof(*clks), GFP_KERNEL); BUG_ON(!cpg || !clks); cpg->data.clks = clks; diff --git a/drivers/clk/renesas/r8a7743-cpg-mssr.c b/drivers/clk/renesas/r8a7743-cpg-mssr.c index d3c8b1e2969f..011c170ec3f9 100644 --- a/drivers/clk/renesas/r8a7743-cpg-mssr.c +++ b/drivers/clk/renesas/r8a7743-cpg-mssr.c @@ -52,7 +52,6 @@ static const struct cpg_core_clk r8a7743_core_clks[] __initconst = { /* Core Clock Outputs */ DEF_BASE("z", R8A7743_CLK_Z, CLK_TYPE_GEN2_Z, CLK_PLL0), - DEF_BASE("lb", R8A7743_CLK_LB, CLK_TYPE_GEN2_LB, CLK_PLL1), DEF_BASE("sdh", R8A7743_CLK_SDH, CLK_TYPE_GEN2_SDH, CLK_PLL1), DEF_BASE("sd0", R8A7743_CLK_SD0, CLK_TYPE_GEN2_SD0, CLK_PLL1), DEF_BASE("qspi", R8A7743_CLK_QSPI, CLK_TYPE_GEN2_QSPI, CLK_PLL1_DIV2), @@ -63,6 +62,7 @@ static const struct cpg_core_clk r8a7743_core_clks[] __initconst = { DEF_FIXED("zs", R8A7743_CLK_ZS, CLK_PLL1, 6, 1), DEF_FIXED("hp", R8A7743_CLK_HP, CLK_PLL1, 12, 1), DEF_FIXED("b", R8A7743_CLK_B, CLK_PLL1, 12, 1), + DEF_FIXED("lb", R8A7743_CLK_LB, CLK_PLL1, 24, 1), DEF_FIXED("p", R8A7743_CLK_P, CLK_PLL1, 24, 1), DEF_FIXED("cl", R8A7743_CLK_CL, CLK_PLL1, 48, 1), DEF_FIXED("m2", R8A7743_CLK_M2, CLK_PLL1, 8, 1), diff --git a/drivers/clk/renesas/r8a7745-cpg-mssr.c b/drivers/clk/renesas/r8a7745-cpg-mssr.c index 87f5a3619e4f..4b0a9243b748 100644 --- a/drivers/clk/renesas/r8a7745-cpg-mssr.c +++ b/drivers/clk/renesas/r8a7745-cpg-mssr.c @@ -51,7 +51,6 @@ static const struct cpg_core_clk r8a7745_core_clks[] __initconst = { DEF_FIXED(".pll1_div2", CLK_PLL1_DIV2, CLK_PLL1, 2, 1), /* Core Clock Outputs */ - DEF_BASE("lb", R8A7745_CLK_LB, CLK_TYPE_GEN2_LB, CLK_PLL1), DEF_BASE("sdh", R8A7745_CLK_SDH, CLK_TYPE_GEN2_SDH, CLK_PLL1), DEF_BASE("sd0", R8A7745_CLK_SD0, CLK_TYPE_GEN2_SD0, CLK_PLL1), DEF_BASE("qspi", R8A7745_CLK_QSPI, CLK_TYPE_GEN2_QSPI, CLK_PLL1_DIV2), @@ -63,6 +62,7 @@ static const struct cpg_core_clk r8a7745_core_clks[] __initconst = { DEF_FIXED("zs", R8A7745_CLK_ZS, CLK_PLL1, 6, 1), DEF_FIXED("hp", R8A7745_CLK_HP, CLK_PLL1, 12, 1), DEF_FIXED("b", R8A7745_CLK_B, CLK_PLL1, 12, 1), + DEF_FIXED("lb", R8A7745_CLK_LB, CLK_PLL1, 24, 1), DEF_FIXED("p", R8A7745_CLK_P, CLK_PLL1, 24, 1), DEF_FIXED("cl", R8A7745_CLK_CL, CLK_PLL1, 48, 1), DEF_FIXED("cp", R8A7745_CLK_CP, CLK_PLL1, 48, 1), diff --git a/drivers/clk/renesas/r8a77470-cpg-mssr.c b/drivers/clk/renesas/r8a77470-cpg-mssr.c new file mode 100644 index 000000000000..ab0fb10b6bf0 --- /dev/null +++ b/drivers/clk/renesas/r8a77470-cpg-mssr.c @@ -0,0 +1,229 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * r8a77470 Clock Pulse Generator / Module Standby and Software Reset + * + * Copyright (C) 2018 Renesas Electronics Corp. + */ + +#include +#include +#include +#include + +#include + +#include "renesas-cpg-mssr.h" +#include "rcar-gen2-cpg.h" + +enum clk_ids { + /* Core Clock Outputs exported to DT */ + LAST_DT_CORE_CLK = R8A77470_CLK_OSC, + + /* External Input Clocks */ + CLK_EXTAL, + CLK_USB_EXTAL, + + /* Internal Core Clocks */ + CLK_MAIN, + CLK_PLL0, + CLK_PLL1, + CLK_PLL3, + CLK_PLL1_DIV2, + + /* Module Clocks */ + MOD_CLK_BASE +}; + +static const struct cpg_core_clk r8a77470_core_clks[] __initconst = { + /* External Clock Inputs */ + DEF_INPUT("extal", CLK_EXTAL), + DEF_INPUT("usb_extal", CLK_USB_EXTAL), + + /* Internal Core Clocks */ + DEF_BASE(".main", CLK_MAIN, CLK_TYPE_GEN2_MAIN, CLK_EXTAL), + DEF_BASE(".pll0", CLK_PLL0, CLK_TYPE_GEN2_PLL0, CLK_MAIN), + DEF_BASE(".pll1", CLK_PLL1, CLK_TYPE_GEN2_PLL1, CLK_MAIN), + DEF_BASE(".pll3", CLK_PLL3, CLK_TYPE_GEN2_PLL3, CLK_MAIN), + + DEF_FIXED(".pll1_div2", CLK_PLL1_DIV2, CLK_PLL1, 2, 1), + + /* Core Clock Outputs */ + DEF_BASE("sdh", R8A77470_CLK_SDH, CLK_TYPE_GEN2_SDH, CLK_PLL1), + DEF_BASE("sd0", R8A77470_CLK_SD0, CLK_TYPE_GEN2_SD0, CLK_PLL1), + DEF_BASE("sd1", R8A77470_CLK_SD1, CLK_TYPE_GEN2_SD1, CLK_PLL1), + DEF_BASE("qspi", R8A77470_CLK_QSPI, CLK_TYPE_GEN2_QSPI, CLK_PLL1_DIV2), + DEF_BASE("rcan", R8A77470_CLK_RCAN, CLK_TYPE_GEN2_RCAN, CLK_USB_EXTAL), + + DEF_FIXED("z2", R8A77470_CLK_Z2, CLK_PLL0, 1, 1), + DEF_FIXED("zx", R8A77470_CLK_ZX, CLK_PLL1, 3, 1), + DEF_FIXED("zs", R8A77470_CLK_ZS, CLK_PLL1, 6, 1), + DEF_FIXED("hp", R8A77470_CLK_HP, CLK_PLL1, 12, 1), + DEF_FIXED("b", R8A77470_CLK_B, CLK_PLL1, 12, 1), + DEF_FIXED("lb", R8A77470_CLK_LB, CLK_PLL1, 24, 1), + DEF_FIXED("p", R8A77470_CLK_P, CLK_PLL1, 24, 1), + DEF_FIXED("cl", R8A77470_CLK_CL, CLK_PLL1, 48, 1), + DEF_FIXED("cp", R8A77470_CLK_CP, CLK_PLL1, 48, 1), + DEF_FIXED("m2", R8A77470_CLK_M2, CLK_PLL1, 8, 1), + DEF_FIXED("zb3", R8A77470_CLK_ZB3, CLK_PLL3, 4, 1), + DEF_FIXED("mp", R8A77470_CLK_MP, CLK_PLL1_DIV2, 15, 1), + DEF_FIXED("cpex", R8A77470_CLK_CPEX, CLK_EXTAL, 2, 1), + DEF_FIXED("r", R8A77470_CLK_R, CLK_PLL1, 49152, 1), + DEF_FIXED("osc", R8A77470_CLK_OSC, CLK_PLL1, 12288, 1), + + DEF_DIV6P1("sd2", R8A77470_CLK_SD2, CLK_PLL1_DIV2, 0x078), +}; + +static const struct mssr_mod_clk r8a77470_mod_clks[] __initconst = { + DEF_MOD("msiof0", 0, R8A77470_CLK_MP), + DEF_MOD("vcp0", 101, R8A77470_CLK_ZS), + DEF_MOD("vpc0", 103, R8A77470_CLK_ZS), + DEF_MOD("tmu1", 111, R8A77470_CLK_P), + DEF_MOD("3dg", 112, R8A77470_CLK_ZS), + DEF_MOD("2d-dmac", 115, R8A77470_CLK_ZS), + DEF_MOD("fdp1-0", 119, R8A77470_CLK_ZS), + DEF_MOD("tmu3", 121, R8A77470_CLK_P), + DEF_MOD("tmu2", 122, R8A77470_CLK_P), + DEF_MOD("cmt0", 124, R8A77470_CLK_R), + DEF_MOD("vsp1du0", 128, R8A77470_CLK_ZS), + DEF_MOD("vsp1-sy", 131, R8A77470_CLK_ZS), + DEF_MOD("msiof2", 205, R8A77470_CLK_MP), + DEF_MOD("msiof1", 208, R8A77470_CLK_MP), + DEF_MOD("sys-dmac1", 218, R8A77470_CLK_ZS), + DEF_MOD("sys-dmac0", 219, R8A77470_CLK_ZS), + DEF_MOD("sdhi2", 312, R8A77470_CLK_SD2), + DEF_MOD("sdhi1", 313, R8A77470_CLK_SD1), + DEF_MOD("sdhi0", 314, R8A77470_CLK_SD0), + DEF_MOD("usbhs-dmac0-ch1", 326, R8A77470_CLK_HP), + DEF_MOD("usbhs-dmac1-ch1", 327, R8A77470_CLK_HP), + DEF_MOD("cmt1", 329, R8A77470_CLK_R), + DEF_MOD("usbhs-dmac0-ch0", 330, R8A77470_CLK_HP), + DEF_MOD("usbhs-dmac1-ch0", 331, R8A77470_CLK_HP), + DEF_MOD("rwdt", 402, R8A77470_CLK_R), + DEF_MOD("irqc", 407, R8A77470_CLK_CP), + DEF_MOD("intc-sys", 408, R8A77470_CLK_ZS), + DEF_MOD("audio-dmac0", 502, R8A77470_CLK_HP), + DEF_MOD("pwm", 523, R8A77470_CLK_P), + DEF_MOD("usb-ehci-0", 703, R8A77470_CLK_MP), + DEF_MOD("usbhs-0", 704, R8A77470_CLK_HP), + DEF_MOD("usb-ehci-1", 705, R8A77470_CLK_MP), + DEF_MOD("usbhs-1", 706, R8A77470_CLK_HP), + DEF_MOD("hscif2", 713, R8A77470_CLK_ZS), + DEF_MOD("scif5", 714, R8A77470_CLK_P), + DEF_MOD("scif4", 715, R8A77470_CLK_P), + DEF_MOD("hscif1", 716, R8A77470_CLK_ZS), + DEF_MOD("hscif0", 717, R8A77470_CLK_ZS), + DEF_MOD("scif3", 718, R8A77470_CLK_P), + DEF_MOD("scif2", 719, R8A77470_CLK_P), + DEF_MOD("scif1", 720, R8A77470_CLK_P), + DEF_MOD("scif0", 721, R8A77470_CLK_P), + DEF_MOD("du1", 723, R8A77470_CLK_ZX), + DEF_MOD("du0", 724, R8A77470_CLK_ZX), + DEF_MOD("ipmmu-sgx", 800, R8A77470_CLK_ZX), + DEF_MOD("etheravb", 812, R8A77470_CLK_HP), + DEF_MOD("ether", 813, R8A77470_CLK_P), + DEF_MOD("gpio5", 907, R8A77470_CLK_CP), + DEF_MOD("gpio4", 908, R8A77470_CLK_CP), + DEF_MOD("gpio3", 909, R8A77470_CLK_CP), + DEF_MOD("gpio2", 910, R8A77470_CLK_CP), + DEF_MOD("gpio1", 911, R8A77470_CLK_CP), + DEF_MOD("gpio0", 912, R8A77470_CLK_CP), + DEF_MOD("can1", 915, R8A77470_CLK_P), + DEF_MOD("can0", 916, R8A77470_CLK_P), + DEF_MOD("qspi_mod-1", 917, R8A77470_CLK_QSPI), + DEF_MOD("qspi_mod-0", 918, R8A77470_CLK_QSPI), + DEF_MOD("i2c4", 927, R8A77470_CLK_HP), + DEF_MOD("i2c3", 928, R8A77470_CLK_HP), + DEF_MOD("i2c2", 929, R8A77470_CLK_HP), + DEF_MOD("i2c1", 930, R8A77470_CLK_HP), + DEF_MOD("i2c0", 931, R8A77470_CLK_HP), + DEF_MOD("ssi-all", 1005, R8A77470_CLK_P), + DEF_MOD("ssi9", 1006, MOD_CLK_ID(1005)), + DEF_MOD("ssi8", 1007, MOD_CLK_ID(1005)), + DEF_MOD("ssi7", 1008, MOD_CLK_ID(1005)), + DEF_MOD("ssi6", 1009, MOD_CLK_ID(1005)), + DEF_MOD("ssi5", 1010, MOD_CLK_ID(1005)), + DEF_MOD("ssi4", 1011, MOD_CLK_ID(1005)), + DEF_MOD("ssi3", 1012, MOD_CLK_ID(1005)), + DEF_MOD("ssi2", 1013, MOD_CLK_ID(1005)), + DEF_MOD("ssi1", 1014, MOD_CLK_ID(1005)), + DEF_MOD("ssi0", 1015, MOD_CLK_ID(1005)), + DEF_MOD("scu-all", 1017, R8A77470_CLK_P), + DEF_MOD("scu-dvc1", 1018, MOD_CLK_ID(1017)), + DEF_MOD("scu-dvc0", 1019, MOD_CLK_ID(1017)), + DEF_MOD("scu-ctu1-mix1", 1020, MOD_CLK_ID(1017)), + DEF_MOD("scu-ctu0-mix0", 1021, MOD_CLK_ID(1017)), + DEF_MOD("scu-src6", 1025, MOD_CLK_ID(1017)), + DEF_MOD("scu-src5", 1026, MOD_CLK_ID(1017)), + DEF_MOD("scu-src4", 1027, MOD_CLK_ID(1017)), + DEF_MOD("scu-src3", 1028, MOD_CLK_ID(1017)), + DEF_MOD("scu-src2", 1029, MOD_CLK_ID(1017)), + DEF_MOD("scu-src1", 1030, MOD_CLK_ID(1017)), +}; + +static const unsigned int r8a77470_crit_mod_clks[] __initconst = { + MOD_CLK_ID(402), /* RWDT */ + MOD_CLK_ID(408), /* INTC-SYS (GIC) */ +}; + +/* + * CPG Clock Data + */ + +/* + * MD EXTAL PLL0 PLL1 PLL3 + * 14 13 (MHz) *1 *2 + *--------------------------------------------------- + * 0 0 20 x80 x78 x50 + * 0 1 26 x60 x60 x56 + * 1 0 Prohibitted setting + * 1 1 30 x52 x52 x50 + * + * *1 : Table 7.4 indicates VCO output (PLL0 = VCO) + * *2 : Table 7.4 indicates VCO output (PLL1 = VCO) + */ +#define CPG_PLL_CONFIG_INDEX(md) ((((md) & BIT(14)) >> 13) | \ + (((md) & BIT(13)) >> 13)) + +static const struct rcar_gen2_cpg_pll_config cpg_pll_configs[4] __initconst = { + /* EXTAL div PLL1 mult x2 PLL3 mult */ + { 1, 156, 50, }, + { 1, 120, 56, }, + { /* Invalid*/ }, + { 1, 104, 50, }, +}; + +static int __init r8a77470_cpg_mssr_init(struct device *dev) +{ + const struct rcar_gen2_cpg_pll_config *cpg_pll_config; + u32 cpg_mode; + int error; + + error = rcar_rst_read_mode_pins(&cpg_mode); + if (error) + return error; + + cpg_pll_config = &cpg_pll_configs[CPG_PLL_CONFIG_INDEX(cpg_mode)]; + + return rcar_gen2_cpg_init(cpg_pll_config, 2, cpg_mode); +} + +const struct cpg_mssr_info r8a77470_cpg_mssr_info __initconst = { + /* Core Clocks */ + .core_clks = r8a77470_core_clks, + .num_core_clks = ARRAY_SIZE(r8a77470_core_clks), + .last_dt_core_clk = LAST_DT_CORE_CLK, + .num_total_core_clks = MOD_CLK_BASE, + + /* Module Clocks */ + .mod_clks = r8a77470_mod_clks, + .num_mod_clks = ARRAY_SIZE(r8a77470_mod_clks), + .num_hw_mod_clks = 12 * 32, + + /* Critical Module Clocks */ + .crit_mod_clks = r8a77470_crit_mod_clks, + .num_crit_mod_clks = ARRAY_SIZE(r8a77470_crit_mod_clks), + + /* Callbacks */ + .init = r8a77470_cpg_mssr_init, + .cpg_clk_register = rcar_gen2_cpg_clk_register, +}; diff --git a/drivers/clk/renesas/r8a7791-cpg-mssr.c b/drivers/clk/renesas/r8a7791-cpg-mssr.c index 820b220b09cc..1b91f03b7598 100644 --- a/drivers/clk/renesas/r8a7791-cpg-mssr.c +++ b/drivers/clk/renesas/r8a7791-cpg-mssr.c @@ -57,7 +57,6 @@ static struct cpg_core_clk r8a7791_core_clks[] __initdata = { /* Core Clock Outputs */ DEF_BASE("z", R8A7791_CLK_Z, CLK_TYPE_GEN2_Z, CLK_PLL0), - DEF_BASE("lb", R8A7791_CLK_LB, CLK_TYPE_GEN2_LB, CLK_PLL1), DEF_BASE("adsp", R8A7791_CLK_ADSP, CLK_TYPE_GEN2_ADSP, CLK_PLL1), DEF_BASE("sdh", R8A7791_CLK_SDH, CLK_TYPE_GEN2_SDH, CLK_PLL1), DEF_BASE("sd0", R8A7791_CLK_SD0, CLK_TYPE_GEN2_SD0, CLK_PLL1), @@ -70,6 +69,7 @@ static struct cpg_core_clk r8a7791_core_clks[] __initdata = { DEF_FIXED("hp", R8A7791_CLK_HP, CLK_PLL1, 12, 1), DEF_FIXED("i", R8A7791_CLK_I, CLK_PLL1, 2, 1), DEF_FIXED("b", R8A7791_CLK_B, CLK_PLL1, 12, 1), + DEF_FIXED("lb", R8A7791_CLK_LB, CLK_PLL1, 24, 1), DEF_FIXED("p", R8A7791_CLK_P, CLK_PLL1, 24, 1), DEF_FIXED("cl", R8A7791_CLK_CL, CLK_PLL1, 48, 1), DEF_FIXED("m2", R8A7791_CLK_M2, CLK_PLL1, 8, 1), diff --git a/drivers/clk/renesas/r8a7792-cpg-mssr.c b/drivers/clk/renesas/r8a7792-cpg-mssr.c index 609a54080496..493e07859f5f 100644 --- a/drivers/clk/renesas/r8a7792-cpg-mssr.c +++ b/drivers/clk/renesas/r8a7792-cpg-mssr.c @@ -53,7 +53,6 @@ static const struct cpg_core_clk r8a7792_core_clks[] __initconst = { DEF_FIXED(".pll1_div2", CLK_PLL1_DIV2, CLK_PLL1, 2, 1), /* Core Clock Outputs */ - DEF_BASE("lb", R8A7792_CLK_LB, CLK_TYPE_GEN2_LB, CLK_PLL1), DEF_BASE("qspi", R8A7792_CLK_QSPI, CLK_TYPE_GEN2_QSPI, CLK_PLL1_DIV2), DEF_FIXED("z", R8A7792_CLK_Z, CLK_PLL0, 1, 1), @@ -63,6 +62,7 @@ static const struct cpg_core_clk r8a7792_core_clks[] __initconst = { DEF_FIXED("hp", R8A7792_CLK_HP, CLK_PLL1, 12, 1), DEF_FIXED("i", R8A7792_CLK_I, CLK_PLL1, 3, 1), DEF_FIXED("b", R8A7792_CLK_B, CLK_PLL1, 12, 1), + DEF_FIXED("lb", R8A7792_CLK_LB, CLK_PLL1, 24, 1), DEF_FIXED("p", R8A7792_CLK_P, CLK_PLL1, 24, 1), DEF_FIXED("cl", R8A7792_CLK_CL, CLK_PLL1, 48, 1), DEF_FIXED("m2", R8A7792_CLK_M2, CLK_PLL1, 8, 1), diff --git a/drivers/clk/renesas/r8a7794-cpg-mssr.c b/drivers/clk/renesas/r8a7794-cpg-mssr.c index 2a40bbeaeeaf..088f4b79fdfc 100644 --- a/drivers/clk/renesas/r8a7794-cpg-mssr.c +++ b/drivers/clk/renesas/r8a7794-cpg-mssr.c @@ -55,7 +55,6 @@ static const struct cpg_core_clk r8a7794_core_clks[] __initconst = { DEF_FIXED(".pll1_div2", CLK_PLL1_DIV2, CLK_PLL1, 2, 1), /* Core Clock Outputs */ - DEF_BASE("lb", R8A7794_CLK_LB, CLK_TYPE_GEN2_LB, CLK_PLL1), DEF_BASE("adsp", R8A7794_CLK_ADSP, CLK_TYPE_GEN2_ADSP, CLK_PLL1), DEF_BASE("sdh", R8A7794_CLK_SDH, CLK_TYPE_GEN2_SDH, CLK_PLL1), DEF_BASE("sd0", R8A7794_CLK_SD0, CLK_TYPE_GEN2_SD0, CLK_PLL1), @@ -69,6 +68,7 @@ static const struct cpg_core_clk r8a7794_core_clks[] __initconst = { DEF_FIXED("hp", R8A7794_CLK_HP, CLK_PLL1, 12, 1), DEF_FIXED("i", R8A7794_CLK_I, CLK_PLL1, 2, 1), DEF_FIXED("b", R8A7794_CLK_B, CLK_PLL1, 12, 1), + DEF_FIXED("lb", R8A7794_CLK_LB, CLK_PLL1, 24, 1), DEF_FIXED("p", R8A7794_CLK_P, CLK_PLL1, 24, 1), DEF_FIXED("cl", R8A7794_CLK_CL, CLK_PLL1, 48, 1), DEF_FIXED("cp", R8A7794_CLK_CP, CLK_PLL1, 48, 1), diff --git a/drivers/clk/renesas/r8a77965-cpg-mssr.c b/drivers/clk/renesas/r8a77965-cpg-mssr.c index b1acfb60351c..8fae5e9c4a77 100644 --- a/drivers/clk/renesas/r8a77965-cpg-mssr.c +++ b/drivers/clk/renesas/r8a77965-cpg-mssr.c @@ -116,6 +116,10 @@ static const struct mssr_mod_clk r8a77965_mod_clks[] __initconst = { DEF_MOD("scif3", 204, R8A77965_CLK_S3D4), DEF_MOD("scif1", 206, R8A77965_CLK_S3D4), DEF_MOD("scif0", 207, R8A77965_CLK_S3D4), + DEF_MOD("msiof3", 208, R8A77965_CLK_MSO), + DEF_MOD("msiof2", 209, R8A77965_CLK_MSO), + DEF_MOD("msiof1", 210, R8A77965_CLK_MSO), + DEF_MOD("msiof0", 211, R8A77965_CLK_MSO), DEF_MOD("sys-dmac2", 217, R8A77965_CLK_S0D3), DEF_MOD("sys-dmac1", 218, R8A77965_CLK_S0D3), DEF_MOD("sys-dmac0", 219, R8A77965_CLK_S0D3), diff --git a/drivers/clk/renesas/r8a77980-cpg-mssr.c b/drivers/clk/renesas/r8a77980-cpg-mssr.c index 7aaae73a321a..d7ebd9ec0059 100644 --- a/drivers/clk/renesas/r8a77980-cpg-mssr.c +++ b/drivers/clk/renesas/r8a77980-cpg-mssr.c @@ -116,7 +116,7 @@ static const struct mssr_mod_clk r8a77980_mod_clks[] __initconst = { DEF_MOD("sys-dmac1", 218, R8A77980_CLK_S0D3), DEF_MOD("tpu0", 304, R8A77980_CLK_S3D4), DEF_MOD("sdif", 314, R8A77980_CLK_SD0), - DEF_MOD("pciec0", 319, R8A77980_CLK_S3D1), + DEF_MOD("pciec0", 319, R8A77980_CLK_S2D2), DEF_MOD("intc-ex", 407, R8A77980_CLK_CP), DEF_MOD("intc-ap", 408, R8A77980_CLK_S0D3), DEF_MOD("hscif3", 517, R8A77980_CLK_S3D1), diff --git a/drivers/clk/renesas/r8a77990-cpg-mssr.c b/drivers/clk/renesas/r8a77990-cpg-mssr.c new file mode 100644 index 000000000000..9e14f1486fbb --- /dev/null +++ b/drivers/clk/renesas/r8a77990-cpg-mssr.c @@ -0,0 +1,289 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * r8a77990 Clock Pulse Generator / Module Standby and Software Reset + * + * Copyright (C) 2018 Renesas Electronics Corp. + * + * Based on r8a7795-cpg-mssr.c + * + * Copyright (C) 2015 Glider bvba + * Copyright (C) 2015 Renesas Electronics Corp. + */ + +#include +#include +#include +#include + +#include + +#include "renesas-cpg-mssr.h" +#include "rcar-gen3-cpg.h" + +enum clk_ids { + /* Core Clock Outputs exported to DT */ + LAST_DT_CORE_CLK = R8A77990_CLK_CPEX, + + /* External Input Clocks */ + CLK_EXTAL, + + /* Internal Core Clocks */ + CLK_MAIN, + CLK_PLL0, + CLK_PLL1, + CLK_PLL3, + CLK_PLL0D4, + CLK_PLL0D6, + CLK_PLL0D8, + CLK_PLL0D20, + CLK_PLL0D24, + CLK_PLL1D2, + CLK_PE, + CLK_S0, + CLK_S1, + CLK_S2, + CLK_S3, + CLK_SDSRC, + + /* Module Clocks */ + MOD_CLK_BASE +}; + +static const struct cpg_core_clk r8a77990_core_clks[] __initconst = { + /* External Clock Inputs */ + DEF_INPUT("extal", CLK_EXTAL), + + /* Internal Core Clocks */ + DEF_BASE(".main", CLK_MAIN, CLK_TYPE_GEN3_MAIN, CLK_EXTAL), + DEF_BASE(".pll1", CLK_PLL1, CLK_TYPE_GEN3_PLL1, CLK_MAIN), + DEF_BASE(".pll3", CLK_PLL3, CLK_TYPE_GEN3_PLL3, CLK_MAIN), + + DEF_FIXED(".pll0", CLK_PLL0, CLK_MAIN, 1, 100), + DEF_FIXED(".pll0d4", CLK_PLL0D4, CLK_PLL0, 4, 1), + DEF_FIXED(".pll0d6", CLK_PLL0D6, CLK_PLL0, 6, 1), + DEF_FIXED(".pll0d8", CLK_PLL0D8, CLK_PLL0, 8, 1), + DEF_FIXED(".pll0d20", CLK_PLL0D20, CLK_PLL0, 20, 1), + DEF_FIXED(".pll0d24", CLK_PLL0D24, CLK_PLL0, 24, 1), + DEF_FIXED(".pll1d2", CLK_PLL1D2, CLK_PLL1, 2, 1), + DEF_FIXED(".pe", CLK_PE, CLK_PLL0D20, 1, 1), + DEF_FIXED(".s0", CLK_S0, CLK_PLL1, 2, 1), + DEF_FIXED(".s1", CLK_S1, CLK_PLL1, 3, 1), + DEF_FIXED(".s2", CLK_S2, CLK_PLL1, 4, 1), + DEF_FIXED(".s3", CLK_S3, CLK_PLL1, 6, 1), + DEF_FIXED(".sdsrc", CLK_SDSRC, CLK_PLL1, 2, 1), + + /* Core Clock Outputs */ + DEF_FIXED("za2", R8A77990_CLK_ZA2, CLK_PLL0D24, 1, 1), + DEF_FIXED("za8", R8A77990_CLK_ZA8, CLK_PLL0D8, 1, 1), + DEF_FIXED("ztr", R8A77990_CLK_ZTR, CLK_PLL1, 6, 1), + DEF_FIXED("zt", R8A77990_CLK_ZT, CLK_PLL1, 4, 1), + DEF_FIXED("zx", R8A77990_CLK_ZX, CLK_PLL1, 3, 1), + DEF_FIXED("s0d1", R8A77990_CLK_S0D1, CLK_S0, 1, 1), + DEF_FIXED("s0d3", R8A77990_CLK_S0D3, CLK_S0, 3, 1), + DEF_FIXED("s0d6", R8A77990_CLK_S0D6, CLK_S0, 6, 1), + DEF_FIXED("s0d12", R8A77990_CLK_S0D12, CLK_S0, 12, 1), + DEF_FIXED("s0d24", R8A77990_CLK_S0D24, CLK_S0, 24, 1), + DEF_FIXED("s1d1", R8A77990_CLK_S1D1, CLK_S1, 1, 1), + DEF_FIXED("s1d2", R8A77990_CLK_S1D2, CLK_S1, 2, 1), + DEF_FIXED("s1d4", R8A77990_CLK_S1D4, CLK_S1, 4, 1), + DEF_FIXED("s2d1", R8A77990_CLK_S2D1, CLK_S2, 1, 1), + DEF_FIXED("s2d2", R8A77990_CLK_S2D2, CLK_S2, 2, 1), + DEF_FIXED("s2d4", R8A77990_CLK_S2D4, CLK_S2, 4, 1), + DEF_FIXED("s3d1", R8A77990_CLK_S3D1, CLK_S3, 1, 1), + DEF_FIXED("s3d2", R8A77990_CLK_S3D2, CLK_S3, 2, 1), + DEF_FIXED("s3d4", R8A77990_CLK_S3D4, CLK_S3, 4, 1), + + DEF_GEN3_SD("sd0", R8A77990_CLK_SD0, CLK_SDSRC, 0x0074), + DEF_GEN3_SD("sd1", R8A77990_CLK_SD1, CLK_SDSRC, 0x0078), + DEF_GEN3_SD("sd3", R8A77990_CLK_SD3, CLK_SDSRC, 0x026c), + + DEF_FIXED("cl", R8A77990_CLK_CL, CLK_PLL1, 48, 1), + DEF_FIXED("cp", R8A77990_CLK_CP, CLK_EXTAL, 2, 1), + DEF_FIXED("cpex", R8A77990_CLK_CPEX, CLK_EXTAL, 4, 1), + DEF_FIXED("osc", R8A77990_CLK_OSC, CLK_EXTAL, 384, 1), + DEF_FIXED("r", R8A77990_CLK_R, CLK_EXTAL, 1536, 1), + + DEF_GEN3_PE("s0d6c", R8A77990_CLK_S0D6C, CLK_S0, 6, CLK_PE, 2), + DEF_GEN3_PE("s3d1c", R8A77990_CLK_S3D1C, CLK_S3, 1, CLK_PE, 1), + DEF_GEN3_PE("s3d2c", R8A77990_CLK_S3D2C, CLK_S3, 2, CLK_PE, 2), + DEF_GEN3_PE("s3d4c", R8A77990_CLK_S3D4C, CLK_S3, 4, CLK_PE, 4), + + DEF_DIV6P1("canfd", R8A77990_CLK_CANFD, CLK_PLL0D6, 0x244), + DEF_DIV6P1("csi0", R8A77990_CLK_CSI0, CLK_PLL1D2, 0x00c), + DEF_DIV6P1("mso", R8A77990_CLK_MSO, CLK_PLL1D2, 0x014), +}; + +static const struct mssr_mod_clk r8a77990_mod_clks[] __initconst = { + DEF_MOD("scif5", 202, R8A77990_CLK_S3D4C), + DEF_MOD("scif4", 203, R8A77990_CLK_S3D4C), + DEF_MOD("scif3", 204, R8A77990_CLK_S3D4C), + DEF_MOD("scif1", 206, R8A77990_CLK_S3D4C), + DEF_MOD("scif0", 207, R8A77990_CLK_S3D4C), + DEF_MOD("msiof3", 208, R8A77990_CLK_MSO), + DEF_MOD("msiof2", 209, R8A77990_CLK_MSO), + DEF_MOD("msiof1", 210, R8A77990_CLK_MSO), + DEF_MOD("msiof0", 211, R8A77990_CLK_MSO), + DEF_MOD("sys-dmac2", 217, R8A77990_CLK_S3D1), + DEF_MOD("sys-dmac1", 218, R8A77990_CLK_S3D1), + DEF_MOD("sys-dmac0", 219, R8A77990_CLK_S3D1), + + DEF_MOD("cmt3", 300, R8A77990_CLK_R), + DEF_MOD("cmt2", 301, R8A77990_CLK_R), + DEF_MOD("cmt1", 302, R8A77990_CLK_R), + DEF_MOD("cmt0", 303, R8A77990_CLK_R), + DEF_MOD("scif2", 310, R8A77990_CLK_S3D4C), + DEF_MOD("sdif3", 311, R8A77990_CLK_SD3), + DEF_MOD("sdif1", 313, R8A77990_CLK_SD1), + DEF_MOD("sdif0", 314, R8A77990_CLK_SD0), + DEF_MOD("pcie0", 319, R8A77990_CLK_S3D1), + DEF_MOD("usb3-if0", 328, R8A77990_CLK_S3D1), + DEF_MOD("usb-dmac0", 330, R8A77990_CLK_S3D1), + DEF_MOD("usb-dmac1", 331, R8A77990_CLK_S3D1), + + DEF_MOD("rwdt", 402, R8A77990_CLK_R), + DEF_MOD("intc-ex", 407, R8A77990_CLK_CP), + DEF_MOD("intc-ap", 408, R8A77990_CLK_S0D3), + + DEF_MOD("audmac0", 502, R8A77990_CLK_S3D4), + DEF_MOD("drif7", 508, R8A77990_CLK_S3D2), + DEF_MOD("drif6", 509, R8A77990_CLK_S3D2), + DEF_MOD("drif5", 510, R8A77990_CLK_S3D2), + DEF_MOD("drif4", 511, R8A77990_CLK_S3D2), + DEF_MOD("drif3", 512, R8A77990_CLK_S3D2), + DEF_MOD("drif2", 513, R8A77990_CLK_S3D2), + DEF_MOD("drif1", 514, R8A77990_CLK_S3D2), + DEF_MOD("drif0", 515, R8A77990_CLK_S3D2), + DEF_MOD("hscif4", 516, R8A77990_CLK_S3D1C), + DEF_MOD("hscif3", 517, R8A77990_CLK_S3D1C), + DEF_MOD("hscif2", 518, R8A77990_CLK_S3D1C), + DEF_MOD("hscif1", 519, R8A77990_CLK_S3D1C), + DEF_MOD("hscif0", 520, R8A77990_CLK_S3D1C), + DEF_MOD("thermal", 522, R8A77990_CLK_CP), + DEF_MOD("pwm", 523, R8A77990_CLK_S3D4C), + + DEF_MOD("fcpvd1", 602, R8A77990_CLK_S1D2), + DEF_MOD("fcpvd0", 603, R8A77990_CLK_S1D2), + DEF_MOD("fcpvb0", 607, R8A77990_CLK_S0D1), + DEF_MOD("fcpvi0", 611, R8A77990_CLK_S0D1), + DEF_MOD("fcpf0", 615, R8A77990_CLK_S0D1), + DEF_MOD("fcpcs", 619, R8A77990_CLK_S0D1), + DEF_MOD("vspd1", 622, R8A77990_CLK_S1D2), + DEF_MOD("vspd0", 623, R8A77990_CLK_S1D2), + DEF_MOD("vspb", 626, R8A77990_CLK_S0D1), + DEF_MOD("vspi0", 631, R8A77990_CLK_S0D1), + + DEF_MOD("ehci0", 703, R8A77990_CLK_S3D4), + DEF_MOD("hsusb", 704, R8A77990_CLK_S3D4), + DEF_MOD("csi40", 716, R8A77990_CLK_CSI0), + DEF_MOD("du1", 723, R8A77990_CLK_S2D1), + DEF_MOD("du0", 724, R8A77990_CLK_S2D1), + DEF_MOD("lvds", 727, R8A77990_CLK_S2D1), + + DEF_MOD("vin5", 806, R8A77990_CLK_S1D2), + DEF_MOD("vin4", 807, R8A77990_CLK_S1D2), + DEF_MOD("etheravb", 812, R8A77990_CLK_S3D2), + + DEF_MOD("gpio6", 906, R8A77990_CLK_S3D4), + DEF_MOD("gpio5", 907, R8A77990_CLK_S3D4), + DEF_MOD("gpio4", 908, R8A77990_CLK_S3D4), + DEF_MOD("gpio3", 909, R8A77990_CLK_S3D4), + DEF_MOD("gpio2", 910, R8A77990_CLK_S3D4), + DEF_MOD("gpio1", 911, R8A77990_CLK_S3D4), + DEF_MOD("gpio0", 912, R8A77990_CLK_S3D4), + DEF_MOD("can-fd", 914, R8A77990_CLK_S3D2), + DEF_MOD("can-if1", 915, R8A77990_CLK_S3D4), + DEF_MOD("can-if0", 916, R8A77990_CLK_S3D4), + DEF_MOD("i2c6", 918, R8A77990_CLK_S3D2), + DEF_MOD("i2c5", 919, R8A77990_CLK_S3D2), + DEF_MOD("i2c-dvfs", 926, R8A77990_CLK_CP), + DEF_MOD("i2c4", 927, R8A77990_CLK_S3D2), + DEF_MOD("i2c3", 928, R8A77990_CLK_S3D2), + DEF_MOD("i2c2", 929, R8A77990_CLK_S3D2), + DEF_MOD("i2c1", 930, R8A77990_CLK_S3D2), + DEF_MOD("i2c0", 931, R8A77990_CLK_S3D2), + + DEF_MOD("ssi-all", 1005, R8A77990_CLK_S3D4), + DEF_MOD("ssi9", 1006, MOD_CLK_ID(1005)), + DEF_MOD("ssi8", 1007, MOD_CLK_ID(1005)), + DEF_MOD("ssi7", 1008, MOD_CLK_ID(1005)), + DEF_MOD("ssi6", 1009, MOD_CLK_ID(1005)), + DEF_MOD("ssi5", 1010, MOD_CLK_ID(1005)), + DEF_MOD("ssi4", 1011, MOD_CLK_ID(1005)), + DEF_MOD("ssi3", 1012, MOD_CLK_ID(1005)), + DEF_MOD("ssi2", 1013, MOD_CLK_ID(1005)), + DEF_MOD("ssi1", 1014, MOD_CLK_ID(1005)), + DEF_MOD("ssi0", 1015, MOD_CLK_ID(1005)), + DEF_MOD("scu-all", 1017, R8A77990_CLK_S3D4), + DEF_MOD("scu-dvc1", 1018, MOD_CLK_ID(1017)), + DEF_MOD("scu-dvc0", 1019, MOD_CLK_ID(1017)), + DEF_MOD("scu-ctu1-mix1", 1020, MOD_CLK_ID(1017)), + DEF_MOD("scu-ctu0-mix0", 1021, MOD_CLK_ID(1017)), + DEF_MOD("scu-src9", 1022, MOD_CLK_ID(1017)), + DEF_MOD("scu-src8", 1023, MOD_CLK_ID(1017)), + DEF_MOD("scu-src7", 1024, MOD_CLK_ID(1017)), + DEF_MOD("scu-src6", 1025, MOD_CLK_ID(1017)), + DEF_MOD("scu-src5", 1026, MOD_CLK_ID(1017)), + DEF_MOD("scu-src4", 1027, MOD_CLK_ID(1017)), + DEF_MOD("scu-src3", 1028, MOD_CLK_ID(1017)), + DEF_MOD("scu-src2", 1029, MOD_CLK_ID(1017)), + DEF_MOD("scu-src1", 1030, MOD_CLK_ID(1017)), + DEF_MOD("scu-src0", 1031, MOD_CLK_ID(1017)), +}; + +static const unsigned int r8a77990_crit_mod_clks[] __initconst = { + MOD_CLK_ID(408), /* INTC-AP (GIC) */ +}; + +/* + * CPG Clock Data + */ + +/* + * MD19 EXTAL (MHz) PLL0 PLL1 PLL3 + *-------------------------------------------------------------------- + * 0 48 x 1 x100/4 x100/3 x100/3 + * 1 48 x 1 x100/4 x100/3 x58/3 + */ +#define CPG_PLL_CONFIG_INDEX(md) (((md) & BIT(19)) >> 19) + +static const struct rcar_gen3_cpg_pll_config cpg_pll_configs[2] __initconst = { + /* EXTAL div PLL1 mult/div PLL3 mult/div */ + { 1, 100, 3, 100, 3, }, + { 1, 100, 3, 58, 3, }, +}; + +static int __init r8a77990_cpg_mssr_init(struct device *dev) +{ + const struct rcar_gen3_cpg_pll_config *cpg_pll_config; + u32 cpg_mode; + int error; + + error = rcar_rst_read_mode_pins(&cpg_mode); + if (error) + return error; + + cpg_pll_config = &cpg_pll_configs[CPG_PLL_CONFIG_INDEX(cpg_mode)]; + + return rcar_gen3_cpg_init(cpg_pll_config, 0, cpg_mode); +} + +const struct cpg_mssr_info r8a77990_cpg_mssr_info __initconst = { + /* Core Clocks */ + .core_clks = r8a77990_core_clks, + .num_core_clks = ARRAY_SIZE(r8a77990_core_clks), + .last_dt_core_clk = LAST_DT_CORE_CLK, + .num_total_core_clks = MOD_CLK_BASE, + + /* Module Clocks */ + .mod_clks = r8a77990_mod_clks, + .num_mod_clks = ARRAY_SIZE(r8a77990_mod_clks), + .num_hw_mod_clks = 12 * 32, + + /* Critical Module Clocks */ + .crit_mod_clks = r8a77990_crit_mod_clks, + .num_crit_mod_clks = ARRAY_SIZE(r8a77990_crit_mod_clks), + + /* Callbacks */ + .init = r8a77990_cpg_mssr_init, + .cpg_clk_register = rcar_gen3_cpg_clk_register, +}; diff --git a/drivers/clk/renesas/rcar-gen2-cpg.c b/drivers/clk/renesas/rcar-gen2-cpg.c index feb14579a71b..daf88bc2cdae 100644 --- a/drivers/clk/renesas/rcar-gen2-cpg.c +++ b/drivers/clk/renesas/rcar-gen2-cpg.c @@ -16,6 +16,7 @@ #include #include #include +#include #include "renesas-cpg-mssr.h" #include "rcar-gen2-cpg.h" @@ -260,6 +261,17 @@ static const struct clk_div_table cpg_sd01_div_table[] = { static const struct rcar_gen2_cpg_pll_config *cpg_pll_config __initdata; static unsigned int cpg_pll0_div __initdata; static u32 cpg_mode __initdata; +static u32 cpg_quirks __initdata; + +#define SD_SKIP_FIRST BIT(0) /* Skip first clock in SD table */ + +static const struct soc_device_attribute cpg_quirks_match[] __initconst = { + { + .soc_id = "r8a77470", + .data = (void *)SD_SKIP_FIRST, + }, + { /* sentinel */ } +}; struct clk * __init rcar_gen2_cpg_clk_register(struct device *dev, const struct cpg_core_clk *core, const struct cpg_mssr_info *info, @@ -327,11 +339,17 @@ struct clk * __init rcar_gen2_cpg_clk_register(struct device *dev, case CLK_TYPE_GEN2_SD0: table = cpg_sd01_div_table; + if (cpg_quirks & SD_SKIP_FIRST) + table++; + shift = 4; break; case CLK_TYPE_GEN2_SD1: table = cpg_sd01_div_table; + if (cpg_quirks & SD_SKIP_FIRST) + table++; + shift = 0; break; @@ -360,9 +378,15 @@ struct clk * __init rcar_gen2_cpg_clk_register(struct device *dev, int __init rcar_gen2_cpg_init(const struct rcar_gen2_cpg_pll_config *config, unsigned int pll0_div, u32 mode) { + const struct soc_device_attribute *attr; + cpg_pll_config = config; cpg_pll0_div = pll0_div; cpg_mode = mode; + attr = soc_device_match(cpg_quirks_match); + if (attr) + cpg_quirks = (uintptr_t)attr->data; + pr_debug("%s: mode = 0x%x quirks = 0x%x\n", __func__, mode, cpg_quirks); spin_lock_init(&cpg_lock); diff --git a/drivers/clk/renesas/renesas-cpg-mssr.c b/drivers/clk/renesas/renesas-cpg-mssr.c index 4e88e980fb76..f4b013e9352d 100644 --- a/drivers/clk/renesas/renesas-cpg-mssr.c +++ b/drivers/clk/renesas/renesas-cpg-mssr.c @@ -258,8 +258,9 @@ struct clk *cpg_mssr_clk_src_twocell_get(struct of_phandle_args *clkspec, dev_err(dev, "Cannot get %s clock %u: %ld", type, clkidx, PTR_ERR(clk)); else - dev_dbg(dev, "clock (%u, %u) is %pC at %pCr Hz\n", - clkspec->args[0], clkspec->args[1], clk, clk); + dev_dbg(dev, "clock (%u, %u) is %pC at %lu Hz\n", + clkspec->args[0], clkspec->args[1], clk, + clk_get_rate(clk)); return clk; } @@ -326,7 +327,7 @@ static void __init cpg_mssr_register_core_clk(const struct cpg_core_clk *core, if (IS_ERR_OR_NULL(clk)) goto fail; - dev_dbg(dev, "Core clock %pC at %pCr Hz\n", clk, clk); + dev_dbg(dev, "Core clock %pC at %lu Hz\n", clk, clk_get_rate(clk)); priv->clks[id] = clk; return; @@ -392,7 +393,7 @@ static void __init cpg_mssr_register_mod_clk(const struct mssr_mod_clk *mod, if (IS_ERR(clk)) goto fail; - dev_dbg(dev, "Module clock %pC at %pCr Hz\n", clk, clk); + dev_dbg(dev, "Module clock %pC at %lu Hz\n", clk, clk_get_rate(clk)); priv->clks[id] = clk; priv->smstpcr_saved[clock->index / 32].mask |= BIT(clock->index % 32); return; @@ -652,6 +653,12 @@ static const struct of_device_id cpg_mssr_match[] = { .data = &r8a7745_cpg_mssr_info, }, #endif +#ifdef CONFIG_CLK_R8A77470 + { + .compatible = "renesas,r8a77470-cpg-mssr", + .data = &r8a77470_cpg_mssr_info, + }, +#endif #ifdef CONFIG_CLK_R8A7790 { .compatible = "renesas,r8a7790-cpg-mssr", @@ -711,6 +718,12 @@ static const struct of_device_id cpg_mssr_match[] = { .data = &r8a77980_cpg_mssr_info, }, #endif +#ifdef CONFIG_CLK_R8A77990 + { + .compatible = "renesas,r8a77990-cpg-mssr", + .data = &r8a77990_cpg_mssr_info, + }, +#endif #ifdef CONFIG_CLK_R8A77995 { .compatible = "renesas,r8a77995-cpg-mssr", diff --git a/drivers/clk/renesas/renesas-cpg-mssr.h b/drivers/clk/renesas/renesas-cpg-mssr.h index 97ccb093c10f..642f720b9b05 100644 --- a/drivers/clk/renesas/renesas-cpg-mssr.h +++ b/drivers/clk/renesas/renesas-cpg-mssr.h @@ -133,6 +133,7 @@ struct cpg_mssr_info { extern const struct cpg_mssr_info r8a7743_cpg_mssr_info; extern const struct cpg_mssr_info r8a7745_cpg_mssr_info; +extern const struct cpg_mssr_info r8a77470_cpg_mssr_info; extern const struct cpg_mssr_info r8a7790_cpg_mssr_info; extern const struct cpg_mssr_info r8a7791_cpg_mssr_info; extern const struct cpg_mssr_info r8a7792_cpg_mssr_info; @@ -142,6 +143,7 @@ extern const struct cpg_mssr_info r8a7796_cpg_mssr_info; extern const struct cpg_mssr_info r8a77965_cpg_mssr_info; extern const struct cpg_mssr_info r8a77970_cpg_mssr_info; extern const struct cpg_mssr_info r8a77980_cpg_mssr_info; +extern const struct cpg_mssr_info r8a77990_cpg_mssr_info; extern const struct cpg_mssr_info r8a77995_cpg_mssr_info; diff --git a/drivers/clk/rockchip/Makefile b/drivers/clk/rockchip/Makefile index 59b8d320960a..98e7b9429b83 100644 --- a/drivers/clk/rockchip/Makefile +++ b/drivers/clk/rockchip/Makefile @@ -3,7 +3,6 @@ # Rockchip Clock specific Makefile # -obj-y += clk-rockchip.o obj-y += clk.o obj-y += clk-pll.o obj-y += clk-cpu.o diff --git a/drivers/clk/rockchip/clk-rockchip.c b/drivers/clk/rockchip/clk-rockchip.c deleted file mode 100644 index 2c9bb81144c9..000000000000 --- a/drivers/clk/rockchip/clk-rockchip.c +++ /dev/null @@ -1,98 +0,0 @@ -/* - * Copyright (c) 2013 MundoReader S.L. - * Author: Heiko Stuebner - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - */ - -#include -#include -#include -#include - -static DEFINE_SPINLOCK(clk_lock); - -/* - * Gate clocks - */ - -static void __init rk2928_gate_clk_init(struct device_node *node) -{ - struct clk_onecell_data *clk_data; - const char *clk_parent; - const char *clk_name; - void __iomem *reg; - void __iomem *reg_idx; - int flags; - int qty; - int reg_bit; - int clkflags = CLK_SET_RATE_PARENT; - int i; - - qty = of_property_count_strings(node, "clock-output-names"); - if (qty < 0) { - pr_err("%s: error in clock-output-names %d\n", __func__, qty); - return; - } - - if (qty == 0) { - pr_info("%s: nothing to do\n", __func__); - return; - } - - reg = of_iomap(node, 0); - if (!reg) - return; - - clk_data = kzalloc(sizeof(struct clk_onecell_data), GFP_KERNEL); - if (!clk_data) { - iounmap(reg); - return; - } - - clk_data->clks = kzalloc(qty * sizeof(struct clk *), GFP_KERNEL); - if (!clk_data->clks) { - kfree(clk_data); - iounmap(reg); - return; - } - - flags = CLK_GATE_HIWORD_MASK | CLK_GATE_SET_TO_DISABLE; - - for (i = 0; i < qty; i++) { - of_property_read_string_index(node, "clock-output-names", - i, &clk_name); - - /* ignore empty slots */ - if (!strcmp("reserved", clk_name)) - continue; - - clk_parent = of_clk_get_parent_name(node, i); - - /* keep all gates untouched for now */ - clkflags |= CLK_IGNORE_UNUSED; - - reg_idx = reg + (4 * (i / 16)); - reg_bit = (i % 16); - - clk_data->clks[i] = clk_register_gate(NULL, clk_name, - clk_parent, clkflags, - reg_idx, reg_bit, - flags, - &clk_lock); - WARN_ON(IS_ERR(clk_data->clks[i])); - } - - clk_data->clk_num = qty; - - of_clk_add_provider(node, of_clk_src_onecell_get, clk_data); -} -CLK_OF_DECLARE(rk2928_gate, "rockchip,rk2928-gate-clk", rk2928_gate_clk_init); diff --git a/drivers/clk/rockchip/clk.c b/drivers/clk/rockchip/clk.c index 3cd8ad59e0b7..326b3fa44f5d 100644 --- a/drivers/clk/rockchip/clk.c +++ b/drivers/clk/rockchip/clk.c @@ -274,18 +274,10 @@ static struct clk *rockchip_clk_register_frac_branch( struct clk_mux *frac_mux = &frac->mux; struct clk_init_data init; struct clk *mux_clk; - int i, ret; - - frac->mux_frac_idx = -1; - for (i = 0; i < child->num_parents; i++) { - if (!strcmp(name, child->parent_names[i])) { - pr_debug("%s: found fractional parent in mux at pos %d\n", - __func__, i); - frac->mux_frac_idx = i; - break; - } - } + int ret; + frac->mux_frac_idx = match_string(child->parent_names, + child->num_parents, name); frac->mux_ops = &clk_mux_ops; frac->clk_nb.notifier_call = rockchip_clk_frac_notifier_cb; @@ -312,6 +304,8 @@ static struct clk *rockchip_clk_register_frac_branch( /* notifier on the fraction divider to catch rate changes */ if (frac->mux_frac_idx >= 0) { + pr_debug("%s: found fractional parent in mux at pos %d\n", + __func__, frac->mux_frac_idx); ret = clk_notifier_register(clk, &frac->clk_nb); if (ret) pr_err("%s: failed to register clock notifier for %s\n", diff --git a/drivers/clk/samsung/clk-exynos-audss.c b/drivers/clk/samsung/clk-exynos-audss.c index b4b057c7301c..f659c5cbf1d5 100644 --- a/drivers/clk/samsung/clk-exynos-audss.c +++ b/drivers/clk/samsung/clk-exynos-audss.c @@ -149,8 +149,8 @@ static int exynos_audss_clk_probe(struct platform_device *pdev) epll = ERR_PTR(-ENODEV); clk_data = devm_kzalloc(dev, - sizeof(*clk_data) + - sizeof(*clk_data->hws) * EXYNOS_AUDSS_MAX_CLKS, + struct_size(clk_data, hws, + EXYNOS_AUDSS_MAX_CLKS), GFP_KERNEL); if (!clk_data) return -ENOMEM; diff --git a/drivers/clk/samsung/clk-exynos-clkout.c b/drivers/clk/samsung/clk-exynos-clkout.c index f29fb5824005..9c95390d2d77 100644 --- a/drivers/clk/samsung/clk-exynos-clkout.c +++ b/drivers/clk/samsung/clk-exynos-clkout.c @@ -61,8 +61,7 @@ static void __init exynos_clkout_init(struct device_node *node, u32 mux_mask) int ret; int i; - clkout = kzalloc(sizeof(*clkout) + - sizeof(*clkout->data.hws) * EXYNOS_CLKOUT_NR_CLKS, + clkout = kzalloc(struct_size(clkout, data.hws, EXYNOS_CLKOUT_NR_CLKS), GFP_KERNEL); if (!clkout) return; diff --git a/drivers/clk/samsung/clk-exynos5433.c b/drivers/clk/samsung/clk-exynos5433.c index 5305ace514b2..162de44df099 100644 --- a/drivers/clk/samsung/clk-exynos5433.c +++ b/drivers/clk/samsung/clk-exynos5433.c @@ -5505,8 +5505,8 @@ static int __init exynos5433_cmu_probe(struct platform_device *pdev) info = of_device_get_match_data(dev); - data = devm_kzalloc(dev, sizeof(*data) + - sizeof(*data->ctx.clk_data.hws) * info->nr_clk_ids, + data = devm_kzalloc(dev, + struct_size(data, ctx.clk_data.hws, info->nr_clk_ids), GFP_KERNEL); if (!data) return -ENOMEM; diff --git a/drivers/clk/samsung/clk-s3c2410-dclk.c b/drivers/clk/samsung/clk-s3c2410-dclk.c index 077df3e539a7..0d92f3e5e3d9 100644 --- a/drivers/clk/samsung/clk-s3c2410-dclk.c +++ b/drivers/clk/samsung/clk-s3c2410-dclk.c @@ -219,8 +219,7 @@ static int s3c24xx_dclk1_div_notify(struct notifier_block *nb, #ifdef CONFIG_PM_SLEEP static int s3c24xx_dclk_suspend(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct s3c24xx_dclk *s3c24xx_dclk = platform_get_drvdata(pdev); + struct s3c24xx_dclk *s3c24xx_dclk = dev_get_drvdata(dev); s3c24xx_dclk->reg_save = readl_relaxed(s3c24xx_dclk->base); return 0; @@ -228,8 +227,7 @@ static int s3c24xx_dclk_suspend(struct device *dev) static int s3c24xx_dclk_resume(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct s3c24xx_dclk *s3c24xx_dclk = platform_get_drvdata(pdev); + struct s3c24xx_dclk *s3c24xx_dclk = dev_get_drvdata(dev); writel_relaxed(s3c24xx_dclk->reg_save, s3c24xx_dclk->base); return 0; @@ -247,9 +245,10 @@ static int s3c24xx_dclk_probe(struct platform_device *pdev) struct clk_hw **clk_table; int ret, i; - s3c24xx_dclk = devm_kzalloc(&pdev->dev, sizeof(*s3c24xx_dclk) + - sizeof(*s3c24xx_dclk->clk_data.hws) * DCLK_MAX_CLKS, - GFP_KERNEL); + s3c24xx_dclk = devm_kzalloc(&pdev->dev, + struct_size(s3c24xx_dclk, clk_data.hws, + DCLK_MAX_CLKS), + GFP_KERNEL); if (!s3c24xx_dclk) return -ENOMEM; diff --git a/drivers/clk/samsung/clk-s5pv210-audss.c b/drivers/clk/samsung/clk-s5pv210-audss.c index b9641414ddc6..22b18e728b88 100644 --- a/drivers/clk/samsung/clk-s5pv210-audss.c +++ b/drivers/clk/samsung/clk-s5pv210-audss.c @@ -81,8 +81,7 @@ static int s5pv210_audss_clk_probe(struct platform_device *pdev) } clk_data = devm_kzalloc(&pdev->dev, - sizeof(*clk_data) + - sizeof(*clk_data->hws) * AUDSS_MAX_CLKS, + struct_size(clk_data, hws, AUDSS_MAX_CLKS), GFP_KERNEL); if (!clk_data) diff --git a/drivers/clk/socfpga/clk-s10.c b/drivers/clk/socfpga/clk-s10.c index 3a11c382a663..72714633e39c 100644 --- a/drivers/clk/socfpga/clk-s10.c +++ b/drivers/clk/socfpga/clk-s10.c @@ -260,46 +260,45 @@ static int s10_clk_register_pll(const struct stratix10_pll_clock *clks, return 0; } -static struct stratix10_clock_data *__socfpga_s10_clk_init(struct device_node *np, +static struct stratix10_clock_data *__socfpga_s10_clk_init(struct platform_device *pdev, int nr_clks) { + struct device_node *np = pdev->dev.of_node; + struct device *dev = &pdev->dev; struct stratix10_clock_data *clk_data; struct clk **clk_table; + struct resource *res; void __iomem *base; - base = of_iomap(np, 0); - if (!base) { + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + base = devm_ioremap_resource(dev, res); + if (IS_ERR(base)) { pr_err("%s: failed to map clock registers\n", __func__); - goto err; + return ERR_CAST(base); } - clk_data = kzalloc(sizeof(*clk_data), GFP_KERNEL); + clk_data = devm_kzalloc(dev, sizeof(*clk_data), GFP_KERNEL); if (!clk_data) - goto err; + return ERR_PTR(-ENOMEM); clk_data->base = base; - clk_table = kcalloc(nr_clks, sizeof(*clk_table), GFP_KERNEL); + clk_table = devm_kcalloc(dev, nr_clks, sizeof(*clk_table), GFP_KERNEL); if (!clk_table) - goto err_data; + return ERR_PTR(-ENOMEM); clk_data->clk_data.clks = clk_table; clk_data->clk_data.clk_num = nr_clks; of_clk_add_provider(np, of_clk_src_onecell_get, &clk_data->clk_data); return clk_data; - -err_data: - kfree(clk_data); -err: - return NULL; } -static int s10_clkmgr_init(struct device_node *np) +static int s10_clkmgr_init(struct platform_device *pdev) { struct stratix10_clock_data *clk_data; - clk_data = __socfpga_s10_clk_init(np, STRATIX10_NUM_CLKS); - if (!clk_data) - return -ENOMEM; + clk_data = __socfpga_s10_clk_init(pdev, STRATIX10_NUM_CLKS); + if (IS_ERR(clk_data)) + return PTR_ERR(clk_data); s10_clk_register_pll(s10_pll_clks, ARRAY_SIZE(s10_pll_clks), clk_data); @@ -317,11 +316,7 @@ static int s10_clkmgr_init(struct device_node *np) static int s10_clkmgr_probe(struct platform_device *pdev) { - struct device_node *np = pdev->dev.of_node; - - s10_clkmgr_init(np); - - return 0; + return s10_clkmgr_init(pdev); } static const struct of_device_id stratix10_clkmgr_match_table[] = { @@ -334,6 +329,7 @@ static struct platform_driver stratix10_clkmgr_driver = { .probe = s10_clkmgr_probe, .driver = { .name = "stratix10-clkmgr", + .suppress_bind_attrs = true, .of_match_table = stratix10_clkmgr_match_table, }, }; diff --git a/drivers/clk/spear/spear6xx_clock.c b/drivers/clk/spear/spear6xx_clock.c index f911d9f77763..47810be7f15c 100644 --- a/drivers/clk/spear/spear6xx_clock.c +++ b/drivers/clk/spear/spear6xx_clock.c @@ -147,7 +147,7 @@ void __init spear6xx_clk_init(void __iomem *misc_base) clk = clk_register_fixed_factor(NULL, "wdt_clk", "osc_30m_clk", 0, 1, 1); - clk_register_clkdev(clk, NULL, "wdt"); + clk_register_clkdev(clk, NULL, "fc880000.wdt"); /* clock derived from pll1 clk */ clk = clk_register_fixed_factor(NULL, "cpu_clk", "pll1_clk", diff --git a/drivers/clk/st/clkgen-fsyn.c b/drivers/clk/st/clkgen-fsyn.c index 14819d919df1..a79d81985c4e 100644 --- a/drivers/clk/st/clkgen-fsyn.c +++ b/drivers/clk/st/clkgen-fsyn.c @@ -874,7 +874,7 @@ static void __init st_of_create_quadfs_fsynths( return; clk_data->clk_num = QUADFS_MAX_CHAN; - clk_data->clks = kzalloc(QUADFS_MAX_CHAN * sizeof(struct clk *), + clk_data->clks = kcalloc(QUADFS_MAX_CHAN, sizeof(struct clk *), GFP_KERNEL); if (!clk_data->clks) { diff --git a/drivers/clk/st/clkgen-pll.c b/drivers/clk/st/clkgen-pll.c index 25bda48a5d35..7a7106dc80bf 100644 --- a/drivers/clk/st/clkgen-pll.c +++ b/drivers/clk/st/clkgen-pll.c @@ -738,7 +738,7 @@ static void __init clkgen_c32_pll_setup(struct device_node *np, return; clk_data->clk_num = num_odfs; - clk_data->clks = kzalloc(clk_data->clk_num * sizeof(struct clk *), + clk_data->clks = kcalloc(clk_data->clk_num, sizeof(struct clk *), GFP_KERNEL); if (!clk_data->clks) diff --git a/drivers/clk/sunxi-ng/Kconfig b/drivers/clk/sunxi-ng/Kconfig index 79dfd296c3d1..826674d090fd 100644 --- a/drivers/clk/sunxi-ng/Kconfig +++ b/drivers/clk/sunxi-ng/Kconfig @@ -16,6 +16,11 @@ config SUN50I_H6_CCU default ARM64 && ARCH_SUNXI depends on (ARM64 && ARCH_SUNXI) || COMPILE_TEST +config SUN50I_H6_R_CCU + bool "Support for the Allwinner H6 PRCM CCU" + default ARM64 && ARCH_SUNXI + depends on (ARM64 && ARCH_SUNXI) || COMPILE_TEST + config SUN4I_A10_CCU bool "Support for the Allwinner A10/A20 CCU" default MACH_SUN4I diff --git a/drivers/clk/sunxi-ng/Makefile b/drivers/clk/sunxi-ng/Makefile index 128a40ee5c5e..acaa14cfa25c 100644 --- a/drivers/clk/sunxi-ng/Makefile +++ b/drivers/clk/sunxi-ng/Makefile @@ -23,6 +23,7 @@ lib-$(CONFIG_SUNXI_CCU) += ccu_mp.o # SoC support obj-$(CONFIG_SUN50I_A64_CCU) += ccu-sun50i-a64.o obj-$(CONFIG_SUN50I_H6_CCU) += ccu-sun50i-h6.o +obj-$(CONFIG_SUN50I_H6_R_CCU) += ccu-sun50i-h6-r.o obj-$(CONFIG_SUN4I_A10_CCU) += ccu-sun4i-a10.o obj-$(CONFIG_SUN5I_CCU) += ccu-sun5i.o obj-$(CONFIG_SUN6I_A31_CCU) += ccu-sun6i-a31.o diff --git a/drivers/clk/sunxi-ng/ccu-sun50i-h6-r.c b/drivers/clk/sunxi-ng/ccu-sun50i-h6-r.c new file mode 100644 index 000000000000..27554eaf6929 --- /dev/null +++ b/drivers/clk/sunxi-ng/ccu-sun50i-h6-r.c @@ -0,0 +1,207 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2017 Icenowy Zheng + */ + +#include +#include +#include + +#include "ccu_common.h" +#include "ccu_reset.h" + +#include "ccu_div.h" +#include "ccu_gate.h" +#include "ccu_mp.h" +#include "ccu_nm.h" + +#include "ccu-sun50i-h6-r.h" + +/* + * Information about AR100 and AHB/APB clocks in R_CCU are gathered from + * clock definitions in the BSP source code. + */ + +static const char * const ar100_r_apb2_parents[] = { "osc24M", "osc32k", + "pll-periph0", "iosc" }; +static const struct ccu_mux_var_prediv ar100_r_apb2_predivs[] = { + { .index = 2, .shift = 0, .width = 5 }, +}; + +static struct ccu_div ar100_clk = { + .div = _SUNXI_CCU_DIV_FLAGS(8, 2, CLK_DIVIDER_POWER_OF_TWO), + + .mux = { + .shift = 24, + .width = 2, + + .var_predivs = ar100_r_apb2_predivs, + .n_var_predivs = ARRAY_SIZE(ar100_r_apb2_predivs), + }, + + .common = { + .reg = 0x000, + .features = CCU_FEATURE_VARIABLE_PREDIV, + .hw.init = CLK_HW_INIT_PARENTS("ar100", + ar100_r_apb2_parents, + &ccu_div_ops, + 0), + }, +}; + +static CLK_FIXED_FACTOR(r_ahb_clk, "r-ahb", "ar100", 1, 1, 0); + +static struct ccu_div r_apb1_clk = { + .div = _SUNXI_CCU_DIV(0, 2), + + .common = { + .reg = 0x00c, + .hw.init = CLK_HW_INIT("r-apb1", + "r-ahb", + &ccu_div_ops, + 0), + }, +}; + +static struct ccu_div r_apb2_clk = { + .div = _SUNXI_CCU_DIV_FLAGS(8, 2, CLK_DIVIDER_POWER_OF_TWO), + + .mux = { + .shift = 24, + .width = 2, + + .var_predivs = ar100_r_apb2_predivs, + .n_var_predivs = ARRAY_SIZE(ar100_r_apb2_predivs), + }, + + .common = { + .reg = 0x010, + .features = CCU_FEATURE_VARIABLE_PREDIV, + .hw.init = CLK_HW_INIT_PARENTS("r-apb2", + ar100_r_apb2_parents, + &ccu_div_ops, + 0), + }, +}; + +/* + * Information about the gate/resets are gathered from the clock header file + * in the BSP source code, although most of them are unused. The existence + * of the hardware block is verified with "3.1 Memory Mapping" chapter in + * "Allwinner H6 V200 User Manual V1.1"; and the parent APB buses are verified + * with "3.3.2.1 System Bus Tree" chapter inthe same document. + */ +static SUNXI_CCU_GATE(r_apb1_timer_clk, "r-apb1-timer", "r-apb1", + 0x11c, BIT(0), 0); +static SUNXI_CCU_GATE(r_apb1_twd_clk, "r-apb1-twd", "r-apb1", + 0x12c, BIT(0), 0); +static SUNXI_CCU_GATE(r_apb1_pwm_clk, "r-apb1-pwm", "r-apb1", + 0x13c, BIT(0), 0); +static SUNXI_CCU_GATE(r_apb2_uart_clk, "r-apb2-uart", "r-apb2", + 0x18c, BIT(0), 0); +static SUNXI_CCU_GATE(r_apb2_i2c_clk, "r-apb2-i2c", "r-apb2", + 0x19c, BIT(0), 0); +static SUNXI_CCU_GATE(r_apb1_ir_clk, "r-apb1-ir", "r-apb1", + 0x1cc, BIT(0), 0); +static SUNXI_CCU_GATE(r_apb1_w1_clk, "r-apb1-w1", "r-apb1", + 0x1cc, BIT(0), 0); + +/* Information of IR(RX) mod clock is gathered from BSP source code */ +static const char * const r_mod0_default_parents[] = { "osc32k", "osc24M" }; +static SUNXI_CCU_MP_WITH_MUX_GATE(ir_clk, "ir", + r_mod0_default_parents, 0x1c0, + 0, 5, /* M */ + 8, 2, /* P */ + 24, 1, /* mux */ + BIT(31), /* gate */ + 0); + +/* + * BSP didn't use the 1-wire function at all now, and the information about + * this mod clock is guessed from the IR mod clock above. The existence of + * this mod clock is proven by BSP clock header, and the dividers are verified + * by contents in the 1-wire related chapter of the User Manual. + */ + +static SUNXI_CCU_MP_WITH_MUX_GATE(w1_clk, "w1", + r_mod0_default_parents, 0x1e0, + 0, 5, /* M */ + 8, 2, /* P */ + 24, 1, /* mux */ + BIT(31), /* gate */ + 0); + +static struct ccu_common *sun50i_h6_r_ccu_clks[] = { + &ar100_clk.common, + &r_apb1_clk.common, + &r_apb2_clk.common, + &r_apb1_timer_clk.common, + &r_apb1_twd_clk.common, + &r_apb1_pwm_clk.common, + &r_apb2_uart_clk.common, + &r_apb2_i2c_clk.common, + &r_apb1_ir_clk.common, + &r_apb1_w1_clk.common, + &ir_clk.common, + &w1_clk.common, +}; + +static struct clk_hw_onecell_data sun50i_h6_r_hw_clks = { + .hws = { + [CLK_AR100] = &ar100_clk.common.hw, + [CLK_R_AHB] = &r_ahb_clk.hw, + [CLK_R_APB1] = &r_apb1_clk.common.hw, + [CLK_R_APB2] = &r_apb2_clk.common.hw, + [CLK_R_APB1_TIMER] = &r_apb1_timer_clk.common.hw, + [CLK_R_APB1_TWD] = &r_apb1_twd_clk.common.hw, + [CLK_R_APB1_PWM] = &r_apb1_pwm_clk.common.hw, + [CLK_R_APB2_UART] = &r_apb2_uart_clk.common.hw, + [CLK_R_APB2_I2C] = &r_apb2_i2c_clk.common.hw, + [CLK_R_APB1_IR] = &r_apb1_ir_clk.common.hw, + [CLK_R_APB1_W1] = &r_apb1_w1_clk.common.hw, + [CLK_IR] = &ir_clk.common.hw, + [CLK_W1] = &w1_clk.common.hw, + }, + .num = CLK_NUMBER, +}; + +static struct ccu_reset_map sun50i_h6_r_ccu_resets[] = { + [RST_R_APB1_TIMER] = { 0x11c, BIT(16) }, + [RST_R_APB1_TWD] = { 0x12c, BIT(16) }, + [RST_R_APB1_PWM] = { 0x13c, BIT(16) }, + [RST_R_APB2_UART] = { 0x18c, BIT(16) }, + [RST_R_APB2_I2C] = { 0x19c, BIT(16) }, + [RST_R_APB1_IR] = { 0x1cc, BIT(16) }, + [RST_R_APB1_W1] = { 0x1ec, BIT(16) }, +}; + +static const struct sunxi_ccu_desc sun50i_h6_r_ccu_desc = { + .ccu_clks = sun50i_h6_r_ccu_clks, + .num_ccu_clks = ARRAY_SIZE(sun50i_h6_r_ccu_clks), + + .hw_clks = &sun50i_h6_r_hw_clks, + + .resets = sun50i_h6_r_ccu_resets, + .num_resets = ARRAY_SIZE(sun50i_h6_r_ccu_resets), +}; + +static void __init sunxi_r_ccu_init(struct device_node *node, + const struct sunxi_ccu_desc *desc) +{ + void __iomem *reg; + + reg = of_io_request_and_map(node, 0, of_node_full_name(node)); + if (IS_ERR(reg)) { + pr_err("%pOF: Could not map the clock registers\n", node); + return; + } + + sunxi_ccu_probe(node, reg, desc); +} + +static void __init sun50i_h6_r_ccu_setup(struct device_node *node) +{ + sunxi_r_ccu_init(node, &sun50i_h6_r_ccu_desc); +} +CLK_OF_DECLARE(sun50i_h6_r_ccu, "allwinner,sun50i-h6-r-ccu", + sun50i_h6_r_ccu_setup); diff --git a/drivers/clk/sunxi-ng/ccu-sun50i-h6-r.h b/drivers/clk/sunxi-ng/ccu-sun50i-h6-r.h new file mode 100644 index 000000000000..782117dc0b28 --- /dev/null +++ b/drivers/clk/sunxi-ng/ccu-sun50i-h6-r.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright 2017 Icenowy Zheng + */ + +#ifndef _CCU_SUN50I_H6_R_H +#define _CCU_SUN50I_H6_R_H + +#include +#include + +/* AHB/APB bus clocks are not exported except APB1 for R_PIO */ +#define CLK_R_AHB 1 + +#define CLK_R_APB2 3 + +#define CLK_NUMBER (CLK_W1 + 1) + +#endif /* _CCU_SUN50I_H6_R_H */ diff --git a/drivers/clk/sunxi-ng/ccu-sun8i-r40.c b/drivers/clk/sunxi-ng/ccu-sun8i-r40.c index 933f2e68f42a..65ba6455feb7 100644 --- a/drivers/clk/sunxi-ng/ccu-sun8i-r40.c +++ b/drivers/clk/sunxi-ng/ccu-sun8i-r40.c @@ -12,7 +12,8 @@ */ #include -#include +#include +#include #include "ccu_common.h" #include "ccu_reset.h" @@ -1250,17 +1251,45 @@ static struct ccu_mux_nb sun8i_r40_cpu_nb = { .bypass_index = 1, /* index of 24 MHz oscillator */ }; -static void __init sun8i_r40_ccu_setup(struct device_node *node) +/* + * Add a regmap for the GMAC driver (dwmac-sun8i) to access the + * GMAC configuration register. + * Only this register is allowed to be written, in order to + * prevent overriding critical clock configuration. + */ + +#define SUN8I_R40_GMAC_CFG_REG 0x164 +static bool sun8i_r40_ccu_regmap_accessible_reg(struct device *dev, + unsigned int reg) +{ + if (reg == SUN8I_R40_GMAC_CFG_REG) + return true; + return false; +} + +static struct regmap_config sun8i_r40_ccu_regmap_config = { + .reg_bits = 32, + .val_bits = 32, + .reg_stride = 4, + .max_register = 0x320, /* PLL_LOCK_CTRL_REG */ + + /* other devices have no business accessing other registers */ + .readable_reg = sun8i_r40_ccu_regmap_accessible_reg, + .writeable_reg = sun8i_r40_ccu_regmap_accessible_reg, +}; + +static int sun8i_r40_ccu_probe(struct platform_device *pdev) { + struct resource *res; + struct regmap *regmap; void __iomem *reg; u32 val; + int ret; - reg = of_io_request_and_map(node, 0, of_node_full_name(node)); - if (IS_ERR(reg)) { - pr_err("%s: Could not map the clock registers\n", - of_node_full_name(node)); - return; - } + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + reg = devm_ioremap_resource(&pdev->dev, res); + if (IS_ERR(reg)) + return PTR_ERR(reg); /* Force the PLL-Audio-1x divider to 4 */ val = readl(reg + SUN8I_R40_PLL_AUDIO_REG); @@ -1277,7 +1306,14 @@ static void __init sun8i_r40_ccu_setup(struct device_node *node) val &= ~GENMASK(25, 20); writel(val, reg + SUN8I_R40_USB_CLK_REG); - sunxi_ccu_probe(node, reg, &sun8i_r40_ccu_desc); + regmap = devm_regmap_init_mmio(&pdev->dev, reg, + &sun8i_r40_ccu_regmap_config); + if (IS_ERR(regmap)) + return PTR_ERR(regmap); + + ret = sunxi_ccu_probe(pdev->dev.of_node, reg, &sun8i_r40_ccu_desc); + if (ret) + return ret; /* Gate then ungate PLL CPU after any rate changes */ ccu_pll_notifier_register(&sun8i_r40_pll_cpu_nb); @@ -1285,6 +1321,20 @@ static void __init sun8i_r40_ccu_setup(struct device_node *node) /* Reparent CPU during PLL CPU rate changes */ ccu_mux_notifier_register(pll_cpu_clk.common.hw.clk, &sun8i_r40_cpu_nb); + + return 0; } -CLK_OF_DECLARE(sun8i_r40_ccu, "allwinner,sun8i-r40-ccu", - sun8i_r40_ccu_setup); + +static const struct of_device_id sun8i_r40_ccu_ids[] = { + { .compatible = "allwinner,sun8i-r40-ccu" }, + { } +}; + +static struct platform_driver sun8i_r40_ccu_driver = { + .probe = sun8i_r40_ccu_probe, + .driver = { + .name = "sun8i-r40-ccu", + .of_match_table = sun8i_r40_ccu_ids, + }, +}; +builtin_platform_driver(sun8i_r40_ccu_driver); diff --git a/drivers/clk/sunxi/clk-usb.c b/drivers/clk/sunxi/clk-usb.c index fe0c3d169377..917fc27a33dd 100644 --- a/drivers/clk/sunxi/clk-usb.c +++ b/drivers/clk/sunxi/clk-usb.c @@ -122,7 +122,7 @@ static void __init sunxi_usb_clk_setup(struct device_node *node, if (!clk_data) return; - clk_data->clks = kzalloc((qty+1) * sizeof(struct clk *), GFP_KERNEL); + clk_data->clks = kcalloc(qty + 1, sizeof(struct clk *), GFP_KERNEL); if (!clk_data->clks) { kfree(clk_data); return; diff --git a/drivers/clk/tegra/clk-dfll.c b/drivers/clk/tegra/clk-dfll.c index 0a7deee74eea..48ee43734e05 100644 --- a/drivers/clk/tegra/clk-dfll.c +++ b/drivers/clk/tegra/clk-dfll.c @@ -1196,42 +1196,24 @@ static const struct file_operations attr_registers_fops = { .release = single_release, }; -static int dfll_debug_init(struct tegra_dfll *td) +static void dfll_debug_init(struct tegra_dfll *td) { - int ret; + struct dentry *root; if (!td || (td->mode == DFLL_UNINITIALIZED)) - return 0; - - td->debugfs_dir = debugfs_create_dir("tegra_dfll_fcpu", NULL); - if (!td->debugfs_dir) - return -ENOMEM; - - ret = -ENOMEM; - - if (!debugfs_create_file("enable", S_IRUGO | S_IWUSR, - td->debugfs_dir, td, &enable_fops)) - goto err_out; - - if (!debugfs_create_file("lock", S_IRUGO, - td->debugfs_dir, td, &lock_fops)) - goto err_out; + return; - if (!debugfs_create_file("rate", S_IRUGO, - td->debugfs_dir, td, &rate_fops)) - goto err_out; + root = debugfs_create_dir("tegra_dfll_fcpu", NULL); + td->debugfs_dir = root; - if (!debugfs_create_file("registers", S_IRUGO, - td->debugfs_dir, td, &attr_registers_fops)) - goto err_out; - - return 0; - -err_out: - debugfs_remove_recursive(td->debugfs_dir); - return ret; + debugfs_create_file("enable", S_IRUGO | S_IWUSR, root, td, &enable_fops); + debugfs_create_file("lock", S_IRUGO, root, td, &lock_fops); + debugfs_create_file("rate", S_IRUGO, root, td, &rate_fops); + debugfs_create_file("registers", S_IRUGO, root, td, &attr_registers_fops); } +#else +static void inline dfll_debug_init(struct tegra_dfll *td) { } #endif /* CONFIG_DEBUG_FS */ /* @@ -1715,9 +1697,7 @@ int tegra_dfll_register(struct platform_device *pdev, return ret; } -#ifdef CONFIG_DEBUG_FS dfll_debug_init(td); -#endif return 0; } diff --git a/drivers/clk/tegra/clk-tegra114.c b/drivers/clk/tegra/clk-tegra114.c index 5d5a22d529f5..1824f014202b 100644 --- a/drivers/clk/tegra/clk-tegra114.c +++ b/drivers/clk/tegra/clk-tegra114.c @@ -1367,7 +1367,7 @@ static void __init tegra114_clock_init(struct device_node *np) tegra_super_clk_gen4_init(clk_base, pmc_base, tegra114_clks, &pll_x_params); - tegra_add_of_provider(np); + tegra_add_of_provider(np, of_clk_src_onecell_get); tegra_register_devclks(devclks, ARRAY_SIZE(devclks)); tegra_clk_apply_init_table = tegra114_clock_apply_init_table; diff --git a/drivers/clk/tegra/clk-tegra124.c b/drivers/clk/tegra/clk-tegra124.c index 50088e976611..0c69c7970950 100644 --- a/drivers/clk/tegra/clk-tegra124.c +++ b/drivers/clk/tegra/clk-tegra124.c @@ -1479,7 +1479,7 @@ static void __init tegra124_132_clock_init_post(struct device_node *np) &pll_x_params); tegra_init_special_resets(1, tegra124_reset_assert, tegra124_reset_deassert); - tegra_add_of_provider(np); + tegra_add_of_provider(np, of_clk_src_onecell_get); clks[TEGRA124_CLK_EMC] = tegra_clk_register_emc(clk_base, np, &emc_lock); diff --git a/drivers/clk/tegra/clk-tegra20.c b/drivers/clk/tegra/clk-tegra20.c index 0ee56dd04cec..cc857d4d4a86 100644 --- a/drivers/clk/tegra/clk-tegra20.c +++ b/drivers/clk/tegra/clk-tegra20.c @@ -26,6 +26,8 @@ #include "clk.h" #include "clk-id.h" +#define MISC_CLK_ENB 0x48 + #define OSC_CTRL 0x50 #define OSC_CTRL_OSC_FREQ_MASK (3<<30) #define OSC_CTRL_OSC_FREQ_13MHZ (0<<30) @@ -831,15 +833,25 @@ static void __init tegra20_periph_clk_init(void) periph_clk_enb_refcnt); clks[TEGRA20_CLK_PEX] = clk; + /* dev1 OSC divider */ + clk_register_divider(NULL, "dev1_osc_div", "clk_m", + 0, clk_base + MISC_CLK_ENB, 22, 2, + CLK_DIVIDER_POWER_OF_TWO | CLK_DIVIDER_READ_ONLY, + NULL); + + /* dev2 OSC divider */ + clk_register_divider(NULL, "dev2_osc_div", "clk_m", + 0, clk_base + MISC_CLK_ENB, 20, 2, + CLK_DIVIDER_POWER_OF_TWO | CLK_DIVIDER_READ_ONLY, + NULL); + /* cdev1 */ - clk = clk_register_fixed_rate(NULL, "cdev1_fixed", NULL, 0, 26000000); - clk = tegra_clk_register_periph_gate("cdev1", "cdev1_fixed", 0, + clk = tegra_clk_register_periph_gate("cdev1", "cdev1_mux", 0, clk_base, 0, 94, periph_clk_enb_refcnt); clks[TEGRA20_CLK_CDEV1] = clk; /* cdev2 */ - clk = clk_register_fixed_rate(NULL, "cdev2_fixed", NULL, 0, 26000000); - clk = tegra_clk_register_periph_gate("cdev2", "cdev2_fixed", 0, + clk = tegra_clk_register_periph_gate("cdev2", "cdev2_mux", 0, clk_base, 0, 93, periph_clk_enb_refcnt); clks[TEGRA20_CLK_CDEV2] = clk; @@ -1077,6 +1089,36 @@ static const struct of_device_id pmc_match[] __initconst = { { }, }; +static struct clk *tegra20_clk_src_onecell_get(struct of_phandle_args *clkspec, + void *data) +{ + struct clk_hw *parent_hw; + struct clk_hw *hw; + struct clk *clk; + + clk = of_clk_src_onecell_get(clkspec, data); + if (IS_ERR(clk)) + return clk; + + /* + * Tegra20 CDEV1 and CDEV2 clocks are a bit special case, their parent + * clock is created by the pinctrl driver. It is possible for clk user + * to request these clocks before pinctrl driver got probed and hence + * user will get an orphaned clock. That might be undesirable because + * user may expect parent clock to be enabled by the child. + */ + if (clkspec->args[0] == TEGRA20_CLK_CDEV1 || + clkspec->args[0] == TEGRA20_CLK_CDEV2) { + hw = __clk_get_hw(clk); + + parent_hw = clk_hw_get_parent(hw); + if (!parent_hw) + return ERR_PTR(-EPROBE_DEFER); + } + + return clk; +} + static void __init tegra20_clock_init(struct device_node *np) { struct device_node *node; @@ -1115,7 +1157,7 @@ static void __init tegra20_clock_init(struct device_node *np) tegra_init_dup_clks(tegra_clk_duplicates, clks, TEGRA20_CLK_CLK_MAX); - tegra_add_of_provider(np); + tegra_add_of_provider(np, tegra20_clk_src_onecell_get); tegra_register_devclks(devclks, ARRAY_SIZE(devclks)); tegra_clk_apply_init_table = tegra20_clock_apply_init_table; diff --git a/drivers/clk/tegra/clk-tegra210.c b/drivers/clk/tegra/clk-tegra210.c index 9fb5d51ccce4..5435d01c636a 100644 --- a/drivers/clk/tegra/clk-tegra210.c +++ b/drivers/clk/tegra/clk-tegra210.c @@ -3567,7 +3567,7 @@ static void __init tegra210_clock_init(struct device_node *np) tegra_init_special_resets(2, tegra210_reset_assert, tegra210_reset_deassert); - tegra_add_of_provider(np); + tegra_add_of_provider(np, of_clk_src_onecell_get); tegra_register_devclks(devclks, ARRAY_SIZE(devclks)); tegra210_mbist_clk_init(); diff --git a/drivers/clk/tegra/clk-tegra30.c b/drivers/clk/tegra/clk-tegra30.c index b316dfb6f6c7..acfe661b2ae7 100644 --- a/drivers/clk/tegra/clk-tegra30.c +++ b/drivers/clk/tegra/clk-tegra30.c @@ -1349,7 +1349,7 @@ static void __init tegra30_clock_init(struct device_node *np) tegra_init_dup_clks(tegra_clk_duplicates, clks, TEGRA30_CLK_CLK_MAX); - tegra_add_of_provider(np); + tegra_add_of_provider(np, of_clk_src_onecell_get); tegra_register_devclks(devclks, ARRAY_SIZE(devclks)); tegra_clk_apply_init_table = tegra30_clock_apply_init_table; diff --git a/drivers/clk/tegra/clk.c b/drivers/clk/tegra/clk.c index ba923f0d5953..ffaf17f71860 100644 --- a/drivers/clk/tegra/clk.c +++ b/drivers/clk/tegra/clk.c @@ -216,14 +216,15 @@ struct clk ** __init tegra_clk_init(void __iomem *regs, int num, int banks) if (WARN_ON(banks > ARRAY_SIZE(periph_regs))) return NULL; - periph_clk_enb_refcnt = kzalloc(32 * banks * - sizeof(*periph_clk_enb_refcnt), GFP_KERNEL); + periph_clk_enb_refcnt = kcalloc(32 * banks, + sizeof(*periph_clk_enb_refcnt), + GFP_KERNEL); if (!periph_clk_enb_refcnt) return NULL; periph_banks = banks; - clks = kzalloc(num * sizeof(struct clk *), GFP_KERNEL); + clks = kcalloc(num, sizeof(struct clk *), GFP_KERNEL); if (!clks) kfree(periph_clk_enb_refcnt); @@ -298,7 +299,8 @@ static struct reset_controller_dev rst_ctlr = { .of_reset_n_cells = 1, }; -void __init tegra_add_of_provider(struct device_node *np) +void __init tegra_add_of_provider(struct device_node *np, + void *clk_src_onecell_get) { int i; @@ -314,7 +316,7 @@ void __init tegra_add_of_provider(struct device_node *np) clk_data.clks = clks; clk_data.clk_num = clk_num; - of_clk_add_provider(np, of_clk_src_onecell_get, &clk_data); + of_clk_add_provider(np, clk_src_onecell_get, &clk_data); rst_ctlr.of_node = np; rst_ctlr.nr_resets = periph_banks * 32 + num_special_reset; diff --git a/drivers/clk/tegra/clk.h b/drivers/clk/tegra/clk.h index ba7e20e6a82b..e1f88463b600 100644 --- a/drivers/clk/tegra/clk.h +++ b/drivers/clk/tegra/clk.h @@ -763,7 +763,7 @@ struct clk **tegra_clk_init(void __iomem *clk_base, int num, int periph_banks); struct clk **tegra_lookup_dt_id(int clk_id, struct tegra_clk *tegra_clk); -void tegra_add_of_provider(struct device_node *np); +void tegra_add_of_provider(struct device_node *np, void *clk_src_onecell_get); void tegra_register_devclks(struct tegra_devclk *dev_clks, int num); void tegra_audio_clk_init(void __iomem *clk_base, diff --git a/drivers/clk/ti/adpll.c b/drivers/clk/ti/adpll.c index d6036c788fab..688e403333b9 100644 --- a/drivers/clk/ti/adpll.c +++ b/drivers/clk/ti/adpll.c @@ -501,8 +501,9 @@ static int ti_adpll_init_dco(struct ti_adpll_data *d) const char *postfix; int width, err; - d->outputs.clks = devm_kzalloc(d->dev, sizeof(struct clk *) * + d->outputs.clks = devm_kcalloc(d->dev, MAX_ADPLL_OUTPUTS, + sizeof(struct clk *), GFP_KERNEL); if (!d->outputs.clks) return -ENOMEM; @@ -915,8 +916,9 @@ static int ti_adpll_probe(struct platform_device *pdev) if (err) return err; - d->clocks = devm_kzalloc(d->dev, sizeof(struct ti_adpll_clock) * + d->clocks = devm_kcalloc(d->dev, TI_ADPLL_NR_CLOCKS, + sizeof(struct ti_adpll_clock), GFP_KERNEL); if (!d->clocks) return -ENOMEM; diff --git a/drivers/clk/ti/apll.c b/drivers/clk/ti/apll.c index 9498e9363b57..61c126a5d26a 100644 --- a/drivers/clk/ti/apll.c +++ b/drivers/clk/ti/apll.c @@ -206,7 +206,7 @@ static void __init of_dra7_apll_setup(struct device_node *node) goto cleanup; } - parent_names = kzalloc(sizeof(char *) * init->num_parents, GFP_KERNEL); + parent_names = kcalloc(init->num_parents, sizeof(char *), GFP_KERNEL); if (!parent_names) goto cleanup; diff --git a/drivers/clk/ti/divider.c b/drivers/clk/ti/divider.c index aaa277dd6d99..ccfb4d9a152a 100644 --- a/drivers/clk/ti/divider.c +++ b/drivers/clk/ti/divider.c @@ -366,7 +366,7 @@ int ti_clk_parse_divider_data(int *div_table, int num_dividers, int max_div, num_dividers = i; - tmp = kzalloc(sizeof(*tmp) * (valid_div + 1), GFP_KERNEL); + tmp = kcalloc(valid_div + 1, sizeof(*tmp), GFP_KERNEL); if (!tmp) return -ENOMEM; @@ -496,7 +496,7 @@ __init ti_clk_get_div_table(struct device_node *node) return ERR_PTR(-EINVAL); } - table = kzalloc(sizeof(*table) * (valid_div + 1), GFP_KERNEL); + table = kcalloc(valid_div + 1, sizeof(*table), GFP_KERNEL); if (!table) return ERR_PTR(-ENOMEM); diff --git a/drivers/clk/ti/dpll.c b/drivers/clk/ti/dpll.c index 7d33ca9042cb..dc86d07d0921 100644 --- a/drivers/clk/ti/dpll.c +++ b/drivers/clk/ti/dpll.c @@ -309,7 +309,7 @@ static void __init of_ti_dpll_setup(struct device_node *node, goto cleanup; } - parent_names = kzalloc(sizeof(char *) * init->num_parents, GFP_KERNEL); + parent_names = kcalloc(init->num_parents, sizeof(char *), GFP_KERNEL); if (!parent_names) goto cleanup; diff --git a/drivers/clk/uniphier/clk-uniphier-sys.c b/drivers/clk/uniphier/clk-uniphier-sys.c index ebc78ab2df05..4f5ff9fa11fd 100644 --- a/drivers/clk/uniphier/clk-uniphier-sys.c +++ b/drivers/clk/uniphier/clk-uniphier-sys.c @@ -51,6 +51,9 @@ #define UNIPHIER_LD11_SYS_CLK_STDMAC(idx) \ UNIPHIER_CLK_GATE("stdmac", (idx), NULL, 0x210c, 8) +#define UNIPHIER_LD11_SYS_CLK_HSC(idx) \ + UNIPHIER_CLK_GATE("hsc", (idx), NULL, 0x210c, 9) + #define UNIPHIER_PRO4_SYS_CLK_GIO(idx) \ UNIPHIER_CLK_GATE("gio", (idx), NULL, 0x2104, 6) @@ -182,6 +185,7 @@ const struct uniphier_clk_data uniphier_ld11_sys_clk_data[] = { /* Index 5 reserved for eMMC PHY */ UNIPHIER_LD11_SYS_CLK_ETHER(6), UNIPHIER_LD11_SYS_CLK_STDMAC(8), /* HSC, MIO */ + UNIPHIER_LD11_SYS_CLK_HSC(9), UNIPHIER_CLK_FACTOR("usb2", -1, "ref", 24, 25), UNIPHIER_LD11_SYS_CLK_AIO(40), UNIPHIER_LD11_SYS_CLK_EVEA(41), @@ -215,6 +219,7 @@ const struct uniphier_clk_data uniphier_ld20_sys_clk_data[] = { UNIPHIER_LD20_SYS_CLK_SD, UNIPHIER_LD11_SYS_CLK_ETHER(6), UNIPHIER_LD11_SYS_CLK_STDMAC(8), /* HSC */ + UNIPHIER_LD11_SYS_CLK_HSC(9), /* GIO is always clock-enabled: no function for 0x210c bit5 */ /* * clock for USB Link is enabled by the logic "OR" of bit 14 and bit 15. diff --git a/drivers/clk/x86/Makefile b/drivers/clk/x86/Makefile index 1367afb03858..00303bc05415 100644 --- a/drivers/clk/x86/Makefile +++ b/drivers/clk/x86/Makefile @@ -1,3 +1,4 @@ +obj-$(CONFIG_PMC_ATOM) += clk-pmc-atom.o +obj-$(CONFIG_X86_AMD_PLATFORM_DEVICE) += clk-st.o clk-x86-lpss-objs := clk-lpt.o obj-$(CONFIG_X86_INTEL_LPSS) += clk-x86-lpss.o -obj-$(CONFIG_PMC_ATOM) += clk-pmc-atom.o diff --git a/drivers/clk/x86/clk-st.c b/drivers/clk/x86/clk-st.c new file mode 100644 index 000000000000..fb62f3938008 --- /dev/null +++ b/drivers/clk/x86/clk-st.c @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: MIT +/* + * clock framework for AMD Stoney based clocks + * + * Copyright 2018 Advanced Micro Devices, Inc. + */ + +#include +#include +#include +#include +#include + +/* Clock Driving Strength 2 register */ +#define CLKDRVSTR2 0x28 +/* Clock Control 1 register */ +#define MISCCLKCNTL1 0x40 +/* Auxiliary clock1 enable bit */ +#define OSCCLKENB 2 +/* 25Mhz auxiliary output clock freq bit */ +#define OSCOUT1CLK25MHZ 16 + +#define ST_CLK_48M 0 +#define ST_CLK_25M 1 +#define ST_CLK_MUX 2 +#define ST_CLK_GATE 3 +#define ST_MAX_CLKS 4 + +static const char * const clk_oscout1_parents[] = { "clk48MHz", "clk25MHz" }; +static struct clk_hw *hws[ST_MAX_CLKS]; + +static int st_clk_probe(struct platform_device *pdev) +{ + struct st_clk_data *st_data; + + st_data = dev_get_platdata(&pdev->dev); + if (!st_data || !st_data->base) + return -EINVAL; + + hws[ST_CLK_48M] = clk_hw_register_fixed_rate(NULL, "clk48MHz", NULL, 0, + 48000000); + hws[ST_CLK_25M] = clk_hw_register_fixed_rate(NULL, "clk25MHz", NULL, 0, + 25000000); + + hws[ST_CLK_MUX] = clk_hw_register_mux(NULL, "oscout1_mux", + clk_oscout1_parents, ARRAY_SIZE(clk_oscout1_parents), + 0, st_data->base + CLKDRVSTR2, OSCOUT1CLK25MHZ, 3, 0, NULL); + + clk_set_parent(hws[ST_CLK_MUX]->clk, hws[ST_CLK_25M]->clk); + + hws[ST_CLK_GATE] = clk_hw_register_gate(NULL, "oscout1", "oscout1_mux", + 0, st_data->base + MISCCLKCNTL1, OSCCLKENB, + CLK_GATE_SET_TO_DISABLE, NULL); + + clk_hw_register_clkdev(hws[ST_CLK_GATE], "oscout1", NULL); + + return 0; +} + +static int st_clk_remove(struct platform_device *pdev) +{ + int i; + + for (i = 0; i < ST_MAX_CLKS; i++) + clk_hw_unregister(hws[i]); + return 0; +} + +static struct platform_driver st_clk_driver = { + .driver = { + .name = "clk-st", + .suppress_bind_attrs = true, + }, + .probe = st_clk_probe, + .remove = st_clk_remove, +}; +builtin_platform_driver(st_clk_driver); diff --git a/drivers/clocksource/Kconfig b/drivers/clocksource/Kconfig index 8e8a09755d10..dec0dd88ec15 100644 --- a/drivers/clocksource/Kconfig +++ b/drivers/clocksource/Kconfig @@ -450,8 +450,10 @@ config MTK_TIMER Support for Mediatek timer driver. config SPRD_TIMER - bool "Spreadtrum timer driver" if COMPILE_TEST + bool "Spreadtrum timer driver" if EXPERT depends on HAS_IOMEM + depends on (ARCH_SPRD || COMPILE_TEST) + default ARCH_SPRD select TIMER_OF help Enables support for the Spreadtrum timer driver. diff --git a/drivers/clocksource/arc_timer.c b/drivers/clocksource/arc_timer.c index 471b428d8034..20da9b1d7f7d 100644 --- a/drivers/clocksource/arc_timer.c +++ b/drivers/clocksource/arc_timer.c @@ -61,6 +61,20 @@ static u64 arc_read_gfrc(struct clocksource *cs) unsigned long flags; u32 l, h; + /* + * From a programming model pov, there seems to be just one instance of + * MCIP_CMD/MCIP_READBACK however micro-architecturally there's + * an instance PER ARC CORE (not per cluster), and there are dedicated + * hardware decode logic (per core) inside ARConnect to handle + * simultaneous read/write accesses from cores via those two registers. + * So several concurrent commands to ARConnect are OK if they are + * trying to access two different sub-components (like GFRC, + * inter-core interrupt, etc...). HW also supports simultaneously + * accessing GFRC by multiple cores. + * That's why it is safe to disable hard interrupts on the local CPU + * before access to GFRC instead of taking global MCIP spinlock + * defined in arch/arc/kernel/mcip.c + */ local_irq_save(flags); __mcip_cmd(CMD_GFRC_READ_LO, 0); diff --git a/drivers/clocksource/mips-gic-timer.c b/drivers/clocksource/mips-gic-timer.c index 986b6796b631..54f8a331b53a 100644 --- a/drivers/clocksource/mips-gic-timer.c +++ b/drivers/clocksource/mips-gic-timer.c @@ -5,6 +5,9 @@ * * Copyright (C) 2012 MIPS Technologies, Inc. All rights reserved. */ + +#define pr_fmt(fmt) "mips-gic-timer: " fmt + #include #include #include @@ -136,8 +139,7 @@ static int gic_clockevent_init(void) ret = setup_percpu_irq(gic_timer_irq, &gic_compare_irqaction); if (ret < 0) { - pr_err("GIC timer IRQ %d setup failed: %d\n", - gic_timer_irq, ret); + pr_err("IRQ %d setup failed (%d)\n", gic_timer_irq, ret); return ret; } @@ -176,7 +178,7 @@ static int __init __gic_clocksource_init(void) ret = clocksource_register_hz(&gic_clocksource, gic_frequency); if (ret < 0) - pr_warn("GIC: Unable to register clocksource\n"); + pr_warn("Unable to register clocksource\n"); return ret; } @@ -188,7 +190,7 @@ static int __init gic_clocksource_of_init(struct device_node *node) if (!mips_gic_present() || !node->parent || !of_device_is_compatible(node->parent, "mti,gic")) { - pr_warn("No DT definition for the mips gic driver\n"); + pr_warn("No DT definition\n"); return -ENXIO; } @@ -196,7 +198,7 @@ static int __init gic_clocksource_of_init(struct device_node *node) if (!IS_ERR(clk)) { ret = clk_prepare_enable(clk); if (ret < 0) { - pr_err("GIC failed to enable clock\n"); + pr_err("Failed to enable clock\n"); clk_put(clk); return ret; } @@ -204,12 +206,12 @@ static int __init gic_clocksource_of_init(struct device_node *node) gic_frequency = clk_get_rate(clk); } else if (of_property_read_u32(node, "clock-frequency", &gic_frequency)) { - pr_err("GIC frequency not specified.\n"); + pr_err("Frequency not specified\n"); return -EINVAL; } gic_timer_irq = irq_of_parse_and_map(node, 0); if (!gic_timer_irq) { - pr_err("GIC timer IRQ not specified.\n"); + pr_err("IRQ not specified\n"); return -EINVAL; } @@ -220,7 +222,7 @@ static int __init gic_clocksource_of_init(struct device_node *node) ret = gic_clockevent_init(); if (!ret && !IS_ERR(clk)) { if (clk_notifier_register(clk, &gic_clk_nb) < 0) - pr_warn("GIC: Unable to register clock notifier\n"); + pr_warn("Unable to register clock notifier\n"); } /* And finally start the counter */ diff --git a/drivers/clocksource/mxs_timer.c b/drivers/clocksource/mxs_timer.c index a03434e9fe8f..f6ddae30933f 100644 --- a/drivers/clocksource/mxs_timer.c +++ b/drivers/clocksource/mxs_timer.c @@ -1,24 +1,10 @@ -/* - * Copyright (C) 2000-2001 Deep Blue Solutions - * Copyright (C) 2002 Shane Nay (shane@minirl.com) - * Copyright (C) 2006-2007 Pavel Pisa (ppisa@pikron.com) - * Copyright (C) 2008 Juergen Beisert (kernel@pengutronix.de) - * Copyright (C) 2010 Freescale Semiconductor, Inc. All Rights Reserved. - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version 2 - * of the License, or (at your option) any later version. - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - * MA 02110-1301, USA. - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright (C) 2000-2001 Deep Blue Solutions +// Copyright (C) 2002 Shane Nay (shane@minirl.com) +// Copyright (C) 2006-2007 Pavel Pisa (ppisa@pikron.com) +// Copyright (C) 2008 Juergen Beisert (kernel@pengutronix.de) +// Copyright (C) 2010 Freescale Semiconductor, Inc. All Rights Reserved. #include #include diff --git a/drivers/clocksource/sh_cmt.c b/drivers/clocksource/sh_cmt.c index 70b3cf8e23d0..bbbf37c471a3 100644 --- a/drivers/clocksource/sh_cmt.c +++ b/drivers/clocksource/sh_cmt.c @@ -1000,7 +1000,7 @@ static int sh_cmt_setup(struct sh_cmt_device *cmt, struct platform_device *pdev) /* Allocate and setup the channels. */ cmt->num_channels = hweight8(cmt->hw_channels); - cmt->channels = kzalloc(cmt->num_channels * sizeof(*cmt->channels), + cmt->channels = kcalloc(cmt->num_channels, sizeof(*cmt->channels), GFP_KERNEL); if (cmt->channels == NULL) { ret = -ENOMEM; diff --git a/drivers/clocksource/sh_mtu2.c b/drivers/clocksource/sh_mtu2.c index 53aa7e92a7d7..6812e099b6a3 100644 --- a/drivers/clocksource/sh_mtu2.c +++ b/drivers/clocksource/sh_mtu2.c @@ -418,7 +418,7 @@ static int sh_mtu2_setup(struct sh_mtu2_device *mtu, /* Allocate and setup the channels. */ mtu->num_channels = 3; - mtu->channels = kzalloc(sizeof(*mtu->channels) * mtu->num_channels, + mtu->channels = kcalloc(mtu->num_channels, sizeof(*mtu->channels), GFP_KERNEL); if (mtu->channels == NULL) { ret = -ENOMEM; diff --git a/drivers/clocksource/sh_tmu.c b/drivers/clocksource/sh_tmu.c index 31d881621e41..c74a6c543ca2 100644 --- a/drivers/clocksource/sh_tmu.c +++ b/drivers/clocksource/sh_tmu.c @@ -569,7 +569,7 @@ static int sh_tmu_setup(struct sh_tmu_device *tmu, struct platform_device *pdev) } /* Allocate and setup the channels. */ - tmu->channels = kzalloc(sizeof(*tmu->channels) * tmu->num_channels, + tmu->channels = kcalloc(tmu->num_channels, sizeof(*tmu->channels), GFP_KERNEL); if (tmu->channels == NULL) { ret = -ENOMEM; diff --git a/drivers/clocksource/timer-imx-gpt.c b/drivers/clocksource/timer-imx-gpt.c index 6ec6d79b237c..165fbbb1c9a0 100644 --- a/drivers/clocksource/timer-imx-gpt.c +++ b/drivers/clocksource/timer-imx-gpt.c @@ -1,25 +1,9 @@ -/* - * linux/arch/arm/plat-mxc/time.c - * - * Copyright (C) 2000-2001 Deep Blue Solutions - * Copyright (C) 2002 Shane Nay (shane@minirl.com) - * Copyright (C) 2006-2007 Pavel Pisa (ppisa@pikron.com) - * Copyright (C) 2008 Juergen Beisert (kernel@pengutronix.de) - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version 2 - * of the License, or (at your option) any later version. - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - * MA 02110-1301, USA. - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright (C) 2000-2001 Deep Blue Solutions +// Copyright (C) 2002 Shane Nay (shane@minirl.com) +// Copyright (C) 2006-2007 Pavel Pisa (ppisa@pikron.com) +// Copyright (C) 2008 Juergen Beisert (kernel@pengutronix.de) #include #include diff --git a/drivers/clocksource/timer-imx-tpm.c b/drivers/clocksource/timer-imx-tpm.c index 6c8318470b48..b7aa2b817078 100644 --- a/drivers/clocksource/timer-imx-tpm.c +++ b/drivers/clocksource/timer-imx-tpm.c @@ -1,12 +1,7 @@ -/* - * Copyright 2016 Freescale Semiconductor, Inc. - * Copyright 2017 NXP - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version 2 - * of the License, or (at your option) any later version. - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// Copyright 2016 Freescale Semiconductor, Inc. +// Copyright 2017 NXP #include #include diff --git a/drivers/clocksource/timer-sp804.c b/drivers/clocksource/timer-sp804.c index 3ac9dec9a038..e01222ea888f 100644 --- a/drivers/clocksource/timer-sp804.c +++ b/drivers/clocksource/timer-sp804.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include @@ -245,7 +246,7 @@ static int __init sp804_of_init(struct device_node *np) clk1 = NULL; /* Get the 2nd clock if the timer has 3 timer clocks */ - if (of_count_phandle_with_args(np, "clocks", "#clock-cells") == 3) { + if (of_clk_get_parent_count(np) == 3) { clk2 = of_clk_get(np, 1); if (IS_ERR(clk2)) { pr_err("sp804: %s clock not found: %d\n", np->name, diff --git a/drivers/connector/cn_proc.c b/drivers/connector/cn_proc.c index a782ce87715c..ed5e42461094 100644 --- a/drivers/connector/cn_proc.c +++ b/drivers/connector/cn_proc.c @@ -262,6 +262,8 @@ void proc_coredump_connector(struct task_struct *task) ev->what = PROC_EVENT_COREDUMP; ev->event_data.coredump.process_pid = task->pid; ev->event_data.coredump.process_tgid = task->tgid; + ev->event_data.coredump.parent_pid = task->real_parent->pid; + ev->event_data.coredump.parent_tgid = task->real_parent->tgid; memcpy(&msg->id, &cn_proc_event_id, sizeof(msg->id)); msg->ack = 0; /* not used */ @@ -288,6 +290,8 @@ void proc_exit_connector(struct task_struct *task) ev->event_data.exit.process_tgid = task->tgid; ev->event_data.exit.exit_code = task->exit_code; ev->event_data.exit.exit_signal = task->exit_signal; + ev->event_data.exit.parent_pid = task->real_parent->pid; + ev->event_data.exit.parent_tgid = task->real_parent->tgid; memcpy(&msg->id, &cn_proc_event_id, sizeof(msg->id)); msg->ack = 0; /* not used */ diff --git a/drivers/connector/connector.c b/drivers/connector/connector.c index 8615594bd065..e718b8c69a56 100644 --- a/drivers/connector/connector.c +++ b/drivers/connector/connector.c @@ -260,19 +260,6 @@ static int cn_proc_show(struct seq_file *m, void *v) return 0; } -static int cn_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, cn_proc_show, NULL); -} - -static const struct file_operations cn_file_ops = { - .owner = THIS_MODULE, - .open = cn_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release -}; - static struct cn_dev cdev = { .input = cn_rx_skb, }; @@ -297,7 +284,7 @@ static int cn_init(void) cn_already_initialized = 1; - proc_create("connector", S_IRUGO, init_net.proc_net, &cn_file_ops); + proc_create_single("connector", S_IRUGO, init_net.proc_net, cn_proc_show); return 0; } diff --git a/drivers/cpufreq/Kconfig.arm b/drivers/cpufreq/Kconfig.arm index 96b35b8b3606..52f5f1a2040c 100644 --- a/drivers/cpufreq/Kconfig.arm +++ b/drivers/cpufreq/Kconfig.arm @@ -124,6 +124,17 @@ config ARM_OMAP2PLUS_CPUFREQ depends on ARCH_OMAP2PLUS default ARCH_OMAP2PLUS +config ARM_QCOM_CPUFREQ_KRYO + tristate "Qualcomm Kryo based CPUFreq" + depends on ARM64 + depends on QCOM_QFPROM + depends on QCOM_SMEM + select PM_OPP + help + This adds the CPUFreq driver for Qualcomm Kryo SoC based boards. + + If in doubt, say N. + config ARM_S3C_CPUFREQ bool help @@ -264,7 +275,7 @@ config ARM_TANGO_CPUFREQ default y config ARM_TEGRA20_CPUFREQ - bool "Tegra20 CPUFreq support" + tristate "Tegra20 CPUFreq support" depends on ARCH_TEGRA default y help diff --git a/drivers/cpufreq/Makefile b/drivers/cpufreq/Makefile index 8d24ade3bd02..fb4a2ecac43b 100644 --- a/drivers/cpufreq/Makefile +++ b/drivers/cpufreq/Makefile @@ -65,6 +65,7 @@ obj-$(CONFIG_MACH_MVEBU_V7) += mvebu-cpufreq.o obj-$(CONFIG_ARM_OMAP2PLUS_CPUFREQ) += omap-cpufreq.o obj-$(CONFIG_ARM_PXA2xx_CPUFREQ) += pxa2xx-cpufreq.o obj-$(CONFIG_PXA3xx) += pxa3xx-cpufreq.o +obj-$(CONFIG_ARM_QCOM_CPUFREQ_KRYO) += qcom-cpufreq-kryo.o obj-$(CONFIG_ARM_S3C2410_CPUFREQ) += s3c2410-cpufreq.o obj-$(CONFIG_ARM_S3C2412_CPUFREQ) += s3c2412-cpufreq.o obj-$(CONFIG_ARM_S3C2416_CPUFREQ) += s3c2416-cpufreq.o diff --git a/drivers/cpufreq/acpi-cpufreq.c b/drivers/cpufreq/acpi-cpufreq.c index 9449657d72f0..b61f4ec43e06 100644 --- a/drivers/cpufreq/acpi-cpufreq.c +++ b/drivers/cpufreq/acpi-cpufreq.c @@ -465,8 +465,8 @@ static int acpi_cpufreq_target(struct cpufreq_policy *policy, return result; } -unsigned int acpi_cpufreq_fast_switch(struct cpufreq_policy *policy, - unsigned int target_freq) +static unsigned int acpi_cpufreq_fast_switch(struct cpufreq_policy *policy, + unsigned int target_freq) { struct acpi_cpufreq_data *data = policy->driver_data; struct acpi_processor_performance *perf; @@ -759,8 +759,8 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy) goto err_unreg; } - freq_table = kzalloc(sizeof(*freq_table) * - (perf->state_count+1), GFP_KERNEL); + freq_table = kcalloc(perf->state_count + 1, sizeof(*freq_table), + GFP_KERNEL); if (!freq_table) { result = -ENOMEM; goto err_unreg; diff --git a/drivers/cpufreq/arm_big_little.c b/drivers/cpufreq/arm_big_little.c index 1d7ef5fc1977..cf62a1f64dd7 100644 --- a/drivers/cpufreq/arm_big_little.c +++ b/drivers/cpufreq/arm_big_little.c @@ -280,7 +280,7 @@ static int merge_cluster_tables(void) for (i = 0; i < MAX_CLUSTERS; i++) count += get_table_count(freq_table[i]); - table = kzalloc(sizeof(*table) * count, GFP_KERNEL); + table = kcalloc(count, sizeof(*table), GFP_KERNEL); if (!table) return -ENOMEM; diff --git a/drivers/cpufreq/armada-37xx-cpufreq.c b/drivers/cpufreq/armada-37xx-cpufreq.c index 72a2975499db..739da90ff3f6 100644 --- a/drivers/cpufreq/armada-37xx-cpufreq.c +++ b/drivers/cpufreq/armada-37xx-cpufreq.c @@ -23,6 +23,8 @@ #include #include +#include "cpufreq-dt.h" + /* Power management in North Bridge register set */ #define ARMADA_37XX_NB_L0L1 0x18 #define ARMADA_37XX_NB_L2L3 0x1C @@ -56,6 +58,16 @@ */ #define LOAD_LEVEL_NR 4 +struct armada37xx_cpufreq_state { + struct regmap *regmap; + u32 nb_l0l1; + u32 nb_l2l3; + u32 nb_dyn_mod; + u32 nb_cpu_load; +}; + +static struct armada37xx_cpufreq_state *armada37xx_cpufreq_state; + struct armada_37xx_dvfs { u32 cpu_freq_max; u8 divider[LOAD_LEVEL_NR]; @@ -136,7 +148,7 @@ static void __init armada37xx_cpufreq_dvfs_setup(struct regmap *base, clk_set_parent(clk, parent); } -static void __init armada37xx_cpufreq_disable_dvfs(struct regmap *base) +static void armada37xx_cpufreq_disable_dvfs(struct regmap *base) { unsigned int reg = ARMADA_37XX_NB_DYN_MOD, mask = ARMADA_37XX_NB_DFS_EN; @@ -162,10 +174,47 @@ static void __init armada37xx_cpufreq_enable_dvfs(struct regmap *base) regmap_update_bits(base, reg, mask, mask); } +static int armada37xx_cpufreq_suspend(struct cpufreq_policy *policy) +{ + struct armada37xx_cpufreq_state *state = armada37xx_cpufreq_state; + + regmap_read(state->regmap, ARMADA_37XX_NB_L0L1, &state->nb_l0l1); + regmap_read(state->regmap, ARMADA_37XX_NB_L2L3, &state->nb_l2l3); + regmap_read(state->regmap, ARMADA_37XX_NB_CPU_LOAD, + &state->nb_cpu_load); + regmap_read(state->regmap, ARMADA_37XX_NB_DYN_MOD, &state->nb_dyn_mod); + + return 0; +} + +static int armada37xx_cpufreq_resume(struct cpufreq_policy *policy) +{ + struct armada37xx_cpufreq_state *state = armada37xx_cpufreq_state; + + /* Ensure DVFS is disabled otherwise the following registers are RO */ + armada37xx_cpufreq_disable_dvfs(state->regmap); + + regmap_write(state->regmap, ARMADA_37XX_NB_L0L1, state->nb_l0l1); + regmap_write(state->regmap, ARMADA_37XX_NB_L2L3, state->nb_l2l3); + regmap_write(state->regmap, ARMADA_37XX_NB_CPU_LOAD, + state->nb_cpu_load); + + /* + * NB_DYN_MOD register is the one that actually enable back DVFS if it + * was enabled before the suspend operation. This must be done last + * otherwise other registers are not writable. + */ + regmap_write(state->regmap, ARMADA_37XX_NB_DYN_MOD, state->nb_dyn_mod); + + return 0; +} + static int __init armada37xx_cpufreq_driver_init(void) { + struct cpufreq_dt_platform_data pdata; struct armada_37xx_dvfs *dvfs; struct platform_device *pdev; + unsigned long freq; unsigned int cur_frequency; struct regmap *nb_pm_base; struct device *cpu_dev; @@ -207,33 +256,58 @@ static int __init armada37xx_cpufreq_driver_init(void) } dvfs = armada_37xx_cpu_freq_info_get(cur_frequency); - if (!dvfs) + if (!dvfs) { + clk_put(clk); return -EINVAL; + } + + armada37xx_cpufreq_state = kmalloc(sizeof(*armada37xx_cpufreq_state), + GFP_KERNEL); + if (!armada37xx_cpufreq_state) { + clk_put(clk); + return -ENOMEM; + } + + armada37xx_cpufreq_state->regmap = nb_pm_base; armada37xx_cpufreq_dvfs_setup(nb_pm_base, clk, dvfs->divider); clk_put(clk); for (load_lvl = ARMADA_37XX_DVFS_LOAD_0; load_lvl < LOAD_LEVEL_NR; load_lvl++) { - unsigned long freq = cur_frequency / dvfs->divider[load_lvl]; + freq = cur_frequency / dvfs->divider[load_lvl]; ret = dev_pm_opp_add(cpu_dev, freq, 0); - if (ret) { - /* clean-up the already added opp before leaving */ - while (load_lvl-- > ARMADA_37XX_DVFS_LOAD_0) { - freq = cur_frequency / dvfs->divider[load_lvl]; - dev_pm_opp_remove(cpu_dev, freq); - } - return ret; - } + if (ret) + goto remove_opp; } /* Now that everything is setup, enable the DVFS at hardware level */ armada37xx_cpufreq_enable_dvfs(nb_pm_base); - pdev = platform_device_register_simple("cpufreq-dt", -1, NULL, 0); + pdata.suspend = armada37xx_cpufreq_suspend; + pdata.resume = armada37xx_cpufreq_resume; + + pdev = platform_device_register_data(NULL, "cpufreq-dt", -1, &pdata, + sizeof(pdata)); + ret = PTR_ERR_OR_ZERO(pdev); + if (ret) + goto disable_dvfs; + + return 0; + +disable_dvfs: + armada37xx_cpufreq_disable_dvfs(nb_pm_base); +remove_opp: + /* clean-up the already added opp before leaving */ + while (load_lvl-- > ARMADA_37XX_DVFS_LOAD_0) { + freq = cur_frequency / dvfs->divider[load_lvl]; + dev_pm_opp_remove(cpu_dev, freq); + } + + kfree(armada37xx_cpufreq_state); - return PTR_ERR_OR_ZERO(pdev); + return ret; } /* late_initcall, to guarantee the driver is loaded after A37xx clock driver */ late_initcall(armada37xx_cpufreq_driver_init); diff --git a/drivers/cpufreq/bmips-cpufreq.c b/drivers/cpufreq/bmips-cpufreq.c index 1653151b77df..56a4ebbf00e0 100644 --- a/drivers/cpufreq/bmips-cpufreq.c +++ b/drivers/cpufreq/bmips-cpufreq.c @@ -71,7 +71,7 @@ bmips_cpufreq_get_freq_table(const struct cpufreq_policy *policy) cpu_freq = htp_freq_to_cpu_freq(priv->clk_mult); - table = kmalloc((priv->max_freqs + 1) * sizeof(*table), GFP_KERNEL); + table = kmalloc_array(priv->max_freqs + 1, sizeof(*table), GFP_KERNEL); if (!table) return ERR_PTR(-ENOMEM); diff --git a/drivers/cpufreq/brcmstb-avs-cpufreq.c b/drivers/cpufreq/brcmstb-avs-cpufreq.c index b07559b9ed99..e6f9cbe5835f 100644 --- a/drivers/cpufreq/brcmstb-avs-cpufreq.c +++ b/drivers/cpufreq/brcmstb-avs-cpufreq.c @@ -410,7 +410,7 @@ brcm_avs_get_freq_table(struct device *dev, struct private_data *priv) if (ret) return ERR_PTR(ret); - table = devm_kzalloc(dev, (AVS_PSTATE_MAX + 1) * sizeof(*table), + table = devm_kcalloc(dev, AVS_PSTATE_MAX + 1, sizeof(*table), GFP_KERNEL); if (!table) return ERR_PTR(-ENOMEM); diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index b15115a48775..a9d3eec32795 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -42,9 +42,6 @@ */ static struct cppc_cpudata **all_cpu_data; -/* Capture the max KHz from DMI */ -static u64 cppc_dmi_max_khz; - /* Callback function used to retrieve the max frequency from DMI */ static void cppc_find_dmi_mhz(const struct dmi_header *dm, void *private) { @@ -75,6 +72,64 @@ static u64 cppc_get_dmi_max_khz(void) return (1000 * mhz); } +/* + * If CPPC lowest_freq and nominal_freq registers are exposed then we can + * use them to convert perf to freq and vice versa + * + * If the perf/freq point lies between Nominal and Lowest, we can treat + * (Low perf, Low freq) and (Nom Perf, Nom freq) as 2D co-ordinates of a line + * and extrapolate the rest + * For perf/freq > Nominal, we use the ratio perf:freq at Nominal for conversion + */ +static unsigned int cppc_cpufreq_perf_to_khz(struct cppc_cpudata *cpu, + unsigned int perf) +{ + static u64 max_khz; + struct cppc_perf_caps *caps = &cpu->perf_caps; + u64 mul, div; + + if (caps->lowest_freq && caps->nominal_freq) { + if (perf >= caps->nominal_perf) { + mul = caps->nominal_freq; + div = caps->nominal_perf; + } else { + mul = caps->nominal_freq - caps->lowest_freq; + div = caps->nominal_perf - caps->lowest_perf; + } + } else { + if (!max_khz) + max_khz = cppc_get_dmi_max_khz(); + mul = max_khz; + div = cpu->perf_caps.highest_perf; + } + return (u64)perf * mul / div; +} + +static unsigned int cppc_cpufreq_khz_to_perf(struct cppc_cpudata *cpu, + unsigned int freq) +{ + static u64 max_khz; + struct cppc_perf_caps *caps = &cpu->perf_caps; + u64 mul, div; + + if (caps->lowest_freq && caps->nominal_freq) { + if (freq >= caps->nominal_freq) { + mul = caps->nominal_perf; + div = caps->nominal_freq; + } else { + mul = caps->lowest_perf; + div = caps->lowest_freq; + } + } else { + if (!max_khz) + max_khz = cppc_get_dmi_max_khz(); + mul = cpu->perf_caps.highest_perf; + div = max_khz; + } + + return (u64)freq * mul / div; +} + static int cppc_cpufreq_set_target(struct cpufreq_policy *policy, unsigned int target_freq, unsigned int relation) @@ -86,7 +141,7 @@ static int cppc_cpufreq_set_target(struct cpufreq_policy *policy, cpu = all_cpu_data[policy->cpu]; - desired_perf = (u64)target_freq * cpu->perf_caps.highest_perf / cppc_dmi_max_khz; + desired_perf = cppc_cpufreq_khz_to_perf(cpu, target_freq); /* Return if it is exactly the same perf */ if (desired_perf == cpu->perf_ctrls.desired_perf) return ret; @@ -186,24 +241,24 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) return ret; } - cppc_dmi_max_khz = cppc_get_dmi_max_khz(); + /* Convert the lowest and nominal freq from MHz to KHz */ + cpu->perf_caps.lowest_freq *= 1000; + cpu->perf_caps.nominal_freq *= 1000; /* * Set min to lowest nonlinear perf to avoid any efficiency penalty (see * Section 8.4.7.1.1.5 of ACPI 6.1 spec) */ - policy->min = cpu->perf_caps.lowest_nonlinear_perf * cppc_dmi_max_khz / - cpu->perf_caps.highest_perf; - policy->max = cppc_dmi_max_khz; + policy->min = cppc_cpufreq_perf_to_khz(cpu, cpu->perf_caps.lowest_nonlinear_perf); + policy->max = cppc_cpufreq_perf_to_khz(cpu, cpu->perf_caps.highest_perf); /* * Set cpuinfo.min_freq to Lowest to make the full range of performance * available if userspace wants to use any perf between lowest & lowest * nonlinear perf */ - policy->cpuinfo.min_freq = cpu->perf_caps.lowest_perf * cppc_dmi_max_khz / - cpu->perf_caps.highest_perf; - policy->cpuinfo.max_freq = cppc_dmi_max_khz; + policy->cpuinfo.min_freq = cppc_cpufreq_perf_to_khz(cpu, cpu->perf_caps.lowest_perf); + policy->cpuinfo.max_freq = cppc_cpufreq_perf_to_khz(cpu, cpu->perf_caps.highest_perf); policy->transition_delay_us = cppc_cpufreq_get_transition_delay_us(cpu_num); policy->shared_type = cpu->shared_type; @@ -229,7 +284,8 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) cpu->cur_policy = policy; /* Set policy->cur to max now. The governors will adjust later. */ - policy->cur = cppc_dmi_max_khz; + policy->cur = cppc_cpufreq_perf_to_khz(cpu, + cpu->perf_caps.highest_perf); cpu->perf_ctrls.desired_perf = cpu->perf_caps.highest_perf; ret = cppc_set_perf(cpu_num, &cpu->perf_ctrls); @@ -257,7 +313,8 @@ static int __init cppc_cpufreq_init(void) if (acpi_disabled) return -ENODEV; - all_cpu_data = kzalloc(sizeof(void *) * num_possible_cpus(), GFP_KERNEL); + all_cpu_data = kcalloc(num_possible_cpus(), sizeof(void *), + GFP_KERNEL); if (!all_cpu_data) return -ENOMEM; diff --git a/drivers/cpufreq/cpufreq-dt-platdev.c b/drivers/cpufreq/cpufreq-dt-platdev.c index 3b585e4bfac5..fe14c57de6ca 100644 --- a/drivers/cpufreq/cpufreq-dt-platdev.c +++ b/drivers/cpufreq/cpufreq-dt-platdev.c @@ -66,8 +66,6 @@ static const struct of_device_id whitelist[] __initconst = { { .compatible = "renesas,r8a7792", }, { .compatible = "renesas,r8a7793", }, { .compatible = "renesas,r8a7794", }, - { .compatible = "renesas,r8a7795", }, - { .compatible = "renesas,r8a7796", }, { .compatible = "renesas,sh73a0", }, { .compatible = "rockchip,rk2928", }, @@ -118,6 +116,9 @@ static const struct of_device_id blacklist[] __initconst = { { .compatible = "nvidia,tegra124", }, + { .compatible = "qcom,apq8096", }, + { .compatible = "qcom,msm8996", }, + { .compatible = "st,stih407", }, { .compatible = "st,stih410", }, diff --git a/drivers/cpufreq/cpufreq-dt.c b/drivers/cpufreq/cpufreq-dt.c index 190ea0dccb79..0a9ebf00be46 100644 --- a/drivers/cpufreq/cpufreq-dt.c +++ b/drivers/cpufreq/cpufreq-dt.c @@ -346,8 +346,14 @@ static int dt_cpufreq_probe(struct platform_device *pdev) if (ret) return ret; - if (data && data->have_governor_per_policy) - dt_cpufreq_driver.flags |= CPUFREQ_HAVE_GOVERNOR_PER_POLICY; + if (data) { + if (data->have_governor_per_policy) + dt_cpufreq_driver.flags |= CPUFREQ_HAVE_GOVERNOR_PER_POLICY; + + dt_cpufreq_driver.resume = data->resume; + if (data->suspend) + dt_cpufreq_driver.suspend = data->suspend; + } ret = cpufreq_register_driver(&dt_cpufreq_driver); if (ret) diff --git a/drivers/cpufreq/cpufreq-dt.h b/drivers/cpufreq/cpufreq-dt.h index 54d774e46c43..d5aeea13433e 100644 --- a/drivers/cpufreq/cpufreq-dt.h +++ b/drivers/cpufreq/cpufreq-dt.h @@ -12,8 +12,13 @@ #include +struct cpufreq_policy; + struct cpufreq_dt_platform_data { bool have_governor_per_policy; + + int (*suspend)(struct cpufreq_policy *policy); + int (*resume)(struct cpufreq_policy *policy); }; #endif /* __CPUFREQ_DT_H__ */ diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index 075d18f6ba7a..b0dfd3222013 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -89,16 +89,7 @@ static void cpufreq_governor_limits(struct cpufreq_policy *policy); * The mutex locks both lists. */ static BLOCKING_NOTIFIER_HEAD(cpufreq_policy_notifier_list); -static struct srcu_notifier_head cpufreq_transition_notifier_list; - -static bool init_cpufreq_transition_notifier_list_called; -static int __init init_cpufreq_transition_notifier_list(void) -{ - srcu_init_notifier_head(&cpufreq_transition_notifier_list); - init_cpufreq_transition_notifier_list_called = true; - return 0; -} -pure_initcall(init_cpufreq_transition_notifier_list); +SRCU_NOTIFIER_HEAD_STATIC(cpufreq_transition_notifier_list); static int off __read_mostly; static int cpufreq_disabled(void) @@ -300,8 +291,19 @@ static void adjust_jiffies(unsigned long val, struct cpufreq_freqs *ci) #endif } -static void __cpufreq_notify_transition(struct cpufreq_policy *policy, - struct cpufreq_freqs *freqs, unsigned int state) +/** + * cpufreq_notify_transition - Notify frequency transition and adjust_jiffies. + * @policy: cpufreq policy to enable fast frequency switching for. + * @freqs: contain details of the frequency update. + * @state: set to CPUFREQ_PRECHANGE or CPUFREQ_POSTCHANGE. + * + * This function calls the transition notifiers and the "adjust_jiffies" + * function. It is called twice on all CPU frequency changes that have + * external effects. + */ +static void cpufreq_notify_transition(struct cpufreq_policy *policy, + struct cpufreq_freqs *freqs, + unsigned int state) { BUG_ON(irqs_disabled()); @@ -313,54 +315,44 @@ static void __cpufreq_notify_transition(struct cpufreq_policy *policy, state, freqs->new); switch (state) { - case CPUFREQ_PRECHANGE: - /* detect if the driver reported a value as "old frequency" + /* + * Detect if the driver reported a value as "old frequency" * which is not equal to what the cpufreq core thinks is * "old frequency". */ if (!(cpufreq_driver->flags & CPUFREQ_CONST_LOOPS)) { - if ((policy) && (policy->cpu == freqs->cpu) && - (policy->cur) && (policy->cur != freqs->old)) { + if (policy->cur && (policy->cur != freqs->old)) { pr_debug("Warning: CPU frequency is %u, cpufreq assumed %u kHz\n", freqs->old, policy->cur); freqs->old = policy->cur; } } - srcu_notifier_call_chain(&cpufreq_transition_notifier_list, - CPUFREQ_PRECHANGE, freqs); + + for_each_cpu(freqs->cpu, policy->cpus) { + srcu_notifier_call_chain(&cpufreq_transition_notifier_list, + CPUFREQ_PRECHANGE, freqs); + } + adjust_jiffies(CPUFREQ_PRECHANGE, freqs); break; case CPUFREQ_POSTCHANGE: adjust_jiffies(CPUFREQ_POSTCHANGE, freqs); - pr_debug("FREQ: %lu - CPU: %lu\n", - (unsigned long)freqs->new, (unsigned long)freqs->cpu); - trace_cpu_frequency(freqs->new, freqs->cpu); + pr_debug("FREQ: %u - CPUs: %*pbl\n", freqs->new, + cpumask_pr_args(policy->cpus)); + + for_each_cpu(freqs->cpu, policy->cpus) { + trace_cpu_frequency(freqs->new, freqs->cpu); + srcu_notifier_call_chain(&cpufreq_transition_notifier_list, + CPUFREQ_POSTCHANGE, freqs); + } + cpufreq_stats_record_transition(policy, freqs->new); - srcu_notifier_call_chain(&cpufreq_transition_notifier_list, - CPUFREQ_POSTCHANGE, freqs); - if (likely(policy) && likely(policy->cpu == freqs->cpu)) - policy->cur = freqs->new; - break; + policy->cur = freqs->new; } } -/** - * cpufreq_notify_transition - call notifier chain and adjust_jiffies - * on frequency transition. - * - * This function calls the transition notifiers and the "adjust_jiffies" - * function. It is called twice on all CPU frequency changes that have - * external effects. - */ -static void cpufreq_notify_transition(struct cpufreq_policy *policy, - struct cpufreq_freqs *freqs, unsigned int state) -{ - for_each_cpu(freqs->cpu, policy->cpus) - __cpufreq_notify_transition(policy, freqs, state); -} - /* Do post notifications when there are chances that transition has failed */ static void cpufreq_notify_post_transition(struct cpufreq_policy *policy, struct cpufreq_freqs *freqs, int transition_failed) @@ -696,6 +688,8 @@ static ssize_t store_##file_name \ struct cpufreq_policy new_policy; \ \ memcpy(&new_policy, policy, sizeof(*policy)); \ + new_policy.min = policy->user_policy.min; \ + new_policy.max = policy->user_policy.max; \ \ ret = sscanf(buf, "%u", &new_policy.object); \ if (ret != 1) \ @@ -1764,8 +1758,6 @@ int cpufreq_register_notifier(struct notifier_block *nb, unsigned int list) if (cpufreq_disabled()) return -EINVAL; - WARN_ON(!init_cpufreq_transition_notifier_list_called); - switch (list) { case CPUFREQ_TRANSITION_NOTIFIER: mutex_lock(&cpufreq_fast_switch_lock); diff --git a/drivers/cpufreq/cpufreq_governor.c b/drivers/cpufreq/cpufreq_governor.c index ca38229b045a..1d50e97d49f1 100644 --- a/drivers/cpufreq/cpufreq_governor.c +++ b/drivers/cpufreq/cpufreq_governor.c @@ -165,7 +165,7 @@ unsigned int dbs_update(struct cpufreq_policy *policy) * calls, so the previous load value can be used then. */ load = j_cdbs->prev_load; - } else if (unlikely(time_elapsed > 2 * sampling_rate && + } else if (unlikely((int)idle_time > 2 * sampling_rate && j_cdbs->prev_load)) { /* * If the CPU had gone completely idle and a task has @@ -185,10 +185,8 @@ unsigned int dbs_update(struct cpufreq_policy *policy) * clear prev_load to guarantee that the load will be * computed again next time. * - * Detecting this situation is easy: the governor's - * utilization update handler would not have run during - * CPU-idle periods. Hence, an unusually large - * 'time_elapsed' (as compared to the sampling rate) + * Detecting this situation is easy: an unusually large + * 'idle_time' (as compared to the sampling rate) * indicates this scenario. */ load = j_cdbs->prev_load; @@ -217,8 +215,8 @@ unsigned int dbs_update(struct cpufreq_policy *policy) j_cdbs->prev_load = load; } - if (time_elapsed > 2 * sampling_rate) { - unsigned int periods = time_elapsed / sampling_rate; + if (unlikely((int)idle_time > 2 * sampling_rate)) { + unsigned int periods = idle_time / sampling_rate; if (periods < idle_periods) idle_periods = periods; @@ -278,7 +276,7 @@ static void dbs_update_util_handler(struct update_util_data *data, u64 time, struct policy_dbs_info *policy_dbs = cdbs->policy_dbs; u64 delta_ns, lst; - if (!cpufreq_can_do_remote_dvfs(policy_dbs->policy)) + if (!cpufreq_this_cpu_can_update(policy_dbs->policy)) return; /* diff --git a/drivers/cpufreq/ia64-acpi-cpufreq.c b/drivers/cpufreq/ia64-acpi-cpufreq.c index 7974a2fdb760..dd5440d3372d 100644 --- a/drivers/cpufreq/ia64-acpi-cpufreq.c +++ b/drivers/cpufreq/ia64-acpi-cpufreq.c @@ -241,8 +241,8 @@ acpi_cpufreq_cpu_init ( } /* alloc freq_table */ - freq_table = kzalloc(sizeof(*freq_table) * - (data->acpi_data.state_count + 1), + freq_table = kcalloc(data->acpi_data.state_count + 1, + sizeof(*freq_table), GFP_KERNEL); if (!freq_table) { result = -ENOMEM; diff --git a/drivers/cpufreq/imx6q-cpufreq.c b/drivers/cpufreq/imx6q-cpufreq.c index 83cf631fc9bc..8b3c2a79ad6c 100644 --- a/drivers/cpufreq/imx6q-cpufreq.c +++ b/drivers/cpufreq/imx6q-cpufreq.c @@ -266,6 +266,8 @@ put_node: } #define OCOTP_CFG3_6UL_SPEED_696MHZ 0x2 +#define OCOTP_CFG3_6ULL_SPEED_792MHZ 0x2 +#define OCOTP_CFG3_6ULL_SPEED_900MHZ 0x3 static void imx6ul_opp_check_speed_grading(struct device *dev) { @@ -287,16 +289,30 @@ static void imx6ul_opp_check_speed_grading(struct device *dev) * Speed GRADING[1:0] defines the max speed of ARM: * 2b'00: Reserved; * 2b'01: 528000000Hz; - * 2b'10: 696000000Hz; - * 2b'11: Reserved; + * 2b'10: 696000000Hz on i.MX6UL, 792000000Hz on i.MX6ULL; + * 2b'11: 900000000Hz on i.MX6ULL only; * We need to set the max speed of ARM according to fuse map. */ val = readl_relaxed(base + OCOTP_CFG3); val >>= OCOTP_CFG3_SPEED_SHIFT; val &= 0x3; - if (val != OCOTP_CFG3_6UL_SPEED_696MHZ) - if (dev_pm_opp_disable(dev, 696000000)) - dev_warn(dev, "failed to disable 696MHz OPP\n"); + + if (of_machine_is_compatible("fsl,imx6ul")) { + if (val != OCOTP_CFG3_6UL_SPEED_696MHZ) + if (dev_pm_opp_disable(dev, 696000000)) + dev_warn(dev, "failed to disable 696MHz OPP\n"); + } + + if (of_machine_is_compatible("fsl,imx6ull")) { + if (val != OCOTP_CFG3_6ULL_SPEED_792MHZ) + if (dev_pm_opp_disable(dev, 792000000)) + dev_warn(dev, "failed to disable 792MHz OPP\n"); + + if (val != OCOTP_CFG3_6ULL_SPEED_900MHZ) + if (dev_pm_opp_disable(dev, 900000000)) + dev_warn(dev, "failed to disable 900MHz OPP\n"); + } + iounmap(base); put_node: of_node_put(np); @@ -356,7 +372,8 @@ static int imx6q_cpufreq_probe(struct platform_device *pdev) goto put_reg; } - if (of_machine_is_compatible("fsl,imx6ul")) + if (of_machine_is_compatible("fsl,imx6ul") || + of_machine_is_compatible("fsl,imx6ull")) imx6ul_opp_check_speed_grading(cpu_dev); else imx6q_opp_check_speed_grading(cpu_dev); @@ -377,7 +394,8 @@ static int imx6q_cpufreq_probe(struct platform_device *pdev) } /* Make imx6_soc_volt array's size same as arm opp number */ - imx6_soc_volt = devm_kzalloc(cpu_dev, sizeof(*imx6_soc_volt) * num, GFP_KERNEL); + imx6_soc_volt = devm_kcalloc(cpu_dev, num, sizeof(*imx6_soc_volt), + GFP_KERNEL); if (imx6_soc_volt == NULL) { ret = -ENOMEM; goto free_freq_table; diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c index 17e566afbb41..ece120da3353 100644 --- a/drivers/cpufreq/intel_pstate.c +++ b/drivers/cpufreq/intel_pstate.c @@ -221,6 +221,11 @@ struct global_params { * preference/bias * @epp_saved: Saved EPP/EPB during system suspend or CPU offline * operation + * @hwp_req_cached: Cached value of the last HWP Request MSR + * @hwp_cap_cached: Cached value of the last HWP Capabilities MSR + * @last_io_update: Last time when IO wake flag was set + * @sched_flags: Store scheduler flags for possible cross CPU update + * @hwp_boost_min: Last HWP boosted min performance * * This structure stores per CPU instance data for all CPUs. */ @@ -253,6 +258,11 @@ struct cpudata { s16 epp_policy; s16 epp_default; s16 epp_saved; + u64 hwp_req_cached; + u64 hwp_cap_cached; + u64 last_io_update; + unsigned int sched_flags; + u32 hwp_boost_min; }; static struct cpudata **all_cpu_data; @@ -284,7 +294,9 @@ struct pstate_funcs { static struct pstate_funcs pstate_funcs __read_mostly; static int hwp_active __read_mostly; +static int hwp_mode_bdw __read_mostly; static bool per_cpu_limits __read_mostly; +static bool hwp_boost __read_mostly; static struct cpufreq_driver *intel_pstate_driver __read_mostly; @@ -689,6 +701,7 @@ static void intel_pstate_get_hwp_max(unsigned int cpu, int *phy_max, u64 cap; rdmsrl_on_cpu(cpu, MSR_HWP_CAPABILITIES, &cap); + WRITE_ONCE(all_cpu_data[cpu]->hwp_cap_cached, cap); if (global.no_turbo) *current_max = HWP_GUARANTEED_PERF(cap); else @@ -763,6 +776,7 @@ update_epp: intel_pstate_set_epb(cpu, epp); } skip_epp: + WRITE_ONCE(cpu_data->hwp_req_cached, value); wrmsrl_on_cpu(cpu, MSR_HWP_REQUEST, value); } @@ -1020,6 +1034,30 @@ static ssize_t store_min_perf_pct(struct kobject *a, struct attribute *b, return count; } +static ssize_t show_hwp_dynamic_boost(struct kobject *kobj, + struct attribute *attr, char *buf) +{ + return sprintf(buf, "%u\n", hwp_boost); +} + +static ssize_t store_hwp_dynamic_boost(struct kobject *a, struct attribute *b, + const char *buf, size_t count) +{ + unsigned int input; + int ret; + + ret = kstrtouint(buf, 10, &input); + if (ret) + return ret; + + mutex_lock(&intel_pstate_driver_lock); + hwp_boost = !!input; + intel_pstate_update_policies(); + mutex_unlock(&intel_pstate_driver_lock); + + return count; +} + show_one(max_perf_pct, max_perf_pct); show_one(min_perf_pct, min_perf_pct); @@ -1029,6 +1067,7 @@ define_one_global_rw(max_perf_pct); define_one_global_rw(min_perf_pct); define_one_global_ro(turbo_pct); define_one_global_ro(num_pstates); +define_one_global_rw(hwp_dynamic_boost); static struct attribute *intel_pstate_attributes[] = { &status.attr, @@ -1069,6 +1108,11 @@ static void __init intel_pstate_sysfs_expose_params(void) rc = sysfs_create_file(intel_pstate_kobject, &min_perf_pct.attr); WARN_ON(rc); + if (hwp_active) { + rc = sysfs_create_file(intel_pstate_kobject, + &hwp_dynamic_boost.attr); + WARN_ON(rc); + } } /************************** sysfs end ************************/ @@ -1370,7 +1414,15 @@ static void intel_pstate_get_cpu_pstates(struct cpudata *cpu) cpu->pstate.turbo_pstate = pstate_funcs.get_turbo(); cpu->pstate.scaling = pstate_funcs.get_scaling(); cpu->pstate.max_freq = cpu->pstate.max_pstate * cpu->pstate.scaling; - cpu->pstate.turbo_freq = cpu->pstate.turbo_pstate * cpu->pstate.scaling; + + if (hwp_active && !hwp_mode_bdw) { + unsigned int phy_max, current_max; + + intel_pstate_get_hwp_max(cpu->cpu, &phy_max, ¤t_max); + cpu->pstate.turbo_freq = phy_max * cpu->pstate.scaling; + } else { + cpu->pstate.turbo_freq = cpu->pstate.turbo_pstate * cpu->pstate.scaling; + } if (pstate_funcs.get_aperf_mperf_shift) cpu->aperf_mperf_shift = pstate_funcs.get_aperf_mperf_shift(); @@ -1381,6 +1433,116 @@ static void intel_pstate_get_cpu_pstates(struct cpudata *cpu) intel_pstate_set_min_pstate(cpu); } +/* + * Long hold time will keep high perf limits for long time, + * which negatively impacts perf/watt for some workloads, + * like specpower. 3ms is based on experiements on some + * workoads. + */ +static int hwp_boost_hold_time_ns = 3 * NSEC_PER_MSEC; + +static inline void intel_pstate_hwp_boost_up(struct cpudata *cpu) +{ + u64 hwp_req = READ_ONCE(cpu->hwp_req_cached); + u32 max_limit = (hwp_req & 0xff00) >> 8; + u32 min_limit = (hwp_req & 0xff); + u32 boost_level1; + + /* + * Cases to consider (User changes via sysfs or boot time): + * If, P0 (Turbo max) = P1 (Guaranteed max) = min: + * No boost, return. + * If, P0 (Turbo max) > P1 (Guaranteed max) = min: + * Should result in one level boost only for P0. + * If, P0 (Turbo max) = P1 (Guaranteed max) > min: + * Should result in two level boost: + * (min + p1)/2 and P1. + * If, P0 (Turbo max) > P1 (Guaranteed max) > min: + * Should result in three level boost: + * (min + p1)/2, P1 and P0. + */ + + /* If max and min are equal or already at max, nothing to boost */ + if (max_limit == min_limit || cpu->hwp_boost_min >= max_limit) + return; + + if (!cpu->hwp_boost_min) + cpu->hwp_boost_min = min_limit; + + /* level at half way mark between min and guranteed */ + boost_level1 = (HWP_GUARANTEED_PERF(cpu->hwp_cap_cached) + min_limit) >> 1; + + if (cpu->hwp_boost_min < boost_level1) + cpu->hwp_boost_min = boost_level1; + else if (cpu->hwp_boost_min < HWP_GUARANTEED_PERF(cpu->hwp_cap_cached)) + cpu->hwp_boost_min = HWP_GUARANTEED_PERF(cpu->hwp_cap_cached); + else if (cpu->hwp_boost_min == HWP_GUARANTEED_PERF(cpu->hwp_cap_cached) && + max_limit != HWP_GUARANTEED_PERF(cpu->hwp_cap_cached)) + cpu->hwp_boost_min = max_limit; + else + return; + + hwp_req = (hwp_req & ~GENMASK_ULL(7, 0)) | cpu->hwp_boost_min; + wrmsrl(MSR_HWP_REQUEST, hwp_req); + cpu->last_update = cpu->sample.time; +} + +static inline void intel_pstate_hwp_boost_down(struct cpudata *cpu) +{ + if (cpu->hwp_boost_min) { + bool expired; + + /* Check if we are idle for hold time to boost down */ + expired = time_after64(cpu->sample.time, cpu->last_update + + hwp_boost_hold_time_ns); + if (expired) { + wrmsrl(MSR_HWP_REQUEST, cpu->hwp_req_cached); + cpu->hwp_boost_min = 0; + } + } + cpu->last_update = cpu->sample.time; +} + +static inline void intel_pstate_update_util_hwp_local(struct cpudata *cpu, + u64 time) +{ + cpu->sample.time = time; + + if (cpu->sched_flags & SCHED_CPUFREQ_IOWAIT) { + bool do_io = false; + + cpu->sched_flags = 0; + /* + * Set iowait_boost flag and update time. Since IO WAIT flag + * is set all the time, we can't just conclude that there is + * some IO bound activity is scheduled on this CPU with just + * one occurrence. If we receive at least two in two + * consecutive ticks, then we treat as boost candidate. + */ + if (time_before64(time, cpu->last_io_update + 2 * TICK_NSEC)) + do_io = true; + + cpu->last_io_update = time; + + if (do_io) + intel_pstate_hwp_boost_up(cpu); + + } else { + intel_pstate_hwp_boost_down(cpu); + } +} + +static inline void intel_pstate_update_util_hwp(struct update_util_data *data, + u64 time, unsigned int flags) +{ + struct cpudata *cpu = container_of(data, struct cpudata, update_util); + + cpu->sched_flags |= flags; + + if (smp_processor_id() == cpu->cpu) + intel_pstate_update_util_hwp_local(cpu, time); +} + static inline void intel_pstate_calc_avg_perf(struct cpudata *cpu) { struct sample *sample = &cpu->sample; @@ -1641,6 +1803,12 @@ static const struct x86_cpu_id intel_pstate_cpu_ee_disable_ids[] = { {} }; +static const struct x86_cpu_id intel_pstate_hwp_boost_ids[] = { + ICPU(INTEL_FAM6_SKYLAKE_X, core_funcs), + ICPU(INTEL_FAM6_SKYLAKE_DESKTOP, core_funcs), + {} +}; + static int intel_pstate_init_cpu(unsigned int cpunum) { struct cpudata *cpu; @@ -1671,6 +1839,10 @@ static int intel_pstate_init_cpu(unsigned int cpunum) intel_pstate_disable_ee(cpunum); intel_pstate_hwp_enable(cpu); + + id = x86_match_cpu(intel_pstate_hwp_boost_ids); + if (id) + hwp_boost = true; } intel_pstate_get_cpu_pstates(cpu); @@ -1684,7 +1856,7 @@ static void intel_pstate_set_update_util_hook(unsigned int cpu_num) { struct cpudata *cpu = all_cpu_data[cpu_num]; - if (hwp_active) + if (hwp_active && !hwp_boost) return; if (cpu->update_util_set) @@ -1693,7 +1865,9 @@ static void intel_pstate_set_update_util_hook(unsigned int cpu_num) /* Prevent intel_pstate_update_util() from using stale data. */ cpu->sample.time = 0; cpufreq_add_update_util_hook(cpu_num, &cpu->update_util, - intel_pstate_update_util); + (hwp_active ? + intel_pstate_update_util_hwp : + intel_pstate_update_util)); cpu->update_util_set = true; } @@ -1805,8 +1979,16 @@ static int intel_pstate_set_policy(struct cpufreq_policy *policy) intel_pstate_set_update_util_hook(policy->cpu); } - if (hwp_active) + if (hwp_active) { + /* + * When hwp_boost was active before and dynamically it + * was turned off, in that case we need to clear the + * update util hook. + */ + if (!hwp_boost) + intel_pstate_clear_update_util_hook(policy->cpu); intel_pstate_hwp_set(policy->cpu); + } mutex_unlock(&intel_pstate_limits_lock); @@ -1939,13 +2121,51 @@ static int intel_cpufreq_verify_policy(struct cpufreq_policy *policy) return 0; } +/* Use of trace in passive mode: + * + * In passive mode the trace core_busy field (also known as the + * performance field, and lablelled as such on the graphs; also known as + * core_avg_perf) is not needed and so is re-assigned to indicate if the + * driver call was via the normal or fast switch path. Various graphs + * output from the intel_pstate_tracer.py utility that include core_busy + * (or performance or core_avg_perf) have a fixed y-axis from 0 to 100%, + * so we use 10 to indicate the the normal path through the driver, and + * 90 to indicate the fast switch path through the driver. + * The scaled_busy field is not used, and is set to 0. + */ + +#define INTEL_PSTATE_TRACE_TARGET 10 +#define INTEL_PSTATE_TRACE_FAST_SWITCH 90 + +static void intel_cpufreq_trace(struct cpudata *cpu, unsigned int trace_type, int old_pstate) +{ + struct sample *sample; + + if (!trace_pstate_sample_enabled()) + return; + + if (!intel_pstate_sample(cpu, ktime_get())) + return; + + sample = &cpu->sample; + trace_pstate_sample(trace_type, + 0, + old_pstate, + cpu->pstate.current_pstate, + sample->mperf, + sample->aperf, + sample->tsc, + get_avg_frequency(cpu), + fp_toint(cpu->iowait_boost * 100)); +} + static int intel_cpufreq_target(struct cpufreq_policy *policy, unsigned int target_freq, unsigned int relation) { struct cpudata *cpu = all_cpu_data[policy->cpu]; struct cpufreq_freqs freqs; - int target_pstate; + int target_pstate, old_pstate; update_turbo_state(); @@ -1965,12 +2185,14 @@ static int intel_cpufreq_target(struct cpufreq_policy *policy, break; } target_pstate = intel_pstate_prepare_request(cpu, target_pstate); + old_pstate = cpu->pstate.current_pstate; if (target_pstate != cpu->pstate.current_pstate) { cpu->pstate.current_pstate = target_pstate; wrmsrl_on_cpu(policy->cpu, MSR_IA32_PERF_CTL, pstate_funcs.get_val(cpu, target_pstate)); } freqs.new = target_pstate * cpu->pstate.scaling; + intel_cpufreq_trace(cpu, INTEL_PSTATE_TRACE_TARGET, old_pstate); cpufreq_freq_transition_end(policy, &freqs, false); return 0; @@ -1980,13 +2202,15 @@ static unsigned int intel_cpufreq_fast_switch(struct cpufreq_policy *policy, unsigned int target_freq) { struct cpudata *cpu = all_cpu_data[policy->cpu]; - int target_pstate; + int target_pstate, old_pstate; update_turbo_state(); target_pstate = DIV_ROUND_UP(target_freq, cpu->pstate.scaling); target_pstate = intel_pstate_prepare_request(cpu, target_pstate); + old_pstate = cpu->pstate.current_pstate; intel_pstate_update_pstate(cpu, target_pstate); + intel_cpufreq_trace(cpu, INTEL_PSTATE_TRACE_FAST_SWITCH, old_pstate); return target_pstate * cpu->pstate.scaling; } @@ -2252,28 +2476,36 @@ static inline bool intel_pstate_has_acpi_ppc(void) { return false; } static inline void intel_pstate_request_control_from_smm(void) {} #endif /* CONFIG_ACPI */ +#define INTEL_PSTATE_HWP_BROADWELL 0x01 + +#define ICPU_HWP(model, hwp_mode) \ + { X86_VENDOR_INTEL, 6, model, X86_FEATURE_HWP, hwp_mode } + static const struct x86_cpu_id hwp_support_ids[] __initconst = { - { X86_VENDOR_INTEL, 6, X86_MODEL_ANY, X86_FEATURE_HWP }, + ICPU_HWP(INTEL_FAM6_BROADWELL_X, INTEL_PSTATE_HWP_BROADWELL), + ICPU_HWP(INTEL_FAM6_BROADWELL_XEON_D, INTEL_PSTATE_HWP_BROADWELL), + ICPU_HWP(X86_MODEL_ANY, 0), {} }; static int __init intel_pstate_init(void) { + const struct x86_cpu_id *id; int rc; if (no_load) return -ENODEV; - if (x86_match_cpu(hwp_support_ids)) { + id = x86_match_cpu(hwp_support_ids); + if (id) { copy_cpu_funcs(&core_funcs); if (!no_hwp) { hwp_active++; + hwp_mode_bdw = id->driver_data; intel_pstate.attr = hwp_cpufreq_attrs; goto hwp_cpu_matched; } } else { - const struct x86_cpu_id *id; - id = x86_match_cpu(intel_pstate_cpu_ids); if (!id) return -ENODEV; @@ -2297,7 +2529,7 @@ hwp_cpu_matched: pr_info("Intel P-state driver initializing\n"); - all_cpu_data = vzalloc(sizeof(void *) * num_possible_cpus()); + all_cpu_data = vzalloc(array_size(sizeof(void *), num_possible_cpus())); if (!all_cpu_data) return -ENOMEM; diff --git a/drivers/cpufreq/longhaul.c b/drivers/cpufreq/longhaul.c index 61a4c5b08219..279bd9e9fa95 100644 --- a/drivers/cpufreq/longhaul.c +++ b/drivers/cpufreq/longhaul.c @@ -474,8 +474,8 @@ static int longhaul_get_ranges(void) return -EINVAL; } - longhaul_table = kzalloc((numscales + 1) * sizeof(*longhaul_table), - GFP_KERNEL); + longhaul_table = kcalloc(numscales + 1, sizeof(*longhaul_table), + GFP_KERNEL); if (!longhaul_table) return -ENOMEM; diff --git a/drivers/cpufreq/pxa3xx-cpufreq.c b/drivers/cpufreq/pxa3xx-cpufreq.c index 7acc7fa4536d..9daa2cc318bb 100644 --- a/drivers/cpufreq/pxa3xx-cpufreq.c +++ b/drivers/cpufreq/pxa3xx-cpufreq.c @@ -93,7 +93,7 @@ static int setup_freqs_table(struct cpufreq_policy *policy, struct cpufreq_frequency_table *table; int i; - table = kzalloc((num + 1) * sizeof(*table), GFP_KERNEL); + table = kcalloc(num + 1, sizeof(*table), GFP_KERNEL); if (table == NULL) return -ENOMEM; diff --git a/drivers/cpufreq/qcom-cpufreq-kryo.c b/drivers/cpufreq/qcom-cpufreq-kryo.c new file mode 100644 index 000000000000..01bddacf5c3b --- /dev/null +++ b/drivers/cpufreq/qcom-cpufreq-kryo.c @@ -0,0 +1,231 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2018, The Linux Foundation. All rights reserved. + */ + +/* + * In Certain QCOM SoCs like apq8096 and msm8996 that have KRYO processors, + * the CPU frequency subset and voltage value of each OPP varies + * based on the silicon variant in use. Qualcomm Process Voltage Scaling Tables + * defines the voltage and frequency value based on the msm-id in SMEM + * and speedbin blown in the efuse combination. + * The qcom-cpufreq-kryo driver reads the msm-id and efuse value from the SoC + * to provide the OPP framework with required information. + * This is used to determine the voltage and frequency value for each OPP of + * operating-points-v2 table when it is parsed by the OPP framework. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MSM_ID_SMEM 137 + +enum _msm_id { + MSM8996V3 = 0xF6ul, + APQ8096V3 = 0x123ul, + MSM8996SG = 0x131ul, + APQ8096SG = 0x138ul, +}; + +enum _msm8996_version { + MSM8996_V3, + MSM8996_SG, + NUM_OF_MSM8996_VERSIONS, +}; + +struct platform_device *cpufreq_dt_pdev, *kryo_cpufreq_pdev; + +static enum _msm8996_version __init qcom_cpufreq_kryo_get_msm_id(void) +{ + size_t len; + u32 *msm_id; + enum _msm8996_version version; + + msm_id = qcom_smem_get(QCOM_SMEM_HOST_ANY, MSM_ID_SMEM, &len); + if (IS_ERR(msm_id)) + return NUM_OF_MSM8996_VERSIONS; + + /* The first 4 bytes are format, next to them is the actual msm-id */ + msm_id++; + + switch ((enum _msm_id)*msm_id) { + case MSM8996V3: + case APQ8096V3: + version = MSM8996_V3; + break; + case MSM8996SG: + case APQ8096SG: + version = MSM8996_SG; + break; + default: + version = NUM_OF_MSM8996_VERSIONS; + } + + return version; +} + +static int qcom_cpufreq_kryo_probe(struct platform_device *pdev) +{ + struct opp_table *opp_tables[NR_CPUS] = {0}; + enum _msm8996_version msm8996_version; + struct nvmem_cell *speedbin_nvmem; + struct device_node *np; + struct device *cpu_dev; + unsigned cpu; + u8 *speedbin; + u32 versions; + size_t len; + int ret; + + cpu_dev = get_cpu_device(0); + if (NULL == cpu_dev) + ret = -ENODEV; + + msm8996_version = qcom_cpufreq_kryo_get_msm_id(); + if (NUM_OF_MSM8996_VERSIONS == msm8996_version) { + dev_err(cpu_dev, "Not Snapdragon 820/821!"); + return -ENODEV; + } + + np = dev_pm_opp_of_get_opp_desc_node(cpu_dev); + if (IS_ERR(np)) + return PTR_ERR(np); + + ret = of_device_is_compatible(np, "operating-points-v2-kryo-cpu"); + if (!ret) { + of_node_put(np); + return -ENOENT; + } + + speedbin_nvmem = of_nvmem_cell_get(np, NULL); + of_node_put(np); + if (IS_ERR(speedbin_nvmem)) { + dev_err(cpu_dev, "Could not get nvmem cell: %ld\n", + PTR_ERR(speedbin_nvmem)); + return PTR_ERR(speedbin_nvmem); + } + + speedbin = nvmem_cell_read(speedbin_nvmem, &len); + nvmem_cell_put(speedbin_nvmem); + if (IS_ERR(speedbin)) + return PTR_ERR(speedbin); + + switch (msm8996_version) { + case MSM8996_V3: + versions = 1 << (unsigned int)(*speedbin); + break; + case MSM8996_SG: + versions = 1 << ((unsigned int)(*speedbin) + 4); + break; + default: + BUG(); + break; + } + kfree(speedbin); + + for_each_possible_cpu(cpu) { + cpu_dev = get_cpu_device(cpu); + if (NULL == cpu_dev) { + ret = -ENODEV; + goto free_opp; + } + + opp_tables[cpu] = dev_pm_opp_set_supported_hw(cpu_dev, + &versions, 1); + if (IS_ERR(opp_tables[cpu])) { + ret = PTR_ERR(opp_tables[cpu]); + dev_err(cpu_dev, "Failed to set supported hardware\n"); + goto free_opp; + } + } + + cpufreq_dt_pdev = platform_device_register_simple("cpufreq-dt", -1, + NULL, 0); + if (!IS_ERR(cpufreq_dt_pdev)) + return 0; + + ret = PTR_ERR(cpufreq_dt_pdev); + dev_err(cpu_dev, "Failed to register platform device\n"); + +free_opp: + for_each_possible_cpu(cpu) { + if (IS_ERR_OR_NULL(opp_tables[cpu])) + break; + dev_pm_opp_put_supported_hw(opp_tables[cpu]); + } + + return ret; +} + +static int qcom_cpufreq_kryo_remove(struct platform_device *pdev) +{ + platform_device_unregister(cpufreq_dt_pdev); + return 0; +} + +static struct platform_driver qcom_cpufreq_kryo_driver = { + .probe = qcom_cpufreq_kryo_probe, + .remove = qcom_cpufreq_kryo_remove, + .driver = { + .name = "qcom-cpufreq-kryo", + }, +}; + +static const struct of_device_id qcom_cpufreq_kryo_match_list[] __initconst = { + { .compatible = "qcom,apq8096", }, + { .compatible = "qcom,msm8996", }, +}; + +/* + * Since the driver depends on smem and nvmem drivers, which may + * return EPROBE_DEFER, all the real activity is done in the probe, + * which may be defered as well. The init here is only registering + * the driver and the platform device. + */ +static int __init qcom_cpufreq_kryo_init(void) +{ + struct device_node *np = of_find_node_by_path("/"); + const struct of_device_id *match; + int ret; + + if (!np) + return -ENODEV; + + match = of_match_node(qcom_cpufreq_kryo_match_list, np); + of_node_put(np); + if (!match) + return -ENODEV; + + ret = platform_driver_register(&qcom_cpufreq_kryo_driver); + if (unlikely(ret < 0)) + return ret; + + kryo_cpufreq_pdev = platform_device_register_simple( + "qcom-cpufreq-kryo", -1, NULL, 0); + ret = PTR_ERR_OR_ZERO(kryo_cpufreq_pdev); + if (0 == ret) + return 0; + + platform_driver_unregister(&qcom_cpufreq_kryo_driver); + return ret; +} +module_init(qcom_cpufreq_kryo_init); + +static void __init qcom_cpufreq_kryo_exit(void) +{ + platform_device_unregister(kryo_cpufreq_pdev); + platform_driver_unregister(&qcom_cpufreq_kryo_driver); +} +module_exit(qcom_cpufreq_kryo_exit); + +MODULE_DESCRIPTION("Qualcomm Technologies, Inc. Kryo CPUfreq driver"); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/cpufreq/s3c2440-cpufreq.c b/drivers/cpufreq/s3c2440-cpufreq.c index d0d75b65ddd6..d2f67b7a20dd 100644 --- a/drivers/cpufreq/s3c2440-cpufreq.c +++ b/drivers/cpufreq/s3c2440-cpufreq.c @@ -143,7 +143,7 @@ static void s3c2440_cpufreq_setdivs(struct s3c_cpufreq_config *cfg) { unsigned long clkdiv, camdiv; - s3c_freq_dbg("%s: divsiors: h=%d, p=%d\n", __func__, + s3c_freq_dbg("%s: divisors: h=%d, p=%d\n", __func__, cfg->divs.h_divisor, cfg->divs.p_divisor); clkdiv = __raw_readl(S3C2410_CLKDIVN); diff --git a/drivers/cpufreq/s3c24xx-cpufreq.c b/drivers/cpufreq/s3c24xx-cpufreq.c index 909bd6e27639..3b291a2b0cb3 100644 --- a/drivers/cpufreq/s3c24xx-cpufreq.c +++ b/drivers/cpufreq/s3c24xx-cpufreq.c @@ -562,7 +562,7 @@ static int s3c_cpufreq_build_freq(void) size = cpu_cur.info->calc_freqtable(&cpu_cur, NULL, 0); size++; - ftab = kzalloc(sizeof(*ftab) * size, GFP_KERNEL); + ftab = kcalloc(size, sizeof(*ftab), GFP_KERNEL); if (!ftab) return -ENOMEM; diff --git a/drivers/cpufreq/scmi-cpufreq.c b/drivers/cpufreq/scmi-cpufreq.c index b4dbc77459b6..50b1551ba894 100644 --- a/drivers/cpufreq/scmi-cpufreq.c +++ b/drivers/cpufreq/scmi-cpufreq.c @@ -117,7 +117,7 @@ static int scmi_cpufreq_init(struct cpufreq_policy *policy) return -ENODEV; } - ret = handle->perf_ops->add_opps_to_device(handle, cpu_dev); + ret = handle->perf_ops->device_opps_add(handle, cpu_dev); if (ret) { dev_warn(cpu_dev, "failed to add opps to the device\n"); return ret; @@ -164,7 +164,7 @@ static int scmi_cpufreq_init(struct cpufreq_policy *policy) /* SCMI allows DVFS request for any domain from any CPU */ policy->dvfs_possible_from_any_cpu = true; - latency = handle->perf_ops->get_transition_latency(handle, cpu_dev); + latency = handle->perf_ops->transition_latency_get(handle, cpu_dev); if (!latency) latency = CPUFREQ_ETERNAL; diff --git a/drivers/cpufreq/sfi-cpufreq.c b/drivers/cpufreq/sfi-cpufreq.c index 9767afe05da2..978770432b13 100644 --- a/drivers/cpufreq/sfi-cpufreq.c +++ b/drivers/cpufreq/sfi-cpufreq.c @@ -95,8 +95,8 @@ static int __init sfi_cpufreq_init(void) if (ret) return ret; - freq_table = kzalloc(sizeof(*freq_table) * - (num_freq_table_entries + 1), GFP_KERNEL); + freq_table = kcalloc(num_freq_table_entries + 1, sizeof(*freq_table), + GFP_KERNEL); if (!freq_table) { ret = -ENOMEM; goto err_free_array; diff --git a/drivers/cpufreq/spear-cpufreq.c b/drivers/cpufreq/spear-cpufreq.c index 195f27f9c1cb..4074e2615522 100644 --- a/drivers/cpufreq/spear-cpufreq.c +++ b/drivers/cpufreq/spear-cpufreq.c @@ -195,7 +195,7 @@ static int spear_cpufreq_probe(struct platform_device *pdev) cnt = prop->length / sizeof(u32); val = prop->value; - freq_tbl = kzalloc(sizeof(*freq_tbl) * (cnt + 1), GFP_KERNEL); + freq_tbl = kcalloc(cnt + 1, sizeof(*freq_tbl), GFP_KERNEL); if (!freq_tbl) { ret = -ENOMEM; goto out_put_node; diff --git a/drivers/cpufreq/speedstep-lib.c b/drivers/cpufreq/speedstep-lib.c index e3a9962ee410..cabb6f48eb77 100644 --- a/drivers/cpufreq/speedstep-lib.c +++ b/drivers/cpufreq/speedstep-lib.c @@ -252,7 +252,7 @@ EXPORT_SYMBOL_GPL(speedstep_get_frequency); *********************************************************************/ /* Keep in sync with the x86_cpu_id tables in the different modules */ -unsigned int speedstep_detect_processor(void) +enum speedstep_processor speedstep_detect_processor(void) { struct cpuinfo_x86 *c = &cpu_data(0); u32 ebx, msr_lo, msr_hi; diff --git a/drivers/cpufreq/tegra20-cpufreq.c b/drivers/cpufreq/tegra20-cpufreq.c index 2bd62845e9d5..05f57dcd5215 100644 --- a/drivers/cpufreq/tegra20-cpufreq.c +++ b/drivers/cpufreq/tegra20-cpufreq.c @@ -16,16 +16,13 @@ * */ -#include -#include -#include -#include +#include #include -#include -#include #include -#include -#include +#include +#include +#include +#include static struct cpufreq_frequency_table freq_table[] = { { .frequency = 216000 }, @@ -39,25 +36,27 @@ static struct cpufreq_frequency_table freq_table[] = { { .frequency = CPUFREQ_TABLE_END }, }; -#define NUM_CPUS 2 - -static struct clk *cpu_clk; -static struct clk *pll_x_clk; -static struct clk *pll_p_clk; -static struct clk *emc_clk; -static bool pll_x_prepared; +struct tegra20_cpufreq { + struct device *dev; + struct cpufreq_driver driver; + struct clk *cpu_clk; + struct clk *pll_x_clk; + struct clk *pll_p_clk; + bool pll_x_prepared; +}; static unsigned int tegra_get_intermediate(struct cpufreq_policy *policy, unsigned int index) { - unsigned int ifreq = clk_get_rate(pll_p_clk) / 1000; + struct tegra20_cpufreq *cpufreq = cpufreq_get_driver_data(); + unsigned int ifreq = clk_get_rate(cpufreq->pll_p_clk) / 1000; /* * Don't switch to intermediate freq if: * - we are already at it, i.e. policy->cur == ifreq * - index corresponds to ifreq */ - if ((freq_table[index].frequency == ifreq) || (policy->cur == ifreq)) + if (freq_table[index].frequency == ifreq || policy->cur == ifreq) return 0; return ifreq; @@ -66,6 +65,7 @@ static unsigned int tegra_get_intermediate(struct cpufreq_policy *policy, static int tegra_target_intermediate(struct cpufreq_policy *policy, unsigned int index) { + struct tegra20_cpufreq *cpufreq = cpufreq_get_driver_data(); int ret; /* @@ -78,47 +78,37 @@ static int tegra_target_intermediate(struct cpufreq_policy *policy, * Also, we wouldn't be using pll_x anymore and must not take extra * reference to it, as it can be disabled now to save some power. */ - clk_prepare_enable(pll_x_clk); + clk_prepare_enable(cpufreq->pll_x_clk); - ret = clk_set_parent(cpu_clk, pll_p_clk); + ret = clk_set_parent(cpufreq->cpu_clk, cpufreq->pll_p_clk); if (ret) - clk_disable_unprepare(pll_x_clk); + clk_disable_unprepare(cpufreq->pll_x_clk); else - pll_x_prepared = true; + cpufreq->pll_x_prepared = true; return ret; } static int tegra_target(struct cpufreq_policy *policy, unsigned int index) { + struct tegra20_cpufreq *cpufreq = cpufreq_get_driver_data(); unsigned long rate = freq_table[index].frequency; - unsigned int ifreq = clk_get_rate(pll_p_clk) / 1000; - int ret = 0; - - /* - * Vote on memory bus frequency based on cpu frequency - * This sets the minimum frequency, display or avp may request higher - */ - if (rate >= 816000) - clk_set_rate(emc_clk, 600000000); /* cpu 816 MHz, emc max */ - else if (rate >= 456000) - clk_set_rate(emc_clk, 300000000); /* cpu 456 MHz, emc 150Mhz */ - else - clk_set_rate(emc_clk, 100000000); /* emc 50Mhz */ + unsigned int ifreq = clk_get_rate(cpufreq->pll_p_clk) / 1000; + int ret; /* * target freq == pll_p, don't need to take extra reference to pll_x_clk * as it isn't used anymore. */ if (rate == ifreq) - return clk_set_parent(cpu_clk, pll_p_clk); + return clk_set_parent(cpufreq->cpu_clk, cpufreq->pll_p_clk); - ret = clk_set_rate(pll_x_clk, rate * 1000); + ret = clk_set_rate(cpufreq->pll_x_clk, rate * 1000); /* Restore to earlier frequency on error, i.e. pll_x */ if (ret) - pr_err("Failed to change pll_x to %lu\n", rate); + dev_err(cpufreq->dev, "Failed to change pll_x to %lu\n", rate); - ret = clk_set_parent(cpu_clk, pll_x_clk); + ret = clk_set_parent(cpufreq->cpu_clk, cpufreq->pll_x_clk); /* This shouldn't fail while changing or restoring */ WARN_ON(ret); @@ -126,9 +116,9 @@ static int tegra_target(struct cpufreq_policy *policy, unsigned int index) * Drop count to pll_x clock only if we switched to intermediate freq * earlier while transitioning to a target frequency. */ - if (pll_x_prepared) { - clk_disable_unprepare(pll_x_clk); - pll_x_prepared = false; + if (cpufreq->pll_x_prepared) { + clk_disable_unprepare(cpufreq->pll_x_clk); + cpufreq->pll_x_prepared = false; } return ret; @@ -136,81 +126,111 @@ static int tegra_target(struct cpufreq_policy *policy, unsigned int index) static int tegra_cpu_init(struct cpufreq_policy *policy) { + struct tegra20_cpufreq *cpufreq = cpufreq_get_driver_data(); int ret; - if (policy->cpu >= NUM_CPUS) - return -EINVAL; - - clk_prepare_enable(emc_clk); - clk_prepare_enable(cpu_clk); + clk_prepare_enable(cpufreq->cpu_clk); /* FIXME: what's the actual transition time? */ ret = cpufreq_generic_init(policy, freq_table, 300 * 1000); if (ret) { - clk_disable_unprepare(cpu_clk); - clk_disable_unprepare(emc_clk); + clk_disable_unprepare(cpufreq->cpu_clk); return ret; } - policy->clk = cpu_clk; + policy->clk = cpufreq->cpu_clk; policy->suspend_freq = freq_table[0].frequency; return 0; } static int tegra_cpu_exit(struct cpufreq_policy *policy) { - clk_disable_unprepare(cpu_clk); - clk_disable_unprepare(emc_clk); + struct tegra20_cpufreq *cpufreq = cpufreq_get_driver_data(); + + clk_disable_unprepare(cpufreq->cpu_clk); return 0; } -static struct cpufreq_driver tegra_cpufreq_driver = { - .flags = CPUFREQ_NEED_INITIAL_FREQ_CHECK, - .verify = cpufreq_generic_frequency_table_verify, - .get_intermediate = tegra_get_intermediate, - .target_intermediate = tegra_target_intermediate, - .target_index = tegra_target, - .get = cpufreq_generic_get, - .init = tegra_cpu_init, - .exit = tegra_cpu_exit, - .name = "tegra", - .attr = cpufreq_generic_attr, - .suspend = cpufreq_generic_suspend, -}; - -static int __init tegra_cpufreq_init(void) +static int tegra20_cpufreq_probe(struct platform_device *pdev) { - cpu_clk = clk_get_sys(NULL, "cclk"); - if (IS_ERR(cpu_clk)) - return PTR_ERR(cpu_clk); - - pll_x_clk = clk_get_sys(NULL, "pll_x"); - if (IS_ERR(pll_x_clk)) - return PTR_ERR(pll_x_clk); - - pll_p_clk = clk_get_sys(NULL, "pll_p"); - if (IS_ERR(pll_p_clk)) - return PTR_ERR(pll_p_clk); - - emc_clk = clk_get_sys("cpu", "emc"); - if (IS_ERR(emc_clk)) { - clk_put(cpu_clk); - return PTR_ERR(emc_clk); + struct tegra20_cpufreq *cpufreq; + int err; + + cpufreq = devm_kzalloc(&pdev->dev, sizeof(*cpufreq), GFP_KERNEL); + if (!cpufreq) + return -ENOMEM; + + cpufreq->cpu_clk = clk_get_sys(NULL, "cclk"); + if (IS_ERR(cpufreq->cpu_clk)) + return PTR_ERR(cpufreq->cpu_clk); + + cpufreq->pll_x_clk = clk_get_sys(NULL, "pll_x"); + if (IS_ERR(cpufreq->pll_x_clk)) { + err = PTR_ERR(cpufreq->pll_x_clk); + goto put_cpu; + } + + cpufreq->pll_p_clk = clk_get_sys(NULL, "pll_p"); + if (IS_ERR(cpufreq->pll_p_clk)) { + err = PTR_ERR(cpufreq->pll_p_clk); + goto put_pll_x; } - return cpufreq_register_driver(&tegra_cpufreq_driver); + cpufreq->dev = &pdev->dev; + cpufreq->driver.get = cpufreq_generic_get; + cpufreq->driver.attr = cpufreq_generic_attr; + cpufreq->driver.init = tegra_cpu_init; + cpufreq->driver.exit = tegra_cpu_exit; + cpufreq->driver.flags = CPUFREQ_NEED_INITIAL_FREQ_CHECK; + cpufreq->driver.verify = cpufreq_generic_frequency_table_verify; + cpufreq->driver.suspend = cpufreq_generic_suspend; + cpufreq->driver.driver_data = cpufreq; + cpufreq->driver.target_index = tegra_target; + cpufreq->driver.get_intermediate = tegra_get_intermediate; + cpufreq->driver.target_intermediate = tegra_target_intermediate; + snprintf(cpufreq->driver.name, CPUFREQ_NAME_LEN, "tegra"); + + err = cpufreq_register_driver(&cpufreq->driver); + if (err) + goto put_pll_p; + + platform_set_drvdata(pdev, cpufreq); + + return 0; + +put_pll_p: + clk_put(cpufreq->pll_p_clk); +put_pll_x: + clk_put(cpufreq->pll_x_clk); +put_cpu: + clk_put(cpufreq->cpu_clk); + + return err; } -static void __exit tegra_cpufreq_exit(void) +static int tegra20_cpufreq_remove(struct platform_device *pdev) { - cpufreq_unregister_driver(&tegra_cpufreq_driver); - clk_put(emc_clk); - clk_put(cpu_clk); + struct tegra20_cpufreq *cpufreq = platform_get_drvdata(pdev); + + cpufreq_unregister_driver(&cpufreq->driver); + + clk_put(cpufreq->pll_p_clk); + clk_put(cpufreq->pll_x_clk); + clk_put(cpufreq->cpu_clk); + + return 0; } +static struct platform_driver tegra20_cpufreq_driver = { + .probe = tegra20_cpufreq_probe, + .remove = tegra20_cpufreq_remove, + .driver = { + .name = "tegra20-cpufreq", + }, +}; +module_platform_driver(tegra20_cpufreq_driver); +MODULE_ALIAS("platform:tegra20-cpufreq"); MODULE_AUTHOR("Colin Cross "); -MODULE_DESCRIPTION("cpufreq driver for Nvidia Tegra2"); +MODULE_DESCRIPTION("NVIDIA Tegra20 cpufreq driver"); MODULE_LICENSE("GPL"); -module_init(tegra_cpufreq_init); -module_exit(tegra_cpufreq_exit); diff --git a/drivers/cpufreq/ti-cpufreq.c b/drivers/cpufreq/ti-cpufreq.c index 6ba709b6f095..3f0e2a14895a 100644 --- a/drivers/cpufreq/ti-cpufreq.c +++ b/drivers/cpufreq/ti-cpufreq.c @@ -217,7 +217,7 @@ static int ti_cpufreq_probe(struct platform_device *pdev) if (!match) return -ENODEV; - opp_data = kzalloc(sizeof(*opp_data), GFP_KERNEL); + opp_data = devm_kzalloc(&pdev->dev, sizeof(*opp_data), GFP_KERNEL); if (!opp_data) return -ENOMEM; @@ -226,8 +226,7 @@ static int ti_cpufreq_probe(struct platform_device *pdev) opp_data->cpu_dev = get_cpu_device(0); if (!opp_data->cpu_dev) { pr_err("%s: Failed to get device for CPU0\n", __func__); - ret = ENODEV; - goto free_opp_data; + return -ENODEV; } opp_data->opp_node = dev_pm_opp_of_get_opp_desc_node(opp_data->cpu_dev); @@ -285,8 +284,6 @@ register_cpufreq_dt: fail_put_node: of_node_put(opp_data->opp_node); -free_opp_data: - kfree(opp_data); return ret; } diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c index 1a8234e706bc..d29e4f041efe 100644 --- a/drivers/cpuidle/cpuidle-powernv.c +++ b/drivers/cpuidle/cpuidle-powernv.c @@ -43,9 +43,31 @@ struct stop_psscr_table { static struct stop_psscr_table stop_psscr_table[CPUIDLE_STATE_MAX] __read_mostly; -static u64 snooze_timeout __read_mostly; +static u64 default_snooze_timeout __read_mostly; static bool snooze_timeout_en __read_mostly; +static u64 get_snooze_timeout(struct cpuidle_device *dev, + struct cpuidle_driver *drv, + int index) +{ + int i; + + if (unlikely(!snooze_timeout_en)) + return default_snooze_timeout; + + for (i = index + 1; i < drv->state_count; i++) { + struct cpuidle_state *s = &drv->states[i]; + struct cpuidle_state_usage *su = &dev->states_usage[i]; + + if (s->disabled || su->disable) + continue; + + return s->target_residency * tb_ticks_per_usec; + } + + return default_snooze_timeout; +} + static int snooze_loop(struct cpuidle_device *dev, struct cpuidle_driver *drv, int index) @@ -56,7 +78,7 @@ static int snooze_loop(struct cpuidle_device *dev, local_irq_enable(); - snooze_exit_time = get_tb() + snooze_timeout; + snooze_exit_time = get_tb() + get_snooze_timeout(dev, drv, index); ppc64_runlatch_off(); HMT_very_low(); while (!need_resched()) { @@ -465,11 +487,9 @@ static int powernv_idle_probe(void) cpuidle_state_table = powernv_states; /* Device tree can indicate more idle states */ max_idle_state = powernv_add_idle_states(); - if (max_idle_state > 1) { + default_snooze_timeout = TICK_USEC * tb_ticks_per_usec; + if (max_idle_state > 1) snooze_timeout_en = true; - snooze_timeout = powernv_states[1].target_residency * - tb_ticks_per_usec; - } } else return -ENODEV; diff --git a/drivers/cpuidle/governor.c b/drivers/cpuidle/governor.c index 5d359aff3cc5..9fed1b829292 100644 --- a/drivers/cpuidle/governor.c +++ b/drivers/cpuidle/governor.c @@ -8,8 +8,10 @@ * This code is licenced under the GPL. */ -#include +#include #include +#include +#include #include "cpuidle.h" @@ -93,3 +95,16 @@ int cpuidle_register_governor(struct cpuidle_governor *gov) return ret; } + +/** + * cpuidle_governor_latency_req - Compute a latency constraint for CPU + * @cpu: Target CPU + */ +int cpuidle_governor_latency_req(unsigned int cpu) +{ + int global_req = pm_qos_request(PM_QOS_CPU_DMA_LATENCY); + struct device *device = get_cpu_device(cpu); + int device_req = dev_pm_qos_raw_read_value(device); + + return device_req < global_req ? device_req : global_req; +} diff --git a/drivers/cpuidle/governors/ladder.c b/drivers/cpuidle/governors/ladder.c index b24883f85c99..704880a6612a 100644 --- a/drivers/cpuidle/governors/ladder.c +++ b/drivers/cpuidle/governors/ladder.c @@ -14,10 +14,8 @@ #include #include -#include #include #include -#include #include #include @@ -69,16 +67,10 @@ static int ladder_select_state(struct cpuidle_driver *drv, struct cpuidle_device *dev, bool *dummy) { struct ladder_device *ldev = this_cpu_ptr(&ladder_devices); - struct device *device = get_cpu_device(dev->cpu); struct ladder_device_state *last_state; int last_residency, last_idx = ldev->last_state_idx; int first_idx = drv->states[0].flags & CPUIDLE_FLAG_POLLING ? 1 : 0; - int latency_req = pm_qos_request(PM_QOS_CPU_DMA_LATENCY); - int resume_latency = dev_pm_qos_raw_read_value(device); - - if (resume_latency < latency_req && - resume_latency != PM_QOS_RESUME_LATENCY_NO_CONSTRAINT) - latency_req = resume_latency; + int latency_req = cpuidle_governor_latency_req(dev->cpu); /* Special case when user has set very strict latency requirement */ if (unlikely(latency_req == 0)) { diff --git a/drivers/cpuidle/governors/menu.c b/drivers/cpuidle/governors/menu.c index 1bfe03ceb236..1aef60d160eb 100644 --- a/drivers/cpuidle/governors/menu.c +++ b/drivers/cpuidle/governors/menu.c @@ -12,7 +12,6 @@ #include #include -#include #include #include #include @@ -21,7 +20,6 @@ #include #include #include -#include /* * Please note when changing the tuning values: @@ -286,15 +284,13 @@ static int menu_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, bool *stop_tick) { struct menu_device *data = this_cpu_ptr(&menu_devices); - struct device *device = get_cpu_device(dev->cpu); - int latency_req = pm_qos_request(PM_QOS_CPU_DMA_LATENCY); + int latency_req = cpuidle_governor_latency_req(dev->cpu); int i; int first_idx; int idx; unsigned int interactivity_req; unsigned int expected_interval; unsigned long nr_iowaiters, cpu_load; - int resume_latency = dev_pm_qos_raw_read_value(device); ktime_t delta_next; if (data->needs_update) { @@ -302,10 +298,6 @@ static int menu_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, data->needs_update = 0; } - if (resume_latency < latency_req && - resume_latency != PM_QOS_RESUME_LATENCY_NO_CONSTRAINT) - latency_req = resume_latency; - /* Special case when user has set very strict latency requirement */ if (unlikely(latency_req == 0)) { *stop_tick = false; diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig index d1ea1a07cecb..43cccf6aff61 100644 --- a/drivers/crypto/Kconfig +++ b/drivers/crypto/Kconfig @@ -302,6 +302,7 @@ config CRYPTO_DEV_PPC4XX select CRYPTO_AEAD select CRYPTO_AES select CRYPTO_CCM + select CRYPTO_CTR select CRYPTO_GCM select CRYPTO_BLKCIPHER help @@ -419,7 +420,7 @@ config CRYPTO_DEV_EXYNOS_RNG config CRYPTO_DEV_S5P tristate "Support for Samsung S5PV210/Exynos crypto accelerator" depends on ARCH_S5PV210 || ARCH_EXYNOS || COMPILE_TEST - depends on HAS_IOMEM && HAS_DMA + depends on HAS_IOMEM select CRYPTO_AES select CRYPTO_BLKCIPHER help @@ -466,7 +467,6 @@ endif # if CRYPTO_DEV_UX500 config CRYPTO_DEV_ATMEL_AUTHENC tristate "Support for Atmel IPSEC/SSL hw accelerator" - depends on HAS_DMA depends on ARCH_AT91 || COMPILE_TEST select CRYPTO_AUTHENC select CRYPTO_DEV_ATMEL_AES @@ -479,7 +479,6 @@ config CRYPTO_DEV_ATMEL_AUTHENC config CRYPTO_DEV_ATMEL_AES tristate "Support for Atmel AES hw accelerator" - depends on HAS_DMA depends on ARCH_AT91 || COMPILE_TEST select CRYPTO_AES select CRYPTO_AEAD @@ -494,7 +493,6 @@ config CRYPTO_DEV_ATMEL_AES config CRYPTO_DEV_ATMEL_TDES tristate "Support for Atmel DES/TDES hw accelerator" - depends on HAS_DMA depends on ARCH_AT91 || COMPILE_TEST select CRYPTO_DES select CRYPTO_BLKCIPHER @@ -508,7 +506,6 @@ config CRYPTO_DEV_ATMEL_TDES config CRYPTO_DEV_ATMEL_SHA tristate "Support for Atmel SHA hw accelerator" - depends on HAS_DMA depends on ARCH_AT91 || COMPILE_TEST select CRYPTO_HASH help @@ -574,7 +571,8 @@ config CRYPTO_DEV_CAVIUM_ZIP config CRYPTO_DEV_QCE tristate "Qualcomm crypto engine accelerator" - depends on (ARCH_QCOM || COMPILE_TEST) && HAS_DMA && HAS_IOMEM + depends on ARCH_QCOM || COMPILE_TEST + depends on HAS_IOMEM select CRYPTO_AES select CRYPTO_DES select CRYPTO_ECB @@ -598,7 +596,6 @@ source "drivers/crypto/vmx/Kconfig" config CRYPTO_DEV_IMGTEC_HASH tristate "Imagination Technologies hardware hash accelerator" depends on MIPS || COMPILE_TEST - depends on HAS_DMA select CRYPTO_MD5 select CRYPTO_SHA1 select CRYPTO_SHA256 @@ -650,7 +647,6 @@ config CRYPTO_DEV_ROCKCHIP config CRYPTO_DEV_MEDIATEK tristate "MediaTek's EIP97 Cryptographic Engine driver" - depends on HAS_DMA depends on (ARM && ARCH_MEDIATEK) || COMPILE_TEST select CRYPTO_AES select CRYPTO_AEAD @@ -688,9 +684,10 @@ source "drivers/crypto/stm32/Kconfig" config CRYPTO_DEV_SAFEXCEL tristate "Inside Secure's SafeXcel cryptographic engine driver" - depends on HAS_DMA && OF + depends on OF depends on (ARM64 && ARCH_MVEBU) || (COMPILE_TEST && 64BIT) select CRYPTO_AES + select CRYPTO_AUTHENC select CRYPTO_BLKCIPHER select CRYPTO_HASH select CRYPTO_HMAC @@ -706,7 +703,6 @@ config CRYPTO_DEV_SAFEXCEL config CRYPTO_DEV_ARTPEC6 tristate "Support for Axis ARTPEC-6/7 hardware crypto acceleration." depends on ARM && (ARCH_ARTPEC || COMPILE_TEST) - depends on HAS_DMA depends on OF select CRYPTO_AEAD select CRYPTO_AES diff --git a/drivers/crypto/amcc/crypto4xx_alg.c b/drivers/crypto/amcc/crypto4xx_alg.c index ea83d0bff0e9..f5c07498ea4f 100644 --- a/drivers/crypto/amcc/crypto4xx_alg.c +++ b/drivers/crypto/amcc/crypto4xx_alg.c @@ -31,6 +31,7 @@ #include #include #include +#include #include "crypto4xx_reg_def.h" #include "crypto4xx_core.h" #include "crypto4xx_sa.h" @@ -74,51 +75,57 @@ static void set_dynamic_sa_command_1(struct dynamic_sa_ctl *sa, u32 cm, sa->sa_command_1.bf.copy_hdr = cp_hdr; } -int crypto4xx_encrypt(struct ablkcipher_request *req) +static inline int crypto4xx_crypt(struct skcipher_request *req, + const unsigned int ivlen, bool decrypt) { - struct crypto4xx_ctx *ctx = crypto_tfm_ctx(req->base.tfm); - unsigned int ivlen = crypto_ablkcipher_ivsize( - crypto_ablkcipher_reqtfm(req)); - __le32 iv[ivlen]; + struct crypto_skcipher *cipher = crypto_skcipher_reqtfm(req); + struct crypto4xx_ctx *ctx = crypto_skcipher_ctx(cipher); + __le32 iv[AES_IV_SIZE]; if (ivlen) - crypto4xx_memcpy_to_le32(iv, req->info, ivlen); + crypto4xx_memcpy_to_le32(iv, req->iv, ivlen); return crypto4xx_build_pd(&req->base, ctx, req->src, req->dst, - req->nbytes, iv, ivlen, ctx->sa_out, ctx->sa_len, 0); + req->cryptlen, iv, ivlen, decrypt ? ctx->sa_in : ctx->sa_out, + ctx->sa_len, 0, NULL); } -int crypto4xx_decrypt(struct ablkcipher_request *req) +int crypto4xx_encrypt_noiv(struct skcipher_request *req) { - struct crypto4xx_ctx *ctx = crypto_tfm_ctx(req->base.tfm); - unsigned int ivlen = crypto_ablkcipher_ivsize( - crypto_ablkcipher_reqtfm(req)); - __le32 iv[ivlen]; + return crypto4xx_crypt(req, 0, false); +} - if (ivlen) - crypto4xx_memcpy_to_le32(iv, req->info, ivlen); +int crypto4xx_encrypt_iv(struct skcipher_request *req) +{ + return crypto4xx_crypt(req, AES_IV_SIZE, false); +} - return crypto4xx_build_pd(&req->base, ctx, req->src, req->dst, - req->nbytes, iv, ivlen, ctx->sa_in, ctx->sa_len, 0); +int crypto4xx_decrypt_noiv(struct skcipher_request *req) +{ + return crypto4xx_crypt(req, 0, true); +} + +int crypto4xx_decrypt_iv(struct skcipher_request *req) +{ + return crypto4xx_crypt(req, AES_IV_SIZE, true); } /** * AES Functions */ -static int crypto4xx_setkey_aes(struct crypto_ablkcipher *cipher, +static int crypto4xx_setkey_aes(struct crypto_skcipher *cipher, const u8 *key, unsigned int keylen, unsigned char cm, u8 fb) { - struct crypto_tfm *tfm = crypto_ablkcipher_tfm(cipher); - struct crypto4xx_ctx *ctx = crypto_tfm_ctx(tfm); + struct crypto4xx_ctx *ctx = crypto_skcipher_ctx(cipher); struct dynamic_sa_ctl *sa; int rc; if (keylen != AES_KEYSIZE_256 && keylen != AES_KEYSIZE_192 && keylen != AES_KEYSIZE_128) { - crypto_ablkcipher_set_flags(cipher, + crypto_skcipher_set_flags(cipher, CRYPTO_TFM_RES_BAD_KEY_LEN); return -EINVAL; } @@ -134,7 +141,8 @@ static int crypto4xx_setkey_aes(struct crypto_ablkcipher *cipher, /* Setup SA */ sa = ctx->sa_in; - set_dynamic_sa_command_0(sa, SA_NOT_SAVE_HASH, SA_NOT_SAVE_IV, + set_dynamic_sa_command_0(sa, SA_NOT_SAVE_HASH, (cm == CRYPTO_MODE_CBC ? + SA_SAVE_IV : SA_NOT_SAVE_IV), SA_LOAD_HASH_FROM_SA, SA_LOAD_IV_FROM_STATE, SA_NO_HEADER_PROC, SA_HASH_ALG_NULL, SA_CIPHER_ALG_AES, SA_PAD_TYPE_ZERO, @@ -158,39 +166,38 @@ static int crypto4xx_setkey_aes(struct crypto_ablkcipher *cipher, return 0; } -int crypto4xx_setkey_aes_cbc(struct crypto_ablkcipher *cipher, +int crypto4xx_setkey_aes_cbc(struct crypto_skcipher *cipher, const u8 *key, unsigned int keylen) { return crypto4xx_setkey_aes(cipher, key, keylen, CRYPTO_MODE_CBC, CRYPTO_FEEDBACK_MODE_NO_FB); } -int crypto4xx_setkey_aes_cfb(struct crypto_ablkcipher *cipher, +int crypto4xx_setkey_aes_cfb(struct crypto_skcipher *cipher, const u8 *key, unsigned int keylen) { return crypto4xx_setkey_aes(cipher, key, keylen, CRYPTO_MODE_CFB, CRYPTO_FEEDBACK_MODE_128BIT_CFB); } -int crypto4xx_setkey_aes_ecb(struct crypto_ablkcipher *cipher, +int crypto4xx_setkey_aes_ecb(struct crypto_skcipher *cipher, const u8 *key, unsigned int keylen) { return crypto4xx_setkey_aes(cipher, key, keylen, CRYPTO_MODE_ECB, CRYPTO_FEEDBACK_MODE_NO_FB); } -int crypto4xx_setkey_aes_ofb(struct crypto_ablkcipher *cipher, +int crypto4xx_setkey_aes_ofb(struct crypto_skcipher *cipher, const u8 *key, unsigned int keylen) { return crypto4xx_setkey_aes(cipher, key, keylen, CRYPTO_MODE_OFB, CRYPTO_FEEDBACK_MODE_64BIT_OFB); } -int crypto4xx_setkey_rfc3686(struct crypto_ablkcipher *cipher, +int crypto4xx_setkey_rfc3686(struct crypto_skcipher *cipher, const u8 *key, unsigned int keylen) { - struct crypto_tfm *tfm = crypto_ablkcipher_tfm(cipher); - struct crypto4xx_ctx *ctx = crypto_tfm_ctx(tfm); + struct crypto4xx_ctx *ctx = crypto_skcipher_ctx(cipher); int rc; rc = crypto4xx_setkey_aes(cipher, key, keylen - CTR_RFC3686_NONCE_SIZE, @@ -204,35 +211,117 @@ int crypto4xx_setkey_rfc3686(struct crypto_ablkcipher *cipher, return 0; } -int crypto4xx_rfc3686_encrypt(struct ablkcipher_request *req) +int crypto4xx_rfc3686_encrypt(struct skcipher_request *req) { - struct crypto4xx_ctx *ctx = crypto_tfm_ctx(req->base.tfm); + struct crypto_skcipher *cipher = crypto_skcipher_reqtfm(req); + struct crypto4xx_ctx *ctx = crypto_skcipher_ctx(cipher); __le32 iv[AES_IV_SIZE / 4] = { ctx->iv_nonce, - cpu_to_le32p((u32 *) req->info), - cpu_to_le32p((u32 *) (req->info + 4)), + cpu_to_le32p((u32 *) req->iv), + cpu_to_le32p((u32 *) (req->iv + 4)), cpu_to_le32(1) }; return crypto4xx_build_pd(&req->base, ctx, req->src, req->dst, - req->nbytes, iv, AES_IV_SIZE, - ctx->sa_out, ctx->sa_len, 0); + req->cryptlen, iv, AES_IV_SIZE, + ctx->sa_out, ctx->sa_len, 0, NULL); } -int crypto4xx_rfc3686_decrypt(struct ablkcipher_request *req) +int crypto4xx_rfc3686_decrypt(struct skcipher_request *req) { - struct crypto4xx_ctx *ctx = crypto_tfm_ctx(req->base.tfm); + struct crypto_skcipher *cipher = crypto_skcipher_reqtfm(req); + struct crypto4xx_ctx *ctx = crypto_skcipher_ctx(cipher); __le32 iv[AES_IV_SIZE / 4] = { ctx->iv_nonce, - cpu_to_le32p((u32 *) req->info), - cpu_to_le32p((u32 *) (req->info + 4)), + cpu_to_le32p((u32 *) req->iv), + cpu_to_le32p((u32 *) (req->iv + 4)), cpu_to_le32(1) }; return crypto4xx_build_pd(&req->base, ctx, req->src, req->dst, - req->nbytes, iv, AES_IV_SIZE, - ctx->sa_out, ctx->sa_len, 0); + req->cryptlen, iv, AES_IV_SIZE, + ctx->sa_out, ctx->sa_len, 0, NULL); +} + +static int +crypto4xx_ctr_crypt(struct skcipher_request *req, bool encrypt) +{ + struct crypto_skcipher *cipher = crypto_skcipher_reqtfm(req); + struct crypto4xx_ctx *ctx = crypto_skcipher_ctx(cipher); + size_t iv_len = crypto_skcipher_ivsize(cipher); + unsigned int counter = be32_to_cpup((__be32 *)(req->iv + iv_len - 4)); + unsigned int nblks = ALIGN(req->cryptlen, AES_BLOCK_SIZE) / + AES_BLOCK_SIZE; + + /* + * The hardware uses only the last 32-bits as the counter while the + * kernel tests (aes_ctr_enc_tv_template[4] for example) expect that + * the whole IV is a counter. So fallback if the counter is going to + * overlow. + */ + if (counter + nblks < counter) { + struct skcipher_request *subreq = skcipher_request_ctx(req); + int ret; + + skcipher_request_set_tfm(subreq, ctx->sw_cipher.cipher); + skcipher_request_set_callback(subreq, req->base.flags, + NULL, NULL); + skcipher_request_set_crypt(subreq, req->src, req->dst, + req->cryptlen, req->iv); + ret = encrypt ? crypto_skcipher_encrypt(subreq) + : crypto_skcipher_decrypt(subreq); + skcipher_request_zero(subreq); + return ret; + } + + return encrypt ? crypto4xx_encrypt_iv(req) + : crypto4xx_decrypt_iv(req); +} + +static int crypto4xx_sk_setup_fallback(struct crypto4xx_ctx *ctx, + struct crypto_skcipher *cipher, + const u8 *key, + unsigned int keylen) +{ + int rc; + + crypto_skcipher_clear_flags(ctx->sw_cipher.cipher, + CRYPTO_TFM_REQ_MASK); + crypto_skcipher_set_flags(ctx->sw_cipher.cipher, + crypto_skcipher_get_flags(cipher) & CRYPTO_TFM_REQ_MASK); + rc = crypto_skcipher_setkey(ctx->sw_cipher.cipher, key, keylen); + crypto_skcipher_clear_flags(cipher, CRYPTO_TFM_RES_MASK); + crypto_skcipher_set_flags(cipher, + crypto_skcipher_get_flags(ctx->sw_cipher.cipher) & + CRYPTO_TFM_RES_MASK); + + return rc; +} + +int crypto4xx_setkey_aes_ctr(struct crypto_skcipher *cipher, + const u8 *key, unsigned int keylen) +{ + struct crypto4xx_ctx *ctx = crypto_skcipher_ctx(cipher); + int rc; + + rc = crypto4xx_sk_setup_fallback(ctx, cipher, key, keylen); + if (rc) + return rc; + + return crypto4xx_setkey_aes(cipher, key, keylen, + CRYPTO_MODE_CTR, CRYPTO_FEEDBACK_MODE_NO_FB); +} + +int crypto4xx_encrypt_ctr(struct skcipher_request *req) +{ + return crypto4xx_ctr_crypt(req, true); +} + +int crypto4xx_decrypt_ctr(struct skcipher_request *req) +{ + return crypto4xx_ctr_crypt(req, false); } static inline bool crypto4xx_aead_need_fallback(struct aead_request *req, + unsigned int len, bool is_ccm, bool decrypt) { struct crypto_aead *aead = crypto_aead_reqtfm(req); @@ -242,14 +331,14 @@ static inline bool crypto4xx_aead_need_fallback(struct aead_request *req, return true; /* - * hardware does not handle cases where cryptlen - * is less than a block + * hardware does not handle cases where plaintext + * is less than a block. */ - if (req->cryptlen < AES_BLOCK_SIZE) + if (len < AES_BLOCK_SIZE) return true; - /* assoc len needs to be a multiple of 4 */ - if (req->assoclen & 0x3) + /* assoc len needs to be a multiple of 4 and <= 1020 */ + if (req->assoclen & 0x3 || req->assoclen > 1020) return true; /* CCM supports only counter field length of 2 and 4 bytes */ @@ -262,13 +351,7 @@ static inline bool crypto4xx_aead_need_fallback(struct aead_request *req, static int crypto4xx_aead_fallback(struct aead_request *req, struct crypto4xx_ctx *ctx, bool do_decrypt) { - char aead_req_data[sizeof(struct aead_request) + - crypto_aead_reqsize(ctx->sw_cipher.aead)] - __aligned(__alignof__(struct aead_request)); - - struct aead_request *subreq = (void *) aead_req_data; - - memset(subreq, 0, sizeof(aead_req_data)); + struct aead_request *subreq = aead_request_ctx(req); aead_request_set_tfm(subreq, ctx->sw_cipher.aead); aead_request_set_callback(subreq, req->base.flags, @@ -280,10 +363,10 @@ static int crypto4xx_aead_fallback(struct aead_request *req, crypto_aead_encrypt(subreq); } -static int crypto4xx_setup_fallback(struct crypto4xx_ctx *ctx, - struct crypto_aead *cipher, - const u8 *key, - unsigned int keylen) +static int crypto4xx_aead_setup_fallback(struct crypto4xx_ctx *ctx, + struct crypto_aead *cipher, + const u8 *key, + unsigned int keylen) { int rc; @@ -311,7 +394,7 @@ int crypto4xx_setkey_aes_ccm(struct crypto_aead *cipher, const u8 *key, struct dynamic_sa_ctl *sa; int rc = 0; - rc = crypto4xx_setup_fallback(ctx, cipher, key, keylen); + rc = crypto4xx_aead_setup_fallback(ctx, cipher, key, keylen); if (rc) return rc; @@ -366,19 +449,20 @@ int crypto4xx_setkey_aes_ccm(struct crypto_aead *cipher, const u8 *key, static int crypto4xx_crypt_aes_ccm(struct aead_request *req, bool decrypt) { struct crypto4xx_ctx *ctx = crypto_tfm_ctx(req->base.tfm); + struct crypto4xx_aead_reqctx *rctx = aead_request_ctx(req); struct crypto_aead *aead = crypto_aead_reqtfm(req); - unsigned int len = req->cryptlen; __le32 iv[16]; - u32 tmp_sa[ctx->sa_len * 4]; + u32 tmp_sa[SA_AES128_CCM_LEN + 4]; struct dynamic_sa_ctl *sa = (struct dynamic_sa_ctl *)tmp_sa; - - if (crypto4xx_aead_need_fallback(req, true, decrypt)) - return crypto4xx_aead_fallback(req, ctx, decrypt); + unsigned int len = req->cryptlen; if (decrypt) len -= crypto_aead_authsize(aead); - memcpy(tmp_sa, decrypt ? ctx->sa_in : ctx->sa_out, sizeof(tmp_sa)); + if (crypto4xx_aead_need_fallback(req, len, true, decrypt)) + return crypto4xx_aead_fallback(req, ctx, decrypt); + + memcpy(tmp_sa, decrypt ? ctx->sa_in : ctx->sa_out, ctx->sa_len * 4); sa->sa_command_0.bf.digest_len = crypto_aead_authsize(aead) >> 2; if (req->iv[0] == 1) { @@ -391,7 +475,7 @@ static int crypto4xx_crypt_aes_ccm(struct aead_request *req, bool decrypt) return crypto4xx_build_pd(&req->base, ctx, req->src, req->dst, len, iv, sizeof(iv), - sa, ctx->sa_len, req->assoclen); + sa, ctx->sa_len, req->assoclen, rctx->dst); } int crypto4xx_encrypt_aes_ccm(struct aead_request *req) @@ -470,7 +554,7 @@ int crypto4xx_setkey_aes_gcm(struct crypto_aead *cipher, return -EINVAL; } - rc = crypto4xx_setup_fallback(ctx, cipher, key, keylen); + rc = crypto4xx_aead_setup_fallback(ctx, cipher, key, keylen); if (rc) return rc; @@ -523,22 +607,23 @@ static inline int crypto4xx_crypt_aes_gcm(struct aead_request *req, bool decrypt) { struct crypto4xx_ctx *ctx = crypto_tfm_ctx(req->base.tfm); - unsigned int len = req->cryptlen; + struct crypto4xx_aead_reqctx *rctx = aead_request_ctx(req); __le32 iv[4]; + unsigned int len = req->cryptlen; + + if (decrypt) + len -= crypto_aead_authsize(crypto_aead_reqtfm(req)); - if (crypto4xx_aead_need_fallback(req, false, decrypt)) + if (crypto4xx_aead_need_fallback(req, len, false, decrypt)) return crypto4xx_aead_fallback(req, ctx, decrypt); crypto4xx_memcpy_to_le32(iv, req->iv, GCM_AES_IV_SIZE); iv[3] = cpu_to_le32(1); - if (decrypt) - len -= crypto_aead_authsize(crypto_aead_reqtfm(req)); - return crypto4xx_build_pd(&req->base, ctx, req->src, req->dst, len, iv, sizeof(iv), decrypt ? ctx->sa_in : ctx->sa_out, - ctx->sa_len, req->assoclen); + ctx->sa_len, req->assoclen, rctx->dst); } int crypto4xx_encrypt_aes_gcm(struct aead_request *req) @@ -623,7 +708,7 @@ int crypto4xx_hash_update(struct ahash_request *req) return crypto4xx_build_pd(&req->base, ctx, req->src, &dst, req->nbytes, NULL, 0, ctx->sa_in, - ctx->sa_len, 0); + ctx->sa_len, 0, NULL); } int crypto4xx_hash_final(struct ahash_request *req) @@ -642,7 +727,7 @@ int crypto4xx_hash_digest(struct ahash_request *req) return crypto4xx_build_pd(&req->base, ctx, req->src, &dst, req->nbytes, NULL, 0, ctx->sa_in, - ctx->sa_len, 0); + ctx->sa_len, 0, NULL); } /** diff --git a/drivers/crypto/amcc/crypto4xx_core.c b/drivers/crypto/amcc/crypto4xx_core.c index 76f459ad2821..05981ccd9901 100644 --- a/drivers/crypto/amcc/crypto4xx_core.c +++ b/drivers/crypto/amcc/crypto4xx_core.c @@ -41,6 +41,7 @@ #include #include #include +#include #include #include #include "crypto4xx_reg_def.h" @@ -140,11 +141,11 @@ static void crypto4xx_hw_init(struct crypto4xx_device *dev) int crypto4xx_alloc_sa(struct crypto4xx_ctx *ctx, u32 size) { - ctx->sa_in = kzalloc(size * 4, GFP_ATOMIC); + ctx->sa_in = kcalloc(size, 4, GFP_ATOMIC); if (ctx->sa_in == NULL) return -ENOMEM; - ctx->sa_out = kzalloc(size * 4, GFP_ATOMIC); + ctx->sa_out = kcalloc(size, 4, GFP_ATOMIC); if (ctx->sa_out == NULL) { kfree(ctx->sa_in); ctx->sa_in = NULL; @@ -179,8 +180,8 @@ static u32 crypto4xx_build_pdr(struct crypto4xx_device *dev) if (!dev->pdr) return -ENOMEM; - dev->pdr_uinfo = kzalloc(sizeof(struct pd_uinfo) * PPC4XX_NUM_PD, - GFP_KERNEL); + dev->pdr_uinfo = kcalloc(PPC4XX_NUM_PD, sizeof(struct pd_uinfo), + GFP_KERNEL); if (!dev->pdr_uinfo) { dma_free_coherent(dev->core_dev->device, sizeof(struct ce_pd) * PPC4XX_NUM_PD, @@ -526,31 +527,38 @@ static void crypto4xx_ret_sg_desc(struct crypto4xx_device *dev, } } -static void crypto4xx_ablkcipher_done(struct crypto4xx_device *dev, +static void crypto4xx_cipher_done(struct crypto4xx_device *dev, struct pd_uinfo *pd_uinfo, struct ce_pd *pd) { - struct crypto4xx_ctx *ctx; - struct ablkcipher_request *ablk_req; + struct skcipher_request *req; struct scatterlist *dst; dma_addr_t addr; - ablk_req = ablkcipher_request_cast(pd_uinfo->async_req); - ctx = crypto_tfm_ctx(ablk_req->base.tfm); + req = skcipher_request_cast(pd_uinfo->async_req); if (pd_uinfo->using_sd) { - crypto4xx_copy_pkt_to_dst(dev, pd, pd_uinfo, ablk_req->nbytes, - ablk_req->dst); + crypto4xx_copy_pkt_to_dst(dev, pd, pd_uinfo, + req->cryptlen, req->dst); } else { dst = pd_uinfo->dest_va; addr = dma_map_page(dev->core_dev->device, sg_page(dst), dst->offset, dst->length, DMA_FROM_DEVICE); } + + if (pd_uinfo->sa_va->sa_command_0.bf.save_iv == SA_SAVE_IV) { + struct crypto_skcipher *skcipher = crypto_skcipher_reqtfm(req); + + crypto4xx_memcpy_from_le32((u32 *)req->iv, + pd_uinfo->sr_va->save_iv, + crypto_skcipher_ivsize(skcipher)); + } + crypto4xx_ret_sg_desc(dev, pd_uinfo); if (pd_uinfo->state & PD_ENTRY_BUSY) - ablkcipher_request_complete(ablk_req, -EINPROGRESS); - ablkcipher_request_complete(ablk_req, 0); + skcipher_request_complete(req, -EINPROGRESS); + skcipher_request_complete(req, 0); } static void crypto4xx_ahash_done(struct crypto4xx_device *dev, @@ -580,7 +588,7 @@ static void crypto4xx_aead_done(struct crypto4xx_device *dev, struct scatterlist *dst = pd_uinfo->dest_va; size_t cp_len = crypto_aead_authsize( crypto_aead_reqtfm(aead_req)); - u32 icv[cp_len]; + u32 icv[AES_BLOCK_SIZE]; int err = 0; if (pd_uinfo->using_sd) { @@ -595,7 +603,7 @@ static void crypto4xx_aead_done(struct crypto4xx_device *dev, if (pd_uinfo->sa_va->sa_command_0.bf.dir == DIR_OUTBOUND) { /* append icv at the end */ crypto4xx_memcpy_from_le32(icv, pd_uinfo->sr_va->save_digest, - cp_len); + sizeof(icv)); scatterwalk_map_and_copy(icv, dst, aead_req->cryptlen, cp_len, 1); @@ -605,7 +613,7 @@ static void crypto4xx_aead_done(struct crypto4xx_device *dev, aead_req->assoclen + aead_req->cryptlen - cp_len, cp_len, 0); - crypto4xx_memcpy_from_le32(icv, icv, cp_len); + crypto4xx_memcpy_from_le32(icv, icv, sizeof(icv)); if (crypto_memneq(icv, pd_uinfo->sr_va->save_digest, cp_len)) err = -EBADMSG; @@ -641,8 +649,8 @@ static void crypto4xx_pd_done(struct crypto4xx_device *dev, u32 idx) struct pd_uinfo *pd_uinfo = &dev->pdr_uinfo[idx]; switch (crypto_tfm_alg_type(pd_uinfo->async_req->tfm)) { - case CRYPTO_ALG_TYPE_ABLKCIPHER: - crypto4xx_ablkcipher_done(dev, pd_uinfo, pd); + case CRYPTO_ALG_TYPE_SKCIPHER: + crypto4xx_cipher_done(dev, pd_uinfo, pd); break; case CRYPTO_ALG_TYPE_AEAD: crypto4xx_aead_done(dev, pd_uinfo, pd); @@ -687,9 +695,9 @@ int crypto4xx_build_pd(struct crypto_async_request *req, const __le32 *iv, const u32 iv_len, const struct dynamic_sa_ctl *req_sa, const unsigned int sa_len, - const unsigned int assoclen) + const unsigned int assoclen, + struct scatterlist *_dst) { - struct scatterlist _dst[2]; struct crypto4xx_device *dev = ctx->dev; struct dynamic_sa_ctl *sa; struct ce_gd *gd; @@ -936,15 +944,27 @@ static void crypto4xx_ctx_init(struct crypto4xx_alg *amcc_alg, ctx->sa_len = 0; } -static int crypto4xx_ablk_init(struct crypto_tfm *tfm) +static int crypto4xx_sk_init(struct crypto_skcipher *sk) { - struct crypto_alg *alg = tfm->__crt_alg; + struct skcipher_alg *alg = crypto_skcipher_alg(sk); struct crypto4xx_alg *amcc_alg; - struct crypto4xx_ctx *ctx = crypto_tfm_ctx(tfm); + struct crypto4xx_ctx *ctx = crypto_skcipher_ctx(sk); + + if (alg->base.cra_flags & CRYPTO_ALG_NEED_FALLBACK) { + ctx->sw_cipher.cipher = + crypto_alloc_skcipher(alg->base.cra_name, 0, + CRYPTO_ALG_NEED_FALLBACK | + CRYPTO_ALG_ASYNC); + if (IS_ERR(ctx->sw_cipher.cipher)) + return PTR_ERR(ctx->sw_cipher.cipher); + + crypto_skcipher_set_reqsize(sk, + sizeof(struct skcipher_request) + 32 + + crypto_skcipher_reqsize(ctx->sw_cipher.cipher)); + } amcc_alg = container_of(alg, struct crypto4xx_alg, alg.u.cipher); crypto4xx_ctx_init(amcc_alg, ctx); - tfm->crt_ablkcipher.reqsize = sizeof(struct crypto4xx_ctx); return 0; } @@ -953,9 +973,13 @@ static void crypto4xx_common_exit(struct crypto4xx_ctx *ctx) crypto4xx_free_sa(ctx); } -static void crypto4xx_ablk_exit(struct crypto_tfm *tfm) +static void crypto4xx_sk_exit(struct crypto_skcipher *sk) { - crypto4xx_common_exit(crypto_tfm_ctx(tfm)); + struct crypto4xx_ctx *ctx = crypto_skcipher_ctx(sk); + + crypto4xx_common_exit(ctx); + if (ctx->sw_cipher.cipher) + crypto_free_skcipher(ctx->sw_cipher.cipher); } static int crypto4xx_aead_init(struct crypto_aead *tfm) @@ -972,9 +996,9 @@ static int crypto4xx_aead_init(struct crypto_aead *tfm) amcc_alg = container_of(alg, struct crypto4xx_alg, alg.u.aead); crypto4xx_ctx_init(amcc_alg, ctx); - crypto_aead_set_reqsize(tfm, sizeof(struct aead_request) + - max(sizeof(struct crypto4xx_ctx), 32 + - crypto_aead_reqsize(ctx->sw_cipher.aead))); + crypto_aead_set_reqsize(tfm, max(sizeof(struct aead_request) + 32 + + crypto_aead_reqsize(ctx->sw_cipher.aead), + sizeof(struct crypto4xx_aead_reqctx))); return 0; } @@ -1012,7 +1036,7 @@ static int crypto4xx_register_alg(struct crypto4xx_device *sec_dev, break; default: - rc = crypto_register_alg(&alg->alg.u.cipher); + rc = crypto_register_skcipher(&alg->alg.u.cipher); break; } @@ -1041,7 +1065,7 @@ static void crypto4xx_unregister_alg(struct crypto4xx_device *sec_dev) break; default: - crypto_unregister_alg(&alg->alg.u.cipher); + crypto_unregister_skcipher(&alg->alg.u.cipher); } kfree(alg); } @@ -1103,126 +1127,131 @@ static irqreturn_t crypto4xx_ce_interrupt_handler_revb(int irq, void *data) */ static struct crypto4xx_alg_common crypto4xx_alg[] = { /* Crypto AES modes */ - { .type = CRYPTO_ALG_TYPE_ABLKCIPHER, .u.cipher = { - .cra_name = "cbc(aes)", - .cra_driver_name = "cbc-aes-ppc4xx", - .cra_priority = CRYPTO4XX_CRYPTO_PRIORITY, - .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER | - CRYPTO_ALG_ASYNC | - CRYPTO_ALG_KERN_DRIVER_ONLY, - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto4xx_ctx), - .cra_type = &crypto_ablkcipher_type, - .cra_init = crypto4xx_ablk_init, - .cra_exit = crypto4xx_ablk_exit, - .cra_module = THIS_MODULE, - .cra_u = { - .ablkcipher = { - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .ivsize = AES_IV_SIZE, - .setkey = crypto4xx_setkey_aes_cbc, - .encrypt = crypto4xx_encrypt, - .decrypt = crypto4xx_decrypt, - } - } - }}, - { .type = CRYPTO_ALG_TYPE_ABLKCIPHER, .u.cipher = { - .cra_name = "cfb(aes)", - .cra_driver_name = "cfb-aes-ppc4xx", - .cra_priority = CRYPTO4XX_CRYPTO_PRIORITY, - .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER | - CRYPTO_ALG_ASYNC | - CRYPTO_ALG_KERN_DRIVER_ONLY, - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto4xx_ctx), - .cra_type = &crypto_ablkcipher_type, - .cra_init = crypto4xx_ablk_init, - .cra_exit = crypto4xx_ablk_exit, - .cra_module = THIS_MODULE, - .cra_u = { - .ablkcipher = { - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .ivsize = AES_IV_SIZE, - .setkey = crypto4xx_setkey_aes_cfb, - .encrypt = crypto4xx_encrypt, - .decrypt = crypto4xx_decrypt, - } - } + { .type = CRYPTO_ALG_TYPE_SKCIPHER, .u.cipher = { + .base = { + .cra_name = "cbc(aes)", + .cra_driver_name = "cbc-aes-ppc4xx", + .cra_priority = CRYPTO4XX_CRYPTO_PRIORITY, + .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | + CRYPTO_ALG_ASYNC | + CRYPTO_ALG_KERN_DRIVER_ONLY, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct crypto4xx_ctx), + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_IV_SIZE, + .setkey = crypto4xx_setkey_aes_cbc, + .encrypt = crypto4xx_encrypt_iv, + .decrypt = crypto4xx_decrypt_iv, + .init = crypto4xx_sk_init, + .exit = crypto4xx_sk_exit, } }, - { .type = CRYPTO_ALG_TYPE_ABLKCIPHER, .u.cipher = { - .cra_name = "rfc3686(ctr(aes))", - .cra_driver_name = "rfc3686-ctr-aes-ppc4xx", - .cra_priority = CRYPTO4XX_CRYPTO_PRIORITY, - .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER | - CRYPTO_ALG_ASYNC | - CRYPTO_ALG_KERN_DRIVER_ONLY, - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto4xx_ctx), - .cra_type = &crypto_ablkcipher_type, - .cra_init = crypto4xx_ablk_init, - .cra_exit = crypto4xx_ablk_exit, - .cra_module = THIS_MODULE, - .cra_u = { - .ablkcipher = { - .min_keysize = AES_MIN_KEY_SIZE + - CTR_RFC3686_NONCE_SIZE, - .max_keysize = AES_MAX_KEY_SIZE + - CTR_RFC3686_NONCE_SIZE, - .ivsize = CTR_RFC3686_IV_SIZE, - .setkey = crypto4xx_setkey_rfc3686, - .encrypt = crypto4xx_rfc3686_encrypt, - .decrypt = crypto4xx_rfc3686_decrypt, - } - } + { .type = CRYPTO_ALG_TYPE_SKCIPHER, .u.cipher = { + .base = { + .cra_name = "cfb(aes)", + .cra_driver_name = "cfb-aes-ppc4xx", + .cra_priority = CRYPTO4XX_CRYPTO_PRIORITY, + .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | + CRYPTO_ALG_ASYNC | + CRYPTO_ALG_KERN_DRIVER_ONLY, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct crypto4xx_ctx), + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_IV_SIZE, + .setkey = crypto4xx_setkey_aes_cfb, + .encrypt = crypto4xx_encrypt_iv, + .decrypt = crypto4xx_decrypt_iv, + .init = crypto4xx_sk_init, + .exit = crypto4xx_sk_exit, } }, - { .type = CRYPTO_ALG_TYPE_ABLKCIPHER, .u.cipher = { - .cra_name = "ecb(aes)", - .cra_driver_name = "ecb-aes-ppc4xx", - .cra_priority = CRYPTO4XX_CRYPTO_PRIORITY, - .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER | - CRYPTO_ALG_ASYNC | - CRYPTO_ALG_KERN_DRIVER_ONLY, - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto4xx_ctx), - .cra_type = &crypto_ablkcipher_type, - .cra_init = crypto4xx_ablk_init, - .cra_exit = crypto4xx_ablk_exit, - .cra_module = THIS_MODULE, - .cra_u = { - .ablkcipher = { - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .setkey = crypto4xx_setkey_aes_ecb, - .encrypt = crypto4xx_encrypt, - .decrypt = crypto4xx_decrypt, - } - } + { .type = CRYPTO_ALG_TYPE_SKCIPHER, .u.cipher = { + .base = { + .cra_name = "ctr(aes)", + .cra_driver_name = "ctr-aes-ppc4xx", + .cra_priority = CRYPTO4XX_CRYPTO_PRIORITY, + .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | + CRYPTO_ALG_NEED_FALLBACK | + CRYPTO_ALG_ASYNC | + CRYPTO_ALG_KERN_DRIVER_ONLY, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct crypto4xx_ctx), + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_IV_SIZE, + .setkey = crypto4xx_setkey_aes_ctr, + .encrypt = crypto4xx_encrypt_ctr, + .decrypt = crypto4xx_decrypt_ctr, + .init = crypto4xx_sk_init, + .exit = crypto4xx_sk_exit, } }, - { .type = CRYPTO_ALG_TYPE_ABLKCIPHER, .u.cipher = { - .cra_name = "ofb(aes)", - .cra_driver_name = "ofb-aes-ppc4xx", - .cra_priority = CRYPTO4XX_CRYPTO_PRIORITY, - .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER | - CRYPTO_ALG_ASYNC | - CRYPTO_ALG_KERN_DRIVER_ONLY, - .cra_blocksize = AES_BLOCK_SIZE, - .cra_ctxsize = sizeof(struct crypto4xx_ctx), - .cra_type = &crypto_ablkcipher_type, - .cra_init = crypto4xx_ablk_init, - .cra_exit = crypto4xx_ablk_exit, - .cra_module = THIS_MODULE, - .cra_u = { - .ablkcipher = { - .min_keysize = AES_MIN_KEY_SIZE, - .max_keysize = AES_MAX_KEY_SIZE, - .ivsize = AES_IV_SIZE, - .setkey = crypto4xx_setkey_aes_ofb, - .encrypt = crypto4xx_encrypt, - .decrypt = crypto4xx_decrypt, - } - } + { .type = CRYPTO_ALG_TYPE_SKCIPHER, .u.cipher = { + .base = { + .cra_name = "rfc3686(ctr(aes))", + .cra_driver_name = "rfc3686-ctr-aes-ppc4xx", + .cra_priority = CRYPTO4XX_CRYPTO_PRIORITY, + .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | + CRYPTO_ALG_ASYNC | + CRYPTO_ALG_KERN_DRIVER_ONLY, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct crypto4xx_ctx), + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE + CTR_RFC3686_NONCE_SIZE, + .max_keysize = AES_MAX_KEY_SIZE + CTR_RFC3686_NONCE_SIZE, + .ivsize = CTR_RFC3686_IV_SIZE, + .setkey = crypto4xx_setkey_rfc3686, + .encrypt = crypto4xx_rfc3686_encrypt, + .decrypt = crypto4xx_rfc3686_decrypt, + .init = crypto4xx_sk_init, + .exit = crypto4xx_sk_exit, + } }, + { .type = CRYPTO_ALG_TYPE_SKCIPHER, .u.cipher = { + .base = { + .cra_name = "ecb(aes)", + .cra_driver_name = "ecb-aes-ppc4xx", + .cra_priority = CRYPTO4XX_CRYPTO_PRIORITY, + .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | + CRYPTO_ALG_ASYNC | + CRYPTO_ALG_KERN_DRIVER_ONLY, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct crypto4xx_ctx), + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .setkey = crypto4xx_setkey_aes_ecb, + .encrypt = crypto4xx_encrypt_noiv, + .decrypt = crypto4xx_decrypt_noiv, + .init = crypto4xx_sk_init, + .exit = crypto4xx_sk_exit, + } }, + { .type = CRYPTO_ALG_TYPE_SKCIPHER, .u.cipher = { + .base = { + .cra_name = "ofb(aes)", + .cra_driver_name = "ofb-aes-ppc4xx", + .cra_priority = CRYPTO4XX_CRYPTO_PRIORITY, + .cra_flags = CRYPTO_ALG_TYPE_SKCIPHER | + CRYPTO_ALG_ASYNC | + CRYPTO_ALG_KERN_DRIVER_ONLY, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct crypto4xx_ctx), + .cra_module = THIS_MODULE, + }, + .min_keysize = AES_MIN_KEY_SIZE, + .max_keysize = AES_MAX_KEY_SIZE, + .ivsize = AES_IV_SIZE, + .setkey = crypto4xx_setkey_aes_ofb, + .encrypt = crypto4xx_encrypt_iv, + .decrypt = crypto4xx_decrypt_iv, + .init = crypto4xx_sk_init, + .exit = crypto4xx_sk_exit, } }, /* AEAD */ diff --git a/drivers/crypto/amcc/crypto4xx_core.h b/drivers/crypto/amcc/crypto4xx_core.h index 23b726da6534..e2ca56722f07 100644 --- a/drivers/crypto/amcc/crypto4xx_core.h +++ b/drivers/crypto/amcc/crypto4xx_core.h @@ -25,6 +25,7 @@ #include #include #include +#include #include "crypto4xx_reg_def.h" #include "crypto4xx_sa.h" @@ -127,14 +128,19 @@ struct crypto4xx_ctx { __le32 iv_nonce; u32 sa_len; union { + struct crypto_skcipher *cipher; struct crypto_aead *aead; } sw_cipher; }; +struct crypto4xx_aead_reqctx { + struct scatterlist dst[2]; +}; + struct crypto4xx_alg_common { u32 type; union { - struct crypto_alg cipher; + struct skcipher_alg cipher; struct ahash_alg hash; struct aead_alg aead; } u; @@ -157,21 +163,28 @@ int crypto4xx_build_pd(struct crypto_async_request *req, const __le32 *iv, const u32 iv_len, const struct dynamic_sa_ctl *sa, const unsigned int sa_len, - const unsigned int assoclen); -int crypto4xx_setkey_aes_cbc(struct crypto_ablkcipher *cipher, + const unsigned int assoclen, + struct scatterlist *dst_tmp); +int crypto4xx_setkey_aes_cbc(struct crypto_skcipher *cipher, + const u8 *key, unsigned int keylen); +int crypto4xx_setkey_aes_cfb(struct crypto_skcipher *cipher, const u8 *key, unsigned int keylen); -int crypto4xx_setkey_aes_cfb(struct crypto_ablkcipher *cipher, +int crypto4xx_setkey_aes_ctr(struct crypto_skcipher *cipher, const u8 *key, unsigned int keylen); -int crypto4xx_setkey_aes_ecb(struct crypto_ablkcipher *cipher, +int crypto4xx_setkey_aes_ecb(struct crypto_skcipher *cipher, const u8 *key, unsigned int keylen); -int crypto4xx_setkey_aes_ofb(struct crypto_ablkcipher *cipher, +int crypto4xx_setkey_aes_ofb(struct crypto_skcipher *cipher, const u8 *key, unsigned int keylen); -int crypto4xx_setkey_rfc3686(struct crypto_ablkcipher *cipher, +int crypto4xx_setkey_rfc3686(struct crypto_skcipher *cipher, const u8 *key, unsigned int keylen); -int crypto4xx_encrypt(struct ablkcipher_request *req); -int crypto4xx_decrypt(struct ablkcipher_request *req); -int crypto4xx_rfc3686_encrypt(struct ablkcipher_request *req); -int crypto4xx_rfc3686_decrypt(struct ablkcipher_request *req); +int crypto4xx_encrypt_ctr(struct skcipher_request *req); +int crypto4xx_decrypt_ctr(struct skcipher_request *req); +int crypto4xx_encrypt_iv(struct skcipher_request *req); +int crypto4xx_decrypt_iv(struct skcipher_request *req); +int crypto4xx_encrypt_noiv(struct skcipher_request *req); +int crypto4xx_decrypt_noiv(struct skcipher_request *req); +int crypto4xx_rfc3686_encrypt(struct skcipher_request *req); +int crypto4xx_rfc3686_decrypt(struct skcipher_request *req); int crypto4xx_sha1_alg_init(struct crypto_tfm *tfm); int crypto4xx_hash_digest(struct ahash_request *req); int crypto4xx_hash_final(struct ahash_request *req); diff --git a/drivers/crypto/caam/caamalg.c b/drivers/crypto/caam/caamalg.c index 7207a535942d..d67667970f7e 100644 --- a/drivers/crypto/caam/caamalg.c +++ b/drivers/crypto/caam/caamalg.c @@ -769,15 +769,18 @@ struct aead_edesc { * @src_nents: number of segments in input s/w scatterlist * @dst_nents: number of segments in output s/w scatterlist * @iv_dma: dma address of iv for checking continuity and link table + * @iv_dir: DMA mapping direction for IV * @sec4_sg_bytes: length of dma mapped sec4_sg space * @sec4_sg_dma: bus physical mapped address of h/w link table * @sec4_sg: pointer to h/w link table * @hw_desc: the h/w job descriptor followed by any referenced link tables + * and IV */ struct ablkcipher_edesc { int src_nents; int dst_nents; dma_addr_t iv_dma; + enum dma_data_direction iv_dir; int sec4_sg_bytes; dma_addr_t sec4_sg_dma; struct sec4_sg_entry *sec4_sg; @@ -787,7 +790,8 @@ struct ablkcipher_edesc { static void caam_unmap(struct device *dev, struct scatterlist *src, struct scatterlist *dst, int src_nents, int dst_nents, - dma_addr_t iv_dma, int ivsize, dma_addr_t sec4_sg_dma, + dma_addr_t iv_dma, int ivsize, + enum dma_data_direction iv_dir, dma_addr_t sec4_sg_dma, int sec4_sg_bytes) { if (dst != src) { @@ -799,7 +803,7 @@ static void caam_unmap(struct device *dev, struct scatterlist *src, } if (iv_dma) - dma_unmap_single(dev, iv_dma, ivsize, DMA_TO_DEVICE); + dma_unmap_single(dev, iv_dma, ivsize, iv_dir); if (sec4_sg_bytes) dma_unmap_single(dev, sec4_sg_dma, sec4_sg_bytes, DMA_TO_DEVICE); @@ -810,7 +814,7 @@ static void aead_unmap(struct device *dev, struct aead_request *req) { caam_unmap(dev, req->src, req->dst, - edesc->src_nents, edesc->dst_nents, 0, 0, + edesc->src_nents, edesc->dst_nents, 0, 0, DMA_NONE, edesc->sec4_sg_dma, edesc->sec4_sg_bytes); } @@ -823,7 +827,7 @@ static void ablkcipher_unmap(struct device *dev, caam_unmap(dev, req->src, req->dst, edesc->src_nents, edesc->dst_nents, - edesc->iv_dma, ivsize, + edesc->iv_dma, ivsize, edesc->iv_dir, edesc->sec4_sg_dma, edesc->sec4_sg_bytes); } @@ -912,6 +916,18 @@ static void ablkcipher_encrypt_done(struct device *jrdev, u32 *desc, u32 err, scatterwalk_map_and_copy(req->info, req->dst, req->nbytes - ivsize, ivsize, 0); + /* In case initial IV was generated, copy it in GIVCIPHER request */ + if (edesc->iv_dir == DMA_FROM_DEVICE) { + u8 *iv; + struct skcipher_givcrypt_request *greq; + + greq = container_of(req, struct skcipher_givcrypt_request, + creq); + iv = (u8 *)edesc->hw_desc + desc_bytes(edesc->hw_desc) + + edesc->sec4_sg_bytes; + memcpy(greq->giv, iv, ivsize); + } + kfree(edesc); ablkcipher_request_complete(req, err); @@ -922,10 +938,10 @@ static void ablkcipher_decrypt_done(struct device *jrdev, u32 *desc, u32 err, { struct ablkcipher_request *req = context; struct ablkcipher_edesc *edesc; +#ifdef DEBUG struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req); int ivsize = crypto_ablkcipher_ivsize(ablkcipher); -#ifdef DEBUG dev_err(jrdev, "%s %d: err 0x%x\n", __func__, __LINE__, err); #endif @@ -943,14 +959,6 @@ static void ablkcipher_decrypt_done(struct device *jrdev, u32 *desc, u32 err, edesc->dst_nents > 1 ? 100 : req->nbytes, 1); ablkcipher_unmap(jrdev, edesc, req); - - /* - * The crypto API expects us to set the IV (req->info) to the last - * ciphertext block. - */ - scatterwalk_map_and_copy(req->info, req->src, req->nbytes - ivsize, - ivsize, 0); - kfree(edesc); ablkcipher_request_complete(req, err); @@ -1099,15 +1107,14 @@ static void init_authenc_job(struct aead_request *req, */ static void init_ablkcipher_job(u32 *sh_desc, dma_addr_t ptr, struct ablkcipher_edesc *edesc, - struct ablkcipher_request *req, - bool iv_contig) + struct ablkcipher_request *req) { struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req); int ivsize = crypto_ablkcipher_ivsize(ablkcipher); u32 *desc = edesc->hw_desc; - u32 out_options = 0, in_options; - dma_addr_t dst_dma, src_dma; - int len, sec4_sg_index = 0; + u32 out_options = 0; + dma_addr_t dst_dma; + int len; #ifdef DEBUG print_hex_dump(KERN_ERR, "presciv@"__stringify(__LINE__)": ", @@ -1123,30 +1130,18 @@ static void init_ablkcipher_job(u32 *sh_desc, dma_addr_t ptr, len = desc_len(sh_desc); init_job_desc_shared(desc, ptr, len, HDR_SHARE_DEFER | HDR_REVERSE); - if (iv_contig) { - src_dma = edesc->iv_dma; - in_options = 0; - } else { - src_dma = edesc->sec4_sg_dma; - sec4_sg_index += edesc->src_nents + 1; - in_options = LDST_SGF; - } - append_seq_in_ptr(desc, src_dma, req->nbytes + ivsize, in_options); + append_seq_in_ptr(desc, edesc->sec4_sg_dma, req->nbytes + ivsize, + LDST_SGF); if (likely(req->src == req->dst)) { - if (edesc->src_nents == 1 && iv_contig) { - dst_dma = sg_dma_address(req->src); - } else { - dst_dma = edesc->sec4_sg_dma + - sizeof(struct sec4_sg_entry); - out_options = LDST_SGF; - } + dst_dma = edesc->sec4_sg_dma + sizeof(struct sec4_sg_entry); + out_options = LDST_SGF; } else { if (edesc->dst_nents == 1) { dst_dma = sg_dma_address(req->dst); } else { - dst_dma = edesc->sec4_sg_dma + - sec4_sg_index * sizeof(struct sec4_sg_entry); + dst_dma = edesc->sec4_sg_dma + (edesc->src_nents + 1) * + sizeof(struct sec4_sg_entry); out_options = LDST_SGF; } } @@ -1158,13 +1153,12 @@ static void init_ablkcipher_job(u32 *sh_desc, dma_addr_t ptr, */ static void init_ablkcipher_giv_job(u32 *sh_desc, dma_addr_t ptr, struct ablkcipher_edesc *edesc, - struct ablkcipher_request *req, - bool iv_contig) + struct ablkcipher_request *req) { struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req); int ivsize = crypto_ablkcipher_ivsize(ablkcipher); u32 *desc = edesc->hw_desc; - u32 out_options, in_options; + u32 in_options; dma_addr_t dst_dma, src_dma; int len, sec4_sg_index = 0; @@ -1190,15 +1184,9 @@ static void init_ablkcipher_giv_job(u32 *sh_desc, dma_addr_t ptr, } append_seq_in_ptr(desc, src_dma, req->nbytes, in_options); - if (iv_contig) { - dst_dma = edesc->iv_dma; - out_options = 0; - } else { - dst_dma = edesc->sec4_sg_dma + - sec4_sg_index * sizeof(struct sec4_sg_entry); - out_options = LDST_SGF; - } - append_seq_out_ptr(desc, dst_dma, req->nbytes + ivsize, out_options); + dst_dma = edesc->sec4_sg_dma + sec4_sg_index * + sizeof(struct sec4_sg_entry); + append_seq_out_ptr(desc, dst_dma, req->nbytes + ivsize, LDST_SGF); } /* @@ -1287,7 +1275,7 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req, GFP_DMA | flags); if (!edesc) { caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, 0, - 0, 0, 0); + 0, DMA_NONE, 0, 0); return ERR_PTR(-ENOMEM); } @@ -1491,8 +1479,7 @@ static int aead_decrypt(struct aead_request *req) * allocate and map the ablkcipher extended descriptor for ablkcipher */ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request - *req, int desc_bytes, - bool *iv_contig_out) + *req, int desc_bytes) { struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req); struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher); @@ -1501,8 +1488,8 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request GFP_KERNEL : GFP_ATOMIC; int src_nents, mapped_src_nents, dst_nents = 0, mapped_dst_nents = 0; struct ablkcipher_edesc *edesc; - dma_addr_t iv_dma = 0; - bool in_contig; + dma_addr_t iv_dma; + u8 *iv; int ivsize = crypto_ablkcipher_ivsize(ablkcipher); int dst_sg_idx, sec4_sg_ents, sec4_sg_bytes; @@ -1546,33 +1533,20 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request } } - iv_dma = dma_map_single(jrdev, req->info, ivsize, DMA_TO_DEVICE); - if (dma_mapping_error(jrdev, iv_dma)) { - dev_err(jrdev, "unable to map IV\n"); - caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, 0, - 0, 0, 0); - return ERR_PTR(-ENOMEM); - } - - if (mapped_src_nents == 1 && - iv_dma + ivsize == sg_dma_address(req->src)) { - in_contig = true; - sec4_sg_ents = 0; - } else { - in_contig = false; - sec4_sg_ents = 1 + mapped_src_nents; - } + sec4_sg_ents = 1 + mapped_src_nents; dst_sg_idx = sec4_sg_ents; sec4_sg_ents += mapped_dst_nents > 1 ? mapped_dst_nents : 0; sec4_sg_bytes = sec4_sg_ents * sizeof(struct sec4_sg_entry); - /* allocate space for base edesc and hw desc commands, link tables */ - edesc = kzalloc(sizeof(*edesc) + desc_bytes + sec4_sg_bytes, + /* + * allocate space for base edesc and hw desc commands, link tables, IV + */ + edesc = kzalloc(sizeof(*edesc) + desc_bytes + sec4_sg_bytes + ivsize, GFP_DMA | flags); if (!edesc) { dev_err(jrdev, "could not allocate extended descriptor\n"); - caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, - iv_dma, ivsize, 0, 0); + caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, 0, + 0, DMA_NONE, 0, 0); return ERR_PTR(-ENOMEM); } @@ -1581,13 +1555,24 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request edesc->sec4_sg_bytes = sec4_sg_bytes; edesc->sec4_sg = (void *)edesc + sizeof(struct ablkcipher_edesc) + desc_bytes; + edesc->iv_dir = DMA_TO_DEVICE; - if (!in_contig) { - dma_to_sec4_sg_one(edesc->sec4_sg, iv_dma, ivsize, 0); - sg_to_sec4_sg_last(req->src, mapped_src_nents, - edesc->sec4_sg + 1, 0); + /* Make sure IV is located in a DMAable area */ + iv = (u8 *)edesc->hw_desc + desc_bytes + sec4_sg_bytes; + memcpy(iv, req->info, ivsize); + + iv_dma = dma_map_single(jrdev, iv, ivsize, DMA_TO_DEVICE); + if (dma_mapping_error(jrdev, iv_dma)) { + dev_err(jrdev, "unable to map IV\n"); + caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, 0, + 0, DMA_NONE, 0, 0); + kfree(edesc); + return ERR_PTR(-ENOMEM); } + dma_to_sec4_sg_one(edesc->sec4_sg, iv_dma, ivsize, 0); + sg_to_sec4_sg_last(req->src, mapped_src_nents, edesc->sec4_sg + 1, 0); + if (mapped_dst_nents > 1) { sg_to_sec4_sg_last(req->dst, mapped_dst_nents, edesc->sec4_sg + dst_sg_idx, 0); @@ -1598,7 +1583,7 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request if (dma_mapping_error(jrdev, edesc->sec4_sg_dma)) { dev_err(jrdev, "unable to map S/G table\n"); caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, - iv_dma, ivsize, 0, 0); + iv_dma, ivsize, DMA_TO_DEVICE, 0, 0); kfree(edesc); return ERR_PTR(-ENOMEM); } @@ -1611,7 +1596,6 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request sec4_sg_bytes, 1); #endif - *iv_contig_out = in_contig; return edesc; } @@ -1621,19 +1605,16 @@ static int ablkcipher_encrypt(struct ablkcipher_request *req) struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req); struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher); struct device *jrdev = ctx->jrdev; - bool iv_contig; u32 *desc; int ret = 0; /* allocate extended descriptor */ - edesc = ablkcipher_edesc_alloc(req, DESC_JOB_IO_LEN * - CAAM_CMD_SZ, &iv_contig); + edesc = ablkcipher_edesc_alloc(req, DESC_JOB_IO_LEN * CAAM_CMD_SZ); if (IS_ERR(edesc)) return PTR_ERR(edesc); /* Create and submit job descriptor*/ - init_ablkcipher_job(ctx->sh_desc_enc, - ctx->sh_desc_enc_dma, edesc, req, iv_contig); + init_ablkcipher_job(ctx->sh_desc_enc, ctx->sh_desc_enc_dma, edesc, req); #ifdef DEBUG print_hex_dump(KERN_ERR, "ablkcipher jobdesc@"__stringify(__LINE__)": ", DUMP_PREFIX_ADDRESS, 16, 4, edesc->hw_desc, @@ -1657,20 +1638,25 @@ static int ablkcipher_decrypt(struct ablkcipher_request *req) struct ablkcipher_edesc *edesc; struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req); struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher); + int ivsize = crypto_ablkcipher_ivsize(ablkcipher); struct device *jrdev = ctx->jrdev; - bool iv_contig; u32 *desc; int ret = 0; /* allocate extended descriptor */ - edesc = ablkcipher_edesc_alloc(req, DESC_JOB_IO_LEN * - CAAM_CMD_SZ, &iv_contig); + edesc = ablkcipher_edesc_alloc(req, DESC_JOB_IO_LEN * CAAM_CMD_SZ); if (IS_ERR(edesc)) return PTR_ERR(edesc); + /* + * The crypto API expects us to set the IV (req->info) to the last + * ciphertext block. + */ + scatterwalk_map_and_copy(req->info, req->src, req->nbytes - ivsize, + ivsize, 0); + /* Create and submit job descriptor*/ - init_ablkcipher_job(ctx->sh_desc_dec, - ctx->sh_desc_dec_dma, edesc, req, iv_contig); + init_ablkcipher_job(ctx->sh_desc_dec, ctx->sh_desc_dec_dma, edesc, req); desc = edesc->hw_desc; #ifdef DEBUG print_hex_dump(KERN_ERR, "ablkcipher jobdesc@"__stringify(__LINE__)": ", @@ -1695,8 +1681,7 @@ static int ablkcipher_decrypt(struct ablkcipher_request *req) */ static struct ablkcipher_edesc *ablkcipher_giv_edesc_alloc( struct skcipher_givcrypt_request *greq, - int desc_bytes, - bool *iv_contig_out) + int desc_bytes) { struct ablkcipher_request *req = &greq->creq; struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req); @@ -1706,8 +1691,8 @@ static struct ablkcipher_edesc *ablkcipher_giv_edesc_alloc( GFP_KERNEL : GFP_ATOMIC; int src_nents, mapped_src_nents, dst_nents, mapped_dst_nents; struct ablkcipher_edesc *edesc; - dma_addr_t iv_dma = 0; - bool out_contig; + dma_addr_t iv_dma; + u8 *iv; int ivsize = crypto_ablkcipher_ivsize(ablkcipher); int dst_sg_idx, sec4_sg_ents, sec4_sg_bytes; @@ -1752,36 +1737,20 @@ static struct ablkcipher_edesc *ablkcipher_giv_edesc_alloc( } } - /* - * Check if iv can be contiguous with source and destination. - * If so, include it. If not, create scatterlist. - */ - iv_dma = dma_map_single(jrdev, greq->giv, ivsize, DMA_TO_DEVICE); - if (dma_mapping_error(jrdev, iv_dma)) { - dev_err(jrdev, "unable to map IV\n"); - caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, 0, - 0, 0, 0); - return ERR_PTR(-ENOMEM); - } - sec4_sg_ents = mapped_src_nents > 1 ? mapped_src_nents : 0; dst_sg_idx = sec4_sg_ents; - if (mapped_dst_nents == 1 && - iv_dma + ivsize == sg_dma_address(req->dst)) { - out_contig = true; - } else { - out_contig = false; - sec4_sg_ents += 1 + mapped_dst_nents; - } + sec4_sg_ents += 1 + mapped_dst_nents; - /* allocate space for base edesc and hw desc commands, link tables */ + /* + * allocate space for base edesc and hw desc commands, link tables, IV + */ sec4_sg_bytes = sec4_sg_ents * sizeof(struct sec4_sg_entry); - edesc = kzalloc(sizeof(*edesc) + desc_bytes + sec4_sg_bytes, + edesc = kzalloc(sizeof(*edesc) + desc_bytes + sec4_sg_bytes + ivsize, GFP_DMA | flags); if (!edesc) { dev_err(jrdev, "could not allocate extended descriptor\n"); - caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, - iv_dma, ivsize, 0, 0); + caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, 0, + 0, DMA_NONE, 0, 0); return ERR_PTR(-ENOMEM); } @@ -1790,24 +1759,33 @@ static struct ablkcipher_edesc *ablkcipher_giv_edesc_alloc( edesc->sec4_sg_bytes = sec4_sg_bytes; edesc->sec4_sg = (void *)edesc + sizeof(struct ablkcipher_edesc) + desc_bytes; + edesc->iv_dir = DMA_FROM_DEVICE; + + /* Make sure IV is located in a DMAable area */ + iv = (u8 *)edesc->hw_desc + desc_bytes + sec4_sg_bytes; + iv_dma = dma_map_single(jrdev, iv, ivsize, DMA_FROM_DEVICE); + if (dma_mapping_error(jrdev, iv_dma)) { + dev_err(jrdev, "unable to map IV\n"); + caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, 0, + 0, DMA_NONE, 0, 0); + kfree(edesc); + return ERR_PTR(-ENOMEM); + } if (mapped_src_nents > 1) sg_to_sec4_sg_last(req->src, mapped_src_nents, edesc->sec4_sg, 0); - if (!out_contig) { - dma_to_sec4_sg_one(edesc->sec4_sg + dst_sg_idx, - iv_dma, ivsize, 0); - sg_to_sec4_sg_last(req->dst, mapped_dst_nents, - edesc->sec4_sg + dst_sg_idx + 1, 0); - } + dma_to_sec4_sg_one(edesc->sec4_sg + dst_sg_idx, iv_dma, ivsize, 0); + sg_to_sec4_sg_last(req->dst, mapped_dst_nents, edesc->sec4_sg + + dst_sg_idx + 1, 0); edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg, sec4_sg_bytes, DMA_TO_DEVICE); if (dma_mapping_error(jrdev, edesc->sec4_sg_dma)) { dev_err(jrdev, "unable to map S/G table\n"); caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, - iv_dma, ivsize, 0, 0); + iv_dma, ivsize, DMA_FROM_DEVICE, 0, 0); kfree(edesc); return ERR_PTR(-ENOMEM); } @@ -1820,7 +1798,6 @@ static struct ablkcipher_edesc *ablkcipher_giv_edesc_alloc( sec4_sg_bytes, 1); #endif - *iv_contig_out = out_contig; return edesc; } @@ -1831,19 +1808,17 @@ static int ablkcipher_givencrypt(struct skcipher_givcrypt_request *creq) struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req); struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher); struct device *jrdev = ctx->jrdev; - bool iv_contig = false; u32 *desc; int ret = 0; /* allocate extended descriptor */ - edesc = ablkcipher_giv_edesc_alloc(creq, DESC_JOB_IO_LEN * - CAAM_CMD_SZ, &iv_contig); + edesc = ablkcipher_giv_edesc_alloc(creq, DESC_JOB_IO_LEN * CAAM_CMD_SZ); if (IS_ERR(edesc)) return PTR_ERR(edesc); /* Create and submit job descriptor*/ init_ablkcipher_giv_job(ctx->sh_desc_givenc, ctx->sh_desc_givenc_dma, - edesc, req, iv_contig); + edesc, req); #ifdef DEBUG print_hex_dump(KERN_ERR, "ablkcipher jobdesc@" __stringify(__LINE__) ": ", diff --git a/drivers/crypto/caam/caamalg_desc.c b/drivers/crypto/caam/caamalg_desc.c index 8ae7a1be7dfd..a408edd84f34 100644 --- a/drivers/crypto/caam/caamalg_desc.c +++ b/drivers/crypto/caam/caamalg_desc.c @@ -1093,7 +1093,7 @@ void cnstr_shdsc_rfc4543_encap(u32 * const desc, struct alginfo *cdata, read_move_cmd = append_move(desc, MOVE_SRC_DESCBUF | MOVE_DEST_MATH3 | (0x6 << MOVE_LEN_SHIFT)); write_move_cmd = append_move(desc, MOVE_SRC_MATH3 | MOVE_DEST_DESCBUF | - (0x8 << MOVE_LEN_SHIFT)); + (0x8 << MOVE_LEN_SHIFT) | MOVE_WAITCOMP); /* Will read assoclen + cryptlen bytes */ append_math_sub(desc, VARSEQINLEN, SEQINLEN, REG0, CAAM_CMD_SZ); @@ -1178,7 +1178,7 @@ void cnstr_shdsc_rfc4543_decap(u32 * const desc, struct alginfo *cdata, read_move_cmd = append_move(desc, MOVE_SRC_DESCBUF | MOVE_DEST_MATH3 | (0x6 << MOVE_LEN_SHIFT)); write_move_cmd = append_move(desc, MOVE_SRC_MATH3 | MOVE_DEST_DESCBUF | - (0x8 << MOVE_LEN_SHIFT)); + (0x8 << MOVE_LEN_SHIFT) | MOVE_WAITCOMP); /* Will read assoclen + cryptlen bytes */ append_math_sub(desc, VARSEQINLEN, SEQOUTLEN, REG0, CAAM_CMD_SZ); diff --git a/drivers/crypto/caam/caamalg_qi.c b/drivers/crypto/caam/caamalg_qi.c index cacda0831390..6e61cc93c2b0 100644 --- a/drivers/crypto/caam/caamalg_qi.c +++ b/drivers/crypto/caam/caamalg_qi.c @@ -728,7 +728,7 @@ badkey: * @assoclen: associated data length, in CAAM endianness * @assoclen_dma: bus physical mapped address of req->assoclen * @drv_req: driver-specific request structure - * @sgt: the h/w link table + * @sgt: the h/w link table, followed by IV */ struct aead_edesc { int src_nents; @@ -739,9 +739,6 @@ struct aead_edesc { unsigned int assoclen; dma_addr_t assoclen_dma; struct caam_drv_req drv_req; -#define CAAM_QI_MAX_AEAD_SG \ - ((CAAM_QI_MEMCACHE_SIZE - offsetof(struct aead_edesc, sgt)) / \ - sizeof(struct qm_sg_entry)) struct qm_sg_entry sgt[0]; }; @@ -753,7 +750,7 @@ struct aead_edesc { * @qm_sg_bytes: length of dma mapped h/w link table * @qm_sg_dma: bus physical mapped address of h/w link table * @drv_req: driver-specific request structure - * @sgt: the h/w link table + * @sgt: the h/w link table, followed by IV */ struct ablkcipher_edesc { int src_nents; @@ -762,9 +759,6 @@ struct ablkcipher_edesc { int qm_sg_bytes; dma_addr_t qm_sg_dma; struct caam_drv_req drv_req; -#define CAAM_QI_MAX_ABLKCIPHER_SG \ - ((CAAM_QI_MEMCACHE_SIZE - offsetof(struct ablkcipher_edesc, sgt)) / \ - sizeof(struct qm_sg_entry)) struct qm_sg_entry sgt[0]; }; @@ -986,17 +980,8 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req, } } - if ((alg->caam.rfc3686 && encrypt) || !alg->caam.geniv) { + if ((alg->caam.rfc3686 && encrypt) || !alg->caam.geniv) ivsize = crypto_aead_ivsize(aead); - iv_dma = dma_map_single(qidev, req->iv, ivsize, DMA_TO_DEVICE); - if (dma_mapping_error(qidev, iv_dma)) { - dev_err(qidev, "unable to map IV\n"); - caam_unmap(qidev, req->src, req->dst, src_nents, - dst_nents, 0, 0, op_type, 0, 0); - qi_cache_free(edesc); - return ERR_PTR(-ENOMEM); - } - } /* * Create S/G table: req->assoclen, [IV,] req->src [, req->dst]. @@ -1004,16 +989,33 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req, */ qm_sg_ents = 1 + !!ivsize + mapped_src_nents + (mapped_dst_nents > 1 ? mapped_dst_nents : 0); - if (unlikely(qm_sg_ents > CAAM_QI_MAX_AEAD_SG)) { - dev_err(qidev, "Insufficient S/G entries: %d > %zu\n", - qm_sg_ents, CAAM_QI_MAX_AEAD_SG); - caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, - iv_dma, ivsize, op_type, 0, 0); + sg_table = &edesc->sgt[0]; + qm_sg_bytes = qm_sg_ents * sizeof(*sg_table); + if (unlikely(offsetof(struct aead_edesc, sgt) + qm_sg_bytes + ivsize > + CAAM_QI_MEMCACHE_SIZE)) { + dev_err(qidev, "No space for %d S/G entries and/or %dB IV\n", + qm_sg_ents, ivsize); + caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0, + 0, 0, 0, 0); qi_cache_free(edesc); return ERR_PTR(-ENOMEM); } - sg_table = &edesc->sgt[0]; - qm_sg_bytes = qm_sg_ents * sizeof(*sg_table); + + if (ivsize) { + u8 *iv = (u8 *)(sg_table + qm_sg_ents); + + /* Make sure IV is located in a DMAable area */ + memcpy(iv, req->iv, ivsize); + + iv_dma = dma_map_single(qidev, iv, ivsize, DMA_TO_DEVICE); + if (dma_mapping_error(qidev, iv_dma)) { + dev_err(qidev, "unable to map IV\n"); + caam_unmap(qidev, req->src, req->dst, src_nents, + dst_nents, 0, 0, 0, 0, 0); + qi_cache_free(edesc); + return ERR_PTR(-ENOMEM); + } + } edesc->src_nents = src_nents; edesc->dst_nents = dst_nents; @@ -1166,15 +1168,27 @@ static void ablkcipher_done(struct caam_drv_req *drv_req, u32 status) #endif ablkcipher_unmap(qidev, edesc, req); - qi_cache_free(edesc); + + /* In case initial IV was generated, copy it in GIVCIPHER request */ + if (edesc->drv_req.drv_ctx->op_type == GIVENCRYPT) { + u8 *iv; + struct skcipher_givcrypt_request *greq; + + greq = container_of(req, struct skcipher_givcrypt_request, + creq); + iv = (u8 *)edesc->sgt + edesc->qm_sg_bytes; + memcpy(greq->giv, iv, ivsize); + } /* * The crypto API expects us to set the IV (req->info) to the last * ciphertext block. This is used e.g. by the CTS mode. */ - scatterwalk_map_and_copy(req->info, req->dst, req->nbytes - ivsize, - ivsize, 0); + if (edesc->drv_req.drv_ctx->op_type != DECRYPT) + scatterwalk_map_and_copy(req->info, req->dst, req->nbytes - + ivsize, ivsize, 0); + qi_cache_free(edesc); ablkcipher_request_complete(req, status); } @@ -1189,9 +1203,9 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request int src_nents, mapped_src_nents, dst_nents = 0, mapped_dst_nents = 0; struct ablkcipher_edesc *edesc; dma_addr_t iv_dma; - bool in_contig; + u8 *iv; int ivsize = crypto_ablkcipher_ivsize(ablkcipher); - int dst_sg_idx, qm_sg_ents; + int dst_sg_idx, qm_sg_ents, qm_sg_bytes; struct qm_sg_entry *sg_table, *fd_sgt; struct caam_drv_ctx *drv_ctx; enum optype op_type = encrypt ? ENCRYPT : DECRYPT; @@ -1238,55 +1252,53 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request } } - iv_dma = dma_map_single(qidev, req->info, ivsize, DMA_TO_DEVICE); - if (dma_mapping_error(qidev, iv_dma)) { - dev_err(qidev, "unable to map IV\n"); - caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0, - 0, 0, 0, 0); - return ERR_PTR(-ENOMEM); - } - - if (mapped_src_nents == 1 && - iv_dma + ivsize == sg_dma_address(req->src)) { - in_contig = true; - qm_sg_ents = 0; - } else { - in_contig = false; - qm_sg_ents = 1 + mapped_src_nents; - } + qm_sg_ents = 1 + mapped_src_nents; dst_sg_idx = qm_sg_ents; qm_sg_ents += mapped_dst_nents > 1 ? mapped_dst_nents : 0; - if (unlikely(qm_sg_ents > CAAM_QI_MAX_ABLKCIPHER_SG)) { - dev_err(qidev, "Insufficient S/G entries: %d > %zu\n", - qm_sg_ents, CAAM_QI_MAX_ABLKCIPHER_SG); - caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, - iv_dma, ivsize, op_type, 0, 0); + qm_sg_bytes = qm_sg_ents * sizeof(struct qm_sg_entry); + if (unlikely(offsetof(struct ablkcipher_edesc, sgt) + qm_sg_bytes + + ivsize > CAAM_QI_MEMCACHE_SIZE)) { + dev_err(qidev, "No space for %d S/G entries and/or %dB IV\n", + qm_sg_ents, ivsize); + caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0, + 0, 0, 0, 0); return ERR_PTR(-ENOMEM); } - /* allocate space for base edesc and link tables */ + /* allocate space for base edesc, link tables and IV */ edesc = qi_cache_alloc(GFP_DMA | flags); if (unlikely(!edesc)) { dev_err(qidev, "could not allocate extended descriptor\n"); - caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, - iv_dma, ivsize, op_type, 0, 0); + caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0, + 0, 0, 0, 0); + return ERR_PTR(-ENOMEM); + } + + /* Make sure IV is located in a DMAable area */ + sg_table = &edesc->sgt[0]; + iv = (u8 *)(sg_table + qm_sg_ents); + memcpy(iv, req->info, ivsize); + + iv_dma = dma_map_single(qidev, iv, ivsize, DMA_TO_DEVICE); + if (dma_mapping_error(qidev, iv_dma)) { + dev_err(qidev, "unable to map IV\n"); + caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0, + 0, 0, 0, 0); + qi_cache_free(edesc); return ERR_PTR(-ENOMEM); } edesc->src_nents = src_nents; edesc->dst_nents = dst_nents; edesc->iv_dma = iv_dma; - sg_table = &edesc->sgt[0]; - edesc->qm_sg_bytes = qm_sg_ents * sizeof(*sg_table); + edesc->qm_sg_bytes = qm_sg_bytes; edesc->drv_req.app_ctx = req; edesc->drv_req.cbk = ablkcipher_done; edesc->drv_req.drv_ctx = drv_ctx; - if (!in_contig) { - dma_to_qm_sg_one(sg_table, iv_dma, ivsize, 0); - sg_to_qm_sg_last(req->src, mapped_src_nents, sg_table + 1, 0); - } + dma_to_qm_sg_one(sg_table, iv_dma, ivsize, 0); + sg_to_qm_sg_last(req->src, mapped_src_nents, sg_table + 1, 0); if (mapped_dst_nents > 1) sg_to_qm_sg_last(req->dst, mapped_dst_nents, sg_table + @@ -1304,20 +1316,12 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request fd_sgt = &edesc->drv_req.fd_sgt[0]; - if (!in_contig) - dma_to_qm_sg_one_last_ext(&fd_sgt[1], edesc->qm_sg_dma, - ivsize + req->nbytes, 0); - else - dma_to_qm_sg_one_last(&fd_sgt[1], iv_dma, ivsize + req->nbytes, - 0); + dma_to_qm_sg_one_last_ext(&fd_sgt[1], edesc->qm_sg_dma, + ivsize + req->nbytes, 0); if (req->src == req->dst) { - if (!in_contig) - dma_to_qm_sg_one_ext(&fd_sgt[0], edesc->qm_sg_dma + - sizeof(*sg_table), req->nbytes, 0); - else - dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->src), - req->nbytes, 0); + dma_to_qm_sg_one_ext(&fd_sgt[0], edesc->qm_sg_dma + + sizeof(*sg_table), req->nbytes, 0); } else if (mapped_dst_nents > 1) { dma_to_qm_sg_one_ext(&fd_sgt[0], edesc->qm_sg_dma + dst_sg_idx * sizeof(*sg_table), req->nbytes, 0); @@ -1341,10 +1345,10 @@ static struct ablkcipher_edesc *ablkcipher_giv_edesc_alloc( int src_nents, mapped_src_nents, dst_nents, mapped_dst_nents; struct ablkcipher_edesc *edesc; dma_addr_t iv_dma; - bool out_contig; + u8 *iv; int ivsize = crypto_ablkcipher_ivsize(ablkcipher); struct qm_sg_entry *sg_table, *fd_sgt; - int dst_sg_idx, qm_sg_ents; + int dst_sg_idx, qm_sg_ents, qm_sg_bytes; struct caam_drv_ctx *drv_ctx; drv_ctx = get_drv_ctx(ctx, GIVENCRYPT); @@ -1392,46 +1396,45 @@ static struct ablkcipher_edesc *ablkcipher_giv_edesc_alloc( mapped_dst_nents = src_nents; } - iv_dma = dma_map_single(qidev, creq->giv, ivsize, DMA_FROM_DEVICE); - if (dma_mapping_error(qidev, iv_dma)) { - dev_err(qidev, "unable to map IV\n"); - caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0, - 0, 0, 0, 0); - return ERR_PTR(-ENOMEM); - } - qm_sg_ents = mapped_src_nents > 1 ? mapped_src_nents : 0; dst_sg_idx = qm_sg_ents; - if (mapped_dst_nents == 1 && - iv_dma + ivsize == sg_dma_address(req->dst)) { - out_contig = true; - } else { - out_contig = false; - qm_sg_ents += 1 + mapped_dst_nents; - } - if (unlikely(qm_sg_ents > CAAM_QI_MAX_ABLKCIPHER_SG)) { - dev_err(qidev, "Insufficient S/G entries: %d > %zu\n", - qm_sg_ents, CAAM_QI_MAX_ABLKCIPHER_SG); - caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, - iv_dma, ivsize, GIVENCRYPT, 0, 0); + qm_sg_ents += 1 + mapped_dst_nents; + qm_sg_bytes = qm_sg_ents * sizeof(struct qm_sg_entry); + if (unlikely(offsetof(struct ablkcipher_edesc, sgt) + qm_sg_bytes + + ivsize > CAAM_QI_MEMCACHE_SIZE)) { + dev_err(qidev, "No space for %d S/G entries and/or %dB IV\n", + qm_sg_ents, ivsize); + caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0, + 0, 0, 0, 0); return ERR_PTR(-ENOMEM); } - /* allocate space for base edesc and link tables */ + /* allocate space for base edesc, link tables and IV */ edesc = qi_cache_alloc(GFP_DMA | flags); if (!edesc) { dev_err(qidev, "could not allocate extended descriptor\n"); - caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, - iv_dma, ivsize, GIVENCRYPT, 0, 0); + caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0, + 0, 0, 0, 0); + return ERR_PTR(-ENOMEM); + } + + /* Make sure IV is located in a DMAable area */ + sg_table = &edesc->sgt[0]; + iv = (u8 *)(sg_table + qm_sg_ents); + iv_dma = dma_map_single(qidev, iv, ivsize, DMA_FROM_DEVICE); + if (dma_mapping_error(qidev, iv_dma)) { + dev_err(qidev, "unable to map IV\n"); + caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0, + 0, 0, 0, 0); + qi_cache_free(edesc); return ERR_PTR(-ENOMEM); } edesc->src_nents = src_nents; edesc->dst_nents = dst_nents; edesc->iv_dma = iv_dma; - sg_table = &edesc->sgt[0]; - edesc->qm_sg_bytes = qm_sg_ents * sizeof(*sg_table); + edesc->qm_sg_bytes = qm_sg_bytes; edesc->drv_req.app_ctx = req; edesc->drv_req.cbk = ablkcipher_done; edesc->drv_req.drv_ctx = drv_ctx; @@ -1439,11 +1442,9 @@ static struct ablkcipher_edesc *ablkcipher_giv_edesc_alloc( if (mapped_src_nents > 1) sg_to_qm_sg_last(req->src, mapped_src_nents, sg_table, 0); - if (!out_contig) { - dma_to_qm_sg_one(sg_table + dst_sg_idx, iv_dma, ivsize, 0); - sg_to_qm_sg_last(req->dst, mapped_dst_nents, sg_table + - dst_sg_idx + 1, 0); - } + dma_to_qm_sg_one(sg_table + dst_sg_idx, iv_dma, ivsize, 0); + sg_to_qm_sg_last(req->dst, mapped_dst_nents, sg_table + dst_sg_idx + 1, + 0); edesc->qm_sg_dma = dma_map_single(qidev, sg_table, edesc->qm_sg_bytes, DMA_TO_DEVICE); @@ -1464,13 +1465,8 @@ static struct ablkcipher_edesc *ablkcipher_giv_edesc_alloc( dma_to_qm_sg_one(&fd_sgt[1], sg_dma_address(req->src), req->nbytes, 0); - if (!out_contig) - dma_to_qm_sg_one_ext(&fd_sgt[0], edesc->qm_sg_dma + dst_sg_idx * - sizeof(*sg_table), ivsize + req->nbytes, - 0); - else - dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->dst), - ivsize + req->nbytes, 0); + dma_to_qm_sg_one_ext(&fd_sgt[0], edesc->qm_sg_dma + dst_sg_idx * + sizeof(*sg_table), ivsize + req->nbytes, 0); return edesc; } @@ -1480,6 +1476,7 @@ static inline int ablkcipher_crypt(struct ablkcipher_request *req, bool encrypt) struct ablkcipher_edesc *edesc; struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req); struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher); + int ivsize = crypto_ablkcipher_ivsize(ablkcipher); int ret; if (unlikely(caam_congested)) @@ -1490,6 +1487,14 @@ static inline int ablkcipher_crypt(struct ablkcipher_request *req, bool encrypt) if (IS_ERR(edesc)) return PTR_ERR(edesc); + /* + * The crypto API expects us to set the IV (req->info) to the last + * ciphertext block. + */ + if (!encrypt) + scatterwalk_map_and_copy(req->info, req->src, req->nbytes - + ivsize, ivsize, 0); + ret = caam_qi_enqueue(ctx->qidev, &edesc->drv_req); if (!ret) { ret = -EINPROGRESS; diff --git a/drivers/crypto/caam/caampkc.c b/drivers/crypto/caam/caampkc.c index 7a897209f181..578ea63a3109 100644 --- a/drivers/crypto/caam/caampkc.c +++ b/drivers/crypto/caam/caampkc.c @@ -66,7 +66,7 @@ static void rsa_priv_f2_unmap(struct device *dev, struct rsa_edesc *edesc, struct caam_rsa_key *key = &ctx->key; struct rsa_priv_f2_pdb *pdb = &edesc->pdb.priv_f2; size_t p_sz = key->p_sz; - size_t q_sz = key->p_sz; + size_t q_sz = key->q_sz; dma_unmap_single(dev, pdb->d_dma, key->d_sz, DMA_TO_DEVICE); dma_unmap_single(dev, pdb->p_dma, p_sz, DMA_TO_DEVICE); @@ -83,7 +83,7 @@ static void rsa_priv_f3_unmap(struct device *dev, struct rsa_edesc *edesc, struct caam_rsa_key *key = &ctx->key; struct rsa_priv_f3_pdb *pdb = &edesc->pdb.priv_f3; size_t p_sz = key->p_sz; - size_t q_sz = key->p_sz; + size_t q_sz = key->q_sz; dma_unmap_single(dev, pdb->p_dma, p_sz, DMA_TO_DEVICE); dma_unmap_single(dev, pdb->q_dma, q_sz, DMA_TO_DEVICE); @@ -166,18 +166,71 @@ static void rsa_priv_f3_done(struct device *dev, u32 *desc, u32 err, akcipher_request_complete(req, err); } +static int caam_rsa_count_leading_zeros(struct scatterlist *sgl, + unsigned int nbytes, + unsigned int flags) +{ + struct sg_mapping_iter miter; + int lzeros, ents; + unsigned int len; + unsigned int tbytes = nbytes; + const u8 *buff; + + ents = sg_nents_for_len(sgl, nbytes); + if (ents < 0) + return ents; + + sg_miter_start(&miter, sgl, ents, SG_MITER_FROM_SG | flags); + + lzeros = 0; + len = 0; + while (nbytes > 0) { + while (len && !*buff) { + lzeros++; + len--; + buff++; + } + + if (len && *buff) + break; + + sg_miter_next(&miter); + buff = miter.addr; + len = miter.length; + + nbytes -= lzeros; + lzeros = 0; + } + + miter.consumed = lzeros; + sg_miter_stop(&miter); + nbytes -= lzeros; + + return tbytes - nbytes; +} + static struct rsa_edesc *rsa_edesc_alloc(struct akcipher_request *req, size_t desclen) { struct crypto_akcipher *tfm = crypto_akcipher_reqtfm(req); struct caam_rsa_ctx *ctx = akcipher_tfm_ctx(tfm); struct device *dev = ctx->dev; + struct caam_rsa_req_ctx *req_ctx = akcipher_request_ctx(req); struct rsa_edesc *edesc; gfp_t flags = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ? GFP_KERNEL : GFP_ATOMIC; + int sg_flags = (flags == GFP_ATOMIC) ? SG_MITER_ATOMIC : 0; int sgc; int sec4_sg_index, sec4_sg_len = 0, sec4_sg_bytes; int src_nents, dst_nents; + int lzeros; + + lzeros = caam_rsa_count_leading_zeros(req->src, req->src_len, sg_flags); + if (lzeros < 0) + return ERR_PTR(lzeros); + + req->src_len -= lzeros; + req->src = scatterwalk_ffwd(req_ctx->src, req->src, lzeros); src_nents = sg_nents_for_len(req->src, req->src_len); dst_nents = sg_nents_for_len(req->dst, req->dst_len); @@ -344,7 +397,7 @@ static int set_rsa_priv_f2_pdb(struct akcipher_request *req, struct rsa_priv_f2_pdb *pdb = &edesc->pdb.priv_f2; int sec4_sg_index = 0; size_t p_sz = key->p_sz; - size_t q_sz = key->p_sz; + size_t q_sz = key->q_sz; pdb->d_dma = dma_map_single(dev, key->d, key->d_sz, DMA_TO_DEVICE); if (dma_mapping_error(dev, pdb->d_dma)) { @@ -419,7 +472,7 @@ static int set_rsa_priv_f3_pdb(struct akcipher_request *req, struct rsa_priv_f3_pdb *pdb = &edesc->pdb.priv_f3; int sec4_sg_index = 0; size_t p_sz = key->p_sz; - size_t q_sz = key->p_sz; + size_t q_sz = key->q_sz; pdb->p_dma = dma_map_single(dev, key->p, p_sz, DMA_TO_DEVICE); if (dma_mapping_error(dev, pdb->p_dma)) { @@ -730,19 +783,12 @@ static u8 *caam_read_rsa_crt(const u8 *ptr, size_t nbytes, size_t dstlen) */ static inline u8 *caam_read_raw_data(const u8 *buf, size_t *nbytes) { - u8 *val; caam_rsa_drop_leading_zeros(&buf, nbytes); if (!*nbytes) return NULL; - val = kzalloc(*nbytes, GFP_DMA | GFP_KERNEL); - if (!val) - return NULL; - - memcpy(val, buf, *nbytes); - - return val; + return kmemdup(buf, *nbytes, GFP_DMA | GFP_KERNEL); } static int caam_rsa_check_key_length(unsigned int len) @@ -953,6 +999,7 @@ static struct akcipher_alg caam_rsa = { .max_size = caam_rsa_max_size, .init = caam_rsa_init_tfm, .exit = caam_rsa_exit_tfm, + .reqsize = sizeof(struct caam_rsa_req_ctx), .base = { .cra_name = "rsa", .cra_driver_name = "rsa-caam", diff --git a/drivers/crypto/caam/caampkc.h b/drivers/crypto/caam/caampkc.h index fd145c46eae1..82645bcf8b27 100644 --- a/drivers/crypto/caam/caampkc.h +++ b/drivers/crypto/caam/caampkc.h @@ -95,6 +95,14 @@ struct caam_rsa_ctx { struct device *dev; }; +/** + * caam_rsa_req_ctx - per request context. + * @src: input scatterlist (stripped of leading zeros) + */ +struct caam_rsa_req_ctx { + struct scatterlist src[2]; +}; + /** * rsa_edesc - s/w-extended rsa descriptor * @src_nents : number of segments in input scatterlist diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c index e4cc636e1104..538c01f428c1 100644 --- a/drivers/crypto/caam/ctrl.c +++ b/drivers/crypto/caam/ctrl.c @@ -322,9 +322,9 @@ static int caam_remove(struct platform_device *pdev) /* * De-initialize RNG state handles initialized by this driver. - * In case of DPAA 2.x, RNG is managed by MC firmware. + * In case of SoCs with Management Complex, RNG is managed by MC f/w. */ - if (!caam_dpaa2 && ctrlpriv->rng4_sh_init) + if (!ctrlpriv->mc_en && ctrlpriv->rng4_sh_init) deinstantiate_rng(ctrldev, ctrlpriv->rng4_sh_init); /* Shut down debug views */ @@ -396,11 +396,56 @@ start_rng: clrsetbits_32(&r4tst->rtmctl, RTMCTL_PRGM, RTMCTL_SAMP_MODE_RAW_ES_SC); } +static int caam_get_era_from_hw(struct caam_ctrl __iomem *ctrl) +{ + static const struct { + u16 ip_id; + u8 maj_rev; + u8 era; + } id[] = { + {0x0A10, 1, 1}, + {0x0A10, 2, 2}, + {0x0A12, 1, 3}, + {0x0A14, 1, 3}, + {0x0A14, 2, 4}, + {0x0A16, 1, 4}, + {0x0A10, 3, 4}, + {0x0A11, 1, 4}, + {0x0A18, 1, 4}, + {0x0A11, 2, 5}, + {0x0A12, 2, 5}, + {0x0A13, 1, 5}, + {0x0A1C, 1, 5} + }; + u32 ccbvid, id_ms; + u8 maj_rev, era; + u16 ip_id; + int i; + + ccbvid = rd_reg32(&ctrl->perfmon.ccb_id); + era = (ccbvid & CCBVID_ERA_MASK) >> CCBVID_ERA_SHIFT; + if (era) /* This is '0' prior to CAAM ERA-6 */ + return era; + + id_ms = rd_reg32(&ctrl->perfmon.caam_id_ms); + ip_id = (id_ms & SECVID_MS_IPID_MASK) >> SECVID_MS_IPID_SHIFT; + maj_rev = (id_ms & SECVID_MS_MAJ_REV_MASK) >> SECVID_MS_MAJ_REV_SHIFT; + + for (i = 0; i < ARRAY_SIZE(id); i++) + if (id[i].ip_id == ip_id && id[i].maj_rev == maj_rev) + return id[i].era; + + return -ENOTSUPP; +} + /** * caam_get_era() - Return the ERA of the SEC on SoC, based - * on "sec-era" propery in the DTS. This property is updated by u-boot. + * on "sec-era" optional property in the DTS. This property is updated + * by u-boot. + * In case this property is not passed an attempt to retrieve the CAAM + * era via register reads will be made. **/ -int caam_get_era(void) +static int caam_get_era(struct caam_ctrl __iomem *ctrl) { struct device_node *caam_node; int ret; @@ -410,9 +455,11 @@ int caam_get_era(void) ret = of_property_read_u32(caam_node, "fsl,sec-era", &prop); of_node_put(caam_node); - return ret ? -ENOTSUPP : prop; + if (!ret) + return prop; + else + return caam_get_era_from_hw(ctrl); } -EXPORT_SYMBOL(caam_get_era); static const struct of_device_id caam_match[] = { { @@ -571,11 +618,15 @@ static int caam_probe(struct platform_device *pdev) /* * Enable DECO watchdogs and, if this is a PHYS_ADDR_T_64BIT kernel, * long pointers in master configuration register. - * In case of DPAA 2.x, Management Complex firmware performs + * In case of SoCs with Management Complex, MC f/w performs * the configuration. */ caam_dpaa2 = !!(comp_params & CTPR_MS_DPAA2); - if (!caam_dpaa2) + np = of_find_compatible_node(NULL, NULL, "fsl,qoriq-mc"); + ctrlpriv->mc_en = !!np; + of_node_put(np); + + if (!ctrlpriv->mc_en) clrsetbits_32(&ctrl->mcr, MCFGR_AWCACHE_MASK | MCFGR_LONG_PTR, MCFGR_AWCACHE_CACH | MCFGR_AWCACHE_BUFF | MCFGR_WDENABLE | MCFGR_LARGE_BURST | @@ -623,7 +674,7 @@ static int caam_probe(struct platform_device *pdev) goto iounmap_ctrl; } - ctrlpriv->era = caam_get_era(); + ctrlpriv->era = caam_get_era(ctrl); ret = of_platform_populate(nprop, caam_match, NULL, dev); if (ret) { @@ -686,9 +737,9 @@ static int caam_probe(struct platform_device *pdev) /* * If SEC has RNG version >= 4 and RNG state handle has not been * already instantiated, do RNG instantiation - * In case of DPAA 2.x, RNG is managed by MC firmware. + * In case of SoCs with Management Complex, RNG is managed by MC f/w. */ - if (!caam_dpaa2 && + if (!ctrlpriv->mc_en && (cha_vid_ls & CHA_ID_LS_RNG_MASK) >> CHA_ID_LS_RNG_SHIFT >= 4) { ctrlpriv->rng4_sh_init = rd_reg32(&ctrl->r4tst[0].rdsta); @@ -757,9 +808,8 @@ static int caam_probe(struct platform_device *pdev) /* Report "alive" for developer to see */ dev_info(dev, "device ID = 0x%016llx (Era %d)\n", caam_id, ctrlpriv->era); - dev_info(dev, "job rings = %d, qi = %d, dpaa2 = %s\n", - ctrlpriv->total_jobrs, ctrlpriv->qi_present, - caam_dpaa2 ? "yes" : "no"); + dev_info(dev, "job rings = %d, qi = %d\n", + ctrlpriv->total_jobrs, ctrlpriv->qi_present); #ifdef CONFIG_DEBUG_FS debugfs_create_file("rq_dequeued", S_IRUSR | S_IRGRP | S_IROTH, diff --git a/drivers/crypto/caam/ctrl.h b/drivers/crypto/caam/ctrl.h index be693a2cc25e..f3ecd67922a7 100644 --- a/drivers/crypto/caam/ctrl.h +++ b/drivers/crypto/caam/ctrl.h @@ -9,8 +9,6 @@ #define CTRL_H /* Prototypes for backend-level services exposed to APIs */ -int caam_get_era(void); - extern bool caam_dpaa2; #endif /* CTRL_H */ diff --git a/drivers/crypto/caam/intern.h b/drivers/crypto/caam/intern.h index 7696a774a362..babc78abd155 100644 --- a/drivers/crypto/caam/intern.h +++ b/drivers/crypto/caam/intern.h @@ -82,6 +82,7 @@ struct caam_drv_private { */ u8 total_jobrs; /* Total Job Rings in device */ u8 qi_present; /* Nonzero if QI present in device */ + u8 mc_en; /* Nonzero if MC f/w is active */ int secvio_irq; /* Security violation interrupt number */ int virt_en; /* Virtualization enabled in CAAM */ int era; /* CAAM Era (internal HW revision) */ diff --git a/drivers/crypto/caam/qi.c b/drivers/crypto/caam/qi.c index b9480828da38..67f7f8c42c93 100644 --- a/drivers/crypto/caam/qi.c +++ b/drivers/crypto/caam/qi.c @@ -657,9 +657,8 @@ static int init_cgr(struct device *qidev) { int ret; struct qm_mcc_initcgr opts; - const u64 cpus = *(u64 *)qman_affine_cpus(); - const int num_cpus = hweight64(cpus); - const u64 val = num_cpus * MAX_RSP_FQ_BACKLOG_PER_CPU; + const u64 val = (u64)cpumask_weight(qman_affine_cpus()) * + MAX_RSP_FQ_BACKLOG_PER_CPU; ret = qman_alloc_cgrid(&qipriv.cgr.cgrid); if (ret) { diff --git a/drivers/crypto/caam/regs.h b/drivers/crypto/caam/regs.h index fee363865d88..4fb91ba39c36 100644 --- a/drivers/crypto/caam/regs.h +++ b/drivers/crypto/caam/regs.h @@ -312,11 +312,17 @@ struct caam_perfmon { /* Component Instantiation Parameters fe0-fff */ u32 rtic_id; /* RVID - RTIC Version ID */ +#define CCBVID_ERA_MASK 0xff000000 +#define CCBVID_ERA_SHIFT 24 u32 ccb_id; /* CCBVID - CCB Version ID */ u32 cha_id_ms; /* CHAVID - CHA Version ID Most Significant*/ u32 cha_id_ls; /* CHAVID - CHA Version ID Least Significant*/ u32 cha_num_ms; /* CHANUM - CHA Number Most Significant */ u32 cha_num_ls; /* CHANUM - CHA Number Least Significant*/ +#define SECVID_MS_IPID_MASK 0xffff0000 +#define SECVID_MS_IPID_SHIFT 16 +#define SECVID_MS_MAJ_REV_MASK 0x0000ff00 +#define SECVID_MS_MAJ_REV_SHIFT 8 u32 caam_id_ms; /* CAAMVID - CAAM Version ID MS */ u32 caam_id_ls; /* CAAMVID - CAAM Version ID LS */ }; diff --git a/drivers/crypto/cavium/nitrox/nitrox_isr.c b/drivers/crypto/cavium/nitrox/nitrox_isr.c index dbead5f45df3..ee0d70ba25d5 100644 --- a/drivers/crypto/cavium/nitrox/nitrox_isr.c +++ b/drivers/crypto/cavium/nitrox/nitrox_isr.c @@ -254,7 +254,7 @@ static int nitrox_enable_msix(struct nitrox_device *ndev) * Entry 192: NPS_CORE_INT_ACTIVE */ nr_entries = (ndev->nr_queues * NR_RING_VECTORS) + 1; - entries = kzalloc_node(nr_entries * sizeof(struct msix_entry), + entries = kcalloc_node(nr_entries, sizeof(struct msix_entry), GFP_KERNEL, ndev->node); if (!entries) return -ENOMEM; diff --git a/drivers/crypto/cavium/zip/common.h b/drivers/crypto/cavium/zip/common.h index dc451e0a43c5..58fb3ed6e644 100644 --- a/drivers/crypto/cavium/zip/common.h +++ b/drivers/crypto/cavium/zip/common.h @@ -46,8 +46,10 @@ #ifndef __COMMON_H__ #define __COMMON_H__ +#include #include #include +#include #include #include #include @@ -149,6 +151,25 @@ struct zip_operation { u32 sizeofzops; }; +static inline int zip_poll_result(union zip_zres_s *result) +{ + int retries = 1000; + + while (!result->s.compcode) { + if (!--retries) { + pr_err("ZIP ERR: request timed out"); + return -ETIMEDOUT; + } + udelay(10); + /* + * Force re-reading of compcode which is updated + * by the ZIP coprocessor. + */ + rmb(); + } + return 0; +} + /* error messages */ #define zip_err(fmt, args...) pr_err("ZIP ERR:%s():%d: " \ fmt "\n", __func__, __LINE__, ## args) diff --git a/drivers/crypto/cavium/zip/zip_crypto.c b/drivers/crypto/cavium/zip/zip_crypto.c index 8df4d26cf9d4..b92b6e7e100f 100644 --- a/drivers/crypto/cavium/zip/zip_crypto.c +++ b/drivers/crypto/cavium/zip/zip_crypto.c @@ -124,7 +124,7 @@ int zip_compress(const u8 *src, unsigned int slen, struct zip_kernel_ctx *zip_ctx) { struct zip_operation *zip_ops = NULL; - struct zip_state zip_state; + struct zip_state *zip_state; struct zip_device *zip = NULL; int ret; @@ -135,20 +135,23 @@ int zip_compress(const u8 *src, unsigned int slen, if (!zip) return -ENODEV; - memset(&zip_state, 0, sizeof(struct zip_state)); + zip_state = kzalloc(sizeof(*zip_state), GFP_ATOMIC); + if (!zip_state) + return -ENOMEM; + zip_ops = &zip_ctx->zip_comp; zip_ops->input_len = slen; zip_ops->output_len = *dlen; memcpy(zip_ops->input, src, slen); - ret = zip_deflate(zip_ops, &zip_state, zip); + ret = zip_deflate(zip_ops, zip_state, zip); if (!ret) { *dlen = zip_ops->output_len; memcpy(dst, zip_ops->output, *dlen); } - + kfree(zip_state); return ret; } @@ -157,7 +160,7 @@ int zip_decompress(const u8 *src, unsigned int slen, struct zip_kernel_ctx *zip_ctx) { struct zip_operation *zip_ops = NULL; - struct zip_state zip_state; + struct zip_state *zip_state; struct zip_device *zip = NULL; int ret; @@ -168,7 +171,10 @@ int zip_decompress(const u8 *src, unsigned int slen, if (!zip) return -ENODEV; - memset(&zip_state, 0, sizeof(struct zip_state)); + zip_state = kzalloc(sizeof(*zip_state), GFP_ATOMIC); + if (!zip_state) + return -ENOMEM; + zip_ops = &zip_ctx->zip_decomp; memcpy(zip_ops->input, src, slen); @@ -179,13 +185,13 @@ int zip_decompress(const u8 *src, unsigned int slen, zip_ops->input_len = slen; zip_ops->output_len = *dlen; - ret = zip_inflate(zip_ops, &zip_state, zip); + ret = zip_inflate(zip_ops, zip_state, zip); if (!ret) { *dlen = zip_ops->output_len; memcpy(dst, zip_ops->output, *dlen); } - + kfree(zip_state); return ret; } diff --git a/drivers/crypto/cavium/zip/zip_deflate.c b/drivers/crypto/cavium/zip/zip_deflate.c index 9a944b8c1e29..d7133f857d67 100644 --- a/drivers/crypto/cavium/zip/zip_deflate.c +++ b/drivers/crypto/cavium/zip/zip_deflate.c @@ -129,8 +129,8 @@ int zip_deflate(struct zip_operation *zip_ops, struct zip_state *s, /* Stats update for compression requests submitted */ atomic64_inc(&zip_dev->stats.comp_req_submit); - while (!result_ptr->s.compcode) - continue; + /* Wait for completion or error */ + zip_poll_result(result_ptr); /* Stats update for compression requests completed */ atomic64_inc(&zip_dev->stats.comp_req_complete); diff --git a/drivers/crypto/cavium/zip/zip_device.c b/drivers/crypto/cavium/zip/zip_device.c index ccf21fb91513..f174ec29ed69 100644 --- a/drivers/crypto/cavium/zip/zip_device.c +++ b/drivers/crypto/cavium/zip/zip_device.c @@ -87,12 +87,12 @@ u32 zip_load_instr(union zip_inst_s *instr, * Distribute the instructions between the enabled queues based on * the CPU id. */ - if (smp_processor_id() % 2 == 0) + if (raw_smp_processor_id() % 2 == 0) queue = 0; else queue = 1; - zip_dbg("CPU Core: %d Queue number:%d", smp_processor_id(), queue); + zip_dbg("CPU Core: %d Queue number:%d", raw_smp_processor_id(), queue); /* Take cmd buffer lock */ spin_lock(&zip_dev->iq[queue].lock); diff --git a/drivers/crypto/cavium/zip/zip_inflate.c b/drivers/crypto/cavium/zip/zip_inflate.c index 50cbdd83dbf2..7e0d73e2f89e 100644 --- a/drivers/crypto/cavium/zip/zip_inflate.c +++ b/drivers/crypto/cavium/zip/zip_inflate.c @@ -143,8 +143,8 @@ int zip_inflate(struct zip_operation *zip_ops, struct zip_state *s, /* Decompression requests submitted stats update */ atomic64_inc(&zip_dev->stats.decomp_req_submit); - while (!result_ptr->s.compcode) - continue; + /* Wait for completion or error */ + zip_poll_result(result_ptr); /* Decompression requests completed stats update */ atomic64_inc(&zip_dev->stats.decomp_req_complete); diff --git a/drivers/crypto/cavium/zip/zip_main.c b/drivers/crypto/cavium/zip/zip_main.c index 1cd8aa488185..be055b9547f6 100644 --- a/drivers/crypto/cavium/zip/zip_main.c +++ b/drivers/crypto/cavium/zip/zip_main.c @@ -113,7 +113,7 @@ struct zip_device *zip_get_device(int node) */ int zip_get_node_id(void) { - return cpu_to_node(smp_processor_id()); + return cpu_to_node(raw_smp_processor_id()); } /* Initializes the ZIP h/w sub-system */ @@ -469,6 +469,8 @@ static int zip_show_stats(struct seq_file *s, void *unused) struct zip_stats *st; for (index = 0; index < MAX_ZIP_DEVICES; index++) { + u64 pending = 0; + if (zip_dev[index]) { zip = zip_dev[index]; st = &zip->stats; @@ -476,16 +478,15 @@ static int zip_show_stats(struct seq_file *s, void *unused) /* Get all the pending requests */ for (q = 0; q < ZIP_NUM_QUEUES; q++) { val = zip_reg_read((zip->reg_base + - ZIP_DBG_COREX_STA(q))); - val = (val >> 32); - val = val & 0xffffff; - atomic64_add(val, &st->pending_req); + ZIP_DBG_QUEX_STA(q))); + pending += val >> 32 & 0xffffff; } - avg_chunk = (atomic64_read(&st->comp_in_bytes) / - atomic64_read(&st->comp_req_complete)); - avg_cr = (atomic64_read(&st->comp_in_bytes) / - atomic64_read(&st->comp_out_bytes)); + val = atomic64_read(&st->comp_req_complete); + avg_chunk = (val) ? atomic64_read(&st->comp_in_bytes) / val : 0; + + val = atomic64_read(&st->comp_out_bytes); + avg_cr = (val) ? atomic64_read(&st->comp_in_bytes) / val : 0; seq_printf(s, " ZIP Device %d Stats\n" "-----------------------------------\n" "Comp Req Submitted : \t%lld\n" @@ -513,10 +514,7 @@ static int zip_show_stats(struct seq_file *s, void *unused) (u64)atomic64_read(&st->decomp_in_bytes), (u64)atomic64_read(&st->decomp_out_bytes), (u64)atomic64_read(&st->decomp_bad_reqs), - (u64)atomic64_read(&st->pending_req)); - - /* Reset pending requests count */ - atomic64_set(&st->pending_req, 0); + pending); } } return 0; diff --git a/drivers/crypto/cavium/zip/zip_main.h b/drivers/crypto/cavium/zip/zip_main.h index 64e051f60784..e1e4fa92ce80 100644 --- a/drivers/crypto/cavium/zip/zip_main.h +++ b/drivers/crypto/cavium/zip/zip_main.h @@ -74,7 +74,6 @@ struct zip_stats { atomic64_t comp_req_complete; atomic64_t decomp_req_submit; atomic64_t decomp_req_complete; - atomic64_t pending_req; atomic64_t comp_in_bytes; atomic64_t comp_out_bytes; atomic64_t decomp_in_bytes; diff --git a/drivers/crypto/cavium/zip/zip_regs.h b/drivers/crypto/cavium/zip/zip_regs.h index d0be682305c1..874e0236c87e 100644 --- a/drivers/crypto/cavium/zip/zip_regs.h +++ b/drivers/crypto/cavium/zip/zip_regs.h @@ -443,7 +443,7 @@ union zip_corex_bist_status { static inline u64 ZIP_COREX_BIST_STATUS(u64 param1) { - if (((param1 <= 1))) + if (param1 <= 1) return 0x0520ull + (param1 & 1) * 0x8ull; pr_err("ZIP_COREX_BIST_STATUS: %llu\n", param1); return 0; @@ -537,7 +537,7 @@ union zip_dbg_corex_inst { static inline u64 ZIP_DBG_COREX_INST(u64 param1) { - if (((param1 <= 1))) + if (param1 <= 1) return 0x0640ull + (param1 & 1) * 0x8ull; pr_err("ZIP_DBG_COREX_INST: %llu\n", param1); return 0; @@ -568,7 +568,7 @@ union zip_dbg_corex_sta { static inline u64 ZIP_DBG_COREX_STA(u64 param1) { - if (((param1 <= 1))) + if (param1 <= 1) return 0x0680ull + (param1 & 1) * 0x8ull; pr_err("ZIP_DBG_COREX_STA: %llu\n", param1); return 0; @@ -599,7 +599,7 @@ union zip_dbg_quex_sta { static inline u64 ZIP_DBG_QUEX_STA(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x1800ull + (param1 & 7) * 0x8ull; pr_err("ZIP_DBG_QUEX_STA: %llu\n", param1); return 0; @@ -817,7 +817,7 @@ union zip_msix_pbax { static inline u64 ZIP_MSIX_PBAX(u64 param1) { - if (((param1 == 0))) + if (param1 == 0) return 0x0000838000FF0000ull; pr_err("ZIP_MSIX_PBAX: %llu\n", param1); return 0; @@ -846,7 +846,7 @@ union zip_msix_vecx_addr { static inline u64 ZIP_MSIX_VECX_ADDR(u64 param1) { - if (((param1 <= 17))) + if (param1 <= 17) return 0x0000838000F00000ull + (param1 & 31) * 0x10ull; pr_err("ZIP_MSIX_VECX_ADDR: %llu\n", param1); return 0; @@ -875,7 +875,7 @@ union zip_msix_vecx_ctl { static inline u64 ZIP_MSIX_VECX_CTL(u64 param1) { - if (((param1 <= 17))) + if (param1 <= 17) return 0x0000838000F00008ull + (param1 & 31) * 0x10ull; pr_err("ZIP_MSIX_VECX_CTL: %llu\n", param1); return 0; @@ -900,7 +900,7 @@ union zip_quex_done { static inline u64 ZIP_QUEX_DONE(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x2000ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_DONE: %llu\n", param1); return 0; @@ -925,7 +925,7 @@ union zip_quex_done_ack { static inline u64 ZIP_QUEX_DONE_ACK(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x2200ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_DONE_ACK: %llu\n", param1); return 0; @@ -950,7 +950,7 @@ union zip_quex_done_ena_w1c { static inline u64 ZIP_QUEX_DONE_ENA_W1C(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x2600ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_DONE_ENA_W1C: %llu\n", param1); return 0; @@ -975,7 +975,7 @@ union zip_quex_done_ena_w1s { static inline u64 ZIP_QUEX_DONE_ENA_W1S(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x2400ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_DONE_ENA_W1S: %llu\n", param1); return 0; @@ -1004,7 +1004,7 @@ union zip_quex_done_wait { static inline u64 ZIP_QUEX_DONE_WAIT(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x2800ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_DONE_WAIT: %llu\n", param1); return 0; @@ -1029,7 +1029,7 @@ union zip_quex_doorbell { static inline u64 ZIP_QUEX_DOORBELL(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x4000ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_DOORBELL: %llu\n", param1); return 0; @@ -1058,7 +1058,7 @@ union zip_quex_err_ena_w1c { static inline u64 ZIP_QUEX_ERR_ENA_W1C(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x3600ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_ERR_ENA_W1C: %llu\n", param1); return 0; @@ -1087,7 +1087,7 @@ union zip_quex_err_ena_w1s { static inline u64 ZIP_QUEX_ERR_ENA_W1S(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x3400ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_ERR_ENA_W1S: %llu\n", param1); return 0; @@ -1120,7 +1120,7 @@ union zip_quex_err_int { static inline u64 ZIP_QUEX_ERR_INT(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x3000ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_ERR_INT: %llu\n", param1); return 0; @@ -1150,7 +1150,7 @@ union zip_quex_err_int_w1s { static inline u64 ZIP_QUEX_ERR_INT_W1S(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x3200ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_ERR_INT_W1S: %llu\n", param1); return 0; @@ -1179,7 +1179,7 @@ union zip_quex_gcfg { static inline u64 ZIP_QUEX_GCFG(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x1A00ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_GCFG: %llu\n", param1); return 0; @@ -1204,7 +1204,7 @@ union zip_quex_map { static inline u64 ZIP_QUEX_MAP(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x1400ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_MAP: %llu\n", param1); return 0; @@ -1236,7 +1236,7 @@ union zip_quex_sbuf_addr { static inline u64 ZIP_QUEX_SBUF_ADDR(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x1000ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_SBUF_ADDR: %llu\n", param1); return 0; @@ -1276,7 +1276,7 @@ union zip_quex_sbuf_ctl { static inline u64 ZIP_QUEX_SBUF_CTL(u64 param1) { - if (((param1 <= 7))) + if (param1 <= 7) return 0x1200ull + (param1 & 7) * 0x8ull; pr_err("ZIP_QUEX_SBUF_CTL: %llu\n", param1); return 0; diff --git a/drivers/crypto/ccp/psp-dev.c b/drivers/crypto/ccp/psp-dev.c index d95ec526587a..ff478d826d7d 100644 --- a/drivers/crypto/ccp/psp-dev.c +++ b/drivers/crypto/ccp/psp-dev.c @@ -22,11 +22,17 @@ #include #include #include +#include #include "sp-dev.h" #include "psp-dev.h" +#define SEV_VERSION_GREATER_OR_EQUAL(_maj, _min) \ + ((psp_master->api_major) >= _maj && \ + (psp_master->api_minor) >= _min) + #define DEVICE_NAME "sev" +#define SEV_FW_FILE "amd/sev.fw" static DEFINE_MUTEX(sev_cmd_mutex); static struct sev_misc_dev *misc_dev; @@ -112,6 +118,8 @@ static int sev_cmd_buffer_len(int cmd) case SEV_CMD_RECEIVE_UPDATE_DATA: return sizeof(struct sev_data_receive_update_data); case SEV_CMD_RECEIVE_UPDATE_VMSA: return sizeof(struct sev_data_receive_update_vmsa); case SEV_CMD_LAUNCH_UPDATE_SECRET: return sizeof(struct sev_data_launch_secret); + case SEV_CMD_DOWNLOAD_FIRMWARE: return sizeof(struct sev_data_download_firmware); + case SEV_CMD_GET_ID: return sizeof(struct sev_data_get_id); default: return 0; } @@ -378,6 +386,79 @@ void *psp_copy_user_blob(u64 __user uaddr, u32 len) } EXPORT_SYMBOL_GPL(psp_copy_user_blob); +static int sev_get_api_version(void) +{ + struct sev_user_data_status *status; + int error, ret; + + status = &psp_master->status_cmd_buf; + ret = sev_platform_status(status, &error); + if (ret) { + dev_err(psp_master->dev, + "SEV: failed to get status. Error: %#x\n", error); + return 1; + } + + psp_master->api_major = status->api_major; + psp_master->api_minor = status->api_minor; + psp_master->build = status->build; + + return 0; +} + +/* Don't fail if SEV FW couldn't be updated. Continue with existing SEV FW */ +static int sev_update_firmware(struct device *dev) +{ + struct sev_data_download_firmware *data; + const struct firmware *firmware; + int ret, error, order; + struct page *p; + u64 data_size; + + ret = request_firmware(&firmware, SEV_FW_FILE, dev); + if (ret < 0) + return -1; + + /* + * SEV FW expects the physical address given to it to be 32 + * byte aligned. Memory allocated has structure placed at the + * beginning followed by the firmware being passed to the SEV + * FW. Allocate enough memory for data structure + alignment + * padding + SEV FW. + */ + data_size = ALIGN(sizeof(struct sev_data_download_firmware), 32); + + order = get_order(firmware->size + data_size); + p = alloc_pages(GFP_KERNEL, order); + if (!p) { + ret = -1; + goto fw_err; + } + + /* + * Copy firmware data to a kernel allocated contiguous + * memory region. + */ + data = page_address(p); + memcpy(page_address(p) + data_size, firmware->data, firmware->size); + + data->address = __psp_pa(page_address(p) + data_size); + data->len = firmware->size; + + ret = sev_do_cmd(SEV_CMD_DOWNLOAD_FIRMWARE, data, &error); + if (ret) + dev_dbg(dev, "Failed to update SEV firmware: %#x\n", error); + else + dev_info(dev, "SEV firmware update successful\n"); + + __free_pages(p, order); + +fw_err: + release_firmware(firmware); + + return ret; +} + static int sev_ioctl_do_pek_import(struct sev_issue_cmd *argp) { struct sev_user_data_pek_cert_import input; @@ -430,6 +511,46 @@ e_free: return ret; } +static int sev_ioctl_do_get_id(struct sev_issue_cmd *argp) +{ + struct sev_data_get_id *data; + u64 data_size, user_size; + void *id_blob, *mem; + int ret; + + /* SEV GET_ID available from SEV API v0.16 and up */ + if (!SEV_VERSION_GREATER_OR_EQUAL(0, 16)) + return -ENOTSUPP; + + /* SEV FW expects the buffer it fills with the ID to be + * 8-byte aligned. Memory allocated should be enough to + * hold data structure + alignment padding + memory + * where SEV FW writes the ID. + */ + data_size = ALIGN(sizeof(struct sev_data_get_id), 8); + user_size = sizeof(struct sev_user_data_get_id); + + mem = kzalloc(data_size + user_size, GFP_KERNEL); + if (!mem) + return -ENOMEM; + + data = mem; + id_blob = mem + data_size; + + data->address = __psp_pa(id_blob); + data->len = user_size; + + ret = __sev_do_cmd_locked(SEV_CMD_GET_ID, data, &argp->error); + if (!ret) { + if (copy_to_user((void __user *)argp->data, id_blob, data->len)) + ret = -EFAULT; + } + + kfree(mem); + + return ret; +} + static int sev_ioctl_do_pdh_export(struct sev_issue_cmd *argp) { struct sev_user_data_pdh_cert_export input; @@ -567,6 +688,9 @@ static long sev_ioctl(struct file *file, unsigned int ioctl, unsigned long arg) case SEV_PDH_CERT_EXPORT: ret = sev_ioctl_do_pdh_export(&input); break; + case SEV_GET_ID: + ret = sev_ioctl_do_get_id(&input); + break; default: ret = -EINVAL; goto out; @@ -750,7 +874,6 @@ EXPORT_SYMBOL_GPL(sev_issue_cmd_external_user); void psp_pci_init(void) { - struct sev_user_data_status *status; struct sp_device *sp; int error, rc; @@ -760,6 +883,13 @@ void psp_pci_init(void) psp_master = sp->psp_data; + if (sev_get_api_version()) + goto err; + + if (SEV_VERSION_GREATER_OR_EQUAL(0, 15) && + sev_update_firmware(psp_master->dev) == 0) + sev_get_api_version(); + /* Initialize the platform */ rc = sev_platform_init(&error); if (rc) { @@ -767,16 +897,9 @@ void psp_pci_init(void) goto err; } - /* Display SEV firmware version */ - status = &psp_master->status_cmd_buf; - rc = sev_platform_status(status, &error); - if (rc) { - dev_err(sp->dev, "SEV: failed to get status error %#x\n", error); - goto err; - } + dev_info(sp->dev, "SEV API:%d.%d build:%d\n", psp_master->api_major, + psp_master->api_minor, psp_master->build); - dev_info(sp->dev, "SEV API:%d.%d build:%d\n", status->api_major, - status->api_minor, status->build); return; err: diff --git a/drivers/crypto/ccp/psp-dev.h b/drivers/crypto/ccp/psp-dev.h index c81f0b11287a..c7e9098a233c 100644 --- a/drivers/crypto/ccp/psp-dev.h +++ b/drivers/crypto/ccp/psp-dev.h @@ -78,6 +78,10 @@ struct psp_device { struct sev_misc_dev *sev_misc; struct sev_user_data_status status_cmd_buf; struct sev_data_init init_cmd_buf; + + u8 api_major; + u8 api_minor; + u8 build; }; #endif /* __PSP_DEV_H */ diff --git a/drivers/crypto/ccree/cc_cipher.c b/drivers/crypto/ccree/cc_cipher.c index df98f7afe645..d2810c183b73 100644 --- a/drivers/crypto/ccree/cc_cipher.c +++ b/drivers/crypto/ccree/cc_cipher.c @@ -42,6 +42,7 @@ struct cc_cipher_ctx { int cipher_mode; int flow_mode; unsigned int flags; + bool hw_key; struct cc_user_key_info user; struct cc_hw_key_info hw; struct crypto_shash *shash_tfm; @@ -49,6 +50,13 @@ struct cc_cipher_ctx { static void cc_cipher_complete(struct device *dev, void *cc_req, int err); +static inline bool cc_is_hw_key(struct crypto_tfm *tfm) +{ + struct cc_cipher_ctx *ctx_p = crypto_tfm_ctx(tfm); + + return ctx_p->hw_key; +} + static int validate_keys_sizes(struct cc_cipher_ctx *ctx_p, u32 size) { switch (ctx_p->flow_mode) { @@ -211,7 +219,7 @@ struct tdes_keys { u8 key3[DES_KEY_SIZE]; }; -static enum cc_hw_crypto_key hw_key_to_cc_hw_key(int slot_num) +static enum cc_hw_crypto_key cc_slot_to_hw_key(int slot_num) { switch (slot_num) { case 0: @@ -226,69 +234,100 @@ static enum cc_hw_crypto_key hw_key_to_cc_hw_key(int slot_num) return END_OF_KEYS; } -static int cc_cipher_setkey(struct crypto_skcipher *sktfm, const u8 *key, - unsigned int keylen) +static int cc_cipher_sethkey(struct crypto_skcipher *sktfm, const u8 *key, + unsigned int keylen) { struct crypto_tfm *tfm = crypto_skcipher_tfm(sktfm); struct cc_cipher_ctx *ctx_p = crypto_tfm_ctx(tfm); struct device *dev = drvdata_to_dev(ctx_p->drvdata); - u32 tmp[DES3_EDE_EXPKEY_WORDS]; - struct cc_crypto_alg *cc_alg = - container_of(tfm->__crt_alg, struct cc_crypto_alg, - skcipher_alg.base); - unsigned int max_key_buf_size = cc_alg->skcipher_alg.max_keysize; + struct cc_hkey_info hki; - dev_dbg(dev, "Setting key in context @%p for %s. keylen=%u\n", + dev_dbg(dev, "Setting HW key in context @%p for %s. keylen=%u\n", ctx_p, crypto_tfm_alg_name(tfm), keylen); dump_byte_array("key", (u8 *)key, keylen); /* STAT_PHASE_0: Init and sanity checks */ + /* This check the size of the hardware key token */ + if (keylen != sizeof(hki)) { + dev_err(dev, "Unsupported HW key size %d.\n", keylen); + crypto_tfm_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; + } + + if (ctx_p->flow_mode != S_DIN_to_AES) { + dev_err(dev, "HW key not supported for non-AES flows\n"); + return -EINVAL; + } + + memcpy(&hki, key, keylen); + + /* The real key len for crypto op is the size of the HW key + * referenced by the HW key slot, not the hardware key token + */ + keylen = hki.keylen; + if (validate_keys_sizes(ctx_p, keylen)) { dev_err(dev, "Unsupported key size %d.\n", keylen); crypto_tfm_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN); return -EINVAL; } - if (cc_is_hw_key(tfm)) { - /* setting HW key slots */ - struct arm_hw_key_info *hki = (struct arm_hw_key_info *)key; + ctx_p->hw.key1_slot = cc_slot_to_hw_key(hki.hw_key1); + if (ctx_p->hw.key1_slot == END_OF_KEYS) { + dev_err(dev, "Unsupported hw key1 number (%d)\n", hki.hw_key1); + return -EINVAL; + } - if (ctx_p->flow_mode != S_DIN_to_AES) { - dev_err(dev, "HW key not supported for non-AES flows\n"); + if (ctx_p->cipher_mode == DRV_CIPHER_XTS || + ctx_p->cipher_mode == DRV_CIPHER_ESSIV || + ctx_p->cipher_mode == DRV_CIPHER_BITLOCKER) { + if (hki.hw_key1 == hki.hw_key2) { + dev_err(dev, "Illegal hw key numbers (%d,%d)\n", + hki.hw_key1, hki.hw_key2); return -EINVAL; } - - ctx_p->hw.key1_slot = hw_key_to_cc_hw_key(hki->hw_key1); - if (ctx_p->hw.key1_slot == END_OF_KEYS) { - dev_err(dev, "Unsupported hw key1 number (%d)\n", - hki->hw_key1); + ctx_p->hw.key2_slot = cc_slot_to_hw_key(hki.hw_key2); + if (ctx_p->hw.key2_slot == END_OF_KEYS) { + dev_err(dev, "Unsupported hw key2 number (%d)\n", + hki.hw_key2); return -EINVAL; } + } - if (ctx_p->cipher_mode == DRV_CIPHER_XTS || - ctx_p->cipher_mode == DRV_CIPHER_ESSIV || - ctx_p->cipher_mode == DRV_CIPHER_BITLOCKER) { - if (hki->hw_key1 == hki->hw_key2) { - dev_err(dev, "Illegal hw key numbers (%d,%d)\n", - hki->hw_key1, hki->hw_key2); - return -EINVAL; - } - ctx_p->hw.key2_slot = - hw_key_to_cc_hw_key(hki->hw_key2); - if (ctx_p->hw.key2_slot == END_OF_KEYS) { - dev_err(dev, "Unsupported hw key2 number (%d)\n", - hki->hw_key2); - return -EINVAL; - } - } + ctx_p->keylen = keylen; + ctx_p->hw_key = true; + dev_dbg(dev, "cc_is_hw_key ret 0"); + + return 0; +} + +static int cc_cipher_setkey(struct crypto_skcipher *sktfm, const u8 *key, + unsigned int keylen) +{ + struct crypto_tfm *tfm = crypto_skcipher_tfm(sktfm); + struct cc_cipher_ctx *ctx_p = crypto_tfm_ctx(tfm); + struct device *dev = drvdata_to_dev(ctx_p->drvdata); + u32 tmp[DES3_EDE_EXPKEY_WORDS]; + struct cc_crypto_alg *cc_alg = + container_of(tfm->__crt_alg, struct cc_crypto_alg, + skcipher_alg.base); + unsigned int max_key_buf_size = cc_alg->skcipher_alg.max_keysize; + + dev_dbg(dev, "Setting key in context @%p for %s. keylen=%u\n", + ctx_p, crypto_tfm_alg_name(tfm), keylen); + dump_byte_array("key", (u8 *)key, keylen); - ctx_p->keylen = keylen; - dev_dbg(dev, "cc_is_hw_key ret 0"); + /* STAT_PHASE_0: Init and sanity checks */ - return 0; + if (validate_keys_sizes(ctx_p, keylen)) { + dev_err(dev, "Unsupported key size %d.\n", keylen); + crypto_tfm_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN); + return -EINVAL; } + ctx_p->hw_key = false; + /* * Verify DES weak keys * Note that we're dropping the expanded key since the @@ -734,6 +773,241 @@ static int cc_cipher_decrypt(struct skcipher_request *req) /* Block cipher alg */ static const struct cc_alg_template skcipher_algs[] = { + { + .name = "xts(paes)", + .driver_name = "xts-paes-ccree", + .blocksize = AES_BLOCK_SIZE, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_XTS, + .flow_mode = S_DIN_to_AES, + .min_hw_rev = CC_HW_REV_630, + }, + { + .name = "xts512(paes)", + .driver_name = "xts-paes-du512-ccree", + .blocksize = AES_BLOCK_SIZE, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_XTS, + .flow_mode = S_DIN_to_AES, + .data_unit = 512, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "xts4096(paes)", + .driver_name = "xts-paes-du4096-ccree", + .blocksize = AES_BLOCK_SIZE, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_XTS, + .flow_mode = S_DIN_to_AES, + .data_unit = 4096, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "essiv(paes)", + .driver_name = "essiv-paes-ccree", + .blocksize = AES_BLOCK_SIZE, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_ESSIV, + .flow_mode = S_DIN_to_AES, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "essiv512(paes)", + .driver_name = "essiv-paes-du512-ccree", + .blocksize = AES_BLOCK_SIZE, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_ESSIV, + .flow_mode = S_DIN_to_AES, + .data_unit = 512, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "essiv4096(paes)", + .driver_name = "essiv-paes-du4096-ccree", + .blocksize = AES_BLOCK_SIZE, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_ESSIV, + .flow_mode = S_DIN_to_AES, + .data_unit = 4096, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "bitlocker(paes)", + .driver_name = "bitlocker-paes-ccree", + .blocksize = AES_BLOCK_SIZE, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_BITLOCKER, + .flow_mode = S_DIN_to_AES, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "bitlocker512(paes)", + .driver_name = "bitlocker-paes-du512-ccree", + .blocksize = AES_BLOCK_SIZE, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_BITLOCKER, + .flow_mode = S_DIN_to_AES, + .data_unit = 512, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "bitlocker4096(paes)", + .driver_name = "bitlocker-paes-du4096-ccree", + .blocksize = AES_BLOCK_SIZE, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_BITLOCKER, + .flow_mode = S_DIN_to_AES, + .data_unit = 4096, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "ecb(paes)", + .driver_name = "ecb-paes-ccree", + .blocksize = AES_BLOCK_SIZE, + .type = CRYPTO_ALG_TYPE_ABLKCIPHER, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = 0, + }, + .cipher_mode = DRV_CIPHER_ECB, + .flow_mode = S_DIN_to_AES, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "cbc(paes)", + .driver_name = "cbc-paes-ccree", + .blocksize = AES_BLOCK_SIZE, + .type = CRYPTO_ALG_TYPE_ABLKCIPHER, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_CBC, + .flow_mode = S_DIN_to_AES, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "ofb(paes)", + .driver_name = "ofb-paes-ccree", + .blocksize = AES_BLOCK_SIZE, + .type = CRYPTO_ALG_TYPE_ABLKCIPHER, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_OFB, + .flow_mode = S_DIN_to_AES, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "cts1(cbc(paes))", + .driver_name = "cts1-cbc-paes-ccree", + .blocksize = AES_BLOCK_SIZE, + .type = CRYPTO_ALG_TYPE_ABLKCIPHER, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_CBC_CTS, + .flow_mode = S_DIN_to_AES, + .min_hw_rev = CC_HW_REV_712, + }, + { + .name = "ctr(paes)", + .driver_name = "ctr-paes-ccree", + .blocksize = 1, + .type = CRYPTO_ALG_TYPE_ABLKCIPHER, + .template_skcipher = { + .setkey = cc_cipher_sethkey, + .encrypt = cc_cipher_encrypt, + .decrypt = cc_cipher_decrypt, + .min_keysize = CC_HW_KEY_SIZE, + .max_keysize = CC_HW_KEY_SIZE, + .ivsize = AES_BLOCK_SIZE, + }, + .cipher_mode = DRV_CIPHER_CTR, + .flow_mode = S_DIN_to_AES, + .min_hw_rev = CC_HW_REV_712, + }, { .name = "xts(aes)", .driver_name = "xts-aes-ccree", diff --git a/drivers/crypto/ccree/cc_cipher.h b/drivers/crypto/ccree/cc_cipher.h index 2a2a6f46c515..68444cfa936b 100644 --- a/drivers/crypto/ccree/cc_cipher.h +++ b/drivers/crypto/ccree/cc_cipher.h @@ -13,18 +13,6 @@ #include "cc_driver.h" #include "cc_buffer_mgr.h" -/* Crypto cipher flags */ -#define CC_CRYPTO_CIPHER_KEY_KFDE0 BIT(0) -#define CC_CRYPTO_CIPHER_KEY_KFDE1 BIT(1) -#define CC_CRYPTO_CIPHER_KEY_KFDE2 BIT(2) -#define CC_CRYPTO_CIPHER_KEY_KFDE3 BIT(3) -#define CC_CRYPTO_CIPHER_DU_SIZE_512B BIT(4) - -#define CC_CRYPTO_CIPHER_KEY_KFDE_MASK (CC_CRYPTO_CIPHER_KEY_KFDE0 | \ - CC_CRYPTO_CIPHER_KEY_KFDE1 | \ - CC_CRYPTO_CIPHER_KEY_KFDE2 | \ - CC_CRYPTO_CIPHER_KEY_KFDE3) - struct cipher_req_ctx { struct async_gen_req_ctx gen_ctx; enum cc_req_dma_buf_type dma_buf_type; @@ -42,18 +30,12 @@ int cc_cipher_alloc(struct cc_drvdata *drvdata); int cc_cipher_free(struct cc_drvdata *drvdata); -struct arm_hw_key_info { - int hw_key1; - int hw_key2; -}; +struct cc_hkey_info { + u16 keylen; + u8 hw_key1; + u8 hw_key2; +} __packed; -/* - * This is a stub function that will replaced when we - * implement secure keys - */ -static inline bool cc_is_hw_key(struct crypto_tfm *tfm) -{ - return false; -} +#define CC_HW_KEY_SIZE sizeof(struct cc_hkey_info) #endif /*__CC_CIPHER_H__*/ diff --git a/drivers/crypto/ccree/cc_debugfs.c b/drivers/crypto/ccree/cc_debugfs.c index 08f8db489cf0..5ca184e42483 100644 --- a/drivers/crypto/ccree/cc_debugfs.c +++ b/drivers/crypto/ccree/cc_debugfs.c @@ -26,7 +26,8 @@ struct cc_debugfs_ctx { static struct dentry *cc_debugfs_dir; static struct debugfs_reg32 debug_regs[] = { - CC_DEBUG_REG(HOST_SIGNATURE), + { .name = "SIGNATURE" }, /* Must be 0th */ + { .name = "VERSION" }, /* Must be 1st */ CC_DEBUG_REG(HOST_IRR), CC_DEBUG_REG(HOST_POWER_DOWN_EN), CC_DEBUG_REG(AXIM_MON_ERR), @@ -34,7 +35,6 @@ static struct debugfs_reg32 debug_regs[] = { CC_DEBUG_REG(HOST_IMR), CC_DEBUG_REG(AXIM_CFG), CC_DEBUG_REG(AXIM_CACHE_PARAMS), - CC_DEBUG_REG(HOST_VERSION), CC_DEBUG_REG(GPR_HOST), CC_DEBUG_REG(AXIM_MON_COMP), }; @@ -58,6 +58,9 @@ int cc_debugfs_init(struct cc_drvdata *drvdata) struct debugfs_regset32 *regset; struct dentry *file; + debug_regs[0].offset = drvdata->sig_offset; + debug_regs[1].offset = drvdata->ver_offset; + ctx = devm_kzalloc(dev, sizeof(*ctx), GFP_KERNEL); if (!ctx) return -ENOMEM; diff --git a/drivers/crypto/ccree/cc_driver.c b/drivers/crypto/ccree/cc_driver.c index 89ce013ae093..bd974fef05e4 100644 --- a/drivers/crypto/ccree/cc_driver.c +++ b/drivers/crypto/ccree/cc_driver.c @@ -168,14 +168,14 @@ int init_cc_regs(struct cc_drvdata *drvdata, bool is_probe) val = cc_ioread(drvdata, CC_REG(AXIM_CACHE_PARAMS)); if (is_probe) - dev_info(dev, "Cache params previous: 0x%08X\n", val); + dev_dbg(dev, "Cache params previous: 0x%08X\n", val); cc_iowrite(drvdata, CC_REG(AXIM_CACHE_PARAMS), cache_params); val = cc_ioread(drvdata, CC_REG(AXIM_CACHE_PARAMS)); if (is_probe) - dev_info(dev, "Cache params current: 0x%08X (expect: 0x%08X)\n", - val, cache_params); + dev_dbg(dev, "Cache params current: 0x%08X (expect: 0x%08X)\n", + val, cache_params); return 0; } @@ -190,6 +190,7 @@ static int init_cc_resources(struct platform_device *plat_dev) u64 dma_mask; const struct cc_hw_data *hw_rev; const struct of_device_id *dev_id; + struct clk *clk; int rc = 0; new_drvdata = devm_kzalloc(dev, sizeof(*new_drvdata), GFP_KERNEL); @@ -207,15 +208,36 @@ static int init_cc_resources(struct platform_device *plat_dev) if (hw_rev->rev >= CC_HW_REV_712) { new_drvdata->hash_len_sz = HASH_LEN_SIZE_712; new_drvdata->axim_mon_offset = CC_REG(AXIM_MON_COMP); + new_drvdata->sig_offset = CC_REG(HOST_SIGNATURE_712); + new_drvdata->ver_offset = CC_REG(HOST_VERSION_712); } else { new_drvdata->hash_len_sz = HASH_LEN_SIZE_630; new_drvdata->axim_mon_offset = CC_REG(AXIM_MON_COMP8); + new_drvdata->sig_offset = CC_REG(HOST_SIGNATURE_630); + new_drvdata->ver_offset = CC_REG(HOST_VERSION_630); } platform_set_drvdata(plat_dev, new_drvdata); new_drvdata->plat_dev = plat_dev; - new_drvdata->clk = of_clk_get(np, 0); + clk = devm_clk_get(dev, NULL); + if (IS_ERR(clk)) + switch (PTR_ERR(clk)) { + /* Clock is optional so this might be fine */ + case -ENOENT: + break; + + /* Clock not available, let's try again soon */ + case -EPROBE_DEFER: + return -EPROBE_DEFER; + + default: + dev_err(dev, "Error getting clock: %ld\n", + PTR_ERR(clk)); + return PTR_ERR(clk); + } + new_drvdata->clk = clk; + new_drvdata->coherent = of_dma_is_coherent(np); /* Get device resources */ @@ -265,7 +287,7 @@ static int init_cc_resources(struct platform_device *plat_dev) } if (rc) { - dev_err(dev, "Failed in dma_set_mask, mask=%pad\n", &dma_mask); + dev_err(dev, "Failed in dma_set_mask, mask=%llx\n", dma_mask); return rc; } @@ -276,7 +298,7 @@ static int init_cc_resources(struct platform_device *plat_dev) } /* Verify correct mapping */ - signature_val = cc_ioread(new_drvdata, CC_REG(HOST_SIGNATURE)); + signature_val = cc_ioread(new_drvdata, new_drvdata->sig_offset); if (signature_val != hw_rev->sig) { dev_err(dev, "Invalid CC signature: SIGNATURE=0x%08X != expected=0x%08X\n", signature_val, hw_rev->sig); @@ -287,7 +309,7 @@ static int init_cc_resources(struct platform_device *plat_dev) /* Display HW versions */ dev_info(dev, "ARM CryptoCell %s Driver: HW version 0x%08X, Driver version %s\n", - hw_rev->name, cc_ioread(new_drvdata, CC_REG(HOST_VERSION)), + hw_rev->name, cc_ioread(new_drvdata, new_drvdata->ver_offset), DRV_MODULE_VERSION); rc = init_cc_regs(new_drvdata, true); diff --git a/drivers/crypto/ccree/cc_driver.h b/drivers/crypto/ccree/cc_driver.h index 2048fdeb9579..95f82b2d1e70 100644 --- a/drivers/crypto/ccree/cc_driver.h +++ b/drivers/crypto/ccree/cc_driver.h @@ -129,6 +129,8 @@ struct cc_drvdata { enum cc_hw_rev hw_rev; u32 hash_len_sz; u32 axim_mon_offset; + u32 sig_offset; + u32 ver_offset; }; struct cc_crypto_alg { diff --git a/drivers/crypto/ccree/cc_host_regs.h b/drivers/crypto/ccree/cc_host_regs.h index f51001898ca1..616b2e1c41ba 100644 --- a/drivers/crypto/ccree/cc_host_regs.h +++ b/drivers/crypto/ccree/cc_host_regs.h @@ -45,7 +45,8 @@ #define CC_HOST_ICR_DSCRPTR_WATERMARK_QUEUE0_CLEAR_BIT_SIZE 0x1UL #define CC_HOST_ICR_AXIM_COMP_INT_CLEAR_BIT_SHIFT 0x17UL #define CC_HOST_ICR_AXIM_COMP_INT_CLEAR_BIT_SIZE 0x1UL -#define CC_HOST_SIGNATURE_REG_OFFSET 0xA24UL +#define CC_HOST_SIGNATURE_712_REG_OFFSET 0xA24UL +#define CC_HOST_SIGNATURE_630_REG_OFFSET 0xAC8UL #define CC_HOST_SIGNATURE_VALUE_BIT_SHIFT 0x0UL #define CC_HOST_SIGNATURE_VALUE_BIT_SIZE 0x20UL #define CC_HOST_BOOT_REG_OFFSET 0xA28UL @@ -105,7 +106,8 @@ #define CC_HOST_BOOT_ONLY_ENCRYPT_LOCAL_BIT_SIZE 0x1UL #define CC_HOST_BOOT_AES_EXISTS_LOCAL_BIT_SHIFT 0x1EUL #define CC_HOST_BOOT_AES_EXISTS_LOCAL_BIT_SIZE 0x1UL -#define CC_HOST_VERSION_REG_OFFSET 0xA40UL +#define CC_HOST_VERSION_712_REG_OFFSET 0xA40UL +#define CC_HOST_VERSION_630_REG_OFFSET 0xAD8UL #define CC_HOST_VERSION_VALUE_BIT_SHIFT 0x0UL #define CC_HOST_VERSION_VALUE_BIT_SIZE 0x20UL #define CC_HOST_KFDE0_VALID_REG_OFFSET 0xA60UL diff --git a/drivers/crypto/chelsio/chcr_algo.c b/drivers/crypto/chelsio/chcr_algo.c index 59fe6631e73e..b916c4eb608c 100644 --- a/drivers/crypto/chelsio/chcr_algo.c +++ b/drivers/crypto/chelsio/chcr_algo.c @@ -203,13 +203,8 @@ static inline void chcr_handle_aead_resp(struct aead_request *req, int err) { struct chcr_aead_reqctx *reqctx = aead_request_ctx(req); - struct crypto_aead *tfm = crypto_aead_reqtfm(req); - struct uld_ctx *u_ctx = ULD_CTX(a_ctx(tfm)); - chcr_aead_dma_unmap(&u_ctx->lldi.pdev->dev, req, reqctx->op); - if (reqctx->b0_dma) - dma_unmap_single(&u_ctx->lldi.pdev->dev, reqctx->b0_dma, - reqctx->b0_len, DMA_BIDIRECTIONAL); + chcr_aead_common_exit(req); if (reqctx->verify == VERIFY_SW) { chcr_verify_tag(req, input, &err); reqctx->verify = VERIFY_HW; @@ -638,7 +633,6 @@ static int chcr_sg_ent_in_wr(struct scatterlist *src, src = sg_next(src); srcskip = 0; } - if (sg_dma_len(dst) == dstskip) { dst = sg_next(dst); dstskip = 0; @@ -688,6 +682,7 @@ static int chcr_cipher_fallback(struct crypto_skcipher *cipher, int err; SKCIPHER_REQUEST_ON_STACK(subreq, cipher); + skcipher_request_set_tfm(subreq, cipher); skcipher_request_set_callback(subreq, flags, NULL, NULL); skcipher_request_set_crypt(subreq, src, dst, @@ -760,13 +755,13 @@ static struct sk_buff *create_cipher_wr(struct cipher_wr_param *wrparam) nents = sg_nents_xlen(reqctx->dstsg, wrparam->bytes, CHCR_DST_SG_SIZE, reqctx->dst_ofst); - dst_size = get_space_for_phys_dsgl(nents + 1); + dst_size = get_space_for_phys_dsgl(nents); kctx_len = roundup(ablkctx->enckey_len, 16); transhdr_len = CIPHER_TRANSHDR_SIZE(kctx_len, dst_size); nents = sg_nents_xlen(reqctx->srcsg, wrparam->bytes, CHCR_SRC_SG_SIZE, reqctx->src_ofst); - temp = reqctx->imm ? roundup(IV + wrparam->req->nbytes, 16) : - (sgl_len(nents + MIN_CIPHER_SG) * 8); + temp = reqctx->imm ? roundup(wrparam->bytes, 16) : + (sgl_len(nents) * 8); transhdr_len += temp; transhdr_len = roundup(transhdr_len, 16); skb = alloc_skb(SGE_MAX_WR_LEN, flags); @@ -788,7 +783,7 @@ static struct sk_buff *create_cipher_wr(struct cipher_wr_param *wrparam) ablkctx->ciph_mode, 0, 0, IV >> 1); chcr_req->sec_cpl.ivgen_hdrlen = FILL_SEC_CPL_IVGEN_HDRLEN(0, 0, 0, - 0, 0, dst_size); + 0, 1, dst_size); chcr_req->key_ctx.ctx_hdr = ablkctx->key_ctx_hdr; if ((reqctx->op == CHCR_DECRYPT_OP) && @@ -818,8 +813,8 @@ static struct sk_buff *create_cipher_wr(struct cipher_wr_param *wrparam) chcr_add_cipher_dst_ent(wrparam->req, phys_cpl, wrparam, wrparam->qid); atomic_inc(&adap->chcr_stats.cipher_rqst); - temp = sizeof(struct cpl_rx_phys_dsgl) + dst_size + kctx_len - +(reqctx->imm ? (IV + wrparam->bytes) : 0); + temp = sizeof(struct cpl_rx_phys_dsgl) + dst_size + kctx_len + IV + + (reqctx->imm ? (wrparam->bytes) : 0); create_wreq(c_ctx(tfm), chcr_req, &(wrparam->req->base), reqctx->imm, 0, transhdr_len, temp, ablkctx->ciph_mode == CHCR_SCMD_CIPHER_MODE_AES_CBC); @@ -1022,7 +1017,7 @@ static int chcr_update_tweak(struct ablkcipher_request *req, u8 *iv, ret = crypto_cipher_setkey(cipher, key, keylen); if (ret) goto out; - /*H/W sends the encrypted IV in dsgl when AADIVDROP bit is 0*/ + crypto_cipher_encrypt_one(cipher, iv, iv); for (i = 0; i < round8; i++) gf128mul_x8_ble((le128 *)iv, (le128 *)iv); @@ -1113,16 +1108,8 @@ static int chcr_handle_cipher_resp(struct ablkcipher_request *req, goto complete; } - if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0], - c_ctx(tfm)->tx_qidx))) { - if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG)) { - err = -EBUSY; - goto unmap; - } - - } if (!reqctx->imm) { - bytes = chcr_sg_ent_in_wr(reqctx->srcsg, reqctx->dstsg, 1, + bytes = chcr_sg_ent_in_wr(reqctx->srcsg, reqctx->dstsg, 0, CIP_SPACE_LEFT(ablkctx->enckey_len), reqctx->src_ofst, reqctx->dst_ofst); if ((bytes + reqctx->processed) >= req->nbytes) @@ -1133,11 +1120,7 @@ static int chcr_handle_cipher_resp(struct ablkcipher_request *req, /*CTR mode counter overfloa*/ bytes = req->nbytes - reqctx->processed; } - dma_sync_single_for_cpu(&ULD_CTX(c_ctx(tfm))->lldi.pdev->dev, - reqctx->iv_dma, IV, DMA_BIDIRECTIONAL); err = chcr_update_cipher_iv(req, fw6_pld, reqctx->iv); - dma_sync_single_for_device(&ULD_CTX(c_ctx(tfm))->lldi.pdev->dev, - reqctx->iv_dma, IV, DMA_BIDIRECTIONAL); if (err) goto unmap; @@ -1212,7 +1195,6 @@ static int process_cipher(struct ablkcipher_request *req, dnents = sg_nents_xlen(req->dst, req->nbytes, CHCR_DST_SG_SIZE, 0); - dnents += 1; // IV phys_dsgl = get_space_for_phys_dsgl(dnents); kctx_len = roundup(ablkctx->enckey_len, 16); transhdr_len = CIPHER_TRANSHDR_SIZE(kctx_len, phys_dsgl); @@ -1225,8 +1207,7 @@ static int process_cipher(struct ablkcipher_request *req, } if (!reqctx->imm) { - bytes = chcr_sg_ent_in_wr(req->src, req->dst, - MIN_CIPHER_SG, + bytes = chcr_sg_ent_in_wr(req->src, req->dst, 0, CIP_SPACE_LEFT(ablkctx->enckey_len), 0, 0); if ((bytes + reqctx->processed) >= req->nbytes) @@ -1293,13 +1274,14 @@ static int chcr_aes_encrypt(struct ablkcipher_request *req) { struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req); struct sk_buff *skb = NULL; - int err; + int err, isfull = 0; struct uld_ctx *u_ctx = ULD_CTX(c_ctx(tfm)); if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0], c_ctx(tfm)->tx_qidx))) { + isfull = 1; if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG)) - return -EBUSY; + return -ENOSPC; } err = process_cipher(req, u_ctx->lldi.rxq_ids[c_ctx(tfm)->rx_qidx], @@ -1309,7 +1291,7 @@ static int chcr_aes_encrypt(struct ablkcipher_request *req) skb->dev = u_ctx->lldi.ports[0]; set_wr_txq(skb, CPL_PRIORITY_DATA, c_ctx(tfm)->tx_qidx); chcr_send_wr(skb); - return -EINPROGRESS; + return isfull ? -EBUSY : -EINPROGRESS; } static int chcr_aes_decrypt(struct ablkcipher_request *req) @@ -1317,12 +1299,13 @@ static int chcr_aes_decrypt(struct ablkcipher_request *req) struct crypto_ablkcipher *tfm = crypto_ablkcipher_reqtfm(req); struct uld_ctx *u_ctx = ULD_CTX(c_ctx(tfm)); struct sk_buff *skb = NULL; - int err; + int err, isfull = 0; if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0], c_ctx(tfm)->tx_qidx))) { + isfull = 1; if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG)) - return -EBUSY; + return -ENOSPC; } err = process_cipher(req, u_ctx->lldi.rxq_ids[c_ctx(tfm)->rx_qidx], @@ -1332,7 +1315,7 @@ static int chcr_aes_decrypt(struct ablkcipher_request *req) skb->dev = u_ctx->lldi.ports[0]; set_wr_txq(skb, CPL_PRIORITY_DATA, c_ctx(tfm)->tx_qidx); chcr_send_wr(skb); - return -EINPROGRESS; + return isfull ? -EBUSY : -EINPROGRESS; } static int chcr_device_init(struct chcr_context *ctx) @@ -1574,14 +1557,15 @@ static int chcr_ahash_update(struct ahash_request *req) u8 remainder = 0, bs; unsigned int nbytes = req->nbytes; struct hash_wr_param params; - int error; + int error, isfull = 0; bs = crypto_tfm_alg_blocksize(crypto_ahash_tfm(rtfm)); u_ctx = ULD_CTX(h_ctx(rtfm)); if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0], h_ctx(rtfm)->tx_qidx))) { + isfull = 1; if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG)) - return -EBUSY; + return -ENOSPC; } if (nbytes + req_ctx->reqlen >= bs) { @@ -1633,7 +1617,7 @@ static int chcr_ahash_update(struct ahash_request *req) set_wr_txq(skb, CPL_PRIORITY_DATA, h_ctx(rtfm)->tx_qidx); chcr_send_wr(skb); - return -EINPROGRESS; + return isfull ? -EBUSY : -EINPROGRESS; unmap: chcr_hash_dma_unmap(&u_ctx->lldi.pdev->dev, req); return error; @@ -1710,15 +1694,16 @@ static int chcr_ahash_finup(struct ahash_request *req) struct sk_buff *skb; struct hash_wr_param params; u8 bs; - int error; + int error, isfull = 0; bs = crypto_tfm_alg_blocksize(crypto_ahash_tfm(rtfm)); u_ctx = ULD_CTX(h_ctx(rtfm)); if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0], h_ctx(rtfm)->tx_qidx))) { + isfull = 1; if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG)) - return -EBUSY; + return -ENOSPC; } chcr_init_hctx_per_wr(req_ctx); error = chcr_hash_dma_map(&u_ctx->lldi.pdev->dev, req); @@ -1777,7 +1762,7 @@ static int chcr_ahash_finup(struct ahash_request *req) set_wr_txq(skb, CPL_PRIORITY_DATA, h_ctx(rtfm)->tx_qidx); chcr_send_wr(skb); - return -EINPROGRESS; + return isfull ? -EBUSY : -EINPROGRESS; unmap: chcr_hash_dma_unmap(&u_ctx->lldi.pdev->dev, req); return error; @@ -1791,7 +1776,7 @@ static int chcr_ahash_digest(struct ahash_request *req) struct sk_buff *skb; struct hash_wr_param params; u8 bs; - int error; + int error, isfull = 0; rtfm->init(req); bs = crypto_tfm_alg_blocksize(crypto_ahash_tfm(rtfm)); @@ -1799,8 +1784,9 @@ static int chcr_ahash_digest(struct ahash_request *req) u_ctx = ULD_CTX(h_ctx(rtfm)); if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0], h_ctx(rtfm)->tx_qidx))) { + isfull = 1; if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG)) - return -EBUSY; + return -ENOSPC; } chcr_init_hctx_per_wr(req_ctx); @@ -1856,7 +1842,7 @@ static int chcr_ahash_digest(struct ahash_request *req) skb->dev = u_ctx->lldi.ports[0]; set_wr_txq(skb, CPL_PRIORITY_DATA, h_ctx(rtfm)->tx_qidx); chcr_send_wr(skb); - return -EINPROGRESS; + return isfull ? -EBUSY : -EINPROGRESS; unmap: chcr_hash_dma_unmap(&u_ctx->lldi.pdev->dev, req); return error; @@ -1875,11 +1861,6 @@ static int chcr_ahash_continue(struct ahash_request *req) bs = crypto_tfm_alg_blocksize(crypto_ahash_tfm(rtfm)); u_ctx = ULD_CTX(h_ctx(rtfm)); - if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0], - h_ctx(rtfm)->tx_qidx))) { - if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG)) - return -EBUSY; - } get_alg_config(¶ms.alg_prm, crypto_ahash_digestsize(rtfm)); params.kctx_len = roundup(params.alg_prm.result_size, 16); if (is_hmac(crypto_ahash_tfm(rtfm))) { @@ -2192,22 +2173,35 @@ static void chcr_hmac_cra_exit(struct crypto_tfm *tfm) } } -static int chcr_aead_common_init(struct aead_request *req, - unsigned short op_type) +inline void chcr_aead_common_exit(struct aead_request *req) +{ + struct chcr_aead_reqctx *reqctx = aead_request_ctx(req); + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct uld_ctx *u_ctx = ULD_CTX(a_ctx(tfm)); + + chcr_aead_dma_unmap(&u_ctx->lldi.pdev->dev, req, reqctx->op); +} + +static int chcr_aead_common_init(struct aead_request *req) { struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct chcr_aead_ctx *aeadctx = AEAD_CTX(a_ctx(tfm)); struct chcr_aead_reqctx *reqctx = aead_request_ctx(req); - int error = -EINVAL; unsigned int authsize = crypto_aead_authsize(tfm); + int error = -EINVAL; /* validate key size */ if (aeadctx->enckey_len == 0) goto err; - if (op_type && req->cryptlen < authsize) + if (reqctx->op && req->cryptlen < authsize) goto err; + if (reqctx->b0_len) + reqctx->scratch_pad = reqctx->iv + IV; + else + reqctx->scratch_pad = NULL; + error = chcr_aead_dma_map(&ULD_CTX(a_ctx(tfm))->lldi.pdev->dev, req, - op_type); + reqctx->op); if (error) { error = -ENOMEM; goto err; @@ -2244,7 +2238,7 @@ static int chcr_aead_fallback(struct aead_request *req, unsigned short op_type) aead_request_set_tfm(subreq, aeadctx->sw_cipher); aead_request_set_callback(subreq, req->base.flags, req->base.complete, req->base.data); - aead_request_set_crypt(subreq, req->src, req->dst, req->cryptlen, + aead_request_set_crypt(subreq, req->src, req->dst, req->cryptlen, req->iv); aead_request_set_ad(subreq, req->assoclen); return op_type ? crypto_aead_decrypt(subreq) : @@ -2253,8 +2247,7 @@ static int chcr_aead_fallback(struct aead_request *req, unsigned short op_type) static struct sk_buff *create_authenc_wr(struct aead_request *req, unsigned short qid, - int size, - unsigned short op_type) + int size) { struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct chcr_aead_ctx *aeadctx = AEAD_CTX(a_ctx(tfm)); @@ -2278,18 +2271,20 @@ static struct sk_buff *create_authenc_wr(struct aead_request *req, if (req->cryptlen == 0) return NULL; - reqctx->b0_dma = 0; + reqctx->b0_len = 0; + error = chcr_aead_common_init(req); + if (error) + return ERR_PTR(error); + if (subtype == CRYPTO_ALG_SUB_TYPE_CBC_NULL || - subtype == CRYPTO_ALG_SUB_TYPE_CTR_NULL) { + subtype == CRYPTO_ALG_SUB_TYPE_CTR_NULL) { null = 1; assoclen = 0; + reqctx->aad_nents = 0; } - error = chcr_aead_common_init(req, op_type); - if (error) - return ERR_PTR(error); dnents = sg_nents_xlen(req->dst, assoclen, CHCR_DST_SG_SIZE, 0); dnents += sg_nents_xlen(req->dst, req->cryptlen + - (op_type ? -authsize : authsize), CHCR_DST_SG_SIZE, + (reqctx->op ? -authsize : authsize), CHCR_DST_SG_SIZE, req->assoclen); dnents += MIN_AUTH_SG; // For IV @@ -2306,11 +2301,10 @@ static struct sk_buff *create_authenc_wr(struct aead_request *req, transhdr_len = roundup(transhdr_len, 16); if (chcr_aead_need_fallback(req, dnents, T6_MAX_AAD_SIZE, - transhdr_len, op_type)) { + transhdr_len, reqctx->op)) { atomic_inc(&adap->chcr_stats.fallback); - chcr_aead_dma_unmap(&ULD_CTX(a_ctx(tfm))->lldi.pdev->dev, req, - op_type); - return ERR_PTR(chcr_aead_fallback(req, op_type)); + chcr_aead_common_exit(req); + return ERR_PTR(chcr_aead_fallback(req, reqctx->op)); } skb = alloc_skb(SGE_MAX_WR_LEN, flags); if (!skb) { @@ -2320,7 +2314,7 @@ static struct sk_buff *create_authenc_wr(struct aead_request *req, chcr_req = __skb_put_zero(skb, transhdr_len); - temp = (op_type == CHCR_ENCRYPT_OP) ? 0 : authsize; + temp = (reqctx->op == CHCR_ENCRYPT_OP) ? 0 : authsize; /* * Input order is AAD,IV and Payload. where IV should be included as @@ -2344,8 +2338,8 @@ static struct sk_buff *create_authenc_wr(struct aead_request *req, temp = CHCR_SCMD_CIPHER_MODE_AES_CTR; else temp = CHCR_SCMD_CIPHER_MODE_AES_CBC; - chcr_req->sec_cpl.seqno_numivs = FILL_SEC_CPL_SCMD0_SEQNO(op_type, - (op_type == CHCR_ENCRYPT_OP) ? 1 : 0, + chcr_req->sec_cpl.seqno_numivs = FILL_SEC_CPL_SCMD0_SEQNO(reqctx->op, + (reqctx->op == CHCR_ENCRYPT_OP) ? 1 : 0, temp, actx->auth_mode, aeadctx->hmac_ctrl, IV >> 1); @@ -2353,7 +2347,7 @@ static struct sk_buff *create_authenc_wr(struct aead_request *req, 0, 0, dst_size); chcr_req->key_ctx.ctx_hdr = aeadctx->key_ctx_hdr; - if (op_type == CHCR_ENCRYPT_OP || + if (reqctx->op == CHCR_ENCRYPT_OP || subtype == CRYPTO_ALG_SUB_TYPE_CTR_SHA || subtype == CRYPTO_ALG_SUB_TYPE_CTR_NULL) memcpy(chcr_req->key_ctx.key, aeadctx->key, @@ -2376,20 +2370,18 @@ static struct sk_buff *create_authenc_wr(struct aead_request *req, } phys_cpl = (struct cpl_rx_phys_dsgl *)((u8 *)(chcr_req + 1) + kctx_len); ulptx = (struct ulptx_sgl *)((u8 *)(phys_cpl + 1) + dst_size); - chcr_add_aead_dst_ent(req, phys_cpl, assoclen, op_type, qid); - chcr_add_aead_src_ent(req, ulptx, assoclen, op_type); + chcr_add_aead_dst_ent(req, phys_cpl, assoclen, qid); + chcr_add_aead_src_ent(req, ulptx, assoclen); atomic_inc(&adap->chcr_stats.cipher_rqst); temp = sizeof(struct cpl_rx_phys_dsgl) + dst_size + kctx_len + (reqctx->imm ? (assoclen + IV + req->cryptlen) : 0); create_wreq(a_ctx(tfm), chcr_req, &req->base, reqctx->imm, size, transhdr_len, temp, 0); reqctx->skb = skb; - reqctx->op = op_type; return skb; err: - chcr_aead_dma_unmap(&ULD_CTX(a_ctx(tfm))->lldi.pdev->dev, req, - op_type); + chcr_aead_common_exit(req); return ERR_PTR(error); } @@ -2408,11 +2400,14 @@ int chcr_aead_dma_map(struct device *dev, -authsize : authsize); if (!req->cryptlen || !dst_size) return 0; - reqctx->iv_dma = dma_map_single(dev, reqctx->iv, IV, + reqctx->iv_dma = dma_map_single(dev, reqctx->iv, (IV + reqctx->b0_len), DMA_BIDIRECTIONAL); if (dma_mapping_error(dev, reqctx->iv_dma)) return -ENOMEM; - + if (reqctx->b0_len) + reqctx->b0_dma = reqctx->iv_dma + IV; + else + reqctx->b0_dma = 0; if (req->src == req->dst) { error = dma_map_sg(dev, req->src, sg_nents(req->src), DMA_BIDIRECTIONAL); @@ -2452,7 +2447,7 @@ void chcr_aead_dma_unmap(struct device *dev, if (!req->cryptlen || !dst_size) return; - dma_unmap_single(dev, reqctx->iv_dma, IV, + dma_unmap_single(dev, reqctx->iv_dma, (IV + reqctx->b0_len), DMA_BIDIRECTIONAL); if (req->src == req->dst) { dma_unmap_sg(dev, req->src, sg_nents(req->src), @@ -2467,8 +2462,7 @@ void chcr_aead_dma_unmap(struct device *dev, void chcr_add_aead_src_ent(struct aead_request *req, struct ulptx_sgl *ulptx, - unsigned int assoclen, - unsigned short op_type) + unsigned int assoclen) { struct ulptx_walk ulp_walk; struct chcr_aead_reqctx *reqctx = aead_request_ctx(req); @@ -2476,7 +2470,7 @@ void chcr_add_aead_src_ent(struct aead_request *req, if (reqctx->imm) { u8 *buf = (u8 *)ulptx; - if (reqctx->b0_dma) { + if (reqctx->b0_len) { memcpy(buf, reqctx->scratch_pad, reqctx->b0_len); buf += reqctx->b0_len; } @@ -2489,7 +2483,7 @@ void chcr_add_aead_src_ent(struct aead_request *req, buf, req->cryptlen, req->assoclen); } else { ulptx_walk_init(&ulp_walk, ulptx); - if (reqctx->b0_dma) + if (reqctx->b0_len) ulptx_walk_add_page(&ulp_walk, reqctx->b0_len, &reqctx->b0_dma); ulptx_walk_add_sg(&ulp_walk, req->src, assoclen, 0); @@ -2503,7 +2497,6 @@ void chcr_add_aead_src_ent(struct aead_request *req, void chcr_add_aead_dst_ent(struct aead_request *req, struct cpl_rx_phys_dsgl *phys_cpl, unsigned int assoclen, - unsigned short op_type, unsigned short qid) { struct chcr_aead_reqctx *reqctx = aead_request_ctx(req); @@ -2513,32 +2506,30 @@ void chcr_add_aead_dst_ent(struct aead_request *req, u32 temp; dsgl_walk_init(&dsgl_walk, phys_cpl); - if (reqctx->b0_dma) + if (reqctx->b0_len) dsgl_walk_add_page(&dsgl_walk, reqctx->b0_len, &reqctx->b0_dma); dsgl_walk_add_sg(&dsgl_walk, req->dst, assoclen, 0); dsgl_walk_add_page(&dsgl_walk, IV, &reqctx->iv_dma); - temp = req->cryptlen + (op_type ? -authsize : authsize); + temp = req->cryptlen + (reqctx->op ? -authsize : authsize); dsgl_walk_add_sg(&dsgl_walk, req->dst, temp, req->assoclen); dsgl_walk_end(&dsgl_walk, qid); } void chcr_add_cipher_src_ent(struct ablkcipher_request *req, - struct ulptx_sgl *ulptx, + void *ulptx, struct cipher_wr_param *wrparam) { struct ulptx_walk ulp_walk; struct chcr_blkcipher_req_ctx *reqctx = ablkcipher_request_ctx(req); + u8 *buf = ulptx; + memcpy(buf, reqctx->iv, IV); + buf += IV; if (reqctx->imm) { - u8 *buf = (u8 *)ulptx; - - memcpy(buf, reqctx->iv, IV); - buf += IV; sg_pcopy_to_buffer(req->src, sg_nents(req->src), buf, wrparam->bytes, reqctx->processed); } else { - ulptx_walk_init(&ulp_walk, ulptx); - ulptx_walk_add_page(&ulp_walk, IV, &reqctx->iv_dma); + ulptx_walk_init(&ulp_walk, (struct ulptx_sgl *)buf); ulptx_walk_add_sg(&ulp_walk, reqctx->srcsg, wrparam->bytes, reqctx->src_ofst); reqctx->srcsg = ulp_walk.last_sg; @@ -2556,7 +2547,6 @@ void chcr_add_cipher_dst_ent(struct ablkcipher_request *req, struct dsgl_walk dsgl_walk; dsgl_walk_init(&dsgl_walk, phys_cpl); - dsgl_walk_add_page(&dsgl_walk, IV, &reqctx->iv_dma); dsgl_walk_add_sg(&dsgl_walk, reqctx->dstsg, wrparam->bytes, reqctx->dst_ofst); reqctx->dstsg = dsgl_walk.last_sg; @@ -2630,12 +2620,6 @@ int chcr_cipher_dma_map(struct device *dev, struct ablkcipher_request *req) { int error; - struct chcr_blkcipher_req_ctx *reqctx = ablkcipher_request_ctx(req); - - reqctx->iv_dma = dma_map_single(dev, reqctx->iv, IV, - DMA_BIDIRECTIONAL); - if (dma_mapping_error(dev, reqctx->iv_dma)) - return -ENOMEM; if (req->src == req->dst) { error = dma_map_sg(dev, req->src, sg_nents(req->src), @@ -2658,17 +2642,12 @@ int chcr_cipher_dma_map(struct device *dev, return 0; err: - dma_unmap_single(dev, reqctx->iv_dma, IV, DMA_BIDIRECTIONAL); return -ENOMEM; } void chcr_cipher_dma_unmap(struct device *dev, struct ablkcipher_request *req) { - struct chcr_blkcipher_req_ctx *reqctx = ablkcipher_request_ctx(req); - - dma_unmap_single(dev, reqctx->iv_dma, IV, - DMA_BIDIRECTIONAL); if (req->src == req->dst) { dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_BIDIRECTIONAL); @@ -2738,7 +2717,8 @@ static inline int crypto_ccm_check_iv(const u8 *iv) static int ccm_format_packet(struct aead_request *req, struct chcr_aead_ctx *aeadctx, unsigned int sub_type, - unsigned short op_type) + unsigned short op_type, + unsigned int assoclen) { struct chcr_aead_reqctx *reqctx = aead_request_ctx(req); int rc = 0; @@ -2748,13 +2728,13 @@ static int ccm_format_packet(struct aead_request *req, memcpy(reqctx->iv + 1, &aeadctx->salt[0], 3); memcpy(reqctx->iv + 4, req->iv, 8); memset(reqctx->iv + 12, 0, 4); - *((unsigned short *)(reqctx->scratch_pad + 16)) = - htons(req->assoclen - 8); } else { memcpy(reqctx->iv, req->iv, 16); - *((unsigned short *)(reqctx->scratch_pad + 16)) = - htons(req->assoclen); } + if (assoclen) + *((unsigned short *)(reqctx->scratch_pad + 16)) = + htons(assoclen); + generate_b0(req, aeadctx, op_type); /* zero the ctr value */ memset(reqctx->iv + 15 - reqctx->iv[0], 0, reqctx->iv[0] + 1); @@ -2836,8 +2816,7 @@ static int aead_ccm_validate_input(unsigned short op_type, static struct sk_buff *create_aead_ccm_wr(struct aead_request *req, unsigned short qid, - int size, - unsigned short op_type) + int size) { struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct chcr_aead_ctx *aeadctx = AEAD_CTX(a_ctx(tfm)); @@ -2855,22 +2834,20 @@ static struct sk_buff *create_aead_ccm_wr(struct aead_request *req, GFP_ATOMIC; struct adapter *adap = padap(a_ctx(tfm)->dev); - reqctx->b0_dma = 0; sub_type = get_aead_subtype(tfm); if (sub_type == CRYPTO_ALG_SUB_TYPE_AEAD_RFC4309) assoclen -= 8; - error = chcr_aead_common_init(req, op_type); + reqctx->b0_len = CCM_B0_SIZE + (assoclen ? CCM_AAD_FIELD_SIZE : 0); + error = chcr_aead_common_init(req); if (error) return ERR_PTR(error); - - reqctx->b0_len = CCM_B0_SIZE + (assoclen ? CCM_AAD_FIELD_SIZE : 0); - error = aead_ccm_validate_input(op_type, req, aeadctx, sub_type); + error = aead_ccm_validate_input(reqctx->op, req, aeadctx, sub_type); if (error) goto err; dnents = sg_nents_xlen(req->dst, assoclen, CHCR_DST_SG_SIZE, 0); dnents += sg_nents_xlen(req->dst, req->cryptlen - + (op_type ? -authsize : authsize), + + (reqctx->op ? -authsize : authsize), CHCR_DST_SG_SIZE, req->assoclen); dnents += MIN_CCM_SG; // For IV and B0 dst_size = get_space_for_phys_dsgl(dnents); @@ -2886,11 +2863,10 @@ static struct sk_buff *create_aead_ccm_wr(struct aead_request *req, transhdr_len = roundup(transhdr_len, 16); if (chcr_aead_need_fallback(req, dnents, T6_MAX_AAD_SIZE - - reqctx->b0_len, transhdr_len, op_type)) { + reqctx->b0_len, transhdr_len, reqctx->op)) { atomic_inc(&adap->chcr_stats.fallback); - chcr_aead_dma_unmap(&ULD_CTX(a_ctx(tfm))->lldi.pdev->dev, req, - op_type); - return ERR_PTR(chcr_aead_fallback(req, op_type)); + chcr_aead_common_exit(req); + return ERR_PTR(chcr_aead_fallback(req, reqctx->op)); } skb = alloc_skb(SGE_MAX_WR_LEN, flags); @@ -2901,7 +2877,7 @@ static struct sk_buff *create_aead_ccm_wr(struct aead_request *req, chcr_req = (struct chcr_wr *) __skb_put_zero(skb, transhdr_len); - fill_sec_cpl_for_aead(&chcr_req->sec_cpl, dst_size, req, op_type); + fill_sec_cpl_for_aead(&chcr_req->sec_cpl, dst_size, req, reqctx->op); chcr_req->key_ctx.ctx_hdr = aeadctx->key_ctx_hdr; memcpy(chcr_req->key_ctx.key, aeadctx->key, aeadctx->enckey_len); @@ -2910,21 +2886,11 @@ static struct sk_buff *create_aead_ccm_wr(struct aead_request *req, phys_cpl = (struct cpl_rx_phys_dsgl *)((u8 *)(chcr_req + 1) + kctx_len); ulptx = (struct ulptx_sgl *)((u8 *)(phys_cpl + 1) + dst_size); - error = ccm_format_packet(req, aeadctx, sub_type, op_type); + error = ccm_format_packet(req, aeadctx, sub_type, reqctx->op, assoclen); if (error) goto dstmap_fail; - - reqctx->b0_dma = dma_map_single(&ULD_CTX(a_ctx(tfm))->lldi.pdev->dev, - &reqctx->scratch_pad, reqctx->b0_len, - DMA_BIDIRECTIONAL); - if (dma_mapping_error(&ULD_CTX(a_ctx(tfm))->lldi.pdev->dev, - reqctx->b0_dma)) { - error = -ENOMEM; - goto dstmap_fail; - } - - chcr_add_aead_dst_ent(req, phys_cpl, assoclen, op_type, qid); - chcr_add_aead_src_ent(req, ulptx, assoclen, op_type); + chcr_add_aead_dst_ent(req, phys_cpl, assoclen, qid); + chcr_add_aead_src_ent(req, ulptx, assoclen); atomic_inc(&adap->chcr_stats.aead_rqst); temp = sizeof(struct cpl_rx_phys_dsgl) + dst_size + @@ -2933,20 +2899,18 @@ static struct sk_buff *create_aead_ccm_wr(struct aead_request *req, create_wreq(a_ctx(tfm), chcr_req, &req->base, reqctx->imm, 0, transhdr_len, temp, 0); reqctx->skb = skb; - reqctx->op = op_type; return skb; dstmap_fail: kfree_skb(skb); err: - chcr_aead_dma_unmap(&ULD_CTX(a_ctx(tfm))->lldi.pdev->dev, req, op_type); + chcr_aead_common_exit(req); return ERR_PTR(error); } static struct sk_buff *create_gcm_wr(struct aead_request *req, unsigned short qid, - int size, - unsigned short op_type) + int size) { struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct chcr_aead_ctx *aeadctx = AEAD_CTX(a_ctx(tfm)); @@ -2966,13 +2930,13 @@ static struct sk_buff *create_gcm_wr(struct aead_request *req, if (get_aead_subtype(tfm) == CRYPTO_ALG_SUB_TYPE_AEAD_RFC4106) assoclen = req->assoclen - 8; - reqctx->b0_dma = 0; - error = chcr_aead_common_init(req, op_type); + reqctx->b0_len = 0; + error = chcr_aead_common_init(req); if (error) return ERR_PTR(error); dnents = sg_nents_xlen(req->dst, assoclen, CHCR_DST_SG_SIZE, 0); dnents += sg_nents_xlen(req->dst, req->cryptlen + - (op_type ? -authsize : authsize), + (reqctx->op ? -authsize : authsize), CHCR_DST_SG_SIZE, req->assoclen); dnents += MIN_GCM_SG; // For IV dst_size = get_space_for_phys_dsgl(dnents); @@ -2986,11 +2950,11 @@ static struct sk_buff *create_gcm_wr(struct aead_request *req, transhdr_len += temp; transhdr_len = roundup(transhdr_len, 16); if (chcr_aead_need_fallback(req, dnents, T6_MAX_AAD_SIZE, - transhdr_len, op_type)) { + transhdr_len, reqctx->op)) { + atomic_inc(&adap->chcr_stats.fallback); - chcr_aead_dma_unmap(&ULD_CTX(a_ctx(tfm))->lldi.pdev->dev, req, - op_type); - return ERR_PTR(chcr_aead_fallback(req, op_type)); + chcr_aead_common_exit(req); + return ERR_PTR(chcr_aead_fallback(req, reqctx->op)); } skb = alloc_skb(SGE_MAX_WR_LEN, flags); if (!skb) { @@ -3001,7 +2965,7 @@ static struct sk_buff *create_gcm_wr(struct aead_request *req, chcr_req = __skb_put_zero(skb, transhdr_len); //Offset of tag from end - temp = (op_type == CHCR_ENCRYPT_OP) ? 0 : authsize; + temp = (reqctx->op == CHCR_ENCRYPT_OP) ? 0 : authsize; chcr_req->sec_cpl.op_ivinsrtofst = FILL_SEC_CPL_OP_IVINSR( a_ctx(tfm)->dev->rx_channel_id, 2, (assoclen + 1)); @@ -3014,7 +2978,7 @@ static struct sk_buff *create_gcm_wr(struct aead_request *req, FILL_SEC_CPL_AUTHINSERT(0, assoclen + IV + 1, temp, temp); chcr_req->sec_cpl.seqno_numivs = - FILL_SEC_CPL_SCMD0_SEQNO(op_type, (op_type == + FILL_SEC_CPL_SCMD0_SEQNO(reqctx->op, (reqctx->op == CHCR_ENCRYPT_OP) ? 1 : 0, CHCR_SCMD_CIPHER_MODE_AES_GCM, CHCR_SCMD_AUTH_MODE_GHASH, @@ -3040,19 +3004,18 @@ static struct sk_buff *create_gcm_wr(struct aead_request *req, phys_cpl = (struct cpl_rx_phys_dsgl *)((u8 *)(chcr_req + 1) + kctx_len); ulptx = (struct ulptx_sgl *)((u8 *)(phys_cpl + 1) + dst_size); - chcr_add_aead_dst_ent(req, phys_cpl, assoclen, op_type, qid); - chcr_add_aead_src_ent(req, ulptx, assoclen, op_type); + chcr_add_aead_dst_ent(req, phys_cpl, assoclen, qid); + chcr_add_aead_src_ent(req, ulptx, assoclen); atomic_inc(&adap->chcr_stats.aead_rqst); temp = sizeof(struct cpl_rx_phys_dsgl) + dst_size + kctx_len + (reqctx->imm ? (assoclen + IV + req->cryptlen) : 0); create_wreq(a_ctx(tfm), chcr_req, &req->base, reqctx->imm, size, transhdr_len, temp, reqctx->verify); reqctx->skb = skb; - reqctx->op = op_type; return skb; err: - chcr_aead_dma_unmap(&ULD_CTX(a_ctx(tfm))->lldi.pdev->dev, req, op_type); + chcr_aead_common_exit(req); return ERR_PTR(error); } @@ -3461,6 +3424,7 @@ static int chcr_authenc_setkey(struct crypto_aead *authenc, const u8 *key, } { SHASH_DESC_ON_STACK(shash, base_hash); + shash->tfm = base_hash; shash->flags = crypto_shash_get_flags(base_hash); bs = crypto_shash_blocksize(base_hash); @@ -3585,13 +3549,13 @@ out: } static int chcr_aead_op(struct aead_request *req, - unsigned short op_type, int size, create_wr_t create_wr_fn) { struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct uld_ctx *u_ctx; struct sk_buff *skb; + int isfull = 0; if (!a_ctx(tfm)->dev) { pr_err("chcr : %s : No crypto device.\n", __func__); @@ -3600,13 +3564,13 @@ static int chcr_aead_op(struct aead_request *req, u_ctx = ULD_CTX(a_ctx(tfm)); if (cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0], a_ctx(tfm)->tx_qidx)) { + isfull = 1; if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG)) - return -EBUSY; + return -ENOSPC; } /* Form a WR from req */ - skb = create_wr_fn(req, u_ctx->lldi.rxq_ids[a_ctx(tfm)->rx_qidx], size, - op_type); + skb = create_wr_fn(req, u_ctx->lldi.rxq_ids[a_ctx(tfm)->rx_qidx], size); if (IS_ERR(skb) || !skb) return PTR_ERR(skb); @@ -3614,7 +3578,7 @@ static int chcr_aead_op(struct aead_request *req, skb->dev = u_ctx->lldi.ports[0]; set_wr_txq(skb, CPL_PRIORITY_DATA, a_ctx(tfm)->tx_qidx); chcr_send_wr(skb); - return -EINPROGRESS; + return isfull ? -EBUSY : -EINPROGRESS; } static int chcr_aead_encrypt(struct aead_request *req) @@ -3623,21 +3587,19 @@ static int chcr_aead_encrypt(struct aead_request *req) struct chcr_aead_reqctx *reqctx = aead_request_ctx(req); reqctx->verify = VERIFY_HW; + reqctx->op = CHCR_ENCRYPT_OP; switch (get_aead_subtype(tfm)) { case CRYPTO_ALG_SUB_TYPE_CTR_SHA: case CRYPTO_ALG_SUB_TYPE_CBC_SHA: case CRYPTO_ALG_SUB_TYPE_CBC_NULL: case CRYPTO_ALG_SUB_TYPE_CTR_NULL: - return chcr_aead_op(req, CHCR_ENCRYPT_OP, 0, - create_authenc_wr); + return chcr_aead_op(req, 0, create_authenc_wr); case CRYPTO_ALG_SUB_TYPE_AEAD_CCM: case CRYPTO_ALG_SUB_TYPE_AEAD_RFC4309: - return chcr_aead_op(req, CHCR_ENCRYPT_OP, 0, - create_aead_ccm_wr); + return chcr_aead_op(req, 0, create_aead_ccm_wr); default: - return chcr_aead_op(req, CHCR_ENCRYPT_OP, 0, - create_gcm_wr); + return chcr_aead_op(req, 0, create_gcm_wr); } } @@ -3655,21 +3617,18 @@ static int chcr_aead_decrypt(struct aead_request *req) size = 0; reqctx->verify = VERIFY_HW; } - + reqctx->op = CHCR_DECRYPT_OP; switch (get_aead_subtype(tfm)) { case CRYPTO_ALG_SUB_TYPE_CBC_SHA: case CRYPTO_ALG_SUB_TYPE_CTR_SHA: case CRYPTO_ALG_SUB_TYPE_CBC_NULL: case CRYPTO_ALG_SUB_TYPE_CTR_NULL: - return chcr_aead_op(req, CHCR_DECRYPT_OP, size, - create_authenc_wr); + return chcr_aead_op(req, size, create_authenc_wr); case CRYPTO_ALG_SUB_TYPE_AEAD_CCM: case CRYPTO_ALG_SUB_TYPE_AEAD_RFC4309: - return chcr_aead_op(req, CHCR_DECRYPT_OP, size, - create_aead_ccm_wr); + return chcr_aead_op(req, size, create_aead_ccm_wr); default: - return chcr_aead_op(req, CHCR_DECRYPT_OP, size, - create_gcm_wr); + return chcr_aead_op(req, size, create_gcm_wr); } } diff --git a/drivers/crypto/chelsio/chcr_algo.h b/drivers/crypto/chelsio/chcr_algo.h index dba3dff1e209..1871500309e2 100644 --- a/drivers/crypto/chelsio/chcr_algo.h +++ b/drivers/crypto/chelsio/chcr_algo.h @@ -146,7 +146,7 @@ kctx_len) #define CIPHER_TRANSHDR_SIZE(kctx_len, sge_pairs) \ (TRANSHDR_SIZE((kctx_len)) + (sge_pairs) +\ - sizeof(struct cpl_rx_phys_dsgl)) + sizeof(struct cpl_rx_phys_dsgl) + AES_BLOCK_SIZE) #define HASH_TRANSHDR_SIZE(kctx_len)\ (TRANSHDR_SIZE(kctx_len) + DUMMY_BYTES) @@ -259,7 +259,6 @@ ULP_TX_SC_MORE_V((immdatalen))) #define MAX_NK 8 #define MAX_DSGL_ENT 32 -#define MIN_CIPHER_SG 1 /* IV */ #define MIN_AUTH_SG 1 /* IV */ #define MIN_GCM_SG 1 /* IV */ #define MIN_DIGEST_SG 1 /*Partial Buffer*/ diff --git a/drivers/crypto/chelsio/chcr_core.h b/drivers/crypto/chelsio/chcr_core.h index 1a20424e18c6..de3a9c085daf 100644 --- a/drivers/crypto/chelsio/chcr_core.h +++ b/drivers/crypto/chelsio/chcr_core.h @@ -56,7 +56,7 @@ #define MAX_SALT 4 #define CIP_WR_MIN_LEN (sizeof(struct chcr_wr) + \ sizeof(struct cpl_rx_phys_dsgl) + \ - sizeof(struct ulptx_sgl)) + sizeof(struct ulptx_sgl) + 16) //IV #define HASH_WR_MIN_LEN (sizeof(struct chcr_wr) + \ DUMMY_BYTES + \ diff --git a/drivers/crypto/chelsio/chcr_crypto.h b/drivers/crypto/chelsio/chcr_crypto.h index c8e8972af283..54835cb109e5 100644 --- a/drivers/crypto/chelsio/chcr_crypto.h +++ b/drivers/crypto/chelsio/chcr_crypto.h @@ -190,8 +190,8 @@ struct chcr_aead_reqctx { short int dst_nents; u16 imm; u16 verify; - u8 iv[CHCR_MAX_CRYPTO_IV_LEN]; - unsigned char scratch_pad[MAX_SCRATCH_PAD_SIZE]; + u8 iv[CHCR_MAX_CRYPTO_IV_LEN + MAX_SCRATCH_PAD_SIZE]; + u8 *scratch_pad; }; struct ulptx_walk { @@ -295,7 +295,6 @@ struct chcr_blkcipher_req_ctx { unsigned int src_ofst; unsigned int dst_ofst; unsigned int op; - dma_addr_t iv_dma; u16 imm; u8 iv[CHCR_MAX_CRYPTO_IV_LEN]; }; @@ -312,8 +311,7 @@ struct chcr_alg_template { typedef struct sk_buff *(*create_wr_t)(struct aead_request *req, unsigned short qid, - int size, - unsigned short op_type); + int size); void chcr_verify_tag(struct aead_request *req, u8 *input, int *err); int chcr_aead_dma_map(struct device *dev, struct aead_request *req, @@ -322,12 +320,12 @@ void chcr_aead_dma_unmap(struct device *dev, struct aead_request *req, unsigned short op_type); void chcr_add_aead_dst_ent(struct aead_request *req, struct cpl_rx_phys_dsgl *phys_cpl, - unsigned int assoclen, unsigned short op_type, + unsigned int assoclen, unsigned short qid); void chcr_add_aead_src_ent(struct aead_request *req, struct ulptx_sgl *ulptx, - unsigned int assoclen, unsigned short op_type); + unsigned int assoclen); void chcr_add_cipher_src_ent(struct ablkcipher_request *req, - struct ulptx_sgl *ulptx, + void *ulptx, struct cipher_wr_param *wrparam); int chcr_cipher_dma_map(struct device *dev, struct ablkcipher_request *req); void chcr_cipher_dma_unmap(struct device *dev, struct ablkcipher_request *req); @@ -340,4 +338,5 @@ void chcr_add_hash_src_ent(struct ahash_request *req, struct ulptx_sgl *ulptx, struct hash_wr_param *param); int chcr_hash_dma_map(struct device *dev, struct ahash_request *req); void chcr_hash_dma_unmap(struct device *dev, struct ahash_request *req); +void chcr_aead_common_exit(struct aead_request *req); #endif /* __CHCR_CRYPTO_H__ */ diff --git a/drivers/crypto/chelsio/chcr_ipsec.c b/drivers/crypto/chelsio/chcr_ipsec.c index 8e0aa3f175c9..461b97e2f1fd 100644 --- a/drivers/crypto/chelsio/chcr_ipsec.c +++ b/drivers/crypto/chelsio/chcr_ipsec.c @@ -346,18 +346,23 @@ inline void *copy_cpltx_pktxt(struct sk_buff *skb, struct net_device *dev, void *pos) { + struct cpl_tx_pkt_core *cpl; + struct sge_eth_txq *q; struct adapter *adap; struct port_info *pi; - struct sge_eth_txq *q; - struct cpl_tx_pkt_core *cpl; - u64 cntrl = 0; u32 ctrl0, qidx; + u64 cntrl = 0; + int left; pi = netdev_priv(dev); adap = pi->adapter; qidx = skb->queue_mapping; q = &adap->sge.ethtxq[qidx + pi->first_qset]; + left = (void *)q->q.stat - pos; + if (!left) + pos = q->q.desc; + cpl = (struct cpl_tx_pkt_core *)pos; cntrl = TXPKT_L4CSUM_DIS_F | TXPKT_IPCSUM_DIS_F; @@ -382,18 +387,17 @@ inline void *copy_key_cpltx_pktxt(struct sk_buff *skb, void *pos, struct ipsec_sa_entry *sa_entry) { - struct adapter *adap; - struct port_info *pi; - struct sge_eth_txq *q; - unsigned int len, qidx; struct _key_ctx *key_ctx; int left, eoq, key_len; + struct sge_eth_txq *q; + struct adapter *adap; + struct port_info *pi; + unsigned int qidx; pi = netdev_priv(dev); adap = pi->adapter; qidx = skb->queue_mapping; q = &adap->sge.ethtxq[qidx + pi->first_qset]; - len = sa_entry->enckey_len + sizeof(struct cpl_tx_pkt_core); key_len = sa_entry->kctx_len; /* end of queue, reset pos to start of queue */ @@ -411,19 +415,14 @@ inline void *copy_key_cpltx_pktxt(struct sk_buff *skb, pos += sizeof(struct _key_ctx); left -= sizeof(struct _key_ctx); - if (likely(len <= left)) { + if (likely(key_len <= left)) { memcpy(key_ctx->key, sa_entry->key, key_len); pos += key_len; } else { - if (key_len <= left) { - memcpy(pos, sa_entry->key, key_len); - pos += key_len; - } else { - memcpy(pos, sa_entry->key, left); - memcpy(q->q.desc, sa_entry->key + left, - key_len - left); - pos = (u8 *)q->q.desc + (key_len - left); - } + memcpy(pos, sa_entry->key, left); + memcpy(q->q.desc, sa_entry->key + left, + key_len - left); + pos = (u8 *)q->q.desc + (key_len - left); } /* Copy CPL TX PKT XT */ pos = copy_cpltx_pktxt(skb, dev, pos); diff --git a/drivers/crypto/chelsio/chtls/chtls.h b/drivers/crypto/chelsio/chtls/chtls.h index f4b8f1ec0061..a53a0e6ba024 100644 --- a/drivers/crypto/chelsio/chtls/chtls.h +++ b/drivers/crypto/chelsio/chtls/chtls.h @@ -67,11 +67,6 @@ enum { CPL_RET_UNKNOWN_TID = 4 /* unexpected unknown TID */ }; -#define TLS_RCV_ST_READ_HEADER 0xF0 -#define TLS_RCV_ST_READ_BODY 0xF1 -#define TLS_RCV_ST_READ_DONE 0xF2 -#define TLS_RCV_ST_READ_NB 0xF3 - #define LISTEN_INFO_HASH_SIZE 32 #define RSPQ_HASH_BITS 5 struct listen_info { @@ -149,6 +144,7 @@ struct chtls_dev { struct list_head rcu_node; struct list_head na_node; unsigned int send_page_order; + int max_host_sndbuf; struct key_map kmap; }; @@ -278,6 +274,7 @@ struct tlsrx_cmp_hdr { #define TLSRX_HDR_PKT_MAC_ERROR_F TLSRX_HDR_PKT_MAC_ERROR_V(1U) #define TLSRX_HDR_PKT_ERROR_M 0x1F +#define CONTENT_TYPE_ERROR 0x7F struct ulp_mem_rw { __be32 cmd; @@ -347,8 +344,8 @@ enum { ULPCB_FLAG_HOLD = 1 << 3, /* skb not ready for Tx yet */ ULPCB_FLAG_COMPL = 1 << 4, /* request WR completion */ ULPCB_FLAG_URG = 1 << 5, /* urgent data */ - ULPCB_FLAG_TLS_ND = 1 << 6, /* payload of zero length */ - ULPCB_FLAG_NO_HDR = 1 << 7, /* not a ofld wr */ + ULPCB_FLAG_TLS_HDR = 1 << 6, /* payload with tls hdr */ + ULPCB_FLAG_NO_HDR = 1 << 7, /* not a ofld wr */ }; /* The ULP mode/submode of an skbuff */ diff --git a/drivers/crypto/chelsio/chtls/chtls_cm.c b/drivers/crypto/chelsio/chtls/chtls_cm.c index 82a473a0cefa..2bb6f0380758 100644 --- a/drivers/crypto/chelsio/chtls/chtls_cm.c +++ b/drivers/crypto/chelsio/chtls/chtls_cm.c @@ -1537,6 +1537,10 @@ static int chtls_rx_data(struct chtls_dev *cdev, struct sk_buff *skb) struct sock *sk; sk = lookup_tid(cdev->tids, hwtid); + if (unlikely(!sk)) { + pr_err("can't find conn. for hwtid %u.\n", hwtid); + return -EINVAL; + } skb_dst_set(skb, NULL); process_cpl_msg(chtls_recv_data, sk, skb); return 0; @@ -1585,6 +1589,10 @@ static int chtls_rx_pdu(struct chtls_dev *cdev, struct sk_buff *skb) struct sock *sk; sk = lookup_tid(cdev->tids, hwtid); + if (unlikely(!sk)) { + pr_err("can't find conn. for hwtid %u.\n", hwtid); + return -EINVAL; + } skb_dst_set(skb, NULL); process_cpl_msg(chtls_recv_pdu, sk, skb); return 0; @@ -1600,12 +1608,14 @@ static void chtls_set_hdrlen(struct sk_buff *skb, unsigned int nlen) static void chtls_rx_hdr(struct sock *sk, struct sk_buff *skb) { - struct cpl_rx_tls_cmp *cmp_cpl = cplhdr(skb); + struct tlsrx_cmp_hdr *tls_hdr_pkt; + struct cpl_rx_tls_cmp *cmp_cpl; struct sk_buff *skb_rec; struct chtls_sock *csk; struct chtls_hws *tlsk; struct tcp_sock *tp; + cmp_cpl = cplhdr(skb); csk = rcu_dereference_sk_user_data(sk); tlsk = &csk->tlshws; tp = tcp_sk(sk); @@ -1615,16 +1625,18 @@ static void chtls_rx_hdr(struct sock *sk, struct sk_buff *skb) skb_reset_transport_header(skb); __skb_pull(skb, sizeof(*cmp_cpl)); + tls_hdr_pkt = (struct tlsrx_cmp_hdr *)skb->data; + if (tls_hdr_pkt->res_to_mac_error & TLSRX_HDR_PKT_ERROR_M) + tls_hdr_pkt->type = CONTENT_TYPE_ERROR; if (!skb->data_len) - __skb_trim(skb, CPL_RX_TLS_CMP_LENGTH_G - (ntohl(cmp_cpl->pdulength_length))); + __skb_trim(skb, TLS_HEADER_LENGTH); tp->rcv_nxt += CPL_RX_TLS_CMP_PDULENGTH_G(ntohl(cmp_cpl->pdulength_length)); + ULP_SKB_CB(skb)->flags |= ULPCB_FLAG_TLS_HDR; skb_rec = __skb_dequeue(&tlsk->sk_recv_queue); if (!skb_rec) { - ULP_SKB_CB(skb)->flags |= ULPCB_FLAG_TLS_ND; __skb_queue_tail(&sk->sk_receive_queue, skb); } else { chtls_set_hdrlen(skb, tlsk->pldlen); @@ -1646,6 +1658,10 @@ static int chtls_rx_cmp(struct chtls_dev *cdev, struct sk_buff *skb) struct sock *sk; sk = lookup_tid(cdev->tids, hwtid); + if (unlikely(!sk)) { + pr_err("can't find conn. for hwtid %u.\n", hwtid); + return -EINVAL; + } skb_dst_set(skb, NULL); process_cpl_msg(chtls_rx_hdr, sk, skb); @@ -2105,6 +2121,10 @@ static int chtls_wr_ack(struct chtls_dev *cdev, struct sk_buff *skb) struct sock *sk; sk = lookup_tid(cdev->tids, hwtid); + if (unlikely(!sk)) { + pr_err("can't find conn. for hwtid %u.\n", hwtid); + return -EINVAL; + } process_cpl_msg(chtls_rx_ack, sk, skb); return 0; diff --git a/drivers/crypto/chelsio/chtls/chtls_hw.c b/drivers/crypto/chelsio/chtls/chtls_hw.c index 54a13aa99121..55d50140f9e5 100644 --- a/drivers/crypto/chelsio/chtls/chtls_hw.c +++ b/drivers/crypto/chelsio/chtls/chtls_hw.c @@ -213,7 +213,7 @@ static int chtls_key_info(struct chtls_sock *csk, struct _key_ctx *kctx, u32 keylen, u32 optname) { - unsigned char key[CHCR_KEYCTX_CIPHER_KEY_SIZE_256]; + unsigned char key[AES_KEYSIZE_128]; struct tls12_crypto_info_aes_gcm_128 *gcm_ctx; unsigned char ghash_h[AEAD_H_SIZE]; struct crypto_cipher *cipher; @@ -228,10 +228,6 @@ static int chtls_key_info(struct chtls_sock *csk, if (keylen == AES_KEYSIZE_128) { ck_size = CHCR_KEYCTX_CIPHER_KEY_SIZE_128; - } else if (keylen == AES_KEYSIZE_192) { - ck_size = CHCR_KEYCTX_CIPHER_KEY_SIZE_192; - } else if (keylen == AES_KEYSIZE_256) { - ck_size = CHCR_KEYCTX_CIPHER_KEY_SIZE_256; } else { pr_err("GCM: Invalid key length %d\n", keylen); return -EINVAL; diff --git a/drivers/crypto/chelsio/chtls/chtls_io.c b/drivers/crypto/chelsio/chtls/chtls_io.c index 5a75be43950f..00c7aab8e7d0 100644 --- a/drivers/crypto/chelsio/chtls/chtls_io.c +++ b/drivers/crypto/chelsio/chtls/chtls_io.c @@ -240,7 +240,7 @@ static int tls_copy_ivs(struct sock *sk, struct sk_buff *skb) } /* generate the IVs */ - ivs = kmalloc(number_of_ivs * CIPHER_BLOCK_SIZE, GFP_ATOMIC); + ivs = kmalloc_array(CIPHER_BLOCK_SIZE, number_of_ivs, GFP_ATOMIC); if (!ivs) return -ENOMEM; get_random_bytes(ivs, number_of_ivs * CIPHER_BLOCK_SIZE); @@ -907,11 +907,83 @@ static int chtls_skb_copy_to_page_nocache(struct sock *sk, } /* Read TLS header to find content type and data length */ -static u16 tls_header_read(struct tls_hdr *thdr, struct iov_iter *from) +static int tls_header_read(struct tls_hdr *thdr, struct iov_iter *from) { if (copy_from_iter(thdr, sizeof(*thdr), from) != sizeof(*thdr)) return -EFAULT; - return (__force u16)cpu_to_be16(thdr->length); + return (__force int)cpu_to_be16(thdr->length); +} + +static int csk_mem_free(struct chtls_dev *cdev, struct sock *sk) +{ + return (cdev->max_host_sndbuf - sk->sk_wmem_queued); +} + +static int csk_wait_memory(struct chtls_dev *cdev, + struct sock *sk, long *timeo_p) +{ + DEFINE_WAIT_FUNC(wait, woken_wake_function); + int sndbuf, err = 0; + long current_timeo; + long vm_wait = 0; + bool noblock; + + current_timeo = *timeo_p; + noblock = (*timeo_p ? false : true); + sndbuf = cdev->max_host_sndbuf; + if (csk_mem_free(cdev, sk)) { + current_timeo = (prandom_u32() % (HZ / 5)) + 2; + vm_wait = (prandom_u32() % (HZ / 5)) + 2; + } + + add_wait_queue(sk_sleep(sk), &wait); + while (1) { + sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk); + + if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) + goto do_error; + if (!*timeo_p) { + if (noblock) + set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); + goto do_nonblock; + } + if (signal_pending(current)) + goto do_interrupted; + sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); + if (csk_mem_free(cdev, sk) && !vm_wait) + break; + + set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); + sk->sk_write_pending++; + sk_wait_event(sk, ¤t_timeo, sk->sk_err || + (sk->sk_shutdown & SEND_SHUTDOWN) || + (csk_mem_free(cdev, sk) && !vm_wait), &wait); + sk->sk_write_pending--; + + if (vm_wait) { + vm_wait -= current_timeo; + current_timeo = *timeo_p; + if (current_timeo != MAX_SCHEDULE_TIMEOUT) { + current_timeo -= vm_wait; + if (current_timeo < 0) + current_timeo = 0; + } + vm_wait = 0; + } + *timeo_p = current_timeo; + } +do_rm_wq: + remove_wait_queue(sk_sleep(sk), &wait); + return err; +do_error: + err = -EPIPE; + goto do_rm_wq; +do_nonblock: + err = -EAGAIN; + goto do_rm_wq; +do_interrupted: + err = sock_intr_errno(*timeo_p); + goto do_rm_wq; } int chtls_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) @@ -952,6 +1024,8 @@ int chtls_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) copy = mss - skb->len; skb->ip_summed = CHECKSUM_UNNECESSARY; } + if (!csk_mem_free(cdev, sk)) + goto wait_for_sndbuf; if (is_tls_tx(csk) && !csk->tlshws.txleft) { struct tls_hdr hdr; @@ -1009,9 +1083,10 @@ new_buf: int off = TCP_OFF(sk); bool merge; - if (page) - pg_size <<= compound_order(page); + if (!page) + goto wait_for_memory; + pg_size <<= compound_order(page); if (off < pg_size && skb_can_coalesce(skb, i, page, off)) { merge = 1; @@ -1099,8 +1174,10 @@ copy: if (ULP_SKB_CB(skb)->flags & ULPCB_FLAG_NO_APPEND) push_frames_if_head(sk); continue; +wait_for_sndbuf: + set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); wait_for_memory: - err = sk_stream_wait_memory(sk, &timeo); + err = csk_wait_memory(cdev, sk, &timeo); if (err) goto do_error; } @@ -1131,6 +1208,7 @@ int chtls_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { struct chtls_sock *csk; + struct chtls_dev *cdev; int mss, err, copied; struct tcp_sock *tp; long timeo; @@ -1138,6 +1216,7 @@ int chtls_sendpage(struct sock *sk, struct page *page, tp = tcp_sk(sk); copied = 0; csk = rcu_dereference_sk_user_data(sk); + cdev = csk->cdev; timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); err = sk_stream_wait_connect(sk, &timeo); @@ -1152,10 +1231,11 @@ int chtls_sendpage(struct sock *sk, struct page *page, struct sk_buff *skb = skb_peek_tail(&csk->txq); int copy, i; - copy = mss - skb->len; if (!skb || (ULP_SKB_CB(skb)->flags & ULPCB_FLAG_NO_APPEND) || - copy <= 0) { + (copy = mss - skb->len) <= 0) { new_buf: + if (!csk_mem_free(cdev, sk)) + goto wait_for_sndbuf; if (is_tls_tx(csk)) { skb = get_record_skb(sk, @@ -1167,7 +1247,7 @@ new_buf: skb = get_tx_skb(sk, 0); } if (!skb) - goto do_error; + goto wait_for_memory; copy = mss; } if (copy > size) @@ -1206,8 +1286,12 @@ new_buf: if (unlikely(ULP_SKB_CB(skb)->flags & ULPCB_FLAG_NO_APPEND)) push_frames_if_head(sk); continue; - +wait_for_sndbuf: set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); +wait_for_memory: + err = csk_wait_memory(cdev, sk, &timeo); + if (err) + goto do_error; } out: csk_reset_flag(csk, CSK_TX_MORE_DATA); @@ -1409,7 +1493,7 @@ static int chtls_pt_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, break; chtls_cleanup_rbuf(sk, copied); sk_wait_data(sk, &timeo, NULL); - continue; + continue; found_ok_skb: if (!skb->len) { skb_dst_set(skb, NULL); @@ -1449,31 +1533,13 @@ found_ok_skb: } } } - if (hws->rstate == TLS_RCV_ST_READ_BODY) { - if (skb_copy_datagram_msg(skb, offset, - msg, avail)) { - if (!copied) { - copied = -EFAULT; - break; - } - } - } else { - struct tlsrx_cmp_hdr *tls_hdr_pkt = - (struct tlsrx_cmp_hdr *)skb->data; - - if ((tls_hdr_pkt->res_to_mac_error & - TLSRX_HDR_PKT_ERROR_M)) - tls_hdr_pkt->type = 0x7F; - - /* CMP pld len is for recv seq */ - hws->rcvpld = skb->hdr_len; - if (skb_copy_datagram_msg(skb, offset, msg, avail)) { - if (!copied) { - copied = -EFAULT; - break; - } + if (skb_copy_datagram_msg(skb, offset, msg, avail)) { + if (!copied) { + copied = -EFAULT; + break; } } + copied += avail; len -= avail; hws->copied_seq += avail; @@ -1481,32 +1547,20 @@ skip_copy: if (tp->urg_data && after(tp->copied_seq, tp->urg_seq)) tp->urg_data = 0; - if (hws->rstate == TLS_RCV_ST_READ_BODY && - (avail + offset) >= skb->len) { + if ((avail + offset) >= skb->len) { if (likely(skb)) chtls_free_skb(sk, skb); buffers_freed++; - hws->rstate = TLS_RCV_ST_READ_HEADER; - atomic_inc(&adap->chcr_stats.tls_pdu_rx); - tp->copied_seq += hws->rcvpld; + if (ULP_SKB_CB(skb)->flags & ULPCB_FLAG_TLS_HDR) { + tp->copied_seq += skb->len; + hws->rcvpld = skb->hdr_len; + } else { + tp->copied_seq += hws->rcvpld; + } hws->copied_seq = 0; if (copied >= target && !skb_peek(&sk->sk_receive_queue)) break; - } else { - if (likely(skb)) { - if (ULP_SKB_CB(skb)->flags & - ULPCB_FLAG_TLS_ND) - hws->rstate = - TLS_RCV_ST_READ_HEADER; - else - hws->rstate = - TLS_RCV_ST_READ_BODY; - chtls_free_skb(sk, skb); - } - buffers_freed++; - tp->copied_seq += avail; - hws->copied_seq = 0; } } while (len > 0); diff --git a/drivers/crypto/chelsio/chtls/chtls_main.c b/drivers/crypto/chelsio/chtls/chtls_main.c index 007c45c38fc7..9b07f9165658 100644 --- a/drivers/crypto/chelsio/chtls/chtls_main.c +++ b/drivers/crypto/chelsio/chtls/chtls_main.c @@ -216,7 +216,6 @@ static void *chtls_uld_add(const struct cxgb4_lld_info *info) cdev->lldi = lldi; cdev->pdev = lldi->pdev; cdev->tids = lldi->tids; - cdev->ports = (struct net_device **)(cdev + 1); cdev->ports = lldi->ports; cdev->mtus = lldi->mtus; cdev->tids = lldi->tids; @@ -239,6 +238,7 @@ static void *chtls_uld_add(const struct cxgb4_lld_info *info) spin_lock_init(&cdev->idr_lock); cdev->send_page_order = min_t(uint, get_order(32768), send_page_order); + cdev->max_host_sndbuf = 48 * 1024; if (lldi->vr->key.size) if (chtls_init_kmap(cdev, lldi)) @@ -250,7 +250,7 @@ static void *chtls_uld_add(const struct cxgb4_lld_info *info) return cdev; out_rspq_skb: - for (j = 0; j <= i; j++) + for (j = 0; j < i; j++) kfree_skb(cdev->rspq_skb_cache[j]); kfree_skb(cdev->askb); out_skb: @@ -441,7 +441,7 @@ nomem: static int do_chtls_getsockopt(struct sock *sk, char __user *optval, int __user *optlen) { - struct tls_crypto_info crypto_info; + struct tls_crypto_info crypto_info = { 0 }; crypto_info.version = TLS_1_2_VERSION; if (copy_to_user(optval, &crypto_info, sizeof(struct tls_crypto_info))) @@ -491,9 +491,13 @@ static int do_chtls_setsockopt(struct sock *sk, int optname, switch (tmp_crypto_info.cipher_type) { case TLS_CIPHER_AES_GCM_128: { - rc = copy_from_user(crypto_info, optval, - sizeof(struct - tls12_crypto_info_aes_gcm_128)); + /* Obtain version and type from previous copy */ + crypto_info[0] = tmp_crypto_info; + /* Now copy the following data */ + rc = copy_from_user((char *)crypto_info + sizeof(*crypto_info), + optval + sizeof(*crypto_info), + sizeof(struct tls12_crypto_info_aes_gcm_128) + - sizeof(*crypto_info)); if (rc) { rc = -EFAULT; diff --git a/drivers/crypto/exynos-rng.c b/drivers/crypto/exynos-rng.c index 86f5f459762e..2cfabb99cb6e 100644 --- a/drivers/crypto/exynos-rng.c +++ b/drivers/crypto/exynos-rng.c @@ -319,8 +319,7 @@ static int exynos_rng_remove(struct platform_device *pdev) static int __maybe_unused exynos_rng_suspend(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct exynos_rng_dev *rng = platform_get_drvdata(pdev); + struct exynos_rng_dev *rng = dev_get_drvdata(dev); int ret; /* If we were never seeded then after resume it will be the same */ @@ -350,8 +349,7 @@ static int __maybe_unused exynos_rng_suspend(struct device *dev) static int __maybe_unused exynos_rng_resume(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct exynos_rng_dev *rng = platform_get_drvdata(pdev); + struct exynos_rng_dev *rng = dev_get_drvdata(dev); int ret; /* Never seeded so nothing to do */ diff --git a/drivers/crypto/inside-secure/safexcel.c b/drivers/crypto/inside-secure/safexcel.c index d4a81be0d7d2..4e86f864a952 100644 --- a/drivers/crypto/inside-secure/safexcel.c +++ b/drivers/crypto/inside-secure/safexcel.c @@ -20,6 +20,7 @@ #include #include +#include #include #include @@ -152,8 +153,8 @@ static int eip197_load_firmwares(struct safexcel_crypto_priv *priv) EIP197_PE_ICE_SCRATCH_CTRL_CHANGE_ACCESS; writel(val, EIP197_PE(priv) + EIP197_PE_ICE_SCRATCH_CTRL); - memset(EIP197_PE(priv) + EIP197_PE_ICE_SCRATCH_RAM, 0, - EIP197_NUM_OF_SCRATCH_BLOCKS * sizeof(u32)); + memset_io(EIP197_PE(priv) + EIP197_PE_ICE_SCRATCH_RAM, 0, + EIP197_NUM_OF_SCRATCH_BLOCKS * sizeof(u32)); eip197_write_firmware(priv, fw[FW_IFPP], EIP197_PE_ICE_FPP_CTRL, EIP197_PE_ICE_RAM_CTRL_FPP_PROG_EN); @@ -352,6 +353,7 @@ static int safexcel_hw_init(struct safexcel_crypto_priv *priv) /* H/W capabilities selection */ val = EIP197_FUNCTION_RSVD; val |= EIP197_PROTOCOL_ENCRYPT_ONLY | EIP197_PROTOCOL_HASH_ONLY; + val |= EIP197_PROTOCOL_ENCRYPT_HASH | EIP197_PROTOCOL_HASH_DECRYPT; val |= EIP197_ALG_AES_ECB | EIP197_ALG_AES_CBC; val |= EIP197_ALG_SHA1 | EIP197_ALG_HMAC_SHA1; val |= EIP197_ALG_SHA2 | EIP197_ALG_HMAC_SHA2; @@ -537,6 +539,27 @@ finalize: EIP197_HIA_CDR(priv, ring) + EIP197_HIA_xDR_PREP_COUNT); } +inline int safexcel_rdesc_check_errors(struct safexcel_crypto_priv *priv, + struct safexcel_result_desc *rdesc) +{ + if (likely(!rdesc->result_data.error_code)) + return 0; + + if (rdesc->result_data.error_code & 0x407f) { + /* Fatal error (bits 0-7, 14) */ + dev_err(priv->dev, + "cipher: result: result descriptor error (%d)\n", + rdesc->result_data.error_code); + return -EIO; + } else if (rdesc->result_data.error_code == BIT(9)) { + /* Authentication failed */ + return -EBADMSG; + } + + /* All other non-fatal errors */ + return -EINVAL; +} + void safexcel_complete(struct safexcel_crypto_priv *priv, int ring) { struct safexcel_command_desc *cdesc; @@ -770,6 +793,9 @@ static struct safexcel_alg_template *safexcel_algs[] = { &safexcel_alg_hmac_sha1, &safexcel_alg_hmac_sha224, &safexcel_alg_hmac_sha256, + &safexcel_alg_authenc_hmac_sha1_cbc_aes, + &safexcel_alg_authenc_hmac_sha224_cbc_aes, + &safexcel_alg_authenc_hmac_sha256_cbc_aes, }; static int safexcel_register_algorithms(struct safexcel_crypto_priv *priv) @@ -781,6 +807,8 @@ static int safexcel_register_algorithms(struct safexcel_crypto_priv *priv) if (safexcel_algs[i]->type == SAFEXCEL_ALG_TYPE_SKCIPHER) ret = crypto_register_skcipher(&safexcel_algs[i]->alg.skcipher); + else if (safexcel_algs[i]->type == SAFEXCEL_ALG_TYPE_AEAD) + ret = crypto_register_aead(&safexcel_algs[i]->alg.aead); else ret = crypto_register_ahash(&safexcel_algs[i]->alg.ahash); @@ -794,6 +822,8 @@ fail: for (j = 0; j < i; j++) { if (safexcel_algs[j]->type == SAFEXCEL_ALG_TYPE_SKCIPHER) crypto_unregister_skcipher(&safexcel_algs[j]->alg.skcipher); + else if (safexcel_algs[j]->type == SAFEXCEL_ALG_TYPE_AEAD) + crypto_unregister_aead(&safexcel_algs[j]->alg.aead); else crypto_unregister_ahash(&safexcel_algs[j]->alg.ahash); } @@ -808,6 +838,8 @@ static void safexcel_unregister_algorithms(struct safexcel_crypto_priv *priv) for (i = 0; i < ARRAY_SIZE(safexcel_algs); i++) { if (safexcel_algs[i]->type == SAFEXCEL_ALG_TYPE_SKCIPHER) crypto_unregister_skcipher(&safexcel_algs[i]->alg.skcipher); + else if (safexcel_algs[i]->type == SAFEXCEL_ALG_TYPE_AEAD) + crypto_unregister_aead(&safexcel_algs[i]->alg.aead); else crypto_unregister_ahash(&safexcel_algs[i]->alg.ahash); } diff --git a/drivers/crypto/inside-secure/safexcel.h b/drivers/crypto/inside-secure/safexcel.h index b470a849721f..8b3ee9b59f53 100644 --- a/drivers/crypto/inside-secure/safexcel.h +++ b/drivers/crypto/inside-secure/safexcel.h @@ -11,8 +11,10 @@ #ifndef __SAFEXCEL_H__ #define __SAFEXCEL_H__ +#include #include #include +#include #include #define EIP197_HIA_VERSION_LE 0xca35 @@ -20,7 +22,7 @@ /* Static configuration */ #define EIP197_DEFAULT_RING_SIZE 400 -#define EIP197_MAX_TOKENS 5 +#define EIP197_MAX_TOKENS 8 #define EIP197_MAX_RINGS 4 #define EIP197_FETCH_COUNT 1 #define EIP197_MAX_BATCH_SZ 64 @@ -28,6 +30,17 @@ #define EIP197_GFP_FLAGS(base) ((base).flags & CRYPTO_TFM_REQ_MAY_SLEEP ? \ GFP_KERNEL : GFP_ATOMIC) +/* Custom on-stack requests (for invalidation) */ +#define EIP197_SKCIPHER_REQ_SIZE sizeof(struct skcipher_request) + \ + sizeof(struct safexcel_cipher_req) +#define EIP197_AHASH_REQ_SIZE sizeof(struct ahash_request) + \ + sizeof(struct safexcel_ahash_req) +#define EIP197_AEAD_REQ_SIZE sizeof(struct aead_request) + \ + sizeof(struct safexcel_cipher_req) +#define EIP197_REQUEST_ON_STACK(name, type, size) \ + char __##name##_desc[size] CRYPTO_MINALIGN_ATTR; \ + struct type##_request *name = (void *)__##name##_desc + /* Register base offsets */ #define EIP197_HIA_AIC(priv) ((priv)->base + (priv)->offsets.hia_aic) #define EIP197_HIA_AIC_G(priv) ((priv)->base + (priv)->offsets.hia_aic_g) @@ -274,7 +287,7 @@ struct safexcel_context_record { u32 control0; u32 control1; - __le32 data[12]; + __le32 data[24]; } __packed; /* control0 */ @@ -286,8 +299,8 @@ struct safexcel_context_record { #define CONTEXT_CONTROL_TYPE_CRYPTO_IN 0x5 #define CONTEXT_CONTROL_TYPE_ENCRYPT_HASH_OUT 0x6 #define CONTEXT_CONTROL_TYPE_DECRYPT_HASH_IN 0x7 -#define CONTEXT_CONTROL_TYPE_HASH_ENCRYPT_OUT 0x14 -#define CONTEXT_CONTROL_TYPE_HASH_DECRYPT_OUT 0x15 +#define CONTEXT_CONTROL_TYPE_HASH_ENCRYPT_OUT 0xe +#define CONTEXT_CONTROL_TYPE_HASH_DECRYPT_IN 0xf #define CONTEXT_CONTROL_RESTART_HASH BIT(4) #define CONTEXT_CONTROL_NO_FINISH_HASH BIT(5) #define CONTEXT_CONTROL_SIZE(n) ((n) << 8) @@ -391,11 +404,15 @@ struct safexcel_token { u8 opcode:4; } __packed; +#define EIP197_TOKEN_HASH_RESULT_VERIFY BIT(16) + #define EIP197_TOKEN_STAT_LAST_HASH BIT(0) #define EIP197_TOKEN_STAT_LAST_PACKET BIT(1) #define EIP197_TOKEN_OPCODE_DIRECTION 0x0 #define EIP197_TOKEN_OPCODE_INSERT 0x2 #define EIP197_TOKEN_OPCODE_NOOP EIP197_TOKEN_OPCODE_INSERT +#define EIP197_TOKEN_OPCODE_RETRIEVE 0x4 +#define EIP197_TOKEN_OPCODE_VERIFY 0xd #define EIP197_TOKEN_OPCODE_BYPASS GENMASK(3, 0) static inline void eip197_noop_token(struct safexcel_token *token) @@ -479,6 +496,7 @@ struct safexcel_ring { enum safexcel_alg_type { SAFEXCEL_ALG_TYPE_SKCIPHER, + SAFEXCEL_ALG_TYPE_AEAD, SAFEXCEL_ALG_TYPE_AHASH, }; @@ -581,6 +599,16 @@ struct safexcel_context { bool exit_inv; }; +struct safexcel_ahash_export_state { + u64 len; + u64 processed; + + u32 digest; + + u32 state[SHA256_DIGEST_SIZE / sizeof(u32)]; + u8 cache[SHA256_BLOCK_SIZE]; +}; + /* * Template structure to describe the algorithms in order to register them. * It also has the purpose to contain our private structure and is actually @@ -591,6 +619,7 @@ struct safexcel_alg_template { enum safexcel_alg_type type; union { struct skcipher_alg skcipher; + struct aead_alg aead; struct ahash_alg ahash; } alg; }; @@ -601,6 +630,8 @@ struct safexcel_inv_result { }; void safexcel_dequeue(struct safexcel_crypto_priv *priv, int ring); +int safexcel_rdesc_check_errors(struct safexcel_crypto_priv *priv, + struct safexcel_result_desc *rdesc); void safexcel_complete(struct safexcel_crypto_priv *priv, int ring); int safexcel_invalidate_cache(struct crypto_async_request *async, struct safexcel_crypto_priv *priv, @@ -625,6 +656,8 @@ struct safexcel_result_desc *safexcel_add_rdesc(struct safexcel_crypto_priv *pri bool first, bool last, dma_addr_t data, u32 len); void safexcel_inv_complete(struct crypto_async_request *req, int error); +int safexcel_hmac_setkey(const char *alg, const u8 *key, unsigned int keylen, + void *istate, void *ostate); /* available algorithms */ extern struct safexcel_alg_template safexcel_alg_ecb_aes; @@ -635,5 +668,8 @@ extern struct safexcel_alg_template safexcel_alg_sha256; extern struct safexcel_alg_template safexcel_alg_hmac_sha1; extern struct safexcel_alg_template safexcel_alg_hmac_sha224; extern struct safexcel_alg_template safexcel_alg_hmac_sha256; +extern struct safexcel_alg_template safexcel_alg_authenc_hmac_sha1_cbc_aes; +extern struct safexcel_alg_template safexcel_alg_authenc_hmac_sha224_cbc_aes; +extern struct safexcel_alg_template safexcel_alg_authenc_hmac_sha256_cbc_aes; #endif diff --git a/drivers/crypto/inside-secure/safexcel_cipher.c b/drivers/crypto/inside-secure/safexcel_cipher.c index bafb60505fab..6bb60fda2043 100644 --- a/drivers/crypto/inside-secure/safexcel_cipher.c +++ b/drivers/crypto/inside-secure/safexcel_cipher.c @@ -12,8 +12,12 @@ #include #include +#include #include +#include +#include #include +#include #include #include "safexcel.h" @@ -28,9 +32,16 @@ struct safexcel_cipher_ctx { struct safexcel_crypto_priv *priv; u32 mode; + bool aead; __le32 key[8]; unsigned int key_len; + + /* All the below is AEAD specific */ + u32 alg; + u32 state_sz; + u32 ipad[SHA256_DIGEST_SIZE / sizeof(u32)]; + u32 opad[SHA256_DIGEST_SIZE / sizeof(u32)]; }; struct safexcel_cipher_req { @@ -38,18 +49,16 @@ struct safexcel_cipher_req { bool needs_inv; }; -static void safexcel_cipher_token(struct safexcel_cipher_ctx *ctx, - struct crypto_async_request *async, - struct safexcel_command_desc *cdesc, - u32 length) +static void safexcel_skcipher_token(struct safexcel_cipher_ctx *ctx, u8 *iv, + struct safexcel_command_desc *cdesc, + u32 length) { - struct skcipher_request *req = skcipher_request_cast(async); struct safexcel_token *token; unsigned offset = 0; if (ctx->mode == CONTEXT_CONTROL_CRYPTO_MODE_CBC) { offset = AES_BLOCK_SIZE / sizeof(u32); - memcpy(cdesc->control_data.token, req->iv, AES_BLOCK_SIZE); + memcpy(cdesc->control_data.token, iv, AES_BLOCK_SIZE); cdesc->control_data.options |= EIP197_OPTION_4_TOKEN_IV_CMD; } @@ -65,8 +74,64 @@ static void safexcel_cipher_token(struct safexcel_cipher_ctx *ctx, EIP197_TOKEN_INS_TYPE_OUTPUT; } -static int safexcel_aes_setkey(struct crypto_skcipher *ctfm, const u8 *key, - unsigned int len) +static void safexcel_aead_token(struct safexcel_cipher_ctx *ctx, u8 *iv, + struct safexcel_command_desc *cdesc, + enum safexcel_cipher_direction direction, + u32 cryptlen, u32 assoclen, u32 digestsize) +{ + struct safexcel_token *token; + unsigned offset = 0; + + if (ctx->mode == CONTEXT_CONTROL_CRYPTO_MODE_CBC) { + offset = AES_BLOCK_SIZE / sizeof(u32); + memcpy(cdesc->control_data.token, iv, AES_BLOCK_SIZE); + + cdesc->control_data.options |= EIP197_OPTION_4_TOKEN_IV_CMD; + } + + token = (struct safexcel_token *)(cdesc->control_data.token + offset); + + if (direction == SAFEXCEL_DECRYPT) + cryptlen -= digestsize; + + token[0].opcode = EIP197_TOKEN_OPCODE_DIRECTION; + token[0].packet_length = assoclen; + token[0].instructions = EIP197_TOKEN_INS_TYPE_HASH | + EIP197_TOKEN_INS_TYPE_OUTPUT; + + token[1].opcode = EIP197_TOKEN_OPCODE_DIRECTION; + token[1].packet_length = cryptlen; + token[1].stat = EIP197_TOKEN_STAT_LAST_HASH; + token[1].instructions = EIP197_TOKEN_INS_LAST | + EIP197_TOKEN_INS_TYPE_CRYTO | + EIP197_TOKEN_INS_TYPE_HASH | + EIP197_TOKEN_INS_TYPE_OUTPUT; + + if (direction == SAFEXCEL_ENCRYPT) { + token[2].opcode = EIP197_TOKEN_OPCODE_INSERT; + token[2].packet_length = digestsize; + token[2].stat = EIP197_TOKEN_STAT_LAST_HASH | + EIP197_TOKEN_STAT_LAST_PACKET; + token[2].instructions = EIP197_TOKEN_INS_TYPE_OUTPUT | + EIP197_TOKEN_INS_INSERT_HASH_DIGEST; + } else { + token[2].opcode = EIP197_TOKEN_OPCODE_RETRIEVE; + token[2].packet_length = digestsize; + token[2].stat = EIP197_TOKEN_STAT_LAST_HASH | + EIP197_TOKEN_STAT_LAST_PACKET; + token[2].instructions = EIP197_TOKEN_INS_INSERT_HASH_DIGEST; + + token[3].opcode = EIP197_TOKEN_OPCODE_VERIFY; + token[3].packet_length = digestsize | + EIP197_TOKEN_HASH_RESULT_VERIFY; + token[3].stat = EIP197_TOKEN_STAT_LAST_HASH | + EIP197_TOKEN_STAT_LAST_PACKET; + token[3].instructions = EIP197_TOKEN_INS_TYPE_OUTPUT; + } +} + +static int safexcel_skcipher_aes_setkey(struct crypto_skcipher *ctfm, + const u8 *key, unsigned int len) { struct crypto_tfm *tfm = crypto_skcipher_tfm(ctfm); struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm); @@ -98,41 +163,123 @@ static int safexcel_aes_setkey(struct crypto_skcipher *ctfm, const u8 *key, return 0; } +static int safexcel_aead_aes_setkey(struct crypto_aead *ctfm, const u8 *key, + unsigned int len) +{ + struct crypto_tfm *tfm = crypto_aead_tfm(ctfm); + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm); + struct safexcel_ahash_export_state istate, ostate; + struct safexcel_crypto_priv *priv = ctx->priv; + struct crypto_authenc_keys keys; + + if (crypto_authenc_extractkeys(&keys, key, len) != 0) + goto badkey; + + if (keys.enckeylen > sizeof(ctx->key)) + goto badkey; + + /* Encryption key */ + if (priv->version == EIP197 && ctx->base.ctxr_dma && + memcmp(ctx->key, keys.enckey, keys.enckeylen)) + ctx->base.needs_inv = true; + + /* Auth key */ + switch (ctx->alg) { + case CONTEXT_CONTROL_CRYPTO_ALG_SHA1: + if (safexcel_hmac_setkey("safexcel-sha1", keys.authkey, + keys.authkeylen, &istate, &ostate)) + goto badkey; + break; + case CONTEXT_CONTROL_CRYPTO_ALG_SHA224: + if (safexcel_hmac_setkey("safexcel-sha224", keys.authkey, + keys.authkeylen, &istate, &ostate)) + goto badkey; + break; + case CONTEXT_CONTROL_CRYPTO_ALG_SHA256: + if (safexcel_hmac_setkey("safexcel-sha256", keys.authkey, + keys.authkeylen, &istate, &ostate)) + goto badkey; + break; + default: + dev_err(priv->dev, "aead: unsupported hash algorithm\n"); + goto badkey; + } + + crypto_aead_set_flags(ctfm, crypto_aead_get_flags(ctfm) & + CRYPTO_TFM_RES_MASK); + + if (priv->version == EIP197 && ctx->base.ctxr_dma && + (memcmp(ctx->ipad, istate.state, ctx->state_sz) || + memcmp(ctx->opad, ostate.state, ctx->state_sz))) + ctx->base.needs_inv = true; + + /* Now copy the keys into the context */ + memcpy(ctx->key, keys.enckey, keys.enckeylen); + ctx->key_len = keys.enckeylen; + + memcpy(ctx->ipad, &istate.state, ctx->state_sz); + memcpy(ctx->opad, &ostate.state, ctx->state_sz); + + memzero_explicit(&keys, sizeof(keys)); + return 0; + +badkey: + crypto_aead_set_flags(ctfm, CRYPTO_TFM_RES_BAD_KEY_LEN); + memzero_explicit(&keys, sizeof(keys)); + return -EINVAL; +} + static int safexcel_context_control(struct safexcel_cipher_ctx *ctx, struct crypto_async_request *async, + struct safexcel_cipher_req *sreq, struct safexcel_command_desc *cdesc) { struct safexcel_crypto_priv *priv = ctx->priv; - struct skcipher_request *req = skcipher_request_cast(async); - struct safexcel_cipher_req *sreq = skcipher_request_ctx(req); int ctrl_size; - if (sreq->direction == SAFEXCEL_ENCRYPT) + if (ctx->aead) { + if (sreq->direction == SAFEXCEL_ENCRYPT) + cdesc->control_data.control0 |= CONTEXT_CONTROL_TYPE_ENCRYPT_HASH_OUT; + else + cdesc->control_data.control0 |= CONTEXT_CONTROL_TYPE_HASH_DECRYPT_IN; + } else { cdesc->control_data.control0 |= CONTEXT_CONTROL_TYPE_CRYPTO_OUT; - else - cdesc->control_data.control0 |= CONTEXT_CONTROL_TYPE_CRYPTO_IN; + + /* The decryption control type is a combination of the + * encryption type and CONTEXT_CONTROL_TYPE_NULL_IN, for all + * types. + */ + if (sreq->direction == SAFEXCEL_DECRYPT) + cdesc->control_data.control0 |= CONTEXT_CONTROL_TYPE_NULL_IN; + } cdesc->control_data.control0 |= CONTEXT_CONTROL_KEY_EN; cdesc->control_data.control1 |= ctx->mode; + if (ctx->aead) + cdesc->control_data.control0 |= CONTEXT_CONTROL_DIGEST_HMAC | + ctx->alg; + switch (ctx->key_len) { case AES_KEYSIZE_128: cdesc->control_data.control0 |= CONTEXT_CONTROL_CRYPTO_ALG_AES128; - ctrl_size = 4; break; case AES_KEYSIZE_192: cdesc->control_data.control0 |= CONTEXT_CONTROL_CRYPTO_ALG_AES192; - ctrl_size = 6; break; case AES_KEYSIZE_256: cdesc->control_data.control0 |= CONTEXT_CONTROL_CRYPTO_ALG_AES256; - ctrl_size = 8; break; default: dev_err(priv->dev, "aes keysize not supported: %u\n", ctx->key_len); return -EINVAL; } + + ctrl_size = ctx->key_len / sizeof(u32); + if (ctx->aead) + /* Take in account the ipad+opad digests */ + ctrl_size += ctx->state_sz / sizeof(u32) * 2; cdesc->control_data.control0 |= CONTEXT_CONTROL_SIZE(ctrl_size); return 0; @@ -140,9 +287,12 @@ static int safexcel_context_control(struct safexcel_cipher_ctx *ctx, static int safexcel_handle_req_result(struct safexcel_crypto_priv *priv, int ring, struct crypto_async_request *async, + struct scatterlist *src, + struct scatterlist *dst, + unsigned int cryptlen, + struct safexcel_cipher_req *sreq, bool *should_complete, int *ret) { - struct skcipher_request *req = skcipher_request_cast(async); struct safexcel_result_desc *rdesc; int ndesc = 0; @@ -158,12 +308,8 @@ static int safexcel_handle_req_result(struct safexcel_crypto_priv *priv, int rin break; } - if (rdesc->result_data.error_code) { - dev_err(priv->dev, - "cipher: result: result descriptor error (%d)\n", - rdesc->result_data.error_code); - *ret = -EIO; - } + if (likely(!*ret)) + *ret = safexcel_rdesc_check_errors(priv, rdesc); ndesc++; } while (!rdesc->last_seg); @@ -171,16 +317,16 @@ static int safexcel_handle_req_result(struct safexcel_crypto_priv *priv, int rin safexcel_complete(priv, ring); spin_unlock_bh(&priv->ring[ring].egress_lock); - if (req->src == req->dst) { - dma_unmap_sg(priv->dev, req->src, - sg_nents_for_len(req->src, req->cryptlen), + if (src == dst) { + dma_unmap_sg(priv->dev, src, + sg_nents_for_len(src, cryptlen), DMA_BIDIRECTIONAL); } else { - dma_unmap_sg(priv->dev, req->src, - sg_nents_for_len(req->src, req->cryptlen), + dma_unmap_sg(priv->dev, src, + sg_nents_for_len(src, cryptlen), DMA_TO_DEVICE); - dma_unmap_sg(priv->dev, req->dst, - sg_nents_for_len(req->dst, req->cryptlen), + dma_unmap_sg(priv->dev, dst, + sg_nents_for_len(dst, cryptlen), DMA_FROM_DEVICE); } @@ -189,39 +335,43 @@ static int safexcel_handle_req_result(struct safexcel_crypto_priv *priv, int rin return ndesc; } -static int safexcel_aes_send(struct crypto_async_request *async, - int ring, struct safexcel_request *request, - int *commands, int *results) +static int safexcel_aes_send(struct crypto_async_request *base, int ring, + struct safexcel_request *request, + struct safexcel_cipher_req *sreq, + struct scatterlist *src, struct scatterlist *dst, + unsigned int cryptlen, unsigned int assoclen, + unsigned int digestsize, u8 *iv, int *commands, + int *results) { - struct skcipher_request *req = skcipher_request_cast(async); - struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(req->base.tfm); + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(base->tfm); struct safexcel_crypto_priv *priv = ctx->priv; struct safexcel_command_desc *cdesc; struct safexcel_result_desc *rdesc; struct scatterlist *sg; - int nr_src, nr_dst, n_cdesc = 0, n_rdesc = 0, queued = req->cryptlen; + unsigned int totlen = cryptlen + assoclen; + int nr_src, nr_dst, n_cdesc = 0, n_rdesc = 0, queued = totlen; int i, ret = 0; - if (req->src == req->dst) { - nr_src = dma_map_sg(priv->dev, req->src, - sg_nents_for_len(req->src, req->cryptlen), + if (src == dst) { + nr_src = dma_map_sg(priv->dev, src, + sg_nents_for_len(src, totlen), DMA_BIDIRECTIONAL); nr_dst = nr_src; if (!nr_src) return -EINVAL; } else { - nr_src = dma_map_sg(priv->dev, req->src, - sg_nents_for_len(req->src, req->cryptlen), + nr_src = dma_map_sg(priv->dev, src, + sg_nents_for_len(src, totlen), DMA_TO_DEVICE); if (!nr_src) return -EINVAL; - nr_dst = dma_map_sg(priv->dev, req->dst, - sg_nents_for_len(req->dst, req->cryptlen), + nr_dst = dma_map_sg(priv->dev, dst, + sg_nents_for_len(dst, totlen), DMA_FROM_DEVICE); if (!nr_dst) { - dma_unmap_sg(priv->dev, req->src, - sg_nents_for_len(req->src, req->cryptlen), + dma_unmap_sg(priv->dev, src, + sg_nents_for_len(src, totlen), DMA_TO_DEVICE); return -EINVAL; } @@ -229,10 +379,17 @@ static int safexcel_aes_send(struct crypto_async_request *async, memcpy(ctx->base.ctxr->data, ctx->key, ctx->key_len); + if (ctx->aead) { + memcpy(ctx->base.ctxr->data + ctx->key_len / sizeof(u32), + ctx->ipad, ctx->state_sz); + memcpy(ctx->base.ctxr->data + (ctx->key_len + ctx->state_sz) / sizeof(u32), + ctx->opad, ctx->state_sz); + } + spin_lock_bh(&priv->ring[ring].egress_lock); /* command descriptors */ - for_each_sg(req->src, sg, nr_src, i) { + for_each_sg(src, sg, nr_src, i) { int len = sg_dma_len(sg); /* Do not overflow the request */ @@ -240,7 +397,7 @@ static int safexcel_aes_send(struct crypto_async_request *async, len = queued; cdesc = safexcel_add_cdesc(priv, ring, !n_cdesc, !(queued - len), - sg_dma_address(sg), len, req->cryptlen, + sg_dma_address(sg), len, totlen, ctx->base.ctxr_dma); if (IS_ERR(cdesc)) { /* No space left in the command descriptor ring */ @@ -250,8 +407,14 @@ static int safexcel_aes_send(struct crypto_async_request *async, n_cdesc++; if (n_cdesc == 1) { - safexcel_context_control(ctx, async, cdesc); - safexcel_cipher_token(ctx, async, cdesc, req->cryptlen); + safexcel_context_control(ctx, base, sreq, cdesc); + if (ctx->aead) + safexcel_aead_token(ctx, iv, cdesc, + sreq->direction, cryptlen, + assoclen, digestsize); + else + safexcel_skcipher_token(ctx, iv, cdesc, + cryptlen); } queued -= len; @@ -260,7 +423,7 @@ static int safexcel_aes_send(struct crypto_async_request *async, } /* result descriptors */ - for_each_sg(req->dst, sg, nr_dst, i) { + for_each_sg(dst, sg, nr_dst, i) { bool first = !i, last = (i == nr_dst - 1); u32 len = sg_dma_len(sg); @@ -276,7 +439,7 @@ static int safexcel_aes_send(struct crypto_async_request *async, spin_unlock_bh(&priv->ring[ring].egress_lock); - request->req = &req->base; + request->req = base; *commands = n_cdesc; *results = n_rdesc; @@ -291,16 +454,16 @@ cdesc_rollback: spin_unlock_bh(&priv->ring[ring].egress_lock); - if (req->src == req->dst) { - dma_unmap_sg(priv->dev, req->src, - sg_nents_for_len(req->src, req->cryptlen), + if (src == dst) { + dma_unmap_sg(priv->dev, src, + sg_nents_for_len(src, totlen), DMA_BIDIRECTIONAL); } else { - dma_unmap_sg(priv->dev, req->src, - sg_nents_for_len(req->src, req->cryptlen), + dma_unmap_sg(priv->dev, src, + sg_nents_for_len(src, totlen), DMA_TO_DEVICE); - dma_unmap_sg(priv->dev, req->dst, - sg_nents_for_len(req->dst, req->cryptlen), + dma_unmap_sg(priv->dev, dst, + sg_nents_for_len(dst, totlen), DMA_FROM_DEVICE); } @@ -309,11 +472,10 @@ cdesc_rollback: static int safexcel_handle_inv_result(struct safexcel_crypto_priv *priv, int ring, - struct crypto_async_request *async, + struct crypto_async_request *base, bool *should_complete, int *ret) { - struct skcipher_request *req = skcipher_request_cast(async); - struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(req->base.tfm); + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(base->tfm); struct safexcel_result_desc *rdesc; int ndesc = 0, enq_ret; @@ -354,7 +516,7 @@ static int safexcel_handle_inv_result(struct safexcel_crypto_priv *priv, ctx->base.ring = ring; spin_lock_bh(&priv->ring[ring].queue_lock); - enq_ret = crypto_enqueue_request(&priv->ring[ring].queue, async); + enq_ret = crypto_enqueue_request(&priv->ring[ring].queue, base); spin_unlock_bh(&priv->ring[ring].queue_lock); if (enq_ret != -EINPROGRESS) @@ -368,9 +530,10 @@ static int safexcel_handle_inv_result(struct safexcel_crypto_priv *priv, return ndesc; } -static int safexcel_handle_result(struct safexcel_crypto_priv *priv, int ring, - struct crypto_async_request *async, - bool *should_complete, int *ret) +static int safexcel_skcipher_handle_result(struct safexcel_crypto_priv *priv, + int ring, + struct crypto_async_request *async, + bool *should_complete, int *ret) { struct skcipher_request *req = skcipher_request_cast(async); struct safexcel_cipher_req *sreq = skcipher_request_ctx(req); @@ -381,24 +544,48 @@ static int safexcel_handle_result(struct safexcel_crypto_priv *priv, int ring, err = safexcel_handle_inv_result(priv, ring, async, should_complete, ret); } else { - err = safexcel_handle_req_result(priv, ring, async, + err = safexcel_handle_req_result(priv, ring, async, req->src, + req->dst, req->cryptlen, sreq, should_complete, ret); } return err; } -static int safexcel_cipher_send_inv(struct crypto_async_request *async, +static int safexcel_aead_handle_result(struct safexcel_crypto_priv *priv, + int ring, + struct crypto_async_request *async, + bool *should_complete, int *ret) +{ + struct aead_request *req = aead_request_cast(async); + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct safexcel_cipher_req *sreq = aead_request_ctx(req); + int err; + + if (sreq->needs_inv) { + sreq->needs_inv = false; + err = safexcel_handle_inv_result(priv, ring, async, + should_complete, ret); + } else { + err = safexcel_handle_req_result(priv, ring, async, req->src, + req->dst, + req->cryptlen + crypto_aead_authsize(tfm), + sreq, should_complete, ret); + } + + return err; +} + +static int safexcel_cipher_send_inv(struct crypto_async_request *base, int ring, struct safexcel_request *request, int *commands, int *results) { - struct skcipher_request *req = skcipher_request_cast(async); - struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(req->base.tfm); + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(base->tfm); struct safexcel_crypto_priv *priv = ctx->priv; int ret; - ret = safexcel_invalidate_cache(async, priv, - ctx->base.ctxr_dma, ring, request); + ret = safexcel_invalidate_cache(base, priv, ctx->base.ctxr_dma, ring, + request); if (unlikely(ret)) return ret; @@ -408,9 +595,9 @@ static int safexcel_cipher_send_inv(struct crypto_async_request *async, return 0; } -static int safexcel_send(struct crypto_async_request *async, - int ring, struct safexcel_request *request, - int *commands, int *results) +static int safexcel_skcipher_send(struct crypto_async_request *async, int ring, + struct safexcel_request *request, + int *commands, int *results) { struct skcipher_request *req = skcipher_request_cast(async); struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(req->base.tfm); @@ -421,59 +608,108 @@ static int safexcel_send(struct crypto_async_request *async, BUG_ON(priv->version == EIP97 && sreq->needs_inv); if (sreq->needs_inv) - ret = safexcel_cipher_send_inv(async, ring, request, - commands, results); + ret = safexcel_cipher_send_inv(async, ring, request, commands, + results); + else + ret = safexcel_aes_send(async, ring, request, sreq, req->src, + req->dst, req->cryptlen, 0, 0, req->iv, + commands, results); + return ret; +} + +static int safexcel_aead_send(struct crypto_async_request *async, int ring, + struct safexcel_request *request, int *commands, + int *results) +{ + struct aead_request *req = aead_request_cast(async); + struct crypto_aead *tfm = crypto_aead_reqtfm(req); + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(req->base.tfm); + struct safexcel_cipher_req *sreq = aead_request_ctx(req); + struct safexcel_crypto_priv *priv = ctx->priv; + int ret; + + BUG_ON(priv->version == EIP97 && sreq->needs_inv); + + if (sreq->needs_inv) + ret = safexcel_cipher_send_inv(async, ring, request, commands, + results); else - ret = safexcel_aes_send(async, ring, request, + ret = safexcel_aes_send(async, ring, request, sreq, req->src, + req->dst, req->cryptlen, req->assoclen, + crypto_aead_authsize(tfm), req->iv, commands, results); return ret; } -static int safexcel_cipher_exit_inv(struct crypto_tfm *tfm) +static int safexcel_cipher_exit_inv(struct crypto_tfm *tfm, + struct crypto_async_request *base, + struct safexcel_cipher_req *sreq, + struct safexcel_inv_result *result) { struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm); struct safexcel_crypto_priv *priv = ctx->priv; - SKCIPHER_REQUEST_ON_STACK(req, __crypto_skcipher_cast(tfm)); - struct safexcel_cipher_req *sreq = skcipher_request_ctx(req); - struct safexcel_inv_result result = {}; int ring = ctx->base.ring; - memset(req, 0, sizeof(struct skcipher_request)); + init_completion(&result->completion); - /* create invalidation request */ - init_completion(&result.completion); - skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, - safexcel_inv_complete, &result); - - skcipher_request_set_tfm(req, __crypto_skcipher_cast(tfm)); - ctx = crypto_tfm_ctx(req->base.tfm); + ctx = crypto_tfm_ctx(base->tfm); ctx->base.exit_inv = true; sreq->needs_inv = true; spin_lock_bh(&priv->ring[ring].queue_lock); - crypto_enqueue_request(&priv->ring[ring].queue, &req->base); + crypto_enqueue_request(&priv->ring[ring].queue, base); spin_unlock_bh(&priv->ring[ring].queue_lock); queue_work(priv->ring[ring].workqueue, &priv->ring[ring].work_data.work); - wait_for_completion(&result.completion); + wait_for_completion(&result->completion); - if (result.error) { + if (result->error) { dev_warn(priv->dev, "cipher: sync: invalidate: completion error %d\n", - result.error); - return result.error; + result->error); + return result->error; } return 0; } -static int safexcel_aes(struct skcipher_request *req, - enum safexcel_cipher_direction dir, u32 mode) +static int safexcel_skcipher_exit_inv(struct crypto_tfm *tfm) { - struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(req->base.tfm); + EIP197_REQUEST_ON_STACK(req, skcipher, EIP197_SKCIPHER_REQ_SIZE); struct safexcel_cipher_req *sreq = skcipher_request_ctx(req); + struct safexcel_inv_result result = {}; + + memset(req, 0, sizeof(struct skcipher_request)); + + skcipher_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, + safexcel_inv_complete, &result); + skcipher_request_set_tfm(req, __crypto_skcipher_cast(tfm)); + + return safexcel_cipher_exit_inv(tfm, &req->base, sreq, &result); +} + +static int safexcel_aead_exit_inv(struct crypto_tfm *tfm) +{ + EIP197_REQUEST_ON_STACK(req, aead, EIP197_AEAD_REQ_SIZE); + struct safexcel_cipher_req *sreq = aead_request_ctx(req); + struct safexcel_inv_result result = {}; + + memset(req, 0, sizeof(struct aead_request)); + + aead_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, + safexcel_inv_complete, &result); + aead_request_set_tfm(req, __crypto_aead_cast(tfm)); + + return safexcel_cipher_exit_inv(tfm, &req->base, sreq, &result); +} + +static int safexcel_aes(struct crypto_async_request *base, + struct safexcel_cipher_req *sreq, + enum safexcel_cipher_direction dir, u32 mode) +{ + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(base->tfm); struct safexcel_crypto_priv *priv = ctx->priv; int ret, ring; @@ -489,7 +725,7 @@ static int safexcel_aes(struct skcipher_request *req, } else { ctx->base.ring = safexcel_select_ring(priv); ctx->base.ctxr = dma_pool_zalloc(priv->context_pool, - EIP197_GFP_FLAGS(req->base), + EIP197_GFP_FLAGS(*base), &ctx->base.ctxr_dma); if (!ctx->base.ctxr) return -ENOMEM; @@ -498,7 +734,7 @@ static int safexcel_aes(struct skcipher_request *req, ring = ctx->base.ring; spin_lock_bh(&priv->ring[ring].queue_lock); - ret = crypto_enqueue_request(&priv->ring[ring].queue, &req->base); + ret = crypto_enqueue_request(&priv->ring[ring].queue, base); spin_unlock_bh(&priv->ring[ring].queue_lock); queue_work(priv->ring[ring].workqueue, @@ -509,14 +745,14 @@ static int safexcel_aes(struct skcipher_request *req, static int safexcel_ecb_aes_encrypt(struct skcipher_request *req) { - return safexcel_aes(req, SAFEXCEL_ENCRYPT, - CONTEXT_CONTROL_CRYPTO_MODE_ECB); + return safexcel_aes(&req->base, skcipher_request_ctx(req), + SAFEXCEL_ENCRYPT, CONTEXT_CONTROL_CRYPTO_MODE_ECB); } static int safexcel_ecb_aes_decrypt(struct skcipher_request *req) { - return safexcel_aes(req, SAFEXCEL_DECRYPT, - CONTEXT_CONTROL_CRYPTO_MODE_ECB); + return safexcel_aes(&req->base, skcipher_request_ctx(req), + SAFEXCEL_DECRYPT, CONTEXT_CONTROL_CRYPTO_MODE_ECB); } static int safexcel_skcipher_cra_init(struct crypto_tfm *tfm) @@ -526,34 +762,64 @@ static int safexcel_skcipher_cra_init(struct crypto_tfm *tfm) container_of(tfm->__crt_alg, struct safexcel_alg_template, alg.skcipher.base); - ctx->priv = tmpl->priv; - ctx->base.send = safexcel_send; - ctx->base.handle_result = safexcel_handle_result; - crypto_skcipher_set_reqsize(__crypto_skcipher_cast(tfm), sizeof(struct safexcel_cipher_req)); + ctx->priv = tmpl->priv; + + ctx->base.send = safexcel_skcipher_send; + ctx->base.handle_result = safexcel_skcipher_handle_result; return 0; } -static void safexcel_skcipher_cra_exit(struct crypto_tfm *tfm) +static int safexcel_cipher_cra_exit(struct crypto_tfm *tfm) { struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm); - struct safexcel_crypto_priv *priv = ctx->priv; - int ret; - memzero_explicit(ctx->key, 8 * sizeof(u32)); + memzero_explicit(ctx->key, sizeof(ctx->key)); /* context not allocated, skip invalidation */ if (!ctx->base.ctxr) + return -ENOMEM; + + memzero_explicit(ctx->base.ctxr->data, sizeof(ctx->base.ctxr->data)); + return 0; +} + +static void safexcel_skcipher_cra_exit(struct crypto_tfm *tfm) +{ + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm); + struct safexcel_crypto_priv *priv = ctx->priv; + int ret; + + if (safexcel_cipher_cra_exit(tfm)) return; - memzero_explicit(ctx->base.ctxr->data, 8 * sizeof(u32)); + if (priv->version == EIP197) { + ret = safexcel_skcipher_exit_inv(tfm); + if (ret) + dev_warn(priv->dev, "skcipher: invalidation error %d\n", + ret); + } else { + dma_pool_free(priv->context_pool, ctx->base.ctxr, + ctx->base.ctxr_dma); + } +} + +static void safexcel_aead_cra_exit(struct crypto_tfm *tfm) +{ + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm); + struct safexcel_crypto_priv *priv = ctx->priv; + int ret; + + if (safexcel_cipher_cra_exit(tfm)) + return; if (priv->version == EIP197) { - ret = safexcel_cipher_exit_inv(tfm); + ret = safexcel_aead_exit_inv(tfm); if (ret) - dev_warn(priv->dev, "cipher: invalidation error %d\n", ret); + dev_warn(priv->dev, "aead: invalidation error %d\n", + ret); } else { dma_pool_free(priv->context_pool, ctx->base.ctxr, ctx->base.ctxr_dma); @@ -563,7 +829,7 @@ static void safexcel_skcipher_cra_exit(struct crypto_tfm *tfm) struct safexcel_alg_template safexcel_alg_ecb_aes = { .type = SAFEXCEL_ALG_TYPE_SKCIPHER, .alg.skcipher = { - .setkey = safexcel_aes_setkey, + .setkey = safexcel_skcipher_aes_setkey, .encrypt = safexcel_ecb_aes_encrypt, .decrypt = safexcel_ecb_aes_decrypt, .min_keysize = AES_MIN_KEY_SIZE, @@ -586,20 +852,20 @@ struct safexcel_alg_template safexcel_alg_ecb_aes = { static int safexcel_cbc_aes_encrypt(struct skcipher_request *req) { - return safexcel_aes(req, SAFEXCEL_ENCRYPT, - CONTEXT_CONTROL_CRYPTO_MODE_CBC); + return safexcel_aes(&req->base, skcipher_request_ctx(req), + SAFEXCEL_ENCRYPT, CONTEXT_CONTROL_CRYPTO_MODE_CBC); } static int safexcel_cbc_aes_decrypt(struct skcipher_request *req) { - return safexcel_aes(req, SAFEXCEL_DECRYPT, - CONTEXT_CONTROL_CRYPTO_MODE_CBC); + return safexcel_aes(&req->base, skcipher_request_ctx(req), + SAFEXCEL_DECRYPT, CONTEXT_CONTROL_CRYPTO_MODE_CBC); } struct safexcel_alg_template safexcel_alg_cbc_aes = { .type = SAFEXCEL_ALG_TYPE_SKCIPHER, .alg.skcipher = { - .setkey = safexcel_aes_setkey, + .setkey = safexcel_skcipher_aes_setkey, .encrypt = safexcel_cbc_aes_encrypt, .decrypt = safexcel_cbc_aes_decrypt, .min_keysize = AES_MIN_KEY_SIZE, @@ -620,3 +886,139 @@ struct safexcel_alg_template safexcel_alg_cbc_aes = { }, }, }; + +static int safexcel_aead_encrypt(struct aead_request *req) +{ + struct safexcel_cipher_req *creq = aead_request_ctx(req); + + return safexcel_aes(&req->base, creq, SAFEXCEL_ENCRYPT, + CONTEXT_CONTROL_CRYPTO_MODE_CBC); +} + +static int safexcel_aead_decrypt(struct aead_request *req) +{ + struct safexcel_cipher_req *creq = aead_request_ctx(req); + + return safexcel_aes(&req->base, creq, SAFEXCEL_DECRYPT, + CONTEXT_CONTROL_CRYPTO_MODE_CBC); +} + +static int safexcel_aead_cra_init(struct crypto_tfm *tfm) +{ + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm); + struct safexcel_alg_template *tmpl = + container_of(tfm->__crt_alg, struct safexcel_alg_template, + alg.aead.base); + + crypto_aead_set_reqsize(__crypto_aead_cast(tfm), + sizeof(struct safexcel_cipher_req)); + + ctx->priv = tmpl->priv; + + ctx->aead = true; + ctx->base.send = safexcel_aead_send; + ctx->base.handle_result = safexcel_aead_handle_result; + return 0; +} + +static int safexcel_aead_sha1_cra_init(struct crypto_tfm *tfm) +{ + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm); + + safexcel_aead_cra_init(tfm); + ctx->alg = CONTEXT_CONTROL_CRYPTO_ALG_SHA1; + ctx->state_sz = SHA1_DIGEST_SIZE; + return 0; +} + +struct safexcel_alg_template safexcel_alg_authenc_hmac_sha1_cbc_aes = { + .type = SAFEXCEL_ALG_TYPE_AEAD, + .alg.aead = { + .setkey = safexcel_aead_aes_setkey, + .encrypt = safexcel_aead_encrypt, + .decrypt = safexcel_aead_decrypt, + .ivsize = AES_BLOCK_SIZE, + .maxauthsize = SHA1_DIGEST_SIZE, + .base = { + .cra_name = "authenc(hmac(sha1),cbc(aes))", + .cra_driver_name = "safexcel-authenc-hmac-sha1-cbc-aes", + .cra_priority = 300, + .cra_flags = CRYPTO_ALG_TYPE_AEAD | CRYPTO_ALG_ASYNC | + CRYPTO_ALG_KERN_DRIVER_ONLY, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct safexcel_cipher_ctx), + .cra_alignmask = 0, + .cra_init = safexcel_aead_sha1_cra_init, + .cra_exit = safexcel_aead_cra_exit, + .cra_module = THIS_MODULE, + }, + }, +}; + +static int safexcel_aead_sha256_cra_init(struct crypto_tfm *tfm) +{ + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm); + + safexcel_aead_cra_init(tfm); + ctx->alg = CONTEXT_CONTROL_CRYPTO_ALG_SHA256; + ctx->state_sz = SHA256_DIGEST_SIZE; + return 0; +} + +struct safexcel_alg_template safexcel_alg_authenc_hmac_sha256_cbc_aes = { + .type = SAFEXCEL_ALG_TYPE_AEAD, + .alg.aead = { + .setkey = safexcel_aead_aes_setkey, + .encrypt = safexcel_aead_encrypt, + .decrypt = safexcel_aead_decrypt, + .ivsize = AES_BLOCK_SIZE, + .maxauthsize = SHA256_DIGEST_SIZE, + .base = { + .cra_name = "authenc(hmac(sha256),cbc(aes))", + .cra_driver_name = "safexcel-authenc-hmac-sha256-cbc-aes", + .cra_priority = 300, + .cra_flags = CRYPTO_ALG_TYPE_AEAD | CRYPTO_ALG_ASYNC | + CRYPTO_ALG_KERN_DRIVER_ONLY, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct safexcel_cipher_ctx), + .cra_alignmask = 0, + .cra_init = safexcel_aead_sha256_cra_init, + .cra_exit = safexcel_aead_cra_exit, + .cra_module = THIS_MODULE, + }, + }, +}; + +static int safexcel_aead_sha224_cra_init(struct crypto_tfm *tfm) +{ + struct safexcel_cipher_ctx *ctx = crypto_tfm_ctx(tfm); + + safexcel_aead_cra_init(tfm); + ctx->alg = CONTEXT_CONTROL_CRYPTO_ALG_SHA224; + ctx->state_sz = SHA256_DIGEST_SIZE; + return 0; +} + +struct safexcel_alg_template safexcel_alg_authenc_hmac_sha224_cbc_aes = { + .type = SAFEXCEL_ALG_TYPE_AEAD, + .alg.aead = { + .setkey = safexcel_aead_aes_setkey, + .encrypt = safexcel_aead_encrypt, + .decrypt = safexcel_aead_decrypt, + .ivsize = AES_BLOCK_SIZE, + .maxauthsize = SHA224_DIGEST_SIZE, + .base = { + .cra_name = "authenc(hmac(sha224),cbc(aes))", + .cra_driver_name = "safexcel-authenc-hmac-sha224-cbc-aes", + .cra_priority = 300, + .cra_flags = CRYPTO_ALG_TYPE_AEAD | CRYPTO_ALG_ASYNC | + CRYPTO_ALG_KERN_DRIVER_ONLY, + .cra_blocksize = AES_BLOCK_SIZE, + .cra_ctxsize = sizeof(struct safexcel_cipher_ctx), + .cra_alignmask = 0, + .cra_init = safexcel_aead_sha224_cra_init, + .cra_exit = safexcel_aead_cra_exit, + .cra_module = THIS_MODULE, + }, + }, +}; diff --git a/drivers/crypto/inside-secure/safexcel_hash.c b/drivers/crypto/inside-secure/safexcel_hash.c index 317b9e480312..c77b0e1655a8 100644 --- a/drivers/crypto/inside-secure/safexcel_hash.c +++ b/drivers/crypto/inside-secure/safexcel_hash.c @@ -50,16 +50,6 @@ struct safexcel_ahash_req { u8 cache_next[SHA256_BLOCK_SIZE] __aligned(sizeof(u32)); }; -struct safexcel_ahash_export_state { - u64 len; - u64 processed; - - u32 digest; - - u32 state[SHA256_DIGEST_SIZE / sizeof(u32)]; - u8 cache[SHA256_BLOCK_SIZE]; -}; - static void safexcel_hash_token(struct safexcel_command_desc *cdesc, u32 input_length, u32 result_length) { @@ -146,11 +136,8 @@ static int safexcel_handle_req_result(struct safexcel_crypto_priv *priv, int rin dev_err(priv->dev, "hash: result: could not retrieve the result descriptor\n"); *ret = PTR_ERR(rdesc); - } else if (rdesc->result_data.error_code) { - dev_err(priv->dev, - "hash: result: result descriptor error (%d)\n", - rdesc->result_data.error_code); - *ret = -EINVAL; + } else { + *ret = safexcel_rdesc_check_errors(priv, rdesc); } safexcel_complete(priv, ring); @@ -480,7 +467,7 @@ static int safexcel_ahash_exit_inv(struct crypto_tfm *tfm) { struct safexcel_ahash_ctx *ctx = crypto_tfm_ctx(tfm); struct safexcel_crypto_priv *priv = ctx->priv; - AHASH_REQUEST_ON_STACK(req, __crypto_ahash_cast(tfm)); + EIP197_REQUEST_ON_STACK(req, ahash, EIP197_AHASH_REQ_SIZE); struct safexcel_ahash_req *rctx = ahash_request_ctx(req); struct safexcel_inv_result result = {}; int ring = ctx->base.ring; @@ -912,8 +899,8 @@ static int safexcel_hmac_init_iv(struct ahash_request *areq, return crypto_ahash_export(areq, state); } -static int safexcel_hmac_setkey(const char *alg, const u8 *key, - unsigned int keylen, void *istate, void *ostate) +int safexcel_hmac_setkey(const char *alg, const u8 *key, unsigned int keylen, + void *istate, void *ostate) { struct ahash_request *areq; struct crypto_ahash *tfm; @@ -935,7 +922,7 @@ static int safexcel_hmac_setkey(const char *alg, const u8 *key, crypto_ahash_clear_flags(tfm, ~0); blocksize = crypto_tfm_alg_blocksize(crypto_ahash_tfm(tfm)); - ipad = kzalloc(2 * blocksize, GFP_KERNEL); + ipad = kcalloc(2, blocksize, GFP_KERNEL); if (!ipad) { ret = -ENOMEM; goto free_request; diff --git a/drivers/crypto/marvell/cesa.c b/drivers/crypto/marvell/cesa.c index f81fa4a3e66b..a4aa6813de4b 100644 --- a/drivers/crypto/marvell/cesa.c +++ b/drivers/crypto/marvell/cesa.c @@ -471,7 +471,7 @@ static int mv_cesa_probe(struct platform_device *pdev) sram_size = CESA_SA_MIN_SRAM_SIZE; cesa->sram_size = sram_size; - cesa->engines = devm_kzalloc(dev, caps->nengines * sizeof(*engines), + cesa->engines = devm_kcalloc(dev, caps->nengines, sizeof(*engines), GFP_KERNEL); if (!cesa->engines) return -ENOMEM; diff --git a/drivers/crypto/marvell/hash.c b/drivers/crypto/marvell/hash.c index e61b08566093..e34d80b6b7e5 100644 --- a/drivers/crypto/marvell/hash.c +++ b/drivers/crypto/marvell/hash.c @@ -1198,7 +1198,7 @@ static int mv_cesa_ahmac_setkey(const char *hash_alg_name, blocksize = crypto_tfm_alg_blocksize(crypto_ahash_tfm(tfm)); - ipad = kzalloc(2 * blocksize, GFP_KERNEL); + ipad = kcalloc(2, blocksize, GFP_KERNEL); if (!ipad) { ret = -ENOMEM; goto free_req; diff --git a/drivers/crypto/n2_core.c b/drivers/crypto/n2_core.c index 80e9c842aad4..ab6235b7ff22 100644 --- a/drivers/crypto/n2_core.c +++ b/drivers/crypto/n2_core.c @@ -1919,12 +1919,12 @@ static int grab_global_resources(void) goto out_hvapi_release; err = -ENOMEM; - cpu_to_cwq = kzalloc(sizeof(struct spu_queue *) * NR_CPUS, + cpu_to_cwq = kcalloc(NR_CPUS, sizeof(struct spu_queue *), GFP_KERNEL); if (!cpu_to_cwq) goto out_queue_cache_destroy; - cpu_to_mau = kzalloc(sizeof(struct spu_queue *) * NR_CPUS, + cpu_to_mau = kcalloc(NR_CPUS, sizeof(struct spu_queue *), GFP_KERNEL); if (!cpu_to_mau) goto out_free_cwq_table; diff --git a/drivers/crypto/nx/nx-842-powernv.c b/drivers/crypto/nx/nx-842-powernv.c index 1e87637c412d..36afd6d8753c 100644 --- a/drivers/crypto/nx/nx-842-powernv.c +++ b/drivers/crypto/nx/nx-842-powernv.c @@ -334,7 +334,7 @@ static int wait_for_csb(struct nx842_workmem *wmem, return -EPROTO; case CSB_CC_SEQUENCE: /* should not happen, we don't use chained CRBs */ - CSB_ERR(csb, "CRB seqeunce number error"); + CSB_ERR(csb, "CRB sequence number error"); return -EPROTO; case CSB_CC_UNKNOWN_CODE: CSB_ERR(csb, "Unknown subfunction code"); diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c index ad02aa63b519..d1a1c74fb56a 100644 --- a/drivers/crypto/omap-sham.c +++ b/drivers/crypto/omap-sham.c @@ -1087,7 +1087,7 @@ static void omap_sham_finish_req(struct ahash_request *req, int err) if (test_bit(FLAGS_SGS_COPIED, &dd->flags)) free_pages((unsigned long)sg_virt(ctx->sg), - get_order(ctx->sg->length)); + get_order(ctx->sg->length + ctx->bufcnt)); if (test_bit(FLAGS_SGS_ALLOCED, &dd->flags)) kfree(ctx->sg); diff --git a/drivers/crypto/picoxcell_crypto.c b/drivers/crypto/picoxcell_crypto.c index a4df966adbf6..321d5e2ac833 100644 --- a/drivers/crypto/picoxcell_crypto.c +++ b/drivers/crypto/picoxcell_crypto.c @@ -1169,8 +1169,7 @@ static void spacc_spacc_complete(unsigned long data) #ifdef CONFIG_PM static int spacc_suspend(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct spacc_engine *engine = platform_get_drvdata(pdev); + struct spacc_engine *engine = dev_get_drvdata(dev); /* * We only support standby mode. All we have to do is gate the clock to @@ -1184,8 +1183,7 @@ static int spacc_suspend(struct device *dev) static int spacc_resume(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct spacc_engine *engine = platform_get_drvdata(pdev); + struct spacc_engine *engine = dev_get_drvdata(dev); return clk_enable(engine->clk); } diff --git a/drivers/crypto/qat/qat_c3xxx/adf_drv.c b/drivers/crypto/qat/qat_c3xxx/adf_drv.c index f172171668ee..ba197f34c252 100644 --- a/drivers/crypto/qat/qat_c3xxx/adf_drv.c +++ b/drivers/crypto/qat/qat_c3xxx/adf_drv.c @@ -329,5 +329,7 @@ module_exit(adfdrv_release); MODULE_LICENSE("Dual BSD/GPL"); MODULE_AUTHOR("Intel"); +MODULE_FIRMWARE(ADF_C3XXX_FW); +MODULE_FIRMWARE(ADF_C3XXX_MMP); MODULE_DESCRIPTION("Intel(R) QuickAssist Technology"); MODULE_VERSION(ADF_DRV_VERSION); diff --git a/drivers/crypto/qat/qat_c62x/adf_drv.c b/drivers/crypto/qat/qat_c62x/adf_drv.c index 58a984c9c3ec..59a5a0df50b6 100644 --- a/drivers/crypto/qat/qat_c62x/adf_drv.c +++ b/drivers/crypto/qat/qat_c62x/adf_drv.c @@ -329,5 +329,7 @@ module_exit(adfdrv_release); MODULE_LICENSE("Dual BSD/GPL"); MODULE_AUTHOR("Intel"); +MODULE_FIRMWARE(ADF_C62X_FW); +MODULE_FIRMWARE(ADF_C62X_MMP); MODULE_DESCRIPTION("Intel(R) QuickAssist Technology"); MODULE_VERSION(ADF_DRV_VERSION); diff --git a/drivers/crypto/qat/qat_common/adf_isr.c b/drivers/crypto/qat/qat_common/adf_isr.c index 06d49017a52b..cd1cdf5305bc 100644 --- a/drivers/crypto/qat/qat_common/adf_isr.c +++ b/drivers/crypto/qat/qat_common/adf_isr.c @@ -238,7 +238,7 @@ static int adf_isr_alloc_msix_entry_table(struct adf_accel_dev *accel_dev) if (!accel_dev->pf.vf_info) msix_num_entries += hw_data->num_banks; - entries = kzalloc_node(msix_num_entries * sizeof(*entries), + entries = kcalloc_node(msix_num_entries, sizeof(*entries), GFP_KERNEL, dev_to_node(&GET_DEV(accel_dev))); if (!entries) return -ENOMEM; diff --git a/drivers/crypto/qat/qat_common/qat_uclo.c b/drivers/crypto/qat/qat_common/qat_uclo.c index 98d22c2096e3..6bd8f6a2a24f 100644 --- a/drivers/crypto/qat/qat_common/qat_uclo.c +++ b/drivers/crypto/qat/qat_common/qat_uclo.c @@ -1162,8 +1162,9 @@ static int qat_uclo_map_suof(struct icp_qat_fw_loader_handle *handle, suof_handle->img_table.num_simgs = suof_ptr->num_chunks - 1; if (suof_handle->img_table.num_simgs != 0) { - suof_img_hdr = kzalloc(suof_handle->img_table.num_simgs * - sizeof(img_header), GFP_KERNEL); + suof_img_hdr = kcalloc(suof_handle->img_table.num_simgs, + sizeof(img_header), + GFP_KERNEL); if (!suof_img_hdr) return -ENOMEM; suof_handle->img_table.simg_hdr = suof_img_hdr; diff --git a/drivers/crypto/qat/qat_dh895xcc/adf_drv.c b/drivers/crypto/qat/qat_dh895xcc/adf_drv.c index 2ce01f010c74..be5c5a988ca5 100644 --- a/drivers/crypto/qat/qat_dh895xcc/adf_drv.c +++ b/drivers/crypto/qat/qat_dh895xcc/adf_drv.c @@ -332,5 +332,6 @@ module_exit(adfdrv_release); MODULE_LICENSE("Dual BSD/GPL"); MODULE_AUTHOR("Intel"); MODULE_FIRMWARE(ADF_DH895XCC_FW); +MODULE_FIRMWARE(ADF_DH895XCC_MMP); MODULE_DESCRIPTION("Intel(R) QuickAssist Technology"); MODULE_VERSION(ADF_DRV_VERSION); diff --git a/drivers/crypto/stm32/stm32-hash.c b/drivers/crypto/stm32/stm32-hash.c index 981e45692695..cdc96f1bb917 100644 --- a/drivers/crypto/stm32/stm32-hash.c +++ b/drivers/crypto/stm32/stm32-hash.c @@ -970,8 +970,9 @@ static int stm32_hash_export(struct ahash_request *req, void *out) while (!(stm32_hash_read(hdev, HASH_SR) & HASH_SR_DATA_INPUT_READY)) cpu_relax(); - rctx->hw_context = kmalloc(sizeof(u32) * (3 + HASH_CSR_REGISTER_NUMBER), - GFP_KERNEL); + rctx->hw_context = kmalloc_array(3 + HASH_CSR_REGISTER_NUMBER, + sizeof(u32), + GFP_KERNEL); preg = rctx->hw_context; diff --git a/drivers/crypto/talitos.c b/drivers/crypto/talitos.c index 7cebf0a6ffbc..cf14f099ce4a 100644 --- a/drivers/crypto/talitos.c +++ b/drivers/crypto/talitos.c @@ -3393,8 +3393,10 @@ static int talitos_probe(struct platform_device *ofdev) } } - priv->chan = devm_kzalloc(dev, sizeof(struct talitos_channel) * - priv->num_channels, GFP_KERNEL); + priv->chan = devm_kcalloc(dev, + priv->num_channels, + sizeof(struct talitos_channel), + GFP_KERNEL); if (!priv->chan) { dev_err(dev, "failed to allocate channel management space\n"); err = -ENOMEM; @@ -3411,9 +3413,10 @@ static int talitos_probe(struct platform_device *ofdev) spin_lock_init(&priv->chan[i].head_lock); spin_lock_init(&priv->chan[i].tail_lock); - priv->chan[i].fifo = devm_kzalloc(dev, - sizeof(struct talitos_request) * - priv->fifo_len, GFP_KERNEL); + priv->chan[i].fifo = devm_kcalloc(dev, + priv->fifo_len, + sizeof(struct talitos_request), + GFP_KERNEL); if (!priv->chan[i].fifo) { dev_err(dev, "failed to allocate request fifo %d\n", i); err = -ENOMEM; diff --git a/drivers/crypto/virtio/virtio_crypto_algs.c b/drivers/crypto/virtio/virtio_crypto_algs.c index ba190cfa7aa1..af6a908dfa7a 100644 --- a/drivers/crypto/virtio/virtio_crypto_algs.c +++ b/drivers/crypto/virtio/virtio_crypto_algs.c @@ -371,7 +371,7 @@ __virtio_crypto_ablkcipher_do_req(struct virtio_crypto_sym_request *vc_sym_req, /* Why 3? outhdr + iv + inhdr */ sg_total = src_nents + dst_nents + 3; - sgs = kzalloc_node(sg_total * sizeof(*sgs), GFP_ATOMIC, + sgs = kcalloc_node(sg_total, sizeof(*sgs), GFP_ATOMIC, dev_to_node(&vcrypto->vdev->dev)); if (!sgs) return -ENOMEM; diff --git a/drivers/crypto/vmx/aes.c b/drivers/crypto/vmx/aes.c index 96072b9b55c4..d7316f7a3a69 100644 --- a/drivers/crypto/vmx/aes.c +++ b/drivers/crypto/vmx/aes.c @@ -48,8 +48,6 @@ static int p8_aes_init(struct crypto_tfm *tfm) alg, PTR_ERR(fallback)); return PTR_ERR(fallback); } - printk(KERN_INFO "Using '%s' as fallback implementation.\n", - crypto_tfm_alg_driver_name((struct crypto_tfm *) fallback)); crypto_cipher_set_flags(fallback, crypto_cipher_get_flags((struct diff --git a/drivers/crypto/vmx/aes_cbc.c b/drivers/crypto/vmx/aes_cbc.c index 7394d35d5936..5285ece4f33a 100644 --- a/drivers/crypto/vmx/aes_cbc.c +++ b/drivers/crypto/vmx/aes_cbc.c @@ -52,9 +52,6 @@ static int p8_aes_cbc_init(struct crypto_tfm *tfm) alg, PTR_ERR(fallback)); return PTR_ERR(fallback); } - printk(KERN_INFO "Using '%s' as fallback implementation.\n", - crypto_skcipher_driver_name(fallback)); - crypto_skcipher_set_flags( fallback, diff --git a/drivers/crypto/vmx/aes_ctr.c b/drivers/crypto/vmx/aes_ctr.c index fc60d00a2e84..cd777c75291d 100644 --- a/drivers/crypto/vmx/aes_ctr.c +++ b/drivers/crypto/vmx/aes_ctr.c @@ -50,8 +50,6 @@ static int p8_aes_ctr_init(struct crypto_tfm *tfm) alg, PTR_ERR(fallback)); return PTR_ERR(fallback); } - printk(KERN_INFO "Using '%s' as fallback implementation.\n", - crypto_skcipher_driver_name(fallback)); crypto_skcipher_set_flags( fallback, diff --git a/drivers/crypto/vmx/aes_xts.c b/drivers/crypto/vmx/aes_xts.c index 8cd6e62e4c90..8bd9aff0f55f 100644 --- a/drivers/crypto/vmx/aes_xts.c +++ b/drivers/crypto/vmx/aes_xts.c @@ -53,8 +53,6 @@ static int p8_aes_xts_init(struct crypto_tfm *tfm) alg, PTR_ERR(fallback)); return PTR_ERR(fallback); } - printk(KERN_INFO "Using '%s' as fallback implementation.\n", - crypto_skcipher_driver_name(fallback)); crypto_skcipher_set_flags( fallback, diff --git a/drivers/crypto/vmx/aesp8-ppc.pl b/drivers/crypto/vmx/aesp8-ppc.pl index 0b4a293b8a1e..d6a9f63d65ba 100644 --- a/drivers/crypto/vmx/aesp8-ppc.pl +++ b/drivers/crypto/vmx/aesp8-ppc.pl @@ -1,12 +1,51 @@ #! /usr/bin/env perl -# Copyright 2014-2016 The OpenSSL Project Authors. All Rights Reserved. +# SPDX-License-Identifier: GPL-2.0 + +# This code is taken from CRYPTOGAMs[1] and is included here using the option +# in the license to distribute the code under the GPL. Therefore this program +# is free software; you can redistribute it and/or modify it under the terms of +# the GNU General Public License version 2 as published by the Free Software +# Foundation. # -# Licensed under the OpenSSL license (the "License"). You may not use -# this file except in compliance with the License. You can obtain a copy -# in the file LICENSE in the source distribution or at -# https://www.openssl.org/source/license.html +# [1] https://www.openssl.org/~appro/cryptogams/ +# Copyright (c) 2006-2017, CRYPTOGAMS by +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain copyright notices, +# this list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above +# copyright notice, this list of conditions and the following +# disclaimer in the documentation and/or other materials +# provided with the distribution. # +# * Neither the name of the CRYPTOGAMS nor the names of its +# copyright holder and contributors may be used to endorse or +# promote products derived from this software without specific +# prior written permission. +# +# ALTERNATIVELY, provided that this notice is retained in full, this +# product may be distributed under the terms of the GNU General Public +# License (GPL), in which case the provisions of the GPL apply INSTEAD OF +# those given above. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + # ==================================================================== # Written by Andy Polyakov for the OpenSSL # project. The module is, however, dual licensed under OpenSSL and diff --git a/drivers/crypto/vmx/ghash.c b/drivers/crypto/vmx/ghash.c index 27a94a119009..1c4b5b889fba 100644 --- a/drivers/crypto/vmx/ghash.c +++ b/drivers/crypto/vmx/ghash.c @@ -64,8 +64,6 @@ static int p8_ghash_init_tfm(struct crypto_tfm *tfm) alg, PTR_ERR(fallback)); return PTR_ERR(fallback); } - printk(KERN_INFO "Using '%s' as fallback implementation.\n", - crypto_tfm_alg_driver_name(crypto_shash_tfm(fallback))); crypto_shash_set_flags(fallback, crypto_shash_get_flags((struct crypto_shash diff --git a/drivers/crypto/vmx/ghashp8-ppc.pl b/drivers/crypto/vmx/ghashp8-ppc.pl index d8429cb71f02..f746af271460 100644 --- a/drivers/crypto/vmx/ghashp8-ppc.pl +++ b/drivers/crypto/vmx/ghashp8-ppc.pl @@ -1,5 +1,14 @@ #!/usr/bin/env perl +# SPDX-License-Identifier: GPL-2.0 + +# This code is taken from the OpenSSL project but the author (Andy Polyakov) +# has relicensed it under the GPLv2. Therefore this program is free software; +# you can redistribute it and/or modify it under the terms of the GNU General +# Public License version 2 as published by the Free Software Foundation. # +# The original headers, including the original license headers, are +# included below for completeness. + # ==================================================================== # Written by Andy Polyakov for the OpenSSL # project. The module is, however, dual licensed under OpenSSL and diff --git a/drivers/dax/device.c b/drivers/dax/device.c index aff2c1594220..de2f8297a210 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -594,7 +594,7 @@ struct dev_dax *devm_create_dev_dax(struct dax_region *dax_region, if (!count) return ERR_PTR(-EINVAL); - dev_dax = kzalloc(sizeof(*dev_dax) + sizeof(*res) * count, GFP_KERNEL); + dev_dax = kzalloc(struct_size(dev_dax, res, count), GFP_KERNEL); if (!dev_dax) return ERR_PTR(-ENOMEM); diff --git a/drivers/dax/super.c b/drivers/dax/super.c index 2b2332b605e4..903d9c473749 100644 --- a/drivers/dax/super.c +++ b/drivers/dax/super.c @@ -74,42 +74,43 @@ EXPORT_SYMBOL_GPL(fs_dax_get_by_bdev); /** * __bdev_dax_supported() - Check if the device supports dax for filesystem - * @sb: The superblock of the device + * @bdev: block device to check * @blocksize: The block size of the device * * This is a library function for filesystems to check if the block device * can be mounted with dax option. * - * Return: negative errno if unsupported, 0 if supported. + * Return: true if supported, false if unsupported */ -int __bdev_dax_supported(struct super_block *sb, int blocksize) +bool __bdev_dax_supported(struct block_device *bdev, int blocksize) { - struct block_device *bdev = sb->s_bdev; struct dax_device *dax_dev; + bool dax_enabled = false; pgoff_t pgoff; int err, id; void *kaddr; pfn_t pfn; long len; + char buf[BDEVNAME_SIZE]; if (blocksize != PAGE_SIZE) { - pr_debug("VFS (%s): error: unsupported blocksize for dax\n", - sb->s_id); - return -EINVAL; + pr_debug("%s: error: unsupported blocksize for dax\n", + bdevname(bdev, buf)); + return false; } err = bdev_dax_pgoff(bdev, 0, PAGE_SIZE, &pgoff); if (err) { - pr_debug("VFS (%s): error: unaligned partition for dax\n", - sb->s_id); - return err; + pr_debug("%s: error: unaligned partition for dax\n", + bdevname(bdev, buf)); + return false; } dax_dev = dax_get_by_host(bdev->bd_disk->disk_name); if (!dax_dev) { - pr_debug("VFS (%s): error: device does not support dax\n", - sb->s_id); - return -EOPNOTSUPP; + pr_debug("%s: error: device does not support dax\n", + bdevname(bdev, buf)); + return false; } id = dax_read_lock(); @@ -119,9 +120,9 @@ int __bdev_dax_supported(struct super_block *sb, int blocksize) put_dax(dax_dev); if (len < 1) { - pr_debug("VFS (%s): error: dax access failed (%ld)\n", - sb->s_id, len); - return len < 0 ? len : -EIO; + pr_debug("%s: error: dax access failed (%ld)\n", + bdevname(bdev, buf), len); + return false; } if (IS_ENABLED(CONFIG_FS_DAX_LIMITED) && pfn_t_special(pfn)) { @@ -134,15 +135,22 @@ int __bdev_dax_supported(struct super_block *sb, int blocksize) * on being able to do (page_address(pfn_to_page())). */ WARN_ON(IS_ENABLED(CONFIG_ARCH_HAS_PMEM_API)); + dax_enabled = true; } else if (pfn_t_devmap(pfn)) { - /* pass */; - } else { - pr_debug("VFS (%s): error: dax support not enabled\n", - sb->s_id); - return -EOPNOTSUPP; + struct dev_pagemap *pgmap; + + pgmap = get_dev_pagemap(pfn_t_to_pfn(pfn), NULL); + if (pgmap && pgmap->type == MEMORY_DEVICE_FS_DAX) + dax_enabled = true; + put_dev_pagemap(pgmap); } - return 0; + if (!dax_enabled) { + pr_debug("%s: error: dax support not enabled\n", + bdevname(bdev, buf)); + return false; + } + return true; } EXPORT_SYMBOL_GPL(__bdev_dax_supported); #endif @@ -182,8 +190,7 @@ static ssize_t write_cache_show(struct device *dev, if (!dax_dev) return -ENXIO; - rc = sprintf(buf, "%d\n", !!test_bit(DAXDEV_WRITE_CACHE, - &dax_dev->flags)); + rc = sprintf(buf, "%d\n", !!dax_write_cache_enabled(dax_dev)); put_dax(dax_dev); return rc; } @@ -201,10 +208,8 @@ static ssize_t write_cache_store(struct device *dev, if (rc) len = rc; - else if (write_cache) - set_bit(DAXDEV_WRITE_CACHE, &dax_dev->flags); else - clear_bit(DAXDEV_WRITE_CACHE, &dax_dev->flags); + dax_write_cache(dax_dev, write_cache); put_dax(dax_dev); return len; @@ -282,11 +287,21 @@ size_t dax_copy_from_iter(struct dax_device *dax_dev, pgoff_t pgoff, void *addr, } EXPORT_SYMBOL_GPL(dax_copy_from_iter); +size_t dax_copy_to_iter(struct dax_device *dax_dev, pgoff_t pgoff, void *addr, + size_t bytes, struct iov_iter *i) +{ + if (!dax_alive(dax_dev)) + return 0; + + return dax_dev->ops->copy_to_iter(dax_dev, pgoff, addr, bytes, i); +} +EXPORT_SYMBOL_GPL(dax_copy_to_iter); + #ifdef CONFIG_ARCH_HAS_PMEM_API void arch_wb_cache_pmem(void *addr, size_t size); void dax_flush(struct dax_device *dax_dev, void *addr, size_t size) { - if (unlikely(!test_bit(DAXDEV_WRITE_CACHE, &dax_dev->flags))) + if (unlikely(!dax_write_cache_enabled(dax_dev))) return; arch_wb_cache_pmem(addr, size); diff --git a/drivers/dca/dca-core.c b/drivers/dca/dca-core.c index 7afbb28d6a0f..1bc5ffb338c8 100644 --- a/drivers/dca/dca-core.c +++ b/drivers/dca/dca-core.c @@ -270,7 +270,7 @@ EXPORT_SYMBOL_GPL(dca_remove_requester); * @dev - the device that wants dca service * @cpu - the cpuid as returned by get_cpu() */ -u8 dca_common_get_tag(struct device *dev, int cpu) +static u8 dca_common_get_tag(struct device *dev, int cpu) { struct dca_provider *dca; u8 tag; diff --git a/drivers/devfreq/devfreq.c b/drivers/devfreq/devfreq.c index fe2af6aa88fc..0b5b3abe054e 100644 --- a/drivers/devfreq/devfreq.c +++ b/drivers/devfreq/devfreq.c @@ -628,14 +628,15 @@ struct devfreq *devfreq_add_device(struct device *dev, goto err_dev; } - devfreq->trans_table = devm_kzalloc(&devfreq->dev, - sizeof(unsigned int) * - devfreq->profile->max_state * - devfreq->profile->max_state, - GFP_KERNEL); - devfreq->time_in_state = devm_kzalloc(&devfreq->dev, - sizeof(unsigned long) * + devfreq->trans_table = + devm_kzalloc(&devfreq->dev, + array3_size(sizeof(unsigned int), + devfreq->profile->max_state, + devfreq->profile->max_state), + GFP_KERNEL); + devfreq->time_in_state = devm_kcalloc(&devfreq->dev, devfreq->profile->max_state, + sizeof(unsigned long), GFP_KERNEL); devfreq->last_stat_updated = jiffies; diff --git a/drivers/devfreq/event/exynos-ppmu.c b/drivers/devfreq/event/exynos-ppmu.c index d96e3dc71cf8..3cd6a184fe7c 100644 --- a/drivers/devfreq/event/exynos-ppmu.c +++ b/drivers/devfreq/event/exynos-ppmu.c @@ -518,7 +518,7 @@ static int of_get_devfreq_events(struct device_node *np, event_ops = exynos_bus_get_ops(np); count = of_get_child_count(events_np); - desc = devm_kzalloc(dev, sizeof(*desc) * count, GFP_KERNEL); + desc = devm_kcalloc(dev, count, sizeof(*desc), GFP_KERNEL); if (!desc) return -ENOMEM; info->num_events = count; diff --git a/drivers/dma-buf/sync_debug.h b/drivers/dma-buf/sync_debug.h index d615a89f774c..05e33f937ad0 100644 --- a/drivers/dma-buf/sync_debug.h +++ b/drivers/dma-buf/sync_debug.h @@ -62,8 +62,6 @@ struct sync_pt { struct rb_node node; }; -#ifdef CONFIG_SW_SYNC - extern const struct file_operations sw_sync_debugfs_fops; void sync_timeline_debug_add(struct sync_timeline *obj); @@ -72,12 +70,4 @@ void sync_file_debug_add(struct sync_file *fence); void sync_file_debug_remove(struct sync_file *fence); void sync_dump(void); -#else -# define sync_timeline_debug_add(obj) -# define sync_timeline_debug_remove(obj) -# define sync_file_debug_add(fence) -# define sync_file_debug_remove(fence) -# define sync_dump() -#endif - #endif /* _LINUX_SYNC_H */ diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig index 6d61cd023633..ca1680afa20a 100644 --- a/drivers/dma/Kconfig +++ b/drivers/dma/Kconfig @@ -151,13 +151,6 @@ config DMA_JZ4780 If you have a board based on such a SoC and wish to use DMA for devices which can use the DMA controller, say Y or M here. -config DMA_OMAP - tristate "OMAP DMA support" - depends on ARCH_OMAP || COMPILE_TEST - select DMA_ENGINE - select DMA_VIRTUAL_CHANNELS - select TI_DMA_CROSSBAR if (SOC_DRA7XX || COMPILE_TEST) - config DMA_SA11X0 tristate "SA-11x0 DMA support" depends on ARCH_SA1100 || COMPILE_TEST @@ -574,28 +567,6 @@ config TIMB_DMA help Enable support for the Timberdale FPGA DMA engine. -config TI_CPPI41 - tristate "CPPI 4.1 DMA support" - depends on (ARCH_OMAP || ARCH_DAVINCI_DA8XX) - select DMA_ENGINE - help - The Communications Port Programming Interface (CPPI) 4.1 DMA engine - is currently used by the USB driver on AM335x and DA8xx platforms. - -config TI_DMA_CROSSBAR - bool - -config TI_EDMA - bool "TI EDMA support" - depends on ARCH_DAVINCI || ARCH_OMAP || ARCH_KEYSTONE || COMPILE_TEST - select DMA_ENGINE - select DMA_VIRTUAL_CHANNELS - select TI_DMA_CROSSBAR if (ARCH_OMAP || COMPILE_TEST) - default n - help - Enable support for the TI EDMA controller. This DMA - engine is found on TI DaVinci and AM33xx parts. - config XGENE_DMA tristate "APM X-Gene DMA support" depends on ARCH_XGENE || COMPILE_TEST @@ -653,6 +624,8 @@ source "drivers/dma/hsu/Kconfig" source "drivers/dma/sh/Kconfig" +source "drivers/dma/ti/Kconfig" + # clients comment "DMA Clients" depends on DMA_ENGINE diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile index 0f62a4d49aab..203a99d68315 100644 --- a/drivers/dma/Makefile +++ b/drivers/dma/Makefile @@ -24,7 +24,6 @@ obj-$(CONFIG_COH901318) += coh901318.o coh901318_lli.o obj-$(CONFIG_DMA_BCM2835) += bcm2835-dma.o obj-$(CONFIG_DMA_JZ4740) += dma-jz4740.o obj-$(CONFIG_DMA_JZ4780) += dma-jz4780.o -obj-$(CONFIG_DMA_OMAP) += omap-dma.o obj-$(CONFIG_DMA_SA11X0) += sa11x0-dma.o obj-$(CONFIG_DMA_SUN4I) += sun4i-dma.o obj-$(CONFIG_DMA_SUN6I) += sun6i-dma.o @@ -69,13 +68,11 @@ obj-$(CONFIG_TXX9_DMAC) += txx9dmac.o obj-$(CONFIG_TEGRA20_APB_DMA) += tegra20-apb-dma.o obj-$(CONFIG_TEGRA210_ADMA) += tegra210-adma.o obj-$(CONFIG_TIMB_DMA) += timb_dma.o -obj-$(CONFIG_TI_CPPI41) += cppi41.o -obj-$(CONFIG_TI_DMA_CROSSBAR) += ti-dma-crossbar.o -obj-$(CONFIG_TI_EDMA) += edma.o obj-$(CONFIG_XGENE_DMA) += xgene-dma.o obj-$(CONFIG_ZX_DMA) += zx_dma.o obj-$(CONFIG_ST_FDMA) += st_fdma.o obj-y += mediatek/ obj-y += qcom/ +obj-y += ti/ obj-y += xilinx/ diff --git a/drivers/dma/at_hdmac.c b/drivers/dma/at_hdmac.c index a861b5b4d443..75f38d19fcbe 100644 --- a/drivers/dma/at_hdmac.c +++ b/drivers/dma/at_hdmac.c @@ -2041,8 +2041,7 @@ static void at_dma_shutdown(struct platform_device *pdev) static int at_dma_prepare(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct at_dma *atdma = platform_get_drvdata(pdev); + struct at_dma *atdma = dev_get_drvdata(dev); struct dma_chan *chan, *_chan; list_for_each_entry_safe(chan, _chan, &atdma->dma_common.channels, @@ -2076,8 +2075,7 @@ static void atc_suspend_cyclic(struct at_dma_chan *atchan) static int at_dma_suspend_noirq(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct at_dma *atdma = platform_get_drvdata(pdev); + struct at_dma *atdma = dev_get_drvdata(dev); struct dma_chan *chan, *_chan; /* preserve data */ @@ -2118,8 +2116,7 @@ static void atc_resume_cyclic(struct at_dma_chan *atchan) static int at_dma_resume_noirq(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct at_dma *atdma = platform_get_drvdata(pdev); + struct at_dma *atdma = dev_get_drvdata(dev); struct dma_chan *chan, *_chan; /* bring back DMA controller */ diff --git a/drivers/dma/at_xdmac.c b/drivers/dma/at_xdmac.c index 94236ec9d410..4bf72561667c 100644 --- a/drivers/dma/at_xdmac.c +++ b/drivers/dma/at_xdmac.c @@ -1833,8 +1833,7 @@ static void at_xdmac_free_chan_resources(struct dma_chan *chan) #ifdef CONFIG_PM static int atmel_xdmac_prepare(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct at_xdmac *atxdmac = platform_get_drvdata(pdev); + struct at_xdmac *atxdmac = dev_get_drvdata(dev); struct dma_chan *chan, *_chan; list_for_each_entry_safe(chan, _chan, &atxdmac->dma.channels, device_node) { @@ -1853,8 +1852,7 @@ static int atmel_xdmac_prepare(struct device *dev) #ifdef CONFIG_PM_SLEEP static int atmel_xdmac_suspend(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct at_xdmac *atxdmac = platform_get_drvdata(pdev); + struct at_xdmac *atxdmac = dev_get_drvdata(dev); struct dma_chan *chan, *_chan; list_for_each_entry_safe(chan, _chan, &atxdmac->dma.channels, device_node) { @@ -1878,8 +1876,7 @@ static int atmel_xdmac_suspend(struct device *dev) static int atmel_xdmac_resume(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct at_xdmac *atxdmac = platform_get_drvdata(pdev); + struct at_xdmac *atxdmac = dev_get_drvdata(dev); struct at_xdmac_chan *atchan; struct dma_chan *chan, *_chan; int i; diff --git a/drivers/dma/bcm-sba-raid.c b/drivers/dma/bcm-sba-raid.c index 3956a018bf5a..72878ac5c78d 100644 --- a/drivers/dma/bcm-sba-raid.c +++ b/drivers/dma/bcm-sba-raid.c @@ -1499,9 +1499,8 @@ static int sba_prealloc_channel_resources(struct sba_device *sba) for (i = 0; i < sba->max_req; i++) { req = devm_kzalloc(sba->dev, - sizeof(*req) + - sba->max_cmd_per_req * sizeof(req->cmds[0]), - GFP_KERNEL); + struct_size(req, cmds, sba->max_cmd_per_req), + GFP_KERNEL); if (!req) { ret = -ENOMEM; goto fail_free_cmds_pool; diff --git a/drivers/dma/bestcomm/bestcomm.c b/drivers/dma/bestcomm/bestcomm.c index 7a67b8345092..d91cbbe7a48f 100644 --- a/drivers/dma/bestcomm/bestcomm.c +++ b/drivers/dma/bestcomm/bestcomm.c @@ -87,7 +87,8 @@ bcom_task_alloc(int bd_count, int bd_size, int priv_size) /* Init the BDs, if needed */ if (bd_count) { - tsk->cookie = kmalloc(sizeof(void*) * bd_count, GFP_KERNEL); + tsk->cookie = kmalloc_array(bd_count, sizeof(void *), + GFP_KERNEL); if (!tsk->cookie) goto error; diff --git a/drivers/dma/dma-axi-dmac.c b/drivers/dma/dma-axi-dmac.c index 2419fe524daa..15b2453d2647 100644 --- a/drivers/dma/dma-axi-dmac.c +++ b/drivers/dma/dma-axi-dmac.c @@ -687,7 +687,7 @@ static int axi_dmac_probe(struct platform_device *pdev) if (ret) goto err_unregister_device; - ret = request_irq(dmac->irq, axi_dmac_interrupt_handler, 0, + ret = request_irq(dmac->irq, axi_dmac_interrupt_handler, IRQF_SHARED, dev_name(&pdev->dev), dmac); if (ret) goto err_unregister_of; diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c index b451354735d3..08ba8473a284 100644 --- a/drivers/dma/dmaengine.c +++ b/drivers/dma/dmaengine.c @@ -38,7 +38,7 @@ * Each device has a channels list, which runs unlocked but is never modified * once the device is registered, it's just setup by the driver. * - * See Documentation/dmaengine.txt for more details + * See Documentation/driver-api/dmaengine for more details */ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt diff --git a/drivers/dma/dmatest.c b/drivers/dma/dmatest.c index b9339524d5bd..aa1712beb0cc 100644 --- a/drivers/dma/dmatest.c +++ b/drivers/dma/dmatest.c @@ -468,6 +468,8 @@ static int dmatest_func(void *data) unsigned long long total_len = 0; u8 align = 0; bool is_memset = false; + dma_addr_t *srcs; + dma_addr_t *dma_pq; set_freezable(); @@ -551,6 +553,14 @@ static int dmatest_func(void *data) set_user_nice(current, 10); + srcs = kcalloc(src_cnt, sizeof(dma_addr_t), GFP_KERNEL); + if (!srcs) + goto err_dstbuf; + + dma_pq = kcalloc(dst_cnt, sizeof(dma_addr_t), GFP_KERNEL); + if (!dma_pq) + goto err_srcs_array; + /* * src and dst buffers are freed by ourselves below */ @@ -561,7 +571,6 @@ static int dmatest_func(void *data) && !(params->iterations && total_tests >= params->iterations)) { struct dma_async_tx_descriptor *tx = NULL; struct dmaengine_unmap_data *um; - dma_addr_t srcs[src_cnt]; dma_addr_t *dsts; unsigned int src_off, dst_off, len; @@ -676,8 +685,6 @@ static int dmatest_func(void *data) srcs, src_cnt, len, flags); else if (thread->type == DMA_PQ) { - dma_addr_t dma_pq[dst_cnt]; - for (i = 0; i < dst_cnt; i++) dma_pq[i] = dsts[i] + dst_off; tx = dev->device_prep_dma_pq(chan, dma_pq, srcs, @@ -779,6 +786,9 @@ static int dmatest_func(void *data) runtime = ktime_to_us(ktime); ret = 0; + kfree(dma_pq); +err_srcs_array: + kfree(srcs); err_dstbuf: for (i = 0; thread->udsts[i]; i++) kfree(thread->udsts[i]); diff --git a/drivers/dma/dw/platform.c b/drivers/dma/dw/platform.c index bc31fe802061..f62dd0944908 100644 --- a/drivers/dma/dw/platform.c +++ b/drivers/dma/dw/platform.c @@ -293,8 +293,7 @@ MODULE_DEVICE_TABLE(acpi, dw_dma_acpi_id_table); static int dw_suspend_late(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct dw_dma_chip *chip = platform_get_drvdata(pdev); + struct dw_dma_chip *chip = dev_get_drvdata(dev); dw_dma_disable(chip); clk_disable_unprepare(chip->clk); @@ -304,8 +303,7 @@ static int dw_suspend_late(struct device *dev) static int dw_resume_early(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct dw_dma_chip *chip = platform_get_drvdata(pdev); + struct dw_dma_chip *chip = dev_get_drvdata(dev); int ret; ret = clk_prepare_enable(chip->clk); diff --git a/drivers/dma/fsldma.c b/drivers/dma/fsldma.c index 3eaece888e75..1117b5123a6f 100644 --- a/drivers/dma/fsldma.c +++ b/drivers/dma/fsldma.c @@ -1328,8 +1328,7 @@ static int fsldma_of_remove(struct platform_device *op) #ifdef CONFIG_PM static int fsldma_suspend_late(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct fsldma_device *fdev = platform_get_drvdata(pdev); + struct fsldma_device *fdev = dev_get_drvdata(dev); struct fsldma_chan *chan; int i; @@ -1360,8 +1359,7 @@ out: static int fsldma_resume_early(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct fsldma_device *fdev = platform_get_drvdata(pdev); + struct fsldma_device *fdev = dev_get_drvdata(dev); struct fsldma_chan *chan; u32 mode; int i; diff --git a/drivers/dma/idma64.c b/drivers/dma/idma64.c index 1953e57505f4..e5c911200bdb 100644 --- a/drivers/dma/idma64.c +++ b/drivers/dma/idma64.c @@ -670,8 +670,7 @@ static int idma64_platform_remove(struct platform_device *pdev) static int idma64_pm_suspend(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct idma64_chip *chip = platform_get_drvdata(pdev); + struct idma64_chip *chip = dev_get_drvdata(dev); idma64_off(chip->idma64); return 0; @@ -679,8 +678,7 @@ static int idma64_pm_suspend(struct device *dev) static int idma64_pm_resume(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct idma64_chip *chip = platform_get_drvdata(pdev); + struct idma64_chip *chip = dev_get_drvdata(dev); idma64_on(chip->idma64); return 0; diff --git a/drivers/dma/imx-dma.c b/drivers/dma/imx-dma.c index 715b39ae5a46..75b6ff0415ee 100644 --- a/drivers/dma/imx-dma.c +++ b/drivers/dma/imx-dma.c @@ -1,19 +1,13 @@ -/* - * drivers/dma/imx-dma.c - * - * This file contains a driver for the Freescale i.MX DMA engine - * found on i.MX1/21/27 - * - * Copyright 2010 Sascha Hauer, Pengutronix - * Copyright 2012 Javier Martin, Vista Silicon - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// drivers/dma/imx-dma.c +// +// This file contains a driver for the Freescale i.MX DMA engine +// found on i.MX1/21/27 +// +// Copyright 2010 Sascha Hauer, Pengutronix +// Copyright 2012 Javier Martin, Vista Silicon + #include #include #include diff --git a/drivers/dma/imx-sdma.c b/drivers/dma/imx-sdma.c index ccd03c3cedfe..f077992635c2 100644 --- a/drivers/dma/imx-sdma.c +++ b/drivers/dma/imx-sdma.c @@ -1,21 +1,14 @@ -/* - * drivers/dma/imx-sdma.c - * - * This file contains a driver for the Freescale Smart DMA engine - * - * Copyright 2010 Sascha Hauer, Pengutronix - * - * Based on code from Freescale: - * - * Copyright 2004-2009 Freescale Semiconductor, Inc. All Rights Reserved. - * - * The code contained herein is licensed under the GNU General Public - * License. You may obtain a copy of the GNU General Public License - * Version 2 or later at the following locations: - * - * http://www.opensource.org/licenses/gpl-license.html - * http://www.gnu.org/copyleft/gpl.html - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// drivers/dma/imx-sdma.c +// +// This file contains a driver for the Freescale Smart DMA engine +// +// Copyright 2010 Sascha Hauer, Pengutronix +// +// Based on code from Freescale: +// +// Copyright 2004-2009 Freescale Semiconductor, Inc. All Rights Reserved. #include #include diff --git a/drivers/dma/ioat/init.c b/drivers/dma/ioat/init.c index 7792a9186f9c..4fa4c06c9edb 100644 --- a/drivers/dma/ioat/init.c +++ b/drivers/dma/ioat/init.c @@ -322,10 +322,10 @@ static int ioat_dma_self_test(struct ioatdma_device *ioat_dma) unsigned long tmo; unsigned long flags; - src = kzalloc(sizeof(u8) * IOAT_TEST_SIZE, GFP_KERNEL); + src = kzalloc(IOAT_TEST_SIZE, GFP_KERNEL); if (!src) return -ENOMEM; - dest = kzalloc(sizeof(u8) * IOAT_TEST_SIZE, GFP_KERNEL); + dest = kzalloc(IOAT_TEST_SIZE, GFP_KERNEL); if (!dest) { kfree(src); return -ENOMEM; diff --git a/drivers/dma/ipu/ipu_idmac.c b/drivers/dma/ipu/ipu_idmac.c index ed76044ce4b9..bbff52be4f0f 100644 --- a/drivers/dma/ipu/ipu_idmac.c +++ b/drivers/dma/ipu/ipu_idmac.c @@ -910,7 +910,8 @@ out: /* Called with ichan->chan_mutex held */ static int idmac_desc_alloc(struct idmac_channel *ichan, int n) { - struct idmac_tx_desc *desc = vmalloc(n * sizeof(struct idmac_tx_desc)); + struct idmac_tx_desc *desc = + vmalloc(array_size(n, sizeof(struct idmac_tx_desc))); struct idmac *idmac = to_idmac(ichan->dma_chan.device); if (!desc) diff --git a/drivers/dma/k3dma.c b/drivers/dma/k3dma.c index 26b67455208f..fa31cccbe04f 100644 --- a/drivers/dma/k3dma.c +++ b/drivers/dma/k3dma.c @@ -848,8 +848,8 @@ static int k3_dma_probe(struct platform_device *op) return -ENOMEM; /* init phy channel */ - d->phy = devm_kzalloc(&op->dev, - d->dma_channels * sizeof(struct k3_dma_phy), GFP_KERNEL); + d->phy = devm_kcalloc(&op->dev, + d->dma_channels, sizeof(struct k3_dma_phy), GFP_KERNEL); if (d->phy == NULL) return -ENOMEM; @@ -879,8 +879,8 @@ static int k3_dma_probe(struct platform_device *op) d->slave.copy_align = DMAENGINE_ALIGN_8_BYTES; /* init virtual channel */ - d->chans = devm_kzalloc(&op->dev, - d->dma_requests * sizeof(struct k3_dma_chan), GFP_KERNEL); + d->chans = devm_kcalloc(&op->dev, + d->dma_requests, sizeof(struct k3_dma_chan), GFP_KERNEL); if (d->chans == NULL) return -ENOMEM; diff --git a/drivers/dma/mic_x100_dma.c b/drivers/dma/mic_x100_dma.c index 94d7bd7d2880..68dd79783b54 100644 --- a/drivers/dma/mic_x100_dma.c +++ b/drivers/dma/mic_x100_dma.c @@ -385,7 +385,8 @@ static int mic_dma_alloc_desc_ring(struct mic_dma_chan *ch) if (dma_mapping_error(dev, ch->desc_ring_micpa)) goto map_error; - ch->tx_array = vzalloc(MIC_DMA_DESC_RX_SIZE * sizeof(*ch->tx_array)); + ch->tx_array = vzalloc(array_size(MIC_DMA_DESC_RX_SIZE, + sizeof(*ch->tx_array))); if (!ch->tx_array) goto tx_error; return 0; diff --git a/drivers/dma/moxart-dma.c b/drivers/dma/moxart-dma.c index e1a5c2242f6f..e04499c1f27f 100644 --- a/drivers/dma/moxart-dma.c +++ b/drivers/dma/moxart-dma.c @@ -309,7 +309,7 @@ static struct dma_async_tx_descriptor *moxart_prep_slave_sg( return NULL; } - d = kzalloc(sizeof(*d) + sg_len * sizeof(d->sg[0]), GFP_ATOMIC); + d = kzalloc(struct_size(d, sg, sg_len), GFP_ATOMIC); if (!d) return NULL; diff --git a/drivers/dma/mv_xor.c b/drivers/dma/mv_xor.c index 1993889003fd..969534c1a6c6 100644 --- a/drivers/dma/mv_xor.c +++ b/drivers/dma/mv_xor.c @@ -777,11 +777,11 @@ static int mv_chan_memcpy_self_test(struct mv_xor_chan *mv_chan) struct dmaengine_unmap_data *unmap; int err = 0; - src = kmalloc(sizeof(u8) * PAGE_SIZE, GFP_KERNEL); + src = kmalloc(PAGE_SIZE, GFP_KERNEL); if (!src) return -ENOMEM; - dest = kzalloc(sizeof(u8) * PAGE_SIZE, GFP_KERNEL); + dest = kzalloc(PAGE_SIZE, GFP_KERNEL); if (!dest) { kfree(src); return -ENOMEM; diff --git a/drivers/dma/mv_xor_v2.c b/drivers/dma/mv_xor_v2.c index 3548caa9e933..c6589ccf1b9a 100644 --- a/drivers/dma/mv_xor_v2.c +++ b/drivers/dma/mv_xor_v2.c @@ -809,8 +809,9 @@ static int mv_xor_v2_probe(struct platform_device *pdev) } /* alloc memory for the SW descriptors */ - xor_dev->sw_desq = devm_kzalloc(&pdev->dev, sizeof(*sw_desc) * - MV_XOR_V2_DESC_NUM, GFP_KERNEL); + xor_dev->sw_desq = devm_kcalloc(&pdev->dev, + MV_XOR_V2_DESC_NUM, sizeof(*sw_desc), + GFP_KERNEL); if (!xor_dev->sw_desq) { ret = -ENOMEM; goto free_hw_desq; diff --git a/drivers/dma/mxs-dma.c b/drivers/dma/mxs-dma.c index 41d167921fab..ae5182ff0128 100644 --- a/drivers/dma/mxs-dma.c +++ b/drivers/dma/mxs-dma.c @@ -1,12 +1,8 @@ -/* - * Copyright 2011 Freescale Semiconductor, Inc. All Rights Reserved. - * - * Refer to drivers/dma/imx-sdma.c - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ +// SPDX-License-Identifier: GPL-2.0 +// +// Copyright 2011 Freescale Semiconductor, Inc. All Rights Reserved. +// +// Refer to drivers/dma/imx-sdma.c #include #include diff --git a/drivers/dma/nbpfaxi.c b/drivers/dma/nbpfaxi.c index 50559338239b..2f9974ddfbb2 100644 --- a/drivers/dma/nbpfaxi.c +++ b/drivers/dma/nbpfaxi.c @@ -1305,8 +1305,8 @@ static int nbpf_probe(struct platform_device *pdev) cfg = of_device_get_match_data(dev); num_channels = cfg->num_channels; - nbpf = devm_kzalloc(dev, sizeof(*nbpf) + num_channels * - sizeof(nbpf->chan[0]), GFP_KERNEL); + nbpf = devm_kzalloc(dev, struct_size(nbpf, chan, num_channels), + GFP_KERNEL); if (!nbpf) return -ENOMEM; diff --git a/drivers/dma/pl330.c b/drivers/dma/pl330.c index de1fd59fe136..defcdde4d358 100644 --- a/drivers/dma/pl330.c +++ b/drivers/dma/pl330.c @@ -27,6 +27,7 @@ #include #include #include +#include #include "dmaengine.h" #define PL330_MAX_CHAN 8 @@ -1094,51 +1095,96 @@ static inline int _ldst_memtomem(unsigned dry_run, u8 buf[], return off; } -static inline int _ldst_devtomem(struct pl330_dmac *pl330, unsigned dry_run, - u8 buf[], const struct _xfer_spec *pxs, - int cyc) +static u32 _emit_load(unsigned int dry_run, u8 buf[], + enum pl330_cond cond, enum dma_transfer_direction direction, + u8 peri) { int off = 0; - enum pl330_cond cond; - if (pl330->quirks & PL330_QUIRK_BROKEN_NO_FLUSHP) - cond = BURST; - else - cond = SINGLE; + switch (direction) { + case DMA_MEM_TO_MEM: + /* fall through */ + case DMA_MEM_TO_DEV: + off += _emit_LD(dry_run, &buf[off], cond); + break; - while (cyc--) { - off += _emit_WFP(dry_run, &buf[off], cond, pxs->desc->peri); - off += _emit_LDP(dry_run, &buf[off], cond, pxs->desc->peri); - off += _emit_ST(dry_run, &buf[off], ALWAYS); + case DMA_DEV_TO_MEM: + if (cond == ALWAYS) { + off += _emit_LDP(dry_run, &buf[off], SINGLE, + peri); + off += _emit_LDP(dry_run, &buf[off], BURST, + peri); + } else { + off += _emit_LDP(dry_run, &buf[off], cond, + peri); + } + break; - if (!(pl330->quirks & PL330_QUIRK_BROKEN_NO_FLUSHP)) - off += _emit_FLUSHP(dry_run, &buf[off], - pxs->desc->peri); + default: + /* this code should be unreachable */ + WARN_ON(1); + break; } return off; } -static inline int _ldst_memtodev(struct pl330_dmac *pl330, +static inline u32 _emit_store(unsigned int dry_run, u8 buf[], + enum pl330_cond cond, enum dma_transfer_direction direction, + u8 peri) +{ + int off = 0; + + switch (direction) { + case DMA_MEM_TO_MEM: + /* fall through */ + case DMA_DEV_TO_MEM: + off += _emit_ST(dry_run, &buf[off], cond); + break; + + case DMA_MEM_TO_DEV: + if (cond == ALWAYS) { + off += _emit_STP(dry_run, &buf[off], SINGLE, + peri); + off += _emit_STP(dry_run, &buf[off], BURST, + peri); + } else { + off += _emit_STP(dry_run, &buf[off], cond, + peri); + } + break; + + default: + /* this code should be unreachable */ + WARN_ON(1); + break; + } + + return off; +} + +static inline int _ldst_peripheral(struct pl330_dmac *pl330, unsigned dry_run, u8 buf[], - const struct _xfer_spec *pxs, int cyc) + const struct _xfer_spec *pxs, int cyc, + enum pl330_cond cond) { int off = 0; - enum pl330_cond cond; if (pl330->quirks & PL330_QUIRK_BROKEN_NO_FLUSHP) cond = BURST; - else - cond = SINGLE; + /* + * do FLUSHP at beginning to clear any stale dma requests before the + * first WFP. + */ + if (!(pl330->quirks & PL330_QUIRK_BROKEN_NO_FLUSHP)) + off += _emit_FLUSHP(dry_run, &buf[off], pxs->desc->peri); while (cyc--) { off += _emit_WFP(dry_run, &buf[off], cond, pxs->desc->peri); - off += _emit_LD(dry_run, &buf[off], ALWAYS); - off += _emit_STP(dry_run, &buf[off], cond, pxs->desc->peri); - - if (!(pl330->quirks & PL330_QUIRK_BROKEN_NO_FLUSHP)) - off += _emit_FLUSHP(dry_run, &buf[off], - pxs->desc->peri); + off += _emit_load(dry_run, &buf[off], cond, pxs->desc->rqtype, + pxs->desc->peri); + off += _emit_store(dry_run, &buf[off], cond, pxs->desc->rqtype, + pxs->desc->peri); } return off; @@ -1148,19 +1194,65 @@ static int _bursts(struct pl330_dmac *pl330, unsigned dry_run, u8 buf[], const struct _xfer_spec *pxs, int cyc) { int off = 0; + enum pl330_cond cond = BRST_LEN(pxs->ccr) > 1 ? BURST : SINGLE; switch (pxs->desc->rqtype) { case DMA_MEM_TO_DEV: - off += _ldst_memtodev(pl330, dry_run, &buf[off], pxs, cyc); - break; + /* fall through */ case DMA_DEV_TO_MEM: - off += _ldst_devtomem(pl330, dry_run, &buf[off], pxs, cyc); + off += _ldst_peripheral(pl330, dry_run, &buf[off], pxs, cyc, + cond); break; + case DMA_MEM_TO_MEM: off += _ldst_memtomem(dry_run, &buf[off], pxs, cyc); break; + + default: + /* this code should be unreachable */ + WARN_ON(1); + break; + } + + return off; +} + +/* + * transfer dregs with single transfers to peripheral, or a reduced size burst + * for mem-to-mem. + */ +static int _dregs(struct pl330_dmac *pl330, unsigned int dry_run, u8 buf[], + const struct _xfer_spec *pxs, int transfer_length) +{ + int off = 0; + int dregs_ccr; + + if (transfer_length == 0) + return off; + + switch (pxs->desc->rqtype) { + case DMA_MEM_TO_DEV: + /* fall through */ + case DMA_DEV_TO_MEM: + off += _ldst_peripheral(pl330, dry_run, &buf[off], pxs, + transfer_length, SINGLE); + break; + + case DMA_MEM_TO_MEM: + dregs_ccr = pxs->ccr; + dregs_ccr &= ~((0xf << CC_SRCBRSTLEN_SHFT) | + (0xf << CC_DSTBRSTLEN_SHFT)); + dregs_ccr |= (((transfer_length - 1) & 0xf) << + CC_SRCBRSTLEN_SHFT); + dregs_ccr |= (((transfer_length - 1) & 0xf) << + CC_DSTBRSTLEN_SHFT); + off += _emit_MOV(dry_run, &buf[off], CCR, dregs_ccr); + off += _ldst_memtomem(dry_run, &buf[off], pxs, 1); + break; + default: - off += 0x40000000; /* Scare off the Client */ + /* this code should be unreachable */ + WARN_ON(1); break; } @@ -1256,6 +1348,8 @@ static inline int _setup_loops(struct pl330_dmac *pl330, struct pl330_xfer *x = &pxs->desc->px; u32 ccr = pxs->ccr; unsigned long c, bursts = BYTE_TO_BURST(x->bytes, ccr); + int num_dregs = (x->bytes - BURST_TO_BYTE(bursts, ccr)) / + BRST_SIZE(ccr); int off = 0; while (bursts) { @@ -1263,6 +1357,7 @@ static inline int _setup_loops(struct pl330_dmac *pl330, off += _loop(pl330, dry_run, &buf[off], &c, pxs); bursts -= c; } + off += _dregs(pl330, dry_run, &buf[off], pxs, num_dregs); return off; } @@ -1294,7 +1389,6 @@ static int _setup_req(struct pl330_dmac *pl330, unsigned dry_run, struct _xfer_spec *pxs) { struct _pl330_req *req = &thrd->req[index]; - struct pl330_xfer *x; u8 *buf = req->mc_cpu; int off = 0; @@ -1303,11 +1397,6 @@ static int _setup_req(struct pl330_dmac *pl330, unsigned dry_run, /* DMAMOV CCR, ccr */ off += _emit_MOV(dry_run, &buf[off], CCR, pxs->ccr); - x = &pxs->desc->px; - /* Error if xfer length is not aligned at burst size */ - if (x->bytes % (BRST_SIZE(pxs->ccr) * BRST_LEN(pxs->ccr))) - return -EINVAL; - off += _setup_xfer(pl330, dry_run, &buf[off], pxs); /* DMASEV peripheral/event */ @@ -1365,6 +1454,20 @@ static int pl330_submit_req(struct pl330_thread *thrd, u32 ccr; int ret = 0; + switch (desc->rqtype) { + case DMA_MEM_TO_DEV: + break; + + case DMA_DEV_TO_MEM: + break; + + case DMA_MEM_TO_MEM: + break; + + default: + return -ENOTSUPP; + } + if (pl330->state == DYING || pl330->dmac_tbd.reset_chan & (1 << thrd->id)) { dev_info(thrd->dmac->ddma.dev, "%s:%d\n", @@ -1763,7 +1866,7 @@ static int dmac_alloc_threads(struct pl330_dmac *pl330) int i; /* Allocate 1 Manager and 'chans' Channel threads */ - pl330->channels = kzalloc((1 + chans) * sizeof(*thrd), + pl330->channels = kcalloc(1 + chans, sizeof(*thrd), GFP_KERNEL); if (!pl330->channels) return -ENOMEM; @@ -2106,6 +2209,18 @@ static bool pl330_prep_slave_fifo(struct dma_pl330_chan *pch, return true; } +static int fixup_burst_len(int max_burst_len, int quirks) +{ + if (quirks & PL330_QUIRK_BROKEN_NO_FLUSHP) + return 1; + else if (max_burst_len > PL330_MAX_BURST) + return PL330_MAX_BURST; + else if (max_burst_len < 1) + return 1; + else + return max_burst_len; +} + static int pl330_config(struct dma_chan *chan, struct dma_slave_config *slave_config) { @@ -2117,15 +2232,15 @@ static int pl330_config(struct dma_chan *chan, pch->fifo_addr = slave_config->dst_addr; if (slave_config->dst_addr_width) pch->burst_sz = __ffs(slave_config->dst_addr_width); - if (slave_config->dst_maxburst) - pch->burst_len = slave_config->dst_maxburst; + pch->burst_len = fixup_burst_len(slave_config->dst_maxburst, + pch->dmac->quirks); } else if (slave_config->direction == DMA_DEV_TO_MEM) { if (slave_config->src_addr) pch->fifo_addr = slave_config->src_addr; if (slave_config->src_addr_width) pch->burst_sz = __ffs(slave_config->src_addr_width); - if (slave_config->src_maxburst) - pch->burst_len = slave_config->src_maxburst; + pch->burst_len = fixup_burst_len(slave_config->src_maxburst, + pch->dmac->quirks); } return 0; @@ -2519,14 +2634,8 @@ static inline int get_burst_len(struct dma_pl330_desc *desc, size_t len) burst_len >>= desc->rqcfg.brst_size; /* src/dst_burst_len can't be more than 16 */ - if (burst_len > 16) - burst_len = 16; - - while (burst_len > 1) { - if (!(len % (burst_len << desc->rqcfg.brst_size))) - break; - burst_len--; - } + if (burst_len > PL330_MAX_BURST) + burst_len = PL330_MAX_BURST; return burst_len; } @@ -2598,7 +2707,7 @@ static struct dma_async_tx_descriptor *pl330_prep_dma_cyclic( desc->rqtype = direction; desc->rqcfg.brst_size = pch->burst_sz; - desc->rqcfg.brst_len = 1; + desc->rqcfg.brst_len = pch->burst_len; desc->bytes_requested = period_len; fill_px(&desc->px, dst, src, period_len); @@ -2743,7 +2852,7 @@ pl330_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl, } desc->rqcfg.brst_size = pch->burst_sz; - desc->rqcfg.brst_len = 1; + desc->rqcfg.brst_len = pch->burst_len; desc->rqtype = direction; desc->bytes_requested = sg_dma_len(sg); } @@ -2881,7 +2990,7 @@ pl330_probe(struct amba_device *adev, const struct amba_id *id) pl330->num_peripherals = num_chan; - pl330->peripherals = kzalloc(num_chan * sizeof(*pch), GFP_KERNEL); + pl330->peripherals = kcalloc(num_chan, sizeof(*pch), GFP_KERNEL); if (!pl330->peripherals) { ret = -ENOMEM; goto probe_err2; diff --git a/drivers/dma/qcom/bam_dma.c b/drivers/dma/qcom/bam_dma.c index 4a828c18099a..1617715aa6e0 100644 --- a/drivers/dma/qcom/bam_dma.c +++ b/drivers/dma/qcom/bam_dma.c @@ -451,6 +451,7 @@ static void bam_reset_channel(struct bam_chan *bchan) /** * bam_chan_init_hw - Initialize channel hardware * @bchan: bam channel + * @dir: DMA transfer direction * * This function resets and initializes the BAM channel */ @@ -673,7 +674,7 @@ static struct dma_async_tx_descriptor *bam_prep_slave_sg(struct dma_chan *chan, remainder = 0; } - async_desc->length += desc->size; + async_desc->length += le16_to_cpu(desc->size); desc++; } while (remainder > 0); } @@ -687,7 +688,7 @@ err_out: /** * bam_dma_terminate_all - terminate all transactions on a channel - * @bchan: bam dma channel + * @chan: bam dma channel * * Dequeues and frees all transactions * No callbacks are done @@ -918,7 +919,8 @@ static enum dma_status bam_tx_status(struct dma_chan *chan, dma_cookie_t cookie, continue; for (i = 0; i < async_desc->num_desc; i++) - residue += async_desc->curr_desc[i].size; + residue += le16_to_cpu( + async_desc->curr_desc[i].size); } } @@ -958,7 +960,7 @@ static void bam_apply_new_config(struct bam_chan *bchan, /** * bam_start_dma - start next transaction - * @bchan - bam dma channel + * @bchan: bam dma channel */ static void bam_start_dma(struct bam_chan *bchan) { diff --git a/drivers/dma/qcom/hidma.c b/drivers/dma/qcom/hidma.c index 963cc5228d05..43d4b00b8138 100644 --- a/drivers/dma/qcom/hidma.c +++ b/drivers/dma/qcom/hidma.c @@ -616,8 +616,7 @@ static irqreturn_t hidma_chirq_handler_msi(int chirq, void *arg) static ssize_t hidma_show_values(struct device *dev, struct device_attribute *attr, char *buf) { - struct platform_device *pdev = to_platform_device(dev); - struct hidma_dev *mdev = platform_get_drvdata(pdev); + struct hidma_dev *mdev = dev_get_drvdata(dev); buf[0] = 0; diff --git a/drivers/dma/qcom/hidma_mgmt.c b/drivers/dma/qcom/hidma_mgmt.c index 000c7019ca7d..d64edeb6771a 100644 --- a/drivers/dma/qcom/hidma_mgmt.c +++ b/drivers/dma/qcom/hidma_mgmt.c @@ -398,7 +398,7 @@ static int __init hidma_mgmt_of_populate_channels(struct device_node *np) } of_node_get(child); new_pdev->dev.of_node = child; - of_dma_configure(&new_pdev->dev, child); + of_dma_configure(&new_pdev->dev, child, true); /* * It is assumed that calling of_msi_configure is safe on * platforms with or without MSI support. diff --git a/drivers/dma/qcom/hidma_mgmt_sys.c b/drivers/dma/qcom/hidma_mgmt_sys.c index d61f1068a34b..cbb89eafd844 100644 --- a/drivers/dma/qcom/hidma_mgmt_sys.c +++ b/drivers/dma/qcom/hidma_mgmt_sys.c @@ -107,8 +107,7 @@ static struct hidma_mgmt_fileinfo hidma_mgmt_files[] = { static ssize_t show_values(struct device *dev, struct device_attribute *attr, char *buf) { - struct platform_device *pdev = to_platform_device(dev); - struct hidma_mgmt_dev *mdev = platform_get_drvdata(pdev); + struct hidma_mgmt_dev *mdev = dev_get_drvdata(dev); unsigned int i; buf[0] = 0; @@ -125,8 +124,7 @@ static ssize_t show_values(struct device *dev, struct device_attribute *attr, static ssize_t set_values(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) { - struct platform_device *pdev = to_platform_device(dev); - struct hidma_mgmt_dev *mdev = platform_get_drvdata(pdev); + struct hidma_mgmt_dev *mdev = dev_get_drvdata(dev); unsigned long tmp; unsigned int i; int rc; diff --git a/drivers/dma/s3c24xx-dma.c b/drivers/dma/s3c24xx-dma.c index cd92d696bcf9..7056fe7513b4 100644 --- a/drivers/dma/s3c24xx-dma.c +++ b/drivers/dma/s3c24xx-dma.c @@ -1223,9 +1223,9 @@ static int s3c24xx_dma_probe(struct platform_device *pdev) if (IS_ERR(s3cdma->base)) return PTR_ERR(s3cdma->base); - s3cdma->phy_chans = devm_kzalloc(&pdev->dev, - sizeof(struct s3c24xx_dma_phy) * - pdata->num_phy_channels, + s3cdma->phy_chans = devm_kcalloc(&pdev->dev, + pdata->num_phy_channels, + sizeof(struct s3c24xx_dma_phy), GFP_KERNEL); if (!s3cdma->phy_chans) return -ENOMEM; diff --git a/drivers/dma/sa11x0-dma.c b/drivers/dma/sa11x0-dma.c index c7a89c22890e..b31d07c7d93c 100644 --- a/drivers/dma/sa11x0-dma.c +++ b/drivers/dma/sa11x0-dma.c @@ -557,7 +557,7 @@ static struct dma_async_tx_descriptor *sa11x0_dma_prep_slave_sg( } } - txd = kzalloc(sizeof(*txd) + j * sizeof(txd->sg[0]), GFP_ATOMIC); + txd = kzalloc(struct_size(txd, sg, j), GFP_ATOMIC); if (!txd) { dev_dbg(chan->device->dev, "vchan %p: kzalloc failed\n", &c->vc); return NULL; @@ -627,7 +627,7 @@ static struct dma_async_tx_descriptor *sa11x0_dma_prep_dma_cyclic( if (sglen == 0) return NULL; - txd = kzalloc(sizeof(*txd) + sglen * sizeof(txd->sg[0]), GFP_ATOMIC); + txd = kzalloc(struct_size(txd, sg, sglen), GFP_ATOMIC); if (!txd) { dev_dbg(chan->device->dev, "vchan %p: kzalloc failed\n", &c->vc); return NULL; diff --git a/drivers/dma/sh/shdma-base.c b/drivers/dma/sh/shdma-base.c index 12fa48e380cf..6b5626e299b2 100644 --- a/drivers/dma/sh/shdma-base.c +++ b/drivers/dma/sh/shdma-base.c @@ -1045,8 +1045,9 @@ EXPORT_SYMBOL(shdma_cleanup); static int __init shdma_enter(void) { - shdma_slave_used = kzalloc(DIV_ROUND_UP(slave_num, BITS_PER_LONG) * - sizeof(long), GFP_KERNEL); + shdma_slave_used = kcalloc(DIV_ROUND_UP(slave_num, BITS_PER_LONG), + sizeof(long), + GFP_KERNEL); if (!shdma_slave_used) return -ENOMEM; return 0; diff --git a/drivers/dma/sh/shdmac.c b/drivers/dma/sh/shdmac.c index c94ffab0d25c..04a74e0a95b7 100644 --- a/drivers/dma/sh/shdmac.c +++ b/drivers/dma/sh/shdmac.c @@ -443,7 +443,6 @@ static bool sh_dmae_reset(struct sh_dmae_device *shdev) return ret; } -#if defined(CONFIG_CPU_SH4) || defined(CONFIG_ARCH_SHMOBILE) static irqreturn_t sh_dmae_err(int irq, void *data) { struct sh_dmae_device *shdev = data; @@ -454,7 +453,6 @@ static irqreturn_t sh_dmae_err(int irq, void *data) sh_dmae_reset(shdev); return IRQ_HANDLED; } -#endif static bool sh_dmae_desc_completed(struct shdma_chan *schan, struct shdma_desc *sdesc) @@ -686,11 +684,8 @@ static int sh_dmae_probe(struct platform_device *pdev) const struct sh_dmae_pdata *pdata; unsigned long chan_flag[SH_DMAE_MAX_CHANNELS] = {}; int chan_irq[SH_DMAE_MAX_CHANNELS]; -#if defined(CONFIG_CPU_SH4) || defined(CONFIG_ARCH_SHMOBILE) unsigned long irqflags = 0; - int errirq; -#endif - int err, i, irq_cnt = 0, irqres = 0, irq_cap = 0; + int err, errirq, i, irq_cnt = 0, irqres = 0, irq_cap = 0; struct sh_dmae_device *shdev; struct dma_device *dma_dev; struct resource *chan, *dmars, *errirq_res, *chanirq_res; @@ -792,33 +787,32 @@ static int sh_dmae_probe(struct platform_device *pdev) if (err) goto rst_err; -#if defined(CONFIG_CPU_SH4) || defined(CONFIG_ARCH_SHMOBILE) - chanirq_res = platform_get_resource(pdev, IORESOURCE_IRQ, 1); + if (IS_ENABLED(CONFIG_CPU_SH4) || IS_ENABLED(CONFIG_ARCH_RENESAS)) { + chanirq_res = platform_get_resource(pdev, IORESOURCE_IRQ, 1); - if (!chanirq_res) - chanirq_res = errirq_res; - else - irqres++; + if (!chanirq_res) + chanirq_res = errirq_res; + else + irqres++; - if (chanirq_res == errirq_res || - (errirq_res->flags & IORESOURCE_BITS) == IORESOURCE_IRQ_SHAREABLE) - irqflags = IRQF_SHARED; + if (chanirq_res == errirq_res || + (errirq_res->flags & IORESOURCE_BITS) == IORESOURCE_IRQ_SHAREABLE) + irqflags = IRQF_SHARED; - errirq = errirq_res->start; + errirq = errirq_res->start; - err = devm_request_irq(&pdev->dev, errirq, sh_dmae_err, irqflags, - "DMAC Address Error", shdev); - if (err) { - dev_err(&pdev->dev, - "DMA failed requesting irq #%d, error %d\n", - errirq, err); - goto eirq_err; + err = devm_request_irq(&pdev->dev, errirq, sh_dmae_err, + irqflags, "DMAC Address Error", shdev); + if (err) { + dev_err(&pdev->dev, + "DMA failed requesting irq #%d, error %d\n", + errirq, err); + goto eirq_err; + } + } else { + chanirq_res = errirq_res; } -#else - chanirq_res = errirq_res; -#endif /* CONFIG_CPU_SH4 || CONFIG_ARCH_SHMOBILE */ - if (chanirq_res->start == chanirq_res->end && !platform_get_resource(pdev, IORESOURCE_IRQ, 1)) { /* Special case - all multiplexed */ @@ -884,9 +878,7 @@ edmadevreg: chan_probe_err: sh_dmae_chan_remove(shdev); -#if defined(CONFIG_CPU_SH4) || defined(CONFIG_ARCH_SHMOBILE) eirq_err: -#endif rst_err: spin_lock_irq(&sh_dmae_lock); list_del_rcu(&shdev->node); diff --git a/drivers/dma/sh/usb-dmac.c b/drivers/dma/sh/usb-dmac.c index 31a145154e9f..1bb1a8e09025 100644 --- a/drivers/dma/sh/usb-dmac.c +++ b/drivers/dma/sh/usb-dmac.c @@ -269,7 +269,7 @@ static int usb_dmac_desc_alloc(struct usb_dmac_chan *chan, unsigned int sg_len, struct usb_dmac_desc *desc; unsigned long flags; - desc = kzalloc(sizeof(*desc) + sg_len * sizeof(desc->sg[0]), gfp); + desc = kzalloc(struct_size(desc, sg, sg_len), gfp); if (!desc) return -ENOMEM; diff --git a/drivers/dma/sprd-dma.c b/drivers/dma/sprd-dma.c index b106e8a60af6..55df0d41355b 100644 --- a/drivers/dma/sprd-dma.c +++ b/drivers/dma/sprd-dma.c @@ -6,6 +6,7 @@ #include #include +#include #include #include #include @@ -116,57 +117,21 @@ #define SPRD_DMA_SRC_TRSF_STEP_OFFSET 0 #define SPRD_DMA_TRSF_STEP_MASK GENMASK(15, 0) +/* define the DMA transfer step type */ +#define SPRD_DMA_NONE_STEP 0 +#define SPRD_DMA_BYTE_STEP 1 +#define SPRD_DMA_SHORT_STEP 2 +#define SPRD_DMA_WORD_STEP 4 +#define SPRD_DMA_DWORD_STEP 8 + #define SPRD_DMA_SOFTWARE_UID 0 -/* - * enum sprd_dma_req_mode: define the DMA request mode - * @SPRD_DMA_FRAG_REQ: fragment request mode - * @SPRD_DMA_BLK_REQ: block request mode - * @SPRD_DMA_TRANS_REQ: transaction request mode - * @SPRD_DMA_LIST_REQ: link-list request mode - * - * We have 4 types request mode: fragment mode, block mode, transaction mode - * and linklist mode. One transaction can contain several blocks, one block can - * contain several fragments. Link-list mode means we can save several DMA - * configuration into one reserved memory, then DMA can fetch each DMA - * configuration automatically to start transfer. - */ -enum sprd_dma_req_mode { - SPRD_DMA_FRAG_REQ, - SPRD_DMA_BLK_REQ, - SPRD_DMA_TRANS_REQ, - SPRD_DMA_LIST_REQ, -}; - -/* - * enum sprd_dma_int_type: define the DMA interrupt type - * @SPRD_DMA_NO_INT: do not need generate DMA interrupts. - * @SPRD_DMA_FRAG_INT: fragment done interrupt when one fragment request - * is done. - * @SPRD_DMA_BLK_INT: block done interrupt when one block request is done. - * @SPRD_DMA_BLK_FRAG_INT: block and fragment interrupt when one fragment - * or one block request is done. - * @SPRD_DMA_TRANS_INT: tansaction done interrupt when one transaction - * request is done. - * @SPRD_DMA_TRANS_FRAG_INT: transaction and fragment interrupt when one - * transaction request or fragment request is done. - * @SPRD_DMA_TRANS_BLK_INT: transaction and block interrupt when one - * transaction request or block request is done. - * @SPRD_DMA_LIST_INT: link-list done interrupt when one link-list request - * is done. - * @SPRD_DMA_CFGERR_INT: configure error interrupt when configuration is - * incorrect. - */ -enum sprd_dma_int_type { - SPRD_DMA_NO_INT, - SPRD_DMA_FRAG_INT, - SPRD_DMA_BLK_INT, - SPRD_DMA_BLK_FRAG_INT, - SPRD_DMA_TRANS_INT, - SPRD_DMA_TRANS_FRAG_INT, - SPRD_DMA_TRANS_BLK_INT, - SPRD_DMA_LIST_INT, - SPRD_DMA_CFGERR_INT, +/* dma data width values */ +enum sprd_dma_datawidth { + SPRD_DMA_DATAWIDTH_1_BYTE, + SPRD_DMA_DATAWIDTH_2_BYTES, + SPRD_DMA_DATAWIDTH_4_BYTES, + SPRD_DMA_DATAWIDTH_8_BYTES, }; /* dma channel hardware configuration */ @@ -199,6 +164,7 @@ struct sprd_dma_desc { struct sprd_dma_chn { struct virt_dma_chan vc; void __iomem *chn_base; + struct dma_slave_config slave_cfg; u32 chn_num; u32 dev_id; struct sprd_dma_desc *cur_desc; @@ -587,52 +553,97 @@ static void sprd_dma_issue_pending(struct dma_chan *chan) spin_unlock_irqrestore(&schan->vc.lock, flags); } -static int sprd_dma_config(struct dma_chan *chan, struct sprd_dma_desc *sdesc, - dma_addr_t dest, dma_addr_t src, size_t len) +static int sprd_dma_get_datawidth(enum dma_slave_buswidth buswidth) +{ + switch (buswidth) { + case DMA_SLAVE_BUSWIDTH_1_BYTE: + case DMA_SLAVE_BUSWIDTH_2_BYTES: + case DMA_SLAVE_BUSWIDTH_4_BYTES: + case DMA_SLAVE_BUSWIDTH_8_BYTES: + return ffs(buswidth) - 1; + + default: + return -EINVAL; + } +} + +static int sprd_dma_get_step(enum dma_slave_buswidth buswidth) +{ + switch (buswidth) { + case DMA_SLAVE_BUSWIDTH_1_BYTE: + case DMA_SLAVE_BUSWIDTH_2_BYTES: + case DMA_SLAVE_BUSWIDTH_4_BYTES: + case DMA_SLAVE_BUSWIDTH_8_BYTES: + return buswidth; + + default: + return -EINVAL; + } +} + +static int sprd_dma_fill_desc(struct dma_chan *chan, + struct sprd_dma_desc *sdesc, + dma_addr_t src, dma_addr_t dst, u32 len, + enum dma_transfer_direction dir, + unsigned long flags, + struct dma_slave_config *slave_cfg) { struct sprd_dma_dev *sdev = to_sprd_dma_dev(chan); + struct sprd_dma_chn *schan = to_sprd_dma_chan(chan); struct sprd_dma_chn_hw *hw = &sdesc->chn_hw; - u32 datawidth, src_step, des_step, fragment_len; - u32 block_len, req_mode, irq_mode, transcation_len; - u32 fix_mode = 0, fix_en = 0; - - if (IS_ALIGNED(len, 4)) { - datawidth = 2; - src_step = 4; - des_step = 4; - } else if (IS_ALIGNED(len, 2)) { - datawidth = 1; - src_step = 2; - des_step = 2; + u32 req_mode = (flags >> SPRD_DMA_REQ_SHIFT) & SPRD_DMA_REQ_MODE_MASK; + u32 int_mode = flags & SPRD_DMA_INT_MASK; + int src_datawidth, dst_datawidth, src_step, dst_step; + u32 temp, fix_mode = 0, fix_en = 0; + + if (dir == DMA_MEM_TO_DEV) { + src_step = sprd_dma_get_step(slave_cfg->src_addr_width); + if (src_step < 0) { + dev_err(sdev->dma_dev.dev, "invalid source step\n"); + return src_step; + } + dst_step = SPRD_DMA_NONE_STEP; } else { - datawidth = 0; - src_step = 1; - des_step = 1; + dst_step = sprd_dma_get_step(slave_cfg->dst_addr_width); + if (dst_step < 0) { + dev_err(sdev->dma_dev.dev, "invalid destination step\n"); + return dst_step; + } + src_step = SPRD_DMA_NONE_STEP; } - fragment_len = SPRD_DMA_MEMCPY_MIN_SIZE; - if (len <= SPRD_DMA_BLK_LEN_MASK) { - block_len = len; - transcation_len = 0; - req_mode = SPRD_DMA_BLK_REQ; - irq_mode = SPRD_DMA_BLK_INT; - } else { - block_len = SPRD_DMA_MEMCPY_MIN_SIZE; - transcation_len = len; - req_mode = SPRD_DMA_TRANS_REQ; - irq_mode = SPRD_DMA_TRANS_INT; + src_datawidth = sprd_dma_get_datawidth(slave_cfg->src_addr_width); + if (src_datawidth < 0) { + dev_err(sdev->dma_dev.dev, "invalid source datawidth\n"); + return src_datawidth; } + dst_datawidth = sprd_dma_get_datawidth(slave_cfg->dst_addr_width); + if (dst_datawidth < 0) { + dev_err(sdev->dma_dev.dev, "invalid destination datawidth\n"); + return dst_datawidth; + } + + if (slave_cfg->slave_id) + schan->dev_id = slave_cfg->slave_id; + hw->cfg = SPRD_DMA_DONOT_WAIT_BDONE << SPRD_DMA_WAIT_BDONE_OFFSET; - hw->wrap_ptr = (u32)((src >> SPRD_DMA_HIGH_ADDR_OFFSET) & - SPRD_DMA_HIGH_ADDR_MASK); - hw->wrap_to = (u32)((dest >> SPRD_DMA_HIGH_ADDR_OFFSET) & - SPRD_DMA_HIGH_ADDR_MASK); - hw->src_addr = (u32)(src & SPRD_DMA_LOW_ADDR_MASK); - hw->des_addr = (u32)(dest & SPRD_DMA_LOW_ADDR_MASK); + /* + * wrap_ptr and wrap_to will save the high 4 bits source address and + * destination address. + */ + hw->wrap_ptr = (src >> SPRD_DMA_HIGH_ADDR_OFFSET) & SPRD_DMA_HIGH_ADDR_MASK; + hw->wrap_to = (dst >> SPRD_DMA_HIGH_ADDR_OFFSET) & SPRD_DMA_HIGH_ADDR_MASK; + hw->src_addr = src & SPRD_DMA_LOW_ADDR_MASK; + hw->des_addr = dst & SPRD_DMA_LOW_ADDR_MASK; - if ((src_step != 0 && des_step != 0) || (src_step | des_step) == 0) { + /* + * If the src step and dst step both are 0 or both are not 0, that means + * we can not enable the fix mode. If one is 0 and another one is not, + * we can enable the fix mode. + */ + if ((src_step != 0 && dst_step != 0) || (src_step | dst_step) == 0) { fix_en = 0; } else { fix_en = 1; @@ -642,87 +653,119 @@ static int sprd_dma_config(struct dma_chan *chan, struct sprd_dma_desc *sdesc, fix_mode = 0; } - hw->frg_len = datawidth << SPRD_DMA_SRC_DATAWIDTH_OFFSET | - datawidth << SPRD_DMA_DES_DATAWIDTH_OFFSET | - req_mode << SPRD_DMA_REQ_MODE_OFFSET | - fix_mode << SPRD_DMA_FIX_SEL_OFFSET | - fix_en << SPRD_DMA_FIX_EN_OFFSET | - (fragment_len & SPRD_DMA_FRG_LEN_MASK); - hw->blk_len = block_len & SPRD_DMA_BLK_LEN_MASK; - - hw->intc = SPRD_DMA_CFG_ERR_INT_EN; - - switch (irq_mode) { - case SPRD_DMA_NO_INT: - break; - - case SPRD_DMA_FRAG_INT: - hw->intc |= SPRD_DMA_FRAG_INT_EN; - break; + hw->intc = int_mode | SPRD_DMA_CFG_ERR_INT_EN; - case SPRD_DMA_BLK_INT: - hw->intc |= SPRD_DMA_BLK_INT_EN; - break; + temp = src_datawidth << SPRD_DMA_SRC_DATAWIDTH_OFFSET; + temp |= dst_datawidth << SPRD_DMA_DES_DATAWIDTH_OFFSET; + temp |= req_mode << SPRD_DMA_REQ_MODE_OFFSET; + temp |= fix_mode << SPRD_DMA_FIX_SEL_OFFSET; + temp |= fix_en << SPRD_DMA_FIX_EN_OFFSET; + temp |= slave_cfg->src_maxburst & SPRD_DMA_FRG_LEN_MASK; + hw->frg_len = temp; - case SPRD_DMA_BLK_FRAG_INT: - hw->intc |= SPRD_DMA_BLK_INT_EN | SPRD_DMA_FRAG_INT_EN; - break; + hw->blk_len = len & SPRD_DMA_BLK_LEN_MASK; + hw->trsc_len = len & SPRD_DMA_TRSC_LEN_MASK; - case SPRD_DMA_TRANS_INT: - hw->intc |= SPRD_DMA_TRANS_INT_EN; - break; + temp = (dst_step & SPRD_DMA_TRSF_STEP_MASK) << SPRD_DMA_DEST_TRSF_STEP_OFFSET; + temp |= (src_step & SPRD_DMA_TRSF_STEP_MASK) << SPRD_DMA_SRC_TRSF_STEP_OFFSET; + hw->trsf_step = temp; - case SPRD_DMA_TRANS_FRAG_INT: - hw->intc |= SPRD_DMA_TRANS_INT_EN | SPRD_DMA_FRAG_INT_EN; - break; + hw->frg_step = 0; + hw->src_blk_step = 0; + hw->des_blk_step = 0; + return 0; +} - case SPRD_DMA_TRANS_BLK_INT: - hw->intc |= SPRD_DMA_TRANS_INT_EN | SPRD_DMA_BLK_INT_EN; - break; +static struct dma_async_tx_descriptor * +sprd_dma_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dest, dma_addr_t src, + size_t len, unsigned long flags) +{ + struct sprd_dma_chn *schan = to_sprd_dma_chan(chan); + struct sprd_dma_desc *sdesc; + struct sprd_dma_chn_hw *hw; + enum sprd_dma_datawidth datawidth; + u32 step, temp; - case SPRD_DMA_LIST_INT: - hw->intc |= SPRD_DMA_LIST_INT_EN; - break; + sdesc = kzalloc(sizeof(*sdesc), GFP_NOWAIT); + if (!sdesc) + return NULL; - case SPRD_DMA_CFGERR_INT: - hw->intc |= SPRD_DMA_CFG_ERR_INT_EN; - break; + hw = &sdesc->chn_hw; - default: - dev_err(sdev->dma_dev.dev, "invalid irq mode\n"); - return -EINVAL; + hw->cfg = SPRD_DMA_DONOT_WAIT_BDONE << SPRD_DMA_WAIT_BDONE_OFFSET; + hw->intc = SPRD_DMA_TRANS_INT | SPRD_DMA_CFG_ERR_INT_EN; + hw->src_addr = src & SPRD_DMA_LOW_ADDR_MASK; + hw->des_addr = dest & SPRD_DMA_LOW_ADDR_MASK; + hw->wrap_ptr = (src >> SPRD_DMA_HIGH_ADDR_OFFSET) & + SPRD_DMA_HIGH_ADDR_MASK; + hw->wrap_to = (dest >> SPRD_DMA_HIGH_ADDR_OFFSET) & + SPRD_DMA_HIGH_ADDR_MASK; + + if (IS_ALIGNED(len, 8)) { + datawidth = SPRD_DMA_DATAWIDTH_8_BYTES; + step = SPRD_DMA_DWORD_STEP; + } else if (IS_ALIGNED(len, 4)) { + datawidth = SPRD_DMA_DATAWIDTH_4_BYTES; + step = SPRD_DMA_WORD_STEP; + } else if (IS_ALIGNED(len, 2)) { + datawidth = SPRD_DMA_DATAWIDTH_2_BYTES; + step = SPRD_DMA_SHORT_STEP; + } else { + datawidth = SPRD_DMA_DATAWIDTH_1_BYTE; + step = SPRD_DMA_BYTE_STEP; } - if (transcation_len == 0) - hw->trsc_len = block_len & SPRD_DMA_TRSC_LEN_MASK; - else - hw->trsc_len = transcation_len & SPRD_DMA_TRSC_LEN_MASK; + temp = datawidth << SPRD_DMA_SRC_DATAWIDTH_OFFSET; + temp |= datawidth << SPRD_DMA_DES_DATAWIDTH_OFFSET; + temp |= SPRD_DMA_TRANS_REQ << SPRD_DMA_REQ_MODE_OFFSET; + temp |= len & SPRD_DMA_FRG_LEN_MASK; + hw->frg_len = temp; - hw->trsf_step = (des_step & SPRD_DMA_TRSF_STEP_MASK) << - SPRD_DMA_DEST_TRSF_STEP_OFFSET | - (src_step & SPRD_DMA_TRSF_STEP_MASK) << - SPRD_DMA_SRC_TRSF_STEP_OFFSET; + hw->blk_len = len & SPRD_DMA_BLK_LEN_MASK; + hw->trsc_len = len & SPRD_DMA_TRSC_LEN_MASK; - hw->frg_step = 0; - hw->src_blk_step = 0; - hw->des_blk_step = 0; - hw->src_blk_step = 0; - return 0; + temp = (step & SPRD_DMA_TRSF_STEP_MASK) << SPRD_DMA_DEST_TRSF_STEP_OFFSET; + temp |= (step & SPRD_DMA_TRSF_STEP_MASK) << SPRD_DMA_SRC_TRSF_STEP_OFFSET; + hw->trsf_step = temp; + + return vchan_tx_prep(&schan->vc, &sdesc->vd, flags); } static struct dma_async_tx_descriptor * -sprd_dma_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dest, dma_addr_t src, - size_t len, unsigned long flags) +sprd_dma_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl, + unsigned int sglen, enum dma_transfer_direction dir, + unsigned long flags, void *context) { struct sprd_dma_chn *schan = to_sprd_dma_chan(chan); + struct dma_slave_config *slave_cfg = &schan->slave_cfg; + dma_addr_t src = 0, dst = 0; struct sprd_dma_desc *sdesc; - int ret; + struct scatterlist *sg; + u32 len = 0; + int ret, i; + + /* TODO: now we only support one sg for each DMA configuration. */ + if (!is_slave_direction(dir) || sglen > 1) + return NULL; sdesc = kzalloc(sizeof(*sdesc), GFP_NOWAIT); if (!sdesc) return NULL; - ret = sprd_dma_config(chan, sdesc, dest, src, len); + for_each_sg(sgl, sg, sglen, i) { + len = sg_dma_len(sg); + + if (dir == DMA_MEM_TO_DEV) { + src = sg_dma_address(sg); + dst = slave_cfg->dst_addr; + } else { + src = slave_cfg->src_addr; + dst = sg_dma_address(sg); + } + } + + ret = sprd_dma_fill_desc(chan, sdesc, src, dst, len, dir, flags, + slave_cfg); if (ret) { kfree(sdesc); return NULL; @@ -731,6 +774,19 @@ sprd_dma_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dest, dma_addr_t src, return vchan_tx_prep(&schan->vc, &sdesc->vd, flags); } +static int sprd_dma_slave_config(struct dma_chan *chan, + struct dma_slave_config *config) +{ + struct sprd_dma_chn *schan = to_sprd_dma_chan(chan); + struct dma_slave_config *slave_cfg = &schan->slave_cfg; + + if (!is_slave_direction(config->direction)) + return -EINVAL; + + memcpy(slave_cfg, config, sizeof(*config)); + return 0; +} + static int sprd_dma_pause(struct dma_chan *chan) { struct sprd_dma_chn *schan = to_sprd_dma_chan(chan); @@ -805,8 +861,8 @@ static int sprd_dma_probe(struct platform_device *pdev) return ret; } - sdev = devm_kzalloc(&pdev->dev, sizeof(*sdev) + - sizeof(*dma_chn) * chn_count, + sdev = devm_kzalloc(&pdev->dev, + struct_size(sdev, channels, chn_count), GFP_KERNEL); if (!sdev) return -ENOMEM; @@ -842,10 +898,9 @@ static int sprd_dma_probe(struct platform_device *pdev) } res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - sdev->glb_base = devm_ioremap_nocache(&pdev->dev, res->start, - resource_size(res)); - if (!sdev->glb_base) - return -ENOMEM; + sdev->glb_base = devm_ioremap_resource(&pdev->dev, res); + if (IS_ERR(sdev->glb_base)) + return PTR_ERR(sdev->glb_base); dma_cap_set(DMA_MEMCPY, sdev->dma_dev.cap_mask); sdev->total_chns = chn_count; @@ -858,6 +913,8 @@ static int sprd_dma_probe(struct platform_device *pdev) sdev->dma_dev.device_tx_status = sprd_dma_tx_status; sdev->dma_dev.device_issue_pending = sprd_dma_issue_pending; sdev->dma_dev.device_prep_dma_memcpy = sprd_dma_prep_dma_memcpy; + sdev->dma_dev.device_prep_slave_sg = sprd_dma_prep_slave_sg; + sdev->dma_dev.device_config = sprd_dma_slave_config; sdev->dma_dev.device_pause = sprd_dma_pause; sdev->dma_dev.device_resume = sprd_dma_resume; sdev->dma_dev.device_terminate_all = sprd_dma_terminate_all; diff --git a/drivers/dma/ste_dma40.c b/drivers/dma/ste_dma40.c index c2b089af0420..1bc149af990e 100644 --- a/drivers/dma/ste_dma40.c +++ b/drivers/dma/ste_dma40.c @@ -2889,8 +2889,7 @@ static int __init d40_dmaengine_init(struct d40_base *base, #ifdef CONFIG_PM_SLEEP static int dma40_suspend(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct d40_base *base = platform_get_drvdata(pdev); + struct d40_base *base = dev_get_drvdata(dev); int ret; ret = pm_runtime_force_suspend(dev); @@ -2904,8 +2903,7 @@ static int dma40_suspend(struct device *dev) static int dma40_resume(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct d40_base *base = platform_get_drvdata(pdev); + struct d40_base *base = dev_get_drvdata(dev); int ret = 0; if (base->lcpa_regulator) { @@ -2970,8 +2968,7 @@ static void d40_save_restore_registers(struct d40_base *base, bool save) static int dma40_runtime_suspend(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct d40_base *base = platform_get_drvdata(pdev); + struct d40_base *base = dev_get_drvdata(dev); d40_save_restore_registers(base, true); @@ -2985,8 +2982,7 @@ static int dma40_runtime_suspend(struct device *dev) static int dma40_runtime_resume(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct d40_base *base = platform_get_drvdata(pdev); + struct d40_base *base = dev_get_drvdata(dev); d40_save_restore_registers(base, false); diff --git a/drivers/dma/stm32-mdma.c b/drivers/dma/stm32-mdma.c index daa1602eb9f5..9dc450b7ace6 100644 --- a/drivers/dma/stm32-mdma.c +++ b/drivers/dma/stm32-mdma.c @@ -252,13 +252,17 @@ struct stm32_mdma_hwdesc { u32 cmdr; } __aligned(64); +struct stm32_mdma_desc_node { + struct stm32_mdma_hwdesc *hwdesc; + dma_addr_t hwdesc_phys; +}; + struct stm32_mdma_desc { struct virt_dma_desc vdesc; u32 ccr; - struct stm32_mdma_hwdesc *hwdesc; - dma_addr_t hwdesc_phys; bool cyclic; u32 count; + struct stm32_mdma_desc_node node[]; }; struct stm32_mdma_chan { @@ -344,30 +348,42 @@ static struct stm32_mdma_desc *stm32_mdma_alloc_desc( struct stm32_mdma_chan *chan, u32 count) { struct stm32_mdma_desc *desc; + int i; - desc = kzalloc(sizeof(*desc), GFP_NOWAIT); + desc = kzalloc(offsetof(typeof(*desc), node[count]), GFP_NOWAIT); if (!desc) return NULL; - desc->hwdesc = dma_pool_alloc(chan->desc_pool, GFP_NOWAIT, - &desc->hwdesc_phys); - if (!desc->hwdesc) { - dev_err(chan2dev(chan), "Failed to allocate descriptor\n"); - kfree(desc); - return NULL; + for (i = 0; i < count; i++) { + desc->node[i].hwdesc = + dma_pool_alloc(chan->desc_pool, GFP_NOWAIT, + &desc->node[i].hwdesc_phys); + if (!desc->node[i].hwdesc) + goto err; } desc->count = count; return desc; + +err: + dev_err(chan2dev(chan), "Failed to allocate descriptor\n"); + while (--i >= 0) + dma_pool_free(chan->desc_pool, desc->node[i].hwdesc, + desc->node[i].hwdesc_phys); + kfree(desc); + return NULL; } static void stm32_mdma_desc_free(struct virt_dma_desc *vdesc) { struct stm32_mdma_desc *desc = to_stm32_mdma_desc(vdesc); struct stm32_mdma_chan *chan = to_stm32_mdma_chan(vdesc->tx.chan); + int i; - dma_pool_free(chan->desc_pool, desc->hwdesc, desc->hwdesc_phys); + for (i = 0; i < desc->count; i++) + dma_pool_free(chan->desc_pool, desc->node[i].hwdesc, + desc->node[i].hwdesc_phys); kfree(desc); } @@ -410,13 +426,10 @@ static enum dma_slave_buswidth stm32_mdma_get_max_width(dma_addr_t addr, static u32 stm32_mdma_get_best_burst(u32 buf_len, u32 tlen, u32 max_burst, enum dma_slave_buswidth width) { - u32 best_burst = max_burst; - u32 burst_len = best_burst * width; + u32 best_burst; - while ((burst_len > 0) && (tlen % burst_len)) { - best_burst = best_burst >> 1; - burst_len = best_burst * width; - } + best_burst = min((u32)1 << __ffs(tlen | buf_len), + max_burst * width) / width; return (best_burst > 0) ? best_burst : 1; } @@ -669,18 +682,18 @@ static int stm32_mdma_set_xfer_param(struct stm32_mdma_chan *chan, } static void stm32_mdma_dump_hwdesc(struct stm32_mdma_chan *chan, - struct stm32_mdma_hwdesc *hwdesc) + struct stm32_mdma_desc_node *node) { - dev_dbg(chan2dev(chan), "hwdesc: 0x%p\n", hwdesc); - dev_dbg(chan2dev(chan), "CTCR: 0x%08x\n", hwdesc->ctcr); - dev_dbg(chan2dev(chan), "CBNDTR: 0x%08x\n", hwdesc->cbndtr); - dev_dbg(chan2dev(chan), "CSAR: 0x%08x\n", hwdesc->csar); - dev_dbg(chan2dev(chan), "CDAR: 0x%08x\n", hwdesc->cdar); - dev_dbg(chan2dev(chan), "CBRUR: 0x%08x\n", hwdesc->cbrur); - dev_dbg(chan2dev(chan), "CLAR: 0x%08x\n", hwdesc->clar); - dev_dbg(chan2dev(chan), "CTBR: 0x%08x\n", hwdesc->ctbr); - dev_dbg(chan2dev(chan), "CMAR: 0x%08x\n", hwdesc->cmar); - dev_dbg(chan2dev(chan), "CMDR: 0x%08x\n\n", hwdesc->cmdr); + dev_dbg(chan2dev(chan), "hwdesc: %pad\n", &node->hwdesc_phys); + dev_dbg(chan2dev(chan), "CTCR: 0x%08x\n", node->hwdesc->ctcr); + dev_dbg(chan2dev(chan), "CBNDTR: 0x%08x\n", node->hwdesc->cbndtr); + dev_dbg(chan2dev(chan), "CSAR: 0x%08x\n", node->hwdesc->csar); + dev_dbg(chan2dev(chan), "CDAR: 0x%08x\n", node->hwdesc->cdar); + dev_dbg(chan2dev(chan), "CBRUR: 0x%08x\n", node->hwdesc->cbrur); + dev_dbg(chan2dev(chan), "CLAR: 0x%08x\n", node->hwdesc->clar); + dev_dbg(chan2dev(chan), "CTBR: 0x%08x\n", node->hwdesc->ctbr); + dev_dbg(chan2dev(chan), "CMAR: 0x%08x\n", node->hwdesc->cmar); + dev_dbg(chan2dev(chan), "CMDR: 0x%08x\n\n", node->hwdesc->cmdr); } static void stm32_mdma_setup_hwdesc(struct stm32_mdma_chan *chan, @@ -694,7 +707,7 @@ static void stm32_mdma_setup_hwdesc(struct stm32_mdma_chan *chan, struct stm32_mdma_hwdesc *hwdesc; u32 next = count + 1; - hwdesc = &desc->hwdesc[count]; + hwdesc = desc->node[count].hwdesc; hwdesc->ctcr = ctcr; hwdesc->cbndtr &= ~(STM32_MDMA_CBNDTR_BRC_MK | STM32_MDMA_CBNDTR_BRDUM | @@ -704,19 +717,20 @@ static void stm32_mdma_setup_hwdesc(struct stm32_mdma_chan *chan, hwdesc->csar = src_addr; hwdesc->cdar = dst_addr; hwdesc->cbrur = 0; - hwdesc->clar = desc->hwdesc_phys + next * sizeof(*hwdesc); hwdesc->ctbr = ctbr; hwdesc->cmar = config->mask_addr; hwdesc->cmdr = config->mask_data; if (is_last) { if (is_cyclic) - hwdesc->clar = desc->hwdesc_phys; + hwdesc->clar = desc->node[0].hwdesc_phys; else hwdesc->clar = 0; + } else { + hwdesc->clar = desc->node[next].hwdesc_phys; } - stm32_mdma_dump_hwdesc(chan, hwdesc); + stm32_mdma_dump_hwdesc(chan, &desc->node[count]); } static int stm32_mdma_setup_xfer(struct stm32_mdma_chan *chan, @@ -780,7 +794,7 @@ stm32_mdma_prep_slave_sg(struct dma_chan *c, struct scatterlist *sgl, { struct stm32_mdma_chan *chan = to_stm32_mdma_chan(c); struct stm32_mdma_desc *desc; - int ret; + int i, ret; /* * Once DMA is in setup cyclic mode the channel we cannot assign this @@ -806,7 +820,9 @@ stm32_mdma_prep_slave_sg(struct dma_chan *c, struct scatterlist *sgl, return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags); xfer_setup_err: - dma_pool_free(chan->desc_pool, &desc->hwdesc, desc->hwdesc_phys); + for (i = 0; i < desc->count; i++) + dma_pool_free(chan->desc_pool, desc->node[i].hwdesc, + desc->node[i].hwdesc_phys); kfree(desc); return NULL; } @@ -895,7 +911,9 @@ stm32_mdma_prep_dma_cyclic(struct dma_chan *c, dma_addr_t buf_addr, return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags); xfer_setup_err: - dma_pool_free(chan->desc_pool, &desc->hwdesc, desc->hwdesc_phys); + for (i = 0; i < desc->count; i++) + dma_pool_free(chan->desc_pool, desc->node[i].hwdesc, + desc->node[i].hwdesc_phys); kfree(desc); return NULL; } @@ -1009,7 +1027,7 @@ stm32_mdma_prep_dma_memcpy(struct dma_chan *c, dma_addr_t dest, dma_addr_t src, ctcr |= STM32_MDMA_CTCR_PKE; /* Prepare hardware descriptor */ - hwdesc = desc->hwdesc; + hwdesc = desc->node[0].hwdesc; hwdesc->ctcr = ctcr; hwdesc->cbndtr = cbndtr; hwdesc->csar = src; @@ -1020,7 +1038,7 @@ stm32_mdma_prep_dma_memcpy(struct dma_chan *c, dma_addr_t dest, dma_addr_t src, hwdesc->cmar = 0; hwdesc->cmdr = 0; - stm32_mdma_dump_hwdesc(chan, hwdesc); + stm32_mdma_dump_hwdesc(chan, &desc->node[0]); } else { /* Setup a LLI transfer */ ctcr |= STM32_MDMA_CTCR_TRGM(STM32_MDMA_LINKED_LIST) | @@ -1120,7 +1138,7 @@ static void stm32_mdma_start_transfer(struct stm32_mdma_chan *chan) } chan->desc = to_stm32_mdma_desc(vdesc); - hwdesc = chan->desc->hwdesc; + hwdesc = chan->desc->node[0].hwdesc; chan->curr_hwdesc = 0; stm32_mdma_write(dmadev, STM32_MDMA_CCR(id), chan->desc->ccr); @@ -1198,7 +1216,7 @@ static int stm32_mdma_resume(struct dma_chan *c) unsigned long flags; u32 status, reg; - hwdesc = &chan->desc->hwdesc[chan->curr_hwdesc]; + hwdesc = chan->desc->node[chan->curr_hwdesc].hwdesc; spin_lock_irqsave(&chan->vchan.lock, flags); @@ -1268,13 +1286,13 @@ static size_t stm32_mdma_desc_residue(struct stm32_mdma_chan *chan, u32 curr_hwdesc) { struct stm32_mdma_device *dmadev = stm32_mdma_get_dev(chan); + struct stm32_mdma_hwdesc *hwdesc = desc->node[0].hwdesc; u32 cbndtr, residue, modulo, burst_size; int i; residue = 0; for (i = curr_hwdesc + 1; i < desc->count; i++) { - struct stm32_mdma_hwdesc *hwdesc = &desc->hwdesc[i]; - + hwdesc = desc->node[i].hwdesc; residue += STM32_MDMA_CBNDTR_BNDT(hwdesc->cbndtr); } cbndtr = stm32_mdma_read(dmadev, STM32_MDMA_CBNDTR(chan->id)); @@ -1503,7 +1521,7 @@ static struct dma_chan *stm32_mdma_of_xlate(struct of_phandle_args *dma_spec, c = dma_get_any_slave_channel(&dmadev->ddev); if (!c) { - dev_err(mdma2dev(dmadev), "No more channel avalaible\n"); + dev_err(mdma2dev(dmadev), "No more channels available\n"); return NULL; } diff --git a/drivers/dma/ti/Kconfig b/drivers/dma/ti/Kconfig new file mode 100644 index 000000000000..e5e74e1361dc --- /dev/null +++ b/drivers/dma/ti/Kconfig @@ -0,0 +1,37 @@ +# +# Texas Instruments DMA drivers +# + +config TI_CPPI41 + tristate "Texas Instruments CPPI 4.1 DMA support" + depends on (ARCH_OMAP || ARCH_DAVINCI_DA8XX) + select DMA_ENGINE + help + The Communications Port Programming Interface (CPPI) 4.1 DMA engine + is currently used by the USB driver on AM335x and DA8xx platforms. + +config TI_EDMA + tristate "Texas Instruments EDMA support" + depends on ARCH_DAVINCI || ARCH_OMAP || ARCH_KEYSTONE || COMPILE_TEST + select DMA_ENGINE + select DMA_VIRTUAL_CHANNELS + select TI_DMA_CROSSBAR if (ARCH_OMAP || COMPILE_TEST) + default y + help + Enable support for the TI EDMA (Enhanced DMA) controller. This DMA + engine is found on TI DaVinci, AM33xx, AM43xx, DRA7xx and Keystone 2 + parts. + +config DMA_OMAP + tristate "Texas Instruments sDMA (omap-dma) support" + depends on ARCH_OMAP || COMPILE_TEST + select DMA_ENGINE + select DMA_VIRTUAL_CHANNELS + select TI_DMA_CROSSBAR if (SOC_DRA7XX || COMPILE_TEST) + default y + help + Enable support for the TI sDMA (System DMA or DMA4) controller. This + DMA engine is found on OMAP and DRA7xx parts. + +config TI_DMA_CROSSBAR + bool diff --git a/drivers/dma/ti/Makefile b/drivers/dma/ti/Makefile new file mode 100644 index 000000000000..113e59ec9c32 --- /dev/null +++ b/drivers/dma/ti/Makefile @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_TI_CPPI41) += cppi41.o +obj-$(CONFIG_TI_EDMA) += edma.o +obj-$(CONFIG_DMA_OMAP) += omap-dma.o +obj-$(CONFIG_TI_DMA_CROSSBAR) += dma-crossbar.o diff --git a/drivers/dma/cppi41.c b/drivers/dma/ti/cppi41.c similarity index 99% rename from drivers/dma/cppi41.c rename to drivers/dma/ti/cppi41.c index d9bee65a18a4..1497da367710 100644 --- a/drivers/dma/cppi41.c +++ b/drivers/dma/ti/cppi41.c @@ -11,7 +11,7 @@ #include #include #include -#include "dmaengine.h" +#include "../dmaengine.h" #define DESC_TYPE 27 #define DESC_TYPE_HOST 0x10 diff --git a/drivers/dma/ti-dma-crossbar.c b/drivers/dma/ti/dma-crossbar.c similarity index 100% rename from drivers/dma/ti-dma-crossbar.c rename to drivers/dma/ti/dma-crossbar.c diff --git a/drivers/dma/edma.c b/drivers/dma/ti/edma.c similarity index 99% rename from drivers/dma/edma.c rename to drivers/dma/ti/edma.c index 85ea92fcea54..ceabdea40ae0 100644 --- a/drivers/dma/edma.c +++ b/drivers/dma/ti/edma.c @@ -33,8 +33,8 @@ #include -#include "dmaengine.h" -#include "virt-dma.h" +#include "../dmaengine.h" +#include "../virt-dma.h" /* Offsets matching "struct edmacc_param" */ #define PARM_OPT 0x00 @@ -1074,8 +1074,7 @@ static struct dma_async_tx_descriptor *edma_prep_slave_sg( return NULL; } - edesc = kzalloc(sizeof(*edesc) + sg_len * sizeof(edesc->pset[0]), - GFP_ATOMIC); + edesc = kzalloc(struct_size(edesc, pset, sg_len), GFP_ATOMIC); if (!edesc) return NULL; @@ -1192,8 +1191,7 @@ static struct dma_async_tx_descriptor *edma_prep_dma_memcpy( nslots = 2; } - edesc = kzalloc(sizeof(*edesc) + nslots * sizeof(edesc->pset[0]), - GFP_ATOMIC); + edesc = kzalloc(struct_size(edesc, pset, nslots), GFP_ATOMIC); if (!edesc) return NULL; @@ -1315,8 +1313,7 @@ static struct dma_async_tx_descriptor *edma_prep_dma_cyclic( } } - edesc = kzalloc(sizeof(*edesc) + nslots * sizeof(edesc->pset[0]), - GFP_ATOMIC); + edesc = kzalloc(struct_size(edesc, pset, nslots), GFP_ATOMIC); if (!edesc) return NULL; diff --git a/drivers/dma/omap-dma.c b/drivers/dma/ti/omap-dma.c similarity index 99% rename from drivers/dma/omap-dma.c rename to drivers/dma/ti/omap-dma.c index d21c19822feb..9b5ca8691f27 100644 --- a/drivers/dma/omap-dma.c +++ b/drivers/dma/ti/omap-dma.c @@ -21,7 +21,7 @@ #include #include -#include "virt-dma.h" +#include "../virt-dma.h" #define OMAP_SDMA_REQUESTS 127 #define OMAP_SDMA_CHANNELS 32 @@ -917,7 +917,7 @@ static struct dma_async_tx_descriptor *omap_dma_prep_slave_sg( } /* Now allocate and setup the descriptor. */ - d = kzalloc(sizeof(*d) + sglen * sizeof(d->sg[0]), GFP_ATOMIC); + d = kzalloc(struct_size(d, sg, sglen), GFP_ATOMIC); if (!d) return NULL; diff --git a/drivers/dma/txx9dmac.c b/drivers/dma/txx9dmac.c index 4d8c7b9078fd..eb45af71d3a3 100644 --- a/drivers/dma/txx9dmac.c +++ b/drivers/dma/txx9dmac.c @@ -1244,8 +1244,7 @@ static void txx9dmac_shutdown(struct platform_device *pdev) static int txx9dmac_suspend_noirq(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct txx9dmac_dev *ddev = platform_get_drvdata(pdev); + struct txx9dmac_dev *ddev = dev_get_drvdata(dev); txx9dmac_off(ddev); return 0; @@ -1253,9 +1252,8 @@ static int txx9dmac_suspend_noirq(struct device *dev) static int txx9dmac_resume_noirq(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - struct txx9dmac_dev *ddev = platform_get_drvdata(pdev); - struct txx9dmac_platform_data *pdata = dev_get_platdata(&pdev->dev); + struct txx9dmac_dev *ddev = dev_get_drvdata(dev); + struct txx9dmac_platform_data *pdata = dev_get_platdata(dev); u32 mcr; mcr = TXX9_DMA_MCR_MSTEN | MCR_LE; diff --git a/drivers/dma/xilinx/zynqmp_dma.c b/drivers/dma/xilinx/zynqmp_dma.c index f14645817ed8..c74a88b65039 100644 --- a/drivers/dma/xilinx/zynqmp_dma.c +++ b/drivers/dma/xilinx/zynqmp_dma.c @@ -471,7 +471,7 @@ static int zynqmp_dma_alloc_chan_resources(struct dma_chan *dchan) if (ret < 0) return ret; - chan->sw_desc_pool = kzalloc(sizeof(*desc) * ZYNQMP_DMA_NUM_DESCS, + chan->sw_desc_pool = kcalloc(ZYNQMP_DMA_NUM_DESCS, sizeof(*desc), GFP_KERNEL); if (!chan->sw_desc_pool) return -ENOMEM; diff --git a/drivers/dma/zx_dma.c b/drivers/dma/zx_dma.c index 2bb695315300..2571bc7693df 100644 --- a/drivers/dma/zx_dma.c +++ b/drivers/dma/zx_dma.c @@ -798,8 +798,8 @@ static int zx_dma_probe(struct platform_device *op) return -ENOMEM; /* init phy channel */ - d->phy = devm_kzalloc(&op->dev, - d->dma_channels * sizeof(struct zx_dma_phy), GFP_KERNEL); + d->phy = devm_kcalloc(&op->dev, + d->dma_channels, sizeof(struct zx_dma_phy), GFP_KERNEL); if (!d->phy) return -ENOMEM; @@ -834,8 +834,8 @@ static int zx_dma_probe(struct platform_device *op) d->slave.residue_granularity = DMA_RESIDUE_GRANULARITY_SEGMENT; /* init virtual channel */ - d->chans = devm_kzalloc(&op->dev, - d->dma_requests * sizeof(struct zx_dma_chan), GFP_KERNEL); + d->chans = devm_kcalloc(&op->dev, + d->dma_requests, sizeof(struct zx_dma_chan), GFP_KERNEL); if (!d->chans) return -ENOMEM; diff --git a/drivers/edac/Kconfig b/drivers/edac/Kconfig index da2da53bca6d..57304b2e989f 100644 --- a/drivers/edac/Kconfig +++ b/drivers/edac/Kconfig @@ -232,6 +232,7 @@ config EDAC_SBRIDGE config EDAC_SKX tristate "Intel Skylake server Integrated MC" depends on PCI && X86_64 && X86_MCE_INTEL && PCI_MMCONFIG + depends on ACPI_NFIT || !ACPI_NFIT # if ACPI_NFIT=m, EDAC_SKX can't be y select DMI help Support for error detection and correction the Intel @@ -374,7 +375,7 @@ config EDAC_THUNDERX config EDAC_ALTERA bool "Altera SOCFPGA ECC" - depends on EDAC=y && ARCH_SOCFPGA + depends on EDAC=y && (ARCH_SOCFPGA || ARCH_STRATIX10) help Support for error detection and correction on the Altera SOCs. This must be selected for SDRAM ECC. diff --git a/drivers/edac/altera_edac.c b/drivers/edac/altera_edac.c index 11d6419788c2..d0d5c4dbe097 100644 --- a/drivers/edac/altera_edac.c +++ b/drivers/edac/altera_edac.c @@ -1,20 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* + * Copyright (C) 2017-2018, Intel Corporation. All rights reserved * Copyright Altera Corporation (C) 2014-2016. All rights reserved. * Copyright 2011-2012 Calxeda, Inc. - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . - * - * Adapted from the highbank_mc_edac driver. */ #include @@ -26,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -80,6 +69,25 @@ static const struct altr_sdram_prv_data a10_data = { .ue_set_mask = A10_DIAGINT_TDERRA_MASK, }; +static const struct altr_sdram_prv_data s10_data = { + .ecc_ctrl_offset = S10_ECCCTRL1_OFST, + .ecc_ctl_en_mask = A10_ECCCTRL1_ECC_EN, + .ecc_stat_offset = S10_INTSTAT_OFST, + .ecc_stat_ce_mask = A10_INTSTAT_SBEERR, + .ecc_stat_ue_mask = A10_INTSTAT_DBEERR, + .ecc_saddr_offset = S10_SERRADDR_OFST, + .ecc_daddr_offset = S10_DERRADDR_OFST, + .ecc_irq_en_offset = S10_ERRINTEN_OFST, + .ecc_irq_en_mask = A10_ECC_IRQ_EN_MASK, + .ecc_irq_clr_offset = S10_INTSTAT_OFST, + .ecc_irq_clr_mask = (A10_INTSTAT_SBEERR | A10_INTSTAT_DBEERR), + .ecc_cnt_rst_offset = S10_ECCCTRL1_OFST, + .ecc_cnt_rst_mask = A10_ECC_CNT_RESET_MASK, + .ce_ue_trgr_offset = S10_DIAGINTTEST_OFST, + .ce_set_mask = A10_DIAGINT_TSERRA_MASK, + .ue_set_mask = A10_DIAGINT_TDERRA_MASK, +}; + /*********************** EDAC Memory Controller Functions ****************/ /* The SDRAM controller uses the EDAC Memory Controller framework. */ @@ -231,6 +239,7 @@ static unsigned long get_total_mem(void) static const struct of_device_id altr_sdram_ctrl_of_match[] = { { .compatible = "altr,sdram-edac", .data = &c5_data}, { .compatible = "altr,sdram-edac-a10", .data = &a10_data}, + { .compatible = "altr,sdram-edac-s10", .data = &s10_data}, {}, }; MODULE_DEVICE_TABLE(of, altr_sdram_ctrl_of_match); @@ -477,6 +486,292 @@ static int altr_sdram_remove(struct platform_device *pdev) return 0; } +/**************** Stratix 10 EDAC Memory Controller Functions ************/ + +/** + * s10_protected_reg_write + * Write to a protected SMC register. + * @context: Not used. + * @reg: Address of register + * @value: Value to write + * Return: INTEL_SIP_SMC_STATUS_OK (0) on success + * INTEL_SIP_SMC_REG_ERROR on error + * INTEL_SIP_SMC_RETURN_UNKNOWN_FUNCTION if not supported + */ +static int s10_protected_reg_write(void *context, unsigned int reg, + unsigned int val) +{ + struct arm_smccc_res result; + + arm_smccc_smc(INTEL_SIP_SMC_REG_WRITE, reg, val, 0, 0, + 0, 0, 0, &result); + + return (int)result.a0; +} + +/** + * s10_protected_reg_read + * Read the status of a protected SMC register + * @context: Not used. + * @reg: Address of register + * @value: Value read. + * Return: INTEL_SIP_SMC_STATUS_OK (0) on success + * INTEL_SIP_SMC_REG_ERROR on error + * INTEL_SIP_SMC_RETURN_UNKNOWN_FUNCTION if not supported + */ +static int s10_protected_reg_read(void *context, unsigned int reg, + unsigned int *val) +{ + struct arm_smccc_res result; + + arm_smccc_smc(INTEL_SIP_SMC_REG_READ, reg, 0, 0, 0, + 0, 0, 0, &result); + + *val = (unsigned int)result.a1; + + return (int)result.a0; +} + +static bool s10_sdram_writeable_reg(struct device *dev, unsigned int reg) +{ + switch (reg) { + case S10_ECCCTRL1_OFST: + case S10_ERRINTEN_OFST: + case S10_INTMODE_OFST: + case S10_INTSTAT_OFST: + case S10_DIAGINTTEST_OFST: + case S10_SYSMGR_ECC_INTMASK_VAL_OFST: + case S10_SYSMGR_ECC_INTMASK_SET_OFST: + case S10_SYSMGR_ECC_INTMASK_CLR_OFST: + return true; + } + return false; +} + +static bool s10_sdram_readable_reg(struct device *dev, unsigned int reg) +{ + switch (reg) { + case S10_ECCCTRL1_OFST: + case S10_ERRINTEN_OFST: + case S10_INTMODE_OFST: + case S10_INTSTAT_OFST: + case S10_DERRADDR_OFST: + case S10_SERRADDR_OFST: + case S10_DIAGINTTEST_OFST: + case S10_SYSMGR_ECC_INTMASK_VAL_OFST: + case S10_SYSMGR_ECC_INTMASK_SET_OFST: + case S10_SYSMGR_ECC_INTMASK_CLR_OFST: + case S10_SYSMGR_ECC_INTSTAT_SERR_OFST: + case S10_SYSMGR_ECC_INTSTAT_DERR_OFST: + return true; + } + return false; +} + +static bool s10_sdram_volatile_reg(struct device *dev, unsigned int reg) +{ + switch (reg) { + case S10_ECCCTRL1_OFST: + case S10_ERRINTEN_OFST: + case S10_INTMODE_OFST: + case S10_INTSTAT_OFST: + case S10_DERRADDR_OFST: + case S10_SERRADDR_OFST: + case S10_DIAGINTTEST_OFST: + case S10_SYSMGR_ECC_INTMASK_VAL_OFST: + case S10_SYSMGR_ECC_INTMASK_SET_OFST: + case S10_SYSMGR_ECC_INTMASK_CLR_OFST: + case S10_SYSMGR_ECC_INTSTAT_SERR_OFST: + case S10_SYSMGR_ECC_INTSTAT_DERR_OFST: + return true; + } + return false; +} + +static const struct regmap_config s10_sdram_regmap_cfg = { + .name = "s10_ddr", + .reg_bits = 32, + .reg_stride = 4, + .val_bits = 32, + .max_register = 0xffffffff, + .writeable_reg = s10_sdram_writeable_reg, + .readable_reg = s10_sdram_readable_reg, + .volatile_reg = s10_sdram_volatile_reg, + .reg_read = s10_protected_reg_read, + .reg_write = s10_protected_reg_write, + .use_single_rw = true, +}; + +static int altr_s10_sdram_probe(struct platform_device *pdev) +{ + const struct of_device_id *id; + struct edac_mc_layer layers[2]; + struct mem_ctl_info *mci; + struct altr_sdram_mc_data *drvdata; + const struct altr_sdram_prv_data *priv; + struct regmap *regmap; + struct dimm_info *dimm; + u32 read_reg; + int irq, ret = 0; + unsigned long mem_size; + + id = of_match_device(altr_sdram_ctrl_of_match, &pdev->dev); + if (!id) + return -ENODEV; + + /* Grab specific offsets and masks for Stratix10 */ + priv = of_match_node(altr_sdram_ctrl_of_match, + pdev->dev.of_node)->data; + + regmap = devm_regmap_init(&pdev->dev, NULL, (void *)priv, + &s10_sdram_regmap_cfg); + if (IS_ERR(regmap)) + return PTR_ERR(regmap); + + /* Validate the SDRAM controller has ECC enabled */ + if (regmap_read(regmap, priv->ecc_ctrl_offset, &read_reg) || + ((read_reg & priv->ecc_ctl_en_mask) != priv->ecc_ctl_en_mask)) { + edac_printk(KERN_ERR, EDAC_MC, + "No ECC/ECC disabled [0x%08X]\n", read_reg); + return -ENODEV; + } + + /* Grab memory size from device tree. */ + mem_size = get_total_mem(); + if (!mem_size) { + edac_printk(KERN_ERR, EDAC_MC, "Unable to calculate memory size\n"); + return -ENODEV; + } + + /* Ensure the SDRAM Interrupt is disabled */ + if (regmap_update_bits(regmap, priv->ecc_irq_en_offset, + priv->ecc_irq_en_mask, 0)) { + edac_printk(KERN_ERR, EDAC_MC, + "Error disabling SDRAM ECC IRQ\n"); + return -ENODEV; + } + + /* Toggle to clear the SDRAM Error count */ + if (regmap_update_bits(regmap, priv->ecc_cnt_rst_offset, + priv->ecc_cnt_rst_mask, + priv->ecc_cnt_rst_mask)) { + edac_printk(KERN_ERR, EDAC_MC, + "Error clearing SDRAM ECC count\n"); + return -ENODEV; + } + + if (regmap_update_bits(regmap, priv->ecc_cnt_rst_offset, + priv->ecc_cnt_rst_mask, 0)) { + edac_printk(KERN_ERR, EDAC_MC, + "Error clearing SDRAM ECC count\n"); + return -ENODEV; + } + + irq = platform_get_irq(pdev, 0); + if (irq < 0) { + edac_printk(KERN_ERR, EDAC_MC, + "No irq %d in DT\n", irq); + return -ENODEV; + } + + layers[0].type = EDAC_MC_LAYER_CHIP_SELECT; + layers[0].size = 1; + layers[0].is_virt_csrow = true; + layers[1].type = EDAC_MC_LAYER_CHANNEL; + layers[1].size = 1; + layers[1].is_virt_csrow = false; + mci = edac_mc_alloc(0, ARRAY_SIZE(layers), layers, + sizeof(struct altr_sdram_mc_data)); + if (!mci) + return -ENOMEM; + + mci->pdev = &pdev->dev; + drvdata = mci->pvt_info; + drvdata->mc_vbase = regmap; + drvdata->data = priv; + platform_set_drvdata(pdev, mci); + + if (!devres_open_group(&pdev->dev, NULL, GFP_KERNEL)) { + edac_printk(KERN_ERR, EDAC_MC, + "Unable to get managed device resource\n"); + ret = -ENOMEM; + goto free; + } + + mci->mtype_cap = MEM_FLAG_DDR3; + mci->edac_ctl_cap = EDAC_FLAG_NONE | EDAC_FLAG_SECDED; + mci->edac_cap = EDAC_FLAG_SECDED; + mci->mod_name = EDAC_MOD_STR; + mci->ctl_name = dev_name(&pdev->dev); + mci->scrub_mode = SCRUB_SW_SRC; + mci->dev_name = dev_name(&pdev->dev); + + dimm = *mci->dimms; + dimm->nr_pages = ((mem_size - 1) >> PAGE_SHIFT) + 1; + dimm->grain = 8; + dimm->dtype = DEV_X8; + dimm->mtype = MEM_DDR3; + dimm->edac_mode = EDAC_SECDED; + + ret = edac_mc_add_mc(mci); + if (ret < 0) + goto err; + + ret = devm_request_irq(&pdev->dev, irq, altr_sdram_mc_err_handler, + IRQF_SHARED, dev_name(&pdev->dev), mci); + if (ret < 0) { + edac_mc_printk(mci, KERN_ERR, + "Unable to request irq %d\n", irq); + ret = -ENODEV; + goto err2; + } + + if (regmap_write(regmap, S10_SYSMGR_ECC_INTMASK_CLR_OFST, + S10_DDR0_IRQ_MASK)) { + edac_printk(KERN_ERR, EDAC_MC, + "Error clearing SDRAM ECC count\n"); + return -ENODEV; + } + + if (regmap_update_bits(drvdata->mc_vbase, priv->ecc_irq_en_offset, + priv->ecc_irq_en_mask, priv->ecc_irq_en_mask)) { + edac_mc_printk(mci, KERN_ERR, + "Error enabling SDRAM ECC IRQ\n"); + ret = -ENODEV; + goto err2; + } + + altr_sdr_mc_create_debugfs_nodes(mci); + + devres_close_group(&pdev->dev, NULL); + + return 0; + +err2: + edac_mc_del_mc(&pdev->dev); +err: + devres_release_group(&pdev->dev, NULL); +free: + edac_mc_free(mci); + edac_printk(KERN_ERR, EDAC_MC, + "EDAC Probe Failed; Error %d\n", ret); + + return ret; +} + +static int altr_s10_sdram_remove(struct platform_device *pdev) +{ + struct mem_ctl_info *mci = platform_get_drvdata(pdev); + + edac_mc_del_mc(&pdev->dev); + edac_mc_free(mci); + platform_set_drvdata(pdev, NULL); + + return 0; +} + +/************** ***********/ + /* * If you want to suspend, need to disable EDAC by removing it * from the device tree or defconfig. @@ -508,6 +803,20 @@ static struct platform_driver altr_sdram_edac_driver = { module_platform_driver(altr_sdram_edac_driver); +static struct platform_driver altr_s10_sdram_edac_driver = { + .probe = altr_s10_sdram_probe, + .remove = altr_s10_sdram_remove, + .driver = { + .name = "altr_s10_sdram_edac", +#ifdef CONFIG_PM + .pm = &altr_sdram_pm_ops, +#endif + .of_match_table = altr_sdram_ctrl_of_match, + }, +}; + +module_platform_driver(altr_s10_sdram_edac_driver); + /************************* EDAC Parent Probe *************************/ static const struct of_device_id altr_edac_device_of_match[]; @@ -1106,7 +1415,7 @@ static void *ocram_alloc_mem(size_t size, void **other) static void ocram_free_mem(void *p, size_t size, void *other) { - gen_pool_free((struct gen_pool *)other, (u32)p, size); + gen_pool_free((struct gen_pool *)other, (unsigned long)p, size); } static const struct edac_device_prv_data ocramecc_data = { @@ -1925,6 +2234,171 @@ static struct platform_driver altr_edac_a10_driver = { }; module_platform_driver(altr_edac_a10_driver); +/************** Stratix 10 EDAC Device Controller Functions> ************/ + +#define to_s10edac(p, m) container_of(p, struct altr_stratix10_edac, m) + +/* + * The double bit error is handled through SError which is fatal. This is + * called as a panic notifier to printout ECC error info as part of the panic. + */ +static int s10_edac_dberr_handler(struct notifier_block *this, + unsigned long event, void *ptr) +{ + struct altr_stratix10_edac *edac = to_s10edac(this, panic_notifier); + int err_addr, dberror; + + s10_protected_reg_read(edac, S10_SYSMGR_ECC_INTSTAT_DERR_OFST, + &dberror); + /* Remember the UE Errors for a reboot */ + s10_protected_reg_write(edac, S10_SYSMGR_UE_VAL_OFST, dberror); + if (dberror & S10_DDR0_IRQ_MASK) { + s10_protected_reg_read(edac, S10_DERRADDR_OFST, &err_addr); + /* Remember the UE Error address */ + s10_protected_reg_write(edac, S10_SYSMGR_UE_ADDR_OFST, + err_addr); + edac_printk(KERN_ERR, EDAC_MC, + "EDAC: [Uncorrectable errors @ 0x%08X]\n\n", + err_addr); + } + + return NOTIFY_DONE; +} + +static void altr_edac_s10_irq_handler(struct irq_desc *desc) +{ + struct altr_stratix10_edac *edac = irq_desc_get_handler_data(desc); + struct irq_chip *chip = irq_desc_get_chip(desc); + int irq = irq_desc_get_irq(desc); + int bit, sm_offset, irq_status; + + sm_offset = S10_SYSMGR_ECC_INTSTAT_SERR_OFST; + + chained_irq_enter(chip, desc); + + s10_protected_reg_read(NULL, sm_offset, &irq_status); + + for_each_set_bit(bit, (unsigned long *)&irq_status, 32) { + irq = irq_linear_revmap(edac->domain, bit); + if (irq) + generic_handle_irq(irq); + } + + chained_irq_exit(chip, desc); +} + +static void s10_eccmgr_irq_mask(struct irq_data *d) +{ + struct altr_stratix10_edac *edac = irq_data_get_irq_chip_data(d); + + s10_protected_reg_write(edac, S10_SYSMGR_ECC_INTMASK_SET_OFST, + BIT(d->hwirq)); +} + +static void s10_eccmgr_irq_unmask(struct irq_data *d) +{ + struct altr_stratix10_edac *edac = irq_data_get_irq_chip_data(d); + + s10_protected_reg_write(edac, S10_SYSMGR_ECC_INTMASK_CLR_OFST, + BIT(d->hwirq)); +} + +static int s10_eccmgr_irqdomain_map(struct irq_domain *d, unsigned int irq, + irq_hw_number_t hwirq) +{ + struct altr_stratix10_edac *edac = d->host_data; + + irq_set_chip_and_handler(irq, &edac->irq_chip, handle_simple_irq); + irq_set_chip_data(irq, edac); + irq_set_noprobe(irq); + + return 0; +} + +static const struct irq_domain_ops s10_eccmgr_ic_ops = { + .map = s10_eccmgr_irqdomain_map, + .xlate = irq_domain_xlate_twocell, +}; + +static int altr_edac_s10_probe(struct platform_device *pdev) +{ + struct altr_stratix10_edac *edac; + struct device_node *child; + int dberror, err_addr; + + edac = devm_kzalloc(&pdev->dev, sizeof(*edac), GFP_KERNEL); + if (!edac) + return -ENOMEM; + + edac->dev = &pdev->dev; + platform_set_drvdata(pdev, edac); + INIT_LIST_HEAD(&edac->s10_ecc_devices); + + edac->irq_chip.name = pdev->dev.of_node->name; + edac->irq_chip.irq_mask = s10_eccmgr_irq_mask; + edac->irq_chip.irq_unmask = s10_eccmgr_irq_unmask; + edac->domain = irq_domain_add_linear(pdev->dev.of_node, 64, + &s10_eccmgr_ic_ops, edac); + if (!edac->domain) { + dev_err(&pdev->dev, "Error adding IRQ domain\n"); + return -ENOMEM; + } + + edac->sb_irq = platform_get_irq(pdev, 0); + if (edac->sb_irq < 0) { + dev_err(&pdev->dev, "No SBERR IRQ resource\n"); + return edac->sb_irq; + } + + irq_set_chained_handler_and_data(edac->sb_irq, + altr_edac_s10_irq_handler, + edac); + + edac->panic_notifier.notifier_call = s10_edac_dberr_handler; + atomic_notifier_chain_register(&panic_notifier_list, + &edac->panic_notifier); + + /* Printout a message if uncorrectable error previously. */ + s10_protected_reg_read(edac, S10_SYSMGR_UE_VAL_OFST, &dberror); + if (dberror) { + s10_protected_reg_read(edac, S10_SYSMGR_UE_ADDR_OFST, + &err_addr); + edac_printk(KERN_ERR, EDAC_DEVICE, + "Previous Boot UE detected[0x%X] @ 0x%X\n", + dberror, err_addr); + /* Reset the sticky registers */ + s10_protected_reg_write(edac, S10_SYSMGR_UE_VAL_OFST, 0); + s10_protected_reg_write(edac, S10_SYSMGR_UE_ADDR_OFST, 0); + } + + for_each_child_of_node(pdev->dev.of_node, child) { + if (!of_device_is_available(child)) + continue; + + if (of_device_is_compatible(child, "altr,sdram-edac-s10")) + of_platform_populate(pdev->dev.of_node, + altr_sdram_ctrl_of_match, + NULL, &pdev->dev); + } + + return 0; +} + +static const struct of_device_id altr_edac_s10_of_match[] = { + { .compatible = "altr,socfpga-s10-ecc-manager" }, + {}, +}; +MODULE_DEVICE_TABLE(of, altr_edac_s10_of_match); + +static struct platform_driver altr_edac_s10_driver = { + .probe = altr_edac_s10_probe, + .driver = { + .name = "socfpga_s10_ecc_manager", + .of_match_table = altr_edac_s10_of_match, + }, +}; +module_platform_driver(altr_edac_s10_driver); + MODULE_LICENSE("GPL v2"); MODULE_AUTHOR("Thor Thayer"); MODULE_DESCRIPTION("EDAC Driver for Altera Memories"); diff --git a/drivers/edac/altera_edac.h b/drivers/edac/altera_edac.h index cbc96290f743..81f0554e09de 100644 --- a/drivers/edac/altera_edac.h +++ b/drivers/edac/altera_edac.h @@ -1,23 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* - * + * Copyright (C) 2017-2018, Intel Corporation * Copyright (C) 2015 Altera Corporation - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #ifndef _ALTERA_EDAC_H #define _ALTERA_EDAC_H +#include #include #include @@ -94,6 +84,7 @@ /* SDRAM Controller Address Width Register */ #define CV_DRAMADDRW 0xFFC2502C #define A10_DRAMADDRW 0xFFCFA0A8 +#define S10_DRAMADDRW 0xF80110E0 /* SDRAM Controller Address Widths Field Register */ #define DRAMADDRW_COLBIT_MASK 0x001F @@ -115,6 +106,7 @@ /* SDRAM Controller Interface Data Width Register */ #define CV_DRAMIFWIDTH 0xFFC25030 #define A10_DRAMIFWIDTH 0xFFCFB008 +#define S10_DRAMIFWIDTH 0xF8011008 /* SDRAM Controller Interface Data Width Defines */ #define CV_DRAMIFWIDTH_16B_ECC 24 @@ -164,6 +156,34 @@ #define A10_INTMASK_CLR_OFST 0x10 #define A10_DDR0_IRQ_MASK BIT(17) +/************* Stratix10 Defines **************/ + +/* SDRAM Controller EccCtrl Register */ +#define S10_ECCCTRL1_OFST 0xF8011100 + +/* SDRAM Controller DRAM IRQ Register */ +#define S10_ERRINTEN_OFST 0xF8011110 + +/* SDRAM Interrupt Mode Register */ +#define S10_INTMODE_OFST 0xF801111C + +/* SDRAM Controller Error Status Register */ +#define S10_INTSTAT_OFST 0xF8011120 + +/* SDRAM Controller ECC Error Address Register */ +#define S10_DERRADDR_OFST 0xF801112C +#define S10_SERRADDR_OFST 0xF8011130 + +/* SDRAM Controller ECC Diagnostic Register */ +#define S10_DIAGINTTEST_OFST 0xF8011124 + +/* SDRAM Single Bit Error Count Compare Set Register */ +#define S10_SERRCNTREG_OFST 0xF801113C + +/* Sticky registers for Uncorrected Errors */ +#define S10_SYSMGR_UE_VAL_OFST 0xFFD12220 +#define S10_SYSMGR_UE_ADDR_OFST 0xFFD12224 + struct altr_sdram_prv_data { int ecc_ctrl_offset; int ecc_ctl_en_mask; @@ -296,6 +316,18 @@ struct altr_sdram_mc_data { /* A10 ECC Controller memory initialization timeout */ #define ALTR_A10_ECC_INIT_WATCHDOG_10US 10000 +/************* Stratix10 Defines **************/ + +/* Stratix10 ECC Manager Defines */ +#define S10_SYSMGR_ECC_INTMASK_VAL_OFST 0xFFD12090 +#define S10_SYSMGR_ECC_INTMASK_SET_OFST 0xFFD12094 +#define S10_SYSMGR_ECC_INTMASK_CLR_OFST 0xFFD12098 + +#define S10_SYSMGR_ECC_INTSTAT_SERR_OFST 0xFFD1209C +#define S10_SYSMGR_ECC_INTSTAT_DERR_OFST 0xFFD120A0 + +#define S10_DDR0_IRQ_MASK BIT(16) + struct altr_edac_device_dev; struct edac_device_prv_data { @@ -340,4 +372,78 @@ struct altr_arria10_edac { struct list_head a10_ecc_devices; }; +/* + * Functions specified by ARM SMC Calling convention: + * + * FAST call executes atomic operations, returns when the requested operation + * has completed. + * STD call starts a operation which can be preempted by a non-secure + * interrupt. The call can return before the requested operation has + * completed. + * + * a0..a7 is used as register names in the descriptions below, on arm32 + * that translates to r0..r7 and on arm64 to w0..w7. + */ + +#define INTEL_SIP_SMC_STD_CALL_VAL(func_num) \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_STD_CALL, ARM_SMCCC_SMC_64, \ + ARM_SMCCC_OWNER_SIP, (func_num)) + +#define INTEL_SIP_SMC_FAST_CALL_VAL(func_num) \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, ARM_SMCCC_SMC_64, \ + ARM_SMCCC_OWNER_SIP, (func_num)) + +#define INTEL_SIP_SMC_RETURN_UNKNOWN_FUNCTION 0xFFFFFFFF +#define INTEL_SIP_SMC_STATUS_OK 0x0 +#define INTEL_SIP_SMC_REG_ERROR 0x5 + +/* + * Request INTEL_SIP_SMC_REG_READ + * + * Read a protected register using SMCCC + * + * Call register usage: + * a0: INTEL_SIP_SMC_REG_READ. + * a1: register address. + * a2-7: not used. + * + * Return status: + * a0: INTEL_SIP_SMC_STATUS_OK, INTEL_SIP_SMC_REG_ERROR, or + * INTEL_SIP_SMC_RETURN_UNKNOWN_FUNCTION + * a1: Value in the register + * a2-3: not used. + */ +#define INTEL_SIP_SMC_FUNCID_REG_READ 7 +#define INTEL_SIP_SMC_REG_READ \ + INTEL_SIP_SMC_FAST_CALL_VAL(INTEL_SIP_SMC_FUNCID_REG_READ) + +/* + * Request INTEL_SIP_SMC_REG_WRITE + * + * Write a protected register using SMCCC + * + * Call register usage: + * a0: INTEL_SIP_SMC_REG_WRITE. + * a1: register address + * a2: value to program into register. + * a3-7: not used. + * + * Return status: + * a0: INTEL_SIP_SMC_STATUS_OK, INTEL_SIP_SMC_REG_ERROR, or + * INTEL_SIP_SMC_RETURN_UNKNOWN_FUNCTION + * a1-3: not used. + */ +#define INTEL_SIP_SMC_FUNCID_REG_WRITE 8 +#define INTEL_SIP_SMC_REG_WRITE \ + INTEL_SIP_SMC_FAST_CALL_VAL(INTEL_SIP_SMC_FUNCID_REG_WRITE) + +struct altr_stratix10_edac { + struct device *dev; + int sb_irq; + struct irq_domain *domain; + struct irq_chip irq_chip; + struct list_head s10_ecc_devices; + struct notifier_block panic_notifier; +}; + #endif /* #ifndef _ALTERA_EDAC_H */ diff --git a/drivers/edac/amd64_edac.c b/drivers/edac/amd64_edac.c index 329cb96f886f..18aeabb1d5ee 100644 --- a/drivers/edac/amd64_edac.c +++ b/drivers/edac/amd64_edac.c @@ -3451,7 +3451,7 @@ static int __init amd64_edac_init(void) opstate_init(); err = -ENOMEM; - ecc_stngs = kzalloc(amd_nb_num() * sizeof(ecc_stngs[0]), GFP_KERNEL); + ecc_stngs = kcalloc(amd_nb_num(), sizeof(ecc_stngs[0]), GFP_KERNEL); if (!ecc_stngs) goto err_free; diff --git a/drivers/edac/ghes_edac.c b/drivers/edac/ghes_edac.c index 68b6ee18bea6..473aeec4b1da 100644 --- a/drivers/edac/ghes_edac.c +++ b/drivers/edac/ghes_edac.c @@ -91,6 +91,7 @@ static void ghes_edac_dmidecode(const struct dmi_header *dh, void *arg) struct dimm_info *dimm = EDAC_DIMM_PTR(mci->layers, mci->dimms, mci->n_layers, dimm_fill->count, 0, 0); + u16 rdr_mask = BIT(7) | BIT(13); if (entry->size == 0xffff) { pr_info("Can't get DIMM%i size\n", @@ -99,22 +100,21 @@ static void ghes_edac_dmidecode(const struct dmi_header *dh, void *arg) } else if (entry->size == 0x7fff) { dimm->nr_pages = MiB_TO_PAGES(entry->extended_size); } else { - if (entry->size & 1 << 15) - dimm->nr_pages = MiB_TO_PAGES((entry->size & - 0x7fff) << 10); + if (entry->size & BIT(15)) + dimm->nr_pages = MiB_TO_PAGES((entry->size & 0x7fff) << 10); else dimm->nr_pages = MiB_TO_PAGES(entry->size); } switch (entry->memory_type) { case 0x12: - if (entry->type_detail & 1 << 13) + if (entry->type_detail & BIT(13)) dimm->mtype = MEM_RDDR; else dimm->mtype = MEM_DDR; break; case 0x13: - if (entry->type_detail & 1 << 13) + if (entry->type_detail & BIT(13)) dimm->mtype = MEM_RDDR2; else dimm->mtype = MEM_DDR2; @@ -123,20 +123,29 @@ static void ghes_edac_dmidecode(const struct dmi_header *dh, void *arg) dimm->mtype = MEM_FB_DDR2; break; case 0x18: - if (entry->type_detail & 1 << 13) + if (entry->type_detail & BIT(12)) + dimm->mtype = MEM_NVDIMM; + else if (entry->type_detail & BIT(13)) dimm->mtype = MEM_RDDR3; else dimm->mtype = MEM_DDR3; break; + case 0x1a: + if (entry->type_detail & BIT(12)) + dimm->mtype = MEM_NVDIMM; + else if (entry->type_detail & BIT(13)) + dimm->mtype = MEM_RDDR4; + else + dimm->mtype = MEM_DDR4; + break; default: - if (entry->type_detail & 1 << 6) + if (entry->type_detail & BIT(6)) dimm->mtype = MEM_RMBS; - else if ((entry->type_detail & ((1 << 7) | (1 << 13))) - == ((1 << 7) | (1 << 13))) + else if ((entry->type_detail & rdr_mask) == rdr_mask) dimm->mtype = MEM_RDR; - else if (entry->type_detail & 1 << 7) + else if (entry->type_detail & BIT(7)) dimm->mtype = MEM_SDR; - else if (entry->type_detail & 1 << 9) + else if (entry->type_detail & BIT(9)) dimm->mtype = MEM_EDO; else dimm->mtype = MEM_UNKNOWN; @@ -172,8 +181,7 @@ static void ghes_edac_dmidecode(const struct dmi_header *dh, void *arg) } } -void ghes_edac_report_mem_error(struct ghes *ghes, int sev, - struct cper_sec_mem_err *mem_err) +void ghes_edac_report_mem_error(int sev, struct cper_sec_mem_err *mem_err) { enum hw_event_mc_err_type type; struct edac_raw_error_desc *e; @@ -183,10 +191,8 @@ void ghes_edac_report_mem_error(struct ghes *ghes, int sev, char *p; u8 grain_bits; - if (!pvt) { - pr_err("Internal error: Can't find EDAC structure\n"); + if (!pvt) return; - } /* * We can do the locking below because GHES defers error processing @@ -434,12 +440,16 @@ int ghes_edac_register(struct ghes *ghes, struct device *dev) struct mem_ctl_info *mci; struct edac_mc_layer layers[1]; struct ghes_edac_dimm_fill dimm_fill; - int idx; + int idx = -1; - /* Check if safe to enable on this system */ - idx = acpi_match_platform_list(plat_list); - if (!force_load && idx < 0) - return 0; + if (IS_ENABLED(CONFIG_X86)) { + /* Check if safe to enable on this system */ + idx = acpi_match_platform_list(plat_list); + if (!force_load && idx < 0) + return -ENODEV; + } else { + idx = 0; + } /* * We have only one logical memory controller to which all DIMMs belong. @@ -519,6 +529,9 @@ void ghes_edac_unregister(struct ghes *ghes) { struct mem_ctl_info *mci; + if (!ghes_pvt) + return; + mci = ghes_pvt->mci; edac_mc_del_mc(mci->pdev); edac_mc_free(mci); diff --git a/drivers/edac/i7core_edac.c b/drivers/edac/i7core_edac.c index 8c5540160a23..8ed4dd9c571b 100644 --- a/drivers/edac/i7core_edac.c +++ b/drivers/edac/i7core_edac.c @@ -461,7 +461,7 @@ static struct i7core_dev *alloc_i7core_dev(u8 socket, if (!i7core_dev) return NULL; - i7core_dev->pdev = kzalloc(sizeof(*i7core_dev->pdev) * table->n_devs, + i7core_dev->pdev = kcalloc(table->n_devs, sizeof(*i7core_dev->pdev), GFP_KERNEL); if (!i7core_dev->pdev) { kfree(i7core_dev); @@ -1743,7 +1743,7 @@ static void i7core_mce_output_error(struct mem_ctl_info *mci, err = "write parity error"; break; case 19: - err = "redundacy loss"; + err = "redundancy loss"; break; case 20: err = "reserved"; diff --git a/drivers/extcon/extcon.c b/drivers/extcon/extcon.c index 8bff5fd18185..af83ad58819c 100644 --- a/drivers/extcon/extcon.c +++ b/drivers/extcon/extcon.c @@ -1126,8 +1126,9 @@ int extcon_dev_register(struct extcon_dev *edev) char *str; struct extcon_cable *cable; - edev->cables = kzalloc(sizeof(struct extcon_cable) * - edev->max_supported, GFP_KERNEL); + edev->cables = kcalloc(edev->max_supported, + sizeof(struct extcon_cable), + GFP_KERNEL); if (!edev->cables) { ret = -ENOMEM; goto err_sysfs_alloc; @@ -1136,7 +1137,7 @@ int extcon_dev_register(struct extcon_dev *edev) cable = &edev->cables[index]; snprintf(buf, 10, "cable.%d", index); - str = kzalloc(sizeof(char) * (strlen(buf) + 1), + str = kzalloc(strlen(buf) + 1, GFP_KERNEL); if (!str) { for (index--; index >= 0; index--) { @@ -1177,15 +1178,17 @@ int extcon_dev_register(struct extcon_dev *edev) for (index = 0; edev->mutually_exclusive[index]; index++) ; - edev->attrs_muex = kzalloc(sizeof(struct attribute *) * - (index + 1), GFP_KERNEL); + edev->attrs_muex = kcalloc(index + 1, + sizeof(struct attribute *), + GFP_KERNEL); if (!edev->attrs_muex) { ret = -ENOMEM; goto err_muex; } - edev->d_attrs_muex = kzalloc(sizeof(struct device_attribute) * - index, GFP_KERNEL); + edev->d_attrs_muex = kcalloc(index, + sizeof(struct device_attribute), + GFP_KERNEL); if (!edev->d_attrs_muex) { ret = -ENOMEM; kfree(edev->attrs_muex); @@ -1194,7 +1197,7 @@ int extcon_dev_register(struct extcon_dev *edev) for (index = 0; edev->mutually_exclusive[index]; index++) { sprintf(buf, "0x%x", edev->mutually_exclusive[index]); - name = kzalloc(sizeof(char) * (strlen(buf) + 1), + name = kzalloc(strlen(buf) + 1, GFP_KERNEL); if (!name) { for (index--; index >= 0; index--) { @@ -1220,8 +1223,9 @@ int extcon_dev_register(struct extcon_dev *edev) if (edev->max_supported) { edev->extcon_dev_type.groups = - kzalloc(sizeof(struct attribute_group *) * - (edev->max_supported + 2), GFP_KERNEL); + kcalloc(edev->max_supported + 2, + sizeof(struct attribute_group *), + GFP_KERNEL); if (!edev->extcon_dev_type.groups) { ret = -ENOMEM; goto err_alloc_groups; diff --git a/drivers/firewire/core-iso.c b/drivers/firewire/core-iso.c index 38c0aa60b2cb..051327a951b1 100644 --- a/drivers/firewire/core-iso.c +++ b/drivers/firewire/core-iso.c @@ -45,8 +45,8 @@ int fw_iso_buffer_alloc(struct fw_iso_buffer *buffer, int page_count) buffer->page_count = 0; buffer->page_count_mapped = 0; - buffer->pages = kmalloc(page_count * sizeof(buffer->pages[0]), - GFP_KERNEL); + buffer->pages = kmalloc_array(page_count, sizeof(buffer->pages[0]), + GFP_KERNEL); if (buffer->pages == NULL) return -ENOMEM; diff --git a/drivers/firewire/core-topology.c b/drivers/firewire/core-topology.c index 939d259ddf19..7db234d3fbdd 100644 --- a/drivers/firewire/core-topology.c +++ b/drivers/firewire/core-topology.c @@ -112,8 +112,7 @@ static struct fw_node *fw_node_create(u32 sid, int port_count, int color) { struct fw_node *node; - node = kzalloc(sizeof(*node) + port_count * sizeof(node->ports[0]), - GFP_ATOMIC); + node = kzalloc(struct_size(node, ports, port_count), GFP_ATOMIC); if (node == NULL) return NULL; diff --git a/drivers/firewire/net.c b/drivers/firewire/net.c index 60e75e6d9104..82ba110d9d1a 100644 --- a/drivers/firewire/net.c +++ b/drivers/firewire/net.c @@ -1121,7 +1121,7 @@ static int fwnet_broadcast_start(struct fwnet_device *dev) max_receive = 1U << (dev->card->max_receive + 1); num_packets = (FWNET_ISO_PAGE_COUNT * PAGE_SIZE) / max_receive; - ptrptr = kmalloc(sizeof(void *) * num_packets, GFP_KERNEL); + ptrptr = kmalloc_array(num_packets, sizeof(void *), GFP_KERNEL); if (!ptrptr) { retval = -ENOMEM; goto failed; diff --git a/drivers/firmware/arm_scmi/base.c b/drivers/firmware/arm_scmi/base.c index 0d3806c0d432..9dff33ea6416 100644 --- a/drivers/firmware/arm_scmi/base.c +++ b/drivers/firmware/arm_scmi/base.c @@ -26,7 +26,7 @@ struct scmi_msg_resp_base_attributes { * scmi_base_attributes_get() - gets the implementation details * that are associated with the base protocol. * - * @handle - SCMI entity handle + * @handle: SCMI entity handle * * Return: 0 on success, else appropriate SCMI error. */ @@ -37,7 +37,7 @@ static int scmi_base_attributes_get(const struct scmi_handle *handle) struct scmi_msg_resp_base_attributes *attr_info; struct scmi_revision_info *rev = handle->version; - ret = scmi_one_xfer_init(handle, PROTOCOL_ATTRIBUTES, + ret = scmi_xfer_get_init(handle, PROTOCOL_ATTRIBUTES, SCMI_PROTOCOL_BASE, 0, sizeof(*attr_info), &t); if (ret) return ret; @@ -49,15 +49,16 @@ static int scmi_base_attributes_get(const struct scmi_handle *handle) rev->num_agents = attr_info->num_agents; } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); + return ret; } /** * scmi_base_vendor_id_get() - gets vendor/subvendor identifier ASCII string. * - * @handle - SCMI entity handle - * @sub_vendor - specify true if sub-vendor ID is needed + * @handle: SCMI entity handle + * @sub_vendor: specify true if sub-vendor ID is needed * * Return: 0 on success, else appropriate SCMI error. */ @@ -80,7 +81,7 @@ scmi_base_vendor_id_get(const struct scmi_handle *handle, bool sub_vendor) size = ARRAY_SIZE(rev->vendor_id); } - ret = scmi_one_xfer_init(handle, cmd, SCMI_PROTOCOL_BASE, 0, size, &t); + ret = scmi_xfer_get_init(handle, cmd, SCMI_PROTOCOL_BASE, 0, size, &t); if (ret) return ret; @@ -88,7 +89,8 @@ scmi_base_vendor_id_get(const struct scmi_handle *handle, bool sub_vendor) if (!ret) memcpy(vendor_id, t->rx.buf, size); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); + return ret; } @@ -97,7 +99,7 @@ scmi_base_vendor_id_get(const struct scmi_handle *handle, bool sub_vendor) * implementation 32-bit version. The format of the version number is * vendor-specific * - * @handle - SCMI entity handle + * @handle: SCMI entity handle * * Return: 0 on success, else appropriate SCMI error. */ @@ -109,7 +111,7 @@ scmi_base_implementation_version_get(const struct scmi_handle *handle) struct scmi_xfer *t; struct scmi_revision_info *rev = handle->version; - ret = scmi_one_xfer_init(handle, BASE_DISCOVER_IMPLEMENT_VERSION, + ret = scmi_xfer_get_init(handle, BASE_DISCOVER_IMPLEMENT_VERSION, SCMI_PROTOCOL_BASE, 0, sizeof(*impl_ver), &t); if (ret) return ret; @@ -120,7 +122,8 @@ scmi_base_implementation_version_get(const struct scmi_handle *handle) rev->impl_ver = le32_to_cpu(*impl_ver); } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); + return ret; } @@ -128,8 +131,8 @@ scmi_base_implementation_version_get(const struct scmi_handle *handle) * scmi_base_implementation_list_get() - gets the list of protocols it is * OSPM is allowed to access * - * @handle - SCMI entity handle - * @protocols_imp - pointer to hold the list of protocol identifiers + * @handle: SCMI entity handle + * @protocols_imp: pointer to hold the list of protocol identifiers * * Return: 0 on success, else appropriate SCMI error. */ @@ -143,7 +146,7 @@ static int scmi_base_implementation_list_get(const struct scmi_handle *handle, u32 tot_num_ret = 0, loop_num_ret; struct device *dev = handle->dev; - ret = scmi_one_xfer_init(handle, BASE_DISCOVER_LIST_PROTOCOLS, + ret = scmi_xfer_get_init(handle, BASE_DISCOVER_LIST_PROTOCOLS, SCMI_PROTOCOL_BASE, sizeof(*num_skip), 0, &t); if (ret) return ret; @@ -172,16 +175,17 @@ static int scmi_base_implementation_list_get(const struct scmi_handle *handle, tot_num_ret += loop_num_ret; } while (loop_num_ret); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); + return ret; } /** * scmi_base_discover_agent_get() - discover the name of an agent * - * @handle - SCMI entity handle - * @id - Agent identifier - * @name - Agent identifier ASCII string + * @handle: SCMI entity handle + * @id: Agent identifier + * @name: Agent identifier ASCII string * * An agent id of 0 is reserved to identify the platform itself. * Generally operating system is represented as "OSPM" @@ -194,7 +198,7 @@ static int scmi_base_discover_agent_get(const struct scmi_handle *handle, int ret; struct scmi_xfer *t; - ret = scmi_one_xfer_init(handle, BASE_DISCOVER_AGENT, + ret = scmi_xfer_get_init(handle, BASE_DISCOVER_AGENT, SCMI_PROTOCOL_BASE, sizeof(__le32), SCMI_MAX_STR_SIZE, &t); if (ret) @@ -206,7 +210,8 @@ static int scmi_base_discover_agent_get(const struct scmi_handle *handle, if (!ret) memcpy(name, t->rx.buf, SCMI_MAX_STR_SIZE); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); + return ret; } diff --git a/drivers/firmware/arm_scmi/bus.c b/drivers/firmware/arm_scmi/bus.c index f2760a596c28..472c88ae1c0f 100644 --- a/drivers/firmware/arm_scmi/bus.c +++ b/drivers/firmware/arm_scmi/bus.c @@ -125,13 +125,13 @@ scmi_device_create(struct device_node *np, struct device *parent, int protocol) int id, retval; struct scmi_device *scmi_dev; - id = ida_simple_get(&scmi_bus_id, 1, 0, GFP_KERNEL); - if (id < 0) - return NULL; - scmi_dev = kzalloc(sizeof(*scmi_dev), GFP_KERNEL); if (!scmi_dev) - goto no_mem; + return NULL; + + id = ida_simple_get(&scmi_bus_id, 1, 0, GFP_KERNEL); + if (id < 0) + goto free_mem; scmi_dev->id = id; scmi_dev->protocol_id = protocol; @@ -141,13 +141,15 @@ scmi_device_create(struct device_node *np, struct device *parent, int protocol) dev_set_name(&scmi_dev->dev, "scmi_dev.%d", id); retval = device_register(&scmi_dev->dev); - if (!retval) - return scmi_dev; + if (retval) + goto put_dev; + return scmi_dev; +put_dev: put_device(&scmi_dev->dev); - kfree(scmi_dev); -no_mem: ida_simple_remove(&scmi_bus_id, id); +free_mem: + kfree(scmi_dev); return NULL; } @@ -171,9 +173,9 @@ int scmi_protocol_register(int protocol_id, scmi_prot_init_fn_t fn) spin_lock(&protocol_lock); ret = idr_alloc(&scmi_protocols, fn, protocol_id, protocol_id + 1, GFP_ATOMIC); + spin_unlock(&protocol_lock); if (ret != protocol_id) pr_err("unable to allocate SCMI idr slot, err %d\n", ret); - spin_unlock(&protocol_lock); return ret; } diff --git a/drivers/firmware/arm_scmi/clock.c b/drivers/firmware/arm_scmi/clock.c index 2b90606452a2..e4119eb34986 100644 --- a/drivers/firmware/arm_scmi/clock.c +++ b/drivers/firmware/arm_scmi/clock.c @@ -77,7 +77,7 @@ static int scmi_clock_protocol_attributes_get(const struct scmi_handle *handle, struct scmi_xfer *t; struct scmi_msg_resp_clock_protocol_attributes *attr; - ret = scmi_one_xfer_init(handle, PROTOCOL_ATTRIBUTES, + ret = scmi_xfer_get_init(handle, PROTOCOL_ATTRIBUTES, SCMI_PROTOCOL_CLOCK, 0, sizeof(*attr), &t); if (ret) return ret; @@ -90,7 +90,7 @@ static int scmi_clock_protocol_attributes_get(const struct scmi_handle *handle, ci->max_async_req = attr->max_async_req; } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -101,7 +101,7 @@ static int scmi_clock_attributes_get(const struct scmi_handle *handle, struct scmi_xfer *t; struct scmi_msg_resp_clock_attributes *attr; - ret = scmi_one_xfer_init(handle, CLOCK_ATTRIBUTES, SCMI_PROTOCOL_CLOCK, + ret = scmi_xfer_get_init(handle, CLOCK_ATTRIBUTES, SCMI_PROTOCOL_CLOCK, sizeof(clk_id), sizeof(*attr), &t); if (ret) return ret; @@ -115,7 +115,7 @@ static int scmi_clock_attributes_get(const struct scmi_handle *handle, else clk->name[0] = '\0'; - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -132,7 +132,7 @@ scmi_clock_describe_rates_get(const struct scmi_handle *handle, u32 clk_id, struct scmi_msg_clock_describe_rates *clk_desc; struct scmi_msg_resp_clock_describe_rates *rlist; - ret = scmi_one_xfer_init(handle, CLOCK_DESCRIBE_RATES, + ret = scmi_xfer_get_init(handle, CLOCK_DESCRIBE_RATES, SCMI_PROTOCOL_CLOCK, sizeof(*clk_desc), 0, &t); if (ret) return ret; @@ -186,7 +186,7 @@ scmi_clock_describe_rates_get(const struct scmi_handle *handle, u32 clk_id, clk->list.num_rates = tot_rate_cnt; err: - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -196,7 +196,7 @@ scmi_clock_rate_get(const struct scmi_handle *handle, u32 clk_id, u64 *value) int ret; struct scmi_xfer *t; - ret = scmi_one_xfer_init(handle, CLOCK_RATE_GET, SCMI_PROTOCOL_CLOCK, + ret = scmi_xfer_get_init(handle, CLOCK_RATE_GET, SCMI_PROTOCOL_CLOCK, sizeof(__le32), sizeof(u64), &t); if (ret) return ret; @@ -211,7 +211,7 @@ scmi_clock_rate_get(const struct scmi_handle *handle, u32 clk_id, u64 *value) *value |= (u64)le32_to_cpu(*(pval + 1)) << 32; } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -222,7 +222,7 @@ static int scmi_clock_rate_set(const struct scmi_handle *handle, u32 clk_id, struct scmi_xfer *t; struct scmi_clock_set_rate *cfg; - ret = scmi_one_xfer_init(handle, CLOCK_RATE_SET, SCMI_PROTOCOL_CLOCK, + ret = scmi_xfer_get_init(handle, CLOCK_RATE_SET, SCMI_PROTOCOL_CLOCK, sizeof(*cfg), 0, &t); if (ret) return ret; @@ -235,7 +235,7 @@ static int scmi_clock_rate_set(const struct scmi_handle *handle, u32 clk_id, ret = scmi_do_xfer(handle, t); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -246,7 +246,7 @@ scmi_clock_config_set(const struct scmi_handle *handle, u32 clk_id, u32 config) struct scmi_xfer *t; struct scmi_clock_set_config *cfg; - ret = scmi_one_xfer_init(handle, CLOCK_CONFIG_SET, SCMI_PROTOCOL_CLOCK, + ret = scmi_xfer_get_init(handle, CLOCK_CONFIG_SET, SCMI_PROTOCOL_CLOCK, sizeof(*cfg), 0, &t); if (ret) return ret; @@ -257,7 +257,7 @@ scmi_clock_config_set(const struct scmi_handle *handle, u32 clk_id, u32 config) ret = scmi_do_xfer(handle, t); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h index 0c30234f9098..937a930ce87d 100644 --- a/drivers/firmware/arm_scmi/common.h +++ b/drivers/firmware/arm_scmi/common.h @@ -7,6 +7,7 @@ * Copyright (C) 2018 ARM Ltd. */ +#include #include #include #include @@ -14,10 +15,10 @@ #include #include -#define PROTOCOL_REV_MINOR_BITS 16 -#define PROTOCOL_REV_MINOR_MASK ((1U << PROTOCOL_REV_MINOR_BITS) - 1) -#define PROTOCOL_REV_MAJOR(x) ((x) >> PROTOCOL_REV_MINOR_BITS) -#define PROTOCOL_REV_MINOR(x) ((x) & PROTOCOL_REV_MINOR_MASK) +#define PROTOCOL_REV_MINOR_MASK GENMASK(15, 0) +#define PROTOCOL_REV_MAJOR_MASK GENMASK(31, 16) +#define PROTOCOL_REV_MAJOR(x) (u16)(FIELD_GET(PROTOCOL_REV_MAJOR_MASK, (x))) +#define PROTOCOL_REV_MINOR(x) (u16)(FIELD_GET(PROTOCOL_REV_MINOR_MASK, (x))) #define MAX_PROTOCOLS_IMP 16 #define MAX_OPPS 16 @@ -50,8 +51,11 @@ struct scmi_msg_resp_prot_version { * @id: The identifier of the command being sent * @protocol_id: The identifier of the protocol used to send @id command * @seq: The token to identify the message. when a message/command returns, - * the platform returns the whole message header unmodified including - * the token. + * the platform returns the whole message header unmodified including + * the token + * @status: Status of the transfer once it's complete + * @poll_completion: Indicate if the transfer needs to be polled for + * completion or interrupt mode is used */ struct scmi_msg_hdr { u8 id; @@ -82,18 +86,16 @@ struct scmi_msg { * buffer for the rx path as we use for the tx path. * @done: completion event */ - struct scmi_xfer { - void *con_priv; struct scmi_msg_hdr hdr; struct scmi_msg tx; struct scmi_msg rx; struct completion done; }; -void scmi_one_xfer_put(const struct scmi_handle *h, struct scmi_xfer *xfer); +void scmi_xfer_put(const struct scmi_handle *h, struct scmi_xfer *xfer); int scmi_do_xfer(const struct scmi_handle *h, struct scmi_xfer *xfer); -int scmi_one_xfer_init(const struct scmi_handle *h, u8 msg_id, u8 prot_id, +int scmi_xfer_get_init(const struct scmi_handle *h, u8 msg_id, u8 prot_id, size_t tx_size, size_t rx_size, struct scmi_xfer **p); int scmi_handle_put(const struct scmi_handle *handle); struct scmi_handle *scmi_handle_get(struct device *dev); diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c index 2455be8cbc4f..8f952f2f1a29 100644 --- a/drivers/firmware/arm_scmi/driver.c +++ b/drivers/firmware/arm_scmi/driver.c @@ -29,16 +29,12 @@ #include "common.h" -#define MSG_ID_SHIFT 0 -#define MSG_ID_MASK 0xff -#define MSG_TYPE_SHIFT 8 -#define MSG_TYPE_MASK 0x3 -#define MSG_PROTOCOL_ID_SHIFT 10 -#define MSG_PROTOCOL_ID_MASK 0xff -#define MSG_TOKEN_ID_SHIFT 18 -#define MSG_TOKEN_ID_MASK 0x3ff -#define MSG_XTRACT_TOKEN(header) \ - (((header) >> MSG_TOKEN_ID_SHIFT) & MSG_TOKEN_ID_MASK) +#define MSG_ID_MASK GENMASK(7, 0) +#define MSG_TYPE_MASK GENMASK(9, 8) +#define MSG_PROTOCOL_ID_MASK GENMASK(17, 10) +#define MSG_TOKEN_ID_MASK GENMASK(27, 18) +#define MSG_XTRACT_TOKEN(hdr) FIELD_GET(MSG_TOKEN_ID_MASK, (hdr)) +#define MSG_TOKEN_MAX (MSG_XTRACT_TOKEN(MSG_TOKEN_ID_MASK) + 1) enum scmi_error_codes { SCMI_SUCCESS = 0, /* Success */ @@ -55,7 +51,7 @@ enum scmi_error_codes { SCMI_ERR_MAX }; -/* List of all SCMI devices active in system */ +/* List of all SCMI devices active in system */ static LIST_HEAD(scmi_list); /* Protection for the entire list */ static DEFINE_MUTEX(scmi_list_mutex); @@ -72,7 +68,6 @@ static DEFINE_MUTEX(scmi_list_mutex); struct scmi_xfers_info { struct scmi_xfer *xfer_block; unsigned long *xfer_alloc_table; - /* protect transfer allocation */ spinlock_t xfer_lock; }; @@ -98,6 +93,7 @@ struct scmi_desc { * @payload: Transmit/Receive mailbox channel payload area * @dev: Reference to device in the SCMI hierarchy corresponding to this * channel + * @handle: Pointer to SCMI entity handle */ struct scmi_chan_info { struct mbox_client cl; @@ -108,7 +104,7 @@ struct scmi_chan_info { }; /** - * struct scmi_info - Structure representing a SCMI instance + * struct scmi_info - Structure representing a SCMI instance * * @dev: Device pointer * @desc: SoC description for this instance @@ -117,9 +113,9 @@ struct scmi_chan_info { * implementation version and (sub-)vendor identification. * @minfo: Message info * @tx_idr: IDR object to map protocol id to channel info pointer - * @protocols_imp: list of protocols implemented, currently maximum of + * @protocols_imp: List of protocols implemented, currently maximum of * MAX_PROTOCOLS_IMP elements allocated by the base protocol - * @node: list head + * @node: List head * @users: Number of users of this instance */ struct scmi_info { @@ -225,9 +221,7 @@ static void scmi_rx_callback(struct mbox_client *cl, void *m) xfer_id = MSG_XTRACT_TOKEN(ioread32(&mem->msg_header)); - /* - * Are we even expecting this? - */ + /* Are we even expecting this? */ if (!test_bit(xfer_id, minfo->xfer_alloc_table)) { dev_err(dev, "message for %d is not expected!\n", xfer_id); return; @@ -252,12 +246,14 @@ static void scmi_rx_callback(struct mbox_client *cl, void *m) * * @hdr: pointer to header containing all the information on message id, * protocol id and sequence id. + * + * Return: 32-bit packed command header to be sent to the platform. */ static inline u32 pack_scmi_header(struct scmi_msg_hdr *hdr) { - return ((hdr->id & MSG_ID_MASK) << MSG_ID_SHIFT) | - ((hdr->seq & MSG_TOKEN_ID_MASK) << MSG_TOKEN_ID_SHIFT) | - ((hdr->protocol_id & MSG_PROTOCOL_ID_MASK) << MSG_PROTOCOL_ID_SHIFT); + return FIELD_PREP(MSG_ID_MASK, hdr->id) | + FIELD_PREP(MSG_TOKEN_ID_MASK, hdr->seq) | + FIELD_PREP(MSG_PROTOCOL_ID_MASK, hdr->protocol_id); } /** @@ -286,9 +282,9 @@ static void scmi_tx_prepare(struct mbox_client *cl, void *m) } /** - * scmi_one_xfer_get() - Allocate one message + * scmi_xfer_get() - Allocate one message * - * @handle: SCMI entity handle + * @handle: Pointer to SCMI entity handle * * Helper function which is used by various command functions that are * exposed to clients of this driver for allocating a message traffic event. @@ -299,7 +295,7 @@ static void scmi_tx_prepare(struct mbox_client *cl, void *m) * * Return: 0 if all went fine, else corresponding error. */ -static struct scmi_xfer *scmi_one_xfer_get(const struct scmi_handle *handle) +static struct scmi_xfer *scmi_xfer_get(const struct scmi_handle *handle) { u16 xfer_id; struct scmi_xfer *xfer; @@ -328,14 +324,14 @@ static struct scmi_xfer *scmi_one_xfer_get(const struct scmi_handle *handle) } /** - * scmi_one_xfer_put() - Release a message + * scmi_xfer_put() - Release a message * - * @minfo: transfer info pointer - * @xfer: message that was reserved by scmi_one_xfer_get + * @handle: Pointer to SCMI entity handle + * @xfer: message that was reserved by scmi_xfer_get * * This holds a spinlock to maintain integrity of internal data structures. */ -void scmi_one_xfer_put(const struct scmi_handle *handle, struct scmi_xfer *xfer) +void scmi_xfer_put(const struct scmi_handle *handle, struct scmi_xfer *xfer) { unsigned long flags; struct scmi_info *info = handle_to_scmi_info(handle); @@ -378,12 +374,12 @@ static bool scmi_xfer_done_no_timeout(const struct scmi_chan_info *cinfo, /** * scmi_do_xfer() - Do one transfer * - * @info: Pointer to SCMI entity information + * @handle: Pointer to SCMI entity handle * @xfer: Transfer to initiate and wait for response * * Return: -ETIMEDOUT in case of no response, if transmit error, - * return corresponding error, else if all goes well, - * return 0. + * return corresponding error, else if all goes well, + * return 0. */ int scmi_do_xfer(const struct scmi_handle *handle, struct scmi_xfer *xfer) { @@ -440,22 +436,22 @@ int scmi_do_xfer(const struct scmi_handle *handle, struct scmi_xfer *xfer) } /** - * scmi_one_xfer_init() - Allocate and initialise one message + * scmi_xfer_get_init() - Allocate and initialise one message * - * @handle: SCMI entity handle + * @handle: Pointer to SCMI entity handle * @msg_id: Message identifier - * @msg_prot_id: Protocol identifier for the message + * @prot_id: Protocol identifier for the message * @tx_size: transmit message size * @rx_size: receive message size * @p: pointer to the allocated and initialised message * - * This function allocates the message using @scmi_one_xfer_get and + * This function allocates the message using @scmi_xfer_get and * initialise the header. * * Return: 0 if all went fine with @p pointing to message, else * corresponding error. */ -int scmi_one_xfer_init(const struct scmi_handle *handle, u8 msg_id, u8 prot_id, +int scmi_xfer_get_init(const struct scmi_handle *handle, u8 msg_id, u8 prot_id, size_t tx_size, size_t rx_size, struct scmi_xfer **p) { int ret; @@ -468,7 +464,7 @@ int scmi_one_xfer_init(const struct scmi_handle *handle, u8 msg_id, u8 prot_id, tx_size > info->desc->max_msg_size) return -ERANGE; - xfer = scmi_one_xfer_get(handle); + xfer = scmi_xfer_get(handle); if (IS_ERR(xfer)) { ret = PTR_ERR(xfer); dev_err(dev, "failed to get free message slot(%d)\n", ret); @@ -482,13 +478,16 @@ int scmi_one_xfer_init(const struct scmi_handle *handle, u8 msg_id, u8 prot_id, xfer->hdr.poll_completion = false; *p = xfer; + return 0; } /** * scmi_version_get() - command to get the revision of the SCMI entity * - * @handle: Handle to SCMI entity information + * @handle: Pointer to SCMI entity handle + * @protocol: Protocol identifier for the message + * @version: Holds returned version of protocol. * * Updates the SCMI information in the internal data structure. * @@ -501,7 +500,7 @@ int scmi_version_get(const struct scmi_handle *handle, u8 protocol, __le32 *rev_info; struct scmi_xfer *t; - ret = scmi_one_xfer_init(handle, PROTOCOL_VERSION, protocol, 0, + ret = scmi_xfer_get_init(handle, PROTOCOL_VERSION, protocol, 0, sizeof(*version), &t); if (ret) return ret; @@ -512,7 +511,7 @@ int scmi_version_get(const struct scmi_handle *handle, u8 protocol, *version = le32_to_cpu(*rev_info); } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -540,12 +539,12 @@ scmi_is_protocol_implemented(const struct scmi_handle *handle, u8 prot_id) } /** - * scmi_handle_get() - Get the SCMI handle for a device + * scmi_handle_get() - Get the SCMI handle for a device * * @dev: pointer to device for which we want SCMI handle * * NOTE: The function does not track individual clients of the framework - * and is expected to be maintained by caller of SCMI protocol library. + * and is expected to be maintained by caller of SCMI protocol library. * scmi_handle_put must be balanced with successful scmi_handle_get * * Return: pointer to handle if successful, NULL on error @@ -576,7 +575,7 @@ struct scmi_handle *scmi_handle_get(struct device *dev) * @handle: handle acquired by scmi_handle_get * * NOTE: The function does not track individual clients of the framework - * and is expected to be maintained by caller of SCMI protocol library. + * and is expected to be maintained by caller of SCMI protocol library. * scmi_handle_put must be balanced with successful scmi_handle_get * * Return: 0 is successfully released @@ -599,7 +598,7 @@ int scmi_handle_put(const struct scmi_handle *handle) } static const struct scmi_desc scmi_generic_desc = { - .max_rx_timeout_ms = 30, /* we may increase this if required */ + .max_rx_timeout_ms = 30, /* We may increase this if required */ .max_msg = 20, /* Limited by MBOX_TX_QUEUE_LEN */ .max_msg_size = 128, }; @@ -621,9 +620,9 @@ static int scmi_xfer_info_init(struct scmi_info *sinfo) struct scmi_xfers_info *info = &sinfo->minfo; /* Pre-allocated messages, no more than what hdr.seq can support */ - if (WARN_ON(desc->max_msg >= (MSG_TOKEN_ID_MASK + 1))) { - dev_err(dev, "Maximum message of %d exceeds supported %d\n", - desc->max_msg, MSG_TOKEN_ID_MASK + 1); + if (WARN_ON(desc->max_msg >= MSG_TOKEN_MAX)) { + dev_err(dev, "Maximum message of %d exceeds supported %ld\n", + desc->max_msg, MSG_TOKEN_MAX); return -EINVAL; } @@ -637,8 +636,6 @@ static int scmi_xfer_info_init(struct scmi_info *sinfo) if (!info->xfer_alloc_table) return -ENOMEM; - bitmap_zero(info->xfer_alloc_table, desc->max_msg); - /* Pre-initialize the buffer pointer to pre-allocated buffers */ for (i = 0, xfer = info->xfer_block; i < desc->max_msg; i++, xfer++) { xfer->rx.buf = devm_kcalloc(dev, sizeof(u8), desc->max_msg_size, @@ -690,11 +687,12 @@ static int scmi_remove(struct platform_device *pdev) list_del(&info->node); mutex_unlock(&scmi_list_mutex); - if (!ret) { - /* Safe to free channels since no more users */ - ret = idr_for_each(idr, scmi_mbox_free_channel, idr); - idr_destroy(&info->tx_idr); - } + if (ret) + return ret; + + /* Safe to free channels since no more users */ + ret = idr_for_each(idr, scmi_mbox_free_channel, idr); + idr_destroy(&info->tx_idr); return ret; } @@ -841,7 +839,8 @@ static int scmi_probe(struct platform_device *pdev) if (of_property_read_u32(child, "reg", &prot_id)) continue; - prot_id &= MSG_PROTOCOL_ID_MASK; + if (!FIELD_FIT(MSG_PROTOCOL_ID_MASK, prot_id)) + dev_err(dev, "Out of range protocol %d\n", prot_id); if (!scmi_is_protocol_implemented(handle, prot_id)) { dev_err(dev, "SCMI protocol %d not implemented\n", diff --git a/drivers/firmware/arm_scmi/perf.c b/drivers/firmware/arm_scmi/perf.c index 987c64d19801..2a219b1261b1 100644 --- a/drivers/firmware/arm_scmi/perf.c +++ b/drivers/firmware/arm_scmi/perf.c @@ -115,7 +115,7 @@ static int scmi_perf_attributes_get(const struct scmi_handle *handle, struct scmi_xfer *t; struct scmi_msg_resp_perf_attributes *attr; - ret = scmi_one_xfer_init(handle, PROTOCOL_ATTRIBUTES, + ret = scmi_xfer_get_init(handle, PROTOCOL_ATTRIBUTES, SCMI_PROTOCOL_PERF, 0, sizeof(*attr), &t); if (ret) return ret; @@ -133,7 +133,7 @@ static int scmi_perf_attributes_get(const struct scmi_handle *handle, pi->stats_size = le32_to_cpu(attr->stats_size); } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -145,7 +145,7 @@ scmi_perf_domain_attributes_get(const struct scmi_handle *handle, u32 domain, struct scmi_xfer *t; struct scmi_msg_resp_perf_domain_attributes *attr; - ret = scmi_one_xfer_init(handle, PERF_DOMAIN_ATTRIBUTES, + ret = scmi_xfer_get_init(handle, PERF_DOMAIN_ATTRIBUTES, SCMI_PROTOCOL_PERF, sizeof(domain), sizeof(*attr), &t); if (ret) @@ -171,7 +171,7 @@ scmi_perf_domain_attributes_get(const struct scmi_handle *handle, u32 domain, memcpy(dom_info->name, attr->name, SCMI_MAX_STR_SIZE); } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -194,7 +194,7 @@ scmi_perf_describe_levels_get(const struct scmi_handle *handle, u32 domain, struct scmi_msg_perf_describe_levels *dom_info; struct scmi_msg_resp_perf_describe_levels *level_info; - ret = scmi_one_xfer_init(handle, PERF_DESCRIBE_LEVELS, + ret = scmi_xfer_get_init(handle, PERF_DESCRIBE_LEVELS, SCMI_PROTOCOL_PERF, sizeof(*dom_info), 0, &t); if (ret) return ret; @@ -237,7 +237,7 @@ scmi_perf_describe_levels_get(const struct scmi_handle *handle, u32 domain, } while (num_returned && num_remaining); perf_dom->opp_count = tot_opp_cnt; - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); sort(perf_dom->opp, tot_opp_cnt, sizeof(*opp), opp_cmp_func, NULL); return ret; @@ -250,7 +250,7 @@ static int scmi_perf_limits_set(const struct scmi_handle *handle, u32 domain, struct scmi_xfer *t; struct scmi_perf_set_limits *limits; - ret = scmi_one_xfer_init(handle, PERF_LIMITS_SET, SCMI_PROTOCOL_PERF, + ret = scmi_xfer_get_init(handle, PERF_LIMITS_SET, SCMI_PROTOCOL_PERF, sizeof(*limits), 0, &t); if (ret) return ret; @@ -262,7 +262,7 @@ static int scmi_perf_limits_set(const struct scmi_handle *handle, u32 domain, ret = scmi_do_xfer(handle, t); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -273,7 +273,7 @@ static int scmi_perf_limits_get(const struct scmi_handle *handle, u32 domain, struct scmi_xfer *t; struct scmi_perf_get_limits *limits; - ret = scmi_one_xfer_init(handle, PERF_LIMITS_GET, SCMI_PROTOCOL_PERF, + ret = scmi_xfer_get_init(handle, PERF_LIMITS_GET, SCMI_PROTOCOL_PERF, sizeof(__le32), 0, &t); if (ret) return ret; @@ -288,7 +288,7 @@ static int scmi_perf_limits_get(const struct scmi_handle *handle, u32 domain, *min_perf = le32_to_cpu(limits->min_level); } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -299,7 +299,7 @@ static int scmi_perf_level_set(const struct scmi_handle *handle, u32 domain, struct scmi_xfer *t; struct scmi_perf_set_level *lvl; - ret = scmi_one_xfer_init(handle, PERF_LEVEL_SET, SCMI_PROTOCOL_PERF, + ret = scmi_xfer_get_init(handle, PERF_LEVEL_SET, SCMI_PROTOCOL_PERF, sizeof(*lvl), 0, &t); if (ret) return ret; @@ -311,7 +311,7 @@ static int scmi_perf_level_set(const struct scmi_handle *handle, u32 domain, ret = scmi_do_xfer(handle, t); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -321,7 +321,7 @@ static int scmi_perf_level_get(const struct scmi_handle *handle, u32 domain, int ret; struct scmi_xfer *t; - ret = scmi_one_xfer_init(handle, PERF_LEVEL_GET, SCMI_PROTOCOL_PERF, + ret = scmi_xfer_get_init(handle, PERF_LEVEL_GET, SCMI_PROTOCOL_PERF, sizeof(u32), sizeof(u32), &t); if (ret) return ret; @@ -333,7 +333,7 @@ static int scmi_perf_level_get(const struct scmi_handle *handle, u32 domain, if (!ret) *level = le32_to_cpu(*(__le32 *)t->rx.buf); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -349,8 +349,8 @@ static int scmi_dev_domain_id(struct device *dev) return clkspec.args[0]; } -static int scmi_dvfs_add_opps_to_device(const struct scmi_handle *handle, - struct device *dev) +static int scmi_dvfs_device_opps_add(const struct scmi_handle *handle, + struct device *dev) { int idx, ret, domain; unsigned long freq; @@ -383,7 +383,7 @@ static int scmi_dvfs_add_opps_to_device(const struct scmi_handle *handle, return 0; } -static int scmi_dvfs_get_transition_latency(const struct scmi_handle *handle, +static int scmi_dvfs_transition_latency_get(const struct scmi_handle *handle, struct device *dev) { struct perf_dom_info *dom; @@ -432,8 +432,8 @@ static struct scmi_perf_ops perf_ops = { .level_set = scmi_perf_level_set, .level_get = scmi_perf_level_get, .device_domain_id = scmi_dev_domain_id, - .get_transition_latency = scmi_dvfs_get_transition_latency, - .add_opps_to_device = scmi_dvfs_add_opps_to_device, + .transition_latency_get = scmi_dvfs_transition_latency_get, + .device_opps_add = scmi_dvfs_device_opps_add, .freq_set = scmi_dvfs_freq_set, .freq_get = scmi_dvfs_freq_get, }; diff --git a/drivers/firmware/arm_scmi/power.c b/drivers/firmware/arm_scmi/power.c index 087c2876cdf2..cfa033b05aed 100644 --- a/drivers/firmware/arm_scmi/power.c +++ b/drivers/firmware/arm_scmi/power.c @@ -63,7 +63,7 @@ static int scmi_power_attributes_get(const struct scmi_handle *handle, struct scmi_xfer *t; struct scmi_msg_resp_power_attributes *attr; - ret = scmi_one_xfer_init(handle, PROTOCOL_ATTRIBUTES, + ret = scmi_xfer_get_init(handle, PROTOCOL_ATTRIBUTES, SCMI_PROTOCOL_POWER, 0, sizeof(*attr), &t); if (ret) return ret; @@ -78,7 +78,7 @@ static int scmi_power_attributes_get(const struct scmi_handle *handle, pi->stats_size = le32_to_cpu(attr->stats_size); } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -90,7 +90,7 @@ scmi_power_domain_attributes_get(const struct scmi_handle *handle, u32 domain, struct scmi_xfer *t; struct scmi_msg_resp_power_domain_attributes *attr; - ret = scmi_one_xfer_init(handle, POWER_DOMAIN_ATTRIBUTES, + ret = scmi_xfer_get_init(handle, POWER_DOMAIN_ATTRIBUTES, SCMI_PROTOCOL_POWER, sizeof(domain), sizeof(*attr), &t); if (ret) @@ -109,7 +109,7 @@ scmi_power_domain_attributes_get(const struct scmi_handle *handle, u32 domain, memcpy(dom_info->name, attr->name, SCMI_MAX_STR_SIZE); } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -120,7 +120,7 @@ scmi_power_state_set(const struct scmi_handle *handle, u32 domain, u32 state) struct scmi_xfer *t; struct scmi_power_set_state *st; - ret = scmi_one_xfer_init(handle, POWER_STATE_SET, SCMI_PROTOCOL_POWER, + ret = scmi_xfer_get_init(handle, POWER_STATE_SET, SCMI_PROTOCOL_POWER, sizeof(*st), 0, &t); if (ret) return ret; @@ -132,7 +132,7 @@ scmi_power_state_set(const struct scmi_handle *handle, u32 domain, u32 state) ret = scmi_do_xfer(handle, t); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -142,7 +142,7 @@ scmi_power_state_get(const struct scmi_handle *handle, u32 domain, u32 *state) int ret; struct scmi_xfer *t; - ret = scmi_one_xfer_init(handle, POWER_STATE_GET, SCMI_PROTOCOL_POWER, + ret = scmi_xfer_get_init(handle, POWER_STATE_GET, SCMI_PROTOCOL_POWER, sizeof(u32), sizeof(u32), &t); if (ret) return ret; @@ -153,7 +153,7 @@ scmi_power_state_get(const struct scmi_handle *handle, u32 domain, u32 *state) if (!ret) *state = le32_to_cpu(*(__le32 *)t->rx.buf); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } diff --git a/drivers/firmware/arm_scmi/sensors.c b/drivers/firmware/arm_scmi/sensors.c index bbb469fea0ed..27f2092b9882 100644 --- a/drivers/firmware/arm_scmi/sensors.c +++ b/drivers/firmware/arm_scmi/sensors.c @@ -79,7 +79,7 @@ static int scmi_sensor_attributes_get(const struct scmi_handle *handle, struct scmi_xfer *t; struct scmi_msg_resp_sensor_attributes *attr; - ret = scmi_one_xfer_init(handle, PROTOCOL_ATTRIBUTES, + ret = scmi_xfer_get_init(handle, PROTOCOL_ATTRIBUTES, SCMI_PROTOCOL_SENSOR, 0, sizeof(*attr), &t); if (ret) return ret; @@ -95,7 +95,7 @@ static int scmi_sensor_attributes_get(const struct scmi_handle *handle, si->reg_size = le32_to_cpu(attr->reg_size); } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -108,7 +108,7 @@ static int scmi_sensor_description_get(const struct scmi_handle *handle, struct scmi_xfer *t; struct scmi_msg_resp_sensor_description *buf; - ret = scmi_one_xfer_init(handle, SENSOR_DESCRIPTION_GET, + ret = scmi_xfer_get_init(handle, SENSOR_DESCRIPTION_GET, SCMI_PROTOCOL_SENSOR, sizeof(__le32), 0, &t); if (ret) return ret; @@ -150,7 +150,7 @@ static int scmi_sensor_description_get(const struct scmi_handle *handle, */ } while (num_returned && num_remaining); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -162,7 +162,7 @@ scmi_sensor_configuration_set(const struct scmi_handle *handle, u32 sensor_id) struct scmi_xfer *t; struct scmi_msg_set_sensor_config *cfg; - ret = scmi_one_xfer_init(handle, SENSOR_CONFIG_SET, + ret = scmi_xfer_get_init(handle, SENSOR_CONFIG_SET, SCMI_PROTOCOL_SENSOR, sizeof(*cfg), 0, &t); if (ret) return ret; @@ -173,7 +173,7 @@ scmi_sensor_configuration_set(const struct scmi_handle *handle, u32 sensor_id) ret = scmi_do_xfer(handle, t); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -185,7 +185,7 @@ static int scmi_sensor_trip_point_set(const struct scmi_handle *handle, struct scmi_xfer *t; struct scmi_msg_set_sensor_trip_point *trip; - ret = scmi_one_xfer_init(handle, SENSOR_TRIP_POINT_SET, + ret = scmi_xfer_get_init(handle, SENSOR_TRIP_POINT_SET, SCMI_PROTOCOL_SENSOR, sizeof(*trip), 0, &t); if (ret) return ret; @@ -198,7 +198,7 @@ static int scmi_sensor_trip_point_set(const struct scmi_handle *handle, ret = scmi_do_xfer(handle, t); - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } @@ -209,7 +209,7 @@ static int scmi_sensor_reading_get(const struct scmi_handle *handle, struct scmi_xfer *t; struct scmi_msg_sensor_reading_get *sensor; - ret = scmi_one_xfer_init(handle, SENSOR_READING_GET, + ret = scmi_xfer_get_init(handle, SENSOR_READING_GET, SCMI_PROTOCOL_SENSOR, sizeof(*sensor), sizeof(u64), &t); if (ret) @@ -227,7 +227,7 @@ static int scmi_sensor_reading_get(const struct scmi_handle *handle, *value |= (u64)le32_to_cpu(*(pval + 1)) << 32; } - scmi_one_xfer_put(handle, t); + scmi_xfer_put(handle, t); return ret; } diff --git a/drivers/firmware/arm_scpi.c b/drivers/firmware/arm_scpi.c index 6d7a6c0a5e07..c7d06a36b23a 100644 --- a/drivers/firmware/arm_scpi.c +++ b/drivers/firmware/arm_scpi.c @@ -890,7 +890,7 @@ static int scpi_alloc_xfer_list(struct device *dev, struct scpi_chan *ch) int i; struct scpi_xfer *xfers; - xfers = devm_kzalloc(dev, MAX_SCPI_XFERS * sizeof(*xfers), GFP_KERNEL); + xfers = devm_kcalloc(dev, MAX_SCPI_XFERS, sizeof(*xfers), GFP_KERNEL); if (!xfers) return -ENOMEM; diff --git a/drivers/firmware/broadcom/bcm47xx_nvram.c b/drivers/firmware/broadcom/bcm47xx_nvram.c index 0b631e5b5b84..d25f080fcb0d 100644 --- a/drivers/firmware/broadcom/bcm47xx_nvram.c +++ b/drivers/firmware/broadcom/bcm47xx_nvram.c @@ -36,7 +36,7 @@ struct nvram_header { static char nvram_buf[NVRAM_SPACE]; static size_t nvram_len; -static const u32 nvram_sizes[] = {0x8000, 0xF000, 0x10000}; +static const u32 nvram_sizes[] = {0x6000, 0x8000, 0xF000, 0x10000}; static u32 find_nvram_size(void __iomem *end) { diff --git a/drivers/firmware/dell_rbu.c b/drivers/firmware/dell_rbu.c index 2f452f1f7c8a..fb8af5cb7c9b 100644 --- a/drivers/firmware/dell_rbu.c +++ b/drivers/firmware/dell_rbu.c @@ -146,7 +146,7 @@ static int create_packet(void *data, size_t length) packet_array_size = max( (unsigned int)(allocation_floor / rbu_data.packetsize), (unsigned int)1); - invalid_addr_packet_array = kzalloc(packet_array_size * sizeof(void*), + invalid_addr_packet_array = kcalloc(packet_array_size, sizeof(void *), GFP_KERNEL); if (!invalid_addr_packet_array) { diff --git a/drivers/firmware/dmi-id.c b/drivers/firmware/dmi-id.c index 951b6c79f166..624a11cb07e2 100644 --- a/drivers/firmware/dmi-id.c +++ b/drivers/firmware/dmi-id.c @@ -47,6 +47,7 @@ DEFINE_DMI_ATTR_WITH_SHOW(product_name, 0444, DMI_PRODUCT_NAME); DEFINE_DMI_ATTR_WITH_SHOW(product_version, 0444, DMI_PRODUCT_VERSION); DEFINE_DMI_ATTR_WITH_SHOW(product_serial, 0400, DMI_PRODUCT_SERIAL); DEFINE_DMI_ATTR_WITH_SHOW(product_uuid, 0400, DMI_PRODUCT_UUID); +DEFINE_DMI_ATTR_WITH_SHOW(product_sku, 0444, DMI_PRODUCT_SKU); DEFINE_DMI_ATTR_WITH_SHOW(product_family, 0444, DMI_PRODUCT_FAMILY); DEFINE_DMI_ATTR_WITH_SHOW(board_vendor, 0444, DMI_BOARD_VENDOR); DEFINE_DMI_ATTR_WITH_SHOW(board_name, 0444, DMI_BOARD_NAME); @@ -193,6 +194,7 @@ static void __init dmi_id_init_attr_table(void) ADD_DMI_ATTR(product_serial, DMI_PRODUCT_SERIAL); ADD_DMI_ATTR(product_uuid, DMI_PRODUCT_UUID); ADD_DMI_ATTR(product_family, DMI_PRODUCT_FAMILY); + ADD_DMI_ATTR(product_sku, DMI_PRODUCT_SKU); ADD_DMI_ATTR(board_vendor, DMI_BOARD_VENDOR); ADD_DMI_ATTR(board_name, DMI_BOARD_NAME); ADD_DMI_ATTR(board_version, DMI_BOARD_VERSION); diff --git a/drivers/firmware/dmi_scan.c b/drivers/firmware/dmi_scan.c index 54e66adef252..f2483548cde9 100644 --- a/drivers/firmware/dmi_scan.c +++ b/drivers/firmware/dmi_scan.c @@ -447,6 +447,7 @@ static void __init dmi_decode(const struct dmi_header *dm, void *dummy) dmi_save_ident(dm, DMI_PRODUCT_VERSION, 6); dmi_save_ident(dm, DMI_PRODUCT_SERIAL, 7); dmi_save_uuid(dm, DMI_PRODUCT_UUID, 8); + dmi_save_ident(dm, DMI_PRODUCT_SKU, 25); dmi_save_ident(dm, DMI_PRODUCT_FAMILY, 26); break; case 2: /* Base Board Information */ diff --git a/drivers/firmware/efi/Kconfig b/drivers/firmware/efi/Kconfig index 3098410abad8..781a4a337557 100644 --- a/drivers/firmware/efi/Kconfig +++ b/drivers/firmware/efi/Kconfig @@ -174,6 +174,11 @@ config UEFI_CPER_ARM depends on UEFI_CPER && ( ARM || ARM64 ) default y +config UEFI_CPER_X86 + bool + depends on UEFI_CPER && X86 + default y + config EFI_DEV_PATH_PARSER bool depends on ACPI diff --git a/drivers/firmware/efi/Makefile b/drivers/firmware/efi/Makefile index cb805374f4bc..5f9f5039de50 100644 --- a/drivers/firmware/efi/Makefile +++ b/drivers/firmware/efi/Makefile @@ -31,3 +31,4 @@ obj-$(CONFIG_ARM) += $(arm-obj-y) obj-$(CONFIG_ARM64) += $(arm-obj-y) obj-$(CONFIG_EFI_CAPSULE_LOADER) += capsule-loader.o obj-$(CONFIG_UEFI_CPER_ARM) += cper-arm.o +obj-$(CONFIG_UEFI_CPER_X86) += cper-x86.o diff --git a/drivers/firmware/efi/apple-properties.c b/drivers/firmware/efi/apple-properties.c index adaa9a3714b9..60a95719ecb8 100644 --- a/drivers/firmware/efi/apple-properties.c +++ b/drivers/firmware/efi/apple-properties.c @@ -13,6 +13,9 @@ * * You should have received a copy of the GNU General Public License * along with this program; if not, see . + * + * Note, all properties are considered as u8 arrays. + * To get a value of any of them the caller must use device_property_read_u8_array(). */ #define pr_fmt(fmt) "apple-properties: " fmt @@ -96,12 +99,13 @@ static void __init unmarshal_key_value_pairs(struct dev_header *dev_header, entry[i].name = key; entry[i].length = val_len - sizeof(val_len); entry[i].is_array = !!entry[i].length; - entry[i].pointer.raw_data = ptr + key_len + sizeof(val_len); + entry[i].type = DEV_PROP_U8; + entry[i].pointer.u8_data = ptr + key_len + sizeof(val_len); if (dump_properties) { dev_info(dev, "property: %s\n", entry[i].name); print_hex_dump(KERN_INFO, pr_fmt(), DUMP_PREFIX_OFFSET, - 16, 1, entry[i].pointer.raw_data, + 16, 1, entry[i].pointer.u8_data, entry[i].length, true); } diff --git a/drivers/firmware/efi/arm-init.c b/drivers/firmware/efi/arm-init.c index 80d1a885def5..b5214c143fee 100644 --- a/drivers/firmware/efi/arm-init.c +++ b/drivers/firmware/efi/arm-init.c @@ -193,7 +193,7 @@ static __init void reserve_regions(void) * uses its own memory map instead. */ memblock_dump_all(); - memblock_remove(0, (phys_addr_t)ULLONG_MAX); + memblock_remove(0, PHYS_ADDR_MAX); for_each_efi_memory_desc(md) { paddr = md->phys_addr; diff --git a/drivers/firmware/efi/capsule-loader.c b/drivers/firmware/efi/capsule-loader.c index e456f4602df1..96688986da56 100644 --- a/drivers/firmware/efi/capsule-loader.c +++ b/drivers/firmware/efi/capsule-loader.c @@ -134,10 +134,16 @@ static ssize_t efi_capsule_submit_update(struct capsule_info *cap_info) /* Indicate capsule binary uploading is done */ cap_info->index = NO_FURTHER_WRITE_ACTION; - pr_info("Successfully upload capsule file with reboot type '%s'\n", - !cap_info->reset_type ? "RESET_COLD" : - cap_info->reset_type == 1 ? "RESET_WARM" : - "RESET_SHUTDOWN"); + + if (cap_info->header.flags & EFI_CAPSULE_PERSIST_ACROSS_RESET) { + pr_info("Successfully uploaded capsule file with reboot type '%s'\n", + !cap_info->reset_type ? "RESET_COLD" : + cap_info->reset_type == 1 ? "RESET_WARM" : + "RESET_SHUTDOWN"); + } else { + pr_info("Successfully processed capsule file\n"); + } + return 0; } diff --git a/drivers/firmware/efi/capsule.c b/drivers/firmware/efi/capsule.c index 901b9306bf94..4938c29b7c5d 100644 --- a/drivers/firmware/efi/capsule.c +++ b/drivers/firmware/efi/capsule.c @@ -231,7 +231,7 @@ int efi_capsule_update(efi_capsule_header_t *capsule, phys_addr_t *pages) count = DIV_ROUND_UP(imagesize, PAGE_SIZE); sg_count = sg_pages_num(count); - sg_pages = kzalloc(sg_count * sizeof(*sg_pages), GFP_KERNEL); + sg_pages = kcalloc(sg_count, sizeof(*sg_pages), GFP_KERNEL); if (!sg_pages) return -ENOMEM; diff --git a/drivers/firmware/efi/cper-arm.c b/drivers/firmware/efi/cper-arm.c index 698e5c8e0c8d..502811344e81 100644 --- a/drivers/firmware/efi/cper-arm.c +++ b/drivers/firmware/efi/cper-arm.c @@ -30,8 +30,6 @@ #include #include -#define INDENT_SP " " - static const char * const arm_reg_ctx_strs[] = { "AArch32 general purpose registers", "AArch32 EL1 context registers", @@ -283,7 +281,7 @@ void cper_print_proc_arm(const char *pfx, pfx, proc->psci_state); } - snprintf(newpfx, sizeof(newpfx), "%s%s", pfx, INDENT_SP); + snprintf(newpfx, sizeof(newpfx), "%s ", pfx); err_info = (struct cper_arm_err_info *)(proc + 1); for (i = 0; i < proc->err_info_num; i++) { @@ -310,7 +308,7 @@ void cper_print_proc_arm(const char *pfx, if (err_info->validation_bits & CPER_ARM_INFO_VALID_ERR_INFO) { printk("%serror_info: 0x%016llx\n", newpfx, err_info->error_info); - snprintf(infopfx, sizeof(infopfx), "%s%s", newpfx, INDENT_SP); + snprintf(infopfx, sizeof(infopfx), "%s ", newpfx); cper_print_arm_err_info(infopfx, err_info->type, err_info->error_info); } diff --git a/drivers/firmware/efi/cper-x86.c b/drivers/firmware/efi/cper-x86.c new file mode 100644 index 000000000000..2531de49f56c --- /dev/null +++ b/drivers/firmware/efi/cper-x86.c @@ -0,0 +1,356 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018, Advanced Micro Devices, Inc. + +#include + +/* + * We don't need a "CPER_IA" prefix since these are all locally defined. + * This will save us a lot of line space. + */ +#define VALID_LAPIC_ID BIT_ULL(0) +#define VALID_CPUID_INFO BIT_ULL(1) +#define VALID_PROC_ERR_INFO_NUM(bits) (((bits) & GENMASK_ULL(7, 2)) >> 2) +#define VALID_PROC_CXT_INFO_NUM(bits) (((bits) & GENMASK_ULL(13, 8)) >> 8) + +#define INFO_ERR_STRUCT_TYPE_CACHE \ + GUID_INIT(0xA55701F5, 0xE3EF, 0x43DE, 0xAC, 0x72, 0x24, 0x9B, \ + 0x57, 0x3F, 0xAD, 0x2C) +#define INFO_ERR_STRUCT_TYPE_TLB \ + GUID_INIT(0xFC06B535, 0x5E1F, 0x4562, 0x9F, 0x25, 0x0A, 0x3B, \ + 0x9A, 0xDB, 0x63, 0xC3) +#define INFO_ERR_STRUCT_TYPE_BUS \ + GUID_INIT(0x1CF3F8B3, 0xC5B1, 0x49a2, 0xAA, 0x59, 0x5E, 0xEF, \ + 0x92, 0xFF, 0xA6, 0x3C) +#define INFO_ERR_STRUCT_TYPE_MS \ + GUID_INIT(0x48AB7F57, 0xDC34, 0x4f6c, 0xA7, 0xD3, 0xB0, 0xB5, \ + 0xB0, 0xA7, 0x43, 0x14) + +#define INFO_VALID_CHECK_INFO BIT_ULL(0) +#define INFO_VALID_TARGET_ID BIT_ULL(1) +#define INFO_VALID_REQUESTOR_ID BIT_ULL(2) +#define INFO_VALID_RESPONDER_ID BIT_ULL(3) +#define INFO_VALID_IP BIT_ULL(4) + +#define CHECK_VALID_TRANS_TYPE BIT_ULL(0) +#define CHECK_VALID_OPERATION BIT_ULL(1) +#define CHECK_VALID_LEVEL BIT_ULL(2) +#define CHECK_VALID_PCC BIT_ULL(3) +#define CHECK_VALID_UNCORRECTED BIT_ULL(4) +#define CHECK_VALID_PRECISE_IP BIT_ULL(5) +#define CHECK_VALID_RESTARTABLE_IP BIT_ULL(6) +#define CHECK_VALID_OVERFLOW BIT_ULL(7) + +#define CHECK_VALID_BUS_PART_TYPE BIT_ULL(8) +#define CHECK_VALID_BUS_TIME_OUT BIT_ULL(9) +#define CHECK_VALID_BUS_ADDR_SPACE BIT_ULL(10) + +#define CHECK_VALID_BITS(check) (((check) & GENMASK_ULL(15, 0))) +#define CHECK_TRANS_TYPE(check) (((check) & GENMASK_ULL(17, 16)) >> 16) +#define CHECK_OPERATION(check) (((check) & GENMASK_ULL(21, 18)) >> 18) +#define CHECK_LEVEL(check) (((check) & GENMASK_ULL(24, 22)) >> 22) +#define CHECK_PCC BIT_ULL(25) +#define CHECK_UNCORRECTED BIT_ULL(26) +#define CHECK_PRECISE_IP BIT_ULL(27) +#define CHECK_RESTARTABLE_IP BIT_ULL(28) +#define CHECK_OVERFLOW BIT_ULL(29) + +#define CHECK_BUS_PART_TYPE(check) (((check) & GENMASK_ULL(31, 30)) >> 30) +#define CHECK_BUS_TIME_OUT BIT_ULL(32) +#define CHECK_BUS_ADDR_SPACE(check) (((check) & GENMASK_ULL(34, 33)) >> 33) + +#define CHECK_VALID_MS_ERR_TYPE BIT_ULL(0) +#define CHECK_VALID_MS_PCC BIT_ULL(1) +#define CHECK_VALID_MS_UNCORRECTED BIT_ULL(2) +#define CHECK_VALID_MS_PRECISE_IP BIT_ULL(3) +#define CHECK_VALID_MS_RESTARTABLE_IP BIT_ULL(4) +#define CHECK_VALID_MS_OVERFLOW BIT_ULL(5) + +#define CHECK_MS_ERR_TYPE(check) (((check) & GENMASK_ULL(18, 16)) >> 16) +#define CHECK_MS_PCC BIT_ULL(19) +#define CHECK_MS_UNCORRECTED BIT_ULL(20) +#define CHECK_MS_PRECISE_IP BIT_ULL(21) +#define CHECK_MS_RESTARTABLE_IP BIT_ULL(22) +#define CHECK_MS_OVERFLOW BIT_ULL(23) + +#define CTX_TYPE_MSR 1 +#define CTX_TYPE_MMREG 7 + +enum err_types { + ERR_TYPE_CACHE = 0, + ERR_TYPE_TLB, + ERR_TYPE_BUS, + ERR_TYPE_MS, + N_ERR_TYPES +}; + +static enum err_types cper_get_err_type(const guid_t *err_type) +{ + if (guid_equal(err_type, &INFO_ERR_STRUCT_TYPE_CACHE)) + return ERR_TYPE_CACHE; + else if (guid_equal(err_type, &INFO_ERR_STRUCT_TYPE_TLB)) + return ERR_TYPE_TLB; + else if (guid_equal(err_type, &INFO_ERR_STRUCT_TYPE_BUS)) + return ERR_TYPE_BUS; + else if (guid_equal(err_type, &INFO_ERR_STRUCT_TYPE_MS)) + return ERR_TYPE_MS; + else + return N_ERR_TYPES; +} + +static const char * const ia_check_trans_type_strs[] = { + "Instruction", + "Data Access", + "Generic", +}; + +static const char * const ia_check_op_strs[] = { + "generic error", + "generic read", + "generic write", + "data read", + "data write", + "instruction fetch", + "prefetch", + "eviction", + "snoop", +}; + +static const char * const ia_check_bus_part_type_strs[] = { + "Local Processor originated request", + "Local Processor responded to request", + "Local Processor observed", + "Generic", +}; + +static const char * const ia_check_bus_addr_space_strs[] = { + "Memory Access", + "Reserved", + "I/O", + "Other Transaction", +}; + +static const char * const ia_check_ms_error_type_strs[] = { + "No Error", + "Unclassified", + "Microcode ROM Parity Error", + "External Error", + "FRC Error", + "Internal Unclassified", +}; + +static const char * const ia_reg_ctx_strs[] = { + "Unclassified Data", + "MSR Registers (Machine Check and other MSRs)", + "32-bit Mode Execution Context", + "64-bit Mode Execution Context", + "FXSAVE Context", + "32-bit Mode Debug Registers (DR0-DR7)", + "64-bit Mode Debug Registers (DR0-DR7)", + "Memory Mapped Registers", +}; + +static inline void print_bool(char *str, const char *pfx, u64 check, u64 bit) +{ + printk("%s%s: %s\n", pfx, str, (check & bit) ? "true" : "false"); +} + +static void print_err_info_ms(const char *pfx, u16 validation_bits, u64 check) +{ + if (validation_bits & CHECK_VALID_MS_ERR_TYPE) { + u8 err_type = CHECK_MS_ERR_TYPE(check); + + printk("%sError Type: %u, %s\n", pfx, err_type, + err_type < ARRAY_SIZE(ia_check_ms_error_type_strs) ? + ia_check_ms_error_type_strs[err_type] : "unknown"); + } + + if (validation_bits & CHECK_VALID_MS_PCC) + print_bool("Processor Context Corrupt", pfx, check, CHECK_MS_PCC); + + if (validation_bits & CHECK_VALID_MS_UNCORRECTED) + print_bool("Uncorrected", pfx, check, CHECK_MS_UNCORRECTED); + + if (validation_bits & CHECK_VALID_MS_PRECISE_IP) + print_bool("Precise IP", pfx, check, CHECK_MS_PRECISE_IP); + + if (validation_bits & CHECK_VALID_MS_RESTARTABLE_IP) + print_bool("Restartable IP", pfx, check, CHECK_MS_RESTARTABLE_IP); + + if (validation_bits & CHECK_VALID_MS_OVERFLOW) + print_bool("Overflow", pfx, check, CHECK_MS_OVERFLOW); +} + +static void print_err_info(const char *pfx, u8 err_type, u64 check) +{ + u16 validation_bits = CHECK_VALID_BITS(check); + + /* + * The MS Check structure varies a lot from the others, so use a + * separate function for decoding. + */ + if (err_type == ERR_TYPE_MS) + return print_err_info_ms(pfx, validation_bits, check); + + if (validation_bits & CHECK_VALID_TRANS_TYPE) { + u8 trans_type = CHECK_TRANS_TYPE(check); + + printk("%sTransaction Type: %u, %s\n", pfx, trans_type, + trans_type < ARRAY_SIZE(ia_check_trans_type_strs) ? + ia_check_trans_type_strs[trans_type] : "unknown"); + } + + if (validation_bits & CHECK_VALID_OPERATION) { + u8 op = CHECK_OPERATION(check); + + /* + * CACHE has more operation types than TLB or BUS, though the + * name and the order are the same. + */ + u8 max_ops = (err_type == ERR_TYPE_CACHE) ? 9 : 7; + + printk("%sOperation: %u, %s\n", pfx, op, + op < max_ops ? ia_check_op_strs[op] : "unknown"); + } + + if (validation_bits & CHECK_VALID_LEVEL) + printk("%sLevel: %llu\n", pfx, CHECK_LEVEL(check)); + + if (validation_bits & CHECK_VALID_PCC) + print_bool("Processor Context Corrupt", pfx, check, CHECK_PCC); + + if (validation_bits & CHECK_VALID_UNCORRECTED) + print_bool("Uncorrected", pfx, check, CHECK_UNCORRECTED); + + if (validation_bits & CHECK_VALID_PRECISE_IP) + print_bool("Precise IP", pfx, check, CHECK_PRECISE_IP); + + if (validation_bits & CHECK_VALID_RESTARTABLE_IP) + print_bool("Restartable IP", pfx, check, CHECK_RESTARTABLE_IP); + + if (validation_bits & CHECK_VALID_OVERFLOW) + print_bool("Overflow", pfx, check, CHECK_OVERFLOW); + + if (err_type != ERR_TYPE_BUS) + return; + + if (validation_bits & CHECK_VALID_BUS_PART_TYPE) { + u8 part_type = CHECK_BUS_PART_TYPE(check); + + printk("%sParticipation Type: %u, %s\n", pfx, part_type, + part_type < ARRAY_SIZE(ia_check_bus_part_type_strs) ? + ia_check_bus_part_type_strs[part_type] : "unknown"); + } + + if (validation_bits & CHECK_VALID_BUS_TIME_OUT) + print_bool("Time Out", pfx, check, CHECK_BUS_TIME_OUT); + + if (validation_bits & CHECK_VALID_BUS_ADDR_SPACE) { + u8 addr_space = CHECK_BUS_ADDR_SPACE(check); + + printk("%sAddress Space: %u, %s\n", pfx, addr_space, + addr_space < ARRAY_SIZE(ia_check_bus_addr_space_strs) ? + ia_check_bus_addr_space_strs[addr_space] : "unknown"); + } +} + +void cper_print_proc_ia(const char *pfx, const struct cper_sec_proc_ia *proc) +{ + int i; + struct cper_ia_err_info *err_info; + struct cper_ia_proc_ctx *ctx_info; + char newpfx[64], infopfx[64]; + u8 err_type; + + if (proc->validation_bits & VALID_LAPIC_ID) + printk("%sLocal APIC_ID: 0x%llx\n", pfx, proc->lapic_id); + + if (proc->validation_bits & VALID_CPUID_INFO) { + printk("%sCPUID Info:\n", pfx); + print_hex_dump(pfx, "", DUMP_PREFIX_OFFSET, 16, 4, proc->cpuid, + sizeof(proc->cpuid), 0); + } + + snprintf(newpfx, sizeof(newpfx), "%s ", pfx); + + err_info = (struct cper_ia_err_info *)(proc + 1); + for (i = 0; i < VALID_PROC_ERR_INFO_NUM(proc->validation_bits); i++) { + printk("%sError Information Structure %d:\n", pfx, i); + + err_type = cper_get_err_type(&err_info->err_type); + printk("%sError Structure Type: %s\n", newpfx, + err_type < ARRAY_SIZE(cper_proc_error_type_strs) ? + cper_proc_error_type_strs[err_type] : "unknown"); + + if (err_type >= N_ERR_TYPES) { + printk("%sError Structure Type: %pUl\n", newpfx, + &err_info->err_type); + } + + if (err_info->validation_bits & INFO_VALID_CHECK_INFO) { + printk("%sCheck Information: 0x%016llx\n", newpfx, + err_info->check_info); + + if (err_type < N_ERR_TYPES) { + snprintf(infopfx, sizeof(infopfx), "%s ", + newpfx); + + print_err_info(infopfx, err_type, + err_info->check_info); + } + } + + if (err_info->validation_bits & INFO_VALID_TARGET_ID) { + printk("%sTarget Identifier: 0x%016llx\n", + newpfx, err_info->target_id); + } + + if (err_info->validation_bits & INFO_VALID_REQUESTOR_ID) { + printk("%sRequestor Identifier: 0x%016llx\n", + newpfx, err_info->requestor_id); + } + + if (err_info->validation_bits & INFO_VALID_RESPONDER_ID) { + printk("%sResponder Identifier: 0x%016llx\n", + newpfx, err_info->responder_id); + } + + if (err_info->validation_bits & INFO_VALID_IP) { + printk("%sInstruction Pointer: 0x%016llx\n", + newpfx, err_info->ip); + } + + err_info++; + } + + ctx_info = (struct cper_ia_proc_ctx *)err_info; + for (i = 0; i < VALID_PROC_CXT_INFO_NUM(proc->validation_bits); i++) { + int size = sizeof(*ctx_info) + ctx_info->reg_arr_size; + int groupsize = 4; + + printk("%sContext Information Structure %d:\n", pfx, i); + + printk("%sRegister Context Type: %s\n", newpfx, + ctx_info->reg_ctx_type < ARRAY_SIZE(ia_reg_ctx_strs) ? + ia_reg_ctx_strs[ctx_info->reg_ctx_type] : "unknown"); + + printk("%sRegister Array Size: 0x%04x\n", newpfx, + ctx_info->reg_arr_size); + + if (ctx_info->reg_ctx_type == CTX_TYPE_MSR) { + groupsize = 8; /* MSRs are 8 bytes wide. */ + printk("%sMSR Address: 0x%08x\n", newpfx, + ctx_info->msr_addr); + } + + if (ctx_info->reg_ctx_type == CTX_TYPE_MMREG) { + printk("%sMM Register Address: 0x%016llx\n", newpfx, + ctx_info->mm_reg_addr); + } + + printk("%sRegister Array:\n", newpfx); + print_hex_dump(newpfx, "", DUMP_PREFIX_OFFSET, 16, groupsize, + (ctx_info + 1), ctx_info->reg_arr_size, 0); + + ctx_info = (struct cper_ia_proc_ctx *)((long)ctx_info + size); + } +} diff --git a/drivers/firmware/efi/cper.c b/drivers/firmware/efi/cper.c index c165933ebf38..3bf0dca378a6 100644 --- a/drivers/firmware/efi/cper.c +++ b/drivers/firmware/efi/cper.c @@ -37,8 +37,6 @@ #include #include -#define INDENT_SP " " - static char rcd_decode_str[CPER_REC_LEN]; /* @@ -433,7 +431,7 @@ cper_estatus_print_section(const char *pfx, struct acpi_hest_generic_data *gdata if (gdata->validation_bits & CPER_SEC_VALID_FRU_TEXT) printk("%s""fru_text: %.20s\n", pfx, gdata->fru_text); - snprintf(newpfx, sizeof(newpfx), "%s%s", pfx, INDENT_SP); + snprintf(newpfx, sizeof(newpfx), "%s ", pfx); if (guid_equal(sec_type, &CPER_SEC_PROC_GENERIC)) { struct cper_sec_proc_generic *proc_err = acpi_hest_get_payload(gdata); @@ -469,6 +467,16 @@ cper_estatus_print_section(const char *pfx, struct acpi_hest_generic_data *gdata cper_print_proc_arm(newpfx, arm_err); else goto err_section_too_small; +#endif +#if defined(CONFIG_UEFI_CPER_X86) + } else if (guid_equal(sec_type, &CPER_SEC_PROC_IA)) { + struct cper_sec_proc_ia *ia_err = acpi_hest_get_payload(gdata); + + printk("%ssection_type: IA32/X64 processor error\n", newpfx); + if (gdata->error_data_length >= sizeof(*ia_err)) + cper_print_proc_ia(newpfx, ia_err); + else + goto err_section_too_small; #endif } else { const void *err = acpi_hest_get_payload(gdata); @@ -500,7 +508,7 @@ void cper_estatus_print(const char *pfx, "It has been corrected by h/w " "and requires no further action"); printk("%s""event severity: %s\n", pfx, cper_severity_str(severity)); - snprintf(newpfx, sizeof(newpfx), "%s%s", pfx, INDENT_SP); + snprintf(newpfx, sizeof(newpfx), "%s ", pfx); apei_estatus_for_each_section(estatus, gdata) { cper_estatus_print_section(newpfx, gdata, sec_no); diff --git a/drivers/firmware/efi/efi-pstore.c b/drivers/firmware/efi/efi-pstore.c index 5a0fa939d70f..cfe87b465819 100644 --- a/drivers/firmware/efi/efi-pstore.c +++ b/drivers/firmware/efi/efi-pstore.c @@ -28,10 +28,9 @@ static int efi_pstore_close(struct pstore_info *psi) return 0; } -static inline u64 generic_id(unsigned long timestamp, - unsigned int part, int count) +static inline u64 generic_id(u64 timestamp, unsigned int part, int count) { - return ((u64) timestamp * 100 + part) * 1000 + count; + return (timestamp * 100 + part) * 1000 + count; } static int efi_pstore_read_func(struct efivar_entry *entry, @@ -42,7 +41,8 @@ static int efi_pstore_read_func(struct efivar_entry *entry, int i; int cnt; unsigned int part; - unsigned long time, size; + unsigned long size; + u64 time; if (efi_guidcmp(entry->var.VendorGuid, vendor)) return 0; @@ -50,7 +50,7 @@ static int efi_pstore_read_func(struct efivar_entry *entry, for (i = 0; i < DUMP_NAME_LEN; i++) name[i] = entry->var.VariableName[i]; - if (sscanf(name, "dump-type%u-%u-%d-%lu-%c", + if (sscanf(name, "dump-type%u-%u-%d-%llu-%c", &record->type, &part, &cnt, &time, &data_type) == 5) { record->id = generic_id(time, part, cnt); record->part = part; @@ -62,7 +62,7 @@ static int efi_pstore_read_func(struct efivar_entry *entry, else record->compressed = false; record->ecc_notice_size = 0; - } else if (sscanf(name, "dump-type%u-%u-%d-%lu", + } else if (sscanf(name, "dump-type%u-%u-%d-%llu", &record->type, &part, &cnt, &time) == 4) { record->id = generic_id(time, part, cnt); record->part = part; @@ -71,7 +71,7 @@ static int efi_pstore_read_func(struct efivar_entry *entry, record->time.tv_nsec = 0; record->compressed = false; record->ecc_notice_size = 0; - } else if (sscanf(name, "dump-type%u-%u-%lu", + } else if (sscanf(name, "dump-type%u-%u-%llu", &record->type, &part, &time) == 3) { /* * Check if an old format, @@ -250,9 +250,10 @@ static int efi_pstore_write(struct pstore_record *record) /* Since we copy the entire length of name, make sure it is wiped. */ memset(name, 0, sizeof(name)); - snprintf(name, sizeof(name), "dump-type%u-%u-%d-%lu-%c", + snprintf(name, sizeof(name), "dump-type%u-%u-%d-%lld-%c", record->type, record->part, record->count, - record->time.tv_sec, record->compressed ? 'C' : 'D'); + (long long)record->time.tv_sec, + record->compressed ? 'C' : 'D'); for (i = 0; i < DUMP_NAME_LEN; i++) efi_name[i] = name[i]; @@ -327,15 +328,15 @@ static int efi_pstore_erase(struct pstore_record *record) char name[DUMP_NAME_LEN]; int ret; - snprintf(name, sizeof(name), "dump-type%u-%u-%d-%lu", + snprintf(name, sizeof(name), "dump-type%u-%u-%d-%lld", record->type, record->part, record->count, - record->time.tv_sec); + (long long)record->time.tv_sec); ret = efi_pstore_erase_name(name); if (ret != -ENOENT) return ret; - snprintf(name, sizeof(name), "dump-type%u-%u-%lu", - record->type, record->part, record->time.tv_sec); + snprintf(name, sizeof(name), "dump-type%u-%u-%lld", + record->type, record->part, (long long)record->time.tv_sec); ret = efi_pstore_erase_name(name); return ret; diff --git a/drivers/firmware/efi/libstub/secureboot.c b/drivers/firmware/efi/libstub/secureboot.c index 8f07eb414c00..72d9dfbebf08 100644 --- a/drivers/firmware/efi/libstub/secureboot.c +++ b/drivers/firmware/efi/libstub/secureboot.c @@ -30,6 +30,9 @@ static const efi_char16_t shim_MokSBState_name[] = L"MokSBState"; /* * Determine whether we're in secure boot mode. + * + * Please keep the logic in sync with + * arch/x86/xen/efi.c:xen_efi_get_secureboot(). */ enum efi_secureboot_mode efi_get_secureboot(efi_system_table_t *sys_table_arg) { diff --git a/drivers/firmware/efi/libstub/tpm.c b/drivers/firmware/efi/libstub/tpm.c index 9d08cea3f1b0..caa37a6dd9d4 100644 --- a/drivers/firmware/efi/libstub/tpm.c +++ b/drivers/firmware/efi/libstub/tpm.c @@ -59,7 +59,7 @@ void efi_enable_reset_attack_mitigation(efi_system_table_t *sys_table_arg) #endif -void efi_retrieve_tpm2_eventlog_1_2(efi_system_table_t *sys_table_arg) +static void efi_retrieve_tpm2_eventlog_1_2(efi_system_table_t *sys_table_arg) { efi_guid_t tcg2_guid = EFI_TCG2_PROTOCOL_GUID; efi_guid_t linux_eventlog_guid = LINUX_EFI_TPM_EVENT_LOG_GUID; diff --git a/drivers/firmware/efi/runtime-map.c b/drivers/firmware/efi/runtime-map.c index f377609ff141..84a11d0a8023 100644 --- a/drivers/firmware/efi/runtime-map.c +++ b/drivers/firmware/efi/runtime-map.c @@ -166,7 +166,7 @@ int __init efi_runtime_map_init(struct kobject *efi_kobj) if (!efi_enabled(EFI_MEMMAP)) return 0; - map_entries = kzalloc(efi.memmap.nr_map * sizeof(entry), GFP_KERNEL); + map_entries = kcalloc(efi.memmap.nr_map, sizeof(entry), GFP_KERNEL); if (!map_entries) { ret = -ENOMEM; goto out; diff --git a/drivers/firmware/google/Kconfig b/drivers/firmware/google/Kconfig index f16b381a569c..a456a000048b 100644 --- a/drivers/firmware/google/Kconfig +++ b/drivers/firmware/google/Kconfig @@ -55,6 +55,14 @@ config GOOGLE_MEMCONSOLE_X86_LEGACY the EBDA on Google servers. If found, this log is exported to userland in the file /sys/firmware/log. +config GOOGLE_FRAMEBUFFER_COREBOOT + tristate "Coreboot Framebuffer" + depends on FB_SIMPLE + depends on GOOGLE_COREBOOT_TABLE + help + This option enables the kernel to search for a framebuffer in + the coreboot table. If found, it is registered with simplefb. + config GOOGLE_MEMCONSOLE_COREBOOT tristate "Firmware Memory Console" depends on GOOGLE_COREBOOT_TABLE diff --git a/drivers/firmware/google/Makefile b/drivers/firmware/google/Makefile index dcd3675efcfc..d0b3fba96194 100644 --- a/drivers/firmware/google/Makefile +++ b/drivers/firmware/google/Makefile @@ -4,6 +4,7 @@ obj-$(CONFIG_GOOGLE_SMI) += gsmi.o obj-$(CONFIG_GOOGLE_COREBOOT_TABLE) += coreboot_table.o obj-$(CONFIG_GOOGLE_COREBOOT_TABLE_ACPI) += coreboot_table-acpi.o obj-$(CONFIG_GOOGLE_COREBOOT_TABLE_OF) += coreboot_table-of.o +obj-$(CONFIG_GOOGLE_FRAMEBUFFER_COREBOOT) += framebuffer-coreboot.o obj-$(CONFIG_GOOGLE_MEMCONSOLE) += memconsole.o obj-$(CONFIG_GOOGLE_MEMCONSOLE_COREBOOT) += memconsole-coreboot.o obj-$(CONFIG_GOOGLE_MEMCONSOLE_X86_LEGACY) += memconsole-x86-legacy.o diff --git a/drivers/firmware/google/coreboot_table-acpi.c b/drivers/firmware/google/coreboot_table-acpi.c index fb98db2d20e2..77197fe3d42f 100644 --- a/drivers/firmware/google/coreboot_table-acpi.c +++ b/drivers/firmware/google/coreboot_table-acpi.c @@ -53,7 +53,7 @@ static int coreboot_table_acpi_probe(struct platform_device *pdev) if (!ptr) return -ENOMEM; - return coreboot_table_init(ptr); + return coreboot_table_init(&pdev->dev, ptr); } static int coreboot_table_acpi_remove(struct platform_device *pdev) diff --git a/drivers/firmware/google/coreboot_table-of.c b/drivers/firmware/google/coreboot_table-of.c index 727acdc83e83..f15bf404c579 100644 --- a/drivers/firmware/google/coreboot_table-of.c +++ b/drivers/firmware/google/coreboot_table-of.c @@ -34,7 +34,7 @@ static int coreboot_table_of_probe(struct platform_device *pdev) if (!ptr) return -ENOMEM; - return coreboot_table_init(ptr); + return coreboot_table_init(&pdev->dev, ptr); } static int coreboot_table_of_remove(struct platform_device *pdev) diff --git a/drivers/firmware/google/coreboot_table.c b/drivers/firmware/google/coreboot_table.c index 0019d3ec18dd..19db5709ae28 100644 --- a/drivers/firmware/google/coreboot_table.c +++ b/drivers/firmware/google/coreboot_table.c @@ -4,6 +4,7 @@ * Module providing coreboot table access. * * Copyright 2017 Google Inc. + * Copyright 2017 Samuel Holland * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License v2.0 as published by @@ -15,37 +16,96 @@ * GNU General Public License for more details. */ +#include #include #include #include #include #include +#include #include "coreboot_table.h" -struct coreboot_table_entry { - u32 tag; - u32 size; -}; +#define CB_DEV(d) container_of(d, struct coreboot_device, dev) +#define CB_DRV(d) container_of(d, struct coreboot_driver, drv) static struct coreboot_table_header __iomem *ptr_header; -/* - * This function parses the coreboot table for an entry that contains the base - * address of the given entry tag. The coreboot table consists of a header - * directly followed by a number of small, variable-sized entries, which each - * contain an identifying tag and their length as the first two fields. - */ -int coreboot_table_find(int tag, void *data, size_t data_size) +static int coreboot_bus_match(struct device *dev, struct device_driver *drv) { - struct coreboot_table_header header; - struct coreboot_table_entry entry; - void *ptr_entry; - int i; + struct coreboot_device *device = CB_DEV(dev); + struct coreboot_driver *driver = CB_DRV(drv); - if (!ptr_header) - return -EPROBE_DEFER; + return device->entry.tag == driver->tag; +} +static int coreboot_bus_probe(struct device *dev) +{ + int ret = -ENODEV; + struct coreboot_device *device = CB_DEV(dev); + struct coreboot_driver *driver = CB_DRV(dev->driver); + + if (driver->probe) + ret = driver->probe(device); + + return ret; +} + +static int coreboot_bus_remove(struct device *dev) +{ + int ret = 0; + struct coreboot_device *device = CB_DEV(dev); + struct coreboot_driver *driver = CB_DRV(dev->driver); + + if (driver->remove) + ret = driver->remove(device); + + return ret; +} + +static struct bus_type coreboot_bus_type = { + .name = "coreboot", + .match = coreboot_bus_match, + .probe = coreboot_bus_probe, + .remove = coreboot_bus_remove, +}; + +static int __init coreboot_bus_init(void) +{ + return bus_register(&coreboot_bus_type); +} +module_init(coreboot_bus_init); + +static void coreboot_device_release(struct device *dev) +{ + struct coreboot_device *device = CB_DEV(dev); + + kfree(device); +} + +int coreboot_driver_register(struct coreboot_driver *driver) +{ + driver->drv.bus = &coreboot_bus_type; + + return driver_register(&driver->drv); +} +EXPORT_SYMBOL(coreboot_driver_register); + +void coreboot_driver_unregister(struct coreboot_driver *driver) +{ + driver_unregister(&driver->drv); +} +EXPORT_SYMBOL(coreboot_driver_unregister); + +int coreboot_table_init(struct device *dev, void __iomem *ptr) +{ + int i, ret; + void *ptr_entry; + struct coreboot_device *device; + struct coreboot_table_entry entry; + struct coreboot_table_header header; + + ptr_header = ptr; memcpy_fromio(&header, ptr_header, sizeof(header)); if (strncmp(header.signature, "LBIO", sizeof(header.signature))) { @@ -54,37 +114,41 @@ int coreboot_table_find(int tag, void *data, size_t data_size) } ptr_entry = (void *)ptr_header + header.header_bytes; - for (i = 0; i < header.table_entries; i++) { memcpy_fromio(&entry, ptr_entry, sizeof(entry)); - if (entry.tag == tag) { - if (data_size < entry.size) - return -EINVAL; - memcpy_fromio(data, ptr_entry, entry.size); + device = kzalloc(sizeof(struct device) + entry.size, GFP_KERNEL); + if (!device) { + ret = -ENOMEM; + break; + } + + dev_set_name(&device->dev, "coreboot%d", i); + device->dev.parent = dev; + device->dev.bus = &coreboot_bus_type; + device->dev.release = coreboot_device_release; + memcpy_fromio(&device->entry, ptr_entry, entry.size); - return 0; + ret = device_register(&device->dev); + if (ret) { + put_device(&device->dev); + break; } ptr_entry += entry.size; } - return -ENOENT; -} -EXPORT_SYMBOL(coreboot_table_find); - -int coreboot_table_init(void __iomem *ptr) -{ - ptr_header = ptr; - - return 0; + return ret; } EXPORT_SYMBOL(coreboot_table_init); int coreboot_table_exit(void) { - if (ptr_header) + if (ptr_header) { + bus_unregister(&coreboot_bus_type); iounmap(ptr_header); + ptr_header = NULL; + } return 0; } diff --git a/drivers/firmware/google/coreboot_table.h b/drivers/firmware/google/coreboot_table.h index 6eff1ae0c5d3..8ad95a94481b 100644 --- a/drivers/firmware/google/coreboot_table.h +++ b/drivers/firmware/google/coreboot_table.h @@ -3,7 +3,9 @@ * * Internal header for coreboot table access. * + * Copyright 2014 Gerd Hoffmann * Copyright 2017 Google Inc. + * Copyright 2017 Samuel Holland * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License v2.0 as published by @@ -20,14 +22,6 @@ #include -/* List of coreboot entry structures that is used */ -struct lb_cbmem_ref { - uint32_t tag; - uint32_t size; - - uint64_t cbmem_addr; -}; - /* Coreboot table header structure */ struct coreboot_table_header { char signature[4]; @@ -38,11 +32,67 @@ struct coreboot_table_header { u32 table_entries; }; -/* Retrieve coreboot table entry with tag *tag* and copy it to data */ -int coreboot_table_find(int tag, void *data, size_t data_size); +/* List of coreboot entry structures that is used */ +/* Generic */ +struct coreboot_table_entry { + u32 tag; + u32 size; +}; + +/* Points to a CBMEM entry */ +struct lb_cbmem_ref { + u32 tag; + u32 size; + + u64 cbmem_addr; +}; + +/* Describes framebuffer setup by coreboot */ +struct lb_framebuffer { + u32 tag; + u32 size; + + u64 physical_address; + u32 x_resolution; + u32 y_resolution; + u32 bytes_per_line; + u8 bits_per_pixel; + u8 red_mask_pos; + u8 red_mask_size; + u8 green_mask_pos; + u8 green_mask_size; + u8 blue_mask_pos; + u8 blue_mask_size; + u8 reserved_mask_pos; + u8 reserved_mask_size; +}; + +/* A device, additionally with information from coreboot. */ +struct coreboot_device { + struct device dev; + union { + struct coreboot_table_entry entry; + struct lb_cbmem_ref cbmem_ref; + struct lb_framebuffer framebuffer; + }; +}; + +/* A driver for handling devices described in coreboot tables. */ +struct coreboot_driver { + int (*probe)(struct coreboot_device *); + int (*remove)(struct coreboot_device *); + struct device_driver drv; + u32 tag; +}; + +/* Register a driver that uses the data from a coreboot table. */ +int coreboot_driver_register(struct coreboot_driver *driver); + +/* Unregister a driver that uses the data from a coreboot table. */ +void coreboot_driver_unregister(struct coreboot_driver *driver); /* Initialize coreboot table module given a pointer to iomem */ -int coreboot_table_init(void __iomem *ptr); +int coreboot_table_init(struct device *dev, void __iomem *ptr); /* Cleanup coreboot table module */ int coreboot_table_exit(void); diff --git a/drivers/firmware/google/framebuffer-coreboot.c b/drivers/firmware/google/framebuffer-coreboot.c new file mode 100644 index 000000000000..b8b49c067157 --- /dev/null +++ b/drivers/firmware/google/framebuffer-coreboot.c @@ -0,0 +1,115 @@ +/* + * framebuffer-coreboot.c + * + * Memory based framebuffer accessed through coreboot table. + * + * Copyright 2012-2013 David Herrmann + * Copyright 2017 Google Inc. + * Copyright 2017 Samuel Holland + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License v2.0 as published by + * the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + */ + +#include +#include +#include +#include +#include +#include + +#include "coreboot_table.h" + +#define CB_TAG_FRAMEBUFFER 0x12 + +static const struct simplefb_format formats[] = SIMPLEFB_FORMATS; + +static int framebuffer_probe(struct coreboot_device *dev) +{ + int i; + u32 length; + struct lb_framebuffer *fb = &dev->framebuffer; + struct platform_device *pdev; + struct resource res; + struct simplefb_platform_data pdata = { + .width = fb->x_resolution, + .height = fb->y_resolution, + .stride = fb->bytes_per_line, + .format = NULL, + }; + + for (i = 0; i < ARRAY_SIZE(formats); ++i) { + if (fb->bits_per_pixel == formats[i].bits_per_pixel && + fb->red_mask_pos == formats[i].red.offset && + fb->red_mask_size == formats[i].red.length && + fb->green_mask_pos == formats[i].green.offset && + fb->green_mask_size == formats[i].green.length && + fb->blue_mask_pos == formats[i].blue.offset && + fb->blue_mask_size == formats[i].blue.length && + fb->reserved_mask_pos == formats[i].transp.offset && + fb->reserved_mask_size == formats[i].transp.length) + pdata.format = formats[i].name; + } + if (!pdata.format) + return -ENODEV; + + memset(&res, 0, sizeof(res)); + res.flags = IORESOURCE_MEM | IORESOURCE_BUSY; + res.name = "Coreboot Framebuffer"; + res.start = fb->physical_address; + length = PAGE_ALIGN(fb->y_resolution * fb->bytes_per_line); + res.end = res.start + length - 1; + if (res.end <= res.start) + return -EINVAL; + + pdev = platform_device_register_resndata(&dev->dev, + "simple-framebuffer", 0, + &res, 1, &pdata, + sizeof(pdata)); + if (IS_ERR(pdev)) + pr_warn("coreboot: could not register framebuffer\n"); + else + dev_set_drvdata(&dev->dev, pdev); + + return PTR_ERR_OR_ZERO(pdev); +} + +static int framebuffer_remove(struct coreboot_device *dev) +{ + struct platform_device *pdev = dev_get_drvdata(&dev->dev); + + platform_device_unregister(pdev); + + return 0; +} + +static struct coreboot_driver framebuffer_driver = { + .probe = framebuffer_probe, + .remove = framebuffer_remove, + .drv = { + .name = "framebuffer", + }, + .tag = CB_TAG_FRAMEBUFFER, +}; + +static int __init coreboot_framebuffer_init(void) +{ + return coreboot_driver_register(&framebuffer_driver); +} + +static void coreboot_framebuffer_exit(void) +{ + coreboot_driver_unregister(&framebuffer_driver); +} + +module_init(coreboot_framebuffer_init); +module_exit(coreboot_framebuffer_exit); + +MODULE_AUTHOR("Samuel Holland "); +MODULE_LICENSE("GPL"); diff --git a/drivers/firmware/google/memconsole-coreboot.c b/drivers/firmware/google/memconsole-coreboot.c index 52738887735c..b29e10757bfb 100644 --- a/drivers/firmware/google/memconsole-coreboot.c +++ b/drivers/firmware/google/memconsole-coreboot.c @@ -15,9 +15,9 @@ * GNU General Public License for more details. */ +#include #include #include -#include #include "memconsole.h" #include "coreboot_table.h" @@ -73,18 +73,19 @@ static ssize_t memconsole_coreboot_read(char *buf, loff_t pos, size_t count) return done; } -static int memconsole_coreboot_init(phys_addr_t physaddr) +static int memconsole_probe(struct coreboot_device *dev) { struct cbmem_cons __iomem *tmp_cbmc; - tmp_cbmc = memremap(physaddr, sizeof(*tmp_cbmc), MEMREMAP_WB); + tmp_cbmc = memremap(dev->cbmem_ref.cbmem_addr, + sizeof(*tmp_cbmc), MEMREMAP_WB); if (!tmp_cbmc) return -ENOMEM; /* Read size only once to prevent overrun attack through /dev/mem. */ cbmem_console_size = tmp_cbmc->size_dont_access_after_boot; - cbmem_console = memremap(physaddr, + cbmem_console = memremap(dev->cbmem_ref.cbmem_addr, cbmem_console_size + sizeof(*cbmem_console), MEMREMAP_WB); memunmap(tmp_cbmc); @@ -93,26 +94,11 @@ static int memconsole_coreboot_init(phys_addr_t physaddr) return -ENOMEM; memconsole_setup(memconsole_coreboot_read); - return 0; -} - -static int memconsole_probe(struct platform_device *pdev) -{ - int ret; - struct lb_cbmem_ref entry; - - ret = coreboot_table_find(CB_TAG_CBMEM_CONSOLE, &entry, sizeof(entry)); - if (ret) - return ret; - - ret = memconsole_coreboot_init(entry.cbmem_addr); - if (ret) - return ret; return memconsole_sysfs_init(); } -static int memconsole_remove(struct platform_device *pdev) +static int memconsole_remove(struct coreboot_device *dev) { memconsole_exit(); @@ -122,28 +108,27 @@ static int memconsole_remove(struct platform_device *pdev) return 0; } -static struct platform_driver memconsole_driver = { +static struct coreboot_driver memconsole_driver = { .probe = memconsole_probe, .remove = memconsole_remove, - .driver = { + .drv = { .name = "memconsole", }, + .tag = CB_TAG_CBMEM_CONSOLE, }; -static int __init platform_memconsole_init(void) +static void coreboot_memconsole_exit(void) { - struct platform_device *pdev; - - pdev = platform_device_register_simple("memconsole", -1, NULL, 0); - if (IS_ERR(pdev)) - return PTR_ERR(pdev); - - platform_driver_register(&memconsole_driver); + coreboot_driver_unregister(&memconsole_driver); +} - return 0; +static int __init coreboot_memconsole_init(void) +{ + return coreboot_driver_register(&memconsole_driver); } -module_init(platform_memconsole_init); +module_exit(coreboot_memconsole_exit); +module_init(coreboot_memconsole_init); MODULE_AUTHOR("Google, Inc."); MODULE_LICENSE("GPL"); diff --git a/drivers/firmware/google/vpd.c b/drivers/firmware/google/vpd.c index e4b40f2b4627..e9db895916c3 100644 --- a/drivers/firmware/google/vpd.c +++ b/drivers/firmware/google/vpd.c @@ -286,20 +286,15 @@ static int vpd_sections_init(phys_addr_t physaddr) return 0; } -static int vpd_probe(struct platform_device *pdev) +static int vpd_probe(struct coreboot_device *dev) { int ret; - struct lb_cbmem_ref entry; - - ret = coreboot_table_find(CB_TAG_VPD, &entry, sizeof(entry)); - if (ret) - return ret; vpd_kobj = kobject_create_and_add("vpd", firmware_kobj); if (!vpd_kobj) return -ENOMEM; - ret = vpd_sections_init(entry.cbmem_addr); + ret = vpd_sections_init(dev->cbmem_ref.cbmem_addr); if (ret) { kobject_put(vpd_kobj); return ret; @@ -308,7 +303,7 @@ static int vpd_probe(struct platform_device *pdev) return 0; } -static int vpd_remove(struct platform_device *pdev) +static int vpd_remove(struct coreboot_device *dev) { vpd_section_destroy(&ro_vpd); vpd_section_destroy(&rw_vpd); @@ -318,41 +313,27 @@ static int vpd_remove(struct platform_device *pdev) return 0; } -static struct platform_driver vpd_driver = { +static struct coreboot_driver vpd_driver = { .probe = vpd_probe, .remove = vpd_remove, - .driver = { + .drv = { .name = "vpd", }, + .tag = CB_TAG_VPD, }; -static struct platform_device *vpd_pdev; - -static int __init vpd_platform_init(void) +static int __init coreboot_vpd_init(void) { - int ret; - - ret = platform_driver_register(&vpd_driver); - if (ret) - return ret; - - vpd_pdev = platform_device_register_simple("vpd", -1, NULL, 0); - if (IS_ERR(vpd_pdev)) { - platform_driver_unregister(&vpd_driver); - return PTR_ERR(vpd_pdev); - } - - return 0; + return coreboot_driver_register(&vpd_driver); } -static void __exit vpd_platform_exit(void) +static void __exit coreboot_vpd_exit(void) { - platform_device_unregister(vpd_pdev); - platform_driver_unregister(&vpd_driver); + coreboot_driver_unregister(&vpd_driver); } -module_init(vpd_platform_init); -module_exit(vpd_platform_exit); +module_init(coreboot_vpd_init); +module_exit(coreboot_vpd_exit); MODULE_AUTHOR("Google, Inc."); MODULE_LICENSE("GPL"); diff --git a/drivers/firmware/qcom_scm.c b/drivers/firmware/qcom_scm.c index 5a7d693009ef..e778af766fae 100644 --- a/drivers/firmware/qcom_scm.c +++ b/drivers/firmware/qcom_scm.c @@ -603,6 +603,9 @@ static const struct of_device_id qcom_scm_dt_match[] = { { .compatible = "qcom,scm-msm8996", .data = NULL, /* no clocks */ }, + { .compatible = "qcom,scm-ipq4019", + .data = NULL, /* no clocks */ + }, { .compatible = "qcom,scm", .data = (void *)(SCM_HAS_CORE_CLK | SCM_HAS_IFACE_CLK diff --git a/drivers/firmware/ti_sci.c b/drivers/firmware/ti_sci.c index 5229036dcfbf..7fa744793bc5 100644 --- a/drivers/firmware/ti_sci.c +++ b/drivers/firmware/ti_sci.c @@ -1,17 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Texas Instruments System Control Interface Protocol Driver * * Copyright (C) 2015-2016 Texas Instruments Incorporated - http://www.ti.com/ * Nishanth Menon - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed "as is" WITHOUT ANY WARRANTY of any - * kind, whether express or implied; without even the implied warranty - * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. */ #define pr_fmt(fmt) "%s: " fmt, __func__ @@ -1862,9 +1854,9 @@ static int ti_sci_probe(struct platform_device *pdev) if (!minfo->xfer_block) return -ENOMEM; - minfo->xfer_alloc_table = devm_kzalloc(dev, - BITS_TO_LONGS(desc->max_msgs) - * sizeof(unsigned long), + minfo->xfer_alloc_table = devm_kcalloc(dev, + BITS_TO_LONGS(desc->max_msgs), + sizeof(unsigned long), GFP_KERNEL); if (!minfo->xfer_alloc_table) return -ENOMEM; diff --git a/drivers/firmware/ti_sci.h b/drivers/firmware/ti_sci.h index 9b611e9e6f6d..12bf316b68df 100644 --- a/drivers/firmware/ti_sci.h +++ b/drivers/firmware/ti_sci.h @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: BSD-3-Clause /* * Texas Instruments System Control Interface (TISCI) Protocol * @@ -6,35 +7,6 @@ * See: http://processors.wiki.ti.com/index.php/TISCI for details * * Copyright (C) 2015-2016 Texas Instruments Incorporated - http://www.ti.com/ - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * - * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the - * distribution. - * - * Neither the name of Texas Instruments Incorporated nor the names of - * its contributors may be used to endorse or promote products derived - * from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * */ #ifndef __TI_SCI_H diff --git a/drivers/fmc/fmc-sdb.c b/drivers/fmc/fmc-sdb.c index ffdc1762b580..d0e65b86dc22 100644 --- a/drivers/fmc/fmc-sdb.c +++ b/drivers/fmc/fmc-sdb.c @@ -48,8 +48,8 @@ static struct sdb_array *__fmc_scan_sdb_tree(struct fmc_device *fmc, arr = kzalloc(sizeof(*arr), GFP_KERNEL); if (!arr) return ERR_PTR(-ENOMEM); - arr->record = kzalloc(sizeof(arr->record[0]) * n, GFP_KERNEL); - arr->subtree = kzalloc(sizeof(arr->subtree[0]) * n, GFP_KERNEL); + arr->record = kcalloc(n, sizeof(arr->record[0]), GFP_KERNEL); + arr->subtree = kcalloc(n, sizeof(arr->subtree[0]), GFP_KERNEL); if (!arr->record || !arr->subtree) { kfree(arr->record); kfree(arr->subtree); diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig index f47ef848bcd0..ee9c5420c47f 100644 --- a/drivers/fpga/Kconfig +++ b/drivers/fpga/Kconfig @@ -53,7 +53,6 @@ config FPGA_MGR_ALTERA_CVP config FPGA_MGR_ZYNQ_FPGA tristate "Xilinx Zynq FPGA" depends on ARCH_ZYNQ || COMPILE_TEST - depends on HAS_DMA help FPGA manager driver support for Xilinx Zynq FPGAs. @@ -70,6 +69,13 @@ config FPGA_MGR_ICE40_SPI help FPGA manager driver support for Lattice iCE40 FPGAs over SPI. +config FPGA_MGR_MACHXO2_SPI + tristate "Lattice MachXO2 SPI" + depends on SPI + help + FPGA manager driver support for Lattice MachXO2 configuration + over slave SPI interface. + config FPGA_MGR_TS73XX tristate "Technologic Systems TS-73xx SBC FPGA Manager" depends on ARCH_EP93XX && MACH_TS72XX diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile index 3cb276a0f88d..f9803dad6919 100644 --- a/drivers/fpga/Makefile +++ b/drivers/fpga/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_FPGA) += fpga-mgr.o obj-$(CONFIG_FPGA_MGR_ALTERA_CVP) += altera-cvp.o obj-$(CONFIG_FPGA_MGR_ALTERA_PS_SPI) += altera-ps-spi.o obj-$(CONFIG_FPGA_MGR_ICE40_SPI) += ice40-spi.o +obj-$(CONFIG_FPGA_MGR_MACHXO2_SPI) += machxo2-spi.o obj-$(CONFIG_FPGA_MGR_SOCFPGA) += socfpga.o obj-$(CONFIG_FPGA_MGR_SOCFPGA_A10) += socfpga-a10.o obj-$(CONFIG_FPGA_MGR_TS73XX) += ts73xx-fpga.o diff --git a/drivers/fpga/altera-cvp.c b/drivers/fpga/altera-cvp.c index 77b04e4b3254..dd4edd8f22ce 100644 --- a/drivers/fpga/altera-cvp.c +++ b/drivers/fpga/altera-cvp.c @@ -401,6 +401,7 @@ static int altera_cvp_probe(struct pci_dev *pdev, const struct pci_device_id *dev_id) { struct altera_cvp_conf *conf; + struct fpga_manager *mgr; u16 cmd, val; int ret; @@ -452,16 +453,24 @@ static int altera_cvp_probe(struct pci_dev *pdev, snprintf(conf->mgr_name, sizeof(conf->mgr_name), "%s @%s", ALTERA_CVP_MGR_NAME, pci_name(pdev)); - ret = fpga_mgr_register(&pdev->dev, conf->mgr_name, - &altera_cvp_ops, conf); - if (ret) + mgr = fpga_mgr_create(&pdev->dev, conf->mgr_name, + &altera_cvp_ops, conf); + if (!mgr) + return -ENOMEM; + + pci_set_drvdata(pdev, mgr); + + ret = fpga_mgr_register(mgr); + if (ret) { + fpga_mgr_free(mgr); goto err_unmap; + } ret = driver_create_file(&altera_cvp_driver.driver, &driver_attr_chkcfg); if (ret) { dev_err(&pdev->dev, "Can't create sysfs chkcfg file\n"); - fpga_mgr_unregister(&pdev->dev); + fpga_mgr_unregister(mgr); goto err_unmap; } @@ -483,7 +492,7 @@ static void altera_cvp_remove(struct pci_dev *pdev) u16 cmd; driver_remove_file(&altera_cvp_driver.driver, &driver_attr_chkcfg); - fpga_mgr_unregister(&pdev->dev); + fpga_mgr_unregister(mgr); pci_iounmap(pdev, conf->map); pci_release_region(pdev, CVP_BAR); pci_read_config_word(pdev, PCI_COMMAND, &cmd); diff --git a/drivers/fpga/altera-fpga2sdram.c b/drivers/fpga/altera-fpga2sdram.c index d4eeb74388da..23660ccd634b 100644 --- a/drivers/fpga/altera-fpga2sdram.c +++ b/drivers/fpga/altera-fpga2sdram.c @@ -1,19 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * FPGA to SDRAM Bridge Driver for Altera SoCFPGA Devices * * Copyright (C) 2013-2016 Altera Corporation, All Rights Reserved. - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ /* @@ -106,6 +95,7 @@ static int alt_fpga_bridge_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct alt_fpga2sdram_data *priv; + struct fpga_bridge *br; u32 enable; struct regmap *sysmgr; int ret = 0; @@ -131,10 +121,18 @@ static int alt_fpga_bridge_probe(struct platform_device *pdev) /* Get f2s bridge configuration saved in handoff register */ regmap_read(sysmgr, SYSMGR_ISWGRP_HANDOFF3, &priv->mask); - ret = fpga_bridge_register(dev, F2S_BRIDGE_NAME, - &altera_fpga2sdram_br_ops, priv); - if (ret) + br = fpga_bridge_create(dev, F2S_BRIDGE_NAME, + &altera_fpga2sdram_br_ops, priv); + if (!br) + return -ENOMEM; + + platform_set_drvdata(pdev, br); + + ret = fpga_bridge_register(br); + if (ret) { + fpga_bridge_free(br); return ret; + } dev_info(dev, "driver initialized with handoff %08x\n", priv->mask); @@ -146,7 +144,7 @@ static int alt_fpga_bridge_probe(struct platform_device *pdev) (enable ? "enabling" : "disabling")); ret = _alt_fpga2sdram_enable_set(priv, enable); if (ret) { - fpga_bridge_unregister(&pdev->dev); + fpga_bridge_unregister(br); return ret; } } @@ -157,7 +155,9 @@ static int alt_fpga_bridge_probe(struct platform_device *pdev) static int alt_fpga_bridge_remove(struct platform_device *pdev) { - fpga_bridge_unregister(&pdev->dev); + struct fpga_bridge *br = platform_get_drvdata(pdev); + + fpga_bridge_unregister(br); return 0; } diff --git a/drivers/fpga/altera-freeze-bridge.c b/drivers/fpga/altera-freeze-bridge.c index 6159cfcf78a2..ffd586c48ecf 100644 --- a/drivers/fpga/altera-freeze-bridge.c +++ b/drivers/fpga/altera-freeze-bridge.c @@ -1,19 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * FPGA Freeze Bridge Controller * * Copyright (C) 2016 Altera Corporation. All rights reserved. - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include #include @@ -221,8 +210,10 @@ static int altera_freeze_br_probe(struct platform_device *pdev) struct device_node *np = pdev->dev.of_node; void __iomem *base_addr; struct altera_freeze_br_data *priv; + struct fpga_bridge *br; struct resource *res; u32 status, revision; + int ret; if (!np) return -ENODEV; @@ -254,13 +245,27 @@ static int altera_freeze_br_probe(struct platform_device *pdev) priv->base_addr = base_addr; - return fpga_bridge_register(dev, FREEZE_BRIDGE_NAME, - &altera_freeze_br_br_ops, priv); + br = fpga_bridge_create(dev, FREEZE_BRIDGE_NAME, + &altera_freeze_br_br_ops, priv); + if (!br) + return -ENOMEM; + + platform_set_drvdata(pdev, br); + + ret = fpga_bridge_register(br); + if (ret) { + fpga_bridge_free(br); + return ret; + } + + return 0; } static int altera_freeze_br_remove(struct platform_device *pdev) { - fpga_bridge_unregister(&pdev->dev); + struct fpga_bridge *br = platform_get_drvdata(pdev); + + fpga_bridge_unregister(br); return 0; } diff --git a/drivers/fpga/altera-hps2fpga.c b/drivers/fpga/altera-hps2fpga.c index 406d2f10741f..a974d3f60321 100644 --- a/drivers/fpga/altera-hps2fpga.c +++ b/drivers/fpga/altera-hps2fpga.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * FPGA to/from HPS Bridge Driver for Altera SoCFPGA Devices * @@ -6,18 +7,6 @@ * Includes this patch from the mailing list: * fpga: altera-hps2fpga: fix HPS2FPGA bridge visibility to L3 masters * Signed-off-by: Anatolij Gustschin - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ /* @@ -139,6 +128,7 @@ static int alt_fpga_bridge_probe(struct platform_device *pdev) struct device *dev = &pdev->dev; struct altera_hps2fpga_data *priv; const struct of_device_id *of_id; + struct fpga_bridge *br; u32 enable; int ret; @@ -190,11 +180,24 @@ static int alt_fpga_bridge_probe(struct platform_device *pdev) } } - ret = fpga_bridge_register(dev, priv->name, &altera_hps2fpga_br_ops, - priv); -err: + br = fpga_bridge_create(dev, priv->name, &altera_hps2fpga_br_ops, priv); + if (!br) { + ret = -ENOMEM; + goto err; + } + + platform_set_drvdata(pdev, br); + + ret = fpga_bridge_register(br); if (ret) - clk_disable_unprepare(priv->clk); + goto err_free; + + return 0; + +err_free: + fpga_bridge_free(br); +err: + clk_disable_unprepare(priv->clk); return ret; } @@ -204,7 +207,7 @@ static int alt_fpga_bridge_remove(struct platform_device *pdev) struct fpga_bridge *bridge = platform_get_drvdata(pdev); struct altera_hps2fpga_data *priv = bridge->priv; - fpga_bridge_unregister(&pdev->dev); + fpga_bridge_unregister(bridge); clk_disable_unprepare(priv->clk); diff --git a/drivers/fpga/altera-pr-ip-core-plat.c b/drivers/fpga/altera-pr-ip-core-plat.c index 8fb36b8b4648..b293d83143f1 100644 --- a/drivers/fpga/altera-pr-ip-core-plat.c +++ b/drivers/fpga/altera-pr-ip-core-plat.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Driver for Altera Partial Reconfiguration IP Core * @@ -5,18 +6,6 @@ * * Based on socfpga-a10.c Copyright (C) 2015-2016 Altera Corporation * by Alan Tull - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include #include diff --git a/drivers/fpga/altera-pr-ip-core.c b/drivers/fpga/altera-pr-ip-core.c index a7b31f9797ce..65e0b6a2c031 100644 --- a/drivers/fpga/altera-pr-ip-core.c +++ b/drivers/fpga/altera-pr-ip-core.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * Driver for Altera Partial Reconfiguration IP Core * @@ -5,18 +6,6 @@ * * Based on socfpga-a10.c Copyright (C) 2015-2016 Altera Corporation * by Alan Tull - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include #include @@ -187,6 +176,8 @@ static const struct fpga_manager_ops alt_pr_ops = { int alt_pr_register(struct device *dev, void __iomem *reg_base) { struct alt_pr_priv *priv; + struct fpga_manager *mgr; + int ret; u32 val; priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL); @@ -201,15 +192,27 @@ int alt_pr_register(struct device *dev, void __iomem *reg_base) (val & ALT_PR_CSR_STATUS_MSK) >> ALT_PR_CSR_STATUS_SFT, (int)(val & ALT_PR_CSR_PR_START)); - return fpga_mgr_register(dev, dev_name(dev), &alt_pr_ops, priv); + mgr = fpga_mgr_create(dev, dev_name(dev), &alt_pr_ops, priv); + if (!mgr) + return -ENOMEM; + + dev_set_drvdata(dev, mgr); + + ret = fpga_mgr_register(mgr); + if (ret) + fpga_mgr_free(mgr); + + return ret; } EXPORT_SYMBOL_GPL(alt_pr_register); int alt_pr_unregister(struct device *dev) { + struct fpga_manager *mgr = dev_get_drvdata(dev); + dev_dbg(dev, "%s\n", __func__); - fpga_mgr_unregister(dev); + fpga_mgr_unregister(mgr); return 0; } diff --git a/drivers/fpga/altera-ps-spi.c b/drivers/fpga/altera-ps-spi.c index 06d212a3d49d..24b25c626036 100644 --- a/drivers/fpga/altera-ps-spi.c +++ b/drivers/fpga/altera-ps-spi.c @@ -238,6 +238,8 @@ static int altera_ps_probe(struct spi_device *spi) { struct altera_ps_conf *conf; const struct of_device_id *of_id; + struct fpga_manager *mgr; + int ret; conf = devm_kzalloc(&spi->dev, sizeof(*conf), GFP_KERNEL); if (!conf) @@ -273,13 +275,25 @@ static int altera_ps_probe(struct spi_device *spi) snprintf(conf->mgr_name, sizeof(conf->mgr_name), "%s %s", dev_driver_string(&spi->dev), dev_name(&spi->dev)); - return fpga_mgr_register(&spi->dev, conf->mgr_name, - &altera_ps_ops, conf); + mgr = fpga_mgr_create(&spi->dev, conf->mgr_name, + &altera_ps_ops, conf); + if (!mgr) + return -ENOMEM; + + spi_set_drvdata(spi, mgr); + + ret = fpga_mgr_register(mgr); + if (ret) + fpga_mgr_free(mgr); + + return ret; } static int altera_ps_remove(struct spi_device *spi) { - fpga_mgr_unregister(&spi->dev); + struct fpga_manager *mgr = spi_get_drvdata(spi); + + fpga_mgr_unregister(mgr); return 0; } diff --git a/drivers/fpga/fpga-bridge.c b/drivers/fpga/fpga-bridge.c index 31bd2c59c305..24b8f98b73ec 100644 --- a/drivers/fpga/fpga-bridge.c +++ b/drivers/fpga/fpga-bridge.c @@ -1,20 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * FPGA Bridge Framework Driver * * Copyright (C) 2013-2016 Altera Corporation, All Rights Reserved. * Copyright (C) 2017 Intel Corporation - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include #include @@ -132,6 +121,7 @@ static int fpga_bridge_dev_match(struct device *dev, const void *data) /** * fpga_bridge_get - get an exclusive reference to a fpga bridge * @dev: parent device that fpga bridge was registered with + * @info: fpga manager info * * Given a device, get an exclusive reference to a fpga bridge. * @@ -328,28 +318,29 @@ static struct attribute *fpga_bridge_attrs[] = { ATTRIBUTE_GROUPS(fpga_bridge); /** - * fpga_bridge_register - register a fpga bridge driver + * fpga_bridge_create - create and initialize a struct fpga_bridge * @dev: FPGA bridge device from pdev * @name: FPGA bridge name * @br_ops: pointer to structure of fpga bridge ops * @priv: FPGA bridge private data * - * Return: 0 for success, error code otherwise. + * Return: struct fpga_bridge or NULL */ -int fpga_bridge_register(struct device *dev, const char *name, - const struct fpga_bridge_ops *br_ops, void *priv) +struct fpga_bridge *fpga_bridge_create(struct device *dev, const char *name, + const struct fpga_bridge_ops *br_ops, + void *priv) { struct fpga_bridge *bridge; int id, ret = 0; if (!name || !strlen(name)) { dev_err(dev, "Attempt to register with no name!\n"); - return -EINVAL; + return NULL; } bridge = kzalloc(sizeof(*bridge), GFP_KERNEL); if (!bridge) - return -ENOMEM; + return NULL; id = ida_simple_get(&fpga_bridge_ida, 0, 0, GFP_KERNEL); if (id < 0) { @@ -370,40 +361,62 @@ int fpga_bridge_register(struct device *dev, const char *name, bridge->dev.parent = dev; bridge->dev.of_node = dev->of_node; bridge->dev.id = id; - dev_set_drvdata(dev, bridge); ret = dev_set_name(&bridge->dev, "br%d", id); if (ret) goto error_device; - ret = device_add(&bridge->dev); - if (ret) - goto error_device; - - of_platform_populate(dev->of_node, NULL, NULL, dev); - - dev_info(bridge->dev.parent, "fpga bridge [%s] registered\n", - bridge->name); - - return 0; + return bridge; error_device: ida_simple_remove(&fpga_bridge_ida, id); error_kfree: kfree(bridge); - return ret; + return NULL; } -EXPORT_SYMBOL_GPL(fpga_bridge_register); +EXPORT_SYMBOL_GPL(fpga_bridge_create); /** - * fpga_bridge_unregister - unregister a fpga bridge driver - * @dev: FPGA bridge device from pdev + * fpga_bridge_free - free a fpga bridge and its id + * @bridge: FPGA bridge struct created by fpga_bridge_create */ -void fpga_bridge_unregister(struct device *dev) +void fpga_bridge_free(struct fpga_bridge *bridge) { - struct fpga_bridge *bridge = dev_get_drvdata(dev); + ida_simple_remove(&fpga_bridge_ida, bridge->dev.id); + kfree(bridge); +} +EXPORT_SYMBOL_GPL(fpga_bridge_free); +/** + * fpga_bridge_register - register a fpga bridge + * @bridge: FPGA bridge struct created by fpga_bridge_create + * + * Return: 0 for success, error code otherwise. + */ +int fpga_bridge_register(struct fpga_bridge *bridge) +{ + struct device *dev = &bridge->dev; + int ret; + + ret = device_add(dev); + if (ret) + return ret; + + of_platform_populate(dev->of_node, NULL, NULL, dev); + + dev_info(dev->parent, "fpga bridge [%s] registered\n", bridge->name); + + return 0; +} +EXPORT_SYMBOL_GPL(fpga_bridge_register); + +/** + * fpga_bridge_unregister - unregister and free a fpga bridge + * @bridge: FPGA bridge struct created by fpga_bridge_create + */ +void fpga_bridge_unregister(struct fpga_bridge *bridge) +{ /* * If the low level driver provides a method for putting bridge into * a desired state upon unregister, do it. @@ -419,8 +432,7 @@ static void fpga_bridge_dev_release(struct device *dev) { struct fpga_bridge *bridge = to_fpga_bridge(dev); - ida_simple_remove(&fpga_bridge_ida, bridge->dev.id); - kfree(bridge); + fpga_bridge_free(bridge); } static int __init fpga_bridge_dev_init(void) diff --git a/drivers/fpga/fpga-mgr.c b/drivers/fpga/fpga-mgr.c index 9939d2cbc9a6..c1564cf827fe 100644 --- a/drivers/fpga/fpga-mgr.c +++ b/drivers/fpga/fpga-mgr.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * FPGA Manager Core * @@ -6,18 +7,6 @@ * * With code from the mailing list: * Copyright (C) 2013 Xilinx, Inc. - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include #include @@ -32,6 +21,12 @@ static DEFINE_IDA(fpga_mgr_ida); static struct class *fpga_mgr_class; +/** + * fpga_image_info_alloc - Allocate a FPGA image info struct + * @dev: owning device + * + * Return: struct fpga_image_info or NULL + */ struct fpga_image_info *fpga_image_info_alloc(struct device *dev) { struct fpga_image_info *info; @@ -50,6 +45,10 @@ struct fpga_image_info *fpga_image_info_alloc(struct device *dev) } EXPORT_SYMBOL_GPL(fpga_image_info_alloc); +/** + * fpga_image_info_free - Free a FPGA image info struct + * @info: FPGA image info struct to free + */ void fpga_image_info_free(struct fpga_image_info *info) { struct device *dev; @@ -234,7 +233,7 @@ static int fpga_mgr_buf_load_mapped(struct fpga_manager *mgr, /** * fpga_mgr_buf_load - load fpga from image in buffer * @mgr: fpga manager - * @flags: flags setting fpga confuration modes + * @info: fpga image info * @buf: buffer contain fpga image * @count: byte count of buf * @@ -343,6 +342,16 @@ static int fpga_mgr_firmware_load(struct fpga_manager *mgr, return ret; } +/** + * fpga_mgr_load - load FPGA from scatter/gather table, buffer, or firmware + * @mgr: fpga manager + * @info: fpga image information. + * + * Load the FPGA from an image which is indicated in @info. If successful, the + * FPGA ends up in operating mode. + * + * Return: 0 on success, negative error code otherwise. + */ int fpga_mgr_load(struct fpga_manager *mgr, struct fpga_image_info *info) { if (info->sgt) @@ -429,11 +438,9 @@ static int fpga_mgr_dev_match(struct device *dev, const void *data) } /** - * fpga_mgr_get - get a reference to a fpga mgr + * fpga_mgr_get - Given a device, get a reference to a fpga mgr. * @dev: parent device that fpga mgr was registered with * - * Given a device, get a reference to a fpga mgr. - * * Return: fpga manager struct or IS_ERR() condition containing error code. */ struct fpga_manager *fpga_mgr_get(struct device *dev) @@ -453,10 +460,9 @@ static int fpga_mgr_of_node_match(struct device *dev, const void *data) } /** - * of_fpga_mgr_get - get a reference to a fpga mgr - * @node: device node + * of_fpga_mgr_get - Given a device node, get a reference to a fpga mgr. * - * Given a device node, get a reference to a fpga mgr. + * @node: device node * * Return: fpga manager struct or IS_ERR() condition containing error code. */ @@ -489,7 +495,10 @@ EXPORT_SYMBOL_GPL(fpga_mgr_put); * @mgr: fpga manager * * Given a pointer to FPGA Manager (from fpga_mgr_get() or - * of_fpga_mgr_put()) attempt to get the mutex. + * of_fpga_mgr_put()) attempt to get the mutex. The user should call + * fpga_mgr_lock() and verify that it returns 0 before attempting to + * program the FPGA. Likewise, the user should call fpga_mgr_unlock + * when done programming the FPGA. * * Return: 0 for success or -EBUSY */ @@ -505,7 +514,7 @@ int fpga_mgr_lock(struct fpga_manager *mgr) EXPORT_SYMBOL_GPL(fpga_mgr_lock); /** - * fpga_mgr_unlock - Unlock FPGA manager + * fpga_mgr_unlock - Unlock FPGA manager after done programming * @mgr: fpga manager */ void fpga_mgr_unlock(struct fpga_manager *mgr) @@ -515,17 +524,17 @@ void fpga_mgr_unlock(struct fpga_manager *mgr) EXPORT_SYMBOL_GPL(fpga_mgr_unlock); /** - * fpga_mgr_register - register a low level fpga manager driver + * fpga_mgr_create - create and initialize a FPGA manager struct * @dev: fpga manager device from pdev * @name: fpga manager name * @mops: pointer to structure of fpga manager ops * @priv: fpga manager private data * - * Return: 0 on success, negative error code otherwise. + * Return: pointer to struct fpga_manager or NULL */ -int fpga_mgr_register(struct device *dev, const char *name, - const struct fpga_manager_ops *mops, - void *priv) +struct fpga_manager *fpga_mgr_create(struct device *dev, const char *name, + const struct fpga_manager_ops *mops, + void *priv) { struct fpga_manager *mgr; int id, ret; @@ -534,17 +543,17 @@ int fpga_mgr_register(struct device *dev, const char *name, !mops->write_init || (!mops->write && !mops->write_sg) || (mops->write && mops->write_sg)) { dev_err(dev, "Attempt to register without fpga_manager_ops\n"); - return -EINVAL; + return NULL; } if (!name || !strlen(name)) { dev_err(dev, "Attempt to register with no name!\n"); - return -EINVAL; + return NULL; } mgr = kzalloc(sizeof(*mgr), GFP_KERNEL); if (!mgr) - return -ENOMEM; + return NULL; id = ida_simple_get(&fpga_mgr_ida, 0, 0, GFP_KERNEL); if (id < 0) { @@ -558,25 +567,56 @@ int fpga_mgr_register(struct device *dev, const char *name, mgr->mops = mops; mgr->priv = priv; - /* - * Initialize framework state by requesting low level driver read state - * from device. FPGA may be in reset mode or may have been programmed - * by bootloader or EEPROM. - */ - mgr->state = mgr->mops->state(mgr); - device_initialize(&mgr->dev); mgr->dev.class = fpga_mgr_class; mgr->dev.groups = mops->groups; mgr->dev.parent = dev; mgr->dev.of_node = dev->of_node; mgr->dev.id = id; - dev_set_drvdata(dev, mgr); ret = dev_set_name(&mgr->dev, "fpga%d", id); if (ret) goto error_device; + return mgr; + +error_device: + ida_simple_remove(&fpga_mgr_ida, id); +error_kfree: + kfree(mgr); + + return NULL; +} +EXPORT_SYMBOL_GPL(fpga_mgr_create); + +/** + * fpga_mgr_free - deallocate a FPGA manager + * @mgr: fpga manager struct created by fpga_mgr_create + */ +void fpga_mgr_free(struct fpga_manager *mgr) +{ + ida_simple_remove(&fpga_mgr_ida, mgr->dev.id); + kfree(mgr); +} +EXPORT_SYMBOL_GPL(fpga_mgr_free); + +/** + * fpga_mgr_register - register a FPGA manager + * @mgr: fpga manager struct created by fpga_mgr_create + * + * Return: 0 on success, negative error code otherwise. + */ +int fpga_mgr_register(struct fpga_manager *mgr) +{ + int ret; + + /* + * Initialize framework state by requesting low level driver read state + * from device. FPGA may be in reset mode or may have been programmed + * by bootloader or EEPROM. + */ + mgr->state = mgr->mops->state(mgr); + ret = device_add(&mgr->dev); if (ret) goto error_device; @@ -586,22 +626,18 @@ int fpga_mgr_register(struct device *dev, const char *name, return 0; error_device: - ida_simple_remove(&fpga_mgr_ida, id); -error_kfree: - kfree(mgr); + ida_simple_remove(&fpga_mgr_ida, mgr->dev.id); return ret; } EXPORT_SYMBOL_GPL(fpga_mgr_register); /** - * fpga_mgr_unregister - unregister a low level fpga manager driver - * @dev: fpga manager device from pdev + * fpga_mgr_unregister - unregister and free a FPGA manager + * @mgr: fpga manager struct */ -void fpga_mgr_unregister(struct device *dev) +void fpga_mgr_unregister(struct fpga_manager *mgr) { - struct fpga_manager *mgr = dev_get_drvdata(dev); - dev_info(&mgr->dev, "%s %s\n", __func__, mgr->name); /* @@ -619,8 +655,7 @@ static void fpga_mgr_dev_release(struct device *dev) { struct fpga_manager *mgr = to_fpga_manager(dev); - ida_simple_remove(&fpga_mgr_ida, mgr->dev.id); - kfree(mgr); + fpga_mgr_free(mgr); } static int __init fpga_mgr_class_init(void) diff --git a/drivers/fpga/fpga-region.c b/drivers/fpga/fpga-region.c index edab2a2e03ef..6d214d75c7be 100644 --- a/drivers/fpga/fpga-region.c +++ b/drivers/fpga/fpga-region.c @@ -1,22 +1,10 @@ +// SPDX-License-Identifier: GPL-2.0 /* * FPGA Region - Device Tree support for FPGA programming under Linux * * Copyright (C) 2013-2016 Altera Corporation * Copyright (C) 2017 Intel Corporation - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ - #include #include #include @@ -93,8 +81,16 @@ static void fpga_region_put(struct fpga_region *region) /** * fpga_region_program_fpga - program FPGA + * * @region: FPGA region + * * Program an FPGA using fpga image info (region->info). + * If the region has a get_bridges function, the exclusive reference for the + * bridges will be held if programming succeeds. This is intended to prevent + * reprogramming the region until the caller considers it safe to do so. + * The caller will need to call fpga_bridges_put() before attempting to + * reprogram the region. + * * Return 0 for success or negative error code. */ int fpga_region_program_fpga(struct fpga_region *region) @@ -162,45 +158,86 @@ err_put_region: } EXPORT_SYMBOL_GPL(fpga_region_program_fpga); -int fpga_region_register(struct device *dev, struct fpga_region *region) +/** + * fpga_region_create - alloc and init a struct fpga_region + * @dev: device parent + * @mgr: manager that programs this region + * @get_bridges: optional function to get bridges to a list + * + * Return: struct fpga_region or NULL + */ +struct fpga_region +*fpga_region_create(struct device *dev, + struct fpga_manager *mgr, + int (*get_bridges)(struct fpga_region *)) { + struct fpga_region *region; int id, ret = 0; + region = kzalloc(sizeof(*region), GFP_KERNEL); + if (!region) + return NULL; + id = ida_simple_get(&fpga_region_ida, 0, 0, GFP_KERNEL); if (id < 0) - return id; + goto err_free; + region->mgr = mgr; + region->get_bridges = get_bridges; mutex_init(®ion->mutex); INIT_LIST_HEAD(®ion->bridge_list); + device_initialize(®ion->dev); - region->dev.groups = region->groups; region->dev.class = fpga_region_class; region->dev.parent = dev; region->dev.of_node = dev->of_node; region->dev.id = id; - dev_set_drvdata(dev, region); ret = dev_set_name(®ion->dev, "region%d", id); if (ret) goto err_remove; - ret = device_add(®ion->dev); - if (ret) - goto err_remove; - - return 0; + return region; err_remove: ida_simple_remove(&fpga_region_ida, id); - return ret; +err_free: + kfree(region); + + return NULL; +} +EXPORT_SYMBOL_GPL(fpga_region_create); + +/** + * fpga_region_free - free a struct fpga_region + * @region: FPGA region created by fpga_region_create + */ +void fpga_region_free(struct fpga_region *region) +{ + ida_simple_remove(&fpga_region_ida, region->dev.id); + kfree(region); +} +EXPORT_SYMBOL_GPL(fpga_region_free); + +/** + * fpga_region_register - register a FPGA region + * @region: FPGA region created by fpga_region_create + * Return: 0 or -errno + */ +int fpga_region_register(struct fpga_region *region) +{ + return device_add(®ion->dev); + } EXPORT_SYMBOL_GPL(fpga_region_register); -int fpga_region_unregister(struct fpga_region *region) +/** + * fpga_region_unregister - unregister and free a FPGA region + * @region: FPGA region + */ +void fpga_region_unregister(struct fpga_region *region) { device_unregister(®ion->dev); - - return 0; } EXPORT_SYMBOL_GPL(fpga_region_unregister); @@ -208,7 +245,7 @@ static void fpga_region_dev_release(struct device *dev) { struct fpga_region *region = to_fpga_region(dev); - ida_simple_remove(&fpga_region_ida, region->dev.id); + fpga_region_free(region); } /** diff --git a/drivers/fpga/ice40-spi.c b/drivers/fpga/ice40-spi.c index 7fca82023062..5981c7ee7a7d 100644 --- a/drivers/fpga/ice40-spi.c +++ b/drivers/fpga/ice40-spi.c @@ -133,6 +133,7 @@ static int ice40_fpga_probe(struct spi_device *spi) { struct device *dev = &spi->dev; struct ice40_fpga_priv *priv; + struct fpga_manager *mgr; int ret; priv = devm_kzalloc(&spi->dev, sizeof(*priv), GFP_KERNEL); @@ -174,14 +175,26 @@ static int ice40_fpga_probe(struct spi_device *spi) return ret; } - /* Register with the FPGA manager */ - return fpga_mgr_register(dev, "Lattice iCE40 FPGA Manager", - &ice40_fpga_ops, priv); + mgr = fpga_mgr_create(dev, "Lattice iCE40 FPGA Manager", + &ice40_fpga_ops, priv); + if (!mgr) + return -ENOMEM; + + spi_set_drvdata(spi, mgr); + + ret = fpga_mgr_register(mgr); + if (ret) + fpga_mgr_free(mgr); + + return ret; } static int ice40_fpga_remove(struct spi_device *spi) { - fpga_mgr_unregister(&spi->dev); + struct fpga_manager *mgr = spi_get_drvdata(spi); + + fpga_mgr_unregister(mgr); + return 0; } diff --git a/drivers/fpga/machxo2-spi.c b/drivers/fpga/machxo2-spi.c new file mode 100644 index 000000000000..a582e0000c97 --- /dev/null +++ b/drivers/fpga/machxo2-spi.c @@ -0,0 +1,415 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Lattice MachXO2 Slave SPI Driver + * + * Manage Lattice FPGA firmware that is loaded over SPI using + * the slave serial configuration interface. + * + * Copyright (C) 2018 Paolo Pisati + */ + +#include +#include +#include +#include +#include +#include + +/* MachXO2 Programming Guide - sysCONFIG Programming Commands */ +#define IDCODE_PUB {0xe0, 0x00, 0x00, 0x00} +#define ISC_ENABLE {0xc6, 0x08, 0x00, 0x00} +#define ISC_ERASE {0x0e, 0x04, 0x00, 0x00} +#define ISC_PROGRAMDONE {0x5e, 0x00, 0x00, 0x00} +#define LSC_INITADDRESS {0x46, 0x00, 0x00, 0x00} +#define LSC_PROGINCRNV {0x70, 0x00, 0x00, 0x01} +#define LSC_READ_STATUS {0x3c, 0x00, 0x00, 0x00} +#define LSC_REFRESH {0x79, 0x00, 0x00, 0x00} + +/* + * Max CCLK in Slave SPI mode according to 'MachXO2 Family Data + * Sheet' sysCONFIG Port Timing Specifications (3-36) + */ +#define MACHXO2_MAX_SPEED 66000000 + +#define MACHXO2_LOW_DELAY_USEC 5 +#define MACHXO2_HIGH_DELAY_USEC 200 +#define MACHXO2_REFRESH_USEC 4800 +#define MACHXO2_MAX_BUSY_LOOP 128 +#define MACHXO2_MAX_REFRESH_LOOP 16 + +#define MACHXO2_PAGE_SIZE 16 +#define MACHXO2_BUF_SIZE (MACHXO2_PAGE_SIZE + 4) + +/* Status register bits, errors and error mask */ +#define BUSY 12 +#define DONE 8 +#define DVER 27 +#define ENAB 9 +#define ERRBITS 23 +#define ERRMASK 7 +#define FAIL 13 + +#define ENOERR 0 /* no error */ +#define EID 1 +#define ECMD 2 +#define ECRC 3 +#define EPREAM 4 /* preamble error */ +#define EABRT 5 /* abort error */ +#define EOVERFL 6 /* overflow error */ +#define ESDMEOF 7 /* SDM EOF */ + +static inline u8 get_err(unsigned long *status) +{ + return (*status >> ERRBITS) & ERRMASK; +} + +static int get_status(struct spi_device *spi, unsigned long *status) +{ + struct spi_message msg; + struct spi_transfer rx, tx; + static const u8 cmd[] = LSC_READ_STATUS; + int ret; + + memset(&rx, 0, sizeof(rx)); + memset(&tx, 0, sizeof(tx)); + tx.tx_buf = cmd; + tx.len = sizeof(cmd); + rx.rx_buf = status; + rx.len = 4; + spi_message_init(&msg); + spi_message_add_tail(&tx, &msg); + spi_message_add_tail(&rx, &msg); + ret = spi_sync(spi, &msg); + if (ret) + return ret; + + *status = be32_to_cpu(*status); + + return 0; +} + +#ifdef DEBUG +static const char *get_err_string(u8 err) +{ + switch (err) { + case ENOERR: return "No Error"; + case EID: return "ID ERR"; + case ECMD: return "CMD ERR"; + case ECRC: return "CRC ERR"; + case EPREAM: return "Preamble ERR"; + case EABRT: return "Abort ERR"; + case EOVERFL: return "Overflow ERR"; + case ESDMEOF: return "SDM EOF"; + } + + return "Default switch case"; +} +#endif + +static void dump_status_reg(unsigned long *status) +{ +#ifdef DEBUG + pr_debug("machxo2 status: 0x%08lX - done=%d, cfgena=%d, busy=%d, fail=%d, devver=%d, err=%s\n", + *status, test_bit(DONE, status), test_bit(ENAB, status), + test_bit(BUSY, status), test_bit(FAIL, status), + test_bit(DVER, status), get_err_string(get_err(status))); +#endif +} + +static int wait_until_not_busy(struct spi_device *spi) +{ + unsigned long status; + int ret, loop = 0; + + do { + ret = get_status(spi, &status); + if (ret) + return ret; + if (++loop >= MACHXO2_MAX_BUSY_LOOP) + return -EBUSY; + } while (test_bit(BUSY, &status)); + + return 0; +} + +static int machxo2_cleanup(struct fpga_manager *mgr) +{ + struct spi_device *spi = mgr->priv; + struct spi_message msg; + struct spi_transfer tx[2]; + static const u8 erase[] = ISC_ERASE; + static const u8 refresh[] = LSC_REFRESH; + int ret; + + memset(tx, 0, sizeof(tx)); + spi_message_init(&msg); + tx[0].tx_buf = &erase; + tx[0].len = sizeof(erase); + spi_message_add_tail(&tx[0], &msg); + ret = spi_sync(spi, &msg); + if (ret) + goto fail; + + ret = wait_until_not_busy(spi); + if (ret) + goto fail; + + spi_message_init(&msg); + tx[1].tx_buf = &refresh; + tx[1].len = sizeof(refresh); + tx[1].delay_usecs = MACHXO2_REFRESH_USEC; + spi_message_add_tail(&tx[1], &msg); + ret = spi_sync(spi, &msg); + if (ret) + goto fail; + + return 0; +fail: + dev_err(&mgr->dev, "Cleanup failed\n"); + + return ret; +} + +static enum fpga_mgr_states machxo2_spi_state(struct fpga_manager *mgr) +{ + struct spi_device *spi = mgr->priv; + unsigned long status; + + get_status(spi, &status); + if (!test_bit(BUSY, &status) && test_bit(DONE, &status) && + get_err(&status) == ENOERR) + return FPGA_MGR_STATE_OPERATING; + + return FPGA_MGR_STATE_UNKNOWN; +} + +static int machxo2_write_init(struct fpga_manager *mgr, + struct fpga_image_info *info, + const char *buf, size_t count) +{ + struct spi_device *spi = mgr->priv; + struct spi_message msg; + struct spi_transfer tx[3]; + static const u8 enable[] = ISC_ENABLE; + static const u8 erase[] = ISC_ERASE; + static const u8 initaddr[] = LSC_INITADDRESS; + unsigned long status; + int ret; + + if ((info->flags & FPGA_MGR_PARTIAL_RECONFIG)) { + dev_err(&mgr->dev, + "Partial reconfiguration is not supported\n"); + return -ENOTSUPP; + } + + get_status(spi, &status); + dump_status_reg(&status); + memset(tx, 0, sizeof(tx)); + spi_message_init(&msg); + tx[0].tx_buf = &enable; + tx[0].len = sizeof(enable); + tx[0].delay_usecs = MACHXO2_LOW_DELAY_USEC; + spi_message_add_tail(&tx[0], &msg); + + tx[1].tx_buf = &erase; + tx[1].len = sizeof(erase); + spi_message_add_tail(&tx[1], &msg); + ret = spi_sync(spi, &msg); + if (ret) + goto fail; + + ret = wait_until_not_busy(spi); + if (ret) + goto fail; + + get_status(spi, &status); + if (test_bit(FAIL, &status)) + goto fail; + dump_status_reg(&status); + + spi_message_init(&msg); + tx[2].tx_buf = &initaddr; + tx[2].len = sizeof(initaddr); + spi_message_add_tail(&tx[2], &msg); + ret = spi_sync(spi, &msg); + if (ret) + goto fail; + + get_status(spi, &status); + dump_status_reg(&status); + + return 0; +fail: + dev_err(&mgr->dev, "Error during FPGA init.\n"); + + return ret; +} + +static int machxo2_write(struct fpga_manager *mgr, const char *buf, + size_t count) +{ + struct spi_device *spi = mgr->priv; + struct spi_message msg; + struct spi_transfer tx; + static const u8 progincr[] = LSC_PROGINCRNV; + u8 payload[MACHXO2_BUF_SIZE]; + unsigned long status; + int i, ret; + + if (count % MACHXO2_PAGE_SIZE != 0) { + dev_err(&mgr->dev, "Malformed payload.\n"); + return -EINVAL; + } + get_status(spi, &status); + dump_status_reg(&status); + memcpy(payload, &progincr, sizeof(progincr)); + for (i = 0; i < count; i += MACHXO2_PAGE_SIZE) { + memcpy(&payload[sizeof(progincr)], &buf[i], MACHXO2_PAGE_SIZE); + memset(&tx, 0, sizeof(tx)); + spi_message_init(&msg); + tx.tx_buf = payload; + tx.len = MACHXO2_BUF_SIZE; + tx.delay_usecs = MACHXO2_HIGH_DELAY_USEC; + spi_message_add_tail(&tx, &msg); + ret = spi_sync(spi, &msg); + if (ret) { + dev_err(&mgr->dev, "Error loading the bitstream.\n"); + return ret; + } + } + get_status(spi, &status); + dump_status_reg(&status); + + return 0; +} + +static int machxo2_write_complete(struct fpga_manager *mgr, + struct fpga_image_info *info) +{ + struct spi_device *spi = mgr->priv; + struct spi_message msg; + struct spi_transfer tx[2]; + static const u8 progdone[] = ISC_PROGRAMDONE; + static const u8 refresh[] = LSC_REFRESH; + unsigned long status; + int ret, refreshloop = 0; + + memset(tx, 0, sizeof(tx)); + spi_message_init(&msg); + tx[0].tx_buf = &progdone; + tx[0].len = sizeof(progdone); + spi_message_add_tail(&tx[0], &msg); + ret = spi_sync(spi, &msg); + if (ret) + goto fail; + ret = wait_until_not_busy(spi); + if (ret) + goto fail; + + get_status(spi, &status); + dump_status_reg(&status); + if (!test_bit(DONE, &status)) { + machxo2_cleanup(mgr); + goto fail; + } + + do { + spi_message_init(&msg); + tx[1].tx_buf = &refresh; + tx[1].len = sizeof(refresh); + tx[1].delay_usecs = MACHXO2_REFRESH_USEC; + spi_message_add_tail(&tx[1], &msg); + ret = spi_sync(spi, &msg); + if (ret) + goto fail; + + /* check refresh status */ + get_status(spi, &status); + dump_status_reg(&status); + if (!test_bit(BUSY, &status) && test_bit(DONE, &status) && + get_err(&status) == ENOERR) + break; + if (++refreshloop == MACHXO2_MAX_REFRESH_LOOP) { + machxo2_cleanup(mgr); + goto fail; + } + } while (1); + + get_status(spi, &status); + dump_status_reg(&status); + + return 0; +fail: + dev_err(&mgr->dev, "Refresh failed.\n"); + + return ret; +} + +static const struct fpga_manager_ops machxo2_ops = { + .state = machxo2_spi_state, + .write_init = machxo2_write_init, + .write = machxo2_write, + .write_complete = machxo2_write_complete, +}; + +static int machxo2_spi_probe(struct spi_device *spi) +{ + struct device *dev = &spi->dev; + struct fpga_manager *mgr; + int ret; + + if (spi->max_speed_hz > MACHXO2_MAX_SPEED) { + dev_err(dev, "Speed is too high\n"); + return -EINVAL; + } + + mgr = fpga_mgr_create(dev, "Lattice MachXO2 SPI FPGA Manager", + &machxo2_ops, spi); + if (!mgr) + return -ENOMEM; + + spi_set_drvdata(spi, mgr); + + ret = fpga_mgr_register(mgr); + if (ret) + fpga_mgr_free(mgr); + + return ret; +} + +static int machxo2_spi_remove(struct spi_device *spi) +{ + struct fpga_manager *mgr = spi_get_drvdata(spi); + + fpga_mgr_unregister(mgr); + + return 0; +} + +static const struct of_device_id of_match[] = { + { .compatible = "lattice,machxo2-slave-spi", }, + {} +}; +MODULE_DEVICE_TABLE(of, of_match); + +static const struct spi_device_id lattice_ids[] = { + { "machxo2-slave-spi", 0 }, + { }, +}; +MODULE_DEVICE_TABLE(spi, lattice_ids); + +static struct spi_driver machxo2_spi_driver = { + .driver = { + .name = "machxo2-slave-spi", + .of_match_table = of_match_ptr(of_match), + }, + .probe = machxo2_spi_probe, + .remove = machxo2_spi_remove, + .id_table = lattice_ids, +}; + +module_spi_driver(machxo2_spi_driver) + +MODULE_AUTHOR("Paolo Pisati "); +MODULE_DESCRIPTION("Load Lattice FPGA firmware over SPI"); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/fpga/of-fpga-region.c b/drivers/fpga/of-fpga-region.c index 119ff75522f1..35fabb8083fb 100644 --- a/drivers/fpga/of-fpga-region.c +++ b/drivers/fpga/of-fpga-region.c @@ -1,22 +1,10 @@ +// SPDX-License-Identifier: GPL-2.0 /* * FPGA Region - Device Tree support for FPGA programming under Linux * * Copyright (C) 2013-2016 Altera Corporation * Copyright (C) 2017 Intel Corporation - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ - #include #include #include @@ -422,27 +410,25 @@ static int of_fpga_region_probe(struct platform_device *pdev) if (IS_ERR(mgr)) return -EPROBE_DEFER; - region = devm_kzalloc(dev, sizeof(*region), GFP_KERNEL); + region = fpga_region_create(dev, mgr, of_fpga_region_get_bridges); if (!region) { ret = -ENOMEM; goto eprobe_mgr_put; } - region->mgr = mgr; - - /* Specify how to get bridges for this type of region. */ - region->get_bridges = of_fpga_region_get_bridges; - - ret = fpga_region_register(dev, region); + ret = fpga_region_register(region); if (ret) - goto eprobe_mgr_put; + goto eprobe_free; of_platform_populate(np, fpga_region_of_match, NULL, ®ion->dev); + dev_set_drvdata(dev, region); dev_info(dev, "FPGA Region probed\n"); return 0; +eprobe_free: + fpga_region_free(region); eprobe_mgr_put: fpga_mgr_put(mgr); return ret; diff --git a/drivers/fpga/socfpga-a10.c b/drivers/fpga/socfpga-a10.c index a46e343a5b72..be30c48eb6e4 100644 --- a/drivers/fpga/socfpga-a10.c +++ b/drivers/fpga/socfpga-a10.c @@ -1,21 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 /* * FPGA Manager Driver for Altera Arria10 SoCFPGA * * Copyright (C) 2015-2016 Altera Corporation - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ - #include #include #include @@ -482,6 +470,7 @@ static int socfpga_a10_fpga_probe(struct platform_device *pdev) struct device *dev = &pdev->dev; struct a10_fpga_priv *priv; void __iomem *reg_base; + struct fpga_manager *mgr; struct resource *res; int ret; @@ -519,9 +508,16 @@ static int socfpga_a10_fpga_probe(struct platform_device *pdev) return -EBUSY; } - ret = fpga_mgr_register(dev, "SoCFPGA Arria10 FPGA Manager", - &socfpga_a10_fpga_mgr_ops, priv); + mgr = fpga_mgr_create(dev, "SoCFPGA Arria10 FPGA Manager", + &socfpga_a10_fpga_mgr_ops, priv); + if (!mgr) + return -ENOMEM; + + platform_set_drvdata(pdev, mgr); + + ret = fpga_mgr_register(mgr); if (ret) { + fpga_mgr_free(mgr); clk_disable_unprepare(priv->clk); return ret; } @@ -534,7 +530,7 @@ static int socfpga_a10_fpga_remove(struct platform_device *pdev) struct fpga_manager *mgr = platform_get_drvdata(pdev); struct a10_fpga_priv *priv = mgr->priv; - fpga_mgr_unregister(&pdev->dev); + fpga_mgr_unregister(mgr); clk_disable_unprepare(priv->clk); return 0; diff --git a/drivers/fpga/socfpga.c b/drivers/fpga/socfpga.c index b6672e66cda6..959d71f26896 100644 --- a/drivers/fpga/socfpga.c +++ b/drivers/fpga/socfpga.c @@ -1,19 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 /* * FPGA Manager Driver for Altera SOCFPGA * * Copyright (C) 2013-2015 Altera Corporation - * - * This program is free software; you can redistribute it and/or modify it - * under the terms and conditions of the GNU General Public License, - * version 2, as published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for - * more details. - * - * You should have received a copy of the GNU General Public License along with - * this program. If not, see . */ #include #include @@ -555,6 +544,7 @@ static int socfpga_fpga_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct socfpga_fpga_priv *priv; + struct fpga_manager *mgr; struct resource *res; int ret; @@ -581,13 +571,25 @@ static int socfpga_fpga_probe(struct platform_device *pdev) if (ret) return ret; - return fpga_mgr_register(dev, "Altera SOCFPGA FPGA Manager", - &socfpga_fpga_ops, priv); + mgr = fpga_mgr_create(dev, "Altera SOCFPGA FPGA Manager", + &socfpga_fpga_ops, priv); + if (!mgr) + return -ENOMEM; + + platform_set_drvdata(pdev, mgr); + + ret = fpga_mgr_register(mgr); + if (ret) + fpga_mgr_free(mgr); + + return ret; } static int socfpga_fpga_remove(struct platform_device *pdev) { - fpga_mgr_unregister(&pdev->dev); + struct fpga_manager *mgr = platform_get_drvdata(pdev); + + fpga_mgr_unregister(mgr); return 0; } diff --git a/drivers/fpga/ts73xx-fpga.c b/drivers/fpga/ts73xx-fpga.c index f6a96b42e2ca..08efd1895b1b 100644 --- a/drivers/fpga/ts73xx-fpga.c +++ b/drivers/fpga/ts73xx-fpga.c @@ -116,7 +116,9 @@ static int ts73xx_fpga_probe(struct platform_device *pdev) { struct device *kdev = &pdev->dev; struct ts73xx_fpga_priv *priv; + struct fpga_manager *mgr; struct resource *res; + int ret; priv = devm_kzalloc(kdev, sizeof(*priv), GFP_KERNEL); if (!priv) @@ -131,13 +133,25 @@ static int ts73xx_fpga_probe(struct platform_device *pdev) return PTR_ERR(priv->io_base); } - return fpga_mgr_register(kdev, "TS-73xx FPGA Manager", - &ts73xx_fpga_ops, priv); + mgr = fpga_mgr_create(kdev, "TS-73xx FPGA Manager", + &ts73xx_fpga_ops, priv); + if (!mgr) + return -ENOMEM; + + platform_set_drvdata(pdev, mgr); + + ret = fpga_mgr_register(mgr); + if (ret) + fpga_mgr_free(mgr); + + return ret; } static int ts73xx_fpga_remove(struct platform_device *pdev) { - fpga_mgr_unregister(&pdev->dev); + struct fpga_manager *mgr = platform_get_drvdata(pdev); + + fpga_mgr_unregister(mgr); return 0; } diff --git a/drivers/fpga/xilinx-pr-decoupler.c b/drivers/fpga/xilinx-pr-decoupler.c index 0d7743089414..07ba1539e82c 100644 --- a/drivers/fpga/xilinx-pr-decoupler.c +++ b/drivers/fpga/xilinx-pr-decoupler.c @@ -94,6 +94,7 @@ MODULE_DEVICE_TABLE(of, xlnx_pr_decoupler_of_match); static int xlnx_pr_decoupler_probe(struct platform_device *pdev) { struct xlnx_pr_decoupler_data *priv; + struct fpga_bridge *br; int err; struct resource *res; @@ -120,16 +121,27 @@ static int xlnx_pr_decoupler_probe(struct platform_device *pdev) clk_disable(priv->clk); - err = fpga_bridge_register(&pdev->dev, "Xilinx PR Decoupler", - &xlnx_pr_decoupler_br_ops, priv); + br = fpga_bridge_create(&pdev->dev, "Xilinx PR Decoupler", + &xlnx_pr_decoupler_br_ops, priv); + if (!br) { + err = -ENOMEM; + goto err_clk; + } + + platform_set_drvdata(pdev, br); + err = fpga_bridge_register(br); if (err) { dev_err(&pdev->dev, "unable to register Xilinx PR Decoupler"); - clk_unprepare(priv->clk); - return err; + goto err_clk; } return 0; + +err_clk: + clk_unprepare(priv->clk); + + return err; } static int xlnx_pr_decoupler_remove(struct platform_device *pdev) @@ -137,7 +149,7 @@ static int xlnx_pr_decoupler_remove(struct platform_device *pdev) struct fpga_bridge *bridge = platform_get_drvdata(pdev); struct xlnx_pr_decoupler_data *p = bridge->priv; - fpga_bridge_unregister(&pdev->dev); + fpga_bridge_unregister(bridge); clk_unprepare(p->clk); diff --git a/drivers/fpga/xilinx-spi.c b/drivers/fpga/xilinx-spi.c index 9b62a4c2a3df..8d1945966533 100644 --- a/drivers/fpga/xilinx-spi.c +++ b/drivers/fpga/xilinx-spi.c @@ -143,6 +143,8 @@ static const struct fpga_manager_ops xilinx_spi_ops = { static int xilinx_spi_probe(struct spi_device *spi) { struct xilinx_spi_conf *conf; + struct fpga_manager *mgr; + int ret; conf = devm_kzalloc(&spi->dev, sizeof(*conf), GFP_KERNEL); if (!conf) @@ -165,13 +167,25 @@ static int xilinx_spi_probe(struct spi_device *spi) return PTR_ERR(conf->done); } - return fpga_mgr_register(&spi->dev, "Xilinx Slave Serial FPGA Manager", - &xilinx_spi_ops, conf); + mgr = fpga_mgr_create(&spi->dev, "Xilinx Slave Serial FPGA Manager", + &xilinx_spi_ops, conf); + if (!mgr) + return -ENOMEM; + + spi_set_drvdata(spi, mgr); + + ret = fpga_mgr_register(mgr); + if (ret) + fpga_mgr_free(mgr); + + return ret; } static int xilinx_spi_remove(struct spi_device *spi) { - fpga_mgr_unregister(&spi->dev); + struct fpga_manager *mgr = spi_get_drvdata(spi); + + fpga_mgr_unregister(mgr); return 0; } diff --git a/drivers/fpga/zynq-fpga.c b/drivers/fpga/zynq-fpga.c index 70b15b303471..3110e00121ca 100644 --- a/drivers/fpga/zynq-fpga.c +++ b/drivers/fpga/zynq-fpga.c @@ -558,6 +558,7 @@ static int zynq_fpga_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct zynq_fpga_priv *priv; + struct fpga_manager *mgr; struct resource *res; int err; @@ -613,10 +614,17 @@ static int zynq_fpga_probe(struct platform_device *pdev) clk_disable(priv->clk); - err = fpga_mgr_register(dev, "Xilinx Zynq FPGA Manager", - &zynq_fpga_ops, priv); + mgr = fpga_mgr_create(dev, "Xilinx Zynq FPGA Manager", + &zynq_fpga_ops, priv); + if (!mgr) + return -ENOMEM; + + platform_set_drvdata(pdev, mgr); + + err = fpga_mgr_register(mgr); if (err) { dev_err(dev, "unable to register FPGA manager\n"); + fpga_mgr_free(mgr); clk_unprepare(priv->clk); return err; } @@ -632,7 +640,7 @@ static int zynq_fpga_remove(struct platform_device *pdev) mgr = platform_get_drvdata(pdev); priv = mgr->priv; - fpga_mgr_unregister(&pdev->dev); + fpga_mgr_unregister(mgr); clk_unprepare(priv->clk); diff --git a/drivers/gpio/Kconfig b/drivers/gpio/Kconfig index b960f6f35abd..71c0ab46f216 100644 --- a/drivers/gpio/Kconfig +++ b/drivers/gpio/Kconfig @@ -22,6 +22,18 @@ menuconfig GPIOLIB if GPIOLIB +config GPIOLIB_FASTPATH_LIMIT + int "Maximum number of GPIOs for fast path" + range 32 512 + default 512 + help + This adjusts the point at which certain APIs will switch from + using a stack allocated buffer to a dynamically allocated buffer. + + You shouldn't need to change this unless you really need to + optimize either stack space or performance. Change this carefully + since setting an incorrect value could cause stack corruption. + config OF_GPIO def_bool y depends on OF diff --git a/drivers/gpio/gpio-104-dio-48e.c b/drivers/gpio/gpio-104-dio-48e.c index 31e22c93e844..9c4e07fcb74b 100644 --- a/drivers/gpio/gpio-104-dio-48e.c +++ b/drivers/gpio/gpio-104-dio-48e.c @@ -188,7 +188,7 @@ static int dio48e_gpio_get_multiple(struct gpio_chip *chip, unsigned long *mask, { struct dio48e_gpio *const dio48egpio = gpiochip_get_data(chip); size_t i; - const size_t ports[] = { 0, 1, 2, 4, 5, 6 }; + static const size_t ports[] = { 0, 1, 2, 4, 5, 6 }; const unsigned int gpio_reg_size = 8; unsigned int bits_offset; size_t word_index; diff --git a/drivers/gpio/gpio-104-idi-48.c b/drivers/gpio/gpio-104-idi-48.c index f35632609379..2c9738adb3a6 100644 --- a/drivers/gpio/gpio-104-idi-48.c +++ b/drivers/gpio/gpio-104-idi-48.c @@ -94,7 +94,7 @@ static int idi_48_gpio_get_multiple(struct gpio_chip *chip, unsigned long *mask, { struct idi_48_gpio *const idi48gpio = gpiochip_get_data(chip); size_t i; - const size_t ports[] = { 0, 1, 2, 4, 5, 6 }; + static const size_t ports[] = { 0, 1, 2, 4, 5, 6 }; const unsigned int gpio_reg_size = 8; unsigned int bits_offset; size_t word_index; diff --git a/drivers/gpio/gpio-74xx-mmio.c b/drivers/gpio/gpio-74xx-mmio.c index 0475e8ec96d0..49616ec815ee 100644 --- a/drivers/gpio/gpio-74xx-mmio.c +++ b/drivers/gpio/gpio-74xx-mmio.c @@ -105,27 +105,22 @@ static int mmio_74xx_dir_out(struct gpio_chip *gc, unsigned int gpio, int val) static int mmio_74xx_gpio_probe(struct platform_device *pdev) { - const struct of_device_id *of_id; struct mmio_74xx_gpio_priv *priv; struct resource *res; void __iomem *dat; int err; - of_id = of_match_device(mmio_74xx_gpio_ids, &pdev->dev); - if (!of_id) - return -ENODEV; - priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL); if (!priv) return -ENOMEM; + priv->flags = (uintptr_t)of_device_get_match_data(&pdev->dev); + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); dat = devm_ioremap_resource(&pdev->dev, res); if (IS_ERR(dat)) return PTR_ERR(dat); - priv->flags = (uintptr_t) of_id->data; - err = bgpio_init(&priv->gc, &pdev->dev, DIV_ROUND_UP(MMIO_74XX_BIT_CNT(priv->flags), 8), dat, NULL, NULL, NULL, NULL, 0); diff --git a/drivers/gpio/gpio-adnp.c b/drivers/gpio/gpio-adnp.c index 44c09904daa6..91b90c0cea73 100644 --- a/drivers/gpio/gpio-adnp.c +++ b/drivers/gpio/gpio-adnp.c @@ -427,7 +427,7 @@ static int adnp_irq_setup(struct adnp *adnp) * is chosen to match the register layout of the hardware in that * each segment contains the corresponding bits for all interrupts. */ - adnp->irq_enable = devm_kzalloc(chip->parent, num_regs * 6, + adnp->irq_enable = devm_kcalloc(chip->parent, num_regs, 6, GFP_KERNEL); if (!adnp->irq_enable) return -ENOMEM; diff --git a/drivers/gpio/gpio-aspeed.c b/drivers/gpio/gpio-aspeed.c index 6f693b7d5220..b31ae16170e7 100644 --- a/drivers/gpio/gpio-aspeed.c +++ b/drivers/gpio/gpio-aspeed.c @@ -54,6 +54,8 @@ struct aspeed_gpio { u8 *offset_timer; unsigned int timer_users[4]; struct clk *clk; + + u32 *dcache; }; struct aspeed_gpio_bank { @@ -231,12 +233,13 @@ static void __aspeed_gpio_set(struct gpio_chip *gc, unsigned int offset, u32 reg; addr = bank_val_reg(gpio, bank, GPIO_DATA); - reg = ioread32(addr); + reg = gpio->dcache[GPIO_BANK(offset)]; if (val) reg |= GPIO_BIT(offset); else reg &= ~GPIO_BIT(offset); + gpio->dcache[GPIO_BANK(offset)] = reg; iowrite32(reg, addr); } @@ -287,11 +290,10 @@ static int aspeed_gpio_dir_out(struct gpio_chip *gc, spin_lock_irqsave(&gpio->lock, flags); + __aspeed_gpio_set(gc, offset, val); reg = ioread32(bank_val_reg(gpio, bank, GPIO_DIR)); iowrite32(reg | GPIO_BIT(offset), bank_val_reg(gpio, bank, GPIO_DIR)); - __aspeed_gpio_set(gc, offset, val); - spin_unlock_irqrestore(&gpio->lock, flags); return 0; @@ -852,7 +854,7 @@ static int __init aspeed_gpio_probe(struct platform_device *pdev) const struct of_device_id *gpio_id; struct aspeed_gpio *gpio; struct resource *res; - int rc; + int rc, i, banks; gpio = devm_kzalloc(&pdev->dev, sizeof(*gpio), GFP_KERNEL); if (!gpio) @@ -893,6 +895,20 @@ static int __init aspeed_gpio_probe(struct platform_device *pdev) gpio->chip.base = -1; gpio->chip.irq.need_valid_mask = true; + /* Allocate a cache of the output registers */ + banks = gpio->config->nr_gpios >> 5; + gpio->dcache = devm_kcalloc(&pdev->dev, + banks, sizeof(u32), GFP_KERNEL); + if (!gpio->dcache) + return -ENOMEM; + + /* Populate it with initial values read from the HW */ + for (i = 0; i < banks; i++) { + const struct aspeed_gpio_bank *bank = &aspeed_gpio_banks[i]; + gpio->dcache[i] = ioread32(gpio->base + bank->val_regs + + GPIO_DATA); + } + rc = devm_gpiochip_add_data(&pdev->dev, &gpio->chip, gpio); if (rc < 0) return rc; diff --git a/drivers/gpio/gpio-bcm-kona.c b/drivers/gpio/gpio-bcm-kona.c index eb8369b21e90..00272fa7cc4f 100644 --- a/drivers/gpio/gpio-bcm-kona.c +++ b/drivers/gpio/gpio-bcm-kona.c @@ -601,9 +601,10 @@ static int bcm_kona_gpio_probe(struct platform_device *pdev) GPIO_MAX_BANK_NUM); return -ENXIO; } - kona_gpio->banks = devm_kzalloc(dev, - kona_gpio->num_bank * - sizeof(*kona_gpio->banks), GFP_KERNEL); + kona_gpio->banks = devm_kcalloc(dev, + kona_gpio->num_bank, + sizeof(*kona_gpio->banks), + GFP_KERNEL); if (!kona_gpio->banks) return -ENOMEM; diff --git a/drivers/gpio/gpio-davinci.c b/drivers/gpio/gpio-davinci.c index 987126c4c6f6..035a454eca43 100644 --- a/drivers/gpio/gpio-davinci.c +++ b/drivers/gpio/gpio-davinci.c @@ -198,8 +198,8 @@ static int davinci_gpio_probe(struct platform_device *pdev) ngpio = ARCH_NR_GPIOS; nbank = DIV_ROUND_UP(ngpio, 32); - chips = devm_kzalloc(dev, - nbank * sizeof(struct davinci_gpio_controller), + chips = devm_kcalloc(dev, + nbank, sizeof(struct davinci_gpio_controller), GFP_KERNEL); if (!chips) return -ENOMEM; @@ -610,14 +610,12 @@ done: return 0; } -#if IS_ENABLED(CONFIG_OF) static const struct of_device_id davinci_gpio_ids[] = { { .compatible = "ti,keystone-gpio", keystone_gpio_get_irq_chip}, { .compatible = "ti,dm6441-gpio", davinci_gpio_get_irq_chip}, { /* sentinel */ }, }; MODULE_DEVICE_TABLE(of, davinci_gpio_ids); -#endif static struct platform_driver davinci_gpio_driver = { .probe = davinci_gpio_probe, diff --git a/drivers/gpio/gpio-dwapb.c b/drivers/gpio/gpio-dwapb.c index 226977f78482..7a2de3de6571 100644 --- a/drivers/gpio/gpio-dwapb.c +++ b/drivers/gpio/gpio-dwapb.c @@ -441,14 +441,19 @@ static void dwapb_configure_irqs(struct dwapb_gpio *gpio, irq_gc->chip_types[1].handler = handle_edge_irq; if (!pp->irq_shared) { - irq_set_chained_handler_and_data(pp->irq, dwapb_irq_handler, - gpio); + int i; + + for (i = 0; i < pp->ngpio; i++) { + if (pp->irq[i] >= 0) + irq_set_chained_handler_and_data(pp->irq[i], + dwapb_irq_handler, gpio); + } } else { /* * Request a shared IRQ since where MFD would have devices * using the same irq pin */ - err = devm_request_irq(gpio->dev, pp->irq, + err = devm_request_irq(gpio->dev, pp->irq[0], dwapb_irq_handler_mfd, IRQF_SHARED, "gpio-dwapb-mfd", gpio); if (err) { @@ -524,7 +529,7 @@ static int dwapb_gpio_add_port(struct dwapb_gpio *gpio, if (pp->idx == 0) port->gc.set_config = dwapb_gpio_set_config; - if (pp->irq) + if (pp->has_irq) dwapb_configure_irqs(gpio, port, pp); err = gpiochip_add_data(&port->gc, port); @@ -535,7 +540,7 @@ static int dwapb_gpio_add_port(struct dwapb_gpio *gpio, port->is_registered = true; /* Add GPIO-signaled ACPI event support */ - if (pp->irq) + if (pp->has_irq) acpi_gpiochip_request_interrupts(&port->gc); return err; @@ -557,7 +562,7 @@ dwapb_gpio_get_pdata(struct device *dev) struct dwapb_platform_data *pdata; struct dwapb_port_property *pp; int nports; - int i; + int i, j; nports = device_get_child_node_count(dev); if (nports == 0) @@ -575,6 +580,8 @@ dwapb_gpio_get_pdata(struct device *dev) i = 0; device_for_each_child_node(dev, fwnode) { + struct device_node *np = NULL; + pp = &pdata->properties[i++]; pp->fwnode = fwnode; @@ -594,23 +601,35 @@ dwapb_gpio_get_pdata(struct device *dev) pp->ngpio = 32; } + pp->irq_shared = false; + pp->gpio_base = -1; + /* * Only port A can provide interrupts in all configurations of * the IP. */ - if (dev->of_node && pp->idx == 0 && - fwnode_property_read_bool(fwnode, + if (pp->idx != 0) + continue; + + if (dev->of_node && fwnode_property_read_bool(fwnode, "interrupt-controller")) { - pp->irq = irq_of_parse_and_map(to_of_node(fwnode), 0); - if (!pp->irq) - dev_warn(dev, "no irq for port%d\n", pp->idx); + np = to_of_node(fwnode); } - if (has_acpi_companion(dev) && pp->idx == 0) - pp->irq = platform_get_irq(to_platform_device(dev), 0); + for (j = 0; j < pp->ngpio; j++) { + pp->irq[j] = -ENXIO; - pp->irq_shared = false; - pp->gpio_base = -1; + if (np) + pp->irq[j] = of_irq_get(np, j); + else if (has_acpi_companion(dev)) + pp->irq[j] = platform_get_irq(to_platform_device(dev), j); + + if (pp->irq[j] >= 0) + pp->has_irq = true; + } + + if (!pp->has_irq) + dev_warn(dev, "no irq for port%d\n", pp->idx); } return pdata; @@ -684,13 +703,7 @@ static int dwapb_gpio_probe(struct platform_device *pdev) gpio->flags = 0; if (dev->of_node) { - const struct of_device_id *of_devid; - - of_devid = of_match_device(dwapb_of_match, dev); - if (of_devid) { - if (of_devid->data) - gpio->flags = (uintptr_t)of_devid->data; - } + gpio->flags = (uintptr_t)of_device_get_match_data(dev); } else if (has_acpi_companion(dev)) { const struct acpi_device_id *acpi_id; diff --git a/drivers/gpio/gpio-eic-sprd.c b/drivers/gpio/gpio-eic-sprd.c index de7dd939c043..e0d6a0a7bc69 100644 --- a/drivers/gpio/gpio-eic-sprd.c +++ b/drivers/gpio/gpio-eic-sprd.c @@ -300,6 +300,7 @@ static int sprd_eic_irq_set_type(struct irq_data *data, unsigned int flow_type) struct gpio_chip *chip = irq_data_get_irq_chip_data(data); struct sprd_eic *sprd_eic = gpiochip_get_data(chip); u32 offset = irqd_to_hwirq(data); + int state; switch (sprd_eic->type) { case SPRD_EIC_DEBOUNCE: @@ -310,6 +311,17 @@ static int sprd_eic_irq_set_type(struct irq_data *data, unsigned int flow_type) case IRQ_TYPE_LEVEL_LOW: sprd_eic_update(chip, offset, SPRD_EIC_DBNC_IEV, 0); break; + case IRQ_TYPE_EDGE_RISING: + case IRQ_TYPE_EDGE_FALLING: + case IRQ_TYPE_EDGE_BOTH: + state = sprd_eic_get(chip, offset); + if (state) + sprd_eic_update(chip, offset, + SPRD_EIC_DBNC_IEV, 0); + else + sprd_eic_update(chip, offset, + SPRD_EIC_DBNC_IEV, 1); + break; default: return -ENOTSUPP; } @@ -324,6 +336,17 @@ static int sprd_eic_irq_set_type(struct irq_data *data, unsigned int flow_type) case IRQ_TYPE_LEVEL_LOW: sprd_eic_update(chip, offset, SPRD_EIC_LATCH_INTPOL, 1); break; + case IRQ_TYPE_EDGE_RISING: + case IRQ_TYPE_EDGE_FALLING: + case IRQ_TYPE_EDGE_BOTH: + state = sprd_eic_get(chip, offset); + if (state) + sprd_eic_update(chip, offset, + SPRD_EIC_LATCH_INTPOL, 0); + else + sprd_eic_update(chip, offset, + SPRD_EIC_LATCH_INTPOL, 1); + break; default: return -ENOTSUPP; } @@ -405,6 +428,55 @@ static int sprd_eic_irq_set_type(struct irq_data *data, unsigned int flow_type) return 0; } +static void sprd_eic_toggle_trigger(struct gpio_chip *chip, unsigned int irq, + unsigned int offset) +{ + struct sprd_eic *sprd_eic = gpiochip_get_data(chip); + struct irq_data *data = irq_get_irq_data(irq); + u32 trigger = irqd_get_trigger_type(data); + int state, post_state; + + /* + * The debounce EIC and latch EIC can only support level trigger, so we + * can toggle the level trigger to emulate the edge trigger. + */ + if ((sprd_eic->type != SPRD_EIC_DEBOUNCE && + sprd_eic->type != SPRD_EIC_LATCH) || + !(trigger & IRQ_TYPE_EDGE_BOTH)) + return; + + sprd_eic_irq_mask(data); + state = sprd_eic_get(chip, offset); + +retry: + switch (sprd_eic->type) { + case SPRD_EIC_DEBOUNCE: + if (state) + sprd_eic_update(chip, offset, SPRD_EIC_DBNC_IEV, 0); + else + sprd_eic_update(chip, offset, SPRD_EIC_DBNC_IEV, 1); + break; + case SPRD_EIC_LATCH: + if (state) + sprd_eic_update(chip, offset, SPRD_EIC_LATCH_INTPOL, 0); + else + sprd_eic_update(chip, offset, SPRD_EIC_LATCH_INTPOL, 1); + break; + default: + sprd_eic_irq_unmask(data); + return; + } + + post_state = sprd_eic_get(chip, offset); + if (state != post_state) { + dev_warn(chip->parent, "EIC level was changed.\n"); + state = post_state; + goto retry; + } + + sprd_eic_irq_unmask(data); +} + static int sprd_eic_match_chip_by_type(struct gpio_chip *chip, void *data) { enum sprd_eic_type type = *(enum sprd_eic_type *)data; @@ -448,6 +520,7 @@ static void sprd_eic_handle_one_type(struct gpio_chip *chip) bank * SPRD_EIC_PER_BANK_NR + n); generic_handle_irq(girq); + sprd_eic_toggle_trigger(chip, girq, n); } } } diff --git a/drivers/gpio/gpio-ge.c b/drivers/gpio/gpio-ge.c index 1fe2d3418f2f..636952769bc8 100644 --- a/drivers/gpio/gpio-ge.c +++ b/drivers/gpio/gpio-ge.c @@ -52,8 +52,6 @@ MODULE_DEVICE_TABLE(of, gef_gpio_ids); static int __init gef_gpio_probe(struct platform_device *pdev) { - const struct of_device_id *of_id = - of_match_device(gef_gpio_ids, &pdev->dev); struct gpio_chip *gc; void __iomem *regs; int ret; @@ -82,7 +80,7 @@ static int __init gef_gpio_probe(struct platform_device *pdev) } gc->base = -1; - gc->ngpio = (u16)(uintptr_t)of_id->data; + gc->ngpio = (u16)(uintptr_t)of_device_get_match_data(&pdev->dev); gc->of_gpio_n_cells = 2; gc->of_node = pdev->dev.of_node; diff --git a/drivers/gpio/gpio-gpio-mm.c b/drivers/gpio/gpio-gpio-mm.c index d496cc56c2a2..b56ff2efbf36 100644 --- a/drivers/gpio/gpio-gpio-mm.c +++ b/drivers/gpio/gpio-gpio-mm.c @@ -177,7 +177,7 @@ static int gpiomm_gpio_get_multiple(struct gpio_chip *chip, unsigned long *mask, { struct gpiomm_gpio *const gpiommgpio = gpiochip_get_data(chip); size_t i; - const size_t ports[] = { 0, 1, 2, 4, 5, 6 }; + static const size_t ports[] = { 0, 1, 2, 4, 5, 6 }; const unsigned int gpio_reg_size = 8; unsigned int bits_offset; size_t word_index; diff --git a/drivers/gpio/gpio-htc-egpio.c b/drivers/gpio/gpio-htc-egpio.c index 516383934945..ad6e5b518669 100644 --- a/drivers/gpio/gpio-htc-egpio.c +++ b/drivers/gpio/gpio-htc-egpio.c @@ -321,8 +321,8 @@ static int __init egpio_probe(struct platform_device *pdev) platform_set_drvdata(pdev, ei); ei->nchips = pdata->num_chips; - ei->chip = devm_kzalloc(&pdev->dev, - sizeof(struct egpio_chip) * ei->nchips, + ei->chip = devm_kcalloc(&pdev->dev, + ei->nchips, sizeof(struct egpio_chip), GFP_KERNEL); if (!ei->chip) { ret = -ENOMEM; diff --git a/drivers/gpio/gpio-ingenic.c b/drivers/gpio/gpio-ingenic.c index 15fb2bc796a8..e738e384a5ca 100644 --- a/drivers/gpio/gpio-ingenic.c +++ b/drivers/gpio/gpio-ingenic.c @@ -285,8 +285,6 @@ MODULE_DEVICE_TABLE(of, ingenic_gpio_of_match); static int ingenic_gpio_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; - const struct of_device_id *of_id = of_match_device( - ingenic_gpio_of_match, dev); struct ingenic_gpio_chip *jzgc; u32 bank; int err; @@ -323,7 +321,7 @@ static int ingenic_gpio_probe(struct platform_device *pdev) jzgc->gc.parent = dev; jzgc->gc.of_node = dev->of_node; jzgc->gc.owner = THIS_MODULE; - jzgc->version = (enum jz_version)of_id->data; + jzgc->version = (enum jz_version)of_device_get_match_data(dev); jzgc->gc.set = ingenic_gpio_set; jzgc->gc.get = ingenic_gpio_get; diff --git a/drivers/gpio/gpio-loongson.c b/drivers/gpio/gpio-loongson.c index 92c4fe7b2677..16cfbe9e72fe 100644 --- a/drivers/gpio/gpio-loongson.c +++ b/drivers/gpio/gpio-loongson.c @@ -17,9 +17,11 @@ #include #include #include +#include +#include +#include #include #include -#include #define STLS2F_N_GPIO 4 #define STLS3A_N_GPIO 16 @@ -30,86 +32,108 @@ #define LOONGSON_N_GPIO STLS2F_N_GPIO #endif +/* + * Offset into the register where we read lines, we write them from offset 0. + * This offset is the only thing that stand between us and using + * GPIO_GENERIC. + */ #define LOONGSON_GPIO_IN_OFFSET 16 static DEFINE_SPINLOCK(gpio_lock); -static int loongson_gpio_direction_input(struct gpio_chip *chip, unsigned gpio) +static int loongson_gpio_get_value(struct gpio_chip *chip, unsigned gpio) { - u32 temp; - u32 mask; + u32 val; spin_lock(&gpio_lock); - mask = 1 << gpio; - temp = LOONGSON_GPIOIE; - temp |= mask; - LOONGSON_GPIOIE = temp; + val = LOONGSON_GPIODATA; spin_unlock(&gpio_lock); - return 0; + return !!(val & BIT(gpio + LOONGSON_GPIO_IN_OFFSET)); } -static int loongson_gpio_direction_output(struct gpio_chip *chip, - unsigned gpio, int level) +static void loongson_gpio_set_value(struct gpio_chip *chip, + unsigned gpio, int value) +{ + u32 val; + + spin_lock(&gpio_lock); + val = LOONGSON_GPIODATA; + if (value) + val |= BIT(gpio); + else + val &= ~BIT(gpio); + LOONGSON_GPIODATA = val; + spin_unlock(&gpio_lock); +} + +static int loongson_gpio_direction_input(struct gpio_chip *chip, unsigned gpio) { u32 temp; - u32 mask; - gpio_set_value(gpio, level); spin_lock(&gpio_lock); - mask = 1 << gpio; temp = LOONGSON_GPIOIE; - temp &= (~mask); + temp |= BIT(gpio); LOONGSON_GPIOIE = temp; spin_unlock(&gpio_lock); return 0; } -static int loongson_gpio_get_value(struct gpio_chip *chip, unsigned gpio) +static int loongson_gpio_direction_output(struct gpio_chip *chip, + unsigned gpio, int level) { - u32 val; - u32 mask; + u32 temp; - mask = 1 << (gpio + LOONGSON_GPIO_IN_OFFSET); + loongson_gpio_set_value(chip, gpio, level); spin_lock(&gpio_lock); - val = LOONGSON_GPIODATA; + temp = LOONGSON_GPIOIE; + temp &= ~BIT(gpio); + LOONGSON_GPIOIE = temp; spin_unlock(&gpio_lock); - return (val & mask) != 0; + return 0; } -static void loongson_gpio_set_value(struct gpio_chip *chip, - unsigned gpio, int value) +static int loongson_gpio_probe(struct platform_device *pdev) { - u32 val; - u32 mask; - - mask = 1 << gpio; - - spin_lock(&gpio_lock); - val = LOONGSON_GPIODATA; - if (value) - val |= mask; - else - val &= (~mask); - LOONGSON_GPIODATA = val; - spin_unlock(&gpio_lock); + struct gpio_chip *gc; + struct device *dev = &pdev->dev; + + gc = devm_kzalloc(dev, sizeof(*gc), GFP_KERNEL); + if (!gc) + return -ENOMEM; + + gc->label = "loongson-gpio-chip"; + gc->base = 0; + gc->ngpio = LOONGSON_N_GPIO; + gc->get = loongson_gpio_get_value; + gc->set = loongson_gpio_set_value; + gc->direction_input = loongson_gpio_direction_input; + gc->direction_output = loongson_gpio_direction_output; + + return gpiochip_add_data(gc, NULL); } -static struct gpio_chip loongson_chip = { - .label = "Loongson-gpio-chip", - .direction_input = loongson_gpio_direction_input, - .get = loongson_gpio_get_value, - .direction_output = loongson_gpio_direction_output, - .set = loongson_gpio_set_value, - .base = 0, - .ngpio = LOONGSON_N_GPIO, - .can_sleep = false, +static struct platform_driver loongson_gpio_driver = { + .driver = { + .name = "loongson-gpio", + }, + .probe = loongson_gpio_probe, }; static int __init loongson_gpio_setup(void) { - return gpiochip_add_data(&loongson_chip, NULL); + struct platform_device *pdev; + int ret; + + ret = platform_driver_register(&loongson_gpio_driver); + if (ret) { + pr_err("error registering loongson GPIO driver\n"); + return ret; + } + + pdev = platform_device_register_simple("loongson-gpio", -1, NULL, 0); + return PTR_ERR_OR_ZERO(pdev); } postcore_initcall(loongson_gpio_setup); diff --git a/drivers/gpio/gpio-lp3943.c b/drivers/gpio/gpio-lp3943.c index 6dc6725403ec..c3a3b9b7b553 100644 --- a/drivers/gpio/gpio-lp3943.c +++ b/drivers/gpio/gpio-lp3943.c @@ -12,7 +12,7 @@ #include #include -#include +#include #include #include #include diff --git a/drivers/gpio/gpio-lp873x.c b/drivers/gpio/gpio-lp873x.c index df0ad2cef0d2..801995dd9b26 100644 --- a/drivers/gpio/gpio-lp873x.c +++ b/drivers/gpio/gpio-lp873x.c @@ -14,7 +14,7 @@ * Based on the TPS65218 driver */ -#include +#include #include #include #include diff --git a/drivers/gpio/gpio-lpc32xx.c b/drivers/gpio/gpio-lpc32xx.c index 92b3ae2a6735..aa74cc4d8b14 100644 --- a/drivers/gpio/gpio-lpc32xx.c +++ b/drivers/gpio/gpio-lpc32xx.c @@ -20,9 +20,8 @@ #include #include #include -#include +#include #include -#include #include #include diff --git a/drivers/gpio/gpio-lynxpoint.c b/drivers/gpio/gpio-lynxpoint.c index 1e557b10d73e..b5b5e500e72c 100644 --- a/drivers/gpio/gpio-lynxpoint.c +++ b/drivers/gpio/gpio-lynxpoint.c @@ -25,7 +25,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/gpio/gpio-max730x.c b/drivers/gpio/gpio-max730x.c index 946d09195598..198a36b07773 100644 --- a/drivers/gpio/gpio-max730x.c +++ b/drivers/gpio/gpio-max730x.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include /* diff --git a/drivers/gpio/gpio-mc33880.c b/drivers/gpio/gpio-mc33880.c index 0f0df7956264..18a5a58d634a 100644 --- a/drivers/gpio/gpio-mc33880.c +++ b/drivers/gpio/gpio-mc33880.c @@ -24,7 +24,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/gpio/gpio-mc9s08dz60.c b/drivers/gpio/gpio-mc9s08dz60.c index 2fcad5b9cca5..d8d846d2189a 100644 --- a/drivers/gpio/gpio-mc9s08dz60.c +++ b/drivers/gpio/gpio-mc9s08dz60.c @@ -18,7 +18,7 @@ #include #include #include -#include +#include #define GPIO_GROUP_NUM 2 #define GPIO_NUM_PER_GROUP 8 diff --git a/drivers/gpio/gpio-ml-ioh.c b/drivers/gpio/gpio-ml-ioh.c index b3678bd1c120..b23d9a36be1f 100644 --- a/drivers/gpio/gpio-ml-ioh.c +++ b/drivers/gpio/gpio-ml-ioh.c @@ -18,7 +18,7 @@ #include #include #include -#include +#include #include #include @@ -443,7 +443,7 @@ static int ioh_gpio_probe(struct pci_dev *pdev, goto err_iomap; } - chip_save = kzalloc(sizeof(*chip) * 8, GFP_KERNEL); + chip_save = kcalloc(8, sizeof(*chip), GFP_KERNEL); if (chip_save == NULL) { ret = -ENOMEM; goto err_kzalloc; diff --git a/drivers/gpio/gpio-mm-lantiq.c b/drivers/gpio/gpio-mm-lantiq.c index b1cf76dd84ba..b0754fe69e77 100644 --- a/drivers/gpio/gpio-mm-lantiq.c +++ b/drivers/gpio/gpio-mm-lantiq.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/gpio/gpio-mockup.c b/drivers/gpio/gpio-mockup.c index 76c2fe91a901..d66b7a768ecd 100644 --- a/drivers/gpio/gpio-mockup.c +++ b/drivers/gpio/gpio-mockup.c @@ -1,15 +1,10 @@ +// SPDX-License-Identifier: GPL-2.0+ /* * GPIO Testing Device Driver * * Copyright (C) 2014 Kamlakant Patel * Copyright (C) 2015-2016 Bamvor Jian Zhang * Copyright (C) 2017 Bartosz Golaszewski - * - * This program is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License as published by the - * Free Software Foundation; either version 2 of the License, or (at your - * option) any later version. - * */ #include diff --git a/drivers/gpio/gpio-msic.c b/drivers/gpio/gpio-msic.c index 6cb67595d15f..3b34dbecef99 100644 --- a/drivers/gpio/gpio-msic.c +++ b/drivers/gpio/gpio-msic.c @@ -24,7 +24,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/gpio/gpio-mvebu.c b/drivers/gpio/gpio-mvebu.c index 45c65f805fd6..6e02148c208b 100644 --- a/drivers/gpio/gpio-mvebu.c +++ b/drivers/gpio/gpio-mvebu.c @@ -36,7 +36,8 @@ #include #include #include -#include +#include +#include #include #include #include @@ -51,8 +52,6 @@ #include #include -#include "gpiolib.h" - /* * GPIO unit register offsets. */ @@ -608,19 +607,16 @@ static int mvebu_pwm_request(struct pwm_chip *chip, struct pwm_device *pwm) if (mvpwm->gpiod) { ret = -EBUSY; } else { - desc = gpio_to_desc(mvchip->chip.base + pwm->hwpwm); - if (!desc) { - ret = -ENODEV; + desc = gpiochip_request_own_desc(&mvchip->chip, + pwm->hwpwm, "mvebu-pwm"); + if (IS_ERR(desc)) { + ret = PTR_ERR(desc); goto out; } - ret = gpiod_request(desc, "mvebu-pwm"); - if (ret) - goto out; - ret = gpiod_direction_output(desc, 0); if (ret) { - gpiod_free(desc); + gpiochip_free_own_desc(desc); goto out; } @@ -637,7 +633,7 @@ static void mvebu_pwm_free(struct pwm_chip *chip, struct pwm_device *pwm) unsigned long flags; spin_lock_irqsave(&mvpwm->lock, flags); - gpiod_free(mvpwm->gpiod); + gpiochip_free_own_desc(mvpwm->gpiod); mvpwm->gpiod = NULL; spin_unlock_irqrestore(&mvpwm->lock, flags); } diff --git a/drivers/gpio/gpio-mxc.c b/drivers/gpio/gpio-mxc.c index 5245a2fe62ae..2f2829966d4c 100644 --- a/drivers/gpio/gpio-mxc.c +++ b/drivers/gpio/gpio-mxc.c @@ -1,25 +1,13 @@ -/* - * MXC GPIO support. (c) 2008 Daniel Mack - * Copyright 2008 Juergen Beisert, kernel@pengutronix.de - * - * Based on code from Freescale Semiconductor, - * Authors: Daniel Mack, Juergen Beisert. - * Copyright (C) 2004-2010 Freescale Semiconductor, Inc. All Rights Reserved. - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version 2 - * of the License, or (at your option) any later version. - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - */ - +// SPDX-License-Identifier: GPL-2.0+ +// +// MXC GPIO support. (c) 2008 Daniel Mack +// Copyright 2008 Juergen Beisert, kernel@pengutronix.de +// +// Based on code from Freescale Semiconductor, +// Authors: Daniel Mack, Juergen Beisert. +// Copyright (C) 2004-2010 Freescale Semiconductor, Inc. All Rights Reserved. + +#include #include #include #include @@ -30,8 +18,6 @@ #include #include #include -/* FIXME: for gpio_get_value() replace this with direct register read */ -#include #include #include #include @@ -62,6 +48,7 @@ struct mxc_gpio_hwdata { struct mxc_gpio_port { struct list_head node; void __iomem *base; + struct clk *clk; int irq; int irq_high; struct irq_domain *domain; @@ -174,7 +161,6 @@ static int gpio_set_irq_type(struct irq_data *d, u32 type) struct mxc_gpio_port *port = gc->private; u32 bit, val; u32 gpio_idx = d->hwirq; - u32 gpio = port->gc.base + gpio_idx; int edge; void __iomem *reg = port->base; @@ -190,13 +176,13 @@ static int gpio_set_irq_type(struct irq_data *d, u32 type) if (GPIO_EDGE_SEL >= 0) { edge = GPIO_INT_BOTH_EDGES; } else { - val = gpio_get_value(gpio); + val = port->gc.get(&port->gc, gpio_idx); if (val) { edge = GPIO_INT_LOW_LEV; - pr_debug("mxc: set GPIO %d to low trigger\n", gpio); + pr_debug("mxc: set GPIO %d to low trigger\n", gpio_idx); } else { edge = GPIO_INT_HIGH_LEV; - pr_debug("mxc: set GPIO %d to high trigger\n", gpio); + pr_debug("mxc: set GPIO %d to high trigger\n", gpio_idx); } port->both_edges |= 1 << gpio_idx; } @@ -437,6 +423,17 @@ static int mxc_gpio_probe(struct platform_device *pdev) if (port->irq < 0) return port->irq; + /* the controller clock is optional */ + port->clk = devm_clk_get(&pdev->dev, NULL); + if (IS_ERR(port->clk)) + port->clk = NULL; + + err = clk_prepare_enable(port->clk); + if (err) { + dev_err(&pdev->dev, "Unable to enable clock.\n"); + return err; + } + /* disable the interrupt and clear the status */ writel(0, port->base + GPIO_IMR); writel(~0, port->base + GPIO_ISR); @@ -505,6 +502,7 @@ static int mxc_gpio_probe(struct platform_device *pdev) out_irqdomain_remove: irq_domain_remove(port->domain); out_bgio: + clk_disable_unprepare(port->clk); dev_info(&pdev->dev, "%s failed with errno %d\n", __func__, err); return err; } diff --git a/drivers/gpio/gpio-mxs.c b/drivers/gpio/gpio-mxs.c index 435def22445d..e2831ee70cdc 100644 --- a/drivers/gpio/gpio-mxs.c +++ b/drivers/gpio/gpio-mxs.c @@ -1,24 +1,10 @@ -/* - * MXC GPIO support. (c) 2008 Daniel Mack - * Copyright 2008 Juergen Beisert, kernel@pengutronix.de - * - * Based on code from Freescale, - * Copyright (C) 2004-2010 Freescale Semiconductor, Inc. All Rights Reserved. - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version 2 - * of the License, or (at your option) any later version. - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, - * MA 02110-1301, USA. - */ +// SPDX-License-Identifier: GPL-2.0+ +// +// MXC GPIO support. (c) 2008 Daniel Mack +// Copyright 2008 Juergen Beisert, kernel@pengutronix.de +// +// Based on code from Freescale, +// Copyright (C) 2004-2010 Freescale Semiconductor, Inc. All Rights Reserved. #include #include @@ -290,8 +276,6 @@ MODULE_DEVICE_TABLE(of, mxs_gpio_dt_ids); static int mxs_gpio_probe(struct platform_device *pdev) { - const struct of_device_id *of_id = - of_match_device(mxs_gpio_dt_ids, &pdev->dev); struct device_node *np = pdev->dev.of_node; struct device_node *parent; static void __iomem *base; @@ -306,7 +290,7 @@ static int mxs_gpio_probe(struct platform_device *pdev) port->id = of_alias_get_id(np, "gpio"); if (port->id < 0) return port->id; - port->devid = (enum mxs_gpio_id) of_id->data; + port->devid = (enum mxs_gpio_id)of_device_get_match_data(&pdev->dev); port->dev = &pdev->dev; port->irq = platform_get_irq(pdev, 0); if (port->irq < 0) diff --git a/drivers/gpio/gpio-octeon.c b/drivers/gpio/gpio-octeon.c index 96a8a8cb2729..1b19c88ea7bb 100644 --- a/drivers/gpio/gpio-octeon.c +++ b/drivers/gpio/gpio-octeon.c @@ -9,7 +9,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/gpio/gpio-omap.c b/drivers/gpio/gpio-omap.c index 35971a341c40..d1afedf4dcbf 100644 --- a/drivers/gpio/gpio-omap.c +++ b/drivers/gpio/gpio-omap.c @@ -24,7 +24,7 @@ #include #include #include -#include +#include #include #include diff --git a/drivers/gpio/gpio-palmas.c b/drivers/gpio/gpio-palmas.c index 3d818195e351..05b0cd5dcf11 100644 --- a/drivers/gpio/gpio-palmas.c +++ b/drivers/gpio/gpio-palmas.c @@ -18,7 +18,7 @@ * along with this program. If not, see . */ -#include +#include #include #include #include @@ -159,13 +159,9 @@ static int palmas_gpio_probe(struct platform_device *pdev) struct palmas_platform_data *palmas_pdata; struct palmas_gpio *palmas_gpio; int ret; - const struct of_device_id *match; const struct palmas_device_data *dev_data; - match = of_match_device(of_palmas_gpio_match, &pdev->dev); - if (!match) - return -ENODEV; - dev_data = match->data; + dev_data = of_device_get_match_data(&pdev->dev); if (!dev_data) dev_data = &palmas_dev_data; diff --git a/drivers/gpio/gpio-pca953x.c b/drivers/gpio/gpio-pca953x.c index d2ead4b1cf61..c55ad157e820 100644 --- a/drivers/gpio/gpio-pca953x.c +++ b/drivers/gpio/gpio-pca953x.c @@ -12,7 +12,7 @@ */ #include -#include +#include #include #include #include @@ -25,29 +25,44 @@ #include -#define PCA953X_INPUT 0 -#define PCA953X_OUTPUT 1 -#define PCA953X_INVERT 2 -#define PCA953X_DIRECTION 3 +#define PCA953X_INPUT 0x00 +#define PCA953X_OUTPUT 0x01 +#define PCA953X_INVERT 0x02 +#define PCA953X_DIRECTION 0x03 #define REG_ADDR_AI 0x80 -#define PCA957X_IN 0 -#define PCA957X_INVRT 1 -#define PCA957X_BKEN 2 -#define PCA957X_PUPD 3 -#define PCA957X_CFG 4 -#define PCA957X_OUT 5 -#define PCA957X_MSK 6 -#define PCA957X_INTS 7 - -#define PCAL953X_IN_LATCH 34 -#define PCAL953X_INT_MASK 37 -#define PCAL953X_INT_STAT 38 +#define PCA957X_IN 0x00 +#define PCA957X_INVRT 0x01 +#define PCA957X_BKEN 0x02 +#define PCA957X_PUPD 0x03 +#define PCA957X_CFG 0x04 +#define PCA957X_OUT 0x05 +#define PCA957X_MSK 0x06 +#define PCA957X_INTS 0x07 + +#define PCAL953X_OUT_STRENGTH 0x20 +#define PCAL953X_IN_LATCH 0x22 +#define PCAL953X_PULL_EN 0x23 +#define PCAL953X_PULL_SEL 0x24 +#define PCAL953X_INT_MASK 0x25 +#define PCAL953X_INT_STAT 0x26 +#define PCAL953X_OUT_CONF 0x27 + +#define PCAL6524_INT_EDGE 0x28 +#define PCAL6524_INT_CLR 0x2a +#define PCAL6524_IN_STATUS 0x2b +#define PCAL6524_OUT_INDCONF 0x2c +#define PCAL6524_DEBOUNCE 0x2d #define PCA_GPIO_MASK 0x00FF + +#define PCAL_GPIO_MASK 0x1f +#define PCAL_PINCTRL_MASK 0xe0 + #define PCA_INT 0x0100 #define PCA_PCAL 0x0200 +#define PCA_LATCH_INT (PCA_PCAL | PCA_INT) #define PCA953X_TYPE 0x1000 #define PCA957X_TYPE 0x2000 #define PCA_TYPE_MASK 0xF000 @@ -207,9 +222,11 @@ static int pca957x_write_regs_16(struct pca953x_chip *chip, int reg, u8 *val) static int pca953x_write_regs_24(struct pca953x_chip *chip, int reg, u8 *val) { int bank_shift = fls((chip->gpio_chip.ngpio - 1) / BANK_SZ); + int addr = (reg & PCAL_GPIO_MASK) << bank_shift; + int pinctrl = (reg & PCAL_PINCTRL_MASK) << 1; return i2c_smbus_write_i2c_block_data(chip->client, - (reg << bank_shift) | REG_ADDR_AI, + pinctrl | addr | REG_ADDR_AI, NBANK(chip), val); } @@ -249,9 +266,11 @@ static int pca953x_read_regs_16(struct pca953x_chip *chip, int reg, u8 *val) static int pca953x_read_regs_24(struct pca953x_chip *chip, int reg, u8 *val) { int bank_shift = fls((chip->gpio_chip.ngpio - 1) / BANK_SZ); + int addr = (reg & PCAL_GPIO_MASK) << bank_shift; + int pinctrl = (reg & PCAL_PINCTRL_MASK) << 1; return i2c_smbus_read_i2c_block_data(chip->client, - (reg << bank_shift) | REG_ADDR_AI, + pinctrl | addr | REG_ADDR_AI, NBANK(chip), val); } @@ -522,6 +541,15 @@ static int pca953x_irq_set_type(struct irq_data *d, unsigned int type) return 0; } +static void pca953x_irq_shutdown(struct irq_data *d) +{ + struct pca953x_chip *chip = irq_data_get_irq_chip_data(d); + u8 mask = 1 << (d->hwirq % BANK_SZ); + + chip->irq_trig_raise[d->hwirq / BANK_SZ] &= ~mask; + chip->irq_trig_fall[d->hwirq / BANK_SZ] &= ~mask; +} + static struct irq_chip pca953x_irq_chip = { .name = "pca953x", .irq_mask = pca953x_irq_mask, @@ -529,6 +557,7 @@ static struct irq_chip pca953x_irq_chip = { .irq_bus_lock = pca953x_irq_bus_lock, .irq_bus_sync_unlock = pca953x_irq_bus_sync_unlock, .irq_set_type = pca953x_irq_set_type, + .irq_shutdown = pca953x_irq_shutdown, }; static bool pca953x_irq_pending(struct pca953x_chip *chip, u8 *pending) @@ -810,13 +839,11 @@ static int pca953x_probe(struct i2c_client *client, chip->driver_data = i2c_id->driver_data; } else { const struct acpi_device_id *acpi_id; - const struct of_device_id *match; + struct device *dev = &client->dev; - match = of_match_device(pca953x_dt_ids, &client->dev); - if (match) { - chip->driver_data = (int)(uintptr_t)match->data; - } else { - acpi_id = acpi_match_device(pca953x_acpi_ids, &client->dev); + chip->driver_data = (uintptr_t)of_device_get_match_data(dev); + if (!chip->driver_data) { + acpi_id = acpi_match_device(pca953x_acpi_ids, dev); if (!acpi_id) { ret = -ENODEV; goto err_exit; @@ -936,8 +963,8 @@ static const struct of_device_id pca953x_dt_ids[] = { { .compatible = "nxp,pca9575", .data = OF_957X(16, PCA_INT), }, { .compatible = "nxp,pca9698", .data = OF_953X(40, 0), }, - { .compatible = "nxp,pcal6524", .data = OF_953X(24, PCA_INT), }, - { .compatible = "nxp,pcal9555a", .data = OF_953X(16, PCA_INT), }, + { .compatible = "nxp,pcal6524", .data = OF_953X(24, PCA_LATCH_INT), }, + { .compatible = "nxp,pcal9555a", .data = OF_953X(16, PCA_LATCH_INT), }, { .compatible = "maxim,max7310", .data = OF_953X( 8, 0), }, { .compatible = "maxim,max7312", .data = OF_953X(16, PCA_INT), }, diff --git a/drivers/gpio/gpio-pcf857x.c b/drivers/gpio/gpio-pcf857x.c index 38fbb420c6cd..adf72dda25a2 100644 --- a/drivers/gpio/gpio-pcf857x.c +++ b/drivers/gpio/gpio-pcf857x.c @@ -18,7 +18,7 @@ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ -#include +#include #include #include #include diff --git a/drivers/gpio/gpio-pch.c b/drivers/gpio/gpio-pch.c index 68c6d0c5a6d1..ffce0ab912ed 100644 --- a/drivers/gpio/gpio-pch.c +++ b/drivers/gpio/gpio-pch.c @@ -17,7 +17,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/gpio/gpio-pl061.c b/drivers/gpio/gpio-pl061.c index b70974cb9ef1..2afd9de84a0d 100644 --- a/drivers/gpio/gpio-pl061.c +++ b/drivers/gpio/gpio-pl061.c @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/drivers/gpio/gpio-pmic-eic-sprd.c b/drivers/gpio/gpio-pmic-eic-sprd.c index 66d68d991162..29e044ff4b17 100644 --- a/drivers/gpio/gpio-pmic-eic-sprd.c +++ b/drivers/gpio/gpio-pmic-eic-sprd.c @@ -178,6 +178,14 @@ static int sprd_pmic_eic_irq_set_type(struct irq_data *data, case IRQ_TYPE_LEVEL_LOW: pmic_eic->reg[REG_IEV] = 0; break; + case IRQ_TYPE_EDGE_RISING: + case IRQ_TYPE_EDGE_FALLING: + case IRQ_TYPE_EDGE_BOTH: + /* + * Will set the trigger level according to current EIC level + * in irq_bus_sync_unlock() interface, so here nothing to do. + */ + break; default: return -ENOTSUPP; } @@ -197,11 +205,22 @@ static void sprd_pmic_eic_bus_sync_unlock(struct irq_data *data) { struct gpio_chip *chip = irq_data_get_irq_chip_data(data); struct sprd_pmic_eic *pmic_eic = gpiochip_get_data(chip); + u32 trigger = irqd_get_trigger_type(data); u32 offset = irqd_to_hwirq(data); + int state; /* Set irq type */ - sprd_pmic_eic_update(chip, offset, SPRD_PMIC_EIC_IEV, - pmic_eic->reg[REG_IEV]); + if (trigger & IRQ_TYPE_EDGE_BOTH) { + state = sprd_pmic_eic_get(chip, offset); + if (state) + sprd_pmic_eic_update(chip, offset, SPRD_PMIC_EIC_IEV, 0); + else + sprd_pmic_eic_update(chip, offset, SPRD_PMIC_EIC_IEV, 1); + } else { + sprd_pmic_eic_update(chip, offset, SPRD_PMIC_EIC_IEV, + pmic_eic->reg[REG_IEV]); + } + /* Set irq unmask */ sprd_pmic_eic_update(chip, offset, SPRD_PMIC_EIC_IE, pmic_eic->reg[REG_IE]); @@ -212,6 +231,35 @@ static void sprd_pmic_eic_bus_sync_unlock(struct irq_data *data) mutex_unlock(&pmic_eic->buslock); } +static void sprd_pmic_eic_toggle_trigger(struct gpio_chip *chip, + unsigned int irq, unsigned int offset) +{ + u32 trigger = irq_get_trigger_type(irq); + int state, post_state; + + if (!(trigger & IRQ_TYPE_EDGE_BOTH)) + return; + + state = sprd_pmic_eic_get(chip, offset); +retry: + if (state) + sprd_pmic_eic_update(chip, offset, SPRD_PMIC_EIC_IEV, 0); + else + sprd_pmic_eic_update(chip, offset, SPRD_PMIC_EIC_IEV, 1); + + post_state = sprd_pmic_eic_get(chip, offset); + if (state != post_state) { + dev_warn(chip->parent, "PMIC EIC level was changed.\n"); + state = post_state; + goto retry; + } + + /* Set irq unmask */ + sprd_pmic_eic_update(chip, offset, SPRD_PMIC_EIC_IE, 1); + /* Generate trigger start pulse for debounce EIC */ + sprd_pmic_eic_update(chip, offset, SPRD_PMIC_EIC_TRIG, 1); +} + static irqreturn_t sprd_pmic_eic_irq_handler(int irq, void *data) { struct sprd_pmic_eic *pmic_eic = data; @@ -233,6 +281,12 @@ static irqreturn_t sprd_pmic_eic_irq_handler(int irq, void *data) girq = irq_find_mapping(chip->irq.domain, n); handle_nested_irq(girq); + + /* + * The PMIC EIC can only support level trigger, so we can + * toggle the level trigger to emulate the edge trigger. + */ + sprd_pmic_eic_toggle_trigger(chip, girq, n); } return IRQ_HANDLED; diff --git a/drivers/gpio/gpio-pxa.c b/drivers/gpio/gpio-pxa.c index f480fb896963..1e66f808051c 100644 --- a/drivers/gpio/gpio-pxa.c +++ b/drivers/gpio/gpio-pxa.c @@ -14,7 +14,7 @@ #include #include #include -#include +#include #include #include #include @@ -579,15 +579,9 @@ static int pxa_gpio_probe_dt(struct platform_device *pdev, struct pxa_gpio_chip *pchip) { int nr_gpios; - const struct of_device_id *of_id = - of_match_device(pxa_gpio_dt_ids, &pdev->dev); const struct pxa_gpio_id *gpio_id; - if (!of_id || !of_id->data) { - dev_err(&pdev->dev, "Failed to find gpio controller\n"); - return -EFAULT; - } - gpio_id = of_id->data; + gpio_id = of_device_get_match_data(&pdev->dev); gpio_type = gpio_id->type; nr_gpios = gpio_id->gpio_nums; diff --git a/drivers/gpio/gpio-stmpe.c b/drivers/gpio/gpio-stmpe.c index f8d7d1cd8488..8d6a5a7e612d 100644 --- a/drivers/gpio/gpio-stmpe.c +++ b/drivers/gpio/gpio-stmpe.c @@ -363,13 +363,15 @@ static struct irq_chip stmpe_gpio_irq_chip = { .irq_set_type = stmpe_gpio_irq_set_type, }; +#define MAX_GPIOS 24 + static irqreturn_t stmpe_gpio_irq(int irq, void *dev) { struct stmpe_gpio *stmpe_gpio = dev; struct stmpe *stmpe = stmpe_gpio->stmpe; u8 statmsbreg; int num_banks = DIV_ROUND_UP(stmpe->num_gpios, 8); - u8 status[num_banks]; + u8 status[DIV_ROUND_UP(MAX_GPIOS, 8)]; int ret; int i; @@ -434,6 +436,11 @@ static int stmpe_gpio_probe(struct platform_device *pdev) struct stmpe_gpio *stmpe_gpio; int ret, irq; + if (stmpe->num_gpios > MAX_GPIOS) { + dev_err(&pdev->dev, "Need to increase maximum GPIO number\n"); + return -EINVAL; + } + stmpe_gpio = kzalloc(sizeof(*stmpe_gpio), GFP_KERNEL); if (!stmpe_gpio) return -ENOMEM; diff --git a/drivers/gpio/gpio-syscon.c b/drivers/gpio/gpio-syscon.c index 537cec7583fc..8b0a69c5ba88 100644 --- a/drivers/gpio/gpio-syscon.c +++ b/drivers/gpio/gpio-syscon.c @@ -182,20 +182,15 @@ MODULE_DEVICE_TABLE(of, syscon_gpio_ids); static int syscon_gpio_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; - const struct of_device_id *of_id; struct syscon_gpio_priv *priv; struct device_node *np = dev->of_node; int ret; - of_id = of_match_device(syscon_gpio_ids, dev); - if (!of_id) - return -ENODEV; - priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL); if (!priv) return -ENOMEM; - priv->data = of_id->data; + priv->data = of_device_get_match_data(dev); if (priv->data->compatible) { priv->syscon = syscon_regmap_lookup_by_compatible( @@ -205,6 +200,8 @@ static int syscon_gpio_probe(struct platform_device *pdev) } else { priv->syscon = syscon_regmap_lookup_by_phandle(np, "gpio,syscon-dev"); + if (IS_ERR(priv->syscon) && np->parent) + priv->syscon = syscon_node_to_regmap(np->parent); if (IS_ERR(priv->syscon)) return PTR_ERR(priv->syscon); diff --git a/drivers/gpio/gpio-thunderx.c b/drivers/gpio/gpio-thunderx.c index d16e9d4a129b..1306722faa5a 100644 --- a/drivers/gpio/gpio-thunderx.c +++ b/drivers/gpio/gpio-thunderx.c @@ -504,16 +504,17 @@ static int thunderx_gpio_probe(struct pci_dev *pdev, txgpio->base_msi = (c >> 8) & 0xff; } - txgpio->msix_entries = devm_kzalloc(dev, - sizeof(struct msix_entry) * ngpio, + txgpio->msix_entries = devm_kcalloc(dev, + ngpio, sizeof(struct msix_entry), GFP_KERNEL); if (!txgpio->msix_entries) { err = -ENOMEM; goto out; } - txgpio->line_entries = devm_kzalloc(dev, - sizeof(struct thunderx_line) * ngpio, + txgpio->line_entries = devm_kcalloc(dev, + ngpio, + sizeof(struct thunderx_line), GFP_KERNEL); if (!txgpio->line_entries) { err = -ENOMEM; diff --git a/drivers/gpio/gpio-ts4900.c b/drivers/gpio/gpio-ts4900.c index 5bd21725e604..1da8d0586329 100644 --- a/drivers/gpio/gpio-ts4900.c +++ b/drivers/gpio/gpio-ts4900.c @@ -128,15 +128,10 @@ MODULE_DEVICE_TABLE(of, ts4900_gpio_of_match_table); static int ts4900_gpio_probe(struct i2c_client *client, const struct i2c_device_id *id) { - const struct of_device_id *match; struct ts4900_gpio_priv *priv; u32 ngpio; int ret; - match = of_match_device(ts4900_gpio_of_match_table, &client->dev); - if (!match) - return -EINVAL; - if (of_property_read_u32(client->dev.of_node, "ngpios", &ngpio)) ngpio = DEFAULT_PIN_NUMBER; @@ -148,7 +143,7 @@ static int ts4900_gpio_probe(struct i2c_client *client, priv->gpio_chip.label = "ts4900-gpio"; priv->gpio_chip.ngpio = ngpio; priv->gpio_chip.parent = &client->dev; - priv->input_bit = (uintptr_t)match->data; + priv->input_bit = (uintptr_t)of_device_get_match_data(&client->dev); priv->regmap = devm_regmap_init_i2c(client, &ts4900_regmap_config); if (IS_ERR(priv->regmap)) { diff --git a/drivers/gpio/gpio-uniphier.c b/drivers/gpio/gpio-uniphier.c index 761d8279abca..d3cf9502e7e7 100644 --- a/drivers/gpio/gpio-uniphier.c +++ b/drivers/gpio/gpio-uniphier.c @@ -371,8 +371,7 @@ static int uniphier_gpio_probe(struct platform_device *pdev) return ret; nregs = uniphier_gpio_get_nbanks(ngpios) * 2 + 3; - priv = devm_kzalloc(dev, - sizeof(*priv) + sizeof(priv->saved_vals[0]) * nregs, + priv = devm_kzalloc(dev, struct_size(priv, saved_vals, nregs), GFP_KERNEL); if (!priv) return -ENOMEM; diff --git a/drivers/gpio/gpio-vf610.c b/drivers/gpio/gpio-vf610.c index 4610cc2938ad..d4ad6d0e02a2 100644 --- a/drivers/gpio/gpio-vf610.c +++ b/drivers/gpio/gpio-vf610.c @@ -254,8 +254,6 @@ static struct irq_chip vf610_gpio_irq_chip = { static int vf610_gpio_probe(struct platform_device *pdev) { - const struct of_device_id *of_id = of_match_device(vf610_gpio_dt_ids, - &pdev->dev); struct device *dev = &pdev->dev; struct device_node *np = dev->of_node; struct vf610_gpio_port *port; @@ -267,7 +265,7 @@ static int vf610_gpio_probe(struct platform_device *pdev) if (!port) return -ENOMEM; - port->sdata = of_id->data; + port->sdata = of_device_get_match_data(dev); iores = platform_get_resource(pdev, IORESOURCE_MEM, 0); port->base = devm_ioremap_resource(dev, iores); if (IS_ERR(port->base)) diff --git a/drivers/gpio/gpio-xlp.c b/drivers/gpio/gpio-xlp.c index e74bd43a6974..8e4275eaa7d7 100644 --- a/drivers/gpio/gpio-xlp.c +++ b/drivers/gpio/gpio-xlp.c @@ -322,14 +322,7 @@ static int xlp_gpio_probe(struct platform_device *pdev) return irq; if (pdev->dev.of_node) { - const struct of_device_id *of_id; - - of_id = of_match_device(xlp_gpio_of_ids, &pdev->dev); - if (!of_id) { - dev_err(&pdev->dev, "Unable to match OF ID\n"); - return -ENODEV; - } - soc_type = (uintptr_t) of_id->data; + soc_type = (uintptr_t)of_device_get_match_data(&pdev->dev); } else { const struct acpi_device_id *acpi_id; diff --git a/drivers/gpio/gpio-xra1403.c b/drivers/gpio/gpio-xra1403.c index 8d4c8e99b251..8711a7907568 100644 --- a/drivers/gpio/gpio-xra1403.c +++ b/drivers/gpio/gpio-xra1403.c @@ -39,6 +39,7 @@ #define XRA_REIR 0x10 /* Input Rising Edge Interrupt Enable */ #define XRA_FEIR 0x12 /* Input Falling Edge Interrupt Enable */ #define XRA_IFR 0x14 /* Input Filter Enable/Disable */ +#define XRA_LAST 0x15 /* Bounds */ struct xra1403 { struct gpio_chip chip; @@ -50,7 +51,7 @@ static const struct regmap_config xra1403_regmap_cfg = { .pad_bits = 1, .val_bits = 8, - .max_register = XRA_IFR | 0x01, + .max_register = XRA_LAST, }; static unsigned int to_reg(unsigned int reg, unsigned int offset) @@ -126,21 +127,16 @@ static void xra1403_dbg_show(struct seq_file *s, struct gpio_chip *chip) { int reg; struct xra1403 *xra = gpiochip_get_data(chip); - int *value; + int value[XRA_LAST]; int i; unsigned int gcr; unsigned int gsr; - value = kmalloc_array(xra1403_regmap_cfg.max_register, sizeof(*value), - GFP_KERNEL); - if (!value) - return; - seq_puts(s, "xra reg:"); - for (reg = 0; reg <= xra1403_regmap_cfg.max_register; reg++) + for (reg = 0; reg <= XRA_LAST; reg++) seq_printf(s, " %2.2x", reg); seq_puts(s, "\n value:"); - for (reg = 0; reg < xra1403_regmap_cfg.max_register; reg++) { + for (reg = 0; reg < XRA_LAST; reg++) { regmap_read(xra->regmap, reg, &value[reg]); seq_printf(s, " %2.2x", value[reg]); } @@ -159,7 +155,6 @@ static void xra1403_dbg_show(struct seq_file *s, struct gpio_chip *chip) (gcr & BIT(i)) ? "in" : "out", (gsr & BIT(i)) ? "hi" : "lo"); } - kfree(value); } #else #define xra1403_dbg_show NULL diff --git a/drivers/gpio/gpio-zynq.c b/drivers/gpio/gpio-zynq.c index 75ee877e5cd5..3f5fcdd5a429 100644 --- a/drivers/gpio/gpio-zynq.c +++ b/drivers/gpio/gpio-zynq.c @@ -665,10 +665,8 @@ static void zynq_gpio_restore_context(struct zynq_gpio *gpio) static int __maybe_unused zynq_gpio_suspend(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - int irq = platform_get_irq(pdev, 0); - struct irq_data *data = irq_get_irq_data(irq); - struct zynq_gpio *gpio = platform_get_drvdata(pdev); + struct zynq_gpio *gpio = dev_get_drvdata(dev); + struct irq_data *data = irq_get_irq_data(gpio->irq); if (!irqd_is_wakeup_set(data)) { zynq_gpio_save_context(gpio); @@ -680,10 +678,8 @@ static int __maybe_unused zynq_gpio_suspend(struct device *dev) static int __maybe_unused zynq_gpio_resume(struct device *dev) { - struct platform_device *pdev = to_platform_device(dev); - int irq = platform_get_irq(pdev, 0); - struct irq_data *data = irq_get_irq_data(irq); - struct zynq_gpio *gpio = platform_get_drvdata(pdev); + struct zynq_gpio *gpio = dev_get_drvdata(dev); + struct irq_data *data = irq_get_irq_data(gpio->irq); int ret; if (!irqd_is_wakeup_set(data)) { @@ -831,7 +827,7 @@ static int zynq_gpio_probe(struct platform_device *pdev) chip->free = zynq_gpio_free; chip->direction_input = zynq_gpio_dir_in; chip->direction_output = zynq_gpio_dir_out; - chip->base = -1; + chip->base = of_alias_get_id(pdev->dev.of_node, "gpio"); chip->ngpio = gpio->p_data->ngpio; /* Retrieve GPIO clock */ diff --git a/drivers/gpio/gpiolib-of.c b/drivers/gpio/gpiolib-of.c index 586d15137c03..28d968088131 100644 --- a/drivers/gpio/gpiolib-of.c +++ b/drivers/gpio/gpiolib-of.c @@ -210,11 +210,8 @@ static struct gpio_desc *of_find_regulator_gpio(struct device *dev, const char * if (!con_id) return ERR_PTR(-ENOENT); - for (i = 0; i < ARRAY_SIZE(whitelist); i++) - if (!strcmp(con_id, whitelist[i])) - break; - - if (i == ARRAY_SIZE(whitelist)) + i = match_string(whitelist, ARRAY_SIZE(whitelist), con_id); + if (i < 0) return ERR_PTR(-ENOENT); desc = of_get_named_gpiod_flags(np, con_id, 0, of_flags); diff --git a/drivers/gpio/gpiolib.c b/drivers/gpio/gpiolib.c index d8ccb500872f..e11a3bb03820 100644 --- a/drivers/gpio/gpiolib.c +++ b/drivers/gpio/gpiolib.c @@ -61,6 +61,11 @@ static struct bus_type gpio_bus_type = { .name = "gpio", }; +/* + * Number of GPIOs to use for the fast path in set array + */ +#define FASTPATH_NGPIO CONFIG_GPIOLIB_FASTPATH_LIMIT + /* gpio_lock prevents conflicts during gpio_desc[] table updates. * While any GPIO is requested, its gpio_chip is not removable; * each GPIO's "requested" flag serves as a lock and refcount. @@ -71,6 +76,9 @@ static DEFINE_MUTEX(gpio_lookup_lock); static LIST_HEAD(gpio_lookup_list); LIST_HEAD(gpio_devices); +static DEFINE_MUTEX(gpio_machine_hogs_mutex); +static LIST_HEAD(gpio_machine_hogs); + static void gpiochip_free_hogs(struct gpio_chip *chip); static int gpiochip_add_irqchip(struct gpio_chip *gpiochip, struct lock_class_key *lock_key, @@ -450,12 +458,11 @@ static long linehandle_ioctl(struct file *filep, unsigned int cmd, vals[i] = !!ghd.values[i]; /* Reuse the array setting function */ - gpiod_set_array_value_complex(false, + return gpiod_set_array_value_complex(false, true, lh->numdescs, lh->descs, vals); - return 0; } return -EINVAL; } @@ -1172,6 +1179,41 @@ err_remove_device: return status; } +static void gpiochip_machine_hog(struct gpio_chip *chip, struct gpiod_hog *hog) +{ + struct gpio_desc *desc; + int rv; + + desc = gpiochip_get_desc(chip, hog->chip_hwnum); + if (IS_ERR(desc)) { + pr_err("%s: unable to get GPIO desc: %ld\n", + __func__, PTR_ERR(desc)); + return; + } + + if (test_bit(FLAG_IS_HOGGED, &desc->flags)) + return; + + rv = gpiod_hog(desc, hog->line_name, hog->lflags, hog->dflags); + if (rv) + pr_err("%s: unable to hog GPIO line (%s:%u): %d\n", + __func__, chip->label, hog->chip_hwnum, rv); +} + +static void machine_gpiochip_add(struct gpio_chip *chip) +{ + struct gpiod_hog *hog; + + mutex_lock(&gpio_machine_hogs_mutex); + + list_for_each_entry(hog, &gpio_machine_hogs, list) { + if (!strcmp(chip->label, hog->chip_label)) + gpiochip_machine_hog(chip, hog); + } + + mutex_unlock(&gpio_machine_hogs_mutex); +} + static void gpiochip_setup_devs(void) { struct gpio_device *gdev; @@ -1244,6 +1286,10 @@ int gpiochip_add_data_with_key(struct gpio_chip *chip, void *data, goto err_free_descs; } + if (chip->ngpio > FASTPATH_NGPIO) + chip_warn(chip, "line cnt %u is greater than fast path cnt %u\n", + chip->ngpio, FASTPATH_NGPIO); + gdev->label = kstrdup_const(chip->label ?: "unknown", GFP_KERNEL); if (!gdev->label) { status = -ENOMEM; @@ -1327,6 +1373,8 @@ int gpiochip_add_data_with_key(struct gpio_chip *chip, void *data, acpi_gpiochip_add(chip); + machine_gpiochip_add(chip); + /* * By first adding the chardev, and then adding the device, * we get a device node entry in sysfs under @@ -2078,6 +2126,11 @@ EXPORT_SYMBOL_GPL(gpiochip_generic_config); * @pctldev: the pin controller to map to * @gpio_offset: the start offset in the current gpio_chip number space * @pin_group: name of the pin group inside the pin controller + * + * Calling this function directly from a DeviceTree-supported + * pinctrl driver is DEPRECATED. Please see Section 2.1 of + * Documentation/devicetree/bindings/gpio/gpio.txt on how to + * bind pinctrl and gpio drivers via the "gpio-ranges" property. */ int gpiochip_add_pingroup_range(struct gpio_chip *chip, struct pinctrl_dev *pctldev, @@ -2131,6 +2184,11 @@ EXPORT_SYMBOL_GPL(gpiochip_add_pingroup_range); * * Returns: * 0 on success, or a negative error-code on failure. + * + * Calling this function directly from a DeviceTree-supported + * pinctrl driver is DEPRECATED. Please see Section 2.1 of + * Documentation/devicetree/bindings/gpio/gpio.txt on how to + * bind pinctrl and gpio drivers via the "gpio-ranges" property. */ int gpiochip_add_pin_range(struct gpio_chip *chip, const char *pinctl_name, unsigned int gpio_offset, unsigned int pin_offset, @@ -2719,16 +2777,28 @@ int gpiod_get_array_value_complex(bool raw, bool can_sleep, while (i < array_size) { struct gpio_chip *chip = desc_array[i]->gdev->chip; - unsigned long mask[BITS_TO_LONGS(chip->ngpio)]; - unsigned long bits[BITS_TO_LONGS(chip->ngpio)]; + unsigned long fastpath[2 * BITS_TO_LONGS(FASTPATH_NGPIO)]; + unsigned long *mask, *bits; int first, j, ret; + if (likely(chip->ngpio <= FASTPATH_NGPIO)) { + mask = fastpath; + } else { + mask = kmalloc_array(2 * BITS_TO_LONGS(chip->ngpio), + sizeof(*mask), + can_sleep ? GFP_KERNEL : GFP_ATOMIC); + if (!mask) + return -ENOMEM; + } + + bits = mask + BITS_TO_LONGS(chip->ngpio); + bitmap_zero(mask, chip->ngpio); + if (!can_sleep) WARN_ON(chip->can_sleep); /* collect all inputs belonging to the same chip */ first = i; - memset(mask, 0, sizeof(mask)); do { const struct gpio_desc *desc = desc_array[i]; int hwgpio = gpio_chip_hwgpio(desc); @@ -2739,8 +2809,11 @@ int gpiod_get_array_value_complex(bool raw, bool can_sleep, (desc_array[i]->gdev->chip == chip)); ret = gpio_chip_get_multiple(chip, mask, bits); - if (ret) + if (ret) { + if (mask != fastpath) + kfree(mask); return ret; + } for (j = first; j < i; j++) { const struct gpio_desc *desc = desc_array[j]; @@ -2752,6 +2825,9 @@ int gpiod_get_array_value_complex(bool raw, bool can_sleep, value_array[j] = value; trace_gpio_value(desc_to_gpio(desc), 1, value); } + + if (mask != fastpath) + kfree(mask); } return 0; } @@ -2935,7 +3011,7 @@ static void gpio_chip_set_multiple(struct gpio_chip *chip, } } -void gpiod_set_array_value_complex(bool raw, bool can_sleep, +int gpiod_set_array_value_complex(bool raw, bool can_sleep, unsigned int array_size, struct gpio_desc **desc_array, int *value_array) @@ -2944,14 +3020,26 @@ void gpiod_set_array_value_complex(bool raw, bool can_sleep, while (i < array_size) { struct gpio_chip *chip = desc_array[i]->gdev->chip; - unsigned long mask[BITS_TO_LONGS(chip->ngpio)]; - unsigned long bits[BITS_TO_LONGS(chip->ngpio)]; + unsigned long fastpath[2 * BITS_TO_LONGS(FASTPATH_NGPIO)]; + unsigned long *mask, *bits; int count = 0; + if (likely(chip->ngpio <= FASTPATH_NGPIO)) { + mask = fastpath; + } else { + mask = kmalloc_array(2 * BITS_TO_LONGS(chip->ngpio), + sizeof(*mask), + can_sleep ? GFP_KERNEL : GFP_ATOMIC); + if (!mask) + return -ENOMEM; + } + + bits = mask + BITS_TO_LONGS(chip->ngpio); + bitmap_zero(mask, chip->ngpio); + if (!can_sleep) WARN_ON(chip->can_sleep); - memset(mask, 0, sizeof(mask)); do { struct gpio_desc *desc = desc_array[i]; int hwgpio = gpio_chip_hwgpio(desc); @@ -2982,7 +3070,11 @@ void gpiod_set_array_value_complex(bool raw, bool can_sleep, /* push collected bits to outputs */ if (count != 0) gpio_chip_set_multiple(chip, mask, bits); + + if (mask != fastpath) + kfree(mask); } + return 0; } /** @@ -3057,13 +3149,13 @@ EXPORT_SYMBOL_GPL(gpiod_set_value); * This function should be called from contexts where we cannot sleep, and will * complain if the GPIO chip functions potentially sleep. */ -void gpiod_set_raw_array_value(unsigned int array_size, +int gpiod_set_raw_array_value(unsigned int array_size, struct gpio_desc **desc_array, int *value_array) { if (!desc_array) - return; - gpiod_set_array_value_complex(true, false, array_size, desc_array, - value_array); + return -EINVAL; + return gpiod_set_array_value_complex(true, false, array_size, + desc_array, value_array); } EXPORT_SYMBOL_GPL(gpiod_set_raw_array_value); @@ -3383,14 +3475,14 @@ EXPORT_SYMBOL_GPL(gpiod_set_value_cansleep); * * This function is to be called from contexts that can sleep. */ -void gpiod_set_raw_array_value_cansleep(unsigned int array_size, +int gpiod_set_raw_array_value_cansleep(unsigned int array_size, struct gpio_desc **desc_array, int *value_array) { might_sleep_if(extra_checks); if (!desc_array) - return; - gpiod_set_array_value_complex(true, true, array_size, desc_array, + return -EINVAL; + return gpiod_set_array_value_complex(true, true, array_size, desc_array, value_array); } EXPORT_SYMBOL_GPL(gpiod_set_raw_array_value_cansleep); @@ -3463,6 +3555,33 @@ void gpiod_remove_lookup_table(struct gpiod_lookup_table *table) } EXPORT_SYMBOL_GPL(gpiod_remove_lookup_table); +/** + * gpiod_add_hogs() - register a set of GPIO hogs from machine code + * @hogs: table of gpio hog entries with a zeroed sentinel at the end + */ +void gpiod_add_hogs(struct gpiod_hog *hogs) +{ + struct gpio_chip *chip; + struct gpiod_hog *hog; + + mutex_lock(&gpio_machine_hogs_mutex); + + for (hog = &hogs[0]; hog->chip_label; hog++) { + list_add_tail(&hog->list, &gpio_machine_hogs); + + /* + * The chip may have been registered earlier, so check if it + * exists and, if so, try to hog the line now. + */ + chip = find_chip_by_name(hog->chip_label); + if (chip) + gpiochip_machine_hog(chip, hog); + } + + mutex_unlock(&gpio_machine_hogs_mutex); +} +EXPORT_SYMBOL_GPL(gpiod_add_hogs); + static struct gpiod_lookup_table *gpiod_find_lookup_table(struct device *dev) { const char *dev_id = dev ? dev_name(dev) : NULL; @@ -4023,8 +4142,7 @@ struct gpio_descs *__must_check gpiod_get_array(struct device *dev, if (count < 0) return ERR_PTR(count); - descs = kzalloc(sizeof(*descs) + sizeof(descs->desc[0]) * count, - GFP_KERNEL); + descs = kzalloc(struct_size(descs, desc, count), GFP_KERNEL); if (!descs) return ERR_PTR(-ENOMEM); diff --git a/drivers/gpio/gpiolib.h b/drivers/gpio/gpiolib.h index ad456b6f9d8b..1a8e20363861 100644 --- a/drivers/gpio/gpiolib.h +++ b/drivers/gpio/gpiolib.h @@ -188,7 +188,7 @@ int gpiod_get_array_value_complex(bool raw, bool can_sleep, unsigned int array_size, struct gpio_desc **desc_array, int *value_array); -void gpiod_set_array_value_complex(bool raw, bool can_sleep, +int gpiod_set_array_value_complex(bool raw, bool can_sleep, unsigned int array_size, struct gpio_desc **desc_array, int *value_array); diff --git a/drivers/gpu/drm/Kconfig b/drivers/gpu/drm/Kconfig index deeefa7a1773..2a72d2feb76d 100644 --- a/drivers/gpu/drm/Kconfig +++ b/drivers/gpu/drm/Kconfig @@ -49,16 +49,17 @@ config DRM_DEBUG_MM If in doubt, say "N". -config DRM_DEBUG_MM_SELFTEST - tristate "kselftests for DRM range manager (struct drm_mm)" +config DRM_DEBUG_SELFTEST + tristate "kselftests for DRM" depends on DRM depends on DEBUG_KERNEL select PRIME_NUMBERS select DRM_LIB_RANDOM + select DRM_KMS_HELPER default n help - This option provides a kernel module that can be used to test - the DRM range manager (drm_mm) and its API. This option is not + This option provides kernel modules that can be used to run + various selftests on parts of the DRM api. This option is not useful for distributions or general kernels, but only for kernel developers working on DRM and associated drivers. @@ -267,6 +268,8 @@ source "drivers/gpu/drm/amd/amdkfd/Kconfig" source "drivers/gpu/drm/imx/Kconfig" +source "drivers/gpu/drm/v3d/Kconfig" + source "drivers/gpu/drm/vc4/Kconfig" source "drivers/gpu/drm/etnaviv/Kconfig" @@ -289,6 +292,8 @@ source "drivers/gpu/drm/pl111/Kconfig" source "drivers/gpu/drm/tve200/Kconfig" +source "drivers/gpu/drm/xen/Kconfig" + # Keep legacy drivers last menuconfig DRM_LEGACY diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile index 50093ff4479b..ef9f3dab287f 100644 --- a/drivers/gpu/drm/Makefile +++ b/drivers/gpu/drm/Makefile @@ -43,7 +43,7 @@ drm_kms_helper-$(CONFIG_DRM_KMS_CMA_HELPER) += drm_fb_cma_helper.o drm_kms_helper-$(CONFIG_DRM_DP_AUX_CHARDEV) += drm_dp_aux_dev.o obj-$(CONFIG_DRM_KMS_HELPER) += drm_kms_helper.o -obj-$(CONFIG_DRM_DEBUG_MM_SELFTEST) += selftests/ +obj-$(CONFIG_DRM_DEBUG_SELFTEST) += selftests/ obj-$(CONFIG_DRM) += drm.o obj-$(CONFIG_DRM_MIPI_DSI) += drm_mipi_dsi.o @@ -61,6 +61,7 @@ obj-$(CONFIG_DRM_MGA) += mga/ obj-$(CONFIG_DRM_I810) += i810/ obj-$(CONFIG_DRM_I915) += i915/ obj-$(CONFIG_DRM_MGAG200) += mgag200/ +obj-$(CONFIG_DRM_V3D) += v3d/ obj-$(CONFIG_DRM_VC4) += vc4/ obj-$(CONFIG_DRM_CIRRUS_QEMU) += cirrus/ obj-$(CONFIG_DRM_SIS) += sis/ @@ -103,3 +104,4 @@ obj-$(CONFIG_DRM_MXSFB) += mxsfb/ obj-$(CONFIG_DRM_TINYDRM) += tinydrm/ obj-$(CONFIG_DRM_PL111) += pl111/ obj-$(CONFIG_DRM_TVE200) += tve200/ +obj-$(CONFIG_DRM_XEN) += xen/ diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile b/drivers/gpu/drm/amd/amdgpu/Makefile index 2ca2b5154d52..bfd332c95b61 100644 --- a/drivers/gpu/drm/amd/amdgpu/Makefile +++ b/drivers/gpu/drm/amd/amdgpu/Makefile @@ -56,13 +56,18 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \ # add asic specific block amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \ - ci_smc.o ci_dpm.o dce_v8_0.o gfx_v7_0.o cik_sdma.o uvd_v4_2.o vce_v2_0.o \ - amdgpu_amdkfd_gfx_v7.o + ci_smc.o ci_dpm.o dce_v8_0.o gfx_v7_0.o cik_sdma.o uvd_v4_2.o vce_v2_0.o amdgpu-$(CONFIG_DRM_AMDGPU_SI)+= si.o gmc_v6_0.o gfx_v6_0.o si_ih.o si_dma.o dce_v6_0.o si_dpm.o si_smc.o amdgpu-y += \ - vi.o mxgpu_vi.o nbio_v6_1.o soc15.o emu_soc.o mxgpu_ai.o nbio_v7_0.o vega10_reg_init.o + vi.o mxgpu_vi.o nbio_v6_1.o soc15.o emu_soc.o mxgpu_ai.o nbio_v7_0.o vega10_reg_init.o \ + vega20_reg_init.o + +# add DF block +amdgpu-y += \ + df_v1_7.o \ + df_v3_6.o # add GMC block amdgpu-y += \ @@ -126,11 +131,20 @@ amdgpu-y += \ vcn_v1_0.o # add amdkfd interfaces +amdgpu-y += amdgpu_amdkfd.o + +ifneq ($(CONFIG_HSA_AMD),) amdgpu-y += \ - amdgpu_amdkfd.o \ amdgpu_amdkfd_fence.o \ amdgpu_amdkfd_gpuvm.o \ - amdgpu_amdkfd_gfx_v8.o + amdgpu_amdkfd_gfx_v8.o \ + amdgpu_amdkfd_gfx_v9.o + +ifneq ($(CONFIG_DRM_AMDGPU_CIK),) +amdgpu-y += amdgpu_amdkfd_gfx_v7.o +endif + +endif # add cgs amdgpu-y += amdgpu_cgs.o diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h index c8b605f3dc05..a59c07590cee 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h @@ -129,6 +129,7 @@ extern int amdgpu_lbpw; extern int amdgpu_compute_multipipe; extern int amdgpu_gpu_recovery; extern int amdgpu_emu_mode; +extern uint amdgpu_smu_memory_pool_size; #ifdef CONFIG_DRM_AMDGPU_SI extern int amdgpu_si_support; @@ -137,6 +138,7 @@ extern int amdgpu_si_support; extern int amdgpu_cik_support; #endif +#define AMDGPU_SG_THRESHOLD (256*1024*1024) #define AMDGPU_DEFAULT_GTT_SIZE_MB 3072ULL /* 3GB by default */ #define AMDGPU_WAIT_IDLE_TIMEOUT_IN_MS 3000 #define AMDGPU_MAX_USEC_TIMEOUT 100000 /* 100 ms */ @@ -222,10 +224,10 @@ enum amdgpu_kiq_irq { AMDGPU_CP_KIQ_IRQ_LAST }; -int amdgpu_device_ip_set_clockgating_state(struct amdgpu_device *adev, +int amdgpu_device_ip_set_clockgating_state(void *dev, enum amd_ip_block_type block_type, enum amd_clockgating_state state); -int amdgpu_device_ip_set_powergating_state(struct amdgpu_device *adev, +int amdgpu_device_ip_set_powergating_state(void *dev, enum amd_ip_block_type block_type, enum amd_powergating_state state); void amdgpu_device_ip_get_clockgating_state(struct amdgpu_device *adev, @@ -681,6 +683,8 @@ int amdgpu_ctx_ioctl(struct drm_device *dev, void *data, int amdgpu_ctx_wait_prev_fence(struct amdgpu_ctx *ctx, unsigned ring_id); void amdgpu_ctx_mgr_init(struct amdgpu_ctx_mgr *mgr); +void amdgpu_ctx_mgr_entity_cleanup(struct amdgpu_ctx_mgr *mgr); +void amdgpu_ctx_mgr_entity_fini(struct amdgpu_ctx_mgr *mgr); void amdgpu_ctx_mgr_fini(struct amdgpu_ctx_mgr *mgr); @@ -771,9 +775,18 @@ struct amdgpu_rlc { u32 starting_offsets_start; u32 reg_list_format_size_bytes; u32 reg_list_size_bytes; + u32 reg_list_format_direct_reg_list_length; + u32 save_restore_list_cntl_size_bytes; + u32 save_restore_list_gpm_size_bytes; + u32 save_restore_list_srm_size_bytes; u32 *register_list_format; u32 *register_restore; + u8 *save_restore_list_cntl; + u8 *save_restore_list_gpm; + u8 *save_restore_list_srm; + + bool is_rlc_v2_1; }; #define AMDGPU_MAX_COMPUTE_QUEUES KGD_MAX_QUEUES @@ -867,6 +880,8 @@ struct amdgpu_gfx_config { /* gfx configure feature */ uint32_t double_offchip_lds_buf; + /* cached value of DB_DEBUG2 */ + uint32_t db_debug2; }; struct amdgpu_cu_info { @@ -938,6 +953,12 @@ struct amdgpu_gfx { uint32_t ce_feature_version; uint32_t pfp_feature_version; uint32_t rlc_feature_version; + uint32_t rlc_srlc_fw_version; + uint32_t rlc_srlc_feature_version; + uint32_t rlc_srlg_fw_version; + uint32_t rlc_srlg_feature_version; + uint32_t rlc_srls_fw_version; + uint32_t rlc_srls_feature_version; uint32_t mec_feature_version; uint32_t mec2_feature_version; struct amdgpu_ring gfx_ring[AMDGPU_MAX_GFX_RINGS]; @@ -1204,6 +1225,8 @@ struct amdgpu_asic_funcs { /* invalidate hdp read cache */ void (*invalidate_hdp)(struct amdgpu_device *adev, struct amdgpu_ring *ring); + /* check if the asic needs a full reset of if soft reset will work */ + bool (*need_full_reset)(struct amdgpu_device *adev); }; /* @@ -1368,7 +1391,19 @@ struct amdgpu_nbio_funcs { void (*detect_hw_virt)(struct amdgpu_device *adev); }; - +struct amdgpu_df_funcs { + void (*init)(struct amdgpu_device *adev); + void (*enable_broadcast_mode)(struct amdgpu_device *adev, + bool enable); + u32 (*get_fb_channel_number)(struct amdgpu_device *adev); + u32 (*get_hbm_channel_number)(struct amdgpu_device *adev); + void (*update_medium_grain_clock_gating)(struct amdgpu_device *adev, + bool enable); + void (*get_clockgating_state)(struct amdgpu_device *adev, + u32 *flags); + void (*enable_ecc_force_par_wr_rmw)(struct amdgpu_device *adev, + bool enable); +}; /* Define the HW IP blocks will be used in driver , add more if necessary */ enum amd_hw_ip_block_type { GC_HWIP = 1, @@ -1398,6 +1433,7 @@ enum amd_hw_ip_block_type { struct amd_powerplay { void *pp_handle; const struct amd_pm_funcs *pp_funcs; + uint32_t pp_feature; }; #define AMDGPU_RESET_MAGIC_NUM 64 @@ -1590,6 +1626,7 @@ struct amdgpu_device { uint32_t *reg_offset[MAX_HWIP][HWIP_MAX_INSTANCE]; const struct amdgpu_nbio_funcs *nbio_funcs; + const struct amdgpu_df_funcs *df_funcs; /* delayed work_func for deferring clockgating during resume */ struct delayed_work late_init_work; @@ -1764,6 +1801,7 @@ amdgpu_get_sdma_instance(struct amdgpu_ring *ring) #define amdgpu_asic_get_config_memsize(adev) (adev)->asic_funcs->get_config_memsize((adev)) #define amdgpu_asic_flush_hdp(adev, r) (adev)->asic_funcs->flush_hdp((adev), (r)) #define amdgpu_asic_invalidate_hdp(adev, r) (adev)->asic_funcs->invalidate_hdp((adev), (r)) +#define amdgpu_asic_need_full_reset(adev) (adev)->asic_funcs->need_full_reset((adev)) #define amdgpu_gmc_flush_gpu_tlb(adev, vmid) (adev)->gmc.gmc_funcs->flush_gpu_tlb((adev), (vmid)) #define amdgpu_gmc_emit_flush_gpu_tlb(r, vmid, addr) (r)->adev->gmc.gmc_funcs->emit_flush_gpu_tlb((r), (vmid), (addr)) #define amdgpu_gmc_emit_pasid_mapping(r, vmid, pasid) (r)->adev->gmc.gmc_funcs->emit_pasid_mapping((r), (vmid), (pasid)) @@ -1790,6 +1828,7 @@ amdgpu_get_sdma_instance(struct amdgpu_ring *ring) #define amdgpu_ring_emit_rreg(r, d) (r)->funcs->emit_rreg((r), (d)) #define amdgpu_ring_emit_wreg(r, d, v) (r)->funcs->emit_wreg((r), (d), (v)) #define amdgpu_ring_emit_reg_wait(r, d, v, m) (r)->funcs->emit_reg_wait((r), (d), (v), (m)) +#define amdgpu_ring_emit_reg_write_reg_wait(r, d0, d1, v, m) (r)->funcs->emit_reg_write_reg_wait((r), (d0), (d1), (v), (m)) #define amdgpu_ring_emit_tmz(r, b) (r)->funcs->emit_tmz((r), (b)) #define amdgpu_ring_pad_ib(r, ib) ((r)->funcs->pad_ib((r), (ib))) #define amdgpu_ring_init_cond_exec(r) (r)->funcs->init_cond_exec((r)) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c index a29362f9ef41..f4c474a95875 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c @@ -290,12 +290,11 @@ static int acp_hw_init(void *handle) else if (r) return r; - r = cgs_get_pci_resource(adev->acp.cgs_device, CGS_RESOURCE_TYPE_MMIO, - 0x5289, 0, &acp_base); - if (r == -ENODEV) - return 0; - else if (r) - return r; + if (adev->rmmio_size == 0 || adev->rmmio_size < 0x5289) + return -EINVAL; + + acp_base = adev->rmmio_base; + if (adev->asic_type != CHIP_STONEY) { adev->acp.acp_genpd = kzalloc(sizeof(struct acp_pm_domain), GFP_KERNEL); if (adev->acp.acp_genpd == NULL) @@ -311,20 +310,20 @@ static int acp_hw_init(void *handle) pm_genpd_init(&adev->acp.acp_genpd->gpd, NULL, false); } - adev->acp.acp_cell = kzalloc(sizeof(struct mfd_cell) * ACP_DEVS, + adev->acp.acp_cell = kcalloc(ACP_DEVS, sizeof(struct mfd_cell), GFP_KERNEL); if (adev->acp.acp_cell == NULL) return -ENOMEM; - adev->acp.acp_res = kzalloc(sizeof(struct resource) * 4, GFP_KERNEL); + adev->acp.acp_res = kcalloc(4, sizeof(struct resource), GFP_KERNEL); if (adev->acp.acp_res == NULL) { kfree(adev->acp.acp_cell); return -ENOMEM; } - i2s_pdata = kzalloc(sizeof(struct i2s_platform_data) * 2, GFP_KERNEL); + i2s_pdata = kcalloc(2, sizeof(struct i2s_platform_data), GFP_KERNEL); if (i2s_pdata == NULL) { kfree(adev->acp.acp_res); kfree(adev->acp.acp_cell); @@ -513,7 +512,7 @@ static int acp_hw_fini(void *handle) if (adev->acp.acp_genpd) { for (i = 0; i < ACP_DEVS ; i++) { dev = get_mfd_cell_dev(adev->acp.acp_cell[i].name, i); - ret = pm_genpd_remove_device(&adev->acp.acp_genpd->gpd, dev); + ret = pm_genpd_remove_device(dev); /* If removal fails, dont giveup and try rest */ if (ret) dev_err(dev, "remove dev from genpd failed\n"); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c index 4d36203ffb11..305143fcc1ce 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c @@ -50,15 +50,21 @@ int amdgpu_amdkfd_init(void) kgd2kfd = NULL; } + #elif defined(CONFIG_HSA_AMD) + ret = kgd2kfd_init(KFD_INTERFACE_VERSION, &kgd2kfd); if (ret) kgd2kfd = NULL; #else + kgd2kfd = NULL; ret = -ENOENT; #endif + +#if defined(CONFIG_HSA_AMD_MODULE) || defined(CONFIG_HSA_AMD) amdgpu_amdkfd_gpuvm_init_mem_limits(); +#endif return ret; } @@ -92,8 +98,12 @@ void amdgpu_amdkfd_device_probe(struct amdgpu_device *adev) case CHIP_POLARIS11: kfd2kgd = amdgpu_amdkfd_gfx_8_0_get_functions(); break; + case CHIP_VEGA10: + case CHIP_RAVEN: + kfd2kgd = amdgpu_amdkfd_gfx_9_0_get_functions(); + break; default: - dev_dbg(adev->dev, "kfd not supported on this ASIC\n"); + dev_info(adev->dev, "kfd not supported on this ASIC\n"); return; } @@ -175,6 +185,28 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev) &gpu_resources.doorbell_physical_address, &gpu_resources.doorbell_aperture_size, &gpu_resources.doorbell_start_offset); + if (adev->asic_type >= CHIP_VEGA10) { + /* On SOC15 the BIF is involved in routing + * doorbells using the low 12 bits of the + * address. Communicate the assignments to + * KFD. KFD uses two doorbell pages per + * process in case of 64-bit doorbells so we + * can use each doorbell assignment twice. + */ + gpu_resources.sdma_doorbell[0][0] = + AMDGPU_DOORBELL64_sDMA_ENGINE0; + gpu_resources.sdma_doorbell[0][1] = + AMDGPU_DOORBELL64_sDMA_ENGINE0 + 0x200; + gpu_resources.sdma_doorbell[1][0] = + AMDGPU_DOORBELL64_sDMA_ENGINE1; + gpu_resources.sdma_doorbell[1][1] = + AMDGPU_DOORBELL64_sDMA_ENGINE1 + 0x200; + /* Doorbells 0x0f0-0ff and 0x2f0-2ff are reserved for + * SDMA, IH and VCN. So don't use them for the CP. + */ + gpu_resources.reserved_doorbell_mask = 0x1f0; + gpu_resources.reserved_doorbell_val = 0x0f0; + } kgd2kfd->device_init(adev->kfd, &gpu_resources); } @@ -217,13 +249,19 @@ int alloc_gtt_mem(struct kgd_dev *kgd, size_t size, { struct amdgpu_device *adev = (struct amdgpu_device *)kgd; struct amdgpu_bo *bo = NULL; + struct amdgpu_bo_param bp; int r; uint64_t gpu_addr_tmp = 0; void *cpu_ptr_tmp = NULL; - r = amdgpu_bo_create(adev, size, PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT, - AMDGPU_GEM_CREATE_CPU_GTT_USWC, ttm_bo_type_kernel, - NULL, &bo); + memset(&bp, 0, sizeof(bp)); + bp.size = size; + bp.byte_align = PAGE_SIZE; + bp.domain = AMDGPU_GEM_DOMAIN_GTT; + bp.flags = AMDGPU_GEM_CREATE_CPU_GTT_USWC; + bp.type = ttm_bo_type_kernel; + bp.resv = NULL; + r = amdgpu_bo_create(adev, &bp, &bo); if (r) { dev_err(adev->dev, "failed to allocate BO for amdkfd (%d)\n", r); @@ -304,15 +342,12 @@ void get_local_mem_info(struct kgd_dev *kgd, mem_info->local_mem_size_public, mem_info->local_mem_size_private); - if (amdgpu_emu_mode == 1) { - mem_info->mem_clk_max = 100; - return; - } - if (amdgpu_sriov_vf(adev)) mem_info->mem_clk_max = adev->clock.default_mclk / 100; - else + else if (adev->powerplay.pp_funcs) mem_info->mem_clk_max = amdgpu_dpm_get_mclk(adev, false) / 100; + else + mem_info->mem_clk_max = 100; } uint64_t get_gpu_clock_counter(struct kgd_dev *kgd) @@ -329,13 +364,12 @@ uint32_t get_max_engine_clock_in_mhz(struct kgd_dev *kgd) struct amdgpu_device *adev = (struct amdgpu_device *)kgd; /* the sclk is in quantas of 10kHz */ - if (amdgpu_emu_mode == 1) - return 100; - if (amdgpu_sriov_vf(adev)) return adev->clock.default_sclk / 100; - - return amdgpu_dpm_get_sclk(adev, false) / 100; + else if (adev->powerplay.pp_funcs) + return amdgpu_dpm_get_sclk(adev, false) / 100; + else + return 100; } void get_cu_info(struct kgd_dev *kgd, struct kfd_cu_info *cu_info) @@ -432,3 +466,44 @@ bool amdgpu_amdkfd_is_kfd_vmid(struct amdgpu_device *adev, u32 vmid) return false; } + +#if !defined(CONFIG_HSA_AMD_MODULE) && !defined(CONFIG_HSA_AMD) +bool amdkfd_fence_check_mm(struct dma_fence *f, struct mm_struct *mm) +{ + return false; +} + +void amdgpu_amdkfd_unreserve_system_memory_limit(struct amdgpu_bo *bo) +{ +} + +void amdgpu_amdkfd_gpuvm_destroy_cb(struct amdgpu_device *adev, + struct amdgpu_vm *vm) +{ +} + +struct amdgpu_amdkfd_fence *to_amdgpu_amdkfd_fence(struct dma_fence *f) +{ + return NULL; +} + +int amdgpu_amdkfd_evict_userptr(struct kgd_mem *mem, struct mm_struct *mm) +{ + return 0; +} + +struct kfd2kgd_calls *amdgpu_amdkfd_gfx_7_get_functions(void) +{ + return NULL; +} + +struct kfd2kgd_calls *amdgpu_amdkfd_gfx_8_0_get_functions(void) +{ + return NULL; +} + +struct kfd2kgd_calls *amdgpu_amdkfd_gfx_9_0_get_functions(void) +{ + return NULL; +} +#endif diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h index c2c2bea731e0..a8418a3f4e9d 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include "amdgpu_sync.h" @@ -59,7 +60,9 @@ struct kgd_mem { uint32_t mapping_flags; + atomic_t invalid; struct amdkfd_process_info *process_info; + struct page **user_pages; struct amdgpu_sync sync; @@ -84,6 +87,9 @@ struct amdkfd_process_info { struct list_head vm_list_head; /* List head for all KFD BOs that belong to a KFD process. */ struct list_head kfd_bo_list; + /* List of userptr BOs that are valid or invalid */ + struct list_head userptr_valid_list; + struct list_head userptr_inval_list; /* Lock to protect kfd_bo_list */ struct mutex lock; @@ -91,6 +97,11 @@ struct amdkfd_process_info { unsigned int n_vms; /* Eviction Fence */ struct amdgpu_amdkfd_fence *eviction_fence; + + /* MMU-notifier related fields */ + atomic_t evicted_bos; + struct delayed_work restore_userptr_work; + struct pid *pid; }; int amdgpu_amdkfd_init(void); @@ -104,12 +115,14 @@ void amdgpu_amdkfd_device_probe(struct amdgpu_device *adev); void amdgpu_amdkfd_device_init(struct amdgpu_device *adev); void amdgpu_amdkfd_device_fini(struct amdgpu_device *adev); +int amdgpu_amdkfd_evict_userptr(struct kgd_mem *mem, struct mm_struct *mm); int amdgpu_amdkfd_submit_ib(struct kgd_dev *kgd, enum kgd_engine_type engine, uint32_t vmid, uint64_t gpu_addr, uint32_t *ib_cmd, uint32_t ib_len); struct kfd2kgd_calls *amdgpu_amdkfd_gfx_7_get_functions(void); struct kfd2kgd_calls *amdgpu_amdkfd_gfx_8_0_get_functions(void); +struct kfd2kgd_calls *amdgpu_amdkfd_gfx_9_0_get_functions(void); bool amdgpu_amdkfd_is_kfd_vmid(struct amdgpu_device *adev, u32 vmid); @@ -143,14 +156,14 @@ uint64_t amdgpu_amdkfd_get_vram_usage(struct kgd_dev *kgd); /* GPUVM API */ int amdgpu_amdkfd_gpuvm_create_process_vm(struct kgd_dev *kgd, void **vm, - void **process_info, - struct dma_fence **ef); + void **process_info, + struct dma_fence **ef); int amdgpu_amdkfd_gpuvm_acquire_process_vm(struct kgd_dev *kgd, - struct file *filp, - void **vm, void **process_info, - struct dma_fence **ef); + struct file *filp, + void **vm, void **process_info, + struct dma_fence **ef); void amdgpu_amdkfd_gpuvm_destroy_cb(struct amdgpu_device *adev, - struct amdgpu_vm *vm); + struct amdgpu_vm *vm); void amdgpu_amdkfd_gpuvm_destroy_process_vm(struct kgd_dev *kgd, void *vm); uint32_t amdgpu_amdkfd_gpuvm_get_process_page_dir(void *vm); int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v7.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v7.c index ea54e53172b9..ea79908dac4c 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v7.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v7.c @@ -98,8 +98,6 @@ static void kgd_program_sh_mem_settings(struct kgd_dev *kgd, uint32_t vmid, static int kgd_set_pasid_vmid_mapping(struct kgd_dev *kgd, unsigned int pasid, unsigned int vmid); -static int kgd_init_pipeline(struct kgd_dev *kgd, uint32_t pipe_id, - uint32_t hpd_size, uint64_t hpd_gpu_addr); static int kgd_init_interrupts(struct kgd_dev *kgd, uint32_t pipe_id); static int kgd_hqd_load(struct kgd_dev *kgd, void *mqd, uint32_t pipe_id, uint32_t queue_id, uint32_t __user *wptr, @@ -183,7 +181,6 @@ static const struct kfd2kgd_calls kfd2kgd = { .free_pasid = amdgpu_pasid_free, .program_sh_mem_settings = kgd_program_sh_mem_settings, .set_pasid_vmid_mapping = kgd_set_pasid_vmid_mapping, - .init_pipeline = kgd_init_pipeline, .init_interrupts = kgd_init_interrupts, .hqd_load = kgd_hqd_load, .hqd_sdma_load = kgd_hqd_sdma_load, @@ -309,13 +306,6 @@ static int kgd_set_pasid_vmid_mapping(struct kgd_dev *kgd, unsigned int pasid, return 0; } -static int kgd_init_pipeline(struct kgd_dev *kgd, uint32_t pipe_id, - uint32_t hpd_size, uint64_t hpd_gpu_addr) -{ - /* amdgpu owns the per-pipe state */ - return 0; -} - static int kgd_init_interrupts(struct kgd_dev *kgd, uint32_t pipe_id) { struct amdgpu_device *adev = get_amdgpu_device(kgd); @@ -417,7 +407,7 @@ static int kgd_hqd_dump(struct kgd_dev *kgd, (*dump)[i++][1] = RREG32(addr); \ } while (0) - *dump = kmalloc(HQD_N_REGS*2*sizeof(uint32_t), GFP_KERNEL); + *dump = kmalloc_array(HQD_N_REGS * 2, sizeof(uint32_t), GFP_KERNEL); if (*dump == NULL) return -ENOMEM; @@ -514,7 +504,7 @@ static int kgd_hqd_sdma_dump(struct kgd_dev *kgd, #undef HQD_N_REGS #define HQD_N_REGS (19+4) - *dump = kmalloc(HQD_N_REGS*2*sizeof(uint32_t), GFP_KERNEL); + *dump = kmalloc_array(HQD_N_REGS * 2, sizeof(uint32_t), GFP_KERNEL); if (*dump == NULL) return -ENOMEM; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v8.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v8.c index 89264c9a5e9f..19dd665e7307 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v8.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v8.c @@ -57,8 +57,6 @@ static void kgd_program_sh_mem_settings(struct kgd_dev *kgd, uint32_t vmid, uint32_t sh_mem_bases); static int kgd_set_pasid_vmid_mapping(struct kgd_dev *kgd, unsigned int pasid, unsigned int vmid); -static int kgd_init_pipeline(struct kgd_dev *kgd, uint32_t pipe_id, - uint32_t hpd_size, uint64_t hpd_gpu_addr); static int kgd_init_interrupts(struct kgd_dev *kgd, uint32_t pipe_id); static int kgd_hqd_load(struct kgd_dev *kgd, void *mqd, uint32_t pipe_id, uint32_t queue_id, uint32_t __user *wptr, @@ -141,7 +139,6 @@ static const struct kfd2kgd_calls kfd2kgd = { .free_pasid = amdgpu_pasid_free, .program_sh_mem_settings = kgd_program_sh_mem_settings, .set_pasid_vmid_mapping = kgd_set_pasid_vmid_mapping, - .init_pipeline = kgd_init_pipeline, .init_interrupts = kgd_init_interrupts, .hqd_load = kgd_hqd_load, .hqd_sdma_load = kgd_hqd_sdma_load, @@ -270,13 +267,6 @@ static int kgd_set_pasid_vmid_mapping(struct kgd_dev *kgd, unsigned int pasid, return 0; } -static int kgd_init_pipeline(struct kgd_dev *kgd, uint32_t pipe_id, - uint32_t hpd_size, uint64_t hpd_gpu_addr) -{ - /* amdgpu owns the per-pipe state */ - return 0; -} - static int kgd_init_interrupts(struct kgd_dev *kgd, uint32_t pipe_id) { struct amdgpu_device *adev = get_amdgpu_device(kgd); @@ -405,7 +395,7 @@ static int kgd_hqd_dump(struct kgd_dev *kgd, (*dump)[i++][1] = RREG32(addr); \ } while (0) - *dump = kmalloc(HQD_N_REGS*2*sizeof(uint32_t), GFP_KERNEL); + *dump = kmalloc_array(HQD_N_REGS * 2, sizeof(uint32_t), GFP_KERNEL); if (*dump == NULL) return -ENOMEM; @@ -501,7 +491,7 @@ static int kgd_hqd_sdma_dump(struct kgd_dev *kgd, #undef HQD_N_REGS #define HQD_N_REGS (19+4+2+3+7) - *dump = kmalloc(HQD_N_REGS*2*sizeof(uint32_t), GFP_KERNEL); + *dump = kmalloc_array(HQD_N_REGS * 2, sizeof(uint32_t), GFP_KERNEL); if (*dump == NULL) return -ENOMEM; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c new file mode 100644 index 000000000000..1db60aa5b7f0 --- /dev/null +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v9.c @@ -0,0 +1,1043 @@ +/* + * Copyright 2014-2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#define pr_fmt(fmt) "kfd2kgd: " fmt + +#include +#include +#include +#include +#include +#include "amdgpu.h" +#include "amdgpu_amdkfd.h" +#include "amdgpu_ucode.h" +#include "soc15_hw_ip.h" +#include "gc/gc_9_0_offset.h" +#include "gc/gc_9_0_sh_mask.h" +#include "vega10_enum.h" +#include "sdma0/sdma0_4_0_offset.h" +#include "sdma0/sdma0_4_0_sh_mask.h" +#include "sdma1/sdma1_4_0_offset.h" +#include "sdma1/sdma1_4_0_sh_mask.h" +#include "athub/athub_1_0_offset.h" +#include "athub/athub_1_0_sh_mask.h" +#include "oss/osssys_4_0_offset.h" +#include "oss/osssys_4_0_sh_mask.h" +#include "soc15_common.h" +#include "v9_structs.h" +#include "soc15.h" +#include "soc15d.h" + +/* HACK: MMHUB and GC both have VM-related register with the same + * names but different offsets. Define the MMHUB register we need here + * with a prefix. A proper solution would be to move the functions + * programming these registers into gfx_v9_0.c and mmhub_v1_0.c + * respectively. + */ +#define mmMMHUB_VM_INVALIDATE_ENG16_REQ 0x06f3 +#define mmMMHUB_VM_INVALIDATE_ENG16_REQ_BASE_IDX 0 + +#define mmMMHUB_VM_INVALIDATE_ENG16_ACK 0x0705 +#define mmMMHUB_VM_INVALIDATE_ENG16_ACK_BASE_IDX 0 + +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32 0x072b +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32_BASE_IDX 0 +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_HI32 0x072c +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_HI32_BASE_IDX 0 + +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_START_ADDR_LO32 0x074b +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_START_ADDR_LO32_BASE_IDX 0 +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_START_ADDR_HI32 0x074c +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_START_ADDR_HI32_BASE_IDX 0 + +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_END_ADDR_LO32 0x076b +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_END_ADDR_LO32_BASE_IDX 0 +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_END_ADDR_HI32 0x076c +#define mmMMHUB_VM_CONTEXT0_PAGE_TABLE_END_ADDR_HI32_BASE_IDX 0 + +#define mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_LO32 0x0727 +#define mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_LO32_BASE_IDX 0 +#define mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_HI32 0x0728 +#define mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_HI32_BASE_IDX 0 + +#define V9_PIPE_PER_MEC (4) +#define V9_QUEUES_PER_PIPE_MEC (8) + +enum hqd_dequeue_request_type { + NO_ACTION = 0, + DRAIN_PIPE, + RESET_WAVES +}; + +/* + * Register access functions + */ + +static void kgd_program_sh_mem_settings(struct kgd_dev *kgd, uint32_t vmid, + uint32_t sh_mem_config, + uint32_t sh_mem_ape1_base, uint32_t sh_mem_ape1_limit, + uint32_t sh_mem_bases); +static int kgd_set_pasid_vmid_mapping(struct kgd_dev *kgd, unsigned int pasid, + unsigned int vmid); +static int kgd_init_interrupts(struct kgd_dev *kgd, uint32_t pipe_id); +static int kgd_hqd_load(struct kgd_dev *kgd, void *mqd, uint32_t pipe_id, + uint32_t queue_id, uint32_t __user *wptr, + uint32_t wptr_shift, uint32_t wptr_mask, + struct mm_struct *mm); +static int kgd_hqd_dump(struct kgd_dev *kgd, + uint32_t pipe_id, uint32_t queue_id, + uint32_t (**dump)[2], uint32_t *n_regs); +static int kgd_hqd_sdma_load(struct kgd_dev *kgd, void *mqd, + uint32_t __user *wptr, struct mm_struct *mm); +static int kgd_hqd_sdma_dump(struct kgd_dev *kgd, + uint32_t engine_id, uint32_t queue_id, + uint32_t (**dump)[2], uint32_t *n_regs); +static bool kgd_hqd_is_occupied(struct kgd_dev *kgd, uint64_t queue_address, + uint32_t pipe_id, uint32_t queue_id); +static bool kgd_hqd_sdma_is_occupied(struct kgd_dev *kgd, void *mqd); +static int kgd_hqd_destroy(struct kgd_dev *kgd, void *mqd, + enum kfd_preempt_type reset_type, + unsigned int utimeout, uint32_t pipe_id, + uint32_t queue_id); +static int kgd_hqd_sdma_destroy(struct kgd_dev *kgd, void *mqd, + unsigned int utimeout); +static int kgd_address_watch_disable(struct kgd_dev *kgd); +static int kgd_address_watch_execute(struct kgd_dev *kgd, + unsigned int watch_point_id, + uint32_t cntl_val, + uint32_t addr_hi, + uint32_t addr_lo); +static int kgd_wave_control_execute(struct kgd_dev *kgd, + uint32_t gfx_index_val, + uint32_t sq_cmd); +static uint32_t kgd_address_watch_get_offset(struct kgd_dev *kgd, + unsigned int watch_point_id, + unsigned int reg_offset); + +static bool get_atc_vmid_pasid_mapping_valid(struct kgd_dev *kgd, + uint8_t vmid); +static uint16_t get_atc_vmid_pasid_mapping_pasid(struct kgd_dev *kgd, + uint8_t vmid); +static void set_vm_context_page_table_base(struct kgd_dev *kgd, uint32_t vmid, + uint32_t page_table_base); +static uint16_t get_fw_version(struct kgd_dev *kgd, enum kgd_engine_type type); +static void set_scratch_backing_va(struct kgd_dev *kgd, + uint64_t va, uint32_t vmid); +static int invalidate_tlbs(struct kgd_dev *kgd, uint16_t pasid); +static int invalidate_tlbs_vmid(struct kgd_dev *kgd, uint16_t vmid); + +/* Because of REG_GET_FIELD() being used, we put this function in the + * asic specific file. + */ +static int amdgpu_amdkfd_get_tile_config(struct kgd_dev *kgd, + struct tile_config *config) +{ + struct amdgpu_device *adev = (struct amdgpu_device *)kgd; + + config->gb_addr_config = adev->gfx.config.gb_addr_config; + + config->tile_config_ptr = adev->gfx.config.tile_mode_array; + config->num_tile_configs = + ARRAY_SIZE(adev->gfx.config.tile_mode_array); + config->macro_tile_config_ptr = + adev->gfx.config.macrotile_mode_array; + config->num_macro_tile_configs = + ARRAY_SIZE(adev->gfx.config.macrotile_mode_array); + + return 0; +} + +static const struct kfd2kgd_calls kfd2kgd = { + .init_gtt_mem_allocation = alloc_gtt_mem, + .free_gtt_mem = free_gtt_mem, + .get_local_mem_info = get_local_mem_info, + .get_gpu_clock_counter = get_gpu_clock_counter, + .get_max_engine_clock_in_mhz = get_max_engine_clock_in_mhz, + .alloc_pasid = amdgpu_pasid_alloc, + .free_pasid = amdgpu_pasid_free, + .program_sh_mem_settings = kgd_program_sh_mem_settings, + .set_pasid_vmid_mapping = kgd_set_pasid_vmid_mapping, + .init_interrupts = kgd_init_interrupts, + .hqd_load = kgd_hqd_load, + .hqd_sdma_load = kgd_hqd_sdma_load, + .hqd_dump = kgd_hqd_dump, + .hqd_sdma_dump = kgd_hqd_sdma_dump, + .hqd_is_occupied = kgd_hqd_is_occupied, + .hqd_sdma_is_occupied = kgd_hqd_sdma_is_occupied, + .hqd_destroy = kgd_hqd_destroy, + .hqd_sdma_destroy = kgd_hqd_sdma_destroy, + .address_watch_disable = kgd_address_watch_disable, + .address_watch_execute = kgd_address_watch_execute, + .wave_control_execute = kgd_wave_control_execute, + .address_watch_get_offset = kgd_address_watch_get_offset, + .get_atc_vmid_pasid_mapping_pasid = + get_atc_vmid_pasid_mapping_pasid, + .get_atc_vmid_pasid_mapping_valid = + get_atc_vmid_pasid_mapping_valid, + .get_fw_version = get_fw_version, + .set_scratch_backing_va = set_scratch_backing_va, + .get_tile_config = amdgpu_amdkfd_get_tile_config, + .get_cu_info = get_cu_info, + .get_vram_usage = amdgpu_amdkfd_get_vram_usage, + .create_process_vm = amdgpu_amdkfd_gpuvm_create_process_vm, + .acquire_process_vm = amdgpu_amdkfd_gpuvm_acquire_process_vm, + .destroy_process_vm = amdgpu_amdkfd_gpuvm_destroy_process_vm, + .get_process_page_dir = amdgpu_amdkfd_gpuvm_get_process_page_dir, + .set_vm_context_page_table_base = set_vm_context_page_table_base, + .alloc_memory_of_gpu = amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu, + .free_memory_of_gpu = amdgpu_amdkfd_gpuvm_free_memory_of_gpu, + .map_memory_to_gpu = amdgpu_amdkfd_gpuvm_map_memory_to_gpu, + .unmap_memory_to_gpu = amdgpu_amdkfd_gpuvm_unmap_memory_from_gpu, + .sync_memory = amdgpu_amdkfd_gpuvm_sync_memory, + .map_gtt_bo_to_kernel = amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel, + .restore_process_bos = amdgpu_amdkfd_gpuvm_restore_process_bos, + .invalidate_tlbs = invalidate_tlbs, + .invalidate_tlbs_vmid = invalidate_tlbs_vmid, + .submit_ib = amdgpu_amdkfd_submit_ib, +}; + +struct kfd2kgd_calls *amdgpu_amdkfd_gfx_9_0_get_functions(void) +{ + return (struct kfd2kgd_calls *)&kfd2kgd; +} + +static inline struct amdgpu_device *get_amdgpu_device(struct kgd_dev *kgd) +{ + return (struct amdgpu_device *)kgd; +} + +static void lock_srbm(struct kgd_dev *kgd, uint32_t mec, uint32_t pipe, + uint32_t queue, uint32_t vmid) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + + mutex_lock(&adev->srbm_mutex); + soc15_grbm_select(adev, mec, pipe, queue, vmid); +} + +static void unlock_srbm(struct kgd_dev *kgd) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + + soc15_grbm_select(adev, 0, 0, 0, 0); + mutex_unlock(&adev->srbm_mutex); +} + +static void acquire_queue(struct kgd_dev *kgd, uint32_t pipe_id, + uint32_t queue_id) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + + uint32_t mec = (pipe_id / adev->gfx.mec.num_pipe_per_mec) + 1; + uint32_t pipe = (pipe_id % adev->gfx.mec.num_pipe_per_mec); + + lock_srbm(kgd, mec, pipe, queue_id, 0); +} + +static uint32_t get_queue_mask(struct amdgpu_device *adev, + uint32_t pipe_id, uint32_t queue_id) +{ + unsigned int bit = (pipe_id * adev->gfx.mec.num_queue_per_pipe + + queue_id) & 31; + + return ((uint32_t)1) << bit; +} + +static void release_queue(struct kgd_dev *kgd) +{ + unlock_srbm(kgd); +} + +static void kgd_program_sh_mem_settings(struct kgd_dev *kgd, uint32_t vmid, + uint32_t sh_mem_config, + uint32_t sh_mem_ape1_base, + uint32_t sh_mem_ape1_limit, + uint32_t sh_mem_bases) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + + lock_srbm(kgd, 0, 0, 0, vmid); + + WREG32(SOC15_REG_OFFSET(GC, 0, mmSH_MEM_CONFIG), sh_mem_config); + WREG32(SOC15_REG_OFFSET(GC, 0, mmSH_MEM_BASES), sh_mem_bases); + /* APE1 no longer exists on GFX9 */ + + unlock_srbm(kgd); +} + +static int kgd_set_pasid_vmid_mapping(struct kgd_dev *kgd, unsigned int pasid, + unsigned int vmid) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + + /* + * We have to assume that there is no outstanding mapping. + * The ATC_VMID_PASID_MAPPING_UPDATE_STATUS bit could be 0 because + * a mapping is in progress or because a mapping finished + * and the SW cleared it. + * So the protocol is to always wait & clear. + */ + uint32_t pasid_mapping = (pasid == 0) ? 0 : (uint32_t)pasid | + ATC_VMID0_PASID_MAPPING__VALID_MASK; + + /* + * need to do this twice, once for gfx and once for mmhub + * for ATC add 16 to VMID for mmhub, for IH different registers. + * ATC_VMID0..15 registers are separate from ATC_VMID16..31. + */ + + WREG32(SOC15_REG_OFFSET(ATHUB, 0, mmATC_VMID0_PASID_MAPPING) + vmid, + pasid_mapping); + + while (!(RREG32(SOC15_REG_OFFSET( + ATHUB, 0, + mmATC_VMID_PASID_MAPPING_UPDATE_STATUS)) & + (1U << vmid))) + cpu_relax(); + + WREG32(SOC15_REG_OFFSET(ATHUB, 0, + mmATC_VMID_PASID_MAPPING_UPDATE_STATUS), + 1U << vmid); + + /* Mapping vmid to pasid also for IH block */ + WREG32(SOC15_REG_OFFSET(OSSSYS, 0, mmIH_VMID_0_LUT) + vmid, + pasid_mapping); + + WREG32(SOC15_REG_OFFSET(ATHUB, 0, mmATC_VMID16_PASID_MAPPING) + vmid, + pasid_mapping); + + while (!(RREG32(SOC15_REG_OFFSET( + ATHUB, 0, + mmATC_VMID_PASID_MAPPING_UPDATE_STATUS)) & + (1U << (vmid + 16)))) + cpu_relax(); + + WREG32(SOC15_REG_OFFSET(ATHUB, 0, + mmATC_VMID_PASID_MAPPING_UPDATE_STATUS), + 1U << (vmid + 16)); + + /* Mapping vmid to pasid also for IH block */ + WREG32(SOC15_REG_OFFSET(OSSSYS, 0, mmIH_VMID_0_LUT_MM) + vmid, + pasid_mapping); + return 0; +} + +/* TODO - RING0 form of field is obsolete, seems to date back to SI + * but still works + */ + +static int kgd_init_interrupts(struct kgd_dev *kgd, uint32_t pipe_id) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + uint32_t mec; + uint32_t pipe; + + mec = (pipe_id / adev->gfx.mec.num_pipe_per_mec) + 1; + pipe = (pipe_id % adev->gfx.mec.num_pipe_per_mec); + + lock_srbm(kgd, mec, pipe, 0, 0); + + WREG32(SOC15_REG_OFFSET(GC, 0, mmCPC_INT_CNTL), + CP_INT_CNTL_RING0__TIME_STAMP_INT_ENABLE_MASK | + CP_INT_CNTL_RING0__OPCODE_ERROR_INT_ENABLE_MASK); + + unlock_srbm(kgd); + + return 0; +} + +static uint32_t get_sdma_base_addr(struct amdgpu_device *adev, + unsigned int engine_id, + unsigned int queue_id) +{ + uint32_t base[2] = { + SOC15_REG_OFFSET(SDMA0, 0, + mmSDMA0_RLC0_RB_CNTL) - mmSDMA0_RLC0_RB_CNTL, + SOC15_REG_OFFSET(SDMA1, 0, + mmSDMA1_RLC0_RB_CNTL) - mmSDMA1_RLC0_RB_CNTL + }; + uint32_t retval; + + retval = base[engine_id] + queue_id * (mmSDMA0_RLC1_RB_CNTL - + mmSDMA0_RLC0_RB_CNTL); + + pr_debug("sdma base address: 0x%x\n", retval); + + return retval; +} + +static inline struct v9_mqd *get_mqd(void *mqd) +{ + return (struct v9_mqd *)mqd; +} + +static inline struct v9_sdma_mqd *get_sdma_mqd(void *mqd) +{ + return (struct v9_sdma_mqd *)mqd; +} + +static int kgd_hqd_load(struct kgd_dev *kgd, void *mqd, uint32_t pipe_id, + uint32_t queue_id, uint32_t __user *wptr, + uint32_t wptr_shift, uint32_t wptr_mask, + struct mm_struct *mm) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + struct v9_mqd *m; + uint32_t *mqd_hqd; + uint32_t reg, hqd_base, data; + + m = get_mqd(mqd); + + acquire_queue(kgd, pipe_id, queue_id); + + /* HIQ is set during driver init period with vmid set to 0*/ + if (m->cp_hqd_vmid == 0) { + uint32_t value, mec, pipe; + + mec = (pipe_id / adev->gfx.mec.num_pipe_per_mec) + 1; + pipe = (pipe_id % adev->gfx.mec.num_pipe_per_mec); + + pr_debug("kfd: set HIQ, mec:%d, pipe:%d, queue:%d.\n", + mec, pipe, queue_id); + value = RREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_CP_SCHEDULERS)); + value = REG_SET_FIELD(value, RLC_CP_SCHEDULERS, scheduler1, + ((mec << 5) | (pipe << 3) | queue_id | 0x80)); + WREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_CP_SCHEDULERS), value); + } + + /* HQD registers extend from CP_MQD_BASE_ADDR to CP_HQD_EOP_WPTR_MEM. */ + mqd_hqd = &m->cp_mqd_base_addr_lo; + hqd_base = SOC15_REG_OFFSET(GC, 0, mmCP_MQD_BASE_ADDR); + + for (reg = hqd_base; + reg <= SOC15_REG_OFFSET(GC, 0, mmCP_HQD_PQ_WPTR_HI); reg++) + WREG32(reg, mqd_hqd[reg - hqd_base]); + + + /* Activate doorbell logic before triggering WPTR poll. */ + data = REG_SET_FIELD(m->cp_hqd_pq_doorbell_control, + CP_HQD_PQ_DOORBELL_CONTROL, DOORBELL_EN, 1); + WREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_PQ_DOORBELL_CONTROL), data); + + if (wptr) { + /* Don't read wptr with get_user because the user + * context may not be accessible (if this function + * runs in a work queue). Instead trigger a one-shot + * polling read from memory in the CP. This assumes + * that wptr is GPU-accessible in the queue's VMID via + * ATC or SVM. WPTR==RPTR before starting the poll so + * the CP starts fetching new commands from the right + * place. + * + * Guessing a 64-bit WPTR from a 32-bit RPTR is a bit + * tricky. Assume that the queue didn't overflow. The + * number of valid bits in the 32-bit RPTR depends on + * the queue size. The remaining bits are taken from + * the saved 64-bit WPTR. If the WPTR wrapped, add the + * queue size. + */ + uint32_t queue_size = + 2 << REG_GET_FIELD(m->cp_hqd_pq_control, + CP_HQD_PQ_CONTROL, QUEUE_SIZE); + uint64_t guessed_wptr = m->cp_hqd_pq_rptr & (queue_size - 1); + + if ((m->cp_hqd_pq_wptr_lo & (queue_size - 1)) < guessed_wptr) + guessed_wptr += queue_size; + guessed_wptr += m->cp_hqd_pq_wptr_lo & ~(queue_size - 1); + guessed_wptr += (uint64_t)m->cp_hqd_pq_wptr_hi << 32; + + WREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_PQ_WPTR_LO), + lower_32_bits(guessed_wptr)); + WREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_PQ_WPTR_HI), + upper_32_bits(guessed_wptr)); + WREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_PQ_WPTR_POLL_ADDR), + lower_32_bits((uintptr_t)wptr)); + WREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_PQ_WPTR_POLL_ADDR_HI), + upper_32_bits((uintptr_t)wptr)); + WREG32(SOC15_REG_OFFSET(GC, 0, mmCP_PQ_WPTR_POLL_CNTL1), + get_queue_mask(adev, pipe_id, queue_id)); + } + + /* Start the EOP fetcher */ + WREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_EOP_RPTR), + REG_SET_FIELD(m->cp_hqd_eop_rptr, + CP_HQD_EOP_RPTR, INIT_FETCHER, 1)); + + data = REG_SET_FIELD(m->cp_hqd_active, CP_HQD_ACTIVE, ACTIVE, 1); + WREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_ACTIVE), data); + + release_queue(kgd); + + return 0; +} + +static int kgd_hqd_dump(struct kgd_dev *kgd, + uint32_t pipe_id, uint32_t queue_id, + uint32_t (**dump)[2], uint32_t *n_regs) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + uint32_t i = 0, reg; +#define HQD_N_REGS 56 +#define DUMP_REG(addr) do { \ + if (WARN_ON_ONCE(i >= HQD_N_REGS)) \ + break; \ + (*dump)[i][0] = (addr) << 2; \ + (*dump)[i++][1] = RREG32(addr); \ + } while (0) + + *dump = kmalloc_array(HQD_N_REGS * 2, sizeof(uint32_t), GFP_KERNEL); + if (*dump == NULL) + return -ENOMEM; + + acquire_queue(kgd, pipe_id, queue_id); + + for (reg = SOC15_REG_OFFSET(GC, 0, mmCP_MQD_BASE_ADDR); + reg <= SOC15_REG_OFFSET(GC, 0, mmCP_HQD_PQ_WPTR_HI); reg++) + DUMP_REG(reg); + + release_queue(kgd); + + WARN_ON_ONCE(i != HQD_N_REGS); + *n_regs = i; + + return 0; +} + +static int kgd_hqd_sdma_load(struct kgd_dev *kgd, void *mqd, + uint32_t __user *wptr, struct mm_struct *mm) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + struct v9_sdma_mqd *m; + uint32_t sdma_base_addr, sdmax_gfx_context_cntl; + unsigned long end_jiffies; + uint32_t data; + uint64_t data64; + uint64_t __user *wptr64 = (uint64_t __user *)wptr; + + m = get_sdma_mqd(mqd); + sdma_base_addr = get_sdma_base_addr(adev, m->sdma_engine_id, + m->sdma_queue_id); + sdmax_gfx_context_cntl = m->sdma_engine_id ? + SOC15_REG_OFFSET(SDMA1, 0, mmSDMA1_GFX_CONTEXT_CNTL) : + SOC15_REG_OFFSET(SDMA0, 0, mmSDMA0_GFX_CONTEXT_CNTL); + + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_CNTL, + m->sdmax_rlcx_rb_cntl & (~SDMA0_RLC0_RB_CNTL__RB_ENABLE_MASK)); + + end_jiffies = msecs_to_jiffies(2000) + jiffies; + while (true) { + data = RREG32(sdma_base_addr + mmSDMA0_RLC0_CONTEXT_STATUS); + if (data & SDMA0_RLC0_CONTEXT_STATUS__IDLE_MASK) + break; + if (time_after(jiffies, end_jiffies)) + return -ETIME; + usleep_range(500, 1000); + } + data = RREG32(sdmax_gfx_context_cntl); + data = REG_SET_FIELD(data, SDMA0_GFX_CONTEXT_CNTL, + RESUME_CTX, 0); + WREG32(sdmax_gfx_context_cntl, data); + + WREG32(sdma_base_addr + mmSDMA0_RLC0_DOORBELL_OFFSET, + m->sdmax_rlcx_doorbell_offset); + + data = REG_SET_FIELD(m->sdmax_rlcx_doorbell, SDMA0_RLC0_DOORBELL, + ENABLE, 1); + WREG32(sdma_base_addr + mmSDMA0_RLC0_DOORBELL, data); + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_RPTR, m->sdmax_rlcx_rb_rptr); + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_RPTR_HI, + m->sdmax_rlcx_rb_rptr_hi); + + WREG32(sdma_base_addr + mmSDMA0_RLC0_MINOR_PTR_UPDATE, 1); + if (read_user_wptr(mm, wptr64, data64)) { + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_WPTR, + lower_32_bits(data64)); + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_WPTR_HI, + upper_32_bits(data64)); + } else { + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_WPTR, + m->sdmax_rlcx_rb_rptr); + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_WPTR_HI, + m->sdmax_rlcx_rb_rptr_hi); + } + WREG32(sdma_base_addr + mmSDMA0_RLC0_MINOR_PTR_UPDATE, 0); + + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_BASE, m->sdmax_rlcx_rb_base); + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_BASE_HI, + m->sdmax_rlcx_rb_base_hi); + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_RPTR_ADDR_LO, + m->sdmax_rlcx_rb_rptr_addr_lo); + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_RPTR_ADDR_HI, + m->sdmax_rlcx_rb_rptr_addr_hi); + + data = REG_SET_FIELD(m->sdmax_rlcx_rb_cntl, SDMA0_RLC0_RB_CNTL, + RB_ENABLE, 1); + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_CNTL, data); + + return 0; +} + +static int kgd_hqd_sdma_dump(struct kgd_dev *kgd, + uint32_t engine_id, uint32_t queue_id, + uint32_t (**dump)[2], uint32_t *n_regs) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + uint32_t sdma_base_addr = get_sdma_base_addr(adev, engine_id, queue_id); + uint32_t i = 0, reg; +#undef HQD_N_REGS +#define HQD_N_REGS (19+6+7+10) + + *dump = kmalloc_array(HQD_N_REGS * 2, sizeof(uint32_t), GFP_KERNEL); + if (*dump == NULL) + return -ENOMEM; + + for (reg = mmSDMA0_RLC0_RB_CNTL; reg <= mmSDMA0_RLC0_DOORBELL; reg++) + DUMP_REG(sdma_base_addr + reg); + for (reg = mmSDMA0_RLC0_STATUS; reg <= mmSDMA0_RLC0_CSA_ADDR_HI; reg++) + DUMP_REG(sdma_base_addr + reg); + for (reg = mmSDMA0_RLC0_IB_SUB_REMAIN; + reg <= mmSDMA0_RLC0_MINOR_PTR_UPDATE; reg++) + DUMP_REG(sdma_base_addr + reg); + for (reg = mmSDMA0_RLC0_MIDCMD_DATA0; + reg <= mmSDMA0_RLC0_MIDCMD_CNTL; reg++) + DUMP_REG(sdma_base_addr + reg); + + WARN_ON_ONCE(i != HQD_N_REGS); + *n_regs = i; + + return 0; +} + +static bool kgd_hqd_is_occupied(struct kgd_dev *kgd, uint64_t queue_address, + uint32_t pipe_id, uint32_t queue_id) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + uint32_t act; + bool retval = false; + uint32_t low, high; + + acquire_queue(kgd, pipe_id, queue_id); + act = RREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_ACTIVE)); + if (act) { + low = lower_32_bits(queue_address >> 8); + high = upper_32_bits(queue_address >> 8); + + if (low == RREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_PQ_BASE)) && + high == RREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_PQ_BASE_HI))) + retval = true; + } + release_queue(kgd); + return retval; +} + +static bool kgd_hqd_sdma_is_occupied(struct kgd_dev *kgd, void *mqd) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + struct v9_sdma_mqd *m; + uint32_t sdma_base_addr; + uint32_t sdma_rlc_rb_cntl; + + m = get_sdma_mqd(mqd); + sdma_base_addr = get_sdma_base_addr(adev, m->sdma_engine_id, + m->sdma_queue_id); + + sdma_rlc_rb_cntl = RREG32(sdma_base_addr + mmSDMA0_RLC0_RB_CNTL); + + if (sdma_rlc_rb_cntl & SDMA0_RLC0_RB_CNTL__RB_ENABLE_MASK) + return true; + + return false; +} + +static int kgd_hqd_destroy(struct kgd_dev *kgd, void *mqd, + enum kfd_preempt_type reset_type, + unsigned int utimeout, uint32_t pipe_id, + uint32_t queue_id) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + enum hqd_dequeue_request_type type; + unsigned long end_jiffies; + uint32_t temp; + struct v9_mqd *m = get_mqd(mqd); + + acquire_queue(kgd, pipe_id, queue_id); + + if (m->cp_hqd_vmid == 0) + WREG32_FIELD15(GC, 0, RLC_CP_SCHEDULERS, scheduler1, 0); + + switch (reset_type) { + case KFD_PREEMPT_TYPE_WAVEFRONT_DRAIN: + type = DRAIN_PIPE; + break; + case KFD_PREEMPT_TYPE_WAVEFRONT_RESET: + type = RESET_WAVES; + break; + default: + type = DRAIN_PIPE; + break; + } + + WREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_DEQUEUE_REQUEST), type); + + end_jiffies = (utimeout * HZ / 1000) + jiffies; + while (true) { + temp = RREG32(SOC15_REG_OFFSET(GC, 0, mmCP_HQD_ACTIVE)); + if (!(temp & CP_HQD_ACTIVE__ACTIVE_MASK)) + break; + if (time_after(jiffies, end_jiffies)) { + pr_err("cp queue preemption time out.\n"); + release_queue(kgd); + return -ETIME; + } + usleep_range(500, 1000); + } + + release_queue(kgd); + return 0; +} + +static int kgd_hqd_sdma_destroy(struct kgd_dev *kgd, void *mqd, + unsigned int utimeout) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + struct v9_sdma_mqd *m; + uint32_t sdma_base_addr; + uint32_t temp; + unsigned long end_jiffies = (utimeout * HZ / 1000) + jiffies; + + m = get_sdma_mqd(mqd); + sdma_base_addr = get_sdma_base_addr(adev, m->sdma_engine_id, + m->sdma_queue_id); + + temp = RREG32(sdma_base_addr + mmSDMA0_RLC0_RB_CNTL); + temp = temp & ~SDMA0_RLC0_RB_CNTL__RB_ENABLE_MASK; + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_CNTL, temp); + + while (true) { + temp = RREG32(sdma_base_addr + mmSDMA0_RLC0_CONTEXT_STATUS); + if (temp & SDMA0_RLC0_CONTEXT_STATUS__IDLE_MASK) + break; + if (time_after(jiffies, end_jiffies)) + return -ETIME; + usleep_range(500, 1000); + } + + WREG32(sdma_base_addr + mmSDMA0_RLC0_DOORBELL, 0); + WREG32(sdma_base_addr + mmSDMA0_RLC0_RB_CNTL, + RREG32(sdma_base_addr + mmSDMA0_RLC0_RB_CNTL) | + SDMA0_RLC0_RB_CNTL__RB_ENABLE_MASK); + + m->sdmax_rlcx_rb_rptr = RREG32(sdma_base_addr + mmSDMA0_RLC0_RB_RPTR); + m->sdmax_rlcx_rb_rptr_hi = + RREG32(sdma_base_addr + mmSDMA0_RLC0_RB_RPTR_HI); + + return 0; +} + +static bool get_atc_vmid_pasid_mapping_valid(struct kgd_dev *kgd, + uint8_t vmid) +{ + uint32_t reg; + struct amdgpu_device *adev = (struct amdgpu_device *) kgd; + + reg = RREG32(SOC15_REG_OFFSET(ATHUB, 0, mmATC_VMID0_PASID_MAPPING) + + vmid); + return reg & ATC_VMID0_PASID_MAPPING__VALID_MASK; +} + +static uint16_t get_atc_vmid_pasid_mapping_pasid(struct kgd_dev *kgd, + uint8_t vmid) +{ + uint32_t reg; + struct amdgpu_device *adev = (struct amdgpu_device *) kgd; + + reg = RREG32(SOC15_REG_OFFSET(ATHUB, 0, mmATC_VMID0_PASID_MAPPING) + + vmid); + return reg & ATC_VMID0_PASID_MAPPING__PASID_MASK; +} + +static void write_vmid_invalidate_request(struct kgd_dev *kgd, uint8_t vmid) +{ + struct amdgpu_device *adev = (struct amdgpu_device *) kgd; + uint32_t req = (1 << vmid) | + (0 << VM_INVALIDATE_ENG16_REQ__FLUSH_TYPE__SHIFT) | /* legacy */ + VM_INVALIDATE_ENG16_REQ__INVALIDATE_L2_PTES_MASK | + VM_INVALIDATE_ENG16_REQ__INVALIDATE_L2_PDE0_MASK | + VM_INVALIDATE_ENG16_REQ__INVALIDATE_L2_PDE1_MASK | + VM_INVALIDATE_ENG16_REQ__INVALIDATE_L2_PDE2_MASK | + VM_INVALIDATE_ENG16_REQ__INVALIDATE_L1_PTES_MASK; + + mutex_lock(&adev->srbm_mutex); + + /* Use legacy mode tlb invalidation. + * + * Currently on Raven the code below is broken for anything but + * legacy mode due to a MMHUB power gating problem. A workaround + * is for MMHUB to wait until the condition PER_VMID_INVALIDATE_REQ + * == PER_VMID_INVALIDATE_ACK instead of simply waiting for the ack + * bit. + * + * TODO 1: agree on the right set of invalidation registers for + * KFD use. Use the last one for now. Invalidate both GC and + * MMHUB. + * + * TODO 2: support range-based invalidation, requires kfg2kgd + * interface change + */ + WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_INVALIDATE_ENG16_ADDR_RANGE_LO32), + 0xffffffff); + WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_INVALIDATE_ENG16_ADDR_RANGE_HI32), + 0x0000001f); + + WREG32(SOC15_REG_OFFSET(MMHUB, 0, + mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_LO32), + 0xffffffff); + WREG32(SOC15_REG_OFFSET(MMHUB, 0, + mmMMHUB_VM_INVALIDATE_ENG16_ADDR_RANGE_HI32), + 0x0000001f); + + WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_INVALIDATE_ENG16_REQ), req); + + WREG32(SOC15_REG_OFFSET(MMHUB, 0, mmMMHUB_VM_INVALIDATE_ENG16_REQ), + req); + + while (!(RREG32(SOC15_REG_OFFSET(GC, 0, mmVM_INVALIDATE_ENG16_ACK)) & + (1 << vmid))) + cpu_relax(); + + while (!(RREG32(SOC15_REG_OFFSET(MMHUB, 0, + mmMMHUB_VM_INVALIDATE_ENG16_ACK)) & + (1 << vmid))) + cpu_relax(); + + mutex_unlock(&adev->srbm_mutex); + +} + +static int invalidate_tlbs_with_kiq(struct amdgpu_device *adev, uint16_t pasid) +{ + signed long r; + uint32_t seq; + struct amdgpu_ring *ring = &adev->gfx.kiq.ring; + + spin_lock(&adev->gfx.kiq.ring_lock); + amdgpu_ring_alloc(ring, 12); /* fence + invalidate_tlbs package*/ + amdgpu_ring_write(ring, PACKET3(PACKET3_INVALIDATE_TLBS, 0)); + amdgpu_ring_write(ring, + PACKET3_INVALIDATE_TLBS_DST_SEL(1) | + PACKET3_INVALIDATE_TLBS_ALL_HUB(1) | + PACKET3_INVALIDATE_TLBS_PASID(pasid) | + PACKET3_INVALIDATE_TLBS_FLUSH_TYPE(0)); /* legacy */ + amdgpu_fence_emit_polling(ring, &seq); + amdgpu_ring_commit(ring); + spin_unlock(&adev->gfx.kiq.ring_lock); + + r = amdgpu_fence_wait_polling(ring, seq, adev->usec_timeout); + if (r < 1) { + DRM_ERROR("wait for kiq fence error: %ld.\n", r); + return -ETIME; + } + + return 0; +} + +static int invalidate_tlbs(struct kgd_dev *kgd, uint16_t pasid) +{ + struct amdgpu_device *adev = (struct amdgpu_device *) kgd; + int vmid; + struct amdgpu_ring *ring = &adev->gfx.kiq.ring; + + if (ring->ready) + return invalidate_tlbs_with_kiq(adev, pasid); + + for (vmid = 0; vmid < 16; vmid++) { + if (!amdgpu_amdkfd_is_kfd_vmid(adev, vmid)) + continue; + if (get_atc_vmid_pasid_mapping_valid(kgd, vmid)) { + if (get_atc_vmid_pasid_mapping_pasid(kgd, vmid) + == pasid) { + write_vmid_invalidate_request(kgd, vmid); + break; + } + } + } + + return 0; +} + +static int invalidate_tlbs_vmid(struct kgd_dev *kgd, uint16_t vmid) +{ + struct amdgpu_device *adev = (struct amdgpu_device *) kgd; + + if (!amdgpu_amdkfd_is_kfd_vmid(adev, vmid)) { + pr_err("non kfd vmid %d\n", vmid); + return 0; + } + + write_vmid_invalidate_request(kgd, vmid); + return 0; +} + +static int kgd_address_watch_disable(struct kgd_dev *kgd) +{ + return 0; +} + +static int kgd_address_watch_execute(struct kgd_dev *kgd, + unsigned int watch_point_id, + uint32_t cntl_val, + uint32_t addr_hi, + uint32_t addr_lo) +{ + return 0; +} + +static int kgd_wave_control_execute(struct kgd_dev *kgd, + uint32_t gfx_index_val, + uint32_t sq_cmd) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + uint32_t data = 0; + + mutex_lock(&adev->grbm_idx_mutex); + + WREG32(SOC15_REG_OFFSET(GC, 0, mmGRBM_GFX_INDEX), gfx_index_val); + WREG32(SOC15_REG_OFFSET(GC, 0, mmSQ_CMD), sq_cmd); + + data = REG_SET_FIELD(data, GRBM_GFX_INDEX, + INSTANCE_BROADCAST_WRITES, 1); + data = REG_SET_FIELD(data, GRBM_GFX_INDEX, + SH_BROADCAST_WRITES, 1); + data = REG_SET_FIELD(data, GRBM_GFX_INDEX, + SE_BROADCAST_WRITES, 1); + + WREG32(SOC15_REG_OFFSET(GC, 0, mmGRBM_GFX_INDEX), data); + mutex_unlock(&adev->grbm_idx_mutex); + + return 0; +} + +static uint32_t kgd_address_watch_get_offset(struct kgd_dev *kgd, + unsigned int watch_point_id, + unsigned int reg_offset) +{ + return 0; +} + +static void set_scratch_backing_va(struct kgd_dev *kgd, + uint64_t va, uint32_t vmid) +{ + /* No longer needed on GFXv9. The scratch base address is + * passed to the shader by the CP. It's the user mode driver's + * responsibility. + */ +} + +/* FIXME: Does this need to be ASIC-specific code? */ +static uint16_t get_fw_version(struct kgd_dev *kgd, enum kgd_engine_type type) +{ + struct amdgpu_device *adev = (struct amdgpu_device *) kgd; + const union amdgpu_firmware_header *hdr; + + switch (type) { + case KGD_ENGINE_PFP: + hdr = (const union amdgpu_firmware_header *)adev->gfx.pfp_fw->data; + break; + + case KGD_ENGINE_ME: + hdr = (const union amdgpu_firmware_header *)adev->gfx.me_fw->data; + break; + + case KGD_ENGINE_CE: + hdr = (const union amdgpu_firmware_header *)adev->gfx.ce_fw->data; + break; + + case KGD_ENGINE_MEC1: + hdr = (const union amdgpu_firmware_header *)adev->gfx.mec_fw->data; + break; + + case KGD_ENGINE_MEC2: + hdr = (const union amdgpu_firmware_header *)adev->gfx.mec2_fw->data; + break; + + case KGD_ENGINE_RLC: + hdr = (const union amdgpu_firmware_header *)adev->gfx.rlc_fw->data; + break; + + case KGD_ENGINE_SDMA1: + hdr = (const union amdgpu_firmware_header *)adev->sdma.instance[0].fw->data; + break; + + case KGD_ENGINE_SDMA2: + hdr = (const union amdgpu_firmware_header *)adev->sdma.instance[1].fw->data; + break; + + default: + return 0; + } + + if (hdr == NULL) + return 0; + + /* Only 12 bit in use*/ + return hdr->common.ucode_version; +} + +static void set_vm_context_page_table_base(struct kgd_dev *kgd, uint32_t vmid, + uint32_t page_table_base) +{ + struct amdgpu_device *adev = get_amdgpu_device(kgd); + uint64_t base = (uint64_t)page_table_base << PAGE_SHIFT | + AMDGPU_PTE_VALID; + + if (!amdgpu_amdkfd_is_kfd_vmid(adev, vmid)) { + pr_err("trying to set page table base for wrong VMID %u\n", + vmid); + return; + } + + /* TODO: take advantage of per-process address space size. For + * now, all processes share the same address space size, like + * on GFX8 and older. + */ + WREG32(SOC15_REG_OFFSET(MMHUB, 0, mmMMHUB_VM_CONTEXT0_PAGE_TABLE_START_ADDR_LO32) + (vmid*2), 0); + WREG32(SOC15_REG_OFFSET(MMHUB, 0, mmMMHUB_VM_CONTEXT0_PAGE_TABLE_START_ADDR_HI32) + (vmid*2), 0); + + WREG32(SOC15_REG_OFFSET(MMHUB, 0, mmMMHUB_VM_CONTEXT0_PAGE_TABLE_END_ADDR_LO32) + (vmid*2), + lower_32_bits(adev->vm_manager.max_pfn - 1)); + WREG32(SOC15_REG_OFFSET(MMHUB, 0, mmMMHUB_VM_CONTEXT0_PAGE_TABLE_END_ADDR_HI32) + (vmid*2), + upper_32_bits(adev->vm_manager.max_pfn - 1)); + + WREG32(SOC15_REG_OFFSET(MMHUB, 0, mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32) + (vmid*2), lower_32_bits(base)); + WREG32(SOC15_REG_OFFSET(MMHUB, 0, mmMMHUB_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_HI32) + (vmid*2), upper_32_bits(base)); + + WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_CONTEXT0_PAGE_TABLE_START_ADDR_LO32) + (vmid*2), 0); + WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_CONTEXT0_PAGE_TABLE_START_ADDR_HI32) + (vmid*2), 0); + + WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_CONTEXT0_PAGE_TABLE_END_ADDR_LO32) + (vmid*2), + lower_32_bits(adev->vm_manager.max_pfn - 1)); + WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_CONTEXT0_PAGE_TABLE_END_ADDR_HI32) + (vmid*2), + upper_32_bits(adev->vm_manager.max_pfn - 1)); + + WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LO32) + (vmid*2), lower_32_bits(base)); + WREG32(SOC15_REG_OFFSET(GC, 0, mmVM_CONTEXT0_PAGE_TABLE_BASE_ADDR_HI32) + (vmid*2), upper_32_bits(base)); +} diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c index 1d6e1479da38..ff8fd75f7ca5 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c @@ -23,6 +23,8 @@ #define pr_fmt(fmt) "kfd2kgd: " fmt #include +#include +#include #include #include "amdgpu_object.h" #include "amdgpu_vm.h" @@ -33,10 +35,20 @@ */ #define VI_BO_SIZE_ALIGN (0x8000) +/* BO flag to indicate a KFD userptr BO */ +#define AMDGPU_AMDKFD_USERPTR_BO (1ULL << 63) + +/* Userptr restore delay, just long enough to allow consecutive VM + * changes to accumulate + */ +#define AMDGPU_USERPTR_RESTORE_DELAY_MS 1 + /* Impose limit on how much memory KFD can use */ static struct { uint64_t max_system_mem_limit; + uint64_t max_userptr_mem_limit; int64_t system_mem_used; + int64_t userptr_mem_used; spinlock_t mem_limit_lock; } kfd_mem_limit; @@ -57,6 +69,7 @@ static const char * const domain_bit_to_string[] = { #define domain_string(domain) domain_bit_to_string[ffs(domain)-1] +static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work); static inline struct amdgpu_device *get_amdgpu_device(struct kgd_dev *kgd) @@ -78,6 +91,7 @@ static bool check_if_add_bo_to_vm(struct amdgpu_vm *avm, /* Set memory usage limits. Current, limits are * System (kernel) memory - 3/8th System RAM + * Userptr memory - 3/4th System RAM */ void amdgpu_amdkfd_gpuvm_init_mem_limits(void) { @@ -90,8 +104,10 @@ void amdgpu_amdkfd_gpuvm_init_mem_limits(void) spin_lock_init(&kfd_mem_limit.mem_limit_lock); kfd_mem_limit.max_system_mem_limit = (mem >> 1) - (mem >> 3); - pr_debug("Kernel memory limit %lluM\n", - (kfd_mem_limit.max_system_mem_limit >> 20)); + kfd_mem_limit.max_userptr_mem_limit = mem - (mem >> 2); + pr_debug("Kernel memory limit %lluM, userptr limit %lluM\n", + (kfd_mem_limit.max_system_mem_limit >> 20), + (kfd_mem_limit.max_userptr_mem_limit >> 20)); } static int amdgpu_amdkfd_reserve_system_mem_limit(struct amdgpu_device *adev, @@ -111,6 +127,16 @@ static int amdgpu_amdkfd_reserve_system_mem_limit(struct amdgpu_device *adev, goto err_no_mem; } kfd_mem_limit.system_mem_used += (acc_size + size); + } else if (domain == AMDGPU_GEM_DOMAIN_CPU) { + if ((kfd_mem_limit.system_mem_used + acc_size > + kfd_mem_limit.max_system_mem_limit) || + (kfd_mem_limit.userptr_mem_used + (size + acc_size) > + kfd_mem_limit.max_userptr_mem_limit)) { + ret = -ENOMEM; + goto err_no_mem; + } + kfd_mem_limit.system_mem_used += acc_size; + kfd_mem_limit.userptr_mem_used += size; } err_no_mem: spin_unlock(&kfd_mem_limit.mem_limit_lock); @@ -126,10 +152,16 @@ static void unreserve_system_mem_limit(struct amdgpu_device *adev, sizeof(struct amdgpu_bo)); spin_lock(&kfd_mem_limit.mem_limit_lock); - if (domain == AMDGPU_GEM_DOMAIN_GTT) + if (domain == AMDGPU_GEM_DOMAIN_GTT) { kfd_mem_limit.system_mem_used -= (acc_size + size); + } else if (domain == AMDGPU_GEM_DOMAIN_CPU) { + kfd_mem_limit.system_mem_used -= acc_size; + kfd_mem_limit.userptr_mem_used -= size; + } WARN_ONCE(kfd_mem_limit.system_mem_used < 0, "kfd system memory accounting unbalanced"); + WARN_ONCE(kfd_mem_limit.userptr_mem_used < 0, + "kfd userptr memory accounting unbalanced"); spin_unlock(&kfd_mem_limit.mem_limit_lock); } @@ -138,12 +170,17 @@ void amdgpu_amdkfd_unreserve_system_memory_limit(struct amdgpu_bo *bo) { spin_lock(&kfd_mem_limit.mem_limit_lock); - if (bo->preferred_domains == AMDGPU_GEM_DOMAIN_GTT) { + if (bo->flags & AMDGPU_AMDKFD_USERPTR_BO) { + kfd_mem_limit.system_mem_used -= bo->tbo.acc_size; + kfd_mem_limit.userptr_mem_used -= amdgpu_bo_size(bo); + } else if (bo->preferred_domains == AMDGPU_GEM_DOMAIN_GTT) { kfd_mem_limit.system_mem_used -= (bo->tbo.acc_size + amdgpu_bo_size(bo)); } WARN_ONCE(kfd_mem_limit.system_mem_used < 0, "kfd system memory accounting unbalanced"); + WARN_ONCE(kfd_mem_limit.userptr_mem_used < 0, + "kfd userptr memory accounting unbalanced"); spin_unlock(&kfd_mem_limit.mem_limit_lock); } @@ -506,7 +543,8 @@ static void remove_bo_from_vm(struct amdgpu_device *adev, } static void add_kgd_mem_to_kfd_bo_list(struct kgd_mem *mem, - struct amdkfd_process_info *process_info) + struct amdkfd_process_info *process_info, + bool userptr) { struct ttm_validate_buffer *entry = &mem->validate_list; struct amdgpu_bo *bo = mem->bo; @@ -515,10 +553,95 @@ static void add_kgd_mem_to_kfd_bo_list(struct kgd_mem *mem, entry->shared = true; entry->bo = &bo->tbo; mutex_lock(&process_info->lock); - list_add_tail(&entry->head, &process_info->kfd_bo_list); + if (userptr) + list_add_tail(&entry->head, &process_info->userptr_valid_list); + else + list_add_tail(&entry->head, &process_info->kfd_bo_list); mutex_unlock(&process_info->lock); } +/* Initializes user pages. It registers the MMU notifier and validates + * the userptr BO in the GTT domain. + * + * The BO must already be on the userptr_valid_list. Otherwise an + * eviction and restore may happen that leaves the new BO unmapped + * with the user mode queues running. + * + * Takes the process_info->lock to protect against concurrent restore + * workers. + * + * Returns 0 for success, negative errno for errors. + */ +static int init_user_pages(struct kgd_mem *mem, struct mm_struct *mm, + uint64_t user_addr) +{ + struct amdkfd_process_info *process_info = mem->process_info; + struct amdgpu_bo *bo = mem->bo; + struct ttm_operation_ctx ctx = { true, false }; + int ret = 0; + + mutex_lock(&process_info->lock); + + ret = amdgpu_ttm_tt_set_userptr(bo->tbo.ttm, user_addr, 0); + if (ret) { + pr_err("%s: Failed to set userptr: %d\n", __func__, ret); + goto out; + } + + ret = amdgpu_mn_register(bo, user_addr); + if (ret) { + pr_err("%s: Failed to register MMU notifier: %d\n", + __func__, ret); + goto out; + } + + /* If no restore worker is running concurrently, user_pages + * should not be allocated + */ + WARN(mem->user_pages, "Leaking user_pages array"); + + mem->user_pages = kvmalloc_array(bo->tbo.ttm->num_pages, + sizeof(struct page *), + GFP_KERNEL | __GFP_ZERO); + if (!mem->user_pages) { + pr_err("%s: Failed to allocate pages array\n", __func__); + ret = -ENOMEM; + goto unregister_out; + } + + ret = amdgpu_ttm_tt_get_user_pages(bo->tbo.ttm, mem->user_pages); + if (ret) { + pr_err("%s: Failed to get user pages: %d\n", __func__, ret); + goto free_out; + } + + amdgpu_ttm_tt_set_user_pages(bo->tbo.ttm, mem->user_pages); + + ret = amdgpu_bo_reserve(bo, true); + if (ret) { + pr_err("%s: Failed to reserve BO\n", __func__); + goto release_out; + } + amdgpu_ttm_placement_from_domain(bo, mem->domain); + ret = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx); + if (ret) + pr_err("%s: failed to validate BO\n", __func__); + amdgpu_bo_unreserve(bo); + +release_out: + if (ret) + release_pages(mem->user_pages, bo->tbo.ttm->num_pages); +free_out: + kvfree(mem->user_pages); + mem->user_pages = NULL; +unregister_out: + if (ret) + amdgpu_mn_unregister(bo); +out: + mutex_unlock(&process_info->lock); + return ret; +} + /* Reserving a BO and its page table BOs must happen atomically to * avoid deadlocks. Some operations update multiple VMs at once. Track * all the reservation info in a context structure. Optionally a sync @@ -748,7 +871,8 @@ static int update_gpuvm_pte(struct amdgpu_device *adev, } static int map_bo_to_gpuvm(struct amdgpu_device *adev, - struct kfd_bo_va_list *entry, struct amdgpu_sync *sync) + struct kfd_bo_va_list *entry, struct amdgpu_sync *sync, + bool no_update_pte) { int ret; @@ -762,6 +886,9 @@ static int map_bo_to_gpuvm(struct amdgpu_device *adev, return ret; } + if (no_update_pte) + return 0; + ret = update_gpuvm_pte(adev, entry, sync); if (ret) { pr_err("update_gpuvm_pte() failed\n"); @@ -820,6 +947,8 @@ static int init_kfd_vm(struct amdgpu_vm *vm, void **process_info, mutex_init(&info->lock); INIT_LIST_HEAD(&info->vm_list_head); INIT_LIST_HEAD(&info->kfd_bo_list); + INIT_LIST_HEAD(&info->userptr_valid_list); + INIT_LIST_HEAD(&info->userptr_inval_list); info->eviction_fence = amdgpu_amdkfd_fence_create(dma_fence_context_alloc(1), @@ -830,6 +959,11 @@ static int init_kfd_vm(struct amdgpu_vm *vm, void **process_info, goto create_evict_fence_fail; } + info->pid = get_task_pid(current->group_leader, PIDTYPE_PID); + atomic_set(&info->evicted_bos, 0); + INIT_DELAYED_WORK(&info->restore_userptr_work, + amdgpu_amdkfd_restore_userptr_worker); + *process_info = info; *ef = dma_fence_get(&info->eviction_fence->base); } @@ -872,6 +1006,7 @@ reserve_pd_fail: dma_fence_put(*ef); *ef = NULL; *process_info = NULL; + put_pid(info->pid); create_evict_fence_fail: mutex_destroy(&info->lock); kfree(info); @@ -967,8 +1102,12 @@ void amdgpu_amdkfd_gpuvm_destroy_cb(struct amdgpu_device *adev, /* Release per-process resources when last compute VM is destroyed */ if (!process_info->n_vms) { WARN_ON(!list_empty(&process_info->kfd_bo_list)); + WARN_ON(!list_empty(&process_info->userptr_valid_list)); + WARN_ON(!list_empty(&process_info->userptr_inval_list)); dma_fence_put(&process_info->eviction_fence->base); + cancel_delayed_work_sync(&process_info->restore_userptr_work); + put_pid(process_info->pid); mutex_destroy(&process_info->lock); kfree(process_info); } @@ -1003,9 +1142,11 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( { struct amdgpu_device *adev = get_amdgpu_device(kgd); struct amdgpu_vm *avm = (struct amdgpu_vm *)vm; + uint64_t user_addr = 0; struct amdgpu_bo *bo; + struct amdgpu_bo_param bp; int byte_align; - u32 alloc_domain; + u32 domain, alloc_domain; u64 alloc_flags; uint32_t mapping_flags; int ret; @@ -1014,14 +1155,21 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( * Check on which domain to allocate BO */ if (flags & ALLOC_MEM_FLAGS_VRAM) { - alloc_domain = AMDGPU_GEM_DOMAIN_VRAM; + domain = alloc_domain = AMDGPU_GEM_DOMAIN_VRAM; alloc_flags = AMDGPU_GEM_CREATE_VRAM_CLEARED; alloc_flags |= (flags & ALLOC_MEM_FLAGS_PUBLIC) ? AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED : AMDGPU_GEM_CREATE_NO_CPU_ACCESS; } else if (flags & ALLOC_MEM_FLAGS_GTT) { - alloc_domain = AMDGPU_GEM_DOMAIN_GTT; + domain = alloc_domain = AMDGPU_GEM_DOMAIN_GTT; + alloc_flags = 0; + } else if (flags & ALLOC_MEM_FLAGS_USERPTR) { + domain = AMDGPU_GEM_DOMAIN_GTT; + alloc_domain = AMDGPU_GEM_DOMAIN_CPU; alloc_flags = 0; + if (!offset || !*offset) + return -EINVAL; + user_addr = *offset; } else { return -EINVAL; } @@ -1069,8 +1217,14 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( pr_debug("\tcreate BO VA 0x%llx size 0x%llx domain %s\n", va, size, domain_string(alloc_domain)); - ret = amdgpu_bo_create(adev, size, byte_align, - alloc_domain, alloc_flags, ttm_bo_type_device, NULL, &bo); + memset(&bp, 0, sizeof(bp)); + bp.size = size; + bp.byte_align = byte_align; + bp.domain = alloc_domain; + bp.flags = alloc_flags; + bp.type = ttm_bo_type_device; + bp.resv = NULL; + ret = amdgpu_bo_create(adev, &bp, &bo); if (ret) { pr_debug("Failed to create BO on domain %s. ret %d\n", domain_string(alloc_domain), ret); @@ -1078,18 +1232,34 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( } bo->kfd_bo = *mem; (*mem)->bo = bo; + if (user_addr) + bo->flags |= AMDGPU_AMDKFD_USERPTR_BO; (*mem)->va = va; - (*mem)->domain = alloc_domain; + (*mem)->domain = domain; (*mem)->mapped_to_gpu_memory = 0; (*mem)->process_info = avm->process_info; - add_kgd_mem_to_kfd_bo_list(*mem, avm->process_info); + add_kgd_mem_to_kfd_bo_list(*mem, avm->process_info, user_addr); + + if (user_addr) { + ret = init_user_pages(*mem, current->mm, user_addr); + if (ret) { + mutex_lock(&avm->process_info->lock); + list_del(&(*mem)->validate_list.head); + mutex_unlock(&avm->process_info->lock); + goto allocate_init_user_pages_failed; + } + } if (offset) *offset = amdgpu_bo_mmap_offset(bo); return 0; +allocate_init_user_pages_failed: + amdgpu_bo_unref(&bo); + /* Don't unreserve system mem limit twice */ + goto err_reserve_system_mem; err_bo_create: unreserve_system_mem_limit(adev, size, alloc_domain); err_reserve_system_mem: @@ -1122,12 +1292,24 @@ int amdgpu_amdkfd_gpuvm_free_memory_of_gpu( * be freed anyway */ + /* No more MMU notifiers */ + amdgpu_mn_unregister(mem->bo); + /* Make sure restore workers don't access the BO any more */ bo_list_entry = &mem->validate_list; mutex_lock(&process_info->lock); list_del(&bo_list_entry->head); mutex_unlock(&process_info->lock); + /* Free user pages if necessary */ + if (mem->user_pages) { + pr_debug("%s: Freeing user_pages array\n", __func__); + if (mem->user_pages[0]) + release_pages(mem->user_pages, + mem->bo->tbo.ttm->num_pages); + kvfree(mem->user_pages); + } + ret = reserve_bo_and_cond_vms(mem, NULL, BO_VM_ALL, &ctx); if (unlikely(ret)) return ret; @@ -1173,21 +1355,32 @@ int amdgpu_amdkfd_gpuvm_map_memory_to_gpu( struct kfd_bo_va_list *bo_va_entry = NULL; struct kfd_bo_va_list *bo_va_entry_aql = NULL; unsigned long bo_size; - - /* Make sure restore is not running concurrently. - */ - mutex_lock(&mem->process_info->lock); - - mutex_lock(&mem->lock); + bool is_invalid_userptr = false; bo = mem->bo; - if (!bo) { pr_err("Invalid BO when mapping memory to GPU\n"); - ret = -EINVAL; - goto out; + return -EINVAL; } + /* Make sure restore is not running concurrently. Since we + * don't map invalid userptr BOs, we rely on the next restore + * worker to do the mapping + */ + mutex_lock(&mem->process_info->lock); + + /* Lock mmap-sem. If we find an invalid userptr BO, we can be + * sure that the MMU notifier is no longer running + * concurrently and the queues are actually stopped + */ + if (amdgpu_ttm_tt_get_usermm(bo->tbo.ttm)) { + down_write(¤t->mm->mmap_sem); + is_invalid_userptr = atomic_read(&mem->invalid); + up_write(¤t->mm->mmap_sem); + } + + mutex_lock(&mem->lock); + domain = mem->domain; bo_size = bo->tbo.mem.size; @@ -1200,6 +1393,14 @@ int amdgpu_amdkfd_gpuvm_map_memory_to_gpu( if (unlikely(ret)) goto out; + /* Userptr can be marked as "not invalid", but not actually be + * validated yet (still in the system domain). In that case + * the queues are still stopped and we can leave mapping for + * the next restore worker + */ + if (bo->tbo.mem.mem_type == TTM_PL_SYSTEM) + is_invalid_userptr = true; + if (check_if_add_bo_to_vm(avm, mem)) { ret = add_bo_to_vm(adev, mem, avm, false, &bo_va_entry); @@ -1217,7 +1418,8 @@ int amdgpu_amdkfd_gpuvm_map_memory_to_gpu( goto add_bo_to_vm_failed; } - if (mem->mapped_to_gpu_memory == 0) { + if (mem->mapped_to_gpu_memory == 0 && + !amdgpu_ttm_tt_get_usermm(bo->tbo.ttm)) { /* Validate BO only once. The eviction fence gets added to BO * the first time it is mapped. Validate will wait for all * background evictions to complete. @@ -1235,7 +1437,8 @@ int amdgpu_amdkfd_gpuvm_map_memory_to_gpu( entry->va, entry->va + bo_size, entry); - ret = map_bo_to_gpuvm(adev, entry, ctx.sync); + ret = map_bo_to_gpuvm(adev, entry, ctx.sync, + is_invalid_userptr); if (ret) { pr_err("Failed to map radeon bo to gpuvm\n"); goto map_bo_to_gpuvm_failed; @@ -1418,6 +1621,337 @@ bo_reserve_failed: return ret; } +/* Evict a userptr BO by stopping the queues if necessary + * + * Runs in MMU notifier, may be in RECLAIM_FS context. This means it + * cannot do any memory allocations, and cannot take any locks that + * are held elsewhere while allocating memory. Therefore this is as + * simple as possible, using atomic counters. + * + * It doesn't do anything to the BO itself. The real work happens in + * restore, where we get updated page addresses. This function only + * ensures that GPU access to the BO is stopped. + */ +int amdgpu_amdkfd_evict_userptr(struct kgd_mem *mem, + struct mm_struct *mm) +{ + struct amdkfd_process_info *process_info = mem->process_info; + int invalid, evicted_bos; + int r = 0; + + invalid = atomic_inc_return(&mem->invalid); + evicted_bos = atomic_inc_return(&process_info->evicted_bos); + if (evicted_bos == 1) { + /* First eviction, stop the queues */ + r = kgd2kfd->quiesce_mm(mm); + if (r) + pr_err("Failed to quiesce KFD\n"); + schedule_delayed_work(&process_info->restore_userptr_work, + msecs_to_jiffies(AMDGPU_USERPTR_RESTORE_DELAY_MS)); + } + + return r; +} + +/* Update invalid userptr BOs + * + * Moves invalidated (evicted) userptr BOs from userptr_valid_list to + * userptr_inval_list and updates user pages for all BOs that have + * been invalidated since their last update. + */ +static int update_invalid_user_pages(struct amdkfd_process_info *process_info, + struct mm_struct *mm) +{ + struct kgd_mem *mem, *tmp_mem; + struct amdgpu_bo *bo; + struct ttm_operation_ctx ctx = { false, false }; + int invalid, ret; + + /* Move all invalidated BOs to the userptr_inval_list and + * release their user pages by migration to the CPU domain + */ + list_for_each_entry_safe(mem, tmp_mem, + &process_info->userptr_valid_list, + validate_list.head) { + if (!atomic_read(&mem->invalid)) + continue; /* BO is still valid */ + + bo = mem->bo; + + if (amdgpu_bo_reserve(bo, true)) + return -EAGAIN; + amdgpu_ttm_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_CPU); + ret = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx); + amdgpu_bo_unreserve(bo); + if (ret) { + pr_err("%s: Failed to invalidate userptr BO\n", + __func__); + return -EAGAIN; + } + + list_move_tail(&mem->validate_list.head, + &process_info->userptr_inval_list); + } + + if (list_empty(&process_info->userptr_inval_list)) + return 0; /* All evicted userptr BOs were freed */ + + /* Go through userptr_inval_list and update any invalid user_pages */ + list_for_each_entry(mem, &process_info->userptr_inval_list, + validate_list.head) { + invalid = atomic_read(&mem->invalid); + if (!invalid) + /* BO hasn't been invalidated since the last + * revalidation attempt. Keep its BO list. + */ + continue; + + bo = mem->bo; + + if (!mem->user_pages) { + mem->user_pages = + kvmalloc_array(bo->tbo.ttm->num_pages, + sizeof(struct page *), + GFP_KERNEL | __GFP_ZERO); + if (!mem->user_pages) { + pr_err("%s: Failed to allocate pages array\n", + __func__); + return -ENOMEM; + } + } else if (mem->user_pages[0]) { + release_pages(mem->user_pages, bo->tbo.ttm->num_pages); + } + + /* Get updated user pages */ + ret = amdgpu_ttm_tt_get_user_pages(bo->tbo.ttm, + mem->user_pages); + if (ret) { + mem->user_pages[0] = NULL; + pr_info("%s: Failed to get user pages: %d\n", + __func__, ret); + /* Pretend it succeeded. It will fail later + * with a VM fault if the GPU tries to access + * it. Better than hanging indefinitely with + * stalled user mode queues. + */ + } + + /* Mark the BO as valid unless it was invalidated + * again concurrently + */ + if (atomic_cmpxchg(&mem->invalid, invalid, 0) != invalid) + return -EAGAIN; + } + + return 0; +} + +/* Validate invalid userptr BOs + * + * Validates BOs on the userptr_inval_list, and moves them back to the + * userptr_valid_list. Also updates GPUVM page tables with new page + * addresses and waits for the page table updates to complete. + */ +static int validate_invalid_user_pages(struct amdkfd_process_info *process_info) +{ + struct amdgpu_bo_list_entry *pd_bo_list_entries; + struct list_head resv_list, duplicates; + struct ww_acquire_ctx ticket; + struct amdgpu_sync sync; + + struct amdgpu_vm *peer_vm; + struct kgd_mem *mem, *tmp_mem; + struct amdgpu_bo *bo; + struct ttm_operation_ctx ctx = { false, false }; + int i, ret; + + pd_bo_list_entries = kcalloc(process_info->n_vms, + sizeof(struct amdgpu_bo_list_entry), + GFP_KERNEL); + if (!pd_bo_list_entries) { + pr_err("%s: Failed to allocate PD BO list entries\n", __func__); + return -ENOMEM; + } + + INIT_LIST_HEAD(&resv_list); + INIT_LIST_HEAD(&duplicates); + + /* Get all the page directory BOs that need to be reserved */ + i = 0; + list_for_each_entry(peer_vm, &process_info->vm_list_head, + vm_list_node) + amdgpu_vm_get_pd_bo(peer_vm, &resv_list, + &pd_bo_list_entries[i++]); + /* Add the userptr_inval_list entries to resv_list */ + list_for_each_entry(mem, &process_info->userptr_inval_list, + validate_list.head) { + list_add_tail(&mem->resv_list.head, &resv_list); + mem->resv_list.bo = mem->validate_list.bo; + mem->resv_list.shared = mem->validate_list.shared; + } + + /* Reserve all BOs and page tables for validation */ + ret = ttm_eu_reserve_buffers(&ticket, &resv_list, false, &duplicates); + WARN(!list_empty(&duplicates), "Duplicates should be empty"); + if (ret) + goto out; + + amdgpu_sync_create(&sync); + + /* Avoid triggering eviction fences when unmapping invalid + * userptr BOs (waits for all fences, doesn't use + * FENCE_OWNER_VM) + */ + list_for_each_entry(peer_vm, &process_info->vm_list_head, + vm_list_node) + amdgpu_amdkfd_remove_eviction_fence(peer_vm->root.base.bo, + process_info->eviction_fence, + NULL, NULL); + + ret = process_validate_vms(process_info); + if (ret) + goto unreserve_out; + + /* Validate BOs and update GPUVM page tables */ + list_for_each_entry_safe(mem, tmp_mem, + &process_info->userptr_inval_list, + validate_list.head) { + struct kfd_bo_va_list *bo_va_entry; + + bo = mem->bo; + + /* Copy pages array and validate the BO if we got user pages */ + if (mem->user_pages[0]) { + amdgpu_ttm_tt_set_user_pages(bo->tbo.ttm, + mem->user_pages); + amdgpu_ttm_placement_from_domain(bo, mem->domain); + ret = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx); + if (ret) { + pr_err("%s: failed to validate BO\n", __func__); + goto unreserve_out; + } + } + + /* Validate succeeded, now the BO owns the pages, free + * our copy of the pointer array. Put this BO back on + * the userptr_valid_list. If we need to revalidate + * it, we need to start from scratch. + */ + kvfree(mem->user_pages); + mem->user_pages = NULL; + list_move_tail(&mem->validate_list.head, + &process_info->userptr_valid_list); + + /* Update mapping. If the BO was not validated + * (because we couldn't get user pages), this will + * clear the page table entries, which will result in + * VM faults if the GPU tries to access the invalid + * memory. + */ + list_for_each_entry(bo_va_entry, &mem->bo_va_list, bo_list) { + if (!bo_va_entry->is_mapped) + continue; + + ret = update_gpuvm_pte((struct amdgpu_device *) + bo_va_entry->kgd_dev, + bo_va_entry, &sync); + if (ret) { + pr_err("%s: update PTE failed\n", __func__); + /* make sure this gets validated again */ + atomic_inc(&mem->invalid); + goto unreserve_out; + } + } + } + + /* Update page directories */ + ret = process_update_pds(process_info, &sync); + +unreserve_out: + list_for_each_entry(peer_vm, &process_info->vm_list_head, + vm_list_node) + amdgpu_bo_fence(peer_vm->root.base.bo, + &process_info->eviction_fence->base, true); + ttm_eu_backoff_reservation(&ticket, &resv_list); + amdgpu_sync_wait(&sync, false); + amdgpu_sync_free(&sync); +out: + kfree(pd_bo_list_entries); + + return ret; +} + +/* Worker callback to restore evicted userptr BOs + * + * Tries to update and validate all userptr BOs. If successful and no + * concurrent evictions happened, the queues are restarted. Otherwise, + * reschedule for another attempt later. + */ +static void amdgpu_amdkfd_restore_userptr_worker(struct work_struct *work) +{ + struct delayed_work *dwork = to_delayed_work(work); + struct amdkfd_process_info *process_info = + container_of(dwork, struct amdkfd_process_info, + restore_userptr_work); + struct task_struct *usertask; + struct mm_struct *mm; + int evicted_bos; + + evicted_bos = atomic_read(&process_info->evicted_bos); + if (!evicted_bos) + return; + + /* Reference task and mm in case of concurrent process termination */ + usertask = get_pid_task(process_info->pid, PIDTYPE_PID); + if (!usertask) + return; + mm = get_task_mm(usertask); + if (!mm) { + put_task_struct(usertask); + return; + } + + mutex_lock(&process_info->lock); + + if (update_invalid_user_pages(process_info, mm)) + goto unlock_out; + /* userptr_inval_list can be empty if all evicted userptr BOs + * have been freed. In that case there is nothing to validate + * and we can just restart the queues. + */ + if (!list_empty(&process_info->userptr_inval_list)) { + if (atomic_read(&process_info->evicted_bos) != evicted_bos) + goto unlock_out; /* Concurrent eviction, try again */ + + if (validate_invalid_user_pages(process_info)) + goto unlock_out; + } + /* Final check for concurrent evicton and atomic update. If + * another eviction happens after successful update, it will + * be a first eviction that calls quiesce_mm. The eviction + * reference counting inside KFD will handle this case. + */ + if (atomic_cmpxchg(&process_info->evicted_bos, evicted_bos, 0) != + evicted_bos) + goto unlock_out; + evicted_bos = 0; + if (kgd2kfd->resume_mm(mm)) { + pr_err("%s: Failed to resume KFD\n", __func__); + /* No recovery from this failure. Probably the CP is + * hanging. No point trying again. + */ + } +unlock_out: + mutex_unlock(&process_info->lock); + mmput(mm); + put_task_struct(usertask); + + /* If validation failed, reschedule another attempt */ + if (evicted_bos) + schedule_delayed_work(&process_info->restore_userptr_work, + msecs_to_jiffies(AMDGPU_USERPTR_RESTORE_DELAY_MS)); +} + /** amdgpu_amdkfd_gpuvm_restore_process_bos - Restore all BOs for the given * KFD process identified by process_info * diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.c index a0f48cb9b8f0..236915849cfe 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.c @@ -322,3 +322,47 @@ int amdgpu_atomfirmware_get_clock_info(struct amdgpu_device *adev) return ret; } + +union gfx_info { + struct atom_gfx_info_v2_4 v24; +}; + +int amdgpu_atomfirmware_get_gfx_info(struct amdgpu_device *adev) +{ + struct amdgpu_mode_info *mode_info = &adev->mode_info; + int index; + uint8_t frev, crev; + uint16_t data_offset; + + index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1, + gfx_info); + if (amdgpu_atom_parse_data_header(mode_info->atom_context, index, NULL, + &frev, &crev, &data_offset)) { + union gfx_info *gfx_info = (union gfx_info *) + (mode_info->atom_context->bios + data_offset); + switch (crev) { + case 4: + adev->gfx.config.max_shader_engines = gfx_info->v24.gc_num_se; + adev->gfx.config.max_cu_per_sh = gfx_info->v24.gc_num_cu_per_sh; + adev->gfx.config.max_sh_per_se = gfx_info->v24.gc_num_sh_per_se; + adev->gfx.config.max_backends_per_se = gfx_info->v24.gc_num_rb_per_se; + adev->gfx.config.max_texture_channel_caches = gfx_info->v24.gc_num_tccs; + adev->gfx.config.max_gprs = le16_to_cpu(gfx_info->v24.gc_num_gprs); + adev->gfx.config.max_gs_threads = gfx_info->v24.gc_num_max_gs_thds; + adev->gfx.config.gs_vgt_table_depth = gfx_info->v24.gc_gs_table_depth; + adev->gfx.config.gs_prim_buffer_depth = + le16_to_cpu(gfx_info->v24.gc_gsprim_buff_depth); + adev->gfx.config.double_offchip_lds_buf = + gfx_info->v24.gc_double_offchip_lds_buffer; + adev->gfx.cu_info.wave_front_size = le16_to_cpu(gfx_info->v24.gc_wave_size); + adev->gfx.cu_info.max_waves_per_simd = le16_to_cpu(gfx_info->v24.gc_max_waves_per_simd); + adev->gfx.cu_info.max_scratch_slots_per_cu = gfx_info->v24.gc_max_scratch_slots_per_cu; + adev->gfx.cu_info.lds_size = le16_to_cpu(gfx_info->v24.gc_lds_size); + return 0; + default: + return -EINVAL; + } + + } + return -EINVAL; +} diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.h index 7689c961c4ef..20f158fd3b76 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_atomfirmware.h @@ -30,5 +30,6 @@ int amdgpu_atomfirmware_allocate_fb_scratch(struct amdgpu_device *adev); int amdgpu_atomfirmware_get_vram_width(struct amdgpu_device *adev); int amdgpu_atomfirmware_get_vram_type(struct amdgpu_device *adev); int amdgpu_atomfirmware_get_clock_info(struct amdgpu_device *adev); +int amdgpu_atomfirmware_get_gfx_info(struct amdgpu_device *adev); #endif diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_atpx_handler.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_atpx_handler.c index 1ae5ae8c45a4..daa06e7c5bb7 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_atpx_handler.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_atpx_handler.c @@ -550,7 +550,7 @@ static int amdgpu_atpx_init(void) * look up whether we are the integrated or discrete GPU (all asics). * Returns the client id. */ -static int amdgpu_atpx_get_client_id(struct pci_dev *pdev) +static enum vga_switcheroo_client_id amdgpu_atpx_get_client_id(struct pci_dev *pdev) { if (amdgpu_atpx_priv.dhandle == ACPI_HANDLE(&pdev->dev)) return VGA_SWITCHEROO_IGD; @@ -569,7 +569,6 @@ static const struct amdgpu_px_quirk amdgpu_px_quirk_list[] = { { 0x1002, 0x6900, 0x1002, 0x0124, AMDGPU_PX_QUIRK_FORCE_ATPX }, { 0x1002, 0x6900, 0x1028, 0x0812, AMDGPU_PX_QUIRK_FORCE_ATPX }, { 0x1002, 0x6900, 0x1028, 0x0813, AMDGPU_PX_QUIRK_FORCE_ATPX }, - { 0x1002, 0x67DF, 0x1028, 0x0774, AMDGPU_PX_QUIRK_FORCE_ATPX }, { 0, 0, 0, 0, 0 }, }; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.c index 02b849be083b..19cfff31f2e1 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.c @@ -75,13 +75,20 @@ static void amdgpu_benchmark_move(struct amdgpu_device *adev, unsigned size, { struct amdgpu_bo *dobj = NULL; struct amdgpu_bo *sobj = NULL; + struct amdgpu_bo_param bp; uint64_t saddr, daddr; int r, n; int time; + memset(&bp, 0, sizeof(bp)); + bp.size = size; + bp.byte_align = PAGE_SIZE; + bp.domain = sdomain; + bp.flags = 0; + bp.type = ttm_bo_type_kernel; + bp.resv = NULL; n = AMDGPU_BENCHMARK_ITERATIONS; - r = amdgpu_bo_create(adev, size, PAGE_SIZE,sdomain, 0, - ttm_bo_type_kernel, NULL, &sobj); + r = amdgpu_bo_create(adev, &bp, &sobj); if (r) { goto out_cleanup; } @@ -93,8 +100,8 @@ static void amdgpu_benchmark_move(struct amdgpu_device *adev, unsigned size, if (r) { goto out_cleanup; } - r = amdgpu_bo_create(adev, size, PAGE_SIZE, ddomain, 0, - ttm_bo_type_kernel, NULL, &dobj); + bp.domain = ddomain; + r = amdgpu_bo_create(adev, &bp, &dobj); if (r) { goto out_cleanup; } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cgs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cgs.c index 71a57b2f7f04..e950730f1933 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cgs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cgs.c @@ -23,7 +23,6 @@ */ #include #include -#include #include #include #include @@ -109,121 +108,6 @@ static void amdgpu_cgs_write_ind_register(struct cgs_device *cgs_device, WARN(1, "Invalid indirect register space"); } -static int amdgpu_cgs_get_pci_resource(struct cgs_device *cgs_device, - enum cgs_resource_type resource_type, - uint64_t size, - uint64_t offset, - uint64_t *resource_base) -{ - CGS_FUNC_ADEV; - - if (resource_base == NULL) - return -EINVAL; - - switch (resource_type) { - case CGS_RESOURCE_TYPE_MMIO: - if (adev->rmmio_size == 0) - return -ENOENT; - if ((offset + size) > adev->rmmio_size) - return -EINVAL; - *resource_base = adev->rmmio_base; - return 0; - case CGS_RESOURCE_TYPE_DOORBELL: - if (adev->doorbell.size == 0) - return -ENOENT; - if ((offset + size) > adev->doorbell.size) - return -EINVAL; - *resource_base = adev->doorbell.base; - return 0; - case CGS_RESOURCE_TYPE_FB: - case CGS_RESOURCE_TYPE_IO: - case CGS_RESOURCE_TYPE_ROM: - default: - return -EINVAL; - } -} - -static const void *amdgpu_cgs_atom_get_data_table(struct cgs_device *cgs_device, - unsigned table, uint16_t *size, - uint8_t *frev, uint8_t *crev) -{ - CGS_FUNC_ADEV; - uint16_t data_start; - - if (amdgpu_atom_parse_data_header( - adev->mode_info.atom_context, table, size, - frev, crev, &data_start)) - return (uint8_t*)adev->mode_info.atom_context->bios + - data_start; - - return NULL; -} - -static int amdgpu_cgs_atom_get_cmd_table_revs(struct cgs_device *cgs_device, unsigned table, - uint8_t *frev, uint8_t *crev) -{ - CGS_FUNC_ADEV; - - if (amdgpu_atom_parse_cmd_header( - adev->mode_info.atom_context, table, - frev, crev)) - return 0; - - return -EINVAL; -} - -static int amdgpu_cgs_atom_exec_cmd_table(struct cgs_device *cgs_device, unsigned table, - void *args) -{ - CGS_FUNC_ADEV; - - return amdgpu_atom_execute_table( - adev->mode_info.atom_context, table, args); -} - -static int amdgpu_cgs_set_clockgating_state(struct cgs_device *cgs_device, - enum amd_ip_block_type block_type, - enum amd_clockgating_state state) -{ - CGS_FUNC_ADEV; - int i, r = -1; - - for (i = 0; i < adev->num_ip_blocks; i++) { - if (!adev->ip_blocks[i].status.valid) - continue; - - if (adev->ip_blocks[i].version->type == block_type) { - r = adev->ip_blocks[i].version->funcs->set_clockgating_state( - (void *)adev, - state); - break; - } - } - return r; -} - -static int amdgpu_cgs_set_powergating_state(struct cgs_device *cgs_device, - enum amd_ip_block_type block_type, - enum amd_powergating_state state) -{ - CGS_FUNC_ADEV; - int i, r = -1; - - for (i = 0; i < adev->num_ip_blocks; i++) { - if (!adev->ip_blocks[i].status.valid) - continue; - - if (adev->ip_blocks[i].version->type == block_type) { - r = adev->ip_blocks[i].version->funcs->set_powergating_state( - (void *)adev, - state); - break; - } - } - return r; -} - - static uint32_t fw_type_convert(struct cgs_device *cgs_device, uint32_t fw_type) { CGS_FUNC_ADEV; @@ -271,18 +155,6 @@ static uint32_t fw_type_convert(struct cgs_device *cgs_device, uint32_t fw_type) return result; } -static int amdgpu_cgs_rel_firmware(struct cgs_device *cgs_device, enum cgs_ucode_id type) -{ - CGS_FUNC_ADEV; - if ((CGS_UCODE_ID_SMU == type) || (CGS_UCODE_ID_SMU_SK == type)) { - release_firmware(adev->pm.fw); - adev->pm.fw = NULL; - return 0; - } - /* cannot release other firmware because they are not created by cgs */ - return -EINVAL; -} - static uint16_t amdgpu_get_firmware_version(struct cgs_device *cgs_device, enum cgs_ucode_id type) { @@ -326,34 +198,6 @@ static uint16_t amdgpu_get_firmware_version(struct cgs_device *cgs_device, return fw_version; } -static int amdgpu_cgs_enter_safe_mode(struct cgs_device *cgs_device, - bool en) -{ - CGS_FUNC_ADEV; - - if (adev->gfx.rlc.funcs->enter_safe_mode == NULL || - adev->gfx.rlc.funcs->exit_safe_mode == NULL) - return 0; - - if (en) - adev->gfx.rlc.funcs->enter_safe_mode(adev); - else - adev->gfx.rlc.funcs->exit_safe_mode(adev); - - return 0; -} - -static void amdgpu_cgs_lock_grbm_idx(struct cgs_device *cgs_device, - bool lock) -{ - CGS_FUNC_ADEV; - - if (lock) - mutex_lock(&adev->grbm_idx_mutex); - else - mutex_unlock(&adev->grbm_idx_mutex); -} - static int amdgpu_cgs_get_firmware_info(struct cgs_device *cgs_device, enum cgs_ucode_id type, struct cgs_firmware_info *info) @@ -541,6 +385,9 @@ static int amdgpu_cgs_get_firmware_info(struct cgs_device *cgs_device, case CHIP_POLARIS12: strcpy(fw_name, "amdgpu/polaris12_smc.bin"); break; + case CHIP_VEGAM: + strcpy(fw_name, "amdgpu/vegam_smc.bin"); + break; case CHIP_VEGA10: if ((adev->pdev->device == 0x687f) && ((adev->pdev->revision == 0xc0) || @@ -553,6 +400,9 @@ static int amdgpu_cgs_get_firmware_info(struct cgs_device *cgs_device, case CHIP_VEGA12: strcpy(fw_name, "amdgpu/vega12_smc.bin"); break; + case CHIP_VEGA20: + strcpy(fw_name, "amdgpu/vega20_smc.bin"); + break; default: DRM_ERROR("SMC firmware not supported\n"); return -EINVAL; @@ -598,97 +448,12 @@ static int amdgpu_cgs_get_firmware_info(struct cgs_device *cgs_device, return 0; } -static int amdgpu_cgs_is_virtualization_enabled(void *cgs_device) -{ - CGS_FUNC_ADEV; - return amdgpu_sriov_vf(adev); -} - -static int amdgpu_cgs_get_active_displays_info(struct cgs_device *cgs_device, - struct cgs_display_info *info) -{ - CGS_FUNC_ADEV; - struct cgs_mode_info *mode_info; - - if (info == NULL) - return -EINVAL; - - mode_info = info->mode_info; - if (mode_info) - /* if the displays are off, vblank time is max */ - mode_info->vblank_time_us = 0xffffffff; - - if (!amdgpu_device_has_dc_support(adev)) { - struct amdgpu_crtc *amdgpu_crtc; - struct drm_device *ddev = adev->ddev; - struct drm_crtc *crtc; - uint32_t line_time_us, vblank_lines; - - if (adev->mode_info.num_crtc && adev->mode_info.mode_config_initialized) { - list_for_each_entry(crtc, - &ddev->mode_config.crtc_list, head) { - amdgpu_crtc = to_amdgpu_crtc(crtc); - if (crtc->enabled) { - info->active_display_mask |= (1 << amdgpu_crtc->crtc_id); - info->display_count++; - } - if (mode_info != NULL && - crtc->enabled && amdgpu_crtc->enabled && - amdgpu_crtc->hw_mode.clock) { - line_time_us = (amdgpu_crtc->hw_mode.crtc_htotal * 1000) / - amdgpu_crtc->hw_mode.clock; - vblank_lines = amdgpu_crtc->hw_mode.crtc_vblank_end - - amdgpu_crtc->hw_mode.crtc_vdisplay + - (amdgpu_crtc->v_border * 2); - mode_info->vblank_time_us = vblank_lines * line_time_us; - mode_info->refresh_rate = drm_mode_vrefresh(&amdgpu_crtc->hw_mode); - /* we have issues with mclk switching with refresh rates - * over 120 hz on the non-DC code. - */ - if (mode_info->refresh_rate > 120) - mode_info->vblank_time_us = 0; - mode_info = NULL; - } - } - } - } else { - info->display_count = adev->pm.pm_display_cfg.num_display; - if (mode_info != NULL) { - mode_info->vblank_time_us = adev->pm.pm_display_cfg.min_vblank_time; - mode_info->refresh_rate = adev->pm.pm_display_cfg.vrefresh; - } - } - return 0; -} - - -static int amdgpu_cgs_notify_dpm_enabled(struct cgs_device *cgs_device, bool enabled) -{ - CGS_FUNC_ADEV; - - adev->pm.dpm_enabled = enabled; - - return 0; -} - static const struct cgs_ops amdgpu_cgs_ops = { .read_register = amdgpu_cgs_read_register, .write_register = amdgpu_cgs_write_register, .read_ind_register = amdgpu_cgs_read_ind_register, .write_ind_register = amdgpu_cgs_write_ind_register, - .get_pci_resource = amdgpu_cgs_get_pci_resource, - .atom_get_data_table = amdgpu_cgs_atom_get_data_table, - .atom_get_cmd_table_revs = amdgpu_cgs_atom_get_cmd_table_revs, - .atom_exec_cmd_table = amdgpu_cgs_atom_exec_cmd_table, .get_firmware_info = amdgpu_cgs_get_firmware_info, - .rel_firmware = amdgpu_cgs_rel_firmware, - .set_powergating_state = amdgpu_cgs_set_powergating_state, - .set_clockgating_state = amdgpu_cgs_set_clockgating_state, - .get_active_displays_info = amdgpu_cgs_get_active_displays_info, - .notify_dpm_enabled = amdgpu_cgs_notify_dpm_enabled, - .is_virtualization_enabled = amdgpu_cgs_is_virtualization_enabled, - .enter_safe_mode = amdgpu_cgs_enter_safe_mode, - .lock_grbm_idx = amdgpu_cgs_lock_grbm_idx, }; struct cgs_device *amdgpu_cgs_create_device(struct amdgpu_device *adev) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_connectors.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_connectors.c index 96501ff0e55b..8e66851eb427 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_connectors.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_connectors.c @@ -691,7 +691,7 @@ static int amdgpu_connector_lvds_get_modes(struct drm_connector *connector) return ret; } -static int amdgpu_connector_lvds_mode_valid(struct drm_connector *connector, +static enum drm_mode_status amdgpu_connector_lvds_mode_valid(struct drm_connector *connector, struct drm_display_mode *mode) { struct drm_encoder *encoder = amdgpu_connector_best_single_encoder(connector); @@ -843,7 +843,7 @@ static int amdgpu_connector_vga_get_modes(struct drm_connector *connector) return ret; } -static int amdgpu_connector_vga_mode_valid(struct drm_connector *connector, +static enum drm_mode_status amdgpu_connector_vga_mode_valid(struct drm_connector *connector, struct drm_display_mode *mode) { struct drm_device *dev = connector->dev; @@ -1172,7 +1172,7 @@ static void amdgpu_connector_dvi_force(struct drm_connector *connector) amdgpu_connector->use_digital = true; } -static int amdgpu_connector_dvi_mode_valid(struct drm_connector *connector, +static enum drm_mode_status amdgpu_connector_dvi_mode_valid(struct drm_connector *connector, struct drm_display_mode *mode) { struct drm_device *dev = connector->dev; @@ -1448,7 +1448,7 @@ out: return ret; } -static int amdgpu_connector_dp_mode_valid(struct drm_connector *connector, +static enum drm_mode_status amdgpu_connector_dp_mode_valid(struct drm_connector *connector, struct drm_display_mode *mode) { struct amdgpu_connector *amdgpu_connector = to_amdgpu_connector(connector); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c index dc34b50e6b29..82312a7bc6ad 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c @@ -382,8 +382,7 @@ retry: p->bytes_moved += ctx.bytes_moved; if (adev->gmc.visible_vram_size < adev->gmc.real_vram_size && - bo->tbo.mem.mem_type == TTM_PL_VRAM && - bo->tbo.mem.start < adev->gmc.visible_vram_size >> PAGE_SHIFT) + amdgpu_bo_in_cpu_visible_vram(bo)) p->bytes_moved_vis += ctx.bytes_moved; if (unlikely(r == -ENOMEM) && domain != bo->allowed_domains) { @@ -411,7 +410,6 @@ static bool amdgpu_cs_try_evict(struct amdgpu_cs_parser *p, struct amdgpu_bo_list_entry *candidate = p->evictable; struct amdgpu_bo *bo = candidate->robj; struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev); - u64 initial_bytes_moved, bytes_moved; bool update_bytes_moved_vis; uint32_t other; @@ -435,18 +433,14 @@ static bool amdgpu_cs_try_evict(struct amdgpu_cs_parser *p, continue; /* Good we can try to move this BO somewhere else */ - amdgpu_ttm_placement_from_domain(bo, other); update_bytes_moved_vis = adev->gmc.visible_vram_size < adev->gmc.real_vram_size && - bo->tbo.mem.mem_type == TTM_PL_VRAM && - bo->tbo.mem.start < adev->gmc.visible_vram_size >> PAGE_SHIFT; - initial_bytes_moved = atomic64_read(&adev->num_bytes_moved); + amdgpu_bo_in_cpu_visible_vram(bo); + amdgpu_ttm_placement_from_domain(bo, other); r = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx); - bytes_moved = atomic64_read(&adev->num_bytes_moved) - - initial_bytes_moved; - p->bytes_moved += bytes_moved; + p->bytes_moved += ctx.bytes_moved; if (update_bytes_moved_vis) - p->bytes_moved_vis += bytes_moved; + p->bytes_moved_vis += ctx.bytes_moved; if (unlikely(r)) break; @@ -528,6 +522,9 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, struct amdgpu_bo_list_entry *e; struct list_head duplicates; unsigned i, tries = 10; + struct amdgpu_bo *gds; + struct amdgpu_bo *gws; + struct amdgpu_bo *oa; int r; INIT_LIST_HEAD(&p->validated); @@ -536,7 +533,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, if (p->bo_list) { amdgpu_bo_list_get_list(p->bo_list, &p->validated); if (p->bo_list->first_userptr != p->bo_list->num_entries) - p->mn = amdgpu_mn_get(p->adev); + p->mn = amdgpu_mn_get(p->adev, AMDGPU_MN_TYPE_GFX); } INIT_LIST_HEAD(&duplicates); @@ -658,31 +655,36 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, amdgpu_cs_report_moved_bytes(p->adev, p->bytes_moved, p->bytes_moved_vis); + if (p->bo_list) { - struct amdgpu_bo *gds = p->bo_list->gds_obj; - struct amdgpu_bo *gws = p->bo_list->gws_obj; - struct amdgpu_bo *oa = p->bo_list->oa_obj; struct amdgpu_vm *vm = &fpriv->vm; unsigned i; + gds = p->bo_list->gds_obj; + gws = p->bo_list->gws_obj; + oa = p->bo_list->oa_obj; for (i = 0; i < p->bo_list->num_entries; i++) { struct amdgpu_bo *bo = p->bo_list->array[i].robj; p->bo_list->array[i].bo_va = amdgpu_vm_bo_find(vm, bo); } + } else { + gds = p->adev->gds.gds_gfx_bo; + gws = p->adev->gds.gws_gfx_bo; + oa = p->adev->gds.oa_gfx_bo; + } - if (gds) { - p->job->gds_base = amdgpu_bo_gpu_offset(gds); - p->job->gds_size = amdgpu_bo_size(gds); - } - if (gws) { - p->job->gws_base = amdgpu_bo_gpu_offset(gws); - p->job->gws_size = amdgpu_bo_size(gws); - } - if (oa) { - p->job->oa_base = amdgpu_bo_gpu_offset(oa); - p->job->oa_size = amdgpu_bo_size(oa); - } + if (gds) { + p->job->gds_base = amdgpu_bo_gpu_offset(gds); + p->job->gds_size = amdgpu_bo_size(gds); + } + if (gws) { + p->job->gws_base = amdgpu_bo_gpu_offset(gws); + p->job->gws_size = amdgpu_bo_size(gws); + } + if (oa) { + p->job->oa_base = amdgpu_bo_gpu_offset(oa); + p->job->oa_size = amdgpu_bo_size(oa); } if (!r && p->uf_entry.robj) { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c index 3fabf9f97022..c5bb36275e93 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c @@ -91,7 +91,7 @@ static int amdgpu_ctx_init(struct amdgpu_device *adev, continue; r = drm_sched_entity_init(&ring->sched, &ctx->rings[i].entity, - rq, amdgpu_sched_jobs, &ctx->guilty); + rq, &ctx->guilty); if (r) goto failed; } @@ -111,8 +111,9 @@ failed: return r; } -static void amdgpu_ctx_fini(struct amdgpu_ctx *ctx) +static void amdgpu_ctx_fini(struct kref *ref) { + struct amdgpu_ctx *ctx = container_of(ref, struct amdgpu_ctx, refcount); struct amdgpu_device *adev = ctx->adev; unsigned i, j; @@ -125,13 +126,11 @@ static void amdgpu_ctx_fini(struct amdgpu_ctx *ctx) kfree(ctx->fences); ctx->fences = NULL; - for (i = 0; i < adev->num_rings; i++) - drm_sched_entity_fini(&adev->rings[i]->sched, - &ctx->rings[i].entity); - amdgpu_queue_mgr_fini(adev, &ctx->queue_mgr); mutex_destroy(&ctx->lock); + + kfree(ctx); } static int amdgpu_ctx_alloc(struct amdgpu_device *adev, @@ -170,12 +169,20 @@ static int amdgpu_ctx_alloc(struct amdgpu_device *adev, static void amdgpu_ctx_do_release(struct kref *ref) { struct amdgpu_ctx *ctx; + u32 i; ctx = container_of(ref, struct amdgpu_ctx, refcount); - amdgpu_ctx_fini(ctx); + for (i = 0; i < ctx->adev->num_rings; i++) { - kfree(ctx); + if (ctx->adev->rings[i] == &ctx->adev->gfx.kiq.ring) + continue; + + drm_sched_entity_fini(&ctx->adev->rings[i]->sched, + &ctx->rings[i].entity); + } + + amdgpu_ctx_fini(ref); } static int amdgpu_ctx_free(struct amdgpu_fpriv *fpriv, uint32_t id) @@ -437,16 +444,72 @@ void amdgpu_ctx_mgr_init(struct amdgpu_ctx_mgr *mgr) idr_init(&mgr->ctx_handles); } +void amdgpu_ctx_mgr_entity_fini(struct amdgpu_ctx_mgr *mgr) +{ + struct amdgpu_ctx *ctx; + struct idr *idp; + uint32_t id, i; + + idp = &mgr->ctx_handles; + + idr_for_each_entry(idp, ctx, id) { + + if (!ctx->adev) + return; + + for (i = 0; i < ctx->adev->num_rings; i++) { + + if (ctx->adev->rings[i] == &ctx->adev->gfx.kiq.ring) + continue; + + if (kref_read(&ctx->refcount) == 1) + drm_sched_entity_do_release(&ctx->adev->rings[i]->sched, + &ctx->rings[i].entity); + else + DRM_ERROR("ctx %p is still alive\n", ctx); + } + } +} + +void amdgpu_ctx_mgr_entity_cleanup(struct amdgpu_ctx_mgr *mgr) +{ + struct amdgpu_ctx *ctx; + struct idr *idp; + uint32_t id, i; + + idp = &mgr->ctx_handles; + + idr_for_each_entry(idp, ctx, id) { + + if (!ctx->adev) + return; + + for (i = 0; i < ctx->adev->num_rings; i++) { + + if (ctx->adev->rings[i] == &ctx->adev->gfx.kiq.ring) + continue; + + if (kref_read(&ctx->refcount) == 1) + drm_sched_entity_cleanup(&ctx->adev->rings[i]->sched, + &ctx->rings[i].entity); + else + DRM_ERROR("ctx %p is still alive\n", ctx); + } + } +} + void amdgpu_ctx_mgr_fini(struct amdgpu_ctx_mgr *mgr) { struct amdgpu_ctx *ctx; struct idr *idp; uint32_t id; + amdgpu_ctx_mgr_entity_cleanup(mgr); + idp = &mgr->ctx_handles; idr_for_each_entry(idp, ctx, id) { - if (kref_put(&ctx->refcount, amdgpu_ctx_do_release) != 1) + if (kref_put(&ctx->refcount, amdgpu_ctx_fini) != 1) DRM_ERROR("ctx %p is still alive\n", ctx); } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c index 448d69fe3756..f5fb93795a69 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c @@ -28,8 +28,13 @@ #include #include "amdgpu.h" -/* - * Debugfs +/** + * amdgpu_debugfs_add_files - Add simple debugfs entries + * + * @adev: Device to attach debugfs entries to + * @files: Array of function callbacks that respond to reads + * @nfiles: Number of callbacks to register + * */ int amdgpu_debugfs_add_files(struct amdgpu_device *adev, const struct drm_info_list *files, @@ -64,7 +69,33 @@ int amdgpu_debugfs_add_files(struct amdgpu_device *adev, #if defined(CONFIG_DEBUG_FS) - +/** + * amdgpu_debugfs_process_reg_op - Handle MMIO register reads/writes + * + * @read: True if reading + * @f: open file handle + * @buf: User buffer to write/read to + * @size: Number of bytes to write/read + * @pos: Offset to seek to + * + * This debugfs entry has special meaning on the offset being sought. + * Various bits have different meanings: + * + * Bit 62: Indicates a GRBM bank switch is needed + * Bit 61: Indicates a SRBM bank switch is needed (implies bit 62 is + * zero) + * Bits 24..33: The SE or ME selector if needed + * Bits 34..43: The SH (or SA) or PIPE selector if needed + * Bits 44..53: The INSTANCE (or CU/WGP) or QUEUE selector if needed + * + * Bit 23: Indicates that the PM power gating lock should be held + * This is necessary to read registers that might be + * unreliable during a power gating transistion. + * + * The lower bits are the BYTE offset of the register to read. This + * allows reading multiple registers in a single call and having + * the returned size reflect that. + */ static int amdgpu_debugfs_process_reg_op(bool read, struct file *f, char __user *buf, size_t size, loff_t *pos) { @@ -164,19 +195,37 @@ end: return result; } - +/** + * amdgpu_debugfs_regs_read - Callback for reading MMIO registers + */ static ssize_t amdgpu_debugfs_regs_read(struct file *f, char __user *buf, size_t size, loff_t *pos) { return amdgpu_debugfs_process_reg_op(true, f, buf, size, pos); } +/** + * amdgpu_debugfs_regs_write - Callback for writing MMIO registers + */ static ssize_t amdgpu_debugfs_regs_write(struct file *f, const char __user *buf, size_t size, loff_t *pos) { return amdgpu_debugfs_process_reg_op(false, f, (char __user *)buf, size, pos); } + +/** + * amdgpu_debugfs_regs_pcie_read - Read from a PCIE register + * + * @f: open file handle + * @buf: User buffer to store read data in + * @size: Number of bytes to read + * @pos: Offset to seek to + * + * The lower bits are the BYTE offset of the register to read. This + * allows reading multiple registers in a single call and having + * the returned size reflect that. + */ static ssize_t amdgpu_debugfs_regs_pcie_read(struct file *f, char __user *buf, size_t size, loff_t *pos) { @@ -204,6 +253,18 @@ static ssize_t amdgpu_debugfs_regs_pcie_read(struct file *f, char __user *buf, return result; } +/** + * amdgpu_debugfs_regs_pcie_write - Write to a PCIE register + * + * @f: open file handle + * @buf: User buffer to write data from + * @size: Number of bytes to write + * @pos: Offset to seek to + * + * The lower bits are the BYTE offset of the register to write. This + * allows writing multiple registers in a single call and having + * the returned size reflect that. + */ static ssize_t amdgpu_debugfs_regs_pcie_write(struct file *f, const char __user *buf, size_t size, loff_t *pos) { @@ -232,6 +293,18 @@ static ssize_t amdgpu_debugfs_regs_pcie_write(struct file *f, const char __user return result; } +/** + * amdgpu_debugfs_regs_didt_read - Read from a DIDT register + * + * @f: open file handle + * @buf: User buffer to store read data in + * @size: Number of bytes to read + * @pos: Offset to seek to + * + * The lower bits are the BYTE offset of the register to read. This + * allows reading multiple registers in a single call and having + * the returned size reflect that. + */ static ssize_t amdgpu_debugfs_regs_didt_read(struct file *f, char __user *buf, size_t size, loff_t *pos) { @@ -259,6 +332,18 @@ static ssize_t amdgpu_debugfs_regs_didt_read(struct file *f, char __user *buf, return result; } +/** + * amdgpu_debugfs_regs_didt_write - Write to a DIDT register + * + * @f: open file handle + * @buf: User buffer to write data from + * @size: Number of bytes to write + * @pos: Offset to seek to + * + * The lower bits are the BYTE offset of the register to write. This + * allows writing multiple registers in a single call and having + * the returned size reflect that. + */ static ssize_t amdgpu_debugfs_regs_didt_write(struct file *f, const char __user *buf, size_t size, loff_t *pos) { @@ -287,6 +372,18 @@ static ssize_t amdgpu_debugfs_regs_didt_write(struct file *f, const char __user return result; } +/** + * amdgpu_debugfs_regs_smc_read - Read from a SMC register + * + * @f: open file handle + * @buf: User buffer to store read data in + * @size: Number of bytes to read + * @pos: Offset to seek to + * + * The lower bits are the BYTE offset of the register to read. This + * allows reading multiple registers in a single call and having + * the returned size reflect that. + */ static ssize_t amdgpu_debugfs_regs_smc_read(struct file *f, char __user *buf, size_t size, loff_t *pos) { @@ -314,6 +411,18 @@ static ssize_t amdgpu_debugfs_regs_smc_read(struct file *f, char __user *buf, return result; } +/** + * amdgpu_debugfs_regs_smc_write - Write to a SMC register + * + * @f: open file handle + * @buf: User buffer to write data from + * @size: Number of bytes to write + * @pos: Offset to seek to + * + * The lower bits are the BYTE offset of the register to write. This + * allows writing multiple registers in a single call and having + * the returned size reflect that. + */ static ssize_t amdgpu_debugfs_regs_smc_write(struct file *f, const char __user *buf, size_t size, loff_t *pos) { @@ -342,6 +451,20 @@ static ssize_t amdgpu_debugfs_regs_smc_write(struct file *f, const char __user * return result; } +/** + * amdgpu_debugfs_gca_config_read - Read from gfx config data + * + * @f: open file handle + * @buf: User buffer to store read data in + * @size: Number of bytes to read + * @pos: Offset to seek to + * + * This file is used to access configuration data in a somewhat + * stable fashion. The format is a series of DWORDs with the first + * indicating which revision it is. New content is appended to the + * end so that older software can still read the data. + */ + static ssize_t amdgpu_debugfs_gca_config_read(struct file *f, char __user *buf, size_t size, loff_t *pos) { @@ -418,6 +541,19 @@ static ssize_t amdgpu_debugfs_gca_config_read(struct file *f, char __user *buf, return result; } +/** + * amdgpu_debugfs_sensor_read - Read from the powerplay sensors + * + * @f: open file handle + * @buf: User buffer to store read data in + * @size: Number of bytes to read + * @pos: Offset to seek to + * + * The offset is treated as the BYTE address of one of the sensors + * enumerated in amd/include/kgd_pp_interface.h under the + * 'amd_pp_sensors' enumeration. For instance to read the UVD VCLK + * you would use the offset 3 * 4 = 12. + */ static ssize_t amdgpu_debugfs_sensor_read(struct file *f, char __user *buf, size_t size, loff_t *pos) { @@ -428,7 +564,7 @@ static ssize_t amdgpu_debugfs_sensor_read(struct file *f, char __user *buf, if (size & 3 || *pos & 0x3) return -EINVAL; - if (amdgpu_dpm == 0) + if (!adev->pm.dpm_enabled) return -EINVAL; /* convert offset to sensor number */ @@ -457,6 +593,27 @@ static ssize_t amdgpu_debugfs_sensor_read(struct file *f, char __user *buf, return !r ? outsize : r; } +/** amdgpu_debugfs_wave_read - Read WAVE STATUS data + * + * @f: open file handle + * @buf: User buffer to store read data in + * @size: Number of bytes to read + * @pos: Offset to seek to + * + * The offset being sought changes which wave that the status data + * will be returned for. The bits are used as follows: + * + * Bits 0..6: Byte offset into data + * Bits 7..14: SE selector + * Bits 15..22: SH/SA selector + * Bits 23..30: CU/{WGP+SIMD} selector + * Bits 31..36: WAVE ID selector + * Bits 37..44: SIMD ID selector + * + * The returned data begins with one DWORD of version information + * Followed by WAVE STATUS registers relevant to the GFX IP version + * being used. See gfx_v8_0_read_wave_data() for an example output. + */ static ssize_t amdgpu_debugfs_wave_read(struct file *f, char __user *buf, size_t size, loff_t *pos) { @@ -507,6 +664,28 @@ static ssize_t amdgpu_debugfs_wave_read(struct file *f, char __user *buf, return result; } +/** amdgpu_debugfs_gpr_read - Read wave gprs + * + * @f: open file handle + * @buf: User buffer to store read data in + * @size: Number of bytes to read + * @pos: Offset to seek to + * + * The offset being sought changes which wave that the status data + * will be returned for. The bits are used as follows: + * + * Bits 0..11: Byte offset into data + * Bits 12..19: SE selector + * Bits 20..27: SH/SA selector + * Bits 28..35: CU/{WGP+SIMD} selector + * Bits 36..43: WAVE ID selector + * Bits 37..44: SIMD ID selector + * Bits 52..59: Thread selector + * Bits 60..61: Bank selector (VGPR=0,SGPR=1) + * + * The return data comes from the SGPR or VGPR register bank for + * the selected operational unit. + */ static ssize_t amdgpu_debugfs_gpr_read(struct file *f, char __user *buf, size_t size, loff_t *pos) { @@ -637,6 +816,12 @@ static const char *debugfs_regs_names[] = { "amdgpu_gpr", }; +/** + * amdgpu_debugfs_regs_init - Initialize debugfs entries that provide + * register access. + * + * @adev: The device to attach the debugfs entries to + */ int amdgpu_debugfs_regs_init(struct amdgpu_device *adev) { struct drm_minor *minor = adev->ddev->primary; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 34af664b9f93..3317d1536f4f 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -83,8 +83,10 @@ static const char *amdgpu_asic_name[] = { "POLARIS10", "POLARIS11", "POLARIS12", + "VEGAM", "VEGA10", "VEGA12", + "VEGA20", "RAVEN", "LAST", }; @@ -690,6 +692,8 @@ void amdgpu_device_gart_location(struct amdgpu_device *adev, { u64 size_af, size_bf; + mc->gart_size += adev->pm.smu_prv_buffer_size; + size_af = adev->gmc.mc_mask - mc->vram_end; size_bf = mc->vram_start; if (size_bf > size_af) { @@ -907,6 +911,46 @@ static void amdgpu_device_check_vm_size(struct amdgpu_device *adev) } } +static void amdgpu_device_check_smu_prv_buffer_size(struct amdgpu_device *adev) +{ + struct sysinfo si; + bool is_os_64 = (sizeof(void *) == 8) ? true : false; + uint64_t total_memory; + uint64_t dram_size_seven_GB = 0x1B8000000; + uint64_t dram_size_three_GB = 0xB8000000; + + if (amdgpu_smu_memory_pool_size == 0) + return; + + if (!is_os_64) { + DRM_WARN("Not 64-bit OS, feature not supported\n"); + goto def_value; + } + si_meminfo(&si); + total_memory = (uint64_t)si.totalram * si.mem_unit; + + if ((amdgpu_smu_memory_pool_size == 1) || + (amdgpu_smu_memory_pool_size == 2)) { + if (total_memory < dram_size_three_GB) + goto def_value1; + } else if ((amdgpu_smu_memory_pool_size == 4) || + (amdgpu_smu_memory_pool_size == 8)) { + if (total_memory < dram_size_seven_GB) + goto def_value1; + } else { + DRM_WARN("Smu memory pool size not supported\n"); + goto def_value; + } + adev->pm.smu_prv_buffer_size = amdgpu_smu_memory_pool_size << 28; + + return; + +def_value1: + DRM_WARN("No enough system memory\n"); +def_value: + adev->pm.smu_prv_buffer_size = 0; +} + /** * amdgpu_device_check_arguments - validate module params * @@ -948,6 +992,8 @@ static void amdgpu_device_check_arguments(struct amdgpu_device *adev) amdgpu_vm_fragment_size = -1; } + amdgpu_device_check_smu_prv_buffer_size(adev); + amdgpu_device_check_vm_size(adev); amdgpu_device_check_block_size(adev); @@ -1039,10 +1085,11 @@ static const struct vga_switcheroo_client_ops amdgpu_switcheroo_ops = { * the hardware IP specified. * Returns the error code from the last instance. */ -int amdgpu_device_ip_set_clockgating_state(struct amdgpu_device *adev, +int amdgpu_device_ip_set_clockgating_state(void *dev, enum amd_ip_block_type block_type, enum amd_clockgating_state state) { + struct amdgpu_device *adev = dev; int i, r = 0; for (i = 0; i < adev->num_ip_blocks; i++) { @@ -1072,10 +1119,11 @@ int amdgpu_device_ip_set_clockgating_state(struct amdgpu_device *adev, * the hardware IP specified. * Returns the error code from the last instance. */ -int amdgpu_device_ip_set_powergating_state(struct amdgpu_device *adev, +int amdgpu_device_ip_set_powergating_state(void *dev, enum amd_ip_block_type block_type, enum amd_powergating_state state) { + struct amdgpu_device *adev = dev; int i, r = 0; for (i = 0; i < adev->num_ip_blocks; i++) { @@ -1320,9 +1368,10 @@ static int amdgpu_device_parse_gpu_info_fw(struct amdgpu_device *adev) case CHIP_TOPAZ: case CHIP_TONGA: case CHIP_FIJI: - case CHIP_POLARIS11: case CHIP_POLARIS10: + case CHIP_POLARIS11: case CHIP_POLARIS12: + case CHIP_VEGAM: case CHIP_CARRIZO: case CHIP_STONEY: #ifdef CONFIG_DRM_AMDGPU_SI @@ -1339,6 +1388,7 @@ static int amdgpu_device_parse_gpu_info_fw(struct amdgpu_device *adev) case CHIP_KABINI: case CHIP_MULLINS: #endif + case CHIP_VEGA20: default: return 0; case CHIP_VEGA10: @@ -1428,9 +1478,10 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) case CHIP_TOPAZ: case CHIP_TONGA: case CHIP_FIJI: - case CHIP_POLARIS11: case CHIP_POLARIS10: + case CHIP_POLARIS11: case CHIP_POLARIS12: + case CHIP_VEGAM: case CHIP_CARRIZO: case CHIP_STONEY: if (adev->asic_type == CHIP_CARRIZO || adev->asic_type == CHIP_STONEY) @@ -1472,6 +1523,7 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) #endif case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: case CHIP_RAVEN: if (adev->asic_type == CHIP_RAVEN) adev->family = AMDGPU_FAMILY_RV; @@ -1499,6 +1551,8 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) return -EAGAIN; } + adev->powerplay.pp_feature = amdgpu_pp_feature_mask; + for (i = 0; i < adev->num_ip_blocks; i++) { if ((amdgpu_ip_block_mask & (1 << i)) == 0) { DRM_ERROR("disabled ip block: %d <%s>\n", @@ -1654,12 +1708,17 @@ static int amdgpu_device_ip_late_set_cg_state(struct amdgpu_device *adev) if (amdgpu_emu_mode == 1) return 0; + r = amdgpu_ib_ring_tests(adev); + if (r) + DRM_ERROR("ib ring test failed (%d).\n", r); + for (i = 0; i < adev->num_ip_blocks; i++) { if (!adev->ip_blocks[i].status.valid) continue; /* skip CG for VCE/UVD, it's handled specially */ if (adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_UVD && adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_VCE && + adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_VCN && adev->ip_blocks[i].version->funcs->set_clockgating_state) { /* enable clockgating to save power */ r = adev->ip_blocks[i].version->funcs->set_clockgating_state((void *)adev, @@ -1671,6 +1730,18 @@ static int amdgpu_device_ip_late_set_cg_state(struct amdgpu_device *adev) } } } + + if (adev->powerplay.pp_feature & PP_GFXOFF_MASK) { + /* enable gfx powergating */ + amdgpu_device_ip_set_powergating_state(adev, + AMD_IP_BLOCK_TYPE_GFX, + AMD_PG_STATE_GATE); + /* enable gfxoff */ + amdgpu_device_ip_set_powergating_state(adev, + AMD_IP_BLOCK_TYPE_SMC, + AMD_PG_STATE_GATE); + } + return 0; } @@ -1704,8 +1775,8 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev) } } - mod_delayed_work(system_wq, &adev->late_init_work, - msecs_to_jiffies(AMDGPU_RESUME_MS)); + queue_delayed_work(system_wq, &adev->late_init_work, + msecs_to_jiffies(AMDGPU_RESUME_MS)); amdgpu_device_fill_reset_magic(adev); @@ -1759,6 +1830,7 @@ static int amdgpu_device_ip_fini(struct amdgpu_device *adev) if (adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_UVD && adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_VCE && + adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_VCN && adev->ip_blocks[i].version->funcs->set_clockgating_state) { /* ungate blocks before hw fini so that we can shutdown the blocks safely */ r = adev->ip_blocks[i].version->funcs->set_clockgating_state((void *)adev, @@ -1850,6 +1922,12 @@ int amdgpu_device_ip_suspend(struct amdgpu_device *adev) if (amdgpu_sriov_vf(adev)) amdgpu_virt_request_full_gpu(adev, false); + /* ungate SMC block powergating */ + if (adev->powerplay.pp_feature & PP_GFXOFF_MASK) + amdgpu_device_ip_set_powergating_state(adev, + AMD_IP_BLOCK_TYPE_SMC, + AMD_CG_STATE_UNGATE); + /* ungate SMC block first */ r = amdgpu_device_ip_set_clockgating_state(adev, AMD_IP_BLOCK_TYPE_SMC, AMD_CG_STATE_UNGATE); @@ -2086,16 +2164,15 @@ bool amdgpu_device_asic_has_dc_support(enum amd_asic_type asic_type) case CHIP_MULLINS: case CHIP_CARRIZO: case CHIP_STONEY: - case CHIP_POLARIS11: case CHIP_POLARIS10: + case CHIP_POLARIS11: case CHIP_POLARIS12: + case CHIP_VEGAM: case CHIP_TONGA: case CHIP_FIJI: -#if defined(CONFIG_DRM_AMD_DC_PRE_VEGA) - return amdgpu_dc != 0; -#endif case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: #if defined(CONFIG_DRM_AMD_DC_DCN1_0) case CHIP_RAVEN: #endif @@ -2375,10 +2452,6 @@ fence_driver_init: goto failed; } - r = amdgpu_ib_ring_tests(adev); - if (r) - DRM_ERROR("ib ring test failed (%d).\n", r); - if (amdgpu_sriov_vf(adev)) amdgpu_virt_init_data_exchange(adev); @@ -2539,7 +2612,7 @@ int amdgpu_device_suspend(struct drm_device *dev, bool suspend, bool fbcon) /* unpin the front buffers and cursors */ list_for_each_entry(crtc, &dev->mode_config.crtc_list, head) { struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc); - struct amdgpu_framebuffer *rfb = to_amdgpu_framebuffer(crtc->primary->fb); + struct drm_framebuffer *fb = crtc->primary->fb; struct amdgpu_bo *robj; if (amdgpu_crtc->cursor_bo) { @@ -2551,10 +2624,10 @@ int amdgpu_device_suspend(struct drm_device *dev, bool suspend, bool fbcon) } } - if (rfb == NULL || rfb->obj == NULL) { + if (fb == NULL || fb->obj[0] == NULL) { continue; } - robj = gem_to_amdgpu_bo(rfb->obj); + robj = gem_to_amdgpu_bo(fb->obj[0]); /* don't unpin kernel fb objects */ if (!amdgpu_fbdev_robj_is_fb(adev, robj)) { r = amdgpu_bo_reserve(robj, true); @@ -2640,11 +2713,6 @@ int amdgpu_device_resume(struct drm_device *dev, bool resume, bool fbcon) } amdgpu_fence_driver_resume(adev); - if (resume) { - r = amdgpu_ib_ring_tests(adev); - if (r) - DRM_ERROR("ib ring test failed (%d).\n", r); - } r = amdgpu_device_ip_late_init(adev); if (r) @@ -2736,6 +2804,9 @@ static bool amdgpu_device_ip_check_soft_reset(struct amdgpu_device *adev) if (amdgpu_sriov_vf(adev)) return true; + if (amdgpu_asic_need_full_reset(adev)) + return true; + for (i = 0; i < adev->num_ip_blocks; i++) { if (!adev->ip_blocks[i].status.valid) continue; @@ -2792,6 +2863,9 @@ static bool amdgpu_device_ip_need_full_reset(struct amdgpu_device *adev) { int i; + if (amdgpu_asic_need_full_reset(adev)) + return true; + for (i = 0; i < adev->num_ip_blocks; i++) { if (!adev->ip_blocks[i].status.valid) continue; @@ -3087,20 +3161,19 @@ static int amdgpu_device_reset_sriov(struct amdgpu_device *adev, /* now we are okay to resume SMC/CP/SDMA */ r = amdgpu_device_ip_reinit_late_sriov(adev); - amdgpu_virt_release_full_gpu(adev, true); if (r) goto error; amdgpu_irq_gpu_reset_resume_helper(adev); r = amdgpu_ib_ring_tests(adev); +error: + amdgpu_virt_release_full_gpu(adev, true); if (!r && adev->virt.gim_feature & AMDGIM_FEATURE_GIM_FLR_VRAMLOST) { atomic_inc(&adev->vram_lost_counter); r = amdgpu_device_handle_vram_lost(adev); } -error: - return r; } @@ -3117,7 +3190,6 @@ error: int amdgpu_device_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job *job, bool force) { - struct drm_atomic_state *state = NULL; int i, r, resched; if (!force && !amdgpu_device_ip_check_soft_reset(adev)) { @@ -3140,10 +3212,6 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, /* block TTM */ resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev); - /* store modesetting */ - if (amdgpu_device_has_dc_support(adev)) - state = drm_atomic_helper_suspend(adev->ddev); - /* block all schedulers and reset given job's ring */ for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = adev->rings[i]; @@ -3183,10 +3251,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, kthread_unpark(ring->sched.thread); } - if (amdgpu_device_has_dc_support(adev)) { - if (drm_atomic_helper_resume(adev->ddev, state)) - dev_info(adev->dev, "drm resume failed:%d\n", r); - } else { + if (!amdgpu_device_has_dc_support(adev)) { drm_helper_resume_force_mode(adev->ddev); } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_display.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_display.c index 93f700ab1bfb..76ee8e04ff11 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_display.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_display.c @@ -35,6 +35,7 @@ #include #include #include +#include #include static void amdgpu_display_flip_callback(struct dma_fence *f, @@ -151,8 +152,6 @@ int amdgpu_display_crtc_page_flip_target(struct drm_crtc *crtc, struct drm_device *dev = crtc->dev; struct amdgpu_device *adev = dev->dev_private; struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc); - struct amdgpu_framebuffer *old_amdgpu_fb; - struct amdgpu_framebuffer *new_amdgpu_fb; struct drm_gem_object *obj; struct amdgpu_flip_work *work; struct amdgpu_bo *new_abo; @@ -174,15 +173,13 @@ int amdgpu_display_crtc_page_flip_target(struct drm_crtc *crtc, work->async = (page_flip_flags & DRM_MODE_PAGE_FLIP_ASYNC) != 0; /* schedule unpin of the old buffer */ - old_amdgpu_fb = to_amdgpu_framebuffer(crtc->primary->fb); - obj = old_amdgpu_fb->obj; + obj = crtc->primary->fb->obj[0]; /* take a reference to the old object */ work->old_abo = gem_to_amdgpu_bo(obj); amdgpu_bo_ref(work->old_abo); - new_amdgpu_fb = to_amdgpu_framebuffer(fb); - obj = new_amdgpu_fb->obj; + obj = fb->obj[0]; new_abo = gem_to_amdgpu_bo(obj); /* pin the new buffer */ @@ -192,7 +189,7 @@ int amdgpu_display_crtc_page_flip_target(struct drm_crtc *crtc, goto cleanup; } - r = amdgpu_bo_pin(new_abo, amdgpu_display_framebuffer_domains(adev), &base); + r = amdgpu_bo_pin(new_abo, amdgpu_display_supported_domains(adev), &base); if (unlikely(r != 0)) { DRM_ERROR("failed to pin new abo buffer before flip\n"); goto unreserve; @@ -482,31 +479,12 @@ bool amdgpu_display_ddc_probe(struct amdgpu_connector *amdgpu_connector, return true; } -static void amdgpu_display_user_framebuffer_destroy(struct drm_framebuffer *fb) -{ - struct amdgpu_framebuffer *amdgpu_fb = to_amdgpu_framebuffer(fb); - - drm_gem_object_put_unlocked(amdgpu_fb->obj); - drm_framebuffer_cleanup(fb); - kfree(amdgpu_fb); -} - -static int amdgpu_display_user_framebuffer_create_handle( - struct drm_framebuffer *fb, - struct drm_file *file_priv, - unsigned int *handle) -{ - struct amdgpu_framebuffer *amdgpu_fb = to_amdgpu_framebuffer(fb); - - return drm_gem_handle_create(file_priv, amdgpu_fb->obj, handle); -} - static const struct drm_framebuffer_funcs amdgpu_fb_funcs = { - .destroy = amdgpu_display_user_framebuffer_destroy, - .create_handle = amdgpu_display_user_framebuffer_create_handle, + .destroy = drm_gem_fb_destroy, + .create_handle = drm_gem_fb_create_handle, }; -uint32_t amdgpu_display_framebuffer_domains(struct amdgpu_device *adev) +uint32_t amdgpu_display_supported_domains(struct amdgpu_device *adev) { uint32_t domain = AMDGPU_GEM_DOMAIN_VRAM; @@ -526,11 +504,11 @@ int amdgpu_display_framebuffer_init(struct drm_device *dev, struct drm_gem_object *obj) { int ret; - rfb->obj = obj; + rfb->base.obj[0] = obj; drm_helper_mode_fill_fb_struct(dev, &rfb->base, mode_cmd); ret = drm_framebuffer_init(dev, &rfb->base, &amdgpu_fb_funcs); if (ret) { - rfb->obj = NULL; + rfb->base.obj[0] = NULL; return ret; } return 0; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_display.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_display.h index 2b11d808f297..f66e3e3fef0a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_display.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_display.h @@ -23,7 +23,7 @@ #ifndef __AMDGPU_DISPLAY_H__ #define __AMDGPU_DISPLAY_H__ -uint32_t amdgpu_display_framebuffer_domains(struct amdgpu_device *adev); +uint32_t amdgpu_display_supported_domains(struct amdgpu_device *adev); struct drm_framebuffer * amdgpu_display_user_framebuffer_create(struct drm_device *dev, struct drm_file *file_priv, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.c index e997ebbe43ea..77ad59ade85c 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.c @@ -115,6 +115,26 @@ void amdgpu_dpm_print_ps_status(struct amdgpu_device *adev, pr_cont("\n"); } +void amdgpu_dpm_get_active_displays(struct amdgpu_device *adev) +{ + struct drm_device *ddev = adev->ddev; + struct drm_crtc *crtc; + struct amdgpu_crtc *amdgpu_crtc; + + adev->pm.dpm.new_active_crtcs = 0; + adev->pm.dpm.new_active_crtc_count = 0; + if (adev->mode_info.num_crtc && adev->mode_info.mode_config_initialized) { + list_for_each_entry(crtc, + &ddev->mode_config.crtc_list, head) { + amdgpu_crtc = to_amdgpu_crtc(crtc); + if (amdgpu_crtc->enabled) { + adev->pm.dpm.new_active_crtcs |= (1 << amdgpu_crtc->crtc_id); + adev->pm.dpm.new_active_crtc_count++; + } + } + } +} + u32 amdgpu_dpm_get_vblank_time(struct amdgpu_device *adev) { @@ -432,7 +452,7 @@ int amdgpu_parse_extended_power_table(struct amdgpu_device *adev) ATOM_PPLIB_PhaseSheddingLimits_Record *entry; adev->pm.dpm.dyn_state.phase_shedding_limits_table.entries = - kzalloc(psl->ucNumEntries * + kcalloc(psl->ucNumEntries, sizeof(struct amdgpu_phase_shedding_limits_entry), GFP_KERNEL); if (!adev->pm.dpm.dyn_state.phase_shedding_limits_table.entries) { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.h index 643d008410c6..dd6203a0a6b7 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dpm.h @@ -52,8 +52,6 @@ enum amdgpu_dpm_event_src { AMDGPU_DPM_EVENT_SRC_DIGIAL_OR_EXTERNAL = 4 }; -#define SCLK_DEEP_SLEEP_MASK 0x8 - struct amdgpu_ps { u32 caps; /* vbios flags */ u32 class; /* vbios flags */ @@ -349,12 +347,6 @@ enum amdgpu_pcie_gen { ((adev)->powerplay.pp_funcs->set_clockgating_by_smu(\ (adev)->powerplay.pp_handle, msg_id)) -#define amdgpu_dpm_notify_smu_memory_info(adev, virtual_addr_low, \ - virtual_addr_hi, mc_addr_low, mc_addr_hi, size) \ - ((adev)->powerplay.pp_funcs->notify_smu_memory_info)( \ - (adev)->powerplay.pp_handle, virtual_addr_low, \ - virtual_addr_hi, mc_addr_low, mc_addr_hi, size) - #define amdgpu_dpm_get_power_profile_mode(adev, buf) \ ((adev)->powerplay.pp_funcs->get_power_profile_mode(\ (adev)->powerplay.pp_handle, buf)) @@ -445,6 +437,8 @@ struct amdgpu_pm { uint32_t pcie_gen_mask; uint32_t pcie_mlw_mask; struct amd_pp_display_configuration pm_display_cfg;/* set by dc */ + uint32_t smu_prv_buffer_size; + struct amdgpu_bo *smu_prv_buffer; }; #define R600_SSTU_DFLT 0 @@ -482,6 +476,7 @@ void amdgpu_dpm_print_ps_status(struct amdgpu_device *adev, struct amdgpu_ps *rps); u32 amdgpu_dpm_get_vblank_time(struct amdgpu_device *adev); u32 amdgpu_dpm_get_vrefresh(struct amdgpu_device *adev); +void amdgpu_dpm_get_active_displays(struct amdgpu_device *adev); bool amdgpu_is_uvd_state(u32 class, u32 class2); void amdgpu_calculate_u_and_p(u32 i, u32 r_c, u32 p_b, u32 *p, u32 *u); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index 0b19482b36b8..b0bf2f24da48 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c @@ -75,9 +75,10 @@ * - 3.23.0 - Add query for VRAM lost counter * - 3.24.0 - Add high priority compute support for gfx9 * - 3.25.0 - Add support for sensor query info (stable pstate sclk/mclk). + * - 3.26.0 - GFX9: Process AMDGPU_IB_FLAG_TC_WB_NOT_INVALIDATE. */ #define KMS_DRIVER_MAJOR 3 -#define KMS_DRIVER_MINOR 25 +#define KMS_DRIVER_MINOR 26 #define KMS_DRIVER_PATCHLEVEL 0 int amdgpu_vram_limit = 0; @@ -121,7 +122,7 @@ uint amdgpu_pg_mask = 0xffffffff; uint amdgpu_sdma_phase_quantum = 32; char *amdgpu_disable_cu = NULL; char *amdgpu_virtual_display = NULL; -uint amdgpu_pp_feature_mask = 0xffffbfff; +uint amdgpu_pp_feature_mask = 0xffff3fff; /* gfxoff (bit 15) disabled by default */ int amdgpu_ngg = 0; int amdgpu_prim_buf_per_se = 0; int amdgpu_pos_buf_per_se = 0; @@ -132,6 +133,7 @@ int amdgpu_lbpw = -1; int amdgpu_compute_multipipe = -1; int amdgpu_gpu_recovery = -1; /* auto */ int amdgpu_emu_mode = 0; +uint amdgpu_smu_memory_pool_size = 0; MODULE_PARM_DESC(vramlimit, "Restrict VRAM for testing, in megabytes"); module_param_named(vramlimit, amdgpu_vram_limit, int, 0600); @@ -316,6 +318,11 @@ MODULE_PARM_DESC(cik_support, "CIK support (1 = enabled (default), 0 = disabled) module_param_named(cik_support, amdgpu_cik_support, int, 0444); #endif +MODULE_PARM_DESC(smu_memory_pool_size, + "reserve gtt for smu debug usage, 0 = disable," + "0x1 = 256Mbyte, 0x2 = 512Mbyte, 0x4 = 1 Gbyte, 0x8 = 2GByte"); +module_param_named(smu_memory_pool_size, amdgpu_smu_memory_pool_size, uint, 0444); + static const struct pci_device_id pciidlist[] = { #ifdef CONFIG_DRM_AMDGPU_SI {0x1002, 0x6780, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TAHITI}, @@ -534,6 +541,9 @@ static const struct pci_device_id pciidlist[] = { {0x1002, 0x6995, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12}, {0x1002, 0x6997, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12}, {0x1002, 0x699F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12}, + /* VEGAM */ + {0x1002, 0x694C, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGAM}, + {0x1002, 0x694E, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGAM}, /* Vega 10 */ {0x1002, 0x6860, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10}, {0x1002, 0x6861, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA10}, @@ -550,6 +560,13 @@ static const struct pci_device_id pciidlist[] = { {0x1002, 0x69A2, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA12}, {0x1002, 0x69A3, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA12}, {0x1002, 0x69AF, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA12}, + /* Vega 20 */ + {0x1002, 0x66A0, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA20|AMD_EXP_HW_SUPPORT}, + {0x1002, 0x66A1, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA20|AMD_EXP_HW_SUPPORT}, + {0x1002, 0x66A2, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA20|AMD_EXP_HW_SUPPORT}, + {0x1002, 0x66A3, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA20|AMD_EXP_HW_SUPPORT}, + {0x1002, 0x66A7, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA20|AMD_EXP_HW_SUPPORT}, + {0x1002, 0x66AF, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_VEGA20|AMD_EXP_HW_SUPPORT}, /* Raven */ {0x1002, 0x15dd, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_RAVEN|AMD_IS_APU}, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fb.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_fb.c index 12063019751b..bc5fd8ebab5d 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fb.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fb.c @@ -137,7 +137,7 @@ static int amdgpufb_create_pinned_object(struct amdgpu_fbdev *rfbdev, /* need to align pitch with crtc limits */ mode_cmd->pitches[0] = amdgpu_align_pitch(adev, mode_cmd->width, cpp, fb_tiled); - domain = amdgpu_display_framebuffer_domains(adev); + domain = amdgpu_display_supported_domains(adev); height = ALIGN(mode_cmd->height, 8); size = mode_cmd->pitches[0] * height; @@ -292,9 +292,9 @@ static int amdgpu_fbdev_destroy(struct drm_device *dev, struct amdgpu_fbdev *rfb drm_fb_helper_unregister_fbi(&rfbdev->helper); - if (rfb->obj) { - amdgpufb_destroy_pinned_object(rfb->obj); - rfb->obj = NULL; + if (rfb->base.obj[0]) { + amdgpufb_destroy_pinned_object(rfb->base.obj[0]); + rfb->base.obj[0] = NULL; drm_framebuffer_unregister_private(&rfb->base); drm_framebuffer_cleanup(&rfb->base); } @@ -377,7 +377,7 @@ int amdgpu_fbdev_total_size(struct amdgpu_device *adev) if (!adev->mode_info.rfbdev) return 0; - robj = gem_to_amdgpu_bo(adev->mode_info.rfbdev->rfb.obj); + robj = gem_to_amdgpu_bo(adev->mode_info.rfbdev->rfb.base.obj[0]); size += amdgpu_bo_size(robj); return size; } @@ -386,7 +386,7 @@ bool amdgpu_fbdev_robj_is_fb(struct amdgpu_device *adev, struct amdgpu_bo *robj) { if (!adev->mode_info.rfbdev) return false; - if (robj == gem_to_amdgpu_bo(adev->mode_info.rfbdev->rfb.obj)) + if (robj == gem_to_amdgpu_bo(adev->mode_info.rfbdev->rfb.base.obj[0])) return true; return false; } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c index 97449e06a242..39ec6b8890a1 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c @@ -131,7 +131,8 @@ static u32 amdgpu_fence_read(struct amdgpu_ring *ring) * Emits a fence command on the requested ring (all asics). * Returns 0 on success, -ENOMEM on failure. */ -int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **f) +int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **f, + unsigned flags) { struct amdgpu_device *adev = ring->adev; struct amdgpu_fence *fence; @@ -149,7 +150,7 @@ int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **f) adev->fence_context + ring->idx, seq); amdgpu_ring_emit_fence(ring, ring->fence_drv.gpu_addr, - seq, AMDGPU_FENCE_FLAG_INT); + seq, flags | AMDGPU_FENCE_FLAG_INT); ptr = &ring->fence_drv.fences[seq & ring->fence_drv.num_fences_mask]; /* This function can't be called concurrently anyway, otherwise @@ -375,14 +376,14 @@ int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring, struct amdgpu_device *adev = ring->adev; uint64_t index; - if (ring != &adev->uvd.ring) { + if (ring != &adev->uvd.inst[ring->me].ring) { ring->fence_drv.cpu_addr = &adev->wb.wb[ring->fence_offs]; ring->fence_drv.gpu_addr = adev->wb.gpu_addr + (ring->fence_offs * 4); } else { /* put fence directly behind firmware */ index = ALIGN(adev->uvd.fw->size, 8); - ring->fence_drv.cpu_addr = adev->uvd.cpu_addr + index; - ring->fence_drv.gpu_addr = adev->uvd.gpu_addr + index; + ring->fence_drv.cpu_addr = adev->uvd.inst[ring->me].cpu_addr + index; + ring->fence_drv.gpu_addr = adev->uvd.inst[ring->me].gpu_addr + index; } amdgpu_fence_write(ring, atomic_read(&ring->fence_drv.last_seq)); amdgpu_irq_get(adev, irq_src, irq_type); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c index cf0f186c6092..dd11b7313ca0 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c @@ -113,12 +113,17 @@ int amdgpu_gart_table_vram_alloc(struct amdgpu_device *adev) int r; if (adev->gart.robj == NULL) { - r = amdgpu_bo_create(adev, adev->gart.table_size, PAGE_SIZE, - AMDGPU_GEM_DOMAIN_VRAM, - AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED | - AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS, - ttm_bo_type_kernel, NULL, - &adev->gart.robj); + struct amdgpu_bo_param bp; + + memset(&bp, 0, sizeof(bp)); + bp.size = adev->gart.table_size; + bp.byte_align = PAGE_SIZE; + bp.domain = AMDGPU_GEM_DOMAIN_VRAM; + bp.flags = AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED | + AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS; + bp.type = ttm_bo_type_kernel; + bp.resv = NULL; + r = amdgpu_bo_create(adev, &bp, &adev->gart.robj); if (r) { return r; } @@ -364,7 +369,8 @@ int amdgpu_gart_init(struct amdgpu_device *adev) #ifdef CONFIG_DRM_AMDGPU_GART_DEBUGFS /* Allocate pages table */ - adev->gart.pages = vzalloc(sizeof(void *) * adev->gart.num_cpu_pages); + adev->gart.pages = vzalloc(array_size(sizeof(void *), + adev->gart.num_cpu_pages)); if (adev->gart.pages == NULL) return -ENOMEM; #endif diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c index 46b9ea4e6103..5fb156a01774 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c @@ -30,6 +30,7 @@ #include #include #include "amdgpu.h" +#include "amdgpu_display.h" void amdgpu_gem_object_free(struct drm_gem_object *gobj) { @@ -48,17 +49,25 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size, struct drm_gem_object **obj) { struct amdgpu_bo *bo; + struct amdgpu_bo_param bp; int r; + memset(&bp, 0, sizeof(bp)); *obj = NULL; /* At least align on page size */ if (alignment < PAGE_SIZE) { alignment = PAGE_SIZE; } + bp.size = size; + bp.byte_align = alignment; + bp.type = type; + bp.resv = resv; + bp.preferred_domain = initial_domain; retry: - r = amdgpu_bo_create(adev, size, alignment, initial_domain, - flags, type, resv, &bo); + bp.flags = flags; + bp.domain = initial_domain; + r = amdgpu_bo_create(adev, &bp, &bo); if (r) { if (r != -ERESTARTSYS) { if (flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) { @@ -221,17 +230,19 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data, return -EINVAL; /* reject invalid gem domains */ - if (args->in.domains & ~(AMDGPU_GEM_DOMAIN_CPU | - AMDGPU_GEM_DOMAIN_GTT | - AMDGPU_GEM_DOMAIN_VRAM | - AMDGPU_GEM_DOMAIN_GDS | - AMDGPU_GEM_DOMAIN_GWS | - AMDGPU_GEM_DOMAIN_OA)) + if (args->in.domains & ~AMDGPU_GEM_DOMAIN_MASK) return -EINVAL; /* create a gem object to contain this object in */ if (args->in.domains & (AMDGPU_GEM_DOMAIN_GDS | AMDGPU_GEM_DOMAIN_GWS | AMDGPU_GEM_DOMAIN_OA)) { + if (flags & AMDGPU_GEM_CREATE_VM_ALWAYS_VALID) { + /* if gds bo is created from user space, it must be + * passed to bo list + */ + DRM_ERROR("GDS bo cannot be per-vm-bo\n"); + return -EINVAL; + } flags |= AMDGPU_GEM_CREATE_NO_CPU_ACCESS; if (args->in.domains == AMDGPU_GEM_DOMAIN_GDS) size = size << AMDGPU_GDS_SHIFT; @@ -746,15 +757,16 @@ int amdgpu_mode_dumb_create(struct drm_file *file_priv, struct amdgpu_device *adev = dev->dev_private; struct drm_gem_object *gobj; uint32_t handle; + u32 domain; int r; args->pitch = amdgpu_align_pitch(adev, args->width, DIV_ROUND_UP(args->bpp, 8), 0); args->size = (u64)args->pitch * args->height; args->size = ALIGN(args->size, PAGE_SIZE); - - r = amdgpu_gem_object_create(adev, args->size, 0, - AMDGPU_GEM_DOMAIN_VRAM, + domain = amdgpu_bo_get_preferred_pin_domain(adev, + amdgpu_display_supported_domains(adev)); + r = amdgpu_gem_object_create(adev, args->size, 0, domain, AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED, false, NULL, &gobj); if (r) @@ -771,16 +783,23 @@ int amdgpu_mode_dumb_create(struct drm_file *file_priv, } #if defined(CONFIG_DEBUG_FS) + +#define amdgpu_debugfs_gem_bo_print_flag(m, bo, flag) \ + if (bo->flags & (AMDGPU_GEM_CREATE_ ## flag)) { \ + seq_printf((m), " " #flag); \ + } + static int amdgpu_debugfs_gem_bo_info(int id, void *ptr, void *data) { struct drm_gem_object *gobj = ptr; struct amdgpu_bo *bo = gem_to_amdgpu_bo(gobj); struct seq_file *m = data; + struct dma_buf_attachment *attachment; + struct dma_buf *dma_buf; unsigned domain; const char *placement; unsigned pin_count; - uint64_t offset; domain = amdgpu_mem_type_to_domain(bo->tbo.mem.mem_type); switch (domain) { @@ -798,13 +817,27 @@ static int amdgpu_debugfs_gem_bo_info(int id, void *ptr, void *data) seq_printf(m, "\t0x%08x: %12ld byte %s", id, amdgpu_bo_size(bo), placement); - offset = READ_ONCE(bo->tbo.mem.start); - if (offset != AMDGPU_BO_INVALID_OFFSET) - seq_printf(m, " @ 0x%010Lx", offset); - pin_count = READ_ONCE(bo->pin_count); if (pin_count) seq_printf(m, " pin count %d", pin_count); + + dma_buf = READ_ONCE(bo->gem_base.dma_buf); + attachment = READ_ONCE(bo->gem_base.import_attach); + + if (attachment) + seq_printf(m, " imported from %p", dma_buf); + else if (dma_buf) + seq_printf(m, " exported as %p", dma_buf); + + amdgpu_debugfs_gem_bo_print_flag(m, bo, CPU_ACCESS_REQUIRED); + amdgpu_debugfs_gem_bo_print_flag(m, bo, NO_CPU_ACCESS); + amdgpu_debugfs_gem_bo_print_flag(m, bo, CPU_GTT_USWC); + amdgpu_debugfs_gem_bo_print_flag(m, bo, VRAM_CLEARED); + amdgpu_debugfs_gem_bo_print_flag(m, bo, SHADOW); + amdgpu_debugfs_gem_bo_print_flag(m, bo, VRAM_CONTIGUOUS); + amdgpu_debugfs_gem_bo_print_flag(m, bo, VM_ALWAYS_VALID); + amdgpu_debugfs_gem_bo_print_flag(m, bo, EXPLICIT_SYNC); + seq_printf(m, "\n"); return 0; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c index 311589e02d17..f70eeed9ed76 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c @@ -127,6 +127,7 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned num_ibs, struct amdgpu_vm *vm; uint64_t fence_ctx; uint32_t status = 0, alloc_size; + unsigned fence_flags = 0; unsigned i; int r = 0; @@ -227,7 +228,10 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned num_ibs, #endif amdgpu_asic_invalidate_hdp(adev, ring); - r = amdgpu_fence_emit(ring, f); + if (ib->flags & AMDGPU_IB_FLAG_TC_WB_NOT_INVALIDATE) + fence_flags |= AMDGPU_FENCE_FLAG_TC_WB_ONLY; + + r = amdgpu_fence_emit(ring, f, fence_flags); if (r) { dev_err(adev->dev, "failed to emit fence (%d)\n", r); if (job && job->vmid) @@ -242,7 +246,7 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned num_ibs, /* wrap the last IB with fence */ if (job && job->uf_addr) { amdgpu_ring_emit_fence(ring, job->uf_addr, job->uf_sequence, - AMDGPU_FENCE_FLAG_64BIT); + fence_flags | AMDGPU_FENCE_FLAG_64BIT); } if (patch_offset != ~0 && ring->funcs->patch_cond_exec) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c index 4b7824d30e73..91517b166a3b 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c @@ -31,6 +31,7 @@ #include "amdgpu_sched.h" #include "amdgpu_uvd.h" #include "amdgpu_vce.h" +#include "atom.h" #include #include @@ -214,6 +215,18 @@ static int amdgpu_firmware_info(struct drm_amdgpu_info_firmware *fw_info, fw_info->ver = adev->gfx.rlc_fw_version; fw_info->feature = adev->gfx.rlc_feature_version; break; + case AMDGPU_INFO_FW_GFX_RLC_RESTORE_LIST_CNTL: + fw_info->ver = adev->gfx.rlc_srlc_fw_version; + fw_info->feature = adev->gfx.rlc_srlc_feature_version; + break; + case AMDGPU_INFO_FW_GFX_RLC_RESTORE_LIST_GPM_MEM: + fw_info->ver = adev->gfx.rlc_srlg_fw_version; + fw_info->feature = adev->gfx.rlc_srlg_feature_version; + break; + case AMDGPU_INFO_FW_GFX_RLC_RESTORE_LIST_SRM_MEM: + fw_info->ver = adev->gfx.rlc_srls_fw_version; + fw_info->feature = adev->gfx.rlc_srls_feature_version; + break; case AMDGPU_INFO_FW_GFX_MEC: if (query_fw->index == 0) { fw_info->ver = adev->gfx.mec_fw_version; @@ -273,12 +286,15 @@ static int amdgpu_info_ioctl(struct drm_device *dev, void *data, struct drm_file struct drm_crtc *crtc; uint32_t ui32 = 0; uint64_t ui64 = 0; - int i, found; + int i, j, found; int ui32_size = sizeof(ui32); if (!info->return_size || !info->return_pointer) return -EINVAL; + /* Ensure IB tests are run on ring */ + flush_delayed_work(&adev->late_init_work); + switch (info->query) { case AMDGPU_INFO_ACCEL_WORKING: ui32 = adev->accel_working; @@ -332,7 +348,8 @@ static int amdgpu_info_ioctl(struct drm_device *dev, void *data, struct drm_file break; case AMDGPU_HW_IP_UVD: type = AMD_IP_BLOCK_TYPE_UVD; - ring_mask = adev->uvd.ring.ready ? 1 : 0; + for (i = 0; i < adev->uvd.num_uvd_inst; i++) + ring_mask |= ((adev->uvd.inst[i].ring.ready ? 1 : 0) << i); ib_start_alignment = AMDGPU_GPU_PAGE_SIZE; ib_size_alignment = 16; break; @@ -345,8 +362,11 @@ static int amdgpu_info_ioctl(struct drm_device *dev, void *data, struct drm_file break; case AMDGPU_HW_IP_UVD_ENC: type = AMD_IP_BLOCK_TYPE_UVD; - for (i = 0; i < adev->uvd.num_enc_rings; i++) - ring_mask |= ((adev->uvd.ring_enc[i].ready ? 1 : 0) << i); + for (i = 0; i < adev->uvd.num_uvd_inst; i++) + for (j = 0; j < adev->uvd.num_enc_rings; j++) + ring_mask |= + ((adev->uvd.inst[i].ring_enc[j].ready ? 1 : 0) << + (j + i * adev->uvd.num_enc_rings)); ib_start_alignment = AMDGPU_GPU_PAGE_SIZE; ib_size_alignment = 1; break; @@ -701,10 +721,7 @@ static int amdgpu_info_ioctl(struct drm_device *dev, void *data, struct drm_file } } case AMDGPU_INFO_SENSOR: { - struct pp_gpu_power query = {0}; - int query_size = sizeof(query); - - if (amdgpu_dpm == 0) + if (!adev->pm.dpm_enabled) return -ENOENT; switch (info->sensor_info.type) { @@ -746,10 +763,10 @@ static int amdgpu_info_ioctl(struct drm_device *dev, void *data, struct drm_file /* get average GPU power */ if (amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_GPU_POWER, - (void *)&query, &query_size)) { + (void *)&ui32, &ui32_size)) { return -EINVAL; } - ui32 = query.average_gpu_power >> 8; + ui32 >>= 8; break; case AMDGPU_INFO_SENSOR_VDDNB: /* get VDDNB in millivolts */ @@ -913,8 +930,7 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev, return; pm_runtime_get_sync(dev->dev); - - amdgpu_ctx_mgr_fini(&fpriv->ctx_mgr); + amdgpu_ctx_mgr_entity_fini(&fpriv->ctx_mgr); if (adev->asic_type != CHIP_RAVEN) { amdgpu_uvd_free_handles(adev, file_priv); @@ -935,6 +951,8 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev, pd = amdgpu_bo_ref(fpriv->vm.root.base.bo); amdgpu_vm_fini(adev, &fpriv->vm); + amdgpu_ctx_mgr_fini(&fpriv->ctx_mgr); + if (pasid) amdgpu_pasid_free_delayed(pd->tbo.resv, pasid); amdgpu_bo_unref(&pd); @@ -1088,6 +1106,7 @@ static int amdgpu_debugfs_firmware_info(struct seq_file *m, void *data) struct amdgpu_device *adev = dev->dev_private; struct drm_amdgpu_info_firmware fw_info; struct drm_amdgpu_query_fw query_fw; + struct atom_context *ctx = adev->mode_info.atom_context; int ret, i; /* VCE */ @@ -1146,6 +1165,30 @@ static int amdgpu_debugfs_firmware_info(struct seq_file *m, void *data) seq_printf(m, "RLC feature version: %u, firmware version: 0x%08x\n", fw_info.feature, fw_info.ver); + /* RLC SAVE RESTORE LIST CNTL */ + query_fw.fw_type = AMDGPU_INFO_FW_GFX_RLC_RESTORE_LIST_CNTL; + ret = amdgpu_firmware_info(&fw_info, &query_fw, adev); + if (ret) + return ret; + seq_printf(m, "RLC SRLC feature version: %u, firmware version: 0x%08x\n", + fw_info.feature, fw_info.ver); + + /* RLC SAVE RESTORE LIST GPM MEM */ + query_fw.fw_type = AMDGPU_INFO_FW_GFX_RLC_RESTORE_LIST_GPM_MEM; + ret = amdgpu_firmware_info(&fw_info, &query_fw, adev); + if (ret) + return ret; + seq_printf(m, "RLC SRLG feature version: %u, firmware version: 0x%08x\n", + fw_info.feature, fw_info.ver); + + /* RLC SAVE RESTORE LIST SRM MEM */ + query_fw.fw_type = AMDGPU_INFO_FW_GFX_RLC_RESTORE_LIST_SRM_MEM; + ret = amdgpu_firmware_info(&fw_info, &query_fw, adev); + if (ret) + return ret; + seq_printf(m, "RLC SRLS feature version: %u, firmware version: 0x%08x\n", + fw_info.feature, fw_info.ver); + /* MEC */ query_fw.fw_type = AMDGPU_INFO_FW_GFX_MEC; query_fw.index = 0; @@ -1210,6 +1253,9 @@ static int amdgpu_debugfs_firmware_info(struct seq_file *m, void *data) seq_printf(m, "VCN feature version: %u, firmware version: 0x%08x\n", fw_info.feature, fw_info.ver); + + seq_printf(m, "VBIOS version: %s\n", ctx->vbios_version); + return 0; } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c index bd67f4cb8e6c..83e344fbb50a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c @@ -36,12 +36,14 @@ #include #include "amdgpu.h" +#include "amdgpu_amdkfd.h" struct amdgpu_mn { /* constant after initialisation */ struct amdgpu_device *adev; struct mm_struct *mm; struct mmu_notifier mn; + enum amdgpu_mn_type type; /* only used on destruction */ struct work_struct work; @@ -185,7 +187,7 @@ static void amdgpu_mn_invalidate_node(struct amdgpu_mn_node *node, } /** - * amdgpu_mn_invalidate_range_start - callback to notify about mm change + * amdgpu_mn_invalidate_range_start_gfx - callback to notify about mm change * * @mn: our notifier * @mn: the mm this callback is about @@ -195,10 +197,10 @@ static void amdgpu_mn_invalidate_node(struct amdgpu_mn_node *node, * We block for all BOs between start and end to be idle and * unmap them by move them into system domain again. */ -static void amdgpu_mn_invalidate_range_start(struct mmu_notifier *mn, - struct mm_struct *mm, - unsigned long start, - unsigned long end) +static void amdgpu_mn_invalidate_range_start_gfx(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, + unsigned long end) { struct amdgpu_mn *rmn = container_of(mn, struct amdgpu_mn, mn); struct interval_tree_node *it; @@ -219,6 +221,49 @@ static void amdgpu_mn_invalidate_range_start(struct mmu_notifier *mn, } } +/** + * amdgpu_mn_invalidate_range_start_hsa - callback to notify about mm change + * + * @mn: our notifier + * @mn: the mm this callback is about + * @start: start of updated range + * @end: end of updated range + * + * We temporarily evict all BOs between start and end. This + * necessitates evicting all user-mode queues of the process. The BOs + * are restorted in amdgpu_mn_invalidate_range_end_hsa. + */ +static void amdgpu_mn_invalidate_range_start_hsa(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + struct amdgpu_mn *rmn = container_of(mn, struct amdgpu_mn, mn); + struct interval_tree_node *it; + + /* notification is exclusive, but interval is inclusive */ + end -= 1; + + amdgpu_mn_read_lock(rmn); + + it = interval_tree_iter_first(&rmn->objects, start, end); + while (it) { + struct amdgpu_mn_node *node; + struct amdgpu_bo *bo; + + node = container_of(it, struct amdgpu_mn_node, it); + it = interval_tree_iter_next(it, start, end); + + list_for_each_entry(bo, &node->bos, mn_list) { + struct kgd_mem *mem = bo->kfd_bo; + + if (amdgpu_ttm_tt_affect_userptr(bo->tbo.ttm, + start, end)) + amdgpu_amdkfd_evict_userptr(mem, mm); + } + } +} + /** * amdgpu_mn_invalidate_range_end - callback to notify about mm change * @@ -239,23 +284,39 @@ static void amdgpu_mn_invalidate_range_end(struct mmu_notifier *mn, amdgpu_mn_read_unlock(rmn); } -static const struct mmu_notifier_ops amdgpu_mn_ops = { - .release = amdgpu_mn_release, - .invalidate_range_start = amdgpu_mn_invalidate_range_start, - .invalidate_range_end = amdgpu_mn_invalidate_range_end, +static const struct mmu_notifier_ops amdgpu_mn_ops[] = { + [AMDGPU_MN_TYPE_GFX] = { + .release = amdgpu_mn_release, + .invalidate_range_start = amdgpu_mn_invalidate_range_start_gfx, + .invalidate_range_end = amdgpu_mn_invalidate_range_end, + }, + [AMDGPU_MN_TYPE_HSA] = { + .release = amdgpu_mn_release, + .invalidate_range_start = amdgpu_mn_invalidate_range_start_hsa, + .invalidate_range_end = amdgpu_mn_invalidate_range_end, + }, }; +/* Low bits of any reasonable mm pointer will be unused due to struct + * alignment. Use these bits to make a unique key from the mm pointer + * and notifier type. + */ +#define AMDGPU_MN_KEY(mm, type) ((unsigned long)(mm) + (type)) + /** * amdgpu_mn_get - create notifier context * * @adev: amdgpu device pointer + * @type: type of MMU notifier context * * Creates a notifier context for current->mm. */ -struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev) +struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, + enum amdgpu_mn_type type) { struct mm_struct *mm = current->mm; struct amdgpu_mn *rmn; + unsigned long key = AMDGPU_MN_KEY(mm, type); int r; mutex_lock(&adev->mn_lock); @@ -264,8 +325,8 @@ struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev) return ERR_PTR(-EINTR); } - hash_for_each_possible(adev->mn_hash, rmn, node, (unsigned long)mm) - if (rmn->mm == mm) + hash_for_each_possible(adev->mn_hash, rmn, node, key) + if (AMDGPU_MN_KEY(rmn->mm, rmn->type) == key) goto release_locks; rmn = kzalloc(sizeof(*rmn), GFP_KERNEL); @@ -276,8 +337,9 @@ struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev) rmn->adev = adev; rmn->mm = mm; - rmn->mn.ops = &amdgpu_mn_ops; init_rwsem(&rmn->lock); + rmn->type = type; + rmn->mn.ops = &amdgpu_mn_ops[type]; rmn->objects = RB_ROOT_CACHED; mutex_init(&rmn->read_lock); atomic_set(&rmn->recursion, 0); @@ -286,7 +348,7 @@ struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev) if (r) goto free_rmn; - hash_add(adev->mn_hash, &rmn->node, (unsigned long)mm); + hash_add(adev->mn_hash, &rmn->node, AMDGPU_MN_KEY(mm, type)); release_locks: up_write(&mm->mmap_sem); @@ -315,15 +377,21 @@ int amdgpu_mn_register(struct amdgpu_bo *bo, unsigned long addr) { unsigned long end = addr + amdgpu_bo_size(bo) - 1; struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev); + enum amdgpu_mn_type type = + bo->kfd_bo ? AMDGPU_MN_TYPE_HSA : AMDGPU_MN_TYPE_GFX; struct amdgpu_mn *rmn; - struct amdgpu_mn_node *node = NULL; + struct amdgpu_mn_node *node = NULL, *new_node; struct list_head bos; struct interval_tree_node *it; - rmn = amdgpu_mn_get(adev); + rmn = amdgpu_mn_get(adev, type); if (IS_ERR(rmn)) return PTR_ERR(rmn); + new_node = kmalloc(sizeof(*new_node), GFP_KERNEL); + if (!new_node) + return -ENOMEM; + INIT_LIST_HEAD(&bos); down_write(&rmn->lock); @@ -337,13 +405,10 @@ int amdgpu_mn_register(struct amdgpu_bo *bo, unsigned long addr) list_splice(&node->bos, &bos); } - if (!node) { - node = kmalloc(sizeof(struct amdgpu_mn_node), GFP_KERNEL); - if (!node) { - up_write(&rmn->lock); - return -ENOMEM; - } - } + if (!node) + node = new_node; + else + kfree(new_node); bo->mn = rmn; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h index d0095a3793b8..eb0f432f78fe 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h @@ -29,16 +29,23 @@ */ struct amdgpu_mn; +enum amdgpu_mn_type { + AMDGPU_MN_TYPE_GFX, + AMDGPU_MN_TYPE_HSA, +}; + #if defined(CONFIG_MMU_NOTIFIER) void amdgpu_mn_lock(struct amdgpu_mn *mn); void amdgpu_mn_unlock(struct amdgpu_mn *mn); -struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev); +struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, + enum amdgpu_mn_type type); int amdgpu_mn_register(struct amdgpu_bo *bo, unsigned long addr); void amdgpu_mn_unregister(struct amdgpu_bo *bo); #else static inline void amdgpu_mn_lock(struct amdgpu_mn *mn) {} static inline void amdgpu_mn_unlock(struct amdgpu_mn *mn) {} -static inline struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev) +static inline struct amdgpu_mn *amdgpu_mn_get(struct amdgpu_device *adev, + enum amdgpu_mn_type type) { return NULL; } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mode.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_mode.h index d6416ee52e32..b9e9e8b02fb7 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mode.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mode.h @@ -308,7 +308,6 @@ struct amdgpu_display_funcs { struct amdgpu_framebuffer { struct drm_framebuffer base; - struct drm_gem_object *obj; /* caching for later use */ uint64_t address; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c index 6d08cde8443c..5e4e1bd90383 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c @@ -191,14 +191,21 @@ int amdgpu_bo_create_reserved(struct amdgpu_device *adev, u32 domain, struct amdgpu_bo **bo_ptr, u64 *gpu_addr, void **cpu_addr) { + struct amdgpu_bo_param bp; bool free = false; int r; + memset(&bp, 0, sizeof(bp)); + bp.size = size; + bp.byte_align = align; + bp.domain = domain; + bp.flags = AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED | + AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS; + bp.type = ttm_bo_type_kernel; + bp.resv = NULL; + if (!*bo_ptr) { - r = amdgpu_bo_create(adev, size, align, domain, - AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED | - AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS, - ttm_bo_type_kernel, NULL, bo_ptr); + r = amdgpu_bo_create(adev, &bp, bo_ptr); if (r) { dev_err(adev->dev, "(%d) failed to allocate kernel bo\n", r); @@ -341,27 +348,25 @@ fail: return false; } -static int amdgpu_bo_do_create(struct amdgpu_device *adev, unsigned long size, - int byte_align, u32 domain, - u64 flags, enum ttm_bo_type type, - struct reservation_object *resv, +static int amdgpu_bo_do_create(struct amdgpu_device *adev, + struct amdgpu_bo_param *bp, struct amdgpu_bo **bo_ptr) { struct ttm_operation_ctx ctx = { - .interruptible = (type != ttm_bo_type_kernel), + .interruptible = (bp->type != ttm_bo_type_kernel), .no_wait_gpu = false, - .resv = resv, + .resv = bp->resv, .flags = TTM_OPT_FLAG_ALLOW_RES_EVICT }; struct amdgpu_bo *bo; - unsigned long page_align; + unsigned long page_align, size = bp->size; size_t acc_size; int r; - page_align = roundup(byte_align, PAGE_SIZE) >> PAGE_SHIFT; + page_align = roundup(bp->byte_align, PAGE_SIZE) >> PAGE_SHIFT; size = ALIGN(size, PAGE_SIZE); - if (!amdgpu_bo_validate_size(adev, size, domain)) + if (!amdgpu_bo_validate_size(adev, size, bp->domain)) return -ENOMEM; *bo_ptr = NULL; @@ -375,18 +380,14 @@ static int amdgpu_bo_do_create(struct amdgpu_device *adev, unsigned long size, drm_gem_private_object_init(adev->ddev, &bo->gem_base, size); INIT_LIST_HEAD(&bo->shadow_list); INIT_LIST_HEAD(&bo->va); - bo->preferred_domains = domain & (AMDGPU_GEM_DOMAIN_VRAM | - AMDGPU_GEM_DOMAIN_GTT | - AMDGPU_GEM_DOMAIN_CPU | - AMDGPU_GEM_DOMAIN_GDS | - AMDGPU_GEM_DOMAIN_GWS | - AMDGPU_GEM_DOMAIN_OA); + bo->preferred_domains = bp->preferred_domain ? bp->preferred_domain : + bp->domain; bo->allowed_domains = bo->preferred_domains; - if (type != ttm_bo_type_kernel && + if (bp->type != ttm_bo_type_kernel && bo->allowed_domains == AMDGPU_GEM_DOMAIN_VRAM) bo->allowed_domains |= AMDGPU_GEM_DOMAIN_GTT; - bo->flags = flags; + bo->flags = bp->flags; #ifdef CONFIG_X86_32 /* XXX: Write-combined CPU mappings of GTT seem broken on 32-bit @@ -417,11 +418,13 @@ static int amdgpu_bo_do_create(struct amdgpu_device *adev, unsigned long size, #endif bo->tbo.bdev = &adev->mman.bdev; - amdgpu_ttm_placement_from_domain(bo, domain); + amdgpu_ttm_placement_from_domain(bo, bp->domain); + if (bp->type == ttm_bo_type_kernel) + bo->tbo.priority = 1; - r = ttm_bo_init_reserved(&adev->mman.bdev, &bo->tbo, size, type, + r = ttm_bo_init_reserved(&adev->mman.bdev, &bo->tbo, size, bp->type, &bo->placement, page_align, &ctx, acc_size, - NULL, resv, &amdgpu_ttm_bo_destroy); + NULL, bp->resv, &amdgpu_ttm_bo_destroy); if (unlikely(r != 0)) return r; @@ -433,10 +436,7 @@ static int amdgpu_bo_do_create(struct amdgpu_device *adev, unsigned long size, else amdgpu_cs_report_moved_bytes(adev, ctx.bytes_moved, 0); - if (type == ttm_bo_type_kernel) - bo->tbo.priority = 1; - - if (flags & AMDGPU_GEM_CREATE_VRAM_CLEARED && + if (bp->flags & AMDGPU_GEM_CREATE_VRAM_CLEARED && bo->tbo.mem.placement & TTM_PL_FLAG_VRAM) { struct dma_fence *fence; @@ -449,20 +449,20 @@ static int amdgpu_bo_do_create(struct amdgpu_device *adev, unsigned long size, bo->tbo.moving = dma_fence_get(fence); dma_fence_put(fence); } - if (!resv) + if (!bp->resv) amdgpu_bo_unreserve(bo); *bo_ptr = bo; trace_amdgpu_bo_create(bo); /* Treat CPU_ACCESS_REQUIRED only as a hint if given by UMD */ - if (type == ttm_bo_type_device) + if (bp->type == ttm_bo_type_device) bo->flags &= ~AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED; return 0; fail_unreserve: - if (!resv) + if (!bp->resv) ww_mutex_unlock(&bo->tbo.resv->lock); amdgpu_bo_unref(&bo); return r; @@ -472,16 +472,22 @@ static int amdgpu_bo_create_shadow(struct amdgpu_device *adev, unsigned long size, int byte_align, struct amdgpu_bo *bo) { + struct amdgpu_bo_param bp; int r; if (bo->shadow) return 0; - r = amdgpu_bo_do_create(adev, size, byte_align, AMDGPU_GEM_DOMAIN_GTT, - AMDGPU_GEM_CREATE_CPU_GTT_USWC | - AMDGPU_GEM_CREATE_SHADOW, - ttm_bo_type_kernel, - bo->tbo.resv, &bo->shadow); + memset(&bp, 0, sizeof(bp)); + bp.size = size; + bp.byte_align = byte_align; + bp.domain = AMDGPU_GEM_DOMAIN_GTT; + bp.flags = AMDGPU_GEM_CREATE_CPU_GTT_USWC | + AMDGPU_GEM_CREATE_SHADOW; + bp.type = ttm_bo_type_kernel; + bp.resv = bo->tbo.resv; + + r = amdgpu_bo_do_create(adev, &bp, &bo->shadow); if (!r) { bo->shadow->parent = amdgpu_bo_ref(bo); mutex_lock(&adev->shadow_list_lock); @@ -492,28 +498,26 @@ static int amdgpu_bo_create_shadow(struct amdgpu_device *adev, return r; } -int amdgpu_bo_create(struct amdgpu_device *adev, unsigned long size, - int byte_align, u32 domain, - u64 flags, enum ttm_bo_type type, - struct reservation_object *resv, +int amdgpu_bo_create(struct amdgpu_device *adev, + struct amdgpu_bo_param *bp, struct amdgpu_bo **bo_ptr) { - uint64_t parent_flags = flags & ~AMDGPU_GEM_CREATE_SHADOW; + u64 flags = bp->flags; int r; - r = amdgpu_bo_do_create(adev, size, byte_align, domain, - parent_flags, type, resv, bo_ptr); + bp->flags = bp->flags & ~AMDGPU_GEM_CREATE_SHADOW; + r = amdgpu_bo_do_create(adev, bp, bo_ptr); if (r) return r; if ((flags & AMDGPU_GEM_CREATE_SHADOW) && amdgpu_need_backup(adev)) { - if (!resv) + if (!bp->resv) WARN_ON(reservation_object_lock((*bo_ptr)->tbo.resv, NULL)); - r = amdgpu_bo_create_shadow(adev, size, byte_align, (*bo_ptr)); + r = amdgpu_bo_create_shadow(adev, bp->size, bp->byte_align, (*bo_ptr)); - if (!resv) + if (!bp->resv) reservation_object_unlock((*bo_ptr)->tbo.resv); if (r) @@ -689,8 +693,17 @@ int amdgpu_bo_pin_restricted(struct amdgpu_bo *bo, u32 domain, return -EINVAL; /* A shared bo cannot be migrated to VRAM */ - if (bo->prime_shared_count && (domain == AMDGPU_GEM_DOMAIN_VRAM)) - return -EINVAL; + if (bo->prime_shared_count) { + if (domain & AMDGPU_GEM_DOMAIN_GTT) + domain = AMDGPU_GEM_DOMAIN_GTT; + else + return -EINVAL; + } + + /* This assumes only APU display buffers are pinned with (VRAM|GTT). + * See function amdgpu_display_supported_domains() + */ + domain = amdgpu_bo_get_preferred_pin_domain(adev, domain); if (bo->pin_count) { uint32_t mem_type = bo->tbo.mem.mem_type; @@ -838,6 +851,13 @@ int amdgpu_bo_init(struct amdgpu_device *adev) return amdgpu_ttm_init(adev); } +int amdgpu_bo_late_init(struct amdgpu_device *adev) +{ + amdgpu_ttm_late_init(adev); + + return 0; +} + void amdgpu_bo_fini(struct amdgpu_device *adev) { amdgpu_ttm_fini(adev); @@ -1042,3 +1062,14 @@ u64 amdgpu_bo_gpu_offset(struct amdgpu_bo *bo) return bo->tbo.offset; } + +uint32_t amdgpu_bo_get_preferred_pin_domain(struct amdgpu_device *adev, + uint32_t domain) +{ + if (domain == (AMDGPU_GEM_DOMAIN_VRAM | AMDGPU_GEM_DOMAIN_GTT)) { + domain = AMDGPU_GEM_DOMAIN_VRAM; + if (adev->gmc.real_vram_size <= AMDGPU_SG_THRESHOLD) + domain = AMDGPU_GEM_DOMAIN_GTT; + } + return domain; +} diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h index 546f77cb7882..731748033878 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h @@ -33,6 +33,16 @@ #define AMDGPU_BO_INVALID_OFFSET LONG_MAX +struct amdgpu_bo_param { + unsigned long size; + int byte_align; + u32 domain; + u32 preferred_domain; + u64 flags; + enum ttm_bo_type type; + struct reservation_object *resv; +}; + /* bo virtual addresses in a vm */ struct amdgpu_bo_va_mapping { struct amdgpu_bo_va *bo_va; @@ -195,6 +205,27 @@ static inline bool amdgpu_bo_gpu_accessible(struct amdgpu_bo *bo) } } +/** + * amdgpu_bo_in_cpu_visible_vram - check if BO is (partly) in visible VRAM + */ +static inline bool amdgpu_bo_in_cpu_visible_vram(struct amdgpu_bo *bo) +{ + struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev); + unsigned fpfn = adev->gmc.visible_vram_size >> PAGE_SHIFT; + struct drm_mm_node *node = bo->tbo.mem.mm_node; + unsigned long pages_left; + + if (bo->tbo.mem.mem_type != TTM_PL_VRAM) + return false; + + for (pages_left = bo->tbo.mem.num_pages; pages_left; + pages_left -= node->size, node++) + if (node->start < fpfn) + return true; + + return false; +} + /** * amdgpu_bo_explicit_sync - return whether the bo is explicitly synced */ @@ -203,10 +234,8 @@ static inline bool amdgpu_bo_explicit_sync(struct amdgpu_bo *bo) return bo->flags & AMDGPU_GEM_CREATE_EXPLICIT_SYNC; } -int amdgpu_bo_create(struct amdgpu_device *adev, unsigned long size, - int byte_align, u32 domain, - u64 flags, enum ttm_bo_type type, - struct reservation_object *resv, +int amdgpu_bo_create(struct amdgpu_device *adev, + struct amdgpu_bo_param *bp, struct amdgpu_bo **bo_ptr); int amdgpu_bo_create_reserved(struct amdgpu_device *adev, unsigned long size, int align, @@ -230,6 +259,7 @@ int amdgpu_bo_pin_restricted(struct amdgpu_bo *bo, u32 domain, int amdgpu_bo_unpin(struct amdgpu_bo *bo); int amdgpu_bo_evict_vram(struct amdgpu_device *adev); int amdgpu_bo_init(struct amdgpu_device *adev); +int amdgpu_bo_late_init(struct amdgpu_device *adev); void amdgpu_bo_fini(struct amdgpu_device *adev); int amdgpu_bo_fbdev_mmap(struct amdgpu_bo *bo, struct vm_area_struct *vma); @@ -259,7 +289,8 @@ int amdgpu_bo_restore_from_shadow(struct amdgpu_device *adev, struct reservation_object *resv, struct dma_fence **fence, bool direct); - +uint32_t amdgpu_bo_get_preferred_pin_domain(struct amdgpu_device *adev, + uint32_t domain); /* * sub allocation diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_pm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_pm.c index 361975cf45a9..b455da487782 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_pm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_pm.c @@ -77,6 +77,37 @@ void amdgpu_pm_acpi_event_handler(struct amdgpu_device *adev) } } +/** + * DOC: power_dpm_state + * + * This is a legacy interface and is only provided for backwards compatibility. + * The amdgpu driver provides a sysfs API for adjusting certain power + * related parameters. The file power_dpm_state is used for this. + * It accepts the following arguments: + * - battery + * - balanced + * - performance + * + * battery + * + * On older GPUs, the vbios provided a special power state for battery + * operation. Selecting battery switched to this state. This is no + * longer provided on newer GPUs so the option does nothing in that case. + * + * balanced + * + * On older GPUs, the vbios provided a special power state for balanced + * operation. Selecting balanced switched to this state. This is no + * longer provided on newer GPUs so the option does nothing in that case. + * + * performance + * + * On older GPUs, the vbios provided a special power state for performance + * operation. Selecting performance switched to this state. This is no + * longer provided on newer GPUs so the option does nothing in that case. + * + */ + static ssize_t amdgpu_get_dpm_state(struct device *dev, struct device_attribute *attr, char *buf) @@ -131,6 +162,59 @@ fail: return count; } + +/** + * DOC: power_dpm_force_performance_level + * + * The amdgpu driver provides a sysfs API for adjusting certain power + * related parameters. The file power_dpm_force_performance_level is + * used for this. It accepts the following arguments: + * - auto + * - low + * - high + * - manual + * - GPU fan + * - profile_standard + * - profile_min_sclk + * - profile_min_mclk + * - profile_peak + * + * auto + * + * When auto is selected, the driver will attempt to dynamically select + * the optimal power profile for current conditions in the driver. + * + * low + * + * When low is selected, the clocks are forced to the lowest power state. + * + * high + * + * When high is selected, the clocks are forced to the highest power state. + * + * manual + * + * When manual is selected, the user can manually adjust which power states + * are enabled for each clock domain via the sysfs pp_dpm_mclk, pp_dpm_sclk, + * and pp_dpm_pcie files and adjust the power state transition heuristics + * via the pp_power_profile_mode sysfs file. + * + * profile_standard + * profile_min_sclk + * profile_min_mclk + * profile_peak + * + * When the profiling modes are selected, clock and power gating are + * disabled and the clocks are set for different profiling cases. This + * mode is recommended for profiling specific work loads where you do + * not want clock or power gating for clock fluctuation to interfere + * with your results. profile_standard sets the clocks to a fixed clock + * level which varies from asic to asic. profile_min_sclk forces the sclk + * to the lowest level. profile_min_mclk forces the mclk to the lowest level. + * profile_peak sets all clocks (mclk, sclk, pcie) to the highest levels. + * + */ + static ssize_t amdgpu_get_dpm_forced_performance_level(struct device *dev, struct device_attribute *attr, char *buf) @@ -324,6 +408,17 @@ fail: return count; } +/** + * DOC: pp_table + * + * The amdgpu driver provides a sysfs API for uploading new powerplay + * tables. The file pp_table is used for this. Reading the file + * will dump the current power play table. Writing to the file + * will attempt to upload a new powerplay table and re-initialize + * powerplay using that new table. + * + */ + static ssize_t amdgpu_get_pp_table(struct device *dev, struct device_attribute *attr, char *buf) @@ -360,6 +455,29 @@ static ssize_t amdgpu_set_pp_table(struct device *dev, return count; } +/** + * DOC: pp_od_clk_voltage + * + * The amdgpu driver provides a sysfs API for adjusting the clocks and voltages + * in each power level within a power state. The pp_od_clk_voltage is used for + * this. + * + * Reading the file will display: + * - a list of engine clock levels and voltages labeled OD_SCLK + * - a list of memory clock levels and voltages labeled OD_MCLK + * - a list of valid ranges for sclk, mclk, and voltage labeled OD_RANGE + * + * To manually adjust these settings, first select manual using + * power_dpm_force_performance_level. Enter a new value for each + * level by writing a string that contains "s/m level clock voltage" to + * the file. E.g., "s 1 500 820" will update sclk level 1 to be 500 MHz + * at 820 mV; "m 0 350 810" will update mclk level 0 to be 350 MHz at + * 810 mV. When you have edited all of the states as needed, write + * "c" (commit) to the file to commit your changes. If you want to reset to the + * default power levels, write "r" (reset) to the file to reset them. + * + */ + static ssize_t amdgpu_set_pp_od_clk_voltage(struct device *dev, struct device_attribute *attr, const char *buf, @@ -437,6 +555,7 @@ static ssize_t amdgpu_get_pp_od_clk_voltage(struct device *dev, if (adev->powerplay.pp_funcs->print_clock_levels) { size = amdgpu_dpm_print_clock_levels(adev, OD_SCLK, buf); size += amdgpu_dpm_print_clock_levels(adev, OD_MCLK, buf+size); + size += amdgpu_dpm_print_clock_levels(adev, OD_RANGE, buf+size); return size; } else { return snprintf(buf, PAGE_SIZE, "\n"); @@ -444,6 +563,23 @@ static ssize_t amdgpu_get_pp_od_clk_voltage(struct device *dev, } +/** + * DOC: pp_dpm_sclk pp_dpm_mclk pp_dpm_pcie + * + * The amdgpu driver provides a sysfs API for adjusting what power levels + * are enabled for a given power state. The files pp_dpm_sclk, pp_dpm_mclk, + * and pp_dpm_pcie are used for this. + * + * Reading back the files will show you the available power levels within + * the power state and the clock information for those levels. + * + * To manually adjust these states, first select manual using + * power_dpm_force_performance_level. + * Secondly,Enter a new value for each level by inputing a string that + * contains " echo xx xx xx > pp_dpm_sclk/mclk/pcie" + * E.g., echo 4 5 6 to > pp_dpm_sclk will enable sclk levels 4, 5, and 6. + */ + static ssize_t amdgpu_get_pp_dpm_sclk(struct device *dev, struct device_attribute *attr, char *buf) @@ -466,23 +602,27 @@ static ssize_t amdgpu_set_pp_dpm_sclk(struct device *dev, struct amdgpu_device *adev = ddev->dev_private; int ret; long level; - uint32_t i, mask = 0; - char sub_str[2]; + uint32_t mask = 0; + char *sub_str = NULL; + char *tmp; + char buf_cpy[count]; + const char delimiter[3] = {' ', '\n', '\0'}; - for (i = 0; i < strlen(buf); i++) { - if (*(buf + i) == '\n') - continue; - sub_str[0] = *(buf + i); - sub_str[1] = '\0'; - ret = kstrtol(sub_str, 0, &level); + memcpy(buf_cpy, buf, count+1); + tmp = buf_cpy; + while (tmp[0]) { + sub_str = strsep(&tmp, delimiter); + if (strlen(sub_str)) { + ret = kstrtol(sub_str, 0, &level); - if (ret) { - count = -EINVAL; - goto fail; - } - mask |= 1 << level; + if (ret) { + count = -EINVAL; + goto fail; + } + mask |= 1 << level; + } else + break; } - if (adev->powerplay.pp_funcs->force_clock_level) amdgpu_dpm_force_clock_level(adev, PP_SCLK, mask); @@ -512,21 +652,26 @@ static ssize_t amdgpu_set_pp_dpm_mclk(struct device *dev, struct amdgpu_device *adev = ddev->dev_private; int ret; long level; - uint32_t i, mask = 0; - char sub_str[2]; + uint32_t mask = 0; + char *sub_str = NULL; + char *tmp; + char buf_cpy[count]; + const char delimiter[3] = {' ', '\n', '\0'}; - for (i = 0; i < strlen(buf); i++) { - if (*(buf + i) == '\n') - continue; - sub_str[0] = *(buf + i); - sub_str[1] = '\0'; - ret = kstrtol(sub_str, 0, &level); + memcpy(buf_cpy, buf, count+1); + tmp = buf_cpy; + while (tmp[0]) { + sub_str = strsep(&tmp, delimiter); + if (strlen(sub_str)) { + ret = kstrtol(sub_str, 0, &level); - if (ret) { - count = -EINVAL; - goto fail; - } - mask |= 1 << level; + if (ret) { + count = -EINVAL; + goto fail; + } + mask |= 1 << level; + } else + break; } if (adev->powerplay.pp_funcs->force_clock_level) amdgpu_dpm_force_clock_level(adev, PP_MCLK, mask); @@ -557,21 +702,27 @@ static ssize_t amdgpu_set_pp_dpm_pcie(struct device *dev, struct amdgpu_device *adev = ddev->dev_private; int ret; long level; - uint32_t i, mask = 0; - char sub_str[2]; + uint32_t mask = 0; + char *sub_str = NULL; + char *tmp; + char buf_cpy[count]; + const char delimiter[3] = {' ', '\n', '\0'}; - for (i = 0; i < strlen(buf); i++) { - if (*(buf + i) == '\n') - continue; - sub_str[0] = *(buf + i); - sub_str[1] = '\0'; - ret = kstrtol(sub_str, 0, &level); + memcpy(buf_cpy, buf, count+1); + tmp = buf_cpy; - if (ret) { - count = -EINVAL; - goto fail; - } - mask |= 1 << level; + while (tmp[0]) { + sub_str = strsep(&tmp, delimiter); + if (strlen(sub_str)) { + ret = kstrtol(sub_str, 0, &level); + + if (ret) { + count = -EINVAL; + goto fail; + } + mask |= 1 << level; + } else + break; } if (adev->powerplay.pp_funcs->force_clock_level) amdgpu_dpm_force_clock_level(adev, PP_PCIE, mask); @@ -668,6 +819,26 @@ fail: return count; } +/** + * DOC: pp_power_profile_mode + * + * The amdgpu driver provides a sysfs API for adjusting the heuristics + * related to switching between power levels in a power state. The file + * pp_power_profile_mode is used for this. + * + * Reading this file outputs a list of all of the predefined power profiles + * and the relevant heuristics settings for that profile. + * + * To select a profile or create a custom profile, first select manual using + * power_dpm_force_performance_level. Writing the number of a predefined + * profile to pp_power_profile_mode will enable those heuristics. To + * create a custom set of heuristics, write a string of numbers to the file + * starting with the number of the custom profile along with a setting + * for each heuristic parameter. Due to differences across asic families + * the heuristic parameters vary from family to family. + * + */ + static ssize_t amdgpu_get_pp_power_profile_mode(struct device *dev, struct device_attribute *attr, char *buf) @@ -1020,8 +1191,8 @@ static ssize_t amdgpu_hwmon_show_power_avg(struct device *dev, { struct amdgpu_device *adev = dev_get_drvdata(dev); struct drm_device *ddev = adev->ddev; - struct pp_gpu_power query = {0}; - int r, size = sizeof(query); + u32 query = 0; + int r, size = sizeof(u32); unsigned uw; /* Can't get power when the card is off */ @@ -1041,7 +1212,7 @@ static ssize_t amdgpu_hwmon_show_power_avg(struct device *dev, return r; /* convert to microwatts */ - uw = (query.average_gpu_power >> 8) * 1000000; + uw = (query >> 8) * 1000000 + (query & 0xff) * 1000; return snprintf(buf, PAGE_SIZE, "%u\n", uw); } @@ -1109,6 +1280,46 @@ static ssize_t amdgpu_hwmon_set_power_cap(struct device *dev, return count; } + +/** + * DOC: hwmon + * + * The amdgpu driver exposes the following sensor interfaces: + * - GPU temperature (via the on-die sensor) + * - GPU voltage + * - Northbridge voltage (APUs only) + * - GPU power + * - GPU fan + * + * hwmon interfaces for GPU temperature: + * - temp1_input: the on die GPU temperature in millidegrees Celsius + * - temp1_crit: temperature critical max value in millidegrees Celsius + * - temp1_crit_hyst: temperature hysteresis for critical limit in millidegrees Celsius + * + * hwmon interfaces for GPU voltage: + * - in0_input: the voltage on the GPU in millivolts + * - in1_input: the voltage on the Northbridge in millivolts + * + * hwmon interfaces for GPU power: + * - power1_average: average power used by the GPU in microWatts + * - power1_cap_min: minimum cap supported in microWatts + * - power1_cap_max: maximum cap supported in microWatts + * - power1_cap: selected power cap in microWatts + * + * hwmon interfaces for GPU fan: + * - pwm1: pulse width modulation fan level (0-255) + * - pwm1_enable: pulse width modulation fan control method + * 0: no fan speed control + * 1: manual fan speed control using pwm interface + * 2: automatic fan speed control + * - pwm1_min: pulse width modulation fan control minimum level (0) + * - pwm1_max: pulse width modulation fan control maximum level (255) + * - fan1_input: fan speed in RPM + * + * You can use hwmon tools like sensors to view this information on your system. + * + */ + static SENSOR_DEVICE_ATTR(temp1_input, S_IRUGO, amdgpu_hwmon_show_temp, NULL, 0); static SENSOR_DEVICE_ATTR(temp1_crit, S_IRUGO, amdgpu_hwmon_show_temp_thresh, NULL, 0); static SENSOR_DEVICE_ATTR(temp1_crit_hyst, S_IRUGO, amdgpu_hwmon_show_temp_thresh, NULL, 1); @@ -1153,19 +1364,14 @@ static umode_t hwmon_attributes_visible(struct kobject *kobj, struct amdgpu_device *adev = dev_get_drvdata(dev); umode_t effective_mode = attr->mode; - /* handle non-powerplay limitations */ - if (!adev->powerplay.pp_handle) { - /* Skip fan attributes if fan is not present */ - if (adev->pm.no_fan && - (attr == &sensor_dev_attr_pwm1.dev_attr.attr || - attr == &sensor_dev_attr_pwm1_enable.dev_attr.attr || - attr == &sensor_dev_attr_pwm1_max.dev_attr.attr || - attr == &sensor_dev_attr_pwm1_min.dev_attr.attr)) - return 0; - /* requires powerplay */ - if (attr == &sensor_dev_attr_fan1_input.dev_attr.attr) - return 0; - } + + /* Skip fan attributes if fan is not present */ + if (adev->pm.no_fan && (attr == &sensor_dev_attr_pwm1.dev_attr.attr || + attr == &sensor_dev_attr_pwm1_enable.dev_attr.attr || + attr == &sensor_dev_attr_pwm1_max.dev_attr.attr || + attr == &sensor_dev_attr_pwm1_min.dev_attr.attr || + attr == &sensor_dev_attr_fan1_input.dev_attr.attr)) + return 0; /* Skip limit attributes if DPM is not enabled */ if (!adev->pm.dpm_enabled && @@ -1658,9 +1864,6 @@ void amdgpu_pm_sysfs_fini(struct amdgpu_device *adev) void amdgpu_pm_compute_clocks(struct amdgpu_device *adev) { - struct drm_device *ddev = adev->ddev; - struct drm_crtc *crtc; - struct amdgpu_crtc *amdgpu_crtc; int i = 0; if (!adev->pm.dpm_enabled) @@ -1676,21 +1879,25 @@ void amdgpu_pm_compute_clocks(struct amdgpu_device *adev) } if (adev->powerplay.pp_funcs->dispatch_tasks) { + if (!amdgpu_device_has_dc_support(adev)) { + mutex_lock(&adev->pm.mutex); + amdgpu_dpm_get_active_displays(adev); + adev->pm.pm_display_cfg.num_display = adev->pm.dpm.new_active_crtcs; + adev->pm.pm_display_cfg.vrefresh = amdgpu_dpm_get_vrefresh(adev); + adev->pm.pm_display_cfg.min_vblank_time = amdgpu_dpm_get_vblank_time(adev); + /* we have issues with mclk switching with refresh rates over 120 hz on the non-DC code. */ + if (adev->pm.pm_display_cfg.vrefresh > 120) + adev->pm.pm_display_cfg.min_vblank_time = 0; + if (adev->powerplay.pp_funcs->display_configuration_change) + adev->powerplay.pp_funcs->display_configuration_change( + adev->powerplay.pp_handle, + &adev->pm.pm_display_cfg); + mutex_unlock(&adev->pm.mutex); + } amdgpu_dpm_dispatch_task(adev, AMD_PP_TASK_DISPLAY_CONFIG_CHANGE, NULL); } else { mutex_lock(&adev->pm.mutex); - adev->pm.dpm.new_active_crtcs = 0; - adev->pm.dpm.new_active_crtc_count = 0; - if (adev->mode_info.num_crtc && adev->mode_info.mode_config_initialized) { - list_for_each_entry(crtc, - &ddev->mode_config.crtc_list, head) { - amdgpu_crtc = to_amdgpu_crtc(crtc); - if (amdgpu_crtc->enabled) { - adev->pm.dpm.new_active_crtcs |= (1 << amdgpu_crtc->crtc_id); - adev->pm.dpm.new_active_crtc_count++; - } - } - } + amdgpu_dpm_get_active_displays(adev); /* update battery/ac status */ if (power_supply_is_system_supplied() > 0) adev->pm.dpm.ac_power = true; @@ -1711,7 +1918,7 @@ void amdgpu_pm_compute_clocks(struct amdgpu_device *adev) static int amdgpu_debugfs_pm_info_pp(struct seq_file *m, struct amdgpu_device *adev) { uint32_t value; - struct pp_gpu_power query = {0}; + uint32_t query = 0; int size; /* sanity check PP is enabled */ @@ -1734,17 +1941,9 @@ static int amdgpu_debugfs_pm_info_pp(struct seq_file *m, struct amdgpu_device *a seq_printf(m, "\t%u mV (VDDGFX)\n", value); if (!amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_VDDNB, (void *)&value, &size)) seq_printf(m, "\t%u mV (VDDNB)\n", value); - size = sizeof(query); - if (!amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_GPU_POWER, (void *)&query, &size)) { - seq_printf(m, "\t%u.%u W (VDDC)\n", query.vddc_power >> 8, - query.vddc_power & 0xff); - seq_printf(m, "\t%u.%u W (VDDCI)\n", query.vddci_power >> 8, - query.vddci_power & 0xff); - seq_printf(m, "\t%u.%u W (max GPU)\n", query.max_gpu_power >> 8, - query.max_gpu_power & 0xff); - seq_printf(m, "\t%u.%u W (average GPU)\n", query.average_gpu_power >> 8, - query.average_gpu_power & 0xff); - } + size = sizeof(uint32_t); + if (!amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_GPU_POWER, (void *)&query, &size)) + seq_printf(m, "\t%u.%u W (average GPU)\n", query >> 8, query & 0xff); size = sizeof(value); seq_printf(m, "\n"); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_prime.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_prime.c index 4b584cb75bf4..4683626b065f 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_prime.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_prime.c @@ -102,12 +102,18 @@ amdgpu_gem_prime_import_sg_table(struct drm_device *dev, struct reservation_object *resv = attach->dmabuf->resv; struct amdgpu_device *adev = dev->dev_private; struct amdgpu_bo *bo; + struct amdgpu_bo_param bp; int ret; + memset(&bp, 0, sizeof(bp)); + bp.size = attach->dmabuf->size; + bp.byte_align = PAGE_SIZE; + bp.domain = AMDGPU_GEM_DOMAIN_CPU; + bp.flags = 0; + bp.type = ttm_bo_type_sg; + bp.resv = resv; ww_mutex_lock(&resv->lock, NULL); - ret = amdgpu_bo_create(adev, attach->dmabuf->size, PAGE_SIZE, - AMDGPU_GEM_DOMAIN_CPU, 0, ttm_bo_type_sg, - resv, &bo); + ret = amdgpu_bo_create(adev, &bp, &bo); if (ret) goto error; @@ -209,7 +215,7 @@ static int amdgpu_gem_begin_cpu_access(struct dma_buf *dma_buf, struct amdgpu_bo *bo = gem_to_amdgpu_bo(dma_buf->priv); struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev); struct ttm_operation_ctx ctx = { true, false }; - u32 domain = amdgpu_display_framebuffer_domains(adev); + u32 domain = amdgpu_display_supported_domains(adev); int ret; bool reads = (direction == DMA_BIDIRECTIONAL || direction == DMA_FROM_DEVICE); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c index c7d43e064fc7..9f1a5bd39ae8 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c @@ -52,6 +52,7 @@ static int psp_sw_init(void *handle) switch (adev->asic_type) { case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: psp_v3_1_set_psp_funcs(psp); break; case CHIP_RAVEN: diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_queue_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_queue_mgr.c index 262c1267249e..8af16e81c7d4 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_queue_mgr.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_queue_mgr.c @@ -66,6 +66,8 @@ static int amdgpu_identity_map(struct amdgpu_device *adev, u32 ring, struct amdgpu_ring **out_ring) { + u32 instance; + switch (mapper->hw_ip) { case AMDGPU_HW_IP_GFX: *out_ring = &adev->gfx.gfx_ring[ring]; @@ -77,13 +79,16 @@ static int amdgpu_identity_map(struct amdgpu_device *adev, *out_ring = &adev->sdma.instance[ring].ring; break; case AMDGPU_HW_IP_UVD: - *out_ring = &adev->uvd.ring; + instance = ring; + *out_ring = &adev->uvd.inst[instance].ring; break; case AMDGPU_HW_IP_VCE: *out_ring = &adev->vce.ring[ring]; break; case AMDGPU_HW_IP_UVD_ENC: - *out_ring = &adev->uvd.ring_enc[ring]; + instance = ring / adev->uvd.num_enc_rings; + *out_ring = + &adev->uvd.inst[instance].ring_enc[ring%adev->uvd.num_enc_rings]; break; case AMDGPU_HW_IP_VCN_DEC: *out_ring = &adev->vcn.ring_dec; @@ -240,13 +245,14 @@ int amdgpu_queue_mgr_map(struct amdgpu_device *adev, ip_num_rings = adev->sdma.num_instances; break; case AMDGPU_HW_IP_UVD: - ip_num_rings = 1; + ip_num_rings = adev->uvd.num_uvd_inst; break; case AMDGPU_HW_IP_VCE: ip_num_rings = adev->vce.num_rings; break; case AMDGPU_HW_IP_UVD_ENC: - ip_num_rings = adev->uvd.num_enc_rings; + ip_num_rings = + adev->uvd.num_enc_rings * adev->uvd.num_uvd_inst; break; case AMDGPU_HW_IP_VCN_DEC: ip_num_rings = 1; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c index d5f526f38e50..c6850b629d0e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c @@ -362,6 +362,7 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring) dma_fence_put(ring->vmid_wait); ring->vmid_wait = NULL; + ring->me = 0; ring->adev->rings[ring->idx] = NULL; } @@ -459,6 +460,26 @@ void amdgpu_ring_lru_touch(struct amdgpu_device *adev, struct amdgpu_ring *ring) spin_unlock(&adev->ring_lru_list_lock); } +/** + * amdgpu_ring_emit_reg_write_reg_wait_helper - ring helper + * + * @adev: amdgpu_device pointer + * @reg0: register to write + * @reg1: register to wait on + * @ref: reference value to write/wait on + * @mask: mask to wait on + * + * Helper for rings that don't support write and wait in a + * single oneshot packet. + */ +void amdgpu_ring_emit_reg_write_reg_wait_helper(struct amdgpu_ring *ring, + uint32_t reg0, uint32_t reg1, + uint32_t ref, uint32_t mask) +{ + amdgpu_ring_emit_wreg(ring, reg0, ref); + amdgpu_ring_emit_reg_wait(ring, reg1, mask, mask); +} + /* * Debugfs info */ diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h index 1a5911882657..1513124c5659 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h @@ -29,7 +29,7 @@ #include /* max number of rings */ -#define AMDGPU_MAX_RINGS 18 +#define AMDGPU_MAX_RINGS 21 #define AMDGPU_MAX_GFX_RINGS 1 #define AMDGPU_MAX_COMPUTE_RINGS 8 #define AMDGPU_MAX_VCE_RINGS 3 @@ -42,6 +42,7 @@ #define AMDGPU_FENCE_FLAG_64BIT (1 << 0) #define AMDGPU_FENCE_FLAG_INT (1 << 1) +#define AMDGPU_FENCE_FLAG_TC_WB_ONLY (1 << 2) enum amdgpu_ring_type { AMDGPU_RING_TYPE_GFX, @@ -90,7 +91,8 @@ int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring, unsigned irq_type); void amdgpu_fence_driver_suspend(struct amdgpu_device *adev); void amdgpu_fence_driver_resume(struct amdgpu_device *adev); -int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **fence); +int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **fence, + unsigned flags); int amdgpu_fence_emit_polling(struct amdgpu_ring *ring, uint32_t *s); void amdgpu_fence_process(struct amdgpu_ring *ring); int amdgpu_fence_wait_empty(struct amdgpu_ring *ring); @@ -154,6 +156,9 @@ struct amdgpu_ring_funcs { void (*emit_wreg)(struct amdgpu_ring *ring, uint32_t reg, uint32_t val); void (*emit_reg_wait)(struct amdgpu_ring *ring, uint32_t reg, uint32_t val, uint32_t mask); + void (*emit_reg_write_reg_wait)(struct amdgpu_ring *ring, + uint32_t reg0, uint32_t reg1, + uint32_t ref, uint32_t mask); void (*emit_tmz)(struct amdgpu_ring *ring, bool start); /* priority functions */ void (*set_priority) (struct amdgpu_ring *ring, @@ -228,6 +233,10 @@ int amdgpu_ring_lru_get(struct amdgpu_device *adev, int type, int *blacklist, int num_blacklist, bool lru_pipe_order, struct amdgpu_ring **ring); void amdgpu_ring_lru_touch(struct amdgpu_device *adev, struct amdgpu_ring *ring); +void amdgpu_ring_emit_reg_write_reg_wait_helper(struct amdgpu_ring *ring, + uint32_t reg0, uint32_t val0, + uint32_t reg1, uint32_t val1); + static inline void amdgpu_ring_clear_ring(struct amdgpu_ring *ring) { int i = 0; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_test.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_test.c index 2dbe87591f81..e3878256743a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_test.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_test.c @@ -33,6 +33,7 @@ static void amdgpu_do_test_moves(struct amdgpu_device *adev) struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring; struct amdgpu_bo *vram_obj = NULL; struct amdgpu_bo **gtt_obj = NULL; + struct amdgpu_bo_param bp; uint64_t gart_addr, vram_addr; unsigned n, size; int i, r; @@ -52,15 +53,21 @@ static void amdgpu_do_test_moves(struct amdgpu_device *adev) n -= adev->irq.ih.ring_size; n /= size; - gtt_obj = kzalloc(n * sizeof(*gtt_obj), GFP_KERNEL); + gtt_obj = kcalloc(n, sizeof(*gtt_obj), GFP_KERNEL); if (!gtt_obj) { DRM_ERROR("Failed to allocate %d pointers\n", n); r = 1; goto out_cleanup; } - - r = amdgpu_bo_create(adev, size, PAGE_SIZE, AMDGPU_GEM_DOMAIN_VRAM, 0, - ttm_bo_type_kernel, NULL, &vram_obj); + memset(&bp, 0, sizeof(bp)); + bp.size = size; + bp.byte_align = PAGE_SIZE; + bp.domain = AMDGPU_GEM_DOMAIN_VRAM; + bp.flags = 0; + bp.type = ttm_bo_type_kernel; + bp.resv = NULL; + + r = amdgpu_bo_create(adev, &bp, &vram_obj); if (r) { DRM_ERROR("Failed to create VRAM object\n"); goto out_cleanup; @@ -79,9 +86,8 @@ static void amdgpu_do_test_moves(struct amdgpu_device *adev) void **vram_start, **vram_end; struct dma_fence *fence = NULL; - r = amdgpu_bo_create(adev, size, PAGE_SIZE, - AMDGPU_GEM_DOMAIN_GTT, 0, - ttm_bo_type_kernel, NULL, gtt_obj + i); + bp.domain = AMDGPU_GEM_DOMAIN_GTT; + r = amdgpu_bo_create(adev, &bp, gtt_obj + i); if (r) { DRM_ERROR("Failed to create GTT object %d\n", i); goto out_lclean; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h index 532263ab6e16..e96e26d3f3b0 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h @@ -275,7 +275,7 @@ TRACE_EVENT(amdgpu_vm_bo_unmap, ), TP_fast_assign( - __entry->bo = bo_va->base.bo; + __entry->bo = bo_va ? bo_va->base.bo : NULL; __entry->start = mapping->start; __entry->last = mapping->last; __entry->offset = mapping->offset; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c index 205da3ff9cd0..e93a0a237dc3 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c @@ -63,16 +63,44 @@ static void amdgpu_ttm_debugfs_fini(struct amdgpu_device *adev); /* * Global memory. */ + +/** + * amdgpu_ttm_mem_global_init - Initialize and acquire reference to + * memory object + * + * @ref: Object for initialization. + * + * This is called by drm_global_item_ref() when an object is being + * initialized. + */ static int amdgpu_ttm_mem_global_init(struct drm_global_reference *ref) { return ttm_mem_global_init(ref->object); } +/** + * amdgpu_ttm_mem_global_release - Drop reference to a memory object + * + * @ref: Object being removed + * + * This is called by drm_global_item_unref() when an object is being + * released. + */ static void amdgpu_ttm_mem_global_release(struct drm_global_reference *ref) { ttm_mem_global_release(ref->object); } +/** + * amdgpu_ttm_global_init - Initialize global TTM memory reference + * structures. + * + * @adev: AMDGPU device for which the global structures need to be + * registered. + * + * This is called as part of the AMDGPU ttm init from amdgpu_ttm_init() + * during bring up. + */ static int amdgpu_ttm_global_init(struct amdgpu_device *adev) { struct drm_global_reference *global_ref; @@ -80,7 +108,9 @@ static int amdgpu_ttm_global_init(struct amdgpu_device *adev) struct drm_sched_rq *rq; int r; + /* ensure reference is false in case init fails */ adev->mman.mem_global_referenced = false; + global_ref = &adev->mman.mem_global_ref; global_ref->global_type = DRM_GLOBAL_TTM_MEM; global_ref->size = sizeof(struct ttm_mem_global); @@ -111,7 +141,7 @@ static int amdgpu_ttm_global_init(struct amdgpu_device *adev) ring = adev->mman.buffer_funcs_ring; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_KERNEL]; r = drm_sched_entity_init(&ring->sched, &adev->mman.entity, - rq, amdgpu_sched_jobs, NULL); + rq, NULL); if (r) { DRM_ERROR("Failed setting up TTM BO move run queue.\n"); goto error_entity; @@ -146,6 +176,18 @@ static int amdgpu_invalidate_caches(struct ttm_bo_device *bdev, uint32_t flags) return 0; } +/** + * amdgpu_init_mem_type - Initialize a memory manager for a specific + * type of memory request. + * + * @bdev: The TTM BO device object (contains a reference to + * amdgpu_device) + * @type: The type of memory requested + * @man: + * + * This is called by ttm_bo_init_mm() when a buffer object is being + * initialized. + */ static int amdgpu_init_mem_type(struct ttm_bo_device *bdev, uint32_t type, struct ttm_mem_type_manager *man) { @@ -161,6 +203,7 @@ static int amdgpu_init_mem_type(struct ttm_bo_device *bdev, uint32_t type, man->default_caching = TTM_PL_FLAG_CACHED; break; case TTM_PL_TT: + /* GTT memory */ man->func = &amdgpu_gtt_mgr_func; man->gpu_offset = adev->gmc.gart_start; man->available_caching = TTM_PL_MASK_CACHING; @@ -193,6 +236,14 @@ static int amdgpu_init_mem_type(struct ttm_bo_device *bdev, uint32_t type, return 0; } +/** + * amdgpu_evict_flags - Compute placement flags + * + * @bo: The buffer object to evict + * @placement: Possible destination(s) for evicted BO + * + * Fill in placement data when ttm_bo_evict() is called + */ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, struct ttm_placement *placement) { @@ -204,12 +255,14 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, .flags = TTM_PL_MASK_CACHING | TTM_PL_FLAG_SYSTEM }; + /* Don't handle scatter gather BOs */ if (bo->type == ttm_bo_type_sg) { placement->num_placement = 0; placement->num_busy_placement = 0; return; } + /* Object isn't an AMDGPU object so ignore */ if (!amdgpu_ttm_bo_is_amdgpu_bo(bo)) { placement->placement = &placements; placement->busy_placement = &placements; @@ -217,26 +270,16 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, placement->num_busy_placement = 1; return; } + abo = ttm_to_amdgpu_bo(bo); switch (bo->mem.mem_type) { case TTM_PL_VRAM: if (!adev->mman.buffer_funcs_enabled) { + /* Move to system memory */ amdgpu_ttm_placement_from_domain(abo, AMDGPU_GEM_DOMAIN_CPU); } else if (adev->gmc.visible_vram_size < adev->gmc.real_vram_size && - !(abo->flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED)) { - unsigned fpfn = adev->gmc.visible_vram_size >> PAGE_SHIFT; - struct drm_mm_node *node = bo->mem.mm_node; - unsigned long pages_left; - - for (pages_left = bo->mem.num_pages; - pages_left; - pages_left -= node->size, node++) { - if (node->start < fpfn) - break; - } - - if (!pages_left) - goto gtt; + !(abo->flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) && + amdgpu_bo_in_cpu_visible_vram(abo)) { /* Try evicting to the CPU inaccessible part of VRAM * first, but only set GTT as busy placement, so this @@ -245,12 +288,12 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, */ amdgpu_ttm_placement_from_domain(abo, AMDGPU_GEM_DOMAIN_VRAM | AMDGPU_GEM_DOMAIN_GTT); - abo->placements[0].fpfn = fpfn; + abo->placements[0].fpfn = adev->gmc.visible_vram_size >> PAGE_SHIFT; abo->placements[0].lpfn = 0; abo->placement.busy_placement = &abo->placements[1]; abo->placement.num_busy_placement = 1; } else { -gtt: + /* Move to GTT memory */ amdgpu_ttm_placement_from_domain(abo, AMDGPU_GEM_DOMAIN_GTT); } break; @@ -261,6 +304,15 @@ gtt: *placement = abo->placement; } +/** + * amdgpu_verify_access - Verify access for a mmap call + * + * @bo: The buffer object to map + * @filp: The file pointer from the process performing the mmap + * + * This is called by ttm_bo_mmap() to verify whether a process + * has the right to mmap a BO to their process space. + */ static int amdgpu_verify_access(struct ttm_buffer_object *bo, struct file *filp) { struct amdgpu_bo *abo = ttm_to_amdgpu_bo(bo); @@ -278,6 +330,15 @@ static int amdgpu_verify_access(struct ttm_buffer_object *bo, struct file *filp) filp->private_data); } +/** + * amdgpu_move_null - Register memory for a buffer object + * + * @bo: The bo to assign the memory to + * @new_mem: The memory to be assigned. + * + * Assign the memory from new_mem to the memory of the buffer object + * bo. + */ static void amdgpu_move_null(struct ttm_buffer_object *bo, struct ttm_mem_reg *new_mem) { @@ -288,6 +349,10 @@ static void amdgpu_move_null(struct ttm_buffer_object *bo, new_mem->mm_node = NULL; } +/** + * amdgpu_mm_node_addr - Compute the GPU relative offset of a GTT + * buffer. + */ static uint64_t amdgpu_mm_node_addr(struct ttm_buffer_object *bo, struct drm_mm_node *mm_node, struct ttm_mem_reg *mem) @@ -302,9 +367,10 @@ static uint64_t amdgpu_mm_node_addr(struct ttm_buffer_object *bo, } /** - * amdgpu_find_mm_node - Helper function finds the drm_mm_node - * corresponding to @offset. It also modifies the offset to be - * within the drm_mm_node returned + * amdgpu_find_mm_node - Helper function finds the drm_mm_node + * corresponding to @offset. It also modifies + * the offset to be within the drm_mm_node + * returned */ static struct drm_mm_node *amdgpu_find_mm_node(struct ttm_mem_reg *mem, unsigned long *offset) @@ -443,7 +509,12 @@ error: return r; } - +/** + * amdgpu_move_blit - Copy an entire buffer to another buffer + * + * This is a helper called by amdgpu_bo_move() and + * amdgpu_move_vram_ram() to help move buffers to and from VRAM. + */ static int amdgpu_move_blit(struct ttm_buffer_object *bo, bool evict, bool no_wait_gpu, struct ttm_mem_reg *new_mem, @@ -478,6 +549,11 @@ error: return r; } +/** + * amdgpu_move_vram_ram - Copy VRAM buffer to RAM buffer + * + * Called by amdgpu_bo_move(). + */ static int amdgpu_move_vram_ram(struct ttm_buffer_object *bo, bool evict, struct ttm_operation_ctx *ctx, struct ttm_mem_reg *new_mem) @@ -490,6 +566,8 @@ static int amdgpu_move_vram_ram(struct ttm_buffer_object *bo, bool evict, int r; adev = amdgpu_ttm_adev(bo->bdev); + + /* create space/pages for new_mem in GTT space */ tmp_mem = *new_mem; tmp_mem.mm_node = NULL; placement.num_placement = 1; @@ -504,25 +582,36 @@ static int amdgpu_move_vram_ram(struct ttm_buffer_object *bo, bool evict, return r; } + /* set caching flags */ r = ttm_tt_set_placement_caching(bo->ttm, tmp_mem.placement); if (unlikely(r)) { goto out_cleanup; } + /* Bind the memory to the GTT space */ r = ttm_tt_bind(bo->ttm, &tmp_mem, ctx); if (unlikely(r)) { goto out_cleanup; } + + /* blit VRAM to GTT */ r = amdgpu_move_blit(bo, true, ctx->no_wait_gpu, &tmp_mem, old_mem); if (unlikely(r)) { goto out_cleanup; } + + /* move BO (in tmp_mem) to new_mem */ r = ttm_bo_move_ttm(bo, ctx, new_mem); out_cleanup: ttm_bo_mem_put(bo, &tmp_mem); return r; } +/** + * amdgpu_move_ram_vram - Copy buffer from RAM to VRAM + * + * Called by amdgpu_bo_move(). + */ static int amdgpu_move_ram_vram(struct ttm_buffer_object *bo, bool evict, struct ttm_operation_ctx *ctx, struct ttm_mem_reg *new_mem) @@ -535,6 +624,8 @@ static int amdgpu_move_ram_vram(struct ttm_buffer_object *bo, bool evict, int r; adev = amdgpu_ttm_adev(bo->bdev); + + /* make space in GTT for old_mem buffer */ tmp_mem = *new_mem; tmp_mem.mm_node = NULL; placement.num_placement = 1; @@ -548,10 +639,14 @@ static int amdgpu_move_ram_vram(struct ttm_buffer_object *bo, bool evict, if (unlikely(r)) { return r; } + + /* move/bind old memory to GTT space */ r = ttm_bo_move_ttm(bo, ctx, &tmp_mem); if (unlikely(r)) { goto out_cleanup; } + + /* copy to VRAM */ r = amdgpu_move_blit(bo, true, ctx->no_wait_gpu, new_mem, old_mem); if (unlikely(r)) { goto out_cleanup; @@ -561,6 +656,11 @@ out_cleanup: return r; } +/** + * amdgpu_bo_move - Move a buffer object to a new memory location + * + * Called by ttm_bo_handle_move_mem() + */ static int amdgpu_bo_move(struct ttm_buffer_object *bo, bool evict, struct ttm_operation_ctx *ctx, struct ttm_mem_reg *new_mem) @@ -626,6 +726,11 @@ memcpy: return 0; } +/** + * amdgpu_ttm_io_mem_reserve - Reserve a block of memory during a fault + * + * Called by ttm_mem_io_reserve() ultimately via ttm_bo_vm_fault() + */ static int amdgpu_ttm_io_mem_reserve(struct ttm_bo_device *bdev, struct ttm_mem_reg *mem) { struct ttm_mem_type_manager *man = &bdev->man[mem->mem_type]; @@ -695,7 +800,7 @@ struct amdgpu_ttm_tt { struct ttm_dma_tt ttm; u64 offset; uint64_t userptr; - struct mm_struct *usermm; + struct task_struct *usertask; uint32_t userflags; spinlock_t guptasklock; struct list_head guptasks; @@ -703,17 +808,29 @@ struct amdgpu_ttm_tt { uint32_t last_set_pages; }; +/** + * amdgpu_ttm_tt_get_user_pages - Pin pages of memory pointed to + * by a USERPTR pointer to memory + * + * Called by amdgpu_gem_userptr_ioctl() and amdgpu_cs_parser_bos(). + * This provides a wrapper around the get_user_pages() call to provide + * device accessible pages that back user memory. + */ int amdgpu_ttm_tt_get_user_pages(struct ttm_tt *ttm, struct page **pages) { struct amdgpu_ttm_tt *gtt = (void *)ttm; + struct mm_struct *mm = gtt->usertask->mm; unsigned int flags = 0; unsigned pinned = 0; int r; + if (!mm) /* Happens during process shutdown */ + return -ESRCH; + if (!(gtt->userflags & AMDGPU_GEM_USERPTR_READONLY)) flags |= FOLL_WRITE; - down_read(¤t->mm->mmap_sem); + down_read(&mm->mmap_sem); if (gtt->userflags & AMDGPU_GEM_USERPTR_ANONONLY) { /* check that we only use anonymous memory @@ -721,13 +838,14 @@ int amdgpu_ttm_tt_get_user_pages(struct ttm_tt *ttm, struct page **pages) unsigned long end = gtt->userptr + ttm->num_pages * PAGE_SIZE; struct vm_area_struct *vma; - vma = find_vma(gtt->usermm, gtt->userptr); + vma = find_vma(mm, gtt->userptr); if (!vma || vma->vm_file || vma->vm_end < end) { - up_read(¤t->mm->mmap_sem); + up_read(&mm->mmap_sem); return -EPERM; } } + /* loop enough times using contiguous pages of memory */ do { unsigned num_pages = ttm->num_pages - pinned; uint64_t userptr = gtt->userptr + pinned * PAGE_SIZE; @@ -739,7 +857,12 @@ int amdgpu_ttm_tt_get_user_pages(struct ttm_tt *ttm, struct page **pages) list_add(&guptask.list, >t->guptasks); spin_unlock(>t->guptasklock); - r = get_user_pages(userptr, num_pages, flags, p, NULL); + if (mm == current->mm) + r = get_user_pages(userptr, num_pages, flags, p, NULL); + else + r = get_user_pages_remote(gtt->usertask, + mm, userptr, num_pages, + flags, p, NULL, NULL); spin_lock(>t->guptasklock); list_del(&guptask.list); @@ -752,15 +875,23 @@ int amdgpu_ttm_tt_get_user_pages(struct ttm_tt *ttm, struct page **pages) } while (pinned < ttm->num_pages); - up_read(¤t->mm->mmap_sem); + up_read(&mm->mmap_sem); return 0; release_pages: release_pages(pages, pinned); - up_read(¤t->mm->mmap_sem); + up_read(&mm->mmap_sem); return r; } +/** + * amdgpu_ttm_tt_set_user_pages - Copy pages in, putting old pages + * as necessary. + * + * Called by amdgpu_cs_list_validate(). This creates the page list + * that backs user memory and will ultimately be mapped into the device + * address space. + */ void amdgpu_ttm_tt_set_user_pages(struct ttm_tt *ttm, struct page **pages) { struct amdgpu_ttm_tt *gtt = (void *)ttm; @@ -775,6 +906,11 @@ void amdgpu_ttm_tt_set_user_pages(struct ttm_tt *ttm, struct page **pages) } } +/** + * amdgpu_ttm_tt_mark_user_page - Mark pages as dirty + * + * Called while unpinning userptr pages + */ void amdgpu_ttm_tt_mark_user_pages(struct ttm_tt *ttm) { struct amdgpu_ttm_tt *gtt = (void *)ttm; @@ -793,7 +929,12 @@ void amdgpu_ttm_tt_mark_user_pages(struct ttm_tt *ttm) } } -/* prepare the sg table with the user pages */ +/** + * amdgpu_ttm_tt_pin_userptr - prepare the sg table with the + * user pages + * + * Called by amdgpu_ttm_backend_bind() + **/ static int amdgpu_ttm_tt_pin_userptr(struct ttm_tt *ttm) { struct amdgpu_device *adev = amdgpu_ttm_adev(ttm->bdev); @@ -805,17 +946,20 @@ static int amdgpu_ttm_tt_pin_userptr(struct ttm_tt *ttm) enum dma_data_direction direction = write ? DMA_BIDIRECTIONAL : DMA_TO_DEVICE; + /* Allocate an SG array and squash pages into it */ r = sg_alloc_table_from_pages(ttm->sg, ttm->pages, ttm->num_pages, 0, ttm->num_pages << PAGE_SHIFT, GFP_KERNEL); if (r) goto release_sg; + /* Map SG to device */ r = -ENOMEM; nents = dma_map_sg(adev->dev, ttm->sg->sgl, ttm->sg->nents, direction); if (nents != ttm->sg->nents) goto release_sg; + /* convert SG to linear array of pages and dma addresses */ drm_prime_sg_to_page_addr_arrays(ttm->sg, ttm->pages, gtt->ttm.dma_address, ttm->num_pages); @@ -826,6 +970,9 @@ release_sg: return r; } +/** + * amdgpu_ttm_tt_unpin_userptr - Unpin and unmap userptr pages + */ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) { struct amdgpu_device *adev = amdgpu_ttm_adev(ttm->bdev); @@ -839,14 +986,60 @@ static void amdgpu_ttm_tt_unpin_userptr(struct ttm_tt *ttm) if (!ttm->sg->sgl) return; - /* free the sg table and pages again */ + /* unmap the pages mapped to the device */ dma_unmap_sg(adev->dev, ttm->sg->sgl, ttm->sg->nents, direction); + /* mark the pages as dirty */ amdgpu_ttm_tt_mark_user_pages(ttm); sg_free_table(ttm->sg); } +int amdgpu_ttm_gart_bind(struct amdgpu_device *adev, + struct ttm_buffer_object *tbo, + uint64_t flags) +{ + struct amdgpu_bo *abo = ttm_to_amdgpu_bo(tbo); + struct ttm_tt *ttm = tbo->ttm; + struct amdgpu_ttm_tt *gtt = (void *)ttm; + int r; + + if (abo->flags & AMDGPU_GEM_CREATE_MQD_GFX9) { + uint64_t page_idx = 1; + + r = amdgpu_gart_bind(adev, gtt->offset, page_idx, + ttm->pages, gtt->ttm.dma_address, flags); + if (r) + goto gart_bind_fail; + + /* Patch mtype of the second part BO */ + flags &= ~AMDGPU_PTE_MTYPE_MASK; + flags |= AMDGPU_PTE_MTYPE(AMDGPU_MTYPE_NC); + + r = amdgpu_gart_bind(adev, + gtt->offset + (page_idx << PAGE_SHIFT), + ttm->num_pages - page_idx, + &ttm->pages[page_idx], + &(gtt->ttm.dma_address[page_idx]), flags); + } else { + r = amdgpu_gart_bind(adev, gtt->offset, ttm->num_pages, + ttm->pages, gtt->ttm.dma_address, flags); + } + +gart_bind_fail: + if (r) + DRM_ERROR("failed to bind %lu pages at 0x%08llX\n", + ttm->num_pages, gtt->offset); + + return r; +} + +/** + * amdgpu_ttm_backend_bind - Bind GTT memory + * + * Called by ttm_tt_bind() on behalf of ttm_bo_handle_move_mem(). + * This handles binding GTT memory to the device address space. + */ static int amdgpu_ttm_backend_bind(struct ttm_tt *ttm, struct ttm_mem_reg *bo_mem) { @@ -877,7 +1070,10 @@ static int amdgpu_ttm_backend_bind(struct ttm_tt *ttm, return 0; } + /* compute PTE flags relevant to this BO memory */ flags = amdgpu_ttm_tt_pte_flags(adev, ttm, bo_mem); + + /* bind pages into GART page tables */ gtt->offset = (u64)bo_mem->start << PAGE_SHIFT; r = amdgpu_gart_bind(adev, gtt->offset, ttm->num_pages, ttm->pages, gtt->ttm.dma_address, flags); @@ -888,6 +1084,9 @@ static int amdgpu_ttm_backend_bind(struct ttm_tt *ttm, return r; } +/** + * amdgpu_ttm_alloc_gart - Allocate GART memory for buffer object + */ int amdgpu_ttm_alloc_gart(struct ttm_buffer_object *bo) { struct amdgpu_device *adev = amdgpu_ttm_adev(bo->bdev); @@ -903,6 +1102,7 @@ int amdgpu_ttm_alloc_gart(struct ttm_buffer_object *bo) amdgpu_gtt_mgr_has_gart_addr(&bo->mem)) return 0; + /* allocate GTT space */ tmp = bo->mem; tmp.mm_node = NULL; placement.num_placement = 1; @@ -918,10 +1118,12 @@ int amdgpu_ttm_alloc_gart(struct ttm_buffer_object *bo) if (unlikely(r)) return r; + /* compute PTE flags for this buffer object */ flags = amdgpu_ttm_tt_pte_flags(adev, bo->ttm, &tmp); + + /* Bind pages */ gtt->offset = (u64)tmp.start << PAGE_SHIFT; - r = amdgpu_gart_bind(adev, gtt->offset, bo->ttm->num_pages, - bo->ttm->pages, gtt->ttm.dma_address, flags); + r = amdgpu_ttm_gart_bind(adev, bo, flags); if (unlikely(r)) { ttm_bo_mem_put(bo, &tmp); return r; @@ -935,31 +1137,40 @@ int amdgpu_ttm_alloc_gart(struct ttm_buffer_object *bo) return 0; } +/** + * amdgpu_ttm_recover_gart - Rebind GTT pages + * + * Called by amdgpu_gtt_mgr_recover() from amdgpu_device_reset() to + * rebind GTT pages during a GPU reset. + */ int amdgpu_ttm_recover_gart(struct ttm_buffer_object *tbo) { struct amdgpu_device *adev = amdgpu_ttm_adev(tbo->bdev); - struct amdgpu_ttm_tt *gtt = (void *)tbo->ttm; uint64_t flags; int r; - if (!gtt) + if (!tbo->ttm) return 0; - flags = amdgpu_ttm_tt_pte_flags(adev, >t->ttm.ttm, &tbo->mem); - r = amdgpu_gart_bind(adev, gtt->offset, gtt->ttm.ttm.num_pages, - gtt->ttm.ttm.pages, gtt->ttm.dma_address, flags); - if (r) - DRM_ERROR("failed to bind %lu pages at 0x%08llX\n", - gtt->ttm.ttm.num_pages, gtt->offset); + flags = amdgpu_ttm_tt_pte_flags(adev, tbo->ttm, &tbo->mem); + r = amdgpu_ttm_gart_bind(adev, tbo, flags); + return r; } +/** + * amdgpu_ttm_backend_unbind - Unbind GTT mapped pages + * + * Called by ttm_tt_unbind() on behalf of ttm_bo_move_ttm() and + * ttm_tt_destroy(). + */ static int amdgpu_ttm_backend_unbind(struct ttm_tt *ttm) { struct amdgpu_device *adev = amdgpu_ttm_adev(ttm->bdev); struct amdgpu_ttm_tt *gtt = (void *)ttm; int r; + /* if the pages have userptr pinning then clear that first */ if (gtt->userptr) amdgpu_ttm_tt_unpin_userptr(ttm); @@ -978,6 +1189,9 @@ static void amdgpu_ttm_backend_destroy(struct ttm_tt *ttm) { struct amdgpu_ttm_tt *gtt = (void *)ttm; + if (gtt->usertask) + put_task_struct(gtt->usertask); + ttm_dma_tt_fini(>t->ttm); kfree(gtt); } @@ -988,6 +1202,13 @@ static struct ttm_backend_func amdgpu_backend_func = { .destroy = &amdgpu_ttm_backend_destroy, }; +/** + * amdgpu_ttm_tt_create - Create a ttm_tt object for a given BO + * + * @bo: The buffer object to create a GTT ttm_tt object around + * + * Called by ttm_tt_create(). + */ static struct ttm_tt *amdgpu_ttm_tt_create(struct ttm_buffer_object *bo, uint32_t page_flags) { @@ -1001,6 +1222,8 @@ static struct ttm_tt *amdgpu_ttm_tt_create(struct ttm_buffer_object *bo, return NULL; } gtt->ttm.ttm.func = &amdgpu_backend_func; + + /* allocate space for the uninitialized page entries */ if (ttm_sg_tt_init(>t->ttm, bo, page_flags)) { kfree(gtt); return NULL; @@ -1008,6 +1231,12 @@ static struct ttm_tt *amdgpu_ttm_tt_create(struct ttm_buffer_object *bo, return >t->ttm.ttm; } +/** + * amdgpu_ttm_tt_populate - Map GTT pages visible to the device + * + * Map the pages of a ttm_tt object to an address space visible + * to the underlying device. + */ static int amdgpu_ttm_tt_populate(struct ttm_tt *ttm, struct ttm_operation_ctx *ctx) { @@ -1015,6 +1244,7 @@ static int amdgpu_ttm_tt_populate(struct ttm_tt *ttm, struct amdgpu_ttm_tt *gtt = (void *)ttm; bool slave = !!(ttm->page_flags & TTM_PAGE_FLAG_SG); + /* user pages are bound by amdgpu_ttm_tt_pin_userptr() */ if (gtt && gtt->userptr) { ttm->sg = kzalloc(sizeof(struct sg_table), GFP_KERNEL); if (!ttm->sg) @@ -1039,9 +1269,17 @@ static int amdgpu_ttm_tt_populate(struct ttm_tt *ttm, } #endif + /* fall back to generic helper to populate the page array + * and map them to the device */ return ttm_populate_and_map_pages(adev->dev, >t->ttm, ctx); } +/** + * amdgpu_ttm_tt_unpopulate - unmap GTT pages and unpopulate page arrays + * + * Unmaps pages of a ttm_tt object from the device address space and + * unpopulates the page array backing it. + */ static void amdgpu_ttm_tt_unpopulate(struct ttm_tt *ttm) { struct amdgpu_device *adev; @@ -1067,9 +1305,21 @@ static void amdgpu_ttm_tt_unpopulate(struct ttm_tt *ttm) } #endif + /* fall back to generic helper to unmap and unpopulate array */ ttm_unmap_and_unpopulate_pages(adev->dev, >t->ttm); } +/** + * amdgpu_ttm_tt_set_userptr - Initialize userptr GTT ttm_tt + * for the current task + * + * @ttm: The ttm_tt object to bind this userptr object to + * @addr: The address in the current tasks VM space to use + * @flags: Requirements of userptr object. + * + * Called by amdgpu_gem_userptr_ioctl() to bind userptr pages + * to current task + */ int amdgpu_ttm_tt_set_userptr(struct ttm_tt *ttm, uint64_t addr, uint32_t flags) { @@ -1079,8 +1329,13 @@ int amdgpu_ttm_tt_set_userptr(struct ttm_tt *ttm, uint64_t addr, return -EINVAL; gtt->userptr = addr; - gtt->usermm = current->mm; gtt->userflags = flags; + + if (gtt->usertask) + put_task_struct(gtt->usertask); + gtt->usertask = current->group_leader; + get_task_struct(gtt->usertask); + spin_lock_init(>t->guptasklock); INIT_LIST_HEAD(>t->guptasks); atomic_set(>t->mmu_invalidations, 0); @@ -1089,6 +1344,9 @@ int amdgpu_ttm_tt_set_userptr(struct ttm_tt *ttm, uint64_t addr, return 0; } +/** + * amdgpu_ttm_tt_get_usermm - Return memory manager for ttm_tt object + */ struct mm_struct *amdgpu_ttm_tt_get_usermm(struct ttm_tt *ttm) { struct amdgpu_ttm_tt *gtt = (void *)ttm; @@ -1096,9 +1354,18 @@ struct mm_struct *amdgpu_ttm_tt_get_usermm(struct ttm_tt *ttm) if (gtt == NULL) return NULL; - return gtt->usermm; + if (gtt->usertask == NULL) + return NULL; + + return gtt->usertask->mm; } +/** + * amdgpu_ttm_tt_affect_userptr - Determine if a ttm_tt object lays + * inside an address range for the + * current task. + * + */ bool amdgpu_ttm_tt_affect_userptr(struct ttm_tt *ttm, unsigned long start, unsigned long end) { @@ -1109,10 +1376,16 @@ bool amdgpu_ttm_tt_affect_userptr(struct ttm_tt *ttm, unsigned long start, if (gtt == NULL || !gtt->userptr) return false; + /* Return false if no part of the ttm_tt object lies within + * the range + */ size = (unsigned long)gtt->ttm.ttm.num_pages * PAGE_SIZE; if (gtt->userptr > end || gtt->userptr + size <= start) return false; + /* Search the lists of tasks that hold this mapping and see + * if current is one of them. If it is return false. + */ spin_lock(>t->guptasklock); list_for_each_entry(entry, >t->guptasks, list) { if (entry->task == current) { @@ -1127,6 +1400,10 @@ bool amdgpu_ttm_tt_affect_userptr(struct ttm_tt *ttm, unsigned long start, return true; } +/** + * amdgpu_ttm_tt_userptr_invalidated - Has the ttm_tt object been + * invalidated? + */ bool amdgpu_ttm_tt_userptr_invalidated(struct ttm_tt *ttm, int *last_invalidated) { @@ -1137,6 +1414,12 @@ bool amdgpu_ttm_tt_userptr_invalidated(struct ttm_tt *ttm, return prev_invalidated != *last_invalidated; } +/** + * amdgpu_ttm_tt_userptr_needs_pages - Have the pages backing this + * ttm_tt object been invalidated + * since the last time they've + * been set? + */ bool amdgpu_ttm_tt_userptr_needs_pages(struct ttm_tt *ttm) { struct amdgpu_ttm_tt *gtt = (void *)ttm; @@ -1147,6 +1430,9 @@ bool amdgpu_ttm_tt_userptr_needs_pages(struct ttm_tt *ttm) return atomic_read(>t->mmu_invalidations) != gtt->last_set_pages; } +/** + * amdgpu_ttm_tt_is_readonly - Is the ttm_tt object read only? + */ bool amdgpu_ttm_tt_is_readonly(struct ttm_tt *ttm) { struct amdgpu_ttm_tt *gtt = (void *)ttm; @@ -1157,6 +1443,12 @@ bool amdgpu_ttm_tt_is_readonly(struct ttm_tt *ttm) return !!(gtt->userflags & AMDGPU_GEM_USERPTR_READONLY); } +/** + * amdgpu_ttm_tt_pte_flags - Compute PTE flags for ttm_tt object + * + * @ttm: The ttm_tt object to compute the flags for + * @mem: The memory registry backing this ttm_tt object + */ uint64_t amdgpu_ttm_tt_pte_flags(struct amdgpu_device *adev, struct ttm_tt *ttm, struct ttm_mem_reg *mem) { @@ -1181,6 +1473,16 @@ uint64_t amdgpu_ttm_tt_pte_flags(struct amdgpu_device *adev, struct ttm_tt *ttm, return flags; } +/** + * amdgpu_ttm_bo_eviction_valuable - Check to see if we can evict + * a buffer object. + * + * Return true if eviction is sensible. Called by + * ttm_mem_evict_first() on behalf of ttm_bo_mem_force_space() + * which tries to evict buffer objects until it can find space + * for a new object and by ttm_bo_force_list_clean() which is + * used to clean out a memory space. + */ static bool amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo, const struct ttm_place *place) { @@ -1227,6 +1529,19 @@ static bool amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo, return ttm_bo_eviction_valuable(bo, place); } +/** + * amdgpu_ttm_access_memory - Read or Write memory that backs a + * buffer object. + * + * @bo: The buffer object to read/write + * @offset: Offset into buffer object + * @buf: Secondary buffer to write/read from + * @len: Length in bytes of access + * @write: true if writing + * + * This is used to access VRAM that backs a buffer object via MMIO + * access for debugging purposes. + */ static int amdgpu_ttm_access_memory(struct ttm_buffer_object *bo, unsigned long offset, void *buf, int len, int write) @@ -1329,6 +1644,7 @@ static void amdgpu_ttm_fw_reserve_vram_fini(struct amdgpu_device *adev) static int amdgpu_ttm_fw_reserve_vram_init(struct amdgpu_device *adev) { struct ttm_operation_ctx ctx = { false, false }; + struct amdgpu_bo_param bp; int r = 0; int i; u64 vram_size = adev->gmc.visible_vram_size; @@ -1336,17 +1652,21 @@ static int amdgpu_ttm_fw_reserve_vram_init(struct amdgpu_device *adev) u64 size = adev->fw_vram_usage.size; struct amdgpu_bo *bo; + memset(&bp, 0, sizeof(bp)); + bp.size = adev->fw_vram_usage.size; + bp.byte_align = PAGE_SIZE; + bp.domain = AMDGPU_GEM_DOMAIN_VRAM; + bp.flags = AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED | + AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS; + bp.type = ttm_bo_type_kernel; + bp.resv = NULL; adev->fw_vram_usage.va = NULL; adev->fw_vram_usage.reserved_bo = NULL; if (adev->fw_vram_usage.size > 0 && adev->fw_vram_usage.size <= vram_size) { - r = amdgpu_bo_create(adev, adev->fw_vram_usage.size, PAGE_SIZE, - AMDGPU_GEM_DOMAIN_VRAM, - AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED | - AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS, - ttm_bo_type_kernel, NULL, + r = amdgpu_bo_create(adev, &bp, &adev->fw_vram_usage.reserved_bo); if (r) goto error_create; @@ -1398,13 +1718,22 @@ error_create: adev->fw_vram_usage.reserved_bo = NULL; return r; } - +/** + * amdgpu_ttm_init - Init the memory management (ttm) as well as + * various gtt/vram related fields. + * + * This initializes all of the memory space pools that the TTM layer + * will need such as the GTT space (system memory mapped to the device), + * VRAM (on-board memory), and on-chip memories (GDS, GWS, OA) which + * can be mapped per VMID. + */ int amdgpu_ttm_init(struct amdgpu_device *adev) { uint64_t gtt_size; int r; u64 vis_vram_limit; + /* initialize global references for vram/gtt */ r = amdgpu_ttm_global_init(adev); if (r) { return r; @@ -1425,6 +1754,7 @@ int amdgpu_ttm_init(struct amdgpu_device *adev) /* We opt to avoid OOM on system pages allocations */ adev->mman.bdev.no_retry = true; + /* Initialize VRAM pool with all of VRAM divided into pages */ r = ttm_bo_init_mm(&adev->mman.bdev, TTM_PL_VRAM, adev->gmc.real_vram_size >> PAGE_SHIFT); if (r) { @@ -1454,15 +1784,23 @@ int amdgpu_ttm_init(struct amdgpu_device *adev) return r; } - r = amdgpu_bo_create_kernel(adev, adev->gmc.stolen_size, PAGE_SIZE, - AMDGPU_GEM_DOMAIN_VRAM, - &adev->stolen_vga_memory, - NULL, NULL); - if (r) - return r; + /* allocate memory as required for VGA + * This is used for VGA emulation and pre-OS scanout buffers to + * avoid display artifacts while transitioning between pre-OS + * and driver. */ + if (adev->gmc.stolen_size) { + r = amdgpu_bo_create_kernel(adev, adev->gmc.stolen_size, PAGE_SIZE, + AMDGPU_GEM_DOMAIN_VRAM, + &adev->stolen_vga_memory, + NULL, NULL); + if (r) + return r; + } DRM_INFO("amdgpu: %uM of VRAM memory ready\n", (unsigned) (adev->gmc.real_vram_size / (1024 * 1024))); + /* Compute GTT size, either bsaed on 3/4th the size of RAM size + * or whatever the user passed on module init */ if (amdgpu_gtt_size == -1) { struct sysinfo si; @@ -1473,6 +1811,8 @@ int amdgpu_ttm_init(struct amdgpu_device *adev) } else gtt_size = (uint64_t)amdgpu_gtt_size << 20; + + /* Initialize GTT memory pool */ r = ttm_bo_init_mm(&adev->mman.bdev, TTM_PL_TT, gtt_size >> PAGE_SHIFT); if (r) { DRM_ERROR("Failed initializing GTT heap.\n"); @@ -1481,6 +1821,7 @@ int amdgpu_ttm_init(struct amdgpu_device *adev) DRM_INFO("amdgpu: %uM of GTT memory ready.\n", (unsigned)(gtt_size / (1024 * 1024))); + /* Initialize various on-chip memory pools */ adev->gds.mem.total_size = adev->gds.mem.total_size << AMDGPU_GDS_SHIFT; adev->gds.mem.gfx_partition_size = adev->gds.mem.gfx_partition_size << AMDGPU_GDS_SHIFT; adev->gds.mem.cs_partition_size = adev->gds.mem.cs_partition_size << AMDGPU_GDS_SHIFT; @@ -1520,6 +1861,7 @@ int amdgpu_ttm_init(struct amdgpu_device *adev) } } + /* Register debugfs entries for amdgpu_ttm */ r = amdgpu_ttm_debugfs_init(adev); if (r) { DRM_ERROR("Failed to init debugfs\n"); @@ -1528,13 +1870,25 @@ int amdgpu_ttm_init(struct amdgpu_device *adev) return 0; } +/** + * amdgpu_ttm_late_init - Handle any late initialization for + * amdgpu_ttm + */ +void amdgpu_ttm_late_init(struct amdgpu_device *adev) +{ + /* return the VGA stolen memory (if any) back to VRAM */ + amdgpu_bo_free_kernel(&adev->stolen_vga_memory, NULL, NULL); +} + +/** + * amdgpu_ttm_fini - De-initialize the TTM memory pools + */ void amdgpu_ttm_fini(struct amdgpu_device *adev) { if (!adev->mman.initialized) return; amdgpu_ttm_debugfs_fini(adev); - amdgpu_bo_free_kernel(&adev->stolen_vga_memory, NULL, NULL); amdgpu_ttm_fw_reserve_vram_fini(adev); if (adev->mman.aper_base_kaddr) iounmap(adev->mman.aper_base_kaddr); @@ -1856,6 +2210,11 @@ static const struct drm_info_list amdgpu_ttm_debugfs_list[] = { #endif }; +/** + * amdgpu_ttm_vram_read - Linear read access to VRAM + * + * Accesses VRAM via MMIO for debugging purposes. + */ static ssize_t amdgpu_ttm_vram_read(struct file *f, char __user *buf, size_t size, loff_t *pos) { @@ -1895,6 +2254,11 @@ static ssize_t amdgpu_ttm_vram_read(struct file *f, char __user *buf, return result; } +/** + * amdgpu_ttm_vram_write - Linear write access to VRAM + * + * Accesses VRAM via MMIO for debugging purposes. + */ static ssize_t amdgpu_ttm_vram_write(struct file *f, const char __user *buf, size_t size, loff_t *pos) { @@ -1943,6 +2307,9 @@ static const struct file_operations amdgpu_ttm_vram_fops = { #ifdef CONFIG_DRM_AMDGPU_GART_DEBUGFS +/** + * amdgpu_ttm_gtt_read - Linear read access to GTT memory + */ static ssize_t amdgpu_ttm_gtt_read(struct file *f, char __user *buf, size_t size, loff_t *pos) { @@ -1990,6 +2357,13 @@ static const struct file_operations amdgpu_ttm_gtt_fops = { #endif +/** + * amdgpu_iomem_read - Virtual read access to GPU mapped memory + * + * This function is used to read memory that has been mapped to the + * GPU and the known addresses are not physical addresses but instead + * bus addresses (e.g., what you'd put in an IB or ring buffer). + */ static ssize_t amdgpu_iomem_read(struct file *f, char __user *buf, size_t size, loff_t *pos) { @@ -1998,6 +2372,7 @@ static ssize_t amdgpu_iomem_read(struct file *f, char __user *buf, ssize_t result = 0; int r; + /* retrieve the IOMMU domain if any for this device */ dom = iommu_get_domain_for_dev(adev->dev); while (size) { @@ -2010,6 +2385,10 @@ static ssize_t amdgpu_iomem_read(struct file *f, char __user *buf, bytes = bytes < size ? bytes : size; + /* Translate the bus address to a physical address. If + * the domain is NULL it means there is no IOMMU active + * and the address translation is the identity + */ addr = dom ? iommu_iova_to_phys(dom, addr) : addr; pfn = addr >> PAGE_SHIFT; @@ -2034,6 +2413,13 @@ static ssize_t amdgpu_iomem_read(struct file *f, char __user *buf, return result; } +/** + * amdgpu_iomem_write - Virtual write access to GPU mapped memory + * + * This function is used to write memory that has been mapped to the + * GPU and the known addresses are not physical addresses but instead + * bus addresses (e.g., what you'd put in an IB or ring buffer). + */ static ssize_t amdgpu_iomem_write(struct file *f, const char __user *buf, size_t size, loff_t *pos) { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h index 6ea7de863041..e969c879d87e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h @@ -77,6 +77,7 @@ uint64_t amdgpu_vram_mgr_usage(struct ttm_mem_type_manager *man); uint64_t amdgpu_vram_mgr_vis_usage(struct ttm_mem_type_manager *man); int amdgpu_ttm_init(struct amdgpu_device *adev); +void amdgpu_ttm_late_init(struct amdgpu_device *adev); void amdgpu_ttm_fini(struct amdgpu_device *adev); void amdgpu_ttm_set_buffer_funcs_status(struct amdgpu_device *adev, bool enable); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c index 5916cc25e28b..f55f72a37ca8 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c @@ -161,8 +161,38 @@ void amdgpu_ucode_print_rlc_hdr(const struct common_firmware_header *hdr) le32_to_cpu(rlc_hdr->reg_list_format_separate_array_offset_bytes)); DRM_DEBUG("reg_list_separate_size_bytes: %u\n", le32_to_cpu(rlc_hdr->reg_list_separate_size_bytes)); - DRM_DEBUG("reg_list_separate_size_bytes: %u\n", - le32_to_cpu(rlc_hdr->reg_list_separate_size_bytes)); + DRM_DEBUG("reg_list_separate_array_offset_bytes: %u\n", + le32_to_cpu(rlc_hdr->reg_list_separate_array_offset_bytes)); + if (version_minor == 1) { + const struct rlc_firmware_header_v2_1 *v2_1 = + container_of(rlc_hdr, struct rlc_firmware_header_v2_1, v2_0); + DRM_DEBUG("reg_list_format_direct_reg_list_length: %u\n", + le32_to_cpu(v2_1->reg_list_format_direct_reg_list_length)); + DRM_DEBUG("save_restore_list_cntl_ucode_ver: %u\n", + le32_to_cpu(v2_1->save_restore_list_cntl_ucode_ver)); + DRM_DEBUG("save_restore_list_cntl_feature_ver: %u\n", + le32_to_cpu(v2_1->save_restore_list_cntl_feature_ver)); + DRM_DEBUG("save_restore_list_cntl_size_bytes %u\n", + le32_to_cpu(v2_1->save_restore_list_cntl_size_bytes)); + DRM_DEBUG("save_restore_list_cntl_offset_bytes: %u\n", + le32_to_cpu(v2_1->save_restore_list_cntl_offset_bytes)); + DRM_DEBUG("save_restore_list_gpm_ucode_ver: %u\n", + le32_to_cpu(v2_1->save_restore_list_gpm_ucode_ver)); + DRM_DEBUG("save_restore_list_gpm_feature_ver: %u\n", + le32_to_cpu(v2_1->save_restore_list_gpm_feature_ver)); + DRM_DEBUG("save_restore_list_gpm_size_bytes %u\n", + le32_to_cpu(v2_1->save_restore_list_gpm_size_bytes)); + DRM_DEBUG("save_restore_list_gpm_offset_bytes: %u\n", + le32_to_cpu(v2_1->save_restore_list_gpm_offset_bytes)); + DRM_DEBUG("save_restore_list_srm_ucode_ver: %u\n", + le32_to_cpu(v2_1->save_restore_list_srm_ucode_ver)); + DRM_DEBUG("save_restore_list_srm_feature_ver: %u\n", + le32_to_cpu(v2_1->save_restore_list_srm_feature_ver)); + DRM_DEBUG("save_restore_list_srm_size_bytes %u\n", + le32_to_cpu(v2_1->save_restore_list_srm_size_bytes)); + DRM_DEBUG("save_restore_list_srm_offset_bytes: %u\n", + le32_to_cpu(v2_1->save_restore_list_srm_offset_bytes)); + } } else { DRM_ERROR("Unknown RLC ucode version: %u.%u\n", version_major, version_minor); } @@ -265,6 +295,7 @@ amdgpu_ucode_get_load_type(struct amdgpu_device *adev, int load_type) case CHIP_POLARIS10: case CHIP_POLARIS11: case CHIP_POLARIS12: + case CHIP_VEGAM: if (!load_type) return AMDGPU_FW_LOAD_DIRECT; else @@ -276,6 +307,8 @@ amdgpu_ucode_get_load_type(struct amdgpu_device *adev, int load_type) return AMDGPU_FW_LOAD_DIRECT; else return AMDGPU_FW_LOAD_PSP; + case CHIP_VEGA20: + return AMDGPU_FW_LOAD_DIRECT; default: DRM_ERROR("Unknown firmware load type\n"); } @@ -307,7 +340,10 @@ static int amdgpu_ucode_init_single_fw(struct amdgpu_device *adev, (ucode->ucode_id != AMDGPU_UCODE_ID_CP_MEC1 && ucode->ucode_id != AMDGPU_UCODE_ID_CP_MEC2 && ucode->ucode_id != AMDGPU_UCODE_ID_CP_MEC1_JT && - ucode->ucode_id != AMDGPU_UCODE_ID_CP_MEC2_JT)) { + ucode->ucode_id != AMDGPU_UCODE_ID_CP_MEC2_JT && + ucode->ucode_id != AMDGPU_UCODE_ID_RLC_RESTORE_LIST_CNTL && + ucode->ucode_id != AMDGPU_UCODE_ID_RLC_RESTORE_LIST_GPM_MEM && + ucode->ucode_id != AMDGPU_UCODE_ID_RLC_RESTORE_LIST_SRM_MEM)) { ucode->ucode_size = le32_to_cpu(header->ucode_size_bytes); memcpy(ucode->kaddr, (void *)((uint8_t *)ucode->fw->data + @@ -329,6 +365,18 @@ static int amdgpu_ucode_init_single_fw(struct amdgpu_device *adev, le32_to_cpu(header->ucode_array_offset_bytes) + le32_to_cpu(cp_hdr->jt_offset) * 4), ucode->ucode_size); + } else if (ucode->ucode_id == AMDGPU_UCODE_ID_RLC_RESTORE_LIST_CNTL) { + ucode->ucode_size = adev->gfx.rlc.save_restore_list_cntl_size_bytes; + memcpy(ucode->kaddr, adev->gfx.rlc.save_restore_list_cntl, + ucode->ucode_size); + } else if (ucode->ucode_id == AMDGPU_UCODE_ID_RLC_RESTORE_LIST_GPM_MEM) { + ucode->ucode_size = adev->gfx.rlc.save_restore_list_gpm_size_bytes; + memcpy(ucode->kaddr, adev->gfx.rlc.save_restore_list_gpm, + ucode->ucode_size); + } else if (ucode->ucode_id == AMDGPU_UCODE_ID_RLC_RESTORE_LIST_SRM_MEM) { + ucode->ucode_size = adev->gfx.rlc.save_restore_list_srm_size_bytes; + memcpy(ucode->kaddr, adev->gfx.rlc.save_restore_list_srm, + ucode->ucode_size); } return 0; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.h index 30b5500dc152..08e38579af24 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.h @@ -98,6 +98,24 @@ struct rlc_firmware_header_v2_0 { uint32_t reg_list_separate_array_offset_bytes; /* payload offset from the start of the header */ }; +/* version_major=2, version_minor=1 */ +struct rlc_firmware_header_v2_1 { + struct rlc_firmware_header_v2_0 v2_0; + uint32_t reg_list_format_direct_reg_list_length; /* length of direct reg list format array */ + uint32_t save_restore_list_cntl_ucode_ver; + uint32_t save_restore_list_cntl_feature_ver; + uint32_t save_restore_list_cntl_size_bytes; + uint32_t save_restore_list_cntl_offset_bytes; + uint32_t save_restore_list_gpm_ucode_ver; + uint32_t save_restore_list_gpm_feature_ver; + uint32_t save_restore_list_gpm_size_bytes; + uint32_t save_restore_list_gpm_offset_bytes; + uint32_t save_restore_list_srm_ucode_ver; + uint32_t save_restore_list_srm_feature_ver; + uint32_t save_restore_list_srm_size_bytes; + uint32_t save_restore_list_srm_offset_bytes; +}; + /* version_major=1, version_minor=0 */ struct sdma_firmware_header_v1_0 { struct common_firmware_header header; @@ -148,6 +166,7 @@ union amdgpu_firmware_header { struct gfx_firmware_header_v1_0 gfx; struct rlc_firmware_header_v1_0 rlc; struct rlc_firmware_header_v2_0 rlc_v2_0; + struct rlc_firmware_header_v2_1 rlc_v2_1; struct sdma_firmware_header_v1_0 sdma; struct sdma_firmware_header_v1_1 sdma_v1_1; struct gpu_info_firmware_header_v1_0 gpu_info; @@ -168,6 +187,9 @@ enum AMDGPU_UCODE_ID { AMDGPU_UCODE_ID_CP_MEC2, AMDGPU_UCODE_ID_CP_MEC2_JT, AMDGPU_UCODE_ID_RLC_G, + AMDGPU_UCODE_ID_RLC_RESTORE_LIST_CNTL, + AMDGPU_UCODE_ID_RLC_RESTORE_LIST_GPM_MEM, + AMDGPU_UCODE_ID_RLC_RESTORE_LIST_SRM_MEM, AMDGPU_UCODE_ID_STORAGE, AMDGPU_UCODE_ID_SMC, AMDGPU_UCODE_ID_UVD, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c index 627542b22ae4..bcf68f80bbf0 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c @@ -66,15 +66,18 @@ #define FIRMWARE_POLARIS10 "amdgpu/polaris10_uvd.bin" #define FIRMWARE_POLARIS11 "amdgpu/polaris11_uvd.bin" #define FIRMWARE_POLARIS12 "amdgpu/polaris12_uvd.bin" +#define FIRMWARE_VEGAM "amdgpu/vegam_uvd.bin" #define FIRMWARE_VEGA10 "amdgpu/vega10_uvd.bin" #define FIRMWARE_VEGA12 "amdgpu/vega12_uvd.bin" +#define FIRMWARE_VEGA20 "amdgpu/vega20_uvd.bin" -#define mmUVD_GPCOM_VCPU_DATA0_VEGA10 (0x03c4 + 0x7e00) -#define mmUVD_GPCOM_VCPU_DATA1_VEGA10 (0x03c5 + 0x7e00) -#define mmUVD_GPCOM_VCPU_CMD_VEGA10 (0x03c3 + 0x7e00) -#define mmUVD_NO_OP_VEGA10 (0x03ff + 0x7e00) -#define mmUVD_ENGINE_CNTL_VEGA10 (0x03c6 + 0x7e00) +/* These are common relative offsets for all asics, from uvd_7_0_offset.h, */ +#define UVD_GPCOM_VCPU_CMD 0x03c3 +#define UVD_GPCOM_VCPU_DATA0 0x03c4 +#define UVD_GPCOM_VCPU_DATA1 0x03c5 +#define UVD_NO_OP 0x03ff +#define UVD_BASE_SI 0x3800 /** * amdgpu_uvd_cs_ctx - Command submission parser context @@ -109,9 +112,11 @@ MODULE_FIRMWARE(FIRMWARE_STONEY); MODULE_FIRMWARE(FIRMWARE_POLARIS10); MODULE_FIRMWARE(FIRMWARE_POLARIS11); MODULE_FIRMWARE(FIRMWARE_POLARIS12); +MODULE_FIRMWARE(FIRMWARE_VEGAM); MODULE_FIRMWARE(FIRMWARE_VEGA10); MODULE_FIRMWARE(FIRMWARE_VEGA12); +MODULE_FIRMWARE(FIRMWARE_VEGA20); static void amdgpu_uvd_idle_work_handler(struct work_struct *work); @@ -123,9 +128,9 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev) const char *fw_name; const struct common_firmware_header *hdr; unsigned version_major, version_minor, family_id; - int i, r; + int i, j, r; - INIT_DELAYED_WORK(&adev->uvd.idle_work, amdgpu_uvd_idle_work_handler); + INIT_DELAYED_WORK(&adev->uvd.inst->idle_work, amdgpu_uvd_idle_work_handler); switch (adev->asic_type) { #ifdef CONFIG_DRM_AMDGPU_CIK @@ -172,6 +177,12 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev) case CHIP_VEGA12: fw_name = FIRMWARE_VEGA12; break; + case CHIP_VEGAM: + fw_name = FIRMWARE_VEGAM; + break; + case CHIP_VEGA20: + fw_name = FIRMWARE_VEGA20; + break; default: return -EINVAL; } @@ -226,28 +237,30 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev) if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP) bo_size += AMDGPU_GPU_PAGE_ALIGN(le32_to_cpu(hdr->ucode_size_bytes) + 8); - r = amdgpu_bo_create_kernel(adev, bo_size, PAGE_SIZE, - AMDGPU_GEM_DOMAIN_VRAM, &adev->uvd.vcpu_bo, - &adev->uvd.gpu_addr, &adev->uvd.cpu_addr); - if (r) { - dev_err(adev->dev, "(%d) failed to allocate UVD bo\n", r); - return r; - } + for (j = 0; j < adev->uvd.num_uvd_inst; j++) { - ring = &adev->uvd.ring; - rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL]; - r = drm_sched_entity_init(&ring->sched, &adev->uvd.entity, - rq, amdgpu_sched_jobs, NULL); - if (r != 0) { - DRM_ERROR("Failed setting up UVD run queue.\n"); - return r; - } + r = amdgpu_bo_create_kernel(adev, bo_size, PAGE_SIZE, + AMDGPU_GEM_DOMAIN_VRAM, &adev->uvd.inst[j].vcpu_bo, + &adev->uvd.inst[j].gpu_addr, &adev->uvd.inst[j].cpu_addr); + if (r) { + dev_err(adev->dev, "(%d) failed to allocate UVD bo\n", r); + return r; + } - for (i = 0; i < adev->uvd.max_handles; ++i) { - atomic_set(&adev->uvd.handles[i], 0); - adev->uvd.filp[i] = NULL; - } + ring = &adev->uvd.inst[j].ring; + rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL]; + r = drm_sched_entity_init(&ring->sched, &adev->uvd.inst[j].entity, + rq, NULL); + if (r != 0) { + DRM_ERROR("Failed setting up UVD(%d) run queue.\n", j); + return r; + } + for (i = 0; i < adev->uvd.max_handles; ++i) { + atomic_set(&adev->uvd.inst[j].handles[i], 0); + adev->uvd.inst[j].filp[i] = NULL; + } + } /* from uvd v5.0 HW addressing capacity increased to 64 bits */ if (!amdgpu_device_ip_block_version_cmp(adev, AMD_IP_BLOCK_TYPE_UVD, 5, 0)) adev->uvd.address_64_bit = true; @@ -274,20 +287,22 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev) int amdgpu_uvd_sw_fini(struct amdgpu_device *adev) { - int i; - kfree(adev->uvd.saved_bo); + int i, j; - drm_sched_entity_fini(&adev->uvd.ring.sched, &adev->uvd.entity); + for (j = 0; j < adev->uvd.num_uvd_inst; ++j) { + kfree(adev->uvd.inst[j].saved_bo); - amdgpu_bo_free_kernel(&adev->uvd.vcpu_bo, - &adev->uvd.gpu_addr, - (void **)&adev->uvd.cpu_addr); + drm_sched_entity_fini(&adev->uvd.inst[j].ring.sched, &adev->uvd.inst[j].entity); - amdgpu_ring_fini(&adev->uvd.ring); + amdgpu_bo_free_kernel(&adev->uvd.inst[j].vcpu_bo, + &adev->uvd.inst[j].gpu_addr, + (void **)&adev->uvd.inst[j].cpu_addr); - for (i = 0; i < AMDGPU_MAX_UVD_ENC_RINGS; ++i) - amdgpu_ring_fini(&adev->uvd.ring_enc[i]); + amdgpu_ring_fini(&adev->uvd.inst[j].ring); + for (i = 0; i < AMDGPU_MAX_UVD_ENC_RINGS; ++i) + amdgpu_ring_fini(&adev->uvd.inst[j].ring_enc[i]); + } release_firmware(adev->uvd.fw); return 0; @@ -297,32 +312,33 @@ int amdgpu_uvd_suspend(struct amdgpu_device *adev) { unsigned size; void *ptr; - int i; - - if (adev->uvd.vcpu_bo == NULL) - return 0; + int i, j; - cancel_delayed_work_sync(&adev->uvd.idle_work); + for (j = 0; j < adev->uvd.num_uvd_inst; ++j) { + if (adev->uvd.inst[j].vcpu_bo == NULL) + continue; - /* only valid for physical mode */ - if (adev->asic_type < CHIP_POLARIS10) { - for (i = 0; i < adev->uvd.max_handles; ++i) - if (atomic_read(&adev->uvd.handles[i])) - break; + cancel_delayed_work_sync(&adev->uvd.inst[j].idle_work); - if (i == adev->uvd.max_handles) - return 0; - } + /* only valid for physical mode */ + if (adev->asic_type < CHIP_POLARIS10) { + for (i = 0; i < adev->uvd.max_handles; ++i) + if (atomic_read(&adev->uvd.inst[j].handles[i])) + break; - size = amdgpu_bo_size(adev->uvd.vcpu_bo); - ptr = adev->uvd.cpu_addr; + if (i == adev->uvd.max_handles) + continue; + } - adev->uvd.saved_bo = kmalloc(size, GFP_KERNEL); - if (!adev->uvd.saved_bo) - return -ENOMEM; + size = amdgpu_bo_size(adev->uvd.inst[j].vcpu_bo); + ptr = adev->uvd.inst[j].cpu_addr; - memcpy_fromio(adev->uvd.saved_bo, ptr, size); + adev->uvd.inst[j].saved_bo = kmalloc(size, GFP_KERNEL); + if (!adev->uvd.inst[j].saved_bo) + return -ENOMEM; + memcpy_fromio(adev->uvd.inst[j].saved_bo, ptr, size); + } return 0; } @@ -330,59 +346,65 @@ int amdgpu_uvd_resume(struct amdgpu_device *adev) { unsigned size; void *ptr; + int i; - if (adev->uvd.vcpu_bo == NULL) - return -EINVAL; + for (i = 0; i < adev->uvd.num_uvd_inst; i++) { + if (adev->uvd.inst[i].vcpu_bo == NULL) + return -EINVAL; - size = amdgpu_bo_size(adev->uvd.vcpu_bo); - ptr = adev->uvd.cpu_addr; + size = amdgpu_bo_size(adev->uvd.inst[i].vcpu_bo); + ptr = adev->uvd.inst[i].cpu_addr; - if (adev->uvd.saved_bo != NULL) { - memcpy_toio(ptr, adev->uvd.saved_bo, size); - kfree(adev->uvd.saved_bo); - adev->uvd.saved_bo = NULL; - } else { - const struct common_firmware_header *hdr; - unsigned offset; - - hdr = (const struct common_firmware_header *)adev->uvd.fw->data; - if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP) { - offset = le32_to_cpu(hdr->ucode_array_offset_bytes); - memcpy_toio(adev->uvd.cpu_addr, adev->uvd.fw->data + offset, - le32_to_cpu(hdr->ucode_size_bytes)); - size -= le32_to_cpu(hdr->ucode_size_bytes); - ptr += le32_to_cpu(hdr->ucode_size_bytes); + if (adev->uvd.inst[i].saved_bo != NULL) { + memcpy_toio(ptr, adev->uvd.inst[i].saved_bo, size); + kfree(adev->uvd.inst[i].saved_bo); + adev->uvd.inst[i].saved_bo = NULL; + } else { + const struct common_firmware_header *hdr; + unsigned offset; + + hdr = (const struct common_firmware_header *)adev->uvd.fw->data; + if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP) { + offset = le32_to_cpu(hdr->ucode_array_offset_bytes); + memcpy_toio(adev->uvd.inst[i].cpu_addr, adev->uvd.fw->data + offset, + le32_to_cpu(hdr->ucode_size_bytes)); + size -= le32_to_cpu(hdr->ucode_size_bytes); + ptr += le32_to_cpu(hdr->ucode_size_bytes); + } + memset_io(ptr, 0, size); + /* to restore uvd fence seq */ + amdgpu_fence_driver_force_completion(&adev->uvd.inst[i].ring); } - memset_io(ptr, 0, size); - /* to restore uvd fence seq */ - amdgpu_fence_driver_force_completion(&adev->uvd.ring); } - return 0; } void amdgpu_uvd_free_handles(struct amdgpu_device *adev, struct drm_file *filp) { - struct amdgpu_ring *ring = &adev->uvd.ring; - int i, r; + struct amdgpu_ring *ring; + int i, j, r; - for (i = 0; i < adev->uvd.max_handles; ++i) { - uint32_t handle = atomic_read(&adev->uvd.handles[i]); - if (handle != 0 && adev->uvd.filp[i] == filp) { - struct dma_fence *fence; - - r = amdgpu_uvd_get_destroy_msg(ring, handle, - false, &fence); - if (r) { - DRM_ERROR("Error destroying UVD (%d)!\n", r); - continue; - } + for (j = 0; j < adev->uvd.num_uvd_inst; j++) { + ring = &adev->uvd.inst[j].ring; + + for (i = 0; i < adev->uvd.max_handles; ++i) { + uint32_t handle = atomic_read(&adev->uvd.inst[j].handles[i]); + if (handle != 0 && adev->uvd.inst[j].filp[i] == filp) { + struct dma_fence *fence; + + r = amdgpu_uvd_get_destroy_msg(ring, handle, + false, &fence); + if (r) { + DRM_ERROR("Error destroying UVD(%d) %d!\n", j, r); + continue; + } - dma_fence_wait(fence, false); - dma_fence_put(fence); + dma_fence_wait(fence, false); + dma_fence_put(fence); - adev->uvd.filp[i] = NULL; - atomic_set(&adev->uvd.handles[i], 0); + adev->uvd.inst[j].filp[i] = NULL; + atomic_set(&adev->uvd.inst[j].handles[i], 0); + } } } } @@ -657,15 +679,16 @@ static int amdgpu_uvd_cs_msg(struct amdgpu_uvd_cs_ctx *ctx, void *ptr; long r; int i; + uint32_t ip_instance = ctx->parser->job->ring->me; if (offset & 0x3F) { - DRM_ERROR("UVD messages must be 64 byte aligned!\n"); + DRM_ERROR("UVD(%d) messages must be 64 byte aligned!\n", ip_instance); return -EINVAL; } r = amdgpu_bo_kmap(bo, &ptr); if (r) { - DRM_ERROR("Failed mapping the UVD message (%ld)!\n", r); + DRM_ERROR("Failed mapping the UVD(%d) message (%ld)!\n", ip_instance, r); return r; } @@ -675,7 +698,7 @@ static int amdgpu_uvd_cs_msg(struct amdgpu_uvd_cs_ctx *ctx, handle = msg[2]; if (handle == 0) { - DRM_ERROR("Invalid UVD handle!\n"); + DRM_ERROR("Invalid UVD(%d) handle!\n", ip_instance); return -EINVAL; } @@ -686,18 +709,18 @@ static int amdgpu_uvd_cs_msg(struct amdgpu_uvd_cs_ctx *ctx, /* try to alloc a new handle */ for (i = 0; i < adev->uvd.max_handles; ++i) { - if (atomic_read(&adev->uvd.handles[i]) == handle) { - DRM_ERROR("Handle 0x%x already in use!\n", handle); + if (atomic_read(&adev->uvd.inst[ip_instance].handles[i]) == handle) { + DRM_ERROR("(%d)Handle 0x%x already in use!\n", ip_instance, handle); return -EINVAL; } - if (!atomic_cmpxchg(&adev->uvd.handles[i], 0, handle)) { - adev->uvd.filp[i] = ctx->parser->filp; + if (!atomic_cmpxchg(&adev->uvd.inst[ip_instance].handles[i], 0, handle)) { + adev->uvd.inst[ip_instance].filp[i] = ctx->parser->filp; return 0; } } - DRM_ERROR("No more free UVD handles!\n"); + DRM_ERROR("No more free UVD(%d) handles!\n", ip_instance); return -ENOSPC; case 1: @@ -709,27 +732,27 @@ static int amdgpu_uvd_cs_msg(struct amdgpu_uvd_cs_ctx *ctx, /* validate the handle */ for (i = 0; i < adev->uvd.max_handles; ++i) { - if (atomic_read(&adev->uvd.handles[i]) == handle) { - if (adev->uvd.filp[i] != ctx->parser->filp) { - DRM_ERROR("UVD handle collision detected!\n"); + if (atomic_read(&adev->uvd.inst[ip_instance].handles[i]) == handle) { + if (adev->uvd.inst[ip_instance].filp[i] != ctx->parser->filp) { + DRM_ERROR("UVD(%d) handle collision detected!\n", ip_instance); return -EINVAL; } return 0; } } - DRM_ERROR("Invalid UVD handle 0x%x!\n", handle); + DRM_ERROR("Invalid UVD(%d) handle 0x%x!\n", ip_instance, handle); return -ENOENT; case 2: /* it's a destroy msg, free the handle */ for (i = 0; i < adev->uvd.max_handles; ++i) - atomic_cmpxchg(&adev->uvd.handles[i], handle, 0); + atomic_cmpxchg(&adev->uvd.inst[ip_instance].handles[i], handle, 0); amdgpu_bo_kunmap(bo); return 0; default: - DRM_ERROR("Illegal UVD message type (%d)!\n", msg_type); + DRM_ERROR("Illegal UVD(%d) message type (%d)!\n", ip_instance, msg_type); return -EINVAL; } BUG(); @@ -800,7 +823,7 @@ static int amdgpu_uvd_cs_pass2(struct amdgpu_uvd_cs_ctx *ctx) } if ((cmd == 0 || cmd == 0x3) && - (start >> 28) != (ctx->parser->adev->uvd.gpu_addr >> 28)) { + (start >> 28) != (ctx->parser->adev->uvd.inst->gpu_addr >> 28)) { DRM_ERROR("msg/fb buffer %LX-%LX out of 256MB segment!\n", start, end); return -EINVAL; @@ -968,6 +991,8 @@ static int amdgpu_uvd_send_msg(struct amdgpu_ring *ring, struct amdgpu_bo *bo, uint64_t addr; long r; int i; + unsigned offset_idx = 0; + unsigned offset[3] = { UVD_BASE_SI, 0, 0 }; amdgpu_bo_kunmap(bo); amdgpu_bo_unpin(bo); @@ -987,17 +1012,16 @@ static int amdgpu_uvd_send_msg(struct amdgpu_ring *ring, struct amdgpu_bo *bo, goto err; if (adev->asic_type >= CHIP_VEGA10) { - data[0] = PACKET0(mmUVD_GPCOM_VCPU_DATA0_VEGA10, 0); - data[1] = PACKET0(mmUVD_GPCOM_VCPU_DATA1_VEGA10, 0); - data[2] = PACKET0(mmUVD_GPCOM_VCPU_CMD_VEGA10, 0); - data[3] = PACKET0(mmUVD_NO_OP_VEGA10, 0); - } else { - data[0] = PACKET0(mmUVD_GPCOM_VCPU_DATA0, 0); - data[1] = PACKET0(mmUVD_GPCOM_VCPU_DATA1, 0); - data[2] = PACKET0(mmUVD_GPCOM_VCPU_CMD, 0); - data[3] = PACKET0(mmUVD_NO_OP, 0); + offset_idx = 1 + ring->me; + offset[1] = adev->reg_offset[UVD_HWIP][0][1]; + offset[2] = adev->reg_offset[UVD_HWIP][1][1]; } + data[0] = PACKET0(offset[offset_idx] + UVD_GPCOM_VCPU_DATA0, 0); + data[1] = PACKET0(offset[offset_idx] + UVD_GPCOM_VCPU_DATA1, 0); + data[2] = PACKET0(offset[offset_idx] + UVD_GPCOM_VCPU_CMD, 0); + data[3] = PACKET0(offset[offset_idx] + UVD_NO_OP, 0); + ib = &job->ibs[0]; addr = amdgpu_bo_gpu_offset(bo); ib->ptr[0] = data[0]; @@ -1033,7 +1057,7 @@ static int amdgpu_uvd_send_msg(struct amdgpu_ring *ring, struct amdgpu_bo *bo, if (r) goto err_free; - r = amdgpu_job_submit(job, ring, &adev->uvd.entity, + r = amdgpu_job_submit(job, ring, &adev->uvd.inst[ring->me].entity, AMDGPU_FENCE_OWNER_UNDEFINED, &f); if (r) goto err_free; @@ -1121,8 +1145,15 @@ int amdgpu_uvd_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle, static void amdgpu_uvd_idle_work_handler(struct work_struct *work) { struct amdgpu_device *adev = - container_of(work, struct amdgpu_device, uvd.idle_work.work); - unsigned fences = amdgpu_fence_count_emitted(&adev->uvd.ring); + container_of(work, struct amdgpu_device, uvd.inst->idle_work.work); + unsigned fences = 0, i, j; + + for (i = 0; i < adev->uvd.num_uvd_inst; ++i) { + fences += amdgpu_fence_count_emitted(&adev->uvd.inst[i].ring); + for (j = 0; j < adev->uvd.num_enc_rings; ++j) { + fences += amdgpu_fence_count_emitted(&adev->uvd.inst[i].ring_enc[j]); + } + } if (fences == 0) { if (adev->pm.dpm_enabled) { @@ -1136,7 +1167,7 @@ static void amdgpu_uvd_idle_work_handler(struct work_struct *work) AMD_CG_STATE_GATE); } } else { - schedule_delayed_work(&adev->uvd.idle_work, UVD_IDLE_TIMEOUT); + schedule_delayed_work(&adev->uvd.inst->idle_work, UVD_IDLE_TIMEOUT); } } @@ -1148,7 +1179,7 @@ void amdgpu_uvd_ring_begin_use(struct amdgpu_ring *ring) if (amdgpu_sriov_vf(adev)) return; - set_clocks = !cancel_delayed_work_sync(&adev->uvd.idle_work); + set_clocks = !cancel_delayed_work_sync(&adev->uvd.inst->idle_work); if (set_clocks) { if (adev->pm.dpm_enabled) { amdgpu_dpm_enable_uvd(adev, true); @@ -1165,7 +1196,7 @@ void amdgpu_uvd_ring_begin_use(struct amdgpu_ring *ring) void amdgpu_uvd_ring_end_use(struct amdgpu_ring *ring) { if (!amdgpu_sriov_vf(ring->adev)) - schedule_delayed_work(&ring->adev->uvd.idle_work, UVD_IDLE_TIMEOUT); + schedule_delayed_work(&ring->adev->uvd.inst->idle_work, UVD_IDLE_TIMEOUT); } /** @@ -1179,27 +1210,28 @@ int amdgpu_uvd_ring_test_ib(struct amdgpu_ring *ring, long timeout) { struct dma_fence *fence; long r; + uint32_t ip_instance = ring->me; r = amdgpu_uvd_get_create_msg(ring, 1, NULL); if (r) { - DRM_ERROR("amdgpu: failed to get create msg (%ld).\n", r); + DRM_ERROR("amdgpu: (%d)failed to get create msg (%ld).\n", ip_instance, r); goto error; } r = amdgpu_uvd_get_destroy_msg(ring, 1, true, &fence); if (r) { - DRM_ERROR("amdgpu: failed to get destroy ib (%ld).\n", r); + DRM_ERROR("amdgpu: (%d)failed to get destroy ib (%ld).\n", ip_instance, r); goto error; } r = dma_fence_wait_timeout(fence, false, timeout); if (r == 0) { - DRM_ERROR("amdgpu: IB test timed out.\n"); + DRM_ERROR("amdgpu: (%d)IB test timed out.\n", ip_instance); r = -ETIMEDOUT; } else if (r < 0) { - DRM_ERROR("amdgpu: fence wait failed (%ld).\n", r); + DRM_ERROR("amdgpu: (%d)fence wait failed (%ld).\n", ip_instance, r); } else { - DRM_DEBUG("ib test on ring %d succeeded\n", ring->idx); + DRM_DEBUG("ib test on (%d)ring %d succeeded\n", ip_instance, ring->idx); r = 0; } @@ -1227,7 +1259,7 @@ uint32_t amdgpu_uvd_used_handles(struct amdgpu_device *adev) * necessarily linear. So we need to count * all non-zero handles. */ - if (atomic_read(&adev->uvd.handles[i])) + if (atomic_read(&adev->uvd.inst->handles[i])) used_handles++; } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.h index 32ea20b99e53..b1579fba134c 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.h @@ -31,30 +31,37 @@ #define AMDGPU_UVD_SESSION_SIZE (50*1024) #define AMDGPU_UVD_FIRMWARE_OFFSET 256 +#define AMDGPU_MAX_UVD_INSTANCES 2 + #define AMDGPU_UVD_FIRMWARE_SIZE(adev) \ (AMDGPU_GPU_PAGE_ALIGN(le32_to_cpu(((const struct common_firmware_header *)(adev)->uvd.fw->data)->ucode_size_bytes) + \ 8) - AMDGPU_UVD_FIRMWARE_OFFSET) -struct amdgpu_uvd { +struct amdgpu_uvd_inst { struct amdgpu_bo *vcpu_bo; void *cpu_addr; uint64_t gpu_addr; - unsigned fw_version; void *saved_bo; - unsigned max_handles; atomic_t handles[AMDGPU_MAX_UVD_HANDLES]; struct drm_file *filp[AMDGPU_MAX_UVD_HANDLES]; struct delayed_work idle_work; - const struct firmware *fw; /* UVD firmware */ struct amdgpu_ring ring; struct amdgpu_ring ring_enc[AMDGPU_MAX_UVD_ENC_RINGS]; struct amdgpu_irq_src irq; - bool address_64_bit; - bool use_ctx_buf; struct drm_sched_entity entity; struct drm_sched_entity entity_enc; uint32_t srbm_soft_reset; +}; + +struct amdgpu_uvd { + const struct firmware *fw; /* UVD firmware */ + unsigned fw_version; + unsigned max_handles; unsigned num_enc_rings; + uint8_t num_uvd_inst; + bool address_64_bit; + bool use_ctx_buf; + struct amdgpu_uvd_inst inst[AMDGPU_MAX_UVD_INSTANCES]; }; int amdgpu_uvd_sw_init(struct amdgpu_device *adev); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c index a33804bd3314..23d960ec1cf2 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c @@ -51,11 +51,13 @@ #define FIRMWARE_FIJI "amdgpu/fiji_vce.bin" #define FIRMWARE_STONEY "amdgpu/stoney_vce.bin" #define FIRMWARE_POLARIS10 "amdgpu/polaris10_vce.bin" -#define FIRMWARE_POLARIS11 "amdgpu/polaris11_vce.bin" -#define FIRMWARE_POLARIS12 "amdgpu/polaris12_vce.bin" +#define FIRMWARE_POLARIS11 "amdgpu/polaris11_vce.bin" +#define FIRMWARE_POLARIS12 "amdgpu/polaris12_vce.bin" +#define FIRMWARE_VEGAM "amdgpu/vegam_vce.bin" #define FIRMWARE_VEGA10 "amdgpu/vega10_vce.bin" #define FIRMWARE_VEGA12 "amdgpu/vega12_vce.bin" +#define FIRMWARE_VEGA20 "amdgpu/vega20_vce.bin" #ifdef CONFIG_DRM_AMDGPU_CIK MODULE_FIRMWARE(FIRMWARE_BONAIRE); @@ -71,9 +73,11 @@ MODULE_FIRMWARE(FIRMWARE_STONEY); MODULE_FIRMWARE(FIRMWARE_POLARIS10); MODULE_FIRMWARE(FIRMWARE_POLARIS11); MODULE_FIRMWARE(FIRMWARE_POLARIS12); +MODULE_FIRMWARE(FIRMWARE_VEGAM); MODULE_FIRMWARE(FIRMWARE_VEGA10); MODULE_FIRMWARE(FIRMWARE_VEGA12); +MODULE_FIRMWARE(FIRMWARE_VEGA20); static void amdgpu_vce_idle_work_handler(struct work_struct *work); @@ -132,12 +136,18 @@ int amdgpu_vce_sw_init(struct amdgpu_device *adev, unsigned long size) case CHIP_POLARIS12: fw_name = FIRMWARE_POLARIS12; break; + case CHIP_VEGAM: + fw_name = FIRMWARE_VEGAM; + break; case CHIP_VEGA10: fw_name = FIRMWARE_VEGA10; break; case CHIP_VEGA12: fw_name = FIRMWARE_VEGA12; break; + case CHIP_VEGA20: + fw_name = FIRMWARE_VEGA20; + break; default: return -EINVAL; @@ -181,7 +191,7 @@ int amdgpu_vce_sw_init(struct amdgpu_device *adev, unsigned long size) ring = &adev->vce.ring[0]; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL]; r = drm_sched_entity_init(&ring->sched, &adev->vce.entity, - rq, amdgpu_sched_jobs, NULL); + rq, NULL); if (r != 0) { DRM_ERROR("Failed setting up VCE run queue.\n"); return r; @@ -755,6 +765,18 @@ int amdgpu_vce_ring_parse_cs(struct amdgpu_cs_parser *p, uint32_t ib_idx) if (r) goto out; break; + + case 0x0500000d: /* MV buffer */ + r = amdgpu_vce_validate_bo(p, ib_idx, idx + 3, + idx + 2, 0, 0); + if (r) + goto out; + + r = amdgpu_vce_validate_bo(p, ib_idx, idx + 8, + idx + 7, 0, 0); + if (r) + goto out; + break; } idx += len / 4; @@ -860,6 +882,18 @@ int amdgpu_vce_ring_parse_cs(struct amdgpu_cs_parser *p, uint32_t ib_idx) goto out; break; + case 0x0500000d: /* MV buffer */ + r = amdgpu_vce_cs_reloc(p, ib_idx, idx + 3, + idx + 2, *size, 0); + if (r) + goto out; + + r = amdgpu_vce_cs_reloc(p, ib_idx, idx + 8, + idx + 7, *size / 12, 0); + if (r) + goto out; + break; + default: DRM_ERROR("invalid VCE command (0x%x)!\n", cmd); r = -EINVAL; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c index 58e495330b38..127e87b470ff 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c @@ -49,8 +49,6 @@ static void amdgpu_vcn_idle_work_handler(struct work_struct *work); int amdgpu_vcn_sw_init(struct amdgpu_device *adev) { - struct amdgpu_ring *ring; - struct drm_sched_rq *rq; unsigned long bo_size; const char *fw_name; const struct common_firmware_header *hdr; @@ -84,6 +82,7 @@ int amdgpu_vcn_sw_init(struct amdgpu_device *adev) } hdr = (const struct common_firmware_header *)adev->vcn.fw->data; + adev->vcn.fw_version = le32_to_cpu(hdr->ucode_version); family_id = le32_to_cpu(hdr->ucode_version) & 0xff; version_major = (le32_to_cpu(hdr->ucode_version) >> 24) & 0xff; version_minor = (le32_to_cpu(hdr->ucode_version) >> 8) & 0xff; @@ -102,24 +101,6 @@ int amdgpu_vcn_sw_init(struct amdgpu_device *adev) return r; } - ring = &adev->vcn.ring_dec; - rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL]; - r = drm_sched_entity_init(&ring->sched, &adev->vcn.entity_dec, - rq, amdgpu_sched_jobs, NULL); - if (r != 0) { - DRM_ERROR("Failed setting up VCN dec run queue.\n"); - return r; - } - - ring = &adev->vcn.ring_enc[0]; - rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL]; - r = drm_sched_entity_init(&ring->sched, &adev->vcn.entity_enc, - rq, amdgpu_sched_jobs, NULL); - if (r != 0) { - DRM_ERROR("Failed setting up VCN enc run queue.\n"); - return r; - } - return 0; } @@ -129,10 +110,6 @@ int amdgpu_vcn_sw_fini(struct amdgpu_device *adev) kfree(adev->vcn.saved_bo); - drm_sched_entity_fini(&adev->vcn.ring_dec.sched, &adev->vcn.entity_dec); - - drm_sched_entity_fini(&adev->vcn.ring_enc[0].sched, &adev->vcn.entity_enc); - amdgpu_bo_free_kernel(&adev->vcn.vcpu_bo, &adev->vcn.gpu_addr, (void **)&adev->vcn.cpu_addr); @@ -205,13 +182,18 @@ static void amdgpu_vcn_idle_work_handler(struct work_struct *work) struct amdgpu_device *adev = container_of(work, struct amdgpu_device, vcn.idle_work.work); unsigned fences = amdgpu_fence_count_emitted(&adev->vcn.ring_dec); + unsigned i; + + for (i = 0; i < adev->vcn.num_enc_rings; ++i) { + fences += amdgpu_fence_count_emitted(&adev->vcn.ring_enc[i]); + } if (fences == 0) { - if (adev->pm.dpm_enabled) { - /* might be used when with pg/cg + if (adev->pm.dpm_enabled) amdgpu_dpm_enable_uvd(adev, false); - */ - } + else + amdgpu_device_ip_set_powergating_state(adev, AMD_IP_BLOCK_TYPE_VCN, + AMD_PG_STATE_GATE); } else { schedule_delayed_work(&adev->vcn.idle_work, VCN_IDLE_TIMEOUT); } @@ -223,9 +205,11 @@ void amdgpu_vcn_ring_begin_use(struct amdgpu_ring *ring) bool set_clocks = !cancel_delayed_work_sync(&adev->vcn.idle_work); if (set_clocks && adev->pm.dpm_enabled) { - /* might be used when with pg/cg - amdgpu_dpm_enable_uvd(adev, true); - */ + if (adev->pm.dpm_enabled) + amdgpu_dpm_enable_uvd(adev, true); + else + amdgpu_device_ip_set_powergating_state(adev, AMD_IP_BLOCK_TYPE_VCN, + AMD_PG_STATE_UNGATE); } } @@ -271,7 +255,7 @@ int amdgpu_vcn_dec_ring_test_ring(struct amdgpu_ring *ring) } static int amdgpu_vcn_dec_send_msg(struct amdgpu_ring *ring, - struct amdgpu_bo *bo, bool direct, + struct amdgpu_bo *bo, struct dma_fence **fence) { struct amdgpu_device *adev = ring->adev; @@ -299,19 +283,12 @@ static int amdgpu_vcn_dec_send_msg(struct amdgpu_ring *ring, } ib->length_dw = 16; - if (direct) { - r = amdgpu_ib_schedule(ring, 1, ib, NULL, &f); - job->fence = dma_fence_get(f); - if (r) - goto err_free; + r = amdgpu_ib_schedule(ring, 1, ib, NULL, &f); + job->fence = dma_fence_get(f); + if (r) + goto err_free; - amdgpu_job_free(job); - } else { - r = amdgpu_job_submit(job, ring, &adev->vcn.entity_dec, - AMDGPU_FENCE_OWNER_UNDEFINED, &f); - if (r) - goto err_free; - } + amdgpu_job_free(job); amdgpu_bo_fence(bo, f, false); amdgpu_bo_unreserve(bo); @@ -363,11 +340,11 @@ static int amdgpu_vcn_dec_get_create_msg(struct amdgpu_ring *ring, uint32_t hand for (i = 14; i < 1024; ++i) msg[i] = cpu_to_le32(0x0); - return amdgpu_vcn_dec_send_msg(ring, bo, true, fence); + return amdgpu_vcn_dec_send_msg(ring, bo, fence); } static int amdgpu_vcn_dec_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle, - bool direct, struct dma_fence **fence) + struct dma_fence **fence) { struct amdgpu_device *adev = ring->adev; struct amdgpu_bo *bo = NULL; @@ -389,7 +366,7 @@ static int amdgpu_vcn_dec_get_destroy_msg(struct amdgpu_ring *ring, uint32_t han for (i = 6; i < 1024; ++i) msg[i] = cpu_to_le32(0x0); - return amdgpu_vcn_dec_send_msg(ring, bo, direct, fence); + return amdgpu_vcn_dec_send_msg(ring, bo, fence); } int amdgpu_vcn_dec_ring_test_ib(struct amdgpu_ring *ring, long timeout) @@ -403,7 +380,7 @@ int amdgpu_vcn_dec_ring_test_ib(struct amdgpu_ring *ring, long timeout) goto error; } - r = amdgpu_vcn_dec_get_destroy_msg(ring, 1, true, &fence); + r = amdgpu_vcn_dec_get_destroy_msg(ring, 1, &fence); if (r) { DRM_ERROR("amdgpu: failed to get destroy ib (%ld).\n", r); goto error; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h index 2fd7db891689..773010b9ff15 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h @@ -45,6 +45,17 @@ #define VCN_ENC_CMD_REG_WRITE 0x0000000b #define VCN_ENC_CMD_REG_WAIT 0x0000000c +enum engine_status_constants { + UVD_PGFSM_STATUS__UVDM_UVDU_PWR_ON = 0x2AAAA0, + UVD_PGFSM_CONFIG__UVDM_UVDU_PWR_ON = 0x00000002, + UVD_STATUS__UVD_BUSY = 0x00000004, + GB_ADDR_CONFIG_DEFAULT = 0x26010011, + UVD_STATUS__IDLE = 0x2, + UVD_STATUS__BUSY = 0x5, + UVD_POWER_STATUS__UVD_POWER_STATUS_TILES_OFF = 0x1, + UVD_STATUS__RBC_BUSY = 0x1, +}; + struct amdgpu_vcn { struct amdgpu_bo *vcpu_bo; void *cpu_addr; @@ -56,8 +67,6 @@ struct amdgpu_vcn { struct amdgpu_ring ring_dec; struct amdgpu_ring ring_enc[AMDGPU_VCN_MAX_ENC_RINGS]; struct amdgpu_irq_src irq; - struct drm_sched_entity entity_dec; - struct drm_sched_entity entity_enc; unsigned num_enc_rings; }; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c index da55a78d7380..b0eb2f537392 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c @@ -94,6 +94,34 @@ struct amdgpu_prt_cb { struct dma_fence_cb cb; }; +static void amdgpu_vm_bo_base_init(struct amdgpu_vm_bo_base *base, + struct amdgpu_vm *vm, + struct amdgpu_bo *bo) +{ + base->vm = vm; + base->bo = bo; + INIT_LIST_HEAD(&base->bo_list); + INIT_LIST_HEAD(&base->vm_status); + + if (!bo) + return; + list_add_tail(&base->bo_list, &bo->va); + + if (bo->tbo.resv != vm->root.base.bo->tbo.resv) + return; + + if (bo->preferred_domains & + amdgpu_mem_type_to_domain(bo->tbo.mem.mem_type)) + return; + + /* + * we checked all the prerequisites, but it looks like this per vm bo + * is currently evicted. add the bo to the evicted list to make sure it + * is validated on next vm use to avoid fault. + * */ + list_move_tail(&base->vm_status, &vm->evicted); +} + /** * amdgpu_vm_level_shift - return the addr shift for each level * @@ -196,24 +224,16 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm, void *param) { struct ttm_bo_global *glob = adev->mman.bdev.glob; - int r; + struct amdgpu_vm_bo_base *bo_base, *tmp; + int r = 0; - spin_lock(&vm->status_lock); - while (!list_empty(&vm->evicted)) { - struct amdgpu_vm_bo_base *bo_base; - struct amdgpu_bo *bo; + list_for_each_entry_safe(bo_base, tmp, &vm->evicted, vm_status) { + struct amdgpu_bo *bo = bo_base->bo; - bo_base = list_first_entry(&vm->evicted, - struct amdgpu_vm_bo_base, - vm_status); - spin_unlock(&vm->status_lock); - - bo = bo_base->bo; - BUG_ON(!bo); if (bo->parent) { r = validate(param, bo); if (r) - return r; + break; spin_lock(&glob->lru_lock); ttm_bo_move_to_lru_tail(&bo->tbo); @@ -222,22 +242,29 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm, spin_unlock(&glob->lru_lock); } - if (bo->tbo.type == ttm_bo_type_kernel && - vm->use_cpu_for_update) { - r = amdgpu_bo_kmap(bo, NULL); - if (r) - return r; - } - - spin_lock(&vm->status_lock); - if (bo->tbo.type != ttm_bo_type_kernel) + if (bo->tbo.type != ttm_bo_type_kernel) { + spin_lock(&vm->moved_lock); list_move(&bo_base->vm_status, &vm->moved); - else + spin_unlock(&vm->moved_lock); + } else { list_move(&bo_base->vm_status, &vm->relocated); + } } - spin_unlock(&vm->status_lock); - return 0; + spin_lock(&glob->lru_lock); + list_for_each_entry(bo_base, &vm->idle, vm_status) { + struct amdgpu_bo *bo = bo_base->bo; + + if (!bo->parent) + continue; + + ttm_bo_move_to_lru_tail(&bo->tbo); + if (bo->shadow) + ttm_bo_move_to_lru_tail(&bo->shadow->tbo); + } + spin_unlock(&glob->lru_lock); + + return r; } /** @@ -249,13 +276,7 @@ int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm, */ bool amdgpu_vm_ready(struct amdgpu_vm *vm) { - bool ready; - - spin_lock(&vm->status_lock); - ready = list_empty(&vm->evicted); - spin_unlock(&vm->status_lock); - - return ready; + return list_empty(&vm->evicted); } /** @@ -412,11 +433,16 @@ static int amdgpu_vm_alloc_levels(struct amdgpu_device *adev, struct amdgpu_bo *pt; if (!entry->base.bo) { - r = amdgpu_bo_create(adev, - amdgpu_vm_bo_size(adev, level), - AMDGPU_GPU_PAGE_SIZE, - AMDGPU_GEM_DOMAIN_VRAM, flags, - ttm_bo_type_kernel, resv, &pt); + struct amdgpu_bo_param bp; + + memset(&bp, 0, sizeof(bp)); + bp.size = amdgpu_vm_bo_size(adev, level); + bp.byte_align = AMDGPU_GPU_PAGE_SIZE; + bp.domain = AMDGPU_GEM_DOMAIN_VRAM; + bp.flags = flags; + bp.type = ttm_bo_type_kernel; + bp.resv = resv; + r = amdgpu_bo_create(adev, &bp, &pt); if (r) return r; @@ -441,12 +467,8 @@ static int amdgpu_vm_alloc_levels(struct amdgpu_device *adev, */ pt->parent = amdgpu_bo_ref(parent->base.bo); - entry->base.vm = vm; - entry->base.bo = pt; - list_add_tail(&entry->base.bo_list, &pt->va); - spin_lock(&vm->status_lock); - list_add(&entry->base.vm_status, &vm->relocated); - spin_unlock(&vm->status_lock); + amdgpu_vm_bo_base_init(&entry->base, vm, pt); + list_move(&entry->base.vm_status, &vm->relocated); } if (level < AMDGPU_VM_PTB) { @@ -628,7 +650,7 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, bool need_ amdgpu_gmc_emit_pasid_mapping(ring, job->vmid, job->pasid); if (vm_flush_needed || pasid_mapping_needed) { - r = amdgpu_fence_emit(ring, &fence); + r = amdgpu_fence_emit(ring, &fence, 0); if (r) return r; } @@ -893,10 +915,8 @@ static void amdgpu_vm_invalidate_level(struct amdgpu_device *adev, if (!entry->base.bo) continue; - spin_lock(&vm->status_lock); - if (list_empty(&entry->base.vm_status)) - list_add(&entry->base.vm_status, &vm->relocated); - spin_unlock(&vm->status_lock); + if (!entry->base.moved) + list_move(&entry->base.vm_status, &vm->relocated); amdgpu_vm_invalidate_level(adev, vm, entry, level + 1); } } @@ -926,6 +946,14 @@ restart: params.adev = adev; if (vm->use_cpu_for_update) { + struct amdgpu_vm_bo_base *bo_base; + + list_for_each_entry(bo_base, &vm->relocated, vm_status) { + r = amdgpu_bo_kmap(bo_base->bo, NULL); + if (unlikely(r)) + return r; + } + r = amdgpu_vm_wait_pd(adev, vm, AMDGPU_FENCE_OWNER_VM); if (unlikely(r)) return r; @@ -941,7 +969,6 @@ restart: params.func = amdgpu_vm_do_set_ptes; } - spin_lock(&vm->status_lock); while (!list_empty(&vm->relocated)) { struct amdgpu_vm_bo_base *bo_base, *parent; struct amdgpu_vm_pt *pt, *entry; @@ -950,14 +977,12 @@ restart: bo_base = list_first_entry(&vm->relocated, struct amdgpu_vm_bo_base, vm_status); - list_del_init(&bo_base->vm_status); - spin_unlock(&vm->status_lock); + bo_base->moved = false; + list_move(&bo_base->vm_status, &vm->idle); bo = bo_base->bo->parent; - if (!bo) { - spin_lock(&vm->status_lock); + if (!bo) continue; - } parent = list_first_entry(&bo->va, struct amdgpu_vm_bo_base, bo_list); @@ -966,12 +991,10 @@ restart: amdgpu_vm_update_pde(¶ms, vm, pt, entry); - spin_lock(&vm->status_lock); if (!vm->use_cpu_for_update && (ndw - params.ib->length_dw) < 32) break; } - spin_unlock(&vm->status_lock); if (vm->use_cpu_for_update) { /* Flush HDP */ @@ -1074,9 +1097,7 @@ static void amdgpu_vm_handle_huge_pages(struct amdgpu_pte_update_params *p, if (entry->huge) { /* Add the entry to the relocated list to update it. */ entry->huge = false; - spin_lock(&p->vm->status_lock); list_move(&entry->base.vm_status, &p->vm->relocated); - spin_unlock(&p->vm->status_lock); } return; } @@ -1555,9 +1576,22 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev, amdgpu_asic_flush_hdp(adev, NULL); } - spin_lock(&vm->status_lock); + spin_lock(&vm->moved_lock); list_del_init(&bo_va->base.vm_status); - spin_unlock(&vm->status_lock); + spin_unlock(&vm->moved_lock); + + /* If the BO is not in its preferred location add it back to + * the evicted list so that it gets validated again on the + * next command submission. + */ + if (bo && bo->tbo.resv == vm->root.base.bo->tbo.resv) { + uint32_t mem_type = bo->tbo.mem.mem_type; + + if (!(bo->preferred_domains & amdgpu_mem_type_to_domain(mem_type))) + list_add_tail(&bo_va->base.vm_status, &vm->evicted); + else + list_add(&bo_va->base.vm_status, &vm->idle); + } list_splice_init(&bo_va->invalids, &bo_va->valids); bo_va->cleared = clear; @@ -1766,19 +1800,18 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev, int amdgpu_vm_handle_moved(struct amdgpu_device *adev, struct amdgpu_vm *vm) { + struct amdgpu_bo_va *bo_va, *tmp; + struct list_head moved; bool clear; - int r = 0; - - spin_lock(&vm->status_lock); - while (!list_empty(&vm->moved)) { - struct amdgpu_bo_va *bo_va; - struct reservation_object *resv; + int r; - bo_va = list_first_entry(&vm->moved, - struct amdgpu_bo_va, base.vm_status); - spin_unlock(&vm->status_lock); + INIT_LIST_HEAD(&moved); + spin_lock(&vm->moved_lock); + list_splice_init(&vm->moved, &moved); + spin_unlock(&vm->moved_lock); - resv = bo_va->base.bo->tbo.resv; + list_for_each_entry_safe(bo_va, tmp, &moved, base.vm_status) { + struct reservation_object *resv = bo_va->base.bo->tbo.resv; /* Per VM BOs never need to bo cleared in the page tables */ if (resv == vm->root.base.bo->tbo.resv) @@ -1791,17 +1824,19 @@ int amdgpu_vm_handle_moved(struct amdgpu_device *adev, clear = true; r = amdgpu_vm_bo_update(adev, bo_va, clear); - if (r) + if (r) { + spin_lock(&vm->moved_lock); + list_splice(&moved, &vm->moved); + spin_unlock(&vm->moved_lock); return r; + } if (!clear && resv != vm->root.base.bo->tbo.resv) reservation_object_unlock(resv); - spin_lock(&vm->status_lock); } - spin_unlock(&vm->status_lock); - return r; + return 0; } /** @@ -1827,36 +1862,12 @@ struct amdgpu_bo_va *amdgpu_vm_bo_add(struct amdgpu_device *adev, if (bo_va == NULL) { return NULL; } - bo_va->base.vm = vm; - bo_va->base.bo = bo; - INIT_LIST_HEAD(&bo_va->base.bo_list); - INIT_LIST_HEAD(&bo_va->base.vm_status); + amdgpu_vm_bo_base_init(&bo_va->base, vm, bo); bo_va->ref_count = 1; INIT_LIST_HEAD(&bo_va->valids); INIT_LIST_HEAD(&bo_va->invalids); - if (!bo) - return bo_va; - - list_add_tail(&bo_va->base.bo_list, &bo->va); - - if (bo->tbo.resv != vm->root.base.bo->tbo.resv) - return bo_va; - - if (bo->preferred_domains & - amdgpu_mem_type_to_domain(bo->tbo.mem.mem_type)) - return bo_va; - - /* - * We checked all the prerequisites, but it looks like this per VM BO - * is currently evicted. add the BO to the evicted list to make sure it - * is validated on next VM use to avoid fault. - * */ - spin_lock(&vm->status_lock); - list_move_tail(&bo_va->base.vm_status, &vm->evicted); - spin_unlock(&vm->status_lock); - return bo_va; } @@ -1884,11 +1895,11 @@ static void amdgpu_vm_bo_insert_map(struct amdgpu_device *adev, if (mapping->flags & AMDGPU_PTE_PRT) amdgpu_vm_prt_get(adev); - if (bo && bo->tbo.resv == vm->root.base.bo->tbo.resv) { - spin_lock(&vm->status_lock); - if (list_empty(&bo_va->base.vm_status)) - list_add(&bo_va->base.vm_status, &vm->moved); - spin_unlock(&vm->status_lock); + if (bo && bo->tbo.resv == vm->root.base.bo->tbo.resv && + !bo_va->base.moved) { + spin_lock(&vm->moved_lock); + list_move(&bo_va->base.vm_status, &vm->moved); + spin_unlock(&vm->moved_lock); } trace_amdgpu_vm_bo_map(bo_va, mapping); } @@ -2112,7 +2123,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev, before->last = saddr - 1; before->offset = tmp->offset; before->flags = tmp->flags; - list_add(&before->list, &tmp->list); + before->bo_va = tmp->bo_va; + list_add(&before->list, &tmp->bo_va->invalids); } /* Remember mapping split at the end */ @@ -2122,7 +2134,8 @@ int amdgpu_vm_bo_clear_mappings(struct amdgpu_device *adev, after->offset = tmp->offset; after->offset += after->start - tmp->start; after->flags = tmp->flags; - list_add(&after->list, &tmp->list); + after->bo_va = tmp->bo_va; + list_add(&after->list, &tmp->bo_va->invalids); } list_del(&tmp->list); @@ -2198,9 +2211,9 @@ void amdgpu_vm_bo_rmv(struct amdgpu_device *adev, list_del(&bo_va->base.bo_list); - spin_lock(&vm->status_lock); + spin_lock(&vm->moved_lock); list_del(&bo_va->base.vm_status); - spin_unlock(&vm->status_lock); + spin_unlock(&vm->moved_lock); list_for_each_entry_safe(mapping, next, &bo_va->valids, list) { list_del(&mapping->list); @@ -2234,33 +2247,34 @@ void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev, { struct amdgpu_vm_bo_base *bo_base; + /* shadow bo doesn't have bo base, its validation needs its parent */ + if (bo->parent && bo->parent->shadow == bo) + bo = bo->parent; + list_for_each_entry(bo_base, &bo->va, bo_list) { struct amdgpu_vm *vm = bo_base->vm; + bool was_moved = bo_base->moved; bo_base->moved = true; if (evicted && bo->tbo.resv == vm->root.base.bo->tbo.resv) { - spin_lock(&bo_base->vm->status_lock); if (bo->tbo.type == ttm_bo_type_kernel) list_move(&bo_base->vm_status, &vm->evicted); else list_move_tail(&bo_base->vm_status, &vm->evicted); - spin_unlock(&bo_base->vm->status_lock); continue; } - if (bo->tbo.type == ttm_bo_type_kernel) { - spin_lock(&bo_base->vm->status_lock); - if (list_empty(&bo_base->vm_status)) - list_add(&bo_base->vm_status, &vm->relocated); - spin_unlock(&bo_base->vm->status_lock); + if (was_moved) continue; - } - spin_lock(&bo_base->vm->status_lock); - if (list_empty(&bo_base->vm_status)) - list_add(&bo_base->vm_status, &vm->moved); - spin_unlock(&bo_base->vm->status_lock); + if (bo->tbo.type == ttm_bo_type_kernel) { + list_move(&bo_base->vm_status, &vm->relocated); + } else { + spin_lock(&bo_base->vm->moved_lock); + list_move(&bo_base->vm_status, &vm->moved); + spin_unlock(&bo_base->vm->moved_lock); + } } } @@ -2355,6 +2369,8 @@ void amdgpu_vm_adjust_size(struct amdgpu_device *adev, uint32_t vm_size, int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, int vm_context, unsigned int pasid) { + struct amdgpu_bo_param bp; + struct amdgpu_bo *root; const unsigned align = min(AMDGPU_VM_PTB_ALIGN_SIZE, AMDGPU_VM_PTE_COUNT(adev) * 8); unsigned ring_instance; @@ -2367,10 +2383,11 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, vm->va = RB_ROOT_CACHED; for (i = 0; i < AMDGPU_MAX_VMHUBS; i++) vm->reserved_vmid[i] = NULL; - spin_lock_init(&vm->status_lock); INIT_LIST_HEAD(&vm->evicted); INIT_LIST_HEAD(&vm->relocated); + spin_lock_init(&vm->moved_lock); INIT_LIST_HEAD(&vm->moved); + INIT_LIST_HEAD(&vm->idle); INIT_LIST_HEAD(&vm->freed); /* create scheduler entity for page table updates */ @@ -2380,7 +2397,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, ring = adev->vm_manager.vm_pte_rings[ring_instance]; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_KERNEL]; r = drm_sched_entity_init(&ring->sched, &vm->entity, - rq, amdgpu_sched_jobs, NULL); + rq, NULL); if (r) return r; @@ -2409,24 +2426,28 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, flags |= AMDGPU_GEM_CREATE_SHADOW; size = amdgpu_vm_bo_size(adev, adev->vm_manager.root_level); - r = amdgpu_bo_create(adev, size, align, AMDGPU_GEM_DOMAIN_VRAM, flags, - ttm_bo_type_kernel, NULL, &vm->root.base.bo); + memset(&bp, 0, sizeof(bp)); + bp.size = size; + bp.byte_align = align; + bp.domain = AMDGPU_GEM_DOMAIN_VRAM; + bp.flags = flags; + bp.type = ttm_bo_type_kernel; + bp.resv = NULL; + r = amdgpu_bo_create(adev, &bp, &root); if (r) goto error_free_sched_entity; - r = amdgpu_bo_reserve(vm->root.base.bo, true); + r = amdgpu_bo_reserve(root, true); if (r) goto error_free_root; - r = amdgpu_vm_clear_bo(adev, vm, vm->root.base.bo, + r = amdgpu_vm_clear_bo(adev, vm, root, adev->vm_manager.root_level, vm->pte_support_ats); if (r) goto error_unreserve; - vm->root.base.vm = vm; - list_add_tail(&vm->root.base.bo_list, &vm->root.base.bo->va); - list_add_tail(&vm->root.base.vm_status, &vm->evicted); + amdgpu_vm_bo_base_init(&vm->root.base, vm, root); amdgpu_bo_unreserve(vm->root.base.bo); if (pasid) { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h index 30f080364c97..061b99a18cb8 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h @@ -75,11 +75,12 @@ struct amdgpu_bo_list_entry; /* PDE Block Fragment Size for VEGA10 */ #define AMDGPU_PDE_BFS(a) ((uint64_t)a << 59) -/* VEGA10 only */ + +/* For GFX9 */ #define AMDGPU_PTE_MTYPE(a) ((uint64_t)a << 57) #define AMDGPU_PTE_MTYPE_MASK AMDGPU_PTE_MTYPE(3ULL) -/* For Raven */ +#define AMDGPU_MTYPE_NC 0 #define AMDGPU_MTYPE_CC 2 #define AMDGPU_PTE_DEFAULT_ATC (AMDGPU_PTE_SYSTEM \ @@ -167,9 +168,6 @@ struct amdgpu_vm { /* tree of virtual addresses mapped */ struct rb_root_cached va; - /* protecting invalidated */ - spinlock_t status_lock; - /* BOs who needs a validation */ struct list_head evicted; @@ -178,6 +176,10 @@ struct amdgpu_vm { /* BOs moved, but not yet updated in the PT */ struct list_head moved; + spinlock_t moved_lock; + + /* All BOs of this VM not currently in the state machine */ + struct list_head idle; /* BO mappings freed, but not yet updated in the PT */ struct list_head freed; @@ -186,9 +188,6 @@ struct amdgpu_vm { struct amdgpu_vm_pt root; struct dma_fence *last_update; - /* protecting freed */ - spinlock_t freed_lock; - /* Scheduler entity for page table updates */ struct drm_sched_entity entity; diff --git a/drivers/gpu/drm/amd/amdgpu/atom.c b/drivers/gpu/drm/amd/amdgpu/atom.c index 69500a8b4e2d..e9934de1b9cf 100644 --- a/drivers/gpu/drm/amd/amdgpu/atom.c +++ b/drivers/gpu/drm/amd/amdgpu/atom.c @@ -1221,7 +1221,7 @@ static int amdgpu_atom_execute_table_locked(struct atom_context *ctx, int index, ectx.abort = false; ectx.last_jump = 0; if (ws) - ectx.ws = kzalloc(4 * ws, GFP_KERNEL); + ectx.ws = kcalloc(4, ws, GFP_KERNEL); else ectx.ws = NULL; diff --git a/drivers/gpu/drm/amd/amdgpu/ci_dpm.c b/drivers/gpu/drm/amd/amdgpu/ci_dpm.c index 47ef3e6e7178..7fbad2f5f0bd 100644 --- a/drivers/gpu/drm/amd/amdgpu/ci_dpm.c +++ b/drivers/gpu/drm/amd/amdgpu/ci_dpm.c @@ -5679,8 +5679,9 @@ static int ci_parse_power_table(struct amdgpu_device *adev) (mode_info->atom_context->bios + data_offset + le16_to_cpu(power_info->pplib.usNonClockInfoArrayOffset)); - adev->pm.dpm.ps = kzalloc(sizeof(struct amdgpu_ps) * - state_array->ucNumEntries, GFP_KERNEL); + adev->pm.dpm.ps = kcalloc(state_array->ucNumEntries, + sizeof(struct amdgpu_ps), + GFP_KERNEL); if (!adev->pm.dpm.ps) return -ENOMEM; power_state_offset = (u8 *)state_array->states; @@ -5903,7 +5904,7 @@ static int ci_dpm_init(struct amdgpu_device *adev) pi->pcie_dpm_key_disabled = 0; pi->thermal_sclk_dpm_enabled = 0; - if (amdgpu_pp_feature_mask & SCLK_DEEP_SLEEP_MASK) + if (adev->powerplay.pp_feature & PP_SCLK_DEEP_SLEEP_MASK) pi->caps_sclk_ds = true; else pi->caps_sclk_ds = false; @@ -5927,7 +5928,9 @@ static int ci_dpm_init(struct amdgpu_device *adev) ci_set_private_data_variables_based_on_pptable(adev); adev->pm.dpm.dyn_state.vddc_dependency_on_dispclk.entries = - kzalloc(4 * sizeof(struct amdgpu_clock_voltage_dependency_entry), GFP_KERNEL); + kcalloc(4, + sizeof(struct amdgpu_clock_voltage_dependency_entry), + GFP_KERNEL); if (!adev->pm.dpm.dyn_state.vddc_dependency_on_dispclk.entries) { ci_dpm_fini(adev); return -ENOMEM; @@ -6255,7 +6258,7 @@ static int ci_dpm_late_init(void *handle) int ret; struct amdgpu_device *adev = (struct amdgpu_device *)handle; - if (!amdgpu_dpm) + if (!adev->pm.dpm_enabled) return 0; /* init the sysfs and debugfs files late */ diff --git a/drivers/gpu/drm/amd/amdgpu/cik.c b/drivers/gpu/drm/amd/amdgpu/cik.c index 0df22030e713..8ff4c60d1b59 100644 --- a/drivers/gpu/drm/amd/amdgpu/cik.c +++ b/drivers/gpu/drm/amd/amdgpu/cik.c @@ -1735,6 +1735,12 @@ static void cik_invalidate_hdp(struct amdgpu_device *adev, } } +static bool cik_need_full_reset(struct amdgpu_device *adev) +{ + /* change this when we support soft reset */ + return true; +} + static const struct amdgpu_asic_funcs cik_asic_funcs = { .read_disabled_bios = &cik_read_disabled_bios, @@ -1748,6 +1754,7 @@ static const struct amdgpu_asic_funcs cik_asic_funcs = .get_config_memsize = &cik_get_config_memsize, .flush_hdp = &cik_flush_hdp, .invalidate_hdp = &cik_invalidate_hdp, + .need_full_reset = &cik_need_full_reset, }; static int cik_common_early_init(void *handle) diff --git a/drivers/gpu/drm/amd/amdgpu/dce_v10_0.c b/drivers/gpu/drm/amd/amdgpu/dce_v10_0.c index 452f88ea46a2..ada241bfeee9 100644 --- a/drivers/gpu/drm/amd/amdgpu/dce_v10_0.c +++ b/drivers/gpu/drm/amd/amdgpu/dce_v10_0.c @@ -1823,7 +1823,6 @@ static int dce_v10_0_crtc_do_set_base(struct drm_crtc *crtc, struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc); struct drm_device *dev = crtc->dev; struct amdgpu_device *adev = dev->dev_private; - struct amdgpu_framebuffer *amdgpu_fb; struct drm_framebuffer *target_fb; struct drm_gem_object *obj; struct amdgpu_bo *abo; @@ -1842,18 +1841,15 @@ static int dce_v10_0_crtc_do_set_base(struct drm_crtc *crtc, return 0; } - if (atomic) { - amdgpu_fb = to_amdgpu_framebuffer(fb); + if (atomic) target_fb = fb; - } else { - amdgpu_fb = to_amdgpu_framebuffer(crtc->primary->fb); + else target_fb = crtc->primary->fb; - } /* If atomic, assume fb object is pinned & idle & fenced and * just update base pointers */ - obj = amdgpu_fb->obj; + obj = target_fb->obj[0]; abo = gem_to_amdgpu_bo(obj); r = amdgpu_bo_reserve(abo, false); if (unlikely(r != 0)) @@ -2043,8 +2039,7 @@ static int dce_v10_0_crtc_do_set_base(struct drm_crtc *crtc, WREG32(mmMASTER_UPDATE_MODE + amdgpu_crtc->crtc_offset, 0); if (!atomic && fb && fb != crtc->primary->fb) { - amdgpu_fb = to_amdgpu_framebuffer(fb); - abo = gem_to_amdgpu_bo(amdgpu_fb->obj); + abo = gem_to_amdgpu_bo(fb->obj[0]); r = amdgpu_bo_reserve(abo, true); if (unlikely(r != 0)) return r; @@ -2526,11 +2521,9 @@ static void dce_v10_0_crtc_disable(struct drm_crtc *crtc) dce_v10_0_crtc_dpms(crtc, DRM_MODE_DPMS_OFF); if (crtc->primary->fb) { int r; - struct amdgpu_framebuffer *amdgpu_fb; struct amdgpu_bo *abo; - amdgpu_fb = to_amdgpu_framebuffer(crtc->primary->fb); - abo = gem_to_amdgpu_bo(amdgpu_fb->obj); + abo = gem_to_amdgpu_bo(crtc->primary->fb->obj[0]); r = amdgpu_bo_reserve(abo, true); if (unlikely(r)) DRM_ERROR("failed to reserve abo before unpin\n"); diff --git a/drivers/gpu/drm/amd/amdgpu/dce_v11_0.c b/drivers/gpu/drm/amd/amdgpu/dce_v11_0.c index a7c1c584a191..a5b96eac3033 100644 --- a/drivers/gpu/drm/amd/amdgpu/dce_v11_0.c +++ b/drivers/gpu/drm/amd/amdgpu/dce_v11_0.c @@ -173,6 +173,7 @@ static void dce_v11_0_init_golden_registers(struct amdgpu_device *adev) ARRAY_SIZE(polaris11_golden_settings_a11)); break; case CHIP_POLARIS10: + case CHIP_VEGAM: amdgpu_device_program_register_sequence(adev, polaris10_golden_settings_a11, ARRAY_SIZE(polaris10_golden_settings_a11)); @@ -473,6 +474,7 @@ static int dce_v11_0_get_num_crtc (struct amdgpu_device *adev) num_crtc = 2; break; case CHIP_POLARIS10: + case CHIP_VEGAM: num_crtc = 6; break; case CHIP_POLARIS11: @@ -1445,6 +1447,7 @@ static int dce_v11_0_audio_init(struct amdgpu_device *adev) adev->mode_info.audio.num_pins = 7; break; case CHIP_POLARIS10: + case CHIP_VEGAM: adev->mode_info.audio.num_pins = 8; break; case CHIP_POLARIS11: @@ -1862,7 +1865,6 @@ static int dce_v11_0_crtc_do_set_base(struct drm_crtc *crtc, struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc); struct drm_device *dev = crtc->dev; struct amdgpu_device *adev = dev->dev_private; - struct amdgpu_framebuffer *amdgpu_fb; struct drm_framebuffer *target_fb; struct drm_gem_object *obj; struct amdgpu_bo *abo; @@ -1881,18 +1883,15 @@ static int dce_v11_0_crtc_do_set_base(struct drm_crtc *crtc, return 0; } - if (atomic) { - amdgpu_fb = to_amdgpu_framebuffer(fb); + if (atomic) target_fb = fb; - } else { - amdgpu_fb = to_amdgpu_framebuffer(crtc->primary->fb); + else target_fb = crtc->primary->fb; - } /* If atomic, assume fb object is pinned & idle & fenced and * just update base pointers */ - obj = amdgpu_fb->obj; + obj = target_fb->obj[0]; abo = gem_to_amdgpu_bo(obj); r = amdgpu_bo_reserve(abo, false); if (unlikely(r != 0)) @@ -2082,8 +2081,7 @@ static int dce_v11_0_crtc_do_set_base(struct drm_crtc *crtc, WREG32(mmCRTC_MASTER_UPDATE_MODE + amdgpu_crtc->crtc_offset, 0); if (!atomic && fb && fb != crtc->primary->fb) { - amdgpu_fb = to_amdgpu_framebuffer(fb); - abo = gem_to_amdgpu_bo(amdgpu_fb->obj); + abo = gem_to_amdgpu_bo(fb->obj[0]); r = amdgpu_bo_reserve(abo, true); if (unlikely(r != 0)) return r; @@ -2253,7 +2251,8 @@ static u32 dce_v11_0_pick_pll(struct drm_crtc *crtc) if ((adev->asic_type == CHIP_POLARIS10) || (adev->asic_type == CHIP_POLARIS11) || - (adev->asic_type == CHIP_POLARIS12)) { + (adev->asic_type == CHIP_POLARIS12) || + (adev->asic_type == CHIP_VEGAM)) { struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(amdgpu_crtc->encoder); struct amdgpu_encoder_atom_dig *dig = amdgpu_encoder->enc_priv; @@ -2601,11 +2600,9 @@ static void dce_v11_0_crtc_disable(struct drm_crtc *crtc) dce_v11_0_crtc_dpms(crtc, DRM_MODE_DPMS_OFF); if (crtc->primary->fb) { int r; - struct amdgpu_framebuffer *amdgpu_fb; struct amdgpu_bo *abo; - amdgpu_fb = to_amdgpu_framebuffer(crtc->primary->fb); - abo = gem_to_amdgpu_bo(amdgpu_fb->obj); + abo = gem_to_amdgpu_bo(crtc->primary->fb->obj[0]); r = amdgpu_bo_reserve(abo, true); if (unlikely(r)) DRM_ERROR("failed to reserve abo before unpin\n"); @@ -2673,7 +2670,8 @@ static int dce_v11_0_crtc_mode_set(struct drm_crtc *crtc, if ((adev->asic_type == CHIP_POLARIS10) || (adev->asic_type == CHIP_POLARIS11) || - (adev->asic_type == CHIP_POLARIS12)) { + (adev->asic_type == CHIP_POLARIS12) || + (adev->asic_type == CHIP_VEGAM)) { struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(amdgpu_crtc->encoder); int encoder_mode = @@ -2830,6 +2828,7 @@ static int dce_v11_0_early_init(void *handle) adev->mode_info.num_dig = 9; break; case CHIP_POLARIS10: + case CHIP_VEGAM: adev->mode_info.num_hpd = 6; adev->mode_info.num_dig = 6; break; @@ -2949,7 +2948,8 @@ static int dce_v11_0_hw_init(void *handle) amdgpu_atombios_encoder_init_dig(adev); if ((adev->asic_type == CHIP_POLARIS10) || (adev->asic_type == CHIP_POLARIS11) || - (adev->asic_type == CHIP_POLARIS12)) { + (adev->asic_type == CHIP_POLARIS12) || + (adev->asic_type == CHIP_VEGAM)) { amdgpu_atombios_crtc_set_dce_clock(adev, adev->clock.default_dispclk, DCE_CLOCK_TYPE_DISPCLK, ATOM_GCK_DFS); amdgpu_atombios_crtc_set_dce_clock(adev, 0, diff --git a/drivers/gpu/drm/amd/amdgpu/dce_v6_0.c b/drivers/gpu/drm/amd/amdgpu/dce_v6_0.c index 9f67b7fd3487..394cc1e8fe20 100644 --- a/drivers/gpu/drm/amd/amdgpu/dce_v6_0.c +++ b/drivers/gpu/drm/amd/amdgpu/dce_v6_0.c @@ -1780,7 +1780,6 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc); struct drm_device *dev = crtc->dev; struct amdgpu_device *adev = dev->dev_private; - struct amdgpu_framebuffer *amdgpu_fb; struct drm_framebuffer *target_fb; struct drm_gem_object *obj; struct amdgpu_bo *abo; @@ -1798,18 +1797,15 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, return 0; } - if (atomic) { - amdgpu_fb = to_amdgpu_framebuffer(fb); + if (atomic) target_fb = fb; - } else { - amdgpu_fb = to_amdgpu_framebuffer(crtc->primary->fb); + else target_fb = crtc->primary->fb; - } /* If atomic, assume fb object is pinned & idle & fenced and * just update base pointers */ - obj = amdgpu_fb->obj; + obj = target_fb->obj[0]; abo = gem_to_amdgpu_bo(obj); r = amdgpu_bo_reserve(abo, false); if (unlikely(r != 0)) @@ -1978,8 +1974,7 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, WREG32(mmMASTER_UPDATE_MODE + amdgpu_crtc->crtc_offset, 0); if (!atomic && fb && fb != crtc->primary->fb) { - amdgpu_fb = to_amdgpu_framebuffer(fb); - abo = gem_to_amdgpu_bo(amdgpu_fb->obj); + abo = gem_to_amdgpu_bo(fb->obj[0]); r = amdgpu_bo_reserve(abo, true); if (unlikely(r != 0)) return r; @@ -2414,11 +2409,9 @@ static void dce_v6_0_crtc_disable(struct drm_crtc *crtc) dce_v6_0_crtc_dpms(crtc, DRM_MODE_DPMS_OFF); if (crtc->primary->fb) { int r; - struct amdgpu_framebuffer *amdgpu_fb; struct amdgpu_bo *abo; - amdgpu_fb = to_amdgpu_framebuffer(crtc->primary->fb); - abo = gem_to_amdgpu_bo(amdgpu_fb->obj); + abo = gem_to_amdgpu_bo(crtc->primary->fb->obj[0]); r = amdgpu_bo_reserve(abo, true); if (unlikely(r)) DRM_ERROR("failed to reserve abo before unpin\n"); diff --git a/drivers/gpu/drm/amd/amdgpu/dce_v8_0.c b/drivers/gpu/drm/amd/amdgpu/dce_v8_0.c index f55422cbd77a..c9b9ab8f1b05 100644 --- a/drivers/gpu/drm/amd/amdgpu/dce_v8_0.c +++ b/drivers/gpu/drm/amd/amdgpu/dce_v8_0.c @@ -1754,7 +1754,6 @@ static int dce_v8_0_crtc_do_set_base(struct drm_crtc *crtc, struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc); struct drm_device *dev = crtc->dev; struct amdgpu_device *adev = dev->dev_private; - struct amdgpu_framebuffer *amdgpu_fb; struct drm_framebuffer *target_fb; struct drm_gem_object *obj; struct amdgpu_bo *abo; @@ -1773,18 +1772,15 @@ static int dce_v8_0_crtc_do_set_base(struct drm_crtc *crtc, return 0; } - if (atomic) { - amdgpu_fb = to_amdgpu_framebuffer(fb); + if (atomic) target_fb = fb; - } else { - amdgpu_fb = to_amdgpu_framebuffer(crtc->primary->fb); + else target_fb = crtc->primary->fb; - } /* If atomic, assume fb object is pinned & idle & fenced and * just update base pointers */ - obj = amdgpu_fb->obj; + obj = target_fb->obj[0]; abo = gem_to_amdgpu_bo(obj); r = amdgpu_bo_reserve(abo, false); if (unlikely(r != 0)) @@ -1955,8 +1951,7 @@ static int dce_v8_0_crtc_do_set_base(struct drm_crtc *crtc, WREG32(mmMASTER_UPDATE_MODE + amdgpu_crtc->crtc_offset, 0); if (!atomic && fb && fb != crtc->primary->fb) { - amdgpu_fb = to_amdgpu_framebuffer(fb); - abo = gem_to_amdgpu_bo(amdgpu_fb->obj); + abo = gem_to_amdgpu_bo(fb->obj[0]); r = amdgpu_bo_reserve(abo, true); if (unlikely(r != 0)) return r; @@ -2430,11 +2425,9 @@ static void dce_v8_0_crtc_disable(struct drm_crtc *crtc) dce_v8_0_crtc_dpms(crtc, DRM_MODE_DPMS_OFF); if (crtc->primary->fb) { int r; - struct amdgpu_framebuffer *amdgpu_fb; struct amdgpu_bo *abo; - amdgpu_fb = to_amdgpu_framebuffer(crtc->primary->fb); - abo = gem_to_amdgpu_bo(amdgpu_fb->obj); + abo = gem_to_amdgpu_bo(crtc->primary->fb->obj[0]); r = amdgpu_bo_reserve(abo, true); if (unlikely(r)) DRM_ERROR("failed to reserve abo before unpin\n"); diff --git a/drivers/gpu/drm/amd/amdgpu/dce_virtual.c b/drivers/gpu/drm/amd/amdgpu/dce_virtual.c index b51f05dc9582..dbf2ccd0c744 100644 --- a/drivers/gpu/drm/amd/amdgpu/dce_virtual.c +++ b/drivers/gpu/drm/amd/amdgpu/dce_virtual.c @@ -168,11 +168,9 @@ static void dce_virtual_crtc_disable(struct drm_crtc *crtc) dce_virtual_crtc_dpms(crtc, DRM_MODE_DPMS_OFF); if (crtc->primary->fb) { int r; - struct amdgpu_framebuffer *amdgpu_fb; struct amdgpu_bo *abo; - amdgpu_fb = to_amdgpu_framebuffer(crtc->primary->fb); - abo = gem_to_amdgpu_bo(amdgpu_fb->obj); + abo = gem_to_amdgpu_bo(crtc->primary->fb->obj[0]); r = amdgpu_bo_reserve(abo, true); if (unlikely(r)) DRM_ERROR("failed to reserve abo before unpin\n"); @@ -329,7 +327,7 @@ static int dce_virtual_get_modes(struct drm_connector *connector) return 0; } -static int dce_virtual_mode_valid(struct drm_connector *connector, +static enum drm_mode_status dce_virtual_mode_valid(struct drm_connector *connector, struct drm_display_mode *mode) { return MODE_OK; @@ -462,8 +460,9 @@ static int dce_virtual_hw_init(void *handle) break; case CHIP_CARRIZO: case CHIP_STONEY: - case CHIP_POLARIS11: case CHIP_POLARIS10: + case CHIP_POLARIS11: + case CHIP_VEGAM: dce_v11_0_disable_dce(adev); break; case CHIP_TOPAZ: @@ -474,6 +473,7 @@ static int dce_virtual_hw_init(void *handle) break; case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: break; default: DRM_ERROR("Virtual display unsupported ASIC type: 0x%X\n", adev->asic_type); diff --git a/drivers/gpu/drm/amd/amdgpu/df_v1_7.c b/drivers/gpu/drm/amd/amdgpu/df_v1_7.c new file mode 100644 index 000000000000..9935371db7ce --- /dev/null +++ b/drivers/gpu/drm/amd/amdgpu/df_v1_7.c @@ -0,0 +1,120 @@ +/* + * Copyright 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ +#include "amdgpu.h" +#include "df_v1_7.h" + +#include "df/df_1_7_default.h" +#include "df/df_1_7_offset.h" +#include "df/df_1_7_sh_mask.h" + +static u32 df_v1_7_channel_number[] = {1, 2, 0, 4, 0, 8, 0, 16, 2}; + +static void df_v1_7_init (struct amdgpu_device *adev) +{ +} + +static void df_v1_7_enable_broadcast_mode(struct amdgpu_device *adev, + bool enable) +{ + u32 tmp; + + if (enable) { + tmp = RREG32_SOC15(DF, 0, mmFabricConfigAccessControl); + tmp &= ~FabricConfigAccessControl__CfgRegInstAccEn_MASK; + WREG32_SOC15(DF, 0, mmFabricConfigAccessControl, tmp); + } else + WREG32_SOC15(DF, 0, mmFabricConfigAccessControl, + mmFabricConfigAccessControl_DEFAULT); +} + +static u32 df_v1_7_get_fb_channel_number(struct amdgpu_device *adev) +{ + u32 tmp; + + tmp = RREG32_SOC15(DF, 0, mmDF_CS_AON0_DramBaseAddress0); + tmp &= DF_CS_AON0_DramBaseAddress0__IntLvNumChan_MASK; + tmp >>= DF_CS_AON0_DramBaseAddress0__IntLvNumChan__SHIFT; + + return tmp; +} + +static u32 df_v1_7_get_hbm_channel_number(struct amdgpu_device *adev) +{ + int fb_channel_number; + + fb_channel_number = adev->df_funcs->get_fb_channel_number(adev); + + return df_v1_7_channel_number[fb_channel_number]; +} + +static void df_v1_7_update_medium_grain_clock_gating(struct amdgpu_device *adev, + bool enable) +{ + u32 tmp; + + /* Put DF on broadcast mode */ + adev->df_funcs->enable_broadcast_mode(adev, true); + + if (enable && (adev->cg_flags & AMD_CG_SUPPORT_DF_MGCG)) { + tmp = RREG32_SOC15(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater); + tmp &= ~DF_PIE_AON0_DfGlobalClkGater__MGCGMode_MASK; + tmp |= DF_V1_7_MGCG_ENABLE_15_CYCLE_DELAY; + WREG32_SOC15(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater, tmp); + } else { + tmp = RREG32_SOC15(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater); + tmp &= ~DF_PIE_AON0_DfGlobalClkGater__MGCGMode_MASK; + tmp |= DF_V1_7_MGCG_DISABLE; + WREG32_SOC15(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater, tmp); + } + + /* Exit boradcast mode */ + adev->df_funcs->enable_broadcast_mode(adev, false); +} + +static void df_v1_7_get_clockgating_state(struct amdgpu_device *adev, + u32 *flags) +{ + u32 tmp; + + /* AMD_CG_SUPPORT_DF_MGCG */ + tmp = RREG32_SOC15(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater); + if (tmp & DF_V1_7_MGCG_ENABLE_15_CYCLE_DELAY) + *flags |= AMD_CG_SUPPORT_DF_MGCG; +} + +static void df_v1_7_enable_ecc_force_par_wr_rmw(struct amdgpu_device *adev, + bool enable) +{ + WREG32_FIELD15(DF, 0, DF_CS_AON0_CoherentSlaveModeCtrlA0, + ForceParWrRMW, enable); +} + +const struct amdgpu_df_funcs df_v1_7_funcs = { + .init = df_v1_7_init, + .enable_broadcast_mode = df_v1_7_enable_broadcast_mode, + .get_fb_channel_number = df_v1_7_get_fb_channel_number, + .get_hbm_channel_number = df_v1_7_get_hbm_channel_number, + .update_medium_grain_clock_gating = df_v1_7_update_medium_grain_clock_gating, + .get_clockgating_state = df_v1_7_get_clockgating_state, + .enable_ecc_force_par_wr_rmw = df_v1_7_enable_ecc_force_par_wr_rmw, +}; diff --git a/drivers/gpu/drm/amd/amdgpu/df_v1_7.h b/drivers/gpu/drm/amd/amdgpu/df_v1_7.h new file mode 100644 index 000000000000..74621104c487 --- /dev/null +++ b/drivers/gpu/drm/amd/amdgpu/df_v1_7.h @@ -0,0 +1,40 @@ +/* + * Copyright 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ + +#ifndef __DF_V1_7_H__ +#define __DF_V1_7_H__ + +#include "soc15_common.h" +enum DF_V1_7_MGCG +{ + DF_V1_7_MGCG_DISABLE = 0, + DF_V1_7_MGCG_ENABLE_00_CYCLE_DELAY =1, + DF_V1_7_MGCG_ENABLE_01_CYCLE_DELAY =2, + DF_V1_7_MGCG_ENABLE_15_CYCLE_DELAY =13, + DF_V1_7_MGCG_ENABLE_31_CYCLE_DELAY =14, + DF_V1_7_MGCG_ENABLE_63_CYCLE_DELAY =15 +}; + +extern const struct amdgpu_df_funcs df_v1_7_funcs; + +#endif diff --git a/drivers/gpu/drm/amd/amdgpu/df_v3_6.c b/drivers/gpu/drm/amd/amdgpu/df_v3_6.c new file mode 100644 index 000000000000..d5ebe566809b --- /dev/null +++ b/drivers/gpu/drm/amd/amdgpu/df_v3_6.c @@ -0,0 +1,116 @@ +/* + * Copyright 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ +#include "amdgpu.h" +#include "df_v3_6.h" + +#include "df/df_3_6_default.h" +#include "df/df_3_6_offset.h" +#include "df/df_3_6_sh_mask.h" + +static u32 df_v3_6_channel_number[] = {1, 2, 0, 4, 0, 8, 0, + 16, 32, 0, 0, 0, 2, 4, 8}; + +static void df_v3_6_init(struct amdgpu_device *adev) +{ +} + +static void df_v3_6_enable_broadcast_mode(struct amdgpu_device *adev, + bool enable) +{ + u32 tmp; + + if (enable) { + tmp = RREG32_SOC15(DF, 0, mmFabricConfigAccessControl); + tmp &= ~FabricConfigAccessControl__CfgRegInstAccEn_MASK; + WREG32_SOC15(DF, 0, mmFabricConfigAccessControl, tmp); + } else + WREG32_SOC15(DF, 0, mmFabricConfigAccessControl, + mmFabricConfigAccessControl_DEFAULT); +} + +static u32 df_v3_6_get_fb_channel_number(struct amdgpu_device *adev) +{ + u32 tmp; + + tmp = RREG32_SOC15(DF, 0, mmDF_CS_UMC_AON0_DramBaseAddress0); + tmp &= DF_CS_UMC_AON0_DramBaseAddress0__IntLvNumChan_MASK; + tmp >>= DF_CS_UMC_AON0_DramBaseAddress0__IntLvNumChan__SHIFT; + + return tmp; +} + +static u32 df_v3_6_get_hbm_channel_number(struct amdgpu_device *adev) +{ + int fb_channel_number; + + fb_channel_number = adev->df_funcs->get_fb_channel_number(adev); + if (fb_channel_number >= ARRAY_SIZE(df_v3_6_channel_number)) + fb_channel_number = 0; + + return df_v3_6_channel_number[fb_channel_number]; +} + +static void df_v3_6_update_medium_grain_clock_gating(struct amdgpu_device *adev, + bool enable) +{ + u32 tmp; + + /* Put DF on broadcast mode */ + adev->df_funcs->enable_broadcast_mode(adev, true); + + if (enable && (adev->cg_flags & AMD_CG_SUPPORT_DF_MGCG)) { + tmp = RREG32_SOC15(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater); + tmp &= ~DF_PIE_AON0_DfGlobalClkGater__MGCGMode_MASK; + tmp |= DF_V3_6_MGCG_ENABLE_15_CYCLE_DELAY; + WREG32_SOC15(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater, tmp); + } else { + tmp = RREG32_SOC15(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater); + tmp &= ~DF_PIE_AON0_DfGlobalClkGater__MGCGMode_MASK; + tmp |= DF_V3_6_MGCG_DISABLE; + WREG32_SOC15(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater, tmp); + } + + /* Exit broadcast mode */ + adev->df_funcs->enable_broadcast_mode(adev, false); +} + +static void df_v3_6_get_clockgating_state(struct amdgpu_device *adev, + u32 *flags) +{ + u32 tmp; + + /* AMD_CG_SUPPORT_DF_MGCG */ + tmp = RREG32_SOC15(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater); + if (tmp & DF_V3_6_MGCG_ENABLE_15_CYCLE_DELAY) + *flags |= AMD_CG_SUPPORT_DF_MGCG; +} + +const struct amdgpu_df_funcs df_v3_6_funcs = { + .init = df_v3_6_init, + .enable_broadcast_mode = df_v3_6_enable_broadcast_mode, + .get_fb_channel_number = df_v3_6_get_fb_channel_number, + .get_hbm_channel_number = df_v3_6_get_hbm_channel_number, + .update_medium_grain_clock_gating = + df_v3_6_update_medium_grain_clock_gating, + .get_clockgating_state = df_v3_6_get_clockgating_state, +}; diff --git a/drivers/gpu/drm/amd/amdgpu/df_v3_6.h b/drivers/gpu/drm/amd/amdgpu/df_v3_6.h new file mode 100644 index 000000000000..e79c58e5efcb --- /dev/null +++ b/drivers/gpu/drm/amd/amdgpu/df_v3_6.h @@ -0,0 +1,40 @@ +/* + * Copyright 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ + +#ifndef __DF_V3_6_H__ +#define __DF_V3_6_H__ + +#include "soc15_common.h" + +enum DF_V3_6_MGCG { + DF_V3_6_MGCG_DISABLE = 0, + DF_V3_6_MGCG_ENABLE_00_CYCLE_DELAY = 1, + DF_V3_6_MGCG_ENABLE_01_CYCLE_DELAY = 2, + DF_V3_6_MGCG_ENABLE_15_CYCLE_DELAY = 13, + DF_V3_6_MGCG_ENABLE_31_CYCLE_DELAY = 14, + DF_V3_6_MGCG_ENABLE_63_CYCLE_DELAY = 15 +}; + +extern const struct amdgpu_df_funcs df_v3_6_funcs; + +#endif diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c index e14263fca1c9..818874b13c99 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c @@ -125,18 +125,6 @@ MODULE_FIRMWARE("amdgpu/fiji_mec.bin"); MODULE_FIRMWARE("amdgpu/fiji_mec2.bin"); MODULE_FIRMWARE("amdgpu/fiji_rlc.bin"); -MODULE_FIRMWARE("amdgpu/polaris11_ce.bin"); -MODULE_FIRMWARE("amdgpu/polaris11_ce_2.bin"); -MODULE_FIRMWARE("amdgpu/polaris11_pfp.bin"); -MODULE_FIRMWARE("amdgpu/polaris11_pfp_2.bin"); -MODULE_FIRMWARE("amdgpu/polaris11_me.bin"); -MODULE_FIRMWARE("amdgpu/polaris11_me_2.bin"); -MODULE_FIRMWARE("amdgpu/polaris11_mec.bin"); -MODULE_FIRMWARE("amdgpu/polaris11_mec_2.bin"); -MODULE_FIRMWARE("amdgpu/polaris11_mec2.bin"); -MODULE_FIRMWARE("amdgpu/polaris11_mec2_2.bin"); -MODULE_FIRMWARE("amdgpu/polaris11_rlc.bin"); - MODULE_FIRMWARE("amdgpu/polaris10_ce.bin"); MODULE_FIRMWARE("amdgpu/polaris10_ce_2.bin"); MODULE_FIRMWARE("amdgpu/polaris10_pfp.bin"); @@ -149,6 +137,18 @@ MODULE_FIRMWARE("amdgpu/polaris10_mec2.bin"); MODULE_FIRMWARE("amdgpu/polaris10_mec2_2.bin"); MODULE_FIRMWARE("amdgpu/polaris10_rlc.bin"); +MODULE_FIRMWARE("amdgpu/polaris11_ce.bin"); +MODULE_FIRMWARE("amdgpu/polaris11_ce_2.bin"); +MODULE_FIRMWARE("amdgpu/polaris11_pfp.bin"); +MODULE_FIRMWARE("amdgpu/polaris11_pfp_2.bin"); +MODULE_FIRMWARE("amdgpu/polaris11_me.bin"); +MODULE_FIRMWARE("amdgpu/polaris11_me_2.bin"); +MODULE_FIRMWARE("amdgpu/polaris11_mec.bin"); +MODULE_FIRMWARE("amdgpu/polaris11_mec_2.bin"); +MODULE_FIRMWARE("amdgpu/polaris11_mec2.bin"); +MODULE_FIRMWARE("amdgpu/polaris11_mec2_2.bin"); +MODULE_FIRMWARE("amdgpu/polaris11_rlc.bin"); + MODULE_FIRMWARE("amdgpu/polaris12_ce.bin"); MODULE_FIRMWARE("amdgpu/polaris12_ce_2.bin"); MODULE_FIRMWARE("amdgpu/polaris12_pfp.bin"); @@ -161,6 +161,13 @@ MODULE_FIRMWARE("amdgpu/polaris12_mec2.bin"); MODULE_FIRMWARE("amdgpu/polaris12_mec2_2.bin"); MODULE_FIRMWARE("amdgpu/polaris12_rlc.bin"); +MODULE_FIRMWARE("amdgpu/vegam_ce.bin"); +MODULE_FIRMWARE("amdgpu/vegam_pfp.bin"); +MODULE_FIRMWARE("amdgpu/vegam_me.bin"); +MODULE_FIRMWARE("amdgpu/vegam_mec.bin"); +MODULE_FIRMWARE("amdgpu/vegam_mec2.bin"); +MODULE_FIRMWARE("amdgpu/vegam_rlc.bin"); + static const struct amdgpu_gds_reg_offset amdgpu_gds_reg_offset[] = { {mmGDS_VMID0_BASE, mmGDS_VMID0_SIZE, mmGDS_GWS_VMID0, mmGDS_OA_VMID0}, @@ -292,6 +299,37 @@ static const u32 tonga_mgcg_cgcg_init[] = mmCP_MEM_SLP_CNTL, 0x00000001, 0x00000001, }; +static const u32 golden_settings_vegam_a11[] = +{ + mmCB_HW_CONTROL, 0x0001f3cf, 0x00007208, + mmCB_HW_CONTROL_2, 0x0f000000, 0x0d000000, + mmCB_HW_CONTROL_3, 0x000001ff, 0x00000040, + mmDB_DEBUG2, 0xf00fffff, 0x00000400, + mmPA_SC_ENHANCE, 0xffffffff, 0x20000001, + mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000, + mmPA_SC_RASTER_CONFIG, 0x3f3fffff, 0x3a00161a, + mmPA_SC_RASTER_CONFIG_1, 0x0000003f, 0x0000002e, + mmRLC_CGCG_CGLS_CTRL, 0x00000003, 0x0001003c, + mmRLC_CGCG_CGLS_CTRL_3D, 0xffffffff, 0x0001003c, + mmSQ_CONFIG, 0x07f80000, 0x01180000, + mmTA_CNTL_AUX, 0x000f000f, 0x000b0000, + mmTCC_CTRL, 0x00100000, 0xf31fff7f, + mmTCP_ADDR_CONFIG, 0x000003ff, 0x000000f7, + mmTCP_CHAN_STEER_HI, 0xffffffff, 0x00000000, + mmTCP_CHAN_STEER_LO, 0xffffffff, 0x32761054, + mmVGT_RESET_DEBUG, 0x00000004, 0x00000004, +}; + +static const u32 vegam_golden_common_all[] = +{ + mmGRBM_GFX_INDEX, 0xffffffff, 0xe0000000, + mmGB_ADDR_CONFIG, 0xffffffff, 0x22011003, + mmSPI_RESOURCE_RESERVE_CU_0, 0xffffffff, 0x00000800, + mmSPI_RESOURCE_RESERVE_CU_1, 0xffffffff, 0x00000800, + mmSPI_RESOURCE_RESERVE_EN_CU_0, 0xffffffff, 0x00FF7FBF, + mmSPI_RESOURCE_RESERVE_EN_CU_1, 0xffffffff, 0x00FF7FAF, +}; + static const u32 golden_settings_polaris11_a11[] = { mmCB_HW_CONTROL, 0x0000f3cf, 0x00007208, @@ -712,6 +750,14 @@ static void gfx_v8_0_init_golden_registers(struct amdgpu_device *adev) tonga_golden_common_all, ARRAY_SIZE(tonga_golden_common_all)); break; + case CHIP_VEGAM: + amdgpu_device_program_register_sequence(adev, + golden_settings_vegam_a11, + ARRAY_SIZE(golden_settings_vegam_a11)); + amdgpu_device_program_register_sequence(adev, + vegam_golden_common_all, + ARRAY_SIZE(vegam_golden_common_all)); + break; case CHIP_POLARIS11: case CHIP_POLARIS12: amdgpu_device_program_register_sequence(adev, @@ -918,17 +964,20 @@ static int gfx_v8_0_init_microcode(struct amdgpu_device *adev) case CHIP_FIJI: chip_name = "fiji"; break; - case CHIP_POLARIS11: - chip_name = "polaris11"; + case CHIP_STONEY: + chip_name = "stoney"; break; case CHIP_POLARIS10: chip_name = "polaris10"; break; + case CHIP_POLARIS11: + chip_name = "polaris11"; + break; case CHIP_POLARIS12: chip_name = "polaris12"; break; - case CHIP_STONEY: - chip_name = "stoney"; + case CHIP_VEGAM: + chip_name = "vegam"; break; default: BUG(); @@ -1770,6 +1819,7 @@ static int gfx_v8_0_gpu_early_init(struct amdgpu_device *adev) gb_addr_config = POLARIS11_GB_ADDR_CONFIG_GOLDEN; break; case CHIP_POLARIS10: + case CHIP_VEGAM: ret = amdgpu_atombios_get_gfx_info(adev); if (ret) return ret; @@ -1957,12 +2007,13 @@ static int gfx_v8_0_sw_init(void *handle) struct amdgpu_device *adev = (struct amdgpu_device *)handle; switch (adev->asic_type) { - case CHIP_FIJI: case CHIP_TONGA: + case CHIP_CARRIZO: + case CHIP_FIJI: + case CHIP_POLARIS10: case CHIP_POLARIS11: case CHIP_POLARIS12: - case CHIP_POLARIS10: - case CHIP_CARRIZO: + case CHIP_VEGAM: adev->gfx.mec.num_mec = 2; break; case CHIP_TOPAZ: @@ -2323,6 +2374,7 @@ static void gfx_v8_0_tiling_mode_table_init(struct amdgpu_device *adev) break; case CHIP_FIJI: + case CHIP_VEGAM: modearray[0] = (ARRAY_MODE(ARRAY_2D_TILED_THIN1) | PIPE_CONFIG(ADDR_SURF_P16_32x32_16x16) | TILE_SPLIT(ADDR_SURF_TILE_SPLIT_64B) | @@ -3504,6 +3556,7 @@ gfx_v8_0_raster_config(struct amdgpu_device *adev, u32 *rconf, u32 *rconf1) { switch (adev->asic_type) { case CHIP_FIJI: + case CHIP_VEGAM: *rconf |= RB_MAP_PKR0(2) | RB_MAP_PKR1(2) | RB_XSEL2(1) | PKR_MAP(2) | PKR_XSEL(1) | PKR_YSEL(1) | @@ -4071,7 +4124,8 @@ static void gfx_v8_0_init_pg(struct amdgpu_device *adev) gfx_v8_0_init_power_gating(adev); WREG32(mmRLC_PG_ALWAYS_ON_CU_MASK, adev->gfx.cu_info.ao_cu_mask); } else if ((adev->asic_type == CHIP_POLARIS11) || - (adev->asic_type == CHIP_POLARIS12)) { + (adev->asic_type == CHIP_POLARIS12) || + (adev->asic_type == CHIP_VEGAM)) { gfx_v8_0_init_csb(adev); gfx_v8_0_init_save_restore_list(adev); gfx_v8_0_enable_save_restore_machine(adev); @@ -4146,7 +4200,8 @@ static int gfx_v8_0_rlc_resume(struct amdgpu_device *adev) WREG32(mmRLC_CGCG_CGLS_CTRL, tmp); if (adev->asic_type == CHIP_POLARIS11 || adev->asic_type == CHIP_POLARIS10 || - adev->asic_type == CHIP_POLARIS12) { + adev->asic_type == CHIP_POLARIS12 || + adev->asic_type == CHIP_VEGAM) { tmp = RREG32(mmRLC_CGCG_CGLS_CTRL_3D); tmp &= ~0x3; WREG32(mmRLC_CGCG_CGLS_CTRL_3D, tmp); @@ -5498,7 +5553,8 @@ static void gfx_v8_0_enable_gfx_static_mg_power_gating(struct amdgpu_device *ade bool enable) { if ((adev->asic_type == CHIP_POLARIS11) || - (adev->asic_type == CHIP_POLARIS12)) + (adev->asic_type == CHIP_POLARIS12) || + (adev->asic_type == CHIP_VEGAM)) /* Send msg to SMU via Powerplay */ amdgpu_device_ip_set_powergating_state(adev, AMD_IP_BLOCK_TYPE_SMC, @@ -5588,6 +5644,7 @@ static int gfx_v8_0_set_powergating_state(void *handle, break; case CHIP_POLARIS11: case CHIP_POLARIS12: + case CHIP_VEGAM: if ((adev->pg_flags & AMD_PG_SUPPORT_GFX_SMG) && enable) gfx_v8_0_enable_gfx_static_mg_power_gating(adev, true); else @@ -6154,6 +6211,7 @@ static int gfx_v8_0_set_clockgating_state(void *handle, case CHIP_POLARIS10: case CHIP_POLARIS11: case CHIP_POLARIS12: + case CHIP_VEGAM: gfx_v8_0_polaris_update_gfx_clock_gating(adev, state); break; default: diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c index 9d39fd5b1822..a69153435ea7 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c @@ -27,6 +27,7 @@ #include "amdgpu_gfx.h" #include "soc15.h" #include "soc15d.h" +#include "amdgpu_atomfirmware.h" #include "gc/gc_9_0_offset.h" #include "gc/gc_9_0_sh_mask.h" @@ -41,7 +42,6 @@ #define GFX9_MEC_HPD_SIZE 2048 #define RLCG_UCODE_LOADING_START_ADDRESS 0x00002000L #define RLC_SAVE_RESTORE_ADDR_STARTING_OFFSET 0x00000000L -#define GFX9_RLC_FORMAT_DIRECT_REG_LIST_LENGTH 34 #define mmPWR_MISC_CNTL_STATUS 0x0183 #define mmPWR_MISC_CNTL_STATUS_BASE_IDX 0 @@ -64,6 +64,13 @@ MODULE_FIRMWARE("amdgpu/vega12_mec.bin"); MODULE_FIRMWARE("amdgpu/vega12_mec2.bin"); MODULE_FIRMWARE("amdgpu/vega12_rlc.bin"); +MODULE_FIRMWARE("amdgpu/vega20_ce.bin"); +MODULE_FIRMWARE("amdgpu/vega20_pfp.bin"); +MODULE_FIRMWARE("amdgpu/vega20_me.bin"); +MODULE_FIRMWARE("amdgpu/vega20_mec.bin"); +MODULE_FIRMWARE("amdgpu/vega20_mec2.bin"); +MODULE_FIRMWARE("amdgpu/vega20_rlc.bin"); + MODULE_FIRMWARE("amdgpu/raven_ce.bin"); MODULE_FIRMWARE("amdgpu/raven_pfp.bin"); MODULE_FIRMWARE("amdgpu/raven_me.bin"); @@ -73,29 +80,22 @@ MODULE_FIRMWARE("amdgpu/raven_rlc.bin"); static const struct soc15_reg_golden golden_settings_gc_9_0[] = { - SOC15_REG_GOLDEN_VALUE(GC, 0, mmCPC_UTCL1_CNTL, 0x08000000, 0x08000080), - SOC15_REG_GOLDEN_VALUE(GC, 0, mmCPF_UTCL1_CNTL, 0x08000000, 0x08000080), - SOC15_REG_GOLDEN_VALUE(GC, 0, mmCPG_UTCL1_CNTL, 0x08000000, 0x08000080), SOC15_REG_GOLDEN_VALUE(GC, 0, mmDB_DEBUG2, 0xf00fffff, 0x00000420), SOC15_REG_GOLDEN_VALUE(GC, 0, mmGB_GPU_ID, 0x0000000f, 0x00000000), - SOC15_REG_GOLDEN_VALUE(GC, 0, mmIA_UTCL1_CNTL, 0x08000000, 0x08000080), SOC15_REG_GOLDEN_VALUE(GC, 0, mmPA_SC_BINNER_EVENT_CNTL_3, 0x00000003, 0x82400024), SOC15_REG_GOLDEN_VALUE(GC, 0, mmPA_SC_ENHANCE, 0x3fffffff, 0x00000001), SOC15_REG_GOLDEN_VALUE(GC, 0, mmPA_SC_LINE_STIPPLE_STATE, 0x0000ff0f, 0x00000000), - SOC15_REG_GOLDEN_VALUE(GC, 0, mmRLC_GPM_UTCL1_CNTL_0, 0x08000000, 0x08000080), - SOC15_REG_GOLDEN_VALUE(GC, 0, mmRLC_GPM_UTCL1_CNTL_1, 0x08000000, 0x08000080), - SOC15_REG_GOLDEN_VALUE(GC, 0, mmRLC_GPM_UTCL1_CNTL_2, 0x08000000, 0x08000080), - SOC15_REG_GOLDEN_VALUE(GC, 0, mmRLC_PREWALKER_UTCL1_CNTL, 0x08000000, 0x08000080), - SOC15_REG_GOLDEN_VALUE(GC, 0, mmRLC_SPM_UTCL1_CNTL, 0x08000000, 0x08000080), SOC15_REG_GOLDEN_VALUE(GC, 0, mmSH_MEM_CONFIG, 0x00001000, 0x00001000), - SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_CONFIG_CNTL_1, 0x0000000f, 0x01000107), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_RESOURCE_RESERVE_CU_0, 0x0007ffff, 0x00000800), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_RESOURCE_RESERVE_CU_1, 0x0007ffff, 0x00000800), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_RESOURCE_RESERVE_EN_CU_0, 0x01ffffff, 0x0000ff87), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_RESOURCE_RESERVE_EN_CU_1, 0x01ffffff, 0x0000ff8f), SOC15_REG_GOLDEN_VALUE(GC, 0, mmSQC_CONFIG, 0x03000000, 0x020a2000), SOC15_REG_GOLDEN_VALUE(GC, 0, mmTA_CNTL_AUX, 0xfffffeef, 0x010b0000), SOC15_REG_GOLDEN_VALUE(GC, 0, mmTCP_CHAN_STEER_HI, 0xffffffff, 0x4a2c0e68), SOC15_REG_GOLDEN_VALUE(GC, 0, mmTCP_CHAN_STEER_LO, 0xffffffff, 0xb5d3f197), SOC15_REG_GOLDEN_VALUE(GC, 0, mmVGT_CACHE_INVALIDATION, 0x3fff3af3, 0x19200000), - SOC15_REG_GOLDEN_VALUE(GC, 0, mmVGT_GS_MAX_WAVE_ID, 0x00000fff, 0x000003ff), - SOC15_REG_GOLDEN_VALUE(GC, 0, mmWD_UTCL1_CNTL, 0x08000000, 0x08000080) + SOC15_REG_GOLDEN_VALUE(GC, 0, mmVGT_GS_MAX_WAVE_ID, 0x00000fff, 0x000003ff) }; static const struct soc15_reg_golden golden_settings_gc_9_0_vg10[] = @@ -109,6 +109,21 @@ static const struct soc15_reg_golden golden_settings_gc_9_0_vg10[] = SOC15_REG_GOLDEN_VALUE(GC, 0, mmTD_CNTL, 0x00001800, 0x00000800) }; +static const struct soc15_reg_golden golden_settings_gc_9_0_vg20[] = +{ + SOC15_REG_GOLDEN_VALUE(GC, 0, mmCB_DCC_CONFIG, 0x0f000080, 0x04000080), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmCB_HW_CONTROL_2, 0x0f000000, 0x0a000000), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmCB_HW_CONTROL_3, 0x30000000, 0x10000000), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmGB_ADDR_CONFIG, 0xf3e777ff, 0x22014042), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmGB_ADDR_CONFIG_READ, 0xf3e777ff, 0x22014042), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmDB_DEBUG2, 0x00003e00, 0x00000400), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmPA_SC_ENHANCE_1, 0xff840000, 0x04040000), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmRMI_UTCL1_CNTL2, 0x00030000, 0x00030000), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmSPI_CONFIG_CNTL_1, 0xffff010f, 0x01000107), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmTA_CNTL_AUX, 0x000b0000, 0x000b0000), + SOC15_REG_GOLDEN_VALUE(GC, 0, mmTD_CNTL, 0x01000000, 0x01000000) +}; + static const struct soc15_reg_golden golden_settings_gc_9_1[] = { SOC15_REG_GOLDEN_VALUE(GC, 0, mmCB_HW_CONTROL, 0xfffdf3cf, 0x00014104), @@ -185,6 +200,30 @@ static const struct soc15_reg_golden golden_settings_gc_9_2_1_vg12[] = SOC15_REG_GOLDEN_VALUE(GC, 0, mmTD_CNTL, 0x01bd9f33, 0x01000000) }; +static const u32 GFX_RLC_SRM_INDEX_CNTL_ADDR_OFFSETS[] = +{ + mmRLC_SRM_INDEX_CNTL_ADDR_0 - mmRLC_SRM_INDEX_CNTL_ADDR_0, + mmRLC_SRM_INDEX_CNTL_ADDR_1 - mmRLC_SRM_INDEX_CNTL_ADDR_0, + mmRLC_SRM_INDEX_CNTL_ADDR_2 - mmRLC_SRM_INDEX_CNTL_ADDR_0, + mmRLC_SRM_INDEX_CNTL_ADDR_3 - mmRLC_SRM_INDEX_CNTL_ADDR_0, + mmRLC_SRM_INDEX_CNTL_ADDR_4 - mmRLC_SRM_INDEX_CNTL_ADDR_0, + mmRLC_SRM_INDEX_CNTL_ADDR_5 - mmRLC_SRM_INDEX_CNTL_ADDR_0, + mmRLC_SRM_INDEX_CNTL_ADDR_6 - mmRLC_SRM_INDEX_CNTL_ADDR_0, + mmRLC_SRM_INDEX_CNTL_ADDR_7 - mmRLC_SRM_INDEX_CNTL_ADDR_0, +}; + +static const u32 GFX_RLC_SRM_INDEX_CNTL_DATA_OFFSETS[] = +{ + mmRLC_SRM_INDEX_CNTL_DATA_0 - mmRLC_SRM_INDEX_CNTL_DATA_0, + mmRLC_SRM_INDEX_CNTL_DATA_1 - mmRLC_SRM_INDEX_CNTL_DATA_0, + mmRLC_SRM_INDEX_CNTL_DATA_2 - mmRLC_SRM_INDEX_CNTL_DATA_0, + mmRLC_SRM_INDEX_CNTL_DATA_3 - mmRLC_SRM_INDEX_CNTL_DATA_0, + mmRLC_SRM_INDEX_CNTL_DATA_4 - mmRLC_SRM_INDEX_CNTL_DATA_0, + mmRLC_SRM_INDEX_CNTL_DATA_5 - mmRLC_SRM_INDEX_CNTL_DATA_0, + mmRLC_SRM_INDEX_CNTL_DATA_6 - mmRLC_SRM_INDEX_CNTL_DATA_0, + mmRLC_SRM_INDEX_CNTL_DATA_7 - mmRLC_SRM_INDEX_CNTL_DATA_0, +}; + #define VEGA10_GB_ADDR_CONFIG_GOLDEN 0x2a114042 #define VEGA12_GB_ADDR_CONFIG_GOLDEN 0x24104041 #define RAVEN_GB_ADDR_CONFIG_GOLDEN 0x24000042 @@ -218,6 +257,14 @@ static void gfx_v9_0_init_golden_registers(struct amdgpu_device *adev) golden_settings_gc_9_2_1_vg12, ARRAY_SIZE(golden_settings_gc_9_2_1_vg12)); break; + case CHIP_VEGA20: + soc15_program_register_sequence(adev, + golden_settings_gc_9_0, + ARRAY_SIZE(golden_settings_gc_9_0)); + soc15_program_register_sequence(adev, + golden_settings_gc_9_0_vg20, + ARRAY_SIZE(golden_settings_gc_9_0_vg20)); + break; case CHIP_RAVEN: soc15_program_register_sequence(adev, golden_settings_gc_9_1, @@ -401,6 +448,27 @@ static void gfx_v9_0_free_microcode(struct amdgpu_device *adev) kfree(adev->gfx.rlc.register_list_format); } +static void gfx_v9_0_init_rlc_ext_microcode(struct amdgpu_device *adev) +{ + const struct rlc_firmware_header_v2_1 *rlc_hdr; + + rlc_hdr = (const struct rlc_firmware_header_v2_1 *)adev->gfx.rlc_fw->data; + adev->gfx.rlc_srlc_fw_version = le32_to_cpu(rlc_hdr->save_restore_list_cntl_ucode_ver); + adev->gfx.rlc_srlc_feature_version = le32_to_cpu(rlc_hdr->save_restore_list_cntl_feature_ver); + adev->gfx.rlc.save_restore_list_cntl_size_bytes = le32_to_cpu(rlc_hdr->save_restore_list_cntl_size_bytes); + adev->gfx.rlc.save_restore_list_cntl = (u8 *)rlc_hdr + le32_to_cpu(rlc_hdr->save_restore_list_cntl_offset_bytes); + adev->gfx.rlc_srlg_fw_version = le32_to_cpu(rlc_hdr->save_restore_list_gpm_ucode_ver); + adev->gfx.rlc_srlg_feature_version = le32_to_cpu(rlc_hdr->save_restore_list_gpm_feature_ver); + adev->gfx.rlc.save_restore_list_gpm_size_bytes = le32_to_cpu(rlc_hdr->save_restore_list_gpm_size_bytes); + adev->gfx.rlc.save_restore_list_gpm = (u8 *)rlc_hdr + le32_to_cpu(rlc_hdr->save_restore_list_gpm_offset_bytes); + adev->gfx.rlc_srls_fw_version = le32_to_cpu(rlc_hdr->save_restore_list_srm_ucode_ver); + adev->gfx.rlc_srls_feature_version = le32_to_cpu(rlc_hdr->save_restore_list_srm_feature_ver); + adev->gfx.rlc.save_restore_list_srm_size_bytes = le32_to_cpu(rlc_hdr->save_restore_list_srm_size_bytes); + adev->gfx.rlc.save_restore_list_srm = (u8 *)rlc_hdr + le32_to_cpu(rlc_hdr->save_restore_list_srm_offset_bytes); + adev->gfx.rlc.reg_list_format_direct_reg_list_length = + le32_to_cpu(rlc_hdr->reg_list_format_direct_reg_list_length); +} + static int gfx_v9_0_init_microcode(struct amdgpu_device *adev) { const char *chip_name; @@ -412,6 +480,8 @@ static int gfx_v9_0_init_microcode(struct amdgpu_device *adev) const struct rlc_firmware_header_v2_0 *rlc_hdr; unsigned int *tmp = NULL; unsigned int i = 0; + uint16_t version_major; + uint16_t version_minor; DRM_DEBUG("\n"); @@ -422,6 +492,9 @@ static int gfx_v9_0_init_microcode(struct amdgpu_device *adev) case CHIP_VEGA12: chip_name = "vega12"; break; + case CHIP_VEGA20: + chip_name = "vega20"; + break; case CHIP_RAVEN: chip_name = "raven"; break; @@ -468,6 +541,12 @@ static int gfx_v9_0_init_microcode(struct amdgpu_device *adev) goto out; err = amdgpu_ucode_validate(adev->gfx.rlc_fw); rlc_hdr = (const struct rlc_firmware_header_v2_0 *)adev->gfx.rlc_fw->data; + + version_major = le16_to_cpu(rlc_hdr->header.header_version_major); + version_minor = le16_to_cpu(rlc_hdr->header.header_version_minor); + if (version_major == 2 && version_minor == 1) + adev->gfx.rlc.is_rlc_v2_1 = true; + adev->gfx.rlc_fw_version = le32_to_cpu(rlc_hdr->header.ucode_version); adev->gfx.rlc_feature_version = le32_to_cpu(rlc_hdr->ucode_feature_version); adev->gfx.rlc.save_and_restore_offset = @@ -508,6 +587,9 @@ static int gfx_v9_0_init_microcode(struct amdgpu_device *adev) for (i = 0 ; i < (rlc_hdr->reg_list_size_bytes >> 2); i++) adev->gfx.rlc.register_restore[i] = le32_to_cpu(tmp[i]); + if (adev->gfx.rlc.is_rlc_v2_1) + gfx_v9_0_init_rlc_ext_microcode(adev); + snprintf(fw_name, sizeof(fw_name), "amdgpu/%s_mec.bin", chip_name); err = request_firmware(&adev->gfx.mec_fw, fw_name, adev->dev); if (err) @@ -566,6 +648,26 @@ static int gfx_v9_0_init_microcode(struct amdgpu_device *adev) adev->firmware.fw_size += ALIGN(le32_to_cpu(header->ucode_size_bytes), PAGE_SIZE); + if (adev->gfx.rlc.is_rlc_v2_1) { + info = &adev->firmware.ucode[AMDGPU_UCODE_ID_RLC_RESTORE_LIST_CNTL]; + info->ucode_id = AMDGPU_UCODE_ID_RLC_RESTORE_LIST_CNTL; + info->fw = adev->gfx.rlc_fw; + adev->firmware.fw_size += + ALIGN(adev->gfx.rlc.save_restore_list_cntl_size_bytes, PAGE_SIZE); + + info = &adev->firmware.ucode[AMDGPU_UCODE_ID_RLC_RESTORE_LIST_GPM_MEM]; + info->ucode_id = AMDGPU_UCODE_ID_RLC_RESTORE_LIST_GPM_MEM; + info->fw = adev->gfx.rlc_fw; + adev->firmware.fw_size += + ALIGN(adev->gfx.rlc.save_restore_list_gpm_size_bytes, PAGE_SIZE); + + info = &adev->firmware.ucode[AMDGPU_UCODE_ID_RLC_RESTORE_LIST_SRM_MEM]; + info->ucode_id = AMDGPU_UCODE_ID_RLC_RESTORE_LIST_SRM_MEM; + info->fw = adev->gfx.rlc_fw; + adev->firmware.fw_size += + ALIGN(adev->gfx.rlc.save_restore_list_srm_size_bytes, PAGE_SIZE); + } + info = &adev->firmware.ucode[AMDGPU_UCODE_ID_CP_MEC1]; info->ucode_id = AMDGPU_UCODE_ID_CP_MEC1; info->fw = adev->gfx.mec_fw; @@ -1013,9 +1115,10 @@ static const struct amdgpu_gfx_funcs gfx_v9_0_gfx_funcs = { .select_me_pipe_q = &gfx_v9_0_select_me_pipe_q }; -static void gfx_v9_0_gpu_early_init(struct amdgpu_device *adev) +static int gfx_v9_0_gpu_early_init(struct amdgpu_device *adev) { u32 gb_addr_config; + int err; adev->gfx.funcs = &gfx_v9_0_gfx_funcs; @@ -1037,6 +1140,20 @@ static void gfx_v9_0_gpu_early_init(struct amdgpu_device *adev) gb_addr_config = VEGA12_GB_ADDR_CONFIG_GOLDEN; DRM_INFO("fix gfx.config for vega12\n"); break; + case CHIP_VEGA20: + adev->gfx.config.max_hw_contexts = 8; + adev->gfx.config.sc_prim_fifo_size_frontend = 0x20; + adev->gfx.config.sc_prim_fifo_size_backend = 0x100; + adev->gfx.config.sc_hiz_tile_fifo_size = 0x30; + adev->gfx.config.sc_earlyz_tile_fifo_size = 0x4C0; + gb_addr_config = RREG32_SOC15(GC, 0, mmGB_ADDR_CONFIG); + gb_addr_config &= ~0xf3e777ff; + gb_addr_config |= 0x22014042; + /* check vbios table if gpu info is not available */ + err = amdgpu_atomfirmware_get_gfx_info(adev); + if (err) + return err; + break; case CHIP_RAVEN: adev->gfx.config.max_hw_contexts = 8; adev->gfx.config.sc_prim_fifo_size_frontend = 0x20; @@ -1086,6 +1203,8 @@ static void gfx_v9_0_gpu_early_init(struct amdgpu_device *adev) adev->gfx.config.gb_addr_config, GB_ADDR_CONFIG, PIPE_INTERLEAVE_SIZE)); + + return 0; } static int gfx_v9_0_ngg_create_buf(struct amdgpu_device *adev, @@ -1319,6 +1438,7 @@ static int gfx_v9_0_sw_init(void *handle) switch (adev->asic_type) { case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: case CHIP_RAVEN: adev->gfx.mec.num_mec = 2; break; @@ -1446,7 +1566,9 @@ static int gfx_v9_0_sw_init(void *handle) adev->gfx.ce_ram_size = 0x8000; - gfx_v9_0_gpu_early_init(adev); + r = gfx_v9_0_gpu_early_init(adev); + if (r) + return r; r = gfx_v9_0_ngg_init(adev); if (r) @@ -1600,6 +1722,7 @@ static void gfx_v9_0_gpu_init(struct amdgpu_device *adev) gfx_v9_0_setup_rb(adev); gfx_v9_0_get_cu_info(adev, &adev->gfx.cu_info); + adev->gfx.config.db_debug2 = RREG32_SOC15(GC, 0, mmDB_DEBUG2); /* XXX SH_MEM regs */ /* where to put LDS, scratch, GPUVM in FSA64 space */ @@ -1616,7 +1739,10 @@ static void gfx_v9_0_gpu_init(struct amdgpu_device *adev) tmp = REG_SET_FIELD(0, SH_MEM_CONFIG, ALIGNMENT_MODE, SH_MEM_ALIGNMENT_MODE_UNALIGNED); WREG32_SOC15(GC, 0, mmSH_MEM_CONFIG, tmp); - tmp = adev->gmc.shared_aperture_start >> 48; + tmp = REG_SET_FIELD(0, SH_MEM_BASES, PRIVATE_BASE, + (adev->gmc.private_aperture_start >> 48)); + tmp = REG_SET_FIELD(tmp, SH_MEM_BASES, SHARED_BASE, + (adev->gmc.shared_aperture_start >> 48)); WREG32_SOC15(GC, 0, mmSH_MEM_BASES, tmp); } } @@ -1708,55 +1834,44 @@ static void gfx_v9_0_init_csb(struct amdgpu_device *adev) adev->gfx.rlc.clear_state_size); } -static void gfx_v9_0_parse_ind_reg_list(int *register_list_format, +static void gfx_v9_1_parse_ind_reg_list(int *register_list_format, int indirect_offset, int list_size, int *unique_indirect_regs, - int *unique_indirect_reg_count, - int max_indirect_reg_count, + int unique_indirect_reg_count, int *indirect_start_offsets, int *indirect_start_offsets_count, - int max_indirect_start_offsets_count) + int max_start_offsets_count) { int idx; - bool new_entry = true; for (; indirect_offset < list_size; indirect_offset++) { + WARN_ON(*indirect_start_offsets_count >= max_start_offsets_count); + indirect_start_offsets[*indirect_start_offsets_count] = indirect_offset; + *indirect_start_offsets_count = *indirect_start_offsets_count + 1; + + while (register_list_format[indirect_offset] != 0xFFFFFFFF) { + indirect_offset += 2; + + /* look for the matching indice */ + for (idx = 0; idx < unique_indirect_reg_count; idx++) { + if (unique_indirect_regs[idx] == + register_list_format[indirect_offset] || + !unique_indirect_regs[idx]) + break; + } - if (new_entry) { - new_entry = false; - indirect_start_offsets[*indirect_start_offsets_count] = indirect_offset; - *indirect_start_offsets_count = *indirect_start_offsets_count + 1; - BUG_ON(*indirect_start_offsets_count >= max_indirect_start_offsets_count); - } - - if (register_list_format[indirect_offset] == 0xFFFFFFFF) { - new_entry = true; - continue; - } - - indirect_offset += 2; + BUG_ON(idx >= unique_indirect_reg_count); - /* look for the matching indice */ - for (idx = 0; idx < *unique_indirect_reg_count; idx++) { - if (unique_indirect_regs[idx] == - register_list_format[indirect_offset]) - break; - } + if (!unique_indirect_regs[idx]) + unique_indirect_regs[idx] = register_list_format[indirect_offset]; - if (idx >= *unique_indirect_reg_count) { - unique_indirect_regs[*unique_indirect_reg_count] = - register_list_format[indirect_offset]; - idx = *unique_indirect_reg_count; - *unique_indirect_reg_count = *unique_indirect_reg_count + 1; - BUG_ON(*unique_indirect_reg_count >= max_indirect_reg_count); + indirect_offset++; } - - register_list_format[indirect_offset] = idx; } } -static int gfx_v9_0_init_rlc_save_restore_list(struct amdgpu_device *adev) +static int gfx_v9_1_init_rlc_save_restore_list(struct amdgpu_device *adev) { int unique_indirect_regs[] = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}; int unique_indirect_reg_count = 0; @@ -1765,7 +1880,7 @@ static int gfx_v9_0_init_rlc_save_restore_list(struct amdgpu_device *adev) int indirect_start_offsets_count = 0; int list_size = 0; - int i = 0; + int i = 0, j = 0; u32 tmp = 0; u32 *register_list_format = @@ -1776,15 +1891,15 @@ static int gfx_v9_0_init_rlc_save_restore_list(struct amdgpu_device *adev) adev->gfx.rlc.reg_list_format_size_bytes); /* setup unique_indirect_regs array and indirect_start_offsets array */ - gfx_v9_0_parse_ind_reg_list(register_list_format, - GFX9_RLC_FORMAT_DIRECT_REG_LIST_LENGTH, - adev->gfx.rlc.reg_list_format_size_bytes >> 2, - unique_indirect_regs, - &unique_indirect_reg_count, - ARRAY_SIZE(unique_indirect_regs), - indirect_start_offsets, - &indirect_start_offsets_count, - ARRAY_SIZE(indirect_start_offsets)); + unique_indirect_reg_count = ARRAY_SIZE(unique_indirect_regs); + gfx_v9_1_parse_ind_reg_list(register_list_format, + adev->gfx.rlc.reg_list_format_direct_reg_list_length, + adev->gfx.rlc.reg_list_format_size_bytes >> 2, + unique_indirect_regs, + unique_indirect_reg_count, + indirect_start_offsets, + &indirect_start_offsets_count, + ARRAY_SIZE(indirect_start_offsets)); /* enable auto inc in case it is disabled */ tmp = RREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_SRM_CNTL)); @@ -1798,19 +1913,37 @@ static int gfx_v9_0_init_rlc_save_restore_list(struct amdgpu_device *adev) WREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_SRM_ARAM_DATA), adev->gfx.rlc.register_restore[i]); - /* load direct register */ - WREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_SRM_ARAM_ADDR), 0); - for (i = 0; i < adev->gfx.rlc.reg_list_size_bytes >> 2; i++) - WREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_SRM_ARAM_DATA), - adev->gfx.rlc.register_restore[i]); - /* load indirect register */ WREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_GPM_SCRATCH_ADDR), adev->gfx.rlc.reg_list_format_start); - for (i = 0; i < adev->gfx.rlc.reg_list_format_size_bytes >> 2; i++) + + /* direct register portion */ + for (i = 0; i < adev->gfx.rlc.reg_list_format_direct_reg_list_length; i++) WREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_GPM_SCRATCH_DATA), register_list_format[i]); + /* indirect register portion */ + while (i < (adev->gfx.rlc.reg_list_format_size_bytes >> 2)) { + if (register_list_format[i] == 0xFFFFFFFF) { + WREG32_SOC15(GC, 0, mmRLC_GPM_SCRATCH_DATA, register_list_format[i++]); + continue; + } + + WREG32_SOC15(GC, 0, mmRLC_GPM_SCRATCH_DATA, register_list_format[i++]); + WREG32_SOC15(GC, 0, mmRLC_GPM_SCRATCH_DATA, register_list_format[i++]); + + for (j = 0; j < unique_indirect_reg_count; j++) { + if (register_list_format[i] == unique_indirect_regs[j]) { + WREG32_SOC15(GC, 0, mmRLC_GPM_SCRATCH_DATA, j); + break; + } + } + + BUG_ON(j >= unique_indirect_reg_count); + + i++; + } + /* set save/restore list size */ list_size = adev->gfx.rlc.reg_list_size_bytes >> 2; list_size = list_size >> 1; @@ -1823,14 +1956,19 @@ static int gfx_v9_0_init_rlc_save_restore_list(struct amdgpu_device *adev) adev->gfx.rlc.starting_offsets_start); for (i = 0; i < ARRAY_SIZE(indirect_start_offsets); i++) WREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_GPM_SCRATCH_DATA), - indirect_start_offsets[i]); + indirect_start_offsets[i]); /* load unique indirect regs*/ for (i = 0; i < ARRAY_SIZE(unique_indirect_regs); i++) { - WREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_SRM_INDEX_CNTL_ADDR_0) + i, - unique_indirect_regs[i] & 0x3FFFF); - WREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_SRM_INDEX_CNTL_DATA_0) + i, - unique_indirect_regs[i] >> 20); + if (unique_indirect_regs[i] != 0) { + WREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_SRM_INDEX_CNTL_ADDR_0) + + GFX_RLC_SRM_INDEX_CNTL_ADDR_OFFSETS[i], + unique_indirect_regs[i] & 0x3FFFF); + + WREG32(SOC15_REG_OFFSET(GC, 0, mmRLC_SRM_INDEX_CNTL_DATA_0) + + GFX_RLC_SRM_INDEX_CNTL_DATA_OFFSETS[i], + unique_indirect_regs[i] >> 20); + } } kfree(register_list_format); @@ -2010,6 +2148,9 @@ static void gfx_v9_0_enable_gfx_dynamic_mg_power_gating(struct amdgpu_device *ad static void gfx_v9_0_init_pg(struct amdgpu_device *adev) { + if (!adev->gfx.rlc.is_rlc_v2_1) + return; + if (adev->pg_flags & (AMD_PG_SUPPORT_GFX_PG | AMD_PG_SUPPORT_GFX_SMG | AMD_PG_SUPPORT_GFX_DMG | @@ -2017,27 +2158,12 @@ static void gfx_v9_0_init_pg(struct amdgpu_device *adev) AMD_PG_SUPPORT_GDS | AMD_PG_SUPPORT_RLC_SMU_HS)) { gfx_v9_0_init_csb(adev); - gfx_v9_0_init_rlc_save_restore_list(adev); + gfx_v9_1_init_rlc_save_restore_list(adev); gfx_v9_0_enable_save_restore_machine(adev); - if (adev->asic_type == CHIP_RAVEN) { - WREG32(mmRLC_JUMP_TABLE_RESTORE, - adev->gfx.rlc.cp_table_gpu_addr >> 8); - gfx_v9_0_init_gfx_power_gating(adev); - - if (adev->pg_flags & AMD_PG_SUPPORT_RLC_SMU_HS) { - gfx_v9_0_enable_sck_slow_down_on_power_up(adev, true); - gfx_v9_0_enable_sck_slow_down_on_power_down(adev, true); - } else { - gfx_v9_0_enable_sck_slow_down_on_power_up(adev, false); - gfx_v9_0_enable_sck_slow_down_on_power_down(adev, false); - } - - if (adev->pg_flags & AMD_PG_SUPPORT_CP) - gfx_v9_0_enable_cp_power_gating(adev, true); - else - gfx_v9_0_enable_cp_power_gating(adev, false); - } + WREG32(mmRLC_JUMP_TABLE_RESTORE, + adev->gfx.rlc.cp_table_gpu_addr >> 8); + gfx_v9_0_init_gfx_power_gating(adev); } } @@ -3061,6 +3187,9 @@ static int gfx_v9_0_hw_fini(void *handle) struct amdgpu_device *adev = (struct amdgpu_device *)handle; int i; + amdgpu_device_ip_set_powergating_state(adev, AMD_IP_BLOCK_TYPE_GFX, + AMD_PG_STATE_UNGATE); + amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0); amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0); @@ -3339,8 +3468,7 @@ static void gfx_v9_0_exit_rlc_safe_mode(struct amdgpu_device *adev) static void gfx_v9_0_update_gfx_cg_power_gating(struct amdgpu_device *adev, bool enable) { - /* TODO: double check if we need to perform under safe mdoe */ - /* gfx_v9_0_enter_rlc_safe_mode(adev); */ + gfx_v9_0_enter_rlc_safe_mode(adev); if ((adev->pg_flags & AMD_PG_SUPPORT_GFX_PG) && enable) { gfx_v9_0_enable_gfx_cg_power_gating(adev, true); @@ -3351,7 +3479,7 @@ static void gfx_v9_0_update_gfx_cg_power_gating(struct amdgpu_device *adev, gfx_v9_0_enable_gfx_pipeline_powergating(adev, false); } - /* gfx_v9_0_exit_rlc_safe_mode(adev); */ + gfx_v9_0_exit_rlc_safe_mode(adev); } static void gfx_v9_0_update_gfx_mg_power_gating(struct amdgpu_device *adev, @@ -3605,6 +3733,7 @@ static int gfx_v9_0_set_clockgating_state(void *handle, switch (adev->asic_type) { case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: case CHIP_RAVEN: gfx_v9_0_update_gfx_clock_gating(adev, state == AMD_CG_STATE_GATE ? true : false); @@ -3742,7 +3871,7 @@ static void gfx_v9_0_ring_emit_ib_gfx(struct amdgpu_ring *ring, } amdgpu_ring_write(ring, header); -BUG_ON(ib->gpu_addr & 0x3); /* Dword align */ + BUG_ON(ib->gpu_addr & 0x3); /* Dword align */ amdgpu_ring_write(ring, #ifdef __BIG_ENDIAN (2 << 0) | @@ -3774,13 +3903,16 @@ static void gfx_v9_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, { bool write64bit = flags & AMDGPU_FENCE_FLAG_64BIT; bool int_sel = flags & AMDGPU_FENCE_FLAG_INT; + bool writeback = flags & AMDGPU_FENCE_FLAG_TC_WB_ONLY; /* RELEASE_MEM - flush caches, send int */ amdgpu_ring_write(ring, PACKET3(PACKET3_RELEASE_MEM, 6)); - amdgpu_ring_write(ring, (EOP_TCL1_ACTION_EN | - EOP_TC_ACTION_EN | - EOP_TC_WB_ACTION_EN | - EOP_TC_MD_ACTION_EN | + amdgpu_ring_write(ring, ((writeback ? (EOP_TC_WB_ACTION_EN | + EOP_TC_NC_ACTION_EN) : + (EOP_TCL1_ACTION_EN | + EOP_TC_ACTION_EN | + EOP_TC_WB_ACTION_EN | + EOP_TC_MD_ACTION_EN)) | EVENT_TYPE(CACHE_FLUSH_AND_INV_TS_EVENT) | EVENT_INDEX(5))); amdgpu_ring_write(ring, DATA_SEL(write64bit ? 2 : 1) | INT_SEL(int_sel ? 2 : 0)); @@ -4137,6 +4269,20 @@ static void gfx_v9_0_ring_emit_reg_wait(struct amdgpu_ring *ring, uint32_t reg, gfx_v9_0_wait_reg_mem(ring, 0, 0, 0, reg, 0, val, mask, 0x20); } +static void gfx_v9_0_ring_emit_reg_write_reg_wait(struct amdgpu_ring *ring, + uint32_t reg0, uint32_t reg1, + uint32_t ref, uint32_t mask) +{ + int usepfp = (ring->funcs->type == AMDGPU_RING_TYPE_GFX); + + if (amdgpu_sriov_vf(ring->adev)) + gfx_v9_0_wait_reg_mem(ring, usepfp, 0, 1, reg0, reg1, + ref, mask, 0x20); + else + amdgpu_ring_emit_reg_write_reg_wait_helper(ring, reg0, reg1, + ref, mask); +} + static void gfx_v9_0_set_gfx_eop_interrupt_state(struct amdgpu_device *adev, enum amdgpu_interrupt_state state) { @@ -4458,6 +4604,7 @@ static const struct amdgpu_ring_funcs gfx_v9_0_ring_funcs_gfx = { .emit_tmz = gfx_v9_0_ring_emit_tmz, .emit_wreg = gfx_v9_0_ring_emit_wreg, .emit_reg_wait = gfx_v9_0_ring_emit_reg_wait, + .emit_reg_write_reg_wait = gfx_v9_0_ring_emit_reg_write_reg_wait, }; static const struct amdgpu_ring_funcs gfx_v9_0_ring_funcs_compute = { @@ -4492,6 +4639,7 @@ static const struct amdgpu_ring_funcs gfx_v9_0_ring_funcs_compute = { .set_priority = gfx_v9_0_ring_set_priority_compute, .emit_wreg = gfx_v9_0_ring_emit_wreg, .emit_reg_wait = gfx_v9_0_ring_emit_reg_wait, + .emit_reg_write_reg_wait = gfx_v9_0_ring_emit_reg_write_reg_wait, }; static const struct amdgpu_ring_funcs gfx_v9_0_ring_funcs_kiq = { @@ -4522,6 +4670,7 @@ static const struct amdgpu_ring_funcs gfx_v9_0_ring_funcs_kiq = { .emit_rreg = gfx_v9_0_ring_emit_rreg, .emit_wreg = gfx_v9_0_ring_emit_wreg, .emit_reg_wait = gfx_v9_0_ring_emit_reg_wait, + .emit_reg_write_reg_wait = gfx_v9_0_ring_emit_reg_write_reg_wait, }; static void gfx_v9_0_set_ring_funcs(struct amdgpu_device *adev) @@ -4577,6 +4726,7 @@ static void gfx_v9_0_set_rlc_funcs(struct amdgpu_device *adev) switch (adev->asic_type) { case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: case CHIP_RAVEN: adev->gfx.rlc.funcs = &gfx_v9_0_rlc_funcs; break; @@ -4686,6 +4836,7 @@ static int gfx_v9_0_get_cu_info(struct amdgpu_device *adev, cu_info->number = active_cu_number; cu_info->ao_cu_mask = ao_cu_mask; + cu_info->simd_per_cu = NUM_SIMD_PER_CU; return 0; } diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c index 5617cf62c566..79f9ac29019b 100644 --- a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c @@ -819,12 +819,33 @@ static int gmc_v6_0_late_init(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; + amdgpu_bo_late_init(adev); + if (amdgpu_vm_fault_stop != AMDGPU_VM_FAULT_STOP_ALWAYS) return amdgpu_irq_get(adev, &adev->gmc.vm_fault, 0); else return 0; } +static unsigned gmc_v6_0_get_vbios_fb_size(struct amdgpu_device *adev) +{ + u32 d1vga_control = RREG32(mmD1VGA_CONTROL); + unsigned size; + + if (REG_GET_FIELD(d1vga_control, D1VGA_CONTROL, D1VGA_MODE_ENABLE)) { + size = 9 * 1024 * 1024; /* reserve 8MB for vga emulator and 1 MB for FB */ + } else { + u32 viewport = RREG32(mmVIEWPORT_SIZE); + size = (REG_GET_FIELD(viewport, VIEWPORT_SIZE, VIEWPORT_HEIGHT) * + REG_GET_FIELD(viewport, VIEWPORT_SIZE, VIEWPORT_WIDTH) * + 4); + } + /* return 0 if the pre-OS buffer uses up most of vram */ + if ((adev->gmc.real_vram_size - size) < (8 * 1024 * 1024)) + return 0; + return size; +} + static int gmc_v6_0_sw_init(void *handle) { int r; @@ -851,8 +872,6 @@ static int gmc_v6_0_sw_init(void *handle) adev->gmc.mc_mask = 0xffffffffffULL; - adev->gmc.stolen_size = 256 * 1024; - adev->need_dma32 = false; dma_bits = adev->need_dma32 ? 32 : 40; r = pci_set_dma_mask(adev->pdev, DMA_BIT_MASK(dma_bits)); @@ -878,6 +897,8 @@ static int gmc_v6_0_sw_init(void *handle) if (r) return r; + adev->gmc.stolen_size = gmc_v6_0_get_vbios_fb_size(adev); + r = amdgpu_bo_init(adev); if (r) return r; diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c index 80054f36e487..7147bfe25a23 100644 --- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c @@ -958,12 +958,33 @@ static int gmc_v7_0_late_init(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; + amdgpu_bo_late_init(adev); + if (amdgpu_vm_fault_stop != AMDGPU_VM_FAULT_STOP_ALWAYS) return amdgpu_irq_get(adev, &adev->gmc.vm_fault, 0); else return 0; } +static unsigned gmc_v7_0_get_vbios_fb_size(struct amdgpu_device *adev) +{ + u32 d1vga_control = RREG32(mmD1VGA_CONTROL); + unsigned size; + + if (REG_GET_FIELD(d1vga_control, D1VGA_CONTROL, D1VGA_MODE_ENABLE)) { + size = 9 * 1024 * 1024; /* reserve 8MB for vga emulator and 1 MB for FB */ + } else { + u32 viewport = RREG32(mmVIEWPORT_SIZE); + size = (REG_GET_FIELD(viewport, VIEWPORT_SIZE, VIEWPORT_HEIGHT) * + REG_GET_FIELD(viewport, VIEWPORT_SIZE, VIEWPORT_WIDTH) * + 4); + } + /* return 0 if the pre-OS buffer uses up most of vram */ + if ((adev->gmc.real_vram_size - size) < (8 * 1024 * 1024)) + return 0; + return size; +} + static int gmc_v7_0_sw_init(void *handle) { int r; @@ -998,8 +1019,6 @@ static int gmc_v7_0_sw_init(void *handle) */ adev->gmc.mc_mask = 0xffffffffffULL; /* 40 bit MC */ - adev->gmc.stolen_size = 256 * 1024; - /* set DMA mask + need_dma32 flags. * PCIE - can handle 40-bits. * IGP - can handle 40-bits @@ -1030,6 +1049,8 @@ static int gmc_v7_0_sw_init(void *handle) if (r) return r; + adev->gmc.stolen_size = gmc_v7_0_get_vbios_fb_size(adev); + /* Memory manager */ r = amdgpu_bo_init(adev); if (r) diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c index d71d4cb68f9c..1edbe6b477b5 100644 --- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c @@ -138,6 +138,7 @@ static void gmc_v8_0_init_golden_registers(struct amdgpu_device *adev) break; case CHIP_POLARIS11: case CHIP_POLARIS12: + case CHIP_VEGAM: amdgpu_device_program_register_sequence(adev, golden_settings_polaris11_a11, ARRAY_SIZE(golden_settings_polaris11_a11)); @@ -231,6 +232,7 @@ static int gmc_v8_0_init_microcode(struct amdgpu_device *adev) case CHIP_FIJI: case CHIP_CARRIZO: case CHIP_STONEY: + case CHIP_VEGAM: return 0; default: BUG(); } @@ -567,9 +569,10 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev) /* set the gart size */ if (amdgpu_gart_size == -1) { switch (adev->asic_type) { - case CHIP_POLARIS11: /* all engines support GPUVM */ case CHIP_POLARIS10: /* all engines support GPUVM */ + case CHIP_POLARIS11: /* all engines support GPUVM */ case CHIP_POLARIS12: /* all engines support GPUVM */ + case CHIP_VEGAM: /* all engines support GPUVM */ default: adev->gmc.gart_size = 256ULL << 20; break; @@ -1049,12 +1052,33 @@ static int gmc_v8_0_late_init(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; + amdgpu_bo_late_init(adev); + if (amdgpu_vm_fault_stop != AMDGPU_VM_FAULT_STOP_ALWAYS) return amdgpu_irq_get(adev, &adev->gmc.vm_fault, 0); else return 0; } +static unsigned gmc_v8_0_get_vbios_fb_size(struct amdgpu_device *adev) +{ + u32 d1vga_control = RREG32(mmD1VGA_CONTROL); + unsigned size; + + if (REG_GET_FIELD(d1vga_control, D1VGA_CONTROL, D1VGA_MODE_ENABLE)) { + size = 9 * 1024 * 1024; /* reserve 8MB for vga emulator and 1 MB for FB */ + } else { + u32 viewport = RREG32(mmVIEWPORT_SIZE); + size = (REG_GET_FIELD(viewport, VIEWPORT_SIZE, VIEWPORT_HEIGHT) * + REG_GET_FIELD(viewport, VIEWPORT_SIZE, VIEWPORT_WIDTH) * + 4); + } + /* return 0 if the pre-OS buffer uses up most of vram */ + if ((adev->gmc.real_vram_size - size) < (8 * 1024 * 1024)) + return 0; + return size; +} + #define mmMC_SEQ_MISC0_FIJI 0xA71 static int gmc_v8_0_sw_init(void *handle) @@ -1068,7 +1092,8 @@ static int gmc_v8_0_sw_init(void *handle) } else { u32 tmp; - if (adev->asic_type == CHIP_FIJI) + if ((adev->asic_type == CHIP_FIJI) || + (adev->asic_type == CHIP_VEGAM)) tmp = RREG32(mmMC_SEQ_MISC0_FIJI); else tmp = RREG32(mmMC_SEQ_MISC0); @@ -1096,8 +1121,6 @@ static int gmc_v8_0_sw_init(void *handle) */ adev->gmc.mc_mask = 0xffffffffffULL; /* 40 bit MC */ - adev->gmc.stolen_size = 256 * 1024; - /* set DMA mask + need_dma32 flags. * PCIE - can handle 40-bits. * IGP - can handle 40-bits @@ -1128,6 +1151,8 @@ static int gmc_v8_0_sw_init(void *handle) if (r) return r; + adev->gmc.stolen_size = gmc_v8_0_get_vbios_fb_size(adev); + /* Memory manager */ r = amdgpu_bo_init(adev); if (r) diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c index e687363900bb..3c0a85d4e4ab 100644 --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c @@ -43,19 +43,13 @@ #include "gfxhub_v1_0.h" #include "mmhub_v1_0.h" -#define mmDF_CS_AON0_DramBaseAddress0 0x0044 -#define mmDF_CS_AON0_DramBaseAddress0_BASE_IDX 0 -//DF_CS_AON0_DramBaseAddress0 -#define DF_CS_AON0_DramBaseAddress0__AddrRngVal__SHIFT 0x0 -#define DF_CS_AON0_DramBaseAddress0__LgcyMmioHoleEn__SHIFT 0x1 -#define DF_CS_AON0_DramBaseAddress0__IntLvNumChan__SHIFT 0x4 -#define DF_CS_AON0_DramBaseAddress0__IntLvAddrSel__SHIFT 0x8 -#define DF_CS_AON0_DramBaseAddress0__DramBaseAddr__SHIFT 0xc -#define DF_CS_AON0_DramBaseAddress0__AddrRngVal_MASK 0x00000001L -#define DF_CS_AON0_DramBaseAddress0__LgcyMmioHoleEn_MASK 0x00000002L -#define DF_CS_AON0_DramBaseAddress0__IntLvNumChan_MASK 0x000000F0L -#define DF_CS_AON0_DramBaseAddress0__IntLvAddrSel_MASK 0x00000700L -#define DF_CS_AON0_DramBaseAddress0__DramBaseAddr_MASK 0xFFFFF000L +/* add these here since we already include dce12 headers and these are for DCN */ +#define mmHUBP0_DCSURF_PRI_VIEWPORT_DIMENSION 0x055d +#define mmHUBP0_DCSURF_PRI_VIEWPORT_DIMENSION_BASE_IDX 2 +#define HUBP0_DCSURF_PRI_VIEWPORT_DIMENSION__PRI_VIEWPORT_WIDTH__SHIFT 0x0 +#define HUBP0_DCSURF_PRI_VIEWPORT_DIMENSION__PRI_VIEWPORT_HEIGHT__SHIFT 0x10 +#define HUBP0_DCSURF_PRI_VIEWPORT_DIMENSION__PRI_VIEWPORT_WIDTH_MASK 0x00003FFFL +#define HUBP0_DCSURF_PRI_VIEWPORT_DIMENSION__PRI_VIEWPORT_HEIGHT_MASK 0x3FFF0000L /* XXX Move this macro to VEGA10 header file, which is like vid.h for VI.*/ #define AMDGPU_NUM_OF_VMIDS 8 @@ -385,11 +379,9 @@ static uint64_t gmc_v9_0_emit_flush_gpu_tlb(struct amdgpu_ring *ring, amdgpu_ring_emit_wreg(ring, hub->ctx0_ptb_addr_hi32 + (2 * vmid), upper_32_bits(pd_addr)); - amdgpu_ring_emit_wreg(ring, hub->vm_inv_eng0_req + eng, req); - - /* wait for the invalidate to complete */ - amdgpu_ring_emit_reg_wait(ring, hub->vm_inv_eng0_ack + eng, - 1 << vmid, 1 << vmid); + amdgpu_ring_emit_reg_write_reg_wait(ring, hub->vm_inv_eng0_req + eng, + hub->vm_inv_eng0_ack + eng, + req, 1 << vmid); return pd_addr; } @@ -556,8 +548,7 @@ static int gmc_v9_0_early_init(void *handle) adev->gmc.shared_aperture_start = 0x2000000000000000ULL; adev->gmc.shared_aperture_end = adev->gmc.shared_aperture_start + (4ULL << 30) - 1; - adev->gmc.private_aperture_start = - adev->gmc.shared_aperture_end + 1; + adev->gmc.private_aperture_start = 0x1000000000000000ULL; adev->gmc.private_aperture_end = adev->gmc.private_aperture_start + (4ULL << 30) - 1; @@ -659,6 +650,11 @@ static int gmc_v9_0_late_init(void *handle) unsigned i; int r; + /* + * TODO - Uncomment once GART corruption issue is fixed. + */ + /* amdgpu_bo_late_init(adev); */ + for(i = 0; i < adev->num_rings; ++i) { struct amdgpu_ring *ring = adev->rings[i]; unsigned vmhub = ring->funcs->vmhub; @@ -679,6 +675,7 @@ static int gmc_v9_0_late_init(void *handle) DRM_INFO("ECC is active.\n"); } else if (r == 0) { DRM_INFO("ECC is not present.\n"); + adev->df_funcs->enable_ecc_force_par_wr_rmw(adev, false); } else { DRM_ERROR("gmc_v9_0_ecc_available() failed. r: %d\n", r); return r; @@ -697,10 +694,7 @@ static void gmc_v9_0_vram_gtt_location(struct amdgpu_device *adev, amdgpu_device_vram_location(adev, &adev->gmc, base); amdgpu_device_gart_location(adev, mc); /* base offset of vram pages */ - if (adev->flags & AMD_IS_APU) - adev->vm_manager.vram_base_offset = gfxhub_v1_0_get_mc_fb_offset(adev); - else - adev->vm_manager.vram_base_offset = 0; + adev->vm_manager.vram_base_offset = gfxhub_v1_0_get_mc_fb_offset(adev); } /** @@ -714,7 +708,6 @@ static void gmc_v9_0_vram_gtt_location(struct amdgpu_device *adev, */ static int gmc_v9_0_mc_init(struct amdgpu_device *adev) { - u32 tmp; int chansize, numchan; int r; @@ -727,39 +720,7 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev) else chansize = 128; - tmp = RREG32_SOC15(DF, 0, mmDF_CS_AON0_DramBaseAddress0); - tmp &= DF_CS_AON0_DramBaseAddress0__IntLvNumChan_MASK; - tmp >>= DF_CS_AON0_DramBaseAddress0__IntLvNumChan__SHIFT; - switch (tmp) { - case 0: - default: - numchan = 1; - break; - case 1: - numchan = 2; - break; - case 2: - numchan = 0; - break; - case 3: - numchan = 4; - break; - case 4: - numchan = 0; - break; - case 5: - numchan = 8; - break; - case 6: - numchan = 0; - break; - case 7: - numchan = 16; - break; - case 8: - numchan = 2; - break; - } + numchan = adev->df_funcs->get_hbm_channel_number(adev); adev->gmc.vram_width = numchan * chansize; } @@ -792,6 +753,7 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev) switch (adev->asic_type) { case CHIP_VEGA10: /* all engines support GPUVM */ case CHIP_VEGA12: /* all engines support GPUVM */ + case CHIP_VEGA20: default: adev->gmc.gart_size = 512ULL << 20; break; @@ -826,6 +788,52 @@ static int gmc_v9_0_gart_init(struct amdgpu_device *adev) return amdgpu_gart_table_vram_alloc(adev); } +static unsigned gmc_v9_0_get_vbios_fb_size(struct amdgpu_device *adev) +{ +#if 0 + u32 d1vga_control = RREG32_SOC15(DCE, 0, mmD1VGA_CONTROL); +#endif + unsigned size; + + /* + * TODO Remove once GART corruption is resolved + * Check related code in gmc_v9_0_sw_fini + * */ + size = 9 * 1024 * 1024; + +#if 0 + if (REG_GET_FIELD(d1vga_control, D1VGA_CONTROL, D1VGA_MODE_ENABLE)) { + size = 9 * 1024 * 1024; /* reserve 8MB for vga emulator and 1 MB for FB */ + } else { + u32 viewport; + + switch (adev->asic_type) { + case CHIP_RAVEN: + viewport = RREG32_SOC15(DCE, 0, mmHUBP0_DCSURF_PRI_VIEWPORT_DIMENSION); + size = (REG_GET_FIELD(viewport, + HUBP0_DCSURF_PRI_VIEWPORT_DIMENSION, PRI_VIEWPORT_HEIGHT) * + REG_GET_FIELD(viewport, + HUBP0_DCSURF_PRI_VIEWPORT_DIMENSION, PRI_VIEWPORT_WIDTH) * + 4); + break; + case CHIP_VEGA10: + case CHIP_VEGA12: + default: + viewport = RREG32_SOC15(DCE, 0, mmSCL0_VIEWPORT_SIZE); + size = (REG_GET_FIELD(viewport, SCL0_VIEWPORT_SIZE, VIEWPORT_HEIGHT) * + REG_GET_FIELD(viewport, SCL0_VIEWPORT_SIZE, VIEWPORT_WIDTH) * + 4); + break; + } + } + /* return 0 if the pre-OS buffer uses up most of vram */ + if ((adev->gmc.real_vram_size - size) < (8 * 1024 * 1024)) + return 0; + +#endif + return size; +} + static int gmc_v9_0_sw_init(void *handle) { int r; @@ -851,6 +859,7 @@ static int gmc_v9_0_sw_init(void *handle) break; case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: /* * To fulfill 4-level page support, * vm size is 256TB (48bit), maximum size of Vega10, @@ -877,12 +886,6 @@ static int gmc_v9_0_sw_init(void *handle) */ adev->gmc.mc_mask = 0xffffffffffffULL; /* 48 bit MC */ - /* - * It needs to reserve 8M stolen memory for vega10 - * TODO: Figure out how to avoid that... - */ - adev->gmc.stolen_size = 8 * 1024 * 1024; - /* set DMA mask + need_dma32 flags. * PCIE - can handle 44-bits. * IGP - can handle 44-bits @@ -907,6 +910,8 @@ static int gmc_v9_0_sw_init(void *handle) if (r) return r; + adev->gmc.stolen_size = gmc_v9_0_get_vbios_fb_size(adev); + /* Memory manager */ r = amdgpu_bo_init(adev); if (r) @@ -950,6 +955,18 @@ static int gmc_v9_0_sw_fini(void *handle) amdgpu_gem_force_release(adev); amdgpu_vm_manager_fini(adev); gmc_v9_0_gart_fini(adev); + + /* + * TODO: + * Currently there is a bug where some memory client outside + * of the driver writes to first 8M of VRAM on S3 resume, + * this overrides GART which by default gets placed in first 8M and + * causes VM_FAULTS once GTT is accessed. + * Keep the stolen memory reservation until the while this is not solved. + * Also check code in gmc_v9_0_get_vbios_fb_size and gmc_v9_0_late_init + */ + amdgpu_bo_free_kernel(&adev->stolen_vga_memory, NULL, NULL); + amdgpu_bo_fini(adev); return 0; @@ -960,6 +977,7 @@ static void gmc_v9_0_init_golden_registers(struct amdgpu_device *adev) switch (adev->asic_type) { case CHIP_VEGA10: + case CHIP_VEGA20: soc15_program_register_sequence(adev, golden_settings_mmhub_1_0_0, ARRAY_SIZE(golden_settings_mmhub_1_0_0)); diff --git a/drivers/gpu/drm/amd/amdgpu/kv_dpm.c b/drivers/gpu/drm/amd/amdgpu/kv_dpm.c index 26ba984ab2b7..7a1e77c93bf1 100644 --- a/drivers/gpu/drm/amd/amdgpu/kv_dpm.c +++ b/drivers/gpu/drm/amd/amdgpu/kv_dpm.c @@ -2727,8 +2727,9 @@ static int kv_parse_power_table(struct amdgpu_device *adev) (mode_info->atom_context->bios + data_offset + le16_to_cpu(power_info->pplib.usNonClockInfoArrayOffset)); - adev->pm.dpm.ps = kzalloc(sizeof(struct amdgpu_ps) * - state_array->ucNumEntries, GFP_KERNEL); + adev->pm.dpm.ps = kcalloc(state_array->ucNumEntries, + sizeof(struct amdgpu_ps), + GFP_KERNEL); if (!adev->pm.dpm.ps) return -ENOMEM; power_state_offset = (u8 *)state_array->states; @@ -2817,7 +2818,7 @@ static int kv_dpm_init(struct amdgpu_device *adev) pi->caps_tcp_ramping = true; } - if (amdgpu_pp_feature_mask & SCLK_DEEP_SLEEP_MASK) + if (adev->powerplay.pp_feature & PP_SCLK_DEEP_SLEEP_MASK) pi->caps_sclk_ds = true; else pi->caps_sclk_ds = false; @@ -2974,7 +2975,7 @@ static int kv_dpm_late_init(void *handle) /* powerdown unused blocks for now */ struct amdgpu_device *adev = (struct amdgpu_device *)handle; - if (!amdgpu_dpm) + if (!adev->pm.dpm_enabled) return 0; kv_dpm_powergate_acp(adev, true); diff --git a/drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.c b/drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.c index 43f925773b57..3d53c4413f13 100644 --- a/drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.c +++ b/drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.c @@ -734,6 +734,7 @@ int mmhub_v1_0_set_clockgating(struct amdgpu_device *adev, switch (adev->asic_type) { case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: case CHIP_RAVEN: mmhub_v1_0_update_medium_grain_clock_gating(adev, state == AMD_CG_STATE_GATE ? true : false); diff --git a/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c b/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c index 493348672475..078f70faedcb 100644 --- a/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +++ b/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c @@ -260,8 +260,10 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work) } while (timeout > 1); flr_done: - if (locked) + if (locked) { + adev->in_gpu_reset = 0; mutex_unlock(&adev->lock_reset); + } /* Trigger recovery for world switch failure if no TDR */ if (amdgpu_lockup_timeout == 0) diff --git a/drivers/gpu/drm/amd/amdgpu/nbio_v7_0.c b/drivers/gpu/drm/amd/amdgpu/nbio_v7_0.c index df34dc79d444..365517c0121e 100644 --- a/drivers/gpu/drm/amd/amdgpu/nbio_v7_0.c +++ b/drivers/gpu/drm/amd/amdgpu/nbio_v7_0.c @@ -34,10 +34,19 @@ #define smnCPM_CONTROL 0x11180460 #define smnPCIE_CNTL2 0x11180070 +/* vega20 */ +#define mmRCC_DEV0_EPF0_STRAP0_VG20 0x0011 +#define mmRCC_DEV0_EPF0_STRAP0_VG20_BASE_IDX 2 + static u32 nbio_v7_0_get_rev_id(struct amdgpu_device *adev) { u32 tmp = RREG32_SOC15(NBIO, 0, mmRCC_DEV0_EPF0_STRAP0); + if (adev->asic_type == CHIP_VEGA20) + tmp = RREG32_SOC15(NBIO, 0, mmRCC_DEV0_EPF0_STRAP0_VG20); + else + tmp = RREG32_SOC15(NBIO, 0, mmRCC_DEV0_EPF0_STRAP0); + tmp &= RCC_DEV0_EPF0_STRAP0__STRAP_ATI_REV_ID_DEV0_F0_MASK; tmp >>= RCC_DEV0_EPF0_STRAP0__STRAP_ATI_REV_ID_DEV0_F0__SHIFT; @@ -75,10 +84,14 @@ static void nbio_v7_0_sdma_doorbell_range(struct amdgpu_device *adev, int instan SOC15_REG_OFFSET(NBIO, 0, mmBIF_SDMA1_DOORBELL_RANGE); u32 doorbell_range = RREG32(reg); + u32 range = 2; + + if (adev->asic_type == CHIP_VEGA20) + range = 8; if (use_doorbell) { doorbell_range = REG_SET_FIELD(doorbell_range, BIF_SDMA0_DOORBELL_RANGE, OFFSET, doorbell_index); - doorbell_range = REG_SET_FIELD(doorbell_range, BIF_SDMA0_DOORBELL_RANGE, SIZE, 2); + doorbell_range = REG_SET_FIELD(doorbell_range, BIF_SDMA0_DOORBELL_RANGE, SIZE, range); } else doorbell_range = REG_SET_FIELD(doorbell_range, BIF_SDMA0_DOORBELL_RANGE, SIZE, 0); @@ -133,6 +146,9 @@ static void nbio_v7_0_update_medium_grain_clock_gating(struct amdgpu_device *ade { uint32_t def, data; + if (adev->asic_type == CHIP_VEGA20) + return; + /* NBIF_MGCG_CTRL_LCLK */ def = data = RREG32_PCIE(smnNBIF_MGCG_CTRL_LCLK); diff --git a/drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h b/drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h index 8da6da90b1c9..0cf48d26c676 100644 --- a/drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h +++ b/drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h @@ -40,11 +40,20 @@ enum psp_gfx_crtl_cmd_id GFX_CTRL_CMD_ID_INIT_GPCOM_RING = 0x00020000, /* initialize GPCOM ring */ GFX_CTRL_CMD_ID_DESTROY_RINGS = 0x00030000, /* destroy rings */ GFX_CTRL_CMD_ID_CAN_INIT_RINGS = 0x00040000, /* is it allowed to initialized the rings */ + GFX_CTRL_CMD_ID_ENABLE_INT = 0x00050000, /* enable PSP-to-Gfx interrupt */ + GFX_CTRL_CMD_ID_DISABLE_INT = 0x00060000, /* disable PSP-to-Gfx interrupt */ + GFX_CTRL_CMD_ID_MODE1_RST = 0x00070000, /* trigger the Mode 1 reset */ GFX_CTRL_CMD_ID_MAX = 0x000F0000, /* max command ID */ }; +/*----------------------------------------------------------------------------- + NOTE: All physical addresses used in this interface are actually + GPU Virtual Addresses. +*/ + + /* Control registers of the TEE Gfx interface. These are located in * SRBM-to-PSP mailbox registers (total 8 registers). */ @@ -55,8 +64,8 @@ struct psp_gfx_ctrl volatile uint32_t rbi_rptr; /* +8 Read pointer (index) of RBI ring */ volatile uint32_t gpcom_wptr; /* +12 Write pointer (index) of GPCOM ring */ volatile uint32_t gpcom_rptr; /* +16 Read pointer (index) of GPCOM ring */ - volatile uint32_t ring_addr_lo; /* +20 bits [31:0] of physical address of ring buffer */ - volatile uint32_t ring_addr_hi; /* +24 bits [63:32] of physical address of ring buffer */ + volatile uint32_t ring_addr_lo; /* +20 bits [31:0] of GPU Virtual of ring buffer (VMID=0)*/ + volatile uint32_t ring_addr_hi; /* +24 bits [63:32] of GPU Virtual of ring buffer (VMID=0) */ volatile uint32_t ring_buf_size; /* +28 Ring buffer size (in bytes) */ }; @@ -78,6 +87,8 @@ enum psp_gfx_cmd_id GFX_CMD_ID_LOAD_ASD = 0x00000004, /* load ASD Driver */ GFX_CMD_ID_SETUP_TMR = 0x00000005, /* setup TMR region */ GFX_CMD_ID_LOAD_IP_FW = 0x00000006, /* load HW IP FW */ + GFX_CMD_ID_DESTROY_TMR = 0x00000007, /* destroy TMR region */ + GFX_CMD_ID_SAVE_RESTORE = 0x00000008, /* save/restore HW IP FW */ }; @@ -85,11 +96,11 @@ enum psp_gfx_cmd_id /* Command to load Trusted Application binary into PSP OS. */ struct psp_gfx_cmd_load_ta { - uint32_t app_phy_addr_lo; /* bits [31:0] of the physical address of the TA binary (must be 4 KB aligned) */ - uint32_t app_phy_addr_hi; /* bits [63:32] of the physical address of the TA binary */ + uint32_t app_phy_addr_lo; /* bits [31:0] of the GPU Virtual address of the TA binary (must be 4 KB aligned) */ + uint32_t app_phy_addr_hi; /* bits [63:32] of the GPU Virtual address of the TA binary */ uint32_t app_len; /* length of the TA binary in bytes */ - uint32_t cmd_buf_phy_addr_lo; /* bits [31:0] of the physical address of CMD buffer (must be 4 KB aligned) */ - uint32_t cmd_buf_phy_addr_hi; /* bits [63:32] of the physical address of CMD buffer */ + uint32_t cmd_buf_phy_addr_lo; /* bits [31:0] of the GPU Virtual address of CMD buffer (must be 4 KB aligned) */ + uint32_t cmd_buf_phy_addr_hi; /* bits [63:32] of the GPU Virtual address of CMD buffer */ uint32_t cmd_buf_len; /* length of the CMD buffer in bytes; must be multiple of 4 KB */ /* Note: CmdBufLen can be set to 0. In this case no persistent CMD buffer is provided @@ -111,8 +122,8 @@ struct psp_gfx_cmd_unload_ta */ struct psp_gfx_buf_desc { - uint32_t buf_phy_addr_lo; /* bits [31:0] of physical address of the buffer (must be 4 KB aligned) */ - uint32_t buf_phy_addr_hi; /* bits [63:32] of physical address of the buffer */ + uint32_t buf_phy_addr_lo; /* bits [31:0] of GPU Virtual address of the buffer (must be 4 KB aligned) */ + uint32_t buf_phy_addr_hi; /* bits [63:32] of GPU Virtual address of the buffer */ uint32_t buf_size; /* buffer size in bytes (must be multiple of 4 KB and no bigger than 64 MB) */ }; @@ -145,8 +156,8 @@ struct psp_gfx_cmd_invoke_cmd /* Command to setup TMR region. */ struct psp_gfx_cmd_setup_tmr { - uint32_t buf_phy_addr_lo; /* bits [31:0] of physical address of TMR buffer (must be 4 KB aligned) */ - uint32_t buf_phy_addr_hi; /* bits [63:32] of physical address of TMR buffer */ + uint32_t buf_phy_addr_lo; /* bits [31:0] of GPU Virtual address of TMR buffer (must be 4 KB aligned) */ + uint32_t buf_phy_addr_hi; /* bits [63:32] of GPU Virtual address of TMR buffer */ uint32_t buf_size; /* buffer size in bytes (must be multiple of 4 KB) */ }; @@ -174,18 +185,32 @@ enum psp_gfx_fw_type GFX_FW_TYPE_ISP = 16, GFX_FW_TYPE_ACP = 17, GFX_FW_TYPE_SMU = 18, + GFX_FW_TYPE_MMSCH = 19, + GFX_FW_TYPE_RLC_RESTORE_LIST_GPM_MEM = 20, + GFX_FW_TYPE_RLC_RESTORE_LIST_SRM_MEM = 21, + GFX_FW_TYPE_RLC_RESTORE_LIST_CNTL = 22, + GFX_FW_TYPE_MAX = 23 }; /* Command to load HW IP FW. */ struct psp_gfx_cmd_load_ip_fw { - uint32_t fw_phy_addr_lo; /* bits [31:0] of physical address of FW location (must be 4 KB aligned) */ - uint32_t fw_phy_addr_hi; /* bits [63:32] of physical address of FW location */ + uint32_t fw_phy_addr_lo; /* bits [31:0] of GPU Virtual address of FW location (must be 4 KB aligned) */ + uint32_t fw_phy_addr_hi; /* bits [63:32] of GPU Virtual address of FW location */ uint32_t fw_size; /* FW buffer size in bytes */ enum psp_gfx_fw_type fw_type; /* FW type */ }; +/* Command to save/restore HW IP FW. */ +struct psp_gfx_cmd_save_restore_ip_fw +{ + uint32_t save_fw; /* if set, command is used for saving fw otherwise for resetoring*/ + uint32_t save_restore_addr_lo; /* bits [31:0] of FB address of GART memory used as save/restore buffer (must be 4 KB aligned) */ + uint32_t save_restore_addr_hi; /* bits [63:32] of FB address of GART memory used as save/restore buffer */ + uint32_t buf_size; /* Size of the save/restore buffer in bytes */ + enum psp_gfx_fw_type fw_type; /* FW type */ +}; /* All GFX ring buffer commands. */ union psp_gfx_commands @@ -195,7 +220,7 @@ union psp_gfx_commands struct psp_gfx_cmd_invoke_cmd cmd_invoke_cmd; struct psp_gfx_cmd_setup_tmr cmd_setup_tmr; struct psp_gfx_cmd_load_ip_fw cmd_load_ip_fw; - + struct psp_gfx_cmd_save_restore_ip_fw cmd_save_restore_ip_fw; }; @@ -226,8 +251,8 @@ struct psp_gfx_cmd_resp /* These fields are used for RBI only. They are all 0 in GPCOM commands */ - uint32_t resp_buf_addr_lo; /* +12 bits [31:0] of physical address of response buffer (must be 4 KB aligned) */ - uint32_t resp_buf_addr_hi; /* +16 bits [63:32] of physical address of response buffer */ + uint32_t resp_buf_addr_lo; /* +12 bits [31:0] of GPU Virtual address of response buffer (must be 4 KB aligned) */ + uint32_t resp_buf_addr_hi; /* +16 bits [63:32] of GPU Virtual address of response buffer */ uint32_t resp_offset; /* +20 offset within response buffer */ uint32_t resp_buf_size; /* +24 total size of the response buffer in bytes */ @@ -251,19 +276,19 @@ struct psp_gfx_cmd_resp /* Structure of the Ring Buffer Frame */ struct psp_gfx_rb_frame { - uint32_t cmd_buf_addr_lo; /* +0 bits [31:0] of physical address of command buffer (must be 4 KB aligned) */ - uint32_t cmd_buf_addr_hi; /* +4 bits [63:32] of physical address of command buffer */ + uint32_t cmd_buf_addr_lo; /* +0 bits [31:0] of GPU Virtual address of command buffer (must be 4 KB aligned) */ + uint32_t cmd_buf_addr_hi; /* +4 bits [63:32] of GPU Virtual address of command buffer */ uint32_t cmd_buf_size; /* +8 command buffer size in bytes */ - uint32_t fence_addr_lo; /* +12 bits [31:0] of physical address of Fence for this frame */ - uint32_t fence_addr_hi; /* +16 bits [63:32] of physical address of Fence for this frame */ + uint32_t fence_addr_lo; /* +12 bits [31:0] of GPU Virtual address of Fence for this frame */ + uint32_t fence_addr_hi; /* +16 bits [63:32] of GPU Virtual address of Fence for this frame */ uint32_t fence_value; /* +20 Fence value */ uint32_t sid_lo; /* +24 bits [31:0] of SID value (used only for RBI frames) */ uint32_t sid_hi; /* +28 bits [63:32] of SID value (used only for RBI frames) */ uint8_t vmid; /* +32 VMID value used for mapping of all addresses for this frame */ uint8_t frame_type; /* +33 1: destory context frame, 0: all other frames; used only for RBI frames */ uint8_t reserved1[2]; /* +34 reserved, must be 0 */ - uint32_t reserved2[7]; /* +40 reserved, must be 0 */ - /* total 64 bytes */ + uint32_t reserved2[7]; /* +36 reserved, must be 0 */ + /* total 64 bytes */ }; #endif /* _PSP_TEE_GFX_IF_H_ */ diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v10_0.c b/drivers/gpu/drm/amd/amdgpu/psp_v10_0.c index 8873d833a7f7..0ff136d02d9b 100644 --- a/drivers/gpu/drm/amd/amdgpu/psp_v10_0.c +++ b/drivers/gpu/drm/amd/amdgpu/psp_v10_0.c @@ -70,6 +70,15 @@ psp_v10_0_get_fw_type(struct amdgpu_firmware_info *ucode, enum psp_gfx_fw_type * case AMDGPU_UCODE_ID_RLC_G: *type = GFX_FW_TYPE_RLC_G; break; + case AMDGPU_UCODE_ID_RLC_RESTORE_LIST_CNTL: + *type = GFX_FW_TYPE_RLC_RESTORE_LIST_CNTL; + break; + case AMDGPU_UCODE_ID_RLC_RESTORE_LIST_GPM_MEM: + *type = GFX_FW_TYPE_RLC_RESTORE_LIST_GPM_MEM; + break; + case AMDGPU_UCODE_ID_RLC_RESTORE_LIST_SRM_MEM: + *type = GFX_FW_TYPE_RLC_RESTORE_LIST_SRM_MEM; + break; case AMDGPU_UCODE_ID_SMC: *type = GFX_FW_TYPE_SMU; break; diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c b/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c index 196e75def1f2..727071fee6f6 100644 --- a/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c +++ b/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c @@ -41,9 +41,14 @@ MODULE_FIRMWARE("amdgpu/vega10_sos.bin"); MODULE_FIRMWARE("amdgpu/vega10_asd.bin"); MODULE_FIRMWARE("amdgpu/vega12_sos.bin"); MODULE_FIRMWARE("amdgpu/vega12_asd.bin"); +MODULE_FIRMWARE("amdgpu/vega20_sos.bin"); +MODULE_FIRMWARE("amdgpu/vega20_asd.bin"); + #define smnMP1_FIRMWARE_FLAGS 0x3010028 +static uint32_t sos_old_versions[] = {1517616, 1510592, 1448594, 1446554}; + static int psp_v3_1_get_fw_type(struct amdgpu_firmware_info *ucode, enum psp_gfx_fw_type *type) { @@ -207,12 +212,31 @@ static int psp_v3_1_bootloader_load_sysdrv(struct psp_context *psp) return ret; } +static bool psp_v3_1_match_version(struct amdgpu_device *adev, uint32_t ver) +{ + int i; + + if (ver == adev->psp.sos_fw_version) + return true; + + /* + * Double check if the latest four legacy versions. + * If yes, it is still the right version. + */ + for (i = 0; i < sizeof(sos_old_versions) / sizeof(uint32_t); i++) { + if (sos_old_versions[i] == adev->psp.sos_fw_version) + return true; + } + + return false; +} + static int psp_v3_1_bootloader_load_sos(struct psp_context *psp) { int ret; unsigned int psp_gfxdrv_command_reg = 0; struct amdgpu_device *adev = psp->adev; - uint32_t sol_reg; + uint32_t sol_reg, ver; /* Check sOS sign of life register to confirm sys driver and sOS * are already been loaded. @@ -245,6 +269,10 @@ static int psp_v3_1_bootloader_load_sos(struct psp_context *psp) RREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_81), 0, true); + ver = RREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_58); + if (!psp_v3_1_match_version(adev, ver)) + DRM_WARN("SOS version doesn't match\n"); + return ret; } diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c index be20a387d961..aa9ab299fd32 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c @@ -62,6 +62,8 @@ MODULE_FIRMWARE("amdgpu/polaris11_sdma.bin"); MODULE_FIRMWARE("amdgpu/polaris11_sdma1.bin"); MODULE_FIRMWARE("amdgpu/polaris12_sdma.bin"); MODULE_FIRMWARE("amdgpu/polaris12_sdma1.bin"); +MODULE_FIRMWARE("amdgpu/vegam_sdma.bin"); +MODULE_FIRMWARE("amdgpu/vegam_sdma1.bin"); static const u32 sdma_offsets[SDMA_MAX_INSTANCE] = @@ -209,6 +211,7 @@ static void sdma_v3_0_init_golden_registers(struct amdgpu_device *adev) break; case CHIP_POLARIS11: case CHIP_POLARIS12: + case CHIP_VEGAM: amdgpu_device_program_register_sequence(adev, golden_settings_polaris11_a11, ARRAY_SIZE(golden_settings_polaris11_a11)); @@ -275,15 +278,18 @@ static int sdma_v3_0_init_microcode(struct amdgpu_device *adev) case CHIP_FIJI: chip_name = "fiji"; break; - case CHIP_POLARIS11: - chip_name = "polaris11"; - break; case CHIP_POLARIS10: chip_name = "polaris10"; break; + case CHIP_POLARIS11: + chip_name = "polaris11"; + break; case CHIP_POLARIS12: chip_name = "polaris12"; break; + case CHIP_VEGAM: + chip_name = "vegam"; + break; case CHIP_CARRIZO: chip_name = "carrizo"; break; diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c index 399f876f9cad..ca53b3fba422 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c @@ -42,6 +42,8 @@ MODULE_FIRMWARE("amdgpu/vega10_sdma.bin"); MODULE_FIRMWARE("amdgpu/vega10_sdma1.bin"); MODULE_FIRMWARE("amdgpu/vega12_sdma.bin"); MODULE_FIRMWARE("amdgpu/vega12_sdma1.bin"); +MODULE_FIRMWARE("amdgpu/vega20_sdma.bin"); +MODULE_FIRMWARE("amdgpu/vega20_sdma1.bin"); MODULE_FIRMWARE("amdgpu/raven_sdma.bin"); #define SDMA0_POWER_CNTL__ON_OFF_CONDITION_HOLD_TIME_MASK 0x000000F8L @@ -107,6 +109,28 @@ static const struct soc15_reg_golden golden_settings_sdma_4_1[] = SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_UTCL1_PAGE, 0x000003ff, 0x000003c0) }; +static const struct soc15_reg_golden golden_settings_sdma_4_2[] = +{ + SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_CHICKEN_BITS, 0xfe931f07, 0x02831d07), + SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_CLK_CTRL, 0xffffffff, 0x3f000100), + SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_GB_ADDR_CONFIG, 0x0000773f, 0x00004002), + SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_GB_ADDR_CONFIG_READ, 0x0000773f, 0x00004002), + SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_GFX_RB_WPTR_POLL_CNTL, 0xfffffff7, 0x00403000), + SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_PAGE_RB_WPTR_POLL_CNTL, 0xfffffff7, 0x00403000), + SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_RLC0_RB_WPTR_POLL_CNTL, 0xfffffff0, 0x00403000), + SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_RLC1_RB_WPTR_POLL_CNTL, 0xfffffff7, 0x00403000), + SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_UTCL1_PAGE, 0x000003ff, 0x000003c0), + SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_CHICKEN_BITS, 0xfe931f07, 0x02831d07), + SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_CLK_CTRL, 0xffffffff, 0x3f000100), + SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_GB_ADDR_CONFIG, 0x0000773f, 0x00004002), + SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_GB_ADDR_CONFIG_READ, 0x0000773f, 0x00004002), + SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_GFX_RB_WPTR_POLL_CNTL, 0xfffffff7, 0x00403000), + SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_PAGE_RB_WPTR_POLL_CNTL, 0xfffffff7, 0x00403000), + SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_RLC0_RB_WPTR_POLL_CNTL, 0xfffffff7, 0x00403000), + SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_RLC1_RB_WPTR_POLL_CNTL, 0xfffffff7, 0x00403000), + SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_UTCL1_PAGE, 0x000003ff, 0x000003c0) +}; + static const struct soc15_reg_golden golden_settings_sdma_rv1[] = { SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_GB_ADDR_CONFIG, 0x0018773f, 0x00000002), @@ -139,6 +163,11 @@ static void sdma_v4_0_init_golden_registers(struct amdgpu_device *adev) golden_settings_sdma_vg12, ARRAY_SIZE(golden_settings_sdma_vg12)); break; + case CHIP_VEGA20: + soc15_program_register_sequence(adev, + golden_settings_sdma_4_2, + ARRAY_SIZE(golden_settings_sdma_4_2)); + break; case CHIP_RAVEN: soc15_program_register_sequence(adev, golden_settings_sdma_4_1, @@ -182,6 +211,9 @@ static int sdma_v4_0_init_microcode(struct amdgpu_device *adev) case CHIP_VEGA12: chip_name = "vega12"; break; + case CHIP_VEGA20: + chip_name = "vega20"; + break; case CHIP_RAVEN: chip_name = "raven"; break; @@ -360,6 +392,31 @@ static void sdma_v4_0_ring_emit_ib(struct amdgpu_ring *ring, } +static void sdma_v4_0_wait_reg_mem(struct amdgpu_ring *ring, + int mem_space, int hdp, + uint32_t addr0, uint32_t addr1, + uint32_t ref, uint32_t mask, + uint32_t inv) +{ + amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_POLL_REGMEM) | + SDMA_PKT_POLL_REGMEM_HEADER_HDP_FLUSH(hdp) | + SDMA_PKT_POLL_REGMEM_HEADER_MEM_POLL(mem_space) | + SDMA_PKT_POLL_REGMEM_HEADER_FUNC(3)); /* == */ + if (mem_space) { + /* memory */ + amdgpu_ring_write(ring, addr0); + amdgpu_ring_write(ring, addr1); + } else { + /* registers */ + amdgpu_ring_write(ring, addr0 << 2); + amdgpu_ring_write(ring, addr1 << 2); + } + amdgpu_ring_write(ring, ref); /* reference */ + amdgpu_ring_write(ring, mask); /* mask */ + amdgpu_ring_write(ring, SDMA_PKT_POLL_REGMEM_DW5_RETRY_COUNT(0xfff) | + SDMA_PKT_POLL_REGMEM_DW5_INTERVAL(inv)); /* retry count, poll interval */ +} + /** * sdma_v4_0_ring_emit_hdp_flush - emit an hdp flush on the DMA ring * @@ -378,15 +435,10 @@ static void sdma_v4_0_ring_emit_hdp_flush(struct amdgpu_ring *ring) else ref_and_mask = nbio_hf_reg->ref_and_mask_sdma1; - amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_POLL_REGMEM) | - SDMA_PKT_POLL_REGMEM_HEADER_HDP_FLUSH(1) | - SDMA_PKT_POLL_REGMEM_HEADER_FUNC(3)); /* == */ - amdgpu_ring_write(ring, (adev->nbio_funcs->get_hdp_flush_done_offset(adev)) << 2); - amdgpu_ring_write(ring, (adev->nbio_funcs->get_hdp_flush_req_offset(adev)) << 2); - amdgpu_ring_write(ring, ref_and_mask); /* reference */ - amdgpu_ring_write(ring, ref_and_mask); /* mask */ - amdgpu_ring_write(ring, SDMA_PKT_POLL_REGMEM_DW5_RETRY_COUNT(0xfff) | - SDMA_PKT_POLL_REGMEM_DW5_INTERVAL(10)); /* retry count, poll interval */ + sdma_v4_0_wait_reg_mem(ring, 0, 1, + adev->nbio_funcs->get_hdp_flush_done_offset(adev), + adev->nbio_funcs->get_hdp_flush_req_offset(adev), + ref_and_mask, ref_and_mask, 10); } /** @@ -1114,16 +1166,10 @@ static void sdma_v4_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring) uint64_t addr = ring->fence_drv.gpu_addr; /* wait for idle */ - amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_POLL_REGMEM) | - SDMA_PKT_POLL_REGMEM_HEADER_HDP_FLUSH(0) | - SDMA_PKT_POLL_REGMEM_HEADER_FUNC(3) | /* equal */ - SDMA_PKT_POLL_REGMEM_HEADER_MEM_POLL(1)); - amdgpu_ring_write(ring, addr & 0xfffffffc); - amdgpu_ring_write(ring, upper_32_bits(addr) & 0xffffffff); - amdgpu_ring_write(ring, seq); /* reference */ - amdgpu_ring_write(ring, 0xffffffff); /* mask */ - amdgpu_ring_write(ring, SDMA_PKT_POLL_REGMEM_DW5_RETRY_COUNT(0xfff) | - SDMA_PKT_POLL_REGMEM_DW5_INTERVAL(4)); /* retry count, poll interval */ + sdma_v4_0_wait_reg_mem(ring, 1, 0, + addr & 0xfffffffc, + upper_32_bits(addr) & 0xffffffff, + seq, 0xffffffff, 4); } @@ -1154,15 +1200,7 @@ static void sdma_v4_0_ring_emit_wreg(struct amdgpu_ring *ring, static void sdma_v4_0_ring_emit_reg_wait(struct amdgpu_ring *ring, uint32_t reg, uint32_t val, uint32_t mask) { - amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_POLL_REGMEM) | - SDMA_PKT_POLL_REGMEM_HEADER_HDP_FLUSH(0) | - SDMA_PKT_POLL_REGMEM_HEADER_FUNC(3)); /* equal */ - amdgpu_ring_write(ring, reg << 2); - amdgpu_ring_write(ring, 0); - amdgpu_ring_write(ring, val); /* reference */ - amdgpu_ring_write(ring, mask); /* mask */ - amdgpu_ring_write(ring, SDMA_PKT_POLL_REGMEM_DW5_RETRY_COUNT(0xfff) | - SDMA_PKT_POLL_REGMEM_DW5_INTERVAL(10)); + sdma_v4_0_wait_reg_mem(ring, 0, 0, reg, 0, val, mask, 10); } static int sdma_v4_0_early_init(void *handle) @@ -1510,6 +1548,7 @@ static int sdma_v4_0_set_clockgating_state(void *handle, switch (adev->asic_type) { case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: case CHIP_RAVEN: sdma_v4_0_update_medium_grain_clock_gating(adev, state == AMD_CG_STATE_GATE ? true : false); @@ -1605,6 +1644,7 @@ static const struct amdgpu_ring_funcs sdma_v4_0_ring_funcs = { .pad_ib = sdma_v4_0_ring_pad_ib, .emit_wreg = sdma_v4_0_ring_emit_wreg, .emit_reg_wait = sdma_v4_0_ring_emit_reg_wait, + .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, }; static void sdma_v4_0_set_ring_funcs(struct amdgpu_device *adev) diff --git a/drivers/gpu/drm/amd/amdgpu/si.c b/drivers/gpu/drm/amd/amdgpu/si.c index a675ec6d2811..c364ef94cc36 100644 --- a/drivers/gpu/drm/amd/amdgpu/si.c +++ b/drivers/gpu/drm/amd/amdgpu/si.c @@ -1252,6 +1252,12 @@ static void si_invalidate_hdp(struct amdgpu_device *adev, } } +static bool si_need_full_reset(struct amdgpu_device *adev) +{ + /* change this when we support soft reset */ + return true; +} + static int si_get_pcie_lanes(struct amdgpu_device *adev) { u32 link_width_cntl; @@ -1332,6 +1338,7 @@ static const struct amdgpu_asic_funcs si_asic_funcs = .get_config_memsize = &si_get_config_memsize, .flush_hdp = &si_flush_hdp, .invalidate_hdp = &si_invalidate_hdp, + .need_full_reset = &si_need_full_reset, }; static uint32_t si_get_rev_id(struct amdgpu_device *adev) diff --git a/drivers/gpu/drm/amd/amdgpu/si_dpm.c b/drivers/gpu/drm/amd/amdgpu/si_dpm.c index 797d505bf9ee..5c97a3671726 100644 --- a/drivers/gpu/drm/amd/amdgpu/si_dpm.c +++ b/drivers/gpu/drm/amd/amdgpu/si_dpm.c @@ -7242,8 +7242,9 @@ static int si_parse_power_table(struct amdgpu_device *adev) (mode_info->atom_context->bios + data_offset + le16_to_cpu(power_info->pplib.usNonClockInfoArrayOffset)); - adev->pm.dpm.ps = kzalloc(sizeof(struct amdgpu_ps) * - state_array->ucNumEntries, GFP_KERNEL); + adev->pm.dpm.ps = kcalloc(state_array->ucNumEntries, + sizeof(struct amdgpu_ps), + GFP_KERNEL); if (!adev->pm.dpm.ps) return -ENOMEM; power_state_offset = (u8 *)state_array->states; @@ -7346,7 +7347,9 @@ static int si_dpm_init(struct amdgpu_device *adev) return ret; adev->pm.dpm.dyn_state.vddc_dependency_on_dispclk.entries = - kzalloc(4 * sizeof(struct amdgpu_clock_voltage_dependency_entry), GFP_KERNEL); + kcalloc(4, + sizeof(struct amdgpu_clock_voltage_dependency_entry), + GFP_KERNEL); if (!adev->pm.dpm.dyn_state.vddc_dependency_on_dispclk.entries) { amdgpu_free_extended_power_table(adev); return -ENOMEM; @@ -7580,7 +7583,7 @@ static int si_dpm_late_init(void *handle) int ret; struct amdgpu_device *adev = (struct amdgpu_device *)handle; - if (!amdgpu_dpm) + if (!adev->pm.dpm_enabled) return 0; ret = si_set_temperature_range(adev); diff --git a/drivers/gpu/drm/amd/amdgpu/soc15.c b/drivers/gpu/drm/amd/amdgpu/soc15.c index 51cf8a30f6c2..83f2717fcf81 100644 --- a/drivers/gpu/drm/amd/amdgpu/soc15.c +++ b/drivers/gpu/drm/amd/amdgpu/soc15.c @@ -41,8 +41,6 @@ #include "sdma1/sdma1_4_0_offset.h" #include "hdp/hdp_4_0_offset.h" #include "hdp/hdp_4_0_sh_mask.h" -#include "mp/mp_9_0_offset.h" -#include "mp/mp_9_0_sh_mask.h" #include "smuio/smuio_9_0_offset.h" #include "smuio/smuio_9_0_sh_mask.h" @@ -52,6 +50,8 @@ #include "gmc_v9_0.h" #include "gfxhub_v1_0.h" #include "mmhub_v1_0.h" +#include "df_v1_7.h" +#include "df_v3_6.h" #include "vega10_ih.h" #include "sdma_v4_0.h" #include "uvd_v7_0.h" @@ -60,33 +60,6 @@ #include "dce_virtual.h" #include "mxgpu_ai.h" -#define mmFabricConfigAccessControl 0x0410 -#define mmFabricConfigAccessControl_BASE_IDX 0 -#define mmFabricConfigAccessControl_DEFAULT 0x00000000 -//FabricConfigAccessControl -#define FabricConfigAccessControl__CfgRegInstAccEn__SHIFT 0x0 -#define FabricConfigAccessControl__CfgRegInstAccRegLock__SHIFT 0x1 -#define FabricConfigAccessControl__CfgRegInstID__SHIFT 0x10 -#define FabricConfigAccessControl__CfgRegInstAccEn_MASK 0x00000001L -#define FabricConfigAccessControl__CfgRegInstAccRegLock_MASK 0x00000002L -#define FabricConfigAccessControl__CfgRegInstID_MASK 0x00FF0000L - - -#define mmDF_PIE_AON0_DfGlobalClkGater 0x00fc -#define mmDF_PIE_AON0_DfGlobalClkGater_BASE_IDX 0 -//DF_PIE_AON0_DfGlobalClkGater -#define DF_PIE_AON0_DfGlobalClkGater__MGCGMode__SHIFT 0x0 -#define DF_PIE_AON0_DfGlobalClkGater__MGCGMode_MASK 0x0000000FL - -enum { - DF_MGCG_DISABLE = 0, - DF_MGCG_ENABLE_00_CYCLE_DELAY =1, - DF_MGCG_ENABLE_01_CYCLE_DELAY =2, - DF_MGCG_ENABLE_15_CYCLE_DELAY =13, - DF_MGCG_ENABLE_31_CYCLE_DELAY =14, - DF_MGCG_ENABLE_63_CYCLE_DELAY =15 -}; - #define mmMP0_MISC_CGTT_CTRL0 0x01b9 #define mmMP0_MISC_CGTT_CTRL0_BASE_IDX 0 #define mmMP0_MISC_LIGHT_SLEEP_CTRL 0x01ba @@ -313,6 +286,7 @@ static struct soc15_allowed_register_entry soc15_allowed_read_registers[] = { { SOC15_REG_ENTRY(GC, 0, mmCP_CPC_STALLED_STAT1)}, { SOC15_REG_ENTRY(GC, 0, mmCP_CPC_STATUS)}, { SOC15_REG_ENTRY(GC, 0, mmGB_ADDR_CONFIG)}, + { SOC15_REG_ENTRY(GC, 0, mmDB_DEBUG2)}, }; static uint32_t soc15_read_indexed_register(struct amdgpu_device *adev, u32 se_num, @@ -341,6 +315,8 @@ static uint32_t soc15_get_register_value(struct amdgpu_device *adev, } else { if (reg_offset == SOC15_REG_OFFSET(GC, 0, mmGB_ADDR_CONFIG)) return adev->gfx.config.gb_addr_config; + else if (reg_offset == SOC15_REG_OFFSET(GC, 0, mmDB_DEBUG2)) + return adev->gfx.config.db_debug2; return RREG32(reg_offset); } } @@ -512,15 +488,24 @@ int soc15_set_ip_blocks(struct amdgpu_device *adev) case CHIP_RAVEN: vega10_reg_base_init(adev); break; + case CHIP_VEGA20: + vega20_reg_base_init(adev); + break; default: return -EINVAL; } if (adev->flags & AMD_IS_APU) adev->nbio_funcs = &nbio_v7_0_funcs; + else if (adev->asic_type == CHIP_VEGA20) + adev->nbio_funcs = &nbio_v7_0_funcs; else adev->nbio_funcs = &nbio_v6_1_funcs; + if (adev->asic_type == CHIP_VEGA20) + adev->df_funcs = &df_v3_6_funcs; + else + adev->df_funcs = &df_v1_7_funcs; adev->nbio_funcs->detect_hw_virt(adev); if (amdgpu_sriov_vf(adev)) @@ -529,12 +514,15 @@ int soc15_set_ip_blocks(struct amdgpu_device *adev) switch (adev->asic_type) { case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: amdgpu_device_ip_block_add(adev, &vega10_common_ip_block); amdgpu_device_ip_block_add(adev, &gmc_v9_0_ip_block); amdgpu_device_ip_block_add(adev, &vega10_ih_ip_block); - amdgpu_device_ip_block_add(adev, &psp_v3_1_ip_block); - if (!amdgpu_sriov_vf(adev)) - amdgpu_device_ip_block_add(adev, &pp_smu_ip_block); + if (adev->asic_type != CHIP_VEGA20) { + amdgpu_device_ip_block_add(adev, &psp_v3_1_ip_block); + if (!amdgpu_sriov_vf(adev)) + amdgpu_device_ip_block_add(adev, &pp_smu_ip_block); + } if (adev->enable_virtual_display || amdgpu_sriov_vf(adev)) amdgpu_device_ip_block_add(adev, &dce_virtual_ip_block); #if defined(CONFIG_DRM_AMD_DC) @@ -593,6 +581,12 @@ static void soc15_invalidate_hdp(struct amdgpu_device *adev, HDP, 0, mmHDP_READ_CACHE_INVALIDATE), 1); } +static bool soc15_need_full_reset(struct amdgpu_device *adev) +{ + /* change this when we implement soft reset */ + return true; +} + static const struct amdgpu_asic_funcs soc15_asic_funcs = { .read_disabled_bios = &soc15_read_disabled_bios, @@ -606,6 +600,7 @@ static const struct amdgpu_asic_funcs soc15_asic_funcs = .get_config_memsize = &soc15_get_config_memsize, .flush_hdp = &soc15_flush_hdp, .invalidate_hdp = &soc15_invalidate_hdp, + .need_full_reset = &soc15_need_full_reset, }; static int soc15_common_early_init(void *handle) @@ -675,6 +670,28 @@ static int soc15_common_early_init(void *handle) adev->pg_flags = 0; adev->external_rev_id = adev->rev_id + 0x14; break; + case CHIP_VEGA20: + adev->cg_flags = AMD_CG_SUPPORT_GFX_MGCG | + AMD_CG_SUPPORT_GFX_MGLS | + AMD_CG_SUPPORT_GFX_CGCG | + AMD_CG_SUPPORT_GFX_CGLS | + AMD_CG_SUPPORT_GFX_3D_CGCG | + AMD_CG_SUPPORT_GFX_3D_CGLS | + AMD_CG_SUPPORT_GFX_CP_LS | + AMD_CG_SUPPORT_MC_LS | + AMD_CG_SUPPORT_MC_MGCG | + AMD_CG_SUPPORT_SDMA_MGCG | + AMD_CG_SUPPORT_SDMA_LS | + AMD_CG_SUPPORT_BIF_MGCG | + AMD_CG_SUPPORT_BIF_LS | + AMD_CG_SUPPORT_HDP_MGCG | + AMD_CG_SUPPORT_HDP_LS | + AMD_CG_SUPPORT_ROM_MGCG | + AMD_CG_SUPPORT_VCE_MGCG | + AMD_CG_SUPPORT_UVD_MGCG; + adev->pg_flags = 0; + adev->external_rev_id = adev->rev_id + 0x28; + break; case CHIP_RAVEN: adev->cg_flags = AMD_CG_SUPPORT_GFX_MGCG | AMD_CG_SUPPORT_GFX_MGLS | @@ -694,8 +711,15 @@ static int soc15_common_early_init(void *handle) AMD_CG_SUPPORT_MC_MGCG | AMD_CG_SUPPORT_MC_LS | AMD_CG_SUPPORT_SDMA_MGCG | - AMD_CG_SUPPORT_SDMA_LS; - adev->pg_flags = AMD_PG_SUPPORT_SDMA; + AMD_CG_SUPPORT_SDMA_LS | + AMD_CG_SUPPORT_VCN_MGCG; + + adev->pg_flags = AMD_PG_SUPPORT_SDMA | AMD_PG_SUPPORT_VCN; + + if (adev->powerplay.pp_feature & PP_GFXOFF_MASK) + adev->pg_flags |= AMD_PG_SUPPORT_GFX_PG | + AMD_PG_SUPPORT_CP | + AMD_PG_SUPPORT_RLC_SMU_HS; adev->external_rev_id = 0x1; break; @@ -871,32 +895,6 @@ static void soc15_update_rom_medium_grain_clock_gating(struct amdgpu_device *ade WREG32(SOC15_REG_OFFSET(SMUIO, 0, mmCGTT_ROM_CLK_CTRL0), data); } -static void soc15_update_df_medium_grain_clock_gating(struct amdgpu_device *adev, - bool enable) -{ - uint32_t data; - - /* Put DF on broadcast mode */ - data = RREG32(SOC15_REG_OFFSET(DF, 0, mmFabricConfigAccessControl)); - data &= ~FabricConfigAccessControl__CfgRegInstAccEn_MASK; - WREG32(SOC15_REG_OFFSET(DF, 0, mmFabricConfigAccessControl), data); - - if (enable && (adev->cg_flags & AMD_CG_SUPPORT_DF_MGCG)) { - data = RREG32(SOC15_REG_OFFSET(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater)); - data &= ~DF_PIE_AON0_DfGlobalClkGater__MGCGMode_MASK; - data |= DF_MGCG_ENABLE_15_CYCLE_DELAY; - WREG32(SOC15_REG_OFFSET(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater), data); - } else { - data = RREG32(SOC15_REG_OFFSET(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater)); - data &= ~DF_PIE_AON0_DfGlobalClkGater__MGCGMode_MASK; - data |= DF_MGCG_DISABLE; - WREG32(SOC15_REG_OFFSET(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater), data); - } - - WREG32(SOC15_REG_OFFSET(DF, 0, mmFabricConfigAccessControl), - mmFabricConfigAccessControl_DEFAULT); -} - static int soc15_common_set_clockgating_state(void *handle, enum amd_clockgating_state state) { @@ -908,6 +906,7 @@ static int soc15_common_set_clockgating_state(void *handle, switch (adev->asic_type) { case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: adev->nbio_funcs->update_medium_grain_clock_gating(adev, state == AMD_CG_STATE_GATE ? true : false); adev->nbio_funcs->update_medium_grain_light_sleep(adev, @@ -920,7 +919,7 @@ static int soc15_common_set_clockgating_state(void *handle, state == AMD_CG_STATE_GATE ? true : false); soc15_update_rom_medium_grain_clock_gating(adev, state == AMD_CG_STATE_GATE ? true : false); - soc15_update_df_medium_grain_clock_gating(adev, + adev->df_funcs->update_medium_grain_clock_gating(adev, state == AMD_CG_STATE_GATE ? true : false); break; case CHIP_RAVEN: @@ -973,10 +972,7 @@ static void soc15_common_get_clockgating_state(void *handle, u32 *flags) if (!(data & CGTT_ROM_CLK_CTRL0__SOFT_OVERRIDE0_MASK)) *flags |= AMD_CG_SUPPORT_ROM_MGCG; - /* AMD_CG_SUPPORT_DF_MGCG */ - data = RREG32(SOC15_REG_OFFSET(DF, 0, mmDF_PIE_AON0_DfGlobalClkGater)); - if (data & DF_MGCG_ENABLE_15_CYCLE_DELAY) - *flags |= AMD_CG_SUPPORT_DF_MGCG; + adev->df_funcs->get_clockgating_state(adev, flags); } static int soc15_common_set_powergating_state(void *handle, diff --git a/drivers/gpu/drm/amd/amdgpu/soc15.h b/drivers/gpu/drm/amd/amdgpu/soc15.h index f70da8a29f86..1f714b7af520 100644 --- a/drivers/gpu/drm/amd/amdgpu/soc15.h +++ b/drivers/gpu/drm/amd/amdgpu/soc15.h @@ -55,5 +55,6 @@ void soc15_program_register_sequence(struct amdgpu_device *adev, const u32 array_size); int vega10_reg_base_init(struct amdgpu_device *adev); +int vega20_reg_base_init(struct amdgpu_device *adev); #endif diff --git a/drivers/gpu/drm/amd/amdgpu/soc15_common.h b/drivers/gpu/drm/amd/amdgpu/soc15_common.h index def865067edd..0942f492d2e1 100644 --- a/drivers/gpu/drm/amd/amdgpu/soc15_common.h +++ b/drivers/gpu/drm/amd/amdgpu/soc15_common.h @@ -47,6 +47,21 @@ #define WREG32_SOC15_OFFSET(ip, inst, reg, offset, value) \ WREG32((adev->reg_offset[ip##_HWIP][inst][reg##_BASE_IDX] + reg) + offset, value) +#define SOC15_WAIT_ON_RREG(ip, inst, reg, expected_value, mask, ret) \ + do { \ + uint32_t tmp_ = RREG32(adev->reg_offset[ip##_HWIP][inst][reg##_BASE_IDX] + reg); \ + uint32_t loop = adev->usec_timeout; \ + while ((tmp_ & (mask)) != (expected_value)) { \ + udelay(2); \ + tmp_ = RREG32(adev->reg_offset[ip##_HWIP][inst][reg##_BASE_IDX] + reg); \ + loop--; \ + if (!loop) { \ + ret = -ETIMEDOUT; \ + break; \ + } \ + } \ + } while (0) + #endif diff --git a/drivers/gpu/drm/amd/amdgpu/soc15d.h b/drivers/gpu/drm/amd/amdgpu/soc15d.h index 7f408f85fdb6..8dc29107228f 100644 --- a/drivers/gpu/drm/amd/amdgpu/soc15d.h +++ b/drivers/gpu/drm/amd/amdgpu/soc15d.h @@ -159,6 +159,7 @@ #define EOP_TC_WB_ACTION_EN (1 << 15) /* L2 */ #define EOP_TCL1_ACTION_EN (1 << 16) #define EOP_TC_ACTION_EN (1 << 17) /* L2 */ +#define EOP_TC_NC_ACTION_EN (1 << 19) #define EOP_TC_MD_ACTION_EN (1 << 21) /* L2 metadata */ #define DATA_SEL(x) ((x) << 29) @@ -268,6 +269,11 @@ * x=1: tmz_end */ +#define PACKET3_INVALIDATE_TLBS 0x98 +# define PACKET3_INVALIDATE_TLBS_DST_SEL(x) ((x) << 0) +# define PACKET3_INVALIDATE_TLBS_ALL_HUB(x) ((x) << 4) +# define PACKET3_INVALIDATE_TLBS_PASID(x) ((x) << 5) +# define PACKET3_INVALIDATE_TLBS_FLUSH_TYPE(x) ((x) << 29) #define PACKET3_SET_RESOURCES 0xA0 /* 1. header * 2. CONTROL diff --git a/drivers/gpu/drm/amd/amdgpu/uvd_v4_2.c b/drivers/gpu/drm/amd/amdgpu/uvd_v4_2.c index 948bb9437757..6fed3d7797a8 100644 --- a/drivers/gpu/drm/amd/amdgpu/uvd_v4_2.c +++ b/drivers/gpu/drm/amd/amdgpu/uvd_v4_2.c @@ -93,6 +93,7 @@ static void uvd_v4_2_ring_set_wptr(struct amdgpu_ring *ring) static int uvd_v4_2_early_init(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; + adev->uvd.num_uvd_inst = 1; uvd_v4_2_set_ring_funcs(adev); uvd_v4_2_set_irq_funcs(adev); @@ -107,7 +108,7 @@ static int uvd_v4_2_sw_init(void *handle) int r; /* UVD TRAP */ - r = amdgpu_irq_add_id(adev, AMDGPU_IH_CLIENTID_LEGACY, 124, &adev->uvd.irq); + r = amdgpu_irq_add_id(adev, AMDGPU_IH_CLIENTID_LEGACY, 124, &adev->uvd.inst->irq); if (r) return r; @@ -119,9 +120,9 @@ static int uvd_v4_2_sw_init(void *handle) if (r) return r; - ring = &adev->uvd.ring; + ring = &adev->uvd.inst->ring; sprintf(ring->name, "uvd"); - r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.irq, 0); + r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.inst->irq, 0); return r; } @@ -150,7 +151,7 @@ static void uvd_v4_2_enable_mgcg(struct amdgpu_device *adev, static int uvd_v4_2_hw_init(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - struct amdgpu_ring *ring = &adev->uvd.ring; + struct amdgpu_ring *ring = &adev->uvd.inst->ring; uint32_t tmp; int r; @@ -208,7 +209,7 @@ done: static int uvd_v4_2_hw_fini(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - struct amdgpu_ring *ring = &adev->uvd.ring; + struct amdgpu_ring *ring = &adev->uvd.inst->ring; if (RREG32(mmUVD_STATUS) != 0) uvd_v4_2_stop(adev); @@ -251,7 +252,7 @@ static int uvd_v4_2_resume(void *handle) */ static int uvd_v4_2_start(struct amdgpu_device *adev) { - struct amdgpu_ring *ring = &adev->uvd.ring; + struct amdgpu_ring *ring = &adev->uvd.inst->ring; uint32_t rb_bufsz; int i, j, r; u32 tmp; @@ -523,6 +524,18 @@ static void uvd_v4_2_ring_emit_ib(struct amdgpu_ring *ring, amdgpu_ring_write(ring, ib->length_dw); } +static void uvd_v4_2_ring_insert_nop(struct amdgpu_ring *ring, uint32_t count) +{ + int i; + + WARN_ON(ring->wptr % 2 || count % 2); + + for (i = 0; i < count / 2; i++) { + amdgpu_ring_write(ring, PACKET0(mmUVD_NO_OP, 0)); + amdgpu_ring_write(ring, 0); + } +} + /** * uvd_v4_2_mc_resume - memory controller programming * @@ -536,7 +549,7 @@ static void uvd_v4_2_mc_resume(struct amdgpu_device *adev) uint32_t size; /* programm the VCPU memory controller bits 0-27 */ - addr = (adev->uvd.gpu_addr + AMDGPU_UVD_FIRMWARE_OFFSET) >> 3; + addr = (adev->uvd.inst->gpu_addr + AMDGPU_UVD_FIRMWARE_OFFSET) >> 3; size = AMDGPU_UVD_FIRMWARE_SIZE(adev) >> 3; WREG32(mmUVD_VCPU_CACHE_OFFSET0, addr); WREG32(mmUVD_VCPU_CACHE_SIZE0, size); @@ -553,11 +566,11 @@ static void uvd_v4_2_mc_resume(struct amdgpu_device *adev) WREG32(mmUVD_VCPU_CACHE_SIZE2, size); /* bits 28-31 */ - addr = (adev->uvd.gpu_addr >> 28) & 0xF; + addr = (adev->uvd.inst->gpu_addr >> 28) & 0xF; WREG32(mmUVD_LMI_ADDR_EXT, (addr << 12) | (addr << 0)); /* bits 32-39 */ - addr = (adev->uvd.gpu_addr >> 32) & 0xFF; + addr = (adev->uvd.inst->gpu_addr >> 32) & 0xFF; WREG32(mmUVD_LMI_EXT40_ADDR, addr | (0x9 << 16) | (0x1 << 31)); WREG32(mmUVD_UDEC_ADDR_CONFIG, adev->gfx.config.gb_addr_config); @@ -664,7 +677,7 @@ static int uvd_v4_2_process_interrupt(struct amdgpu_device *adev, struct amdgpu_iv_entry *entry) { DRM_DEBUG("IH: UVD TRAP\n"); - amdgpu_fence_process(&adev->uvd.ring); + amdgpu_fence_process(&adev->uvd.inst->ring); return 0; } @@ -688,7 +701,7 @@ static int uvd_v4_2_set_powergating_state(void *handle, if (state == AMD_PG_STATE_GATE) { uvd_v4_2_stop(adev); - if (adev->pg_flags & AMD_PG_SUPPORT_UVD && amdgpu_dpm == 0) { + if (adev->pg_flags & AMD_PG_SUPPORT_UVD && !adev->pm.dpm_enabled) { if (!(RREG32_SMC(ixCURRENT_PG_STATUS) & CURRENT_PG_STATUS__UVD_PG_STATUS_MASK)) { WREG32(mmUVD_PGFSM_CONFIG, (UVD_PGFSM_CONFIG__UVD_PGFSM_FSM_ADDR_MASK | @@ -699,7 +712,7 @@ static int uvd_v4_2_set_powergating_state(void *handle, } return 0; } else { - if (adev->pg_flags & AMD_PG_SUPPORT_UVD && amdgpu_dpm == 0) { + if (adev->pg_flags & AMD_PG_SUPPORT_UVD && !adev->pm.dpm_enabled) { if (RREG32_SMC(ixCURRENT_PG_STATUS) & CURRENT_PG_STATUS__UVD_PG_STATUS_MASK) { WREG32(mmUVD_PGFSM_CONFIG, (UVD_PGFSM_CONFIG__UVD_PGFSM_FSM_ADDR_MASK | @@ -732,7 +745,6 @@ static const struct amd_ip_funcs uvd_v4_2_ip_funcs = { static const struct amdgpu_ring_funcs uvd_v4_2_ring_funcs = { .type = AMDGPU_RING_TYPE_UVD, .align_mask = 0xf, - .nop = PACKET0(mmUVD_NO_OP, 0), .support_64bit_ptrs = false, .get_rptr = uvd_v4_2_ring_get_rptr, .get_wptr = uvd_v4_2_ring_get_wptr, @@ -745,7 +757,7 @@ static const struct amdgpu_ring_funcs uvd_v4_2_ring_funcs = { .emit_fence = uvd_v4_2_ring_emit_fence, .test_ring = uvd_v4_2_ring_test_ring, .test_ib = amdgpu_uvd_ring_test_ib, - .insert_nop = amdgpu_ring_insert_nop, + .insert_nop = uvd_v4_2_ring_insert_nop, .pad_ib = amdgpu_ring_generic_pad_ib, .begin_use = amdgpu_uvd_ring_begin_use, .end_use = amdgpu_uvd_ring_end_use, @@ -753,7 +765,7 @@ static const struct amdgpu_ring_funcs uvd_v4_2_ring_funcs = { static void uvd_v4_2_set_ring_funcs(struct amdgpu_device *adev) { - adev->uvd.ring.funcs = &uvd_v4_2_ring_funcs; + adev->uvd.inst->ring.funcs = &uvd_v4_2_ring_funcs; } static const struct amdgpu_irq_src_funcs uvd_v4_2_irq_funcs = { @@ -763,8 +775,8 @@ static const struct amdgpu_irq_src_funcs uvd_v4_2_irq_funcs = { static void uvd_v4_2_set_irq_funcs(struct amdgpu_device *adev) { - adev->uvd.irq.num_types = 1; - adev->uvd.irq.funcs = &uvd_v4_2_irq_funcs; + adev->uvd.inst->irq.num_types = 1; + adev->uvd.inst->irq.funcs = &uvd_v4_2_irq_funcs; } const struct amdgpu_ip_block_version uvd_v4_2_ip_block = diff --git a/drivers/gpu/drm/amd/amdgpu/uvd_v5_0.c b/drivers/gpu/drm/amd/amdgpu/uvd_v5_0.c index 6445d55e7d5a..341ee6d55ce8 100644 --- a/drivers/gpu/drm/amd/amdgpu/uvd_v5_0.c +++ b/drivers/gpu/drm/amd/amdgpu/uvd_v5_0.c @@ -89,6 +89,7 @@ static void uvd_v5_0_ring_set_wptr(struct amdgpu_ring *ring) static int uvd_v5_0_early_init(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; + adev->uvd.num_uvd_inst = 1; uvd_v5_0_set_ring_funcs(adev); uvd_v5_0_set_irq_funcs(adev); @@ -103,7 +104,7 @@ static int uvd_v5_0_sw_init(void *handle) int r; /* UVD TRAP */ - r = amdgpu_irq_add_id(adev, AMDGPU_IH_CLIENTID_LEGACY, 124, &adev->uvd.irq); + r = amdgpu_irq_add_id(adev, AMDGPU_IH_CLIENTID_LEGACY, 124, &adev->uvd.inst->irq); if (r) return r; @@ -115,9 +116,9 @@ static int uvd_v5_0_sw_init(void *handle) if (r) return r; - ring = &adev->uvd.ring; + ring = &adev->uvd.inst->ring; sprintf(ring->name, "uvd"); - r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.irq, 0); + r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.inst->irq, 0); return r; } @@ -144,7 +145,7 @@ static int uvd_v5_0_sw_fini(void *handle) static int uvd_v5_0_hw_init(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - struct amdgpu_ring *ring = &adev->uvd.ring; + struct amdgpu_ring *ring = &adev->uvd.inst->ring; uint32_t tmp; int r; @@ -204,7 +205,7 @@ done: static int uvd_v5_0_hw_fini(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - struct amdgpu_ring *ring = &adev->uvd.ring; + struct amdgpu_ring *ring = &adev->uvd.inst->ring; if (RREG32(mmUVD_STATUS) != 0) uvd_v5_0_stop(adev); @@ -253,9 +254,9 @@ static void uvd_v5_0_mc_resume(struct amdgpu_device *adev) /* programm memory controller bits 0-27 */ WREG32(mmUVD_LMI_VCPU_CACHE_64BIT_BAR_LOW, - lower_32_bits(adev->uvd.gpu_addr)); + lower_32_bits(adev->uvd.inst->gpu_addr)); WREG32(mmUVD_LMI_VCPU_CACHE_64BIT_BAR_HIGH, - upper_32_bits(adev->uvd.gpu_addr)); + upper_32_bits(adev->uvd.inst->gpu_addr)); offset = AMDGPU_UVD_FIRMWARE_OFFSET; size = AMDGPU_UVD_FIRMWARE_SIZE(adev); @@ -287,7 +288,7 @@ static void uvd_v5_0_mc_resume(struct amdgpu_device *adev) */ static int uvd_v5_0_start(struct amdgpu_device *adev) { - struct amdgpu_ring *ring = &adev->uvd.ring; + struct amdgpu_ring *ring = &adev->uvd.inst->ring; uint32_t rb_bufsz, tmp; uint32_t lmi_swap_cntl; uint32_t mp_swap_cntl; @@ -540,6 +541,18 @@ static void uvd_v5_0_ring_emit_ib(struct amdgpu_ring *ring, amdgpu_ring_write(ring, ib->length_dw); } +static void uvd_v5_0_ring_insert_nop(struct amdgpu_ring *ring, uint32_t count) +{ + int i; + + WARN_ON(ring->wptr % 2 || count % 2); + + for (i = 0; i < count / 2; i++) { + amdgpu_ring_write(ring, PACKET0(mmUVD_NO_OP, 0)); + amdgpu_ring_write(ring, 0); + } +} + static bool uvd_v5_0_is_idle(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; @@ -586,7 +599,7 @@ static int uvd_v5_0_process_interrupt(struct amdgpu_device *adev, struct amdgpu_iv_entry *entry) { DRM_DEBUG("IH: UVD TRAP\n"); - amdgpu_fence_process(&adev->uvd.ring); + amdgpu_fence_process(&adev->uvd.inst->ring); return 0; } @@ -840,7 +853,6 @@ static const struct amd_ip_funcs uvd_v5_0_ip_funcs = { static const struct amdgpu_ring_funcs uvd_v5_0_ring_funcs = { .type = AMDGPU_RING_TYPE_UVD, .align_mask = 0xf, - .nop = PACKET0(mmUVD_NO_OP, 0), .support_64bit_ptrs = false, .get_rptr = uvd_v5_0_ring_get_rptr, .get_wptr = uvd_v5_0_ring_get_wptr, @@ -853,7 +865,7 @@ static const struct amdgpu_ring_funcs uvd_v5_0_ring_funcs = { .emit_fence = uvd_v5_0_ring_emit_fence, .test_ring = uvd_v5_0_ring_test_ring, .test_ib = amdgpu_uvd_ring_test_ib, - .insert_nop = amdgpu_ring_insert_nop, + .insert_nop = uvd_v5_0_ring_insert_nop, .pad_ib = amdgpu_ring_generic_pad_ib, .begin_use = amdgpu_uvd_ring_begin_use, .end_use = amdgpu_uvd_ring_end_use, @@ -861,7 +873,7 @@ static const struct amdgpu_ring_funcs uvd_v5_0_ring_funcs = { static void uvd_v5_0_set_ring_funcs(struct amdgpu_device *adev) { - adev->uvd.ring.funcs = &uvd_v5_0_ring_funcs; + adev->uvd.inst->ring.funcs = &uvd_v5_0_ring_funcs; } static const struct amdgpu_irq_src_funcs uvd_v5_0_irq_funcs = { @@ -871,8 +883,8 @@ static const struct amdgpu_irq_src_funcs uvd_v5_0_irq_funcs = { static void uvd_v5_0_set_irq_funcs(struct amdgpu_device *adev) { - adev->uvd.irq.num_types = 1; - adev->uvd.irq.funcs = &uvd_v5_0_irq_funcs; + adev->uvd.inst->irq.num_types = 1; + adev->uvd.inst->irq.funcs = &uvd_v5_0_irq_funcs; } const struct amdgpu_ip_block_version uvd_v5_0_ip_block = diff --git a/drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c b/drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c index f26f515db2fb..bfddf97dd13e 100644 --- a/drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c +++ b/drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c @@ -62,7 +62,7 @@ static void uvd_v6_0_enable_mgcg(struct amdgpu_device *adev, static inline bool uvd_v6_0_enc_support(struct amdgpu_device *adev) { return ((adev->asic_type >= CHIP_POLARIS10) && - (adev->asic_type <= CHIP_POLARIS12) && + (adev->asic_type <= CHIP_VEGAM) && (!adev->uvd.fw_version || adev->uvd.fw_version >= FW_1_130_16)); } @@ -91,7 +91,7 @@ static uint64_t uvd_v6_0_enc_ring_get_rptr(struct amdgpu_ring *ring) { struct amdgpu_device *adev = ring->adev; - if (ring == &adev->uvd.ring_enc[0]) + if (ring == &adev->uvd.inst->ring_enc[0]) return RREG32(mmUVD_RB_RPTR); else return RREG32(mmUVD_RB_RPTR2); @@ -121,7 +121,7 @@ static uint64_t uvd_v6_0_enc_ring_get_wptr(struct amdgpu_ring *ring) { struct amdgpu_device *adev = ring->adev; - if (ring == &adev->uvd.ring_enc[0]) + if (ring == &adev->uvd.inst->ring_enc[0]) return RREG32(mmUVD_RB_WPTR); else return RREG32(mmUVD_RB_WPTR2); @@ -152,7 +152,7 @@ static void uvd_v6_0_enc_ring_set_wptr(struct amdgpu_ring *ring) { struct amdgpu_device *adev = ring->adev; - if (ring == &adev->uvd.ring_enc[0]) + if (ring == &adev->uvd.inst->ring_enc[0]) WREG32(mmUVD_RB_WPTR, lower_32_bits(ring->wptr)); else @@ -375,6 +375,7 @@ error: static int uvd_v6_0_early_init(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; + adev->uvd.num_uvd_inst = 1; if (!(adev->flags & AMD_IS_APU) && (RREG32_SMC(ixCC_HARVEST_FUSES) & CC_HARVEST_FUSES__UVD_DISABLE_MASK)) @@ -399,14 +400,14 @@ static int uvd_v6_0_sw_init(void *handle) struct amdgpu_device *adev = (struct amdgpu_device *)handle; /* UVD TRAP */ - r = amdgpu_irq_add_id(adev, AMDGPU_IH_CLIENTID_LEGACY, 124, &adev->uvd.irq); + r = amdgpu_irq_add_id(adev, AMDGPU_IH_CLIENTID_LEGACY, 124, &adev->uvd.inst->irq); if (r) return r; /* UVD ENC TRAP */ if (uvd_v6_0_enc_support(adev)) { for (i = 0; i < adev->uvd.num_enc_rings; ++i) { - r = amdgpu_irq_add_id(adev, AMDGPU_IH_CLIENTID_LEGACY, i + 119, &adev->uvd.irq); + r = amdgpu_irq_add_id(adev, AMDGPU_IH_CLIENTID_LEGACY, i + 119, &adev->uvd.inst->irq); if (r) return r; } @@ -418,18 +419,18 @@ static int uvd_v6_0_sw_init(void *handle) if (!uvd_v6_0_enc_support(adev)) { for (i = 0; i < adev->uvd.num_enc_rings; ++i) - adev->uvd.ring_enc[i].funcs = NULL; + adev->uvd.inst->ring_enc[i].funcs = NULL; - adev->uvd.irq.num_types = 1; + adev->uvd.inst->irq.num_types = 1; adev->uvd.num_enc_rings = 0; DRM_INFO("UVD ENC is disabled\n"); } else { struct drm_sched_rq *rq; - ring = &adev->uvd.ring_enc[0]; + ring = &adev->uvd.inst->ring_enc[0]; rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL]; - r = drm_sched_entity_init(&ring->sched, &adev->uvd.entity_enc, - rq, amdgpu_sched_jobs, NULL); + r = drm_sched_entity_init(&ring->sched, &adev->uvd.inst->entity_enc, + rq, NULL); if (r) { DRM_ERROR("Failed setting up UVD ENC run queue.\n"); return r; @@ -440,17 +441,17 @@ static int uvd_v6_0_sw_init(void *handle) if (r) return r; - ring = &adev->uvd.ring; + ring = &adev->uvd.inst->ring; sprintf(ring->name, "uvd"); - r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.irq, 0); + r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.inst->irq, 0); if (r) return r; if (uvd_v6_0_enc_support(adev)) { for (i = 0; i < adev->uvd.num_enc_rings; ++i) { - ring = &adev->uvd.ring_enc[i]; + ring = &adev->uvd.inst->ring_enc[i]; sprintf(ring->name, "uvd_enc%d", i); - r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.irq, 0); + r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.inst->irq, 0); if (r) return r; } @@ -469,10 +470,10 @@ static int uvd_v6_0_sw_fini(void *handle) return r; if (uvd_v6_0_enc_support(adev)) { - drm_sched_entity_fini(&adev->uvd.ring_enc[0].sched, &adev->uvd.entity_enc); + drm_sched_entity_fini(&adev->uvd.inst->ring_enc[0].sched, &adev->uvd.inst->entity_enc); for (i = 0; i < adev->uvd.num_enc_rings; ++i) - amdgpu_ring_fini(&adev->uvd.ring_enc[i]); + amdgpu_ring_fini(&adev->uvd.inst->ring_enc[i]); } return amdgpu_uvd_sw_fini(adev); @@ -488,7 +489,7 @@ static int uvd_v6_0_sw_fini(void *handle) static int uvd_v6_0_hw_init(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - struct amdgpu_ring *ring = &adev->uvd.ring; + struct amdgpu_ring *ring = &adev->uvd.inst->ring; uint32_t tmp; int i, r; @@ -532,7 +533,7 @@ static int uvd_v6_0_hw_init(void *handle) if (uvd_v6_0_enc_support(adev)) { for (i = 0; i < adev->uvd.num_enc_rings; ++i) { - ring = &adev->uvd.ring_enc[i]; + ring = &adev->uvd.inst->ring_enc[i]; ring->ready = true; r = amdgpu_ring_test_ring(ring); if (r) { @@ -563,7 +564,7 @@ done: static int uvd_v6_0_hw_fini(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - struct amdgpu_ring *ring = &adev->uvd.ring; + struct amdgpu_ring *ring = &adev->uvd.inst->ring; if (RREG32(mmUVD_STATUS) != 0) uvd_v6_0_stop(adev); @@ -611,9 +612,9 @@ static void uvd_v6_0_mc_resume(struct amdgpu_device *adev) /* programm memory controller bits 0-27 */ WREG32(mmUVD_LMI_VCPU_CACHE_64BIT_BAR_LOW, - lower_32_bits(adev->uvd.gpu_addr)); + lower_32_bits(adev->uvd.inst->gpu_addr)); WREG32(mmUVD_LMI_VCPU_CACHE_64BIT_BAR_HIGH, - upper_32_bits(adev->uvd.gpu_addr)); + upper_32_bits(adev->uvd.inst->gpu_addr)); offset = AMDGPU_UVD_FIRMWARE_OFFSET; size = AMDGPU_UVD_FIRMWARE_SIZE(adev); @@ -726,7 +727,7 @@ static void cz_set_uvd_clock_gating_branches(struct amdgpu_device *adev, */ static int uvd_v6_0_start(struct amdgpu_device *adev) { - struct amdgpu_ring *ring = &adev->uvd.ring; + struct amdgpu_ring *ring = &adev->uvd.inst->ring; uint32_t rb_bufsz, tmp; uint32_t lmi_swap_cntl; uint32_t mp_swap_cntl; @@ -866,14 +867,14 @@ static int uvd_v6_0_start(struct amdgpu_device *adev) WREG32_FIELD(UVD_RBC_RB_CNTL, RB_NO_FETCH, 0); if (uvd_v6_0_enc_support(adev)) { - ring = &adev->uvd.ring_enc[0]; + ring = &adev->uvd.inst->ring_enc[0]; WREG32(mmUVD_RB_RPTR, lower_32_bits(ring->wptr)); WREG32(mmUVD_RB_WPTR, lower_32_bits(ring->wptr)); WREG32(mmUVD_RB_BASE_LO, ring->gpu_addr); WREG32(mmUVD_RB_BASE_HI, upper_32_bits(ring->gpu_addr)); WREG32(mmUVD_RB_SIZE, ring->ring_size / 4); - ring = &adev->uvd.ring_enc[1]; + ring = &adev->uvd.inst->ring_enc[1]; WREG32(mmUVD_RB_RPTR2, lower_32_bits(ring->wptr)); WREG32(mmUVD_RB_WPTR2, lower_32_bits(ring->wptr)); WREG32(mmUVD_RB_BASE_LO2, ring->gpu_addr); @@ -963,6 +964,16 @@ static void uvd_v6_0_enc_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, amdgpu_ring_write(ring, HEVC_ENC_CMD_TRAP); } +/** + * uvd_v6_0_ring_emit_hdp_flush - skip HDP flushing + * + * @ring: amdgpu_ring pointer + */ +static void uvd_v6_0_ring_emit_hdp_flush(struct amdgpu_ring *ring) +{ + /* The firmware doesn't seem to like touching registers at this point. */ +} + /** * uvd_v6_0_ring_test_ring - register write test * @@ -1089,6 +1100,18 @@ static void uvd_v6_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring) amdgpu_ring_write(ring, 0xE); } +static void uvd_v6_0_ring_insert_nop(struct amdgpu_ring *ring, uint32_t count) +{ + int i; + + WARN_ON(ring->wptr % 2 || count % 2); + + for (i = 0; i < count / 2; i++) { + amdgpu_ring_write(ring, PACKET0(mmUVD_NO_OP, 0)); + amdgpu_ring_write(ring, 0); + } +} + static void uvd_v6_0_enc_ring_emit_pipeline_sync(struct amdgpu_ring *ring) { uint32_t seq = ring->fence_drv.sync_seq; @@ -1148,10 +1171,10 @@ static bool uvd_v6_0_check_soft_reset(void *handle) srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET, SOFT_RESET_UVD, 1); if (srbm_soft_reset) { - adev->uvd.srbm_soft_reset = srbm_soft_reset; + adev->uvd.inst->srbm_soft_reset = srbm_soft_reset; return true; } else { - adev->uvd.srbm_soft_reset = 0; + adev->uvd.inst->srbm_soft_reset = 0; return false; } } @@ -1160,7 +1183,7 @@ static int uvd_v6_0_pre_soft_reset(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - if (!adev->uvd.srbm_soft_reset) + if (!adev->uvd.inst->srbm_soft_reset) return 0; uvd_v6_0_stop(adev); @@ -1172,9 +1195,9 @@ static int uvd_v6_0_soft_reset(void *handle) struct amdgpu_device *adev = (struct amdgpu_device *)handle; u32 srbm_soft_reset; - if (!adev->uvd.srbm_soft_reset) + if (!adev->uvd.inst->srbm_soft_reset) return 0; - srbm_soft_reset = adev->uvd.srbm_soft_reset; + srbm_soft_reset = adev->uvd.inst->srbm_soft_reset; if (srbm_soft_reset) { u32 tmp; @@ -1202,7 +1225,7 @@ static int uvd_v6_0_post_soft_reset(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - if (!adev->uvd.srbm_soft_reset) + if (!adev->uvd.inst->srbm_soft_reset) return 0; mdelay(5); @@ -1228,17 +1251,17 @@ static int uvd_v6_0_process_interrupt(struct amdgpu_device *adev, switch (entry->src_id) { case 124: - amdgpu_fence_process(&adev->uvd.ring); + amdgpu_fence_process(&adev->uvd.inst->ring); break; case 119: if (likely(uvd_v6_0_enc_support(adev))) - amdgpu_fence_process(&adev->uvd.ring_enc[0]); + amdgpu_fence_process(&adev->uvd.inst->ring_enc[0]); else int_handled = false; break; case 120: if (likely(uvd_v6_0_enc_support(adev))) - amdgpu_fence_process(&adev->uvd.ring_enc[1]); + amdgpu_fence_process(&adev->uvd.inst->ring_enc[1]); else int_handled = false; break; @@ -1521,22 +1544,22 @@ static const struct amd_ip_funcs uvd_v6_0_ip_funcs = { static const struct amdgpu_ring_funcs uvd_v6_0_ring_phys_funcs = { .type = AMDGPU_RING_TYPE_UVD, .align_mask = 0xf, - .nop = PACKET0(mmUVD_NO_OP, 0), .support_64bit_ptrs = false, .get_rptr = uvd_v6_0_ring_get_rptr, .get_wptr = uvd_v6_0_ring_get_wptr, .set_wptr = uvd_v6_0_ring_set_wptr, .parse_cs = amdgpu_uvd_ring_parse_cs, .emit_frame_size = - 6 + 6 + /* hdp flush / invalidate */ + 6 + /* hdp invalidate */ 10 + /* uvd_v6_0_ring_emit_pipeline_sync */ 14, /* uvd_v6_0_ring_emit_fence x1 no user fence */ .emit_ib_size = 8, /* uvd_v6_0_ring_emit_ib */ .emit_ib = uvd_v6_0_ring_emit_ib, .emit_fence = uvd_v6_0_ring_emit_fence, + .emit_hdp_flush = uvd_v6_0_ring_emit_hdp_flush, .test_ring = uvd_v6_0_ring_test_ring, .test_ib = amdgpu_uvd_ring_test_ib, - .insert_nop = amdgpu_ring_insert_nop, + .insert_nop = uvd_v6_0_ring_insert_nop, .pad_ib = amdgpu_ring_generic_pad_ib, .begin_use = amdgpu_uvd_ring_begin_use, .end_use = amdgpu_uvd_ring_end_use, @@ -1552,7 +1575,7 @@ static const struct amdgpu_ring_funcs uvd_v6_0_ring_vm_funcs = { .get_wptr = uvd_v6_0_ring_get_wptr, .set_wptr = uvd_v6_0_ring_set_wptr, .emit_frame_size = - 6 + 6 + /* hdp flush / invalidate */ + 6 + /* hdp invalidate */ 10 + /* uvd_v6_0_ring_emit_pipeline_sync */ VI_FLUSH_GPU_TLB_NUM_WREG * 6 + 8 + /* uvd_v6_0_ring_emit_vm_flush */ 14 + 14, /* uvd_v6_0_ring_emit_fence x2 vm fence */ @@ -1561,6 +1584,7 @@ static const struct amdgpu_ring_funcs uvd_v6_0_ring_vm_funcs = { .emit_fence = uvd_v6_0_ring_emit_fence, .emit_vm_flush = uvd_v6_0_ring_emit_vm_flush, .emit_pipeline_sync = uvd_v6_0_ring_emit_pipeline_sync, + .emit_hdp_flush = uvd_v6_0_ring_emit_hdp_flush, .test_ring = uvd_v6_0_ring_test_ring, .test_ib = amdgpu_uvd_ring_test_ib, .insert_nop = amdgpu_ring_insert_nop, @@ -1600,10 +1624,10 @@ static const struct amdgpu_ring_funcs uvd_v6_0_enc_ring_vm_funcs = { static void uvd_v6_0_set_ring_funcs(struct amdgpu_device *adev) { if (adev->asic_type >= CHIP_POLARIS10) { - adev->uvd.ring.funcs = &uvd_v6_0_ring_vm_funcs; + adev->uvd.inst->ring.funcs = &uvd_v6_0_ring_vm_funcs; DRM_INFO("UVD is enabled in VM mode\n"); } else { - adev->uvd.ring.funcs = &uvd_v6_0_ring_phys_funcs; + adev->uvd.inst->ring.funcs = &uvd_v6_0_ring_phys_funcs; DRM_INFO("UVD is enabled in physical mode\n"); } } @@ -1613,7 +1637,7 @@ static void uvd_v6_0_set_enc_ring_funcs(struct amdgpu_device *adev) int i; for (i = 0; i < adev->uvd.num_enc_rings; ++i) - adev->uvd.ring_enc[i].funcs = &uvd_v6_0_enc_ring_vm_funcs; + adev->uvd.inst->ring_enc[i].funcs = &uvd_v6_0_enc_ring_vm_funcs; DRM_INFO("UVD ENC is enabled in VM mode\n"); } @@ -1626,11 +1650,11 @@ static const struct amdgpu_irq_src_funcs uvd_v6_0_irq_funcs = { static void uvd_v6_0_set_irq_funcs(struct amdgpu_device *adev) { if (uvd_v6_0_enc_support(adev)) - adev->uvd.irq.num_types = adev->uvd.num_enc_rings + 1; + adev->uvd.inst->irq.num_types = adev->uvd.num_enc_rings + 1; else - adev->uvd.irq.num_types = 1; + adev->uvd.inst->irq.num_types = 1; - adev->uvd.irq.funcs = &uvd_v6_0_irq_funcs; + adev->uvd.inst->irq.funcs = &uvd_v6_0_irq_funcs; } const struct amdgpu_ip_block_version uvd_v6_0_ip_block = diff --git a/drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c b/drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c index eddc57f3b72a..57d32f21b3a6 100644 --- a/drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c +++ b/drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c @@ -40,6 +40,8 @@ #include "mmhub/mmhub_1_0_offset.h" #include "mmhub/mmhub_1_0_sh_mask.h" +#define UVD7_MAX_HW_INSTANCES_VEGA20 2 + static void uvd_v7_0_set_ring_funcs(struct amdgpu_device *adev); static void uvd_v7_0_set_enc_ring_funcs(struct amdgpu_device *adev); static void uvd_v7_0_set_irq_funcs(struct amdgpu_device *adev); @@ -47,6 +49,11 @@ static int uvd_v7_0_start(struct amdgpu_device *adev); static void uvd_v7_0_stop(struct amdgpu_device *adev); static int uvd_v7_0_sriov_start(struct amdgpu_device *adev); +static int amdgpu_ih_clientid_uvds[] = { + SOC15_IH_CLIENTID_UVD, + SOC15_IH_CLIENTID_UVD1 +}; + /** * uvd_v7_0_ring_get_rptr - get read pointer * @@ -58,7 +65,7 @@ static uint64_t uvd_v7_0_ring_get_rptr(struct amdgpu_ring *ring) { struct amdgpu_device *adev = ring->adev; - return RREG32_SOC15(UVD, 0, mmUVD_RBC_RB_RPTR); + return RREG32_SOC15(UVD, ring->me, mmUVD_RBC_RB_RPTR); } /** @@ -72,10 +79,10 @@ static uint64_t uvd_v7_0_enc_ring_get_rptr(struct amdgpu_ring *ring) { struct amdgpu_device *adev = ring->adev; - if (ring == &adev->uvd.ring_enc[0]) - return RREG32_SOC15(UVD, 0, mmUVD_RB_RPTR); + if (ring == &adev->uvd.inst[ring->me].ring_enc[0]) + return RREG32_SOC15(UVD, ring->me, mmUVD_RB_RPTR); else - return RREG32_SOC15(UVD, 0, mmUVD_RB_RPTR2); + return RREG32_SOC15(UVD, ring->me, mmUVD_RB_RPTR2); } /** @@ -89,7 +96,7 @@ static uint64_t uvd_v7_0_ring_get_wptr(struct amdgpu_ring *ring) { struct amdgpu_device *adev = ring->adev; - return RREG32_SOC15(UVD, 0, mmUVD_RBC_RB_WPTR); + return RREG32_SOC15(UVD, ring->me, mmUVD_RBC_RB_WPTR); } /** @@ -106,10 +113,10 @@ static uint64_t uvd_v7_0_enc_ring_get_wptr(struct amdgpu_ring *ring) if (ring->use_doorbell) return adev->wb.wb[ring->wptr_offs]; - if (ring == &adev->uvd.ring_enc[0]) - return RREG32_SOC15(UVD, 0, mmUVD_RB_WPTR); + if (ring == &adev->uvd.inst[ring->me].ring_enc[0]) + return RREG32_SOC15(UVD, ring->me, mmUVD_RB_WPTR); else - return RREG32_SOC15(UVD, 0, mmUVD_RB_WPTR2); + return RREG32_SOC15(UVD, ring->me, mmUVD_RB_WPTR2); } /** @@ -123,7 +130,7 @@ static void uvd_v7_0_ring_set_wptr(struct amdgpu_ring *ring) { struct amdgpu_device *adev = ring->adev; - WREG32_SOC15(UVD, 0, mmUVD_RBC_RB_WPTR, lower_32_bits(ring->wptr)); + WREG32_SOC15(UVD, ring->me, mmUVD_RBC_RB_WPTR, lower_32_bits(ring->wptr)); } /** @@ -144,11 +151,11 @@ static void uvd_v7_0_enc_ring_set_wptr(struct amdgpu_ring *ring) return; } - if (ring == &adev->uvd.ring_enc[0]) - WREG32_SOC15(UVD, 0, mmUVD_RB_WPTR, + if (ring == &adev->uvd.inst[ring->me].ring_enc[0]) + WREG32_SOC15(UVD, ring->me, mmUVD_RB_WPTR, lower_32_bits(ring->wptr)); else - WREG32_SOC15(UVD, 0, mmUVD_RB_WPTR2, + WREG32_SOC15(UVD, ring->me, mmUVD_RB_WPTR2, lower_32_bits(ring->wptr)); } @@ -170,8 +177,8 @@ static int uvd_v7_0_enc_ring_test_ring(struct amdgpu_ring *ring) r = amdgpu_ring_alloc(ring, 16); if (r) { - DRM_ERROR("amdgpu: uvd enc failed to lock ring %d (%d).\n", - ring->idx, r); + DRM_ERROR("amdgpu: uvd enc failed to lock (%d)ring %d (%d).\n", + ring->me, ring->idx, r); return r; } amdgpu_ring_write(ring, HEVC_ENC_CMD_END); @@ -184,11 +191,11 @@ static int uvd_v7_0_enc_ring_test_ring(struct amdgpu_ring *ring) } if (i < adev->usec_timeout) { - DRM_DEBUG("ring test on %d succeeded in %d usecs\n", - ring->idx, i); + DRM_DEBUG("(%d)ring test on %d succeeded in %d usecs\n", + ring->me, ring->idx, i); } else { - DRM_ERROR("amdgpu: ring %d test failed\n", - ring->idx); + DRM_ERROR("amdgpu: (%d)ring %d test failed\n", + ring->me, ring->idx); r = -ETIMEDOUT; } @@ -342,24 +349,24 @@ static int uvd_v7_0_enc_ring_test_ib(struct amdgpu_ring *ring, long timeout) r = uvd_v7_0_enc_get_create_msg(ring, 1, NULL); if (r) { - DRM_ERROR("amdgpu: failed to get create msg (%ld).\n", r); + DRM_ERROR("amdgpu: (%d)failed to get create msg (%ld).\n", ring->me, r); goto error; } r = uvd_v7_0_enc_get_destroy_msg(ring, 1, true, &fence); if (r) { - DRM_ERROR("amdgpu: failed to get destroy ib (%ld).\n", r); + DRM_ERROR("amdgpu: (%d)failed to get destroy ib (%ld).\n", ring->me, r); goto error; } r = dma_fence_wait_timeout(fence, false, timeout); if (r == 0) { - DRM_ERROR("amdgpu: IB test timed out.\n"); + DRM_ERROR("amdgpu: (%d)IB test timed out.\n", ring->me); r = -ETIMEDOUT; } else if (r < 0) { - DRM_ERROR("amdgpu: fence wait failed (%ld).\n", r); + DRM_ERROR("amdgpu: (%d)fence wait failed (%ld).\n", ring->me, r); } else { - DRM_DEBUG("ib test on ring %d succeeded\n", ring->idx); + DRM_DEBUG("ib test on (%d)ring %d succeeded\n", ring->me, ring->idx); r = 0; } error: @@ -370,6 +377,10 @@ error: static int uvd_v7_0_early_init(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; + if (adev->asic_type == CHIP_VEGA20) + adev->uvd.num_uvd_inst = UVD7_MAX_HW_INSTANCES_VEGA20; + else + adev->uvd.num_uvd_inst = 1; if (amdgpu_sriov_vf(adev)) adev->uvd.num_enc_rings = 1; @@ -386,19 +397,21 @@ static int uvd_v7_0_sw_init(void *handle) { struct amdgpu_ring *ring; struct drm_sched_rq *rq; - int i, r; + int i, j, r; struct amdgpu_device *adev = (struct amdgpu_device *)handle; - /* UVD TRAP */ - r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_UVD, 124, &adev->uvd.irq); - if (r) - return r; - - /* UVD ENC TRAP */ - for (i = 0; i < adev->uvd.num_enc_rings; ++i) { - r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_UVD, i + 119, &adev->uvd.irq); + for (j = 0; j < adev->uvd.num_uvd_inst; j++) { + /* UVD TRAP */ + r = amdgpu_irq_add_id(adev, amdgpu_ih_clientid_uvds[j], 124, &adev->uvd.inst[j].irq); if (r) return r; + + /* UVD ENC TRAP */ + for (i = 0; i < adev->uvd.num_enc_rings; ++i) { + r = amdgpu_irq_add_id(adev, amdgpu_ih_clientid_uvds[j], i + 119, &adev->uvd.inst[j].irq); + if (r) + return r; + } } r = amdgpu_uvd_sw_init(adev); @@ -415,43 +428,48 @@ static int uvd_v7_0_sw_init(void *handle) DRM_INFO("PSP loading UVD firmware\n"); } - ring = &adev->uvd.ring_enc[0]; - rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL]; - r = drm_sched_entity_init(&ring->sched, &adev->uvd.entity_enc, - rq, amdgpu_sched_jobs, NULL); - if (r) { - DRM_ERROR("Failed setting up UVD ENC run queue.\n"); - return r; + for (j = 0; j < adev->uvd.num_uvd_inst; j++) { + ring = &adev->uvd.inst[j].ring_enc[0]; + rq = &ring->sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL]; + r = drm_sched_entity_init(&ring->sched, &adev->uvd.inst[j].entity_enc, + rq, NULL); + if (r) { + DRM_ERROR("(%d)Failed setting up UVD ENC run queue.\n", j); + return r; + } } r = amdgpu_uvd_resume(adev); if (r) return r; - if (!amdgpu_sriov_vf(adev)) { - ring = &adev->uvd.ring; - sprintf(ring->name, "uvd"); - r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.irq, 0); - if (r) - return r; - } - for (i = 0; i < adev->uvd.num_enc_rings; ++i) { - ring = &adev->uvd.ring_enc[i]; - sprintf(ring->name, "uvd_enc%d", i); - if (amdgpu_sriov_vf(adev)) { - ring->use_doorbell = true; - - /* currently only use the first enconding ring for - * sriov, so set unused location for other unused rings. - */ - if (i == 0) - ring->doorbell_index = AMDGPU_DOORBELL64_UVD_RING0_1 * 2; - else - ring->doorbell_index = AMDGPU_DOORBELL64_UVD_RING2_3 * 2 + 1; + for (j = 0; j < adev->uvd.num_uvd_inst; j++) { + if (!amdgpu_sriov_vf(adev)) { + ring = &adev->uvd.inst[j].ring; + sprintf(ring->name, "uvd<%d>", j); + r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.inst[j].irq, 0); + if (r) + return r; + } + + for (i = 0; i < adev->uvd.num_enc_rings; ++i) { + ring = &adev->uvd.inst[j].ring_enc[i]; + sprintf(ring->name, "uvd_enc%d<%d>", i, j); + if (amdgpu_sriov_vf(adev)) { + ring->use_doorbell = true; + + /* currently only use the first enconding ring for + * sriov, so set unused location for other unused rings. + */ + if (i == 0) + ring->doorbell_index = AMDGPU_DOORBELL64_UVD_RING0_1 * 2; + else + ring->doorbell_index = AMDGPU_DOORBELL64_UVD_RING2_3 * 2 + 1; + } + r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.inst[j].irq, 0); + if (r) + return r; } - r = amdgpu_ring_init(adev, ring, 512, &adev->uvd.irq, 0); - if (r) - return r; } r = amdgpu_virt_alloc_mm_table(adev); @@ -463,7 +481,7 @@ static int uvd_v7_0_sw_init(void *handle) static int uvd_v7_0_sw_fini(void *handle) { - int i, r; + int i, j, r; struct amdgpu_device *adev = (struct amdgpu_device *)handle; amdgpu_virt_free_mm_table(adev); @@ -472,11 +490,12 @@ static int uvd_v7_0_sw_fini(void *handle) if (r) return r; - drm_sched_entity_fini(&adev->uvd.ring_enc[0].sched, &adev->uvd.entity_enc); - - for (i = 0; i < adev->uvd.num_enc_rings; ++i) - amdgpu_ring_fini(&adev->uvd.ring_enc[i]); + for (j = 0; j < adev->uvd.num_uvd_inst; ++j) { + drm_sched_entity_fini(&adev->uvd.inst[j].ring_enc[0].sched, &adev->uvd.inst[j].entity_enc); + for (i = 0; i < adev->uvd.num_enc_rings; ++i) + amdgpu_ring_fini(&adev->uvd.inst[j].ring_enc[i]); + } return amdgpu_uvd_sw_fini(adev); } @@ -490,9 +509,9 @@ static int uvd_v7_0_sw_fini(void *handle) static int uvd_v7_0_hw_init(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - struct amdgpu_ring *ring = &adev->uvd.ring; + struct amdgpu_ring *ring; uint32_t tmp; - int i, r; + int i, j, r; if (amdgpu_sriov_vf(adev)) r = uvd_v7_0_sriov_start(adev); @@ -501,57 +520,60 @@ static int uvd_v7_0_hw_init(void *handle) if (r) goto done; - if (!amdgpu_sriov_vf(adev)) { - ring->ready = true; - r = amdgpu_ring_test_ring(ring); - if (r) { - ring->ready = false; - goto done; + for (j = 0; j < adev->uvd.num_uvd_inst; ++j) { + ring = &adev->uvd.inst[j].ring; + + if (!amdgpu_sriov_vf(adev)) { + ring->ready = true; + r = amdgpu_ring_test_ring(ring); + if (r) { + ring->ready = false; + goto done; + } + + r = amdgpu_ring_alloc(ring, 10); + if (r) { + DRM_ERROR("amdgpu: (%d)ring failed to lock UVD ring (%d).\n", j, r); + goto done; + } + + tmp = PACKET0(SOC15_REG_OFFSET(UVD, j, + mmUVD_SEMA_WAIT_FAULT_TIMEOUT_CNTL), 0); + amdgpu_ring_write(ring, tmp); + amdgpu_ring_write(ring, 0xFFFFF); + + tmp = PACKET0(SOC15_REG_OFFSET(UVD, j, + mmUVD_SEMA_WAIT_INCOMPLETE_TIMEOUT_CNTL), 0); + amdgpu_ring_write(ring, tmp); + amdgpu_ring_write(ring, 0xFFFFF); + + tmp = PACKET0(SOC15_REG_OFFSET(UVD, j, + mmUVD_SEMA_SIGNAL_INCOMPLETE_TIMEOUT_CNTL), 0); + amdgpu_ring_write(ring, tmp); + amdgpu_ring_write(ring, 0xFFFFF); + + /* Clear timeout status bits */ + amdgpu_ring_write(ring, PACKET0(SOC15_REG_OFFSET(UVD, j, + mmUVD_SEMA_TIMEOUT_STATUS), 0)); + amdgpu_ring_write(ring, 0x8); + + amdgpu_ring_write(ring, PACKET0(SOC15_REG_OFFSET(UVD, j, + mmUVD_SEMA_CNTL), 0)); + amdgpu_ring_write(ring, 3); + + amdgpu_ring_commit(ring); } - r = amdgpu_ring_alloc(ring, 10); - if (r) { - DRM_ERROR("amdgpu: ring failed to lock UVD ring (%d).\n", r); - goto done; + for (i = 0; i < adev->uvd.num_enc_rings; ++i) { + ring = &adev->uvd.inst[j].ring_enc[i]; + ring->ready = true; + r = amdgpu_ring_test_ring(ring); + if (r) { + ring->ready = false; + goto done; + } } - - tmp = PACKET0(SOC15_REG_OFFSET(UVD, 0, - mmUVD_SEMA_WAIT_FAULT_TIMEOUT_CNTL), 0); - amdgpu_ring_write(ring, tmp); - amdgpu_ring_write(ring, 0xFFFFF); - - tmp = PACKET0(SOC15_REG_OFFSET(UVD, 0, - mmUVD_SEMA_WAIT_INCOMPLETE_TIMEOUT_CNTL), 0); - amdgpu_ring_write(ring, tmp); - amdgpu_ring_write(ring, 0xFFFFF); - - tmp = PACKET0(SOC15_REG_OFFSET(UVD, 0, - mmUVD_SEMA_SIGNAL_INCOMPLETE_TIMEOUT_CNTL), 0); - amdgpu_ring_write(ring, tmp); - amdgpu_ring_write(ring, 0xFFFFF); - - /* Clear timeout status bits */ - amdgpu_ring_write(ring, PACKET0(SOC15_REG_OFFSET(UVD, 0, - mmUVD_SEMA_TIMEOUT_STATUS), 0)); - amdgpu_ring_write(ring, 0x8); - - amdgpu_ring_write(ring, PACKET0(SOC15_REG_OFFSET(UVD, 0, - mmUVD_SEMA_CNTL), 0)); - amdgpu_ring_write(ring, 3); - - amdgpu_ring_commit(ring); } - - for (i = 0; i < adev->uvd.num_enc_rings; ++i) { - ring = &adev->uvd.ring_enc[i]; - ring->ready = true; - r = amdgpu_ring_test_ring(ring); - if (r) { - ring->ready = false; - goto done; - } - } - done: if (!r) DRM_INFO("UVD and UVD ENC initialized successfully.\n"); @@ -569,7 +591,7 @@ done: static int uvd_v7_0_hw_fini(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - struct amdgpu_ring *ring = &adev->uvd.ring; + int i; if (!amdgpu_sriov_vf(adev)) uvd_v7_0_stop(adev); @@ -578,7 +600,8 @@ static int uvd_v7_0_hw_fini(void *handle) DRM_DEBUG("For SRIOV client, shouldn't do anything.\n"); } - ring->ready = false; + for (i = 0; i < adev->uvd.num_uvd_inst; ++i) + adev->uvd.inst[i].ring.ready = false; return 0; } @@ -618,48 +641,51 @@ static void uvd_v7_0_mc_resume(struct amdgpu_device *adev) { uint32_t size = AMDGPU_UVD_FIRMWARE_SIZE(adev); uint32_t offset; + int i; - if (adev->firmware.load_type == AMDGPU_FW_LOAD_PSP) { - WREG32_SOC15(UVD, 0, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_LOW, - lower_32_bits(adev->firmware.ucode[AMDGPU_UCODE_ID_UVD].mc_addr)); - WREG32_SOC15(UVD, 0, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_HIGH, - upper_32_bits(adev->firmware.ucode[AMDGPU_UCODE_ID_UVD].mc_addr)); - offset = 0; - } else { - WREG32_SOC15(UVD, 0, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_LOW, - lower_32_bits(adev->uvd.gpu_addr)); - WREG32_SOC15(UVD, 0, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_HIGH, - upper_32_bits(adev->uvd.gpu_addr)); - offset = size; - } + for (i = 0; i < adev->uvd.num_uvd_inst; ++i) { + if (adev->firmware.load_type == AMDGPU_FW_LOAD_PSP) { + WREG32_SOC15(UVD, i, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_LOW, + lower_32_bits(adev->firmware.ucode[AMDGPU_UCODE_ID_UVD].mc_addr)); + WREG32_SOC15(UVD, i, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_HIGH, + upper_32_bits(adev->firmware.ucode[AMDGPU_UCODE_ID_UVD].mc_addr)); + offset = 0; + } else { + WREG32_SOC15(UVD, i, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_LOW, + lower_32_bits(adev->uvd.inst[i].gpu_addr)); + WREG32_SOC15(UVD, i, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_HIGH, + upper_32_bits(adev->uvd.inst[i].gpu_addr)); + offset = size; + } - WREG32_SOC15(UVD, 0, mmUVD_VCPU_CACHE_OFFSET0, - AMDGPU_UVD_FIRMWARE_OFFSET >> 3); - WREG32_SOC15(UVD, 0, mmUVD_VCPU_CACHE_SIZE0, size); - - WREG32_SOC15(UVD, 0, mmUVD_LMI_VCPU_CACHE1_64BIT_BAR_LOW, - lower_32_bits(adev->uvd.gpu_addr + offset)); - WREG32_SOC15(UVD, 0, mmUVD_LMI_VCPU_CACHE1_64BIT_BAR_HIGH, - upper_32_bits(adev->uvd.gpu_addr + offset)); - WREG32_SOC15(UVD, 0, mmUVD_VCPU_CACHE_OFFSET1, (1 << 21)); - WREG32_SOC15(UVD, 0, mmUVD_VCPU_CACHE_SIZE1, AMDGPU_UVD_HEAP_SIZE); - - WREG32_SOC15(UVD, 0, mmUVD_LMI_VCPU_CACHE2_64BIT_BAR_LOW, - lower_32_bits(adev->uvd.gpu_addr + offset + AMDGPU_UVD_HEAP_SIZE)); - WREG32_SOC15(UVD, 0, mmUVD_LMI_VCPU_CACHE2_64BIT_BAR_HIGH, - upper_32_bits(adev->uvd.gpu_addr + offset + AMDGPU_UVD_HEAP_SIZE)); - WREG32_SOC15(UVD, 0, mmUVD_VCPU_CACHE_OFFSET2, (2 << 21)); - WREG32_SOC15(UVD, 0, mmUVD_VCPU_CACHE_SIZE2, - AMDGPU_UVD_STACK_SIZE + (AMDGPU_UVD_SESSION_SIZE * 40)); - - WREG32_SOC15(UVD, 0, mmUVD_UDEC_ADDR_CONFIG, - adev->gfx.config.gb_addr_config); - WREG32_SOC15(UVD, 0, mmUVD_UDEC_DB_ADDR_CONFIG, - adev->gfx.config.gb_addr_config); - WREG32_SOC15(UVD, 0, mmUVD_UDEC_DBW_ADDR_CONFIG, - adev->gfx.config.gb_addr_config); - - WREG32_SOC15(UVD, 0, mmUVD_GP_SCRATCH4, adev->uvd.max_handles); + WREG32_SOC15(UVD, i, mmUVD_VCPU_CACHE_OFFSET0, + AMDGPU_UVD_FIRMWARE_OFFSET >> 3); + WREG32_SOC15(UVD, i, mmUVD_VCPU_CACHE_SIZE0, size); + + WREG32_SOC15(UVD, i, mmUVD_LMI_VCPU_CACHE1_64BIT_BAR_LOW, + lower_32_bits(adev->uvd.inst[i].gpu_addr + offset)); + WREG32_SOC15(UVD, i, mmUVD_LMI_VCPU_CACHE1_64BIT_BAR_HIGH, + upper_32_bits(adev->uvd.inst[i].gpu_addr + offset)); + WREG32_SOC15(UVD, i, mmUVD_VCPU_CACHE_OFFSET1, (1 << 21)); + WREG32_SOC15(UVD, i, mmUVD_VCPU_CACHE_SIZE1, AMDGPU_UVD_HEAP_SIZE); + + WREG32_SOC15(UVD, i, mmUVD_LMI_VCPU_CACHE2_64BIT_BAR_LOW, + lower_32_bits(adev->uvd.inst[i].gpu_addr + offset + AMDGPU_UVD_HEAP_SIZE)); + WREG32_SOC15(UVD, i, mmUVD_LMI_VCPU_CACHE2_64BIT_BAR_HIGH, + upper_32_bits(adev->uvd.inst[i].gpu_addr + offset + AMDGPU_UVD_HEAP_SIZE)); + WREG32_SOC15(UVD, i, mmUVD_VCPU_CACHE_OFFSET2, (2 << 21)); + WREG32_SOC15(UVD, i, mmUVD_VCPU_CACHE_SIZE2, + AMDGPU_UVD_STACK_SIZE + (AMDGPU_UVD_SESSION_SIZE * 40)); + + WREG32_SOC15(UVD, i, mmUVD_UDEC_ADDR_CONFIG, + adev->gfx.config.gb_addr_config); + WREG32_SOC15(UVD, i, mmUVD_UDEC_DB_ADDR_CONFIG, + adev->gfx.config.gb_addr_config); + WREG32_SOC15(UVD, i, mmUVD_UDEC_DBW_ADDR_CONFIG, + adev->gfx.config.gb_addr_config); + + WREG32_SOC15(UVD, i, mmUVD_GP_SCRATCH4, adev->uvd.max_handles); + } } static int uvd_v7_0_mmsch_start(struct amdgpu_device *adev, @@ -669,6 +695,7 @@ static int uvd_v7_0_mmsch_start(struct amdgpu_device *adev, uint64_t addr = table->gpu_addr; struct mmsch_v1_0_init_header *header = (struct mmsch_v1_0_init_header *)table->cpu_addr; uint32_t size; + int i; size = header->header_size + header->vce_table_size + header->uvd_table_size; @@ -688,11 +715,12 @@ static int uvd_v7_0_mmsch_start(struct amdgpu_device *adev, /* 4, set resp to zero */ WREG32_SOC15(VCE, 0, mmVCE_MMSCH_VF_MAILBOX_RESP, 0); - WDOORBELL32(adev->uvd.ring_enc[0].doorbell_index, 0); - adev->wb.wb[adev->uvd.ring_enc[0].wptr_offs] = 0; - adev->uvd.ring_enc[0].wptr = 0; - adev->uvd.ring_enc[0].wptr_old = 0; - + for (i = 0; i < adev->uvd.num_uvd_inst; ++i) { + WDOORBELL32(adev->uvd.inst[i].ring_enc[0].doorbell_index, 0); + adev->wb.wb[adev->uvd.inst[i].ring_enc[0].wptr_offs] = 0; + adev->uvd.inst[i].ring_enc[0].wptr = 0; + adev->uvd.inst[i].ring_enc[0].wptr_old = 0; + } /* 5, kick off the initialization and wait until VCE_MMSCH_VF_MAILBOX_RESP becomes non-zero */ WREG32_SOC15(VCE, 0, mmVCE_MMSCH_VF_MAILBOX_HOST, 0x10000001); @@ -725,6 +753,7 @@ static int uvd_v7_0_sriov_start(struct amdgpu_device *adev) struct mmsch_v1_0_cmd_end end = { {0} }; uint32_t *init_table = adev->virt.mm_table.cpu_addr; struct mmsch_v1_0_init_header *header = (struct mmsch_v1_0_init_header *)init_table; + uint8_t i = 0; direct_wt.cmd_header.command_type = MMSCH_COMMAND__DIRECT_REG_WRITE; direct_rd_mod_wt.cmd_header.command_type = MMSCH_COMMAND__DIRECT_REG_READ_MODIFY_WRITE; @@ -742,120 +771,121 @@ static int uvd_v7_0_sriov_start(struct amdgpu_device *adev) init_table += header->uvd_table_offset; - ring = &adev->uvd.ring; - ring->wptr = 0; - size = AMDGPU_GPU_PAGE_ALIGN(adev->uvd.fw->size + 4); - - MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_STATUS), - 0xFFFFFFFF, 0x00000004); - /* mc resume*/ - if (adev->firmware.load_type == AMDGPU_FW_LOAD_PSP) { - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_LOW), - lower_32_bits(adev->firmware.ucode[AMDGPU_UCODE_ID_UVD].mc_addr)); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_HIGH), - upper_32_bits(adev->firmware.ucode[AMDGPU_UCODE_ID_UVD].mc_addr)); - offset = 0; - } else { - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_LOW), - lower_32_bits(adev->uvd.gpu_addr)); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_HIGH), - upper_32_bits(adev->uvd.gpu_addr)); - offset = size; + for (i = 0; i < adev->uvd.num_uvd_inst; ++i) { + ring = &adev->uvd.inst[i].ring; + ring->wptr = 0; + size = AMDGPU_GPU_PAGE_ALIGN(adev->uvd.fw->size + 4); + + MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_STATUS), + 0xFFFFFFFF, 0x00000004); + /* mc resume*/ + if (adev->firmware.load_type == AMDGPU_FW_LOAD_PSP) { + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_LOW), + lower_32_bits(adev->firmware.ucode[AMDGPU_UCODE_ID_UVD].mc_addr)); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_HIGH), + upper_32_bits(adev->firmware.ucode[AMDGPU_UCODE_ID_UVD].mc_addr)); + offset = 0; + } else { + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_LOW), + lower_32_bits(adev->uvd.inst[i].gpu_addr)); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_VCPU_CACHE_64BIT_BAR_HIGH), + upper_32_bits(adev->uvd.inst[i].gpu_addr)); + offset = size; + } + + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_VCPU_CACHE_OFFSET0), + AMDGPU_UVD_FIRMWARE_OFFSET >> 3); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_VCPU_CACHE_SIZE0), size); + + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_VCPU_CACHE1_64BIT_BAR_LOW), + lower_32_bits(adev->uvd.inst[i].gpu_addr + offset)); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_VCPU_CACHE1_64BIT_BAR_HIGH), + upper_32_bits(adev->uvd.inst[i].gpu_addr + offset)); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_VCPU_CACHE_OFFSET1), (1 << 21)); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_VCPU_CACHE_SIZE1), AMDGPU_UVD_HEAP_SIZE); + + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_VCPU_CACHE2_64BIT_BAR_LOW), + lower_32_bits(adev->uvd.inst[i].gpu_addr + offset + AMDGPU_UVD_HEAP_SIZE)); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_VCPU_CACHE2_64BIT_BAR_HIGH), + upper_32_bits(adev->uvd.inst[i].gpu_addr + offset + AMDGPU_UVD_HEAP_SIZE)); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_VCPU_CACHE_OFFSET2), (2 << 21)); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_VCPU_CACHE_SIZE2), + AMDGPU_UVD_STACK_SIZE + (AMDGPU_UVD_SESSION_SIZE * 40)); + + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_GP_SCRATCH4), adev->uvd.max_handles); + /* mc resume end*/ + + /* disable clock gating */ + MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_CGC_CTRL), + ~UVD_CGC_CTRL__DYN_CLOCK_MODE_MASK, 0); + + /* disable interupt */ + MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_MASTINT_EN), + ~UVD_MASTINT_EN__VCPU_EN_MASK, 0); + + /* stall UMC and register bus before resetting VCPU */ + MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_CTRL2), + ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK, + UVD_LMI_CTRL2__STALL_ARB_UMC_MASK); + + /* put LMI, VCPU, RBC etc... into reset */ + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_SOFT_RESET), + (uint32_t)(UVD_SOFT_RESET__LMI_SOFT_RESET_MASK | + UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK | + UVD_SOFT_RESET__LBSI_SOFT_RESET_MASK | + UVD_SOFT_RESET__RBC_SOFT_RESET_MASK | + UVD_SOFT_RESET__CSM_SOFT_RESET_MASK | + UVD_SOFT_RESET__CXW_SOFT_RESET_MASK | + UVD_SOFT_RESET__TAP_SOFT_RESET_MASK | + UVD_SOFT_RESET__LMI_UMC_SOFT_RESET_MASK)); + + /* initialize UVD memory controller */ + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_CTRL), + (uint32_t)((0x40 << UVD_LMI_CTRL__WRITE_CLEAN_TIMER__SHIFT) | + UVD_LMI_CTRL__WRITE_CLEAN_TIMER_EN_MASK | + UVD_LMI_CTRL__DATA_COHERENCY_EN_MASK | + UVD_LMI_CTRL__VCPU_DATA_COHERENCY_EN_MASK | + UVD_LMI_CTRL__REQ_MODE_MASK | + 0x00100000L)); + + /* take all subblocks out of reset, except VCPU */ + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_SOFT_RESET), + UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK); + + /* enable VCPU clock */ + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_VCPU_CNTL), + UVD_VCPU_CNTL__CLK_EN_MASK); + + /* enable master interrupt */ + MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_MASTINT_EN), + ~(UVD_MASTINT_EN__VCPU_EN_MASK|UVD_MASTINT_EN__SYS_EN_MASK), + (UVD_MASTINT_EN__VCPU_EN_MASK|UVD_MASTINT_EN__SYS_EN_MASK)); + + /* clear the bit 4 of UVD_STATUS */ + MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_STATUS), + ~(2 << UVD_STATUS__VCPU_REPORT__SHIFT), 0); + + /* force RBC into idle state */ + size = order_base_2(ring->ring_size); + tmp = REG_SET_FIELD(0, UVD_RBC_RB_CNTL, RB_BUFSZ, size); + tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_NO_FETCH, 1); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_RBC_RB_CNTL), tmp); + + ring = &adev->uvd.inst[i].ring_enc[0]; + ring->wptr = 0; + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_RB_BASE_LO), ring->gpu_addr); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_RB_BASE_HI), upper_32_bits(ring->gpu_addr)); + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_RB_SIZE), ring->ring_size / 4); + + /* boot up the VCPU */ + MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_SOFT_RESET), 0); + + /* enable UMC */ + MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_CTRL2), + ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK, 0); + + MMSCH_V1_0_INSERT_DIRECT_POLL(SOC15_REG_OFFSET(UVD, i, mmUVD_STATUS), 0x02, 0x02); } - - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_VCPU_CACHE_OFFSET0), - AMDGPU_UVD_FIRMWARE_OFFSET >> 3); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_VCPU_CACHE_SIZE0), size); - - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_VCPU_CACHE1_64BIT_BAR_LOW), - lower_32_bits(adev->uvd.gpu_addr + offset)); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_VCPU_CACHE1_64BIT_BAR_HIGH), - upper_32_bits(adev->uvd.gpu_addr + offset)); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_VCPU_CACHE_OFFSET1), (1 << 21)); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_VCPU_CACHE_SIZE1), AMDGPU_UVD_HEAP_SIZE); - - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_VCPU_CACHE2_64BIT_BAR_LOW), - lower_32_bits(adev->uvd.gpu_addr + offset + AMDGPU_UVD_HEAP_SIZE)); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_VCPU_CACHE2_64BIT_BAR_HIGH), - upper_32_bits(adev->uvd.gpu_addr + offset + AMDGPU_UVD_HEAP_SIZE)); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_VCPU_CACHE_OFFSET2), (2 << 21)); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_VCPU_CACHE_SIZE2), - AMDGPU_UVD_STACK_SIZE + (AMDGPU_UVD_SESSION_SIZE * 40)); - - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_GP_SCRATCH4), adev->uvd.max_handles); - /* mc resume end*/ - - /* disable clock gating */ - MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_CGC_CTRL), - ~UVD_CGC_CTRL__DYN_CLOCK_MODE_MASK, 0); - - /* disable interupt */ - MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_MASTINT_EN), - ~UVD_MASTINT_EN__VCPU_EN_MASK, 0); - - /* stall UMC and register bus before resetting VCPU */ - MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_CTRL2), - ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK, - UVD_LMI_CTRL2__STALL_ARB_UMC_MASK); - - /* put LMI, VCPU, RBC etc... into reset */ - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_SOFT_RESET), - (uint32_t)(UVD_SOFT_RESET__LMI_SOFT_RESET_MASK | - UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK | - UVD_SOFT_RESET__LBSI_SOFT_RESET_MASK | - UVD_SOFT_RESET__RBC_SOFT_RESET_MASK | - UVD_SOFT_RESET__CSM_SOFT_RESET_MASK | - UVD_SOFT_RESET__CXW_SOFT_RESET_MASK | - UVD_SOFT_RESET__TAP_SOFT_RESET_MASK | - UVD_SOFT_RESET__LMI_UMC_SOFT_RESET_MASK)); - - /* initialize UVD memory controller */ - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_CTRL), - (uint32_t)((0x40 << UVD_LMI_CTRL__WRITE_CLEAN_TIMER__SHIFT) | - UVD_LMI_CTRL__WRITE_CLEAN_TIMER_EN_MASK | - UVD_LMI_CTRL__DATA_COHERENCY_EN_MASK | - UVD_LMI_CTRL__VCPU_DATA_COHERENCY_EN_MASK | - UVD_LMI_CTRL__REQ_MODE_MASK | - 0x00100000L)); - - /* take all subblocks out of reset, except VCPU */ - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_SOFT_RESET), - UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK); - - /* enable VCPU clock */ - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_VCPU_CNTL), - UVD_VCPU_CNTL__CLK_EN_MASK); - - /* enable master interrupt */ - MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_MASTINT_EN), - ~(UVD_MASTINT_EN__VCPU_EN_MASK|UVD_MASTINT_EN__SYS_EN_MASK), - (UVD_MASTINT_EN__VCPU_EN_MASK|UVD_MASTINT_EN__SYS_EN_MASK)); - - /* clear the bit 4 of UVD_STATUS */ - MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_STATUS), - ~(2 << UVD_STATUS__VCPU_REPORT__SHIFT), 0); - - /* force RBC into idle state */ - size = order_base_2(ring->ring_size); - tmp = REG_SET_FIELD(0, UVD_RBC_RB_CNTL, RB_BUFSZ, size); - tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_NO_FETCH, 1); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_RBC_RB_CNTL), tmp); - - ring = &adev->uvd.ring_enc[0]; - ring->wptr = 0; - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_RB_BASE_LO), ring->gpu_addr); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_RB_BASE_HI), upper_32_bits(ring->gpu_addr)); - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_RB_SIZE), ring->ring_size / 4); - - /* boot up the VCPU */ - MMSCH_V1_0_INSERT_DIRECT_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_SOFT_RESET), 0); - - /* enable UMC */ - MMSCH_V1_0_INSERT_DIRECT_RD_MOD_WT(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_CTRL2), - ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK, 0); - - MMSCH_V1_0_INSERT_DIRECT_POLL(SOC15_REG_OFFSET(UVD, 0, mmUVD_STATUS), 0x02, 0x02); - /* add end packet */ memcpy((void *)init_table, &end, sizeof(struct mmsch_v1_0_cmd_end)); table_size += sizeof(struct mmsch_v1_0_cmd_end) / 4; @@ -874,15 +904,17 @@ static int uvd_v7_0_sriov_start(struct amdgpu_device *adev) */ static int uvd_v7_0_start(struct amdgpu_device *adev) { - struct amdgpu_ring *ring = &adev->uvd.ring; + struct amdgpu_ring *ring; uint32_t rb_bufsz, tmp; uint32_t lmi_swap_cntl; uint32_t mp_swap_cntl; - int i, j, r; + int i, j, k, r; - /* disable DPG */ - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_POWER_STATUS), 0, - ~UVD_POWER_STATUS__UVD_PG_MODE_MASK); + for (k = 0; k < adev->uvd.num_uvd_inst; ++k) { + /* disable DPG */ + WREG32_P(SOC15_REG_OFFSET(UVD, k, mmUVD_POWER_STATUS), 0, + ~UVD_POWER_STATUS__UVD_PG_MODE_MASK); + } /* disable byte swapping */ lmi_swap_cntl = 0; @@ -890,157 +922,159 @@ static int uvd_v7_0_start(struct amdgpu_device *adev) uvd_v7_0_mc_resume(adev); - /* disable clock gating */ - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_CGC_CTRL), 0, - ~UVD_CGC_CTRL__DYN_CLOCK_MODE_MASK); - - /* disable interupt */ - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_MASTINT_EN), 0, - ~UVD_MASTINT_EN__VCPU_EN_MASK); - - /* stall UMC and register bus before resetting VCPU */ - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_CTRL2), - UVD_LMI_CTRL2__STALL_ARB_UMC_MASK, - ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK); - mdelay(1); - - /* put LMI, VCPU, RBC etc... into reset */ - WREG32_SOC15(UVD, 0, mmUVD_SOFT_RESET, - UVD_SOFT_RESET__LMI_SOFT_RESET_MASK | - UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK | - UVD_SOFT_RESET__LBSI_SOFT_RESET_MASK | - UVD_SOFT_RESET__RBC_SOFT_RESET_MASK | - UVD_SOFT_RESET__CSM_SOFT_RESET_MASK | - UVD_SOFT_RESET__CXW_SOFT_RESET_MASK | - UVD_SOFT_RESET__TAP_SOFT_RESET_MASK | - UVD_SOFT_RESET__LMI_UMC_SOFT_RESET_MASK); - mdelay(5); + for (k = 0; k < adev->uvd.num_uvd_inst; ++k) { + ring = &adev->uvd.inst[k].ring; + /* disable clock gating */ + WREG32_P(SOC15_REG_OFFSET(UVD, k, mmUVD_CGC_CTRL), 0, + ~UVD_CGC_CTRL__DYN_CLOCK_MODE_MASK); - /* initialize UVD memory controller */ - WREG32_SOC15(UVD, 0, mmUVD_LMI_CTRL, - (0x40 << UVD_LMI_CTRL__WRITE_CLEAN_TIMER__SHIFT) | - UVD_LMI_CTRL__WRITE_CLEAN_TIMER_EN_MASK | - UVD_LMI_CTRL__DATA_COHERENCY_EN_MASK | - UVD_LMI_CTRL__VCPU_DATA_COHERENCY_EN_MASK | - UVD_LMI_CTRL__REQ_MODE_MASK | - 0x00100000L); + /* disable interupt */ + WREG32_P(SOC15_REG_OFFSET(UVD, k, mmUVD_MASTINT_EN), 0, + ~UVD_MASTINT_EN__VCPU_EN_MASK); + + /* stall UMC and register bus before resetting VCPU */ + WREG32_P(SOC15_REG_OFFSET(UVD, k, mmUVD_LMI_CTRL2), + UVD_LMI_CTRL2__STALL_ARB_UMC_MASK, + ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK); + mdelay(1); + + /* put LMI, VCPU, RBC etc... into reset */ + WREG32_SOC15(UVD, k, mmUVD_SOFT_RESET, + UVD_SOFT_RESET__LMI_SOFT_RESET_MASK | + UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK | + UVD_SOFT_RESET__LBSI_SOFT_RESET_MASK | + UVD_SOFT_RESET__RBC_SOFT_RESET_MASK | + UVD_SOFT_RESET__CSM_SOFT_RESET_MASK | + UVD_SOFT_RESET__CXW_SOFT_RESET_MASK | + UVD_SOFT_RESET__TAP_SOFT_RESET_MASK | + UVD_SOFT_RESET__LMI_UMC_SOFT_RESET_MASK); + mdelay(5); + + /* initialize UVD memory controller */ + WREG32_SOC15(UVD, k, mmUVD_LMI_CTRL, + (0x40 << UVD_LMI_CTRL__WRITE_CLEAN_TIMER__SHIFT) | + UVD_LMI_CTRL__WRITE_CLEAN_TIMER_EN_MASK | + UVD_LMI_CTRL__DATA_COHERENCY_EN_MASK | + UVD_LMI_CTRL__VCPU_DATA_COHERENCY_EN_MASK | + UVD_LMI_CTRL__REQ_MODE_MASK | + 0x00100000L); #ifdef __BIG_ENDIAN - /* swap (8 in 32) RB and IB */ - lmi_swap_cntl = 0xa; - mp_swap_cntl = 0; + /* swap (8 in 32) RB and IB */ + lmi_swap_cntl = 0xa; + mp_swap_cntl = 0; #endif - WREG32_SOC15(UVD, 0, mmUVD_LMI_SWAP_CNTL, lmi_swap_cntl); - WREG32_SOC15(UVD, 0, mmUVD_MP_SWAP_CNTL, mp_swap_cntl); - - WREG32_SOC15(UVD, 0, mmUVD_MPC_SET_MUXA0, 0x40c2040); - WREG32_SOC15(UVD, 0, mmUVD_MPC_SET_MUXA1, 0x0); - WREG32_SOC15(UVD, 0, mmUVD_MPC_SET_MUXB0, 0x40c2040); - WREG32_SOC15(UVD, 0, mmUVD_MPC_SET_MUXB1, 0x0); - WREG32_SOC15(UVD, 0, mmUVD_MPC_SET_ALU, 0); - WREG32_SOC15(UVD, 0, mmUVD_MPC_SET_MUX, 0x88); - - /* take all subblocks out of reset, except VCPU */ - WREG32_SOC15(UVD, 0, mmUVD_SOFT_RESET, - UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK); - mdelay(5); + WREG32_SOC15(UVD, k, mmUVD_LMI_SWAP_CNTL, lmi_swap_cntl); + WREG32_SOC15(UVD, k, mmUVD_MP_SWAP_CNTL, mp_swap_cntl); - /* enable VCPU clock */ - WREG32_SOC15(UVD, 0, mmUVD_VCPU_CNTL, - UVD_VCPU_CNTL__CLK_EN_MASK); + WREG32_SOC15(UVD, k, mmUVD_MPC_SET_MUXA0, 0x40c2040); + WREG32_SOC15(UVD, k, mmUVD_MPC_SET_MUXA1, 0x0); + WREG32_SOC15(UVD, k, mmUVD_MPC_SET_MUXB0, 0x40c2040); + WREG32_SOC15(UVD, k, mmUVD_MPC_SET_MUXB1, 0x0); + WREG32_SOC15(UVD, k, mmUVD_MPC_SET_ALU, 0); + WREG32_SOC15(UVD, k, mmUVD_MPC_SET_MUX, 0x88); - /* enable UMC */ - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_CTRL2), 0, - ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK); + /* take all subblocks out of reset, except VCPU */ + WREG32_SOC15(UVD, k, mmUVD_SOFT_RESET, + UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK); + mdelay(5); - /* boot up the VCPU */ - WREG32_SOC15(UVD, 0, mmUVD_SOFT_RESET, 0); - mdelay(10); + /* enable VCPU clock */ + WREG32_SOC15(UVD, k, mmUVD_VCPU_CNTL, + UVD_VCPU_CNTL__CLK_EN_MASK); - for (i = 0; i < 10; ++i) { - uint32_t status; + /* enable UMC */ + WREG32_P(SOC15_REG_OFFSET(UVD, k, mmUVD_LMI_CTRL2), 0, + ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK); - for (j = 0; j < 100; ++j) { - status = RREG32_SOC15(UVD, 0, mmUVD_STATUS); + /* boot up the VCPU */ + WREG32_SOC15(UVD, k, mmUVD_SOFT_RESET, 0); + mdelay(10); + + for (i = 0; i < 10; ++i) { + uint32_t status; + + for (j = 0; j < 100; ++j) { + status = RREG32_SOC15(UVD, k, mmUVD_STATUS); + if (status & 2) + break; + mdelay(10); + } + r = 0; if (status & 2) break; + + DRM_ERROR("UVD(%d) not responding, trying to reset the VCPU!!!\n", k); + WREG32_P(SOC15_REG_OFFSET(UVD, k, mmUVD_SOFT_RESET), + UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK, + ~UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK); mdelay(10); + WREG32_P(SOC15_REG_OFFSET(UVD, k, mmUVD_SOFT_RESET), 0, + ~UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK); + mdelay(10); + r = -1; } - r = 0; - if (status & 2) - break; - - DRM_ERROR("UVD not responding, trying to reset the VCPU!!!\n"); - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_SOFT_RESET), - UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK, - ~UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK); - mdelay(10); - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_SOFT_RESET), 0, - ~UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK); - mdelay(10); - r = -1; - } - - if (r) { - DRM_ERROR("UVD not responding, giving up!!!\n"); - return r; - } - /* enable master interrupt */ - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_MASTINT_EN), - (UVD_MASTINT_EN__VCPU_EN_MASK|UVD_MASTINT_EN__SYS_EN_MASK), - ~(UVD_MASTINT_EN__VCPU_EN_MASK|UVD_MASTINT_EN__SYS_EN_MASK)); - - /* clear the bit 4 of UVD_STATUS */ - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_STATUS), 0, - ~(2 << UVD_STATUS__VCPU_REPORT__SHIFT)); - - /* force RBC into idle state */ - rb_bufsz = order_base_2(ring->ring_size); - tmp = REG_SET_FIELD(0, UVD_RBC_RB_CNTL, RB_BUFSZ, rb_bufsz); - tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_BLKSZ, 1); - tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_NO_FETCH, 1); - tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_WPTR_POLL_EN, 0); - tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_NO_UPDATE, 1); - tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_RPTR_WR_EN, 1); - WREG32_SOC15(UVD, 0, mmUVD_RBC_RB_CNTL, tmp); - - /* set the write pointer delay */ - WREG32_SOC15(UVD, 0, mmUVD_RBC_RB_WPTR_CNTL, 0); - - /* set the wb address */ - WREG32_SOC15(UVD, 0, mmUVD_RBC_RB_RPTR_ADDR, - (upper_32_bits(ring->gpu_addr) >> 2)); - - /* programm the RB_BASE for ring buffer */ - WREG32_SOC15(UVD, 0, mmUVD_LMI_RBC_RB_64BIT_BAR_LOW, - lower_32_bits(ring->gpu_addr)); - WREG32_SOC15(UVD, 0, mmUVD_LMI_RBC_RB_64BIT_BAR_HIGH, - upper_32_bits(ring->gpu_addr)); - - /* Initialize the ring buffer's read and write pointers */ - WREG32_SOC15(UVD, 0, mmUVD_RBC_RB_RPTR, 0); - - ring->wptr = RREG32_SOC15(UVD, 0, mmUVD_RBC_RB_RPTR); - WREG32_SOC15(UVD, 0, mmUVD_RBC_RB_WPTR, - lower_32_bits(ring->wptr)); - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_RBC_RB_CNTL), 0, - ~UVD_RBC_RB_CNTL__RB_NO_FETCH_MASK); - - ring = &adev->uvd.ring_enc[0]; - WREG32_SOC15(UVD, 0, mmUVD_RB_RPTR, lower_32_bits(ring->wptr)); - WREG32_SOC15(UVD, 0, mmUVD_RB_WPTR, lower_32_bits(ring->wptr)); - WREG32_SOC15(UVD, 0, mmUVD_RB_BASE_LO, ring->gpu_addr); - WREG32_SOC15(UVD, 0, mmUVD_RB_BASE_HI, upper_32_bits(ring->gpu_addr)); - WREG32_SOC15(UVD, 0, mmUVD_RB_SIZE, ring->ring_size / 4); + if (r) { + DRM_ERROR("UVD(%d) not responding, giving up!!!\n", k); + return r; + } + /* enable master interrupt */ + WREG32_P(SOC15_REG_OFFSET(UVD, k, mmUVD_MASTINT_EN), + (UVD_MASTINT_EN__VCPU_EN_MASK|UVD_MASTINT_EN__SYS_EN_MASK), + ~(UVD_MASTINT_EN__VCPU_EN_MASK|UVD_MASTINT_EN__SYS_EN_MASK)); - ring = &adev->uvd.ring_enc[1]; - WREG32_SOC15(UVD, 0, mmUVD_RB_RPTR2, lower_32_bits(ring->wptr)); - WREG32_SOC15(UVD, 0, mmUVD_RB_WPTR2, lower_32_bits(ring->wptr)); - WREG32_SOC15(UVD, 0, mmUVD_RB_BASE_LO2, ring->gpu_addr); - WREG32_SOC15(UVD, 0, mmUVD_RB_BASE_HI2, upper_32_bits(ring->gpu_addr)); - WREG32_SOC15(UVD, 0, mmUVD_RB_SIZE2, ring->ring_size / 4); + /* clear the bit 4 of UVD_STATUS */ + WREG32_P(SOC15_REG_OFFSET(UVD, k, mmUVD_STATUS), 0, + ~(2 << UVD_STATUS__VCPU_REPORT__SHIFT)); + /* force RBC into idle state */ + rb_bufsz = order_base_2(ring->ring_size); + tmp = REG_SET_FIELD(0, UVD_RBC_RB_CNTL, RB_BUFSZ, rb_bufsz); + tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_BLKSZ, 1); + tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_NO_FETCH, 1); + tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_WPTR_POLL_EN, 0); + tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_NO_UPDATE, 1); + tmp = REG_SET_FIELD(tmp, UVD_RBC_RB_CNTL, RB_RPTR_WR_EN, 1); + WREG32_SOC15(UVD, k, mmUVD_RBC_RB_CNTL, tmp); + + /* set the write pointer delay */ + WREG32_SOC15(UVD, k, mmUVD_RBC_RB_WPTR_CNTL, 0); + + /* set the wb address */ + WREG32_SOC15(UVD, k, mmUVD_RBC_RB_RPTR_ADDR, + (upper_32_bits(ring->gpu_addr) >> 2)); + + /* programm the RB_BASE for ring buffer */ + WREG32_SOC15(UVD, k, mmUVD_LMI_RBC_RB_64BIT_BAR_LOW, + lower_32_bits(ring->gpu_addr)); + WREG32_SOC15(UVD, k, mmUVD_LMI_RBC_RB_64BIT_BAR_HIGH, + upper_32_bits(ring->gpu_addr)); + + /* Initialize the ring buffer's read and write pointers */ + WREG32_SOC15(UVD, k, mmUVD_RBC_RB_RPTR, 0); + + ring->wptr = RREG32_SOC15(UVD, k, mmUVD_RBC_RB_RPTR); + WREG32_SOC15(UVD, k, mmUVD_RBC_RB_WPTR, + lower_32_bits(ring->wptr)); + + WREG32_P(SOC15_REG_OFFSET(UVD, k, mmUVD_RBC_RB_CNTL), 0, + ~UVD_RBC_RB_CNTL__RB_NO_FETCH_MASK); + + ring = &adev->uvd.inst[k].ring_enc[0]; + WREG32_SOC15(UVD, k, mmUVD_RB_RPTR, lower_32_bits(ring->wptr)); + WREG32_SOC15(UVD, k, mmUVD_RB_WPTR, lower_32_bits(ring->wptr)); + WREG32_SOC15(UVD, k, mmUVD_RB_BASE_LO, ring->gpu_addr); + WREG32_SOC15(UVD, k, mmUVD_RB_BASE_HI, upper_32_bits(ring->gpu_addr)); + WREG32_SOC15(UVD, k, mmUVD_RB_SIZE, ring->ring_size / 4); + + ring = &adev->uvd.inst[k].ring_enc[1]; + WREG32_SOC15(UVD, k, mmUVD_RB_RPTR2, lower_32_bits(ring->wptr)); + WREG32_SOC15(UVD, k, mmUVD_RB_WPTR2, lower_32_bits(ring->wptr)); + WREG32_SOC15(UVD, k, mmUVD_RB_BASE_LO2, ring->gpu_addr); + WREG32_SOC15(UVD, k, mmUVD_RB_BASE_HI2, upper_32_bits(ring->gpu_addr)); + WREG32_SOC15(UVD, k, mmUVD_RB_SIZE2, ring->ring_size / 4); + } return 0; } @@ -1053,26 +1087,30 @@ static int uvd_v7_0_start(struct amdgpu_device *adev) */ static void uvd_v7_0_stop(struct amdgpu_device *adev) { - /* force RBC into idle state */ - WREG32_SOC15(UVD, 0, mmUVD_RBC_RB_CNTL, 0x11010101); - - /* Stall UMC and register bus before resetting VCPU */ - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_CTRL2), - UVD_LMI_CTRL2__STALL_ARB_UMC_MASK, - ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK); - mdelay(1); - - /* put VCPU into reset */ - WREG32_SOC15(UVD, 0, mmUVD_SOFT_RESET, - UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK); - mdelay(5); + uint8_t i = 0; - /* disable VCPU clock */ - WREG32_SOC15(UVD, 0, mmUVD_VCPU_CNTL, 0x0); + for (i = 0; i < adev->uvd.num_uvd_inst; ++i) { + /* force RBC into idle state */ + WREG32_SOC15(UVD, i, mmUVD_RBC_RB_CNTL, 0x11010101); - /* Unstall UMC and register bus */ - WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_CTRL2), 0, - ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK); + /* Stall UMC and register bus before resetting VCPU */ + WREG32_P(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_CTRL2), + UVD_LMI_CTRL2__STALL_ARB_UMC_MASK, + ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK); + mdelay(1); + + /* put VCPU into reset */ + WREG32_SOC15(UVD, i, mmUVD_SOFT_RESET, + UVD_SOFT_RESET__VCPU_SOFT_RESET_MASK); + mdelay(5); + + /* disable VCPU clock */ + WREG32_SOC15(UVD, i, mmUVD_VCPU_CNTL, 0x0); + + /* Unstall UMC and register bus */ + WREG32_P(SOC15_REG_OFFSET(UVD, i, mmUVD_LMI_CTRL2), 0, + ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK); + } } /** @@ -1091,26 +1129,26 @@ static void uvd_v7_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 seq WARN_ON(flags & AMDGPU_FENCE_FLAG_64BIT); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_CONTEXT_ID), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_CONTEXT_ID), 0)); amdgpu_ring_write(ring, seq); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_DATA0), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_DATA0), 0)); amdgpu_ring_write(ring, addr & 0xffffffff); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_DATA1), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_DATA1), 0)); amdgpu_ring_write(ring, upper_32_bits(addr) & 0xff); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_CMD), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_CMD), 0)); amdgpu_ring_write(ring, 0); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_DATA0), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_DATA0), 0)); amdgpu_ring_write(ring, 0); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_DATA1), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_DATA1), 0)); amdgpu_ring_write(ring, 0); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_CMD), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_CMD), 0)); amdgpu_ring_write(ring, 2); } @@ -1135,6 +1173,16 @@ static void uvd_v7_0_enc_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, amdgpu_ring_write(ring, HEVC_ENC_CMD_TRAP); } +/** + * uvd_v7_0_ring_emit_hdp_flush - skip HDP flushing + * + * @ring: amdgpu_ring pointer + */ +static void uvd_v7_0_ring_emit_hdp_flush(struct amdgpu_ring *ring) +{ + /* The firmware doesn't seem to like touching registers at this point. */ +} + /** * uvd_v7_0_ring_test_ring - register write test * @@ -1149,30 +1197,30 @@ static int uvd_v7_0_ring_test_ring(struct amdgpu_ring *ring) unsigned i; int r; - WREG32_SOC15(UVD, 0, mmUVD_CONTEXT_ID, 0xCAFEDEAD); + WREG32_SOC15(UVD, ring->me, mmUVD_CONTEXT_ID, 0xCAFEDEAD); r = amdgpu_ring_alloc(ring, 3); if (r) { - DRM_ERROR("amdgpu: cp failed to lock ring %d (%d).\n", - ring->idx, r); + DRM_ERROR("amdgpu: (%d)cp failed to lock ring %d (%d).\n", + ring->me, ring->idx, r); return r; } amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_CONTEXT_ID), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_CONTEXT_ID), 0)); amdgpu_ring_write(ring, 0xDEADBEEF); amdgpu_ring_commit(ring); for (i = 0; i < adev->usec_timeout; i++) { - tmp = RREG32_SOC15(UVD, 0, mmUVD_CONTEXT_ID); + tmp = RREG32_SOC15(UVD, ring->me, mmUVD_CONTEXT_ID); if (tmp == 0xDEADBEEF) break; DRM_UDELAY(1); } if (i < adev->usec_timeout) { - DRM_DEBUG("ring test on %d succeeded in %d usecs\n", - ring->idx, i); + DRM_DEBUG("(%d)ring test on %d succeeded in %d usecs\n", + ring->me, ring->idx, i); } else { - DRM_ERROR("amdgpu: ring %d test failed (0x%08X)\n", - ring->idx, tmp); + DRM_ERROR("(%d)amdgpu: ring %d test failed (0x%08X)\n", + ring->me, ring->idx, tmp); r = -EINVAL; } return r; @@ -1193,17 +1241,17 @@ static void uvd_v7_0_ring_emit_ib(struct amdgpu_ring *ring, struct amdgpu_device *adev = ring->adev; amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_RBC_IB_VMID), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_LMI_RBC_IB_VMID), 0)); amdgpu_ring_write(ring, vmid); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_RBC_IB_64BIT_BAR_LOW), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_LMI_RBC_IB_64BIT_BAR_LOW), 0)); amdgpu_ring_write(ring, lower_32_bits(ib->gpu_addr)); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_RBC_IB_64BIT_BAR_HIGH), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_LMI_RBC_IB_64BIT_BAR_HIGH), 0)); amdgpu_ring_write(ring, upper_32_bits(ib->gpu_addr)); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_RBC_IB_SIZE), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_RBC_IB_SIZE), 0)); amdgpu_ring_write(ring, ib->length_dw); } @@ -1231,13 +1279,13 @@ static void uvd_v7_0_ring_emit_wreg(struct amdgpu_ring *ring, struct amdgpu_device *adev = ring->adev; amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_DATA0), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_DATA0), 0)); amdgpu_ring_write(ring, reg << 2); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_DATA1), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_DATA1), 0)); amdgpu_ring_write(ring, val); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_CMD), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_CMD), 0)); amdgpu_ring_write(ring, 8); } @@ -1247,16 +1295,16 @@ static void uvd_v7_0_ring_emit_reg_wait(struct amdgpu_ring *ring, uint32_t reg, struct amdgpu_device *adev = ring->adev; amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_DATA0), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_DATA0), 0)); amdgpu_ring_write(ring, reg << 2); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_DATA1), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_DATA1), 0)); amdgpu_ring_write(ring, val); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GP_SCRATCH8), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GP_SCRATCH8), 0)); amdgpu_ring_write(ring, mask); amdgpu_ring_write(ring, - PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_GPCOM_VCPU_CMD), 0)); + PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_GPCOM_VCPU_CMD), 0)); amdgpu_ring_write(ring, 12); } @@ -1277,12 +1325,15 @@ static void uvd_v7_0_ring_emit_vm_flush(struct amdgpu_ring *ring, static void uvd_v7_0_ring_insert_nop(struct amdgpu_ring *ring, uint32_t count) { - int i; struct amdgpu_device *adev = ring->adev; + int i; - for (i = 0; i < count; i++) - amdgpu_ring_write(ring, PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_NO_OP), 0)); + WARN_ON(ring->wptr % 2 || count % 2); + for (i = 0; i < count / 2; i++) { + amdgpu_ring_write(ring, PACKET0(SOC15_REG_OFFSET(UVD, ring->me, mmUVD_NO_OP), 0)); + amdgpu_ring_write(ring, 0); + } } static void uvd_v7_0_enc_ring_insert_end(struct amdgpu_ring *ring) @@ -1349,16 +1400,16 @@ static bool uvd_v7_0_check_soft_reset(void *handle) if (REG_GET_FIELD(tmp, SRBM_STATUS, UVD_RQ_PENDING) || REG_GET_FIELD(tmp, SRBM_STATUS, UVD_BUSY) || - (RREG32_SOC15(UVD, 0, mmUVD_STATUS) & + (RREG32_SOC15(UVD, ring->me, mmUVD_STATUS) & AMDGPU_UVD_STATUS_BUSY_MASK)) srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET, SOFT_RESET_UVD, 1); if (srbm_soft_reset) { - adev->uvd.srbm_soft_reset = srbm_soft_reset; + adev->uvd.inst[ring->me].srbm_soft_reset = srbm_soft_reset; return true; } else { - adev->uvd.srbm_soft_reset = 0; + adev->uvd.inst[ring->me].srbm_soft_reset = 0; return false; } } @@ -1367,7 +1418,7 @@ static int uvd_v7_0_pre_soft_reset(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - if (!adev->uvd.srbm_soft_reset) + if (!adev->uvd.inst[ring->me].srbm_soft_reset) return 0; uvd_v7_0_stop(adev); @@ -1379,9 +1430,9 @@ static int uvd_v7_0_soft_reset(void *handle) struct amdgpu_device *adev = (struct amdgpu_device *)handle; u32 srbm_soft_reset; - if (!adev->uvd.srbm_soft_reset) + if (!adev->uvd.inst[ring->me].srbm_soft_reset) return 0; - srbm_soft_reset = adev->uvd.srbm_soft_reset; + srbm_soft_reset = adev->uvd.inst[ring->me].srbm_soft_reset; if (srbm_soft_reset) { u32 tmp; @@ -1409,7 +1460,7 @@ static int uvd_v7_0_post_soft_reset(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - if (!adev->uvd.srbm_soft_reset) + if (!adev->uvd.inst[ring->me].srbm_soft_reset) return 0; mdelay(5); @@ -1431,17 +1482,32 @@ static int uvd_v7_0_process_interrupt(struct amdgpu_device *adev, struct amdgpu_irq_src *source, struct amdgpu_iv_entry *entry) { + uint32_t ip_instance; + + switch (entry->client_id) { + case SOC15_IH_CLIENTID_UVD: + ip_instance = 0; + break; + case SOC15_IH_CLIENTID_UVD1: + ip_instance = 1; + break; + default: + DRM_ERROR("Unhandled client id: %d\n", entry->client_id); + return 0; + } + DRM_DEBUG("IH: UVD TRAP\n"); + switch (entry->src_id) { case 124: - amdgpu_fence_process(&adev->uvd.ring); + amdgpu_fence_process(&adev->uvd.inst[ip_instance].ring); break; case 119: - amdgpu_fence_process(&adev->uvd.ring_enc[0]); + amdgpu_fence_process(&adev->uvd.inst[ip_instance].ring_enc[0]); break; case 120: if (!amdgpu_sriov_vf(adev)) - amdgpu_fence_process(&adev->uvd.ring_enc[1]); + amdgpu_fence_process(&adev->uvd.inst[ip_instance].ring_enc[1]); break; default: DRM_ERROR("Unhandled interrupt: %d %d\n", @@ -1457,9 +1523,9 @@ static void uvd_v7_0_set_sw_clock_gating(struct amdgpu_device *adev) { uint32_t data, data1, data2, suvd_flags; - data = RREG32_SOC15(UVD, 0, mmUVD_CGC_CTRL); - data1 = RREG32_SOC15(UVD, 0, mmUVD_SUVD_CGC_GATE); - data2 = RREG32_SOC15(UVD, 0, mmUVD_SUVD_CGC_CTRL); + data = RREG32_SOC15(UVD, ring->me, mmUVD_CGC_CTRL); + data1 = RREG32_SOC15(UVD, ring->me, mmUVD_SUVD_CGC_GATE); + data2 = RREG32_SOC15(UVD, ring->me, mmUVD_SUVD_CGC_CTRL); data &= ~(UVD_CGC_CTRL__CLK_OFF_DELAY_MASK | UVD_CGC_CTRL__CLK_GATE_DLY_TIMER_MASK); @@ -1503,18 +1569,18 @@ static void uvd_v7_0_set_sw_clock_gating(struct amdgpu_device *adev) UVD_SUVD_CGC_CTRL__SDB_MODE_MASK); data1 |= suvd_flags; - WREG32_SOC15(UVD, 0, mmUVD_CGC_CTRL, data); - WREG32_SOC15(UVD, 0, mmUVD_CGC_GATE, 0); - WREG32_SOC15(UVD, 0, mmUVD_SUVD_CGC_GATE, data1); - WREG32_SOC15(UVD, 0, mmUVD_SUVD_CGC_CTRL, data2); + WREG32_SOC15(UVD, ring->me, mmUVD_CGC_CTRL, data); + WREG32_SOC15(UVD, ring->me, mmUVD_CGC_GATE, 0); + WREG32_SOC15(UVD, ring->me, mmUVD_SUVD_CGC_GATE, data1); + WREG32_SOC15(UVD, ring->me, mmUVD_SUVD_CGC_CTRL, data2); } static void uvd_v7_0_set_hw_clock_gating(struct amdgpu_device *adev) { uint32_t data, data1, cgc_flags, suvd_flags; - data = RREG32_SOC15(UVD, 0, mmUVD_CGC_GATE); - data1 = RREG32_SOC15(UVD, 0, mmUVD_SUVD_CGC_GATE); + data = RREG32_SOC15(UVD, ring->me, mmUVD_CGC_GATE); + data1 = RREG32_SOC15(UVD, ring->me, mmUVD_SUVD_CGC_GATE); cgc_flags = UVD_CGC_GATE__SYS_MASK | UVD_CGC_GATE__UDEC_MASK | @@ -1546,8 +1612,8 @@ static void uvd_v7_0_set_hw_clock_gating(struct amdgpu_device *adev) data |= cgc_flags; data1 |= suvd_flags; - WREG32_SOC15(UVD, 0, mmUVD_CGC_GATE, data); - WREG32_SOC15(UVD, 0, mmUVD_SUVD_CGC_GATE, data1); + WREG32_SOC15(UVD, ring->me, mmUVD_CGC_GATE, data); + WREG32_SOC15(UVD, ring->me, mmUVD_SUVD_CGC_GATE, data1); } static void uvd_v7_0_set_bypass_mode(struct amdgpu_device *adev, bool enable) @@ -1606,7 +1672,7 @@ static int uvd_v7_0_set_powergating_state(void *handle, if (!(adev->pg_flags & AMD_PG_SUPPORT_UVD)) return 0; - WREG32_SOC15(UVD, 0, mmUVD_POWER_STATUS, UVD_POWER_STATUS__UVD_PG_EN_MASK); + WREG32_SOC15(UVD, ring->me, mmUVD_POWER_STATUS, UVD_POWER_STATUS__UVD_PG_EN_MASK); if (state == AMD_PG_STATE_GATE) { uvd_v7_0_stop(adev); @@ -1647,14 +1713,13 @@ const struct amd_ip_funcs uvd_v7_0_ip_funcs = { static const struct amdgpu_ring_funcs uvd_v7_0_ring_vm_funcs = { .type = AMDGPU_RING_TYPE_UVD, .align_mask = 0xf, - .nop = PACKET0(0x81ff, 0), .support_64bit_ptrs = false, .vmhub = AMDGPU_MMHUB, .get_rptr = uvd_v7_0_ring_get_rptr, .get_wptr = uvd_v7_0_ring_get_wptr, .set_wptr = uvd_v7_0_ring_set_wptr, .emit_frame_size = - 6 + 6 + /* hdp flush / invalidate */ + 6 + /* hdp invalidate */ SOC15_FLUSH_GPU_TLB_NUM_WREG * 6 + SOC15_FLUSH_GPU_TLB_NUM_REG_WAIT * 8 + 8 + /* uvd_v7_0_ring_emit_vm_flush */ @@ -1663,6 +1728,7 @@ static const struct amdgpu_ring_funcs uvd_v7_0_ring_vm_funcs = { .emit_ib = uvd_v7_0_ring_emit_ib, .emit_fence = uvd_v7_0_ring_emit_fence, .emit_vm_flush = uvd_v7_0_ring_emit_vm_flush, + .emit_hdp_flush = uvd_v7_0_ring_emit_hdp_flush, .test_ring = uvd_v7_0_ring_test_ring, .test_ib = amdgpu_uvd_ring_test_ib, .insert_nop = uvd_v7_0_ring_insert_nop, @@ -1671,6 +1737,7 @@ static const struct amdgpu_ring_funcs uvd_v7_0_ring_vm_funcs = { .end_use = amdgpu_uvd_ring_end_use, .emit_wreg = uvd_v7_0_ring_emit_wreg, .emit_reg_wait = uvd_v7_0_ring_emit_reg_wait, + .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, }; static const struct amdgpu_ring_funcs uvd_v7_0_enc_ring_vm_funcs = { @@ -1702,22 +1769,32 @@ static const struct amdgpu_ring_funcs uvd_v7_0_enc_ring_vm_funcs = { .end_use = amdgpu_uvd_ring_end_use, .emit_wreg = uvd_v7_0_enc_ring_emit_wreg, .emit_reg_wait = uvd_v7_0_enc_ring_emit_reg_wait, + .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, }; static void uvd_v7_0_set_ring_funcs(struct amdgpu_device *adev) { - adev->uvd.ring.funcs = &uvd_v7_0_ring_vm_funcs; - DRM_INFO("UVD is enabled in VM mode\n"); + int i; + + for (i = 0; i < adev->uvd.num_uvd_inst; i++) { + adev->uvd.inst[i].ring.funcs = &uvd_v7_0_ring_vm_funcs; + adev->uvd.inst[i].ring.me = i; + DRM_INFO("UVD(%d) is enabled in VM mode\n", i); + } } static void uvd_v7_0_set_enc_ring_funcs(struct amdgpu_device *adev) { - int i; + int i, j; - for (i = 0; i < adev->uvd.num_enc_rings; ++i) - adev->uvd.ring_enc[i].funcs = &uvd_v7_0_enc_ring_vm_funcs; + for (j = 0; j < adev->uvd.num_uvd_inst; j++) { + for (i = 0; i < adev->uvd.num_enc_rings; ++i) { + adev->uvd.inst[j].ring_enc[i].funcs = &uvd_v7_0_enc_ring_vm_funcs; + adev->uvd.inst[j].ring_enc[i].me = j; + } - DRM_INFO("UVD ENC is enabled in VM mode\n"); + DRM_INFO("UVD(%d) ENC is enabled in VM mode\n", j); + } } static const struct amdgpu_irq_src_funcs uvd_v7_0_irq_funcs = { @@ -1727,8 +1804,12 @@ static const struct amdgpu_irq_src_funcs uvd_v7_0_irq_funcs = { static void uvd_v7_0_set_irq_funcs(struct amdgpu_device *adev) { - adev->uvd.irq.num_types = adev->uvd.num_enc_rings + 1; - adev->uvd.irq.funcs = &uvd_v7_0_irq_funcs; + int i; + + for (i = 0; i < adev->uvd.num_uvd_inst; i++) { + adev->uvd.inst[i].irq.num_types = adev->uvd.num_enc_rings + 1; + adev->uvd.inst[i].irq.funcs = &uvd_v7_0_irq_funcs; + } } const struct amdgpu_ip_block_version uvd_v7_0_ip_block = diff --git a/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c b/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c index 428d1928e44e..0999c843f623 100644 --- a/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c +++ b/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c @@ -388,7 +388,8 @@ static unsigned vce_v3_0_get_harvest_config(struct amdgpu_device *adev) default: if ((adev->asic_type == CHIP_POLARIS10) || (adev->asic_type == CHIP_POLARIS11) || - (adev->asic_type == CHIP_POLARIS12)) + (adev->asic_type == CHIP_POLARIS12) || + (adev->asic_type == CHIP_VEGAM)) return AMDGPU_VCE_HARVEST_VCE1; return 0; @@ -467,8 +468,8 @@ static int vce_v3_0_hw_init(void *handle) struct amdgpu_device *adev = (struct amdgpu_device *)handle; vce_v3_0_override_vce_clock_gating(adev, true); - if (!(adev->flags & AMD_IS_APU)) - amdgpu_asic_set_vce_clocks(adev, 10000, 10000); + + amdgpu_asic_set_vce_clocks(adev, 10000, 10000); for (i = 0; i < adev->vce.num_rings; i++) adev->vce.ring[i].ready = false; diff --git a/drivers/gpu/drm/amd/amdgpu/vce_v4_0.c b/drivers/gpu/drm/amd/amdgpu/vce_v4_0.c index 73fd48d6c756..8fd1b742985a 100644 --- a/drivers/gpu/drm/amd/amdgpu/vce_v4_0.c +++ b/drivers/gpu/drm/amd/amdgpu/vce_v4_0.c @@ -1081,6 +1081,7 @@ static const struct amdgpu_ring_funcs vce_v4_0_ring_vm_funcs = { .end_use = amdgpu_vce_ring_end_use, .emit_wreg = vce_v4_0_emit_wreg, .emit_reg_wait = vce_v4_0_emit_reg_wait, + .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, }; static void vce_v4_0_set_ring_funcs(struct amdgpu_device *adev) diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v1_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v1_0.c index 8c132673bc79..29684c3ea4ef 100644 --- a/drivers/gpu/drm/amd/amdgpu/vcn_v1_0.c +++ b/drivers/gpu/drm/amd/amdgpu/vcn_v1_0.c @@ -35,7 +35,6 @@ #include "mmhub/mmhub_9_1_offset.h" #include "mmhub/mmhub_9_1_sh_mask.h" -static int vcn_v1_0_start(struct amdgpu_device *adev); static int vcn_v1_0_stop(struct amdgpu_device *adev); static void vcn_v1_0_set_dec_ring_funcs(struct amdgpu_device *adev); static void vcn_v1_0_set_enc_ring_funcs(struct amdgpu_device *adev); @@ -146,10 +145,6 @@ static int vcn_v1_0_hw_init(void *handle) struct amdgpu_ring *ring = &adev->vcn.ring_dec; int i, r; - r = vcn_v1_0_start(adev); - if (r) - goto done; - ring->ready = true; r = amdgpu_ring_test_ring(ring); if (r) { @@ -185,11 +180,9 @@ static int vcn_v1_0_hw_fini(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; struct amdgpu_ring *ring = &adev->vcn.ring_dec; - int r; - r = vcn_v1_0_stop(adev); - if (r) - return r; + if (RREG32_SOC15(VCN, 0, mmUVD_STATUS)) + vcn_v1_0_stop(adev); ring->ready = false; @@ -288,14 +281,14 @@ static void vcn_v1_0_mc_resume(struct amdgpu_device *adev) * * Disable clock gating for VCN block */ -static void vcn_v1_0_disable_clock_gating(struct amdgpu_device *adev, bool sw) +static void vcn_v1_0_disable_clock_gating(struct amdgpu_device *adev) { uint32_t data; /* JPEG disable CGC */ data = RREG32_SOC15(VCN, 0, mmJPEG_CGC_CTRL); - if (sw) + if (adev->cg_flags & AMD_CG_SUPPORT_VCN_MGCG) data |= 1 << JPEG_CGC_CTRL__DYN_CLOCK_MODE__SHIFT; else data &= ~JPEG_CGC_CTRL__DYN_CLOCK_MODE_MASK; @@ -310,7 +303,7 @@ static void vcn_v1_0_disable_clock_gating(struct amdgpu_device *adev, bool sw) /* UVD disable CGC */ data = RREG32_SOC15(VCN, 0, mmUVD_CGC_CTRL); - if (sw) + if (adev->cg_flags & AMD_CG_SUPPORT_VCN_MGCG) data |= 1 << UVD_CGC_CTRL__DYN_CLOCK_MODE__SHIFT; else data &= ~ UVD_CGC_CTRL__DYN_CLOCK_MODE_MASK; @@ -415,13 +408,13 @@ static void vcn_v1_0_disable_clock_gating(struct amdgpu_device *adev, bool sw) * * Enable clock gating for VCN block */ -static void vcn_v1_0_enable_clock_gating(struct amdgpu_device *adev, bool sw) +static void vcn_v1_0_enable_clock_gating(struct amdgpu_device *adev) { uint32_t data = 0; /* enable JPEG CGC */ data = RREG32_SOC15(VCN, 0, mmJPEG_CGC_CTRL); - if (sw) + if (adev->cg_flags & AMD_CG_SUPPORT_VCN_MGCG) data |= 1 << JPEG_CGC_CTRL__DYN_CLOCK_MODE__SHIFT; else data |= 0 << JPEG_CGC_CTRL__DYN_CLOCK_MODE__SHIFT; @@ -435,7 +428,7 @@ static void vcn_v1_0_enable_clock_gating(struct amdgpu_device *adev, bool sw) /* enable UVD CGC */ data = RREG32_SOC15(VCN, 0, mmUVD_CGC_CTRL); - if (sw) + if (adev->cg_flags & AMD_CG_SUPPORT_VCN_MGCG) data |= 1 << UVD_CGC_CTRL__DYN_CLOCK_MODE__SHIFT; else data |= 0 << UVD_CGC_CTRL__DYN_CLOCK_MODE__SHIFT; @@ -480,6 +473,94 @@ static void vcn_v1_0_enable_clock_gating(struct amdgpu_device *adev, bool sw) WREG32_SOC15(VCN, 0, mmUVD_SUVD_CGC_CTRL, data); } +static void vcn_1_0_disable_static_power_gating(struct amdgpu_device *adev) +{ + uint32_t data = 0; + int ret; + + if (adev->pg_flags & AMD_PG_SUPPORT_VCN) { + data = (1 << UVD_PGFSM_CONFIG__UVDM_PWR_CONFIG__SHIFT + | 1 << UVD_PGFSM_CONFIG__UVDU_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDF_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDC_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDB_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDIL_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDIR_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDTD_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDTE_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDE_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDW_PWR_CONFIG__SHIFT); + + WREG32_SOC15(VCN, 0, mmUVD_PGFSM_CONFIG, data); + SOC15_WAIT_ON_RREG(VCN, 0, mmUVD_PGFSM_STATUS, UVD_PGFSM_STATUS__UVDM_UVDU_PWR_ON, 0xFFFFFF, ret); + } else { + data = (1 << UVD_PGFSM_CONFIG__UVDM_PWR_CONFIG__SHIFT + | 1 << UVD_PGFSM_CONFIG__UVDU_PWR_CONFIG__SHIFT + | 1 << UVD_PGFSM_CONFIG__UVDF_PWR_CONFIG__SHIFT + | 1 << UVD_PGFSM_CONFIG__UVDC_PWR_CONFIG__SHIFT + | 1 << UVD_PGFSM_CONFIG__UVDB_PWR_CONFIG__SHIFT + | 1 << UVD_PGFSM_CONFIG__UVDIL_PWR_CONFIG__SHIFT + | 1 << UVD_PGFSM_CONFIG__UVDIR_PWR_CONFIG__SHIFT + | 1 << UVD_PGFSM_CONFIG__UVDTD_PWR_CONFIG__SHIFT + | 1 << UVD_PGFSM_CONFIG__UVDTE_PWR_CONFIG__SHIFT + | 1 << UVD_PGFSM_CONFIG__UVDE_PWR_CONFIG__SHIFT + | 1 << UVD_PGFSM_CONFIG__UVDW_PWR_CONFIG__SHIFT); + WREG32_SOC15(VCN, 0, mmUVD_PGFSM_CONFIG, data); + SOC15_WAIT_ON_RREG(VCN, 0, mmUVD_PGFSM_STATUS, 0, 0xFFFFFFFF, ret); + } + + /* polling UVD_PGFSM_STATUS to confirm UVDM_PWR_STATUS , UVDU_PWR_STATUS are 0 (power on) */ + + data = RREG32_SOC15(VCN, 0, mmUVD_POWER_STATUS); + data &= ~0x103; + if (adev->pg_flags & AMD_PG_SUPPORT_VCN) + data |= UVD_PGFSM_CONFIG__UVDM_UVDU_PWR_ON | UVD_POWER_STATUS__UVD_PG_EN_MASK; + + WREG32_SOC15(VCN, 0, mmUVD_POWER_STATUS, data); +} + +static void vcn_1_0_enable_static_power_gating(struct amdgpu_device *adev) +{ + uint32_t data = 0; + int ret; + + if (adev->pg_flags & AMD_PG_SUPPORT_VCN) { + /* Before power off, this indicator has to be turned on */ + data = RREG32_SOC15(VCN, 0, mmUVD_POWER_STATUS); + data &= ~UVD_POWER_STATUS__UVD_POWER_STATUS_MASK; + data |= UVD_POWER_STATUS__UVD_POWER_STATUS_TILES_OFF; + WREG32_SOC15(VCN, 0, mmUVD_POWER_STATUS, data); + + + data = (2 << UVD_PGFSM_CONFIG__UVDM_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDU_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDF_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDC_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDB_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDIL_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDIR_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDTD_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDTE_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDE_PWR_CONFIG__SHIFT + | 2 << UVD_PGFSM_CONFIG__UVDW_PWR_CONFIG__SHIFT); + + WREG32_SOC15(VCN, 0, mmUVD_PGFSM_CONFIG, data); + + data = (2 << UVD_PGFSM_STATUS__UVDM_PWR_STATUS__SHIFT + | 2 << UVD_PGFSM_STATUS__UVDU_PWR_STATUS__SHIFT + | 2 << UVD_PGFSM_STATUS__UVDF_PWR_STATUS__SHIFT + | 2 << UVD_PGFSM_STATUS__UVDC_PWR_STATUS__SHIFT + | 2 << UVD_PGFSM_STATUS__UVDB_PWR_STATUS__SHIFT + | 2 << UVD_PGFSM_STATUS__UVDIL_PWR_STATUS__SHIFT + | 2 << UVD_PGFSM_STATUS__UVDIR_PWR_STATUS__SHIFT + | 2 << UVD_PGFSM_STATUS__UVDTD_PWR_STATUS__SHIFT + | 2 << UVD_PGFSM_STATUS__UVDTE_PWR_STATUS__SHIFT + | 2 << UVD_PGFSM_STATUS__UVDE_PWR_STATUS__SHIFT + | 2 << UVD_PGFSM_STATUS__UVDW_PWR_STATUS__SHIFT); + SOC15_WAIT_ON_RREG(VCN, 0, mmUVD_PGFSM_STATUS, data, 0xFFFFFFFF, ret); + } +} + /** * vcn_v1_0_start - start VCN block * @@ -499,8 +580,9 @@ static int vcn_v1_0_start(struct amdgpu_device *adev) vcn_v1_0_mc_resume(adev); + vcn_1_0_disable_static_power_gating(adev); /* disable clock gating */ - vcn_v1_0_disable_clock_gating(adev, true); + vcn_v1_0_disable_clock_gating(adev); /* disable interupt */ WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_MASTINT_EN), 0, @@ -680,16 +762,45 @@ static int vcn_v1_0_stop(struct amdgpu_device *adev) WREG32_P(SOC15_REG_OFFSET(UVD, 0, mmUVD_LMI_CTRL2), 0, ~UVD_LMI_CTRL2__STALL_ARB_UMC_MASK); - /* enable clock gating */ - vcn_v1_0_enable_clock_gating(adev, true); + WREG32_SOC15(VCN, 0, mmUVD_STATUS, 0); + vcn_v1_0_enable_clock_gating(adev); + vcn_1_0_enable_static_power_gating(adev); return 0; } +static bool vcn_v1_0_is_idle(void *handle) +{ + struct amdgpu_device *adev = (struct amdgpu_device *)handle; + + return (RREG32_SOC15(VCN, 0, mmUVD_STATUS) == 0x2); +} + +static int vcn_v1_0_wait_for_idle(void *handle) +{ + struct amdgpu_device *adev = (struct amdgpu_device *)handle; + int ret = 0; + + SOC15_WAIT_ON_RREG(VCN, 0, mmUVD_STATUS, 0x2, 0x2, ret); + + return ret; +} + static int vcn_v1_0_set_clockgating_state(void *handle, enum amd_clockgating_state state) { - /* needed for driver unload*/ + struct amdgpu_device *adev = (struct amdgpu_device *)handle; + bool enable = (state == AMD_CG_STATE_GATE) ? true : false; + + if (enable) { + /* wait for STATUS to clear */ + if (vcn_v1_0_is_idle(handle)) + return -EBUSY; + vcn_v1_0_enable_clock_gating(adev); + } else { + /* disable HW gating and enable Sw gating */ + vcn_v1_0_disable_clock_gating(adev); + } return 0; } @@ -1048,16 +1159,36 @@ static int vcn_v1_0_process_interrupt(struct amdgpu_device *adev, return 0; } -static void vcn_v1_0_ring_insert_nop(struct amdgpu_ring *ring, uint32_t count) +static void vcn_v1_0_dec_ring_insert_nop(struct amdgpu_ring *ring, uint32_t count) { - int i; struct amdgpu_device *adev = ring->adev; + int i; - for (i = 0; i < count; i++) - amdgpu_ring_write(ring, PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_NO_OP), 0)); + WARN_ON(ring->wptr % 2 || count % 2); + for (i = 0; i < count / 2; i++) { + amdgpu_ring_write(ring, PACKET0(SOC15_REG_OFFSET(UVD, 0, mmUVD_NO_OP), 0)); + amdgpu_ring_write(ring, 0); + } } +static int vcn_v1_0_set_powergating_state(void *handle, + enum amd_powergating_state state) +{ + /* This doesn't actually powergate the VCN block. + * That's done in the dpm code via the SMC. This + * just re-inits the block as necessary. The actual + * gating still happens in the dpm code. We should + * revisit this when there is a cleaner line between + * the smc and the hw blocks + */ + struct amdgpu_device *adev = (struct amdgpu_device *)handle; + + if (state == AMD_PG_STATE_GATE) + return vcn_v1_0_stop(adev); + else + return vcn_v1_0_start(adev); +} static const struct amd_ip_funcs vcn_v1_0_ip_funcs = { .name = "vcn_v1_0", @@ -1069,20 +1200,19 @@ static const struct amd_ip_funcs vcn_v1_0_ip_funcs = { .hw_fini = vcn_v1_0_hw_fini, .suspend = vcn_v1_0_suspend, .resume = vcn_v1_0_resume, - .is_idle = NULL /* vcn_v1_0_is_idle */, - .wait_for_idle = NULL /* vcn_v1_0_wait_for_idle */, + .is_idle = vcn_v1_0_is_idle, + .wait_for_idle = vcn_v1_0_wait_for_idle, .check_soft_reset = NULL /* vcn_v1_0_check_soft_reset */, .pre_soft_reset = NULL /* vcn_v1_0_pre_soft_reset */, .soft_reset = NULL /* vcn_v1_0_soft_reset */, .post_soft_reset = NULL /* vcn_v1_0_post_soft_reset */, .set_clockgating_state = vcn_v1_0_set_clockgating_state, - .set_powergating_state = NULL /* vcn_v1_0_set_powergating_state */, + .set_powergating_state = vcn_v1_0_set_powergating_state, }; static const struct amdgpu_ring_funcs vcn_v1_0_dec_ring_vm_funcs = { .type = AMDGPU_RING_TYPE_VCN_DEC, .align_mask = 0xf, - .nop = PACKET0(0x81ff, 0), .support_64bit_ptrs = false, .vmhub = AMDGPU_MMHUB, .get_rptr = vcn_v1_0_dec_ring_get_rptr, @@ -1101,7 +1231,7 @@ static const struct amdgpu_ring_funcs vcn_v1_0_dec_ring_vm_funcs = { .emit_vm_flush = vcn_v1_0_dec_ring_emit_vm_flush, .test_ring = amdgpu_vcn_dec_ring_test_ring, .test_ib = amdgpu_vcn_dec_ring_test_ib, - .insert_nop = vcn_v1_0_ring_insert_nop, + .insert_nop = vcn_v1_0_dec_ring_insert_nop, .insert_start = vcn_v1_0_dec_ring_insert_start, .insert_end = vcn_v1_0_dec_ring_insert_end, .pad_ib = amdgpu_ring_generic_pad_ib, @@ -1109,6 +1239,7 @@ static const struct amdgpu_ring_funcs vcn_v1_0_dec_ring_vm_funcs = { .end_use = amdgpu_vcn_ring_end_use, .emit_wreg = vcn_v1_0_dec_ring_emit_wreg, .emit_reg_wait = vcn_v1_0_dec_ring_emit_reg_wait, + .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, }; static const struct amdgpu_ring_funcs vcn_v1_0_enc_ring_vm_funcs = { @@ -1139,6 +1270,7 @@ static const struct amdgpu_ring_funcs vcn_v1_0_enc_ring_vm_funcs = { .end_use = amdgpu_vcn_ring_end_use, .emit_wreg = vcn_v1_0_enc_ring_emit_wreg, .emit_reg_wait = vcn_v1_0_enc_ring_emit_reg_wait, + .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, }; static void vcn_v1_0_set_dec_ring_funcs(struct amdgpu_device *adev) diff --git a/drivers/gpu/drm/amd/amdgpu/vega20_reg_init.c b/drivers/gpu/drm/amd/amdgpu/vega20_reg_init.c new file mode 100644 index 000000000000..52778de93ab0 --- /dev/null +++ b/drivers/gpu/drm/amd/amdgpu/vega20_reg_init.c @@ -0,0 +1,53 @@ +/* + * Copyright 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ +#include "amdgpu.h" +#include "soc15.h" + +#include "soc15_common.h" +#include "soc15_hw_ip.h" +#include "vega20_ip_offset.h" + +int vega20_reg_base_init(struct amdgpu_device *adev) +{ + /* HW has more IP blocks, only initialized the blocke beend by our driver */ + uint32_t i; + for (i = 0 ; i < MAX_INSTANCE ; ++i) { + adev->reg_offset[GC_HWIP][i] = (uint32_t *)(&(GC_BASE.instance[i])); + adev->reg_offset[HDP_HWIP][i] = (uint32_t *)(&(HDP_BASE.instance[i])); + adev->reg_offset[MMHUB_HWIP][i] = (uint32_t *)(&(MMHUB_BASE.instance[i])); + adev->reg_offset[ATHUB_HWIP][i] = (uint32_t *)(&(ATHUB_BASE.instance[i])); + adev->reg_offset[NBIO_HWIP][i] = (uint32_t *)(&(NBIO_BASE.instance[i])); + adev->reg_offset[MP0_HWIP][i] = (uint32_t *)(&(MP0_BASE.instance[i])); + adev->reg_offset[UVD_HWIP][i] = (uint32_t *)(&(UVD_BASE.instance[i])); + adev->reg_offset[VCE_HWIP][i] = (uint32_t *)(&(VCE_BASE.instance[i])); + adev->reg_offset[DF_HWIP][i] = (uint32_t *)(&(DF_BASE.instance[i])); + adev->reg_offset[DCE_HWIP][i] = (uint32_t *)(&(DCE_BASE.instance[i])); + adev->reg_offset[OSSSYS_HWIP][i] = (uint32_t *)(&(OSSSYS_BASE.instance[i])); + adev->reg_offset[SDMA0_HWIP][i] = (uint32_t *)(&(SDMA0_BASE.instance[i])); + adev->reg_offset[SDMA1_HWIP][i] = (uint32_t *)(&(SDMA1_BASE.instance[i])); + adev->reg_offset[SMUIO_HWIP][i] = (uint32_t *)(&(SMUIO_BASE.instance[i])); + } + return 0; +} + + diff --git a/drivers/gpu/drm/amd/amdgpu/vi.c b/drivers/gpu/drm/amd/amdgpu/vi.c index 126f1276d347..4ac1288ab7df 100644 --- a/drivers/gpu/drm/amd/amdgpu/vi.c +++ b/drivers/gpu/drm/amd/amdgpu/vi.c @@ -305,9 +305,10 @@ static void vi_init_golden_registers(struct amdgpu_device *adev) stoney_mgcg_cgcg_init, ARRAY_SIZE(stoney_mgcg_cgcg_init)); break; - case CHIP_POLARIS11: case CHIP_POLARIS10: + case CHIP_POLARIS11: case CHIP_POLARIS12: + case CHIP_VEGAM: default: break; } @@ -728,33 +729,59 @@ static int vi_set_uvd_clock(struct amdgpu_device *adev, u32 clock, return r; tmp = RREG32_SMC(cntl_reg); - tmp &= ~(CG_DCLK_CNTL__DCLK_DIR_CNTL_EN_MASK | - CG_DCLK_CNTL__DCLK_DIVIDER_MASK); + + if (adev->flags & AMD_IS_APU) + tmp &= ~CG_DCLK_CNTL__DCLK_DIVIDER_MASK; + else + tmp &= ~(CG_DCLK_CNTL__DCLK_DIR_CNTL_EN_MASK | + CG_DCLK_CNTL__DCLK_DIVIDER_MASK); tmp |= dividers.post_divider; WREG32_SMC(cntl_reg, tmp); for (i = 0; i < 100; i++) { - if (RREG32_SMC(status_reg) & CG_DCLK_STATUS__DCLK_STATUS_MASK) - break; + tmp = RREG32_SMC(status_reg); + if (adev->flags & AMD_IS_APU) { + if (tmp & 0x10000) + break; + } else { + if (tmp & CG_DCLK_STATUS__DCLK_STATUS_MASK) + break; + } mdelay(10); } if (i == 100) return -ETIMEDOUT; - return 0; } +#define ixGNB_CLK1_DFS_CNTL 0xD82200F0 +#define ixGNB_CLK1_STATUS 0xD822010C +#define ixGNB_CLK2_DFS_CNTL 0xD8220110 +#define ixGNB_CLK2_STATUS 0xD822012C +#define ixGNB_CLK3_DFS_CNTL 0xD8220130 +#define ixGNB_CLK3_STATUS 0xD822014C + static int vi_set_uvd_clocks(struct amdgpu_device *adev, u32 vclk, u32 dclk) { int r; - r = vi_set_uvd_clock(adev, vclk, ixCG_VCLK_CNTL, ixCG_VCLK_STATUS); - if (r) - return r; + if (adev->flags & AMD_IS_APU) { + r = vi_set_uvd_clock(adev, vclk, ixGNB_CLK2_DFS_CNTL, ixGNB_CLK2_STATUS); + if (r) + return r; - r = vi_set_uvd_clock(adev, dclk, ixCG_DCLK_CNTL, ixCG_DCLK_STATUS); - if (r) - return r; + r = vi_set_uvd_clock(adev, dclk, ixGNB_CLK1_DFS_CNTL, ixGNB_CLK1_STATUS); + if (r) + return r; + } else { + r = vi_set_uvd_clock(adev, vclk, ixCG_VCLK_CNTL, ixCG_VCLK_STATUS); + if (r) + return r; + + r = vi_set_uvd_clock(adev, dclk, ixCG_DCLK_CNTL, ixCG_DCLK_STATUS); + if (r) + return r; + } return 0; } @@ -764,6 +791,22 @@ static int vi_set_vce_clocks(struct amdgpu_device *adev, u32 evclk, u32 ecclk) int r, i; struct atom_clock_dividers dividers; u32 tmp; + u32 reg_ctrl; + u32 reg_status; + u32 status_mask; + u32 reg_mask; + + if (adev->flags & AMD_IS_APU) { + reg_ctrl = ixGNB_CLK3_DFS_CNTL; + reg_status = ixGNB_CLK3_STATUS; + status_mask = 0x00010000; + reg_mask = CG_ECLK_CNTL__ECLK_DIVIDER_MASK; + } else { + reg_ctrl = ixCG_ECLK_CNTL; + reg_status = ixCG_ECLK_STATUS; + status_mask = CG_ECLK_STATUS__ECLK_STATUS_MASK; + reg_mask = CG_ECLK_CNTL__ECLK_DIR_CNTL_EN_MASK | CG_ECLK_CNTL__ECLK_DIVIDER_MASK; + } r = amdgpu_atombios_get_clock_dividers(adev, COMPUTE_GPUCLK_INPUT_FLAG_DEFAULT_GPUCLK, @@ -772,24 +815,25 @@ static int vi_set_vce_clocks(struct amdgpu_device *adev, u32 evclk, u32 ecclk) return r; for (i = 0; i < 100; i++) { - if (RREG32_SMC(ixCG_ECLK_STATUS) & CG_ECLK_STATUS__ECLK_STATUS_MASK) + if (RREG32_SMC(reg_status) & status_mask) break; mdelay(10); } + if (i == 100) return -ETIMEDOUT; - tmp = RREG32_SMC(ixCG_ECLK_CNTL); - tmp &= ~(CG_ECLK_CNTL__ECLK_DIR_CNTL_EN_MASK | - CG_ECLK_CNTL__ECLK_DIVIDER_MASK); + tmp = RREG32_SMC(reg_ctrl); + tmp &= ~reg_mask; tmp |= dividers.post_divider; - WREG32_SMC(ixCG_ECLK_CNTL, tmp); + WREG32_SMC(reg_ctrl, tmp); for (i = 0; i < 100; i++) { - if (RREG32_SMC(ixCG_ECLK_STATUS) & CG_ECLK_STATUS__ECLK_STATUS_MASK) + if (RREG32_SMC(reg_status) & status_mask) break; mdelay(10); } + if (i == 100) return -ETIMEDOUT; @@ -876,6 +920,27 @@ static void vi_invalidate_hdp(struct amdgpu_device *adev, } } +static bool vi_need_full_reset(struct amdgpu_device *adev) +{ + switch (adev->asic_type) { + case CHIP_CARRIZO: + case CHIP_STONEY: + /* CZ has hang issues with full reset at the moment */ + return false; + case CHIP_FIJI: + case CHIP_TONGA: + /* XXX: soft reset should work on fiji and tonga */ + return true; + case CHIP_POLARIS10: + case CHIP_POLARIS11: + case CHIP_POLARIS12: + case CHIP_TOPAZ: + default: + /* change this when we support soft reset */ + return true; + } +} + static const struct amdgpu_asic_funcs vi_asic_funcs = { .read_disabled_bios = &vi_read_disabled_bios, @@ -889,6 +954,7 @@ static const struct amdgpu_asic_funcs vi_asic_funcs = .get_config_memsize = &vi_get_config_memsize, .flush_hdp = &vi_flush_hdp, .invalidate_hdp = &vi_invalidate_hdp, + .need_full_reset = &vi_need_full_reset, }; #define CZ_REV_BRISTOL(rev) \ @@ -1031,6 +1097,30 @@ static int vi_common_early_init(void *handle) adev->pg_flags = 0; adev->external_rev_id = adev->rev_id + 0x64; break; + case CHIP_VEGAM: + adev->cg_flags = 0; + /*AMD_CG_SUPPORT_GFX_MGCG | + AMD_CG_SUPPORT_GFX_RLC_LS | + AMD_CG_SUPPORT_GFX_CP_LS | + AMD_CG_SUPPORT_GFX_CGCG | + AMD_CG_SUPPORT_GFX_CGLS | + AMD_CG_SUPPORT_GFX_3D_CGCG | + AMD_CG_SUPPORT_GFX_3D_CGLS | + AMD_CG_SUPPORT_SDMA_MGCG | + AMD_CG_SUPPORT_SDMA_LS | + AMD_CG_SUPPORT_BIF_MGCG | + AMD_CG_SUPPORT_BIF_LS | + AMD_CG_SUPPORT_HDP_MGCG | + AMD_CG_SUPPORT_HDP_LS | + AMD_CG_SUPPORT_ROM_MGCG | + AMD_CG_SUPPORT_MC_MGCG | + AMD_CG_SUPPORT_MC_LS | + AMD_CG_SUPPORT_DRM_LS | + AMD_CG_SUPPORT_UVD_MGCG | + AMD_CG_SUPPORT_VCE_MGCG;*/ + adev->pg_flags = 0; + adev->external_rev_id = adev->rev_id + 0x6E; + break; case CHIP_CARRIZO: adev->cg_flags = AMD_CG_SUPPORT_UVD_MGCG | AMD_CG_SUPPORT_GFX_MGCG | @@ -1422,6 +1512,7 @@ static int vi_common_set_clockgating_state(void *handle, case CHIP_POLARIS10: case CHIP_POLARIS11: case CHIP_POLARIS12: + case CHIP_VEGAM: vi_common_set_clockgating_state_by_smu(adev, state); default: break; @@ -1551,9 +1642,10 @@ int vi_set_ip_blocks(struct amdgpu_device *adev) amdgpu_device_ip_block_add(adev, &vce_v3_0_ip_block); } break; - case CHIP_POLARIS11: case CHIP_POLARIS10: + case CHIP_POLARIS11: case CHIP_POLARIS12: + case CHIP_VEGAM: amdgpu_device_ip_block_add(adev, &vi_common_ip_block); amdgpu_device_ip_block_add(adev, &gmc_v8_1_ip_block); amdgpu_device_ip_block_add(adev, &tonga_ih_ip_block); diff --git a/drivers/gpu/drm/amd/amdkfd/Makefile b/drivers/gpu/drm/amd/amdkfd/Makefile index 0d0242240c47..ffd096fffc1c 100644 --- a/drivers/gpu/drm/amd/amdkfd/Makefile +++ b/drivers/gpu/drm/amd/amdkfd/Makefile @@ -30,12 +30,14 @@ amdkfd-y := kfd_module.o kfd_device.o kfd_chardev.o kfd_topology.o \ kfd_pasid.o kfd_doorbell.o kfd_flat_memory.o \ kfd_process.o kfd_queue.o kfd_mqd_manager.o \ kfd_mqd_manager_cik.o kfd_mqd_manager_vi.o \ + kfd_mqd_manager_v9.o \ kfd_kernel_queue.o kfd_kernel_queue_cik.o \ - kfd_kernel_queue_vi.o kfd_packet_manager.o \ - kfd_process_queue_manager.o kfd_device_queue_manager.o \ - kfd_device_queue_manager_cik.o kfd_device_queue_manager_vi.o \ + kfd_kernel_queue_vi.o kfd_kernel_queue_v9.o \ + kfd_packet_manager.o kfd_process_queue_manager.o \ + kfd_device_queue_manager.o kfd_device_queue_manager_cik.o \ + kfd_device_queue_manager_vi.o kfd_device_queue_manager_v9.o \ kfd_interrupt.o kfd_events.o cik_event_interrupt.o \ - kfd_dbgdev.o kfd_dbgmgr.o kfd_crat.o + kfd_int_process_v9.o kfd_dbgdev.o kfd_dbgmgr.o kfd_crat.o ifneq ($(CONFIG_AMD_IOMMU_V2),) amdkfd-y += kfd_iommu.o diff --git a/drivers/gpu/drm/amd/amdkfd/cik_event_interrupt.c b/drivers/gpu/drm/amd/amdkfd/cik_event_interrupt.c index 3d5ccb3755d4..49df6c791cfc 100644 --- a/drivers/gpu/drm/amd/amdkfd/cik_event_interrupt.c +++ b/drivers/gpu/drm/amd/amdkfd/cik_event_interrupt.c @@ -27,18 +27,28 @@ static bool cik_event_interrupt_isr(struct kfd_dev *dev, const uint32_t *ih_ring_entry) { - unsigned int pasid; const struct cik_ih_ring_entry *ihre = (const struct cik_ih_ring_entry *)ih_ring_entry; + unsigned int vmid, pasid; + + /* Only handle interrupts from KFD VMIDs */ + vmid = (ihre->ring_id & 0x0000ff00) >> 8; + if (vmid < dev->vm_info.first_vmid_kfd || + vmid > dev->vm_info.last_vmid_kfd) + return 0; + /* If there is no valid PASID, it's likely a firmware bug */ pasid = (ihre->ring_id & 0xffff0000) >> 16; + if (WARN_ONCE(pasid == 0, "FW bug: No PASID in KFD interrupt")) + return 0; - /* Do not process in ISR, just request it to be forwarded to WQ. */ - return (pasid != 0) && - (ihre->source_id == CIK_INTSRC_CP_END_OF_PIPE || + /* Interrupt types we care about: various signals and faults. + * They will be forwarded to a work queue (see below). + */ + return ihre->source_id == CIK_INTSRC_CP_END_OF_PIPE || ihre->source_id == CIK_INTSRC_SDMA_TRAP || ihre->source_id == CIK_INTSRC_SQ_INTERRUPT_MSG || - ihre->source_id == CIK_INTSRC_CP_BAD_OPCODE); + ihre->source_id == CIK_INTSRC_CP_BAD_OPCODE; } static void cik_event_interrupt_wq(struct kfd_dev *dev, diff --git a/drivers/gpu/drm/amd/amdkfd/cik_regs.h b/drivers/gpu/drm/amd/amdkfd/cik_regs.h index 48769d12dd7b..37ce6dd65391 100644 --- a/drivers/gpu/drm/amd/amdkfd/cik_regs.h +++ b/drivers/gpu/drm/amd/amdkfd/cik_regs.h @@ -33,7 +33,8 @@ #define APE1_MTYPE(x) ((x) << 7) /* valid for both DEFAULT_MTYPE and APE1_MTYPE */ -#define MTYPE_CACHED 0 +#define MTYPE_CACHED_NV 0 +#define MTYPE_CACHED 1 #define MTYPE_NONCACHED 3 #define DEFAULT_CP_HQD_PERSISTENT_STATE (0x33U << 8) diff --git a/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler.h b/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler.h new file mode 100644 index 000000000000..f68aef02fc1f --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler.h @@ -0,0 +1,560 @@ +/* + * Copyright 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +static const uint32_t cwsr_trap_gfx8_hex[] = { + 0xbf820001, 0xbf820125, + 0xb8f4f802, 0x89748674, + 0xb8f5f803, 0x8675ff75, + 0x00000400, 0xbf850011, + 0xc00a1e37, 0x00000000, + 0xbf8c007f, 0x87777978, + 0xbf840002, 0xb974f802, + 0xbe801d78, 0xb8f5f803, + 0x8675ff75, 0x000001ff, + 0xbf850002, 0x80708470, + 0x82718071, 0x8671ff71, + 0x0000ffff, 0xb974f802, + 0xbe801f70, 0xb8f5f803, + 0x8675ff75, 0x00000100, + 0xbf840006, 0xbefa0080, + 0xb97a0203, 0x8671ff71, + 0x0000ffff, 0x80f08870, + 0x82f18071, 0xbefa0080, + 0xb97a0283, 0xbef60068, + 0xbef70069, 0xb8fa1c07, + 0x8e7a9c7a, 0x87717a71, + 0xb8fa03c7, 0x8e7a9b7a, + 0x87717a71, 0xb8faf807, + 0x867aff7a, 0x00007fff, + 0xb97af807, 0xbef2007e, + 0xbef3007f, 0xbefe0180, + 0xbf900004, 0x877a8474, + 0xb97af802, 0xbf8e0002, + 0xbf88fffe, 0xbef8007e, + 0x8679ff7f, 0x0000ffff, + 0x8779ff79, 0x00040000, + 0xbefa0080, 0xbefb00ff, + 0x00807fac, 0x867aff7f, + 0x08000000, 0x8f7a837a, + 0x877b7a7b, 0x867aff7f, + 0x70000000, 0x8f7a817a, + 0x877b7a7b, 0xbeef007c, + 0xbeee0080, 0xb8ee2a05, + 0x806e816e, 0x8e6e8a6e, + 0xb8fa1605, 0x807a817a, + 0x8e7a867a, 0x806e7a6e, + 0xbefa0084, 0xbefa00ff, + 0x01000000, 0xbefe007c, + 0xbefc006e, 0xc0611bfc, + 0x0000007c, 0x806e846e, + 0xbefc007e, 0xbefe007c, + 0xbefc006e, 0xc0611c3c, + 0x0000007c, 0x806e846e, + 0xbefc007e, 0xbefe007c, + 0xbefc006e, 0xc0611c7c, + 0x0000007c, 0x806e846e, + 0xbefc007e, 0xbefe007c, + 0xbefc006e, 0xc0611cbc, + 0x0000007c, 0x806e846e, + 0xbefc007e, 0xbefe007c, + 0xbefc006e, 0xc0611cfc, + 0x0000007c, 0x806e846e, + 0xbefc007e, 0xbefe007c, + 0xbefc006e, 0xc0611d3c, + 0x0000007c, 0x806e846e, + 0xbefc007e, 0xb8f5f803, + 0xbefe007c, 0xbefc006e, + 0xc0611d7c, 0x0000007c, + 0x806e846e, 0xbefc007e, + 0xbefe007c, 0xbefc006e, + 0xc0611dbc, 0x0000007c, + 0x806e846e, 0xbefc007e, + 0xbefe007c, 0xbefc006e, + 0xc0611dfc, 0x0000007c, + 0x806e846e, 0xbefc007e, + 0xb8eff801, 0xbefe007c, + 0xbefc006e, 0xc0611bfc, + 0x0000007c, 0x806e846e, + 0xbefc007e, 0xbefe007c, + 0xbefc006e, 0xc0611b3c, + 0x0000007c, 0x806e846e, + 0xbefc007e, 0xbefe007c, + 0xbefc006e, 0xc0611b7c, + 0x0000007c, 0x806e846e, + 0xbefc007e, 0x867aff7f, + 0x04000000, 0xbef30080, + 0x8773737a, 0xb8ee2a05, + 0x806e816e, 0x8e6e8a6e, + 0xb8f51605, 0x80758175, + 0x8e758475, 0x8e7a8275, + 0xbefa00ff, 0x01000000, + 0xbef60178, 0x80786e78, + 0x82798079, 0xbefc0080, + 0xbe802b00, 0xbe822b02, + 0xbe842b04, 0xbe862b06, + 0xbe882b08, 0xbe8a2b0a, + 0xbe8c2b0c, 0xbe8e2b0e, + 0xc06b003c, 0x00000000, + 0xc06b013c, 0x00000010, + 0xc06b023c, 0x00000020, + 0xc06b033c, 0x00000030, + 0x8078c078, 0x82798079, + 0x807c907c, 0xbf0a757c, + 0xbf85ffeb, 0xbef80176, + 0xbeee0080, 0xbefe00c1, + 0xbeff00c1, 0xbefa00ff, + 0x01000000, 0xe0724000, + 0x6e1e0000, 0xe0724100, + 0x6e1e0100, 0xe0724200, + 0x6e1e0200, 0xe0724300, + 0x6e1e0300, 0xbefe00c1, + 0xbeff00c1, 0xb8f54306, + 0x8675c175, 0xbf84002c, + 0xbf8a0000, 0x867aff73, + 0x04000000, 0xbf840028, + 0x8e758675, 0x8e758275, + 0xbefa0075, 0xb8ee2a05, + 0x806e816e, 0x8e6e8a6e, + 0xb8fa1605, 0x807a817a, + 0x8e7a867a, 0x806e7a6e, + 0x806eff6e, 0x00000080, + 0xbefa00ff, 0x01000000, + 0xbefc0080, 0xd28c0002, + 0x000100c1, 0xd28d0003, + 0x000204c1, 0xd1060002, + 0x00011103, 0x7e0602ff, + 0x00000200, 0xbefc00ff, + 0x00010000, 0xbe80007b, + 0x867bff7b, 0xff7fffff, + 0x877bff7b, 0x00058000, + 0xd8ec0000, 0x00000002, + 0xbf8c007f, 0xe0765000, + 0x6e1e0002, 0x32040702, + 0xd0c9006a, 0x0000eb02, + 0xbf87fff7, 0xbefb0000, + 0xbeee00ff, 0x00000400, + 0xbefe00c1, 0xbeff00c1, + 0xb8f52a05, 0x80758175, + 0x8e758275, 0x8e7a8875, + 0xbefa00ff, 0x01000000, + 0xbefc0084, 0xbf0a757c, + 0xbf840015, 0xbf11017c, + 0x8075ff75, 0x00001000, + 0x7e000300, 0x7e020301, + 0x7e040302, 0x7e060303, + 0xe0724000, 0x6e1e0000, + 0xe0724100, 0x6e1e0100, + 0xe0724200, 0x6e1e0200, + 0xe0724300, 0x6e1e0300, + 0x807c847c, 0x806eff6e, + 0x00000400, 0xbf0a757c, + 0xbf85ffef, 0xbf9c0000, + 0xbf8200ca, 0xbef8007e, + 0x8679ff7f, 0x0000ffff, + 0x8779ff79, 0x00040000, + 0xbefa0080, 0xbefb00ff, + 0x00807fac, 0x8676ff7f, + 0x08000000, 0x8f768376, + 0x877b767b, 0x8676ff7f, + 0x70000000, 0x8f768176, + 0x877b767b, 0x8676ff7f, + 0x04000000, 0xbf84001e, + 0xbefe00c1, 0xbeff00c1, + 0xb8f34306, 0x8673c173, + 0xbf840019, 0x8e738673, + 0x8e738273, 0xbefa0073, + 0xb8f22a05, 0x80728172, + 0x8e728a72, 0xb8f61605, + 0x80768176, 0x8e768676, + 0x80727672, 0x8072ff72, + 0x00000080, 0xbefa00ff, + 0x01000000, 0xbefc0080, + 0xe0510000, 0x721e0000, + 0xe0510100, 0x721e0000, + 0x807cff7c, 0x00000200, + 0x8072ff72, 0x00000200, + 0xbf0a737c, 0xbf85fff6, + 0xbef20080, 0xbefe00c1, + 0xbeff00c1, 0xb8f32a05, + 0x80738173, 0x8e738273, + 0x8e7a8873, 0xbefa00ff, + 0x01000000, 0xbef60072, + 0x8072ff72, 0x00000400, + 0xbefc0084, 0xbf11087c, + 0x8073ff73, 0x00008000, + 0xe0524000, 0x721e0000, + 0xe0524100, 0x721e0100, + 0xe0524200, 0x721e0200, + 0xe0524300, 0x721e0300, + 0xbf8c0f70, 0x7e000300, + 0x7e020301, 0x7e040302, + 0x7e060303, 0x807c847c, + 0x8072ff72, 0x00000400, + 0xbf0a737c, 0xbf85ffee, + 0xbf9c0000, 0xe0524000, + 0x761e0000, 0xe0524100, + 0x761e0100, 0xe0524200, + 0x761e0200, 0xe0524300, + 0x761e0300, 0xb8f22a05, + 0x80728172, 0x8e728a72, + 0xb8f61605, 0x80768176, + 0x8e768676, 0x80727672, + 0x80f2c072, 0xb8f31605, + 0x80738173, 0x8e738473, + 0x8e7a8273, 0xbefa00ff, + 0x01000000, 0xbefc0073, + 0xc031003c, 0x00000072, + 0x80f2c072, 0xbf8c007f, + 0x80fc907c, 0xbe802d00, + 0xbe822d02, 0xbe842d04, + 0xbe862d06, 0xbe882d08, + 0xbe8a2d0a, 0xbe8c2d0c, + 0xbe8e2d0e, 0xbf06807c, + 0xbf84fff1, 0xb8f22a05, + 0x80728172, 0x8e728a72, + 0xb8f61605, 0x80768176, + 0x8e768676, 0x80727672, + 0xbefa0084, 0xbefa00ff, + 0x01000000, 0xc0211cfc, + 0x00000072, 0x80728472, + 0xc0211c3c, 0x00000072, + 0x80728472, 0xc0211c7c, + 0x00000072, 0x80728472, + 0xc0211bbc, 0x00000072, + 0x80728472, 0xc0211bfc, + 0x00000072, 0x80728472, + 0xc0211d3c, 0x00000072, + 0x80728472, 0xc0211d7c, + 0x00000072, 0x80728472, + 0xc0211a3c, 0x00000072, + 0x80728472, 0xc0211a7c, + 0x00000072, 0x80728472, + 0xc0211dfc, 0x00000072, + 0x80728472, 0xc0211b3c, + 0x00000072, 0x80728472, + 0xc0211b7c, 0x00000072, + 0x80728472, 0xbf8c007f, + 0xbefc0073, 0xbefe006e, + 0xbeff006f, 0x867375ff, + 0x000003ff, 0xb9734803, + 0x867375ff, 0xfffff800, + 0x8f738b73, 0xb973a2c3, + 0xb977f801, 0x8673ff71, + 0xf0000000, 0x8f739c73, + 0x8e739073, 0xbef60080, + 0x87767376, 0x8673ff71, + 0x08000000, 0x8f739b73, + 0x8e738f73, 0x87767376, + 0x8673ff74, 0x00800000, + 0x8f739773, 0xb976f807, + 0x8671ff71, 0x0000ffff, + 0x86fe7e7e, 0x86ea6a6a, + 0xb974f802, 0xbf8a0000, + 0x95807370, 0xbf810000, +}; + + +static const uint32_t cwsr_trap_gfx9_hex[] = { + 0xbf820001, 0xbf82015a, + 0xb8f8f802, 0x89788678, + 0xb8f1f803, 0x866eff71, + 0x00000400, 0xbf850034, + 0x866eff71, 0x00000800, + 0xbf850003, 0x866eff71, + 0x00000100, 0xbf840008, + 0x866eff78, 0x00002000, + 0xbf840001, 0xbf810000, + 0x8778ff78, 0x00002000, + 0x80ec886c, 0x82ed806d, + 0xb8eef807, 0x866fff6e, + 0x001f8000, 0x8e6f8b6f, + 0x8977ff77, 0xfc000000, + 0x87776f77, 0x896eff6e, + 0x001f8000, 0xb96ef807, + 0xb8f0f812, 0xb8f1f813, + 0x8ef08870, 0xc0071bb8, + 0x00000000, 0xbf8cc07f, + 0xc0071c38, 0x00000008, + 0xbf8cc07f, 0x86ee6e6e, + 0xbf840001, 0xbe801d6e, + 0xb8f1f803, 0x8671ff71, + 0x000001ff, 0xbf850002, + 0x806c846c, 0x826d806d, + 0x866dff6d, 0x0000ffff, + 0x8f6e8b77, 0x866eff6e, + 0x001f8000, 0xb96ef807, + 0x86fe7e7e, 0x86ea6a6a, + 0xb978f802, 0xbe801f6c, + 0x866dff6d, 0x0000ffff, + 0xbef00080, 0xb9700283, + 0xb8f02407, 0x8e709c70, + 0x876d706d, 0xb8f003c7, + 0x8e709b70, 0x876d706d, + 0xb8f0f807, 0x8670ff70, + 0x00007fff, 0xb970f807, + 0xbeee007e, 0xbeef007f, + 0xbefe0180, 0xbf900004, + 0x87708478, 0xb970f802, + 0xbf8e0002, 0xbf88fffe, + 0xb8f02a05, 0x80708170, + 0x8e708a70, 0xb8f11605, + 0x80718171, 0x8e718671, + 0x80707170, 0x80707e70, + 0x8271807f, 0x8671ff71, + 0x0000ffff, 0xc0471cb8, + 0x00000040, 0xbf8cc07f, + 0xc04b1d38, 0x00000048, + 0xbf8cc07f, 0xc0431e78, + 0x00000058, 0xbf8cc07f, + 0xc0471eb8, 0x0000005c, + 0xbf8cc07f, 0xbef4007e, + 0x8675ff7f, 0x0000ffff, + 0x8775ff75, 0x00040000, + 0xbef60080, 0xbef700ff, + 0x00807fac, 0x8670ff7f, + 0x08000000, 0x8f708370, + 0x87777077, 0x8670ff7f, + 0x70000000, 0x8f708170, + 0x87777077, 0xbefb007c, + 0xbefa0080, 0xb8fa2a05, + 0x807a817a, 0x8e7a8a7a, + 0xb8f01605, 0x80708170, + 0x8e708670, 0x807a707a, + 0xbef60084, 0xbef600ff, + 0x01000000, 0xbefe007c, + 0xbefc007a, 0xc0611efa, + 0x0000007c, 0xbf8cc07f, + 0x807a847a, 0xbefc007e, + 0xbefe007c, 0xbefc007a, + 0xc0611b3a, 0x0000007c, + 0xbf8cc07f, 0x807a847a, + 0xbefc007e, 0xbefe007c, + 0xbefc007a, 0xc0611b7a, + 0x0000007c, 0xbf8cc07f, + 0x807a847a, 0xbefc007e, + 0xbefe007c, 0xbefc007a, + 0xc0611bba, 0x0000007c, + 0xbf8cc07f, 0x807a847a, + 0xbefc007e, 0xbefe007c, + 0xbefc007a, 0xc0611bfa, + 0x0000007c, 0xbf8cc07f, + 0x807a847a, 0xbefc007e, + 0xbefe007c, 0xbefc007a, + 0xc0611e3a, 0x0000007c, + 0xbf8cc07f, 0x807a847a, + 0xbefc007e, 0xb8f1f803, + 0xbefe007c, 0xbefc007a, + 0xc0611c7a, 0x0000007c, + 0xbf8cc07f, 0x807a847a, + 0xbefc007e, 0xbefe007c, + 0xbefc007a, 0xc0611a3a, + 0x0000007c, 0xbf8cc07f, + 0x807a847a, 0xbefc007e, + 0xbefe007c, 0xbefc007a, + 0xc0611a7a, 0x0000007c, + 0xbf8cc07f, 0x807a847a, + 0xbefc007e, 0xb8fbf801, + 0xbefe007c, 0xbefc007a, + 0xc0611efa, 0x0000007c, + 0xbf8cc07f, 0x807a847a, + 0xbefc007e, 0x8670ff7f, + 0x04000000, 0xbeef0080, + 0x876f6f70, 0xb8fa2a05, + 0x807a817a, 0x8e7a8a7a, + 0xb8f11605, 0x80718171, + 0x8e718471, 0x8e768271, + 0xbef600ff, 0x01000000, + 0xbef20174, 0x80747a74, + 0x82758075, 0xbefc0080, + 0xbf800000, 0xbe802b00, + 0xbe822b02, 0xbe842b04, + 0xbe862b06, 0xbe882b08, + 0xbe8a2b0a, 0xbe8c2b0c, + 0xbe8e2b0e, 0xc06b003a, + 0x00000000, 0xbf8cc07f, + 0xc06b013a, 0x00000010, + 0xbf8cc07f, 0xc06b023a, + 0x00000020, 0xbf8cc07f, + 0xc06b033a, 0x00000030, + 0xbf8cc07f, 0x8074c074, + 0x82758075, 0x807c907c, + 0xbf0a717c, 0xbf85ffe7, + 0xbef40172, 0xbefa0080, + 0xbefe00c1, 0xbeff00c1, + 0xbee80080, 0xbee90080, + 0xbef600ff, 0x01000000, + 0xe0724000, 0x7a1d0000, + 0xe0724100, 0x7a1d0100, + 0xe0724200, 0x7a1d0200, + 0xe0724300, 0x7a1d0300, + 0xbefe00c1, 0xbeff00c1, + 0xb8f14306, 0x8671c171, + 0xbf84002c, 0xbf8a0000, + 0x8670ff6f, 0x04000000, + 0xbf840028, 0x8e718671, + 0x8e718271, 0xbef60071, + 0xb8fa2a05, 0x807a817a, + 0x8e7a8a7a, 0xb8f01605, + 0x80708170, 0x8e708670, + 0x807a707a, 0x807aff7a, + 0x00000080, 0xbef600ff, + 0x01000000, 0xbefc0080, + 0xd28c0002, 0x000100c1, + 0xd28d0003, 0x000204c1, + 0xd1060002, 0x00011103, + 0x7e0602ff, 0x00000200, + 0xbefc00ff, 0x00010000, + 0xbe800077, 0x8677ff77, + 0xff7fffff, 0x8777ff77, + 0x00058000, 0xd8ec0000, + 0x00000002, 0xbf8cc07f, + 0xe0765000, 0x7a1d0002, + 0x68040702, 0xd0c9006a, + 0x0000e302, 0xbf87fff7, + 0xbef70000, 0xbefa00ff, + 0x00000400, 0xbefe00c1, + 0xbeff00c1, 0xb8f12a05, + 0x80718171, 0x8e718271, + 0x8e768871, 0xbef600ff, + 0x01000000, 0xbefc0084, + 0xbf0a717c, 0xbf840015, + 0xbf11017c, 0x8071ff71, + 0x00001000, 0x7e000300, + 0x7e020301, 0x7e040302, + 0x7e060303, 0xe0724000, + 0x7a1d0000, 0xe0724100, + 0x7a1d0100, 0xe0724200, + 0x7a1d0200, 0xe0724300, + 0x7a1d0300, 0x807c847c, + 0x807aff7a, 0x00000400, + 0xbf0a717c, 0xbf85ffef, + 0xbf9c0000, 0xbf8200d9, + 0xbef4007e, 0x8675ff7f, + 0x0000ffff, 0x8775ff75, + 0x00040000, 0xbef60080, + 0xbef700ff, 0x00807fac, + 0x866eff7f, 0x08000000, + 0x8f6e836e, 0x87776e77, + 0x866eff7f, 0x70000000, + 0x8f6e816e, 0x87776e77, + 0x866eff7f, 0x04000000, + 0xbf84001e, 0xbefe00c1, + 0xbeff00c1, 0xb8ef4306, + 0x866fc16f, 0xbf840019, + 0x8e6f866f, 0x8e6f826f, + 0xbef6006f, 0xb8f82a05, + 0x80788178, 0x8e788a78, + 0xb8ee1605, 0x806e816e, + 0x8e6e866e, 0x80786e78, + 0x8078ff78, 0x00000080, + 0xbef600ff, 0x01000000, + 0xbefc0080, 0xe0510000, + 0x781d0000, 0xe0510100, + 0x781d0000, 0x807cff7c, + 0x00000200, 0x8078ff78, + 0x00000200, 0xbf0a6f7c, + 0xbf85fff6, 0xbef80080, + 0xbefe00c1, 0xbeff00c1, + 0xb8ef2a05, 0x806f816f, + 0x8e6f826f, 0x8e76886f, + 0xbef600ff, 0x01000000, + 0xbeee0078, 0x8078ff78, + 0x00000400, 0xbefc0084, + 0xbf11087c, 0x806fff6f, + 0x00008000, 0xe0524000, + 0x781d0000, 0xe0524100, + 0x781d0100, 0xe0524200, + 0x781d0200, 0xe0524300, + 0x781d0300, 0xbf8c0f70, + 0x7e000300, 0x7e020301, + 0x7e040302, 0x7e060303, + 0x807c847c, 0x8078ff78, + 0x00000400, 0xbf0a6f7c, + 0xbf85ffee, 0xbf9c0000, + 0xe0524000, 0x6e1d0000, + 0xe0524100, 0x6e1d0100, + 0xe0524200, 0x6e1d0200, + 0xe0524300, 0x6e1d0300, + 0xb8f82a05, 0x80788178, + 0x8e788a78, 0xb8ee1605, + 0x806e816e, 0x8e6e866e, + 0x80786e78, 0x80f8c078, + 0xb8ef1605, 0x806f816f, + 0x8e6f846f, 0x8e76826f, + 0xbef600ff, 0x01000000, + 0xbefc006f, 0xc031003a, + 0x00000078, 0x80f8c078, + 0xbf8cc07f, 0x80fc907c, + 0xbf800000, 0xbe802d00, + 0xbe822d02, 0xbe842d04, + 0xbe862d06, 0xbe882d08, + 0xbe8a2d0a, 0xbe8c2d0c, + 0xbe8e2d0e, 0xbf06807c, + 0xbf84fff0, 0xb8f82a05, + 0x80788178, 0x8e788a78, + 0xb8ee1605, 0x806e816e, + 0x8e6e866e, 0x80786e78, + 0xbef60084, 0xbef600ff, + 0x01000000, 0xc0211bfa, + 0x00000078, 0x80788478, + 0xc0211b3a, 0x00000078, + 0x80788478, 0xc0211b7a, + 0x00000078, 0x80788478, + 0xc0211eba, 0x00000078, + 0x80788478, 0xc0211efa, + 0x00000078, 0x80788478, + 0xc0211c3a, 0x00000078, + 0x80788478, 0xc0211c7a, + 0x00000078, 0x80788478, + 0xc0211a3a, 0x00000078, + 0x80788478, 0xc0211a7a, + 0x00000078, 0x80788478, + 0xc0211cfa, 0x00000078, + 0x80788478, 0xbf8cc07f, + 0xbefc006f, 0xbefe007a, + 0xbeff007b, 0x866f71ff, + 0x000003ff, 0xb96f4803, + 0x866f71ff, 0xfffff800, + 0x8f6f8b6f, 0xb96fa2c3, + 0xb973f801, 0xb8ee2a05, + 0x806e816e, 0x8e6e8a6e, + 0xb8ef1605, 0x806f816f, + 0x8e6f866f, 0x806e6f6e, + 0x806e746e, 0x826f8075, + 0x866fff6f, 0x0000ffff, + 0xc0071cb7, 0x00000040, + 0xc00b1d37, 0x00000048, + 0xc0031e77, 0x00000058, + 0xc0071eb7, 0x0000005c, + 0xbf8cc07f, 0x866fff6d, + 0xf0000000, 0x8f6f9c6f, + 0x8e6f906f, 0xbeee0080, + 0x876e6f6e, 0x866fff6d, + 0x08000000, 0x8f6f9b6f, + 0x8e6f8f6f, 0x876e6f6e, + 0x866fff70, 0x00800000, + 0x8f6f976f, 0xb96ef807, + 0x866dff6d, 0x0000ffff, + 0x86fe7e7e, 0x86ea6a6a, + 0xb970f802, 0xbf8a0000, + 0x95806f6c, 0xbf810000, +}; diff --git a/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler_gfx8.asm b/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler_gfx8.asm index 997a383dcb8b..a2a04bb64096 100644 --- a/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler_gfx8.asm +++ b/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler_gfx8.asm @@ -20,9 +20,12 @@ * OTHER DEALINGS IN THE SOFTWARE. */ -#if 0 -HW (VI) source code for CWSR trap handler -#Version 18 + multiple trap handler +/* To compile this assembly code: + * PROJECT=vi ./sp3 cwsr_trap_handler_gfx8.asm -hex tmp.hex + */ + +/* HW (VI) source code for CWSR trap handler */ +/* Version 18 + multiple trap handler */ // this performance-optimal version was originally from Seven Xu at SRDC @@ -98,6 +101,7 @@ var SWIZZLE_EN = 0 //whether we use swi /**************************************************************************/ var SQ_WAVE_STATUS_INST_ATC_SHIFT = 23 var SQ_WAVE_STATUS_INST_ATC_MASK = 0x00800000 +var SQ_WAVE_STATUS_SPI_PRIO_SHIFT = 1 var SQ_WAVE_STATUS_SPI_PRIO_MASK = 0x00000006 var SQ_WAVE_LDS_ALLOC_LDS_SIZE_SHIFT = 12 @@ -149,7 +153,7 @@ var s_save_spi_init_lo = exec_lo var s_save_spi_init_hi = exec_hi //tba_lo and tba_hi need to be saved/restored -var s_save_pc_lo = ttmp0 //{TTMP1, TTMP0} = {3??h0,pc_rewind[3:0], HT[0],trapID[7:0], PC[47:0]} +var s_save_pc_lo = ttmp0 //{TTMP1, TTMP0} = {3'h0,pc_rewind[3:0], HT[0],trapID[7:0], PC[47:0]} var s_save_pc_hi = ttmp1 var s_save_exec_lo = ttmp2 var s_save_exec_hi = ttmp3 @@ -319,6 +323,10 @@ end s_sendmsg sendmsg(MSG_SAVEWAVE) //send SPI a message and wait for SPI's write to EXEC end + // Set SPI_PRIO=2 to avoid starving instruction fetch in the waves we're waiting for. + s_or_b32 s_save_tmp, s_save_status, (2 << SQ_WAVE_STATUS_SPI_PRIO_SHIFT) + s_setreg_b32 hwreg(HW_REG_STATUS), s_save_tmp + L_SLEEP: s_sleep 0x2 // sleep 1 (64clk) is not enough for 8 waves per SIMD, which will cause SQ hang, since the 7,8th wave could not get arbit to exec inst, while other waves are stuck into the sleep-loop and waiting for wrexec!=0 @@ -1007,8 +1015,6 @@ end s_waitcnt lgkmcnt(0) //from now on, it is safe to restore STATUS and IB_STS - s_and_b32 s_restore_pc_hi, s_restore_pc_hi, 0x0000ffff //pc[47:32] //Do it here in order not to affect STATUS - //for normal save & restore, the saved PC points to the next inst to execute, no adjustment needs to be made, otherwise: if ((EMU_RUN_HACK) && (!EMU_RUN_HACK_RESTORE_NORMAL)) s_add_u32 s_restore_pc_lo, s_restore_pc_lo, 8 //pc[31:0]+8 //two back-to-back s_trap are used (first for save and second for restore) @@ -1044,6 +1050,7 @@ end s_lshr_b32 s_restore_m0, s_restore_m0, SQ_WAVE_STATUS_INST_ATC_SHIFT s_setreg_b32 hwreg(HW_REG_IB_STS), s_restore_tmp + s_and_b32 s_restore_pc_hi, s_restore_pc_hi, 0x0000ffff //pc[47:32] //Do it here in order not to affect STATUS s_and_b64 exec, exec, exec // Restore STATUS.EXECZ, not writable by s_setreg_b32 s_and_b64 vcc, vcc, vcc // Restore STATUS.VCCZ, not writable by s_setreg_b32 s_setreg_b32 hwreg(HW_REG_STATUS), s_restore_status // SCC is included, which is changed by previous salu @@ -1127,258 +1134,3 @@ end function get_hwreg_size_bytes return 128 //HWREG size 128 bytes end - - -#endif - -static const uint32_t cwsr_trap_gfx8_hex[] = { - 0xbf820001, 0xbf820123, - 0xb8f4f802, 0x89748674, - 0xb8f5f803, 0x8675ff75, - 0x00000400, 0xbf850011, - 0xc00a1e37, 0x00000000, - 0xbf8c007f, 0x87777978, - 0xbf840002, 0xb974f802, - 0xbe801d78, 0xb8f5f803, - 0x8675ff75, 0x000001ff, - 0xbf850002, 0x80708470, - 0x82718071, 0x8671ff71, - 0x0000ffff, 0xb974f802, - 0xbe801f70, 0xb8f5f803, - 0x8675ff75, 0x00000100, - 0xbf840006, 0xbefa0080, - 0xb97a0203, 0x8671ff71, - 0x0000ffff, 0x80f08870, - 0x82f18071, 0xbefa0080, - 0xb97a0283, 0xbef60068, - 0xbef70069, 0xb8fa1c07, - 0x8e7a9c7a, 0x87717a71, - 0xb8fa03c7, 0x8e7a9b7a, - 0x87717a71, 0xb8faf807, - 0x867aff7a, 0x00007fff, - 0xb97af807, 0xbef2007e, - 0xbef3007f, 0xbefe0180, - 0xbf900004, 0xbf8e0002, - 0xbf88fffe, 0xbef8007e, - 0x8679ff7f, 0x0000ffff, - 0x8779ff79, 0x00040000, - 0xbefa0080, 0xbefb00ff, - 0x00807fac, 0x867aff7f, - 0x08000000, 0x8f7a837a, - 0x877b7a7b, 0x867aff7f, - 0x70000000, 0x8f7a817a, - 0x877b7a7b, 0xbeef007c, - 0xbeee0080, 0xb8ee2a05, - 0x806e816e, 0x8e6e8a6e, - 0xb8fa1605, 0x807a817a, - 0x8e7a867a, 0x806e7a6e, - 0xbefa0084, 0xbefa00ff, - 0x01000000, 0xbefe007c, - 0xbefc006e, 0xc0611bfc, - 0x0000007c, 0x806e846e, - 0xbefc007e, 0xbefe007c, - 0xbefc006e, 0xc0611c3c, - 0x0000007c, 0x806e846e, - 0xbefc007e, 0xbefe007c, - 0xbefc006e, 0xc0611c7c, - 0x0000007c, 0x806e846e, - 0xbefc007e, 0xbefe007c, - 0xbefc006e, 0xc0611cbc, - 0x0000007c, 0x806e846e, - 0xbefc007e, 0xbefe007c, - 0xbefc006e, 0xc0611cfc, - 0x0000007c, 0x806e846e, - 0xbefc007e, 0xbefe007c, - 0xbefc006e, 0xc0611d3c, - 0x0000007c, 0x806e846e, - 0xbefc007e, 0xb8f5f803, - 0xbefe007c, 0xbefc006e, - 0xc0611d7c, 0x0000007c, - 0x806e846e, 0xbefc007e, - 0xbefe007c, 0xbefc006e, - 0xc0611dbc, 0x0000007c, - 0x806e846e, 0xbefc007e, - 0xbefe007c, 0xbefc006e, - 0xc0611dfc, 0x0000007c, - 0x806e846e, 0xbefc007e, - 0xb8eff801, 0xbefe007c, - 0xbefc006e, 0xc0611bfc, - 0x0000007c, 0x806e846e, - 0xbefc007e, 0xbefe007c, - 0xbefc006e, 0xc0611b3c, - 0x0000007c, 0x806e846e, - 0xbefc007e, 0xbefe007c, - 0xbefc006e, 0xc0611b7c, - 0x0000007c, 0x806e846e, - 0xbefc007e, 0x867aff7f, - 0x04000000, 0xbef30080, - 0x8773737a, 0xb8ee2a05, - 0x806e816e, 0x8e6e8a6e, - 0xb8f51605, 0x80758175, - 0x8e758475, 0x8e7a8275, - 0xbefa00ff, 0x01000000, - 0xbef60178, 0x80786e78, - 0x82798079, 0xbefc0080, - 0xbe802b00, 0xbe822b02, - 0xbe842b04, 0xbe862b06, - 0xbe882b08, 0xbe8a2b0a, - 0xbe8c2b0c, 0xbe8e2b0e, - 0xc06b003c, 0x00000000, - 0xc06b013c, 0x00000010, - 0xc06b023c, 0x00000020, - 0xc06b033c, 0x00000030, - 0x8078c078, 0x82798079, - 0x807c907c, 0xbf0a757c, - 0xbf85ffeb, 0xbef80176, - 0xbeee0080, 0xbefe00c1, - 0xbeff00c1, 0xbefa00ff, - 0x01000000, 0xe0724000, - 0x6e1e0000, 0xe0724100, - 0x6e1e0100, 0xe0724200, - 0x6e1e0200, 0xe0724300, - 0x6e1e0300, 0xbefe00c1, - 0xbeff00c1, 0xb8f54306, - 0x8675c175, 0xbf84002c, - 0xbf8a0000, 0x867aff73, - 0x04000000, 0xbf840028, - 0x8e758675, 0x8e758275, - 0xbefa0075, 0xb8ee2a05, - 0x806e816e, 0x8e6e8a6e, - 0xb8fa1605, 0x807a817a, - 0x8e7a867a, 0x806e7a6e, - 0x806eff6e, 0x00000080, - 0xbefa00ff, 0x01000000, - 0xbefc0080, 0xd28c0002, - 0x000100c1, 0xd28d0003, - 0x000204c1, 0xd1060002, - 0x00011103, 0x7e0602ff, - 0x00000200, 0xbefc00ff, - 0x00010000, 0xbe80007b, - 0x867bff7b, 0xff7fffff, - 0x877bff7b, 0x00058000, - 0xd8ec0000, 0x00000002, - 0xbf8c007f, 0xe0765000, - 0x6e1e0002, 0x32040702, - 0xd0c9006a, 0x0000eb02, - 0xbf87fff7, 0xbefb0000, - 0xbeee00ff, 0x00000400, - 0xbefe00c1, 0xbeff00c1, - 0xb8f52a05, 0x80758175, - 0x8e758275, 0x8e7a8875, - 0xbefa00ff, 0x01000000, - 0xbefc0084, 0xbf0a757c, - 0xbf840015, 0xbf11017c, - 0x8075ff75, 0x00001000, - 0x7e000300, 0x7e020301, - 0x7e040302, 0x7e060303, - 0xe0724000, 0x6e1e0000, - 0xe0724100, 0x6e1e0100, - 0xe0724200, 0x6e1e0200, - 0xe0724300, 0x6e1e0300, - 0x807c847c, 0x806eff6e, - 0x00000400, 0xbf0a757c, - 0xbf85ffef, 0xbf9c0000, - 0xbf8200ca, 0xbef8007e, - 0x8679ff7f, 0x0000ffff, - 0x8779ff79, 0x00040000, - 0xbefa0080, 0xbefb00ff, - 0x00807fac, 0x8676ff7f, - 0x08000000, 0x8f768376, - 0x877b767b, 0x8676ff7f, - 0x70000000, 0x8f768176, - 0x877b767b, 0x8676ff7f, - 0x04000000, 0xbf84001e, - 0xbefe00c1, 0xbeff00c1, - 0xb8f34306, 0x8673c173, - 0xbf840019, 0x8e738673, - 0x8e738273, 0xbefa0073, - 0xb8f22a05, 0x80728172, - 0x8e728a72, 0xb8f61605, - 0x80768176, 0x8e768676, - 0x80727672, 0x8072ff72, - 0x00000080, 0xbefa00ff, - 0x01000000, 0xbefc0080, - 0xe0510000, 0x721e0000, - 0xe0510100, 0x721e0000, - 0x807cff7c, 0x00000200, - 0x8072ff72, 0x00000200, - 0xbf0a737c, 0xbf85fff6, - 0xbef20080, 0xbefe00c1, - 0xbeff00c1, 0xb8f32a05, - 0x80738173, 0x8e738273, - 0x8e7a8873, 0xbefa00ff, - 0x01000000, 0xbef60072, - 0x8072ff72, 0x00000400, - 0xbefc0084, 0xbf11087c, - 0x8073ff73, 0x00008000, - 0xe0524000, 0x721e0000, - 0xe0524100, 0x721e0100, - 0xe0524200, 0x721e0200, - 0xe0524300, 0x721e0300, - 0xbf8c0f70, 0x7e000300, - 0x7e020301, 0x7e040302, - 0x7e060303, 0x807c847c, - 0x8072ff72, 0x00000400, - 0xbf0a737c, 0xbf85ffee, - 0xbf9c0000, 0xe0524000, - 0x761e0000, 0xe0524100, - 0x761e0100, 0xe0524200, - 0x761e0200, 0xe0524300, - 0x761e0300, 0xb8f22a05, - 0x80728172, 0x8e728a72, - 0xb8f61605, 0x80768176, - 0x8e768676, 0x80727672, - 0x80f2c072, 0xb8f31605, - 0x80738173, 0x8e738473, - 0x8e7a8273, 0xbefa00ff, - 0x01000000, 0xbefc0073, - 0xc031003c, 0x00000072, - 0x80f2c072, 0xbf8c007f, - 0x80fc907c, 0xbe802d00, - 0xbe822d02, 0xbe842d04, - 0xbe862d06, 0xbe882d08, - 0xbe8a2d0a, 0xbe8c2d0c, - 0xbe8e2d0e, 0xbf06807c, - 0xbf84fff1, 0xb8f22a05, - 0x80728172, 0x8e728a72, - 0xb8f61605, 0x80768176, - 0x8e768676, 0x80727672, - 0xbefa0084, 0xbefa00ff, - 0x01000000, 0xc0211cfc, - 0x00000072, 0x80728472, - 0xc0211c3c, 0x00000072, - 0x80728472, 0xc0211c7c, - 0x00000072, 0x80728472, - 0xc0211bbc, 0x00000072, - 0x80728472, 0xc0211bfc, - 0x00000072, 0x80728472, - 0xc0211d3c, 0x00000072, - 0x80728472, 0xc0211d7c, - 0x00000072, 0x80728472, - 0xc0211a3c, 0x00000072, - 0x80728472, 0xc0211a7c, - 0x00000072, 0x80728472, - 0xc0211dfc, 0x00000072, - 0x80728472, 0xc0211b3c, - 0x00000072, 0x80728472, - 0xc0211b7c, 0x00000072, - 0x80728472, 0xbf8c007f, - 0x8671ff71, 0x0000ffff, - 0xbefc0073, 0xbefe006e, - 0xbeff006f, 0x867375ff, - 0x000003ff, 0xb9734803, - 0x867375ff, 0xfffff800, - 0x8f738b73, 0xb973a2c3, - 0xb977f801, 0x8673ff71, - 0xf0000000, 0x8f739c73, - 0x8e739073, 0xbef60080, - 0x87767376, 0x8673ff71, - 0x08000000, 0x8f739b73, - 0x8e738f73, 0x87767376, - 0x8673ff74, 0x00800000, - 0x8f739773, 0xb976f807, - 0x86fe7e7e, 0x86ea6a6a, - 0xb974f802, 0xbf8a0000, - 0x95807370, 0xbf810000, -}; - diff --git a/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler_gfx9.asm b/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler_gfx9.asm new file mode 100644 index 000000000000..998be96be736 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler_gfx9.asm @@ -0,0 +1,1214 @@ +/* + * Copyright 2016 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +/* To compile this assembly code: + * PROJECT=greenland ./sp3 cwsr_trap_handler_gfx9.asm -hex tmp.hex + */ + +/* HW (GFX9) source code for CWSR trap handler */ +/* Version 18 + multiple trap handler */ + +// this performance-optimal version was originally from Seven Xu at SRDC + +// Revison #18 --... +/* Rev History +** #1. Branch from gc dv. //gfxip/gfx9/main/src/test/suites/block/cs/sr/cs_trap_handler.sp3#1,#50, #51, #52-53(Skip, Already Fixed by PV), #54-56(merged),#57-58(mergerd, skiped-already fixed by PV) +** #4. SR Memory Layout: +** 1. VGPR-SGPR-HWREG-{LDS} +** 2. tba_hi.bits.26 - reconfigured as the first wave in tg bits, for defer Save LDS for a threadgroup.. performance concern.. +** #5. Update: 1. Accurate g8sr_ts_save_d timestamp +** #6. Update: 1. Fix s_barrier usage; 2. VGPR s/r using swizzle buffer?(NoNeed, already matched the swizzle pattern, more investigation) +** #7. Update: 1. don't barrier if noLDS +** #8. Branch: 1. Branch to ver#0, which is very similar to gc dv version +** 2. Fix SQ issue by s_sleep 2 +** #9. Update: 1. Fix scc restore failed issue, restore wave_status at last +** 2. optimize s_buffer save by burst 16sgprs... +** #10. Update 1. Optimize restore sgpr by busrt 16 sgprs. +** #11. Update 1. Add 2 more timestamp for debug version +** #12. Update 1. Add VGPR SR using DWx4, some case improve and some case drop performance +** #13. Integ 1. Always use MUBUF for PV trap shader... +** #14. Update 1. s_buffer_store soft clause... +** #15. Update 1. PERF - sclar write with glc:0/mtype0 to allow L2 combine. perf improvement a lot. +** #16. Update 1. PRRF - UNROLL LDS_DMA got 2500cycle save in IP tree +** #17. Update 1. FUNC - LDS_DMA has issues while ATC, replace with ds_read/buffer_store for save part[TODO restore part] +** 2. PERF - Save LDS before save VGPR to cover LDS save long latency... +** #18. Update 1. FUNC - Implicitly estore STATUS.VCCZ, which is not writable by s_setreg_b32 +** 2. FUNC - Handle non-CWSR traps +*/ + +var G8SR_WDMEM_HWREG_OFFSET = 0 +var G8SR_WDMEM_SGPR_OFFSET = 128 // in bytes + +// Keep definition same as the app shader, These 2 time stamps are part of the app shader... Should before any Save and after restore. + +var G8SR_DEBUG_TIMESTAMP = 0 +var G8SR_DEBUG_TS_SAVE_D_OFFSET = 40*4 // ts_save_d timestamp offset relative to SGPR_SR_memory_offset +var s_g8sr_ts_save_s = s[34:35] // save start +var s_g8sr_ts_sq_save_msg = s[36:37] // The save shader send SAVEWAVE msg to spi +var s_g8sr_ts_spi_wrexec = s[38:39] // the SPI write the sr address to SQ +var s_g8sr_ts_save_d = s[40:41] // save end +var s_g8sr_ts_restore_s = s[42:43] // restore start +var s_g8sr_ts_restore_d = s[44:45] // restore end + +var G8SR_VGPR_SR_IN_DWX4 = 0 +var G8SR_SAVE_BUF_RSRC_WORD1_STRIDE_DWx4 = 0x00100000 // DWx4 stride is 4*4Bytes +var G8SR_RESTORE_BUF_RSRC_WORD1_STRIDE_DWx4 = G8SR_SAVE_BUF_RSRC_WORD1_STRIDE_DWx4 + + +/*************************************************************************/ +/* control on how to run the shader */ +/*************************************************************************/ +//any hack that needs to be made to run this code in EMU (either because various EMU code are not ready or no compute save & restore in EMU run) +var EMU_RUN_HACK = 0 +var EMU_RUN_HACK_RESTORE_NORMAL = 0 +var EMU_RUN_HACK_SAVE_NORMAL_EXIT = 0 +var EMU_RUN_HACK_SAVE_SINGLE_WAVE = 0 +var EMU_RUN_HACK_SAVE_FIRST_TIME = 0 //for interrupted restore in which the first save is through EMU_RUN_HACK +var SAVE_LDS = 1 +var WG_BASE_ADDR_LO = 0x9000a000 +var WG_BASE_ADDR_HI = 0x0 +var WAVE_SPACE = 0x5000 //memory size that each wave occupies in workgroup state mem +var CTX_SAVE_CONTROL = 0x0 +var CTX_RESTORE_CONTROL = CTX_SAVE_CONTROL +var SIM_RUN_HACK = 0 //any hack that needs to be made to run this code in SIM (either because various RTL code are not ready or no compute save & restore in RTL run) +var SGPR_SAVE_USE_SQC = 1 //use SQC D$ to do the write +var USE_MTBUF_INSTEAD_OF_MUBUF = 0 //because TC EMU currently asserts on 0 of // overload DFMT field to carry 4 more bits of stride for MUBUF opcodes +var SWIZZLE_EN = 0 //whether we use swizzled buffer addressing +var ACK_SQC_STORE = 1 //workaround for suspected SQC store bug causing incorrect stores under concurrency + +/**************************************************************************/ +/* variables */ +/**************************************************************************/ +var SQ_WAVE_STATUS_INST_ATC_SHIFT = 23 +var SQ_WAVE_STATUS_INST_ATC_MASK = 0x00800000 +var SQ_WAVE_STATUS_SPI_PRIO_SHIFT = 1 +var SQ_WAVE_STATUS_SPI_PRIO_MASK = 0x00000006 +var SQ_WAVE_STATUS_HALT_MASK = 0x2000 + +var SQ_WAVE_LDS_ALLOC_LDS_SIZE_SHIFT = 12 +var SQ_WAVE_LDS_ALLOC_LDS_SIZE_SIZE = 9 +var SQ_WAVE_GPR_ALLOC_VGPR_SIZE_SHIFT = 8 +var SQ_WAVE_GPR_ALLOC_VGPR_SIZE_SIZE = 6 +var SQ_WAVE_GPR_ALLOC_SGPR_SIZE_SHIFT = 24 +var SQ_WAVE_GPR_ALLOC_SGPR_SIZE_SIZE = 3 //FIXME sq.blk still has 4 bits at this time while SQ programming guide has 3 bits + +var SQ_WAVE_TRAPSTS_SAVECTX_MASK = 0x400 +var SQ_WAVE_TRAPSTS_EXCE_MASK = 0x1FF // Exception mask +var SQ_WAVE_TRAPSTS_SAVECTX_SHIFT = 10 +var SQ_WAVE_TRAPSTS_MEM_VIOL_MASK = 0x100 +var SQ_WAVE_TRAPSTS_MEM_VIOL_SHIFT = 8 +var SQ_WAVE_TRAPSTS_PRE_SAVECTX_MASK = 0x3FF +var SQ_WAVE_TRAPSTS_PRE_SAVECTX_SHIFT = 0x0 +var SQ_WAVE_TRAPSTS_PRE_SAVECTX_SIZE = 10 +var SQ_WAVE_TRAPSTS_POST_SAVECTX_MASK = 0xFFFFF800 +var SQ_WAVE_TRAPSTS_POST_SAVECTX_SHIFT = 11 +var SQ_WAVE_TRAPSTS_POST_SAVECTX_SIZE = 21 +var SQ_WAVE_TRAPSTS_ILLEGAL_INST_MASK = 0x800 + +var SQ_WAVE_IB_STS_RCNT_SHIFT = 16 //FIXME +var SQ_WAVE_IB_STS_FIRST_REPLAY_SHIFT = 15 //FIXME +var SQ_WAVE_IB_STS_RCNT_FIRST_REPLAY_MASK = 0x1F8000 +var SQ_WAVE_IB_STS_RCNT_FIRST_REPLAY_MASK_NEG = 0x00007FFF //FIXME + +var SQ_BUF_RSRC_WORD1_ATC_SHIFT = 24 +var SQ_BUF_RSRC_WORD3_MTYPE_SHIFT = 27 + +var TTMP11_SAVE_RCNT_FIRST_REPLAY_SHIFT = 26 // bits [31:26] unused by SPI debug data +var TTMP11_SAVE_RCNT_FIRST_REPLAY_MASK = 0xFC000000 + +/* Save */ +var S_SAVE_BUF_RSRC_WORD1_STRIDE = 0x00040000 //stride is 4 bytes +var S_SAVE_BUF_RSRC_WORD3_MISC = 0x00807FAC //SQ_SEL_X/Y/Z/W, BUF_NUM_FORMAT_FLOAT, (0 for MUBUF stride[17:14] when ADD_TID_ENABLE and BUF_DATA_FORMAT_32 for MTBUF), ADD_TID_ENABLE + +var S_SAVE_SPI_INIT_ATC_MASK = 0x08000000 //bit[27]: ATC bit +var S_SAVE_SPI_INIT_ATC_SHIFT = 27 +var S_SAVE_SPI_INIT_MTYPE_MASK = 0x70000000 //bit[30:28]: Mtype +var S_SAVE_SPI_INIT_MTYPE_SHIFT = 28 +var S_SAVE_SPI_INIT_FIRST_WAVE_MASK = 0x04000000 //bit[26]: FirstWaveInTG +var S_SAVE_SPI_INIT_FIRST_WAVE_SHIFT = 26 + +var S_SAVE_PC_HI_RCNT_SHIFT = 28 //FIXME check with Brian to ensure all fields other than PC[47:0] can be used +var S_SAVE_PC_HI_RCNT_MASK = 0xF0000000 //FIXME +var S_SAVE_PC_HI_FIRST_REPLAY_SHIFT = 27 //FIXME +var S_SAVE_PC_HI_FIRST_REPLAY_MASK = 0x08000000 //FIXME + +var s_save_spi_init_lo = exec_lo +var s_save_spi_init_hi = exec_hi + +var s_save_pc_lo = ttmp0 //{TTMP1, TTMP0} = {3'h0,pc_rewind[3:0], HT[0],trapID[7:0], PC[47:0]} +var s_save_pc_hi = ttmp1 +var s_save_exec_lo = ttmp2 +var s_save_exec_hi = ttmp3 +var s_save_tmp = ttmp4 +var s_save_trapsts = ttmp5 //not really used until the end of the SAVE routine +var s_save_xnack_mask_lo = ttmp6 +var s_save_xnack_mask_hi = ttmp7 +var s_save_buf_rsrc0 = ttmp8 +var s_save_buf_rsrc1 = ttmp9 +var s_save_buf_rsrc2 = ttmp10 +var s_save_buf_rsrc3 = ttmp11 +var s_save_status = ttmp12 +var s_save_mem_offset = ttmp14 +var s_save_alloc_size = s_save_trapsts //conflict +var s_save_m0 = ttmp15 +var s_save_ttmps_lo = s_save_tmp //no conflict +var s_save_ttmps_hi = s_save_trapsts //no conflict + +/* Restore */ +var S_RESTORE_BUF_RSRC_WORD1_STRIDE = S_SAVE_BUF_RSRC_WORD1_STRIDE +var S_RESTORE_BUF_RSRC_WORD3_MISC = S_SAVE_BUF_RSRC_WORD3_MISC + +var S_RESTORE_SPI_INIT_ATC_MASK = 0x08000000 //bit[27]: ATC bit +var S_RESTORE_SPI_INIT_ATC_SHIFT = 27 +var S_RESTORE_SPI_INIT_MTYPE_MASK = 0x70000000 //bit[30:28]: Mtype +var S_RESTORE_SPI_INIT_MTYPE_SHIFT = 28 +var S_RESTORE_SPI_INIT_FIRST_WAVE_MASK = 0x04000000 //bit[26]: FirstWaveInTG +var S_RESTORE_SPI_INIT_FIRST_WAVE_SHIFT = 26 + +var S_RESTORE_PC_HI_RCNT_SHIFT = S_SAVE_PC_HI_RCNT_SHIFT +var S_RESTORE_PC_HI_RCNT_MASK = S_SAVE_PC_HI_RCNT_MASK +var S_RESTORE_PC_HI_FIRST_REPLAY_SHIFT = S_SAVE_PC_HI_FIRST_REPLAY_SHIFT +var S_RESTORE_PC_HI_FIRST_REPLAY_MASK = S_SAVE_PC_HI_FIRST_REPLAY_MASK + +var s_restore_spi_init_lo = exec_lo +var s_restore_spi_init_hi = exec_hi + +var s_restore_mem_offset = ttmp12 +var s_restore_alloc_size = ttmp3 +var s_restore_tmp = ttmp2 +var s_restore_mem_offset_save = s_restore_tmp //no conflict + +var s_restore_m0 = s_restore_alloc_size //no conflict + +var s_restore_mode = ttmp7 + +var s_restore_pc_lo = ttmp0 +var s_restore_pc_hi = ttmp1 +var s_restore_exec_lo = ttmp14 +var s_restore_exec_hi = ttmp15 +var s_restore_status = ttmp4 +var s_restore_trapsts = ttmp5 +var s_restore_xnack_mask_lo = xnack_mask_lo +var s_restore_xnack_mask_hi = xnack_mask_hi +var s_restore_buf_rsrc0 = ttmp8 +var s_restore_buf_rsrc1 = ttmp9 +var s_restore_buf_rsrc2 = ttmp10 +var s_restore_buf_rsrc3 = ttmp11 +var s_restore_ttmps_lo = s_restore_tmp //no conflict +var s_restore_ttmps_hi = s_restore_alloc_size //no conflict + +/**************************************************************************/ +/* trap handler entry points */ +/**************************************************************************/ +/* Shader Main*/ + +shader main + asic(GFX9) + type(CS) + + + if ((EMU_RUN_HACK) && (!EMU_RUN_HACK_RESTORE_NORMAL)) //hack to use trap_id for determining save/restore + //FIXME VCCZ un-init assertion s_getreg_b32 s_save_status, hwreg(HW_REG_STATUS) //save STATUS since we will change SCC + s_and_b32 s_save_tmp, s_save_pc_hi, 0xffff0000 //change SCC + s_cmp_eq_u32 s_save_tmp, 0x007e0000 //Save: trap_id = 0x7e. Restore: trap_id = 0x7f. + s_cbranch_scc0 L_JUMP_TO_RESTORE //do not need to recover STATUS here since we are going to RESTORE + //FIXME s_setreg_b32 hwreg(HW_REG_STATUS), s_save_status //need to recover STATUS since we are going to SAVE + s_branch L_SKIP_RESTORE //NOT restore, SAVE actually + else + s_branch L_SKIP_RESTORE //NOT restore. might be a regular trap or save + end + +L_JUMP_TO_RESTORE: + s_branch L_RESTORE //restore + +L_SKIP_RESTORE: + + s_getreg_b32 s_save_status, hwreg(HW_REG_STATUS) //save STATUS since we will change SCC + s_andn2_b32 s_save_status, s_save_status, SQ_WAVE_STATUS_SPI_PRIO_MASK //check whether this is for save + s_getreg_b32 s_save_trapsts, hwreg(HW_REG_TRAPSTS) + s_and_b32 ttmp2, s_save_trapsts, SQ_WAVE_TRAPSTS_SAVECTX_MASK //check whether this is for save + s_cbranch_scc1 L_SAVE //this is the operation for save + + // ********* Handle non-CWSR traps ******************* +if (!EMU_RUN_HACK) + // Illegal instruction is a non-maskable exception which blocks context save. + // Halt the wavefront and return from the trap. + s_and_b32 ttmp2, s_save_trapsts, SQ_WAVE_TRAPSTS_ILLEGAL_INST_MASK + s_cbranch_scc1 L_HALT_WAVE + + // If STATUS.MEM_VIOL is asserted then we cannot fetch from the TMA. + // Instead, halt the wavefront and return from the trap. + s_and_b32 ttmp2, s_save_trapsts, SQ_WAVE_TRAPSTS_MEM_VIOL_MASK + s_cbranch_scc0 L_FETCH_2ND_TRAP + +L_HALT_WAVE: + // If STATUS.HALT is set then this fault must come from SQC instruction fetch. + // We cannot prevent further faults so just terminate the wavefront. + s_and_b32 ttmp2, s_save_status, SQ_WAVE_STATUS_HALT_MASK + s_cbranch_scc0 L_NOT_ALREADY_HALTED + s_endpgm +L_NOT_ALREADY_HALTED: + s_or_b32 s_save_status, s_save_status, SQ_WAVE_STATUS_HALT_MASK + + // If the PC points to S_ENDPGM then context save will fail if STATUS.HALT is set. + // Rewind the PC to prevent this from occurring. The debugger compensates for this. + s_sub_u32 ttmp0, ttmp0, 0x8 + s_subb_u32 ttmp1, ttmp1, 0x0 + +L_FETCH_2ND_TRAP: + // Preserve and clear scalar XNACK state before issuing scalar reads. + // Save IB_STS.FIRST_REPLAY[15] and IB_STS.RCNT[20:16] into unused space ttmp11[31:26]. + s_getreg_b32 ttmp2, hwreg(HW_REG_IB_STS) + s_and_b32 ttmp3, ttmp2, SQ_WAVE_IB_STS_RCNT_FIRST_REPLAY_MASK + s_lshl_b32 ttmp3, ttmp3, (TTMP11_SAVE_RCNT_FIRST_REPLAY_SHIFT - SQ_WAVE_IB_STS_FIRST_REPLAY_SHIFT) + s_andn2_b32 ttmp11, ttmp11, TTMP11_SAVE_RCNT_FIRST_REPLAY_MASK + s_or_b32 ttmp11, ttmp11, ttmp3 + + s_andn2_b32 ttmp2, ttmp2, SQ_WAVE_IB_STS_RCNT_FIRST_REPLAY_MASK + s_setreg_b32 hwreg(HW_REG_IB_STS), ttmp2 + + // Read second-level TBA/TMA from first-level TMA and jump if available. + // ttmp[2:5] and ttmp12 can be used (others hold SPI-initialized debug data) + // ttmp12 holds SQ_WAVE_STATUS + s_getreg_b32 ttmp4, hwreg(HW_REG_SQ_SHADER_TMA_LO) + s_getreg_b32 ttmp5, hwreg(HW_REG_SQ_SHADER_TMA_HI) + s_lshl_b64 [ttmp4, ttmp5], [ttmp4, ttmp5], 0x8 + s_load_dwordx2 [ttmp2, ttmp3], [ttmp4, ttmp5], 0x0 glc:1 // second-level TBA + s_waitcnt lgkmcnt(0) + s_load_dwordx2 [ttmp4, ttmp5], [ttmp4, ttmp5], 0x8 glc:1 // second-level TMA + s_waitcnt lgkmcnt(0) + s_and_b64 [ttmp2, ttmp3], [ttmp2, ttmp3], [ttmp2, ttmp3] + s_cbranch_scc0 L_NO_NEXT_TRAP // second-level trap handler not been set + s_setpc_b64 [ttmp2, ttmp3] // jump to second-level trap handler + +L_NO_NEXT_TRAP: + s_getreg_b32 s_save_trapsts, hwreg(HW_REG_TRAPSTS) + s_and_b32 s_save_trapsts, s_save_trapsts, SQ_WAVE_TRAPSTS_EXCE_MASK // Check whether it is an exception + s_cbranch_scc1 L_EXCP_CASE // Exception, jump back to the shader program directly. + s_add_u32 ttmp0, ttmp0, 4 // S_TRAP case, add 4 to ttmp0 + s_addc_u32 ttmp1, ttmp1, 0 +L_EXCP_CASE: + s_and_b32 ttmp1, ttmp1, 0xFFFF + + // Restore SQ_WAVE_IB_STS. + s_lshr_b32 ttmp2, ttmp11, (TTMP11_SAVE_RCNT_FIRST_REPLAY_SHIFT - SQ_WAVE_IB_STS_FIRST_REPLAY_SHIFT) + s_and_b32 ttmp2, ttmp2, SQ_WAVE_IB_STS_RCNT_FIRST_REPLAY_MASK + s_setreg_b32 hwreg(HW_REG_IB_STS), ttmp2 + + // Restore SQ_WAVE_STATUS. + s_and_b64 exec, exec, exec // Restore STATUS.EXECZ, not writable by s_setreg_b32 + s_and_b64 vcc, vcc, vcc // Restore STATUS.VCCZ, not writable by s_setreg_b32 + s_setreg_b32 hwreg(HW_REG_STATUS), s_save_status + + s_rfe_b64 [ttmp0, ttmp1] +end + // ********* End handling of non-CWSR traps ******************* + +/**************************************************************************/ +/* save routine */ +/**************************************************************************/ + +L_SAVE: + +if G8SR_DEBUG_TIMESTAMP + s_memrealtime s_g8sr_ts_save_s + s_waitcnt lgkmcnt(0) //FIXME, will cause xnack?? +end + + s_and_b32 s_save_pc_hi, s_save_pc_hi, 0x0000ffff //pc[47:32] + + s_mov_b32 s_save_tmp, 0 //clear saveCtx bit + s_setreg_b32 hwreg(HW_REG_TRAPSTS, SQ_WAVE_TRAPSTS_SAVECTX_SHIFT, 1), s_save_tmp //clear saveCtx bit + + s_getreg_b32 s_save_tmp, hwreg(HW_REG_IB_STS, SQ_WAVE_IB_STS_RCNT_SHIFT, SQ_WAVE_IB_STS_RCNT_SIZE) //save RCNT + s_lshl_b32 s_save_tmp, s_save_tmp, S_SAVE_PC_HI_RCNT_SHIFT + s_or_b32 s_save_pc_hi, s_save_pc_hi, s_save_tmp + s_getreg_b32 s_save_tmp, hwreg(HW_REG_IB_STS, SQ_WAVE_IB_STS_FIRST_REPLAY_SHIFT, SQ_WAVE_IB_STS_FIRST_REPLAY_SIZE) //save FIRST_REPLAY + s_lshl_b32 s_save_tmp, s_save_tmp, S_SAVE_PC_HI_FIRST_REPLAY_SHIFT + s_or_b32 s_save_pc_hi, s_save_pc_hi, s_save_tmp + s_getreg_b32 s_save_tmp, hwreg(HW_REG_IB_STS) //clear RCNT and FIRST_REPLAY in IB_STS + s_and_b32 s_save_tmp, s_save_tmp, SQ_WAVE_IB_STS_RCNT_FIRST_REPLAY_MASK_NEG + + s_setreg_b32 hwreg(HW_REG_IB_STS), s_save_tmp + + /* inform SPI the readiness and wait for SPI's go signal */ + s_mov_b32 s_save_exec_lo, exec_lo //save EXEC and use EXEC for the go signal from SPI + s_mov_b32 s_save_exec_hi, exec_hi + s_mov_b64 exec, 0x0 //clear EXEC to get ready to receive + +if G8SR_DEBUG_TIMESTAMP + s_memrealtime s_g8sr_ts_sq_save_msg + s_waitcnt lgkmcnt(0) +end + + if (EMU_RUN_HACK) + + else + s_sendmsg sendmsg(MSG_SAVEWAVE) //send SPI a message and wait for SPI's write to EXEC + end + + // Set SPI_PRIO=2 to avoid starving instruction fetch in the waves we're waiting for. + s_or_b32 s_save_tmp, s_save_status, (2 << SQ_WAVE_STATUS_SPI_PRIO_SHIFT) + s_setreg_b32 hwreg(HW_REG_STATUS), s_save_tmp + + L_SLEEP: + s_sleep 0x2 // sleep 1 (64clk) is not enough for 8 waves per SIMD, which will cause SQ hang, since the 7,8th wave could not get arbit to exec inst, while other waves are stuck into the sleep-loop and waiting for wrexec!=0 + + if (EMU_RUN_HACK) + + else + s_cbranch_execz L_SLEEP + end + +if G8SR_DEBUG_TIMESTAMP + s_memrealtime s_g8sr_ts_spi_wrexec + s_waitcnt lgkmcnt(0) +end + + if ((EMU_RUN_HACK) && (!EMU_RUN_HACK_SAVE_SINGLE_WAVE)) + //calculate wd_addr using absolute thread id + v_readlane_b32 s_save_tmp, v9, 0 + s_lshr_b32 s_save_tmp, s_save_tmp, 6 + s_mul_i32 s_save_tmp, s_save_tmp, WAVE_SPACE + s_add_i32 s_save_spi_init_lo, s_save_tmp, WG_BASE_ADDR_LO + s_mov_b32 s_save_spi_init_hi, WG_BASE_ADDR_HI + s_and_b32 s_save_spi_init_hi, s_save_spi_init_hi, CTX_SAVE_CONTROL + else + end + if ((EMU_RUN_HACK) && (EMU_RUN_HACK_SAVE_SINGLE_WAVE)) + s_add_i32 s_save_spi_init_lo, s_save_tmp, WG_BASE_ADDR_LO + s_mov_b32 s_save_spi_init_hi, WG_BASE_ADDR_HI + s_and_b32 s_save_spi_init_hi, s_save_spi_init_hi, CTX_SAVE_CONTROL + else + end + + // Save trap temporaries 6-11, 13-15 initialized by SPI debug dispatch logic + // ttmp SR memory offset : size(VGPR)+size(SGPR)+0x40 + get_vgpr_size_bytes(s_save_ttmps_lo) + get_sgpr_size_bytes(s_save_ttmps_hi) + s_add_u32 s_save_ttmps_lo, s_save_ttmps_lo, s_save_ttmps_hi + s_add_u32 s_save_ttmps_lo, s_save_ttmps_lo, s_save_spi_init_lo + s_addc_u32 s_save_ttmps_hi, s_save_spi_init_hi, 0x0 + s_and_b32 s_save_ttmps_hi, s_save_ttmps_hi, 0xFFFF + s_store_dwordx2 [ttmp6, ttmp7], [s_save_ttmps_lo, s_save_ttmps_hi], 0x40 glc:1 + ack_sqc_store_workaround() + s_store_dwordx4 [ttmp8, ttmp9, ttmp10, ttmp11], [s_save_ttmps_lo, s_save_ttmps_hi], 0x48 glc:1 + ack_sqc_store_workaround() + s_store_dword ttmp13, [s_save_ttmps_lo, s_save_ttmps_hi], 0x58 glc:1 + ack_sqc_store_workaround() + s_store_dwordx2 [ttmp14, ttmp15], [s_save_ttmps_lo, s_save_ttmps_hi], 0x5C glc:1 + ack_sqc_store_workaround() + + /* setup Resource Contants */ + s_mov_b32 s_save_buf_rsrc0, s_save_spi_init_lo //base_addr_lo + s_and_b32 s_save_buf_rsrc1, s_save_spi_init_hi, 0x0000FFFF //base_addr_hi + s_or_b32 s_save_buf_rsrc1, s_save_buf_rsrc1, S_SAVE_BUF_RSRC_WORD1_STRIDE + s_mov_b32 s_save_buf_rsrc2, 0 //NUM_RECORDS initial value = 0 (in bytes) although not neccessarily inited + s_mov_b32 s_save_buf_rsrc3, S_SAVE_BUF_RSRC_WORD3_MISC + s_and_b32 s_save_tmp, s_save_spi_init_hi, S_SAVE_SPI_INIT_ATC_MASK + s_lshr_b32 s_save_tmp, s_save_tmp, (S_SAVE_SPI_INIT_ATC_SHIFT-SQ_BUF_RSRC_WORD1_ATC_SHIFT) //get ATC bit into position + s_or_b32 s_save_buf_rsrc3, s_save_buf_rsrc3, s_save_tmp //or ATC + s_and_b32 s_save_tmp, s_save_spi_init_hi, S_SAVE_SPI_INIT_MTYPE_MASK + s_lshr_b32 s_save_tmp, s_save_tmp, (S_SAVE_SPI_INIT_MTYPE_SHIFT-SQ_BUF_RSRC_WORD3_MTYPE_SHIFT) //get MTYPE bits into position + s_or_b32 s_save_buf_rsrc3, s_save_buf_rsrc3, s_save_tmp //or MTYPE + + //FIXME right now s_save_m0/s_save_mem_offset use tma_lo/tma_hi (might need to save them before using them?) + s_mov_b32 s_save_m0, m0 //save M0 + + /* global mem offset */ + s_mov_b32 s_save_mem_offset, 0x0 //mem offset initial value = 0 + + + + + /* save HW registers */ + ////////////////////////////// + + L_SAVE_HWREG: + // HWREG SR memory offset : size(VGPR)+size(SGPR) + get_vgpr_size_bytes(s_save_mem_offset) + get_sgpr_size_bytes(s_save_tmp) + s_add_u32 s_save_mem_offset, s_save_mem_offset, s_save_tmp + + + s_mov_b32 s_save_buf_rsrc2, 0x4 //NUM_RECORDS in bytes + if (SWIZZLE_EN) + s_add_u32 s_save_buf_rsrc2, s_save_buf_rsrc2, 0x0 //FIXME need to use swizzle to enable bounds checking? + else + s_mov_b32 s_save_buf_rsrc2, 0x1000000 //NUM_RECORDS in bytes + end + + + write_hwreg_to_mem(s_save_m0, s_save_buf_rsrc0, s_save_mem_offset) //M0 + + if ((EMU_RUN_HACK) && (EMU_RUN_HACK_SAVE_FIRST_TIME)) + s_add_u32 s_save_pc_lo, s_save_pc_lo, 4 //pc[31:0]+4 + s_addc_u32 s_save_pc_hi, s_save_pc_hi, 0x0 //carry bit over + end + + write_hwreg_to_mem(s_save_pc_lo, s_save_buf_rsrc0, s_save_mem_offset) //PC + write_hwreg_to_mem(s_save_pc_hi, s_save_buf_rsrc0, s_save_mem_offset) + write_hwreg_to_mem(s_save_exec_lo, s_save_buf_rsrc0, s_save_mem_offset) //EXEC + write_hwreg_to_mem(s_save_exec_hi, s_save_buf_rsrc0, s_save_mem_offset) + write_hwreg_to_mem(s_save_status, s_save_buf_rsrc0, s_save_mem_offset) //STATUS + + //s_save_trapsts conflicts with s_save_alloc_size + s_getreg_b32 s_save_trapsts, hwreg(HW_REG_TRAPSTS) + write_hwreg_to_mem(s_save_trapsts, s_save_buf_rsrc0, s_save_mem_offset) //TRAPSTS + + write_hwreg_to_mem(xnack_mask_lo, s_save_buf_rsrc0, s_save_mem_offset) //XNACK_MASK_LO + write_hwreg_to_mem(xnack_mask_hi, s_save_buf_rsrc0, s_save_mem_offset) //XNACK_MASK_HI + + //use s_save_tmp would introduce conflict here between s_save_tmp and s_save_buf_rsrc2 + s_getreg_b32 s_save_m0, hwreg(HW_REG_MODE) //MODE + write_hwreg_to_mem(s_save_m0, s_save_buf_rsrc0, s_save_mem_offset) + + + + /* the first wave in the threadgroup */ + s_and_b32 s_save_tmp, s_save_spi_init_hi, S_SAVE_SPI_INIT_FIRST_WAVE_MASK // extract fisrt wave bit + s_mov_b32 s_save_exec_hi, 0x0 + s_or_b32 s_save_exec_hi, s_save_tmp, s_save_exec_hi // save first wave bit to s_save_exec_hi.bits[26] + + + /* save SGPRs */ + // Save SGPR before LDS save, then the s0 to s4 can be used during LDS save... + ////////////////////////////// + + // SGPR SR memory offset : size(VGPR) + get_vgpr_size_bytes(s_save_mem_offset) + // TODO, change RSRC word to rearrange memory layout for SGPRS + + s_getreg_b32 s_save_alloc_size, hwreg(HW_REG_GPR_ALLOC,SQ_WAVE_GPR_ALLOC_SGPR_SIZE_SHIFT,SQ_WAVE_GPR_ALLOC_SGPR_SIZE_SIZE) //spgr_size + s_add_u32 s_save_alloc_size, s_save_alloc_size, 1 + s_lshl_b32 s_save_alloc_size, s_save_alloc_size, 4 //Number of SGPRs = (sgpr_size + 1) * 16 (non-zero value) + + if (SGPR_SAVE_USE_SQC) + s_lshl_b32 s_save_buf_rsrc2, s_save_alloc_size, 2 //NUM_RECORDS in bytes + else + s_lshl_b32 s_save_buf_rsrc2, s_save_alloc_size, 8 //NUM_RECORDS in bytes (64 threads) + end + + if (SWIZZLE_EN) + s_add_u32 s_save_buf_rsrc2, s_save_buf_rsrc2, 0x0 //FIXME need to use swizzle to enable bounds checking? + else + s_mov_b32 s_save_buf_rsrc2, 0x1000000 //NUM_RECORDS in bytes + end + + + // backup s_save_buf_rsrc0,1 to s_save_pc_lo/hi, since write_16sgpr_to_mem function will change the rsrc0 + //s_mov_b64 s_save_pc_lo, s_save_buf_rsrc0 + s_mov_b64 s_save_xnack_mask_lo, s_save_buf_rsrc0 + s_add_u32 s_save_buf_rsrc0, s_save_buf_rsrc0, s_save_mem_offset + s_addc_u32 s_save_buf_rsrc1, s_save_buf_rsrc1, 0 + + s_mov_b32 m0, 0x0 //SGPR initial index value =0 + s_nop 0x0 //Manually inserted wait states + L_SAVE_SGPR_LOOP: + // SGPR is allocated in 16 SGPR granularity + s_movrels_b64 s0, s0 //s0 = s[0+m0], s1 = s[1+m0] + s_movrels_b64 s2, s2 //s2 = s[2+m0], s3 = s[3+m0] + s_movrels_b64 s4, s4 //s4 = s[4+m0], s5 = s[5+m0] + s_movrels_b64 s6, s6 //s6 = s[6+m0], s7 = s[7+m0] + s_movrels_b64 s8, s8 //s8 = s[8+m0], s9 = s[9+m0] + s_movrels_b64 s10, s10 //s10 = s[10+m0], s11 = s[11+m0] + s_movrels_b64 s12, s12 //s12 = s[12+m0], s13 = s[13+m0] + s_movrels_b64 s14, s14 //s14 = s[14+m0], s15 = s[15+m0] + + write_16sgpr_to_mem(s0, s_save_buf_rsrc0, s_save_mem_offset) //PV: the best performance should be using s_buffer_store_dwordx4 + s_add_u32 m0, m0, 16 //next sgpr index + s_cmp_lt_u32 m0, s_save_alloc_size //scc = (m0 < s_save_alloc_size) ? 1 : 0 + s_cbranch_scc1 L_SAVE_SGPR_LOOP //SGPR save is complete? + // restore s_save_buf_rsrc0,1 + //s_mov_b64 s_save_buf_rsrc0, s_save_pc_lo + s_mov_b64 s_save_buf_rsrc0, s_save_xnack_mask_lo + + + + + /* save first 4 VGPR, then LDS save could use */ + // each wave will alloc 4 vgprs at least... + ///////////////////////////////////////////////////////////////////////////////////// + + s_mov_b32 s_save_mem_offset, 0 + s_mov_b32 exec_lo, 0xFFFFFFFF //need every thread from now on + s_mov_b32 exec_hi, 0xFFFFFFFF + s_mov_b32 xnack_mask_lo, 0x0 + s_mov_b32 xnack_mask_hi, 0x0 + + if (SWIZZLE_EN) + s_add_u32 s_save_buf_rsrc2, s_save_buf_rsrc2, 0x0 //FIXME need to use swizzle to enable bounds checking? + else + s_mov_b32 s_save_buf_rsrc2, 0x1000000 //NUM_RECORDS in bytes + end + + + // VGPR Allocated in 4-GPR granularity + +if G8SR_VGPR_SR_IN_DWX4 + // the const stride for DWx4 is 4*4 bytes + s_and_b32 s_save_buf_rsrc1, s_save_buf_rsrc1, 0x0000FFFF // reset const stride to 0 + s_or_b32 s_save_buf_rsrc1, s_save_buf_rsrc1, G8SR_SAVE_BUF_RSRC_WORD1_STRIDE_DWx4 // const stride to 4*4 bytes + + buffer_store_dwordx4 v0, v0, s_save_buf_rsrc0, s_save_mem_offset slc:1 glc:1 + + s_and_b32 s_save_buf_rsrc1, s_save_buf_rsrc1, 0x0000FFFF // reset const stride to 0 + s_or_b32 s_save_buf_rsrc1, s_save_buf_rsrc1, S_SAVE_BUF_RSRC_WORD1_STRIDE // reset const stride to 4 bytes +else + buffer_store_dword v0, v0, s_save_buf_rsrc0, s_save_mem_offset slc:1 glc:1 + buffer_store_dword v1, v0, s_save_buf_rsrc0, s_save_mem_offset slc:1 glc:1 offset:256 + buffer_store_dword v2, v0, s_save_buf_rsrc0, s_save_mem_offset slc:1 glc:1 offset:256*2 + buffer_store_dword v3, v0, s_save_buf_rsrc0, s_save_mem_offset slc:1 glc:1 offset:256*3 +end + + + + /* save LDS */ + ////////////////////////////// + + L_SAVE_LDS: + + // Change EXEC to all threads... + s_mov_b32 exec_lo, 0xFFFFFFFF //need every thread from now on + s_mov_b32 exec_hi, 0xFFFFFFFF + + s_getreg_b32 s_save_alloc_size, hwreg(HW_REG_LDS_ALLOC,SQ_WAVE_LDS_ALLOC_LDS_SIZE_SHIFT,SQ_WAVE_LDS_ALLOC_LDS_SIZE_SIZE) //lds_size + s_and_b32 s_save_alloc_size, s_save_alloc_size, 0xFFFFFFFF //lds_size is zero? + s_cbranch_scc0 L_SAVE_LDS_DONE //no lds used? jump to L_SAVE_DONE + + s_barrier //LDS is used? wait for other waves in the same TG + s_and_b32 s_save_tmp, s_save_exec_hi, S_SAVE_SPI_INIT_FIRST_WAVE_MASK //exec is still used here + s_cbranch_scc0 L_SAVE_LDS_DONE + + // first wave do LDS save; + + s_lshl_b32 s_save_alloc_size, s_save_alloc_size, 6 //LDS size in dwords = lds_size * 64dw + s_lshl_b32 s_save_alloc_size, s_save_alloc_size, 2 //LDS size in bytes + s_mov_b32 s_save_buf_rsrc2, s_save_alloc_size //NUM_RECORDS in bytes + + // LDS at offset: size(VGPR)+SIZE(SGPR)+SIZE(HWREG) + // + get_vgpr_size_bytes(s_save_mem_offset) + get_sgpr_size_bytes(s_save_tmp) + s_add_u32 s_save_mem_offset, s_save_mem_offset, s_save_tmp + s_add_u32 s_save_mem_offset, s_save_mem_offset, get_hwreg_size_bytes() + + + if (SWIZZLE_EN) + s_add_u32 s_save_buf_rsrc2, s_save_buf_rsrc2, 0x0 //FIXME need to use swizzle to enable bounds checking? + else + s_mov_b32 s_save_buf_rsrc2, 0x1000000 //NUM_RECORDS in bytes + end + + s_mov_b32 m0, 0x0 //lds_offset initial value = 0 + + +var LDS_DMA_ENABLE = 0 +var UNROLL = 0 +if UNROLL==0 && LDS_DMA_ENABLE==1 + s_mov_b32 s3, 256*2 + s_nop 0 + s_nop 0 + s_nop 0 + L_SAVE_LDS_LOOP: + //TODO: looks the 2 buffer_store/load clause for s/r will hurt performance.??? + if (SAVE_LDS) //SPI always alloc LDS space in 128DW granularity + buffer_store_lds_dword s_save_buf_rsrc0, s_save_mem_offset lds:1 // first 64DW + buffer_store_lds_dword s_save_buf_rsrc0, s_save_mem_offset lds:1 offset:256 // second 64DW + end + + s_add_u32 m0, m0, s3 //every buffer_store_lds does 256 bytes + s_add_u32 s_save_mem_offset, s_save_mem_offset, s3 //mem offset increased by 256 bytes + s_cmp_lt_u32 m0, s_save_alloc_size //scc=(m0 < s_save_alloc_size) ? 1 : 0 + s_cbranch_scc1 L_SAVE_LDS_LOOP //LDS save is complete? + +elsif LDS_DMA_ENABLE==1 && UNROLL==1 // UNROOL , has ichace miss + // store from higest LDS address to lowest + s_mov_b32 s3, 256*2 + s_sub_u32 m0, s_save_alloc_size, s3 + s_add_u32 s_save_mem_offset, s_save_mem_offset, m0 + s_lshr_b32 s_save_alloc_size, s_save_alloc_size, 9 // how many 128 trunks... + s_sub_u32 s_save_alloc_size, 128, s_save_alloc_size // store from higheset addr to lowest + s_mul_i32 s_save_alloc_size, s_save_alloc_size, 6*4 // PC offset increment, each LDS save block cost 6*4 Bytes instruction + s_add_u32 s_save_alloc_size, s_save_alloc_size, 3*4 //2is the below 2 inst...//s_addc and s_setpc + s_nop 0 + s_nop 0 + s_nop 0 //pad 3 dw to let LDS_DMA align with 64Bytes + s_getpc_b64 s[0:1] // reuse s[0:1], since s[0:1] already saved + s_add_u32 s0, s0,s_save_alloc_size + s_addc_u32 s1, s1, 0 + s_setpc_b64 s[0:1] + + + for var i =0; i< 128; i++ + // be careful to make here a 64Byte aligned address, which could improve performance... + buffer_store_lds_dword s_save_buf_rsrc0, s_save_mem_offset lds:1 offset:0 // first 64DW + buffer_store_lds_dword s_save_buf_rsrc0, s_save_mem_offset lds:1 offset:256 // second 64DW + + if i!=127 + s_sub_u32 m0, m0, s3 // use a sgpr to shrink 2DW-inst to 1DW inst to improve performance , i.e. pack more LDS_DMA inst to one Cacheline + s_sub_u32 s_save_mem_offset, s_save_mem_offset, s3 + end + end + +else // BUFFER_STORE + v_mbcnt_lo_u32_b32 v2, 0xffffffff, 0x0 + v_mbcnt_hi_u32_b32 v3, 0xffffffff, v2 // tid + v_mul_i32_i24 v2, v3, 8 // tid*8 + v_mov_b32 v3, 256*2 + s_mov_b32 m0, 0x10000 + s_mov_b32 s0, s_save_buf_rsrc3 + s_and_b32 s_save_buf_rsrc3, s_save_buf_rsrc3, 0xFF7FFFFF // disable add_tid + s_or_b32 s_save_buf_rsrc3, s_save_buf_rsrc3, 0x58000 //DFMT + +L_SAVE_LDS_LOOP_VECTOR: + ds_read_b64 v[0:1], v2 //x =LDS[a], byte address + s_waitcnt lgkmcnt(0) + buffer_store_dwordx2 v[0:1], v2, s_save_buf_rsrc0, s_save_mem_offset offen:1 glc:1 slc:1 +// s_waitcnt vmcnt(0) +// v_add_u32 v2, vcc[0:1], v2, v3 + v_add_u32 v2, v2, v3 + v_cmp_lt_u32 vcc[0:1], v2, s_save_alloc_size + s_cbranch_vccnz L_SAVE_LDS_LOOP_VECTOR + + // restore rsrc3 + s_mov_b32 s_save_buf_rsrc3, s0 + +end + +L_SAVE_LDS_DONE: + + + /* save VGPRs - set the Rest VGPRs */ + ////////////////////////////////////////////////////////////////////////////////////// + L_SAVE_VGPR: + // VGPR SR memory offset: 0 + // TODO rearrange the RSRC words to use swizzle for VGPR save... + + s_mov_b32 s_save_mem_offset, (0+256*4) // for the rest VGPRs + s_mov_b32 exec_lo, 0xFFFFFFFF //need every thread from now on + s_mov_b32 exec_hi, 0xFFFFFFFF + + s_getreg_b32 s_save_alloc_size, hwreg(HW_REG_GPR_ALLOC,SQ_WAVE_GPR_ALLOC_VGPR_SIZE_SHIFT,SQ_WAVE_GPR_ALLOC_VGPR_SIZE_SIZE) //vpgr_size + s_add_u32 s_save_alloc_size, s_save_alloc_size, 1 + s_lshl_b32 s_save_alloc_size, s_save_alloc_size, 2 //Number of VGPRs = (vgpr_size + 1) * 4 (non-zero value) //FIXME for GFX, zero is possible + s_lshl_b32 s_save_buf_rsrc2, s_save_alloc_size, 8 //NUM_RECORDS in bytes (64 threads*4) + if (SWIZZLE_EN) + s_add_u32 s_save_buf_rsrc2, s_save_buf_rsrc2, 0x0 //FIXME need to use swizzle to enable bounds checking? + else + s_mov_b32 s_save_buf_rsrc2, 0x1000000 //NUM_RECORDS in bytes + end + + + // VGPR Allocated in 4-GPR granularity + +if G8SR_VGPR_SR_IN_DWX4 + // the const stride for DWx4 is 4*4 bytes + s_and_b32 s_save_buf_rsrc1, s_save_buf_rsrc1, 0x0000FFFF // reset const stride to 0 + s_or_b32 s_save_buf_rsrc1, s_save_buf_rsrc1, G8SR_SAVE_BUF_RSRC_WORD1_STRIDE_DWx4 // const stride to 4*4 bytes + + s_mov_b32 m0, 4 // skip first 4 VGPRs + s_cmp_lt_u32 m0, s_save_alloc_size + s_cbranch_scc0 L_SAVE_VGPR_LOOP_END // no more vgprs + + s_set_gpr_idx_on m0, 0x1 // This will change M0 + s_add_u32 s_save_alloc_size, s_save_alloc_size, 0x1000 // because above inst change m0 +L_SAVE_VGPR_LOOP: + v_mov_b32 v0, v0 // v0 = v[0+m0] + v_mov_b32 v1, v1 + v_mov_b32 v2, v2 + v_mov_b32 v3, v3 + + + buffer_store_dwordx4 v0, v0, s_save_buf_rsrc0, s_save_mem_offset slc:1 glc:1 + s_add_u32 m0, m0, 4 + s_add_u32 s_save_mem_offset, s_save_mem_offset, 256*4 + s_cmp_lt_u32 m0, s_save_alloc_size + s_cbranch_scc1 L_SAVE_VGPR_LOOP //VGPR save is complete? + s_set_gpr_idx_off +L_SAVE_VGPR_LOOP_END: + + s_and_b32 s_save_buf_rsrc1, s_save_buf_rsrc1, 0x0000FFFF // reset const stride to 0 + s_or_b32 s_save_buf_rsrc1, s_save_buf_rsrc1, S_SAVE_BUF_RSRC_WORD1_STRIDE // reset const stride to 4 bytes +else + // VGPR store using dw burst + s_mov_b32 m0, 0x4 //VGPR initial index value =0 + s_cmp_lt_u32 m0, s_save_alloc_size + s_cbranch_scc0 L_SAVE_VGPR_END + + + s_set_gpr_idx_on m0, 0x1 //M0[7:0] = M0[7:0] and M0[15:12] = 0x1 + s_add_u32 s_save_alloc_size, s_save_alloc_size, 0x1000 //add 0x1000 since we compare m0 against it later + + L_SAVE_VGPR_LOOP: + v_mov_b32 v0, v0 //v0 = v[0+m0] + v_mov_b32 v1, v1 //v0 = v[0+m0] + v_mov_b32 v2, v2 //v0 = v[0+m0] + v_mov_b32 v3, v3 //v0 = v[0+m0] + + if(USE_MTBUF_INSTEAD_OF_MUBUF) + tbuffer_store_format_x v0, v0, s_save_buf_rsrc0, s_save_mem_offset format:BUF_NUM_FORMAT_FLOAT format: BUF_DATA_FORMAT_32 slc:1 glc:1 + else + buffer_store_dword v0, v0, s_save_buf_rsrc0, s_save_mem_offset slc:1 glc:1 + buffer_store_dword v1, v0, s_save_buf_rsrc0, s_save_mem_offset slc:1 glc:1 offset:256 + buffer_store_dword v2, v0, s_save_buf_rsrc0, s_save_mem_offset slc:1 glc:1 offset:256*2 + buffer_store_dword v3, v0, s_save_buf_rsrc0, s_save_mem_offset slc:1 glc:1 offset:256*3 + end + + s_add_u32 m0, m0, 4 //next vgpr index + s_add_u32 s_save_mem_offset, s_save_mem_offset, 256*4 //every buffer_store_dword does 256 bytes + s_cmp_lt_u32 m0, s_save_alloc_size //scc = (m0 < s_save_alloc_size) ? 1 : 0 + s_cbranch_scc1 L_SAVE_VGPR_LOOP //VGPR save is complete? + s_set_gpr_idx_off +end + +L_SAVE_VGPR_END: + + + + + + + /* S_PGM_END_SAVED */ //FIXME graphics ONLY + if ((EMU_RUN_HACK) && (!EMU_RUN_HACK_SAVE_NORMAL_EXIT)) + s_and_b32 s_save_pc_hi, s_save_pc_hi, 0x0000ffff //pc[47:32] + s_add_u32 s_save_pc_lo, s_save_pc_lo, 4 //pc[31:0]+4 + s_addc_u32 s_save_pc_hi, s_save_pc_hi, 0x0 //carry bit over + s_rfe_b64 s_save_pc_lo //Return to the main shader program + else + end + +// Save Done timestamp +if G8SR_DEBUG_TIMESTAMP + s_memrealtime s_g8sr_ts_save_d + // SGPR SR memory offset : size(VGPR) + get_vgpr_size_bytes(s_save_mem_offset) + s_add_u32 s_save_mem_offset, s_save_mem_offset, G8SR_DEBUG_TS_SAVE_D_OFFSET + s_waitcnt lgkmcnt(0) //FIXME, will cause xnack?? + // Need reset rsrc2?? + s_mov_b32 m0, s_save_mem_offset + s_mov_b32 s_save_buf_rsrc2, 0x1000000 //NUM_RECORDS in bytes + s_buffer_store_dwordx2 s_g8sr_ts_save_d, s_save_buf_rsrc0, m0 glc:1 +end + + + s_branch L_END_PGM + + + +/**************************************************************************/ +/* restore routine */ +/**************************************************************************/ + +L_RESTORE: + /* Setup Resource Contants */ + if ((EMU_RUN_HACK) && (!EMU_RUN_HACK_RESTORE_NORMAL)) + //calculate wd_addr using absolute thread id + v_readlane_b32 s_restore_tmp, v9, 0 + s_lshr_b32 s_restore_tmp, s_restore_tmp, 6 + s_mul_i32 s_restore_tmp, s_restore_tmp, WAVE_SPACE + s_add_i32 s_restore_spi_init_lo, s_restore_tmp, WG_BASE_ADDR_LO + s_mov_b32 s_restore_spi_init_hi, WG_BASE_ADDR_HI + s_and_b32 s_restore_spi_init_hi, s_restore_spi_init_hi, CTX_RESTORE_CONTROL + else + end + +if G8SR_DEBUG_TIMESTAMP + s_memrealtime s_g8sr_ts_restore_s + s_waitcnt lgkmcnt(0) //FIXME, will cause xnack?? + // tma_lo/hi are sgpr 110, 111, which will not used for 112 SGPR allocated case... + s_mov_b32 s_restore_pc_lo, s_g8sr_ts_restore_s[0] + s_mov_b32 s_restore_pc_hi, s_g8sr_ts_restore_s[1] //backup ts to ttmp0/1, sicne exec will be finally restored.. +end + + + + s_mov_b32 s_restore_buf_rsrc0, s_restore_spi_init_lo //base_addr_lo + s_and_b32 s_restore_buf_rsrc1, s_restore_spi_init_hi, 0x0000FFFF //base_addr_hi + s_or_b32 s_restore_buf_rsrc1, s_restore_buf_rsrc1, S_RESTORE_BUF_RSRC_WORD1_STRIDE + s_mov_b32 s_restore_buf_rsrc2, 0 //NUM_RECORDS initial value = 0 (in bytes) + s_mov_b32 s_restore_buf_rsrc3, S_RESTORE_BUF_RSRC_WORD3_MISC + s_and_b32 s_restore_tmp, s_restore_spi_init_hi, S_RESTORE_SPI_INIT_ATC_MASK + s_lshr_b32 s_restore_tmp, s_restore_tmp, (S_RESTORE_SPI_INIT_ATC_SHIFT-SQ_BUF_RSRC_WORD1_ATC_SHIFT) //get ATC bit into position + s_or_b32 s_restore_buf_rsrc3, s_restore_buf_rsrc3, s_restore_tmp //or ATC + s_and_b32 s_restore_tmp, s_restore_spi_init_hi, S_RESTORE_SPI_INIT_MTYPE_MASK + s_lshr_b32 s_restore_tmp, s_restore_tmp, (S_RESTORE_SPI_INIT_MTYPE_SHIFT-SQ_BUF_RSRC_WORD3_MTYPE_SHIFT) //get MTYPE bits into position + s_or_b32 s_restore_buf_rsrc3, s_restore_buf_rsrc3, s_restore_tmp //or MTYPE + + /* global mem offset */ +// s_mov_b32 s_restore_mem_offset, 0x0 //mem offset initial value = 0 + + /* the first wave in the threadgroup */ + s_and_b32 s_restore_tmp, s_restore_spi_init_hi, S_RESTORE_SPI_INIT_FIRST_WAVE_MASK + s_cbranch_scc0 L_RESTORE_VGPR + + /* restore LDS */ + ////////////////////////////// + L_RESTORE_LDS: + + s_mov_b32 exec_lo, 0xFFFFFFFF //need every thread from now on //be consistent with SAVE although can be moved ahead + s_mov_b32 exec_hi, 0xFFFFFFFF + + s_getreg_b32 s_restore_alloc_size, hwreg(HW_REG_LDS_ALLOC,SQ_WAVE_LDS_ALLOC_LDS_SIZE_SHIFT,SQ_WAVE_LDS_ALLOC_LDS_SIZE_SIZE) //lds_size + s_and_b32 s_restore_alloc_size, s_restore_alloc_size, 0xFFFFFFFF //lds_size is zero? + s_cbranch_scc0 L_RESTORE_VGPR //no lds used? jump to L_RESTORE_VGPR + s_lshl_b32 s_restore_alloc_size, s_restore_alloc_size, 6 //LDS size in dwords = lds_size * 64dw + s_lshl_b32 s_restore_alloc_size, s_restore_alloc_size, 2 //LDS size in bytes + s_mov_b32 s_restore_buf_rsrc2, s_restore_alloc_size //NUM_RECORDS in bytes + + // LDS at offset: size(VGPR)+SIZE(SGPR)+SIZE(HWREG) + // + get_vgpr_size_bytes(s_restore_mem_offset) + get_sgpr_size_bytes(s_restore_tmp) + s_add_u32 s_restore_mem_offset, s_restore_mem_offset, s_restore_tmp + s_add_u32 s_restore_mem_offset, s_restore_mem_offset, get_hwreg_size_bytes() //FIXME, Check if offset overflow??? + + + if (SWIZZLE_EN) + s_add_u32 s_restore_buf_rsrc2, s_restore_buf_rsrc2, 0x0 //FIXME need to use swizzle to enable bounds checking? + else + s_mov_b32 s_restore_buf_rsrc2, 0x1000000 //NUM_RECORDS in bytes + end + s_mov_b32 m0, 0x0 //lds_offset initial value = 0 + + L_RESTORE_LDS_LOOP: + if (SAVE_LDS) + buffer_load_dword v0, v0, s_restore_buf_rsrc0, s_restore_mem_offset lds:1 // first 64DW + buffer_load_dword v0, v0, s_restore_buf_rsrc0, s_restore_mem_offset lds:1 offset:256 // second 64DW + end + s_add_u32 m0, m0, 256*2 // 128 DW + s_add_u32 s_restore_mem_offset, s_restore_mem_offset, 256*2 //mem offset increased by 128DW + s_cmp_lt_u32 m0, s_restore_alloc_size //scc=(m0 < s_restore_alloc_size) ? 1 : 0 + s_cbranch_scc1 L_RESTORE_LDS_LOOP //LDS restore is complete? + + + /* restore VGPRs */ + ////////////////////////////// + L_RESTORE_VGPR: + // VGPR SR memory offset : 0 + s_mov_b32 s_restore_mem_offset, 0x0 + s_mov_b32 exec_lo, 0xFFFFFFFF //need every thread from now on //be consistent with SAVE although can be moved ahead + s_mov_b32 exec_hi, 0xFFFFFFFF + + s_getreg_b32 s_restore_alloc_size, hwreg(HW_REG_GPR_ALLOC,SQ_WAVE_GPR_ALLOC_VGPR_SIZE_SHIFT,SQ_WAVE_GPR_ALLOC_VGPR_SIZE_SIZE) //vpgr_size + s_add_u32 s_restore_alloc_size, s_restore_alloc_size, 1 + s_lshl_b32 s_restore_alloc_size, s_restore_alloc_size, 2 //Number of VGPRs = (vgpr_size + 1) * 4 (non-zero value) + s_lshl_b32 s_restore_buf_rsrc2, s_restore_alloc_size, 8 //NUM_RECORDS in bytes (64 threads*4) + if (SWIZZLE_EN) + s_add_u32 s_restore_buf_rsrc2, s_restore_buf_rsrc2, 0x0 //FIXME need to use swizzle to enable bounds checking? + else + s_mov_b32 s_restore_buf_rsrc2, 0x1000000 //NUM_RECORDS in bytes + end + +if G8SR_VGPR_SR_IN_DWX4 + get_vgpr_size_bytes(s_restore_mem_offset) + s_sub_u32 s_restore_mem_offset, s_restore_mem_offset, 256*4 + + // the const stride for DWx4 is 4*4 bytes + s_and_b32 s_restore_buf_rsrc1, s_restore_buf_rsrc1, 0x0000FFFF // reset const stride to 0 + s_or_b32 s_restore_buf_rsrc1, s_restore_buf_rsrc1, G8SR_RESTORE_BUF_RSRC_WORD1_STRIDE_DWx4 // const stride to 4*4 bytes + + s_mov_b32 m0, s_restore_alloc_size + s_set_gpr_idx_on m0, 0x8 // Note.. This will change m0 + +L_RESTORE_VGPR_LOOP: + buffer_load_dwordx4 v0, v0, s_restore_buf_rsrc0, s_restore_mem_offset slc:1 glc:1 + s_waitcnt vmcnt(0) + s_sub_u32 m0, m0, 4 + v_mov_b32 v0, v0 // v[0+m0] = v0 + v_mov_b32 v1, v1 + v_mov_b32 v2, v2 + v_mov_b32 v3, v3 + s_sub_u32 s_restore_mem_offset, s_restore_mem_offset, 256*4 + s_cmp_eq_u32 m0, 0x8000 + s_cbranch_scc0 L_RESTORE_VGPR_LOOP + s_set_gpr_idx_off + + s_and_b32 s_restore_buf_rsrc1, s_restore_buf_rsrc1, 0x0000FFFF // reset const stride to 0 + s_or_b32 s_restore_buf_rsrc1, s_restore_buf_rsrc1, S_RESTORE_BUF_RSRC_WORD1_STRIDE // const stride to 4*4 bytes + +else + // VGPR load using dw burst + s_mov_b32 s_restore_mem_offset_save, s_restore_mem_offset // restore start with v1, v0 will be the last + s_add_u32 s_restore_mem_offset, s_restore_mem_offset, 256*4 + s_mov_b32 m0, 4 //VGPR initial index value = 1 + s_set_gpr_idx_on m0, 0x8 //M0[7:0] = M0[7:0] and M0[15:12] = 0x8 + s_add_u32 s_restore_alloc_size, s_restore_alloc_size, 0x8000 //add 0x8000 since we compare m0 against it later + + L_RESTORE_VGPR_LOOP: + if(USE_MTBUF_INSTEAD_OF_MUBUF) + tbuffer_load_format_x v0, v0, s_restore_buf_rsrc0, s_restore_mem_offset format:BUF_NUM_FORMAT_FLOAT format: BUF_DATA_FORMAT_32 slc:1 glc:1 + else + buffer_load_dword v0, v0, s_restore_buf_rsrc0, s_restore_mem_offset slc:1 glc:1 + buffer_load_dword v1, v0, s_restore_buf_rsrc0, s_restore_mem_offset slc:1 glc:1 offset:256 + buffer_load_dword v2, v0, s_restore_buf_rsrc0, s_restore_mem_offset slc:1 glc:1 offset:256*2 + buffer_load_dword v3, v0, s_restore_buf_rsrc0, s_restore_mem_offset slc:1 glc:1 offset:256*3 + end + s_waitcnt vmcnt(0) //ensure data ready + v_mov_b32 v0, v0 //v[0+m0] = v0 + v_mov_b32 v1, v1 + v_mov_b32 v2, v2 + v_mov_b32 v3, v3 + s_add_u32 m0, m0, 4 //next vgpr index + s_add_u32 s_restore_mem_offset, s_restore_mem_offset, 256*4 //every buffer_load_dword does 256 bytes + s_cmp_lt_u32 m0, s_restore_alloc_size //scc = (m0 < s_restore_alloc_size) ? 1 : 0 + s_cbranch_scc1 L_RESTORE_VGPR_LOOP //VGPR restore (except v0) is complete? + s_set_gpr_idx_off + /* VGPR restore on v0 */ + if(USE_MTBUF_INSTEAD_OF_MUBUF) + tbuffer_load_format_x v0, v0, s_restore_buf_rsrc0, s_restore_mem_offset_save format:BUF_NUM_FORMAT_FLOAT format: BUF_DATA_FORMAT_32 slc:1 glc:1 + else + buffer_load_dword v0, v0, s_restore_buf_rsrc0, s_restore_mem_offset_save slc:1 glc:1 + buffer_load_dword v1, v0, s_restore_buf_rsrc0, s_restore_mem_offset_save slc:1 glc:1 offset:256 + buffer_load_dword v2, v0, s_restore_buf_rsrc0, s_restore_mem_offset_save slc:1 glc:1 offset:256*2 + buffer_load_dword v3, v0, s_restore_buf_rsrc0, s_restore_mem_offset_save slc:1 glc:1 offset:256*3 + end + +end + + /* restore SGPRs */ + ////////////////////////////// + + // SGPR SR memory offset : size(VGPR) + get_vgpr_size_bytes(s_restore_mem_offset) + get_sgpr_size_bytes(s_restore_tmp) + s_add_u32 s_restore_mem_offset, s_restore_mem_offset, s_restore_tmp + s_sub_u32 s_restore_mem_offset, s_restore_mem_offset, 16*4 // restore SGPR from S[n] to S[0], by 16 sgprs group + // TODO, change RSRC word to rearrange memory layout for SGPRS + + s_getreg_b32 s_restore_alloc_size, hwreg(HW_REG_GPR_ALLOC,SQ_WAVE_GPR_ALLOC_SGPR_SIZE_SHIFT,SQ_WAVE_GPR_ALLOC_SGPR_SIZE_SIZE) //spgr_size + s_add_u32 s_restore_alloc_size, s_restore_alloc_size, 1 + s_lshl_b32 s_restore_alloc_size, s_restore_alloc_size, 4 //Number of SGPRs = (sgpr_size + 1) * 16 (non-zero value) + + if (SGPR_SAVE_USE_SQC) + s_lshl_b32 s_restore_buf_rsrc2, s_restore_alloc_size, 2 //NUM_RECORDS in bytes + else + s_lshl_b32 s_restore_buf_rsrc2, s_restore_alloc_size, 8 //NUM_RECORDS in bytes (64 threads) + end + if (SWIZZLE_EN) + s_add_u32 s_restore_buf_rsrc2, s_restore_buf_rsrc2, 0x0 //FIXME need to use swizzle to enable bounds checking? + else + s_mov_b32 s_restore_buf_rsrc2, 0x1000000 //NUM_RECORDS in bytes + end + + s_mov_b32 m0, s_restore_alloc_size + + L_RESTORE_SGPR_LOOP: + read_16sgpr_from_mem(s0, s_restore_buf_rsrc0, s_restore_mem_offset) //PV: further performance improvement can be made + s_waitcnt lgkmcnt(0) //ensure data ready + + s_sub_u32 m0, m0, 16 // Restore from S[n] to S[0] + s_nop 0 // hazard SALU M0=> S_MOVREL + + s_movreld_b64 s0, s0 //s[0+m0] = s0 + s_movreld_b64 s2, s2 + s_movreld_b64 s4, s4 + s_movreld_b64 s6, s6 + s_movreld_b64 s8, s8 + s_movreld_b64 s10, s10 + s_movreld_b64 s12, s12 + s_movreld_b64 s14, s14 + + s_cmp_eq_u32 m0, 0 //scc = (m0 < s_restore_alloc_size) ? 1 : 0 + s_cbranch_scc0 L_RESTORE_SGPR_LOOP //SGPR restore (except s0) is complete? + + /* restore HW registers */ + ////////////////////////////// + L_RESTORE_HWREG: + + +if G8SR_DEBUG_TIMESTAMP + s_mov_b32 s_g8sr_ts_restore_s[0], s_restore_pc_lo + s_mov_b32 s_g8sr_ts_restore_s[1], s_restore_pc_hi +end + + // HWREG SR memory offset : size(VGPR)+size(SGPR) + get_vgpr_size_bytes(s_restore_mem_offset) + get_sgpr_size_bytes(s_restore_tmp) + s_add_u32 s_restore_mem_offset, s_restore_mem_offset, s_restore_tmp + + + s_mov_b32 s_restore_buf_rsrc2, 0x4 //NUM_RECORDS in bytes + if (SWIZZLE_EN) + s_add_u32 s_restore_buf_rsrc2, s_restore_buf_rsrc2, 0x0 //FIXME need to use swizzle to enable bounds checking? + else + s_mov_b32 s_restore_buf_rsrc2, 0x1000000 //NUM_RECORDS in bytes + end + + read_hwreg_from_mem(s_restore_m0, s_restore_buf_rsrc0, s_restore_mem_offset) //M0 + read_hwreg_from_mem(s_restore_pc_lo, s_restore_buf_rsrc0, s_restore_mem_offset) //PC + read_hwreg_from_mem(s_restore_pc_hi, s_restore_buf_rsrc0, s_restore_mem_offset) + read_hwreg_from_mem(s_restore_exec_lo, s_restore_buf_rsrc0, s_restore_mem_offset) //EXEC + read_hwreg_from_mem(s_restore_exec_hi, s_restore_buf_rsrc0, s_restore_mem_offset) + read_hwreg_from_mem(s_restore_status, s_restore_buf_rsrc0, s_restore_mem_offset) //STATUS + read_hwreg_from_mem(s_restore_trapsts, s_restore_buf_rsrc0, s_restore_mem_offset) //TRAPSTS + read_hwreg_from_mem(xnack_mask_lo, s_restore_buf_rsrc0, s_restore_mem_offset) //XNACK_MASK_LO + read_hwreg_from_mem(xnack_mask_hi, s_restore_buf_rsrc0, s_restore_mem_offset) //XNACK_MASK_HI + read_hwreg_from_mem(s_restore_mode, s_restore_buf_rsrc0, s_restore_mem_offset) //MODE + + s_waitcnt lgkmcnt(0) //from now on, it is safe to restore STATUS and IB_STS + + //for normal save & restore, the saved PC points to the next inst to execute, no adjustment needs to be made, otherwise: + if ((EMU_RUN_HACK) && (!EMU_RUN_HACK_RESTORE_NORMAL)) + s_add_u32 s_restore_pc_lo, s_restore_pc_lo, 8 //pc[31:0]+8 //two back-to-back s_trap are used (first for save and second for restore) + s_addc_u32 s_restore_pc_hi, s_restore_pc_hi, 0x0 //carry bit over + end + if ((EMU_RUN_HACK) && (EMU_RUN_HACK_RESTORE_NORMAL)) + s_add_u32 s_restore_pc_lo, s_restore_pc_lo, 4 //pc[31:0]+4 // save is hack through s_trap but restore is normal + s_addc_u32 s_restore_pc_hi, s_restore_pc_hi, 0x0 //carry bit over + end + + s_mov_b32 m0, s_restore_m0 + s_mov_b32 exec_lo, s_restore_exec_lo + s_mov_b32 exec_hi, s_restore_exec_hi + + s_and_b32 s_restore_m0, SQ_WAVE_TRAPSTS_PRE_SAVECTX_MASK, s_restore_trapsts + s_setreg_b32 hwreg(HW_REG_TRAPSTS, SQ_WAVE_TRAPSTS_PRE_SAVECTX_SHIFT, SQ_WAVE_TRAPSTS_PRE_SAVECTX_SIZE), s_restore_m0 + s_and_b32 s_restore_m0, SQ_WAVE_TRAPSTS_POST_SAVECTX_MASK, s_restore_trapsts + s_lshr_b32 s_restore_m0, s_restore_m0, SQ_WAVE_TRAPSTS_POST_SAVECTX_SHIFT + s_setreg_b32 hwreg(HW_REG_TRAPSTS, SQ_WAVE_TRAPSTS_POST_SAVECTX_SHIFT, SQ_WAVE_TRAPSTS_POST_SAVECTX_SIZE), s_restore_m0 + //s_setreg_b32 hwreg(HW_REG_TRAPSTS), s_restore_trapsts //don't overwrite SAVECTX bit as it may be set through external SAVECTX during restore + s_setreg_b32 hwreg(HW_REG_MODE), s_restore_mode + + // Restore trap temporaries 6-11, 13-15 initialized by SPI debug dispatch logic + // ttmp SR memory offset : size(VGPR)+size(SGPR)+0x40 + get_vgpr_size_bytes(s_restore_ttmps_lo) + get_sgpr_size_bytes(s_restore_ttmps_hi) + s_add_u32 s_restore_ttmps_lo, s_restore_ttmps_lo, s_restore_ttmps_hi + s_add_u32 s_restore_ttmps_lo, s_restore_ttmps_lo, s_restore_buf_rsrc0 + s_addc_u32 s_restore_ttmps_hi, s_restore_buf_rsrc1, 0x0 + s_and_b32 s_restore_ttmps_hi, s_restore_ttmps_hi, 0xFFFF + s_load_dwordx2 [ttmp6, ttmp7], [s_restore_ttmps_lo, s_restore_ttmps_hi], 0x40 glc:1 + s_load_dwordx4 [ttmp8, ttmp9, ttmp10, ttmp11], [s_restore_ttmps_lo, s_restore_ttmps_hi], 0x48 glc:1 + s_load_dword ttmp13, [s_restore_ttmps_lo, s_restore_ttmps_hi], 0x58 glc:1 + s_load_dwordx2 [ttmp14, ttmp15], [s_restore_ttmps_lo, s_restore_ttmps_hi], 0x5C glc:1 + s_waitcnt lgkmcnt(0) + + //reuse s_restore_m0 as a temp register + s_and_b32 s_restore_m0, s_restore_pc_hi, S_SAVE_PC_HI_RCNT_MASK + s_lshr_b32 s_restore_m0, s_restore_m0, S_SAVE_PC_HI_RCNT_SHIFT + s_lshl_b32 s_restore_m0, s_restore_m0, SQ_WAVE_IB_STS_RCNT_SHIFT + s_mov_b32 s_restore_tmp, 0x0 //IB_STS is zero + s_or_b32 s_restore_tmp, s_restore_tmp, s_restore_m0 + s_and_b32 s_restore_m0, s_restore_pc_hi, S_SAVE_PC_HI_FIRST_REPLAY_MASK + s_lshr_b32 s_restore_m0, s_restore_m0, S_SAVE_PC_HI_FIRST_REPLAY_SHIFT + s_lshl_b32 s_restore_m0, s_restore_m0, SQ_WAVE_IB_STS_FIRST_REPLAY_SHIFT + s_or_b32 s_restore_tmp, s_restore_tmp, s_restore_m0 + s_and_b32 s_restore_m0, s_restore_status, SQ_WAVE_STATUS_INST_ATC_MASK + s_lshr_b32 s_restore_m0, s_restore_m0, SQ_WAVE_STATUS_INST_ATC_SHIFT + s_setreg_b32 hwreg(HW_REG_IB_STS), s_restore_tmp + + s_and_b32 s_restore_pc_hi, s_restore_pc_hi, 0x0000ffff //pc[47:32] //Do it here in order not to affect STATUS + s_and_b64 exec, exec, exec // Restore STATUS.EXECZ, not writable by s_setreg_b32 + s_and_b64 vcc, vcc, vcc // Restore STATUS.VCCZ, not writable by s_setreg_b32 + s_setreg_b32 hwreg(HW_REG_STATUS), s_restore_status // SCC is included, which is changed by previous salu + + s_barrier //barrier to ensure the readiness of LDS before access attempts from any other wave in the same TG //FIXME not performance-optimal at this time + +if G8SR_DEBUG_TIMESTAMP + s_memrealtime s_g8sr_ts_restore_d + s_waitcnt lgkmcnt(0) +end + +// s_rfe_b64 s_restore_pc_lo //Return to the main shader program and resume execution + s_rfe_restore_b64 s_restore_pc_lo, s_restore_m0 // s_restore_m0[0] is used to set STATUS.inst_atc + + +/**************************************************************************/ +/* the END */ +/**************************************************************************/ +L_END_PGM: + s_endpgm + +end + + +/**************************************************************************/ +/* the helper functions */ +/**************************************************************************/ + +//Only for save hwreg to mem +function write_hwreg_to_mem(s, s_rsrc, s_mem_offset) + s_mov_b32 exec_lo, m0 //assuming exec_lo is not needed anymore from this point on + s_mov_b32 m0, s_mem_offset + s_buffer_store_dword s, s_rsrc, m0 glc:1 + ack_sqc_store_workaround() + s_add_u32 s_mem_offset, s_mem_offset, 4 + s_mov_b32 m0, exec_lo +end + + +// HWREG are saved before SGPRs, so all HWREG could be use. +function write_16sgpr_to_mem(s, s_rsrc, s_mem_offset) + + s_buffer_store_dwordx4 s[0], s_rsrc, 0 glc:1 + ack_sqc_store_workaround() + s_buffer_store_dwordx4 s[4], s_rsrc, 16 glc:1 + ack_sqc_store_workaround() + s_buffer_store_dwordx4 s[8], s_rsrc, 32 glc:1 + ack_sqc_store_workaround() + s_buffer_store_dwordx4 s[12], s_rsrc, 48 glc:1 + ack_sqc_store_workaround() + s_add_u32 s_rsrc[0], s_rsrc[0], 4*16 + s_addc_u32 s_rsrc[1], s_rsrc[1], 0x0 // +scc +end + + +function read_hwreg_from_mem(s, s_rsrc, s_mem_offset) + s_buffer_load_dword s, s_rsrc, s_mem_offset glc:1 + s_add_u32 s_mem_offset, s_mem_offset, 4 +end + +function read_16sgpr_from_mem(s, s_rsrc, s_mem_offset) + s_buffer_load_dwordx16 s, s_rsrc, s_mem_offset glc:1 + s_sub_u32 s_mem_offset, s_mem_offset, 4*16 +end + + + +function get_lds_size_bytes(s_lds_size_byte) + // SQ LDS granularity is 64DW, while PGM_RSRC2.lds_size is in granularity 128DW + s_getreg_b32 s_lds_size_byte, hwreg(HW_REG_LDS_ALLOC, SQ_WAVE_LDS_ALLOC_LDS_SIZE_SHIFT, SQ_WAVE_LDS_ALLOC_LDS_SIZE_SIZE) // lds_size + s_lshl_b32 s_lds_size_byte, s_lds_size_byte, 8 //LDS size in dwords = lds_size * 64 *4Bytes // granularity 64DW +end + +function get_vgpr_size_bytes(s_vgpr_size_byte) + s_getreg_b32 s_vgpr_size_byte, hwreg(HW_REG_GPR_ALLOC,SQ_WAVE_GPR_ALLOC_VGPR_SIZE_SHIFT,SQ_WAVE_GPR_ALLOC_VGPR_SIZE_SIZE) //vpgr_size + s_add_u32 s_vgpr_size_byte, s_vgpr_size_byte, 1 + s_lshl_b32 s_vgpr_size_byte, s_vgpr_size_byte, (2+8) //Number of VGPRs = (vgpr_size + 1) * 4 * 64 * 4 (non-zero value) //FIXME for GFX, zero is possible +end + +function get_sgpr_size_bytes(s_sgpr_size_byte) + s_getreg_b32 s_sgpr_size_byte, hwreg(HW_REG_GPR_ALLOC,SQ_WAVE_GPR_ALLOC_SGPR_SIZE_SHIFT,SQ_WAVE_GPR_ALLOC_SGPR_SIZE_SIZE) //spgr_size + s_add_u32 s_sgpr_size_byte, s_sgpr_size_byte, 1 + s_lshl_b32 s_sgpr_size_byte, s_sgpr_size_byte, 6 //Number of SGPRs = (sgpr_size + 1) * 16 *4 (non-zero value) +end + +function get_hwreg_size_bytes + return 128 //HWREG size 128 bytes +end + +function ack_sqc_store_workaround + if ACK_SQC_STORE + s_waitcnt lgkmcnt(0) + end +end diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c index 59808a39ecf4..f64c5551cdba 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c @@ -233,7 +233,7 @@ static int set_queue_properties_from_user(struct queue_properties *q_properties, pr_debug("Queue Size: 0x%llX, %u\n", q_properties->queue_size, args->ring_size); - pr_debug("Queue r/w Pointers: %p, %p\n", + pr_debug("Queue r/w Pointers: %px, %px\n", q_properties->read_ptr, q_properties->write_ptr); @@ -292,8 +292,16 @@ static int kfd_ioctl_create_queue(struct file *filep, struct kfd_process *p, /* Return gpu_id as doorbell offset for mmap usage */ - args->doorbell_offset = (KFD_MMAP_DOORBELL_MASK | args->gpu_id); + args->doorbell_offset = KFD_MMAP_TYPE_DOORBELL; + args->doorbell_offset |= KFD_MMAP_GPU_ID(args->gpu_id); args->doorbell_offset <<= PAGE_SHIFT; + if (KFD_IS_SOC15(dev->device_info->asic_family)) + /* On SOC15 ASICs, doorbell allocation must be + * per-device, and independent from the per-process + * queue_id. Return the doorbell offset within the + * doorbell aperture to user mode. + */ + args->doorbell_offset |= q_properties.doorbell_off; mutex_unlock(&p->mutex); @@ -1296,8 +1304,8 @@ static int kfd_ioctl_map_memory_to_gpu(struct file *filep, return -EINVAL; } - devices_arr = kmalloc(args->n_devices * sizeof(*devices_arr), - GFP_KERNEL); + devices_arr = kmalloc_array(args->n_devices, sizeof(*devices_arr), + GFP_KERNEL); if (!devices_arr) return -ENOMEM; @@ -1405,8 +1413,8 @@ static int kfd_ioctl_unmap_memory_from_gpu(struct file *filep, return -EINVAL; } - devices_arr = kmalloc(args->n_devices * sizeof(*devices_arr), - GFP_KERNEL); + devices_arr = kmalloc_array(args->n_devices, sizeof(*devices_arr), + GFP_KERNEL); if (!devices_arr) return -ENOMEM; @@ -1645,23 +1653,33 @@ err_i1: static int kfd_mmap(struct file *filp, struct vm_area_struct *vma) { struct kfd_process *process; + struct kfd_dev *dev = NULL; + unsigned long vm_pgoff; + unsigned int gpu_id; process = kfd_get_process(current); if (IS_ERR(process)) return PTR_ERR(process); - if ((vma->vm_pgoff & KFD_MMAP_DOORBELL_MASK) == - KFD_MMAP_DOORBELL_MASK) { - vma->vm_pgoff = vma->vm_pgoff ^ KFD_MMAP_DOORBELL_MASK; - return kfd_doorbell_mmap(process, vma); - } else if ((vma->vm_pgoff & KFD_MMAP_EVENTS_MASK) == - KFD_MMAP_EVENTS_MASK) { - vma->vm_pgoff = vma->vm_pgoff ^ KFD_MMAP_EVENTS_MASK; + vm_pgoff = vma->vm_pgoff; + vma->vm_pgoff = KFD_MMAP_OFFSET_VALUE_GET(vm_pgoff); + gpu_id = KFD_MMAP_GPU_ID_GET(vm_pgoff); + if (gpu_id) + dev = kfd_device_by_id(gpu_id); + + switch (vm_pgoff & KFD_MMAP_TYPE_MASK) { + case KFD_MMAP_TYPE_DOORBELL: + if (!dev) + return -ENODEV; + return kfd_doorbell_mmap(dev, process, vma); + + case KFD_MMAP_TYPE_EVENTS: return kfd_event_mmap(process, vma); - } else if ((vma->vm_pgoff & KFD_MMAP_RESERVED_MEM_MASK) == - KFD_MMAP_RESERVED_MEM_MASK) { - vma->vm_pgoff = vma->vm_pgoff ^ KFD_MMAP_RESERVED_MEM_MASK; - return kfd_reserved_mem_mmap(process, vma); + + case KFD_MMAP_TYPE_RESERVED_MEM: + if (!dev) + return -ENODEV; + return kfd_reserved_mem_mmap(dev, process, vma); } return -EFAULT; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c index 4f126ef6139b..296b3f230280 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c @@ -132,6 +132,9 @@ static struct kfd_gpu_cache_info carrizo_cache_info[] = { #define fiji_cache_info carrizo_cache_info #define polaris10_cache_info carrizo_cache_info #define polaris11_cache_info carrizo_cache_info +/* TODO - check & update Vega10 cache details */ +#define vega10_cache_info carrizo_cache_info +#define raven_cache_info carrizo_cache_info static void kfd_populated_cu_info_cpu(struct kfd_topology_device *dev, struct crat_subtype_computeunit *cu) @@ -603,6 +606,14 @@ static int kfd_fill_gpu_cache_info(struct kfd_dev *kdev, pcache_info = polaris11_cache_info; num_of_cache_types = ARRAY_SIZE(polaris11_cache_info); break; + case CHIP_VEGA10: + pcache_info = vega10_cache_info; + num_of_cache_types = ARRAY_SIZE(vega10_cache_info); + break; + case CHIP_RAVEN: + pcache_info = raven_cache_info; + num_of_cache_types = ARRAY_SIZE(raven_cache_info); + break; default: return -EINVAL; } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c index 3346699960dd..7ee6cec2c060 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c @@ -20,16 +20,13 @@ * OTHER DEALINGS IN THE SOFTWARE. */ -#if defined(CONFIG_AMD_IOMMU_V2_MODULE) || defined(CONFIG_AMD_IOMMU_V2) -#include -#endif #include #include #include #include "kfd_priv.h" #include "kfd_device_queue_manager.h" #include "kfd_pm4_headers_vi.h" -#include "cwsr_trap_handler_gfx8.asm" +#include "cwsr_trap_handler.h" #include "kfd_iommu.h" #define MQD_SIZE_ALIGNED 768 @@ -41,6 +38,7 @@ static const struct kfd_device_info kaveri_device_info = { .max_pasid_bits = 16, /* max num of queues for KV.TODO should be a dynamic value */ .max_no_of_hqd = 24, + .doorbell_size = 4, .ih_ring_entry_size = 4 * sizeof(uint32_t), .event_interrupt_class = &event_interrupt_class_cik, .num_of_watch_points = 4, @@ -55,6 +53,7 @@ static const struct kfd_device_info carrizo_device_info = { .max_pasid_bits = 16, /* max num of queues for CZ.TODO should be a dynamic value */ .max_no_of_hqd = 24, + .doorbell_size = 4, .ih_ring_entry_size = 4 * sizeof(uint32_t), .event_interrupt_class = &event_interrupt_class_cik, .num_of_watch_points = 4, @@ -70,6 +69,7 @@ static const struct kfd_device_info hawaii_device_info = { .max_pasid_bits = 16, /* max num of queues for KV.TODO should be a dynamic value */ .max_no_of_hqd = 24, + .doorbell_size = 4, .ih_ring_entry_size = 4 * sizeof(uint32_t), .event_interrupt_class = &event_interrupt_class_cik, .num_of_watch_points = 4, @@ -83,6 +83,7 @@ static const struct kfd_device_info tonga_device_info = { .asic_family = CHIP_TONGA, .max_pasid_bits = 16, .max_no_of_hqd = 24, + .doorbell_size = 4, .ih_ring_entry_size = 4 * sizeof(uint32_t), .event_interrupt_class = &event_interrupt_class_cik, .num_of_watch_points = 4, @@ -96,6 +97,7 @@ static const struct kfd_device_info tonga_vf_device_info = { .asic_family = CHIP_TONGA, .max_pasid_bits = 16, .max_no_of_hqd = 24, + .doorbell_size = 4, .ih_ring_entry_size = 4 * sizeof(uint32_t), .event_interrupt_class = &event_interrupt_class_cik, .num_of_watch_points = 4, @@ -109,6 +111,7 @@ static const struct kfd_device_info fiji_device_info = { .asic_family = CHIP_FIJI, .max_pasid_bits = 16, .max_no_of_hqd = 24, + .doorbell_size = 4, .ih_ring_entry_size = 4 * sizeof(uint32_t), .event_interrupt_class = &event_interrupt_class_cik, .num_of_watch_points = 4, @@ -122,6 +125,7 @@ static const struct kfd_device_info fiji_vf_device_info = { .asic_family = CHIP_FIJI, .max_pasid_bits = 16, .max_no_of_hqd = 24, + .doorbell_size = 4, .ih_ring_entry_size = 4 * sizeof(uint32_t), .event_interrupt_class = &event_interrupt_class_cik, .num_of_watch_points = 4, @@ -136,6 +140,7 @@ static const struct kfd_device_info polaris10_device_info = { .asic_family = CHIP_POLARIS10, .max_pasid_bits = 16, .max_no_of_hqd = 24, + .doorbell_size = 4, .ih_ring_entry_size = 4 * sizeof(uint32_t), .event_interrupt_class = &event_interrupt_class_cik, .num_of_watch_points = 4, @@ -149,6 +154,7 @@ static const struct kfd_device_info polaris10_vf_device_info = { .asic_family = CHIP_POLARIS10, .max_pasid_bits = 16, .max_no_of_hqd = 24, + .doorbell_size = 4, .ih_ring_entry_size = 4 * sizeof(uint32_t), .event_interrupt_class = &event_interrupt_class_cik, .num_of_watch_points = 4, @@ -162,6 +168,7 @@ static const struct kfd_device_info polaris11_device_info = { .asic_family = CHIP_POLARIS11, .max_pasid_bits = 16, .max_no_of_hqd = 24, + .doorbell_size = 4, .ih_ring_entry_size = 4 * sizeof(uint32_t), .event_interrupt_class = &event_interrupt_class_cik, .num_of_watch_points = 4, @@ -171,6 +178,34 @@ static const struct kfd_device_info polaris11_device_info = { .needs_pci_atomics = true, }; +static const struct kfd_device_info vega10_device_info = { + .asic_family = CHIP_VEGA10, + .max_pasid_bits = 16, + .max_no_of_hqd = 24, + .doorbell_size = 8, + .ih_ring_entry_size = 8 * sizeof(uint32_t), + .event_interrupt_class = &event_interrupt_class_v9, + .num_of_watch_points = 4, + .mqd_size_aligned = MQD_SIZE_ALIGNED, + .supports_cwsr = true, + .needs_iommu_device = false, + .needs_pci_atomics = false, +}; + +static const struct kfd_device_info vega10_vf_device_info = { + .asic_family = CHIP_VEGA10, + .max_pasid_bits = 16, + .max_no_of_hqd = 24, + .doorbell_size = 8, + .ih_ring_entry_size = 8 * sizeof(uint32_t), + .event_interrupt_class = &event_interrupt_class_v9, + .num_of_watch_points = 4, + .mqd_size_aligned = MQD_SIZE_ALIGNED, + .supports_cwsr = true, + .needs_iommu_device = false, + .needs_pci_atomics = false, +}; + struct kfd_deviceid { unsigned short did; @@ -250,6 +285,15 @@ static const struct kfd_deviceid supported_devices[] = { { 0x67EB, &polaris11_device_info }, /* Polaris11 */ { 0x67EF, &polaris11_device_info }, /* Polaris11 */ { 0x67FF, &polaris11_device_info }, /* Polaris11 */ + { 0x6860, &vega10_device_info }, /* Vega10 */ + { 0x6861, &vega10_device_info }, /* Vega10 */ + { 0x6862, &vega10_device_info }, /* Vega10 */ + { 0x6863, &vega10_device_info }, /* Vega10 */ + { 0x6864, &vega10_device_info }, /* Vega10 */ + { 0x6867, &vega10_device_info }, /* Vega10 */ + { 0x6868, &vega10_device_info }, /* Vega10 */ + { 0x686C, &vega10_vf_device_info }, /* Vega10 vf*/ + { 0x687F, &vega10_device_info }, /* Vega10 */ }; static int kfd_gtt_sa_init(struct kfd_dev *kfd, unsigned int buf_size, @@ -279,7 +323,7 @@ struct kfd_dev *kgd2kfd_probe(struct kgd_dev *kgd, struct pci_dev *pdev, const struct kfd2kgd_calls *f2g) { struct kfd_dev *kfd; - + int ret; const struct kfd_device_info *device_info = lookup_device_info(pdev->device); @@ -288,19 +332,18 @@ struct kfd_dev *kgd2kfd_probe(struct kgd_dev *kgd, return NULL; } - if (device_info->needs_pci_atomics) { - /* Allow BIF to recode atomics to PCIe 3.0 - * AtomicOps. 32 and 64-bit requests are possible and - * must be supported. - */ - if (pci_enable_atomic_ops_to_root(pdev, - PCI_EXP_DEVCAP2_ATOMIC_COMP32 | - PCI_EXP_DEVCAP2_ATOMIC_COMP64) < 0) { - dev_info(kfd_device, - "skipped device %x:%x, PCI rejects atomics", - pdev->vendor, pdev->device); - return NULL; - } + /* Allow BIF to recode atomics to PCIe 3.0 AtomicOps. + * 32 and 64-bit requests are possible and must be + * supported. + */ + ret = pci_enable_atomic_ops_to_root(pdev, + PCI_EXP_DEVCAP2_ATOMIC_COMP32 | + PCI_EXP_DEVCAP2_ATOMIC_COMP64); + if (device_info->needs_pci_atomics && ret < 0) { + dev_info(kfd_device, + "skipped device %x:%x, PCI rejects atomics\n", + pdev->vendor, pdev->device); + return NULL; } kfd = kzalloc(sizeof(*kfd), GFP_KERNEL); @@ -323,10 +366,16 @@ struct kfd_dev *kgd2kfd_probe(struct kgd_dev *kgd, static void kfd_cwsr_init(struct kfd_dev *kfd) { if (cwsr_enable && kfd->device_info->supports_cwsr) { - BUILD_BUG_ON(sizeof(cwsr_trap_gfx8_hex) > PAGE_SIZE); + if (kfd->device_info->asic_family < CHIP_VEGA10) { + BUILD_BUG_ON(sizeof(cwsr_trap_gfx8_hex) > PAGE_SIZE); + kfd->cwsr_isa = cwsr_trap_gfx8_hex; + kfd->cwsr_isa_size = sizeof(cwsr_trap_gfx8_hex); + } else { + BUILD_BUG_ON(sizeof(cwsr_trap_gfx9_hex) > PAGE_SIZE); + kfd->cwsr_isa = cwsr_trap_gfx9_hex; + kfd->cwsr_isa_size = sizeof(cwsr_trap_gfx9_hex); + } - kfd->cwsr_isa = cwsr_trap_gfx8_hex; - kfd->cwsr_isa_size = sizeof(cwsr_trap_gfx8_hex); kfd->cwsr_enabled = true; } } @@ -541,6 +590,44 @@ void kgd2kfd_interrupt(struct kfd_dev *kfd, const void *ih_ring_entry) spin_unlock(&kfd->interrupt_lock); } +int kgd2kfd_quiesce_mm(struct mm_struct *mm) +{ + struct kfd_process *p; + int r; + + /* Because we are called from arbitrary context (workqueue) as opposed + * to process context, kfd_process could attempt to exit while we are + * running so the lookup function increments the process ref count. + */ + p = kfd_lookup_process_by_mm(mm); + if (!p) + return -ESRCH; + + r = kfd_process_evict_queues(p); + + kfd_unref_process(p); + return r; +} + +int kgd2kfd_resume_mm(struct mm_struct *mm) +{ + struct kfd_process *p; + int r; + + /* Because we are called from arbitrary context (workqueue) as opposed + * to process context, kfd_process could attempt to exit while we are + * running so the lookup function increments the process ref count. + */ + p = kfd_lookup_process_by_mm(mm); + if (!p) + return -ESRCH; + + r = kfd_process_restore_queues(p); + + kfd_unref_process(p); + return r; +} + /** kgd2kfd_schedule_evict_and_restore_process - Schedules work queue that will * prepare for safe eviction of KFD BOs that belong to the specified * process. @@ -652,7 +739,7 @@ int kfd_gtt_sa_allocate(struct kfd_dev *kfd, unsigned int size, if (size > kfd->gtt_sa_num_of_chunks * kfd->gtt_sa_chunk_size) return -ENOMEM; - *mem_obj = kmalloc(sizeof(struct kfd_mem_obj), GFP_KERNEL); + *mem_obj = kzalloc(sizeof(struct kfd_mem_obj), GFP_NOIO); if ((*mem_obj) == NULL) return -ENOMEM; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c index d55d29d31da4..668ad07ebe1f 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c @@ -110,6 +110,57 @@ void program_sh_mem_settings(struct device_queue_manager *dqm, qpd->sh_mem_bases); } +static int allocate_doorbell(struct qcm_process_device *qpd, struct queue *q) +{ + struct kfd_dev *dev = qpd->dqm->dev; + + if (!KFD_IS_SOC15(dev->device_info->asic_family)) { + /* On pre-SOC15 chips we need to use the queue ID to + * preserve the user mode ABI. + */ + q->doorbell_id = q->properties.queue_id; + } else if (q->properties.type == KFD_QUEUE_TYPE_SDMA) { + /* For SDMA queues on SOC15, use static doorbell + * assignments based on the engine and queue. + */ + q->doorbell_id = dev->shared_resources.sdma_doorbell + [q->properties.sdma_engine_id] + [q->properties.sdma_queue_id]; + } else { + /* For CP queues on SOC15 reserve a free doorbell ID */ + unsigned int found; + + found = find_first_zero_bit(qpd->doorbell_bitmap, + KFD_MAX_NUM_OF_QUEUES_PER_PROCESS); + if (found >= KFD_MAX_NUM_OF_QUEUES_PER_PROCESS) { + pr_debug("No doorbells available"); + return -EBUSY; + } + set_bit(found, qpd->doorbell_bitmap); + q->doorbell_id = found; + } + + q->properties.doorbell_off = + kfd_doorbell_id_to_offset(dev, q->process, + q->doorbell_id); + + return 0; +} + +static void deallocate_doorbell(struct qcm_process_device *qpd, + struct queue *q) +{ + unsigned int old; + struct kfd_dev *dev = qpd->dqm->dev; + + if (!KFD_IS_SOC15(dev->device_info->asic_family) || + q->properties.type == KFD_QUEUE_TYPE_SDMA) + return; + + old = test_and_clear_bit(q->doorbell_id, qpd->doorbell_bitmap); + WARN_ON(!old); +} + static int allocate_vmid(struct device_queue_manager *dqm, struct qcm_process_device *qpd, struct queue *q) @@ -145,15 +196,19 @@ static int allocate_vmid(struct device_queue_manager *dqm, static int flush_texture_cache_nocpsch(struct kfd_dev *kdev, struct qcm_process_device *qpd) { - uint32_t len; + const struct packet_manager_funcs *pmf = qpd->dqm->packets.pmf; + int ret; if (!qpd->ib_kaddr) return -ENOMEM; - len = pm_create_release_mem(qpd->ib_base, (uint32_t *)qpd->ib_kaddr); + ret = pmf->release_mem(qpd->ib_base, (uint32_t *)qpd->ib_kaddr); + if (ret) + return ret; return kdev->kfd2kgd->submit_ib(kdev->kgd, KGD_ENGINE_MEC1, qpd->vmid, - qpd->ib_base, (uint32_t *)qpd->ib_kaddr, len); + qpd->ib_base, (uint32_t *)qpd->ib_kaddr, + pmf->release_mem_size / sizeof(uint32_t)); } static void deallocate_vmid(struct device_queue_manager *dqm, @@ -301,10 +356,14 @@ static int create_compute_queue_nocpsch(struct device_queue_manager *dqm, if (retval) return retval; + retval = allocate_doorbell(qpd, q); + if (retval) + goto out_deallocate_hqd; + retval = mqd->init_mqd(mqd, &q->mqd, &q->mqd_mem_obj, &q->gart_mqd_addr, &q->properties); if (retval) - goto out_deallocate_hqd; + goto out_deallocate_doorbell; pr_debug("Loading mqd to hqd on pipe %d, queue %d\n", q->pipe, q->queue); @@ -324,6 +383,8 @@ static int create_compute_queue_nocpsch(struct device_queue_manager *dqm, out_uninit_mqd: mqd->uninit_mqd(mqd, q->mqd, q->mqd_mem_obj); +out_deallocate_doorbell: + deallocate_doorbell(qpd, q); out_deallocate_hqd: deallocate_hqd(dqm, q); @@ -357,6 +418,8 @@ static int destroy_queue_nocpsch_locked(struct device_queue_manager *dqm, } dqm->total_queue_count--; + deallocate_doorbell(qpd, q); + retval = mqd->destroy_mqd(mqd, q->mqd, KFD_PREEMPT_TYPE_WAVEFRONT_RESET, KFD_UNMAP_LATENCY_MS, @@ -861,6 +924,10 @@ static int create_sdma_queue_nocpsch(struct device_queue_manager *dqm, q->properties.sdma_queue_id = q->sdma_id / CIK_SDMA_QUEUES_PER_ENGINE; q->properties.sdma_engine_id = q->sdma_id % CIK_SDMA_QUEUES_PER_ENGINE; + retval = allocate_doorbell(qpd, q); + if (retval) + goto out_deallocate_sdma_queue; + pr_debug("SDMA id is: %d\n", q->sdma_id); pr_debug("SDMA queue id: %d\n", q->properties.sdma_queue_id); pr_debug("SDMA engine id: %d\n", q->properties.sdma_engine_id); @@ -869,7 +936,7 @@ static int create_sdma_queue_nocpsch(struct device_queue_manager *dqm, retval = mqd->init_mqd(mqd, &q->mqd, &q->mqd_mem_obj, &q->gart_mqd_addr, &q->properties); if (retval) - goto out_deallocate_sdma_queue; + goto out_deallocate_doorbell; retval = mqd->load_mqd(mqd, q->mqd, 0, 0, &q->properties, NULL); if (retval) @@ -879,6 +946,8 @@ static int create_sdma_queue_nocpsch(struct device_queue_manager *dqm, out_uninit_mqd: mqd->uninit_mqd(mqd, q->mqd, q->mqd_mem_obj); +out_deallocate_doorbell: + deallocate_doorbell(qpd, q); out_deallocate_sdma_queue: deallocate_sdma_queue(dqm, q->sdma_id); @@ -1070,12 +1139,17 @@ static int create_queue_cpsch(struct device_queue_manager *dqm, struct queue *q, q->properties.sdma_engine_id = q->sdma_id % CIK_SDMA_QUEUES_PER_ENGINE; } + + retval = allocate_doorbell(qpd, q); + if (retval) + goto out_deallocate_sdma_queue; + mqd = dqm->ops.get_mqd_manager(dqm, get_mqd_type_from_queue_type(q->properties.type)); if (!mqd) { retval = -ENOMEM; - goto out_deallocate_sdma_queue; + goto out_deallocate_doorbell; } /* * Eviction state logic: we only mark active queues as evicted @@ -1093,7 +1167,7 @@ static int create_queue_cpsch(struct device_queue_manager *dqm, struct queue *q, retval = mqd->init_mqd(mqd, &q->mqd, &q->mqd_mem_obj, &q->gart_mqd_addr, &q->properties); if (retval) - goto out_deallocate_sdma_queue; + goto out_deallocate_doorbell; list_add(&q->list, &qpd->queues_list); qpd->queue_count++; @@ -1117,6 +1191,8 @@ static int create_queue_cpsch(struct device_queue_manager *dqm, struct queue *q, mutex_unlock(&dqm->lock); return retval; +out_deallocate_doorbell: + deallocate_doorbell(qpd, q); out_deallocate_sdma_queue: if (q->properties.type == KFD_QUEUE_TYPE_SDMA) deallocate_sdma_queue(dqm, q->sdma_id); @@ -1257,6 +1333,8 @@ static int destroy_queue_cpsch(struct device_queue_manager *dqm, goto failed; } + deallocate_doorbell(qpd, q); + if (q->properties.type == KFD_QUEUE_TYPE_SDMA) { dqm->sdma_queue_count--; deallocate_sdma_queue(dqm, q->sdma_id); @@ -1308,7 +1386,10 @@ static bool set_cache_memory_policy(struct device_queue_manager *dqm, void __user *alternate_aperture_base, uint64_t alternate_aperture_size) { - bool retval; + bool retval = true; + + if (!dqm->asic_ops.set_cache_memory_policy) + return retval; mutex_lock(&dqm->lock); @@ -1577,6 +1658,11 @@ struct device_queue_manager *device_queue_manager_init(struct kfd_dev *dev) case CHIP_POLARIS11: device_queue_manager_init_vi_tonga(&dqm->asic_ops); break; + + case CHIP_VEGA10: + case CHIP_RAVEN: + device_queue_manager_init_v9(&dqm->asic_ops); + break; default: WARN(1, "Unexpected ASIC family %u", dev->device_info->asic_family); @@ -1627,6 +1713,18 @@ int dqm_debugfs_hqds(struct seq_file *m, void *data) int pipe, queue; int r = 0; + r = dqm->dev->kfd2kgd->hqd_dump(dqm->dev->kgd, + KFD_CIK_HIQ_PIPE, KFD_CIK_HIQ_QUEUE, &dump, &n_regs); + if (!r) { + seq_printf(m, " HIQ on MEC %d Pipe %d Queue %d\n", + KFD_CIK_HIQ_PIPE/get_pipes_per_mec(dqm)+1, + KFD_CIK_HIQ_PIPE%get_pipes_per_mec(dqm), + KFD_CIK_HIQ_QUEUE); + seq_reg_dump(m, dump, n_regs); + + kfree(dump); + } + for (pipe = 0; pipe < get_pipes_per_mec(dqm); pipe++) { int pipe_offset = pipe * get_queues_per_pipe(dqm); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h index 412beff3281d..59a6b1956932 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h @@ -200,6 +200,8 @@ void device_queue_manager_init_vi( struct device_queue_manager_asic_ops *asic_ops); void device_queue_manager_init_vi_tonga( struct device_queue_manager_asic_ops *asic_ops); +void device_queue_manager_init_v9( + struct device_queue_manager_asic_ops *asic_ops); void program_sh_mem_settings(struct device_queue_manager *dqm, struct qcm_process_device *qpd); unsigned int get_queues_num(struct device_queue_manager *dqm); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager_v9.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager_v9.c new file mode 100644 index 000000000000..79e5bcf6367c --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager_v9.c @@ -0,0 +1,84 @@ +/* + * Copyright 2016-2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ + +#include "kfd_device_queue_manager.h" +#include "vega10_enum.h" +#include "gc/gc_9_0_offset.h" +#include "gc/gc_9_0_sh_mask.h" +#include "sdma0/sdma0_4_0_sh_mask.h" + +static int update_qpd_v9(struct device_queue_manager *dqm, + struct qcm_process_device *qpd); +static void init_sdma_vm_v9(struct device_queue_manager *dqm, struct queue *q, + struct qcm_process_device *qpd); + +void device_queue_manager_init_v9( + struct device_queue_manager_asic_ops *asic_ops) +{ + asic_ops->update_qpd = update_qpd_v9; + asic_ops->init_sdma_vm = init_sdma_vm_v9; +} + +static uint32_t compute_sh_mem_bases_64bit(struct kfd_process_device *pdd) +{ + uint32_t shared_base = pdd->lds_base >> 48; + uint32_t private_base = pdd->scratch_base >> 48; + + return (shared_base << SH_MEM_BASES__SHARED_BASE__SHIFT) | + private_base; +} + +static int update_qpd_v9(struct device_queue_manager *dqm, + struct qcm_process_device *qpd) +{ + struct kfd_process_device *pdd; + + pdd = qpd_to_pdd(qpd); + + /* check if sh_mem_config register already configured */ + if (qpd->sh_mem_config == 0) { + qpd->sh_mem_config = + SH_MEM_ALIGNMENT_MODE_UNALIGNED << + SH_MEM_CONFIG__ALIGNMENT_MODE__SHIFT; + if (vega10_noretry && + !dqm->dev->device_info->needs_iommu_device) + qpd->sh_mem_config |= + 1 << SH_MEM_CONFIG__RETRY_DISABLE__SHIFT; + + qpd->sh_mem_ape1_limit = 0; + qpd->sh_mem_ape1_base = 0; + } + + qpd->sh_mem_bases = compute_sh_mem_bases_64bit(pdd); + + pr_debug("sh_mem_bases 0x%X\n", qpd->sh_mem_bases); + + return 0; +} + +static void init_sdma_vm_v9(struct device_queue_manager *dqm, struct queue *q, + struct qcm_process_device *qpd) +{ + /* Not needed on SDMAv4 any more */ + q->properties.sdma_vm_addr = 0; +} diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c b/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c index ebb4da14e3df..c3744d89352c 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c @@ -33,7 +33,6 @@ static DEFINE_IDA(doorbell_ida); static unsigned int max_doorbell_slices; -#define KFD_SIZE_OF_DOORBELL_IN_BYTES 4 /* * Each device exposes a doorbell aperture, a PCI MMIO aperture that @@ -50,9 +49,9 @@ static unsigned int max_doorbell_slices; */ /* # of doorbell bytes allocated for each process. */ -static inline size_t doorbell_process_allocation(void) +size_t kfd_doorbell_process_slice(struct kfd_dev *kfd) { - return roundup(KFD_SIZE_OF_DOORBELL_IN_BYTES * + return roundup(kfd->device_info->doorbell_size * KFD_MAX_NUM_OF_QUEUES_PER_PROCESS, PAGE_SIZE); } @@ -72,16 +71,16 @@ int kfd_doorbell_init(struct kfd_dev *kfd) doorbell_start_offset = roundup(kfd->shared_resources.doorbell_start_offset, - doorbell_process_allocation()); + kfd_doorbell_process_slice(kfd)); doorbell_aperture_size = rounddown(kfd->shared_resources.doorbell_aperture_size, - doorbell_process_allocation()); + kfd_doorbell_process_slice(kfd)); if (doorbell_aperture_size > doorbell_start_offset) doorbell_process_limit = (doorbell_aperture_size - doorbell_start_offset) / - doorbell_process_allocation(); + kfd_doorbell_process_slice(kfd); else return -ENOSPC; @@ -95,7 +94,7 @@ int kfd_doorbell_init(struct kfd_dev *kfd) kfd->doorbell_id_offset = doorbell_start_offset / sizeof(u32); kfd->doorbell_kernel_ptr = ioremap(kfd->doorbell_base, - doorbell_process_allocation()); + kfd_doorbell_process_slice(kfd)); if (!kfd->doorbell_kernel_ptr) return -ENOMEM; @@ -127,21 +126,16 @@ void kfd_doorbell_fini(struct kfd_dev *kfd) iounmap(kfd->doorbell_kernel_ptr); } -int kfd_doorbell_mmap(struct kfd_process *process, struct vm_area_struct *vma) +int kfd_doorbell_mmap(struct kfd_dev *dev, struct kfd_process *process, + struct vm_area_struct *vma) { phys_addr_t address; - struct kfd_dev *dev; /* * For simplicitly we only allow mapping of the entire doorbell * allocation of a single device & process. */ - if (vma->vm_end - vma->vm_start != doorbell_process_allocation()) - return -EINVAL; - - /* Find kfd device according to gpu id */ - dev = kfd_device_by_id(vma->vm_pgoff); - if (!dev) + if (vma->vm_end - vma->vm_start != kfd_doorbell_process_slice(dev)) return -EINVAL; /* Calculate physical address of doorbell */ @@ -158,19 +152,19 @@ int kfd_doorbell_mmap(struct kfd_process *process, struct vm_area_struct *vma) " vm_flags == 0x%04lX\n" " size == 0x%04lX\n", (unsigned long long) vma->vm_start, address, vma->vm_flags, - doorbell_process_allocation()); + kfd_doorbell_process_slice(dev)); return io_remap_pfn_range(vma, vma->vm_start, address >> PAGE_SHIFT, - doorbell_process_allocation(), + kfd_doorbell_process_slice(dev), vma->vm_page_prot); } /* get kernel iomem pointer for a doorbell */ -u32 __iomem *kfd_get_kernel_doorbell(struct kfd_dev *kfd, +void __iomem *kfd_get_kernel_doorbell(struct kfd_dev *kfd, unsigned int *doorbell_off) { u32 inx; @@ -185,6 +179,8 @@ u32 __iomem *kfd_get_kernel_doorbell(struct kfd_dev *kfd, if (inx >= KFD_MAX_NUM_OF_QUEUES_PER_PROCESS) return NULL; + inx *= kfd->device_info->doorbell_size / sizeof(u32); + /* * Calculating the kernel doorbell offset using the first * doorbell page. @@ -210,7 +206,7 @@ void kfd_release_kernel_doorbell(struct kfd_dev *kfd, u32 __iomem *db_addr) mutex_unlock(&kfd->doorbell_mutex); } -inline void write_kernel_doorbell(u32 __iomem *db, u32 value) +void write_kernel_doorbell(void __iomem *db, u32 value) { if (db) { writel(value, db); @@ -218,30 +214,37 @@ inline void write_kernel_doorbell(u32 __iomem *db, u32 value) } } -/* - * queue_ids are in the range [0,MAX_PROCESS_QUEUES) and are mapped 1:1 - * to doorbells with the process's doorbell page - */ -unsigned int kfd_queue_id_to_doorbell(struct kfd_dev *kfd, +void write_kernel_doorbell64(void __iomem *db, u64 value) +{ + if (db) { + WARN(((unsigned long)db & 7) != 0, + "Unaligned 64-bit doorbell"); + writeq(value, (u64 __iomem *)db); + pr_debug("writing %llu to doorbell address %p\n", value, db); + } +} + +unsigned int kfd_doorbell_id_to_offset(struct kfd_dev *kfd, struct kfd_process *process, - unsigned int queue_id) + unsigned int doorbell_id) { /* * doorbell_id_offset accounts for doorbells taken by KGD. - * index * doorbell_process_allocation/sizeof(u32) adjusts to - * the process's doorbells. + * index * kfd_doorbell_process_slice/sizeof(u32) adjusts to + * the process's doorbells. The offset returned is in dword + * units regardless of the ASIC-dependent doorbell size. */ return kfd->doorbell_id_offset + process->doorbell_index - * doorbell_process_allocation() / sizeof(u32) + - queue_id; + * kfd_doorbell_process_slice(kfd) / sizeof(u32) + + doorbell_id * kfd->device_info->doorbell_size / sizeof(u32); } uint64_t kfd_get_number_elems(struct kfd_dev *kfd) { uint64_t num_of_elems = (kfd->shared_resources.doorbell_aperture_size - kfd->shared_resources.doorbell_start_offset) / - doorbell_process_allocation() + 1; + kfd_doorbell_process_slice(kfd) + 1; return num_of_elems; @@ -251,7 +254,7 @@ phys_addr_t kfd_get_process_doorbells(struct kfd_dev *dev, struct kfd_process *process) { return dev->doorbell_base + - process->doorbell_index * doorbell_process_allocation(); + process->doorbell_index * kfd_doorbell_process_slice(dev); } int kfd_alloc_process_doorbells(struct kfd_process *process) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_events.c index 4890a90f1e44..5562e94e786a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c @@ -345,7 +345,7 @@ int kfd_event_create(struct file *devkfd, struct kfd_process *p, case KFD_EVENT_TYPE_DEBUG: ret = create_signal_event(devkfd, p, ev); if (!ret) { - *event_page_offset = KFD_MMAP_EVENTS_MASK; + *event_page_offset = KFD_MMAP_TYPE_EVENTS; *event_page_offset <<= PAGE_SHIFT; *event_slot_index = ev->event_id; } @@ -496,7 +496,7 @@ void kfd_signal_event_interrupt(unsigned int pasid, uint32_t partial_id, pr_debug_ratelimited("Partial ID invalid: %u (%u valid bits)\n", partial_id, valid_id_bits); - if (p->signal_event_count < KFD_SIGNAL_EVENT_LIMIT/2) { + if (p->signal_event_count < KFD_SIGNAL_EVENT_LIMIT / 64) { /* With relatively few events, it's faster to * iterate over the event IDR */ diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c index 66852de410c8..97d5423c5673 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c @@ -275,23 +275,35 @@ * for FLAT_* / S_LOAD operations. */ -#define MAKE_GPUVM_APP_BASE(gpu_num) \ +#define MAKE_GPUVM_APP_BASE_VI(gpu_num) \ (((uint64_t)(gpu_num) << 61) + 0x1000000000000L) #define MAKE_GPUVM_APP_LIMIT(base, size) \ (((uint64_t)(base) & 0xFFFFFF0000000000UL) + (size) - 1) -#define MAKE_SCRATCH_APP_BASE() \ +#define MAKE_SCRATCH_APP_BASE_VI() \ (((uint64_t)(0x1UL) << 61) + 0x100000000L) #define MAKE_SCRATCH_APP_LIMIT(base) \ (((uint64_t)base & 0xFFFFFFFF00000000UL) | 0xFFFFFFFF) -#define MAKE_LDS_APP_BASE() \ +#define MAKE_LDS_APP_BASE_VI() \ (((uint64_t)(0x1UL) << 61) + 0x0) #define MAKE_LDS_APP_LIMIT(base) \ (((uint64_t)(base) & 0xFFFFFFFF00000000UL) | 0xFFFFFFFF) +/* On GFXv9 the LDS and scratch apertures are programmed independently + * using the high 16 bits of the 64-bit virtual address. They must be + * in the hole, which will be the case as long as the high 16 bits are + * not 0. + * + * The aperture sizes are still 4GB implicitly. + * + * A GPUVM aperture is not applicable on GFXv9. + */ +#define MAKE_LDS_APP_BASE_V9() ((uint64_t)(0x1UL) << 48) +#define MAKE_SCRATCH_APP_BASE_V9() ((uint64_t)(0x2UL) << 48) + /* User mode manages most of the SVM aperture address space. The low * 16MB are reserved for kernel use (CWSR trap handler and kernel IB * for now). @@ -300,6 +312,55 @@ #define SVM_CWSR_BASE (SVM_USER_BASE - KFD_CWSR_TBA_TMA_SIZE) #define SVM_IB_BASE (SVM_CWSR_BASE - PAGE_SIZE) +static void kfd_init_apertures_vi(struct kfd_process_device *pdd, uint8_t id) +{ + /* + * node id couldn't be 0 - the three MSB bits of + * aperture shoudn't be 0 + */ + pdd->lds_base = MAKE_LDS_APP_BASE_VI(); + pdd->lds_limit = MAKE_LDS_APP_LIMIT(pdd->lds_base); + + if (!pdd->dev->device_info->needs_iommu_device) { + /* dGPUs: SVM aperture starting at 0 + * with small reserved space for kernel. + * Set them to CANONICAL addresses. + */ + pdd->gpuvm_base = SVM_USER_BASE; + pdd->gpuvm_limit = + pdd->dev->shared_resources.gpuvm_size - 1; + } else { + /* set them to non CANONICAL addresses, and no SVM is + * allocated. + */ + pdd->gpuvm_base = MAKE_GPUVM_APP_BASE_VI(id + 1); + pdd->gpuvm_limit = MAKE_GPUVM_APP_LIMIT(pdd->gpuvm_base, + pdd->dev->shared_resources.gpuvm_size); + } + + pdd->scratch_base = MAKE_SCRATCH_APP_BASE_VI(); + pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base); +} + +static void kfd_init_apertures_v9(struct kfd_process_device *pdd, uint8_t id) +{ + pdd->lds_base = MAKE_LDS_APP_BASE_V9(); + pdd->lds_limit = MAKE_LDS_APP_LIMIT(pdd->lds_base); + + /* Raven needs SVM to support graphic handle, etc. Leave the small + * reserved space before SVM on Raven as well, even though we don't + * have to. + * Set gpuvm_base and gpuvm_limit to CANONICAL addresses so that they + * are used in Thunk to reserve SVM. + */ + pdd->gpuvm_base = SVM_USER_BASE; + pdd->gpuvm_limit = + pdd->dev->shared_resources.gpuvm_size - 1; + + pdd->scratch_base = MAKE_SCRATCH_APP_BASE_V9(); + pdd->scratch_limit = MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base); +} + int kfd_init_apertures(struct kfd_process *process) { uint8_t id = 0; @@ -307,9 +368,7 @@ int kfd_init_apertures(struct kfd_process *process) struct kfd_process_device *pdd; /*Iterating over all devices*/ - while (kfd_topology_enum_kfd_devices(id, &dev) == 0 && - id < NUM_OF_SUPPORTED_GPUS) { - + while (kfd_topology_enum_kfd_devices(id, &dev) == 0) { if (!dev) { id++; /* Skip non GPU devices */ continue; @@ -318,7 +377,7 @@ int kfd_init_apertures(struct kfd_process *process) pdd = kfd_create_process_device_data(dev, process); if (!pdd) { pr_err("Failed to create process device data\n"); - return -1; + return -ENOMEM; } /* * For 64 bit process apertures will be statically reserved in @@ -330,32 +389,30 @@ int kfd_init_apertures(struct kfd_process *process) pdd->gpuvm_base = pdd->gpuvm_limit = 0; pdd->scratch_base = pdd->scratch_limit = 0; } else { - /* Same LDS and scratch apertures can be used - * on all GPUs. This allows using more dGPUs - * than placement options for apertures. - */ - pdd->lds_base = MAKE_LDS_APP_BASE(); - pdd->lds_limit = MAKE_LDS_APP_LIMIT(pdd->lds_base); - - pdd->scratch_base = MAKE_SCRATCH_APP_BASE(); - pdd->scratch_limit = - MAKE_SCRATCH_APP_LIMIT(pdd->scratch_base); + switch (dev->device_info->asic_family) { + case CHIP_KAVERI: + case CHIP_HAWAII: + case CHIP_CARRIZO: + case CHIP_TONGA: + case CHIP_FIJI: + case CHIP_POLARIS10: + case CHIP_POLARIS11: + kfd_init_apertures_vi(pdd, id); + break; + case CHIP_VEGA10: + case CHIP_RAVEN: + kfd_init_apertures_v9(pdd, id); + break; + default: + WARN(1, "Unexpected ASIC family %u", + dev->device_info->asic_family); + return -EINVAL; + } - if (dev->device_info->needs_iommu_device) { - /* APUs: GPUVM aperture in - * non-canonical address space - */ - pdd->gpuvm_base = MAKE_GPUVM_APP_BASE(id + 1); - pdd->gpuvm_limit = MAKE_GPUVM_APP_LIMIT( - pdd->gpuvm_base, - dev->shared_resources.gpuvm_size); - } else { - /* dGPUs: SVM aperture starting at 0 - * with small reserved space for kernel + if (!dev->device_info->needs_iommu_device) { + /* dGPUs: the reserved space for kernel + * before SVM */ - pdd->gpuvm_base = SVM_USER_BASE; - pdd->gpuvm_limit = - dev->shared_resources.gpuvm_size - 1; pdd->qpd.cwsr_base = SVM_CWSR_BASE; pdd->qpd.ib_base = SVM_IB_BASE; } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c b/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c new file mode 100644 index 000000000000..37029baa3346 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c @@ -0,0 +1,92 @@ +/* + * Copyright 2016-2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include "kfd_priv.h" +#include "kfd_events.h" +#include "soc15_int.h" + + +static bool event_interrupt_isr_v9(struct kfd_dev *dev, + const uint32_t *ih_ring_entry) +{ + uint16_t source_id, client_id, pasid, vmid; + const uint32_t *data = ih_ring_entry; + + /* Only handle interrupts from KFD VMIDs */ + vmid = SOC15_VMID_FROM_IH_ENTRY(ih_ring_entry); + if (vmid < dev->vm_info.first_vmid_kfd || + vmid > dev->vm_info.last_vmid_kfd) + return 0; + + /* If there is no valid PASID, it's likely a firmware bug */ + pasid = SOC15_PASID_FROM_IH_ENTRY(ih_ring_entry); + if (WARN_ONCE(pasid == 0, "FW bug: No PASID in KFD interrupt")) + return 0; + + source_id = SOC15_SOURCE_ID_FROM_IH_ENTRY(ih_ring_entry); + client_id = SOC15_CLIENT_ID_FROM_IH_ENTRY(ih_ring_entry); + + pr_debug("client id 0x%x, source id %d, pasid 0x%x. raw data:\n", + client_id, source_id, pasid); + pr_debug("%8X, %8X, %8X, %8X, %8X, %8X, %8X, %8X.\n", + data[0], data[1], data[2], data[3], + data[4], data[5], data[6], data[7]); + + /* Interrupt types we care about: various signals and faults. + * They will be forwarded to a work queue (see below). + */ + return source_id == SOC15_INTSRC_CP_END_OF_PIPE || + source_id == SOC15_INTSRC_SDMA_TRAP || + source_id == SOC15_INTSRC_SQ_INTERRUPT_MSG || + source_id == SOC15_INTSRC_CP_BAD_OPCODE; +} + +static void event_interrupt_wq_v9(struct kfd_dev *dev, + const uint32_t *ih_ring_entry) +{ + uint16_t source_id, client_id, pasid, vmid; + uint32_t context_id; + + source_id = SOC15_SOURCE_ID_FROM_IH_ENTRY(ih_ring_entry); + client_id = SOC15_CLIENT_ID_FROM_IH_ENTRY(ih_ring_entry); + pasid = SOC15_PASID_FROM_IH_ENTRY(ih_ring_entry); + vmid = SOC15_VMID_FROM_IH_ENTRY(ih_ring_entry); + context_id = SOC15_CONTEXT_ID0_FROM_IH_ENTRY(ih_ring_entry); + + if (source_id == SOC15_INTSRC_CP_END_OF_PIPE) + kfd_signal_event_interrupt(pasid, context_id, 32); + else if (source_id == SOC15_INTSRC_SDMA_TRAP) + kfd_signal_event_interrupt(pasid, context_id & 0xfffffff, 28); + else if (source_id == SOC15_INTSRC_SQ_INTERRUPT_MSG) + kfd_signal_event_interrupt(pasid, context_id & 0xffffff, 24); + else if (source_id == SOC15_INTSRC_CP_BAD_OPCODE) + kfd_signal_hw_exception_event(pasid); + else if (client_id == SOC15_IH_CLIENTID_VMC || + client_id == SOC15_IH_CLIENTID_UTCL2) { + /* TODO */ + } +} + +const struct kfd_event_interrupt_class event_interrupt_class_v9 = { + .interrupt_isr = event_interrupt_isr_v9, + .interrupt_wq = event_interrupt_wq_v9, +}; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c b/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c index 035c351f47c5..db6d9336b80d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c @@ -139,10 +139,12 @@ static void interrupt_wq(struct work_struct *work) { struct kfd_dev *dev = container_of(work, struct kfd_dev, interrupt_work); + uint32_t ih_ring_entry[KFD_MAX_RING_ENTRY_SIZE]; - uint32_t ih_ring_entry[DIV_ROUND_UP( - dev->device_info->ih_ring_entry_size, - sizeof(uint32_t))]; + if (dev->device_info->ih_ring_entry_size > sizeof(ih_ring_entry)) { + dev_err_once(kfd_chardev(), "Ring entry too small\n"); + return; + } while (dequeue_ih_ring_entry(dev, ih_ring_entry)) dev->device_info->event_interrupt_class->interrupt_wq(dev, diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c index 69f496485331..476951d8c91c 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c @@ -99,7 +99,7 @@ static bool initialize(struct kernel_queue *kq, struct kfd_dev *dev, kq->rptr_kernel = kq->rptr_mem->cpu_ptr; kq->rptr_gpu_addr = kq->rptr_mem->gpu_addr; - retval = kfd_gtt_sa_allocate(dev, sizeof(*kq->wptr_kernel), + retval = kfd_gtt_sa_allocate(dev, dev->device_info->doorbell_size, &kq->wptr_mem); if (retval != 0) @@ -208,6 +208,7 @@ static int acquire_packet_buffer(struct kernel_queue *kq, size_t available_size; size_t queue_size_dwords; uint32_t wptr, rptr; + uint64_t wptr64; unsigned int *queue_address; /* When rptr == wptr, the buffer is empty. @@ -216,7 +217,8 @@ static int acquire_packet_buffer(struct kernel_queue *kq, * the opposite. So we can only use up to queue_size_dwords - 1 dwords. */ rptr = *kq->rptr_kernel; - wptr = *kq->wptr_kernel; + wptr = kq->pending_wptr; + wptr64 = kq->pending_wptr64; queue_address = (unsigned int *)kq->pq_kernel_addr; queue_size_dwords = kq->queue->properties.queue_size / 4; @@ -232,29 +234,33 @@ static int acquire_packet_buffer(struct kernel_queue *kq, * make sure calling functions know * acquire_packet_buffer() failed */ - *buffer_ptr = NULL; - return -ENOMEM; + goto err_no_space; } if (wptr + packet_size_in_dwords >= queue_size_dwords) { /* make sure after rolling back to position 0, there is * still enough space. */ - if (packet_size_in_dwords >= rptr) { - *buffer_ptr = NULL; - return -ENOMEM; - } + if (packet_size_in_dwords >= rptr) + goto err_no_space; + /* fill nops, roll back and start at position 0 */ while (wptr > 0) { queue_address[wptr] = kq->nop_packet; wptr = (wptr + 1) % queue_size_dwords; + wptr64++; } } *buffer_ptr = &queue_address[wptr]; kq->pending_wptr = wptr + packet_size_in_dwords; + kq->pending_wptr64 = wptr64 + packet_size_in_dwords; return 0; + +err_no_space: + *buffer_ptr = NULL; + return -ENOMEM; } static void submit_packet(struct kernel_queue *kq) @@ -270,14 +276,18 @@ static void submit_packet(struct kernel_queue *kq) pr_debug("\n"); #endif - *kq->wptr_kernel = kq->pending_wptr; - write_kernel_doorbell(kq->queue->properties.doorbell_ptr, - kq->pending_wptr); + kq->ops_asic_specific.submit_packet(kq); } static void rollback_packet(struct kernel_queue *kq) { - kq->pending_wptr = *kq->queue->properties.write_ptr; + if (kq->dev->device_info->doorbell_size == 8) { + kq->pending_wptr64 = *kq->wptr64_kernel; + kq->pending_wptr = *kq->wptr_kernel % + (kq->queue->properties.queue_size / 4); + } else { + kq->pending_wptr = *kq->wptr_kernel; + } } struct kernel_queue *kernel_queue_init(struct kfd_dev *dev, @@ -308,6 +318,11 @@ struct kernel_queue *kernel_queue_init(struct kfd_dev *dev, case CHIP_HAWAII: kernel_queue_init_cik(&kq->ops_asic_specific); break; + + case CHIP_VEGA10: + case CHIP_RAVEN: + kernel_queue_init_v9(&kq->ops_asic_specific); + break; default: WARN(1, "Unexpected ASIC family %u", dev->device_info->asic_family); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h index 594053136ee4..97aff2041a5d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.h @@ -72,6 +72,7 @@ struct kernel_queue { struct kfd_dev *dev; struct mqd_manager *mqd; struct queue *queue; + uint64_t pending_wptr64; uint32_t pending_wptr; unsigned int nop_packet; @@ -79,7 +80,10 @@ struct kernel_queue { uint32_t *rptr_kernel; uint64_t rptr_gpu_addr; struct kfd_mem_obj *wptr_mem; - uint32_t *wptr_kernel; + union { + uint64_t *wptr64_kernel; + uint32_t *wptr_kernel; + }; uint64_t wptr_gpu_addr; struct kfd_mem_obj *pq; uint64_t pq_gpu_addr; @@ -97,5 +101,6 @@ struct kernel_queue { void kernel_queue_init_cik(struct kernel_queue_ops *ops); void kernel_queue_init_vi(struct kernel_queue_ops *ops); +void kernel_queue_init_v9(struct kernel_queue_ops *ops); #endif /* KFD_KERNEL_QUEUE_H_ */ diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_cik.c b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_cik.c index a90eb440b1fb..19e54acb4125 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_cik.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_cik.c @@ -26,11 +26,13 @@ static bool initialize_cik(struct kernel_queue *kq, struct kfd_dev *dev, enum kfd_queue_type type, unsigned int queue_size); static void uninitialize_cik(struct kernel_queue *kq); +static void submit_packet_cik(struct kernel_queue *kq); void kernel_queue_init_cik(struct kernel_queue_ops *ops) { ops->initialize = initialize_cik; ops->uninitialize = uninitialize_cik; + ops->submit_packet = submit_packet_cik; } static bool initialize_cik(struct kernel_queue *kq, struct kfd_dev *dev, @@ -42,3 +44,10 @@ static bool initialize_cik(struct kernel_queue *kq, struct kfd_dev *dev, static void uninitialize_cik(struct kernel_queue *kq) { } + +static void submit_packet_cik(struct kernel_queue *kq) +{ + *kq->wptr_kernel = kq->pending_wptr; + write_kernel_doorbell(kq->queue->properties.doorbell_ptr, + kq->pending_wptr); +} diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c new file mode 100644 index 000000000000..684a3bf07efd --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_v9.c @@ -0,0 +1,340 @@ +/* + * Copyright 2016-2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ + +#include "kfd_kernel_queue.h" +#include "kfd_device_queue_manager.h" +#include "kfd_pm4_headers_ai.h" +#include "kfd_pm4_opcodes.h" + +static bool initialize_v9(struct kernel_queue *kq, struct kfd_dev *dev, + enum kfd_queue_type type, unsigned int queue_size); +static void uninitialize_v9(struct kernel_queue *kq); +static void submit_packet_v9(struct kernel_queue *kq); + +void kernel_queue_init_v9(struct kernel_queue_ops *ops) +{ + ops->initialize = initialize_v9; + ops->uninitialize = uninitialize_v9; + ops->submit_packet = submit_packet_v9; +} + +static bool initialize_v9(struct kernel_queue *kq, struct kfd_dev *dev, + enum kfd_queue_type type, unsigned int queue_size) +{ + int retval; + + retval = kfd_gtt_sa_allocate(dev, PAGE_SIZE, &kq->eop_mem); + if (retval) + return false; + + kq->eop_gpu_addr = kq->eop_mem->gpu_addr; + kq->eop_kernel_addr = kq->eop_mem->cpu_ptr; + + memset(kq->eop_kernel_addr, 0, PAGE_SIZE); + + return true; +} + +static void uninitialize_v9(struct kernel_queue *kq) +{ + kfd_gtt_sa_free(kq->dev, kq->eop_mem); +} + +static void submit_packet_v9(struct kernel_queue *kq) +{ + *kq->wptr64_kernel = kq->pending_wptr64; + write_kernel_doorbell64(kq->queue->properties.doorbell_ptr, + kq->pending_wptr64); +} + +static int pm_map_process_v9(struct packet_manager *pm, + uint32_t *buffer, struct qcm_process_device *qpd) +{ + struct pm4_mes_map_process *packet; + uint64_t vm_page_table_base_addr = + (uint64_t)(qpd->page_table_base) << 12; + + packet = (struct pm4_mes_map_process *)buffer; + memset(buffer, 0, sizeof(struct pm4_mes_map_process)); + + packet->header.u32All = pm_build_pm4_header(IT_MAP_PROCESS, + sizeof(struct pm4_mes_map_process)); + packet->bitfields2.diq_enable = (qpd->is_debug) ? 1 : 0; + packet->bitfields2.process_quantum = 1; + packet->bitfields2.pasid = qpd->pqm->process->pasid; + packet->bitfields14.gds_size = qpd->gds_size; + packet->bitfields14.num_gws = qpd->num_gws; + packet->bitfields14.num_oac = qpd->num_oac; + packet->bitfields14.sdma_enable = 1; + packet->bitfields14.num_queues = (qpd->is_debug) ? 0 : qpd->queue_count; + + packet->sh_mem_config = qpd->sh_mem_config; + packet->sh_mem_bases = qpd->sh_mem_bases; + packet->sq_shader_tba_lo = lower_32_bits(qpd->tba_addr >> 8); + packet->sq_shader_tba_hi = upper_32_bits(qpd->tba_addr >> 8); + packet->sq_shader_tma_lo = lower_32_bits(qpd->tma_addr >> 8); + packet->sq_shader_tma_hi = upper_32_bits(qpd->tma_addr >> 8); + + packet->gds_addr_lo = lower_32_bits(qpd->gds_context_area); + packet->gds_addr_hi = upper_32_bits(qpd->gds_context_area); + + packet->vm_context_page_table_base_addr_lo32 = + lower_32_bits(vm_page_table_base_addr); + packet->vm_context_page_table_base_addr_hi32 = + upper_32_bits(vm_page_table_base_addr); + + return 0; +} + +static int pm_runlist_v9(struct packet_manager *pm, uint32_t *buffer, + uint64_t ib, size_t ib_size_in_dwords, bool chain) +{ + struct pm4_mes_runlist *packet; + + int concurrent_proc_cnt = 0; + struct kfd_dev *kfd = pm->dqm->dev; + + /* Determine the number of processes to map together to HW: + * it can not exceed the number of VMIDs available to the + * scheduler, and it is determined by the smaller of the number + * of processes in the runlist and kfd module parameter + * hws_max_conc_proc. + * Note: the arbitration between the number of VMIDs and + * hws_max_conc_proc has been done in + * kgd2kfd_device_init(). + */ + concurrent_proc_cnt = min(pm->dqm->processes_count, + kfd->max_proc_per_quantum); + + packet = (struct pm4_mes_runlist *)buffer; + + memset(buffer, 0, sizeof(struct pm4_mes_runlist)); + packet->header.u32All = pm_build_pm4_header(IT_RUN_LIST, + sizeof(struct pm4_mes_runlist)); + + packet->bitfields4.ib_size = ib_size_in_dwords; + packet->bitfields4.chain = chain ? 1 : 0; + packet->bitfields4.offload_polling = 0; + packet->bitfields4.valid = 1; + packet->bitfields4.process_cnt = concurrent_proc_cnt; + packet->ordinal2 = lower_32_bits(ib); + packet->ib_base_hi = upper_32_bits(ib); + + return 0; +} + +static int pm_map_queues_v9(struct packet_manager *pm, uint32_t *buffer, + struct queue *q, bool is_static) +{ + struct pm4_mes_map_queues *packet; + bool use_static = is_static; + + packet = (struct pm4_mes_map_queues *)buffer; + memset(buffer, 0, sizeof(struct pm4_mes_map_queues)); + + packet->header.u32All = pm_build_pm4_header(IT_MAP_QUEUES, + sizeof(struct pm4_mes_map_queues)); + packet->bitfields2.alloc_format = + alloc_format__mes_map_queues__one_per_pipe_vi; + packet->bitfields2.num_queues = 1; + packet->bitfields2.queue_sel = + queue_sel__mes_map_queues__map_to_hws_determined_queue_slots_vi; + + packet->bitfields2.engine_sel = + engine_sel__mes_map_queues__compute_vi; + packet->bitfields2.queue_type = + queue_type__mes_map_queues__normal_compute_vi; + + switch (q->properties.type) { + case KFD_QUEUE_TYPE_COMPUTE: + if (use_static) + packet->bitfields2.queue_type = + queue_type__mes_map_queues__normal_latency_static_queue_vi; + break; + case KFD_QUEUE_TYPE_DIQ: + packet->bitfields2.queue_type = + queue_type__mes_map_queues__debug_interface_queue_vi; + break; + case KFD_QUEUE_TYPE_SDMA: + packet->bitfields2.engine_sel = q->properties.sdma_engine_id + + engine_sel__mes_map_queues__sdma0_vi; + use_static = false; /* no static queues under SDMA */ + break; + default: + WARN(1, "queue type %d", q->properties.type); + return -EINVAL; + } + packet->bitfields3.doorbell_offset = + q->properties.doorbell_off; + + packet->mqd_addr_lo = + lower_32_bits(q->gart_mqd_addr); + + packet->mqd_addr_hi = + upper_32_bits(q->gart_mqd_addr); + + packet->wptr_addr_lo = + lower_32_bits((uint64_t)q->properties.write_ptr); + + packet->wptr_addr_hi = + upper_32_bits((uint64_t)q->properties.write_ptr); + + return 0; +} + +static int pm_unmap_queues_v9(struct packet_manager *pm, uint32_t *buffer, + enum kfd_queue_type type, + enum kfd_unmap_queues_filter filter, + uint32_t filter_param, bool reset, + unsigned int sdma_engine) +{ + struct pm4_mes_unmap_queues *packet; + + packet = (struct pm4_mes_unmap_queues *)buffer; + memset(buffer, 0, sizeof(struct pm4_mes_unmap_queues)); + + packet->header.u32All = pm_build_pm4_header(IT_UNMAP_QUEUES, + sizeof(struct pm4_mes_unmap_queues)); + switch (type) { + case KFD_QUEUE_TYPE_COMPUTE: + case KFD_QUEUE_TYPE_DIQ: + packet->bitfields2.engine_sel = + engine_sel__mes_unmap_queues__compute; + break; + case KFD_QUEUE_TYPE_SDMA: + packet->bitfields2.engine_sel = + engine_sel__mes_unmap_queues__sdma0 + sdma_engine; + break; + default: + WARN(1, "queue type %d", type); + return -EINVAL; + } + + if (reset) + packet->bitfields2.action = + action__mes_unmap_queues__reset_queues; + else + packet->bitfields2.action = + action__mes_unmap_queues__preempt_queues; + + switch (filter) { + case KFD_UNMAP_QUEUES_FILTER_SINGLE_QUEUE: + packet->bitfields2.queue_sel = + queue_sel__mes_unmap_queues__perform_request_on_specified_queues; + packet->bitfields2.num_queues = 1; + packet->bitfields3b.doorbell_offset0 = filter_param; + break; + case KFD_UNMAP_QUEUES_FILTER_BY_PASID: + packet->bitfields2.queue_sel = + queue_sel__mes_unmap_queues__perform_request_on_pasid_queues; + packet->bitfields3a.pasid = filter_param; + break; + case KFD_UNMAP_QUEUES_FILTER_ALL_QUEUES: + packet->bitfields2.queue_sel = + queue_sel__mes_unmap_queues__unmap_all_queues; + break; + case KFD_UNMAP_QUEUES_FILTER_DYNAMIC_QUEUES: + /* in this case, we do not preempt static queues */ + packet->bitfields2.queue_sel = + queue_sel__mes_unmap_queues__unmap_all_non_static_queues; + break; + default: + WARN(1, "filter %d", filter); + return -EINVAL; + } + + return 0; + +} + +static int pm_query_status_v9(struct packet_manager *pm, uint32_t *buffer, + uint64_t fence_address, uint32_t fence_value) +{ + struct pm4_mes_query_status *packet; + + packet = (struct pm4_mes_query_status *)buffer; + memset(buffer, 0, sizeof(struct pm4_mes_query_status)); + + + packet->header.u32All = pm_build_pm4_header(IT_QUERY_STATUS, + sizeof(struct pm4_mes_query_status)); + + packet->bitfields2.context_id = 0; + packet->bitfields2.interrupt_sel = + interrupt_sel__mes_query_status__completion_status; + packet->bitfields2.command = + command__mes_query_status__fence_only_after_write_ack; + + packet->addr_hi = upper_32_bits((uint64_t)fence_address); + packet->addr_lo = lower_32_bits((uint64_t)fence_address); + packet->data_hi = upper_32_bits((uint64_t)fence_value); + packet->data_lo = lower_32_bits((uint64_t)fence_value); + + return 0; +} + + +static int pm_release_mem_v9(uint64_t gpu_addr, uint32_t *buffer) +{ + struct pm4_mec_release_mem *packet; + + packet = (struct pm4_mec_release_mem *)buffer; + memset(buffer, 0, sizeof(struct pm4_mec_release_mem)); + + packet->header.u32All = pm_build_pm4_header(IT_RELEASE_MEM, + sizeof(struct pm4_mec_release_mem)); + + packet->bitfields2.event_type = CACHE_FLUSH_AND_INV_TS_EVENT; + packet->bitfields2.event_index = event_index__mec_release_mem__end_of_pipe; + packet->bitfields2.tcl1_action_ena = 1; + packet->bitfields2.tc_action_ena = 1; + packet->bitfields2.cache_policy = cache_policy__mec_release_mem__lru; + + packet->bitfields3.data_sel = data_sel__mec_release_mem__send_32_bit_low; + packet->bitfields3.int_sel = + int_sel__mec_release_mem__send_interrupt_after_write_confirm; + + packet->bitfields4.address_lo_32b = (gpu_addr & 0xffffffff) >> 2; + packet->address_hi = upper_32_bits(gpu_addr); + + packet->data_lo = 0; + + return 0; +} + +const struct packet_manager_funcs kfd_v9_pm_funcs = { + .map_process = pm_map_process_v9, + .runlist = pm_runlist_v9, + .set_resources = pm_set_resources_vi, + .map_queues = pm_map_queues_v9, + .unmap_queues = pm_unmap_queues_v9, + .query_status = pm_query_status_v9, + .release_mem = pm_release_mem_v9, + .map_process_size = sizeof(struct pm4_mes_map_process), + .runlist_size = sizeof(struct pm4_mes_runlist), + .set_resources_size = sizeof(struct pm4_mes_set_resources), + .map_queues_size = sizeof(struct pm4_mes_map_queues), + .unmap_queues_size = sizeof(struct pm4_mes_unmap_queues), + .query_status_size = sizeof(struct pm4_mes_query_status), + .release_mem_size = sizeof(struct pm4_mec_release_mem) +}; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_vi.c b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_vi.c index f1d48281e322..bf20c6d32ef3 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_vi.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue_vi.c @@ -22,15 +22,20 @@ */ #include "kfd_kernel_queue.h" +#include "kfd_device_queue_manager.h" +#include "kfd_pm4_headers_vi.h" +#include "kfd_pm4_opcodes.h" static bool initialize_vi(struct kernel_queue *kq, struct kfd_dev *dev, enum kfd_queue_type type, unsigned int queue_size); static void uninitialize_vi(struct kernel_queue *kq); +static void submit_packet_vi(struct kernel_queue *kq); void kernel_queue_init_vi(struct kernel_queue_ops *ops) { ops->initialize = initialize_vi; ops->uninitialize = uninitialize_vi; + ops->submit_packet = submit_packet_vi; } static bool initialize_vi(struct kernel_queue *kq, struct kfd_dev *dev, @@ -54,3 +59,317 @@ static void uninitialize_vi(struct kernel_queue *kq) { kfd_gtt_sa_free(kq->dev, kq->eop_mem); } + +static void submit_packet_vi(struct kernel_queue *kq) +{ + *kq->wptr_kernel = kq->pending_wptr; + write_kernel_doorbell(kq->queue->properties.doorbell_ptr, + kq->pending_wptr); +} + +unsigned int pm_build_pm4_header(unsigned int opcode, size_t packet_size) +{ + union PM4_MES_TYPE_3_HEADER header; + + header.u32All = 0; + header.opcode = opcode; + header.count = packet_size / 4 - 2; + header.type = PM4_TYPE_3; + + return header.u32All; +} + +static int pm_map_process_vi(struct packet_manager *pm, uint32_t *buffer, + struct qcm_process_device *qpd) +{ + struct pm4_mes_map_process *packet; + + packet = (struct pm4_mes_map_process *)buffer; + + memset(buffer, 0, sizeof(struct pm4_mes_map_process)); + + packet->header.u32All = pm_build_pm4_header(IT_MAP_PROCESS, + sizeof(struct pm4_mes_map_process)); + packet->bitfields2.diq_enable = (qpd->is_debug) ? 1 : 0; + packet->bitfields2.process_quantum = 1; + packet->bitfields2.pasid = qpd->pqm->process->pasid; + packet->bitfields3.page_table_base = qpd->page_table_base; + packet->bitfields10.gds_size = qpd->gds_size; + packet->bitfields10.num_gws = qpd->num_gws; + packet->bitfields10.num_oac = qpd->num_oac; + packet->bitfields10.num_queues = (qpd->is_debug) ? 0 : qpd->queue_count; + + packet->sh_mem_config = qpd->sh_mem_config; + packet->sh_mem_bases = qpd->sh_mem_bases; + packet->sh_mem_ape1_base = qpd->sh_mem_ape1_base; + packet->sh_mem_ape1_limit = qpd->sh_mem_ape1_limit; + + packet->sh_hidden_private_base_vmid = qpd->sh_hidden_private_base; + + packet->gds_addr_lo = lower_32_bits(qpd->gds_context_area); + packet->gds_addr_hi = upper_32_bits(qpd->gds_context_area); + + return 0; +} + +static int pm_runlist_vi(struct packet_manager *pm, uint32_t *buffer, + uint64_t ib, size_t ib_size_in_dwords, bool chain) +{ + struct pm4_mes_runlist *packet; + int concurrent_proc_cnt = 0; + struct kfd_dev *kfd = pm->dqm->dev; + + if (WARN_ON(!ib)) + return -EFAULT; + + /* Determine the number of processes to map together to HW: + * it can not exceed the number of VMIDs available to the + * scheduler, and it is determined by the smaller of the number + * of processes in the runlist and kfd module parameter + * hws_max_conc_proc. + * Note: the arbitration between the number of VMIDs and + * hws_max_conc_proc has been done in + * kgd2kfd_device_init(). + */ + concurrent_proc_cnt = min(pm->dqm->processes_count, + kfd->max_proc_per_quantum); + + packet = (struct pm4_mes_runlist *)buffer; + + memset(buffer, 0, sizeof(struct pm4_mes_runlist)); + packet->header.u32All = pm_build_pm4_header(IT_RUN_LIST, + sizeof(struct pm4_mes_runlist)); + + packet->bitfields4.ib_size = ib_size_in_dwords; + packet->bitfields4.chain = chain ? 1 : 0; + packet->bitfields4.offload_polling = 0; + packet->bitfields4.valid = 1; + packet->bitfields4.process_cnt = concurrent_proc_cnt; + packet->ordinal2 = lower_32_bits(ib); + packet->bitfields3.ib_base_hi = upper_32_bits(ib); + + return 0; +} + +int pm_set_resources_vi(struct packet_manager *pm, uint32_t *buffer, + struct scheduling_resources *res) +{ + struct pm4_mes_set_resources *packet; + + packet = (struct pm4_mes_set_resources *)buffer; + memset(buffer, 0, sizeof(struct pm4_mes_set_resources)); + + packet->header.u32All = pm_build_pm4_header(IT_SET_RESOURCES, + sizeof(struct pm4_mes_set_resources)); + + packet->bitfields2.queue_type = + queue_type__mes_set_resources__hsa_interface_queue_hiq; + packet->bitfields2.vmid_mask = res->vmid_mask; + packet->bitfields2.unmap_latency = KFD_UNMAP_LATENCY_MS / 100; + packet->bitfields7.oac_mask = res->oac_mask; + packet->bitfields8.gds_heap_base = res->gds_heap_base; + packet->bitfields8.gds_heap_size = res->gds_heap_size; + + packet->gws_mask_lo = lower_32_bits(res->gws_mask); + packet->gws_mask_hi = upper_32_bits(res->gws_mask); + + packet->queue_mask_lo = lower_32_bits(res->queue_mask); + packet->queue_mask_hi = upper_32_bits(res->queue_mask); + + return 0; +} + +static int pm_map_queues_vi(struct packet_manager *pm, uint32_t *buffer, + struct queue *q, bool is_static) +{ + struct pm4_mes_map_queues *packet; + bool use_static = is_static; + + packet = (struct pm4_mes_map_queues *)buffer; + memset(buffer, 0, sizeof(struct pm4_mes_map_queues)); + + packet->header.u32All = pm_build_pm4_header(IT_MAP_QUEUES, + sizeof(struct pm4_mes_map_queues)); + packet->bitfields2.alloc_format = + alloc_format__mes_map_queues__one_per_pipe_vi; + packet->bitfields2.num_queues = 1; + packet->bitfields2.queue_sel = + queue_sel__mes_map_queues__map_to_hws_determined_queue_slots_vi; + + packet->bitfields2.engine_sel = + engine_sel__mes_map_queues__compute_vi; + packet->bitfields2.queue_type = + queue_type__mes_map_queues__normal_compute_vi; + + switch (q->properties.type) { + case KFD_QUEUE_TYPE_COMPUTE: + if (use_static) + packet->bitfields2.queue_type = + queue_type__mes_map_queues__normal_latency_static_queue_vi; + break; + case KFD_QUEUE_TYPE_DIQ: + packet->bitfields2.queue_type = + queue_type__mes_map_queues__debug_interface_queue_vi; + break; + case KFD_QUEUE_TYPE_SDMA: + packet->bitfields2.engine_sel = q->properties.sdma_engine_id + + engine_sel__mes_map_queues__sdma0_vi; + use_static = false; /* no static queues under SDMA */ + break; + default: + WARN(1, "queue type %d", q->properties.type); + return -EINVAL; + } + packet->bitfields3.doorbell_offset = + q->properties.doorbell_off; + + packet->mqd_addr_lo = + lower_32_bits(q->gart_mqd_addr); + + packet->mqd_addr_hi = + upper_32_bits(q->gart_mqd_addr); + + packet->wptr_addr_lo = + lower_32_bits((uint64_t)q->properties.write_ptr); + + packet->wptr_addr_hi = + upper_32_bits((uint64_t)q->properties.write_ptr); + + return 0; +} + +static int pm_unmap_queues_vi(struct packet_manager *pm, uint32_t *buffer, + enum kfd_queue_type type, + enum kfd_unmap_queues_filter filter, + uint32_t filter_param, bool reset, + unsigned int sdma_engine) +{ + struct pm4_mes_unmap_queues *packet; + + packet = (struct pm4_mes_unmap_queues *)buffer; + memset(buffer, 0, sizeof(struct pm4_mes_unmap_queues)); + + packet->header.u32All = pm_build_pm4_header(IT_UNMAP_QUEUES, + sizeof(struct pm4_mes_unmap_queues)); + switch (type) { + case KFD_QUEUE_TYPE_COMPUTE: + case KFD_QUEUE_TYPE_DIQ: + packet->bitfields2.engine_sel = + engine_sel__mes_unmap_queues__compute; + break; + case KFD_QUEUE_TYPE_SDMA: + packet->bitfields2.engine_sel = + engine_sel__mes_unmap_queues__sdma0 + sdma_engine; + break; + default: + WARN(1, "queue type %d", type); + return -EINVAL; + } + + if (reset) + packet->bitfields2.action = + action__mes_unmap_queues__reset_queues; + else + packet->bitfields2.action = + action__mes_unmap_queues__preempt_queues; + + switch (filter) { + case KFD_UNMAP_QUEUES_FILTER_SINGLE_QUEUE: + packet->bitfields2.queue_sel = + queue_sel__mes_unmap_queues__perform_request_on_specified_queues; + packet->bitfields2.num_queues = 1; + packet->bitfields3b.doorbell_offset0 = filter_param; + break; + case KFD_UNMAP_QUEUES_FILTER_BY_PASID: + packet->bitfields2.queue_sel = + queue_sel__mes_unmap_queues__perform_request_on_pasid_queues; + packet->bitfields3a.pasid = filter_param; + break; + case KFD_UNMAP_QUEUES_FILTER_ALL_QUEUES: + packet->bitfields2.queue_sel = + queue_sel__mes_unmap_queues__unmap_all_queues; + break; + case KFD_UNMAP_QUEUES_FILTER_DYNAMIC_QUEUES: + /* in this case, we do not preempt static queues */ + packet->bitfields2.queue_sel = + queue_sel__mes_unmap_queues__unmap_all_non_static_queues; + break; + default: + WARN(1, "filter %d", filter); + return -EINVAL; + } + + return 0; + +} + +static int pm_query_status_vi(struct packet_manager *pm, uint32_t *buffer, + uint64_t fence_address, uint32_t fence_value) +{ + struct pm4_mes_query_status *packet; + + packet = (struct pm4_mes_query_status *)buffer; + memset(buffer, 0, sizeof(struct pm4_mes_query_status)); + + packet->header.u32All = pm_build_pm4_header(IT_QUERY_STATUS, + sizeof(struct pm4_mes_query_status)); + + packet->bitfields2.context_id = 0; + packet->bitfields2.interrupt_sel = + interrupt_sel__mes_query_status__completion_status; + packet->bitfields2.command = + command__mes_query_status__fence_only_after_write_ack; + + packet->addr_hi = upper_32_bits((uint64_t)fence_address); + packet->addr_lo = lower_32_bits((uint64_t)fence_address); + packet->data_hi = upper_32_bits((uint64_t)fence_value); + packet->data_lo = lower_32_bits((uint64_t)fence_value); + + return 0; +} + +static int pm_release_mem_vi(uint64_t gpu_addr, uint32_t *buffer) +{ + struct pm4_mec_release_mem *packet; + + packet = (struct pm4_mec_release_mem *)buffer; + memset(buffer, 0, sizeof(*packet)); + + packet->header.u32All = pm_build_pm4_header(IT_RELEASE_MEM, + sizeof(*packet)); + + packet->bitfields2.event_type = CACHE_FLUSH_AND_INV_TS_EVENT; + packet->bitfields2.event_index = event_index___release_mem__end_of_pipe; + packet->bitfields2.tcl1_action_ena = 1; + packet->bitfields2.tc_action_ena = 1; + packet->bitfields2.cache_policy = cache_policy___release_mem__lru; + packet->bitfields2.atc = 0; + + packet->bitfields3.data_sel = data_sel___release_mem__send_32_bit_low; + packet->bitfields3.int_sel = + int_sel___release_mem__send_interrupt_after_write_confirm; + + packet->bitfields4.address_lo_32b = (gpu_addr & 0xffffffff) >> 2; + packet->address_hi = upper_32_bits(gpu_addr); + + packet->data_lo = 0; + + return 0; +} + +const struct packet_manager_funcs kfd_vi_pm_funcs = { + .map_process = pm_map_process_vi, + .runlist = pm_runlist_vi, + .set_resources = pm_set_resources_vi, + .map_queues = pm_map_queues_vi, + .unmap_queues = pm_unmap_queues_vi, + .query_status = pm_query_status_vi, + .release_mem = pm_release_mem_vi, + .map_process_size = sizeof(struct pm4_mes_map_process), + .runlist_size = sizeof(struct pm4_mes_runlist), + .set_resources_size = sizeof(struct pm4_mes_set_resources), + .map_queues_size = sizeof(struct pm4_mes_map_queues), + .unmap_queues_size = sizeof(struct pm4_mes_unmap_queues), + .query_status_size = sizeof(struct pm4_mes_query_status), + .release_mem_size = sizeof(struct pm4_mec_release_mem) +}; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_module.c b/drivers/gpu/drm/amd/amdkfd/kfd_module.c index e0c07d24d251..76bf2dc8aec4 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_module.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_module.c @@ -43,6 +43,8 @@ static const struct kgd2kfd_calls kgd2kfd = { .interrupt = kgd2kfd_interrupt, .suspend = kgd2kfd_suspend, .resume = kgd2kfd_resume, + .quiesce_mm = kgd2kfd_quiesce_mm, + .resume_mm = kgd2kfd_resume_mm, .schedule_evict_and_restore_process = kgd2kfd_schedule_evict_and_restore_process, }; @@ -81,6 +83,11 @@ module_param(ignore_crat, int, 0444); MODULE_PARM_DESC(ignore_crat, "Ignore CRAT table during KFD initialization (0 = use CRAT (default), 1 = ignore CRAT)"); +int vega10_noretry; +module_param_named(noretry, vega10_noretry, int, 0644); +MODULE_PARM_DESC(noretry, + "Set sh_mem_config.retry_disable on Vega10 (0 = retry enabled (default), 1 = retry disabled)"); + static int amdkfd_init_completed; int kgd2kfd_init(unsigned int interface_version, diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c index ee7061e1c466..4b8eb506642b 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c @@ -38,6 +38,9 @@ struct mqd_manager *mqd_manager_init(enum KFD_MQD_TYPE type, case CHIP_POLARIS10: case CHIP_POLARIS11: return mqd_manager_init_vi_tonga(type, dev); + case CHIP_VEGA10: + case CHIP_RAVEN: + return mqd_manager_init_v9(type, dev); default: WARN(1, "Unexpected ASIC family %u", dev->device_info->asic_family); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_cik.c b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_cik.c index c00c325ed3c9..06eaa218eba6 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_cik.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_cik.c @@ -79,10 +79,6 @@ static int init_mqd(struct mqd_manager *mm, void **mqd, m->cp_mqd_base_addr_lo = lower_32_bits(addr); m->cp_mqd_base_addr_hi = upper_32_bits(addr); - m->cp_hqd_ib_control = DEFAULT_MIN_IB_AVAIL_SIZE | IB_ATC_EN; - /* Although WinKFD writes this, I suspect it should not be necessary */ - m->cp_hqd_ib_control = IB_ATC_EN | DEFAULT_MIN_IB_AVAIL_SIZE; - m->cp_hqd_quantum = QUANTUM_EN | QUANTUM_SCALE_1MS | QUANTUM_DURATION(10); @@ -412,7 +408,7 @@ struct mqd_manager *mqd_manager_init_cik(enum KFD_MQD_TYPE type, if (WARN_ON(type >= KFD_MQD_TYPE_MAX)) return NULL; - mqd = kzalloc(sizeof(*mqd), GFP_KERNEL); + mqd = kzalloc(sizeof(*mqd), GFP_NOIO); if (!mqd) return NULL; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c new file mode 100644 index 000000000000..684054ff02cd --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c @@ -0,0 +1,443 @@ +/* + * Copyright 2016-2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ + +#include +#include +#include +#include "kfd_priv.h" +#include "kfd_mqd_manager.h" +#include "v9_structs.h" +#include "gc/gc_9_0_offset.h" +#include "gc/gc_9_0_sh_mask.h" +#include "sdma0/sdma0_4_0_sh_mask.h" + +static inline struct v9_mqd *get_mqd(void *mqd) +{ + return (struct v9_mqd *)mqd; +} + +static inline struct v9_sdma_mqd *get_sdma_mqd(void *mqd) +{ + return (struct v9_sdma_mqd *)mqd; +} + +static int init_mqd(struct mqd_manager *mm, void **mqd, + struct kfd_mem_obj **mqd_mem_obj, uint64_t *gart_addr, + struct queue_properties *q) +{ + int retval; + uint64_t addr; + struct v9_mqd *m; + struct kfd_dev *kfd = mm->dev; + + /* From V9, for CWSR, the control stack is located on the next page + * boundary after the mqd, we will use the gtt allocation function + * instead of sub-allocation function. + */ + if (kfd->cwsr_enabled && (q->type == KFD_QUEUE_TYPE_COMPUTE)) { + *mqd_mem_obj = kzalloc(sizeof(struct kfd_mem_obj), GFP_NOIO); + if (!*mqd_mem_obj) + return -ENOMEM; + retval = kfd->kfd2kgd->init_gtt_mem_allocation(kfd->kgd, + ALIGN(q->ctl_stack_size, PAGE_SIZE) + + ALIGN(sizeof(struct v9_mqd), PAGE_SIZE), + &((*mqd_mem_obj)->gtt_mem), + &((*mqd_mem_obj)->gpu_addr), + (void *)&((*mqd_mem_obj)->cpu_ptr)); + } else + retval = kfd_gtt_sa_allocate(mm->dev, sizeof(struct v9_mqd), + mqd_mem_obj); + if (retval != 0) + return -ENOMEM; + + m = (struct v9_mqd *) (*mqd_mem_obj)->cpu_ptr; + addr = (*mqd_mem_obj)->gpu_addr; + + memset(m, 0, sizeof(struct v9_mqd)); + + m->header = 0xC0310800; + m->compute_pipelinestat_enable = 1; + m->compute_static_thread_mgmt_se0 = 0xFFFFFFFF; + m->compute_static_thread_mgmt_se1 = 0xFFFFFFFF; + m->compute_static_thread_mgmt_se2 = 0xFFFFFFFF; + m->compute_static_thread_mgmt_se3 = 0xFFFFFFFF; + + m->cp_hqd_persistent_state = CP_HQD_PERSISTENT_STATE__PRELOAD_REQ_MASK | + 0x53 << CP_HQD_PERSISTENT_STATE__PRELOAD_SIZE__SHIFT; + + m->cp_mqd_control = 1 << CP_MQD_CONTROL__PRIV_STATE__SHIFT; + + m->cp_mqd_base_addr_lo = lower_32_bits(addr); + m->cp_mqd_base_addr_hi = upper_32_bits(addr); + + m->cp_hqd_quantum = 1 << CP_HQD_QUANTUM__QUANTUM_EN__SHIFT | + 1 << CP_HQD_QUANTUM__QUANTUM_SCALE__SHIFT | + 10 << CP_HQD_QUANTUM__QUANTUM_DURATION__SHIFT; + + m->cp_hqd_pipe_priority = 1; + m->cp_hqd_queue_priority = 15; + + if (q->format == KFD_QUEUE_FORMAT_AQL) { + m->cp_hqd_aql_control = + 1 << CP_HQD_AQL_CONTROL__CONTROL0__SHIFT; + } + + if (q->tba_addr) { + m->compute_pgm_rsrc2 |= + (1 << COMPUTE_PGM_RSRC2__TRAP_PRESENT__SHIFT); + } + + if (mm->dev->cwsr_enabled && q->ctx_save_restore_area_address) { + m->cp_hqd_persistent_state |= + (1 << CP_HQD_PERSISTENT_STATE__QSWITCH_MODE__SHIFT); + m->cp_hqd_ctx_save_base_addr_lo = + lower_32_bits(q->ctx_save_restore_area_address); + m->cp_hqd_ctx_save_base_addr_hi = + upper_32_bits(q->ctx_save_restore_area_address); + m->cp_hqd_ctx_save_size = q->ctx_save_restore_area_size; + m->cp_hqd_cntl_stack_size = q->ctl_stack_size; + m->cp_hqd_cntl_stack_offset = q->ctl_stack_size; + m->cp_hqd_wg_state_offset = q->ctl_stack_size; + } + + *mqd = m; + if (gart_addr) + *gart_addr = addr; + retval = mm->update_mqd(mm, m, q); + + return retval; +} + +static int load_mqd(struct mqd_manager *mm, void *mqd, + uint32_t pipe_id, uint32_t queue_id, + struct queue_properties *p, struct mm_struct *mms) +{ + /* AQL write pointer counts in 64B packets, PM4/CP counts in dwords. */ + uint32_t wptr_shift = (p->format == KFD_QUEUE_FORMAT_AQL ? 4 : 0); + + return mm->dev->kfd2kgd->hqd_load(mm->dev->kgd, mqd, pipe_id, queue_id, + (uint32_t __user *)p->write_ptr, + wptr_shift, 0, mms); +} + +static int update_mqd(struct mqd_manager *mm, void *mqd, + struct queue_properties *q) +{ + struct v9_mqd *m; + + m = get_mqd(mqd); + + m->cp_hqd_pq_control = 5 << CP_HQD_PQ_CONTROL__RPTR_BLOCK_SIZE__SHIFT; + m->cp_hqd_pq_control |= order_base_2(q->queue_size / 4) - 1; + pr_debug("cp_hqd_pq_control 0x%x\n", m->cp_hqd_pq_control); + + m->cp_hqd_pq_base_lo = lower_32_bits((uint64_t)q->queue_address >> 8); + m->cp_hqd_pq_base_hi = upper_32_bits((uint64_t)q->queue_address >> 8); + + m->cp_hqd_pq_rptr_report_addr_lo = lower_32_bits((uint64_t)q->read_ptr); + m->cp_hqd_pq_rptr_report_addr_hi = upper_32_bits((uint64_t)q->read_ptr); + m->cp_hqd_pq_wptr_poll_addr_lo = lower_32_bits((uint64_t)q->write_ptr); + m->cp_hqd_pq_wptr_poll_addr_hi = upper_32_bits((uint64_t)q->write_ptr); + + m->cp_hqd_pq_doorbell_control = + q->doorbell_off << + CP_HQD_PQ_DOORBELL_CONTROL__DOORBELL_OFFSET__SHIFT; + pr_debug("cp_hqd_pq_doorbell_control 0x%x\n", + m->cp_hqd_pq_doorbell_control); + + m->cp_hqd_ib_control = + 3 << CP_HQD_IB_CONTROL__MIN_IB_AVAIL_SIZE__SHIFT | + 1 << CP_HQD_IB_CONTROL__IB_EXE_DISABLE__SHIFT; + + /* + * HW does not clamp this field correctly. Maximum EOP queue size + * is constrained by per-SE EOP done signal count, which is 8-bit. + * Limit is 0xFF EOP entries (= 0x7F8 dwords). CP will not submit + * more than (EOP entry count - 1) so a queue size of 0x800 dwords + * is safe, giving a maximum field value of 0xA. + */ + m->cp_hqd_eop_control = min(0xA, + order_base_2(q->eop_ring_buffer_size / 4) - 1); + m->cp_hqd_eop_base_addr_lo = + lower_32_bits(q->eop_ring_buffer_address >> 8); + m->cp_hqd_eop_base_addr_hi = + upper_32_bits(q->eop_ring_buffer_address >> 8); + + m->cp_hqd_iq_timer = 0; + + m->cp_hqd_vmid = q->vmid; + + if (q->format == KFD_QUEUE_FORMAT_AQL) { + m->cp_hqd_pq_control |= CP_HQD_PQ_CONTROL__NO_UPDATE_RPTR_MASK | + 2 << CP_HQD_PQ_CONTROL__SLOT_BASED_WPTR__SHIFT | + 1 << CP_HQD_PQ_CONTROL__QUEUE_FULL_EN__SHIFT | + 1 << CP_HQD_PQ_CONTROL__WPP_CLAMP_EN__SHIFT; + m->cp_hqd_pq_doorbell_control |= 1 << + CP_HQD_PQ_DOORBELL_CONTROL__DOORBELL_BIF_DROP__SHIFT; + } + if (mm->dev->cwsr_enabled && q->ctx_save_restore_area_address) + m->cp_hqd_ctx_save_control = 0; + + q->is_active = (q->queue_size > 0 && + q->queue_address != 0 && + q->queue_percent > 0 && + !q->is_evicted); + + return 0; +} + + +static int destroy_mqd(struct mqd_manager *mm, void *mqd, + enum kfd_preempt_type type, + unsigned int timeout, uint32_t pipe_id, + uint32_t queue_id) +{ + return mm->dev->kfd2kgd->hqd_destroy + (mm->dev->kgd, mqd, type, timeout, + pipe_id, queue_id); +} + +static void uninit_mqd(struct mqd_manager *mm, void *mqd, + struct kfd_mem_obj *mqd_mem_obj) +{ + struct kfd_dev *kfd = mm->dev; + + if (mqd_mem_obj->gtt_mem) { + kfd->kfd2kgd->free_gtt_mem(kfd->kgd, mqd_mem_obj->gtt_mem); + kfree(mqd_mem_obj); + } else { + kfd_gtt_sa_free(mm->dev, mqd_mem_obj); + } +} + +static bool is_occupied(struct mqd_manager *mm, void *mqd, + uint64_t queue_address, uint32_t pipe_id, + uint32_t queue_id) +{ + return mm->dev->kfd2kgd->hqd_is_occupied( + mm->dev->kgd, queue_address, + pipe_id, queue_id); +} + +static int init_mqd_hiq(struct mqd_manager *mm, void **mqd, + struct kfd_mem_obj **mqd_mem_obj, uint64_t *gart_addr, + struct queue_properties *q) +{ + struct v9_mqd *m; + int retval = init_mqd(mm, mqd, mqd_mem_obj, gart_addr, q); + + if (retval != 0) + return retval; + + m = get_mqd(*mqd); + + m->cp_hqd_pq_control |= 1 << CP_HQD_PQ_CONTROL__PRIV_STATE__SHIFT | + 1 << CP_HQD_PQ_CONTROL__KMD_QUEUE__SHIFT; + + return retval; +} + +static int update_mqd_hiq(struct mqd_manager *mm, void *mqd, + struct queue_properties *q) +{ + struct v9_mqd *m; + int retval = update_mqd(mm, mqd, q); + + if (retval != 0) + return retval; + + /* TODO: what's the point? update_mqd already does this. */ + m = get_mqd(mqd); + m->cp_hqd_vmid = q->vmid; + return retval; +} + +static int init_mqd_sdma(struct mqd_manager *mm, void **mqd, + struct kfd_mem_obj **mqd_mem_obj, uint64_t *gart_addr, + struct queue_properties *q) +{ + int retval; + struct v9_sdma_mqd *m; + + + retval = kfd_gtt_sa_allocate(mm->dev, + sizeof(struct v9_sdma_mqd), + mqd_mem_obj); + + if (retval != 0) + return -ENOMEM; + + m = (struct v9_sdma_mqd *) (*mqd_mem_obj)->cpu_ptr; + + memset(m, 0, sizeof(struct v9_sdma_mqd)); + + *mqd = m; + if (gart_addr) + *gart_addr = (*mqd_mem_obj)->gpu_addr; + + retval = mm->update_mqd(mm, m, q); + + return retval; +} + +static void uninit_mqd_sdma(struct mqd_manager *mm, void *mqd, + struct kfd_mem_obj *mqd_mem_obj) +{ + kfd_gtt_sa_free(mm->dev, mqd_mem_obj); +} + +static int load_mqd_sdma(struct mqd_manager *mm, void *mqd, + uint32_t pipe_id, uint32_t queue_id, + struct queue_properties *p, struct mm_struct *mms) +{ + return mm->dev->kfd2kgd->hqd_sdma_load(mm->dev->kgd, mqd, + (uint32_t __user *)p->write_ptr, + mms); +} + +#define SDMA_RLC_DUMMY_DEFAULT 0xf + +static int update_mqd_sdma(struct mqd_manager *mm, void *mqd, + struct queue_properties *q) +{ + struct v9_sdma_mqd *m; + + m = get_sdma_mqd(mqd); + m->sdmax_rlcx_rb_cntl = order_base_2(q->queue_size / 4) + << SDMA0_RLC0_RB_CNTL__RB_SIZE__SHIFT | + q->vmid << SDMA0_RLC0_RB_CNTL__RB_VMID__SHIFT | + 1 << SDMA0_RLC0_RB_CNTL__RPTR_WRITEBACK_ENABLE__SHIFT | + 6 << SDMA0_RLC0_RB_CNTL__RPTR_WRITEBACK_TIMER__SHIFT; + + m->sdmax_rlcx_rb_base = lower_32_bits(q->queue_address >> 8); + m->sdmax_rlcx_rb_base_hi = upper_32_bits(q->queue_address >> 8); + m->sdmax_rlcx_rb_rptr_addr_lo = lower_32_bits((uint64_t)q->read_ptr); + m->sdmax_rlcx_rb_rptr_addr_hi = upper_32_bits((uint64_t)q->read_ptr); + m->sdmax_rlcx_doorbell_offset = + q->doorbell_off << SDMA0_RLC0_DOORBELL_OFFSET__OFFSET__SHIFT; + + m->sdma_engine_id = q->sdma_engine_id; + m->sdma_queue_id = q->sdma_queue_id; + m->sdmax_rlcx_dummy_reg = SDMA_RLC_DUMMY_DEFAULT; + + q->is_active = (q->queue_size > 0 && + q->queue_address != 0 && + q->queue_percent > 0 && + !q->is_evicted); + + return 0; +} + +/* + * * preempt type here is ignored because there is only one way + * * to preempt sdma queue + */ +static int destroy_mqd_sdma(struct mqd_manager *mm, void *mqd, + enum kfd_preempt_type type, + unsigned int timeout, uint32_t pipe_id, + uint32_t queue_id) +{ + return mm->dev->kfd2kgd->hqd_sdma_destroy(mm->dev->kgd, mqd, timeout); +} + +static bool is_occupied_sdma(struct mqd_manager *mm, void *mqd, + uint64_t queue_address, uint32_t pipe_id, + uint32_t queue_id) +{ + return mm->dev->kfd2kgd->hqd_sdma_is_occupied(mm->dev->kgd, mqd); +} + +#if defined(CONFIG_DEBUG_FS) + +static int debugfs_show_mqd(struct seq_file *m, void *data) +{ + seq_hex_dump(m, " ", DUMP_PREFIX_OFFSET, 32, 4, + data, sizeof(struct v9_mqd), false); + return 0; +} + +static int debugfs_show_mqd_sdma(struct seq_file *m, void *data) +{ + seq_hex_dump(m, " ", DUMP_PREFIX_OFFSET, 32, 4, + data, sizeof(struct v9_sdma_mqd), false); + return 0; +} + +#endif + +struct mqd_manager *mqd_manager_init_v9(enum KFD_MQD_TYPE type, + struct kfd_dev *dev) +{ + struct mqd_manager *mqd; + + if (WARN_ON(type >= KFD_MQD_TYPE_MAX)) + return NULL; + + mqd = kzalloc(sizeof(*mqd), GFP_NOIO); + if (!mqd) + return NULL; + + mqd->dev = dev; + + switch (type) { + case KFD_MQD_TYPE_CP: + case KFD_MQD_TYPE_COMPUTE: + mqd->init_mqd = init_mqd; + mqd->uninit_mqd = uninit_mqd; + mqd->load_mqd = load_mqd; + mqd->update_mqd = update_mqd; + mqd->destroy_mqd = destroy_mqd; + mqd->is_occupied = is_occupied; +#if defined(CONFIG_DEBUG_FS) + mqd->debugfs_show_mqd = debugfs_show_mqd; +#endif + break; + case KFD_MQD_TYPE_HIQ: + mqd->init_mqd = init_mqd_hiq; + mqd->uninit_mqd = uninit_mqd; + mqd->load_mqd = load_mqd; + mqd->update_mqd = update_mqd_hiq; + mqd->destroy_mqd = destroy_mqd; + mqd->is_occupied = is_occupied; +#if defined(CONFIG_DEBUG_FS) + mqd->debugfs_show_mqd = debugfs_show_mqd; +#endif + break; + case KFD_MQD_TYPE_SDMA: + mqd->init_mqd = init_mqd_sdma; + mqd->uninit_mqd = uninit_mqd_sdma; + mqd->load_mqd = load_mqd_sdma; + mqd->update_mqd = update_mqd_sdma; + mqd->destroy_mqd = destroy_mqd_sdma; + mqd->is_occupied = is_occupied_sdma; +#if defined(CONFIG_DEBUG_FS) + mqd->debugfs_show_mqd = debugfs_show_mqd_sdma; +#endif + break; + default: + kfree(mqd); + return NULL; + } + + return mqd; +} diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_vi.c b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_vi.c index 89e4242e43e7..481307b8b4db 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_vi.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_vi.c @@ -394,7 +394,7 @@ struct mqd_manager *mqd_manager_init_vi(enum KFD_MQD_TYPE type, if (WARN_ON(type >= KFD_MQD_TYPE_MAX)) return NULL; - mqd = kzalloc(sizeof(*mqd), GFP_KERNEL); + mqd = kzalloc(sizeof(*mqd), GFP_NOIO); if (!mqd) return NULL; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_packet_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_packet_manager.c index 89ba4c670ec5..c317feb43f69 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_packet_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_packet_manager.c @@ -26,8 +26,6 @@ #include "kfd_device_queue_manager.h" #include "kfd_kernel_queue.h" #include "kfd_priv.h" -#include "kfd_pm4_headers_vi.h" -#include "kfd_pm4_opcodes.h" static inline void inc_wptr(unsigned int *wptr, unsigned int increment_bytes, unsigned int buffer_size_bytes) @@ -39,18 +37,6 @@ static inline void inc_wptr(unsigned int *wptr, unsigned int increment_bytes, *wptr = temp; } -static unsigned int build_pm4_header(unsigned int opcode, size_t packet_size) -{ - union PM4_MES_TYPE_3_HEADER header; - - header.u32All = 0; - header.opcode = opcode; - header.count = packet_size / 4 - 2; - header.type = PM4_TYPE_3; - - return header.u32All; -} - static void pm_calc_rlib_size(struct packet_manager *pm, unsigned int *rlib_size, bool *over_subscription) @@ -80,9 +66,9 @@ static void pm_calc_rlib_size(struct packet_manager *pm, pr_debug("Over subscribed runlist\n"); } - map_queue_size = sizeof(struct pm4_mes_map_queues); + map_queue_size = pm->pmf->map_queues_size; /* calculate run list ib allocation size */ - *rlib_size = process_count * sizeof(struct pm4_mes_map_process) + + *rlib_size = process_count * pm->pmf->map_process_size + queue_count * map_queue_size; /* @@ -90,7 +76,7 @@ static void pm_calc_rlib_size(struct packet_manager *pm, * when over subscription */ if (*over_subscription) - *rlib_size += sizeof(struct pm4_mes_runlist); + *rlib_size += pm->pmf->runlist_size; pr_debug("runlist ib size %d\n", *rlib_size); } @@ -108,12 +94,14 @@ static int pm_allocate_runlist_ib(struct packet_manager *pm, pm_calc_rlib_size(pm, rl_buffer_size, is_over_subscription); + mutex_lock(&pm->lock); + retval = kfd_gtt_sa_allocate(pm->dqm->dev, *rl_buffer_size, &pm->ib_buffer_obj); if (retval) { pr_err("Failed to allocate runlist IB\n"); - return retval; + goto out; } *(void **)rl_buffer = pm->ib_buffer_obj->cpu_ptr; @@ -121,138 +109,10 @@ static int pm_allocate_runlist_ib(struct packet_manager *pm, memset(*rl_buffer, 0, *rl_buffer_size); pm->allocated = true; - return retval; -} - -static int pm_create_runlist(struct packet_manager *pm, uint32_t *buffer, - uint64_t ib, size_t ib_size_in_dwords, bool chain) -{ - struct pm4_mes_runlist *packet; - int concurrent_proc_cnt = 0; - struct kfd_dev *kfd = pm->dqm->dev; - - if (WARN_ON(!ib)) - return -EFAULT; - - /* Determine the number of processes to map together to HW: - * it can not exceed the number of VMIDs available to the - * scheduler, and it is determined by the smaller of the number - * of processes in the runlist and kfd module parameter - * hws_max_conc_proc. - * Note: the arbitration between the number of VMIDs and - * hws_max_conc_proc has been done in - * kgd2kfd_device_init(). - */ - concurrent_proc_cnt = min(pm->dqm->processes_count, - kfd->max_proc_per_quantum); - - packet = (struct pm4_mes_runlist *)buffer; - - memset(buffer, 0, sizeof(struct pm4_mes_runlist)); - packet->header.u32All = build_pm4_header(IT_RUN_LIST, - sizeof(struct pm4_mes_runlist)); - - packet->bitfields4.ib_size = ib_size_in_dwords; - packet->bitfields4.chain = chain ? 1 : 0; - packet->bitfields4.offload_polling = 0; - packet->bitfields4.valid = 1; - packet->bitfields4.process_cnt = concurrent_proc_cnt; - packet->ordinal2 = lower_32_bits(ib); - packet->bitfields3.ib_base_hi = upper_32_bits(ib); - - return 0; -} - -static int pm_create_map_process(struct packet_manager *pm, uint32_t *buffer, - struct qcm_process_device *qpd) -{ - struct pm4_mes_map_process *packet; - - packet = (struct pm4_mes_map_process *)buffer; - memset(buffer, 0, sizeof(struct pm4_mes_map_process)); - - packet->header.u32All = build_pm4_header(IT_MAP_PROCESS, - sizeof(struct pm4_mes_map_process)); - packet->bitfields2.diq_enable = (qpd->is_debug) ? 1 : 0; - packet->bitfields2.process_quantum = 1; - packet->bitfields2.pasid = qpd->pqm->process->pasid; - packet->bitfields3.page_table_base = qpd->page_table_base; - packet->bitfields10.gds_size = qpd->gds_size; - packet->bitfields10.num_gws = qpd->num_gws; - packet->bitfields10.num_oac = qpd->num_oac; - packet->bitfields10.num_queues = (qpd->is_debug) ? 0 : qpd->queue_count; - - packet->sh_mem_config = qpd->sh_mem_config; - packet->sh_mem_bases = qpd->sh_mem_bases; - packet->sh_mem_ape1_base = qpd->sh_mem_ape1_base; - packet->sh_mem_ape1_limit = qpd->sh_mem_ape1_limit; - - packet->sh_hidden_private_base_vmid = qpd->sh_hidden_private_base; - - packet->gds_addr_lo = lower_32_bits(qpd->gds_context_area); - packet->gds_addr_hi = upper_32_bits(qpd->gds_context_area); - - return 0; -} - -static int pm_create_map_queue(struct packet_manager *pm, uint32_t *buffer, - struct queue *q, bool is_static) -{ - struct pm4_mes_map_queues *packet; - bool use_static = is_static; - - packet = (struct pm4_mes_map_queues *)buffer; - memset(buffer, 0, sizeof(struct pm4_mes_map_queues)); - - packet->header.u32All = build_pm4_header(IT_MAP_QUEUES, - sizeof(struct pm4_mes_map_queues)); - packet->bitfields2.alloc_format = - alloc_format__mes_map_queues__one_per_pipe_vi; - packet->bitfields2.num_queues = 1; - packet->bitfields2.queue_sel = - queue_sel__mes_map_queues__map_to_hws_determined_queue_slots_vi; - - packet->bitfields2.engine_sel = - engine_sel__mes_map_queues__compute_vi; - packet->bitfields2.queue_type = - queue_type__mes_map_queues__normal_compute_vi; - - switch (q->properties.type) { - case KFD_QUEUE_TYPE_COMPUTE: - if (use_static) - packet->bitfields2.queue_type = - queue_type__mes_map_queues__normal_latency_static_queue_vi; - break; - case KFD_QUEUE_TYPE_DIQ: - packet->bitfields2.queue_type = - queue_type__mes_map_queues__debug_interface_queue_vi; - break; - case KFD_QUEUE_TYPE_SDMA: - packet->bitfields2.engine_sel = q->properties.sdma_engine_id + - engine_sel__mes_map_queues__sdma0_vi; - use_static = false; /* no static queues under SDMA */ - break; - default: - WARN(1, "queue type %d", q->properties.type); - return -EINVAL; - } - packet->bitfields3.doorbell_offset = - q->properties.doorbell_off; - - packet->mqd_addr_lo = - lower_32_bits(q->gart_mqd_addr); - - packet->mqd_addr_hi = - upper_32_bits(q->gart_mqd_addr); - - packet->wptr_addr_lo = - lower_32_bits((uint64_t)q->properties.write_ptr); - - packet->wptr_addr_hi = - upper_32_bits((uint64_t)q->properties.write_ptr); - - return 0; +out: + mutex_unlock(&pm->lock); + return retval; } static int pm_create_runlist_ib(struct packet_manager *pm, @@ -292,12 +152,12 @@ static int pm_create_runlist_ib(struct packet_manager *pm, return -ENOMEM; } - retval = pm_create_map_process(pm, &rl_buffer[rl_wptr], qpd); + retval = pm->pmf->map_process(pm, &rl_buffer[rl_wptr], qpd); if (retval) return retval; proccesses_mapped++; - inc_wptr(&rl_wptr, sizeof(struct pm4_mes_map_process), + inc_wptr(&rl_wptr, pm->pmf->map_process_size, alloc_size_bytes); list_for_each_entry(kq, &qpd->priv_queue_list, list) { @@ -307,7 +167,7 @@ static int pm_create_runlist_ib(struct packet_manager *pm, pr_debug("static_queue, mapping kernel q %d, is debug status %d\n", kq->queue->queue, qpd->is_debug); - retval = pm_create_map_queue(pm, + retval = pm->pmf->map_queues(pm, &rl_buffer[rl_wptr], kq->queue, qpd->is_debug); @@ -315,7 +175,7 @@ static int pm_create_runlist_ib(struct packet_manager *pm, return retval; inc_wptr(&rl_wptr, - sizeof(struct pm4_mes_map_queues), + pm->pmf->map_queues_size, alloc_size_bytes); } @@ -326,7 +186,7 @@ static int pm_create_runlist_ib(struct packet_manager *pm, pr_debug("static_queue, mapping user queue %d, is debug status %d\n", q->queue, qpd->is_debug); - retval = pm_create_map_queue(pm, + retval = pm->pmf->map_queues(pm, &rl_buffer[rl_wptr], q, qpd->is_debug); @@ -335,7 +195,7 @@ static int pm_create_runlist_ib(struct packet_manager *pm, return retval; inc_wptr(&rl_wptr, - sizeof(struct pm4_mes_map_queues), + pm->pmf->map_queues_size, alloc_size_bytes); } } @@ -343,7 +203,7 @@ static int pm_create_runlist_ib(struct packet_manager *pm, pr_debug("Finished map process and queues to runlist\n"); if (is_over_subscription) - retval = pm_create_runlist(pm, &rl_buffer[rl_wptr], + retval = pm->pmf->runlist(pm, &rl_buffer[rl_wptr], *rl_gpu_addr, alloc_size_bytes / sizeof(uint32_t), true); @@ -355,45 +215,29 @@ static int pm_create_runlist_ib(struct packet_manager *pm, return retval; } -/* pm_create_release_mem - Create a RELEASE_MEM packet and return the size - * of this packet - * @gpu_addr - GPU address of the packet. It's a virtual address. - * @buffer - buffer to fill up with the packet. It's a CPU kernel pointer - * Return - length of the packet - */ -uint32_t pm_create_release_mem(uint64_t gpu_addr, uint32_t *buffer) -{ - struct pm4_mec_release_mem *packet; - - WARN_ON(!buffer); - - packet = (struct pm4_mec_release_mem *)buffer; - memset(buffer, 0, sizeof(*packet)); - - packet->header.u32All = build_pm4_header(IT_RELEASE_MEM, - sizeof(*packet)); - - packet->bitfields2.event_type = CACHE_FLUSH_AND_INV_TS_EVENT; - packet->bitfields2.event_index = event_index___release_mem__end_of_pipe; - packet->bitfields2.tcl1_action_ena = 1; - packet->bitfields2.tc_action_ena = 1; - packet->bitfields2.cache_policy = cache_policy___release_mem__lru; - packet->bitfields2.atc = 0; - - packet->bitfields3.data_sel = data_sel___release_mem__send_32_bit_low; - packet->bitfields3.int_sel = - int_sel___release_mem__send_interrupt_after_write_confirm; - - packet->bitfields4.address_lo_32b = (gpu_addr & 0xffffffff) >> 2; - packet->address_hi = upper_32_bits(gpu_addr); - - packet->data_lo = 0; - - return sizeof(*packet) / sizeof(unsigned int); -} - int pm_init(struct packet_manager *pm, struct device_queue_manager *dqm) { + switch (dqm->dev->device_info->asic_family) { + case CHIP_KAVERI: + case CHIP_HAWAII: + /* PM4 packet structures on CIK are the same as on VI */ + case CHIP_CARRIZO: + case CHIP_TONGA: + case CHIP_FIJI: + case CHIP_POLARIS10: + case CHIP_POLARIS11: + pm->pmf = &kfd_vi_pm_funcs; + break; + case CHIP_VEGA10: + case CHIP_RAVEN: + pm->pmf = &kfd_v9_pm_funcs; + break; + default: + WARN(1, "Unexpected ASIC family %u", + dqm->dev->device_info->asic_family); + return -EINVAL; + } + pm->dqm = dqm; mutex_init(&pm->lock); pm->priv_queue = kernel_queue_init(dqm->dev, KFD_QUEUE_TYPE_HIQ); @@ -415,38 +259,25 @@ void pm_uninit(struct packet_manager *pm) int pm_send_set_resources(struct packet_manager *pm, struct scheduling_resources *res) { - struct pm4_mes_set_resources *packet; + uint32_t *buffer, size; int retval = 0; + size = pm->pmf->set_resources_size; mutex_lock(&pm->lock); pm->priv_queue->ops.acquire_packet_buffer(pm->priv_queue, - sizeof(*packet) / sizeof(uint32_t), - (unsigned int **)&packet); - if (!packet) { + size / sizeof(uint32_t), + (unsigned int **)&buffer); + if (!buffer) { pr_err("Failed to allocate buffer on kernel queue\n"); retval = -ENOMEM; goto out; } - memset(packet, 0, sizeof(struct pm4_mes_set_resources)); - packet->header.u32All = build_pm4_header(IT_SET_RESOURCES, - sizeof(struct pm4_mes_set_resources)); - - packet->bitfields2.queue_type = - queue_type__mes_set_resources__hsa_interface_queue_hiq; - packet->bitfields2.vmid_mask = res->vmid_mask; - packet->bitfields2.unmap_latency = KFD_UNMAP_LATENCY_MS / 100; - packet->bitfields7.oac_mask = res->oac_mask; - packet->bitfields8.gds_heap_base = res->gds_heap_base; - packet->bitfields8.gds_heap_size = res->gds_heap_size; - - packet->gws_mask_lo = lower_32_bits(res->gws_mask); - packet->gws_mask_hi = upper_32_bits(res->gws_mask); - - packet->queue_mask_lo = lower_32_bits(res->queue_mask); - packet->queue_mask_hi = upper_32_bits(res->queue_mask); - - pm->priv_queue->ops.submit_packet(pm->priv_queue); + retval = pm->pmf->set_resources(pm, buffer, res); + if (!retval) + pm->priv_queue->ops.submit_packet(pm->priv_queue); + else + pm->priv_queue->ops.rollback_packet(pm->priv_queue); out: mutex_unlock(&pm->lock); @@ -468,7 +299,7 @@ int pm_send_runlist(struct packet_manager *pm, struct list_head *dqm_queues) pr_debug("runlist IB address: 0x%llX\n", rl_gpu_ib_addr); - packet_size_dwords = sizeof(struct pm4_mes_runlist) / sizeof(uint32_t); + packet_size_dwords = pm->pmf->runlist_size / sizeof(uint32_t); mutex_lock(&pm->lock); retval = pm->priv_queue->ops.acquire_packet_buffer(pm->priv_queue, @@ -476,7 +307,7 @@ int pm_send_runlist(struct packet_manager *pm, struct list_head *dqm_queues) if (retval) goto fail_acquire_packet_buffer; - retval = pm_create_runlist(pm, rl_buffer, rl_gpu_ib_addr, + retval = pm->pmf->runlist(pm, rl_buffer, rl_gpu_ib_addr, rl_ib_size / sizeof(uint32_t), false); if (retval) goto fail_create_runlist; @@ -499,37 +330,29 @@ fail_create_runlist_ib: int pm_send_query_status(struct packet_manager *pm, uint64_t fence_address, uint32_t fence_value) { - int retval; - struct pm4_mes_query_status *packet; + uint32_t *buffer, size; + int retval = 0; if (WARN_ON(!fence_address)) return -EFAULT; + size = pm->pmf->query_status_size; mutex_lock(&pm->lock); - retval = pm->priv_queue->ops.acquire_packet_buffer( - pm->priv_queue, - sizeof(struct pm4_mes_query_status) / sizeof(uint32_t), - (unsigned int **)&packet); - if (retval) - goto fail_acquire_packet_buffer; - - packet->header.u32All = build_pm4_header(IT_QUERY_STATUS, - sizeof(struct pm4_mes_query_status)); - - packet->bitfields2.context_id = 0; - packet->bitfields2.interrupt_sel = - interrupt_sel__mes_query_status__completion_status; - packet->bitfields2.command = - command__mes_query_status__fence_only_after_write_ack; - - packet->addr_hi = upper_32_bits((uint64_t)fence_address); - packet->addr_lo = lower_32_bits((uint64_t)fence_address); - packet->data_hi = upper_32_bits((uint64_t)fence_value); - packet->data_lo = lower_32_bits((uint64_t)fence_value); + pm->priv_queue->ops.acquire_packet_buffer(pm->priv_queue, + size / sizeof(uint32_t), (unsigned int **)&buffer); + if (!buffer) { + pr_err("Failed to allocate buffer on kernel queue\n"); + retval = -ENOMEM; + goto out; + } - pm->priv_queue->ops.submit_packet(pm->priv_queue); + retval = pm->pmf->query_status(pm, buffer, fence_address, fence_value); + if (!retval) + pm->priv_queue->ops.submit_packet(pm->priv_queue); + else + pm->priv_queue->ops.rollback_packet(pm->priv_queue); -fail_acquire_packet_buffer: +out: mutex_unlock(&pm->lock); return retval; } @@ -539,82 +362,27 @@ int pm_send_unmap_queue(struct packet_manager *pm, enum kfd_queue_type type, uint32_t filter_param, bool reset, unsigned int sdma_engine) { - int retval; - uint32_t *buffer; - struct pm4_mes_unmap_queues *packet; + uint32_t *buffer, size; + int retval = 0; + size = pm->pmf->unmap_queues_size; mutex_lock(&pm->lock); - retval = pm->priv_queue->ops.acquire_packet_buffer( - pm->priv_queue, - sizeof(struct pm4_mes_unmap_queues) / sizeof(uint32_t), - &buffer); - if (retval) - goto err_acquire_packet_buffer; - - packet = (struct pm4_mes_unmap_queues *)buffer; - memset(buffer, 0, sizeof(struct pm4_mes_unmap_queues)); - pr_debug("static_queue: unmapping queues: filter is %d , reset is %d , type is %d\n", - filter, reset, type); - packet->header.u32All = build_pm4_header(IT_UNMAP_QUEUES, - sizeof(struct pm4_mes_unmap_queues)); - switch (type) { - case KFD_QUEUE_TYPE_COMPUTE: - case KFD_QUEUE_TYPE_DIQ: - packet->bitfields2.engine_sel = - engine_sel__mes_unmap_queues__compute; - break; - case KFD_QUEUE_TYPE_SDMA: - packet->bitfields2.engine_sel = - engine_sel__mes_unmap_queues__sdma0 + sdma_engine; - break; - default: - WARN(1, "queue type %d", type); - retval = -EINVAL; - goto err_invalid; + pm->priv_queue->ops.acquire_packet_buffer(pm->priv_queue, + size / sizeof(uint32_t), (unsigned int **)&buffer); + if (!buffer) { + pr_err("Failed to allocate buffer on kernel queue\n"); + retval = -ENOMEM; + goto out; } - if (reset) - packet->bitfields2.action = - action__mes_unmap_queues__reset_queues; + retval = pm->pmf->unmap_queues(pm, buffer, type, filter, filter_param, + reset, sdma_engine); + if (!retval) + pm->priv_queue->ops.submit_packet(pm->priv_queue); else - packet->bitfields2.action = - action__mes_unmap_queues__preempt_queues; - - switch (filter) { - case KFD_UNMAP_QUEUES_FILTER_SINGLE_QUEUE: - packet->bitfields2.queue_sel = - queue_sel__mes_unmap_queues__perform_request_on_specified_queues; - packet->bitfields2.num_queues = 1; - packet->bitfields3b.doorbell_offset0 = filter_param; - break; - case KFD_UNMAP_QUEUES_FILTER_BY_PASID: - packet->bitfields2.queue_sel = - queue_sel__mes_unmap_queues__perform_request_on_pasid_queues; - packet->bitfields3a.pasid = filter_param; - break; - case KFD_UNMAP_QUEUES_FILTER_ALL_QUEUES: - packet->bitfields2.queue_sel = - queue_sel__mes_unmap_queues__unmap_all_queues; - break; - case KFD_UNMAP_QUEUES_FILTER_DYNAMIC_QUEUES: - /* in this case, we do not preempt static queues */ - packet->bitfields2.queue_sel = - queue_sel__mes_unmap_queues__unmap_all_non_static_queues; - break; - default: - WARN(1, "filter %d", filter); - retval = -EINVAL; - goto err_invalid; - } + pm->priv_queue->ops.rollback_packet(pm->priv_queue); - pm->priv_queue->ops.submit_packet(pm->priv_queue); - - mutex_unlock(&pm->lock); - return 0; - -err_invalid: - pm->priv_queue->ops.rollback_packet(pm->priv_queue); -err_acquire_packet_buffer: +out: mutex_unlock(&pm->lock); return retval; } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_pm4_headers_ai.h b/drivers/gpu/drm/amd/amdkfd/kfd_pm4_headers_ai.h new file mode 100644 index 000000000000..f2bcf5c092ea --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/kfd_pm4_headers_ai.h @@ -0,0 +1,583 @@ +/* + * Copyright 2016 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ + +#ifndef F32_MES_PM4_PACKETS_H +#define F32_MES_PM4_PACKETS_H + +#ifndef PM4_MES_HEADER_DEFINED +#define PM4_MES_HEADER_DEFINED +union PM4_MES_TYPE_3_HEADER { + struct { + uint32_t reserved1 : 8; /* < reserved */ + uint32_t opcode : 8; /* < IT opcode */ + uint32_t count : 14;/* < number of DWORDs - 1 in the + * information body. + */ + uint32_t type : 2; /* < packet identifier. + * It should be 3 for type 3 packets + */ + }; + uint32_t u32All; +}; +#endif /* PM4_MES_HEADER_DEFINED */ + +/*--------------------MES_SET_RESOURCES--------------------*/ + +#ifndef PM4_MES_SET_RESOURCES_DEFINED +#define PM4_MES_SET_RESOURCES_DEFINED +enum mes_set_resources_queue_type_enum { + queue_type__mes_set_resources__kernel_interface_queue_kiq = 0, + queue_type__mes_set_resources__hsa_interface_queue_hiq = 1, + queue_type__mes_set_resources__hsa_debug_interface_queue = 4 +}; + + +struct pm4_mes_set_resources { + union { + union PM4_MES_TYPE_3_HEADER header; /* header */ + uint32_t ordinal1; + }; + + union { + struct { + uint32_t vmid_mask:16; + uint32_t unmap_latency:8; + uint32_t reserved1:5; + enum mes_set_resources_queue_type_enum queue_type:3; + } bitfields2; + uint32_t ordinal2; + }; + + uint32_t queue_mask_lo; + uint32_t queue_mask_hi; + uint32_t gws_mask_lo; + uint32_t gws_mask_hi; + + union { + struct { + uint32_t oac_mask:16; + uint32_t reserved2:16; + } bitfields7; + uint32_t ordinal7; + }; + + union { + struct { + uint32_t gds_heap_base:6; + uint32_t reserved3:5; + uint32_t gds_heap_size:6; + uint32_t reserved4:15; + } bitfields8; + uint32_t ordinal8; + }; + +}; +#endif + +/*--------------------MES_RUN_LIST--------------------*/ + +#ifndef PM4_MES_RUN_LIST_DEFINED +#define PM4_MES_RUN_LIST_DEFINED + +struct pm4_mes_runlist { + union { + union PM4_MES_TYPE_3_HEADER header; /* header */ + uint32_t ordinal1; + }; + + union { + struct { + uint32_t reserved1:2; + uint32_t ib_base_lo:30; + } bitfields2; + uint32_t ordinal2; + }; + + uint32_t ib_base_hi; + + union { + struct { + uint32_t ib_size:20; + uint32_t chain:1; + uint32_t offload_polling:1; + uint32_t reserved2:1; + uint32_t valid:1; + uint32_t process_cnt:4; + uint32_t reserved3:4; + } bitfields4; + uint32_t ordinal4; + }; + +}; +#endif + +/*--------------------MES_MAP_PROCESS--------------------*/ + +#ifndef PM4_MES_MAP_PROCESS_DEFINED +#define PM4_MES_MAP_PROCESS_DEFINED + +struct pm4_mes_map_process { + union { + union PM4_MES_TYPE_3_HEADER header; /* header */ + uint32_t ordinal1; + }; + + union { + struct { + uint32_t pasid:16; + uint32_t reserved1:8; + uint32_t diq_enable:1; + uint32_t process_quantum:7; + } bitfields2; + uint32_t ordinal2; + }; + + uint32_t vm_context_page_table_base_addr_lo32; + + uint32_t vm_context_page_table_base_addr_hi32; + + uint32_t sh_mem_bases; + + uint32_t sh_mem_config; + + uint32_t sq_shader_tba_lo; + + uint32_t sq_shader_tba_hi; + + uint32_t sq_shader_tma_lo; + + uint32_t sq_shader_tma_hi; + + uint32_t reserved6; + + uint32_t gds_addr_lo; + + uint32_t gds_addr_hi; + + union { + struct { + uint32_t num_gws:6; + uint32_t reserved7:1; + uint32_t sdma_enable:1; + uint32_t num_oac:4; + uint32_t reserved8:4; + uint32_t gds_size:6; + uint32_t num_queues:10; + } bitfields14; + uint32_t ordinal14; + }; + + uint32_t completion_signal_lo; + + uint32_t completion_signal_hi; + +}; + +#endif + +/*--------------------MES_MAP_PROCESS_VM--------------------*/ + +#ifndef PM4_MES_MAP_PROCESS_VM_DEFINED +#define PM4_MES_MAP_PROCESS_VM_DEFINED + +struct PM4_MES_MAP_PROCESS_VM { + union { + union PM4_MES_TYPE_3_HEADER header; /* header */ + uint32_t ordinal1; + }; + + uint32_t reserved1; + + uint32_t vm_context_cntl; + + uint32_t reserved2; + + uint32_t vm_context_page_table_end_addr_lo32; + + uint32_t vm_context_page_table_end_addr_hi32; + + uint32_t vm_context_page_table_start_addr_lo32; + + uint32_t vm_context_page_table_start_addr_hi32; + + uint32_t reserved3; + + uint32_t reserved4; + + uint32_t reserved5; + + uint32_t reserved6; + + uint32_t reserved7; + + uint32_t reserved8; + + uint32_t completion_signal_lo32; + + uint32_t completion_signal_hi32; + +}; +#endif + +/*--------------------MES_MAP_QUEUES--------------------*/ + +#ifndef PM4_MES_MAP_QUEUES_VI_DEFINED +#define PM4_MES_MAP_QUEUES_VI_DEFINED +enum mes_map_queues_queue_sel_enum { + queue_sel__mes_map_queues__map_to_specified_queue_slots_vi = 0, +queue_sel__mes_map_queues__map_to_hws_determined_queue_slots_vi = 1 +}; + +enum mes_map_queues_queue_type_enum { + queue_type__mes_map_queues__normal_compute_vi = 0, + queue_type__mes_map_queues__debug_interface_queue_vi = 1, + queue_type__mes_map_queues__normal_latency_static_queue_vi = 2, +queue_type__mes_map_queues__low_latency_static_queue_vi = 3 +}; + +enum mes_map_queues_alloc_format_enum { + alloc_format__mes_map_queues__one_per_pipe_vi = 0, +alloc_format__mes_map_queues__all_on_one_pipe_vi = 1 +}; + +enum mes_map_queues_engine_sel_enum { + engine_sel__mes_map_queues__compute_vi = 0, + engine_sel__mes_map_queues__sdma0_vi = 2, + engine_sel__mes_map_queues__sdma1_vi = 3 +}; + + +struct pm4_mes_map_queues { + union { + union PM4_MES_TYPE_3_HEADER header; /* header */ + uint32_t ordinal1; + }; + + union { + struct { + uint32_t reserved1:4; + enum mes_map_queues_queue_sel_enum queue_sel:2; + uint32_t reserved2:15; + enum mes_map_queues_queue_type_enum queue_type:3; + enum mes_map_queues_alloc_format_enum alloc_format:2; + enum mes_map_queues_engine_sel_enum engine_sel:3; + uint32_t num_queues:3; + } bitfields2; + uint32_t ordinal2; + }; + + union { + struct { + uint32_t reserved3:1; + uint32_t check_disable:1; + uint32_t doorbell_offset:26; + uint32_t reserved4:4; + } bitfields3; + uint32_t ordinal3; + }; + + uint32_t mqd_addr_lo; + uint32_t mqd_addr_hi; + uint32_t wptr_addr_lo; + uint32_t wptr_addr_hi; +}; +#endif + +/*--------------------MES_QUERY_STATUS--------------------*/ + +#ifndef PM4_MES_QUERY_STATUS_DEFINED +#define PM4_MES_QUERY_STATUS_DEFINED +enum mes_query_status_interrupt_sel_enum { + interrupt_sel__mes_query_status__completion_status = 0, + interrupt_sel__mes_query_status__process_status = 1, + interrupt_sel__mes_query_status__queue_status = 2 +}; + +enum mes_query_status_command_enum { + command__mes_query_status__interrupt_only = 0, + command__mes_query_status__fence_only_immediate = 1, + command__mes_query_status__fence_only_after_write_ack = 2, + command__mes_query_status__fence_wait_for_write_ack_send_interrupt = 3 +}; + +enum mes_query_status_engine_sel_enum { + engine_sel__mes_query_status__compute = 0, + engine_sel__mes_query_status__sdma0_queue = 2, + engine_sel__mes_query_status__sdma1_queue = 3 +}; + +struct pm4_mes_query_status { + union { + union PM4_MES_TYPE_3_HEADER header; /* header */ + uint32_t ordinal1; + }; + + union { + struct { + uint32_t context_id:28; + enum mes_query_status_interrupt_sel_enum interrupt_sel:2; + enum mes_query_status_command_enum command:2; + } bitfields2; + uint32_t ordinal2; + }; + + union { + struct { + uint32_t pasid:16; + uint32_t reserved1:16; + } bitfields3a; + struct { + uint32_t reserved2:2; + uint32_t doorbell_offset:26; + enum mes_query_status_engine_sel_enum engine_sel:3; + uint32_t reserved3:1; + } bitfields3b; + uint32_t ordinal3; + }; + + uint32_t addr_lo; + uint32_t addr_hi; + uint32_t data_lo; + uint32_t data_hi; +}; +#endif + +/*--------------------MES_UNMAP_QUEUES--------------------*/ + +#ifndef PM4_MES_UNMAP_QUEUES_DEFINED +#define PM4_MES_UNMAP_QUEUES_DEFINED +enum mes_unmap_queues_action_enum { + action__mes_unmap_queues__preempt_queues = 0, + action__mes_unmap_queues__reset_queues = 1, + action__mes_unmap_queues__disable_process_queues = 2, + action__mes_unmap_queues__reserved = 3 +}; + +enum mes_unmap_queues_queue_sel_enum { + queue_sel__mes_unmap_queues__perform_request_on_specified_queues = 0, + queue_sel__mes_unmap_queues__perform_request_on_pasid_queues = 1, + queue_sel__mes_unmap_queues__unmap_all_queues = 2, + queue_sel__mes_unmap_queues__unmap_all_non_static_queues = 3 +}; + +enum mes_unmap_queues_engine_sel_enum { + engine_sel__mes_unmap_queues__compute = 0, + engine_sel__mes_unmap_queues__sdma0 = 2, + engine_sel__mes_unmap_queues__sdmal = 3 +}; + +struct pm4_mes_unmap_queues { + union { + union PM4_MES_TYPE_3_HEADER header; /* header */ + uint32_t ordinal1; + }; + + union { + struct { + enum mes_unmap_queues_action_enum action:2; + uint32_t reserved1:2; + enum mes_unmap_queues_queue_sel_enum queue_sel:2; + uint32_t reserved2:20; + enum mes_unmap_queues_engine_sel_enum engine_sel:3; + uint32_t num_queues:3; + } bitfields2; + uint32_t ordinal2; + }; + + union { + struct { + uint32_t pasid:16; + uint32_t reserved3:16; + } bitfields3a; + struct { + uint32_t reserved4:2; + uint32_t doorbell_offset0:26; + int32_t reserved5:4; + } bitfields3b; + uint32_t ordinal3; + }; + + union { + struct { + uint32_t reserved6:2; + uint32_t doorbell_offset1:26; + uint32_t reserved7:4; + } bitfields4; + uint32_t ordinal4; + }; + + union { + struct { + uint32_t reserved8:2; + uint32_t doorbell_offset2:26; + uint32_t reserved9:4; + } bitfields5; + uint32_t ordinal5; + }; + + union { + struct { + uint32_t reserved10:2; + uint32_t doorbell_offset3:26; + uint32_t reserved11:4; + } bitfields6; + uint32_t ordinal6; + }; +}; +#endif + +#ifndef PM4_MEC_RELEASE_MEM_DEFINED +#define PM4_MEC_RELEASE_MEM_DEFINED + +enum mec_release_mem_event_index_enum { + event_index__mec_release_mem__end_of_pipe = 5, + event_index__mec_release_mem__shader_done = 6 +}; + +enum mec_release_mem_cache_policy_enum { + cache_policy__mec_release_mem__lru = 0, + cache_policy__mec_release_mem__stream = 1 +}; + +enum mec_release_mem_pq_exe_status_enum { + pq_exe_status__mec_release_mem__default = 0, + pq_exe_status__mec_release_mem__phase_update = 1 +}; + +enum mec_release_mem_dst_sel_enum { + dst_sel__mec_release_mem__memory_controller = 0, + dst_sel__mec_release_mem__tc_l2 = 1, + dst_sel__mec_release_mem__queue_write_pointer_register = 2, + dst_sel__mec_release_mem__queue_write_pointer_poll_mask_bit = 3 +}; + +enum mec_release_mem_int_sel_enum { + int_sel__mec_release_mem__none = 0, + int_sel__mec_release_mem__send_interrupt_only = 1, + int_sel__mec_release_mem__send_interrupt_after_write_confirm = 2, + int_sel__mec_release_mem__send_data_after_write_confirm = 3, + int_sel__mec_release_mem__unconditionally_send_int_ctxid = 4, + int_sel__mec_release_mem__conditionally_send_int_ctxid_based_on_32_bit_compare = 5, + int_sel__mec_release_mem__conditionally_send_int_ctxid_based_on_64_bit_compare = 6 +}; + +enum mec_release_mem_data_sel_enum { + data_sel__mec_release_mem__none = 0, + data_sel__mec_release_mem__send_32_bit_low = 1, + data_sel__mec_release_mem__send_64_bit_data = 2, + data_sel__mec_release_mem__send_gpu_clock_counter = 3, + data_sel__mec_release_mem__send_cp_perfcounter_hi_lo = 4, + data_sel__mec_release_mem__store_gds_data_to_memory = 5 +}; + +struct pm4_mec_release_mem { + union { + union PM4_MES_TYPE_3_HEADER header; /*header */ + unsigned int ordinal1; + }; + + union { + struct { + unsigned int event_type:6; + unsigned int reserved1:2; + enum mec_release_mem_event_index_enum event_index:4; + unsigned int tcl1_vol_action_ena:1; + unsigned int tc_vol_action_ena:1; + unsigned int reserved2:1; + unsigned int tc_wb_action_ena:1; + unsigned int tcl1_action_ena:1; + unsigned int tc_action_ena:1; + uint32_t reserved3:1; + uint32_t tc_nc_action_ena:1; + uint32_t tc_wc_action_ena:1; + uint32_t tc_md_action_ena:1; + uint32_t reserved4:3; + enum mec_release_mem_cache_policy_enum cache_policy:2; + uint32_t reserved5:2; + enum mec_release_mem_pq_exe_status_enum pq_exe_status:1; + uint32_t reserved6:2; + } bitfields2; + unsigned int ordinal2; + }; + + union { + struct { + uint32_t reserved7:16; + enum mec_release_mem_dst_sel_enum dst_sel:2; + uint32_t reserved8:6; + enum mec_release_mem_int_sel_enum int_sel:3; + uint32_t reserved9:2; + enum mec_release_mem_data_sel_enum data_sel:3; + } bitfields3; + unsigned int ordinal3; + }; + + union { + struct { + uint32_t reserved10:2; + unsigned int address_lo_32b:30; + } bitfields4; + struct { + uint32_t reserved11:3; + uint32_t address_lo_64b:29; + } bitfields4b; + uint32_t reserved12; + unsigned int ordinal4; + }; + + union { + uint32_t address_hi; + uint32_t reserved13; + uint32_t ordinal5; + }; + + union { + uint32_t data_lo; + uint32_t cmp_data_lo; + struct { + uint32_t dw_offset:16; + uint32_t num_dwords:16; + } bitfields6c; + uint32_t reserved14; + uint32_t ordinal6; + }; + + union { + uint32_t data_hi; + uint32_t cmp_data_hi; + uint32_t reserved15; + uint32_t reserved16; + uint32_t ordinal7; + }; + + uint32_t int_ctxid; + +}; + +#endif + +enum { + CACHE_FLUSH_AND_INV_TS_EVENT = 0x00000014 +}; +#endif + diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index 96a9cc0f02c9..5e3990bb4c4b 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -39,11 +39,37 @@ #include "amd_shared.h" +#define KFD_MAX_RING_ENTRY_SIZE 8 + #define KFD_SYSFS_FILE_MODE 0444 -#define KFD_MMAP_DOORBELL_MASK 0x8000000000000ull -#define KFD_MMAP_EVENTS_MASK 0x4000000000000ull -#define KFD_MMAP_RESERVED_MEM_MASK 0x2000000000000ull +/* GPU ID hash width in bits */ +#define KFD_GPU_ID_HASH_WIDTH 16 + +/* Use upper bits of mmap offset to store KFD driver specific information. + * BITS[63:62] - Encode MMAP type + * BITS[61:46] - Encode gpu_id. To identify to which GPU the offset belongs to + * BITS[45:0] - MMAP offset value + * + * NOTE: struct vm_area_struct.vm_pgoff uses offset in pages. Hence, these + * defines are w.r.t to PAGE_SIZE + */ +#define KFD_MMAP_TYPE_SHIFT (62 - PAGE_SHIFT) +#define KFD_MMAP_TYPE_MASK (0x3ULL << KFD_MMAP_TYPE_SHIFT) +#define KFD_MMAP_TYPE_DOORBELL (0x3ULL << KFD_MMAP_TYPE_SHIFT) +#define KFD_MMAP_TYPE_EVENTS (0x2ULL << KFD_MMAP_TYPE_SHIFT) +#define KFD_MMAP_TYPE_RESERVED_MEM (0x1ULL << KFD_MMAP_TYPE_SHIFT) + +#define KFD_MMAP_GPU_ID_SHIFT (46 - PAGE_SHIFT) +#define KFD_MMAP_GPU_ID_MASK (((1ULL << KFD_GPU_ID_HASH_WIDTH) - 1) \ + << KFD_MMAP_GPU_ID_SHIFT) +#define KFD_MMAP_GPU_ID(gpu_id) ((((uint64_t)gpu_id) << KFD_MMAP_GPU_ID_SHIFT)\ + & KFD_MMAP_GPU_ID_MASK) +#define KFD_MMAP_GPU_ID_GET(offset) ((offset & KFD_MMAP_GPU_ID_MASK) \ + >> KFD_MMAP_GPU_ID_SHIFT) + +#define KFD_MMAP_OFFSET_VALUE_MASK (0x3FFFFFFFFFFFULL >> PAGE_SHIFT) +#define KFD_MMAP_OFFSET_VALUE_GET(offset) (offset & KFD_MMAP_OFFSET_VALUE_MASK) /* * When working with cp scheduler we should assign the HIQ manually or via @@ -55,9 +81,6 @@ #define KFD_CIK_HIQ_PIPE 4 #define KFD_CIK_HIQ_QUEUE 0 -/* GPU ID hash width in bits */ -#define KFD_GPU_ID_HASH_WIDTH 16 - /* Macro for allocating structures */ #define kfd_alloc_struct(ptr_to_struct) \ ((typeof(ptr_to_struct)) kzalloc(sizeof(*ptr_to_struct), GFP_KERNEL)) @@ -116,6 +139,11 @@ extern int debug_largebar; */ extern int ignore_crat; +/* + * Set sh_mem_config.retry_disable on Vega10 + */ +extern int vega10_noretry; + /** * enum kfd_sched_policy * @@ -148,6 +176,8 @@ enum cache_policy { cache_policy_noncoherent }; +#define KFD_IS_SOC15(chip) ((chip) >= CHIP_VEGA10) + struct kfd_event_interrupt_class { bool (*interrupt_isr)(struct kfd_dev *dev, const uint32_t *ih_ring_entry); @@ -160,6 +190,7 @@ struct kfd_device_info { const struct kfd_event_interrupt_class *event_interrupt_class; unsigned int max_pasid_bits; unsigned int max_no_of_hqd; + unsigned int doorbell_size; size_t ih_ring_entry_size; uint8_t num_of_watch_points; uint16_t mqd_size_aligned; @@ -173,6 +204,7 @@ struct kfd_mem_obj { uint32_t range_end; uint64_t gpu_addr; uint32_t *cpu_ptr; + void *gtt_mem; }; struct kfd_vmid_info { @@ -364,7 +396,7 @@ struct queue_properties { uint32_t queue_percent; uint32_t *read_ptr; uint32_t *write_ptr; - uint32_t __iomem *doorbell_ptr; + void __iomem *doorbell_ptr; uint32_t doorbell_off; bool is_interop; bool is_evicted; @@ -427,6 +459,7 @@ struct queue { uint32_t queue; unsigned int sdma_id; + unsigned int doorbell_id; struct kfd_process *process; struct kfd_dev *device; @@ -501,6 +534,9 @@ struct qcm_process_device { /* IB memory */ uint64_t ib_base; void *ib_kaddr; + + /* doorbell resources per process per device */ + unsigned long *doorbell_bitmap; }; /* KFD Memory Eviction */ @@ -512,6 +548,8 @@ struct qcm_process_device { /* Approx. time before evicting the process again */ #define PROCESS_ACTIVE_TIME_MS 10 +int kgd2kfd_quiesce_mm(struct mm_struct *mm); +int kgd2kfd_resume_mm(struct mm_struct *mm); int kgd2kfd_schedule_evict_and_restore_process(struct mm_struct *mm, struct dma_fence *fence); @@ -681,6 +719,8 @@ struct kfd_process *kfd_get_process(const struct task_struct *); struct kfd_process *kfd_lookup_process_by_pasid(unsigned int pasid); struct kfd_process *kfd_lookup_process_by_mm(const struct mm_struct *mm); void kfd_unref_process(struct kfd_process *p); +int kfd_process_evict_queues(struct kfd_process *p); +int kfd_process_restore_queues(struct kfd_process *p); void kfd_suspend_all_processes(void); int kfd_resume_all_processes(void); @@ -693,7 +733,7 @@ struct kfd_process_device *kfd_get_process_device_data(struct kfd_dev *dev, struct kfd_process_device *kfd_create_process_device_data(struct kfd_dev *dev, struct kfd_process *p); -int kfd_reserved_mem_mmap(struct kfd_process *process, +int kfd_reserved_mem_mmap(struct kfd_dev *dev, struct kfd_process *process, struct vm_area_struct *vma); /* KFD process API for creating and translating handles */ @@ -721,17 +761,20 @@ unsigned int kfd_pasid_alloc(void); void kfd_pasid_free(unsigned int pasid); /* Doorbells */ +size_t kfd_doorbell_process_slice(struct kfd_dev *kfd); int kfd_doorbell_init(struct kfd_dev *kfd); void kfd_doorbell_fini(struct kfd_dev *kfd); -int kfd_doorbell_mmap(struct kfd_process *process, struct vm_area_struct *vma); -u32 __iomem *kfd_get_kernel_doorbell(struct kfd_dev *kfd, +int kfd_doorbell_mmap(struct kfd_dev *dev, struct kfd_process *process, + struct vm_area_struct *vma); +void __iomem *kfd_get_kernel_doorbell(struct kfd_dev *kfd, unsigned int *doorbell_off); void kfd_release_kernel_doorbell(struct kfd_dev *kfd, u32 __iomem *db_addr); u32 read_kernel_doorbell(u32 __iomem *db); -void write_kernel_doorbell(u32 __iomem *db, u32 value); -unsigned int kfd_queue_id_to_doorbell(struct kfd_dev *kfd, +void write_kernel_doorbell(void __iomem *db, u32 value); +void write_kernel_doorbell64(void __iomem *db, u64 value); +unsigned int kfd_doorbell_id_to_offset(struct kfd_dev *kfd, struct kfd_process *process, - unsigned int queue_id); + unsigned int doorbell_id); phys_addr_t kfd_get_process_doorbells(struct kfd_dev *dev, struct kfd_process *process); int kfd_alloc_process_doorbells(struct kfd_process *process); @@ -788,6 +831,8 @@ struct mqd_manager *mqd_manager_init_vi(enum KFD_MQD_TYPE type, struct kfd_dev *dev); struct mqd_manager *mqd_manager_init_vi_tonga(enum KFD_MQD_TYPE type, struct kfd_dev *dev); +struct mqd_manager *mqd_manager_init_v9(enum KFD_MQD_TYPE type, + struct kfd_dev *dev); struct device_queue_manager *device_queue_manager_init(struct kfd_dev *dev); void device_queue_manager_uninit(struct device_queue_manager *dqm); struct kernel_queue *kernel_queue_init(struct kfd_dev *dev, @@ -832,8 +877,42 @@ struct packet_manager { bool allocated; struct kfd_mem_obj *ib_buffer_obj; unsigned int ib_size_bytes; + + const struct packet_manager_funcs *pmf; +}; + +struct packet_manager_funcs { + /* Support ASIC-specific packet formats for PM4 packets */ + int (*map_process)(struct packet_manager *pm, uint32_t *buffer, + struct qcm_process_device *qpd); + int (*runlist)(struct packet_manager *pm, uint32_t *buffer, + uint64_t ib, size_t ib_size_in_dwords, bool chain); + int (*set_resources)(struct packet_manager *pm, uint32_t *buffer, + struct scheduling_resources *res); + int (*map_queues)(struct packet_manager *pm, uint32_t *buffer, + struct queue *q, bool is_static); + int (*unmap_queues)(struct packet_manager *pm, uint32_t *buffer, + enum kfd_queue_type type, + enum kfd_unmap_queues_filter mode, + uint32_t filter_param, bool reset, + unsigned int sdma_engine); + int (*query_status)(struct packet_manager *pm, uint32_t *buffer, + uint64_t fence_address, uint32_t fence_value); + int (*release_mem)(uint64_t gpu_addr, uint32_t *buffer); + + /* Packet sizes */ + int map_process_size; + int runlist_size; + int set_resources_size; + int map_queues_size; + int unmap_queues_size; + int query_status_size; + int release_mem_size; }; +extern const struct packet_manager_funcs kfd_vi_pm_funcs; +extern const struct packet_manager_funcs kfd_v9_pm_funcs; + int pm_init(struct packet_manager *pm, struct device_queue_manager *dqm); void pm_uninit(struct packet_manager *pm); int pm_send_set_resources(struct packet_manager *pm, @@ -849,12 +928,17 @@ int pm_send_unmap_queue(struct packet_manager *pm, enum kfd_queue_type type, void pm_release_ib(struct packet_manager *pm); -uint32_t pm_create_release_mem(uint64_t gpu_addr, uint32_t *buffer); +/* Following PM funcs can be shared among VI and AI */ +unsigned int pm_build_pm4_header(unsigned int opcode, size_t packet_size); +int pm_set_resources_vi(struct packet_manager *pm, uint32_t *buffer, + struct scheduling_resources *res); uint64_t kfd_get_number_elems(struct kfd_dev *kfd); /* Events */ extern const struct kfd_event_interrupt_class event_interrupt_class_cik; +extern const struct kfd_event_interrupt_class event_interrupt_class_v9; + extern const struct kfd_device_global_init_class device_global_init_class_cik; void kfd_event_init_process(struct kfd_process *p); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index 1711ad0642f7..1d80b4f7c681 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -332,6 +332,7 @@ static void kfd_process_destroy_pdds(struct kfd_process *p) free_pages((unsigned long)pdd->qpd.cwsr_kaddr, get_order(KFD_CWSR_TBA_TMA_SIZE)); + kfree(pdd->qpd.doorbell_bitmap); idr_destroy(&pdd->alloc_idr); kfree(pdd); @@ -451,7 +452,8 @@ static int kfd_process_init_cwsr_apu(struct kfd_process *p, struct file *filep) if (!dev->cwsr_enabled || qpd->cwsr_kaddr || qpd->cwsr_base) continue; - offset = (dev->id | KFD_MMAP_RESERVED_MEM_MASK) << PAGE_SHIFT; + offset = (KFD_MMAP_TYPE_RESERVED_MEM | KFD_MMAP_GPU_ID(dev->id)) + << PAGE_SHIFT; qpd->tba_addr = (int64_t)vm_mmap(filep, 0, KFD_CWSR_TBA_TMA_SIZE, PROT_READ | PROT_EXEC, MAP_SHARED, offset); @@ -585,6 +587,31 @@ err_alloc_process: return ERR_PTR(err); } +static int init_doorbell_bitmap(struct qcm_process_device *qpd, + struct kfd_dev *dev) +{ + unsigned int i; + + if (!KFD_IS_SOC15(dev->device_info->asic_family)) + return 0; + + qpd->doorbell_bitmap = + kzalloc(DIV_ROUND_UP(KFD_MAX_NUM_OF_QUEUES_PER_PROCESS, + BITS_PER_BYTE), GFP_KERNEL); + if (!qpd->doorbell_bitmap) + return -ENOMEM; + + /* Mask out any reserved doorbells */ + for (i = 0; i < KFD_MAX_NUM_OF_QUEUES_PER_PROCESS; i++) + if ((dev->shared_resources.reserved_doorbell_mask & i) == + dev->shared_resources.reserved_doorbell_val) { + set_bit(i, qpd->doorbell_bitmap); + pr_debug("reserved doorbell 0x%03x\n", i); + } + + return 0; +} + struct kfd_process_device *kfd_get_process_device_data(struct kfd_dev *dev, struct kfd_process *p) { @@ -606,6 +633,12 @@ struct kfd_process_device *kfd_create_process_device_data(struct kfd_dev *dev, if (!pdd) return NULL; + if (init_doorbell_bitmap(&pdd->qpd, dev)) { + pr_err("Failed to init doorbell for process\n"); + kfree(pdd); + return NULL; + } + pdd->dev = dev; INIT_LIST_HEAD(&pdd->qpd.queues_list); INIT_LIST_HEAD(&pdd->qpd.priv_queue_list); @@ -808,7 +841,7 @@ struct kfd_process *kfd_lookup_process_by_mm(const struct mm_struct *mm) * Eviction is reference-counted per process-device. This means multiple * evictions from different sources can be nested safely. */ -static int process_evict_queues(struct kfd_process *p) +int kfd_process_evict_queues(struct kfd_process *p) { struct kfd_process_device *pdd; int r = 0; @@ -844,7 +877,7 @@ fail: } /* process_restore_queues - Restore all user queues of a process */ -static int process_restore_queues(struct kfd_process *p) +int kfd_process_restore_queues(struct kfd_process *p) { struct kfd_process_device *pdd; int r, ret = 0; @@ -886,7 +919,7 @@ static void evict_process_worker(struct work_struct *work) flush_delayed_work(&p->restore_work); pr_debug("Started evicting pasid %d\n", p->pasid); - ret = process_evict_queues(p); + ret = kfd_process_evict_queues(p); if (!ret) { dma_fence_signal(p->ef); dma_fence_put(p->ef); @@ -946,7 +979,7 @@ static void restore_process_worker(struct work_struct *work) return; } - ret = process_restore_queues(p); + ret = kfd_process_restore_queues(p); if (!ret) pr_debug("Finished restoring pasid %d\n", p->pasid); else @@ -963,7 +996,7 @@ void kfd_suspend_all_processes(void) cancel_delayed_work_sync(&p->eviction_work); cancel_delayed_work_sync(&p->restore_work); - if (process_evict_queues(p)) + if (kfd_process_evict_queues(p)) pr_err("Failed to suspend process %d\n", p->pasid); dma_fence_signal(p->ef); dma_fence_put(p->ef); @@ -989,15 +1022,12 @@ int kfd_resume_all_processes(void) return ret; } -int kfd_reserved_mem_mmap(struct kfd_process *process, +int kfd_reserved_mem_mmap(struct kfd_dev *dev, struct kfd_process *process, struct vm_area_struct *vma) { - struct kfd_dev *dev = kfd_device_by_id(vma->vm_pgoff); struct kfd_process_device *pdd; struct qcm_process_device *qpd; - if (!dev) - return -EINVAL; if ((vma->vm_end - vma->vm_start) != KFD_CWSR_TBA_TMA_SIZE) { pr_err("Incorrect CWSR mapping size.\n"); return -EINVAL; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c index 7817e327ea6d..d65ce0436b31 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c @@ -119,9 +119,6 @@ static int create_cp_queue(struct process_queue_manager *pqm, /* Doorbell initialized in user space*/ q_properties->doorbell_ptr = NULL; - q_properties->doorbell_off = - kfd_queue_id_to_doorbell(dev, pqm->process, qid); - /* let DQM handle it*/ q_properties->vmid = 0; q_properties->queue_id = qid; @@ -244,10 +241,20 @@ int pqm_create_queue(struct process_queue_manager *pqm, } if (retval != 0) { - pr_err("DQM create queue failed\n"); + pr_err("Pasid %d DQM create queue %d failed. ret %d\n", + pqm->process->pasid, type, retval); goto err_create_queue; } + if (q) + /* Return the doorbell offset within the doorbell page + * to the caller so it can be passed up to user mode + * (in bytes). + */ + properties->doorbell_off = + (q->properties.doorbell_off * sizeof(uint32_t)) & + (kfd_doorbell_process_slice(dev) - 1); + pr_debug("PQM After DQM create queue\n"); list_add(&pqn->process_queue_list, &pqm->queues); @@ -313,8 +320,11 @@ int pqm_destroy_queue(struct process_queue_manager *pqm, unsigned int qid) dqm = pqn->q->device->dqm; retval = dqm->ops.destroy_queue(dqm, &pdd->qpd, pqn->q); if (retval) { - pr_debug("Destroy queue failed, returned %d\n", retval); - goto err_destroy_queue; + pr_err("Pasid %d destroy queue %d failed, ret %d\n", + pqm->process->pasid, + pqn->q->properties.queue_id, retval); + if (retval != -ETIME) + goto err_destroy_queue; } uninit_queue(pqn->q); } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_queue.c b/drivers/gpu/drm/amd/amdkfd/kfd_queue.c index a5315d4f1c95..6dcd621e5b71 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_queue.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_queue.c @@ -36,8 +36,8 @@ void print_queue_properties(struct queue_properties *q) pr_debug("Queue Address: 0x%llX\n", q->queue_address); pr_debug("Queue Id: %u\n", q->queue_id); pr_debug("Queue Process Vmid: %u\n", q->vmid); - pr_debug("Queue Read Pointer: 0x%p\n", q->read_ptr); - pr_debug("Queue Write Pointer: 0x%p\n", q->write_ptr); + pr_debug("Queue Read Pointer: 0x%px\n", q->read_ptr); + pr_debug("Queue Write Pointer: 0x%px\n", q->write_ptr); pr_debug("Queue Doorbell Pointer: 0x%p\n", q->doorbell_ptr); pr_debug("Queue Doorbell Offset: %u\n", q->doorbell_off); } @@ -53,8 +53,8 @@ void print_queue(struct queue *q) pr_debug("Queue Address: 0x%llX\n", q->properties.queue_address); pr_debug("Queue Id: %u\n", q->properties.queue_id); pr_debug("Queue Process Vmid: %u\n", q->properties.vmid); - pr_debug("Queue Read Pointer: 0x%p\n", q->properties.read_ptr); - pr_debug("Queue Write Pointer: 0x%p\n", q->properties.write_ptr); + pr_debug("Queue Read Pointer: 0x%px\n", q->properties.read_ptr); + pr_debug("Queue Write Pointer: 0x%px\n", q->properties.write_ptr); pr_debug("Queue Doorbell Pointer: 0x%p\n", q->properties.doorbell_ptr); pr_debug("Queue Doorbell Offset: %u\n", q->properties.doorbell_off); pr_debug("Queue MQD Address: 0x%p\n", q->mqd); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c index ac28abc94e57..bc95d4dfee2e 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c @@ -1239,6 +1239,12 @@ int kfd_topology_add_device(struct kfd_dev *gpu) HSA_CAP_DOORBELL_TYPE_TOTALBITS_SHIFT) & HSA_CAP_DOORBELL_TYPE_TOTALBITS_MASK); break; + case CHIP_VEGA10: + case CHIP_RAVEN: + dev->node_props.capability |= ((HSA_CAP_DOORBELL_TYPE_2_0 << + HSA_CAP_DOORBELL_TYPE_TOTALBITS_SHIFT) & + HSA_CAP_DOORBELL_TYPE_TOTALBITS_MASK); + break; default: WARN(1, "Unexpected ASIC family %u", dev->gpu->device_info->asic_family); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.h b/drivers/gpu/drm/amd/amdkfd/kfd_topology.h index eb54cfcaf039..7d9c3f948dff 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.h @@ -45,6 +45,7 @@ #define HSA_CAP_DOORBELL_TYPE_PRE_1_0 0x0 #define HSA_CAP_DOORBELL_TYPE_1_0 0x1 +#define HSA_CAP_DOORBELL_TYPE_2_0 0x2 #define HSA_CAP_AQL_QUEUE_DOUBLE_MAP 0x00004000 struct kfd_node_properties { diff --git a/drivers/gpu/drm/amd/amdkfd/soc15_int.h b/drivers/gpu/drm/amd/amdkfd/soc15_int.h new file mode 100644 index 000000000000..0bc0b25cb410 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/soc15_int.h @@ -0,0 +1,47 @@ +/* + * Copyright 2016-2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#ifndef HSA_SOC15_INT_H_INCLUDED +#define HSA_SOC15_INT_H_INCLUDED + +#include "soc15_ih_clientid.h" + +#define SOC15_INTSRC_CP_END_OF_PIPE 181 +#define SOC15_INTSRC_CP_BAD_OPCODE 183 +#define SOC15_INTSRC_SQ_INTERRUPT_MSG 239 +#define SOC15_INTSRC_VMC_FAULT 0 +#define SOC15_INTSRC_SDMA_TRAP 224 + + +#define SOC15_CLIENT_ID_FROM_IH_ENTRY(entry) (le32_to_cpu(entry[0]) & 0xff) +#define SOC15_SOURCE_ID_FROM_IH_ENTRY(entry) (le32_to_cpu(entry[0]) >> 8 & 0xff) +#define SOC15_RING_ID_FROM_IH_ENTRY(entry) (le32_to_cpu(entry[0]) >> 16 & 0xff) +#define SOC15_VMID_FROM_IH_ENTRY(entry) (le32_to_cpu(entry[0]) >> 24 & 0xf) +#define SOC15_VMID_TYPE_FROM_IH_ENTRY(entry) (le32_to_cpu(entry[0]) >> 31 & 0x1) +#define SOC15_PASID_FROM_IH_ENTRY(entry) (le32_to_cpu(entry[3]) & 0xffff) +#define SOC15_CONTEXT_ID0_FROM_IH_ENTRY(entry) (le32_to_cpu(entry[4])) +#define SOC15_CONTEXT_ID1_FROM_IH_ENTRY(entry) (le32_to_cpu(entry[5])) +#define SOC15_CONTEXT_ID2_FROM_IH_ENTRY(entry) (le32_to_cpu(entry[6])) +#define SOC15_CONTEXT_ID3_FROM_IH_ENTRY(entry) (le32_to_cpu(entry[7])) + +#endif + diff --git a/drivers/gpu/drm/amd/display/Kconfig b/drivers/gpu/drm/amd/display/Kconfig index 5b124a67404c..d5d4586e6176 100644 --- a/drivers/gpu/drm/amd/display/Kconfig +++ b/drivers/gpu/drm/amd/display/Kconfig @@ -9,14 +9,6 @@ config DRM_AMD_DC support for AMDGPU. This adds required support for Vega and Raven ASICs. -config DRM_AMD_DC_PRE_VEGA - bool "DC support for Polaris and older ASICs" - default y - help - Choose this option to enable the new DC support for older asics - by default. This includes Polaris, Carrizo, Tonga, Bonaire, - and Hawaii. - config DRM_AMD_DC_FBC bool "AMD FBC - Enable Frame Buffer Compression" depends on DRM_AMD_DC diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c index 1dd1142246c2..f9add85157e7 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c @@ -46,6 +46,7 @@ #include #include #include +#include #include #include @@ -433,11 +434,6 @@ static int amdgpu_dm_init(struct amdgpu_device *adev) init_data.dce_environment = DCE_ENV_PRODUCTION_DRV; - if (amdgpu_dc_log) - init_data.log_mask = DC_DEFAULT_LOG_MASK; - else - init_data.log_mask = DC_MIN_LOG_MASK; - /* * TODO debug why this doesn't work on Raven */ @@ -649,18 +645,6 @@ amdgpu_dm_find_first_crtc_matching_connector(struct drm_atomic_state *state, static int dm_resume(void *handle) { struct amdgpu_device *adev = handle; - struct amdgpu_display_manager *dm = &adev->dm; - int ret = 0; - - /* power on hardware */ - dc_set_power_state(dm->dc, DC_ACPI_CM_POWER_STATE_D0); - - ret = amdgpu_dm_display_resume(adev); - return ret; -} - -int amdgpu_dm_display_resume(struct amdgpu_device *adev) -{ struct drm_device *ddev = adev->ddev; struct amdgpu_display_manager *dm = &adev->dm; struct amdgpu_dm_connector *aconnector; @@ -671,10 +655,12 @@ int amdgpu_dm_display_resume(struct amdgpu_device *adev) struct drm_plane *plane; struct drm_plane_state *new_plane_state; struct dm_plane_state *dm_new_plane_state; - - int ret = 0; + int ret; int i; + /* power on hardware */ + dc_set_power_state(dm->dc, DC_ACPI_CM_POWER_STATE_D0); + /* program HPD filter */ dc_resume(dm->dc); @@ -688,8 +674,7 @@ int amdgpu_dm_display_resume(struct amdgpu_device *adev) amdgpu_dm_irq_resume_early(adev); /* Do detection*/ - list_for_each_entry(connector, - &ddev->mode_config.connector_list, head) { + list_for_each_entry(connector, &ddev->mode_config.connector_list, head) { aconnector = to_amdgpu_dm_connector(connector); /* @@ -711,7 +696,7 @@ int amdgpu_dm_display_resume(struct amdgpu_device *adev) } /* Force mode set in atomic comit */ - for_each_new_crtc_in_state(adev->dm.cached_state, crtc, new_crtc_state, i) + for_each_new_crtc_in_state(dm->cached_state, crtc, new_crtc_state, i) new_crtc_state->active_changed = true; /* @@ -719,7 +704,7 @@ int amdgpu_dm_display_resume(struct amdgpu_device *adev) * them here, since they were duplicated as part of the suspend * procedure. */ - for_each_new_crtc_in_state(adev->dm.cached_state, crtc, new_crtc_state, i) { + for_each_new_crtc_in_state(dm->cached_state, crtc, new_crtc_state, i) { dm_new_crtc_state = to_dm_crtc_state(new_crtc_state); if (dm_new_crtc_state->stream) { WARN_ON(kref_read(&dm_new_crtc_state->stream->refcount) > 1); @@ -728,7 +713,7 @@ int amdgpu_dm_display_resume(struct amdgpu_device *adev) } } - for_each_new_plane_in_state(adev->dm.cached_state, plane, new_plane_state, i) { + for_each_new_plane_in_state(dm->cached_state, plane, new_plane_state, i) { dm_new_plane_state = to_dm_plane_state(new_plane_state); if (dm_new_plane_state->dc_state) { WARN_ON(kref_read(&dm_new_plane_state->dc_state->refcount) > 1); @@ -737,9 +722,9 @@ int amdgpu_dm_display_resume(struct amdgpu_device *adev) } } - ret = drm_atomic_helper_resume(ddev, adev->dm.cached_state); + ret = drm_atomic_helper_resume(ddev, dm->cached_state); - adev->dm.cached_state = NULL; + dm->cached_state = NULL; amdgpu_dm_irq_resume_late(adev); @@ -927,6 +912,7 @@ amdgpu_dm_update_connector_after_detect(struct amdgpu_dm_connector *aconnector) drm_mode_connector_update_edid_property(connector, NULL); aconnector->num_modes = 0; aconnector->dc_sink = NULL; + aconnector->edid = NULL; } mutex_unlock(&dev->mode_config.mutex); @@ -1131,6 +1117,7 @@ static int dce110_register_irq_handlers(struct amdgpu_device *adev) if (adev->asic_type == CHIP_VEGA10 || adev->asic_type == CHIP_VEGA12 || + adev->asic_type == CHIP_VEGA20 || adev->asic_type == CHIP_RAVEN) client_id = SOC15_IH_CLIENTID_DCE; @@ -1529,8 +1516,10 @@ static int amdgpu_dm_initialize_drm_device(struct amdgpu_device *adev) case CHIP_POLARIS11: case CHIP_POLARIS10: case CHIP_POLARIS12: + case CHIP_VEGAM: case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: if (dce110_register_irq_handlers(dm->adev)) { DRM_ERROR("DM: Failed to initialize IRQ\n"); goto fail; @@ -1549,7 +1538,7 @@ static int amdgpu_dm_initialize_drm_device(struct amdgpu_device *adev) break; #endif default: - DRM_ERROR("Usupported ASIC type: 0x%X\n", adev->asic_type); + DRM_ERROR("Unsupported ASIC type: 0x%X\n", adev->asic_type); goto fail; } @@ -1657,7 +1646,6 @@ static ssize_t s3_debug_store(struct device *device, if (ret == 0) { if (s3_state) { dm_resume(adev); - amdgpu_dm_display_resume(adev); drm_kms_helper_hotplug_event(adev->ddev); } else dm_suspend(adev); @@ -1722,6 +1710,7 @@ static int dm_early_init(void *handle) adev->mode_info.plane_type = dm_plane_type_default; break; case CHIP_POLARIS10: + case CHIP_VEGAM: adev->mode_info.num_crtc = 6; adev->mode_info.num_hpd = 6; adev->mode_info.num_dig = 6; @@ -1729,6 +1718,7 @@ static int dm_early_init(void *handle) break; case CHIP_VEGA10: case CHIP_VEGA12: + case CHIP_VEGA20: adev->mode_info.num_crtc = 6; adev->mode_info.num_hpd = 6; adev->mode_info.num_dig = 6; @@ -1743,7 +1733,7 @@ static int dm_early_init(void *handle) break; #endif default: - DRM_ERROR("Usupported ASIC type: 0x%X\n", adev->asic_type); + DRM_ERROR("Unsupported ASIC type: 0x%X\n", adev->asic_type); return -EINVAL; } @@ -1848,7 +1838,7 @@ static bool fill_rects_from_plane_state(const struct drm_plane_state *state, static int get_fb_info(const struct amdgpu_framebuffer *amdgpu_fb, uint64_t *tiling_flags) { - struct amdgpu_bo *rbo = gem_to_amdgpu_bo(amdgpu_fb->obj); + struct amdgpu_bo *rbo = gem_to_amdgpu_bo(amdgpu_fb->base.obj[0]); int r = amdgpu_bo_reserve(rbo, false); if (unlikely(r)) { @@ -1977,6 +1967,7 @@ static int fill_plane_attributes_from_fb(struct amdgpu_device *adev, if (adev->asic_type == CHIP_VEGA10 || adev->asic_type == CHIP_VEGA12 || + adev->asic_type == CHIP_VEGA20 || adev->asic_type == CHIP_RAVEN) { /* Fill GFX9 params */ plane_state->tiling_info.gfx9.num_pipes = @@ -2017,7 +2008,6 @@ static int fill_plane_attributes(struct amdgpu_device *adev, const struct amdgpu_framebuffer *amdgpu_fb = to_amdgpu_framebuffer(plane_state->fb); const struct drm_crtc *crtc = plane_state->crtc; - struct dc_transfer_func *input_tf; int ret = 0; if (!fill_rects_from_plane_state(plane_state, dc_plane_state)) @@ -2031,13 +2021,6 @@ static int fill_plane_attributes(struct amdgpu_device *adev, if (ret) return ret; - input_tf = dc_create_transfer_func(); - - if (input_tf == NULL) - return -ENOMEM; - - dc_plane_state->in_transfer_func = input_tf; - /* * Always set input transfer function, since plane state is refreshed * every time. @@ -2113,12 +2096,6 @@ convert_color_depth_from_display_info(const struct drm_connector *connector) { uint32_t bpc = connector->display_info.bpc; - /* Limited color depth to 8bit - * TODO: Still need to handle deep color - */ - if (bpc > 8) - bpc = 8; - switch (bpc) { case 0: /* Temporary Work around, DRM don't parse color depth for @@ -2206,7 +2183,6 @@ fill_stream_properties_from_drm_display_mode(struct dc_stream_state *stream, const struct drm_connector *connector) { struct dc_crtc_timing *timing_out = &stream->timing; - struct dc_transfer_func *tf = dc_create_transfer_func(); memset(timing_out, 0, sizeof(struct dc_crtc_timing)); @@ -2250,9 +2226,8 @@ fill_stream_properties_from_drm_display_mode(struct dc_stream_state *stream, stream->output_color_space = get_output_color_space(timing_out); - tf->type = TF_TYPE_PREDEFINED; - tf->tf = TRANSFER_FUNCTION_SRGB; - stream->out_transfer_func = tf; + stream->out_transfer_func->type = TF_TYPE_PREDEFINED; + stream->out_transfer_func->tf = TRANSFER_FUNCTION_SRGB; } static void fill_audio_info(struct audio_info *audio_info, @@ -2336,27 +2311,22 @@ decide_crtc_timing_for_drm_display_mode(struct drm_display_mode *drm_mode, } } -static int create_fake_sink(struct amdgpu_dm_connector *aconnector) +static struct dc_sink * +create_fake_sink(struct amdgpu_dm_connector *aconnector) { - struct dc_sink *sink = NULL; struct dc_sink_init_data sink_init_data = { 0 }; - + struct dc_sink *sink = NULL; sink_init_data.link = aconnector->dc_link; sink_init_data.sink_signal = aconnector->dc_link->connector_signal; sink = dc_sink_create(&sink_init_data); if (!sink) { DRM_ERROR("Failed to create sink!\n"); - return -ENOMEM; + return NULL; } - sink->sink_signal = SIGNAL_TYPE_VIRTUAL; - aconnector->fake_enable = true; - aconnector->dc_sink = sink; - aconnector->dc_link->local_sink = sink; - - return 0; + return sink; } static void set_multisync_trigger_params( @@ -2419,7 +2389,7 @@ create_stream_for_sink(struct amdgpu_dm_connector *aconnector, struct dc_stream_state *stream = NULL; struct drm_display_mode mode = *drm_mode; bool native_mode_found = false; - + struct dc_sink *sink = NULL; if (aconnector == NULL) { DRM_ERROR("aconnector is NULL!\n"); return stream; @@ -2437,15 +2407,18 @@ create_stream_for_sink(struct amdgpu_dm_connector *aconnector, return stream; } - if (create_fake_sink(aconnector)) + sink = create_fake_sink(aconnector); + if (!sink) return stream; + } else { + sink = aconnector->dc_sink; } - stream = dc_create_stream_for_sink(aconnector->dc_sink); + stream = dc_create_stream_for_sink(sink); if (stream == NULL) { DRM_ERROR("Failed to create stream for sink!\n"); - return stream; + goto finish; } list_for_each_entry(preferred_mode, &aconnector->base.modes, head) { @@ -2484,10 +2457,16 @@ create_stream_for_sink(struct amdgpu_dm_connector *aconnector, fill_audio_info( &stream->audio_info, drm_connector, - aconnector->dc_sink); + sink); update_stream_signal(stream); + if (dm_state && dm_state->freesync_capable) + stream->ignore_msa_timing_param = true; +finish: + if (sink && sink->sink_signal == SIGNAL_TYPE_VIRTUAL) + dc_sink_release(sink); + return stream; } @@ -2710,18 +2689,15 @@ static void amdgpu_dm_connector_destroy(struct drm_connector *connector) const struct dc_link *link = aconnector->dc_link; struct amdgpu_device *adev = connector->dev->dev_private; struct amdgpu_display_manager *dm = &adev->dm; + #if defined(CONFIG_BACKLIGHT_CLASS_DEVICE) ||\ defined(CONFIG_BACKLIGHT_CLASS_DEVICE_MODULE) if ((link->connector_signal & (SIGNAL_TYPE_EDP | SIGNAL_TYPE_LVDS)) && - link->type != dc_connection_none) { - amdgpu_dm_register_backlight_device(dm); - - if (dm->backlight_dev) { - backlight_device_unregister(dm->backlight_dev); - dm->backlight_dev = NULL; - } - + link->type != dc_connection_none && + dm->backlight_dev) { + backlight_device_unregister(dm->backlight_dev); + dm->backlight_dev = NULL; } #endif drm_connector_unregister(connector); @@ -2734,6 +2710,9 @@ void amdgpu_dm_connector_funcs_reset(struct drm_connector *connector) struct dm_connector_state *state = to_dm_connector_state(connector->state); + if (connector->state) + __drm_atomic_helper_connector_destroy_state(connector->state); + kfree(state); state = kzalloc(sizeof(*state), GFP_KERNEL); @@ -2744,8 +2723,7 @@ void amdgpu_dm_connector_funcs_reset(struct drm_connector *connector) state->underscan_hborder = 0; state->underscan_vborder = 0; - connector->state = &state->base; - connector->state->connector = connector; + __drm_atomic_helper_connector_reset(connector, &state->base); } } @@ -2855,7 +2833,7 @@ static void handle_edid_mgmt(struct amdgpu_dm_connector *aconnector) create_eml_sink(aconnector); } -int amdgpu_dm_connector_mode_valid(struct drm_connector *connector, +enum drm_mode_status amdgpu_dm_connector_mode_valid(struct drm_connector *connector, struct drm_display_mode *mode) { int result = MODE_ERROR; @@ -3058,8 +3036,7 @@ static int dm_plane_helper_prepare_fb(struct drm_plane *plane, } afb = to_amdgpu_framebuffer(new_state->fb); - - obj = afb->obj; + obj = new_state->fb->obj[0]; rbo = gem_to_amdgpu_bo(obj); adev = amdgpu_ttm_adev(rbo->tbo.bdev); r = amdgpu_bo_reserve(rbo, false); @@ -3067,12 +3044,11 @@ static int dm_plane_helper_prepare_fb(struct drm_plane *plane, return r; if (plane->type != DRM_PLANE_TYPE_CURSOR) - domain = amdgpu_display_framebuffer_domains(adev); + domain = amdgpu_display_supported_domains(adev); else domain = AMDGPU_GEM_DOMAIN_VRAM; r = amdgpu_bo_pin(rbo, domain, &afb->address); - amdgpu_bo_unreserve(rbo); if (unlikely(r != 0)) { @@ -3105,17 +3081,6 @@ static int dm_plane_helper_prepare_fb(struct drm_plane *plane, } } - /* It's a hack for s3 since in 4.9 kernel filter out cursor buffer - * prepare and cleanup in drm_atomic_helper_prepare_planes - * and drm_atomic_helper_cleanup_planes because fb doens't in s3. - * IN 4.10 kernel this code should be removed and amdgpu_device_suspend - * code touching fram buffers should be avoided for DC. - */ - if (plane->type == DRM_PLANE_TYPE_CURSOR) { - struct amdgpu_crtc *acrtc = to_amdgpu_crtc(new_state->crtc); - - acrtc->cursor_bo = obj; - } return 0; } @@ -3123,14 +3088,12 @@ static void dm_plane_helper_cleanup_fb(struct drm_plane *plane, struct drm_plane_state *old_state) { struct amdgpu_bo *rbo; - struct amdgpu_framebuffer *afb; int r; if (!old_state->fb) return; - afb = to_amdgpu_framebuffer(old_state->fb); - rbo = gem_to_amdgpu_bo(afb->obj); + rbo = gem_to_amdgpu_bo(old_state->fb->obj[0]); r = amdgpu_bo_reserve(rbo, false); if (unlikely(r)) { DRM_ERROR("failed to reserve rbo before unpin\n"); @@ -3773,7 +3736,7 @@ static void remove_stream(struct amdgpu_device *adev, static int get_cursor_position(struct drm_plane *plane, struct drm_crtc *crtc, struct dc_cursor_position *position) { - struct amdgpu_crtc *amdgpu_crtc = amdgpu_crtc = to_amdgpu_crtc(crtc); + struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc); int x, y; int xorigin = 0, yorigin = 0; @@ -3905,7 +3868,7 @@ static void amdgpu_dm_do_flip(struct drm_crtc *crtc, int r, vpos, hpos; struct amdgpu_crtc *acrtc = to_amdgpu_crtc(crtc); struct amdgpu_framebuffer *afb = to_amdgpu_framebuffer(fb); - struct amdgpu_bo *abo = gem_to_amdgpu_bo(afb->obj); + struct amdgpu_bo *abo = gem_to_amdgpu_bo(fb->obj[0]); struct amdgpu_device *adev = crtc->dev->dev_private; bool async_flip = (crtc->state->pageflip_flags & DRM_MODE_PAGE_FLIP_ASYNC) != 0; struct dc_flip_addrs addr = { {0} }; @@ -3986,6 +3949,96 @@ static void amdgpu_dm_do_flip(struct drm_crtc *crtc, spin_unlock_irqrestore(&crtc->dev->event_lock, flags); } +/* + * TODO this whole function needs to go + * + * dc_surface_update is needlessly complex. See if we can just replace this + * with a dc_plane_state and follow the atomic model a bit more closely here. + */ +static bool commit_planes_to_stream( + struct dc *dc, + struct dc_plane_state **plane_states, + uint8_t new_plane_count, + struct dm_crtc_state *dm_new_crtc_state, + struct dm_crtc_state *dm_old_crtc_state, + struct dc_state *state) +{ + /* no need to dynamically allocate this. it's pretty small */ + struct dc_surface_update updates[MAX_SURFACES]; + struct dc_flip_addrs *flip_addr; + struct dc_plane_info *plane_info; + struct dc_scaling_info *scaling_info; + int i; + struct dc_stream_state *dc_stream = dm_new_crtc_state->stream; + struct dc_stream_update *stream_update = + kzalloc(sizeof(struct dc_stream_update), GFP_KERNEL); + + if (!stream_update) { + BREAK_TO_DEBUGGER(); + return false; + } + + flip_addr = kcalloc(MAX_SURFACES, sizeof(struct dc_flip_addrs), + GFP_KERNEL); + plane_info = kcalloc(MAX_SURFACES, sizeof(struct dc_plane_info), + GFP_KERNEL); + scaling_info = kcalloc(MAX_SURFACES, sizeof(struct dc_scaling_info), + GFP_KERNEL); + + if (!flip_addr || !plane_info || !scaling_info) { + kfree(flip_addr); + kfree(plane_info); + kfree(scaling_info); + kfree(stream_update); + return false; + } + + memset(updates, 0, sizeof(updates)); + + stream_update->src = dc_stream->src; + stream_update->dst = dc_stream->dst; + stream_update->out_transfer_func = dc_stream->out_transfer_func; + + for (i = 0; i < new_plane_count; i++) { + updates[i].surface = plane_states[i]; + updates[i].gamma = + (struct dc_gamma *)plane_states[i]->gamma_correction; + updates[i].in_transfer_func = plane_states[i]->in_transfer_func; + flip_addr[i].address = plane_states[i]->address; + flip_addr[i].flip_immediate = plane_states[i]->flip_immediate; + plane_info[i].color_space = plane_states[i]->color_space; + plane_info[i].format = plane_states[i]->format; + plane_info[i].plane_size = plane_states[i]->plane_size; + plane_info[i].rotation = plane_states[i]->rotation; + plane_info[i].horizontal_mirror = plane_states[i]->horizontal_mirror; + plane_info[i].stereo_format = plane_states[i]->stereo_format; + plane_info[i].tiling_info = plane_states[i]->tiling_info; + plane_info[i].visible = plane_states[i]->visible; + plane_info[i].per_pixel_alpha = plane_states[i]->per_pixel_alpha; + plane_info[i].dcc = plane_states[i]->dcc; + scaling_info[i].scaling_quality = plane_states[i]->scaling_quality; + scaling_info[i].src_rect = plane_states[i]->src_rect; + scaling_info[i].dst_rect = plane_states[i]->dst_rect; + scaling_info[i].clip_rect = plane_states[i]->clip_rect; + + updates[i].flip_addr = &flip_addr[i]; + updates[i].plane_info = &plane_info[i]; + updates[i].scaling_info = &scaling_info[i]; + } + + dc_commit_updates_for_stream( + dc, + updates, + new_plane_count, + dc_stream, stream_update, plane_states, state); + + kfree(flip_addr); + kfree(plane_info); + kfree(scaling_info); + kfree(stream_update); + return true; +} + static void amdgpu_dm_commit_planes(struct drm_atomic_state *state, struct drm_device *dev, struct amdgpu_display_manager *dm, @@ -4001,6 +4054,8 @@ static void amdgpu_dm_commit_planes(struct drm_atomic_state *state, struct drm_crtc_state *new_pcrtc_state = drm_atomic_get_new_crtc_state(state, pcrtc); struct dm_crtc_state *acrtc_state = to_dm_crtc_state(new_pcrtc_state); + struct dm_crtc_state *dm_old_crtc_state = + to_dm_crtc_state(drm_atomic_get_old_crtc_state(state, pcrtc)); struct dm_atomic_state *dm_state = to_dm_atomic_state(state); int planes_count = 0; unsigned long flags; @@ -4037,7 +4092,7 @@ static void amdgpu_dm_commit_planes(struct drm_atomic_state *state, } spin_unlock_irqrestore(&crtc->dev->event_lock, flags); - if (!pflip_needed) { + if (!pflip_needed || plane->type == DRM_PLANE_TYPE_OVERLAY) { WARN_ON(!dm_new_plane_state->dc_state); plane_states_constructed[planes_count] = dm_new_plane_state->dc_state; @@ -4079,10 +4134,12 @@ static void amdgpu_dm_commit_planes(struct drm_atomic_state *state, spin_unlock_irqrestore(&pcrtc->dev->event_lock, flags); } - if (false == dc_commit_planes_to_stream(dm->dc, + + if (false == commit_planes_to_stream(dm->dc, plane_states_constructed, planes_count, - dc_stream_attach, + acrtc_state, + dm_old_crtc_state, dm_state->context)) dm_error("%s: Failed to attach plane!\n", __func__); } else { @@ -4211,6 +4268,8 @@ static void amdgpu_dm_atomic_commit_tail(struct drm_atomic_state *state) if (dm_old_crtc_state->stream) remove_stream(adev, acrtc, dm_old_crtc_state->stream); + pm_runtime_get_noresume(dev->dev); + acrtc->enabled = true; acrtc->hw_mode = new_crtc_state->mode; crtc->hwmode = new_crtc_state->mode; @@ -4307,8 +4366,10 @@ static void amdgpu_dm_atomic_commit_tail(struct drm_atomic_state *state) struct amdgpu_crtc *acrtc = to_amdgpu_crtc(dm_new_con_state->base.crtc); struct dc_stream_status *status = NULL; - if (acrtc) + if (acrtc) { new_crtc_state = drm_atomic_get_new_crtc_state(state, &acrtc->base); + old_crtc_state = drm_atomic_get_old_crtc_state(state, &acrtc->base); + } /* Skip any modesets/resets */ if (!acrtc || drm_atomic_crtc_needs_modeset(new_crtc_state)) @@ -4331,11 +4392,12 @@ static void amdgpu_dm_atomic_commit_tail(struct drm_atomic_state *state) WARN_ON(!status->plane_count); /*TODO How it works with MPO ?*/ - if (!dc_commit_planes_to_stream( + if (!commit_planes_to_stream( dm->dc, status->plane_states, status->plane_count, - dm_new_crtc_state->stream, + dm_new_crtc_state, + to_dm_crtc_state(old_crtc_state), dm_state->context)) dm_error("%s: Failed to update stream scaling!\n", __func__); } @@ -4396,6 +4458,16 @@ static void amdgpu_dm_atomic_commit_tail(struct drm_atomic_state *state) drm_atomic_helper_wait_for_flip_done(dev, state); drm_atomic_helper_cleanup_planes(dev, state); + + /* Finally, drop a runtime PM reference for each newly disabled CRTC, + * so we can put the GPU into runtime suspend if we're not driving any + * displays anymore + */ + pm_runtime_mark_last_busy(dev->dev); + for_each_oldnew_crtc_in_state(state, crtc, old_crtc_state, new_crtc_state, i) { + if (old_crtc_state->active && !new_crtc_state->active) + pm_runtime_put_autosuspend(dev->dev); + } } @@ -4555,8 +4627,8 @@ static int dm_update_crtcs_state(struct dc *dc, for_each_oldnew_crtc_in_state(state, crtc, old_crtc_state, new_crtc_state, i) { struct amdgpu_crtc *acrtc = NULL; struct amdgpu_dm_connector *aconnector = NULL; - struct drm_connector_state *new_con_state = NULL; - struct dm_connector_state *dm_conn_state = NULL; + struct drm_connector_state *drm_new_conn_state = NULL, *drm_old_conn_state = NULL; + struct dm_connector_state *dm_new_conn_state = NULL, *dm_old_conn_state = NULL; struct drm_plane_state *new_plane_state = NULL; new_stream = NULL; @@ -4577,19 +4649,22 @@ static int dm_update_crtcs_state(struct dc *dc, /* TODO This hack should go away */ if (aconnector && enable) { // Make sure fake sink is created in plug-in scenario - new_con_state = drm_atomic_get_connector_state(state, + drm_new_conn_state = drm_atomic_get_new_connector_state(state, &aconnector->base); + drm_old_conn_state = drm_atomic_get_old_connector_state(state, + &aconnector->base); - if (IS_ERR(new_con_state)) { - ret = PTR_ERR_OR_ZERO(new_con_state); + if (IS_ERR(drm_new_conn_state)) { + ret = PTR_ERR_OR_ZERO(drm_new_conn_state); break; } - dm_conn_state = to_dm_connector_state(new_con_state); + dm_new_conn_state = to_dm_connector_state(drm_new_conn_state); + dm_old_conn_state = to_dm_connector_state(drm_old_conn_state); new_stream = create_stream_for_sink(aconnector, &new_crtc_state->mode, - dm_conn_state); + dm_new_conn_state); /* * we can have no stream on ACTION_SET if a display @@ -4695,20 +4770,30 @@ next_crtc: * We want to do dc stream updates that do not require a * full modeset below. */ - if (!enable || !aconnector || modereset_required(new_crtc_state)) + if (!(enable && aconnector && new_crtc_state->enable && + new_crtc_state->active)) continue; /* * Given above conditions, the dc state cannot be NULL because: - * 1. We're attempting to enable a CRTC. Which has a... - * 2. Valid connector attached, and - * 3. User does not want to reset it (disable or mark inactive, - * which can happen on a CRTC that's already disabled). - * => It currently exists. + * 1. We're in the process of enabling CRTCs (just been added + * to the dc context, or already is on the context) + * 2. Has a valid connector attached, and + * 3. Is currently active and enabled. + * => The dc stream state currently exists. */ BUG_ON(dm_new_crtc_state->stream == NULL); - /* Color managment settings */ - if (dm_new_crtc_state->base.color_mgmt_changed) { + /* Scaling or underscan settings */ + if (is_scaling_state_different(dm_old_conn_state, dm_new_conn_state)) + update_stream_scaling_settings( + &new_crtc_state->mode, dm_new_conn_state, dm_new_crtc_state->stream); + + /* + * Color management settings. We also update color properties + * when a modeset is needed, to ensure it gets reprogrammed. + */ + if (dm_new_crtc_state->base.color_mgmt_changed || + drm_atomic_crtc_needs_modeset(new_crtc_state)) { ret = amdgpu_dm_set_regamma_lut(dm_new_crtc_state); if (ret) goto fail; @@ -4755,7 +4840,8 @@ static int dm_update_planes_state(struct dc *dc, /* Remove any changed/removed planes */ if (!enable) { - if (pflip_needed) + if (pflip_needed && + plane->type != DRM_PLANE_TYPE_OVERLAY) continue; if (!old_plane_crtc) @@ -4802,7 +4888,8 @@ static int dm_update_planes_state(struct dc *dc, if (!dm_new_crtc_state->stream) continue; - if (pflip_needed) + if (pflip_needed && + plane->type != DRM_PLANE_TYPE_OVERLAY) continue; WARN_ON(dm_new_plane_state->dc_state); @@ -5009,17 +5096,24 @@ void amdgpu_dm_add_sink_to_freesync_module(struct drm_connector *connector, struct edid *edid) { int i; - uint64_t val_capable; bool edid_check_required; struct detailed_timing *timing; struct detailed_non_pixel *data; struct detailed_data_monitor_range *range; struct amdgpu_dm_connector *amdgpu_dm_connector = to_amdgpu_dm_connector(connector); + struct dm_connector_state *dm_con_state; struct drm_device *dev = connector->dev; struct amdgpu_device *adev = dev->dev_private; + if (!connector->state) { + DRM_ERROR("%s - Connector has no state", __func__); + return; + } + + dm_con_state = to_dm_connector_state(connector->state); + edid_check_required = false; if (!amdgpu_dm_connector->dc_sink) { DRM_ERROR("dc_sink NULL, could not add free_sync module.\n"); @@ -5038,7 +5132,7 @@ void amdgpu_dm_add_sink_to_freesync_module(struct drm_connector *connector, amdgpu_dm_connector); } } - val_capable = 0; + dm_con_state->freesync_capable = false; if (edid_check_required == true && (edid->version > 1 || (edid->version == 1 && edid->revision > 1))) { for (i = 0; i < 4; i++) { @@ -5074,7 +5168,7 @@ void amdgpu_dm_add_sink_to_freesync_module(struct drm_connector *connector, amdgpu_dm_connector->min_vfreq * 1000000; amdgpu_dm_connector->caps.max_refresh_in_micro_hz = amdgpu_dm_connector->max_vfreq * 1000000; - val_capable = 1; + dm_con_state->freesync_capable = true; } } diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h index b68400c1154b..d5aa89ad5571 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h @@ -28,7 +28,6 @@ #include #include -#include "dc.h" /* * This file contains the definition for amdgpu_display_manager @@ -53,6 +52,7 @@ struct amdgpu_device; struct drm_device; struct amdgpu_dm_irq_handler_data; +struct dc; struct amdgpu_dm_prev_state { struct drm_framebuffer *fb; @@ -220,6 +220,7 @@ struct dm_connector_state { uint8_t underscan_hborder; bool underscan_enable; struct mod_freesync_user_enable user_enable; + bool freesync_capable; }; #define to_dm_connector_state(x)\ @@ -246,7 +247,7 @@ void amdgpu_dm_connector_init_helper(struct amdgpu_display_manager *dm, struct dc_link *link, int link_index); -int amdgpu_dm_connector_mode_valid(struct drm_connector *connector, +enum drm_mode_status amdgpu_dm_connector_mode_valid(struct drm_connector *connector, struct drm_display_mode *mode); void dm_restore_drm_connector_state(struct drm_device *dev, diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_color.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_color.c index 25f064c01038..b329393307e5 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_color.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_color.c @@ -25,6 +25,7 @@ #include "amdgpu_mode.h" #include "amdgpu_dm.h" +#include "dc.h" #include "modules/color/color_gamma.h" #define MAX_DRM_LUT_VALUE 0xFFFF @@ -87,9 +88,9 @@ static void __drm_lut_to_dc_gamma(struct drm_color_lut *lut, g = drm_color_lut_extract(lut[i].green, 16); b = drm_color_lut_extract(lut[i].blue, 16); - gamma->entries.red[i] = dal_fixed31_32_from_int(r); - gamma->entries.green[i] = dal_fixed31_32_from_int(g); - gamma->entries.blue[i] = dal_fixed31_32_from_int(b); + gamma->entries.red[i] = dc_fixpt_from_int(r); + gamma->entries.green[i] = dc_fixpt_from_int(g); + gamma->entries.blue[i] = dc_fixpt_from_int(b); } return; } @@ -100,9 +101,9 @@ static void __drm_lut_to_dc_gamma(struct drm_color_lut *lut, g = drm_color_lut_extract(lut[i].green, 16); b = drm_color_lut_extract(lut[i].blue, 16); - gamma->entries.red[i] = dal_fixed31_32_from_fraction(r, MAX_DRM_LUT_VALUE); - gamma->entries.green[i] = dal_fixed31_32_from_fraction(g, MAX_DRM_LUT_VALUE); - gamma->entries.blue[i] = dal_fixed31_32_from_fraction(b, MAX_DRM_LUT_VALUE); + gamma->entries.red[i] = dc_fixpt_from_fraction(r, MAX_DRM_LUT_VALUE); + gamma->entries.green[i] = dc_fixpt_from_fraction(g, MAX_DRM_LUT_VALUE); + gamma->entries.blue[i] = dc_fixpt_from_fraction(b, MAX_DRM_LUT_VALUE); } } @@ -207,7 +208,7 @@ void amdgpu_dm_set_ctm(struct dm_crtc_state *crtc) for (i = 0; i < 12; i++) { /* Skip 4th element */ if (i % 4 == 3) { - stream->gamut_remap_matrix.matrix[i] = dal_fixed31_32_zero; + stream->gamut_remap_matrix.matrix[i] = dc_fixpt_zero; continue; } diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c index ca0b08bfa2cf..ec304b1a5973 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c @@ -330,11 +330,6 @@ bool dm_helpers_dp_mst_send_payload_allocation( return true; } -bool dm_helpers_dc_conn_log(struct dc_context *ctx, struct log_entry *entry, enum dc_log_type event) -{ - return true; -} - void dm_dtn_log_begin(struct dc_context *ctx) {} @@ -440,7 +435,7 @@ bool dm_helpers_submit_i2c( return false; } - msgs = kzalloc(num * sizeof(struct i2c_msg), GFP_KERNEL); + msgs = kcalloc(num, sizeof(struct i2c_msg), GFP_KERNEL); if (!msgs) return false; diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c index 4be21bf54749..a910f01838ab 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c @@ -555,6 +555,9 @@ static inline int dm_irq_state(struct amdgpu_device *adev, return 0; } + if (acrtc->otg_inst == -1) + return 0; + irq_source = dal_irq_type + acrtc->otg_inst; st = (state == AMDGPU_IRQ_STATE_ENABLE); diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_services.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_services.c index 89342b48be6b..5a3346124a01 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_services.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_services.c @@ -37,8 +37,17 @@ unsigned long long dm_get_timestamp(struct dc_context *ctx) { - /* TODO: return actual timestamp */ - return 0; + struct timespec64 time; + + getrawmonotonic64(&time); + return timespec64_to_ns(&time); +} + +unsigned long long dm_get_elapse_time_in_ns(struct dc_context *ctx, + unsigned long long current_time_stamp, + unsigned long long last_time_stamp) +{ + return current_time_stamp - last_time_stamp; } void dm_perf_trace_timestamp(const char *func_name, unsigned int line) @@ -225,6 +234,33 @@ static void pp_to_dc_clock_levels( } } +static void pp_to_dc_clock_levels_with_latency( + const struct pp_clock_levels_with_latency *pp_clks, + struct dm_pp_clock_levels_with_latency *clk_level_info, + enum dm_pp_clock_type dc_clk_type) +{ + uint32_t i; + + if (pp_clks->num_levels > DM_PP_MAX_CLOCK_LEVELS) { + DRM_INFO("DM_PPLIB: Warning: %s clock: number of levels %d exceeds maximum of %d!\n", + DC_DECODE_PP_CLOCK_TYPE(dc_clk_type), + pp_clks->num_levels, + DM_PP_MAX_CLOCK_LEVELS); + + clk_level_info->num_levels = DM_PP_MAX_CLOCK_LEVELS; + } else + clk_level_info->num_levels = pp_clks->num_levels; + + DRM_DEBUG("DM_PPLIB: values for %s clock\n", + DC_DECODE_PP_CLOCK_TYPE(dc_clk_type)); + + for (i = 0; i < clk_level_info->num_levels; i++) { + DRM_DEBUG("DM_PPLIB:\t %d\n", pp_clks->data[i].clocks_in_khz); + clk_level_info->data[i].clocks_in_khz = pp_clks->data[i].clocks_in_khz; + clk_level_info->data[i].latency_in_us = pp_clks->data[i].latency_in_us; + } +} + bool dm_pp_get_clock_levels_by_type( const struct dc_context *ctx, enum dm_pp_clock_type clk_type, @@ -302,8 +338,22 @@ bool dm_pp_get_clock_levels_by_type_with_latency( enum dm_pp_clock_type clk_type, struct dm_pp_clock_levels_with_latency *clk_level_info) { - /* TODO: to be implemented */ - return false; + struct amdgpu_device *adev = ctx->driver_context; + void *pp_handle = adev->powerplay.pp_handle; + struct pp_clock_levels_with_latency pp_clks = { 0 }; + const struct amd_pm_funcs *pp_funcs = adev->powerplay.pp_funcs; + + if (!pp_funcs || !pp_funcs->get_clock_by_type_with_latency) + return false; + + if (pp_funcs->get_clock_by_type_with_latency(pp_handle, + dc_to_pp_clock_type(clk_type), + &pp_clks)) + return false; + + pp_to_dc_clock_levels_with_latency(&pp_clks, clk_level_info, clk_type); + + return true; } bool dm_pp_get_clock_levels_by_type_with_voltage( diff --git a/drivers/gpu/drm/amd/display/dc/basics/Makefile b/drivers/gpu/drm/amd/display/dc/basics/Makefile index bca33bd9a0d2..b49ea96b5dae 100644 --- a/drivers/gpu/drm/amd/display/dc/basics/Makefile +++ b/drivers/gpu/drm/amd/display/dc/basics/Makefile @@ -24,7 +24,7 @@ # It provides the general basic services required by other DAL # subcomponents. -BASICS = conversion.o fixpt31_32.o fixpt32_32.o \ +BASICS = conversion.o fixpt31_32.o \ logger.o log_helpers.o vector.o AMD_DAL_BASICS = $(addprefix $(AMDDALPATH)/dc/basics/,$(BASICS)) diff --git a/drivers/gpu/drm/amd/display/dc/basics/conversion.c b/drivers/gpu/drm/amd/display/dc/basics/conversion.c index 310964915a83..50b47f11875c 100644 --- a/drivers/gpu/drm/amd/display/dc/basics/conversion.c +++ b/drivers/gpu/drm/amd/display/dc/basics/conversion.c @@ -41,22 +41,22 @@ uint16_t fixed_point_to_int_frac( uint16_t result; - uint16_t d = (uint16_t)dal_fixed31_32_floor( - dal_fixed31_32_abs( + uint16_t d = (uint16_t)dc_fixpt_floor( + dc_fixpt_abs( arg)); if (d <= (uint16_t)(1 << integer_bits) - (1 / (uint16_t)divisor)) - numerator = (uint16_t)dal_fixed31_32_round( - dal_fixed31_32_mul_int( + numerator = (uint16_t)dc_fixpt_round( + dc_fixpt_mul_int( arg, divisor)); else { - numerator = dal_fixed31_32_floor( - dal_fixed31_32_sub( - dal_fixed31_32_from_int( + numerator = dc_fixpt_floor( + dc_fixpt_sub( + dc_fixpt_from_int( 1LL << integer_bits), - dal_fixed31_32_recip( - dal_fixed31_32_from_int( + dc_fixpt_recip( + dc_fixpt_from_int( divisor)))); } @@ -66,8 +66,8 @@ uint16_t fixed_point_to_int_frac( result = (uint16_t)( (1 << (integer_bits + fractional_bits + 1)) + numerator); - if ((result != 0) && dal_fixed31_32_lt( - arg, dal_fixed31_32_zero)) + if ((result != 0) && dc_fixpt_lt( + arg, dc_fixpt_zero)) result |= 1 << (integer_bits + fractional_bits); return result; @@ -84,15 +84,15 @@ void convert_float_matrix( uint32_t buffer_size) { const struct fixed31_32 min_2_13 = - dal_fixed31_32_from_fraction(S2D13_MIN, DIVIDER); + dc_fixpt_from_fraction(S2D13_MIN, DIVIDER); const struct fixed31_32 max_2_13 = - dal_fixed31_32_from_fraction(S2D13_MAX, DIVIDER); + dc_fixpt_from_fraction(S2D13_MAX, DIVIDER); uint32_t i; for (i = 0; i < buffer_size; ++i) { uint32_t reg_value = fixed_point_to_int_frac( - dal_fixed31_32_clamp( + dc_fixpt_clamp( flt[i], min_2_13, max_2_13), diff --git a/drivers/gpu/drm/amd/display/dc/basics/fixpt31_32.c b/drivers/gpu/drm/amd/display/dc/basics/fixpt31_32.c index 8a9bba879207..f28989860fd8 100644 --- a/drivers/gpu/drm/amd/display/dc/basics/fixpt31_32.c +++ b/drivers/gpu/drm/amd/display/dc/basics/fixpt31_32.c @@ -26,13 +26,13 @@ #include "dm_services.h" #include "include/fixed31_32.h" -static inline uint64_t abs_i64( - int64_t arg) +static inline unsigned long long abs_i64( + long long arg) { if (arg > 0) - return (uint64_t)arg; + return (unsigned long long)arg; else - return (uint64_t)(-arg); + return (unsigned long long)(-arg); } /* @@ -40,12 +40,12 @@ static inline uint64_t abs_i64( * result = dividend / divisor * *remainder = dividend % divisor */ -static inline uint64_t complete_integer_division_u64( - uint64_t dividend, - uint64_t divisor, - uint64_t *remainder) +static inline unsigned long long complete_integer_division_u64( + unsigned long long dividend, + unsigned long long divisor, + unsigned long long *remainder) { - uint64_t result; + unsigned long long result; ASSERT(divisor); @@ -64,30 +64,28 @@ static inline uint64_t complete_integer_division_u64( #define GET_FRACTIONAL_PART(x) \ (FRACTIONAL_PART_MASK & (x)) -struct fixed31_32 dal_fixed31_32_from_fraction( - int64_t numerator, - int64_t denominator) +struct fixed31_32 dc_fixpt_from_fraction(long long numerator, long long denominator) { struct fixed31_32 res; bool arg1_negative = numerator < 0; bool arg2_negative = denominator < 0; - uint64_t arg1_value = arg1_negative ? -numerator : numerator; - uint64_t arg2_value = arg2_negative ? -denominator : denominator; + unsigned long long arg1_value = arg1_negative ? -numerator : numerator; + unsigned long long arg2_value = arg2_negative ? -denominator : denominator; - uint64_t remainder; + unsigned long long remainder; /* determine integer part */ - uint64_t res_value = complete_integer_division_u64( + unsigned long long res_value = complete_integer_division_u64( arg1_value, arg2_value, &remainder); ASSERT(res_value <= LONG_MAX); /* determine fractional part */ { - uint32_t i = FIXED31_32_BITS_PER_FRACTIONAL_PART; + unsigned int i = FIXED31_32_BITS_PER_FRACTIONAL_PART; do { remainder <<= 1; @@ -103,14 +101,14 @@ struct fixed31_32 dal_fixed31_32_from_fraction( /* round up LSB */ { - uint64_t summand = (remainder << 1) >= arg2_value; + unsigned long long summand = (remainder << 1) >= arg2_value; ASSERT(res_value <= LLONG_MAX - summand); res_value += summand; } - res.value = (int64_t)res_value; + res.value = (long long)res_value; if (arg1_negative ^ arg2_negative) res.value = -res.value; @@ -118,79 +116,23 @@ struct fixed31_32 dal_fixed31_32_from_fraction( return res; } -struct fixed31_32 dal_fixed31_32_from_int_nonconst( - int64_t arg) -{ - struct fixed31_32 res; - - ASSERT((LONG_MIN <= arg) && (arg <= LONG_MAX)); - - res.value = arg << FIXED31_32_BITS_PER_FRACTIONAL_PART; - - return res; -} - -struct fixed31_32 dal_fixed31_32_shl( - struct fixed31_32 arg, - uint8_t shift) -{ - struct fixed31_32 res; - - ASSERT(((arg.value >= 0) && (arg.value <= LLONG_MAX >> shift)) || - ((arg.value < 0) && (arg.value >= LLONG_MIN >> shift))); - - res.value = arg.value << shift; - - return res; -} - -struct fixed31_32 dal_fixed31_32_add( - struct fixed31_32 arg1, - struct fixed31_32 arg2) -{ - struct fixed31_32 res; - - ASSERT(((arg1.value >= 0) && (LLONG_MAX - arg1.value >= arg2.value)) || - ((arg1.value < 0) && (LLONG_MIN - arg1.value <= arg2.value))); - - res.value = arg1.value + arg2.value; - - return res; -} - -struct fixed31_32 dal_fixed31_32_sub( - struct fixed31_32 arg1, - struct fixed31_32 arg2) -{ - struct fixed31_32 res; - - ASSERT(((arg2.value >= 0) && (LLONG_MIN + arg2.value <= arg1.value)) || - ((arg2.value < 0) && (LLONG_MAX + arg2.value >= arg1.value))); - - res.value = arg1.value - arg2.value; - - return res; -} - -struct fixed31_32 dal_fixed31_32_mul( - struct fixed31_32 arg1, - struct fixed31_32 arg2) +struct fixed31_32 dc_fixpt_mul(struct fixed31_32 arg1, struct fixed31_32 arg2) { struct fixed31_32 res; bool arg1_negative = arg1.value < 0; bool arg2_negative = arg2.value < 0; - uint64_t arg1_value = arg1_negative ? -arg1.value : arg1.value; - uint64_t arg2_value = arg2_negative ? -arg2.value : arg2.value; + unsigned long long arg1_value = arg1_negative ? -arg1.value : arg1.value; + unsigned long long arg2_value = arg2_negative ? -arg2.value : arg2.value; - uint64_t arg1_int = GET_INTEGER_PART(arg1_value); - uint64_t arg2_int = GET_INTEGER_PART(arg2_value); + unsigned long long arg1_int = GET_INTEGER_PART(arg1_value); + unsigned long long arg2_int = GET_INTEGER_PART(arg2_value); - uint64_t arg1_fra = GET_FRACTIONAL_PART(arg1_value); - uint64_t arg2_fra = GET_FRACTIONAL_PART(arg2_value); + unsigned long long arg1_fra = GET_FRACTIONAL_PART(arg1_value); + unsigned long long arg2_fra = GET_FRACTIONAL_PART(arg2_value); - uint64_t tmp; + unsigned long long tmp; res.value = arg1_int * arg2_int; @@ -200,22 +142,22 @@ struct fixed31_32 dal_fixed31_32_mul( tmp = arg1_int * arg2_fra; - ASSERT(tmp <= (uint64_t)(LLONG_MAX - res.value)); + ASSERT(tmp <= (unsigned long long)(LLONG_MAX - res.value)); res.value += tmp; tmp = arg2_int * arg1_fra; - ASSERT(tmp <= (uint64_t)(LLONG_MAX - res.value)); + ASSERT(tmp <= (unsigned long long)(LLONG_MAX - res.value)); res.value += tmp; tmp = arg1_fra * arg2_fra; tmp = (tmp >> FIXED31_32_BITS_PER_FRACTIONAL_PART) + - (tmp >= (uint64_t)dal_fixed31_32_half.value); + (tmp >= (unsigned long long)dc_fixpt_half.value); - ASSERT(tmp <= (uint64_t)(LLONG_MAX - res.value)); + ASSERT(tmp <= (unsigned long long)(LLONG_MAX - res.value)); res.value += tmp; @@ -225,18 +167,17 @@ struct fixed31_32 dal_fixed31_32_mul( return res; } -struct fixed31_32 dal_fixed31_32_sqr( - struct fixed31_32 arg) +struct fixed31_32 dc_fixpt_sqr(struct fixed31_32 arg) { struct fixed31_32 res; - uint64_t arg_value = abs_i64(arg.value); + unsigned long long arg_value = abs_i64(arg.value); - uint64_t arg_int = GET_INTEGER_PART(arg_value); + unsigned long long arg_int = GET_INTEGER_PART(arg_value); - uint64_t arg_fra = GET_FRACTIONAL_PART(arg_value); + unsigned long long arg_fra = GET_FRACTIONAL_PART(arg_value); - uint64_t tmp; + unsigned long long tmp; res.value = arg_int * arg_int; @@ -246,28 +187,27 @@ struct fixed31_32 dal_fixed31_32_sqr( tmp = arg_int * arg_fra; - ASSERT(tmp <= (uint64_t)(LLONG_MAX - res.value)); + ASSERT(tmp <= (unsigned long long)(LLONG_MAX - res.value)); res.value += tmp; - ASSERT(tmp <= (uint64_t)(LLONG_MAX - res.value)); + ASSERT(tmp <= (unsigned long long)(LLONG_MAX - res.value)); res.value += tmp; tmp = arg_fra * arg_fra; tmp = (tmp >> FIXED31_32_BITS_PER_FRACTIONAL_PART) + - (tmp >= (uint64_t)dal_fixed31_32_half.value); + (tmp >= (unsigned long long)dc_fixpt_half.value); - ASSERT(tmp <= (uint64_t)(LLONG_MAX - res.value)); + ASSERT(tmp <= (unsigned long long)(LLONG_MAX - res.value)); res.value += tmp; return res; } -struct fixed31_32 dal_fixed31_32_recip( - struct fixed31_32 arg) +struct fixed31_32 dc_fixpt_recip(struct fixed31_32 arg) { /* * @note @@ -276,41 +216,40 @@ struct fixed31_32 dal_fixed31_32_recip( ASSERT(arg.value); - return dal_fixed31_32_from_fraction( - dal_fixed31_32_one.value, + return dc_fixpt_from_fraction( + dc_fixpt_one.value, arg.value); } -struct fixed31_32 dal_fixed31_32_sinc( - struct fixed31_32 arg) +struct fixed31_32 dc_fixpt_sinc(struct fixed31_32 arg) { struct fixed31_32 square; - struct fixed31_32 res = dal_fixed31_32_one; + struct fixed31_32 res = dc_fixpt_one; - int32_t n = 27; + int n = 27; struct fixed31_32 arg_norm = arg; - if (dal_fixed31_32_le( - dal_fixed31_32_two_pi, - dal_fixed31_32_abs(arg))) { - arg_norm = dal_fixed31_32_sub( + if (dc_fixpt_le( + dc_fixpt_two_pi, + dc_fixpt_abs(arg))) { + arg_norm = dc_fixpt_sub( arg_norm, - dal_fixed31_32_mul_int( - dal_fixed31_32_two_pi, - (int32_t)div64_s64( + dc_fixpt_mul_int( + dc_fixpt_two_pi, + (int)div64_s64( arg_norm.value, - dal_fixed31_32_two_pi.value))); + dc_fixpt_two_pi.value))); } - square = dal_fixed31_32_sqr(arg_norm); + square = dc_fixpt_sqr(arg_norm); do { - res = dal_fixed31_32_sub( - dal_fixed31_32_one, - dal_fixed31_32_div_int( - dal_fixed31_32_mul( + res = dc_fixpt_sub( + dc_fixpt_one, + dc_fixpt_div_int( + dc_fixpt_mul( square, res), n * (n - 1))); @@ -319,37 +258,35 @@ struct fixed31_32 dal_fixed31_32_sinc( } while (n > 2); if (arg.value != arg_norm.value) - res = dal_fixed31_32_div( - dal_fixed31_32_mul(res, arg_norm), + res = dc_fixpt_div( + dc_fixpt_mul(res, arg_norm), arg); return res; } -struct fixed31_32 dal_fixed31_32_sin( - struct fixed31_32 arg) +struct fixed31_32 dc_fixpt_sin(struct fixed31_32 arg) { - return dal_fixed31_32_mul( + return dc_fixpt_mul( arg, - dal_fixed31_32_sinc(arg)); + dc_fixpt_sinc(arg)); } -struct fixed31_32 dal_fixed31_32_cos( - struct fixed31_32 arg) +struct fixed31_32 dc_fixpt_cos(struct fixed31_32 arg) { /* TODO implement argument normalization */ - const struct fixed31_32 square = dal_fixed31_32_sqr(arg); + const struct fixed31_32 square = dc_fixpt_sqr(arg); - struct fixed31_32 res = dal_fixed31_32_one; + struct fixed31_32 res = dc_fixpt_one; - int32_t n = 26; + int n = 26; do { - res = dal_fixed31_32_sub( - dal_fixed31_32_one, - dal_fixed31_32_div_int( - dal_fixed31_32_mul( + res = dc_fixpt_sub( + dc_fixpt_one, + dc_fixpt_div_int( + dc_fixpt_mul( square, res), n * (n - 1))); @@ -367,37 +304,35 @@ struct fixed31_32 dal_fixed31_32_cos( * * Calculated as Taylor series. */ -static struct fixed31_32 fixed31_32_exp_from_taylor_series( - struct fixed31_32 arg) +static struct fixed31_32 fixed31_32_exp_from_taylor_series(struct fixed31_32 arg) { - uint32_t n = 9; + unsigned int n = 9; - struct fixed31_32 res = dal_fixed31_32_from_fraction( + struct fixed31_32 res = dc_fixpt_from_fraction( n + 2, n + 1); /* TODO find correct res */ - ASSERT(dal_fixed31_32_lt(arg, dal_fixed31_32_one)); + ASSERT(dc_fixpt_lt(arg, dc_fixpt_one)); do - res = dal_fixed31_32_add( - dal_fixed31_32_one, - dal_fixed31_32_div_int( - dal_fixed31_32_mul( + res = dc_fixpt_add( + dc_fixpt_one, + dc_fixpt_div_int( + dc_fixpt_mul( arg, res), n)); while (--n != 1); - return dal_fixed31_32_add( - dal_fixed31_32_one, - dal_fixed31_32_mul( + return dc_fixpt_add( + dc_fixpt_one, + dc_fixpt_mul( arg, res)); } -struct fixed31_32 dal_fixed31_32_exp( - struct fixed31_32 arg) +struct fixed31_32 dc_fixpt_exp(struct fixed31_32 arg) { /* * @brief @@ -406,44 +341,43 @@ struct fixed31_32 dal_fixed31_32_exp( * where m = round(x / ln(2)), r = x - m * ln(2) */ - if (dal_fixed31_32_le( - dal_fixed31_32_ln2_div_2, - dal_fixed31_32_abs(arg))) { - int32_t m = dal_fixed31_32_round( - dal_fixed31_32_div( + if (dc_fixpt_le( + dc_fixpt_ln2_div_2, + dc_fixpt_abs(arg))) { + int m = dc_fixpt_round( + dc_fixpt_div( arg, - dal_fixed31_32_ln2)); + dc_fixpt_ln2)); - struct fixed31_32 r = dal_fixed31_32_sub( + struct fixed31_32 r = dc_fixpt_sub( arg, - dal_fixed31_32_mul_int( - dal_fixed31_32_ln2, + dc_fixpt_mul_int( + dc_fixpt_ln2, m)); ASSERT(m != 0); - ASSERT(dal_fixed31_32_lt( - dal_fixed31_32_abs(r), - dal_fixed31_32_one)); + ASSERT(dc_fixpt_lt( + dc_fixpt_abs(r), + dc_fixpt_one)); if (m > 0) - return dal_fixed31_32_shl( + return dc_fixpt_shl( fixed31_32_exp_from_taylor_series(r), - (uint8_t)m); + (unsigned char)m); else - return dal_fixed31_32_div_int( + return dc_fixpt_div_int( fixed31_32_exp_from_taylor_series(r), 1LL << -m); } else if (arg.value != 0) return fixed31_32_exp_from_taylor_series(arg); else - return dal_fixed31_32_one; + return dc_fixpt_one; } -struct fixed31_32 dal_fixed31_32_log( - struct fixed31_32 arg) +struct fixed31_32 dc_fixpt_log(struct fixed31_32 arg) { - struct fixed31_32 res = dal_fixed31_32_neg(dal_fixed31_32_one); + struct fixed31_32 res = dc_fixpt_neg(dc_fixpt_one); /* TODO improve 1st estimation */ struct fixed31_32 error; @@ -453,15 +387,15 @@ struct fixed31_32 dal_fixed31_32_log( /* TODO if arg is zero, return -INF */ do { - struct fixed31_32 res1 = dal_fixed31_32_add( - dal_fixed31_32_sub( + struct fixed31_32 res1 = dc_fixpt_add( + dc_fixpt_sub( res, - dal_fixed31_32_one), - dal_fixed31_32_div( + dc_fixpt_one), + dc_fixpt_div( arg, - dal_fixed31_32_exp(res))); + dc_fixpt_exp(res))); - error = dal_fixed31_32_sub( + error = dc_fixpt_sub( res, res1); @@ -472,78 +406,23 @@ struct fixed31_32 dal_fixed31_32_log( return res; } -struct fixed31_32 dal_fixed31_32_pow( - struct fixed31_32 arg1, - struct fixed31_32 arg2) -{ - return dal_fixed31_32_exp( - dal_fixed31_32_mul( - dal_fixed31_32_log(arg1), - arg2)); -} - -int32_t dal_fixed31_32_floor( - struct fixed31_32 arg) -{ - uint64_t arg_value = abs_i64(arg.value); - - if (arg.value >= 0) - return (int32_t)GET_INTEGER_PART(arg_value); - else - return -(int32_t)GET_INTEGER_PART(arg_value); -} - -int32_t dal_fixed31_32_round( - struct fixed31_32 arg) -{ - uint64_t arg_value = abs_i64(arg.value); - - const int64_t summand = dal_fixed31_32_half.value; - - ASSERT(LLONG_MAX - (int64_t)arg_value >= summand); - - arg_value += summand; - - if (arg.value >= 0) - return (int32_t)GET_INTEGER_PART(arg_value); - else - return -(int32_t)GET_INTEGER_PART(arg_value); -} - -int32_t dal_fixed31_32_ceil( - struct fixed31_32 arg) -{ - uint64_t arg_value = abs_i64(arg.value); - - const int64_t summand = dal_fixed31_32_one.value - - dal_fixed31_32_epsilon.value; - - ASSERT(LLONG_MAX - (int64_t)arg_value >= summand); - - arg_value += summand; - - if (arg.value >= 0) - return (int32_t)GET_INTEGER_PART(arg_value); - else - return -(int32_t)GET_INTEGER_PART(arg_value); -} /* this function is a generic helper to translate fixed point value to * specified integer format that will consist of integer_bits integer part and * fractional_bits fractional part. For example it is used in - * dal_fixed31_32_u2d19 to receive 2 bits integer part and 19 bits fractional + * dc_fixpt_u2d19 to receive 2 bits integer part and 19 bits fractional * part in 32 bits. It is used in hw programming (scaler) */ -static inline uint32_t ux_dy( - int64_t value, - uint32_t integer_bits, - uint32_t fractional_bits) +static inline unsigned int ux_dy( + long long value, + unsigned int integer_bits, + unsigned int fractional_bits) { /* 1. create mask of integer part */ - uint32_t result = (1 << integer_bits) - 1; + unsigned int result = (1 << integer_bits) - 1; /* 2. mask out fractional part */ - uint32_t fractional_part = FRACTIONAL_PART_MASK & value; + unsigned int fractional_part = FRACTIONAL_PART_MASK & value; /* 3. shrink fixed point integer part to be of integer_bits width*/ result &= GET_INTEGER_PART(value); /* 4. make space for fractional part to be filled in after integer */ @@ -554,13 +433,13 @@ static inline uint32_t ux_dy( return result | fractional_part; } -static inline uint32_t clamp_ux_dy( - int64_t value, - uint32_t integer_bits, - uint32_t fractional_bits, - uint32_t min_clamp) +static inline unsigned int clamp_ux_dy( + long long value, + unsigned int integer_bits, + unsigned int fractional_bits, + unsigned int min_clamp) { - uint32_t truncated_val = ux_dy(value, integer_bits, fractional_bits); + unsigned int truncated_val = ux_dy(value, integer_bits, fractional_bits); if (value >= (1LL << (integer_bits + FIXED31_32_BITS_PER_FRACTIONAL_PART))) return (1 << (integer_bits + fractional_bits)) - 1; @@ -570,35 +449,35 @@ static inline uint32_t clamp_ux_dy( return min_clamp; } -uint32_t dal_fixed31_32_u2d19( - struct fixed31_32 arg) +unsigned int dc_fixpt_u3d19(struct fixed31_32 arg) +{ + return ux_dy(arg.value, 3, 19); +} + +unsigned int dc_fixpt_u2d19(struct fixed31_32 arg) { return ux_dy(arg.value, 2, 19); } -uint32_t dal_fixed31_32_u0d19( - struct fixed31_32 arg) +unsigned int dc_fixpt_u0d19(struct fixed31_32 arg) { return ux_dy(arg.value, 0, 19); } -uint32_t dal_fixed31_32_clamp_u0d14( - struct fixed31_32 arg) +unsigned int dc_fixpt_clamp_u0d14(struct fixed31_32 arg) { return clamp_ux_dy(arg.value, 0, 14, 1); } -uint32_t dal_fixed31_32_clamp_u0d10( - struct fixed31_32 arg) +unsigned int dc_fixpt_clamp_u0d10(struct fixed31_32 arg) { return clamp_ux_dy(arg.value, 0, 10, 1); } -int32_t dal_fixed31_32_s4d19( - struct fixed31_32 arg) +int dc_fixpt_s4d19(struct fixed31_32 arg) { if (arg.value < 0) - return -(int32_t)ux_dy(dal_fixed31_32_abs(arg).value, 4, 19); + return -(int)ux_dy(dc_fixpt_abs(arg).value, 4, 19); else return ux_dy(arg.value, 4, 19); } diff --git a/drivers/gpu/drm/amd/display/dc/basics/fixpt32_32.c b/drivers/gpu/drm/amd/display/dc/basics/fixpt32_32.c deleted file mode 100644 index 4d3aaa82a07b..000000000000 --- a/drivers/gpu/drm/amd/display/dc/basics/fixpt32_32.c +++ /dev/null @@ -1,161 +0,0 @@ -/* - * Copyright 2012-15 Advanced Micro Devices, Inc. - * - * Permission is hereby granted, free of charge, to any person obtaining a - * copy of this software and associated documentation files (the "Software"), - * to deal in the Software without restriction, including without limitation - * the rights to use, copy, modify, merge, publish, distribute, sublicense, - * and/or sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following conditions: - * - * The above copyright notice and this permission notice shall be included in - * all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, - * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL - * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR - * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, - * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - * - * Authors: AMD - * - */ - -#include "dm_services.h" -#include "include/fixed32_32.h" - -static uint64_t u64_div(uint64_t n, uint64_t d) -{ - uint32_t i = 0; - uint64_t r; - uint64_t q = div64_u64_rem(n, d, &r); - - for (i = 0; i < 32; ++i) { - uint64_t sbit = q & (1ULL<<63); - - r <<= 1; - r |= sbit ? 1 : 0; - q <<= 1; - if (r >= d) { - r -= d; - q |= 1; - } - } - - if (2*r >= d) - q += 1; - return q; -} - -struct fixed32_32 dal_fixed32_32_from_fraction(uint32_t n, uint32_t d) -{ - struct fixed32_32 fx; - - fx.value = u64_div((uint64_t)n << 32, (uint64_t)d << 32); - return fx; -} - -struct fixed32_32 dal_fixed32_32_add( - struct fixed32_32 lhs, - struct fixed32_32 rhs) -{ - struct fixed32_32 fx = {lhs.value + rhs.value}; - - ASSERT(fx.value >= rhs.value); - return fx; -} - -struct fixed32_32 dal_fixed32_32_add_int(struct fixed32_32 lhs, uint32_t rhs) -{ - struct fixed32_32 fx = {lhs.value + ((uint64_t)rhs << 32)}; - - ASSERT(fx.value >= (uint64_t)rhs << 32); - return fx; - -} -struct fixed32_32 dal_fixed32_32_sub( - struct fixed32_32 lhs, - struct fixed32_32 rhs) -{ - struct fixed32_32 fx; - - ASSERT(lhs.value >= rhs.value); - fx.value = lhs.value - rhs.value; - return fx; -} - -struct fixed32_32 dal_fixed32_32_sub_int(struct fixed32_32 lhs, uint32_t rhs) -{ - struct fixed32_32 fx; - - ASSERT(lhs.value >= ((uint64_t)rhs<<32)); - fx.value = lhs.value - ((uint64_t)rhs<<32); - return fx; -} - -struct fixed32_32 dal_fixed32_32_mul( - struct fixed32_32 lhs, - struct fixed32_32 rhs) -{ - struct fixed32_32 fx; - uint64_t lhs_int = lhs.value>>32; - uint64_t lhs_frac = (uint32_t)lhs.value; - uint64_t rhs_int = rhs.value>>32; - uint64_t rhs_frac = (uint32_t)rhs.value; - uint64_t ahbh = lhs_int * rhs_int; - uint64_t ahbl = lhs_int * rhs_frac; - uint64_t albh = lhs_frac * rhs_int; - uint64_t albl = lhs_frac * rhs_frac; - - ASSERT((ahbh>>32) == 0); - - fx.value = (ahbh<<32) + ahbl + albh + (albl>>32); - return fx; - -} - -struct fixed32_32 dal_fixed32_32_mul_int(struct fixed32_32 lhs, uint32_t rhs) -{ - struct fixed32_32 fx; - uint64_t lhsi = (lhs.value>>32) * (uint64_t)rhs; - uint64_t lhsf; - - ASSERT((lhsi>>32) == 0); - lhsf = ((uint32_t)lhs.value) * (uint64_t)rhs; - ASSERT((lhsi<<32) + lhsf >= lhsf); - fx.value = (lhsi<<32) + lhsf; - return fx; -} - -struct fixed32_32 dal_fixed32_32_div( - struct fixed32_32 lhs, - struct fixed32_32 rhs) -{ - struct fixed32_32 fx; - - fx.value = u64_div(lhs.value, rhs.value); - return fx; -} - -struct fixed32_32 dal_fixed32_32_div_int(struct fixed32_32 lhs, uint32_t rhs) -{ - struct fixed32_32 fx; - - fx.value = u64_div(lhs.value, (uint64_t)rhs << 32); - return fx; -} - -uint32_t dal_fixed32_32_ceil(struct fixed32_32 v) -{ - ASSERT((uint32_t)v.value ? (v.value >> 32) + 1 >= 1 : true); - return (v.value>>32) + ((uint32_t)v.value ? 1 : 0); -} - -uint32_t dal_fixed32_32_round(struct fixed32_32 v) -{ - ASSERT(v.value + (1ULL<<31) >= (1ULL<<31)); - return (v.value + (1ULL<<31))>>32; -} - diff --git a/drivers/gpu/drm/amd/display/dc/basics/log_helpers.c b/drivers/gpu/drm/amd/display/dc/basics/log_helpers.c index 854678a0c54b..021451549ff7 100644 --- a/drivers/gpu/drm/amd/display/dc/basics/log_helpers.c +++ b/drivers/gpu/drm/amd/display/dc/basics/log_helpers.c @@ -94,7 +94,6 @@ void dc_conn_log(struct dc_context *ctx, dm_logger_append(&entry, "%2.2X ", hex_data[i]); dm_logger_append(&entry, "^\n"); - dm_helpers_dc_conn_log(ctx, &entry, event); fail: dm_logger_close(&entry); diff --git a/drivers/gpu/drm/amd/display/dc/basics/logger.c b/drivers/gpu/drm/amd/display/dc/basics/logger.c index 31bee054f43a..0866874ae8c6 100644 --- a/drivers/gpu/drm/amd/display/dc/basics/logger.c +++ b/drivers/gpu/drm/amd/display/dc/basics/logger.c @@ -61,7 +61,7 @@ static const struct dc_log_type_info log_type_info_tbl[] = { {LOG_EVENT_UNDERFLOW, "Underflow"}, {LOG_IF_TRACE, "InterfaceTrace"}, {LOG_DTN, "DTN"}, - {LOG_PROFILING, "Profiling"} + {LOG_DISPLAYSTATS, "DisplayStats"} }; @@ -364,7 +364,7 @@ void dm_logger_open( entry->type = log_type; entry->logger = logger; - entry->buf = kzalloc(DAL_LOGGER_BUFFER_MAX_SIZE * sizeof(char), + entry->buf = kzalloc(DAL_LOGGER_BUFFER_MAX_SIZE, GFP_KERNEL); entry->buf_offset = 0; @@ -402,3 +402,4 @@ cleanup: entry->max_buf_bytes = 0; } } + diff --git a/drivers/gpu/drm/amd/display/dc/basics/vector.c b/drivers/gpu/drm/amd/display/dc/basics/vector.c index 217b8f1f7bf6..d28e9cf0e961 100644 --- a/drivers/gpu/drm/amd/display/dc/basics/vector.c +++ b/drivers/gpu/drm/amd/display/dc/basics/vector.c @@ -40,7 +40,7 @@ bool dal_vector_construct( return false; } - vector->container = kzalloc(struct_size * capacity, GFP_KERNEL); + vector->container = kcalloc(capacity, struct_size, GFP_KERNEL); if (vector->container == NULL) return false; vector->capacity = capacity; @@ -67,7 +67,7 @@ bool dal_vector_presized_costruct( return false; } - vector->container = kzalloc(struct_size * count, GFP_KERNEL); + vector->container = kcalloc(count, struct_size, GFP_KERNEL); if (vector->container == NULL) return false; diff --git a/drivers/gpu/drm/amd/display/dc/bios/bios_parser2.c b/drivers/gpu/drm/amd/display/dc/bios/bios_parser2.c index 10a5807a7e8b..b8cef7af3c4a 100644 --- a/drivers/gpu/drm/amd/display/dc/bios/bios_parser2.c +++ b/drivers/gpu/drm/amd/display/dc/bios/bios_parser2.c @@ -1330,6 +1330,9 @@ static enum bp_result bios_parser_get_firmware_info( case 2: result = get_firmware_info_v3_2(bp, info); break; + case 3: + result = get_firmware_info_v3_2(bp, info); + break; default: break; } diff --git a/drivers/gpu/drm/amd/display/dc/bios/command_table.c b/drivers/gpu/drm/amd/display/dc/bios/command_table.c index 4b5fdd577848..651e1fd4622f 100644 --- a/drivers/gpu/drm/amd/display/dc/bios/command_table.c +++ b/drivers/gpu/drm/amd/display/dc/bios/command_table.c @@ -24,7 +24,7 @@ */ #include "dm_services.h" - +#include "amdgpu.h" #include "atom.h" #include "include/bios_parser_interface.h" @@ -35,16 +35,16 @@ #include "bios_parser_types_internal.h" #define EXEC_BIOS_CMD_TABLE(command, params)\ - (cgs_atom_exec_cmd_table(bp->base.ctx->cgs_device, \ + (amdgpu_atom_execute_table(((struct amdgpu_device *)bp->base.ctx->driver_context)->mode_info.atom_context, \ GetIndexIntoMasterTable(COMMAND, command), \ - ¶ms) == 0) + (uint32_t *)¶ms) == 0) #define BIOS_CMD_TABLE_REVISION(command, frev, crev)\ - cgs_atom_get_cmd_table_revs(bp->base.ctx->cgs_device, \ + amdgpu_atom_parse_cmd_header(((struct amdgpu_device *)bp->base.ctx->driver_context)->mode_info.atom_context, \ GetIndexIntoMasterTable(COMMAND, command), &frev, &crev) #define BIOS_CMD_TABLE_PARA_REVISION(command)\ - bios_cmd_table_para_revision(bp->base.ctx->cgs_device, \ + bios_cmd_table_para_revision(bp->base.ctx->driver_context, \ GetIndexIntoMasterTable(COMMAND, command)) static void init_dig_encoder_control(struct bios_parser *bp); @@ -82,16 +82,18 @@ void dal_bios_parser_init_cmd_tbl(struct bios_parser *bp) init_set_dce_clock(bp); } -static uint32_t bios_cmd_table_para_revision(void *cgs_device, +static uint32_t bios_cmd_table_para_revision(void *dev, uint32_t index) { + struct amdgpu_device *adev = dev; uint8_t frev, crev; - if (cgs_atom_get_cmd_table_revs(cgs_device, + if (amdgpu_atom_parse_cmd_header(adev->mode_info.atom_context, index, - &frev, &crev) != 0) + &frev, &crev)) + return crev; + else return 0; - return crev; } /******************************************************************************* @@ -368,7 +370,7 @@ static void init_transmitter_control(struct bios_parser *bp) uint8_t crev; if (BIOS_CMD_TABLE_REVISION(UNIPHYTransmitterControl, - frev, crev) != 0) + frev, crev) == false) BREAK_TO_DEBUGGER(); switch (crev) { case 2: diff --git a/drivers/gpu/drm/amd/display/dc/bios/command_table2.c b/drivers/gpu/drm/amd/display/dc/bios/command_table2.c index 3f63f712c8a4..752b08a42d3e 100644 --- a/drivers/gpu/drm/amd/display/dc/bios/command_table2.c +++ b/drivers/gpu/drm/amd/display/dc/bios/command_table2.c @@ -26,14 +26,18 @@ #include "dm_services.h" #include "ObjectID.h" -#include "atomfirmware.h" +#include "atomfirmware.h" +#include "atom.h" #include "include/bios_parser_interface.h" #include "command_table2.h" #include "command_table_helper2.h" #include "bios_parser_helper.h" #include "bios_parser_types_internal2.h" +#include "amdgpu.h" + + #define DC_LOGGER \ bp->base.ctx->logger @@ -43,16 +47,16 @@ ->FieldName)-(char *)0)/sizeof(uint16_t)) #define EXEC_BIOS_CMD_TABLE(fname, params)\ - (cgs_atom_exec_cmd_table(bp->base.ctx->cgs_device, \ + (amdgpu_atom_execute_table(((struct amdgpu_device *)bp->base.ctx->driver_context)->mode_info.atom_context, \ GET_INDEX_INTO_MASTER_TABLE(command, fname), \ - ¶ms) == 0) + (uint32_t *)¶ms) == 0) #define BIOS_CMD_TABLE_REVISION(fname, frev, crev)\ - cgs_atom_get_cmd_table_revs(bp->base.ctx->cgs_device, \ + amdgpu_atom_parse_cmd_header(((struct amdgpu_device *)bp->base.ctx->driver_context)->mode_info.atom_context, \ GET_INDEX_INTO_MASTER_TABLE(command, fname), &frev, &crev) #define BIOS_CMD_TABLE_PARA_REVISION(fname)\ - bios_cmd_table_para_revision(bp->base.ctx->cgs_device, \ + bios_cmd_table_para_revision(bp->base.ctx->driver_context, \ GET_INDEX_INTO_MASTER_TABLE(command, fname)) static void init_dig_encoder_control(struct bios_parser *bp); @@ -86,16 +90,18 @@ void dal_firmware_parser_init_cmd_tbl(struct bios_parser *bp) init_get_smu_clock_info(bp); } -static uint32_t bios_cmd_table_para_revision(void *cgs_device, +static uint32_t bios_cmd_table_para_revision(void *dev, uint32_t index) { + struct amdgpu_device *adev = dev; uint8_t frev, crev; - if (cgs_atom_get_cmd_table_revs(cgs_device, + if (amdgpu_atom_parse_cmd_header(adev->mode_info.atom_context, index, - &frev, &crev) != 0) + &frev, &crev)) + return crev; + else return 0; - return crev; } /****************************************************************************** @@ -201,7 +207,7 @@ static void init_transmitter_control(struct bios_parser *bp) uint8_t frev; uint8_t crev; - if (BIOS_CMD_TABLE_REVISION(dig1transmittercontrol, frev, crev) != 0) + if (BIOS_CMD_TABLE_REVISION(dig1transmittercontrol, frev, crev) == false) BREAK_TO_DEBUGGER(); switch (crev) { case 6: diff --git a/drivers/gpu/drm/amd/display/dc/bios/command_table_helper.c b/drivers/gpu/drm/amd/display/dc/bios/command_table_helper.c index 2979358c6a55..253bbb1eea60 100644 --- a/drivers/gpu/drm/amd/display/dc/bios/command_table_helper.c +++ b/drivers/gpu/drm/amd/display/dc/bios/command_table_helper.c @@ -51,6 +51,7 @@ bool dal_bios_parser_init_cmd_tbl_helper( return true; case DCE_VERSION_11_2: + case DCE_VERSION_11_22: *h = dal_cmd_tbl_helper_dce112_get_table(); return true; diff --git a/drivers/gpu/drm/amd/display/dc/bios/command_table_helper2.c b/drivers/gpu/drm/amd/display/dc/bios/command_table_helper2.c index 9a4d30dd4969..bbbcef566c55 100644 --- a/drivers/gpu/drm/amd/display/dc/bios/command_table_helper2.c +++ b/drivers/gpu/drm/amd/display/dc/bios/command_table_helper2.c @@ -52,6 +52,7 @@ bool dal_bios_parser_init_cmd_tbl_helper2( return true; case DCE_VERSION_11_2: + case DCE_VERSION_11_22: *h = dal_cmd_tbl_helper_dce112_get_table2(); return true; #if defined(CONFIG_DRM_AMD_DC_DCN1_0) diff --git a/drivers/gpu/drm/amd/display/dc/calcs/calcs_logger.h b/drivers/gpu/drm/amd/display/dc/calcs/calcs_logger.h new file mode 100644 index 000000000000..fc3f98fb09ea --- /dev/null +++ b/drivers/gpu/drm/amd/display/dc/calcs/calcs_logger.h @@ -0,0 +1,579 @@ +/* + * Copyright 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * Authors: AMD + * + */ + +#ifndef _CALCS_CALCS_LOGGER_H_ +#define _CALCS_CALCS_LOGGER_H_ +#define DC_LOGGER \ + logger + +static void print_bw_calcs_dceip(struct dal_logger *logger, const struct bw_calcs_dceip *dceip) +{ + + DC_LOG_BANDWIDTH_CALCS("#####################################################################"); + DC_LOG_BANDWIDTH_CALCS("struct bw_calcs_dceip"); + DC_LOG_BANDWIDTH_CALCS("#####################################################################"); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_calcs_version version %d", dceip->version); + DC_LOG_BANDWIDTH_CALCS(" [bool] large_cursor: %d", dceip->large_cursor); + DC_LOG_BANDWIDTH_CALCS(" [bool] dmif_pipe_en_fbc_chunk_tracker: %d", dceip->dmif_pipe_en_fbc_chunk_tracker); + DC_LOG_BANDWIDTH_CALCS(" [bool] display_write_back_supported: %d", dceip->display_write_back_supported); + DC_LOG_BANDWIDTH_CALCS(" [bool] argb_compression_support: %d", dceip->argb_compression_support); + DC_LOG_BANDWIDTH_CALCS(" [bool] pre_downscaler_enabled: %d", dceip->pre_downscaler_enabled); + DC_LOG_BANDWIDTH_CALCS(" [bool] underlay_downscale_prefetch_enabled: %d", + dceip->underlay_downscale_prefetch_enabled); + DC_LOG_BANDWIDTH_CALCS(" [bool] graphics_lb_nodownscaling_multi_line_prefetching: %d", + dceip->graphics_lb_nodownscaling_multi_line_prefetching); + DC_LOG_BANDWIDTH_CALCS(" [bool] limit_excessive_outstanding_dmif_requests: %d", + dceip->limit_excessive_outstanding_dmif_requests); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] cursor_max_outstanding_group_num: %d", + dceip->cursor_max_outstanding_group_num); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] lines_interleaved_into_lb: %d", dceip->lines_interleaved_into_lb); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] low_power_tiling_mode: %d", dceip->low_power_tiling_mode); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] chunk_width: %d", dceip->chunk_width); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] number_of_graphics_pipes: %d", dceip->number_of_graphics_pipes); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] number_of_underlay_pipes: %d", dceip->number_of_underlay_pipes); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] max_dmif_buffer_allocated: %d", dceip->max_dmif_buffer_allocated); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] graphics_dmif_size: %d", dceip->graphics_dmif_size); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] underlay_luma_dmif_size: %d", dceip->underlay_luma_dmif_size); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] underlay_chroma_dmif_size: %d", dceip->underlay_chroma_dmif_size); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] scatter_gather_lines_of_pte_prefetching_in_linear_mode: %d", + dceip->scatter_gather_lines_of_pte_prefetching_in_linear_mode); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] display_write_back420_luma_mcifwr_buffer_size: %d", + dceip->display_write_back420_luma_mcifwr_buffer_size); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] display_write_back420_chroma_mcifwr_buffer_size: %d", + dceip->display_write_back420_chroma_mcifwr_buffer_size); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] scatter_gather_pte_request_rows_in_tiling_mode: %d", + dceip->scatter_gather_pte_request_rows_in_tiling_mode); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] underlay_vscaler_efficiency10_bit_per_component: %d", + bw_fixed_to_int(dceip->underlay_vscaler_efficiency10_bit_per_component)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] underlay_vscaler_efficiency12_bit_per_component: %d", + bw_fixed_to_int(dceip->underlay_vscaler_efficiency12_bit_per_component)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] graphics_vscaler_efficiency6_bit_per_component: %d", + bw_fixed_to_int(dceip->graphics_vscaler_efficiency6_bit_per_component)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] graphics_vscaler_efficiency8_bit_per_component: %d", + bw_fixed_to_int(dceip->graphics_vscaler_efficiency8_bit_per_component)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] graphics_vscaler_efficiency10_bit_per_component: %d", + bw_fixed_to_int(dceip->graphics_vscaler_efficiency10_bit_per_component)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] graphics_vscaler_efficiency12_bit_per_component: %d", + bw_fixed_to_int(dceip->graphics_vscaler_efficiency12_bit_per_component)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] alpha_vscaler_efficiency: %d", + bw_fixed_to_int(dceip->alpha_vscaler_efficiency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] lb_write_pixels_per_dispclk: %d", + bw_fixed_to_int(dceip->lb_write_pixels_per_dispclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] lb_size_per_component444: %d", + bw_fixed_to_int(dceip->lb_size_per_component444)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] stutter_and_dram_clock_state_change_gated_before_cursor: %d", + bw_fixed_to_int(dceip->stutter_and_dram_clock_state_change_gated_before_cursor)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] underlay420_luma_lb_size_per_component: %d", + bw_fixed_to_int(dceip->underlay420_luma_lb_size_per_component)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] underlay420_chroma_lb_size_per_component: %d", + bw_fixed_to_int(dceip->underlay420_chroma_lb_size_per_component)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] underlay422_lb_size_per_component: %d", + bw_fixed_to_int(dceip->underlay422_lb_size_per_component)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] cursor_chunk_width: %d", bw_fixed_to_int(dceip->cursor_chunk_width)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] cursor_dcp_buffer_lines: %d", + bw_fixed_to_int(dceip->cursor_dcp_buffer_lines)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] underlay_maximum_width_efficient_for_tiling: %d", + bw_fixed_to_int(dceip->underlay_maximum_width_efficient_for_tiling)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] underlay_maximum_height_efficient_for_tiling: %d", + bw_fixed_to_int(dceip->underlay_maximum_height_efficient_for_tiling)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] peak_pte_request_to_eviction_ratio_limiting_multiple_displays_or_single_rotated_display: %d", + bw_fixed_to_int(dceip->peak_pte_request_to_eviction_ratio_limiting_multiple_displays_or_single_rotated_display)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] peak_pte_request_to_eviction_ratio_limiting_single_display_no_rotation: %d", + bw_fixed_to_int(dceip->peak_pte_request_to_eviction_ratio_limiting_single_display_no_rotation)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] minimum_outstanding_pte_request_limit: %d", + bw_fixed_to_int(dceip->minimum_outstanding_pte_request_limit)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] maximum_total_outstanding_pte_requests_allowed_by_saw: %d", + bw_fixed_to_int(dceip->maximum_total_outstanding_pte_requests_allowed_by_saw)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] linear_mode_line_request_alternation_slice: %d", + bw_fixed_to_int(dceip->linear_mode_line_request_alternation_slice)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] request_efficiency: %d", bw_fixed_to_int(dceip->request_efficiency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dispclk_per_request: %d", bw_fixed_to_int(dceip->dispclk_per_request)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dispclk_ramping_factor: %d", + bw_fixed_to_int(dceip->dispclk_ramping_factor)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] display_pipe_throughput_factor: %d", + bw_fixed_to_int(dceip->display_pipe_throughput_factor)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mcifwr_all_surfaces_burst_time: %d", + bw_fixed_to_int(dceip->mcifwr_all_surfaces_burst_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dmif_request_buffer_size: %d", + bw_fixed_to_int(dceip->dmif_request_buffer_size)); + + +} + +static void print_bw_calcs_vbios(struct dal_logger *logger, const struct bw_calcs_vbios *vbios) +{ + + DC_LOG_BANDWIDTH_CALCS("#####################################################################"); + DC_LOG_BANDWIDTH_CALCS("struct bw_calcs_vbios vbios"); + DC_LOG_BANDWIDTH_CALCS("#####################################################################"); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_defines memory_type: %d", vbios->memory_type); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_defines memory_type: %d", vbios->memory_type); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] dram_channel_width_in_bits: %d", vbios->dram_channel_width_in_bits); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] number_of_dram_channels: %d", vbios->number_of_dram_channels); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] number_of_dram_banks: %d", vbios->number_of_dram_banks); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] low_yclk: %d", bw_fixed_to_int(vbios->low_yclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mid_yclk: %d", bw_fixed_to_int(vbios->mid_yclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] high_yclk: %d", bw_fixed_to_int(vbios->high_yclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] low_sclk: %d", bw_fixed_to_int(vbios->low_sclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mid1_sclk: %d", bw_fixed_to_int(vbios->mid1_sclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mid2_sclk: %d", bw_fixed_to_int(vbios->mid2_sclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mid3_sclk: %d", bw_fixed_to_int(vbios->mid3_sclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mid4_sclk: %d", bw_fixed_to_int(vbios->mid4_sclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mid5_sclk: %d", bw_fixed_to_int(vbios->mid5_sclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mid6_sclk: %d", bw_fixed_to_int(vbios->mid6_sclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] high_sclk: %d", bw_fixed_to_int(vbios->high_sclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] low_voltage_max_dispclk: %d", + bw_fixed_to_int(vbios->low_voltage_max_dispclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mid_voltage_max_dispclk;: %d", + bw_fixed_to_int(vbios->mid_voltage_max_dispclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] high_voltage_max_dispclk;: %d", + bw_fixed_to_int(vbios->high_voltage_max_dispclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] low_voltage_max_phyclk: %d", + bw_fixed_to_int(vbios->low_voltage_max_phyclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mid_voltage_max_phyclk: %d", + bw_fixed_to_int(vbios->mid_voltage_max_phyclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] high_voltage_max_phyclk: %d", + bw_fixed_to_int(vbios->high_voltage_max_phyclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] data_return_bus_width: %d", bw_fixed_to_int(vbios->data_return_bus_width)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] trc: %d", bw_fixed_to_int(vbios->trc)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dmifmc_urgent_latency: %d", bw_fixed_to_int(vbios->dmifmc_urgent_latency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] stutter_self_refresh_exit_latency: %d", + bw_fixed_to_int(vbios->stutter_self_refresh_exit_latency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] stutter_self_refresh_entry_latency: %d", + bw_fixed_to_int(vbios->stutter_self_refresh_entry_latency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] nbp_state_change_latency: %d", + bw_fixed_to_int(vbios->nbp_state_change_latency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mcifwrmc_urgent_latency: %d", + bw_fixed_to_int(vbios->mcifwrmc_urgent_latency)); + DC_LOG_BANDWIDTH_CALCS(" [bool] scatter_gather_enable: %d", vbios->scatter_gather_enable); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] down_spread_percentage: %d", + bw_fixed_to_int(vbios->down_spread_percentage)); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] cursor_width: %d", vbios->cursor_width); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] average_compression_rate: %d", vbios->average_compression_rate); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] number_of_request_slots_gmc_reserves_for_dmif_per_channel: %d", + vbios->number_of_request_slots_gmc_reserves_for_dmif_per_channel); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] blackout_duration: %d", bw_fixed_to_int(vbios->blackout_duration)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] maximum_blackout_recovery_time: %d", + bw_fixed_to_int(vbios->maximum_blackout_recovery_time)); + + +} + +static void print_bw_calcs_data(struct dal_logger *logger, struct bw_calcs_data *data) +{ + + int i, j, k; + + DC_LOG_BANDWIDTH_CALCS("#####################################################################"); + DC_LOG_BANDWIDTH_CALCS("struct bw_calcs_data data"); + DC_LOG_BANDWIDTH_CALCS("#####################################################################"); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] number_of_displays: %d", data->number_of_displays); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_defines underlay_surface_type: %d", data->underlay_surface_type); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_defines panning_and_bezel_adjustment: %d", + data->panning_and_bezel_adjustment); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_defines graphics_tiling_mode: %d", data->graphics_tiling_mode); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] graphics_lb_bpc: %d", data->graphics_lb_bpc); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] underlay_lb_bpc: %d", data->underlay_lb_bpc); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_defines underlay_tiling_mode: %d", data->underlay_tiling_mode); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_defines d0_underlay_mode: %d", data->d0_underlay_mode); + DC_LOG_BANDWIDTH_CALCS(" [bool] d1_display_write_back_dwb_enable: %d", data->d1_display_write_back_dwb_enable); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_defines d1_underlay_mode: %d", data->d1_underlay_mode); + DC_LOG_BANDWIDTH_CALCS(" [bool] cpup_state_change_enable: %d", data->cpup_state_change_enable); + DC_LOG_BANDWIDTH_CALCS(" [bool] cpuc_state_change_enable: %d", data->cpuc_state_change_enable); + DC_LOG_BANDWIDTH_CALCS(" [bool] nbp_state_change_enable: %d", data->nbp_state_change_enable); + DC_LOG_BANDWIDTH_CALCS(" [bool] stutter_mode_enable: %d", data->stutter_mode_enable); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] y_clk_level: %d", data->y_clk_level); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] sclk_level: %d", data->sclk_level); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] number_of_underlay_surfaces: %d", data->number_of_underlay_surfaces); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] number_of_dram_wrchannels: %d", data->number_of_dram_wrchannels); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] chunk_request_delay: %d", data->chunk_request_delay); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] number_of_dram_channels: %d", data->number_of_dram_channels); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_defines underlay_micro_tile_mode: %d", data->underlay_micro_tile_mode); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_defines graphics_micro_tile_mode: %d", data->graphics_micro_tile_mode); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] max_phyclk: %d", bw_fixed_to_int(data->max_phyclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dram_efficiency: %d", bw_fixed_to_int(data->dram_efficiency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] src_width_after_surface_type: %d", + bw_fixed_to_int(data->src_width_after_surface_type)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] src_height_after_surface_type: %d", + bw_fixed_to_int(data->src_height_after_surface_type)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] hsr_after_surface_type: %d", + bw_fixed_to_int(data->hsr_after_surface_type)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] vsr_after_surface_type: %d", bw_fixed_to_int(data->vsr_after_surface_type)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] src_width_after_rotation: %d", + bw_fixed_to_int(data->src_width_after_rotation)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] src_height_after_rotation: %d", + bw_fixed_to_int(data->src_height_after_rotation)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] hsr_after_rotation: %d", bw_fixed_to_int(data->hsr_after_rotation)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] vsr_after_rotation: %d", bw_fixed_to_int(data->vsr_after_rotation)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] source_height_pixels: %d", bw_fixed_to_int(data->source_height_pixels)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] hsr_after_stereo: %d", bw_fixed_to_int(data->hsr_after_stereo)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] vsr_after_stereo: %d", bw_fixed_to_int(data->vsr_after_stereo)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] source_width_in_lb: %d", bw_fixed_to_int(data->source_width_in_lb)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] lb_line_pitch: %d", bw_fixed_to_int(data->lb_line_pitch)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] underlay_maximum_source_efficient_for_tiling: %d", + bw_fixed_to_int(data->underlay_maximum_source_efficient_for_tiling)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] num_lines_at_frame_start: %d", + bw_fixed_to_int(data->num_lines_at_frame_start)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] min_dmif_size_in_time: %d", bw_fixed_to_int(data->min_dmif_size_in_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] min_mcifwr_size_in_time: %d", + bw_fixed_to_int(data->min_mcifwr_size_in_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_requests_for_dmif_size: %d", + bw_fixed_to_int(data->total_requests_for_dmif_size)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] peak_pte_request_to_eviction_ratio_limiting: %d", + bw_fixed_to_int(data->peak_pte_request_to_eviction_ratio_limiting)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] useful_pte_per_pte_request: %d", + bw_fixed_to_int(data->useful_pte_per_pte_request)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] scatter_gather_pte_request_rows: %d", + bw_fixed_to_int(data->scatter_gather_pte_request_rows)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] scatter_gather_row_height: %d", + bw_fixed_to_int(data->scatter_gather_row_height)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] scatter_gather_pte_requests_in_vblank: %d", + bw_fixed_to_int(data->scatter_gather_pte_requests_in_vblank)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] inefficient_linear_pitch_in_bytes: %d", + bw_fixed_to_int(data->inefficient_linear_pitch_in_bytes)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] cursor_total_data: %d", bw_fixed_to_int(data->cursor_total_data)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] cursor_total_request_groups: %d", + bw_fixed_to_int(data->cursor_total_request_groups)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] scatter_gather_total_pte_requests: %d", + bw_fixed_to_int(data->scatter_gather_total_pte_requests)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] scatter_gather_total_pte_request_groups: %d", + bw_fixed_to_int(data->scatter_gather_total_pte_request_groups)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] tile_width_in_pixels: %d", bw_fixed_to_int(data->tile_width_in_pixels)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dmif_total_number_of_data_request_page_close_open: %d", + bw_fixed_to_int(data->dmif_total_number_of_data_request_page_close_open)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mcifwr_total_number_of_data_request_page_close_open: %d", + bw_fixed_to_int(data->mcifwr_total_number_of_data_request_page_close_open)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] bytes_per_page_close_open: %d", + bw_fixed_to_int(data->bytes_per_page_close_open)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mcifwr_total_page_close_open_time: %d", + bw_fixed_to_int(data->mcifwr_total_page_close_open_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_requests_for_adjusted_dmif_size: %d", + bw_fixed_to_int(data->total_requests_for_adjusted_dmif_size)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_dmifmc_urgent_trips: %d", + bw_fixed_to_int(data->total_dmifmc_urgent_trips)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_dmifmc_urgent_latency: %d", + bw_fixed_to_int(data->total_dmifmc_urgent_latency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_display_reads_required_data: %d", + bw_fixed_to_int(data->total_display_reads_required_data)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_display_reads_required_dram_access_data: %d", + bw_fixed_to_int(data->total_display_reads_required_dram_access_data)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_display_writes_required_data: %d", + bw_fixed_to_int(data->total_display_writes_required_data)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_display_writes_required_dram_access_data: %d", + bw_fixed_to_int(data->total_display_writes_required_dram_access_data)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] display_reads_required_data: %d", + bw_fixed_to_int(data->display_reads_required_data)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] display_reads_required_dram_access_data: %d", + bw_fixed_to_int(data->display_reads_required_dram_access_data)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dmif_total_page_close_open_time: %d", + bw_fixed_to_int(data->dmif_total_page_close_open_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] min_cursor_memory_interface_buffer_size_in_time: %d", + bw_fixed_to_int(data->min_cursor_memory_interface_buffer_size_in_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] min_read_buffer_size_in_time: %d", + bw_fixed_to_int(data->min_read_buffer_size_in_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] display_reads_time_for_data_transfer: %d", + bw_fixed_to_int(data->display_reads_time_for_data_transfer)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] display_writes_time_for_data_transfer: %d", + bw_fixed_to_int(data->display_writes_time_for_data_transfer)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dmif_required_dram_bandwidth: %d", + bw_fixed_to_int(data->dmif_required_dram_bandwidth)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mcifwr_required_dram_bandwidth: %d", + bw_fixed_to_int(data->mcifwr_required_dram_bandwidth)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] required_dmifmc_urgent_latency_for_page_close_open: %d", + bw_fixed_to_int(data->required_dmifmc_urgent_latency_for_page_close_open)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] required_mcifmcwr_urgent_latency: %d", + bw_fixed_to_int(data->required_mcifmcwr_urgent_latency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] required_dram_bandwidth_gbyte_per_second: %d", + bw_fixed_to_int(data->required_dram_bandwidth_gbyte_per_second)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dram_bandwidth: %d", bw_fixed_to_int(data->dram_bandwidth)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dmif_required_sclk: %d", bw_fixed_to_int(data->dmif_required_sclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mcifwr_required_sclk: %d", bw_fixed_to_int(data->mcifwr_required_sclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] required_sclk: %d", bw_fixed_to_int(data->required_sclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] downspread_factor: %d", bw_fixed_to_int(data->downspread_factor)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] v_scaler_efficiency: %d", bw_fixed_to_int(data->v_scaler_efficiency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] scaler_limits_factor: %d", bw_fixed_to_int(data->scaler_limits_factor)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] display_pipe_pixel_throughput: %d", + bw_fixed_to_int(data->display_pipe_pixel_throughput)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_dispclk_required_with_ramping: %d", + bw_fixed_to_int(data->total_dispclk_required_with_ramping)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_dispclk_required_without_ramping: %d", + bw_fixed_to_int(data->total_dispclk_required_without_ramping)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_read_request_bandwidth: %d", + bw_fixed_to_int(data->total_read_request_bandwidth)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_write_request_bandwidth: %d", + bw_fixed_to_int(data->total_write_request_bandwidth)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dispclk_required_for_total_read_request_bandwidth: %d", + bw_fixed_to_int(data->dispclk_required_for_total_read_request_bandwidth)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_dispclk_required_with_ramping_with_request_bandwidth: %d", + bw_fixed_to_int(data->total_dispclk_required_with_ramping_with_request_bandwidth)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_dispclk_required_without_ramping_with_request_bandwidth: %d", + bw_fixed_to_int(data->total_dispclk_required_without_ramping_with_request_bandwidth)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dispclk: %d", bw_fixed_to_int(data->dispclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] blackout_recovery_time: %d", bw_fixed_to_int(data->blackout_recovery_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] min_pixels_per_data_fifo_entry: %d", + bw_fixed_to_int(data->min_pixels_per_data_fifo_entry)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] sclk_deep_sleep: %d", bw_fixed_to_int(data->sclk_deep_sleep)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] chunk_request_time: %d", bw_fixed_to_int(data->chunk_request_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] cursor_request_time: %d", bw_fixed_to_int(data->cursor_request_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] line_source_pixels_transfer_time: %d", + bw_fixed_to_int(data->line_source_pixels_transfer_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dmifdram_access_efficiency: %d", + bw_fixed_to_int(data->dmifdram_access_efficiency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mcifwrdram_access_efficiency: %d", + bw_fixed_to_int(data->mcifwrdram_access_efficiency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_average_bandwidth_no_compression: %d", + bw_fixed_to_int(data->total_average_bandwidth_no_compression)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_average_bandwidth: %d", + bw_fixed_to_int(data->total_average_bandwidth)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] total_stutter_cycle_duration: %d", + bw_fixed_to_int(data->total_stutter_cycle_duration)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] stutter_burst_time: %d", bw_fixed_to_int(data->stutter_burst_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] time_in_self_refresh: %d", bw_fixed_to_int(data->time_in_self_refresh)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] stutter_efficiency: %d", bw_fixed_to_int(data->stutter_efficiency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] worst_number_of_trips_to_memory: %d", + bw_fixed_to_int(data->worst_number_of_trips_to_memory)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] immediate_flip_time: %d", bw_fixed_to_int(data->immediate_flip_time)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] latency_for_non_dmif_clients: %d", + bw_fixed_to_int(data->latency_for_non_dmif_clients)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] latency_for_non_mcifwr_clients: %d", + bw_fixed_to_int(data->latency_for_non_mcifwr_clients)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dmifmc_urgent_latency_supported_in_high_sclk_and_yclk: %d", + bw_fixed_to_int(data->dmifmc_urgent_latency_supported_in_high_sclk_and_yclk)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] nbp_state_dram_speed_change_margin: %d", + bw_fixed_to_int(data->nbp_state_dram_speed_change_margin)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] display_reads_time_for_data_transfer_and_urgent_latency: %d", + bw_fixed_to_int(data->display_reads_time_for_data_transfer_and_urgent_latency)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dram_speed_change_margin: %d", + bw_fixed_to_int(data->dram_speed_change_margin)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] min_vblank_dram_speed_change_margin: %d", + bw_fixed_to_int(data->min_vblank_dram_speed_change_margin)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] min_stutter_refresh_duration: %d", + bw_fixed_to_int(data->min_stutter_refresh_duration)); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] total_stutter_dmif_buffer_size: %d", data->total_stutter_dmif_buffer_size); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] total_bytes_requested: %d", data->total_bytes_requested); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] min_stutter_dmif_buffer_size: %d", data->min_stutter_dmif_buffer_size); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] num_stutter_bursts: %d", data->num_stutter_bursts); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] v_blank_nbp_state_dram_speed_change_latency_supported: %d", + bw_fixed_to_int(data->v_blank_nbp_state_dram_speed_change_latency_supported)); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] nbp_state_dram_speed_change_latency_supported: %d", + bw_fixed_to_int(data->nbp_state_dram_speed_change_latency_supported)); + + for (i = 0; i < maximum_number_of_surfaces; i++) { + DC_LOG_BANDWIDTH_CALCS(" [bool] fbc_en[%d]:%d\n", i, data->fbc_en[i]); + DC_LOG_BANDWIDTH_CALCS(" [bool] lpt_en[%d]:%d", i, data->lpt_en[i]); + DC_LOG_BANDWIDTH_CALCS(" [bool] displays_match_flag[%d]:%d", i, data->displays_match_flag[i]); + DC_LOG_BANDWIDTH_CALCS(" [bool] use_alpha[%d]:%d", i, data->use_alpha[i]); + DC_LOG_BANDWIDTH_CALCS(" [bool] orthogonal_rotation[%d]:%d", i, data->orthogonal_rotation[i]); + DC_LOG_BANDWIDTH_CALCS(" [bool] enable[%d]:%d", i, data->enable[i]); + DC_LOG_BANDWIDTH_CALCS(" [bool] access_one_channel_only[%d]:%d", i, data->access_one_channel_only[i]); + DC_LOG_BANDWIDTH_CALCS(" [bool] scatter_gather_enable_for_pipe[%d]:%d", + i, data->scatter_gather_enable_for_pipe[i]); + DC_LOG_BANDWIDTH_CALCS(" [bool] interlace_mode[%d]:%d", + i, data->interlace_mode[i]); + DC_LOG_BANDWIDTH_CALCS(" [bool] display_pstate_change_enable[%d]:%d", + i, data->display_pstate_change_enable[i]); + DC_LOG_BANDWIDTH_CALCS(" [bool] line_buffer_prefetch[%d]:%d", i, data->line_buffer_prefetch[i]); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] bytes_per_pixel[%d]:%d", i, data->bytes_per_pixel[i]); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] max_chunks_non_fbc_mode[%d]:%d", + i, data->max_chunks_non_fbc_mode[i]); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] lb_bpc[%d]:%d", i, data->lb_bpc[i]); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] output_bpphdmi[%d]:%d", i, data->output_bpphdmi[i]); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] output_bppdp4_lane_hbr[%d]:%d", i, data->output_bppdp4_lane_hbr[i]); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] output_bppdp4_lane_hbr2[%d]:%d", + i, data->output_bppdp4_lane_hbr2[i]); + DC_LOG_BANDWIDTH_CALCS(" [uint32_t] output_bppdp4_lane_hbr3[%d]:%d", + i, data->output_bppdp4_lane_hbr3[i]); + DC_LOG_BANDWIDTH_CALCS(" [enum] bw_defines stereo_mode[%d]:%d", i, data->stereo_mode[i]); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dmif_buffer_transfer_time[%d]:%d", + i, bw_fixed_to_int(data->dmif_buffer_transfer_time[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] displays_with_same_mode[%d]:%d", + i, bw_fixed_to_int(data->displays_with_same_mode[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] stutter_dmif_buffer_size[%d]:%d", + i, bw_fixed_to_int(data->stutter_dmif_buffer_size[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] stutter_refresh_duration[%d]:%d", + i, bw_fixed_to_int(data->stutter_refresh_duration[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] stutter_exit_watermark[%d]:%d", + i, bw_fixed_to_int(data->stutter_exit_watermark[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] stutter_entry_watermark[%d]:%d", + i, bw_fixed_to_int(data->stutter_entry_watermark[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] h_total[%d]:%d", i, bw_fixed_to_int(data->h_total[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] v_total[%d]:%d", i, bw_fixed_to_int(data->v_total[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] pixel_rate[%d]:%d", i, bw_fixed_to_int(data->pixel_rate[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] src_width[%d]:%d", i, bw_fixed_to_int(data->src_width[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] pitch_in_pixels[%d]:%d", + i, bw_fixed_to_int(data->pitch_in_pixels[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] pitch_in_pixels_after_surface_type[%d]:%d", + i, bw_fixed_to_int(data->pitch_in_pixels_after_surface_type[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] src_height[%d]:%d", i, bw_fixed_to_int(data->src_height[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] scale_ratio[%d]:%d", i, bw_fixed_to_int(data->scale_ratio[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] h_taps[%d]:%d", i, bw_fixed_to_int(data->h_taps[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] v_taps[%d]:%d", i, bw_fixed_to_int(data->v_taps[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] h_scale_ratio[%d]:%d", i, bw_fixed_to_int(data->h_scale_ratio[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] v_scale_ratio[%d]:%d", i, bw_fixed_to_int(data->v_scale_ratio[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] rotation_angle[%d]:%d", + i, bw_fixed_to_int(data->rotation_angle[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] compression_rate[%d]:%d", + i, bw_fixed_to_int(data->compression_rate[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] hsr[%d]:%d", i, bw_fixed_to_int(data->hsr[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] vsr[%d]:%d", i, bw_fixed_to_int(data->vsr[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] source_width_rounded_up_to_chunks[%d]:%d", + i, bw_fixed_to_int(data->source_width_rounded_up_to_chunks[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] source_width_pixels[%d]:%d", + i, bw_fixed_to_int(data->source_width_pixels[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] source_height_rounded_up_to_chunks[%d]:%d", + i, bw_fixed_to_int(data->source_height_rounded_up_to_chunks[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] display_bandwidth[%d]:%d", + i, bw_fixed_to_int(data->display_bandwidth[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] request_bandwidth[%d]:%d", + i, bw_fixed_to_int(data->request_bandwidth[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] bytes_per_request[%d]:%d", + i, bw_fixed_to_int(data->bytes_per_request[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] useful_bytes_per_request[%d]:%d", + i, bw_fixed_to_int(data->useful_bytes_per_request[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] lines_interleaved_in_mem_access[%d]:%d", + i, bw_fixed_to_int(data->lines_interleaved_in_mem_access[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] latency_hiding_lines[%d]:%d", + i, bw_fixed_to_int(data->latency_hiding_lines[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] lb_partitions[%d]:%d", + i, bw_fixed_to_int(data->lb_partitions[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] lb_partitions_max[%d]:%d", + i, bw_fixed_to_int(data->lb_partitions_max[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dispclk_required_with_ramping[%d]:%d", + i, bw_fixed_to_int(data->dispclk_required_with_ramping[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dispclk_required_without_ramping[%d]:%d", + i, bw_fixed_to_int(data->dispclk_required_without_ramping[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] data_buffer_size[%d]:%d", + i, bw_fixed_to_int(data->data_buffer_size[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] outstanding_chunk_request_limit[%d]:%d", + i, bw_fixed_to_int(data->outstanding_chunk_request_limit[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] urgent_watermark[%d]:%d", + i, bw_fixed_to_int(data->urgent_watermark[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] nbp_state_change_watermark[%d]:%d", + i, bw_fixed_to_int(data->nbp_state_change_watermark[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] v_filter_init[%d]:%d", i, bw_fixed_to_int(data->v_filter_init[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] stutter_cycle_duration[%d]:%d", + i, bw_fixed_to_int(data->stutter_cycle_duration[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] average_bandwidth[%d]:%d", + i, bw_fixed_to_int(data->average_bandwidth[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] average_bandwidth_no_compression[%d]:%d", + i, bw_fixed_to_int(data->average_bandwidth_no_compression[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] scatter_gather_pte_request_limit[%d]:%d", + i, bw_fixed_to_int(data->scatter_gather_pte_request_limit[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] lb_size_per_component[%d]:%d", + i, bw_fixed_to_int(data->lb_size_per_component[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] memory_chunk_size_in_bytes[%d]:%d", + i, bw_fixed_to_int(data->memory_chunk_size_in_bytes[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] pipe_chunk_size_in_bytes[%d]:%d", + i, bw_fixed_to_int(data->pipe_chunk_size_in_bytes[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] number_of_trips_to_memory_for_getting_apte_row[%d]:%d", + i, bw_fixed_to_int(data->number_of_trips_to_memory_for_getting_apte_row[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] adjusted_data_buffer_size[%d]:%d", + i, bw_fixed_to_int(data->adjusted_data_buffer_size[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] adjusted_data_buffer_size_in_memory[%d]:%d", + i, bw_fixed_to_int(data->adjusted_data_buffer_size_in_memory[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] pixels_per_data_fifo_entry[%d]:%d", + i, bw_fixed_to_int(data->pixels_per_data_fifo_entry[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] scatter_gather_pte_requests_in_row[%d]:%d", + i, bw_fixed_to_int(data->scatter_gather_pte_requests_in_row[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] pte_request_per_chunk[%d]:%d", + i, bw_fixed_to_int(data->pte_request_per_chunk[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] scatter_gather_page_width[%d]:%d", + i, bw_fixed_to_int(data->scatter_gather_page_width[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] scatter_gather_page_height[%d]:%d", + i, bw_fixed_to_int(data->scatter_gather_page_height[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] lb_lines_in_per_line_out_in_beginning_of_frame[%d]:%d", + i, bw_fixed_to_int(data->lb_lines_in_per_line_out_in_beginning_of_frame[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] lb_lines_in_per_line_out_in_middle_of_frame[%d]:%d", + i, bw_fixed_to_int(data->lb_lines_in_per_line_out_in_middle_of_frame[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] cursor_width_pixels[%d]:%d", + i, bw_fixed_to_int(data->cursor_width_pixels[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] minimum_latency_hiding[%d]:%d", + i, bw_fixed_to_int(data->minimum_latency_hiding[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] maximum_latency_hiding[%d]:%d", + i, bw_fixed_to_int(data->maximum_latency_hiding[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] minimum_latency_hiding_with_cursor[%d]:%d", + i, bw_fixed_to_int(data->minimum_latency_hiding_with_cursor[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] maximum_latency_hiding_with_cursor[%d]:%d", + i, bw_fixed_to_int(data->maximum_latency_hiding_with_cursor[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] src_pixels_for_first_output_pixel[%d]:%d", + i, bw_fixed_to_int(data->src_pixels_for_first_output_pixel[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] src_pixels_for_last_output_pixel[%d]:%d", + i, bw_fixed_to_int(data->src_pixels_for_last_output_pixel[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] src_data_for_first_output_pixel[%d]:%d", + i, bw_fixed_to_int(data->src_data_for_first_output_pixel[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] src_data_for_last_output_pixel[%d]:%d", + i, bw_fixed_to_int(data->src_data_for_last_output_pixel[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] active_time[%d]:%d", i, bw_fixed_to_int(data->active_time[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] horizontal_blank_and_chunk_granularity_factor[%d]:%d", + i, bw_fixed_to_int(data->horizontal_blank_and_chunk_granularity_factor[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] cursor_latency_hiding[%d]:%d", + i, bw_fixed_to_int(data->cursor_latency_hiding[i])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] v_blank_dram_speed_change_margin[%d]:%d", + i, bw_fixed_to_int(data->v_blank_dram_speed_change_margin[i])); + } + + for (i = 0; i < maximum_number_of_surfaces; i++) { + for (j = 0; j < 3; j++) { + for (k = 0; k < 8; k++) { + + DC_LOG_BANDWIDTH_CALCS("\n [bw_fixed] line_source_transfer_time[%d][%d][%d]:%d", + i, j, k, bw_fixed_to_int(data->line_source_transfer_time[i][j][k])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dram_speed_change_line_source_transfer_time[%d][%d][%d]:%d", + i, j, k, + bw_fixed_to_int(data->dram_speed_change_line_source_transfer_time[i][j][k])); + } + } + } + + for (i = 0; i < 3; i++) { + for (j = 0; j < 8; j++) { + + DC_LOG_BANDWIDTH_CALCS("\n [uint32_t] num_displays_with_margin[%d][%d]:%d", + i, j, data->num_displays_with_margin[i][j]); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dmif_burst_time[%d][%d]:%d", + i, j, bw_fixed_to_int(data->dmif_burst_time[i][j])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] mcifwr_burst_time[%d][%d]:%d", + i, j, bw_fixed_to_int(data->mcifwr_burst_time[i][j])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] min_dram_speed_change_margin[%d][%d]:%d", + i, j, bw_fixed_to_int(data->min_dram_speed_change_margin[i][j])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dispclk_required_for_dram_speed_change[%d][%d]:%d", + i, j, bw_fixed_to_int(data->dispclk_required_for_dram_speed_change[i][j])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] blackout_duration_margin[%d][%d]:%d", + i, j, bw_fixed_to_int(data->blackout_duration_margin[i][j])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dispclk_required_for_blackout_duration[%d][%d]:%d", + i, j, bw_fixed_to_int(data->dispclk_required_for_blackout_duration[i][j])); + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dispclk_required_for_blackout_recovery[%d][%d]:%d", + i, j, bw_fixed_to_int(data->dispclk_required_for_blackout_recovery[i][j])); + } + } + + for (i = 0; i < 6; i++) { + DC_LOG_BANDWIDTH_CALCS(" [bw_fixed] dmif_required_sclk_for_urgent_latency[%d]:%d", + i, bw_fixed_to_int(data->dmif_required_sclk_for_urgent_latency[i])); + } +} +; + +#endif /* _CALCS_CALCS_LOGGER_H_ */ diff --git a/drivers/gpu/drm/amd/display/dc/calcs/custom_float.c b/drivers/gpu/drm/amd/display/dc/calcs/custom_float.c index 7243c37f569e..31d167bc548f 100644 --- a/drivers/gpu/drm/amd/display/dc/calcs/custom_float.c +++ b/drivers/gpu/drm/amd/display/dc/calcs/custom_float.c @@ -36,41 +36,41 @@ static bool build_custom_float( uint32_t exp_offset = (1 << (format->exponenta_bits - 1)) - 1; const struct fixed31_32 mantissa_constant_plus_max_fraction = - dal_fixed31_32_from_fraction( + dc_fixpt_from_fraction( (1LL << (format->mantissa_bits + 1)) - 1, 1LL << format->mantissa_bits); struct fixed31_32 mantiss; - if (dal_fixed31_32_eq( + if (dc_fixpt_eq( value, - dal_fixed31_32_zero)) { + dc_fixpt_zero)) { *negative = false; *mantissa = 0; *exponenta = 0; return true; } - if (dal_fixed31_32_lt( + if (dc_fixpt_lt( value, - dal_fixed31_32_zero)) { + dc_fixpt_zero)) { *negative = format->sign; - value = dal_fixed31_32_neg(value); + value = dc_fixpt_neg(value); } else { *negative = false; } - if (dal_fixed31_32_lt( + if (dc_fixpt_lt( value, - dal_fixed31_32_one)) { + dc_fixpt_one)) { uint32_t i = 1; do { - value = dal_fixed31_32_shl(value, 1); + value = dc_fixpt_shl(value, 1); ++i; - } while (dal_fixed31_32_lt( + } while (dc_fixpt_lt( value, - dal_fixed31_32_one)); + dc_fixpt_one)); --i; @@ -81,15 +81,15 @@ static bool build_custom_float( } *exponenta = exp_offset - i; - } else if (dal_fixed31_32_le( + } else if (dc_fixpt_le( mantissa_constant_plus_max_fraction, value)) { uint32_t i = 1; do { - value = dal_fixed31_32_shr(value, 1); + value = dc_fixpt_shr(value, 1); ++i; - } while (dal_fixed31_32_lt( + } while (dc_fixpt_lt( mantissa_constant_plus_max_fraction, value)); @@ -98,23 +98,23 @@ static bool build_custom_float( *exponenta = exp_offset; } - mantiss = dal_fixed31_32_sub( + mantiss = dc_fixpt_sub( value, - dal_fixed31_32_one); + dc_fixpt_one); - if (dal_fixed31_32_lt( + if (dc_fixpt_lt( mantiss, - dal_fixed31_32_zero) || - dal_fixed31_32_lt( - dal_fixed31_32_one, + dc_fixpt_zero) || + dc_fixpt_lt( + dc_fixpt_one, mantiss)) - mantiss = dal_fixed31_32_zero; + mantiss = dc_fixpt_zero; else - mantiss = dal_fixed31_32_shl( + mantiss = dc_fixpt_shl( mantiss, format->mantissa_bits); - *mantissa = dal_fixed31_32_floor(mantiss); + *mantissa = dc_fixpt_floor(mantiss); return true; } diff --git a/drivers/gpu/drm/amd/display/dc/calcs/dce_calcs.c b/drivers/gpu/drm/amd/display/dc/calcs/dce_calcs.c index 0cbab81ab304..2c4e8f0cb2dc 100644 --- a/drivers/gpu/drm/amd/display/dc/calcs/dce_calcs.c +++ b/drivers/gpu/drm/amd/display/dc/calcs/dce_calcs.c @@ -28,6 +28,7 @@ #include "dc.h" #include "core_types.h" #include "dal_asic_id.h" +#include "calcs_logger.h" /* * NOTE: @@ -52,11 +53,14 @@ static enum bw_calcs_version bw_calcs_version_from_asic_id(struct hw_asic_id asi return BW_CALCS_VERSION_CARRIZO; case FAMILY_VI: + if (ASIC_REV_IS_POLARIS12_V(asic_id.hw_internal_rev)) + return BW_CALCS_VERSION_POLARIS12; if (ASIC_REV_IS_POLARIS10_P(asic_id.hw_internal_rev)) return BW_CALCS_VERSION_POLARIS10; - if (ASIC_REV_IS_POLARIS11_M(asic_id.hw_internal_rev) || - ASIC_REV_IS_POLARIS12_V(asic_id.hw_internal_rev)) + if (ASIC_REV_IS_POLARIS11_M(asic_id.hw_internal_rev)) return BW_CALCS_VERSION_POLARIS11; + if (ASIC_REV_IS_VEGAM(asic_id.hw_internal_rev)) + return BW_CALCS_VERSION_VEGAM; return BW_CALCS_VERSION_INVALID; case FAMILY_AI: @@ -2145,6 +2149,9 @@ void bw_calcs_init(struct bw_calcs_dceip *bw_dceip, dceip.mcifwr_all_surfaces_burst_time = bw_int_to_fixed(0); /* todo: this is a bug*/ break; case BW_CALCS_VERSION_POLARIS10: + /* TODO: Treat VEGAM the same as P10 for now + * Need to tune the para for VEGAM if needed */ + case BW_CALCS_VERSION_VEGAM: vbios.memory_type = bw_def_gddr5; vbios.dram_channel_width_in_bits = 32; vbios.number_of_dram_channels = asic_id.vram_width / vbios.dram_channel_width_in_bits; @@ -2373,6 +2380,122 @@ void bw_calcs_init(struct bw_calcs_dceip *bw_dceip, dceip.scatter_gather_pte_request_rows_in_tiling_mode = 2; dceip.mcifwr_all_surfaces_burst_time = bw_int_to_fixed(0); break; + case BW_CALCS_VERSION_POLARIS12: + vbios.memory_type = bw_def_gddr5; + vbios.dram_channel_width_in_bits = 32; + vbios.number_of_dram_channels = asic_id.vram_width / vbios.dram_channel_width_in_bits; + vbios.number_of_dram_banks = 8; + vbios.high_yclk = bw_int_to_fixed(6000); + vbios.mid_yclk = bw_int_to_fixed(3200); + vbios.low_yclk = bw_int_to_fixed(1000); + vbios.low_sclk = bw_int_to_fixed(678); + vbios.mid1_sclk = bw_int_to_fixed(864); + vbios.mid2_sclk = bw_int_to_fixed(900); + vbios.mid3_sclk = bw_int_to_fixed(920); + vbios.mid4_sclk = bw_int_to_fixed(940); + vbios.mid5_sclk = bw_int_to_fixed(960); + vbios.mid6_sclk = bw_int_to_fixed(980); + vbios.high_sclk = bw_int_to_fixed(1049); + vbios.low_voltage_max_dispclk = bw_int_to_fixed(459); + vbios.mid_voltage_max_dispclk = bw_int_to_fixed(654); + vbios.high_voltage_max_dispclk = bw_int_to_fixed(1108); + vbios.low_voltage_max_phyclk = bw_int_to_fixed(540); + vbios.mid_voltage_max_phyclk = bw_int_to_fixed(810); + vbios.high_voltage_max_phyclk = bw_int_to_fixed(810); + vbios.data_return_bus_width = bw_int_to_fixed(32); + vbios.trc = bw_int_to_fixed(48); + if (vbios.number_of_dram_channels == 2) // 64-bit + vbios.dmifmc_urgent_latency = bw_int_to_fixed(4); + else + vbios.dmifmc_urgent_latency = bw_int_to_fixed(3); + vbios.stutter_self_refresh_exit_latency = bw_int_to_fixed(5); + vbios.stutter_self_refresh_entry_latency = bw_int_to_fixed(0); + vbios.nbp_state_change_latency = bw_int_to_fixed(250); + vbios.mcifwrmc_urgent_latency = bw_int_to_fixed(10); + vbios.scatter_gather_enable = false; + vbios.down_spread_percentage = bw_frc_to_fixed(5, 10); + vbios.cursor_width = 32; + vbios.average_compression_rate = 4; + vbios.number_of_request_slots_gmc_reserves_for_dmif_per_channel = 256; + vbios.blackout_duration = bw_int_to_fixed(0); /* us */ + vbios.maximum_blackout_recovery_time = bw_int_to_fixed(0); + + dceip.max_average_percent_of_ideal_port_bw_display_can_use_in_normal_system_operation = 100; + dceip.max_average_percent_of_ideal_drambw_display_can_use_in_normal_system_operation = 100; + dceip.percent_of_ideal_port_bw_received_after_urgent_latency = 100; + dceip.large_cursor = false; + dceip.dmif_request_buffer_size = bw_int_to_fixed(768); + dceip.dmif_pipe_en_fbc_chunk_tracker = false; + dceip.cursor_max_outstanding_group_num = 1; + dceip.lines_interleaved_into_lb = 2; + dceip.chunk_width = 256; + dceip.number_of_graphics_pipes = 5; + dceip.number_of_underlay_pipes = 0; + dceip.low_power_tiling_mode = 0; + dceip.display_write_back_supported = true; + dceip.argb_compression_support = true; + dceip.underlay_vscaler_efficiency6_bit_per_component = + bw_frc_to_fixed(35556, 10000); + dceip.underlay_vscaler_efficiency8_bit_per_component = + bw_frc_to_fixed(34286, 10000); + dceip.underlay_vscaler_efficiency10_bit_per_component = + bw_frc_to_fixed(32, 10); + dceip.underlay_vscaler_efficiency12_bit_per_component = + bw_int_to_fixed(3); + dceip.graphics_vscaler_efficiency6_bit_per_component = + bw_frc_to_fixed(35, 10); + dceip.graphics_vscaler_efficiency8_bit_per_component = + bw_frc_to_fixed(34286, 10000); + dceip.graphics_vscaler_efficiency10_bit_per_component = + bw_frc_to_fixed(32, 10); + dceip.graphics_vscaler_efficiency12_bit_per_component = + bw_int_to_fixed(3); + dceip.alpha_vscaler_efficiency = bw_int_to_fixed(3); + dceip.max_dmif_buffer_allocated = 4; + dceip.graphics_dmif_size = 12288; + dceip.underlay_luma_dmif_size = 19456; + dceip.underlay_chroma_dmif_size = 23552; + dceip.pre_downscaler_enabled = true; + dceip.underlay_downscale_prefetch_enabled = true; + dceip.lb_write_pixels_per_dispclk = bw_int_to_fixed(1); + dceip.lb_size_per_component444 = bw_int_to_fixed(245952); + dceip.graphics_lb_nodownscaling_multi_line_prefetching = true; + dceip.stutter_and_dram_clock_state_change_gated_before_cursor = + bw_int_to_fixed(1); + dceip.underlay420_luma_lb_size_per_component = bw_int_to_fixed( + 82176); + dceip.underlay420_chroma_lb_size_per_component = + bw_int_to_fixed(164352); + dceip.underlay422_lb_size_per_component = bw_int_to_fixed( + 82176); + dceip.cursor_chunk_width = bw_int_to_fixed(64); + dceip.cursor_dcp_buffer_lines = bw_int_to_fixed(4); + dceip.underlay_maximum_width_efficient_for_tiling = + bw_int_to_fixed(1920); + dceip.underlay_maximum_height_efficient_for_tiling = + bw_int_to_fixed(1080); + dceip.peak_pte_request_to_eviction_ratio_limiting_multiple_displays_or_single_rotated_display = + bw_frc_to_fixed(3, 10); + dceip.peak_pte_request_to_eviction_ratio_limiting_single_display_no_rotation = + bw_int_to_fixed(25); + dceip.minimum_outstanding_pte_request_limit = bw_int_to_fixed( + 2); + dceip.maximum_total_outstanding_pte_requests_allowed_by_saw = + bw_int_to_fixed(128); + dceip.limit_excessive_outstanding_dmif_requests = true; + dceip.linear_mode_line_request_alternation_slice = + bw_int_to_fixed(64); + dceip.scatter_gather_lines_of_pte_prefetching_in_linear_mode = + 32; + dceip.display_write_back420_luma_mcifwr_buffer_size = 12288; + dceip.display_write_back420_chroma_mcifwr_buffer_size = 8192; + dceip.request_efficiency = bw_frc_to_fixed(8, 10); + dceip.dispclk_per_request = bw_int_to_fixed(2); + dceip.dispclk_ramping_factor = bw_frc_to_fixed(105, 100); + dceip.display_pipe_throughput_factor = bw_frc_to_fixed(105, 100); + dceip.scatter_gather_pte_request_rows_in_tiling_mode = 2; + dceip.mcifwr_all_surfaces_burst_time = bw_int_to_fixed(0); + break; case BW_CALCS_VERSION_STONEY: vbios.memory_type = bw_def_gddr5; vbios.dram_channel_width_in_bits = 64; @@ -2815,6 +2938,19 @@ static void populate_initial_data( data->bytes_per_pixel[num_displays + 4] = 4; break; } + } else if (pipe[i].stream->dst.width != 0 && + pipe[i].stream->dst.height != 0 && + pipe[i].stream->src.width != 0 && + pipe[i].stream->src.height != 0) { + data->src_width[num_displays + 4] = bw_int_to_fixed(pipe[i].stream->src.width); + data->pitch_in_pixels[num_displays + 4] = data->src_width[num_displays + 4]; + data->src_height[num_displays + 4] = bw_int_to_fixed(pipe[i].stream->src.height); + data->h_taps[num_displays + 4] = pipe[i].stream->src.width == pipe[i].stream->dst.width ? bw_int_to_fixed(1) : bw_int_to_fixed(2); + data->v_taps[num_displays + 4] = pipe[i].stream->src.height == pipe[i].stream->dst.height ? bw_int_to_fixed(1) : bw_int_to_fixed(2); + data->h_scale_ratio[num_displays + 4] = bw_frc_to_fixed(pipe[i].stream->src.width, pipe[i].stream->dst.width); + data->v_scale_ratio[num_displays + 4] = bw_frc_to_fixed(pipe[i].stream->src.height, pipe[i].stream->dst.height); + data->rotation_angle[num_displays + 4] = bw_int_to_fixed(0); + data->bytes_per_pixel[num_displays + 4] = 4; } else { data->src_width[num_displays + 4] = bw_int_to_fixed(pipe[i].stream->timing.h_addressable); data->pitch_in_pixels[num_displays + 4] = data->src_width[num_displays + 4]; @@ -2873,6 +3009,11 @@ bool bw_calcs(struct dc_context *ctx, struct bw_fixed mid_yclk = vbios->mid_yclk; struct bw_fixed low_yclk = vbios->low_yclk; + if (ctx->dc->debug.bandwidth_calcs_trace) { + print_bw_calcs_dceip(ctx->logger, dceip); + print_bw_calcs_vbios(ctx->logger, vbios); + print_bw_calcs_data(ctx->logger, data); + } calculate_bandwidth(dceip, vbios, data); yclk_lvl = data->y_clk_level; @@ -2968,7 +3109,33 @@ bool bw_calcs(struct dc_context *ctx, bw_fixed_to_int(bw_mul(data-> stutter_exit_watermark[9], bw_int_to_fixed(1000))); - + calcs_output->stutter_entry_wm_ns[0].a_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[4], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[1].a_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[5], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[2].a_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[6], bw_int_to_fixed(1000))); + if (ctx->dc->caps.max_slave_planes) { + calcs_output->stutter_entry_wm_ns[3].a_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[0], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[4].a_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[1], bw_int_to_fixed(1000))); + } else { + calcs_output->stutter_entry_wm_ns[3].a_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[7], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[4].a_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[8], bw_int_to_fixed(1000))); + } + calcs_output->stutter_entry_wm_ns[5].a_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[9], bw_int_to_fixed(1000))); calcs_output->urgent_wm_ns[0].a_mark = bw_fixed_to_int(bw_mul(data-> @@ -3063,7 +3230,33 @@ bool bw_calcs(struct dc_context *ctx, bw_fixed_to_int(bw_mul(data-> stutter_exit_watermark[9], bw_int_to_fixed(1000))); - + calcs_output->stutter_entry_wm_ns[0].b_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[4], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[1].b_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[5], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[2].b_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[6], bw_int_to_fixed(1000))); + if (ctx->dc->caps.max_slave_planes) { + calcs_output->stutter_entry_wm_ns[3].b_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[0], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[4].b_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[1], bw_int_to_fixed(1000))); + } else { + calcs_output->stutter_entry_wm_ns[3].b_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[7], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[4].b_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[8], bw_int_to_fixed(1000))); + } + calcs_output->stutter_entry_wm_ns[5].b_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[9], bw_int_to_fixed(1000))); calcs_output->urgent_wm_ns[0].b_mark = bw_fixed_to_int(bw_mul(data-> @@ -3156,6 +3349,34 @@ bool bw_calcs(struct dc_context *ctx, bw_fixed_to_int(bw_mul(data-> stutter_exit_watermark[9], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[0].c_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[4], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[1].c_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[5], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[2].c_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[6], bw_int_to_fixed(1000))); + if (ctx->dc->caps.max_slave_planes) { + calcs_output->stutter_entry_wm_ns[3].c_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[0], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[4].c_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[1], bw_int_to_fixed(1000))); + } else { + calcs_output->stutter_entry_wm_ns[3].c_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[7], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[4].c_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[8], bw_int_to_fixed(1000))); + } + calcs_output->stutter_entry_wm_ns[5].c_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[9], bw_int_to_fixed(1000))); + calcs_output->urgent_wm_ns[0].c_mark = bw_fixed_to_int(bw_mul(data-> urgent_watermark[4], bw_int_to_fixed(1000))); @@ -3260,6 +3481,33 @@ bool bw_calcs(struct dc_context *ctx, bw_fixed_to_int(bw_mul(data-> stutter_exit_watermark[9], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[0].d_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[4], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[1].d_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[5], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[2].d_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[6], bw_int_to_fixed(1000))); + if (ctx->dc->caps.max_slave_planes) { + calcs_output->stutter_entry_wm_ns[3].d_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[0], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[4].d_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[1], bw_int_to_fixed(1000))); + } else { + calcs_output->stutter_entry_wm_ns[3].d_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[7], bw_int_to_fixed(1000))); + calcs_output->stutter_entry_wm_ns[4].d_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[8], bw_int_to_fixed(1000))); + } + calcs_output->stutter_entry_wm_ns[5].d_mark = + bw_fixed_to_int(bw_mul(data-> + stutter_entry_watermark[9], bw_int_to_fixed(1000))); calcs_output->urgent_wm_ns[0].d_mark = bw_fixed_to_int(bw_mul(data-> diff --git a/drivers/gpu/drm/amd/display/dc/calcs/dcn_calcs.c b/drivers/gpu/drm/amd/display/dc/calcs/dcn_calcs.c index 4bb43a371292..49a4ea45466d 100644 --- a/drivers/gpu/drm/amd/display/dc/calcs/dcn_calcs.c +++ b/drivers/gpu/drm/amd/display/dc/calcs/dcn_calcs.c @@ -873,14 +873,14 @@ bool dcn_validate_bandwidth( } if (pipe->plane_state->rotation % 2 == 0) { - ASSERT(pipe->plane_res.scl_data.ratios.horz.value != dal_fixed31_32_one.value + ASSERT(pipe->plane_res.scl_data.ratios.horz.value != dc_fixpt_one.value || v->scaler_rec_out_width[input_idx] == v->viewport_width[input_idx]); - ASSERT(pipe->plane_res.scl_data.ratios.vert.value != dal_fixed31_32_one.value + ASSERT(pipe->plane_res.scl_data.ratios.vert.value != dc_fixpt_one.value || v->scaler_recout_height[input_idx] == v->viewport_height[input_idx]); } else { - ASSERT(pipe->plane_res.scl_data.ratios.horz.value != dal_fixed31_32_one.value + ASSERT(pipe->plane_res.scl_data.ratios.horz.value != dc_fixpt_one.value || v->scaler_recout_height[input_idx] == v->viewport_width[input_idx]); - ASSERT(pipe->plane_res.scl_data.ratios.vert.value != dal_fixed31_32_one.value + ASSERT(pipe->plane_res.scl_data.ratios.vert.value != dc_fixpt_one.value || v->scaler_rec_out_width[input_idx] == v->viewport_height[input_idx]); } v->dcc_enable[input_idx] = pipe->plane_state->dcc.enable ? dcn_bw_yes : dcn_bw_no; @@ -1459,39 +1459,39 @@ void dcn_bw_notify_pplib_of_wm_ranges(struct dc *dc) void dcn_bw_sync_calcs_and_dml(struct dc *dc) { kernel_fpu_begin(); - DC_LOG_BANDWIDTH_CALCS("sr_exit_time: %d ns\n" - "sr_enter_plus_exit_time: %d ns\n" - "urgent_latency: %d ns\n" - "write_back_latency: %d ns\n" - "percent_of_ideal_drambw_received_after_urg_latency: %d %\n" + DC_LOG_BANDWIDTH_CALCS("sr_exit_time: %f ns\n" + "sr_enter_plus_exit_time: %f ns\n" + "urgent_latency: %f ns\n" + "write_back_latency: %f ns\n" + "percent_of_ideal_drambw_received_after_urg_latency: %f %%\n" "max_request_size: %d bytes\n" - "dcfclkv_max0p9: %d kHz\n" - "dcfclkv_nom0p8: %d kHz\n" - "dcfclkv_mid0p72: %d kHz\n" - "dcfclkv_min0p65: %d kHz\n" - "max_dispclk_vmax0p9: %d kHz\n" - "max_dispclk_vnom0p8: %d kHz\n" - "max_dispclk_vmid0p72: %d kHz\n" - "max_dispclk_vmin0p65: %d kHz\n" - "max_dppclk_vmax0p9: %d kHz\n" - "max_dppclk_vnom0p8: %d kHz\n" - "max_dppclk_vmid0p72: %d kHz\n" - "max_dppclk_vmin0p65: %d kHz\n" - "socclk: %d kHz\n" - "fabric_and_dram_bandwidth_vmax0p9: %d MB/s\n" - "fabric_and_dram_bandwidth_vnom0p8: %d MB/s\n" - "fabric_and_dram_bandwidth_vmid0p72: %d MB/s\n" - "fabric_and_dram_bandwidth_vmin0p65: %d MB/s\n" - "phyclkv_max0p9: %d kHz\n" - "phyclkv_nom0p8: %d kHz\n" - "phyclkv_mid0p72: %d kHz\n" - "phyclkv_min0p65: %d kHz\n" - "downspreading: %d %\n" + "dcfclkv_max0p9: %f kHz\n" + "dcfclkv_nom0p8: %f kHz\n" + "dcfclkv_mid0p72: %f kHz\n" + "dcfclkv_min0p65: %f kHz\n" + "max_dispclk_vmax0p9: %f kHz\n" + "max_dispclk_vnom0p8: %f kHz\n" + "max_dispclk_vmid0p72: %f kHz\n" + "max_dispclk_vmin0p65: %f kHz\n" + "max_dppclk_vmax0p9: %f kHz\n" + "max_dppclk_vnom0p8: %f kHz\n" + "max_dppclk_vmid0p72: %f kHz\n" + "max_dppclk_vmin0p65: %f kHz\n" + "socclk: %f kHz\n" + "fabric_and_dram_bandwidth_vmax0p9: %f MB/s\n" + "fabric_and_dram_bandwidth_vnom0p8: %f MB/s\n" + "fabric_and_dram_bandwidth_vmid0p72: %f MB/s\n" + "fabric_and_dram_bandwidth_vmin0p65: %f MB/s\n" + "phyclkv_max0p9: %f kHz\n" + "phyclkv_nom0p8: %f kHz\n" + "phyclkv_mid0p72: %f kHz\n" + "phyclkv_min0p65: %f kHz\n" + "downspreading: %f %%\n" "round_trip_ping_latency_cycles: %d DCFCLK Cycles\n" "urgent_out_of_order_return_per_channel: %d Bytes\n" "number_of_channels: %d\n" "vmm_page_size: %d Bytes\n" - "dram_clock_change_latency: %d ns\n" + "dram_clock_change_latency: %f ns\n" "return_bus_width: %d Bytes\n", dc->dcn_soc->sr_exit_time * 1000, dc->dcn_soc->sr_enter_plus_exit_time * 1000, @@ -1527,11 +1527,11 @@ void dcn_bw_sync_calcs_and_dml(struct dc *dc) dc->dcn_soc->vmm_page_size, dc->dcn_soc->dram_clock_change_latency * 1000, dc->dcn_soc->return_bus_width); - DC_LOG_BANDWIDTH_CALCS("rob_buffer_size_in_kbyte: %d\n" - "det_buffer_size_in_kbyte: %d\n" - "dpp_output_buffer_pixels: %d\n" - "opp_output_buffer_lines: %d\n" - "pixel_chunk_size_in_kbyte: %d\n" + DC_LOG_BANDWIDTH_CALCS("rob_buffer_size_in_kbyte: %f\n" + "det_buffer_size_in_kbyte: %f\n" + "dpp_output_buffer_pixels: %f\n" + "opp_output_buffer_lines: %f\n" + "pixel_chunk_size_in_kbyte: %f\n" "pte_enable: %d\n" "pte_chunk_size: %d kbytes\n" "meta_chunk_size: %d kbytes\n" @@ -1550,13 +1550,13 @@ void dcn_bw_sync_calcs_and_dml(struct dc *dc) "max_pscl_tolb_throughput: %d pixels/dppclk\n" "max_lb_tovscl_throughput: %d pixels/dppclk\n" "max_vscl_tohscl_throughput: %d pixels/dppclk\n" - "max_hscl_ratio: %d\n" - "max_vscl_ratio: %d\n" + "max_hscl_ratio: %f\n" + "max_vscl_ratio: %f\n" "max_hscl_taps: %d\n" "max_vscl_taps: %d\n" "pte_buffer_size_in_requests: %d\n" - "dispclk_ramping_margin: %d %\n" - "under_scan_factor: %d %\n" + "dispclk_ramping_margin: %f %%\n" + "under_scan_factor: %f %%\n" "max_inter_dcn_tile_repeaters: %d\n" "can_vstartup_lines_exceed_vsync_plus_back_porch_lines_minus_one: %d\n" "bug_forcing_luma_and_chroma_request_to_same_size_fixed: %d\n" diff --git a/drivers/gpu/drm/amd/display/dc/core/dc.c b/drivers/gpu/drm/amd/display/dc/core/dc.c index 9cd3566def8d..644b2187507b 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc.c @@ -936,95 +936,6 @@ bool dc_post_update_surfaces_to_stream(struct dc *dc) return true; } -/* - * TODO this whole function needs to go - * - * dc_surface_update is needlessly complex. See if we can just replace this - * with a dc_plane_state and follow the atomic model a bit more closely here. - */ -bool dc_commit_planes_to_stream( - struct dc *dc, - struct dc_plane_state **plane_states, - uint8_t new_plane_count, - struct dc_stream_state *dc_stream, - struct dc_state *state) -{ - /* no need to dynamically allocate this. it's pretty small */ - struct dc_surface_update updates[MAX_SURFACES]; - struct dc_flip_addrs *flip_addr; - struct dc_plane_info *plane_info; - struct dc_scaling_info *scaling_info; - int i; - struct dc_stream_update *stream_update = - kzalloc(sizeof(struct dc_stream_update), GFP_KERNEL); - - if (!stream_update) { - BREAK_TO_DEBUGGER(); - return false; - } - - flip_addr = kcalloc(MAX_SURFACES, sizeof(struct dc_flip_addrs), - GFP_KERNEL); - plane_info = kcalloc(MAX_SURFACES, sizeof(struct dc_plane_info), - GFP_KERNEL); - scaling_info = kcalloc(MAX_SURFACES, sizeof(struct dc_scaling_info), - GFP_KERNEL); - - if (!flip_addr || !plane_info || !scaling_info) { - kfree(flip_addr); - kfree(plane_info); - kfree(scaling_info); - kfree(stream_update); - return false; - } - - memset(updates, 0, sizeof(updates)); - - stream_update->src = dc_stream->src; - stream_update->dst = dc_stream->dst; - stream_update->out_transfer_func = dc_stream->out_transfer_func; - - for (i = 0; i < new_plane_count; i++) { - updates[i].surface = plane_states[i]; - updates[i].gamma = - (struct dc_gamma *)plane_states[i]->gamma_correction; - updates[i].in_transfer_func = plane_states[i]->in_transfer_func; - flip_addr[i].address = plane_states[i]->address; - flip_addr[i].flip_immediate = plane_states[i]->flip_immediate; - plane_info[i].color_space = plane_states[i]->color_space; - plane_info[i].input_tf = plane_states[i]->input_tf; - plane_info[i].format = plane_states[i]->format; - plane_info[i].plane_size = plane_states[i]->plane_size; - plane_info[i].rotation = plane_states[i]->rotation; - plane_info[i].horizontal_mirror = plane_states[i]->horizontal_mirror; - plane_info[i].stereo_format = plane_states[i]->stereo_format; - plane_info[i].tiling_info = plane_states[i]->tiling_info; - plane_info[i].visible = plane_states[i]->visible; - plane_info[i].per_pixel_alpha = plane_states[i]->per_pixel_alpha; - plane_info[i].dcc = plane_states[i]->dcc; - scaling_info[i].scaling_quality = plane_states[i]->scaling_quality; - scaling_info[i].src_rect = plane_states[i]->src_rect; - scaling_info[i].dst_rect = plane_states[i]->dst_rect; - scaling_info[i].clip_rect = plane_states[i]->clip_rect; - - updates[i].flip_addr = &flip_addr[i]; - updates[i].plane_info = &plane_info[i]; - updates[i].scaling_info = &scaling_info[i]; - } - - dc_commit_updates_for_stream( - dc, - updates, - new_plane_count, - dc_stream, stream_update, plane_states, state); - - kfree(flip_addr); - kfree(plane_info); - kfree(scaling_info); - kfree(stream_update); - return true; -} - struct dc_state *dc_create_state(void) { struct dc_state *context = kzalloc(sizeof(struct dc_state), @@ -1107,9 +1018,6 @@ static enum surface_update_type get_plane_info_update_type(const struct dc_surfa if (u->plane_info->color_space != u->surface->color_space) update_flags->bits.color_space_change = 1; - if (u->plane_info->input_tf != u->surface->input_tf) - update_flags->bits.input_tf_change = 1; - if (u->plane_info->horizontal_mirror != u->surface->horizontal_mirror) update_flags->bits.horizontal_mirror_change = 1; @@ -1243,12 +1151,20 @@ static enum surface_update_type det_surface_update(const struct dc *dc, if (u->input_csc_color_matrix) update_flags->bits.input_csc_change = 1; - if (update_flags->bits.in_transfer_func_change - || update_flags->bits.input_csc_change) { + if (u->coeff_reduction_factor) + update_flags->bits.coeff_reduction_change = 1; + + if (update_flags->bits.in_transfer_func_change) { type = UPDATE_TYPE_MED; elevate_update_type(&overall_type, type); } + if (update_flags->bits.input_csc_change + || update_flags->bits.coeff_reduction_change) { + type = UPDATE_TYPE_FULL; + elevate_update_type(&overall_type, type); + } + return overall_type; } @@ -1297,7 +1213,7 @@ enum surface_update_type dc_check_update_surfaces_for_stream( type = check_update_surfaces_for_stream(dc, updates, surface_count, stream_update, stream_status); if (type == UPDATE_TYPE_FULL) for (i = 0; i < surface_count; i++) - updates[i].surface->update_flags.bits.full_update = 1; + updates[i].surface->update_flags.raw = 0xFFFFFFFF; return type; } @@ -1375,6 +1291,12 @@ static void commit_planes_for_stream(struct dc *dc, pipe_ctx->stream_res.abm->funcs->set_abm_level( pipe_ctx->stream_res.abm, stream->abm_level); } + + if (stream_update && stream_update->periodic_fn_vsync_delta && + pipe_ctx->stream_res.tg->funcs->program_vline_interrupt) + pipe_ctx->stream_res.tg->funcs->program_vline_interrupt( + pipe_ctx->stream_res.tg, &pipe_ctx->stream->timing, + pipe_ctx->stream->periodic_fn_vsync_delta); } } diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_debug.c b/drivers/gpu/drm/amd/display/dc/core/dc_debug.c index 5a552cb3f8a7..267c76766dea 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc_debug.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc_debug.c @@ -36,8 +36,9 @@ #include "hw_sequencer.h" #include "resource.h" -#define DC_LOGGER \ - logger + +#define DC_LOGGER_INIT(logger) + #define SURFACE_TRACE(...) do {\ if (dc->debug.surface_trace) \ @@ -60,8 +61,7 @@ void pre_surface_trace( int surface_count) { int i; - struct dc *core_dc = dc; - struct dal_logger *logger = core_dc->ctx->logger; + DC_LOGGER_INIT(dc->ctx->logger); for (i = 0; i < surface_count; i++) { const struct dc_plane_state *plane_state = plane_states[i]; @@ -72,8 +72,8 @@ void pre_surface_trace( "plane_state->visible = %d;\n" "plane_state->flip_immediate = %d;\n" "plane_state->address.type = %d;\n" - "plane_state->address.grph.addr.quad_part = 0x%X;\n" - "plane_state->address.grph.meta_addr.quad_part = 0x%X;\n" + "plane_state->address.grph.addr.quad_part = 0x%llX;\n" + "plane_state->address.grph.meta_addr.quad_part = 0x%llX;\n" "plane_state->scaling_quality.h_taps = %d;\n" "plane_state->scaling_quality.v_taps = %d;\n" "plane_state->scaling_quality.h_taps_c = %d;\n" @@ -155,7 +155,6 @@ void pre_surface_trace( "plane_state->tiling_info.gfx8.pipe_config = %d;\n" "plane_state->tiling_info.gfx8.array_mode = %d;\n" "plane_state->color_space = %d;\n" - "plane_state->input_tf = %d;\n" "plane_state->dcc.enable = %d;\n" "plane_state->format = %d;\n" "plane_state->rotation = %d;\n" @@ -163,7 +162,6 @@ void pre_surface_trace( plane_state->tiling_info.gfx8.pipe_config, plane_state->tiling_info.gfx8.array_mode, plane_state->color_space, - plane_state->input_tf, plane_state->dcc.enable, plane_state->format, plane_state->rotation, @@ -183,8 +181,7 @@ void update_surface_trace( int surface_count) { int i; - struct dc *core_dc = dc; - struct dal_logger *logger = core_dc->ctx->logger; + DC_LOGGER_INIT(dc->ctx->logger); for (i = 0; i < surface_count; i++) { const struct dc_surface_update *update = &updates[i]; @@ -192,8 +189,8 @@ void update_surface_trace( SURFACE_TRACE("Update %d\n", i); if (update->flip_addr) { SURFACE_TRACE("flip_addr->address.type = %d;\n" - "flip_addr->address.grph.addr.quad_part = 0x%X;\n" - "flip_addr->address.grph.meta_addr.quad_part = 0x%X;\n" + "flip_addr->address.grph.addr.quad_part = 0x%llX;\n" + "flip_addr->address.grph.meta_addr.quad_part = 0x%llX;\n" "flip_addr->flip_immediate = %d;\n", update->flip_addr->address.type, update->flip_addr->address.grph.addr.quad_part, @@ -204,16 +201,15 @@ void update_surface_trace( if (update->plane_info) { SURFACE_TRACE( "plane_info->color_space = %d;\n" - "plane_info->input_tf = %d;\n" "plane_info->format = %d;\n" "plane_info->plane_size.grph.surface_pitch = %d;\n" "plane_info->plane_size.grph.surface_size.height = %d;\n" "plane_info->plane_size.grph.surface_size.width = %d;\n" "plane_info->plane_size.grph.surface_size.x = %d;\n" "plane_info->plane_size.grph.surface_size.y = %d;\n" - "plane_info->rotation = %d;\n", + "plane_info->rotation = %d;\n" + "plane_info->stereo_format = %d;\n", update->plane_info->color_space, - update->plane_info->input_tf, update->plane_info->format, update->plane_info->plane_size.grph.surface_pitch, update->plane_info->plane_size.grph.surface_size.height, @@ -303,8 +299,7 @@ void update_surface_trace( void post_surface_trace(struct dc *dc) { - struct dc *core_dc = dc; - struct dal_logger *logger = core_dc->ctx->logger; + DC_LOGGER_INIT(dc->ctx->logger); SURFACE_TRACE("post surface process.\n"); @@ -316,10 +311,10 @@ void context_timing_trace( { int i; struct dc *core_dc = dc; - struct dal_logger *logger = core_dc->ctx->logger; int h_pos[MAX_PIPES], v_pos[MAX_PIPES]; struct crtc_position position; unsigned int underlay_idx = core_dc->res_pool->underlay_pipe_index; + DC_LOGGER_INIT(dc->ctx->logger); for (i = 0; i < core_dc->res_pool->pipe_count; i++) { @@ -354,9 +349,7 @@ void context_clock_trace( struct dc_state *context) { #if defined(CONFIG_DRM_AMD_DC_DCN1_0) - struct dc *core_dc = dc; - struct dal_logger *logger = core_dc->ctx->logger; - + DC_LOGGER_INIT(dc->ctx->logger); CLOCK_TRACE("Current: dispclk_khz:%d max_dppclk_khz:%d dcfclk_khz:%d\n" "dcfclk_deep_sleep_khz:%d fclk_khz:%d socclk_khz:%d\n", context->bw.dcn.calc_clk.dispclk_khz, @@ -371,6 +364,7 @@ void context_clock_trace( context->bw.dcn.calc_clk.dppclk_khz, context->bw.dcn.calc_clk.dcfclk_khz, context->bw.dcn.calc_clk.dcfclk_deep_sleep_khz, - context->bw.dcn.calc_clk.fclk_khz); + context->bw.dcn.calc_clk.fclk_khz, + context->bw.dcn.calc_clk.socclk_khz); #endif } diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c b/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c index ebc96b720083..83d121510ef5 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c @@ -208,6 +208,7 @@ void color_space_to_black_color( case COLOR_SPACE_YCBCR709: case COLOR_SPACE_YCBCR601_LIMITED: case COLOR_SPACE_YCBCR709_LIMITED: + case COLOR_SPACE_2020_YCBCR: *black_color = black_color_format[BLACK_COLOR_FORMAT_YUV_CV]; break; @@ -216,7 +217,25 @@ void color_space_to_black_color( black_color_format[BLACK_COLOR_FORMAT_RGB_LIMITED]; break; - default: + /** + * Remove default and add case for all color space + * so when we forget to add new color space + * compiler will give a warning + */ + case COLOR_SPACE_UNKNOWN: + case COLOR_SPACE_SRGB: + case COLOR_SPACE_XR_RGB: + case COLOR_SPACE_MSREF_SCRGB: + case COLOR_SPACE_XV_YCC_709: + case COLOR_SPACE_XV_YCC_601: + case COLOR_SPACE_2020_RGB_FULLRANGE: + case COLOR_SPACE_2020_RGB_LIMITEDRANGE: + case COLOR_SPACE_ADOBERGB: + case COLOR_SPACE_DCIP3: + case COLOR_SPACE_DISPLAYNATIVE: + case COLOR_SPACE_DOLBYVISION: + case COLOR_SPACE_APPCTRL: + case COLOR_SPACE_CUSTOMPOINTS: /* fefault is sRGB black (full range). */ *black_color = black_color_format[BLACK_COLOR_FORMAT_RGB_FULLRANGE]; @@ -230,6 +249,9 @@ bool hwss_wait_for_blank_complete( { int counter; + /* Not applicable if the pipe is not primary, save 300ms of boot time */ + if (!tg->funcs->is_blanked) + return true; for (counter = 0; counter < 100; counter++) { if (tg->funcs->is_blanked(tg)) break; diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_link.c b/drivers/gpu/drm/amd/display/dc/core/dc_link.c index 6d1c4981a185..2fa521812d23 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc_link.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc_link.c @@ -45,8 +45,9 @@ #include "dce/dce_11_0_d.h" #include "dce/dce_11_0_enum.h" #include "dce/dce_11_0_sh_mask.h" -#define DC_LOGGER \ - dc_ctx->logger + +#define DC_LOGGER_INIT(logger) + #define LINK_INFO(...) \ DC_LOG_HW_HOTPLUG( \ @@ -468,6 +469,13 @@ static void link_disconnect_sink(struct dc_link *link) link->dpcd_sink_count = 0; } +static void link_disconnect_remap(struct dc_sink *prev_sink, struct dc_link *link) +{ + dc_sink_release(link->local_sink); + link->local_sink = prev_sink; +} + + static bool detect_dp( struct dc_link *link, struct display_sink_capability *sink_caps, @@ -550,6 +558,17 @@ static bool detect_dp( return true; } +static bool is_same_edid(struct dc_edid *old_edid, struct dc_edid *new_edid) +{ + if (old_edid->length != new_edid->length) + return false; + + if (new_edid->length == 0) + return false; + + return (memcmp(old_edid->raw_edid, new_edid->raw_edid, new_edid->length) == 0); +} + bool dc_link_detect(struct dc_link *link, enum dc_detect_reason reason) { struct dc_sink_init_data sink_init_data = { 0 }; @@ -557,11 +576,15 @@ bool dc_link_detect(struct dc_link *link, enum dc_detect_reason reason) uint8_t i; bool converter_disable_audio = false; struct audio_support *aud_support = &link->dc->res_pool->audio_support; + bool same_edid = false; enum dc_edid_status edid_status; struct dc_context *dc_ctx = link->ctx; struct dc_sink *sink = NULL; + struct dc_sink *prev_sink = NULL; + struct dpcd_caps prev_dpcd_caps; + bool same_dpcd = true; enum dc_connection_type new_connection_type = dc_connection_none; - + DC_LOGGER_INIT(link->ctx->logger); if (link->connector_signal == SIGNAL_TYPE_VIRTUAL) return false; @@ -574,6 +597,11 @@ bool dc_link_detect(struct dc_link *link, enum dc_detect_reason reason) link->local_sink) return true; + prev_sink = link->local_sink; + if (prev_sink != NULL) { + dc_sink_retain(prev_sink); + memcpy(&prev_dpcd_caps, &link->dpcd_caps, sizeof(struct dpcd_caps)); + } link_disconnect_sink(link); if (new_connection_type != dc_connection_none) { @@ -615,14 +643,25 @@ bool dc_link_detect(struct dc_link *link, enum dc_detect_reason reason) link, &sink_caps, &converter_disable_audio, - aud_support, reason)) + aud_support, reason)) { + if (prev_sink != NULL) + dc_sink_release(prev_sink); return false; + } + // Check if dpcp block is the same + if (prev_sink != NULL) { + if (memcmp(&link->dpcd_caps, &prev_dpcd_caps, sizeof(struct dpcd_caps))) + same_dpcd = false; + } /* Active dongle downstream unplug */ if (link->type == dc_connection_active_dongle && link->dpcd_caps.sink_count. - bits.SINK_COUNT == 0) + bits.SINK_COUNT == 0) { + if (prev_sink != NULL) + dc_sink_release(prev_sink); return true; + } if (link->type == dc_connection_mst_branch) { LINK_INFO("link=%d, mst branch is now Connected\n", @@ -630,9 +669,11 @@ bool dc_link_detect(struct dc_link *link, enum dc_detect_reason reason) /* Need to setup mst link_cap struct here * otherwise dc_link_detect() will leave mst link_cap * empty which leads to allocate_mst_payload() has "0" - * pbn_per_slot value leading to exception on dal_fixed31_32_div() + * pbn_per_slot value leading to exception on dc_fixpt_div() */ link->verified_link_cap = link->reported_link_cap; + if (prev_sink != NULL) + dc_sink_release(prev_sink); return false; } @@ -642,6 +683,8 @@ bool dc_link_detect(struct dc_link *link, enum dc_detect_reason reason) default: DC_ERROR("Invalid connector type! signal:%d\n", link->connector_signal); + if (prev_sink != NULL) + dc_sink_release(prev_sink); return false; } /* switch() */ @@ -664,6 +707,8 @@ bool dc_link_detect(struct dc_link *link, enum dc_detect_reason reason) sink = dc_sink_create(&sink_init_data); if (!sink) { DC_ERROR("Failed to create sink!\n"); + if (prev_sink != NULL) + dc_sink_release(prev_sink); return false; } @@ -687,22 +732,33 @@ bool dc_link_detect(struct dc_link *link, enum dc_detect_reason reason) break; } - if (link->connector_signal == SIGNAL_TYPE_DISPLAY_PORT && - sink_caps.transaction_type == - DDC_TRANSACTION_TYPE_I2C_OVER_AUX) { - /* - * TODO debug why Dell 2413 doesn't like - * two link trainings - */ + // Check if edid is the same + if ((prev_sink != NULL) && ((edid_status == EDID_THE_SAME) || (edid_status == EDID_OK))) + same_edid = is_same_edid(&prev_sink->dc_edid, &sink->dc_edid); - /* deal with non-mst cases */ - dp_hbr_verify_link_cap(link, &link->reported_link_cap); - } + // If both edid and dpcd are the same, then discard new sink and revert back to original sink + if ((same_edid) && (same_dpcd)) { + link_disconnect_remap(prev_sink, link); + sink = prev_sink; + prev_sink = NULL; + } else { + if (link->connector_signal == SIGNAL_TYPE_DISPLAY_PORT && + sink_caps.transaction_type == + DDC_TRANSACTION_TYPE_I2C_OVER_AUX) { + /* + * TODO debug why Dell 2413 doesn't like + * two link trainings + */ - /* HDMI-DVI Dongle */ - if (sink->sink_signal == SIGNAL_TYPE_HDMI_TYPE_A && - !sink->edid_caps.edid_hdmi) - sink->sink_signal = SIGNAL_TYPE_DVI_SINGLE_LINK; + /* deal with non-mst cases */ + dp_hbr_verify_link_cap(link, &link->reported_link_cap); + } + + /* HDMI-DVI Dongle */ + if (sink->sink_signal == SIGNAL_TYPE_HDMI_TYPE_A && + !sink->edid_caps.edid_hdmi) + sink->sink_signal = SIGNAL_TYPE_DVI_SINGLE_LINK; + } /* Connectivity log: detection */ for (i = 0; i < sink->dc_edid.length / EDID_BLOCK_SIZE; i++) { @@ -761,10 +817,14 @@ bool dc_link_detect(struct dc_link *link, enum dc_detect_reason reason) sink_caps.signal = SIGNAL_TYPE_NONE; } - LINK_INFO("link=%d, dc_sink_in=%p is now %s\n", + LINK_INFO("link=%d, dc_sink_in=%p is now %s prev_sink=%p dpcd same=%d edid same=%d\n", link->link_index, sink, (sink_caps.signal == SIGNAL_TYPE_NONE ? - "Disconnected":"Connected")); + "Disconnected":"Connected"), prev_sink, + same_dpcd, same_edid); + + if (prev_sink != NULL) + dc_sink_release(prev_sink); return true; } @@ -927,6 +987,7 @@ static bool construct( struct integrated_info info = {{{ 0 }}}; struct dc_bios *bios = init_params->dc->ctx->dc_bios; const struct dc_vbios_funcs *bp_funcs = bios->funcs; + DC_LOGGER_INIT(dc_ctx->logger); link->irq_source_hpd = DC_IRQ_SOURCE_INVALID; link->irq_source_hpd_rx = DC_IRQ_SOURCE_INVALID; @@ -1135,7 +1196,8 @@ static void dpcd_configure_panel_mode( { union dpcd_edp_config edp_config_set; bool panel_mode_edp = false; - struct dc_context *dc_ctx = link->ctx; + DC_LOGGER_INIT(link->ctx->logger); + memset(&edp_config_set, '\0', sizeof(union dpcd_edp_config)); if (DP_PANEL_MODE_DEFAULT != panel_mode) { @@ -1183,16 +1245,21 @@ static void enable_stream_features(struct pipe_ctx *pipe_ctx) { struct dc_stream_state *stream = pipe_ctx->stream; struct dc_link *link = stream->sink->link; - union down_spread_ctrl downspread; + union down_spread_ctrl old_downspread; + union down_spread_ctrl new_downspread; core_link_read_dpcd(link, DP_DOWNSPREAD_CTRL, - &downspread.raw, sizeof(downspread)); + &old_downspread.raw, sizeof(old_downspread)); - downspread.bits.IGNORE_MSA_TIMING_PARAM = + new_downspread.raw = old_downspread.raw; + + new_downspread.bits.IGNORE_MSA_TIMING_PARAM = (stream->ignore_msa_timing_param) ? 1 : 0; - core_link_write_dpcd(link, DP_DOWNSPREAD_CTRL, - &downspread.raw, sizeof(downspread)); + if (new_downspread.raw != old_downspread.raw) { + core_link_write_dpcd(link, DP_DOWNSPREAD_CTRL, + &new_downspread.raw, sizeof(new_downspread)); + } } static enum dc_status enable_link_dp( @@ -1843,9 +1910,22 @@ static void disable_link(struct dc_link *link, enum signal_type signal) static bool dp_active_dongle_validate_timing( const struct dc_crtc_timing *timing, - const struct dc_dongle_caps *dongle_caps) + const struct dpcd_caps *dpcd_caps) { unsigned int required_pix_clk = timing->pix_clk_khz; + const struct dc_dongle_caps *dongle_caps = &dpcd_caps->dongle_caps; + + switch (dpcd_caps->dongle_type) { + case DISPLAY_DONGLE_DP_VGA_CONVERTER: + case DISPLAY_DONGLE_DP_DVI_CONVERTER: + case DISPLAY_DONGLE_DP_DVI_DONGLE: + if (timing->pixel_encoding == PIXEL_ENCODING_RGB) + return true; + else + return false; + default: + break; + } if (dongle_caps->dongle_type != DISPLAY_DONGLE_DP_HDMI_CONVERTER || dongle_caps->extendedCapValid == false) @@ -1911,7 +1991,7 @@ enum dc_status dc_link_validate_mode_timing( const struct dc_crtc_timing *timing) { uint32_t max_pix_clk = stream->sink->dongle_max_pix_clk; - struct dc_dongle_caps *dongle_caps = &link->dpcd_caps.dongle_caps; + struct dpcd_caps *dpcd_caps = &link->dpcd_caps; /* A hack to avoid failing any modes for EDID override feature on * topology change such as lower quality cable for DP or different dongle @@ -1924,7 +2004,7 @@ enum dc_status dc_link_validate_mode_timing( return DC_EXCEED_DONGLE_CAP; /* Active Dongle*/ - if (!dp_active_dongle_validate_timing(timing, dongle_caps)) + if (!dp_active_dongle_validate_timing(timing, dpcd_caps)) return DC_EXCEED_DONGLE_CAP; switch (stream->signal) { @@ -1950,10 +2030,10 @@ bool dc_link_set_backlight_level(const struct dc_link *link, uint32_t level, struct dc *core_dc = link->ctx->dc; struct abm *abm = core_dc->res_pool->abm; struct dmcu *dmcu = core_dc->res_pool->dmcu; - struct dc_context *dc_ctx = link->ctx; unsigned int controller_id = 0; bool use_smooth_brightness = true; int i; + DC_LOGGER_INIT(link->ctx->logger); if ((dmcu == NULL) || (abm == NULL) || @@ -1961,7 +2041,7 @@ bool dc_link_set_backlight_level(const struct dc_link *link, uint32_t level, return false; if (stream) { - if (stream->bl_pwm_level == 0) + if (stream->bl_pwm_level == EDP_BACKLIGHT_RAMP_DISABLE_LEVEL) frame_ramp = 0; ((struct dc_stream_state *)stream)->bl_pwm_level = level; @@ -2038,10 +2118,10 @@ static struct fixed31_32 get_pbn_per_slot(struct dc_stream_state *stream) &stream->sink->link->cur_link_settings; uint32_t link_rate_in_mbps = link_settings->link_rate * LINK_RATE_REF_FREQ_IN_MHZ; - struct fixed31_32 mbps = dal_fixed31_32_from_int( + struct fixed31_32 mbps = dc_fixpt_from_int( link_rate_in_mbps * link_settings->lane_count); - return dal_fixed31_32_div_int(mbps, 54); + return dc_fixpt_div_int(mbps, 54); } static int get_color_depth(enum dc_color_depth color_depth) @@ -2082,7 +2162,7 @@ static struct fixed31_32 get_pbn_from_timing(struct pipe_ctx *pipe_ctx) numerator = 64 * PEAK_FACTOR_X1000; denominator = 54 * 8 * 1000 * 1000; kbps *= numerator; - peak_kbps = dal_fixed31_32_from_fraction(kbps, denominator); + peak_kbps = dc_fixpt_from_fraction(kbps, denominator); return peak_kbps; } @@ -2149,8 +2229,8 @@ static enum dc_status allocate_mst_payload(struct pipe_ctx *pipe_ctx) struct fixed31_32 avg_time_slots_per_mtp; struct fixed31_32 pbn; struct fixed31_32 pbn_per_slot; - struct dc_context *dc_ctx = link->ctx; uint8_t i; + DC_LOGGER_INIT(link->ctx->logger); /* enable_link_dp_mst already check link->enabled_stream_count * and stream is in link->stream[]. This is called during set mode, @@ -2178,11 +2258,11 @@ static enum dc_status allocate_mst_payload(struct pipe_ctx *pipe_ctx) link->mst_stream_alloc_table.stream_count); for (i = 0; i < MAX_CONTROLLER_NUM; i++) { - DC_LOG_MST("stream_enc[%d]: 0x%x " + DC_LOG_MST("stream_enc[%d]: %p " "stream[%d].vcp_id: %d " "stream[%d].slot_count: %d\n", i, - link->mst_stream_alloc_table.stream_allocations[i].stream_enc, + (void *) link->mst_stream_alloc_table.stream_allocations[i].stream_enc, i, link->mst_stream_alloc_table.stream_allocations[i].vcp_id, i, @@ -2209,7 +2289,7 @@ static enum dc_status allocate_mst_payload(struct pipe_ctx *pipe_ctx) /* slot X.Y for only current stream */ pbn_per_slot = get_pbn_per_slot(stream); pbn = get_pbn_from_timing(pipe_ctx); - avg_time_slots_per_mtp = dal_fixed31_32_div(pbn, pbn_per_slot); + avg_time_slots_per_mtp = dc_fixpt_div(pbn, pbn_per_slot); stream_encoder->funcs->set_mst_bandwidth( stream_encoder, @@ -2226,10 +2306,10 @@ static enum dc_status deallocate_mst_payload(struct pipe_ctx *pipe_ctx) struct link_encoder *link_encoder = link->link_enc; struct stream_encoder *stream_encoder = pipe_ctx->stream_res.stream_enc; struct dp_mst_stream_allocation_table proposed_table = {0}; - struct fixed31_32 avg_time_slots_per_mtp = dal_fixed31_32_from_int(0); + struct fixed31_32 avg_time_slots_per_mtp = dc_fixpt_from_int(0); uint8_t i; bool mst_mode = (link->type == dc_connection_mst_branch); - struct dc_context *dc_ctx = link->ctx; + DC_LOGGER_INIT(link->ctx->logger); /* deallocate_mst_payload is called before disable link. When mode or * disable/enable monitor, new stream is created which is not in link @@ -2268,11 +2348,11 @@ static enum dc_status deallocate_mst_payload(struct pipe_ctx *pipe_ctx) link->mst_stream_alloc_table.stream_count); for (i = 0; i < MAX_CONTROLLER_NUM; i++) { - DC_LOG_MST("stream_enc[%d]: 0x%x " + DC_LOG_MST("stream_enc[%d]: %p " "stream[%d].vcp_id: %d " "stream[%d].slot_count: %d\n", i, - link->mst_stream_alloc_table.stream_allocations[i].stream_enc, + (void *) link->mst_stream_alloc_table.stream_allocations[i].stream_enc, i, link->mst_stream_alloc_table.stream_allocations[i].vcp_id, i, @@ -2302,8 +2382,8 @@ void core_link_enable_stream( struct pipe_ctx *pipe_ctx) { struct dc *core_dc = pipe_ctx->stream->ctx->dc; - struct dc_context *dc_ctx = pipe_ctx->stream->ctx; enum dc_status status; + DC_LOGGER_INIT(pipe_ctx->stream->ctx->logger); /* eDP lit up by bios already, no need to enable again. */ if (pipe_ctx->stream->signal == SIGNAL_TYPE_EDP && diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_link_dp.c b/drivers/gpu/drm/amd/display/dc/core/dc_link_dp.c index 3b5053570229..7857cb42b3e6 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc_link_dp.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc_link_dp.c @@ -1378,8 +1378,8 @@ static uint32_t bandwidth_in_kbps_from_timing( { uint32_t bits_per_channel = 0; uint32_t kbps; - switch (timing->display_color_depth) { + switch (timing->display_color_depth) { case COLOR_DEPTH_666: bits_per_channel = 6; break; @@ -1401,14 +1401,20 @@ static uint32_t bandwidth_in_kbps_from_timing( default: break; } + ASSERT(bits_per_channel != 0); kbps = timing->pix_clk_khz; kbps *= bits_per_channel; - if (timing->flags.Y_ONLY != 1) + if (timing->flags.Y_ONLY != 1) { /*Only YOnly make reduce bandwidth by 1/3 compares to RGB*/ kbps *= 3; + if (timing->pixel_encoding == PIXEL_ENCODING_YCBCR420) + kbps /= 2; + else if (timing->pixel_encoding == PIXEL_ENCODING_YCBCR422) + kbps = kbps * 2 / 3; + } return kbps; @@ -1624,17 +1630,42 @@ static enum dc_status read_hpd_rx_irq_data( struct dc_link *link, union hpd_irq_data *irq_data) { + static enum dc_status retval; + /* The HW reads 16 bytes from 200h on HPD, * but if we get an AUX_DEFER, the HW cannot retry * and this causes the CTS tests 4.3.2.1 - 3.2.4 to * fail, so we now explicitly read 6 bytes which is * the req from the above mentioned test cases. + * + * For DP 1.4 we need to read those from 2002h range. */ - return core_link_read_dpcd( - link, - DP_SINK_COUNT, - irq_data->raw, - sizeof(union hpd_irq_data)); + if (link->dpcd_caps.dpcd_rev.raw < DPCD_REV_14) + retval = core_link_read_dpcd( + link, + DP_SINK_COUNT, + irq_data->raw, + sizeof(union hpd_irq_data)); + else { + /* Read 2 bytes at this location,... */ + retval = core_link_read_dpcd( + link, + DP_SINK_COUNT_ESI, + irq_data->raw, + 2); + + if (retval != DC_OK) + return retval; + + /* ... then read remaining 4 at the other location */ + retval = core_link_read_dpcd( + link, + DP_LANE0_1_STATUS_ESI, + &irq_data->raw[2], + 4); + } + + return retval; } static bool allow_hpd_rx_irq(const struct dc_link *link) @@ -2272,12 +2303,14 @@ static void dp_wa_power_up_0010FA(struct dc_link *link, uint8_t *dpcd_data, static bool retrieve_link_cap(struct dc_link *link) { - uint8_t dpcd_data[DP_TRAINING_AUX_RD_INTERVAL - DP_DPCD_REV + 1]; + uint8_t dpcd_data[DP_ADAPTER_CAP - DP_DPCD_REV + 1]; union down_stream_port_count down_strm_port_count; union edp_configuration_cap edp_config_cap; union dp_downstream_port_present ds_port = { 0 }; enum dc_status status = DC_ERROR_UNEXPECTED; + uint32_t read_dpcd_retry_cnt = 3; + int i; memset(dpcd_data, '\0', sizeof(dpcd_data)); memset(&down_strm_port_count, @@ -2285,11 +2318,15 @@ static bool retrieve_link_cap(struct dc_link *link) memset(&edp_config_cap, '\0', sizeof(union edp_configuration_cap)); - status = core_link_read_dpcd( - link, - DP_DPCD_REV, - dpcd_data, - sizeof(dpcd_data)); + for (i = 0; i < read_dpcd_retry_cnt; i++) { + status = core_link_read_dpcd( + link, + DP_DPCD_REV, + dpcd_data, + sizeof(dpcd_data)); + if (status == DC_OK) + break; + } if (status != DC_OK) { dm_error("%s: Read dpcd data failed.\n", __func__); @@ -2376,6 +2413,10 @@ bool detect_dp_sink_caps(struct dc_link *link) void detect_edp_sink_caps(struct dc_link *link) { retrieve_link_cap(link); + + if (link->reported_link_cap.link_rate == LINK_RATE_UNKNOWN) + link->reported_link_cap.link_rate = LINK_RATE_HIGH2; + link->verified_link_cap = link->reported_link_cap; } diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_link_hwss.c b/drivers/gpu/drm/amd/display/dc/core/dc_link_hwss.c index 7c866a7d5e77..82cd1d6e6e59 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc_link_hwss.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc_link_hwss.c @@ -11,8 +11,6 @@ #include "dc_link_dp.h" #include "dc_link_ddc.h" #include "dm_helpers.h" -#include "dce/dce_link_encoder.h" -#include "dce/dce_stream_encoder.h" #include "dpcd_defs.h" enum dc_status core_link_read_dpcd( diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_resource.c b/drivers/gpu/drm/amd/display/dc/core/dc_resource.c index ba3487e97361..751f3ac9d921 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc_resource.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc_resource.c @@ -45,8 +45,9 @@ #include "dcn10/dcn10_resource.h" #endif #include "dce120/dce120_resource.h" -#define DC_LOGGER \ - ctx->logger + +#define DC_LOGGER_INIT(logger) + enum dce_version resource_parse_asic_id(struct hw_asic_id asic_id) { enum dce_version dc_version = DCE_VERSION_UNKNOWN; @@ -78,6 +79,8 @@ enum dce_version resource_parse_asic_id(struct hw_asic_id asic_id) ASIC_REV_IS_POLARIS12_V(asic_id.hw_internal_rev)) { dc_version = DCE_VERSION_11_2; } + if (ASIC_REV_IS_VEGAM(asic_id.hw_internal_rev)) + dc_version = DCE_VERSION_11_22; break; case FAMILY_AI: dc_version = DCE_VERSION_12_0; @@ -124,6 +127,7 @@ struct resource_pool *dc_create_resource_pool( num_virtual_links, dc, asic_id); break; case DCE_VERSION_11_2: + case DCE_VERSION_11_22: res_pool = dce112_create_resource_pool( num_virtual_links, dc); break; @@ -492,9 +496,9 @@ static void calculate_viewport(struct pipe_ctx *pipe_ctx) data->viewport_c.x = data->viewport.x / vpc_div; data->viewport_c.y = data->viewport.y / vpc_div; data->inits.h_c = (data->viewport.x % vpc_div) != 0 ? - dal_fixed31_32_half : dal_fixed31_32_zero; + dc_fixpt_half : dc_fixpt_zero; data->inits.v_c = (data->viewport.y % vpc_div) != 0 ? - dal_fixed31_32_half : dal_fixed31_32_zero; + dc_fixpt_half : dc_fixpt_zero; /* Round up, assume original video size always even dimensions */ data->viewport_c.width = (data->viewport.width + vpc_div - 1) / vpc_div; data->viewport_c.height = (data->viewport.height + vpc_div - 1) / vpc_div; @@ -623,10 +627,10 @@ static void calculate_scaling_ratios(struct pipe_ctx *pipe_ctx) pipe_ctx->plane_state->rotation == ROTATION_ANGLE_270) rect_swap_helper(&surf_src); - pipe_ctx->plane_res.scl_data.ratios.horz = dal_fixed31_32_from_fraction( + pipe_ctx->plane_res.scl_data.ratios.horz = dc_fixpt_from_fraction( surf_src.width, plane_state->dst_rect.width); - pipe_ctx->plane_res.scl_data.ratios.vert = dal_fixed31_32_from_fraction( + pipe_ctx->plane_res.scl_data.ratios.vert = dc_fixpt_from_fraction( surf_src.height, plane_state->dst_rect.height); @@ -648,6 +652,14 @@ static void calculate_scaling_ratios(struct pipe_ctx *pipe_ctx) pipe_ctx->plane_res.scl_data.ratios.horz_c.value /= 2; pipe_ctx->plane_res.scl_data.ratios.vert_c.value /= 2; } + pipe_ctx->plane_res.scl_data.ratios.horz = dc_fixpt_truncate( + pipe_ctx->plane_res.scl_data.ratios.horz, 19); + pipe_ctx->plane_res.scl_data.ratios.vert = dc_fixpt_truncate( + pipe_ctx->plane_res.scl_data.ratios.vert, 19); + pipe_ctx->plane_res.scl_data.ratios.horz_c = dc_fixpt_truncate( + pipe_ctx->plane_res.scl_data.ratios.horz_c, 19); + pipe_ctx->plane_res.scl_data.ratios.vert_c = dc_fixpt_truncate( + pipe_ctx->plane_res.scl_data.ratios.vert_c, 19); } static void calculate_inits_and_adj_vp(struct pipe_ctx *pipe_ctx, struct view *recout_skip) @@ -684,32 +696,33 @@ static void calculate_inits_and_adj_vp(struct pipe_ctx *pipe_ctx, struct view *r * init_bot = init + scaling_ratio * init_c = init + truncated_vp_c_offset(from calculate viewport) */ - data->inits.h = dal_fixed31_32_div_int( - dal_fixed31_32_add_int(data->ratios.horz, data->taps.h_taps + 1), 2); + data->inits.h = dc_fixpt_truncate(dc_fixpt_div_int( + dc_fixpt_add_int(data->ratios.horz, data->taps.h_taps + 1), 2), 19); - data->inits.h_c = dal_fixed31_32_add(data->inits.h_c, dal_fixed31_32_div_int( - dal_fixed31_32_add_int(data->ratios.horz_c, data->taps.h_taps_c + 1), 2)); + data->inits.h_c = dc_fixpt_truncate(dc_fixpt_add(data->inits.h_c, dc_fixpt_div_int( + dc_fixpt_add_int(data->ratios.horz_c, data->taps.h_taps_c + 1), 2)), 19); - data->inits.v = dal_fixed31_32_div_int( - dal_fixed31_32_add_int(data->ratios.vert, data->taps.v_taps + 1), 2); + data->inits.v = dc_fixpt_truncate(dc_fixpt_div_int( + dc_fixpt_add_int(data->ratios.vert, data->taps.v_taps + 1), 2), 19); + + data->inits.v_c = dc_fixpt_truncate(dc_fixpt_add(data->inits.v_c, dc_fixpt_div_int( + dc_fixpt_add_int(data->ratios.vert_c, data->taps.v_taps_c + 1), 2)), 19); - data->inits.v_c = dal_fixed31_32_add(data->inits.v_c, dal_fixed31_32_div_int( - dal_fixed31_32_add_int(data->ratios.vert_c, data->taps.v_taps_c + 1), 2)); /* Adjust for viewport end clip-off */ if ((data->viewport.x + data->viewport.width) < (src.x + src.width) && !flip_horz_scan_dir) { int vp_clip = src.x + src.width - data->viewport.width - data->viewport.x; - int int_part = dal_fixed31_32_floor( - dal_fixed31_32_sub(data->inits.h, data->ratios.horz)); + int int_part = dc_fixpt_floor( + dc_fixpt_sub(data->inits.h, data->ratios.horz)); int_part = int_part > 0 ? int_part : 0; data->viewport.width += int_part < vp_clip ? int_part : vp_clip; } if ((data->viewport.y + data->viewport.height) < (src.y + src.height) && !flip_vert_scan_dir) { int vp_clip = src.y + src.height - data->viewport.height - data->viewport.y; - int int_part = dal_fixed31_32_floor( - dal_fixed31_32_sub(data->inits.v, data->ratios.vert)); + int int_part = dc_fixpt_floor( + dc_fixpt_sub(data->inits.v, data->ratios.vert)); int_part = int_part > 0 ? int_part : 0; data->viewport.height += int_part < vp_clip ? int_part : vp_clip; @@ -717,8 +730,8 @@ static void calculate_inits_and_adj_vp(struct pipe_ctx *pipe_ctx, struct view *r if ((data->viewport_c.x + data->viewport_c.width) < (src.x + src.width) / vpc_div && !flip_horz_scan_dir) { int vp_clip = (src.x + src.width) / vpc_div - data->viewport_c.width - data->viewport_c.x; - int int_part = dal_fixed31_32_floor( - dal_fixed31_32_sub(data->inits.h_c, data->ratios.horz_c)); + int int_part = dc_fixpt_floor( + dc_fixpt_sub(data->inits.h_c, data->ratios.horz_c)); int_part = int_part > 0 ? int_part : 0; data->viewport_c.width += int_part < vp_clip ? int_part : vp_clip; @@ -726,8 +739,8 @@ static void calculate_inits_and_adj_vp(struct pipe_ctx *pipe_ctx, struct view *r if ((data->viewport_c.y + data->viewport_c.height) < (src.y + src.height) / vpc_div && !flip_vert_scan_dir) { int vp_clip = (src.y + src.height) / vpc_div - data->viewport_c.height - data->viewport_c.y; - int int_part = dal_fixed31_32_floor( - dal_fixed31_32_sub(data->inits.v_c, data->ratios.vert_c)); + int int_part = dc_fixpt_floor( + dc_fixpt_sub(data->inits.v_c, data->ratios.vert_c)); int_part = int_part > 0 ? int_part : 0; data->viewport_c.height += int_part < vp_clip ? int_part : vp_clip; @@ -737,9 +750,9 @@ static void calculate_inits_and_adj_vp(struct pipe_ctx *pipe_ctx, struct view *r if (data->viewport.x && !flip_horz_scan_dir) { int int_part; - data->inits.h = dal_fixed31_32_add(data->inits.h, dal_fixed31_32_mul_int( + data->inits.h = dc_fixpt_add(data->inits.h, dc_fixpt_mul_int( data->ratios.horz, recout_skip->width)); - int_part = dal_fixed31_32_floor(data->inits.h) - data->viewport.x; + int_part = dc_fixpt_floor(data->inits.h) - data->viewport.x; if (int_part < data->taps.h_taps) { int int_adj = data->viewport.x >= (data->taps.h_taps - int_part) ? (data->taps.h_taps - int_part) : data->viewport.x; @@ -752,15 +765,15 @@ static void calculate_inits_and_adj_vp(struct pipe_ctx *pipe_ctx, struct view *r int_part = data->taps.h_taps; } data->inits.h.value &= 0xffffffff; - data->inits.h = dal_fixed31_32_add_int(data->inits.h, int_part); + data->inits.h = dc_fixpt_add_int(data->inits.h, int_part); } if (data->viewport_c.x && !flip_horz_scan_dir) { int int_part; - data->inits.h_c = dal_fixed31_32_add(data->inits.h_c, dal_fixed31_32_mul_int( + data->inits.h_c = dc_fixpt_add(data->inits.h_c, dc_fixpt_mul_int( data->ratios.horz_c, recout_skip->width)); - int_part = dal_fixed31_32_floor(data->inits.h_c) - data->viewport_c.x; + int_part = dc_fixpt_floor(data->inits.h_c) - data->viewport_c.x; if (int_part < data->taps.h_taps_c) { int int_adj = data->viewport_c.x >= (data->taps.h_taps_c - int_part) ? (data->taps.h_taps_c - int_part) : data->viewport_c.x; @@ -773,15 +786,15 @@ static void calculate_inits_and_adj_vp(struct pipe_ctx *pipe_ctx, struct view *r int_part = data->taps.h_taps_c; } data->inits.h_c.value &= 0xffffffff; - data->inits.h_c = dal_fixed31_32_add_int(data->inits.h_c, int_part); + data->inits.h_c = dc_fixpt_add_int(data->inits.h_c, int_part); } if (data->viewport.y && !flip_vert_scan_dir) { int int_part; - data->inits.v = dal_fixed31_32_add(data->inits.v, dal_fixed31_32_mul_int( + data->inits.v = dc_fixpt_add(data->inits.v, dc_fixpt_mul_int( data->ratios.vert, recout_skip->height)); - int_part = dal_fixed31_32_floor(data->inits.v) - data->viewport.y; + int_part = dc_fixpt_floor(data->inits.v) - data->viewport.y; if (int_part < data->taps.v_taps) { int int_adj = data->viewport.y >= (data->taps.v_taps - int_part) ? (data->taps.v_taps - int_part) : data->viewport.y; @@ -794,15 +807,15 @@ static void calculate_inits_and_adj_vp(struct pipe_ctx *pipe_ctx, struct view *r int_part = data->taps.v_taps; } data->inits.v.value &= 0xffffffff; - data->inits.v = dal_fixed31_32_add_int(data->inits.v, int_part); + data->inits.v = dc_fixpt_add_int(data->inits.v, int_part); } if (data->viewport_c.y && !flip_vert_scan_dir) { int int_part; - data->inits.v_c = dal_fixed31_32_add(data->inits.v_c, dal_fixed31_32_mul_int( + data->inits.v_c = dc_fixpt_add(data->inits.v_c, dc_fixpt_mul_int( data->ratios.vert_c, recout_skip->height)); - int_part = dal_fixed31_32_floor(data->inits.v_c) - data->viewport_c.y; + int_part = dc_fixpt_floor(data->inits.v_c) - data->viewport_c.y; if (int_part < data->taps.v_taps_c) { int int_adj = data->viewport_c.y >= (data->taps.v_taps_c - int_part) ? (data->taps.v_taps_c - int_part) : data->viewport_c.y; @@ -815,12 +828,12 @@ static void calculate_inits_and_adj_vp(struct pipe_ctx *pipe_ctx, struct view *r int_part = data->taps.v_taps_c; } data->inits.v_c.value &= 0xffffffff; - data->inits.v_c = dal_fixed31_32_add_int(data->inits.v_c, int_part); + data->inits.v_c = dc_fixpt_add_int(data->inits.v_c, int_part); } /* Interlaced inits based on final vert inits */ - data->inits.v_bot = dal_fixed31_32_add(data->inits.v, data->ratios.vert); - data->inits.v_c_bot = dal_fixed31_32_add(data->inits.v_c, data->ratios.vert_c); + data->inits.v_bot = dc_fixpt_add(data->inits.v, data->ratios.vert); + data->inits.v_c_bot = dc_fixpt_add(data->inits.v_c, data->ratios.vert_c); if (pipe_ctx->plane_state->rotation == ROTATION_ANGLE_90 || pipe_ctx->plane_state->rotation == ROTATION_ANGLE_270) { @@ -835,7 +848,7 @@ bool resource_build_scaling_params(struct pipe_ctx *pipe_ctx) struct dc_crtc_timing *timing = &pipe_ctx->stream->timing; struct view recout_skip = { 0 }; bool res = false; - struct dc_context *ctx = pipe_ctx->stream->ctx; + DC_LOGGER_INIT(pipe_ctx->stream->ctx->logger); /* Important: scaling ratio calculation requires pixel format, * lb depth calculation requires recout and taps require scaling ratios. * Inits require viewport, taps, ratios and recout of split pipe @@ -843,6 +856,9 @@ bool resource_build_scaling_params(struct pipe_ctx *pipe_ctx) pipe_ctx->plane_res.scl_data.format = convert_pixel_format_to_dalsurface( pipe_ctx->plane_state->format); + if (pipe_ctx->stream->timing.flags.INTERLACE) + pipe_ctx->stream->dst.height *= 2; + calculate_scaling_ratios(pipe_ctx); calculate_viewport(pipe_ctx); @@ -863,6 +879,8 @@ bool resource_build_scaling_params(struct pipe_ctx *pipe_ctx) pipe_ctx->plane_res.scl_data.h_active = timing->h_addressable + timing->h_border_left + timing->h_border_right; pipe_ctx->plane_res.scl_data.v_active = timing->v_addressable + timing->v_border_top + timing->v_border_bottom; + if (pipe_ctx->stream->timing.flags.INTERLACE) + pipe_ctx->plane_res.scl_data.v_active *= 2; /* Taps calculations */ @@ -908,6 +926,9 @@ bool resource_build_scaling_params(struct pipe_ctx *pipe_ctx) plane_state->dst_rect.x, plane_state->dst_rect.y); + if (pipe_ctx->stream->timing.flags.INTERLACE) + pipe_ctx->stream->dst.height /= 2; + return res; } @@ -1294,6 +1315,19 @@ bool dc_add_all_planes_for_stream( } +static bool is_hdr_static_meta_changed(struct dc_stream_state *cur_stream, + struct dc_stream_state *new_stream) +{ + if (cur_stream == NULL) + return true; + + if (memcmp(&cur_stream->hdr_static_metadata, + &new_stream->hdr_static_metadata, + sizeof(struct dc_info_packet)) != 0) + return true; + + return false; +} static bool is_timing_changed(struct dc_stream_state *cur_stream, struct dc_stream_state *new_stream) @@ -1329,6 +1363,9 @@ static bool are_stream_backends_same( if (is_timing_changed(stream_a, stream_b)) return false; + if (is_hdr_static_meta_changed(stream_a, stream_b)) + return false; + return true; } @@ -1599,18 +1636,6 @@ enum dc_status dc_remove_stream_from_ctx( return DC_OK; } -static void copy_pipe_ctx( - const struct pipe_ctx *from_pipe_ctx, struct pipe_ctx *to_pipe_ctx) -{ - struct dc_plane_state *plane_state = to_pipe_ctx->plane_state; - struct dc_stream_state *stream = to_pipe_ctx->stream; - - *to_pipe_ctx = *from_pipe_ctx; - to_pipe_ctx->stream = stream; - if (plane_state != NULL) - to_pipe_ctx->plane_state = plane_state; -} - static struct dc_stream_state *find_pll_sharable_stream( struct dc_stream_state *stream_needs_pll, struct dc_state *context) @@ -1703,7 +1728,7 @@ enum dc_status resource_map_pool_resources( pipe_idx = acquire_first_split_pipe(&context->res_ctx, pool, stream); #endif - if (pipe_idx < 0) + if (pipe_idx < 0 || context->res_ctx.pipe_ctx[pipe_idx].stream_res.tg == NULL) return DC_NO_CONTROLLER_RESOURCE; pipe_ctx = &context->res_ctx.pipe_ctx[pipe_idx]; @@ -1752,26 +1777,6 @@ enum dc_status resource_map_pool_resources( return DC_ERROR_UNEXPECTED; } -/* first stream in the context is used to populate the rest */ -void validate_guaranteed_copy_streams( - struct dc_state *context, - int max_streams) -{ - int i; - - for (i = 1; i < max_streams; i++) { - context->streams[i] = context->streams[0]; - - copy_pipe_ctx(&context->res_ctx.pipe_ctx[0], - &context->res_ctx.pipe_ctx[i]); - context->res_ctx.pipe_ctx[i].stream = - context->res_ctx.pipe_ctx[0].stream; - - dc_stream_retain(context->streams[i]); - context->stream_count++; - } -} - void dc_resource_state_copy_construct_current( const struct dc *dc, struct dc_state *dst_ctx) @@ -1798,9 +1803,9 @@ enum dc_status dc_validate_global_state( return DC_ERROR_UNEXPECTED; if (dc->res_pool->funcs->validate_global) { - result = dc->res_pool->funcs->validate_global(dc, new_ctx); - if (result != DC_OK) - return result; + result = dc->res_pool->funcs->validate_global(dc, new_ctx); + if (result != DC_OK) + return result; } for (i = 0; i < new_ctx->stream_count; i++) { @@ -1843,7 +1848,7 @@ enum dc_status dc_validate_global_state( } static void patch_gamut_packet_checksum( - struct encoder_info_packet *gamut_packet) + struct dc_info_packet *gamut_packet) { /* For gamut we recalc checksum */ if (gamut_packet->valid) { @@ -1862,12 +1867,11 @@ static void patch_gamut_packet_checksum( } static void set_avi_info_frame( - struct encoder_info_packet *info_packet, + struct dc_info_packet *info_packet, struct pipe_ctx *pipe_ctx) { struct dc_stream_state *stream = pipe_ctx->stream; enum dc_color_space color_space = COLOR_SPACE_UNKNOWN; - struct info_frame info_frame = { {0} }; uint32_t pixel_encoding = 0; enum scanning_type scan_type = SCANNING_TYPE_NODATA; enum dc_aspect_ratio aspect = ASPECT_RATIO_NO_DATA; @@ -1877,22 +1881,24 @@ static void set_avi_info_frame( unsigned int cn0_cn1_value = 0; uint8_t *check_sum = NULL; uint8_t byte_index = 0; - union hdmi_info_packet *hdmi_info = &info_frame.avi_info_packet.info_packet_hdmi; + union hdmi_info_packet hdmi_info; union display_content_support support = {0}; unsigned int vic = pipe_ctx->stream->timing.vic; enum dc_timing_3d_format format; + memset(&hdmi_info, 0, sizeof(union hdmi_info_packet)); + color_space = pipe_ctx->stream->output_color_space; if (color_space == COLOR_SPACE_UNKNOWN) color_space = (stream->timing.pixel_encoding == PIXEL_ENCODING_RGB) ? COLOR_SPACE_SRGB:COLOR_SPACE_YCBCR709; /* Initialize header */ - hdmi_info->bits.header.info_frame_type = HDMI_INFOFRAME_TYPE_AVI; + hdmi_info.bits.header.info_frame_type = HDMI_INFOFRAME_TYPE_AVI; /* InfoFrameVersion_3 is defined by CEA861F (Section 6.4), but shall * not be used in HDMI 2.0 (Section 10.1) */ - hdmi_info->bits.header.version = 2; - hdmi_info->bits.header.length = HDMI_AVI_INFOFRAME_SIZE; + hdmi_info.bits.header.version = 2; + hdmi_info.bits.header.length = HDMI_AVI_INFOFRAME_SIZE; /* * IDO-defined (Y2,Y1,Y0 = 1,1,1) shall not be used by devices built @@ -1918,39 +1924,39 @@ static void set_avi_info_frame( /* Y0_Y1_Y2 : The pixel encoding */ /* H14b AVI InfoFrame has extension on Y-field from 2 bits to 3 bits */ - hdmi_info->bits.Y0_Y1_Y2 = pixel_encoding; + hdmi_info.bits.Y0_Y1_Y2 = pixel_encoding; /* A0 = 1 Active Format Information valid */ - hdmi_info->bits.A0 = ACTIVE_FORMAT_VALID; + hdmi_info.bits.A0 = ACTIVE_FORMAT_VALID; /* B0, B1 = 3; Bar info data is valid */ - hdmi_info->bits.B0_B1 = BAR_INFO_BOTH_VALID; + hdmi_info.bits.B0_B1 = BAR_INFO_BOTH_VALID; - hdmi_info->bits.SC0_SC1 = PICTURE_SCALING_UNIFORM; + hdmi_info.bits.SC0_SC1 = PICTURE_SCALING_UNIFORM; /* S0, S1 : Underscan / Overscan */ /* TODO: un-hardcode scan type */ scan_type = SCANNING_TYPE_UNDERSCAN; - hdmi_info->bits.S0_S1 = scan_type; + hdmi_info.bits.S0_S1 = scan_type; /* C0, C1 : Colorimetry */ if (color_space == COLOR_SPACE_YCBCR709 || color_space == COLOR_SPACE_YCBCR709_LIMITED) - hdmi_info->bits.C0_C1 = COLORIMETRY_ITU709; + hdmi_info.bits.C0_C1 = COLORIMETRY_ITU709; else if (color_space == COLOR_SPACE_YCBCR601 || color_space == COLOR_SPACE_YCBCR601_LIMITED) - hdmi_info->bits.C0_C1 = COLORIMETRY_ITU601; + hdmi_info.bits.C0_C1 = COLORIMETRY_ITU601; else { - hdmi_info->bits.C0_C1 = COLORIMETRY_NO_DATA; + hdmi_info.bits.C0_C1 = COLORIMETRY_NO_DATA; } if (color_space == COLOR_SPACE_2020_RGB_FULLRANGE || color_space == COLOR_SPACE_2020_RGB_LIMITEDRANGE || color_space == COLOR_SPACE_2020_YCBCR) { - hdmi_info->bits.EC0_EC2 = COLORIMETRYEX_BT2020RGBYCBCR; - hdmi_info->bits.C0_C1 = COLORIMETRY_EXTENDED; + hdmi_info.bits.EC0_EC2 = COLORIMETRYEX_BT2020RGBYCBCR; + hdmi_info.bits.C0_C1 = COLORIMETRY_EXTENDED; } else if (color_space == COLOR_SPACE_ADOBERGB) { - hdmi_info->bits.EC0_EC2 = COLORIMETRYEX_ADOBERGB; - hdmi_info->bits.C0_C1 = COLORIMETRY_EXTENDED; + hdmi_info.bits.EC0_EC2 = COLORIMETRYEX_ADOBERGB; + hdmi_info.bits.C0_C1 = COLORIMETRY_EXTENDED; } /* TODO: un-hardcode aspect ratio */ @@ -1959,18 +1965,18 @@ static void set_avi_info_frame( switch (aspect) { case ASPECT_RATIO_4_3: case ASPECT_RATIO_16_9: - hdmi_info->bits.M0_M1 = aspect; + hdmi_info.bits.M0_M1 = aspect; break; case ASPECT_RATIO_NO_DATA: case ASPECT_RATIO_64_27: case ASPECT_RATIO_256_135: default: - hdmi_info->bits.M0_M1 = 0; + hdmi_info.bits.M0_M1 = 0; } /* Active Format Aspect ratio - same as Picture Aspect Ratio. */ - hdmi_info->bits.R0_R3 = ACTIVE_FORMAT_ASPECT_RATIO_SAME_AS_PICTURE; + hdmi_info.bits.R0_R3 = ACTIVE_FORMAT_ASPECT_RATIO_SAME_AS_PICTURE; /* TODO: un-hardcode cn0_cn1 and itc */ @@ -2013,8 +2019,8 @@ static void set_avi_info_frame( } } } - hdmi_info->bits.CN0_CN1 = cn0_cn1_value; - hdmi_info->bits.ITC = itc_value; + hdmi_info.bits.CN0_CN1 = cn0_cn1_value; + hdmi_info.bits.ITC = itc_value; } /* TODO : We should handle YCC quantization */ @@ -2023,19 +2029,19 @@ static void set_avi_info_frame( stream->sink->edid_caps.qy_bit == 1) { if (color_space == COLOR_SPACE_SRGB || color_space == COLOR_SPACE_2020_RGB_FULLRANGE) { - hdmi_info->bits.Q0_Q1 = RGB_QUANTIZATION_FULL_RANGE; - hdmi_info->bits.YQ0_YQ1 = YYC_QUANTIZATION_FULL_RANGE; + hdmi_info.bits.Q0_Q1 = RGB_QUANTIZATION_FULL_RANGE; + hdmi_info.bits.YQ0_YQ1 = YYC_QUANTIZATION_FULL_RANGE; } else if (color_space == COLOR_SPACE_SRGB_LIMITED || color_space == COLOR_SPACE_2020_RGB_LIMITEDRANGE) { - hdmi_info->bits.Q0_Q1 = RGB_QUANTIZATION_LIMITED_RANGE; - hdmi_info->bits.YQ0_YQ1 = YYC_QUANTIZATION_LIMITED_RANGE; + hdmi_info.bits.Q0_Q1 = RGB_QUANTIZATION_LIMITED_RANGE; + hdmi_info.bits.YQ0_YQ1 = YYC_QUANTIZATION_LIMITED_RANGE; } else { - hdmi_info->bits.Q0_Q1 = RGB_QUANTIZATION_DEFAULT_RANGE; - hdmi_info->bits.YQ0_YQ1 = YYC_QUANTIZATION_LIMITED_RANGE; + hdmi_info.bits.Q0_Q1 = RGB_QUANTIZATION_DEFAULT_RANGE; + hdmi_info.bits.YQ0_YQ1 = YYC_QUANTIZATION_LIMITED_RANGE; } } else { - hdmi_info->bits.Q0_Q1 = RGB_QUANTIZATION_DEFAULT_RANGE; - hdmi_info->bits.YQ0_YQ1 = YYC_QUANTIZATION_LIMITED_RANGE; + hdmi_info.bits.Q0_Q1 = RGB_QUANTIZATION_DEFAULT_RANGE; + hdmi_info.bits.YQ0_YQ1 = YYC_QUANTIZATION_LIMITED_RANGE; } ///VIC @@ -2060,51 +2066,49 @@ static void set_avi_info_frame( break; } } - hdmi_info->bits.VIC0_VIC7 = vic; + hdmi_info.bits.VIC0_VIC7 = vic; /* pixel repetition * PR0 - PR3 start from 0 whereas pHwPathMode->mode.timing.flags.pixel * repetition start from 1 */ - hdmi_info->bits.PR0_PR3 = 0; + hdmi_info.bits.PR0_PR3 = 0; /* Bar Info * barTop: Line Number of End of Top Bar. * barBottom: Line Number of Start of Bottom Bar. * barLeft: Pixel Number of End of Left Bar. * barRight: Pixel Number of Start of Right Bar. */ - hdmi_info->bits.bar_top = stream->timing.v_border_top; - hdmi_info->bits.bar_bottom = (stream->timing.v_total + hdmi_info.bits.bar_top = stream->timing.v_border_top; + hdmi_info.bits.bar_bottom = (stream->timing.v_total - stream->timing.v_border_bottom + 1); - hdmi_info->bits.bar_left = stream->timing.h_border_left; - hdmi_info->bits.bar_right = (stream->timing.h_total + hdmi_info.bits.bar_left = stream->timing.h_border_left; + hdmi_info.bits.bar_right = (stream->timing.h_total - stream->timing.h_border_right + 1); /* check_sum - Calculate AFMT_AVI_INFO0 ~ AFMT_AVI_INFO3 */ - check_sum = &info_frame.avi_info_packet.info_packet_hdmi.packet_raw_data.sb[0]; + check_sum = &hdmi_info.packet_raw_data.sb[0]; *check_sum = HDMI_INFOFRAME_TYPE_AVI + HDMI_AVI_INFOFRAME_SIZE + 2; for (byte_index = 1; byte_index <= HDMI_AVI_INFOFRAME_SIZE; byte_index++) - *check_sum += hdmi_info->packet_raw_data.sb[byte_index]; + *check_sum += hdmi_info.packet_raw_data.sb[byte_index]; /* one byte complement */ *check_sum = (uint8_t) (0x100 - *check_sum); /* Store in hw_path_mode */ - info_packet->hb0 = hdmi_info->packet_raw_data.hb0; - info_packet->hb1 = hdmi_info->packet_raw_data.hb1; - info_packet->hb2 = hdmi_info->packet_raw_data.hb2; + info_packet->hb0 = hdmi_info.packet_raw_data.hb0; + info_packet->hb1 = hdmi_info.packet_raw_data.hb1; + info_packet->hb2 = hdmi_info.packet_raw_data.hb2; - for (byte_index = 0; byte_index < sizeof(info_frame.avi_info_packet. - info_packet_hdmi.packet_raw_data.sb); byte_index++) - info_packet->sb[byte_index] = info_frame.avi_info_packet. - info_packet_hdmi.packet_raw_data.sb[byte_index]; + for (byte_index = 0; byte_index < sizeof(hdmi_info.packet_raw_data.sb); byte_index++) + info_packet->sb[byte_index] = hdmi_info.packet_raw_data.sb[byte_index]; info_packet->valid = true; } static void set_vendor_info_packet( - struct encoder_info_packet *info_packet, + struct dc_info_packet *info_packet, struct dc_stream_state *stream) { uint32_t length = 0; @@ -2217,7 +2221,7 @@ static void set_vendor_info_packet( } static void set_spd_info_packet( - struct encoder_info_packet *info_packet, + struct dc_info_packet *info_packet, struct dc_stream_state *stream) { /* SPD info packet for FreeSync */ @@ -2338,104 +2342,19 @@ static void set_spd_info_packet( } static void set_hdr_static_info_packet( - struct encoder_info_packet *info_packet, + struct dc_info_packet *info_packet, struct dc_stream_state *stream) { - uint16_t i = 0; - enum signal_type signal = stream->signal; - uint32_t data; + /* HDR Static Metadata info packet for HDR10 */ - if (!stream->hdr_static_metadata.hdr_supported) + if (!stream->hdr_static_metadata.valid) return; - if (dc_is_hdmi_signal(signal)) { - info_packet->valid = true; - - info_packet->hb0 = 0x87; - info_packet->hb1 = 0x01; - info_packet->hb2 = 0x1A; - i = 1; - } else if (dc_is_dp_signal(signal)) { - info_packet->valid = true; - - info_packet->hb0 = 0x00; - info_packet->hb1 = 0x87; - info_packet->hb2 = 0x1D; - info_packet->hb3 = (0x13 << 2); - i = 2; - } - - data = stream->hdr_static_metadata.is_hdr; - info_packet->sb[i++] = data ? 0x02 : 0x00; - info_packet->sb[i++] = 0x00; - - data = stream->hdr_static_metadata.chromaticity_green_x / 2; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - data = stream->hdr_static_metadata.chromaticity_green_y / 2; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - data = stream->hdr_static_metadata.chromaticity_blue_x / 2; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - data = stream->hdr_static_metadata.chromaticity_blue_y / 2; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - data = stream->hdr_static_metadata.chromaticity_red_x / 2; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - data = stream->hdr_static_metadata.chromaticity_red_y / 2; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - data = stream->hdr_static_metadata.chromaticity_white_point_x / 2; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - data = stream->hdr_static_metadata.chromaticity_white_point_y / 2; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - data = stream->hdr_static_metadata.max_luminance; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - data = stream->hdr_static_metadata.min_luminance; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - data = stream->hdr_static_metadata.maximum_content_light_level; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - data = stream->hdr_static_metadata.maximum_frame_average_light_level; - info_packet->sb[i++] = data & 0xFF; - info_packet->sb[i++] = (data & 0xFF00) >> 8; - - if (dc_is_hdmi_signal(signal)) { - uint32_t checksum = 0; - - checksum += info_packet->hb0; - checksum += info_packet->hb1; - checksum += info_packet->hb2; - - for (i = 1; i <= info_packet->hb2; i++) - checksum += info_packet->sb[i]; - - info_packet->sb[0] = 0x100 - checksum; - } else if (dc_is_dp_signal(signal)) { - info_packet->sb[0] = 0x01; - info_packet->sb[1] = 0x1A; - } + *info_packet = stream->hdr_static_metadata; } static void set_vsc_info_packet( - struct encoder_info_packet *info_packet, + struct dc_info_packet *info_packet, struct dc_stream_state *stream) { unsigned int vscPacketRevision = 0; @@ -2650,6 +2569,8 @@ bool pipe_need_reprogram( if (is_timing_changed(pipe_ctx_old->stream, pipe_ctx->stream)) return true; + if (is_hdr_static_meta_changed(pipe_ctx_old->stream, pipe_ctx->stream)) + return true; return false; } diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_stream.c b/drivers/gpu/drm/amd/display/dc/core/dc_stream.c index ce0747ed0f00..3732a1de9d6c 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc_stream.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc_stream.c @@ -101,14 +101,16 @@ static void construct(struct dc_stream_state *stream, stream->status.link = stream->sink->link; update_stream_signal(stream); + + stream->out_transfer_func = dc_create_transfer_func(); + stream->out_transfer_func->type = TF_TYPE_BYPASS; } static void destruct(struct dc_stream_state *stream) { dc_sink_release(stream->sink); if (stream->out_transfer_func != NULL) { - dc_transfer_func_release( - stream->out_transfer_func); + dc_transfer_func_release(stream->out_transfer_func); stream->out_transfer_func = NULL; } } @@ -176,6 +178,7 @@ bool dc_stream_set_cursor_attributes( int i; struct dc *core_dc; struct resource_context *res_ctx; + struct pipe_ctx *pipe_to_program = NULL; if (NULL == stream) { dm_error("DC: dc_stream is NULL!\n"); @@ -203,9 +206,17 @@ bool dc_stream_set_cursor_attributes( if (pipe_ctx->top_pipe && pipe_ctx->plane_state != pipe_ctx->top_pipe->plane_state) continue; + if (!pipe_to_program) { + pipe_to_program = pipe_ctx; + core_dc->hwss.pipe_control_lock(core_dc, pipe_to_program, true); + } core_dc->hwss.set_cursor_attribute(pipe_ctx); } + + if (pipe_to_program) + core_dc->hwss.pipe_control_lock(core_dc, pipe_to_program, false); + return true; } @@ -216,6 +227,7 @@ bool dc_stream_set_cursor_position( int i; struct dc *core_dc; struct resource_context *res_ctx; + struct pipe_ctx *pipe_to_program = NULL; if (NULL == stream) { dm_error("DC: dc_stream is NULL!\n"); @@ -241,9 +253,17 @@ bool dc_stream_set_cursor_position( !pipe_ctx->plane_res.ipp) continue; + if (!pipe_to_program) { + pipe_to_program = pipe_ctx; + core_dc->hwss.pipe_control_lock(core_dc, pipe_to_program, true); + } + core_dc->hwss.set_cursor_position(pipe_ctx); } + if (pipe_to_program) + core_dc->hwss.pipe_control_lock(core_dc, pipe_to_program, false); + return true; } diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_surface.c b/drivers/gpu/drm/amd/display/dc/core/dc_surface.c index 132eef3826e2..68a71adeb12e 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc_surface.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc_surface.c @@ -38,6 +38,12 @@ static void construct(struct dc_context *ctx, struct dc_plane_state *plane_state) { plane_state->ctx = ctx; + + plane_state->gamma_correction = dc_create_gamma(); + plane_state->gamma_correction->is_identity = true; + + plane_state->in_transfer_func = dc_create_transfer_func(); + plane_state->in_transfer_func->type = TF_TYPE_BYPASS; } static void destruct(struct dc_plane_state *plane_state) @@ -175,7 +181,7 @@ void dc_transfer_func_release(struct dc_transfer_func *tf) kref_put(&tf->refcount, dc_transfer_func_free); } -struct dc_transfer_func *dc_create_transfer_func(void) +struct dc_transfer_func *dc_create_transfer_func() { struct dc_transfer_func *tf = kvzalloc(sizeof(*tf), GFP_KERNEL); diff --git a/drivers/gpu/drm/amd/display/dc/dc.h b/drivers/gpu/drm/amd/display/dc/dc.h index fa4b3c8b3bb7..9cfde0ccf4e9 100644 --- a/drivers/gpu/drm/amd/display/dc/dc.h +++ b/drivers/gpu/drm/amd/display/dc/dc.h @@ -38,7 +38,7 @@ #include "inc/compressor.h" #include "dml/display_mode_lib.h" -#define DC_VER "3.1.38" +#define DC_VER "3.1.44" #define MAX_SURFACES 3 #define MAX_STREAMS 6 @@ -75,6 +75,7 @@ struct dc_caps { bool dynamic_audio; bool is_apu; bool dual_link_dvi; + bool post_blend_color_processing; }; struct dc_dcc_surface_param { @@ -202,6 +203,8 @@ struct dc_debug { bool timing_trace; bool clock_trace; bool validation_trace; + bool bandwidth_calcs_trace; + int max_downscale_src_width; /* stutter efficiency related */ bool disable_stutter; @@ -238,6 +241,8 @@ struct dc_debug { bool az_endpoint_mute_only; bool always_use_regamma; bool p010_mpo_support; + bool recovery_enabled; + }; struct dc_state; struct resource_pool; @@ -332,20 +337,6 @@ enum { TRANSFER_FUNC_POINTS = 1025 }; -// Moved here from color module for linux -enum color_transfer_func { - transfer_func_unknown, - transfer_func_srgb, - transfer_func_bt709, - transfer_func_pq2084, - transfer_func_pq2084_interim, - transfer_func_linear_0_1, - transfer_func_linear_0_125, - transfer_func_dolbyvision, - transfer_func_gamma_22, - transfer_func_gamma_26 -}; - struct dc_hdr_static_metadata { /* display chromaticities and white point in units of 0.00001 */ unsigned int chromaticity_green_x; @@ -361,9 +352,6 @@ struct dc_hdr_static_metadata { uint32_t max_luminance; uint32_t maximum_content_light_level; uint32_t maximum_frame_average_light_level; - - bool hdr_supported; - bool is_hdr; }; enum dc_transfer_func_type { @@ -419,7 +407,6 @@ union surface_update_flags { /* Medium updates */ uint32_t dcc_change:1; uint32_t color_space_change:1; - uint32_t input_tf_change:1; uint32_t horizontal_mirror_change:1; uint32_t per_pixel_alpha_change:1; uint32_t rotation_change:1; @@ -428,6 +415,7 @@ union surface_update_flags { uint32_t position_change:1; uint32_t in_transfer_func_change:1; uint32_t input_csc_change:1; + uint32_t coeff_reduction_change:1; uint32_t output_tf_change:1; uint32_t pixel_format_change:1; @@ -460,7 +448,7 @@ struct dc_plane_state { struct dc_gamma *gamma_correction; struct dc_transfer_func *in_transfer_func; struct dc_bias_and_scale *bias_and_scale; - struct csc_transform input_csc_color_matrix; + struct dc_csc_transform input_csc_color_matrix; struct fixed31_32 coeff_reduction_factor; uint32_t sdr_white_level; @@ -468,7 +456,6 @@ struct dc_plane_state { struct dc_hdr_static_metadata hdr_static_ctx; enum dc_color_space color_space; - enum color_transfer_func input_tf; enum surface_pixel_format format; enum dc_rotation_angle rotation; @@ -498,7 +485,6 @@ struct dc_plane_info { enum dc_rotation_angle rotation; enum plane_stereo_format stereo_format; enum dc_color_space color_space; - enum color_transfer_func input_tf; unsigned int sdr_white_level; bool horizontal_mirror; bool visible; @@ -517,19 +503,18 @@ struct dc_surface_update { struct dc_plane_state *surface; /* isr safe update parameters. null means no updates */ - struct dc_flip_addrs *flip_addr; - struct dc_plane_info *plane_info; - struct dc_scaling_info *scaling_info; + const struct dc_flip_addrs *flip_addr; + const struct dc_plane_info *plane_info; + const struct dc_scaling_info *scaling_info; /* following updates require alloc/sleep/spin that is not isr safe, * null means no updates */ - struct dc_gamma *gamma; - enum color_transfer_func color_input_tf; - struct dc_transfer_func *in_transfer_func; + const struct dc_gamma *gamma; + const struct dc_transfer_func *in_transfer_func; - struct csc_transform *input_csc_color_matrix; - struct fixed31_32 *coeff_reduction_factor; + const struct dc_csc_transform *input_csc_color_matrix; + const struct fixed31_32 *coeff_reduction_factor; }; /* @@ -699,6 +684,7 @@ struct dc_cursor { struct dc_cursor_attributes attributes; }; + /******************************************************************************* * Interrupt interfaces ******************************************************************************/ diff --git a/drivers/gpu/drm/amd/display/dc/dc_dp_types.h b/drivers/gpu/drm/amd/display/dc/dc_dp_types.h index 2726b02e006b..90bccd5ccaa2 100644 --- a/drivers/gpu/drm/amd/display/dc/dc_dp_types.h +++ b/drivers/gpu/drm/amd/display/dc/dc_dp_types.h @@ -26,6 +26,8 @@ #ifndef DC_DP_TYPES_H #define DC_DP_TYPES_H +#include "os_types.h" + enum dc_lane_count { LANE_COUNT_UNKNOWN = 0, LANE_COUNT_ONE = 1, diff --git a/drivers/gpu/drm/amd/display/dc/dc_helper.c b/drivers/gpu/drm/amd/display/dc/dc_helper.c index 48e1fcf53d43..bd0fda0ceb91 100644 --- a/drivers/gpu/drm/amd/display/dc/dc_helper.c +++ b/drivers/gpu/drm/amd/display/dc/dc_helper.c @@ -117,6 +117,65 @@ uint32_t generic_reg_get5(const struct dc_context *ctx, uint32_t addr, return reg_val; } +uint32_t generic_reg_get6(const struct dc_context *ctx, uint32_t addr, + uint8_t shift1, uint32_t mask1, uint32_t *field_value1, + uint8_t shift2, uint32_t mask2, uint32_t *field_value2, + uint8_t shift3, uint32_t mask3, uint32_t *field_value3, + uint8_t shift4, uint32_t mask4, uint32_t *field_value4, + uint8_t shift5, uint32_t mask5, uint32_t *field_value5, + uint8_t shift6, uint32_t mask6, uint32_t *field_value6) +{ + uint32_t reg_val = dm_read_reg(ctx, addr); + *field_value1 = get_reg_field_value_ex(reg_val, mask1, shift1); + *field_value2 = get_reg_field_value_ex(reg_val, mask2, shift2); + *field_value3 = get_reg_field_value_ex(reg_val, mask3, shift3); + *field_value4 = get_reg_field_value_ex(reg_val, mask4, shift4); + *field_value5 = get_reg_field_value_ex(reg_val, mask5, shift5); + *field_value6 = get_reg_field_value_ex(reg_val, mask6, shift6); + return reg_val; +} + +uint32_t generic_reg_get7(const struct dc_context *ctx, uint32_t addr, + uint8_t shift1, uint32_t mask1, uint32_t *field_value1, + uint8_t shift2, uint32_t mask2, uint32_t *field_value2, + uint8_t shift3, uint32_t mask3, uint32_t *field_value3, + uint8_t shift4, uint32_t mask4, uint32_t *field_value4, + uint8_t shift5, uint32_t mask5, uint32_t *field_value5, + uint8_t shift6, uint32_t mask6, uint32_t *field_value6, + uint8_t shift7, uint32_t mask7, uint32_t *field_value7) +{ + uint32_t reg_val = dm_read_reg(ctx, addr); + *field_value1 = get_reg_field_value_ex(reg_val, mask1, shift1); + *field_value2 = get_reg_field_value_ex(reg_val, mask2, shift2); + *field_value3 = get_reg_field_value_ex(reg_val, mask3, shift3); + *field_value4 = get_reg_field_value_ex(reg_val, mask4, shift4); + *field_value5 = get_reg_field_value_ex(reg_val, mask5, shift5); + *field_value6 = get_reg_field_value_ex(reg_val, mask6, shift6); + *field_value7 = get_reg_field_value_ex(reg_val, mask7, shift7); + return reg_val; +} + +uint32_t generic_reg_get8(const struct dc_context *ctx, uint32_t addr, + uint8_t shift1, uint32_t mask1, uint32_t *field_value1, + uint8_t shift2, uint32_t mask2, uint32_t *field_value2, + uint8_t shift3, uint32_t mask3, uint32_t *field_value3, + uint8_t shift4, uint32_t mask4, uint32_t *field_value4, + uint8_t shift5, uint32_t mask5, uint32_t *field_value5, + uint8_t shift6, uint32_t mask6, uint32_t *field_value6, + uint8_t shift7, uint32_t mask7, uint32_t *field_value7, + uint8_t shift8, uint32_t mask8, uint32_t *field_value8) +{ + uint32_t reg_val = dm_read_reg(ctx, addr); + *field_value1 = get_reg_field_value_ex(reg_val, mask1, shift1); + *field_value2 = get_reg_field_value_ex(reg_val, mask2, shift2); + *field_value3 = get_reg_field_value_ex(reg_val, mask3, shift3); + *field_value4 = get_reg_field_value_ex(reg_val, mask4, shift4); + *field_value5 = get_reg_field_value_ex(reg_val, mask5, shift5); + *field_value6 = get_reg_field_value_ex(reg_val, mask6, shift6); + *field_value7 = get_reg_field_value_ex(reg_val, mask7, shift7); + *field_value8 = get_reg_field_value_ex(reg_val, mask8, shift8); + return reg_val; +} /* note: va version of this is pretty bad idea, since there is a output parameter pass by pointer * compiler won't be able to check for size match and is prone to stack corruption type of bugs diff --git a/drivers/gpu/drm/amd/display/dc/dc_hw_types.h b/drivers/gpu/drm/amd/display/dc/dc_hw_types.h index b83a7dc2f5a9..b1f70579d61b 100644 --- a/drivers/gpu/drm/amd/display/dc/dc_hw_types.h +++ b/drivers/gpu/drm/amd/display/dc/dc_hw_types.h @@ -423,6 +423,11 @@ enum dc_gamma_type { GAMMA_CS_TFM_1D = 3, }; +struct dc_csc_transform { + uint16_t matrix[12]; + bool enable_adjustment; +}; + struct dc_gamma { struct kref refcount; enum dc_gamma_type type; diff --git a/drivers/gpu/drm/amd/display/dc/dc_link.h b/drivers/gpu/drm/amd/display/dc/dc_link.h index dc34515ef01f..8a716baa1203 100644 --- a/drivers/gpu/drm/amd/display/dc/dc_link.h +++ b/drivers/gpu/drm/amd/display/dc/dc_link.h @@ -51,6 +51,14 @@ struct link_mst_stream_allocation_table { struct link_mst_stream_allocation stream_allocations[MAX_CONTROLLER_NUM]; }; +struct time_stamp { + uint64_t edp_poweroff; + uint64_t edp_poweron; +}; + +struct link_trace { + struct time_stamp time_stamp; +}; /* * A link contains one or more sinks and their connected status. * The currently active signal type (HDMI, DP-SST, DP-MST) is also reported. @@ -114,6 +122,7 @@ struct dc_link { struct dc_link_status link_status; + struct link_trace link_trace; }; const struct dc_link_status *dc_link_get_status(const struct dc_link *dc_link); diff --git a/drivers/gpu/drm/amd/display/dc/dc_stream.h b/drivers/gpu/drm/amd/display/dc/dc_stream.h index d017df56b2ba..d7e6d53bb383 100644 --- a/drivers/gpu/drm/amd/display/dc/dc_stream.h +++ b/drivers/gpu/drm/amd/display/dc/dc_stream.h @@ -58,18 +58,20 @@ struct dc_stream_state { struct freesync_context freesync_ctx; - struct dc_hdr_static_metadata hdr_static_metadata; + struct dc_info_packet hdr_static_metadata; struct dc_transfer_func *out_transfer_func; struct colorspace_transform gamut_remap_matrix; - struct csc_transform csc_color_matrix; + struct dc_csc_transform csc_color_matrix; enum dc_color_space output_color_space; enum dc_dither_option dither_option; enum view_3d_format view_format; - enum color_transfer_func output_tf; bool ignore_msa_timing_param; + + unsigned long long periodic_fn_vsync_delta; + /* TODO: custom INFO packets */ /* TODO: ABM info (DMCU) */ /* PSR info */ @@ -110,9 +112,10 @@ struct dc_stream_update { struct rect src; struct rect dst; struct dc_transfer_func *out_transfer_func; - struct dc_hdr_static_metadata *hdr_static_metadata; - enum color_transfer_func color_output_tf; + struct dc_info_packet *hdr_static_metadata; unsigned int *abm_level; + + unsigned long long *periodic_fn_vsync_delta; }; bool dc_is_stream_unchanged( @@ -131,13 +134,6 @@ bool dc_is_stream_scaling_unchanged( * This does not trigger a flip. No surface address is programmed. */ -bool dc_commit_planes_to_stream( - struct dc *dc, - struct dc_plane_state **plane_states, - uint8_t new_plane_count, - struct dc_stream_state *dc_stream, - struct dc_state *state); - void dc_commit_updates_for_stream(struct dc *dc, struct dc_surface_update *srf_updates, int surface_count, @@ -208,14 +204,6 @@ bool dc_add_all_planes_for_stream( enum dc_status dc_validate_stream(struct dc *dc, struct dc_stream_state *stream); -/* - * This function takes a stream and checks if it is guaranteed to be supported. - * Guaranteed means that MAX_COFUNC similar streams are supported. - * - * After this call: - * No hardware is programmed for call. Only validation is done. - */ - /* * Set up streams and links associated to drive sinks * The streams parameter is an absolute set of all active streams. diff --git a/drivers/gpu/drm/amd/display/dc/dc_types.h b/drivers/gpu/drm/amd/display/dc/dc_types.h index 9441305d3ab5..76df2534c4a4 100644 --- a/drivers/gpu/drm/amd/display/dc/dc_types.h +++ b/drivers/gpu/drm/amd/display/dc/dc_types.h @@ -25,7 +25,7 @@ #ifndef DC_TYPES_H_ #define DC_TYPES_H_ -#include "fixed32_32.h" +#include "os_types.h" #include "fixed31_32.h" #include "irq_types.h" #include "dc_dp_types.h" @@ -370,12 +370,6 @@ struct dc_csc_adjustments { struct fixed31_32 hue; }; -enum { - MAX_LANES = 2, - MAX_COFUNC_PATH = 6, - LAYER_INDEX_PRIMARY = -1, -}; - enum dpcd_downstream_port_max_bpc { DOWN_STREAM_MAX_8BPC = 0, DOWN_STREAM_MAX_10BPC, @@ -530,6 +524,15 @@ struct vrr_params { uint32_t frame_counter; }; +struct dc_info_packet { + bool valid; + uint8_t hb0; + uint8_t hb1; + uint8_t hb2; + uint8_t hb3; + uint8_t sb[32]; +}; + #define DC_PLANE_UPDATE_TIMES_MAX 10 struct dc_plane_flip_time { diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_abm.c b/drivers/gpu/drm/amd/display/dc/dce/dce_abm.c index fe92a1222803..29294db1a96b 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_abm.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_abm.c @@ -26,7 +26,7 @@ #include "dce_abm.h" #include "dm_services.h" #include "reg_helper.h" -#include "fixed32_32.h" +#include "fixed31_32.h" #include "dc.h" #include "atom.h" diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_audio.c b/drivers/gpu/drm/amd/display/dc/dce/dce_audio.c index 6d5cdcdc8ec9..7f6d724686f1 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_audio.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_audio.c @@ -33,8 +33,9 @@ #define CTX \ aud->base.ctx -#define DC_LOGGER \ - aud->base.ctx->logger + +#define DC_LOGGER_INIT() + #define REG(reg)\ (aud->regs->reg) @@ -348,8 +349,8 @@ static void set_audio_latency( void dce_aud_az_enable(struct audio *audio) { - struct dce_audio *aud = DCE_AUD(audio); uint32_t value = AZ_REG_READ(AZALIA_F0_CODEC_PIN_CONTROL_HOT_PLUG_CONTROL); + DC_LOGGER_INIT(); set_reg_field_value(value, 1, AZALIA_F0_CODEC_PIN_CONTROL_HOT_PLUG_CONTROL, @@ -371,7 +372,7 @@ void dce_aud_az_enable(struct audio *audio) void dce_aud_az_disable(struct audio *audio) { uint32_t value; - struct dce_audio *aud = DCE_AUD(audio); + DC_LOGGER_INIT(); value = AZ_REG_READ(AZALIA_F0_CODEC_PIN_CONTROL_HOT_PLUG_CONTROL); set_reg_field_value(value, 1, diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c b/drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c index 0aa2cda60890..88b09dd758ba 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c @@ -41,8 +41,9 @@ #define CTX \ clk_src->base.ctx -#define DC_LOGGER \ - calc_pll_cs->ctx->logger + +#define DC_LOGGER_INIT() + #undef FN #define FN(reg_name, field_name) \ clk_src->cs_shift->field_name, clk_src->cs_mask->field_name @@ -467,7 +468,7 @@ static uint32_t dce110_get_pix_clk_dividers_helper ( { uint32_t field = 0; uint32_t pll_calc_error = MAX_PLL_CALC_ERROR; - struct calc_pll_clock_source *calc_pll_cs = &clk_src->calc_pll; + DC_LOGGER_INIT(); /* Check if reference clock is external (not pcie/xtalin) * HW Dce80 spec: * 00 - PCIE_REFCLK, 01 - XTALIN, 02 - GENERICA, 03 - GENERICB @@ -557,8 +558,8 @@ static uint32_t dce110_get_pix_clk_dividers( struct pll_settings *pll_settings) { struct dce110_clk_src *clk_src = TO_DCE110_CLK_SRC(cs); - struct calc_pll_clock_source *calc_pll_cs = &clk_src->calc_pll; uint32_t pll_calc_error = MAX_PLL_CALC_ERROR; + DC_LOGGER_INIT(); if (pix_clk_params == NULL || pll_settings == NULL || pix_clk_params->requested_pix_clk == 0) { @@ -589,6 +590,7 @@ static uint32_t dce110_get_pix_clk_dividers( pll_settings, pix_clk_params); break; case DCE_VERSION_11_2: + case DCE_VERSION_11_22: case DCE_VERSION_12_0: #if defined(CONFIG_DRM_AMD_DC_DCN1_0) case DCN_VERSION_1_0: @@ -655,12 +657,12 @@ static uint32_t dce110_get_d_to_pixel_rate_in_hz( return 0; } - pix_rate = dal_fixed31_32_from_int(clk_src->ref_freq_khz); - pix_rate = dal_fixed31_32_mul_int(pix_rate, 1000); - pix_rate = dal_fixed31_32_mul_int(pix_rate, phase); - pix_rate = dal_fixed31_32_div_int(pix_rate, modulo); + pix_rate = dc_fixpt_from_int(clk_src->ref_freq_khz); + pix_rate = dc_fixpt_mul_int(pix_rate, 1000); + pix_rate = dc_fixpt_mul_int(pix_rate, phase); + pix_rate = dc_fixpt_div_int(pix_rate, modulo); - return dal_fixed31_32_round(pix_rate); + return dc_fixpt_round(pix_rate); } else { return dce110_get_dp_pixel_rate_from_combo_phy_pll(cs, pix_clk_params, pll_settings); } @@ -709,12 +711,12 @@ static bool calculate_ss( const struct spread_spectrum_data *ss_data, struct delta_sigma_data *ds_data) { - struct fixed32_32 fb_div; - struct fixed32_32 ss_amount; - struct fixed32_32 ss_nslip_amount; - struct fixed32_32 ss_ds_frac_amount; - struct fixed32_32 ss_step_size; - struct fixed32_32 modulation_time; + struct fixed31_32 fb_div; + struct fixed31_32 ss_amount; + struct fixed31_32 ss_nslip_amount; + struct fixed31_32 ss_ds_frac_amount; + struct fixed31_32 ss_step_size; + struct fixed31_32 modulation_time; if (ds_data == NULL) return false; @@ -729,42 +731,42 @@ static bool calculate_ss( /* compute SS_AMOUNT_FBDIV & SS_AMOUNT_NFRAC_SLIP & SS_AMOUNT_DSFRAC*/ /* 6 decimal point support in fractional feedback divider */ - fb_div = dal_fixed32_32_from_fraction( + fb_div = dc_fixpt_from_fraction( pll_settings->fract_feedback_divider, 1000000); - fb_div = dal_fixed32_32_add_int(fb_div, pll_settings->feedback_divider); + fb_div = dc_fixpt_add_int(fb_div, pll_settings->feedback_divider); ds_data->ds_frac_amount = 0; /*spreadSpectrumPercentage is in the unit of .01%, * so have to divided by 100 * 100*/ - ss_amount = dal_fixed32_32_mul( - fb_div, dal_fixed32_32_from_fraction(ss_data->percentage, + ss_amount = dc_fixpt_mul( + fb_div, dc_fixpt_from_fraction(ss_data->percentage, 100 * ss_data->percentage_divider)); - ds_data->feedback_amount = dal_fixed32_32_floor(ss_amount); + ds_data->feedback_amount = dc_fixpt_floor(ss_amount); - ss_nslip_amount = dal_fixed32_32_sub(ss_amount, - dal_fixed32_32_from_int(ds_data->feedback_amount)); - ss_nslip_amount = dal_fixed32_32_mul_int(ss_nslip_amount, 10); - ds_data->nfrac_amount = dal_fixed32_32_floor(ss_nslip_amount); + ss_nslip_amount = dc_fixpt_sub(ss_amount, + dc_fixpt_from_int(ds_data->feedback_amount)); + ss_nslip_amount = dc_fixpt_mul_int(ss_nslip_amount, 10); + ds_data->nfrac_amount = dc_fixpt_floor(ss_nslip_amount); - ss_ds_frac_amount = dal_fixed32_32_sub(ss_nslip_amount, - dal_fixed32_32_from_int(ds_data->nfrac_amount)); - ss_ds_frac_amount = dal_fixed32_32_mul_int(ss_ds_frac_amount, 65536); - ds_data->ds_frac_amount = dal_fixed32_32_floor(ss_ds_frac_amount); + ss_ds_frac_amount = dc_fixpt_sub(ss_nslip_amount, + dc_fixpt_from_int(ds_data->nfrac_amount)); + ss_ds_frac_amount = dc_fixpt_mul_int(ss_ds_frac_amount, 65536); + ds_data->ds_frac_amount = dc_fixpt_floor(ss_ds_frac_amount); /* compute SS_STEP_SIZE_DSFRAC */ - modulation_time = dal_fixed32_32_from_fraction( + modulation_time = dc_fixpt_from_fraction( pll_settings->reference_freq * 1000, pll_settings->reference_divider * ss_data->modulation_freq_hz); if (ss_data->flags.CENTER_SPREAD) - modulation_time = dal_fixed32_32_div_int(modulation_time, 4); + modulation_time = dc_fixpt_div_int(modulation_time, 4); else - modulation_time = dal_fixed32_32_div_int(modulation_time, 2); + modulation_time = dc_fixpt_div_int(modulation_time, 2); - ss_step_size = dal_fixed32_32_div(ss_amount, modulation_time); + ss_step_size = dc_fixpt_div(ss_amount, modulation_time); /* SS_STEP_SIZE_DSFRAC_DEC = Int(SS_STEP_SIZE * 2 ^ 16 * 10)*/ - ss_step_size = dal_fixed32_32_mul_int(ss_step_size, 65536 * 10); - ds_data->ds_frac_size = dal_fixed32_32_floor(ss_step_size); + ss_step_size = dc_fixpt_mul_int(ss_step_size, 65536 * 10); + ds_data->ds_frac_size = dc_fixpt_floor(ss_step_size); return true; } @@ -978,6 +980,7 @@ static bool dce110_program_pix_clk( break; case DCE_VERSION_11_2: + case DCE_VERSION_11_22: case DCE_VERSION_12_0: #if defined(CONFIG_DRM_AMD_DC_DCN1_0) case DCN_VERSION_1_0: @@ -1054,7 +1057,7 @@ static void get_ss_info_from_atombios( struct spread_spectrum_info *ss_info_cur; struct spread_spectrum_data *ss_data_cur; uint32_t i; - struct calc_pll_clock_source *calc_pll_cs = &clk_src->calc_pll; + DC_LOGGER_INIT(); if (ss_entries_num == NULL) { DC_LOG_SYNC( "Invalid entry !!!\n"); @@ -1076,13 +1079,15 @@ static void get_ss_info_from_atombios( if (*ss_entries_num == 0) return; - ss_info = kzalloc(sizeof(struct spread_spectrum_info) * (*ss_entries_num), + ss_info = kcalloc(*ss_entries_num, + sizeof(struct spread_spectrum_info), GFP_KERNEL); ss_info_cur = ss_info; if (ss_info == NULL) return; - ss_data = kzalloc(sizeof(struct spread_spectrum_data) * (*ss_entries_num), + ss_data = kcalloc(*ss_entries_num, + sizeof(struct spread_spectrum_data), GFP_KERNEL); if (ss_data == NULL) goto out_free_info; diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_clocks.c b/drivers/gpu/drm/amd/display/dc/dce/dce_clocks.c index 78e6beb6cf26..8a581c67bf2d 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_clocks.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_clocks.c @@ -26,7 +26,7 @@ #include "dce_clocks.h" #include "dm_services.h" #include "reg_helper.h" -#include "fixed32_32.h" +#include "fixed31_32.h" #include "bios_parser_interface.h" #include "dc.h" #include "dmcu.h" @@ -35,7 +35,7 @@ #endif #include "core_types.h" #include "dc_types.h" - +#include "dal_asic_id.h" #define TO_DCE_CLOCKS(clocks)\ container_of(clocks, struct dce_disp_clk, base) @@ -228,19 +228,19 @@ static int dce_clocks_get_dp_ref_freq(struct display_clock *clk) generated according to average value (case as with previous ASICs) */ if (clk_dce->ss_on_dprefclk && clk_dce->dprefclk_ss_divider != 0) { - struct fixed32_32 ss_percentage = dal_fixed32_32_div_int( - dal_fixed32_32_from_fraction( + struct fixed31_32 ss_percentage = dc_fixpt_div_int( + dc_fixpt_from_fraction( clk_dce->dprefclk_ss_percentage, clk_dce->dprefclk_ss_divider), 200); - struct fixed32_32 adj_dp_ref_clk_khz; + struct fixed31_32 adj_dp_ref_clk_khz; - ss_percentage = dal_fixed32_32_sub(dal_fixed32_32_one, + ss_percentage = dc_fixpt_sub(dc_fixpt_one, ss_percentage); adj_dp_ref_clk_khz = - dal_fixed32_32_mul_int( + dc_fixpt_mul_int( ss_percentage, dp_ref_clk_khz); - dp_ref_clk_khz = dal_fixed32_32_floor(adj_dp_ref_clk_khz); + dp_ref_clk_khz = dc_fixpt_floor(adj_dp_ref_clk_khz); } return dp_ref_clk_khz; @@ -256,19 +256,19 @@ static int dce_clocks_get_dp_ref_freq_wrkaround(struct display_clock *clk) int dp_ref_clk_khz = 600000; if (clk_dce->ss_on_dprefclk && clk_dce->dprefclk_ss_divider != 0) { - struct fixed32_32 ss_percentage = dal_fixed32_32_div_int( - dal_fixed32_32_from_fraction( + struct fixed31_32 ss_percentage = dc_fixpt_div_int( + dc_fixpt_from_fraction( clk_dce->dprefclk_ss_percentage, clk_dce->dprefclk_ss_divider), 200); - struct fixed32_32 adj_dp_ref_clk_khz; + struct fixed31_32 adj_dp_ref_clk_khz; - ss_percentage = dal_fixed32_32_sub(dal_fixed32_32_one, + ss_percentage = dc_fixpt_sub(dc_fixpt_one, ss_percentage); adj_dp_ref_clk_khz = - dal_fixed32_32_mul_int( + dc_fixpt_mul_int( ss_percentage, dp_ref_clk_khz); - dp_ref_clk_khz = dal_fixed32_32_floor(adj_dp_ref_clk_khz); + dp_ref_clk_khz = dc_fixpt_floor(adj_dp_ref_clk_khz); } return dp_ref_clk_khz; @@ -413,9 +413,12 @@ static int dce112_set_clock( /*VBIOS will determine DPREFCLK frequency, so we don't set it*/ dce_clk_params.target_clock_frequency = 0; dce_clk_params.clock_type = DCECLOCK_TYPE_DPREFCLK; - dce_clk_params.flags.USE_GENLOCK_AS_SOURCE_FOR_DPREFCLK = + if (!ASICREV_IS_VEGA20_P(clk->ctx->asic_id.hw_internal_rev)) + dce_clk_params.flags.USE_GENLOCK_AS_SOURCE_FOR_DPREFCLK = (dce_clk_params.pll_id == CLOCK_SOURCE_COMBO_DISPLAY_PLL0); + else + dce_clk_params.flags.USE_GENLOCK_AS_SOURCE_FOR_DPREFCLK = false; bp->funcs->set_dce_clock(bp, &dce_clk_params); diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_dmcu.c b/drivers/gpu/drm/amd/display/dc/dce/dce_dmcu.c index 2ee3d9bf1062..a576b8bbb3cd 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_dmcu.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_dmcu.c @@ -28,7 +28,7 @@ #include "dce_dmcu.h" #include "dm_services.h" #include "reg_helper.h" -#include "fixed32_32.h" +#include "fixed31_32.h" #include "dc.h" #define TO_DCE_DMCU(dmcu)\ diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_hwseq.c b/drivers/gpu/drm/amd/display/dc/dce/dce_hwseq.c index 487724345d9d..0275d6d60da4 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_hwseq.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_hwseq.c @@ -53,7 +53,8 @@ void dce_pipe_control_lock(struct dc *dc, struct dce_hwseq *hws = dc->hwseq; /* Not lock pipe when blank */ - if (lock && pipe->stream_res.tg->funcs->is_blanked(pipe->stream_res.tg)) + if (lock && pipe->stream_res.tg->funcs->is_blanked && + pipe->stream_res.tg->funcs->is_blanked(pipe->stream_res.tg)) return; val = REG_GET_4(BLND_V_UPDATE_LOCK[pipe->stream_res.tg->inst], diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_ipp.c b/drivers/gpu/drm/amd/display/dc/dce/dce_ipp.c index d737e911971b..5d9506b3d46b 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_ipp.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_ipp.c @@ -195,13 +195,13 @@ static void dce_ipp_program_input_lut( for (i = 0; i < gamma->num_entries; i++) { REG_SET(DC_LUT_SEQ_COLOR, 0, DC_LUT_SEQ_COLOR, - dal_fixed31_32_round( + dc_fixpt_round( gamma->entries.red[i])); REG_SET(DC_LUT_SEQ_COLOR, 0, DC_LUT_SEQ_COLOR, - dal_fixed31_32_round( + dc_fixpt_round( gamma->entries.green[i])); REG_SET(DC_LUT_SEQ_COLOR, 0, DC_LUT_SEQ_COLOR, - dal_fixed31_32_round( + dc_fixpt_round( gamma->entries.blue[i])); } diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_link_encoder.c b/drivers/gpu/drm/amd/display/dc/dce/dce_link_encoder.c index 8167cad7bcf7..dbe3b26b6d9e 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_link_encoder.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_link_encoder.c @@ -113,6 +113,7 @@ static const struct link_encoder_funcs dce110_lnk_enc_funcs = { .connect_dig_be_to_fe = dce110_link_encoder_connect_dig_be_to_fe, .enable_hpd = dce110_link_encoder_enable_hpd, .disable_hpd = dce110_link_encoder_disable_hpd, + .is_dig_enabled = dce110_is_dig_enabled, .destroy = dce110_link_encoder_destroy }; @@ -535,8 +536,9 @@ void dce110_psr_program_secondary_packet(struct link_encoder *enc, DP_SEC_GSP0_PRIORITY, 1); } -static bool is_dig_enabled(const struct dce110_link_encoder *enc110) +bool dce110_is_dig_enabled(struct link_encoder *enc) { + struct dce110_link_encoder *enc110 = TO_DCE110_LINK_ENC(enc); uint32_t value; REG_GET(DIG_BE_EN_CNTL, DIG_ENABLE, &value); @@ -1031,7 +1033,7 @@ void dce110_link_encoder_disable_output( struct bp_transmitter_control cntl = { 0 }; enum bp_result result; - if (!is_dig_enabled(enc110)) { + if (!dce110_is_dig_enabled(enc)) { /* OF_SKIP_POWER_DOWN_INACTIVE_ENCODER */ return; } diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_link_encoder.h b/drivers/gpu/drm/amd/display/dc/dce/dce_link_encoder.h index 0ec3433d34b6..347069461a22 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_link_encoder.h +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_link_encoder.h @@ -263,4 +263,6 @@ void dce110_psr_program_dp_dphy_fast_training(struct link_encoder *enc, void dce110_psr_program_secondary_packet(struct link_encoder *enc, unsigned int sdp_transmit_line_num_deadline); +bool dce110_is_dig_enabled(struct link_encoder *enc); + #endif /* __DC_LINK_ENCODER__DCE110_H__ */ diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_mem_input.c b/drivers/gpu/drm/amd/display/dc/dce/dce_mem_input.c index 0790f25c7b3b..b235a75355b8 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_mem_input.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_mem_input.c @@ -174,6 +174,25 @@ static void program_urgency_watermark( URGENCY_HIGH_WATERMARK, urgency_high_wm); } +static void dce120_program_urgency_watermark( + struct dce_mem_input *dce_mi, + uint32_t wm_select, + uint32_t urgency_low_wm, + uint32_t urgency_high_wm) +{ + REG_UPDATE(DPG_WATERMARK_MASK_CONTROL, + URGENCY_WATERMARK_MASK, wm_select); + + REG_SET_2(DPG_PIPE_URGENCY_CONTROL, 0, + URGENCY_LOW_WATERMARK, urgency_low_wm, + URGENCY_HIGH_WATERMARK, urgency_high_wm); + + REG_SET_2(DPG_PIPE_URGENT_LEVEL_CONTROL, 0, + URGENT_LEVEL_LOW_WATERMARK, urgency_low_wm, + URGENT_LEVEL_HIGH_WATERMARK, urgency_high_wm); + +} + static void program_nbp_watermark( struct dce_mem_input *dce_mi, uint32_t wm_select, @@ -206,6 +225,25 @@ static void program_nbp_watermark( } } +static void dce120_program_stutter_watermark( + struct dce_mem_input *dce_mi, + uint32_t wm_select, + uint32_t stutter_mark, + uint32_t stutter_entry) +{ + REG_UPDATE(DPG_WATERMARK_MASK_CONTROL, + STUTTER_EXIT_SELF_REFRESH_WATERMARK_MASK, wm_select); + + if (REG(DPG_PIPE_STUTTER_CONTROL2)) + REG_UPDATE_2(DPG_PIPE_STUTTER_CONTROL2, + STUTTER_EXIT_SELF_REFRESH_WATERMARK, stutter_mark, + STUTTER_ENTER_SELF_REFRESH_WATERMARK, stutter_entry); + else + REG_UPDATE_2(DPG_PIPE_STUTTER_CONTROL, + STUTTER_EXIT_SELF_REFRESH_WATERMARK, stutter_mark, + STUTTER_ENTER_SELF_REFRESH_WATERMARK, stutter_entry); +} + static void program_stutter_watermark( struct dce_mem_input *dce_mi, uint32_t wm_select, @@ -225,7 +263,8 @@ static void program_stutter_watermark( static void dce_mi_program_display_marks( struct mem_input *mi, struct dce_watermarks nbp, - struct dce_watermarks stutter, + struct dce_watermarks stutter_exit, + struct dce_watermarks stutter_enter, struct dce_watermarks urgent, uint32_t total_dest_line_time_ns) { @@ -243,13 +282,14 @@ static void dce_mi_program_display_marks( program_nbp_watermark(dce_mi, 2, nbp.a_mark); /* set a */ program_nbp_watermark(dce_mi, 1, nbp.d_mark); /* set d */ - program_stutter_watermark(dce_mi, 2, stutter.a_mark); /* set a */ - program_stutter_watermark(dce_mi, 1, stutter.d_mark); /* set d */ + program_stutter_watermark(dce_mi, 2, stutter_exit.a_mark); /* set a */ + program_stutter_watermark(dce_mi, 1, stutter_exit.d_mark); /* set d */ } -static void dce120_mi_program_display_marks(struct mem_input *mi, +static void dce112_mi_program_display_marks(struct mem_input *mi, struct dce_watermarks nbp, - struct dce_watermarks stutter, + struct dce_watermarks stutter_exit, + struct dce_watermarks stutter_entry, struct dce_watermarks urgent, uint32_t total_dest_line_time_ns) { @@ -273,10 +313,43 @@ static void dce120_mi_program_display_marks(struct mem_input *mi, program_nbp_watermark(dce_mi, 2, nbp.c_mark); /* set c */ program_nbp_watermark(dce_mi, 3, nbp.d_mark); /* set d */ - program_stutter_watermark(dce_mi, 0, stutter.a_mark); /* set a */ - program_stutter_watermark(dce_mi, 1, stutter.b_mark); /* set b */ - program_stutter_watermark(dce_mi, 2, stutter.c_mark); /* set c */ - program_stutter_watermark(dce_mi, 3, stutter.d_mark); /* set d */ + program_stutter_watermark(dce_mi, 0, stutter_exit.a_mark); /* set a */ + program_stutter_watermark(dce_mi, 1, stutter_exit.b_mark); /* set b */ + program_stutter_watermark(dce_mi, 2, stutter_exit.c_mark); /* set c */ + program_stutter_watermark(dce_mi, 3, stutter_exit.d_mark); /* set d */ +} + +static void dce120_mi_program_display_marks(struct mem_input *mi, + struct dce_watermarks nbp, + struct dce_watermarks stutter_exit, + struct dce_watermarks stutter_entry, + struct dce_watermarks urgent, + uint32_t total_dest_line_time_ns) +{ + struct dce_mem_input *dce_mi = TO_DCE_MEM_INPUT(mi); + uint32_t stutter_en = mi->ctx->dc->debug.disable_stutter ? 0 : 1; + + dce120_program_urgency_watermark(dce_mi, 0, /* set a */ + urgent.a_mark, total_dest_line_time_ns); + dce120_program_urgency_watermark(dce_mi, 1, /* set b */ + urgent.b_mark, total_dest_line_time_ns); + dce120_program_urgency_watermark(dce_mi, 2, /* set c */ + urgent.c_mark, total_dest_line_time_ns); + dce120_program_urgency_watermark(dce_mi, 3, /* set d */ + urgent.d_mark, total_dest_line_time_ns); + + REG_UPDATE_2(DPG_PIPE_STUTTER_CONTROL, + STUTTER_ENABLE, stutter_en, + STUTTER_IGNORE_FBC, 1); + program_nbp_watermark(dce_mi, 0, nbp.a_mark); /* set a */ + program_nbp_watermark(dce_mi, 1, nbp.b_mark); /* set b */ + program_nbp_watermark(dce_mi, 2, nbp.c_mark); /* set c */ + program_nbp_watermark(dce_mi, 3, nbp.d_mark); /* set d */ + + dce120_program_stutter_watermark(dce_mi, 0, stutter_exit.a_mark, stutter_entry.a_mark); /* set a */ + dce120_program_stutter_watermark(dce_mi, 1, stutter_exit.b_mark, stutter_entry.b_mark); /* set b */ + dce120_program_stutter_watermark(dce_mi, 2, stutter_exit.c_mark, stutter_entry.c_mark); /* set c */ + dce120_program_stutter_watermark(dce_mi, 3, stutter_exit.d_mark, stutter_entry.d_mark); /* set d */ } static void program_tiling( @@ -694,6 +767,18 @@ void dce112_mem_input_construct( const struct dce_mem_input_registers *regs, const struct dce_mem_input_shift *mi_shift, const struct dce_mem_input_mask *mi_mask) +{ + dce_mem_input_construct(dce_mi, ctx, inst, regs, mi_shift, mi_mask); + dce_mi->base.funcs->mem_input_program_display_marks = dce112_mi_program_display_marks; +} + +void dce120_mem_input_construct( + struct dce_mem_input *dce_mi, + struct dc_context *ctx, + int inst, + const struct dce_mem_input_registers *regs, + const struct dce_mem_input_shift *mi_shift, + const struct dce_mem_input_mask *mi_mask) { dce_mem_input_construct(dce_mi, ctx, inst, regs, mi_shift, mi_mask); dce_mi->base.funcs->mem_input_program_display_marks = dce120_mi_program_display_marks; diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_mem_input.h b/drivers/gpu/drm/amd/display/dc/dce/dce_mem_input.h index 05d39c0cbe87..d15b0d7f47fc 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_mem_input.h +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_mem_input.h @@ -106,6 +106,7 @@ struct dce_mem_input_registers { uint32_t DPG_PIPE_ARBITRATION_CONTROL1; uint32_t DPG_WATERMARK_MASK_CONTROL; uint32_t DPG_PIPE_URGENCY_CONTROL; + uint32_t DPG_PIPE_URGENT_LEVEL_CONTROL; uint32_t DPG_PIPE_NB_PSTATE_CHANGE_CONTROL; uint32_t DPG_PIPE_LOW_POWER_CONTROL; uint32_t DPG_PIPE_STUTTER_CONTROL; @@ -213,6 +214,11 @@ struct dce_mem_input_registers { #define MI_DCE12_DMIF_PG_MASK_SH_LIST(mask_sh, blk)\ SFB(blk, DPG_PIPE_STUTTER_CONTROL2, STUTTER_EXIT_SELF_REFRESH_WATERMARK, mask_sh),\ + SFB(blk, DPG_PIPE_STUTTER_CONTROL2, STUTTER_ENTER_SELF_REFRESH_WATERMARK, mask_sh),\ + SFB(blk, DPG_PIPE_URGENT_LEVEL_CONTROL, URGENT_LEVEL_LOW_WATERMARK, mask_sh),\ + SFB(blk, DPG_PIPE_URGENT_LEVEL_CONTROL, URGENT_LEVEL_HIGH_WATERMARK, mask_sh),\ + SFB(blk, DPG_PIPE_URGENCY_CONTROL, URGENCY_LOW_WATERMARK, mask_sh),\ + SFB(blk, DPG_PIPE_URGENCY_CONTROL, URGENCY_HIGH_WATERMARK, mask_sh),\ SFB(blk, DPG_WATERMARK_MASK_CONTROL, PSTATE_CHANGE_WATERMARK_MASK, mask_sh),\ SFB(blk, DPG_PIPE_LOW_POWER_CONTROL, PSTATE_CHANGE_ENABLE, mask_sh),\ SFB(blk, DPG_PIPE_LOW_POWER_CONTROL, PSTATE_CHANGE_URGENT_DURING_REQUEST, mask_sh),\ @@ -286,6 +292,8 @@ struct dce_mem_input_registers { type STUTTER_EXIT_SELF_REFRESH_WATERMARK_MASK; \ type URGENCY_LOW_WATERMARK; \ type URGENCY_HIGH_WATERMARK; \ + type URGENT_LEVEL_LOW_WATERMARK;\ + type URGENT_LEVEL_HIGH_WATERMARK;\ type NB_PSTATE_CHANGE_ENABLE; \ type NB_PSTATE_CHANGE_URGENT_DURING_REQUEST; \ type NB_PSTATE_CHANGE_NOT_SELF_REFRESH_DURING_REQUEST; \ @@ -297,6 +305,7 @@ struct dce_mem_input_registers { type STUTTER_ENABLE; \ type STUTTER_IGNORE_FBC; \ type STUTTER_EXIT_SELF_REFRESH_WATERMARK; \ + type STUTTER_ENTER_SELF_REFRESH_WATERMARK; \ type DMIF_BUFFERS_ALLOCATED; \ type DMIF_BUFFERS_ALLOCATION_COMPLETED; \ type ENABLE; /* MC_HUB_RDREQ_DMIF_LIMIT */\ @@ -344,4 +353,12 @@ void dce112_mem_input_construct( const struct dce_mem_input_shift *mi_shift, const struct dce_mem_input_mask *mi_mask); +void dce120_mem_input_construct( + struct dce_mem_input *dce_mi, + struct dc_context *ctx, + int inst, + const struct dce_mem_input_registers *regs, + const struct dce_mem_input_shift *mi_shift, + const struct dce_mem_input_mask *mi_mask); + #endif /*__DCE_MEM_INPUT_H__*/ diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_scl_filters.c b/drivers/gpu/drm/amd/display/dc/dce/dce_scl_filters.c index 6243450b41b7..48862bebf29e 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_scl_filters.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_scl_filters.c @@ -1014,11 +1014,11 @@ static const uint16_t filter_8tap_64p_183[264] = { const uint16_t *get_filter_3tap_16p(struct fixed31_32 ratio) { - if (ratio.value < dal_fixed31_32_one.value) + if (ratio.value < dc_fixpt_one.value) return filter_3tap_16p_upscale; - else if (ratio.value < dal_fixed31_32_from_fraction(4, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(4, 3).value) return filter_3tap_16p_117; - else if (ratio.value < dal_fixed31_32_from_fraction(5, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(5, 3).value) return filter_3tap_16p_150; else return filter_3tap_16p_183; @@ -1026,11 +1026,11 @@ const uint16_t *get_filter_3tap_16p(struct fixed31_32 ratio) const uint16_t *get_filter_3tap_64p(struct fixed31_32 ratio) { - if (ratio.value < dal_fixed31_32_one.value) + if (ratio.value < dc_fixpt_one.value) return filter_3tap_64p_upscale; - else if (ratio.value < dal_fixed31_32_from_fraction(4, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(4, 3).value) return filter_3tap_64p_117; - else if (ratio.value < dal_fixed31_32_from_fraction(5, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(5, 3).value) return filter_3tap_64p_150; else return filter_3tap_64p_183; @@ -1038,11 +1038,11 @@ const uint16_t *get_filter_3tap_64p(struct fixed31_32 ratio) const uint16_t *get_filter_4tap_16p(struct fixed31_32 ratio) { - if (ratio.value < dal_fixed31_32_one.value) + if (ratio.value < dc_fixpt_one.value) return filter_4tap_16p_upscale; - else if (ratio.value < dal_fixed31_32_from_fraction(4, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(4, 3).value) return filter_4tap_16p_117; - else if (ratio.value < dal_fixed31_32_from_fraction(5, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(5, 3).value) return filter_4tap_16p_150; else return filter_4tap_16p_183; @@ -1050,11 +1050,11 @@ const uint16_t *get_filter_4tap_16p(struct fixed31_32 ratio) const uint16_t *get_filter_4tap_64p(struct fixed31_32 ratio) { - if (ratio.value < dal_fixed31_32_one.value) + if (ratio.value < dc_fixpt_one.value) return filter_4tap_64p_upscale; - else if (ratio.value < dal_fixed31_32_from_fraction(4, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(4, 3).value) return filter_4tap_64p_117; - else if (ratio.value < dal_fixed31_32_from_fraction(5, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(5, 3).value) return filter_4tap_64p_150; else return filter_4tap_64p_183; @@ -1062,11 +1062,11 @@ const uint16_t *get_filter_4tap_64p(struct fixed31_32 ratio) const uint16_t *get_filter_5tap_64p(struct fixed31_32 ratio) { - if (ratio.value < dal_fixed31_32_one.value) + if (ratio.value < dc_fixpt_one.value) return filter_5tap_64p_upscale; - else if (ratio.value < dal_fixed31_32_from_fraction(4, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(4, 3).value) return filter_5tap_64p_117; - else if (ratio.value < dal_fixed31_32_from_fraction(5, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(5, 3).value) return filter_5tap_64p_150; else return filter_5tap_64p_183; @@ -1074,11 +1074,11 @@ const uint16_t *get_filter_5tap_64p(struct fixed31_32 ratio) const uint16_t *get_filter_6tap_64p(struct fixed31_32 ratio) { - if (ratio.value < dal_fixed31_32_one.value) + if (ratio.value < dc_fixpt_one.value) return filter_6tap_64p_upscale; - else if (ratio.value < dal_fixed31_32_from_fraction(4, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(4, 3).value) return filter_6tap_64p_117; - else if (ratio.value < dal_fixed31_32_from_fraction(5, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(5, 3).value) return filter_6tap_64p_150; else return filter_6tap_64p_183; @@ -1086,11 +1086,11 @@ const uint16_t *get_filter_6tap_64p(struct fixed31_32 ratio) const uint16_t *get_filter_7tap_64p(struct fixed31_32 ratio) { - if (ratio.value < dal_fixed31_32_one.value) + if (ratio.value < dc_fixpt_one.value) return filter_7tap_64p_upscale; - else if (ratio.value < dal_fixed31_32_from_fraction(4, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(4, 3).value) return filter_7tap_64p_117; - else if (ratio.value < dal_fixed31_32_from_fraction(5, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(5, 3).value) return filter_7tap_64p_150; else return filter_7tap_64p_183; @@ -1098,11 +1098,11 @@ const uint16_t *get_filter_7tap_64p(struct fixed31_32 ratio) const uint16_t *get_filter_8tap_64p(struct fixed31_32 ratio) { - if (ratio.value < dal_fixed31_32_one.value) + if (ratio.value < dc_fixpt_one.value) return filter_8tap_64p_upscale; - else if (ratio.value < dal_fixed31_32_from_fraction(4, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(4, 3).value) return filter_8tap_64p_117; - else if (ratio.value < dal_fixed31_32_from_fraction(5, 3).value) + else if (ratio.value < dc_fixpt_from_fraction(5, 3).value) return filter_8tap_64p_150; else return filter_8tap_64p_183; diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_stream_encoder.c b/drivers/gpu/drm/amd/display/dc/dce/dce_stream_encoder.c index 162f6a6c4208..c0e813c7ddd4 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_stream_encoder.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_stream_encoder.c @@ -26,27 +26,10 @@ #include "dc_bios_types.h" #include "dce_stream_encoder.h" #include "reg_helper.h" +#include "hw_shared.h" + #define DC_LOGGER \ enc110->base.ctx->logger -enum DP_PIXEL_ENCODING { -DP_PIXEL_ENCODING_RGB444 = 0x00000000, -DP_PIXEL_ENCODING_YCBCR422 = 0x00000001, -DP_PIXEL_ENCODING_YCBCR444 = 0x00000002, -DP_PIXEL_ENCODING_RGB_WIDE_GAMUT = 0x00000003, -DP_PIXEL_ENCODING_Y_ONLY = 0x00000004, -DP_PIXEL_ENCODING_YCBCR420 = 0x00000005, -DP_PIXEL_ENCODING_RESERVED = 0x00000006, -}; - - -enum DP_COMPONENT_DEPTH { -DP_COMPONENT_DEPTH_6BPC = 0x00000000, -DP_COMPONENT_DEPTH_8BPC = 0x00000001, -DP_COMPONENT_DEPTH_10BPC = 0x00000002, -DP_COMPONENT_DEPTH_12BPC = 0x00000003, -DP_COMPONENT_DEPTH_16BPC = 0x00000004, -DP_COMPONENT_DEPTH_RESERVED = 0x00000005, -}; #define REG(reg)\ @@ -80,7 +63,7 @@ enum { static void dce110_update_generic_info_packet( struct dce110_stream_encoder *enc110, uint32_t packet_index, - const struct encoder_info_packet *info_packet) + const struct dc_info_packet *info_packet) { uint32_t regval; /* TODOFPGA Figure out a proper number for max_retries polling for lock @@ -89,7 +72,8 @@ static void dce110_update_generic_info_packet( uint32_t max_retries = 50; /*we need turn on clock before programming AFMT block*/ - REG_UPDATE(AFMT_CNTL, AFMT_AUDIO_CLOCK_EN, 1); + if (REG(AFMT_CNTL)) + REG_UPDATE(AFMT_CNTL, AFMT_AUDIO_CLOCK_EN, 1); if (REG(AFMT_VBI_PACKET_CONTROL1)) { if (packet_index >= 8) @@ -196,7 +180,7 @@ static void dce110_update_generic_info_packet( static void dce110_update_hdmi_info_packet( struct dce110_stream_encoder *enc110, uint32_t packet_index, - const struct encoder_info_packet *info_packet) + const struct dc_info_packet *info_packet) { uint32_t cont, send, line; @@ -314,11 +298,11 @@ static void dce110_stream_encoder_dp_set_stream_attribute( switch (crtc_timing->pixel_encoding) { case PIXEL_ENCODING_YCBCR422: REG_UPDATE(DP_PIXEL_FORMAT, DP_PIXEL_ENCODING, - DP_PIXEL_ENCODING_YCBCR422); + DP_PIXEL_ENCODING_TYPE_YCBCR422); break; case PIXEL_ENCODING_YCBCR444: REG_UPDATE(DP_PIXEL_FORMAT, DP_PIXEL_ENCODING, - DP_PIXEL_ENCODING_YCBCR444); + DP_PIXEL_ENCODING_TYPE_YCBCR444); if (crtc_timing->flags.Y_ONLY) if (crtc_timing->display_color_depth != COLOR_DEPTH_666) @@ -326,7 +310,7 @@ static void dce110_stream_encoder_dp_set_stream_attribute( * Color depth of Y-only could be * 8, 10, 12, 16 bits */ REG_UPDATE(DP_PIXEL_FORMAT, DP_PIXEL_ENCODING, - DP_PIXEL_ENCODING_Y_ONLY); + DP_PIXEL_ENCODING_TYPE_Y_ONLY); /* Note: DP_MSA_MISC1 bit 7 is the indicator * of Y-only mode. * This bit is set in HW if register @@ -334,7 +318,7 @@ static void dce110_stream_encoder_dp_set_stream_attribute( break; case PIXEL_ENCODING_YCBCR420: REG_UPDATE(DP_PIXEL_FORMAT, DP_PIXEL_ENCODING, - DP_PIXEL_ENCODING_YCBCR420); + DP_PIXEL_ENCODING_TYPE_YCBCR420); if (enc110->se_mask->DP_VID_M_DOUBLE_VALUE_EN) REG_UPDATE(DP_VID_TIMING, DP_VID_M_DOUBLE_VALUE_EN, 1); @@ -345,7 +329,7 @@ static void dce110_stream_encoder_dp_set_stream_attribute( break; default: REG_UPDATE(DP_PIXEL_FORMAT, DP_PIXEL_ENCODING, - DP_PIXEL_ENCODING_RGB444); + DP_PIXEL_ENCODING_TYPE_RGB444); break; } @@ -363,20 +347,20 @@ static void dce110_stream_encoder_dp_set_stream_attribute( break; case COLOR_DEPTH_888: REG_UPDATE(DP_PIXEL_FORMAT, DP_COMPONENT_DEPTH, - DP_COMPONENT_DEPTH_8BPC); + DP_COMPONENT_PIXEL_DEPTH_8BPC); break; case COLOR_DEPTH_101010: REG_UPDATE(DP_PIXEL_FORMAT, DP_COMPONENT_DEPTH, - DP_COMPONENT_DEPTH_10BPC); + DP_COMPONENT_PIXEL_DEPTH_10BPC); break; case COLOR_DEPTH_121212: REG_UPDATE(DP_PIXEL_FORMAT, DP_COMPONENT_DEPTH, - DP_COMPONENT_DEPTH_12BPC); + DP_COMPONENT_PIXEL_DEPTH_12BPC); break; default: REG_UPDATE(DP_PIXEL_FORMAT, DP_COMPONENT_DEPTH, - DP_COMPONENT_DEPTH_6BPC); + DP_COMPONENT_PIXEL_DEPTH_6BPC); break; } @@ -700,11 +684,11 @@ static void dce110_stream_encoder_set_mst_bandwidth( struct fixed31_32 avg_time_slots_per_mtp) { struct dce110_stream_encoder *enc110 = DCE110STRENC_FROM_STRENC(enc); - uint32_t x = dal_fixed31_32_floor( + uint32_t x = dc_fixpt_floor( avg_time_slots_per_mtp); - uint32_t y = dal_fixed31_32_ceil( - dal_fixed31_32_shl( - dal_fixed31_32_sub_int( + uint32_t y = dc_fixpt_ceil( + dc_fixpt_shl( + dc_fixpt_sub_int( avg_time_slots_per_mtp, x), 26)); @@ -736,7 +720,8 @@ static void dce110_stream_encoder_update_hdmi_info_packets( const uint32_t *content = (const uint32_t *) &info_frame->avi.sb[0]; /*we need turn on clock before programming AFMT block*/ - REG_UPDATE(AFMT_CNTL, AFMT_AUDIO_CLOCK_EN, 1); + if (REG(AFMT_CNTL)) + REG_UPDATE(AFMT_CNTL, AFMT_AUDIO_CLOCK_EN, 1); REG_WRITE(AFMT_AVI_INFO0, content[0]); @@ -836,7 +821,7 @@ static void dce110_stream_encoder_update_dp_info_packets( const struct encoder_info_frame *info_frame) { struct dce110_stream_encoder *enc110 = DCE110STRENC_FROM_STRENC(enc); - uint32_t value = REG_READ(DP_SEC_CNTL); + uint32_t value = 0; if (info_frame->vsc.valid) dce110_update_generic_info_packet( @@ -870,6 +855,7 @@ static void dce110_stream_encoder_update_dp_info_packets( * Therefore we need to enable master bit * if at least on of the fields is not 0 */ + value = REG_READ(DP_SEC_CNTL); if (value) REG_UPDATE(DP_SEC_CNTL, DP_SEC_STREAM_ENABLE, 1); } @@ -879,7 +865,7 @@ static void dce110_stream_encoder_stop_dp_info_packets( { /* stop generic packets on DP */ struct dce110_stream_encoder *enc110 = DCE110STRENC_FROM_STRENC(enc); - uint32_t value = REG_READ(DP_SEC_CNTL); + uint32_t value = 0; if (enc110->se_mask->DP_SEC_AVI_ENABLE) { REG_SET_7(DP_SEC_CNTL, 0, @@ -892,25 +878,10 @@ static void dce110_stream_encoder_stop_dp_info_packets( DP_SEC_STREAM_ENABLE, 0); } -#if defined(CONFIG_DRM_AMD_DC_DCN1_0) - if (enc110->se_mask->DP_SEC_GSP7_ENABLE) { - REG_SET_10(DP_SEC_CNTL, 0, - DP_SEC_GSP0_ENABLE, 0, - DP_SEC_GSP1_ENABLE, 0, - DP_SEC_GSP2_ENABLE, 0, - DP_SEC_GSP3_ENABLE, 0, - DP_SEC_GSP4_ENABLE, 0, - DP_SEC_GSP5_ENABLE, 0, - DP_SEC_GSP6_ENABLE, 0, - DP_SEC_GSP7_ENABLE, 0, - DP_SEC_MPG_ENABLE, 0, - DP_SEC_STREAM_ENABLE, 0); - } -#endif /* this register shared with audio info frame. * therefore we need to keep master enabled * if at least one of the fields is not 0 */ - + value = REG_READ(DP_SEC_CNTL); if (value) REG_UPDATE(DP_SEC_CNTL, DP_SEC_STREAM_ENABLE, 1); @@ -1513,7 +1484,7 @@ static void dce110_se_disable_dp_audio( struct stream_encoder *enc) { struct dce110_stream_encoder *enc110 = DCE110STRENC_FROM_STRENC(enc); - uint32_t value = REG_READ(DP_SEC_CNTL); + uint32_t value = 0; /* Disable Audio packets */ REG_UPDATE_5(DP_SEC_CNTL, @@ -1525,6 +1496,7 @@ static void dce110_se_disable_dp_audio( /* This register shared with encoder info frame. Therefore we need to keep master enabled if at least on of the fields is not 0 */ + value = REG_READ(DP_SEC_CNTL); if (value != 0) REG_UPDATE(DP_SEC_CNTL, DP_SEC_STREAM_ENABLE, 1); diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_transform.c b/drivers/gpu/drm/amd/display/dc/dce/dce_transform.c index 832c5daada35..a02e719d7794 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_transform.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_transform.c @@ -41,7 +41,7 @@ #define DC_LOGGER \ xfm_dce->base.ctx->logger -#define IDENTITY_RATIO(ratio) (dal_fixed31_32_u2d19(ratio) == (1 << 19)) +#define IDENTITY_RATIO(ratio) (dc_fixpt_u2d19(ratio) == (1 << 19)) #define GAMUT_MATRIX_SIZE 12 #define SCL_PHASES 16 @@ -256,27 +256,27 @@ static void calculate_inits( struct fixed31_32 v_init; inits->h_int_scale_ratio = - dal_fixed31_32_u2d19(data->ratios.horz) << 5; + dc_fixpt_u2d19(data->ratios.horz) << 5; inits->v_int_scale_ratio = - dal_fixed31_32_u2d19(data->ratios.vert) << 5; + dc_fixpt_u2d19(data->ratios.vert) << 5; h_init = - dal_fixed31_32_div_int( - dal_fixed31_32_add( + dc_fixpt_div_int( + dc_fixpt_add( data->ratios.horz, - dal_fixed31_32_from_int(data->taps.h_taps + 1)), + dc_fixpt_from_int(data->taps.h_taps + 1)), 2); - inits->h_init.integer = dal_fixed31_32_floor(h_init); - inits->h_init.fraction = dal_fixed31_32_u0d19(h_init) << 5; + inits->h_init.integer = dc_fixpt_floor(h_init); + inits->h_init.fraction = dc_fixpt_u0d19(h_init) << 5; v_init = - dal_fixed31_32_div_int( - dal_fixed31_32_add( + dc_fixpt_div_int( + dc_fixpt_add( data->ratios.vert, - dal_fixed31_32_from_int(data->taps.v_taps + 1)), + dc_fixpt_from_int(data->taps.v_taps + 1)), 2); - inits->v_init.integer = dal_fixed31_32_floor(v_init); - inits->v_init.fraction = dal_fixed31_32_u0d19(v_init) << 5; + inits->v_init.integer = dc_fixpt_floor(v_init); + inits->v_init.fraction = dc_fixpt_u0d19(v_init) << 5; } static void program_scl_ratios_inits( diff --git a/drivers/gpu/drm/amd/display/dc/dce100/dce100_resource.c b/drivers/gpu/drm/amd/display/dc/dce100/dce100_resource.c index 3092f76bdb75..38ec0d609297 100644 --- a/drivers/gpu/drm/amd/display/dc/dce100/dce100_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dce100/dce100_resource.c @@ -733,38 +733,6 @@ enum dc_status dce100_add_stream_to_ctx( return result; } -enum dc_status dce100_validate_guaranteed( - struct dc *dc, - struct dc_stream_state *dc_stream, - struct dc_state *context) -{ - enum dc_status result = DC_ERROR_UNEXPECTED; - - context->streams[0] = dc_stream; - dc_stream_retain(context->streams[0]); - context->stream_count++; - - result = resource_map_pool_resources(dc, context, dc_stream); - - if (result == DC_OK) - result = resource_map_clock_resources(dc, context, dc_stream); - - if (result == DC_OK) - result = build_mapped_resource(dc, context, dc_stream); - - if (result == DC_OK) { - validate_guaranteed_copy_streams( - context, dc->caps.max_streams); - result = resource_build_scaling_params_for_context(dc, context); - } - - if (result == DC_OK) - if (!dce100_validate_bandwidth(dc, context)) - result = DC_FAIL_BANDWIDTH_VALIDATE; - - return result; -} - static void dce100_destroy_resource_pool(struct resource_pool **pool) { struct dce110_resource_pool *dce110_pool = TO_DCE110_RES_POOL(*pool); @@ -786,7 +754,6 @@ enum dc_status dce100_validate_plane(const struct dc_plane_state *plane_state, s static const struct resource_funcs dce100_res_pool_funcs = { .destroy = dce100_destroy_resource_pool, .link_enc_create = dce100_link_encoder_create, - .validate_guaranteed = dce100_validate_guaranteed, .validate_bandwidth = dce100_validate_bandwidth, .validate_plane = dce100_validate_plane, .add_stream_to_ctx = dce100_add_stream_to_ctx, diff --git a/drivers/gpu/drm/amd/display/dc/dce110/dce110_compressor.c b/drivers/gpu/drm/amd/display/dc/dce110/dce110_compressor.c index 9150d2694450..e2994d337044 100644 --- a/drivers/gpu/drm/amd/display/dc/dce110/dce110_compressor.c +++ b/drivers/gpu/drm/amd/display/dc/dce110/dce110_compressor.c @@ -121,10 +121,10 @@ static void reset_lb_on_vblank(struct dc_context *ctx) frame_count = dm_read_reg(ctx, mmCRTC_STATUS_FRAME_COUNT); - for (retry = 100; retry > 0; retry--) { + for (retry = 10000; retry > 0; retry--) { if (frame_count != dm_read_reg(ctx, mmCRTC_STATUS_FRAME_COUNT)) break; - msleep(1); + udelay(10); } if (!retry) dm_error("Frame count did not increase for 100ms.\n"); @@ -147,14 +147,14 @@ static void wait_for_fbc_state_changed( uint32_t addr = mmFBC_STATUS; uint32_t value; - while (counter < 10) { + while (counter < 1000) { value = dm_read_reg(cp110->base.ctx, addr); if (get_reg_field_value( value, FBC_STATUS, FBC_ENABLE_STATUS) == enabled) break; - msleep(10); + udelay(100); counter++; } diff --git a/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c b/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c index d0575999f172..c29052b6da5a 100644 --- a/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c +++ b/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c @@ -70,8 +70,9 @@ #define CTX \ hws->ctx -#define DC_LOGGER \ - ctx->logger + +#define DC_LOGGER_INIT() + #define REG(reg)\ hws->regs->reg @@ -279,7 +280,9 @@ dce110_set_input_transfer_func(struct pipe_ctx *pipe_ctx, build_prescale_params(&prescale_params, plane_state); ipp->funcs->ipp_program_prescale(ipp, &prescale_params); - if (plane_state->gamma_correction && dce_use_lut(plane_state->format)) + if (plane_state->gamma_correction && + !plane_state->gamma_correction->is_identity && + dce_use_lut(plane_state->format)) ipp->funcs->ipp_program_input_lut(ipp, plane_state->gamma_correction); if (tf == NULL) { @@ -506,19 +509,19 @@ dce110_translate_regamma_to_hw_format(const struct dc_transfer_func *output_tf, rgb_resulted[hw_points - 1].green = output_tf->tf_pts.green[start_index]; rgb_resulted[hw_points - 1].blue = output_tf->tf_pts.blue[start_index]; - arr_points[0].x = dal_fixed31_32_pow(dal_fixed31_32_from_int(2), - dal_fixed31_32_from_int(region_start)); - arr_points[1].x = dal_fixed31_32_pow(dal_fixed31_32_from_int(2), - dal_fixed31_32_from_int(region_end)); + arr_points[0].x = dc_fixpt_pow(dc_fixpt_from_int(2), + dc_fixpt_from_int(region_start)); + arr_points[1].x = dc_fixpt_pow(dc_fixpt_from_int(2), + dc_fixpt_from_int(region_end)); y_r = rgb_resulted[0].red; y_g = rgb_resulted[0].green; y_b = rgb_resulted[0].blue; - y1_min = dal_fixed31_32_min(y_r, dal_fixed31_32_min(y_g, y_b)); + y1_min = dc_fixpt_min(y_r, dc_fixpt_min(y_g, y_b)); arr_points[0].y = y1_min; - arr_points[0].slope = dal_fixed31_32_div(arr_points[0].y, + arr_points[0].slope = dc_fixpt_div(arr_points[0].y, arr_points[0].x); y_r = rgb_resulted[hw_points - 1].red; @@ -528,21 +531,21 @@ dce110_translate_regamma_to_hw_format(const struct dc_transfer_func *output_tf, /* see comment above, m_arrPoints[1].y should be the Y value for the * region end (m_numOfHwPoints), not last HW point(m_numOfHwPoints - 1) */ - y3_max = dal_fixed31_32_max(y_r, dal_fixed31_32_max(y_g, y_b)); + y3_max = dc_fixpt_max(y_r, dc_fixpt_max(y_g, y_b)); arr_points[1].y = y3_max; - arr_points[1].slope = dal_fixed31_32_zero; + arr_points[1].slope = dc_fixpt_zero; if (output_tf->tf == TRANSFER_FUNCTION_PQ) { /* for PQ, we want to have a straight line from last HW X point, * and the slope to be such that we hit 1.0 at 10000 nits. */ - const struct fixed31_32 end_value = dal_fixed31_32_from_int(125); + const struct fixed31_32 end_value = dc_fixpt_from_int(125); - arr_points[1].slope = dal_fixed31_32_div( - dal_fixed31_32_sub(dal_fixed31_32_one, arr_points[1].y), - dal_fixed31_32_sub(end_value, arr_points[1].x)); + arr_points[1].slope = dc_fixpt_div( + dc_fixpt_sub(dc_fixpt_one, arr_points[1].y), + dc_fixpt_sub(end_value, arr_points[1].x)); } regamma_params->hw_points_num = hw_points; @@ -566,16 +569,16 @@ dce110_translate_regamma_to_hw_format(const struct dc_transfer_func *output_tf, i = 1; while (i != hw_points + 1) { - if (dal_fixed31_32_lt(rgb_plus_1->red, rgb->red)) + if (dc_fixpt_lt(rgb_plus_1->red, rgb->red)) rgb_plus_1->red = rgb->red; - if (dal_fixed31_32_lt(rgb_plus_1->green, rgb->green)) + if (dc_fixpt_lt(rgb_plus_1->green, rgb->green)) rgb_plus_1->green = rgb->green; - if (dal_fixed31_32_lt(rgb_plus_1->blue, rgb->blue)) + if (dc_fixpt_lt(rgb_plus_1->blue, rgb->blue)) rgb_plus_1->blue = rgb->blue; - rgb->delta_red = dal_fixed31_32_sub(rgb_plus_1->red, rgb->red); - rgb->delta_green = dal_fixed31_32_sub(rgb_plus_1->green, rgb->green); - rgb->delta_blue = dal_fixed31_32_sub(rgb_plus_1->blue, rgb->blue); + rgb->delta_red = dc_fixpt_sub(rgb_plus_1->red, rgb->red); + rgb->delta_green = dc_fixpt_sub(rgb_plus_1->green, rgb->green); + rgb->delta_blue = dc_fixpt_sub(rgb_plus_1->blue, rgb->blue); ++rgb_plus_1; ++rgb; @@ -851,6 +854,28 @@ void hwss_edp_power_control( if (power_up != is_panel_powered_on(hwseq)) { /* Send VBIOS command to prompt eDP panel power */ + if (power_up) { + unsigned long long current_ts = dm_get_timestamp(ctx); + unsigned long long duration_in_ms = + dm_get_elapse_time_in_ns( + ctx, + current_ts, + div64_u64(link->link_trace.time_stamp.edp_poweroff, 1000000)); + unsigned long long wait_time_ms = 0; + + /* max 500ms from LCDVDD off to on */ + if (link->link_trace.time_stamp.edp_poweroff == 0) + wait_time_ms = 500; + else if (duration_in_ms < 500) + wait_time_ms = 500 - duration_in_ms; + + if (wait_time_ms) { + msleep(wait_time_ms); + dm_output_to_console("%s: wait %lld ms to power on eDP.\n", + __func__, wait_time_ms); + } + + } DC_LOG_HW_RESUME_S3( "%s: Panel Power action: %s\n", @@ -864,9 +889,14 @@ void hwss_edp_power_control( cntl.coherent = false; cntl.lanes_number = LANE_COUNT_FOUR; cntl.hpd_sel = link->link_enc->hpd_source; - bp_result = link_transmitter_control(ctx->dc_bios, &cntl); + if (!power_up) + /*save driver power off time stamp*/ + link->link_trace.time_stamp.edp_poweroff = dm_get_timestamp(ctx); + else + link->link_trace.time_stamp.edp_poweron = dm_get_timestamp(ctx); + if (bp_result != BP_RESULT_OK) DC_LOG_ERROR( "%s: Panel Power bp_result: %d\n", @@ -974,9 +1004,9 @@ void dce110_disable_stream(struct pipe_ctx *pipe_ctx, int option) /*don't free audio if it is from retrain or internal disable stream*/ if (option == FREE_ACQUIRED_RESOURCE && dc->caps.dynamic_audio == true) { /*we have to dynamic arbitrate the audio endpoints*/ - pipe_ctx->stream_res.audio = NULL; /*we free the resource, need reset is_audio_acquired*/ update_audio_usage(&dc->current_state->res_ctx, dc->res_pool, pipe_ctx->stream_res.audio, false); + pipe_ctx->stream_res.audio = NULL; } /* TODO: notify audio driver for if audio modes list changed @@ -1011,7 +1041,7 @@ void dce110_unblank_stream(struct pipe_ctx *pipe_ctx, if (link->local_sink && link->local_sink->sink_signal == SIGNAL_TYPE_EDP) { link->dc->hwss.edp_backlight_control(link, true); - stream->bl_pwm_level = 0; + stream->bl_pwm_level = EDP_BACKLIGHT_RAMP_DISABLE_LEVEL; } } void dce110_blank_stream(struct pipe_ctx *pipe_ctx) @@ -1203,7 +1233,7 @@ static void program_scaler(const struct dc *dc, &pipe_ctx->plane_res.scl_data); } -static enum dc_status dce110_prog_pixclk_crtc_otg( +static enum dc_status dce110_enable_stream_timing( struct pipe_ctx *pipe_ctx, struct dc_state *context, struct dc *dc) @@ -1269,7 +1299,7 @@ static enum dc_status apply_single_controller_ctx_to_hw( pipe_ctx[pipe_ctx->pipe_idx]; /* */ - dc->hwss.prog_pixclk_crtc_otg(pipe_ctx, context, dc); + dc->hwss.enable_stream_timing(pipe_ctx, context, dc); /* FPGA does not program backend */ if (IS_FPGA_MAXIMUS_DC(dc->ctx->dce_environment)) { @@ -1441,6 +1471,17 @@ static void disable_vga_and_power_gate_all_controllers( } } +static struct dc_link *get_link_for_edp(struct dc *dc) +{ + int i; + + for (i = 0; i < dc->link_count; i++) { + if (dc->links[i]->connector_signal == SIGNAL_TYPE_EDP) + return dc->links[i]; + } + return NULL; +} + static struct dc_link *get_link_for_edp_not_in_use( struct dc *dc, struct dc_state *context) @@ -1475,20 +1516,21 @@ static struct dc_link *get_link_for_edp_not_in_use( */ void dce110_enable_accelerated_mode(struct dc *dc, struct dc_state *context) { - struct dc_bios *dcb = dc->ctx->dc_bios; - - /* vbios already light up eDP, so we can leverage vbios and skip eDP - * programming - */ - bool can_eDP_fast_boot_optimize = - (dcb->funcs->get_vga_enabled_displays(dc->ctx->dc_bios) == ATOM_DISPLAY_LCD1_ACTIVE); - - /* if OS doesn't light up eDP and eDP link is available, we want to disable */ struct dc_link *edp_link_to_turnoff = NULL; + struct dc_link *edp_link = get_link_for_edp(dc); + bool can_eDP_fast_boot_optimize = false; + + if (edp_link) { + can_eDP_fast_boot_optimize = + edp_link->link_enc->funcs->is_dig_enabled(edp_link->link_enc); + } if (can_eDP_fast_boot_optimize) { edp_link_to_turnoff = get_link_for_edp_not_in_use(dc, context); + /* if OS doesn't light up eDP and eDP link is available, we want to disable + * If resume from S4/S5, should optimization. + */ if (!edp_link_to_turnoff) dc->apply_edp_fast_boot_optimization = true; } @@ -1544,6 +1586,7 @@ static void dce110_set_displaymarks( pipe_ctx->plane_res.mi, context->bw.dce.nbp_state_change_wm_ns[num_pipes], context->bw.dce.stutter_exit_wm_ns[num_pipes], + context->bw.dce.stutter_entry_wm_ns[num_pipes], context->bw.dce.urgent_wm_ns[num_pipes], total_dest_line_time_ns); if (i == underlay_idx) { @@ -1569,6 +1612,7 @@ static void set_safe_displaymarks( MAX_WATERMARK, MAX_WATERMARK, MAX_WATERMARK, MAX_WATERMARK }; struct dce_watermarks nbp_marks = { SAFE_NBP_MARK, SAFE_NBP_MARK, SAFE_NBP_MARK, SAFE_NBP_MARK }; + struct dce_watermarks min_marks = { 0, 0, 0, 0}; for (i = 0; i < MAX_PIPES; i++) { if (res_ctx->pipe_ctx[i].stream == NULL || res_ctx->pipe_ctx[i].plane_res.mi == NULL) @@ -1578,6 +1622,7 @@ static void set_safe_displaymarks( res_ctx->pipe_ctx[i].plane_res.mi, nbp_marks, max_marks, + min_marks, max_marks, MAX_WATERMARK); @@ -1803,6 +1848,9 @@ static bool should_enable_fbc(struct dc *dc, } } + /* Pipe context should be found */ + ASSERT(pipe_ctx); + /* Only supports eDP */ if (pipe_ctx->stream->sink->link->connector_signal != SIGNAL_TYPE_EDP) return false; @@ -2221,74 +2269,6 @@ static void program_gamut_remap(struct pipe_ctx *pipe_ctx) pipe_ctx->plane_res.xfm->funcs->transform_set_gamut_remap(pipe_ctx->plane_res.xfm, &adjust); } - -/** - * TODO REMOVE, USE UPDATE INSTEAD - */ -static void set_plane_config( - const struct dc *dc, - struct pipe_ctx *pipe_ctx, - struct resource_context *res_ctx) -{ - struct mem_input *mi = pipe_ctx->plane_res.mi; - struct dc_plane_state *plane_state = pipe_ctx->plane_state; - struct xfm_grph_csc_adjustment adjust; - struct out_csc_color_matrix tbl_entry; - unsigned int i; - - memset(&adjust, 0, sizeof(adjust)); - memset(&tbl_entry, 0, sizeof(tbl_entry)); - adjust.gamut_adjust_type = GRAPHICS_GAMUT_ADJUST_TYPE_BYPASS; - - dce_enable_fe_clock(dc->hwseq, mi->inst, true); - - set_default_colors(pipe_ctx); - if (pipe_ctx->stream->csc_color_matrix.enable_adjustment == true) { - tbl_entry.color_space = - pipe_ctx->stream->output_color_space; - - for (i = 0; i < 12; i++) - tbl_entry.regval[i] = - pipe_ctx->stream->csc_color_matrix.matrix[i]; - - pipe_ctx->plane_res.xfm->funcs->opp_set_csc_adjustment - (pipe_ctx->plane_res.xfm, &tbl_entry); - } - - if (pipe_ctx->stream->gamut_remap_matrix.enable_remap == true) { - adjust.gamut_adjust_type = GRAPHICS_GAMUT_ADJUST_TYPE_SW; - - for (i = 0; i < CSC_TEMPERATURE_MATRIX_SIZE; i++) - adjust.temperature_matrix[i] = - pipe_ctx->stream->gamut_remap_matrix.matrix[i]; - } - - pipe_ctx->plane_res.xfm->funcs->transform_set_gamut_remap(pipe_ctx->plane_res.xfm, &adjust); - - pipe_ctx->plane_res.scl_data.lb_params.alpha_en = pipe_ctx->bottom_pipe != 0; - program_scaler(dc, pipe_ctx); - - program_surface_visibility(dc, pipe_ctx); - - mi->funcs->mem_input_program_surface_config( - mi, - plane_state->format, - &plane_state->tiling_info, - &plane_state->plane_size, - plane_state->rotation, - NULL, - false); - if (mi->funcs->set_blank) - mi->funcs->set_blank(mi, pipe_ctx->plane_state->visible); - - if (dc->config.gpu_vm_support) - mi->funcs->mem_input_program_pte_vm( - pipe_ctx->plane_res.mi, - plane_state->format, - &plane_state->tiling_info, - plane_state->rotation); -} - static void update_plane_addr(const struct dc *dc, struct pipe_ctx *pipe_ctx) { @@ -2699,8 +2679,11 @@ static void dce110_program_front_end_for_pipe( struct dc_plane_state *plane_state = pipe_ctx->plane_state; struct xfm_grph_csc_adjustment adjust; struct out_csc_color_matrix tbl_entry; +#if defined(CONFIG_DRM_AMD_DC_FBC) + unsigned int underlay_idx = dc->res_pool->underlay_pipe_index; +#endif unsigned int i; - struct dc_context *ctx = dc->ctx; + DC_LOGGER_INIT(); memset(&tbl_entry, 0, sizeof(tbl_entry)); if (dc->current_state) @@ -2740,7 +2723,9 @@ static void dce110_program_front_end_for_pipe( program_scaler(dc, pipe_ctx); #if defined(CONFIG_DRM_AMD_DC_FBC) - if (dc->fbc_compressor && old_pipe->stream) { + /* fbc not applicable on Underlay pipe */ + if (dc->fbc_compressor && old_pipe->stream && + pipe_ctx->pipe_idx != underlay_idx) { if (plane_state->tiling_info.gfx8.array_mode == DC_ARRAY_LINEAR_GENERAL) dc->fbc_compressor->funcs->disable_fbc(dc->fbc_compressor); else @@ -2776,13 +2761,13 @@ static void dce110_program_front_end_for_pipe( dc->hwss.set_output_transfer_func(pipe_ctx, pipe_ctx->stream); DC_LOG_SURFACE( - "Pipe:%d 0x%x: addr hi:0x%x, " + "Pipe:%d %p: addr hi:0x%x, " "addr low:0x%x, " "src: %d, %d, %d," " %d; dst: %d, %d, %d, %d;" "clip: %d, %d, %d, %d\n", pipe_ctx->pipe_idx, - pipe_ctx->plane_state, + (void *) pipe_ctx->plane_state, pipe_ctx->plane_state->address.grph.addr.high_part, pipe_ctx->plane_state->address.grph.addr.low_part, pipe_ctx->plane_state->src_rect.x, @@ -2970,7 +2955,6 @@ static const struct hw_sequencer_funcs dce110_funcs = { .init_hw = init_hw, .apply_ctx_to_hw = dce110_apply_ctx_to_hw, .apply_ctx_for_surface = dce110_apply_ctx_for_surface, - .set_plane_config = set_plane_config, .update_plane_addr = update_plane_addr, .update_pending_status = dce110_update_pending_status, .set_input_transfer_func = dce110_set_input_transfer_func, @@ -2993,7 +2977,7 @@ static const struct hw_sequencer_funcs dce110_funcs = { .get_position = get_position, .set_static_screen_control = set_static_screen_control, .reset_hw_ctx_wrap = dce110_reset_hw_ctx_wrap, - .prog_pixclk_crtc_otg = dce110_prog_pixclk_crtc_otg, + .enable_stream_timing = dce110_enable_stream_timing, .setup_stereo = NULL, .set_avmute = dce110_set_avmute, .wait_for_mpcc_disconnect = dce110_wait_for_mpcc_disconnect, diff --git a/drivers/gpu/drm/amd/display/dc/dce110/dce110_mem_input_v.c b/drivers/gpu/drm/amd/display/dc/dce110/dce110_mem_input_v.c index 7bab8c6d2a73..0564c8e31252 100644 --- a/drivers/gpu/drm/amd/display/dc/dce110/dce110_mem_input_v.c +++ b/drivers/gpu/drm/amd/display/dc/dce110/dce110_mem_input_v.c @@ -923,6 +923,7 @@ void dce_mem_input_v_program_display_marks( struct mem_input *mem_input, struct dce_watermarks nbp, struct dce_watermarks stutter, + struct dce_watermarks stutter_enter, struct dce_watermarks urgent, uint32_t total_dest_line_time_ns) { diff --git a/drivers/gpu/drm/amd/display/dc/dce110/dce110_resource.c b/drivers/gpu/drm/amd/display/dc/dce110/dce110_resource.c index b1f14be20fdf..ee33786bdef6 100644 --- a/drivers/gpu/drm/amd/display/dc/dce110/dce110_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dce110/dce110_resource.c @@ -930,38 +930,6 @@ static enum dc_status dce110_add_stream_to_ctx( return result; } -static enum dc_status dce110_validate_guaranteed( - struct dc *dc, - struct dc_stream_state *dc_stream, - struct dc_state *context) -{ - enum dc_status result = DC_ERROR_UNEXPECTED; - - context->streams[0] = dc_stream; - dc_stream_retain(context->streams[0]); - context->stream_count++; - - result = resource_map_pool_resources(dc, context, dc_stream); - - if (result == DC_OK) - result = resource_map_clock_resources(dc, context, dc_stream); - - if (result == DC_OK) - result = build_mapped_resource(dc, context, dc_stream); - - if (result == DC_OK) { - validate_guaranteed_copy_streams( - context, dc->caps.max_streams); - result = resource_build_scaling_params_for_context(dc, context); - } - - if (result == DC_OK) - if (!dce110_validate_bandwidth(dc, context)) - result = DC_FAIL_BANDWIDTH_VALIDATE; - - return result; -} - static struct pipe_ctx *dce110_acquire_underlay( struct dc_state *context, const struct resource_pool *pool, @@ -1036,7 +1004,6 @@ static void dce110_destroy_resource_pool(struct resource_pool **pool) static const struct resource_funcs dce110_res_pool_funcs = { .destroy = dce110_destroy_resource_pool, .link_enc_create = dce110_link_encoder_create, - .validate_guaranteed = dce110_validate_guaranteed, .validate_bandwidth = dce110_validate_bandwidth, .validate_plane = dce110_validate_plane, .acquire_idle_pipe_for_layer = dce110_acquire_underlay, diff --git a/drivers/gpu/drm/amd/display/dc/dce110/dce110_timing_generator.c b/drivers/gpu/drm/amd/display/dc/dce110/dce110_timing_generator.c index be7153924a70..1b2fe0df347f 100644 --- a/drivers/gpu/drm/amd/display/dc/dce110/dce110_timing_generator.c +++ b/drivers/gpu/drm/amd/display/dc/dce110/dce110_timing_generator.c @@ -431,14 +431,6 @@ void dce110_timing_generator_set_drr( 0, CRTC_V_TOTAL_CONTROL, CRTC_SET_V_TOTAL_MIN_MASK); - set_reg_field_value(v_total_min, - 0, - CRTC_V_TOTAL_MIN, - CRTC_V_TOTAL_MIN); - set_reg_field_value(v_total_max, - 0, - CRTC_V_TOTAL_MAX, - CRTC_V_TOTAL_MAX); set_reg_field_value(v_total_cntl, 0, CRTC_V_TOTAL_CONTROL, @@ -447,6 +439,14 @@ void dce110_timing_generator_set_drr( 0, CRTC_V_TOTAL_CONTROL, CRTC_V_TOTAL_MAX_SEL); + set_reg_field_value(v_total_min, + 0, + CRTC_V_TOTAL_MIN, + CRTC_V_TOTAL_MIN); + set_reg_field_value(v_total_max, + 0, + CRTC_V_TOTAL_MAX, + CRTC_V_TOTAL_MAX); set_reg_field_value(v_total_cntl, 0, CRTC_V_TOTAL_CONTROL, diff --git a/drivers/gpu/drm/amd/display/dc/dce110/dce110_timing_generator_v.c b/drivers/gpu/drm/amd/display/dc/dce110/dce110_timing_generator_v.c index 8ad04816e7d3..a3cef60380ed 100644 --- a/drivers/gpu/drm/amd/display/dc/dce110/dce110_timing_generator_v.c +++ b/drivers/gpu/drm/amd/display/dc/dce110/dce110_timing_generator_v.c @@ -648,12 +648,6 @@ static void dce110_timing_generator_v_disable_vga( return; } -static bool dce110_tg_v_is_blanked(struct timing_generator *tg) -{ - /* Signal comes from the primary pipe, underlay is never blanked. */ - return false; -} - /** ******************************************************************************************** * * DCE11 Timing Generator Constructor / Destructor @@ -670,7 +664,6 @@ static const struct timing_generator_funcs dce110_tg_v_funcs = { .set_early_control = dce110_timing_generator_v_set_early_control, .wait_for_state = dce110_timing_generator_v_wait_for_state, .set_blank = dce110_timing_generator_v_set_blank, - .is_blanked = dce110_tg_v_is_blanked, .set_colors = dce110_timing_generator_v_set_colors, .set_overscan_blank_color = dce110_timing_generator_v_set_overscan_color_black, diff --git a/drivers/gpu/drm/amd/display/dc/dce110/dce110_transform_v.c b/drivers/gpu/drm/amd/display/dc/dce110/dce110_transform_v.c index 8ba3c12fc608..a7dce060204f 100644 --- a/drivers/gpu/drm/amd/display/dc/dce110/dce110_transform_v.c +++ b/drivers/gpu/drm/amd/display/dc/dce110/dce110_transform_v.c @@ -373,13 +373,13 @@ static void calculate_inits( struct rect *chroma_viewport) { inits->h_int_scale_ratio_luma = - dal_fixed31_32_u2d19(data->ratios.horz) << 5; + dc_fixpt_u2d19(data->ratios.horz) << 5; inits->v_int_scale_ratio_luma = - dal_fixed31_32_u2d19(data->ratios.vert) << 5; + dc_fixpt_u2d19(data->ratios.vert) << 5; inits->h_int_scale_ratio_chroma = - dal_fixed31_32_u2d19(data->ratios.horz_c) << 5; + dc_fixpt_u2d19(data->ratios.horz_c) << 5; inits->v_int_scale_ratio_chroma = - dal_fixed31_32_u2d19(data->ratios.vert_c) << 5; + dc_fixpt_u2d19(data->ratios.vert_c) << 5; inits->h_init_luma.integer = 1; inits->v_init_luma.integer = 1; diff --git a/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c b/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c index cd1e3f72c44e..00c0a1ef15eb 100644 --- a/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c @@ -430,7 +430,7 @@ static struct stream_encoder *dce112_stream_encoder_create( if (!enc110) return NULL; - + dce110_stream_encoder_construct(enc110, ctx, ctx->dc_bios, eng_id, &stream_enc_regs[eng_id], &se_shift, &se_mask); @@ -867,38 +867,6 @@ enum dc_status dce112_add_stream_to_ctx( return result; } -enum dc_status dce112_validate_guaranteed( - struct dc *dc, - struct dc_stream_state *stream, - struct dc_state *context) -{ - enum dc_status result = DC_ERROR_UNEXPECTED; - - context->streams[0] = stream; - dc_stream_retain(context->streams[0]); - context->stream_count++; - - result = resource_map_pool_resources(dc, context, stream); - - if (result == DC_OK) - result = resource_map_phy_clock_resources(dc, context, stream); - - if (result == DC_OK) - result = build_mapped_resource(dc, context, stream); - - if (result == DC_OK) { - validate_guaranteed_copy_streams( - context, dc->caps.max_streams); - result = resource_build_scaling_params_for_context(dc, context); - } - - if (result == DC_OK) - if (!dce112_validate_bandwidth(dc, context)) - result = DC_FAIL_BANDWIDTH_VALIDATE; - - return result; -} - enum dc_status dce112_validate_global( struct dc *dc, struct dc_state *context) @@ -921,7 +889,6 @@ static void dce112_destroy_resource_pool(struct resource_pool **pool) static const struct resource_funcs dce112_res_pool_funcs = { .destroy = dce112_destroy_resource_pool, .link_enc_create = dce112_link_encoder_create, - .validate_guaranteed = dce112_validate_guaranteed, .validate_bandwidth = dce112_validate_bandwidth, .validate_plane = dce100_validate_plane, .add_stream_to_ctx = dce112_add_stream_to_ctx, diff --git a/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.h b/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.h index d5c19d34eb0a..95a403396219 100644 --- a/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.h +++ b/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.h @@ -42,11 +42,6 @@ enum dc_status dce112_validate_with_context( struct dc_state *context, struct dc_state *old_context); -enum dc_status dce112_validate_guaranteed( - struct dc *dc, - struct dc_stream_state *dc_stream, - struct dc_state *context); - bool dce112_validate_bandwidth( struct dc *dc, struct dc_state *context); diff --git a/drivers/gpu/drm/amd/display/dc/dce120/dce120_resource.c b/drivers/gpu/drm/amd/display/dc/dce120/dce120_resource.c index 4659a4bfabaa..2d58daccc005 100644 --- a/drivers/gpu/drm/amd/display/dc/dce120/dce120_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dce120/dce120_resource.c @@ -652,7 +652,7 @@ static struct mem_input *dce120_mem_input_create( return NULL; } - dce112_mem_input_construct(dce_mi, ctx, inst, &mi_regs[inst], &mi_shifts, &mi_masks); + dce120_mem_input_construct(dce_mi, ctx, inst, &mi_regs[inst], &mi_shifts, &mi_masks); return &dce_mi->base; } @@ -684,7 +684,6 @@ static void dce120_destroy_resource_pool(struct resource_pool **pool) static const struct resource_funcs dce120_res_pool_funcs = { .destroy = dce120_destroy_resource_pool, .link_enc_create = dce120_link_encoder_create, - .validate_guaranteed = dce112_validate_guaranteed, .validate_bandwidth = dce112_validate_bandwidth, .validate_plane = dce100_validate_plane, .add_stream_to_ctx = dce112_add_stream_to_ctx @@ -815,14 +814,25 @@ static void bw_calcs_data_update_from_pplib(struct dc *dc) dm_pp_notify_wm_clock_changes(dc->ctx, &clk_ranges); } +static uint32_t read_pipe_fuses(struct dc_context *ctx) +{ + uint32_t value = dm_read_reg_soc15(ctx, mmCC_DC_PIPE_DIS, 0); + /* VG20 support max 6 pipes */ + value = value & 0x3f; + return value; +} + static bool construct( uint8_t num_virtual_links, struct dc *dc, struct dce110_resource_pool *pool) { unsigned int i; + int j; struct dc_context *ctx = dc->ctx; struct irq_service_init_data irq_init_data; + bool harvest_enabled = ASICREV_IS_VEGA20_P(ctx->asic_id.hw_internal_rev); + uint32_t pipe_fuses; ctx->dc_bios->regs = &bios_regs; @@ -916,28 +926,41 @@ static bool construct( if (!pool->base.irqs) goto irqs_create_fail; + /* retrieve valid pipe fuses */ + if (harvest_enabled) + pipe_fuses = read_pipe_fuses(ctx); + + /* index to valid pipe resource */ + j = 0; for (i = 0; i < pool->base.pipe_count; i++) { - pool->base.timing_generators[i] = + if (harvest_enabled) { + if ((pipe_fuses & (1 << i)) != 0) { + dm_error("DC: skip invalid pipe %d!\n", i); + continue; + } + } + + pool->base.timing_generators[j] = dce120_timing_generator_create( ctx, i, &dce120_tg_offsets[i]); - if (pool->base.timing_generators[i] == NULL) { + if (pool->base.timing_generators[j] == NULL) { BREAK_TO_DEBUGGER(); dm_error("DC: failed to create tg!\n"); goto controller_create_fail; } - pool->base.mis[i] = dce120_mem_input_create(ctx, i); + pool->base.mis[j] = dce120_mem_input_create(ctx, i); - if (pool->base.mis[i] == NULL) { + if (pool->base.mis[j] == NULL) { BREAK_TO_DEBUGGER(); dm_error( "DC: failed to create memory input!\n"); goto controller_create_fail; } - pool->base.ipps[i] = dce120_ipp_create(ctx, i); + pool->base.ipps[j] = dce120_ipp_create(ctx, i); if (pool->base.ipps[i] == NULL) { BREAK_TO_DEBUGGER(); dm_error( @@ -945,7 +968,7 @@ static bool construct( goto controller_create_fail; } - pool->base.transforms[i] = dce120_transform_create(ctx, i); + pool->base.transforms[j] = dce120_transform_create(ctx, i); if (pool->base.transforms[i] == NULL) { BREAK_TO_DEBUGGER(); dm_error( @@ -953,16 +976,23 @@ static bool construct( goto res_create_fail; } - pool->base.opps[i] = dce120_opp_create( + pool->base.opps[j] = dce120_opp_create( ctx, i); - if (pool->base.opps[i] == NULL) { + if (pool->base.opps[j] == NULL) { BREAK_TO_DEBUGGER(); dm_error( "DC: failed to create output pixel processor!\n"); } + + /* check next valid pipe */ + j++; } + /* valid pipe num */ + pool->base.pipe_count = j; + pool->base.timing_generator_count = j; + if (!resource_construct(num_virtual_links, dc, &pool->base, &res_create_funcs)) goto res_create_fail; diff --git a/drivers/gpu/drm/amd/display/dc/dce120/dce120_timing_generator.c b/drivers/gpu/drm/amd/display/dc/dce120/dce120_timing_generator.c index 7bee78172d85..2ea490f8482e 100644 --- a/drivers/gpu/drm/amd/display/dc/dce120/dce120_timing_generator.c +++ b/drivers/gpu/drm/amd/display/dc/dce120/dce120_timing_generator.c @@ -570,18 +570,18 @@ void dce120_timing_generator_set_drr( 0x180); } else { - CRTC_REG_UPDATE( - CRTC0_CRTC_V_TOTAL_MIN, - CRTC_V_TOTAL_MIN, 0); - CRTC_REG_UPDATE( - CRTC0_CRTC_V_TOTAL_MAX, - CRTC_V_TOTAL_MAX, 0); CRTC_REG_SET_N(CRTC0_CRTC_V_TOTAL_CONTROL, 5, FD(CRTC0_CRTC_V_TOTAL_CONTROL__CRTC_V_TOTAL_MIN_SEL), 0, FD(CRTC0_CRTC_V_TOTAL_CONTROL__CRTC_V_TOTAL_MAX_SEL), 0, FD(CRTC0_CRTC_V_TOTAL_CONTROL__CRTC_FORCE_LOCK_ON_EVENT), 0, FD(CRTC0_CRTC_V_TOTAL_CONTROL__CRTC_FORCE_LOCK_TO_MASTER_VSYNC), 0, FD(CRTC0_CRTC_V_TOTAL_CONTROL__CRTC_SET_V_TOTAL_MIN_MASK), 0); + CRTC_REG_UPDATE( + CRTC0_CRTC_V_TOTAL_MIN, + CRTC_V_TOTAL_MIN, 0); + CRTC_REG_UPDATE( + CRTC0_CRTC_V_TOTAL_MAX, + CRTC_V_TOTAL_MAX, 0); CRTC_REG_UPDATE( CRTC0_CRTC_STATIC_SCREEN_CONTROL, CRTC_STATIC_SCREEN_EVENT_MASK, diff --git a/drivers/gpu/drm/amd/display/dc/dce80/dce80_resource.c b/drivers/gpu/drm/amd/display/dc/dce80/dce80_resource.c index 5d854a37a978..48a068964722 100644 --- a/drivers/gpu/drm/amd/display/dc/dce80/dce80_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dce80/dce80_resource.c @@ -691,23 +691,6 @@ static void destruct(struct dce110_resource_pool *pool) } } -static enum dc_status build_mapped_resource( - const struct dc *dc, - struct dc_state *context, - struct dc_stream_state *stream) -{ - struct pipe_ctx *pipe_ctx = resource_get_head_pipe_for_stream(&context->res_ctx, stream); - - if (!pipe_ctx) - return DC_ERROR_UNEXPECTED; - - dce110_resource_build_pipe_hw_param(pipe_ctx); - - resource_build_info_frame(pipe_ctx); - - return DC_OK; -} - bool dce80_validate_bandwidth( struct dc *dc, struct dc_state *context) @@ -749,37 +732,6 @@ enum dc_status dce80_validate_global( return DC_OK; } -enum dc_status dce80_validate_guaranteed( - struct dc *dc, - struct dc_stream_state *dc_stream, - struct dc_state *context) -{ - enum dc_status result = DC_ERROR_UNEXPECTED; - - context->streams[0] = dc_stream; - dc_stream_retain(context->streams[0]); - context->stream_count++; - - result = resource_map_pool_resources(dc, context, dc_stream); - - if (result == DC_OK) - result = resource_map_clock_resources(dc, context, dc_stream); - - if (result == DC_OK) - result = build_mapped_resource(dc, context, dc_stream); - - if (result == DC_OK) { - validate_guaranteed_copy_streams( - context, dc->caps.max_streams); - result = resource_build_scaling_params_for_context(dc, context); - } - - if (result == DC_OK) - result = dce80_validate_bandwidth(dc, context); - - return result; -} - static void dce80_destroy_resource_pool(struct resource_pool **pool) { struct dce110_resource_pool *dce110_pool = TO_DCE110_RES_POOL(*pool); @@ -792,7 +744,6 @@ static void dce80_destroy_resource_pool(struct resource_pool **pool) static const struct resource_funcs dce80_res_pool_funcs = { .destroy = dce80_destroy_resource_pool, .link_enc_create = dce80_link_encoder_create, - .validate_guaranteed = dce80_validate_guaranteed, .validate_bandwidth = dce80_validate_bandwidth, .validate_plane = dce100_validate_plane, .add_stream_to_ctx = dce100_add_stream_to_ctx, diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/Makefile b/drivers/gpu/drm/amd/display/dc/dcn10/Makefile index 5469bdfe19f3..84f52c63d95c 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/Makefile +++ b/drivers/gpu/drm/amd/display/dc/dcn10/Makefile @@ -26,7 +26,7 @@ DCN10 = dcn10_resource.o dcn10_ipp.o dcn10_hw_sequencer.o \ dcn10_dpp.o dcn10_opp.o dcn10_optc.o \ dcn10_hubp.o dcn10_mpc.o \ dcn10_dpp_dscl.o dcn10_dpp_cm.o dcn10_cm_common.o \ - dcn10_hubbub.o + dcn10_hubbub.o dcn10_stream_encoder.o dcn10_link_encoder.o AMD_DAL_DCN10 = $(addprefix $(AMDDALPATH)/dc/dcn10/,$(DCN10)) diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_cm_common.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_cm_common.c index 881a1bff94d2..5d95a997fd9f 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_cm_common.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_cm_common.c @@ -169,7 +169,7 @@ bool cm_helper_convert_to_custom_float( } if (fixpoint == true) - arr_points[1].custom_float_y = dal_fixed31_32_clamp_u0d14(arr_points[1].y); + arr_points[1].custom_float_y = dc_fixpt_clamp_u0d14(arr_points[1].y); else if (!convert_to_custom_float_format(arr_points[1].y, &fmt, &arr_points[1].custom_float_y)) { BREAK_TO_DEBUGGER(); @@ -327,19 +327,19 @@ bool cm_helper_translate_curve_to_hw_format( rgb_resulted[hw_points - 1].green = output_tf->tf_pts.green[start_index]; rgb_resulted[hw_points - 1].blue = output_tf->tf_pts.blue[start_index]; - arr_points[0].x = dal_fixed31_32_pow(dal_fixed31_32_from_int(2), - dal_fixed31_32_from_int(region_start)); - arr_points[1].x = dal_fixed31_32_pow(dal_fixed31_32_from_int(2), - dal_fixed31_32_from_int(region_end)); + arr_points[0].x = dc_fixpt_pow(dc_fixpt_from_int(2), + dc_fixpt_from_int(region_start)); + arr_points[1].x = dc_fixpt_pow(dc_fixpt_from_int(2), + dc_fixpt_from_int(region_end)); y_r = rgb_resulted[0].red; y_g = rgb_resulted[0].green; y_b = rgb_resulted[0].blue; - y1_min = dal_fixed31_32_min(y_r, dal_fixed31_32_min(y_g, y_b)); + y1_min = dc_fixpt_min(y_r, dc_fixpt_min(y_g, y_b)); arr_points[0].y = y1_min; - arr_points[0].slope = dal_fixed31_32_div(arr_points[0].y, arr_points[0].x); + arr_points[0].slope = dc_fixpt_div(arr_points[0].y, arr_points[0].x); y_r = rgb_resulted[hw_points - 1].red; y_g = rgb_resulted[hw_points - 1].green; y_b = rgb_resulted[hw_points - 1].blue; @@ -347,35 +347,35 @@ bool cm_helper_translate_curve_to_hw_format( /* see comment above, m_arrPoints[1].y should be the Y value for the * region end (m_numOfHwPoints), not last HW point(m_numOfHwPoints - 1) */ - y3_max = dal_fixed31_32_max(y_r, dal_fixed31_32_max(y_g, y_b)); + y3_max = dc_fixpt_max(y_r, dc_fixpt_max(y_g, y_b)); arr_points[1].y = y3_max; - arr_points[1].slope = dal_fixed31_32_zero; + arr_points[1].slope = dc_fixpt_zero; if (output_tf->tf == TRANSFER_FUNCTION_PQ) { /* for PQ, we want to have a straight line from last HW X point, * and the slope to be such that we hit 1.0 at 10000 nits. */ const struct fixed31_32 end_value = - dal_fixed31_32_from_int(125); + dc_fixpt_from_int(125); - arr_points[1].slope = dal_fixed31_32_div( - dal_fixed31_32_sub(dal_fixed31_32_one, arr_points[1].y), - dal_fixed31_32_sub(end_value, arr_points[1].x)); + arr_points[1].slope = dc_fixpt_div( + dc_fixpt_sub(dc_fixpt_one, arr_points[1].y), + dc_fixpt_sub(end_value, arr_points[1].x)); } lut_params->hw_points_num = hw_points; - i = 1; - for (k = 0; k < MAX_REGIONS_NUMBER && i < MAX_REGIONS_NUMBER; k++) { + k = 0; + for (i = 1; i < MAX_REGIONS_NUMBER; i++) { if (seg_distr[k] != -1) { lut_params->arr_curve_points[k].segments_num = seg_distr[k]; lut_params->arr_curve_points[i].offset = lut_params->arr_curve_points[k].offset + (1 << seg_distr[k]); } - i++; + k++; } if (seg_distr[k] != -1) @@ -386,24 +386,24 @@ bool cm_helper_translate_curve_to_hw_format( i = 1; while (i != hw_points + 1) { - if (dal_fixed31_32_lt(rgb_plus_1->red, rgb->red)) + if (dc_fixpt_lt(rgb_plus_1->red, rgb->red)) rgb_plus_1->red = rgb->red; - if (dal_fixed31_32_lt(rgb_plus_1->green, rgb->green)) + if (dc_fixpt_lt(rgb_plus_1->green, rgb->green)) rgb_plus_1->green = rgb->green; - if (dal_fixed31_32_lt(rgb_plus_1->blue, rgb->blue)) + if (dc_fixpt_lt(rgb_plus_1->blue, rgb->blue)) rgb_plus_1->blue = rgb->blue; - rgb->delta_red = dal_fixed31_32_sub(rgb_plus_1->red, rgb->red); - rgb->delta_green = dal_fixed31_32_sub(rgb_plus_1->green, rgb->green); - rgb->delta_blue = dal_fixed31_32_sub(rgb_plus_1->blue, rgb->blue); + rgb->delta_red = dc_fixpt_sub(rgb_plus_1->red, rgb->red); + rgb->delta_green = dc_fixpt_sub(rgb_plus_1->green, rgb->green); + rgb->delta_blue = dc_fixpt_sub(rgb_plus_1->blue, rgb->blue); if (fixpoint == true) { - rgb->delta_red_reg = dal_fixed31_32_clamp_u0d10(rgb->delta_red); - rgb->delta_green_reg = dal_fixed31_32_clamp_u0d10(rgb->delta_green); - rgb->delta_blue_reg = dal_fixed31_32_clamp_u0d10(rgb->delta_blue); - rgb->red_reg = dal_fixed31_32_clamp_u0d14(rgb->red); - rgb->green_reg = dal_fixed31_32_clamp_u0d14(rgb->green); - rgb->blue_reg = dal_fixed31_32_clamp_u0d14(rgb->blue); + rgb->delta_red_reg = dc_fixpt_clamp_u0d10(rgb->delta_red); + rgb->delta_green_reg = dc_fixpt_clamp_u0d10(rgb->delta_green); + rgb->delta_blue_reg = dc_fixpt_clamp_u0d10(rgb->delta_blue); + rgb->red_reg = dc_fixpt_clamp_u0d14(rgb->red); + rgb->green_reg = dc_fixpt_clamp_u0d14(rgb->green); + rgb->blue_reg = dc_fixpt_clamp_u0d14(rgb->blue); } ++rgb_plus_1; @@ -489,19 +489,19 @@ bool cm_helper_translate_curve_to_degamma_hw_format( rgb_resulted[hw_points - 1].green = output_tf->tf_pts.green[start_index]; rgb_resulted[hw_points - 1].blue = output_tf->tf_pts.blue[start_index]; - arr_points[0].x = dal_fixed31_32_pow(dal_fixed31_32_from_int(2), - dal_fixed31_32_from_int(region_start)); - arr_points[1].x = dal_fixed31_32_pow(dal_fixed31_32_from_int(2), - dal_fixed31_32_from_int(region_end)); + arr_points[0].x = dc_fixpt_pow(dc_fixpt_from_int(2), + dc_fixpt_from_int(region_start)); + arr_points[1].x = dc_fixpt_pow(dc_fixpt_from_int(2), + dc_fixpt_from_int(region_end)); y_r = rgb_resulted[0].red; y_g = rgb_resulted[0].green; y_b = rgb_resulted[0].blue; - y1_min = dal_fixed31_32_min(y_r, dal_fixed31_32_min(y_g, y_b)); + y1_min = dc_fixpt_min(y_r, dc_fixpt_min(y_g, y_b)); arr_points[0].y = y1_min; - arr_points[0].slope = dal_fixed31_32_div(arr_points[0].y, arr_points[0].x); + arr_points[0].slope = dc_fixpt_div(arr_points[0].y, arr_points[0].x); y_r = rgb_resulted[hw_points - 1].red; y_g = rgb_resulted[hw_points - 1].green; y_b = rgb_resulted[hw_points - 1].blue; @@ -509,35 +509,35 @@ bool cm_helper_translate_curve_to_degamma_hw_format( /* see comment above, m_arrPoints[1].y should be the Y value for the * region end (m_numOfHwPoints), not last HW point(m_numOfHwPoints - 1) */ - y3_max = dal_fixed31_32_max(y_r, dal_fixed31_32_max(y_g, y_b)); + y3_max = dc_fixpt_max(y_r, dc_fixpt_max(y_g, y_b)); arr_points[1].y = y3_max; - arr_points[1].slope = dal_fixed31_32_zero; + arr_points[1].slope = dc_fixpt_zero; if (output_tf->tf == TRANSFER_FUNCTION_PQ) { /* for PQ, we want to have a straight line from last HW X point, * and the slope to be such that we hit 1.0 at 10000 nits. */ const struct fixed31_32 end_value = - dal_fixed31_32_from_int(125); + dc_fixpt_from_int(125); - arr_points[1].slope = dal_fixed31_32_div( - dal_fixed31_32_sub(dal_fixed31_32_one, arr_points[1].y), - dal_fixed31_32_sub(end_value, arr_points[1].x)); + arr_points[1].slope = dc_fixpt_div( + dc_fixpt_sub(dc_fixpt_one, arr_points[1].y), + dc_fixpt_sub(end_value, arr_points[1].x)); } lut_params->hw_points_num = hw_points; - i = 1; - for (k = 0; k < MAX_REGIONS_NUMBER && i < MAX_REGIONS_NUMBER; k++) { + k = 0; + for (i = 1; i < MAX_REGIONS_NUMBER; i++) { if (seg_distr[k] != -1) { lut_params->arr_curve_points[k].segments_num = seg_distr[k]; lut_params->arr_curve_points[i].offset = lut_params->arr_curve_points[k].offset + (1 << seg_distr[k]); } - i++; + k++; } if (seg_distr[k] != -1) @@ -548,16 +548,16 @@ bool cm_helper_translate_curve_to_degamma_hw_format( i = 1; while (i != hw_points + 1) { - if (dal_fixed31_32_lt(rgb_plus_1->red, rgb->red)) + if (dc_fixpt_lt(rgb_plus_1->red, rgb->red)) rgb_plus_1->red = rgb->red; - if (dal_fixed31_32_lt(rgb_plus_1->green, rgb->green)) + if (dc_fixpt_lt(rgb_plus_1->green, rgb->green)) rgb_plus_1->green = rgb->green; - if (dal_fixed31_32_lt(rgb_plus_1->blue, rgb->blue)) + if (dc_fixpt_lt(rgb_plus_1->blue, rgb->blue)) rgb_plus_1->blue = rgb->blue; - rgb->delta_red = dal_fixed31_32_sub(rgb_plus_1->red, rgb->red); - rgb->delta_green = dal_fixed31_32_sub(rgb_plus_1->green, rgb->green); - rgb->delta_blue = dal_fixed31_32_sub(rgb_plus_1->blue, rgb->blue); + rgb->delta_red = dc_fixpt_sub(rgb_plus_1->red, rgb->red); + rgb->delta_green = dc_fixpt_sub(rgb_plus_1->green, rgb->green); + rgb->delta_blue = dc_fixpt_sub(rgb_plus_1->blue, rgb->blue); ++rgb_plus_1; ++rgb; diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp.c index e305c28c98de..c69fa4bfab0a 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp.c @@ -98,6 +98,30 @@ enum gamut_remap_select { GAMUT_REMAP_COMB_COEFF }; +void dpp_read_state(struct dpp *dpp_base, + struct dcn_dpp_state *s) +{ + struct dcn10_dpp *dpp = TO_DCN10_DPP(dpp_base); + + REG_GET(CM_IGAM_CONTROL, + CM_IGAM_LUT_MODE, &s->igam_lut_mode); + REG_GET(CM_IGAM_CONTROL, + CM_IGAM_INPUT_FORMAT, &s->igam_input_format); + REG_GET(CM_DGAM_CONTROL, + CM_DGAM_LUT_MODE, &s->dgam_lut_mode); + REG_GET(CM_RGAM_CONTROL, + CM_RGAM_LUT_MODE, &s->rgam_lut_mode); + REG_GET(CM_GAMUT_REMAP_CONTROL, + CM_GAMUT_REMAP_MODE, &s->gamut_remap_mode); + + s->gamut_remap_c11_c12 = REG_READ(CM_GAMUT_REMAP_C11_C12); + s->gamut_remap_c13_c14 = REG_READ(CM_GAMUT_REMAP_C13_C14); + s->gamut_remap_c21_c22 = REG_READ(CM_GAMUT_REMAP_C21_C22); + s->gamut_remap_c23_c24 = REG_READ(CM_GAMUT_REMAP_C23_C24); + s->gamut_remap_c31_c32 = REG_READ(CM_GAMUT_REMAP_C31_C32); + s->gamut_remap_c33_c34 = REG_READ(CM_GAMUT_REMAP_C33_C34); +} + /* Program gamut remap in bypass mode */ void dpp_set_gamut_remap_bypass(struct dcn10_dpp *dpp) { @@ -106,10 +130,9 @@ void dpp_set_gamut_remap_bypass(struct dcn10_dpp *dpp) /* Gamut remap in bypass */ } -#define IDENTITY_RATIO(ratio) (dal_fixed31_32_u2d19(ratio) == (1 << 19)) - +#define IDENTITY_RATIO(ratio) (dc_fixpt_u2d19(ratio) == (1 << 19)) -bool dpp_get_optimal_number_of_taps( +static bool dpp_get_optimal_number_of_taps( struct dpp *dpp, struct scaler_data *scl_data, const struct scaling_taps *in_taps) @@ -121,6 +144,18 @@ bool dpp_get_optimal_number_of_taps( else pixel_width = scl_data->viewport.width; + /* Some ASICs does not support FP16 scaling, so we reject modes require this*/ + if (scl_data->viewport.width != scl_data->h_active && + scl_data->viewport.height != scl_data->v_active && + dpp->caps->dscl_data_proc_format == DSCL_DATA_PRCESSING_FIXED_FORMAT && + scl_data->format == PIXEL_FORMAT_FP16) + return false; + + if (scl_data->viewport.width > scl_data->h_active && + dpp->ctx->dc->debug.max_downscale_src_width != 0 && + scl_data->viewport.width > dpp->ctx->dc->debug.max_downscale_src_width) + return false; + /* TODO: add lb check */ /* No support for programming ratio of 4, drop to 3.99999.. */ @@ -257,7 +292,7 @@ void dpp1_cnv_setup ( struct dpp *dpp_base, enum surface_pixel_format format, enum expansion_mode mode, - struct csc_transform input_csc_color_matrix, + struct dc_csc_transform input_csc_color_matrix, enum dc_color_space input_color_space) { uint32_t pixel_format; @@ -416,7 +451,7 @@ void dpp1_set_cursor_position( if (src_x_offset >= (int)param->viewport_width) cur_en = 0; /* not visible beyond right edge*/ - if (src_x_offset + (int)width < 0) + if (src_x_offset + (int)width <= 0) cur_en = 0; /* not visible beyond left edge*/ REG_UPDATE(CURSOR0_CONTROL, @@ -443,6 +478,7 @@ void dpp1_dppclk_control( } static const struct dpp_funcs dcn10_dpp_funcs = { + .dpp_read_state = dpp_read_state, .dpp_reset = dpp_reset, .dpp_set_scaler = dpp1_dscl_set_scaler_manual_scale, .dpp_get_optimal_number_of_taps = dpp_get_optimal_number_of_taps, diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp.h b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp.h index 17b062a8f88a..e862cafa6501 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp.h +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp.h @@ -44,6 +44,10 @@ #define TF_REG_LIST_DCN(id) \ SRI(CM_GAMUT_REMAP_CONTROL, CM, id),\ SRI(CM_GAMUT_REMAP_C11_C12, CM, id),\ + SRI(CM_GAMUT_REMAP_C13_C14, CM, id),\ + SRI(CM_GAMUT_REMAP_C21_C22, CM, id),\ + SRI(CM_GAMUT_REMAP_C23_C24, CM, id),\ + SRI(CM_GAMUT_REMAP_C31_C32, CM, id),\ SRI(CM_GAMUT_REMAP_C33_C34, CM, id),\ SRI(DSCL_EXT_OVERSCAN_LEFT_RIGHT, DSCL, id), \ SRI(DSCL_EXT_OVERSCAN_TOP_BOTTOM, DSCL, id), \ @@ -108,6 +112,8 @@ SRI(CM_DGAM_LUT_DATA, CM, id), \ SRI(CM_CONTROL, CM, id), \ SRI(CM_DGAM_CONTROL, CM, id), \ + SRI(CM_TEST_DEBUG_INDEX, CM, id), \ + SRI(CM_TEST_DEBUG_DATA, CM, id), \ SRI(FORMAT_CONTROL, CNVC_CFG, id), \ SRI(CNVC_SURFACE_PIXEL_FORMAT, CNVC_CFG, id), \ SRI(CURSOR0_CONTROL, CNVC_CUR, id), \ @@ -175,6 +181,14 @@ TF_SF(CM0_CM_GAMUT_REMAP_CONTROL, CM_GAMUT_REMAP_MODE, mask_sh),\ TF_SF(CM0_CM_GAMUT_REMAP_C11_C12, CM_GAMUT_REMAP_C11, mask_sh),\ TF_SF(CM0_CM_GAMUT_REMAP_C11_C12, CM_GAMUT_REMAP_C12, mask_sh),\ + TF_SF(CM0_CM_GAMUT_REMAP_C13_C14, CM_GAMUT_REMAP_C13, mask_sh),\ + TF_SF(CM0_CM_GAMUT_REMAP_C13_C14, CM_GAMUT_REMAP_C14, mask_sh),\ + TF_SF(CM0_CM_GAMUT_REMAP_C21_C22, CM_GAMUT_REMAP_C21, mask_sh),\ + TF_SF(CM0_CM_GAMUT_REMAP_C21_C22, CM_GAMUT_REMAP_C22, mask_sh),\ + TF_SF(CM0_CM_GAMUT_REMAP_C23_C24, CM_GAMUT_REMAP_C23, mask_sh),\ + TF_SF(CM0_CM_GAMUT_REMAP_C23_C24, CM_GAMUT_REMAP_C24, mask_sh),\ + TF_SF(CM0_CM_GAMUT_REMAP_C31_C32, CM_GAMUT_REMAP_C31, mask_sh),\ + TF_SF(CM0_CM_GAMUT_REMAP_C31_C32, CM_GAMUT_REMAP_C32, mask_sh),\ TF_SF(CM0_CM_GAMUT_REMAP_C33_C34, CM_GAMUT_REMAP_C33, mask_sh),\ TF_SF(CM0_CM_GAMUT_REMAP_C33_C34, CM_GAMUT_REMAP_C34, mask_sh),\ TF_SF(DSCL0_DSCL_EXT_OVERSCAN_LEFT_RIGHT, EXT_OVERSCAN_LEFT, mask_sh),\ @@ -300,6 +314,7 @@ TF_SF(CM0_CM_DGAM_LUT_INDEX, CM_DGAM_LUT_INDEX, mask_sh), \ TF_SF(CM0_CM_DGAM_LUT_DATA, CM_DGAM_LUT_DATA, mask_sh), \ TF_SF(CM0_CM_DGAM_CONTROL, CM_DGAM_LUT_MODE, mask_sh), \ + TF_SF(CM0_CM_TEST_DEBUG_INDEX, CM_TEST_DEBUG_INDEX, mask_sh), \ TF_SF(CNVC_CFG0_FORMAT_CONTROL, CNVC_BYPASS, mask_sh), \ TF2_SF(CNVC_CFG0, FORMAT_CONTROL__ALPHA_EN, mask_sh), \ TF_SF(CNVC_CFG0_FORMAT_CONTROL, FORMAT_EXPANSION_MODE, mask_sh), \ @@ -417,6 +432,41 @@ TF_SF(CURSOR0_CURSOR_CONTROL, CURSOR_ENABLE, mask_sh), \ TF_SF(DPP_TOP0_DPP_CONTROL, DPPCLK_RATE_CONTROL, mask_sh) +/* + * + DCN1 CM debug status register definition + + register :ID9_CM_STATUS do + implement_ref :cm + map to: :cmdebugind, at: j + width 32 + disclosure NEVER + + field :ID9_VUPDATE_CFG, [0], R + field :ID9_IGAM_LUT_MODE, [2..1], R + field :ID9_BNS_BYPASS, [3], R + field :ID9_ICSC_MODE, [5..4], R + field :ID9_DGAM_LUT_MODE, [8..6], R + field :ID9_HDR_BYPASS, [9], R + field :ID9_GAMUT_REMAP_MODE, [11..10], R + field :ID9_RGAM_LUT_MODE, [14..12], R + #1 free bit + field :ID9_OCSC_MODE, [18..16], R + field :ID9_DENORM_MODE, [21..19], R + field :ID9_ROUND_TRUNC_MODE, [25..22], R + field :ID9_DITHER_EN, [26], R + field :ID9_DITHER_MODE, [28..27], R + end +*/ + +#define TF_DEBUG_REG_LIST_SH_DCN10 \ + .CM_TEST_DEBUG_DATA_ID9_ICSC_MODE = 4, \ + .CM_TEST_DEBUG_DATA_ID9_OCSC_MODE = 16 + +#define TF_DEBUG_REG_LIST_MASK_DCN10 \ + .CM_TEST_DEBUG_DATA_ID9_ICSC_MODE = 0x30, \ + .CM_TEST_DEBUG_DATA_ID9_OCSC_MODE = 0x70000 + #define TF_REG_FIELD_LIST(type) \ type EXT_OVERSCAN_LEFT; \ type EXT_OVERSCAN_RIGHT; \ @@ -486,6 +536,14 @@ type CM_GAMUT_REMAP_MODE; \ type CM_GAMUT_REMAP_C11; \ type CM_GAMUT_REMAP_C12; \ + type CM_GAMUT_REMAP_C13; \ + type CM_GAMUT_REMAP_C14; \ + type CM_GAMUT_REMAP_C21; \ + type CM_GAMUT_REMAP_C22; \ + type CM_GAMUT_REMAP_C23; \ + type CM_GAMUT_REMAP_C24; \ + type CM_GAMUT_REMAP_C31; \ + type CM_GAMUT_REMAP_C32; \ type CM_GAMUT_REMAP_C33; \ type CM_GAMUT_REMAP_C34; \ type CM_COMA_C11; \ @@ -1010,6 +1068,9 @@ type CUR0_EXPANSION_MODE; \ type CUR0_ENABLE; \ type CM_BYPASS; \ + type CM_TEST_DEBUG_INDEX; \ + type CM_TEST_DEBUG_DATA_ID9_ICSC_MODE; \ + type CM_TEST_DEBUG_DATA_ID9_OCSC_MODE;\ type FORMAT_CONTROL__ALPHA_EN; \ type CUR0_COLOR0; \ type CUR0_COLOR1; \ @@ -1054,6 +1115,10 @@ struct dcn_dpp_mask { uint32_t RECOUT_SIZE; \ uint32_t CM_GAMUT_REMAP_CONTROL; \ uint32_t CM_GAMUT_REMAP_C11_C12; \ + uint32_t CM_GAMUT_REMAP_C13_C14; \ + uint32_t CM_GAMUT_REMAP_C21_C22; \ + uint32_t CM_GAMUT_REMAP_C23_C24; \ + uint32_t CM_GAMUT_REMAP_C31_C32; \ uint32_t CM_GAMUT_REMAP_C33_C34; \ uint32_t CM_COMA_C11_C12; \ uint32_t CM_COMA_C33_C34; \ @@ -1255,6 +1320,8 @@ struct dcn_dpp_mask { uint32_t CM_IGAM_LUT_RW_CONTROL; \ uint32_t CM_IGAM_LUT_RW_INDEX; \ uint32_t CM_IGAM_LUT_SEQ_COLOR; \ + uint32_t CM_TEST_DEBUG_INDEX; \ + uint32_t CM_TEST_DEBUG_DATA; \ uint32_t FORMAT_CONTROL; \ uint32_t CNVC_SURFACE_PIXEL_FORMAT; \ uint32_t CURSOR_CONTROL; \ @@ -1289,8 +1356,8 @@ struct dcn10_dpp { enum dcn10_input_csc_select { INPUT_CSC_SELECT_BYPASS = 0, - INPUT_CSC_SELECT_ICSC, - INPUT_CSC_SELECT_COMA + INPUT_CSC_SELECT_ICSC = 1, + INPUT_CSC_SELECT_COMA = 2 }; void dpp1_set_cursor_attributes( @@ -1357,12 +1424,11 @@ void dpp1_set_degamma( enum ipp_degamma_mode mode); void dpp1_set_degamma_pwl(struct dpp *dpp_base, - const struct pwl_params *params); + const struct pwl_params *params); + -bool dpp_get_optimal_number_of_taps( - struct dpp *dpp, - struct scaler_data *scl_data, - const struct scaling_taps *in_taps); +void dpp_read_state(struct dpp *dpp_base, + struct dcn_dpp_state *s); void dpp_reset(struct dpp *dpp_base); @@ -1408,7 +1474,7 @@ void dpp1_cnv_setup ( struct dpp *dpp_base, enum surface_pixel_format format, enum expansion_mode mode, - struct csc_transform input_csc_color_matrix, + struct dc_csc_transform input_csc_color_matrix, enum dc_color_space input_color_space); void dpp1_full_bypass(struct dpp *dpp_base); diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp_cm.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp_cm.c index fb32975e4b67..116977eb24e2 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp_cm.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp_cm.c @@ -216,41 +216,55 @@ static void dpp1_cm_program_color_matrix( struct dcn10_dpp *dpp, const uint16_t *regval) { - uint32_t mode; + uint32_t ocsc_mode; + uint32_t cur_mode; struct color_matrices_reg gam_regs; - REG_GET(CM_OCSC_CONTROL, CM_OCSC_MODE, &mode); - if (regval == NULL) { BREAK_TO_DEBUGGER(); return; } - mode = 4; + + /* determine which CSC matrix (ocsc or comb) we are using + * currently. select the alternate set to double buffer + * the CSC update so CSC is updated on frame boundary + */ + REG_SET(CM_TEST_DEBUG_INDEX, 0, + CM_TEST_DEBUG_INDEX, 9); + + REG_GET(CM_TEST_DEBUG_DATA, + CM_TEST_DEBUG_DATA_ID9_OCSC_MODE, &cur_mode); + + if (cur_mode != 4) + ocsc_mode = 4; + else + ocsc_mode = 5; + + gam_regs.shifts.csc_c11 = dpp->tf_shift->CM_OCSC_C11; gam_regs.masks.csc_c11 = dpp->tf_mask->CM_OCSC_C11; gam_regs.shifts.csc_c12 = dpp->tf_shift->CM_OCSC_C12; gam_regs.masks.csc_c12 = dpp->tf_mask->CM_OCSC_C12; - if (mode == 4) { + if (ocsc_mode == 4) { gam_regs.csc_c11_c12 = REG(CM_OCSC_C11_C12); gam_regs.csc_c33_c34 = REG(CM_OCSC_C33_C34); - cm_helper_program_color_matrices( - dpp->base.ctx, - regval, - &gam_regs); - } else { gam_regs.csc_c11_c12 = REG(CM_COMB_C11_C12); gam_regs.csc_c33_c34 = REG(CM_COMB_C33_C34); - cm_helper_program_color_matrices( - dpp->base.ctx, - regval, - &gam_regs); } + + cm_helper_program_color_matrices( + dpp->base.ctx, + regval, + &gam_regs); + + REG_SET(CM_OCSC_CONTROL, 0, CM_OCSC_MODE, ocsc_mode); + } void dpp1_cm_set_output_csc_default( @@ -260,15 +274,14 @@ void dpp1_cm_set_output_csc_default( struct dcn10_dpp *dpp = TO_DCN10_DPP(dpp_base); const uint16_t *regval = NULL; int arr_size; - uint32_t ocsc_mode = 4; regval = find_color_matrix(colorspace, &arr_size); if (regval == NULL) { BREAK_TO_DEBUGGER(); return; } + dpp1_cm_program_color_matrix(dpp, regval); - REG_SET(CM_OCSC_CONTROL, 0, CM_OCSC_MODE, ocsc_mode); } static void dpp1_cm_get_reg_field( @@ -329,9 +342,8 @@ void dpp1_cm_set_output_csc_adjustment( const uint16_t *regval) { struct dcn10_dpp *dpp = TO_DCN10_DPP(dpp_base); - uint32_t ocsc_mode = 4; + dpp1_cm_program_color_matrix(dpp, regval); - REG_SET(CM_OCSC_CONTROL, 0, CM_OCSC_MODE, ocsc_mode); } void dpp1_cm_power_on_regamma_lut(struct dpp *dpp_base, @@ -437,17 +449,18 @@ void dpp1_cm_program_regamma_lutb_settings( void dpp1_program_input_csc( struct dpp *dpp_base, enum dc_color_space color_space, - enum dcn10_input_csc_select select, + enum dcn10_input_csc_select input_select, const struct out_csc_color_matrix *tbl_entry) { struct dcn10_dpp *dpp = TO_DCN10_DPP(dpp_base); int i; int arr_size = sizeof(dcn10_input_csc_matrix)/sizeof(struct dcn10_input_csc_matrix); const uint16_t *regval = NULL; - uint32_t selection = 1; + uint32_t cur_select = 0; + enum dcn10_input_csc_select select; struct color_matrices_reg gam_regs; - if (select == INPUT_CSC_SELECT_BYPASS) { + if (input_select == INPUT_CSC_SELECT_BYPASS) { REG_SET(CM_ICSC_CONTROL, 0, CM_ICSC_MODE, 0); return; } @@ -467,36 +480,45 @@ void dpp1_program_input_csc( regval = tbl_entry->regval; } - if (select == INPUT_CSC_SELECT_COMA) - selection = 2; - REG_SET(CM_ICSC_CONTROL, 0, - CM_ICSC_MODE, selection); + /* determine which CSC matrix (icsc or coma) we are using + * currently. select the alternate set to double buffer + * the CSC update so CSC is updated on frame boundary + */ + REG_SET(CM_TEST_DEBUG_INDEX, 0, + CM_TEST_DEBUG_INDEX, 9); + + REG_GET(CM_TEST_DEBUG_DATA, + CM_TEST_DEBUG_DATA_ID9_ICSC_MODE, &cur_select); + + if (cur_select != INPUT_CSC_SELECT_ICSC) + select = INPUT_CSC_SELECT_ICSC; + else + select = INPUT_CSC_SELECT_COMA; gam_regs.shifts.csc_c11 = dpp->tf_shift->CM_ICSC_C11; gam_regs.masks.csc_c11 = dpp->tf_mask->CM_ICSC_C11; gam_regs.shifts.csc_c12 = dpp->tf_shift->CM_ICSC_C12; gam_regs.masks.csc_c12 = dpp->tf_mask->CM_ICSC_C12; - if (select == INPUT_CSC_SELECT_ICSC) { gam_regs.csc_c11_c12 = REG(CM_ICSC_C11_C12); gam_regs.csc_c33_c34 = REG(CM_ICSC_C33_C34); - cm_helper_program_color_matrices( - dpp->base.ctx, - regval, - &gam_regs); } else { gam_regs.csc_c11_c12 = REG(CM_COMA_C11_C12); gam_regs.csc_c33_c34 = REG(CM_COMA_C33_C34); - cm_helper_program_color_matrices( - dpp->base.ctx, - regval, - &gam_regs); } + + cm_helper_program_color_matrices( + dpp->base.ctx, + regval, + &gam_regs); + + REG_SET(CM_ICSC_CONTROL, 0, + CM_ICSC_MODE, select); } //keep here for now, decide multi dce support later @@ -789,13 +811,13 @@ void dpp1_program_input_lut( REG_UPDATE(CM_IGAM_LUT_RW_INDEX, CM_IGAM_LUT_RW_INDEX, 0); for (i = 0; i < gamma->num_entries; i++) { REG_SET(CM_IGAM_LUT_SEQ_COLOR, 0, CM_IGAM_LUT_SEQ_COLOR, - dal_fixed31_32_round( + dc_fixpt_round( gamma->entries.red[i])); REG_SET(CM_IGAM_LUT_SEQ_COLOR, 0, CM_IGAM_LUT_SEQ_COLOR, - dal_fixed31_32_round( + dc_fixpt_round( gamma->entries.green[i])); REG_SET(CM_IGAM_LUT_SEQ_COLOR, 0, CM_IGAM_LUT_SEQ_COLOR, - dal_fixed31_32_round( + dc_fixpt_round( gamma->entries.blue[i])); } // Power off LUT memory diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp_dscl.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp_dscl.c index 3eb824debf43..f862fd148cca 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp_dscl.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_dpp_dscl.c @@ -169,7 +169,7 @@ static enum dscl_mode_sel dpp1_dscl_get_dscl_mode( const struct scaler_data *data, bool dbg_always_scale) { - const long long one = dal_fixed31_32_one.value; + const long long one = dc_fixpt_one.value; if (dpp_base->caps->dscl_data_proc_format == DSCL_DATA_PRCESSING_FIXED_FORMAT) { /* DSCL is processing data in fixed format */ @@ -464,8 +464,8 @@ static enum lb_memory_config dpp1_dscl_find_lb_memory_config(struct dcn10_dpp *d int num_part_y, num_part_c; int vtaps = scl_data->taps.v_taps; int vtaps_c = scl_data->taps.v_taps_c; - int ceil_vratio = dal_fixed31_32_ceil(scl_data->ratios.vert); - int ceil_vratio_c = dal_fixed31_32_ceil(scl_data->ratios.vert_c); + int ceil_vratio = dc_fixpt_ceil(scl_data->ratios.vert); + int ceil_vratio_c = dc_fixpt_ceil(scl_data->ratios.vert_c); enum lb_memory_config mem_cfg = LB_MEMORY_CONFIG_0; if (dpp->base.ctx->dc->debug.use_max_lb) @@ -565,52 +565,52 @@ static void dpp1_dscl_set_manual_ratio_init( uint32_t init_int = 0; REG_SET(SCL_HORZ_FILTER_SCALE_RATIO, 0, - SCL_H_SCALE_RATIO, dal_fixed31_32_u2d19(data->ratios.horz) << 5); + SCL_H_SCALE_RATIO, dc_fixpt_u3d19(data->ratios.horz) << 5); REG_SET(SCL_VERT_FILTER_SCALE_RATIO, 0, - SCL_V_SCALE_RATIO, dal_fixed31_32_u2d19(data->ratios.vert) << 5); + SCL_V_SCALE_RATIO, dc_fixpt_u3d19(data->ratios.vert) << 5); REG_SET(SCL_HORZ_FILTER_SCALE_RATIO_C, 0, - SCL_H_SCALE_RATIO_C, dal_fixed31_32_u2d19(data->ratios.horz_c) << 5); + SCL_H_SCALE_RATIO_C, dc_fixpt_u3d19(data->ratios.horz_c) << 5); REG_SET(SCL_VERT_FILTER_SCALE_RATIO_C, 0, - SCL_V_SCALE_RATIO_C, dal_fixed31_32_u2d19(data->ratios.vert_c) << 5); + SCL_V_SCALE_RATIO_C, dc_fixpt_u3d19(data->ratios.vert_c) << 5); /* * 0.24 format for fraction, first five bits zeroed */ - init_frac = dal_fixed31_32_u0d19(data->inits.h) << 5; - init_int = dal_fixed31_32_floor(data->inits.h); + init_frac = dc_fixpt_u0d19(data->inits.h) << 5; + init_int = dc_fixpt_floor(data->inits.h); REG_SET_2(SCL_HORZ_FILTER_INIT, 0, SCL_H_INIT_FRAC, init_frac, SCL_H_INIT_INT, init_int); - init_frac = dal_fixed31_32_u0d19(data->inits.h_c) << 5; - init_int = dal_fixed31_32_floor(data->inits.h_c); + init_frac = dc_fixpt_u0d19(data->inits.h_c) << 5; + init_int = dc_fixpt_floor(data->inits.h_c); REG_SET_2(SCL_HORZ_FILTER_INIT_C, 0, SCL_H_INIT_FRAC_C, init_frac, SCL_H_INIT_INT_C, init_int); - init_frac = dal_fixed31_32_u0d19(data->inits.v) << 5; - init_int = dal_fixed31_32_floor(data->inits.v); + init_frac = dc_fixpt_u0d19(data->inits.v) << 5; + init_int = dc_fixpt_floor(data->inits.v); REG_SET_2(SCL_VERT_FILTER_INIT, 0, SCL_V_INIT_FRAC, init_frac, SCL_V_INIT_INT, init_int); - init_frac = dal_fixed31_32_u0d19(data->inits.v_bot) << 5; - init_int = dal_fixed31_32_floor(data->inits.v_bot); + init_frac = dc_fixpt_u0d19(data->inits.v_bot) << 5; + init_int = dc_fixpt_floor(data->inits.v_bot); REG_SET_2(SCL_VERT_FILTER_INIT_BOT, 0, SCL_V_INIT_FRAC_BOT, init_frac, SCL_V_INIT_INT_BOT, init_int); - init_frac = dal_fixed31_32_u0d19(data->inits.v_c) << 5; - init_int = dal_fixed31_32_floor(data->inits.v_c); + init_frac = dc_fixpt_u0d19(data->inits.v_c) << 5; + init_int = dc_fixpt_floor(data->inits.v_c); REG_SET_2(SCL_VERT_FILTER_INIT_C, 0, SCL_V_INIT_FRAC_C, init_frac, SCL_V_INIT_INT_C, init_int); - init_frac = dal_fixed31_32_u0d19(data->inits.v_c_bot) << 5; - init_int = dal_fixed31_32_floor(data->inits.v_c_bot); + init_frac = dc_fixpt_u0d19(data->inits.v_c_bot) << 5; + init_int = dc_fixpt_floor(data->inits.v_c_bot); REG_SET_2(SCL_VERT_FILTER_INIT_BOT_C, 0, SCL_V_INIT_FRAC_BOT_C, init_frac, SCL_V_INIT_INT_BOT_C, init_int); diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubbub.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubbub.c index 738f67ffd1b4..943143efbb82 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubbub.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubbub.c @@ -476,8 +476,235 @@ void hubbub1_toggle_watermark_change_req(struct hubbub *hubbub) DCHUBBUB_ARB_WATERMARK_CHANGE_REQUEST, watermark_change_req); } +void hubbub1_soft_reset(struct hubbub *hubbub, bool reset) +{ + uint32_t reset_en = reset ? 1 : 0; + + REG_UPDATE(DCHUBBUB_SOFT_RESET, + DCHUBBUB_GLOBAL_SOFT_RESET, reset_en); +} + +static bool hubbub1_dcc_support_swizzle( + enum swizzle_mode_values swizzle, + unsigned int bytes_per_element, + enum segment_order *segment_order_horz, + enum segment_order *segment_order_vert) +{ + bool standard_swizzle = false; + bool display_swizzle = false; + + switch (swizzle) { + case DC_SW_4KB_S: + case DC_SW_64KB_S: + case DC_SW_VAR_S: + case DC_SW_4KB_S_X: + case DC_SW_64KB_S_X: + case DC_SW_VAR_S_X: + standard_swizzle = true; + break; + case DC_SW_4KB_D: + case DC_SW_64KB_D: + case DC_SW_VAR_D: + case DC_SW_4KB_D_X: + case DC_SW_64KB_D_X: + case DC_SW_VAR_D_X: + display_swizzle = true; + break; + default: + break; + } + + if (bytes_per_element == 1 && standard_swizzle) { + *segment_order_horz = segment_order__contiguous; + *segment_order_vert = segment_order__na; + return true; + } + if (bytes_per_element == 2 && standard_swizzle) { + *segment_order_horz = segment_order__non_contiguous; + *segment_order_vert = segment_order__contiguous; + return true; + } + if (bytes_per_element == 4 && standard_swizzle) { + *segment_order_horz = segment_order__non_contiguous; + *segment_order_vert = segment_order__contiguous; + return true; + } + if (bytes_per_element == 8 && standard_swizzle) { + *segment_order_horz = segment_order__na; + *segment_order_vert = segment_order__contiguous; + return true; + } + if (bytes_per_element == 8 && display_swizzle) { + *segment_order_horz = segment_order__contiguous; + *segment_order_vert = segment_order__non_contiguous; + return true; + } + + return false; +} + +static bool hubbub1_dcc_support_pixel_format( + enum surface_pixel_format format, + unsigned int *bytes_per_element) +{ + /* DML: get_bytes_per_element */ + switch (format) { + case SURFACE_PIXEL_FORMAT_GRPH_ARGB1555: + case SURFACE_PIXEL_FORMAT_GRPH_RGB565: + *bytes_per_element = 2; + return true; + case SURFACE_PIXEL_FORMAT_GRPH_ARGB8888: + case SURFACE_PIXEL_FORMAT_GRPH_ABGR8888: + case SURFACE_PIXEL_FORMAT_GRPH_ARGB2101010: + case SURFACE_PIXEL_FORMAT_GRPH_ABGR2101010: + *bytes_per_element = 4; + return true; + case SURFACE_PIXEL_FORMAT_GRPH_ARGB16161616: + case SURFACE_PIXEL_FORMAT_GRPH_ARGB16161616F: + case SURFACE_PIXEL_FORMAT_GRPH_ABGR16161616F: + *bytes_per_element = 8; + return true; + default: + return false; + } +} + +static void hubbub1_get_blk256_size(unsigned int *blk256_width, unsigned int *blk256_height, + unsigned int bytes_per_element) +{ + /* copied from DML. might want to refactor DML to leverage from DML */ + /* DML : get_blk256_size */ + if (bytes_per_element == 1) { + *blk256_width = 16; + *blk256_height = 16; + } else if (bytes_per_element == 2) { + *blk256_width = 16; + *blk256_height = 8; + } else if (bytes_per_element == 4) { + *blk256_width = 8; + *blk256_height = 8; + } else if (bytes_per_element == 8) { + *blk256_width = 8; + *blk256_height = 4; + } +} + +static void hubbub1_det_request_size( + unsigned int height, + unsigned int width, + unsigned int bpe, + bool *req128_horz_wc, + bool *req128_vert_wc) +{ + unsigned int detile_buf_size = 164 * 1024; /* 164KB for DCN1.0 */ + + unsigned int blk256_height = 0; + unsigned int blk256_width = 0; + unsigned int swath_bytes_horz_wc, swath_bytes_vert_wc; + + hubbub1_get_blk256_size(&blk256_width, &blk256_height, bpe); + + swath_bytes_horz_wc = height * blk256_height * bpe; + swath_bytes_vert_wc = width * blk256_width * bpe; + + *req128_horz_wc = (2 * swath_bytes_horz_wc <= detile_buf_size) ? + false : /* full 256B request */ + true; /* half 128b request */ + + *req128_vert_wc = (2 * swath_bytes_vert_wc <= detile_buf_size) ? + false : /* full 256B request */ + true; /* half 128b request */ +} + +static bool hubbub1_get_dcc_compression_cap(struct hubbub *hubbub, + const struct dc_dcc_surface_param *input, + struct dc_surface_dcc_cap *output) +{ + struct dc *dc = hubbub->ctx->dc; + /* implement section 1.6.2.1 of DCN1_Programming_Guide.docx */ + enum dcc_control dcc_control; + unsigned int bpe; + enum segment_order segment_order_horz, segment_order_vert; + bool req128_horz_wc, req128_vert_wc; + + memset(output, 0, sizeof(*output)); + + if (dc->debug.disable_dcc == DCC_DISABLE) + return false; + + if (!hubbub->funcs->dcc_support_pixel_format(input->format, &bpe)) + return false; + + if (!hubbub->funcs->dcc_support_swizzle(input->swizzle_mode, bpe, + &segment_order_horz, &segment_order_vert)) + return false; + + hubbub1_det_request_size(input->surface_size.height, input->surface_size.width, + bpe, &req128_horz_wc, &req128_vert_wc); + + if (!req128_horz_wc && !req128_vert_wc) { + dcc_control = dcc_control__256_256_xxx; + } else if (input->scan == SCAN_DIRECTION_HORIZONTAL) { + if (!req128_horz_wc) + dcc_control = dcc_control__256_256_xxx; + else if (segment_order_horz == segment_order__contiguous) + dcc_control = dcc_control__128_128_xxx; + else + dcc_control = dcc_control__256_64_64; + } else if (input->scan == SCAN_DIRECTION_VERTICAL) { + if (!req128_vert_wc) + dcc_control = dcc_control__256_256_xxx; + else if (segment_order_vert == segment_order__contiguous) + dcc_control = dcc_control__128_128_xxx; + else + dcc_control = dcc_control__256_64_64; + } else { + if ((req128_horz_wc && + segment_order_horz == segment_order__non_contiguous) || + (req128_vert_wc && + segment_order_vert == segment_order__non_contiguous)) + /* access_dir not known, must use most constraining */ + dcc_control = dcc_control__256_64_64; + else + /* reg128 is true for either horz and vert + * but segment_order is contiguous + */ + dcc_control = dcc_control__128_128_xxx; + } + + if (dc->debug.disable_dcc == DCC_HALF_REQ_DISALBE && + dcc_control != dcc_control__256_256_xxx) + return false; + + switch (dcc_control) { + case dcc_control__256_256_xxx: + output->grph.rgb.max_uncompressed_blk_size = 256; + output->grph.rgb.max_compressed_blk_size = 256; + output->grph.rgb.independent_64b_blks = false; + break; + case dcc_control__128_128_xxx: + output->grph.rgb.max_uncompressed_blk_size = 128; + output->grph.rgb.max_compressed_blk_size = 128; + output->grph.rgb.independent_64b_blks = false; + break; + case dcc_control__256_64_64: + output->grph.rgb.max_uncompressed_blk_size = 256; + output->grph.rgb.max_compressed_blk_size = 64; + output->grph.rgb.independent_64b_blks = true; + break; + } + + output->capable = true; + output->const_color_support = false; + + return true; +} + static const struct hubbub_funcs hubbub1_funcs = { - .update_dchub = hubbub1_update_dchub + .update_dchub = hubbub1_update_dchub, + .dcc_support_swizzle = hubbub1_dcc_support_swizzle, + .dcc_support_pixel_format = hubbub1_dcc_support_pixel_format, + .get_dcc_compression_cap = hubbub1_get_dcc_compression_cap, }; void hubbub1_construct(struct hubbub *hubbub, diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubbub.h b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubbub.h index a16e908821a0..6315a0e6b0d6 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubbub.h +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubbub.h @@ -27,6 +27,7 @@ #define __DC_HUBBUB_DCN10_H__ #include "core_types.h" +#include "dchubbub.h" #define HUBHUB_REG_LIST_DCN()\ SR(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_A),\ @@ -47,7 +48,8 @@ SR(DCHUBBUB_ARB_DF_REQ_OUTSTAND),\ SR(DCHUBBUB_GLOBAL_TIMER_CNTL), \ SR(DCHUBBUB_TEST_DEBUG_INDEX), \ - SR(DCHUBBUB_TEST_DEBUG_DATA) + SR(DCHUBBUB_TEST_DEBUG_DATA),\ + SR(DCHUBBUB_SOFT_RESET) #define HUBBUB_SR_WATERMARK_REG_LIST()\ SR(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_A),\ @@ -104,6 +106,7 @@ struct dcn_hubbub_registers { uint32_t DCHUBBUB_SDPIF_AGP_BOT; uint32_t DCHUBBUB_SDPIF_AGP_TOP; uint32_t DCHUBBUB_CRC_CTRL; + uint32_t DCHUBBUB_SOFT_RESET; }; /* set field name */ @@ -113,6 +116,7 @@ struct dcn_hubbub_registers { #define HUBBUB_MASK_SH_LIST_DCN(mask_sh)\ HUBBUB_SF(DCHUBBUB_GLOBAL_TIMER_CNTL, DCHUBBUB_GLOBAL_TIMER_ENABLE, mask_sh), \ + HUBBUB_SF(DCHUBBUB_SOFT_RESET, DCHUBBUB_GLOBAL_SOFT_RESET, mask_sh), \ HUBBUB_SF(DCHUBBUB_ARB_WATERMARK_CHANGE_CNTL, DCHUBBUB_ARB_WATERMARK_CHANGE_REQUEST, mask_sh), \ HUBBUB_SF(DCHUBBUB_ARB_WATERMARK_CHANGE_CNTL, DCHUBBUB_ARB_WATERMARK_CHANGE_DONE_INTERRUPT_DISABLE, mask_sh), \ HUBBUB_SF(DCHUBBUB_ARB_DRAM_STATE_CNTL, DCHUBBUB_ARB_ALLOW_SELF_REFRESH_FORCE_VALUE, mask_sh), \ @@ -142,6 +146,7 @@ struct dcn_hubbub_registers { type DCHUBBUB_ARB_SAT_LEVEL;\ type DCHUBBUB_ARB_MIN_REQ_OUTSTAND;\ type DCHUBBUB_GLOBAL_TIMER_REFDIV;\ + type DCHUBBUB_GLOBAL_SOFT_RESET; \ type SDPIF_FB_TOP;\ type SDPIF_FB_BASE;\ type SDPIF_FB_OFFSET;\ @@ -173,12 +178,6 @@ struct dcn_hubbub_wm { struct dcn_hubbub_wm_set sets[4]; }; -struct hubbub_funcs { - void (*update_dchub)( - struct hubbub *hubbub, - struct dchub_init_data *dh_data); -}; - struct hubbub { const struct hubbub_funcs *funcs; struct dc_context *ctx; @@ -206,6 +205,7 @@ void hubbub1_toggle_watermark_change_req( void hubbub1_wm_read_state(struct hubbub *hubbub, struct dcn_hubbub_wm *wm); +void hubbub1_soft_reset(struct hubbub *hubbub, bool reset); void hubbub1_construct(struct hubbub *hubbub, struct dc_context *ctx, const struct dcn_hubbub_registers *hubbub_regs, diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.c index 39b72f696ae9..c28085be39ff 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.c @@ -78,6 +78,27 @@ static void hubp1_disconnect(struct hubp *hubp) CURSOR_ENABLE, 0); } +static void hubp1_disable_control(struct hubp *hubp, bool disable_hubp) +{ + struct dcn10_hubp *hubp1 = TO_DCN10_HUBP(hubp); + uint32_t disable = disable_hubp ? 1 : 0; + + REG_UPDATE(DCHUBP_CNTL, + HUBP_DISABLE, disable); +} + +static unsigned int hubp1_get_underflow_status(struct hubp *hubp) +{ + uint32_t hubp_underflow = 0; + struct dcn10_hubp *hubp1 = TO_DCN10_HUBP(hubp); + + REG_GET(DCHUBP_CNTL, + HUBP_UNDERFLOW_STATUS, + &hubp_underflow); + + return hubp_underflow; +} + static void hubp1_set_hubp_blank_en(struct hubp *hubp, bool blank) { struct dcn10_hubp *hubp1 = TO_DCN10_HUBP(hubp); @@ -146,6 +167,9 @@ void hubp1_program_size_and_rotation( * 444 or 420 luma */ if (format >= SURFACE_PIXEL_FORMAT_VIDEO_BEGIN) { + ASSERT(plane_size->video.chroma_pitch != 0); + /* Chroma pitch zero can cause system hang! */ + pitch = plane_size->video.luma_pitch - 1; meta_pitch = dcc->video.meta_pitch_l - 1; pitch_c = plane_size->video.chroma_pitch - 1; @@ -372,11 +396,15 @@ bool hubp1_program_surface_flip_and_addr( if (address->grph_stereo.right_addr.quad_part == 0) break; - REG_UPDATE_4(DCSURF_SURFACE_CONTROL, + REG_UPDATE_8(DCSURF_SURFACE_CONTROL, PRIMARY_SURFACE_TMZ, address->tmz_surface, PRIMARY_SURFACE_TMZ_C, address->tmz_surface, PRIMARY_META_SURFACE_TMZ, address->tmz_surface, - PRIMARY_META_SURFACE_TMZ_C, address->tmz_surface); + PRIMARY_META_SURFACE_TMZ_C, address->tmz_surface, + SECONDARY_SURFACE_TMZ, address->tmz_surface, + SECONDARY_SURFACE_TMZ_C, address->tmz_surface, + SECONDARY_META_SURFACE_TMZ, address->tmz_surface, + SECONDARY_META_SURFACE_TMZ_C, address->tmz_surface); if (address->grph_stereo.right_meta_addr.quad_part != 0) { @@ -435,9 +463,11 @@ void hubp1_dcc_control(struct hubp *hubp, bool enable, uint32_t dcc_ind_64b_blk = independent_64b_blks ? 1 : 0; struct dcn10_hubp *hubp1 = TO_DCN10_HUBP(hubp); - REG_UPDATE_2(DCSURF_SURFACE_CONTROL, + REG_UPDATE_4(DCSURF_SURFACE_CONTROL, PRIMARY_SURFACE_DCC_EN, dcc_en, - PRIMARY_SURFACE_DCC_IND_64B_BLK, dcc_ind_64b_blk); + PRIMARY_SURFACE_DCC_IND_64B_BLK, dcc_ind_64b_blk, + SECONDARY_SURFACE_DCC_EN, dcc_en, + SECONDARY_SURFACE_DCC_IND_64B_BLK, dcc_ind_64b_blk); } void hubp1_program_surface_config( @@ -535,11 +565,13 @@ void hubp1_program_deadline( REG_SET(VBLANK_PARAMETERS_3, 0, REFCYC_PER_META_CHUNK_VBLANK_L, dlg_attr->refcyc_per_meta_chunk_vblank_l); - REG_SET(NOM_PARAMETERS_0, 0, - DST_Y_PER_PTE_ROW_NOM_L, dlg_attr->dst_y_per_pte_row_nom_l); + if (REG(NOM_PARAMETERS_0)) + REG_SET(NOM_PARAMETERS_0, 0, + DST_Y_PER_PTE_ROW_NOM_L, dlg_attr->dst_y_per_pte_row_nom_l); - REG_SET(NOM_PARAMETERS_1, 0, - REFCYC_PER_PTE_GROUP_NOM_L, dlg_attr->refcyc_per_pte_group_nom_l); + if (REG(NOM_PARAMETERS_1)) + REG_SET(NOM_PARAMETERS_1, 0, + REFCYC_PER_PTE_GROUP_NOM_L, dlg_attr->refcyc_per_pte_group_nom_l); REG_SET(NOM_PARAMETERS_4, 0, DST_Y_PER_META_ROW_NOM_L, dlg_attr->dst_y_per_meta_row_nom_l); @@ -568,11 +600,13 @@ void hubp1_program_deadline( REG_SET(VBLANK_PARAMETERS_4, 0, REFCYC_PER_META_CHUNK_VBLANK_C, dlg_attr->refcyc_per_meta_chunk_vblank_c); - REG_SET(NOM_PARAMETERS_2, 0, - DST_Y_PER_PTE_ROW_NOM_C, dlg_attr->dst_y_per_pte_row_nom_c); + if (REG(NOM_PARAMETERS_2)) + REG_SET(NOM_PARAMETERS_2, 0, + DST_Y_PER_PTE_ROW_NOM_C, dlg_attr->dst_y_per_pte_row_nom_c); - REG_SET(NOM_PARAMETERS_3, 0, - REFCYC_PER_PTE_GROUP_NOM_C, dlg_attr->refcyc_per_pte_group_nom_c); + if (REG(NOM_PARAMETERS_3)) + REG_SET(NOM_PARAMETERS_3, 0, + REFCYC_PER_PTE_GROUP_NOM_C, dlg_attr->refcyc_per_pte_group_nom_c); REG_SET(NOM_PARAMETERS_6, 0, DST_Y_PER_META_ROW_NOM_C, dlg_attr->dst_y_per_meta_row_nom_c); @@ -609,6 +643,13 @@ void hubp1_program_deadline( REG_SET(DCN_SURF1_TTU_CNTL1, 0, REFCYC_PER_REQ_DELIVERY_PRE, ttu_attr->refcyc_per_req_delivery_pre_c); + + REG_SET_3(DCN_CUR0_TTU_CNTL0, 0, + REFCYC_PER_REQ_DELIVERY, ttu_attr->refcyc_per_req_delivery_cur0, + QoS_LEVEL_FIXED, ttu_attr->qos_level_fixed_cur0, + QoS_RAMP_DISABLE, ttu_attr->qos_ramp_disable_cur0); + REG_SET(DCN_CUR0_TTU_CNTL1, 0, + REFCYC_PER_REQ_DELIVERY_PRE, ttu_attr->refcyc_per_req_delivery_pre_cur0); } static void hubp1_setup( @@ -752,9 +793,159 @@ void min_set_viewport( PRI_VIEWPORT_Y_START_C, viewport_c->y); } -void hubp1_read_state(struct dcn10_hubp *hubp1, - struct dcn_hubp_state *s) +void hubp1_read_state(struct hubp *hubp) { + struct dcn10_hubp *hubp1 = TO_DCN10_HUBP(hubp); + struct dcn_hubp_state *s = &hubp1->state; + struct _vcs_dpi_display_dlg_regs_st *dlg_attr = &s->dlg_attr; + struct _vcs_dpi_display_ttu_regs_st *ttu_attr = &s->ttu_attr; + struct _vcs_dpi_display_rq_regs_st *rq_regs = &s->rq_regs; + + /* Requester */ + REG_GET(HUBPRET_CONTROL, + DET_BUF_PLANE1_BASE_ADDRESS, &rq_regs->plane1_base_address); + REG_GET_4(DCN_EXPANSION_MODE, + DRQ_EXPANSION_MODE, &rq_regs->drq_expansion_mode, + PRQ_EXPANSION_MODE, &rq_regs->prq_expansion_mode, + MRQ_EXPANSION_MODE, &rq_regs->mrq_expansion_mode, + CRQ_EXPANSION_MODE, &rq_regs->crq_expansion_mode); + REG_GET_8(DCHUBP_REQ_SIZE_CONFIG, + CHUNK_SIZE, &rq_regs->rq_regs_l.chunk_size, + MIN_CHUNK_SIZE, &rq_regs->rq_regs_l.min_chunk_size, + META_CHUNK_SIZE, &rq_regs->rq_regs_l.meta_chunk_size, + MIN_META_CHUNK_SIZE, &rq_regs->rq_regs_l.min_meta_chunk_size, + DPTE_GROUP_SIZE, &rq_regs->rq_regs_l.dpte_group_size, + MPTE_GROUP_SIZE, &rq_regs->rq_regs_l.mpte_group_size, + SWATH_HEIGHT, &rq_regs->rq_regs_l.swath_height, + PTE_ROW_HEIGHT_LINEAR, &rq_regs->rq_regs_l.pte_row_height_linear); + REG_GET_8(DCHUBP_REQ_SIZE_CONFIG_C, + CHUNK_SIZE_C, &rq_regs->rq_regs_c.chunk_size, + MIN_CHUNK_SIZE_C, &rq_regs->rq_regs_c.min_chunk_size, + META_CHUNK_SIZE_C, &rq_regs->rq_regs_c.meta_chunk_size, + MIN_META_CHUNK_SIZE_C, &rq_regs->rq_regs_c.min_meta_chunk_size, + DPTE_GROUP_SIZE_C, &rq_regs->rq_regs_c.dpte_group_size, + MPTE_GROUP_SIZE_C, &rq_regs->rq_regs_c.mpte_group_size, + SWATH_HEIGHT_C, &rq_regs->rq_regs_c.swath_height, + PTE_ROW_HEIGHT_LINEAR_C, &rq_regs->rq_regs_c.pte_row_height_linear); + + /* DLG - Per hubp */ + REG_GET_2(BLANK_OFFSET_0, + REFCYC_H_BLANK_END, &dlg_attr->refcyc_h_blank_end, + DLG_V_BLANK_END, &dlg_attr->dlg_vblank_end); + + REG_GET(BLANK_OFFSET_1, + MIN_DST_Y_NEXT_START, &dlg_attr->min_dst_y_next_start); + + REG_GET(DST_DIMENSIONS, + REFCYC_PER_HTOTAL, &dlg_attr->refcyc_per_htotal); + + REG_GET_2(DST_AFTER_SCALER, + REFCYC_X_AFTER_SCALER, &dlg_attr->refcyc_x_after_scaler, + DST_Y_AFTER_SCALER, &dlg_attr->dst_y_after_scaler); + + if (REG(PREFETCH_SETTINS)) + REG_GET_2(PREFETCH_SETTINS, + DST_Y_PREFETCH, &dlg_attr->dst_y_prefetch, + VRATIO_PREFETCH, &dlg_attr->vratio_prefetch); + else + REG_GET_2(PREFETCH_SETTINGS, + DST_Y_PREFETCH, &dlg_attr->dst_y_prefetch, + VRATIO_PREFETCH, &dlg_attr->vratio_prefetch); + + REG_GET_2(VBLANK_PARAMETERS_0, + DST_Y_PER_VM_VBLANK, &dlg_attr->dst_y_per_vm_vblank, + DST_Y_PER_ROW_VBLANK, &dlg_attr->dst_y_per_row_vblank); + + REG_GET(REF_FREQ_TO_PIX_FREQ, + REF_FREQ_TO_PIX_FREQ, &dlg_attr->ref_freq_to_pix_freq); + + /* DLG - Per luma/chroma */ + REG_GET(VBLANK_PARAMETERS_1, + REFCYC_PER_PTE_GROUP_VBLANK_L, &dlg_attr->refcyc_per_pte_group_vblank_l); + + REG_GET(VBLANK_PARAMETERS_3, + REFCYC_PER_META_CHUNK_VBLANK_L, &dlg_attr->refcyc_per_meta_chunk_vblank_l); + + if (REG(NOM_PARAMETERS_0)) + REG_GET(NOM_PARAMETERS_0, + DST_Y_PER_PTE_ROW_NOM_L, &dlg_attr->dst_y_per_pte_row_nom_l); + + if (REG(NOM_PARAMETERS_1)) + REG_GET(NOM_PARAMETERS_1, + REFCYC_PER_PTE_GROUP_NOM_L, &dlg_attr->refcyc_per_pte_group_nom_l); + + REG_GET(NOM_PARAMETERS_4, + DST_Y_PER_META_ROW_NOM_L, &dlg_attr->dst_y_per_meta_row_nom_l); + + REG_GET(NOM_PARAMETERS_5, + REFCYC_PER_META_CHUNK_NOM_L, &dlg_attr->refcyc_per_meta_chunk_nom_l); + + REG_GET_2(PER_LINE_DELIVERY_PRE, + REFCYC_PER_LINE_DELIVERY_PRE_L, &dlg_attr->refcyc_per_line_delivery_pre_l, + REFCYC_PER_LINE_DELIVERY_PRE_C, &dlg_attr->refcyc_per_line_delivery_pre_c); + + REG_GET_2(PER_LINE_DELIVERY, + REFCYC_PER_LINE_DELIVERY_L, &dlg_attr->refcyc_per_line_delivery_l, + REFCYC_PER_LINE_DELIVERY_C, &dlg_attr->refcyc_per_line_delivery_c); + + if (REG(PREFETCH_SETTINS_C)) + REG_GET(PREFETCH_SETTINS_C, + VRATIO_PREFETCH_C, &dlg_attr->vratio_prefetch_c); + else + REG_GET(PREFETCH_SETTINGS_C, + VRATIO_PREFETCH_C, &dlg_attr->vratio_prefetch_c); + + REG_GET(VBLANK_PARAMETERS_2, + REFCYC_PER_PTE_GROUP_VBLANK_C, &dlg_attr->refcyc_per_pte_group_vblank_c); + + REG_GET(VBLANK_PARAMETERS_4, + REFCYC_PER_META_CHUNK_VBLANK_C, &dlg_attr->refcyc_per_meta_chunk_vblank_c); + + if (REG(NOM_PARAMETERS_2)) + REG_GET(NOM_PARAMETERS_2, + DST_Y_PER_PTE_ROW_NOM_C, &dlg_attr->dst_y_per_pte_row_nom_c); + + if (REG(NOM_PARAMETERS_3)) + REG_GET(NOM_PARAMETERS_3, + REFCYC_PER_PTE_GROUP_NOM_C, &dlg_attr->refcyc_per_pte_group_nom_c); + + REG_GET(NOM_PARAMETERS_6, + DST_Y_PER_META_ROW_NOM_C, &dlg_attr->dst_y_per_meta_row_nom_c); + + REG_GET(NOM_PARAMETERS_7, + REFCYC_PER_META_CHUNK_NOM_C, &dlg_attr->refcyc_per_meta_chunk_nom_c); + + /* TTU - per hubp */ + REG_GET_2(DCN_TTU_QOS_WM, + QoS_LEVEL_LOW_WM, &ttu_attr->qos_level_low_wm, + QoS_LEVEL_HIGH_WM, &ttu_attr->qos_level_high_wm); + + REG_GET_2(DCN_GLOBAL_TTU_CNTL, + MIN_TTU_VBLANK, &ttu_attr->min_ttu_vblank, + QoS_LEVEL_FLIP, &ttu_attr->qos_level_flip); + + /* TTU - per luma/chroma */ + /* Assumed surf0 is luma and 1 is chroma */ + + REG_GET_3(DCN_SURF0_TTU_CNTL0, + REFCYC_PER_REQ_DELIVERY, &ttu_attr->refcyc_per_req_delivery_l, + QoS_LEVEL_FIXED, &ttu_attr->qos_level_fixed_l, + QoS_RAMP_DISABLE, &ttu_attr->qos_ramp_disable_l); + + REG_GET(DCN_SURF0_TTU_CNTL1, + REFCYC_PER_REQ_DELIVERY_PRE, + &ttu_attr->refcyc_per_req_delivery_pre_l); + + REG_GET_3(DCN_SURF1_TTU_CNTL0, + REFCYC_PER_REQ_DELIVERY, &ttu_attr->refcyc_per_req_delivery_c, + QoS_LEVEL_FIXED, &ttu_attr->qos_level_fixed_c, + QoS_RAMP_DISABLE, &ttu_attr->qos_ramp_disable_c); + + REG_GET(DCN_SURF1_TTU_CNTL1, + REFCYC_PER_REQ_DELIVERY_PRE, + &ttu_attr->refcyc_per_req_delivery_pre_c); + + /* Rest of hubp */ REG_GET(DCSURF_SURFACE_CONFIG, SURFACE_PIXEL_FORMAT, &s->pixel_format); @@ -890,14 +1081,14 @@ void hubp1_cursor_set_position( ASSERT(param->h_scale_ratio.value); if (param->h_scale_ratio.value) - dst_x_offset = dal_fixed31_32_floor(dal_fixed31_32_div( - dal_fixed31_32_from_int(dst_x_offset), + dst_x_offset = dc_fixpt_floor(dc_fixpt_div( + dc_fixpt_from_int(dst_x_offset), param->h_scale_ratio)); if (src_x_offset >= (int)param->viewport_width) cur_en = 0; /* not visible beyond right edge*/ - if (src_x_offset + (int)hubp->curs_attr.width < 0) + if (src_x_offset + (int)hubp->curs_attr.width <= 0) cur_en = 0; /* not visible beyond left edge*/ if (cur_en && REG_READ(CURSOR_SURFACE_ADDRESS) == 0) @@ -952,6 +1143,10 @@ static struct hubp_funcs dcn10_hubp_funcs = { .hubp_disconnect = hubp1_disconnect, .hubp_clk_cntl = hubp1_clk_cntl, .hubp_vtg_sel = hubp1_vtg_sel, + .hubp_read_state = hubp1_read_state, + .hubp_disable_control = hubp1_disable_control, + .hubp_get_underflow_status = hubp1_get_underflow_status, + }; /*****************************************/ diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.h b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.h index 4a3703e12ea1..d901d5092969 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.h +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.h @@ -30,6 +30,7 @@ #define TO_DCN10_HUBP(hubp)\ container_of(hubp, struct dcn10_hubp, base) +/* Register address initialization macro for all ASICs (including those with reduced functionality) */ #define HUBP_REG_LIST_DCN(id)\ SRI(DCHUBP_CNTL, HUBP, id),\ SRI(HUBPREQ_DEBUG_DB, HUBP, id),\ @@ -78,16 +79,12 @@ SRI(REF_FREQ_TO_PIX_FREQ, HUBPREQ, id),\ SRI(VBLANK_PARAMETERS_1, HUBPREQ, id),\ SRI(VBLANK_PARAMETERS_3, HUBPREQ, id),\ - SRI(NOM_PARAMETERS_0, HUBPREQ, id),\ - SRI(NOM_PARAMETERS_1, HUBPREQ, id),\ SRI(NOM_PARAMETERS_4, HUBPREQ, id),\ SRI(NOM_PARAMETERS_5, HUBPREQ, id),\ SRI(PER_LINE_DELIVERY_PRE, HUBPREQ, id),\ SRI(PER_LINE_DELIVERY, HUBPREQ, id),\ SRI(VBLANK_PARAMETERS_2, HUBPREQ, id),\ SRI(VBLANK_PARAMETERS_4, HUBPREQ, id),\ - SRI(NOM_PARAMETERS_2, HUBPREQ, id),\ - SRI(NOM_PARAMETERS_3, HUBPREQ, id),\ SRI(NOM_PARAMETERS_6, HUBPREQ, id),\ SRI(NOM_PARAMETERS_7, HUBPREQ, id),\ SRI(DCN_TTU_QOS_WM, HUBPREQ, id),\ @@ -96,11 +93,21 @@ SRI(DCN_SURF0_TTU_CNTL1, HUBPREQ, id),\ SRI(DCN_SURF1_TTU_CNTL0, HUBPREQ, id),\ SRI(DCN_SURF1_TTU_CNTL1, HUBPREQ, id),\ - SRI(DCN_VM_MX_L1_TLB_CNTL, HUBPREQ, id),\ + SRI(DCN_CUR0_TTU_CNTL0, HUBPREQ, id),\ + SRI(DCN_CUR0_TTU_CNTL1, HUBPREQ, id),\ SRI(HUBP_CLK_CNTL, HUBP, id) +/* Register address initialization macro for ASICs with VM */ +#define HUBP_REG_LIST_DCN_VM(id)\ + SRI(NOM_PARAMETERS_0, HUBPREQ, id),\ + SRI(NOM_PARAMETERS_1, HUBPREQ, id),\ + SRI(NOM_PARAMETERS_2, HUBPREQ, id),\ + SRI(NOM_PARAMETERS_3, HUBPREQ, id),\ + SRI(DCN_VM_MX_L1_TLB_CNTL, HUBPREQ, id) + #define HUBP_REG_LIST_DCN10(id)\ HUBP_REG_LIST_DCN(id),\ + HUBP_REG_LIST_DCN_VM(id),\ SRI(PREFETCH_SETTINS, HUBPREQ, id),\ SRI(PREFETCH_SETTINS_C, HUBPREQ, id),\ SRI(DCN_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_MSB, HUBPREQ, id),\ @@ -198,6 +205,8 @@ uint32_t DCN_SURF0_TTU_CNTL1; \ uint32_t DCN_SURF1_TTU_CNTL0; \ uint32_t DCN_SURF1_TTU_CNTL1; \ + uint32_t DCN_CUR0_TTU_CNTL0; \ + uint32_t DCN_CUR0_TTU_CNTL1; \ uint32_t DCN_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_MSB; \ uint32_t DCN_VM_CONTEXT0_PAGE_TABLE_BASE_ADDR_LSB; \ uint32_t DCN_VM_CONTEXT0_PAGE_TABLE_START_ADDR_MSB; \ @@ -237,12 +246,14 @@ #define HUBP_SF(reg_name, field_name, post_fix)\ .field_name = reg_name ## __ ## field_name ## post_fix +/* Mask/shift struct generation macro for all ASICs (including those with reduced functionality) */ #define HUBP_MASK_SH_LIST_DCN(mask_sh)\ HUBP_SF(HUBP0_DCHUBP_CNTL, HUBP_BLANK_EN, mask_sh),\ HUBP_SF(HUBP0_DCHUBP_CNTL, HUBP_TTU_DISABLE, mask_sh),\ HUBP_SF(HUBP0_DCHUBP_CNTL, HUBP_UNDERFLOW_STATUS, mask_sh),\ HUBP_SF(HUBP0_DCHUBP_CNTL, HUBP_NO_OUTSTANDING_REQ, mask_sh),\ HUBP_SF(HUBP0_DCHUBP_CNTL, HUBP_VTG_SEL, mask_sh),\ + HUBP_SF(HUBP0_DCHUBP_CNTL, HUBP_DISABLE, mask_sh),\ HUBP_SF(HUBP0_DCSURF_ADDR_CONFIG, NUM_PIPES, mask_sh),\ HUBP_SF(HUBP0_DCSURF_ADDR_CONFIG, NUM_BANKS, mask_sh),\ HUBP_SF(HUBP0_DCSURF_ADDR_CONFIG, PIPE_INTERLEAVE, mask_sh),\ @@ -301,6 +312,12 @@ HUBP_SF(HUBPREQ0_DCSURF_SURFACE_CONTROL, PRIMARY_META_SURFACE_TMZ_C, mask_sh),\ HUBP_SF(HUBPREQ0_DCSURF_SURFACE_CONTROL, PRIMARY_SURFACE_DCC_EN, mask_sh),\ HUBP_SF(HUBPREQ0_DCSURF_SURFACE_CONTROL, PRIMARY_SURFACE_DCC_IND_64B_BLK, mask_sh),\ + HUBP_SF(HUBPREQ0_DCSURF_SURFACE_CONTROL, SECONDARY_SURFACE_TMZ, mask_sh),\ + HUBP_SF(HUBPREQ0_DCSURF_SURFACE_CONTROL, SECONDARY_SURFACE_TMZ_C, mask_sh),\ + HUBP_SF(HUBPREQ0_DCSURF_SURFACE_CONTROL, SECONDARY_META_SURFACE_TMZ, mask_sh),\ + HUBP_SF(HUBPREQ0_DCSURF_SURFACE_CONTROL, SECONDARY_META_SURFACE_TMZ_C, mask_sh),\ + HUBP_SF(HUBPREQ0_DCSURF_SURFACE_CONTROL, SECONDARY_SURFACE_DCC_EN, mask_sh),\ + HUBP_SF(HUBPREQ0_DCSURF_SURFACE_CONTROL, SECONDARY_SURFACE_DCC_IND_64B_BLK, mask_sh),\ HUBP_SF(HUBPRET0_HUBPRET_CONTROL, DET_BUF_PLANE1_BASE_ADDRESS, mask_sh),\ HUBP_SF(HUBPRET0_HUBPRET_CONTROL, CROSSBAR_SRC_CB_B, mask_sh),\ HUBP_SF(HUBPRET0_HUBPRET_CONTROL, CROSSBAR_SRC_CR_R, mask_sh),\ @@ -335,8 +352,6 @@ HUBP_SF(HUBPREQ0_REF_FREQ_TO_PIX_FREQ, REF_FREQ_TO_PIX_FREQ, mask_sh),\ HUBP_SF(HUBPREQ0_VBLANK_PARAMETERS_1, REFCYC_PER_PTE_GROUP_VBLANK_L, mask_sh),\ HUBP_SF(HUBPREQ0_VBLANK_PARAMETERS_3, REFCYC_PER_META_CHUNK_VBLANK_L, mask_sh),\ - HUBP_SF(HUBPREQ0_NOM_PARAMETERS_0, DST_Y_PER_PTE_ROW_NOM_L, mask_sh),\ - HUBP_SF(HUBPREQ0_NOM_PARAMETERS_1, REFCYC_PER_PTE_GROUP_NOM_L, mask_sh),\ HUBP_SF(HUBPREQ0_NOM_PARAMETERS_4, DST_Y_PER_META_ROW_NOM_L, mask_sh),\ HUBP_SF(HUBPREQ0_NOM_PARAMETERS_5, REFCYC_PER_META_CHUNK_NOM_L, mask_sh),\ HUBP_SF(HUBPREQ0_PER_LINE_DELIVERY_PRE, REFCYC_PER_LINE_DELIVERY_PRE_L, mask_sh),\ @@ -345,8 +360,6 @@ HUBP_SF(HUBPREQ0_PER_LINE_DELIVERY, REFCYC_PER_LINE_DELIVERY_C, mask_sh),\ HUBP_SF(HUBPREQ0_VBLANK_PARAMETERS_2, REFCYC_PER_PTE_GROUP_VBLANK_C, mask_sh),\ HUBP_SF(HUBPREQ0_VBLANK_PARAMETERS_4, REFCYC_PER_META_CHUNK_VBLANK_C, mask_sh),\ - HUBP_SF(HUBPREQ0_NOM_PARAMETERS_2, DST_Y_PER_PTE_ROW_NOM_C, mask_sh),\ - HUBP_SF(HUBPREQ0_NOM_PARAMETERS_3, REFCYC_PER_PTE_GROUP_NOM_C, mask_sh),\ HUBP_SF(HUBPREQ0_NOM_PARAMETERS_6, DST_Y_PER_META_ROW_NOM_C, mask_sh),\ HUBP_SF(HUBPREQ0_NOM_PARAMETERS_7, REFCYC_PER_META_CHUNK_NOM_C, mask_sh),\ HUBP_SF(HUBPREQ0_DCN_TTU_QOS_WM, QoS_LEVEL_LOW_WM, mask_sh),\ @@ -357,12 +370,24 @@ HUBP_SF(HUBPREQ0_DCN_SURF0_TTU_CNTL0, QoS_LEVEL_FIXED, mask_sh),\ HUBP_SF(HUBPREQ0_DCN_SURF0_TTU_CNTL0, QoS_RAMP_DISABLE, mask_sh),\ HUBP_SF(HUBPREQ0_DCN_SURF0_TTU_CNTL1, REFCYC_PER_REQ_DELIVERY_PRE, mask_sh),\ + HUBP_SF(HUBP0_HUBP_CLK_CNTL, HUBP_CLOCK_ENABLE, mask_sh) + +/* Mask/shift struct generation macro for ASICs with VM */ +#define HUBP_MASK_SH_LIST_DCN_VM(mask_sh)\ + HUBP_SF(HUBPREQ0_NOM_PARAMETERS_0, DST_Y_PER_PTE_ROW_NOM_L, mask_sh),\ + HUBP_SF(HUBPREQ0_NOM_PARAMETERS_1, REFCYC_PER_PTE_GROUP_NOM_L, mask_sh),\ + HUBP_SF(HUBPREQ0_NOM_PARAMETERS_2, DST_Y_PER_PTE_ROW_NOM_C, mask_sh),\ + HUBP_SF(HUBPREQ0_NOM_PARAMETERS_3, REFCYC_PER_PTE_GROUP_NOM_C, mask_sh),\ HUBP_SF(HUBPREQ0_DCN_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB, mask_sh),\ HUBP_SF(HUBPREQ0_DCN_VM_MX_L1_TLB_CNTL, SYSTEM_ACCESS_MODE, mask_sh),\ - HUBP_SF(HUBP0_HUBP_CLK_CNTL, HUBP_CLOCK_ENABLE, mask_sh) + HUBP_SF(HUBPREQ0_DCN_CUR0_TTU_CNTL0, REFCYC_PER_REQ_DELIVERY, mask_sh),\ + HUBP_SF(HUBPREQ0_DCN_CUR0_TTU_CNTL0, QoS_LEVEL_FIXED, mask_sh),\ + HUBP_SF(HUBPREQ0_DCN_CUR0_TTU_CNTL0, QoS_RAMP_DISABLE, mask_sh),\ + HUBP_SF(HUBPREQ0_DCN_CUR0_TTU_CNTL1, REFCYC_PER_REQ_DELIVERY_PRE, mask_sh) #define HUBP_MASK_SH_LIST_DCN10(mask_sh)\ HUBP_MASK_SH_LIST_DCN(mask_sh),\ + HUBP_MASK_SH_LIST_DCN_VM(mask_sh),\ HUBP_SF(HUBPREQ0_PREFETCH_SETTINS, DST_Y_PREFETCH, mask_sh),\ HUBP_SF(HUBPREQ0_PREFETCH_SETTINS, VRATIO_PREFETCH, mask_sh),\ HUBP_SF(HUBPREQ0_PREFETCH_SETTINS_C, VRATIO_PREFETCH_C, mask_sh),\ @@ -403,6 +428,7 @@ #define DCN_HUBP_REG_FIELD_LIST(type) \ type HUBP_BLANK_EN;\ + type HUBP_DISABLE;\ type HUBP_TTU_DISABLE;\ type HUBP_NO_OUTSTANDING_REQ;\ type HUBP_VTG_SEL;\ @@ -469,6 +495,8 @@ type SECONDARY_META_SURFACE_TMZ_C;\ type PRIMARY_SURFACE_DCC_EN;\ type PRIMARY_SURFACE_DCC_IND_64B_BLK;\ + type SECONDARY_SURFACE_DCC_EN;\ + type SECONDARY_SURFACE_DCC_IND_64B_BLK;\ type DET_BUF_PLANE1_BASE_ADDRESS;\ type CROSSBAR_SRC_CB_B;\ type CROSSBAR_SRC_CR_R;\ @@ -601,8 +629,29 @@ struct dcn_mi_mask { DCN_HUBP_REG_FIELD_LIST(uint32_t); }; +struct dcn_hubp_state { + struct _vcs_dpi_display_dlg_regs_st dlg_attr; + struct _vcs_dpi_display_ttu_regs_st ttu_attr; + struct _vcs_dpi_display_rq_regs_st rq_regs; + uint32_t pixel_format; + uint32_t inuse_addr_hi; + uint32_t viewport_width; + uint32_t viewport_height; + uint32_t rotation_angle; + uint32_t h_mirror_en; + uint32_t sw_mode; + uint32_t dcc_en; + uint32_t blank_en; + uint32_t underflow_status; + uint32_t ttu_disable; + uint32_t min_ttu_vblank; + uint32_t qos_level_low_wm; + uint32_t qos_level_high_wm; +}; + struct dcn10_hubp { struct hubp base; + struct dcn_hubp_state state; const struct dcn_mi_registers *hubp_regs; const struct dcn_mi_shift *hubp_shift; const struct dcn_mi_mask *hubp_mask; @@ -680,26 +729,9 @@ void dcn10_hubp_construct( const struct dcn_mi_shift *hubp_shift, const struct dcn_mi_mask *hubp_mask); - -struct dcn_hubp_state { - uint32_t pixel_format; - uint32_t inuse_addr_hi; - uint32_t viewport_width; - uint32_t viewport_height; - uint32_t rotation_angle; - uint32_t h_mirror_en; - uint32_t sw_mode; - uint32_t dcc_en; - uint32_t blank_en; - uint32_t underflow_status; - uint32_t ttu_disable; - uint32_t min_ttu_vblank; - uint32_t qos_level_low_wm; - uint32_t qos_level_high_wm; -}; -void hubp1_read_state(struct dcn10_hubp *hubp1, - struct dcn_hubp_state *s); +void hubp1_read_state(struct hubp *hubp); enum cursor_pitch hubp1_get_cursor_pitch(unsigned int pitch); + #endif diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c index 8b0f6b8a5627..f8e0576af6e0 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c @@ -45,8 +45,8 @@ #include "dcn10_hubbub.h" #include "dcn10_cm_common.h" -#define DC_LOGGER \ - ctx->logger +#define DC_LOGGER_INIT(logger) + #define CTX \ hws->ctx #define REG(reg)\ @@ -56,16 +56,17 @@ #define FN(reg_name, field_name) \ hws->shifts->field_name, hws->masks->field_name +/*print is 17 wide, first two characters are spaces*/ #define DTN_INFO_MICRO_SEC(ref_cycle) \ print_microsec(dc_ctx, ref_cycle) void print_microsec(struct dc_context *dc_ctx, uint32_t ref_cycle) { - static const uint32_t ref_clk_mhz = 48; - static const unsigned int frac = 10; + const uint32_t ref_clk_mhz = dc_ctx->dc->res_pool->ref_clock_inKhz / 1000; + static const unsigned int frac = 1000; uint32_t us_x10 = (ref_cycle * frac) / ref_clk_mhz; - DTN_INFO("%d.%d \t ", + DTN_INFO(" %11d.%03d", us_x10 / frac, us_x10 % frac); } @@ -92,14 +93,14 @@ void dcn10_log_hubbub_state(struct dc *dc) hubbub1_wm_read_state(dc->res_pool->hubbub, &wm); - DTN_INFO("HUBBUB WM: \t data_urgent \t pte_meta_urgent \t " - "sr_enter \t sr_exit \t dram_clk_change \n"); + DTN_INFO("HUBBUB WM: data_urgent pte_meta_urgent" + " sr_enter sr_exit dram_clk_change\n"); for (i = 0; i < 4; i++) { struct dcn_hubbub_wm_set *s; s = &wm.sets[i]; - DTN_INFO("WM_Set[%d]:\t ", s->wm_set); + DTN_INFO("WM_Set[%d]:", s->wm_set); DTN_INFO_MICRO_SEC(s->data_urgent); DTN_INFO_MICRO_SEC(s->pte_meta_urgent); DTN_INFO_MICRO_SEC(s->sr_enter); @@ -111,6 +112,121 @@ void dcn10_log_hubbub_state(struct dc *dc) DTN_INFO("\n"); } +static void dcn10_log_hubp_states(struct dc *dc) +{ + struct dc_context *dc_ctx = dc->ctx; + struct resource_pool *pool = dc->res_pool; + int i; + + DTN_INFO("HUBP: format addr_hi width height" + " rot mir sw_mode dcc_en blank_en ttu_dis underflow" + " min_ttu_vblank qos_low_wm qos_high_wm\n"); + for (i = 0; i < pool->pipe_count; i++) { + struct hubp *hubp = pool->hubps[i]; + struct dcn_hubp_state *s = &(TO_DCN10_HUBP(hubp)->state); + + hubp->funcs->hubp_read_state(hubp); + + if (!s->blank_en) { + DTN_INFO("[%2d]: %5xh %6xh %5d %6d %2xh %2xh %6xh" + " %6d %8d %7d %8xh", + hubp->inst, + s->pixel_format, + s->inuse_addr_hi, + s->viewport_width, + s->viewport_height, + s->rotation_angle, + s->h_mirror_en, + s->sw_mode, + s->dcc_en, + s->blank_en, + s->ttu_disable, + s->underflow_status); + DTN_INFO_MICRO_SEC(s->min_ttu_vblank); + DTN_INFO_MICRO_SEC(s->qos_level_low_wm); + DTN_INFO_MICRO_SEC(s->qos_level_high_wm); + DTN_INFO("\n"); + } + } + + DTN_INFO("\n=========RQ========\n"); + DTN_INFO("HUBP: drq_exp_m prq_exp_m mrq_exp_m crq_exp_m plane1_ba L:chunk_s min_chu_s meta_ch_s" + " min_m_c_s dpte_gr_s mpte_gr_s swath_hei pte_row_h C:chunk_s min_chu_s meta_ch_s" + " min_m_c_s dpte_gr_s mpte_gr_s swath_hei pte_row_h\n"); + for (i = 0; i < pool->pipe_count; i++) { + struct dcn_hubp_state *s = &(TO_DCN10_HUBP(pool->hubps[i])->state); + struct _vcs_dpi_display_rq_regs_st *rq_regs = &s->rq_regs; + + if (!s->blank_en) + DTN_INFO("[%2d]: %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh\n", + pool->hubps[i]->inst, rq_regs->drq_expansion_mode, rq_regs->prq_expansion_mode, rq_regs->mrq_expansion_mode, + rq_regs->crq_expansion_mode, rq_regs->plane1_base_address, rq_regs->rq_regs_l.chunk_size, + rq_regs->rq_regs_l.min_chunk_size, rq_regs->rq_regs_l.meta_chunk_size, + rq_regs->rq_regs_l.min_meta_chunk_size, rq_regs->rq_regs_l.dpte_group_size, + rq_regs->rq_regs_l.mpte_group_size, rq_regs->rq_regs_l.swath_height, + rq_regs->rq_regs_l.pte_row_height_linear, rq_regs->rq_regs_c.chunk_size, rq_regs->rq_regs_c.min_chunk_size, + rq_regs->rq_regs_c.meta_chunk_size, rq_regs->rq_regs_c.min_meta_chunk_size, + rq_regs->rq_regs_c.dpte_group_size, rq_regs->rq_regs_c.mpte_group_size, + rq_regs->rq_regs_c.swath_height, rq_regs->rq_regs_c.pte_row_height_linear); + } + + DTN_INFO("========DLG========\n"); + DTN_INFO("HUBP: rc_hbe dlg_vbe min_d_y_n rc_per_ht rc_x_a_s " + " dst_y_a_s dst_y_pf dst_y_vvb dst_y_rvb dst_y_vfl dst_y_rfl rf_pix_fq" + " vratio_pf vrat_pf_c rc_pg_vbl rc_pg_vbc rc_mc_vbl rc_mc_vbc rc_pg_fll" + " rc_pg_flc rc_mc_fll rc_mc_flc pr_nom_l pr_nom_c rc_pg_nl rc_pg_nc " + " mr_nom_l mr_nom_c rc_mc_nl rc_mc_nc rc_ld_pl rc_ld_pc rc_ld_l " + " rc_ld_c cha_cur0 ofst_cur1 cha_cur1 vr_af_vc0 ddrq_limt x_rt_dlay" + " x_rp_dlay x_rr_sfl\n"); + for (i = 0; i < pool->pipe_count; i++) { + struct dcn_hubp_state *s = &(TO_DCN10_HUBP(pool->hubps[i])->state); + struct _vcs_dpi_display_dlg_regs_st *dlg_regs = &s->dlg_attr; + + if (!s->blank_en) + DTN_INFO("[%2d]: %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh" + "% 8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh" + " %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh\n", + pool->hubps[i]->inst, dlg_regs->refcyc_h_blank_end, dlg_regs->dlg_vblank_end, dlg_regs->min_dst_y_next_start, + dlg_regs->refcyc_per_htotal, dlg_regs->refcyc_x_after_scaler, dlg_regs->dst_y_after_scaler, + dlg_regs->dst_y_prefetch, dlg_regs->dst_y_per_vm_vblank, dlg_regs->dst_y_per_row_vblank, + dlg_regs->dst_y_per_vm_flip, dlg_regs->dst_y_per_row_flip, dlg_regs->ref_freq_to_pix_freq, + dlg_regs->vratio_prefetch, dlg_regs->vratio_prefetch_c, dlg_regs->refcyc_per_pte_group_vblank_l, + dlg_regs->refcyc_per_pte_group_vblank_c, dlg_regs->refcyc_per_meta_chunk_vblank_l, + dlg_regs->refcyc_per_meta_chunk_vblank_c, dlg_regs->refcyc_per_pte_group_flip_l, + dlg_regs->refcyc_per_pte_group_flip_c, dlg_regs->refcyc_per_meta_chunk_flip_l, + dlg_regs->refcyc_per_meta_chunk_flip_c, dlg_regs->dst_y_per_pte_row_nom_l, + dlg_regs->dst_y_per_pte_row_nom_c, dlg_regs->refcyc_per_pte_group_nom_l, + dlg_regs->refcyc_per_pte_group_nom_c, dlg_regs->dst_y_per_meta_row_nom_l, + dlg_regs->dst_y_per_meta_row_nom_c, dlg_regs->refcyc_per_meta_chunk_nom_l, + dlg_regs->refcyc_per_meta_chunk_nom_c, dlg_regs->refcyc_per_line_delivery_pre_l, + dlg_regs->refcyc_per_line_delivery_pre_c, dlg_regs->refcyc_per_line_delivery_l, + dlg_regs->refcyc_per_line_delivery_c, dlg_regs->chunk_hdl_adjust_cur0, dlg_regs->dst_y_offset_cur1, + dlg_regs->chunk_hdl_adjust_cur1, dlg_regs->vready_after_vcount0, dlg_regs->dst_y_delta_drq_limit, + dlg_regs->xfc_reg_transfer_delay, dlg_regs->xfc_reg_precharge_delay, + dlg_regs->xfc_reg_remote_surface_flip_latency); + } + + DTN_INFO("========TTU========\n"); + DTN_INFO("HUBP: qos_ll_wm qos_lh_wm mn_ttu_vb qos_l_flp rc_rd_p_l rc_rd_l rc_rd_p_c" + " rc_rd_c rc_rd_c0 rc_rd_pc0 rc_rd_c1 rc_rd_pc1 qos_lf_l qos_rds_l" + " qos_lf_c qos_rds_c qos_lf_c0 qos_rds_c0 qos_lf_c1 qos_rds_c1\n"); + for (i = 0; i < pool->pipe_count; i++) { + struct dcn_hubp_state *s = &(TO_DCN10_HUBP(pool->hubps[i])->state); + struct _vcs_dpi_display_ttu_regs_st *ttu_regs = &s->ttu_attr; + + if (!s->blank_en) + DTN_INFO("[%2d]: %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh\n", + pool->hubps[i]->inst, ttu_regs->qos_level_low_wm, ttu_regs->qos_level_high_wm, ttu_regs->min_ttu_vblank, + ttu_regs->qos_level_flip, ttu_regs->refcyc_per_req_delivery_pre_l, ttu_regs->refcyc_per_req_delivery_l, + ttu_regs->refcyc_per_req_delivery_pre_c, ttu_regs->refcyc_per_req_delivery_c, ttu_regs->refcyc_per_req_delivery_cur0, + ttu_regs->refcyc_per_req_delivery_pre_cur0, ttu_regs->refcyc_per_req_delivery_cur1, + ttu_regs->refcyc_per_req_delivery_pre_cur1, ttu_regs->qos_level_fixed_l, ttu_regs->qos_ramp_disable_l, + ttu_regs->qos_level_fixed_c, ttu_regs->qos_ramp_disable_c, ttu_regs->qos_level_fixed_cur0, + ttu_regs->qos_ramp_disable_cur0, ttu_regs->qos_level_fixed_cur1, ttu_regs->qos_ramp_disable_cur1); + } + DTN_INFO("\n"); +} + void dcn10_log_hw_state(struct dc *dc) { struct dc_context *dc_ctx = dc->ctx; @@ -121,41 +237,64 @@ void dcn10_log_hw_state(struct dc *dc) dcn10_log_hubbub_state(dc); - DTN_INFO("HUBP:\t format \t addr_hi \t width \t height \t " - "rotation \t mirror \t sw_mode \t " - "dcc_en \t blank_en \t ttu_dis \t underflow \t " - "min_ttu_vblank \t qos_low_wm \t qos_high_wm \n"); + dcn10_log_hubp_states(dc); + DTN_INFO("DPP: IGAM format IGAM mode DGAM mode RGAM mode" + " GAMUT mode C11 C12 C13 C14 C21 C22 C23 C24 " + "C31 C32 C33 C34\n"); for (i = 0; i < pool->pipe_count; i++) { - struct hubp *hubp = pool->hubps[i]; - struct dcn_hubp_state s; - - hubp1_read_state(TO_DCN10_HUBP(hubp), &s); - - DTN_INFO("[%d]:\t %xh \t %xh \t %d \t %d \t " - "%xh \t %xh \t %xh \t " - "%d \t %d \t %d \t %xh \t", - hubp->inst, - s.pixel_format, - s.inuse_addr_hi, - s.viewport_width, - s.viewport_height, - s.rotation_angle, - s.h_mirror_en, - s.sw_mode, - s.dcc_en, - s.blank_en, - s.ttu_disable, - s.underflow_status); - DTN_INFO_MICRO_SEC(s.min_ttu_vblank); - DTN_INFO_MICRO_SEC(s.qos_level_low_wm); - DTN_INFO_MICRO_SEC(s.qos_level_high_wm); + struct dpp *dpp = pool->dpps[i]; + struct dcn_dpp_state s; + + dpp->funcs->dpp_read_state(dpp, &s); + + DTN_INFO("[%2d]: %11xh %-11s %-11s %-11s" + "%8x %08xh %08xh %08xh %08xh %08xh %08xh", + dpp->inst, + s.igam_input_format, + (s.igam_lut_mode == 0) ? "BypassFixed" : + ((s.igam_lut_mode == 1) ? "BypassFloat" : + ((s.igam_lut_mode == 2) ? "RAM" : + ((s.igam_lut_mode == 3) ? "RAM" : + "Unknown"))), + (s.dgam_lut_mode == 0) ? "Bypass" : + ((s.dgam_lut_mode == 1) ? "sRGB" : + ((s.dgam_lut_mode == 2) ? "Ycc" : + ((s.dgam_lut_mode == 3) ? "RAM" : + ((s.dgam_lut_mode == 4) ? "RAM" : + "Unknown")))), + (s.rgam_lut_mode == 0) ? "Bypass" : + ((s.rgam_lut_mode == 1) ? "sRGB" : + ((s.rgam_lut_mode == 2) ? "Ycc" : + ((s.rgam_lut_mode == 3) ? "RAM" : + ((s.rgam_lut_mode == 4) ? "RAM" : + "Unknown")))), + s.gamut_remap_mode, + s.gamut_remap_c11_c12, + s.gamut_remap_c13_c14, + s.gamut_remap_c21_c22, + s.gamut_remap_c23_c24, + s.gamut_remap_c31_c32, + s.gamut_remap_c33_c34); DTN_INFO("\n"); } DTN_INFO("\n"); - DTN_INFO("OTG:\t v_bs \t v_be \t v_ss \t v_se \t vpol \t vmax \t vmin \t " - "h_bs \t h_be \t h_ss \t h_se \t hpol \t htot \t vtot \t underflow\n"); + DTN_INFO("MPCC: OPP DPP MPCCBOT MODE ALPHA_MODE PREMULT OVERLAP_ONLY IDLE\n"); + for (i = 0; i < pool->pipe_count; i++) { + struct mpcc_state s = {0}; + + pool->mpc->funcs->read_mpcc_state(pool->mpc, i, &s); + if (s.opp_id != 0xf) + DTN_INFO("[%2d]: %2xh %2xh %6xh %4d %10d %7d %12d %4d\n", + i, s.opp_id, s.dpp_id, s.bot_mpcc_id, + s.mode, s.alpha_mode, s.pre_multiplied_alpha, s.overlap_only, + s.idle); + } + DTN_INFO("\n"); + + DTN_INFO("OTG: v_bs v_be v_ss v_se vpol vmax vmin vmax_sel vmin_sel" + " h_bs h_be h_ss h_se hpol htot vtot underflow\n"); for (i = 0; i < pool->timing_generator_count; i++) { struct timing_generator *tg = pool->timing_generators[i]; @@ -167,9 +306,8 @@ void dcn10_log_hw_state(struct dc *dc) if ((s.otg_enabled & 1) == 0) continue; - DTN_INFO("[%d]:\t %d \t %d \t %d \t %d \t " - "%d \t %d \t %d \t %d \t %d \t %d \t " - "%d \t %d \t %d \t %d \t %d \t ", + DTN_INFO("[%d]: %5d %5d %5d %5d %5d %5d %5d %9d %9d %5d %5d %5d" + " %5d %5d %5d %5d %9d\n", tg->inst, s.v_blank_start, s.v_blank_end, @@ -178,6 +316,8 @@ void dcn10_log_hw_state(struct dc *dc) s.v_sync_a_pol, s.v_total_max, s.v_total_min, + s.v_total_max_sel, + s.v_total_min_sel, s.h_blank_start, s.h_blank_end, s.h_sync_a_start, @@ -186,10 +326,25 @@ void dcn10_log_hw_state(struct dc *dc) s.h_total, s.v_total, s.underflow_occurred_status); - DTN_INFO("\n"); + + // Clear underflow for debug purposes + // We want to keep underflow sticky bit on for the longevity tests outside of test environment. + // This function is called only from Windows or Diags test environment, hence it's safe to clear + // it from here without affecting the original intent. + tg->funcs->clear_optc_underflow(tg); } DTN_INFO("\n"); + DTN_INFO("\nCALCULATED Clocks: dcfclk_khz:%d dcfclk_deep_sleep_khz:%d dispclk_khz:%d\n" + "dppclk_khz:%d max_supported_dppclk_khz:%d fclk_khz:%d socclk_khz:%d\n\n", + dc->current_state->bw.dcn.calc_clk.dcfclk_khz, + dc->current_state->bw.dcn.calc_clk.dcfclk_deep_sleep_khz, + dc->current_state->bw.dcn.calc_clk.dispclk_khz, + dc->current_state->bw.dcn.calc_clk.dppclk_khz, + dc->current_state->bw.dcn.calc_clk.max_supported_dppclk_khz, + dc->current_state->bw.dcn.calc_clk.fclk_khz, + dc->current_state->bw.dcn.calc_clk.socclk_khz); + log_mpc_crc(dc); DTN_INFO_END(); @@ -354,7 +509,7 @@ static void power_on_plane( struct dce_hwseq *hws, int plane_id) { - struct dc_context *ctx = hws->ctx; + DC_LOGGER_INIT(hws->ctx->logger); if (REG(DC_IP_REQUEST_CNTL)) { REG_SET(DC_IP_REQUEST_CNTL, 0, IP_REQUEST_EN, 1); @@ -461,7 +616,7 @@ static void false_optc_underflow_wa( tg->funcs->clear_optc_underflow(tg); } -static enum dc_status dcn10_prog_pixclk_crtc_otg( +static enum dc_status dcn10_enable_stream_timing( struct pipe_ctx *pipe_ctx, struct dc_state *context, struct dc *dc) @@ -553,7 +708,7 @@ static void reset_back_end_for_pipe( struct dc_state *context) { int i; - struct dc_context *ctx = dc->ctx; + DC_LOGGER_INIT(dc->ctx->logger); if (pipe_ctx->stream_res.stream_enc == NULL) { pipe_ctx->stream = NULL; return; @@ -603,6 +758,90 @@ static void reset_back_end_for_pipe( pipe_ctx->pipe_idx, pipe_ctx->stream_res.tg->inst); } +static bool dcn10_hw_wa_force_recovery(struct dc *dc) +{ + struct hubp *hubp ; + unsigned int i; + bool need_recover = true; + + if (!dc->debug.recovery_enabled) + return false; + + for (i = 0; i < dc->res_pool->pipe_count; i++) { + struct pipe_ctx *pipe_ctx = + &dc->current_state->res_ctx.pipe_ctx[i]; + if (pipe_ctx != NULL) { + hubp = pipe_ctx->plane_res.hubp; + if (hubp != NULL) { + if (hubp->funcs->hubp_get_underflow_status(hubp) != 0) { + /* one pipe underflow, we will reset all the pipes*/ + need_recover = true; + } + } + } + } + if (!need_recover) + return false; + /* + DCHUBP_CNTL:HUBP_BLANK_EN=1 + DCHUBBUB_SOFT_RESET:DCHUBBUB_GLOBAL_SOFT_RESET=1 + DCHUBP_CNTL:HUBP_DISABLE=1 + DCHUBP_CNTL:HUBP_DISABLE=0 + DCHUBBUB_SOFT_RESET:DCHUBBUB_GLOBAL_SOFT_RESET=0 + DCSURF_PRIMARY_SURFACE_ADDRESS + DCHUBP_CNTL:HUBP_BLANK_EN=0 + */ + + for (i = 0; i < dc->res_pool->pipe_count; i++) { + struct pipe_ctx *pipe_ctx = + &dc->current_state->res_ctx.pipe_ctx[i]; + if (pipe_ctx != NULL) { + hubp = pipe_ctx->plane_res.hubp; + /*DCHUBP_CNTL:HUBP_BLANK_EN=1*/ + if (hubp != NULL) + hubp->funcs->set_hubp_blank_en(hubp, true); + } + } + /*DCHUBBUB_SOFT_RESET:DCHUBBUB_GLOBAL_SOFT_RESET=1*/ + hubbub1_soft_reset(dc->res_pool->hubbub, true); + + for (i = 0; i < dc->res_pool->pipe_count; i++) { + struct pipe_ctx *pipe_ctx = + &dc->current_state->res_ctx.pipe_ctx[i]; + if (pipe_ctx != NULL) { + hubp = pipe_ctx->plane_res.hubp; + /*DCHUBP_CNTL:HUBP_DISABLE=1*/ + if (hubp != NULL) + hubp->funcs->hubp_disable_control(hubp, true); + } + } + for (i = 0; i < dc->res_pool->pipe_count; i++) { + struct pipe_ctx *pipe_ctx = + &dc->current_state->res_ctx.pipe_ctx[i]; + if (pipe_ctx != NULL) { + hubp = pipe_ctx->plane_res.hubp; + /*DCHUBP_CNTL:HUBP_DISABLE=0*/ + if (hubp != NULL) + hubp->funcs->hubp_disable_control(hubp, true); + } + } + /*DCHUBBUB_SOFT_RESET:DCHUBBUB_GLOBAL_SOFT_RESET=0*/ + hubbub1_soft_reset(dc->res_pool->hubbub, false); + for (i = 0; i < dc->res_pool->pipe_count; i++) { + struct pipe_ctx *pipe_ctx = + &dc->current_state->res_ctx.pipe_ctx[i]; + if (pipe_ctx != NULL) { + hubp = pipe_ctx->plane_res.hubp; + /*DCHUBP_CNTL:HUBP_BLANK_EN=0*/ + if (hubp != NULL) + hubp->funcs->set_hubp_blank_en(hubp, true); + } + } + return true; + +} + + static void dcn10_verify_allow_pstate_change_high(struct dc *dc) { static bool should_log_hw_state; /* prevent hw state log by default */ @@ -611,13 +850,17 @@ static void dcn10_verify_allow_pstate_change_high(struct dc *dc) if (should_log_hw_state) { dcn10_log_hw_state(dc); } - BREAK_TO_DEBUGGER(); + if (dcn10_hw_wa_force_recovery(dc)) { + /*check again*/ + if (!hubbub1_verify_allow_pstate_change_high(dc->res_pool->hubbub)) + BREAK_TO_DEBUGGER(); + } } } /* trigger HW to start disconnect plane from stream on the next vsync */ -static void plane_atomic_disconnect(struct dc *dc, struct pipe_ctx *pipe_ctx) +void hwss1_plane_atomic_disconnect(struct dc *dc, struct pipe_ctx *pipe_ctx) { struct hubp *hubp = pipe_ctx->plane_res.hubp; int dpp_id = pipe_ctx->plane_res.dpp->inst; @@ -649,7 +892,7 @@ static void plane_atomic_power_down(struct dc *dc, struct pipe_ctx *pipe_ctx) { struct dce_hwseq *hws = dc->hwseq; struct dpp *dpp = pipe_ctx->plane_res.dpp; - struct dc_context *ctx = dc->ctx; + DC_LOGGER_INIT(dc->ctx->logger); if (REG(DC_IP_REQUEST_CNTL)) { REG_SET(DC_IP_REQUEST_CNTL, 0, @@ -699,7 +942,7 @@ static void plane_atomic_disable(struct dc *dc, struct pipe_ctx *pipe_ctx) static void dcn10_disable_plane(struct dc *dc, struct pipe_ctx *pipe_ctx) { - struct dc_context *ctx = dc->ctx; + DC_LOGGER_INIT(dc->ctx->logger); if (!pipe_ctx->plane_res.hubp || pipe_ctx->plane_res.hubp->power_gated) return; @@ -800,7 +1043,7 @@ static void dcn10_init_hw(struct dc *dc) dc->res_pool->opps[i]->mpcc_disconnect_pending[pipe_ctx->plane_res.mpcc_inst] = true; pipe_ctx->stream_res.opp = dc->res_pool->opps[i]; - plane_atomic_disconnect(dc, pipe_ctx); + hwss1_plane_atomic_disconnect(dc, pipe_ctx); } for (i = 0; i < dc->res_pool->pipe_count; i++) { @@ -945,9 +1188,8 @@ static bool dcn10_set_input_transfer_func(struct pipe_ctx *pipe_ctx, tf = plane_state->in_transfer_func; if (plane_state->gamma_correction && - plane_state->gamma_correction->is_identity) - dpp_base->funcs->dpp_set_degamma(dpp_base, IPP_DEGAMMA_MODE_BYPASS); - else if (plane_state->gamma_correction && dce_use_lut(plane_state->format)) + !plane_state->gamma_correction->is_identity + && dce_use_lut(plane_state->format)) dpp_base->funcs->dpp_program_input_lut(dpp_base, plane_state->gamma_correction); if (tf == NULL) @@ -1433,7 +1675,7 @@ static void program_csc_matrix(struct pipe_ctx *pipe_ctx, } } -static void program_output_csc(struct dc *dc, +static void dcn10_program_output_csc(struct dc *dc, struct pipe_ctx *pipe_ctx, enum dc_color_space colorspace, uint16_t *matrix, @@ -1542,22 +1784,22 @@ static uint16_t fixed_point_to_int_frac( uint16_t result; - uint16_t d = (uint16_t)dal_fixed31_32_floor( - dal_fixed31_32_abs( + uint16_t d = (uint16_t)dc_fixpt_floor( + dc_fixpt_abs( arg)); if (d <= (uint16_t)(1 << integer_bits) - (1 / (uint16_t)divisor)) - numerator = (uint16_t)dal_fixed31_32_floor( - dal_fixed31_32_mul_int( + numerator = (uint16_t)dc_fixpt_floor( + dc_fixpt_mul_int( arg, divisor)); else { - numerator = dal_fixed31_32_floor( - dal_fixed31_32_sub( - dal_fixed31_32_from_int( + numerator = dc_fixpt_floor( + dc_fixpt_sub( + dc_fixpt_from_int( 1LL << integer_bits), - dal_fixed31_32_recip( - dal_fixed31_32_from_int( + dc_fixpt_recip( + dc_fixpt_from_int( divisor)))); } @@ -1567,8 +1809,8 @@ static uint16_t fixed_point_to_int_frac( result = (uint16_t)( (1 << (integer_bits + fractional_bits + 1)) + numerator); - if ((result != 0) && dal_fixed31_32_lt( - arg, dal_fixed31_32_zero)) + if ((result != 0) && dc_fixpt_lt( + arg, dc_fixpt_zero)) result |= 1 << (integer_bits + fractional_bits); return result; @@ -1582,8 +1824,8 @@ void build_prescale_params(struct dc_bias_and_scale *bias_and_scale, && plane_state->input_csc_color_matrix.enable_adjustment && plane_state->coeff_reduction_factor.value != 0) { bias_and_scale->scale_blue = fixed_point_to_int_frac( - dal_fixed31_32_mul(plane_state->coeff_reduction_factor, - dal_fixed31_32_from_fraction(256, 255)), + dc_fixpt_mul(plane_state->coeff_reduction_factor, + dc_fixpt_from_fraction(256, 255)), 2, 13); bias_and_scale->scale_red = bias_and_scale->scale_blue; @@ -1623,6 +1865,8 @@ static void update_mpcc(struct dc *dc, struct pipe_ctx *pipe_ctx) struct mpc *mpc = dc->res_pool->mpc; struct mpc_tree *mpc_tree_params = &(pipe_ctx->stream_res.opp->mpc_tree_params); + + /* TODO: proper fix once fpga works */ if (dc->debug.surface_visual_confirm) @@ -1649,6 +1893,7 @@ static void update_mpcc(struct dc *dc, struct pipe_ctx *pipe_ctx) pipe_ctx->stream->output_color_space) && per_pixel_alpha; + /* * TODO: remove hack * Note: currently there is a bug in init_hw such that @@ -1659,6 +1904,12 @@ static void update_mpcc(struct dc *dc, struct pipe_ctx *pipe_ctx) */ mpcc_id = hubp->inst; + /* If there is no full update, don't need to touch MPC tree*/ + if (!pipe_ctx->plane_state->update_flags.bits.full_update) { + mpc->funcs->update_blending(mpc, &blnd_cfg, mpcc_id); + return; + } + /* check if this MPCC is already being used */ new_mpcc = mpc->funcs->get_mpcc_for_dpp(mpc_tree_params, mpcc_id); /* remove MPCC if being used */ @@ -1777,7 +2028,7 @@ static void update_dchubp_dpp( /*gamut remap*/ program_gamut_remap(pipe_ctx); - program_output_csc(dc, + dc->hwss.program_output_csc(dc, pipe_ctx, pipe_ctx->stream->output_color_space, pipe_ctx->stream->csc_color_matrix.matrix, @@ -1810,9 +2061,9 @@ static void update_dchubp_dpp( hubp->funcs->set_blank(hubp, false); } -static void dcn10_otg_blank( +static void dcn10_blank_pixel_data( struct dc *dc, - struct stream_resource stream_res, + struct stream_resource *stream_res, struct dc_stream_state *stream, bool blank) { @@ -1823,27 +2074,27 @@ static void dcn10_otg_blank( color_space = stream->output_color_space; color_space_to_black_color(dc, color_space, &black_color); - if (stream_res.tg->funcs->set_blank_color) - stream_res.tg->funcs->set_blank_color( - stream_res.tg, + if (stream_res->tg->funcs->set_blank_color) + stream_res->tg->funcs->set_blank_color( + stream_res->tg, &black_color); if (!blank) { - if (stream_res.tg->funcs->set_blank) - stream_res.tg->funcs->set_blank(stream_res.tg, blank); - if (stream_res.abm) - stream_res.abm->funcs->set_abm_level(stream_res.abm, stream->abm_level); + if (stream_res->tg->funcs->set_blank) + stream_res->tg->funcs->set_blank(stream_res->tg, blank); + if (stream_res->abm) + stream_res->abm->funcs->set_abm_level(stream_res->abm, stream->abm_level); } else if (blank) { - if (stream_res.abm) - stream_res.abm->funcs->set_abm_immediate_disable(stream_res.abm); - if (stream_res.tg->funcs->set_blank) - stream_res.tg->funcs->set_blank(stream_res.tg, blank); + if (stream_res->abm) + stream_res->abm->funcs->set_abm_immediate_disable(stream_res->abm); + if (stream_res->tg->funcs->set_blank) + stream_res->tg->funcs->set_blank(stream_res->tg, blank); } } static void set_hdr_multiplier(struct pipe_ctx *pipe_ctx) { - struct fixed31_32 multiplier = dal_fixed31_32_from_fraction( + struct fixed31_32 multiplier = dc_fixpt_from_fraction( pipe_ctx->plane_state->sdr_white_level, 80); uint32_t hw_mult = 0x1f000; // 1.0 default multiplier struct custom_float_format fmt; @@ -1876,7 +2127,7 @@ static void program_all_pipe_in_tree( pipe_ctx->stream_res.tg->funcs->program_global_sync( pipe_ctx->stream_res.tg); - dcn10_otg_blank(dc, pipe_ctx->stream_res, + dc->hwss.blank_pixel_data(dc, &pipe_ctx->stream_res, pipe_ctx->stream, blank); } @@ -1983,9 +2234,9 @@ static void dcn10_apply_ctx_for_surface( bool removed_pipe[4] = { false }; unsigned int ref_clk_mhz = dc->res_pool->ref_clock_inKhz/1000; bool program_water_mark = false; - struct dc_context *ctx = dc->ctx; struct pipe_ctx *top_pipe_to_program = find_top_pipe_for_stream(dc, context, stream); + DC_LOGGER_INIT(dc->ctx->logger); if (!top_pipe_to_program) return; @@ -1996,7 +2247,7 @@ static void dcn10_apply_ctx_for_surface( if (num_planes == 0) { /* OTG blank before remove all front end */ - dcn10_otg_blank(dc, top_pipe_to_program->stream_res, top_pipe_to_program->stream, true); + dc->hwss.blank_pixel_data(dc, &top_pipe_to_program->stream_res, top_pipe_to_program->stream, true); } /* Disconnect unused mpcc */ @@ -2027,7 +2278,7 @@ static void dcn10_apply_ctx_for_surface( old_pipe_ctx->plane_state && old_pipe_ctx->stream_res.tg == tg) { - plane_atomic_disconnect(dc, old_pipe_ctx); + hwss1_plane_atomic_disconnect(dc, old_pipe_ctx); removed_pipe[i] = true; DC_LOG_DC( @@ -2335,15 +2586,6 @@ static void set_static_screen_control(struct pipe_ctx **pipe_ctx, set_static_screen_control(pipe_ctx[i]->stream_res.tg, value); } -static void set_plane_config( - const struct dc *dc, - struct pipe_ctx *pipe_ctx, - struct resource_context *res_ctx) -{ - /* TODO */ - program_gamut_remap(pipe_ctx); -} - static void dcn10_config_stereo_parameters( struct dc_stream_state *stream, struct crtc_stereo_flags *flags) { @@ -2521,12 +2763,12 @@ static const struct hw_sequencer_funcs dcn10_funcs = { .init_hw = dcn10_init_hw, .apply_ctx_to_hw = dce110_apply_ctx_to_hw, .apply_ctx_for_surface = dcn10_apply_ctx_for_surface, - .set_plane_config = set_plane_config, .update_plane_addr = dcn10_update_plane_addr, .update_dchub = dcn10_update_dchub, .update_pending_status = dcn10_update_pending_status, .set_input_transfer_func = dcn10_set_input_transfer_func, .set_output_transfer_func = dcn10_set_output_transfer_func, + .program_output_csc = dcn10_program_output_csc, .power_down = dce110_power_down, .enable_accelerated_mode = dce110_enable_accelerated_mode, .enable_timing_synchronization = dcn10_enable_timing_synchronization, @@ -2538,10 +2780,11 @@ static const struct hw_sequencer_funcs dcn10_funcs = { .blank_stream = dce110_blank_stream, .enable_display_power_gating = dcn10_dummy_display_power_gating, .disable_plane = dcn10_disable_plane, + .blank_pixel_data = dcn10_blank_pixel_data, .pipe_control_lock = dcn10_pipe_control_lock, .set_bandwidth = dcn10_set_bandwidth, .reset_hw_ctx_wrap = reset_hw_ctx_wrap, - .prog_pixclk_crtc_otg = dcn10_prog_pixclk_crtc_otg, + .enable_stream_timing = dcn10_enable_stream_timing, .set_drr = set_drr, .get_position = get_position, .set_static_screen_control = set_static_screen_control, diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.h b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.h index 6c526b5095d9..44f734b73f9e 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.h +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.h @@ -37,4 +37,6 @@ extern void fill_display_configs( bool is_rgb_cspace(enum dc_color_space output_color_space); +void hwss1_plane_atomic_disconnect(struct dc *dc, struct pipe_ctx *pipe_ctx); + #endif /* __DC_HWSS_DCN10_H__ */ diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_link_encoder.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_link_encoder.c new file mode 100644 index 000000000000..21fa40ac0786 --- /dev/null +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_link_encoder.c @@ -0,0 +1,1362 @@ +/* + * Copyright 2012-15 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * Authors: AMD + * + */ + +#include "reg_helper.h" + +#include "core_types.h" +#include "link_encoder.h" +#include "dcn10_link_encoder.h" +#include "stream_encoder.h" +#include "i2caux_interface.h" +#include "dc_bios_types.h" + +#include "gpio_service_interface.h" + +#define CTX \ + enc10->base.ctx +#define DC_LOGGER \ + enc10->base.ctx->logger + +#define REG(reg)\ + (enc10->link_regs->reg) + +#undef FN +#define FN(reg_name, field_name) \ + enc10->link_shift->field_name, enc10->link_mask->field_name + + +/* + * @brief + * Trigger Source Select + * ASIC-dependent, actual values for register programming + */ +#define DCN10_DIG_FE_SOURCE_SELECT_INVALID 0x0 +#define DCN10_DIG_FE_SOURCE_SELECT_DIGA 0x1 +#define DCN10_DIG_FE_SOURCE_SELECT_DIGB 0x2 +#define DCN10_DIG_FE_SOURCE_SELECT_DIGC 0x4 +#define DCN10_DIG_FE_SOURCE_SELECT_DIGD 0x08 +#define DCN10_DIG_FE_SOURCE_SELECT_DIGE 0x10 +#define DCN10_DIG_FE_SOURCE_SELECT_DIGF 0x20 +#define DCN10_DIG_FE_SOURCE_SELECT_DIGG 0x40 + +enum { + DP_MST_UPDATE_MAX_RETRY = 50 +}; + + + +static void aux_initialize(struct dcn10_link_encoder *enc10); + + +static const struct link_encoder_funcs dcn10_lnk_enc_funcs = { + .validate_output_with_stream = + dcn10_link_encoder_validate_output_with_stream, + .hw_init = dcn10_link_encoder_hw_init, + .setup = dcn10_link_encoder_setup, + .enable_tmds_output = dcn10_link_encoder_enable_tmds_output, + .enable_dp_output = dcn10_link_encoder_enable_dp_output, + .enable_dp_mst_output = dcn10_link_encoder_enable_dp_mst_output, + .disable_output = dcn10_link_encoder_disable_output, + .dp_set_lane_settings = dcn10_link_encoder_dp_set_lane_settings, + .dp_set_phy_pattern = dcn10_link_encoder_dp_set_phy_pattern, + .update_mst_stream_allocation_table = + dcn10_link_encoder_update_mst_stream_allocation_table, + .psr_program_dp_dphy_fast_training = + dcn10_psr_program_dp_dphy_fast_training, + .psr_program_secondary_packet = dcn10_psr_program_secondary_packet, + .connect_dig_be_to_fe = dcn10_link_encoder_connect_dig_be_to_fe, + .enable_hpd = dcn10_link_encoder_enable_hpd, + .disable_hpd = dcn10_link_encoder_disable_hpd, + .is_dig_enabled = dcn10_is_dig_enabled, + .destroy = dcn10_link_encoder_destroy +}; + +static enum bp_result link_transmitter_control( + struct dcn10_link_encoder *enc10, + struct bp_transmitter_control *cntl) +{ + enum bp_result result; + struct dc_bios *bp = enc10->base.ctx->dc_bios; + + result = bp->funcs->transmitter_control(bp, cntl); + + return result; +} + +static void enable_phy_bypass_mode( + struct dcn10_link_encoder *enc10, + bool enable) +{ + /* This register resides in DP back end block; + * transmitter is used for the offset + */ + REG_UPDATE(DP_DPHY_CNTL, DPHY_BYPASS, enable); + +} + +static void disable_prbs_symbols( + struct dcn10_link_encoder *enc10, + bool disable) +{ + /* This register resides in DP back end block; + * transmitter is used for the offset + */ + REG_UPDATE_4(DP_DPHY_CNTL, + DPHY_ATEST_SEL_LANE0, disable, + DPHY_ATEST_SEL_LANE1, disable, + DPHY_ATEST_SEL_LANE2, disable, + DPHY_ATEST_SEL_LANE3, disable); +} + +static void disable_prbs_mode( + struct dcn10_link_encoder *enc10) +{ + REG_UPDATE(DP_DPHY_PRBS_CNTL, DPHY_PRBS_EN, 0); +} + +static void program_pattern_symbols( + struct dcn10_link_encoder *enc10, + uint16_t pattern_symbols[8]) +{ + /* This register resides in DP back end block; + * transmitter is used for the offset + */ + REG_SET_3(DP_DPHY_SYM0, 0, + DPHY_SYM1, pattern_symbols[0], + DPHY_SYM2, pattern_symbols[1], + DPHY_SYM3, pattern_symbols[2]); + + /* This register resides in DP back end block; + * transmitter is used for the offset + */ + REG_SET_3(DP_DPHY_SYM1, 0, + DPHY_SYM4, pattern_symbols[3], + DPHY_SYM5, pattern_symbols[4], + DPHY_SYM6, pattern_symbols[5]); + + /* This register resides in DP back end block; + * transmitter is used for the offset + */ + REG_SET_2(DP_DPHY_SYM2, 0, + DPHY_SYM7, pattern_symbols[6], + DPHY_SYM8, pattern_symbols[7]); +} + +static void set_dp_phy_pattern_d102( + struct dcn10_link_encoder *enc10) +{ + /* Disable PHY Bypass mode to setup the test pattern */ + enable_phy_bypass_mode(enc10, false); + + /* For 10-bit PRBS or debug symbols + * please use the following sequence: + * + * Enable debug symbols on the lanes + */ + disable_prbs_symbols(enc10, true); + + /* Disable PRBS mode */ + disable_prbs_mode(enc10); + + /* Program debug symbols to be output */ + { + uint16_t pattern_symbols[8] = { + 0x2AA, 0x2AA, 0x2AA, 0x2AA, + 0x2AA, 0x2AA, 0x2AA, 0x2AA + }; + + program_pattern_symbols(enc10, pattern_symbols); + } + + /* Enable phy bypass mode to enable the test pattern */ + + enable_phy_bypass_mode(enc10, true); +} + +static void set_link_training_complete( + struct dcn10_link_encoder *enc10, + bool complete) +{ + /* This register resides in DP back end block; + * transmitter is used for the offset + */ + REG_UPDATE(DP_LINK_CNTL, DP_LINK_TRAINING_COMPLETE, complete); + +} + +void dcn10_link_encoder_set_dp_phy_pattern_training_pattern( + struct link_encoder *enc, + uint32_t index) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + /* Write Training Pattern */ + + REG_WRITE(DP_DPHY_TRAINING_PATTERN_SEL, index); + + /* Set HW Register Training Complete to false */ + + set_link_training_complete(enc10, false); + + /* Disable PHY Bypass mode to output Training Pattern */ + + enable_phy_bypass_mode(enc10, false); + + /* Disable PRBS mode */ + disable_prbs_mode(enc10); +} + +static void setup_panel_mode( + struct dcn10_link_encoder *enc10, + enum dp_panel_mode panel_mode) +{ + uint32_t value; + + ASSERT(REG(DP_DPHY_INTERNAL_CTRL)); + value = REG_READ(DP_DPHY_INTERNAL_CTRL); + + switch (panel_mode) { + case DP_PANEL_MODE_EDP: + value = 0x1; + break; + case DP_PANEL_MODE_SPECIAL: + value = 0x11; + break; + default: + value = 0x0; + break; + } + + REG_WRITE(DP_DPHY_INTERNAL_CTRL, value); +} + +static void set_dp_phy_pattern_symbol_error( + struct dcn10_link_encoder *enc10) +{ + /* Disable PHY Bypass mode to setup the test pattern */ + enable_phy_bypass_mode(enc10, false); + + /* program correct panel mode*/ + setup_panel_mode(enc10, DP_PANEL_MODE_DEFAULT); + + /* A PRBS23 pattern is used for most DP electrical measurements. */ + + /* Enable PRBS symbols on the lanes */ + disable_prbs_symbols(enc10, false); + + /* For PRBS23 Set bit DPHY_PRBS_SEL=1 and Set bit DPHY_PRBS_EN=1 */ + REG_UPDATE_2(DP_DPHY_PRBS_CNTL, + DPHY_PRBS_SEL, 1, + DPHY_PRBS_EN, 1); + + /* Enable phy bypass mode to enable the test pattern */ + enable_phy_bypass_mode(enc10, true); +} + +static void set_dp_phy_pattern_prbs7( + struct dcn10_link_encoder *enc10) +{ + /* Disable PHY Bypass mode to setup the test pattern */ + enable_phy_bypass_mode(enc10, false); + + /* A PRBS7 pattern is used for most DP electrical measurements. */ + + /* Enable PRBS symbols on the lanes */ + disable_prbs_symbols(enc10, false); + + /* For PRBS7 Set bit DPHY_PRBS_SEL=0 and Set bit DPHY_PRBS_EN=1 */ + REG_UPDATE_2(DP_DPHY_PRBS_CNTL, + DPHY_PRBS_SEL, 0, + DPHY_PRBS_EN, 1); + + /* Enable phy bypass mode to enable the test pattern */ + enable_phy_bypass_mode(enc10, true); +} + +static void set_dp_phy_pattern_80bit_custom( + struct dcn10_link_encoder *enc10, + const uint8_t *pattern) +{ + /* Disable PHY Bypass mode to setup the test pattern */ + enable_phy_bypass_mode(enc10, false); + + /* Enable debug symbols on the lanes */ + + disable_prbs_symbols(enc10, true); + + /* Enable PHY bypass mode to enable the test pattern */ + /* TODO is it really needed ? */ + + enable_phy_bypass_mode(enc10, true); + + /* Program 80 bit custom pattern */ + { + uint16_t pattern_symbols[8]; + + pattern_symbols[0] = + ((pattern[1] & 0x03) << 8) | pattern[0]; + pattern_symbols[1] = + ((pattern[2] & 0x0f) << 6) | ((pattern[1] >> 2) & 0x3f); + pattern_symbols[2] = + ((pattern[3] & 0x3f) << 4) | ((pattern[2] >> 4) & 0x0f); + pattern_symbols[3] = + (pattern[4] << 2) | ((pattern[3] >> 6) & 0x03); + pattern_symbols[4] = + ((pattern[6] & 0x03) << 8) | pattern[5]; + pattern_symbols[5] = + ((pattern[7] & 0x0f) << 6) | ((pattern[6] >> 2) & 0x3f); + pattern_symbols[6] = + ((pattern[8] & 0x3f) << 4) | ((pattern[7] >> 4) & 0x0f); + pattern_symbols[7] = + (pattern[9] << 2) | ((pattern[8] >> 6) & 0x03); + + program_pattern_symbols(enc10, pattern_symbols); + } + + /* Enable phy bypass mode to enable the test pattern */ + + enable_phy_bypass_mode(enc10, true); +} + +static void set_dp_phy_pattern_hbr2_compliance_cp2520_2( + struct dcn10_link_encoder *enc10, + unsigned int cp2520_pattern) +{ + + /* previously there is a register DP_HBR2_EYE_PATTERN + * that is enabled to get the pattern. + * But it does not work with the latest spec change, + * so we are programming the following registers manually. + * + * The following settings have been confirmed + * by Nick Chorney and Sandra Liu + */ + + /* Disable PHY Bypass mode to setup the test pattern */ + + enable_phy_bypass_mode(enc10, false); + + /* Setup DIG encoder in DP SST mode */ + enc10->base.funcs->setup(&enc10->base, SIGNAL_TYPE_DISPLAY_PORT); + + /* ensure normal panel mode. */ + setup_panel_mode(enc10, DP_PANEL_MODE_DEFAULT); + + /* no vbid after BS (SR) + * DP_LINK_FRAMING_CNTL changed history Sandra Liu + * 11000260 / 11000104 / 110000FC + */ + REG_UPDATE_3(DP_LINK_FRAMING_CNTL, + DP_IDLE_BS_INTERVAL, 0xFC, + DP_VBID_DISABLE, 1, + DP_VID_ENHANCED_FRAME_MODE, 1); + + /* swap every BS with SR */ + REG_UPDATE(DP_DPHY_SCRAM_CNTL, DPHY_SCRAMBLER_BS_COUNT, 0); + + /* select cp2520 patterns */ + if (REG(DP_DPHY_HBR2_PATTERN_CONTROL)) + REG_UPDATE(DP_DPHY_HBR2_PATTERN_CONTROL, + DP_DPHY_HBR2_PATTERN_CONTROL, cp2520_pattern); + else + /* pre-DCE11 can only generate CP2520 pattern 2 */ + ASSERT(cp2520_pattern == 2); + + /* set link training complete */ + set_link_training_complete(enc10, true); + + /* disable video stream */ + REG_UPDATE(DP_VID_STREAM_CNTL, DP_VID_STREAM_ENABLE, 0); + + /* Disable PHY Bypass mode to setup the test pattern */ + enable_phy_bypass_mode(enc10, false); +} + +static void set_dp_phy_pattern_passthrough_mode( + struct dcn10_link_encoder *enc10, + enum dp_panel_mode panel_mode) +{ + /* program correct panel mode */ + setup_panel_mode(enc10, panel_mode); + + /* restore LINK_FRAMING_CNTL and DPHY_SCRAMBLER_BS_COUNT + * in case we were doing HBR2 compliance pattern before + */ + REG_UPDATE_3(DP_LINK_FRAMING_CNTL, + DP_IDLE_BS_INTERVAL, 0x2000, + DP_VBID_DISABLE, 0, + DP_VID_ENHANCED_FRAME_MODE, 1); + + REG_UPDATE(DP_DPHY_SCRAM_CNTL, DPHY_SCRAMBLER_BS_COUNT, 0x1FF); + + /* set link training complete */ + set_link_training_complete(enc10, true); + + /* Disable PHY Bypass mode to setup the test pattern */ + enable_phy_bypass_mode(enc10, false); + + /* Disable PRBS mode */ + disable_prbs_mode(enc10); +} + +/* return value is bit-vector */ +static uint8_t get_frontend_source( + enum engine_id engine) +{ + switch (engine) { + case ENGINE_ID_DIGA: + return DCN10_DIG_FE_SOURCE_SELECT_DIGA; + case ENGINE_ID_DIGB: + return DCN10_DIG_FE_SOURCE_SELECT_DIGB; + case ENGINE_ID_DIGC: + return DCN10_DIG_FE_SOURCE_SELECT_DIGC; + case ENGINE_ID_DIGD: + return DCN10_DIG_FE_SOURCE_SELECT_DIGD; + case ENGINE_ID_DIGE: + return DCN10_DIG_FE_SOURCE_SELECT_DIGE; + case ENGINE_ID_DIGF: + return DCN10_DIG_FE_SOURCE_SELECT_DIGF; + case ENGINE_ID_DIGG: + return DCN10_DIG_FE_SOURCE_SELECT_DIGG; + default: + ASSERT_CRITICAL(false); + return DCN10_DIG_FE_SOURCE_SELECT_INVALID; + } +} + +static void configure_encoder( + struct dcn10_link_encoder *enc10, + const struct dc_link_settings *link_settings) +{ + /* set number of lanes */ + + REG_SET(DP_CONFIG, 0, + DP_UDI_LANES, link_settings->lane_count - LANE_COUNT_ONE); + + /* setup scrambler */ + REG_UPDATE(DP_DPHY_SCRAM_CNTL, DPHY_SCRAMBLER_ADVANCE, 1); +} + +void dcn10_psr_program_dp_dphy_fast_training(struct link_encoder *enc, + bool exit_link_training_required) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + + if (exit_link_training_required) + REG_UPDATE(DP_DPHY_FAST_TRAINING, + DPHY_RX_FAST_TRAINING_CAPABLE, 1); + else { + REG_UPDATE(DP_DPHY_FAST_TRAINING, + DPHY_RX_FAST_TRAINING_CAPABLE, 0); + /*In DCE 11, we are able to pre-program a Force SR register + * to be able to trigger SR symbol after 5 idle patterns + * transmitted. Upon PSR Exit, DMCU can trigger + * DPHY_LOAD_BS_COUNT_START = 1. Upon writing 1 to + * DPHY_LOAD_BS_COUNT_START and the internal counter + * reaches DPHY_LOAD_BS_COUNT, the next BS symbol will be + * replaced by SR symbol once. + */ + + REG_UPDATE(DP_DPHY_BS_SR_SWAP_CNTL, DPHY_LOAD_BS_COUNT, 0x5); + } +} + +void dcn10_psr_program_secondary_packet(struct link_encoder *enc, + unsigned int sdp_transmit_line_num_deadline) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + + REG_UPDATE_2(DP_SEC_CNTL1, + DP_SEC_GSP0_LINE_NUM, sdp_transmit_line_num_deadline, + DP_SEC_GSP0_PRIORITY, 1); +} + +bool dcn10_is_dig_enabled(struct link_encoder *enc) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + uint32_t value; + + REG_GET(DIG_BE_EN_CNTL, DIG_ENABLE, &value); + return value; +} + +static void link_encoder_disable(struct dcn10_link_encoder *enc10) +{ + /* reset training pattern */ + REG_SET(DP_DPHY_TRAINING_PATTERN_SEL, 0, + DPHY_TRAINING_PATTERN_SEL, 0); + + /* reset training complete */ + REG_UPDATE(DP_LINK_CNTL, DP_LINK_TRAINING_COMPLETE, 0); + + /* reset panel mode */ + setup_panel_mode(enc10, DP_PANEL_MODE_DEFAULT); +} + +static void hpd_initialize( + struct dcn10_link_encoder *enc10) +{ + /* Associate HPD with DIG_BE */ + enum hpd_source_id hpd_source = enc10->base.hpd_source; + + REG_UPDATE(DIG_BE_CNTL, DIG_HPD_SELECT, hpd_source); +} + +bool dcn10_link_encoder_validate_dvi_output( + const struct dcn10_link_encoder *enc10, + enum signal_type connector_signal, + enum signal_type signal, + const struct dc_crtc_timing *crtc_timing) +{ + uint32_t max_pixel_clock = TMDS_MAX_PIXEL_CLOCK; + + if (signal == SIGNAL_TYPE_DVI_DUAL_LINK) + max_pixel_clock *= 2; + + /* This handles the case of HDMI downgrade to DVI we don't want to + * we don't want to cap the pixel clock if the DDI is not DVI. + */ + if (connector_signal != SIGNAL_TYPE_DVI_DUAL_LINK && + connector_signal != SIGNAL_TYPE_DVI_SINGLE_LINK) + max_pixel_clock = enc10->base.features.max_hdmi_pixel_clock; + + /* DVI only support RGB pixel encoding */ + if (crtc_timing->pixel_encoding != PIXEL_ENCODING_RGB) + return false; + + /*connect DVI via adpater's HDMI connector*/ + if ((connector_signal == SIGNAL_TYPE_DVI_SINGLE_LINK || + connector_signal == SIGNAL_TYPE_HDMI_TYPE_A) && + signal != SIGNAL_TYPE_HDMI_TYPE_A && + crtc_timing->pix_clk_khz > TMDS_MAX_PIXEL_CLOCK) + return false; + if (crtc_timing->pix_clk_khz < TMDS_MIN_PIXEL_CLOCK) + return false; + + if (crtc_timing->pix_clk_khz > max_pixel_clock) + return false; + + /* DVI supports 6/8bpp single-link and 10/16bpp dual-link */ + switch (crtc_timing->display_color_depth) { + case COLOR_DEPTH_666: + case COLOR_DEPTH_888: + break; + case COLOR_DEPTH_101010: + case COLOR_DEPTH_161616: + if (signal != SIGNAL_TYPE_DVI_DUAL_LINK) + return false; + break; + default: + return false; + } + + return true; +} + +static bool dcn10_link_encoder_validate_hdmi_output( + const struct dcn10_link_encoder *enc10, + const struct dc_crtc_timing *crtc_timing, + int adjusted_pix_clk_khz) +{ + enum dc_color_depth max_deep_color = + enc10->base.features.max_hdmi_deep_color; + + if (max_deep_color < crtc_timing->display_color_depth) + return false; + + if (crtc_timing->display_color_depth < COLOR_DEPTH_888) + return false; + if (adjusted_pix_clk_khz < TMDS_MIN_PIXEL_CLOCK) + return false; + + if ((adjusted_pix_clk_khz == 0) || + (adjusted_pix_clk_khz > enc10->base.features.max_hdmi_pixel_clock)) + return false; + + /* DCE11 HW does not support 420 */ + if (!enc10->base.features.ycbcr420_supported && + crtc_timing->pixel_encoding == PIXEL_ENCODING_YCBCR420) + return false; + + if (!enc10->base.features.flags.bits.HDMI_6GB_EN && + adjusted_pix_clk_khz >= 300000) + return false; + return true; +} + +bool dcn10_link_encoder_validate_dp_output( + const struct dcn10_link_encoder *enc10, + const struct dc_crtc_timing *crtc_timing) +{ + /* default RGB only */ + if (crtc_timing->pixel_encoding == PIXEL_ENCODING_RGB) + return true; + + if (enc10->base.features.flags.bits.IS_YCBCR_CAPABLE) + return true; + + /* for DCE 8.x or later DP Y-only feature, + * we need ASIC cap + FeatureSupportDPYonly, not support 666 + */ + if (crtc_timing->flags.Y_ONLY && + enc10->base.features.flags.bits.IS_YCBCR_CAPABLE && + crtc_timing->display_color_depth != COLOR_DEPTH_666) + return true; + + return false; +} + +void dcn10_link_encoder_construct( + struct dcn10_link_encoder *enc10, + const struct encoder_init_data *init_data, + const struct encoder_feature_support *enc_features, + const struct dcn10_link_enc_registers *link_regs, + const struct dcn10_link_enc_aux_registers *aux_regs, + const struct dcn10_link_enc_hpd_registers *hpd_regs, + const struct dcn10_link_enc_shift *link_shift, + const struct dcn10_link_enc_mask *link_mask) +{ + struct bp_encoder_cap_info bp_cap_info = {0}; + const struct dc_vbios_funcs *bp_funcs = init_data->ctx->dc_bios->funcs; + enum bp_result result = BP_RESULT_OK; + + enc10->base.funcs = &dcn10_lnk_enc_funcs; + enc10->base.ctx = init_data->ctx; + enc10->base.id = init_data->encoder; + + enc10->base.hpd_source = init_data->hpd_source; + enc10->base.connector = init_data->connector; + + enc10->base.preferred_engine = ENGINE_ID_UNKNOWN; + + enc10->base.features = *enc_features; + + enc10->base.transmitter = init_data->transmitter; + + /* set the flag to indicate whether driver poll the I2C data pin + * while doing the DP sink detect + */ + +/* if (dal_adapter_service_is_feature_supported(as, + FEATURE_DP_SINK_DETECT_POLL_DATA_PIN)) + enc10->base.features.flags.bits. + DP_SINK_DETECT_POLL_DATA_PIN = true;*/ + + enc10->base.output_signals = + SIGNAL_TYPE_DVI_SINGLE_LINK | + SIGNAL_TYPE_DVI_DUAL_LINK | + SIGNAL_TYPE_LVDS | + SIGNAL_TYPE_DISPLAY_PORT | + SIGNAL_TYPE_DISPLAY_PORT_MST | + SIGNAL_TYPE_EDP | + SIGNAL_TYPE_HDMI_TYPE_A; + + /* For DCE 8.0 and 8.1, by design, UNIPHY is hardwired to DIG_BE. + * SW always assign DIG_FE 1:1 mapped to DIG_FE for non-MST UNIPHY. + * SW assign DIG_FE to non-MST UNIPHY first and MST last. So prefer + * DIG is per UNIPHY and used by SST DP, eDP, HDMI, DVI and LVDS. + * Prefer DIG assignment is decided by board design. + * For DCE 8.0, there are only max 6 UNIPHYs, we assume board design + * and VBIOS will filter out 7 UNIPHY for DCE 8.0. + * By this, adding DIGG should not hurt DCE 8.0. + * This will let DCE 8.1 share DCE 8.0 as much as possible + */ + + enc10->link_regs = link_regs; + enc10->aux_regs = aux_regs; + enc10->hpd_regs = hpd_regs; + enc10->link_shift = link_shift; + enc10->link_mask = link_mask; + + switch (enc10->base.transmitter) { + case TRANSMITTER_UNIPHY_A: + enc10->base.preferred_engine = ENGINE_ID_DIGA; + break; + case TRANSMITTER_UNIPHY_B: + enc10->base.preferred_engine = ENGINE_ID_DIGB; + break; + case TRANSMITTER_UNIPHY_C: + enc10->base.preferred_engine = ENGINE_ID_DIGC; + break; + case TRANSMITTER_UNIPHY_D: + enc10->base.preferred_engine = ENGINE_ID_DIGD; + break; + case TRANSMITTER_UNIPHY_E: + enc10->base.preferred_engine = ENGINE_ID_DIGE; + break; + case TRANSMITTER_UNIPHY_F: + enc10->base.preferred_engine = ENGINE_ID_DIGF; + break; + case TRANSMITTER_UNIPHY_G: + enc10->base.preferred_engine = ENGINE_ID_DIGG; + break; + default: + ASSERT_CRITICAL(false); + enc10->base.preferred_engine = ENGINE_ID_UNKNOWN; + } + + /* default to one to mirror Windows behavior */ + enc10->base.features.flags.bits.HDMI_6GB_EN = 1; + + result = bp_funcs->get_encoder_cap_info(enc10->base.ctx->dc_bios, + enc10->base.id, &bp_cap_info); + + /* Override features with DCE-specific values */ + if (result == BP_RESULT_OK) { + enc10->base.features.flags.bits.IS_HBR2_CAPABLE = + bp_cap_info.DP_HBR2_EN; + enc10->base.features.flags.bits.IS_HBR3_CAPABLE = + bp_cap_info.DP_HBR3_EN; + enc10->base.features.flags.bits.HDMI_6GB_EN = bp_cap_info.HDMI_6GB_EN; + } else { + DC_LOG_WARNING("%s: Failed to get encoder_cap_info from VBIOS with error code %d!\n", + __func__, + result); + } +} + +bool dcn10_link_encoder_validate_output_with_stream( + struct link_encoder *enc, + const struct dc_stream_state *stream) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + bool is_valid; + + switch (stream->signal) { + case SIGNAL_TYPE_DVI_SINGLE_LINK: + case SIGNAL_TYPE_DVI_DUAL_LINK: + is_valid = dcn10_link_encoder_validate_dvi_output( + enc10, + stream->sink->link->connector_signal, + stream->signal, + &stream->timing); + break; + case SIGNAL_TYPE_HDMI_TYPE_A: + is_valid = dcn10_link_encoder_validate_hdmi_output( + enc10, + &stream->timing, + stream->phy_pix_clk); + break; + case SIGNAL_TYPE_DISPLAY_PORT: + case SIGNAL_TYPE_DISPLAY_PORT_MST: + is_valid = dcn10_link_encoder_validate_dp_output( + enc10, &stream->timing); + break; + case SIGNAL_TYPE_EDP: + is_valid = (stream->timing.pixel_encoding == PIXEL_ENCODING_RGB) ? true : false; + break; + case SIGNAL_TYPE_VIRTUAL: + is_valid = true; + break; + default: + is_valid = false; + break; + } + + return is_valid; +} + +void dcn10_link_encoder_hw_init( + struct link_encoder *enc) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + struct bp_transmitter_control cntl = { 0 }; + enum bp_result result; + + cntl.action = TRANSMITTER_CONTROL_INIT; + cntl.engine_id = ENGINE_ID_UNKNOWN; + cntl.transmitter = enc10->base.transmitter; + cntl.connector_obj_id = enc10->base.connector; + cntl.lanes_number = LANE_COUNT_FOUR; + cntl.coherent = false; + cntl.hpd_sel = enc10->base.hpd_source; + + if (enc10->base.connector.id == CONNECTOR_ID_EDP) + cntl.signal = SIGNAL_TYPE_EDP; + + result = link_transmitter_control(enc10, &cntl); + + if (result != BP_RESULT_OK) { + DC_LOG_ERROR("%s: Failed to execute VBIOS command table!\n", + __func__); + BREAK_TO_DEBUGGER(); + return; + } + + if (enc10->base.connector.id == CONNECTOR_ID_LVDS) { + cntl.action = TRANSMITTER_CONTROL_BACKLIGHT_BRIGHTNESS; + + result = link_transmitter_control(enc10, &cntl); + + ASSERT(result == BP_RESULT_OK); + + } + aux_initialize(enc10); + + /* reinitialize HPD. + * hpd_initialize() will pass DIG_FE id to HW context. + * All other routine within HW context will use fe_engine_offset + * as DIG_FE id even caller pass DIG_FE id. + * So this routine must be called first. + */ + hpd_initialize(enc10); +} + +void dcn10_link_encoder_destroy(struct link_encoder **enc) +{ + kfree(TO_DCN10_LINK_ENC(*enc)); + *enc = NULL; +} + +void dcn10_link_encoder_setup( + struct link_encoder *enc, + enum signal_type signal) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + + switch (signal) { + case SIGNAL_TYPE_EDP: + case SIGNAL_TYPE_DISPLAY_PORT: + /* DP SST */ + REG_UPDATE(DIG_BE_CNTL, DIG_MODE, 0); + break; + case SIGNAL_TYPE_LVDS: + /* LVDS */ + REG_UPDATE(DIG_BE_CNTL, DIG_MODE, 1); + break; + case SIGNAL_TYPE_DVI_SINGLE_LINK: + case SIGNAL_TYPE_DVI_DUAL_LINK: + /* TMDS-DVI */ + REG_UPDATE(DIG_BE_CNTL, DIG_MODE, 2); + break; + case SIGNAL_TYPE_HDMI_TYPE_A: + /* TMDS-HDMI */ + REG_UPDATE(DIG_BE_CNTL, DIG_MODE, 3); + break; + case SIGNAL_TYPE_DISPLAY_PORT_MST: + /* DP MST */ + REG_UPDATE(DIG_BE_CNTL, DIG_MODE, 5); + break; + default: + ASSERT_CRITICAL(false); + /* invalid mode ! */ + break; + } + +} + +/* TODO: still need depth or just pass in adjusted pixel clock? */ +void dcn10_link_encoder_enable_tmds_output( + struct link_encoder *enc, + enum clock_source_id clock_source, + enum dc_color_depth color_depth, + enum signal_type signal, + uint32_t pixel_clock) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + struct bp_transmitter_control cntl = { 0 }; + enum bp_result result; + + /* Enable the PHY */ + + cntl.action = TRANSMITTER_CONTROL_ENABLE; + cntl.engine_id = enc->preferred_engine; + cntl.transmitter = enc10->base.transmitter; + cntl.pll_id = clock_source; + cntl.signal = signal; + if (cntl.signal == SIGNAL_TYPE_DVI_DUAL_LINK) + cntl.lanes_number = 8; + else + cntl.lanes_number = 4; + + cntl.hpd_sel = enc10->base.hpd_source; + + cntl.pixel_clock = pixel_clock; + cntl.color_depth = color_depth; + + result = link_transmitter_control(enc10, &cntl); + + if (result != BP_RESULT_OK) { + DC_LOG_ERROR("%s: Failed to execute VBIOS command table!\n", + __func__); + BREAK_TO_DEBUGGER(); + } +} + +/* enables DP PHY output */ +void dcn10_link_encoder_enable_dp_output( + struct link_encoder *enc, + const struct dc_link_settings *link_settings, + enum clock_source_id clock_source) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + struct bp_transmitter_control cntl = { 0 }; + enum bp_result result; + + /* Enable the PHY */ + + /* number_of_lanes is used for pixel clock adjust, + * but it's not passed to asic_control. + * We need to set number of lanes manually. + */ + configure_encoder(enc10, link_settings); + + cntl.action = TRANSMITTER_CONTROL_ENABLE; + cntl.engine_id = enc->preferred_engine; + cntl.transmitter = enc10->base.transmitter; + cntl.pll_id = clock_source; + cntl.signal = SIGNAL_TYPE_DISPLAY_PORT; + cntl.lanes_number = link_settings->lane_count; + cntl.hpd_sel = enc10->base.hpd_source; + cntl.pixel_clock = link_settings->link_rate + * LINK_RATE_REF_FREQ_IN_KHZ; + /* TODO: check if undefined works */ + cntl.color_depth = COLOR_DEPTH_UNDEFINED; + + result = link_transmitter_control(enc10, &cntl); + + if (result != BP_RESULT_OK) { + DC_LOG_ERROR("%s: Failed to execute VBIOS command table!\n", + __func__); + BREAK_TO_DEBUGGER(); + } +} + +/* enables DP PHY output in MST mode */ +void dcn10_link_encoder_enable_dp_mst_output( + struct link_encoder *enc, + const struct dc_link_settings *link_settings, + enum clock_source_id clock_source) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + struct bp_transmitter_control cntl = { 0 }; + enum bp_result result; + + /* Enable the PHY */ + + /* number_of_lanes is used for pixel clock adjust, + * but it's not passed to asic_control. + * We need to set number of lanes manually. + */ + configure_encoder(enc10, link_settings); + + cntl.action = TRANSMITTER_CONTROL_ENABLE; + cntl.engine_id = ENGINE_ID_UNKNOWN; + cntl.transmitter = enc10->base.transmitter; + cntl.pll_id = clock_source; + cntl.signal = SIGNAL_TYPE_DISPLAY_PORT_MST; + cntl.lanes_number = link_settings->lane_count; + cntl.hpd_sel = enc10->base.hpd_source; + cntl.pixel_clock = link_settings->link_rate + * LINK_RATE_REF_FREQ_IN_KHZ; + /* TODO: check if undefined works */ + cntl.color_depth = COLOR_DEPTH_UNDEFINED; + + result = link_transmitter_control(enc10, &cntl); + + if (result != BP_RESULT_OK) { + DC_LOG_ERROR("%s: Failed to execute VBIOS command table!\n", + __func__); + BREAK_TO_DEBUGGER(); + } +} +/* + * @brief + * Disable transmitter and its encoder + */ +void dcn10_link_encoder_disable_output( + struct link_encoder *enc, + enum signal_type signal) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + struct bp_transmitter_control cntl = { 0 }; + enum bp_result result; + + if (!dcn10_is_dig_enabled(enc)) { + /* OF_SKIP_POWER_DOWN_INACTIVE_ENCODER */ + return; + } + /* Power-down RX and disable GPU PHY should be paired. + * Disabling PHY without powering down RX may cause + * symbol lock loss, on which we will get DP Sink interrupt. + */ + + /* There is a case for the DP active dongles + * where we want to disable the PHY but keep RX powered, + * for those we need to ignore DP Sink interrupt + * by checking lane count that has been set + * on the last do_enable_output(). + */ + + /* disable transmitter */ + cntl.action = TRANSMITTER_CONTROL_DISABLE; + cntl.transmitter = enc10->base.transmitter; + cntl.hpd_sel = enc10->base.hpd_source; + cntl.signal = signal; + cntl.connector_obj_id = enc10->base.connector; + + result = link_transmitter_control(enc10, &cntl); + + if (result != BP_RESULT_OK) { + DC_LOG_ERROR("%s: Failed to execute VBIOS command table!\n", + __func__); + BREAK_TO_DEBUGGER(); + return; + } + + /* disable encoder */ + if (dc_is_dp_signal(signal)) + link_encoder_disable(enc10); +} + +void dcn10_link_encoder_dp_set_lane_settings( + struct link_encoder *enc, + const struct link_training_settings *link_settings) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + union dpcd_training_lane_set training_lane_set = { { 0 } }; + int32_t lane = 0; + struct bp_transmitter_control cntl = { 0 }; + + if (!link_settings) { + BREAK_TO_DEBUGGER(); + return; + } + + cntl.action = TRANSMITTER_CONTROL_SET_VOLTAGE_AND_PREEMPASIS; + cntl.transmitter = enc10->base.transmitter; + cntl.connector_obj_id = enc10->base.connector; + cntl.lanes_number = link_settings->link_settings.lane_count; + cntl.hpd_sel = enc10->base.hpd_source; + cntl.pixel_clock = link_settings->link_settings.link_rate * + LINK_RATE_REF_FREQ_IN_KHZ; + + for (lane = 0; lane < link_settings->link_settings.lane_count; lane++) { + /* translate lane settings */ + + training_lane_set.bits.VOLTAGE_SWING_SET = + link_settings->lane_settings[lane].VOLTAGE_SWING; + training_lane_set.bits.PRE_EMPHASIS_SET = + link_settings->lane_settings[lane].PRE_EMPHASIS; + + /* post cursor 2 setting only applies to HBR2 link rate */ + if (link_settings->link_settings.link_rate == LINK_RATE_HIGH2) { + /* this is passed to VBIOS + * to program post cursor 2 level + */ + training_lane_set.bits.POST_CURSOR2_SET = + link_settings->lane_settings[lane].POST_CURSOR2; + } + + cntl.lane_select = lane; + cntl.lane_settings = training_lane_set.raw; + + /* call VBIOS table to set voltage swing and pre-emphasis */ + link_transmitter_control(enc10, &cntl); + } +} + +/* set DP PHY test and training patterns */ +void dcn10_link_encoder_dp_set_phy_pattern( + struct link_encoder *enc, + const struct encoder_set_dp_phy_pattern_param *param) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + + switch (param->dp_phy_pattern) { + case DP_TEST_PATTERN_TRAINING_PATTERN1: + dcn10_link_encoder_set_dp_phy_pattern_training_pattern(enc, 0); + break; + case DP_TEST_PATTERN_TRAINING_PATTERN2: + dcn10_link_encoder_set_dp_phy_pattern_training_pattern(enc, 1); + break; + case DP_TEST_PATTERN_TRAINING_PATTERN3: + dcn10_link_encoder_set_dp_phy_pattern_training_pattern(enc, 2); + break; + case DP_TEST_PATTERN_TRAINING_PATTERN4: + dcn10_link_encoder_set_dp_phy_pattern_training_pattern(enc, 3); + break; + case DP_TEST_PATTERN_D102: + set_dp_phy_pattern_d102(enc10); + break; + case DP_TEST_PATTERN_SYMBOL_ERROR: + set_dp_phy_pattern_symbol_error(enc10); + break; + case DP_TEST_PATTERN_PRBS7: + set_dp_phy_pattern_prbs7(enc10); + break; + case DP_TEST_PATTERN_80BIT_CUSTOM: + set_dp_phy_pattern_80bit_custom( + enc10, param->custom_pattern); + break; + case DP_TEST_PATTERN_CP2520_1: + set_dp_phy_pattern_hbr2_compliance_cp2520_2(enc10, 1); + break; + case DP_TEST_PATTERN_CP2520_2: + set_dp_phy_pattern_hbr2_compliance_cp2520_2(enc10, 2); + break; + case DP_TEST_PATTERN_CP2520_3: + set_dp_phy_pattern_hbr2_compliance_cp2520_2(enc10, 3); + break; + case DP_TEST_PATTERN_VIDEO_MODE: { + set_dp_phy_pattern_passthrough_mode( + enc10, param->dp_panel_mode); + break; + } + + default: + /* invalid phy pattern */ + ASSERT_CRITICAL(false); + break; + } +} + +static void fill_stream_allocation_row_info( + const struct link_mst_stream_allocation *stream_allocation, + uint32_t *src, + uint32_t *slots) +{ + const struct stream_encoder *stream_enc = stream_allocation->stream_enc; + + if (stream_enc) { + *src = stream_enc->id; + *slots = stream_allocation->slot_count; + } else { + *src = 0; + *slots = 0; + } +} + +/* programs DP MST VC payload allocation */ +void dcn10_link_encoder_update_mst_stream_allocation_table( + struct link_encoder *enc, + const struct link_mst_stream_allocation_table *table) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + uint32_t value0 = 0; + uint32_t value1 = 0; + uint32_t value2 = 0; + uint32_t slots = 0; + uint32_t src = 0; + uint32_t retries = 0; + + /* For CZ, there are only 3 pipes. So Virtual channel is up 3.*/ + + /* --- Set MSE Stream Attribute - + * Setup VC Payload Table on Tx Side, + * Issue allocation change trigger + * to commit payload on both tx and rx side + */ + + /* we should clean-up table each time */ + + if (table->stream_count >= 1) { + fill_stream_allocation_row_info( + &table->stream_allocations[0], + &src, + &slots); + } else { + src = 0; + slots = 0; + } + + REG_UPDATE_2(DP_MSE_SAT0, + DP_MSE_SAT_SRC0, src, + DP_MSE_SAT_SLOT_COUNT0, slots); + + if (table->stream_count >= 2) { + fill_stream_allocation_row_info( + &table->stream_allocations[1], + &src, + &slots); + } else { + src = 0; + slots = 0; + } + + REG_UPDATE_2(DP_MSE_SAT0, + DP_MSE_SAT_SRC1, src, + DP_MSE_SAT_SLOT_COUNT1, slots); + + if (table->stream_count >= 3) { + fill_stream_allocation_row_info( + &table->stream_allocations[2], + &src, + &slots); + } else { + src = 0; + slots = 0; + } + + REG_UPDATE_2(DP_MSE_SAT1, + DP_MSE_SAT_SRC2, src, + DP_MSE_SAT_SLOT_COUNT2, slots); + + if (table->stream_count >= 4) { + fill_stream_allocation_row_info( + &table->stream_allocations[3], + &src, + &slots); + } else { + src = 0; + slots = 0; + } + + REG_UPDATE_2(DP_MSE_SAT1, + DP_MSE_SAT_SRC3, src, + DP_MSE_SAT_SLOT_COUNT3, slots); + + /* --- wait for transaction finish */ + + /* send allocation change trigger (ACT) ? + * this step first sends the ACT, + * then double buffers the SAT into the hardware + * making the new allocation active on the DP MST mode link + */ + + /* DP_MSE_SAT_UPDATE: + * 0 - No Action + * 1 - Update SAT with trigger + * 2 - Update SAT without trigger + */ + REG_UPDATE(DP_MSE_SAT_UPDATE, + DP_MSE_SAT_UPDATE, 1); + + /* wait for update to complete + * (i.e. DP_MSE_SAT_UPDATE field is reset to 0) + * then wait for the transmission + * of at least 16 MTP headers on immediate local link. + * i.e. DP_MSE_16_MTP_KEEPOUT field (read only) is reset to 0 + * a value of 1 indicates that DP MST mode + * is in the 16 MTP keepout region after a VC has been added. + * MST stream bandwidth (VC rate) can be configured + * after this bit is cleared + */ + do { + udelay(10); + + value0 = REG_READ(DP_MSE_SAT_UPDATE); + + REG_GET(DP_MSE_SAT_UPDATE, + DP_MSE_SAT_UPDATE, &value1); + + REG_GET(DP_MSE_SAT_UPDATE, + DP_MSE_16_MTP_KEEPOUT, &value2); + + /* bit field DP_MSE_SAT_UPDATE is set to 1 already */ + if (!value1 && !value2) + break; + ++retries; + } while (retries < DP_MST_UPDATE_MAX_RETRY); +} + +void dcn10_link_encoder_connect_dig_be_to_fe( + struct link_encoder *enc, + enum engine_id engine, + bool connect) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + uint32_t field; + + if (engine != ENGINE_ID_UNKNOWN) { + + REG_GET(DIG_BE_CNTL, DIG_FE_SOURCE_SELECT, &field); + + if (connect) + field |= get_frontend_source(engine); + else + field &= ~get_frontend_source(engine); + + REG_UPDATE(DIG_BE_CNTL, DIG_FE_SOURCE_SELECT, field); + } +} + + +#define HPD_REG(reg)\ + (enc10->hpd_regs->reg) + +#define HPD_REG_READ(reg_name) \ + dm_read_reg(CTX, HPD_REG(reg_name)) + +#define HPD_REG_UPDATE_N(reg_name, n, ...) \ + generic_reg_update_ex(CTX, \ + HPD_REG(reg_name), \ + HPD_REG_READ(reg_name), \ + n, __VA_ARGS__) + +#define HPD_REG_UPDATE(reg_name, field, val) \ + HPD_REG_UPDATE_N(reg_name, 1, \ + FN(reg_name, field), val) + +void dcn10_link_encoder_enable_hpd(struct link_encoder *enc) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + + HPD_REG_UPDATE(DC_HPD_CONTROL, + DC_HPD_EN, 1); +} + +void dcn10_link_encoder_disable_hpd(struct link_encoder *enc) +{ + struct dcn10_link_encoder *enc10 = TO_DCN10_LINK_ENC(enc); + + HPD_REG_UPDATE(DC_HPD_CONTROL, + DC_HPD_EN, 0); +} + + +#define AUX_REG(reg)\ + (enc10->aux_regs->reg) + +#define AUX_REG_READ(reg_name) \ + dm_read_reg(CTX, AUX_REG(reg_name)) + +#define AUX_REG_UPDATE_N(reg_name, n, ...) \ + generic_reg_update_ex(CTX, \ + AUX_REG(reg_name), \ + AUX_REG_READ(reg_name), \ + n, __VA_ARGS__) + +#define AUX_REG_UPDATE(reg_name, field, val) \ + AUX_REG_UPDATE_N(reg_name, 1, \ + FN(reg_name, field), val) + +#define AUX_REG_UPDATE_2(reg, f1, v1, f2, v2) \ + AUX_REG_UPDATE_N(reg, 2,\ + FN(reg, f1), v1,\ + FN(reg, f2), v2) + +static void aux_initialize( + struct dcn10_link_encoder *enc10) +{ + enum hpd_source_id hpd_source = enc10->base.hpd_source; + + AUX_REG_UPDATE_2(AUX_CONTROL, + AUX_HPD_SEL, hpd_source, + AUX_LS_READ_EN, 0); + + /* 1/4 window (the maximum allowed) */ + AUX_REG_UPDATE(AUX_DPHY_RX_CONTROL0, + AUX_RX_RECEIVE_WINDOW, 1); +} diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_link_encoder.h b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_link_encoder.h new file mode 100644 index 000000000000..2a97cdb2cfbb --- /dev/null +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_link_encoder.h @@ -0,0 +1,330 @@ +/* + * Copyright 2012-15 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * Authors: AMD + * + */ + +#ifndef __DC_LINK_ENCODER__DCN10_H__ +#define __DC_LINK_ENCODER__DCN10_H__ + +#include "link_encoder.h" + +#define TO_DCN10_LINK_ENC(link_encoder)\ + container_of(link_encoder, struct dcn10_link_encoder, base) + + +#define AUX_REG_LIST(id)\ + SRI(AUX_CONTROL, DP_AUX, id), \ + SRI(AUX_DPHY_RX_CONTROL0, DP_AUX, id) + +#define HPD_REG_LIST(id)\ + SRI(DC_HPD_CONTROL, HPD, id) + +#define LE_DCN_COMMON_REG_LIST(id) \ + SRI(DIG_BE_CNTL, DIG, id), \ + SRI(DIG_BE_EN_CNTL, DIG, id), \ + SRI(DP_CONFIG, DP, id), \ + SRI(DP_DPHY_CNTL, DP, id), \ + SRI(DP_DPHY_PRBS_CNTL, DP, id), \ + SRI(DP_DPHY_SCRAM_CNTL, DP, id),\ + SRI(DP_DPHY_SYM0, DP, id), \ + SRI(DP_DPHY_SYM1, DP, id), \ + SRI(DP_DPHY_SYM2, DP, id), \ + SRI(DP_DPHY_TRAINING_PATTERN_SEL, DP, id), \ + SRI(DP_LINK_CNTL, DP, id), \ + SRI(DP_LINK_FRAMING_CNTL, DP, id), \ + SRI(DP_MSE_SAT0, DP, id), \ + SRI(DP_MSE_SAT1, DP, id), \ + SRI(DP_MSE_SAT2, DP, id), \ + SRI(DP_MSE_SAT_UPDATE, DP, id), \ + SRI(DP_SEC_CNTL, DP, id), \ + SRI(DP_VID_STREAM_CNTL, DP, id), \ + SRI(DP_DPHY_FAST_TRAINING, DP, id), \ + SRI(DP_SEC_CNTL1, DP, id), \ + SRI(DP_DPHY_BS_SR_SWAP_CNTL, DP, id), \ + SRI(DP_DPHY_INTERNAL_CTRL, DP, id), \ + SRI(DP_DPHY_HBR2_PATTERN_CONTROL, DP, id) + +#define LE_DCN10_REG_LIST(id)\ + LE_DCN_COMMON_REG_LIST(id) + +struct dcn10_link_enc_aux_registers { + uint32_t AUX_CONTROL; + uint32_t AUX_DPHY_RX_CONTROL0; +}; + +struct dcn10_link_enc_hpd_registers { + uint32_t DC_HPD_CONTROL; +}; + +struct dcn10_link_enc_registers { + uint32_t DIG_BE_CNTL; + uint32_t DIG_BE_EN_CNTL; + uint32_t DP_CONFIG; + uint32_t DP_DPHY_CNTL; + uint32_t DP_DPHY_INTERNAL_CTRL; + uint32_t DP_DPHY_PRBS_CNTL; + uint32_t DP_DPHY_SCRAM_CNTL; + uint32_t DP_DPHY_SYM0; + uint32_t DP_DPHY_SYM1; + uint32_t DP_DPHY_SYM2; + uint32_t DP_DPHY_TRAINING_PATTERN_SEL; + uint32_t DP_LINK_CNTL; + uint32_t DP_LINK_FRAMING_CNTL; + uint32_t DP_MSE_SAT0; + uint32_t DP_MSE_SAT1; + uint32_t DP_MSE_SAT2; + uint32_t DP_MSE_SAT_UPDATE; + uint32_t DP_SEC_CNTL; + uint32_t DP_VID_STREAM_CNTL; + uint32_t DP_DPHY_FAST_TRAINING; + uint32_t DP_DPHY_BS_SR_SWAP_CNTL; + uint32_t DP_DPHY_HBR2_PATTERN_CONTROL; + uint32_t DP_SEC_CNTL1; +}; + +#define LE_SF(reg_name, field_name, post_fix)\ + .field_name = reg_name ## __ ## field_name ## post_fix + +#define LINK_ENCODER_MASK_SH_LIST_DCN10(mask_sh)\ + LE_SF(DIG0_DIG_BE_EN_CNTL, DIG_ENABLE, mask_sh),\ + LE_SF(DIG0_DIG_BE_CNTL, DIG_HPD_SELECT, mask_sh),\ + LE_SF(DIG0_DIG_BE_CNTL, DIG_MODE, mask_sh),\ + LE_SF(DIG0_DIG_BE_CNTL, DIG_FE_SOURCE_SELECT, mask_sh),\ + LE_SF(DP0_DP_DPHY_CNTL, DPHY_BYPASS, mask_sh),\ + LE_SF(DP0_DP_DPHY_CNTL, DPHY_ATEST_SEL_LANE0, mask_sh),\ + LE_SF(DP0_DP_DPHY_CNTL, DPHY_ATEST_SEL_LANE1, mask_sh),\ + LE_SF(DP0_DP_DPHY_CNTL, DPHY_ATEST_SEL_LANE2, mask_sh),\ + LE_SF(DP0_DP_DPHY_CNTL, DPHY_ATEST_SEL_LANE3, mask_sh),\ + LE_SF(DP0_DP_DPHY_PRBS_CNTL, DPHY_PRBS_EN, mask_sh),\ + LE_SF(DP0_DP_DPHY_PRBS_CNTL, DPHY_PRBS_SEL, mask_sh),\ + LE_SF(DP0_DP_DPHY_SYM0, DPHY_SYM1, mask_sh),\ + LE_SF(DP0_DP_DPHY_SYM0, DPHY_SYM2, mask_sh),\ + LE_SF(DP0_DP_DPHY_SYM0, DPHY_SYM3, mask_sh),\ + LE_SF(DP0_DP_DPHY_SYM1, DPHY_SYM4, mask_sh),\ + LE_SF(DP0_DP_DPHY_SYM1, DPHY_SYM5, mask_sh),\ + LE_SF(DP0_DP_DPHY_SYM1, DPHY_SYM6, mask_sh),\ + LE_SF(DP0_DP_DPHY_SYM2, DPHY_SYM7, mask_sh),\ + LE_SF(DP0_DP_DPHY_SYM2, DPHY_SYM8, mask_sh),\ + LE_SF(DP0_DP_DPHY_SCRAM_CNTL, DPHY_SCRAMBLER_BS_COUNT, mask_sh),\ + LE_SF(DP0_DP_DPHY_SCRAM_CNTL, DPHY_SCRAMBLER_ADVANCE, mask_sh),\ + LE_SF(DP0_DP_DPHY_FAST_TRAINING, DPHY_RX_FAST_TRAINING_CAPABLE, mask_sh),\ + LE_SF(DP0_DP_DPHY_BS_SR_SWAP_CNTL, DPHY_LOAD_BS_COUNT, mask_sh),\ + LE_SF(DP0_DP_DPHY_TRAINING_PATTERN_SEL, DPHY_TRAINING_PATTERN_SEL, mask_sh),\ + LE_SF(DP0_DP_DPHY_HBR2_PATTERN_CONTROL, DP_DPHY_HBR2_PATTERN_CONTROL, mask_sh),\ + LE_SF(DP0_DP_LINK_CNTL, DP_LINK_TRAINING_COMPLETE, mask_sh),\ + LE_SF(DP0_DP_LINK_FRAMING_CNTL, DP_IDLE_BS_INTERVAL, mask_sh),\ + LE_SF(DP0_DP_LINK_FRAMING_CNTL, DP_VBID_DISABLE, mask_sh),\ + LE_SF(DP0_DP_LINK_FRAMING_CNTL, DP_VID_ENHANCED_FRAME_MODE, mask_sh),\ + LE_SF(DP0_DP_VID_STREAM_CNTL, DP_VID_STREAM_ENABLE, mask_sh),\ + LE_SF(DP0_DP_CONFIG, DP_UDI_LANES, mask_sh),\ + LE_SF(DP0_DP_SEC_CNTL1, DP_SEC_GSP0_LINE_NUM, mask_sh),\ + LE_SF(DP0_DP_SEC_CNTL1, DP_SEC_GSP0_PRIORITY, mask_sh),\ + LE_SF(DP0_DP_MSE_SAT0, DP_MSE_SAT_SRC0, mask_sh),\ + LE_SF(DP0_DP_MSE_SAT0, DP_MSE_SAT_SRC1, mask_sh),\ + LE_SF(DP0_DP_MSE_SAT0, DP_MSE_SAT_SLOT_COUNT0, mask_sh),\ + LE_SF(DP0_DP_MSE_SAT0, DP_MSE_SAT_SLOT_COUNT1, mask_sh),\ + LE_SF(DP0_DP_MSE_SAT1, DP_MSE_SAT_SRC2, mask_sh),\ + LE_SF(DP0_DP_MSE_SAT1, DP_MSE_SAT_SRC3, mask_sh),\ + LE_SF(DP0_DP_MSE_SAT1, DP_MSE_SAT_SLOT_COUNT2, mask_sh),\ + LE_SF(DP0_DP_MSE_SAT1, DP_MSE_SAT_SLOT_COUNT3, mask_sh),\ + LE_SF(DP0_DP_MSE_SAT_UPDATE, DP_MSE_SAT_UPDATE, mask_sh),\ + LE_SF(DP0_DP_MSE_SAT_UPDATE, DP_MSE_16_MTP_KEEPOUT, mask_sh),\ + LE_SF(DP_AUX0_AUX_CONTROL, AUX_HPD_SEL, mask_sh),\ + LE_SF(DP_AUX0_AUX_CONTROL, AUX_LS_READ_EN, mask_sh),\ + LE_SF(DP_AUX0_AUX_DPHY_RX_CONTROL0, AUX_RX_RECEIVE_WINDOW, mask_sh),\ + LE_SF(HPD0_DC_HPD_CONTROL, DC_HPD_EN, mask_sh) + +#define DCN_LINK_ENCODER_REG_FIELD_LIST(type) \ + type DIG_ENABLE;\ + type DIG_HPD_SELECT;\ + type DIG_MODE;\ + type DIG_FE_SOURCE_SELECT;\ + type DPHY_BYPASS;\ + type DPHY_ATEST_SEL_LANE0;\ + type DPHY_ATEST_SEL_LANE1;\ + type DPHY_ATEST_SEL_LANE2;\ + type DPHY_ATEST_SEL_LANE3;\ + type DPHY_PRBS_EN;\ + type DPHY_PRBS_SEL;\ + type DPHY_SYM1;\ + type DPHY_SYM2;\ + type DPHY_SYM3;\ + type DPHY_SYM4;\ + type DPHY_SYM5;\ + type DPHY_SYM6;\ + type DPHY_SYM7;\ + type DPHY_SYM8;\ + type DPHY_SCRAMBLER_BS_COUNT;\ + type DPHY_SCRAMBLER_ADVANCE;\ + type DPHY_RX_FAST_TRAINING_CAPABLE;\ + type DPHY_LOAD_BS_COUNT;\ + type DPHY_TRAINING_PATTERN_SEL;\ + type DP_DPHY_HBR2_PATTERN_CONTROL;\ + type DP_LINK_TRAINING_COMPLETE;\ + type DP_IDLE_BS_INTERVAL;\ + type DP_VBID_DISABLE;\ + type DP_VID_ENHANCED_FRAME_MODE;\ + type DP_VID_STREAM_ENABLE;\ + type DP_UDI_LANES;\ + type DP_SEC_GSP0_LINE_NUM;\ + type DP_SEC_GSP0_PRIORITY;\ + type DP_MSE_SAT_SRC0;\ + type DP_MSE_SAT_SRC1;\ + type DP_MSE_SAT_SRC2;\ + type DP_MSE_SAT_SRC3;\ + type DP_MSE_SAT_SLOT_COUNT0;\ + type DP_MSE_SAT_SLOT_COUNT1;\ + type DP_MSE_SAT_SLOT_COUNT2;\ + type DP_MSE_SAT_SLOT_COUNT3;\ + type DP_MSE_SAT_UPDATE;\ + type DP_MSE_16_MTP_KEEPOUT;\ + type AUX_HPD_SEL;\ + type AUX_LS_READ_EN;\ + type AUX_RX_RECEIVE_WINDOW;\ + type DC_HPD_EN + +struct dcn10_link_enc_shift { + DCN_LINK_ENCODER_REG_FIELD_LIST(uint8_t); +}; + +struct dcn10_link_enc_mask { + DCN_LINK_ENCODER_REG_FIELD_LIST(uint32_t); +}; + +struct dcn10_link_encoder { + struct link_encoder base; + const struct dcn10_link_enc_registers *link_regs; + const struct dcn10_link_enc_aux_registers *aux_regs; + const struct dcn10_link_enc_hpd_registers *hpd_regs; + const struct dcn10_link_enc_shift *link_shift; + const struct dcn10_link_enc_mask *link_mask; +}; + + +void dcn10_link_encoder_construct( + struct dcn10_link_encoder *enc10, + const struct encoder_init_data *init_data, + const struct encoder_feature_support *enc_features, + const struct dcn10_link_enc_registers *link_regs, + const struct dcn10_link_enc_aux_registers *aux_regs, + const struct dcn10_link_enc_hpd_registers *hpd_regs, + const struct dcn10_link_enc_shift *link_shift, + const struct dcn10_link_enc_mask *link_mask); + +bool dcn10_link_encoder_validate_dvi_output( + const struct dcn10_link_encoder *enc10, + enum signal_type connector_signal, + enum signal_type signal, + const struct dc_crtc_timing *crtc_timing); + +bool dcn10_link_encoder_validate_rgb_output( + const struct dcn10_link_encoder *enc10, + const struct dc_crtc_timing *crtc_timing); + +bool dcn10_link_encoder_validate_dp_output( + const struct dcn10_link_encoder *enc10, + const struct dc_crtc_timing *crtc_timing); + +bool dcn10_link_encoder_validate_wireless_output( + const struct dcn10_link_encoder *enc10, + const struct dc_crtc_timing *crtc_timing); + +bool dcn10_link_encoder_validate_output_with_stream( + struct link_encoder *enc, + const struct dc_stream_state *stream); + +/****************** HW programming ************************/ + +/* initialize HW */ /* why do we initialze aux in here? */ +void dcn10_link_encoder_hw_init(struct link_encoder *enc); + +void dcn10_link_encoder_destroy(struct link_encoder **enc); + +/* program DIG_MODE in DIG_BE */ +/* TODO can this be combined with enable_output? */ +void dcn10_link_encoder_setup( + struct link_encoder *enc, + enum signal_type signal); + +/* enables TMDS PHY output */ +/* TODO: still need depth or just pass in adjusted pixel clock? */ +void dcn10_link_encoder_enable_tmds_output( + struct link_encoder *enc, + enum clock_source_id clock_source, + enum dc_color_depth color_depth, + enum signal_type signal, + uint32_t pixel_clock); + +/* enables DP PHY output */ +void dcn10_link_encoder_enable_dp_output( + struct link_encoder *enc, + const struct dc_link_settings *link_settings, + enum clock_source_id clock_source); + +/* enables DP PHY output in MST mode */ +void dcn10_link_encoder_enable_dp_mst_output( + struct link_encoder *enc, + const struct dc_link_settings *link_settings, + enum clock_source_id clock_source); + +/* disable PHY output */ +void dcn10_link_encoder_disable_output( + struct link_encoder *enc, + enum signal_type signal); + +/* set DP lane settings */ +void dcn10_link_encoder_dp_set_lane_settings( + struct link_encoder *enc, + const struct link_training_settings *link_settings); + +void dcn10_link_encoder_dp_set_phy_pattern( + struct link_encoder *enc, + const struct encoder_set_dp_phy_pattern_param *param); + +/* programs DP MST VC payload allocation */ +void dcn10_link_encoder_update_mst_stream_allocation_table( + struct link_encoder *enc, + const struct link_mst_stream_allocation_table *table); + +void dcn10_link_encoder_connect_dig_be_to_fe( + struct link_encoder *enc, + enum engine_id engine, + bool connect); + +void dcn10_link_encoder_set_dp_phy_pattern_training_pattern( + struct link_encoder *enc, + uint32_t index); + +void dcn10_link_encoder_enable_hpd(struct link_encoder *enc); + +void dcn10_link_encoder_disable_hpd(struct link_encoder *enc); + +void dcn10_psr_program_dp_dphy_fast_training(struct link_encoder *enc, + bool exit_link_training_required); + +void dcn10_psr_program_secondary_packet(struct link_encoder *enc, + unsigned int sdp_transmit_line_num_deadline); + +bool dcn10_is_dig_enabled(struct link_encoder *enc); + +#endif /* __DC_LINK_ENCODER__DCN10_H__ */ diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_mpc.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_mpc.c index 179890b1a8c4..9ca51ae46de7 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_mpc.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_mpc.c @@ -65,6 +65,7 @@ static void mpc1_update_blending( int mpcc_id) { struct dcn10_mpc *mpc10 = TO_DCN10_MPC(mpc); + struct mpcc *mpcc = mpc1_get_mpcc(mpc, mpcc_id); REG_UPDATE_5(MPCC_CONTROL[mpcc_id], MPCC_ALPHA_BLND_MODE, blnd_cfg->alpha_mode, @@ -74,6 +75,7 @@ static void mpc1_update_blending( MPCC_GLOBAL_GAIN, blnd_cfg->global_gain); mpc1_set_bg_color(mpc, &blnd_cfg->black_color, mpcc_id); + mpcc->blnd_cfg = *blnd_cfg; } void mpc1_update_stereo_mix( @@ -235,8 +237,7 @@ struct mpcc *mpc1_insert_plane( } /* update the blending configuration */ - new_mpcc->blnd_cfg = *blnd_cfg; - mpc->funcs->update_blending(mpc, &new_mpcc->blnd_cfg, mpcc_id); + mpc->funcs->update_blending(mpc, blnd_cfg, mpcc_id); /* update the stereo mix settings, if provided */ if (sm_cfg != NULL) { @@ -409,7 +410,26 @@ void mpc1_init_mpcc_list_from_hw( } } +void mpc1_read_mpcc_state( + struct mpc *mpc, + int mpcc_inst, + struct mpcc_state *s) +{ + struct dcn10_mpc *mpc10 = TO_DCN10_MPC(mpc); + + REG_GET(MPCC_OPP_ID[mpcc_inst], MPCC_OPP_ID, &s->opp_id); + REG_GET(MPCC_TOP_SEL[mpcc_inst], MPCC_TOP_SEL, &s->dpp_id); + REG_GET(MPCC_BOT_SEL[mpcc_inst], MPCC_BOT_SEL, &s->bot_mpcc_id); + REG_GET_4(MPCC_CONTROL[mpcc_inst], MPCC_MODE, &s->mode, + MPCC_ALPHA_BLND_MODE, &s->alpha_mode, + MPCC_ALPHA_MULTIPLIED_MODE, &s->pre_multiplied_alpha, + MPCC_BLND_ACTIVE_OVERLAP_ONLY, &s->overlap_only); + REG_GET_2(MPCC_STATUS[mpcc_inst], MPCC_IDLE, &s->idle, + MPCC_BUSY, &s->busy); +} + const struct mpc_funcs dcn10_mpc_funcs = { + .read_mpcc_state = mpc1_read_mpcc_state, .insert_plane = mpc1_insert_plane, .remove_mpcc = mpc1_remove_mpcc, .mpc_init = mpc1_mpc_init, diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_mpc.h b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_mpc.h index 267a2995ef6e..d3d16c4cbea3 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_mpc.h +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_mpc.h @@ -183,4 +183,9 @@ struct mpcc *mpc1_get_mpcc_for_dpp( struct mpc_tree *tree, int dpp_id); +void mpc1_read_mpcc_state( + struct mpc *mpc, + int mpcc_inst, + struct mpcc_state *s); + #endif diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_optc.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_optc.c index 4bf64d1b2c60..f2fbce0e3fc5 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_optc.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_optc.c @@ -93,6 +93,81 @@ static void optc1_disable_stereo(struct timing_generator *optc) OTG_3D_STRUCTURE_STEREO_SEL_OVR, 0); } +static uint32_t get_start_vline(struct timing_generator *optc, const struct dc_crtc_timing *dc_crtc_timing) +{ + struct dc_crtc_timing patched_crtc_timing; + int vesa_sync_start; + int asic_blank_end; + int interlace_factor; + int vertical_line_start; + + patched_crtc_timing = *dc_crtc_timing; + optc1_apply_front_porch_workaround(optc, &patched_crtc_timing); + + vesa_sync_start = patched_crtc_timing.h_addressable + + patched_crtc_timing.h_border_right + + patched_crtc_timing.h_front_porch; + + asic_blank_end = patched_crtc_timing.h_total - + vesa_sync_start - + patched_crtc_timing.h_border_left; + + interlace_factor = patched_crtc_timing.flags.INTERLACE ? 2 : 1; + + vesa_sync_start = patched_crtc_timing.v_addressable + + patched_crtc_timing.v_border_bottom + + patched_crtc_timing.v_front_porch; + + asic_blank_end = (patched_crtc_timing.v_total - + vesa_sync_start - + patched_crtc_timing.v_border_top) + * interlace_factor; + + vertical_line_start = asic_blank_end - optc->dlg_otg_param.vstartup_start + 1; + if (vertical_line_start < 0) { + ASSERT(0); + vertical_line_start = 0; + } + + return vertical_line_start; +} + +void optc1_program_vline_interrupt( + struct timing_generator *optc, + const struct dc_crtc_timing *dc_crtc_timing, + unsigned long long vsync_delta) +{ + + struct optc *optc1 = DCN10TG_FROM_TG(optc); + + unsigned long long req_delta_tens_of_usec = div64_u64((vsync_delta + 9999), 10000); + unsigned long long pix_clk_hundreds_khz = div64_u64((dc_crtc_timing->pix_clk_khz + 99), 100); + uint32_t req_delta_lines = (uint32_t) div64_u64( + (req_delta_tens_of_usec * pix_clk_hundreds_khz + dc_crtc_timing->h_total - 1), + dc_crtc_timing->h_total); + + uint32_t vsync_line = get_start_vline(optc, dc_crtc_timing); + uint32_t start_line = 0; + uint32_t endLine = 0; + + if (req_delta_lines != 0) + req_delta_lines--; + + if (req_delta_lines > vsync_line) + start_line = dc_crtc_timing->v_total - (req_delta_lines - vsync_line) - 1; + else + start_line = vsync_line - req_delta_lines; + + endLine = start_line + 2; + + if (endLine >= dc_crtc_timing->v_total) + endLine = 2; + + REG_SET_2(OTG_VERTICAL_INTERRUPT0_POSITION, 0, + OTG_VERTICAL_INTERRUPT0_LINE_START, start_line, + OTG_VERTICAL_INTERRUPT0_LINE_END, endLine); +} + /** * program_timing_generator used by mode timing set * Program CRTC Timing Registers - OTG_H_*, OTG_V_*, Pixel repetition. @@ -285,7 +360,7 @@ void optc1_program_timing( } -static void optc1_set_blank_data_double_buffer(struct timing_generator *optc, bool enable) +void optc1_set_blank_data_double_buffer(struct timing_generator *optc, bool enable) { struct optc *optc1 = DCN10TG_FROM_TG(optc); @@ -780,17 +855,17 @@ void optc1_set_drr( OTG_SET_V_TOTAL_MIN_MASK_EN, 0, OTG_SET_V_TOTAL_MIN_MASK, 0); } else { - REG_SET(OTG_V_TOTAL_MIN, 0, - OTG_V_TOTAL_MIN, 0); - - REG_SET(OTG_V_TOTAL_MAX, 0, - OTG_V_TOTAL_MAX, 0); - REG_UPDATE_4(OTG_V_TOTAL_CONTROL, OTG_SET_V_TOTAL_MIN_MASK, 0, OTG_V_TOTAL_MIN_SEL, 0, OTG_V_TOTAL_MAX_SEL, 0, OTG_FORCE_LOCK_ON_EVENT, 0); + + REG_SET(OTG_V_TOTAL_MIN, 0, + OTG_V_TOTAL_MIN, 0); + + REG_SET(OTG_V_TOTAL_MAX, 0, + OTG_V_TOTAL_MAX, 0); } } @@ -1154,6 +1229,12 @@ void optc1_read_otg_state(struct optc *optc1, REG_GET(OTG_V_TOTAL_MIN, OTG_V_TOTAL_MIN, &s->v_total_min); + REG_GET(OTG_V_TOTAL_CONTROL, + OTG_V_TOTAL_MAX_SEL, &s->v_total_max_sel); + + REG_GET(OTG_V_TOTAL_CONTROL, + OTG_V_TOTAL_MIN_SEL, &s->v_total_min_sel); + REG_GET_2(OTG_V_SYNC_A, OTG_V_SYNC_A_START, &s->v_sync_a_start, OTG_V_SYNC_A_END, &s->v_sync_a_end); @@ -1176,20 +1257,20 @@ void optc1_read_otg_state(struct optc *optc1, OPTC_UNDERFLOW_OCCURRED_STATUS, &s->underflow_occurred_status); } -static void optc1_clear_optc_underflow(struct timing_generator *optc) +void optc1_clear_optc_underflow(struct timing_generator *optc) { struct optc *optc1 = DCN10TG_FROM_TG(optc); REG_UPDATE(OPTC_INPUT_GLOBAL_CONTROL, OPTC_UNDERFLOW_CLEAR, 1); } -static void optc1_tg_init(struct timing_generator *optc) +void optc1_tg_init(struct timing_generator *optc) { optc1_set_blank_data_double_buffer(optc, true); optc1_clear_optc_underflow(optc); } -static bool optc1_is_tg_enabled(struct timing_generator *optc) +bool optc1_is_tg_enabled(struct timing_generator *optc) { struct optc *optc1 = DCN10TG_FROM_TG(optc); uint32_t otg_enabled = 0; @@ -1200,7 +1281,7 @@ static bool optc1_is_tg_enabled(struct timing_generator *optc) } -static bool optc1_is_optc_underflow_occurred(struct timing_generator *optc) +bool optc1_is_optc_underflow_occurred(struct timing_generator *optc) { struct optc *optc1 = DCN10TG_FROM_TG(optc); uint32_t underflow_occurred = 0; @@ -1215,6 +1296,7 @@ static bool optc1_is_optc_underflow_occurred(struct timing_generator *optc) static const struct timing_generator_funcs dcn10_tg_funcs = { .validate_timing = optc1_validate_timing, .program_timing = optc1_program_timing, + .program_vline_interrupt = optc1_program_vline_interrupt, .program_global_sync = optc1_program_global_sync, .enable_crtc = optc1_enable_crtc, .disable_crtc = optc1_disable_crtc, diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_optc.h b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_optc.h index d25e7bf0d0d7..c62052f46460 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_optc.h +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_optc.h @@ -65,6 +65,8 @@ SRI(OTG_NOM_VERT_POSITION, OTG, inst),\ SRI(OTG_BLACK_COLOR, OTG, inst),\ SRI(OTG_CLOCK_CONTROL, OTG, inst),\ + SRI(OTG_VERTICAL_INTERRUPT0_CONTROL, OTG, inst),\ + SRI(OTG_VERTICAL_INTERRUPT0_POSITION, OTG, inst),\ SRI(OTG_VERTICAL_INTERRUPT2_CONTROL, OTG, inst),\ SRI(OTG_VERTICAL_INTERRUPT2_POSITION, OTG, inst),\ SRI(OPTC_INPUT_CLOCK_CONTROL, ODM, inst),\ @@ -124,6 +126,8 @@ struct dcn_optc_registers { uint32_t OTG_TEST_PATTERN_CONTROL; uint32_t OTG_TEST_PATTERN_COLOR; uint32_t OTG_CLOCK_CONTROL; + uint32_t OTG_VERTICAL_INTERRUPT0_CONTROL; + uint32_t OTG_VERTICAL_INTERRUPT0_POSITION; uint32_t OTG_VERTICAL_INTERRUPT2_CONTROL; uint32_t OTG_VERTICAL_INTERRUPT2_POSITION; uint32_t OPTC_INPUT_CLOCK_CONTROL; @@ -206,6 +210,9 @@ struct dcn_optc_registers { SF(OTG0_OTG_CLOCK_CONTROL, OTG_CLOCK_EN, mask_sh),\ SF(OTG0_OTG_CLOCK_CONTROL, OTG_CLOCK_ON, mask_sh),\ SF(OTG0_OTG_CLOCK_CONTROL, OTG_CLOCK_GATE_DIS, mask_sh),\ + SF(OTG0_OTG_VERTICAL_INTERRUPT0_CONTROL, OTG_VERTICAL_INTERRUPT0_INT_ENABLE, mask_sh),\ + SF(OTG0_OTG_VERTICAL_INTERRUPT0_POSITION, OTG_VERTICAL_INTERRUPT0_LINE_START, mask_sh),\ + SF(OTG0_OTG_VERTICAL_INTERRUPT0_POSITION, OTG_VERTICAL_INTERRUPT0_LINE_END, mask_sh),\ SF(OTG0_OTG_VERTICAL_INTERRUPT2_CONTROL, OTG_VERTICAL_INTERRUPT2_INT_ENABLE, mask_sh),\ SF(OTG0_OTG_VERTICAL_INTERRUPT2_POSITION, OTG_VERTICAL_INTERRUPT2_LINE_START, mask_sh),\ SF(ODM0_OPTC_INPUT_CLOCK_CONTROL, OPTC_INPUT_CLK_EN, mask_sh),\ @@ -323,6 +330,9 @@ struct dcn_optc_registers { type OTG_CLOCK_EN;\ type OTG_CLOCK_ON;\ type OTG_CLOCK_GATE_DIS;\ + type OTG_VERTICAL_INTERRUPT0_INT_ENABLE;\ + type OTG_VERTICAL_INTERRUPT0_LINE_START;\ + type OTG_VERTICAL_INTERRUPT0_LINE_END;\ type OTG_VERTICAL_INTERRUPT2_INT_ENABLE;\ type OTG_VERTICAL_INTERRUPT2_LINE_START;\ type OPTC_INPUT_CLK_EN;\ @@ -396,6 +406,8 @@ struct dcn_otg_state { uint32_t v_total; uint32_t v_total_max; uint32_t v_total_min; + uint32_t v_total_min_sel; + uint32_t v_total_max_sel; uint32_t v_sync_a_start; uint32_t v_sync_a_end; uint32_t h_blank_start; @@ -420,6 +432,10 @@ void optc1_program_timing( const struct dc_crtc_timing *dc_crtc_timing, bool use_vbios); +void optc1_program_vline_interrupt(struct timing_generator *optc, + const struct dc_crtc_timing *dc_crtc_timing, + unsigned long long vsync_delta); + void optc1_program_global_sync( struct timing_generator *optc); @@ -481,4 +497,14 @@ void optc1_program_stereo(struct timing_generator *optc, bool optc1_is_stereo_left_eye(struct timing_generator *optc); +void optc1_clear_optc_underflow(struct timing_generator *optc); + +void optc1_tg_init(struct timing_generator *optc); + +bool optc1_is_tg_enabled(struct timing_generator *optc); + +bool optc1_is_optc_underflow_occurred(struct timing_generator *optc); + +void optc1_set_blank_data_double_buffer(struct timing_generator *optc, bool enable); + #endif /* __DC_TIMING_GENERATOR_DCN10_H__ */ diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_resource.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_resource.c index 02bd664aed3e..df5cb2d1d164 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_resource.c @@ -38,8 +38,8 @@ #include "dcn10/dcn10_hw_sequencer.h" #include "dce110/dce110_hw_sequencer.h" #include "dcn10/dcn10_opp.h" -#include "dce/dce_link_encoder.h" -#include "dce/dce_stream_encoder.h" +#include "dcn10/dcn10_link_encoder.h" +#include "dcn10/dcn10_stream_encoder.h" #include "dce/dce_clocks.h" #include "dce/dce_clock_source.h" #include "dce/dce_audio.h" @@ -166,36 +166,22 @@ static const struct dce_abm_mask abm_mask = { #define stream_enc_regs(id)\ [id] = {\ - SE_DCN_REG_LIST(id),\ - .TMDS_CNTL = 0,\ - .AFMT_AVI_INFO0 = 0,\ - .AFMT_AVI_INFO1 = 0,\ - .AFMT_AVI_INFO2 = 0,\ - .AFMT_AVI_INFO3 = 0,\ + SE_DCN_REG_LIST(id)\ } -static const struct dce110_stream_enc_registers stream_enc_regs[] = { +static const struct dcn10_stream_enc_registers stream_enc_regs[] = { stream_enc_regs(0), stream_enc_regs(1), stream_enc_regs(2), stream_enc_regs(3), }; -static const struct dce_stream_encoder_shift se_shift = { +static const struct dcn10_stream_encoder_shift se_shift = { SE_COMMON_MASK_SH_LIST_DCN10(__SHIFT) }; -static const struct dce_stream_encoder_mask se_mask = { - SE_COMMON_MASK_SH_LIST_DCN10(_MASK), - .AFMT_GENERIC0_UPDATE = 0, - .AFMT_GENERIC2_UPDATE = 0, - .DP_DYN_RANGE = 0, - .DP_YCBCR_RANGE = 0, - .HDMI_AVI_INFO_SEND = 0, - .HDMI_AVI_INFO_CONT = 0, - .HDMI_AVI_INFO_LINE = 0, - .DP_SEC_AVI_ENABLE = 0, - .AFMT_AVI_INFO_VERSION = 0 +static const struct dcn10_stream_encoder_mask se_mask = { + SE_COMMON_MASK_SH_LIST_DCN10(_MASK) }; #define audio_regs(id)\ @@ -228,13 +214,11 @@ static const struct dce_aduio_mask audio_mask = { AUX_REG_LIST(id)\ } -static const struct dce110_link_enc_aux_registers link_enc_aux_regs[] = { +static const struct dcn10_link_enc_aux_registers link_enc_aux_regs[] = { aux_regs(0), aux_regs(1), aux_regs(2), - aux_regs(3), - aux_regs(4), - aux_regs(5) + aux_regs(3) }; #define hpd_regs(id)\ @@ -242,13 +226,11 @@ static const struct dce110_link_enc_aux_registers link_enc_aux_regs[] = { HPD_REG_LIST(id)\ } -static const struct dce110_link_enc_hpd_registers link_enc_hpd_regs[] = { +static const struct dcn10_link_enc_hpd_registers link_enc_hpd_regs[] = { hpd_regs(0), hpd_regs(1), hpd_regs(2), - hpd_regs(3), - hpd_regs(4), - hpd_regs(5) + hpd_regs(3) }; #define link_regs(id)\ @@ -257,14 +239,19 @@ static const struct dce110_link_enc_hpd_registers link_enc_hpd_regs[] = { SRI(DP_DPHY_INTERNAL_CTRL, DP, id) \ } -static const struct dce110_link_enc_registers link_enc_regs[] = { +static const struct dcn10_link_enc_registers link_enc_regs[] = { link_regs(0), link_regs(1), link_regs(2), - link_regs(3), - link_regs(4), - link_regs(5), - link_regs(6), + link_regs(3) +}; + +static const struct dcn10_link_enc_shift le_shift = { + LINK_ENCODER_MASK_SH_LIST_DCN10(__SHIFT) +}; + +static const struct dcn10_link_enc_mask le_mask = { + LINK_ENCODER_MASK_SH_LIST_DCN10(_MASK) }; #define ipp_regs(id)\ @@ -320,11 +307,14 @@ static const struct dcn_dpp_registers tf_regs[] = { }; static const struct dcn_dpp_shift tf_shift = { - TF_REG_LIST_SH_MASK_DCN10(__SHIFT) + TF_REG_LIST_SH_MASK_DCN10(__SHIFT), + TF_DEBUG_REG_LIST_SH_DCN10 + }; static const struct dcn_dpp_mask tf_mask = { TF_REG_LIST_SH_MASK_DCN10(_MASK), + TF_DEBUG_REG_LIST_MASK_DCN10 }; static const struct dcn_mpc_registers mpc_regs = { @@ -457,6 +447,8 @@ static const struct dc_debug debug_defaults_drv = { .vsr_support = true, .performance_trace = false, .az_endpoint_mute_only = true, + .recovery_enabled = false, /*enable this by default after testing.*/ + .max_downscale_src_width = 3840, }; static const struct dc_debug debug_defaults_diags = { @@ -592,20 +584,22 @@ static const struct encoder_feature_support link_enc_feature = { struct link_encoder *dcn10_link_encoder_create( const struct encoder_init_data *enc_init_data) { - struct dce110_link_encoder *enc110 = - kzalloc(sizeof(struct dce110_link_encoder), GFP_KERNEL); + struct dcn10_link_encoder *enc10 = + kzalloc(sizeof(struct dcn10_link_encoder), GFP_KERNEL); - if (!enc110) + if (!enc10) return NULL; - dce110_link_encoder_construct(enc110, + dcn10_link_encoder_construct(enc10, enc_init_data, &link_enc_feature, &link_enc_regs[enc_init_data->transmitter], &link_enc_aux_regs[enc_init_data->channel - 1], - &link_enc_hpd_regs[enc_init_data->hpd_source]); + &link_enc_hpd_regs[enc_init_data->hpd_source], + &le_shift, + &le_mask); - return &enc110->base; + return &enc10->base; } struct clock_source *dcn10_clock_source_create( @@ -650,16 +644,16 @@ static struct stream_encoder *dcn10_stream_encoder_create( enum engine_id eng_id, struct dc_context *ctx) { - struct dce110_stream_encoder *enc110 = - kzalloc(sizeof(struct dce110_stream_encoder), GFP_KERNEL); + struct dcn10_stream_encoder *enc1 = + kzalloc(sizeof(struct dcn10_stream_encoder), GFP_KERNEL); - if (!enc110) + if (!enc1) return NULL; - dce110_stream_encoder_construct(enc110, ctx, ctx->dc_bios, eng_id, + dcn10_stream_encoder_construct(enc1, ctx, ctx->dc_bios, eng_id, &stream_enc_regs[eng_id], &se_shift, &se_mask); - return &enc110->base; + return &enc1->base; } static const struct dce_hwseq_registers hwseq_reg = { @@ -918,36 +912,6 @@ enum dc_status dcn10_add_stream_to_ctx( return result; } -enum dc_status dcn10_validate_guaranteed( - struct dc *dc, - struct dc_stream_state *dc_stream, - struct dc_state *context) -{ - enum dc_status result = DC_ERROR_UNEXPECTED; - - context->streams[0] = dc_stream; - dc_stream_retain(context->streams[0]); - context->stream_count++; - - result = resource_map_pool_resources(dc, context, dc_stream); - - if (result == DC_OK) - result = resource_map_phy_clock_resources(dc, context, dc_stream); - - if (result == DC_OK) - result = build_mapped_resource(dc, context, dc_stream); - - if (result == DC_OK) { - validate_guaranteed_copy_streams( - context, dc->caps.max_streams); - result = resource_build_scaling_params_for_context(dc, context); - } - if (result == DC_OK && !dcn_validate_bandwidth(dc, context)) - return DC_FAIL_BANDWIDTH_VALIDATE; - - return result; -} - static struct pipe_ctx *dcn10_acquire_idle_pipe_for_layer( struct dc_state *context, const struct resource_pool *pool, @@ -978,235 +942,16 @@ static struct pipe_ctx *dcn10_acquire_idle_pipe_for_layer( return idle_pipe; } -enum dcc_control { - dcc_control__256_256_xxx, - dcc_control__128_128_xxx, - dcc_control__256_64_64, -}; - -enum segment_order { - segment_order__na, - segment_order__contiguous, - segment_order__non_contiguous, -}; - -static bool dcc_support_pixel_format( - enum surface_pixel_format format, - unsigned int *bytes_per_element) -{ - /* DML: get_bytes_per_element */ - switch (format) { - case SURFACE_PIXEL_FORMAT_GRPH_ARGB1555: - case SURFACE_PIXEL_FORMAT_GRPH_RGB565: - *bytes_per_element = 2; - return true; - case SURFACE_PIXEL_FORMAT_GRPH_ARGB8888: - case SURFACE_PIXEL_FORMAT_GRPH_ABGR8888: - case SURFACE_PIXEL_FORMAT_GRPH_ARGB2101010: - case SURFACE_PIXEL_FORMAT_GRPH_ABGR2101010: - *bytes_per_element = 4; - return true; - case SURFACE_PIXEL_FORMAT_GRPH_ARGB16161616: - case SURFACE_PIXEL_FORMAT_GRPH_ARGB16161616F: - case SURFACE_PIXEL_FORMAT_GRPH_ABGR16161616F: - *bytes_per_element = 8; - return true; - default: - return false; - } -} - -static bool dcc_support_swizzle( - enum swizzle_mode_values swizzle, - unsigned int bytes_per_element, - enum segment_order *segment_order_horz, - enum segment_order *segment_order_vert) -{ - bool standard_swizzle = false; - bool display_swizzle = false; - - switch (swizzle) { - case DC_SW_4KB_S: - case DC_SW_64KB_S: - case DC_SW_VAR_S: - case DC_SW_4KB_S_X: - case DC_SW_64KB_S_X: - case DC_SW_VAR_S_X: - standard_swizzle = true; - break; - case DC_SW_4KB_D: - case DC_SW_64KB_D: - case DC_SW_VAR_D: - case DC_SW_4KB_D_X: - case DC_SW_64KB_D_X: - case DC_SW_VAR_D_X: - display_swizzle = true; - break; - default: - break; - } - - if (bytes_per_element == 1 && standard_swizzle) { - *segment_order_horz = segment_order__contiguous; - *segment_order_vert = segment_order__na; - return true; - } - if (bytes_per_element == 2 && standard_swizzle) { - *segment_order_horz = segment_order__non_contiguous; - *segment_order_vert = segment_order__contiguous; - return true; - } - if (bytes_per_element == 4 && standard_swizzle) { - *segment_order_horz = segment_order__non_contiguous; - *segment_order_vert = segment_order__contiguous; - return true; - } - if (bytes_per_element == 8 && standard_swizzle) { - *segment_order_horz = segment_order__na; - *segment_order_vert = segment_order__contiguous; - return true; - } - if (bytes_per_element == 8 && display_swizzle) { - *segment_order_horz = segment_order__contiguous; - *segment_order_vert = segment_order__non_contiguous; - return true; - } - - return false; -} - -static void get_blk256_size(unsigned int *blk256_width, unsigned int *blk256_height, - unsigned int bytes_per_element) -{ - /* copied from DML. might want to refactor DML to leverage from DML */ - /* DML : get_blk256_size */ - if (bytes_per_element == 1) { - *blk256_width = 16; - *blk256_height = 16; - } else if (bytes_per_element == 2) { - *blk256_width = 16; - *blk256_height = 8; - } else if (bytes_per_element == 4) { - *blk256_width = 8; - *blk256_height = 8; - } else if (bytes_per_element == 8) { - *blk256_width = 8; - *blk256_height = 4; - } -} - -static void det_request_size( - unsigned int height, - unsigned int width, - unsigned int bpe, - bool *req128_horz_wc, - bool *req128_vert_wc) -{ - unsigned int detile_buf_size = 164 * 1024; /* 164KB for DCN1.0 */ - - unsigned int blk256_height = 0; - unsigned int blk256_width = 0; - unsigned int swath_bytes_horz_wc, swath_bytes_vert_wc; - - get_blk256_size(&blk256_width, &blk256_height, bpe); - - swath_bytes_horz_wc = height * blk256_height * bpe; - swath_bytes_vert_wc = width * blk256_width * bpe; - - *req128_horz_wc = (2 * swath_bytes_horz_wc <= detile_buf_size) ? - false : /* full 256B request */ - true; /* half 128b request */ - - *req128_vert_wc = (2 * swath_bytes_vert_wc <= detile_buf_size) ? - false : /* full 256B request */ - true; /* half 128b request */ -} - -static bool get_dcc_compression_cap(const struct dc *dc, +static bool dcn10_get_dcc_compression_cap(const struct dc *dc, const struct dc_dcc_surface_param *input, struct dc_surface_dcc_cap *output) { - /* implement section 1.6.2.1 of DCN1_Programming_Guide.docx */ - enum dcc_control dcc_control; - unsigned int bpe; - enum segment_order segment_order_horz, segment_order_vert; - bool req128_horz_wc, req128_vert_wc; - - memset(output, 0, sizeof(*output)); - - if (dc->debug.disable_dcc == DCC_DISABLE) - return false; - - if (!dcc_support_pixel_format(input->format, - &bpe)) - return false; - - if (!dcc_support_swizzle(input->swizzle_mode, bpe, - &segment_order_horz, &segment_order_vert)) - return false; - - det_request_size(input->surface_size.height, input->surface_size.width, - bpe, &req128_horz_wc, &req128_vert_wc); - - if (!req128_horz_wc && !req128_vert_wc) { - dcc_control = dcc_control__256_256_xxx; - } else if (input->scan == SCAN_DIRECTION_HORIZONTAL) { - if (!req128_horz_wc) - dcc_control = dcc_control__256_256_xxx; - else if (segment_order_horz == segment_order__contiguous) - dcc_control = dcc_control__128_128_xxx; - else - dcc_control = dcc_control__256_64_64; - } else if (input->scan == SCAN_DIRECTION_VERTICAL) { - if (!req128_vert_wc) - dcc_control = dcc_control__256_256_xxx; - else if (segment_order_vert == segment_order__contiguous) - dcc_control = dcc_control__128_128_xxx; - else - dcc_control = dcc_control__256_64_64; - } else { - if ((req128_horz_wc && - segment_order_horz == segment_order__non_contiguous) || - (req128_vert_wc && - segment_order_vert == segment_order__non_contiguous)) - /* access_dir not known, must use most constraining */ - dcc_control = dcc_control__256_64_64; - else - /* reg128 is true for either horz and vert - * but segment_order is contiguous - */ - dcc_control = dcc_control__128_128_xxx; - } - - if (dc->debug.disable_dcc == DCC_HALF_REQ_DISALBE && - dcc_control != dcc_control__256_256_xxx) - return false; - - switch (dcc_control) { - case dcc_control__256_256_xxx: - output->grph.rgb.max_uncompressed_blk_size = 256; - output->grph.rgb.max_compressed_blk_size = 256; - output->grph.rgb.independent_64b_blks = false; - break; - case dcc_control__128_128_xxx: - output->grph.rgb.max_uncompressed_blk_size = 128; - output->grph.rgb.max_compressed_blk_size = 128; - output->grph.rgb.independent_64b_blks = false; - break; - case dcc_control__256_64_64: - output->grph.rgb.max_uncompressed_blk_size = 256; - output->grph.rgb.max_compressed_blk_size = 64; - output->grph.rgb.independent_64b_blks = true; - break; - } - - output->capable = true; - output->const_color_support = false; - - return true; + return dc->res_pool->hubbub->funcs->get_dcc_compression_cap( + dc->res_pool->hubbub, + input, + output); } - static void dcn10_destroy_resource_pool(struct resource_pool **pool) { struct dcn10_resource_pool *dcn10_pool = TO_DCN10_RES_POOL(*pool); @@ -1227,13 +972,12 @@ static enum dc_status dcn10_validate_plane(const struct dc_plane_state *plane_st } static struct dc_cap_funcs cap_funcs = { - .get_dcc_compression_cap = get_dcc_compression_cap + .get_dcc_compression_cap = dcn10_get_dcc_compression_cap }; static struct resource_funcs dcn10_res_pool_funcs = { .destroy = dcn10_destroy_resource_pool, .link_enc_create = dcn10_link_encoder_create, - .validate_guaranteed = dcn10_validate_guaranteed, .validate_bandwidth = dcn_validate_bandwidth, .acquire_idle_pipe_for_layer = dcn10_acquire_idle_pipe_for_layer, .validate_plane = dcn10_validate_plane, @@ -1282,6 +1026,7 @@ static bool construct( dc->caps.max_cursor_size = 256; dc->caps.max_slave_planes = 1; dc->caps.is_apu = true; + dc->caps.post_blend_color_processing = false; if (dc->ctx->dce_environment == DCE_ENV_PRODUCTION_DRV) dc->debug = debug_defaults_drv; diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_stream_encoder.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_stream_encoder.c new file mode 100644 index 000000000000..c928ee4cd382 --- /dev/null +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_stream_encoder.c @@ -0,0 +1,1494 @@ +/* + * Copyright 2012-15 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * Authors: AMD + * + */ + + +#include "dc_bios_types.h" +#include "dcn10_stream_encoder.h" +#include "reg_helper.h" +#include "hw_shared.h" + +#define DC_LOGGER \ + enc1->base.ctx->logger + + +#define REG(reg)\ + (enc1->regs->reg) + +#undef FN +#define FN(reg_name, field_name) \ + enc1->se_shift->field_name, enc1->se_mask->field_name + +#define VBI_LINE_0 0 +#define DP_BLANK_MAX_RETRY 20 +#define HDMI_CLOCK_CHANNEL_RATE_MORE_340M 340000 + + +enum { + DP_MST_UPDATE_MAX_RETRY = 50 +}; + +#define CTX \ + enc1->base.ctx + +void enc1_update_generic_info_packet( + struct dcn10_stream_encoder *enc1, + uint32_t packet_index, + const struct dc_info_packet *info_packet) +{ + uint32_t regval; + /* TODOFPGA Figure out a proper number for max_retries polling for lock + * use 50 for now. + */ + uint32_t max_retries = 50; + + /*we need turn on clock before programming AFMT block*/ + REG_UPDATE(AFMT_CNTL, AFMT_AUDIO_CLOCK_EN, 1); + + if (packet_index >= 8) + ASSERT(0); + + /* poll dig_update_lock is not locked -> asic internal signal + * assume otg master lock will unlock it + */ +/* REG_WAIT(AFMT_VBI_PACKET_CONTROL, AFMT_GENERIC_LOCK_STATUS, + 0, 10, max_retries);*/ + + /* check if HW reading GSP memory */ + REG_WAIT(AFMT_VBI_PACKET_CONTROL, AFMT_GENERIC_CONFLICT, + 0, 10, max_retries); + + /* HW does is not reading GSP memory not reading too long -> + * something wrong. clear GPS memory access and notify? + * hw SW is writing to GSP memory + */ + REG_UPDATE(AFMT_VBI_PACKET_CONTROL, AFMT_GENERIC_CONFLICT_CLR, 1); + + /* choose which generic packet to use */ + regval = REG_READ(AFMT_VBI_PACKET_CONTROL); + REG_UPDATE(AFMT_VBI_PACKET_CONTROL, + AFMT_GENERIC_INDEX, packet_index); + + /* write generic packet header + * (4th byte is for GENERIC0 only) + */ + REG_SET_4(AFMT_GENERIC_HDR, 0, + AFMT_GENERIC_HB0, info_packet->hb0, + AFMT_GENERIC_HB1, info_packet->hb1, + AFMT_GENERIC_HB2, info_packet->hb2, + AFMT_GENERIC_HB3, info_packet->hb3); + + /* write generic packet contents + * (we never use last 4 bytes) + * there are 8 (0-7) mmDIG0_AFMT_GENERIC0_x registers + */ + { + const uint32_t *content = + (const uint32_t *) &info_packet->sb[0]; + + REG_WRITE(AFMT_GENERIC_0, *content++); + REG_WRITE(AFMT_GENERIC_1, *content++); + REG_WRITE(AFMT_GENERIC_2, *content++); + REG_WRITE(AFMT_GENERIC_3, *content++); + REG_WRITE(AFMT_GENERIC_4, *content++); + REG_WRITE(AFMT_GENERIC_5, *content++); + REG_WRITE(AFMT_GENERIC_6, *content++); + REG_WRITE(AFMT_GENERIC_7, *content); + } + + switch (packet_index) { + case 0: + REG_UPDATE(AFMT_VBI_PACKET_CONTROL1, + AFMT_GENERIC0_FRAME_UPDATE, 1); + break; + case 1: + REG_UPDATE(AFMT_VBI_PACKET_CONTROL1, + AFMT_GENERIC1_FRAME_UPDATE, 1); + break; + case 2: + REG_UPDATE(AFMT_VBI_PACKET_CONTROL1, + AFMT_GENERIC2_FRAME_UPDATE, 1); + break; + case 3: + REG_UPDATE(AFMT_VBI_PACKET_CONTROL1, + AFMT_GENERIC3_FRAME_UPDATE, 1); + break; + case 4: + REG_UPDATE(AFMT_VBI_PACKET_CONTROL1, + AFMT_GENERIC4_FRAME_UPDATE, 1); + break; + case 5: + REG_UPDATE(AFMT_VBI_PACKET_CONTROL1, + AFMT_GENERIC5_FRAME_UPDATE, 1); + break; + case 6: + REG_UPDATE(AFMT_VBI_PACKET_CONTROL1, + AFMT_GENERIC6_FRAME_UPDATE, 1); + break; + case 7: + REG_UPDATE(AFMT_VBI_PACKET_CONTROL1, + AFMT_GENERIC7_FRAME_UPDATE, 1); + break; + default: + break; + } +} + +static void enc1_update_hdmi_info_packet( + struct dcn10_stream_encoder *enc1, + uint32_t packet_index, + const struct dc_info_packet *info_packet) +{ + uint32_t cont, send, line; + + if (info_packet->valid) { + enc1_update_generic_info_packet( + enc1, + packet_index, + info_packet); + + /* enable transmission of packet(s) - + * packet transmission begins on the next frame + */ + cont = 1; + /* send packet(s) every frame */ + send = 1; + /* select line number to send packets on */ + line = 2; + } else { + cont = 0; + send = 0; + line = 0; + } + + /* choose which generic packet control to use */ + switch (packet_index) { + case 0: + REG_UPDATE_3(HDMI_GENERIC_PACKET_CONTROL0, + HDMI_GENERIC0_CONT, cont, + HDMI_GENERIC0_SEND, send, + HDMI_GENERIC0_LINE, line); + break; + case 1: + REG_UPDATE_3(HDMI_GENERIC_PACKET_CONTROL0, + HDMI_GENERIC1_CONT, cont, + HDMI_GENERIC1_SEND, send, + HDMI_GENERIC1_LINE, line); + break; + case 2: + REG_UPDATE_3(HDMI_GENERIC_PACKET_CONTROL1, + HDMI_GENERIC0_CONT, cont, + HDMI_GENERIC0_SEND, send, + HDMI_GENERIC0_LINE, line); + break; + case 3: + REG_UPDATE_3(HDMI_GENERIC_PACKET_CONTROL1, + HDMI_GENERIC1_CONT, cont, + HDMI_GENERIC1_SEND, send, + HDMI_GENERIC1_LINE, line); + break; + case 4: + REG_UPDATE_3(HDMI_GENERIC_PACKET_CONTROL2, + HDMI_GENERIC0_CONT, cont, + HDMI_GENERIC0_SEND, send, + HDMI_GENERIC0_LINE, line); + break; + case 5: + REG_UPDATE_3(HDMI_GENERIC_PACKET_CONTROL2, + HDMI_GENERIC1_CONT, cont, + HDMI_GENERIC1_SEND, send, + HDMI_GENERIC1_LINE, line); + break; + case 6: + REG_UPDATE_3(HDMI_GENERIC_PACKET_CONTROL3, + HDMI_GENERIC0_CONT, cont, + HDMI_GENERIC0_SEND, send, + HDMI_GENERIC0_LINE, line); + break; + case 7: + REG_UPDATE_3(HDMI_GENERIC_PACKET_CONTROL3, + HDMI_GENERIC1_CONT, cont, + HDMI_GENERIC1_SEND, send, + HDMI_GENERIC1_LINE, line); + break; + default: + /* invalid HW packet index */ + DC_LOG_WARNING( + "Invalid HW packet index: %s()\n", + __func__); + return; + } +} + +/* setup stream encoder in dp mode */ +void enc1_stream_encoder_dp_set_stream_attribute( + struct stream_encoder *enc, + struct dc_crtc_timing *crtc_timing, + enum dc_color_space output_color_space) +{ + uint32_t h_active_start; + uint32_t v_active_start; + uint32_t misc0 = 0; + uint32_t misc1 = 0; + uint32_t h_blank; + uint32_t h_back_porch; + uint8_t synchronous_clock = 0; /* asynchronous mode */ + uint8_t colorimetry_bpc; + uint8_t dynamic_range_rgb = 0; /*full range*/ + uint8_t dynamic_range_ycbcr = 1; /*bt709*/ + + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + + REG_UPDATE(DP_DB_CNTL, DP_DB_DISABLE, 1); + + /* set pixel encoding */ + switch (crtc_timing->pixel_encoding) { + case PIXEL_ENCODING_YCBCR422: + REG_UPDATE(DP_PIXEL_FORMAT, DP_PIXEL_ENCODING, + DP_PIXEL_ENCODING_TYPE_YCBCR422); + break; + case PIXEL_ENCODING_YCBCR444: + REG_UPDATE(DP_PIXEL_FORMAT, DP_PIXEL_ENCODING, + DP_PIXEL_ENCODING_TYPE_YCBCR444); + + if (crtc_timing->flags.Y_ONLY) + if (crtc_timing->display_color_depth != COLOR_DEPTH_666) + /* HW testing only, no use case yet. + * Color depth of Y-only could be + * 8, 10, 12, 16 bits + */ + REG_UPDATE(DP_PIXEL_FORMAT, DP_PIXEL_ENCODING, + DP_PIXEL_ENCODING_TYPE_Y_ONLY); + /* Note: DP_MSA_MISC1 bit 7 is the indicator + * of Y-only mode. + * This bit is set in HW if register + * DP_PIXEL_ENCODING is programmed to 0x4 + */ + break; + case PIXEL_ENCODING_YCBCR420: + REG_UPDATE(DP_PIXEL_FORMAT, DP_PIXEL_ENCODING, + DP_PIXEL_ENCODING_TYPE_YCBCR420); + REG_UPDATE(DP_VID_TIMING, DP_VID_N_MUL, 1); + break; + default: + REG_UPDATE(DP_PIXEL_FORMAT, DP_PIXEL_ENCODING, + DP_PIXEL_ENCODING_TYPE_RGB444); + break; + } + + misc1 = REG_READ(DP_MSA_MISC); + + /* set color depth */ + + switch (crtc_timing->display_color_depth) { + case COLOR_DEPTH_666: + REG_UPDATE(DP_PIXEL_FORMAT, DP_COMPONENT_DEPTH, + 0); + break; + case COLOR_DEPTH_888: + REG_UPDATE(DP_PIXEL_FORMAT, DP_COMPONENT_DEPTH, + DP_COMPONENT_PIXEL_DEPTH_8BPC); + break; + case COLOR_DEPTH_101010: + REG_UPDATE(DP_PIXEL_FORMAT, DP_COMPONENT_DEPTH, + DP_COMPONENT_PIXEL_DEPTH_10BPC); + + break; + case COLOR_DEPTH_121212: + REG_UPDATE(DP_PIXEL_FORMAT, DP_COMPONENT_DEPTH, + DP_COMPONENT_PIXEL_DEPTH_12BPC); + break; + case COLOR_DEPTH_161616: + REG_UPDATE(DP_PIXEL_FORMAT, DP_COMPONENT_DEPTH, + DP_COMPONENT_PIXEL_DEPTH_16BPC); + break; + default: + REG_UPDATE(DP_PIXEL_FORMAT, DP_COMPONENT_DEPTH, + DP_COMPONENT_PIXEL_DEPTH_6BPC); + break; + } + + /* set dynamic range and YCbCr range */ + + switch (crtc_timing->display_color_depth) { + case COLOR_DEPTH_666: + colorimetry_bpc = 0; + break; + case COLOR_DEPTH_888: + colorimetry_bpc = 1; + break; + case COLOR_DEPTH_101010: + colorimetry_bpc = 2; + break; + case COLOR_DEPTH_121212: + colorimetry_bpc = 3; + break; + default: + colorimetry_bpc = 0; + break; + } + + misc0 = misc0 | synchronous_clock; + misc0 = colorimetry_bpc << 5; + + switch (output_color_space) { + case COLOR_SPACE_SRGB: + misc0 = misc0 | 0x0; + misc1 = misc1 & ~0x80; /* bit7 = 0*/ + dynamic_range_rgb = 0; /*full range*/ + break; + case COLOR_SPACE_SRGB_LIMITED: + misc0 = misc0 | 0x8; /* bit3=1 */ + misc1 = misc1 & ~0x80; /* bit7 = 0*/ + dynamic_range_rgb = 1; /*limited range*/ + break; + case COLOR_SPACE_YCBCR601: + case COLOR_SPACE_YCBCR601_LIMITED: + misc0 = misc0 | 0x8; /* bit3=1, bit4=0 */ + misc1 = misc1 & ~0x80; /* bit7 = 0*/ + dynamic_range_ycbcr = 0; /*bt601*/ + if (crtc_timing->pixel_encoding == PIXEL_ENCODING_YCBCR422) + misc0 = misc0 | 0x2; /* bit2=0, bit1=1 */ + else if (crtc_timing->pixel_encoding == PIXEL_ENCODING_YCBCR444) + misc0 = misc0 | 0x4; /* bit2=1, bit1=0 */ + break; + case COLOR_SPACE_YCBCR709: + case COLOR_SPACE_YCBCR709_LIMITED: + misc0 = misc0 | 0x18; /* bit3=1, bit4=1 */ + misc1 = misc1 & ~0x80; /* bit7 = 0*/ + dynamic_range_ycbcr = 1; /*bt709*/ + if (crtc_timing->pixel_encoding == PIXEL_ENCODING_YCBCR422) + misc0 = misc0 | 0x2; /* bit2=0, bit1=1 */ + else if (crtc_timing->pixel_encoding == PIXEL_ENCODING_YCBCR444) + misc0 = misc0 | 0x4; /* bit2=1, bit1=0 */ + break; + case COLOR_SPACE_2020_RGB_LIMITEDRANGE: + dynamic_range_rgb = 1; /*limited range*/ + break; + case COLOR_SPACE_2020_RGB_FULLRANGE: + case COLOR_SPACE_2020_YCBCR: + case COLOR_SPACE_XR_RGB: + case COLOR_SPACE_MSREF_SCRGB: + case COLOR_SPACE_ADOBERGB: + case COLOR_SPACE_DCIP3: + case COLOR_SPACE_XV_YCC_709: + case COLOR_SPACE_XV_YCC_601: + case COLOR_SPACE_DISPLAYNATIVE: + case COLOR_SPACE_DOLBYVISION: + case COLOR_SPACE_APPCTRL: + case COLOR_SPACE_CUSTOMPOINTS: + case COLOR_SPACE_UNKNOWN: + /* do nothing */ + break; + } + + REG_SET(DP_MSA_COLORIMETRY, 0, DP_MSA_MISC0, misc0); + REG_WRITE(DP_MSA_MISC, misc1); /* MSA_MISC1 */ + + /* dcn new register + * dc_crtc_timing is vesa dmt struct. data from edid + */ + REG_SET_2(DP_MSA_TIMING_PARAM1, 0, + DP_MSA_HTOTAL, crtc_timing->h_total, + DP_MSA_VTOTAL, crtc_timing->v_total); + + /* calculate from vesa timing parameters + * h_active_start related to leading edge of sync + */ + + h_blank = crtc_timing->h_total - crtc_timing->h_border_left - + crtc_timing->h_addressable - crtc_timing->h_border_right; + + h_back_porch = h_blank - crtc_timing->h_front_porch - + crtc_timing->h_sync_width; + + /* start at beginning of left border */ + h_active_start = crtc_timing->h_sync_width + h_back_porch; + + + v_active_start = crtc_timing->v_total - crtc_timing->v_border_top - + crtc_timing->v_addressable - crtc_timing->v_border_bottom - + crtc_timing->v_front_porch; + + + /* start at beginning of left border */ + REG_SET_2(DP_MSA_TIMING_PARAM2, 0, + DP_MSA_HSTART, h_active_start, + DP_MSA_VSTART, v_active_start); + + REG_SET_4(DP_MSA_TIMING_PARAM3, 0, + DP_MSA_HSYNCWIDTH, + crtc_timing->h_sync_width, + DP_MSA_HSYNCPOLARITY, + !crtc_timing->flags.HSYNC_POSITIVE_POLARITY, + DP_MSA_VSYNCWIDTH, + crtc_timing->v_sync_width, + DP_MSA_VSYNCPOLARITY, + !crtc_timing->flags.VSYNC_POSITIVE_POLARITY); + + /* HWDITH include border or overscan */ + REG_SET_2(DP_MSA_TIMING_PARAM4, 0, + DP_MSA_HWIDTH, crtc_timing->h_border_left + + crtc_timing->h_addressable + crtc_timing->h_border_right, + DP_MSA_VHEIGHT, crtc_timing->v_border_top + + crtc_timing->v_addressable + crtc_timing->v_border_bottom); +} + +static void enc1_stream_encoder_set_stream_attribute_helper( + struct dcn10_stream_encoder *enc1, + struct dc_crtc_timing *crtc_timing) +{ + switch (crtc_timing->pixel_encoding) { + case PIXEL_ENCODING_YCBCR422: + REG_UPDATE(DIG_FE_CNTL, TMDS_PIXEL_ENCODING, 1); + break; + default: + REG_UPDATE(DIG_FE_CNTL, TMDS_PIXEL_ENCODING, 0); + break; + } + REG_UPDATE(DIG_FE_CNTL, TMDS_COLOR_FORMAT, 0); +} + +/* setup stream encoder in hdmi mode */ +void enc1_stream_encoder_hdmi_set_stream_attribute( + struct stream_encoder *enc, + struct dc_crtc_timing *crtc_timing, + int actual_pix_clk_khz, + bool enable_audio) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + struct bp_encoder_control cntl = {0}; + + cntl.action = ENCODER_CONTROL_SETUP; + cntl.engine_id = enc1->base.id; + cntl.signal = SIGNAL_TYPE_HDMI_TYPE_A; + cntl.enable_dp_audio = enable_audio; + cntl.pixel_clock = actual_pix_clk_khz; + cntl.lanes_number = LANE_COUNT_FOUR; + + if (enc1->base.bp->funcs->encoder_control( + enc1->base.bp, &cntl) != BP_RESULT_OK) + return; + + enc1_stream_encoder_set_stream_attribute_helper(enc1, crtc_timing); + + /* setup HDMI engine */ + REG_UPDATE_5(HDMI_CONTROL, + HDMI_PACKET_GEN_VERSION, 1, + HDMI_KEEPOUT_MODE, 1, + HDMI_DEEP_COLOR_ENABLE, 0, + HDMI_DATA_SCRAMBLE_EN, 0, + HDMI_CLOCK_CHANNEL_RATE, 0); + + + switch (crtc_timing->display_color_depth) { + case COLOR_DEPTH_888: + REG_UPDATE(HDMI_CONTROL, HDMI_DEEP_COLOR_DEPTH, 0); + break; + case COLOR_DEPTH_101010: + if (crtc_timing->pixel_encoding == PIXEL_ENCODING_YCBCR422) { + REG_UPDATE_2(HDMI_CONTROL, + HDMI_DEEP_COLOR_DEPTH, 1, + HDMI_DEEP_COLOR_ENABLE, 0); + } else { + REG_UPDATE_2(HDMI_CONTROL, + HDMI_DEEP_COLOR_DEPTH, 1, + HDMI_DEEP_COLOR_ENABLE, 1); + } + break; + case COLOR_DEPTH_121212: + if (crtc_timing->pixel_encoding == PIXEL_ENCODING_YCBCR422) { + REG_UPDATE_2(HDMI_CONTROL, + HDMI_DEEP_COLOR_DEPTH, 2, + HDMI_DEEP_COLOR_ENABLE, 0); + } else { + REG_UPDATE_2(HDMI_CONTROL, + HDMI_DEEP_COLOR_DEPTH, 2, + HDMI_DEEP_COLOR_ENABLE, 1); + } + break; + case COLOR_DEPTH_161616: + REG_UPDATE_2(HDMI_CONTROL, + HDMI_DEEP_COLOR_DEPTH, 3, + HDMI_DEEP_COLOR_ENABLE, 1); + break; + default: + break; + } + + if (actual_pix_clk_khz >= HDMI_CLOCK_CHANNEL_RATE_MORE_340M) { + /* enable HDMI data scrambler + * HDMI_CLOCK_CHANNEL_RATE_MORE_340M + * Clock channel frequency is 1/4 of character rate. + */ + REG_UPDATE_2(HDMI_CONTROL, + HDMI_DATA_SCRAMBLE_EN, 1, + HDMI_CLOCK_CHANNEL_RATE, 1); + } else if (crtc_timing->flags.LTE_340MCSC_SCRAMBLE) { + + /* TODO: New feature for DCE11, still need to implement */ + + /* enable HDMI data scrambler + * HDMI_CLOCK_CHANNEL_FREQ_EQUAL_TO_CHAR_RATE + * Clock channel frequency is the same + * as character rate + */ + REG_UPDATE_2(HDMI_CONTROL, + HDMI_DATA_SCRAMBLE_EN, 1, + HDMI_CLOCK_CHANNEL_RATE, 0); + } + + + REG_UPDATE_3(HDMI_VBI_PACKET_CONTROL, + HDMI_GC_CONT, 1, + HDMI_GC_SEND, 1, + HDMI_NULL_SEND, 1); + + /* following belongs to audio */ + REG_UPDATE(HDMI_INFOFRAME_CONTROL0, HDMI_AUDIO_INFO_SEND, 1); + + REG_UPDATE(AFMT_INFOFRAME_CONTROL0, AFMT_AUDIO_INFO_UPDATE, 1); + + REG_UPDATE(HDMI_INFOFRAME_CONTROL1, HDMI_AUDIO_INFO_LINE, + VBI_LINE_0 + 2); + + REG_UPDATE(HDMI_GC, HDMI_GC_AVMUTE, 0); +} + +/* setup stream encoder in dvi mode */ +void enc1_stream_encoder_dvi_set_stream_attribute( + struct stream_encoder *enc, + struct dc_crtc_timing *crtc_timing, + bool is_dual_link) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + struct bp_encoder_control cntl = {0}; + + cntl.action = ENCODER_CONTROL_SETUP; + cntl.engine_id = enc1->base.id; + cntl.signal = is_dual_link ? + SIGNAL_TYPE_DVI_DUAL_LINK : SIGNAL_TYPE_DVI_SINGLE_LINK; + cntl.enable_dp_audio = false; + cntl.pixel_clock = crtc_timing->pix_clk_khz; + cntl.lanes_number = (is_dual_link) ? LANE_COUNT_EIGHT : LANE_COUNT_FOUR; + + if (enc1->base.bp->funcs->encoder_control( + enc1->base.bp, &cntl) != BP_RESULT_OK) + return; + + ASSERT(crtc_timing->pixel_encoding == PIXEL_ENCODING_RGB); + ASSERT(crtc_timing->display_color_depth == COLOR_DEPTH_888); + enc1_stream_encoder_set_stream_attribute_helper(enc1, crtc_timing); +} + +void enc1_stream_encoder_set_mst_bandwidth( + struct stream_encoder *enc, + struct fixed31_32 avg_time_slots_per_mtp) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + uint32_t x = dc_fixpt_floor( + avg_time_slots_per_mtp); + uint32_t y = dc_fixpt_ceil( + dc_fixpt_shl( + dc_fixpt_sub_int( + avg_time_slots_per_mtp, + x), + 26)); + + REG_SET_2(DP_MSE_RATE_CNTL, 0, + DP_MSE_RATE_X, x, + DP_MSE_RATE_Y, y); + + /* wait for update to be completed on the link */ + /* i.e. DP_MSE_RATE_UPDATE_PENDING field (read only) */ + /* is reset to 0 (not pending) */ + REG_WAIT(DP_MSE_RATE_UPDATE, DP_MSE_RATE_UPDATE_PENDING, + 0, + 10, DP_MST_UPDATE_MAX_RETRY); +} + +static void enc1_stream_encoder_update_hdmi_info_packets( + struct stream_encoder *enc, + const struct encoder_info_frame *info_frame) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + + /* for bring up, disable dp double TODO */ + REG_UPDATE(HDMI_DB_CONTROL, HDMI_DB_DISABLE, 1); + + enc1_update_hdmi_info_packet(enc1, 0, &info_frame->avi); + enc1_update_hdmi_info_packet(enc1, 1, &info_frame->vendor); + enc1_update_hdmi_info_packet(enc1, 2, &info_frame->gamut); + enc1_update_hdmi_info_packet(enc1, 3, &info_frame->spd); + enc1_update_hdmi_info_packet(enc1, 4, &info_frame->hdrsmd); +} + +static void enc1_stream_encoder_stop_hdmi_info_packets( + struct stream_encoder *enc) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + + /* stop generic packets 0 & 1 on HDMI */ + REG_SET_6(HDMI_GENERIC_PACKET_CONTROL0, 0, + HDMI_GENERIC1_CONT, 0, + HDMI_GENERIC1_LINE, 0, + HDMI_GENERIC1_SEND, 0, + HDMI_GENERIC0_CONT, 0, + HDMI_GENERIC0_LINE, 0, + HDMI_GENERIC0_SEND, 0); + + /* stop generic packets 2 & 3 on HDMI */ + REG_SET_6(HDMI_GENERIC_PACKET_CONTROL1, 0, + HDMI_GENERIC0_CONT, 0, + HDMI_GENERIC0_LINE, 0, + HDMI_GENERIC0_SEND, 0, + HDMI_GENERIC1_CONT, 0, + HDMI_GENERIC1_LINE, 0, + HDMI_GENERIC1_SEND, 0); + + /* stop generic packets 2 & 3 on HDMI */ + REG_SET_6(HDMI_GENERIC_PACKET_CONTROL2, 0, + HDMI_GENERIC0_CONT, 0, + HDMI_GENERIC0_LINE, 0, + HDMI_GENERIC0_SEND, 0, + HDMI_GENERIC1_CONT, 0, + HDMI_GENERIC1_LINE, 0, + HDMI_GENERIC1_SEND, 0); + + REG_SET_6(HDMI_GENERIC_PACKET_CONTROL3, 0, + HDMI_GENERIC0_CONT, 0, + HDMI_GENERIC0_LINE, 0, + HDMI_GENERIC0_SEND, 0, + HDMI_GENERIC1_CONT, 0, + HDMI_GENERIC1_LINE, 0, + HDMI_GENERIC1_SEND, 0); +} + +void enc1_stream_encoder_update_dp_info_packets( + struct stream_encoder *enc, + const struct encoder_info_frame *info_frame) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + uint32_t value = 0; + + if (info_frame->vsc.valid) + enc1_update_generic_info_packet( + enc1, + 0, /* packetIndex */ + &info_frame->vsc); + + if (info_frame->spd.valid) + enc1_update_generic_info_packet( + enc1, + 2, /* packetIndex */ + &info_frame->spd); + + if (info_frame->hdrsmd.valid) + enc1_update_generic_info_packet( + enc1, + 3, /* packetIndex */ + &info_frame->hdrsmd); + + /* enable/disable transmission of packet(s). + * If enabled, packet transmission begins on the next frame + */ + REG_UPDATE(DP_SEC_CNTL, DP_SEC_GSP0_ENABLE, info_frame->vsc.valid); + REG_UPDATE(DP_SEC_CNTL, DP_SEC_GSP2_ENABLE, info_frame->spd.valid); + REG_UPDATE(DP_SEC_CNTL, DP_SEC_GSP3_ENABLE, info_frame->hdrsmd.valid); + + + /* This bit is the master enable bit. + * When enabling secondary stream engine, + * this master bit must also be set. + * This register shared with audio info frame. + * Therefore we need to enable master bit + * if at least on of the fields is not 0 + */ + value = REG_READ(DP_SEC_CNTL); + if (value) + REG_UPDATE(DP_SEC_CNTL, DP_SEC_STREAM_ENABLE, 1); +} + +void enc1_stream_encoder_stop_dp_info_packets( + struct stream_encoder *enc) +{ + /* stop generic packets on DP */ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + uint32_t value = 0; + + REG_SET_10(DP_SEC_CNTL, 0, + DP_SEC_GSP0_ENABLE, 0, + DP_SEC_GSP1_ENABLE, 0, + DP_SEC_GSP2_ENABLE, 0, + DP_SEC_GSP3_ENABLE, 0, + DP_SEC_GSP4_ENABLE, 0, + DP_SEC_GSP5_ENABLE, 0, + DP_SEC_GSP6_ENABLE, 0, + DP_SEC_GSP7_ENABLE, 0, + DP_SEC_MPG_ENABLE, 0, + DP_SEC_STREAM_ENABLE, 0); + + /* this register shared with audio info frame. + * therefore we need to keep master enabled + * if at least one of the fields is not 0 */ + value = REG_READ(DP_SEC_CNTL); + if (value) + REG_UPDATE(DP_SEC_CNTL, DP_SEC_STREAM_ENABLE, 1); + +} + +void enc1_stream_encoder_dp_blank( + struct stream_encoder *enc) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + uint32_t retries = 0; + uint32_t reg1 = 0; + uint32_t max_retries = DP_BLANK_MAX_RETRY * 10; + + /* Note: For CZ, we are changing driver default to disable + * stream deferred to next VBLANK. If results are positive, we + * will make the same change to all DCE versions. There are a + * handful of panels that cannot handle disable stream at + * HBLANK and will result in a white line flash across the + * screen on stream disable. + */ + REG_GET(DP_VID_STREAM_CNTL, DP_VID_STREAM_ENABLE, ®1); + if ((reg1 & 0x1) == 0) + /*stream not enabled*/ + return; + /* Specify the video stream disable point + * (2 = start of the next vertical blank) + */ + REG_UPDATE(DP_VID_STREAM_CNTL, DP_VID_STREAM_DIS_DEFER, 2); + /* Larger delay to wait until VBLANK - use max retry of + * 10us*3000=30ms. This covers 16.6ms of typical 60 Hz mode + + * a little more because we may not trust delay accuracy. + */ + max_retries = DP_BLANK_MAX_RETRY * 150; + + /* disable DP stream */ + REG_UPDATE(DP_VID_STREAM_CNTL, DP_VID_STREAM_ENABLE, 0); + + /* the encoder stops sending the video stream + * at the start of the vertical blanking. + * Poll for DP_VID_STREAM_STATUS == 0 + */ + + REG_WAIT(DP_VID_STREAM_CNTL, DP_VID_STREAM_STATUS, + 0, + 10, max_retries); + + ASSERT(retries <= max_retries); + + /* Tell the DP encoder to ignore timing from CRTC, must be done after + * the polling. If we set DP_STEER_FIFO_RESET before DP stream blank is + * complete, stream status will be stuck in video stream enabled state, + * i.e. DP_VID_STREAM_STATUS stuck at 1. + */ + + REG_UPDATE(DP_STEER_FIFO, DP_STEER_FIFO_RESET, true); +} + +/* output video stream to link encoder */ +void enc1_stream_encoder_dp_unblank( + struct stream_encoder *enc, + const struct encoder_unblank_param *param) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + + if (param->link_settings.link_rate != LINK_RATE_UNKNOWN) { + uint32_t n_vid = 0x8000; + uint32_t m_vid; + + /* M / N = Fstream / Flink + * m_vid / n_vid = pixel rate / link rate + */ + + uint64_t m_vid_l = n_vid; + + m_vid_l *= param->pixel_clk_khz; + m_vid_l = div_u64(m_vid_l, + param->link_settings.link_rate + * LINK_RATE_REF_FREQ_IN_KHZ); + + m_vid = (uint32_t) m_vid_l; + + /* enable auto measurement */ + + REG_UPDATE(DP_VID_TIMING, DP_VID_M_N_GEN_EN, 0); + + /* auto measurement need 1 full 0x8000 symbol cycle to kick in, + * therefore program initial value for Mvid and Nvid + */ + + REG_UPDATE(DP_VID_N, DP_VID_N, n_vid); + + REG_UPDATE(DP_VID_M, DP_VID_M, m_vid); + + REG_UPDATE(DP_VID_TIMING, DP_VID_M_N_GEN_EN, 1); + } + + /* set DIG_START to 0x1 to resync FIFO */ + + REG_UPDATE(DIG_FE_CNTL, DIG_START, 1); + + /* switch DP encoder to CRTC data */ + + REG_UPDATE(DP_STEER_FIFO, DP_STEER_FIFO_RESET, 0); + + /* wait 100us for DIG/DP logic to prime + * (i.e. a few video lines) + */ + udelay(100); + + /* the hardware would start sending video at the start of the next DP + * frame (i.e. rising edge of the vblank). + * NOTE: We used to program DP_VID_STREAM_DIS_DEFER = 2 here, but this + * register has no effect on enable transition! HW always guarantees + * VID_STREAM enable at start of next frame, and this is not + * programmable + */ + + REG_UPDATE(DP_VID_STREAM_CNTL, DP_VID_STREAM_ENABLE, true); +} + +void enc1_stream_encoder_set_avmute( + struct stream_encoder *enc, + bool enable) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + unsigned int value = enable ? 1 : 0; + + REG_UPDATE(HDMI_GC, HDMI_GC_AVMUTE, value); +} + + +#define DP_SEC_AUD_N__DP_SEC_AUD_N__DEFAULT 0x8000 +#define DP_SEC_TIMESTAMP__DP_SEC_TIMESTAMP_MODE__AUTO_CALC 1 + +#include "include/audio_types.h" + +/** +* speakersToChannels +* +* @brief +* translate speakers to channels +* +* FL - Front Left +* FR - Front Right +* RL - Rear Left +* RR - Rear Right +* RC - Rear Center +* FC - Front Center +* FLC - Front Left Center +* FRC - Front Right Center +* RLC - Rear Left Center +* RRC - Rear Right Center +* LFE - Low Freq Effect +* +* FC +* FLC FRC +* FL FR +* +* LFE +* () +* +* +* RL RR +* RLC RRC +* RC +* +* ch 8 7 6 5 4 3 2 1 +* 0b00000011 - - - - - - FR FL +* 0b00000111 - - - - - LFE FR FL +* 0b00001011 - - - - FC - FR FL +* 0b00001111 - - - - FC LFE FR FL +* 0b00010011 - - - RC - - FR FL +* 0b00010111 - - - RC - LFE FR FL +* 0b00011011 - - - RC FC - FR FL +* 0b00011111 - - - RC FC LFE FR FL +* 0b00110011 - - RR RL - - FR FL +* 0b00110111 - - RR RL - LFE FR FL +* 0b00111011 - - RR RL FC - FR FL +* 0b00111111 - - RR RL FC LFE FR FL +* 0b01110011 - RC RR RL - - FR FL +* 0b01110111 - RC RR RL - LFE FR FL +* 0b01111011 - RC RR RL FC - FR FL +* 0b01111111 - RC RR RL FC LFE FR FL +* 0b11110011 RRC RLC RR RL - - FR FL +* 0b11110111 RRC RLC RR RL - LFE FR FL +* 0b11111011 RRC RLC RR RL FC - FR FL +* 0b11111111 RRC RLC RR RL FC LFE FR FL +* 0b11000011 FRC FLC - - - - FR FL +* 0b11000111 FRC FLC - - - LFE FR FL +* 0b11001011 FRC FLC - - FC - FR FL +* 0b11001111 FRC FLC - - FC LFE FR FL +* 0b11010011 FRC FLC - RC - - FR FL +* 0b11010111 FRC FLC - RC - LFE FR FL +* 0b11011011 FRC FLC - RC FC - FR FL +* 0b11011111 FRC FLC - RC FC LFE FR FL +* 0b11110011 FRC FLC RR RL - - FR FL +* 0b11110111 FRC FLC RR RL - LFE FR FL +* 0b11111011 FRC FLC RR RL FC - FR FL +* 0b11111111 FRC FLC RR RL FC LFE FR FL +* +* @param +* speakers - speaker information as it comes from CEA audio block +*/ +/* translate speakers to channels */ + +union audio_cea_channels { + uint8_t all; + struct audio_cea_channels_bits { + uint32_t FL:1; + uint32_t FR:1; + uint32_t LFE:1; + uint32_t FC:1; + uint32_t RL_RC:1; + uint32_t RR:1; + uint32_t RC_RLC_FLC:1; + uint32_t RRC_FRC:1; + } channels; +}; + +struct audio_clock_info { + /* pixel clock frequency*/ + uint32_t pixel_clock_in_10khz; + /* N - 32KHz audio */ + uint32_t n_32khz; + /* CTS - 32KHz audio*/ + uint32_t cts_32khz; + uint32_t n_44khz; + uint32_t cts_44khz; + uint32_t n_48khz; + uint32_t cts_48khz; +}; + +/* 25.2MHz/1.001*/ +/* 25.2MHz/1.001*/ +/* 25.2MHz*/ +/* 27MHz */ +/* 27MHz*1.001*/ +/* 27MHz*1.001*/ +/* 54MHz*/ +/* 54MHz*1.001*/ +/* 74.25MHz/1.001*/ +/* 74.25MHz*/ +/* 148.5MHz/1.001*/ +/* 148.5MHz*/ + +static const struct audio_clock_info audio_clock_info_table[16] = { + {2517, 4576, 28125, 7007, 31250, 6864, 28125}, + {2518, 4576, 28125, 7007, 31250, 6864, 28125}, + {2520, 4096, 25200, 6272, 28000, 6144, 25200}, + {2700, 4096, 27000, 6272, 30000, 6144, 27000}, + {2702, 4096, 27027, 6272, 30030, 6144, 27027}, + {2703, 4096, 27027, 6272, 30030, 6144, 27027}, + {5400, 4096, 54000, 6272, 60000, 6144, 54000}, + {5405, 4096, 54054, 6272, 60060, 6144, 54054}, + {7417, 11648, 210937, 17836, 234375, 11648, 140625}, + {7425, 4096, 74250, 6272, 82500, 6144, 74250}, + {14835, 11648, 421875, 8918, 234375, 5824, 140625}, + {14850, 4096, 148500, 6272, 165000, 6144, 148500}, + {29670, 5824, 421875, 4459, 234375, 5824, 281250}, + {29700, 3072, 222750, 4704, 247500, 5120, 247500}, + {59340, 5824, 843750, 8918, 937500, 5824, 562500}, + {59400, 3072, 445500, 9408, 990000, 6144, 594000} +}; + +static const struct audio_clock_info audio_clock_info_table_36bpc[14] = { + {2517, 9152, 84375, 7007, 48875, 9152, 56250}, + {2518, 9152, 84375, 7007, 48875, 9152, 56250}, + {2520, 4096, 37800, 6272, 42000, 6144, 37800}, + {2700, 4096, 40500, 6272, 45000, 6144, 40500}, + {2702, 8192, 81081, 6272, 45045, 8192, 54054}, + {2703, 8192, 81081, 6272, 45045, 8192, 54054}, + {5400, 4096, 81000, 6272, 90000, 6144, 81000}, + {5405, 4096, 81081, 6272, 90090, 6144, 81081}, + {7417, 11648, 316406, 17836, 351562, 11648, 210937}, + {7425, 4096, 111375, 6272, 123750, 6144, 111375}, + {14835, 11648, 632812, 17836, 703125, 11648, 421875}, + {14850, 4096, 222750, 6272, 247500, 6144, 222750}, + {29670, 5824, 632812, 8918, 703125, 5824, 421875}, + {29700, 4096, 445500, 4704, 371250, 5120, 371250} +}; + +static const struct audio_clock_info audio_clock_info_table_48bpc[14] = { + {2517, 4576, 56250, 7007, 62500, 6864, 56250}, + {2518, 4576, 56250, 7007, 62500, 6864, 56250}, + {2520, 4096, 50400, 6272, 56000, 6144, 50400}, + {2700, 4096, 54000, 6272, 60000, 6144, 54000}, + {2702, 4096, 54054, 6267, 60060, 8192, 54054}, + {2703, 4096, 54054, 6272, 60060, 8192, 54054}, + {5400, 4096, 108000, 6272, 120000, 6144, 108000}, + {5405, 4096, 108108, 6272, 120120, 6144, 108108}, + {7417, 11648, 421875, 17836, 468750, 11648, 281250}, + {7425, 4096, 148500, 6272, 165000, 6144, 148500}, + {14835, 11648, 843750, 8918, 468750, 11648, 281250}, + {14850, 4096, 297000, 6272, 330000, 6144, 297000}, + {29670, 5824, 843750, 4459, 468750, 5824, 562500}, + {29700, 3072, 445500, 4704, 495000, 5120, 495000} + + +}; + +static union audio_cea_channels speakers_to_channels( + struct audio_speaker_flags speaker_flags) +{ + union audio_cea_channels cea_channels = {0}; + + /* these are one to one */ + cea_channels.channels.FL = speaker_flags.FL_FR; + cea_channels.channels.FR = speaker_flags.FL_FR; + cea_channels.channels.LFE = speaker_flags.LFE; + cea_channels.channels.FC = speaker_flags.FC; + + /* if Rear Left and Right exist move RC speaker to channel 7 + * otherwise to channel 5 + */ + if (speaker_flags.RL_RR) { + cea_channels.channels.RL_RC = speaker_flags.RL_RR; + cea_channels.channels.RR = speaker_flags.RL_RR; + cea_channels.channels.RC_RLC_FLC = speaker_flags.RC; + } else { + cea_channels.channels.RL_RC = speaker_flags.RC; + } + + /* FRONT Left Right Center and REAR Left Right Center are exclusive */ + if (speaker_flags.FLC_FRC) { + cea_channels.channels.RC_RLC_FLC = speaker_flags.FLC_FRC; + cea_channels.channels.RRC_FRC = speaker_flags.FLC_FRC; + } else { + cea_channels.channels.RC_RLC_FLC = speaker_flags.RLC_RRC; + cea_channels.channels.RRC_FRC = speaker_flags.RLC_RRC; + } + + return cea_channels; +} + +static uint32_t calc_max_audio_packets_per_line( + const struct audio_crtc_info *crtc_info) +{ + uint32_t max_packets_per_line; + + max_packets_per_line = + crtc_info->h_total - crtc_info->h_active; + + if (crtc_info->pixel_repetition) + max_packets_per_line *= crtc_info->pixel_repetition; + + /* for other hdmi features */ + max_packets_per_line -= 58; + /* for Control Period */ + max_packets_per_line -= 16; + /* Number of Audio Packets per Line */ + max_packets_per_line /= 32; + + return max_packets_per_line; +} + +static void get_audio_clock_info( + enum dc_color_depth color_depth, + uint32_t crtc_pixel_clock_in_khz, + uint32_t actual_pixel_clock_in_khz, + struct audio_clock_info *audio_clock_info) +{ + const struct audio_clock_info *clock_info; + uint32_t index; + uint32_t crtc_pixel_clock_in_10khz = crtc_pixel_clock_in_khz / 10; + uint32_t audio_array_size; + + switch (color_depth) { + case COLOR_DEPTH_161616: + clock_info = audio_clock_info_table_48bpc; + audio_array_size = ARRAY_SIZE( + audio_clock_info_table_48bpc); + break; + case COLOR_DEPTH_121212: + clock_info = audio_clock_info_table_36bpc; + audio_array_size = ARRAY_SIZE( + audio_clock_info_table_36bpc); + break; + default: + clock_info = audio_clock_info_table; + audio_array_size = ARRAY_SIZE( + audio_clock_info_table); + break; + } + + if (clock_info != NULL) { + /* search for exact pixel clock in table */ + for (index = 0; index < audio_array_size; index++) { + if (clock_info[index].pixel_clock_in_10khz > + crtc_pixel_clock_in_10khz) + break; /* not match */ + else if (clock_info[index].pixel_clock_in_10khz == + crtc_pixel_clock_in_10khz) { + /* match found */ + *audio_clock_info = clock_info[index]; + return; + } + } + } + + /* not found */ + if (actual_pixel_clock_in_khz == 0) + actual_pixel_clock_in_khz = crtc_pixel_clock_in_khz; + + /* See HDMI spec the table entry under + * pixel clock of "Other". */ + audio_clock_info->pixel_clock_in_10khz = + actual_pixel_clock_in_khz / 10; + audio_clock_info->cts_32khz = actual_pixel_clock_in_khz; + audio_clock_info->cts_44khz = actual_pixel_clock_in_khz; + audio_clock_info->cts_48khz = actual_pixel_clock_in_khz; + + audio_clock_info->n_32khz = 4096; + audio_clock_info->n_44khz = 6272; + audio_clock_info->n_48khz = 6144; +} + +static void enc1_se_audio_setup( + struct stream_encoder *enc, + unsigned int az_inst, + struct audio_info *audio_info) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + + uint32_t speakers = 0; + uint32_t channels = 0; + + ASSERT(audio_info); + if (audio_info == NULL) + /* This should not happen.it does so we don't get BSOD*/ + return; + + speakers = audio_info->flags.info.ALLSPEAKERS; + channels = speakers_to_channels(audio_info->flags.speaker_flags).all; + + /* setup the audio stream source select (audio -> dig mapping) */ + REG_SET(AFMT_AUDIO_SRC_CONTROL, 0, AFMT_AUDIO_SRC_SELECT, az_inst); + + /* Channel allocation */ + REG_UPDATE(AFMT_AUDIO_PACKET_CONTROL2, AFMT_AUDIO_CHANNEL_ENABLE, channels); +} + +static void enc1_se_setup_hdmi_audio( + struct stream_encoder *enc, + const struct audio_crtc_info *crtc_info) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + + struct audio_clock_info audio_clock_info = {0}; + uint32_t max_packets_per_line; + + /* For now still do calculation, although this field is ignored when + * above HDMI_PACKET_GEN_VERSION set to 1 + */ + max_packets_per_line = calc_max_audio_packets_per_line(crtc_info); + + /* HDMI_AUDIO_PACKET_CONTROL */ + REG_UPDATE_2(HDMI_AUDIO_PACKET_CONTROL, + HDMI_AUDIO_PACKETS_PER_LINE, max_packets_per_line, + HDMI_AUDIO_DELAY_EN, 1); + + /* AFMT_AUDIO_PACKET_CONTROL */ + REG_UPDATE(AFMT_AUDIO_PACKET_CONTROL, AFMT_60958_CS_UPDATE, 1); + + /* AFMT_AUDIO_PACKET_CONTROL2 */ + REG_UPDATE_2(AFMT_AUDIO_PACKET_CONTROL2, + AFMT_AUDIO_LAYOUT_OVRD, 0, + AFMT_60958_OSF_OVRD, 0); + + /* HDMI_ACR_PACKET_CONTROL */ + REG_UPDATE_3(HDMI_ACR_PACKET_CONTROL, + HDMI_ACR_AUTO_SEND, 1, + HDMI_ACR_SOURCE, 0, + HDMI_ACR_AUDIO_PRIORITY, 0); + + /* Program audio clock sample/regeneration parameters */ + get_audio_clock_info(crtc_info->color_depth, + crtc_info->requested_pixel_clock, + crtc_info->calculated_pixel_clock, + &audio_clock_info); + DC_LOG_HW_AUDIO( + "\n%s:Input::requested_pixel_clock = %d" \ + "calculated_pixel_clock = %d \n", __func__, \ + crtc_info->requested_pixel_clock, \ + crtc_info->calculated_pixel_clock); + + /* HDMI_ACR_32_0__HDMI_ACR_CTS_32_MASK */ + REG_UPDATE(HDMI_ACR_32_0, HDMI_ACR_CTS_32, audio_clock_info.cts_32khz); + + /* HDMI_ACR_32_1__HDMI_ACR_N_32_MASK */ + REG_UPDATE(HDMI_ACR_32_1, HDMI_ACR_N_32, audio_clock_info.n_32khz); + + /* HDMI_ACR_44_0__HDMI_ACR_CTS_44_MASK */ + REG_UPDATE(HDMI_ACR_44_0, HDMI_ACR_CTS_44, audio_clock_info.cts_44khz); + + /* HDMI_ACR_44_1__HDMI_ACR_N_44_MASK */ + REG_UPDATE(HDMI_ACR_44_1, HDMI_ACR_N_44, audio_clock_info.n_44khz); + + /* HDMI_ACR_48_0__HDMI_ACR_CTS_48_MASK */ + REG_UPDATE(HDMI_ACR_48_0, HDMI_ACR_CTS_48, audio_clock_info.cts_48khz); + + /* HDMI_ACR_48_1__HDMI_ACR_N_48_MASK */ + REG_UPDATE(HDMI_ACR_48_1, HDMI_ACR_N_48, audio_clock_info.n_48khz); + + /* Video driver cannot know in advance which sample rate will + * be used by HD Audio driver + * HDMI_ACR_PACKET_CONTROL__HDMI_ACR_N_MULTIPLE field is + * programmed below in interruppt callback + */ + + /* AFMT_60958_0__AFMT_60958_CS_CHANNEL_NUMBER_L_MASK & + * AFMT_60958_0__AFMT_60958_CS_CLOCK_ACCURACY_MASK + */ + REG_UPDATE_2(AFMT_60958_0, + AFMT_60958_CS_CHANNEL_NUMBER_L, 1, + AFMT_60958_CS_CLOCK_ACCURACY, 0); + + /* AFMT_60958_1 AFMT_60958_CS_CHALNNEL_NUMBER_R */ + REG_UPDATE(AFMT_60958_1, AFMT_60958_CS_CHANNEL_NUMBER_R, 2); + + /* AFMT_60958_2 now keep this settings until + * Programming guide comes out + */ + REG_UPDATE_6(AFMT_60958_2, + AFMT_60958_CS_CHANNEL_NUMBER_2, 3, + AFMT_60958_CS_CHANNEL_NUMBER_3, 4, + AFMT_60958_CS_CHANNEL_NUMBER_4, 5, + AFMT_60958_CS_CHANNEL_NUMBER_5, 6, + AFMT_60958_CS_CHANNEL_NUMBER_6, 7, + AFMT_60958_CS_CHANNEL_NUMBER_7, 8); +} + +static void enc1_se_setup_dp_audio( + struct stream_encoder *enc) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + + /* --- DP Audio packet configurations --- */ + + /* ATP Configuration */ + REG_SET(DP_SEC_AUD_N, 0, + DP_SEC_AUD_N, DP_SEC_AUD_N__DP_SEC_AUD_N__DEFAULT); + + /* Async/auto-calc timestamp mode */ + REG_SET(DP_SEC_TIMESTAMP, 0, DP_SEC_TIMESTAMP_MODE, + DP_SEC_TIMESTAMP__DP_SEC_TIMESTAMP_MODE__AUTO_CALC); + + /* --- The following are the registers + * copied from the SetupHDMI --- + */ + + /* AFMT_AUDIO_PACKET_CONTROL */ + REG_UPDATE(AFMT_AUDIO_PACKET_CONTROL, AFMT_60958_CS_UPDATE, 1); + + /* AFMT_AUDIO_PACKET_CONTROL2 */ + /* Program the ATP and AIP next */ + REG_UPDATE_2(AFMT_AUDIO_PACKET_CONTROL2, + AFMT_AUDIO_LAYOUT_OVRD, 0, + AFMT_60958_OSF_OVRD, 0); + + /* AFMT_INFOFRAME_CONTROL0 */ + REG_UPDATE(AFMT_INFOFRAME_CONTROL0, AFMT_AUDIO_INFO_UPDATE, 1); + + /* AFMT_60958_0__AFMT_60958_CS_CLOCK_ACCURACY_MASK */ + REG_UPDATE(AFMT_60958_0, AFMT_60958_CS_CLOCK_ACCURACY, 0); +} + +static void enc1_se_enable_audio_clock( + struct stream_encoder *enc, + bool enable) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + + if (REG(AFMT_CNTL) == 0) + return; /* DCE8/10 does not have this register */ + + REG_UPDATE(AFMT_CNTL, AFMT_AUDIO_CLOCK_EN, !!enable); + + /* wait for AFMT clock to turn on, + * expectation: this should complete in 1-2 reads + * + * REG_WAIT(AFMT_CNTL, AFMT_AUDIO_CLOCK_ON, !!enable, 1, 10); + * + * TODO: wait for clock_on does not work well. May need HW + * program sequence. But audio seems work normally even without wait + * for clock_on status change + */ +} + +static void enc1_se_enable_dp_audio( + struct stream_encoder *enc) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + + /* Enable Audio packets */ + REG_UPDATE(DP_SEC_CNTL, DP_SEC_ASP_ENABLE, 1); + + /* Program the ATP and AIP next */ + REG_UPDATE_2(DP_SEC_CNTL, + DP_SEC_ATP_ENABLE, 1, + DP_SEC_AIP_ENABLE, 1); + + /* Program STREAM_ENABLE after all the other enables. */ + REG_UPDATE(DP_SEC_CNTL, DP_SEC_STREAM_ENABLE, 1); +} + +static void enc1_se_disable_dp_audio( + struct stream_encoder *enc) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + uint32_t value = 0; + + /* Disable Audio packets */ + REG_UPDATE_5(DP_SEC_CNTL, + DP_SEC_ASP_ENABLE, 0, + DP_SEC_ATP_ENABLE, 0, + DP_SEC_AIP_ENABLE, 0, + DP_SEC_ACM_ENABLE, 0, + DP_SEC_STREAM_ENABLE, 0); + + /* This register shared with encoder info frame. Therefore we need to + * keep master enabled if at least on of the fields is not 0 + */ + value = REG_READ(DP_SEC_CNTL); + if (value != 0) + REG_UPDATE(DP_SEC_CNTL, DP_SEC_STREAM_ENABLE, 1); + +} + +void enc1_se_audio_mute_control( + struct stream_encoder *enc, + bool mute) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + + REG_UPDATE(AFMT_AUDIO_PACKET_CONTROL, AFMT_AUDIO_SAMPLE_SEND, !mute); +} + +void enc1_se_dp_audio_setup( + struct stream_encoder *enc, + unsigned int az_inst, + struct audio_info *info) +{ + enc1_se_audio_setup(enc, az_inst, info); +} + +void enc1_se_dp_audio_enable( + struct stream_encoder *enc) +{ + enc1_se_enable_audio_clock(enc, true); + enc1_se_setup_dp_audio(enc); + enc1_se_enable_dp_audio(enc); +} + +void enc1_se_dp_audio_disable( + struct stream_encoder *enc) +{ + enc1_se_disable_dp_audio(enc); + enc1_se_enable_audio_clock(enc, false); +} + +void enc1_se_hdmi_audio_setup( + struct stream_encoder *enc, + unsigned int az_inst, + struct audio_info *info, + struct audio_crtc_info *audio_crtc_info) +{ + enc1_se_enable_audio_clock(enc, true); + enc1_se_setup_hdmi_audio(enc, audio_crtc_info); + enc1_se_audio_setup(enc, az_inst, info); +} + +void enc1_se_hdmi_audio_disable( + struct stream_encoder *enc) +{ + enc1_se_enable_audio_clock(enc, false); +} + + +void enc1_setup_stereo_sync( + struct stream_encoder *enc, + int tg_inst, bool enable) +{ + struct dcn10_stream_encoder *enc1 = DCN10STRENC_FROM_STRENC(enc); + REG_UPDATE(DIG_FE_CNTL, DIG_STEREOSYNC_SELECT, tg_inst); + REG_UPDATE(DIG_FE_CNTL, DIG_STEREOSYNC_GATE_EN, !enable); +} + + +static const struct stream_encoder_funcs dcn10_str_enc_funcs = { + .dp_set_stream_attribute = + enc1_stream_encoder_dp_set_stream_attribute, + .hdmi_set_stream_attribute = + enc1_stream_encoder_hdmi_set_stream_attribute, + .dvi_set_stream_attribute = + enc1_stream_encoder_dvi_set_stream_attribute, + .set_mst_bandwidth = + enc1_stream_encoder_set_mst_bandwidth, + .update_hdmi_info_packets = + enc1_stream_encoder_update_hdmi_info_packets, + .stop_hdmi_info_packets = + enc1_stream_encoder_stop_hdmi_info_packets, + .update_dp_info_packets = + enc1_stream_encoder_update_dp_info_packets, + .stop_dp_info_packets = + enc1_stream_encoder_stop_dp_info_packets, + .dp_blank = + enc1_stream_encoder_dp_blank, + .dp_unblank = + enc1_stream_encoder_dp_unblank, + .audio_mute_control = enc1_se_audio_mute_control, + + .dp_audio_setup = enc1_se_dp_audio_setup, + .dp_audio_enable = enc1_se_dp_audio_enable, + .dp_audio_disable = enc1_se_dp_audio_disable, + + .hdmi_audio_setup = enc1_se_hdmi_audio_setup, + .hdmi_audio_disable = enc1_se_hdmi_audio_disable, + .setup_stereo_sync = enc1_setup_stereo_sync, + .set_avmute = enc1_stream_encoder_set_avmute, +}; + +void dcn10_stream_encoder_construct( + struct dcn10_stream_encoder *enc1, + struct dc_context *ctx, + struct dc_bios *bp, + enum engine_id eng_id, + const struct dcn10_stream_enc_registers *regs, + const struct dcn10_stream_encoder_shift *se_shift, + const struct dcn10_stream_encoder_mask *se_mask) +{ + enc1->base.funcs = &dcn10_str_enc_funcs; + enc1->base.ctx = ctx; + enc1->base.id = eng_id; + enc1->base.bp = bp; + enc1->regs = regs; + enc1->se_shift = se_shift; + enc1->se_mask = se_mask; +} + diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_stream_encoder.h b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_stream_encoder.h new file mode 100644 index 000000000000..6b3e4ded155b --- /dev/null +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_stream_encoder.h @@ -0,0 +1,524 @@ +/* + * Copyright 2012-15 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * Authors: AMD + * + */ + +#ifndef __DC_STREAM_ENCODER_DCN10_H__ +#define __DC_STREAM_ENCODER_DCN10_H__ + +#include "stream_encoder.h" + +#define DCN10STRENC_FROM_STRENC(stream_encoder)\ + container_of(stream_encoder, struct dcn10_stream_encoder, base) + +#define SE_COMMON_DCN_REG_LIST(id) \ + SRI(AFMT_CNTL, DIG, id), \ + SRI(AFMT_GENERIC_0, DIG, id), \ + SRI(AFMT_GENERIC_1, DIG, id), \ + SRI(AFMT_GENERIC_2, DIG, id), \ + SRI(AFMT_GENERIC_3, DIG, id), \ + SRI(AFMT_GENERIC_4, DIG, id), \ + SRI(AFMT_GENERIC_5, DIG, id), \ + SRI(AFMT_GENERIC_6, DIG, id), \ + SRI(AFMT_GENERIC_7, DIG, id), \ + SRI(AFMT_GENERIC_HDR, DIG, id), \ + SRI(AFMT_INFOFRAME_CONTROL0, DIG, id), \ + SRI(AFMT_VBI_PACKET_CONTROL, DIG, id), \ + SRI(AFMT_VBI_PACKET_CONTROL1, DIG, id), \ + SRI(AFMT_AUDIO_PACKET_CONTROL, DIG, id), \ + SRI(AFMT_AUDIO_PACKET_CONTROL2, DIG, id), \ + SRI(AFMT_AUDIO_SRC_CONTROL, DIG, id), \ + SRI(AFMT_60958_0, DIG, id), \ + SRI(AFMT_60958_1, DIG, id), \ + SRI(AFMT_60958_2, DIG, id), \ + SRI(DIG_FE_CNTL, DIG, id), \ + SRI(HDMI_CONTROL, DIG, id), \ + SRI(HDMI_DB_CONTROL, DIG, id), \ + SRI(HDMI_GC, DIG, id), \ + SRI(HDMI_GENERIC_PACKET_CONTROL0, DIG, id), \ + SRI(HDMI_GENERIC_PACKET_CONTROL1, DIG, id), \ + SRI(HDMI_GENERIC_PACKET_CONTROL2, DIG, id), \ + SRI(HDMI_GENERIC_PACKET_CONTROL3, DIG, id), \ + SRI(HDMI_INFOFRAME_CONTROL0, DIG, id), \ + SRI(HDMI_INFOFRAME_CONTROL1, DIG, id), \ + SRI(HDMI_VBI_PACKET_CONTROL, DIG, id), \ + SRI(HDMI_AUDIO_PACKET_CONTROL, DIG, id),\ + SRI(HDMI_ACR_PACKET_CONTROL, DIG, id),\ + SRI(HDMI_ACR_32_0, DIG, id),\ + SRI(HDMI_ACR_32_1, DIG, id),\ + SRI(HDMI_ACR_44_0, DIG, id),\ + SRI(HDMI_ACR_44_1, DIG, id),\ + SRI(HDMI_ACR_48_0, DIG, id),\ + SRI(HDMI_ACR_48_1, DIG, id),\ + SRI(DP_DB_CNTL, DP, id), \ + SRI(DP_MSA_MISC, DP, id), \ + SRI(DP_MSA_COLORIMETRY, DP, id), \ + SRI(DP_MSA_TIMING_PARAM1, DP, id), \ + SRI(DP_MSA_TIMING_PARAM2, DP, id), \ + SRI(DP_MSA_TIMING_PARAM3, DP, id), \ + SRI(DP_MSA_TIMING_PARAM4, DP, id), \ + SRI(DP_MSE_RATE_CNTL, DP, id), \ + SRI(DP_MSE_RATE_UPDATE, DP, id), \ + SRI(DP_PIXEL_FORMAT, DP, id), \ + SRI(DP_SEC_CNTL, DP, id), \ + SRI(DP_STEER_FIFO, DP, id), \ + SRI(DP_VID_M, DP, id), \ + SRI(DP_VID_N, DP, id), \ + SRI(DP_VID_STREAM_CNTL, DP, id), \ + SRI(DP_VID_TIMING, DP, id), \ + SRI(DP_SEC_AUD_N, DP, id), \ + SRI(DP_SEC_TIMESTAMP, DP, id) + +#define SE_DCN_REG_LIST(id)\ + SE_COMMON_DCN_REG_LIST(id) + + +struct dcn10_stream_enc_registers { + uint32_t AFMT_CNTL; + uint32_t AFMT_AVI_INFO0; + uint32_t AFMT_AVI_INFO1; + uint32_t AFMT_AVI_INFO2; + uint32_t AFMT_AVI_INFO3; + uint32_t AFMT_GENERIC_0; + uint32_t AFMT_GENERIC_1; + uint32_t AFMT_GENERIC_2; + uint32_t AFMT_GENERIC_3; + uint32_t AFMT_GENERIC_4; + uint32_t AFMT_GENERIC_5; + uint32_t AFMT_GENERIC_6; + uint32_t AFMT_GENERIC_7; + uint32_t AFMT_GENERIC_HDR; + uint32_t AFMT_INFOFRAME_CONTROL0; + uint32_t AFMT_VBI_PACKET_CONTROL; + uint32_t AFMT_VBI_PACKET_CONTROL1; + uint32_t AFMT_AUDIO_PACKET_CONTROL; + uint32_t AFMT_AUDIO_PACKET_CONTROL2; + uint32_t AFMT_AUDIO_SRC_CONTROL; + uint32_t AFMT_60958_0; + uint32_t AFMT_60958_1; + uint32_t AFMT_60958_2; + uint32_t DIG_FE_CNTL; + uint32_t DP_MSE_RATE_CNTL; + uint32_t DP_MSE_RATE_UPDATE; + uint32_t DP_PIXEL_FORMAT; + uint32_t DP_SEC_CNTL; + uint32_t DP_STEER_FIFO; + uint32_t DP_VID_M; + uint32_t DP_VID_N; + uint32_t DP_VID_STREAM_CNTL; + uint32_t DP_VID_TIMING; + uint32_t DP_SEC_AUD_N; + uint32_t DP_SEC_TIMESTAMP; + uint32_t HDMI_CONTROL; + uint32_t HDMI_GC; + uint32_t HDMI_GENERIC_PACKET_CONTROL0; + uint32_t HDMI_GENERIC_PACKET_CONTROL1; + uint32_t HDMI_GENERIC_PACKET_CONTROL2; + uint32_t HDMI_GENERIC_PACKET_CONTROL3; + uint32_t HDMI_GENERIC_PACKET_CONTROL4; + uint32_t HDMI_GENERIC_PACKET_CONTROL5; + uint32_t HDMI_INFOFRAME_CONTROL0; + uint32_t HDMI_INFOFRAME_CONTROL1; + uint32_t HDMI_VBI_PACKET_CONTROL; + uint32_t HDMI_AUDIO_PACKET_CONTROL; + uint32_t HDMI_ACR_PACKET_CONTROL; + uint32_t HDMI_ACR_32_0; + uint32_t HDMI_ACR_32_1; + uint32_t HDMI_ACR_44_0; + uint32_t HDMI_ACR_44_1; + uint32_t HDMI_ACR_48_0; + uint32_t HDMI_ACR_48_1; + uint32_t DP_DB_CNTL; + uint32_t DP_MSA_MISC; + uint32_t DP_MSA_COLORIMETRY; + uint32_t DP_MSA_TIMING_PARAM1; + uint32_t DP_MSA_TIMING_PARAM2; + uint32_t DP_MSA_TIMING_PARAM3; + uint32_t DP_MSA_TIMING_PARAM4; + uint32_t HDMI_DB_CONTROL; +}; + + +#define SE_SF(reg_name, field_name, post_fix)\ + .field_name = reg_name ## __ ## field_name ## post_fix + +#define SE_COMMON_MASK_SH_LIST_SOC_BASE(mask_sh)\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL, AFMT_GENERIC_INDEX, mask_sh),\ + SE_SF(DIG0_AFMT_GENERIC_HDR, AFMT_GENERIC_HB0, mask_sh),\ + SE_SF(DIG0_AFMT_GENERIC_HDR, AFMT_GENERIC_HB1, mask_sh),\ + SE_SF(DIG0_AFMT_GENERIC_HDR, AFMT_GENERIC_HB2, mask_sh),\ + SE_SF(DIG0_AFMT_GENERIC_HDR, AFMT_GENERIC_HB3, mask_sh),\ + SE_SF(DP0_DP_PIXEL_FORMAT, DP_PIXEL_ENCODING, mask_sh),\ + SE_SF(DP0_DP_PIXEL_FORMAT, DP_COMPONENT_DEPTH, mask_sh),\ + SE_SF(DIG0_HDMI_CONTROL, HDMI_PACKET_GEN_VERSION, mask_sh),\ + SE_SF(DIG0_HDMI_CONTROL, HDMI_KEEPOUT_MODE, mask_sh),\ + SE_SF(DIG0_HDMI_CONTROL, HDMI_DEEP_COLOR_ENABLE, mask_sh),\ + SE_SF(DIG0_HDMI_CONTROL, HDMI_DEEP_COLOR_DEPTH, mask_sh),\ + SE_SF(DIG0_HDMI_CONTROL, HDMI_DATA_SCRAMBLE_EN, mask_sh),\ + SE_SF(DIG0_HDMI_VBI_PACKET_CONTROL, HDMI_GC_CONT, mask_sh),\ + SE_SF(DIG0_HDMI_VBI_PACKET_CONTROL, HDMI_GC_SEND, mask_sh),\ + SE_SF(DIG0_HDMI_VBI_PACKET_CONTROL, HDMI_NULL_SEND, mask_sh),\ + SE_SF(DIG0_HDMI_INFOFRAME_CONTROL0, HDMI_AUDIO_INFO_SEND, mask_sh),\ + SE_SF(DIG0_AFMT_INFOFRAME_CONTROL0, AFMT_AUDIO_INFO_UPDATE, mask_sh),\ + SE_SF(DIG0_HDMI_INFOFRAME_CONTROL1, HDMI_AUDIO_INFO_LINE, mask_sh),\ + SE_SF(DIG0_HDMI_GC, HDMI_GC_AVMUTE, mask_sh),\ + SE_SF(DP0_DP_MSE_RATE_CNTL, DP_MSE_RATE_X, mask_sh),\ + SE_SF(DP0_DP_MSE_RATE_CNTL, DP_MSE_RATE_Y, mask_sh),\ + SE_SF(DP0_DP_MSE_RATE_UPDATE, DP_MSE_RATE_UPDATE_PENDING, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_GSP0_ENABLE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_STREAM_ENABLE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_GSP1_ENABLE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_GSP2_ENABLE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_GSP3_ENABLE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_MPG_ENABLE, mask_sh),\ + SE_SF(DP0_DP_VID_STREAM_CNTL, DP_VID_STREAM_DIS_DEFER, mask_sh),\ + SE_SF(DP0_DP_VID_STREAM_CNTL, DP_VID_STREAM_ENABLE, mask_sh),\ + SE_SF(DP0_DP_VID_STREAM_CNTL, DP_VID_STREAM_STATUS, mask_sh),\ + SE_SF(DP0_DP_STEER_FIFO, DP_STEER_FIFO_RESET, mask_sh),\ + SE_SF(DP0_DP_VID_TIMING, DP_VID_M_N_GEN_EN, mask_sh),\ + SE_SF(DP0_DP_VID_N, DP_VID_N, mask_sh),\ + SE_SF(DP0_DP_VID_M, DP_VID_M, mask_sh),\ + SE_SF(DIG0_DIG_FE_CNTL, DIG_START, mask_sh),\ + SE_SF(DIG0_AFMT_AUDIO_SRC_CONTROL, AFMT_AUDIO_SRC_SELECT, mask_sh),\ + SE_SF(DIG0_AFMT_AUDIO_PACKET_CONTROL2, AFMT_AUDIO_CHANNEL_ENABLE, mask_sh),\ + SE_SF(DIG0_HDMI_AUDIO_PACKET_CONTROL, HDMI_AUDIO_PACKETS_PER_LINE, mask_sh),\ + SE_SF(DIG0_HDMI_AUDIO_PACKET_CONTROL, HDMI_AUDIO_DELAY_EN, mask_sh),\ + SE_SF(DIG0_AFMT_AUDIO_PACKET_CONTROL, AFMT_60958_CS_UPDATE, mask_sh),\ + SE_SF(DIG0_AFMT_AUDIO_PACKET_CONTROL2, AFMT_AUDIO_LAYOUT_OVRD, mask_sh),\ + SE_SF(DIG0_AFMT_AUDIO_PACKET_CONTROL2, AFMT_60958_OSF_OVRD, mask_sh),\ + SE_SF(DIG0_HDMI_ACR_PACKET_CONTROL, HDMI_ACR_AUTO_SEND, mask_sh),\ + SE_SF(DIG0_HDMI_ACR_PACKET_CONTROL, HDMI_ACR_SOURCE, mask_sh),\ + SE_SF(DIG0_HDMI_ACR_PACKET_CONTROL, HDMI_ACR_AUDIO_PRIORITY, mask_sh),\ + SE_SF(DIG0_HDMI_ACR_32_0, HDMI_ACR_CTS_32, mask_sh),\ + SE_SF(DIG0_HDMI_ACR_32_1, HDMI_ACR_N_32, mask_sh),\ + SE_SF(DIG0_HDMI_ACR_44_0, HDMI_ACR_CTS_44, mask_sh),\ + SE_SF(DIG0_HDMI_ACR_44_1, HDMI_ACR_N_44, mask_sh),\ + SE_SF(DIG0_HDMI_ACR_48_0, HDMI_ACR_CTS_48, mask_sh),\ + SE_SF(DIG0_HDMI_ACR_48_1, HDMI_ACR_N_48, mask_sh),\ + SE_SF(DIG0_AFMT_60958_0, AFMT_60958_CS_CHANNEL_NUMBER_L, mask_sh),\ + SE_SF(DIG0_AFMT_60958_0, AFMT_60958_CS_CLOCK_ACCURACY, mask_sh),\ + SE_SF(DIG0_AFMT_60958_1, AFMT_60958_CS_CHANNEL_NUMBER_R, mask_sh),\ + SE_SF(DIG0_AFMT_60958_2, AFMT_60958_CS_CHANNEL_NUMBER_2, mask_sh),\ + SE_SF(DIG0_AFMT_60958_2, AFMT_60958_CS_CHANNEL_NUMBER_3, mask_sh),\ + SE_SF(DIG0_AFMT_60958_2, AFMT_60958_CS_CHANNEL_NUMBER_4, mask_sh),\ + SE_SF(DIG0_AFMT_60958_2, AFMT_60958_CS_CHANNEL_NUMBER_5, mask_sh),\ + SE_SF(DIG0_AFMT_60958_2, AFMT_60958_CS_CHANNEL_NUMBER_6, mask_sh),\ + SE_SF(DIG0_AFMT_60958_2, AFMT_60958_CS_CHANNEL_NUMBER_7, mask_sh),\ + SE_SF(DP0_DP_SEC_AUD_N, DP_SEC_AUD_N, mask_sh),\ + SE_SF(DP0_DP_SEC_TIMESTAMP, DP_SEC_TIMESTAMP_MODE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_ASP_ENABLE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_ATP_ENABLE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_AIP_ENABLE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_ACM_ENABLE, mask_sh),\ + SE_SF(DIG0_AFMT_AUDIO_PACKET_CONTROL, AFMT_AUDIO_SAMPLE_SEND, mask_sh),\ + SE_SF(DIG0_AFMT_CNTL, AFMT_AUDIO_CLOCK_EN, mask_sh),\ + SE_SF(DIG0_HDMI_CONTROL, HDMI_CLOCK_CHANNEL_RATE, mask_sh),\ + SE_SF(DIG0_DIG_FE_CNTL, TMDS_PIXEL_ENCODING, mask_sh),\ + SE_SF(DIG0_DIG_FE_CNTL, TMDS_COLOR_FORMAT, mask_sh),\ + SE_SF(DIG0_DIG_FE_CNTL, DIG_STEREOSYNC_SELECT, mask_sh),\ + SE_SF(DIG0_DIG_FE_CNTL, DIG_STEREOSYNC_GATE_EN, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL, AFMT_GENERIC_LOCK_STATUS, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL, AFMT_GENERIC_CONFLICT, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL, AFMT_GENERIC_CONFLICT_CLR, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC0_FRAME_UPDATE_PENDING, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC1_FRAME_UPDATE_PENDING, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC2_FRAME_UPDATE_PENDING, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC3_FRAME_UPDATE_PENDING, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC4_FRAME_UPDATE_PENDING, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC5_FRAME_UPDATE_PENDING, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC6_FRAME_UPDATE_PENDING, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC7_FRAME_UPDATE_PENDING, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC0_FRAME_UPDATE, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC1_FRAME_UPDATE, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC2_FRAME_UPDATE, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC3_FRAME_UPDATE, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC4_FRAME_UPDATE, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC5_FRAME_UPDATE, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC6_FRAME_UPDATE, mask_sh),\ + SE_SF(DIG0_AFMT_VBI_PACKET_CONTROL1, AFMT_GENERIC7_FRAME_UPDATE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_GSP4_ENABLE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_GSP5_ENABLE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_GSP6_ENABLE, mask_sh),\ + SE_SF(DP0_DP_SEC_CNTL, DP_SEC_GSP7_ENABLE, mask_sh),\ + SE_SF(DP0_DP_DB_CNTL, DP_DB_DISABLE, mask_sh),\ + SE_SF(DP0_DP_MSA_COLORIMETRY, DP_MSA_MISC0, mask_sh),\ + SE_SF(DP0_DP_MSA_TIMING_PARAM1, DP_MSA_HTOTAL, mask_sh),\ + SE_SF(DP0_DP_MSA_TIMING_PARAM1, DP_MSA_VTOTAL, mask_sh),\ + SE_SF(DP0_DP_MSA_TIMING_PARAM2, DP_MSA_HSTART, mask_sh),\ + SE_SF(DP0_DP_MSA_TIMING_PARAM2, DP_MSA_VSTART, mask_sh),\ + SE_SF(DP0_DP_MSA_TIMING_PARAM3, DP_MSA_HSYNCWIDTH, mask_sh),\ + SE_SF(DP0_DP_MSA_TIMING_PARAM3, DP_MSA_HSYNCPOLARITY, mask_sh),\ + SE_SF(DP0_DP_MSA_TIMING_PARAM3, DP_MSA_VSYNCWIDTH, mask_sh),\ + SE_SF(DP0_DP_MSA_TIMING_PARAM3, DP_MSA_VSYNCPOLARITY, mask_sh),\ + SE_SF(DP0_DP_MSA_TIMING_PARAM4, DP_MSA_HWIDTH, mask_sh),\ + SE_SF(DP0_DP_MSA_TIMING_PARAM4, DP_MSA_VHEIGHT, mask_sh),\ + SE_SF(DIG0_HDMI_DB_CONTROL, HDMI_DB_DISABLE, mask_sh),\ + SE_SF(DP0_DP_VID_TIMING, DP_VID_N_MUL, mask_sh) + +#define SE_COMMON_MASK_SH_LIST_SOC(mask_sh)\ + SE_COMMON_MASK_SH_LIST_SOC_BASE(mask_sh) + +#define SE_COMMON_MASK_SH_LIST_DCN10(mask_sh)\ + SE_COMMON_MASK_SH_LIST_SOC(mask_sh),\ + SE_SF(DIG0_HDMI_GENERIC_PACKET_CONTROL0, HDMI_GENERIC0_CONT, mask_sh),\ + SE_SF(DIG0_HDMI_GENERIC_PACKET_CONTROL0, HDMI_GENERIC0_SEND, mask_sh),\ + SE_SF(DIG0_HDMI_GENERIC_PACKET_CONTROL0, HDMI_GENERIC0_LINE, mask_sh),\ + SE_SF(DIG0_HDMI_GENERIC_PACKET_CONTROL0, HDMI_GENERIC1_CONT, mask_sh),\ + SE_SF(DIG0_HDMI_GENERIC_PACKET_CONTROL0, HDMI_GENERIC1_SEND, mask_sh),\ + SE_SF(DIG0_HDMI_GENERIC_PACKET_CONTROL0, HDMI_GENERIC1_LINE, mask_sh) + + +#define SE_REG_FIELD_LIST_DCN1_0(type) \ + type AFMT_GENERIC_INDEX;\ + type AFMT_GENERIC_HB0;\ + type AFMT_GENERIC_HB1;\ + type AFMT_GENERIC_HB2;\ + type AFMT_GENERIC_HB3;\ + type AFMT_GENERIC_LOCK_STATUS;\ + type AFMT_GENERIC_CONFLICT;\ + type AFMT_GENERIC_CONFLICT_CLR;\ + type AFMT_GENERIC0_FRAME_UPDATE_PENDING;\ + type AFMT_GENERIC1_FRAME_UPDATE_PENDING;\ + type AFMT_GENERIC2_FRAME_UPDATE_PENDING;\ + type AFMT_GENERIC3_FRAME_UPDATE_PENDING;\ + type AFMT_GENERIC4_FRAME_UPDATE_PENDING;\ + type AFMT_GENERIC5_FRAME_UPDATE_PENDING;\ + type AFMT_GENERIC6_FRAME_UPDATE_PENDING;\ + type AFMT_GENERIC7_FRAME_UPDATE_PENDING;\ + type AFMT_GENERIC0_FRAME_UPDATE;\ + type AFMT_GENERIC1_FRAME_UPDATE;\ + type AFMT_GENERIC2_FRAME_UPDATE;\ + type AFMT_GENERIC3_FRAME_UPDATE;\ + type AFMT_GENERIC4_FRAME_UPDATE;\ + type AFMT_GENERIC5_FRAME_UPDATE;\ + type AFMT_GENERIC6_FRAME_UPDATE;\ + type AFMT_GENERIC7_FRAME_UPDATE;\ + type HDMI_GENERIC0_CONT;\ + type HDMI_GENERIC0_SEND;\ + type HDMI_GENERIC0_LINE;\ + type HDMI_GENERIC1_CONT;\ + type HDMI_GENERIC1_SEND;\ + type HDMI_GENERIC1_LINE;\ + type HDMI_GENERIC2_CONT;\ + type HDMI_GENERIC2_SEND;\ + type HDMI_GENERIC2_LINE;\ + type HDMI_GENERIC3_CONT;\ + type HDMI_GENERIC3_SEND;\ + type HDMI_GENERIC3_LINE;\ + type HDMI_GENERIC4_CONT;\ + type HDMI_GENERIC4_SEND;\ + type HDMI_GENERIC4_LINE;\ + type HDMI_GENERIC5_CONT;\ + type HDMI_GENERIC5_SEND;\ + type HDMI_GENERIC5_LINE;\ + type HDMI_GENERIC6_CONT;\ + type HDMI_GENERIC6_SEND;\ + type HDMI_GENERIC6_LINE;\ + type HDMI_GENERIC7_CONT;\ + type HDMI_GENERIC7_SEND;\ + type HDMI_GENERIC7_LINE;\ + type DP_PIXEL_ENCODING;\ + type DP_COMPONENT_DEPTH;\ + type HDMI_PACKET_GEN_VERSION;\ + type HDMI_KEEPOUT_MODE;\ + type HDMI_DEEP_COLOR_ENABLE;\ + type HDMI_CLOCK_CHANNEL_RATE;\ + type HDMI_DEEP_COLOR_DEPTH;\ + type HDMI_GC_CONT;\ + type HDMI_GC_SEND;\ + type HDMI_NULL_SEND;\ + type HDMI_DATA_SCRAMBLE_EN;\ + type HDMI_AUDIO_INFO_SEND;\ + type AFMT_AUDIO_INFO_UPDATE;\ + type HDMI_AUDIO_INFO_LINE;\ + type HDMI_GC_AVMUTE;\ + type DP_MSE_RATE_X;\ + type DP_MSE_RATE_Y;\ + type DP_MSE_RATE_UPDATE_PENDING;\ + type DP_SEC_GSP0_ENABLE;\ + type DP_SEC_STREAM_ENABLE;\ + type DP_SEC_GSP1_ENABLE;\ + type DP_SEC_GSP2_ENABLE;\ + type DP_SEC_GSP3_ENABLE;\ + type DP_SEC_GSP4_ENABLE;\ + type DP_SEC_GSP5_ENABLE;\ + type DP_SEC_GSP6_ENABLE;\ + type DP_SEC_GSP7_ENABLE;\ + type DP_SEC_MPG_ENABLE;\ + type DP_VID_STREAM_DIS_DEFER;\ + type DP_VID_STREAM_ENABLE;\ + type DP_VID_STREAM_STATUS;\ + type DP_STEER_FIFO_RESET;\ + type DP_VID_M_N_GEN_EN;\ + type DP_VID_N;\ + type DP_VID_M;\ + type DIG_START;\ + type AFMT_AUDIO_SRC_SELECT;\ + type AFMT_AUDIO_CHANNEL_ENABLE;\ + type HDMI_AUDIO_PACKETS_PER_LINE;\ + type HDMI_AUDIO_DELAY_EN;\ + type AFMT_60958_CS_UPDATE;\ + type AFMT_AUDIO_LAYOUT_OVRD;\ + type AFMT_60958_OSF_OVRD;\ + type HDMI_ACR_AUTO_SEND;\ + type HDMI_ACR_SOURCE;\ + type HDMI_ACR_AUDIO_PRIORITY;\ + type HDMI_ACR_CTS_32;\ + type HDMI_ACR_N_32;\ + type HDMI_ACR_CTS_44;\ + type HDMI_ACR_N_44;\ + type HDMI_ACR_CTS_48;\ + type HDMI_ACR_N_48;\ + type AFMT_60958_CS_CHANNEL_NUMBER_L;\ + type AFMT_60958_CS_CLOCK_ACCURACY;\ + type AFMT_60958_CS_CHANNEL_NUMBER_R;\ + type AFMT_60958_CS_CHANNEL_NUMBER_2;\ + type AFMT_60958_CS_CHANNEL_NUMBER_3;\ + type AFMT_60958_CS_CHANNEL_NUMBER_4;\ + type AFMT_60958_CS_CHANNEL_NUMBER_5;\ + type AFMT_60958_CS_CHANNEL_NUMBER_6;\ + type AFMT_60958_CS_CHANNEL_NUMBER_7;\ + type DP_SEC_AUD_N;\ + type DP_SEC_TIMESTAMP_MODE;\ + type DP_SEC_ASP_ENABLE;\ + type DP_SEC_ATP_ENABLE;\ + type DP_SEC_AIP_ENABLE;\ + type DP_SEC_ACM_ENABLE;\ + type AFMT_AUDIO_SAMPLE_SEND;\ + type AFMT_AUDIO_CLOCK_EN;\ + type TMDS_PIXEL_ENCODING;\ + type TMDS_COLOR_FORMAT;\ + type DIG_STEREOSYNC_SELECT;\ + type DIG_STEREOSYNC_GATE_EN;\ + type DP_DB_DISABLE;\ + type DP_MSA_MISC0;\ + type DP_MSA_HTOTAL;\ + type DP_MSA_VTOTAL;\ + type DP_MSA_HSTART;\ + type DP_MSA_VSTART;\ + type DP_MSA_HSYNCWIDTH;\ + type DP_MSA_HSYNCPOLARITY;\ + type DP_MSA_VSYNCWIDTH;\ + type DP_MSA_VSYNCPOLARITY;\ + type DP_MSA_HWIDTH;\ + type DP_MSA_VHEIGHT;\ + type HDMI_DB_DISABLE;\ + type DP_VID_N_MUL;\ + type DP_VID_M_DOUBLE_VALUE_EN + +struct dcn10_stream_encoder_shift { + SE_REG_FIELD_LIST_DCN1_0(uint8_t); +}; + +struct dcn10_stream_encoder_mask { + SE_REG_FIELD_LIST_DCN1_0(uint32_t); +}; + +struct dcn10_stream_encoder { + struct stream_encoder base; + const struct dcn10_stream_enc_registers *regs; + const struct dcn10_stream_encoder_shift *se_shift; + const struct dcn10_stream_encoder_mask *se_mask; +}; + +void dcn10_stream_encoder_construct( + struct dcn10_stream_encoder *enc1, + struct dc_context *ctx, + struct dc_bios *bp, + enum engine_id eng_id, + const struct dcn10_stream_enc_registers *regs, + const struct dcn10_stream_encoder_shift *se_shift, + const struct dcn10_stream_encoder_mask *se_mask); + +void enc1_update_generic_info_packet( + struct dcn10_stream_encoder *enc1, + uint32_t packet_index, + const struct dc_info_packet *info_packet); + +void enc1_stream_encoder_dp_set_stream_attribute( + struct stream_encoder *enc, + struct dc_crtc_timing *crtc_timing, + enum dc_color_space output_color_space); + +void enc1_stream_encoder_hdmi_set_stream_attribute( + struct stream_encoder *enc, + struct dc_crtc_timing *crtc_timing, + int actual_pix_clk_khz, + bool enable_audio); + +void enc1_stream_encoder_dvi_set_stream_attribute( + struct stream_encoder *enc, + struct dc_crtc_timing *crtc_timing, + bool is_dual_link); + +void enc1_stream_encoder_set_mst_bandwidth( + struct stream_encoder *enc, + struct fixed31_32 avg_time_slots_per_mtp); + +void enc1_stream_encoder_update_dp_info_packets( + struct stream_encoder *enc, + const struct encoder_info_frame *info_frame); + +void enc1_stream_encoder_stop_dp_info_packets( + struct stream_encoder *enc); + +void enc1_stream_encoder_dp_blank( + struct stream_encoder *enc); + +void enc1_stream_encoder_dp_unblank( + struct stream_encoder *enc, + const struct encoder_unblank_param *param); + +void enc1_setup_stereo_sync( + struct stream_encoder *enc, + int tg_inst, bool enable); + +void enc1_stream_encoder_set_avmute( + struct stream_encoder *enc, + bool enable); + +void enc1_se_audio_mute_control( + struct stream_encoder *enc, + bool mute); + +void enc1_se_dp_audio_setup( + struct stream_encoder *enc, + unsigned int az_inst, + struct audio_info *info); + +void enc1_se_dp_audio_enable( + struct stream_encoder *enc); + +void enc1_se_dp_audio_disable( + struct stream_encoder *enc); + +void enc1_se_hdmi_audio_setup( + struct stream_encoder *enc, + unsigned int az_inst, + struct audio_info *info, + struct audio_crtc_info *audio_crtc_info); + +void enc1_se_hdmi_audio_disable( + struct stream_encoder *enc); + +#endif /* __DC_STREAM_ENCODER_DCN10_H__ */ diff --git a/drivers/gpu/drm/amd/display/dc/dm_services.h b/drivers/gpu/drm/amd/display/dc/dm_services.h index 22e7ee7dcd26..4ff9b2bba178 100644 --- a/drivers/gpu/drm/amd/display/dc/dm_services.h +++ b/drivers/gpu/drm/amd/display/dc/dm_services.h @@ -341,6 +341,10 @@ bool dm_dmcu_set_pipe(struct dc_context *ctx, unsigned int controller_id); unsigned long long dm_get_timestamp(struct dc_context *ctx); +unsigned long long dm_get_elapse_time_in_ns(struct dc_context *ctx, + unsigned long long current_time_stamp, + unsigned long long last_time_stamp); + /* * performance tracing */ @@ -351,10 +355,6 @@ void dm_perf_trace_timestamp(const char *func_name, unsigned int line); /* * Debug and verification hooks */ -bool dm_helpers_dc_conn_log( - struct dc_context *ctx, - struct log_entry *entry, - enum dc_log_type event); void dm_dtn_log_begin(struct dc_context *ctx); void dm_dtn_log_append_v(struct dc_context *ctx, const char *msg, ...); diff --git a/drivers/gpu/drm/amd/display/dc/dml/display_mode_enums.h b/drivers/gpu/drm/amd/display/dc/dml/display_mode_enums.h index b1ad3553f900..47c19f8fe7d1 100644 --- a/drivers/gpu/drm/amd/display/dc/dml/display_mode_enums.h +++ b/drivers/gpu/drm/amd/display/dc/dml/display_mode_enums.h @@ -108,4 +108,17 @@ enum output_standard { dm_std_uninitialized = 0, dm_std_cvtr2, dm_std_cvt }; +enum mpc_combine_affinity { + dm_mpc_always_when_possible, + dm_mpc_reduce_voltage, + dm_mpc_reduce_voltage_and_clocks +}; + +enum self_refresh_affinity { + dm_try_to_allow_self_refresh_and_mclk_switch, + dm_allow_self_refresh_and_mclk_switch, + dm_allow_self_refresh, + dm_neither_self_refresh_nor_mclk_switch +}; + #endif diff --git a/drivers/gpu/drm/amd/display/dc/dml/display_mode_lib.c b/drivers/gpu/drm/amd/display/dc/dml/display_mode_lib.c index c109b2c34c8f..fd9d97aab071 100644 --- a/drivers/gpu/drm/amd/display/dc/dml/display_mode_lib.c +++ b/drivers/gpu/drm/amd/display/dc/dml/display_mode_lib.c @@ -26,75 +26,89 @@ #include "display_mode_lib.h" #include "dc_features.h" +static const struct _vcs_dpi_ip_params_st dcn1_0_ip = { + .rob_buffer_size_kbytes = 64, + .det_buffer_size_kbytes = 164, + .dpte_buffer_size_in_pte_reqs = 42, + .dpp_output_buffer_pixels = 2560, + .opp_output_buffer_lines = 1, + .pixel_chunk_size_kbytes = 8, + .pte_enable = 1, + .pte_chunk_size_kbytes = 2, + .meta_chunk_size_kbytes = 2, + .writeback_chunk_size_kbytes = 2, + .line_buffer_size_bits = 589824, + .max_line_buffer_lines = 12, + .IsLineBufferBppFixed = 0, + .LineBufferFixedBpp = -1, + .writeback_luma_buffer_size_kbytes = 12, + .writeback_chroma_buffer_size_kbytes = 8, + .max_num_dpp = 4, + .max_num_wb = 2, + .max_dchub_pscl_bw_pix_per_clk = 4, + .max_pscl_lb_bw_pix_per_clk = 2, + .max_lb_vscl_bw_pix_per_clk = 4, + .max_vscl_hscl_bw_pix_per_clk = 4, + .max_hscl_ratio = 4, + .max_vscl_ratio = 4, + .hscl_mults = 4, + .vscl_mults = 4, + .max_hscl_taps = 8, + .max_vscl_taps = 8, + .dispclk_ramp_margin_percent = 1, + .underscan_factor = 1.10, + .min_vblank_lines = 14, + .dppclk_delay_subtotal = 90, + .dispclk_delay_subtotal = 42, + .dcfclk_cstate_latency = 10, + .max_inter_dcn_tile_repeaters = 8, + .can_vstartup_lines_exceed_vsync_plus_back_porch_lines_minus_one = 0, + .bug_forcing_LC_req_same_size_fixed = 0, +}; + +static const struct _vcs_dpi_soc_bounding_box_st dcn1_0_soc = { + .sr_exit_time_us = 9.0, + .sr_enter_plus_exit_time_us = 11.0, + .urgent_latency_us = 4.0, + .writeback_latency_us = 12.0, + .ideal_dram_bw_after_urgent_percent = 80.0, + .max_request_size_bytes = 256, + .downspread_percent = 0.5, + .dram_page_open_time_ns = 50.0, + .dram_rw_turnaround_time_ns = 17.5, + .dram_return_buffer_per_channel_bytes = 8192, + .round_trip_ping_latency_dcfclk_cycles = 128, + .urgent_out_of_order_return_per_channel_bytes = 256, + .channel_interleave_bytes = 256, + .num_banks = 8, + .num_chans = 2, + .vmm_page_size_bytes = 4096, + .dram_clock_change_latency_us = 17.0, + .writeback_dram_clock_change_latency_us = 23.0, + .return_bus_width_bytes = 64, +}; + static void set_soc_bounding_box(struct _vcs_dpi_soc_bounding_box_st *soc, enum dml_project project) { - if (project == DML_PROJECT_RAVEN1) { - soc->sr_exit_time_us = 9.0; - soc->sr_enter_plus_exit_time_us = 11.0; - soc->urgent_latency_us = 4.0; - soc->writeback_latency_us = 12.0; - soc->ideal_dram_bw_after_urgent_percent = 80.0; - soc->max_request_size_bytes = 256; - soc->downspread_percent = 0.5; - soc->dram_page_open_time_ns = 50.0; - soc->dram_rw_turnaround_time_ns = 17.5; - soc->dram_return_buffer_per_channel_bytes = 8192; - soc->round_trip_ping_latency_dcfclk_cycles = 128; - soc->urgent_out_of_order_return_per_channel_bytes = 256; - soc->channel_interleave_bytes = 256; - soc->num_banks = 8; - soc->num_chans = 2; - soc->vmm_page_size_bytes = 4096; - soc->dram_clock_change_latency_us = 17.0; - soc->writeback_dram_clock_change_latency_us = 23.0; - soc->return_bus_width_bytes = 64; - } else { - BREAK_TO_DEBUGGER(); /* Invalid Project Specified */ + switch (project) { + case DML_PROJECT_RAVEN1: + *soc = dcn1_0_soc; + break; + default: + ASSERT(0); + break; } } static void set_ip_params(struct _vcs_dpi_ip_params_st *ip, enum dml_project project) { - if (project == DML_PROJECT_RAVEN1) { - ip->rob_buffer_size_kbytes = 64; - ip->det_buffer_size_kbytes = 164; - ip->dpte_buffer_size_in_pte_reqs = 42; - ip->dpp_output_buffer_pixels = 2560; - ip->opp_output_buffer_lines = 1; - ip->pixel_chunk_size_kbytes = 8; - ip->pte_enable = 1; - ip->pte_chunk_size_kbytes = 2; - ip->meta_chunk_size_kbytes = 2; - ip->writeback_chunk_size_kbytes = 2; - ip->line_buffer_size_bits = 589824; - ip->max_line_buffer_lines = 12; - ip->IsLineBufferBppFixed = 0; - ip->LineBufferFixedBpp = -1; - ip->writeback_luma_buffer_size_kbytes = 12; - ip->writeback_chroma_buffer_size_kbytes = 8; - ip->max_num_dpp = 4; - ip->max_num_wb = 2; - ip->max_dchub_pscl_bw_pix_per_clk = 4; - ip->max_pscl_lb_bw_pix_per_clk = 2; - ip->max_lb_vscl_bw_pix_per_clk = 4; - ip->max_vscl_hscl_bw_pix_per_clk = 4; - ip->max_hscl_ratio = 4; - ip->max_vscl_ratio = 4; - ip->hscl_mults = 4; - ip->vscl_mults = 4; - ip->max_hscl_taps = 8; - ip->max_vscl_taps = 8; - ip->dispclk_ramp_margin_percent = 1; - ip->underscan_factor = 1.10; - ip->min_vblank_lines = 14; - ip->dppclk_delay_subtotal = 90; - ip->dispclk_delay_subtotal = 42; - ip->dcfclk_cstate_latency = 10; - ip->max_inter_dcn_tile_repeaters = 8; - ip->can_vstartup_lines_exceed_vsync_plus_back_porch_lines_minus_one = 0; - ip->bug_forcing_LC_req_same_size_fixed = 0; - } else { - BREAK_TO_DEBUGGER(); /* Invalid Project Specified */ + switch (project) { + case DML_PROJECT_RAVEN1: + *ip = dcn1_0_ip; + break; + default: + ASSERT(0); + break; } } diff --git a/drivers/gpu/drm/amd/display/dc/dml/display_mode_structs.h b/drivers/gpu/drm/amd/display/dc/dml/display_mode_structs.h index 09affa16cc43..7fa0375939ae 100644 --- a/drivers/gpu/drm/amd/display/dc/dml/display_mode_structs.h +++ b/drivers/gpu/drm/amd/display/dc/dml/display_mode_structs.h @@ -25,39 +25,39 @@ #ifndef __DISPLAY_MODE_STRUCTS_H__ #define __DISPLAY_MODE_STRUCTS_H__ -typedef struct _vcs_dpi_voltage_scaling_st voltage_scaling_st; -typedef struct _vcs_dpi_soc_bounding_box_st soc_bounding_box_st; -typedef struct _vcs_dpi_ip_params_st ip_params_st; -typedef struct _vcs_dpi_display_pipe_source_params_st display_pipe_source_params_st; -typedef struct _vcs_dpi_display_output_params_st display_output_params_st; -typedef struct _vcs_dpi_display_bandwidth_st display_bandwidth_st; -typedef struct _vcs_dpi_scaler_ratio_depth_st scaler_ratio_depth_st; -typedef struct _vcs_dpi_scaler_taps_st scaler_taps_st; -typedef struct _vcs_dpi_display_pipe_dest_params_st display_pipe_dest_params_st; -typedef struct _vcs_dpi_display_pipe_params_st display_pipe_params_st; -typedef struct _vcs_dpi_display_clocks_and_cfg_st display_clocks_and_cfg_st; -typedef struct _vcs_dpi_display_e2e_pipe_params_st display_e2e_pipe_params_st; -typedef struct _vcs_dpi_dchub_buffer_sizing_st dchub_buffer_sizing_st; -typedef struct _vcs_dpi_watermarks_perf_st watermarks_perf_st; -typedef struct _vcs_dpi_cstate_pstate_watermarks_st cstate_pstate_watermarks_st; -typedef struct _vcs_dpi_wm_calc_pipe_params_st wm_calc_pipe_params_st; -typedef struct _vcs_dpi_vratio_pre_st vratio_pre_st; -typedef struct _vcs_dpi_display_data_rq_misc_params_st display_data_rq_misc_params_st; -typedef struct _vcs_dpi_display_data_rq_sizing_params_st display_data_rq_sizing_params_st; -typedef struct _vcs_dpi_display_data_rq_dlg_params_st display_data_rq_dlg_params_st; -typedef struct _vcs_dpi_display_cur_rq_dlg_params_st display_cur_rq_dlg_params_st; -typedef struct _vcs_dpi_display_rq_dlg_params_st display_rq_dlg_params_st; -typedef struct _vcs_dpi_display_rq_sizing_params_st display_rq_sizing_params_st; -typedef struct _vcs_dpi_display_rq_misc_params_st display_rq_misc_params_st; -typedef struct _vcs_dpi_display_rq_params_st display_rq_params_st; -typedef struct _vcs_dpi_display_dlg_regs_st display_dlg_regs_st; -typedef struct _vcs_dpi_display_ttu_regs_st display_ttu_regs_st; -typedef struct _vcs_dpi_display_data_rq_regs_st display_data_rq_regs_st; -typedef struct _vcs_dpi_display_rq_regs_st display_rq_regs_st; -typedef struct _vcs_dpi_display_dlg_sys_params_st display_dlg_sys_params_st; -typedef struct _vcs_dpi_display_dlg_prefetch_param_st display_dlg_prefetch_param_st; -typedef struct _vcs_dpi_display_pipe_clock_st display_pipe_clock_st; -typedef struct _vcs_dpi_display_arb_params_st display_arb_params_st; +typedef struct _vcs_dpi_voltage_scaling_st voltage_scaling_st; +typedef struct _vcs_dpi_soc_bounding_box_st soc_bounding_box_st; +typedef struct _vcs_dpi_ip_params_st ip_params_st; +typedef struct _vcs_dpi_display_pipe_source_params_st display_pipe_source_params_st; +typedef struct _vcs_dpi_display_output_params_st display_output_params_st; +typedef struct _vcs_dpi_display_bandwidth_st display_bandwidth_st; +typedef struct _vcs_dpi_scaler_ratio_depth_st scaler_ratio_depth_st; +typedef struct _vcs_dpi_scaler_taps_st scaler_taps_st; +typedef struct _vcs_dpi_display_pipe_dest_params_st display_pipe_dest_params_st; +typedef struct _vcs_dpi_display_pipe_params_st display_pipe_params_st; +typedef struct _vcs_dpi_display_clocks_and_cfg_st display_clocks_and_cfg_st; +typedef struct _vcs_dpi_display_e2e_pipe_params_st display_e2e_pipe_params_st; +typedef struct _vcs_dpi_dchub_buffer_sizing_st dchub_buffer_sizing_st; +typedef struct _vcs_dpi_watermarks_perf_st watermarks_perf_st; +typedef struct _vcs_dpi_cstate_pstate_watermarks_st cstate_pstate_watermarks_st; +typedef struct _vcs_dpi_wm_calc_pipe_params_st wm_calc_pipe_params_st; +typedef struct _vcs_dpi_vratio_pre_st vratio_pre_st; +typedef struct _vcs_dpi_display_data_rq_misc_params_st display_data_rq_misc_params_st; +typedef struct _vcs_dpi_display_data_rq_sizing_params_st display_data_rq_sizing_params_st; +typedef struct _vcs_dpi_display_data_rq_dlg_params_st display_data_rq_dlg_params_st; +typedef struct _vcs_dpi_display_cur_rq_dlg_params_st display_cur_rq_dlg_params_st; +typedef struct _vcs_dpi_display_rq_dlg_params_st display_rq_dlg_params_st; +typedef struct _vcs_dpi_display_rq_sizing_params_st display_rq_sizing_params_st; +typedef struct _vcs_dpi_display_rq_misc_params_st display_rq_misc_params_st; +typedef struct _vcs_dpi_display_rq_params_st display_rq_params_st; +typedef struct _vcs_dpi_display_dlg_regs_st display_dlg_regs_st; +typedef struct _vcs_dpi_display_ttu_regs_st display_ttu_regs_st; +typedef struct _vcs_dpi_display_data_rq_regs_st display_data_rq_regs_st; +typedef struct _vcs_dpi_display_rq_regs_st display_rq_regs_st; +typedef struct _vcs_dpi_display_dlg_sys_params_st display_dlg_sys_params_st; +typedef struct _vcs_dpi_display_dlg_prefetch_param_st display_dlg_prefetch_param_st; +typedef struct _vcs_dpi_display_pipe_clock_st display_pipe_clock_st; +typedef struct _vcs_dpi_display_arb_params_st display_arb_params_st; struct _vcs_dpi_voltage_scaling_st { int state; @@ -72,89 +72,107 @@ struct _vcs_dpi_voltage_scaling_st { double dppclk_mhz; }; -struct _vcs_dpi_soc_bounding_box_st { - double sr_exit_time_us; - double sr_enter_plus_exit_time_us; - double urgent_latency_us; - double writeback_latency_us; - double ideal_dram_bw_after_urgent_percent; - unsigned int max_request_size_bytes; - double downspread_percent; - double dram_page_open_time_ns; - double dram_rw_turnaround_time_ns; - double dram_return_buffer_per_channel_bytes; - double dram_channel_width_bytes; +struct _vcs_dpi_soc_bounding_box_st { + double sr_exit_time_us; + double sr_enter_plus_exit_time_us; + double urgent_latency_us; + double urgent_latency_pixel_data_only_us; + double urgent_latency_pixel_mixed_with_vm_data_us; + double urgent_latency_vm_data_only_us; + double writeback_latency_us; + double ideal_dram_bw_after_urgent_percent; + double pct_ideal_dram_sdp_bw_after_urgent_pixel_only; // PercentOfIdealDRAMFabricAndSDPPortBWReceivedAfterUrgLatencyPixelDataOnly + double pct_ideal_dram_sdp_bw_after_urgent_pixel_and_vm; + double pct_ideal_dram_sdp_bw_after_urgent_vm_only; + double max_avg_sdp_bw_use_normal_percent; + double max_avg_dram_bw_use_normal_percent; + unsigned int max_request_size_bytes; + double downspread_percent; + double dram_page_open_time_ns; + double dram_rw_turnaround_time_ns; + double dram_return_buffer_per_channel_bytes; + double dram_channel_width_bytes; double fabric_datapath_to_dcn_data_return_bytes; double dcn_downspread_percent; double dispclk_dppclk_vco_speed_mhz; double dfs_vco_period_ps; - unsigned int round_trip_ping_latency_dcfclk_cycles; - unsigned int urgent_out_of_order_return_per_channel_bytes; - unsigned int channel_interleave_bytes; - unsigned int num_banks; - unsigned int num_chans; - unsigned int vmm_page_size_bytes; - double dram_clock_change_latency_us; - double writeback_dram_clock_change_latency_us; - unsigned int return_bus_width_bytes; - unsigned int voltage_override; - double xfc_bus_transport_time_us; - double xfc_xbuf_latency_tolerance_us; + unsigned int urgent_out_of_order_return_per_channel_pixel_only_bytes; + unsigned int urgent_out_of_order_return_per_channel_pixel_and_vm_bytes; + unsigned int urgent_out_of_order_return_per_channel_vm_only_bytes; + unsigned int round_trip_ping_latency_dcfclk_cycles; + unsigned int urgent_out_of_order_return_per_channel_bytes; + unsigned int channel_interleave_bytes; + unsigned int num_banks; + unsigned int num_chans; + unsigned int vmm_page_size_bytes; + double dram_clock_change_latency_us; + double writeback_dram_clock_change_latency_us; + unsigned int return_bus_width_bytes; + unsigned int voltage_override; + double xfc_bus_transport_time_us; + double xfc_xbuf_latency_tolerance_us; + int use_urgent_burst_bw; struct _vcs_dpi_voltage_scaling_st clock_limits[7]; }; -struct _vcs_dpi_ip_params_st { - unsigned int max_inter_dcn_tile_repeaters; - unsigned int num_dsc; - unsigned int odm_capable; - unsigned int rob_buffer_size_kbytes; - unsigned int det_buffer_size_kbytes; - unsigned int dpte_buffer_size_in_pte_reqs; - unsigned int pde_proc_buffer_size_64k_reqs; - unsigned int dpp_output_buffer_pixels; - unsigned int opp_output_buffer_lines; - unsigned int pixel_chunk_size_kbytes; - unsigned char pte_enable; - unsigned int pte_chunk_size_kbytes; - unsigned int meta_chunk_size_kbytes; - unsigned int writeback_chunk_size_kbytes; - unsigned int line_buffer_size_bits; - unsigned int max_line_buffer_lines; - unsigned int writeback_luma_buffer_size_kbytes; - unsigned int writeback_chroma_buffer_size_kbytes; - unsigned int writeback_chroma_line_buffer_width_pixels; - unsigned int max_page_table_levels; - unsigned int max_num_dpp; - unsigned int max_num_otg; - unsigned int cursor_chunk_size; - unsigned int cursor_buffer_size; - unsigned int max_num_wb; - unsigned int max_dchub_pscl_bw_pix_per_clk; - unsigned int max_pscl_lb_bw_pix_per_clk; - unsigned int max_lb_vscl_bw_pix_per_clk; - unsigned int max_vscl_hscl_bw_pix_per_clk; - double max_hscl_ratio; - double max_vscl_ratio; - unsigned int hscl_mults; - unsigned int vscl_mults; - unsigned int max_hscl_taps; - unsigned int max_vscl_taps; - unsigned int xfc_supported; - unsigned int xfc_fill_constant_bytes; - double dispclk_ramp_margin_percent; - double xfc_fill_bw_overhead_percent; - double underscan_factor; - unsigned int min_vblank_lines; - unsigned int dppclk_delay_subtotal; - unsigned int dispclk_delay_subtotal; - unsigned int dcfclk_cstate_latency; - unsigned int dppclk_delay_scl; - unsigned int dppclk_delay_scl_lb_only; - unsigned int dppclk_delay_cnvc_formatter; - unsigned int dppclk_delay_cnvc_cursor; - unsigned int is_line_buffer_bpp_fixed; - unsigned int line_buffer_fixed_bpp; - unsigned int dcc_supported; +struct _vcs_dpi_ip_params_st { + bool gpuvm_enable; + bool hostvm_enable; + unsigned int gpuvm_max_page_table_levels; + unsigned int hostvm_max_page_table_levels; + unsigned int hostvm_cached_page_table_levels; + unsigned int pte_group_size_bytes; + unsigned int max_inter_dcn_tile_repeaters; + unsigned int num_dsc; + unsigned int odm_capable; + unsigned int rob_buffer_size_kbytes; + unsigned int det_buffer_size_kbytes; + unsigned int dpte_buffer_size_in_pte_reqs; + unsigned int pde_proc_buffer_size_64k_reqs; + unsigned int dpp_output_buffer_pixels; + unsigned int opp_output_buffer_lines; + unsigned int pixel_chunk_size_kbytes; + unsigned char pte_enable; + unsigned int pte_chunk_size_kbytes; + unsigned int meta_chunk_size_kbytes; + unsigned int writeback_chunk_size_kbytes; + unsigned int line_buffer_size_bits; + unsigned int max_line_buffer_lines; + unsigned int writeback_luma_buffer_size_kbytes; + unsigned int writeback_chroma_buffer_size_kbytes; + unsigned int writeback_chroma_line_buffer_width_pixels; + unsigned int max_page_table_levels; + unsigned int max_num_dpp; + unsigned int max_num_otg; + unsigned int cursor_chunk_size; + unsigned int cursor_buffer_size; + unsigned int max_num_wb; + unsigned int max_dchub_pscl_bw_pix_per_clk; + unsigned int max_pscl_lb_bw_pix_per_clk; + unsigned int max_lb_vscl_bw_pix_per_clk; + unsigned int max_vscl_hscl_bw_pix_per_clk; + double max_hscl_ratio; + double max_vscl_ratio; + unsigned int hscl_mults; + unsigned int vscl_mults; + unsigned int max_hscl_taps; + unsigned int max_vscl_taps; + unsigned int xfc_supported; + unsigned int xfc_fill_constant_bytes; + double dispclk_ramp_margin_percent; + double xfc_fill_bw_overhead_percent; + double underscan_factor; + unsigned int min_vblank_lines; + unsigned int dppclk_delay_subtotal; + unsigned int dispclk_delay_subtotal; + unsigned int dcfclk_cstate_latency; + unsigned int dppclk_delay_scl; + unsigned int dppclk_delay_scl_lb_only; + unsigned int dppclk_delay_cnvc_formatter; + unsigned int dppclk_delay_cnvc_cursor; + unsigned int is_line_buffer_bpp_fixed; + unsigned int line_buffer_fixed_bpp; + unsigned int dcc_supported; unsigned int IsLineBufferBppFixed; unsigned int LineBufferFixedBpp; @@ -169,41 +187,45 @@ struct _vcs_dpi_display_xfc_params_st { int xfc_slv_chunk_size_bytes; }; -struct _vcs_dpi_display_pipe_source_params_st { - int source_format; - unsigned char dcc; - unsigned int dcc_override; - unsigned int dcc_rate; - unsigned char dcc_use_global; - unsigned char vm; - unsigned char vm_levels_force_en; - unsigned int vm_levels_force; - int source_scan; - int sw_mode; - int macro_tile_size; - unsigned char is_display_sw; - unsigned int viewport_width; - unsigned int viewport_height; - unsigned int viewport_y_y; - unsigned int viewport_y_c; - unsigned int viewport_width_c; - unsigned int viewport_height_c; - unsigned int data_pitch; - unsigned int data_pitch_c; - unsigned int meta_pitch; - unsigned int meta_pitch_c; - unsigned int cur0_src_width; - int cur0_bpp; - unsigned int cur1_src_width; - int cur1_bpp; - int num_cursors; - unsigned char is_hsplit; - unsigned char dynamic_metadata_enable; - unsigned int dynamic_metadata_lines_before_active; - unsigned int dynamic_metadata_xmit_bytes; - unsigned int hsplit_grp; - unsigned char xfc_enable; - unsigned char xfc_slave; +struct _vcs_dpi_display_pipe_source_params_st { + int source_format; + unsigned char dcc; + unsigned int dcc_override; + unsigned int dcc_rate; + unsigned char dcc_use_global; + unsigned char vm; + bool gpuvm; // gpuvm enabled + bool hostvm; // hostvm enabled + bool gpuvm_levels_force_en; + unsigned int gpuvm_levels_force; + bool hostvm_levels_force_en; + unsigned int hostvm_levels_force; + int source_scan; + int sw_mode; + int macro_tile_size; + unsigned char is_display_sw; + unsigned int viewport_width; + unsigned int viewport_height; + unsigned int viewport_y_y; + unsigned int viewport_y_c; + unsigned int viewport_width_c; + unsigned int viewport_height_c; + unsigned int data_pitch; + unsigned int data_pitch_c; + unsigned int meta_pitch; + unsigned int meta_pitch_c; + unsigned int cur0_src_width; + int cur0_bpp; + unsigned int cur1_src_width; + int cur1_bpp; + int num_cursors; + unsigned char is_hsplit; + unsigned char dynamic_metadata_enable; + unsigned int dynamic_metadata_lines_before_active; + unsigned int dynamic_metadata_xmit_bytes; + unsigned int hsplit_grp; + unsigned char xfc_enable; + unsigned char xfc_slave; struct _vcs_dpi_display_xfc_params_st xfc_params; }; struct writeback_st { @@ -215,338 +237,339 @@ struct writeback_st { int wb_vtaps_luma; int wb_htaps_chroma; int wb_vtaps_chroma; - int wb_hratio; - int wb_vratio; -}; - -struct _vcs_dpi_display_output_params_st { - int dp_lanes; - int output_bpp; - int dsc_enable; - int wb_enable; - int opp_input_bpc; - int output_type; - int output_format; - int output_standard; - int dsc_slices; + double wb_hratio; + double wb_vratio; +}; + +struct _vcs_dpi_display_output_params_st { + int dp_lanes; + int output_bpp; + int dsc_enable; + int wb_enable; + int num_active_wb; + int output_bpc; + int output_type; + int output_format; + int output_standard; + int dsc_slices; struct writeback_st wb; }; -struct _vcs_dpi_display_bandwidth_st { - double total_bw_consumed_gbps; - double guaranteed_urgent_return_bw_gbps; -}; - -struct _vcs_dpi_scaler_ratio_depth_st { - double hscl_ratio; - double vscl_ratio; - double hscl_ratio_c; - double vscl_ratio_c; - double vinit; - double vinit_c; - double vinit_bot; - double vinit_bot_c; - int lb_depth; - int scl_enable; -}; - -struct _vcs_dpi_scaler_taps_st { - unsigned int htaps; - unsigned int vtaps; - unsigned int htaps_c; - unsigned int vtaps_c; -}; - -struct _vcs_dpi_display_pipe_dest_params_st { - unsigned int recout_width; - unsigned int recout_height; - unsigned int full_recout_width; - unsigned int full_recout_height; - unsigned int hblank_start; - unsigned int hblank_end; - unsigned int vblank_start; - unsigned int vblank_end; - unsigned int htotal; - unsigned int vtotal; - unsigned int vactive; - unsigned int hactive; - unsigned int vstartup_start; - unsigned int vupdate_offset; - unsigned int vupdate_width; - unsigned int vready_offset; - unsigned char interlaced; - unsigned char underscan; - double pixel_rate_mhz; - unsigned char synchronized_vblank_all_planes; - unsigned char otg_inst; - unsigned char odm_split_cnt; - unsigned char odm_combine; -}; - -struct _vcs_dpi_display_pipe_params_st { - display_pipe_source_params_st src; - display_pipe_dest_params_st dest; - scaler_ratio_depth_st scale_ratio_depth; - scaler_taps_st scale_taps; -}; - -struct _vcs_dpi_display_clocks_and_cfg_st { - int voltage; - double dppclk_mhz; - double refclk_mhz; - double dispclk_mhz; - double dcfclk_mhz; - double socclk_mhz; -}; - -struct _vcs_dpi_display_e2e_pipe_params_st { - display_pipe_params_st pipe; - display_output_params_st dout; - display_clocks_and_cfg_st clks_cfg; -}; - -struct _vcs_dpi_dchub_buffer_sizing_st { - unsigned int swath_width_y; - unsigned int swath_height_y; - unsigned int swath_height_c; - unsigned int detail_buffer_size_y; -}; - -struct _vcs_dpi_watermarks_perf_st { - double stutter_eff_in_active_region_percent; - double urgent_latency_supported_us; - double non_urgent_latency_supported_us; - double dram_clock_change_margin_us; - double dram_access_eff_percent; -}; - -struct _vcs_dpi_cstate_pstate_watermarks_st { - double cstate_exit_us; - double cstate_enter_plus_exit_us; - double pstate_change_us; -}; - -struct _vcs_dpi_wm_calc_pipe_params_st { - unsigned int num_dpp; - int voltage; - int output_type; - double dcfclk_mhz; - double socclk_mhz; - double dppclk_mhz; - double pixclk_mhz; - unsigned char interlace_en; - unsigned char pte_enable; - unsigned char dcc_enable; - double dcc_rate; - double bytes_per_pixel_c; - double bytes_per_pixel_y; - unsigned int swath_width_y; - unsigned int swath_height_y; - unsigned int swath_height_c; - unsigned int det_buffer_size_y; - double h_ratio; - double v_ratio; - unsigned int h_taps; - unsigned int h_total; - unsigned int v_total; - unsigned int v_active; - unsigned int e2e_index; - double display_pipe_line_delivery_time; - double read_bw; - unsigned int lines_in_det_y; - unsigned int lines_in_det_y_rounded_down_to_swath; - double full_det_buffering_time; - double dcfclk_deepsleep_mhz_per_plane; -}; - -struct _vcs_dpi_vratio_pre_st { - double vratio_pre_l; - double vratio_pre_c; -}; - -struct _vcs_dpi_display_data_rq_misc_params_st { - unsigned int full_swath_bytes; - unsigned int stored_swath_bytes; - unsigned int blk256_height; - unsigned int blk256_width; - unsigned int req_height; - unsigned int req_width; -}; - -struct _vcs_dpi_display_data_rq_sizing_params_st { - unsigned int chunk_bytes; - unsigned int min_chunk_bytes; - unsigned int meta_chunk_bytes; - unsigned int min_meta_chunk_bytes; - unsigned int mpte_group_bytes; - unsigned int dpte_group_bytes; -}; - -struct _vcs_dpi_display_data_rq_dlg_params_st { - unsigned int swath_width_ub; - unsigned int swath_height; - unsigned int req_per_swath_ub; - unsigned int meta_pte_bytes_per_frame_ub; - unsigned int dpte_req_per_row_ub; - unsigned int dpte_groups_per_row_ub; - unsigned int dpte_row_height; - unsigned int dpte_bytes_per_row_ub; - unsigned int meta_chunks_per_row_ub; - unsigned int meta_req_per_row_ub; - unsigned int meta_row_height; - unsigned int meta_bytes_per_row_ub; -}; - -struct _vcs_dpi_display_cur_rq_dlg_params_st { - unsigned char enable; - unsigned int swath_height; - unsigned int req_per_line; -}; - -struct _vcs_dpi_display_rq_dlg_params_st { - display_data_rq_dlg_params_st rq_l; - display_data_rq_dlg_params_st rq_c; - display_cur_rq_dlg_params_st rq_cur0; -}; - -struct _vcs_dpi_display_rq_sizing_params_st { - display_data_rq_sizing_params_st rq_l; - display_data_rq_sizing_params_st rq_c; -}; - -struct _vcs_dpi_display_rq_misc_params_st { - display_data_rq_misc_params_st rq_l; - display_data_rq_misc_params_st rq_c; -}; - -struct _vcs_dpi_display_rq_params_st { - unsigned char yuv420; - unsigned char yuv420_10bpc; - display_rq_misc_params_st misc; - display_rq_sizing_params_st sizing; - display_rq_dlg_params_st dlg; -}; - -struct _vcs_dpi_display_dlg_regs_st { - unsigned int refcyc_h_blank_end; - unsigned int dlg_vblank_end; - unsigned int min_dst_y_next_start; - unsigned int refcyc_per_htotal; - unsigned int refcyc_x_after_scaler; - unsigned int dst_y_after_scaler; - unsigned int dst_y_prefetch; - unsigned int dst_y_per_vm_vblank; - unsigned int dst_y_per_row_vblank; - unsigned int dst_y_per_vm_flip; - unsigned int dst_y_per_row_flip; - unsigned int ref_freq_to_pix_freq; - unsigned int vratio_prefetch; - unsigned int vratio_prefetch_c; - unsigned int refcyc_per_pte_group_vblank_l; - unsigned int refcyc_per_pte_group_vblank_c; - unsigned int refcyc_per_meta_chunk_vblank_l; - unsigned int refcyc_per_meta_chunk_vblank_c; - unsigned int refcyc_per_pte_group_flip_l; - unsigned int refcyc_per_pte_group_flip_c; - unsigned int refcyc_per_meta_chunk_flip_l; - unsigned int refcyc_per_meta_chunk_flip_c; - unsigned int dst_y_per_pte_row_nom_l; - unsigned int dst_y_per_pte_row_nom_c; - unsigned int refcyc_per_pte_group_nom_l; - unsigned int refcyc_per_pte_group_nom_c; - unsigned int dst_y_per_meta_row_nom_l; - unsigned int dst_y_per_meta_row_nom_c; - unsigned int refcyc_per_meta_chunk_nom_l; - unsigned int refcyc_per_meta_chunk_nom_c; - unsigned int refcyc_per_line_delivery_pre_l; - unsigned int refcyc_per_line_delivery_pre_c; - unsigned int refcyc_per_line_delivery_l; - unsigned int refcyc_per_line_delivery_c; - unsigned int chunk_hdl_adjust_cur0; - unsigned int chunk_hdl_adjust_cur1; - unsigned int vready_after_vcount0; - unsigned int dst_y_offset_cur0; - unsigned int dst_y_offset_cur1; - unsigned int xfc_reg_transfer_delay; - unsigned int xfc_reg_precharge_delay; - unsigned int xfc_reg_remote_surface_flip_latency; - unsigned int xfc_reg_prefetch_margin; - unsigned int dst_y_delta_drq_limit; -}; - -struct _vcs_dpi_display_ttu_regs_st { - unsigned int qos_level_low_wm; - unsigned int qos_level_high_wm; - unsigned int min_ttu_vblank; - unsigned int qos_level_flip; - unsigned int refcyc_per_req_delivery_l; - unsigned int refcyc_per_req_delivery_c; - unsigned int refcyc_per_req_delivery_cur0; - unsigned int refcyc_per_req_delivery_cur1; - unsigned int refcyc_per_req_delivery_pre_l; - unsigned int refcyc_per_req_delivery_pre_c; - unsigned int refcyc_per_req_delivery_pre_cur0; - unsigned int refcyc_per_req_delivery_pre_cur1; - unsigned int qos_level_fixed_l; - unsigned int qos_level_fixed_c; - unsigned int qos_level_fixed_cur0; - unsigned int qos_level_fixed_cur1; - unsigned int qos_ramp_disable_l; - unsigned int qos_ramp_disable_c; - unsigned int qos_ramp_disable_cur0; - unsigned int qos_ramp_disable_cur1; -}; - -struct _vcs_dpi_display_data_rq_regs_st { - unsigned int chunk_size; - unsigned int min_chunk_size; - unsigned int meta_chunk_size; - unsigned int min_meta_chunk_size; - unsigned int dpte_group_size; - unsigned int mpte_group_size; - unsigned int swath_height; - unsigned int pte_row_height_linear; -}; - -struct _vcs_dpi_display_rq_regs_st { - display_data_rq_regs_st rq_regs_l; - display_data_rq_regs_st rq_regs_c; - unsigned int drq_expansion_mode; - unsigned int prq_expansion_mode; - unsigned int mrq_expansion_mode; - unsigned int crq_expansion_mode; - unsigned int plane1_base_address; -}; - -struct _vcs_dpi_display_dlg_sys_params_st { - double t_mclk_wm_us; - double t_urg_wm_us; - double t_sr_wm_us; - double t_extra_us; - double mem_trip_us; - double t_srx_delay_us; - double deepsleep_dcfclk_mhz; - double total_flip_bw; - unsigned int total_flip_bytes; -}; - -struct _vcs_dpi_display_dlg_prefetch_param_st { - double prefetch_bw; - unsigned int flip_bytes; -}; - -struct _vcs_dpi_display_pipe_clock_st { - double dcfclk_mhz; - double dispclk_mhz; - double socclk_mhz; - double dscclk_mhz[6]; - double dppclk_mhz[6]; -}; - -struct _vcs_dpi_display_arb_params_st { - int max_req_outstanding; - int min_req_outstanding; - int sat_level_us; +struct _vcs_dpi_display_bandwidth_st { + double total_bw_consumed_gbps; + double guaranteed_urgent_return_bw_gbps; +}; + +struct _vcs_dpi_scaler_ratio_depth_st { + double hscl_ratio; + double vscl_ratio; + double hscl_ratio_c; + double vscl_ratio_c; + double vinit; + double vinit_c; + double vinit_bot; + double vinit_bot_c; + int lb_depth; + int scl_enable; +}; + +struct _vcs_dpi_scaler_taps_st { + unsigned int htaps; + unsigned int vtaps; + unsigned int htaps_c; + unsigned int vtaps_c; +}; + +struct _vcs_dpi_display_pipe_dest_params_st { + unsigned int recout_width; + unsigned int recout_height; + unsigned int full_recout_width; + unsigned int full_recout_height; + unsigned int hblank_start; + unsigned int hblank_end; + unsigned int vblank_start; + unsigned int vblank_end; + unsigned int htotal; + unsigned int vtotal; + unsigned int vactive; + unsigned int hactive; + unsigned int vstartup_start; + unsigned int vupdate_offset; + unsigned int vupdate_width; + unsigned int vready_offset; + unsigned char interlaced; + unsigned char underscan; + double pixel_rate_mhz; + unsigned char synchronized_vblank_all_planes; + unsigned char otg_inst; + unsigned char odm_split_cnt; + unsigned char odm_combine; +}; + +struct _vcs_dpi_display_pipe_params_st { + display_pipe_source_params_st src; + display_pipe_dest_params_st dest; + scaler_ratio_depth_st scale_ratio_depth; + scaler_taps_st scale_taps; +}; + +struct _vcs_dpi_display_clocks_and_cfg_st { + int voltage; + double dppclk_mhz; + double refclk_mhz; + double dispclk_mhz; + double dcfclk_mhz; + double socclk_mhz; +}; + +struct _vcs_dpi_display_e2e_pipe_params_st { + display_pipe_params_st pipe; + display_output_params_st dout; + display_clocks_and_cfg_st clks_cfg; +}; + +struct _vcs_dpi_dchub_buffer_sizing_st { + unsigned int swath_width_y; + unsigned int swath_height_y; + unsigned int swath_height_c; + unsigned int detail_buffer_size_y; +}; + +struct _vcs_dpi_watermarks_perf_st { + double stutter_eff_in_active_region_percent; + double urgent_latency_supported_us; + double non_urgent_latency_supported_us; + double dram_clock_change_margin_us; + double dram_access_eff_percent; +}; + +struct _vcs_dpi_cstate_pstate_watermarks_st { + double cstate_exit_us; + double cstate_enter_plus_exit_us; + double pstate_change_us; +}; + +struct _vcs_dpi_wm_calc_pipe_params_st { + unsigned int num_dpp; + int voltage; + int output_type; + double dcfclk_mhz; + double socclk_mhz; + double dppclk_mhz; + double pixclk_mhz; + unsigned char interlace_en; + unsigned char pte_enable; + unsigned char dcc_enable; + double dcc_rate; + double bytes_per_pixel_c; + double bytes_per_pixel_y; + unsigned int swath_width_y; + unsigned int swath_height_y; + unsigned int swath_height_c; + unsigned int det_buffer_size_y; + double h_ratio; + double v_ratio; + unsigned int h_taps; + unsigned int h_total; + unsigned int v_total; + unsigned int v_active; + unsigned int e2e_index; + double display_pipe_line_delivery_time; + double read_bw; + unsigned int lines_in_det_y; + unsigned int lines_in_det_y_rounded_down_to_swath; + double full_det_buffering_time; + double dcfclk_deepsleep_mhz_per_plane; +}; + +struct _vcs_dpi_vratio_pre_st { + double vratio_pre_l; + double vratio_pre_c; +}; + +struct _vcs_dpi_display_data_rq_misc_params_st { + unsigned int full_swath_bytes; + unsigned int stored_swath_bytes; + unsigned int blk256_height; + unsigned int blk256_width; + unsigned int req_height; + unsigned int req_width; +}; + +struct _vcs_dpi_display_data_rq_sizing_params_st { + unsigned int chunk_bytes; + unsigned int min_chunk_bytes; + unsigned int meta_chunk_bytes; + unsigned int min_meta_chunk_bytes; + unsigned int mpte_group_bytes; + unsigned int dpte_group_bytes; +}; + +struct _vcs_dpi_display_data_rq_dlg_params_st { + unsigned int swath_width_ub; + unsigned int swath_height; + unsigned int req_per_swath_ub; + unsigned int meta_pte_bytes_per_frame_ub; + unsigned int dpte_req_per_row_ub; + unsigned int dpte_groups_per_row_ub; + unsigned int dpte_row_height; + unsigned int dpte_bytes_per_row_ub; + unsigned int meta_chunks_per_row_ub; + unsigned int meta_req_per_row_ub; + unsigned int meta_row_height; + unsigned int meta_bytes_per_row_ub; +}; + +struct _vcs_dpi_display_cur_rq_dlg_params_st { + unsigned char enable; + unsigned int swath_height; + unsigned int req_per_line; +}; + +struct _vcs_dpi_display_rq_dlg_params_st { + display_data_rq_dlg_params_st rq_l; + display_data_rq_dlg_params_st rq_c; + display_cur_rq_dlg_params_st rq_cur0; +}; + +struct _vcs_dpi_display_rq_sizing_params_st { + display_data_rq_sizing_params_st rq_l; + display_data_rq_sizing_params_st rq_c; +}; + +struct _vcs_dpi_display_rq_misc_params_st { + display_data_rq_misc_params_st rq_l; + display_data_rq_misc_params_st rq_c; +}; + +struct _vcs_dpi_display_rq_params_st { + unsigned char yuv420; + unsigned char yuv420_10bpc; + display_rq_misc_params_st misc; + display_rq_sizing_params_st sizing; + display_rq_dlg_params_st dlg; +}; + +struct _vcs_dpi_display_dlg_regs_st { + unsigned int refcyc_h_blank_end; + unsigned int dlg_vblank_end; + unsigned int min_dst_y_next_start; + unsigned int refcyc_per_htotal; + unsigned int refcyc_x_after_scaler; + unsigned int dst_y_after_scaler; + unsigned int dst_y_prefetch; + unsigned int dst_y_per_vm_vblank; + unsigned int dst_y_per_row_vblank; + unsigned int dst_y_per_vm_flip; + unsigned int dst_y_per_row_flip; + unsigned int ref_freq_to_pix_freq; + unsigned int vratio_prefetch; + unsigned int vratio_prefetch_c; + unsigned int refcyc_per_pte_group_vblank_l; + unsigned int refcyc_per_pte_group_vblank_c; + unsigned int refcyc_per_meta_chunk_vblank_l; + unsigned int refcyc_per_meta_chunk_vblank_c; + unsigned int refcyc_per_pte_group_flip_l; + unsigned int refcyc_per_pte_group_flip_c; + unsigned int refcyc_per_meta_chunk_flip_l; + unsigned int refcyc_per_meta_chunk_flip_c; + unsigned int dst_y_per_pte_row_nom_l; + unsigned int dst_y_per_pte_row_nom_c; + unsigned int refcyc_per_pte_group_nom_l; + unsigned int refcyc_per_pte_group_nom_c; + unsigned int dst_y_per_meta_row_nom_l; + unsigned int dst_y_per_meta_row_nom_c; + unsigned int refcyc_per_meta_chunk_nom_l; + unsigned int refcyc_per_meta_chunk_nom_c; + unsigned int refcyc_per_line_delivery_pre_l; + unsigned int refcyc_per_line_delivery_pre_c; + unsigned int refcyc_per_line_delivery_l; + unsigned int refcyc_per_line_delivery_c; + unsigned int chunk_hdl_adjust_cur0; + unsigned int chunk_hdl_adjust_cur1; + unsigned int vready_after_vcount0; + unsigned int dst_y_offset_cur0; + unsigned int dst_y_offset_cur1; + unsigned int xfc_reg_transfer_delay; + unsigned int xfc_reg_precharge_delay; + unsigned int xfc_reg_remote_surface_flip_latency; + unsigned int xfc_reg_prefetch_margin; + unsigned int dst_y_delta_drq_limit; +}; + +struct _vcs_dpi_display_ttu_regs_st { + unsigned int qos_level_low_wm; + unsigned int qos_level_high_wm; + unsigned int min_ttu_vblank; + unsigned int qos_level_flip; + unsigned int refcyc_per_req_delivery_l; + unsigned int refcyc_per_req_delivery_c; + unsigned int refcyc_per_req_delivery_cur0; + unsigned int refcyc_per_req_delivery_cur1; + unsigned int refcyc_per_req_delivery_pre_l; + unsigned int refcyc_per_req_delivery_pre_c; + unsigned int refcyc_per_req_delivery_pre_cur0; + unsigned int refcyc_per_req_delivery_pre_cur1; + unsigned int qos_level_fixed_l; + unsigned int qos_level_fixed_c; + unsigned int qos_level_fixed_cur0; + unsigned int qos_level_fixed_cur1; + unsigned int qos_ramp_disable_l; + unsigned int qos_ramp_disable_c; + unsigned int qos_ramp_disable_cur0; + unsigned int qos_ramp_disable_cur1; +}; + +struct _vcs_dpi_display_data_rq_regs_st { + unsigned int chunk_size; + unsigned int min_chunk_size; + unsigned int meta_chunk_size; + unsigned int min_meta_chunk_size; + unsigned int dpte_group_size; + unsigned int mpte_group_size; + unsigned int swath_height; + unsigned int pte_row_height_linear; +}; + +struct _vcs_dpi_display_rq_regs_st { + display_data_rq_regs_st rq_regs_l; + display_data_rq_regs_st rq_regs_c; + unsigned int drq_expansion_mode; + unsigned int prq_expansion_mode; + unsigned int mrq_expansion_mode; + unsigned int crq_expansion_mode; + unsigned int plane1_base_address; +}; + +struct _vcs_dpi_display_dlg_sys_params_st { + double t_mclk_wm_us; + double t_urg_wm_us; + double t_sr_wm_us; + double t_extra_us; + double mem_trip_us; + double t_srx_delay_us; + double deepsleep_dcfclk_mhz; + double total_flip_bw; + unsigned int total_flip_bytes; +}; + +struct _vcs_dpi_display_dlg_prefetch_param_st { + double prefetch_bw; + unsigned int flip_bytes; +}; + +struct _vcs_dpi_display_pipe_clock_st { + double dcfclk_mhz; + double dispclk_mhz; + double socclk_mhz; + double dscclk_mhz[6]; + double dppclk_mhz[6]; +}; + +struct _vcs_dpi_display_arb_params_st { + int max_req_outstanding; + int min_req_outstanding; + int sat_level_us; }; #endif /*__DISPLAY_MODE_STRUCTS_H__*/ diff --git a/drivers/gpu/drm/amd/display/dc/dml/dml_inline_defs.h b/drivers/gpu/drm/amd/display/dc/dml/dml_inline_defs.h index f9cf08357989..e8ce08567cd8 100644 --- a/drivers/gpu/drm/amd/display/dc/dml/dml_inline_defs.h +++ b/drivers/gpu/drm/amd/display/dc/dml/dml_inline_defs.h @@ -35,6 +35,16 @@ static inline double dml_min(double a, double b) return (double) dcn_bw_min2(a, b); } +static inline double dml_min3(double a, double b, double c) +{ + return dml_min(dml_min(a, b), c); +} + +static inline double dml_min4(double a, double b, double c, double d) +{ + return dml_min(dml_min(a, b), dml_min(c, d)); +} + static inline double dml_max(double a, double b) { return (double) dcn_bw_max2(a, b); diff --git a/drivers/gpu/drm/amd/display/dc/gpio/gpio_service.c b/drivers/gpu/drm/amd/display/dc/gpio/gpio_service.c index 80038e0e610f..ab5483c0c502 100644 --- a/drivers/gpu/drm/amd/display/dc/gpio/gpio_service.c +++ b/drivers/gpu/drm/amd/display/dc/gpio/gpio_service.c @@ -98,7 +98,8 @@ struct gpio_service *dal_gpio_service_create( if (number_of_bits) { uint32_t index_of_uint = 0; - slot = kzalloc(number_of_uints * sizeof(uint32_t), + slot = kcalloc(number_of_uints, + sizeof(uint32_t), GFP_KERNEL); if (!slot) { diff --git a/drivers/gpu/drm/amd/display/dc/gpio/hw_factory.c b/drivers/gpu/drm/amd/display/dc/gpio/hw_factory.c index 87b580fa4bc9..0caee3523017 100644 --- a/drivers/gpu/drm/amd/display/dc/gpio/hw_factory.c +++ b/drivers/gpu/drm/amd/display/dc/gpio/hw_factory.c @@ -75,6 +75,7 @@ bool dal_hw_factory_init( return true; case DCE_VERSION_11_0: case DCE_VERSION_11_2: + case DCE_VERSION_11_22: dal_hw_factory_dce110_init(factory); return true; case DCE_VERSION_12_0: diff --git a/drivers/gpu/drm/amd/display/dc/gpio/hw_translate.c b/drivers/gpu/drm/amd/display/dc/gpio/hw_translate.c index 0ae8ace25739..55c707488541 100644 --- a/drivers/gpu/drm/amd/display/dc/gpio/hw_translate.c +++ b/drivers/gpu/drm/amd/display/dc/gpio/hw_translate.c @@ -72,6 +72,7 @@ bool dal_hw_translate_init( case DCE_VERSION_10_0: case DCE_VERSION_11_0: case DCE_VERSION_11_2: + case DCE_VERSION_11_22: dal_hw_translate_dce110_init(translate); return true; case DCE_VERSION_12_0: diff --git a/drivers/gpu/drm/amd/display/dc/i2caux/dce110/i2c_hw_engine_dce110.c b/drivers/gpu/drm/amd/display/dc/i2caux/dce110/i2c_hw_engine_dce110.c index abd0095ced30..b7256f595052 100644 --- a/drivers/gpu/drm/amd/display/dc/i2caux/dce110/i2c_hw_engine_dce110.c +++ b/drivers/gpu/drm/amd/display/dc/i2caux/dce110/i2c_hw_engine_dce110.c @@ -527,7 +527,7 @@ static void construct( REG_GET(MICROSECOND_TIME_BASE_DIV, XTAL_REF_DIV, &xtal_ref_div); if (xtal_ref_div == 0) { - DC_LOG_WARNING("Invalid base timer divider\n", + DC_LOG_WARNING("Invalid base timer divider [%s]\n", __func__); xtal_ref_div = 2; } diff --git a/drivers/gpu/drm/amd/display/dc/i2caux/i2caux.c b/drivers/gpu/drm/amd/display/dc/i2caux/i2caux.c index 5cbf6626b8d4..14dc8c94d862 100644 --- a/drivers/gpu/drm/amd/display/dc/i2caux/i2caux.c +++ b/drivers/gpu/drm/amd/display/dc/i2caux/i2caux.c @@ -83,6 +83,7 @@ struct i2caux *dal_i2caux_create( case DCE_VERSION_8_3: return dal_i2caux_dce80_create(ctx); case DCE_VERSION_11_2: + case DCE_VERSION_11_22: return dal_i2caux_dce112_create(ctx); case DCE_VERSION_11_0: return dal_i2caux_dce110_create(ctx); diff --git a/drivers/gpu/drm/amd/display/dc/inc/core_types.h b/drivers/gpu/drm/amd/display/dc/inc/core_types.h index 8c51ad70cace..a94942d4e66b 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/core_types.h +++ b/drivers/gpu/drm/amd/display/dc/inc/core_types.h @@ -95,11 +95,6 @@ struct resource_funcs { struct link_encoder *(*link_enc_create)( const struct encoder_init_data *init); - enum dc_status (*validate_guaranteed)( - struct dc *dc, - struct dc_stream_state *stream, - struct dc_state *context); - bool (*validate_bandwidth)( struct dc *dc, struct dc_state *context); @@ -250,6 +245,7 @@ struct dce_bw_output { bool all_displays_in_sync; struct dce_watermarks urgent_wm_ns[MAX_PIPES]; struct dce_watermarks stutter_exit_wm_ns[MAX_PIPES]; + struct dce_watermarks stutter_entry_wm_ns[MAX_PIPES]; struct dce_watermarks nbp_state_change_wm_ns[MAX_PIPES]; int sclk_khz; int sclk_deep_sleep_khz; diff --git a/drivers/gpu/drm/amd/display/dc/inc/dce_calcs.h b/drivers/gpu/drm/amd/display/dc/inc/dce_calcs.h index a9bfe9ff8ce6..eece165206f9 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/dce_calcs.h +++ b/drivers/gpu/drm/amd/display/dc/inc/dce_calcs.h @@ -42,6 +42,8 @@ enum bw_calcs_version { BW_CALCS_VERSION_CARRIZO, BW_CALCS_VERSION_POLARIS10, BW_CALCS_VERSION_POLARIS11, + BW_CALCS_VERSION_POLARIS12, + BW_CALCS_VERSION_VEGAM, BW_CALCS_VERSION_STONEY, BW_CALCS_VERSION_VEGA10 }; diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/dchubbub.h b/drivers/gpu/drm/amd/display/dc/inc/hw/dchubbub.h new file mode 100644 index 000000000000..02f757dd70d4 --- /dev/null +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/dchubbub.h @@ -0,0 +1,64 @@ +/* + * Copyright 2012-15 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * Authors: AMD + * + */ + +#ifndef __DAL_DCHUBBUB_H__ +#define __DAL_DCHUBBUB_H__ + + +enum dcc_control { + dcc_control__256_256_xxx, + dcc_control__128_128_xxx, + dcc_control__256_64_64, +}; + +enum segment_order { + segment_order__na, + segment_order__contiguous, + segment_order__non_contiguous, +}; + + +struct hubbub_funcs { + void (*update_dchub)( + struct hubbub *hubbub, + struct dchub_init_data *dh_data); + + bool (*get_dcc_compression_cap)(struct hubbub *hubbub, + const struct dc_dcc_surface_param *input, + struct dc_surface_dcc_cap *output); + + bool (*dcc_support_swizzle)( + enum swizzle_mode_values swizzle, + unsigned int bytes_per_element, + enum segment_order *segment_order_horz, + enum segment_order *segment_order_vert); + + bool (*dcc_support_pixel_format)( + enum surface_pixel_format format, + unsigned int *bytes_per_element); +}; + + +#endif diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/dpp.h b/drivers/gpu/drm/amd/display/dc/inc/hw/dpp.h index 99995608b620..582458f028f8 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw/dpp.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/dpp.h @@ -44,7 +44,23 @@ struct dpp_grph_csc_adjustment { enum graphics_gamut_adjust_type gamut_adjust_type; }; +struct dcn_dpp_state { + uint32_t igam_lut_mode; + uint32_t igam_input_format; + uint32_t dgam_lut_mode; + uint32_t rgam_lut_mode; + uint32_t gamut_remap_mode; + uint32_t gamut_remap_c11_c12; + uint32_t gamut_remap_c13_c14; + uint32_t gamut_remap_c21_c22; + uint32_t gamut_remap_c23_c24; + uint32_t gamut_remap_c31_c32; + uint32_t gamut_remap_c33_c34; +}; + struct dpp_funcs { + void (*dpp_read_state)(struct dpp *dpp, struct dcn_dpp_state *s); + void (*dpp_reset)(struct dpp *dpp); void (*dpp_set_scaler)(struct dpp *dpp, @@ -117,7 +133,7 @@ struct dpp_funcs { struct dpp *dpp_base, enum surface_pixel_format format, enum expansion_mode mode, - struct csc_transform input_csc_color_matrix, + struct dc_csc_transform input_csc_color_matrix, enum dc_color_space input_color_space); void (*dpp_full_bypass)(struct dpp *dpp_base); diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/hubp.h b/drivers/gpu/drm/amd/display/dc/inc/hw/hubp.h index 9ced254e652c..97df82cddf82 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw/hubp.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/hubp.h @@ -56,7 +56,6 @@ struct hubp { bool power_gated; }; - struct hubp_funcs { void (*hubp_setup)( struct hubp *hubp, @@ -121,6 +120,9 @@ struct hubp_funcs { void (*hubp_clk_cntl)(struct hubp *hubp, bool enable); void (*hubp_vtg_sel)(struct hubp *hubp, uint32_t otg_inst); + void (*hubp_read_state)(struct hubp *hubp); + void (*hubp_disable_control)(struct hubp *hubp, bool disable_hubp); + unsigned int (*hubp_get_underflow_status)(struct hubp *hubp); }; diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/hw_shared.h b/drivers/gpu/drm/amd/display/dc/inc/hw/hw_shared.h index b22158190262..cf7433ebf91a 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw/hw_shared.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/hw_shared.h @@ -140,11 +140,6 @@ enum opp_regamma { OPP_REGAMMA_USER }; -struct csc_transform { - uint16_t matrix[12]; - bool enable_adjustment; -}; - struct dc_bias_and_scale { uint16_t scale_red; uint16_t bias_red; @@ -191,4 +186,9 @@ enum controller_dp_test_pattern { CONTROLLER_DP_TEST_PATTERN_COLORSQUARES_CEA }; +enum dc_lut_mode { + LUT_BYPASS, + LUT_RAM_A, + LUT_RAM_B +}; #endif /* __DAL_HW_SHARED_H__ */ diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/ipp.h b/drivers/gpu/drm/amd/display/dc/inc/hw/ipp.h index 2109eac20a3d..b2fa4c4cd920 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw/ipp.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/ipp.h @@ -87,7 +87,7 @@ struct ipp_funcs { struct input_pixel_processor *ipp, enum surface_pixel_format format, enum expansion_mode mode, - struct csc_transform input_csc_color_matrix, + struct dc_csc_transform input_csc_color_matrix, enum dc_color_space input_color_space); /* DCE function to setup IPP. TODO: see if we can consolidate to setup */ diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/link_encoder.h b/drivers/gpu/drm/amd/display/dc/inc/hw/link_encoder.h index 54d8a1386142..cf6df2e7beb2 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw/link_encoder.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/link_encoder.h @@ -149,6 +149,7 @@ struct link_encoder_funcs { bool connect); void (*enable_hpd)(struct link_encoder *enc); void (*disable_hpd)(struct link_encoder *enc); + bool (*is_dig_enabled)(struct link_encoder *enc); void (*destroy)(struct link_encoder **enc); }; diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/mem_input.h b/drivers/gpu/drm/amd/display/dc/inc/hw/mem_input.h index 3e1e7e6a8792..47f1dc5a43b7 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw/mem_input.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/mem_input.h @@ -104,6 +104,7 @@ struct mem_input_funcs { struct mem_input *mem_input, struct dce_watermarks nbp, struct dce_watermarks stutter, + struct dce_watermarks stutter_enter, struct dce_watermarks urgent, uint32_t total_dest_line_time_ns); diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/mpc.h b/drivers/gpu/drm/amd/display/dc/inc/hw/mpc.h index 23a8d5e53a89..caf74e3c836f 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw/mpc.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/mpc.h @@ -105,7 +105,24 @@ struct mpc { struct mpcc mpcc_array[MAX_MPCC]; }; +struct mpcc_state { + uint32_t opp_id; + uint32_t dpp_id; + uint32_t bot_mpcc_id; + uint32_t mode; + uint32_t alpha_mode; + uint32_t pre_multiplied_alpha; + uint32_t overlap_only; + uint32_t idle; + uint32_t busy; +}; + struct mpc_funcs { + void (*read_mpcc_state)( + struct mpc *mpc, + int mpcc_inst, + struct mpcc_state *s); + /* * Insert DPP into MPC tree based on specified blending position. * Only used for planes that are part of blending chain for OPP output diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/stream_encoder.h b/drivers/gpu/drm/amd/display/dc/inc/hw/stream_encoder.h index b5db1692393c..cfa7ec9517ae 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw/stream_encoder.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/stream_encoder.h @@ -29,31 +29,40 @@ #define STREAM_ENCODER_H_ #include "audio_types.h" +#include "hw_shared.h" struct dc_bios; struct dc_context; struct dc_crtc_timing; -struct encoder_info_packet { - bool valid; - uint8_t hb0; - uint8_t hb1; - uint8_t hb2; - uint8_t hb3; - uint8_t sb[32]; +enum dp_pixel_encoding_type { + DP_PIXEL_ENCODING_TYPE_RGB444 = 0x00000000, + DP_PIXEL_ENCODING_TYPE_YCBCR422 = 0x00000001, + DP_PIXEL_ENCODING_TYPE_YCBCR444 = 0x00000002, + DP_PIXEL_ENCODING_TYPE_RGB_WIDE_GAMUT = 0x00000003, + DP_PIXEL_ENCODING_TYPE_Y_ONLY = 0x00000004, + DP_PIXEL_ENCODING_TYPE_YCBCR420 = 0x00000005 +}; + +enum dp_component_depth { + DP_COMPONENT_PIXEL_DEPTH_6BPC = 0x00000000, + DP_COMPONENT_PIXEL_DEPTH_8BPC = 0x00000001, + DP_COMPONENT_PIXEL_DEPTH_10BPC = 0x00000002, + DP_COMPONENT_PIXEL_DEPTH_12BPC = 0x00000003, + DP_COMPONENT_PIXEL_DEPTH_16BPC = 0x00000004 }; struct encoder_info_frame { /* auxiliary video information */ - struct encoder_info_packet avi; - struct encoder_info_packet gamut; - struct encoder_info_packet vendor; + struct dc_info_packet avi; + struct dc_info_packet gamut; + struct dc_info_packet vendor; /* source product description */ - struct encoder_info_packet spd; + struct dc_info_packet spd; /* video stream configuration */ - struct encoder_info_packet vsc; + struct dc_info_packet vsc; /* HDR Static MetaData */ - struct encoder_info_packet hdrsmd; + struct dc_info_packet hdrsmd; }; struct encoder_unblank_param { @@ -147,6 +156,7 @@ struct stream_encoder_funcs { void (*set_avmute)( struct stream_encoder *enc, bool enable); + }; #endif /* STREAM_ENCODER_H_ */ diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/timing_generator.h b/drivers/gpu/drm/amd/display/dc/inc/hw/timing_generator.h index 3217b5bf6c7a..69cb0a105300 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw/timing_generator.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/timing_generator.h @@ -140,6 +140,9 @@ struct timing_generator_funcs { void (*program_timing)(struct timing_generator *tg, const struct dc_crtc_timing *timing, bool use_vbios); + void (*program_vline_interrupt)(struct timing_generator *optc, + const struct dc_crtc_timing *dc_crtc_timing, + unsigned long long vsync_delta); bool (*enable_crtc)(struct timing_generator *tg); bool (*disable_crtc)(struct timing_generator *tg); bool (*is_counter_moving)(struct timing_generator *tg); diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/transform.h b/drivers/gpu/drm/amd/display/dc/inc/hw/transform.h index c5b3623bcbd9..fecc80c47c26 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw/transform.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/transform.h @@ -252,7 +252,7 @@ struct transform_funcs { struct transform *xfm_base, enum surface_pixel_format format, enum expansion_mode mode, - struct csc_transform input_csc_color_matrix, + struct dc_csc_transform input_csc_color_matrix, enum dc_color_space input_color_space); void (*ipp_full_bypass)(struct transform *xfm_base); diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw_sequencer.h b/drivers/gpu/drm/amd/display/dc/inc/hw_sequencer.h index e764cbad881b..63fc6c499789 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw_sequencer.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw_sequencer.h @@ -32,6 +32,8 @@ #include "inc/hw/link_encoder.h" #include "core_status.h" +#define EDP_BACKLIGHT_RAMP_DISABLE_LEVEL 0xFFFFFFFF + enum pipe_gating_control { PIPE_GATING_CONTROL_DISABLE = 0, PIPE_GATING_CONTROL_ENABLE, @@ -63,6 +65,7 @@ struct dchub_init_data; struct dc_static_screen_events; struct resource_pool; struct resource_context; +struct stream_resource; struct hw_sequencer_funcs { @@ -80,11 +83,6 @@ struct hw_sequencer_funcs { int num_planes, struct dc_state *context); - void (*set_plane_config)( - const struct dc *dc, - struct pipe_ctx *pipe_ctx, - struct resource_context *res_ctx); - void (*program_gamut_remap)( struct pipe_ctx *pipe_ctx); @@ -93,6 +91,12 @@ struct hw_sequencer_funcs { enum dc_color_space colorspace, uint16_t *matrix); + void (*program_output_csc)(struct dc *dc, + struct pipe_ctx *pipe_ctx, + enum dc_color_space colorspace, + uint16_t *matrix, + int opp_id); + void (*update_plane_addr)( const struct dc *dc, struct pipe_ctx *pipe_ctx); @@ -154,6 +158,11 @@ struct hw_sequencer_funcs { struct dc *dc, struct pipe_ctx *pipe, bool lock); + void (*blank_pixel_data)( + struct dc *dc, + struct stream_resource *stream_res, + struct dc_stream_state *stream, + bool blank); void (*set_bandwidth)( struct dc *dc, @@ -169,7 +178,7 @@ struct hw_sequencer_funcs { void (*set_static_screen_control)(struct pipe_ctx **pipe_ctx, int num_pipes, const struct dc_static_screen_events *events); - enum dc_status (*prog_pixclk_crtc_otg)( + enum dc_status (*enable_stream_timing)( struct pipe_ctx *pipe_ctx, struct dc_state *context, struct dc *dc); @@ -201,6 +210,7 @@ struct hw_sequencer_funcs { void (*set_cursor_position)(struct pipe_ctx *pipe); void (*set_cursor_attribute)(struct pipe_ctx *pipe); + }; void color_space_to_black_color( diff --git a/drivers/gpu/drm/amd/display/dc/inc/reg_helper.h b/drivers/gpu/drm/amd/display/dc/inc/reg_helper.h index 77eb72874e90..3306e7b0b3e3 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/reg_helper.h +++ b/drivers/gpu/drm/amd/display/dc/inc/reg_helper.h @@ -183,6 +183,36 @@ FN(reg_name, f4), v4, \ FN(reg_name, f5), v5) +#define REG_GET_6(reg_name, f1, v1, f2, v2, f3, v3, f4, v4, f5, v5, f6, v6) \ + generic_reg_get6(CTX, REG(reg_name), \ + FN(reg_name, f1), v1, \ + FN(reg_name, f2), v2, \ + FN(reg_name, f3), v3, \ + FN(reg_name, f4), v4, \ + FN(reg_name, f5), v5, \ + FN(reg_name, f6), v6) + +#define REG_GET_7(reg_name, f1, v1, f2, v2, f3, v3, f4, v4, f5, v5, f6, v6, f7, v7) \ + generic_reg_get7(CTX, REG(reg_name), \ + FN(reg_name, f1), v1, \ + FN(reg_name, f2), v2, \ + FN(reg_name, f3), v3, \ + FN(reg_name, f4), v4, \ + FN(reg_name, f5), v5, \ + FN(reg_name, f6), v6, \ + FN(reg_name, f7), v7) + +#define REG_GET_8(reg_name, f1, v1, f2, v2, f3, v3, f4, v4, f5, v5, f6, v6, f7, v7, f8, v8) \ + generic_reg_get8(CTX, REG(reg_name), \ + FN(reg_name, f1), v1, \ + FN(reg_name, f2), v2, \ + FN(reg_name, f3), v3, \ + FN(reg_name, f4), v4, \ + FN(reg_name, f5), v5, \ + FN(reg_name, f6), v6, \ + FN(reg_name, f7), v7, \ + FN(reg_name, f8), v8) + /* macro to poll and wait for a register field to read back given value */ #define REG_WAIT(reg_name, field, val, delay_between_poll_us, max_try) \ @@ -389,4 +419,30 @@ uint32_t generic_reg_get5(const struct dc_context *ctx, uint32_t addr, uint8_t shift4, uint32_t mask4, uint32_t *field_value4, uint8_t shift5, uint32_t mask5, uint32_t *field_value5); +uint32_t generic_reg_get6(const struct dc_context *ctx, uint32_t addr, + uint8_t shift1, uint32_t mask1, uint32_t *field_value1, + uint8_t shift2, uint32_t mask2, uint32_t *field_value2, + uint8_t shift3, uint32_t mask3, uint32_t *field_value3, + uint8_t shift4, uint32_t mask4, uint32_t *field_value4, + uint8_t shift5, uint32_t mask5, uint32_t *field_value5, + uint8_t shift6, uint32_t mask6, uint32_t *field_value6); + +uint32_t generic_reg_get7(const struct dc_context *ctx, uint32_t addr, + uint8_t shift1, uint32_t mask1, uint32_t *field_value1, + uint8_t shift2, uint32_t mask2, uint32_t *field_value2, + uint8_t shift3, uint32_t mask3, uint32_t *field_value3, + uint8_t shift4, uint32_t mask4, uint32_t *field_value4, + uint8_t shift5, uint32_t mask5, uint32_t *field_value5, + uint8_t shift6, uint32_t mask6, uint32_t *field_value6, + uint8_t shift7, uint32_t mask7, uint32_t *field_value7); + +uint32_t generic_reg_get8(const struct dc_context *ctx, uint32_t addr, + uint8_t shift1, uint32_t mask1, uint32_t *field_value1, + uint8_t shift2, uint32_t mask2, uint32_t *field_value2, + uint8_t shift3, uint32_t mask3, uint32_t *field_value3, + uint8_t shift4, uint32_t mask4, uint32_t *field_value4, + uint8_t shift5, uint32_t mask5, uint32_t *field_value5, + uint8_t shift6, uint32_t mask6, uint32_t *field_value6, + uint8_t shift7, uint32_t mask7, uint32_t *field_value7, + uint8_t shift8, uint32_t mask8, uint32_t *field_value8); #endif /* DRIVERS_GPU_DRM_AMD_DC_DEV_DC_INC_REG_HELPER_H_ */ diff --git a/drivers/gpu/drm/amd/display/dc/inc/resource.h b/drivers/gpu/drm/amd/display/dc/inc/resource.h index 5467332faf7b..640a647f4611 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/resource.h +++ b/drivers/gpu/drm/amd/display/dc/inc/resource.h @@ -139,10 +139,6 @@ bool resource_validate_attach_surfaces( struct dc_state *context, const struct resource_pool *pool); -void validate_guaranteed_copy_streams( - struct dc_state *context, - int max_streams); - void resource_validate_ctx_update_pointer_after_copy( const struct dc_state *src_ctx, struct dc_state *dst_ctx); diff --git a/drivers/gpu/drm/amd/display/dc/irq_types.h b/drivers/gpu/drm/amd/display/dc/irq_types.h index a506c2e939f5..0b5f3a278c22 100644 --- a/drivers/gpu/drm/amd/display/dc/irq_types.h +++ b/drivers/gpu/drm/amd/display/dc/irq_types.h @@ -26,6 +26,8 @@ #ifndef __DAL_IRQ_TYPES_H__ #define __DAL_IRQ_TYPES_H__ +#include "os_types.h" + struct dc_context; typedef void (*interrupt_handler)(void *); @@ -135,6 +137,13 @@ enum dc_irq_source { DC_IRQ_SOURCE_VBLANK5, DC_IRQ_SOURCE_VBLANK6, + DC_IRQ_SOURCE_DC1_VLINE0, + DC_IRQ_SOURCE_DC2_VLINE0, + DC_IRQ_SOURCE_DC3_VLINE0, + DC_IRQ_SOURCE_DC4_VLINE0, + DC_IRQ_SOURCE_DC5_VLINE0, + DC_IRQ_SOURCE_DC6_VLINE0, + DAL_IRQ_SOURCES_NUMBER }; diff --git a/drivers/gpu/drm/amd/display/include/dal_asic_id.h b/drivers/gpu/drm/amd/display/include/dal_asic_id.h index 9b0a04f99ac8..25029ed42d89 100644 --- a/drivers/gpu/drm/amd/display/include/dal_asic_id.h +++ b/drivers/gpu/drm/amd/display/include/dal_asic_id.h @@ -86,6 +86,7 @@ #define VI_POLARIS10_P_A0 80 #define VI_POLARIS11_M_A0 90 #define VI_POLARIS12_V_A0 100 +#define VI_VEGAM_A0 110 #define VI_UNKNOWN 0xFF @@ -98,7 +99,9 @@ (eChipRev < VI_POLARIS11_M_A0)) #define ASIC_REV_IS_POLARIS11_M(eChipRev) ((eChipRev >= VI_POLARIS11_M_A0) && \ (eChipRev < VI_POLARIS12_V_A0)) -#define ASIC_REV_IS_POLARIS12_V(eChipRev) (eChipRev >= VI_POLARIS12_V_A0) +#define ASIC_REV_IS_POLARIS12_V(eChipRev) ((eChipRev >= VI_POLARIS12_V_A0) && \ + (eChipRev < VI_VEGAM_A0)) +#define ASIC_REV_IS_VEGAM(eChipRev) (eChipRev >= VI_VEGAM_A0) /* DCE11 */ #define CZ_CARRIZO_A0 0x01 @@ -110,17 +113,19 @@ ((rev >= STONEY_A0) && (rev < CZ_UNKNOWN)) /* DCE12 */ +#define AI_UNKNOWN 0xFF #define AI_GREENLAND_P_A0 1 #define AI_GREENLAND_P_A1 2 #define AI_UNKNOWN 0xFF #define AI_VEGA12_P_A0 20 +#define AI_VEGA20_P_A0 40 #define ASICREV_IS_GREENLAND_M(eChipRev) (eChipRev < AI_VEGA12_P_A0) #define ASICREV_IS_GREENLAND_P(eChipRev) (eChipRev < AI_VEGA12_P_A0) -#define ASICREV_IS_VEGA12_P(eChipRev) ((eChipRev >= AI_VEGA12_P_A0) && (eChipRev < AI_UNKNOWN)) -#define ASICREV_IS_VEGA12_p(eChipRev) ((eChipRev >= AI_VEGA12_P_A0) && (eChipRev < AI_UNKNOWN)) +#define ASICREV_IS_VEGA12_P(eChipRev) ((eChipRev >= AI_VEGA12_P_A0) && (eChipRev < AI_VEGA20_P_A0)) +#define ASICREV_IS_VEGA20_P(eChipRev) ((eChipRev >= AI_VEGA20_P_A0) && (eChipRev < AI_UNKNOWN)) /* DCN1_0 */ #define INTERNAL_REV_RAVEN_A0 0x00 /* First spin of Raven */ diff --git a/drivers/gpu/drm/amd/display/include/dal_types.h b/drivers/gpu/drm/amd/display/include/dal_types.h index fa543965feb5..840142b65f8b 100644 --- a/drivers/gpu/drm/amd/display/include/dal_types.h +++ b/drivers/gpu/drm/amd/display/include/dal_types.h @@ -40,6 +40,7 @@ enum dce_version { DCE_VERSION_10_0, DCE_VERSION_11_0, DCE_VERSION_11_2, + DCE_VERSION_11_22, DCE_VERSION_12_0, DCE_VERSION_MAX, DCN_VERSION_1_0, diff --git a/drivers/gpu/drm/amd/display/include/fixed31_32.h b/drivers/gpu/drm/amd/display/include/fixed31_32.h index 0de258622c12..a981b3e99ab3 100644 --- a/drivers/gpu/drm/amd/display/include/fixed31_32.h +++ b/drivers/gpu/drm/amd/display/include/fixed31_32.h @@ -26,9 +26,13 @@ #ifndef __DAL_FIXED31_32_H__ #define __DAL_FIXED31_32_H__ -#include "os_types.h" - #define FIXED31_32_BITS_PER_FRACTIONAL_PART 32 +#ifndef LLONG_MIN +#define LLONG_MIN (1LL<<63) +#endif +#ifndef LLONG_MAX +#define LLONG_MAX (-1LL>>1) +#endif /* * @brief @@ -44,24 +48,25 @@ */ struct fixed31_32 { - int64_t value; + long long value; }; + /* * @brief * Useful constants */ -static const struct fixed31_32 dal_fixed31_32_zero = { 0 }; -static const struct fixed31_32 dal_fixed31_32_epsilon = { 1LL }; -static const struct fixed31_32 dal_fixed31_32_half = { 0x80000000LL }; -static const struct fixed31_32 dal_fixed31_32_one = { 0x100000000LL }; +static const struct fixed31_32 dc_fixpt_zero = { 0 }; +static const struct fixed31_32 dc_fixpt_epsilon = { 1LL }; +static const struct fixed31_32 dc_fixpt_half = { 0x80000000LL }; +static const struct fixed31_32 dc_fixpt_one = { 0x100000000LL }; -static const struct fixed31_32 dal_fixed31_32_pi = { 13493037705LL }; -static const struct fixed31_32 dal_fixed31_32_two_pi = { 26986075409LL }; -static const struct fixed31_32 dal_fixed31_32_e = { 11674931555LL }; -static const struct fixed31_32 dal_fixed31_32_ln2 = { 2977044471LL }; -static const struct fixed31_32 dal_fixed31_32_ln2_div_2 = { 1488522236LL }; +static const struct fixed31_32 dc_fixpt_pi = { 13493037705LL }; +static const struct fixed31_32 dc_fixpt_two_pi = { 26986075409LL }; +static const struct fixed31_32 dc_fixpt_e = { 11674931555LL }; +static const struct fixed31_32 dc_fixpt_ln2 = { 2977044471LL }; +static const struct fixed31_32 dc_fixpt_ln2_div_2 = { 1488522236LL }; /* * @brief @@ -72,24 +77,19 @@ static const struct fixed31_32 dal_fixed31_32_ln2_div_2 = { 1488522236LL }; * @brief * result = numerator / denominator */ -struct fixed31_32 dal_fixed31_32_from_fraction( - int64_t numerator, - int64_t denominator); +struct fixed31_32 dc_fixpt_from_fraction(long long numerator, long long denominator); /* * @brief * result = arg */ -struct fixed31_32 dal_fixed31_32_from_int_nonconst(int64_t arg); -static inline struct fixed31_32 dal_fixed31_32_from_int(int64_t arg) +static inline struct fixed31_32 dc_fixpt_from_int(int arg) { - if (__builtin_constant_p(arg)) { - struct fixed31_32 res; - BUILD_BUG_ON((LONG_MIN > arg) || (arg > LONG_MAX)); - res.value = arg << FIXED31_32_BITS_PER_FRACTIONAL_PART; - return res; - } else - return dal_fixed31_32_from_int_nonconst(arg); + struct fixed31_32 res; + + res.value = (long long) arg << FIXED31_32_BITS_PER_FRACTIONAL_PART; + + return res; } /* @@ -101,7 +101,7 @@ static inline struct fixed31_32 dal_fixed31_32_from_int(int64_t arg) * @brief * result = -arg */ -static inline struct fixed31_32 dal_fixed31_32_neg(struct fixed31_32 arg) +static inline struct fixed31_32 dc_fixpt_neg(struct fixed31_32 arg) { struct fixed31_32 res; @@ -114,10 +114,10 @@ static inline struct fixed31_32 dal_fixed31_32_neg(struct fixed31_32 arg) * @brief * result = abs(arg) := (arg >= 0) ? arg : -arg */ -static inline struct fixed31_32 dal_fixed31_32_abs(struct fixed31_32 arg) +static inline struct fixed31_32 dc_fixpt_abs(struct fixed31_32 arg) { if (arg.value < 0) - return dal_fixed31_32_neg(arg); + return dc_fixpt_neg(arg); else return arg; } @@ -131,8 +131,7 @@ static inline struct fixed31_32 dal_fixed31_32_abs(struct fixed31_32 arg) * @brief * result = arg1 < arg2 */ -static inline bool dal_fixed31_32_lt(struct fixed31_32 arg1, - struct fixed31_32 arg2) +static inline bool dc_fixpt_lt(struct fixed31_32 arg1, struct fixed31_32 arg2) { return arg1.value < arg2.value; } @@ -141,8 +140,7 @@ static inline bool dal_fixed31_32_lt(struct fixed31_32 arg1, * @brief * result = arg1 <= arg2 */ -static inline bool dal_fixed31_32_le(struct fixed31_32 arg1, - struct fixed31_32 arg2) +static inline bool dc_fixpt_le(struct fixed31_32 arg1, struct fixed31_32 arg2) { return arg1.value <= arg2.value; } @@ -151,8 +149,7 @@ static inline bool dal_fixed31_32_le(struct fixed31_32 arg1, * @brief * result = arg1 == arg2 */ -static inline bool dal_fixed31_32_eq(struct fixed31_32 arg1, - struct fixed31_32 arg2) +static inline bool dc_fixpt_eq(struct fixed31_32 arg1, struct fixed31_32 arg2) { return arg1.value == arg2.value; } @@ -161,8 +158,7 @@ static inline bool dal_fixed31_32_eq(struct fixed31_32 arg1, * @brief * result = min(arg1, arg2) := (arg1 <= arg2) ? arg1 : arg2 */ -static inline struct fixed31_32 dal_fixed31_32_min(struct fixed31_32 arg1, - struct fixed31_32 arg2) +static inline struct fixed31_32 dc_fixpt_min(struct fixed31_32 arg1, struct fixed31_32 arg2) { if (arg1.value <= arg2.value) return arg1; @@ -174,8 +170,7 @@ static inline struct fixed31_32 dal_fixed31_32_min(struct fixed31_32 arg1, * @brief * result = max(arg1, arg2) := (arg1 <= arg2) ? arg2 : arg1 */ -static inline struct fixed31_32 dal_fixed31_32_max(struct fixed31_32 arg1, - struct fixed31_32 arg2) +static inline struct fixed31_32 dc_fixpt_max(struct fixed31_32 arg1, struct fixed31_32 arg2) { if (arg1.value <= arg2.value) return arg2; @@ -189,14 +184,14 @@ static inline struct fixed31_32 dal_fixed31_32_max(struct fixed31_32 arg1, * result = | arg, when min_value < arg < max_value * | max_value, when arg >= max_value */ -static inline struct fixed31_32 dal_fixed31_32_clamp( +static inline struct fixed31_32 dc_fixpt_clamp( struct fixed31_32 arg, struct fixed31_32 min_value, struct fixed31_32 max_value) { - if (dal_fixed31_32_le(arg, min_value)) + if (dc_fixpt_le(arg, min_value)) return min_value; - else if (dal_fixed31_32_le(max_value, arg)) + else if (dc_fixpt_le(max_value, arg)) return max_value; else return arg; @@ -211,21 +206,30 @@ static inline struct fixed31_32 dal_fixed31_32_clamp( * @brief * result = arg << shift */ -struct fixed31_32 dal_fixed31_32_shl( - struct fixed31_32 arg, - uint8_t shift); +static inline struct fixed31_32 dc_fixpt_shl(struct fixed31_32 arg, unsigned char shift) +{ + ASSERT(((arg.value >= 0) && (arg.value <= LLONG_MAX >> shift)) || + ((arg.value < 0) && (arg.value >= ~(LLONG_MAX >> shift)))); + + arg.value = arg.value << shift; + + return arg; +} /* * @brief * result = arg >> shift */ -static inline struct fixed31_32 dal_fixed31_32_shr( - struct fixed31_32 arg, - uint8_t shift) +static inline struct fixed31_32 dc_fixpt_shr(struct fixed31_32 arg, unsigned char shift) { - struct fixed31_32 res; - res.value = arg.value >> shift; - return res; + bool negative = arg.value < 0; + + if (negative) + arg.value = -arg.value; + arg.value = arg.value >> shift; + if (negative) + arg.value = -arg.value; + return arg; } /* @@ -237,38 +241,50 @@ static inline struct fixed31_32 dal_fixed31_32_shr( * @brief * result = arg1 + arg2 */ -struct fixed31_32 dal_fixed31_32_add( - struct fixed31_32 arg1, - struct fixed31_32 arg2); +static inline struct fixed31_32 dc_fixpt_add(struct fixed31_32 arg1, struct fixed31_32 arg2) +{ + struct fixed31_32 res; + + ASSERT(((arg1.value >= 0) && (LLONG_MAX - arg1.value >= arg2.value)) || + ((arg1.value < 0) && (LLONG_MIN - arg1.value <= arg2.value))); + + res.value = arg1.value + arg2.value; + + return res; +} /* * @brief * result = arg1 + arg2 */ -static inline struct fixed31_32 dal_fixed31_32_add_int(struct fixed31_32 arg1, - int32_t arg2) +static inline struct fixed31_32 dc_fixpt_add_int(struct fixed31_32 arg1, int arg2) { - return dal_fixed31_32_add(arg1, - dal_fixed31_32_from_int(arg2)); + return dc_fixpt_add(arg1, dc_fixpt_from_int(arg2)); } /* * @brief * result = arg1 - arg2 */ -struct fixed31_32 dal_fixed31_32_sub( - struct fixed31_32 arg1, - struct fixed31_32 arg2); +static inline struct fixed31_32 dc_fixpt_sub(struct fixed31_32 arg1, struct fixed31_32 arg2) +{ + struct fixed31_32 res; + + ASSERT(((arg2.value >= 0) && (LLONG_MIN + arg2.value <= arg1.value)) || + ((arg2.value < 0) && (LLONG_MAX + arg2.value >= arg1.value))); + + res.value = arg1.value - arg2.value; + + return res; +} /* * @brief * result = arg1 - arg2 */ -static inline struct fixed31_32 dal_fixed31_32_sub_int(struct fixed31_32 arg1, - int32_t arg2) +static inline struct fixed31_32 dc_fixpt_sub_int(struct fixed31_32 arg1, int arg2) { - return dal_fixed31_32_sub(arg1, - dal_fixed31_32_from_int(arg2)); + return dc_fixpt_sub(arg1, dc_fixpt_from_int(arg2)); } @@ -281,49 +297,40 @@ static inline struct fixed31_32 dal_fixed31_32_sub_int(struct fixed31_32 arg1, * @brief * result = arg1 * arg2 */ -struct fixed31_32 dal_fixed31_32_mul( - struct fixed31_32 arg1, - struct fixed31_32 arg2); +struct fixed31_32 dc_fixpt_mul(struct fixed31_32 arg1, struct fixed31_32 arg2); /* * @brief * result = arg1 * arg2 */ -static inline struct fixed31_32 dal_fixed31_32_mul_int(struct fixed31_32 arg1, - int32_t arg2) +static inline struct fixed31_32 dc_fixpt_mul_int(struct fixed31_32 arg1, int arg2) { - return dal_fixed31_32_mul(arg1, - dal_fixed31_32_from_int(arg2)); + return dc_fixpt_mul(arg1, dc_fixpt_from_int(arg2)); } /* * @brief * result = square(arg) := arg * arg */ -struct fixed31_32 dal_fixed31_32_sqr( - struct fixed31_32 arg); +struct fixed31_32 dc_fixpt_sqr(struct fixed31_32 arg); /* * @brief * result = arg1 / arg2 */ -static inline struct fixed31_32 dal_fixed31_32_div_int(struct fixed31_32 arg1, - int64_t arg2) +static inline struct fixed31_32 dc_fixpt_div_int(struct fixed31_32 arg1, long long arg2) { - return dal_fixed31_32_from_fraction(arg1.value, - dal_fixed31_32_from_int(arg2).value); + return dc_fixpt_from_fraction(arg1.value, dc_fixpt_from_int(arg2).value); } /* * @brief * result = arg1 / arg2 */ -static inline struct fixed31_32 dal_fixed31_32_div(struct fixed31_32 arg1, - struct fixed31_32 arg2) +static inline struct fixed31_32 dc_fixpt_div(struct fixed31_32 arg1, struct fixed31_32 arg2) { - return dal_fixed31_32_from_fraction(arg1.value, - arg2.value); + return dc_fixpt_from_fraction(arg1.value, arg2.value); } /* @@ -338,8 +345,7 @@ static inline struct fixed31_32 dal_fixed31_32_div(struct fixed31_32 arg1, * @note * No special actions taken in case argument is zero. */ -struct fixed31_32 dal_fixed31_32_recip( - struct fixed31_32 arg); +struct fixed31_32 dc_fixpt_recip(struct fixed31_32 arg); /* * @brief @@ -354,8 +360,7 @@ struct fixed31_32 dal_fixed31_32_recip( * Argument specified in radians, * internally it's normalized to [-2pi...2pi] range. */ -struct fixed31_32 dal_fixed31_32_sinc( - struct fixed31_32 arg); +struct fixed31_32 dc_fixpt_sinc(struct fixed31_32 arg); /* * @brief @@ -365,8 +370,7 @@ struct fixed31_32 dal_fixed31_32_sinc( * Argument specified in radians, * internally it's normalized to [-2pi...2pi] range. */ -struct fixed31_32 dal_fixed31_32_sin( - struct fixed31_32 arg); +struct fixed31_32 dc_fixpt_sin(struct fixed31_32 arg); /* * @brief @@ -378,8 +382,7 @@ struct fixed31_32 dal_fixed31_32_sin( * passing arguments outside that range * will cause incorrect result! */ -struct fixed31_32 dal_fixed31_32_cos( - struct fixed31_32 arg); +struct fixed31_32 dc_fixpt_cos(struct fixed31_32 arg); /* * @brief @@ -393,8 +396,7 @@ struct fixed31_32 dal_fixed31_32_cos( * @note * Currently, function is verified for abs(arg) <= 1. */ -struct fixed31_32 dal_fixed31_32_exp( - struct fixed31_32 arg); +struct fixed31_32 dc_fixpt_exp(struct fixed31_32 arg); /* * @brief @@ -406,8 +408,7 @@ struct fixed31_32 dal_fixed31_32_exp( * Currently, no special actions taken * in case of invalid argument(s). Take care! */ -struct fixed31_32 dal_fixed31_32_log( - struct fixed31_32 arg); +struct fixed31_32 dc_fixpt_log(struct fixed31_32 arg); /* * @brief @@ -421,9 +422,13 @@ struct fixed31_32 dal_fixed31_32_log( * @note * Currently, abs(arg1) should be less than 1. Take care! */ -struct fixed31_32 dal_fixed31_32_pow( - struct fixed31_32 arg1, - struct fixed31_32 arg2); +static inline struct fixed31_32 dc_fixpt_pow(struct fixed31_32 arg1, struct fixed31_32 arg2) +{ + return dc_fixpt_exp( + dc_fixpt_mul( + dc_fixpt_log(arg1), + arg2)); +} /* * @brief @@ -434,22 +439,56 @@ struct fixed31_32 dal_fixed31_32_pow( * @brief * result = floor(arg) := greatest integer lower than or equal to arg */ -int32_t dal_fixed31_32_floor( - struct fixed31_32 arg); +static inline int dc_fixpt_floor(struct fixed31_32 arg) +{ + unsigned long long arg_value = arg.value > 0 ? arg.value : -arg.value; + + if (arg.value >= 0) + return (int)(arg_value >> FIXED31_32_BITS_PER_FRACTIONAL_PART); + else + return -(int)(arg_value >> FIXED31_32_BITS_PER_FRACTIONAL_PART); +} /* * @brief * result = round(arg) := integer nearest to arg */ -int32_t dal_fixed31_32_round( - struct fixed31_32 arg); +static inline int dc_fixpt_round(struct fixed31_32 arg) +{ + unsigned long long arg_value = arg.value > 0 ? arg.value : -arg.value; + + const long long summand = dc_fixpt_half.value; + + ASSERT(LLONG_MAX - (long long)arg_value >= summand); + + arg_value += summand; + + if (arg.value >= 0) + return (int)(arg_value >> FIXED31_32_BITS_PER_FRACTIONAL_PART); + else + return -(int)(arg_value >> FIXED31_32_BITS_PER_FRACTIONAL_PART); +} /* * @brief * result = ceil(arg) := lowest integer greater than or equal to arg */ -int32_t dal_fixed31_32_ceil( - struct fixed31_32 arg); +static inline int dc_fixpt_ceil(struct fixed31_32 arg) +{ + unsigned long long arg_value = arg.value > 0 ? arg.value : -arg.value; + + const long long summand = dc_fixpt_one.value - + dc_fixpt_epsilon.value; + + ASSERT(LLONG_MAX - (long long)arg_value >= summand); + + arg_value += summand; + + if (arg.value >= 0) + return (int)(arg_value >> FIXED31_32_BITS_PER_FRACTIONAL_PART); + else + return -(int)(arg_value >> FIXED31_32_BITS_PER_FRACTIONAL_PART); +} /* the following two function are used in scaler hw programming to convert fixed * point value to format 2 bits from integer part and 19 bits from fractional @@ -457,20 +496,33 @@ int32_t dal_fixed31_32_ceil( * fractional */ -uint32_t dal_fixed31_32_u2d19( - struct fixed31_32 arg); +unsigned int dc_fixpt_u3d19(struct fixed31_32 arg); + +unsigned int dc_fixpt_u2d19(struct fixed31_32 arg); + +unsigned int dc_fixpt_u0d19(struct fixed31_32 arg); -uint32_t dal_fixed31_32_u0d19( - struct fixed31_32 arg); +unsigned int dc_fixpt_clamp_u0d14(struct fixed31_32 arg); +unsigned int dc_fixpt_clamp_u0d10(struct fixed31_32 arg); -uint32_t dal_fixed31_32_clamp_u0d14( - struct fixed31_32 arg); +int dc_fixpt_s4d19(struct fixed31_32 arg); -uint32_t dal_fixed31_32_clamp_u0d10( - struct fixed31_32 arg); +static inline struct fixed31_32 dc_fixpt_truncate(struct fixed31_32 arg, unsigned int frac_bits) +{ + bool negative = arg.value < 0; -int32_t dal_fixed31_32_s4d19( - struct fixed31_32 arg); + if (frac_bits >= FIXED31_32_BITS_PER_FRACTIONAL_PART) { + ASSERT(frac_bits == FIXED31_32_BITS_PER_FRACTIONAL_PART); + return arg; + } + + if (negative) + arg.value = -arg.value; + arg.value &= (~0LL) << (FIXED31_32_BITS_PER_FRACTIONAL_PART - frac_bits); + if (negative) + arg.value = -arg.value; + return arg; +} #endif diff --git a/drivers/gpu/drm/amd/display/include/fixed32_32.h b/drivers/gpu/drm/amd/display/include/fixed32_32.h deleted file mode 100644 index 9c70341fe026..000000000000 --- a/drivers/gpu/drm/amd/display/include/fixed32_32.h +++ /dev/null @@ -1,129 +0,0 @@ -/* - * Copyright 2012-15 Advanced Micro Devices, Inc. - * - * Permission is hereby granted, free of charge, to any person obtaining a - * copy of this software and associated documentation files (the "Software"), - * to deal in the Software without restriction, including without limitation - * the rights to use, copy, modify, merge, publish, distribute, sublicense, - * and/or sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following conditions: - * - * The above copyright notice and this permission notice shall be included in - * all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, - * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL - * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR - * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, - * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - * - * Authors: AMD - * - */ - - -#ifndef __DAL_FIXED32_32_H__ -#define __DAL_FIXED32_32_H__ - -#include "os_types.h" - -struct fixed32_32 { - uint64_t value; -}; - -static const struct fixed32_32 dal_fixed32_32_zero = { 0 }; -static const struct fixed32_32 dal_fixed32_32_one = { 0x100000000LL }; -static const struct fixed32_32 dal_fixed32_32_half = { 0x80000000LL }; - -struct fixed32_32 dal_fixed32_32_from_fraction(uint32_t n, uint32_t d); -static inline struct fixed32_32 dal_fixed32_32_from_int(uint32_t value) -{ - struct fixed32_32 fx; - - fx.value = (uint64_t)value<<32; - return fx; -} - -struct fixed32_32 dal_fixed32_32_add( - struct fixed32_32 lhs, - struct fixed32_32 rhs); -struct fixed32_32 dal_fixed32_32_add_int( - struct fixed32_32 lhs, - uint32_t rhs); -struct fixed32_32 dal_fixed32_32_sub( - struct fixed32_32 lhs, - struct fixed32_32 rhs); -struct fixed32_32 dal_fixed32_32_sub_int( - struct fixed32_32 lhs, - uint32_t rhs); -struct fixed32_32 dal_fixed32_32_mul( - struct fixed32_32 lhs, - struct fixed32_32 rhs); -struct fixed32_32 dal_fixed32_32_mul_int( - struct fixed32_32 lhs, - uint32_t rhs); -struct fixed32_32 dal_fixed32_32_div( - struct fixed32_32 lhs, - struct fixed32_32 rhs); -struct fixed32_32 dal_fixed32_32_div_int( - struct fixed32_32 lhs, - uint32_t rhs); - -static inline struct fixed32_32 dal_fixed32_32_min(struct fixed32_32 lhs, - struct fixed32_32 rhs) -{ - return (lhs.value < rhs.value) ? lhs : rhs; -} - -static inline struct fixed32_32 dal_fixed32_32_max(struct fixed32_32 lhs, - struct fixed32_32 rhs) -{ - return (lhs.value > rhs.value) ? lhs : rhs; -} - -static inline bool dal_fixed32_32_gt(struct fixed32_32 lhs, struct fixed32_32 rhs) -{ - return lhs.value > rhs.value; -} - -static inline bool dal_fixed32_32_gt_int(struct fixed32_32 lhs, uint32_t rhs) -{ - return lhs.value > ((uint64_t)rhs<<32); -} - -static inline bool dal_fixed32_32_lt(struct fixed32_32 lhs, struct fixed32_32 rhs) -{ - return lhs.value < rhs.value; -} - -static inline bool dal_fixed32_32_lt_int(struct fixed32_32 lhs, uint32_t rhs) -{ - return lhs.value < ((uint64_t)rhs<<32); -} - -static inline bool dal_fixed32_32_le(struct fixed32_32 lhs, struct fixed32_32 rhs) -{ - return lhs.value <= rhs.value; -} - -static inline bool dal_fixed32_32_le_int(struct fixed32_32 lhs, uint32_t rhs) -{ - return lhs.value <= ((uint64_t)rhs<<32); -} - -static inline bool dal_fixed32_32_eq(struct fixed32_32 lhs, struct fixed32_32 rhs) -{ - return lhs.value == rhs.value; -} - -uint32_t dal_fixed32_32_ceil(struct fixed32_32 value); -static inline uint32_t dal_fixed32_32_floor(struct fixed32_32 value) -{ - return value.value>>32; -} - -uint32_t dal_fixed32_32_round(struct fixed32_32 value); - -#endif diff --git a/drivers/gpu/drm/amd/display/include/logger_interface.h b/drivers/gpu/drm/amd/display/include/logger_interface.h index 28dee960d509..dc98d6d4b2bd 100644 --- a/drivers/gpu/drm/amd/display/include/logger_interface.h +++ b/drivers/gpu/drm/amd/display/include/logger_interface.h @@ -190,4 +190,13 @@ void context_clock_trace( } \ } while (0) +#define DISPLAY_STATS_BEGIN(entry) \ + dm_logger_open(dc->ctx->logger, &entry, LOG_DISPLAYSTATS) + +#define DISPLAY_STATS(msg, ...) \ + dm_logger_append(&log_entry, msg, ##__VA_ARGS__) + +#define DISPLAY_STATS_END(entry) \ + dm_logger_close(&entry) + #endif /* __DAL_LOGGER_INTERFACE_H__ */ diff --git a/drivers/gpu/drm/amd/display/include/logger_types.h b/drivers/gpu/drm/amd/display/include/logger_types.h index 427796bdc14a..0a540b9897a6 100644 --- a/drivers/gpu/drm/amd/display/include/logger_types.h +++ b/drivers/gpu/drm/amd/display/include/logger_types.h @@ -29,39 +29,39 @@ #include "os_types.h" #define MAX_NAME_LEN 32 -#define DC_LOG_ERROR(a, ...) dm_logger_write(DC_LOGGER, LOG_ERROR, a, ## __VA_ARGS__) -#define DC_LOG_WARNING(a, ...) dm_logger_write(DC_LOGGER, LOG_WARNING, a, ## __VA_ARGS__) -#define DC_LOG_DEBUG(a, ...) dm_logger_write(DC_LOGGER, LOG_DEBUG, a, ## __VA_ARGS__) -#define DC_LOG_DC(a, ...) dm_logger_write(DC_LOGGER, LOG_DC, a, ## __VA_ARGS__) -#define DC_LOG_DTN(a, ...) dm_logger_write(DC_LOGGER, LOG_DTN, a, ## __VA_ARGS__) -#define DC_LOG_SURFACE(a, ...) dm_logger_write(DC_LOGGER, LOG_SURFACE, a, ## __VA_ARGS__) -#define DC_LOG_HW_HOTPLUG(a, ...) dm_logger_write(DC_LOGGER, LOG_HW_HOTPLUG, a, ## __VA_ARGS__) -#define DC_LOG_HW_LINK_TRAINING(a, ...) dm_logger_write(DC_LOGGER, LOG_HW_LINK_TRAINING, a, ## __VA_ARGS__) -#define DC_LOG_HW_SET_MODE(a, ...) dm_logger_write(DC_LOGGER, LOG_HW_SET_MODE, a, ## __VA_ARGS__) -#define DC_LOG_HW_RESUME_S3(a, ...) dm_logger_write(DC_LOGGER, LOG_HW_RESUME_S3, a, ## __VA_ARGS__) -#define DC_LOG_HW_AUDIO(a, ...) dm_logger_write(DC_LOGGER, LOG_HW_AUDIO, a, ## __VA_ARGS__) -#define DC_LOG_HW_HPD_IRQ(a, ...) dm_logger_write(DC_LOGGER, LOG_HW_HPD_IRQ, a, ## __VA_ARGS__) -#define DC_LOG_MST(a, ...) dm_logger_write(DC_LOGGER, LOG_MST, a, ## __VA_ARGS__) -#define DC_LOG_SCALER(a, ...) dm_logger_write(DC_LOGGER, LOG_SCALER, a, ## __VA_ARGS__) -#define DC_LOG_BIOS(a, ...) dm_logger_write(DC_LOGGER, LOG_BIOS, a, ## __VA_ARGS__) -#define DC_LOG_BANDWIDTH_CALCS(a, ...) dm_logger_write(DC_LOGGER, LOG_BANDWIDTH_CALCS, a, ## __VA_ARGS__) -#define DC_LOG_BANDWIDTH_VALIDATION(a, ...) dm_logger_write(DC_LOGGER, LOG_BANDWIDTH_VALIDATION, a, ## __VA_ARGS__) -#define DC_LOG_I2C_AUX(a, ...) dm_logger_write(DC_LOGGER, LOG_I2C_AUX, a, ## __VA_ARGS__) -#define DC_LOG_SYNC(a, ...) dm_logger_write(DC_LOGGER, LOG_SYNC, a, ## __VA_ARGS__) -#define DC_LOG_BACKLIGHT(a, ...) dm_logger_write(DC_LOGGER, LOG_BACKLIGHT, a, ## __VA_ARGS__) -#define DC_LOG_FEATURE_OVERRIDE(a, ...) dm_logger_write(DC_LOGGER, LOG_FEATURE_OVERRIDE, a, ## __VA_ARGS__) -#define DC_LOG_DETECTION_EDID_PARSER(a, ...) dm_logger_write(DC_LOGGER, LOG_DETECTION_EDID_PARSER, a, ## __VA_ARGS__) -#define DC_LOG_DETECTION_DP_CAPS(a, ...) dm_logger_write(DC_LOGGER, LOG_DETECTION_DP_CAPS, a, ## __VA_ARGS__) -#define DC_LOG_RESOURCE(a, ...) dm_logger_write(DC_LOGGER, LOG_RESOURCE, a, ## __VA_ARGS__) -#define DC_LOG_DML(a, ...) dm_logger_write(DC_LOGGER, LOG_DML, a, ## __VA_ARGS__) -#define DC_LOG_EVENT_MODE_SET(a, ...) dm_logger_write(DC_LOGGER, LOG_EVENT_MODE_SET, a, ## __VA_ARGS__) -#define DC_LOG_EVENT_DETECTION(a, ...) dm_logger_write(DC_LOGGER, LOG_EVENT_DETECTION, a, ## __VA_ARGS__) -#define DC_LOG_EVENT_LINK_TRAINING(a, ...) dm_logger_write(DC_LOGGER, LOG_EVENT_LINK_TRAINING, a, ## __VA_ARGS__) -#define DC_LOG_EVENT_LINK_LOSS(a, ...) dm_logger_write(DC_LOGGER, LOG_EVENT_LINK_LOSS, a, ## __VA_ARGS__) -#define DC_LOG_EVENT_UNDERFLOW(a, ...) dm_logger_write(DC_LOGGER, LOG_EVENT_UNDERFLOW, a, ## __VA_ARGS__) -#define DC_LOG_IF_TRACE(a, ...) dm_logger_write(DC_LOGGER, LOG_IF_TRACE, a, ## __VA_ARGS__) -#define DC_LOG_PERF_TRACE(a, ...) dm_logger_write(DC_LOGGER, LOG_PERF_TRACE, a, ## __VA_ARGS__) +#define DC_LOG_ERROR(...) DRM_ERROR(__VA_ARGS__) +#define DC_LOG_WARNING(...) DRM_WARN(__VA_ARGS__) +#define DC_LOG_DEBUG(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_DC(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_DTN(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_SURFACE(...) pr_debug("[SURFACE]:"__VA_ARGS__) +#define DC_LOG_HW_HOTPLUG(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_HW_LINK_TRAINING(...) pr_debug("[HW_LINK_TRAINING]:"__VA_ARGS__) +#define DC_LOG_HW_SET_MODE(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_HW_RESUME_S3(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_HW_AUDIO(...) pr_debug("[HW_AUDIO]:"__VA_ARGS__) +#define DC_LOG_HW_HPD_IRQ(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_MST(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_SCALER(...) pr_debug("[SCALER]:"__VA_ARGS__) +#define DC_LOG_BIOS(...) pr_debug("[BIOS]:"__VA_ARGS__) +#define DC_LOG_BANDWIDTH_CALCS(...) pr_debug("[BANDWIDTH_CALCS]:"__VA_ARGS__) +#define DC_LOG_BANDWIDTH_VALIDATION(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_I2C_AUX(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_SYNC(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_BACKLIGHT(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_FEATURE_OVERRIDE(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_DETECTION_EDID_PARSER(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_DETECTION_DP_CAPS(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_RESOURCE(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_DML(...) pr_debug("[DML]:"__VA_ARGS__) +#define DC_LOG_EVENT_MODE_SET(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_EVENT_DETECTION(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_EVENT_LINK_TRAINING(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_EVENT_LINK_LOSS(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_EVENT_UNDERFLOW(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_IF_TRACE(...) pr_debug("[IF_TRACE]:"__VA_ARGS__) +#define DC_LOG_PERF_TRACE(...) DRM_DEBUG_KMS(__VA_ARGS__) struct dal_logger; @@ -98,7 +98,7 @@ enum dc_log_type { LOG_EVENT_UNDERFLOW, LOG_IF_TRACE, LOG_PERF_TRACE, - LOG_PROFILING, + LOG_DISPLAYSTATS, LOG_SECTION_TOTAL_COUNT }; diff --git a/drivers/gpu/drm/amd/display/modules/color/color_gamma.c b/drivers/gpu/drm/amd/display/modules/color/color_gamma.c index b3747a019deb..eee0dfad6962 100644 --- a/drivers/gpu/drm/amd/display/modules/color/color_gamma.c +++ b/drivers/gpu/drm/amd/display/modules/color/color_gamma.c @@ -43,7 +43,7 @@ static bool de_pq_initialized; /* = false; */ /* one-time setup of X points */ void setup_x_points_distribution(void) { - struct fixed31_32 region_size = dal_fixed31_32_from_int(128); + struct fixed31_32 region_size = dc_fixpt_from_int(128); int32_t segment; uint32_t seg_offset; uint32_t index; @@ -53,8 +53,8 @@ void setup_x_points_distribution(void) coordinates_x[MAX_HW_POINTS + 1].x = region_size; for (segment = 6; segment > (6 - NUM_REGIONS); segment--) { - region_size = dal_fixed31_32_div_int(region_size, 2); - increment = dal_fixed31_32_div_int(region_size, + region_size = dc_fixpt_div_int(region_size, 2); + increment = dc_fixpt_div_int(region_size, NUM_PTS_IN_REGION); seg_offset = (segment + (NUM_REGIONS - 7)) * NUM_PTS_IN_REGION; coordinates_x[seg_offset].x = region_size; @@ -62,7 +62,7 @@ void setup_x_points_distribution(void) for (index = seg_offset + 1; index < seg_offset + NUM_PTS_IN_REGION; index++) { - coordinates_x[index].x = dal_fixed31_32_add + coordinates_x[index].x = dc_fixpt_add (coordinates_x[index-1].x, increment); } } @@ -72,63 +72,63 @@ static void compute_pq(struct fixed31_32 in_x, struct fixed31_32 *out_y) { /* consts for PQ gamma formula. */ const struct fixed31_32 m1 = - dal_fixed31_32_from_fraction(159301758, 1000000000); + dc_fixpt_from_fraction(159301758, 1000000000); const struct fixed31_32 m2 = - dal_fixed31_32_from_fraction(7884375, 100000); + dc_fixpt_from_fraction(7884375, 100000); const struct fixed31_32 c1 = - dal_fixed31_32_from_fraction(8359375, 10000000); + dc_fixpt_from_fraction(8359375, 10000000); const struct fixed31_32 c2 = - dal_fixed31_32_from_fraction(188515625, 10000000); + dc_fixpt_from_fraction(188515625, 10000000); const struct fixed31_32 c3 = - dal_fixed31_32_from_fraction(186875, 10000); + dc_fixpt_from_fraction(186875, 10000); struct fixed31_32 l_pow_m1; struct fixed31_32 base; - if (dal_fixed31_32_lt(in_x, dal_fixed31_32_zero)) - in_x = dal_fixed31_32_zero; + if (dc_fixpt_lt(in_x, dc_fixpt_zero)) + in_x = dc_fixpt_zero; - l_pow_m1 = dal_fixed31_32_pow(in_x, m1); - base = dal_fixed31_32_div( - dal_fixed31_32_add(c1, - (dal_fixed31_32_mul(c2, l_pow_m1))), - dal_fixed31_32_add(dal_fixed31_32_one, - (dal_fixed31_32_mul(c3, l_pow_m1)))); - *out_y = dal_fixed31_32_pow(base, m2); + l_pow_m1 = dc_fixpt_pow(in_x, m1); + base = dc_fixpt_div( + dc_fixpt_add(c1, + (dc_fixpt_mul(c2, l_pow_m1))), + dc_fixpt_add(dc_fixpt_one, + (dc_fixpt_mul(c3, l_pow_m1)))); + *out_y = dc_fixpt_pow(base, m2); } static void compute_de_pq(struct fixed31_32 in_x, struct fixed31_32 *out_y) { /* consts for dePQ gamma formula. */ const struct fixed31_32 m1 = - dal_fixed31_32_from_fraction(159301758, 1000000000); + dc_fixpt_from_fraction(159301758, 1000000000); const struct fixed31_32 m2 = - dal_fixed31_32_from_fraction(7884375, 100000); + dc_fixpt_from_fraction(7884375, 100000); const struct fixed31_32 c1 = - dal_fixed31_32_from_fraction(8359375, 10000000); + dc_fixpt_from_fraction(8359375, 10000000); const struct fixed31_32 c2 = - dal_fixed31_32_from_fraction(188515625, 10000000); + dc_fixpt_from_fraction(188515625, 10000000); const struct fixed31_32 c3 = - dal_fixed31_32_from_fraction(186875, 10000); + dc_fixpt_from_fraction(186875, 10000); struct fixed31_32 l_pow_m1; struct fixed31_32 base, div; - if (dal_fixed31_32_lt(in_x, dal_fixed31_32_zero)) - in_x = dal_fixed31_32_zero; + if (dc_fixpt_lt(in_x, dc_fixpt_zero)) + in_x = dc_fixpt_zero; - l_pow_m1 = dal_fixed31_32_pow(in_x, - dal_fixed31_32_div(dal_fixed31_32_one, m2)); - base = dal_fixed31_32_sub(l_pow_m1, c1); + l_pow_m1 = dc_fixpt_pow(in_x, + dc_fixpt_div(dc_fixpt_one, m2)); + base = dc_fixpt_sub(l_pow_m1, c1); - if (dal_fixed31_32_lt(base, dal_fixed31_32_zero)) - base = dal_fixed31_32_zero; + if (dc_fixpt_lt(base, dc_fixpt_zero)) + base = dc_fixpt_zero; - div = dal_fixed31_32_sub(c2, dal_fixed31_32_mul(c3, l_pow_m1)); + div = dc_fixpt_sub(c2, dc_fixpt_mul(c3, l_pow_m1)); - *out_y = dal_fixed31_32_pow(dal_fixed31_32_div(base, div), - dal_fixed31_32_div(dal_fixed31_32_one, m1)); + *out_y = dc_fixpt_pow(dc_fixpt_div(base, div), + dc_fixpt_div(dc_fixpt_one, m1)); } /* one-time pre-compute PQ values - only for sdr_white_level 80 */ @@ -138,14 +138,14 @@ void precompute_pq(void) struct fixed31_32 x; const struct hw_x_point *coord_x = coordinates_x + 32; struct fixed31_32 scaling_factor = - dal_fixed31_32_from_fraction(80, 10000); + dc_fixpt_from_fraction(80, 10000); /* pow function has problems with arguments too small */ for (i = 0; i < 32; i++) - pq_table[i] = dal_fixed31_32_zero; + pq_table[i] = dc_fixpt_zero; for (i = 32; i <= MAX_HW_POINTS; i++) { - x = dal_fixed31_32_mul(coord_x->x, scaling_factor); + x = dc_fixpt_mul(coord_x->x, scaling_factor); compute_pq(x, &pq_table[i]); ++coord_x; } @@ -158,7 +158,7 @@ void precompute_de_pq(void) struct fixed31_32 y; uint32_t begin_index, end_index; - struct fixed31_32 scaling_factor = dal_fixed31_32_from_int(125); + struct fixed31_32 scaling_factor = dc_fixpt_from_int(125); /* X points is 2^-25 to 2^7 * De-gamma X is 2^-12 to 2^0 – we are skipping first -12-(-25) = 13 regions @@ -167,11 +167,11 @@ void precompute_de_pq(void) end_index = begin_index + 12 * NUM_PTS_IN_REGION; for (i = 0; i <= begin_index; i++) - de_pq_table[i] = dal_fixed31_32_zero; + de_pq_table[i] = dc_fixpt_zero; for (; i <= end_index; i++) { compute_de_pq(coordinates_x[i].x, &y); - de_pq_table[i] = dal_fixed31_32_mul(y, scaling_factor); + de_pq_table[i] = dc_fixpt_mul(y, scaling_factor); } for (; i <= MAX_HW_POINTS; i++) @@ -185,25 +185,25 @@ struct dividers { static void build_coefficients(struct gamma_coefficients *coefficients, bool is_2_4) { - static const int32_t numerator01[] = { 31308, 180000}; - static const int32_t numerator02[] = { 12920, 4500}; - static const int32_t numerator03[] = { 55, 99}; - static const int32_t numerator04[] = { 55, 99}; - static const int32_t numerator05[] = { 2400, 2200}; + static const int32_t numerator01[] = { 31308, 180000}; + static const int32_t numerator02[] = { 12920, 4500}; + static const int32_t numerator03[] = { 55, 99}; + static const int32_t numerator04[] = { 55, 99}; + static const int32_t numerator05[] = { 2400, 2200}; - uint32_t i = 0; - uint32_t index = is_2_4 == true ? 0:1; + uint32_t i = 0; + uint32_t index = is_2_4 == true ? 0:1; do { - coefficients->a0[i] = dal_fixed31_32_from_fraction( + coefficients->a0[i] = dc_fixpt_from_fraction( numerator01[index], 10000000); - coefficients->a1[i] = dal_fixed31_32_from_fraction( + coefficients->a1[i] = dc_fixpt_from_fraction( numerator02[index], 1000); - coefficients->a2[i] = dal_fixed31_32_from_fraction( + coefficients->a2[i] = dc_fixpt_from_fraction( numerator03[index], 1000); - coefficients->a3[i] = dal_fixed31_32_from_fraction( + coefficients->a3[i] = dc_fixpt_from_fraction( numerator04[index], 1000); - coefficients->user_gamma[i] = dal_fixed31_32_from_fraction( + coefficients->user_gamma[i] = dc_fixpt_from_fraction( numerator05[index], 1000); ++i; @@ -218,33 +218,33 @@ static struct fixed31_32 translate_from_linear_space( struct fixed31_32 a3, struct fixed31_32 gamma) { - const struct fixed31_32 one = dal_fixed31_32_from_int(1); + const struct fixed31_32 one = dc_fixpt_from_int(1); - if (dal_fixed31_32_lt(one, arg)) + if (dc_fixpt_lt(one, arg)) return one; - if (dal_fixed31_32_le(arg, dal_fixed31_32_neg(a0))) - return dal_fixed31_32_sub( + if (dc_fixpt_le(arg, dc_fixpt_neg(a0))) + return dc_fixpt_sub( a2, - dal_fixed31_32_mul( - dal_fixed31_32_add( + dc_fixpt_mul( + dc_fixpt_add( one, a3), - dal_fixed31_32_pow( - dal_fixed31_32_neg(arg), - dal_fixed31_32_recip(gamma)))); - else if (dal_fixed31_32_le(a0, arg)) - return dal_fixed31_32_sub( - dal_fixed31_32_mul( - dal_fixed31_32_add( + dc_fixpt_pow( + dc_fixpt_neg(arg), + dc_fixpt_recip(gamma)))); + else if (dc_fixpt_le(a0, arg)) + return dc_fixpt_sub( + dc_fixpt_mul( + dc_fixpt_add( one, a3), - dal_fixed31_32_pow( + dc_fixpt_pow( arg, - dal_fixed31_32_recip(gamma))), + dc_fixpt_recip(gamma))), a2); else - return dal_fixed31_32_mul( + return dc_fixpt_mul( arg, a1); } @@ -259,25 +259,25 @@ static struct fixed31_32 translate_to_linear_space( { struct fixed31_32 linear; - a0 = dal_fixed31_32_mul(a0, a1); - if (dal_fixed31_32_le(arg, dal_fixed31_32_neg(a0))) + a0 = dc_fixpt_mul(a0, a1); + if (dc_fixpt_le(arg, dc_fixpt_neg(a0))) - linear = dal_fixed31_32_neg( - dal_fixed31_32_pow( - dal_fixed31_32_div( - dal_fixed31_32_sub(a2, arg), - dal_fixed31_32_add( - dal_fixed31_32_one, a3)), gamma)); + linear = dc_fixpt_neg( + dc_fixpt_pow( + dc_fixpt_div( + dc_fixpt_sub(a2, arg), + dc_fixpt_add( + dc_fixpt_one, a3)), gamma)); - else if (dal_fixed31_32_le(dal_fixed31_32_neg(a0), arg) && - dal_fixed31_32_le(arg, a0)) - linear = dal_fixed31_32_div(arg, a1); + else if (dc_fixpt_le(dc_fixpt_neg(a0), arg) && + dc_fixpt_le(arg, a0)) + linear = dc_fixpt_div(arg, a1); else - linear = dal_fixed31_32_pow( - dal_fixed31_32_div( - dal_fixed31_32_add(a2, arg), - dal_fixed31_32_add( - dal_fixed31_32_one, a3)), gamma); + linear = dc_fixpt_pow( + dc_fixpt_div( + dc_fixpt_add(a2, arg), + dc_fixpt_add( + dc_fixpt_one, a3)), gamma); return linear; } @@ -352,8 +352,8 @@ static bool find_software_points( right = axis_x[max_number - 1].b; } - if (dal_fixed31_32_le(left, hw_point) && - dal_fixed31_32_le(hw_point, right)) { + if (dc_fixpt_le(left, hw_point) && + dc_fixpt_le(hw_point, right)) { *index_to_start = i; *index_left = i; @@ -366,7 +366,7 @@ static bool find_software_points( return true; } else if ((i == *index_to_start) && - dal_fixed31_32_le(hw_point, left)) { + dc_fixpt_le(hw_point, left)) { *index_to_start = i; *index_left = i; *index_right = i; @@ -375,7 +375,7 @@ static bool find_software_points( return true; } else if ((i == max_number - 1) && - dal_fixed31_32_le(right, hw_point)) { + dc_fixpt_le(right, hw_point)) { *index_to_start = i; *index_left = i; *index_right = i; @@ -457,17 +457,17 @@ static bool build_custom_gamma_mapping_coefficients_worker( } if (hw_pos == HW_POINT_POSITION_MIDDLE) - point->coeff = dal_fixed31_32_div( - dal_fixed31_32_sub( + point->coeff = dc_fixpt_div( + dc_fixpt_sub( coord_x, left_pos), - dal_fixed31_32_sub( + dc_fixpt_sub( right_pos, left_pos)); else if (hw_pos == HW_POINT_POSITION_LEFT) - point->coeff = dal_fixed31_32_zero; + point->coeff = dc_fixpt_zero; else if (hw_pos == HW_POINT_POSITION_RIGHT) - point->coeff = dal_fixed31_32_from_int(2); + point->coeff = dc_fixpt_from_int(2); else { BREAK_TO_DEBUGGER(); return false; @@ -502,45 +502,45 @@ static struct fixed31_32 calculate_mapped_value( if ((point->left_index < 0) || (point->left_index > max_index)) { BREAK_TO_DEBUGGER(); - return dal_fixed31_32_zero; + return dc_fixpt_zero; } if ((point->right_index < 0) || (point->right_index > max_index)) { BREAK_TO_DEBUGGER(); - return dal_fixed31_32_zero; + return dc_fixpt_zero; } if (point->pos == HW_POINT_POSITION_MIDDLE) if (channel == CHANNEL_NAME_RED) - result = dal_fixed31_32_add( - dal_fixed31_32_mul( + result = dc_fixpt_add( + dc_fixpt_mul( point->coeff, - dal_fixed31_32_sub( + dc_fixpt_sub( rgb[point->right_index].r, rgb[point->left_index].r)), rgb[point->left_index].r); else if (channel == CHANNEL_NAME_GREEN) - result = dal_fixed31_32_add( - dal_fixed31_32_mul( + result = dc_fixpt_add( + dc_fixpt_mul( point->coeff, - dal_fixed31_32_sub( + dc_fixpt_sub( rgb[point->right_index].g, rgb[point->left_index].g)), rgb[point->left_index].g); else - result = dal_fixed31_32_add( - dal_fixed31_32_mul( + result = dc_fixpt_add( + dc_fixpt_mul( point->coeff, - dal_fixed31_32_sub( + dc_fixpt_sub( rgb[point->right_index].b, rgb[point->left_index].b)), rgb[point->left_index].b); else if (point->pos == HW_POINT_POSITION_LEFT) { BREAK_TO_DEBUGGER(); - result = dal_fixed31_32_zero; + result = dc_fixpt_zero; } else { BREAK_TO_DEBUGGER(); - result = dal_fixed31_32_one; + result = dc_fixpt_one; } return result; @@ -558,7 +558,7 @@ static void build_pq(struct pwl_float_data_ex *rgb_regamma, struct fixed31_32 x; struct fixed31_32 output; struct fixed31_32 scaling_factor = - dal_fixed31_32_from_fraction(sdr_white_level, 10000); + dc_fixpt_from_fraction(sdr_white_level, 10000); if (!pq_initialized && sdr_white_level == 80) { precompute_pq(); @@ -579,15 +579,15 @@ static void build_pq(struct pwl_float_data_ex *rgb_regamma, if (sdr_white_level == 80) { output = pq_table[i]; } else { - x = dal_fixed31_32_mul(coord_x->x, scaling_factor); + x = dc_fixpt_mul(coord_x->x, scaling_factor); compute_pq(x, &output); } /* should really not happen? */ - if (dal_fixed31_32_lt(output, dal_fixed31_32_zero)) - output = dal_fixed31_32_zero; - else if (dal_fixed31_32_lt(dal_fixed31_32_one, output)) - output = dal_fixed31_32_one; + if (dc_fixpt_lt(output, dc_fixpt_zero)) + output = dc_fixpt_zero; + else if (dc_fixpt_lt(dc_fixpt_one, output)) + output = dc_fixpt_one; rgb->r = output; rgb->g = output; @@ -605,7 +605,7 @@ static void build_de_pq(struct pwl_float_data_ex *de_pq, uint32_t i; struct fixed31_32 output; - struct fixed31_32 scaling_factor = dal_fixed31_32_from_int(125); + struct fixed31_32 scaling_factor = dc_fixpt_from_int(125); if (!de_pq_initialized) { precompute_de_pq(); @@ -616,9 +616,9 @@ static void build_de_pq(struct pwl_float_data_ex *de_pq, for (i = 0; i <= hw_points_num; i++) { output = de_pq_table[i]; /* should really not happen? */ - if (dal_fixed31_32_lt(output, dal_fixed31_32_zero)) - output = dal_fixed31_32_zero; - else if (dal_fixed31_32_lt(scaling_factor, output)) + if (dc_fixpt_lt(output, dc_fixpt_zero)) + output = dc_fixpt_zero; + else if (dc_fixpt_lt(scaling_factor, output)) output = scaling_factor; de_pq[i].r = output; de_pq[i].g = output; @@ -670,9 +670,9 @@ static void build_degamma(struct pwl_float_data_ex *curve, end_index = begin_index + 12 * NUM_PTS_IN_REGION; while (i != begin_index) { - curve[i].r = dal_fixed31_32_zero; - curve[i].g = dal_fixed31_32_zero; - curve[i].b = dal_fixed31_32_zero; + curve[i].r = dc_fixpt_zero; + curve[i].g = dc_fixpt_zero; + curve[i].b = dc_fixpt_zero; i++; } @@ -684,19 +684,19 @@ static void build_degamma(struct pwl_float_data_ex *curve, i++; } while (i != hw_points_num + 1) { - curve[i].r = dal_fixed31_32_one; - curve[i].g = dal_fixed31_32_one; - curve[i].b = dal_fixed31_32_one; + curve[i].r = dc_fixpt_one; + curve[i].g = dc_fixpt_one; + curve[i].b = dc_fixpt_one; i++; } } -static bool scale_gamma(struct pwl_float_data *pwl_rgb, +static void scale_gamma(struct pwl_float_data *pwl_rgb, const struct dc_gamma *ramp, struct dividers dividers) { - const struct fixed31_32 max_driver = dal_fixed31_32_from_int(0xFFFF); - const struct fixed31_32 max_os = dal_fixed31_32_from_int(0xFF00); + const struct fixed31_32 max_driver = dc_fixpt_from_int(0xFFFF); + const struct fixed31_32 max_os = dc_fixpt_from_int(0xFF00); struct fixed31_32 scaler = max_os; uint32_t i; struct pwl_float_data *rgb = pwl_rgb; @@ -705,9 +705,9 @@ static bool scale_gamma(struct pwl_float_data *pwl_rgb, i = 0; do { - if (dal_fixed31_32_lt(max_os, ramp->entries.red[i]) || - dal_fixed31_32_lt(max_os, ramp->entries.green[i]) || - dal_fixed31_32_lt(max_os, ramp->entries.blue[i])) { + if (dc_fixpt_lt(max_os, ramp->entries.red[i]) || + dc_fixpt_lt(max_os, ramp->entries.green[i]) || + dc_fixpt_lt(max_os, ramp->entries.blue[i])) { scaler = max_driver; break; } @@ -717,109 +717,170 @@ static bool scale_gamma(struct pwl_float_data *pwl_rgb, i = 0; do { - rgb->r = dal_fixed31_32_div( + rgb->r = dc_fixpt_div( ramp->entries.red[i], scaler); - rgb->g = dal_fixed31_32_div( + rgb->g = dc_fixpt_div( ramp->entries.green[i], scaler); - rgb->b = dal_fixed31_32_div( + rgb->b = dc_fixpt_div( ramp->entries.blue[i], scaler); ++rgb; ++i; } while (i != ramp->num_entries); - rgb->r = dal_fixed31_32_mul(rgb_last->r, + rgb->r = dc_fixpt_mul(rgb_last->r, dividers.divider1); - rgb->g = dal_fixed31_32_mul(rgb_last->g, + rgb->g = dc_fixpt_mul(rgb_last->g, dividers.divider1); - rgb->b = dal_fixed31_32_mul(rgb_last->b, + rgb->b = dc_fixpt_mul(rgb_last->b, dividers.divider1); ++rgb; - rgb->r = dal_fixed31_32_mul(rgb_last->r, + rgb->r = dc_fixpt_mul(rgb_last->r, dividers.divider2); - rgb->g = dal_fixed31_32_mul(rgb_last->g, + rgb->g = dc_fixpt_mul(rgb_last->g, dividers.divider2); - rgb->b = dal_fixed31_32_mul(rgb_last->b, + rgb->b = dc_fixpt_mul(rgb_last->b, dividers.divider2); ++rgb; - rgb->r = dal_fixed31_32_mul(rgb_last->r, + rgb->r = dc_fixpt_mul(rgb_last->r, dividers.divider3); - rgb->g = dal_fixed31_32_mul(rgb_last->g, + rgb->g = dc_fixpt_mul(rgb_last->g, dividers.divider3); - rgb->b = dal_fixed31_32_mul(rgb_last->b, + rgb->b = dc_fixpt_mul(rgb_last->b, dividers.divider3); - - return true; } -static bool scale_gamma_dx(struct pwl_float_data *pwl_rgb, +static void scale_gamma_dx(struct pwl_float_data *pwl_rgb, const struct dc_gamma *ramp, struct dividers dividers) { uint32_t i; - struct fixed31_32 min = dal_fixed31_32_zero; - struct fixed31_32 max = dal_fixed31_32_one; + struct fixed31_32 min = dc_fixpt_zero; + struct fixed31_32 max = dc_fixpt_one; - struct fixed31_32 delta = dal_fixed31_32_zero; - struct fixed31_32 offset = dal_fixed31_32_zero; + struct fixed31_32 delta = dc_fixpt_zero; + struct fixed31_32 offset = dc_fixpt_zero; for (i = 0 ; i < ramp->num_entries; i++) { - if (dal_fixed31_32_lt(ramp->entries.red[i], min)) + if (dc_fixpt_lt(ramp->entries.red[i], min)) min = ramp->entries.red[i]; - if (dal_fixed31_32_lt(ramp->entries.green[i], min)) + if (dc_fixpt_lt(ramp->entries.green[i], min)) min = ramp->entries.green[i]; - if (dal_fixed31_32_lt(ramp->entries.blue[i], min)) + if (dc_fixpt_lt(ramp->entries.blue[i], min)) min = ramp->entries.blue[i]; - if (dal_fixed31_32_lt(max, ramp->entries.red[i])) + if (dc_fixpt_lt(max, ramp->entries.red[i])) max = ramp->entries.red[i]; - if (dal_fixed31_32_lt(max, ramp->entries.green[i])) + if (dc_fixpt_lt(max, ramp->entries.green[i])) max = ramp->entries.green[i]; - if (dal_fixed31_32_lt(max, ramp->entries.blue[i])) + if (dc_fixpt_lt(max, ramp->entries.blue[i])) max = ramp->entries.blue[i]; } - if (dal_fixed31_32_lt(min, dal_fixed31_32_zero)) - delta = dal_fixed31_32_neg(min); + if (dc_fixpt_lt(min, dc_fixpt_zero)) + delta = dc_fixpt_neg(min); - offset = dal_fixed31_32_add(min, max); + offset = dc_fixpt_add(min, max); for (i = 0 ; i < ramp->num_entries; i++) { - pwl_rgb[i].r = dal_fixed31_32_div( - dal_fixed31_32_add( + pwl_rgb[i].r = dc_fixpt_div( + dc_fixpt_add( ramp->entries.red[i], delta), offset); - pwl_rgb[i].g = dal_fixed31_32_div( - dal_fixed31_32_add( + pwl_rgb[i].g = dc_fixpt_div( + dc_fixpt_add( ramp->entries.green[i], delta), offset); - pwl_rgb[i].b = dal_fixed31_32_div( - dal_fixed31_32_add( + pwl_rgb[i].b = dc_fixpt_div( + dc_fixpt_add( ramp->entries.blue[i], delta), offset); } - pwl_rgb[i].r = dal_fixed31_32_sub(dal_fixed31_32_mul_int( + pwl_rgb[i].r = dc_fixpt_sub(dc_fixpt_mul_int( pwl_rgb[i-1].r, 2), pwl_rgb[i-2].r); - pwl_rgb[i].g = dal_fixed31_32_sub(dal_fixed31_32_mul_int( + pwl_rgb[i].g = dc_fixpt_sub(dc_fixpt_mul_int( pwl_rgb[i-1].g, 2), pwl_rgb[i-2].g); - pwl_rgb[i].b = dal_fixed31_32_sub(dal_fixed31_32_mul_int( + pwl_rgb[i].b = dc_fixpt_sub(dc_fixpt_mul_int( pwl_rgb[i-1].b, 2), pwl_rgb[i-2].b); ++i; - pwl_rgb[i].r = dal_fixed31_32_sub(dal_fixed31_32_mul_int( + pwl_rgb[i].r = dc_fixpt_sub(dc_fixpt_mul_int( pwl_rgb[i-1].r, 2), pwl_rgb[i-2].r); - pwl_rgb[i].g = dal_fixed31_32_sub(dal_fixed31_32_mul_int( + pwl_rgb[i].g = dc_fixpt_sub(dc_fixpt_mul_int( pwl_rgb[i-1].g, 2), pwl_rgb[i-2].g); - pwl_rgb[i].b = dal_fixed31_32_sub(dal_fixed31_32_mul_int( + pwl_rgb[i].b = dc_fixpt_sub(dc_fixpt_mul_int( pwl_rgb[i-1].b, 2), pwl_rgb[i-2].b); +} - return true; +/* todo: all these scale_gamma functions are inherently the same but + * take different structures as params or different format for ramp + * values. We could probably implement it in a more generic fashion + */ +static void scale_user_regamma_ramp(struct pwl_float_data *pwl_rgb, + const struct regamma_ramp *ramp, + struct dividers dividers) +{ + unsigned short max_driver = 0xFFFF; + unsigned short max_os = 0xFF00; + unsigned short scaler = max_os; + uint32_t i; + struct pwl_float_data *rgb = pwl_rgb; + struct pwl_float_data *rgb_last = rgb + GAMMA_RGB_256_ENTRIES - 1; + + i = 0; + do { + if (ramp->gamma[i] > max_os || + ramp->gamma[i + 256] > max_os || + ramp->gamma[i + 512] > max_os) { + scaler = max_driver; + break; + } + i++; + } while (i != GAMMA_RGB_256_ENTRIES); + + i = 0; + do { + rgb->r = dc_fixpt_from_fraction( + ramp->gamma[i], scaler); + rgb->g = dc_fixpt_from_fraction( + ramp->gamma[i + 256], scaler); + rgb->b = dc_fixpt_from_fraction( + ramp->gamma[i + 512], scaler); + + ++rgb; + ++i; + } while (i != GAMMA_RGB_256_ENTRIES); + + rgb->r = dc_fixpt_mul(rgb_last->r, + dividers.divider1); + rgb->g = dc_fixpt_mul(rgb_last->g, + dividers.divider1); + rgb->b = dc_fixpt_mul(rgb_last->b, + dividers.divider1); + + ++rgb; + + rgb->r = dc_fixpt_mul(rgb_last->r, + dividers.divider2); + rgb->g = dc_fixpt_mul(rgb_last->g, + dividers.divider2); + rgb->b = dc_fixpt_mul(rgb_last->b, + dividers.divider2); + + ++rgb; + + rgb->r = dc_fixpt_mul(rgb_last->r, + dividers.divider3); + rgb->g = dc_fixpt_mul(rgb_last->g, + dividers.divider3); + rgb->b = dc_fixpt_mul(rgb_last->b, + dividers.divider3); } /* @@ -852,7 +913,7 @@ static void apply_lut_1d( struct fixed31_32 lut2; const int max_lut_index = 4095; const struct fixed31_32 max_lut_index_f = - dal_fixed31_32_from_int_nonconst(max_lut_index); + dc_fixpt_from_int(max_lut_index); int32_t index = 0, index_next = 0; struct fixed31_32 index_f; struct fixed31_32 delta_lut; @@ -870,10 +931,10 @@ static void apply_lut_1d( else regamma_y = &tf_pts->blue[i]; - norm_y = dal_fixed31_32_mul(max_lut_index_f, + norm_y = dc_fixpt_mul(max_lut_index_f, *regamma_y); - index = dal_fixed31_32_floor(norm_y); - index_f = dal_fixed31_32_from_int_nonconst(index); + index = dc_fixpt_floor(norm_y); + index_f = dc_fixpt_from_int(index); if (index < 0 || index > max_lut_index) continue; @@ -892,11 +953,11 @@ static void apply_lut_1d( } // we have everything now, so interpolate - delta_lut = dal_fixed31_32_sub(lut2, lut1); - delta_index = dal_fixed31_32_sub(norm_y, index_f); + delta_lut = dc_fixpt_sub(lut2, lut1); + delta_index = dc_fixpt_sub(norm_y, index_f); - *regamma_y = dal_fixed31_32_add(lut1, - dal_fixed31_32_mul(delta_index, delta_lut)); + *regamma_y = dc_fixpt_add(lut1, + dc_fixpt_mul(delta_index, delta_lut)); } } } @@ -912,7 +973,7 @@ static void build_evenly_distributed_points( uint32_t i = 0; do { - struct fixed31_32 value = dal_fixed31_32_from_fraction(i, + struct fixed31_32 value = dc_fixpt_from_fraction(i, numberof_points - 1); p->r = value; @@ -923,21 +984,21 @@ static void build_evenly_distributed_points( ++i; } while (i != numberof_points); - p->r = dal_fixed31_32_div(p_last->r, dividers.divider1); - p->g = dal_fixed31_32_div(p_last->g, dividers.divider1); - p->b = dal_fixed31_32_div(p_last->b, dividers.divider1); + p->r = dc_fixpt_div(p_last->r, dividers.divider1); + p->g = dc_fixpt_div(p_last->g, dividers.divider1); + p->b = dc_fixpt_div(p_last->b, dividers.divider1); ++p; - p->r = dal_fixed31_32_div(p_last->r, dividers.divider2); - p->g = dal_fixed31_32_div(p_last->g, dividers.divider2); - p->b = dal_fixed31_32_div(p_last->b, dividers.divider2); + p->r = dc_fixpt_div(p_last->r, dividers.divider2); + p->g = dc_fixpt_div(p_last->g, dividers.divider2); + p->b = dc_fixpt_div(p_last->b, dividers.divider2); ++p; - p->r = dal_fixed31_32_div(p_last->r, dividers.divider3); - p->g = dal_fixed31_32_div(p_last->g, dividers.divider3); - p->b = dal_fixed31_32_div(p_last->b, dividers.divider3); + p->r = dc_fixpt_div(p_last->r, dividers.divider3); + p->g = dc_fixpt_div(p_last->g, dividers.divider3); + p->b = dc_fixpt_div(p_last->b, dividers.divider3); } static inline void copy_rgb_regamma_to_coordinates_x( @@ -949,7 +1010,7 @@ static inline void copy_rgb_regamma_to_coordinates_x( uint32_t i = 0; const struct pwl_float_data_ex *rgb_regamma = rgb_ex; - while (i <= hw_points_num) { + while (i <= hw_points_num + 1) { coords->regamma_y_red = rgb_regamma->r; coords->regamma_y_green = rgb_regamma->g; coords->regamma_y_blue = rgb_regamma->b; @@ -1002,6 +1063,102 @@ static bool calculate_interpolated_hardware_curve( return true; } +/* The "old" interpolation uses a complicated scheme to build an array of + * coefficients while also using an array of 0-255 normalized to 0-1 + * Then there's another loop using both of the above + new scaled user ramp + * and we concatenate them. It also searches for points of interpolation and + * uses enums for positions. + * + * This function uses a different approach: + * user ramp is always applied on X with 0/255, 1/255, 2/255, ..., 255/255 + * To find index for hwX , we notice the following: + * i/255 <= hwX < (i+1)/255 <=> i <= 255*hwX < i+1 + * See apply_lut_1d which is the same principle, but on 4K entry 1D LUT + * + * Once the index is known, combined Y is simply: + * user_ramp(index) + (hwX-index/255)*(user_ramp(index+1) - user_ramp(index) + * + * We should switch to this method in all cases, it's simpler and faster + * ToDo one day - for now this only applies to ADL regamma to avoid regression + * for regular use cases (sRGB and PQ) + */ +static void interpolate_user_regamma(uint32_t hw_points_num, + struct pwl_float_data *rgb_user, + bool apply_degamma, + struct dc_transfer_func_distributed_points *tf_pts) +{ + uint32_t i; + uint32_t color = 0; + int32_t index; + int32_t index_next; + struct fixed31_32 *tf_point; + struct fixed31_32 hw_x; + struct fixed31_32 norm_factor = + dc_fixpt_from_int(255); + struct fixed31_32 norm_x; + struct fixed31_32 index_f; + struct fixed31_32 lut1; + struct fixed31_32 lut2; + struct fixed31_32 delta_lut; + struct fixed31_32 delta_index; + + i = 0; + /* fixed_pt library has problems handling too small values */ + while (i != 32) { + tf_pts->red[i] = dc_fixpt_zero; + tf_pts->green[i] = dc_fixpt_zero; + tf_pts->blue[i] = dc_fixpt_zero; + ++i; + } + while (i <= hw_points_num + 1) { + for (color = 0; color < 3; color++) { + if (color == 0) + tf_point = &tf_pts->red[i]; + else if (color == 1) + tf_point = &tf_pts->green[i]; + else + tf_point = &tf_pts->blue[i]; + + if (apply_degamma) { + if (color == 0) + hw_x = coordinates_x[i].regamma_y_red; + else if (color == 1) + hw_x = coordinates_x[i].regamma_y_green; + else + hw_x = coordinates_x[i].regamma_y_blue; + } else + hw_x = coordinates_x[i].x; + + norm_x = dc_fixpt_mul(norm_factor, hw_x); + index = dc_fixpt_floor(norm_x); + if (index < 0 || index > 255) + continue; + + index_f = dc_fixpt_from_int(index); + index_next = (index == 255) ? index : index + 1; + + if (color == 0) { + lut1 = rgb_user[index].r; + lut2 = rgb_user[index_next].r; + } else if (color == 1) { + lut1 = rgb_user[index].g; + lut2 = rgb_user[index_next].g; + } else { + lut1 = rgb_user[index].b; + lut2 = rgb_user[index_next].b; + } + + // we have everything now, so interpolate + delta_lut = dc_fixpt_sub(lut2, lut1); + delta_index = dc_fixpt_sub(norm_x, index_f); + + *tf_point = dc_fixpt_add(lut1, + dc_fixpt_mul(delta_index, delta_lut)); + } + ++i; + } +} + static void build_new_custom_resulted_curve( uint32_t hw_points_num, struct dc_transfer_func_distributed_points *tf_pts) @@ -1011,16 +1168,39 @@ static void build_new_custom_resulted_curve( i = 0; while (i != hw_points_num + 1) { - tf_pts->red[i] = dal_fixed31_32_clamp( - tf_pts->red[i], dal_fixed31_32_zero, - dal_fixed31_32_one); - tf_pts->green[i] = dal_fixed31_32_clamp( - tf_pts->green[i], dal_fixed31_32_zero, - dal_fixed31_32_one); - tf_pts->blue[i] = dal_fixed31_32_clamp( - tf_pts->blue[i], dal_fixed31_32_zero, - dal_fixed31_32_one); + tf_pts->red[i] = dc_fixpt_clamp( + tf_pts->red[i], dc_fixpt_zero, + dc_fixpt_one); + tf_pts->green[i] = dc_fixpt_clamp( + tf_pts->green[i], dc_fixpt_zero, + dc_fixpt_one); + tf_pts->blue[i] = dc_fixpt_clamp( + tf_pts->blue[i], dc_fixpt_zero, + dc_fixpt_one); + + ++i; + } +} + +static void apply_degamma_for_user_regamma(struct pwl_float_data_ex *rgb_regamma, + uint32_t hw_points_num) +{ + uint32_t i; + struct gamma_coefficients coeff; + struct pwl_float_data_ex *rgb = rgb_regamma; + const struct hw_x_point *coord_x = coordinates_x; + + build_coefficients(&coeff, true); + + i = 0; + while (i != hw_points_num + 1) { + rgb->r = translate_from_linear_space_ex( + coord_x->x, &coeff, 0); + rgb->g = rgb->r; + rgb->b = rgb->r; + ++coord_x; + ++rgb; ++i; } } @@ -1062,6 +1242,7 @@ static bool map_regamma_hw_to_x_user( } } + /* this should be named differently, all it does is clamp to 0-1 */ build_new_custom_resulted_curve(hw_points_num, tf_pts); return true; @@ -1093,25 +1274,28 @@ bool mod_color_calculate_regamma_params(struct dc_transfer_func *output_tf, output_tf->type = TF_TYPE_DISTRIBUTED_POINTS; - rgb_user = kvzalloc(sizeof(*rgb_user) * (ramp->num_entries + _EXTRA_POINTS), + rgb_user = kvcalloc(ramp->num_entries + _EXTRA_POINTS, + sizeof(*rgb_user), GFP_KERNEL); if (!rgb_user) goto rgb_user_alloc_fail; - rgb_regamma = kvzalloc(sizeof(*rgb_regamma) * (MAX_HW_POINTS + _EXTRA_POINTS), + rgb_regamma = kvcalloc(MAX_HW_POINTS + _EXTRA_POINTS, + sizeof(*rgb_regamma), GFP_KERNEL); if (!rgb_regamma) goto rgb_regamma_alloc_fail; - axix_x = kvzalloc(sizeof(*axix_x) * (ramp->num_entries + 3), + axix_x = kvcalloc(ramp->num_entries + 3, sizeof(*axix_x), GFP_KERNEL); if (!axix_x) goto axix_x_alloc_fail; - coeff = kvzalloc(sizeof(*coeff) * (MAX_HW_POINTS + _EXTRA_POINTS), GFP_KERNEL); + coeff = kvcalloc(MAX_HW_POINTS + _EXTRA_POINTS, sizeof(*coeff), + GFP_KERNEL); if (!coeff) goto coeff_alloc_fail; - dividers.divider1 = dal_fixed31_32_from_fraction(3, 2); - dividers.divider2 = dal_fixed31_32_from_int(2); - dividers.divider3 = dal_fixed31_32_from_fraction(5, 2); + dividers.divider1 = dc_fixpt_from_fraction(3, 2); + dividers.divider2 = dc_fixpt_from_int(2); + dividers.divider3 = dc_fixpt_from_fraction(5, 2); tf = output_tf->tf; @@ -1168,6 +1352,115 @@ rgb_user_alloc_fail: return ret; } +bool calculate_user_regamma_coeff(struct dc_transfer_func *output_tf, + const struct regamma_lut *regamma) +{ + struct gamma_coefficients coeff; + const struct hw_x_point *coord_x = coordinates_x; + uint32_t i = 0; + + do { + coeff.a0[i] = dc_fixpt_from_fraction( + regamma->coeff.A0[i], 10000000); + coeff.a1[i] = dc_fixpt_from_fraction( + regamma->coeff.A1[i], 1000); + coeff.a2[i] = dc_fixpt_from_fraction( + regamma->coeff.A2[i], 1000); + coeff.a3[i] = dc_fixpt_from_fraction( + regamma->coeff.A3[i], 1000); + coeff.user_gamma[i] = dc_fixpt_from_fraction( + regamma->coeff.gamma[i], 1000); + + ++i; + } while (i != 3); + + i = 0; + /* fixed_pt library has problems handling too small values */ + while (i != 32) { + output_tf->tf_pts.red[i] = dc_fixpt_zero; + output_tf->tf_pts.green[i] = dc_fixpt_zero; + output_tf->tf_pts.blue[i] = dc_fixpt_zero; + ++coord_x; + ++i; + } + while (i != MAX_HW_POINTS + 1) { + output_tf->tf_pts.red[i] = translate_from_linear_space_ex( + coord_x->x, &coeff, 0); + output_tf->tf_pts.green[i] = translate_from_linear_space_ex( + coord_x->x, &coeff, 1); + output_tf->tf_pts.blue[i] = translate_from_linear_space_ex( + coord_x->x, &coeff, 2); + ++coord_x; + ++i; + } + + // this function just clamps output to 0-1 + build_new_custom_resulted_curve(MAX_HW_POINTS, &output_tf->tf_pts); + output_tf->type = TF_TYPE_DISTRIBUTED_POINTS; + + return true; +} + +bool calculate_user_regamma_ramp(struct dc_transfer_func *output_tf, + const struct regamma_lut *regamma) +{ + struct dc_transfer_func_distributed_points *tf_pts = &output_tf->tf_pts; + struct dividers dividers; + + struct pwl_float_data *rgb_user = NULL; + struct pwl_float_data_ex *rgb_regamma = NULL; + bool ret = false; + + if (regamma == NULL) + return false; + + output_tf->type = TF_TYPE_DISTRIBUTED_POINTS; + + rgb_user = kcalloc(GAMMA_RGB_256_ENTRIES + _EXTRA_POINTS, + sizeof(*rgb_user), + GFP_KERNEL); + if (!rgb_user) + goto rgb_user_alloc_fail; + + rgb_regamma = kcalloc(MAX_HW_POINTS + _EXTRA_POINTS, + sizeof(*rgb_regamma), + GFP_KERNEL); + if (!rgb_regamma) + goto rgb_regamma_alloc_fail; + + dividers.divider1 = dc_fixpt_from_fraction(3, 2); + dividers.divider2 = dc_fixpt_from_int(2); + dividers.divider3 = dc_fixpt_from_fraction(5, 2); + + scale_user_regamma_ramp(rgb_user, ®amma->ramp, dividers); + + if (regamma->flags.bits.applyDegamma == 1) { + apply_degamma_for_user_regamma(rgb_regamma, MAX_HW_POINTS); + copy_rgb_regamma_to_coordinates_x(coordinates_x, + MAX_HW_POINTS, rgb_regamma); + } + + interpolate_user_regamma(MAX_HW_POINTS, rgb_user, + regamma->flags.bits.applyDegamma, tf_pts); + + // no custom HDR curves! + tf_pts->end_exponent = 0; + tf_pts->x_point_at_y1_red = 1; + tf_pts->x_point_at_y1_green = 1; + tf_pts->x_point_at_y1_blue = 1; + + // this function just clamps output to 0-1 + build_new_custom_resulted_curve(MAX_HW_POINTS, tf_pts); + + ret = true; + + kfree(rgb_regamma); +rgb_regamma_alloc_fail: + kvfree(rgb_user); +rgb_user_alloc_fail: + return ret; +} + bool mod_color_calculate_degamma_params(struct dc_transfer_func *input_tf, const struct dc_gamma *ramp, bool mapUserRamp) { @@ -1192,25 +1485,27 @@ bool mod_color_calculate_degamma_params(struct dc_transfer_func *input_tf, input_tf->type = TF_TYPE_DISTRIBUTED_POINTS; - rgb_user = kvzalloc(sizeof(*rgb_user) * (ramp->num_entries + _EXTRA_POINTS), + rgb_user = kvcalloc(ramp->num_entries + _EXTRA_POINTS, + sizeof(*rgb_user), GFP_KERNEL); if (!rgb_user) goto rgb_user_alloc_fail; - curve = kvzalloc(sizeof(*curve) * (MAX_HW_POINTS + _EXTRA_POINTS), + curve = kvcalloc(MAX_HW_POINTS + _EXTRA_POINTS, sizeof(*curve), GFP_KERNEL); if (!curve) goto curve_alloc_fail; - axix_x = kvzalloc(sizeof(*axix_x) * (ramp->num_entries + _EXTRA_POINTS), + axix_x = kvcalloc(ramp->num_entries + _EXTRA_POINTS, sizeof(*axix_x), GFP_KERNEL); if (!axix_x) goto axix_x_alloc_fail; - coeff = kvzalloc(sizeof(*coeff) * (MAX_HW_POINTS + _EXTRA_POINTS), GFP_KERNEL); + coeff = kvcalloc(MAX_HW_POINTS + _EXTRA_POINTS, sizeof(*coeff), + GFP_KERNEL); if (!coeff) goto coeff_alloc_fail; - dividers.divider1 = dal_fixed31_32_from_fraction(3, 2); - dividers.divider2 = dal_fixed31_32_from_int(2); - dividers.divider3 = dal_fixed31_32_from_fraction(5, 2); + dividers.divider1 = dc_fixpt_from_fraction(3, 2); + dividers.divider2 = dc_fixpt_from_int(2); + dividers.divider3 = dc_fixpt_from_fraction(5, 2); tf = input_tf->tf; @@ -1281,8 +1576,8 @@ bool mod_color_calculate_curve(enum dc_transfer_func_predefined trans, } ret = true; } else if (trans == TRANSFER_FUNCTION_PQ) { - rgb_regamma = kvzalloc(sizeof(*rgb_regamma) * - (MAX_HW_POINTS + _EXTRA_POINTS), + rgb_regamma = kvcalloc(MAX_HW_POINTS + _EXTRA_POINTS, + sizeof(*rgb_regamma), GFP_KERNEL); if (!rgb_regamma) goto rgb_regamma_alloc_fail; @@ -1306,8 +1601,8 @@ bool mod_color_calculate_curve(enum dc_transfer_func_predefined trans, kvfree(rgb_regamma); } else if (trans == TRANSFER_FUNCTION_SRGB || trans == TRANSFER_FUNCTION_BT709) { - rgb_regamma = kvzalloc(sizeof(*rgb_regamma) * - (MAX_HW_POINTS + _EXTRA_POINTS), + rgb_regamma = kvcalloc(MAX_HW_POINTS + _EXTRA_POINTS, + sizeof(*rgb_regamma), GFP_KERNEL); if (!rgb_regamma) goto rgb_regamma_alloc_fail; @@ -1350,8 +1645,8 @@ bool mod_color_calculate_degamma_curve(enum dc_transfer_func_predefined trans, } ret = true; } else if (trans == TRANSFER_FUNCTION_PQ) { - rgb_degamma = kvzalloc(sizeof(*rgb_degamma) * - (MAX_HW_POINTS + _EXTRA_POINTS), + rgb_degamma = kvcalloc(MAX_HW_POINTS + _EXTRA_POINTS, + sizeof(*rgb_degamma), GFP_KERNEL); if (!rgb_degamma) goto rgb_degamma_alloc_fail; @@ -1370,8 +1665,8 @@ bool mod_color_calculate_degamma_curve(enum dc_transfer_func_predefined trans, kvfree(rgb_degamma); } else if (trans == TRANSFER_FUNCTION_SRGB || trans == TRANSFER_FUNCTION_BT709) { - rgb_degamma = kvzalloc(sizeof(*rgb_degamma) * - (MAX_HW_POINTS + _EXTRA_POINTS), + rgb_degamma = kvcalloc(MAX_HW_POINTS + _EXTRA_POINTS, + sizeof(*rgb_degamma), GFP_KERNEL); if (!rgb_degamma) goto rgb_degamma_alloc_fail; diff --git a/drivers/gpu/drm/amd/display/modules/color/color_gamma.h b/drivers/gpu/drm/amd/display/modules/color/color_gamma.h index b7f9bc27d101..b64048991a95 100644 --- a/drivers/gpu/drm/amd/display/modules/color/color_gamma.h +++ b/drivers/gpu/drm/amd/display/modules/color/color_gamma.h @@ -32,6 +32,47 @@ struct dc_transfer_func_distributed_points; struct dc_rgb_fixed; enum dc_transfer_func_predefined; +/* For SetRegamma ADL interface support + * Must match escape type + */ +union regamma_flags { + unsigned int raw; + struct { + unsigned int gammaRampArray :1; // RegammaRamp is in use + unsigned int gammaFromEdid :1; //gamma from edid is in use + unsigned int gammaFromEdidEx :1; //gamma from edid is in use , but only for Display Id 1.2 + unsigned int gammaFromUser :1; //user custom gamma is used + unsigned int coeffFromUser :1; //coeff. A0-A3 from user is in use + unsigned int coeffFromEdid :1; //coeff. A0-A3 from edid is in use + unsigned int applyDegamma :1; //flag for additional degamma correction in driver + unsigned int gammaPredefinedSRGB :1; //flag for SRGB gamma + unsigned int gammaPredefinedPQ :1; //flag for PQ gamma + unsigned int gammaPredefinedPQ2084Interim :1; //flag for PQ gamma, lower max nits + unsigned int gammaPredefined36 :1; //flag for 3.6 gamma + unsigned int gammaPredefinedReset :1; //flag to return to previous gamma + } bits; +}; + +struct regamma_ramp { + unsigned short gamma[256*3]; // gamma ramp packed in same way as OS windows ,r , g & b +}; + +struct regamma_coeff { + int gamma[3]; + int A0[3]; + int A1[3]; + int A2[3]; + int A3[3]; +}; + +struct regamma_lut { + union regamma_flags flags; + union { + struct regamma_ramp ramp; + struct regamma_coeff coeff; + }; +}; + void setup_x_points_distribution(void); void precompute_pq(void); void precompute_de_pq(void); @@ -45,9 +86,14 @@ bool mod_color_calculate_degamma_params(struct dc_transfer_func *output_tf, bool mod_color_calculate_curve(enum dc_transfer_func_predefined trans, struct dc_transfer_func_distributed_points *points); -bool mod_color_calculate_degamma_curve(enum dc_transfer_func_predefined trans, +bool mod_color_calculate_degamma_curve(enum dc_transfer_func_predefined trans, struct dc_transfer_func_distributed_points *points); +bool calculate_user_regamma_coeff(struct dc_transfer_func *output_tf, + const struct regamma_lut *regamma); + +bool calculate_user_regamma_ramp(struct dc_transfer_func *output_tf, + const struct regamma_lut *regamma); #endif /* COLOR_MOD_COLOR_GAMMA_H_ */ diff --git a/drivers/gpu/drm/amd/display/modules/freesync/freesync.c b/drivers/gpu/drm/amd/display/modules/freesync/freesync.c index 27d4003aa2c7..fa344ceafc17 100644 --- a/drivers/gpu/drm/amd/display/modules/freesync/freesync.c +++ b/drivers/gpu/drm/amd/display/modules/freesync/freesync.c @@ -155,7 +155,8 @@ struct mod_freesync *mod_freesync_create(struct dc *dc) if (core_freesync == NULL) goto fail_alloc_context; - core_freesync->map = kzalloc(sizeof(struct freesync_entity) * MOD_FREESYNC_MAX_CONCURRENT_STREAMS, + core_freesync->map = kcalloc(MOD_FREESYNC_MAX_CONCURRENT_STREAMS, + sizeof(struct freesync_entity), GFP_KERNEL); if (core_freesync->map == NULL) diff --git a/drivers/gpu/drm/amd/display/modules/inc/mod_stats.h b/drivers/gpu/drm/amd/display/modules/inc/mod_stats.h index 3230e2adb870..3812094b52e8 100644 --- a/drivers/gpu/drm/amd/display/modules/inc/mod_stats.h +++ b/drivers/gpu/drm/amd/display/modules/inc/mod_stats.h @@ -46,6 +46,10 @@ void mod_stats_dump(struct mod_stats *mod_stats); void mod_stats_reset_data(struct mod_stats *mod_stats); +void mod_stats_update_event(struct mod_stats *mod_stats, + char *event_string, + unsigned int length); + void mod_stats_update_flip(struct mod_stats *mod_stats, unsigned long timestamp_in_ns); diff --git a/drivers/gpu/drm/amd/display/modules/stats/stats.c b/drivers/gpu/drm/amd/display/modules/stats/stats.c index 041f87b73d5f..710852ad03f3 100644 --- a/drivers/gpu/drm/amd/display/modules/stats/stats.c +++ b/drivers/gpu/drm/amd/display/modules/stats/stats.c @@ -36,9 +36,14 @@ #define DAL_STATS_ENTRIES_REGKEY_DEFAULT 0x00350000 #define DAL_STATS_ENTRIES_REGKEY_MAX 0x01000000 +#define DAL_STATS_EVENT_ENTRIES_DEFAULT 0x00000100 + #define MOD_STATS_NUM_VSYNCS 5 +#define MOD_STATS_EVENT_STRING_MAX 512 struct stats_time_cache { + unsigned int entry_id; + unsigned long flip_timestamp_in_ns; unsigned long vupdate_timestamp_in_ns; @@ -63,15 +68,26 @@ struct stats_time_cache { unsigned int flags; }; +struct stats_event_cache { + unsigned int entry_id; + char event_string[MOD_STATS_EVENT_STRING_MAX]; +}; + struct core_stats { struct mod_stats public; struct dc *dc; + bool enabled; + unsigned int entries; + unsigned int event_entries; + unsigned int entry_id; + struct stats_time_cache *time; unsigned int index; - bool enabled; - unsigned int entries; + struct stats_event_cache *events; + unsigned int event_index; + }; #define MOD_STATS_TO_CORE(mod_stats)\ @@ -99,12 +115,12 @@ struct mod_stats *mod_stats_create(struct dc *dc) unsigned int reg_data; int i = 0; + if (dc == NULL) + goto fail_construct; + core_stats = kzalloc(sizeof(struct core_stats), GFP_KERNEL); if (core_stats == NULL) - goto fail_alloc_context; - - if (dc == NULL) goto fail_construct; core_stats->dc = dc; @@ -115,33 +131,53 @@ struct mod_stats *mod_stats_create(struct dc *dc) ®_data, sizeof(unsigned int), &flag)) core_stats->enabled = reg_data; - core_stats->entries = DAL_STATS_ENTRIES_REGKEY_DEFAULT; - if (dm_read_persistent_data(dc->ctx, NULL, NULL, - DAL_STATS_ENTRIES_REGKEY, - ®_data, sizeof(unsigned int), &flag)) { - if (reg_data > DAL_STATS_ENTRIES_REGKEY_MAX) - core_stats->entries = DAL_STATS_ENTRIES_REGKEY_MAX; - else - core_stats->entries = reg_data; - } + if (core_stats->enabled) { + core_stats->entries = DAL_STATS_ENTRIES_REGKEY_DEFAULT; + if (dm_read_persistent_data(dc->ctx, NULL, NULL, + DAL_STATS_ENTRIES_REGKEY, + ®_data, sizeof(unsigned int), &flag)) { + if (reg_data > DAL_STATS_ENTRIES_REGKEY_MAX) + core_stats->entries = DAL_STATS_ENTRIES_REGKEY_MAX; + else + core_stats->entries = reg_data; + } + core_stats->time = kcalloc(core_stats->entries, + sizeof(struct stats_time_cache), + GFP_KERNEL); - core_stats->time = kzalloc(sizeof(struct stats_time_cache) * core_stats->entries, - GFP_KERNEL); + if (core_stats->time == NULL) + goto fail_construct_time; - if (core_stats->time == NULL) - goto fail_construct; + core_stats->event_entries = DAL_STATS_EVENT_ENTRIES_DEFAULT; + core_stats->events = kcalloc(core_stats->event_entries, + sizeof(struct stats_event_cache), + GFP_KERNEL); + + if (core_stats->events == NULL) + goto fail_construct_events; + + } else { + core_stats->entries = 0; + } /* Purposely leave index 0 unused so we don't need special logic to * handle calculation cases that depend on previous flip data. */ core_stats->index = 1; + core_stats->event_index = 0; + + // Keeps track of ordering within the different stats structures + core_stats->entry_id = 0; return &core_stats->public; -fail_construct: +fail_construct_events: + kfree(core_stats->time); + +fail_construct_time: kfree(core_stats); -fail_alloc_context: +fail_construct: return NULL; } @@ -153,6 +189,9 @@ void mod_stats_destroy(struct mod_stats *mod_stats) if (core_stats->time != NULL) kfree(core_stats->time); + if (core_stats->events != NULL) + kfree(core_stats->events); + kfree(core_stats); } } @@ -163,7 +202,11 @@ void mod_stats_dump(struct mod_stats *mod_stats) struct dal_logger *logger = NULL; struct core_stats *core_stats = NULL; struct stats_time_cache *time = NULL; + struct stats_event_cache *events = NULL; + unsigned int time_index = 1; + unsigned int event_index = 0; unsigned int index = 0; + struct log_entry log_entry; if (mod_stats == NULL) return; @@ -172,45 +215,62 @@ void mod_stats_dump(struct mod_stats *mod_stats) dc = core_stats->dc; logger = dc->ctx->logger; time = core_stats->time; - - //LogEntry* pLog = GetLog()->Open(LogMajor_ISR, LogMinor_ISR_FreeSyncSW); - - //if (!pLog->IsDummyEntry()) - { - dm_logger_write(logger, LOG_PROFILING, "==Display Caps==\n"); - dm_logger_write(logger, LOG_PROFILING, "\n"); - dm_logger_write(logger, LOG_PROFILING, "\n"); - - dm_logger_write(logger, LOG_PROFILING, "==Stats==\n"); - dm_logger_write(logger, LOG_PROFILING, - "render avgRender minWindow midPoint maxWindow vsyncToFlip flipToVsync #vsyncBetweenFlip #frame insertDuration vTotalMin vTotalMax eventTrigs vSyncTime1 vSyncTime2 vSyncTime3 vSyncTime4 vSyncTime5 flags\n"); - - for (int i = 0; i < core_stats->index && i < core_stats->entries; i++) { - dm_logger_write(logger, LOG_PROFILING, - "%u %u %u %u %u %u %u %u %u %u %u %u %u %u %u %u %u %u %u\n", - time[i].render_time_in_us, - time[i].avg_render_time_in_us_last_ten, - time[i].min_window, - time[i].lfc_mid_point_in_us, - time[i].max_window, - time[i].vsync_to_flip_time_in_us, - time[i].flip_to_vsync_time_in_us, - time[i].num_vsync_between_flips, - time[i].num_frames_inserted, - time[i].inserted_duration_in_us, - time[i].v_total_min, - time[i].v_total_max, - time[i].event_triggers, - time[i].v_sync_time_in_us[0], - time[i].v_sync_time_in_us[1], - time[i].v_sync_time_in_us[2], - time[i].v_sync_time_in_us[3], - time[i].v_sync_time_in_us[4], - time[i].flags); + events = core_stats->events; + + DISPLAY_STATS_BEGIN(log_entry); + + DISPLAY_STATS("==Display Caps==\n"); + + DISPLAY_STATS("==Display Stats==\n"); + + DISPLAY_STATS("%10s %10s %10s %10s %10s" + " %11s %11s %17s %10s %14s" + " %10s %10s %10s %10s %10s" + " %10s %10s %10s %10s\n", + "render", "avgRender", + "minWindow", "midPoint", "maxWindow", + "vsyncToFlip", "flipToVsync", "vsyncsBetweenFlip", + "numFrame", "insertDuration", + "vTotalMin", "vTotalMax", "eventTrigs", + "vSyncTime1", "vSyncTime2", "vSyncTime3", + "vSyncTime4", "vSyncTime5", "flags"); + + for (int i = 0; i < core_stats->entry_id; i++) { + if (event_index < core_stats->event_index && + i == events[event_index].entry_id) { + DISPLAY_STATS("%s\n", events[event_index].event_string); + event_index++; + } else if (time_index < core_stats->index && + i == time[time_index].entry_id) { + DISPLAY_STATS("%10u %10u %10u %10u %10u" + " %11u %11u %17u %10u %14u" + " %10u %10u %10u %10u %10u" + " %10u %10u %10u %10u\n", + time[time_index].render_time_in_us, + time[time_index].avg_render_time_in_us_last_ten, + time[time_index].min_window, + time[time_index].lfc_mid_point_in_us, + time[time_index].max_window, + time[time_index].vsync_to_flip_time_in_us, + time[time_index].flip_to_vsync_time_in_us, + time[time_index].num_vsync_between_flips, + time[time_index].num_frames_inserted, + time[time_index].inserted_duration_in_us, + time[time_index].v_total_min, + time[time_index].v_total_max, + time[time_index].event_triggers, + time[time_index].v_sync_time_in_us[0], + time[time_index].v_sync_time_in_us[1], + time[time_index].v_sync_time_in_us[2], + time[time_index].v_sync_time_in_us[3], + time[time_index].v_sync_time_in_us[4], + time[time_index].flags); + + time_index++; } } - //GetLog()->Close(pLog); - //GetLog()->UnSetLogMask(LogMajor_ISR, LogMinor_ISR_FreeSyncSW); + + DISPLAY_STATS_END(log_entry); } void mod_stats_reset_data(struct mod_stats *mod_stats) @@ -227,7 +287,46 @@ void mod_stats_reset_data(struct mod_stats *mod_stats) memset(core_stats->time, 0, sizeof(struct stats_time_cache) * core_stats->entries); - core_stats->index = 0; + memset(core_stats->events, 0, + sizeof(struct stats_event_cache) * core_stats->event_entries); + + core_stats->index = 1; + core_stats->event_index = 0; + + // Keeps track of ordering within the different stats structures + core_stats->entry_id = 0; +} + +void mod_stats_update_event(struct mod_stats *mod_stats, + char *event_string, + unsigned int length) +{ + struct core_stats *core_stats = NULL; + struct stats_event_cache *events = NULL; + unsigned int index = 0; + unsigned int copy_length = 0; + + if (mod_stats == NULL) + return; + + core_stats = MOD_STATS_TO_CORE(mod_stats); + + if (core_stats->event_index >= core_stats->event_entries) + return; + + events = core_stats->events; + index = core_stats->event_index; + + copy_length = length; + if (length > MOD_STATS_EVENT_STRING_MAX) + copy_length = MOD_STATS_EVENT_STRING_MAX; + + memcpy(&events[index].event_string, event_string, copy_length); + events[index].event_string[copy_length - 1] = '\0'; + + events[index].entry_id = core_stats->entry_id; + core_stats->event_index++; + core_stats->entry_id++; } void mod_stats_update_flip(struct mod_stats *mod_stats, @@ -250,7 +349,7 @@ void mod_stats_update_flip(struct mod_stats *mod_stats, time[index].flip_timestamp_in_ns = timestamp_in_ns; time[index].render_time_in_us = - timestamp_in_ns - time[index - 1].flip_timestamp_in_ns; + (timestamp_in_ns - time[index - 1].flip_timestamp_in_ns) / 1000; if (index >= 10) { for (unsigned int i = 0; i < 10; i++) @@ -261,12 +360,16 @@ void mod_stats_update_flip(struct mod_stats *mod_stats, if (time[index].num_vsync_between_flips > 0) time[index].vsync_to_flip_time_in_us = - timestamp_in_ns - time[index].vupdate_timestamp_in_ns; + (timestamp_in_ns - + time[index].vupdate_timestamp_in_ns) / 1000; else time[index].vsync_to_flip_time_in_us = - timestamp_in_ns - time[index - 1].vupdate_timestamp_in_ns; + (timestamp_in_ns - + time[index - 1].vupdate_timestamp_in_ns) / 1000; + time[index].entry_id = core_stats->entry_id; core_stats->index++; + core_stats->entry_id++; } void mod_stats_update_vupdate(struct mod_stats *mod_stats, @@ -275,6 +378,8 @@ void mod_stats_update_vupdate(struct mod_stats *mod_stats, struct core_stats *core_stats = NULL; struct stats_time_cache *time = NULL; unsigned int index = 0; + unsigned int num_vsyncs = 0; + unsigned int prev_vsync_in_ns = 0; if (mod_stats == NULL) return; @@ -286,14 +391,27 @@ void mod_stats_update_vupdate(struct mod_stats *mod_stats, time = core_stats->time; index = core_stats->index; + num_vsyncs = time[index].num_vsync_between_flips; + + if (num_vsyncs < MOD_STATS_NUM_VSYNCS) { + if (num_vsyncs == 0) { + prev_vsync_in_ns = + time[index - 1].vupdate_timestamp_in_ns; + + time[index].flip_to_vsync_time_in_us = + (timestamp_in_ns - + time[index - 1].flip_timestamp_in_ns) / + 1000; + } else { + prev_vsync_in_ns = + time[index].vupdate_timestamp_in_ns; + } - time[index].vupdate_timestamp_in_ns = timestamp_in_ns; - if (time[index].num_vsync_between_flips < MOD_STATS_NUM_VSYNCS) - time[index].v_sync_time_in_us[time[index].num_vsync_between_flips] = - timestamp_in_ns - time[index - 1].vupdate_timestamp_in_ns; - time[index].flip_to_vsync_time_in_us = - timestamp_in_ns - time[index - 1].flip_timestamp_in_ns; + time[index].v_sync_time_in_us[num_vsyncs] = + (timestamp_in_ns - prev_vsync_in_ns) / 1000; + } + time[index].vupdate_timestamp_in_ns = timestamp_in_ns; time[index].num_vsync_between_flips++; } diff --git a/drivers/gpu/drm/amd/include/amd_shared.h b/drivers/gpu/drm/amd/include/amd_shared.h index 9fa3aaef3f33..b178176b72ac 100644 --- a/drivers/gpu/drm/amd/include/amd_shared.h +++ b/drivers/gpu/drm/amd/include/amd_shared.h @@ -92,7 +92,7 @@ enum amd_powergating_state { #define AMD_CG_SUPPORT_GFX_3D_CGLS (1 << 21) #define AMD_CG_SUPPORT_DRM_MGCG (1 << 22) #define AMD_CG_SUPPORT_DF_MGCG (1 << 23) - +#define AMD_CG_SUPPORT_VCN_MGCG (1 << 24) /* PG flags */ #define AMD_PG_SUPPORT_GFX_PG (1 << 0) #define AMD_PG_SUPPORT_GFX_SMG (1 << 1) @@ -108,6 +108,27 @@ enum amd_powergating_state { #define AMD_PG_SUPPORT_GFX_QUICK_MG (1 << 11) #define AMD_PG_SUPPORT_GFX_PIPELINE (1 << 12) #define AMD_PG_SUPPORT_MMHUB (1 << 13) +#define AMD_PG_SUPPORT_VCN (1 << 14) + +enum PP_FEATURE_MASK { + PP_SCLK_DPM_MASK = 0x1, + PP_MCLK_DPM_MASK = 0x2, + PP_PCIE_DPM_MASK = 0x4, + PP_SCLK_DEEP_SLEEP_MASK = 0x8, + PP_POWER_CONTAINMENT_MASK = 0x10, + PP_UVD_HANDSHAKE_MASK = 0x20, + PP_SMC_VOLTAGE_CONTROL_MASK = 0x40, + PP_VBI_TIME_SUPPORT_MASK = 0x80, + PP_ULV_MASK = 0x100, + PP_ENABLE_GFX_CG_THRU_SMU = 0x200, + PP_CLOCK_STRETCH_MASK = 0x400, + PP_OD_FUZZY_FAN_CONTROL_MASK = 0x800, + PP_SOCCLK_DPM_MASK = 0x1000, + PP_DCEFCLK_DPM_MASK = 0x2000, + PP_OVERDRIVE_MASK = 0x4000, + PP_GFXOFF_MASK = 0x8000, + PP_ACG_MASK = 0x10000, +}; struct amd_ip_funcs { /* Name of IP block */ diff --git a/drivers/gpu/drm/amd/include/asic_reg/dce/dce_12_0_offset.h b/drivers/gpu/drm/amd/include/asic_reg/dce/dce_12_0_offset.h index f730d0629020..b6f74bf4af02 100644 --- a/drivers/gpu/drm/amd/include/asic_reg/dce/dce_12_0_offset.h +++ b/drivers/gpu/drm/amd/include/asic_reg/dce/dce_12_0_offset.h @@ -2095,6 +2095,18 @@ #define mmDC_GPIO_AUX_CTRL_2_BASE_IDX 2 #define mmDC_GPIO_RXEN 0x212f #define mmDC_GPIO_RXEN_BASE_IDX 2 +#define mmDC_GPIO_AUX_CTRL_3 0x2130 +#define mmDC_GPIO_AUX_CTRL_3_BASE_IDX 2 +#define mmDC_GPIO_AUX_CTRL_4 0x2131 +#define mmDC_GPIO_AUX_CTRL_4_BASE_IDX 2 +#define mmDC_GPIO_AUX_CTRL_5 0x2132 +#define mmDC_GPIO_AUX_CTRL_5_BASE_IDX 2 +#define mmAUXI2C_PAD_ALL_PWR_OK 0x2133 +#define mmAUXI2C_PAD_ALL_PWR_OK_BASE_IDX 2 +#define mmDC_GPIO_PULLUPEN 0x2134 +#define mmDC_GPIO_PULLUPEN_BASE_IDX 2 +#define mmDC_GPIO_AUX_CTRL_6 0x2135 +#define mmDC_GPIO_AUX_CTRL_6_BASE_IDX 2 #define mmBPHYC_DAC_MACRO_CNTL 0x2136 #define mmBPHYC_DAC_MACRO_CNTL_BASE_IDX 2 #define mmDAC_MACRO_CNTL_RESERVED0 0x2136 diff --git a/drivers/gpu/drm/amd/include/asic_reg/dce/dce_12_0_sh_mask.h b/drivers/gpu/drm/amd/include/asic_reg/dce/dce_12_0_sh_mask.h index 6d3162c42957..bcd190a3fcdd 100644 --- a/drivers/gpu/drm/amd/include/asic_reg/dce/dce_12_0_sh_mask.h +++ b/drivers/gpu/drm/amd/include/asic_reg/dce/dce_12_0_sh_mask.h @@ -10971,6 +10971,158 @@ #define DC_GPIO_RXEN__DC_GPIO_BLON_RXEN_MASK 0x00100000L #define DC_GPIO_RXEN__DC_GPIO_DIGON_RXEN_MASK 0x00200000L #define DC_GPIO_RXEN__DC_GPIO_ENA_BL_RXEN_MASK 0x00400000L +//DC_GPIO_AUX_CTRL_3 +#define DC_GPIO_AUX_CTRL_3__AUX1_NEN_RTERM__SHIFT 0x0 +#define DC_GPIO_AUX_CTRL_3__AUX2_NEN_RTERM__SHIFT 0x1 +#define DC_GPIO_AUX_CTRL_3__AUX3_NEN_RTERM__SHIFT 0x2 +#define DC_GPIO_AUX_CTRL_3__AUX4_NEN_RTERM__SHIFT 0x3 +#define DC_GPIO_AUX_CTRL_3__AUX5_NEN_RTERM__SHIFT 0x4 +#define DC_GPIO_AUX_CTRL_3__AUX6_NEN_RTERM__SHIFT 0x5 +#define DC_GPIO_AUX_CTRL_3__AUX1_DP_DN_SWAP__SHIFT 0x8 +#define DC_GPIO_AUX_CTRL_3__AUX2_DP_DN_SWAP__SHIFT 0x9 +#define DC_GPIO_AUX_CTRL_3__AUX3_DP_DN_SWAP__SHIFT 0xa +#define DC_GPIO_AUX_CTRL_3__AUX4_DP_DN_SWAP__SHIFT 0xb +#define DC_GPIO_AUX_CTRL_3__AUX5_DP_DN_SWAP__SHIFT 0xc +#define DC_GPIO_AUX_CTRL_3__AUX6_DP_DN_SWAP__SHIFT 0xd +#define DC_GPIO_AUX_CTRL_3__AUX1_HYS_TUNE__SHIFT 0x10 +#define DC_GPIO_AUX_CTRL_3__AUX2_HYS_TUNE__SHIFT 0x12 +#define DC_GPIO_AUX_CTRL_3__AUX3_HYS_TUNE__SHIFT 0x14 +#define DC_GPIO_AUX_CTRL_3__AUX4_HYS_TUNE__SHIFT 0x16 +#define DC_GPIO_AUX_CTRL_3__AUX5_HYS_TUNE__SHIFT 0x18 +#define DC_GPIO_AUX_CTRL_3__AUX6_HYS_TUNE__SHIFT 0x1a +#define DC_GPIO_AUX_CTRL_3__AUX1_NEN_RTERM_MASK 0x00000001L +#define DC_GPIO_AUX_CTRL_3__AUX2_NEN_RTERM_MASK 0x00000002L +#define DC_GPIO_AUX_CTRL_3__AUX3_NEN_RTERM_MASK 0x00000004L +#define DC_GPIO_AUX_CTRL_3__AUX4_NEN_RTERM_MASK 0x00000008L +#define DC_GPIO_AUX_CTRL_3__AUX5_NEN_RTERM_MASK 0x00000010L +#define DC_GPIO_AUX_CTRL_3__AUX6_NEN_RTERM_MASK 0x00000020L +#define DC_GPIO_AUX_CTRL_3__AUX1_DP_DN_SWAP_MASK 0x00000100L +#define DC_GPIO_AUX_CTRL_3__AUX2_DP_DN_SWAP_MASK 0x00000200L +#define DC_GPIO_AUX_CTRL_3__AUX3_DP_DN_SWAP_MASK 0x00000400L +#define DC_GPIO_AUX_CTRL_3__AUX4_DP_DN_SWAP_MASK 0x00000800L +#define DC_GPIO_AUX_CTRL_3__AUX5_DP_DN_SWAP_MASK 0x00001000L +#define DC_GPIO_AUX_CTRL_3__AUX6_DP_DN_SWAP_MASK 0x00002000L +#define DC_GPIO_AUX_CTRL_3__AUX1_HYS_TUNE_MASK 0x00030000L +#define DC_GPIO_AUX_CTRL_3__AUX2_HYS_TUNE_MASK 0x000C0000L +#define DC_GPIO_AUX_CTRL_3__AUX3_HYS_TUNE_MASK 0x00300000L +#define DC_GPIO_AUX_CTRL_3__AUX4_HYS_TUNE_MASK 0x00C00000L +#define DC_GPIO_AUX_CTRL_3__AUX5_HYS_TUNE_MASK 0x03000000L +#define DC_GPIO_AUX_CTRL_3__AUX6_HYS_TUNE_MASK 0x0C000000L +//DC_GPIO_AUX_CTRL_4 +#define DC_GPIO_AUX_CTRL_4__AUX1_AUX_CTRL__SHIFT 0x0 +#define DC_GPIO_AUX_CTRL_4__AUX2_AUX_CTRL__SHIFT 0x4 +#define DC_GPIO_AUX_CTRL_4__AUX3_AUX_CTRL__SHIFT 0x8 +#define DC_GPIO_AUX_CTRL_4__AUX4_AUX_CTRL__SHIFT 0xc +#define DC_GPIO_AUX_CTRL_4__AUX5_AUX_CTRL__SHIFT 0x10 +#define DC_GPIO_AUX_CTRL_4__AUX6_AUX_CTRL__SHIFT 0x14 +#define DC_GPIO_AUX_CTRL_4__AUX1_AUX_CTRL_MASK 0x0000000FL +#define DC_GPIO_AUX_CTRL_4__AUX2_AUX_CTRL_MASK 0x000000F0L +#define DC_GPIO_AUX_CTRL_4__AUX3_AUX_CTRL_MASK 0x00000F00L +#define DC_GPIO_AUX_CTRL_4__AUX4_AUX_CTRL_MASK 0x0000F000L +#define DC_GPIO_AUX_CTRL_4__AUX5_AUX_CTRL_MASK 0x000F0000L +#define DC_GPIO_AUX_CTRL_4__AUX6_AUX_CTRL_MASK 0x00F00000L +//DC_GPIO_AUX_CTRL_5 +#define DC_GPIO_AUX_CTRL_5__AUX1_VOD_TUNE__SHIFT 0x0 +#define DC_GPIO_AUX_CTRL_5__AUX2_VOD_TUNE__SHIFT 0x2 +#define DC_GPIO_AUX_CTRL_5__AUX3_VOD_TUNE__SHIFT 0x4 +#define DC_GPIO_AUX_CTRL_5__AUX4_VOD_TUNE__SHIFT 0x6 +#define DC_GPIO_AUX_CTRL_5__AUX5_VOD_TUNE__SHIFT 0x8 +#define DC_GPIO_AUX_CTRL_5__AUX6_VOD_TUNE__SHIFT 0xa +#define DC_GPIO_AUX_CTRL_5__DDC_PAD1_I2CMODE__SHIFT 0xc +#define DC_GPIO_AUX_CTRL_5__DDC_PAD2_I2CMODE__SHIFT 0xd +#define DC_GPIO_AUX_CTRL_5__DDC_PAD3_I2CMODE__SHIFT 0xe +#define DC_GPIO_AUX_CTRL_5__DDC_PAD4_I2CMODE__SHIFT 0xf +#define DC_GPIO_AUX_CTRL_5__DDC_PAD5_I2CMODE__SHIFT 0x10 +#define DC_GPIO_AUX_CTRL_5__DDC_PAD6_I2CMODE__SHIFT 0x11 +#define DC_GPIO_AUX_CTRL_5__DDC1_I2C_VPH_1V2_EN__SHIFT 0x12 +#define DC_GPIO_AUX_CTRL_5__DDC2_I2C_VPH_1V2_EN__SHIFT 0x13 +#define DC_GPIO_AUX_CTRL_5__DDC3_I2C_VPH_1V2_EN__SHIFT 0x14 +#define DC_GPIO_AUX_CTRL_5__DDC4_I2C_VPH_1V2_EN__SHIFT 0x15 +#define DC_GPIO_AUX_CTRL_5__DDC5_I2C_VPH_1V2_EN__SHIFT 0x16 +#define DC_GPIO_AUX_CTRL_5__DDC6_I2C_VPH_1V2_EN__SHIFT 0x17 +#define DC_GPIO_AUX_CTRL_5__DDC1_PAD_I2C_CTRL__SHIFT 0x18 +#define DC_GPIO_AUX_CTRL_5__DDC2_PAD_I2C_CTRL__SHIFT 0x19 +#define DC_GPIO_AUX_CTRL_5__DDC3_PAD_I2C_CTRL__SHIFT 0x1a +#define DC_GPIO_AUX_CTRL_5__DDC4_PAD_I2C_CTRL__SHIFT 0x1b +#define DC_GPIO_AUX_CTRL_5__DDC5_PAD_I2C_CTRL__SHIFT 0x1c +#define DC_GPIO_AUX_CTRL_5__DDC6_PAD_I2C_CTRL__SHIFT 0x1d +#define DC_GPIO_AUX_CTRL_5__AUX1_VOD_TUNE_MASK 0x00000003L +#define DC_GPIO_AUX_CTRL_5__AUX2_VOD_TUNE_MASK 0x0000000CL +#define DC_GPIO_AUX_CTRL_5__AUX3_VOD_TUNE_MASK 0x00000030L +#define DC_GPIO_AUX_CTRL_5__AUX4_VOD_TUNE_MASK 0x000000C0L +#define DC_GPIO_AUX_CTRL_5__AUX5_VOD_TUNE_MASK 0x00000300L +#define DC_GPIO_AUX_CTRL_5__AUX6_VOD_TUNE_MASK 0x00000C00L +#define DC_GPIO_AUX_CTRL_5__DDC_PAD1_I2CMODE_MASK 0x00001000L +#define DC_GPIO_AUX_CTRL_5__DDC_PAD2_I2CMODE_MASK 0x00002000L +#define DC_GPIO_AUX_CTRL_5__DDC_PAD3_I2CMODE_MASK 0x00004000L +#define DC_GPIO_AUX_CTRL_5__DDC_PAD4_I2CMODE_MASK 0x00008000L +#define DC_GPIO_AUX_CTRL_5__DDC_PAD5_I2CMODE_MASK 0x00010000L +#define DC_GPIO_AUX_CTRL_5__DDC_PAD6_I2CMODE_MASK 0x00020000L +#define DC_GPIO_AUX_CTRL_5__DDC1_I2C_VPH_1V2_EN_MASK 0x00040000L +#define DC_GPIO_AUX_CTRL_5__DDC2_I2C_VPH_1V2_EN_MASK 0x00080000L +#define DC_GPIO_AUX_CTRL_5__DDC3_I2C_VPH_1V2_EN_MASK 0x00100000L +#define DC_GPIO_AUX_CTRL_5__DDC4_I2C_VPH_1V2_EN_MASK 0x00200000L +#define DC_GPIO_AUX_CTRL_5__DDC5_I2C_VPH_1V2_EN_MASK 0x00400000L +#define DC_GPIO_AUX_CTRL_5__DDC6_I2C_VPH_1V2_EN_MASK 0x00800000L +#define DC_GPIO_AUX_CTRL_5__DDC1_PAD_I2C_CTRL_MASK 0x01000000L +#define DC_GPIO_AUX_CTRL_5__DDC2_PAD_I2C_CTRL_MASK 0x02000000L +#define DC_GPIO_AUX_CTRL_5__DDC3_PAD_I2C_CTRL_MASK 0x04000000L +#define DC_GPIO_AUX_CTRL_5__DDC4_PAD_I2C_CTRL_MASK 0x08000000L +#define DC_GPIO_AUX_CTRL_5__DDC5_PAD_I2C_CTRL_MASK 0x10000000L +#define DC_GPIO_AUX_CTRL_5__DDC6_PAD_I2C_CTRL_MASK 0x20000000L +//AUXI2C_PAD_ALL_PWR_OK +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY1_ALL_PWR_OK__SHIFT 0x0 +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY2_ALL_PWR_OK__SHIFT 0x1 +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY3_ALL_PWR_OK__SHIFT 0x2 +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY4_ALL_PWR_OK__SHIFT 0x3 +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY5_ALL_PWR_OK__SHIFT 0x4 +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY6_ALL_PWR_OK__SHIFT 0x5 +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY1_ALL_PWR_OK_MASK 0x00000001L +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY2_ALL_PWR_OK_MASK 0x00000002L +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY3_ALL_PWR_OK_MASK 0x00000004L +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY4_ALL_PWR_OK_MASK 0x00000008L +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY5_ALL_PWR_OK_MASK 0x00000010L +#define AUXI2C_PAD_ALL_PWR_OK__AUXI2C_PHY6_ALL_PWR_OK_MASK 0x00000020L +//DC_GPIO_PULLUPEN +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICA_PU_EN__SHIFT 0x0 +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICB_PU_EN__SHIFT 0x1 +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICC_PU_EN__SHIFT 0x2 +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICD_PU_EN__SHIFT 0x3 +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICE_PU_EN__SHIFT 0x4 +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICF_PU_EN__SHIFT 0x5 +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICG_PU_EN__SHIFT 0x6 +#define DC_GPIO_PULLUPEN__DC_GPIO_HSYNCA_PU_EN__SHIFT 0x8 +#define DC_GPIO_PULLUPEN__DC_GPIO_VSYNCA_PU_EN__SHIFT 0x9 +#define DC_GPIO_PULLUPEN__DC_GPIO_HPD1_PU_EN__SHIFT 0xe +#define DC_GPIO_PULLUPEN__DC_GPIO_BLON_PU_EN__SHIFT 0x14 +#define DC_GPIO_PULLUPEN__DC_GPIO_DIGON_PU_EN__SHIFT 0x15 +#define DC_GPIO_PULLUPEN__DC_GPIO_ENA_BL_PU_EN__SHIFT 0x16 +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICA_PU_EN_MASK 0x00000001L +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICB_PU_EN_MASK 0x00000002L +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICC_PU_EN_MASK 0x00000004L +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICD_PU_EN_MASK 0x00000008L +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICE_PU_EN_MASK 0x00000010L +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICF_PU_EN_MASK 0x00000020L +#define DC_GPIO_PULLUPEN__DC_GPIO_GENERICG_PU_EN_MASK 0x00000040L +#define DC_GPIO_PULLUPEN__DC_GPIO_HSYNCA_PU_EN_MASK 0x00000100L +#define DC_GPIO_PULLUPEN__DC_GPIO_VSYNCA_PU_EN_MASK 0x00000200L +#define DC_GPIO_PULLUPEN__DC_GPIO_HPD1_PU_EN_MASK 0x00004000L +#define DC_GPIO_PULLUPEN__DC_GPIO_BLON_PU_EN_MASK 0x00100000L +#define DC_GPIO_PULLUPEN__DC_GPIO_DIGON_PU_EN_MASK 0x00200000L +#define DC_GPIO_PULLUPEN__DC_GPIO_ENA_BL_PU_EN_MASK 0x00400000L +//DC_GPIO_AUX_CTRL_6 +#define DC_GPIO_AUX_CTRL_6__AUX1_PAD_RXSEL__SHIFT 0x0 +#define DC_GPIO_AUX_CTRL_6__AUX2_PAD_RXSEL__SHIFT 0x2 +#define DC_GPIO_AUX_CTRL_6__AUX3_PAD_RXSEL__SHIFT 0x4 +#define DC_GPIO_AUX_CTRL_6__AUX4_PAD_RXSEL__SHIFT 0x6 +#define DC_GPIO_AUX_CTRL_6__AUX5_PAD_RXSEL__SHIFT 0x8 +#define DC_GPIO_AUX_CTRL_6__AUX6_PAD_RXSEL__SHIFT 0xa +#define DC_GPIO_AUX_CTRL_6__AUX1_PAD_RXSEL_MASK 0x00000003L +#define DC_GPIO_AUX_CTRL_6__AUX2_PAD_RXSEL_MASK 0x0000000CL +#define DC_GPIO_AUX_CTRL_6__AUX3_PAD_RXSEL_MASK 0x00000030L +#define DC_GPIO_AUX_CTRL_6__AUX4_PAD_RXSEL_MASK 0x000000C0L +#define DC_GPIO_AUX_CTRL_6__AUX5_PAD_RXSEL_MASK 0x00000300L +#define DC_GPIO_AUX_CTRL_6__AUX6_PAD_RXSEL_MASK 0x00000C00L //BPHYC_DAC_MACRO_CNTL #define BPHYC_DAC_MACRO_CNTL__BPHYC_DAC_WHITE_LEVEL__SHIFT 0x0 #define BPHYC_DAC_MACRO_CNTL__BPHYC_DAC_WHITE_FINE_CONTROL__SHIFT 0x8 diff --git a/drivers/gpu/drm/amd/include/asic_reg/dcn/dcn_1_0_offset.h b/drivers/gpu/drm/amd/include/asic_reg/dcn/dcn_1_0_offset.h index 4ccf9681c45d..721c61171045 100644 --- a/drivers/gpu/drm/amd/include/asic_reg/dcn/dcn_1_0_offset.h +++ b/drivers/gpu/drm/amd/include/asic_reg/dcn/dcn_1_0_offset.h @@ -3895,6 +3895,10 @@ #define mmCM0_CM_MEM_PWR_CTRL_BASE_IDX 2 #define mmCM0_CM_MEM_PWR_STATUS 0x0d33 #define mmCM0_CM_MEM_PWR_STATUS_BASE_IDX 2 +#define mmCM0_CM_TEST_DEBUG_INDEX 0x0d35 +#define mmCM0_CM_TEST_DEBUG_INDEX_BASE_IDX 2 +#define mmCM0_CM_TEST_DEBUG_DATA 0x0d36 +#define mmCM0_CM_TEST_DEBUG_DATA_BASE_IDX 2 // addressBlock: dce_dc_dpp0_dispdec_dpp_dcperfmon_dc_perfmon_dispdec @@ -4367,7 +4371,10 @@ #define mmCM1_CM_MEM_PWR_CTRL_BASE_IDX 2 #define mmCM1_CM_MEM_PWR_STATUS 0x0e4e #define mmCM1_CM_MEM_PWR_STATUS_BASE_IDX 2 - +#define mmCM1_CM_TEST_DEBUG_INDEX 0x0e50 +#define mmCM1_CM_TEST_DEBUG_INDEX_BASE_IDX 2 +#define mmCM1_CM_TEST_DEBUG_DATA 0x0e51 +#define mmCM1_CM_TEST_DEBUG_DATA_BASE_IDX 2 // addressBlock: dce_dc_dpp1_dispdec_dpp_dcperfmon_dc_perfmon_dispdec // base address: 0x399c @@ -4839,7 +4846,10 @@ #define mmCM2_CM_MEM_PWR_CTRL_BASE_IDX 2 #define mmCM2_CM_MEM_PWR_STATUS 0x0f69 #define mmCM2_CM_MEM_PWR_STATUS_BASE_IDX 2 - +#define mmCM2_CM_TEST_DEBUG_INDEX 0x0f6b +#define mmCM2_CM_TEST_DEBUG_INDEX_BASE_IDX 2 +#define mmCM2_CM_TEST_DEBUG_DATA 0x0f6c +#define mmCM2_CM_TEST_DEBUG_DATA_BASE_IDX 2 // addressBlock: dce_dc_dpp2_dispdec_dpp_dcperfmon_dc_perfmon_dispdec // base address: 0x3e08 @@ -5311,7 +5321,10 @@ #define mmCM3_CM_MEM_PWR_CTRL_BASE_IDX 2 #define mmCM3_CM_MEM_PWR_STATUS 0x1084 #define mmCM3_CM_MEM_PWR_STATUS_BASE_IDX 2 - +#define mmCM3_CM_TEST_DEBUG_INDEX 0x1086 +#define mmCM3_CM_TEST_DEBUG_INDEX_BASE_IDX 2 +#define mmCM3_CM_TEST_DEBUG_DATA 0x1087 +#define mmCM3_CM_TEST_DEBUG_DATA_BASE_IDX 2 // addressBlock: dce_dc_dpp3_dispdec_dpp_dcperfmon_dc_perfmon_dispdec // base address: 0x4274 diff --git a/drivers/gpu/drm/amd/include/asic_reg/dcn/dcn_1_0_sh_mask.h b/drivers/gpu/drm/amd/include/asic_reg/dcn/dcn_1_0_sh_mask.h index e2a2f114bd8e..e7c0cad41081 100644 --- a/drivers/gpu/drm/amd/include/asic_reg/dcn/dcn_1_0_sh_mask.h +++ b/drivers/gpu/drm/amd/include/asic_reg/dcn/dcn_1_0_sh_mask.h @@ -14049,6 +14049,14 @@ #define CM0_CM_MEM_PWR_STATUS__RGAM_MEM_PWR_STATE__SHIFT 0x2 #define CM0_CM_MEM_PWR_STATUS__SHARED_MEM_PWR_STATE_MASK 0x00000003L #define CM0_CM_MEM_PWR_STATUS__RGAM_MEM_PWR_STATE_MASK 0x0000000CL +//CM0_CM_TEST_DEBUG_INDEX +#define CM0_CM_TEST_DEBUG_INDEX__CM_TEST_DEBUG_INDEX__SHIFT 0x0 +#define CM0_CM_TEST_DEBUG_INDEX__CM_TEST_DEBUG_WRITE_EN__SHIFT 0x8 +#define CM0_CM_TEST_DEBUG_INDEX__CM_TEST_DEBUG_INDEX_MASK 0x000000FFL +#define CM0_CM_TEST_DEBUG_INDEX__CM_TEST_DEBUG_WRITE_EN_MASK 0x00000100L +//CM0_CM_TEST_DEBUG_DATA +#define CM0_CM_TEST_DEBUG_DATA__CM_TEST_DEBUG_DATA__SHIFT 0x0 +#define CM0_CM_TEST_DEBUG_DATA__CM_TEST_DEBUG_DATA_MASK 0xFFFFFFFFL // addressBlock: dce_dc_dpp0_dispdec_dpp_dcperfmon_dc_perfmon_dispdec diff --git a/drivers/gpu/drm/amd/include/asic_reg/df/df_1_7_default.h b/drivers/gpu/drm/amd/include/asic_reg/df/df_1_7_default.h new file mode 100644 index 000000000000..9e19e723081b --- /dev/null +++ b/drivers/gpu/drm/amd/include/asic_reg/df/df_1_7_default.h @@ -0,0 +1,26 @@ +/* + * Copyright (C) 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included + * in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS + * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +#ifndef _df_1_7_DEFAULT_HEADER +#define _df_1_7_DEFAULT_HEADER + +#define mmFabricConfigAccessControl_DEFAULT 0x00000000 + +#endif diff --git a/drivers/gpu/drm/amd/include/asic_reg/df/df_1_7_offset.h b/drivers/gpu/drm/amd/include/asic_reg/df/df_1_7_offset.h new file mode 100644 index 000000000000..e6044e27a913 --- /dev/null +++ b/drivers/gpu/drm/amd/include/asic_reg/df/df_1_7_offset.h @@ -0,0 +1,37 @@ +/* + * Copyright (C) 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included + * in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS + * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +#ifndef _df_1_7_OFFSET_HEADER +#define _df_1_7_OFFSET_HEADER + +#define mmFabricConfigAccessControl 0x0410 +#define mmFabricConfigAccessControl_BASE_IDX 0 + +#define mmDF_PIE_AON0_DfGlobalClkGater 0x00fc +#define mmDF_PIE_AON0_DfGlobalClkGater_BASE_IDX 0 + +#define mmDF_CS_AON0_DramBaseAddress0 0x0044 +#define mmDF_CS_AON0_DramBaseAddress0_BASE_IDX 0 + +#define mmDF_CS_AON0_CoherentSlaveModeCtrlA0 0x0214 +#define mmDF_CS_AON0_CoherentSlaveModeCtrlA0_BASE_IDX 0 + + +#endif diff --git a/drivers/gpu/drm/amd/include/asic_reg/df/df_1_7_sh_mask.h b/drivers/gpu/drm/amd/include/asic_reg/df/df_1_7_sh_mask.h new file mode 100644 index 000000000000..a78c99480e2d --- /dev/null +++ b/drivers/gpu/drm/amd/include/asic_reg/df/df_1_7_sh_mask.h @@ -0,0 +1,52 @@ +/* + * Copyright (C) 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included + * in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS + * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +#ifndef _df_1_7_SH_MASK_HEADER +#define _df_1_7_SH_MASK_HEADER + +/* FabricConfigAccessControl */ +#define FabricConfigAccessControl__CfgRegInstAccEn__SHIFT 0x0 +#define FabricConfigAccessControl__CfgRegInstAccRegLock__SHIFT 0x1 +#define FabricConfigAccessControl__CfgRegInstID__SHIFT 0x10 +#define FabricConfigAccessControl__CfgRegInstAccEn_MASK 0x00000001L +#define FabricConfigAccessControl__CfgRegInstAccRegLock_MASK 0x00000002L +#define FabricConfigAccessControl__CfgRegInstID_MASK 0x00FF0000L + +/* DF_PIE_AON0_DfGlobalClkGater */ +#define DF_PIE_AON0_DfGlobalClkGater__MGCGMode__SHIFT 0x0 +#define DF_PIE_AON0_DfGlobalClkGater__MGCGMode_MASK 0x0000000FL + +/* DF_CS_AON0_DramBaseAddress0 */ +#define DF_CS_AON0_DramBaseAddress0__AddrRngVal__SHIFT 0x0 +#define DF_CS_AON0_DramBaseAddress0__LgcyMmioHoleEn__SHIFT 0x1 +#define DF_CS_AON0_DramBaseAddress0__IntLvNumChan__SHIFT 0x4 +#define DF_CS_AON0_DramBaseAddress0__IntLvAddrSel__SHIFT 0x8 +#define DF_CS_AON0_DramBaseAddress0__DramBaseAddr__SHIFT 0xc +#define DF_CS_AON0_DramBaseAddress0__AddrRngVal_MASK 0x00000001L +#define DF_CS_AON0_DramBaseAddress0__LgcyMmioHoleEn_MASK 0x00000002L +#define DF_CS_AON0_DramBaseAddress0__IntLvNumChan_MASK 0x000000F0L +#define DF_CS_AON0_DramBaseAddress0__IntLvAddrSel_MASK 0x00000700L +#define DF_CS_AON0_DramBaseAddress0__DramBaseAddr_MASK 0xFFFFF000L + +//DF_CS_AON0_CoherentSlaveModeCtrlA0 +#define DF_CS_AON0_CoherentSlaveModeCtrlA0__ForceParWrRMW__SHIFT 0x3 +#define DF_CS_AON0_CoherentSlaveModeCtrlA0__ForceParWrRMW_MASK 0x00000008L + +#endif diff --git a/drivers/gpu/drm/amd/include/asic_reg/df/df_3_6_default.h b/drivers/gpu/drm/amd/include/asic_reg/df/df_3_6_default.h new file mode 100644 index 000000000000..e58c207ac980 --- /dev/null +++ b/drivers/gpu/drm/amd/include/asic_reg/df/df_3_6_default.h @@ -0,0 +1,26 @@ +/* + * Copyright (C) 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included + * in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS + * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +#ifndef _df_3_6_DEFAULT_HEADER +#define _df_3_6_DEFAULT_HEADER + +#define mmFabricConfigAccessControl_DEFAULT 0x00000000 + +#endif diff --git a/drivers/gpu/drm/amd/include/asic_reg/df/df_3_6_offset.h b/drivers/gpu/drm/amd/include/asic_reg/df/df_3_6_offset.h new file mode 100644 index 000000000000..a9575db8d7aa --- /dev/null +++ b/drivers/gpu/drm/amd/include/asic_reg/df/df_3_6_offset.h @@ -0,0 +1,33 @@ +/* + * Copyright (C) 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included + * in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS + * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +#ifndef _df_3_6_OFFSET_HEADER +#define _df_3_6_OFFSET_HEADER + +#define mmFabricConfigAccessControl 0x0410 +#define mmFabricConfigAccessControl_BASE_IDX 0 + +#define mmDF_PIE_AON0_DfGlobalClkGater 0x00fc +#define mmDF_PIE_AON0_DfGlobalClkGater_BASE_IDX 0 + +#define mmDF_CS_UMC_AON0_DramBaseAddress0 0x0044 +#define mmDF_CS_UMC_AON0_DramBaseAddress0_BASE_IDX 0 + +#endif diff --git a/drivers/gpu/drm/amd/include/asic_reg/df/df_3_6_sh_mask.h b/drivers/gpu/drm/amd/include/asic_reg/df/df_3_6_sh_mask.h new file mode 100644 index 000000000000..06fac509e987 --- /dev/null +++ b/drivers/gpu/drm/amd/include/asic_reg/df/df_3_6_sh_mask.h @@ -0,0 +1,48 @@ +/* + * Copyright (C) 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included + * in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS + * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +#ifndef _df_3_6_SH_MASK_HEADER +#define _df_3_6_SH_MASK_HEADER + +/* FabricConfigAccessControl */ +#define FabricConfigAccessControl__CfgRegInstAccEn__SHIFT 0x0 +#define FabricConfigAccessControl__CfgRegInstAccRegLock__SHIFT 0x1 +#define FabricConfigAccessControl__CfgRegInstID__SHIFT 0x10 +#define FabricConfigAccessControl__CfgRegInstAccEn_MASK 0x00000001L +#define FabricConfigAccessControl__CfgRegInstAccRegLock_MASK 0x00000002L +#define FabricConfigAccessControl__CfgRegInstID_MASK 0x00FF0000L + +/* DF_PIE_AON0_DfGlobalClkGater */ +#define DF_PIE_AON0_DfGlobalClkGater__MGCGMode__SHIFT 0x0 +#define DF_PIE_AON0_DfGlobalClkGater__MGCGMode_MASK 0x0000000FL + +/* DF_CS_AON0_DramBaseAddress0 */ +#define DF_CS_UMC_AON0_DramBaseAddress0__AddrRngVal__SHIFT 0x0 +#define DF_CS_UMC_AON0_DramBaseAddress0__LgcyMmioHoleEn__SHIFT 0x1 +#define DF_CS_UMC_AON0_DramBaseAddress0__IntLvNumChan__SHIFT 0x2 +#define DF_CS_UMC_AON0_DramBaseAddress0__IntLvAddrSel__SHIFT 0x9 +#define DF_CS_UMC_AON0_DramBaseAddress0__DramBaseAddr__SHIFT 0xc +#define DF_CS_UMC_AON0_DramBaseAddress0__AddrRngVal_MASK 0x00000001L +#define DF_CS_UMC_AON0_DramBaseAddress0__LgcyMmioHoleEn_MASK 0x00000002L +#define DF_CS_UMC_AON0_DramBaseAddress0__IntLvNumChan_MASK 0x0000003CL +#define DF_CS_UMC_AON0_DramBaseAddress0__IntLvAddrSel_MASK 0x00000E00L +#define DF_CS_UMC_AON0_DramBaseAddress0__DramBaseAddr_MASK 0xFFFFF000L + +#endif diff --git a/drivers/gpu/drm/amd/include/atombios.h b/drivers/gpu/drm/amd/include/atombios.h index f696bbb643ef..7931502fa54f 100644 --- a/drivers/gpu/drm/amd/include/atombios.h +++ b/drivers/gpu/drm/amd/include/atombios.h @@ -632,6 +632,13 @@ typedef struct _COMPUTE_MEMORY_CLOCK_PARAM_PARAMETERS_V2_2 ULONG ulReserved; }COMPUTE_MEMORY_CLOCK_PARAM_PARAMETERS_V2_2; +typedef struct _COMPUTE_MEMORY_CLOCK_PARAM_PARAMETERS_V2_3 +{ + COMPUTE_MEMORY_ENGINE_PLL_PARAMETERS_V4 ulClock; + USHORT usMclk_fcw_frac; //fractional divider of fcw = usSclk_fcw_frac/65536 + USHORT usMclk_fcw_int; //integer divider of fcwc +}COMPUTE_MEMORY_CLOCK_PARAM_PARAMETERS_V2_3; + //Input parameter of DynamicMemorySettingsTable //when ATOM_COMPUTE_CLOCK_FREQ.ulComputeClockFlag = COMPUTE_MEMORY_PLL_PARAM typedef struct _DYNAMICE_MEMORY_SETTINGS_PARAMETER diff --git a/drivers/gpu/drm/amd/include/atomfirmware.h b/drivers/gpu/drm/amd/include/atomfirmware.h index de177ce8ca80..092d800b703a 100644 --- a/drivers/gpu/drm/amd/include/atomfirmware.h +++ b/drivers/gpu/drm/amd/include/atomfirmware.h @@ -1219,6 +1219,41 @@ struct atom_gfx_info_v2_3 { uint32_t rm21_sram_vmin_value; }; +struct atom_gfx_info_v2_4 { + struct atom_common_table_header table_header; + uint8_t gfxip_min_ver; + uint8_t gfxip_max_ver; + uint8_t gc_num_se; + uint8_t max_tile_pipes; + uint8_t gc_num_cu_per_sh; + uint8_t gc_num_sh_per_se; + uint8_t gc_num_rb_per_se; + uint8_t gc_num_tccs; + uint32_t regaddr_cp_dma_src_addr; + uint32_t regaddr_cp_dma_src_addr_hi; + uint32_t regaddr_cp_dma_dst_addr; + uint32_t regaddr_cp_dma_dst_addr_hi; + uint32_t regaddr_cp_dma_command; + uint32_t regaddr_cp_status; + uint32_t regaddr_rlc_gpu_clock_32; + uint32_t rlc_gpu_timer_refclk; + uint8_t active_cu_per_sh; + uint8_t active_rb_per_se; + uint16_t gcgoldenoffset; + uint16_t gc_num_gprs; + uint16_t gc_gsprim_buff_depth; + uint16_t gc_parameter_cache_depth; + uint16_t gc_wave_size; + uint16_t gc_max_waves_per_simd; + uint16_t gc_lds_size; + uint8_t gc_num_max_gs_thds; + uint8_t gc_gs_table_depth; + uint8_t gc_double_offchip_lds_buffer; + uint8_t gc_max_scratch_slots_per_cu; + uint32_t sram_rm_fuses_val; + uint32_t sram_custom_rm_fuses_val; +}; + /* *************************************************************************** Data Table smu_info structure @@ -1991,17 +2026,15 @@ enum atom_smu11_syspll_id { SMU11_SYSPLL3_1_ID = 6, }; - enum atom_smu11_syspll0_clock_id { - SMU11_SYSPLL0_SOCCLK_ID = 0, // SOCCLK - SMU11_SYSPLL0_MP0CLK_ID = 1, // MP0CLK - SMU11_SYSPLL0_DCLK_ID = 2, // DCLK - SMU11_SYSPLL0_VCLK_ID = 3, // VCLK - SMU11_SYSPLL0_ECLK_ID = 4, // ECLK + SMU11_SYSPLL0_ECLK_ID = 0, // ECLK + SMU11_SYSPLL0_SOCCLK_ID = 1, // SOCCLK + SMU11_SYSPLL0_MP0CLK_ID = 2, // MP0CLK + SMU11_SYSPLL0_DCLK_ID = 3, // DCLK + SMU11_SYSPLL0_VCLK_ID = 4, // VCLK SMU11_SYSPLL0_DCEFCLK_ID = 5, // DCEFCLK }; - enum atom_smu11_syspll1_0_clock_id { SMU11_SYSPLL1_0_UCLKA_ID = 0, // UCLK_a }; diff --git a/drivers/gpu/drm/amd/include/cgs_common.h b/drivers/gpu/drm/amd/include/cgs_common.h index f2814ae7ecdd..a69deb3a2ac0 100644 --- a/drivers/gpu/drm/amd/include/cgs_common.h +++ b/drivers/gpu/drm/amd/include/cgs_common.h @@ -42,20 +42,6 @@ enum cgs_ind_reg { CGS_IND_REG__AUDIO_ENDPT }; -/** - * enum cgs_engine - Engines that can be statically power-gated - */ -enum cgs_engine { - CGS_ENGINE__UVD, - CGS_ENGINE__VCE, - CGS_ENGINE__VP8, - CGS_ENGINE__ACP_DMA, - CGS_ENGINE__ACP_DSP0, - CGS_ENGINE__ACP_DSP1, - CGS_ENGINE__ISP, - /* ... */ -}; - /* * enum cgs_ucode_id - Firmware types for different IPs */ @@ -76,17 +62,6 @@ enum cgs_ucode_id { CGS_UCODE_ID_MAXIMUM, }; -/* - * enum cgs_resource_type - GPU resource type - */ -enum cgs_resource_type { - CGS_RESOURCE_TYPE_MMIO = 0, - CGS_RESOURCE_TYPE_FB, - CGS_RESOURCE_TYPE_IO, - CGS_RESOURCE_TYPE_DOORBELL, - CGS_RESOURCE_TYPE_ROM, -}; - /** * struct cgs_firmware_info - Firmware information */ @@ -104,17 +79,6 @@ struct cgs_firmware_info { bool is_kicker; }; -struct cgs_mode_info { - uint32_t refresh_rate; - uint32_t vblank_time_us; -}; - -struct cgs_display_info { - uint32_t display_count; - uint32_t active_display_mask; - struct cgs_mode_info *mode_info; -}; - typedef unsigned long cgs_handle_t; /** @@ -170,119 +134,18 @@ typedef void (*cgs_write_ind_register_t)(struct cgs_device *cgs_device, enum cgs #define CGS_WREG32_FIELD_IND(device, space, reg, field, val) \ cgs_write_ind_register(device, space, ix##reg, (cgs_read_ind_register(device, space, ix##reg) & ~CGS_REG_FIELD_MASK(reg, field)) | (val) << CGS_REG_FIELD_SHIFT(reg, field)) -/** - * cgs_get_pci_resource() - provide access to a device resource (PCI BAR) - * @cgs_device: opaque device handle - * @resource_type: Type of Resource (MMIO, IO, ROM, FB, DOORBELL) - * @size: size of the region - * @offset: offset from the start of the region - * @resource_base: base address (not including offset) returned - * - * Return: 0 on success, -errno otherwise - */ -typedef int (*cgs_get_pci_resource_t)(struct cgs_device *cgs_device, - enum cgs_resource_type resource_type, - uint64_t size, - uint64_t offset, - uint64_t *resource_base); - -/** - * cgs_atom_get_data_table() - Get a pointer to an ATOM BIOS data table - * @cgs_device: opaque device handle - * @table: data table index - * @size: size of the table (output, may be NULL) - * @frev: table format revision (output, may be NULL) - * @crev: table content revision (output, may be NULL) - * - * Return: Pointer to start of the table, or NULL on failure - */ -typedef const void *(*cgs_atom_get_data_table_t)( - struct cgs_device *cgs_device, unsigned table, - uint16_t *size, uint8_t *frev, uint8_t *crev); - -/** - * cgs_atom_get_cmd_table_revs() - Get ATOM BIOS command table revisions - * @cgs_device: opaque device handle - * @table: data table index - * @frev: table format revision (output, may be NULL) - * @crev: table content revision (output, may be NULL) - * - * Return: 0 on success, -errno otherwise - */ -typedef int (*cgs_atom_get_cmd_table_revs_t)(struct cgs_device *cgs_device, unsigned table, - uint8_t *frev, uint8_t *crev); - -/** - * cgs_atom_exec_cmd_table() - Execute an ATOM BIOS command table - * @cgs_device: opaque device handle - * @table: command table index - * @args: arguments - * - * Return: 0 on success, -errno otherwise - */ -typedef int (*cgs_atom_exec_cmd_table_t)(struct cgs_device *cgs_device, - unsigned table, void *args); - -/** - * cgs_get_firmware_info - Get the firmware information from core driver - * @cgs_device: opaque device handle - * @type: the firmware type - * @info: returend firmware information - * - * Return: 0 on success, -errno otherwise - */ typedef int (*cgs_get_firmware_info)(struct cgs_device *cgs_device, enum cgs_ucode_id type, struct cgs_firmware_info *info); -typedef int (*cgs_rel_firmware)(struct cgs_device *cgs_device, - enum cgs_ucode_id type); - -typedef int(*cgs_set_powergating_state)(struct cgs_device *cgs_device, - enum amd_ip_block_type block_type, - enum amd_powergating_state state); - -typedef int(*cgs_set_clockgating_state)(struct cgs_device *cgs_device, - enum amd_ip_block_type block_type, - enum amd_clockgating_state state); - -typedef int(*cgs_get_active_displays_info)( - struct cgs_device *cgs_device, - struct cgs_display_info *info); - -typedef int (*cgs_notify_dpm_enabled)(struct cgs_device *cgs_device, bool enabled); - -typedef int (*cgs_is_virtualization_enabled_t)(void *cgs_device); - -typedef int (*cgs_enter_safe_mode)(struct cgs_device *cgs_device, bool en); - -typedef void (*cgs_lock_grbm_idx)(struct cgs_device *cgs_device, bool lock); - struct cgs_ops { /* MMIO access */ cgs_read_register_t read_register; cgs_write_register_t write_register; cgs_read_ind_register_t read_ind_register; cgs_write_ind_register_t write_ind_register; - /* PCI resources */ - cgs_get_pci_resource_t get_pci_resource; - /* ATOM BIOS */ - cgs_atom_get_data_table_t atom_get_data_table; - cgs_atom_get_cmd_table_revs_t atom_get_cmd_table_revs; - cgs_atom_exec_cmd_table_t atom_exec_cmd_table; /* Firmware Info */ cgs_get_firmware_info get_firmware_info; - cgs_rel_firmware rel_firmware; - /* cg pg interface*/ - cgs_set_powergating_state set_powergating_state; - cgs_set_clockgating_state set_clockgating_state; - /* display manager */ - cgs_get_active_displays_info get_active_displays_info; - /* notify dpm enabled */ - cgs_notify_dpm_enabled notify_dpm_enabled; - cgs_is_virtualization_enabled_t is_virtualization_enabled; - cgs_enter_safe_mode enter_safe_mode; - cgs_lock_grbm_idx lock_grbm_idx; }; struct cgs_os_ops; /* To be define in OS-specific CGS header */ @@ -309,40 +172,7 @@ struct cgs_device #define cgs_write_ind_register(dev,space,index,value) \ CGS_CALL(write_ind_register,dev,space,index,value) -#define cgs_atom_get_data_table(dev,table,size,frev,crev) \ - CGS_CALL(atom_get_data_table,dev,table,size,frev,crev) -#define cgs_atom_get_cmd_table_revs(dev,table,frev,crev) \ - CGS_CALL(atom_get_cmd_table_revs,dev,table,frev,crev) -#define cgs_atom_exec_cmd_table(dev,table,args) \ - CGS_CALL(atom_exec_cmd_table,dev,table,args) - #define cgs_get_firmware_info(dev, type, info) \ CGS_CALL(get_firmware_info, dev, type, info) -#define cgs_rel_firmware(dev, type) \ - CGS_CALL(rel_firmware, dev, type) -#define cgs_set_powergating_state(dev, block_type, state) \ - CGS_CALL(set_powergating_state, dev, block_type, state) -#define cgs_set_clockgating_state(dev, block_type, state) \ - CGS_CALL(set_clockgating_state, dev, block_type, state) -#define cgs_notify_dpm_enabled(dev, enabled) \ - CGS_CALL(notify_dpm_enabled, dev, enabled) - -#define cgs_get_active_displays_info(dev, info) \ - CGS_CALL(get_active_displays_info, dev, info) - -#define cgs_get_pci_resource(cgs_device, resource_type, size, offset, \ - resource_base) \ - CGS_CALL(get_pci_resource, cgs_device, resource_type, size, offset, \ - resource_base) - -#define cgs_is_virtualization_enabled(cgs_device) \ - CGS_CALL(is_virtualization_enabled, cgs_device) - -#define cgs_enter_safe_mode(cgs_device, en) \ - CGS_CALL(enter_safe_mode, cgs_device, en) - -#define cgs_lock_grbm_idx(cgs_device, lock) \ - CGS_CALL(lock_grbm_idx, cgs_device, lock) - #endif /* _CGS_COMMON_H */ diff --git a/drivers/gpu/drm/amd/include/kgd_kfd_interface.h b/drivers/gpu/drm/amd/include/kgd_kfd_interface.h index 237289a72bb7..5733fbee07f7 100644 --- a/drivers/gpu/drm/amd/include/kgd_kfd_interface.h +++ b/drivers/gpu/drm/amd/include/kgd_kfd_interface.h @@ -100,6 +100,21 @@ struct kgd2kfd_shared_resources { /* Bit n == 1 means Queue n is available for KFD */ DECLARE_BITMAP(queue_bitmap, KGD_MAX_QUEUES); + /* Doorbell assignments (SOC15 and later chips only). Only + * specific doorbells are routed to each SDMA engine. Others + * are routed to IH and VCN. They are not usable by the CP. + * + * Any doorbell number D that satisfies the following condition + * is reserved: (D & reserved_doorbell_mask) == reserved_doorbell_val + * + * KFD currently uses 1024 (= 0x3ff) doorbells per process. If + * doorbells 0x0f0-0x0f7 and 0x2f-0x2f7 are reserved, that means + * mask would be set to 0x1f8 and val set to 0x0f0. + */ + unsigned int sdma_doorbell[2][2]; + unsigned int reserved_doorbell_mask; + unsigned int reserved_doorbell_val; + /* Base address of doorbell aperture. */ phys_addr_t doorbell_physical_address; @@ -173,8 +188,6 @@ struct tile_config { * @set_pasid_vmid_mapping: Exposes pasid/vmid pair to the H/W for no cp * scheduling mode. Only used for no cp scheduling mode. * - * @init_pipeline: Initialized the compute pipelines. - * * @hqd_load: Loads the mqd structure to a H/W hqd slot. used only for no cp * sceduling mode. * @@ -274,9 +287,6 @@ struct kfd2kgd_calls { int (*set_pasid_vmid_mapping)(struct kgd_dev *kgd, unsigned int pasid, unsigned int vmid); - int (*init_pipeline)(struct kgd_dev *kgd, uint32_t pipe_id, - uint32_t hpd_size, uint64_t hpd_gpu_addr); - int (*init_interrupts)(struct kgd_dev *kgd, uint32_t pipe_id); int (*hqd_load)(struct kgd_dev *kgd, void *mqd, uint32_t pipe_id, @@ -382,6 +392,10 @@ struct kfd2kgd_calls { * * @resume: Notifies amdkfd about a resume action done to a kgd device * + * @quiesce_mm: Quiesce all user queue access to specified MM address space + * + * @resume_mm: Resume user queue access to specified MM address space + * * @schedule_evict_and_restore_process: Schedules work queue that will prepare * for safe eviction of KFD BOs that belong to the specified process. * @@ -399,6 +413,8 @@ struct kgd2kfd_calls { void (*interrupt)(struct kfd_dev *kfd, const void *ih_ring_entry); void (*suspend)(struct kfd_dev *kfd); int (*resume)(struct kfd_dev *kfd); + int (*quiesce_mm)(struct mm_struct *mm); + int (*resume_mm)(struct mm_struct *mm); int (*schedule_evict_and_restore_process)(struct mm_struct *mm, struct dma_fence *fence); }; diff --git a/drivers/gpu/drm/amd/include/kgd_pp_interface.h b/drivers/gpu/drm/amd/include/kgd_pp_interface.h index 5c840c022b52..06f08f34a110 100644 --- a/drivers/gpu/drm/amd/include/kgd_pp_interface.h +++ b/drivers/gpu/drm/amd/include/kgd_pp_interface.h @@ -94,6 +94,7 @@ enum pp_clock_type { PP_PCIE, OD_SCLK, OD_MCLK, + OD_RANGE, }; enum amd_pp_sensors { @@ -149,13 +150,6 @@ struct pp_states_info { uint32_t states[16]; }; -struct pp_gpu_power { - uint32_t vddc_power; - uint32_t vddci_power; - uint32_t max_gpu_power; - uint32_t average_gpu_power; -}; - #define PP_GROUP_MASK 0xF0000000 #define PP_GROUP_SHIFT 28 @@ -246,11 +240,6 @@ struct amd_pm_funcs { int (*load_firmware)(void *handle); int (*wait_for_fw_loading_complete)(void *handle); int (*set_clockgating_by_smu)(void *handle, uint32_t msg_id); - int (*notify_smu_memory_info)(void *handle, uint32_t virtual_addr_low, - uint32_t virtual_addr_hi, - uint32_t mc_addr_low, - uint32_t mc_addr_hi, - uint32_t size); int (*set_power_limit)(void *handle, uint32_t n); int (*get_power_limit)(void *handle, uint32_t *limit, bool default_limit); /* export to DC */ diff --git a/drivers/gpu/drm/amd/include/soc15_ih_clientid.h b/drivers/gpu/drm/amd/include/soc15_ih_clientid.h index a12d4f27cfa4..12e196c15bbe 100644 --- a/drivers/gpu/drm/amd/include/soc15_ih_clientid.h +++ b/drivers/gpu/drm/amd/include/soc15_ih_clientid.h @@ -43,6 +43,7 @@ enum soc15_ih_clientid { SOC15_IH_CLIENTID_SE2SH = 0x0c, SOC15_IH_CLIENTID_SE3SH = 0x0d, SOC15_IH_CLIENTID_SYSHUB = 0x0e, + SOC15_IH_CLIENTID_UVD1 = 0x0e, SOC15_IH_CLIENTID_THM = 0x0f, SOC15_IH_CLIENTID_UVD = 0x10, SOC15_IH_CLIENTID_VCE0 = 0x11, diff --git a/drivers/gpu/drm/amd/include/v9_structs.h b/drivers/gpu/drm/amd/include/v9_structs.h index 2fb25abaf7c8..ceaf4932258d 100644 --- a/drivers/gpu/drm/amd/include/v9_structs.h +++ b/drivers/gpu/drm/amd/include/v9_structs.h @@ -29,10 +29,10 @@ struct v9_sdma_mqd { uint32_t sdmax_rlcx_rb_base; uint32_t sdmax_rlcx_rb_base_hi; uint32_t sdmax_rlcx_rb_rptr; + uint32_t sdmax_rlcx_rb_rptr_hi; uint32_t sdmax_rlcx_rb_wptr; + uint32_t sdmax_rlcx_rb_wptr_hi; uint32_t sdmax_rlcx_rb_wptr_poll_cntl; - uint32_t sdmax_rlcx_rb_wptr_poll_addr_hi; - uint32_t sdmax_rlcx_rb_wptr_poll_addr_lo; uint32_t sdmax_rlcx_rb_rptr_addr_hi; uint32_t sdmax_rlcx_rb_rptr_addr_lo; uint32_t sdmax_rlcx_ib_cntl; @@ -44,29 +44,29 @@ struct v9_sdma_mqd { uint32_t sdmax_rlcx_skip_cntl; uint32_t sdmax_rlcx_context_status; uint32_t sdmax_rlcx_doorbell; - uint32_t sdmax_rlcx_virtual_addr; - uint32_t sdmax_rlcx_ape1_cntl; + uint32_t sdmax_rlcx_status; uint32_t sdmax_rlcx_doorbell_log; - uint32_t reserved_22; - uint32_t reserved_23; - uint32_t reserved_24; - uint32_t reserved_25; - uint32_t reserved_26; - uint32_t reserved_27; - uint32_t reserved_28; - uint32_t reserved_29; - uint32_t reserved_30; - uint32_t reserved_31; - uint32_t reserved_32; - uint32_t reserved_33; - uint32_t reserved_34; - uint32_t reserved_35; - uint32_t reserved_36; - uint32_t reserved_37; - uint32_t reserved_38; - uint32_t reserved_39; - uint32_t reserved_40; - uint32_t reserved_41; + uint32_t sdmax_rlcx_watermark; + uint32_t sdmax_rlcx_doorbell_offset; + uint32_t sdmax_rlcx_csa_addr_lo; + uint32_t sdmax_rlcx_csa_addr_hi; + uint32_t sdmax_rlcx_ib_sub_remain; + uint32_t sdmax_rlcx_preempt; + uint32_t sdmax_rlcx_dummy_reg; + uint32_t sdmax_rlcx_rb_wptr_poll_addr_hi; + uint32_t sdmax_rlcx_rb_wptr_poll_addr_lo; + uint32_t sdmax_rlcx_rb_aql_cntl; + uint32_t sdmax_rlcx_minor_ptr_update; + uint32_t sdmax_rlcx_midcmd_data0; + uint32_t sdmax_rlcx_midcmd_data1; + uint32_t sdmax_rlcx_midcmd_data2; + uint32_t sdmax_rlcx_midcmd_data3; + uint32_t sdmax_rlcx_midcmd_data4; + uint32_t sdmax_rlcx_midcmd_data5; + uint32_t sdmax_rlcx_midcmd_data6; + uint32_t sdmax_rlcx_midcmd_data7; + uint32_t sdmax_rlcx_midcmd_data8; + uint32_t sdmax_rlcx_midcmd_cntl; uint32_t reserved_42; uint32_t reserved_43; uint32_t reserved_44; diff --git a/drivers/gpu/drm/amd/include/vega20_ip_offset.h b/drivers/gpu/drm/amd/include/vega20_ip_offset.h new file mode 100644 index 000000000000..2a2a9cc8bedb --- /dev/null +++ b/drivers/gpu/drm/amd/include/vega20_ip_offset.h @@ -0,0 +1,1051 @@ +/* + * Copyright (C) 2018 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included + * in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS + * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +#ifndef _vega20_ip_offset_HEADER +#define _vega20_ip_offset_HEADER + +#define MAX_INSTANCE 6 +#define MAX_SEGMENT 6 + + +struct IP_BASE_INSTANCE +{ + unsigned int segment[MAX_SEGMENT]; +}; + +struct IP_BASE +{ + struct IP_BASE_INSTANCE instance[MAX_INSTANCE]; +}; + + +static const struct IP_BASE ATHUB_BASE ={ { { { 0x00000C20, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE CLK_BASE ={ { { { 0x00016C00, 0x00016E00, 0x00017000, 0x00017200, 0x0001B000, 0x0001B200 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE DCE_BASE ={ { { { 0x00000012, 0x000000C0, 0x000034C0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE DF_BASE ={ { { { 0x00007000, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE FUSE_BASE ={ { { { 0x00017400, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE GC_BASE ={ { { { 0x00002000, 0x0000A000, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE HDP_BASE ={ { { { 0x00000F20, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE MMHUB_BASE ={ { { { 0x0001A000, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE MP0_BASE ={ { { { 0x00016000, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE MP1_BASE ={ { { { 0x00016000, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE NBIO_BASE ={ { { { 0x00000000, 0x00000014, 0x00000D20, 0x00010400, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE OSSSYS_BASE ={ { { { 0x000010A0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE SDMA0_BASE ={ { { { 0x00001260, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE SDMA1_BASE ={ { { { 0x00001860, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE SMUIO_BASE ={ { { { 0x00016800, 0x00016A00, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE THM_BASE ={ { { { 0x00016600, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE UMC_BASE ={ { { { 0x00014000, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE UVD_BASE ={ { { { 0x00007800, 0x00007E00, 0, 0, 0, 0 } }, + { { 0, 0x00009000, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +/* Adjust VCE_BASE to make vce_4_1 use vce_4_0 offset header files*/ +static const struct IP_BASE VCE_BASE ={ { { { 0x00007E00/* 0x00008800 */, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE XDMA_BASE ={ { { { 0x00003400, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; +static const struct IP_BASE RSMU_BASE ={ { { { 0x00012000, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } }, + { { 0, 0, 0, 0, 0, 0 } } } }; + + +#define ATHUB_BASE__INST0_SEG0 0x00000C20 +#define ATHUB_BASE__INST0_SEG1 0 +#define ATHUB_BASE__INST0_SEG2 0 +#define ATHUB_BASE__INST0_SEG3 0 +#define ATHUB_BASE__INST0_SEG4 0 +#define ATHUB_BASE__INST0_SEG5 0 + +#define ATHUB_BASE__INST1_SEG0 0 +#define ATHUB_BASE__INST1_SEG1 0 +#define ATHUB_BASE__INST1_SEG2 0 +#define ATHUB_BASE__INST1_SEG3 0 +#define ATHUB_BASE__INST1_SEG4 0 +#define ATHUB_BASE__INST1_SEG5 0 + +#define ATHUB_BASE__INST2_SEG0 0 +#define ATHUB_BASE__INST2_SEG1 0 +#define ATHUB_BASE__INST2_SEG2 0 +#define ATHUB_BASE__INST2_SEG3 0 +#define ATHUB_BASE__INST2_SEG4 0 +#define ATHUB_BASE__INST2_SEG5 0 + +#define ATHUB_BASE__INST3_SEG0 0 +#define ATHUB_BASE__INST3_SEG1 0 +#define ATHUB_BASE__INST3_SEG2 0 +#define ATHUB_BASE__INST3_SEG3 0 +#define ATHUB_BASE__INST3_SEG4 0 +#define ATHUB_BASE__INST3_SEG5 0 + +#define ATHUB_BASE__INST4_SEG0 0 +#define ATHUB_BASE__INST4_SEG1 0 +#define ATHUB_BASE__INST4_SEG2 0 +#define ATHUB_BASE__INST4_SEG3 0 +#define ATHUB_BASE__INST4_SEG4 0 +#define ATHUB_BASE__INST4_SEG5 0 + +#define ATHUB_BASE__INST5_SEG0 0 +#define ATHUB_BASE__INST5_SEG1 0 +#define ATHUB_BASE__INST5_SEG2 0 +#define ATHUB_BASE__INST5_SEG3 0 +#define ATHUB_BASE__INST5_SEG4 0 +#define ATHUB_BASE__INST5_SEG5 0 + +#define CLK_BASE__INST0_SEG0 0x00016C00 +#define CLK_BASE__INST0_SEG1 0x00016E00 +#define CLK_BASE__INST0_SEG2 0x00017000 +#define CLK_BASE__INST0_SEG3 0x00017200 +#define CLK_BASE__INST0_SEG4 0x0001B000 +#define CLK_BASE__INST0_SEG5 0x0001B200 + +#define CLK_BASE__INST1_SEG0 0 +#define CLK_BASE__INST1_SEG1 0 +#define CLK_BASE__INST1_SEG2 0 +#define CLK_BASE__INST1_SEG3 0 +#define CLK_BASE__INST1_SEG4 0 +#define CLK_BASE__INST1_SEG5 0 + +#define CLK_BASE__INST2_SEG0 0 +#define CLK_BASE__INST2_SEG1 0 +#define CLK_BASE__INST2_SEG2 0 +#define CLK_BASE__INST2_SEG3 0 +#define CLK_BASE__INST2_SEG4 0 +#define CLK_BASE__INST2_SEG5 0 + +#define CLK_BASE__INST3_SEG0 0 +#define CLK_BASE__INST3_SEG1 0 +#define CLK_BASE__INST3_SEG2 0 +#define CLK_BASE__INST3_SEG3 0 +#define CLK_BASE__INST3_SEG4 0 +#define CLK_BASE__INST3_SEG5 0 + +#define CLK_BASE__INST4_SEG0 0 +#define CLK_BASE__INST4_SEG1 0 +#define CLK_BASE__INST4_SEG2 0 +#define CLK_BASE__INST4_SEG3 0 +#define CLK_BASE__INST4_SEG4 0 +#define CLK_BASE__INST4_SEG5 0 + +#define CLK_BASE__INST5_SEG0 0 +#define CLK_BASE__INST5_SEG1 0 +#define CLK_BASE__INST5_SEG2 0 +#define CLK_BASE__INST5_SEG3 0 +#define CLK_BASE__INST5_SEG4 0 +#define CLK_BASE__INST5_SEG5 0 + +#define DCE_BASE__INST0_SEG0 0x00000012 +#define DCE_BASE__INST0_SEG1 0x000000C0 +#define DCE_BASE__INST0_SEG2 0x000034C0 +#define DCE_BASE__INST0_SEG3 0 +#define DCE_BASE__INST0_SEG4 0 +#define DCE_BASE__INST0_SEG5 0 + +#define DCE_BASE__INST1_SEG0 0 +#define DCE_BASE__INST1_SEG1 0 +#define DCE_BASE__INST1_SEG2 0 +#define DCE_BASE__INST1_SEG3 0 +#define DCE_BASE__INST1_SEG4 0 +#define DCE_BASE__INST1_SEG5 0 + +#define DCE_BASE__INST2_SEG0 0 +#define DCE_BASE__INST2_SEG1 0 +#define DCE_BASE__INST2_SEG2 0 +#define DCE_BASE__INST2_SEG3 0 +#define DCE_BASE__INST2_SEG4 0 +#define DCE_BASE__INST2_SEG5 0 + +#define DCE_BASE__INST3_SEG0 0 +#define DCE_BASE__INST3_SEG1 0 +#define DCE_BASE__INST3_SEG2 0 +#define DCE_BASE__INST3_SEG3 0 +#define DCE_BASE__INST3_SEG4 0 +#define DCE_BASE__INST3_SEG5 0 + +#define DCE_BASE__INST4_SEG0 0 +#define DCE_BASE__INST4_SEG1 0 +#define DCE_BASE__INST4_SEG2 0 +#define DCE_BASE__INST4_SEG3 0 +#define DCE_BASE__INST4_SEG4 0 +#define DCE_BASE__INST4_SEG5 0 + +#define DCE_BASE__INST5_SEG0 0 +#define DCE_BASE__INST5_SEG1 0 +#define DCE_BASE__INST5_SEG2 0 +#define DCE_BASE__INST5_SEG3 0 +#define DCE_BASE__INST5_SEG4 0 +#define DCE_BASE__INST5_SEG5 0 + +#define DF_BASE__INST0_SEG0 0x00007000 +#define DF_BASE__INST0_SEG1 0 +#define DF_BASE__INST0_SEG2 0 +#define DF_BASE__INST0_SEG3 0 +#define DF_BASE__INST0_SEG4 0 +#define DF_BASE__INST0_SEG5 0 + +#define DF_BASE__INST1_SEG0 0 +#define DF_BASE__INST1_SEG1 0 +#define DF_BASE__INST1_SEG2 0 +#define DF_BASE__INST1_SEG3 0 +#define DF_BASE__INST1_SEG4 0 +#define DF_BASE__INST1_SEG5 0 + +#define DF_BASE__INST2_SEG0 0 +#define DF_BASE__INST2_SEG1 0 +#define DF_BASE__INST2_SEG2 0 +#define DF_BASE__INST2_SEG3 0 +#define DF_BASE__INST2_SEG4 0 +#define DF_BASE__INST2_SEG5 0 + +#define DF_BASE__INST3_SEG0 0 +#define DF_BASE__INST3_SEG1 0 +#define DF_BASE__INST3_SEG2 0 +#define DF_BASE__INST3_SEG3 0 +#define DF_BASE__INST3_SEG4 0 +#define DF_BASE__INST3_SEG5 0 + +#define DF_BASE__INST4_SEG0 0 +#define DF_BASE__INST4_SEG1 0 +#define DF_BASE__INST4_SEG2 0 +#define DF_BASE__INST4_SEG3 0 +#define DF_BASE__INST4_SEG4 0 +#define DF_BASE__INST4_SEG5 0 + +#define DF_BASE__INST5_SEG0 0 +#define DF_BASE__INST5_SEG1 0 +#define DF_BASE__INST5_SEG2 0 +#define DF_BASE__INST5_SEG3 0 +#define DF_BASE__INST5_SEG4 0 +#define DF_BASE__INST5_SEG5 0 + +#define FUSE_BASE__INST0_SEG0 0x00017400 +#define FUSE_BASE__INST0_SEG1 0 +#define FUSE_BASE__INST0_SEG2 0 +#define FUSE_BASE__INST0_SEG3 0 +#define FUSE_BASE__INST0_SEG4 0 +#define FUSE_BASE__INST0_SEG5 0 + +#define FUSE_BASE__INST1_SEG0 0 +#define FUSE_BASE__INST1_SEG1 0 +#define FUSE_BASE__INST1_SEG2 0 +#define FUSE_BASE__INST1_SEG3 0 +#define FUSE_BASE__INST1_SEG4 0 +#define FUSE_BASE__INST1_SEG5 0 + +#define FUSE_BASE__INST2_SEG0 0 +#define FUSE_BASE__INST2_SEG1 0 +#define FUSE_BASE__INST2_SEG2 0 +#define FUSE_BASE__INST2_SEG3 0 +#define FUSE_BASE__INST2_SEG4 0 +#define FUSE_BASE__INST2_SEG5 0 + +#define FUSE_BASE__INST3_SEG0 0 +#define FUSE_BASE__INST3_SEG1 0 +#define FUSE_BASE__INST3_SEG2 0 +#define FUSE_BASE__INST3_SEG3 0 +#define FUSE_BASE__INST3_SEG4 0 +#define FUSE_BASE__INST3_SEG5 0 + +#define FUSE_BASE__INST4_SEG0 0 +#define FUSE_BASE__INST4_SEG1 0 +#define FUSE_BASE__INST4_SEG2 0 +#define FUSE_BASE__INST4_SEG3 0 +#define FUSE_BASE__INST4_SEG4 0 +#define FUSE_BASE__INST4_SEG5 0 + +#define FUSE_BASE__INST5_SEG0 0 +#define FUSE_BASE__INST5_SEG1 0 +#define FUSE_BASE__INST5_SEG2 0 +#define FUSE_BASE__INST5_SEG3 0 +#define FUSE_BASE__INST5_SEG4 0 +#define FUSE_BASE__INST5_SEG5 0 + +#define GC_BASE__INST0_SEG0 0x00002000 +#define GC_BASE__INST0_SEG1 0x0000A000 +#define GC_BASE__INST0_SEG2 0 +#define GC_BASE__INST0_SEG3 0 +#define GC_BASE__INST0_SEG4 0 +#define GC_BASE__INST0_SEG5 0 + +#define GC_BASE__INST1_SEG0 0 +#define GC_BASE__INST1_SEG1 0 +#define GC_BASE__INST1_SEG2 0 +#define GC_BASE__INST1_SEG3 0 +#define GC_BASE__INST1_SEG4 0 +#define GC_BASE__INST1_SEG5 0 + +#define GC_BASE__INST2_SEG0 0 +#define GC_BASE__INST2_SEG1 0 +#define GC_BASE__INST2_SEG2 0 +#define GC_BASE__INST2_SEG3 0 +#define GC_BASE__INST2_SEG4 0 +#define GC_BASE__INST2_SEG5 0 + +#define GC_BASE__INST3_SEG0 0 +#define GC_BASE__INST3_SEG1 0 +#define GC_BASE__INST3_SEG2 0 +#define GC_BASE__INST3_SEG3 0 +#define GC_BASE__INST3_SEG4 0 +#define GC_BASE__INST3_SEG5 0 + +#define GC_BASE__INST4_SEG0 0 +#define GC_BASE__INST4_SEG1 0 +#define GC_BASE__INST4_SEG2 0 +#define GC_BASE__INST4_SEG3 0 +#define GC_BASE__INST4_SEG4 0 +#define GC_BASE__INST4_SEG5 0 + +#define GC_BASE__INST5_SEG0 0 +#define GC_BASE__INST5_SEG1 0 +#define GC_BASE__INST5_SEG2 0 +#define GC_BASE__INST5_SEG3 0 +#define GC_BASE__INST5_SEG4 0 +#define GC_BASE__INST5_SEG5 0 + +#define HDP_BASE__INST0_SEG0 0x00000F20 +#define HDP_BASE__INST0_SEG1 0 +#define HDP_BASE__INST0_SEG2 0 +#define HDP_BASE__INST0_SEG3 0 +#define HDP_BASE__INST0_SEG4 0 +#define HDP_BASE__INST0_SEG5 0 + +#define HDP_BASE__INST1_SEG0 0 +#define HDP_BASE__INST1_SEG1 0 +#define HDP_BASE__INST1_SEG2 0 +#define HDP_BASE__INST1_SEG3 0 +#define HDP_BASE__INST1_SEG4 0 +#define HDP_BASE__INST1_SEG5 0 + +#define HDP_BASE__INST2_SEG0 0 +#define HDP_BASE__INST2_SEG1 0 +#define HDP_BASE__INST2_SEG2 0 +#define HDP_BASE__INST2_SEG3 0 +#define HDP_BASE__INST2_SEG4 0 +#define HDP_BASE__INST2_SEG5 0 + +#define HDP_BASE__INST3_SEG0 0 +#define HDP_BASE__INST3_SEG1 0 +#define HDP_BASE__INST3_SEG2 0 +#define HDP_BASE__INST3_SEG3 0 +#define HDP_BASE__INST3_SEG4 0 +#define HDP_BASE__INST3_SEG5 0 + +#define HDP_BASE__INST4_SEG0 0 +#define HDP_BASE__INST4_SEG1 0 +#define HDP_BASE__INST4_SEG2 0 +#define HDP_BASE__INST4_SEG3 0 +#define HDP_BASE__INST4_SEG4 0 +#define HDP_BASE__INST4_SEG5 0 + +#define HDP_BASE__INST5_SEG0 0 +#define HDP_BASE__INST5_SEG1 0 +#define HDP_BASE__INST5_SEG2 0 +#define HDP_BASE__INST5_SEG3 0 +#define HDP_BASE__INST5_SEG4 0 +#define HDP_BASE__INST5_SEG5 0 + +#define MMHUB_BASE__INST0_SEG0 0x0001A000 +#define MMHUB_BASE__INST0_SEG1 0 +#define MMHUB_BASE__INST0_SEG2 0 +#define MMHUB_BASE__INST0_SEG3 0 +#define MMHUB_BASE__INST0_SEG4 0 +#define MMHUB_BASE__INST0_SEG5 0 + +#define MMHUB_BASE__INST1_SEG0 0 +#define MMHUB_BASE__INST1_SEG1 0 +#define MMHUB_BASE__INST1_SEG2 0 +#define MMHUB_BASE__INST1_SEG3 0 +#define MMHUB_BASE__INST1_SEG4 0 +#define MMHUB_BASE__INST1_SEG5 0 + +#define MMHUB_BASE__INST2_SEG0 0 +#define MMHUB_BASE__INST2_SEG1 0 +#define MMHUB_BASE__INST2_SEG2 0 +#define MMHUB_BASE__INST2_SEG3 0 +#define MMHUB_BASE__INST2_SEG4 0 +#define MMHUB_BASE__INST2_SEG5 0 + +#define MMHUB_BASE__INST3_SEG0 0 +#define MMHUB_BASE__INST3_SEG1 0 +#define MMHUB_BASE__INST3_SEG2 0 +#define MMHUB_BASE__INST3_SEG3 0 +#define MMHUB_BASE__INST3_SEG4 0 +#define MMHUB_BASE__INST3_SEG5 0 + +#define MMHUB_BASE__INST4_SEG0 0 +#define MMHUB_BASE__INST4_SEG1 0 +#define MMHUB_BASE__INST4_SEG2 0 +#define MMHUB_BASE__INST4_SEG3 0 +#define MMHUB_BASE__INST4_SEG4 0 +#define MMHUB_BASE__INST4_SEG5 0 + +#define MMHUB_BASE__INST5_SEG0 0 +#define MMHUB_BASE__INST5_SEG1 0 +#define MMHUB_BASE__INST5_SEG2 0 +#define MMHUB_BASE__INST5_SEG3 0 +#define MMHUB_BASE__INST5_SEG4 0 +#define MMHUB_BASE__INST5_SEG5 0 + +#define MP0_BASE__INST0_SEG0 0x00016000 +#define MP0_BASE__INST0_SEG1 0 +#define MP0_BASE__INST0_SEG2 0 +#define MP0_BASE__INST0_SEG3 0 +#define MP0_BASE__INST0_SEG4 0 +#define MP0_BASE__INST0_SEG5 0 + +#define MP0_BASE__INST1_SEG0 0 +#define MP0_BASE__INST1_SEG1 0 +#define MP0_BASE__INST1_SEG2 0 +#define MP0_BASE__INST1_SEG3 0 +#define MP0_BASE__INST1_SEG4 0 +#define MP0_BASE__INST1_SEG5 0 + +#define MP0_BASE__INST2_SEG0 0 +#define MP0_BASE__INST2_SEG1 0 +#define MP0_BASE__INST2_SEG2 0 +#define MP0_BASE__INST2_SEG3 0 +#define MP0_BASE__INST2_SEG4 0 +#define MP0_BASE__INST2_SEG5 0 + +#define MP0_BASE__INST3_SEG0 0 +#define MP0_BASE__INST3_SEG1 0 +#define MP0_BASE__INST3_SEG2 0 +#define MP0_BASE__INST3_SEG3 0 +#define MP0_BASE__INST3_SEG4 0 +#define MP0_BASE__INST3_SEG5 0 + +#define MP0_BASE__INST4_SEG0 0 +#define MP0_BASE__INST4_SEG1 0 +#define MP0_BASE__INST4_SEG2 0 +#define MP0_BASE__INST4_SEG3 0 +#define MP0_BASE__INST4_SEG4 0 +#define MP0_BASE__INST4_SEG5 0 + +#define MP0_BASE__INST5_SEG0 0 +#define MP0_BASE__INST5_SEG1 0 +#define MP0_BASE__INST5_SEG2 0 +#define MP0_BASE__INST5_SEG3 0 +#define MP0_BASE__INST5_SEG4 0 +#define MP0_BASE__INST5_SEG5 0 + +#define MP1_BASE__INST0_SEG0 0x00016000 +#define MP1_BASE__INST0_SEG1 0 +#define MP1_BASE__INST0_SEG2 0 +#define MP1_BASE__INST0_SEG3 0 +#define MP1_BASE__INST0_SEG4 0 +#define MP1_BASE__INST0_SEG5 0 + +#define MP1_BASE__INST1_SEG0 0 +#define MP1_BASE__INST1_SEG1 0 +#define MP1_BASE__INST1_SEG2 0 +#define MP1_BASE__INST1_SEG3 0 +#define MP1_BASE__INST1_SEG4 0 +#define MP1_BASE__INST1_SEG5 0 + +#define MP1_BASE__INST2_SEG0 0 +#define MP1_BASE__INST2_SEG1 0 +#define MP1_BASE__INST2_SEG2 0 +#define MP1_BASE__INST2_SEG3 0 +#define MP1_BASE__INST2_SEG4 0 +#define MP1_BASE__INST2_SEG5 0 + +#define MP1_BASE__INST3_SEG0 0 +#define MP1_BASE__INST3_SEG1 0 +#define MP1_BASE__INST3_SEG2 0 +#define MP1_BASE__INST3_SEG3 0 +#define MP1_BASE__INST3_SEG4 0 +#define MP1_BASE__INST3_SEG5 0 + +#define MP1_BASE__INST4_SEG0 0 +#define MP1_BASE__INST4_SEG1 0 +#define MP1_BASE__INST4_SEG2 0 +#define MP1_BASE__INST4_SEG3 0 +#define MP1_BASE__INST4_SEG4 0 +#define MP1_BASE__INST4_SEG5 0 + +#define MP1_BASE__INST5_SEG0 0 +#define MP1_BASE__INST5_SEG1 0 +#define MP1_BASE__INST5_SEG2 0 +#define MP1_BASE__INST5_SEG3 0 +#define MP1_BASE__INST5_SEG4 0 +#define MP1_BASE__INST5_SEG5 0 + +#define NBIO_BASE__INST0_SEG0 0x00000000 +#define NBIO_BASE__INST0_SEG1 0x00000014 +#define NBIO_BASE__INST0_SEG2 0x00000D20 +#define NBIO_BASE__INST0_SEG3 0x00010400 +#define NBIO_BASE__INST0_SEG4 0 +#define NBIO_BASE__INST0_SEG5 0 + +#define NBIO_BASE__INST1_SEG0 0 +#define NBIO_BASE__INST1_SEG1 0 +#define NBIO_BASE__INST1_SEG2 0 +#define NBIO_BASE__INST1_SEG3 0 +#define NBIO_BASE__INST1_SEG4 0 +#define NBIO_BASE__INST1_SEG5 0 + +#define NBIO_BASE__INST2_SEG0 0 +#define NBIO_BASE__INST2_SEG1 0 +#define NBIO_BASE__INST2_SEG2 0 +#define NBIO_BASE__INST2_SEG3 0 +#define NBIO_BASE__INST2_SEG4 0 +#define NBIO_BASE__INST2_SEG5 0 + +#define NBIO_BASE__INST3_SEG0 0 +#define NBIO_BASE__INST3_SEG1 0 +#define NBIO_BASE__INST3_SEG2 0 +#define NBIO_BASE__INST3_SEG3 0 +#define NBIO_BASE__INST3_SEG4 0 +#define NBIO_BASE__INST3_SEG5 0 + +#define NBIO_BASE__INST4_SEG0 0 +#define NBIO_BASE__INST4_SEG1 0 +#define NBIO_BASE__INST4_SEG2 0 +#define NBIO_BASE__INST4_SEG3 0 +#define NBIO_BASE__INST4_SEG4 0 +#define NBIO_BASE__INST4_SEG5 0 + +#define NBIO_BASE__INST5_SEG0 0 +#define NBIO_BASE__INST5_SEG1 0 +#define NBIO_BASE__INST5_SEG2 0 +#define NBIO_BASE__INST5_SEG3 0 +#define NBIO_BASE__INST5_SEG4 0 +#define NBIO_BASE__INST5_SEG5 0 + +#define OSSSYS_BASE__INST0_SEG0 0x000010A0 +#define OSSSYS_BASE__INST0_SEG1 0 +#define OSSSYS_BASE__INST0_SEG2 0 +#define OSSSYS_BASE__INST0_SEG3 0 +#define OSSSYS_BASE__INST0_SEG4 0 +#define OSSSYS_BASE__INST0_SEG5 0 + +#define OSSSYS_BASE__INST1_SEG0 0 +#define OSSSYS_BASE__INST1_SEG1 0 +#define OSSSYS_BASE__INST1_SEG2 0 +#define OSSSYS_BASE__INST1_SEG3 0 +#define OSSSYS_BASE__INST1_SEG4 0 +#define OSSSYS_BASE__INST1_SEG5 0 + +#define OSSSYS_BASE__INST2_SEG0 0 +#define OSSSYS_BASE__INST2_SEG1 0 +#define OSSSYS_BASE__INST2_SEG2 0 +#define OSSSYS_BASE__INST2_SEG3 0 +#define OSSSYS_BASE__INST2_SEG4 0 +#define OSSSYS_BASE__INST2_SEG5 0 + +#define OSSSYS_BASE__INST3_SEG0 0 +#define OSSSYS_BASE__INST3_SEG1 0 +#define OSSSYS_BASE__INST3_SEG2 0 +#define OSSSYS_BASE__INST3_SEG3 0 +#define OSSSYS_BASE__INST3_SEG4 0 +#define OSSSYS_BASE__INST3_SEG5 0 + +#define OSSSYS_BASE__INST4_SEG0 0 +#define OSSSYS_BASE__INST4_SEG1 0 +#define OSSSYS_BASE__INST4_SEG2 0 +#define OSSSYS_BASE__INST4_SEG3 0 +#define OSSSYS_BASE__INST4_SEG4 0 +#define OSSSYS_BASE__INST4_SEG5 0 + +#define OSSSYS_BASE__INST5_SEG0 0 +#define OSSSYS_BASE__INST5_SEG1 0 +#define OSSSYS_BASE__INST5_SEG2 0 +#define OSSSYS_BASE__INST5_SEG3 0 +#define OSSSYS_BASE__INST5_SEG4 0 +#define OSSSYS_BASE__INST5_SEG5 0 + +#define SDMA0_BASE__INST0_SEG0 0x00001260 +#define SDMA0_BASE__INST0_SEG1 0 +#define SDMA0_BASE__INST0_SEG2 0 +#define SDMA0_BASE__INST0_SEG3 0 +#define SDMA0_BASE__INST0_SEG4 0 +#define SDMA0_BASE__INST0_SEG5 0 + +#define SDMA0_BASE__INST1_SEG0 0 +#define SDMA0_BASE__INST1_SEG1 0 +#define SDMA0_BASE__INST1_SEG2 0 +#define SDMA0_BASE__INST1_SEG3 0 +#define SDMA0_BASE__INST1_SEG4 0 +#define SDMA0_BASE__INST1_SEG5 0 + +#define SDMA0_BASE__INST2_SEG0 0 +#define SDMA0_BASE__INST2_SEG1 0 +#define SDMA0_BASE__INST2_SEG2 0 +#define SDMA0_BASE__INST2_SEG3 0 +#define SDMA0_BASE__INST2_SEG4 0 +#define SDMA0_BASE__INST2_SEG5 0 + +#define SDMA0_BASE__INST3_SEG0 0 +#define SDMA0_BASE__INST3_SEG1 0 +#define SDMA0_BASE__INST3_SEG2 0 +#define SDMA0_BASE__INST3_SEG3 0 +#define SDMA0_BASE__INST3_SEG4 0 +#define SDMA0_BASE__INST3_SEG5 0 + +#define SDMA0_BASE__INST4_SEG0 0 +#define SDMA0_BASE__INST4_SEG1 0 +#define SDMA0_BASE__INST4_SEG2 0 +#define SDMA0_BASE__INST4_SEG3 0 +#define SDMA0_BASE__INST4_SEG4 0 +#define SDMA0_BASE__INST4_SEG5 0 + +#define SDMA0_BASE__INST5_SEG0 0 +#define SDMA0_BASE__INST5_SEG1 0 +#define SDMA0_BASE__INST5_SEG2 0 +#define SDMA0_BASE__INST5_SEG3 0 +#define SDMA0_BASE__INST5_SEG4 0 +#define SDMA0_BASE__INST5_SEG5 0 + +#define SDMA1_BASE__INST0_SEG0 0x00001860 +#define SDMA1_BASE__INST0_SEG1 0 +#define SDMA1_BASE__INST0_SEG2 0 +#define SDMA1_BASE__INST0_SEG3 0 +#define SDMA1_BASE__INST0_SEG4 0 +#define SDMA1_BASE__INST0_SEG5 0 + +#define SDMA1_BASE__INST1_SEG0 0 +#define SDMA1_BASE__INST1_SEG1 0 +#define SDMA1_BASE__INST1_SEG2 0 +#define SDMA1_BASE__INST1_SEG3 0 +#define SDMA1_BASE__INST1_SEG4 0 +#define SDMA1_BASE__INST1_SEG5 0 + +#define SDMA1_BASE__INST2_SEG0 0 +#define SDMA1_BASE__INST2_SEG1 0 +#define SDMA1_BASE__INST2_SEG2 0 +#define SDMA1_BASE__INST2_SEG3 0 +#define SDMA1_BASE__INST2_SEG4 0 +#define SDMA1_BASE__INST2_SEG5 0 + +#define SDMA1_BASE__INST3_SEG0 0 +#define SDMA1_BASE__INST3_SEG1 0 +#define SDMA1_BASE__INST3_SEG2 0 +#define SDMA1_BASE__INST3_SEG3 0 +#define SDMA1_BASE__INST3_SEG4 0 +#define SDMA1_BASE__INST3_SEG5 0 + +#define SDMA1_BASE__INST4_SEG0 0 +#define SDMA1_BASE__INST4_SEG1 0 +#define SDMA1_BASE__INST4_SEG2 0 +#define SDMA1_BASE__INST4_SEG3 0 +#define SDMA1_BASE__INST4_SEG4 0 +#define SDMA1_BASE__INST4_SEG5 0 + +#define SDMA1_BASE__INST5_SEG0 0 +#define SDMA1_BASE__INST5_SEG1 0 +#define SDMA1_BASE__INST5_SEG2 0 +#define SDMA1_BASE__INST5_SEG3 0 +#define SDMA1_BASE__INST5_SEG4 0 +#define SDMA1_BASE__INST5_SEG5 0 + +#define SMUIO_BASE__INST0_SEG0 0x00016800 +#define SMUIO_BASE__INST0_SEG1 0x00016A00 +#define SMUIO_BASE__INST0_SEG2 0 +#define SMUIO_BASE__INST0_SEG3 0 +#define SMUIO_BASE__INST0_SEG4 0 +#define SMUIO_BASE__INST0_SEG5 0 + +#define SMUIO_BASE__INST1_SEG0 0 +#define SMUIO_BASE__INST1_SEG1 0 +#define SMUIO_BASE__INST1_SEG2 0 +#define SMUIO_BASE__INST1_SEG3 0 +#define SMUIO_BASE__INST1_SEG4 0 +#define SMUIO_BASE__INST1_SEG5 0 + +#define SMUIO_BASE__INST2_SEG0 0 +#define SMUIO_BASE__INST2_SEG1 0 +#define SMUIO_BASE__INST2_SEG2 0 +#define SMUIO_BASE__INST2_SEG3 0 +#define SMUIO_BASE__INST2_SEG4 0 +#define SMUIO_BASE__INST2_SEG5 0 + +#define SMUIO_BASE__INST3_SEG0 0 +#define SMUIO_BASE__INST3_SEG1 0 +#define SMUIO_BASE__INST3_SEG2 0 +#define SMUIO_BASE__INST3_SEG3 0 +#define SMUIO_BASE__INST3_SEG4 0 +#define SMUIO_BASE__INST3_SEG5 0 + +#define SMUIO_BASE__INST4_SEG0 0 +#define SMUIO_BASE__INST4_SEG1 0 +#define SMUIO_BASE__INST4_SEG2 0 +#define SMUIO_BASE__INST4_SEG3 0 +#define SMUIO_BASE__INST4_SEG4 0 +#define SMUIO_BASE__INST4_SEG5 0 + +#define SMUIO_BASE__INST5_SEG0 0 +#define SMUIO_BASE__INST5_SEG1 0 +#define SMUIO_BASE__INST5_SEG2 0 +#define SMUIO_BASE__INST5_SEG3 0 +#define SMUIO_BASE__INST5_SEG4 0 +#define SMUIO_BASE__INST5_SEG5 0 + +#define THM_BASE__INST0_SEG0 0x00016600 +#define THM_BASE__INST0_SEG1 0 +#define THM_BASE__INST0_SEG2 0 +#define THM_BASE__INST0_SEG3 0 +#define THM_BASE__INST0_SEG4 0 +#define THM_BASE__INST0_SEG5 0 + +#define THM_BASE__INST1_SEG0 0 +#define THM_BASE__INST1_SEG1 0 +#define THM_BASE__INST1_SEG2 0 +#define THM_BASE__INST1_SEG3 0 +#define THM_BASE__INST1_SEG4 0 +#define THM_BASE__INST1_SEG5 0 + +#define THM_BASE__INST2_SEG0 0 +#define THM_BASE__INST2_SEG1 0 +#define THM_BASE__INST2_SEG2 0 +#define THM_BASE__INST2_SEG3 0 +#define THM_BASE__INST2_SEG4 0 +#define THM_BASE__INST2_SEG5 0 + +#define THM_BASE__INST3_SEG0 0 +#define THM_BASE__INST3_SEG1 0 +#define THM_BASE__INST3_SEG2 0 +#define THM_BASE__INST3_SEG3 0 +#define THM_BASE__INST3_SEG4 0 +#define THM_BASE__INST3_SEG5 0 + +#define THM_BASE__INST4_SEG0 0 +#define THM_BASE__INST4_SEG1 0 +#define THM_BASE__INST4_SEG2 0 +#define THM_BASE__INST4_SEG3 0 +#define THM_BASE__INST4_SEG4 0 +#define THM_BASE__INST4_SEG5 0 + +#define THM_BASE__INST5_SEG0 0 +#define THM_BASE__INST5_SEG1 0 +#define THM_BASE__INST5_SEG2 0 +#define THM_BASE__INST5_SEG3 0 +#define THM_BASE__INST5_SEG4 0 +#define THM_BASE__INST5_SEG5 0 + +#define UMC_BASE__INST0_SEG0 0x00014000 +#define UMC_BASE__INST0_SEG1 0 +#define UMC_BASE__INST0_SEG2 0 +#define UMC_BASE__INST0_SEG3 0 +#define UMC_BASE__INST0_SEG4 0 +#define UMC_BASE__INST0_SEG5 0 + +#define UMC_BASE__INST1_SEG0 0 +#define UMC_BASE__INST1_SEG1 0 +#define UMC_BASE__INST1_SEG2 0 +#define UMC_BASE__INST1_SEG3 0 +#define UMC_BASE__INST1_SEG4 0 +#define UMC_BASE__INST1_SEG5 0 + +#define UMC_BASE__INST2_SEG0 0 +#define UMC_BASE__INST2_SEG1 0 +#define UMC_BASE__INST2_SEG2 0 +#define UMC_BASE__INST2_SEG3 0 +#define UMC_BASE__INST2_SEG4 0 +#define UMC_BASE__INST2_SEG5 0 + +#define UMC_BASE__INST3_SEG0 0 +#define UMC_BASE__INST3_SEG1 0 +#define UMC_BASE__INST3_SEG2 0 +#define UMC_BASE__INST3_SEG3 0 +#define UMC_BASE__INST3_SEG4 0 +#define UMC_BASE__INST3_SEG5 0 + +#define UMC_BASE__INST4_SEG0 0 +#define UMC_BASE__INST4_SEG1 0 +#define UMC_BASE__INST4_SEG2 0 +#define UMC_BASE__INST4_SEG3 0 +#define UMC_BASE__INST4_SEG4 0 +#define UMC_BASE__INST4_SEG5 0 + +#define UMC_BASE__INST5_SEG0 0 +#define UMC_BASE__INST5_SEG1 0 +#define UMC_BASE__INST5_SEG2 0 +#define UMC_BASE__INST5_SEG3 0 +#define UMC_BASE__INST5_SEG4 0 +#define UMC_BASE__INST5_SEG5 0 + +#define UVD_BASE__INST0_SEG0 0x00007800 +#define UVD_BASE__INST0_SEG1 0x00007E00 +#define UVD_BASE__INST0_SEG2 0 +#define UVD_BASE__INST0_SEG3 0 +#define UVD_BASE__INST0_SEG4 0 +#define UVD_BASE__INST0_SEG5 0 + +#define UVD_BASE__INST1_SEG0 0 +#define UVD_BASE__INST1_SEG1 0x00009000 +#define UVD_BASE__INST1_SEG2 0 +#define UVD_BASE__INST1_SEG3 0 +#define UVD_BASE__INST1_SEG4 0 +#define UVD_BASE__INST1_SEG5 0 + +#define UVD_BASE__INST2_SEG0 0 +#define UVD_BASE__INST2_SEG1 0 +#define UVD_BASE__INST2_SEG2 0 +#define UVD_BASE__INST2_SEG3 0 +#define UVD_BASE__INST2_SEG4 0 +#define UVD_BASE__INST2_SEG5 0 + +#define UVD_BASE__INST3_SEG0 0 +#define UVD_BASE__INST3_SEG1 0 +#define UVD_BASE__INST3_SEG2 0 +#define UVD_BASE__INST3_SEG3 0 +#define UVD_BASE__INST3_SEG4 0 +#define UVD_BASE__INST3_SEG5 0 + +#define UVD_BASE__INST4_SEG0 0 +#define UVD_BASE__INST4_SEG1 0 +#define UVD_BASE__INST4_SEG2 0 +#define UVD_BASE__INST4_SEG3 0 +#define UVD_BASE__INST4_SEG4 0 +#define UVD_BASE__INST4_SEG5 0 + +#define UVD_BASE__INST5_SEG0 0 +#define UVD_BASE__INST5_SEG1 0 +#define UVD_BASE__INST5_SEG2 0 +#define UVD_BASE__INST5_SEG3 0 +#define UVD_BASE__INST5_SEG4 0 +#define UVD_BASE__INST5_SEG5 0 + +#define VCE_BASE__INST0_SEG0 0x00008800 +#define VCE_BASE__INST0_SEG1 0 +#define VCE_BASE__INST0_SEG2 0 +#define VCE_BASE__INST0_SEG3 0 +#define VCE_BASE__INST0_SEG4 0 +#define VCE_BASE__INST0_SEG5 0 + +#define VCE_BASE__INST1_SEG0 0 +#define VCE_BASE__INST1_SEG1 0 +#define VCE_BASE__INST1_SEG2 0 +#define VCE_BASE__INST1_SEG3 0 +#define VCE_BASE__INST1_SEG4 0 +#define VCE_BASE__INST1_SEG5 0 + +#define VCE_BASE__INST2_SEG0 0 +#define VCE_BASE__INST2_SEG1 0 +#define VCE_BASE__INST2_SEG2 0 +#define VCE_BASE__INST2_SEG3 0 +#define VCE_BASE__INST2_SEG4 0 +#define VCE_BASE__INST2_SEG5 0 + +#define VCE_BASE__INST3_SEG0 0 +#define VCE_BASE__INST3_SEG1 0 +#define VCE_BASE__INST3_SEG2 0 +#define VCE_BASE__INST3_SEG3 0 +#define VCE_BASE__INST3_SEG4 0 +#define VCE_BASE__INST3_SEG5 0 + +#define VCE_BASE__INST4_SEG0 0 +#define VCE_BASE__INST4_SEG1 0 +#define VCE_BASE__INST4_SEG2 0 +#define VCE_BASE__INST4_SEG3 0 +#define VCE_BASE__INST4_SEG4 0 +#define VCE_BASE__INST4_SEG5 0 + +#define VCE_BASE__INST5_SEG0 0 +#define VCE_BASE__INST5_SEG1 0 +#define VCE_BASE__INST5_SEG2 0 +#define VCE_BASE__INST5_SEG3 0 +#define VCE_BASE__INST5_SEG4 0 +#define VCE_BASE__INST5_SEG5 0 + +#define XDMA_BASE__INST0_SEG0 0x00003400 +#define XDMA_BASE__INST0_SEG1 0 +#define XDMA_BASE__INST0_SEG2 0 +#define XDMA_BASE__INST0_SEG3 0 +#define XDMA_BASE__INST0_SEG4 0 +#define XDMA_BASE__INST0_SEG5 0 + +#define XDMA_BASE__INST1_SEG0 0 +#define XDMA_BASE__INST1_SEG1 0 +#define XDMA_BASE__INST1_SEG2 0 +#define XDMA_BASE__INST1_SEG3 0 +#define XDMA_BASE__INST1_SEG4 0 +#define XDMA_BASE__INST1_SEG5 0 + +#define XDMA_BASE__INST2_SEG0 0 +#define XDMA_BASE__INST2_SEG1 0 +#define XDMA_BASE__INST2_SEG2 0 +#define XDMA_BASE__INST2_SEG3 0 +#define XDMA_BASE__INST2_SEG4 0 +#define XDMA_BASE__INST2_SEG5 0 + +#define XDMA_BASE__INST3_SEG0 0 +#define XDMA_BASE__INST3_SEG1 0 +#define XDMA_BASE__INST3_SEG2 0 +#define XDMA_BASE__INST3_SEG3 0 +#define XDMA_BASE__INST3_SEG4 0 +#define XDMA_BASE__INST3_SEG5 0 + +#define XDMA_BASE__INST4_SEG0 0 +#define XDMA_BASE__INST4_SEG1 0 +#define XDMA_BASE__INST4_SEG2 0 +#define XDMA_BASE__INST4_SEG3 0 +#define XDMA_BASE__INST4_SEG4 0 +#define XDMA_BASE__INST4_SEG5 0 + +#define XDMA_BASE__INST5_SEG0 0 +#define XDMA_BASE__INST5_SEG1 0 +#define XDMA_BASE__INST5_SEG2 0 +#define XDMA_BASE__INST5_SEG3 0 +#define XDMA_BASE__INST5_SEG4 0 +#define XDMA_BASE__INST5_SEG5 0 + +#define RSMU_BASE__INST0_SEG0 0x00012000 +#define RSMU_BASE__INST0_SEG1 0 +#define RSMU_BASE__INST0_SEG2 0 +#define RSMU_BASE__INST0_SEG3 0 +#define RSMU_BASE__INST0_SEG4 0 +#define RSMU_BASE__INST0_SEG5 0 + +#define RSMU_BASE__INST1_SEG0 0 +#define RSMU_BASE__INST1_SEG1 0 +#define RSMU_BASE__INST1_SEG2 0 +#define RSMU_BASE__INST1_SEG3 0 +#define RSMU_BASE__INST1_SEG4 0 +#define RSMU_BASE__INST1_SEG5 0 + +#define RSMU_BASE__INST2_SEG0 0 +#define RSMU_BASE__INST2_SEG1 0 +#define RSMU_BASE__INST2_SEG2 0 +#define RSMU_BASE__INST2_SEG3 0 +#define RSMU_BASE__INST2_SEG4 0 +#define RSMU_BASE__INST2_SEG5 0 + +#define RSMU_BASE__INST3_SEG0 0 +#define RSMU_BASE__INST3_SEG1 0 +#define RSMU_BASE__INST3_SEG2 0 +#define RSMU_BASE__INST3_SEG3 0 +#define RSMU_BASE__INST3_SEG4 0 +#define RSMU_BASE__INST3_SEG5 0 + +#define RSMU_BASE__INST4_SEG0 0 +#define RSMU_BASE__INST4_SEG1 0 +#define RSMU_BASE__INST4_SEG2 0 +#define RSMU_BASE__INST4_SEG3 0 +#define RSMU_BASE__INST4_SEG4 0 +#define RSMU_BASE__INST4_SEG5 0 + +#define RSMU_BASE__INST5_SEG0 0 +#define RSMU_BASE__INST5_SEG1 0 +#define RSMU_BASE__INST5_SEG2 0 +#define RSMU_BASE__INST5_SEG3 0 +#define RSMU_BASE__INST5_SEG4 0 +#define RSMU_BASE__INST5_SEG5 0 + +#endif + diff --git a/drivers/gpu/drm/amd/powerplay/amd_powerplay.c b/drivers/gpu/drm/amd/powerplay/amd_powerplay.c index 7e8ad30d98e2..d567be49c31b 100644 --- a/drivers/gpu/drm/amd/powerplay/amd_powerplay.c +++ b/drivers/gpu/drm/amd/powerplay/amd_powerplay.c @@ -25,30 +25,16 @@ #include #include #include +#include #include "amd_shared.h" #include "amd_powerplay.h" #include "power_state.h" #include "amdgpu.h" #include "hwmgr.h" -#define PP_DPM_DISABLED 0xCCCC - -static int pp_dpm_dispatch_tasks(void *handle, enum amd_pp_task task_id, - enum amd_pm_state_type *user_state); static const struct amd_pm_funcs pp_dpm_funcs; -static inline int pp_check(struct pp_hwmgr *hwmgr) -{ - if (hwmgr == NULL || hwmgr->smumgr_funcs == NULL) - return -EINVAL; - - if (hwmgr->pm_en == 0 || hwmgr->hwmgr_func == NULL) - return PP_DPM_DISABLED; - - return 0; -} - static int amd_powerplay_create(struct amdgpu_device *adev) { struct pp_hwmgr *hwmgr; @@ -61,19 +47,21 @@ static int amd_powerplay_create(struct amdgpu_device *adev) return -ENOMEM; hwmgr->adev = adev; - hwmgr->pm_en = (amdgpu_dpm != 0 && !amdgpu_sriov_vf(adev)) ? true : false; + hwmgr->not_vf = !amdgpu_sriov_vf(adev); + hwmgr->pm_en = (amdgpu_dpm && hwmgr->not_vf) ? true : false; hwmgr->device = amdgpu_cgs_create_device(adev); mutex_init(&hwmgr->smu_lock); hwmgr->chip_family = adev->family; hwmgr->chip_id = adev->asic_type; - hwmgr->feature_mask = amdgpu_pp_feature_mask; + hwmgr->feature_mask = adev->powerplay.pp_feature; + hwmgr->display_config = &adev->pm.pm_display_cfg; adev->powerplay.pp_handle = hwmgr; adev->powerplay.pp_funcs = &pp_dpm_funcs; return 0; } -static int amd_powerplay_destroy(struct amdgpu_device *adev) +static void amd_powerplay_destroy(struct amdgpu_device *adev) { struct pp_hwmgr *hwmgr = adev->powerplay.pp_handle; @@ -82,8 +70,6 @@ static int amd_powerplay_destroy(struct amdgpu_device *adev) kfree(hwmgr); hwmgr = NULL; - - return 0; } static int pp_early_init(void *handle) @@ -109,18 +95,9 @@ static int pp_sw_init(void *handle) struct pp_hwmgr *hwmgr = adev->powerplay.pp_handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret >= 0) { - if (hwmgr->smumgr_funcs->smu_init == NULL) - return -EINVAL; + ret = hwmgr_sw_init(hwmgr); - ret = hwmgr->smumgr_funcs->smu_init(hwmgr); - - phm_register_irq_handlers(hwmgr); - - pr_debug("amdgpu: powerplay sw initialized\n"); - } + pr_debug("powerplay sw init %s\n", ret ? "failed" : "successfully"); return ret; } @@ -129,16 +106,14 @@ static int pp_sw_fini(void *handle) { struct amdgpu_device *adev = handle; struct pp_hwmgr *hwmgr = adev->powerplay.pp_handle; - int ret = 0; - ret = pp_check(hwmgr); - if (ret >= 0) { - if (hwmgr->smumgr_funcs->smu_fini != NULL) - hwmgr->smumgr_funcs->smu_fini(hwmgr); - } + hwmgr_sw_fini(hwmgr); - if (adev->firmware.load_type == AMDGPU_FW_LOAD_SMU) + if (adev->firmware.load_type == AMDGPU_FW_LOAD_SMU) { + release_firmware(adev->pm.fw); + adev->pm.fw = NULL; amdgpu_ucode_fini_bo(adev); + } return 0; } @@ -152,55 +127,68 @@ static int pp_hw_init(void *handle) if (adev->firmware.load_type == AMDGPU_FW_LOAD_SMU) amdgpu_ucode_init_bo(adev); - ret = pp_check(hwmgr); + ret = hwmgr_hw_init(hwmgr); - if (ret >= 0) { - if (hwmgr->smumgr_funcs->start_smu == NULL) - return -EINVAL; + if (ret) + pr_err("powerplay hw init failed\n"); - if (hwmgr->smumgr_funcs->start_smu(hwmgr)) { - pr_err("smc start failed\n"); - hwmgr->smumgr_funcs->smu_fini(hwmgr); - return -EINVAL; - } - if (ret == PP_DPM_DISABLED) - goto exit; - ret = hwmgr_hw_init(hwmgr); - if (ret) - goto exit; - } return ret; -exit: - hwmgr->pm_en = 0; - cgs_notify_dpm_enabled(hwmgr->device, false); - return 0; - } static int pp_hw_fini(void *handle) { struct amdgpu_device *adev = handle; struct pp_hwmgr *hwmgr = adev->powerplay.pp_handle; - int ret = 0; - ret = pp_check(hwmgr); - if (ret == 0) - hwmgr_hw_fini(hwmgr); + hwmgr_hw_fini(hwmgr); return 0; } +static void pp_reserve_vram_for_smu(struct amdgpu_device *adev) +{ + int r = -EINVAL; + void *cpu_ptr = NULL; + uint64_t gpu_addr; + struct pp_hwmgr *hwmgr = adev->powerplay.pp_handle; + + if (amdgpu_bo_create_kernel(adev, adev->pm.smu_prv_buffer_size, + PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT, + &adev->pm.smu_prv_buffer, + &gpu_addr, + &cpu_ptr)) { + DRM_ERROR("amdgpu: failed to create smu prv buffer\n"); + return; + } + + if (hwmgr->hwmgr_func->notify_cac_buffer_info) + r = hwmgr->hwmgr_func->notify_cac_buffer_info(hwmgr, + lower_32_bits((unsigned long)cpu_ptr), + upper_32_bits((unsigned long)cpu_ptr), + lower_32_bits(gpu_addr), + upper_32_bits(gpu_addr), + adev->pm.smu_prv_buffer_size); + + if (r) { + amdgpu_bo_free_kernel(&adev->pm.smu_prv_buffer, NULL, NULL); + adev->pm.smu_prv_buffer = NULL; + DRM_ERROR("amdgpu: failed to notify SMU buffer address\n"); + } +} + static int pp_late_init(void *handle) { struct amdgpu_device *adev = handle; struct pp_hwmgr *hwmgr = adev->powerplay.pp_handle; - int ret = 0; - - ret = pp_check(hwmgr); - if (ret == 0) - pp_dpm_dispatch_tasks(hwmgr, + if (hwmgr && hwmgr->pm_en) { + mutex_lock(&hwmgr->smu_lock); + hwmgr_handle_task(hwmgr, AMD_PP_TASK_COMPLETE_INIT, NULL); + mutex_unlock(&hwmgr->smu_lock); + } + if (adev->pm.smu_prv_buffer_size != 0) + pp_reserve_vram_for_smu(adev); return 0; } @@ -209,6 +197,8 @@ static void pp_late_fini(void *handle) { struct amdgpu_device *adev = handle; + if (adev->pm.smu_prv_buffer) + amdgpu_bo_free_kernel(&adev->pm.smu_prv_buffer, NULL, NULL); amd_powerplay_destroy(adev); } @@ -233,15 +223,21 @@ static int pp_set_powergating_state(void *handle, { struct amdgpu_device *adev = handle; struct pp_hwmgr *hwmgr = adev->powerplay.pp_handle; - int ret = 0; + int ret; - ret = pp_check(hwmgr); + if (!hwmgr || !hwmgr->pm_en) + return 0; - if (ret) - return ret; + if (hwmgr->hwmgr_func->gfx_off_control) { + /* Enable/disable GFX off through SMU */ + ret = hwmgr->hwmgr_func->gfx_off_control(hwmgr, + state == AMD_PG_STATE_GATE); + if (ret) + pr_err("gfx off control failed!\n"); + } if (hwmgr->hwmgr_func->enable_per_cu_power_gating == NULL) { - pr_info("%s was not implemented.\n", __func__); + pr_debug("%s was not implemented.\n", __func__); return 0; } @@ -254,38 +250,16 @@ static int pp_suspend(void *handle) { struct amdgpu_device *adev = handle; struct pp_hwmgr *hwmgr = adev->powerplay.pp_handle; - int ret = 0; - ret = pp_check(hwmgr); - if (ret == 0) - hwmgr_hw_suspend(hwmgr); - return 0; + return hwmgr_suspend(hwmgr); } static int pp_resume(void *handle) { struct amdgpu_device *adev = handle; struct pp_hwmgr *hwmgr = adev->powerplay.pp_handle; - int ret; - - ret = pp_check(hwmgr); - - if (ret < 0) - return ret; - - if (hwmgr->smumgr_funcs->start_smu == NULL) - return -EINVAL; - - if (hwmgr->smumgr_funcs->start_smu(hwmgr)) { - pr_err("smc start failed\n"); - hwmgr->smumgr_funcs->smu_fini(hwmgr); - return -EINVAL; - } - if (ret == PP_DPM_DISABLED) - return 0; - - return hwmgr_hw_resume(hwmgr); + return hwmgr_resume(hwmgr); } static int pp_set_clockgating_state(void *handle, @@ -334,12 +308,9 @@ static int pp_dpm_fw_loading_complete(void *handle) static int pp_set_clockgating_by_smu(void *handle, uint32_t msg_id) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; - - ret = pp_check(hwmgr); - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->update_clock_gatings == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -362,10 +333,10 @@ static void pp_dpm_en_umd_pstate(struct pp_hwmgr *hwmgr, if (*level & profile_mode_mask) { hwmgr->saved_dpm_level = hwmgr->dpm_level; hwmgr->en_umd_pstate = true; - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_GFX, AMD_CG_STATE_UNGATE); - cgs_set_powergating_state(hwmgr->device, + amdgpu_device_ip_set_powergating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_GFX, AMD_PG_STATE_UNGATE); } @@ -375,10 +346,10 @@ static void pp_dpm_en_umd_pstate(struct pp_hwmgr *hwmgr, if (*level == AMD_DPM_FORCED_LEVEL_PROFILE_EXIT) *level = hwmgr->saved_dpm_level; hwmgr->en_umd_pstate = false; - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_GFX, AMD_CG_STATE_GATE); - cgs_set_powergating_state(hwmgr->device, + amdgpu_device_ip_set_powergating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_GFX, AMD_PG_STATE_GATE); } @@ -389,12 +360,9 @@ static int pp_dpm_force_performance_level(void *handle, enum amd_dpm_forced_level level) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (level == hwmgr->dpm_level) return 0; @@ -412,13 +380,10 @@ static enum amd_dpm_forced_level pp_dpm_get_performance_level( void *handle) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; enum amd_dpm_forced_level level; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; mutex_lock(&hwmgr->smu_lock); level = hwmgr->dpm_level; @@ -429,13 +394,10 @@ static enum amd_dpm_forced_level pp_dpm_get_performance_level( static uint32_t pp_dpm_get_sclk(void *handle, bool low) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; uint32_t clk = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return 0; if (hwmgr->hwmgr_func->get_sclk == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -450,13 +412,10 @@ static uint32_t pp_dpm_get_sclk(void *handle, bool low) static uint32_t pp_dpm_get_mclk(void *handle, bool low) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; uint32_t clk = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return 0; if (hwmgr->hwmgr_func->get_mclk == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -471,11 +430,8 @@ static uint32_t pp_dpm_get_mclk(void *handle, bool low) static void pp_dpm_powergate_vce(void *handle, bool gate) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; - ret = pp_check(hwmgr); - - if (ret) + if (!hwmgr || !hwmgr->pm_en) return; if (hwmgr->hwmgr_func->powergate_vce == NULL) { @@ -490,11 +446,8 @@ static void pp_dpm_powergate_vce(void *handle, bool gate) static void pp_dpm_powergate_uvd(void *handle, bool gate) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; - - ret = pp_check(hwmgr); - if (ret) + if (!hwmgr || !hwmgr->pm_en) return; if (hwmgr->hwmgr_func->powergate_uvd == NULL) { @@ -512,10 +465,8 @@ static int pp_dpm_dispatch_tasks(void *handle, enum amd_pp_task task_id, int ret = 0; struct pp_hwmgr *hwmgr = handle; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; mutex_lock(&hwmgr->smu_lock); ret = hwmgr_handle_task(hwmgr, task_id, user_state); @@ -528,15 +479,9 @@ static enum amd_pm_state_type pp_dpm_get_current_power_state(void *handle) { struct pp_hwmgr *hwmgr = handle; struct pp_power_state *state; - int ret = 0; enum amd_pm_state_type pm_type; - ret = pp_check(hwmgr); - - if (ret) - return ret; - - if (hwmgr->current_ps == NULL) + if (!hwmgr || !hwmgr->pm_en || !hwmgr->current_ps) return -EINVAL; mutex_lock(&hwmgr->smu_lock); @@ -568,11 +513,8 @@ static enum amd_pm_state_type pp_dpm_get_current_power_state(void *handle) static void pp_dpm_set_fan_control_mode(void *handle, uint32_t mode) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; - - ret = pp_check(hwmgr); - if (ret) + if (!hwmgr || !hwmgr->pm_en) return; if (hwmgr->hwmgr_func->set_fan_control_mode == NULL) { @@ -587,13 +529,10 @@ static void pp_dpm_set_fan_control_mode(void *handle, uint32_t mode) static uint32_t pp_dpm_get_fan_control_mode(void *handle) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; uint32_t mode = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return 0; if (hwmgr->hwmgr_func->get_fan_control_mode == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -610,10 +549,8 @@ static int pp_dpm_set_fan_speed_percent(void *handle, uint32_t percent) struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->set_fan_speed_percent == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -630,10 +567,8 @@ static int pp_dpm_get_fan_speed_percent(void *handle, uint32_t *speed) struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->get_fan_speed_percent == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -651,10 +586,8 @@ static int pp_dpm_get_fan_speed_rpm(void *handle, uint32_t *rpm) struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->get_fan_speed_rpm == NULL) return -EINVAL; @@ -670,16 +603,10 @@ static int pp_dpm_get_pp_num_states(void *handle, { struct pp_hwmgr *hwmgr = handle; int i; - int ret = 0; memset(data, 0, sizeof(*data)); - ret = pp_check(hwmgr); - - if (ret) - return ret; - - if (hwmgr->ps == NULL) + if (!hwmgr || !hwmgr->pm_en ||!hwmgr->ps) return -EINVAL; mutex_lock(&hwmgr->smu_lock); @@ -713,15 +640,9 @@ static int pp_dpm_get_pp_num_states(void *handle, static int pp_dpm_get_pp_table(void *handle, char **table) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; int size = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; - - if (!hwmgr->soft_pp_table) + if (!hwmgr || !hwmgr->pm_en ||!hwmgr->soft_pp_table) return -EINVAL; mutex_lock(&hwmgr->smu_lock); @@ -736,10 +657,6 @@ static int amd_powerplay_reset(void *handle) struct pp_hwmgr *hwmgr = handle; int ret; - ret = pp_check(hwmgr); - if (ret) - return ret; - ret = hwmgr_hw_fini(hwmgr); if (ret) return ret; @@ -754,40 +671,38 @@ static int amd_powerplay_reset(void *handle) static int pp_dpm_set_pp_table(void *handle, const char *buf, size_t size) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; - - ret = pp_check(hwmgr); + int ret = -ENOMEM; - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; mutex_lock(&hwmgr->smu_lock); if (!hwmgr->hardcode_pp_table) { hwmgr->hardcode_pp_table = kmemdup(hwmgr->soft_pp_table, hwmgr->soft_pp_table_size, GFP_KERNEL); - if (!hwmgr->hardcode_pp_table) { - mutex_unlock(&hwmgr->smu_lock); - return -ENOMEM; - } + if (!hwmgr->hardcode_pp_table) + goto err; } memcpy(hwmgr->hardcode_pp_table, buf, size); hwmgr->soft_pp_table = hwmgr->hardcode_pp_table; - mutex_unlock(&hwmgr->smu_lock); ret = amd_powerplay_reset(handle); if (ret) - return ret; + goto err; if (hwmgr->hwmgr_func->avfs_control) { ret = hwmgr->hwmgr_func->avfs_control(hwmgr, false); if (ret) - return ret; + goto err; } - + mutex_unlock(&hwmgr->smu_lock); return 0; +err: + mutex_unlock(&hwmgr->smu_lock); + return ret; } static int pp_dpm_force_clock_level(void *handle, @@ -796,10 +711,8 @@ static int pp_dpm_force_clock_level(void *handle, struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->force_clock_level == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -820,10 +733,8 @@ static int pp_dpm_print_clock_levels(void *handle, struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->print_clock_levels == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -840,10 +751,8 @@ static int pp_dpm_get_sclk_od(void *handle) struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->get_sclk_od == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -860,10 +769,8 @@ static int pp_dpm_set_sclk_od(void *handle, uint32_t value) struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->set_sclk_od == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -881,10 +788,8 @@ static int pp_dpm_get_mclk_od(void *handle) struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->get_mclk_od == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -901,10 +806,8 @@ static int pp_dpm_set_mclk_od(void *handle, uint32_t value) struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->set_mclk_od == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -922,11 +825,7 @@ static int pp_dpm_read_sensor(void *handle, int idx, struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - if (ret) - return ret; - - if (value == NULL) + if (!hwmgr || !hwmgr->pm_en || !value) return -EINVAL; switch (idx) { @@ -948,14 +847,11 @@ static struct amd_vce_state* pp_dpm_get_vce_clock_state(void *handle, unsigned idx) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; - ret = pp_check(hwmgr); - - if (ret) + if (!hwmgr || !hwmgr->pm_en) return NULL; - if (hwmgr && idx < hwmgr->num_vce_state_tables) + if (idx < hwmgr->num_vce_state_tables) return &hwmgr->vce_states[idx]; return NULL; } @@ -964,7 +860,7 @@ static int pp_get_power_profile_mode(void *handle, char *buf) { struct pp_hwmgr *hwmgr = handle; - if (!buf || pp_check(hwmgr)) + if (!hwmgr || !hwmgr->pm_en || !buf) return -EINVAL; if (hwmgr->hwmgr_func->get_power_profile_mode == NULL) { @@ -980,12 +876,12 @@ static int pp_set_power_profile_mode(void *handle, long *input, uint32_t size) struct pp_hwmgr *hwmgr = handle; int ret = -EINVAL; - if (pp_check(hwmgr)) - return -EINVAL; + if (!hwmgr || !hwmgr->pm_en) + return ret; if (hwmgr->hwmgr_func->set_power_profile_mode == NULL) { pr_info("%s was not implemented.\n", __func__); - return -EINVAL; + return ret; } mutex_lock(&hwmgr->smu_lock); if (hwmgr->dpm_level == AMD_DPM_FORCED_LEVEL_MANUAL) @@ -998,7 +894,7 @@ static int pp_odn_edit_dpm_table(void *handle, uint32_t type, long *input, uint3 { struct pp_hwmgr *hwmgr = handle; - if (pp_check(hwmgr)) + if (!hwmgr || !hwmgr->pm_en) return -EINVAL; if (hwmgr->hwmgr_func->odn_edit_dpm_table == NULL) { @@ -1016,7 +912,7 @@ static int pp_dpm_switch_power_profile(void *handle, long workload; uint32_t index; - if (pp_check(hwmgr)) + if (!hwmgr || !hwmgr->pm_en) return -EINVAL; if (hwmgr->hwmgr_func->set_power_profile_mode == NULL) { @@ -1048,46 +944,12 @@ static int pp_dpm_switch_power_profile(void *handle, return 0; } -static int pp_dpm_notify_smu_memory_info(void *handle, - uint32_t virtual_addr_low, - uint32_t virtual_addr_hi, - uint32_t mc_addr_low, - uint32_t mc_addr_hi, - uint32_t size) -{ - struct pp_hwmgr *hwmgr = handle; - int ret = 0; - - ret = pp_check(hwmgr); - - if (ret) - return ret; - - if (hwmgr->hwmgr_func->notify_cac_buffer_info == NULL) { - pr_info("%s was not implemented.\n", __func__); - return -EINVAL; - } - - mutex_lock(&hwmgr->smu_lock); - - ret = hwmgr->hwmgr_func->notify_cac_buffer_info(hwmgr, virtual_addr_low, - virtual_addr_hi, mc_addr_low, mc_addr_hi, - size); - - mutex_unlock(&hwmgr->smu_lock); - - return ret; -} - static int pp_set_power_limit(void *handle, uint32_t limit) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; - - ret = pp_check(hwmgr); - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->set_power_limit == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -1104,20 +966,14 @@ static int pp_set_power_limit(void *handle, uint32_t limit) hwmgr->hwmgr_func->set_power_limit(hwmgr, limit); hwmgr->power_limit = limit; mutex_unlock(&hwmgr->smu_lock); - return ret; + return 0; } static int pp_get_power_limit(void *handle, uint32_t *limit, bool default_limit) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; - - ret = pp_check(hwmgr); - - if (ret) - return ret; - if (limit == NULL) + if (!hwmgr || !hwmgr->pm_en ||!limit) return -EINVAL; mutex_lock(&hwmgr->smu_lock); @@ -1129,19 +985,16 @@ static int pp_get_power_limit(void *handle, uint32_t *limit, bool default_limit) mutex_unlock(&hwmgr->smu_lock); - return ret; + return 0; } static int pp_display_configuration_change(void *handle, const struct amd_pp_display_configuration *display_config) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; - - ret = pp_check(hwmgr); - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; mutex_lock(&hwmgr->smu_lock); phm_store_dal_configuration_data(hwmgr, display_config); @@ -1155,12 +1008,7 @@ static int pp_get_display_power_level(void *handle, struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; - - if (output == NULL) + if (!hwmgr || !hwmgr->pm_en ||!output) return -EINVAL; mutex_lock(&hwmgr->smu_lock); @@ -1177,10 +1025,8 @@ static int pp_get_current_clocks(void *handle, struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; mutex_lock(&hwmgr->smu_lock); @@ -1225,10 +1071,8 @@ static int pp_get_clock_by_type(void *handle, enum amd_pp_clock_type type, struc struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (clocks == NULL) return -EINVAL; @@ -1246,11 +1090,7 @@ static int pp_get_clock_by_type_with_latency(void *handle, struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - if (ret) - return ret; - - if (!clocks) + if (!hwmgr || !hwmgr->pm_en ||!clocks) return -EINVAL; mutex_lock(&hwmgr->smu_lock); @@ -1266,11 +1106,7 @@ static int pp_get_clock_by_type_with_voltage(void *handle, struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - if (ret) - return ret; - - if (!clocks) + if (!hwmgr || !hwmgr->pm_en ||!clocks) return -EINVAL; mutex_lock(&hwmgr->smu_lock); @@ -1287,11 +1123,7 @@ static int pp_set_watermarks_for_clocks_ranges(void *handle, struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - if (ret) - return ret; - - if (!wm_with_clock_ranges) + if (!hwmgr || !hwmgr->pm_en ||!wm_with_clock_ranges) return -EINVAL; mutex_lock(&hwmgr->smu_lock); @@ -1308,11 +1140,7 @@ static int pp_display_clock_voltage_request(void *handle, struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - if (ret) - return ret; - - if (!clock) + if (!hwmgr || !hwmgr->pm_en ||!clock) return -EINVAL; mutex_lock(&hwmgr->smu_lock); @@ -1328,12 +1156,7 @@ static int pp_get_display_mode_validation_clocks(void *handle, struct pp_hwmgr *hwmgr = handle; int ret = 0; - ret = pp_check(hwmgr); - - if (ret) - return ret; - - if (clocks == NULL) + if (!hwmgr || !hwmgr->pm_en ||!clocks) return -EINVAL; mutex_lock(&hwmgr->smu_lock); @@ -1348,12 +1171,9 @@ static int pp_get_display_mode_validation_clocks(void *handle, static int pp_set_mmhub_powergating_by_smu(void *handle) { struct pp_hwmgr *hwmgr = handle; - int ret = 0; - - ret = pp_check(hwmgr); - if (ret) - return ret; + if (!hwmgr || !hwmgr->pm_en) + return -EINVAL; if (hwmgr->hwmgr_func->set_mmhub_powergating_by_smu == NULL) { pr_info("%s was not implemented.\n", __func__); @@ -1390,7 +1210,6 @@ static const struct amd_pm_funcs pp_dpm_funcs = { .get_vce_clock_state = pp_dpm_get_vce_clock_state, .switch_power_profile = pp_dpm_switch_power_profile, .set_clockgating_by_smu = pp_set_clockgating_by_smu, - .notify_smu_memory_info = pp_dpm_notify_smu_memory_info, .get_power_profile_mode = pp_get_power_profile_mode, .set_power_profile_mode = pp_set_power_profile_mode, .odn_edit_dpm_table = pp_odn_edit_dpm_table, diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/hardwaremanager.c b/drivers/gpu/drm/amd/powerplay/hwmgr/hardwaremanager.c index ae2e9339dd6b..a0bb921fac22 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/hardwaremanager.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/hardwaremanager.c @@ -75,8 +75,7 @@ int phm_set_power_state(struct pp_hwmgr *hwmgr, int phm_enable_dynamic_state_management(struct pp_hwmgr *hwmgr) { - int ret = 1; - bool enabled; + int ret = -EINVAL;; PHM_FUNC_CHECK(hwmgr); if (smum_is_dpm_running(hwmgr)) { @@ -87,17 +86,12 @@ int phm_enable_dynamic_state_management(struct pp_hwmgr *hwmgr) if (NULL != hwmgr->hwmgr_func->dynamic_state_management_enable) ret = hwmgr->hwmgr_func->dynamic_state_management_enable(hwmgr); - enabled = ret == 0; - - cgs_notify_dpm_enabled(hwmgr->device, enabled); - return ret; } int phm_disable_dynamic_state_management(struct pp_hwmgr *hwmgr) { - int ret = -1; - bool enabled; + int ret = -EINVAL; PHM_FUNC_CHECK(hwmgr); @@ -109,10 +103,6 @@ int phm_disable_dynamic_state_management(struct pp_hwmgr *hwmgr) if (hwmgr->hwmgr_func->dynamic_state_management_disable) ret = hwmgr->hwmgr_func->dynamic_state_management_disable(hwmgr); - enabled = ret == 0 ? false : true; - - cgs_notify_dpm_enabled(hwmgr->device, enabled); - return ret; } @@ -142,6 +132,15 @@ int phm_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, return 0; } +int phm_apply_clock_adjust_rules(struct pp_hwmgr *hwmgr) +{ + PHM_FUNC_CHECK(hwmgr); + + if (hwmgr->hwmgr_func->apply_clocks_adjust_rules != NULL) + return hwmgr->hwmgr_func->apply_clocks_adjust_rules(hwmgr); + return 0; +} + int phm_powerdown_uvd(struct pp_hwmgr *hwmgr) { PHM_FUNC_CHECK(hwmgr); @@ -171,6 +170,16 @@ int phm_disable_clock_power_gatings(struct pp_hwmgr *hwmgr) return 0; } +int phm_pre_display_configuration_changed(struct pp_hwmgr *hwmgr) +{ + PHM_FUNC_CHECK(hwmgr); + + if (NULL != hwmgr->hwmgr_func->pre_display_config_changed) + hwmgr->hwmgr_func->pre_display_config_changed(hwmgr); + + return 0; + +} int phm_display_configuration_changed(struct pp_hwmgr *hwmgr) { @@ -275,13 +284,11 @@ int phm_store_dal_configuration_data(struct pp_hwmgr *hwmgr, if (display_config == NULL) return -EINVAL; - hwmgr->display_config = *display_config; - if (NULL != hwmgr->hwmgr_func->set_deep_sleep_dcefclk) - hwmgr->hwmgr_func->set_deep_sleep_dcefclk(hwmgr, hwmgr->display_config.min_dcef_deep_sleep_set_clk); + hwmgr->hwmgr_func->set_deep_sleep_dcefclk(hwmgr, display_config->min_dcef_deep_sleep_set_clk); - for (index = 0; index < hwmgr->display_config.num_path_including_non_display; index++) { - if (hwmgr->display_config.displays[index].controller_id != 0) + for (index = 0; index < display_config->num_path_including_non_display; index++) { + if (display_config->displays[index].controller_id != 0) number_of_active_display++; } diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/hwmgr.c b/drivers/gpu/drm/amd/powerplay/hwmgr/hwmgr.c index 42982055b161..e63bc47dc715 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/hwmgr.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/hwmgr.c @@ -40,6 +40,7 @@ extern const struct pp_smumgr_func iceland_smu_funcs; extern const struct pp_smumgr_func tonga_smu_funcs; extern const struct pp_smumgr_func fiji_smu_funcs; extern const struct pp_smumgr_func polaris10_smu_funcs; +extern const struct pp_smumgr_func vegam_smu_funcs; extern const struct pp_smumgr_func vega10_smu_funcs; extern const struct pp_smumgr_func vega12_smu_funcs; extern const struct pp_smumgr_func smu10_smu_funcs; @@ -76,7 +77,7 @@ static void hwmgr_init_workload_prority(struct pp_hwmgr *hwmgr) int hwmgr_early_init(struct pp_hwmgr *hwmgr) { - if (hwmgr == NULL) + if (!hwmgr) return -EINVAL; hwmgr->usec_timeout = AMD_MAX_USEC_TIMEOUT; @@ -95,7 +96,8 @@ int hwmgr_early_init(struct pp_hwmgr *hwmgr) hwmgr->smumgr_funcs = &ci_smu_funcs; ci_set_asic_special_caps(hwmgr); hwmgr->feature_mask &= ~(PP_VBI_TIME_SUPPORT_MASK | - PP_ENABLE_GFX_CG_THRU_SMU); + PP_ENABLE_GFX_CG_THRU_SMU | + PP_GFXOFF_MASK); hwmgr->pp_table_version = PP_TABLE_V0; hwmgr->od_enabled = false; smu7_init_function_pointers(hwmgr); @@ -103,9 +105,11 @@ int hwmgr_early_init(struct pp_hwmgr *hwmgr) case AMDGPU_FAMILY_CZ: hwmgr->od_enabled = false; hwmgr->smumgr_funcs = &smu8_smu_funcs; + hwmgr->feature_mask &= ~PP_GFXOFF_MASK; smu8_init_function_pointers(hwmgr); break; case AMDGPU_FAMILY_VI: + hwmgr->feature_mask &= ~PP_GFXOFF_MASK; switch (hwmgr->chip_id) { case CHIP_TOPAZ: hwmgr->smumgr_funcs = &iceland_smu_funcs; @@ -133,14 +137,21 @@ int hwmgr_early_init(struct pp_hwmgr *hwmgr) polaris_set_asic_special_caps(hwmgr); hwmgr->feature_mask &= ~(PP_UVD_HANDSHAKE_MASK); break; + case CHIP_VEGAM: + hwmgr->smumgr_funcs = &vegam_smu_funcs; + polaris_set_asic_special_caps(hwmgr); + hwmgr->feature_mask &= ~(PP_UVD_HANDSHAKE_MASK); + break; default: return -EINVAL; } smu7_init_function_pointers(hwmgr); break; case AMDGPU_FAMILY_AI: + hwmgr->feature_mask &= ~PP_GFXOFF_MASK; switch (hwmgr->chip_id) { case CHIP_VEGA10: + case CHIP_VEGA20: hwmgr->smumgr_funcs = &vega10_smu_funcs; vega10_hwmgr_init(hwmgr); break; @@ -170,22 +181,58 @@ int hwmgr_early_init(struct pp_hwmgr *hwmgr) return 0; } +int hwmgr_sw_init(struct pp_hwmgr *hwmgr) +{ + if (!hwmgr|| !hwmgr->smumgr_funcs || !hwmgr->smumgr_funcs->smu_init) + return -EINVAL; + + phm_register_irq_handlers(hwmgr); + + return hwmgr->smumgr_funcs->smu_init(hwmgr); +} + + +int hwmgr_sw_fini(struct pp_hwmgr *hwmgr) +{ + if (hwmgr && hwmgr->smumgr_funcs && hwmgr->smumgr_funcs->smu_fini) + hwmgr->smumgr_funcs->smu_fini(hwmgr); + + return 0; +} + int hwmgr_hw_init(struct pp_hwmgr *hwmgr) { int ret = 0; - if (hwmgr == NULL) + if (!hwmgr || !hwmgr->smumgr_funcs) return -EINVAL; - if (hwmgr->pptable_func == NULL || - hwmgr->pptable_func->pptable_init == NULL || - hwmgr->hwmgr_func->backend_init == NULL) - return -EINVAL; + if (hwmgr->smumgr_funcs->start_smu) { + ret = hwmgr->smumgr_funcs->start_smu(hwmgr); + if (ret) { + pr_err("smc start failed\n"); + return -EINVAL; + } + } + + if (!hwmgr->pm_en) + return 0; + + if (!hwmgr->pptable_func || + !hwmgr->pptable_func->pptable_init || + !hwmgr->hwmgr_func->backend_init) { + hwmgr->pm_en = false; + pr_info("dpm not supported \n"); + return 0; + } ret = hwmgr->pptable_func->pptable_init(hwmgr); if (ret) goto err; + ((struct amdgpu_device *)hwmgr->adev)->pm.no_fan = + hwmgr->thermal_controller.fanInfo.bNoFan; + ret = hwmgr->hwmgr_func->backend_init(hwmgr); if (ret) goto err1; @@ -206,6 +253,8 @@ int hwmgr_hw_init(struct pp_hwmgr *hwmgr) if (ret) goto err2; + ((struct amdgpu_device *)hwmgr->adev)->pm.dpm_enabled = true; + return 0; err2: if (hwmgr->hwmgr_func->backend_fini) @@ -214,14 +263,13 @@ err1: if (hwmgr->pptable_func->pptable_fini) hwmgr->pptable_func->pptable_fini(hwmgr); err: - pr_err("amdgpu: powerplay initialization failed\n"); return ret; } int hwmgr_hw_fini(struct pp_hwmgr *hwmgr) { - if (hwmgr == NULL) - return -EINVAL; + if (!hwmgr || !hwmgr->pm_en) + return 0; phm_stop_thermal_controller(hwmgr); psm_set_boot_states(hwmgr); @@ -236,12 +284,12 @@ int hwmgr_hw_fini(struct pp_hwmgr *hwmgr) return psm_fini_power_state_table(hwmgr); } -int hwmgr_hw_suspend(struct pp_hwmgr *hwmgr) +int hwmgr_suspend(struct pp_hwmgr *hwmgr) { int ret = 0; - if (hwmgr == NULL) - return -EINVAL; + if (!hwmgr || !hwmgr->pm_en) + return 0; phm_disable_smc_firmware_ctf(hwmgr); ret = psm_set_boot_states(hwmgr); @@ -255,13 +303,23 @@ int hwmgr_hw_suspend(struct pp_hwmgr *hwmgr) return ret; } -int hwmgr_hw_resume(struct pp_hwmgr *hwmgr) +int hwmgr_resume(struct pp_hwmgr *hwmgr) { int ret = 0; - if (hwmgr == NULL) + if (!hwmgr) return -EINVAL; + if (hwmgr->smumgr_funcs && hwmgr->smumgr_funcs->start_smu) { + if (hwmgr->smumgr_funcs->start_smu(hwmgr)) { + pr_err("smc start failed\n"); + return -EINVAL; + } + } + + if (!hwmgr->pm_en) + return 0; + ret = phm_setup_asic(hwmgr); if (ret) return ret; @@ -270,9 +328,6 @@ int hwmgr_hw_resume(struct pp_hwmgr *hwmgr) if (ret) return ret; ret = phm_start_thermal_controller(hwmgr); - if (ret) - return ret; - ret |= psm_set_performance_states(hwmgr); if (ret) return ret; diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/pp_psm.c b/drivers/gpu/drm/amd/powerplay/hwmgr/pp_psm.c index 0f2851b5b368..91ffb7bc4ee7 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/pp_psm.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/pp_psm.c @@ -46,11 +46,11 @@ int psm_init_power_state_table(struct pp_hwmgr *hwmgr) sizeof(struct pp_power_state); if (table_entries == 0 || size == 0) { - pr_warn("Please check whether power state management is suppported on this asic\n"); + pr_warn("Please check whether power state management is supported on this asic\n"); return 0; } - hwmgr->ps = kzalloc(size * table_entries, GFP_KERNEL); + hwmgr->ps = kcalloc(table_entries, size, GFP_KERNEL); if (hwmgr->ps == NULL) return -ENOMEM; @@ -265,10 +265,18 @@ int psm_adjust_power_state_dynamic(struct pp_hwmgr *hwmgr, bool skip, if (skip) return 0; + phm_pre_display_configuration_changed(hwmgr); + phm_display_configuration_changed(hwmgr); if (hwmgr->ps) power_state_management(hwmgr, new_ps); + else + /* + * for vega12/vega20 which does not support power state manager + * DAL clock limits should also be honoured + */ + phm_apply_clock_adjust_rules(hwmgr); phm_notify_smc_display_config_after_ps_adjustment(hwmgr); diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomctrl.c b/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomctrl.c index c6febbf0bf69..7047e29755c3 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomctrl.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomctrl.c @@ -23,7 +23,8 @@ #include "pp_debug.h" #include #include - +#include +#include "atom.h" #include "ppatomctrl.h" #include "atombios.h" #include "cgs_common.h" @@ -128,7 +129,6 @@ static int atomctrl_set_mc_reg_address_table( return 0; } - int atomctrl_initialize_mc_reg_table( struct pp_hwmgr *hwmgr, uint8_t module_index, @@ -141,7 +141,7 @@ int atomctrl_initialize_mc_reg_table( u16 size; vram_info = (ATOM_VRAM_INFO_HEADER_V2_1 *) - cgs_atom_get_data_table(hwmgr->device, + smu_atom_get_data_table(hwmgr->adev, GetIndexIntoMasterTable(DATA, VRAM_Info), &size, &frev, &crev); if (module_index >= vram_info->ucNumOfVRAMModule) { @@ -174,6 +174,8 @@ int atomctrl_set_engine_dram_timings_rv770( uint32_t engine_clock, uint32_t memory_clock) { + struct amdgpu_device *adev = hwmgr->adev; + SET_ENGINE_CLOCK_PS_ALLOCATION engine_clock_parameters; /* They are both in 10KHz Units. */ @@ -184,9 +186,10 @@ int atomctrl_set_engine_dram_timings_rv770( /* in 10 khz units.*/ engine_clock_parameters.sReserved.ulClock = cpu_to_le32(memory_clock & SET_CLOCK_FREQ_MASK); - return cgs_atom_exec_cmd_table(hwmgr->device, + + return amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, DynamicMemorySettings), - &engine_clock_parameters); + (uint32_t *)&engine_clock_parameters); } /** @@ -203,7 +206,7 @@ static ATOM_VOLTAGE_OBJECT_INFO *get_voltage_info_table(void *device) union voltage_object_info *voltage_info; voltage_info = (union voltage_object_info *) - cgs_atom_get_data_table(device, index, + smu_atom_get_data_table(device, index, &size, &frev, &crev); if (voltage_info != NULL) @@ -247,16 +250,16 @@ int atomctrl_get_memory_pll_dividers_si( pp_atomctrl_memory_clock_param *mpll_param, bool strobe_mode) { + struct amdgpu_device *adev = hwmgr->adev; COMPUTE_MEMORY_CLOCK_PARAM_PARAMETERS_V2_1 mpll_parameters; int result; mpll_parameters.ulClock = cpu_to_le32(clock_value); mpll_parameters.ucInputFlag = (uint8_t)((strobe_mode) ? 1 : 0); - result = cgs_atom_exec_cmd_table - (hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ComputeMemoryClockParam), - &mpll_parameters); + (uint32_t *)&mpll_parameters); if (0 == result) { mpll_param->mpll_fb_divider.clk_frac = @@ -295,14 +298,15 @@ int atomctrl_get_memory_pll_dividers_si( int atomctrl_get_memory_pll_dividers_vi(struct pp_hwmgr *hwmgr, uint32_t clock_value, pp_atomctrl_memory_clock_param *mpll_param) { + struct amdgpu_device *adev = hwmgr->adev; COMPUTE_MEMORY_CLOCK_PARAM_PARAMETERS_V2_2 mpll_parameters; int result; mpll_parameters.ulClock.ulClock = cpu_to_le32(clock_value); - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ComputeMemoryClockParam), - &mpll_parameters); + (uint32_t *)&mpll_parameters); if (!result) mpll_param->mpll_post_divider = @@ -311,19 +315,49 @@ int atomctrl_get_memory_pll_dividers_vi(struct pp_hwmgr *hwmgr, return result; } +int atomctrl_get_memory_pll_dividers_ai(struct pp_hwmgr *hwmgr, + uint32_t clock_value, + pp_atomctrl_memory_clock_param_ai *mpll_param) +{ + struct amdgpu_device *adev = hwmgr->adev; + COMPUTE_MEMORY_CLOCK_PARAM_PARAMETERS_V2_3 mpll_parameters = {{0}, 0, 0}; + int result; + + mpll_parameters.ulClock.ulClock = cpu_to_le32(clock_value); + + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, + GetIndexIntoMasterTable(COMMAND, ComputeMemoryClockParam), + (uint32_t *)&mpll_parameters); + + /* VEGAM's mpll takes sometime to finish computing */ + udelay(10); + + if (!result) { + mpll_param->ulMclk_fcw_int = + le16_to_cpu(mpll_parameters.usMclk_fcw_int); + mpll_param->ulMclk_fcw_frac = + le16_to_cpu(mpll_parameters.usMclk_fcw_frac); + mpll_param->ulClock = + le32_to_cpu(mpll_parameters.ulClock.ulClock); + mpll_param->ulPostDiv = mpll_parameters.ulClock.ucPostDiv; + } + + return result; +} + int atomctrl_get_engine_pll_dividers_kong(struct pp_hwmgr *hwmgr, uint32_t clock_value, pp_atomctrl_clock_dividers_kong *dividers) { + struct amdgpu_device *adev = hwmgr->adev; COMPUTE_MEMORY_ENGINE_PLL_PARAMETERS_V4 pll_parameters; int result; pll_parameters.ulClock = cpu_to_le32(clock_value); - result = cgs_atom_exec_cmd_table - (hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ComputeMemoryEnginePLL), - &pll_parameters); + (uint32_t *)&pll_parameters); if (0 == result) { dividers->pll_post_divider = pll_parameters.ucPostDiv; @@ -338,16 +372,16 @@ int atomctrl_get_engine_pll_dividers_vi( uint32_t clock_value, pp_atomctrl_clock_dividers_vi *dividers) { + struct amdgpu_device *adev = hwmgr->adev; COMPUTE_GPU_CLOCK_OUTPUT_PARAMETERS_V1_6 pll_patameters; int result; pll_patameters.ulClock.ulClock = cpu_to_le32(clock_value); pll_patameters.ulClock.ucPostDiv = COMPUTE_GPUCLK_INPUT_FLAG_SCLK; - result = cgs_atom_exec_cmd_table - (hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ComputeMemoryEnginePLL), - &pll_patameters); + (uint32_t *)&pll_patameters); if (0 == result) { dividers->pll_post_divider = @@ -375,16 +409,16 @@ int atomctrl_get_engine_pll_dividers_ai(struct pp_hwmgr *hwmgr, uint32_t clock_value, pp_atomctrl_clock_dividers_ai *dividers) { + struct amdgpu_device *adev = hwmgr->adev; COMPUTE_GPU_CLOCK_OUTPUT_PARAMETERS_V1_7 pll_patameters; int result; pll_patameters.ulClock.ulClock = cpu_to_le32(clock_value); pll_patameters.ulClock.ucPostDiv = COMPUTE_GPUCLK_INPUT_FLAG_SCLK; - result = cgs_atom_exec_cmd_table - (hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ComputeMemoryEnginePLL), - &pll_patameters); + (uint32_t *)&pll_patameters); if (0 == result) { dividers->usSclk_fcw_frac = le16_to_cpu(pll_patameters.usSclk_fcw_frac); @@ -407,6 +441,7 @@ int atomctrl_get_dfs_pll_dividers_vi( uint32_t clock_value, pp_atomctrl_clock_dividers_vi *dividers) { + struct amdgpu_device *adev = hwmgr->adev; COMPUTE_GPU_CLOCK_OUTPUT_PARAMETERS_V1_6 pll_patameters; int result; @@ -414,10 +449,9 @@ int atomctrl_get_dfs_pll_dividers_vi( pll_patameters.ulClock.ucPostDiv = COMPUTE_GPUCLK_INPUT_FLAG_DEFAULT_GPUCLK; - result = cgs_atom_exec_cmd_table - (hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ComputeMemoryEnginePLL), - &pll_patameters); + (uint32_t *)&pll_patameters); if (0 == result) { dividers->pll_post_divider = @@ -452,7 +486,7 @@ uint32_t atomctrl_get_reference_clock(struct pp_hwmgr *hwmgr) uint32_t clock; fw_info = (ATOM_FIRMWARE_INFO *) - cgs_atom_get_data_table(hwmgr->device, + smu_atom_get_data_table(hwmgr->adev, GetIndexIntoMasterTable(DATA, FirmwareInfo), &size, &frev, &crev); @@ -476,7 +510,7 @@ bool atomctrl_is_voltage_controlled_by_gpio_v3( uint8_t voltage_mode) { ATOM_VOLTAGE_OBJECT_INFO_V3_1 *voltage_info = - (ATOM_VOLTAGE_OBJECT_INFO_V3_1 *)get_voltage_info_table(hwmgr->device); + (ATOM_VOLTAGE_OBJECT_INFO_V3_1 *)get_voltage_info_table(hwmgr->adev); bool ret; PP_ASSERT_WITH_CODE((NULL != voltage_info), @@ -495,7 +529,7 @@ int atomctrl_get_voltage_table_v3( pp_atomctrl_voltage_table *voltage_table) { ATOM_VOLTAGE_OBJECT_INFO_V3_1 *voltage_info = - (ATOM_VOLTAGE_OBJECT_INFO_V3_1 *)get_voltage_info_table(hwmgr->device); + (ATOM_VOLTAGE_OBJECT_INFO_V3_1 *)get_voltage_info_table(hwmgr->adev); const ATOM_VOLTAGE_OBJECT_V3 *voltage_object; unsigned int i; @@ -572,7 +606,7 @@ static ATOM_GPIO_PIN_LUT *get_gpio_lookup_table(void *device) void *table_address; table_address = (ATOM_GPIO_PIN_LUT *) - cgs_atom_get_data_table(device, + smu_atom_get_data_table(device, GetIndexIntoMasterTable(DATA, GPIO_Pin_LUT), &size, &frev, &crev); @@ -592,7 +626,7 @@ bool atomctrl_get_pp_assign_pin( { bool bRet = false; ATOM_GPIO_PIN_LUT *gpio_lookup_table = - get_gpio_lookup_table(hwmgr->device); + get_gpio_lookup_table(hwmgr->adev); PP_ASSERT_WITH_CODE((NULL != gpio_lookup_table), "Could not find GPIO lookup Table in BIOS.", return false); @@ -613,7 +647,7 @@ int atomctrl_calculate_voltage_evv_on_sclk( bool debug) { ATOM_ASIC_PROFILING_INFO_V3_4 *getASICProfilingInfo; - + struct amdgpu_device *adev = hwmgr->adev; EFUSE_LINEAR_FUNC_PARAM sRO_fuse; EFUSE_LINEAR_FUNC_PARAM sCACm_fuse; EFUSE_LINEAR_FUNC_PARAM sCACb_fuse; @@ -640,7 +674,7 @@ int atomctrl_calculate_voltage_evv_on_sclk( int result; getASICProfilingInfo = (ATOM_ASIC_PROFILING_INFO_V3_4 *) - cgs_atom_get_data_table(hwmgr->device, + smu_atom_get_data_table(hwmgr->adev, GetIndexIntoMasterTable(DATA, ASIC_ProfilingInfo), NULL, NULL, NULL); @@ -706,9 +740,9 @@ int atomctrl_calculate_voltage_evv_on_sclk( sOutput_FuseValues.sEfuse = sInput_FuseValues; - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ReadEfuseValue), - &sOutput_FuseValues); + (uint32_t *)&sOutput_FuseValues); if (result) return result; @@ -727,9 +761,9 @@ int atomctrl_calculate_voltage_evv_on_sclk( sOutput_FuseValues.sEfuse = sInput_FuseValues; - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ReadEfuseValue), - &sOutput_FuseValues); + (uint32_t *)&sOutput_FuseValues); if (result) return result; @@ -747,9 +781,9 @@ int atomctrl_calculate_voltage_evv_on_sclk( sInput_FuseValues.ucBitLength = sCACb_fuse.ucEfuseLength; sOutput_FuseValues.sEfuse = sInput_FuseValues; - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ReadEfuseValue), - &sOutput_FuseValues); + (uint32_t *)&sOutput_FuseValues); if (result) return result; @@ -768,9 +802,9 @@ int atomctrl_calculate_voltage_evv_on_sclk( sOutput_FuseValues.sEfuse = sInput_FuseValues; - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ReadEfuseValue), - &sOutput_FuseValues); + (uint32_t *)&sOutput_FuseValues); if (result) return result; @@ -790,9 +824,9 @@ int atomctrl_calculate_voltage_evv_on_sclk( sOutput_FuseValues.sEfuse = sInput_FuseValues; - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ReadEfuseValue), - &sOutput_FuseValues); + (uint32_t *)&sOutput_FuseValues); if (result) return result; @@ -811,9 +845,9 @@ int atomctrl_calculate_voltage_evv_on_sclk( sInput_FuseValues.ucBitLength = sKv_b_fuse.ucEfuseLength; sOutput_FuseValues.sEfuse = sInput_FuseValues; - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ReadEfuseValue), - &sOutput_FuseValues); + (uint32_t *)&sOutput_FuseValues); if (result) return result; @@ -842,9 +876,9 @@ int atomctrl_calculate_voltage_evv_on_sclk( sOutput_FuseValues.sEfuse = sInput_FuseValues; - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ReadEfuseValue), - &sOutput_FuseValues); + (uint32_t *)&sOutput_FuseValues); if (result) return result; @@ -1053,8 +1087,9 @@ int atomctrl_get_voltage_evv_on_sclk( uint32_t sclk, uint16_t virtual_voltage_Id, uint16_t *voltage) { - int result; + struct amdgpu_device *adev = hwmgr->adev; GET_VOLTAGE_INFO_INPUT_PARAMETER_V1_2 get_voltage_info_param_space; + int result; get_voltage_info_param_space.ucVoltageType = voltage_type; @@ -1065,14 +1100,12 @@ int atomctrl_get_voltage_evv_on_sclk( get_voltage_info_param_space.ulSCLKFreq = cpu_to_le32(sclk); - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, GetVoltageInfo), - &get_voltage_info_param_space); - - if (0 != result) - return result; + (uint32_t *)&get_voltage_info_param_space); - *voltage = le16_to_cpu(((GET_EVV_VOLTAGE_INFO_OUTPUT_PARAMETER_V1_2 *) + *voltage = result ? 0 : + le16_to_cpu(((GET_EVV_VOLTAGE_INFO_OUTPUT_PARAMETER_V1_2 *) (&get_voltage_info_param_space))->usVoltageLevel); return result; @@ -1088,9 +1121,10 @@ int atomctrl_get_voltage_evv(struct pp_hwmgr *hwmgr, uint16_t virtual_voltage_id, uint16_t *voltage) { + struct amdgpu_device *adev = hwmgr->adev; + GET_VOLTAGE_INFO_INPUT_PARAMETER_V1_2 get_voltage_info_param_space; int result; int entry_id; - GET_VOLTAGE_INFO_INPUT_PARAMETER_V1_2 get_voltage_info_param_space; /* search for leakage voltage ID 0xff01 ~ 0xff08 and sckl */ for (entry_id = 0; entry_id < hwmgr->dyn_state.vddc_dependency_on_sclk->count; entry_id++) { @@ -1111,9 +1145,9 @@ int atomctrl_get_voltage_evv(struct pp_hwmgr *hwmgr, get_voltage_info_param_space.ulSCLKFreq = cpu_to_le32(hwmgr->dyn_state.vddc_dependency_on_sclk->entries[entry_id].clk); - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, GetVoltageInfo), - &get_voltage_info_param_space); + (uint32_t *)&get_voltage_info_param_space); if (0 != result) return result; @@ -1135,7 +1169,7 @@ uint32_t atomctrl_get_mpll_reference_clock(struct pp_hwmgr *hwmgr) u16 size; fw_info = (ATOM_COMMON_TABLE_HEADER *) - cgs_atom_get_data_table(hwmgr->device, + smu_atom_get_data_table(hwmgr->adev, GetIndexIntoMasterTable(DATA, FirmwareInfo), &size, &frev, &crev); @@ -1167,7 +1201,7 @@ static ATOM_ASIC_INTERNAL_SS_INFO *asic_internal_ss_get_ss_table(void *device) u16 size; table = (ATOM_ASIC_INTERNAL_SS_INFO *) - cgs_atom_get_data_table(device, + smu_atom_get_data_table(device, GetIndexIntoMasterTable(DATA, ASIC_InternalSS_Info), &size, &frev, &crev); @@ -1188,7 +1222,7 @@ static int asic_internal_ss_get_ss_asignment(struct pp_hwmgr *hwmgr, memset(ssEntry, 0x00, sizeof(pp_atomctrl_internal_ss_info)); - table = asic_internal_ss_get_ss_table(hwmgr->device); + table = asic_internal_ss_get_ss_table(hwmgr->adev); if (NULL == table) return -1; @@ -1260,9 +1294,10 @@ int atomctrl_get_engine_clock_spread_spectrum( ASIC_INTERNAL_ENGINE_SS, engine_clock, ssInfo); } -int atomctrl_read_efuse(void *device, uint16_t start_index, +int atomctrl_read_efuse(struct pp_hwmgr *hwmgr, uint16_t start_index, uint16_t end_index, uint32_t mask, uint32_t *efuse) { + struct amdgpu_device *adev = hwmgr->adev; int result; READ_EFUSE_VALUE_PARAMETER efuse_param; @@ -1272,11 +1307,10 @@ int atomctrl_read_efuse(void *device, uint16_t start_index, efuse_param.sEfuse.ucBitLength = (uint8_t) ((end_index - start_index) + 1); - result = cgs_atom_exec_cmd_table(device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, ReadEfuseValue), - &efuse_param); - if (!result) - *efuse = le32_to_cpu(efuse_param.ulEfuseValue) & mask; + (uint32_t *)&efuse_param); + *efuse = result ? 0 : le32_to_cpu(efuse_param.ulEfuseValue) & mask; return result; } @@ -1284,6 +1318,7 @@ int atomctrl_read_efuse(void *device, uint16_t start_index, int atomctrl_set_ac_timing_ai(struct pp_hwmgr *hwmgr, uint32_t memory_clock, uint8_t level) { + struct amdgpu_device *adev = hwmgr->adev; DYNAMICE_MEMORY_SETTINGS_PARAMETER_V2_1 memory_clock_parameters; int result; @@ -1293,10 +1328,9 @@ int atomctrl_set_ac_timing_ai(struct pp_hwmgr *hwmgr, uint32_t memory_clock, ADJUST_MC_SETTING_PARAM; memory_clock_parameters.asDPMMCReg.ucMclkDPMState = level; - result = cgs_atom_exec_cmd_table - (hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, DynamicMemorySettings), - &memory_clock_parameters); + (uint32_t *)&memory_clock_parameters); return result; } @@ -1304,7 +1338,7 @@ int atomctrl_set_ac_timing_ai(struct pp_hwmgr *hwmgr, uint32_t memory_clock, int atomctrl_get_voltage_evv_on_sclk_ai(struct pp_hwmgr *hwmgr, uint8_t voltage_type, uint32_t sclk, uint16_t virtual_voltage_Id, uint32_t *voltage) { - + struct amdgpu_device *adev = hwmgr->adev; int result; GET_VOLTAGE_INFO_INPUT_PARAMETER_V1_3 get_voltage_info_param_space; @@ -1313,15 +1347,12 @@ int atomctrl_get_voltage_evv_on_sclk_ai(struct pp_hwmgr *hwmgr, uint8_t voltage_ get_voltage_info_param_space.usVoltageLevel = cpu_to_le16(virtual_voltage_Id); get_voltage_info_param_space.ulSCLKFreq = cpu_to_le32(sclk); - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, GetVoltageInfo), - &get_voltage_info_param_space); + (uint32_t *)&get_voltage_info_param_space); - if (0 != result) - return result; - - *voltage = le32_to_cpu(((GET_EVV_VOLTAGE_INFO_OUTPUT_PARAMETER_V1_3 *) - (&get_voltage_info_param_space))->ulVoltageLevel); + *voltage = result ? 0 : + le32_to_cpu(((GET_EVV_VOLTAGE_INFO_OUTPUT_PARAMETER_V1_3 *)(&get_voltage_info_param_space))->ulVoltageLevel); return result; } @@ -1334,7 +1365,7 @@ int atomctrl_get_smc_sclk_range_table(struct pp_hwmgr *hwmgr, struct pp_atom_ctr u16 size; ATOM_SMU_INFO_V2_1 *psmu_info = - (ATOM_SMU_INFO_V2_1 *)cgs_atom_get_data_table(hwmgr->device, + (ATOM_SMU_INFO_V2_1 *)smu_atom_get_data_table(hwmgr->adev, GetIndexIntoMasterTable(DATA, SMU_Info), &size, &frev, &crev); @@ -1362,7 +1393,7 @@ int atomctrl_get_avfs_information(struct pp_hwmgr *hwmgr, return -EINVAL; profile = (ATOM_ASIC_PROFILING_INFO_V3_6 *) - cgs_atom_get_data_table(hwmgr->device, + smu_atom_get_data_table(hwmgr->adev, GetIndexIntoMasterTable(DATA, ASIC_ProfilingInfo), NULL, NULL, NULL); if (!profile) @@ -1402,7 +1433,7 @@ int atomctrl_get_svi2_info(struct pp_hwmgr *hwmgr, uint8_t voltage_type, uint16_t *load_line) { ATOM_VOLTAGE_OBJECT_INFO_V3_1 *voltage_info = - (ATOM_VOLTAGE_OBJECT_INFO_V3_1 *)get_voltage_info_table(hwmgr->device); + (ATOM_VOLTAGE_OBJECT_INFO_V3_1 *)get_voltage_info_table(hwmgr->adev); const ATOM_VOLTAGE_OBJECT_V3 *voltage_object; @@ -1421,16 +1452,17 @@ int atomctrl_get_svi2_info(struct pp_hwmgr *hwmgr, uint8_t voltage_type, int atomctrl_get_leakage_id_from_efuse(struct pp_hwmgr *hwmgr, uint16_t *virtual_voltage_id) { - int result; + struct amdgpu_device *adev = hwmgr->adev; SET_VOLTAGE_PS_ALLOCATION allocation; SET_VOLTAGE_PARAMETERS_V1_3 *voltage_parameters = (SET_VOLTAGE_PARAMETERS_V1_3 *)&allocation.sASICSetVoltage; + int result; voltage_parameters->ucVoltageMode = ATOM_GET_LEAKAGE_ID; - result = cgs_atom_exec_cmd_table(hwmgr->device, + result = amdgpu_atom_execute_table(adev->mode_info.atom_context, GetIndexIntoMasterTable(COMMAND, SetVoltage), - voltage_parameters); + (uint32_t *)voltage_parameters); *virtual_voltage_id = voltage_parameters->usVoltageLevel; @@ -1453,7 +1485,7 @@ int atomctrl_get_leakage_vddc_base_on_leakage(struct pp_hwmgr *hwmgr, ix = GetIndexIntoMasterTable(DATA, ASIC_ProfilingInfo); profile = (ATOM_ASIC_PROFILING_INFO_V2_1 *) - cgs_atom_get_data_table(hwmgr->device, + smu_atom_get_data_table(hwmgr->adev, ix, NULL, NULL, NULL); if (!profile) @@ -1498,3 +1530,33 @@ int atomctrl_get_leakage_vddc_base_on_leakage(struct pp_hwmgr *hwmgr, return 0; } + +void atomctrl_get_voltage_range(struct pp_hwmgr *hwmgr, uint32_t *max_vddc, + uint32_t *min_vddc) +{ + void *profile; + + profile = smu_atom_get_data_table(hwmgr->adev, + GetIndexIntoMasterTable(DATA, ASIC_ProfilingInfo), + NULL, NULL, NULL); + + if (profile) { + switch (hwmgr->chip_id) { + case CHIP_TONGA: + case CHIP_FIJI: + *max_vddc = le32_to_cpu(((ATOM_ASIC_PROFILING_INFO_V3_3 *)profile)->ulMaxVddc/4); + *min_vddc = le32_to_cpu(((ATOM_ASIC_PROFILING_INFO_V3_3 *)profile)->ulMinVddc/4); + return; + case CHIP_POLARIS11: + case CHIP_POLARIS10: + case CHIP_POLARIS12: + *max_vddc = le32_to_cpu(((ATOM_ASIC_PROFILING_INFO_V3_6 *)profile)->ulMaxVddc/100); + *min_vddc = le32_to_cpu(((ATOM_ASIC_PROFILING_INFO_V3_6 *)profile)->ulMinVddc/100); + return; + default: + break; + } + } + *max_vddc = 0; + *min_vddc = 0; +} diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomctrl.h b/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomctrl.h index c44a92064cf1..3ee54f182943 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomctrl.h +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomctrl.h @@ -146,6 +146,14 @@ struct pp_atomctrl_memory_clock_param { }; typedef struct pp_atomctrl_memory_clock_param pp_atomctrl_memory_clock_param; +struct pp_atomctrl_memory_clock_param_ai { + uint32_t ulClock; + uint32_t ulPostDiv; + uint16_t ulMclk_fcw_frac; + uint16_t ulMclk_fcw_int; +}; +typedef struct pp_atomctrl_memory_clock_param_ai pp_atomctrl_memory_clock_param_ai; + struct pp_atomctrl_internal_ss_info { uint32_t speed_spectrum_percentage; /* in 1/100 percentage */ uint32_t speed_spectrum_rate; /* in KHz */ @@ -295,10 +303,12 @@ extern bool atomctrl_is_voltage_controlled_by_gpio_v3(struct pp_hwmgr *hwmgr, ui extern int atomctrl_get_voltage_table_v3(struct pp_hwmgr *hwmgr, uint8_t voltage_type, uint8_t voltage_mode, pp_atomctrl_voltage_table *voltage_table); extern int atomctrl_get_memory_pll_dividers_vi(struct pp_hwmgr *hwmgr, uint32_t clock_value, pp_atomctrl_memory_clock_param *mpll_param); +extern int atomctrl_get_memory_pll_dividers_ai(struct pp_hwmgr *hwmgr, + uint32_t clock_value, pp_atomctrl_memory_clock_param_ai *mpll_param); extern int atomctrl_get_engine_pll_dividers_kong(struct pp_hwmgr *hwmgr, uint32_t clock_value, pp_atomctrl_clock_dividers_kong *dividers); -extern int atomctrl_read_efuse(void *device, uint16_t start_index, +extern int atomctrl_read_efuse(struct pp_hwmgr *hwmgr, uint16_t start_index, uint16_t end_index, uint32_t mask, uint32_t *efuse); extern int atomctrl_calculate_voltage_evv_on_sclk(struct pp_hwmgr *hwmgr, uint8_t voltage_type, uint32_t sclk, uint16_t virtual_voltage_Id, uint16_t *voltage, uint16_t dpm_level, bool debug); @@ -320,5 +330,8 @@ extern int atomctrl_get_leakage_vddc_base_on_leakage(struct pp_hwmgr *hwmgr, uint16_t virtual_voltage_id, uint16_t efuse_voltage_id); extern int atomctrl_get_leakage_id_from_efuse(struct pp_hwmgr *hwmgr, uint16_t *virtual_voltage_id); + +extern void atomctrl_get_voltage_range(struct pp_hwmgr *hwmgr, uint32_t *max_vddc, + uint32_t *min_vddc); #endif diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomfwctrl.c b/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomfwctrl.c index ad42caac033e..5325661fedff 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomfwctrl.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomfwctrl.c @@ -23,9 +23,9 @@ #include "ppatomfwctrl.h" #include "atomfirmware.h" +#include "atom.h" #include "pp_debug.h" - static const union atom_voltage_object_v4 *pp_atomfwctrl_lookup_voltage_type_v4( const struct atom_voltage_objects_info_v4_1 *voltage_object_info_table, uint8_t voltage_type, uint8_t voltage_mode) @@ -38,35 +38,34 @@ static const union atom_voltage_object_v4 *pp_atomfwctrl_lookup_voltage_type_v4( while (offset < size) { const union atom_voltage_object_v4 *voltage_object = - (const union atom_voltage_object_v4 *)(start + offset); + (const union atom_voltage_object_v4 *)(start + offset); - if (voltage_type == voltage_object->gpio_voltage_obj.header.voltage_type && - voltage_mode == voltage_object->gpio_voltage_obj.header.voltage_mode) - return voltage_object; + if (voltage_type == voltage_object->gpio_voltage_obj.header.voltage_type && + voltage_mode == voltage_object->gpio_voltage_obj.header.voltage_mode) + return voltage_object; - offset += le16_to_cpu(voltage_object->gpio_voltage_obj.header.object_size); + offset += le16_to_cpu(voltage_object->gpio_voltage_obj.header.object_size); - } + } - return NULL; + return NULL; } static struct atom_voltage_objects_info_v4_1 *pp_atomfwctrl_get_voltage_info_table( struct pp_hwmgr *hwmgr) { - const void *table_address; - uint16_t idx; + const void *table_address; + uint16_t idx; - idx = GetIndexIntoMasterDataTable(voltageobject_info); - table_address = cgs_atom_get_data_table(hwmgr->device, - idx, NULL, NULL, NULL); + idx = GetIndexIntoMasterDataTable(voltageobject_info); + table_address = smu_atom_get_data_table(hwmgr->adev, + idx, NULL, NULL, NULL); - PP_ASSERT_WITH_CODE( - table_address, - "Error retrieving BIOS Table Address!", - return NULL); + PP_ASSERT_WITH_CODE(table_address, + "Error retrieving BIOS Table Address!", + return NULL); - return (struct atom_voltage_objects_info_v4_1 *)table_address; + return (struct atom_voltage_objects_info_v4_1 *)table_address; } /** @@ -167,7 +166,7 @@ static struct atom_gpio_pin_lut_v2_1 *pp_atomfwctrl_get_gpio_lookup_table( uint16_t idx; idx = GetIndexIntoMasterDataTable(gpio_pin_lut); - table_address = cgs_atom_get_data_table(hwmgr->device, + table_address = smu_atom_get_data_table(hwmgr->adev, idx, NULL, NULL, NULL); PP_ASSERT_WITH_CODE(table_address, "Error retrieving BIOS Table Address!", @@ -248,28 +247,30 @@ int pp_atomfwctrl_get_gpu_pll_dividers_vega10(struct pp_hwmgr *hwmgr, uint32_t clock_type, uint32_t clock_value, struct pp_atomfwctrl_clock_dividers_soc15 *dividers) { + struct amdgpu_device *adev = hwmgr->adev; struct compute_gpu_clock_input_parameter_v1_8 pll_parameters; struct compute_gpu_clock_output_parameter_v1_8 *pll_output; - int result; uint32_t idx; pll_parameters.gpuclock_10khz = (uint32_t)clock_value; pll_parameters.gpu_clock_type = clock_type; idx = GetIndexIntoMasterCmdTable(computegpuclockparam); - result = cgs_atom_exec_cmd_table(hwmgr->device, idx, &pll_parameters); - - if (!result) { - pll_output = (struct compute_gpu_clock_output_parameter_v1_8 *) - &pll_parameters; - dividers->ulClock = le32_to_cpu(pll_output->gpuclock_10khz); - dividers->ulDid = le32_to_cpu(pll_output->dfs_did); - dividers->ulPll_fb_mult = le32_to_cpu(pll_output->pll_fb_mult); - dividers->ulPll_ss_fbsmult = le32_to_cpu(pll_output->pll_ss_fbsmult); - dividers->usPll_ss_slew_frac = le16_to_cpu(pll_output->pll_ss_slew_frac); - dividers->ucPll_ss_enable = pll_output->pll_ss_enable; - } - return result; + + if (amdgpu_atom_execute_table( + adev->mode_info.atom_context, idx, (uint32_t *)&pll_parameters)) + return -EINVAL; + + pll_output = (struct compute_gpu_clock_output_parameter_v1_8 *) + &pll_parameters; + dividers->ulClock = le32_to_cpu(pll_output->gpuclock_10khz); + dividers->ulDid = le32_to_cpu(pll_output->dfs_did); + dividers->ulPll_fb_mult = le32_to_cpu(pll_output->pll_fb_mult); + dividers->ulPll_ss_fbsmult = le32_to_cpu(pll_output->pll_ss_fbsmult); + dividers->usPll_ss_slew_frac = le16_to_cpu(pll_output->pll_ss_slew_frac); + dividers->ucPll_ss_enable = pll_output->pll_ss_enable; + + return 0; } int pp_atomfwctrl_get_avfs_information(struct pp_hwmgr *hwmgr, @@ -283,7 +284,7 @@ int pp_atomfwctrl_get_avfs_information(struct pp_hwmgr *hwmgr, idx = GetIndexIntoMasterDataTable(asic_profiling_info); profile = (struct atom_asic_profiling_info_v4_1 *) - cgs_atom_get_data_table(hwmgr->device, + smu_atom_get_data_table(hwmgr->adev, idx, NULL, NULL, NULL); if (!profile) @@ -467,7 +468,7 @@ int pp_atomfwctrl_get_gpio_information(struct pp_hwmgr *hwmgr, idx = GetIndexIntoMasterDataTable(smu_info); info = (struct atom_smu_info_v3_1 *) - cgs_atom_get_data_table(hwmgr->device, + smu_atom_get_data_table(hwmgr->adev, idx, NULL, NULL, NULL); if (!info) { @@ -487,23 +488,26 @@ int pp_atomfwctrl_get_gpio_information(struct pp_hwmgr *hwmgr, return 0; } -int pp_atomfwctrl__get_clk_information_by_clkid(struct pp_hwmgr *hwmgr, BIOS_CLKID id, uint32_t *frequency) +int pp_atomfwctrl_get_clk_information_by_clkid(struct pp_hwmgr *hwmgr, BIOS_CLKID id, uint32_t *frequency) { + struct amdgpu_device *adev = hwmgr->adev; struct atom_get_smu_clock_info_parameters_v3_1 parameters; struct atom_get_smu_clock_info_output_parameters_v3_1 *output; uint32_t ix; parameters.clk_id = id; + parameters.syspll_id = 0; parameters.command = GET_SMU_CLOCK_INFO_V3_1_GET_CLOCK_FREQ; + parameters.dfsdid = 0; ix = GetIndexIntoMasterCmdTable(getsmuclockinfo); - if (!cgs_atom_exec_cmd_table(hwmgr->device, ix, ¶meters)) { - output = (struct atom_get_smu_clock_info_output_parameters_v3_1 *)¶meters; - *frequency = output->atom_smu_outputclkfreq.smu_clock_freq_hz / 10000; - } else { - pr_info("Error execute_table getsmuclockinfo!"); - return -1; - } + + if (amdgpu_atom_execute_table( + adev->mode_info.atom_context, ix, (uint32_t *)¶meters)) + return -EINVAL; + + output = (struct atom_get_smu_clock_info_output_parameters_v3_1 *)¶meters; + *frequency = le32_to_cpu(output->atom_smu_outputclkfreq.smu_clock_freq_hz) / 10000; return 0; } @@ -513,11 +517,10 @@ int pp_atomfwctrl_get_vbios_bootup_values(struct pp_hwmgr *hwmgr, { struct atom_firmware_info_v3_1 *info = NULL; uint16_t ix; - uint32_t frequency = 0; ix = GetIndexIntoMasterDataTable(firmwareinfo); info = (struct atom_firmware_info_v3_1 *) - cgs_atom_get_data_table(hwmgr->device, + smu_atom_get_data_table(hwmgr->adev, ix, NULL, NULL, NULL); if (!info) { @@ -536,12 +539,6 @@ int pp_atomfwctrl_get_vbios_bootup_values(struct pp_hwmgr *hwmgr, boot_values->ulSocClk = 0; boot_values->ulDCEFClk = 0; - if (!pp_atomfwctrl__get_clk_information_by_clkid(hwmgr, SMU9_SYSPLL0_SOCCLK_ID, &frequency)) - boot_values->ulSocClk = frequency; - - if (!pp_atomfwctrl__get_clk_information_by_clkid(hwmgr, SMU9_SYSPLL0_DCEFCLK_ID, &frequency)) - boot_values->ulDCEFClk = frequency; - return 0; } @@ -553,7 +550,7 @@ int pp_atomfwctrl_get_smc_dpm_information(struct pp_hwmgr *hwmgr, ix = GetIndexIntoMasterDataTable(smc_dpm_info); info = (struct atom_smc_dpm_info_v4_1 *) - cgs_atom_get_data_table(hwmgr->device, + smu_atom_get_data_table(hwmgr->adev, ix, NULL, NULL, NULL); if (!info) { pr_info("Error retrieving BIOS Table Address!"); diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomfwctrl.h b/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomfwctrl.h index 8df1e84f27c9..fe10aa4db5e6 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomfwctrl.h +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/ppatomfwctrl.h @@ -230,6 +230,8 @@ int pp_atomfwctrl_get_vbios_bootup_values(struct pp_hwmgr *hwmgr, struct pp_atomfwctrl_bios_boot_up_values *boot_values); int pp_atomfwctrl_get_smc_dpm_information(struct pp_hwmgr *hwmgr, struct pp_atomfwctrl_smc_dpm_parameters *param); +int pp_atomfwctrl_get_clk_information_by_clkid(struct pp_hwmgr *hwmgr, + BIOS_CLKID id, uint32_t *frequency); #endif diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/process_pptables_v1_0.c b/drivers/gpu/drm/amd/powerplay/hwmgr/process_pptables_v1_0.c index c9eecce5683f..35bd9870ab10 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/process_pptables_v1_0.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/process_pptables_v1_0.c @@ -141,7 +141,7 @@ static const void *get_powerplay_table(struct pp_hwmgr *hwmgr) if (!table_address) { table_address = (ATOM_Tonga_POWERPLAYTABLE *) - cgs_atom_get_data_table(hwmgr->device, + smu_atom_get_data_table(hwmgr->adev, index, &size, &frev, &crev); hwmgr->soft_pp_table = table_address; /*Cache the result in RAM.*/ hwmgr->soft_pp_table_size = size; @@ -728,6 +728,32 @@ static int get_mm_clock_voltage_table( return 0; } +static int get_gpio_table(struct pp_hwmgr *hwmgr, + struct phm_ppt_v1_gpio_table **pp_tonga_gpio_table, + const ATOM_Tonga_GPIO_Table *atom_gpio_table) +{ + uint32_t table_size; + struct phm_ppt_v1_gpio_table *pp_gpio_table; + struct phm_ppt_v1_information *pp_table_information = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + + table_size = sizeof(struct phm_ppt_v1_gpio_table); + pp_gpio_table = kzalloc(table_size, GFP_KERNEL); + if (!pp_gpio_table) + return -ENOMEM; + + if (pp_table_information->vdd_dep_on_sclk->count < + atom_gpio_table->ucVRHotTriggeredSclkDpmIndex) + PP_ASSERT_WITH_CODE(false, + "SCLK DPM index for VRHot cannot exceed the total sclk level count!",); + else + pp_gpio_table->vrhot_triggered_sclk_dpm_index = + atom_gpio_table->ucVRHotTriggeredSclkDpmIndex; + + *pp_tonga_gpio_table = pp_gpio_table; + + return 0; +} /** * Private Function used during initialization. * Initialize clock voltage dependency @@ -761,11 +787,15 @@ static int init_clock_voltage_dependency( const PPTable_Generic_SubTable_Header *pcie_table = (const PPTable_Generic_SubTable_Header *)(((unsigned long) powerplay_table) + le16_to_cpu(powerplay_table->usPCIETableOffset)); + const ATOM_Tonga_GPIO_Table *gpio_table = + (const ATOM_Tonga_GPIO_Table *)(((unsigned long) powerplay_table) + + le16_to_cpu(powerplay_table->usGPIOTableOffset)); pp_table_information->vdd_dep_on_sclk = NULL; pp_table_information->vdd_dep_on_mclk = NULL; pp_table_information->mm_dep_table = NULL; pp_table_information->pcie_table = NULL; + pp_table_information->gpio_table = NULL; if (powerplay_table->usMMDependencyTableOffset != 0) result = get_mm_clock_voltage_table(hwmgr, @@ -810,6 +840,10 @@ static int init_clock_voltage_dependency( result = get_valid_clk(hwmgr, &pp_table_information->valid_sclk_values, pp_table_information->vdd_dep_on_sclk); + if (!result && gpio_table) + result = get_gpio_table(hwmgr, &pp_table_information->gpio_table, + gpio_table); + return result; } @@ -836,12 +870,6 @@ static int init_over_drive_limits( hwmgr->platform_descriptor.maxOverdriveVDDC = 0; hwmgr->platform_descriptor.overdriveVDDCStep = 0; - if (hwmgr->platform_descriptor.overdriveLimit.engineClock == 0 \ - || hwmgr->platform_descriptor.overdriveLimit.memoryClock == 0) { - hwmgr->od_enabled = false; - pr_debug("OverDrive feature not support by VBIOS\n"); - } - return 0; } @@ -1116,6 +1144,9 @@ static int pp_tables_v1_0_uninitialize(struct pp_hwmgr *hwmgr) kfree(pp_table_information->pcie_table); pp_table_information->pcie_table = NULL; + kfree(pp_table_information->gpio_table); + pp_table_information->gpio_table = NULL; + kfree(hwmgr->pptable); hwmgr->pptable = NULL; diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/processpptables.c b/drivers/gpu/drm/amd/powerplay/hwmgr/processpptables.c index 36ca7c419c90..925e17104f90 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/processpptables.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/processpptables.c @@ -837,7 +837,7 @@ static const ATOM_PPLIB_POWERPLAYTABLE *get_powerplay_table( hwmgr->soft_pp_table = &soft_dummy_pp_table[0]; hwmgr->soft_pp_table_size = sizeof(soft_dummy_pp_table); } else { - table_addr = cgs_atom_get_data_table(hwmgr->device, + table_addr = smu_atom_get_data_table(hwmgr->adev, GetIndexIntoMasterTable(DATA, PowerPlayInfo), &size, &frev, &crev); hwmgr->soft_pp_table = table_addr; @@ -1058,7 +1058,7 @@ static int init_overdrive_limits(struct pp_hwmgr *hwmgr, return 0; /* We assume here that fw_info is unchanged if this call fails.*/ - fw_info = cgs_atom_get_data_table(hwmgr->device, + fw_info = smu_atom_get_data_table(hwmgr->adev, GetIndexIntoMasterTable(DATA, FirmwareInfo), &size, &frev, &crev); @@ -1074,12 +1074,6 @@ static int init_overdrive_limits(struct pp_hwmgr *hwmgr, powerplay_table, (const ATOM_FIRMWARE_INFO_V2_1 *)fw_info); - if (hwmgr->platform_descriptor.overdriveLimit.engineClock == 0 - && hwmgr->platform_descriptor.overdriveLimit.memoryClock == 0) { - hwmgr->od_enabled = false; - pr_debug("OverDrive feature not support by VBIOS\n"); - } - return result; } diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/smu10_hwmgr.c b/drivers/gpu/drm/amd/powerplay/hwmgr/smu10_hwmgr.c index 10253b89b3d8..d4bc83e81389 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/smu10_hwmgr.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/smu10_hwmgr.c @@ -34,7 +34,7 @@ #include "rv_ppsmc.h" #include "smu10_hwmgr.h" #include "power_state.h" -#include "pp_soc15.h" +#include "soc15_common.h" #define SMU10_MAX_DEEPSLEEP_DIVIDER_ID 5 #define SMU10_MINIMUM_ENGINE_CLOCK 800 /* 8Mhz, the low boundary of engine clock allowed on this chip */ @@ -42,12 +42,48 @@ #define SMU10_DISPCLK_BYPASS_THRESHOLD 10000 /* 100Mhz */ #define SMC_RAM_END 0x40000 +#define mmPWR_MISC_CNTL_STATUS 0x0183 +#define mmPWR_MISC_CNTL_STATUS_BASE_IDX 0 +#define PWR_MISC_CNTL_STATUS__PWR_GFX_RLC_CGPG_EN__SHIFT 0x0 +#define PWR_MISC_CNTL_STATUS__PWR_GFXOFF_STATUS__SHIFT 0x1 +#define PWR_MISC_CNTL_STATUS__PWR_GFX_RLC_CGPG_EN_MASK 0x00000001L +#define PWR_MISC_CNTL_STATUS__PWR_GFXOFF_STATUS_MASK 0x00000006L + static const unsigned long SMU10_Magic = (unsigned long) PHM_Rv_Magic; static int smu10_display_clock_voltage_request(struct pp_hwmgr *hwmgr, - struct pp_display_clock_request *clock_req); + struct pp_display_clock_request *clock_req) +{ + struct smu10_hwmgr *smu10_data = (struct smu10_hwmgr *)(hwmgr->backend); + enum amd_pp_clock_type clk_type = clock_req->clock_type; + uint32_t clk_freq = clock_req->clock_freq_in_khz / 1000; + PPSMC_Msg msg; + switch (clk_type) { + case amd_pp_dcf_clock: + if (clk_freq == smu10_data->dcf_actual_hard_min_freq) + return 0; + msg = PPSMC_MSG_SetHardMinDcefclkByFreq; + smu10_data->dcf_actual_hard_min_freq = clk_freq; + break; + case amd_pp_soc_clock: + msg = PPSMC_MSG_SetHardMinSocclkByFreq; + break; + case amd_pp_f_clock: + if (clk_freq == smu10_data->f_actual_hard_min_freq) + return 0; + smu10_data->f_actual_hard_min_freq = clk_freq; + msg = PPSMC_MSG_SetHardMinFclkByFreq; + break; + default: + pr_info("[DisplayClockVoltageRequest]Invalid Clock Type!"); + return -EINVAL; + } + smum_send_msg_to_smc_with_parameter(hwmgr, msg, clk_freq); + + return 0; +} static struct smu10_power_state *cast_smu10_ps(struct pp_hw_power_state *hw_ps) { @@ -74,11 +110,15 @@ static int smu10_initialize_dpm_defaults(struct pp_hwmgr *hwmgr) smu10_data->thermal_auto_throttling_treshold = 0; smu10_data->is_nb_dpm_enabled = 1; smu10_data->dpm_flags = 1; - smu10_data->gfx_off_controled_by_driver = false; smu10_data->need_min_deep_sleep_dcefclk = true; smu10_data->num_active_display = 0; smu10_data->deep_sleep_dcefclk = 0; + if (hwmgr->feature_mask & PP_GFXOFF_MASK) + smu10_data->gfx_off_controled_by_driver = true; + else + smu10_data->gfx_off_controled_by_driver = false; + phm_cap_unset(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_SclkDeepSleep); @@ -161,7 +201,7 @@ static int smu10_set_clock_limit(struct pp_hwmgr *hwmgr, const void *input) struct PP_Clocks clocks = {0}; struct pp_display_clock_request clock_req; - clocks.dcefClock = hwmgr->display_config.min_dcef_set_clk; + clocks.dcefClock = hwmgr->display_config->min_dcef_set_clk; clock_req.clock_type = amd_pp_dcf_clock; clock_req.clock_freq_in_khz = clocks.dcefClock * 10; @@ -206,12 +246,18 @@ static int smu10_set_power_state_tasks(struct pp_hwmgr *hwmgr, const void *input static int smu10_init_power_gate_state(struct pp_hwmgr *hwmgr) { struct smu10_hwmgr *smu10_data = (struct smu10_hwmgr *)(hwmgr->backend); + struct amdgpu_device *adev = hwmgr->adev; smu10_data->vcn_power_gated = true; smu10_data->isp_tileA_power_gated = true; smu10_data->isp_tileB_power_gated = true; - return 0; + if (adev->pg_flags & AMD_PG_SUPPORT_GFX_PG) + return smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_SetGfxCGPG, + true); + else + return 0; } @@ -237,19 +283,37 @@ static int smu10_power_off_asic(struct pp_hwmgr *hwmgr) return smu10_reset_cc6_data(hwmgr); } +static bool smu10_is_gfx_on(struct pp_hwmgr *hwmgr) +{ + uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; + + reg = RREG32_SOC15(PWR, 0, mmPWR_MISC_CNTL_STATUS); + if ((reg & PWR_MISC_CNTL_STATUS__PWR_GFXOFF_STATUS_MASK) == + (0x2 << PWR_MISC_CNTL_STATUS__PWR_GFXOFF_STATUS__SHIFT)) + return true; + + return false; +} + static int smu10_disable_gfx_off(struct pp_hwmgr *hwmgr) { struct smu10_hwmgr *smu10_data = (struct smu10_hwmgr *)(hwmgr->backend); - if (smu10_data->gfx_off_controled_by_driver) + if (smu10_data->gfx_off_controled_by_driver) { smum_send_msg_to_smc(hwmgr, PPSMC_MSG_DisableGfxOff); + /* confirm gfx is back to "on" state */ + while (!smu10_is_gfx_on(hwmgr)) + msleep(1); + } + return 0; } static int smu10_disable_dpm_tasks(struct pp_hwmgr *hwmgr) { - return smu10_disable_gfx_off(hwmgr); + return 0; } static int smu10_enable_gfx_off(struct pp_hwmgr *hwmgr) @@ -264,7 +328,15 @@ static int smu10_enable_gfx_off(struct pp_hwmgr *hwmgr) static int smu10_enable_dpm_tasks(struct pp_hwmgr *hwmgr) { - return smu10_enable_gfx_off(hwmgr); + return 0; +} + +static int smu10_gfx_off_control(struct pp_hwmgr *hwmgr, bool enable) +{ + if (enable) + return smu10_enable_gfx_off(hwmgr); + else + return smu10_disable_gfx_off(hwmgr); } static int smu10_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, @@ -340,7 +412,7 @@ static int smu10_get_clock_voltage_dependency_table(struct pp_hwmgr *hwmgr, static int smu10_populate_clock_table(struct pp_hwmgr *hwmgr) { - int result; + uint32_t result; struct smu10_hwmgr *smu10_data = (struct smu10_hwmgr *)(hwmgr->backend); DpmClocks_t *table = &(smu10_data->clock_table); @@ -386,11 +458,11 @@ static int smu10_populate_clock_table(struct pp_hwmgr *hwmgr) smum_send_msg_to_smc(hwmgr, PPSMC_MSG_GetMinGfxclkFrequency); result = smum_get_argument(hwmgr); - smu10_data->gfx_min_freq_limit = result * 100; + smu10_data->gfx_min_freq_limit = result / 10 * 1000; smum_send_msg_to_smc(hwmgr, PPSMC_MSG_GetMaxGfxclkFrequency); result = smum_get_argument(hwmgr); - smu10_data->gfx_max_freq_limit = result * 100; + smu10_data->gfx_max_freq_limit = result / 10 * 1000; return 0; } @@ -436,8 +508,8 @@ static int smu10_hwmgr_backend_init(struct pp_hwmgr *hwmgr) hwmgr->platform_descriptor.minimumClocksReductionPercentage = 50; - hwmgr->pstate_sclk = SMU10_UMD_PSTATE_GFXCLK; - hwmgr->pstate_mclk = SMU10_UMD_PSTATE_FCLK; + hwmgr->pstate_sclk = SMU10_UMD_PSTATE_GFXCLK * 100; + hwmgr->pstate_mclk = SMU10_UMD_PSTATE_FCLK * 100; return result; } @@ -472,6 +544,8 @@ static int smu10_hwmgr_backend_fini(struct pp_hwmgr *hwmgr) static int smu10_dpm_force_dpm_level(struct pp_hwmgr *hwmgr, enum amd_dpm_forced_level level) { + struct smu10_hwmgr *data = hwmgr->backend; + if (hwmgr->smu_version < 0x1E3700) { pr_info("smu firmware version too old, can not set dpm level\n"); return 0; @@ -482,7 +556,7 @@ static int smu10_dpm_force_dpm_level(struct pp_hwmgr *hwmgr, case AMD_DPM_FORCED_LEVEL_PROFILE_PEAK: smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetHardMinGfxClk, - SMU10_UMD_PSTATE_PEAK_GFXCLK); + data->gfx_max_freq_limit/100); smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetHardMinFclkByFreq, SMU10_UMD_PSTATE_PEAK_FCLK); @@ -495,7 +569,7 @@ static int smu10_dpm_force_dpm_level(struct pp_hwmgr *hwmgr, smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetSoftMaxGfxClk, - SMU10_UMD_PSTATE_PEAK_GFXCLK); + data->gfx_max_freq_limit/100); smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetSoftMaxFclkByFreq, SMU10_UMD_PSTATE_PEAK_FCLK); @@ -509,10 +583,10 @@ static int smu10_dpm_force_dpm_level(struct pp_hwmgr *hwmgr, case AMD_DPM_FORCED_LEVEL_PROFILE_MIN_SCLK: smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetHardMinGfxClk, - SMU10_UMD_PSTATE_MIN_GFXCLK); + data->gfx_min_freq_limit/100); smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetSoftMaxGfxClk, - SMU10_UMD_PSTATE_MIN_GFXCLK); + data->gfx_min_freq_limit/100); break; case AMD_DPM_FORCED_LEVEL_PROFILE_MIN_MCLK: smum_send_msg_to_smc_with_parameter(hwmgr, @@ -552,10 +626,13 @@ static int smu10_dpm_force_dpm_level(struct pp_hwmgr *hwmgr, case AMD_DPM_FORCED_LEVEL_AUTO: smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetHardMinGfxClk, - SMU10_UMD_PSTATE_MIN_GFXCLK); + data->gfx_min_freq_limit/100); smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetHardMinFclkByFreq, + hwmgr->display_config->num_display > 3 ? + SMU10_UMD_PSTATE_PEAK_FCLK : SMU10_UMD_PSTATE_MIN_FCLK); + smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetHardMinSocclkByFreq, SMU10_UMD_PSTATE_MIN_SOCCLK); @@ -565,7 +642,7 @@ static int smu10_dpm_force_dpm_level(struct pp_hwmgr *hwmgr, smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetSoftMaxGfxClk, - SMU10_UMD_PSTATE_PEAK_GFXCLK); + data->gfx_max_freq_limit/100); smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetSoftMaxFclkByFreq, SMU10_UMD_PSTATE_PEAK_FCLK); @@ -579,10 +656,10 @@ static int smu10_dpm_force_dpm_level(struct pp_hwmgr *hwmgr, case AMD_DPM_FORCED_LEVEL_LOW: smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetHardMinGfxClk, - SMU10_UMD_PSTATE_MIN_GFXCLK); + data->gfx_min_freq_limit/100); smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetSoftMaxGfxClk, - SMU10_UMD_PSTATE_MIN_GFXCLK); + data->gfx_min_freq_limit/100); smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetHardMinFclkByFreq, SMU10_UMD_PSTATE_MIN_FCLK); @@ -699,6 +776,16 @@ static int smu10_set_cpu_power_state(struct pp_hwmgr *hwmgr) static int smu10_store_cc6_data(struct pp_hwmgr *hwmgr, uint32_t separation_time, bool cc6_disable, bool pstate_disable, bool pstate_switch_disable) { + struct smu10_hwmgr *data = (struct smu10_hwmgr *)(hwmgr->backend); + + if (separation_time != data->separation_time || + cc6_disable != data->cc6_disable || + pstate_disable != data->pstate_disable) { + data->separation_time = separation_time; + data->cc6_disable = cc6_disable; + data->pstate_disable = pstate_disable; + data->cc6_setting_changed = true; + } return 0; } @@ -711,6 +798,51 @@ static int smu10_get_dal_power_level(struct pp_hwmgr *hwmgr, static int smu10_force_clock_level(struct pp_hwmgr *hwmgr, enum pp_clock_type type, uint32_t mask) { + struct smu10_hwmgr *data = hwmgr->backend; + struct smu10_voltage_dependency_table *mclk_table = + data->clock_vol_info.vdd_dep_on_fclk; + uint32_t low, high; + + low = mask ? (ffs(mask) - 1) : 0; + high = mask ? (fls(mask) - 1) : 0; + + switch (type) { + case PP_SCLK: + if (low > 2 || high > 2) { + pr_info("Currently sclk only support 3 levels on RV\n"); + return -EINVAL; + } + + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_SetHardMinGfxClk, + low == 2 ? data->gfx_max_freq_limit/100 : + low == 1 ? SMU10_UMD_PSTATE_GFXCLK : + data->gfx_min_freq_limit/100); + + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_SetSoftMaxGfxClk, + high == 0 ? data->gfx_min_freq_limit/100 : + high == 1 ? SMU10_UMD_PSTATE_GFXCLK : + data->gfx_max_freq_limit/100); + break; + + case PP_MCLK: + if (low > mclk_table->count - 1 || high > mclk_table->count - 1) + return -EINVAL; + + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_SetHardMinFclkByFreq, + mclk_table->entries[low].clk/100); + + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_SetSoftMaxFclkByFreq, + mclk_table->entries[high].clk/100); + break; + + case PP_PCIE: + default: + break; + } return 0; } @@ -720,21 +852,30 @@ static int smu10_print_clock_levels(struct pp_hwmgr *hwmgr, struct smu10_hwmgr *data = (struct smu10_hwmgr *)(hwmgr->backend); struct smu10_voltage_dependency_table *mclk_table = data->clock_vol_info.vdd_dep_on_fclk; - int i, now, size = 0; + uint32_t i, now, size = 0; switch (type) { case PP_SCLK: smum_send_msg_to_smc(hwmgr, PPSMC_MSG_GetGfxclkFrequency); now = smum_get_argument(hwmgr); + /* driver only know min/max gfx_clk, Add level 1 for all other gfx clks */ + if (now == data->gfx_max_freq_limit/100) + i = 2; + else if (now == data->gfx_min_freq_limit/100) + i = 0; + else + i = 1; + size += sprintf(buf + size, "0: %uMhz %s\n", - data->gfx_min_freq_limit / 100, - ((data->gfx_min_freq_limit / 100) - == now) ? "*" : ""); + data->gfx_min_freq_limit/100, + i == 0 ? "*" : ""); size += sprintf(buf + size, "1: %uMhz %s\n", - data->gfx_max_freq_limit / 100, - ((data->gfx_max_freq_limit / 100) - == now) ? "*" : ""); + i == 1 ? now : SMU10_UMD_PSTATE_GFXCLK, + i == 1 ? "*" : ""); + size += sprintf(buf + size, "2: %uMhz %s\n", + data->gfx_max_freq_limit/100, + i == 2 ? "*" : ""); break; case PP_MCLK: smum_send_msg_to_smc(hwmgr, PPSMC_MSG_GetFclkFrequency); @@ -888,6 +1029,12 @@ static int smu10_get_clock_by_type_with_voltage(struct pp_hwmgr *hwmgr, case amd_pp_soc_clock: pclk_vol_table = pinfo->vdd_dep_on_socclk; break; + case amd_pp_disp_clock: + pclk_vol_table = pinfo->vdd_dep_on_dispclk; + break; + case amd_pp_phy_clock: + pclk_vol_table = pinfo->vdd_dep_on_phyclk; + break; default: return -EINVAL; } @@ -905,39 +1052,7 @@ static int smu10_get_clock_by_type_with_voltage(struct pp_hwmgr *hwmgr, return 0; } -static int smu10_display_clock_voltage_request(struct pp_hwmgr *hwmgr, - struct pp_display_clock_request *clock_req) -{ - struct smu10_hwmgr *smu10_data = (struct smu10_hwmgr *)(hwmgr->backend); - enum amd_pp_clock_type clk_type = clock_req->clock_type; - uint32_t clk_freq = clock_req->clock_freq_in_khz / 1000; - PPSMC_Msg msg; - - switch (clk_type) { - case amd_pp_dcf_clock: - if (clk_freq == smu10_data->dcf_actual_hard_min_freq) - return 0; - msg = PPSMC_MSG_SetHardMinDcefclkByFreq; - smu10_data->dcf_actual_hard_min_freq = clk_freq; - break; - case amd_pp_soc_clock: - msg = PPSMC_MSG_SetHardMinSocclkByFreq; - break; - case amd_pp_f_clock: - if (clk_freq == smu10_data->f_actual_hard_min_freq) - return 0; - smu10_data->f_actual_hard_min_freq = clk_freq; - msg = PPSMC_MSG_SetHardMinFclkByFreq; - break; - default: - pr_info("[DisplayClockVoltageRequest]Invalid Clock Type!"); - return -EINVAL; - } - - smum_send_msg_to_smc_with_parameter(hwmgr, msg, clk_freq); - return 0; -} static int smu10_get_max_high_clocks(struct pp_hwmgr *hwmgr, struct amd_pp_simple_clock_info *clocks) { @@ -947,9 +1062,8 @@ static int smu10_get_max_high_clocks(struct pp_hwmgr *hwmgr, struct amd_pp_simpl static int smu10_thermal_get_temperature(struct pp_hwmgr *hwmgr) { - uint32_t reg_offset = soc15_get_register_offset(THM_HWID, 0, - mmTHM_TCON_CUR_TMP_BASE_IDX, mmTHM_TCON_CUR_TMP); - uint32_t reg_value = cgs_read_register(hwmgr->device, reg_offset); + struct amdgpu_device *adev = hwmgr->adev; + uint32_t reg_value = RREG32_SOC15(THM, 0, mmTHM_TCON_CUR_TMP); int cur_temp = (reg_value & THM_TCON_CUR_TMP__CUR_TEMP_MASK) >> THM_TCON_CUR_TMP__CUR_TEMP__SHIFT; @@ -993,11 +1107,47 @@ static int smu10_read_sensor(struct pp_hwmgr *hwmgr, int idx, return ret; } +static int smu10_set_watermarks_for_clocks_ranges(struct pp_hwmgr *hwmgr, + struct pp_wm_sets_with_clock_ranges_soc15 *wm_with_clock_ranges) +{ + struct smu10_hwmgr *data = hwmgr->backend; + Watermarks_t *table = &(data->water_marks_table); + int result = 0; + + smu_set_watermarks_for_clocks_ranges(table,wm_with_clock_ranges); + smum_smc_table_manager(hwmgr, (uint8_t *)table, (uint16_t)SMU10_WMTABLE, false); + data->water_marks_exist = true; + return result; +} + +static int smu10_smus_notify_pwe(struct pp_hwmgr *hwmgr) +{ + + return smum_send_msg_to_smc(hwmgr, PPSMC_MSG_SetRccPfcPmeRestoreRegister); +} + static int smu10_set_mmhub_powergating_by_smu(struct pp_hwmgr *hwmgr) { return smum_send_msg_to_smc(hwmgr, PPSMC_MSG_PowerGateMmHub); } +static void smu10_powergate_vcn(struct pp_hwmgr *hwmgr, bool bgate) +{ + if (bgate) { + amdgpu_device_ip_set_powergating_state(hwmgr->adev, + AMD_IP_BLOCK_TYPE_VCN, + AMD_PG_STATE_GATE); + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_PowerDownVcn, 0); + } else { + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_PowerUpVcn, 0); + amdgpu_device_ip_set_powergating_state(hwmgr->adev, + AMD_IP_BLOCK_TYPE_VCN, + AMD_PG_STATE_UNGATE); + } +} + static const struct pp_hwmgr_func smu10_hwmgr_funcs = { .backend_init = smu10_hwmgr_backend_init, .backend_fini = smu10_hwmgr_backend_fini, @@ -1006,7 +1156,7 @@ static const struct pp_hwmgr_func smu10_hwmgr_funcs = { .force_dpm_level = smu10_dpm_force_dpm_level, .get_power_state_size = smu10_get_power_state_size, .powerdown_uvd = NULL, - .powergate_uvd = NULL, + .powergate_uvd = smu10_powergate_vcn, .powergate_vce = NULL, .get_mclk = smu10_dpm_get_mclk, .get_sclk = smu10_dpm_get_sclk, @@ -1022,6 +1172,7 @@ static const struct pp_hwmgr_func smu10_hwmgr_funcs = { .get_current_shallow_sleep_clocks = smu10_get_current_shallow_sleep_clocks, .get_clock_by_type_with_latency = smu10_get_clock_by_type_with_latency, .get_clock_by_type_with_voltage = smu10_get_clock_by_type_with_voltage, + .set_watermarks_for_clocks_ranges = smu10_set_watermarks_for_clocks_ranges, .get_max_high_clocks = smu10_get_max_high_clocks, .read_sensor = smu10_read_sensor, .set_active_display_count = smu10_set_active_display_count, @@ -1032,6 +1183,9 @@ static const struct pp_hwmgr_func smu10_hwmgr_funcs = { .power_state_set = smu10_set_power_state_tasks, .dynamic_state_management_disable = smu10_disable_dpm_tasks, .set_mmhub_powergating_by_smu = smu10_set_mmhub_powergating_by_smu, + .smus_notify_pwe = smu10_smus_notify_pwe, + .gfx_off_control = smu10_gfx_off_control, + .display_clock_voltage_request = smu10_display_clock_voltage_request, }; int smu10_init_function_pointers(struct pp_hwmgr *hwmgr) diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/smu10_hwmgr.h b/drivers/gpu/drm/amd/powerplay/hwmgr/smu10_hwmgr.h index 175c3a592b6c..1fb296a996f3 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/smu10_hwmgr.h +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/smu10_hwmgr.h @@ -290,6 +290,7 @@ struct smu10_hwmgr { bool vcn_dpg_mode; bool gfx_off_controled_by_driver; + bool water_marks_exist; Watermarks_t water_marks_table; struct smu10_clock_voltage_information clock_vol_info; DpmClocks_t clock_table; @@ -310,11 +311,9 @@ int smu10_init_function_pointers(struct pp_hwmgr *hwmgr); #define SMU10_UMD_PSTATE_FCLK 933 #define SMU10_UMD_PSTATE_VCE 0x03C00320 -#define SMU10_UMD_PSTATE_PEAK_GFXCLK 1100 #define SMU10_UMD_PSTATE_PEAK_SOCCLK 757 #define SMU10_UMD_PSTATE_PEAK_FCLK 1200 -#define SMU10_UMD_PSTATE_MIN_GFXCLK 200 #define SMU10_UMD_PSTATE_MIN_FCLK 400 #define SMU10_UMD_PSTATE_MIN_SOCCLK 200 #define SMU10_UMD_PSTATE_MIN_VCE 0x0190012C diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_clockpowergating.c b/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_clockpowergating.c index f4cbaee4e2ca..6d72a5600917 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_clockpowergating.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_clockpowergating.c @@ -147,20 +147,20 @@ void smu7_powergate_uvd(struct pp_hwmgr *hwmgr, bool bgate) data->uvd_power_gated = bgate; if (bgate) { - cgs_set_powergating_state(hwmgr->device, + amdgpu_device_ip_set_powergating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_UVD, AMD_PG_STATE_GATE); - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_UVD, AMD_CG_STATE_GATE); smu7_update_uvd_dpm(hwmgr, true); smu7_powerdown_uvd(hwmgr); } else { smu7_powerup_uvd(hwmgr); - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_UVD, AMD_CG_STATE_UNGATE); - cgs_set_powergating_state(hwmgr->device, + amdgpu_device_ip_set_powergating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_UVD, AMD_PG_STATE_UNGATE); smu7_update_uvd_dpm(hwmgr, false); @@ -175,20 +175,20 @@ void smu7_powergate_vce(struct pp_hwmgr *hwmgr, bool bgate) data->vce_power_gated = bgate; if (bgate) { - cgs_set_powergating_state(hwmgr->device, + amdgpu_device_ip_set_powergating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_VCE, AMD_PG_STATE_GATE); - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_VCE, AMD_CG_STATE_GATE); smu7_update_vce_dpm(hwmgr, true); smu7_powerdown_vce(hwmgr); } else { smu7_powerup_vce(hwmgr); - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_VCE, AMD_CG_STATE_UNGATE); - cgs_set_powergating_state(hwmgr->device, + amdgpu_device_ip_set_powergating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_VCE, AMD_PG_STATE_UNGATE); smu7_update_vce_dpm(hwmgr, false); diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_hwmgr.c b/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_hwmgr.c index 18b5b2ff47fe..f8e866ceda02 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_hwmgr.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_hwmgr.c @@ -61,10 +61,6 @@ #define SMC_CG_IND_START 0xc0030000 #define SMC_CG_IND_END 0xc0040000 -#define VOLTAGE_SCALE 4 -#define VOLTAGE_VID_OFFSET_SCALE1 625 -#define VOLTAGE_VID_OFFSET_SCALE2 100 - #define MEM_FREQ_LOW_LATENCY 25000 #define MEM_FREQ_HIGH_LATENCY 80000 @@ -88,6 +84,14 @@ static const struct profile_mode_setting smu7_profiling[6] = {0, 0, 0, 0, 0, 0, 0, 0}, }; +#define PPSMC_MSG_SetVBITimeout_VEGAM ((uint16_t) 0x310) + +#define ixPWR_SVI2_PLANE1_LOAD 0xC0200280 +#define PWR_SVI2_PLANE1_LOAD__PSI1_MASK 0x00000020L +#define PWR_SVI2_PLANE1_LOAD__PSI0_EN_MASK 0x00000040L +#define PWR_SVI2_PLANE1_LOAD__PSI1__SHIFT 0x00000005 +#define PWR_SVI2_PLANE1_LOAD__PSI0_EN__SHIFT 0x00000006 + /** Values for the CG_THERMAL_CTRL::DPM_EVENT_SRC field. */ enum DPM_EVENT_SRC { DPM_EVENT_SRC_ANALOG = 0, @@ -169,6 +173,13 @@ static int smu7_get_current_pcie_lane_number(struct pp_hwmgr *hwmgr) */ static int smu7_enable_smc_voltage_controller(struct pp_hwmgr *hwmgr) { + if (hwmgr->chip_id == CHIP_VEGAM) { + PHM_WRITE_VFPF_INDIRECT_FIELD(hwmgr->device, + CGS_IND_REG__SMC, PWR_SVI2_PLANE1_LOAD, PSI1, 0); + PHM_WRITE_VFPF_INDIRECT_FIELD(hwmgr->device, + CGS_IND_REG__SMC, PWR_SVI2_PLANE1_LOAD, PSI0_EN, 0); + } + if (hwmgr->feature_mask & PP_SMC_VOLTAGE_CONTROL_MASK) smum_send_msg_to_smc(hwmgr, PPSMC_MSG_Voltage_Cntl_Enable); @@ -780,7 +791,8 @@ static int smu7_setup_dpm_tables_v1(struct pp_hwmgr *hwmgr) data->dpm_table.sclk_table.count++; } } - + if (hwmgr->platform_descriptor.overdriveLimit.engineClock == 0) + hwmgr->platform_descriptor.overdriveLimit.engineClock = dep_sclk_table->entries[i-1].clk; /* Initialize Mclk DPM table based on allow Mclk values */ data->dpm_table.mclk_table.count = 0; for (i = 0; i < dep_mclk_table->count; i++) { @@ -795,32 +807,8 @@ static int smu7_setup_dpm_tables_v1(struct pp_hwmgr *hwmgr) } } - return 0; -} - -static int smu7_get_voltage_dependency_table( - const struct phm_ppt_v1_clock_voltage_dependency_table *allowed_dep_table, - struct phm_ppt_v1_clock_voltage_dependency_table *dep_table) -{ - uint8_t i = 0; - PP_ASSERT_WITH_CODE((0 != allowed_dep_table->count), - "Voltage Lookup Table empty", - return -EINVAL); - - dep_table->count = allowed_dep_table->count; - for (i=0; icount; i++) { - dep_table->entries[i].clk = allowed_dep_table->entries[i].clk; - dep_table->entries[i].vddInd = allowed_dep_table->entries[i].vddInd; - dep_table->entries[i].vdd_offset = allowed_dep_table->entries[i].vdd_offset; - dep_table->entries[i].vddc = allowed_dep_table->entries[i].vddc; - dep_table->entries[i].vddgfx = allowed_dep_table->entries[i].vddgfx; - dep_table->entries[i].vddci = allowed_dep_table->entries[i].vddci; - dep_table->entries[i].mvdd = allowed_dep_table->entries[i].mvdd; - dep_table->entries[i].phases = allowed_dep_table->entries[i].phases; - dep_table->entries[i].cks_enable = allowed_dep_table->entries[i].cks_enable; - dep_table->entries[i].cks_voffset = allowed_dep_table->entries[i].cks_voffset; - } - + if (hwmgr->platform_descriptor.overdriveLimit.memoryClock == 0) + hwmgr->platform_descriptor.overdriveLimit.memoryClock = dep_mclk_table->entries[i-1].clk; return 0; } @@ -851,7 +839,7 @@ static int smu7_odn_initial_default_setting(struct pp_hwmgr *hwmgr) entries[i].vddc = dep_sclk_table->entries[i].vddc; } - smu7_get_voltage_dependency_table(dep_sclk_table, + smu_get_voltage_dependency_table_ppt_v1(dep_sclk_table, (struct phm_ppt_v1_clock_voltage_dependency_table *)&(odn_table->vdd_dependency_on_sclk)); odn_table->odn_memory_clock_dpm_levels.num_of_pl = @@ -863,12 +851,40 @@ static int smu7_odn_initial_default_setting(struct pp_hwmgr *hwmgr) entries[i].vddc = dep_mclk_table->entries[i].vddc; } - smu7_get_voltage_dependency_table(dep_mclk_table, + smu_get_voltage_dependency_table_ppt_v1(dep_mclk_table, (struct phm_ppt_v1_clock_voltage_dependency_table *)&(odn_table->vdd_dependency_on_mclk)); return 0; } +static void smu7_setup_voltage_range_from_vbios(struct pp_hwmgr *hwmgr) +{ + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + struct phm_ppt_v1_clock_voltage_dependency_table *dep_sclk_table; + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + uint32_t min_vddc = 0; + uint32_t max_vddc = 0; + + if (!table_info) + return; + + dep_sclk_table = table_info->vdd_dep_on_sclk; + + atomctrl_get_voltage_range(hwmgr, &max_vddc, &min_vddc); + + if (min_vddc == 0 || min_vddc > 2000 + || min_vddc > dep_sclk_table->entries[0].vddc) + min_vddc = dep_sclk_table->entries[0].vddc; + + if (max_vddc == 0 || max_vddc > 2000 + || max_vddc < dep_sclk_table->entries[dep_sclk_table->count-1].vddc) + max_vddc = dep_sclk_table->entries[dep_sclk_table->count-1].vddc; + + data->odn_dpm_table.min_vddc = min_vddc; + data->odn_dpm_table.max_vddc = max_vddc; +} + static int smu7_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) { struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); @@ -887,8 +903,10 @@ static int smu7_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) sizeof(struct smu7_dpm_table)); /* initialize ODN table */ - if (hwmgr->od_enabled) + if (hwmgr->od_enabled) { + smu7_setup_voltage_range_from_vbios(hwmgr); smu7_odn_initial_default_setting(hwmgr); + } return 0; } @@ -966,6 +984,22 @@ static int smu7_disable_deep_sleep_master_switch(struct pp_hwmgr *hwmgr) return 0; } +static int smu7_disable_sclk_vce_handshake(struct pp_hwmgr *hwmgr) +{ + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + uint32_t soft_register_value = 0; + uint32_t handshake_disables_offset = data->soft_regs_start + + smum_get_offsetof(hwmgr, + SMU_SoftRegisters, HandshakeDisables); + + soft_register_value = cgs_read_ind_register(hwmgr->device, + CGS_IND_REG__SMC, handshake_disables_offset); + soft_register_value |= SMU7_VCE_SCLK_HANDSHAKE_DISABLE; + cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, + handshake_disables_offset, soft_register_value); + return 0; +} + static int smu7_disable_handshake_uvd(struct pp_hwmgr *hwmgr) { struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); @@ -988,23 +1022,29 @@ static int smu7_enable_sclk_mclk_dpm(struct pp_hwmgr *hwmgr) struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); /* enable SCLK dpm */ - if (!data->sclk_dpm_key_disabled) + if (!data->sclk_dpm_key_disabled) { + if (hwmgr->chip_id == CHIP_VEGAM) + smu7_disable_sclk_vce_handshake(hwmgr); + PP_ASSERT_WITH_CODE( (0 == smum_send_msg_to_smc(hwmgr, PPSMC_MSG_DPM_Enable)), "Failed to enable SCLK DPM during DPM Start Function!", return -EINVAL); + } /* enable MCLK dpm */ if (0 == data->mclk_dpm_key_disabled) { if (!(hwmgr->feature_mask & PP_UVD_HANDSHAKE_MASK)) smu7_disable_handshake_uvd(hwmgr); + PP_ASSERT_WITH_CODE( (0 == smum_send_msg_to_smc(hwmgr, PPSMC_MSG_MCLKDPM_Enable)), "Failed to enable MCLK DPM during DPM Start Function!", return -EINVAL); - PHM_WRITE_FIELD(hwmgr->device, MC_SEQ_CNTL_3, CAC_EN, 0x1); + if (hwmgr->chip_family != CHIP_VEGAM) + PHM_WRITE_FIELD(hwmgr->device, MC_SEQ_CNTL_3, CAC_EN, 0x1); if (hwmgr->chip_family == AMDGPU_FAMILY_CI) { @@ -1020,8 +1060,13 @@ static int smu7_enable_sclk_mclk_dpm(struct pp_hwmgr *hwmgr) cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixLCAC_MC1_CNTL, 0x5); cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixLCAC_CPL_CNTL, 0x100005); udelay(10); - cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixLCAC_MC0_CNTL, 0x400005); - cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixLCAC_MC1_CNTL, 0x400005); + if (hwmgr->chip_id == CHIP_VEGAM) { + cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixLCAC_MC0_CNTL, 0x400009); + cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixLCAC_MC1_CNTL, 0x400009); + } else { + cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixLCAC_MC0_CNTL, 0x400005); + cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixLCAC_MC1_CNTL, 0x400005); + } cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixLCAC_CPL_CNTL, 0x500005); } } @@ -1230,7 +1275,7 @@ static int smu7_enable_dpm_tasks(struct pp_hwmgr *hwmgr) tmp_result = smu7_construct_voltage_tables(hwmgr); PP_ASSERT_WITH_CODE((0 == tmp_result), - "Failed to contruct voltage tables!", + "Failed to construct voltage tables!", result = tmp_result); } smum_initialize_mc_reg_table(hwmgr); @@ -1262,10 +1307,12 @@ static int smu7_enable_dpm_tasks(struct pp_hwmgr *hwmgr) PP_ASSERT_WITH_CODE((0 == tmp_result), "Failed to process firmware header!", result = tmp_result); - tmp_result = smu7_initial_switch_from_arbf0_to_f1(hwmgr); - PP_ASSERT_WITH_CODE((0 == tmp_result), - "Failed to initialize switch from ArbF0 to F1!", - result = tmp_result); + if (hwmgr->chip_id != CHIP_VEGAM) { + tmp_result = smu7_initial_switch_from_arbf0_to_f1(hwmgr); + PP_ASSERT_WITH_CODE((0 == tmp_result), + "Failed to initialize switch from ArbF0 to F1!", + result = tmp_result); + } result = smu7_setup_default_dpm_tables(hwmgr); PP_ASSERT_WITH_CODE(0 == result, @@ -2755,6 +2802,9 @@ static int smu7_vblank_too_short(struct pp_hwmgr *hwmgr, case CHIP_POLARIS12: switch_limit_us = data->is_memory_gddr5 ? 190 : 150; break; + case CHIP_VEGAM: + switch_limit_us = 30; + break; default: switch_limit_us = data->is_memory_gddr5 ? 450 : 150; break; @@ -2778,8 +2828,6 @@ static int smu7_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, struct PP_Clocks minimum_clocks = {0}; bool disable_mclk_switching; bool disable_mclk_switching_for_frame_lock; - struct cgs_display_info info = {0}; - struct cgs_mode_info mode_info = {0}; const struct phm_clock_and_voltage_limits *max_limits; uint32_t i; struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); @@ -2788,7 +2836,6 @@ static int smu7_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, int32_t count; int32_t stable_pstate_sclk = 0, stable_pstate_mclk = 0; - info.mode_info = &mode_info; data->battery_state = (PP_StateUILabel_Battery == request_ps->classification.ui_label); @@ -2810,10 +2857,8 @@ static int smu7_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, } } - cgs_get_active_displays_info(hwmgr->device, &info); - - minimum_clocks.engineClock = hwmgr->display_config.min_core_set_clock; - minimum_clocks.memoryClock = hwmgr->display_config.min_mem_set_clock; + minimum_clocks.engineClock = hwmgr->display_config->min_core_set_clock; + minimum_clocks.memoryClock = hwmgr->display_config->min_mem_set_clock; if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_StablePState)) { @@ -2844,12 +2889,12 @@ static int smu7_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, PHM_PlatformCaps_DisableMclkSwitchingForFrameLock); - if (info.display_count == 0) + if (hwmgr->display_config->num_display == 0) disable_mclk_switching = false; else - disable_mclk_switching = ((1 < info.display_count) || + disable_mclk_switching = ((1 < hwmgr->display_config->num_display) || disable_mclk_switching_for_frame_lock || - smu7_vblank_too_short(hwmgr, mode_info.vblank_time_us)); + smu7_vblank_too_short(hwmgr, hwmgr->display_config->min_vblank_time)); sclk = smu7_ps->performance_levels[0].engine_clock; mclk = smu7_ps->performance_levels[0].memory_clock; @@ -2958,8 +3003,7 @@ static int smu7_dpm_patch_boot_state(struct pp_hwmgr *hwmgr, /* First retrieve the Boot clocks and VDDC from the firmware info table. * We assume here that fw_info is unchanged if this call fails. */ - fw_info = (ATOM_FIRMWARE_INFO_V2_2 *)cgs_atom_get_data_table( - hwmgr->device, index, + fw_info = (ATOM_FIRMWARE_INFO_V2_2 *)smu_atom_get_data_table(hwmgr->adev, index, &size, &frev, &crev); if (!fw_info) /* During a test, there is no firmware info table. */ @@ -3367,34 +3411,35 @@ static int smu7_get_pp_table_entry(struct pp_hwmgr *hwmgr, return 0; } -static int smu7_get_gpu_power(struct pp_hwmgr *hwmgr, - struct pp_gpu_power *query) +static int smu7_get_gpu_power(struct pp_hwmgr *hwmgr, u32 *query) { - PP_ASSERT_WITH_CODE(!smum_send_msg_to_smc(hwmgr, - PPSMC_MSG_PmStatusLogStart), - "Failed to start pm status log!", - return -1); + int i; + u32 tmp = 0; - /* Sampling period from 50ms to 4sec */ - msleep_interruptible(200); + if (!query) + return -EINVAL; - PP_ASSERT_WITH_CODE(!smum_send_msg_to_smc(hwmgr, - PPSMC_MSG_PmStatusLogSample), - "Failed to sample pm status log!", - return -1); + smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_GetCurrPkgPwr, 0); + tmp = cgs_read_register(hwmgr->device, mmSMC_MSG_ARG_0); + *query = tmp; - query->vddc_power = cgs_read_ind_register(hwmgr->device, - CGS_IND_REG__SMC, - ixSMU_PM_STATUS_40); - query->vddci_power = cgs_read_ind_register(hwmgr->device, - CGS_IND_REG__SMC, - ixSMU_PM_STATUS_49); - query->max_gpu_power = cgs_read_ind_register(hwmgr->device, - CGS_IND_REG__SMC, - ixSMU_PM_STATUS_94); - query->average_gpu_power = cgs_read_ind_register(hwmgr->device, - CGS_IND_REG__SMC, - ixSMU_PM_STATUS_95); + if (tmp != 0) + return 0; + + smum_send_msg_to_smc(hwmgr, PPSMC_MSG_PmStatusLogStart); + cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, + ixSMU_PM_STATUS_94, 0); + + for (i = 0; i < 10; i++) { + mdelay(1); + smum_send_msg_to_smc(hwmgr, PPSMC_MSG_PmStatusLogSample); + tmp = cgs_read_ind_register(hwmgr->device, + CGS_IND_REG__SMC, + ixSMU_PM_STATUS_94); + if (tmp != 0) + break; + } + *query = tmp; return 0; } @@ -3447,10 +3492,7 @@ static int smu7_read_sensor(struct pp_hwmgr *hwmgr, int idx, *size = 4; return 0; case AMDGPU_PP_SENSOR_GPU_POWER: - if (*size < sizeof(struct pp_gpu_power)) - return -EINVAL; - *size = sizeof(struct pp_gpu_power); - return smu7_get_gpu_power(hwmgr, (struct pp_gpu_power *)value); + return smu7_get_gpu_power(hwmgr, (uint32_t *)value); case AMDGPU_PP_SENSOR_VDDGFX: if ((data->vr_config & 0xff) == 0x2) val_vid = PHM_READ_INDIRECT_FIELD(hwmgr->device, @@ -3481,7 +3523,6 @@ static int smu7_find_dpm_states_clocks_in_dpm_table(struct pp_hwmgr *hwmgr, cons [smu7_ps->performance_level_count - 1].memory_clock; struct PP_Clocks min_clocks = {0}; uint32_t i; - struct cgs_display_info info = {0}; for (i = 0; i < sclk_table->count; i++) { if (sclk == sclk_table->dpm_levels[i].value) @@ -3508,9 +3549,8 @@ static int smu7_find_dpm_states_clocks_in_dpm_table(struct pp_hwmgr *hwmgr, cons if (i >= mclk_table->count) data->need_update_smu7_dpm_table |= DPMTABLE_OD_UPDATE_MCLK; - cgs_get_active_displays_info(hwmgr->device, &info); - if (data->display_timing.num_existing_displays != info.display_count) + if (data->display_timing.num_existing_displays != hwmgr->display_config->num_display) data->need_update_smu7_dpm_table |= DPMTABLE_UPDATE_MCLK; return 0; @@ -3715,14 +3755,17 @@ static int smu7_trim_dpm_states(struct pp_hwmgr *hwmgr, static int smu7_generate_dpm_level_enable_mask( struct pp_hwmgr *hwmgr, const void *input) { - int result; + int result = 0; const struct phm_set_power_state_input *states = (const struct phm_set_power_state_input *)input; struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); const struct smu7_power_state *smu7_ps = cast_const_phw_smu7_power_state(states->pnew_state); - result = smu7_trim_dpm_states(hwmgr, smu7_ps); + /*skip the trim if od is enabled*/ + if (!hwmgr->od_enabled) + result = smu7_trim_dpm_states(hwmgr, smu7_ps); + if (result) return result; @@ -3813,9 +3856,14 @@ static int smu7_notify_smc_display(struct pp_hwmgr *hwmgr) { struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); - if (hwmgr->feature_mask & PP_VBI_TIME_SUPPORT_MASK) - smum_send_msg_to_smc_with_parameter(hwmgr, - (PPSMC_Msg)PPSMC_MSG_SetVBITimeout, data->frame_time_x2); + if (hwmgr->feature_mask & PP_VBI_TIME_SUPPORT_MASK) { + if (hwmgr->chip_id == CHIP_VEGAM) + smum_send_msg_to_smc_with_parameter(hwmgr, + (PPSMC_Msg)PPSMC_MSG_SetVBITimeout_VEGAM, data->frame_time_x2); + else + smum_send_msg_to_smc_with_parameter(hwmgr, + (PPSMC_Msg)PPSMC_MSG_SetVBITimeout, data->frame_time_x2); + } return (smum_send_msg_to_smc(hwmgr, (PPSMC_Msg)PPSMC_HasDisplay) == 0) ? 0 : -EINVAL; } @@ -3909,15 +3957,8 @@ smu7_notify_smc_display_change(struct pp_hwmgr *hwmgr, bool has_display) static int smu7_notify_smc_display_config_after_ps_adjustment(struct pp_hwmgr *hwmgr) { - uint32_t num_active_displays = 0; - struct cgs_display_info info = {0}; - - info.mode_info = NULL; - cgs_get_active_displays_info(hwmgr->device, &info); - - num_active_displays = info.display_count; - - if (num_active_displays > 1 && hwmgr->display_config.multi_monitor_in_sync != true) + if (hwmgr->display_config->num_display > 1 && + !hwmgr->display_config->multi_monitor_in_sync) smu7_notify_smc_display_change(hwmgr, false); return 0; @@ -3932,33 +3973,24 @@ smu7_notify_smc_display_config_after_ps_adjustment(struct pp_hwmgr *hwmgr) static int smu7_program_display_gap(struct pp_hwmgr *hwmgr) { struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); - uint32_t num_active_displays = 0; uint32_t display_gap = cgs_read_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixCG_DISPLAY_GAP_CNTL); uint32_t display_gap2; uint32_t pre_vbi_time_in_us; uint32_t frame_time_in_us; - uint32_t ref_clock; - uint32_t refresh_rate = 0; - struct cgs_display_info info = {0}; - struct cgs_mode_info mode_info = {0}; - - info.mode_info = &mode_info; - cgs_get_active_displays_info(hwmgr->device, &info); - num_active_displays = info.display_count; + uint32_t ref_clock, refresh_rate; - display_gap = PHM_SET_FIELD(display_gap, CG_DISPLAY_GAP_CNTL, DISP_GAP, (num_active_displays > 0) ? DISPLAY_GAP_VBLANK_OR_WM : DISPLAY_GAP_IGNORE); + display_gap = PHM_SET_FIELD(display_gap, CG_DISPLAY_GAP_CNTL, DISP_GAP, (hwmgr->display_config->num_display > 0) ? DISPLAY_GAP_VBLANK_OR_WM : DISPLAY_GAP_IGNORE); cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixCG_DISPLAY_GAP_CNTL, display_gap); ref_clock = amdgpu_asic_get_xclk((struct amdgpu_device *)hwmgr->adev); - - refresh_rate = mode_info.refresh_rate; + refresh_rate = hwmgr->display_config->vrefresh; if (0 == refresh_rate) refresh_rate = 60; frame_time_in_us = 1000000 / refresh_rate; - pre_vbi_time_in_us = frame_time_in_us - 200 - mode_info.vblank_time_us; + pre_vbi_time_in_us = frame_time_in_us - 200 - hwmgr->display_config->min_vblank_time; data->frame_time_x2 = frame_time_in_us * 2 / 100; @@ -4038,17 +4070,14 @@ smu7_check_smc_update_required_for_display_configuration(struct pp_hwmgr *hwmgr) { struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); bool is_update_required = false; - struct cgs_display_info info = {0, 0, NULL}; - cgs_get_active_displays_info(hwmgr->device, &info); - - if (data->display_timing.num_existing_displays != info.display_count) + if (data->display_timing.num_existing_displays != hwmgr->display_config->num_display) is_update_required = true; if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_SclkDeepSleep)) { - if (data->display_timing.min_clock_in_sr != hwmgr->display_config.min_core_set_clock_in_sr && + if (data->display_timing.min_clock_in_sr != hwmgr->display_config->min_core_set_clock_in_sr && (data->display_timing.min_clock_in_sr >= SMU7_MINIMUM_ENGINE_CLOCK || - hwmgr->display_config.min_core_set_clock_in_sr >= SMU7_MINIMUM_ENGINE_CLOCK)) + hwmgr->display_config->min_core_set_clock_in_sr >= SMU7_MINIMUM_ENGINE_CLOCK)) is_update_required = true; } return is_update_required; @@ -4103,7 +4132,7 @@ static int smu7_check_states_equal(struct pp_hwmgr *hwmgr, return 0; } -static int smu7_upload_mc_firmware(struct pp_hwmgr *hwmgr) +static int smu7_check_mc_firmware(struct pp_hwmgr *hwmgr) { struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); @@ -4182,13 +4211,9 @@ static int smu7_read_clock_registers(struct pp_hwmgr *hwmgr) static int smu7_get_memory_type(struct pp_hwmgr *hwmgr) { struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); - uint32_t temp; - - temp = cgs_read_register(hwmgr->device, mmMC_SEQ_MISC0); + struct amdgpu_device *adev = hwmgr->adev; - data->is_memory_gddr5 = (MC_SEQ_MISC0_GDDR5_VALUE == - ((temp & MC_SEQ_MISC0_GDDR5_MASK) >> - MC_SEQ_MISC0_GDDR5_SHIFT)); + data->is_memory_gddr5 = (adev->gmc.vram_type == AMDGPU_VRAM_TYPE_GDDR5); return 0; } @@ -4236,7 +4261,7 @@ static int smu7_setup_asic_task(struct pp_hwmgr *hwmgr) { int tmp_result, result = 0; - smu7_upload_mc_firmware(hwmgr); + smu7_check_mc_firmware(hwmgr); tmp_result = smu7_read_clock_registers(hwmgr); PP_ASSERT_WITH_CODE((0 == tmp_result), @@ -4371,22 +4396,36 @@ static int smu7_print_clock_levels(struct pp_hwmgr *hwmgr, break; case OD_SCLK: if (hwmgr->od_enabled) { - size = sprintf(buf, "%s: \n", "OD_SCLK"); + size = sprintf(buf, "%s:\n", "OD_SCLK"); for (i = 0; i < odn_sclk_table->num_of_pl; i++) - size += sprintf(buf + size, "%d: %10uMhz %10u mV\n", - i, odn_sclk_table->entries[i].clock / 100, + size += sprintf(buf + size, "%d: %10uMHz %10umV\n", + i, odn_sclk_table->entries[i].clock/100, odn_sclk_table->entries[i].vddc); } break; case OD_MCLK: if (hwmgr->od_enabled) { - size = sprintf(buf, "%s: \n", "OD_MCLK"); + size = sprintf(buf, "%s:\n", "OD_MCLK"); for (i = 0; i < odn_mclk_table->num_of_pl; i++) - size += sprintf(buf + size, "%d: %10uMhz %10u mV\n", - i, odn_mclk_table->entries[i].clock / 100, + size += sprintf(buf + size, "%d: %10uMHz %10umV\n", + i, odn_mclk_table->entries[i].clock/100, odn_mclk_table->entries[i].vddc); } break; + case OD_RANGE: + if (hwmgr->od_enabled) { + size = sprintf(buf, "%s:\n", "OD_RANGE"); + size += sprintf(buf + size, "SCLK: %7uMHz %10uMHz\n", + data->golden_dpm_table.sclk_table.dpm_levels[0].value/100, + hwmgr->platform_descriptor.overdriveLimit.engineClock/100); + size += sprintf(buf + size, "MCLK: %7uMHz %10uMHz\n", + data->golden_dpm_table.mclk_table.dpm_levels[0].value/100, + hwmgr->platform_descriptor.overdriveLimit.memoryClock/100); + size += sprintf(buf + size, "VDDC: %7umV %11umV\n", + data->odn_dpm_table.min_vddc, + data->odn_dpm_table.max_vddc); + } + break; default: break; } @@ -4670,36 +4709,27 @@ static bool smu7_check_clk_voltage_valid(struct pp_hwmgr *hwmgr, { struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); - struct phm_ppt_v1_information *table_info = - (struct phm_ppt_v1_information *)(hwmgr->pptable); - uint32_t min_vddc; - struct phm_ppt_v1_clock_voltage_dependency_table *dep_sclk_table; - - if (table_info == NULL) - return false; - - dep_sclk_table = table_info->vdd_dep_on_sclk; - min_vddc = dep_sclk_table->entries[0].vddc; - - if (voltage < min_vddc || voltage > 2000) { - pr_info("OD voltage is out of range [%d - 2000] mV\n", min_vddc); + if (voltage < data->odn_dpm_table.min_vddc || voltage > data->odn_dpm_table.max_vddc) { + pr_info("OD voltage is out of range [%d - %d] mV\n", + data->odn_dpm_table.min_vddc, + data->odn_dpm_table.max_vddc); return false; } if (type == PP_OD_EDIT_SCLK_VDDC_TABLE) { - if (data->vbios_boot_state.sclk_bootup_value > clk || + if (data->golden_dpm_table.sclk_table.dpm_levels[0].value > clk || hwmgr->platform_descriptor.overdriveLimit.engineClock < clk) { pr_info("OD engine clock is out of range [%d - %d] MHz\n", - data->vbios_boot_state.sclk_bootup_value, - hwmgr->platform_descriptor.overdriveLimit.engineClock / 100); + data->golden_dpm_table.sclk_table.dpm_levels[0].value/100, + hwmgr->platform_descriptor.overdriveLimit.engineClock/100); return false; } } else if (type == PP_OD_EDIT_MCLK_VDDC_TABLE) { - if (data->vbios_boot_state.mclk_bootup_value > clk || + if (data->golden_dpm_table.mclk_table.dpm_levels[0].value > clk || hwmgr->platform_descriptor.overdriveLimit.memoryClock < clk) { pr_info("OD memory clock is out of range [%d - %d] MHz\n", - data->vbios_boot_state.mclk_bootup_value/100, - hwmgr->platform_descriptor.overdriveLimit.memoryClock / 100); + data->golden_dpm_table.mclk_table.dpm_levels[0].value/100, + hwmgr->platform_descriptor.overdriveLimit.memoryClock/100); return false; } } else { @@ -4748,10 +4778,6 @@ static void smu7_check_dpm_table_updated(struct pp_hwmgr *hwmgr) return; } } - if (i == dep_table->count && data->need_update_smu7_dpm_table & DPMTABLE_OD_UPDATE_VDDC) { - data->need_update_smu7_dpm_table &= ~DPMTABLE_OD_UPDATE_VDDC; - data->need_update_smu7_dpm_table |= DPMTABLE_OD_UPDATE_MCLK; - } dep_table = table_info->vdd_dep_on_sclk; odn_dep_table = (struct phm_ppt_v1_clock_voltage_dependency_table *)&(odn_table->vdd_dependency_on_sclk); @@ -4761,9 +4787,9 @@ static void smu7_check_dpm_table_updated(struct pp_hwmgr *hwmgr) return; } } - if (i == dep_table->count && data->need_update_smu7_dpm_table & DPMTABLE_OD_UPDATE_VDDC) { + if (data->need_update_smu7_dpm_table & DPMTABLE_OD_UPDATE_VDDC) { data->need_update_smu7_dpm_table &= ~DPMTABLE_OD_UPDATE_VDDC; - data->need_update_smu7_dpm_table |= DPMTABLE_OD_UPDATE_SCLK; + data->need_update_smu7_dpm_table |= DPMTABLE_OD_UPDATE_SCLK | DPMTABLE_OD_UPDATE_MCLK; } } diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_hwmgr.h b/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_hwmgr.h index b8d0bb378595..c91e75db6a8e 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_hwmgr.h +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_hwmgr.h @@ -184,6 +184,8 @@ struct smu7_odn_dpm_table { struct smu7_odn_clock_voltage_dependency_table vdd_dependency_on_sclk; struct smu7_odn_clock_voltage_dependency_table vdd_dependency_on_mclk; uint32_t odn_mclk_min_limit; + uint32_t min_vddc; + uint32_t max_vddc; }; struct profile_mode_setting { diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_powertune.c b/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_powertune.c index d9e92e306535..c952845833d7 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_powertune.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/smu7_powertune.c @@ -623,6 +623,190 @@ static const struct gpu_pt_config_reg DIDTConfig_Polaris11_Kicker[] = { 0xFFFFFFFF } /* End of list */ }; +static const struct gpu_pt_config_reg GCCACConfig_VegaM[] = +{ +// --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +// Offset Mask Shift Value Type +// --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + // DIDT_SQ + // + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x00060013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x00860013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x01060013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x01860013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x02060013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x02860013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x03060013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x03860013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x04060013, GPU_CONFIGREG_GC_CAC_IND }, + + // DIDT_TD + // + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x000E0013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x008E0013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x010E0013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x018E0013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x020E0013, GPU_CONFIGREG_GC_CAC_IND }, + + // DIDT_TCP + // + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x00100013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x00900013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x01100013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x01900013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x02100013, GPU_CONFIGREG_GC_CAC_IND }, + { ixGC_CAC_CNTL, 0xFFFFFFFF, 0, 0x02900013, GPU_CONFIGREG_GC_CAC_IND }, + + { 0xFFFFFFFF } // End of list +}; + +static const struct gpu_pt_config_reg DIDTConfig_VegaM[] = +{ +// --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +// Offset Mask Shift Value Type +// --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + // DIDT_SQ + // + { ixDIDT_SQ_WEIGHT0_3, DIDT_SQ_WEIGHT0_3__WEIGHT0_MASK, DIDT_SQ_WEIGHT0_3__WEIGHT0__SHIFT, 0x0073, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_WEIGHT0_3, DIDT_SQ_WEIGHT0_3__WEIGHT1_MASK, DIDT_SQ_WEIGHT0_3__WEIGHT1__SHIFT, 0x00ab, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_WEIGHT0_3, DIDT_SQ_WEIGHT0_3__WEIGHT2_MASK, DIDT_SQ_WEIGHT0_3__WEIGHT2__SHIFT, 0x0084, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_WEIGHT0_3, DIDT_SQ_WEIGHT0_3__WEIGHT3_MASK, DIDT_SQ_WEIGHT0_3__WEIGHT3__SHIFT, 0x005a, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_SQ_WEIGHT4_7, DIDT_SQ_WEIGHT4_7__WEIGHT4_MASK, DIDT_SQ_WEIGHT4_7__WEIGHT4__SHIFT, 0x0067, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_WEIGHT4_7, DIDT_SQ_WEIGHT4_7__WEIGHT5_MASK, DIDT_SQ_WEIGHT4_7__WEIGHT5__SHIFT, 0x0084, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_WEIGHT4_7, DIDT_SQ_WEIGHT4_7__WEIGHT6_MASK, DIDT_SQ_WEIGHT4_7__WEIGHT6__SHIFT, 0x0027, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_WEIGHT4_7, DIDT_SQ_WEIGHT4_7__WEIGHT7_MASK, DIDT_SQ_WEIGHT4_7__WEIGHT7__SHIFT, 0x0046, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_SQ_WEIGHT8_11, DIDT_SQ_WEIGHT8_11__WEIGHT8_MASK, DIDT_SQ_WEIGHT8_11__WEIGHT8__SHIFT, 0x00aa, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_WEIGHT8_11, DIDT_SQ_WEIGHT8_11__WEIGHT9_MASK, DIDT_SQ_WEIGHT8_11__WEIGHT9__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_WEIGHT8_11, DIDT_SQ_WEIGHT8_11__WEIGHT10_MASK, DIDT_SQ_WEIGHT8_11__WEIGHT10__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_WEIGHT8_11, DIDT_SQ_WEIGHT8_11__WEIGHT11_MASK, DIDT_SQ_WEIGHT8_11__WEIGHT11__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_SQ_CTRL1, DIDT_SQ_CTRL1__MIN_POWER_MASK, DIDT_SQ_CTRL1__MIN_POWER__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL1, DIDT_SQ_CTRL1__MAX_POWER_MASK, DIDT_SQ_CTRL1__MAX_POWER__SHIFT, 0xffff, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_SQ_CTRL_OCP, DIDT_SQ_CTRL_OCP__UNUSED_0_MASK, DIDT_SQ_CTRL_OCP__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL_OCP, DIDT_SQ_CTRL_OCP__OCP_MAX_POWER_MASK, DIDT_SQ_CTRL_OCP__OCP_MAX_POWER__SHIFT, 0xffff, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_SQ_CTRL2, DIDT_SQ_CTRL2__MAX_POWER_DELTA_MASK, DIDT_SQ_CTRL2__MAX_POWER_DELTA__SHIFT, 0x3853, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL2, DIDT_SQ_CTRL2__UNUSED_0_MASK, DIDT_SQ_CTRL2__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL2, DIDT_SQ_CTRL2__SHORT_TERM_INTERVAL_SIZE_MASK, DIDT_SQ_CTRL2__SHORT_TERM_INTERVAL_SIZE__SHIFT, 0x005a, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL2, DIDT_SQ_CTRL2__UNUSED_1_MASK, DIDT_SQ_CTRL2__UNUSED_1__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL2, DIDT_SQ_CTRL2__LONG_TERM_INTERVAL_RATIO_MASK, DIDT_SQ_CTRL2__LONG_TERM_INTERVAL_RATIO__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL2, DIDT_SQ_CTRL2__UNUSED_2_MASK, DIDT_SQ_CTRL2__UNUSED_2__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_SQ_STALL_CTRL, DIDT_SQ_STALL_CTRL__DIDT_STALL_CTRL_ENABLE_MASK, DIDT_SQ_STALL_CTRL__DIDT_STALL_CTRL_ENABLE__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_STALL_CTRL, DIDT_SQ_STALL_CTRL__DIDT_STALL_DELAY_HI_MASK, DIDT_SQ_STALL_CTRL__DIDT_STALL_DELAY_HI__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_STALL_CTRL, DIDT_SQ_STALL_CTRL__DIDT_STALL_DELAY_LO_MASK, DIDT_SQ_STALL_CTRL__DIDT_STALL_DELAY_LO__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_STALL_CTRL, DIDT_SQ_STALL_CTRL__DIDT_HI_POWER_THRESHOLD_MASK, DIDT_SQ_STALL_CTRL__DIDT_HI_POWER_THRESHOLD__SHIFT, 0x0ebb, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_STALL_CTRL, DIDT_SQ_STALL_CTRL__UNUSED_0_MASK, DIDT_SQ_STALL_CTRL__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_SQ_TUNING_CTRL, DIDT_SQ_TUNING_CTRL__DIDT_TUNING_ENABLE_MASK, DIDT_SQ_TUNING_CTRL__DIDT_TUNING_ENABLE__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_TUNING_CTRL, DIDT_SQ_TUNING_CTRL__MAX_POWER_DELTA_HI_MASK, DIDT_SQ_TUNING_CTRL__MAX_POWER_DELTA_HI__SHIFT, 0x3853, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_TUNING_CTRL, DIDT_SQ_TUNING_CTRL__MAX_POWER_DELTA_LO_MASK, DIDT_SQ_TUNING_CTRL__MAX_POWER_DELTA_LO__SHIFT, 0x3153, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_TUNING_CTRL, DIDT_SQ_TUNING_CTRL__UNUSED_0_MASK, DIDT_SQ_TUNING_CTRL__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_SQ_CTRL0, DIDT_SQ_CTRL0__DIDT_CTRL_EN_MASK, DIDT_SQ_CTRL0__DIDT_CTRL_EN__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL0, DIDT_SQ_CTRL0__USE_REF_CLOCK_MASK, DIDT_SQ_CTRL0__USE_REF_CLOCK__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL0, DIDT_SQ_CTRL0__PHASE_OFFSET_MASK, DIDT_SQ_CTRL0__PHASE_OFFSET__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL0, DIDT_SQ_CTRL0__DIDT_CTRL_RST_MASK, DIDT_SQ_CTRL0__DIDT_CTRL_RST__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL0, DIDT_SQ_CTRL0__DIDT_CLK_EN_OVERRIDE_MASK, DIDT_SQ_CTRL0__DIDT_CLK_EN_OVERRIDE__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL0, DIDT_SQ_CTRL0__DIDT_MAX_STALLS_ALLOWED_HI_MASK, DIDT_SQ_CTRL0__DIDT_MAX_STALLS_ALLOWED_HI__SHIFT, 0x0010, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL0, DIDT_SQ_CTRL0__DIDT_MAX_STALLS_ALLOWED_LO_MASK, DIDT_SQ_CTRL0__DIDT_MAX_STALLS_ALLOWED_LO__SHIFT, 0x0010, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_SQ_CTRL0, DIDT_SQ_CTRL0__UNUSED_0_MASK, DIDT_SQ_CTRL0__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + // DIDT_TD + // + { ixDIDT_TD_WEIGHT0_3, DIDT_TD_WEIGHT0_3__WEIGHT0_MASK, DIDT_TD_WEIGHT0_3__WEIGHT0__SHIFT, 0x000a, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_WEIGHT0_3, DIDT_TD_WEIGHT0_3__WEIGHT1_MASK, DIDT_TD_WEIGHT0_3__WEIGHT1__SHIFT, 0x0010, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_WEIGHT0_3, DIDT_TD_WEIGHT0_3__WEIGHT2_MASK, DIDT_TD_WEIGHT0_3__WEIGHT2__SHIFT, 0x0017, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_WEIGHT0_3, DIDT_TD_WEIGHT0_3__WEIGHT3_MASK, DIDT_TD_WEIGHT0_3__WEIGHT3__SHIFT, 0x002f, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TD_WEIGHT4_7, DIDT_TD_WEIGHT4_7__WEIGHT4_MASK, DIDT_TD_WEIGHT4_7__WEIGHT4__SHIFT, 0x0046, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_WEIGHT4_7, DIDT_TD_WEIGHT4_7__WEIGHT5_MASK, DIDT_TD_WEIGHT4_7__WEIGHT5__SHIFT, 0x005d, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_WEIGHT4_7, DIDT_TD_WEIGHT4_7__WEIGHT6_MASK, DIDT_TD_WEIGHT4_7__WEIGHT6__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_WEIGHT4_7, DIDT_TD_WEIGHT4_7__WEIGHT7_MASK, DIDT_TD_WEIGHT4_7__WEIGHT7__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TD_CTRL1, DIDT_TD_CTRL1__MIN_POWER_MASK, DIDT_TD_CTRL1__MIN_POWER__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL1, DIDT_TD_CTRL1__MAX_POWER_MASK, DIDT_TD_CTRL1__MAX_POWER__SHIFT, 0xffff, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TD_CTRL_OCP, DIDT_TD_CTRL_OCP__UNUSED_0_MASK, DIDT_TD_CTRL_OCP__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL_OCP, DIDT_TD_CTRL_OCP__OCP_MAX_POWER_MASK, DIDT_TD_CTRL_OCP__OCP_MAX_POWER__SHIFT, 0x00ff, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TD_CTRL2, DIDT_TD_CTRL2__MAX_POWER_DELTA_MASK, DIDT_TD_CTRL2__MAX_POWER_DELTA__SHIFT, 0x3fff, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL2, DIDT_TD_CTRL2__UNUSED_0_MASK, DIDT_TD_CTRL2__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL2, DIDT_TD_CTRL2__SHORT_TERM_INTERVAL_SIZE_MASK, DIDT_TD_CTRL2__SHORT_TERM_INTERVAL_SIZE__SHIFT, 0x000f, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL2, DIDT_TD_CTRL2__UNUSED_1_MASK, DIDT_TD_CTRL2__UNUSED_1__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL2, DIDT_TD_CTRL2__LONG_TERM_INTERVAL_RATIO_MASK, DIDT_TD_CTRL2__LONG_TERM_INTERVAL_RATIO__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL2, DIDT_TD_CTRL2__UNUSED_2_MASK, DIDT_TD_CTRL2__UNUSED_2__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TD_STALL_CTRL, DIDT_TD_STALL_CTRL__DIDT_STALL_CTRL_ENABLE_MASK, DIDT_TD_STALL_CTRL__DIDT_STALL_CTRL_ENABLE__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_STALL_CTRL, DIDT_TD_STALL_CTRL__DIDT_STALL_DELAY_HI_MASK, DIDT_TD_STALL_CTRL__DIDT_STALL_DELAY_HI__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_STALL_CTRL, DIDT_TD_STALL_CTRL__DIDT_STALL_DELAY_LO_MASK, DIDT_TD_STALL_CTRL__DIDT_STALL_DELAY_LO__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_STALL_CTRL, DIDT_TD_STALL_CTRL__DIDT_HI_POWER_THRESHOLD_MASK, DIDT_TD_STALL_CTRL__DIDT_HI_POWER_THRESHOLD__SHIFT, 0x01aa, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_STALL_CTRL, DIDT_TD_STALL_CTRL__UNUSED_0_MASK, DIDT_TD_STALL_CTRL__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TD_TUNING_CTRL, DIDT_TD_TUNING_CTRL__DIDT_TUNING_ENABLE_MASK, DIDT_TD_TUNING_CTRL__DIDT_TUNING_ENABLE__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_TUNING_CTRL, DIDT_TD_TUNING_CTRL__MAX_POWER_DELTA_HI_MASK, DIDT_TD_TUNING_CTRL__MAX_POWER_DELTA_HI__SHIFT, 0x0dde, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_TUNING_CTRL, DIDT_TD_TUNING_CTRL__MAX_POWER_DELTA_LO_MASK, DIDT_TD_TUNING_CTRL__MAX_POWER_DELTA_LO__SHIFT, 0x0dde, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_TUNING_CTRL, DIDT_TD_TUNING_CTRL__UNUSED_0_MASK, DIDT_TD_TUNING_CTRL__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TD_CTRL0, DIDT_TD_CTRL0__DIDT_CTRL_EN_MASK, DIDT_TD_CTRL0__DIDT_CTRL_EN__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL0, DIDT_TD_CTRL0__USE_REF_CLOCK_MASK, DIDT_TD_CTRL0__USE_REF_CLOCK__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL0, DIDT_TD_CTRL0__PHASE_OFFSET_MASK, DIDT_TD_CTRL0__PHASE_OFFSET__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL0, DIDT_TD_CTRL0__DIDT_CTRL_RST_MASK, DIDT_TD_CTRL0__DIDT_CTRL_RST__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL0, DIDT_TD_CTRL0__DIDT_CLK_EN_OVERRIDE_MASK, DIDT_TD_CTRL0__DIDT_CLK_EN_OVERRIDE__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL0, DIDT_TD_CTRL0__DIDT_MAX_STALLS_ALLOWED_HI_MASK, DIDT_TD_CTRL0__DIDT_MAX_STALLS_ALLOWED_HI__SHIFT, 0x0009, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL0, DIDT_TD_CTRL0__DIDT_MAX_STALLS_ALLOWED_LO_MASK, DIDT_TD_CTRL0__DIDT_MAX_STALLS_ALLOWED_LO__SHIFT, 0x0009, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TD_CTRL0, DIDT_TD_CTRL0__UNUSED_0_MASK, DIDT_TD_CTRL0__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + // DIDT_TCP + // + { ixDIDT_TCP_WEIGHT0_3, DIDT_TCP_WEIGHT0_3__WEIGHT0_MASK, DIDT_TCP_WEIGHT0_3__WEIGHT0__SHIFT, 0x0004, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_WEIGHT0_3, DIDT_TCP_WEIGHT0_3__WEIGHT1_MASK, DIDT_TCP_WEIGHT0_3__WEIGHT1__SHIFT, 0x0037, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_WEIGHT0_3, DIDT_TCP_WEIGHT0_3__WEIGHT2_MASK, DIDT_TCP_WEIGHT0_3__WEIGHT2__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_WEIGHT0_3, DIDT_TCP_WEIGHT0_3__WEIGHT3_MASK, DIDT_TCP_WEIGHT0_3__WEIGHT3__SHIFT, 0x00ff, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TCP_WEIGHT4_7, DIDT_TCP_WEIGHT4_7__WEIGHT4_MASK, DIDT_TCP_WEIGHT4_7__WEIGHT4__SHIFT, 0x0054, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_WEIGHT4_7, DIDT_TCP_WEIGHT4_7__WEIGHT5_MASK, DIDT_TCP_WEIGHT4_7__WEIGHT5__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_WEIGHT4_7, DIDT_TCP_WEIGHT4_7__WEIGHT6_MASK, DIDT_TCP_WEIGHT4_7__WEIGHT6__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_WEIGHT4_7, DIDT_TCP_WEIGHT4_7__WEIGHT7_MASK, DIDT_TCP_WEIGHT4_7__WEIGHT7__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TCP_CTRL1, DIDT_TCP_CTRL1__MIN_POWER_MASK, DIDT_TCP_CTRL1__MIN_POWER__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL1, DIDT_TCP_CTRL1__MAX_POWER_MASK, DIDT_TCP_CTRL1__MAX_POWER__SHIFT, 0xffff, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TCP_CTRL_OCP, DIDT_TCP_CTRL_OCP__UNUSED_0_MASK, DIDT_TCP_CTRL_OCP__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL_OCP, DIDT_TCP_CTRL_OCP__OCP_MAX_POWER_MASK, DIDT_TCP_CTRL_OCP__OCP_MAX_POWER__SHIFT, 0xffff, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TCP_CTRL2, DIDT_TCP_CTRL2__MAX_POWER_DELTA_MASK, DIDT_TCP_CTRL2__MAX_POWER_DELTA__SHIFT, 0x3dde, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL2, DIDT_TCP_CTRL2__UNUSED_0_MASK, DIDT_TCP_CTRL2__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL2, DIDT_TCP_CTRL2__SHORT_TERM_INTERVAL_SIZE_MASK, DIDT_TCP_CTRL2__SHORT_TERM_INTERVAL_SIZE__SHIFT, 0x0032, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL2, DIDT_TCP_CTRL2__UNUSED_1_MASK, DIDT_TCP_CTRL2__UNUSED_1__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL2, DIDT_TCP_CTRL2__LONG_TERM_INTERVAL_RATIO_MASK, DIDT_TCP_CTRL2__LONG_TERM_INTERVAL_RATIO__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL2, DIDT_TCP_CTRL2__UNUSED_2_MASK, DIDT_TCP_CTRL2__UNUSED_2__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TCP_STALL_CTRL, DIDT_TCP_STALL_CTRL__DIDT_STALL_CTRL_ENABLE_MASK, DIDT_TCP_STALL_CTRL__DIDT_STALL_CTRL_ENABLE__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_STALL_CTRL, DIDT_TCP_STALL_CTRL__DIDT_STALL_DELAY_HI_MASK, DIDT_TCP_STALL_CTRL__DIDT_STALL_DELAY_HI__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_STALL_CTRL, DIDT_TCP_STALL_CTRL__DIDT_STALL_DELAY_LO_MASK, DIDT_TCP_STALL_CTRL__DIDT_STALL_DELAY_LO__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_STALL_CTRL, DIDT_TCP_STALL_CTRL__DIDT_HI_POWER_THRESHOLD_MASK, DIDT_TCP_STALL_CTRL__DIDT_HI_POWER_THRESHOLD__SHIFT,0x01aa, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_STALL_CTRL, DIDT_TCP_STALL_CTRL__UNUSED_0_MASK, DIDT_TCP_STALL_CTRL__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TCP_TUNING_CTRL, DIDT_TCP_TUNING_CTRL__DIDT_TUNING_ENABLE_MASK, DIDT_TCP_TUNING_CTRL__DIDT_TUNING_ENABLE__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_TUNING_CTRL, DIDT_TCP_TUNING_CTRL__MAX_POWER_DELTA_HI_MASK, DIDT_TCP_TUNING_CTRL__MAX_POWER_DELTA_HI__SHIFT, 0x3dde, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_TUNING_CTRL, DIDT_TCP_TUNING_CTRL__MAX_POWER_DELTA_LO_MASK, DIDT_TCP_TUNING_CTRL__MAX_POWER_DELTA_LO__SHIFT, 0x3dde, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_TUNING_CTRL, DIDT_TCP_TUNING_CTRL__UNUSED_0_MASK, DIDT_TCP_TUNING_CTRL__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { ixDIDT_TCP_CTRL0, DIDT_TCP_CTRL0__DIDT_CTRL_EN_MASK, DIDT_TCP_CTRL0__DIDT_CTRL_EN__SHIFT, 0x0001, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL0, DIDT_TCP_CTRL0__USE_REF_CLOCK_MASK, DIDT_TCP_CTRL0__USE_REF_CLOCK__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL0, DIDT_TCP_CTRL0__PHASE_OFFSET_MASK, DIDT_TCP_CTRL0__PHASE_OFFSET__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL0, DIDT_TCP_CTRL0__DIDT_CTRL_RST_MASK, DIDT_TCP_CTRL0__DIDT_CTRL_RST__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL0, DIDT_TCP_CTRL0__DIDT_CLK_EN_OVERRIDE_MASK, DIDT_TCP_CTRL0__DIDT_CLK_EN_OVERRIDE__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL0, DIDT_TCP_CTRL0__DIDT_MAX_STALLS_ALLOWED_HI_MASK, DIDT_TCP_CTRL0__DIDT_MAX_STALLS_ALLOWED_HI__SHIFT, 0x0010, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL0, DIDT_TCP_CTRL0__DIDT_MAX_STALLS_ALLOWED_LO_MASK, DIDT_TCP_CTRL0__DIDT_MAX_STALLS_ALLOWED_LO__SHIFT, 0x0010, GPU_CONFIGREG_DIDT_IND }, + { ixDIDT_TCP_CTRL0, DIDT_TCP_CTRL0__UNUSED_0_MASK, DIDT_TCP_CTRL0__UNUSED_0__SHIFT, 0x0000, GPU_CONFIGREG_DIDT_IND }, + + { 0xFFFFFFFF } // End of list +}; static int smu7_enable_didt(struct pp_hwmgr *hwmgr, const bool enable) { uint32_t en = enable ? 1 : 0; @@ -740,8 +924,8 @@ int smu7_enable_didt_config(struct pp_hwmgr *hwmgr) PP_CAP(PHM_PlatformCaps_TDRamping) || PP_CAP(PHM_PlatformCaps_TCPRamping)) { - cgs_enter_safe_mode(hwmgr->device, true); - cgs_lock_grbm_idx(hwmgr->device, true); + adev->gfx.rlc.funcs->enter_safe_mode(adev); + mutex_lock(&adev->grbm_idx_mutex); value = 0; value2 = cgs_read_register(hwmgr->device, mmGRBM_GFX_INDEX); for (count = 0; count < num_se; count++) { @@ -752,67 +936,80 @@ int smu7_enable_didt_config(struct pp_hwmgr *hwmgr) if (hwmgr->chip_id == CHIP_POLARIS10) { result = smu7_program_pt_config_registers(hwmgr, GCCACConfig_Polaris10); - PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", return result); + PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", goto error); result = smu7_program_pt_config_registers(hwmgr, DIDTConfig_Polaris10); - PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", return result); + PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", goto error); } else if (hwmgr->chip_id == CHIP_POLARIS11) { result = smu7_program_pt_config_registers(hwmgr, GCCACConfig_Polaris11); - PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", return result); + PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", goto error); if (hwmgr->is_kicker) result = smu7_program_pt_config_registers(hwmgr, DIDTConfig_Polaris11_Kicker); else result = smu7_program_pt_config_registers(hwmgr, DIDTConfig_Polaris11); - PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", return result); + PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", goto error); } else if (hwmgr->chip_id == CHIP_POLARIS12) { result = smu7_program_pt_config_registers(hwmgr, GCCACConfig_Polaris11); - PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", return result); + PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", goto error); result = smu7_program_pt_config_registers(hwmgr, DIDTConfig_Polaris12); - PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", return result); + PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", goto error); + } else if (hwmgr->chip_id == CHIP_VEGAM) { + result = smu7_program_pt_config_registers(hwmgr, GCCACConfig_VegaM); + PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", goto error); + result = smu7_program_pt_config_registers(hwmgr, DIDTConfig_VegaM); + PP_ASSERT_WITH_CODE((result == 0), "DIDT Config failed.", goto error); } } cgs_write_register(hwmgr->device, mmGRBM_GFX_INDEX, value2); result = smu7_enable_didt(hwmgr, true); - PP_ASSERT_WITH_CODE((result == 0), "EnableDiDt failed.", return result); + PP_ASSERT_WITH_CODE((result == 0), "EnableDiDt failed.", goto error); if (hwmgr->chip_id == CHIP_POLARIS11) { result = smum_send_msg_to_smc(hwmgr, (uint16_t)(PPSMC_MSG_EnableDpmDidt)); PP_ASSERT_WITH_CODE((0 == result), - "Failed to enable DPM DIDT.", return result); + "Failed to enable DPM DIDT.", goto error); } - cgs_lock_grbm_idx(hwmgr->device, false); - cgs_enter_safe_mode(hwmgr->device, false); + mutex_unlock(&adev->grbm_idx_mutex); + adev->gfx.rlc.funcs->exit_safe_mode(adev); } return 0; +error: + mutex_unlock(&adev->grbm_idx_mutex); + adev->gfx.rlc.funcs->exit_safe_mode(adev); + return result; } int smu7_disable_didt_config(struct pp_hwmgr *hwmgr) { int result; + struct amdgpu_device *adev = hwmgr->adev; if (PP_CAP(PHM_PlatformCaps_SQRamping) || PP_CAP(PHM_PlatformCaps_DBRamping) || PP_CAP(PHM_PlatformCaps_TDRamping) || PP_CAP(PHM_PlatformCaps_TCPRamping)) { - cgs_enter_safe_mode(hwmgr->device, true); + adev->gfx.rlc.funcs->enter_safe_mode(adev); result = smu7_enable_didt(hwmgr, false); PP_ASSERT_WITH_CODE((result == 0), "Post DIDT enable clock gating failed.", - return result); + goto error); if (hwmgr->chip_id == CHIP_POLARIS11) { result = smum_send_msg_to_smc(hwmgr, (uint16_t)(PPSMC_MSG_DisableDpmDidt)); PP_ASSERT_WITH_CODE((0 == result), - "Failed to disable DPM DIDT.", return result); + "Failed to disable DPM DIDT.", goto error); } - cgs_enter_safe_mode(hwmgr->device, false); + adev->gfx.rlc.funcs->exit_safe_mode(adev); } return 0; +error: + adev->gfx.rlc.funcs->exit_safe_mode(adev); + return result; } int smu7_enable_smc_cac(struct pp_hwmgr *hwmgr) diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/smu8_hwmgr.c b/drivers/gpu/drm/amd/powerplay/hwmgr/smu8_hwmgr.c index 7b26607c646a..50690c72b2ea 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/smu8_hwmgr.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/smu8_hwmgr.c @@ -314,8 +314,7 @@ static int smu8_get_system_info_data(struct pp_hwmgr *hwmgr) uint8_t frev, crev; uint16_t size; - info = (ATOM_INTEGRATED_SYSTEM_INFO_V1_9 *) cgs_atom_get_data_table( - hwmgr->device, + info = (ATOM_INTEGRATED_SYSTEM_INFO_V1_9 *)smu_atom_get_data_table(hwmgr->adev, GetIndexIntoMasterTable(DATA, IntegratedSystemInfo), &size, &frev, &crev); @@ -694,7 +693,7 @@ static int smu8_update_sclk_limit(struct pp_hwmgr *hwmgr) else data->sclk_dpm.soft_max_clk = table->entries[table->count - 1].clk; - clock = hwmgr->display_config.min_core_set_clock; + clock = hwmgr->display_config->min_core_set_clock; if (clock == 0) pr_debug("min_core_set_clock not set\n"); @@ -749,7 +748,7 @@ static int smu8_set_deep_sleep_sclk_threshold(struct pp_hwmgr *hwmgr) { if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_SclkDeepSleep)) { - uint32_t clks = hwmgr->display_config.min_core_set_clock_in_sr; + uint32_t clks = hwmgr->display_config->min_core_set_clock_in_sr; if (clks == 0) clks = SMU8_MIN_DEEP_SLEEP_SCLK; @@ -1041,25 +1040,21 @@ static int smu8_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, struct smu8_hwmgr *data = hwmgr->backend; struct PP_Clocks clocks = {0, 0, 0, 0}; bool force_high; - uint32_t num_of_active_displays = 0; - struct cgs_display_info info = {0}; smu8_ps->need_dfs_bypass = true; data->battery_state = (PP_StateUILabel_Battery == prequest_ps->classification.ui_label); - clocks.memoryClock = hwmgr->display_config.min_mem_set_clock != 0 ? - hwmgr->display_config.min_mem_set_clock : + clocks.memoryClock = hwmgr->display_config->min_mem_set_clock != 0 ? + hwmgr->display_config->min_mem_set_clock : data->sys_info.nbp_memory_clock[1]; - cgs_get_active_displays_info(hwmgr->device, &info); - num_of_active_displays = info.display_count; if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_StablePState)) clocks.memoryClock = hwmgr->dyn_state.max_clock_voltage_on_ac.mclk; force_high = (clocks.memoryClock > data->sys_info.nbp_memory_clock[SMU8_NUM_NBPMEMORYCLOCK - 1]) - || (num_of_active_displays >= 3); + || (hwmgr->display_config->num_display >= 3); smu8_ps->action = smu8_current_ps->action; @@ -1897,20 +1892,20 @@ static void smu8_dpm_powergate_uvd(struct pp_hwmgr *hwmgr, bool bgate) data->uvd_power_gated = bgate; if (bgate) { - cgs_set_powergating_state(hwmgr->device, + amdgpu_device_ip_set_powergating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_UVD, AMD_PG_STATE_GATE); - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_UVD, AMD_CG_STATE_GATE); smu8_dpm_update_uvd_dpm(hwmgr, true); smu8_dpm_powerdown_uvd(hwmgr); } else { smu8_dpm_powerup_uvd(hwmgr); - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_UVD, AMD_CG_STATE_UNGATE); - cgs_set_powergating_state(hwmgr->device, + amdgpu_device_ip_set_powergating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_UVD, AMD_PG_STATE_UNGATE); smu8_dpm_update_uvd_dpm(hwmgr, false); @@ -1923,12 +1918,10 @@ static void smu8_dpm_powergate_vce(struct pp_hwmgr *hwmgr, bool bgate) struct smu8_hwmgr *data = hwmgr->backend; if (bgate) { - cgs_set_powergating_state( - hwmgr->device, + amdgpu_device_ip_set_powergating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_VCE, AMD_PG_STATE_GATE); - cgs_set_clockgating_state( - hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_VCE, AMD_CG_STATE_GATE); smu8_enable_disable_vce_dpm(hwmgr, false); @@ -1937,12 +1930,10 @@ static void smu8_dpm_powergate_vce(struct pp_hwmgr *hwmgr, bool bgate) } else { smu8_dpm_powerup_vce(hwmgr); data->vce_power_gated = false; - cgs_set_clockgating_state( - hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_VCE, AMD_CG_STATE_UNGATE); - cgs_set_powergating_state( - hwmgr->device, + amdgpu_device_ip_set_powergating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_VCE, AMD_PG_STATE_UNGATE); smu8_dpm_update_vce_dpm(hwmgr); diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/smu_helper.c b/drivers/gpu/drm/amd/powerplay/hwmgr/smu_helper.c index 598122854ab5..93a3d022ba47 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/smu_helper.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/smu_helper.c @@ -24,6 +24,7 @@ #include "pp_debug.h" #include "ppatomctrl.h" #include "ppsmc.h" +#include "atom.h" uint8_t convert_to_vid(uint16_t vddc) { @@ -608,3 +609,100 @@ int smu9_register_irq_handlers(struct pp_hwmgr *hwmgr) return 0; } + +void *smu_atom_get_data_table(void *dev, uint32_t table, uint16_t *size, + uint8_t *frev, uint8_t *crev) +{ + struct amdgpu_device *adev = dev; + uint16_t data_start; + + if (amdgpu_atom_parse_data_header( + adev->mode_info.atom_context, table, size, + frev, crev, &data_start)) + return (uint8_t *)adev->mode_info.atom_context->bios + + data_start; + + return NULL; +} + +int smu_get_voltage_dependency_table_ppt_v1( + const struct phm_ppt_v1_clock_voltage_dependency_table *allowed_dep_table, + struct phm_ppt_v1_clock_voltage_dependency_table *dep_table) +{ + uint8_t i = 0; + PP_ASSERT_WITH_CODE((0 != allowed_dep_table->count), + "Voltage Lookup Table empty", + return -EINVAL); + + dep_table->count = allowed_dep_table->count; + for (i=0; icount; i++) { + dep_table->entries[i].clk = allowed_dep_table->entries[i].clk; + dep_table->entries[i].vddInd = allowed_dep_table->entries[i].vddInd; + dep_table->entries[i].vdd_offset = allowed_dep_table->entries[i].vdd_offset; + dep_table->entries[i].vddc = allowed_dep_table->entries[i].vddc; + dep_table->entries[i].vddgfx = allowed_dep_table->entries[i].vddgfx; + dep_table->entries[i].vddci = allowed_dep_table->entries[i].vddci; + dep_table->entries[i].mvdd = allowed_dep_table->entries[i].mvdd; + dep_table->entries[i].phases = allowed_dep_table->entries[i].phases; + dep_table->entries[i].cks_enable = allowed_dep_table->entries[i].cks_enable; + dep_table->entries[i].cks_voffset = allowed_dep_table->entries[i].cks_voffset; + } + + return 0; +} + +int smu_set_watermarks_for_clocks_ranges(void *wt_table, + struct pp_wm_sets_with_clock_ranges_soc15 *wm_with_clock_ranges) +{ + uint32_t i; + struct watermarks *table = wt_table; + + if (!table || !wm_with_clock_ranges) + return -EINVAL; + + if (wm_with_clock_ranges->num_wm_sets_dmif > 4 || wm_with_clock_ranges->num_wm_sets_mcif > 4) + return -EINVAL; + + for (i = 0; i < wm_with_clock_ranges->num_wm_sets_dmif; i++) { + table->WatermarkRow[1][i].MinClock = + cpu_to_le16((uint16_t) + (wm_with_clock_ranges->wm_sets_dmif[i].wm_min_dcefclk_in_khz) / + 100); + table->WatermarkRow[1][i].MaxClock = + cpu_to_le16((uint16_t) + (wm_with_clock_ranges->wm_sets_dmif[i].wm_max_dcefclk_in_khz) / + 100); + table->WatermarkRow[1][i].MinUclk = + cpu_to_le16((uint16_t) + (wm_with_clock_ranges->wm_sets_dmif[i].wm_min_memclk_in_khz) / + 100); + table->WatermarkRow[1][i].MaxUclk = + cpu_to_le16((uint16_t) + (wm_with_clock_ranges->wm_sets_dmif[i].wm_max_memclk_in_khz) / + 100); + table->WatermarkRow[1][i].WmSetting = (uint8_t) + wm_with_clock_ranges->wm_sets_dmif[i].wm_set_id; + } + + for (i = 0; i < wm_with_clock_ranges->num_wm_sets_mcif; i++) { + table->WatermarkRow[0][i].MinClock = + cpu_to_le16((uint16_t) + (wm_with_clock_ranges->wm_sets_mcif[i].wm_min_socclk_in_khz) / + 100); + table->WatermarkRow[0][i].MaxClock = + cpu_to_le16((uint16_t) + (wm_with_clock_ranges->wm_sets_mcif[i].wm_max_socclk_in_khz) / + 100); + table->WatermarkRow[0][i].MinUclk = + cpu_to_le16((uint16_t) + (wm_with_clock_ranges->wm_sets_mcif[i].wm_min_memclk_in_khz) / + 100); + table->WatermarkRow[0][i].MaxUclk = + cpu_to_le16((uint16_t) + (wm_with_clock_ranges->wm_sets_mcif[i].wm_max_memclk_in_khz) / + 100); + table->WatermarkRow[0][i].WmSetting = (uint8_t) + wm_with_clock_ranges->wm_sets_mcif[i].wm_set_id; + } + return 0; +} diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/smu_helper.h b/drivers/gpu/drm/amd/powerplay/hwmgr/smu_helper.h index d37d16e4b613..916cc01e7652 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/smu_helper.h +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/smu_helper.h @@ -26,10 +26,27 @@ struct pp_atomctrl_voltage_table; struct pp_hwmgr; struct phm_ppt_v1_voltage_lookup_table; +struct Watermarks_t; +struct pp_wm_sets_with_clock_ranges_soc15; uint8_t convert_to_vid(uint16_t vddc); uint16_t convert_to_vddc(uint8_t vid); +struct watermark_row_generic_t { + uint16_t MinClock; + uint16_t MaxClock; + uint16_t MinUclk; + uint16_t MaxUclk; + + uint8_t WmSetting; + uint8_t Padding[3]; +}; + +struct watermarks { + struct watermark_row_generic_t WatermarkRow[2][4]; + uint32_t padding[7]; +}; + extern int phm_wait_for_register_unequal(struct pp_hwmgr *hwmgr, uint32_t index, uint32_t value, uint32_t mask); @@ -82,6 +99,16 @@ int phm_irq_process(struct amdgpu_device *adev, int smu9_register_irq_handlers(struct pp_hwmgr *hwmgr); +void *smu_atom_get_data_table(void *dev, uint32_t table, uint16_t *size, + uint8_t *frev, uint8_t *crev); + +int smu_get_voltage_dependency_table_ppt_v1( + const struct phm_ppt_v1_clock_voltage_dependency_table *allowed_dep_table, + struct phm_ppt_v1_clock_voltage_dependency_table *dep_table); + +int smu_set_watermarks_for_clocks_ranges(void *wt_table, + struct pp_wm_sets_with_clock_ranges_soc15 *wm_with_clock_ranges); + #define PHM_FIELD_SHIFT(reg, field) reg##__##field##__SHIFT #define PHM_FIELD_MASK(reg, field) reg##__##field##_MASK diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_hwmgr.c b/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_hwmgr.c index 7cbb56ba6fab..05e680d55dbb 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_hwmgr.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_hwmgr.c @@ -36,7 +36,7 @@ #include "smu9.h" #include "smu9_driver_if.h" #include "vega10_inc.h" -#include "pp_soc15.h" +#include "soc15_common.h" #include "pppcielanes.h" #include "vega10_hwmgr.h" #include "vega10_processpptables.h" @@ -51,10 +51,6 @@ #include "smuio/smuio_9_0_offset.h" #include "smuio/smuio_9_0_sh_mask.h" -#define VOLTAGE_SCALE 4 -#define VOLTAGE_VID_OFFSET_SCALE1 625 -#define VOLTAGE_VID_OFFSET_SCALE2 100 - #define HBM_MEMORY_CHANNEL_WIDTH 128 static const uint32_t channel_number[] = {1, 2, 0, 4, 0, 8, 0, 16, 2}; @@ -79,8 +75,6 @@ static const uint32_t channel_number[] = {1, 2, 0, 4, 0, 8, 0, 16, 2}; #define DF_CS_AON0_DramBaseAddress0__IntLvNumChan_MASK 0x000000F0L #define DF_CS_AON0_DramBaseAddress0__IntLvAddrSel_MASK 0x00000700L #define DF_CS_AON0_DramBaseAddress0__DramBaseAddr_MASK 0xFFFFF000L -static int vega10_force_clock_level(struct pp_hwmgr *hwmgr, - enum pp_clock_type type, uint32_t mask); static const ULONG PhwVega10_Magic = (ULONG)(PHM_VIslands_Magic); @@ -291,6 +285,52 @@ static int vega10_set_features_platform_caps(struct pp_hwmgr *hwmgr) return 0; } +static int vega10_odn_initial_default_setting(struct pp_hwmgr *hwmgr) +{ + struct vega10_hwmgr *data = hwmgr->backend; + struct phm_ppt_v2_information *table_info = + (struct phm_ppt_v2_information *)(hwmgr->pptable); + struct vega10_odn_dpm_table *odn_table = &(data->odn_dpm_table); + struct vega10_odn_vddc_lookup_table *od_lookup_table; + struct phm_ppt_v1_voltage_lookup_table *vddc_lookup_table; + struct phm_ppt_v1_clock_voltage_dependency_table *dep_table[3]; + struct phm_ppt_v1_clock_voltage_dependency_table *od_table[3]; + uint32_t i; + + od_lookup_table = &odn_table->vddc_lookup_table; + vddc_lookup_table = table_info->vddc_lookup_table; + + for (i = 0; i < vddc_lookup_table->count; i++) + od_lookup_table->entries[i].us_vdd = vddc_lookup_table->entries[i].us_vdd; + + od_lookup_table->count = vddc_lookup_table->count; + + dep_table[0] = table_info->vdd_dep_on_sclk; + dep_table[1] = table_info->vdd_dep_on_mclk; + dep_table[2] = table_info->vdd_dep_on_socclk; + od_table[0] = (struct phm_ppt_v1_clock_voltage_dependency_table *)&odn_table->vdd_dep_on_sclk; + od_table[1] = (struct phm_ppt_v1_clock_voltage_dependency_table *)&odn_table->vdd_dep_on_mclk; + od_table[2] = (struct phm_ppt_v1_clock_voltage_dependency_table *)&odn_table->vdd_dep_on_socclk; + + for (i = 0; i < 3; i++) + smu_get_voltage_dependency_table_ppt_v1(dep_table[i], od_table[i]); + + if (odn_table->max_vddc == 0 || odn_table->max_vddc > 2000) + odn_table->max_vddc = dep_table[0]->entries[dep_table[0]->count - 1].vddc; + if (odn_table->min_vddc == 0 || odn_table->min_vddc > 2000) + odn_table->min_vddc = dep_table[0]->entries[0].vddc; + + i = od_table[2]->count - 1; + od_table[2]->entries[i].clk = hwmgr->platform_descriptor.overdriveLimit.memoryClock > od_table[2]->entries[i].clk ? + hwmgr->platform_descriptor.overdriveLimit.memoryClock : + od_table[2]->entries[i].clk; + od_table[2]->entries[i].vddc = odn_table->max_vddc > od_table[2]->entries[i].vddc ? + odn_table->max_vddc : + od_table[2]->entries[i].vddc; + + return 0; +} + static void vega10_init_dpm_defaults(struct pp_hwmgr *hwmgr) { struct vega10_hwmgr *data = hwmgr->backend; @@ -427,7 +467,6 @@ static void vega10_init_dpm_defaults(struct pp_hwmgr *hwmgr) /* ACG firmware has major version 5 */ if ((hwmgr->smu_version & 0xff000000) == 0x5000000) data->smu_features[GNLD_ACG].supported = true; - if (data->registry_data.didt_support) data->smu_features[GNLD_DIDT].supported = true; @@ -754,7 +793,6 @@ static int vega10_hwmgr_backend_init(struct pp_hwmgr *hwmgr) uint32_t config_telemetry = 0; struct pp_atomfwctrl_voltage_table vol_table; struct amdgpu_device *adev = hwmgr->adev; - uint32_t reg; data = kzalloc(sizeof(struct vega10_hwmgr), GFP_KERNEL); if (data == NULL) @@ -860,10 +898,7 @@ static int vega10_hwmgr_backend_init(struct pp_hwmgr *hwmgr) advanceFanControlParameters.usFanPWMMinLimit * hwmgr->thermal_controller.fanInfo.ulMaxRPM / 100; - reg = soc15_get_register_offset(DF_HWID, 0, - mmDF_CS_AON0_DramBaseAddress0_BASE_IDX, - mmDF_CS_AON0_DramBaseAddress0); - data->mem_channels = (cgs_read_register(hwmgr->device, reg) & + data->mem_channels = (RREG32_SOC15(DF, 0, mmDF_CS_AON0_DramBaseAddress0) & DF_CS_AON0_DramBaseAddress0__IntLvNumChan_MASK) >> DF_CS_AON0_DramBaseAddress0__IntLvNumChan__SHIFT; PP_ASSERT_WITH_CODE(data->mem_channels < ARRAY_SIZE(channel_number), @@ -1280,6 +1315,9 @@ static int vega10_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) vega10_setup_default_single_dpm_table(hwmgr, dpm_table, dep_gfx_table); + if (hwmgr->platform_descriptor.overdriveLimit.engineClock == 0) + hwmgr->platform_descriptor.overdriveLimit.engineClock = + dpm_table->dpm_levels[dpm_table->count-1].value; vega10_init_dpm_state(&(dpm_table->dpm_state)); /* Initialize Mclk DPM table based on allow Mclk values */ @@ -1288,6 +1326,10 @@ static int vega10_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) vega10_setup_default_single_dpm_table(hwmgr, dpm_table, dep_mclk_table); + if (hwmgr->platform_descriptor.overdriveLimit.memoryClock == 0) + hwmgr->platform_descriptor.overdriveLimit.memoryClock = + dpm_table->dpm_levels[dpm_table->count-1].value; + vega10_init_dpm_state(&(dpm_table->dpm_state)); data->dpm_table.eclk_table.count = 0; @@ -1370,48 +1412,6 @@ static int vega10_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) memcpy(&(data->golden_dpm_table), &(data->dpm_table), sizeof(struct vega10_dpm_table)); - if (PP_CAP(PHM_PlatformCaps_ODNinACSupport) || - PP_CAP(PHM_PlatformCaps_ODNinDCSupport)) { - data->odn_dpm_table.odn_core_clock_dpm_levels.num_of_pl = - data->dpm_table.gfx_table.count; - for (i = 0; i < data->dpm_table.gfx_table.count; i++) { - data->odn_dpm_table.odn_core_clock_dpm_levels.entries[i].clock = - data->dpm_table.gfx_table.dpm_levels[i].value; - data->odn_dpm_table.odn_core_clock_dpm_levels.entries[i].enabled = true; - } - - data->odn_dpm_table.vdd_dependency_on_sclk.count = - dep_gfx_table->count; - for (i = 0; i < dep_gfx_table->count; i++) { - data->odn_dpm_table.vdd_dependency_on_sclk.entries[i].clk = - dep_gfx_table->entries[i].clk; - data->odn_dpm_table.vdd_dependency_on_sclk.entries[i].vddInd = - dep_gfx_table->entries[i].vddInd; - data->odn_dpm_table.vdd_dependency_on_sclk.entries[i].cks_enable = - dep_gfx_table->entries[i].cks_enable; - data->odn_dpm_table.vdd_dependency_on_sclk.entries[i].cks_voffset = - dep_gfx_table->entries[i].cks_voffset; - } - - data->odn_dpm_table.odn_memory_clock_dpm_levels.num_of_pl = - data->dpm_table.mem_table.count; - for (i = 0; i < data->dpm_table.mem_table.count; i++) { - data->odn_dpm_table.odn_memory_clock_dpm_levels.entries[i].clock = - data->dpm_table.mem_table.dpm_levels[i].value; - data->odn_dpm_table.odn_memory_clock_dpm_levels.entries[i].enabled = true; - } - - data->odn_dpm_table.vdd_dependency_on_mclk.count = dep_mclk_table->count; - for (i = 0; i < dep_mclk_table->count; i++) { - data->odn_dpm_table.vdd_dependency_on_mclk.entries[i].clk = - dep_mclk_table->entries[i].clk; - data->odn_dpm_table.vdd_dependency_on_mclk.entries[i].vddInd = - dep_mclk_table->entries[i].vddInd; - data->odn_dpm_table.vdd_dependency_on_mclk.entries[i].vddci = - dep_mclk_table->entries[i].vddci; - } - } - return 0; } @@ -1514,18 +1514,18 @@ static int vega10_populate_single_gfx_level(struct pp_hwmgr *hwmgr, { struct phm_ppt_v2_information *table_info = (struct phm_ppt_v2_information *)(hwmgr->pptable); - struct phm_ppt_v1_clock_voltage_dependency_table *dep_on_sclk = - table_info->vdd_dep_on_sclk; + struct phm_ppt_v1_clock_voltage_dependency_table *dep_on_sclk; struct vega10_hwmgr *data = hwmgr->backend; struct pp_atomfwctrl_clock_dividers_soc15 dividers; uint32_t gfx_max_clock = hwmgr->platform_descriptor.overdriveLimit.engineClock; uint32_t i = 0; - if (data->apply_overdrive_next_settings_mask & - DPMTABLE_OD_UPDATE_VDDC) + if (hwmgr->od_enabled) dep_on_sclk = (struct phm_ppt_v1_clock_voltage_dependency_table *) - &(data->odn_dpm_table.vdd_dependency_on_sclk); + &(data->odn_dpm_table.vdd_dep_on_sclk); + else + dep_on_sclk = table_info->vdd_dep_on_sclk; PP_ASSERT_WITH_CODE(dep_on_sclk, "Invalid SOC_VDD-GFX_CLK Dependency Table!", @@ -1577,23 +1577,32 @@ static int vega10_populate_single_soc_level(struct pp_hwmgr *hwmgr, uint32_t soc_clock, uint8_t *current_soc_did, uint8_t *current_vol_index) { + struct vega10_hwmgr *data = hwmgr->backend; struct phm_ppt_v2_information *table_info = (struct phm_ppt_v2_information *)(hwmgr->pptable); - struct phm_ppt_v1_clock_voltage_dependency_table *dep_on_soc = - table_info->vdd_dep_on_socclk; + struct phm_ppt_v1_clock_voltage_dependency_table *dep_on_soc; struct pp_atomfwctrl_clock_dividers_soc15 dividers; uint32_t i; - PP_ASSERT_WITH_CODE(dep_on_soc, - "Invalid SOC_VDD-SOC_CLK Dependency Table!", - return -EINVAL); - for (i = 0; i < dep_on_soc->count; i++) { - if (dep_on_soc->entries[i].clk == soc_clock) - break; + if (hwmgr->od_enabled) { + dep_on_soc = (struct phm_ppt_v1_clock_voltage_dependency_table *) + &data->odn_dpm_table.vdd_dep_on_socclk; + for (i = 0; i < dep_on_soc->count; i++) { + if (dep_on_soc->entries[i].clk >= soc_clock) + break; + } + } else { + dep_on_soc = table_info->vdd_dep_on_socclk; + for (i = 0; i < dep_on_soc->count; i++) { + if (dep_on_soc->entries[i].clk == soc_clock) + break; + } } + PP_ASSERT_WITH_CODE(dep_on_soc->count > i, "Cannot find SOC_CLK in SOC_VDD-SOC_CLK Dependency Table", return -EINVAL); + PP_ASSERT_WITH_CODE(!pp_atomfwctrl_get_gpu_pll_dividers_vega10(hwmgr, COMPUTE_GPUCLK_INPUT_FLAG_DEFAULT_GPUCLK, soc_clock, ÷rs), @@ -1602,22 +1611,6 @@ static int vega10_populate_single_soc_level(struct pp_hwmgr *hwmgr, *current_soc_did = (uint8_t)dividers.ulDid; *current_vol_index = (uint8_t)(dep_on_soc->entries[i].vddInd); - - return 0; -} - -uint16_t vega10_locate_vddc_given_clock(struct pp_hwmgr *hwmgr, - uint32_t clk, - struct phm_ppt_v1_clock_voltage_dependency_table *dep_table) -{ - uint16_t i; - - for (i = 0; i < dep_table->count; i++) { - if (dep_table->entries[i].clk == clk) - return dep_table->entries[i].vddc; - } - - pr_info("[LocateVddcGivenClock] Cannot locate SOC Vddc for this clock!"); return 0; } @@ -1631,8 +1624,6 @@ static int vega10_populate_all_graphic_levels(struct pp_hwmgr *hwmgr) struct vega10_hwmgr *data = hwmgr->backend; struct phm_ppt_v2_information *table_info = (struct phm_ppt_v2_information *)(hwmgr->pptable); - struct phm_ppt_v1_clock_voltage_dependency_table *dep_table = - table_info->vdd_dep_on_socclk; PPTable_t *pp_table = &(data->smc_state_table.pp_table); struct vega10_single_dpm_table *dpm_table = &(data->dpm_table.gfx_table); int result = 0; @@ -1663,11 +1654,6 @@ static int vega10_populate_all_graphic_levels(struct pp_hwmgr *hwmgr) dpm_table = &(data->dpm_table.soc_table); for (i = 0; i < dpm_table->count; i++) { - pp_table->SocVid[i] = - (uint8_t)convert_to_vid( - vega10_locate_vddc_given_clock(hwmgr, - dpm_table->dpm_levels[i].value, - dep_table)); result = vega10_populate_single_soc_level(hwmgr, dpm_table->dpm_levels[i].value, &(pp_table->SocclkDid[i]), @@ -1678,7 +1664,6 @@ static int vega10_populate_all_graphic_levels(struct pp_hwmgr *hwmgr) j = i - 1; while (i < NUM_SOCCLK_DPM_LEVELS) { - pp_table->SocVid[i] = pp_table->SocVid[j]; result = vega10_populate_single_soc_level(hwmgr, dpm_table->dpm_levels[j].value, &(pp_table->SocclkDid[i]), @@ -1691,6 +1676,32 @@ static int vega10_populate_all_graphic_levels(struct pp_hwmgr *hwmgr) return result; } +static void vega10_populate_vddc_soc_levels(struct pp_hwmgr *hwmgr) +{ + struct vega10_hwmgr *data = hwmgr->backend; + PPTable_t *pp_table = &(data->smc_state_table.pp_table); + struct phm_ppt_v2_information *table_info = hwmgr->pptable; + struct phm_ppt_v1_voltage_lookup_table *vddc_lookup_table; + + uint8_t soc_vid = 0; + uint32_t i, max_vddc_level; + + if (hwmgr->od_enabled) + vddc_lookup_table = (struct phm_ppt_v1_voltage_lookup_table *)&data->odn_dpm_table.vddc_lookup_table; + else + vddc_lookup_table = table_info->vddc_lookup_table; + + max_vddc_level = vddc_lookup_table->count; + for (i = 0; i < max_vddc_level; i++) { + soc_vid = (uint8_t)convert_to_vid(vddc_lookup_table->entries[i].us_vdd); + pp_table->SocVid[i] = soc_vid; + } + while (i < MAX_REGULAR_DPM_NUMBER) { + pp_table->SocVid[i] = soc_vid; + i++; + } +} + /** * @brief Populates single SMC GFXCLK structure using the provided clock. * @@ -1705,25 +1716,25 @@ static int vega10_populate_single_memory_level(struct pp_hwmgr *hwmgr, struct vega10_hwmgr *data = hwmgr->backend; struct phm_ppt_v2_information *table_info = (struct phm_ppt_v2_information *)(hwmgr->pptable); - struct phm_ppt_v1_clock_voltage_dependency_table *dep_on_mclk = - table_info->vdd_dep_on_mclk; + struct phm_ppt_v1_clock_voltage_dependency_table *dep_on_mclk; struct pp_atomfwctrl_clock_dividers_soc15 dividers; uint32_t mem_max_clock = hwmgr->platform_descriptor.overdriveLimit.memoryClock; uint32_t i = 0; - if (data->apply_overdrive_next_settings_mask & - DPMTABLE_OD_UPDATE_VDDC) + if (hwmgr->od_enabled) dep_on_mclk = (struct phm_ppt_v1_clock_voltage_dependency_table *) - &data->odn_dpm_table.vdd_dependency_on_mclk; + &data->odn_dpm_table.vdd_dep_on_mclk; + else + dep_on_mclk = table_info->vdd_dep_on_mclk; PP_ASSERT_WITH_CODE(dep_on_mclk, "Invalid SOC_VDD-UCLK Dependency Table!", return -EINVAL); - if (data->need_update_dpm_table & DPMTABLE_OD_UPDATE_MCLK) + if (data->need_update_dpm_table & DPMTABLE_OD_UPDATE_MCLK) { mem_clock = mem_clock > mem_max_clock ? mem_max_clock : mem_clock; - else { + } else { for (i = 0; i < dep_on_mclk->count; i++) { if (dep_on_mclk->entries[i].clk == mem_clock) break; @@ -2067,6 +2078,9 @@ static int vega10_populate_avfs_parameters(struct pp_hwmgr *hwmgr) if (data->smu_features[GNLD_AVFS].supported) { result = pp_atomfwctrl_get_avfs_information(hwmgr, &avfs_params); if (!result) { + data->odn_dpm_table.max_vddc = avfs_params.ulMaxVddc; + data->odn_dpm_table.min_vddc = avfs_params.ulMinVddc; + pp_table->MinVoltageVid = (uint8_t) convert_to_vid((uint16_t)(avfs_params.ulMinVddc)); pp_table->MaxVoltageVid = (uint8_t) @@ -2345,6 +2359,22 @@ static int vega10_avfs_enable(struct pp_hwmgr *hwmgr, bool enable) return 0; } +static int vega10_update_avfs(struct pp_hwmgr *hwmgr) +{ + struct vega10_hwmgr *data = hwmgr->backend; + + if (data->need_update_dpm_table & DPMTABLE_OD_UPDATE_VDDC) { + vega10_avfs_enable(hwmgr, false); + } else if (data->need_update_dpm_table) { + vega10_avfs_enable(hwmgr, false); + vega10_avfs_enable(hwmgr, true); + } else { + vega10_avfs_enable(hwmgr, true); + } + + return 0; +} + static int vega10_populate_and_upload_avfs_fuse_override(struct pp_hwmgr *hwmgr) { int result = 0; @@ -2406,6 +2436,10 @@ static int vega10_init_smc_table(struct pp_hwmgr *hwmgr) "Failed to setup default DPM tables!", return result); + /* initialize ODN table */ + if (hwmgr->od_enabled) + vega10_odn_initial_default_setting(hwmgr); + pp_atomfwctrl_get_voltage_table_v4(hwmgr, VOLTAGE_TYPE_VDDC, VOLTAGE_OBJ_SVID2, &voltage_table); pp_table->MaxVidStep = voltage_table.max_vid_step; @@ -2452,6 +2486,8 @@ static int vega10_init_smc_table(struct pp_hwmgr *hwmgr) "Failed to initialize Memory Level!", return result); + vega10_populate_vddc_soc_levels(hwmgr); + result = vega10_populate_all_display_clock_levels(hwmgr); PP_ASSERT_WITH_CODE(!result, "Failed to initialize Display Level!", @@ -2481,6 +2517,12 @@ static int vega10_init_smc_table(struct pp_hwmgr *hwmgr) data->vbios_boot_state.mvddc = boot_up_values.usMvddc; data->vbios_boot_state.gfx_clock = boot_up_values.ulGfxClk; data->vbios_boot_state.mem_clock = boot_up_values.ulUClk; + pp_atomfwctrl_get_clk_information_by_clkid(hwmgr, + SMU9_SYSPLL0_SOCCLK_ID, &boot_up_values.ulSocClk); + + pp_atomfwctrl_get_clk_information_by_clkid(hwmgr, + SMU9_SYSPLL0_DCEFCLK_ID, &boot_up_values.ulDCEFClk); + data->vbios_boot_state.soc_clock = boot_up_values.ulSocClk; data->vbios_boot_state.dcef_clock = boot_up_values.ulDCEFClk; if (0 != boot_up_values.usVddc) { @@ -2829,7 +2871,7 @@ static int vega10_enable_dpm_tasks(struct pp_hwmgr *hwmgr) tmp_result = vega10_construct_voltage_tables(hwmgr); PP_ASSERT_WITH_CODE(!tmp_result, - "Failed to contruct voltage tables!", + "Failed to construct voltage tables!", result = tmp_result); tmp_result = vega10_init_smc_table(hwmgr); @@ -3028,7 +3070,6 @@ static int vega10_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, bool disable_mclk_switching_for_frame_lock; bool disable_mclk_switching_for_vr; bool force_mclk_high; - struct cgs_display_info info = {0}; const struct phm_clock_and_voltage_limits *max_limits; uint32_t i; struct vega10_hwmgr *data = hwmgr->backend; @@ -3063,11 +3104,9 @@ static int vega10_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, } } - cgs_get_active_displays_info(hwmgr->device, &info); - /* result = PHM_CheckVBlankTime(hwmgr, &vblankTooShort);*/ - minimum_clocks.engineClock = hwmgr->display_config.min_core_set_clock; - minimum_clocks.memoryClock = hwmgr->display_config.min_mem_set_clock; + minimum_clocks.engineClock = hwmgr->display_config->min_core_set_clock; + minimum_clocks.memoryClock = hwmgr->display_config->min_mem_set_clock; if (PP_CAP(PHM_PlatformCaps_StablePState)) { stable_pstate_sclk_dpm_percentage = @@ -3107,10 +3146,10 @@ static int vega10_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, PP_CAP(PHM_PlatformCaps_DisableMclkSwitchForVR); force_mclk_high = PP_CAP(PHM_PlatformCaps_ForceMclkHigh); - if (info.display_count == 0) + if (hwmgr->display_config->num_display == 0) disable_mclk_switching = false; else - disable_mclk_switching = (info.display_count > 1) || + disable_mclk_switching = (hwmgr->display_config->num_display > 1) || disable_mclk_switching_for_frame_lock || disable_mclk_switching_for_vr || force_mclk_high; @@ -3171,87 +3210,11 @@ static int vega10_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, static int vega10_find_dpm_states_clocks_in_dpm_table(struct pp_hwmgr *hwmgr, const void *input) { - const struct phm_set_power_state_input *states = - (const struct phm_set_power_state_input *)input; - const struct vega10_power_state *vega10_ps = - cast_const_phw_vega10_power_state(states->pnew_state); struct vega10_hwmgr *data = hwmgr->backend; - struct vega10_single_dpm_table *sclk_table = - &(data->dpm_table.gfx_table); - uint32_t sclk = vega10_ps->performance_levels - [vega10_ps->performance_level_count - 1].gfx_clock; - struct vega10_single_dpm_table *mclk_table = - &(data->dpm_table.mem_table); - uint32_t mclk = vega10_ps->performance_levels - [vega10_ps->performance_level_count - 1].mem_clock; - struct PP_Clocks min_clocks = {0}; - uint32_t i; - struct cgs_display_info info = {0}; - - data->need_update_dpm_table = 0; - - if (PP_CAP(PHM_PlatformCaps_ODNinACSupport) || - PP_CAP(PHM_PlatformCaps_ODNinDCSupport)) { - for (i = 0; i < sclk_table->count; i++) { - if (sclk == sclk_table->dpm_levels[i].value) - break; - } - - if (!(data->apply_overdrive_next_settings_mask & - DPMTABLE_OD_UPDATE_SCLK) && i >= sclk_table->count) { - /* Check SCLK in DAL's minimum clocks - * in case DeepSleep divider update is required. - */ - if (data->display_timing.min_clock_in_sr != - min_clocks.engineClockInSR && - (min_clocks.engineClockInSR >= - VEGA10_MINIMUM_ENGINE_CLOCK || - data->display_timing.min_clock_in_sr >= - VEGA10_MINIMUM_ENGINE_CLOCK)) - data->need_update_dpm_table |= DPMTABLE_UPDATE_SCLK; - } - - cgs_get_active_displays_info(hwmgr->device, &info); - - if (data->display_timing.num_existing_displays != - info.display_count) - data->need_update_dpm_table |= DPMTABLE_UPDATE_MCLK; - } else { - for (i = 0; i < sclk_table->count; i++) { - if (sclk == sclk_table->dpm_levels[i].value) - break; - } - - if (i >= sclk_table->count) - data->need_update_dpm_table |= DPMTABLE_OD_UPDATE_SCLK; - else { - /* Check SCLK in DAL's minimum clocks - * in case DeepSleep divider update is required. - */ - if (data->display_timing.min_clock_in_sr != - min_clocks.engineClockInSR && - (min_clocks.engineClockInSR >= - VEGA10_MINIMUM_ENGINE_CLOCK || - data->display_timing.min_clock_in_sr >= - VEGA10_MINIMUM_ENGINE_CLOCK)) - data->need_update_dpm_table |= DPMTABLE_UPDATE_SCLK; - } - - for (i = 0; i < mclk_table->count; i++) { - if (mclk == mclk_table->dpm_levels[i].value) - break; - } - - cgs_get_active_displays_info(hwmgr->device, &info); - if (i >= mclk_table->count) - data->need_update_dpm_table |= DPMTABLE_OD_UPDATE_MCLK; + if (data->display_timing.num_existing_displays != hwmgr->display_config->num_display) + data->need_update_dpm_table |= DPMTABLE_UPDATE_MCLK; - if (data->display_timing.num_existing_displays != - info.display_count || - i >= mclk_table->count) - data->need_update_dpm_table |= DPMTABLE_UPDATE_MCLK; - } return 0; } @@ -3259,194 +3222,29 @@ static int vega10_populate_and_upload_sclk_mclk_dpm_levels( struct pp_hwmgr *hwmgr, const void *input) { int result = 0; - const struct phm_set_power_state_input *states = - (const struct phm_set_power_state_input *)input; - const struct vega10_power_state *vega10_ps = - cast_const_phw_vega10_power_state(states->pnew_state); struct vega10_hwmgr *data = hwmgr->backend; - uint32_t sclk = vega10_ps->performance_levels - [vega10_ps->performance_level_count - 1].gfx_clock; - uint32_t mclk = vega10_ps->performance_levels - [vega10_ps->performance_level_count - 1].mem_clock; - struct vega10_dpm_table *dpm_table = &data->dpm_table; - struct vega10_dpm_table *golden_dpm_table = - &data->golden_dpm_table; - uint32_t dpm_count, clock_percent; - uint32_t i; - - if (PP_CAP(PHM_PlatformCaps_ODNinACSupport) || - PP_CAP(PHM_PlatformCaps_ODNinDCSupport)) { - - if (!data->need_update_dpm_table && - !data->apply_optimized_settings && - !data->apply_overdrive_next_settings_mask) - return 0; - - if (data->apply_overdrive_next_settings_mask & - DPMTABLE_OD_UPDATE_SCLK) { - for (dpm_count = 0; - dpm_count < dpm_table->gfx_table.count; - dpm_count++) { - dpm_table->gfx_table.dpm_levels[dpm_count].enabled = - data->odn_dpm_table.odn_core_clock_dpm_levels.entries[dpm_count].enabled; - dpm_table->gfx_table.dpm_levels[dpm_count].value = - data->odn_dpm_table.odn_core_clock_dpm_levels.entries[dpm_count].clock; - } - } - - if (data->apply_overdrive_next_settings_mask & - DPMTABLE_OD_UPDATE_MCLK) { - for (dpm_count = 0; - dpm_count < dpm_table->mem_table.count; - dpm_count++) { - dpm_table->mem_table.dpm_levels[dpm_count].enabled = - data->odn_dpm_table.odn_memory_clock_dpm_levels.entries[dpm_count].enabled; - dpm_table->mem_table.dpm_levels[dpm_count].value = - data->odn_dpm_table.odn_memory_clock_dpm_levels.entries[dpm_count].clock; - } - } - if ((data->need_update_dpm_table & DPMTABLE_UPDATE_SCLK) || - data->apply_optimized_settings || - (data->apply_overdrive_next_settings_mask & - DPMTABLE_OD_UPDATE_SCLK)) { - result = vega10_populate_all_graphic_levels(hwmgr); - PP_ASSERT_WITH_CODE(!result, - "Failed to populate SCLK during PopulateNewDPMClocksStates Function!", - return result); - } - - if ((data->need_update_dpm_table & DPMTABLE_UPDATE_MCLK) || - (data->apply_overdrive_next_settings_mask & - DPMTABLE_OD_UPDATE_MCLK)){ - result = vega10_populate_all_memory_levels(hwmgr); - PP_ASSERT_WITH_CODE(!result, - "Failed to populate MCLK during PopulateNewDPMClocksStates Function!", - return result); - } - } else { - if (!data->need_update_dpm_table && - !data->apply_optimized_settings) - return 0; - - if (data->need_update_dpm_table & DPMTABLE_OD_UPDATE_SCLK && - data->smu_features[GNLD_DPM_GFXCLK].supported) { - dpm_table-> - gfx_table.dpm_levels[dpm_table->gfx_table.count - 1]. - value = sclk; - if (hwmgr->od_enabled) { - /* Need to do calculation based on the golden DPM table - * as the Heatmap GPU Clock axis is also based on - * the default values - */ - PP_ASSERT_WITH_CODE( - golden_dpm_table->gfx_table.dpm_levels - [golden_dpm_table->gfx_table.count - 1].value, - "Divide by 0!", - return -1); - - dpm_count = dpm_table->gfx_table.count < 2 ? - 0 : dpm_table->gfx_table.count - 2; - for (i = dpm_count; i > 1; i--) { - if (sclk > golden_dpm_table->gfx_table.dpm_levels - [golden_dpm_table->gfx_table.count - 1].value) { - clock_percent = - ((sclk - golden_dpm_table->gfx_table.dpm_levels - [golden_dpm_table->gfx_table.count - 1].value) * - 100) / - golden_dpm_table->gfx_table.dpm_levels - [golden_dpm_table->gfx_table.count - 1].value; - - dpm_table->gfx_table.dpm_levels[i].value = - golden_dpm_table->gfx_table.dpm_levels[i].value + - (golden_dpm_table->gfx_table.dpm_levels[i].value * - clock_percent) / 100; - } else if (golden_dpm_table-> - gfx_table.dpm_levels[dpm_table->gfx_table.count-1].value > - sclk) { - clock_percent = - ((golden_dpm_table->gfx_table.dpm_levels - [golden_dpm_table->gfx_table.count - 1].value - - sclk) * 100) / - golden_dpm_table->gfx_table.dpm_levels - [golden_dpm_table->gfx_table.count-1].value; - - dpm_table->gfx_table.dpm_levels[i].value = - golden_dpm_table->gfx_table.dpm_levels[i].value - - (golden_dpm_table->gfx_table.dpm_levels[i].value * - clock_percent) / 100; - } else - dpm_table->gfx_table.dpm_levels[i].value = - golden_dpm_table->gfx_table.dpm_levels[i].value; - } - } - } - - if (data->need_update_dpm_table & DPMTABLE_OD_UPDATE_MCLK && - data->smu_features[GNLD_DPM_UCLK].supported) { - dpm_table-> - mem_table.dpm_levels[dpm_table->mem_table.count - 1]. - value = mclk; + if (!data->need_update_dpm_table) + return 0; - if (hwmgr->od_enabled) { - PP_ASSERT_WITH_CODE( - golden_dpm_table->mem_table.dpm_levels - [golden_dpm_table->mem_table.count - 1].value, - "Divide by 0!", - return -1); + if (data->need_update_dpm_table & + (DPMTABLE_OD_UPDATE_SCLK + DPMTABLE_UPDATE_SCLK + DPMTABLE_UPDATE_SOCCLK)) { + result = vega10_populate_all_graphic_levels(hwmgr); + PP_ASSERT_WITH_CODE((0 == result), + "Failed to populate SCLK during PopulateNewDPMClocksStates Function!", + return result); + } - dpm_count = dpm_table->mem_table.count < 2 ? - 0 : dpm_table->mem_table.count - 2; - for (i = dpm_count; i > 1; i--) { - if (mclk > golden_dpm_table->mem_table.dpm_levels - [golden_dpm_table->mem_table.count-1].value) { - clock_percent = ((mclk - - golden_dpm_table->mem_table.dpm_levels - [golden_dpm_table->mem_table.count-1].value) * - 100) / - golden_dpm_table->mem_table.dpm_levels - [golden_dpm_table->mem_table.count-1].value; - - dpm_table->mem_table.dpm_levels[i].value = - golden_dpm_table->mem_table.dpm_levels[i].value + - (golden_dpm_table->mem_table.dpm_levels[i].value * - clock_percent) / 100; - } else if (golden_dpm_table->mem_table.dpm_levels - [dpm_table->mem_table.count-1].value > mclk) { - clock_percent = ((golden_dpm_table->mem_table.dpm_levels - [golden_dpm_table->mem_table.count-1].value - mclk) * - 100) / - golden_dpm_table->mem_table.dpm_levels - [golden_dpm_table->mem_table.count-1].value; - - dpm_table->mem_table.dpm_levels[i].value = - golden_dpm_table->mem_table.dpm_levels[i].value - - (golden_dpm_table->mem_table.dpm_levels[i].value * - clock_percent) / 100; - } else - dpm_table->mem_table.dpm_levels[i].value = - golden_dpm_table->mem_table.dpm_levels[i].value; - } - } - } + if (data->need_update_dpm_table & + (DPMTABLE_OD_UPDATE_MCLK + DPMTABLE_UPDATE_MCLK)) { + result = vega10_populate_all_memory_levels(hwmgr); + PP_ASSERT_WITH_CODE((0 == result), + "Failed to populate MCLK during PopulateNewDPMClocksStates Function!", + return result); + } - if ((data->need_update_dpm_table & - (DPMTABLE_OD_UPDATE_SCLK + DPMTABLE_UPDATE_SCLK)) || - data->apply_optimized_settings) { - result = vega10_populate_all_graphic_levels(hwmgr); - PP_ASSERT_WITH_CODE(!result, - "Failed to populate SCLK during PopulateNewDPMClocksStates Function!", - return result); - } + vega10_populate_vddc_soc_levels(hwmgr); - if (data->need_update_dpm_table & - (DPMTABLE_OD_UPDATE_MCLK + DPMTABLE_UPDATE_MCLK)) { - result = vega10_populate_all_memory_levels(hwmgr); - PP_ASSERT_WITH_CODE(!result, - "Failed to populate MCLK during PopulateNewDPMClocksStates Function!", - return result); - } - } return result; } @@ -3742,8 +3540,9 @@ static int vega10_set_power_state_tasks(struct pp_hwmgr *hwmgr, PP_ASSERT_WITH_CODE(!result, "Failed to upload PPtable!", return result); - data->apply_optimized_settings = false; - data->apply_overdrive_next_settings_mask = 0; + vega10_update_avfs(hwmgr); + + data->need_update_dpm_table &= DPMTABLE_OD_UPDATE_VDDC; return 0; } @@ -3793,16 +3592,18 @@ static uint32_t vega10_dpm_get_mclk(struct pp_hwmgr *hwmgr, bool low) } static int vega10_get_gpu_power(struct pp_hwmgr *hwmgr, - struct pp_gpu_power *query) + uint32_t *query) { uint32_t value; + if (!query) + return -EINVAL; + smum_send_msg_to_smc(hwmgr, PPSMC_MSG_GetCurrPkgPwr); value = smum_get_argument(hwmgr); - /* power value is an integer */ - memset(query, 0, sizeof *query); - query->average_gpu_power = value << 8; + /* SMC returning actual watts, keep consistent with legacy asics, low 8 bit as 8 fractional bits */ + *query = value << 8; return 0; } @@ -3810,22 +3611,18 @@ static int vega10_get_gpu_power(struct pp_hwmgr *hwmgr, static int vega10_read_sensor(struct pp_hwmgr *hwmgr, int idx, void *value, int *size) { - uint32_t sclk_idx, mclk_idx, activity_percent = 0; + struct amdgpu_device *adev = hwmgr->adev; + uint32_t sclk_mhz, mclk_idx, activity_percent = 0; struct vega10_hwmgr *data = hwmgr->backend; struct vega10_dpm_table *dpm_table = &data->dpm_table; int ret = 0; - uint32_t reg, val_vid; + uint32_t val_vid; switch (idx) { case AMDGPU_PP_SENSOR_GFX_SCLK: - smum_send_msg_to_smc(hwmgr, PPSMC_MSG_GetCurrentGfxclkIndex); - sclk_idx = smum_get_argument(hwmgr); - if (sclk_idx < dpm_table->gfx_table.count) { - *((uint32_t *)value) = dpm_table->gfx_table.dpm_levels[sclk_idx].value; - *size = 4; - } else { - ret = -EINVAL; - } + smum_send_msg_to_smc(hwmgr, PPSMC_MSG_GetAverageGfxclkActualFrequency); + sclk_mhz = smum_get_argument(hwmgr); + *((uint32_t *)value) = sclk_mhz * 100; break; case AMDGPU_PP_SENSOR_GFX_MCLK: smum_send_msg_to_smc(hwmgr, PPSMC_MSG_GetCurrentUclkIndex); @@ -3856,18 +3653,10 @@ static int vega10_read_sensor(struct pp_hwmgr *hwmgr, int idx, *size = 4; break; case AMDGPU_PP_SENSOR_GPU_POWER: - if (*size < sizeof(struct pp_gpu_power)) - ret = -EINVAL; - else { - *size = sizeof(struct pp_gpu_power); - ret = vega10_get_gpu_power(hwmgr, (struct pp_gpu_power *)value); - } + ret = vega10_get_gpu_power(hwmgr, (uint32_t *)value); break; case AMDGPU_PP_SENSOR_VDDGFX: - reg = soc15_get_register_offset(SMUIO_HWID, 0, - mmSMUSVI0_PLANE0_CURRENTVID_BASE_IDX, - mmSMUSVI0_PLANE0_CURRENTVID); - val_vid = (cgs_read_register(hwmgr->device, reg) & + val_vid = (RREG32_SOC15(SMUIO, 0, mmSMUSVI0_PLANE0_CURRENTVID) & SMUSVI0_PLANE0_CURRENTVID__CURRENT_SVI0_PLANE0_VID_MASK) >> SMUSVI0_PLANE0_CURRENTVID__CURRENT_SVI0_PLANE0_VID__SHIFT; *((uint32_t *)value) = (uint32_t)convert_to_vddc((uint8_t)val_vid); @@ -3956,26 +3745,18 @@ static int vega10_notify_smc_display_config_after_ps_adjustment( (struct phm_ppt_v2_information *)hwmgr->pptable; struct phm_ppt_v1_clock_voltage_dependency_table *mclk_table = table_info->vdd_dep_on_mclk; uint32_t idx; - uint32_t num_active_disps = 0; - struct cgs_display_info info = {0}; struct PP_Clocks min_clocks = {0}; uint32_t i; struct pp_display_clock_request clock_req; - info.mode_info = NULL; - - cgs_get_active_displays_info(hwmgr->device, &info); - - num_active_disps = info.display_count; - - if (num_active_disps > 1) + if (hwmgr->display_config->num_display > 1) vega10_notify_smc_display_change(hwmgr, false); else vega10_notify_smc_display_change(hwmgr, true); - min_clocks.dcefClock = hwmgr->display_config.min_dcef_set_clk; - min_clocks.dcefClockInSR = hwmgr->display_config.min_dcef_deep_sleep_set_clk; - min_clocks.memoryClock = hwmgr->display_config.min_mem_set_clock; + min_clocks.dcefClock = hwmgr->display_config->min_dcef_set_clk; + min_clocks.dcefClockInSR = hwmgr->display_config->min_dcef_deep_sleep_set_clk; + min_clocks.memoryClock = hwmgr->display_config->min_mem_set_clock; for (i = 0; i < dpm_table->count; i++) { if (dpm_table->dpm_levels[i].value == min_clocks.dcefClock) @@ -4120,6 +3901,47 @@ static void vega10_set_fan_control_mode(struct pp_hwmgr *hwmgr, uint32_t mode) } } +static int vega10_force_clock_level(struct pp_hwmgr *hwmgr, + enum pp_clock_type type, uint32_t mask) +{ + struct vega10_hwmgr *data = hwmgr->backend; + + switch (type) { + case PP_SCLK: + data->smc_state_table.gfx_boot_level = mask ? (ffs(mask) - 1) : 0; + data->smc_state_table.gfx_max_level = mask ? (fls(mask) - 1) : 0; + + PP_ASSERT_WITH_CODE(!vega10_upload_dpm_bootup_level(hwmgr), + "Failed to upload boot level to lowest!", + return -EINVAL); + + PP_ASSERT_WITH_CODE(!vega10_upload_dpm_max_level(hwmgr), + "Failed to upload dpm max level to highest!", + return -EINVAL); + break; + + case PP_MCLK: + data->smc_state_table.mem_boot_level = mask ? (ffs(mask) - 1) : 0; + data->smc_state_table.mem_max_level = mask ? (fls(mask) - 1) : 0; + + PP_ASSERT_WITH_CODE(!vega10_upload_dpm_bootup_level(hwmgr), + "Failed to upload boot level to lowest!", + return -EINVAL); + + PP_ASSERT_WITH_CODE(!vega10_upload_dpm_max_level(hwmgr), + "Failed to upload dpm max level to highest!", + return -EINVAL); + + break; + + case PP_PCIE: + default: + break; + } + + return 0; +} + static int vega10_dpm_force_dpm_level(struct pp_hwmgr *hwmgr, enum amd_dpm_forced_level level) { @@ -4356,97 +4178,15 @@ static int vega10_set_watermarks_for_clocks_ranges(struct pp_hwmgr *hwmgr, struct vega10_hwmgr *data = hwmgr->backend; Watermarks_t *table = &(data->smc_state_table.water_marks_table); int result = 0; - uint32_t i; if (!data->registry_data.disable_water_mark) { - for (i = 0; i < wm_with_clock_ranges->num_wm_sets_dmif; i++) { - table->WatermarkRow[WM_DCEFCLK][i].MinClock = - cpu_to_le16((uint16_t) - (wm_with_clock_ranges->wm_sets_dmif[i].wm_min_dcefclk_in_khz) / - 100); - table->WatermarkRow[WM_DCEFCLK][i].MaxClock = - cpu_to_le16((uint16_t) - (wm_with_clock_ranges->wm_sets_dmif[i].wm_max_dcefclk_in_khz) / - 100); - table->WatermarkRow[WM_DCEFCLK][i].MinUclk = - cpu_to_le16((uint16_t) - (wm_with_clock_ranges->wm_sets_dmif[i].wm_min_memclk_in_khz) / - 100); - table->WatermarkRow[WM_DCEFCLK][i].MaxUclk = - cpu_to_le16((uint16_t) - (wm_with_clock_ranges->wm_sets_dmif[i].wm_max_memclk_in_khz) / - 100); - table->WatermarkRow[WM_DCEFCLK][i].WmSetting = (uint8_t) - wm_with_clock_ranges->wm_sets_dmif[i].wm_set_id; - } - - for (i = 0; i < wm_with_clock_ranges->num_wm_sets_mcif; i++) { - table->WatermarkRow[WM_SOCCLK][i].MinClock = - cpu_to_le16((uint16_t) - (wm_with_clock_ranges->wm_sets_mcif[i].wm_min_socclk_in_khz) / - 100); - table->WatermarkRow[WM_SOCCLK][i].MaxClock = - cpu_to_le16((uint16_t) - (wm_with_clock_ranges->wm_sets_mcif[i].wm_max_socclk_in_khz) / - 100); - table->WatermarkRow[WM_SOCCLK][i].MinUclk = - cpu_to_le16((uint16_t) - (wm_with_clock_ranges->wm_sets_mcif[i].wm_min_memclk_in_khz) / - 100); - table->WatermarkRow[WM_SOCCLK][i].MaxUclk = - cpu_to_le16((uint16_t) - (wm_with_clock_ranges->wm_sets_mcif[i].wm_max_memclk_in_khz) / - 100); - table->WatermarkRow[WM_SOCCLK][i].WmSetting = (uint8_t) - wm_with_clock_ranges->wm_sets_mcif[i].wm_set_id; - } + smu_set_watermarks_for_clocks_ranges(table, wm_with_clock_ranges); data->water_marks_bitmap = WaterMarksExist; } return result; } -static int vega10_force_clock_level(struct pp_hwmgr *hwmgr, - enum pp_clock_type type, uint32_t mask) -{ - struct vega10_hwmgr *data = hwmgr->backend; - - switch (type) { - case PP_SCLK: - data->smc_state_table.gfx_boot_level = mask ? (ffs(mask) - 1) : 0; - data->smc_state_table.gfx_max_level = mask ? (fls(mask) - 1) : 0; - - PP_ASSERT_WITH_CODE(!vega10_upload_dpm_bootup_level(hwmgr), - "Failed to upload boot level to lowest!", - return -EINVAL); - - PP_ASSERT_WITH_CODE(!vega10_upload_dpm_max_level(hwmgr), - "Failed to upload dpm max level to highest!", - return -EINVAL); - break; - - case PP_MCLK: - data->smc_state_table.mem_boot_level = mask ? (ffs(mask) - 1) : 0; - data->smc_state_table.mem_max_level = mask ? (fls(mask) - 1) : 0; - - PP_ASSERT_WITH_CODE(!vega10_upload_dpm_bootup_level(hwmgr), - "Failed to upload boot level to lowest!", - return -EINVAL); - - PP_ASSERT_WITH_CODE(!vega10_upload_dpm_max_level(hwmgr), - "Failed to upload dpm max level to highest!", - return -EINVAL); - - break; - - case PP_PCIE: - default: - break; - } - - return 0; -} - static int vega10_print_clock_levels(struct pp_hwmgr *hwmgr, enum pp_clock_type type, char *buf) { @@ -4454,6 +4194,8 @@ static int vega10_print_clock_levels(struct pp_hwmgr *hwmgr, struct vega10_single_dpm_table *sclk_table = &(data->dpm_table.gfx_table); struct vega10_single_dpm_table *mclk_table = &(data->dpm_table.mem_table); struct vega10_pcie_table *pcie_table = &(data->dpm_table.pcie_table); + struct vega10_odn_clock_voltage_dependency_table *podn_vdd_dep = NULL; + int i, now, size = 0; switch (type) { @@ -4492,6 +4234,40 @@ static int vega10_print_clock_levels(struct pp_hwmgr *hwmgr, (pcie_table->pcie_gen[i] == 2) ? "8.0GT/s, x16" : "", (i == now) ? "*" : ""); break; + case OD_SCLK: + if (hwmgr->od_enabled) { + size = sprintf(buf, "%s:\n", "OD_SCLK"); + podn_vdd_dep = &data->odn_dpm_table.vdd_dep_on_sclk; + for (i = 0; i < podn_vdd_dep->count; i++) + size += sprintf(buf + size, "%d: %10uMhz %10umV\n", + i, podn_vdd_dep->entries[i].clk / 100, + podn_vdd_dep->entries[i].vddc); + } + break; + case OD_MCLK: + if (hwmgr->od_enabled) { + size = sprintf(buf, "%s:\n", "OD_MCLK"); + podn_vdd_dep = &data->odn_dpm_table.vdd_dep_on_mclk; + for (i = 0; i < podn_vdd_dep->count; i++) + size += sprintf(buf + size, "%d: %10uMhz %10umV\n", + i, podn_vdd_dep->entries[i].clk/100, + podn_vdd_dep->entries[i].vddc); + } + break; + case OD_RANGE: + if (hwmgr->od_enabled) { + size = sprintf(buf, "%s:\n", "OD_RANGE"); + size += sprintf(buf + size, "SCLK: %7uMHz %10uMHz\n", + data->golden_dpm_table.gfx_table.dpm_levels[0].value/100, + hwmgr->platform_descriptor.overdriveLimit.engineClock/100); + size += sprintf(buf + size, "MCLK: %7uMHz %10uMHz\n", + data->golden_dpm_table.mem_table.dpm_levels[0].value/100, + hwmgr->platform_descriptor.overdriveLimit.memoryClock/100); + size += sprintf(buf + size, "VDDC: %7umV %11umV\n", + data->odn_dpm_table.min_vddc, + data->odn_dpm_table.max_vddc); + } + break; default: break; } @@ -4501,10 +4277,8 @@ static int vega10_print_clock_levels(struct pp_hwmgr *hwmgr, static int vega10_display_configuration_changed_task(struct pp_hwmgr *hwmgr) { struct vega10_hwmgr *data = hwmgr->backend; - int result = 0; - uint32_t num_turned_on_displays = 1; Watermarks_t *wm_table = &(data->smc_state_table.water_marks_table); - struct cgs_display_info info = {0}; + int result = 0; if ((data->water_marks_bitmap & WaterMarksExist) && !(data->water_marks_bitmap & WaterMarksLoaded)) { @@ -4514,10 +4288,8 @@ static int vega10_display_configuration_changed_task(struct pp_hwmgr *hwmgr) } if (data->water_marks_bitmap & WaterMarksLoaded) { - cgs_get_active_displays_info(hwmgr->device, &info); - num_turned_on_displays = info.display_count; smum_send_msg_to_smc_with_parameter(hwmgr, - PPSMC_MSG_NumOfDisplays, num_turned_on_displays); + PPSMC_MSG_NumOfDisplays, hwmgr->display_config->num_display); } return result; @@ -4603,15 +4375,12 @@ vega10_check_smc_update_required_for_display_configuration(struct pp_hwmgr *hwmg { struct vega10_hwmgr *data = hwmgr->backend; bool is_update_required = false; - struct cgs_display_info info = {0, 0, NULL}; - - cgs_get_active_displays_info(hwmgr->device, &info); - if (data->display_timing.num_existing_displays != info.display_count) + if (data->display_timing.num_existing_displays != hwmgr->display_config->num_display) is_update_required = true; if (PP_CAP(PHM_PlatformCaps_SclkDeepSleep)) { - if (data->display_timing.min_clock_in_sr != hwmgr->display_config.min_core_set_clock_in_sr) + if (data->display_timing.min_clock_in_sr != hwmgr->display_config->min_core_set_clock_in_sr) is_update_required = true; } @@ -4886,6 +4655,200 @@ static int vega10_set_power_profile_mode(struct pp_hwmgr *hwmgr, long *input, ui return 0; } + +static bool vega10_check_clk_voltage_valid(struct pp_hwmgr *hwmgr, + enum PP_OD_DPM_TABLE_COMMAND type, + uint32_t clk, + uint32_t voltage) +{ + struct vega10_hwmgr *data = hwmgr->backend; + struct vega10_odn_dpm_table *odn_table = &(data->odn_dpm_table); + struct vega10_single_dpm_table *golden_table; + + if (voltage < odn_table->min_vddc || voltage > odn_table->max_vddc) { + pr_info("OD voltage is out of range [%d - %d] mV\n", odn_table->min_vddc, odn_table->max_vddc); + return false; + } + + if (type == PP_OD_EDIT_SCLK_VDDC_TABLE) { + golden_table = &(data->golden_dpm_table.gfx_table); + if (golden_table->dpm_levels[0].value > clk || + hwmgr->platform_descriptor.overdriveLimit.engineClock < clk) { + pr_info("OD engine clock is out of range [%d - %d] MHz\n", + golden_table->dpm_levels[0].value/100, + hwmgr->platform_descriptor.overdriveLimit.engineClock/100); + return false; + } + } else if (type == PP_OD_EDIT_MCLK_VDDC_TABLE) { + golden_table = &(data->golden_dpm_table.mem_table); + if (golden_table->dpm_levels[0].value > clk || + hwmgr->platform_descriptor.overdriveLimit.memoryClock < clk) { + pr_info("OD memory clock is out of range [%d - %d] MHz\n", + golden_table->dpm_levels[0].value/100, + hwmgr->platform_descriptor.overdriveLimit.memoryClock/100); + return false; + } + } else { + return false; + } + + return true; +} + +static void vega10_check_dpm_table_updated(struct pp_hwmgr *hwmgr) +{ + struct vega10_hwmgr *data = hwmgr->backend; + struct vega10_odn_dpm_table *odn_table = &(data->odn_dpm_table); + struct phm_ppt_v2_information *table_info = hwmgr->pptable; + struct phm_ppt_v1_clock_voltage_dependency_table *dep_table; + struct phm_ppt_v1_clock_voltage_dependency_table *odn_dep_table; + uint32_t i; + + dep_table = table_info->vdd_dep_on_mclk; + odn_dep_table = (struct phm_ppt_v1_clock_voltage_dependency_table *)&(odn_table->vdd_dep_on_mclk); + + for (i = 0; i < dep_table->count; i++) { + if (dep_table->entries[i].vddc != odn_dep_table->entries[i].vddc) { + data->need_update_dpm_table |= DPMTABLE_OD_UPDATE_VDDC | DPMTABLE_OD_UPDATE_MCLK; + return; + } + } + + dep_table = table_info->vdd_dep_on_sclk; + odn_dep_table = (struct phm_ppt_v1_clock_voltage_dependency_table *)&(odn_table->vdd_dep_on_sclk); + for (i = 0; i < dep_table->count; i++) { + if (dep_table->entries[i].vddc != odn_dep_table->entries[i].vddc) { + data->need_update_dpm_table |= DPMTABLE_OD_UPDATE_VDDC | DPMTABLE_OD_UPDATE_SCLK; + return; + } + } + + if (data->need_update_dpm_table & DPMTABLE_OD_UPDATE_VDDC) { + data->need_update_dpm_table &= ~DPMTABLE_OD_UPDATE_VDDC; + data->need_update_dpm_table |= DPMTABLE_OD_UPDATE_SCLK | DPMTABLE_OD_UPDATE_MCLK; + } +} + +static void vega10_odn_update_soc_table(struct pp_hwmgr *hwmgr, + enum PP_OD_DPM_TABLE_COMMAND type) +{ + struct vega10_hwmgr *data = hwmgr->backend; + struct phm_ppt_v2_information *table_info = hwmgr->pptable; + struct phm_ppt_v1_clock_voltage_dependency_table *dep_table = table_info->vdd_dep_on_socclk; + struct vega10_single_dpm_table *dpm_table = &data->golden_dpm_table.soc_table; + + struct vega10_odn_clock_voltage_dependency_table *podn_vdd_dep_on_socclk = + &data->odn_dpm_table.vdd_dep_on_socclk; + struct vega10_odn_vddc_lookup_table *od_vddc_lookup_table = &data->odn_dpm_table.vddc_lookup_table; + + struct vega10_odn_clock_voltage_dependency_table *podn_vdd_dep; + uint8_t i, j; + + if (type == PP_OD_EDIT_SCLK_VDDC_TABLE) { + podn_vdd_dep = &data->odn_dpm_table.vdd_dep_on_sclk; + for (i = 0; i < podn_vdd_dep->count - 1; i++) + od_vddc_lookup_table->entries[i].us_vdd = podn_vdd_dep->entries[i].vddc; + if (od_vddc_lookup_table->entries[i].us_vdd < podn_vdd_dep->entries[i].vddc) + od_vddc_lookup_table->entries[i].us_vdd = podn_vdd_dep->entries[i].vddc; + } else if (type == PP_OD_EDIT_MCLK_VDDC_TABLE) { + podn_vdd_dep = &data->odn_dpm_table.vdd_dep_on_mclk; + for (i = 0; i < dpm_table->count; i++) { + for (j = 0; j < od_vddc_lookup_table->count; j++) { + if (od_vddc_lookup_table->entries[j].us_vdd > + podn_vdd_dep->entries[i].vddc) + break; + } + if (j == od_vddc_lookup_table->count) { + od_vddc_lookup_table->entries[j-1].us_vdd = + podn_vdd_dep->entries[i].vddc; + data->need_update_dpm_table |= DPMTABLE_OD_UPDATE_VDDC; + } + podn_vdd_dep->entries[i].vddInd = j; + } + dpm_table = &data->dpm_table.soc_table; + for (i = 0; i < dep_table->count; i++) { + if (dep_table->entries[i].vddInd == podn_vdd_dep->entries[dep_table->count-1].vddInd && + dep_table->entries[i].clk < podn_vdd_dep->entries[dep_table->count-1].clk) { + data->need_update_dpm_table |= DPMTABLE_UPDATE_SOCCLK; + podn_vdd_dep_on_socclk->entries[i].clk = podn_vdd_dep->entries[dep_table->count-1].clk; + dpm_table->dpm_levels[i].value = podn_vdd_dep_on_socclk->entries[i].clk; + } + } + if (podn_vdd_dep_on_socclk->entries[podn_vdd_dep_on_socclk->count - 1].clk < + podn_vdd_dep->entries[dep_table->count-1].clk) { + data->need_update_dpm_table |= DPMTABLE_UPDATE_SOCCLK; + podn_vdd_dep_on_socclk->entries[podn_vdd_dep_on_socclk->count - 1].clk = podn_vdd_dep->entries[dep_table->count-1].clk; + dpm_table->dpm_levels[podn_vdd_dep_on_socclk->count - 1].value = podn_vdd_dep->entries[dep_table->count-1].clk; + } + if (podn_vdd_dep_on_socclk->entries[podn_vdd_dep_on_socclk->count - 1].vddInd < + podn_vdd_dep->entries[dep_table->count-1].vddInd) { + data->need_update_dpm_table |= DPMTABLE_UPDATE_SOCCLK; + podn_vdd_dep_on_socclk->entries[podn_vdd_dep_on_socclk->count - 1].vddInd = podn_vdd_dep->entries[dep_table->count-1].vddInd; + } + } +} + +static int vega10_odn_edit_dpm_table(struct pp_hwmgr *hwmgr, + enum PP_OD_DPM_TABLE_COMMAND type, + long *input, uint32_t size) +{ + struct vega10_hwmgr *data = hwmgr->backend; + struct vega10_odn_clock_voltage_dependency_table *podn_vdd_dep_table; + struct vega10_single_dpm_table *dpm_table; + + uint32_t input_clk; + uint32_t input_vol; + uint32_t input_level; + uint32_t i; + + PP_ASSERT_WITH_CODE(input, "NULL user input for clock and voltage", + return -EINVAL); + + if (!hwmgr->od_enabled) { + pr_info("OverDrive feature not enabled\n"); + return -EINVAL; + } + + if (PP_OD_EDIT_SCLK_VDDC_TABLE == type) { + dpm_table = &data->dpm_table.gfx_table; + podn_vdd_dep_table = &data->odn_dpm_table.vdd_dep_on_sclk; + data->need_update_dpm_table |= DPMTABLE_OD_UPDATE_SCLK; + } else if (PP_OD_EDIT_MCLK_VDDC_TABLE == type) { + dpm_table = &data->dpm_table.mem_table; + podn_vdd_dep_table = &data->odn_dpm_table.vdd_dep_on_mclk; + data->need_update_dpm_table |= DPMTABLE_OD_UPDATE_MCLK; + } else if (PP_OD_RESTORE_DEFAULT_TABLE == type) { + memcpy(&(data->dpm_table), &(data->golden_dpm_table), sizeof(struct vega10_dpm_table)); + vega10_odn_initial_default_setting(hwmgr); + return 0; + } else if (PP_OD_COMMIT_DPM_TABLE == type) { + vega10_check_dpm_table_updated(hwmgr); + return 0; + } else { + return -EINVAL; + } + + for (i = 0; i < size; i += 3) { + if (i + 3 > size || input[i] >= podn_vdd_dep_table->count) { + pr_info("invalid clock voltage input\n"); + return 0; + } + input_level = input[i]; + input_clk = input[i+1] * 100; + input_vol = input[i+2]; + + if (vega10_check_clk_voltage_valid(hwmgr, type, input_clk, input_vol)) { + dpm_table->dpm_levels[input_level].value = input_clk; + podn_vdd_dep_table->entries[input_level].clk = input_clk; + podn_vdd_dep_table->entries[input_level].vddc = input_vol; + } else { + return -EINVAL; + } + } + vega10_odn_update_soc_table(hwmgr, type); + return 0; +} + static const struct pp_hwmgr_func vega10_hwmgr_funcs = { .backend_init = vega10_hwmgr_backend_init, .backend_fini = vega10_hwmgr_backend_fini, @@ -4944,6 +4907,7 @@ static const struct pp_hwmgr_func vega10_hwmgr_funcs = { .get_power_profile_mode = vega10_get_power_profile_mode, .set_power_profile_mode = vega10_set_power_profile_mode, .set_power_limit = vega10_set_power_limit, + .odn_edit_dpm_table = vega10_odn_edit_dpm_table, }; int vega10_enable_smc_features(struct pp_hwmgr *hwmgr, diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_hwmgr.h b/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_hwmgr.h index 5339ea1f3dce..aadd6cbc7e85 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_hwmgr.h +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_hwmgr.h @@ -282,15 +282,21 @@ struct vega10_registry_data { struct vega10_odn_clock_voltage_dependency_table { uint32_t count; - struct phm_ppt_v1_clock_voltage_dependency_record - entries[MAX_REGULAR_DPM_NUMBER]; + struct phm_ppt_v1_clock_voltage_dependency_record entries[MAX_REGULAR_DPM_NUMBER]; +}; + +struct vega10_odn_vddc_lookup_table { + uint32_t count; + struct phm_ppt_v1_voltage_lookup_record entries[MAX_REGULAR_DPM_NUMBER]; }; struct vega10_odn_dpm_table { - struct phm_odn_clock_levels odn_core_clock_dpm_levels; - struct phm_odn_clock_levels odn_memory_clock_dpm_levels; - struct vega10_odn_clock_voltage_dependency_table vdd_dependency_on_sclk; - struct vega10_odn_clock_voltage_dependency_table vdd_dependency_on_mclk; + struct vega10_odn_clock_voltage_dependency_table vdd_dep_on_sclk; + struct vega10_odn_clock_voltage_dependency_table vdd_dep_on_mclk; + struct vega10_odn_clock_voltage_dependency_table vdd_dep_on_socclk; + struct vega10_odn_vddc_lookup_table vddc_lookup_table; + uint32_t max_vddc; + uint32_t min_vddc; }; struct vega10_odn_fan_table { @@ -301,8 +307,8 @@ struct vega10_odn_fan_table { }; struct vega10_hwmgr { - struct vega10_dpm_table dpm_table; - struct vega10_dpm_table golden_dpm_table; + struct vega10_dpm_table dpm_table; + struct vega10_dpm_table golden_dpm_table; struct vega10_registry_data registry_data; struct vega10_vbios_boot_state vbios_boot_state; struct vega10_mclk_latency_table mclk_latency_table; @@ -368,12 +374,8 @@ struct vega10_hwmgr { bool need_long_memory_training; /* Internal settings to apply the application power optimization parameters */ - bool apply_optimized_settings; uint32_t disable_dpm_mask; - /* ---- Overdrive next setting ---- */ - uint32_t apply_overdrive_next_settings_mask; - /* ---- SMU9 ---- */ struct smu_features smu_features[GNLD_FEATURES_MAX]; struct vega10_smc_state_table smc_state_table; diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_powertune.c b/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_powertune.c index ba63faefc61f..dbe4b1f66784 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_powertune.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_powertune.c @@ -27,7 +27,7 @@ #include "vega10_ppsmc.h" #include "vega10_inc.h" #include "pp_debug.h" -#include "pp_soc15.h" +#include "soc15_common.h" static const struct vega10_didt_config_reg SEDiDtTuningCtrlConfig_Vega10[] = { @@ -888,36 +888,36 @@ static void vega10_didt_set_mask(struct pp_hwmgr *hwmgr, const bool enable) if (PP_CAP(PHM_PlatformCaps_DiDtEDCEnable)) { if (PP_CAP(PHM_PlatformCaps_SQRamping)) { data = cgs_read_ind_register(hwmgr->device, CGS_IND_REG__DIDT, ixDIDT_SQ_EDC_CTRL); - data = CGS_REG_SET_FIELD(data, DIDT_SQ_EDC_CTRL, EDC_EN, en); - data = CGS_REG_SET_FIELD(data, DIDT_SQ_EDC_CTRL, EDC_SW_RST, ~en); + data = REG_SET_FIELD(data, DIDT_SQ_EDC_CTRL, EDC_EN, en); + data = REG_SET_FIELD(data, DIDT_SQ_EDC_CTRL, EDC_SW_RST, ~en); cgs_write_ind_register(hwmgr->device, CGS_IND_REG__DIDT, ixDIDT_SQ_EDC_CTRL, data); } if (PP_CAP(PHM_PlatformCaps_DBRamping)) { data = cgs_read_ind_register(hwmgr->device, CGS_IND_REG__DIDT, ixDIDT_DB_EDC_CTRL); - data = CGS_REG_SET_FIELD(data, DIDT_DB_EDC_CTRL, EDC_EN, en); - data = CGS_REG_SET_FIELD(data, DIDT_DB_EDC_CTRL, EDC_SW_RST, ~en); + data = REG_SET_FIELD(data, DIDT_DB_EDC_CTRL, EDC_EN, en); + data = REG_SET_FIELD(data, DIDT_DB_EDC_CTRL, EDC_SW_RST, ~en); cgs_write_ind_register(hwmgr->device, CGS_IND_REG__DIDT, ixDIDT_DB_EDC_CTRL, data); } if (PP_CAP(PHM_PlatformCaps_TDRamping)) { data = cgs_read_ind_register(hwmgr->device, CGS_IND_REG__DIDT, ixDIDT_TD_EDC_CTRL); - data = CGS_REG_SET_FIELD(data, DIDT_TD_EDC_CTRL, EDC_EN, en); - data = CGS_REG_SET_FIELD(data, DIDT_TD_EDC_CTRL, EDC_SW_RST, ~en); + data = REG_SET_FIELD(data, DIDT_TD_EDC_CTRL, EDC_EN, en); + data = REG_SET_FIELD(data, DIDT_TD_EDC_CTRL, EDC_SW_RST, ~en); cgs_write_ind_register(hwmgr->device, CGS_IND_REG__DIDT, ixDIDT_TD_EDC_CTRL, data); } if (PP_CAP(PHM_PlatformCaps_TCPRamping)) { data = cgs_read_ind_register(hwmgr->device, CGS_IND_REG__DIDT, ixDIDT_TCP_EDC_CTRL); - data = CGS_REG_SET_FIELD(data, DIDT_TCP_EDC_CTRL, EDC_EN, en); - data = CGS_REG_SET_FIELD(data, DIDT_TCP_EDC_CTRL, EDC_SW_RST, ~en); + data = REG_SET_FIELD(data, DIDT_TCP_EDC_CTRL, EDC_EN, en); + data = REG_SET_FIELD(data, DIDT_TCP_EDC_CTRL, EDC_SW_RST, ~en); cgs_write_ind_register(hwmgr->device, CGS_IND_REG__DIDT, ixDIDT_TCP_EDC_CTRL, data); } if (PP_CAP(PHM_PlatformCaps_DBRRamping)) { data = cgs_read_ind_register(hwmgr->device, CGS_IND_REG__DIDT, ixDIDT_DBR_EDC_CTRL); - data = CGS_REG_SET_FIELD(data, DIDT_DBR_EDC_CTRL, EDC_EN, en); - data = CGS_REG_SET_FIELD(data, DIDT_DBR_EDC_CTRL, EDC_SW_RST, ~en); + data = REG_SET_FIELD(data, DIDT_DBR_EDC_CTRL, EDC_EN, en); + data = REG_SET_FIELD(data, DIDT_DBR_EDC_CTRL, EDC_SW_RST, ~en); cgs_write_ind_register(hwmgr->device, CGS_IND_REG__DIDT, ixDIDT_DBR_EDC_CTRL, data); } } @@ -930,20 +930,18 @@ static void vega10_didt_set_mask(struct pp_hwmgr *hwmgr, const bool enable) static int vega10_enable_cac_driving_se_didt_config(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; int result; uint32_t num_se = 0, count, data; - struct amdgpu_device *adev = hwmgr->adev; - uint32_t reg; num_se = adev->gfx.config.max_shader_engines; - cgs_enter_safe_mode(hwmgr->device, true); + adev->gfx.rlc.funcs->enter_safe_mode(adev); - cgs_lock_grbm_idx(hwmgr->device, true); - reg = soc15_get_register_offset(GC_HWID, 0, mmGRBM_GFX_INDEX_BASE_IDX, mmGRBM_GFX_INDEX); + mutex_lock(&adev->grbm_idx_mutex); for (count = 0; count < num_se; count++) { data = GRBM_GFX_INDEX__INSTANCE_BROADCAST_WRITES_MASK | GRBM_GFX_INDEX__SH_BROADCAST_WRITES_MASK | ( count << GRBM_GFX_INDEX__SE_INDEX__SHIFT); - cgs_write_register(hwmgr->device, reg, data); + WREG32_SOC15(GC, 0, mmGRBM_GFX_INDEX, data); result = vega10_program_didt_config_registers(hwmgr, SEDiDtStallCtrlConfig_vega10, VEGA10_CONFIGREG_DIDT); result |= vega10_program_didt_config_registers(hwmgr, SEDiDtStallPatternConfig_vega10, VEGA10_CONFIGREG_DIDT); @@ -958,43 +956,43 @@ static int vega10_enable_cac_driving_se_didt_config(struct pp_hwmgr *hwmgr) if (0 != result) break; } - cgs_write_register(hwmgr->device, reg, 0xE0000000); - cgs_lock_grbm_idx(hwmgr->device, false); + WREG32_SOC15(GC, 0, mmGRBM_GFX_INDEX, 0xE0000000); + mutex_unlock(&adev->grbm_idx_mutex); vega10_didt_set_mask(hwmgr, true); - cgs_enter_safe_mode(hwmgr->device, false); + adev->gfx.rlc.funcs->exit_safe_mode(adev); return 0; } static int vega10_disable_cac_driving_se_didt_config(struct pp_hwmgr *hwmgr) { - cgs_enter_safe_mode(hwmgr->device, true); + struct amdgpu_device *adev = hwmgr->adev; + + adev->gfx.rlc.funcs->enter_safe_mode(adev); vega10_didt_set_mask(hwmgr, false); - cgs_enter_safe_mode(hwmgr->device, false); + adev->gfx.rlc.funcs->exit_safe_mode(adev); return 0; } static int vega10_enable_psm_gc_didt_config(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; int result; uint32_t num_se = 0, count, data; - struct amdgpu_device *adev = hwmgr->adev; - uint32_t reg; num_se = adev->gfx.config.max_shader_engines; - cgs_enter_safe_mode(hwmgr->device, true); + adev->gfx.rlc.funcs->enter_safe_mode(adev); - cgs_lock_grbm_idx(hwmgr->device, true); - reg = soc15_get_register_offset(GC_HWID, 0, mmGRBM_GFX_INDEX_BASE_IDX, mmGRBM_GFX_INDEX); + mutex_lock(&adev->grbm_idx_mutex); for (count = 0; count < num_se; count++) { data = GRBM_GFX_INDEX__INSTANCE_BROADCAST_WRITES_MASK | GRBM_GFX_INDEX__SH_BROADCAST_WRITES_MASK | ( count << GRBM_GFX_INDEX__SE_INDEX__SHIFT); - cgs_write_register(hwmgr->device, reg, data); + WREG32_SOC15(GC, 0, mmGRBM_GFX_INDEX, data); result = vega10_program_didt_config_registers(hwmgr, SEDiDtStallCtrlConfig_vega10, VEGA10_CONFIGREG_DIDT); result |= vega10_program_didt_config_registers(hwmgr, SEDiDtStallPatternConfig_vega10, VEGA10_CONFIGREG_DIDT); @@ -1003,12 +1001,12 @@ static int vega10_enable_psm_gc_didt_config(struct pp_hwmgr *hwmgr) if (0 != result) break; } - cgs_write_register(hwmgr->device, reg, 0xE0000000); - cgs_lock_grbm_idx(hwmgr->device, false); + WREG32_SOC15(GC, 0, mmGRBM_GFX_INDEX, 0xE0000000); + mutex_unlock(&adev->grbm_idx_mutex); vega10_didt_set_mask(hwmgr, true); - cgs_enter_safe_mode(hwmgr->device, false); + adev->gfx.rlc.funcs->exit_safe_mode(adev); vega10_program_gc_didt_config_registers(hwmgr, GCDiDtDroopCtrlConfig_vega10); if (PP_CAP(PHM_PlatformCaps_GCEDC)) @@ -1022,13 +1020,14 @@ static int vega10_enable_psm_gc_didt_config(struct pp_hwmgr *hwmgr) static int vega10_disable_psm_gc_didt_config(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; uint32_t data; - cgs_enter_safe_mode(hwmgr->device, true); + adev->gfx.rlc.funcs->enter_safe_mode(adev); vega10_didt_set_mask(hwmgr, false); - cgs_enter_safe_mode(hwmgr->device, false); + adev->gfx.rlc.funcs->exit_safe_mode(adev); if (PP_CAP(PHM_PlatformCaps_GCEDC)) { data = 0x00000000; @@ -1043,20 +1042,18 @@ static int vega10_disable_psm_gc_didt_config(struct pp_hwmgr *hwmgr) static int vega10_enable_se_edc_config(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; int result; uint32_t num_se = 0, count, data; - struct amdgpu_device *adev = hwmgr->adev; - uint32_t reg; num_se = adev->gfx.config.max_shader_engines; - cgs_enter_safe_mode(hwmgr->device, true); + adev->gfx.rlc.funcs->enter_safe_mode(adev); - cgs_lock_grbm_idx(hwmgr->device, true); - reg = soc15_get_register_offset(GC_HWID, 0, mmGRBM_GFX_INDEX_BASE_IDX, mmGRBM_GFX_INDEX); + mutex_lock(&adev->grbm_idx_mutex); for (count = 0; count < num_se; count++) { data = GRBM_GFX_INDEX__INSTANCE_BROADCAST_WRITES_MASK | GRBM_GFX_INDEX__SH_BROADCAST_WRITES_MASK | ( count << GRBM_GFX_INDEX__SE_INDEX__SHIFT); - cgs_write_register(hwmgr->device, reg, data); + WREG32_SOC15(GC, 0, mmGRBM_GFX_INDEX, data); result = vega10_program_didt_config_registers(hwmgr, SEDiDtWeightConfig_Vega10, VEGA10_CONFIGREG_DIDT); result |= vega10_program_didt_config_registers(hwmgr, SEEDCStallPatternConfig_Vega10, VEGA10_CONFIGREG_DIDT); result |= vega10_program_didt_config_registers(hwmgr, SEEDCStallDelayConfig_Vega10, VEGA10_CONFIGREG_DIDT); @@ -1067,47 +1064,47 @@ static int vega10_enable_se_edc_config(struct pp_hwmgr *hwmgr) if (0 != result) break; } - cgs_write_register(hwmgr->device, reg, 0xE0000000); - cgs_lock_grbm_idx(hwmgr->device, false); + WREG32_SOC15(GC, 0, mmGRBM_GFX_INDEX, 0xE0000000); + mutex_unlock(&adev->grbm_idx_mutex); vega10_didt_set_mask(hwmgr, true); - cgs_enter_safe_mode(hwmgr->device, false); + adev->gfx.rlc.funcs->exit_safe_mode(adev); return 0; } static int vega10_disable_se_edc_config(struct pp_hwmgr *hwmgr) { - cgs_enter_safe_mode(hwmgr->device, true); + struct amdgpu_device *adev = hwmgr->adev; + + adev->gfx.rlc.funcs->enter_safe_mode(adev); vega10_didt_set_mask(hwmgr, false); - cgs_enter_safe_mode(hwmgr->device, false); + adev->gfx.rlc.funcs->exit_safe_mode(adev); return 0; } static int vega10_enable_psm_gc_edc_config(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; int result; uint32_t num_se = 0; uint32_t count, data; - struct amdgpu_device *adev = hwmgr->adev; - uint32_t reg; num_se = adev->gfx.config.max_shader_engines; - cgs_enter_safe_mode(hwmgr->device, true); + adev->gfx.rlc.funcs->enter_safe_mode(adev); vega10_program_gc_didt_config_registers(hwmgr, AvfsPSMResetConfig_vega10); - cgs_lock_grbm_idx(hwmgr->device, true); - reg = soc15_get_register_offset(GC_HWID, 0, mmGRBM_GFX_INDEX_BASE_IDX, mmGRBM_GFX_INDEX); + mutex_lock(&adev->grbm_idx_mutex); for (count = 0; count < num_se; count++) { data = GRBM_GFX_INDEX__INSTANCE_BROADCAST_WRITES_MASK | GRBM_GFX_INDEX__SH_BROADCAST_WRITES_MASK | ( count << GRBM_GFX_INDEX__SE_INDEX__SHIFT); - cgs_write_register(hwmgr->device, reg, data); - result |= vega10_program_didt_config_registers(hwmgr, PSMSEEDCStallPatternConfig_Vega10, VEGA10_CONFIGREG_DIDT); + WREG32_SOC15(GC, 0, mmGRBM_GFX_INDEX, data); + result = vega10_program_didt_config_registers(hwmgr, PSMSEEDCStallPatternConfig_Vega10, VEGA10_CONFIGREG_DIDT); result |= vega10_program_didt_config_registers(hwmgr, PSMSEEDCStallDelayConfig_Vega10, VEGA10_CONFIGREG_DIDT); result |= vega10_program_didt_config_registers(hwmgr, PSMSEEDCCtrlResetConfig_Vega10, VEGA10_CONFIGREG_DIDT); result |= vega10_program_didt_config_registers(hwmgr, PSMSEEDCCtrlConfig_Vega10, VEGA10_CONFIGREG_DIDT); @@ -1115,12 +1112,12 @@ static int vega10_enable_psm_gc_edc_config(struct pp_hwmgr *hwmgr) if (0 != result) break; } - cgs_write_register(hwmgr->device, reg, 0xE0000000); - cgs_lock_grbm_idx(hwmgr->device, false); + WREG32_SOC15(GC, 0, mmGRBM_GFX_INDEX, 0xE0000000); + mutex_unlock(&adev->grbm_idx_mutex); vega10_didt_set_mask(hwmgr, true); - cgs_enter_safe_mode(hwmgr->device, false); + adev->gfx.rlc.funcs->exit_safe_mode(adev); vega10_program_gc_didt_config_registers(hwmgr, PSMGCEDCDroopCtrlConfig_vega10); @@ -1137,13 +1134,14 @@ static int vega10_enable_psm_gc_edc_config(struct pp_hwmgr *hwmgr) static int vega10_disable_psm_gc_edc_config(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; uint32_t data; - cgs_enter_safe_mode(hwmgr->device, true); + adev->gfx.rlc.funcs->enter_safe_mode(adev); vega10_didt_set_mask(hwmgr, false); - cgs_enter_safe_mode(hwmgr->device, false); + adev->gfx.rlc.funcs->exit_safe_mode(adev); if (PP_CAP(PHM_PlatformCaps_GCEDC)) { data = 0x00000000; @@ -1158,15 +1156,14 @@ static int vega10_disable_psm_gc_edc_config(struct pp_hwmgr *hwmgr) static int vega10_enable_se_edc_force_stall_config(struct pp_hwmgr *hwmgr) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; int result; - cgs_enter_safe_mode(hwmgr->device, true); + adev->gfx.rlc.funcs->enter_safe_mode(adev); - cgs_lock_grbm_idx(hwmgr->device, true); - reg = soc15_get_register_offset(GC_HWID, 0, mmGRBM_GFX_INDEX_BASE_IDX, mmGRBM_GFX_INDEX); - cgs_write_register(hwmgr->device, reg, 0xE0000000); - cgs_lock_grbm_idx(hwmgr->device, false); + mutex_lock(&adev->grbm_idx_mutex); + WREG32_SOC15(GC, 0, mmGRBM_GFX_INDEX, 0xE0000000); + mutex_unlock(&adev->grbm_idx_mutex); result = vega10_program_didt_config_registers(hwmgr, SEEDCForceStallPatternConfig_Vega10, VEGA10_CONFIGREG_DIDT); result |= vega10_program_didt_config_registers(hwmgr, SEEDCCtrlForceStallConfig_Vega10, VEGA10_CONFIGREG_DIDT); @@ -1175,7 +1172,7 @@ static int vega10_enable_se_edc_force_stall_config(struct pp_hwmgr *hwmgr) vega10_didt_set_mask(hwmgr, false); - cgs_enter_safe_mode(hwmgr->device, false); + adev->gfx.rlc.funcs->exit_safe_mode(adev); return 0; } diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_processpptables.c b/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_processpptables.c index c61d0744860d..16b1a9cf6cf0 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_processpptables.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_processpptables.c @@ -52,7 +52,7 @@ static const void *get_powerplay_table(struct pp_hwmgr *hwmgr) if (!table_address) { table_address = (ATOM_Vega10_POWERPLAYTABLE *) - cgs_atom_get_data_table(hwmgr->device, index, + smu_atom_get_data_table(hwmgr->adev, index, &size, &frev, &crev); hwmgr->soft_pp_table = table_address; /*Cache the result in RAM.*/ @@ -267,12 +267,6 @@ static int init_over_drive_limits( hwmgr->platform_descriptor.maxOverdriveVDDC = 0; hwmgr->platform_descriptor.overdriveVDDCStep = 0; - if (hwmgr->platform_descriptor.overdriveLimit.engineClock == 0 || - hwmgr->platform_descriptor.overdriveLimit.memoryClock == 0) { - hwmgr->od_enabled = false; - pr_debug("OverDrive feature not support by VBIOS\n"); - } - return 0; } diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_thermal.c b/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_thermal.c index 9f18226a56ea..aa044c1955fe 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_thermal.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega10_thermal.c @@ -25,7 +25,7 @@ #include "vega10_hwmgr.h" #include "vega10_ppsmc.h" #include "vega10_inc.h" -#include "pp_soc15.h" +#include "soc15_common.h" #include "pp_debug.h" static int vega10_get_current_rpm(struct pp_hwmgr *hwmgr, uint32_t *current_rpm) @@ -89,6 +89,7 @@ int vega10_fan_ctrl_get_fan_speed_percent(struct pp_hwmgr *hwmgr, int vega10_fan_ctrl_get_fan_speed_rpm(struct pp_hwmgr *hwmgr, uint32_t *speed) { + struct amdgpu_device *adev = hwmgr->adev; struct vega10_hwmgr *data = hwmgr->backend; uint32_t tach_period; uint32_t crystal_clock_freq; @@ -100,10 +101,8 @@ int vega10_fan_ctrl_get_fan_speed_rpm(struct pp_hwmgr *hwmgr, uint32_t *speed) if (data->smu_features[GNLD_FAN_CONTROL].supported) { result = vega10_get_current_rpm(hwmgr, speed); } else { - uint32_t reg = soc15_get_register_offset(THM_HWID, 0, - mmCG_TACH_STATUS_BASE_IDX, mmCG_TACH_STATUS); tach_period = - CGS_REG_GET_FIELD(cgs_read_register(hwmgr->device, reg), + REG_GET_FIELD(RREG32_SOC15(THM, 0, mmCG_TACH_STATUS), CG_TACH_STATUS, TACH_PERIOD); @@ -127,26 +126,23 @@ int vega10_fan_ctrl_get_fan_speed_rpm(struct pp_hwmgr *hwmgr, uint32_t *speed) */ int vega10_fan_ctrl_set_static_mode(struct pp_hwmgr *hwmgr, uint32_t mode) { - uint32_t reg; - - reg = soc15_get_register_offset(THM_HWID, 0, - mmCG_FDO_CTRL2_BASE_IDX, mmCG_FDO_CTRL2); + struct amdgpu_device *adev = hwmgr->adev; if (hwmgr->fan_ctrl_is_in_default_mode) { hwmgr->fan_ctrl_default_mode = - CGS_REG_GET_FIELD(cgs_read_register(hwmgr->device, reg), + REG_GET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL2), CG_FDO_CTRL2, FDO_PWM_MODE); hwmgr->tmin = - CGS_REG_GET_FIELD(cgs_read_register(hwmgr->device, reg), + REG_GET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL2), CG_FDO_CTRL2, TMIN); hwmgr->fan_ctrl_is_in_default_mode = false; } - cgs_write_register(hwmgr->device, reg, - CGS_REG_SET_FIELD(cgs_read_register(hwmgr->device, reg), + WREG32_SOC15(THM, 0, mmCG_FDO_CTRL2, + REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL2), CG_FDO_CTRL2, TMIN, 0)); - cgs_write_register(hwmgr->device, reg, - CGS_REG_SET_FIELD(cgs_read_register(hwmgr->device, reg), + WREG32_SOC15(THM, 0, mmCG_FDO_CTRL2, + REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL2), CG_FDO_CTRL2, FDO_PWM_MODE, mode)); return 0; @@ -159,18 +155,15 @@ int vega10_fan_ctrl_set_static_mode(struct pp_hwmgr *hwmgr, uint32_t mode) */ int vega10_fan_ctrl_set_default_mode(struct pp_hwmgr *hwmgr) { - uint32_t reg; - - reg = soc15_get_register_offset(THM_HWID, 0, - mmCG_FDO_CTRL2_BASE_IDX, mmCG_FDO_CTRL2); + struct amdgpu_device *adev = hwmgr->adev; if (!hwmgr->fan_ctrl_is_in_default_mode) { - cgs_write_register(hwmgr->device, reg, - CGS_REG_SET_FIELD(cgs_read_register(hwmgr->device, reg), + WREG32_SOC15(THM, 0, mmCG_FDO_CTRL2, + REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL2), CG_FDO_CTRL2, FDO_PWM_MODE, hwmgr->fan_ctrl_default_mode)); - cgs_write_register(hwmgr->device, reg, - CGS_REG_SET_FIELD(cgs_read_register(hwmgr->device, reg), + WREG32_SOC15(THM, 0, mmCG_FDO_CTRL2, + REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL2), CG_FDO_CTRL2, TMIN, hwmgr->tmin << CG_FDO_CTRL2__TMIN__SHIFT)); hwmgr->fan_ctrl_is_in_default_mode = true; @@ -257,10 +250,10 @@ int vega10_fan_ctrl_stop_smc_fan_control(struct pp_hwmgr *hwmgr) int vega10_fan_ctrl_set_fan_speed_percent(struct pp_hwmgr *hwmgr, uint32_t speed) { + struct amdgpu_device *adev = hwmgr->adev; uint32_t duty100; uint32_t duty; uint64_t tmp64; - uint32_t reg; if (hwmgr->thermal_controller.fanInfo.bNoFan) return 0; @@ -271,10 +264,7 @@ int vega10_fan_ctrl_set_fan_speed_percent(struct pp_hwmgr *hwmgr, if (PP_CAP(PHM_PlatformCaps_MicrocodeFanControl)) vega10_fan_ctrl_stop_smc_fan_control(hwmgr); - reg = soc15_get_register_offset(THM_HWID, 0, - mmCG_FDO_CTRL1_BASE_IDX, mmCG_FDO_CTRL1); - - duty100 = CGS_REG_GET_FIELD(cgs_read_register(hwmgr->device, reg), + duty100 = REG_GET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL1), CG_FDO_CTRL1, FMAX_DUTY100); if (duty100 == 0) @@ -284,10 +274,8 @@ int vega10_fan_ctrl_set_fan_speed_percent(struct pp_hwmgr *hwmgr, do_div(tmp64, 100); duty = (uint32_t)tmp64; - reg = soc15_get_register_offset(THM_HWID, 0, - mmCG_FDO_CTRL0_BASE_IDX, mmCG_FDO_CTRL0); - cgs_write_register(hwmgr->device, reg, - CGS_REG_SET_FIELD(cgs_read_register(hwmgr->device, reg), + WREG32_SOC15(THM, 0, mmCG_FDO_CTRL0, + REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL0), CG_FDO_CTRL0, FDO_STATIC_DUTY, duty)); return vega10_fan_ctrl_set_static_mode(hwmgr, FDO_PWM_MODE_STATIC); @@ -317,10 +305,10 @@ int vega10_fan_ctrl_reset_fan_speed_to_default(struct pp_hwmgr *hwmgr) */ int vega10_fan_ctrl_set_fan_speed_rpm(struct pp_hwmgr *hwmgr, uint32_t speed) { + struct amdgpu_device *adev = hwmgr->adev; uint32_t tach_period; uint32_t crystal_clock_freq; int result = 0; - uint32_t reg; if (hwmgr->thermal_controller.fanInfo.bNoFan || (speed < hwmgr->thermal_controller.fanInfo.ulMinRPM) || @@ -333,10 +321,8 @@ int vega10_fan_ctrl_set_fan_speed_rpm(struct pp_hwmgr *hwmgr, uint32_t speed) if (!result) { crystal_clock_freq = amdgpu_asic_get_xclk((struct amdgpu_device *)hwmgr->adev); tach_period = 60 * crystal_clock_freq * 10000 / (8 * speed); - reg = soc15_get_register_offset(THM_HWID, 0, - mmCG_TACH_STATUS_BASE_IDX, mmCG_TACH_STATUS); - cgs_write_register(hwmgr->device, reg, - CGS_REG_SET_FIELD(cgs_read_register(hwmgr->device, reg), + WREG32_SOC15(THM, 0, mmCG_TACH_STATUS, + REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_TACH_STATUS), CG_TACH_STATUS, TACH_PERIOD, tach_period)); } @@ -350,13 +336,10 @@ int vega10_fan_ctrl_set_fan_speed_rpm(struct pp_hwmgr *hwmgr, uint32_t speed) */ int vega10_thermal_get_temperature(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; int temp; - uint32_t reg; - reg = soc15_get_register_offset(THM_HWID, 0, - mmCG_MULT_THERMAL_STATUS_BASE_IDX, mmCG_MULT_THERMAL_STATUS); - - temp = cgs_read_register(hwmgr->device, reg); + temp = RREG32_SOC15(THM, 0, mmCG_MULT_THERMAL_STATUS); temp = (temp & CG_MULT_THERMAL_STATUS__CTF_TEMP_MASK) >> CG_MULT_THERMAL_STATUS__CTF_TEMP__SHIFT; @@ -379,11 +362,12 @@ int vega10_thermal_get_temperature(struct pp_hwmgr *hwmgr) static int vega10_thermal_set_temperature_range(struct pp_hwmgr *hwmgr, struct PP_TemperatureRange *range) { + struct amdgpu_device *adev = hwmgr->adev; int low = VEGA10_THERMAL_MINIMUM_ALERT_TEMP * PP_TEMPERATURE_UNITS_PER_CENTIGRADES; int high = VEGA10_THERMAL_MAXIMUM_ALERT_TEMP * PP_TEMPERATURE_UNITS_PER_CENTIGRADES; - uint32_t val, reg; + uint32_t val; if (low < range->min) low = range->min; @@ -393,20 +377,17 @@ static int vega10_thermal_set_temperature_range(struct pp_hwmgr *hwmgr, if (low > high) return -EINVAL; - reg = soc15_get_register_offset(THM_HWID, 0, - mmTHM_THERMAL_INT_CTRL_BASE_IDX, mmTHM_THERMAL_INT_CTRL); - - val = cgs_read_register(hwmgr->device, reg); + val = RREG32_SOC15(THM, 0, mmTHM_THERMAL_INT_CTRL); - val = CGS_REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, MAX_IH_CREDIT, 5); - val = CGS_REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, THERM_IH_HW_ENA, 1); - val = CGS_REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, DIG_THERM_INTH, (high / PP_TEMPERATURE_UNITS_PER_CENTIGRADES)); - val = CGS_REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, DIG_THERM_INTL, (low / PP_TEMPERATURE_UNITS_PER_CENTIGRADES)); + val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, MAX_IH_CREDIT, 5); + val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, THERM_IH_HW_ENA, 1); + val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, DIG_THERM_INTH, (high / PP_TEMPERATURE_UNITS_PER_CENTIGRADES)); + val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, DIG_THERM_INTL, (low / PP_TEMPERATURE_UNITS_PER_CENTIGRADES)); val &= (~THM_THERMAL_INT_CTRL__THERM_TRIGGER_MASK_MASK) & (~THM_THERMAL_INT_CTRL__THERM_INTH_MASK_MASK) & (~THM_THERMAL_INT_CTRL__THERM_INTL_MASK_MASK); - cgs_write_register(hwmgr->device, reg, val); + WREG32_SOC15(THM, 0, mmTHM_THERMAL_INT_CTRL, val); return 0; } @@ -418,21 +399,17 @@ static int vega10_thermal_set_temperature_range(struct pp_hwmgr *hwmgr, */ static int vega10_thermal_initialize(struct pp_hwmgr *hwmgr) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; if (hwmgr->thermal_controller.fanInfo.ucTachometerPulsesPerRevolution) { - reg = soc15_get_register_offset(THM_HWID, 0, - mmCG_TACH_CTRL_BASE_IDX, mmCG_TACH_CTRL); - cgs_write_register(hwmgr->device, reg, - CGS_REG_SET_FIELD(cgs_read_register(hwmgr->device, reg), + WREG32_SOC15(THM, 0, mmCG_TACH_CTRL, + REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_TACH_CTRL), CG_TACH_CTRL, EDGE_PER_REV, hwmgr->thermal_controller.fanInfo.ucTachometerPulsesPerRevolution - 1)); } - reg = soc15_get_register_offset(THM_HWID, 0, - mmCG_FDO_CTRL2_BASE_IDX, mmCG_FDO_CTRL2); - cgs_write_register(hwmgr->device, reg, - CGS_REG_SET_FIELD(cgs_read_register(hwmgr->device, reg), + WREG32_SOC15(THM, 0, mmCG_FDO_CTRL2, + REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL2), CG_FDO_CTRL2, TACH_PWM_RESP_RATE, 0x28)); return 0; @@ -445,9 +422,9 @@ static int vega10_thermal_initialize(struct pp_hwmgr *hwmgr) */ static int vega10_thermal_enable_alert(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; struct vega10_hwmgr *data = hwmgr->backend; uint32_t val = 0; - uint32_t reg; if (data->smu_features[GNLD_FW_CTF].supported) { if (data->smu_features[GNLD_FW_CTF].enabled) @@ -465,8 +442,7 @@ static int vega10_thermal_enable_alert(struct pp_hwmgr *hwmgr) val |= (1 << THM_THERMAL_INT_ENA__THERM_INTL_CLR__SHIFT); val |= (1 << THM_THERMAL_INT_ENA__THERM_TRIGGER_CLR__SHIFT); - reg = soc15_get_register_offset(THM_HWID, 0, mmTHM_THERMAL_INT_ENA_BASE_IDX, mmTHM_THERMAL_INT_ENA); - cgs_write_register(hwmgr->device, reg, val); + WREG32_SOC15(THM, 0, mmTHM_THERMAL_INT_ENA, val); return 0; } @@ -477,8 +453,8 @@ static int vega10_thermal_enable_alert(struct pp_hwmgr *hwmgr) */ int vega10_thermal_disable_alert(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; struct vega10_hwmgr *data = hwmgr->backend; - uint32_t reg; if (data->smu_features[GNLD_FW_CTF].supported) { if (!data->smu_features[GNLD_FW_CTF].enabled) @@ -493,8 +469,7 @@ int vega10_thermal_disable_alert(struct pp_hwmgr *hwmgr) data->smu_features[GNLD_FW_CTF].enabled = false; } - reg = soc15_get_register_offset(THM_HWID, 0, mmTHM_THERMAL_INT_ENA_BASE_IDX, mmTHM_THERMAL_INT_ENA); - cgs_write_register(hwmgr->device, reg, 0); + WREG32_SOC15(THM, 0, mmTHM_THERMAL_INT_ENA, 0); return 0; } diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_hwmgr.c b/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_hwmgr.c index 200de46bd06b..782e2098824d 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_hwmgr.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_hwmgr.c @@ -34,7 +34,6 @@ #include "atomfirmware.h" #include "cgs_common.h" #include "vega12_inc.h" -#include "pp_soc15.h" #include "pppcielanes.h" #include "vega12_hwmgr.h" #include "vega12_processpptables.h" @@ -546,6 +545,7 @@ static int vega12_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) return -EINVAL); dpm_table->dpm_levels[i].value = clock; + dpm_table->dpm_levels[i].enabled = true; } vega12_init_dpm_state(&(dpm_table->dpm_state)); @@ -565,6 +565,7 @@ static int vega12_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) return -EINVAL); dpm_table->dpm_levels[i].value = clock; + dpm_table->dpm_levels[i].enabled = true; } vega12_init_dpm_state(&(dpm_table->dpm_state)); @@ -585,6 +586,7 @@ static int vega12_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) return -EINVAL); dpm_table->dpm_levels[i].value = clock; + dpm_table->dpm_levels[i].enabled = true; } vega12_init_dpm_state(&(dpm_table->dpm_state)); @@ -605,6 +607,7 @@ static int vega12_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) return -EINVAL); dpm_table->dpm_levels[i].value = clock; + dpm_table->dpm_levels[i].enabled = true; } vega12_init_dpm_state(&(dpm_table->dpm_state)); @@ -625,6 +628,7 @@ static int vega12_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) return -EINVAL); dpm_table->dpm_levels[i].value = clock; + dpm_table->dpm_levels[i].enabled = true; } vega12_init_dpm_state(&(dpm_table->dpm_state)); @@ -645,6 +649,7 @@ static int vega12_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) return -EINVAL); dpm_table->dpm_levels[i].value = clock; + dpm_table->dpm_levels[i].enabled = true; } vega12_init_dpm_state(&(dpm_table->dpm_state)); @@ -666,6 +671,7 @@ static int vega12_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) return -EINVAL); dpm_table->dpm_levels[i].value = clock; + dpm_table->dpm_levels[i].enabled = true; } vega12_init_dpm_state(&(dpm_table->dpm_state)); @@ -686,6 +692,7 @@ static int vega12_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) return -EINVAL); dpm_table->dpm_levels[i].value = clock; + dpm_table->dpm_levels[i].enabled = true; } vega12_init_dpm_state(&(dpm_table->dpm_state)); @@ -706,6 +713,7 @@ static int vega12_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) return -EINVAL); dpm_table->dpm_levels[i].value = clock; + dpm_table->dpm_levels[i].enabled = true; } vega12_init_dpm_state(&(dpm_table->dpm_state)); @@ -726,6 +734,7 @@ static int vega12_setup_default_dpm_tables(struct pp_hwmgr *hwmgr) return -EINVAL); dpm_table->dpm_levels[i].value = clock; + dpm_table->dpm_levels[i].enabled = true; } vega12_init_dpm_state(&(dpm_table->dpm_state)); @@ -992,15 +1001,55 @@ static uint32_t vega12_find_highest_dpm_level( static int vega12_upload_dpm_min_level(struct pp_hwmgr *hwmgr) { + struct vega12_hwmgr *data = hwmgr->backend; + if (data->smc_state_table.gfx_boot_level != + data->dpm_table.gfx_table.dpm_state.soft_min_level) { + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_SetSoftMinByFreq, + PPCLK_GFXCLK<<16 | data->dpm_table.gfx_table.dpm_levels[data->smc_state_table.gfx_boot_level].value); + data->dpm_table.gfx_table.dpm_state.soft_min_level = + data->smc_state_table.gfx_boot_level; + } + + if (data->smc_state_table.mem_boot_level != + data->dpm_table.mem_table.dpm_state.soft_min_level) { + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_SetSoftMinByFreq, + PPCLK_UCLK<<16 | data->dpm_table.mem_table.dpm_levels[data->smc_state_table.mem_boot_level].value); + data->dpm_table.mem_table.dpm_state.soft_min_level = + data->smc_state_table.mem_boot_level; + } + return 0; + } static int vega12_upload_dpm_max_level(struct pp_hwmgr *hwmgr) { + struct vega12_hwmgr *data = hwmgr->backend; + if (data->smc_state_table.gfx_max_level != + data->dpm_table.gfx_table.dpm_state.soft_max_level) { + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_SetSoftMaxByFreq, + /* plus the vale by 1 to align the resolution */ + PPCLK_GFXCLK<<16 | (data->dpm_table.gfx_table.dpm_levels[data->smc_state_table.gfx_max_level].value + 1)); + data->dpm_table.gfx_table.dpm_state.soft_max_level = + data->smc_state_table.gfx_max_level; + } + + if (data->smc_state_table.mem_max_level != + data->dpm_table.mem_table.dpm_state.soft_max_level) { + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_SetSoftMaxByFreq, + /* plus the vale by 1 to align the resolution */ + PPCLK_UCLK<<16 | (data->dpm_table.mem_table.dpm_levels[data->smc_state_table.mem_max_level].value + 1)); + data->dpm_table.mem_table.dpm_state.soft_max_level = + data->smc_state_table.mem_max_level; + } + return 0; } - int vega12_enable_disable_vce_dpm(struct pp_hwmgr *hwmgr, bool enable) { struct vega12_hwmgr *data = @@ -1064,8 +1113,7 @@ static uint32_t vega12_dpm_get_mclk(struct pp_hwmgr *hwmgr, bool low) return (mem_clk * 100); } -static int vega12_get_gpu_power(struct pp_hwmgr *hwmgr, - struct pp_gpu_power *query) +static int vega12_get_gpu_power(struct pp_hwmgr *hwmgr, uint32_t *query) { #if 0 uint32_t value; @@ -1077,7 +1125,7 @@ static int vega12_get_gpu_power(struct pp_hwmgr *hwmgr, vega12_read_arg_from_smc(hwmgr, &value); /* power value is an integer */ - query->average_gpu_power = value << 8; + *query = value << 8; #endif return 0; } @@ -1186,12 +1234,8 @@ static int vega12_read_sensor(struct pp_hwmgr *hwmgr, int idx, *size = 4; break; case AMDGPU_PP_SENSOR_GPU_POWER: - if (*size < sizeof(struct pp_gpu_power)) - ret = -EINVAL; - else { - *size = sizeof(struct pp_gpu_power); - ret = vega12_get_gpu_power(hwmgr, (struct pp_gpu_power *)value); - } + ret = vega12_get_gpu_power(hwmgr, (uint32_t *)value); + break; default: ret = -EINVAL; @@ -1260,23 +1304,18 @@ static int vega12_notify_smc_display_config_after_ps_adjustment( { struct vega12_hwmgr *data = (struct vega12_hwmgr *)(hwmgr->backend); - uint32_t num_active_disps = 0; - struct cgs_display_info info = {0}; struct PP_Clocks min_clocks = {0}; struct pp_display_clock_request clock_req; uint32_t clk_request; - info.mode_info = NULL; - cgs_get_active_displays_info(hwmgr->device, &info); - num_active_disps = info.display_count; - if (num_active_disps > 1) + if (hwmgr->display_config->num_display > 1) vega12_notify_smc_display_change(hwmgr, false); else vega12_notify_smc_display_change(hwmgr, true); - min_clocks.dcefClock = hwmgr->display_config.min_dcef_set_clk; - min_clocks.dcefClockInSR = hwmgr->display_config.min_dcef_deep_sleep_set_clk; - min_clocks.memoryClock = hwmgr->display_config.min_mem_set_clock; + min_clocks.dcefClock = hwmgr->display_config->min_dcef_set_clk; + min_clocks.dcefClockInSR = hwmgr->display_config->min_dcef_deep_sleep_set_clk; + min_clocks.memoryClock = hwmgr->display_config->min_mem_set_clock; if (data->smu_features[GNLD_DPM_DCEFCLK].supported) { clock_req.clock_type = amd_pp_dcef_clock; @@ -1832,9 +1871,7 @@ static int vega12_display_configuration_changed_task(struct pp_hwmgr *hwmgr) { struct vega12_hwmgr *data = (struct vega12_hwmgr *)(hwmgr->backend); int result = 0; - uint32_t num_turned_on_displays = 1; Watermarks_t *wm_table = &(data->smc_state_table.water_marks_table); - struct cgs_display_info info = {0}; if ((data->water_marks_bitmap & WaterMarksExist) && !(data->water_marks_bitmap & WaterMarksLoaded)) { @@ -1846,12 +1883,9 @@ static int vega12_display_configuration_changed_task(struct pp_hwmgr *hwmgr) if ((data->water_marks_bitmap & WaterMarksExist) && data->smu_features[GNLD_DPM_DCEFCLK].supported && - data->smu_features[GNLD_DPM_SOCCLK].supported) { - cgs_get_active_displays_info(hwmgr->device, &info); - num_turned_on_displays = info.display_count; + data->smu_features[GNLD_DPM_SOCCLK].supported) smum_send_msg_to_smc_with_parameter(hwmgr, - PPSMC_MSG_NumOfDisplays, num_turned_on_displays); - } + PPSMC_MSG_NumOfDisplays, hwmgr->display_config->num_display); return result; } @@ -1894,15 +1928,12 @@ vega12_check_smc_update_required_for_display_configuration(struct pp_hwmgr *hwmg { struct vega12_hwmgr *data = (struct vega12_hwmgr *)(hwmgr->backend); bool is_update_required = false; - struct cgs_display_info info = {0, 0, NULL}; - - cgs_get_active_displays_info(hwmgr->device, &info); - if (data->display_timing.num_existing_displays != info.display_count) + if (data->display_timing.num_existing_displays != hwmgr->display_config->num_display) is_update_required = true; if (data->registry_data.gfx_clk_deep_sleep_support) { - if (data->display_timing.min_clock_in_sr != hwmgr->display_config.min_core_set_clock_in_sr) + if (data->display_timing.min_clock_in_sr != hwmgr->display_config->min_core_set_clock_in_sr) is_update_required = true; } diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_hwmgr.h b/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_hwmgr.h index bc98b1df3b65..e81ded1ec198 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_hwmgr.h +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_hwmgr.h @@ -33,7 +33,7 @@ #define WaterMarksExist 1 #define WaterMarksLoaded 2 -#define VG12_PSUEDO_NUM_GFXCLK_DPM_LEVELS 8 +#define VG12_PSUEDO_NUM_GFXCLK_DPM_LEVELS 16 #define VG12_PSUEDO_NUM_SOCCLK_DPM_LEVELS 8 #define VG12_PSUEDO_NUM_DCEFCLK_DPM_LEVELS 8 #define VG12_PSUEDO_NUM_UCLK_DPM_LEVELS 4 diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_processpptables.c b/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_processpptables.c index b34113f45904..888ddca902d8 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_processpptables.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_processpptables.c @@ -51,7 +51,7 @@ static const void *get_powerplay_table(struct pp_hwmgr *hwmgr) if (!table_address) { table_address = (ATOM_Vega12_POWERPLAYTABLE *) - cgs_atom_get_data_table(hwmgr->device, index, + smu_atom_get_data_table(hwmgr->adev, index, &size, &frev, &crev); hwmgr->soft_pp_table = table_address; /*Cache the result in RAM.*/ @@ -224,6 +224,11 @@ static int append_vbios_pptable(struct pp_hwmgr *hwmgr, PPTable_t *ppsmc_pptable ppsmc_pptable->AcgGfxclkSpreadPercent = smc_dpm_table.acggfxclkspreadpercent; ppsmc_pptable->AcgGfxclkSpreadFreq = smc_dpm_table.acggfxclkspreadfreq; + /* 0xFFFF will disable the ACG feature */ + if (!(hwmgr->feature_mask & PP_ACG_MASK)) { + ppsmc_pptable->AcgThresholdFreqHigh = 0xFFFF; + ppsmc_pptable->AcgThresholdFreqLow = 0xFFFF; + } return 0; } diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_thermal.c b/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_thermal.c index df0fa815cd6e..cfd9e6ccb790 100644 --- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_thermal.c +++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega12_thermal.c @@ -26,7 +26,7 @@ #include "vega12_smumgr.h" #include "vega12_ppsmc.h" #include "vega12_inc.h" -#include "pp_soc15.h" +#include "soc15_common.h" #include "pp_debug.h" static int vega12_get_current_rpm(struct pp_hwmgr *hwmgr, uint32_t *current_rpm) @@ -147,13 +147,10 @@ int vega12_fan_ctrl_reset_fan_speed_to_default(struct pp_hwmgr *hwmgr) */ int vega12_thermal_get_temperature(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; int temp = 0; - uint32_t reg; - reg = soc15_get_register_offset(THM_HWID, 0, - mmCG_MULT_THERMAL_STATUS_BASE_IDX, mmCG_MULT_THERMAL_STATUS); - - temp = cgs_read_register(hwmgr->device, reg); + temp = RREG32_SOC15(THM, 0, mmCG_MULT_THERMAL_STATUS); temp = (temp & CG_MULT_THERMAL_STATUS__CTF_TEMP_MASK) >> CG_MULT_THERMAL_STATUS__CTF_TEMP__SHIFT; @@ -175,11 +172,12 @@ int vega12_thermal_get_temperature(struct pp_hwmgr *hwmgr) static int vega12_thermal_set_temperature_range(struct pp_hwmgr *hwmgr, struct PP_TemperatureRange *range) { + struct amdgpu_device *adev = hwmgr->adev; int low = VEGA12_THERMAL_MINIMUM_ALERT_TEMP * PP_TEMPERATURE_UNITS_PER_CENTIGRADES; int high = VEGA12_THERMAL_MAXIMUM_ALERT_TEMP * PP_TEMPERATURE_UNITS_PER_CENTIGRADES; - uint32_t val, reg; + uint32_t val; if (low < range->min) low = range->min; @@ -189,18 +187,15 @@ static int vega12_thermal_set_temperature_range(struct pp_hwmgr *hwmgr, if (low > high) return -EINVAL; - reg = soc15_get_register_offset(THM_HWID, 0, - mmTHM_THERMAL_INT_CTRL_BASE_IDX, mmTHM_THERMAL_INT_CTRL); - - val = cgs_read_register(hwmgr->device, reg); + val = RREG32_SOC15(THM, 0, mmTHM_THERMAL_INT_CTRL); - val = CGS_REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, MAX_IH_CREDIT, 5); - val = CGS_REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, THERM_IH_HW_ENA, 1); - val = CGS_REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, DIG_THERM_INTH, (high / PP_TEMPERATURE_UNITS_PER_CENTIGRADES)); - val = CGS_REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, DIG_THERM_INTL, (low / PP_TEMPERATURE_UNITS_PER_CENTIGRADES)); + val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, MAX_IH_CREDIT, 5); + val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, THERM_IH_HW_ENA, 1); + val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, DIG_THERM_INTH, (high / PP_TEMPERATURE_UNITS_PER_CENTIGRADES)); + val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, DIG_THERM_INTL, (low / PP_TEMPERATURE_UNITS_PER_CENTIGRADES)); val = val & (~THM_THERMAL_INT_CTRL__THERM_TRIGGER_MASK_MASK); - cgs_write_register(hwmgr->device, reg, val); + WREG32_SOC15(THM, 0, mmTHM_THERMAL_INT_CTRL, val); return 0; } @@ -212,15 +207,14 @@ static int vega12_thermal_set_temperature_range(struct pp_hwmgr *hwmgr, */ static int vega12_thermal_enable_alert(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; uint32_t val = 0; - uint32_t reg; val |= (1 << THM_THERMAL_INT_ENA__THERM_INTH_CLR__SHIFT); val |= (1 << THM_THERMAL_INT_ENA__THERM_INTL_CLR__SHIFT); val |= (1 << THM_THERMAL_INT_ENA__THERM_TRIGGER_CLR__SHIFT); - reg = soc15_get_register_offset(THM_HWID, 0, mmTHM_THERMAL_INT_ENA_BASE_IDX, mmTHM_THERMAL_INT_ENA); - cgs_write_register(hwmgr->device, reg, val); + WREG32_SOC15(THM, 0, mmTHM_THERMAL_INT_ENA, val); return 0; } @@ -231,10 +225,9 @@ static int vega12_thermal_enable_alert(struct pp_hwmgr *hwmgr) */ int vega12_thermal_disable_alert(struct pp_hwmgr *hwmgr) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; - reg = soc15_get_register_offset(THM_HWID, 0, mmTHM_THERMAL_INT_ENA_BASE_IDX, mmTHM_THERMAL_INT_ENA); - cgs_write_register(hwmgr->device, reg, 0); + WREG32_SOC15(THM, 0, mmTHM_THERMAL_INT_ENA, 0); return 0; } diff --git a/drivers/gpu/drm/amd/powerplay/inc/hardwaremanager.h b/drivers/gpu/drm/amd/powerplay/inc/hardwaremanager.h index 8b78bbecd1bc..a202247c9894 100644 --- a/drivers/gpu/drm/amd/powerplay/inc/hardwaremanager.h +++ b/drivers/gpu/drm/amd/powerplay/inc/hardwaremanager.h @@ -377,11 +377,7 @@ struct phm_clocks { #define DPMTABLE_UPDATE_SCLK 0x00000004 #define DPMTABLE_UPDATE_MCLK 0x00000008 #define DPMTABLE_OD_UPDATE_VDDC 0x00000010 - -/* To determine if sclk and mclk are in overdrive state */ -#define SCLK_OVERDRIVE_ENABLED 0x00000001 -#define MCLK_OVERDRIVE_ENABLED 0x00000002 -#define VDDC_OVERDRIVE_ENABLED 0x00000010 +#define DPMTABLE_UPDATE_SOCCLK 0x00000020 struct phm_odn_performance_level { uint32_t clock; @@ -414,7 +410,10 @@ extern int phm_apply_state_adjust_rules(struct pp_hwmgr *hwmgr, struct pp_power_state *adjusted_ps, const struct pp_power_state *current_ps); +extern int phm_apply_clock_adjust_rules(struct pp_hwmgr *hwmgr); + extern int phm_force_dpm_levels(struct pp_hwmgr *hwmgr, enum amd_dpm_forced_level level); +extern int phm_pre_display_configuration_changed(struct pp_hwmgr *hwmgr); extern int phm_display_configuration_changed(struct pp_hwmgr *hwmgr); extern int phm_notify_smc_display_config_after_ps_adjustment(struct pp_hwmgr *hwmgr); extern int phm_register_irq_handlers(struct pp_hwmgr *hwmgr); diff --git a/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h b/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h index 17f811d181c8..b99fb8ac822c 100644 --- a/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h +++ b/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h @@ -38,6 +38,8 @@ struct phm_fan_speed_info; struct pp_atomctrl_voltage_table; #define VOLTAGE_SCALE 4 +#define VOLTAGE_VID_OFFSET_SCALE1 625 +#define VOLTAGE_VID_OFFSET_SCALE2 100 enum DISPLAY_GAP { DISPLAY_GAP_VBLANK_OR_WM = 0, /* Wait for vblank or MCHG watermark. */ @@ -64,24 +66,6 @@ struct vi_dpm_table { #define PCIE_PERF_REQ_GEN2 3 #define PCIE_PERF_REQ_GEN3 4 -enum PP_FEATURE_MASK { - PP_SCLK_DPM_MASK = 0x1, - PP_MCLK_DPM_MASK = 0x2, - PP_PCIE_DPM_MASK = 0x4, - PP_SCLK_DEEP_SLEEP_MASK = 0x8, - PP_POWER_CONTAINMENT_MASK = 0x10, - PP_UVD_HANDSHAKE_MASK = 0x20, - PP_SMC_VOLTAGE_CONTROL_MASK = 0x40, - PP_VBI_TIME_SUPPORT_MASK = 0x80, - PP_ULV_MASK = 0x100, - PP_ENABLE_GFX_CG_THRU_SMU = 0x200, - PP_CLOCK_STRETCH_MASK = 0x400, - PP_OD_FUZZY_FAN_CONTROL_MASK = 0x800, - PP_SOCCLK_DPM_MASK = 0x1000, - PP_DCEFCLK_DPM_MASK = 0x2000, - PP_OVERDRIVE_MASK = 0x4000, -}; - enum PHM_BackEnd_Magic { PHM_Dummy_Magic = 0xAA5555AA, PHM_RV770_Magic = 0xDCBAABCD, @@ -245,6 +229,8 @@ struct pp_hwmgr_func { struct pp_power_state *prequest_ps, const struct pp_power_state *pcurrent_ps); + int (*apply_clocks_adjust_rules)(struct pp_hwmgr *hwmgr); + int (*force_dpm_level)(struct pp_hwmgr *hw_mgr, enum amd_dpm_forced_level level); @@ -268,6 +254,7 @@ struct pp_hwmgr_func { const void *state); int (*enable_clock_power_gating)(struct pp_hwmgr *hwmgr); int (*notify_smc_display_config_after_ps_adjustment)(struct pp_hwmgr *hwmgr); + int (*pre_display_config_changed)(struct pp_hwmgr *hwmgr); int (*display_config_changed)(struct pp_hwmgr *hwmgr); int (*disable_clock_power_gating)(struct pp_hwmgr *hwmgr); int (*update_clock_gatings)(struct pp_hwmgr *hwmgr, @@ -312,6 +299,7 @@ struct pp_hwmgr_func { int (*display_clock_voltage_request)(struct pp_hwmgr *hwmgr, struct pp_display_clock_request *clock); int (*get_max_high_clocks)(struct pp_hwmgr *hwmgr, struct amd_pp_simple_clock_info *clocks); + int (*gfx_off_control)(struct pp_hwmgr *hwmgr, bool enable); int (*power_off_asic)(struct pp_hwmgr *hwmgr); int (*force_clock_level)(struct pp_hwmgr *hwmgr, enum pp_clock_type type, uint32_t mask); int (*print_clock_levels)(struct pp_hwmgr *hwmgr, enum pp_clock_type type, char *buf); @@ -341,6 +329,7 @@ struct pp_hwmgr_func { long *input, uint32_t size); int (*set_power_limit)(struct pp_hwmgr *hwmgr, uint32_t n); int (*set_mmhub_powergating_by_smu)(struct pp_hwmgr *hwmgr); + int (*smus_notify_pwe)(struct pp_hwmgr *hwmgr); }; struct pp_table_func { @@ -718,6 +707,7 @@ struct pp_hwmgr { uint32_t chip_family; uint32_t chip_id; uint32_t smu_version; + bool not_vf; bool pm_en; struct mutex smu_lock; @@ -764,7 +754,7 @@ struct pp_hwmgr { struct pp_power_state *request_ps; struct pp_power_state *boot_ps; struct pp_power_state *uvd_ps; - struct amd_pp_display_configuration display_config; + const struct amd_pp_display_configuration *display_config; uint32_t feature_mask; bool avfs_supported; /* UMD Pstate */ @@ -782,10 +772,13 @@ struct pp_hwmgr { }; int hwmgr_early_init(struct pp_hwmgr *hwmgr); +int hwmgr_sw_init(struct pp_hwmgr *hwmgr); +int hwmgr_sw_fini(struct pp_hwmgr *hwmgr); int hwmgr_hw_init(struct pp_hwmgr *hwmgr); int hwmgr_hw_fini(struct pp_hwmgr *hwmgr); -int hwmgr_hw_suspend(struct pp_hwmgr *hwmgr); -int hwmgr_hw_resume(struct pp_hwmgr *hwmgr); +int hwmgr_suspend(struct pp_hwmgr *hwmgr); +int hwmgr_resume(struct pp_hwmgr *hwmgr); + int hwmgr_handle_task(struct pp_hwmgr *hwmgr, enum amd_pp_task task_id, enum amd_pm_state_type *user_state); diff --git a/drivers/gpu/drm/amd/powerplay/inc/pp_soc15.h b/drivers/gpu/drm/amd/powerplay/inc/pp_soc15.h deleted file mode 100644 index 214f370c5efd..000000000000 --- a/drivers/gpu/drm/amd/powerplay/inc/pp_soc15.h +++ /dev/null @@ -1,52 +0,0 @@ -/* - * Copyright 2016 Advanced Micro Devices, Inc. - * - * Permission is hereby granted, free of charge, to any person obtaining a - * copy of this software and associated documentation files (the "Software"), - * to deal in the Software without restriction, including without limitation - * the rights to use, copy, modify, merge, publish, distribute, sublicense, - * and/or sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following conditions: - * - * The above copyright notice and this permission notice shall be included in - * all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, - * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL - * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR - * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, - * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - * - */ -#ifndef PP_SOC15_H -#define PP_SOC15_H - -#include "soc15_hw_ip.h" -#include "vega10_ip_offset.h" - -inline static uint32_t soc15_get_register_offset( - uint32_t hw_id, - uint32_t inst, - uint32_t segment, - uint32_t offset) -{ - uint32_t reg = 0; - - if (hw_id == THM_HWID) - reg = THM_BASE.instance[inst].segment[segment] + offset; - else if (hw_id == NBIF_HWID) - reg = NBIF_BASE.instance[inst].segment[segment] + offset; - else if (hw_id == MP1_HWID) - reg = MP1_BASE.instance[inst].segment[segment] + offset; - else if (hw_id == DF_HWID) - reg = DF_BASE.instance[inst].segment[segment] + offset; - else if (hw_id == GC_HWID) - reg = GC_BASE.instance[inst].segment[segment] + offset; - else if (hw_id == SMUIO_HWID) - reg = SMUIO_BASE.instance[inst].segment[segment] + offset; - return reg; -} - -#endif diff --git a/drivers/gpu/drm/amd/powerplay/inc/rv_ppsmc.h b/drivers/gpu/drm/amd/powerplay/inc/rv_ppsmc.h index 426bff2aad2b..a2991fa2e6f8 100644 --- a/drivers/gpu/drm/amd/powerplay/inc/rv_ppsmc.h +++ b/drivers/gpu/drm/amd/powerplay/inc/rv_ppsmc.h @@ -75,13 +75,15 @@ #define PPSMC_MSG_GetMinGfxclkFrequency 0x2C #define PPSMC_MSG_GetMaxGfxclkFrequency 0x2D #define PPSMC_MSG_SoftReset 0x2E +#define PPSMC_MSG_SetGfxCGPG 0x2F #define PPSMC_MSG_SetSoftMaxGfxClk 0x30 #define PPSMC_MSG_SetHardMinGfxClk 0x31 #define PPSMC_MSG_SetSoftMaxSocclkByFreq 0x32 #define PPSMC_MSG_SetSoftMaxFclkByFreq 0x33 #define PPSMC_MSG_SetSoftMaxVcn 0x34 #define PPSMC_MSG_PowerGateMmHub 0x35 -#define PPSMC_Message_Count 0x36 +#define PPSMC_MSG_SetRccPfcPmeRestoreRegister 0x36 +#define PPSMC_Message_Count 0x37 typedef uint16_t PPSMC_Result; diff --git a/drivers/gpu/drm/amd/powerplay/inc/smu75.h b/drivers/gpu/drm/amd/powerplay/inc/smu75.h new file mode 100644 index 000000000000..771523001533 --- /dev/null +++ b/drivers/gpu/drm/amd/powerplay/inc/smu75.h @@ -0,0 +1,760 @@ +/* + * Copyright 2017 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ +#ifndef SMU75_H +#define SMU75_H + +#pragma pack(push, 1) + +typedef struct { + uint32_t high; + uint32_t low; +} data_64_t; + +typedef struct { + data_64_t high; + data_64_t low; +} data_128_t; + +#define SMU__DGPU_ONLY + +#define SMU__NUM_SCLK_DPM_STATE 8 +#define SMU__NUM_MCLK_DPM_LEVELS 4 +#define SMU__NUM_LCLK_DPM_LEVELS 8 +#define SMU__NUM_PCIE_DPM_LEVELS 8 + +#define SMU7_CONTEXT_ID_SMC 1 +#define SMU7_CONTEXT_ID_VBIOS 2 + +#define SMU75_MAX_LEVELS_VDDC 16 +#define SMU75_MAX_LEVELS_VDDGFX 16 +#define SMU75_MAX_LEVELS_VDDCI 8 +#define SMU75_MAX_LEVELS_MVDD 4 + +#define SMU_MAX_SMIO_LEVELS 4 + +#define SMU75_MAX_LEVELS_GRAPHICS SMU__NUM_SCLK_DPM_STATE +#define SMU75_MAX_LEVELS_MEMORY SMU__NUM_MCLK_DPM_LEVELS +#define SMU75_MAX_LEVELS_GIO SMU__NUM_LCLK_DPM_LEVELS +#define SMU75_MAX_LEVELS_LINK SMU__NUM_PCIE_DPM_LEVELS +#define SMU75_MAX_LEVELS_UVD 8 +#define SMU75_MAX_LEVELS_VCE 8 +#define SMU75_MAX_LEVELS_ACP 8 +#define SMU75_MAX_LEVELS_SAMU 8 +#define SMU75_MAX_ENTRIES_SMIO 32 + +#define DPM_NO_LIMIT 0 +#define DPM_NO_UP 1 +#define DPM_GO_DOWN 2 +#define DPM_GO_UP 3 + +#define SMU7_FIRST_DPM_GRAPHICS_LEVEL 0 +#define SMU7_FIRST_DPM_MEMORY_LEVEL 0 + +#define GPIO_CLAMP_MODE_VRHOT 1 +#define GPIO_CLAMP_MODE_THERM 2 +#define GPIO_CLAMP_MODE_DC 4 + +#define SCRATCH_B_TARG_PCIE_INDEX_SHIFT 0 +#define SCRATCH_B_TARG_PCIE_INDEX_MASK (0x7<device, mmSMC_RESP_0, 0); cgs_write_register(hwmgr->device, mmSMC_MESSAGE_0, msg); PHM_WAIT_FIELD_UNEQUAL(hwmgr, SMC_RESP_0, SMC_RESP, 0); @@ -1182,7 +1177,6 @@ static int ci_populate_single_memory_level( struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); int result = 0; bool dll_state_on; - struct cgs_display_info info = {0}; uint32_t mclk_edc_wr_enable_threshold = 40000; uint32_t mclk_edc_enable_threshold = 40000; uint32_t mclk_strobe_mode_threshold = 40000; @@ -1236,8 +1230,7 @@ static int ci_populate_single_memory_level( /* default set to low watermark. Highest level will be set to high later.*/ memory_level->DisplayWatermark = PPSMC_DISPLAY_WATERMARK_LOW; - cgs_get_active_displays_info(hwmgr->device, &info); - data->display_timing.num_existing_displays = info.display_count; + data->display_timing.num_existing_displays = hwmgr->display_config->num_display; /* stutter mode not support on ci */ @@ -2784,7 +2777,6 @@ static int ci_smu_fini(struct pp_hwmgr *hwmgr) { kfree(hwmgr->smu_backend); hwmgr->smu_backend = NULL; - cgs_rel_firmware(hwmgr->device, CGS_UCODE_ID_SMU); return 0; } diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/fiji_smumgr.c b/drivers/gpu/drm/amd/powerplay/smumgr/fiji_smumgr.c index faef78321446..53df9405f43a 100644 --- a/drivers/gpu/drm/amd/powerplay/smumgr/fiji_smumgr.c +++ b/drivers/gpu/drm/amd/powerplay/smumgr/fiji_smumgr.c @@ -53,10 +53,7 @@ #define FIJI_SMC_SIZE 0x20000 -#define VOLTAGE_SCALE 4 #define POWERTUNE_DEFAULT_SET_MAX 1 -#define VOLTAGE_VID_OFFSET_SCALE1 625 -#define VOLTAGE_VID_OFFSET_SCALE2 100 #define VDDC_VDDCI_DELTA 300 #define MC_CG_ARB_FREQ_F1 0x0b @@ -288,8 +285,7 @@ static int fiji_start_smu(struct pp_hwmgr *hwmgr) struct fiji_smumgr *priv = (struct fiji_smumgr *)(hwmgr->smu_backend); /* Only start SMC if SMC RAM is not running */ - if (!(smu7_is_smc_ram_running(hwmgr) - || cgs_is_virtualization_enabled(hwmgr->device))) { + if (!smu7_is_smc_ram_running(hwmgr) && hwmgr->not_vf) { /* Check if SMU is running in protected mode */ if (0 == PHM_READ_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, @@ -307,13 +303,13 @@ static int fiji_start_smu(struct pp_hwmgr *hwmgr) } /* To initialize all clock gating before RLC loaded and running.*/ - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_GFX, AMD_CG_STATE_GATE); - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_GMC, AMD_CG_STATE_GATE); - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_SDMA, AMD_CG_STATE_GATE); - cgs_set_clockgating_state(hwmgr->device, + amdgpu_device_ip_set_clockgating_state(hwmgr->adev, AMD_IP_BLOCK_TYPE_COMMON, AMD_CG_STATE_GATE); /* Setup SoftRegsStart here for register lookup in case @@ -335,10 +331,10 @@ static bool fiji_is_hw_avfs_present(struct pp_hwmgr *hwmgr) uint32_t efuse = 0; uint32_t mask = (1 << ((AVFS_EN_MSB - AVFS_EN_LSB) + 1)) - 1; - if (cgs_is_virtualization_enabled(hwmgr->device)) - return 0; + if (!hwmgr->not_vf) + return false; - if (!atomctrl_read_efuse(hwmgr->device, AVFS_EN_LSB, AVFS_EN_MSB, + if (!atomctrl_read_efuse(hwmgr, AVFS_EN_LSB, AVFS_EN_MSB, mask, &efuse)) { if (efuse) return true; @@ -989,11 +985,11 @@ static int fiji_populate_single_graphic_level(struct pp_hwmgr *hwmgr, threshold = clock * data->fast_watermark_threshold / 100; - data->display_timing.min_clock_in_sr = hwmgr->display_config.min_core_set_clock_in_sr; + data->display_timing.min_clock_in_sr = hwmgr->display_config->min_core_set_clock_in_sr; if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_SclkDeepSleep)) level->DeepSleepDivId = smu7_get_sleep_divider_id_from_clock(clock, - hwmgr->display_config.min_core_set_clock_in_sr); + hwmgr->display_config->min_core_set_clock_in_sr); /* Default to slow, highest DPM level will be diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/iceland_smumgr.c b/drivers/gpu/drm/amd/powerplay/smumgr/iceland_smumgr.c index d4bb934e7334..415f691c3fa9 100644 --- a/drivers/gpu/drm/amd/powerplay/smumgr/iceland_smumgr.c +++ b/drivers/gpu/drm/amd/powerplay/smumgr/iceland_smumgr.c @@ -60,10 +60,7 @@ #define ICELAND_SMC_SIZE 0x20000 -#define VOLTAGE_SCALE 4 #define POWERTUNE_DEFAULT_SET_MAX 1 -#define VOLTAGE_VID_OFFSET_SCALE1 625 -#define VOLTAGE_VID_OFFSET_SCALE2 100 #define MC_CG_ARB_FREQ_F1 0x0b #define VDDC_VDDCI_DELTA 200 @@ -932,7 +929,7 @@ static int iceland_populate_single_graphic_level(struct pp_hwmgr *hwmgr, graphic_level->PowerThrottle = 0; data->display_timing.min_clock_in_sr = - hwmgr->display_config.min_core_set_clock_in_sr; + hwmgr->display_config->min_core_set_clock_in_sr; if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_SclkDeepSleep)) @@ -1236,7 +1233,6 @@ static int iceland_populate_single_memory_level( struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); int result = 0; bool dll_state_on; - struct cgs_display_info info = {0}; uint32_t mclk_edc_wr_enable_threshold = 40000; uint32_t mclk_edc_enable_threshold = 40000; uint32_t mclk_strobe_mode_threshold = 40000; @@ -1283,8 +1279,7 @@ static int iceland_populate_single_memory_level( /* default set to low watermark. Highest level will be set to high later.*/ memory_level->DisplayWatermark = PPSMC_DISPLAY_WATERMARK_LOW; - cgs_get_active_displays_info(hwmgr->device, &info); - data->display_timing.num_existing_displays = info.display_count; + data->display_timing.num_existing_displays = hwmgr->display_config->num_display; /* stutter mode not support on iceland */ diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/polaris10_smumgr.c b/drivers/gpu/drm/amd/powerplay/smumgr/polaris10_smumgr.c index 997a777dd35b..a8c6524f07e4 100644 --- a/drivers/gpu/drm/amd/powerplay/smumgr/polaris10_smumgr.c +++ b/drivers/gpu/drm/amd/powerplay/smumgr/polaris10_smumgr.c @@ -52,8 +52,6 @@ #include "dce/dce_10_0_sh_mask.h" #define POLARIS10_SMC_SIZE 0x20000 -#define VOLTAGE_VID_OFFSET_SCALE1 625 -#define VOLTAGE_VID_OFFSET_SCALE2 100 #define POWERTUNE_DEFAULT_SET_MAX 1 #define VDDC_VDDCI_DELTA 200 #define MC_CG_ARB_FREQ_F1 0x0b @@ -295,25 +293,16 @@ static int polaris10_start_smu(struct pp_hwmgr *hwmgr) struct polaris10_smumgr *smu_data = (struct polaris10_smumgr *)(hwmgr->smu_backend); /* Only start SMC if SMC RAM is not running */ - if (!(smu7_is_smc_ram_running(hwmgr) - || cgs_is_virtualization_enabled(hwmgr->device))) { + if (!smu7_is_smc_ram_running(hwmgr) && hwmgr->not_vf) { smu_data->protected_mode = (uint8_t) (PHM_READ_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, SMU_FIRMWARE, SMU_MODE)); smu_data->smu7_data.security_hard_key = (uint8_t) (PHM_READ_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, SMU_FIRMWARE, SMU_SEL)); /* Check if SMU is running in protected mode */ - if (smu_data->protected_mode == 0) { + if (smu_data->protected_mode == 0) result = polaris10_start_smu_in_non_protection_mode(hwmgr); - } else { + else result = polaris10_start_smu_in_protection_mode(hwmgr); - /* If failed, try with different security Key. */ - if (result != 0) { - smu_data->smu7_data.security_hard_key ^= 1; - cgs_rel_firmware(hwmgr->device, CGS_UCODE_ID_SMU); - result = polaris10_start_smu_in_protection_mode(hwmgr); - } - } - if (result != 0) PP_ASSERT_WITH_CODE(0, "Failed to load SMU ucode.", return result); @@ -951,11 +940,11 @@ static int polaris10_populate_single_graphic_level(struct pp_hwmgr *hwmgr, level->DownHyst = data->current_profile_setting.sclk_down_hyst; level->VoltageDownHyst = 0; level->PowerThrottle = 0; - data->display_timing.min_clock_in_sr = hwmgr->display_config.min_core_set_clock_in_sr; + data->display_timing.min_clock_in_sr = hwmgr->display_config->min_core_set_clock_in_sr; if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_SclkDeepSleep)) level->DeepSleepDivId = smu7_get_sleep_divider_id_from_clock(clock, - hwmgr->display_config.min_core_set_clock_in_sr); + hwmgr->display_config->min_core_set_clock_in_sr); /* Default to slow, highest DPM level will be * set to PPSMC_DISPLAY_WATERMARK_LOW later. @@ -1085,11 +1074,9 @@ static int polaris10_populate_single_memory_level(struct pp_hwmgr *hwmgr, struct phm_ppt_v1_information *table_info = (struct phm_ppt_v1_information *)(hwmgr->pptable); int result = 0; - struct cgs_display_info info = {0, 0, NULL}; uint32_t mclk_stutter_mode_threshold = 40000; phm_ppt_v1_clock_voltage_dependency_table *vdd_dep_table = NULL; - cgs_get_active_displays_info(hwmgr->device, &info); if (hwmgr->od_enabled) vdd_dep_table = (phm_ppt_v1_clock_voltage_dependency_table *)&data->odn_dpm_table.vdd_dependency_on_mclk; @@ -1115,7 +1102,7 @@ static int polaris10_populate_single_memory_level(struct pp_hwmgr *hwmgr, mem_level->StutterEnable = false; mem_level->DisplayWatermark = PPSMC_DISPLAY_WATERMARK_LOW; - data->display_timing.num_existing_displays = info.display_count; + data->display_timing.num_existing_displays = hwmgr->display_config->num_display; if (mclk_stutter_mode_threshold && (clock <= mclk_stutter_mode_threshold) && diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/smu10_smumgr.c b/drivers/gpu/drm/amd/powerplay/smumgr/smu10_smumgr.c index bc53f2beda30..0a563f6fe9ea 100644 --- a/drivers/gpu/drm/amd/powerplay/smumgr/smu10_smumgr.c +++ b/drivers/gpu/drm/amd/powerplay/smumgr/smu10_smumgr.c @@ -23,7 +23,7 @@ #include "smumgr.h" #include "smu10_inc.h" -#include "pp_soc15.h" +#include "soc15_common.h" #include "smu10_smumgr.h" #include "ppatomctrl.h" #include "rv_ppsmc.h" @@ -33,8 +33,6 @@ #include "pp_debug.h" -#define VOLTAGE_SCALE 4 - #define BUFFER_SIZE 80000 #define MAX_STRING_SIZE 15 #define BUFFER_SIZETWO 131072 @@ -49,48 +47,41 @@ static uint32_t smu10_wait_for_response(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; uint32_t reg; - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_90_BASE_IDX, mmMP1_SMN_C2PMSG_90); + reg = SOC15_REG_OFFSET(MP1, 0, mmMP1_SMN_C2PMSG_90); phm_wait_for_register_unequal(hwmgr, reg, 0, MP1_C2PMSG_90__CONTENT_MASK); - return cgs_read_register(hwmgr->device, reg); + return RREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90); } static int smu10_send_msg_to_smc_without_waiting(struct pp_hwmgr *hwmgr, uint16_t msg) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_66_BASE_IDX, mmMP1_SMN_C2PMSG_66); - cgs_write_register(hwmgr->device, reg, msg); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_66, msg); return 0; } static int smu10_read_arg_from_smc(struct pp_hwmgr *hwmgr) { - uint32_t reg; - - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_82_BASE_IDX, mmMP1_SMN_C2PMSG_82); + struct amdgpu_device *adev = hwmgr->adev; - return cgs_read_register(hwmgr->device, reg); + return RREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_82); } static int smu10_send_msg_to_smc(struct pp_hwmgr *hwmgr, uint16_t msg) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; smu10_wait_for_response(hwmgr); - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_90_BASE_IDX, mmMP1_SMN_C2PMSG_90); - cgs_write_register(hwmgr->device, reg, 0); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90, 0); smu10_send_msg_to_smc_without_waiting(hwmgr, msg); @@ -104,17 +95,13 @@ static int smu10_send_msg_to_smc(struct pp_hwmgr *hwmgr, uint16_t msg) static int smu10_send_msg_to_smc_with_parameter(struct pp_hwmgr *hwmgr, uint16_t msg, uint32_t parameter) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; smu10_wait_for_response(hwmgr); - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_90_BASE_IDX, mmMP1_SMN_C2PMSG_90); - cgs_write_register(hwmgr->device, reg, 0); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90, 0); - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_82_BASE_IDX, mmMP1_SMN_C2PMSG_82); - cgs_write_register(hwmgr->device, reg, parameter); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_82, parameter); smu10_send_msg_to_smc_without_waiting(hwmgr, msg); diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/smu7_smumgr.c b/drivers/gpu/drm/amd/powerplay/smumgr/smu7_smumgr.c index 0399c10d2be0..d644a9bb9078 100644 --- a/drivers/gpu/drm/amd/powerplay/smumgr/smu7_smumgr.c +++ b/drivers/gpu/drm/amd/powerplay/smumgr/smu7_smumgr.c @@ -167,24 +167,25 @@ int smu7_send_msg_to_smc(struct pp_hwmgr *hwmgr, uint16_t msg) { int ret; - if (!smu7_is_smc_ram_running(hwmgr)) - return -EINVAL; - - PHM_WAIT_FIELD_UNEQUAL(hwmgr, SMC_RESP_0, SMC_RESP, 0); ret = PHM_READ_FIELD(hwmgr->device, SMC_RESP_0, SMC_RESP); - if (ret != 1) - pr_info("\n failed to send pre message %x ret is %d \n", msg, ret); + if (ret == 0xFE) + pr_debug("last message was not supported\n"); + else if (ret != 1) + pr_info("\n last message was failed ret is %d\n", ret); + cgs_write_register(hwmgr->device, mmSMC_RESP_0, 0); cgs_write_register(hwmgr->device, mmSMC_MESSAGE_0, msg); PHM_WAIT_FIELD_UNEQUAL(hwmgr, SMC_RESP_0, SMC_RESP, 0); ret = PHM_READ_FIELD(hwmgr->device, SMC_RESP_0, SMC_RESP); - if (ret != 1) + if (ret == 0xFE) + pr_debug("message %x was not supported\n", msg); + else if (ret != 1) pr_info("\n failed to send message %x ret is %d \n", msg, ret); return 0; @@ -199,10 +200,6 @@ int smu7_send_msg_to_smc_without_waiting(struct pp_hwmgr *hwmgr, uint16_t msg) int smu7_send_msg_to_smc_with_parameter(struct pp_hwmgr *hwmgr, uint16_t msg, uint32_t parameter) { - if (!smu7_is_smc_ram_running(hwmgr)) { - return -EINVAL; - } - PHM_WAIT_FIELD_UNEQUAL(hwmgr, SMC_RESP_0, SMC_RESP, 0); cgs_write_register(hwmgr->device, mmSMC_MSG_ARG_0, parameter); @@ -231,16 +228,6 @@ int smu7_send_msg_to_smc_offset(struct pp_hwmgr *hwmgr) return 0; } -int smu7_wait_for_smc_inactive(struct pp_hwmgr *hwmgr) -{ - if (!smu7_is_smc_ram_running(hwmgr)) - return -EINVAL; - - PHM_WAIT_VFPF_INDIRECT_FIELD(hwmgr, SMC_IND, SMC_SYSCON_CLOCK_CNTL_0, cken, 0); - return 0; -} - - enum cgs_ucode_id smu7_convert_fw_type_to_cgs(uint32_t fw_type) { enum cgs_ucode_id result = CGS_UCODE_ID_MAXIMUM; @@ -296,11 +283,9 @@ int smu7_read_smc_sram_dword(struct pp_hwmgr *hwmgr, uint32_t smc_addr, uint32_t result = smu7_set_smc_sram_address(hwmgr, smc_addr, limit); - if (result) - return result; + *value = result ? 0 : cgs_read_register(hwmgr->device, mmSMC_IND_DATA_11); - *value = cgs_read_register(hwmgr->device, mmSMC_IND_DATA_11); - return 0; + return result; } int smu7_write_smc_sram_dword(struct pp_hwmgr *hwmgr, uint32_t smc_addr, uint32_t value, uint32_t limit) @@ -375,7 +360,7 @@ static int smu7_populate_single_firmware_entry(struct pp_hwmgr *hwmgr, entry->meta_data_addr_low = 0; /* digest need be excluded out */ - if (cgs_is_virtualization_enabled(hwmgr->device)) + if (!hwmgr->not_vf) info.image_size -= 20; entry->data_size_byte = info.image_size; entry->num_register_entries = 0; @@ -409,7 +394,7 @@ int smu7_request_smu_load_fw(struct pp_hwmgr *hwmgr) 0x0); if (hwmgr->chip_id > CHIP_TOPAZ) { /* add support for Topaz */ - if (!cgs_is_virtualization_enabled(hwmgr->device)) { + if (hwmgr->not_vf) { smu7_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SMU_DRAM_ADDR_HI, upper_32_bits(smu_data->smu_buffer.mc_addr)); @@ -467,7 +452,7 @@ int smu7_request_smu_load_fw(struct pp_hwmgr *hwmgr) PP_ASSERT_WITH_CODE(0 == smu7_populate_single_firmware_entry(hwmgr, UCODE_ID_SDMA1, &toc->entry[toc->num_entries++]), "Failed to Get Firmware Entry.", return -EINVAL); - if (cgs_is_virtualization_enabled(hwmgr->device)) + if (!hwmgr->not_vf) PP_ASSERT_WITH_CODE(0 == smu7_populate_single_firmware_entry(hwmgr, UCODE_ID_MEC_STORAGE, &toc->entry[toc->num_entries++]), "Failed to Get Firmware Entry.", return -EINVAL); @@ -608,7 +593,7 @@ int smu7_init(struct pp_hwmgr *hwmgr) smu_data->header = smu_data->header_buffer.kaddr; smu_data->header_buffer.mc_addr = mc_addr; - if (cgs_is_virtualization_enabled(hwmgr->device)) + if (!hwmgr->not_vf) return 0; smu_data->smu_buffer.data_size = 200*4096; @@ -643,13 +628,12 @@ int smu7_smu_fini(struct pp_hwmgr *hwmgr) &smu_data->header_buffer.mc_addr, &smu_data->header_buffer.kaddr); - if (!cgs_is_virtualization_enabled(hwmgr->device)) + if (hwmgr->not_vf) amdgpu_bo_free_kernel(&smu_data->smu_buffer.handle, &smu_data->smu_buffer.mc_addr, &smu_data->smu_buffer.kaddr); kfree(hwmgr->smu_backend); hwmgr->smu_backend = NULL; - cgs_rel_firmware(hwmgr->device, CGS_UCODE_ID_SMU); return 0; } diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/smu7_smumgr.h b/drivers/gpu/drm/amd/powerplay/smumgr/smu7_smumgr.h index 126d300259ba..39c9bfda0ab4 100644 --- a/drivers/gpu/drm/amd/powerplay/smumgr/smu7_smumgr.h +++ b/drivers/gpu/drm/amd/powerplay/smumgr/smu7_smumgr.h @@ -67,7 +67,6 @@ int smu7_send_msg_to_smc_with_parameter(struct pp_hwmgr *hwmgr, uint16_t msg, int smu7_send_msg_to_smc_with_parameter_without_waiting(struct pp_hwmgr *hwmgr, uint16_t msg, uint32_t parameter); int smu7_send_msg_to_smc_offset(struct pp_hwmgr *hwmgr); -int smu7_wait_for_smc_inactive(struct pp_hwmgr *hwmgr); enum cgs_ucode_id smu7_convert_fw_type_to_cgs(uint32_t fw_type); int smu7_read_smc_sram_dword(struct pp_hwmgr *hwmgr, uint32_t smc_addr, diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/smumgr.c b/drivers/gpu/drm/amd/powerplay/smumgr/smumgr.c index c28b60aae5f8..c9837935f0f5 100644 --- a/drivers/gpu/drm/amd/powerplay/smumgr/smumgr.c +++ b/drivers/gpu/drm/amd/powerplay/smumgr/smumgr.c @@ -41,9 +41,11 @@ MODULE_FIRMWARE("amdgpu/polaris11_smc.bin"); MODULE_FIRMWARE("amdgpu/polaris11_smc_sk.bin"); MODULE_FIRMWARE("amdgpu/polaris11_k_smc.bin"); MODULE_FIRMWARE("amdgpu/polaris12_smc.bin"); +MODULE_FIRMWARE("amdgpu/vegam_smc.bin"); MODULE_FIRMWARE("amdgpu/vega10_smc.bin"); MODULE_FIRMWARE("amdgpu/vega10_acg_smc.bin"); MODULE_FIRMWARE("amdgpu/vega12_smc.bin"); +MODULE_FIRMWARE("amdgpu/vega20_smc.bin"); int smum_thermal_avfs_enable(struct pp_hwmgr *hwmgr) { diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/tonga_smumgr.c b/drivers/gpu/drm/amd/powerplay/smumgr/tonga_smumgr.c index b51d7468c3e7..782b19fc2e70 100644 --- a/drivers/gpu/drm/amd/powerplay/smumgr/tonga_smumgr.c +++ b/drivers/gpu/drm/amd/powerplay/smumgr/tonga_smumgr.c @@ -55,11 +55,7 @@ #include "dce/dce_10_0_d.h" #include "dce/dce_10_0_sh_mask.h" - -#define VOLTAGE_SCALE 4 #define POWERTUNE_DEFAULT_SET_MAX 1 -#define VOLTAGE_VID_OFFSET_SCALE1 625 -#define VOLTAGE_VID_OFFSET_SCALE2 100 #define MC_CG_ARB_FREQ_F1 0x0b #define VDDC_VDDCI_DELTA 200 @@ -199,8 +195,7 @@ static int tonga_start_smu(struct pp_hwmgr *hwmgr) int result; /* Only start SMC if SMC RAM is not running */ - if (!(smu7_is_smc_ram_running(hwmgr) || - cgs_is_virtualization_enabled(hwmgr->device))) { + if (!smu7_is_smc_ram_running(hwmgr) && hwmgr->not_vf) { /*Check if SMU is running in protected mode*/ if (0 == PHM_READ_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, SMU_FIRMWARE, SMU_MODE)) { @@ -651,7 +646,7 @@ static int tonga_populate_single_graphic_level(struct pp_hwmgr *hwmgr, graphic_level->PowerThrottle = 0; data->display_timing.min_clock_in_sr = - hwmgr->display_config.min_core_set_clock_in_sr; + hwmgr->display_config->min_core_set_clock_in_sr; if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_SclkDeepSleep)) @@ -957,18 +952,17 @@ static int tonga_populate_single_memory_level( SMU72_Discrete_MemoryLevel *memory_level ) { - uint32_t mvdd = 0; struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); struct phm_ppt_v1_information *pptable_info = (struct phm_ppt_v1_information *)(hwmgr->pptable); - int result = 0; - bool dll_state_on; - struct cgs_display_info info = {0}; uint32_t mclk_edc_wr_enable_threshold = 40000; uint32_t mclk_stutter_mode_threshold = 30000; uint32_t mclk_edc_enable_threshold = 40000; uint32_t mclk_strobe_mode_threshold = 40000; phm_ppt_v1_clock_voltage_dependency_table *vdd_dep_table = NULL; + int result = 0; + bool dll_state_on; + uint32_t mvdd = 0; if (hwmgr->od_enabled) vdd_dep_table = (phm_ppt_v1_clock_voltage_dependency_table *)&data->odn_dpm_table.vdd_dependency_on_mclk; @@ -1009,8 +1003,7 @@ static int tonga_populate_single_memory_level( /* default set to low watermark. Highest level will be set to high later.*/ memory_level->DisplayWatermark = PPSMC_DISPLAY_WATERMARK_LOW; - cgs_get_active_displays_info(hwmgr->device, &info); - data->display_timing.num_existing_displays = info.display_count; + data->display_timing.num_existing_displays = hwmgr->display_config->num_display; if ((mclk_stutter_mode_threshold != 0) && (memory_clock <= mclk_stutter_mode_threshold) && diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/vega10_smumgr.c b/drivers/gpu/drm/amd/powerplay/smumgr/vega10_smumgr.c index 4aafb043bcb0..e84669c448a3 100644 --- a/drivers/gpu/drm/amd/powerplay/smumgr/vega10_smumgr.c +++ b/drivers/gpu/drm/amd/powerplay/smumgr/vega10_smumgr.c @@ -23,7 +23,7 @@ #include "smumgr.h" #include "vega10_inc.h" -#include "pp_soc15.h" +#include "soc15_common.h" #include "vega10_smumgr.h" #include "vega10_hwmgr.h" #include "vega10_ppsmc.h" @@ -35,8 +35,6 @@ #define AVFS_EN_MSB 1568 #define AVFS_EN_LSB 1568 -#define VOLTAGE_SCALE 4 - /* Microcode file is stored in this buffer */ #define BUFFER_SIZE 80000 #define MAX_STRING_SIZE 15 @@ -54,18 +52,13 @@ static bool vega10_is_smc_ram_running(struct pp_hwmgr *hwmgr) { - uint32_t mp1_fw_flags, reg; - - reg = soc15_get_register_offset(NBIF_HWID, 0, - mmPCIE_INDEX2_BASE_IDX, mmPCIE_INDEX2); + struct amdgpu_device *adev = hwmgr->adev; + uint32_t mp1_fw_flags; - cgs_write_register(hwmgr->device, reg, + WREG32_SOC15(NBIF, 0, mmPCIE_INDEX2, (MP1_Public | (smnMP1_FIRMWARE_FLAGS & 0xffffffff))); - reg = soc15_get_register_offset(NBIF_HWID, 0, - mmPCIE_DATA2_BASE_IDX, mmPCIE_DATA2); - - mp1_fw_flags = cgs_read_register(hwmgr->device, reg); + mp1_fw_flags = RREG32_SOC15(NBIF, 0, mmPCIE_DATA2); if (mp1_fw_flags & MP1_FIRMWARE_FLAGS__INTERRUPTS_ENABLED_MASK) return true; @@ -81,11 +74,11 @@ static bool vega10_is_smc_ram_running(struct pp_hwmgr *hwmgr) */ static uint32_t vega10_wait_for_response(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; uint32_t reg; uint32_t ret; - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_90_BASE_IDX, mmMP1_SMN_C2PMSG_90); + reg = SOC15_REG_OFFSET(MP1, 0, mmMP1_SMN_C2PMSG_90); ret = phm_wait_for_register_unequal(hwmgr, reg, 0, MP1_C2PMSG_90__CONTENT_MASK); @@ -93,7 +86,7 @@ static uint32_t vega10_wait_for_response(struct pp_hwmgr *hwmgr) if (ret) pr_err("No response from smu\n"); - return cgs_read_register(hwmgr->device, reg); + return RREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90); } /* @@ -105,11 +98,9 @@ static uint32_t vega10_wait_for_response(struct pp_hwmgr *hwmgr) static int vega10_send_msg_to_smc_without_waiting(struct pp_hwmgr *hwmgr, uint16_t msg) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_66_BASE_IDX, mmMP1_SMN_C2PMSG_66); - cgs_write_register(hwmgr->device, reg, msg); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_66, msg); return 0; } @@ -122,14 +113,12 @@ static int vega10_send_msg_to_smc_without_waiting(struct pp_hwmgr *hwmgr, */ static int vega10_send_msg_to_smc(struct pp_hwmgr *hwmgr, uint16_t msg) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; uint32_t ret; vega10_wait_for_response(hwmgr); - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_90_BASE_IDX, mmMP1_SMN_C2PMSG_90); - cgs_write_register(hwmgr->device, reg, 0); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90, 0); vega10_send_msg_to_smc_without_waiting(hwmgr, msg); @@ -150,18 +139,14 @@ static int vega10_send_msg_to_smc(struct pp_hwmgr *hwmgr, uint16_t msg) static int vega10_send_msg_to_smc_with_parameter(struct pp_hwmgr *hwmgr, uint16_t msg, uint32_t parameter) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; uint32_t ret; vega10_wait_for_response(hwmgr); - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_90_BASE_IDX, mmMP1_SMN_C2PMSG_90); - cgs_write_register(hwmgr->device, reg, 0); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90, 0); - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_82_BASE_IDX, mmMP1_SMN_C2PMSG_82); - cgs_write_register(hwmgr->device, reg, parameter); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_82, parameter); vega10_send_msg_to_smc_without_waiting(hwmgr, msg); @@ -174,12 +159,9 @@ static int vega10_send_msg_to_smc_with_parameter(struct pp_hwmgr *hwmgr, static int vega10_get_argument(struct pp_hwmgr *hwmgr) { - uint32_t reg; - - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_82_BASE_IDX, mmMP1_SMN_C2PMSG_82); + struct amdgpu_device *adev = hwmgr->adev; - return cgs_read_register(hwmgr->device, reg); + return RREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_82); } static int vega10_copy_table_from_smc(struct pp_hwmgr *hwmgr, diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/vega12_smumgr.c b/drivers/gpu/drm/amd/powerplay/smumgr/vega12_smumgr.c index 651a3f28734b..7d9b40e8b1bf 100644 --- a/drivers/gpu/drm/amd/powerplay/smumgr/vega12_smumgr.c +++ b/drivers/gpu/drm/amd/powerplay/smumgr/vega12_smumgr.c @@ -23,7 +23,7 @@ #include "smumgr.h" #include "vega12_inc.h" -#include "pp_soc15.h" +#include "soc15_common.h" #include "vega12_smumgr.h" #include "vega12_ppsmc.h" #include "vega12/smu9_driver_if.h" @@ -44,18 +44,13 @@ static bool vega12_is_smc_ram_running(struct pp_hwmgr *hwmgr) { - uint32_t mp1_fw_flags, reg; + struct amdgpu_device *adev = hwmgr->adev; + uint32_t mp1_fw_flags; - reg = soc15_get_register_offset(NBIF_HWID, 0, - mmPCIE_INDEX2_BASE_IDX, mmPCIE_INDEX2); - - cgs_write_register(hwmgr->device, reg, + WREG32_SOC15(NBIF, 0, mmPCIE_INDEX2, (MP1_Public | (smnMP1_FIRMWARE_FLAGS & 0xffffffff))); - reg = soc15_get_register_offset(NBIF_HWID, 0, - mmPCIE_DATA2_BASE_IDX, mmPCIE_DATA2); - - mp1_fw_flags = cgs_read_register(hwmgr->device, reg); + mp1_fw_flags = RREG32_SOC15(NBIF, 0, mmPCIE_DATA2); if ((mp1_fw_flags & MP1_FIRMWARE_FLAGS__INTERRUPTS_ENABLED_MASK) >> MP1_FIRMWARE_FLAGS__INTERRUPTS_ENABLED__SHIFT) @@ -72,15 +67,15 @@ static bool vega12_is_smc_ram_running(struct pp_hwmgr *hwmgr) */ static uint32_t vega12_wait_for_response(struct pp_hwmgr *hwmgr) { + struct amdgpu_device *adev = hwmgr->adev; uint32_t reg; - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_90_BASE_IDX, mmMP1_SMN_C2PMSG_90); + reg = SOC15_REG_OFFSET(MP1, 0, mmMP1_SMN_C2PMSG_90); phm_wait_for_register_unequal(hwmgr, reg, 0, MP1_C2PMSG_90__CONTENT_MASK); - return cgs_read_register(hwmgr->device, reg); + return RREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90); } /* @@ -92,11 +87,9 @@ static uint32_t vega12_wait_for_response(struct pp_hwmgr *hwmgr) int vega12_send_msg_to_smc_without_waiting(struct pp_hwmgr *hwmgr, uint16_t msg) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_66_BASE_IDX, mmMP1_SMN_C2PMSG_66); - cgs_write_register(hwmgr->device, reg, msg); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_66, msg); return 0; } @@ -109,13 +102,11 @@ int vega12_send_msg_to_smc_without_waiting(struct pp_hwmgr *hwmgr, */ int vega12_send_msg_to_smc(struct pp_hwmgr *hwmgr, uint16_t msg) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; vega12_wait_for_response(hwmgr); - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_90_BASE_IDX, mmMP1_SMN_C2PMSG_90); - cgs_write_register(hwmgr->device, reg, 0); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90, 0); vega12_send_msg_to_smc_without_waiting(hwmgr, msg); @@ -135,17 +126,13 @@ int vega12_send_msg_to_smc(struct pp_hwmgr *hwmgr, uint16_t msg) int vega12_send_msg_to_smc_with_parameter(struct pp_hwmgr *hwmgr, uint16_t msg, uint32_t parameter) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; vega12_wait_for_response(hwmgr); - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_90_BASE_IDX, mmMP1_SMN_C2PMSG_90); - cgs_write_register(hwmgr->device, reg, 0); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90, 0); - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_82_BASE_IDX, mmMP1_SMN_C2PMSG_82); - cgs_write_register(hwmgr->device, reg, parameter); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_82, parameter); vega12_send_msg_to_smc_without_waiting(hwmgr, msg); @@ -166,11 +153,9 @@ int vega12_send_msg_to_smc_with_parameter(struct pp_hwmgr *hwmgr, int vega12_send_msg_to_smc_with_parameter_without_waiting( struct pp_hwmgr *hwmgr, uint16_t msg, uint32_t parameter) { - uint32_t reg; + struct amdgpu_device *adev = hwmgr->adev; - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_66_BASE_IDX, mmMP1_SMN_C2PMSG_66); - cgs_write_register(hwmgr->device, reg, parameter); + WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_66, parameter); return vega12_send_msg_to_smc_without_waiting(hwmgr, msg); } @@ -183,12 +168,9 @@ int vega12_send_msg_to_smc_with_parameter_without_waiting( */ int vega12_read_arg_from_smc(struct pp_hwmgr *hwmgr, uint32_t *arg) { - uint32_t reg; - - reg = soc15_get_register_offset(MP1_HWID, 0, - mmMP1_SMN_C2PMSG_82_BASE_IDX, mmMP1_SMN_C2PMSG_82); + struct amdgpu_device *adev = hwmgr->adev; - *arg = cgs_read_register(hwmgr->device, reg); + *arg = RREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_82); return 0; } diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/vegam_smumgr.c b/drivers/gpu/drm/amd/powerplay/smumgr/vegam_smumgr.c new file mode 100644 index 000000000000..2de48959ac93 --- /dev/null +++ b/drivers/gpu/drm/amd/powerplay/smumgr/vegam_smumgr.c @@ -0,0 +1,2383 @@ +/* + * Copyright 2017 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ +#include "pp_debug.h" +#include "smumgr.h" +#include "smu_ucode_xfer_vi.h" +#include "vegam_smumgr.h" +#include "smu/smu_7_1_3_d.h" +#include "smu/smu_7_1_3_sh_mask.h" +#include "gmc/gmc_8_1_d.h" +#include "gmc/gmc_8_1_sh_mask.h" +#include "oss/oss_3_0_d.h" +#include "gca/gfx_8_0_d.h" +#include "bif/bif_5_0_d.h" +#include "bif/bif_5_0_sh_mask.h" +#include "ppatomctrl.h" +#include "cgs_common.h" +#include "smu7_ppsmc.h" + +#include "smu7_dyn_defaults.h" + +#include "smu7_hwmgr.h" +#include "hardwaremanager.h" +#include "ppatomctrl.h" +#include "atombios.h" +#include "pppcielanes.h" + +#include "dce/dce_11_2_d.h" +#include "dce/dce_11_2_sh_mask.h" + +#define PPVEGAM_TARGETACTIVITY_DFLT 50 + +#define VOLTAGE_VID_OFFSET_SCALE1 625 +#define VOLTAGE_VID_OFFSET_SCALE2 100 +#define POWERTUNE_DEFAULT_SET_MAX 1 +#define VDDC_VDDCI_DELTA 200 +#define MC_CG_ARB_FREQ_F1 0x0b + +#define STRAP_ASIC_RO_LSB 2168 +#define STRAP_ASIC_RO_MSB 2175 + +#define PPSMC_MSG_ApplyAvfsCksOffVoltage ((uint16_t) 0x415) +#define PPSMC_MSG_EnableModeSwitchRLCNotification ((uint16_t) 0x305) + +static const struct vegam_pt_defaults +vegam_power_tune_data_set_array[POWERTUNE_DEFAULT_SET_MAX] = { + /* sviLoadLIneEn, SviLoadLineVddC, TDC_VDDC_ThrottleReleaseLimitPerc, TDC_MAWt, + * TdcWaterfallCtl, DTEAmbientTempBase, DisplayCac, BAPM_TEMP_GRADIENT */ + { 1, 0xF, 0xFD, 0x19, 5, 45, 0, 0xB0000, + { 0x79, 0x253, 0x25D, 0xAE, 0x72, 0x80, 0x83, 0x86, 0x6F, 0xC8, 0xC9, 0xC9, 0x2F, 0x4D, 0x61}, + { 0x17C, 0x172, 0x180, 0x1BC, 0x1B3, 0x1BD, 0x206, 0x200, 0x203, 0x25D, 0x25A, 0x255, 0x2C3, 0x2C5, 0x2B4 } }, +}; + +static const sclkFcwRange_t Range_Table[NUM_SCLK_RANGE] = { + {VCO_2_4, POSTDIV_DIV_BY_16, 75, 160, 112}, + {VCO_3_6, POSTDIV_DIV_BY_16, 112, 224, 160}, + {VCO_2_4, POSTDIV_DIV_BY_8, 75, 160, 112}, + {VCO_3_6, POSTDIV_DIV_BY_8, 112, 224, 160}, + {VCO_2_4, POSTDIV_DIV_BY_4, 75, 160, 112}, + {VCO_3_6, POSTDIV_DIV_BY_4, 112, 216, 160}, + {VCO_2_4, POSTDIV_DIV_BY_2, 75, 160, 108}, + {VCO_3_6, POSTDIV_DIV_BY_2, 112, 216, 160} }; + +static int vegam_smu_init(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data; + + smu_data = kzalloc(sizeof(struct vegam_smumgr), GFP_KERNEL); + if (smu_data == NULL) + return -ENOMEM; + + hwmgr->smu_backend = smu_data; + + if (smu7_init(hwmgr)) { + kfree(smu_data); + return -EINVAL; + } + + return 0; +} + +static int vegam_start_smu_in_protection_mode(struct pp_hwmgr *hwmgr) +{ + int result = 0; + + /* Wait for smc boot up */ + /* PHM_WAIT_VFPF_INDIRECT_FIELD_UNEQUAL(smumgr, SMC_IND, RCU_UC_EVENTS, boot_seq_done, 0) */ + + /* Assert reset */ + PHM_WRITE_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, + SMC_SYSCON_RESET_CNTL, rst_reg, 1); + + result = smu7_upload_smu_firmware_image(hwmgr); + if (result != 0) + return result; + + /* Clear status */ + cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixSMU_STATUS, 0); + + PHM_WRITE_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, + SMC_SYSCON_CLOCK_CNTL_0, ck_disable, 0); + + /* De-assert reset */ + PHM_WRITE_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, + SMC_SYSCON_RESET_CNTL, rst_reg, 0); + + + PHM_WAIT_VFPF_INDIRECT_FIELD(hwmgr, SMC_IND, RCU_UC_EVENTS, INTERRUPTS_ENABLED, 1); + + + /* Call Test SMU message with 0x20000 offset to trigger SMU start */ + smu7_send_msg_to_smc_offset(hwmgr); + + /* Wait done bit to be set */ + /* Check pass/failed indicator */ + + PHM_WAIT_VFPF_INDIRECT_FIELD_UNEQUAL(hwmgr, SMC_IND, SMU_STATUS, SMU_DONE, 0); + + if (1 != PHM_READ_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, + SMU_STATUS, SMU_PASS)) + PP_ASSERT_WITH_CODE(false, "SMU Firmware start failed!", return -1); + + cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixFIRMWARE_FLAGS, 0); + + PHM_WRITE_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, + SMC_SYSCON_RESET_CNTL, rst_reg, 1); + + PHM_WRITE_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, + SMC_SYSCON_RESET_CNTL, rst_reg, 0); + + /* Wait for firmware to initialize */ + PHM_WAIT_VFPF_INDIRECT_FIELD(hwmgr, SMC_IND, FIRMWARE_FLAGS, INTERRUPTS_ENABLED, 1); + + return result; +} + +static int vegam_start_smu_in_non_protection_mode(struct pp_hwmgr *hwmgr) +{ + int result = 0; + + /* wait for smc boot up */ + PHM_WAIT_VFPF_INDIRECT_FIELD_UNEQUAL(hwmgr, SMC_IND, RCU_UC_EVENTS, boot_seq_done, 0); + + /* Clear firmware interrupt enable flag */ + /* PHM_WRITE_VFPF_INDIRECT_FIELD(pSmuMgr, SMC_IND, SMC_SYSCON_MISC_CNTL, pre_fetcher_en, 1); */ + cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, + ixFIRMWARE_FLAGS, 0); + + PHM_WRITE_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, + SMC_SYSCON_RESET_CNTL, + rst_reg, 1); + + result = smu7_upload_smu_firmware_image(hwmgr); + if (result != 0) + return result; + + /* Set smc instruct start point at 0x0 */ + smu7_program_jump_on_start(hwmgr); + + PHM_WRITE_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, + SMC_SYSCON_CLOCK_CNTL_0, ck_disable, 0); + + PHM_WRITE_VFPF_INDIRECT_FIELD(hwmgr->device, CGS_IND_REG__SMC, + SMC_SYSCON_RESET_CNTL, rst_reg, 0); + + /* Wait for firmware to initialize */ + + PHM_WAIT_VFPF_INDIRECT_FIELD(hwmgr, SMC_IND, + FIRMWARE_FLAGS, INTERRUPTS_ENABLED, 1); + + return result; +} + +static int vegam_start_smu(struct pp_hwmgr *hwmgr) +{ + int result = 0; + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + + /* Only start SMC if SMC RAM is not running */ + if (!smu7_is_smc_ram_running(hwmgr) && hwmgr->not_vf) { + smu_data->protected_mode = (uint8_t)(PHM_READ_VFPF_INDIRECT_FIELD(hwmgr->device, + CGS_IND_REG__SMC, SMU_FIRMWARE, SMU_MODE)); + smu_data->smu7_data.security_hard_key = (uint8_t)(PHM_READ_VFPF_INDIRECT_FIELD( + hwmgr->device, CGS_IND_REG__SMC, SMU_FIRMWARE, SMU_SEL)); + + /* Check if SMU is running in protected mode */ + if (smu_data->protected_mode == 0) + result = vegam_start_smu_in_non_protection_mode(hwmgr); + else + result = vegam_start_smu_in_protection_mode(hwmgr); + + if (result != 0) + PP_ASSERT_WITH_CODE(0, "Failed to load SMU ucode.", return result); + } + + /* Setup SoftRegsStart here for register lookup in case DummyBackEnd is used and ProcessFirmwareHeader is not executed */ + smu7_read_smc_sram_dword(hwmgr, + SMU7_FIRMWARE_HEADER_LOCATION + offsetof(SMU75_Firmware_Header, SoftRegisters), + &(smu_data->smu7_data.soft_regs_start), + 0x40000); + + result = smu7_request_smu_load_fw(hwmgr); + + return result; +} + +static int vegam_process_firmware_header(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + uint32_t tmp; + int result; + bool error = false; + + result = smu7_read_smc_sram_dword(hwmgr, + SMU7_FIRMWARE_HEADER_LOCATION + + offsetof(SMU75_Firmware_Header, DpmTable), + &tmp, SMC_RAM_END); + + if (0 == result) + smu_data->smu7_data.dpm_table_start = tmp; + + error |= (0 != result); + + result = smu7_read_smc_sram_dword(hwmgr, + SMU7_FIRMWARE_HEADER_LOCATION + + offsetof(SMU75_Firmware_Header, SoftRegisters), + &tmp, SMC_RAM_END); + + if (!result) { + data->soft_regs_start = tmp; + smu_data->smu7_data.soft_regs_start = tmp; + } + + error |= (0 != result); + + result = smu7_read_smc_sram_dword(hwmgr, + SMU7_FIRMWARE_HEADER_LOCATION + + offsetof(SMU75_Firmware_Header, mcRegisterTable), + &tmp, SMC_RAM_END); + + if (!result) + smu_data->smu7_data.mc_reg_table_start = tmp; + + result = smu7_read_smc_sram_dword(hwmgr, + SMU7_FIRMWARE_HEADER_LOCATION + + offsetof(SMU75_Firmware_Header, FanTable), + &tmp, SMC_RAM_END); + + if (!result) + smu_data->smu7_data.fan_table_start = tmp; + + error |= (0 != result); + + result = smu7_read_smc_sram_dword(hwmgr, + SMU7_FIRMWARE_HEADER_LOCATION + + offsetof(SMU75_Firmware_Header, mcArbDramTimingTable), + &tmp, SMC_RAM_END); + + if (!result) + smu_data->smu7_data.arb_table_start = tmp; + + error |= (0 != result); + + result = smu7_read_smc_sram_dword(hwmgr, + SMU7_FIRMWARE_HEADER_LOCATION + + offsetof(SMU75_Firmware_Header, Version), + &tmp, SMC_RAM_END); + + if (!result) + hwmgr->microcode_version_info.SMC = tmp; + + error |= (0 != result); + + return error ? -1 : 0; +} + +static bool vegam_is_dpm_running(struct pp_hwmgr *hwmgr) +{ + return (1 == PHM_READ_INDIRECT_FIELD(hwmgr->device, + CGS_IND_REG__SMC, FEATURE_STATUS, VOLTAGE_CONTROLLER_ON)) + ? true : false; +} + +static uint32_t vegam_get_mac_definition(uint32_t value) +{ + switch (value) { + case SMU_MAX_LEVELS_GRAPHICS: + return SMU75_MAX_LEVELS_GRAPHICS; + case SMU_MAX_LEVELS_MEMORY: + return SMU75_MAX_LEVELS_MEMORY; + case SMU_MAX_LEVELS_LINK: + return SMU75_MAX_LEVELS_LINK; + case SMU_MAX_ENTRIES_SMIO: + return SMU75_MAX_ENTRIES_SMIO; + case SMU_MAX_LEVELS_VDDC: + return SMU75_MAX_LEVELS_VDDC; + case SMU_MAX_LEVELS_VDDGFX: + return SMU75_MAX_LEVELS_VDDGFX; + case SMU_MAX_LEVELS_VDDCI: + return SMU75_MAX_LEVELS_VDDCI; + case SMU_MAX_LEVELS_MVDD: + return SMU75_MAX_LEVELS_MVDD; + case SMU_UVD_MCLK_HANDSHAKE_DISABLE: + return SMU7_UVD_MCLK_HANDSHAKE_DISABLE | + SMU7_VCE_MCLK_HANDSHAKE_DISABLE; + } + + pr_warn("can't get the mac of %x\n", value); + return 0; +} + +static int vegam_update_uvd_smc_table(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + uint32_t mm_boot_level_offset, mm_boot_level_value; + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + + smu_data->smc_state_table.UvdBootLevel = 0; + if (table_info->mm_dep_table->count > 0) + smu_data->smc_state_table.UvdBootLevel = + (uint8_t) (table_info->mm_dep_table->count - 1); + mm_boot_level_offset = smu_data->smu7_data.dpm_table_start + offsetof(SMU75_Discrete_DpmTable, + UvdBootLevel); + mm_boot_level_offset /= 4; + mm_boot_level_offset *= 4; + mm_boot_level_value = cgs_read_ind_register(hwmgr->device, + CGS_IND_REG__SMC, mm_boot_level_offset); + mm_boot_level_value &= 0x00FFFFFF; + mm_boot_level_value |= smu_data->smc_state_table.UvdBootLevel << 24; + cgs_write_ind_register(hwmgr->device, + CGS_IND_REG__SMC, mm_boot_level_offset, mm_boot_level_value); + + if (!phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_UVDDPM) || + phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_StablePState)) + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_UVDDPM_SetEnabledMask, + (uint32_t)(1 << smu_data->smc_state_table.UvdBootLevel)); + return 0; +} + +static int vegam_update_vce_smc_table(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + uint32_t mm_boot_level_offset, mm_boot_level_value; + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_StablePState)) + smu_data->smc_state_table.VceBootLevel = + (uint8_t) (table_info->mm_dep_table->count - 1); + else + smu_data->smc_state_table.VceBootLevel = 0; + + mm_boot_level_offset = smu_data->smu7_data.dpm_table_start + + offsetof(SMU75_Discrete_DpmTable, VceBootLevel); + mm_boot_level_offset /= 4; + mm_boot_level_offset *= 4; + mm_boot_level_value = cgs_read_ind_register(hwmgr->device, + CGS_IND_REG__SMC, mm_boot_level_offset); + mm_boot_level_value &= 0xFF00FFFF; + mm_boot_level_value |= smu_data->smc_state_table.VceBootLevel << 16; + cgs_write_ind_register(hwmgr->device, + CGS_IND_REG__SMC, mm_boot_level_offset, mm_boot_level_value); + + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_StablePState)) + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_VCEDPM_SetEnabledMask, + (uint32_t)1 << smu_data->smc_state_table.VceBootLevel); + return 0; +} + +static int vegam_update_samu_smc_table(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + uint32_t mm_boot_level_offset, mm_boot_level_value; + + + smu_data->smc_state_table.SamuBootLevel = 0; + mm_boot_level_offset = smu_data->smu7_data.dpm_table_start + + offsetof(SMU75_Discrete_DpmTable, SamuBootLevel); + + mm_boot_level_offset /= 4; + mm_boot_level_offset *= 4; + mm_boot_level_value = cgs_read_ind_register(hwmgr->device, + CGS_IND_REG__SMC, mm_boot_level_offset); + mm_boot_level_value &= 0xFFFFFF00; + mm_boot_level_value |= smu_data->smc_state_table.SamuBootLevel << 0; + cgs_write_ind_register(hwmgr->device, + CGS_IND_REG__SMC, mm_boot_level_offset, mm_boot_level_value); + + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_StablePState)) + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_SAMUDPM_SetEnabledMask, + (uint32_t)(1 << smu_data->smc_state_table.SamuBootLevel)); + return 0; +} + + +static int vegam_update_bif_smc_table(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + struct phm_ppt_v1_pcie_table *pcie_table = table_info->pcie_table; + int max_entry, i; + + max_entry = (SMU75_MAX_LEVELS_LINK < pcie_table->count) ? + SMU75_MAX_LEVELS_LINK : + pcie_table->count; + /* Setup BIF_SCLK levels */ + for (i = 0; i < max_entry; i++) + smu_data->bif_sclk_table[i] = pcie_table->entries[i].pcie_sclk; + return 0; +} + +static int vegam_update_smc_table(struct pp_hwmgr *hwmgr, uint32_t type) +{ + switch (type) { + case SMU_UVD_TABLE: + vegam_update_uvd_smc_table(hwmgr); + break; + case SMU_VCE_TABLE: + vegam_update_vce_smc_table(hwmgr); + break; + case SMU_SAMU_TABLE: + vegam_update_samu_smc_table(hwmgr); + break; + case SMU_BIF_TABLE: + vegam_update_bif_smc_table(hwmgr); + break; + default: + break; + } + return 0; +} + +static void vegam_initialize_power_tune_defaults(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + + if (table_info && + table_info->cac_dtp_table->usPowerTuneDataSetID <= POWERTUNE_DEFAULT_SET_MAX && + table_info->cac_dtp_table->usPowerTuneDataSetID) + smu_data->power_tune_defaults = + &vegam_power_tune_data_set_array + [table_info->cac_dtp_table->usPowerTuneDataSetID - 1]; + else + smu_data->power_tune_defaults = &vegam_power_tune_data_set_array[0]; + +} + +static int vegam_populate_smc_mvdd_table(struct pp_hwmgr *hwmgr, + SMU75_Discrete_DpmTable *table) +{ + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + uint32_t count, level; + + if (SMU7_VOLTAGE_CONTROL_BY_GPIO == data->mvdd_control) { + count = data->mvdd_voltage_table.count; + if (count > SMU_MAX_SMIO_LEVELS) + count = SMU_MAX_SMIO_LEVELS; + for (level = 0; level < count; level++) { + table->SmioTable2.Pattern[level].Voltage = PP_HOST_TO_SMC_US( + data->mvdd_voltage_table.entries[count].value * VOLTAGE_SCALE); + /* Index into DpmTable.Smio. Drive bits from Smio entry to get this voltage level.*/ + table->SmioTable2.Pattern[level].Smio = + (uint8_t) level; + table->Smio[level] |= + data->mvdd_voltage_table.entries[level].smio_low; + } + table->SmioMask2 = data->mvdd_voltage_table.mask_low; + + table->MvddLevelCount = (uint32_t) PP_HOST_TO_SMC_UL(count); + } + + return 0; +} + +static int vegam_populate_smc_vddci_table(struct pp_hwmgr *hwmgr, + struct SMU75_Discrete_DpmTable *table) +{ + uint32_t count, level; + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + + count = data->vddci_voltage_table.count; + + if (SMU7_VOLTAGE_CONTROL_BY_GPIO == data->vddci_control) { + if (count > SMU_MAX_SMIO_LEVELS) + count = SMU_MAX_SMIO_LEVELS; + for (level = 0; level < count; ++level) { + table->SmioTable1.Pattern[level].Voltage = PP_HOST_TO_SMC_US( + data->vddci_voltage_table.entries[level].value * VOLTAGE_SCALE); + table->SmioTable1.Pattern[level].Smio = (uint8_t) level; + + table->Smio[level] |= data->vddci_voltage_table.entries[level].smio_low; + } + } + + table->SmioMask1 = data->vddci_voltage_table.mask_low; + + return 0; +} + +static int vegam_populate_cac_table(struct pp_hwmgr *hwmgr, + struct SMU75_Discrete_DpmTable *table) +{ + uint32_t count; + uint8_t index; + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + struct phm_ppt_v1_voltage_lookup_table *lookup_table = + table_info->vddc_lookup_table; + /* tables is already swapped, so in order to use the value from it, + * we need to swap it back. + * We are populating vddc CAC data to BapmVddc table + * in split and merged mode + */ + for (count = 0; count < lookup_table->count; count++) { + index = phm_get_voltage_index(lookup_table, + data->vddc_voltage_table.entries[count].value); + table->BapmVddcVidLoSidd[count] = + convert_to_vid(lookup_table->entries[index].us_cac_low); + table->BapmVddcVidHiSidd[count] = + convert_to_vid(lookup_table->entries[index].us_cac_mid); + table->BapmVddcVidHiSidd2[count] = + convert_to_vid(lookup_table->entries[index].us_cac_high); + } + + return 0; +} + +static int vegam_populate_smc_voltage_tables(struct pp_hwmgr *hwmgr, + struct SMU75_Discrete_DpmTable *table) +{ + vegam_populate_smc_vddci_table(hwmgr, table); + vegam_populate_smc_mvdd_table(hwmgr, table); + vegam_populate_cac_table(hwmgr, table); + + return 0; +} + +static int vegam_populate_ulv_level(struct pp_hwmgr *hwmgr, + struct SMU75_Discrete_Ulv *state) +{ + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + + state->CcPwrDynRm = 0; + state->CcPwrDynRm1 = 0; + + state->VddcOffset = (uint16_t) table_info->us_ulv_voltage_offset; + state->VddcOffsetVid = (uint8_t)(table_info->us_ulv_voltage_offset * + VOLTAGE_VID_OFFSET_SCALE2 / VOLTAGE_VID_OFFSET_SCALE1); + + state->VddcPhase = data->vddc_phase_shed_control ^ 0x3; + + CONVERT_FROM_HOST_TO_SMC_UL(state->CcPwrDynRm); + CONVERT_FROM_HOST_TO_SMC_UL(state->CcPwrDynRm1); + CONVERT_FROM_HOST_TO_SMC_US(state->VddcOffset); + + return 0; +} + +static int vegam_populate_ulv_state(struct pp_hwmgr *hwmgr, + struct SMU75_Discrete_DpmTable *table) +{ + return vegam_populate_ulv_level(hwmgr, &table->Ulv); +} + +static int vegam_populate_smc_link_level(struct pp_hwmgr *hwmgr, + struct SMU75_Discrete_DpmTable *table) +{ + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + struct vegam_smumgr *smu_data = + (struct vegam_smumgr *)(hwmgr->smu_backend); + struct smu7_dpm_table *dpm_table = &data->dpm_table; + int i; + + /* Index (dpm_table->pcie_speed_table.count) + * is reserved for PCIE boot level. */ + for (i = 0; i <= dpm_table->pcie_speed_table.count; i++) { + table->LinkLevel[i].PcieGenSpeed = + (uint8_t)dpm_table->pcie_speed_table.dpm_levels[i].value; + table->LinkLevel[i].PcieLaneCount = (uint8_t)encode_pcie_lane_width( + dpm_table->pcie_speed_table.dpm_levels[i].param1); + table->LinkLevel[i].EnabledForActivity = 1; + table->LinkLevel[i].SPC = (uint8_t)(data->pcie_spc_cap & 0xff); + table->LinkLevel[i].DownThreshold = PP_HOST_TO_SMC_UL(5); + table->LinkLevel[i].UpThreshold = PP_HOST_TO_SMC_UL(30); + } + + smu_data->smc_state_table.LinkLevelCount = + (uint8_t)dpm_table->pcie_speed_table.count; + +/* To Do move to hwmgr */ + data->dpm_level_enable_mask.pcie_dpm_enable_mask = + phm_get_dpm_level_enable_mask_value(&dpm_table->pcie_speed_table); + + return 0; +} + +static int vegam_get_dependency_volt_by_clk(struct pp_hwmgr *hwmgr, + struct phm_ppt_v1_clock_voltage_dependency_table *dep_table, + uint32_t clock, SMU_VoltageLevel *voltage, uint32_t *mvdd) +{ + uint32_t i; + uint16_t vddci; + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + + *voltage = *mvdd = 0; + + /* clock - voltage dependency table is empty table */ + if (dep_table->count == 0) + return -EINVAL; + + for (i = 0; i < dep_table->count; i++) { + /* find first sclk bigger than request */ + if (dep_table->entries[i].clk >= clock) { + *voltage |= (dep_table->entries[i].vddc * + VOLTAGE_SCALE) << VDDC_SHIFT; + if (SMU7_VOLTAGE_CONTROL_NONE == data->vddci_control) + *voltage |= (data->vbios_boot_state.vddci_bootup_value * + VOLTAGE_SCALE) << VDDCI_SHIFT; + else if (dep_table->entries[i].vddci) + *voltage |= (dep_table->entries[i].vddci * + VOLTAGE_SCALE) << VDDCI_SHIFT; + else { + vddci = phm_find_closest_vddci(&(data->vddci_voltage_table), + (dep_table->entries[i].vddc - + (uint16_t)VDDC_VDDCI_DELTA)); + *voltage |= (vddci * VOLTAGE_SCALE) << VDDCI_SHIFT; + } + + if (SMU7_VOLTAGE_CONTROL_NONE == data->mvdd_control) + *mvdd = data->vbios_boot_state.mvdd_bootup_value * + VOLTAGE_SCALE; + else if (dep_table->entries[i].mvdd) + *mvdd = (uint32_t) dep_table->entries[i].mvdd * + VOLTAGE_SCALE; + + *voltage |= 1 << PHASES_SHIFT; + return 0; + } + } + + /* sclk is bigger than max sclk in the dependence table */ + *voltage |= (dep_table->entries[i - 1].vddc * VOLTAGE_SCALE) << VDDC_SHIFT; + vddci = phm_find_closest_vddci(&(data->vddci_voltage_table), + (dep_table->entries[i - 1].vddc - + (uint16_t)VDDC_VDDCI_DELTA)); + + if (SMU7_VOLTAGE_CONTROL_NONE == data->vddci_control) + *voltage |= (data->vbios_boot_state.vddci_bootup_value * + VOLTAGE_SCALE) << VDDCI_SHIFT; + else if (dep_table->entries[i - 1].vddci) + *voltage |= (dep_table->entries[i - 1].vddci * + VOLTAGE_SCALE) << VDDC_SHIFT; + else + *voltage |= (vddci * VOLTAGE_SCALE) << VDDCI_SHIFT; + + if (SMU7_VOLTAGE_CONTROL_NONE == data->mvdd_control) + *mvdd = data->vbios_boot_state.mvdd_bootup_value * VOLTAGE_SCALE; + else if (dep_table->entries[i].mvdd) + *mvdd = (uint32_t) dep_table->entries[i - 1].mvdd * VOLTAGE_SCALE; + + return 0; +} + +static void vegam_get_sclk_range_table(struct pp_hwmgr *hwmgr, + SMU75_Discrete_DpmTable *table) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + uint32_t i, ref_clk; + + struct pp_atom_ctrl_sclk_range_table range_table_from_vbios = { { {0} } }; + + ref_clk = amdgpu_asic_get_xclk((struct amdgpu_device *)hwmgr->adev); + + if (0 == atomctrl_get_smc_sclk_range_table(hwmgr, &range_table_from_vbios)) { + for (i = 0; i < NUM_SCLK_RANGE; i++) { + table->SclkFcwRangeTable[i].vco_setting = + range_table_from_vbios.entry[i].ucVco_setting; + table->SclkFcwRangeTable[i].postdiv = + range_table_from_vbios.entry[i].ucPostdiv; + table->SclkFcwRangeTable[i].fcw_pcc = + range_table_from_vbios.entry[i].usFcw_pcc; + + table->SclkFcwRangeTable[i].fcw_trans_upper = + range_table_from_vbios.entry[i].usFcw_trans_upper; + table->SclkFcwRangeTable[i].fcw_trans_lower = + range_table_from_vbios.entry[i].usRcw_trans_lower; + + CONVERT_FROM_HOST_TO_SMC_US(table->SclkFcwRangeTable[i].fcw_pcc); + CONVERT_FROM_HOST_TO_SMC_US(table->SclkFcwRangeTable[i].fcw_trans_upper); + CONVERT_FROM_HOST_TO_SMC_US(table->SclkFcwRangeTable[i].fcw_trans_lower); + } + return; + } + + for (i = 0; i < NUM_SCLK_RANGE; i++) { + smu_data->range_table[i].trans_lower_frequency = + (ref_clk * Range_Table[i].fcw_trans_lower) >> Range_Table[i].postdiv; + smu_data->range_table[i].trans_upper_frequency = + (ref_clk * Range_Table[i].fcw_trans_upper) >> Range_Table[i].postdiv; + + table->SclkFcwRangeTable[i].vco_setting = Range_Table[i].vco_setting; + table->SclkFcwRangeTable[i].postdiv = Range_Table[i].postdiv; + table->SclkFcwRangeTable[i].fcw_pcc = Range_Table[i].fcw_pcc; + + table->SclkFcwRangeTable[i].fcw_trans_upper = Range_Table[i].fcw_trans_upper; + table->SclkFcwRangeTable[i].fcw_trans_lower = Range_Table[i].fcw_trans_lower; + + CONVERT_FROM_HOST_TO_SMC_US(table->SclkFcwRangeTable[i].fcw_pcc); + CONVERT_FROM_HOST_TO_SMC_US(table->SclkFcwRangeTable[i].fcw_trans_upper); + CONVERT_FROM_HOST_TO_SMC_US(table->SclkFcwRangeTable[i].fcw_trans_lower); + } +} + +static int vegam_calculate_sclk_params(struct pp_hwmgr *hwmgr, + uint32_t clock, SMU_SclkSetting *sclk_setting) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + const SMU75_Discrete_DpmTable *table = &(smu_data->smc_state_table); + struct pp_atomctrl_clock_dividers_ai dividers; + uint32_t ref_clock; + uint32_t pcc_target_percent, pcc_target_freq, ss_target_percent, ss_target_freq; + uint8_t i; + int result; + uint64_t temp; + + sclk_setting->SclkFrequency = clock; + /* get the engine clock dividers for this clock value */ + result = atomctrl_get_engine_pll_dividers_ai(hwmgr, clock, ÷rs); + if (result == 0) { + sclk_setting->Fcw_int = dividers.usSclk_fcw_int; + sclk_setting->Fcw_frac = dividers.usSclk_fcw_frac; + sclk_setting->Pcc_fcw_int = dividers.usPcc_fcw_int; + sclk_setting->PllRange = dividers.ucSclkPllRange; + sclk_setting->Sclk_slew_rate = 0x400; + sclk_setting->Pcc_up_slew_rate = dividers.usPcc_fcw_slew_frac; + sclk_setting->Pcc_down_slew_rate = 0xffff; + sclk_setting->SSc_En = dividers.ucSscEnable; + sclk_setting->Fcw1_int = dividers.usSsc_fcw1_int; + sclk_setting->Fcw1_frac = dividers.usSsc_fcw1_frac; + sclk_setting->Sclk_ss_slew_rate = dividers.usSsc_fcw_slew_frac; + return result; + } + + ref_clock = amdgpu_asic_get_xclk((struct amdgpu_device *)hwmgr->adev); + + for (i = 0; i < NUM_SCLK_RANGE; i++) { + if (clock > smu_data->range_table[i].trans_lower_frequency + && clock <= smu_data->range_table[i].trans_upper_frequency) { + sclk_setting->PllRange = i; + break; + } + } + + sclk_setting->Fcw_int = (uint16_t) + ((clock << table->SclkFcwRangeTable[sclk_setting->PllRange].postdiv) / + ref_clock); + temp = clock << table->SclkFcwRangeTable[sclk_setting->PllRange].postdiv; + temp <<= 0x10; + do_div(temp, ref_clock); + sclk_setting->Fcw_frac = temp & 0xffff; + + pcc_target_percent = 10; /* Hardcode 10% for now. */ + pcc_target_freq = clock - (clock * pcc_target_percent / 100); + sclk_setting->Pcc_fcw_int = (uint16_t) + ((pcc_target_freq << table->SclkFcwRangeTable[sclk_setting->PllRange].postdiv) / + ref_clock); + + ss_target_percent = 2; /* Hardcode 2% for now. */ + sclk_setting->SSc_En = 0; + if (ss_target_percent) { + sclk_setting->SSc_En = 1; + ss_target_freq = clock - (clock * ss_target_percent / 100); + sclk_setting->Fcw1_int = (uint16_t) + ((ss_target_freq << table->SclkFcwRangeTable[sclk_setting->PllRange].postdiv) / + ref_clock); + temp = ss_target_freq << table->SclkFcwRangeTable[sclk_setting->PllRange].postdiv; + temp <<= 0x10; + do_div(temp, ref_clock); + sclk_setting->Fcw1_frac = temp & 0xffff; + } + + return 0; +} + +static uint8_t vegam_get_sleep_divider_id_from_clock(uint32_t clock, + uint32_t clock_insr) +{ + uint8_t i; + uint32_t temp; + uint32_t min = max(clock_insr, (uint32_t)SMU7_MINIMUM_ENGINE_CLOCK); + + PP_ASSERT_WITH_CODE((clock >= min), + "Engine clock can't satisfy stutter requirement!", + return 0); + for (i = 31; ; i--) { + temp = clock / (i + 1); + + if (temp >= min || i == 0) + break; + } + return i; +} + +static int vegam_populate_single_graphic_level(struct pp_hwmgr *hwmgr, + uint32_t clock, struct SMU75_Discrete_GraphicsLevel *level) +{ + int result; + /* PP_Clocks minClocks; */ + uint32_t mvdd; + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + SMU_SclkSetting curr_sclk_setting = { 0 }; + + result = vegam_calculate_sclk_params(hwmgr, clock, &curr_sclk_setting); + + /* populate graphics levels */ + result = vegam_get_dependency_volt_by_clk(hwmgr, + table_info->vdd_dep_on_sclk, clock, + &level->MinVoltage, &mvdd); + + PP_ASSERT_WITH_CODE((0 == result), + "can not find VDDC voltage value for " + "VDDC engine clock dependency table", + return result); + level->ActivityLevel = (uint16_t)(SclkDPMTuning_VEGAM >> DPMTuning_Activity_Shift); + + level->CcPwrDynRm = 0; + level->CcPwrDynRm1 = 0; + level->EnabledForActivity = 0; + level->EnabledForThrottle = 1; + level->VoltageDownHyst = 0; + level->PowerThrottle = 0; + data->display_timing.min_clock_in_sr = hwmgr->display_config->min_core_set_clock_in_sr; + + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_SclkDeepSleep)) + level->DeepSleepDivId = vegam_get_sleep_divider_id_from_clock(clock, + hwmgr->display_config->min_core_set_clock_in_sr); + + level->SclkSetting = curr_sclk_setting; + + CONVERT_FROM_HOST_TO_SMC_UL(level->MinVoltage); + CONVERT_FROM_HOST_TO_SMC_UL(level->CcPwrDynRm); + CONVERT_FROM_HOST_TO_SMC_UL(level->CcPwrDynRm1); + CONVERT_FROM_HOST_TO_SMC_US(level->ActivityLevel); + CONVERT_FROM_HOST_TO_SMC_UL(level->SclkSetting.SclkFrequency); + CONVERT_FROM_HOST_TO_SMC_US(level->SclkSetting.Fcw_int); + CONVERT_FROM_HOST_TO_SMC_US(level->SclkSetting.Fcw_frac); + CONVERT_FROM_HOST_TO_SMC_US(level->SclkSetting.Pcc_fcw_int); + CONVERT_FROM_HOST_TO_SMC_US(level->SclkSetting.Sclk_slew_rate); + CONVERT_FROM_HOST_TO_SMC_US(level->SclkSetting.Pcc_up_slew_rate); + CONVERT_FROM_HOST_TO_SMC_US(level->SclkSetting.Pcc_down_slew_rate); + CONVERT_FROM_HOST_TO_SMC_US(level->SclkSetting.Fcw1_int); + CONVERT_FROM_HOST_TO_SMC_US(level->SclkSetting.Fcw1_frac); + CONVERT_FROM_HOST_TO_SMC_US(level->SclkSetting.Sclk_ss_slew_rate); + return 0; +} + +static int vegam_populate_all_graphic_levels(struct pp_hwmgr *hwmgr) +{ + struct smu7_hwmgr *hw_data = (struct smu7_hwmgr *)(hwmgr->backend); + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + struct smu7_dpm_table *dpm_table = &hw_data->dpm_table; + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + struct phm_ppt_v1_pcie_table *pcie_table = table_info->pcie_table; + uint8_t pcie_entry_cnt = (uint8_t) hw_data->dpm_table.pcie_speed_table.count; + int result = 0; + uint32_t array = smu_data->smu7_data.dpm_table_start + + offsetof(SMU75_Discrete_DpmTable, GraphicsLevel); + uint32_t array_size = sizeof(struct SMU75_Discrete_GraphicsLevel) * + SMU75_MAX_LEVELS_GRAPHICS; + struct SMU75_Discrete_GraphicsLevel *levels = + smu_data->smc_state_table.GraphicsLevel; + uint32_t i, max_entry; + uint8_t hightest_pcie_level_enabled = 0, + lowest_pcie_level_enabled = 0, + mid_pcie_level_enabled = 0, + count = 0; + + vegam_get_sclk_range_table(hwmgr, &(smu_data->smc_state_table)); + + for (i = 0; i < dpm_table->sclk_table.count; i++) { + + result = vegam_populate_single_graphic_level(hwmgr, + dpm_table->sclk_table.dpm_levels[i].value, + &(smu_data->smc_state_table.GraphicsLevel[i])); + if (result) + return result; + + levels[i].UpHyst = (uint8_t) + (SclkDPMTuning_VEGAM >> DPMTuning_Uphyst_Shift); + levels[i].DownHyst = (uint8_t) + (SclkDPMTuning_VEGAM >> DPMTuning_Downhyst_Shift); + /* Making sure only DPM level 0-1 have Deep Sleep Div ID populated. */ + if (i > 1) + levels[i].DeepSleepDivId = 0; + } + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_SPLLShutdownSupport)) + smu_data->smc_state_table.GraphicsLevel[0].SclkSetting.SSc_En = 0; + + smu_data->smc_state_table.GraphicsDpmLevelCount = + (uint8_t)dpm_table->sclk_table.count; + hw_data->dpm_level_enable_mask.sclk_dpm_enable_mask = + phm_get_dpm_level_enable_mask_value(&dpm_table->sclk_table); + + for (i = 0; i < dpm_table->sclk_table.count; i++) + levels[i].EnabledForActivity = + (hw_data->dpm_level_enable_mask.sclk_dpm_enable_mask >> i) & 0x1; + + if (pcie_table != NULL) { + PP_ASSERT_WITH_CODE((1 <= pcie_entry_cnt), + "There must be 1 or more PCIE levels defined in PPTable.", + return -EINVAL); + max_entry = pcie_entry_cnt - 1; + for (i = 0; i < dpm_table->sclk_table.count; i++) + levels[i].pcieDpmLevel = + (uint8_t) ((i < max_entry) ? i : max_entry); + } else { + while (hw_data->dpm_level_enable_mask.pcie_dpm_enable_mask && + ((hw_data->dpm_level_enable_mask.pcie_dpm_enable_mask & + (1 << (hightest_pcie_level_enabled + 1))) != 0)) + hightest_pcie_level_enabled++; + + while (hw_data->dpm_level_enable_mask.pcie_dpm_enable_mask && + ((hw_data->dpm_level_enable_mask.pcie_dpm_enable_mask & + (1 << lowest_pcie_level_enabled)) == 0)) + lowest_pcie_level_enabled++; + + while ((count < hightest_pcie_level_enabled) && + ((hw_data->dpm_level_enable_mask.pcie_dpm_enable_mask & + (1 << (lowest_pcie_level_enabled + 1 + count))) == 0)) + count++; + + mid_pcie_level_enabled = (lowest_pcie_level_enabled + 1 + count) < + hightest_pcie_level_enabled ? + (lowest_pcie_level_enabled + 1 + count) : + hightest_pcie_level_enabled; + + /* set pcieDpmLevel to hightest_pcie_level_enabled */ + for (i = 2; i < dpm_table->sclk_table.count; i++) + levels[i].pcieDpmLevel = hightest_pcie_level_enabled; + + /* set pcieDpmLevel to lowest_pcie_level_enabled */ + levels[0].pcieDpmLevel = lowest_pcie_level_enabled; + + /* set pcieDpmLevel to mid_pcie_level_enabled */ + levels[1].pcieDpmLevel = mid_pcie_level_enabled; + } + /* level count will send to smc once at init smc table and never change */ + result = smu7_copy_bytes_to_smc(hwmgr, array, (uint8_t *)levels, + (uint32_t)array_size, SMC_RAM_END); + + return result; +} + +static int vegam_calculate_mclk_params(struct pp_hwmgr *hwmgr, + uint32_t clock, struct SMU75_Discrete_MemoryLevel *mem_level) +{ + struct pp_atomctrl_memory_clock_param_ai mpll_param; + + PP_ASSERT_WITH_CODE(!atomctrl_get_memory_pll_dividers_ai(hwmgr, + clock, &mpll_param), + "Failed to retrieve memory pll parameter.", + return -EINVAL); + + mem_level->MclkFrequency = (uint32_t)mpll_param.ulClock; + mem_level->Fcw_int = (uint16_t)mpll_param.ulMclk_fcw_int; + mem_level->Fcw_frac = (uint16_t)mpll_param.ulMclk_fcw_frac; + mem_level->Postdiv = (uint8_t)mpll_param.ulPostDiv; + + return 0; +} + +static int vegam_populate_single_memory_level(struct pp_hwmgr *hwmgr, + uint32_t clock, struct SMU75_Discrete_MemoryLevel *mem_level) +{ + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + int result = 0; + uint32_t mclk_stutter_mode_threshold = 60000; + + + if (table_info->vdd_dep_on_mclk) { + result = vegam_get_dependency_volt_by_clk(hwmgr, + table_info->vdd_dep_on_mclk, clock, + &mem_level->MinVoltage, &mem_level->MinMvdd); + PP_ASSERT_WITH_CODE(!result, + "can not find MinVddc voltage value from memory " + "VDDC voltage dependency table", return result); + } + + result = vegam_calculate_mclk_params(hwmgr, clock, mem_level); + PP_ASSERT_WITH_CODE(!result, + "Failed to calculate mclk params.", + return -EINVAL); + + mem_level->EnabledForThrottle = 1; + mem_level->EnabledForActivity = 0; + mem_level->VoltageDownHyst = 0; + mem_level->ActivityLevel = (uint16_t) + (MemoryDPMTuning_VEGAM >> DPMTuning_Activity_Shift); + mem_level->StutterEnable = false; + mem_level->DisplayWatermark = PPSMC_DISPLAY_WATERMARK_LOW; + + data->display_timing.num_existing_displays = hwmgr->display_config->num_display; + + if (mclk_stutter_mode_threshold && + (clock <= mclk_stutter_mode_threshold) && + (PHM_READ_FIELD(hwmgr->device, DPG_PIPE_STUTTER_CONTROL, + STUTTER_ENABLE) & 0x1)) + mem_level->StutterEnable = true; + + if (!result) { + CONVERT_FROM_HOST_TO_SMC_UL(mem_level->MinMvdd); + CONVERT_FROM_HOST_TO_SMC_UL(mem_level->MclkFrequency); + CONVERT_FROM_HOST_TO_SMC_US(mem_level->Fcw_int); + CONVERT_FROM_HOST_TO_SMC_US(mem_level->Fcw_frac); + CONVERT_FROM_HOST_TO_SMC_US(mem_level->ActivityLevel); + CONVERT_FROM_HOST_TO_SMC_UL(mem_level->MinVoltage); + } + + return result; +} + +static int vegam_populate_all_memory_levels(struct pp_hwmgr *hwmgr) +{ + struct smu7_hwmgr *hw_data = (struct smu7_hwmgr *)(hwmgr->backend); + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + struct smu7_dpm_table *dpm_table = &hw_data->dpm_table; + int result; + /* populate MCLK dpm table to SMU7 */ + uint32_t array = smu_data->smu7_data.dpm_table_start + + offsetof(SMU75_Discrete_DpmTable, MemoryLevel); + uint32_t array_size = sizeof(SMU75_Discrete_MemoryLevel) * + SMU75_MAX_LEVELS_MEMORY; + struct SMU75_Discrete_MemoryLevel *levels = + smu_data->smc_state_table.MemoryLevel; + uint32_t i; + + for (i = 0; i < dpm_table->mclk_table.count; i++) { + PP_ASSERT_WITH_CODE((0 != dpm_table->mclk_table.dpm_levels[i].value), + "can not populate memory level as memory clock is zero", + return -EINVAL); + result = vegam_populate_single_memory_level(hwmgr, + dpm_table->mclk_table.dpm_levels[i].value, + &levels[i]); + + if (result) + return result; + + levels[i].UpHyst = (uint8_t) + (MemoryDPMTuning_VEGAM >> DPMTuning_Uphyst_Shift); + levels[i].DownHyst = (uint8_t) + (MemoryDPMTuning_VEGAM >> DPMTuning_Downhyst_Shift); + } + + smu_data->smc_state_table.MemoryDpmLevelCount = + (uint8_t)dpm_table->mclk_table.count; + hw_data->dpm_level_enable_mask.mclk_dpm_enable_mask = + phm_get_dpm_level_enable_mask_value(&dpm_table->mclk_table); + + for (i = 0; i < dpm_table->mclk_table.count; i++) + levels[i].EnabledForActivity = + (hw_data->dpm_level_enable_mask.mclk_dpm_enable_mask >> i) & 0x1; + + levels[dpm_table->mclk_table.count - 1].DisplayWatermark = + PPSMC_DISPLAY_WATERMARK_HIGH; + + /* level count will send to smc once at init smc table and never change */ + result = smu7_copy_bytes_to_smc(hwmgr, array, (uint8_t *)levels, + (uint32_t)array_size, SMC_RAM_END); + + return result; +} + +static int vegam_populate_mvdd_value(struct pp_hwmgr *hwmgr, + uint32_t mclk, SMIO_Pattern *smio_pat) +{ + const struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + uint32_t i = 0; + + if (SMU7_VOLTAGE_CONTROL_NONE != data->mvdd_control) { + /* find mvdd value which clock is more than request */ + for (i = 0; i < table_info->vdd_dep_on_mclk->count; i++) { + if (mclk <= table_info->vdd_dep_on_mclk->entries[i].clk) { + smio_pat->Voltage = data->mvdd_voltage_table.entries[i].value; + break; + } + } + PP_ASSERT_WITH_CODE(i < table_info->vdd_dep_on_mclk->count, + "MVDD Voltage is outside the supported range.", + return -EINVAL); + } else + return -EINVAL; + + return 0; +} + +static int vegam_populate_smc_acpi_level(struct pp_hwmgr *hwmgr, + SMU75_Discrete_DpmTable *table) +{ + int result = 0; + uint32_t sclk_frequency; + const struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + SMIO_Pattern vol_level; + uint32_t mvdd; + uint16_t us_mvdd; + + table->ACPILevel.Flags &= ~PPSMC_SWSTATE_FLAG_DC; + + /* Get MinVoltage and Frequency from DPM0, + * already converted to SMC_UL */ + sclk_frequency = data->vbios_boot_state.sclk_bootup_value; + result = vegam_get_dependency_volt_by_clk(hwmgr, + table_info->vdd_dep_on_sclk, + sclk_frequency, + &table->ACPILevel.MinVoltage, &mvdd); + PP_ASSERT_WITH_CODE(!result, + "Cannot find ACPI VDDC voltage value " + "in Clock Dependency Table", + ); + + result = vegam_calculate_sclk_params(hwmgr, sclk_frequency, + &(table->ACPILevel.SclkSetting)); + PP_ASSERT_WITH_CODE(!result, + "Error retrieving Engine Clock dividers from VBIOS.", + return result); + + table->ACPILevel.DeepSleepDivId = 0; + table->ACPILevel.CcPwrDynRm = 0; + table->ACPILevel.CcPwrDynRm1 = 0; + + CONVERT_FROM_HOST_TO_SMC_UL(table->ACPILevel.Flags); + CONVERT_FROM_HOST_TO_SMC_UL(table->ACPILevel.MinVoltage); + CONVERT_FROM_HOST_TO_SMC_UL(table->ACPILevel.CcPwrDynRm); + CONVERT_FROM_HOST_TO_SMC_UL(table->ACPILevel.CcPwrDynRm1); + + CONVERT_FROM_HOST_TO_SMC_UL(table->ACPILevel.SclkSetting.SclkFrequency); + CONVERT_FROM_HOST_TO_SMC_US(table->ACPILevel.SclkSetting.Fcw_int); + CONVERT_FROM_HOST_TO_SMC_US(table->ACPILevel.SclkSetting.Fcw_frac); + CONVERT_FROM_HOST_TO_SMC_US(table->ACPILevel.SclkSetting.Pcc_fcw_int); + CONVERT_FROM_HOST_TO_SMC_US(table->ACPILevel.SclkSetting.Sclk_slew_rate); + CONVERT_FROM_HOST_TO_SMC_US(table->ACPILevel.SclkSetting.Pcc_up_slew_rate); + CONVERT_FROM_HOST_TO_SMC_US(table->ACPILevel.SclkSetting.Pcc_down_slew_rate); + CONVERT_FROM_HOST_TO_SMC_US(table->ACPILevel.SclkSetting.Fcw1_int); + CONVERT_FROM_HOST_TO_SMC_US(table->ACPILevel.SclkSetting.Fcw1_frac); + CONVERT_FROM_HOST_TO_SMC_US(table->ACPILevel.SclkSetting.Sclk_ss_slew_rate); + + + /* Get MinVoltage and Frequency from DPM0, already converted to SMC_UL */ + table->MemoryACPILevel.MclkFrequency = data->vbios_boot_state.mclk_bootup_value; + result = vegam_get_dependency_volt_by_clk(hwmgr, + table_info->vdd_dep_on_mclk, + table->MemoryACPILevel.MclkFrequency, + &table->MemoryACPILevel.MinVoltage, &mvdd); + PP_ASSERT_WITH_CODE((0 == result), + "Cannot find ACPI VDDCI voltage value " + "in Clock Dependency Table", + ); + + us_mvdd = 0; + if ((SMU7_VOLTAGE_CONTROL_NONE == data->mvdd_control) || + (data->mclk_dpm_key_disabled)) + us_mvdd = data->vbios_boot_state.mvdd_bootup_value; + else { + if (!vegam_populate_mvdd_value(hwmgr, + data->dpm_table.mclk_table.dpm_levels[0].value, + &vol_level)) + us_mvdd = vol_level.Voltage; + } + + if (!vegam_populate_mvdd_value(hwmgr, 0, &vol_level)) + table->MemoryACPILevel.MinMvdd = PP_HOST_TO_SMC_UL(vol_level.Voltage); + else + table->MemoryACPILevel.MinMvdd = 0; + + table->MemoryACPILevel.StutterEnable = false; + + table->MemoryACPILevel.EnabledForThrottle = 0; + table->MemoryACPILevel.EnabledForActivity = 0; + table->MemoryACPILevel.UpHyst = 0; + table->MemoryACPILevel.DownHyst = 100; + table->MemoryACPILevel.VoltageDownHyst = 0; + table->MemoryACPILevel.ActivityLevel = + PP_HOST_TO_SMC_US(data->current_profile_setting.mclk_activity); + + CONVERT_FROM_HOST_TO_SMC_UL(table->MemoryACPILevel.MclkFrequency); + CONVERT_FROM_HOST_TO_SMC_UL(table->MemoryACPILevel.MinVoltage); + + return result; +} + +static int vegam_populate_smc_vce_level(struct pp_hwmgr *hwmgr, + SMU75_Discrete_DpmTable *table) +{ + int result = -EINVAL; + uint8_t count; + struct pp_atomctrl_clock_dividers_vi dividers; + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + struct phm_ppt_v1_mm_clock_voltage_dependency_table *mm_table = + table_info->mm_dep_table; + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + uint32_t vddci; + + table->VceLevelCount = (uint8_t)(mm_table->count); + table->VceBootLevel = 0; + + for (count = 0; count < table->VceLevelCount; count++) { + table->VceLevel[count].Frequency = mm_table->entries[count].eclk; + table->VceLevel[count].MinVoltage = 0; + table->VceLevel[count].MinVoltage |= + (mm_table->entries[count].vddc * VOLTAGE_SCALE) << VDDC_SHIFT; + + if (SMU7_VOLTAGE_CONTROL_BY_GPIO == data->vddci_control) + vddci = (uint32_t)phm_find_closest_vddci(&(data->vddci_voltage_table), + mm_table->entries[count].vddc - VDDC_VDDCI_DELTA); + else if (SMU7_VOLTAGE_CONTROL_BY_SVID2 == data->vddci_control) + vddci = mm_table->entries[count].vddc - VDDC_VDDCI_DELTA; + else + vddci = (data->vbios_boot_state.vddci_bootup_value * VOLTAGE_SCALE) << VDDCI_SHIFT; + + + table->VceLevel[count].MinVoltage |= + (vddci * VOLTAGE_SCALE) << VDDCI_SHIFT; + table->VceLevel[count].MinVoltage |= 1 << PHASES_SHIFT; + + /*retrieve divider value for VBIOS */ + result = atomctrl_get_dfs_pll_dividers_vi(hwmgr, + table->VceLevel[count].Frequency, ÷rs); + PP_ASSERT_WITH_CODE((0 == result), + "can not find divide id for VCE engine clock", + return result); + + table->VceLevel[count].Divider = (uint8_t)dividers.pll_post_divider; + + CONVERT_FROM_HOST_TO_SMC_UL(table->VceLevel[count].Frequency); + CONVERT_FROM_HOST_TO_SMC_UL(table->VceLevel[count].MinVoltage); + } + return result; +} + +static int vegam_populate_smc_samu_level(struct pp_hwmgr *hwmgr, + SMU75_Discrete_DpmTable *table) +{ + int result = -EINVAL; + uint8_t count; + struct pp_atomctrl_clock_dividers_vi dividers; + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + struct phm_ppt_v1_mm_clock_voltage_dependency_table *mm_table = + table_info->mm_dep_table; + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + uint32_t vddci; + + table->SamuBootLevel = 0; + table->SamuLevelCount = (uint8_t)(mm_table->count); + + for (count = 0; count < table->SamuLevelCount; count++) { + /* not sure whether we need evclk or not */ + table->SamuLevel[count].MinVoltage = 0; + table->SamuLevel[count].Frequency = mm_table->entries[count].samclock; + table->SamuLevel[count].MinVoltage |= (mm_table->entries[count].vddc * + VOLTAGE_SCALE) << VDDC_SHIFT; + + if (SMU7_VOLTAGE_CONTROL_BY_GPIO == data->vddci_control) + vddci = (uint32_t)phm_find_closest_vddci(&(data->vddci_voltage_table), + mm_table->entries[count].vddc - VDDC_VDDCI_DELTA); + else if (SMU7_VOLTAGE_CONTROL_BY_SVID2 == data->vddci_control) + vddci = mm_table->entries[count].vddc - VDDC_VDDCI_DELTA; + else + vddci = (data->vbios_boot_state.vddci_bootup_value * VOLTAGE_SCALE) << VDDCI_SHIFT; + + table->SamuLevel[count].MinVoltage |= (vddci * VOLTAGE_SCALE) << VDDCI_SHIFT; + table->SamuLevel[count].MinVoltage |= 1 << PHASES_SHIFT; + + /* retrieve divider value for VBIOS */ + result = atomctrl_get_dfs_pll_dividers_vi(hwmgr, + table->SamuLevel[count].Frequency, ÷rs); + PP_ASSERT_WITH_CODE((0 == result), + "can not find divide id for samu clock", return result); + + table->SamuLevel[count].Divider = (uint8_t)dividers.pll_post_divider; + + CONVERT_FROM_HOST_TO_SMC_UL(table->SamuLevel[count].Frequency); + CONVERT_FROM_HOST_TO_SMC_UL(table->SamuLevel[count].MinVoltage); + } + return result; +} + +static int vegam_populate_memory_timing_parameters(struct pp_hwmgr *hwmgr, + int32_t eng_clock, int32_t mem_clock, + SMU75_Discrete_MCArbDramTimingTableEntry *arb_regs) +{ + uint32_t dram_timing; + uint32_t dram_timing2; + uint32_t burst_time; + uint32_t rfsh_rate; + uint32_t misc3; + + int result; + + result = atomctrl_set_engine_dram_timings_rv770(hwmgr, + eng_clock, mem_clock); + PP_ASSERT_WITH_CODE(result == 0, + "Error calling VBIOS to set DRAM_TIMING.", + return result); + + dram_timing = cgs_read_register(hwmgr->device, mmMC_ARB_DRAM_TIMING); + dram_timing2 = cgs_read_register(hwmgr->device, mmMC_ARB_DRAM_TIMING2); + burst_time = cgs_read_register(hwmgr->device, mmMC_ARB_BURST_TIME); + rfsh_rate = cgs_read_register(hwmgr->device, mmMC_ARB_RFSH_RATE); + misc3 = cgs_read_register(hwmgr->device, mmMC_ARB_MISC3); + + arb_regs->McArbDramTiming = PP_HOST_TO_SMC_UL(dram_timing); + arb_regs->McArbDramTiming2 = PP_HOST_TO_SMC_UL(dram_timing2); + arb_regs->McArbBurstTime = PP_HOST_TO_SMC_UL(burst_time); + arb_regs->McArbRfshRate = PP_HOST_TO_SMC_UL(rfsh_rate); + arb_regs->McArbMisc3 = PP_HOST_TO_SMC_UL(misc3); + + return 0; +} + +static int vegam_program_memory_timing_parameters(struct pp_hwmgr *hwmgr) +{ + struct smu7_hwmgr *hw_data = (struct smu7_hwmgr *)(hwmgr->backend); + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + struct SMU75_Discrete_MCArbDramTimingTable arb_regs; + uint32_t i, j; + int result = 0; + + memset(&arb_regs, 0, sizeof(SMU75_Discrete_MCArbDramTimingTable)); + + for (i = 0; i < hw_data->dpm_table.sclk_table.count; i++) { + for (j = 0; j < hw_data->dpm_table.mclk_table.count; j++) { + result = vegam_populate_memory_timing_parameters(hwmgr, + hw_data->dpm_table.sclk_table.dpm_levels[i].value, + hw_data->dpm_table.mclk_table.dpm_levels[j].value, + &arb_regs.entries[i][j]); + if (result) + return result; + } + } + + result = smu7_copy_bytes_to_smc( + hwmgr, + smu_data->smu7_data.arb_table_start, + (uint8_t *)&arb_regs, + sizeof(SMU75_Discrete_MCArbDramTimingTable), + SMC_RAM_END); + return result; +} + +static int vegam_populate_smc_uvd_level(struct pp_hwmgr *hwmgr, + struct SMU75_Discrete_DpmTable *table) +{ + int result = -EINVAL; + uint8_t count; + struct pp_atomctrl_clock_dividers_vi dividers; + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + struct phm_ppt_v1_mm_clock_voltage_dependency_table *mm_table = + table_info->mm_dep_table; + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + uint32_t vddci; + + table->UvdLevelCount = (uint8_t)(mm_table->count); + table->UvdBootLevel = 0; + + for (count = 0; count < table->UvdLevelCount; count++) { + table->UvdLevel[count].MinVoltage = 0; + table->UvdLevel[count].VclkFrequency = mm_table->entries[count].vclk; + table->UvdLevel[count].DclkFrequency = mm_table->entries[count].dclk; + table->UvdLevel[count].MinVoltage |= + (mm_table->entries[count].vddc * VOLTAGE_SCALE) << VDDC_SHIFT; + + if (SMU7_VOLTAGE_CONTROL_BY_GPIO == data->vddci_control) + vddci = (uint32_t)phm_find_closest_vddci(&(data->vddci_voltage_table), + mm_table->entries[count].vddc - VDDC_VDDCI_DELTA); + else if (SMU7_VOLTAGE_CONTROL_BY_SVID2 == data->vddci_control) + vddci = mm_table->entries[count].vddc - VDDC_VDDCI_DELTA; + else + vddci = (data->vbios_boot_state.vddci_bootup_value * VOLTAGE_SCALE) << VDDCI_SHIFT; + + table->UvdLevel[count].MinVoltage |= (vddci * VOLTAGE_SCALE) << VDDCI_SHIFT; + table->UvdLevel[count].MinVoltage |= 1 << PHASES_SHIFT; + + /* retrieve divider value for VBIOS */ + result = atomctrl_get_dfs_pll_dividers_vi(hwmgr, + table->UvdLevel[count].VclkFrequency, ÷rs); + PP_ASSERT_WITH_CODE((0 == result), + "can not find divide id for Vclk clock", return result); + + table->UvdLevel[count].VclkDivider = (uint8_t)dividers.pll_post_divider; + + result = atomctrl_get_dfs_pll_dividers_vi(hwmgr, + table->UvdLevel[count].DclkFrequency, ÷rs); + PP_ASSERT_WITH_CODE((0 == result), + "can not find divide id for Dclk clock", return result); + + table->UvdLevel[count].DclkDivider = (uint8_t)dividers.pll_post_divider; + + CONVERT_FROM_HOST_TO_SMC_UL(table->UvdLevel[count].VclkFrequency); + CONVERT_FROM_HOST_TO_SMC_UL(table->UvdLevel[count].DclkFrequency); + CONVERT_FROM_HOST_TO_SMC_UL(table->UvdLevel[count].MinVoltage); + } + + return result; +} + +static int vegam_populate_smc_boot_level(struct pp_hwmgr *hwmgr, + struct SMU75_Discrete_DpmTable *table) +{ + int result = 0; + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + + table->GraphicsBootLevel = 0; + table->MemoryBootLevel = 0; + + /* find boot level from dpm table */ + result = phm_find_boot_level(&(data->dpm_table.sclk_table), + data->vbios_boot_state.sclk_bootup_value, + (uint32_t *)&(table->GraphicsBootLevel)); + + result = phm_find_boot_level(&(data->dpm_table.mclk_table), + data->vbios_boot_state.mclk_bootup_value, + (uint32_t *)&(table->MemoryBootLevel)); + + table->BootVddc = data->vbios_boot_state.vddc_bootup_value * + VOLTAGE_SCALE; + table->BootVddci = data->vbios_boot_state.vddci_bootup_value * + VOLTAGE_SCALE; + table->BootMVdd = data->vbios_boot_state.mvdd_bootup_value * + VOLTAGE_SCALE; + + CONVERT_FROM_HOST_TO_SMC_US(table->BootVddc); + CONVERT_FROM_HOST_TO_SMC_US(table->BootVddci); + CONVERT_FROM_HOST_TO_SMC_US(table->BootMVdd); + + return 0; +} + +static int vegam_populate_smc_initial_state(struct pp_hwmgr *hwmgr) +{ + struct smu7_hwmgr *hw_data = (struct smu7_hwmgr *)(hwmgr->backend); + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + uint8_t count, level; + + count = (uint8_t)(table_info->vdd_dep_on_sclk->count); + + for (level = 0; level < count; level++) { + if (table_info->vdd_dep_on_sclk->entries[level].clk >= + hw_data->vbios_boot_state.sclk_bootup_value) { + smu_data->smc_state_table.GraphicsBootLevel = level; + break; + } + } + + count = (uint8_t)(table_info->vdd_dep_on_mclk->count); + for (level = 0; level < count; level++) { + if (table_info->vdd_dep_on_mclk->entries[level].clk >= + hw_data->vbios_boot_state.mclk_bootup_value) { + smu_data->smc_state_table.MemoryBootLevel = level; + break; + } + } + + return 0; +} + +static uint16_t scale_fan_gain_settings(uint16_t raw_setting) +{ + uint32_t tmp; + tmp = raw_setting * 4096 / 100; + return (uint16_t)tmp; +} + +static int vegam_populate_bapm_parameters_in_dpm_table(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + + const struct vegam_pt_defaults *defaults = smu_data->power_tune_defaults; + SMU75_Discrete_DpmTable *table = &(smu_data->smc_state_table); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + struct phm_cac_tdp_table *cac_dtp_table = table_info->cac_dtp_table; + struct pp_advance_fan_control_parameters *fan_table = + &hwmgr->thermal_controller.advanceFanControlParameters; + int i, j, k; + const uint16_t *pdef1; + const uint16_t *pdef2; + + table->DefaultTdp = PP_HOST_TO_SMC_US((uint16_t)(cac_dtp_table->usTDP * 128)); + table->TargetTdp = PP_HOST_TO_SMC_US((uint16_t)(cac_dtp_table->usTDP * 128)); + + PP_ASSERT_WITH_CODE(cac_dtp_table->usTargetOperatingTemp <= 255, + "Target Operating Temp is out of Range!", + ); + + table->TemperatureLimitEdge = PP_HOST_TO_SMC_US( + cac_dtp_table->usTargetOperatingTemp * 256); + table->TemperatureLimitHotspot = PP_HOST_TO_SMC_US( + cac_dtp_table->usTemperatureLimitHotspot * 256); + table->FanGainEdge = PP_HOST_TO_SMC_US( + scale_fan_gain_settings(fan_table->usFanGainEdge)); + table->FanGainHotspot = PP_HOST_TO_SMC_US( + scale_fan_gain_settings(fan_table->usFanGainHotspot)); + + pdef1 = defaults->BAPMTI_R; + pdef2 = defaults->BAPMTI_RC; + + for (i = 0; i < SMU75_DTE_ITERATIONS; i++) { + for (j = 0; j < SMU75_DTE_SOURCES; j++) { + for (k = 0; k < SMU75_DTE_SINKS; k++) { + table->BAPMTI_R[i][j][k] = PP_HOST_TO_SMC_US(*pdef1); + table->BAPMTI_RC[i][j][k] = PP_HOST_TO_SMC_US(*pdef2); + pdef1++; + pdef2++; + } + } + } + + return 0; +} + +static int vegam_populate_clock_stretcher_data_table(struct pp_hwmgr *hwmgr) +{ + uint32_t ro, efuse, volt_without_cks, volt_with_cks, value, max, min; + struct vegam_smumgr *smu_data = + (struct vegam_smumgr *)(hwmgr->smu_backend); + + uint8_t i, stretch_amount, stretch_amount2, volt_offset = 0; + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + struct phm_ppt_v1_clock_voltage_dependency_table *sclk_table = + table_info->vdd_dep_on_sclk; + uint32_t mask = (1 << ((STRAP_ASIC_RO_MSB - STRAP_ASIC_RO_LSB) + 1)) - 1; + + stretch_amount = (uint8_t)table_info->cac_dtp_table->usClockStretchAmount; + + atomctrl_read_efuse(hwmgr, STRAP_ASIC_RO_LSB, STRAP_ASIC_RO_MSB, + mask, &efuse); + + min = 1200; + max = 2500; + + ro = efuse * (max - min) / 255 + min; + + /* Populate Sclk_CKS_masterEn0_7 and Sclk_voltageOffset */ + for (i = 0; i < sclk_table->count; i++) { + smu_data->smc_state_table.Sclk_CKS_masterEn0_7 |= + sclk_table->entries[i].cks_enable << i; + volt_without_cks = (uint32_t)((2753594000U + (sclk_table->entries[i].clk/100) * + 136418 - (ro - 70) * 1000000) / + (2424180 - (sclk_table->entries[i].clk/100) * 1132925/1000)); + volt_with_cks = (uint32_t)((2797202000U + sclk_table->entries[i].clk/100 * + 3232 - (ro - 65) * 1000000) / + (2522480 - sclk_table->entries[i].clk/100 * 115764/100)); + + if (volt_without_cks >= volt_with_cks) + volt_offset = (uint8_t)(((volt_without_cks - volt_with_cks + + sclk_table->entries[i].cks_voffset) * 100 + 624) / 625); + + smu_data->smc_state_table.Sclk_voltageOffset[i] = volt_offset; + } + + smu_data->smc_state_table.LdoRefSel = + (table_info->cac_dtp_table->ucCKS_LDO_REFSEL != 0) ? + table_info->cac_dtp_table->ucCKS_LDO_REFSEL : 5; + /* Populate CKS Lookup Table */ + if (stretch_amount == 1 || stretch_amount == 2 || stretch_amount == 5) + stretch_amount2 = 0; + else if (stretch_amount == 3 || stretch_amount == 4) + stretch_amount2 = 1; + else { + phm_cap_unset(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_ClockStretcher); + PP_ASSERT_WITH_CODE(false, + "Stretch Amount in PPTable not supported\n", + return -EINVAL); + } + + value = cgs_read_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixPWR_CKS_CNTL); + value &= 0xFFFFFFFE; + cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, ixPWR_CKS_CNTL, value); + + return 0; +} + +static bool vegam_is_hw_avfs_present(struct pp_hwmgr *hwmgr) +{ + uint32_t efuse; + + efuse = cgs_read_ind_register(hwmgr->device, CGS_IND_REG__SMC, + ixSMU_EFUSE_0 + (49 * 4)); + efuse &= 0x00000001; + + if (efuse) + return true; + + return false; +} + +static int vegam_populate_avfs_parameters(struct pp_hwmgr *hwmgr) +{ + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + + SMU75_Discrete_DpmTable *table = &(smu_data->smc_state_table); + int result = 0; + struct pp_atom_ctrl__avfs_parameters avfs_params = {0}; + AVFS_meanNsigma_t AVFS_meanNsigma = { {0} }; + AVFS_Sclk_Offset_t AVFS_SclkOffset = { {0} }; + uint32_t tmp, i; + + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)hwmgr->pptable; + struct phm_ppt_v1_clock_voltage_dependency_table *sclk_table = + table_info->vdd_dep_on_sclk; + + if (!hwmgr->avfs_supported) + return 0; + + result = atomctrl_get_avfs_information(hwmgr, &avfs_params); + + if (0 == result) { + table->BTCGB_VDROOP_TABLE[0].a0 = + PP_HOST_TO_SMC_UL(avfs_params.ulGB_VDROOP_TABLE_CKSON_a0); + table->BTCGB_VDROOP_TABLE[0].a1 = + PP_HOST_TO_SMC_UL(avfs_params.ulGB_VDROOP_TABLE_CKSON_a1); + table->BTCGB_VDROOP_TABLE[0].a2 = + PP_HOST_TO_SMC_UL(avfs_params.ulGB_VDROOP_TABLE_CKSON_a2); + table->BTCGB_VDROOP_TABLE[1].a0 = + PP_HOST_TO_SMC_UL(avfs_params.ulGB_VDROOP_TABLE_CKSOFF_a0); + table->BTCGB_VDROOP_TABLE[1].a1 = + PP_HOST_TO_SMC_UL(avfs_params.ulGB_VDROOP_TABLE_CKSOFF_a1); + table->BTCGB_VDROOP_TABLE[1].a2 = + PP_HOST_TO_SMC_UL(avfs_params.ulGB_VDROOP_TABLE_CKSOFF_a2); + table->AVFSGB_FUSE_TABLE[0].m1 = + PP_HOST_TO_SMC_UL(avfs_params.ulAVFSGB_FUSE_TABLE_CKSON_m1); + table->AVFSGB_FUSE_TABLE[0].m2 = + PP_HOST_TO_SMC_US(avfs_params.usAVFSGB_FUSE_TABLE_CKSON_m2); + table->AVFSGB_FUSE_TABLE[0].b = + PP_HOST_TO_SMC_UL(avfs_params.ulAVFSGB_FUSE_TABLE_CKSON_b); + table->AVFSGB_FUSE_TABLE[0].m1_shift = 24; + table->AVFSGB_FUSE_TABLE[0].m2_shift = 12; + table->AVFSGB_FUSE_TABLE[1].m1 = + PP_HOST_TO_SMC_UL(avfs_params.ulAVFSGB_FUSE_TABLE_CKSOFF_m1); + table->AVFSGB_FUSE_TABLE[1].m2 = + PP_HOST_TO_SMC_US(avfs_params.usAVFSGB_FUSE_TABLE_CKSOFF_m2); + table->AVFSGB_FUSE_TABLE[1].b = + PP_HOST_TO_SMC_UL(avfs_params.ulAVFSGB_FUSE_TABLE_CKSOFF_b); + table->AVFSGB_FUSE_TABLE[1].m1_shift = 24; + table->AVFSGB_FUSE_TABLE[1].m2_shift = 12; + table->MaxVoltage = PP_HOST_TO_SMC_US(avfs_params.usMaxVoltage_0_25mv); + AVFS_meanNsigma.Aconstant[0] = + PP_HOST_TO_SMC_UL(avfs_params.ulAVFS_meanNsigma_Acontant0); + AVFS_meanNsigma.Aconstant[1] = + PP_HOST_TO_SMC_UL(avfs_params.ulAVFS_meanNsigma_Acontant1); + AVFS_meanNsigma.Aconstant[2] = + PP_HOST_TO_SMC_UL(avfs_params.ulAVFS_meanNsigma_Acontant2); + AVFS_meanNsigma.DC_tol_sigma = + PP_HOST_TO_SMC_US(avfs_params.usAVFS_meanNsigma_DC_tol_sigma); + AVFS_meanNsigma.Platform_mean = + PP_HOST_TO_SMC_US(avfs_params.usAVFS_meanNsigma_Platform_mean); + AVFS_meanNsigma.PSM_Age_CompFactor = + PP_HOST_TO_SMC_US(avfs_params.usPSM_Age_ComFactor); + AVFS_meanNsigma.Platform_sigma = + PP_HOST_TO_SMC_US(avfs_params.usAVFS_meanNsigma_Platform_sigma); + + for (i = 0; i < sclk_table->count; i++) { + AVFS_meanNsigma.Static_Voltage_Offset[i] = + (uint8_t)(sclk_table->entries[i].cks_voffset * 100 / 625); + AVFS_SclkOffset.Sclk_Offset[i] = + PP_HOST_TO_SMC_US((uint16_t) + (sclk_table->entries[i].sclk_offset) / 100); + } + + result = smu7_read_smc_sram_dword(hwmgr, + SMU7_FIRMWARE_HEADER_LOCATION + + offsetof(SMU75_Firmware_Header, AvfsMeanNSigma), + &tmp, SMC_RAM_END); + smu7_copy_bytes_to_smc(hwmgr, + tmp, + (uint8_t *)&AVFS_meanNsigma, + sizeof(AVFS_meanNsigma_t), + SMC_RAM_END); + + result = smu7_read_smc_sram_dword(hwmgr, + SMU7_FIRMWARE_HEADER_LOCATION + + offsetof(SMU75_Firmware_Header, AvfsSclkOffsetTable), + &tmp, SMC_RAM_END); + smu7_copy_bytes_to_smc(hwmgr, + tmp, + (uint8_t *)&AVFS_SclkOffset, + sizeof(AVFS_Sclk_Offset_t), + SMC_RAM_END); + + data->avfs_vdroop_override_setting = + (avfs_params.ucEnableGB_VDROOP_TABLE_CKSON << BTCGB0_Vdroop_Enable_SHIFT) | + (avfs_params.ucEnableGB_VDROOP_TABLE_CKSOFF << BTCGB1_Vdroop_Enable_SHIFT) | + (avfs_params.ucEnableGB_FUSE_TABLE_CKSON << AVFSGB0_Vdroop_Enable_SHIFT) | + (avfs_params.ucEnableGB_FUSE_TABLE_CKSOFF << AVFSGB1_Vdroop_Enable_SHIFT); + data->apply_avfs_cks_off_voltage = + (avfs_params.ucEnableApplyAVFS_CKS_OFF_Voltage == 1) ? true : false; + } + return result; +} + +static int vegam_populate_vr_config(struct pp_hwmgr *hwmgr, + struct SMU75_Discrete_DpmTable *table) +{ + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + struct vegam_smumgr *smu_data = + (struct vegam_smumgr *)(hwmgr->smu_backend); + uint16_t config; + + config = VR_MERGED_WITH_VDDC; + table->VRConfig |= (config << VRCONF_VDDGFX_SHIFT); + + /* Set Vddc Voltage Controller */ + if (SMU7_VOLTAGE_CONTROL_BY_SVID2 == data->voltage_control) { + config = VR_SVI2_PLANE_1; + table->VRConfig |= config; + } else { + PP_ASSERT_WITH_CODE(false, + "VDDC should be on SVI2 control in merged mode!", + ); + } + /* Set Vddci Voltage Controller */ + if (SMU7_VOLTAGE_CONTROL_BY_SVID2 == data->vddci_control) { + config = VR_SVI2_PLANE_2; /* only in merged mode */ + table->VRConfig |= (config << VRCONF_VDDCI_SHIFT); + } else if (SMU7_VOLTAGE_CONTROL_BY_GPIO == data->vddci_control) { + config = VR_SMIO_PATTERN_1; + table->VRConfig |= (config << VRCONF_VDDCI_SHIFT); + } else { + config = VR_STATIC_VOLTAGE; + table->VRConfig |= (config << VRCONF_VDDCI_SHIFT); + } + /* Set Mvdd Voltage Controller */ + if (SMU7_VOLTAGE_CONTROL_BY_SVID2 == data->mvdd_control) { + if (config != VR_SVI2_PLANE_2) { + config = VR_SVI2_PLANE_2; + table->VRConfig |= (config << VRCONF_MVDD_SHIFT); + cgs_write_ind_register(hwmgr->device, + CGS_IND_REG__SMC, + smu_data->smu7_data.soft_regs_start + + offsetof(SMU75_SoftRegisters, AllowMvddSwitch), + 0x1); + } else { + PP_ASSERT_WITH_CODE(false, + "SVI2 Plane 2 is already taken, set MVDD as Static",); + config = VR_STATIC_VOLTAGE; + table->VRConfig = (config << VRCONF_MVDD_SHIFT); + } + } else if (SMU7_VOLTAGE_CONTROL_BY_GPIO == data->mvdd_control) { + config = VR_SMIO_PATTERN_2; + table->VRConfig = (config << VRCONF_MVDD_SHIFT); + cgs_write_ind_register(hwmgr->device, + CGS_IND_REG__SMC, + smu_data->smu7_data.soft_regs_start + + offsetof(SMU75_SoftRegisters, AllowMvddSwitch), + 0x1); + } else { + config = VR_STATIC_VOLTAGE; + table->VRConfig |= (config << VRCONF_MVDD_SHIFT); + } + + return 0; +} + +static int vegam_populate_svi_load_line(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + const struct vegam_pt_defaults *defaults = smu_data->power_tune_defaults; + + smu_data->power_tune_table.SviLoadLineEn = defaults->SviLoadLineEn; + smu_data->power_tune_table.SviLoadLineVddC = defaults->SviLoadLineVddC; + smu_data->power_tune_table.SviLoadLineTrimVddC = 3; + smu_data->power_tune_table.SviLoadLineOffsetVddC = 0; + + return 0; +} + +static int vegam_populate_tdc_limit(struct pp_hwmgr *hwmgr) +{ + uint16_t tdc_limit; + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + const struct vegam_pt_defaults *defaults = smu_data->power_tune_defaults; + + tdc_limit = (uint16_t)(table_info->cac_dtp_table->usTDC * 128); + smu_data->power_tune_table.TDC_VDDC_PkgLimit = + CONVERT_FROM_HOST_TO_SMC_US(tdc_limit); + smu_data->power_tune_table.TDC_VDDC_ThrottleReleaseLimitPerc = + defaults->TDC_VDDC_ThrottleReleaseLimitPerc; + smu_data->power_tune_table.TDC_MAWt = defaults->TDC_MAWt; + + return 0; +} + +static int vegam_populate_dw8(struct pp_hwmgr *hwmgr, uint32_t fuse_table_offset) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + const struct vegam_pt_defaults *defaults = smu_data->power_tune_defaults; + uint32_t temp; + + if (smu7_read_smc_sram_dword(hwmgr, + fuse_table_offset + + offsetof(SMU75_Discrete_PmFuses, TdcWaterfallCtl), + (uint32_t *)&temp, SMC_RAM_END)) + PP_ASSERT_WITH_CODE(false, + "Attempt to read PmFuses.DW6 (SviLoadLineEn) from SMC Failed!", + return -EINVAL); + else { + smu_data->power_tune_table.TdcWaterfallCtl = defaults->TdcWaterfallCtl; + smu_data->power_tune_table.LPMLTemperatureMin = + (uint8_t)((temp >> 16) & 0xff); + smu_data->power_tune_table.LPMLTemperatureMax = + (uint8_t)((temp >> 8) & 0xff); + smu_data->power_tune_table.Reserved = (uint8_t)(temp & 0xff); + } + return 0; +} + +static int vegam_populate_temperature_scaler(struct pp_hwmgr *hwmgr) +{ + int i; + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + + /* Currently not used. Set all to zero. */ + for (i = 0; i < 16; i++) + smu_data->power_tune_table.LPMLTemperatureScaler[i] = 0; + + return 0; +} + +static int vegam_populate_fuzzy_fan(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + +/* TO DO move to hwmgr */ + if ((hwmgr->thermal_controller.advanceFanControlParameters.usFanOutputSensitivity & (1 << 15)) + || 0 == hwmgr->thermal_controller.advanceFanControlParameters.usFanOutputSensitivity) + hwmgr->thermal_controller.advanceFanControlParameters.usFanOutputSensitivity = + hwmgr->thermal_controller.advanceFanControlParameters.usDefaultFanOutputSensitivity; + + smu_data->power_tune_table.FuzzyFan_PwmSetDelta = PP_HOST_TO_SMC_US( + hwmgr->thermal_controller.advanceFanControlParameters.usFanOutputSensitivity); + return 0; +} + +static int vegam_populate_gnb_lpml(struct pp_hwmgr *hwmgr) +{ + int i; + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + + /* Currently not used. Set all to zero. */ + for (i = 0; i < 16; i++) + smu_data->power_tune_table.GnbLPML[i] = 0; + + return 0; +} + +static int vegam_populate_bapm_vddc_base_leakage_sidd(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + uint16_t hi_sidd = smu_data->power_tune_table.BapmVddCBaseLeakageHiSidd; + uint16_t lo_sidd = smu_data->power_tune_table.BapmVddCBaseLeakageLoSidd; + struct phm_cac_tdp_table *cac_table = table_info->cac_dtp_table; + + hi_sidd = (uint16_t)(cac_table->usHighCACLeakage / 100 * 256); + lo_sidd = (uint16_t)(cac_table->usLowCACLeakage / 100 * 256); + + smu_data->power_tune_table.BapmVddCBaseLeakageHiSidd = + CONVERT_FROM_HOST_TO_SMC_US(hi_sidd); + smu_data->power_tune_table.BapmVddCBaseLeakageLoSidd = + CONVERT_FROM_HOST_TO_SMC_US(lo_sidd); + + return 0; +} + +static int vegam_populate_pm_fuses(struct pp_hwmgr *hwmgr) +{ + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + uint32_t pm_fuse_table_offset; + + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_PowerContainment)) { + if (smu7_read_smc_sram_dword(hwmgr, + SMU7_FIRMWARE_HEADER_LOCATION + + offsetof(SMU75_Firmware_Header, PmFuseTable), + &pm_fuse_table_offset, SMC_RAM_END)) + PP_ASSERT_WITH_CODE(false, + "Attempt to get pm_fuse_table_offset Failed!", + return -EINVAL); + + if (vegam_populate_svi_load_line(hwmgr)) + PP_ASSERT_WITH_CODE(false, + "Attempt to populate SviLoadLine Failed!", + return -EINVAL); + + if (vegam_populate_tdc_limit(hwmgr)) + PP_ASSERT_WITH_CODE(false, + "Attempt to populate TDCLimit Failed!", return -EINVAL); + + if (vegam_populate_dw8(hwmgr, pm_fuse_table_offset)) + PP_ASSERT_WITH_CODE(false, + "Attempt to populate TdcWaterfallCtl, " + "LPMLTemperature Min and Max Failed!", + return -EINVAL); + + if (0 != vegam_populate_temperature_scaler(hwmgr)) + PP_ASSERT_WITH_CODE(false, + "Attempt to populate LPMLTemperatureScaler Failed!", + return -EINVAL); + + if (vegam_populate_fuzzy_fan(hwmgr)) + PP_ASSERT_WITH_CODE(false, + "Attempt to populate Fuzzy Fan Control parameters Failed!", + return -EINVAL); + + if (vegam_populate_gnb_lpml(hwmgr)) + PP_ASSERT_WITH_CODE(false, + "Attempt to populate GnbLPML Failed!", + return -EINVAL); + + if (vegam_populate_bapm_vddc_base_leakage_sidd(hwmgr)) + PP_ASSERT_WITH_CODE(false, + "Attempt to populate BapmVddCBaseLeakage Hi and Lo " + "Sidd Failed!", return -EINVAL); + + if (smu7_copy_bytes_to_smc(hwmgr, pm_fuse_table_offset, + (uint8_t *)&smu_data->power_tune_table, + (sizeof(struct SMU75_Discrete_PmFuses) - PMFUSES_AVFSSIZE), + SMC_RAM_END)) + PP_ASSERT_WITH_CODE(false, + "Attempt to download PmFuseTable Failed!", + return -EINVAL); + } + return 0; +} + +static int vegam_enable_reconfig_cus(struct pp_hwmgr *hwmgr) +{ + struct amdgpu_device *adev = hwmgr->adev; + + smum_send_msg_to_smc_with_parameter(hwmgr, + PPSMC_MSG_EnableModeSwitchRLCNotification, + adev->gfx.cu_info.number); + + return 0; +} + +static int vegam_init_smc_table(struct pp_hwmgr *hwmgr) +{ + int result; + struct smu7_hwmgr *hw_data = (struct smu7_hwmgr *)(hwmgr->backend); + struct vegam_smumgr *smu_data = (struct vegam_smumgr *)(hwmgr->smu_backend); + + struct phm_ppt_v1_information *table_info = + (struct phm_ppt_v1_information *)(hwmgr->pptable); + struct SMU75_Discrete_DpmTable *table = &(smu_data->smc_state_table); + uint8_t i; + struct pp_atomctrl_gpio_pin_assignment gpio_pin; + struct phm_ppt_v1_gpio_table *gpio_table = + (struct phm_ppt_v1_gpio_table *)table_info->gpio_table; + pp_atomctrl_clock_dividers_vi dividers; + + phm_cap_set(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_AutomaticDCTransition); + + vegam_initialize_power_tune_defaults(hwmgr); + + if (SMU7_VOLTAGE_CONTROL_NONE != hw_data->voltage_control) + vegam_populate_smc_voltage_tables(hwmgr, table); + + table->SystemFlags = 0; + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_AutomaticDCTransition)) + table->SystemFlags |= PPSMC_SYSTEMFLAG_GPIO_DC; + + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_StepVddc)) + table->SystemFlags |= PPSMC_SYSTEMFLAG_STEPVDDC; + + if (hw_data->is_memory_gddr5) + table->SystemFlags |= PPSMC_SYSTEMFLAG_GDDR5; + + if (hw_data->ulv_supported && table_info->us_ulv_voltage_offset) { + result = vegam_populate_ulv_state(hwmgr, table); + PP_ASSERT_WITH_CODE(!result, + "Failed to initialize ULV state!", return result); + cgs_write_ind_register(hwmgr->device, CGS_IND_REG__SMC, + ixCG_ULV_PARAMETER, SMU7_CGULVPARAMETER_DFLT); + } + + result = vegam_populate_smc_link_level(hwmgr, table); + PP_ASSERT_WITH_CODE(!result, + "Failed to initialize Link Level!", return result); + + result = vegam_populate_all_graphic_levels(hwmgr); + PP_ASSERT_WITH_CODE(!result, + "Failed to initialize Graphics Level!", return result); + + result = vegam_populate_all_memory_levels(hwmgr); + PP_ASSERT_WITH_CODE(!result, + "Failed to initialize Memory Level!", return result); + + result = vegam_populate_smc_acpi_level(hwmgr, table); + PP_ASSERT_WITH_CODE(!result, + "Failed to initialize ACPI Level!", return result); + + result = vegam_populate_smc_vce_level(hwmgr, table); + PP_ASSERT_WITH_CODE(!result, + "Failed to initialize VCE Level!", return result); + + result = vegam_populate_smc_samu_level(hwmgr, table); + PP_ASSERT_WITH_CODE(!result, + "Failed to initialize SAMU Level!", return result); + + /* Since only the initial state is completely set up at this point + * (the other states are just copies of the boot state) we only + * need to populate the ARB settings for the initial state. + */ + result = vegam_program_memory_timing_parameters(hwmgr); + PP_ASSERT_WITH_CODE(!result, + "Failed to Write ARB settings for the initial state.", return result); + + result = vegam_populate_smc_uvd_level(hwmgr, table); + PP_ASSERT_WITH_CODE(!result, + "Failed to initialize UVD Level!", return result); + + result = vegam_populate_smc_boot_level(hwmgr, table); + PP_ASSERT_WITH_CODE(!result, + "Failed to initialize Boot Level!", return result); + + result = vegam_populate_smc_initial_state(hwmgr); + PP_ASSERT_WITH_CODE(!result, + "Failed to initialize Boot State!", return result); + + result = vegam_populate_bapm_parameters_in_dpm_table(hwmgr); + PP_ASSERT_WITH_CODE(!result, + "Failed to populate BAPM Parameters!", return result); + + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_ClockStretcher)) { + result = vegam_populate_clock_stretcher_data_table(hwmgr); + PP_ASSERT_WITH_CODE(!result, + "Failed to populate Clock Stretcher Data Table!", + return result); + } + + result = vegam_populate_avfs_parameters(hwmgr); + PP_ASSERT_WITH_CODE(!result, + "Failed to populate AVFS Parameters!", return result;); + + table->CurrSclkPllRange = 0xff; + table->GraphicsVoltageChangeEnable = 1; + table->GraphicsThermThrottleEnable = 1; + table->GraphicsInterval = 1; + table->VoltageInterval = 1; + table->ThermalInterval = 1; + table->TemperatureLimitHigh = + table_info->cac_dtp_table->usTargetOperatingTemp * + SMU7_Q88_FORMAT_CONVERSION_UNIT; + table->TemperatureLimitLow = + (table_info->cac_dtp_table->usTargetOperatingTemp - 1) * + SMU7_Q88_FORMAT_CONVERSION_UNIT; + table->MemoryVoltageChangeEnable = 1; + table->MemoryInterval = 1; + table->VoltageResponseTime = 0; + table->PhaseResponseTime = 0; + table->MemoryThermThrottleEnable = 1; + + PP_ASSERT_WITH_CODE(hw_data->dpm_table.pcie_speed_table.count >= 1, + "There must be 1 or more PCIE levels defined in PPTable.", + return -EINVAL); + table->PCIeBootLinkLevel = + hw_data->dpm_table.pcie_speed_table.count; + table->PCIeGenInterval = 1; + table->VRConfig = 0; + + result = vegam_populate_vr_config(hwmgr, table); + PP_ASSERT_WITH_CODE(!result, + "Failed to populate VRConfig setting!", return result); + + table->ThermGpio = 17; + table->SclkStepSize = 0x4000; + + if (atomctrl_get_pp_assign_pin(hwmgr, + VDDC_VRHOT_GPIO_PINID, &gpio_pin)) { + table->VRHotGpio = gpio_pin.uc_gpio_pin_bit_shift; + if (gpio_table) + table->VRHotLevel = + table_info->gpio_table->vrhot_triggered_sclk_dpm_index; + } else { + table->VRHotGpio = SMU7_UNUSED_GPIO_PIN; + phm_cap_unset(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_RegulatorHot); + } + + if (atomctrl_get_pp_assign_pin(hwmgr, + PP_AC_DC_SWITCH_GPIO_PINID, &gpio_pin)) { + table->AcDcGpio = gpio_pin.uc_gpio_pin_bit_shift; + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_AutomaticDCTransition) && + !smum_send_msg_to_smc(hwmgr, PPSMC_MSG_UseNewGPIOScheme)) + phm_cap_set(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_SMCtoPPLIBAcdcGpioScheme); + } else { + table->AcDcGpio = SMU7_UNUSED_GPIO_PIN; + phm_cap_unset(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_AutomaticDCTransition); + } + + /* Thermal Output GPIO */ + if (atomctrl_get_pp_assign_pin(hwmgr, + THERMAL_INT_OUTPUT_GPIO_PINID, &gpio_pin)) { + table->ThermOutGpio = gpio_pin.uc_gpio_pin_bit_shift; + + /* For porlarity read GPIOPAD_A with assigned Gpio pin + * since VBIOS will program this register to set 'inactive state', + * driver can then determine 'active state' from this and + * program SMU with correct polarity + */ + table->ThermOutPolarity = + (0 == (cgs_read_register(hwmgr->device, mmGPIOPAD_A) & + (1 << gpio_pin.uc_gpio_pin_bit_shift))) ? 1:0; + table->ThermOutMode = SMU7_THERM_OUT_MODE_THERM_ONLY; + + /* if required, combine VRHot/PCC with thermal out GPIO */ + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_RegulatorHot) && + phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_CombinePCCWithThermalSignal)) + table->ThermOutMode = SMU7_THERM_OUT_MODE_THERM_VRHOT; + } else { + table->ThermOutGpio = 17; + table->ThermOutPolarity = 1; + table->ThermOutMode = SMU7_THERM_OUT_MODE_DISABLE; + } + + /* Populate BIF_SCLK levels into SMC DPM table */ + for (i = 0; i <= hw_data->dpm_table.pcie_speed_table.count; i++) { + result = atomctrl_get_dfs_pll_dividers_vi(hwmgr, + smu_data->bif_sclk_table[i], ÷rs); + PP_ASSERT_WITH_CODE(!result, + "Can not find DFS divide id for Sclk", + return result); + + if (i == 0) + table->Ulv.BifSclkDfs = + PP_HOST_TO_SMC_US((uint16_t)(dividers.pll_post_divider)); + else + table->LinkLevel[i - 1].BifSclkDfs = + PP_HOST_TO_SMC_US((uint16_t)(dividers.pll_post_divider)); + } + + for (i = 0; i < SMU75_MAX_ENTRIES_SMIO; i++) + table->Smio[i] = PP_HOST_TO_SMC_UL(table->Smio[i]); + + CONVERT_FROM_HOST_TO_SMC_UL(table->SystemFlags); + CONVERT_FROM_HOST_TO_SMC_UL(table->VRConfig); + CONVERT_FROM_HOST_TO_SMC_UL(table->SmioMask1); + CONVERT_FROM_HOST_TO_SMC_UL(table->SmioMask2); + CONVERT_FROM_HOST_TO_SMC_UL(table->SclkStepSize); + CONVERT_FROM_HOST_TO_SMC_UL(table->CurrSclkPllRange); + CONVERT_FROM_HOST_TO_SMC_US(table->TemperatureLimitHigh); + CONVERT_FROM_HOST_TO_SMC_US(table->TemperatureLimitLow); + CONVERT_FROM_HOST_TO_SMC_US(table->VoltageResponseTime); + CONVERT_FROM_HOST_TO_SMC_US(table->PhaseResponseTime); + + /* Upload all dpm data to SMC memory.(dpm level, dpm level count etc) */ + result = smu7_copy_bytes_to_smc(hwmgr, + smu_data->smu7_data.dpm_table_start + + offsetof(SMU75_Discrete_DpmTable, SystemFlags), + (uint8_t *)&(table->SystemFlags), + sizeof(SMU75_Discrete_DpmTable) - 3 * sizeof(SMU75_PIDController), + SMC_RAM_END); + PP_ASSERT_WITH_CODE(!result, + "Failed to upload dpm data to SMC memory!", return result); + + result = vegam_populate_pm_fuses(hwmgr); + PP_ASSERT_WITH_CODE(!result, + "Failed to populate PM fuses to SMC memory!", return result); + + result = vegam_enable_reconfig_cus(hwmgr); + PP_ASSERT_WITH_CODE(!result, + "Failed to enable reconfigurable CUs!", return result); + + return 0; +} + +static uint32_t vegam_get_offsetof(uint32_t type, uint32_t member) +{ + switch (type) { + case SMU_SoftRegisters: + switch (member) { + case HandshakeDisables: + return offsetof(SMU75_SoftRegisters, HandshakeDisables); + case VoltageChangeTimeout: + return offsetof(SMU75_SoftRegisters, VoltageChangeTimeout); + case AverageGraphicsActivity: + return offsetof(SMU75_SoftRegisters, AverageGraphicsActivity); + case PreVBlankGap: + return offsetof(SMU75_SoftRegisters, PreVBlankGap); + case VBlankTimeout: + return offsetof(SMU75_SoftRegisters, VBlankTimeout); + case UcodeLoadStatus: + return offsetof(SMU75_SoftRegisters, UcodeLoadStatus); + case DRAM_LOG_ADDR_H: + return offsetof(SMU75_SoftRegisters, DRAM_LOG_ADDR_H); + case DRAM_LOG_ADDR_L: + return offsetof(SMU75_SoftRegisters, DRAM_LOG_ADDR_L); + case DRAM_LOG_PHY_ADDR_H: + return offsetof(SMU75_SoftRegisters, DRAM_LOG_PHY_ADDR_H); + case DRAM_LOG_PHY_ADDR_L: + return offsetof(SMU75_SoftRegisters, DRAM_LOG_PHY_ADDR_L); + case DRAM_LOG_BUFF_SIZE: + return offsetof(SMU75_SoftRegisters, DRAM_LOG_BUFF_SIZE); + } + case SMU_Discrete_DpmTable: + switch (member) { + case UvdBootLevel: + return offsetof(SMU75_Discrete_DpmTable, UvdBootLevel); + case VceBootLevel: + return offsetof(SMU75_Discrete_DpmTable, VceBootLevel); + case SamuBootLevel: + return offsetof(SMU75_Discrete_DpmTable, SamuBootLevel); + case LowSclkInterruptThreshold: + return offsetof(SMU75_Discrete_DpmTable, LowSclkInterruptThreshold); + } + } + pr_warn("can't get the offset of type %x member %x\n", type, member); + return 0; +} + +static int vegam_program_mem_timing_parameters(struct pp_hwmgr *hwmgr) +{ + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + + if (data->need_update_smu7_dpm_table & + (DPMTABLE_OD_UPDATE_SCLK + + DPMTABLE_UPDATE_SCLK + + DPMTABLE_UPDATE_MCLK)) + return vegam_program_memory_timing_parameters(hwmgr); + + return 0; +} + +static int vegam_update_sclk_threshold(struct pp_hwmgr *hwmgr) +{ + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + struct vegam_smumgr *smu_data = + (struct vegam_smumgr *)(hwmgr->smu_backend); + int result = 0; + uint32_t low_sclk_interrupt_threshold = 0; + + if (phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_SclkThrottleLowNotification) + && (data->low_sclk_interrupt_threshold != 0)) { + low_sclk_interrupt_threshold = + data->low_sclk_interrupt_threshold; + + CONVERT_FROM_HOST_TO_SMC_UL(low_sclk_interrupt_threshold); + + result = smu7_copy_bytes_to_smc( + hwmgr, + smu_data->smu7_data.dpm_table_start + + offsetof(SMU75_Discrete_DpmTable, + LowSclkInterruptThreshold), + (uint8_t *)&low_sclk_interrupt_threshold, + sizeof(uint32_t), + SMC_RAM_END); + } + PP_ASSERT_WITH_CODE((result == 0), + "Failed to update SCLK threshold!", return result); + + result = vegam_program_mem_timing_parameters(hwmgr); + PP_ASSERT_WITH_CODE((result == 0), + "Failed to program memory timing parameters!", + ); + + return result; +} + +int vegam_thermal_avfs_enable(struct pp_hwmgr *hwmgr) +{ + struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend); + int ret; + + if (!hwmgr->avfs_supported) + return 0; + + ret = smum_send_msg_to_smc(hwmgr, PPSMC_MSG_EnableAvfs); + if (!ret) { + if (data->apply_avfs_cks_off_voltage) + ret = smum_send_msg_to_smc(hwmgr, PPSMC_MSG_ApplyAvfsCksOffVoltage); + } + + return ret; +} + +static int vegam_thermal_setup_fan_table(struct pp_hwmgr *hwmgr) +{ + PP_ASSERT_WITH_CODE(hwmgr->thermal_controller.fanInfo.bNoFan, + "VBIOS fan info is not correct!", + ); + phm_cap_unset(hwmgr->platform_descriptor.platformCaps, + PHM_PlatformCaps_MicrocodeFanControl); + return 0; +} + +const struct pp_smumgr_func vegam_smu_funcs = { + .smu_init = vegam_smu_init, + .smu_fini = smu7_smu_fini, + .start_smu = vegam_start_smu, + .check_fw_load_finish = smu7_check_fw_load_finish, + .request_smu_load_fw = smu7_reload_firmware, + .request_smu_load_specific_fw = NULL, + .send_msg_to_smc = smu7_send_msg_to_smc, + .send_msg_to_smc_with_parameter = smu7_send_msg_to_smc_with_parameter, + .process_firmware_header = vegam_process_firmware_header, + .is_dpm_running = vegam_is_dpm_running, + .get_mac_definition = vegam_get_mac_definition, + .update_smc_table = vegam_update_smc_table, + .init_smc_table = vegam_init_smc_table, + .get_offsetof = vegam_get_offsetof, + .populate_all_graphic_levels = vegam_populate_all_graphic_levels, + .populate_all_memory_levels = vegam_populate_all_memory_levels, + .update_sclk_threshold = vegam_update_sclk_threshold, + .is_hw_avfs_present = vegam_is_hw_avfs_present, + .thermal_avfs_enable = vegam_thermal_avfs_enable, + .thermal_setup_fan_table = vegam_thermal_setup_fan_table, +}; diff --git a/drivers/gpu/drm/amd/powerplay/smumgr/vegam_smumgr.h b/drivers/gpu/drm/amd/powerplay/smumgr/vegam_smumgr.h new file mode 100644 index 000000000000..2b6558238500 --- /dev/null +++ b/drivers/gpu/drm/amd/powerplay/smumgr/vegam_smumgr.h @@ -0,0 +1,75 @@ +/* + * Copyright 2017 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ + +#ifndef _VEGAM_SMUMANAGER_H +#define _VEGAM_SMUMANAGER_H + + +#include +#include "smu75_discrete.h" +#include "smu7_smumgr.h" + +#define SMC_RAM_END 0x40000 + +#define DPMTuning_Uphyst_Shift 0 +#define DPMTuning_Downhyst_Shift 8 +#define DPMTuning_Activity_Shift 16 + +#define GraphicsDPMTuning_VEGAM 0x001e6400 +#define MemoryDPMTuning_VEGAM 0x000f3c0a +#define SclkDPMTuning_VEGAM 0x002d000a +#define MclkDPMTuning_VEGAM 0x001f100a + + +struct vegam_pt_defaults { + uint8_t SviLoadLineEn; + uint8_t SviLoadLineVddC; + uint8_t TDC_VDDC_ThrottleReleaseLimitPerc; + uint8_t TDC_MAWt; + uint8_t TdcWaterfallCtl; + uint8_t DTEAmbientTempBase; + + uint32_t DisplayCac; + uint32_t BAPM_TEMP_GRADIENT; + uint16_t BAPMTI_R[SMU75_DTE_ITERATIONS * SMU75_DTE_SOURCES * SMU75_DTE_SINKS]; + uint16_t BAPMTI_RC[SMU75_DTE_ITERATIONS * SMU75_DTE_SOURCES * SMU75_DTE_SINKS]; +}; + +struct vegam_range_table { + uint32_t trans_lower_frequency; /* in 10khz */ + uint32_t trans_upper_frequency; +}; + +struct vegam_smumgr { + struct smu7_smumgr smu7_data; + uint8_t protected_mode; + SMU75_Discrete_DpmTable smc_state_table; + struct SMU75_Discrete_Ulv ulv_setting; + struct SMU75_Discrete_PmFuses power_tune_table; + struct vegam_range_table range_table[NUM_SCLK_RANGE]; + const struct vegam_pt_defaults *power_tune_defaults; + uint32_t bif_sclk_table[SMU75_MAX_LEVELS_LINK]; +}; + + +#endif diff --git a/drivers/gpu/drm/ast/ast_mode.c b/drivers/gpu/drm/ast/ast_mode.c index 831b73392d82..036dff8a1f33 100644 --- a/drivers/gpu/drm/ast/ast_mode.c +++ b/drivers/gpu/drm/ast/ast_mode.c @@ -799,7 +799,7 @@ static int ast_get_modes(struct drm_connector *connector) return 0; } -static int ast_mode_valid(struct drm_connector *connector, +static enum drm_mode_status ast_mode_valid(struct drm_connector *connector, struct drm_display_mode *mode) { struct ast_private *ast = connector->dev->dev_private; diff --git a/drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_dc.h b/drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_dc.h index ab32d5b268d2..60c937f42114 100644 --- a/drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_dc.h +++ b/drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_dc.h @@ -299,7 +299,6 @@ struct atmel_hlcdc_layer { struct atmel_hlcdc_plane { struct drm_plane base; struct atmel_hlcdc_layer layer; - struct atmel_hlcdc_plane_properties *properties; }; static inline struct atmel_hlcdc_plane * @@ -345,18 +344,6 @@ struct atmel_hlcdc_dc_desc { int nlayers; }; -/** - * Atmel HLCDC Plane properties. - * - * This structure stores plane property definitions. - * - * @alpha: alpha blending (or transparency) property - * @rotation: rotation property - */ -struct atmel_hlcdc_plane_properties { - struct drm_property *alpha; -}; - /** * Atmel HLCDC Display Controller. * diff --git a/drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_plane.c b/drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_plane.c index e18800ed7cd1..73c875db45f4 100644 --- a/drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_plane.c +++ b/drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_plane.c @@ -31,7 +31,6 @@ * @src_y: y buffer position * @src_w: buffer width * @src_h: buffer height - * @alpha: alpha blending of the plane * @disc_x: x discard position * @disc_y: y discard position * @disc_w: discard width @@ -54,8 +53,6 @@ struct atmel_hlcdc_plane_state { uint32_t src_w; uint32_t src_h; - u8 alpha; - int disc_x; int disc_y; int disc_w; @@ -385,7 +382,7 @@ atmel_hlcdc_plane_update_general_settings(struct atmel_hlcdc_plane *plane, cfg |= ATMEL_HLCDC_LAYER_LAEN; else cfg |= ATMEL_HLCDC_LAYER_GAEN | - ATMEL_HLCDC_LAYER_GA(state->alpha); + ATMEL_HLCDC_LAYER_GA(state->base.alpha >> 8); } if (state->disc_h && state->disc_w) @@ -553,7 +550,7 @@ atmel_hlcdc_plane_prepare_disc_area(struct drm_crtc_state *c_state) if (!ovl_s->fb || ovl_s->fb->format->has_alpha || - ovl_state->alpha != 255) + ovl_s->alpha != DRM_BLEND_ALPHA_OPAQUE) continue; /* TODO: implement a smarter hidden area detection */ @@ -829,51 +826,18 @@ static void atmel_hlcdc_plane_destroy(struct drm_plane *p) drm_plane_cleanup(p); } -static int atmel_hlcdc_plane_atomic_set_property(struct drm_plane *p, - struct drm_plane_state *s, - struct drm_property *property, - uint64_t val) -{ - struct atmel_hlcdc_plane *plane = drm_plane_to_atmel_hlcdc_plane(p); - struct atmel_hlcdc_plane_properties *props = plane->properties; - struct atmel_hlcdc_plane_state *state = - drm_plane_state_to_atmel_hlcdc_plane_state(s); - - if (property == props->alpha) - state->alpha = val; - else - return -EINVAL; - - return 0; -} - -static int atmel_hlcdc_plane_atomic_get_property(struct drm_plane *p, - const struct drm_plane_state *s, - struct drm_property *property, - uint64_t *val) -{ - struct atmel_hlcdc_plane *plane = drm_plane_to_atmel_hlcdc_plane(p); - struct atmel_hlcdc_plane_properties *props = plane->properties; - const struct atmel_hlcdc_plane_state *state = - container_of(s, const struct atmel_hlcdc_plane_state, base); - - if (property == props->alpha) - *val = state->alpha; - else - return -EINVAL; - - return 0; -} - -static int atmel_hlcdc_plane_init_properties(struct atmel_hlcdc_plane *plane, - struct atmel_hlcdc_plane_properties *props) +static int atmel_hlcdc_plane_init_properties(struct atmel_hlcdc_plane *plane) { const struct atmel_hlcdc_layer_desc *desc = plane->layer.desc; if (desc->type == ATMEL_HLCDC_OVERLAY_LAYER || - desc->type == ATMEL_HLCDC_CURSOR_LAYER) - drm_object_attach_property(&plane->base.base, - props->alpha, 255); + desc->type == ATMEL_HLCDC_CURSOR_LAYER) { + int ret; + + ret = drm_plane_create_alpha_property(&plane->base); + if (ret) + return ret; + } if (desc->layout.xstride && desc->layout.pstride) { int ret; @@ -988,8 +952,8 @@ static void atmel_hlcdc_plane_reset(struct drm_plane *p) return; } - state->alpha = 255; p->state = &state->base; + p->state->alpha = DRM_BLEND_ALPHA_OPAQUE; p->state->plane = p; } } @@ -1042,13 +1006,10 @@ static const struct drm_plane_funcs layer_plane_funcs = { .reset = atmel_hlcdc_plane_reset, .atomic_duplicate_state = atmel_hlcdc_plane_atomic_duplicate_state, .atomic_destroy_state = atmel_hlcdc_plane_atomic_destroy_state, - .atomic_set_property = atmel_hlcdc_plane_atomic_set_property, - .atomic_get_property = atmel_hlcdc_plane_atomic_get_property, }; static int atmel_hlcdc_plane_create(struct drm_device *dev, - const struct atmel_hlcdc_layer_desc *desc, - struct atmel_hlcdc_plane_properties *props) + const struct atmel_hlcdc_layer_desc *desc) { struct atmel_hlcdc_dc *dc = dev->dev_private; struct atmel_hlcdc_plane *plane; @@ -1060,7 +1021,6 @@ static int atmel_hlcdc_plane_create(struct drm_device *dev, return -ENOMEM; atmel_hlcdc_layer_init(&plane->layer, desc, dc->hlcdc->regmap); - plane->properties = props; if (desc->type == ATMEL_HLCDC_BASE_LAYER) type = DRM_PLANE_TYPE_PRIMARY; @@ -1081,7 +1041,7 @@ static int atmel_hlcdc_plane_create(struct drm_device *dev, &atmel_hlcdc_layer_plane_helper_funcs); /* Set default property values*/ - ret = atmel_hlcdc_plane_init_properties(plane, props); + ret = atmel_hlcdc_plane_init_properties(plane); if (ret) return ret; @@ -1090,34 +1050,13 @@ static int atmel_hlcdc_plane_create(struct drm_device *dev, return 0; } -static struct atmel_hlcdc_plane_properties * -atmel_hlcdc_plane_create_properties(struct drm_device *dev) -{ - struct atmel_hlcdc_plane_properties *props; - - props = devm_kzalloc(dev->dev, sizeof(*props), GFP_KERNEL); - if (!props) - return ERR_PTR(-ENOMEM); - - props->alpha = drm_property_create_range(dev, 0, "alpha", 0, 255); - if (!props->alpha) - return ERR_PTR(-ENOMEM); - - return props; -} - int atmel_hlcdc_create_planes(struct drm_device *dev) { struct atmel_hlcdc_dc *dc = dev->dev_private; - struct atmel_hlcdc_plane_properties *props; const struct atmel_hlcdc_layer_desc *descs = dc->desc->layers; int nlayers = dc->desc->nlayers; int i, ret; - props = atmel_hlcdc_plane_create_properties(dev); - if (IS_ERR(props)) - return PTR_ERR(props); - dc->dscrpool = dmam_pool_create("atmel-hlcdc-dscr", dev->dev, sizeof(struct atmel_hlcdc_dma_channel_dscr), sizeof(u64), 0); @@ -1130,7 +1069,7 @@ int atmel_hlcdc_create_planes(struct drm_device *dev) descs[i].type != ATMEL_HLCDC_CURSOR_LAYER) continue; - ret = atmel_hlcdc_plane_create(dev, &descs[i], props); + ret = atmel_hlcdc_plane_create(dev, &descs[i]); if (ret) return ret; } diff --git a/drivers/gpu/drm/bochs/bochs_kms.c b/drivers/gpu/drm/bochs/bochs_kms.c index a24a18fbd65a..233980a78591 100644 --- a/drivers/gpu/drm/bochs/bochs_kms.c +++ b/drivers/gpu/drm/bochs/bochs_kms.c @@ -188,7 +188,7 @@ static int bochs_connector_get_modes(struct drm_connector *connector) return count; } -static int bochs_connector_mode_valid(struct drm_connector *connector, +static enum drm_mode_status bochs_connector_mode_valid(struct drm_connector *connector, struct drm_display_mode *mode) { struct bochs_device *bochs = diff --git a/drivers/gpu/drm/bridge/Kconfig b/drivers/gpu/drm/bridge/Kconfig index 684ac626ac53..fa2c7997e2fd 100644 --- a/drivers/gpu/drm/bridge/Kconfig +++ b/drivers/gpu/drm/bridge/Kconfig @@ -25,6 +25,16 @@ config DRM_ANALOGIX_ANX78XX the HDMI output of an application processor to MyDP or DisplayPort. +config DRM_CDNS_DSI + tristate "Cadence DPI/DSI bridge" + select DRM_KMS_HELPER + select DRM_MIPI_DSI + select DRM_PANEL_BRIDGE + depends on OF + help + Support Cadence DPI to DSI bridge. This is an internal + bridge and is meant to be directly embedded in a SoC. + config DRM_DUMB_VGA_DAC tristate "Dumb VGA DAC Bridge support" depends on OF @@ -94,6 +104,12 @@ config DRM_SII9234 It is an I2C driver, that detects connection of MHL bridge and starts encapsulation of HDMI signal. +config DRM_THINE_THC63LVD1024 + tristate "Thine THC63LVD1024 LVDS decoder bridge" + depends on OF + ---help--- + Thine THC63LVD1024 LVDS/parallel converter driver. + config DRM_TOSHIBA_TC358767 tristate "Toshiba TC358767 eDP bridge" depends on OF diff --git a/drivers/gpu/drm/bridge/Makefile b/drivers/gpu/drm/bridge/Makefile index 373eb28f31ed..35f88d48ec20 100644 --- a/drivers/gpu/drm/bridge/Makefile +++ b/drivers/gpu/drm/bridge/Makefile @@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_DRM_ANALOGIX_ANX78XX) += analogix-anx78xx.o +obj-$(CONFIG_DRM_CDNS_DSI) += cdns-dsi.o obj-$(CONFIG_DRM_DUMB_VGA_DAC) += dumb-vga-dac.o obj-$(CONFIG_DRM_LVDS_ENCODER) += lvds-encoder.o obj-$(CONFIG_DRM_MEGACHIPS_STDPXXXX_GE_B850V3_FW) += megachips-stdpxxxx-ge-b850v3-fw.o @@ -8,6 +9,7 @@ obj-$(CONFIG_DRM_PARADE_PS8622) += parade-ps8622.o obj-$(CONFIG_DRM_SIL_SII8620) += sil-sii8620.o obj-$(CONFIG_DRM_SII902X) += sii902x.o obj-$(CONFIG_DRM_SII9234) += sii9234.o +obj-$(CONFIG_DRM_THINE_THC63LVD1024) += thc63lvd1024.o obj-$(CONFIG_DRM_TOSHIBA_TC358767) += tc358767.o obj-$(CONFIG_DRM_ANALOGIX_DP) += analogix/ obj-$(CONFIG_DRM_I2C_ADV7511) += adv7511/ diff --git a/drivers/gpu/drm/bridge/adv7511/Kconfig b/drivers/gpu/drm/bridge/adv7511/Kconfig index 592b9d2ec034..944e440c4fde 100644 --- a/drivers/gpu/drm/bridge/adv7511/Kconfig +++ b/drivers/gpu/drm/bridge/adv7511/Kconfig @@ -1,5 +1,5 @@ config DRM_I2C_ADV7511 - tristate "AV7511 encoder" + tristate "ADV7511 encoder" depends on OF select DRM_KMS_HELPER select REGMAP_I2C diff --git a/drivers/gpu/drm/bridge/adv7511/adv7511.h b/drivers/gpu/drm/bridge/adv7511/adv7511.h index d034b2cb5eee..73d8ccb97742 100644 --- a/drivers/gpu/drm/bridge/adv7511/adv7511.h +++ b/drivers/gpu/drm/bridge/adv7511/adv7511.h @@ -93,6 +93,11 @@ #define ADV7511_REG_CHIP_ID_HIGH 0xf5 #define ADV7511_REG_CHIP_ID_LOW 0xf6 +/* Hardware defined default addresses for I2C register maps */ +#define ADV7511_CEC_I2C_ADDR_DEFAULT 0x3c +#define ADV7511_EDID_I2C_ADDR_DEFAULT 0x3f +#define ADV7511_PACKET_I2C_ADDR_DEFAULT 0x38 + #define ADV7511_CSC_ENABLE BIT(7) #define ADV7511_CSC_UPDATE_MODE BIT(5) @@ -321,6 +326,7 @@ enum adv7511_type { struct adv7511 { struct i2c_client *i2c_main; struct i2c_client *i2c_edid; + struct i2c_client *i2c_packet; struct i2c_client *i2c_cec; struct regmap *regmap; diff --git a/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c b/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c index efa29db5fc2b..73021b388e12 100644 --- a/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c +++ b/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c @@ -586,7 +586,7 @@ static int adv7511_get_modes(struct adv7511 *adv7511, /* Reading the EDID only works if the device is powered */ if (!adv7511->powered) { unsigned int edid_i2c_addr = - (adv7511->i2c_main->addr << 1) + 4; + (adv7511->i2c_edid->addr << 1); __adv7511_power_on(adv7511); @@ -654,7 +654,7 @@ adv7511_detect(struct adv7511 *adv7511, struct drm_connector *connector) return status; } -static int adv7511_mode_valid(struct adv7511 *adv7511, +static enum drm_mode_status adv7511_mode_valid(struct adv7511 *adv7511, struct drm_display_mode *mode) { if (mode->clock > 165000) @@ -969,10 +969,10 @@ static int adv7511_init_cec_regmap(struct adv7511 *adv) { int ret; - adv->i2c_cec = i2c_new_dummy(adv->i2c_main->adapter, - adv->i2c_main->addr - 1); + adv->i2c_cec = i2c_new_secondary_device(adv->i2c_main, "cec", + ADV7511_CEC_I2C_ADDR_DEFAULT); if (!adv->i2c_cec) - return -ENOMEM; + return -EINVAL; i2c_set_clientdata(adv->i2c_cec, adv); adv->regmap_cec = devm_regmap_init_i2c(adv->i2c_cec, @@ -1082,8 +1082,6 @@ static int adv7511_probe(struct i2c_client *i2c, const struct i2c_device_id *id) struct adv7511_link_config link_config; struct adv7511 *adv7511; struct device *dev = &i2c->dev; - unsigned int main_i2c_addr = i2c->addr << 1; - unsigned int edid_i2c_addr = main_i2c_addr + 4; unsigned int val; int ret; @@ -1129,7 +1127,7 @@ static int adv7511_probe(struct i2c_client *i2c, const struct i2c_device_id *id) } if (adv7511->gpio_pd) { - mdelay(5); + usleep_range(5000, 6000); gpiod_set_value_cansleep(adv7511->gpio_pd, 0); } @@ -1153,23 +1151,34 @@ static int adv7511_probe(struct i2c_client *i2c, const struct i2c_device_id *id) if (ret) goto uninit_regulators; - regmap_write(adv7511->regmap, ADV7511_REG_EDID_I2C_ADDR, edid_i2c_addr); - regmap_write(adv7511->regmap, ADV7511_REG_PACKET_I2C_ADDR, - main_i2c_addr - 0xa); - regmap_write(adv7511->regmap, ADV7511_REG_CEC_I2C_ADDR, - main_i2c_addr - 2); - adv7511_packet_disable(adv7511, 0xffff); - adv7511->i2c_edid = i2c_new_dummy(i2c->adapter, edid_i2c_addr >> 1); + adv7511->i2c_edid = i2c_new_secondary_device(i2c, "edid", + ADV7511_EDID_I2C_ADDR_DEFAULT); if (!adv7511->i2c_edid) { - ret = -ENOMEM; + ret = -EINVAL; goto uninit_regulators; } + regmap_write(adv7511->regmap, ADV7511_REG_EDID_I2C_ADDR, + adv7511->i2c_edid->addr << 1); + + adv7511->i2c_packet = i2c_new_secondary_device(i2c, "packet", + ADV7511_PACKET_I2C_ADDR_DEFAULT); + if (!adv7511->i2c_packet) { + ret = -EINVAL; + goto err_i2c_unregister_edid; + } + + regmap_write(adv7511->regmap, ADV7511_REG_PACKET_I2C_ADDR, + adv7511->i2c_packet->addr << 1); + ret = adv7511_init_cec_regmap(adv7511); if (ret) - goto err_i2c_unregister_edid; + goto err_i2c_unregister_packet; + + regmap_write(adv7511->regmap, ADV7511_REG_CEC_I2C_ADDR, + adv7511->i2c_cec->addr << 1); INIT_WORK(&adv7511->hpd_work, adv7511_hpd_work); @@ -1207,6 +1216,8 @@ err_unregister_cec: i2c_unregister_device(adv7511->i2c_cec); if (adv7511->cec_clk) clk_disable_unprepare(adv7511->cec_clk); +err_i2c_unregister_packet: + i2c_unregister_device(adv7511->i2c_packet); err_i2c_unregister_edid: i2c_unregister_device(adv7511->i2c_edid); uninit_regulators: @@ -1233,6 +1244,7 @@ static int adv7511_remove(struct i2c_client *i2c) cec_unregister_adapter(adv7511->cec_adap); + i2c_unregister_device(adv7511->i2c_packet); i2c_unregister_device(adv7511->i2c_edid); return 0; diff --git a/drivers/gpu/drm/bridge/analogix/analogix_dp_core.c b/drivers/gpu/drm/bridge/analogix/analogix_dp_core.c index 5c52307146c7..2bcbfadb6ac5 100644 --- a/drivers/gpu/drm/bridge/analogix/analogix_dp_core.c +++ b/drivers/gpu/drm/bridge/analogix/analogix_dp_core.c @@ -43,8 +43,10 @@ struct bridge_init { struct device_node *node; }; -static void analogix_dp_init_dp(struct analogix_dp_device *dp) +static int analogix_dp_init_dp(struct analogix_dp_device *dp) { + int ret; + analogix_dp_reset(dp); analogix_dp_swreset(dp); @@ -56,10 +58,13 @@ static void analogix_dp_init_dp(struct analogix_dp_device *dp) analogix_dp_enable_sw_function(dp); analogix_dp_config_interrupt(dp); - analogix_dp_init_analog_func(dp); + ret = analogix_dp_init_analog_func(dp); + if (ret) + return ret; analogix_dp_init_hpd(dp); analogix_dp_init_aux(dp); + return 0; } static int analogix_dp_detect_hpd(struct analogix_dp_device *dp) @@ -71,7 +76,7 @@ static int analogix_dp_detect_hpd(struct analogix_dp_device *dp) return 0; timeout_loop++; - usleep_range(10, 11); + usleep_range(1000, 1100); } /* @@ -148,87 +153,146 @@ int analogix_dp_disable_psr(struct analogix_dp_device *dp) psr_vsc.DB1 = 0; ret = drm_dp_dpcd_writeb(&dp->aux, DP_SET_POWER, DP_SET_POWER_D0); - if (ret != 1) + if (ret != 1) { dev_err(dp->dev, "Failed to set DP Power0 %d\n", ret); + return ret; + } return analogix_dp_send_psr_spd(dp, &psr_vsc, false); } EXPORT_SYMBOL_GPL(analogix_dp_disable_psr); -static bool analogix_dp_detect_sink_psr(struct analogix_dp_device *dp) +static int analogix_dp_detect_sink_psr(struct analogix_dp_device *dp) { unsigned char psr_version; + int ret; + + ret = drm_dp_dpcd_readb(&dp->aux, DP_PSR_SUPPORT, &psr_version); + if (ret != 1) { + dev_err(dp->dev, "failed to get PSR version, disable it\n"); + return ret; + } - drm_dp_dpcd_readb(&dp->aux, DP_PSR_SUPPORT, &psr_version); dev_dbg(dp->dev, "Panel PSR version : %x\n", psr_version); - return (psr_version & DP_PSR_IS_SUPPORTED) ? true : false; + dp->psr_enable = (psr_version & DP_PSR_IS_SUPPORTED) ? true : false; + + return 0; } -static void analogix_dp_enable_sink_psr(struct analogix_dp_device *dp) +static int analogix_dp_enable_sink_psr(struct analogix_dp_device *dp) { unsigned char psr_en; + int ret; /* Disable psr function */ - drm_dp_dpcd_readb(&dp->aux, DP_PSR_EN_CFG, &psr_en); + ret = drm_dp_dpcd_readb(&dp->aux, DP_PSR_EN_CFG, &psr_en); + if (ret != 1) { + dev_err(dp->dev, "failed to get psr config\n"); + goto end; + } + psr_en &= ~DP_PSR_ENABLE; - drm_dp_dpcd_writeb(&dp->aux, DP_PSR_EN_CFG, psr_en); + ret = drm_dp_dpcd_writeb(&dp->aux, DP_PSR_EN_CFG, psr_en); + if (ret != 1) { + dev_err(dp->dev, "failed to disable panel psr\n"); + goto end; + } /* Main-Link transmitter remains active during PSR active states */ psr_en = DP_PSR_MAIN_LINK_ACTIVE | DP_PSR_CRC_VERIFICATION; - drm_dp_dpcd_writeb(&dp->aux, DP_PSR_EN_CFG, psr_en); + ret = drm_dp_dpcd_writeb(&dp->aux, DP_PSR_EN_CFG, psr_en); + if (ret != 1) { + dev_err(dp->dev, "failed to set panel psr\n"); + goto end; + } /* Enable psr function */ psr_en = DP_PSR_ENABLE | DP_PSR_MAIN_LINK_ACTIVE | DP_PSR_CRC_VERIFICATION; - drm_dp_dpcd_writeb(&dp->aux, DP_PSR_EN_CFG, psr_en); + ret = drm_dp_dpcd_writeb(&dp->aux, DP_PSR_EN_CFG, psr_en); + if (ret != 1) { + dev_err(dp->dev, "failed to set panel psr\n"); + goto end; + } analogix_dp_enable_psr_crc(dp); + + return 0; +end: + dev_err(dp->dev, "enable psr fail, force to disable psr\n"); + dp->psr_enable = false; + + return ret; } -static void +static int analogix_dp_enable_rx_to_enhanced_mode(struct analogix_dp_device *dp, bool enable) { u8 data; + int ret; - drm_dp_dpcd_readb(&dp->aux, DP_LANE_COUNT_SET, &data); + ret = drm_dp_dpcd_readb(&dp->aux, DP_LANE_COUNT_SET, &data); + if (ret != 1) + return ret; if (enable) - drm_dp_dpcd_writeb(&dp->aux, DP_LANE_COUNT_SET, - DP_LANE_COUNT_ENHANCED_FRAME_EN | - DPCD_LANE_COUNT_SET(data)); + ret = drm_dp_dpcd_writeb(&dp->aux, DP_LANE_COUNT_SET, + DP_LANE_COUNT_ENHANCED_FRAME_EN | + DPCD_LANE_COUNT_SET(data)); else - drm_dp_dpcd_writeb(&dp->aux, DP_LANE_COUNT_SET, - DPCD_LANE_COUNT_SET(data)); + ret = drm_dp_dpcd_writeb(&dp->aux, DP_LANE_COUNT_SET, + DPCD_LANE_COUNT_SET(data)); + + return ret < 0 ? ret : 0; } -static int analogix_dp_is_enhanced_mode_available(struct analogix_dp_device *dp) +static int analogix_dp_is_enhanced_mode_available(struct analogix_dp_device *dp, + u8 *enhanced_mode_support) { u8 data; - int retval; + int ret; - drm_dp_dpcd_readb(&dp->aux, DP_MAX_LANE_COUNT, &data); - retval = DPCD_ENHANCED_FRAME_CAP(data); + ret = drm_dp_dpcd_readb(&dp->aux, DP_MAX_LANE_COUNT, &data); + if (ret != 1) { + *enhanced_mode_support = 0; + return ret; + } - return retval; + *enhanced_mode_support = DPCD_ENHANCED_FRAME_CAP(data); + + return 0; } -static void analogix_dp_set_enhanced_mode(struct analogix_dp_device *dp) +static int analogix_dp_set_enhanced_mode(struct analogix_dp_device *dp) { u8 data; + int ret; + + ret = analogix_dp_is_enhanced_mode_available(dp, &data); + if (ret < 0) + return ret; + + ret = analogix_dp_enable_rx_to_enhanced_mode(dp, data); + if (ret < 0) + return ret; - data = analogix_dp_is_enhanced_mode_available(dp); - analogix_dp_enable_rx_to_enhanced_mode(dp, data); analogix_dp_enable_enhanced_mode(dp, data); + + return 0; } -static void analogix_dp_training_pattern_dis(struct analogix_dp_device *dp) +static int analogix_dp_training_pattern_dis(struct analogix_dp_device *dp) { + int ret; + analogix_dp_set_training_pattern(dp, DP_NONE); - drm_dp_dpcd_writeb(&dp->aux, DP_TRAINING_PATTERN_SET, - DP_TRAINING_PATTERN_DISABLE); + ret = drm_dp_dpcd_writeb(&dp->aux, DP_TRAINING_PATTERN_SET, + DP_TRAINING_PATTERN_DISABLE); + + return ret < 0 ? ret : 0; } static void @@ -276,6 +340,12 @@ static int analogix_dp_link_start(struct analogix_dp_device *dp) retval = drm_dp_dpcd_write(&dp->aux, DP_LINK_BW_SET, buf, 2); if (retval < 0) return retval; + /* set enhanced mode if available */ + retval = analogix_dp_set_enhanced_mode(dp); + if (retval < 0) { + dev_err(dp->dev, "failed to set enhance mode\n"); + return retval; + } /* Set TX pre-emphasis to minimum */ for (lane = 0; lane < lane_count; lane++) @@ -531,7 +601,7 @@ static int analogix_dp_process_equalizer_training(struct analogix_dp_device *dp) { int lane, lane_count, retval; u32 reg; - u8 link_align, link_status[2], adjust_request[2], spread; + u8 link_align, link_status[2], adjust_request[2]; usleep_range(400, 401); @@ -560,10 +630,11 @@ static int analogix_dp_process_equalizer_training(struct analogix_dp_device *dp) if (!analogix_dp_channel_eq_ok(link_status, link_align, lane_count)) { /* traing pattern Set to Normal */ - analogix_dp_training_pattern_dis(dp); + retval = analogix_dp_training_pattern_dis(dp); + if (retval < 0) + return retval; dev_info(dp->dev, "Link Training success!\n"); - analogix_dp_get_link_bandwidth(dp, ®); dp->link_train.link_rate = reg; dev_dbg(dp->dev, "final bandwidth = %.2x\n", @@ -574,22 +645,6 @@ static int analogix_dp_process_equalizer_training(struct analogix_dp_device *dp) dev_dbg(dp->dev, "final lane count = %.2x\n", dp->link_train.lane_count); - retval = drm_dp_dpcd_readb(&dp->aux, DP_MAX_DOWNSPREAD, - &spread); - if (retval != 1) { - dev_err(dp->dev, "failed to read downspread %d\n", - retval); - dp->fast_train_support = false; - } else { - dp->fast_train_support = - (spread & DP_NO_AUX_HANDSHAKE_LINK_TRAINING) ? - true : false; - } - dev_dbg(dp->dev, "fast link training %s\n", - dp->fast_train_support ? "supported" : "unsupported"); - - /* set enhanced mode if available */ - analogix_dp_set_enhanced_mode(dp); dp->link_train.lt_state = FINISHED; return 0; @@ -793,7 +848,7 @@ static int analogix_dp_fast_link_train(struct analogix_dp_device *dp) static int analogix_dp_train_link(struct analogix_dp_device *dp) { - if (dp->fast_train_support) + if (dp->fast_train_enable) return analogix_dp_fast_link_train(dp); return analogix_dp_full_link_train(dp, dp->video_info.max_lane_count, @@ -819,11 +874,10 @@ static int analogix_dp_config_video(struct analogix_dp_device *dp) if (analogix_dp_is_slave_video_stream_clock_on(dp) == 0) break; if (timeout_loop > DP_TIMEOUT_LOOP_COUNT) { - dev_err(dp->dev, "Timeout of video streamclk ok\n"); + dev_err(dp->dev, "Timeout of slave video streamclk ok\n"); return -ETIMEDOUT; } - - usleep_range(1, 2); + usleep_range(1000, 1001); } /* Set to use the register calculated M/N video */ @@ -838,6 +892,9 @@ static int analogix_dp_config_video(struct analogix_dp_device *dp) /* Configure video slave mode */ analogix_dp_enable_video_master(dp, 0); + /* Enable video */ + analogix_dp_start_video(dp); + timeout_loop = 0; for (;;) { @@ -850,8 +907,9 @@ static int analogix_dp_config_video(struct analogix_dp_device *dp) done_count = 0; } if (timeout_loop > DP_TIMEOUT_LOOP_COUNT) { - dev_err(dp->dev, "Timeout of video streamclk ok\n"); - return -ETIMEDOUT; + dev_warn(dp->dev, + "Ignoring timeout of video streamclk ok\n"); + break; } usleep_range(1000, 1001); @@ -860,24 +918,32 @@ static int analogix_dp_config_video(struct analogix_dp_device *dp) return 0; } -static void analogix_dp_enable_scramble(struct analogix_dp_device *dp, - bool enable) +static int analogix_dp_enable_scramble(struct analogix_dp_device *dp, + bool enable) { u8 data; + int ret; if (enable) { analogix_dp_enable_scrambling(dp); - drm_dp_dpcd_readb(&dp->aux, DP_TRAINING_PATTERN_SET, &data); - drm_dp_dpcd_writeb(&dp->aux, DP_TRAINING_PATTERN_SET, + ret = drm_dp_dpcd_readb(&dp->aux, DP_TRAINING_PATTERN_SET, + &data); + if (ret != 1) + return ret; + ret = drm_dp_dpcd_writeb(&dp->aux, DP_TRAINING_PATTERN_SET, (u8)(data & ~DP_LINK_SCRAMBLING_DISABLE)); } else { analogix_dp_disable_scrambling(dp); - drm_dp_dpcd_readb(&dp->aux, DP_TRAINING_PATTERN_SET, &data); - drm_dp_dpcd_writeb(&dp->aux, DP_TRAINING_PATTERN_SET, + ret = drm_dp_dpcd_readb(&dp->aux, DP_TRAINING_PATTERN_SET, + &data); + if (ret != 1) + return ret; + ret = drm_dp_dpcd_writeb(&dp->aux, DP_TRAINING_PATTERN_SET, (u8)(data | DP_LINK_SCRAMBLING_DISABLE)); } + return ret < 0 ? ret : 0; } static irqreturn_t analogix_dp_hardirq(int irq, void *arg) @@ -916,7 +982,23 @@ static irqreturn_t analogix_dp_irq_thread(int irq, void *arg) return IRQ_HANDLED; } -static void analogix_dp_commit(struct analogix_dp_device *dp) +static int analogix_dp_fast_link_train_detection(struct analogix_dp_device *dp) +{ + int ret; + u8 spread; + + ret = drm_dp_dpcd_readb(&dp->aux, DP_MAX_DOWNSPREAD, &spread); + if (ret != 1) { + dev_err(dp->dev, "failed to read downspread %d\n", ret); + return ret; + } + dp->fast_train_enable = !!(spread & DP_NO_AUX_HANDSHAKE_LINK_TRAINING); + dev_dbg(dp->dev, "fast link training %s\n", + dp->fast_train_enable ? "supported" : "unsupported"); + return 0; +} + +static int analogix_dp_commit(struct analogix_dp_device *dp) { int ret; @@ -926,34 +1008,50 @@ static void analogix_dp_commit(struct analogix_dp_device *dp) DRM_ERROR("failed to disable the panel\n"); } - ret = readx_poll_timeout(analogix_dp_train_link, dp, ret, !ret, 100, - DP_TIMEOUT_TRAINING_US * 5); + ret = analogix_dp_train_link(dp); if (ret) { dev_err(dp->dev, "unable to do link train, ret=%d\n", ret); - return; + return ret; } - analogix_dp_enable_scramble(dp, 1); - analogix_dp_enable_rx_to_enhanced_mode(dp, 1); - analogix_dp_enable_enhanced_mode(dp, 1); + ret = analogix_dp_enable_scramble(dp, 1); + if (ret < 0) { + dev_err(dp->dev, "can not enable scramble\n"); + return ret; + } analogix_dp_init_video(dp); ret = analogix_dp_config_video(dp); - if (ret) + if (ret) { dev_err(dp->dev, "unable to config video\n"); + return ret; + } /* Safe to enable the panel now */ if (dp->plat_data->panel) { - if (drm_panel_enable(dp->plat_data->panel)) + ret = drm_panel_enable(dp->plat_data->panel); + if (ret) { DRM_ERROR("failed to enable the panel\n"); + return ret; + } } - /* Enable video */ - analogix_dp_start_video(dp); + ret = analogix_dp_detect_sink_psr(dp); + if (ret) + return ret; - dp->psr_enable = analogix_dp_detect_sink_psr(dp); - if (dp->psr_enable) - analogix_dp_enable_sink_psr(dp); + if (dp->psr_enable) { + ret = analogix_dp_enable_sink_psr(dp); + if (ret) + return ret; + } + + /* Check whether panel supports fast training */ + ret = analogix_dp_fast_link_train_detection(dp); + if (ret) + dp->psr_enable = false; + + return ret; } /* @@ -1150,24 +1248,80 @@ static void analogix_dp_bridge_pre_enable(struct drm_bridge *bridge) DRM_ERROR("failed to setup the panel ret = %d\n", ret); } -static void analogix_dp_bridge_enable(struct drm_bridge *bridge) +static int analogix_dp_set_bridge(struct analogix_dp_device *dp) { - struct analogix_dp_device *dp = bridge->driver_private; - - if (dp->dpms_mode == DRM_MODE_DPMS_ON) - return; + int ret; pm_runtime_get_sync(dp->dev); - if (dp->plat_data->power_on) - dp->plat_data->power_on(dp->plat_data); + ret = clk_prepare_enable(dp->clock); + if (ret < 0) { + DRM_ERROR("Failed to prepare_enable the clock clk [%d]\n", ret); + goto out_dp_clk_pre; + } + + if (dp->plat_data->power_on_start) + dp->plat_data->power_on_start(dp->plat_data); phy_power_on(dp->phy); - analogix_dp_init_dp(dp); + + ret = analogix_dp_init_dp(dp); + if (ret) + goto out_dp_init; + + /* + * According to DP spec v1.3 chap 3.5.1.2 Link Training, + * We should first make sure the HPD signal is asserted high by device + * when we want to establish a link with it. + */ + ret = analogix_dp_detect_hpd(dp); + if (ret) { + DRM_ERROR("failed to get hpd single ret = %d\n", ret); + goto out_dp_init; + } + + ret = analogix_dp_commit(dp); + if (ret) { + DRM_ERROR("dp commit error, ret = %d\n", ret); + goto out_dp_init; + } + + if (dp->plat_data->power_on_end) + dp->plat_data->power_on_end(dp->plat_data); + enable_irq(dp->irq); - analogix_dp_commit(dp); + return 0; - dp->dpms_mode = DRM_MODE_DPMS_ON; +out_dp_init: + phy_power_off(dp->phy); + if (dp->plat_data->power_off) + dp->plat_data->power_off(dp->plat_data); + clk_disable_unprepare(dp->clock); +out_dp_clk_pre: + pm_runtime_put_sync(dp->dev); + + return ret; +} + +static void analogix_dp_bridge_enable(struct drm_bridge *bridge) +{ + struct analogix_dp_device *dp = bridge->driver_private; + int timeout_loop = 0; + + if (dp->dpms_mode == DRM_MODE_DPMS_ON) + return; + + while (timeout_loop < MAX_PLL_LOCK_LOOP) { + if (analogix_dp_set_bridge(dp) == 0) { + dp->dpms_mode = DRM_MODE_DPMS_ON; + return; + } + dev_err(dp->dev, "failed to set bridge, retry: %d\n", + timeout_loop); + timeout_loop++; + usleep_range(10, 11); + } + dev_err(dp->dev, "too many times retry set bridge, give it up\n"); } static void analogix_dp_bridge_disable(struct drm_bridge *bridge) @@ -1186,11 +1340,15 @@ static void analogix_dp_bridge_disable(struct drm_bridge *bridge) } disable_irq(dp->irq); - phy_power_off(dp->phy); if (dp->plat_data->power_off) dp->plat_data->power_off(dp->plat_data); + analogix_dp_set_analog_power_down(dp, POWER_ALL, 1); + phy_power_off(dp->phy); + + clk_disable_unprepare(dp->clock); + pm_runtime_put_sync(dp->dev); ret = analogix_dp_prepare_panel(dp, false, true); @@ -1198,6 +1356,7 @@ static void analogix_dp_bridge_disable(struct drm_bridge *bridge) DRM_ERROR("failed to setup the panel ret = %d\n", ret); dp->psr_enable = false; + dp->fast_train_enable = false; dp->dpms_mode = DRM_MODE_DPMS_OFF; } diff --git a/drivers/gpu/drm/bridge/analogix/analogix_dp_core.h b/drivers/gpu/drm/bridge/analogix/analogix_dp_core.h index 6a96ef7e6934..769255dc6e99 100644 --- a/drivers/gpu/drm/bridge/analogix/analogix_dp_core.h +++ b/drivers/gpu/drm/bridge/analogix/analogix_dp_core.h @@ -19,6 +19,7 @@ #define DP_TIMEOUT_LOOP_COUNT 100 #define MAX_CR_LOOP 5 #define MAX_EQ_LOOP 5 +#define MAX_PLL_LOCK_LOOP 5 /* Training takes 22ms if AUX channel comm fails. Use this as retry interval */ #define DP_TIMEOUT_TRAINING_US 22000 @@ -173,7 +174,7 @@ struct analogix_dp_device { int hpd_gpio; bool force_hpd; bool psr_enable; - bool fast_train_support; + bool fast_train_enable; struct mutex panel_lock; bool panel_is_modeset; @@ -197,7 +198,7 @@ void analogix_dp_set_pll_power_down(struct analogix_dp_device *dp, bool enable); void analogix_dp_set_analog_power_down(struct analogix_dp_device *dp, enum analog_power_block block, bool enable); -void analogix_dp_init_analog_func(struct analogix_dp_device *dp); +int analogix_dp_init_analog_func(struct analogix_dp_device *dp); void analogix_dp_init_hpd(struct analogix_dp_device *dp); void analogix_dp_force_hpd(struct analogix_dp_device *dp); enum dp_irq_type analogix_dp_get_irq_type(struct analogix_dp_device *dp); diff --git a/drivers/gpu/drm/bridge/analogix/analogix_dp_reg.c b/drivers/gpu/drm/bridge/analogix/analogix_dp_reg.c index 9df2f3ef000c..a5f2763d72e4 100644 --- a/drivers/gpu/drm/bridge/analogix/analogix_dp_reg.c +++ b/drivers/gpu/drm/bridge/analogix/analogix_dp_reg.c @@ -126,9 +126,14 @@ void analogix_dp_reset(struct analogix_dp_device *dp) analogix_dp_stop_video(dp); analogix_dp_enable_video_mute(dp, 0); - reg = MASTER_VID_FUNC_EN_N | SLAVE_VID_FUNC_EN_N | - AUD_FIFO_FUNC_EN_N | AUD_FUNC_EN_N | - HDCP_FUNC_EN_N | SW_FUNC_EN_N; + if (dp->plat_data && is_rockchip(dp->plat_data->dev_type)) + reg = RK_VID_CAP_FUNC_EN_N | RK_VID_FIFO_FUNC_EN_N | + SW_FUNC_EN_N; + else + reg = MASTER_VID_FUNC_EN_N | SLAVE_VID_FUNC_EN_N | + AUD_FIFO_FUNC_EN_N | AUD_FUNC_EN_N | + HDCP_FUNC_EN_N | SW_FUNC_EN_N; + writel(reg, dp->reg_base + ANALOGIX_DP_FUNC_EN_1); reg = SSC_FUNC_EN_N | AUX_FUNC_EN_N | @@ -230,16 +235,20 @@ enum pll_status analogix_dp_get_pll_lock_status(struct analogix_dp_device *dp) void analogix_dp_set_pll_power_down(struct analogix_dp_device *dp, bool enable) { u32 reg; + u32 mask = DP_PLL_PD; + u32 pd_addr = ANALOGIX_DP_PLL_CTL; - if (enable) { - reg = readl(dp->reg_base + ANALOGIX_DP_PLL_CTL); - reg |= DP_PLL_PD; - writel(reg, dp->reg_base + ANALOGIX_DP_PLL_CTL); - } else { - reg = readl(dp->reg_base + ANALOGIX_DP_PLL_CTL); - reg &= ~DP_PLL_PD; - writel(reg, dp->reg_base + ANALOGIX_DP_PLL_CTL); + if (dp->plat_data && is_rockchip(dp->plat_data->dev_type)) { + pd_addr = ANALOGIX_DP_PD; + mask = RK_PLL_PD; } + + reg = readl(dp->reg_base + pd_addr); + if (enable) + reg |= mask; + else + reg &= ~mask; + writel(reg, dp->reg_base + pd_addr); } void analogix_dp_set_analog_power_down(struct analogix_dp_device *dp, @@ -248,83 +257,98 @@ void analogix_dp_set_analog_power_down(struct analogix_dp_device *dp, { u32 reg; u32 phy_pd_addr = ANALOGIX_DP_PHY_PD; + u32 mask; if (dp->plat_data && is_rockchip(dp->plat_data->dev_type)) phy_pd_addr = ANALOGIX_DP_PD; switch (block) { case AUX_BLOCK: - if (enable) { - reg = readl(dp->reg_base + phy_pd_addr); - reg |= AUX_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } else { - reg = readl(dp->reg_base + phy_pd_addr); - reg &= ~AUX_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } + if (dp->plat_data && is_rockchip(dp->plat_data->dev_type)) + mask = RK_AUX_PD; + else + mask = AUX_PD; + + reg = readl(dp->reg_base + phy_pd_addr); + if (enable) + reg |= mask; + else + reg &= ~mask; + writel(reg, dp->reg_base + phy_pd_addr); break; case CH0_BLOCK: - if (enable) { - reg = readl(dp->reg_base + phy_pd_addr); - reg |= CH0_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } else { - reg = readl(dp->reg_base + phy_pd_addr); - reg &= ~CH0_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } + mask = CH0_PD; + reg = readl(dp->reg_base + phy_pd_addr); + + if (enable) + reg |= mask; + else + reg &= ~mask; + writel(reg, dp->reg_base + phy_pd_addr); break; case CH1_BLOCK: - if (enable) { - reg = readl(dp->reg_base + phy_pd_addr); - reg |= CH1_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } else { - reg = readl(dp->reg_base + phy_pd_addr); - reg &= ~CH1_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } + mask = CH1_PD; + reg = readl(dp->reg_base + phy_pd_addr); + + if (enable) + reg |= mask; + else + reg &= ~mask; + writel(reg, dp->reg_base + phy_pd_addr); break; case CH2_BLOCK: - if (enable) { - reg = readl(dp->reg_base + phy_pd_addr); - reg |= CH2_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } else { - reg = readl(dp->reg_base + phy_pd_addr); - reg &= ~CH2_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } + mask = CH2_PD; + reg = readl(dp->reg_base + phy_pd_addr); + + if (enable) + reg |= mask; + else + reg &= ~mask; + writel(reg, dp->reg_base + phy_pd_addr); break; case CH3_BLOCK: - if (enable) { - reg = readl(dp->reg_base + phy_pd_addr); - reg |= CH3_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } else { - reg = readl(dp->reg_base + phy_pd_addr); - reg &= ~CH3_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } + mask = CH3_PD; + reg = readl(dp->reg_base + phy_pd_addr); + + if (enable) + reg |= mask; + else + reg &= ~mask; + writel(reg, dp->reg_base + phy_pd_addr); break; case ANALOG_TOTAL: - if (enable) { - reg = readl(dp->reg_base + phy_pd_addr); - reg |= DP_PHY_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } else { - reg = readl(dp->reg_base + phy_pd_addr); - reg &= ~DP_PHY_PD; - writel(reg, dp->reg_base + phy_pd_addr); - } + /* + * There is no bit named DP_PHY_PD, so We used DP_INC_BG + * to power off everything instead of DP_PHY_PD in + * Rockchip + */ + if (dp->plat_data && is_rockchip(dp->plat_data->dev_type)) + mask = DP_INC_BG; + else + mask = DP_PHY_PD; + + reg = readl(dp->reg_base + phy_pd_addr); + if (enable) + reg |= mask; + else + reg &= ~mask; + + writel(reg, dp->reg_base + phy_pd_addr); + if (dp->plat_data && is_rockchip(dp->plat_data->dev_type)) + usleep_range(10, 15); break; case POWER_ALL: if (enable) { - reg = DP_PHY_PD | AUX_PD | CH3_PD | CH2_PD | - CH1_PD | CH0_PD; + reg = DP_ALL_PD; writel(reg, dp->reg_base + phy_pd_addr); } else { + reg = DP_ALL_PD; + writel(reg, dp->reg_base + phy_pd_addr); + usleep_range(10, 15); + reg &= ~DP_INC_BG; + writel(reg, dp->reg_base + phy_pd_addr); + usleep_range(10, 15); + writel(0x00, dp->reg_base + phy_pd_addr); } break; @@ -333,7 +357,7 @@ void analogix_dp_set_analog_power_down(struct analogix_dp_device *dp, } } -void analogix_dp_init_analog_func(struct analogix_dp_device *dp) +int analogix_dp_init_analog_func(struct analogix_dp_device *dp) { u32 reg; int timeout_loop = 0; @@ -355,7 +379,7 @@ void analogix_dp_init_analog_func(struct analogix_dp_device *dp) timeout_loop++; if (DP_TIMEOUT_LOOP_COUNT < timeout_loop) { dev_err(dp->dev, "failed to get pll lock status\n"); - return; + return -ETIMEDOUT; } usleep_range(10, 20); } @@ -366,6 +390,7 @@ void analogix_dp_init_analog_func(struct analogix_dp_device *dp) reg &= ~(SERDES_FIFO_FUNC_EN_N | LS_CLK_DOMAIN_FUNC_EN_N | AUX_FUNC_EN_N); writel(reg, dp->reg_base + ANALOGIX_DP_FUNC_EN_2); + return 0; } void analogix_dp_clear_hotplug_interrupts(struct analogix_dp_device *dp) @@ -450,17 +475,22 @@ void analogix_dp_init_aux(struct analogix_dp_device *dp) reg = RPLY_RECEIV | AUX_ERR; writel(reg, dp->reg_base + ANALOGIX_DP_INT_STA); + analogix_dp_set_analog_power_down(dp, AUX_BLOCK, true); + usleep_range(10, 11); + analogix_dp_set_analog_power_down(dp, AUX_BLOCK, false); + analogix_dp_reset_aux(dp); - /* Disable AUX transaction H/W retry */ + /* AUX_BIT_PERIOD_EXPECTED_DELAY doesn't apply to Rockchip IP */ if (dp->plat_data && is_rockchip(dp->plat_data->dev_type)) - reg = AUX_BIT_PERIOD_EXPECTED_DELAY(0) | - AUX_HW_RETRY_COUNT_SEL(3) | - AUX_HW_RETRY_INTERVAL_600_MICROSECONDS; + reg = 0; else - reg = AUX_BIT_PERIOD_EXPECTED_DELAY(3) | - AUX_HW_RETRY_COUNT_SEL(0) | - AUX_HW_RETRY_INTERVAL_600_MICROSECONDS; + reg = AUX_BIT_PERIOD_EXPECTED_DELAY(3); + + /* Disable AUX transaction H/W retry */ + reg |= AUX_HW_RETRY_COUNT_SEL(0) | + AUX_HW_RETRY_INTERVAL_600_MICROSECONDS; + writel(reg, dp->reg_base + ANALOGIX_DP_AUX_HW_RETRY_CTL); /* Receive AUX Channel DEFER commands equal to DEFFER_COUNT*64 */ @@ -947,8 +977,12 @@ void analogix_dp_config_video_slave_mode(struct analogix_dp_device *dp) u32 reg; reg = readl(dp->reg_base + ANALOGIX_DP_FUNC_EN_1); - reg &= ~(MASTER_VID_FUNC_EN_N | SLAVE_VID_FUNC_EN_N); - reg |= MASTER_VID_FUNC_EN_N; + if (dp->plat_data && is_rockchip(dp->plat_data->dev_type)) { + reg &= ~(RK_VID_CAP_FUNC_EN_N | RK_VID_FIFO_FUNC_EN_N); + } else { + reg &= ~(MASTER_VID_FUNC_EN_N | SLAVE_VID_FUNC_EN_N); + reg |= MASTER_VID_FUNC_EN_N; + } writel(reg, dp->reg_base + ANALOGIX_DP_FUNC_EN_1); reg = readl(dp->reg_base + ANALOGIX_DP_VIDEO_CTL_10); @@ -1072,10 +1106,11 @@ ssize_t analogix_dp_transfer(struct analogix_dp_device *dp, struct drm_dp_aux_msg *msg) { u32 reg; + u32 status_reg; u8 *buffer = msg->buffer; - int timeout_loop = 0; unsigned int i; int num_transferred = 0; + int ret; /* Buffer size of AUX CH is 16 bytes */ if (WARN_ON(msg->size > 16)) @@ -1139,17 +1174,20 @@ ssize_t analogix_dp_transfer(struct analogix_dp_device *dp, writel(reg, dp->reg_base + ANALOGIX_DP_AUX_CH_CTL_2); - /* Is AUX CH command reply received? */ + ret = readx_poll_timeout(readl, dp->reg_base + ANALOGIX_DP_AUX_CH_CTL_2, + reg, !(reg & AUX_EN), 25, 500 * 1000); + if (ret) { + dev_err(dp->dev, "AUX CH enable timeout!\n"); + goto aux_error; + } + /* TODO: Wait for an interrupt instead of looping? */ - reg = readl(dp->reg_base + ANALOGIX_DP_INT_STA); - while (!(reg & RPLY_RECEIV)) { - timeout_loop++; - if (timeout_loop > DP_TIMEOUT_LOOP_COUNT) { - dev_err(dp->dev, "AUX CH command reply failed!\n"); - return -ETIMEDOUT; - } - reg = readl(dp->reg_base + ANALOGIX_DP_INT_STA); - usleep_range(10, 11); + /* Is AUX CH command reply received? */ + ret = readx_poll_timeout(readl, dp->reg_base + ANALOGIX_DP_INT_STA, + reg, reg & RPLY_RECEIV, 10, 20 * 1000); + if (ret) { + dev_err(dp->dev, "AUX CH cmd reply timeout!\n"); + goto aux_error; } /* Clear interrupt source for AUX CH command reply */ @@ -1157,17 +1195,13 @@ ssize_t analogix_dp_transfer(struct analogix_dp_device *dp, /* Clear interrupt source for AUX CH access error */ reg = readl(dp->reg_base + ANALOGIX_DP_INT_STA); - if (reg & AUX_ERR) { + status_reg = readl(dp->reg_base + ANALOGIX_DP_AUX_CH_STA); + if ((reg & AUX_ERR) || (status_reg & AUX_STATUS_MASK)) { writel(AUX_ERR, dp->reg_base + ANALOGIX_DP_INT_STA); - return -EREMOTEIO; - } - /* Check AUX CH error access status */ - reg = readl(dp->reg_base + ANALOGIX_DP_AUX_CH_STA); - if ((reg & AUX_STATUS_MASK)) { - dev_err(dp->dev, "AUX CH error happened: %d\n\n", - reg & AUX_STATUS_MASK); - return -EREMOTEIO; + dev_warn(dp->dev, "AUX CH error happened: %#x (%d)\n", + status_reg & AUX_STATUS_MASK, !!(reg & AUX_ERR)); + goto aux_error; } if (msg->request & DP_AUX_I2C_READ) { @@ -1193,4 +1227,10 @@ ssize_t analogix_dp_transfer(struct analogix_dp_device *dp, msg->reply = DP_AUX_NATIVE_REPLY_ACK; return num_transferred > 0 ? num_transferred : -EBUSY; + +aux_error: + /* if aux err happen, reset aux */ + analogix_dp_init_aux(dp); + + return -EREMOTEIO; } diff --git a/drivers/gpu/drm/bridge/analogix/analogix_dp_reg.h b/drivers/gpu/drm/bridge/analogix/analogix_dp_reg.h index 40200c652533..0cf27c731727 100644 --- a/drivers/gpu/drm/bridge/analogix/analogix_dp_reg.h +++ b/drivers/gpu/drm/bridge/analogix/analogix_dp_reg.h @@ -127,7 +127,9 @@ /* ANALOGIX_DP_FUNC_EN_1 */ #define MASTER_VID_FUNC_EN_N (0x1 << 7) +#define RK_VID_CAP_FUNC_EN_N (0x1 << 6) #define SLAVE_VID_FUNC_EN_N (0x1 << 5) +#define RK_VID_FIFO_FUNC_EN_N (0x1 << 5) #define AUD_FIFO_FUNC_EN_N (0x1 << 4) #define AUD_FUNC_EN_N (0x1 << 3) #define HDCP_FUNC_EN_N (0x1 << 2) @@ -342,12 +344,17 @@ #define DP_PLL_REF_BIT_1_2500V (0x7 << 0) /* ANALOGIX_DP_PHY_PD */ +#define DP_INC_BG (0x1 << 7) +#define DP_EXP_BG (0x1 << 6) #define DP_PHY_PD (0x1 << 5) +#define RK_AUX_PD (0x1 << 5) #define AUX_PD (0x1 << 4) +#define RK_PLL_PD (0x1 << 4) #define CH3_PD (0x1 << 3) #define CH2_PD (0x1 << 2) #define CH1_PD (0x1 << 1) #define CH0_PD (0x1 << 0) +#define DP_ALL_PD (0xff) /* ANALOGIX_DP_PHY_TEST */ #define MACRO_RST (0x1 << 5) diff --git a/drivers/gpu/drm/bridge/cdns-dsi.c b/drivers/gpu/drm/bridge/cdns-dsi.c new file mode 100644 index 000000000000..c255fc3e1be5 --- /dev/null +++ b/drivers/gpu/drm/bridge/cdns-dsi.c @@ -0,0 +1,1623 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright: 2017 Cadence Design Systems, Inc. + * + * Author: Boris Brezillon + */ + +#include +#include +#include +#include +#include +#include